2021-02-10T04:10:24.908 INFO:root:teuthology version: 1.1.0-14fad317 2021-02-10T04:10:24.909 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description rados:cephadm:thrash/{0-distro/rhel_latest 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} --archive /home/teuthworker/archive/sage-2021-02-09_22:53:38-rados:cephadm:thrash-wip-sage2-testing-2021-02-09-1332-distro-basic-smithi/5872147 --name sage-2021-02-09_22:53:38-rados:cephadm:thrash-wip-sage2-testing-2021-02-09-1332-distro-basic-smithi --owner scheduled_sage@teuthology -- /home/teuthworker/archive/sage-2021-02-09_22:53:38-rados:cephadm:thrash-wip-sage2-testing-2021-02-09-1332-distro-basic-smithi/5872147/orig.config.yaml 2021-02-10T04:10:24.949 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-02-10T04:10:25.008 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sage-2021-02-09_22:53:38-rados:cephadm:thrash-wip-sage2-testing-2021-02-09-1332-distro-basic-smithi/5872147 branch: wip-sage2-testing-2021-02-09-1332 description: rados:cephadm:thrash/{0-distro/rhel_latest 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} email: null first_in_suite: false job_id: '5872147' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: sage-2021-02-09_22:53:38-rados:cephadm:thrash-wip-sage2-testing-2021-02-09-1332-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: rhel os_version: '8.3' overrides: admin_socket: branch: wip-sage2-testing-2021-02-09-1332 ceph: conf: global: ms bind msgr1: false ms bind msgr2: true ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: c1ea624123d412aff8b9d1430e36cb45fcab76b8 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 cephadm: cephadm_mode: root install: ceph: sha1: c1ea624123d412aff8b9d1430e36cb45fcab76b8 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-sage2-testing-2021-02-09-1332 sha1: c1ea624123d412aff8b9d1430e36cb45fcab76b8 owner: scheduled_sage@teuthology priority: 71 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.realm.zone.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a sha1: c1ea624123d412aff8b9d1430e36cb45fcab76b8 sleep_before_teardown: 0 suite: rados:cephadm:thrash suite_branch: wip-sage2-testing-2021-02-09-1332 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_c1ea624123d412aff8b9d1430e36cb45fcab76b8/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: c1ea624123d412aff8b9d1430e36cb45fcab76b8 targets: smithi085.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDrnOZQw1AEnAwfEoX7WvkBYYg8ksaO7mbrQ/0TNnZ7h5zBnFNlX7FN/PPa9tLdwx/AThGqTtvEgArohk6yKS9mQhUwuKtB9vs4YmyXTs/Qv5HzXC7DfvwzPFM3oaFmiC38dnUgSg88477YRMzKQ5wqzJzFqcIDJ+rJLiqSR/9yup6kPAE5sJYlGI6fc1Qcf2ZrVLeAr6LE6dLJDh3t1nx5yTwKcieIPlwH5kzUfIQfUYi44kIUPCYH1ObWkm10DR2B/ppDaifphfLMwP1adj+ORjkAhxOw64RMauSO6Blwsi0kM+8yDHJqOmGmUY2aUGTCOwlTvqS0tr8tpyjh8kczWsu4F//jNplFUJlc9DGo5Pm55RGQ9x4uPbVnzvECF8wKBw/BM7quRCHTjucPa4L8RMQ0Mgt7asY9qnNbXAqf1JHSBP5KDbauk7tSaGw8XSOZB84Zdkh5x94tyXOIjtbSQdnXxe86vHUl5wNUJBMKYvMm+mAR7O0XY/vtBghws10= smithi164.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDrIq0FOPjl5280XSkDsqea8VBA3/+KpXM1ppsO7GM8Pp+9Njo/+dBqQxBHqzy0VvTzjIy5PmIcXi9W89OfoLff4/8rtRamlhmLMa/UR6k+qoVZA/CuWymVnYJUbv0baOr7CD5ulP03KsAghcG567Wvya2JxbCOiaYx7iTuhDMrd4X9aTxxtQ826qIpfRRkUo7gz1jW0LYowwWQ5ukW3egVYSWXaGWYSUoF3wMPQp/GdDgC6piUH/qve65+7rt+fEHudadIWgnXD3GZIX+FH/ysFQ3AmTCGm8EjVXSsDC63Km/LtnVkxEhw6yaFuyC7kqwb9yKGRWsd7Rq3PW2qy+V7WSL+41F6QcI0NKGrLUnnUfpnXoPqfUC14QKVstqSNiKymOipiSnE1M1e71muKt/7T34pv16NSKlR2kYuWyy/gpVzEUb0uqhVbPKieisrYRD8e4cwKqKQ4o8XbvrAxQ45MaCZatkB8NfjYdHlsWTGZEX759oCbpoaSJzahcBS4EU= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - rados: clients: - client.0 objects: 50 op_weights: copy_from: 50 delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology_branch: master timestamp: 2021-02-09_22:53:38 tube: smithi user: sage verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.29888 2021-02-10T04:10:25.008 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_c1ea624123d412aff8b9d1430e36cb45fcab76b8/qa; will attempt to use it 2021-02-10T04:10:25.019 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_c1ea624123d412aff8b9d1430e36cb45fcab76b8/qa/tasks 2021-02-10T04:10:25.019 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-02-10T04:10:25.021 INFO:teuthology.task.internal:Checking packages... 2021-02-10T04:10:25.085 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'basic' and ceph hash 'c1ea624123d412aff8b9d1430e36cb45fcab76b8' 2021-02-10T04:10:25.085 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-02-10T04:10:25.085 INFO:teuthology.packaging:ref: None 2021-02-10T04:10:25.086 INFO:teuthology.packaging:tag: None 2021-02-10T04:10:25.086 INFO:teuthology.packaging:branch: wip-sage2-testing-2021-02-09-1332 2021-02-10T04:10:25.086 INFO:teuthology.packaging:sha1: c1ea624123d412aff8b9d1430e36cb45fcab76b8 2021-02-10T04:10:25.086 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-sage2-testing-2021-02-09-1332 2021-02-10T04:10:25.264 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-681.gc1ea6241 2021-02-10T04:10:25.264 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-02-10T04:10:25.274 INFO:teuthology.task.internal:no buildpackages task found 2021-02-10T04:10:25.274 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-02-10T04:10:25.287 INFO:teuthology.task.internal:Saving configuration 2021-02-10T04:10:25.301 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-02-10T04:10:25.311 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-10T04:10:25.331 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-02-10 04:05:05.485867', 'description': '/home/teuthworker/archive/sage-2021-02-09_22:53:38-rados:cephadm:thrash-wip-sage2-testing-2021-02-09-1332-distro-basic-smithi/5872147', 'locked_by': 'scheduled_sage@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC33MZXt649Ajz15FzA6a5aRks1Nm5mM860S7f0aPK+R9Zk7EjQuVKWjuQ0hetArsxYGtcJx3WER75ztwso0wNc=', 'os_version': '8.3', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi085.front.sepia.ceph.com'} 2021-02-10T04:10:25.349 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-02-10 04:05:05.485849', 'description': '/home/teuthworker/archive/sage-2021-02-09_22:53:38-rados:cephadm:thrash-wip-sage2-testing-2021-02-09-1332-distro-basic-smithi/5872147', 'locked_by': 'scheduled_sage@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMR5sdYUGgaF8i2d1TwByaoz4uaMnpNKuPuDnXt+TYda80THkymLhtNpRyu9NtX6z93YuXNh7AM0QX+d5XYlFgk=', 'os_version': '8.3', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi164.front.sepia.ceph.com'} 2021-02-10T04:10:25.349 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-02-10T04:10:25.360 INFO:teuthology.task.internal:roles: ubuntu@smithi085.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.realm.zone.a', 'node-exporter.a', 'alertmanager.a'] 2021-02-10T04:10:25.361 INFO:teuthology.task.internal:roles: ubuntu@smithi164.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2021-02-10T04:10:25.361 INFO:teuthology.run_tasks:Running task console_log... 2021-02-10T04:10:25.434 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f64911d1ae8>, signals=[15]) 2021-02-10T04:10:25.434 INFO:teuthology.run_tasks:Running task internal.connect... 2021-02-10T04:10:25.444 INFO:teuthology.task.internal:Opening connections... 2021-02-10T04:10:25.444 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi085.front.sepia.ceph.com 2021-02-10T04:10:25.447 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi085.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-10T04:10:25.523 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi164.front.sepia.ceph.com 2021-02-10T04:10:25.525 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-10T04:10:25.596 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-02-10T04:10:25.603 DEBUG:teuthology.orchestra.run.smithi085:> uname -m 2021-02-10T04:10:25.648 INFO:teuthology.orchestra.run.smithi085.stdout:x86_64 2021-02-10T04:10:25.648 DEBUG:teuthology.orchestra.run.smithi085:> cat /etc/os-release 2021-02-10T04:10:25.701 INFO:teuthology.orchestra.run.smithi085.stdout:NAME="Red Hat Enterprise Linux" 2021-02-10T04:10:25.702 INFO:teuthology.orchestra.run.smithi085.stdout:VERSION="8.3 (Ootpa)" 2021-02-10T04:10:25.702 INFO:teuthology.orchestra.run.smithi085.stdout:ID="rhel" 2021-02-10T04:10:25.702 INFO:teuthology.orchestra.run.smithi085.stdout:ID_LIKE="fedora" 2021-02-10T04:10:25.702 INFO:teuthology.orchestra.run.smithi085.stdout:VERSION_ID="8.3" 2021-02-10T04:10:25.702 INFO:teuthology.orchestra.run.smithi085.stdout:PLATFORM_ID="platform:el8" 2021-02-10T04:10:25.702 INFO:teuthology.orchestra.run.smithi085.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-02-10T04:10:25.702 INFO:teuthology.orchestra.run.smithi085.stdout:ANSI_COLOR="0;31" 2021-02-10T04:10:25.702 INFO:teuthology.orchestra.run.smithi085.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-02-10T04:10:25.702 INFO:teuthology.orchestra.run.smithi085.stdout:HOME_URL="https://www.redhat.com/" 2021-02-10T04:10:25.702 INFO:teuthology.orchestra.run.smithi085.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-02-10T04:10:25.702 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:10:25.702 INFO:teuthology.orchestra.run.smithi085.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-02-10T04:10:25.703 INFO:teuthology.orchestra.run.smithi085.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-02-10T04:10:25.703 INFO:teuthology.orchestra.run.smithi085.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-02-10T04:10:25.703 INFO:teuthology.orchestra.run.smithi085.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-02-10T04:10:25.704 INFO:teuthology.lock.ops:Updating smithi085.front.sepia.ceph.com on lock server 2021-02-10T04:10:25.721 DEBUG:teuthology.orchestra.run.smithi164:> uname -m 2021-02-10T04:10:25.735 INFO:teuthology.orchestra.run.smithi164.stdout:x86_64 2021-02-10T04:10:25.735 DEBUG:teuthology.orchestra.run.smithi164:> cat /etc/os-release 2021-02-10T04:10:25.790 INFO:teuthology.orchestra.run.smithi164.stdout:NAME="Red Hat Enterprise Linux" 2021-02-10T04:10:25.790 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION="8.3 (Ootpa)" 2021-02-10T04:10:25.791 INFO:teuthology.orchestra.run.smithi164.stdout:ID="rhel" 2021-02-10T04:10:25.791 INFO:teuthology.orchestra.run.smithi164.stdout:ID_LIKE="fedora" 2021-02-10T04:10:25.791 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_ID="8.3" 2021-02-10T04:10:25.791 INFO:teuthology.orchestra.run.smithi164.stdout:PLATFORM_ID="platform:el8" 2021-02-10T04:10:25.791 INFO:teuthology.orchestra.run.smithi164.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-02-10T04:10:25.791 INFO:teuthology.orchestra.run.smithi164.stdout:ANSI_COLOR="0;31" 2021-02-10T04:10:25.791 INFO:teuthology.orchestra.run.smithi164.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-02-10T04:10:25.791 INFO:teuthology.orchestra.run.smithi164.stdout:HOME_URL="https://www.redhat.com/" 2021-02-10T04:10:25.791 INFO:teuthology.orchestra.run.smithi164.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-02-10T04:10:25.791 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:10:25.791 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-02-10T04:10:25.791 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-02-10T04:10:25.792 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-02-10T04:10:25.792 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-02-10T04:10:25.792 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server 2021-02-10T04:10:25.809 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-02-10T04:10:25.822 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-02-10T04:10:25.835 INFO:teuthology.task.internal:Checking for old test directory... 2021-02-10T04:10:25.835 DEBUG:teuthology.orchestra.run.smithi085:> test '!' -e /home/ubuntu/cephtest 2021-02-10T04:10:25.837 DEBUG:teuthology.orchestra.run.smithi164:> test '!' -e /home/ubuntu/cephtest 2021-02-10T04:10:25.851 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-02-10T04:10:25.861 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-02-10T04:10:25.862 DEBUG:teuthology.orchestra.run.smithi085:> test -z $(ls -A /var/lib/ceph) 2021-02-10T04:10:25.891 DEBUG:teuthology.orchestra.run.smithi164:> test -z $(ls -A /var/lib/ceph) 2021-02-10T04:10:25.910 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-02-10T04:10:25.960 INFO:teuthology.run_tasks:Running task kernel... 2021-02-10T04:10:25.978 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mgr.y': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'osd.3': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}, 'ceph.rgw.realm.zone.a': {'kdb': True, 'sha1': 'distro'}, 'node-exporter.a': {'kdb': True, 'sha1': 'distro'}, 'alertmanager.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'osd.4': {'kdb': True, 'sha1': 'distro'}, 'osd.5': {'kdb': True, 'sha1': 'distro'}, 'osd.6': {'kdb': True, 'sha1': 'distro'}, 'osd.7': {'kdb': True, 'sha1': 'distro'}, 'client.1': {'kdb': True, 'sha1': 'distro'}, 'prometheus.a': {'kdb': True, 'sha1': 'distro'}, 'grafana.a': {'kdb': True, 'sha1': 'distro'}, 'node-exporter.b': {'kdb': True, 'sha1': 'distro'}, 'ceph.iscsi.iscsi.a': {'kdb': True, 'sha1': 'distro'}} 2021-02-10T04:10:25.979 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-02-10T04:10:25.979 DEBUG:teuthology.orchestra.run.smithi085:> uname -r 2021-02-10T04:10:25.994 INFO:teuthology.orchestra.run.smithi085.stdout:4.18.0-240.1.1.el8_3.x86_64 2021-02-10T04:10:25.994 INFO:teuthology.task.kernel:Running kernel on smithi085: 4.18.0-240.1.1.el8_3.x86_64 2021-02-10T04:10:25.994 DEBUG:teuthology.orchestra.run.smithi085:> sudo yum install -y kernel 2021-02-10T04:10:27.461 INFO:teuthology.orchestra.run.smithi085.stdout:Updating Subscription Management repositories. 2021-02-10T04:10:27.464 INFO:teuthology.orchestra.run.smithi085.stdout:Unable to read consumer identity 2021-02-10T04:10:27.596 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:10:27.596 INFO:teuthology.orchestra.run.smithi085.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-02-10T04:10:27.596 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:10:29.445 INFO:teuthology.orchestra.run.smithi085.stdout:Extra Packages for Enterprise Linux 20 MB/s | 8.8 MB 00:00 2021-02-10T04:10:31.196 INFO:teuthology.orchestra.run.smithi085.stdout:lab-extras 4.2 MB/s | 24 kB 00:00 2021-02-10T04:10:31.691 INFO:teuthology.orchestra.run.smithi085.stdout:Package kernel-4.18.0-240.1.1.el8_3.x86_64 is already installed. 2021-02-10T04:10:31.692 INFO:teuthology.orchestra.run.smithi085.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-02-10T04:10:31.796 INFO:teuthology.orchestra.run.smithi085.stdout:Dependencies resolved. 2021-02-10T04:10:31.804 INFO:teuthology.orchestra.run.smithi085.stdout:Nothing to do. 2021-02-10T04:10:31.805 INFO:teuthology.orchestra.run.smithi085.stdout:Complete! 2021-02-10T04:10:31.895 DEBUG:teuthology.orchestra.run.smithi085:> echo no | sudo yum reinstall kernel || true 2021-02-10T04:10:32.196 INFO:teuthology.orchestra.run.smithi085.stdout:Updating Subscription Management repositories. 2021-02-10T04:10:32.196 INFO:teuthology.orchestra.run.smithi085.stdout:Unable to read consumer identity 2021-02-10T04:10:32.242 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:10:32.243 INFO:teuthology.orchestra.run.smithi085.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-02-10T04:10:32.243 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:10:32.410 INFO:teuthology.orchestra.run.smithi085.stdout:Last metadata expiration check: 0:00:01 ago on Wed 10 Feb 2021 04:10:31 AM UTC. 2021-02-10T04:10:32.549 INFO:teuthology.orchestra.run.smithi085.stdout:Installed package kernel-4.18.0-240.1.1.el8_3.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2021-02-10T04:10:32.549 INFO:teuthology.orchestra.run.smithi085.stdout:Installed package kernel-4.18.0-240.el8.x86_64 (from anaconda) not available. 2021-02-10T04:10:32.582 INFO:teuthology.orchestra.run.smithi085.stderr:Error: No packages marked for reinstall. 2021-02-10T04:10:32.618 DEBUG:teuthology.orchestra.run.smithi085:> sudo yum reinstall -y kernel || true 2021-02-10T04:10:32.879 INFO:teuthology.orchestra.run.smithi085.stdout:Updating Subscription Management repositories. 2021-02-10T04:10:32.880 INFO:teuthology.orchestra.run.smithi085.stdout:Unable to read consumer identity 2021-02-10T04:10:32.931 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:10:32.931 INFO:teuthology.orchestra.run.smithi085.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-02-10T04:10:32.931 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:10:33.099 INFO:teuthology.orchestra.run.smithi085.stdout:Last metadata expiration check: 0:00:02 ago on Wed 10 Feb 2021 04:10:31 AM UTC. 2021-02-10T04:10:33.239 INFO:teuthology.orchestra.run.smithi085.stdout:Installed package kernel-4.18.0-240.1.1.el8_3.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2021-02-10T04:10:33.239 INFO:teuthology.orchestra.run.smithi085.stdout:Installed package kernel-4.18.0-240.el8.x86_64 (from anaconda) not available. 2021-02-10T04:10:33.264 INFO:teuthology.orchestra.run.smithi085.stderr:Error: No packages marked for reinstall. 2021-02-10T04:10:33.301 DEBUG:teuthology.orchestra.run.smithi085:> rpm -q kernel --last | head -n 1 2021-02-10T04:10:33.393 INFO:teuthology.orchestra.run.smithi085.stdout:kernel-4.18.0-240.1.1.el8_3.x86_64 Tue 10 Nov 2020 09:16:49 PM UTC 2021-02-10T04:10:33.395 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-240.1.1.el8_3.x86_64 2021-02-10T04:10:33.395 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-02-10T04:10:33.395 DEBUG:teuthology.orchestra.run.smithi164:> uname -r 2021-02-10T04:10:33.411 INFO:teuthology.orchestra.run.smithi164.stdout:4.18.0-240.1.1.el8_3.x86_64 2021-02-10T04:10:33.412 INFO:teuthology.task.kernel:Running kernel on smithi164: 4.18.0-240.1.1.el8_3.x86_64 2021-02-10T04:10:33.412 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum install -y kernel 2021-02-10T04:10:35.042 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2021-02-10T04:10:35.045 INFO:teuthology.orchestra.run.smithi164.stdout:Unable to read consumer identity 2021-02-10T04:10:35.152 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:10:35.152 INFO:teuthology.orchestra.run.smithi164.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-02-10T04:10:35.152 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:10:36.922 INFO:teuthology.orchestra.run.smithi164.stdout:Extra Packages for Enterprise Linux 20 MB/s | 8.8 MB 00:00 2021-02-10T04:10:38.693 INFO:teuthology.orchestra.run.smithi164.stdout:lab-extras 4.3 MB/s | 24 kB 00:00 2021-02-10T04:10:39.231 INFO:teuthology.orchestra.run.smithi164.stdout:Package kernel-4.18.0-240.1.1.el8_3.x86_64 is already installed. 2021-02-10T04:10:39.232 INFO:teuthology.orchestra.run.smithi164.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-02-10T04:10:39.369 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2021-02-10T04:10:39.380 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2021-02-10T04:10:39.380 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2021-02-10T04:10:39.496 DEBUG:teuthology.orchestra.run.smithi164:> echo no | sudo yum reinstall kernel || true 2021-02-10T04:10:39.762 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2021-02-10T04:10:39.762 INFO:teuthology.orchestra.run.smithi164.stdout:Unable to read consumer identity 2021-02-10T04:10:39.815 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:10:39.815 INFO:teuthology.orchestra.run.smithi164.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-02-10T04:10:39.815 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:10:39.985 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:00:01 ago on Wed 10 Feb 2021 04:10:38 AM UTC. 2021-02-10T04:10:40.144 INFO:teuthology.orchestra.run.smithi164.stdout:Installed package kernel-4.18.0-240.1.1.el8_3.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2021-02-10T04:10:40.144 INFO:teuthology.orchestra.run.smithi164.stdout:Installed package kernel-4.18.0-240.el8.x86_64 (from anaconda) not available. 2021-02-10T04:10:40.189 INFO:teuthology.orchestra.run.smithi164.stderr:Error: No packages marked for reinstall. 2021-02-10T04:10:40.224 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum reinstall -y kernel || true 2021-02-10T04:10:40.488 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2021-02-10T04:10:40.488 INFO:teuthology.orchestra.run.smithi164.stdout:Unable to read consumer identity 2021-02-10T04:10:40.532 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:10:40.532 INFO:teuthology.orchestra.run.smithi164.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-02-10T04:10:40.532 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:10:40.700 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:00:02 ago on Wed 10 Feb 2021 04:10:38 AM UTC. 2021-02-10T04:10:40.835 INFO:teuthology.orchestra.run.smithi164.stdout:Installed package kernel-4.18.0-240.1.1.el8_3.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2021-02-10T04:10:40.835 INFO:teuthology.orchestra.run.smithi164.stdout:Installed package kernel-4.18.0-240.el8.x86_64 (from anaconda) not available. 2021-02-10T04:10:40.862 INFO:teuthology.orchestra.run.smithi164.stderr:Error: No packages marked for reinstall. 2021-02-10T04:10:40.898 DEBUG:teuthology.orchestra.run.smithi164:> rpm -q kernel --last | head -n 1 2021-02-10T04:10:40.988 INFO:teuthology.orchestra.run.smithi164.stdout:kernel-4.18.0-240.1.1.el8_3.x86_64 Tue 10 Nov 2020 09:16:49 PM UTC 2021-02-10T04:10:40.990 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-240.1.1.el8_3.x86_64 2021-02-10T04:10:40.990 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-02-10T04:10:40.991 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2021-02-10T04:10:40.991 DEBUG:teuthology.orchestra.run.smithi085:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-02-10T04:10:41.058 INFO:teuthology.orchestra.run.smithi085.stdout:ttyS1 2021-02-10T04:10:41.071 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2021-02-10T04:10:41.071 DEBUG:teuthology.orchestra.run.smithi164:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-02-10T04:10:41.101 INFO:teuthology.orchestra.run.smithi164.stdout:ttyS1 2021-02-10T04:10:41.120 INFO:teuthology.run_tasks:Running task internal.base... 2021-02-10T04:10:41.129 INFO:teuthology.task.internal:Creating test directory... 2021-02-10T04:10:41.129 DEBUG:teuthology.orchestra.run.smithi085:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-02-10T04:10:41.132 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-02-10T04:10:41.174 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-02-10T04:10:41.283 INFO:teuthology.run_tasks:Running task internal.archive... 2021-02-10T04:10:41.296 INFO:teuthology.task.internal:Creating archive directory... 2021-02-10T04:10:41.296 DEBUG:teuthology.orchestra.run.smithi085:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-02-10T04:10:41.299 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-02-10T04:10:41.333 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-02-10T04:10:41.343 INFO:teuthology.task.internal:Enabling coredump saving... 2021-02-10T04:10:41.344 DEBUG:teuthology.orchestra.run.smithi085:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-02-10T04:10:41.370 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-02-10T04:10:41.398 INFO:teuthology.orchestra.run.smithi085.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-10T04:10:41.405 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-10T04:10:41.408 INFO:teuthology.orchestra.run.smithi085.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-10T04:10:41.415 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-10T04:10:41.416 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-02-10T04:10:46.951 INFO:teuthology.task.internal:Configuring sudo... 2021-02-10T04:10:47.705 DEBUG:teuthology.orchestra.run.smithi085:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-02-10T04:10:47.708 DEBUG:teuthology.orchestra.run.smithi164:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-02-10T04:10:47.739 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-02-10T04:10:47.755 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-02-10T04:10:47.756 DEBUG:teuthology.orchestra.run.smithi085:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-02-10T04:10:47.778 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-02-10T04:10:47.796 DEBUG:teuthology.orchestra.run.smithi085:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-02-10T04:10:47.870 DEBUG:teuthology.orchestra.run.smithi085:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-02-10T04:10:47.951 DEBUG:teuthology.orchestra.run.smithi085:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-02-10T04:10:47.987 DEBUG:teuthology.orchestra.run.smithi085:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-02-10T04:10:48.034 DEBUG:teuthology.orchestra.run.smithi085:> set -ex 2021-02-10T04:10:48.034 DEBUG:teuthology.orchestra.run.smithi085:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-02-10T04:10:48.098 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-02-10T04:10:48.134 DEBUG:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-02-10T04:10:48.217 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-02-10T04:10:48.252 DEBUG:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-02-10T04:10:48.301 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2021-02-10T04:10:48.301 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-02-10T04:10:48.367 DEBUG:teuthology.orchestra.run.smithi085:> sudo service rsyslog restart 2021-02-10T04:10:48.369 DEBUG:teuthology.orchestra.run.smithi164:> sudo service rsyslog restart 2021-02-10T04:10:48.437 INFO:teuthology.orchestra.run.smithi085.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-02-10T04:10:48.473 INFO:teuthology.orchestra.run.smithi164.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-02-10T04:10:48.705 INFO:teuthology.run_tasks:Running task internal.timer... 2021-02-10T04:10:48.716 INFO:teuthology.task.internal:Starting timer... 2021-02-10T04:10:48.716 INFO:teuthology.run_tasks:Running task pcp... 2021-02-10T04:10:48.734 INFO:teuthology.run_tasks:Running task selinux... 2021-02-10T04:10:48.746 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2021-02-10T04:10:48.746 DEBUG:teuthology.orchestra.run.smithi085:> sudo service auditd rotate 2021-02-10T04:10:48.843 INFO:teuthology.orchestra.run.smithi085.stdout:Rotating logs: [ OK ] 2021-02-10T04:10:48.844 DEBUG:teuthology.orchestra.run.smithi164:> sudo service auditd rotate 2021-02-10T04:10:48.930 INFO:teuthology.orchestra.run.smithi164.stdout:Rotating logs: [ OK ] 2021-02-10T04:10:48.931 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-02-10T04:10:48.932 DEBUG:teuthology.orchestra.run.smithi085:> /usr/sbin/getenforce 2021-02-10T04:10:48.961 INFO:teuthology.orchestra.run.smithi085.stdout:Permissive 2021-02-10T04:10:48.961 DEBUG:teuthology.orchestra.run.smithi164:> /usr/sbin/getenforce 2021-02-10T04:10:48.982 INFO:teuthology.orchestra.run.smithi164.stdout:Permissive 2021-02-10T04:10:48.983 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi085.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi164.front.sepia.ceph.com': 'permissive'} 2021-02-10T04:10:48.983 DEBUG:teuthology.orchestra.run.smithi085:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-02-10T04:10:49.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T04:10:49.028 DEBUG:teuthology.orchestra.run.smithi164:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-02-10T04:10:49.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T04:10:49.053 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-02-10T04:10:49.053 DEBUG:teuthology.orchestra.run.smithi085:> sudo /usr/sbin/setenforce permissive 2021-02-10T04:10:49.093 DEBUG:teuthology.orchestra.run.smithi164:> sudo /usr/sbin/setenforce permissive 2021-02-10T04:10:49.119 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-02-10T04:10:49.452 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-02-10T04:10:49.453 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-02-10T04:10:49.475 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2021-02-10T04:10:49.477 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi085.front.sepia.ceph.com,smithi164.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-02-10T04:10:50.452 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-02-10T04:10:52.886 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-02-10T04:10:56.486 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-02-10T04:10:56.534 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:10:56 +0000 (0:00:00.168) 0:00:00.168 **** 2021-02-10T04:10:56.556 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:10:56 +0000 (0:00:00.021) 0:00:00.190 **** 2021-02-10T04:10:58.331 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi085.front.sepia.ceph.com] 2021-02-10T04:10:58.414 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:10:58 +0000 (0:00:01.858) 0:00:02.049 **** 2021-02-10T04:10:58.429 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] 2021-02-10T04:10:58.507 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:10:58 +0000 (0:00:00.092) 0:00:02.141 **** 2021-02-10T04:10:58.508 INFO:teuthology.task.ansible.out: 2021-02-10T04:10:58.739 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-02-10T04:10:58.740 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-02-10T04:10:58.776 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:10:58 +0000 (0:00:00.269) 0:00:02.411 **** 2021-02-10T04:10:58.810 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:10:58 +0000 (0:00:00.033) 0:00:02.444 **** 2021-02-10T04:10:59.233 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-02-10T04:10:59.235 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-02-10T04:10:59.271 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:10:59 +0000 (0:00:00.461) 0:00:02.905 **** 2021-02-10T04:10:59.280 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-02-10T04:10:59.314 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:10:59 +0000 (0:00:00.043) 0:00:02.949 **** 2021-02-10T04:10:59.320 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-02-10T04:10:59.322 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:10:59.355 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:10:59 +0000 (0:00:00.040) 0:00:02.989 **** 2021-02-10T04:10:59.363 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:10:59.400 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:10:59 +0000 (0:00:00.045) 0:00:03.035 **** 2021-02-10T04:10:59.756 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-02-10T04:10:59.757 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T04:10:59.759 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T04:10:59.818 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:10:59 +0000 (0:00:00.418) 0:00:03.453 **** 2021-02-10T04:10:59.875 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:10:59 +0000 (0:00:00.056) 0:00:03.509 **** 2021-02-10T04:11:00.888 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-02-10T04:11:00.987 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612930259.8411033-164673665046557/source", "state": "file", "uid": 0} Wednesday 10 February 2021 04:11:00 +0000 (0:00:01.053) 0:00:04.562 **** changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612930259.8972282-216278722511858/source", "state": "file", "uid": 0} Wednesday 10 February 2021 04:11:00 +0000 (0:00:00.042) 0:00:04.605 **** 2021-02-10T04:11:01.457 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-02-10T04:11:01.458 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-02-10T04:11:01.459 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail 2021-02-10T04:11:01.461 INFO:teuthology.task.ansible.out:.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@ 2021-02-10T04:11:01.462 INFO:teuthology.task.ansible.out:aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-02-10T04:11:01.596 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-02-10T04:11:01.605 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-02-10T04:11:01.787 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:11:01 +0000 (0:00:00.816) 0:00:05.421 **** 2021-02-10T04:11:01.823 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:11:01 +0000 (0:00:00.036) 0:00:05.457 **** 2021-02-10T04:11:02.434 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi085.front.sepia.ceph.com] 2021-02-10T04:11:02.480 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] 2021-02-10T04:11:02.532 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-02-10T04:11:02.534 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:11:02 +0000 (0:00:00.709) 0:00:06.167 **** 2021-02-10T04:11:02.642 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:11:02 +0000 (0:00:00.109) 0:00:06.276 **** 2021-02-10T04:11:03.347 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-02-10T04:11:03.349 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.309885", "end": "2021-02-10 04:11:03.298446", "rc": 0, "start": "2021-02-10 04:11:02.988561", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} changed: [smithi085.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.356776", "end": "2021-02-10 04:11:03.325440", "rc": 0, "start": "2021-02-10 04:11:02.968664", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-02-10T04:11:03.390 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:11:03 +0000 (0:00:00.747) 0:00:07.024 **** 2021-02-10T04:11:03.431 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:11:03 +0000 (0:00:00.041) 0:00:07.066 **** 2021-02-10T04:11:03.590 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-02-10T04:11:03.591 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.010190", "end": "2021-02-10 04:11:03.568436", "failed_when_result": false, "rc": 0, "start": "2021-02-10 04:11:03.558246", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2021-02-10T04:11:03.632 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:11:03 +0000 (0:00:00.200) 0:00:07.266 **** 2021-02-10T04:11:03.653 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.005920", "end": "2021-02-10 04:11:03.624505", "failed_when_result": false, "rc": 0, "start": "2021-02-10 04:11:03.618585", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2021-02-10T04:11:03.693 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:11:03 +0000 (0:00:00.061) 0:00:07.327 **** 2021-02-10T04:11:03.700 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:11:03.755 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:11:03 +0000 (0:00:00.062) 0:00:07.390 **** 2021-02-10T04:11:03.766 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:11:03.812 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:11:03 +0000 (0:00:00.056) 0:00:07.446 **** 2021-02-10T04:11:03.818 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2021-02-10T04:11:03.819 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:11:03.865 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:11:03 +0000 (0:00:00.053) 0:00:07.499 **** 2021-02-10T04:11:03.872 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:11:03.909 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:11:03 +0000 (0:00:00.043) 0:00:07.543 **** 2021-02-10T04:11:03.916 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2021-02-10T04:11:04.200 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 04:11:03 +0000 (0:00:00.060) 0:00:07.603 **** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 04:11:04 +0000 (0:00:00.045) 0:00:07.649 **** TASK [users : Filter the managed_users list] *********************************** skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 04:11:04 +0000 (0:00:00.042) 0:00:07.691 **** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 04:11:04 +0000 (0:00:00.049) 0:00:07.740 **** TASK [users : Filter the managed_admin_users list] ***************************** skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 04:11:04 +0000 (0:00:00.046) 0:00:07.786 **** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:11:04.217 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:11:04 +0000 (0:00:00.064) 0:00:07.851 **** 2021-02-10T04:11:04.387 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-02-10T04:11:04.388 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T04:11:04.441 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:11:04 +0000 (0:00:00.223) 0:00:08.075 **** 2021-02-10T04:11:04.459 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T04:11:04.515 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:11:04 +0000 (0:00:00.074) 0:00:08.149 **** 2021-02-10T04:11:04.705 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2021-02-10T04:11:04.707 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-02-10T04:11:04.781 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-02-10T04:11:04.903 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-02-10T04:11:04.904 INFO:teuthology.task.ansible.out: 2021-02-10T04:11:05.016 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-02-10T04:11:05.080 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-02-10T04:11:05.082 INFO:teuthology.task.ansible.out: 2021-02-10T04:11:05.208 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-02-10T04:11:05.301 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-02-10T04:11:05.394 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-02-10T04:11:05.499 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-02-10T04:11:05.586 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-02-10T04:11:05.694 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-02-10T04:11:05.695 INFO:teuthology.task.ansible.out: 2021-02-10T04:11:05.780 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-02-10T04:11:05.892 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-02-10T04:11:05.967 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-02-10T04:11:06.076 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-02-10T04:11:06.141 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-02-10T04:11:06.268 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-02-10T04:11:06.332 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-02-10T04:11:06.440 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-02-10T04:11:06.504 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-02-10T04:11:06.622 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-02-10T04:11:06.697 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-02-10T04:11:06.804 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-02-10T04:11:06.878 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-02-10T04:11:06.996 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-02-10T04:11:07.071 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-02-10T04:11:07.188 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-02-10T04:11:07.252 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-02-10T04:11:07.369 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-02-10T04:11:07.435 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-02-10T04:11:07.552 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-02-10T04:11:07.616 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-02-10T04:11:07.732 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-02-10T04:11:07.797 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-02-10T04:11:07.913 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-02-10T04:11:07.977 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-02-10T04:11:08.097 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-02-10T04:11:08.161 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-02-10T04:11:08.278 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-02-10T04:11:08.344 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-02-10T04:11:08.472 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-02-10T04:11:08.526 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-02-10T04:11:08.653 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-02-10T04:11:08.707 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-02-10T04:11:08.835 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-02-10T04:11:08.888 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-02-10T04:11:09.016 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-02-10T04:11:09.070 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-02-10T04:11:09.210 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-02-10T04:11:09.255 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-02-10T04:11:09.407 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-02-10T04:11:09.441 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-02-10T04:11:09.590 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-02-10T04:11:09.623 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-02-10T04:11:09.785 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-02-10T04:11:09.807 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-02-10T04:11:09.808 INFO:teuthology.task.ansible.out: 2021-02-10T04:11:09.967 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-02-10T04:11:09.990 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-02-10T04:11:10.149 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-02-10T04:11:10.171 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-02-10T04:11:10.321 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-02-10T04:11:10.354 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-02-10T04:11:10.513 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-02-10T04:11:10.535 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-02-10T04:11:10.705 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-02-10T04:11:10.717 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-02-10T04:11:10.891 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-02-10T04:11:10.903 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-02-10T04:11:11.086 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-02-10T04:11:11.088 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-02-10T04:11:11.268 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-02-10T04:11:11.269 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-02-10T04:11:11.449 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-02-10T04:11:11.450 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-02-10T04:11:11.641 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-02-10T04:11:11.642 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-02-10T04:11:11.811 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-10T04:11:11.823 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-10T04:11:12.003 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-02-10T04:11:12.015 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-02-10T04:11:12.184 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-02-10T04:11:12.196 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-02-10T04:11:12.375 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-02-10T04:11:12.524 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-02-10T04:11:12.545 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-02-10T04:11:12.578 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-02-10T04:11:12.747 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-02-10T04:11:12.771 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-02-10T04:11:12.943 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-02-10T04:11:12.965 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-02-10T04:11:13.130 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-02-10T04:11:13.142 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-02-10T04:11:13.310 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-10T04:11:13.334 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-10T04:11:13.492 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-02-10T04:11:13.513 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-02-10T04:11:13.683 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-02-10T04:11:13.695 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-02-10T04:11:13.696 INFO:teuthology.task.ansible.out: 2021-02-10T04:11:13.865 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-02-10T04:11:13.888 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-02-10T04:11:14.047 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-02-10T04:11:14.080 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-02-10T04:11:14.226 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-02-10T04:11:14.274 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-02-10T04:11:14.413 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-02-10T04:11:14.458 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-02-10T04:11:14.604 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-02-10T04:11:14.657 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-02-10T04:11:14.798 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-02-10T04:11:14.883 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-02-10T04:11:14.968 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-02-10T04:11:15.068 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-02-10T04:11:15.154 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-02-10T04:11:15.249 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-02-10T04:11:15.345 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-02-10T04:11:15.432 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-02-10T04:11:15.527 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-02-10T04:11:15.613 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-02-10T04:11:15.709 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-02-10T04:11:15.794 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-02-10T04:11:15.891 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-02-10T04:11:15.986 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-02-10T04:11:16.092 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-02-10T04:11:16.171 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-02-10T04:11:16.299 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-02-10T04:11:16.364 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-02-10T04:11:16.502 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-10T04:11:16.578 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-10T04:11:16.717 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-02-10T04:11:16.771 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-02-10T04:11:16.931 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-02-10T04:11:16.954 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-02-10T04:11:17.122 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-02-10T04:11:17.144 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-02-10T04:11:17.314 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-02-10T04:11:17.336 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-02-10T04:11:17.498 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-02-10T04:11:17.534 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-02-10T04:11:17.698 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-02-10T04:11:17.731 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-02-10T04:11:17.881 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-02-10T04:11:17.924 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-02-10T04:11:18.073 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-02-10T04:11:18.106 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-02-10T04:11:18.265 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-02-10T04:11:18.287 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-02-10T04:11:18.448 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-02-10T04:11:18.481 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-02-10T04:11:18.630 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-02-10T04:11:18.663 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-02-10T04:11:18.810 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-02-10T04:11:18.833 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-02-10T04:11:19.002 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-02-10T04:11:19.025 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-02-10T04:11:19.186 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-02-10T04:11:19.209 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-02-10T04:11:19.371 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-02-10T04:11:19.393 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-02-10T04:11:19.562 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-02-10T04:11:19.574 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-02-10T04:11:19.576 INFO:teuthology.task.ansible.out: 2021-02-10T04:11:19.734 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-02-10T04:11:19.756 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-02-10T04:11:19.914 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-02-10T04:11:19.947 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-02-10T04:11:20.105 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-02-10T04:11:20.128 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-02-10T04:11:20.129 INFO:teuthology.task.ansible.out: 2021-02-10T04:11:20.287 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-02-10T04:11:20.309 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-02-10T04:11:20.479 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-02-10T04:11:20.503 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-02-10T04:11:20.660 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-02-10T04:11:20.682 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-02-10T04:11:20.851 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-02-10T04:11:20.863 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-02-10T04:11:21.033 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-02-10T04:11:21.045 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-02-10T04:11:21.226 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-02-10T04:11:21.227 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-02-10T04:11:21.409 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-02-10T04:11:21.411 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-02-10T04:11:21.580 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-02-10T04:11:21.602 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-02-10T04:11:21.761 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-02-10T04:11:21.783 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-02-10T04:11:21.962 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-02-10T04:11:21.963 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-02-10T04:11:22.133 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-02-10T04:11:22.144 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-02-10T04:11:22.322 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-02-10T04:11:22.323 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-02-10T04:11:22.503 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-02-10T04:11:22.504 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-02-10T04:11:22.685 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-02-10T04:11:22.687 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-02-10T04:11:22.862 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-02-10T04:11:22.876 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-02-10T04:11:23.045 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-02-10T04:11:23.046 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-02-10T04:11:23.234 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-02-10T04:11:23.235 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-02-10T04:11:23.415 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-02-10T04:11:23.427 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-02-10T04:11:23.596 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-10T04:11:23.607 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-10T04:11:23.776 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-02-10T04:11:23.788 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-02-10T04:11:23.956 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-02-10T04:11:23.968 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-02-10T04:11:24.150 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-02-10T04:11:24.163 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-02-10T04:11:24.353 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-02-10T04:11:24.355 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-02-10T04:11:24.543 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-02-10T04:11:24.544 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-02-10T04:11:24.724 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-02-10T04:11:24.726 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-02-10T04:11:24.915 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-02-10T04:11:24.927 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-02-10T04:11:25.097 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-02-10T04:11:25.098 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-02-10T04:11:25.292 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-02-10T04:11:25.304 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-02-10T04:11:25.474 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-02-10T04:11:25.486 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-02-10T04:11:25.655 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-02-10T04:11:25.667 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-02-10T04:11:25.836 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-02-10T04:11:25.858 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-02-10T04:11:26.016 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-02-10T04:11:26.050 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-02-10T04:11:26.209 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-02-10T04:11:26.231 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-02-10T04:11:26.390 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-02-10T04:11:26.413 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-02-10T04:11:26.593 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-02-10T04:11:26.604 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-02-10T04:11:26.815 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-02-10T04:11:26.848 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-02-10T04:11:27.023 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-02-10T04:11:27.060 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-02-10T04:11:27.220 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-02-10T04:11:27.242 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-02-10T04:11:27.407 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-02-10T04:11:27.442 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-02-10T04:11:27.596 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-02-10T04:11:27.639 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-02-10T04:11:27.798 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-02-10T04:11:27.842 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-02-10T04:11:28.006 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-02-10T04:11:28.007 INFO:teuthology.task.ansible.out: 2021-02-10T04:11:28.029 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-02-10T04:11:28.188 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-02-10T04:11:28.231 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-02-10T04:11:28.379 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-02-10T04:11:28.422 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-02-10T04:11:28.559 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-02-10T04:11:28.603 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-02-10T04:11:28.739 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-02-10T04:11:28.782 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-02-10T04:11:28.931 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-02-10T04:11:28.964 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-02-10T04:11:29.106 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-02-10T04:11:29.149 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-02-10T04:11:29.286 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-02-10T04:11:29.340 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-02-10T04:11:29.476 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-02-10T04:11:29.519 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-02-10T04:11:29.657 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-02-10T04:11:29.700 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-02-10T04:11:29.836 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-02-10T04:11:29.880 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-02-10T04:11:31.314 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-02-10T04:11:31.391 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-02-10T04:11:32.931 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "name": "adking", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-02-10T04:11:33.005 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "name": "adking", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-02-10T04:11:34.360 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "name": "amathuri", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-02-10T04:11:34.444 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "name": "amathuri", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-02-10T04:11:35.778 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-02-10T04:11:35.925 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:11:35 +0000 (0:00:31.409) 0:00:39.559 **** 2021-02-10T04:11:35.932 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-02-10T04:11:36.005 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2021-02-10T04:11:36.039 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:11:36 +0000 (0:00:00.114) 0:00:39.674 **** 2021-02-10T04:11:36.074 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:11:36 +0000 (0:00:00.035) 0:00:39.709 **** 2021-02-10T04:11:36.122 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:11:36 +0000 (0:00:00.047) 0:00:39.756 **** 2021-02-10T04:11:36.142 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2021-02-10T04:11:36.144 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-02-10T04:11:36.145 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-02-10T04:11:36.147 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancic 2021-02-10T04:11:36.148 INFO:teuthology.task.ansible.out:h", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1Wi 2021-02-10T04:11:36.149 INFO:teuthology.task.ansible.out:RenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vum 2021-02-10T04:11:36.151 INFO:teuthology.task.ansible.out:rao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69 2021-02-10T04:11:36.152 INFO:teuthology.task.ansible.out:mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 2021-02-10T04:11:36.152 INFO:teuthology.task.ansible.out:5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d8 2021-02-10T04:11:36.153 INFO:teuthology.task.ansible.out:41a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveirada 2021-02-10T04:11:36.155 INFO:teuthology.task.ansible.out:n@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOe 2021-02-10T04:11:36.156 INFO:teuthology.task.ansible.out:nVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9 2021-02-10T04:11:36.157 INFO:teuthology.task.ansible.out:gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": 2021-02-10T04:11:36.158 INFO:teuthology.task.ansible.out:"dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NI 2021-02-10T04:11:36.160 INFO:teuthology.task.ansible.out:Nj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395 2021-02-10T04:11:36.160 INFO:teuthology.task.ansible.out:862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-02-10T04:11:36.203 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:11:36 +0000 (0:00:00.081) 0:00:39.838 **** 2021-02-10T04:11:36.214 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-02-10T04:11:36.216 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-02-10T04:11:36.217 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancic 2021-02-10T04:11:36.218 INFO:teuthology.task.ansible.out:h", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1Wi 2021-02-10T04:11:36.220 INFO:teuthology.task.ansible.out:RenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vum 2021-02-10T04:11:36.221 INFO:teuthology.task.ansible.out:rao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69 2021-02-10T04:11:36.222 INFO:teuthology.task.ansible.out:mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 2021-02-10T04:11:36.223 INFO:teuthology.task.ansible.out:5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d8 2021-02-10T04:11:36.224 INFO:teuthology.task.ansible.out:41a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveirada 2021-02-10T04:11:36.226 INFO:teuthology.task.ansible.out:n@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOe 2021-02-10T04:11:36.227 INFO:teuthology.task.ansible.out:nVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9 2021-02-10T04:11:36.228 INFO:teuthology.task.ansible.out:gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": 2021-02-10T04:11:36.228 INFO:teuthology.task.ansible.out:"mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": 2021-02-10T04:11:36.230 INFO:teuthology.task.ansible.out:"dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NI 2021-02-10T04:11:36.231 INFO:teuthology.task.ansible.out:Nj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-02-10T04:11:36.282 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:11:36 +0000 (0:00:00.078) 0:00:39.916 **** 2021-02-10T04:11:37.293 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-02-10T04:11:37.296 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com -> localhost] => {"after": "df8946f08e870e2d2565b79099289ca48bd825f3", "attempts": 1, "before": "df8946f08e870e2d2565b79099289ca48bd825f3", "changed": false, "remote_url_changed": false} 2021-02-10T04:11:37.343 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:11:37 +0000 (0:00:01.061) 0:00:40.978 **** 2021-02-10T04:11:37.352 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com -> localhost] => {"after": "df8946f08e870e2d2565b79099289ca48bd825f3", "attempts": 1, "before": "df8946f08e870e2d2565b79099289ca48bd825f3", "changed": false, "remote_url_changed": false} 2021-02-10T04:11:37.387 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:11:37 +0000 (0:00:00.044) 0:00:41.022 **** 2021-02-10T04:11:37.703 INFO:teuthology.task.ansible.out: 2021-02-10T04:11:37.704 INFO:teuthology.task.ansible.out:TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-10T04:11:37.705 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-02-10T04:11:37.717 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-02-10T04:11:37.718 INFO:teuthology.task.ansible.out: 2021-02-10T04:11:37.940 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-02-10T04:11:37.941 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-02-10T04:11:38.179 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-02-10T04:11:38.191 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-02-10T04:11:38.424 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-02-10T04:11:38.425 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-02-10T04:11:38.435 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-02-10T04:11:38.437 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-02-10T04:11:38.678 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-02-10T04:11:38.701 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-02-10T04:11:38.914 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-02-10T04:11:38.915 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-02-10T04:11:38.936 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-02-10T04:11:38.938 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-02-10T04:11:39.160 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-02-10T04:11:39.172 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-02-10T04:11:39.403 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-02-10T04:11:39.414 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-02-10T04:11:39.623 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-02-10T04:11:39.634 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-02-10T04:11:39.635 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:11:39.646 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:11:39.868 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-02-10T04:11:39.869 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-02-10T04:11:39.870 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAA 2021-02-10T04:11:39.871 INFO:teuthology.task.ansible.out:ADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMorbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-02-10T04:11:40.093 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-02-10T04:11:40.095 INFO:teuthology.task.ansible.out:son", "validate_certs": true} skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:11:40.106 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-02-10T04:11:40.107 INFO:teuthology.task.ansible.out:son", "validate_certs": true} skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:11:40.118 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:11:40.129 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:11:40.331 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-02-10T04:11:40.376 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-02-10T04:11:40.554 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-02-10T04:11:40.556 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-02-10T04:11:40.608 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-02-10T04:11:40.609 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-02-10T04:11:40.799 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-02-10T04:11:40.834 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-02-10T04:11:41.027 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2021-02-10T04:11:41.065 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2021-02-10T04:11:41.278 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-02-10T04:11:41.291 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-02-10T04:11:41.518 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-02-10T04:11:41.530 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-02-10T04:11:41.742 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-02-10T04:11:41.754 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-02-10T04:11:41.975 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-02-10T04:11:41.977 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-02-10T04:11:41.987 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-02-10T04:11:41.988 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-02-10T04:11:42.210 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-02-10T04:11:42.211 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-02-10T04:11:42.231 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-02-10T04:11:42.233 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-02-10T04:11:42.432 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-02-10T04:11:42.455 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-02-10T04:11:42.667 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-02-10T04:11:42.679 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-02-10T04:11:42.892 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-02-10T04:11:42.893 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-02-10T04:11:42.914 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-02-10T04:11:42.916 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-02-10T04:11:43.125 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-02-10T04:11:43.146 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-02-10T04:11:43.358 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-02-10T04:11:43.359 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-02-10T04:11:43.579 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-02-10T04:11:43.591 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-02-10T04:11:43.801 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-02-10T04:11:43.813 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-02-10T04:11:44.033 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-02-10T04:11:44.035 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-02-10T04:11:44.266 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-02-10T04:11:44.267 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@c 2021-02-10T04:11:44.269 INFO:teuthology.task.ansible.out:lusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-02-10T04:11:44.489 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-02-10T04:11:44.500 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-02-10T04:11:44.720 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-02-10T04:11:44.721 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-02-10T04:11:44.942 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-02-10T04:11:44.943 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-02-10T04:11:45.176 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-02-10T04:11:45.187 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-02-10T04:11:45.408 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-02-10T04:11:45.645 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-02-10T04:11:45.650 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-02-10T04:11:45.672 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-02-10T04:11:45.884 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-10T04:11:45.906 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-10T04:11:46.121 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-02-10T04:11:46.132 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-02-10T04:11:46.369 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-02-10T04:11:46.402 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-02-10T04:11:46.602 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-02-10T04:11:46.603 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-02-10T04:11:46.656 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-02-10T04:11:46.660 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-02-10T04:11:46.837 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-02-10T04:11:46.838 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-02-10T04:11:46.891 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-02-10T04:11:46.893 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-02-10T04:11:47.077 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-02-10T04:11:47.078 INFO:teuthology.task.ansible.out: 2021-02-10T04:11:47.132 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-02-10T04:11:47.321 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-02-10T04:11:47.374 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-02-10T04:11:47.545 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-02-10T04:11:47.609 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-02-10T04:11:47.793 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-02-10T04:11:47.795 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-10T04:11:47.880 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-02-10T04:11:47.882 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-10T04:11:48.050 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-02-10T04:11:48.051 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-02-10T04:11:48.053 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-02-10T04:11:48.128 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-02-10T04:11:48.129 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-02-10T04:11:48.131 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-02-10T04:11:48.288 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-02-10T04:11:48.363 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-02-10T04:11:48.527 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-02-10T04:11:48.604 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-02-10T04:11:48.764 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-02-10T04:11:48.838 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-02-10T04:11:49.021 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-02-10T04:11:49.076 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-02-10T04:11:49.283 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-02-10T04:11:49.316 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-02-10T04:11:49.545 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-02-10T04:11:49.546 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0b 2021-02-10T04:11:49.548 INFO:teuthology.task.ansible.out:sfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-02-10T04:11:49.779 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-02-10T04:11:49.791 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-02-10T04:11:50.011 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-02-10T04:11:50.022 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-02-10T04:11:50.255 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-02-10T04:11:50.257 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-02-10T04:11:50.266 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-02-10T04:11:50.268 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-02-10T04:11:50.487 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-02-10T04:11:50.488 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-02-10T04:11:50.722 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-02-10T04:11:50.724 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-02-10T04:11:50.725 INFO:teuthology.task.ansible.out: 2021-02-10T04:11:50.958 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-02-10T04:11:50.960 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylM 2021-02-10T04:11:50.961 INFO:teuthology.task.ansible.out:R5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerH 2021-02-10T04:11:50.962 INFO:teuthology.task.ansible.out:COTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-02-10T04:11:51.205 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-02-10T04:11:51.206 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-02-10T04:11:51.434 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-02-10T04:11:51.446 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-02-10T04:11:51.670 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-02-10T04:11:51.681 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-02-10T04:11:51.930 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-10T04:11:51.931 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-10T04:11:52.176 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-02-10T04:11:52.177 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-02-10T04:11:52.399 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-02-10T04:11:52.410 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-02-10T04:11:52.643 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-02-10T04:11:52.643 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-02-10T04:11:52.875 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-02-10T04:11:52.876 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-02-10T04:11:53.110 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-02-10T04:11:53.111 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-02-10T04:11:53.336 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-02-10T04:11:53.356 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-02-10T04:11:53.584 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-02-10T04:11:53.620 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-02-10T04:11:53.833 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-02-10T04:11:53.845 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-02-10T04:11:54.069 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-02-10T04:11:54.071 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-02-10T04:11:54.310 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-02-10T04:11:54.311 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-02-10T04:11:54.534 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-02-10T04:11:54.566 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-02-10T04:11:54.766 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-02-10T04:11:54.808 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-02-10T04:11:55.005 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-02-10T04:11:55.037 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-02-10T04:11:55.267 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-02-10T04:11:55.305 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-02-10T04:11:55.508 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-02-10T04:11:55.542 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-02-10T04:11:55.754 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-02-10T04:11:56.425 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae 2021-02-10T04:11:56.426 INFO:teuthology.task.ansible.out:\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn 2021-02-10T04:11:56.427 INFO:teuthology.task.ansible.out:': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jA 2021-02-10T04:11:56.429 INFO:teuthology.task.ansible.out:dSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed5709 2021-02-10T04:11:56.430 INFO:teuthology.task.ansible.out:83d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-02-10T04:11:56.501 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-02-10T04:11:56.512 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-02-10T04:11:56.747 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-02-10T04:11:56.764 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-02-10T04:11:57.037 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-02-10T04:11:57.048 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-02-10T04:11:57.305 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-02-10T04:11:57.307 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW 2021-02-10T04:11:57.308 INFO:teuthology.task.ansible.out:22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.localdomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-02-10T04:11:57.528 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-02-10T04:11:57.530 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-02-10T04:11:57.540 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-02-10T04:11:57.542 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-02-10T04:11:57.785 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-02-10T04:11:57.796 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-02-10T04:11:58.028 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-02-10T04:11:58.028 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-02-10T04:11:58.264 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-02-10T04:11:58.265 INFO:teuthology.task.ansible.out: 2021-02-10T04:11:58.266 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-02-10T04:11:58.487 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-02-10T04:11:58.499 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-02-10T04:11:58.726 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-02-10T04:11:58.737 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-02-10T04:11:58.980 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-02-10T04:11:58.991 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-02-10T04:11:59.252 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-02-10T04:11:59.254 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-02-10T04:11:59.509 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-02-10T04:11:59.511 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-02-10T04:11:59.520 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-02-10T04:11:59.522 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-02-10T04:11:59.769 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-02-10T04:11:59.794 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-02-10T04:12:00.013 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-02-10T04:12:00.014 INFO:teuthology.task.ansible.out: 2021-02-10T04:12:00.056 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-02-10T04:12:00.280 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-02-10T04:12:00.346 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-02-10T04:12:00.516 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-02-10T04:12:00.591 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-02-10T04:12:00.750 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-02-10T04:12:00.813 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-02-10T04:12:00.982 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-02-10T04:12:01.035 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-02-10T04:12:01.203 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2021-02-10T04:12:01.205 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-10T04:12:01.268 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2021-02-10T04:12:01.269 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-10T04:12:01.447 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-02-10T04:12:01.501 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-02-10T04:12:01.680 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-02-10T04:12:01.744 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-02-10T04:12:01.901 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-02-10T04:12:01.975 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-02-10T04:12:02.132 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-02-10T04:12:02.206 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-02-10T04:12:02.365 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-02-10T04:12:02.430 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-02-10T04:12:02.590 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-02-10T04:12:02.658 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-02-10T04:12:02.828 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-02-10T04:12:02.882 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-02-10T04:12:03.076 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-02-10T04:12:03.109 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-02-10T04:12:03.332 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-02-10T04:12:03.344 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-02-10T04:12:03.581 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-02-10T04:12:03.582 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-02-10T04:12:03.848 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-02-10T04:12:03.849 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-02-10T04:12:04.102 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-02-10T04:12:04.103 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-02-10T04:12:04.345 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-02-10T04:12:04.357 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-02-10T04:12:04.590 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-02-10T04:12:04.601 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-02-10T04:12:04.824 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-02-10T04:12:04.825 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-02-10T04:12:05.059 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-02-10T04:12:05.059 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-02-10T04:12:05.282 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-02-10T04:12:05.283 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-02-10T04:12:05.506 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-02-10T04:12:05.506 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-02-10T04:12:05.739 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-02-10T04:12:05.740 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFT 2021-02-10T04:12:05.742 INFO:teuthology.task.ansible.out:Vsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjLd+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-02-10T04:12:05.962 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-02-10T04:12:05.973 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-02-10T04:12:06.195 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-02-10T04:12:06.196 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-02-10T04:12:06.416 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-02-10T04:12:06.417 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-02-10T04:12:06.648 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-02-10T04:12:06.648 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-02-10T04:12:06.880 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-02-10T04:12:06.880 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-02-10T04:12:07.110 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-02-10T04:12:07.121 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-02-10T04:12:07.342 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-02-10T04:12:07.342 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-02-10T04:12:07.574 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-02-10T04:12:07.574 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-02-10T04:12:07.802 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-02-10T04:12:07.804 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-02-10T04:12:08.043 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-02-10T04:12:08.044 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-02-10T04:12:08.269 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-02-10T04:12:08.302 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-02-10T04:12:08.515 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-02-10T04:12:08.537 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-02-10T04:12:08.739 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-02-10T04:12:08.761 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-02-10T04:12:08.971 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-02-10T04:12:08.993 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-02-10T04:12:09.215 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-02-10T04:12:09.226 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-02-10T04:12:09.447 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-02-10T04:12:09.448 INFO:teuthology.task.ansible.out: 2021-02-10T04:12:09.459 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-02-10T04:12:09.681 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com", "key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-02-10T04:12:09.713 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com", "key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-02-10T04:12:09.915 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-02-10T04:12:09.974 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-02-10T04:12:10.060 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:10 +0000 (0:00:32.672) 0:01:13.694 **** 2021-02-10T04:12:10.099 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:10 +0000 (0:00:00.039) 0:01:13.733 **** 2021-02-10T04:12:10.131 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-10T04:12:10.133 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:10.143 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:10.165 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:10.167 INFO:teuthology.task.ansible.out: skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:10.177 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:10.179 INFO:teuthology.task.ansible.out: skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:10.180 INFO:teuthology.task.ansible.out: 2021-02-10T04:12:10.191 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:10.192 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:10.194 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:10.204 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:10.205 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:10.216 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:10.227 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:10.227 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:10.239 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:10.496 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-02-10T04:12:10.506 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:10.518 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-02-10T04:12:10.518 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:10.520 INFO:teuthology.task.ansible.out: 2021-02-10T04:12:10.541 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:10.563 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:10.747 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-02-10T04:12:10.791 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-02-10T04:12:11.012 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-02-10T04:12:11.025 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.027 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.037 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-02-10T04:12:11.038 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.050 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.051 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.053 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.058 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.067 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.067 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.079 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.080 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.090 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.091 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.103 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.104 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.116 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.138 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.140 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.150 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.152 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.162 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.174 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.175 INFO:teuthology.task.ansible.out: skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.176 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.187 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.187 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.199 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.199 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.200 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.211 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.212 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.223 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.225 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.235 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.236 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.247 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.248 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.249 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.250 INFO:teuthology.task.ansible.out: 2021-02-10T04:12:11.252 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.263 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.265 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.277 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.277 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.278 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.289 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.291 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.301 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.301 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.302 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.303 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.304 INFO:teuthology.task.ansible.out: skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.315 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.316 INFO:teuthology.task.ansible.out: skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.327 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.338 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.340 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.341 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.353 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.354 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.355 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.366 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.367 INFO:teuthology.task.ansible.out: 2021-02-10T04:12:11.368 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.368 INFO:teuthology.task.ansible.out: 2021-02-10T04:12:11.370 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.371 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.381 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.383 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.393 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.394 INFO:teuthology.task.ansible.out: skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.408 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.410 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.423 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.425 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.425 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.426 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQ 2021-02-10T04:12:11.427 INFO:teuthology.task.ansible.out:HujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.438 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.440 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.442 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.443 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.453 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.455 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.466 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.466 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.468 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.478 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.479 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.500 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.505 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.514 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.524 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.526 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.536 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.539 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.548 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.548 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.560 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.560 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.561 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.572 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.574 INFO:teuthology.task.ansible.out: 2021-02-10T04:12:11.575 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.578 INFO:teuthology.task.ansible.out: 2021-02-10T04:12:11.579 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.591 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.602 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.603 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.614 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.615 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.616 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.627 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.628 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.629 INFO:teuthology.task.ansible.out: skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.631 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.632 INFO:teuthology.task.ansible.out: 2021-02-10T04:12:11.644 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.644 INFO:teuthology.task.ansible.out: 2021-02-10T04:12:11.658 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.659 INFO:teuthology.task.ansible.out: 2021-02-10T04:12:11.661 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.663 INFO:teuthology.task.ansible.out: 2021-02-10T04:12:11.663 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.664 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.664 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.666 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.681 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.684 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.685 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.698 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.699 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.703 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.714 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.716 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.716 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7Z 2021-02-10T04:12:11.717 INFO:teuthology.task.ansible.out:Hi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.727 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.729 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.740 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.751 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.752 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.763 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.764 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.775 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.776 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.786 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.788 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.798 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.799 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.800 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.811 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.823 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.825 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.837 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.838 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.839 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.841 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.842 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.858 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.859 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.860 INFO:teuthology.task.ansible.out: skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.863 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.874 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.875 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.887 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.888 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.898 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.909 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.910 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.911 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.922 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.924 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.933 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.934 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.936 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.946 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.948 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.958 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.958 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.960 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.970 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.971 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.982 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.983 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.994 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.994 INFO:teuthology.task.ansible.out: 2021-02-10T04:12:11.995 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:11.996 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.007 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.007 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.009 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.020 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.020 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.032 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.033 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.044 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.045 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.057 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.069 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.070 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.081 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.081 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.083 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.093 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.093 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.104 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.105 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.107 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.107 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.125 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.126 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.161 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:12 +0000 (0:00:02.061) 0:01:15.795 **** 2021-02-10T04:12:12.170 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.171 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.172 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.173 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.185 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.209 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2021-02-10T04:12:12.210 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.247 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:12 +0000 (0:00:00.085) 0:01:15.881 **** 2021-02-10T04:12:12.281 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:12 +0000 (0:00:00.034) 0:01:15.915 **** 2021-02-10T04:12:12.287 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:12.324 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:12 +0000 (0:00:00.043) 0:01:15.959 **** 2021-02-10T04:12:12.511 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-10T04:12:12.512 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-02-10T04:12:12.554 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-02-10T04:12:12.691 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-02-10T04:12:12.724 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-02-10T04:12:12.871 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-02-10T04:12:12.905 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-02-10T04:12:13.053 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-02-10T04:12:13.085 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-02-10T04:12:13.233 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-02-10T04:12:13.255 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-02-10T04:12:13.414 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-02-10T04:12:13.436 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-02-10T04:12:13.438 INFO:teuthology.task.ansible.out: 2021-02-10T04:12:13.598 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-02-10T04:12:13.609 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-02-10T04:12:13.768 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-02-10T04:12:13.791 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-02-10T04:12:13.948 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-02-10T04:12:13.970 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-02-10T04:12:14.127 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-02-10T04:12:14.149 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-02-10T04:12:14.318 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-02-10T04:12:14.330 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-02-10T04:12:14.511 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-02-10T04:12:14.512 INFO:teuthology.task.ansible.out: ok: [smithi085.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-02-10T04:12:14.694 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-02-10T04:12:14.705 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-02-10T04:12:14.878 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-02-10T04:12:14.882 INFO:teuthology.task.ansible.out: 2021-02-10T04:12:14.895 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-02-10T04:12:15.066 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-02-10T04:12:15.078 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-02-10T04:12:15.250 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-02-10T04:12:15.261 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-02-10T04:12:15.438 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-02-10T04:12:15.450 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-02-10T04:12:15.631 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-02-10T04:12:15.633 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-02-10T04:12:15.815 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-02-10T04:12:15.817 INFO:teuthology.task.ansible.out: 2021-02-10T04:12:15.828 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-02-10T04:12:16.000 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-02-10T04:12:16.002 INFO:teuthology.task.ansible.out: 2021-02-10T04:12:16.023 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-02-10T04:12:16.205 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-02-10T04:12:16.217 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-02-10T04:12:16.388 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-02-10T04:12:16.389 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-02-10T04:12:16.570 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-02-10T04:12:16.581 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-02-10T04:12:16.762 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-02-10T04:12:16.763 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-02-10T04:12:16.953 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-02-10T04:12:16.954 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-02-10T04:12:17.133 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-02-10T04:12:17.146 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-02-10T04:12:17.313 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-02-10T04:12:17.314 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-02-10T04:12:17.492 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-02-10T04:12:17.492 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-02-10T04:12:17.675 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-02-10T04:12:17.686 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-02-10T04:12:17.857 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-02-10T04:12:17.859 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-02-10T04:12:18.042 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-02-10T04:12:18.043 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-02-10T04:12:18.213 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-02-10T04:12:18.224 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-02-10T04:12:18.392 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-02-10T04:12:18.404 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-02-10T04:12:18.571 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-02-10T04:12:18.572 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-02-10T04:12:18.762 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-02-10T04:12:18.872 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-02-10T04:12:18.943 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-02-10T04:12:18.955 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-02-10T04:12:19.116 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-02-10T04:12:19.127 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-02-10T04:12:20.469 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "dorinda", "name": "dorinda", "remove": false, "state": "absent"} 2021-02-10T04:12:20.491 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "dorinda", "name": "dorinda", "remove": false, "state": "absent"} 2021-02-10T04:12:22.026 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "zyan", "name": "zyan", "remove": false, "state": "absent"} 2021-02-10T04:12:22.085 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "zyan", "name": "zyan", "remove": false, "state": "absent"} 2021-02-10T04:12:22.275 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-02-10T04:12:22.275 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-02-10T04:12:22.465 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-02-10T04:12:22.497 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-02-10T04:12:22.655 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-02-10T04:12:22.698 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-02-10T04:12:22.838 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-02-10T04:12:22.891 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-02-10T04:12:23.007 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-02-10T04:12:23.071 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-02-10T04:12:23.186 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-02-10T04:12:23.251 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-02-10T04:12:23.368 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-02-10T04:12:23.421 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-02-10T04:12:23.547 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-02-10T04:12:23.614 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-02-10T04:12:23.733 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-02-10T04:12:23.788 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-02-10T04:12:23.926 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-02-10T04:12:23.987 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:23 +0000 (0:00:11.663) 0:01:27.622 **** 2021-02-10T04:12:23.994 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-02-10T04:12:24.048 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:24 +0000 (0:00:00.061) 0:01:27.683 **** 2021-02-10T04:12:24.509 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-02-10T04:12:24.510 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => {"changed": true, "checksum": "5ebc9fcf8859a9c716ab10089a7bec7e44d29368", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "ee06ed012f51c2e3764171d4b9180349", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612930344.0130706-152156004273155/source", "state": "file", "uid": 0} 2021-02-10T04:12:24.546 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:24 +0000 (0:00:00.497) 0:01:28.180 **** 2021-02-10T04:12:24.564 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "checksum": "5ebc9fcf8859a9c716ab10089a7bec7e44d29368", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "ee06ed012f51c2e3764171d4b9180349", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612930344.070126-71110913189622/source", "state": "file", "uid": 0} 2021-02-10T04:12:24.598 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:24 +0000 (0:00:00.051) 0:01:28.232 **** 2021-02-10T04:12:24.604 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-02-10T04:12:24.605 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => { "msg": "Host smithi085.front.sepia.ceph.com is running RedHat 8.3 (Ootpa)" } 2021-02-10T04:12:24.646 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:24 +0000 (0:00:00.048) 0:01:28.280 **** 2021-02-10T04:12:24.654 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => { "msg": "Host smithi164.front.sepia.ceph.com is running RedHat 8.3 (Ootpa)" } 2021-02-10T04:12:24.689 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:24 +0000 (0:00:00.042) 0:01:28.323 **** 2021-02-10T04:12:24.697 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-02-10T04:12:24.698 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"], "rhsm_repos": ["rhel-8-for-x86_64-baseos-rpms", "rhel-8-for-x86_64-appstream-rpms"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"} 2021-02-10T04:12:24.736 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:24 +0000 (0:00:00.047) 0:01:28.371 **** 2021-02-10T04:12:24.743 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"], "rhsm_repos": ["rhel-8-for-x86_64-baseos-rpms", "rhel-8-for-x86_64-appstream-rpms"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"} 2021-02-10T04:12:24.786 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:24 +0000 (0:00:00.050) 0:01:28.421 **** 2021-02-10T04:12:24.796 INFO:teuthology.task.ansible.out: 2021-02-10T04:12:24.797 INFO:teuthology.task.ansible.out:TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:24.833 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:24 +0000 (0:00:00.046) 0:01:28.467 **** 2021-02-10T04:12:24.840 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:24.876 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:24 +0000 (0:00:00.043) 0:01:28.511 **** 2021-02-10T04:12:25.364 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-02-10T04:12:25.365 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.341386", "end": "2021-02-10 04:12:25.339525", "rc": 0, "start": "2021-02-10 04:12:24.998139", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-02-10T04:12:25.402 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:25 +0000 (0:00:00.526) 0:01:29.037 **** 2021-02-10T04:12:25.412 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.340130", "end": "2021-02-10 04:12:25.388508", "rc": 0, "start": "2021-02-10 04:12:25.048378", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-02-10T04:12:25.450 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:25 +0000 (0:00:00.047) 0:01:29.084 **** 2021-02-10T04:12:25.457 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:25.497 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:25 +0000 (0:00:00.047) 0:01:29.132 **** 2021-02-10T04:12:25.507 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:25.547 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:25 +0000 (0:00:00.049) 0:01:29.181 **** 2021-02-10T04:12:25.554 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-02-10T04:12:25.556 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:25.594 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:25 +0000 (0:00:00.046) 0:01:29.228 **** 2021-02-10T04:12:25.601 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:25.641 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:25 +0000 (0:00:00.047) 0:01:29.275 **** 2021-02-10T04:12:25.647 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-02-10T04:12:25.649 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:25.688 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:25 +0000 (0:00:00.046) 0:01:29.322 **** 2021-02-10T04:12:25.697 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:25.736 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:25 +0000 (0:00:00.048) 0:01:29.370 **** 2021-02-10T04:12:25.744 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-02-10T04:12:25.745 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:25.781 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:25 +0000 (0:00:00.045) 0:01:29.416 **** 2021-02-10T04:12:25.788 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:25.825 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:25 +0000 (0:00:00.043) 0:01:29.459 **** 2021-02-10T04:12:25.831 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-02-10T04:12:25.831 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:25.869 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:25 +0000 (0:00:00.043) 0:01:29.503 **** 2021-02-10T04:12:25.877 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:25.915 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:25 +0000 (0:00:00.046) 0:01:29.549 **** 2021-02-10T04:12:25.921 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-02-10T04:12:25.923 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"ansible_facts": {"entitlements_path": "/etc/ansible/secrets/entitlements.yml"}, "changed": false} 2021-02-10T04:12:25.960 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:25 +0000 (0:00:00.045) 0:01:29.594 **** 2021-02-10T04:12:25.967 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"entitlements_path": "/etc/ansible/secrets/entitlements.yml"}, "changed": false} 2021-02-10T04:12:26.006 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:26 +0000 (0:00:00.045) 0:01:29.640 **** 2021-02-10T04:12:26.013 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-02-10T04:12:26.013 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-10T04:12:26.014 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-10T04:12:26.052 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:26 +0000 (0:00:00.045) 0:01:29.686 **** 2021-02-10T04:12:26.059 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-10T04:12:26.060 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-10T04:12:26.101 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:26 +0000 (0:00:00.048) 0:01:29.735 **** 2021-02-10T04:12:26.108 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-02-10T04:12:26.109 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"ansible_facts": {"have_entitlements": true}, "changed": false} 2021-02-10T04:12:26.153 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:26 +0000 (0:00:00.052) 0:01:29.787 **** 2021-02-10T04:12:26.162 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"have_entitlements": true}, "changed": false} 2021-02-10T04:12:26.197 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:26 +0000 (0:00:00.043) 0:01:29.831 **** 2021-02-10T04:12:27.368 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-02-10T04:12:27.369 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed /root/.ansible/tmp/ansible-moduletmp-1612930347.2372625-vwo2d6oi/katello-ca-consumer-latest.noarchzyr3tb62.rpm"]} ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed /root/.ansible/tmp/ansible-moduletmp-1612930347.2434947-_1s4opqf/katello-ca-consumer-latest.noarchiy91q620.rpm"]} 2021-02-10T04:12:27.402 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:27 +0000 (0:00:01.205) 0:01:31.036 **** 2021-02-10T04:12:27.434 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:27 +0000 (0:00:00.031) 0:01:31.068 **** 2021-02-10T04:12:27.947 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-02-10T04:12:27.949 INFO:teuthology.task.ansible.out:fatal: [smithi085.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ...ignoring 2021-02-10T04:12:27.994 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:27 +0000 (0:00:00.559) 0:01:31.628 **** 2021-02-10T04:12:28.022 INFO:teuthology.task.ansible.out:fatal: [smithi164.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ...ignoring 2021-02-10T04:12:28.058 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:28 +0000 (0:00:00.064) 0:01:31.693 **** 2021-02-10T04:12:28.067 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-02-10T04:12:28.069 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": false}, "changed": false} 2021-02-10T04:12:28.119 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:28 +0000 (0:00:00.060) 0:01:31.753 **** 2021-02-10T04:12:28.127 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": false}, "changed": false} 2021-02-10T04:12:28.169 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:28 +0000 (0:00:00.050) 0:01:31.803 **** 2021-02-10T04:12:41.728 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-02-10T04:12:41.730 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"attempts": 1, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-02-10T04:12:41.787 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:41 +0000 (0:00:13.617) 0:01:45.421 **** 2021-02-10T04:12:41.839 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-02-10T04:12:41.841 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": true}, "changed": false} 2021-02-10T04:12:41.874 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:41 +0000 (0:00:00.087) 0:01:45.508 **** 2021-02-10T04:12:42.436 INFO:teuthology.task.ansible.out: 2021-02-10T04:12:42.438 INFO:teuthology.task.ansible.out:TASK [common : List CDN releases available to system] ************************** ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager release --list | grep -E '[0-9]'", "delta": "0:00:00.358432", "end": "2021-02-10 04:12:42.410909", "failed_when_result": false, "rc": 0, "start": "2021-02-10 04:12:42.052477", "stderr": "", "stderr_lines": [], "stdout": "7.4\n7.5\n7.6\n7.7\n7.8\n7.9\n7Server\n8\n8.0\n8.1\n8.2\n8.3", "stdout_lines": ["7.4", "7.5", "7.6", "7.7", "7.8", "7.9", "7Server", "8", "8.0", "8.1", "8.2", "8.3"]} 2021-02-10T04:12:42.473 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:42 +0000 (0:00:00.598) 0:01:46.107 **** 2021-02-10T04:12:42.511 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-02-10T04:12:42.512 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:42.546 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:42 +0000 (0:00:00.073) 0:01:46.180 **** 2021-02-10T04:12:42.584 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-02-10T04:12:42.586 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:42.621 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:42 +0000 (0:00:00.075) 0:01:46.256 **** 2021-02-10T04:12:42.650 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-02-10T04:12:42.651 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-10T04:12:42.695 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:42 +0000 (0:00:00.073) 0:01:46.330 **** 2021-02-10T04:12:42.748 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-02-10T04:12:42.750 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:42.786 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:42 +0000 (0:00:00.090) 0:01:46.420 **** 2021-02-10T04:12:43.012 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** changed: [smithi085.front.sepia.ceph.com] => {"attempts": 1, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-02-10T04:12:43.069 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:43 +0000 (0:00:00.282) 0:01:46.703 **** 2021-02-10T04:12:43.109 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-02-10T04:12:43.110 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": true}, "changed": false} 2021-02-10T04:12:43.144 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:43 +0000 (0:00:00.075) 0:01:46.778 **** 2021-02-10T04:12:43.712 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-02-10T04:12:43.713 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager release --list | grep -E '[0-9]'", "delta": "0:00:00.370520", "end": "2021-02-10 04:12:43.686267", "failed_when_result": false, "rc": 0, "start": "2021-02-10 04:12:43.315747", "stderr": "", "stderr_lines": [], "stdout": "7.4\n7.5\n7.6\n7.7\n7.8\n7.9\n7Server\n8\n8.0\n8.1\n8.2\n8.3", "stdout_lines": ["7.4", "7.5", "7.6", "7.7", "7.8", "7.9", "7Server", "8", "8.0", "8.1", "8.2", "8.3"]} 2021-02-10T04:12:43.786 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:43 +0000 (0:00:00.641) 0:01:47.420 **** 2021-02-10T04:12:43.835 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-02-10T04:12:43.836 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:43.897 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:43 +0000 (0:00:00.111) 0:01:47.532 **** 2021-02-10T04:12:43.956 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-02-10T04:12:43.957 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:43.997 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:43 +0000 (0:00:00.100) 0:01:47.632 **** 2021-02-10T04:12:43.998 INFO:teuthology.task.ansible.out: 2021-02-10T04:12:44.029 INFO:teuthology.task.ansible.out: 2021-02-10T04:12:44.030 INFO:teuthology.task.ansible.out:TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi085.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-10T04:12:44.065 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:44 +0000 (0:00:00.067) 0:01:47.700 **** 2021-02-10T04:12:44.067 INFO:teuthology.task.ansible.out: 2021-02-10T04:12:44.129 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-02-10T04:12:44.131 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:44.168 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:44 +0000 (0:00:00.102) 0:01:47.802 **** 2021-02-10T04:12:45.348 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-02-10T04:12:45.350 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager repos --list | grep -B4 'Enabled:.*1' | grep 'Repo ID:' | sed -e 's/Repo ID:\\s*\\(.*\\)/\\1/' | sort", "delta": "0:00:02.322620", "end": "2021-02-10 04:12:45.328264", "rc": 0, "start": "2021-02-10 04:12:43.005644", "stderr": "", "stderr_lines": [], "stdout": "rhel-8-for-x86_64-appstream-rpms\nrhel-8-for-x86_64-baseos-rpms", "stdout_lines": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]} 2021-02-10T04:12:45.388 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:45 +0000 (0:00:01.219) 0:01:49.022 **** 2021-02-10T04:12:45.437 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-02-10T04:12:45.438 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"repo_list": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]}, "changed": false} 2021-02-10T04:12:45.473 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:45 +0000 (0:00:00.085) 0:01:49.107 **** 2021-02-10T04:12:45.511 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-02-10T04:12:45.512 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:45.547 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:45 +0000 (0:00:00.073) 0:01:49.181 **** 2021-02-10T04:12:45.587 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-02-10T04:12:45.589 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": false}, "changed": false} 2021-02-10T04:12:45.625 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:45 +0000 (0:00:00.077) 0:01:49.259 **** 2021-02-10T04:12:45.664 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-02-10T04:12:45.666 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2021-02-10T04:12:45.705 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:45 +0000 (0:00:00.079) 0:01:49.339 **** 2021-02-10T04:12:46.739 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-02-10T04:12:46.740 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager repos --list | grep -B4 'Enabled:.*1' | grep 'Repo ID:' | sed -e 's/Repo ID:\\s*\\(.*\\)/\\1/' | sort", "delta": "0:00:02.349276", "end": "2021-02-10 04:12:46.717485", "rc": 0, "start": "2021-02-10 04:12:44.368209", "stderr": "", "stderr_lines": [], "stdout": "rhel-8-for-x86_64-appstream-rpms\nrhel-8-for-x86_64-baseos-rpms", "stdout_lines": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]} 2021-02-10T04:12:46.780 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:46 +0000 (0:00:01.075) 0:01:50.414 **** 2021-02-10T04:12:46.829 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-02-10T04:12:46.831 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"ansible_facts": {"repo_list": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]}, "changed": false} 2021-02-10T04:12:46.866 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:46 +0000 (0:00:00.085) 0:01:50.500 **** 2021-02-10T04:12:46.917 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-02-10T04:12:46.919 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:46.964 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:46 +0000 (0:00:00.097) 0:01:50.598 **** 2021-02-10T04:12:47.002 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-02-10T04:12:47.004 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": false}, "changed": false} 2021-02-10T04:12:47.049 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:47 +0000 (0:00:00.085) 0:01:50.683 **** 2021-02-10T04:12:47.087 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-02-10T04:12:47.089 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2021-02-10T04:12:47.121 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:47 +0000 (0:00:00.071) 0:01:50.755 **** 2021-02-10T04:12:48.144 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-02-10T04:12:48.145 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-02-10T04:12:48.192 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:48 +0000 (0:00:01.071) 0:01:51.827 **** 2021-02-10T04:12:49.102 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-02-10T04:12:49.139 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:49 +0000 (0:00:00.947) 0:01:52.774 **** 2021-02-10T04:12:51.257 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-02-10T04:12:51.259 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": ["subscription-manager", "repos", "--enable=rhel-8-for-x86_64-baseos-rpms", "--enable=rhel-8-for-x86_64-appstream-rpms"], "delta": "0:00:02.861679", "end": "2021-02-10 04:12:51.235826", "rc": 0, "start": "2021-02-10 04:12:48.374147", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.\nRepository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.", "Repository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system."]} 2021-02-10T04:12:51.299 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:51 +0000 (0:00:02.160) 0:01:54.934 **** 2021-02-10T04:12:51.794 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-02-10T04:12:51.796 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "path": "/etc/yum.repos.d/cd.repo", "state": "absent"} 2021-02-10T04:12:51.842 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:51 +0000 (0:00:00.542) 0:01:55.477 **** 2021-02-10T04:12:52.037 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-02-10T04:12:52.039 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T04:12:52.072 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:52 +0000 (0:00:00.229) 0:01:55.706 **** 2021-02-10T04:12:52.251 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-02-10T04:12:52.252 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => {"changed": true, "cmd": ["subscription-manager", "repos", "--enable=rhel-8-for-x86_64-baseos-rpms", "--enable=rhel-8-for-x86_64-appstream-rpms"], "delta": "0:00:02.922895", "end": "2021-02-10 04:12:52.232519", "rc": 0, "start": "2021-02-10 04:12:49.309624", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.\nRepository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.", "Repository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system."]} 2021-02-10T04:12:52.291 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:52 +0000 (0:00:00.219) 0:01:55.926 **** 2021-02-10T04:12:52.491 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-02-10T04:12:52.492 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "path": "/etc/yum.repos.d/cd.repo", "state": "absent"} 2021-02-10T04:12:52.528 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:52 +0000 (0:00:00.236) 0:01:56.162 **** 2021-02-10T04:12:52.556 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-02-10T04:12:52.557 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-02-10T04:12:52.717 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-02-10T04:12:52.718 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T04:12:52.753 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:52 +0000 (0:00:00.224) 0:01:56.387 **** 2021-02-10T04:12:52.972 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-02-10T04:12:52.973 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-02-10T04:12:53.009 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:53 +0000 (0:00:00.256) 0:01:56.643 **** 2021-02-10T04:12:53.231 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-02-10T04:12:53.464 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-02-10T04:12:53.466 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-02-10T04:12:53.691 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-02-10T04:12:53.692 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-02-10T04:12:53.743 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:53 +0000 (0:00:00.733) 0:01:57.377 **** 2021-02-10T04:12:53.987 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* ok: [smithi164.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-02-10T04:12:54.024 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:54 +0000 (0:00:00.281) 0:01:57.659 **** 2021-02-10T04:12:54.073 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:54.110 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:54 +0000 (0:00:00.085) 0:01:57.744 **** 2021-02-10T04:12:54.138 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-02-10T04:12:54.140 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:54.178 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:54 +0000 (0:00:00.067) 0:01:57.812 **** 2021-02-10T04:12:54.205 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:54.239 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:54 +0000 (0:00:00.061) 0:01:57.874 **** 2021-02-10T04:12:54.256 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-02-10T04:12:54.257 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-02-10T04:12:54.270 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:54.307 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:54 +0000 (0:00:00.067) 0:01:57.942 **** 2021-02-10T04:12:54.348 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** skipping: [smithi164.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-02-10T04:12:54.359 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-02-10T04:12:54.359 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-02-10T04:12:54.371 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-02-10T04:12:54.417 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:54 +0000 (0:00:00.109) 0:01:58.051 **** 2021-02-10T04:12:54.445 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-02-10T04:12:54.446 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:54.480 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:54 +0000 (0:00:00.063) 0:01:58.114 **** 2021-02-10T04:12:54.519 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-02-10T04:12:54.521 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:54.556 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:54 +0000 (0:00:00.074) 0:01:58.189 **** 2021-02-10T04:12:54.598 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:54.637 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:54 +0000 (0:00:00.082) 0:01:58.272 **** 2021-02-10T04:12:54.676 INFO:teuthology.task.ansible.out: 2021-02-10T04:12:54.676 INFO:teuthology.task.ansible.out:TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-02-10T04:12:54.678 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:54.725 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:54 +0000 (0:00:00.087) 0:01:58.359 **** 2021-02-10T04:12:54.732 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-02-10T04:12:54.734 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-02-10T04:12:54.774 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:54 +0000 (0:00:00.048) 0:01:58.408 **** 2021-02-10T04:12:54.780 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-02-10T04:12:54.781 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:54.818 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:54 +0000 (0:00:00.044) 0:01:58.452 **** 2021-02-10T04:12:54.828 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-02-10T04:12:54.830 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:54.889 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:54 +0000 (0:00:00.070) 0:01:58.523 **** 2021-02-10T04:12:54.920 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-02-10T04:12:54.921 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:54.962 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:54 +0000 (0:00:00.073) 0:01:58.596 **** 2021-02-10T04:12:54.990 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-02-10T04:12:54.992 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:55.029 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:55 +0000 (0:00:00.067) 0:01:58.664 **** 2021-02-10T04:12:55.068 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-02-10T04:12:55.071 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:55.105 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:55 +0000 (0:00:00.075) 0:01:58.739 **** 2021-02-10T04:12:55.143 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-02-10T04:12:55.145 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-02-10T04:12:55.155 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-02-10T04:12:55.155 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-02-10T04:12:55.167 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-02-10T04:12:55.202 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:55 +0000 (0:00:00.097) 0:01:58.836 **** 2021-02-10T04:12:55.230 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-02-10T04:12:55.232 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:55.267 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:55 +0000 (0:00:00.065) 0:01:58.901 **** 2021-02-10T04:12:55.296 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-02-10T04:12:55.297 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:55.330 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:55 +0000 (0:00:00.063) 0:01:58.965 **** 2021-02-10T04:12:55.359 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-02-10T04:12:55.360 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:55.397 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:55 +0000 (0:00:00.066) 0:01:59.031 **** 2021-02-10T04:12:55.435 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-02-10T04:12:55.436 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:55.471 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:55 +0000 (0:00:00.074) 0:01:59.106 **** 2021-02-10T04:12:55.510 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-02-10T04:12:55.512 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:55.545 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:55 +0000 (0:00:00.073) 0:01:59.179 **** 2021-02-10T04:12:55.552 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-02-10T04:12:55.554 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.503465", "end": "2021-02-10 04:12:55.500726", "rc": 0, "start": "2021-02-10 04:12:54.997261", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T04:12:55.597 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:55 +0000 (0:00:00.052) 0:01:59.231 **** 2021-02-10T04:12:57.527 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:01.791499", "end": "2021-02-10 04:12:57.501422", "rc": 0, "start": "2021-02-10 04:12:55.709923", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T04:12:57.570 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:57 +0000 (0:00:01.973) 0:02:01.205 **** 2021-02-10T04:12:58.474 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-02-10T04:12:58.474 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: krb5-workstation"]} 2021-02-10T04:12:58.512 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:58 +0000 (0:00:00.941) 0:02:02.147 **** 2021-02-10T04:12:58.539 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-02-10T04:12:58.540 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:58.576 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:58 +0000 (0:00:00.063) 0:02:02.210 **** 2021-02-10T04:12:58.604 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-02-10T04:12:58.606 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:58.639 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:58 +0000 (0:00:00.063) 0:02:02.273 **** 2021-02-10T04:12:58.666 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-02-10T04:12:58.667 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:58.706 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:58 +0000 (0:00:00.067) 0:02:02.340 **** 2021-02-10T04:12:58.734 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-02-10T04:12:58.735 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:12:58.773 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:58 +0000 (0:00:00.066) 0:02:02.407 **** 2021-02-10T04:12:59.230 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-02-10T04:12:59.232 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-02-10T04:12:59.306 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:12:59 +0000 (0:00:00.531) 0:02:02.939 **** 2021-02-10T04:13:00.433 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-02-10T04:13:00.434 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: krb5-workstation"]} 2021-02-10T04:13:00.473 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:00 +0000 (0:00:01.169) 0:02:04.108 **** 2021-02-10T04:13:00.512 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-02-10T04:13:00.514 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:13:00.547 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:00 +0000 (0:00:00.073) 0:02:04.181 **** 2021-02-10T04:13:00.578 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-02-10T04:13:00.579 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:13:00.612 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:00 +0000 (0:00:00.065) 0:02:04.246 **** 2021-02-10T04:13:00.639 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:13:00.674 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:00 +0000 (0:00:00.062) 0:02:04.308 **** 2021-02-10T04:13:00.702 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-02-10T04:13:00.704 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:13:00.740 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:00 +0000 (0:00:00.065) 0:02:04.374 **** 2021-02-10T04:13:01.179 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-02-10T04:13:01.180 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-02-10T04:13:01.216 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:01 +0000 (0:00:00.475) 0:02:04.850 **** 2021-02-10T04:13:01.789 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-02-10T04:13:01.790 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: smartmontools"]} 2021-02-10T04:13:01.825 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:01 +0000 (0:00:00.609) 0:02:05.460 **** 2021-02-10T04:13:02.336 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-02-10T04:13:02.337 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-02-10T04:13:02.767 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-02-10T04:13:03.211 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-02-10T04:13:03.253 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:03 +0000 (0:00:01.427) 0:02:06.888 **** 2021-02-10T04:13:03.440 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-02-10T04:13:03.441 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T04:13:03.473 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:03 +0000 (0:00:00.220) 0:02:07.108 **** 2021-02-10T04:13:03.669 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-02-10T04:13:03.671 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: smartmontools"]} 2021-02-10T04:13:03.708 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:03 +0000 (0:00:00.234) 0:02:07.342 **** 2021-02-10T04:13:04.015 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-02-10T04:13:05.697 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": true, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "md5sum": "14041bf2666d8f8d277567958c7e9f69", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612930383.4997385-76105494300740/source", "state": "file", "uid": 0} TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [smithi085.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** ok: [smithi164.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [smithi085.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} TASK [common : Upload custom netsaint scripts for 2021-02-10T04:13:05.699 INFO:teuthology.task.ansible.out:raid/disk/smart/monitoring to /usr/libexec/.] *** ok: [smithi164.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} Wednesday 10 February 2021 04:13:04 +0000 (0:00:01.185) 0:02:08.528 **** TASK [common : Include apt_system vars] **************************************** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 04:13:04 +0000 (0:00:00.069) 0:02:08.597 **** TASK [common : Include yum_systems vars] *************************************** ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} Wednesday 10 February 2021 04:13:05 +0000 (0:00:00.081) 0:02:08.678 **** TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [smithi085.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} Wednesday 10 February 2021 04:13:05 +0000 (0:00:00.066) 0:02:08.745 **** TASK [common : Check for epel] ***** 2021-02-10T04:13:05.700 INFO:teuthology.task.ansible.out:******************************************** changed: [smithi164.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} Wednesday 10 February 2021 04:13:05 +0000 (0:00:00.151) 0:02:08.897 **** TASK [common : Create /usr/libexec.] ******************************************* ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} TASK [common : Install nrpe without epel] ************************************** skipping: [smithi164.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 04:13:05 +0000 (0:00:00.085) 0:02:08.982 **** skipping: [smithi164.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7 2021-02-10T04:13:05.702 INFO:teuthology.task.ansible.out:.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 04:13:05 +0000 (0:00:00.056) 0:02:09.039 **** 2021-02-10T04:13:05.881 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-02-10T04:13:05.882 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": true, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "md5sum": "14041bf2666d8f8d277567958c7e9f69", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612930385.372942-138986292432665/source", "state": "file", "uid": 0} 2021-02-10T04:13:06.334 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-02-10T04:13:06.753 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2021-02-10T04:13:06.791 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:06 +0000 (0:00:01.384) 0:02:10.424 **** 2021-02-10T04:13:06.819 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-02-10T04:13:06.821 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:13:06.860 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:06 +0000 (0:00:00.070) 0:02:10.495 **** 2021-02-10T04:13:06.901 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-02-10T04:13:06.902 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-02-10T04:13:06.935 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:06 +0000 (0:00:00.075) 0:02:10.570 **** 2021-02-10T04:13:07.134 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-02-10T04:13:07.135 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-02-10T04:13:07.167 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:07 +0000 (0:00:00.231) 0:02:10.802 **** 2021-02-10T04:13:07.240 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-02-10T04:13:07.242 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-10T04:13:07.252 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-10T04:13:07.263 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-10T04:13:07.274 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-10T04:13:07.312 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:07 +0000 (0:00:00.144) 0:02:10.946 **** 2021-02-10T04:13:07.953 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-02-10T04:13:07.954 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: nagios-common", "Installed: nrpe", "Installed: nagios-plugins", "Installed: nagios-plugins-load"]} 2021-02-10T04:13:07.989 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:07 +0000 (0:00:00.677) 0:02:11.624 **** 2021-02-10T04:13:08.018 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-02-10T04:13:08.020 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:13:08.056 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:08 +0000 (0:00:00.066) 0:02:11.690 **** 2021-02-10T04:13:08.515 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-02-10T04:13:08.516 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-02-10T04:13:08.552 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:08 +0000 (0:00:00.496) 0:02:12.187 **** 2021-02-10T04:13:09.002 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-02-10T04:13:09.003 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-02-10T04:13:09.042 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:09 +0000 (0:00:00.489) 0:02:12.676 **** 2021-02-10T04:13:09.080 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-02-10T04:13:09.081 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-02-10T04:13:09.116 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:09 +0000 (0:00:00.074) 0:02:12.751 **** 2021-02-10T04:13:09.322 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-02-10T04:13:09.324 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T04:13:09.360 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:09 +0000 (0:00:00.244) 0:02:12.995 **** 2021-02-10T04:13:09.651 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-02-10T04:13:09.653 INFO:teuthology.task.ansible.out:fatal: [smithi164.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-02-10T04:13:09.703 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:09 +0000 (0:00:00.342) 0:02:13.337 **** 2021-02-10T04:13:09.756 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:13:09.795 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:09 +0000 (0:00:00.092) 0:02:13.430 **** 2021-02-10T04:13:09.859 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-02-10T04:13:09.861 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: nagios-common", "Installed: nrpe", "Installed: nagios-plugins", "Installed: nagios-plugins-load"]} 2021-02-10T04:13:09.909 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:09 +0000 (0:00:00.113) 0:02:13.543 **** 2021-02-10T04:13:09.937 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-02-10T04:13:09.939 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:13:09.995 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:09 +0000 (0:00:00.086) 0:02:13.629 **** 2021-02-10T04:13:10.271 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-02-10T04:13:10.273 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-02-10T04:13:10.312 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:10 +0000 (0:00:00.316) 0:02:13.946 **** 2021-02-10T04:13:10.460 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-02-10T04:13:10.461 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-02-10T04:13:10.495 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:10 +0000 (0:00:00.183) 0:02:14.130 **** 2021-02-10T04:13:10.956 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-02-10T04:13:10.957 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-02-10T04:13:11.004 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:11 +0000 (0:00:00.508) 0:02:14.638 **** 2021-02-10T04:13:11.057 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-02-10T04:13:11.059 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-02-10T04:13:11.069 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-02-10T04:13:11.115 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:11 +0000 (0:00:00.111) 0:02:14.750 **** 2021-02-10T04:13:11.267 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-02-10T04:13:11.269 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2021-02-10 04:09:44 UTC", "ActiveEnterTimestampMonotonic": "15951112", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target basic.target time-sync.target local-fs.target nss-lookup.target tmp.mount network.target system.slice systemd-tmpfiles-setup.service systemd-journald.socket -.mount var-run.mount", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-02-10 04:09:44 UTC", "AssertTimestampMonotonic": "15936457", "Before": "plymouth-quit.service xdm.service shutdown.target multi-user.target getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-02-10 04:09:44 UTC", "ConditionTimestampMonotonic": "15936457", "ConfigurationDirectoryMode": " 2021-02-10T04:13:11.271 INFO:teuthology.task.ansible.out:0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "980", "ExecMainStartTimestamp": "Wed 2021-02-10 04:09:44 UTC", "ExecMainStartTimestampMonotonic": "15950542", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[Wed 2021-02-10 04:09:44 UTC] ; stop_time=[n/a] ; pid=980 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "running", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-02-10 04:09:44 UTC", "InactiveExitTimestampMonotonic": "15951112", "InvocationID": "9e817928077f4f2388f7fe8abd007a01", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infin 2021-02-10T04:13:11.272 INFO:teuthology.task.ansible.out:ity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127237", "LimitNPROCSoft": "127237", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127237", "LimitSIGPENDINGSoft": "127237", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "980", "MemoryAccounting": "yes", "MemoryCurrent": "1396736", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualSta 2021-02-10T04:13:11.273 INFO:teuthology.task.ansible.out:rt": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2021-02-10 04:09:44 UTC", "StateChangeTimestampMonotonic": "15951112", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203580", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2021-02-10 04:09:44 UTC", "WatchdogTimestampMonotonic": "15951091", "WatchdogUSec": "0"}} 2021-02-10T04:13:11.317 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:11 +0000 (0:00:00.201) 0:02:14.952 **** 2021-02-10T04:13:11.335 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-02-10T04:13:11.337 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T04:13:11.372 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:11 +0000 (0:00:00.054) 0:02:15.006 **** 2021-02-10T04:13:11.507 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-02-10T04:13:11.509 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001731", "end": "2021-02-10 04:13:11.484431", "rc": 0, "start": "2021-02-10 04:13:11.482700", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-02-10T04:13:11.544 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:11 +0000 (0:00:00.171) 0:02:15.178 **** 2021-02-10T04:13:11.647 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-02-10T04:13:11.648 INFO:teuthology.task.ansible.out:fatal: [smithi085.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-02-10T04:13:11.706 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:11 +0000 (0:00:00.161) 0:02:15.340 **** 2021-02-10T04:13:11.737 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-02-10T04:13:11.738 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:13:11.772 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:11 +0000 (0:00:00.065) 0:02:15.406 **** 2021-02-10T04:13:12.233 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-02-10T04:13:12.267 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:12 +0000 (0:00:00.495) 0:02:15.901 **** 2021-02-10T04:13:12.563 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-02-10T04:13:12.564 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2021-02-10 04:09:34 UTC", "ActiveEnterTimestampMonotonic": "15566033", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "basic.target systemd-journald.socket system.slice tmp.mount -.mount network.target time-sync.target nss-lookup.target systemd-tmpfiles-setup.service sysinit.target local-fs.target var-run.mount", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-02-10 04:09:34 UTC", "AssertTimestampMonotonic": "15559786", "Before": "multi-user.target plymouth-quit.service getty@tty1.service shutdown.target xdm.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-02-10 04:09:34 UTC", "ConditionTimestampMonotonic": "15559785", "ConfigurationDirectoryMode": " 2021-02-10T04:13:12.565 INFO:teuthology.task.ansible.out:0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "983", "ExecMainStartTimestamp": "Wed 2021-02-10 04:09:34 UTC", "ExecMainStartTimestampMonotonic": "15565965", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[Wed 2021-02-10 04:09:34 UTC] ; stop_time=[n/a] ; pid=983 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "running", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-02-10 04:09:34 UTC", "InactiveExitTimestampMonotonic": "15566033", "InvocationID": "b6bee0c628524d799aa38e8b69d6e3ba", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infin 2021-02-10T04:13:12.566 INFO:teuthology.task.ansible.out:ity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127237", "LimitNPROCSoft": "127237", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127237", "LimitSIGPENDINGSoft": "127237", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "983", "MemoryAccounting": "yes", "MemoryCurrent": "1372160", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualSta 2021-02-10T04:13:12.568 INFO:teuthology.task.ansible.out:rt": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2021-02-10 04:09:34 UTC", "StateChangeTimestampMonotonic": "15566033", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203580", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2021-02-10 04:09:34 UTC", "WatchdogTimestampMonotonic": "15566032", "WatchdogUSec": "0"}} 2021-02-10T04:13:12.602 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:12 +0000 (0:00:00.334) 0:02:16.236 **** 2021-02-10T04:13:12.788 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-02-10T04:13:12.789 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001709", "end": "2021-02-10 04:13:12.763600", "rc": 0, "start": "2021-02-10 04:13:12.761891", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-02-10T04:13:12.822 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:12 +0000 (0:00:00.220) 0:02:16.456 **** 2021-02-10T04:13:14.050 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-02-10T04:13:14.052 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: python3-libsemanage", "Installed: python3-policycoreutils"]} 2021-02-10T04:13:14.087 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:14 +0000 (0:00:01.264) 0:02:17.721 **** 2021-02-10T04:13:15.325 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: python3-libsemanage", "Installed: python3-policycoreutils"]} 2021-02-10T04:13:15.519 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:15 +0000 (0:00:01.277) 0:02:18.999 **** 2021-02-10T04:13:55.251 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-02-10T04:13:55.253 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-targeted", "Installed: selinux-policy-3.14.3-54.el8_3.2.noarch", "Installed: selinux-policy-targeted-3.14.3-54.el8_3.2.noarch", "Removed: selinux-policy-3.14.3-54.el8.noarch", "Removed: selinux-policy-targeted-3.14.3-54.el8.noarch"]} 2021-02-10T04:13:55.289 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:55 +0000 (0:00:39.924) 0:02:58.923 **** 2021-02-10T04:13:56.009 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-02-10T04:13:56.011 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-02-10T04:13:56.044 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:56 +0000 (0:00:00.755) 0:02:59.679 **** 2021-02-10T04:13:56.261 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-02-10T04:13:56.263 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-targeted", "Installed: selinux-policy-3.14.3-54.el8_3.2.noarch", "Installed: selinux-policy-targeted-3.14.3-54.el8_3.2.noarch", "Removed: selinux-policy-3.14.3-54.el8.noarch", "Removed: selinux-policy-targeted-3.14.3-54.el8.noarch"]} 2021-02-10T04:13:56.316 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:56 +0000 (0:00:00.271) 0:02:59.951 **** 2021-02-10T04:13:56.802 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-02-10T04:13:56.803 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-02-10T04:13:56.838 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:13:56 +0000 (0:00:00.521) 0:03:00.473 **** 2021-02-10T04:14:12.478 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-02-10T04:14:12.480 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.239068", "end": "2021-02-10 04:14:12.449029", "failed_when_result": false, "rc": 0, "start": "2021-02-10 04:13:56.209961", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-02-10T04:14:12.514 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:12 +0000 (0:00:15.675) 0:03:16.148 **** 2021-02-10T04:14:12.968 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-02-10T04:14:12.970 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-02-10T04:14:13.004 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:13 +0000 (0:00:00.489) 0:03:16.638 **** 2021-02-10T04:14:13.160 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-02-10T04:14:13.161 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.131115", "end": "2021-02-10 04:14:13.136519", "failed_when_result": false, "rc": 0, "start": "2021-02-10 04:13:57.005404", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-02-10T04:14:13.197 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:13 +0000 (0:00:00.193) 0:03:16.831 **** 2021-02-10T04:14:13.204 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-02-10T04:14:13.205 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.017911", "end": "2021-02-10 04:14:13.183694", "rc": 0, "start": "2021-02-10 04:14:13.165783", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T04:14:13.241 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:13 +0000 (0:00:00.043) 0:03:16.875 **** 2021-02-10T04:14:13.426 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-02-10T04:14:13.428 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.007238", "end": "2021-02-10 04:14:13.408189", "rc": 0, "start": "2021-02-10 04:14:13.400951", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T04:14:13.462 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:13 +0000 (0:00:00.220) 0:03:17.096 **** 2021-02-10T04:14:13.647 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-02-10T04:14:13.648 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-02-10T04:14:13.685 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:13 +0000 (0:00:00.223) 0:03:17.319 **** 2021-02-10T04:14:13.891 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-02-10T04:14:13.893 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.019049", "end": "2021-02-10 04:14:13.870260", "rc": 0, "start": "2021-02-10 04:14:13.851211", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T04:14:13.928 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:13 +0000 (0:00:00.243) 0:03:17.563 **** 2021-02-10T04:14:14.114 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-02-10T04:14:14.115 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.007328", "end": "2021-02-10 04:14:14.098408", "rc": 0, "start": "2021-02-10 04:14:14.091080", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T04:14:14.148 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:14 +0000 (0:00:00.219) 0:03:17.782 **** 2021-02-10T04:14:29.920 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-02-10T04:14:29.921 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.268017", "end": "2021-02-10 04:14:29.900158", "rc": 0, "start": "2021-02-10 04:14:13.632141", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T04:14:29.956 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:29 +0000 (0:00:15.808) 0:03:33.591 **** 2021-02-10T04:14:30.152 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-02-10T04:14:30.153 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-02-10T04:14:30.186 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:30 +0000 (0:00:00.229) 0:03:33.821 **** 2021-02-10T04:14:30.224 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-02-10T04:14:30.225 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-02-10T04:14:30.259 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:30 +0000 (0:00:00.072) 0:03:33.893 **** 2021-02-10T04:14:30.311 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-02-10T04:14:30.312 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-02-10T04:14:30.346 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:30 +0000 (0:00:00.087) 0:03:33.980 **** 2021-02-10T04:14:30.387 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-02-10T04:14:30.388 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "epel_packages": ["dbench"], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["dnf-utils", "git-all", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "podman", "redhat-lsb", "firewalld", "wget", "podman-docker", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "make", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "autoconf", "gdb", "iozone", "lvm2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"} 2021-02-10T04:14:30.422 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:30 +0000 (0:00:00.076) 0:03:34.057 **** 2021-02-10T04:14:30.462 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-02-10T04:14:30.463 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-02-10T04:14:30.496 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:30 +0000 (0:00:00.073) 0:03:34.130 **** 2021-02-10T04:14:30.503 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-02-10T04:14:30.504 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.140836", "end": "2021-02-10 04:14:30.448943", "rc": 0, "start": "2021-02-10 04:14:14.308107", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T04:14:30.544 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:30 +0000 (0:00:00.047) 0:03:34.178 **** 2021-02-10T04:14:30.688 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-02-10T04:14:30.690 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-02-10T04:14:30.724 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:30 +0000 (0:00:00.180) 0:03:34.358 **** 2021-02-10T04:14:30.741 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-02-10T04:14:30.743 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-02-10T04:14:30.779 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:30 +0000 (0:00:00.054) 0:03:34.413 **** 2021-02-10T04:14:30.807 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-02-10T04:14:30.808 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-02-10T04:14:30.841 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:30 +0000 (0:00:00.062) 0:03:34.476 **** 2021-02-10T04:14:30.879 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* ok: [smithi085.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-02-10T04:14:30.914 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:30 +0000 (0:00:00.072) 0:03:34.548 **** 2021-02-10T04:14:30.921 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-02-10T04:14:30.921 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "name": "ubuntu", "state": "present", "system": false} 2021-02-10T04:14:30.962 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:30 +0000 (0:00:00.048) 0:03:34.597 **** 2021-02-10T04:14:30.969 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-02-10T04:14:30.970 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "epel_packages": ["dbench"], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["dnf-utils", "git-all", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "podman", "redhat-lsb", "firewalld", "wget", "podman-docker", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "make", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "autoconf", "gdb", "iozone", "lvm2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"} 2021-02-10T04:14:31.009 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:31 +0000 (0:00:00.046) 0:03:34.643 **** 2021-02-10T04:14:31.016 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-02-10T04:14:31.018 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:14:31.057 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:31 +0000 (0:00:00.048) 0:03:34.692 **** 2021-02-10T04:14:31.065 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-02-10T04:14:31.067 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-02-10T04:14:31.102 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:31 +0000 (0:00:00.045) 0:03:34.737 **** 2021-02-10T04:14:31.110 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-02-10T04:14:31.110 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:14:31.149 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:31 +0000 (0:00:00.046) 0:03:34.784 **** 2021-02-10T04:14:31.282 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-02-10T04:14:31.283 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-02-10T04:14:31.318 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:31 +0000 (0:00:00.168) 0:03:34.953 **** 2021-02-10T04:14:31.356 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-02-10T04:14:31.358 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-02-10T04:14:31.394 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:31 +0000 (0:00:00.075) 0:03:35.028 **** 2021-02-10T04:14:31.496 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-02-10T04:14:31.497 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "name": "ubuntu", "state": "present", "system": false} 2021-02-10T04:14:31.532 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:31 +0000 (0:00:00.138) 0:03:35.167 **** 2021-02-10T04:14:31.561 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-02-10T04:14:31.563 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:14:31.597 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:31 +0000 (0:00:00.064) 0:03:35.231 **** 2021-02-10T04:14:31.625 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-02-10T04:14:31.626 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:14:31.659 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:31 +0000 (0:00:00.061) 0:03:35.293 **** 2021-02-10T04:14:31.866 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-02-10T04:14:31.867 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-02-10T04:14:31.901 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:31 +0000 (0:00:00.242) 0:03:35.536 **** 2021-02-10T04:14:31.911 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-02-10T04:14:31.911 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-02-10T04:14:31.974 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:31 +0000 (0:00:00.072) 0:03:35.609 **** 2021-02-10T04:14:32.012 INFO:teuthology.task.ansible.out: 2021-02-10T04:14:32.013 INFO:teuthology.task.ansible.out:TASK [testnode : Set hostname var] ********************************************* ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi164"}, "changed": false} 2021-02-10T04:14:32.052 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:32 +0000 (0:00:00.078) 0:03:35.687 **** 2021-02-10T04:14:32.455 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-02-10T04:14:32.457 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-02-10T04:14:32.507 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:32 +0000 (0:00:00.452) 0:03:36.139 **** 2021-02-10T04:14:32.548 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-02-10T04:14:32.549 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi085"}, "changed": false} 2021-02-10T04:14:32.588 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:32 +0000 (0:00:00.083) 0:03:36.223 **** 2021-02-10T04:14:33.208 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-02-10T04:14:33.209 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi085", "ansible_hostname": "smithi085", "ansible_nodename": "smithi085"}, "changed": false, "name": "smithi085"} ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi164", "ansible_hostname": "smithi164", "ansible_nodename": "smithi164"}, "changed": false, "name": "smithi164"} 2021-02-10T04:14:33.243 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:33 +0000 (0:00:00.654) 0:03:36.877 **** 2021-02-10T04:14:33.277 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:33 +0000 (0:00:00.034) 0:03:36.912 **** 2021-02-10T04:14:33.736 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-02-10T04:14:33.737 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-02-10T04:14:33.783 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:33 +0000 (0:00:00.505) 0:03:37.417 **** 2021-02-10T04:14:33.793 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-02-10T04:14:33.831 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:33 +0000 (0:00:00.047) 0:03:37.465 **** 2021-02-10T04:14:34.229 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-02-10T04:14:34.230 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-02-10T04:14:34.267 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:34 +0000 (0:00:00.436) 0:03:37.901 **** 2021-02-10T04:14:34.286 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-02-10T04:14:34.330 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:34 +0000 (0:00:00.063) 0:03:37.965 **** 2021-02-10T04:14:34.925 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-02-10T04:14:34.927 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-02-10T04:14:34.967 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:34 +0000 (0:00:00.636) 0:03:38.601 **** 2021-02-10T04:14:34.974 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-02-10T04:14:35.012 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:35 +0000 (0:00:00.044) 0:03:38.646 **** 2021-02-10T04:14:35.425 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-02-10T04:14:35.426 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-02-10T04:14:35.472 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:35 +0000 (0:00:00.460) 0:03:39.106 **** 2021-02-10T04:14:35.481 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-02-10T04:14:35.518 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:35 +0000 (0:00:00.046) 0:03:39.152 **** 2021-02-10T04:14:35.662 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-10T04:14:35.663 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-02-10T04:14:35.698 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:35 +0000 (0:00:00.180) 0:03:39.333 **** 2021-02-10T04:14:35.717 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-02-10T04:14:35.754 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:35 +0000 (0:00:00.055) 0:03:39.389 **** 2021-02-10T04:14:35.887 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-02-10T04:14:35.889 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-02-10T04:14:35.926 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:35 +0000 (0:00:00.171) 0:03:39.560 **** 2021-02-10T04:14:35.944 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-02-10T04:14:35.989 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:35 +0000 (0:00:00.063) 0:03:39.624 **** 2021-02-10T04:14:36.146 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-02-10T04:14:36.147 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1236, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-02-10T04:14:36.182 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:36 +0000 (0:00:00.192) 0:03:39.816 **** 2021-02-10T04:14:36.213 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1236, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-02-10T04:14:36.271 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:36 +0000 (0:00:00.089) 0:03:39.905 **** 2021-02-10T04:14:36.745 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-10T04:14:36.746 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-02-10T04:14:36.782 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:36 +0000 (0:00:00.510) 0:03:40.416 **** 2021-02-10T04:14:36.790 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-02-10T04:14:36.827 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:36 +0000 (0:00:00.044) 0:03:40.461 **** 2021-02-10T04:14:37.243 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-02-10T04:14:37.245 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-02-10T04:14:37.281 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:37 +0000 (0:00:00.454) 0:03:40.915 **** 2021-02-10T04:14:37.301 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-02-10T04:14:37.336 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:37 +0000 (0:00:00.055) 0:03:40.970 **** 2021-02-10T04:14:39.454 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-10T04:14:39.455 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi085.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.982546", "end": "2021-02-10 04:14:39.428713", "rc": 0, "start": "2021-02-10 04:14:37.446167", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T04:14:39.498 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:39 +0000 (0:00:02.162) 0:03:43.133 **** 2021-02-10T04:14:39.529 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-02-10T04:14:39.530 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:14:39.565 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:39 +0000 (0:00:00.066) 0:03:43.199 **** 2021-02-10T04:14:39.584 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-10T04:14:39.584 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.029131", "end": "2021-02-10 04:14:39.548771", "rc": 0, "start": "2021-02-10 04:14:37.519640", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T04:14:39.625 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:39 +0000 (0:00:00.060) 0:03:43.260 **** 2021-02-10T04:14:39.653 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-02-10T04:14:39.654 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:14:39.687 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:39 +0000 (0:00:00.061) 0:03:43.321 **** 2021-02-10T04:14:40.074 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-02-10T04:14:40.075 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-02-10T04:14:40.117 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:40 +0000 (0:00:00.430) 0:03:43.751 **** 2021-02-10T04:14:40.146 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-02-10T04:14:40.181 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:40 +0000 (0:00:00.064) 0:03:43.815 **** 2021-02-10T04:14:40.189 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-02-10T04:14:40.189 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-02-10T04:14:40.231 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:40 +0000 (0:00:00.049) 0:03:43.865 **** 2021-02-10T04:14:40.238 INFO:teuthology.task.ansible.out: 2021-02-10T04:14:40.240 INFO:teuthology.task.ansible.out:TASK [testnode : Enable copr repos] ******************************************** 2021-02-10T04:14:40.288 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:40 +0000 (0:00:00.057) 0:03:43.923 **** 2021-02-10T04:14:40.301 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-02-10T04:14:40.347 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:40 +0000 (0:00:00.058) 0:03:43.982 **** 2021-02-10T04:14:40.354 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-02-10T04:14:40.356 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:14:40.391 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:40 +0000 (0:00:00.043) 0:03:44.025 **** 2021-02-10T04:14:40.399 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-02-10T04:14:40.449 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:40 +0000 (0:00:00.058) 0:03:44.084 **** 2021-02-10T04:14:40.456 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-02-10T04:14:40.458 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-02-10T04:14:40.497 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:40 +0000 (0:00:00.047) 0:03:44.131 **** 2021-02-10T04:14:40.505 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-02-10T04:14:40.506 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:14:40.542 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:40 +0000 (0:00:00.044) 0:03:44.176 **** 2021-02-10T04:14:40.550 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-02-10T04:14:40.551 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:14:40.592 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:40 +0000 (0:00:00.050) 0:03:44.226 **** 2021-02-10T04:14:40.602 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-02-10T04:14:40.603 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-02-10T04:14:40.604 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-02-10T04:14:40.642 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:40 +0000 (0:00:00.049) 0:03:44.276 **** 2021-02-10T04:14:40.649 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T04:14:40.650 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:14:40.690 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:40 +0000 (0:00:00.048) 0:03:44.324 **** 2021-02-10T04:14:40.698 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-02-10T04:14:40.698 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:14:40.735 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:40 +0000 (0:00:00.045) 0:03:44.369 **** 2021-02-10T04:14:40.742 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-02-10T04:14:40.742 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:14:40.780 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:40 +0000 (0:00:00.044) 0:03:44.414 **** 2021-02-10T04:14:40.787 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T04:14:40.788 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:14:40.832 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:40 +0000 (0:00:00.051) 0:03:44.466 **** 2021-02-10T04:14:40.840 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-02-10T04:14:40.842 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:14:40.878 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:40 +0000 (0:00:00.046) 0:03:44.513 **** 2021-02-10T04:14:40.886 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-02-10T04:14:40.887 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:14:40.924 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:40 +0000 (0:00:00.044) 0:03:44.557 **** 2021-02-10T04:14:40.962 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-02-10T04:14:40.963 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:14:40.996 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:40 +0000 (0:00:00.073) 0:03:44.631 **** 2021-02-10T04:14:41.090 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-02-10T04:14:41.092 INFO:teuthology.task.ansible.out:fatal: [smithi085.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007034", "end": "2021-02-10 04:14:41.065715", "msg": "non-zero return code", "rc": 1, "start": "2021-02-10 04:14:41.058681", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-02-10T04:14:41.139 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:41 +0000 (0:00:00.142) 0:03:44.773 **** 2021-02-10T04:14:41.181 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-02-10T04:14:41.182 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:14:41.234 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:41 +0000 (0:00:00.095) 0:03:44.869 **** 2021-02-10T04:14:41.244 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-02-10T04:14:41.245 INFO:teuthology.task.ansible.out:fatal: [smithi164.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007155", "end": "2021-02-10 04:14:41.182981", "msg": "non-zero return code", "rc": 1, "start": "2021-02-10 04:14:41.175826", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-02-10T04:14:41.282 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:41 +0000 (0:00:00.047) 0:03:44.916 **** 2021-02-10T04:14:41.320 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:14:41.362 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:41 +0000 (0:00:00.079) 0:03:44.996 **** 2021-02-10T04:14:47.035 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-02-10T04:14:47.397 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:4.2.0-34.module+el8.3.0+7976+077be4ec.x86_64"]} Wednesday 10 February 2021 04:14:47 +0000 (0:00:05.709) 0:03:50.706 **** changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:4.2.0-34.module+el8.3.0+7976+077be4ec.x86_64"]} 2021-02-10T04:14:47.402 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:47 +0000 (0:00:00.330) 0:03:51.036 **** 2021-02-10T04:14:52.606 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-02-10T04:14:52.608 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-02-10T04:14:52.643 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:52 +0000 (0:00:05.239) 0:03:56.276 **** 2021-02-10T04:14:52.884 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-02-10T04:14:52.919 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:14:52 +0000 (0:00:00.276) 0:03:56.553 **** 2021-02-10T04:15:29.098 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-10T04:15:29.100 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: dnf-utils", "Installed: git-all", "Installed: sysstat", "Installed: libedit", "Installed: boost-thread", "Installed: xfsprogs", "Installed: gdisk", "Installed: parted", "Installed: libgcrypt", "Installed: fuse-libs", "Installed: openssl", "Installed: libuuid", "Installed: attr", "Installed: ant", "Installed: lsof", "Installed: gettext", "Installed: bc", "Installed: xfsdump", "Installed: blktrace", "Installed: usbredir", "Installed: podman", "Installed: redhat-lsb", "Installed: firewalld", "Installed: wget", "Installed: podman-docker", "Installed: libev-devel", "Installed: valgrind", "Installed: nfs-utils", "Installed: ncurses-devel", "Installed: gcc", "Installed: git", "Installed: make", "Installed: python3-nose", "Installed: python3-virtualenv", "Installed: genisoimage", "Installed: qemu-img", "Installed: qemu-kvm-core", "Installed: qemu-kvm-block-rbd", "Installed: libacl-devel", "Installed: autoconf", "Installed: gdb", "Installed: iozone", "Installed: lvm2", "Installed: perl-Data-Dump-1.23-7.module+el8.3.0+6498+9eecfe51.noarch", "Installed: urw-base35-standard-symbols-ps-fonts-20170801-10.el8.noarch", "Installed: openssl-libs-1:1.1.1g-12.el8_3.x86_64", "Installed: google-droid-sans-fonts-20120715-13.el8.noarch", "Installed: perl-Try-Tiny-0.30-7.module+el8.3.0+6498+9eecfe51.noarch", "Installed: perl-WWW-RobotRules-6.02-18.module+el8.3.0+6498+9eecfe51.noarch", "Installed: perl-Text-Soundex-3.05-8.el8.x86_64", "Installed: jbig2dec-libs-0.14-4.el8_2.x86_64", "Installed: openssl-1:1.1.1g-12.el8_3.x86_64", "Installed: libSM-1.2.3-1.el8.x86_64", "Installed: perl-XML-SAX-1.00-1.el8.noarch", "Installed: qemu-img-15:4.2.0-34.module+el8.3.0+8829+e7a0a3ea.1.x86_64", "Installed: qemu-kvm-core-15:4.2.0-34.module+el8.3.0+8829+e7a0a3ea.1.x86_64", "Installed: qemu-kvm-common-15:4.2.0-34.module+el8.3.0+8829+e7a0a3ea.1.x86_64", "Installed: boost-program-options-1.66.0-10.el8.x8 2021-02-10T04:15:29.101 INFO:teuthology.task.ansible.out:6_64", "Installed: redhat-lsb-languages-4.1-47.el8.x86_64", "Installed: poppler-utils-0.66.0-27.el8.x86_64", "Installed: cups-1:2.2.6-38.el8.x86_64", "Installed: perl-HTTP-Negotiate-6.01-19.module+el8.3.0+6498+9eecfe51.noarch", "Installed: redhat-lsb-desktop-4.1-47.el8.x86_64", "Installed: urw-base35-c059-fonts-20170801-10.el8.noarch", "Installed: qemu-kvm-block-rbd-15:4.2.0-34.module+el8.3.0+8829+e7a0a3ea.1.x86_64", "Installed: urw-base35-bookman-fonts-20170801-10.el8.noarch", "Installed: perl-NTLM-1.09-17.module+el8.3.0+6498+9eecfe51.noarch", "Installed: liberation-mono-fonts-1:2.00.3-7.el8.noarch", "Installed: libpaper-1.1.24-22.el8.x86_64", "Installed: perl-Module-Pluggable-2:5.2-7.el8.noarch", "Installed: poppler-0.66.0-27.el8.x86_64", "Installed: cups-filters-libs-1.20.0-20.el8.x86_64", "Installed: perl-File-CheckTree-4.42-303.el8.noarch", "Installed: urw-base35-nimbus-sans-fonts-20170801-10.el8.noarch", "Installed: perl-Pod-LaTeX-0.61-302.el8.noarch", "Installed: redhat-lsb-cxx-4.1-47.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: libgs-9.25-7.el8.x86_64", "Installed: mesa-libGLU-9.0.0-15.el8.x86_64", "Installed: libXxf86misc-1.0.4-1.el8.x86_64", "Installed: libXt-1.1.5-12.el8.x86_64", "Installed: libXmu-1.1.3-1.el8.x86_64", "Installed: adobe-mappings-pdf-20180407-1.el8.noarch", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: perl-XML-SAX-Base-1.09-4.el8.noarch", "Installed: adobe-mappings-cmap-20171205-3.el8.noarch", "Installed: cups-filesystem-1:2.2.6-38.el8.noarch", "Installed: adobe-mappings-cmap-deprecated-20171205-3.el8.noarch", "Installed: urw-base35-fonts-common-20170801-10.el8.noarch", "Installed: ghostscript-9.25-7.el8.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: cups-filters-1.20.0-20.el8.x86_64", "Installed: liberation-fonts-common-1:2.00.3-7.el8.noarch", "Installed: perl-Net-HTTP-6.17-2.module+el8.3.0+6498+9eecfe51.noarch", "Installed: redhat-lsb-printing-4.1-47.el8.x86_64", "Installed: redh 2021-02-10T04:15:29.103 INFO:teuthology.task.ansible.out:at-lsb-submod-multimedia-4.1-47.el8.x86_64", "Installed: perl-File-Listing-6.04-17.module+el8.3.0+6498+9eecfe51.noarch", "Installed: urw-base35-p052-fonts-20170801-10.el8.noarch", "Installed: urw-base35-nimbus-roman-fonts-20170801-10.el8.noarch", "Installed: perl-B-Lint-1.20-11.el8.noarch", "Installed: perl-Text-Unidecode-1.30-5.el8.noarch", "Installed: perl-XML-NamespaceSupport-1.12-4.el8.noarch", "Installed: urw-base35-z003-fonts-20170801-10.el8.noarch", "Installed: libidn-1.34-5.el8.x86_64", "Installed: perl-Module-Runtime-0.016-2.el8.noarch", "Installed: avahi-glib-0.7-19.el8.x86_64", "Installed: perl-Pod-Plainer-1.04-7.el8.noarch", "Installed: perl-HTTP-Cookies-6.04-2.module+el8.3.0+6498+9eecfe51.noarch", "Installed: libICE-1.0.9-15.el8.x86_64", "Installed: xorg-x11-server-utils-7.7-27.el8.x86_64", "Installed: qpdf-libs-7.1.1-10.el8.x86_64", "Installed: openjpeg2-2.3.1-6.el8.x86_64", "Installed: urw-base35-gothic-fonts-20170801-10.el8.noarch", "Installed: libpng12-1.2.57-5.el8.x86_64", "Installed: cups-ipptool-1:2.2.6-38.el8.x86_64", "Installed: libmcpp-2.7.2-20.el8.x86_64", "Installed: urw-base35-nimbus-mono-ps-fonts-20170801-10.el8.noarch", "Installed: redhat-lsb-4.1-47.el8.x86_64", "Installed: libijs-0.35-5.el8.x86_64", "Installed: avahi-0.7-19.el8.x86_64", "Installed: urw-base35-fonts-20170801-10.el8.noarch", "Installed: perl-libwww-perl-6.34-1.module+el8.3.0+6498+9eecfe51.noarch", "Installed: lcms2-2.9-2.el8.x86_64", "Installed: urw-base35-d050000l-fonts-20170801-10.el8.noarch", "Installed: perl-Class-ISA-0.36-1022.el8.noarch", "Installed: poppler-data-0.4.9-1.el8.noarch", "Installed: mcpp-2.7.2-20.el8.x86_64", "Installed: perl-XML-LibXML-1:2.0132-2.el8.x86_64", "Removed: qemu-img-15:4.2.0-34.module+el8.3.0+7976+077be4ec.x86_64", "Removed: qemu-kvm-common-15:4.2.0-34.module+el8.3.0+7976+077be4ec.x86_64", "Removed: qemu-kvm-core-15:4.2.0-34.module+el8.3.0+7976+077be4ec.x86_64", "Removed: openssl-1:1.1.1g-11.el8.x86_64", "Removed: openssl-libs-1:1.1.1g-11.el 2021-02-10T04:15:29.104 INFO:teuthology.task.ansible.out:8.x86_64"]} 2021-02-10T04:15:29.134 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:29 +0000 (0:00:36.215) 0:04:32.768 **** 2021-02-10T04:15:29.362 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: dnf-utils", "Installed: git-all", "Installed: sysstat", "Installed: libedit", "Installed: boost-thread", "Installed: xfsprogs", "Installed: gdisk", "Installed: parted", "Installed: libgcrypt", "Installed: fuse-libs", "Installed: openssl", "Installed: libuuid", "Installed: attr", "Installed: ant", "Installed: lsof", "Installed: gettext", "Installed: bc", "Installed: xfsdump", "Installed: blktrace", "Installed: usbredir", "Installed: podman", "Installed: redhat-lsb", "Installed: firewalld", "Installed: wget", "Installed: podman-docker", "Installed: libev-devel", "Installed: valgrind", "Installed: nfs-utils", "Installed: ncurses-devel", "Installed: gcc", "Installed: git", "Installed: make", "Installed: python3-nose", "Installed: python3-virtualenv", "Installed: genisoimage", "Installed: qemu-img", "Installed: qemu-kvm-core", "Installed: qemu-kvm-block-rbd", "Installed: libacl-devel", "Installed: autoconf", "Installed: gdb", "Installed: iozone", "Installed: lvm2", "Installed: perl-Data-Dump-1.23-7.module+el8.3.0+6498+9eecfe51.noarch", "Installed: urw-base35-standard-symbols-ps-fonts-20170801-10.el8.noarch", "Installed: openssl-libs-1:1.1.1g-12.el8_3.x86_64", "Installed: google-droid-sans-fonts-20120715-13.el8.noarch", "Installed: perl-Try-Tiny-0.30-7.module+el8.3.0+6498+9eecfe51.noarch", "Installed: perl-WWW-RobotRules-6.02-18.module+el8.3.0+6498+9eecfe51.noarch", "Installed: perl-Text-Soundex-3.05-8.el8.x86_64", "Installed: jbig2dec-libs-0.14-4.el8_2.x86_64", "Installed: openssl-1:1.1.1g-12.el8_3.x86_64", "Installed: libSM-1.2.3-1.el8.x86_64", "Installed: perl-XML-SAX-1.00-1.el8.noarch", "Installed: qemu-img-15:4.2.0-34.module+el8.3.0+8829+e7a0a3ea.1.x86_64", "Installed: qemu-kvm-core-15:4.2.0-34.module+el8.3.0+8829+e7a0a3ea.1.x86_64", "Installed: qemu-kvm-common-15:4.2.0-34.module+el8.3.0+8829+e7a0a3ea.1.x86_64", "Installed: boost-program-options-1.66.0-10.el8.x8 2021-02-10T04:15:29.363 INFO:teuthology.task.ansible.out:6_64", "Installed: redhat-lsb-languages-4.1-47.el8.x86_64", "Installed: poppler-utils-0.66.0-27.el8.x86_64", "Installed: cups-1:2.2.6-38.el8.x86_64", "Installed: perl-HTTP-Negotiate-6.01-19.module+el8.3.0+6498+9eecfe51.noarch", "Installed: redhat-lsb-desktop-4.1-47.el8.x86_64", "Installed: urw-base35-c059-fonts-20170801-10.el8.noarch", "Installed: qemu-kvm-block-rbd-15:4.2.0-34.module+el8.3.0+8829+e7a0a3ea.1.x86_64", "Installed: urw-base35-bookman-fonts-20170801-10.el8.noarch", "Installed: perl-NTLM-1.09-17.module+el8.3.0+6498+9eecfe51.noarch", "Installed: liberation-mono-fonts-1:2.00.3-7.el8.noarch", "Installed: libpaper-1.1.24-22.el8.x86_64", "Installed: perl-Module-Pluggable-2:5.2-7.el8.noarch", "Installed: poppler-0.66.0-27.el8.x86_64", "Installed: cups-filters-libs-1.20.0-20.el8.x86_64", "Installed: perl-File-CheckTree-4.42-303.el8.noarch", "Installed: urw-base35-nimbus-sans-fonts-20170801-10.el8.noarch", "Installed: perl-Pod-LaTeX-0.61-302.el8.noarch", "Installed: redhat-lsb-cxx-4.1-47.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: libgs-9.25-7.el8.x86_64", "Installed: mesa-libGLU-9.0.0-15.el8.x86_64", "Installed: libXxf86misc-1.0.4-1.el8.x86_64", "Installed: libXt-1.1.5-12.el8.x86_64", "Installed: libXmu-1.1.3-1.el8.x86_64", "Installed: adobe-mappings-pdf-20180407-1.el8.noarch", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: perl-XML-SAX-Base-1.09-4.el8.noarch", "Installed: adobe-mappings-cmap-20171205-3.el8.noarch", "Installed: cups-filesystem-1:2.2.6-38.el8.noarch", "Installed: adobe-mappings-cmap-deprecated-20171205-3.el8.noarch", "Installed: urw-base35-fonts-common-20170801-10.el8.noarch", "Installed: ghostscript-9.25-7.el8.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: cups-filters-1.20.0-20.el8.x86_64", "Installed: liberation-fonts-common-1:2.00.3-7.el8.noarch", "Installed: perl-Net-HTTP-6.17-2.module+el8.3.0+6498+9eecfe51.noarch", "Installed: redhat-lsb-printing-4.1-47.el8.x86_64", "Installed: redh 2021-02-10T04:15:29.365 INFO:teuthology.task.ansible.out:at-lsb-submod-multimedia-4.1-47.el8.x86_64", "Installed: perl-File-Listing-6.04-17.module+el8.3.0+6498+9eecfe51.noarch", "Installed: urw-base35-p052-fonts-20170801-10.el8.noarch", "Installed: urw-base35-nimbus-roman-fonts-20170801-10.el8.noarch", "Installed: perl-B-Lint-1.20-11.el8.noarch", "Installed: perl-Text-Unidecode-1.30-5.el8.noarch", "Installed: perl-XML-NamespaceSupport-1.12-4.el8.noarch", "Installed: urw-base35-z003-fonts-20170801-10.el8.noarch", "Installed: libidn-1.34-5.el8.x86_64", "Installed: perl-Module-Runtime-0.016-2.el8.noarch", "Installed: avahi-glib-0.7-19.el8.x86_64", "Installed: perl-Pod-Plainer-1.04-7.el8.noarch", "Installed: perl-HTTP-Cookies-6.04-2.module+el8.3.0+6498+9eecfe51.noarch", "Installed: libICE-1.0.9-15.el8.x86_64", "Installed: xorg-x11-server-utils-7.7-27.el8.x86_64", "Installed: qpdf-libs-7.1.1-10.el8.x86_64", "Installed: openjpeg2-2.3.1-6.el8.x86_64", "Installed: urw-base35-gothic-fonts-20170801-10.el8.noarch", "Installed: libpng12-1.2.57-5.el8.x86_64", "Installed: cups-ipptool-1:2.2.6-38.el8.x86_64", "Installed: libmcpp-2.7.2-20.el8.x86_64", "Installed: urw-base35-nimbus-mono-ps-fonts-20170801-10.el8.noarch", "Installed: redhat-lsb-4.1-47.el8.x86_64", "Installed: libijs-0.35-5.el8.x86_64", "Installed: avahi-0.7-19.el8.x86_64", "Installed: urw-base35-fonts-20170801-10.el8.noarch", "Installed: perl-libwww-perl-6.34-1.module+el8.3.0+6498+9eecfe51.noarch", "Installed: lcms2-2.9-2.el8.x86_64", "Installed: urw-base35-d050000l-fonts-20170801-10.el8.noarch", "Installed: perl-Class-ISA-0.36-1022.el8.noarch", "Installed: poppler-data-0.4.9-1.el8.noarch", "Installed: mcpp-2.7.2-20.el8.x86_64", "Installed: perl-XML-LibXML-1:2.0132-2.el8.x86_64", "Removed: qemu-img-15:4.2.0-34.module+el8.3.0+7976+077be4ec.x86_64", "Removed: qemu-kvm-common-15:4.2.0-34.module+el8.3.0+7976+077be4ec.x86_64", "Removed: qemu-kvm-core-15:4.2.0-34.module+el8.3.0+7976+077be4ec.x86_64", "Removed: openssl-1:1.1.1g-11.el8.x86_64", "Removed: openssl-libs-1:1.1.1g-11.el 2021-02-10T04:15:29.366 INFO:teuthology.task.ansible.out:8.x86_64"]} 2021-02-10T04:15:29.400 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:29 +0000 (0:00:00.266) 0:04:33.035 **** 2021-02-10T04:15:31.666 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-02-10T04:15:31.667 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: dbench"]} 2021-02-10T04:15:31.701 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:31 +0000 (0:00:02.300) 0:04:35.336 **** 2021-02-10T04:15:31.884 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: dbench"]} 2021-02-10T04:15:31.920 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:31 +0000 (0:00:00.218) 0:04:35.554 **** 2021-02-10T04:15:34.244 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-02-10T04:15:34.246 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-02-10T04:15:34.284 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:34 +0000 (0:00:02.364) 0:04:37.919 **** 2021-02-10T04:15:34.322 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-10T04:15:34.324 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:34.357 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:34 +0000 (0:00:00.073) 0:04:37.992 **** 2021-02-10T04:15:34.417 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-02-10T04:15:34.419 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-02-10T04:15:34.456 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:34 +0000 (0:00:00.098) 0:04:38.090 **** 2021-02-10T04:15:34.505 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-10T04:15:34.506 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:34.543 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:34 +0000 (0:00:00.086) 0:04:38.177 **** 2021-02-10T04:15:34.655 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-02-10T04:15:34.657 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "basic.target dbus.socket system.slice dbus.service polkit.service sysinit.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "nftables.service ip6tables.service shutdown.target iptables.service ipset.service ebtables.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "D 2021-02-10T04:15:34.658 INFO:teuthology.task.ansible.out:evicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "Limi 2021-02-10T04:15:34.659 INFO:teuthology.task.ansible.out:tNOFILESoft": "1024", "LimitNPROC": "127237", "LimitNPROCSoft": "127237", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127237", "LimitSIGPENDINGSoft": "127237", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "dbus.socket system.slice sysinit.target", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "S 2021-02-10T04:15:34.661 INFO:teuthology.task.ansible.out:tartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203580", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-02-10T04:15:34.696 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:34 +0000 (0:00:00.153) 0:04:38.331 **** 2021-02-10T04:15:34.842 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "polkit.service dbus.service basic.target dbus.socket system.slice sysinit.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target ip6tables.service ipset.service nftables.service ebtables.service iptables.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "D 2021-02-10T04:15:34.843 INFO:teuthology.task.ansible.out:evicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "Limi 2021-02-10T04:15:34.844 INFO:teuthology.task.ansible.out:tNOFILESoft": "1024", "LimitNPROC": "127237", "LimitNPROCSoft": "127237", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127237", "LimitSIGPENDINGSoft": "127237", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice dbus.socket", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "S 2021-02-10T04:15:34.846 INFO:teuthology.task.ansible.out:tartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203580", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-02-10T04:15:34.878 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:34 +0000 (0:00:00.181) 0:04:38.512 **** 2021-02-10T04:15:34.979 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-02-10T04:15:34.981 INFO:teuthology.task.ansible.out:fatal: [smithi164.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-02-10T04:15:35.017 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:35 +0000 (0:00:00.139) 0:04:38.651 **** 2021-02-10T04:15:35.155 INFO:teuthology.task.ansible.out:fatal: [smithi085.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-02-10T04:15:35.192 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:35 +0000 (0:00:00.174) 0:04:38.826 **** 2021-02-10T04:15:35.824 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-02-10T04:15:35.826 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-02-10T04:15:35.864 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:35 +0000 (0:00:00.672) 0:04:39.498 **** 2021-02-10T04:15:35.896 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:35 +0000 (0:00:00.032) 0:04:39.531 **** 2021-02-10T04:15:35.902 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-02-10T04:15:35.904 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:35.938 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:35 +0000 (0:00:00.041) 0:04:39.572 **** 2021-02-10T04:15:35.944 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:35.980 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:35 +0000 (0:00:00.041) 0:04:39.614 **** 2021-02-10T04:15:35.986 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:36.025 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:36 +0000 (0:00:00.045) 0:04:39.659 **** 2021-02-10T04:15:36.044 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:36.080 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:36 +0000 (0:00:00.055) 0:04:39.714 **** 2021-02-10T04:15:36.437 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 04:15:36 +0000 (0:00:00.055) 0:04:39.770 **** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 04:15:36 +0000 (0:00:00.046) 0:04:39.816 **** TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 04:15:36 +0000 (0:00:00.049) 0:04:39.866 **** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 04:15:36 +0000 (0:00:00.042) 0:04:39.909 **** TASK [testnode : Restart abrtd] ************************************************ skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 04:15:36 +0000 (0:00:00.042) 0:04:39.951 **** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 04:15:36 +0000 (0:00:00.043) 0:04:39.995 **** TASK [testnode : Check for custom repos] *************************************** skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 04:15:36 +0000 (0:00:00.043) 0:04:40.038 **** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:36.448 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:36 +0000 (0:00:00.044) 0:04:40.082 **** 2021-02-10T04:15:36.455 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-02-10T04:15:36.490 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:36 +0000 (0:00:00.042) 0:04:40.125 **** 2021-02-10T04:15:36.537 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:36 +0000 (0:00:00.046) 0:04:40.171 **** 2021-02-10T04:15:36.543 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-02-10T04:15:36.545 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:36.578 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:36 +0000 (0:00:00.041) 0:04:40.213 **** 2021-02-10T04:15:36.588 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:36.623 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:36 +0000 (0:00:00.044) 0:04:40.257 **** 2021-02-10T04:15:36.630 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-02-10T04:15:36.630 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:36.666 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:36 +0000 (0:00:00.043) 0:04:40.300 **** 2021-02-10T04:15:36.673 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:36.711 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:36 +0000 (0:00:00.045) 0:04:40.346 **** 2021-02-10T04:15:36.718 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-02-10T04:15:36.719 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-02-10T04:15:36.756 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:36 +0000 (0:00:00.044) 0:04:40.390 **** 2021-02-10T04:15:36.763 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-02-10T04:15:36.763 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-02-10T04:15:36.799 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:36 +0000 (0:00:00.042) 0:04:40.433 **** 2021-02-10T04:15:36.805 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-02-10T04:15:36.805 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:36.841 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:36 +0000 (0:00:00.042) 0:04:40.475 **** 2021-02-10T04:15:36.848 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:36.883 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:36 +0000 (0:00:00.042) 0:04:40.517 **** 2021-02-10T04:15:36.890 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-02-10T04:15:36.925 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:36 +0000 (0:00:00.042) 0:04:40.560 **** 2021-02-10T04:15:36.968 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:36 +0000 (0:00:00.042) 0:04:40.602 **** 2021-02-10T04:15:36.975 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-02-10T04:15:36.976 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:37.013 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:37 +0000 (0:00:00.044) 0:04:40.647 **** 2021-02-10T04:15:37.020 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:37.055 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:37 +0000 (0:00:00.041) 0:04:40.689 **** 2021-02-10T04:15:37.062 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-02-10T04:15:37.062 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:37.102 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:37 +0000 (0:00:00.047) 0:04:40.737 **** 2021-02-10T04:15:37.109 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:37.154 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:37 +0000 (0:00:00.052) 0:04:40.789 **** 2021-02-10T04:15:37.163 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-02-10T04:15:37.164 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:37.198 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:37 +0000 (0:00:00.043) 0:04:40.832 **** 2021-02-10T04:15:37.205 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:37.240 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:37 +0000 (0:00:00.042) 0:04:40.874 **** 2021-02-10T04:15:37.247 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:37.283 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:37 +0000 (0:00:00.042) 0:04:40.917 **** 2021-02-10T04:15:37.292 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:37.327 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:37 +0000 (0:00:00.044) 0:04:40.961 **** 2021-02-10T04:15:37.334 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-02-10T04:15:37.335 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:37.369 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:37 +0000 (0:00:00.042) 0:04:41.004 **** 2021-02-10T04:15:37.376 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:37.411 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:37 +0000 (0:00:00.041) 0:04:41.045 **** 2021-02-10T04:15:37.418 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-10T04:15:37.419 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:37.453 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:37 +0000 (0:00:00.042) 0:04:41.087 **** 2021-02-10T04:15:37.460 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:37.495 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:37 +0000 (0:00:00.042) 0:04:41.130 **** 2021-02-10T04:15:37.502 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-02-10T04:15:37.502 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:37.537 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:37 +0000 (0:00:00.041) 0:04:41.171 **** 2021-02-10T04:15:37.545 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:37.580 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:37 +0000 (0:00:00.042) 0:04:41.214 **** 2021-02-10T04:15:37.587 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-02-10T04:15:37.588 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:37.622 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:37 +0000 (0:00:00.042) 0:04:41.257 **** 2021-02-10T04:15:37.629 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:37.664 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:37 +0000 (0:00:00.042) 0:04:41.299 **** 2021-02-10T04:15:37.671 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-02-10T04:15:37.672 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:37.707 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:37 +0000 (0:00:00.042) 0:04:41.341 **** 2021-02-10T04:15:37.714 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:37.747 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:37 +0000 (0:00:00.040) 0:04:41.382 **** 2021-02-10T04:15:37.754 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-02-10T04:15:37.756 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:37.789 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:37 +0000 (0:00:00.041) 0:04:41.423 **** 2021-02-10T04:15:37.796 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:37.830 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:37 +0000 (0:00:00.040) 0:04:41.464 **** 2021-02-10T04:15:37.836 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-02-10T04:15:37.836 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:37.872 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:37 +0000 (0:00:00.042) 0:04:41.506 **** 2021-02-10T04:15:37.880 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:37.918 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:37 +0000 (0:00:00.046) 0:04:41.553 **** 2021-02-10T04:15:37.924 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-02-10T04:15:37.926 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:37.960 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:37 +0000 (0:00:00.042) 0:04:41.595 **** 2021-02-10T04:15:37.967 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:38.002 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:37 +0000 (0:00:00.041) 0:04:41.636 **** 2021-02-10T04:15:38.008 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-02-10T04:15:38.010 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:38.045 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:38 +0000 (0:00:00.042) 0:04:41.679 **** 2021-02-10T04:15:38.051 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:38.093 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:38 +0000 (0:00:00.047) 0:04:41.727 **** 2021-02-10T04:15:38.100 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-02-10T04:15:38.100 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:38.138 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:38 +0000 (0:00:00.045) 0:04:41.772 **** 2021-02-10T04:15:38.150 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:38.197 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:38 +0000 (0:00:00.058) 0:04:41.831 **** 2021-02-10T04:15:38.204 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-10T04:15:38.205 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:38.256 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:38 +0000 (0:00:00.059) 0:04:41.890 **** 2021-02-10T04:15:38.265 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:38.305 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:38 +0000 (0:00:00.049) 0:04:41.940 **** 2021-02-10T04:15:38.317 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-02-10T04:15:38.317 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:38.369 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:38 +0000 (0:00:00.064) 0:04:42.004 **** 2021-02-10T04:15:38.381 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:38.425 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:38 +0000 (0:00:00.055) 0:04:42.059 **** 2021-02-10T04:15:38.432 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-02-10T04:15:38.433 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:38.490 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:38 +0000 (0:00:00.065) 0:04:42.125 **** 2021-02-10T04:15:38.500 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:38.537 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:38 +0000 (0:00:00.046) 0:04:42.171 **** 2021-02-10T04:15:38.544 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-10T04:15:38.546 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:38.593 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:38 +0000 (0:00:00.056) 0:04:42.227 **** 2021-02-10T04:15:38.603 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:38.653 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:38 +0000 (0:00:00.059) 0:04:42.287 **** 2021-02-10T04:15:38.662 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-02-10T04:15:38.663 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:38.700 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:38 +0000 (0:00:00.047) 0:04:42.334 **** 2021-02-10T04:15:38.707 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:38.752 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:38 +0000 (0:00:00.052) 0:04:42.387 **** 2021-02-10T04:15:38.761 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-10T04:15:38.762 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:38.810 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:38 +0000 (0:00:00.057) 0:04:42.444 **** 2021-02-10T04:15:38.818 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:38.865 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:38 +0000 (0:00:00.055) 0:04:42.500 **** 2021-02-10T04:15:38.877 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-02-10T04:15:38.878 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:38.914 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:38 +0000 (0:00:00.048) 0:04:42.548 **** 2021-02-10T04:15:38.922 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:38.962 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:38 +0000 (0:00:00.048) 0:04:42.597 **** 2021-02-10T04:15:38.970 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-02-10T04:15:38.972 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:39.007 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:39 +0000 (0:00:00.044) 0:04:42.641 **** 2021-02-10T04:15:39.014 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:39.049 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:39 +0000 (0:00:00.042) 0:04:42.684 **** 2021-02-10T04:15:39.056 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T04:15:39.058 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:39.092 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:39 +0000 (0:00:00.042) 0:04:42.726 **** 2021-02-10T04:15:39.100 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:39.138 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:39 +0000 (0:00:00.045) 0:04:42.772 **** 2021-02-10T04:15:39.150 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-02-10T04:15:39.151 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:39.186 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:39 +0000 (0:00:00.047) 0:04:42.820 **** 2021-02-10T04:15:39.193 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:39.228 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:39 +0000 (0:00:00.042) 0:04:42.863 **** 2021-02-10T04:15:39.235 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-10T04:15:39.236 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:39.273 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:39 +0000 (0:00:00.044) 0:04:42.907 **** 2021-02-10T04:15:39.280 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:39.314 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:39 +0000 (0:00:00.041) 0:04:42.948 **** 2021-02-10T04:15:39.321 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-10T04:15:39.321 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:39.357 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:39 +0000 (0:00:00.042) 0:04:42.991 **** 2021-02-10T04:15:39.364 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:39.402 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:39 +0000 (0:00:00.045) 0:04:43.037 **** 2021-02-10T04:15:39.409 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-02-10T04:15:39.410 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:39.445 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:39 +0000 (0:00:00.042) 0:04:43.079 **** 2021-02-10T04:15:39.451 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:39.487 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:39 +0000 (0:00:00.042) 0:04:43.121 **** 2021-02-10T04:15:39.500 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-10T04:15:39.501 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:39.536 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:39 +0000 (0:00:00.048) 0:04:43.170 **** 2021-02-10T04:15:39.542 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:39.577 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:39 +0000 (0:00:00.041) 0:04:43.211 **** 2021-02-10T04:15:39.584 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-10T04:15:39.586 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:39.619 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:39 +0000 (0:00:00.042) 0:04:43.254 **** 2021-02-10T04:15:39.626 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:39.661 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:39 +0000 (0:00:00.041) 0:04:43.296 **** 2021-02-10T04:15:39.667 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-10T04:15:39.669 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:39.705 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:39 +0000 (0:00:00.043) 0:04:43.340 **** 2021-02-10T04:15:39.713 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:39.757 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:39 +0000 (0:00:00.051) 0:04:43.391 **** 2021-02-10T04:15:39.767 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:39.802 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:39 +0000 (0:00:00.045) 0:04:43.436 **** 2021-02-10T04:15:39.809 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:39.848 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:39 +0000 (0:00:00.045) 0:04:43.482 **** 2021-02-10T04:15:39.854 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-02-10T04:15:39.855 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:39.893 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:39 +0000 (0:00:00.044) 0:04:43.528 **** 2021-02-10T04:15:39.900 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:39.937 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:39 +0000 (0:00:00.043) 0:04:43.572 **** 2021-02-10T04:15:39.944 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-02-10T04:15:39.946 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:39.979 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:39 +0000 (0:00:00.042) 0:04:43.614 **** 2021-02-10T04:15:39.986 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:40.022 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:40 +0000 (0:00:00.042) 0:04:43.656 **** 2021-02-10T04:15:40.028 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-10T04:15:40.030 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:40.064 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:40 +0000 (0:00:00.042) 0:04:43.699 **** 2021-02-10T04:15:40.071 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:40.107 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:40 +0000 (0:00:00.042) 0:04:43.741 **** 2021-02-10T04:15:40.113 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-10T04:15:40.115 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:40.160 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:40 +0000 (0:00:00.053) 0:04:43.794 **** 2021-02-10T04:15:40.171 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:40.207 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:40 +0000 (0:00:00.046) 0:04:43.841 **** 2021-02-10T04:15:40.214 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-10T04:15:40.215 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:40.250 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:40 +0000 (0:00:00.042) 0:04:43.884 **** 2021-02-10T04:15:40.256 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:40.292 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:40 +0000 (0:00:00.042) 0:04:43.926 **** 2021-02-10T04:15:40.298 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-10T04:15:40.300 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:40.337 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:40 +0000 (0:00:00.045) 0:04:43.971 **** 2021-02-10T04:15:40.521 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 04:15:40 +0000 (0:00:00.050) 0:04:44.022 **** TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 04:15:40 +0000 (0:00:00.044) 0:04:44.066 **** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 04:15:40 +0000 (0:00:00.059) 0:04:44.125 **** TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:40.537 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:40 +0000 (0:00:00.045) 0:04:44.171 **** 2021-02-10T04:15:40.547 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:40.583 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:40 +0000 (0:00:00.046) 0:04:44.217 **** 2021-02-10T04:15:40.589 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-02-10T04:15:40.591 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:40.626 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:40 +0000 (0:00:00.042) 0:04:44.260 **** 2021-02-10T04:15:40.635 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:40.669 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:40 +0000 (0:00:00.043) 0:04:44.304 **** 2021-02-10T04:15:40.677 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-02-10T04:15:40.678 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:40.712 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:40 +0000 (0:00:00.042) 0:04:44.346 **** 2021-02-10T04:15:40.719 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:40.759 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:40 +0000 (0:00:00.046) 0:04:44.393 **** 2021-02-10T04:15:40.766 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-10T04:15:40.767 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:40.805 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:40 +0000 (0:00:00.046) 0:04:44.439 **** 2021-02-10T04:15:40.812 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:40.848 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:40 +0000 (0:00:00.043) 0:04:44.482 **** 2021-02-10T04:15:40.855 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-02-10T04:15:40.856 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:40.892 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:40 +0000 (0:00:00.042) 0:04:44.525 **** 2021-02-10T04:15:40.898 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:40.936 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:40 +0000 (0:00:00.043) 0:04:44.569 **** 2021-02-10T04:15:40.942 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-02-10T04:15:40.942 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:40.990 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:40 +0000 (0:00:00.055) 0:04:44.624 **** 2021-02-10T04:15:41.000 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:41.035 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:41 +0000 (0:00:00.044) 0:04:44.669 **** 2021-02-10T04:15:41.036 INFO:teuthology.task.ansible.out: 2021-02-10T04:15:41.043 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-02-10T04:15:41.045 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:41.085 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:41 +0000 (0:00:00.050) 0:04:44.719 **** 2021-02-10T04:15:41.093 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:41.128 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:41 +0000 (0:00:00.042) 0:04:44.762 **** 2021-02-10T04:15:41.134 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-02-10T04:15:41.136 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:41.175 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:41 +0000 (0:00:00.047) 0:04:44.810 **** 2021-02-10T04:15:41.183 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:41.221 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:41 +0000 (0:00:00.043) 0:04:44.853 **** 2021-02-10T04:15:41.226 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-02-10T04:15:41.228 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:41.263 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:41 +0000 (0:00:00.043) 0:04:44.897 **** 2021-02-10T04:15:41.270 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:41.306 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:41 +0000 (0:00:00.043) 0:04:44.940 **** 2021-02-10T04:15:41.313 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-02-10T04:15:41.315 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:41.350 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:41 +0000 (0:00:00.043) 0:04:44.984 **** 2021-02-10T04:15:41.357 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:41.394 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:41 +0000 (0:00:00.044) 0:04:45.028 **** 2021-02-10T04:15:41.401 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-02-10T04:15:41.403 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:41.438 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:41 +0000 (0:00:00.044) 0:04:45.072 **** 2021-02-10T04:15:41.446 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:41.489 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:41 +0000 (0:00:00.051) 0:04:45.124 **** 2021-02-10T04:15:41.498 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-02-10T04:15:41.499 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:41.539 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:41 +0000 (0:00:00.050) 0:04:45.174 **** 2021-02-10T04:15:41.547 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:41.587 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:41 +0000 (0:00:00.047) 0:04:45.221 **** 2021-02-10T04:15:41.594 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-02-10T04:15:41.596 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:41.634 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:41 +0000 (0:00:00.047) 0:04:45.268 **** 2021-02-10T04:15:41.642 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:41.681 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:41 +0000 (0:00:00.047) 0:04:45.315 **** 2021-02-10T04:15:41.689 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-02-10T04:15:41.689 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:41.724 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:41 +0000 (0:00:00.043) 0:04:45.359 **** 2021-02-10T04:15:41.732 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:41.774 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:41 +0000 (0:00:00.049) 0:04:45.408 **** 2021-02-10T04:15:41.781 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T04:15:41.782 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-02-10T04:15:41.817 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:41 +0000 (0:00:00.042) 0:04:45.451 **** 2021-02-10T04:15:41.823 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-02-10T04:15:41.860 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:41 +0000 (0:00:00.043) 0:04:45.494 **** 2021-02-10T04:15:41.868 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-02-10T04:15:41.869 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-02-10T04:15:41.880 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-02-10T04:15:41.883 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-02-10T04:15:41.895 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-02-10T04:15:41.896 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-02-10T04:15:41.907 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-02-10T04:15:41.908 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-02-10T04:15:41.918 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-02-10T04:15:41.920 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-02-10T04:15:41.963 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:41 +0000 (0:00:00.103) 0:04:45.597 **** 2021-02-10T04:15:41.970 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-02-10T04:15:41.971 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-02-10T04:15:41.974 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-02-10T04:15:42.016 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:42 +0000 (0:00:00.052) 0:04:45.650 **** 2021-02-10T04:15:42.299 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-02-10T04:15:42.300 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.007670", "end": "2021-02-10 04:15:42.266367", "rc": 0, "start": "2021-02-10 04:15:42.258697", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock"]} changed: [smithi085.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.006903", "end": "2021-02-10 04:15:42.266404", "rc": 0, "start": "2021-02-10 04:15:42.259501", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock"]} 2021-02-10T04:15:42.337 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:42 +0000 (0:00:00.320) 0:04:45.971 **** 2021-02-10T04:15:42.373 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:42 +0000 (0:00:00.036) 0:04:46.008 **** 2021-02-10T04:15:42.379 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-02-10T04:15:42.381 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:42.420 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:42 +0000 (0:00:00.046) 0:04:46.054 **** 2021-02-10T04:15:42.426 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:42.466 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:42 +0000 (0:00:00.046) 0:04:46.100 **** 2021-02-10T04:15:42.472 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-02-10T04:15:42.473 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:42.522 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:42 +0000 (0:00:00.056) 0:04:46.156 **** 2021-02-10T04:15:42.532 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:42.571 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:42 +0000 (0:00:00.049) 0:04:46.206 **** 2021-02-10T04:15:42.577 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-02-10T04:15:42.579 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:42.622 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:42 +0000 (0:00:00.050) 0:04:46.256 **** 2021-02-10T04:15:42.631 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:42.673 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:42 +0000 (0:00:00.051) 0:04:46.308 **** 2021-02-10T04:15:42.680 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-02-10T04:15:42.681 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:42.718 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:42 +0000 (0:00:00.044) 0:04:46.352 **** 2021-02-10T04:15:42.725 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:42.761 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:42 +0000 (0:00:00.043) 0:04:46.395 **** 2021-02-10T04:15:45.236 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-02-10T04:15:45.237 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: gdisk", "Installed: device-mapper"]} 2021-02-10T04:15:45.271 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:45 +0000 (0:00:02.510) 0:04:48.906 **** 2021-02-10T04:15:45.310 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: gdisk", "Installed: device-mapper"]} 2021-02-10T04:15:45.344 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:45 +0000 (0:00:00.072) 0:04:48.978 **** 2021-02-10T04:15:45.350 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-02-10T04:15:45.351 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930419712000, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227153250, 'block_used': 12941858, 'inode_total': 61054976, 'inode_available': 60986956, 'inode_used': 68020, 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227153250, "block_size": 4096, "block_total": 240095108, "block_used": 12941858, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986956, "inode_total": 61054976, "inode_used": 68020, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930419712000, "size_total": 983429562368, "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}} 2021-02-10T04:15:45.389 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:45 +0000 (0:00:00.044) 0:04:49.023 **** 2021-02-10T04:15:45.396 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930419712000, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227153250, 'block_used': 12941858, 'inode_total': 61054976, 'inode_available': 60986956, 'inode_used': 68020, 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227153250, "block_size": 4096, "block_total": 240095108, "block_used": 12941858, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986956, "inode_total": 61054976, "inode_used": 68020, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930419712000, "size_total": 983429562368, "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}} 2021-02-10T04:15:45.435 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:45 +0000 (0:00:00.045) 0:04:49.069 **** 2021-02-10T04:15:45.572 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-02-10T04:15:45.572 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005130", "end": "2021-02-10 04:15:45.554521", "rc": 0, "start": "2021-02-10 04:15:45.549391", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T04:15:45.610 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:45 +0000 (0:00:00.175) 0:04:49.245 **** 2021-02-10T04:15:45.639 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005081", "end": "2021-02-10 04:15:45.614062", "rc": 0, "start": "2021-02-10 04:15:45.608981", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T04:15:45.676 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:45 +0000 (0:00:00.065) 0:04:49.311 **** 2021-02-10T04:15:45.683 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-02-10T04:15:45.683 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930419712000, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227153250, 'block_used': 12941858, 'inode_total': 61054976, 'inode_available': 60986956, 'inode_used': 68020, 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227153250, "block_size": 4096, "block_total": 240095108, "block_used": 12941858, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986956, "inode_total": 61054976, "inode_used": 68020, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930419712000, "size_total": 983429562368, "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:45.721 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:45 +0000 (0:00:00.045) 0:04:49.356 **** 2021-02-10T04:15:45.729 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930419712000, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227153250, 'block_used': 12941858, 'inode_total': 61054976, 'inode_available': 60986956, 'inode_used': 68020, 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227153250, "block_size": 4096, "block_total": 240095108, "block_used": 12941858, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986956, "inode_total": 61054976, "inode_used": 68020, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930419712000, "size_total": 983429562368, "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:45.765 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:45 +0000 (0:00:00.043) 0:04:49.400 **** 2021-02-10T04:15:45.772 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-02-10T04:15:45.808 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:45 +0000 (0:00:00.042) 0:04:49.442 **** 2021-02-10T04:15:45.809 INFO:teuthology.task.ansible.out: 2021-02-10T04:15:45.851 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:45 +0000 (0:00:00.043) 0:04:49.486 **** 2021-02-10T04:15:47.712 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-02-10T04:15:47.713 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-8fgGDcnYh9E7rmMr5l3WgfT5CtBuqtKBHtRR1F3SjH1oiuiJEWgfCpAJ0oksoPUA'], 'uuids': ['35229488-e1bd-43b2-991e-8fe3638efa7e'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.673262", "end": "2021-02-10 04:15:47.676789", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-8fgGDcnYh9E7rmMr5l3WgfT5CtBuqtKBHtRR1F3SjH1oiuiJEWgfCpAJ0oksoPUA"], "labels": [], "masters": [], "uuids": ["35229488-e1bd-43b2-991e-8fe3638efa7e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 04:15:46.003527", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T04:15:47.975 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-IdediqjzmHzciXPYoZ0LGfbTuVKiPxkTLJAp8dnfjpzPejxrAZ2dRgde0mLOrhDx'], 'uuids': ['7fe23742-b73b-436f-8e92-cae3b21a161b'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.904374", "end": "2021-02-10 04:15:47.942575", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-IdediqjzmHzciXPYoZ0LGfbTuVKiPxkTLJAp8dnfjpzPejxrAZ2dRgde0mLOrhDx"], "labels": [], "masters": [], "uuids": ["7fe23742-b73b-436f-8e92-cae3b21a161b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 04:15:46.038201", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T04:15:48.967 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-gVoRot-dbi4-GjcL-kaQJ-a2tm-Icxa-L5yord', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400X7400BGN', 'nvme-nvme.8086-50484654363230343030583734303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.069062", "end": "2021-02-10 04:15:48.941588", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-gVoRot-dbi4-GjcL-kaQJ-a2tm-Icxa-L5yord", "nvme-INTEL_SSDPEDMD400G4_PHFT620400X7400BGN", "nvme-nvme.8086-50484654363230343030583734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 04:15:47.872526", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-02-10T04:15:48.969 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T04:15:49.209 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-kA9sNi-i9fs-mS93-ZK3B-W4Ej-7b9l-5OkPgI', 'nvme-INTEL_SSDPEDMD400G4_CVFT623300H0400BGN', 'nvme-nvme.8086-43564654363233333030483034303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.050374", "end": "2021-02-10 04:15:49.176493", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-kA9sNi-i9fs-mS93-ZK3B-W4Ej-7b9l-5OkPgI", "nvme-INTEL_SSDPEDMD400G4_CVFT623300H0400BGN", "nvme-nvme.8086-43564654363233333030483034303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 04:15:48.126119", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-02-10T04:15:49.210 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T04:15:50.218 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-8fgGDcnYh9E7rmMr5l3WgfT5CtBuqtKBEgiGhRth30D2ovoAtDdnliqyTd1HMKke'], 'uuids': ['f89aaecc-b501-4937-b558-4dc859bbc59d'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.058872", "end": "2021-02-10 04:15:50.186866", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-8fgGDcnYh9E7rmMr5l3WgfT5CtBuqtKBEgiGhRth30D2ovoAtDdnliqyTd1HMKke"], "labels": [], "masters": [], "uuids": ["f89aaecc-b501-4937-b558-4dc859bbc59d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 04:15:49.127994", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T04:15:50.448 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-IdediqjzmHzciXPYoZ0LGfbTuVKiPxkTfFsdjY5eM5mAoncF3GM6jKQzbrUCfKEs'], 'uuids': ['ba78ad7d-cf58-43cd-9d72-7438b3d5324e'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.063873", "end": "2021-02-10 04:15:50.423006", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-IdediqjzmHzciXPYoZ0LGfbTuVKiPxkTfFsdjY5eM5mAoncF3GM6jKQzbrUCfKEs"], "labels": [], "masters": [], "uuids": ["ba78ad7d-cf58-43cd-9d72-7438b3d5324e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 04:15:49.359133", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T04:15:51.442 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-8fgGDcnYh9E7rmMr5l3WgfT5CtBuqtKBgFL0fm8GFXoGCMkmum0easERhH6u8qS7'], 'uuids': ['559b829f-a0a4-4dc4-af4e-6cfc57d58d1a'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.042673", "end": "2021-02-10 04:15:51.411490", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-8fgGDcnYh9E7rmMr5l3WgfT5CtBuqtKBgFL0fm8GFXoGCMkmum0easERhH6u8qS7"], "labels": [], "masters": [], "uuids": ["559b829f-a0a4-4dc4-af4e-6cfc57d58d1a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 04:15:50.368817", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T04:15:51.696 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-IdediqjzmHzciXPYoZ0LGfbTuVKiPxkTffXHBZK2A6vSWJib1Xm6TBspX3qlqiou'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.062292", "end": "2021-02-10 04:15:51.666884", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-IdediqjzmHzciXPYoZ0LGfbTuVKiPxkTffXHBZK2A6vSWJib1Xm6TBspX3qlqiou"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 04:15:50.604592", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T04:15:52.688 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-8fgGDcnYh9E7rmMr5l3WgfT5CtBuqtKBhCQooGlSYA6vYjHrmeU1cIvMJvgQxCTx'], 'uuids': ['ff51082a-3d1c-41ff-b41f-c0d53b0423d1'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.062139", "end": "2021-02-10 04:15:52.662671", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-8fgGDcnYh9E7rmMr5l3WgfT5CtBuqtKBhCQooGlSYA6vYjHrmeU1cIvMJvgQxCTx"], "labels": [], "masters": [], "uuids": ["ff51082a-3d1c-41ff-b41f-c0d53b0423d1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 04:15:51.600532", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T04:15:52.701 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QA6H', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QA6H', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QA6H', 'scsi-35000c50091e09aef', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QA6H', 'wwn-0x5000c50091e09aef'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5QA6H', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50091e09aef', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QA6H-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QA6H-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QA6H-part1', 'scsi-35000c50091e09aef-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QA6H-part1', 'wwn-0x5000c50091e09aef-part1'], 'uuids': ['782d0384-b480-44b3-9c95-fe95daf886ae'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QA6H", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QA6H", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QA6H", "scsi-35000c50091e09aef", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QA6H", "wwn-0x5000c50091e09aef"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QA6H-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QA6H-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-02-10T04:15:52.702 INFO:teuthology.task.ansible.out:Z1W5QA6H-part1", "scsi-35000c50091e09aef-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QA6H-part1", "wwn-0x5000c50091e09aef-part1"], "labels": [], "masters": [], "uuids": ["782d0384-b480-44b3-9c95-fe95daf886ae"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QA6H", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e09aef"}}, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:52.933 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-IdediqjzmHzciXPYoZ0LGfbTuVKiPxkTMZfVZ14ntixOpauKBnE6fxSBDidDd92Y'], 'uuids': ['8afef42f-59b2-4bfc-93ea-b5508fe2d917'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.048742", "end": "2021-02-10 04:15:52.907269", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-IdediqjzmHzciXPYoZ0LGfbTuVKiPxkTMZfVZ14ntixOpauKBnE6fxSBDidDd92Y"], "labels": [], "masters": [], "uuids": ["8afef42f-59b2-4bfc-93ea-b5508fe2d917"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 04:15:51.858527", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T04:15:52.944 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XTQ7', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XTQ7', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XTQ7', 'scsi-35000c5009293f07d', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XTQ7', 'wwn-0x5000c5009293f07d'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XTQ7', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c5009293f07d', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XTQ7-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XTQ7-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XTQ7-part1', 'scsi-35000c5009293f07d-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XTQ7-part1', 'wwn-0x5000c5009293f07d-part1'], 'uuids': ['782d0384-b480-44b3-9c95-fe95daf886ae'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XTQ7", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XTQ7", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XTQ7", "scsi-35000c5009293f07d", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XTQ7", "wwn-0x5000c5009293f07d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XTQ7-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XTQ7-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-02-10T04:15:52.946 INFO:teuthology.task.ansible.out:Z1W5XTQ7-part1", "scsi-35000c5009293f07d-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XTQ7-part1", "wwn-0x5000c5009293f07d-part1"], "labels": [], "masters": [], "uuids": ["782d0384-b480-44b3-9c95-fe95daf886ae"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XTQ7", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009293f07d"}}, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:53.925 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-8fgGDcnYh9E7rmMr5l3WgfT5CtBuqtKBGtvyQ7MgRK2983ki1qAhulNYneXwwx3M'], 'uuids': ['b3535c2b-2b35-4210-a6b5-d296527df85f'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.046683", "end": "2021-02-10 04:15:53.895851", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-8fgGDcnYh9E7rmMr5l3WgfT5CtBuqtKBGtvyQ7MgRK2983ki1qAhulNYneXwwx3M"], "labels": [], "masters": [], "uuids": ["b3535c2b-2b35-4210-a6b5-d296527df85f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 04:15:52.849168", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T04:15:53.967 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:53 +0000 (0:00:08.115) 0:04:57.601 **** 2021-02-10T04:15:54.192 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-IdediqjzmHzciXPYoZ0LGfbTuVKiPxkTOfN9WLRUHub6TqatFWA7tSJ3oFnBf7Ze'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.059138", "end": "2021-02-10 04:15:54.158568", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-IdediqjzmHzciXPYoZ0LGfbTuVKiPxkTOfN9WLRUHub6TqatFWA7tSJ3oFnBf7Ze"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 04:15:53.099430", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T04:15:54.235 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:54 +0000 (0:00:00.268) 0:04:57.869 **** 2021-02-10T04:15:54.325 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-10T04:15:54.325 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-8fgGDcnYh9E7rmMr5l3WgfT5CtBuqtKBHtRR1F3SjH1oiuiJEWgfCpAJ0oksoPUA'], 'uuids': ['35229488-e1bd-43b2-991e-8fe3638efa7e'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.141267", "end": "2021-02-10 04:15:54.298692", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-8fgGDcnYh9E7rmMr5l3WgfT5CtBuqtKBHtRR1F3SjH1oiuiJEWgfCpAJ0oksoPUA"], "labels": [], "masters": [], "uuids": ["35229488-e1bd-43b2-991e-8fe3638efa7e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 04:15:54.157425", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.13858 s, 832 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.13858 s, 832 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T04:15:54.534 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-gVoRot-dbi4-GjcL-kaQJ-a2tm-Icxa-L5yord', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400X7400BGN', 'nvme-nvme.8086-50484654363230343030583734303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.037548", "end": "2021-02-10 04:15:54.508650", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-gVoRot-dbi4-GjcL-kaQJ-a2tm-Icxa-L5yord", "nvme-INTEL_SSDPEDMD400G4_PHFT620400X7400BGN", "nvme-nvme.8086-50484654363230343030583734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 04:15:54.471102", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0349604 s, 3.3 GB/s", "std 2021-02-10T04:15:54.536 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0349604 s, 3.3 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T04:15:54.598 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-IdediqjzmHzciXPYoZ0LGfbTuVKiPxkTLJAp8dnfjpzPejxrAZ2dRgde0mLOrhDx'], 'uuids': ['7fe23742-b73b-436f-8e92-cae3b21a161b'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.139985", "end": "2021-02-10 04:15:54.567452", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-IdediqjzmHzciXPYoZ0LGfbTuVKiPxkTLJAp8dnfjpzPejxrAZ2dRgde0mLOrhDx"], "labels": [], "masters": [], "uuids": ["7fe23742-b73b-436f-8e92-cae3b21a161b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 04:15:54.427467", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.137244 s, 840 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.137244 s, 840 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T04:15:54.797 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-kA9sNi-i9fs-mS93-ZK3B-W4Ej-7b9l-5OkPgI', 'nvme-INTEL_SSDPEDMD400G4_CVFT623300H0400BGN', 'nvme-nvme.8086-43564654363233333030483034303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.037527", "end": "2021-02-10 04:15:54.776081", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-kA9sNi-i9fs-mS93-ZK3B-W4Ej-7b9l-5OkPgI", "nvme-INTEL_SSDPEDMD400G4_CVFT623300H0400BGN", "nvme-nvme.8086-43564654363233333030483034303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 04:15:54.738554", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0347805 s, 3.3 GB/s", "std 2021-02-10T04:15:54.799 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0347805 s, 3.3 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T04:15:54.852 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-8fgGDcnYh9E7rmMr5l3WgfT5CtBuqtKBEgiGhRth30D2ovoAtDdnliqyTd1HMKke'], 'uuids': ['f89aaecc-b501-4937-b558-4dc859bbc59d'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.146315", "end": "2021-02-10 04:15:54.829947", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-8fgGDcnYh9E7rmMr5l3WgfT5CtBuqtKBEgiGhRth30D2ovoAtDdnliqyTd1HMKke"], "labels": [], "masters": [], "uuids": ["f89aaecc-b501-4937-b558-4dc859bbc59d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 04:15:54.683632", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.143719 s, 803 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.143719 s, 803 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T04:15:55.113 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-IdediqjzmHzciXPYoZ0LGfbTuVKiPxkTfFsdjY5eM5mAoncF3GM6jKQzbrUCfKEs'], 'uuids': ['ba78ad7d-cf58-43cd-9d72-7438b3d5324e'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.139245", "end": "2021-02-10 04:15:55.090690", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-IdediqjzmHzciXPYoZ0LGfbTuVKiPxkTfFsdjY5eM5mAoncF3GM6jKQzbrUCfKEs"], "labels": [], "masters": [], "uuids": ["ba78ad7d-cf58-43cd-9d72-7438b3d5324e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 04:15:54.951445", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.13676 s, 843 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.13676 s, 843 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T04:15:55.166 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-8fgGDcnYh9E7rmMr5l3WgfT5CtBuqtKBgFL0fm8GFXoGCMkmum0easERhH6u8qS7'], 'uuids': ['559b829f-a0a4-4dc4-af4e-6cfc57d58d1a'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.141292", "end": "2021-02-10 04:15:55.138756", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-8fgGDcnYh9E7rmMr5l3WgfT5CtBuqtKBgFL0fm8GFXoGCMkmum0easERhH6u8qS7"], "labels": [], "masters": [], "uuids": ["559b829f-a0a4-4dc4-af4e-6cfc57d58d1a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 04:15:54.997464", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138628 s, 832 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138628 s, 832 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T04:15:55.432 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-IdediqjzmHzciXPYoZ0LGfbTuVKiPxkTffXHBZK2A6vSWJib1Xm6TBspX3qlqiou'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.147243", "end": "2021-02-10 04:15:55.408269", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-IdediqjzmHzciXPYoZ0LGfbTuVKiPxkTffXHBZK2A6vSWJib1Xm6TBspX3qlqiou"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 04:15:55.261026", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.144649 s, 797 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.144649 s, 797 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T04:15:55.477 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-8fgGDcnYh9E7rmMr5l3WgfT5CtBuqtKBhCQooGlSYA6vYjHrmeU1cIvMJvgQxCTx'], 'uuids': ['ff51082a-3d1c-41ff-b41f-c0d53b0423d1'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.142681", "end": "2021-02-10 04:15:55.447581", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-8fgGDcnYh9E7rmMr5l3WgfT5CtBuqtKBhCQooGlSYA6vYjHrmeU1cIvMJvgQxCTx"], "labels": [], "masters": [], "uuids": ["ff51082a-3d1c-41ff-b41f-c0d53b0423d1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 04:15:55.304900", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.140109 s, 823 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.140109 s, 823 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T04:15:55.479 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QA6H', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QA6H', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QA6H', 'scsi-35000c50091e09aef', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QA6H', 'wwn-0x5000c50091e09aef'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5QA6H', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50091e09aef', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QA6H-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QA6H-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QA6H-part1', 'scsi-35000c50091e09aef-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QA6H-part1', 'wwn-0x5000c50091e09aef-part1'], 'uuids': ['782d0384-b480-44b3-9c95-fe95daf886ae'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QA6H", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QA6H", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QA6H", "scsi-35000c50091e09aef", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QA6H", "wwn-0x5000c50091e09aef"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QA6H-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QA6H-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-02-10T04:15:55.481 INFO:teuthology.task.ansible.out:Z1W5QA6H-part1", "scsi-35000c50091e09aef-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QA6H-part1", "wwn-0x5000c50091e09aef-part1"], "labels": [], "masters": [], "uuids": ["782d0384-b480-44b3-9c95-fe95daf886ae"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QA6H", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e09aef"}}, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:55.745 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-IdediqjzmHzciXPYoZ0LGfbTuVKiPxkTMZfVZ14ntixOpauKBnE6fxSBDidDd92Y'], 'uuids': ['8afef42f-59b2-4bfc-93ea-b5508fe2d917'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.142428", "end": "2021-02-10 04:15:55.720634", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-IdediqjzmHzciXPYoZ0LGfbTuVKiPxkTMZfVZ14ntixOpauKBnE6fxSBDidDd92Y"], "labels": [], "masters": [], "uuids": ["8afef42f-59b2-4bfc-93ea-b5508fe2d917"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 04:15:55.578206", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.13995 s, 824 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.13995 s, 824 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T04:15:55.758 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XTQ7', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XTQ7', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XTQ7', 'scsi-35000c5009293f07d', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XTQ7', 'wwn-0x5000c5009293f07d'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XTQ7', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c5009293f07d', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XTQ7-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XTQ7-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XTQ7-part1', 'scsi-35000c5009293f07d-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XTQ7-part1', 'wwn-0x5000c5009293f07d-part1'], 'uuids': ['782d0384-b480-44b3-9c95-fe95daf886ae'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XTQ7", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XTQ7", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XTQ7", "scsi-35000c5009293f07d", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XTQ7", "wwn-0x5000c5009293f07d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XTQ7-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XTQ7-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-02-10T04:15:55.759 INFO:teuthology.task.ansible.out:Z1W5XTQ7-part1", "scsi-35000c5009293f07d-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XTQ7-part1", "wwn-0x5000c5009293f07d-part1"], "labels": [], "masters": [], "uuids": ["782d0384-b480-44b3-9c95-fe95daf886ae"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XTQ7", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009293f07d"}}, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:55.801 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-8fgGDcnYh9E7rmMr5l3WgfT5CtBuqtKBGtvyQ7MgRK2983ki1qAhulNYneXwwx3M'], 'uuids': ['b3535c2b-2b35-4210-a6b5-d296527df85f'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.141364", "end": "2021-02-10 04:15:55.776468", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-8fgGDcnYh9E7rmMr5l3WgfT5CtBuqtKBGtvyQ7MgRK2983ki1qAhulNYneXwwx3M"], "labels": [], "masters": [], "uuids": ["b3535c2b-2b35-4210-a6b5-d296527df85f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 04:15:55.635104", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138801 s, 831 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138801 s, 831 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T04:15:55.848 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:55 +0000 (0:00:01.612) 0:04:59.482 **** 2021-02-10T04:15:56.065 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-IdediqjzmHzciXPYoZ0LGfbTuVKiPxkTOfN9WLRUHub6TqatFWA7tSJ3oFnBf7Ze'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.140253", "end": "2021-02-10 04:15:56.042889", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-IdediqjzmHzciXPYoZ0LGfbTuVKiPxkTOfN9WLRUHub6TqatFWA7tSJ3oFnBf7Ze"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 04:15:55.902636", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.137593 s, 838 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.137593 s, 838 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T04:15:56.105 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:56 +0000 (0:00:00.256) 0:04:59.739 **** 2021-02-10T04:15:56.111 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-02-10T04:15:56.111 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.055442", "end": "2021-02-10 04:15:56.064083", "rc": 0, "start": "2021-02-10 04:15:56.008641", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T04:15:56.147 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:56 +0000 (0:00:00.042) 0:04:59.781 **** 2021-02-10T04:15:56.332 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.046899", "end": "2021-02-10 04:15:56.310439", "rc": 0, "start": "2021-02-10 04:15:56.263540", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T04:15:56.369 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:56 +0000 (0:00:00.221) 0:05:00.003 **** 2021-02-10T04:15:56.387 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-02-10T04:15:56.387 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.049560", "end": "2021-02-10 04:15:56.361394", "rc": 0, "start": "2021-02-10 04:15:56.311834", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T04:15:56.424 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:56 +0000 (0:00:00.055) 0:05:00.058 **** 2021-02-10T04:15:56.451 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-02-10T04:15:56.488 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:56 +0000 (0:00:00.064) 0:05:00.122 **** 2021-02-10T04:15:56.516 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-02-10T04:15:56.517 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:56.554 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:56 +0000 (0:00:00.065) 0:05:00.188 **** 2021-02-10T04:15:56.581 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-02-10T04:15:56.583 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:56.616 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:56 +0000 (0:00:00.061) 0:05:00.250 **** 2021-02-10T04:15:56.625 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-02-10T04:15:56.625 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.043518", "end": "2021-02-10 04:15:56.570253", "rc": 0, "start": "2021-02-10 04:15:56.526735", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T04:15:56.662 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:56 +0000 (0:00:00.046) 0:05:00.297 **** 2021-02-10T04:15:56.669 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-02-10T04:15:56.670 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:56.709 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:56 +0000 (0:00:00.046) 0:05:00.343 **** 2021-02-10T04:15:56.716 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-02-10T04:15:56.755 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:56 +0000 (0:00:00.046) 0:05:00.390 **** 2021-02-10T04:15:56.763 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-02-10T04:15:56.763 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:56.801 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:56 +0000 (0:00:00.045) 0:05:00.435 **** 2021-02-10T04:15:56.808 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-02-10T04:15:56.809 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:56.852 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:56 +0000 (0:00:00.050) 0:05:00.486 **** 2021-02-10T04:15:56.898 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-02-10T04:15:56.899 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:56.933 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:56 +0000 (0:00:00.081) 0:05:00.567 **** 2021-02-10T04:15:56.962 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-02-10T04:15:56.963 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:57.001 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:56 +0000 (0:00:00.067) 0:05:00.635 **** 2021-02-10T04:15:57.009 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-02-10T04:15:57.009 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T04:15:57.050 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:57 +0000 (0:00:00.048) 0:05:00.684 **** 2021-02-10T04:15:57.057 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-02-10T04:15:57.059 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:57.095 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:57 +0000 (0:00:00.045) 0:05:00.730 **** 2021-02-10T04:15:57.101 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-02-10T04:15:57.102 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930419712000, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227153250, 'block_used': 12941858, 'inode_total': 61054976, 'inode_available': 60986956, 'inode_used': 68020, 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227153250, "block_size": 4096, "block_total": 240095108, "block_used": 12941858, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986956, "inode_total": 61054976, "inode_used": 68020, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930419712000, "size_total": 983429562368, "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:57.146 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:57 +0000 (0:00:00.050) 0:05:00.780 **** 2021-02-10T04:15:57.176 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:57.213 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:57 +0000 (0:00:00.067) 0:05:00.848 **** 2021-02-10T04:15:57.243 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T04:15:57.245 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:57.281 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:57 +0000 (0:00:00.068) 0:05:00.916 **** 2021-02-10T04:15:57.296 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-02-10T04:15:57.297 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T04:15:57.332 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:57 +0000 (0:00:00.050) 0:05:00.966 **** 2021-02-10T04:15:57.338 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T04:15:57.339 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:57.377 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:57 +0000 (0:00:00.044) 0:05:01.011 **** 2021-02-10T04:15:57.384 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-02-10T04:15:57.384 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930419712000, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227153250, 'block_used': 12941858, 'inode_total': 61054976, 'inode_available': 60986956, 'inode_used': 68020, 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227153250, "block_size": 4096, "block_total": 240095108, "block_used": 12941858, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986956, "inode_total": 61054976, "inode_used": 68020, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930419712000, "size_total": 983429562368, "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:57.421 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:57 +0000 (0:00:00.043) 0:05:01.055 **** 2021-02-10T04:15:57.427 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:57.463 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:57 +0000 (0:00:00.042) 0:05:01.098 **** 2021-02-10T04:15:57.471 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:57.507 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:57 +0000 (0:00:00.043) 0:05:01.141 **** 2021-02-10T04:15:57.514 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-02-10T04:15:57.516 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:57.551 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:57 +0000 (0:00:00.043) 0:05:01.185 **** 2021-02-10T04:15:57.559 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:57.597 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:57 +0000 (0:00:00.046) 0:05:01.232 **** 2021-02-10T04:15:57.604 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-02-10T04:15:57.605 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:57.643 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:57 +0000 (0:00:00.045) 0:05:01.277 **** 2021-02-10T04:15:57.651 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T04:15:57.652 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:57.694 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:57 +0000 (0:00:00.051) 0:05:01.328 **** 2021-02-10T04:15:57.764 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:57.805 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:57 +0000 (0:00:00.111) 0:05:01.439 **** 2021-02-10T04:15:57.833 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:57.869 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:57 +0000 (0:00:00.063) 0:05:01.503 **** 2021-02-10T04:15:57.908 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-02-10T04:15:57.909 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:15:57.942 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:57 +0000 (0:00:00.072) 0:05:01.576 **** 2021-02-10T04:15:58.405 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-02-10T04:15:58.407 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-02-10T04:15:58.440 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:58 +0000 (0:00:00.498) 0:05:02.074 **** 2021-02-10T04:15:58.448 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-02-10T04:15:58.488 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:15:58 +0000 (0:00:00.047) 0:05:02.122 **** 2021-02-10T04:15:59.070 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-02-10T04:15:59.071 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T04:15:59.072 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T04:15:59.380 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T04:15:59.412 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T04:15:59.709 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T04:15:59.731 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T04:16:00.019 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T04:16:00.052 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T04:16:00.355 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T04:16:00.390 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:00 +0000 (0:00:01.902) 0:05:04.025 **** 2021-02-10T04:16:00.398 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T04:16:00.436 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:00 +0000 (0:00:00.045) 0:05:04.071 **** 2021-02-10T04:16:00.574 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-02-10T04:16:00.576 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-02-10T04:16:00.609 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:00 +0000 (0:00:00.172) 0:05:04.243 **** 2021-02-10T04:16:00.630 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-02-10T04:16:00.677 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:00 +0000 (0:00:00.067) 0:05:04.311 **** 2021-02-10T04:16:00.824 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-02-10T04:16:00.826 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T04:16:00.878 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T04:16:00.994 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T04:16:01.047 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T04:16:01.174 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T04:16:01.217 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T04:16:01.356 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T04:16:01.368 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:01.408 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:01 +0000 (0:00:00.731) 0:05:05.042 **** 2021-02-10T04:16:01.415 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T04:16:01.416 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:01.456 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:01 +0000 (0:00:00.048) 0:05:05.091 **** 2021-02-10T04:16:01.462 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-02-10T04:16:01.464 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:01.499 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:01 +0000 (0:00:00.042) 0:05:05.133 **** 2021-02-10T04:16:01.507 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:01.543 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:01 +0000 (0:00:00.044) 0:05:05.178 **** 2021-02-10T04:16:01.549 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-02-10T04:16:01.550 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:01.589 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:01 +0000 (0:00:00.045) 0:05:05.224 **** 2021-02-10T04:16:01.596 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:01.631 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:01 +0000 (0:00:00.042) 0:05:05.266 **** 2021-02-10T04:16:01.774 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-02-10T04:16:01.775 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T04:16:01.825 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:01 +0000 (0:00:00.193) 0:05:05.459 **** 2021-02-10T04:16:01.832 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T04:16:01.871 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:01 +0000 (0:00:00.045) 0:05:05.505 **** 2021-02-10T04:16:01.880 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-02-10T04:16:01.881 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:01.922 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:01 +0000 (0:00:00.051) 0:05:05.556 **** 2021-02-10T04:16:01.929 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:01.976 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:01 +0000 (0:00:00.054) 0:05:05.611 **** 2021-02-10T04:16:03.514 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-02-10T04:16:03.516 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => {"changed": true} changed: [smithi164.front.sepia.ceph.com] => {"changed": true} 2021-02-10T04:16:03.553 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:03 +0000 (0:00:01.577) 0:05:07.188 **** 2021-02-10T04:16:03.593 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:03 +0000 (0:00:00.039) 0:05:07.228 **** 2021-02-10T04:16:04.127 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-02-10T04:16:04.128 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} changed: [smithi085.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-02-10T04:16:04.162 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:04 +0000 (0:00:00.569) 0:05:07.797 **** 2021-02-10T04:16:04.196 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:04 +0000 (0:00:00.033) 0:05:07.830 **** 2021-02-10T04:16:04.202 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-02-10T04:16:04.204 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:04.250 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:04 +0000 (0:00:00.054) 0:05:07.885 **** 2021-02-10T04:16:04.261 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:04.298 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:04 +0000 (0:00:00.047) 0:05:07.932 **** 2021-02-10T04:16:04.304 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-02-10T04:16:04.306 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:04.339 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:04 +0000 (0:00:00.041) 0:05:07.974 **** 2021-02-10T04:16:04.346 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:04.381 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:04 +0000 (0:00:00.041) 0:05:08.016 **** 2021-02-10T04:16:04.389 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-02-10T04:16:04.391 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:04.423 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:04 +0000 (0:00:00.041) 0:05:08.058 **** 2021-02-10T04:16:04.430 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:04.466 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:04 +0000 (0:00:00.043) 0:05:08.101 **** 2021-02-10T04:16:04.473 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-02-10T04:16:04.474 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:04.509 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:04 +0000 (0:00:00.042) 0:05:08.143 **** 2021-02-10T04:16:04.516 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:04.551 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:04 +0000 (0:00:00.042) 0:05:08.186 **** 2021-02-10T04:16:04.557 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-02-10T04:16:04.559 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:04.595 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:04 +0000 (0:00:00.043) 0:05:08.229 **** 2021-02-10T04:16:04.603 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:04.639 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:04 +0000 (0:00:00.043) 0:05:08.273 **** 2021-02-10T04:16:04.645 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-02-10T04:16:04.645 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:04.692 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:04 +0000 (0:00:00.053) 0:05:08.326 **** 2021-02-10T04:16:04.703 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:04.747 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:04 +0000 (0:00:00.055) 0:05:08.381 **** 2021-02-10T04:16:04.753 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-02-10T04:16:04.755 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:04.789 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:04 +0000 (0:00:00.041) 0:05:08.423 **** 2021-02-10T04:16:04.797 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:04.833 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:04 +0000 (0:00:00.044) 0:05:08.468 **** 2021-02-10T04:16:04.841 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-02-10T04:16:04.841 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:04.886 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:04 +0000 (0:00:00.052) 0:05:08.520 **** 2021-02-10T04:16:04.893 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:04.927 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:04 +0000 (0:00:00.041) 0:05:08.561 **** 2021-02-10T04:16:04.933 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-02-10T04:16:04.935 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:04.968 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:04 +0000 (0:00:00.041) 0:05:08.603 **** 2021-02-10T04:16:04.975 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:05.010 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:05 +0000 (0:00:00.041) 0:05:08.644 **** 2021-02-10T04:16:05.016 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-02-10T04:16:05.018 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:05.051 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:05 +0000 (0:00:00.041) 0:05:08.686 **** 2021-02-10T04:16:05.058 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:05.098 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:05 +0000 (0:00:00.046) 0:05:08.732 **** 2021-02-10T04:16:05.104 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-02-10T04:16:05.106 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:05.143 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:05 +0000 (0:00:00.044) 0:05:08.777 **** 2021-02-10T04:16:05.150 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:05.184 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:05 +0000 (0:00:00.041) 0:05:08.818 **** 2021-02-10T04:16:05.190 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-02-10T04:16:05.191 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:05.226 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:05 +0000 (0:00:00.042) 0:05:08.861 **** 2021-02-10T04:16:05.233 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:05.268 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:05 +0000 (0:00:00.041) 0:05:08.902 **** 2021-02-10T04:16:05.274 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-02-10T04:16:05.275 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:05.311 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:05 +0000 (0:00:00.043) 0:05:08.946 **** 2021-02-10T04:16:05.321 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:05.359 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:05 +0000 (0:00:00.047) 0:05:08.993 **** 2021-02-10T04:16:05.367 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-02-10T04:16:05.368 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:05.402 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:05 +0000 (0:00:00.043) 0:05:09.037 **** 2021-02-10T04:16:05.409 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:05.450 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:05 +0000 (0:00:00.047) 0:05:09.084 **** 2021-02-10T04:16:05.457 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-02-10T04:16:05.459 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:05.502 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:05 +0000 (0:00:00.052) 0:05:09.136 **** 2021-02-10T04:16:05.509 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:05.545 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:05 +0000 (0:00:00.043) 0:05:09.180 **** 2021-02-10T04:16:05.951 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-02-10T04:16:05.988 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:05 +0000 (0:00:00.442) 0:05:09.622 **** 2021-02-10T04:16:06.005 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-02-10T04:16:06.041 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:06 +0000 (0:00:00.053) 0:05:09.675 **** 2021-02-10T04:16:06.042 INFO:teuthology.task.ansible.out: 2021-02-10T04:16:06.282 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-02-10T04:16:06.283 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2021-02-10 04:09:33 UTC", "ActiveEnterTimestampMonotonic": "15270356", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "ntpdate.service system.slice sysinit.target -.mount sntp.service tmp.mount systemd-tmpfiles-setup.service ntpd.service systemd-journald.socket basic.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-02-10 04:09:32 UTC", "AssertTimestampMonotonic": "14282083", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-02-10 04:09:32 UTC", "ConditionTimestampMonotonic": "14282041", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service ntpd.service", "ControlGro 2021-02-10T04:16:06.285 INFO:teuthology.task.ansible.out:up": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "955", "ExecMainStartTimestamp": "Wed 2021-02-10 04:09:33 UTC", "ExecMainStartTimestampMonotonic": "15160010", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-02-10 04:09:32 UTC", "InactiveExitTimestampMonotonic": "14306226", "InvocationID": "21f70bbe10194b63a6ca25d37244de15", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity" 2021-02-10T04:16:06.286 INFO:teuthology.task.ansible.out:, "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127237", "LimitNPROCSoft": "127237", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127237", "LimitSIGPENDINGSoft": "127237", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "955", "MemoryAccounting": "yes", "MemoryCurrent": "3149824", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/var/tmp", "Restart": "no 2021-02-10T04:16:06.287 INFO:teuthology.task.ansible.out:", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2021-02-10 04:09:33 UTC", "StateChangeTimestampMonotonic": "15270356", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203580", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2021-02-10 04:09:33 UTC", "WatchdogTimestampMonotonic": "15160024", "WatchdogUSec": "0"}} 2021-02-10T04:16:06.320 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:06 +0000 (0:00:00.278) 0:05:09.954 **** 2021-02-10T04:16:06.341 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2021-02-10 04:09:44 UTC", "ActiveEnterTimestampMonotonic": "16032878", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket basic.target sntp.service ntpdate.service sysinit.target tmp.mount -.mount ntpd.service systemd-tmpfiles-setup.service system.slice", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-02-10 04:09:43 UTC", "AssertTimestampMonotonic": "14671268", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-02-10 04:09:43 UTC", "ConditionTimestampMonotonic": "14671231", "ConfigurationDirectoryMode": "0755", "Conflicts": "ntpd.service systemd-timesyncd.service shutdown.target", "ControlGro 2021-02-10T04:16:06.342 INFO:teuthology.task.ansible.out:up": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "888", "ExecMainStartTimestamp": "Wed 2021-02-10 04:09:44 UTC", "ExecMainStartTimestampMonotonic": "15893231", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-02-10 04:09:43 UTC", "InactiveExitTimestampMonotonic": "14682178", "InvocationID": "167fa31f4ccc48438147a16082016176", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity" 2021-02-10T04:16:06.344 INFO:teuthology.task.ansible.out:, "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127237", "LimitNPROCSoft": "127237", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127237", "LimitSIGPENDINGSoft": "127237", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "888", "MemoryAccounting": "yes", "MemoryCurrent": "3170304", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp", "Restart": "no 2021-02-10T04:16:06.345 INFO:teuthology.task.ansible.out:", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2021-02-10 04:09:44 UTC", "StateChangeTimestampMonotonic": "16032878", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203580", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2021-02-10 04:09:44 UTC", "WatchdogTimestampMonotonic": "15893242", "WatchdogUSec": "0"}} 2021-02-10T04:16:06.378 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:06 +0000 (0:00:00.058) 0:05:10.013 **** 2021-02-10T04:16:06.386 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T04:16:06.386 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-02-10T04:16:06.424 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:06 +0000 (0:00:00.045) 0:05:10.058 **** 2021-02-10T04:16:06.433 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-02-10T04:16:06.469 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:06 +0000 (0:00:00.045) 0:05:10.104 **** 2021-02-10T04:16:06.476 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T04:16:06.477 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-02-10T04:16:06.520 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:06 +0000 (0:00:00.050) 0:05:10.154 **** 2021-02-10T04:16:06.531 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-02-10T04:16:06.567 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:06 +0000 (0:00:00.047) 0:05:10.202 **** 2021-02-10T04:16:06.576 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-02-10T04:16:06.576 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:06.619 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:06 +0000 (0:00:00.051) 0:05:10.253 **** 2021-02-10T04:16:06.629 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:06.663 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:06 +0000 (0:00:00.043) 0:05:10.297 **** 2021-02-10T04:16:06.669 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-02-10T04:16:06.671 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:06.706 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:06 +0000 (0:00:00.043) 0:05:10.341 **** 2021-02-10T04:16:06.714 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:06.750 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:06 +0000 (0:00:00.043) 0:05:10.384 **** 2021-02-10T04:16:06.756 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-02-10T04:16:06.757 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:06.791 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:06 +0000 (0:00:00.040) 0:05:10.425 **** 2021-02-10T04:16:06.798 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:06.834 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:06 +0000 (0:00:00.043) 0:05:10.468 **** 2021-02-10T04:16:06.993 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-02-10T04:16:06.995 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-02-10T04:16:07.037 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:07 +0000 (0:00:00.203) 0:05:10.672 **** 2021-02-10T04:16:07.048 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-02-10T04:16:07.086 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:07 +0000 (0:00:00.048) 0:05:10.720 **** 2021-02-10T04:16:07.491 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-02-10T04:16:07.492 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-02-10T04:16:07.529 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:07 +0000 (0:00:00.442) 0:05:11.163 **** 2021-02-10T04:16:07.557 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-02-10T04:16:07.592 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:07 +0000 (0:00:00.063) 0:05:11.227 **** 2021-02-10T04:16:08.813 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-02-10T04:16:08.815 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-02-10T04:16:08.851 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:08 +0000 (0:00:01.258) 0:05:12.485 **** 2021-02-10T04:16:08.860 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-02-10T04:16:08.898 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:08 +0000 (0:00:00.046) 0:05:12.532 **** 2021-02-10T04:16:08.905 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-02-10T04:16:08.906 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:08.942 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:08 +0000 (0:00:00.044) 0:05:12.577 **** 2021-02-10T04:16:08.950 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:08.987 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:08 +0000 (0:00:00.044) 0:05:12.621 **** 2021-02-10T04:16:08.994 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-02-10T04:16:08.996 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-02-10T04:16:09.030 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:09 +0000 (0:00:00.043) 0:05:12.664 **** 2021-02-10T04:16:09.037 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-02-10T04:16:09.074 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:09 +0000 (0:00:00.044) 0:05:12.709 **** 2021-02-10T04:16:09.207 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-02-10T04:16:09.209 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-02-10T04:16:09.250 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:09 +0000 (0:00:00.175) 0:05:12.884 **** 2021-02-10T04:16:09.257 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-02-10T04:16:09.293 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:09 +0000 (0:00:00.043) 0:05:12.928 **** 2021-02-10T04:16:09.300 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-02-10T04:16:09.301 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:09.335 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:09 +0000 (0:00:00.041) 0:05:12.969 **** 2021-02-10T04:16:09.342 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:09.385 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:09 +0000 (0:00:00.049) 0:05:13.019 **** 2021-02-10T04:16:09.392 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-02-10T04:16:09.394 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-02-10T04:16:09.434 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:09 +0000 (0:00:00.049) 0:05:13.068 **** 2021-02-10T04:16:09.442 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-02-10T04:16:09.481 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:09 +0000 (0:00:00.047) 0:05:13.115 **** 2021-02-10T04:16:09.488 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-02-10T04:16:09.488 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:09.529 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:09 +0000 (0:00:00.047) 0:05:13.163 **** 2021-02-10T04:16:09.535 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:09.579 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:09 +0000 (0:00:00.050) 0:05:13.214 **** 2021-02-10T04:16:09.707 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-02-10T04:16:09.708 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-02-10T04:16:09.748 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:09 +0000 (0:00:00.169) 0:05:13.383 **** 2021-02-10T04:16:09.766 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-02-10T04:16:09.805 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:09 +0000 (0:00:00.057) 0:05:13.440 **** 2021-02-10T04:16:09.929 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* changed: [smithi164.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-02-10T04:16:09.964 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:09 +0000 (0:00:00.159) 0:05:13.599 **** 2021-02-10T04:16:09.993 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-02-10T04:16:09.994 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:10.027 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:10 +0000 (0:00:00.062) 0:05:13.661 **** 2021-02-10T04:16:10.034 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-02-10T04:16:10.035 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-02-10T04:16:10.076 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:10 +0000 (0:00:00.048) 0:05:13.710 **** 2021-02-10T04:16:10.085 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-02-10T04:16:10.086 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-02-10T04:16:10.088 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-02-10T04:16:10.125 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:10 +0000 (0:00:00.048) 0:05:13.759 **** 2021-02-10T04:16:10.131 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-02-10T04:16:10.132 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:10.167 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:10 +0000 (0:00:00.042) 0:05:13.801 **** 2021-02-10T04:16:10.174 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-02-10T04:16:10.175 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:10.177 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:10.188 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:10.189 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:10.225 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:10 +0000 (0:00:00.058) 0:05:13.860 **** 2021-02-10T04:16:10.231 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-02-10T04:16:10.231 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-02-10T04:16:10.233 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi085.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-02-10T04:16:10.234 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-02-10T04:16:10.270 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:10 +0000 (0:00:00.044) 0:05:13.904 **** 2021-02-10T04:16:10.276 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T04:16:10.277 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:10.318 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:10 +0000 (0:00:00.048) 0:05:13.952 **** 2021-02-10T04:16:10.325 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-02-10T04:16:10.325 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:10.326 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:10.337 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:10.338 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:10.339 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:10.374 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:10 +0000 (0:00:00.056) 0:05:14.008 **** 2021-02-10T04:16:10.381 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-02-10T04:16:10.382 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:10.424 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:10 +0000 (0:00:00.050) 0:05:14.058 **** 2021-02-10T04:16:10.433 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T04:16:10.434 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:10.486 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:10 +0000 (0:00:00.061) 0:05:14.120 **** 2021-02-10T04:16:10.494 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-02-10T04:16:10.494 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:10.532 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:10 +0000 (0:00:00.045) 0:05:14.166 **** 2021-02-10T04:16:10.540 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-02-10T04:16:10.541 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:10.578 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:10 +0000 (0:00:00.046) 0:05:14.212 **** 2021-02-10T04:16:10.586 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-02-10T04:16:10.587 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:10.622 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:10 +0000 (0:00:00.043) 0:05:14.256 **** 2021-02-10T04:16:10.629 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-02-10T04:16:10.630 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:10.670 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:10 +0000 (0:00:00.048) 0:05:14.304 **** 2021-02-10T04:16:10.678 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-02-10T04:16:10.680 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:10.723 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:10 +0000 (0:00:00.053) 0:05:14.358 **** 2021-02-10T04:16:10.737 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-02-10T04:16:10.738 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:10.776 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:10 +0000 (0:00:00.052) 0:05:14.411 **** 2021-02-10T04:16:10.785 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-02-10T04:16:10.786 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:10.830 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:10 +0000 (0:00:00.053) 0:05:14.465 **** 2021-02-10T04:16:10.839 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-02-10T04:16:10.840 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:10.884 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:10 +0000 (0:00:00.053) 0:05:14.518 **** 2021-02-10T04:16:10.895 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-02-10T04:16:10.896 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:10.933 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:10 +0000 (0:00:00.049) 0:05:14.568 **** 2021-02-10T04:16:10.940 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-02-10T04:16:10.941 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:10.986 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:10 +0000 (0:00:00.052) 0:05:14.620 **** 2021-02-10T04:16:10.996 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-02-10T04:16:10.997 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:11.034 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:11 +0000 (0:00:00.047) 0:05:14.668 **** 2021-02-10T04:16:11.041 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-02-10T04:16:11.042 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:11.082 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:11 +0000 (0:00:00.047) 0:05:14.716 **** 2021-02-10T04:16:11.110 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-02-10T04:16:11.112 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:11.145 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:11 +0000 (0:00:00.062) 0:05:14.779 **** 2021-02-10T04:16:12.263 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-02-10T04:16:12.263 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:01.006294", "end": "2021-02-10 04:16:12.220638", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-02-10 04:16:11.214344", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T04:16:12.370 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:01.006162", "end": "2021-02-10 04:16:12.328440", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-02-10 04:16:11.322278", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T04:16:12.445 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.002994", "end": "2021-02-10 04:16:12.412155", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-02-10 04:16:12.409161", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T04:16:12.539 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.002986", "end": "2021-02-10 04:16:12.514305", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-02-10 04:16:12.511319", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T04:16:12.613 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002796", "end": "2021-02-10 04:16:12.579874", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-02-10 04:16:12.577078", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-02-10T04:16:12.707 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002896", "end": "2021-02-10 04:16:12.679436", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-02-10 04:16:12.676540", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-02-10T04:16:12.782 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003165", "end": "2021-02-10 04:16:12.754067", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-02-10 04:16:12.750902", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T04:16:12.828 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:12 +0000 (0:00:01.683) 0:05:16.462 **** 2021-02-10T04:16:12.856 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-02-10T04:16:12.890 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:12 +0000 (0:00:00.062) 0:05:16.525 **** 2021-02-10T04:16:13.068 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-02-10T04:16:13.070 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-02-10T04:16:13.879 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-02-10T04:16:13.880 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:01.004378", "end": "2021-02-10 04:16:13.846165", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-02-10 04:16:12.841787", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T04:16:13.914 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:13 +0000 (0:00:01.023) 0:05:17.548 **** 2021-02-10T04:16:13.941 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-02-10T04:16:13.974 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:13 +0000 (0:00:00.060) 0:05:17.609 **** 2021-02-10T04:16:14.171 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-02-10T04:16:14.172 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-02-10T04:16:14.278 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-02-10T04:16:14.313 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:14 +0000 (0:00:00.338) 0:05:17.947 **** 2021-02-10T04:16:14.348 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:14 +0000 (0:00:00.035) 0:05:17.982 **** 2021-02-10T04:16:14.949 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi085.front.sepia.ceph.com] 2021-02-10T04:16:15.033 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:15 +0000 (0:00:00.684) 0:05:18.667 **** 2021-02-10T04:16:15.047 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] 2021-02-10T04:16:15.089 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T04:16:15.090 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:15.124 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:15 +0000 (0:00:00.090) 0:05:18.758 **** 2021-02-10T04:16:15.159 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:15 +0000 (0:00:00.034) 0:05:18.793 **** 2021-02-10T04:16:15.166 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T04:16:15.167 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-02-10T04:16:15.203 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:15 +0000 (0:00:00.043) 0:05:18.837 **** 2021-02-10T04:16:15.209 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T04:16:15.211 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:15.248 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:15 +0000 (0:00:00.044) 0:05:18.882 **** 2021-02-10T04:16:15.254 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-02-10T04:16:15.254 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-02-10T04:16:15.293 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:15 +0000 (0:00:00.045) 0:05:18.928 **** 2021-02-10T04:16:15.300 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T04:16:15.302 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-02-10T04:16:15.338 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:15 +0000 (0:00:00.044) 0:05:18.972 **** 2021-02-10T04:16:15.375 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-02-10T04:16:15.377 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-02-10T04:16:15.411 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:15 +0000 (0:00:00.072) 0:05:19.045 **** 2021-02-10T04:16:17.800 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-02-10T04:16:17.801 INFO:teuthology.task.ansible.out:ok: [smithi085.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: podman", "Installed: podman-docker"]} 2021-02-10T04:16:17.836 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:17 +0000 (0:00:02.425) 0:05:21.470 **** 2021-02-10T04:16:17.875 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** ok: [smithi085.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-02-10T04:16:17.910 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:17 +0000 (0:00:00.074) 0:05:21.545 **** 2021-02-10T04:16:17.939 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T04:16:17.940 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:17.974 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:17 +0000 (0:00:00.063) 0:05:21.608 **** 2021-02-10T04:16:17.980 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-02-10T04:16:17.981 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: podman", "Installed: podman-docker"]} 2021-02-10T04:16:18.018 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:18 +0000 (0:00:00.044) 0:05:21.653 **** 2021-02-10T04:16:18.056 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T04:16:18.058 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-02-10T04:16:18.092 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:18 +0000 (0:00:00.073) 0:05:21.727 **** 2021-02-10T04:16:18.120 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T04:16:18.121 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:18.156 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:18 +0000 (0:00:00.063) 0:05:21.790 **** 2021-02-10T04:16:18.183 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-02-10T04:16:18.185 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T04:16:18.363 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T04:16:18.374 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T04:16:18.420 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:18 +0000 (0:00:00.263) 0:05:22.054 **** 2021-02-10T04:16:18.541 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T04:16:18.590 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:18 +0000 (0:00:00.170) 0:05:22.225 **** 2021-02-10T04:16:18.952 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-02-10T04:16:18.954 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612930578.4593713-137258401132196/source", "state": "file", "uid": 0} 2021-02-10T04:16:19.112 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612930578.6296167-90940516342302/source", "state": "file", "uid": 0} 2021-02-10T04:16:19.428 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612930578.9576964-134210005586726/source", "state": "file", "uid": 0} 2021-02-10T04:16:19.476 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:19 +0000 (0:00:00.885) 0:05:23.111 **** 2021-02-10T04:16:19.580 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612930579.1145368-32663587388042/source", "state": "file", "uid": 0} 2021-02-10T04:16:19.620 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:19 +0000 (0:00:00.143) 0:05:23.254 **** 2021-02-10T04:16:22.622 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-02-10T04:16:22.623 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-nbjrnqz0-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nInstalling collected packages: toml, docopt, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-nbjrnqz0-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files. 2021-02-10T04:16:22.624 INFO:teuthology.task.ansible.out:pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Installing collected packages: toml, docopt, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-02-10T04:16:22.663 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:22 +0000 (0:00:03.043) 0:05:26.298 **** 2021-02-10T04:16:22.702 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-02-10T04:16:22.704 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:22.743 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:22 +0000 (0:00:00.079) 0:05:26.377 **** 2021-02-10T04:16:22.760 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-02-10T04:16:22.761 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-030ro3ce-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nInstalling collected packages: toml, docopt, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-030ro3ce-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files. 2021-02-10T04:16:22.762 INFO:teuthology.task.ansible.out:pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Installing collected packages: toml, docopt, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-02-10T04:16:22.797 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:22 +0000 (0:00:00.054) 0:05:26.431 **** 2021-02-10T04:16:22.803 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-02-10T04:16:22.805 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:22.843 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:22 +0000 (0:00:00.045) 0:05:26.477 **** 2021-02-10T04:16:22.849 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-02-10T04:16:22.850 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:22.888 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:22 +0000 (0:00:00.045) 0:05:26.522 **** 2021-02-10T04:16:22.925 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-02-10T04:16:22.927 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:22.961 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:22 +0000 (0:00:00.073) 0:05:26.595 **** 2021-02-10T04:16:23.186 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-02-10T04:16:23.188 INFO:teuthology.task.ansible.out:changed: [smithi085.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.153314", "end": "2021-02-10 04:16:23.161787", "rc": 0, "start": "2021-02-10 04:16:23.008473", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T04:16:23.229 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:23 +0000 (0:00:00.267) 0:05:26.863 **** 2021-02-10T04:16:23.257 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-02-10T04:16:23.259 INFO:teuthology.task.ansible.out:skipping: [smithi085.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:23.328 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-02-10T04:16:23.330 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.159718", "end": "2021-02-10 04:16:23.284401", "rc": 0, "start": "2021-02-10 04:16:23.124683", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T04:16:23.363 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 04:16:23 +0000 (0:00:00.133) 0:05:26.997 **** 2021-02-10T04:16:23.400 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-02-10T04:16:23.401 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T04:16:23.505 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-02-10T04:16:23.509 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** 2021-02-10T04:16:23.511 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2021-02-10T04:16:23.514 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* 2021-02-10T04:16:23.516 INFO:teuthology.task.ansible.out:smithi085.front.sepia.ceph.com : ok=136 changed=44 unreachable=0 failed=0 skipped=174 rescued=0 ignored=4 smithi164.front.sepia.ceph.com : ok=136 changed=44 unreachable=0 failed=0 skipped=174 rescued=0 ignored=4 Wednesday 10 February 2021 04:16:23 +0000 (0:00:00.152) 0:05:27.149 **** =============================================================================== common : nrpe - Ensure SELinux policy is up to date -------------------- 39.92s testnode : Install packages -------------------------------------------- 36.22s users : Update authorized_keys using the keys repo --------------------- 32.67s users : Create all admin users with sudo access. ----------------------- 31.41s common : nrpe - Load SELinux policy package ---------------------------- 15.81s common : nrpe - Remove SELinux policy package -------------------------- 15.68s common : Register with subscription-manager. --------------------------- 13.62s users : Remove revoked users ------------------------------------------- 11.66s testnode : Zap all non-root disks --------------------------------------- 8.12s testnode : Ensure ceph packages are not present. ------------------------ 5.71s testnode : Ensure ceph dependency packages are not present. ------------- 5.24s container-host : Install registries-conf-ctl ---------------------------- 3.04s testnode : Make sure rpm dependencies are installed --------------------- 2.51s container-host : Install container packages ----------------------------- 2.43s testnode : Remove packages ---------------------------------------------- 2.36s testnode : Install epel packages ---------------------------------------- 2.30s testnode : Rebuild rpmdb ------------------------------------------------ 2.16s common : Enable necessary rhsm repos ------------------------------------ 2.16s users : Update authorized_keys for each user with literal keys ---------- 2.06s testnode : Create logical volume(s) ------------- 2021-02-10T04:16:23.517 INFO:teuthology.task.ansible.out:------------------------ 1.90s 2021-02-10T04:16:23.845 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi085.front.sepia.ceph.com'), Remote(name='ubuntu@smithi164.front.sepia.ceph.com')] 2021-02-10T04:16:23.846 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-10T04:16:23.847 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi085.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-10T04:16:23.922 DEBUG:teuthology.orchestra.run.smithi085:> true 2021-02-10T04:16:24.015 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-10T04:16:24.016 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-10T04:16:24.085 DEBUG:teuthology.orchestra.run.smithi164:> true 2021-02-10T04:16:24.178 INFO:teuthology.run_tasks:Running task clock... 2021-02-10T04:16:24.196 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-02-10T04:16:24.196 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-02-10T04:16:24.197 DEBUG:teuthology.orchestra.run.smithi085:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-10T04:16:24.228 INFO:teuthology.orchestra.run.smithi085.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-02-10T04:16:24.242 INFO:teuthology.orchestra.run.smithi085.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-02-10T04:16:24.277 INFO:teuthology.orchestra.run.smithi085.stderr:sudo: ntpd: command not found 2021-02-10T04:16:24.287 INFO:teuthology.orchestra.run.smithi085.stdout:506 Cannot talk to daemon 2021-02-10T04:16:24.301 INFO:teuthology.orchestra.run.smithi085.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-02-10T04:16:24.315 INFO:teuthology.orchestra.run.smithi085.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-02-10T04:16:24.374 INFO:teuthology.orchestra.run.smithi085.stderr:bash: ntpq: command not found 2021-02-10T04:16:24.377 INFO:teuthology.orchestra.run.smithi085.stdout:210 Number of sources = 4 2021-02-10T04:16:24.377 INFO:teuthology.orchestra.run.smithi085.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-02-10T04:16:24.377 INFO:teuthology.orchestra.run.smithi085.stdout:=============================================================================== 2021-02-10T04:16:24.377 INFO:teuthology.orchestra.run.smithi085.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-02-10T04:16:24.377 INFO:teuthology.orchestra.run.smithi085.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-02-10T04:16:24.377 INFO:teuthology.orchestra.run.smithi085.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-02-10T04:16:24.377 INFO:teuthology.orchestra.run.smithi085.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-02-10T04:16:24.378 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-02-10T04:16:24.378 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-10T04:16:24.411 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-02-10T04:16:24.426 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-02-10T04:16:24.460 INFO:teuthology.orchestra.run.smithi164.stderr:sudo: ntpd: command not found 2021-02-10T04:16:24.471 INFO:teuthology.orchestra.run.smithi164.stdout:506 Cannot talk to daemon 2021-02-10T04:16:24.485 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-02-10T04:16:24.499 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-02-10T04:16:24.559 INFO:teuthology.orchestra.run.smithi164.stderr:bash: ntpq: command not found 2021-02-10T04:16:24.563 INFO:teuthology.orchestra.run.smithi164.stdout:210 Number of sources = 4 2021-02-10T04:16:24.563 INFO:teuthology.orchestra.run.smithi164.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-02-10T04:16:24.563 INFO:teuthology.orchestra.run.smithi164.stdout:=============================================================================== 2021-02-10T04:16:24.563 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-02-10T04:16:24.563 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-02-10T04:16:24.563 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-02-10T04:16:24.563 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-02-10T04:16:24.563 INFO:teuthology.run_tasks:Running task install... 2021-02-10T04:16:28.636 DEBUG:teuthology.task.install:project ceph 2021-02-10T04:16:28.637 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': 'c1ea624123d412aff8b9d1430e36cb45fcab76b8'}} 2021-02-10T04:16:28.637 DEBUG:teuthology.task.install:config {'sha1': 'c1ea624123d412aff8b9d1430e36cb45fcab76b8'} 2021-02-10T04:16:28.637 INFO:teuthology.task.install:Using flavor: basic 2021-02-10T04:16:28.650 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-02-10T04:16:28.650 INFO:teuthology.task.install:extra packages: [] 2021-02-10T04:16:28.650 DEBUG:teuthology.orchestra.run.smithi085:> sudo lsb_release -is 2021-02-10T04:16:28.696 INFO:teuthology.orchestra.run.smithi085.stdout:RedHatEnterprise 2021-02-10T04:16:28.697 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-02-10T04:16:28.698 DEBUG:teuthology.orchestra.run.smithi164:> sudo lsb_release -is 2021-02-10T04:16:28.698 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c1ea624123d412aff8b9d1430e36cb45fcab76b8', 'tag': None, 'wait_for_package': False} 2021-02-10T04:16:28.698 DEBUG:teuthology.orchestra.run.smithi085:> sudo lsb_release -is 2021-02-10T04:16:28.738 INFO:teuthology.orchestra.run.smithi085.stdout:RedHatEnterprise 2021-02-10T04:16:28.739 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-02-10T04:16:28.740 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c1ea624123d412aff8b9d1430e36cb45fcab76b8 2021-02-10T04:16:28.746 INFO:teuthology.orchestra.run.smithi164.stdout:RedHatEnterprise 2021-02-10T04:16:28.746 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-02-10T04:16:28.747 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c1ea624123d412aff8b9d1430e36cb45fcab76b8', 'tag': None, 'wait_for_package': False} 2021-02-10T04:16:28.747 DEBUG:teuthology.orchestra.run.smithi164:> sudo lsb_release -is 2021-02-10T04:16:28.787 INFO:teuthology.orchestra.run.smithi164.stdout:RedHatEnterprise 2021-02-10T04:16:28.788 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-02-10T04:16:28.788 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c1ea624123d412aff8b9d1430e36cb45fcab76b8 2021-02-10T04:16:28.923 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-02-09-1332/c1ea624123d412aff8b9d1430e36cb45fcab76b8/centos/8/flavors/default/ 2021-02-10T04:16:28.923 INFO:teuthology.task.install.rpm:Package version is 17.0.0-681.gc1ea6241 2021-02-10T04:16:28.958 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-02-09-1332/c1ea624123d412aff8b9d1430e36cb45fcab76b8/centos/8/flavors/default/ 2021-02-10T04:16:28.958 INFO:teuthology.task.install.rpm:Package version is 17.0.0-681.gc1ea6241 2021-02-10T04:16:29.049 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-02-09-1332/c1ea624123d412aff8b9d1430e36cb45fcab76b8/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-02-09-1332/c1ea624123d412aff8b9d1430e36cb45fcab76b8/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-02-09-1332/c1ea624123d412aff8b9d1430e36cb45fcab76b8/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-02-10T04:16:29.049 DEBUG:teuthology.orchestra.run.smithi085:> set -ex 2021-02-10T04:16:29.049 DEBUG:teuthology.orchestra.run.smithi085:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-02-10T04:16:29.076 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-02-09-1332/c1ea624123d412aff8b9d1430e36cb45fcab76b8/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-02-09-1332/c1ea624123d412aff8b9d1430e36cb45fcab76b8/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-sage2-testing-2021-02-09-1332/c1ea624123d412aff8b9d1430e36cb45fcab76b8/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-02-10T04:16:29.077 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2021-02-10T04:16:29.077 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-02-10T04:16:29.078 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-02-10T04:16:29.079 DEBUG:teuthology.orchestra.run.smithi085:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c1ea624123d412aff8b9d1430e36cb45fcab76b8/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-02-10T04:16:29.108 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-02-10T04:16:29.109 DEBUG:teuthology.orchestra.run.smithi164:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c1ea624123d412aff8b9d1430e36cb45fcab76b8/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-02-10T04:16:29.148 DEBUG:teuthology.orchestra.run.smithi085:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-02-10T04:16:29.176 DEBUG:teuthology.orchestra.run.smithi164:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-02-10T04:16:29.225 DEBUG:teuthology.orchestra.run.smithi085:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-02-10T04:16:29.254 DEBUG:teuthology.orchestra.run.smithi164:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-02-10T04:16:29.290 INFO:teuthology.orchestra.run.smithi085.stdout:check_obsoletes = 1 2021-02-10T04:16:29.291 DEBUG:teuthology.orchestra.run.smithi085:> sudo yum clean all 2021-02-10T04:16:29.318 INFO:teuthology.orchestra.run.smithi164.stdout:check_obsoletes = 1 2021-02-10T04:16:29.319 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum clean all 2021-02-10T04:16:29.598 INFO:teuthology.orchestra.run.smithi085.stdout:Updating Subscription Management repositories. 2021-02-10T04:16:29.627 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2021-02-10T04:16:29.856 INFO:teuthology.orchestra.run.smithi085.stdout:120 files removed 2021-02-10T04:16:29.891 INFO:teuthology.orchestra.run.smithi164.stdout:120 files removed 2021-02-10T04:16:29.893 DEBUG:teuthology.orchestra.run.smithi085:> sudo yum -y install ceph-radosgw 2021-02-10T04:16:29.933 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-radosgw 2021-02-10T04:16:30.156 INFO:teuthology.orchestra.run.smithi085.stdout:Updating Subscription Management repositories. 2021-02-10T04:16:30.199 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2021-02-10T04:16:30.892 INFO:teuthology.orchestra.run.smithi085.stdout:ceph packages for x86_64 269 kB/s | 73 kB 00:00 2021-02-10T04:16:30.892 INFO:teuthology.orchestra.run.smithi164.stdout:ceph packages for x86_64 269 kB/s | 73 kB 00:00 2021-02-10T04:16:31.130 INFO:teuthology.orchestra.run.smithi164.stdout:ceph noarch packages 73 kB/s | 15 kB 00:00 2021-02-10T04:16:31.131 INFO:teuthology.orchestra.run.smithi085.stdout:ceph noarch packages 73 kB/s | 15 kB 00:00 2021-02-10T04:16:31.359 INFO:teuthology.orchestra.run.smithi085.stdout:ceph source packages 6.9 kB/s | 1.4 kB 00:00 2021-02-10T04:16:31.362 INFO:teuthology.orchestra.run.smithi164.stdout:ceph source packages 6.8 kB/s | 1.4 kB 00:00 2021-02-10T04:16:31.792 INFO:teuthology.orchestra.run.smithi164.stdout:Extra Packages for Enterprise Linux 21 MB/s | 8.8 MB 00:00 2021-02-10T04:16:31.832 INFO:teuthology.orchestra.run.smithi085.stdout:Extra Packages for Enterprise Linux 19 MB/s | 8.8 MB 00:00 2021-02-10T04:16:33.569 INFO:teuthology.orchestra.run.smithi164.stdout:lab-extras 3.8 MB/s | 24 kB 00:00 2021-02-10T04:16:33.588 INFO:teuthology.orchestra.run.smithi085.stdout:lab-extras 4.3 MB/s | 24 kB 00:00 2021-02-10T04:16:33.838 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 124 MB/s | 27 MB 00:00 2021-02-10T04:16:33.858 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 124 MB/s | 27 MB 00:00 2021-02-10T04:16:37.719 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 115 MB/s | 25 MB 00:00 2021-02-10T04:16:37.798 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 115 MB/s | 25 MB 00:00 2021-02-10T04:16:43.833 INFO:teuthology.orchestra.run.smithi085.stdout:Dependencies resolved. 2021-02-10T04:16:43.844 INFO:teuthology.orchestra.run.smithi085.stdout:=================================================================================================== 2021-02-10T04:16:43.845 INFO:teuthology.orchestra.run.smithi085.stdout: Package Arch Version Repository Size 2021-02-10T04:16:43.845 INFO:teuthology.orchestra.run.smithi085.stdout:=================================================================================================== 2021-02-10T04:16:43.845 INFO:teuthology.orchestra.run.smithi085.stdout:Installing: 2021-02-10T04:16:43.845 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-radosgw x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 13 M 2021-02-10T04:16:43.845 INFO:teuthology.orchestra.run.smithi085.stdout:Upgrading: 2021-02-10T04:16:43.845 INFO:teuthology.orchestra.run.smithi085.stdout: librados2 x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 3.7 M 2021-02-10T04:16:43.845 INFO:teuthology.orchestra.run.smithi085.stdout: librbd1 x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 3.3 M 2021-02-10T04:16:43.845 INFO:teuthology.orchestra.run.smithi085.stdout:Installing dependencies: 2021-02-10T04:16:43.845 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-base x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 6.0 M 2021-02-10T04:16:43.845 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-common x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 24 M 2021-02-10T04:16:43.845 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-selinux x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 25 k 2021-02-10T04:16:43.846 INFO:teuthology.orchestra.run.smithi085.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2021-02-10T04:16:43.846 INFO:teuthology.orchestra.run.smithi085.stdout: gperftools-libs x86_64 2.7-6.el8 epel 315 k 2021-02-10T04:16:43.846 INFO:teuthology.orchestra.run.smithi085.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-02-10T04:16:43.846 INFO:teuthology.orchestra.run.smithi085.stdout: libcephfs2 x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 798 k 2021-02-10T04:16:43.846 INFO:teuthology.orchestra.run.smithi085.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-02-10T04:16:43.846 INFO:teuthology.orchestra.run.smithi085.stdout: librabbitmq x86_64 0.9.0-2.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2021-02-10T04:16:43.846 INFO:teuthology.orchestra.run.smithi085.stdout: libradosstriper1 x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 486 k 2021-02-10T04:16:43.846 INFO:teuthology.orchestra.run.smithi085.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2021-02-10T04:16:43.846 INFO:teuthology.orchestra.run.smithi085.stdout: librgw2 x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 4.0 M 2021-02-10T04:16:43.846 INFO:teuthology.orchestra.run.smithi085.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-02-10T04:16:43.846 INFO:teuthology.orchestra.run.smithi085.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2021-02-10T04:16:43.846 INFO:teuthology.orchestra.run.smithi085.stdout: python3-ceph-argparse x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 44 k 2021-02-10T04:16:43.846 INFO:teuthology.orchestra.run.smithi085.stdout: python3-ceph-common x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 77 k 2021-02-10T04:16:43.847 INFO:teuthology.orchestra.run.smithi085.stdout: python3-cephfs x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 186 k 2021-02-10T04:16:43.847 INFO:teuthology.orchestra.run.smithi085.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2021-02-10T04:16:43.847 INFO:teuthology.orchestra.run.smithi085.stdout: python3-rados x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 387 k 2021-02-10T04:16:43.847 INFO:teuthology.orchestra.run.smithi085.stdout: python3-rbd x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 368 k 2021-02-10T04:16:43.847 INFO:teuthology.orchestra.run.smithi085.stdout: python3-rgw x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 114 k 2021-02-10T04:16:43.847 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:16:43.847 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction Summary 2021-02-10T04:16:43.847 INFO:teuthology.orchestra.run.smithi085.stdout:=================================================================================================== 2021-02-10T04:16:43.847 INFO:teuthology.orchestra.run.smithi085.stdout:Install 22 Packages 2021-02-10T04:16:43.847 INFO:teuthology.orchestra.run.smithi085.stdout:Upgrade 2 Packages 2021-02-10T04:16:43.847 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:16:43.848 INFO:teuthology.orchestra.run.smithi085.stdout:Total download size: 57 M 2021-02-10T04:16:43.848 INFO:teuthology.orchestra.run.smithi085.stdout:Downloading Packages: 2021-02-10T04:16:44.051 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2021-02-10T04:16:44.063 INFO:teuthology.orchestra.run.smithi164.stdout:=================================================================================================== 2021-02-10T04:16:44.063 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2021-02-10T04:16:44.063 INFO:teuthology.orchestra.run.smithi164.stdout:=================================================================================================== 2021-02-10T04:16:44.063 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2021-02-10T04:16:44.063 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-radosgw x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 13 M 2021-02-10T04:16:44.063 INFO:teuthology.orchestra.run.smithi164.stdout:Upgrading: 2021-02-10T04:16:44.063 INFO:teuthology.orchestra.run.smithi164.stdout: librados2 x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 3.7 M 2021-02-10T04:16:44.063 INFO:teuthology.orchestra.run.smithi164.stdout: librbd1 x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 3.3 M 2021-02-10T04:16:44.063 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2021-02-10T04:16:44.063 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-base x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 6.0 M 2021-02-10T04:16:44.063 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-common x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 24 M 2021-02-10T04:16:44.064 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-selinux x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 25 k 2021-02-10T04:16:44.064 INFO:teuthology.orchestra.run.smithi164.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2021-02-10T04:16:44.064 INFO:teuthology.orchestra.run.smithi164.stdout: gperftools-libs x86_64 2.7-6.el8 epel 315 k 2021-02-10T04:16:44.064 INFO:teuthology.orchestra.run.smithi164.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-02-10T04:16:44.064 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs2 x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 798 k 2021-02-10T04:16:44.064 INFO:teuthology.orchestra.run.smithi164.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-02-10T04:16:44.064 INFO:teuthology.orchestra.run.smithi164.stdout: librabbitmq x86_64 0.9.0-2.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2021-02-10T04:16:44.064 INFO:teuthology.orchestra.run.smithi164.stdout: libradosstriper1 x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 486 k 2021-02-10T04:16:44.064 INFO:teuthology.orchestra.run.smithi164.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2021-02-10T04:16:44.064 INFO:teuthology.orchestra.run.smithi164.stdout: librgw2 x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 4.0 M 2021-02-10T04:16:44.064 INFO:teuthology.orchestra.run.smithi164.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-02-10T04:16:44.064 INFO:teuthology.orchestra.run.smithi164.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2021-02-10T04:16:44.065 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-argparse x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 44 k 2021-02-10T04:16:44.065 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-common x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 77 k 2021-02-10T04:16:44.065 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cephfs x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 186 k 2021-02-10T04:16:44.065 INFO:teuthology.orchestra.run.smithi164.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2021-02-10T04:16:44.065 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rados x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 387 k 2021-02-10T04:16:44.065 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rbd x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 368 k 2021-02-10T04:16:44.065 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rgw x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 114 k 2021-02-10T04:16:44.065 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:16:44.065 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2021-02-10T04:16:44.065 INFO:teuthology.orchestra.run.smithi164.stdout:=================================================================================================== 2021-02-10T04:16:44.065 INFO:teuthology.orchestra.run.smithi164.stdout:Install 22 Packages 2021-02-10T04:16:44.065 INFO:teuthology.orchestra.run.smithi164.stdout:Upgrade 2 Packages 2021-02-10T04:16:44.065 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:16:44.066 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 57 M 2021-02-10T04:16:44.066 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2021-02-10T04:16:44.543 INFO:teuthology.orchestra.run.smithi085.stdout:(1/24): ceph-base-17.0.0-681.gc1ea6241.el8.x86_ 8.8 MB/s | 6.0 MB 00:00 2021-02-10T04:16:44.587 INFO:teuthology.orchestra.run.smithi085.stdout:(2/24): ceph-selinux-17.0.0-681.gc1ea6241.el8.x 566 kB/s | 25 kB 00:00 2021-02-10T04:16:44.748 INFO:teuthology.orchestra.run.smithi085.stdout:(3/24): libcephfs2-17.0.0-681.gc1ea6241.el8.x86 4.8 MB/s | 798 kB 00:00 2021-02-10T04:16:44.847 INFO:teuthology.orchestra.run.smithi085.stdout:(4/24): libradosstriper1-17.0.0-681.gc1ea6241.e 4.8 MB/s | 486 kB 00:00 2021-02-10T04:16:45.227 INFO:teuthology.orchestra.run.smithi085.stdout:(5/24): librgw2-17.0.0-681.gc1ea6241.el8.x86_64 11 MB/s | 4.0 MB 00:00 2021-02-10T04:16:45.304 INFO:teuthology.orchestra.run.smithi085.stdout:(6/24): python3-ceph-argparse-17.0.0-681.gc1ea6 578 kB/s | 44 kB 00:00 2021-02-10T04:16:45.357 INFO:teuthology.orchestra.run.smithi085.stdout:(7/24): python3-ceph-common-17.0.0-681.gc1ea624 1.4 MB/s | 77 kB 00:00 2021-02-10T04:16:45.425 INFO:teuthology.orchestra.run.smithi085.stdout:(8/24): python3-cephfs-17.0.0-681.gc1ea6241.el8 2.7 MB/s | 186 kB 00:00 2021-02-10T04:16:45.494 INFO:teuthology.orchestra.run.smithi085.stdout:(9/24): python3-rados-17.0.0-681.gc1ea6241.el8. 5.5 MB/s | 387 kB 00:00 2021-02-10T04:16:45.557 INFO:teuthology.orchestra.run.smithi085.stdout:(10/24): python3-rbd-17.0.0-681.gc1ea6241.el8.x 5.7 MB/s | 368 kB 00:00 2021-02-10T04:16:45.610 INFO:teuthology.orchestra.run.smithi085.stdout:(11/24): python3-rgw-17.0.0-681.gc1ea6241.el8.x 2.1 MB/s | 114 kB 00:00 2021-02-10T04:16:45.794 INFO:teuthology.orchestra.run.smithi085.stdout:(12/24): ceph-radosgw-17.0.0-681.gc1ea6241.el8. 6.5 MB/s | 13 MB 00:01 2021-02-10T04:16:45.794 INFO:teuthology.orchestra.run.smithi085.stdout:(13/24): fmt-6.2.1-1.el8.x86_64.rpm 633 kB/s | 116 kB 00:00 2021-02-10T04:16:45.833 INFO:teuthology.orchestra.run.smithi085.stdout:(14/24): leveldb-1.22-1.el8.x86_64.rpm 4.5 MB/s | 181 kB 00:00 2021-02-10T04:16:45.854 INFO:teuthology.orchestra.run.smithi085.stdout:(15/24): liboath-2.6.2-3.el8.x86_64.rpm 2.9 MB/s | 59 kB 00:00 2021-02-10T04:16:45.874 INFO:teuthology.orchestra.run.smithi085.stdout:(16/24): libunwind-1.3.1-3.el8.x86_64.rpm 3.7 MB/s | 75 kB 00:00 2021-02-10T04:16:45.891 INFO:teuthology.orchestra.run.smithi085.stdout:(17/24): gperftools-libs-2.7-6.el8.x86_64.rpm 3.2 MB/s | 315 kB 00:00 2021-02-10T04:16:45.904 INFO:teuthology.orchestra.run.smithi085.stdout:(18/24): librabbitmq-0.9.0-2.el8.x86_64.rpm 1.5 MB/s | 47 kB 00:00 2021-02-10T04:16:45.926 INFO:teuthology.orchestra.run.smithi085.stdout:(19/24): python3-prettytable-0.7.2-14.el8.noarc 1.2 MB/s | 44 kB 00:00 2021-02-10T04:16:45.935 INFO:teuthology.orchestra.run.smithi085.stdout:(20/24): librdkafka-0.11.4-1.el8.x86_64.rpm 11 MB/s | 353 kB 00:00 2021-02-10T04:16:45.947 INFO:teuthology.orchestra.run.smithi085.stdout:(21/24): lttng-ust-2.8.1-11.el8.x86_64.rpm 12 MB/s | 259 kB 00:00 2021-02-10T04:16:46.598 INFO:teuthology.orchestra.run.smithi085.stdout:(22/24): ceph-common-17.0.0-681.gc1ea6241.el8.x 8.7 MB/s | 24 MB 00:02 2021-02-10T04:16:46.768 INFO:teuthology.orchestra.run.smithi085.stdout:(23/24): librados2-17.0.0-681.gc1ea6241.el8.x86 4.5 MB/s | 3.7 MB 00:00 2021-02-10T04:16:46.830 INFO:teuthology.orchestra.run.smithi085.stdout:(24/24): librbd1-17.0.0-681.gc1ea6241.el8.x86_6 3.7 MB/s | 3.3 MB 00:00 2021-02-10T04:16:46.833 INFO:teuthology.orchestra.run.smithi085.stdout:-------------------------------------------------------------------------------- 2021-02-10T04:16:46.833 INFO:teuthology.orchestra.run.smithi085.stdout:Total 19 MB/s | 57 MB 00:02 2021-02-10T04:16:46.838 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction check 2021-02-10T04:16:46.881 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction check succeeded. 2021-02-10T04:16:46.881 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction test 2021-02-10T04:16:47.316 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction test succeeded. 2021-02-10T04:16:47.320 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction 2021-02-10T04:16:48.041 INFO:teuthology.orchestra.run.smithi164.stdout:(1/24): ceph-base-17.0.0-681.gc1ea6241.el8.x86_ 1.5 MB/s | 6.0 MB 00:03 2021-02-10T04:16:48.064 INFO:teuthology.orchestra.run.smithi164.stdout:(2/24): ceph-selinux-17.0.0-681.gc1ea6241.el8.x 1.0 MB/s | 25 kB 00:00 2021-02-10T04:16:48.271 INFO:teuthology.orchestra.run.smithi164.stdout:(3/24): libcephfs2-17.0.0-681.gc1ea6241.el8.x86 3.8 MB/s | 798 kB 00:00 2021-02-10T04:16:48.356 INFO:teuthology.orchestra.run.smithi085.stdout: Preparing : 1/1 2021-02-10T04:16:48.389 INFO:teuthology.orchestra.run.smithi164.stdout:(4/24): libradosstriper1-17.0.0-681.gc1ea6241.e 4.1 MB/s | 486 kB 00:00 2021-02-10T04:16:48.597 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/26 2021-02-10T04:16:48.792 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/26 2021-02-10T04:16:48.819 INFO:teuthology.orchestra.run.smithi164.stdout:(5/24): ceph-radosgw-17.0.0-681.gc1ea6241.el8.x 2.6 MB/s | 13 MB 00:04 2021-02-10T04:16:48.845 INFO:teuthology.orchestra.run.smithi164.stdout:(6/24): python3-ceph-argparse-17.0.0-681.gc1ea6 1.7 MB/s | 44 kB 00:00 2021-02-10T04:16:48.869 INFO:teuthology.orchestra.run.smithi164.stdout:(7/24): python3-ceph-common-17.0.0-681.gc1ea624 3.2 MB/s | 77 kB 00:00 2021-02-10T04:16:48.917 INFO:teuthology.orchestra.run.smithi164.stdout:(8/24): python3-cephfs-17.0.0-681.gc1ea6241.el8 3.8 MB/s | 186 kB 00:00 2021-02-10T04:16:48.987 INFO:teuthology.orchestra.run.smithi164.stdout:(9/24): python3-rados-17.0.0-681.gc1ea6241.el8. 5.4 MB/s | 387 kB 00:00 2021-02-10T04:16:49.057 INFO:teuthology.orchestra.run.smithi164.stdout:(10/24): python3-rbd-17.0.0-681.gc1ea6241.el8.x 5.1 MB/s | 368 kB 00:00 2021-02-10T04:16:49.082 INFO:teuthology.orchestra.run.smithi164.stdout:(11/24): python3-rgw-17.0.0-681.gc1ea6241.el8.x 4.6 MB/s | 114 kB 00:00 2021-02-10T04:16:49.163 INFO:teuthology.orchestra.run.smithi164.stdout:(12/24): fmt-6.2.1-1.el8.x86_64.rpm 1.4 MB/s | 116 kB 00:00 2021-02-10T04:16:49.196 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/26 2021-02-10T04:16:49.196 INFO:teuthology.orchestra.run.smithi164.stdout:(13/24): gperftools-libs-2.7-6.el8.x86_64.rpm 9.3 MB/s | 315 kB 00:00 2021-02-10T04:16:49.214 INFO:teuthology.orchestra.run.smithi164.stdout:(14/24): leveldb-1.22-1.el8.x86_64.rpm 10 MB/s | 181 kB 00:00 2021-02-10T04:16:49.230 INFO:teuthology.orchestra.run.smithi164.stdout:(15/24): liboath-2.6.2-3.el8.x86_64.rpm 3.5 MB/s | 59 kB 00:00 2021-02-10T04:16:49.248 INFO:teuthology.orchestra.run.smithi164.stdout:(16/24): libunwind-1.3.1-3.el8.x86_64.rpm 4.4 MB/s | 75 kB 00:00 2021-02-10T04:16:49.278 INFO:teuthology.orchestra.run.smithi164.stdout:(17/24): librabbitmq-0.9.0-2.el8.x86_64.rpm 1.5 MB/s | 47 kB 00:00 2021-02-10T04:16:49.295 INFO:teuthology.orchestra.run.smithi164.stdout:(18/24): librgw2-17.0.0-681.gc1ea6241.el8.x86_6 4.5 MB/s | 4.0 MB 00:00 2021-02-10T04:16:49.307 INFO:teuthology.orchestra.run.smithi164.stdout:(19/24): python3-prettytable-0.7.2-14.el8.noarc 1.5 MB/s | 44 kB 00:00 2021-02-10T04:16:49.332 INFO:teuthology.orchestra.run.smithi164.stdout:(20/24): librdkafka-0.11.4-1.el8.x86_64.rpm 9.4 MB/s | 353 kB 00:00 2021-02-10T04:16:49.334 INFO:teuthology.orchestra.run.smithi164.stdout:(21/24): lttng-ust-2.8.1-11.el8.x86_64.rpm 9.5 MB/s | 259 kB 00:00 2021-02-10T04:16:49.392 INFO:teuthology.orchestra.run.smithi085.stdout: Upgrading : librados2-2:17.0.0-681.gc1ea6241.el8.x86_64 3/26 2021-02-10T04:16:49.498 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: librados2-2:17.0.0-681.gc1ea6241.el8.x86_64 3/26 2021-02-10T04:16:49.708 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-rados-2:17.0.0-681.gc1ea6241.el8.x86_64 4/26 2021-02-10T04:16:49.885 INFO:teuthology.orchestra.run.smithi164.stdout:(22/24): librbd1-17.0.0-681.gc1ea6241.el8.x86_6 6.0 MB/s | 3.3 MB 00:00 2021-02-10T04:16:49.923 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/26 2021-02-10T04:16:49.996 INFO:teuthology.orchestra.run.smithi164.stdout:(23/24): librados2-17.0.0-681.gc1ea6241.el8.x86 5.6 MB/s | 3.7 MB 00:00 2021-02-10T04:16:50.117 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : libcephfs2-2:17.0.0-681.gc1ea6241.el8.x86_64 6/26 2021-02-10T04:16:50.471 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: libcephfs2-2:17.0.0-681.gc1ea6241.el8.x86_64 6/26 2021-02-10T04:16:50.683 INFO:teuthology.orchestra.run.smithi085.stdout: Upgrading : librbd1-2:17.0.0-681.gc1ea6241.el8.x86_64 7/26 2021-02-10T04:16:50.762 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: librbd1-2:17.0.0-681.gc1ea6241.el8.x86_64 7/26 2021-02-10T04:16:50.916 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/26 2021-02-10T04:16:51.006 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/26 2021-02-10T04:16:51.517 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : librabbitmq-0.9.0-2.el8.x86_64 9/26 2021-02-10T04:16:51.725 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : librgw2-2:17.0.0-681.gc1ea6241.el8.x86_64 10/26 2021-02-10T04:16:51.791 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: librgw2-2:17.0.0-681.gc1ea6241.el8.x86_64 10/26 2021-02-10T04:16:51.830 INFO:teuthology.orchestra.run.smithi164.stdout:(24/24): ceph-common-17.0.0-681.gc1ea6241.el8.x 3.1 MB/s | 24 MB 00:07 2021-02-10T04:16:51.834 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2021-02-10T04:16:51.834 INFO:teuthology.orchestra.run.smithi164.stdout:Total 7.4 MB/s | 57 MB 00:07 2021-02-10T04:16:51.839 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2021-02-10T04:16:51.886 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2021-02-10T04:16:51.887 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2021-02-10T04:16:51.982 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : leveldb-1.22-1.el8.x86_64 11/26 2021-02-10T04:16:52.126 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-ceph-argparse-2:17.0.0-681.gc1ea6241.el8.x 12/26 2021-02-10T04:16:52.311 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2021-02-10T04:16:52.315 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2021-02-10T04:16:52.327 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-cephfs-2:17.0.0-681.gc1ea6241.el8.x86_64 13/26 2021-02-10T04:16:52.525 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-rgw-2:17.0.0-681.gc1ea6241.el8.x86_64 14/26 2021-02-10T04:16:52.760 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-rbd-2:17.0.0-681.gc1ea6241.el8.x86_64 15/26 2021-02-10T04:16:52.933 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : libradosstriper1-2:17.0.0-681.gc1ea6241.el8.x86_64 16/26 2021-02-10T04:16:53.002 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: libradosstriper1-2:17.0.0-681.gc1ea6241.el8.x86_64 16/26 2021-02-10T04:16:53.193 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 17/26 2021-02-10T04:16:53.360 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2021-02-10T04:16:53.420 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 18/26 2021-02-10T04:16:53.605 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/26 2021-02-10T04:16:53.635 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : gperftools-libs-2.7-6.el8.x86_64 19/26 2021-02-10T04:16:53.765 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/26 2021-02-10T04:16:53.808 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-ceph-common-2:17.0.0-681.gc1ea6241.el8.x86 20/26 2021-02-10T04:16:54.156 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/26 2021-02-10T04:16:54.357 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : librados2-2:17.0.0-681.gc1ea6241.el8.x86_64 3/26 2021-02-10T04:16:54.462 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados2-2:17.0.0-681.gc1ea6241.el8.x86_64 3/26 2021-02-10T04:16:54.680 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rados-2:17.0.0-681.gc1ea6241.el8.x86_64 4/26 2021-02-10T04:16:54.905 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/26 2021-02-10T04:16:55.457 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephfs2-2:17.0.0-681.gc1ea6241.el8.x86_64 6/26 2021-02-10T04:16:55.816 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libcephfs2-2:17.0.0-681.gc1ea6241.el8.x86_64 6/26 2021-02-10T04:16:56.015 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : librbd1-2:17.0.0-681.gc1ea6241.el8.x86_64 7/26 2021-02-10T04:16:56.115 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librbd1-2:17.0.0-681.gc1ea6241.el8.x86_64 7/26 2021-02-10T04:16:56.290 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/26 2021-02-10T04:16:56.395 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/26 2021-02-10T04:16:56.905 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librabbitmq-0.9.0-2.el8.x86_64 9/26 2021-02-10T04:16:57.081 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librgw2-2:17.0.0-681.gc1ea6241.el8.x86_64 10/26 2021-02-10T04:16:57.151 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librgw2-2:17.0.0-681.gc1ea6241.el8.x86_64 10/26 2021-02-10T04:16:57.329 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : leveldb-1.22-1.el8.x86_64 11/26 2021-02-10T04:16:57.449 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ceph-argparse-2:17.0.0-681.gc1ea6241.el8.x 12/26 2021-02-10T04:16:57.642 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cephfs-2:17.0.0-681.gc1ea6241.el8.x86_64 13/26 2021-02-10T04:16:57.840 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rgw-2:17.0.0-681.gc1ea6241.el8.x86_64 14/26 2021-02-10T04:16:57.898 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: ceph-common-2:17.0.0-681.gc1ea6241.el8.x86_64 21/26 2021-02-10T04:16:58.099 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : ceph-common-2:17.0.0-681.gc1ea6241.el8.x86_64 21/26 2021-02-10T04:16:58.100 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rbd-2:17.0.0-681.gc1ea6241.el8.x86_64 15/26 2021-02-10T04:16:58.248 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libradosstriper1-2:17.0.0-681.gc1ea6241.el8.x86_64 16/26 2021-02-10T04:16:58.304 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libradosstriper1-2:17.0.0-681.gc1ea6241.el8.x86_64 16/26 2021-02-10T04:16:58.474 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 17/26 2021-02-10T04:16:58.693 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 18/26 2021-02-10T04:16:58.817 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: ceph-common-2:17.0.0-681.gc1ea6241.el8.x86_64 21/26 2021-02-10T04:16:58.932 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : gperftools-libs-2.7-6.el8.x86_64 19/26 2021-02-10T04:16:59.081 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ceph-common-2:17.0.0-681.gc1ea6241.el8.x86 20/26 2021-02-10T04:16:59.118 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : ceph-base-2:17.0.0-681.gc1ea6241.el8.x86_64 22/26 2021-02-10T04:16:59.208 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: ceph-base-2:17.0.0-681.gc1ea6241.el8.x86_64 22/26 2021-02-10T04:16:59.332 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : ceph-selinux-2:17.0.0-681.gc1ea6241.el8.x86_64 23/26 2021-02-10T04:17:03.158 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-common-2:17.0.0-681.gc1ea6241.el8.x86_64 21/26 2021-02-10T04:17:03.356 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-common-2:17.0.0-681.gc1ea6241.el8.x86_64 21/26 2021-02-10T04:17:04.105 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-common-2:17.0.0-681.gc1ea6241.el8.x86_64 21/26 2021-02-10T04:17:04.334 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-base-2:17.0.0-681.gc1ea6241.el8.x86_64 22/26 2021-02-10T04:17:04.408 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-base-2:17.0.0-681.gc1ea6241.el8.x86_64 22/26 2021-02-10T04:17:04.539 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-selinux-2:17.0.0-681.gc1ea6241.el8.x86_64 23/26 2021-02-10T04:17:16.029 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: ceph-selinux-2:17.0.0-681.gc1ea6241.el8.x86_64 23/26 2021-02-10T04:17:16.029 INFO:teuthology.orchestra.run.smithi085.stdout:skipping the directory /sys 2021-02-10T04:17:16.030 INFO:teuthology.orchestra.run.smithi085.stdout:skipping the directory /proc 2021-02-10T04:17:16.030 INFO:teuthology.orchestra.run.smithi085.stdout:skipping the directory /dev 2021-02-10T04:17:16.030 INFO:teuthology.orchestra.run.smithi085.stdout:skipping the directory /run 2021-02-10T04:17:16.030 INFO:teuthology.orchestra.run.smithi085.stdout:skipping the directory /mnt 2021-02-10T04:17:16.030 INFO:teuthology.orchestra.run.smithi085.stdout:skipping the directory /var/tmp 2021-02-10T04:17:16.031 INFO:teuthology.orchestra.run.smithi085.stdout:skipping the directory /home 2021-02-10T04:17:16.031 INFO:teuthology.orchestra.run.smithi085.stdout:skipping the directory /tmp 2021-02-10T04:17:16.031 INFO:teuthology.orchestra.run.smithi085.stdout:skipping the directory /dev 2021-02-10T04:17:16.031 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:17:17.332 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : ceph-radosgw-2:17.0.0-681.gc1ea6241.el8.x86_64 24/26 2021-02-10T04:17:17.430 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-681.gc1ea6241.el8.x86_64 24/26 2021-02-10T04:17:17.430 INFO:teuthology.orchestra.run.smithi085.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 25/26 2021-02-10T04:17:17.614 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 25/26 2021-02-10T04:17:17.614 INFO:teuthology.orchestra.run.smithi085.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 26/26 2021-02-10T04:17:18.345 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 26/26 2021-02-10T04:17:18.345 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : ceph-base-2:17.0.0-681.gc1ea6241.el8.x86_64 1/26 2021-02-10T04:17:18.345 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : ceph-common-2:17.0.0-681.gc1ea6241.el8.x86_64 2/26 2021-02-10T04:17:18.345 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : ceph-radosgw-2:17.0.0-681.gc1ea6241.el8.x86_64 3/26 2021-02-10T04:17:18.345 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : ceph-selinux-2:17.0.0-681.gc1ea6241.el8.x86_64 4/26 2021-02-10T04:17:18.345 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : libcephfs2-2:17.0.0-681.gc1ea6241.el8.x86_64 5/26 2021-02-10T04:17:18.345 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : libradosstriper1-2:17.0.0-681.gc1ea6241.el8.x86_64 6/26 2021-02-10T04:17:18.345 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : librgw2-2:17.0.0-681.gc1ea6241.el8.x86_64 7/26 2021-02-10T04:17:18.346 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-ceph-argparse-2:17.0.0-681.gc1ea6241.el8.x 8/26 2021-02-10T04:17:18.346 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-ceph-common-2:17.0.0-681.gc1ea6241.el8.x86 9/26 2021-02-10T04:17:18.346 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-cephfs-2:17.0.0-681.gc1ea6241.el8.x86_64 10/26 2021-02-10T04:17:18.346 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-rados-2:17.0.0-681.gc1ea6241.el8.x86_64 11/26 2021-02-10T04:17:18.346 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-rbd-2:17.0.0-681.gc1ea6241.el8.x86_64 12/26 2021-02-10T04:17:18.346 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-rgw-2:17.0.0-681.gc1ea6241.el8.x86_64 13/26 2021-02-10T04:17:18.346 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 14/26 2021-02-10T04:17:18.346 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : gperftools-libs-2.7-6.el8.x86_64 15/26 2021-02-10T04:17:18.346 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : leveldb-1.22-1.el8.x86_64 16/26 2021-02-10T04:17:18.346 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 17/26 2021-02-10T04:17:18.346 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 18/26 2021-02-10T04:17:18.347 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : librabbitmq-0.9.0-2.el8.x86_64 19/26 2021-02-10T04:17:18.347 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 20/26 2021-02-10T04:17:18.347 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 21/26 2021-02-10T04:17:18.347 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 22/26 2021-02-10T04:17:18.347 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : librados2-2:17.0.0-681.gc1ea6241.el8.x86_64 23/26 2021-02-10T04:17:18.347 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 24/26 2021-02-10T04:17:18.347 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : librbd1-2:17.0.0-681.gc1ea6241.el8.x86_64 25/26 2021-02-10T04:17:19.297 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 26/26 2021-02-10T04:17:19.297 INFO:teuthology.orchestra.run.smithi085.stdout:Installed products updated. 2021-02-10T04:17:21.084 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:17:21.084 INFO:teuthology.orchestra.run.smithi085.stdout:Upgraded: 2021-02-10T04:17:21.084 INFO:teuthology.orchestra.run.smithi085.stdout: librados2-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:21.084 INFO:teuthology.orchestra.run.smithi085.stdout: librbd1-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:21.084 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:17:21.084 INFO:teuthology.orchestra.run.smithi085.stdout:Installed: 2021-02-10T04:17:21.084 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-base-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:21.084 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-common-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:21.085 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-radosgw-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:21.085 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-selinux-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:21.085 INFO:teuthology.orchestra.run.smithi085.stdout: fmt-6.2.1-1.el8.x86_64 2021-02-10T04:17:21.085 INFO:teuthology.orchestra.run.smithi085.stdout: gperftools-libs-2.7-6.el8.x86_64 2021-02-10T04:17:21.085 INFO:teuthology.orchestra.run.smithi085.stdout: leveldb-1.22-1.el8.x86_64 2021-02-10T04:17:21.085 INFO:teuthology.orchestra.run.smithi085.stdout: libcephfs2-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:21.085 INFO:teuthology.orchestra.run.smithi085.stdout: liboath-2.6.2-3.el8.x86_64 2021-02-10T04:17:21.085 INFO:teuthology.orchestra.run.smithi085.stdout: librabbitmq-0.9.0-2.el8.x86_64 2021-02-10T04:17:21.085 INFO:teuthology.orchestra.run.smithi085.stdout: libradosstriper1-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:21.085 INFO:teuthology.orchestra.run.smithi085.stdout: librdkafka-0.11.4-1.el8.x86_64 2021-02-10T04:17:21.085 INFO:teuthology.orchestra.run.smithi085.stdout: librgw2-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:21.085 INFO:teuthology.orchestra.run.smithi085.stdout: libunwind-1.3.1-3.el8.x86_64 2021-02-10T04:17:21.086 INFO:teuthology.orchestra.run.smithi085.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-02-10T04:17:21.086 INFO:teuthology.orchestra.run.smithi085.stdout: python3-ceph-argparse-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:21.086 INFO:teuthology.orchestra.run.smithi085.stdout: python3-ceph-common-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:21.086 INFO:teuthology.orchestra.run.smithi085.stdout: python3-cephfs-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:21.086 INFO:teuthology.orchestra.run.smithi085.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-02-10T04:17:21.086 INFO:teuthology.orchestra.run.smithi085.stdout: python3-rados-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:21.086 INFO:teuthology.orchestra.run.smithi085.stdout: python3-rbd-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:21.086 INFO:teuthology.orchestra.run.smithi085.stdout: python3-rgw-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:21.086 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:17:21.086 INFO:teuthology.orchestra.run.smithi085.stdout:Complete! 2021-02-10T04:17:21.218 DEBUG:teuthology.orchestra.run.smithi085:> sudo yum -y install ceph-test 2021-02-10T04:17:21.493 INFO:teuthology.orchestra.run.smithi085.stdout:Updating Subscription Management repositories. 2021-02-10T04:17:21.976 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 2021-02-10T04:17:22.045 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-selinux-2:17.0.0-681.gc1ea6241.el8.x86_64 23/26 2021-02-10T04:17:22.045 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /sys 2021-02-10T04:17:22.045 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /proc 2021-02-10T04:17:22.045 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /dev 2021-02-10T04:17:22.045 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /run 2021-02-10T04:17:22.045 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /mnt 2021-02-10T04:17:22.045 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /var/tmp 2021-02-10T04:17:22.046 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /home 2021-02-10T04:17:22.046 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /tmp 2021-02-10T04:17:22.046 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /dev 2021-02-10T04:17:22.046 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:17:22.087 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 95 kB/s | 2.8 kB 00:00 2021-02-10T04:17:23.389 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-radosgw-2:17.0.0-681.gc1ea6241.el8.x86_64 24/26 2021-02-10T04:17:23.456 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-681.gc1ea6241.el8.x86_64 24/26 2021-02-10T04:17:23.457 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 25/26 2021-02-10T04:17:23.565 INFO:teuthology.orchestra.run.smithi085.stdout:Dependencies resolved. 2021-02-10T04:17:23.566 INFO:teuthology.orchestra.run.smithi085.stdout:======================================================================================== 2021-02-10T04:17:23.566 INFO:teuthology.orchestra.run.smithi085.stdout: Package Arch Version Repository Size 2021-02-10T04:17:23.567 INFO:teuthology.orchestra.run.smithi085.stdout:======================================================================================== 2021-02-10T04:17:23.567 INFO:teuthology.orchestra.run.smithi085.stdout:Installing: 2021-02-10T04:17:23.567 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-test x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 52 M 2021-02-10T04:17:23.567 INFO:teuthology.orchestra.run.smithi085.stdout:Installing dependencies: 2021-02-10T04:17:23.567 INFO:teuthology.orchestra.run.smithi085.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2021-02-10T04:17:23.567 INFO:teuthology.orchestra.run.smithi085.stdout: libxslt x86_64 1.1.32-5.el8 rhel-8-for-x86_64-baseos-rpms 250 k 2021-02-10T04:17:23.567 INFO:teuthology.orchestra.run.smithi085.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2021-02-10T04:17:23.567 INFO:teuthology.orchestra.run.smithi085.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2021-02-10T04:17:23.567 INFO:teuthology.orchestra.run.smithi085.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-02-10T04:17:23.567 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:17:23.567 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction Summary 2021-02-10T04:17:23.567 INFO:teuthology.orchestra.run.smithi085.stdout:======================================================================================== 2021-02-10T04:17:23.568 INFO:teuthology.orchestra.run.smithi085.stdout:Install 6 Packages 2021-02-10T04:17:23.568 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:17:23.568 INFO:teuthology.orchestra.run.smithi085.stdout:Total download size: 53 M 2021-02-10T04:17:23.568 INFO:teuthology.orchestra.run.smithi085.stdout:Installed size: 228 M 2021-02-10T04:17:23.568 INFO:teuthology.orchestra.run.smithi085.stdout:Downloading Packages: 2021-02-10T04:17:23.601 INFO:teuthology.orchestra.run.smithi085.stdout:(1/6): libxslt-1.1.32-5.el8.x86_64.rpm 7.7 MB/s | 250 kB 00:00 2021-02-10T04:17:23.621 INFO:teuthology.orchestra.run.smithi085.stdout:(2/6): jq-1.5-12.el8.x86_64.rpm 7.8 MB/s | 161 kB 00:00 2021-02-10T04:17:23.641 INFO:teuthology.orchestra.run.smithi085.stdout:(3/6): socat-1.7.3.3-2.el8.x86_64.rpm 15 MB/s | 302 kB 00:00 2021-02-10T04:17:23.655 INFO:teuthology.orchestra.run.smithi085.stdout:(4/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 798 kB/s | 69 kB 00:00 2021-02-10T04:17:23.662 INFO:teuthology.orchestra.run.smithi085.stdout:(5/6): oniguruma-6.8.2-2.el8.x86_64.rpm 9.0 MB/s | 187 kB 00:00 2021-02-10T04:17:23.662 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 25/26 2021-02-10T04:17:23.662 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 26/26 2021-02-10T04:17:24.405 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 26/26 2021-02-10T04:17:24.405 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-base-2:17.0.0-681.gc1ea6241.el8.x86_64 1/26 2021-02-10T04:17:24.405 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-common-2:17.0.0-681.gc1ea6241.el8.x86_64 2/26 2021-02-10T04:17:24.405 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-radosgw-2:17.0.0-681.gc1ea6241.el8.x86_64 3/26 2021-02-10T04:17:24.405 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-selinux-2:17.0.0-681.gc1ea6241.el8.x86_64 4/26 2021-02-10T04:17:24.405 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephfs2-2:17.0.0-681.gc1ea6241.el8.x86_64 5/26 2021-02-10T04:17:24.406 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libradosstriper1-2:17.0.0-681.gc1ea6241.el8.x86_64 6/26 2021-02-10T04:17:24.406 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librgw2-2:17.0.0-681.gc1ea6241.el8.x86_64 7/26 2021-02-10T04:17:24.406 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ceph-argparse-2:17.0.0-681.gc1ea6241.el8.x 8/26 2021-02-10T04:17:24.406 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ceph-common-2:17.0.0-681.gc1ea6241.el8.x86 9/26 2021-02-10T04:17:24.406 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cephfs-2:17.0.0-681.gc1ea6241.el8.x86_64 10/26 2021-02-10T04:17:24.406 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rados-2:17.0.0-681.gc1ea6241.el8.x86_64 11/26 2021-02-10T04:17:24.406 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rbd-2:17.0.0-681.gc1ea6241.el8.x86_64 12/26 2021-02-10T04:17:24.407 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rgw-2:17.0.0-681.gc1ea6241.el8.x86_64 13/26 2021-02-10T04:17:24.407 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 14/26 2021-02-10T04:17:24.407 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : gperftools-libs-2.7-6.el8.x86_64 15/26 2021-02-10T04:17:24.407 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : leveldb-1.22-1.el8.x86_64 16/26 2021-02-10T04:17:24.407 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 17/26 2021-02-10T04:17:24.407 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 18/26 2021-02-10T04:17:24.407 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librabbitmq-0.9.0-2.el8.x86_64 19/26 2021-02-10T04:17:24.407 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 20/26 2021-02-10T04:17:24.408 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 21/26 2021-02-10T04:17:24.408 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 22/26 2021-02-10T04:17:24.408 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados2-2:17.0.0-681.gc1ea6241.el8.x86_64 23/26 2021-02-10T04:17:24.408 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 24/26 2021-02-10T04:17:24.408 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librbd1-2:17.0.0-681.gc1ea6241.el8.x86_64 25/26 2021-02-10T04:17:25.310 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 26/26 2021-02-10T04:17:25.310 INFO:teuthology.orchestra.run.smithi164.stdout:Installed products updated. 2021-02-10T04:17:25.989 INFO:teuthology.orchestra.run.smithi085.stdout:(6/6): ceph-test-17.0.0-681.gc1ea6241.el8.x86_6 21 MB/s | 52 MB 00:02 2021-02-10T04:17:25.990 INFO:teuthology.orchestra.run.smithi085.stdout:-------------------------------------------------------------------------------- 2021-02-10T04:17:26.001 INFO:teuthology.orchestra.run.smithi085.stdout:Total 22 MB/s | 53 MB 00:02 2021-02-10T04:17:26.001 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction check 2021-02-10T04:17:26.018 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction check succeeded. 2021-02-10T04:17:26.018 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction test 2021-02-10T04:17:26.292 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction test succeeded. 2021-02-10T04:17:26.296 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction 2021-02-10T04:17:26.772 INFO:teuthology.orchestra.run.smithi085.stdout: Preparing : 1/1 2021-02-10T04:17:26.978 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 1/6 2021-02-10T04:17:27.053 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 1/6 2021-02-10T04:17:27.180 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:17:27.181 INFO:teuthology.orchestra.run.smithi164.stdout:Upgraded: 2021-02-10T04:17:27.181 INFO:teuthology.orchestra.run.smithi164.stdout: librados2-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:27.181 INFO:teuthology.orchestra.run.smithi164.stdout: librbd1-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:27.181 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:17:27.181 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2021-02-10T04:17:27.181 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-base-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:27.181 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-common-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:27.181 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-radosgw-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:27.181 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-selinux-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:27.181 INFO:teuthology.orchestra.run.smithi164.stdout: fmt-6.2.1-1.el8.x86_64 2021-02-10T04:17:27.182 INFO:teuthology.orchestra.run.smithi164.stdout: gperftools-libs-2.7-6.el8.x86_64 2021-02-10T04:17:27.182 INFO:teuthology.orchestra.run.smithi164.stdout: leveldb-1.22-1.el8.x86_64 2021-02-10T04:17:27.182 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs2-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:27.182 INFO:teuthology.orchestra.run.smithi164.stdout: liboath-2.6.2-3.el8.x86_64 2021-02-10T04:17:27.182 INFO:teuthology.orchestra.run.smithi164.stdout: librabbitmq-0.9.0-2.el8.x86_64 2021-02-10T04:17:27.182 INFO:teuthology.orchestra.run.smithi164.stdout: libradosstriper1-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:27.182 INFO:teuthology.orchestra.run.smithi164.stdout: librdkafka-0.11.4-1.el8.x86_64 2021-02-10T04:17:27.182 INFO:teuthology.orchestra.run.smithi164.stdout: librgw2-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:27.182 INFO:teuthology.orchestra.run.smithi164.stdout: libunwind-1.3.1-3.el8.x86_64 2021-02-10T04:17:27.182 INFO:teuthology.orchestra.run.smithi164.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-02-10T04:17:27.182 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-argparse-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:27.182 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-common-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:27.183 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cephfs-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:27.183 INFO:teuthology.orchestra.run.smithi164.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-02-10T04:17:27.183 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rados-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:27.183 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rbd-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:27.183 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rgw-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:27.183 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:17:27.183 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2021-02-10T04:17:27.282 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : jq-1.5-12.el8.x86_64 2/6 2021-02-10T04:17:27.470 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 3/6 2021-02-10T04:17:28.490 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-test 2021-02-10T04:17:28.776 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2021-02-10T04:17:29.028 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : libxslt-1.1.32-5.el8.x86_64 4/6 2021-02-10T04:17:29.291 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 84 kB/s | 2.4 kB 00:00 2021-02-10T04:17:29.408 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 78 kB/s | 2.8 kB 00:00 2021-02-10T04:17:30.902 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2021-02-10T04:17:30.902 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================== 2021-02-10T04:17:30.903 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2021-02-10T04:17:30.903 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================== 2021-02-10T04:17:30.903 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2021-02-10T04:17:30.903 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-test x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 52 M 2021-02-10T04:17:30.903 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2021-02-10T04:17:30.903 INFO:teuthology.orchestra.run.smithi164.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2021-02-10T04:17:30.903 INFO:teuthology.orchestra.run.smithi164.stdout: libxslt x86_64 1.1.32-5.el8 rhel-8-for-x86_64-baseos-rpms 250 k 2021-02-10T04:17:30.903 INFO:teuthology.orchestra.run.smithi164.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2021-02-10T04:17:30.903 INFO:teuthology.orchestra.run.smithi164.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2021-02-10T04:17:30.903 INFO:teuthology.orchestra.run.smithi164.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-02-10T04:17:30.903 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:17:30.903 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2021-02-10T04:17:30.903 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================== 2021-02-10T04:17:30.904 INFO:teuthology.orchestra.run.smithi164.stdout:Install 6 Packages 2021-02-10T04:17:30.904 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:17:30.904 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 53 M 2021-02-10T04:17:30.904 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 228 M 2021-02-10T04:17:30.904 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2021-02-10T04:17:30.939 INFO:teuthology.orchestra.run.smithi164.stdout:(1/6): libxslt-1.1.32-5.el8.x86_64.rpm 7.2 MB/s | 250 kB 00:00 2021-02-10T04:17:30.961 INFO:teuthology.orchestra.run.smithi164.stdout:(2/6): jq-1.5-12.el8.x86_64.rpm 7.3 MB/s | 161 kB 00:00 2021-02-10T04:17:30.973 INFO:teuthology.orchestra.run.smithi164.stdout:(3/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 1.0 MB/s | 69 kB 00:00 2021-02-10T04:17:30.987 INFO:teuthology.orchestra.run.smithi164.stdout:(4/6): socat-1.7.3.3-2.el8.x86_64.rpm 11 MB/s | 302 kB 00:00 2021-02-10T04:17:31.006 INFO:teuthology.orchestra.run.smithi164.stdout:(5/6): oniguruma-6.8.2-2.el8.x86_64.rpm 5.6 MB/s | 187 kB 00:00 2021-02-10T04:17:33.829 INFO:teuthology.orchestra.run.smithi164.stdout:(6/6): ceph-test-17.0.0-681.gc1ea6241.el8.x86_6 18 MB/s | 52 MB 00:02 2021-02-10T04:17:33.831 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2021-02-10T04:17:33.832 INFO:teuthology.orchestra.run.smithi164.stdout:Total 18 MB/s | 53 MB 00:02 2021-02-10T04:17:33.836 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2021-02-10T04:17:33.859 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2021-02-10T04:17:33.859 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2021-02-10T04:17:34.196 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 5/6 2021-02-10T04:17:34.202 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2021-02-10T04:17:34.206 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2021-02-10T04:17:34.416 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : ceph-test-2:17.0.0-681.gc1ea6241.el8.x86_64 6/6 2021-02-10T04:17:34.641 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: ceph-test-2:17.0.0-681.gc1ea6241.el8.x86_64 6/6 2021-02-10T04:17:34.641 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : ceph-test-2:17.0.0-681.gc1ea6241.el8.x86_64 1/6 2021-02-10T04:17:34.642 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 2/6 2021-02-10T04:17:34.642 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : libxslt-1.1.32-5.el8.x86_64 3/6 2021-02-10T04:17:34.642 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : jq-1.5-12.el8.x86_64 4/6 2021-02-10T04:17:34.642 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 5/6 2021-02-10T04:17:35.336 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 6/6 2021-02-10T04:17:35.337 INFO:teuthology.orchestra.run.smithi085.stdout:Installed products updated. 2021-02-10T04:17:35.971 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2021-02-10T04:17:36.134 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 1/6 2021-02-10T04:17:36.202 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 1/6 2021-02-10T04:17:36.437 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : jq-1.5-12.el8.x86_64 2/6 2021-02-10T04:17:36.601 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 3/6 2021-02-10T04:17:36.869 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libxslt-1.1.32-5.el8.x86_64 4/6 2021-02-10T04:17:37.168 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:17:37.168 INFO:teuthology.orchestra.run.smithi085.stdout:Installed: 2021-02-10T04:17:37.168 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-test-2:17.0.0-681.gc1ea6241.el8.x86_64 jq-1.5-12.el8.x86_64 2021-02-10T04:17:37.168 INFO:teuthology.orchestra.run.smithi085.stdout: libxslt-1.1.32-5.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2021-02-10T04:17:37.168 INFO:teuthology.orchestra.run.smithi085.stdout: socat-1.7.3.3-2.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2021-02-10T04:17:37.168 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:17:37.168 INFO:teuthology.orchestra.run.smithi085.stdout:Complete! 2021-02-10T04:17:37.259 DEBUG:teuthology.orchestra.run.smithi085:> sudo yum -y install ceph 2021-02-10T04:17:37.527 INFO:teuthology.orchestra.run.smithi085.stdout:Updating Subscription Management repositories. 2021-02-10T04:17:38.011 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 85 kB/s | 2.4 kB 00:00 2021-02-10T04:17:38.123 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 86 kB/s | 2.8 kB 00:00 2021-02-10T04:17:39.618 INFO:teuthology.orchestra.run.smithi085.stdout:Dependencies resolved. 2021-02-10T04:17:39.622 INFO:teuthology.orchestra.run.smithi085.stdout:====================================================================================================== 2021-02-10T04:17:39.622 INFO:teuthology.orchestra.run.smithi085.stdout: Package Arch Version Repository Size 2021-02-10T04:17:39.622 INFO:teuthology.orchestra.run.smithi085.stdout:====================================================================================================== 2021-02-10T04:17:39.622 INFO:teuthology.orchestra.run.smithi085.stdout:Installing: 2021-02-10T04:17:39.622 INFO:teuthology.orchestra.run.smithi085.stdout: ceph x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 6.4 k 2021-02-10T04:17:39.622 INFO:teuthology.orchestra.run.smithi085.stdout:Installing dependencies: 2021-02-10T04:17:39.622 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-mds x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 2.4 M 2021-02-10T04:17:39.622 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-mgr x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 1.5 M 2021-02-10T04:17:39.623 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-mgr-modules-core noarch 2:17.0.0-681.gc1ea6241.el8 ceph-noarch 222 k 2021-02-10T04:17:39.623 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-mon x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 5.0 M 2021-02-10T04:17:39.623 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-osd x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 18 M 2021-02-10T04:17:39.623 INFO:teuthology.orchestra.run.smithi085.stdout: python3-asn1crypto noarch 0.24.0-3.el8 rhel-8-for-x86_64-baseos-rpms 181 k 2021-02-10T04:17:39.623 INFO:teuthology.orchestra.run.smithi085.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-02-10T04:17:39.623 INFO:teuthology.orchestra.run.smithi085.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-02-10T04:17:39.623 INFO:teuthology.orchestra.run.smithi085.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2021-02-10T04:17:39.623 INFO:teuthology.orchestra.run.smithi085.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-02-10T04:17:39.623 INFO:teuthology.orchestra.run.smithi085.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-02-10T04:17:39.623 INFO:teuthology.orchestra.run.smithi085.stdout: python3-cryptography x86_64 2.3-3.el8 rhel-8-for-x86_64-baseos-rpms 511 k 2021-02-10T04:17:39.623 INFO:teuthology.orchestra.run.smithi085.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-02-10T04:17:39.623 INFO:teuthology.orchestra.run.smithi085.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-02-10T04:17:39.624 INFO:teuthology.orchestra.run.smithi085.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-02-10T04:17:39.624 INFO:teuthology.orchestra.run.smithi085.stdout: python3-lxml x86_64 4.2.3-1.el8 rhel-8-for-x86_64-appstream-rpms 1.5 M 2021-02-10T04:17:39.624 INFO:teuthology.orchestra.run.smithi085.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2021-02-10T04:17:39.624 INFO:teuthology.orchestra.run.smithi085.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2021-02-10T04:17:39.624 INFO:teuthology.orchestra.run.smithi085.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-02-10T04:17:39.624 INFO:teuthology.orchestra.run.smithi085.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-02-10T04:17:39.624 INFO:teuthology.orchestra.run.smithi085.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-02-10T04:17:39.624 INFO:teuthology.orchestra.run.smithi085.stdout: python3-pyOpenSSL noarch 18.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2021-02-10T04:17:39.624 INFO:teuthology.orchestra.run.smithi085.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2021-02-10T04:17:39.624 INFO:teuthology.orchestra.run.smithi085.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2021-02-10T04:17:39.624 INFO:teuthology.orchestra.run.smithi085.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-02-10T04:17:39.624 INFO:teuthology.orchestra.run.smithi085.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-02-10T04:17:39.625 INFO:teuthology.orchestra.run.smithi085.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-02-10T04:17:39.625 INFO:teuthology.orchestra.run.smithi085.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-02-10T04:17:39.625 INFO:teuthology.orchestra.run.smithi085.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-02-10T04:17:39.625 INFO:teuthology.orchestra.run.smithi085.stdout: python3-webencodings noarch 0.5.1-6.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2021-02-10T04:17:39.625 INFO:teuthology.orchestra.run.smithi085.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-02-10T04:17:39.625 INFO:teuthology.orchestra.run.smithi085.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-02-10T04:17:39.625 INFO:teuthology.orchestra.run.smithi085.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2021-02-10T04:17:39.625 INFO:teuthology.orchestra.run.smithi085.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-02-10T04:17:39.625 INFO:teuthology.orchestra.run.smithi085.stdout:Installing weak dependencies: 2021-02-10T04:17:39.625 INFO:teuthology.orchestra.run.smithi085.stdout: python3-cssselect noarch 0.9.2-10.el8 epel 40 k 2021-02-10T04:17:39.625 INFO:teuthology.orchestra.run.smithi085.stdout: python3-html5lib noarch 1:0.999999999-6.el8 rhel-8-for-x86_64-appstream-rpms 214 k 2021-02-10T04:17:39.625 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:17:39.625 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction Summary 2021-02-10T04:17:39.626 INFO:teuthology.orchestra.run.smithi085.stdout:====================================================================================================== 2021-02-10T04:17:39.626 INFO:teuthology.orchestra.run.smithi085.stdout:Install 37 Packages 2021-02-10T04:17:39.626 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:17:39.626 INFO:teuthology.orchestra.run.smithi085.stdout:Total download size: 33 M 2021-02-10T04:17:39.626 INFO:teuthology.orchestra.run.smithi085.stdout:Installed size: 117 M 2021-02-10T04:17:39.626 INFO:teuthology.orchestra.run.smithi085.stdout:Downloading Packages: 2021-02-10T04:17:39.954 INFO:teuthology.orchestra.run.smithi085.stdout:(1/37): ceph-17.0.0-681.gc1ea6241.el8.x86_64.rp 20 kB/s | 6.4 kB 00:00 2021-02-10T04:17:40.460 INFO:teuthology.orchestra.run.smithi085.stdout:(2/37): ceph-mgr-17.0.0-681.gc1ea6241.el8.x86_6 1.8 MB/s | 1.5 MB 00:00 2021-02-10T04:17:40.515 INFO:teuthology.orchestra.run.smithi085.stdout:(3/37): ceph-mds-17.0.0-681.gc1ea6241.el8.x86_6 2.7 MB/s | 2.4 MB 00:00 2021-02-10T04:17:40.616 INFO:teuthology.orchestra.run.smithi085.stdout:(4/37): ceph-mgr-modules-core-17.0.0-681.gc1ea6 2.1 MB/s | 222 kB 00:00 2021-02-10T04:17:40.683 INFO:teuthology.orchestra.run.smithi085.stdout:(5/37): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 661 kB/s | 44 kB 00:00 2021-02-10T04:17:40.731 INFO:teuthology.orchestra.run.smithi085.stdout:(6/37): python3-beautifulsoup4-4.6.3-2.el8.1.no 3.7 MB/s | 185 kB 00:00 2021-02-10T04:17:40.749 INFO:teuthology.orchestra.run.smithi085.stdout:(7/37): python3-cheroot-8.5.2-1.el8.noarch.rpm 9.6 MB/s | 173 kB 00:00 2021-02-10T04:17:40.781 INFO:teuthology.orchestra.run.smithi085.stdout:(8/37): python3-cherrypy-18.4.0-1.el8.noarch.rp 11 MB/s | 384 kB 00:00 2021-02-10T04:17:40.798 INFO:teuthology.orchestra.run.smithi085.stdout:(9/37): python3-cssselect-0.9.2-10.el8.noarch.r 2.4 MB/s | 40 kB 00:00 2021-02-10T04:17:40.814 INFO:teuthology.orchestra.run.smithi085.stdout:(10/37): python3-jaraco-6.2-6.el8.noarch.rpm 690 kB/s | 11 kB 00:00 2021-02-10T04:17:40.838 INFO:teuthology.orchestra.run.smithi085.stdout:(11/37): ceph-mon-17.0.0-681.gc1ea6241.el8.x86_ 5.6 MB/s | 5.0 MB 00:00 2021-02-10T04:17:40.838 INFO:teuthology.orchestra.run.smithi085.stdout:(12/37): python3-jaraco-functools-2.0-4.el8.noa 740 kB/s | 18 kB 00:00 2021-02-10T04:17:40.855 INFO:teuthology.orchestra.run.smithi085.stdout:(13/37): python3-more-itertools-7.2.0-3.el8.noa 3.5 MB/s | 59 kB 00:00 2021-02-10T04:17:40.873 INFO:teuthology.orchestra.run.smithi085.stdout:(14/37): python3-pecan-1.3.2-9.el8.noarch.rpm 15 MB/s | 283 kB 00:00 2021-02-10T04:17:40.889 INFO:teuthology.orchestra.run.smithi085.stdout:(15/37): python3-portend-2.6-1.el8.noarch.rpm 964 kB/s | 16 kB 00:00 2021-02-10T04:17:40.902 INFO:teuthology.orchestra.run.smithi085.stdout:(16/37): python3-logutils-0.3.5-11.el8.noarch.r 759 kB/s | 49 kB 00:00 2021-02-10T04:17:40.906 INFO:teuthology.orchestra.run.smithi085.stdout:(17/37): python3-simplegeneric-0.8.1-17.el8.noa 1.1 MB/s | 19 kB 00:00 2021-02-10T04:17:40.918 INFO:teuthology.orchestra.run.smithi085.stdout:(18/37): python3-singledispatch-3.4.0.3-18.el8. 1.4 MB/s | 24 kB 00:00 2021-02-10T04:17:40.922 INFO:teuthology.orchestra.run.smithi085.stdout:(19/37): python3-tempora-1.14.1-5.el8.noarch.rp 1.7 MB/s | 29 kB 00:00 2021-02-10T04:17:40.935 INFO:teuthology.orchestra.run.smithi085.stdout:(20/37): python3-trustme-0.6.0-4.el8.noarch.rpm 1.6 MB/s | 27 kB 00:00 2021-02-10T04:17:40.941 INFO:teuthology.orchestra.run.smithi085.stdout:(21/37): python3-waitress-1.2.1-2.el8.1.noarch. 13 MB/s | 241 kB 00:00 2021-02-10T04:17:40.958 INFO:teuthology.orchestra.run.smithi085.stdout:(22/37): python3-webtest-2.0.33-1.el8.noarch.rp 5.0 MB/s | 85 kB 00:00 2021-02-10T04:17:40.974 INFO:teuthology.orchestra.run.smithi085.stdout:(23/37): python3-zc-lockfile-2.0-2.el8.noarch.r 1.4 MB/s | 23 kB 00:00 2021-02-10T04:17:40.985 INFO:teuthology.orchestra.run.smithi085.stdout:(24/37): python3-webob-1.8.5-1.el8.1.noarch.rpm 5.0 MB/s | 251 kB 00:00 2021-02-10T04:17:41.010 INFO:teuthology.orchestra.run.smithi085.stdout:(25/37): python3-cffi-1.11.5-5.el8.x86_64.rpm 6.5 MB/s | 238 kB 00:00 2021-02-10T04:17:41.023 INFO:teuthology.orchestra.run.smithi085.stdout:(26/37): python3-asn1crypto-0.24.0-3.el8.noarch 4.6 MB/s | 181 kB 00:00 2021-02-10T04:17:41.035 INFO:teuthology.orchestra.run.smithi085.stdout:(27/37): python3-pycparser-2.14-14.el8.noarch.r 5.3 MB/s | 109 kB 00:00 2021-02-10T04:17:41.055 INFO:teuthology.orchestra.run.smithi085.stdout:(28/37): python3-cryptography-2.3-3.el8.x86_64. 18 MB/s | 511 kB 00:00 2021-02-10T04:17:41.061 INFO:teuthology.orchestra.run.smithi085.stdout:(29/37): python3-webencodings-0.5.1-6.el8.noarc 1.0 MB/s | 27 kB 00:00 2021-02-10T04:17:41.088 INFO:teuthology.orchestra.run.smithi085.stdout:(30/37): python3-lxml-4.2.3-1.el8.x86_64.rpm 45 MB/s | 1.5 MB 00:00 2021-02-10T04:17:41.089 INFO:teuthology.orchestra.run.smithi085.stdout:(31/37): python3-pyOpenSSL-18.0.0-1.el8.noarch. 3.7 MB/s | 103 kB 00:00 2021-02-10T04:17:41.114 INFO:teuthology.orchestra.run.smithi085.stdout:(32/37): python3-mako-1.0.6-13.el8.noarch.rpm 6.2 MB/s | 157 kB 00:00 2021-02-10T04:17:41.124 INFO:teuthology.orchestra.run.smithi085.stdout:(33/37): python3-werkzeug-0.12.2-4.el8.noarch.r 13 MB/s | 457 kB 00:00 2021-02-10T04:17:41.136 INFO:teuthology.orchestra.run.smithi085.stdout:(34/37): python3-html5lib-0.999999999-6.el8.noa 9.6 MB/s | 214 kB 00:00 2021-02-10T04:17:41.147 INFO:teuthology.orchestra.run.smithi085.stdout:(35/37): python3-pytz-2017.2-9.el8.noarch.rpm 2.8 MB/s | 54 kB 00:00 2021-02-10T04:17:41.161 INFO:teuthology.orchestra.run.smithi085.stdout:(36/37): python3-markupsafe-0.23-19.el8.x86_64. 1.5 MB/s | 39 kB 00:00 2021-02-10T04:17:41.781 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 5/6 2021-02-10T04:17:41.821 INFO:teuthology.orchestra.run.smithi085.stdout:(37/37): ceph-osd-17.0.0-681.gc1ea6241.el8.x86_ 13 MB/s | 18 MB 00:01 2021-02-10T04:17:41.824 INFO:teuthology.orchestra.run.smithi085.stdout:-------------------------------------------------------------------------------- 2021-02-10T04:17:41.824 INFO:teuthology.orchestra.run.smithi085.stdout:Total 15 MB/s | 33 MB 00:02 2021-02-10T04:17:41.835 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction check 2021-02-10T04:17:41.874 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction check succeeded. 2021-02-10T04:17:41.875 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction test 2021-02-10T04:17:42.030 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-test-2:17.0.0-681.gc1ea6241.el8.x86_64 6/6 2021-02-10T04:17:42.145 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction test succeeded. 2021-02-10T04:17:42.149 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction 2021-02-10T04:17:42.242 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-test-2:17.0.0-681.gc1ea6241.el8.x86_64 6/6 2021-02-10T04:17:42.242 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-test-2:17.0.0-681.gc1ea6241.el8.x86_64 1/6 2021-02-10T04:17:42.242 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 2/6 2021-02-10T04:17:42.242 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libxslt-1.1.32-5.el8.x86_64 3/6 2021-02-10T04:17:42.242 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : jq-1.5-12.el8.x86_64 4/6 2021-02-10T04:17:42.242 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 5/6 2021-02-10T04:17:42.775 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 6/6 2021-02-10T04:17:42.775 INFO:teuthology.orchestra.run.smithi164.stdout:Installed products updated. 2021-02-10T04:17:43.546 INFO:teuthology.orchestra.run.smithi085.stdout: Preparing : 1/1 2021-02-10T04:17:43.794 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/37 2021-02-10T04:17:43.997 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/37 2021-02-10T04:17:44.207 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 3/37 2021-02-10T04:17:44.435 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 4/37 2021-02-10T04:17:44.594 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:17:44.594 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2021-02-10T04:17:44.594 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-test-2:17.0.0-681.gc1ea6241.el8.x86_64 jq-1.5-12.el8.x86_64 2021-02-10T04:17:44.595 INFO:teuthology.orchestra.run.smithi164.stdout: libxslt-1.1.32-5.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2021-02-10T04:17:44.595 INFO:teuthology.orchestra.run.smithi164.stdout: socat-1.7.3.3-2.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2021-02-10T04:17:44.595 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:17:44.595 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2021-02-10T04:17:44.690 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph 2021-02-10T04:17:44.713 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 5/37 2021-02-10T04:17:44.959 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2021-02-10T04:17:45.195 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/37 2021-02-10T04:17:45.440 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 79 kB/s | 2.4 kB 00:00 2021-02-10T04:17:45.507 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 7/37 2021-02-10T04:17:45.550 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 97 kB/s | 2.8 kB 00:00 2021-02-10T04:17:45.756 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 8/37 2021-02-10T04:17:46.013 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 9/37 2021-02-10T04:17:46.226 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 10/37 2021-02-10T04:17:46.427 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 11/37 2021-02-10T04:17:46.790 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-asn1crypto-0.24.0-3.el8.noarch 12/37 2021-02-10T04:17:47.044 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2021-02-10T04:17:47.048 INFO:teuthology.orchestra.run.smithi164.stdout:====================================================================================================== 2021-02-10T04:17:47.048 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2021-02-10T04:17:47.048 INFO:teuthology.orchestra.run.smithi164.stdout:====================================================================================================== 2021-02-10T04:17:47.048 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2021-02-10T04:17:47.048 INFO:teuthology.orchestra.run.smithi164.stdout: ceph x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 6.4 k 2021-02-10T04:17:47.049 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2021-02-10T04:17:47.049 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mds x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 2.4 M 2021-02-10T04:17:47.049 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 1.5 M 2021-02-10T04:17:47.049 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-modules-core noarch 2:17.0.0-681.gc1ea6241.el8 ceph-noarch 222 k 2021-02-10T04:17:47.049 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mon x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 5.0 M 2021-02-10T04:17:47.049 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-osd x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 18 M 2021-02-10T04:17:47.050 INFO:teuthology.orchestra.run.smithi164.stdout: python3-asn1crypto noarch 0.24.0-3.el8 rhel-8-for-x86_64-baseos-rpms 181 k 2021-02-10T04:17:47.050 INFO:teuthology.orchestra.run.smithi164.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-02-10T04:17:47.050 INFO:teuthology.orchestra.run.smithi164.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-02-10T04:17:47.050 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2021-02-10T04:17:47.050 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-02-10T04:17:47.050 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-02-10T04:17:47.051 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cryptography x86_64 2.3-3.el8 rhel-8-for-x86_64-baseos-rpms 511 k 2021-02-10T04:17:47.051 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-02-10T04:17:47.051 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-02-10T04:17:47.051 INFO:teuthology.orchestra.run.smithi164.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-02-10T04:17:47.051 INFO:teuthology.orchestra.run.smithi164.stdout: python3-lxml x86_64 4.2.3-1.el8 rhel-8-for-x86_64-appstream-rpms 1.5 M 2021-02-10T04:17:47.051 INFO:teuthology.orchestra.run.smithi164.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2021-02-10T04:17:47.052 INFO:teuthology.orchestra.run.smithi164.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2021-02-10T04:17:47.052 INFO:teuthology.orchestra.run.smithi164.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-02-10T04:17:47.052 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-02-10T04:17:47.052 INFO:teuthology.orchestra.run.smithi164.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-02-10T04:17:47.052 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyOpenSSL noarch 18.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2021-02-10T04:17:47.052 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2021-02-10T04:17:47.053 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2021-02-10T04:17:47.053 INFO:teuthology.orchestra.run.smithi164.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-02-10T04:17:47.053 INFO:teuthology.orchestra.run.smithi164.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-02-10T04:17:47.053 INFO:teuthology.orchestra.run.smithi164.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-02-10T04:17:47.053 INFO:teuthology.orchestra.run.smithi164.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-02-10T04:17:47.053 INFO:teuthology.orchestra.run.smithi164.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-02-10T04:17:47.054 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webencodings noarch 0.5.1-6.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2021-02-10T04:17:47.054 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-02-10T04:17:47.054 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-02-10T04:17:47.054 INFO:teuthology.orchestra.run.smithi164.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2021-02-10T04:17:47.054 INFO:teuthology.orchestra.run.smithi164.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-02-10T04:17:47.054 INFO:teuthology.orchestra.run.smithi164.stdout:Installing weak dependencies: 2021-02-10T04:17:47.055 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cssselect noarch 0.9.2-10.el8 epel 40 k 2021-02-10T04:17:47.055 INFO:teuthology.orchestra.run.smithi164.stdout: python3-html5lib noarch 1:0.999999999-6.el8 rhel-8-for-x86_64-appstream-rpms 214 k 2021-02-10T04:17:47.055 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:17:47.055 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2021-02-10T04:17:47.055 INFO:teuthology.orchestra.run.smithi164.stdout:====================================================================================================== 2021-02-10T04:17:47.055 INFO:teuthology.orchestra.run.smithi164.stdout:Install 37 Packages 2021-02-10T04:17:47.056 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:17:47.057 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 33 M 2021-02-10T04:17:47.057 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 117 M 2021-02-10T04:17:47.057 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2021-02-10T04:17:47.105 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-cryptography-2.3-3.el8.x86_64 13/37 2021-02-10T04:17:47.317 INFO:teuthology.orchestra.run.smithi164.stdout:(1/37): ceph-17.0.0-681.gc1ea6241.el8.x86_64.rp 24 kB/s | 6.4 kB 00:00 2021-02-10T04:17:47.335 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-pyOpenSSL-18.0.0-1.el8.noarch 14/37 2021-02-10T04:17:47.534 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 15/37 2021-02-10T04:17:47.765 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 16/37 2021-02-10T04:17:47.999 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 17/37 2021-02-10T04:17:48.171 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 18/37 2021-02-10T04:17:48.322 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 19/37 2021-02-10T04:17:48.512 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 20/37 2021-02-10T04:17:48.664 INFO:teuthology.orchestra.run.smithi164.stdout:(2/37): ceph-mgr-17.0.0-681.gc1ea6241.el8.x86_6 978 kB/s | 1.5 MB 00:01 2021-02-10T04:17:48.690 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 21/37 2021-02-10T04:17:48.929 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 22/37 2021-02-10T04:17:48.965 INFO:teuthology.orchestra.run.smithi164.stdout:(3/37): ceph-mds-17.0.0-681.gc1ea6241.el8.x86_6 1.3 MB/s | 2.4 MB 00:01 2021-02-10T04:17:49.012 INFO:teuthology.orchestra.run.smithi164.stdout:(4/37): ceph-mgr-modules-core-17.0.0-681.gc1ea6 4.6 MB/s | 222 kB 00:00 2021-02-10T04:17:49.090 INFO:teuthology.orchestra.run.smithi164.stdout:(5/37): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 557 kB/s | 44 kB 00:00 2021-02-10T04:17:49.149 INFO:teuthology.orchestra.run.smithi164.stdout:(6/37): python3-beautifulsoup4-4.6.3-2.el8.1.no 3.1 MB/s | 185 kB 00:00 2021-02-10T04:17:49.163 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 23/37 2021-02-10T04:17:49.170 INFO:teuthology.orchestra.run.smithi164.stdout:(7/37): python3-cheroot-8.5.2-1.el8.noarch.rpm 8.0 MB/s | 173 kB 00:00 2021-02-10T04:17:49.210 INFO:teuthology.orchestra.run.smithi164.stdout:(8/37): python3-cherrypy-18.4.0-1.el8.noarch.rp 9.5 MB/s | 384 kB 00:00 2021-02-10T04:17:49.230 INFO:teuthology.orchestra.run.smithi164.stdout:(9/37): python3-cssselect-0.9.2-10.el8.noarch.r 2.0 MB/s | 40 kB 00:00 2021-02-10T04:17:49.249 INFO:teuthology.orchestra.run.smithi164.stdout:(10/37): python3-jaraco-6.2-6.el8.noarch.rpm 573 kB/s | 11 kB 00:00 2021-02-10T04:17:49.269 INFO:teuthology.orchestra.run.smithi164.stdout:(11/37): python3-jaraco-functools-2.0-4.el8.noa 895 kB/s | 18 kB 00:00 2021-02-10T04:17:49.289 INFO:teuthology.orchestra.run.smithi164.stdout:(12/37): python3-logutils-0.3.5-11.el8.noarch.r 2.4 MB/s | 49 kB 00:00 2021-02-10T04:17:49.309 INFO:teuthology.orchestra.run.smithi164.stdout:(13/37): python3-more-itertools-7.2.0-3.el8.noa 2.9 MB/s | 59 kB 00:00 2021-02-10T04:17:49.323 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 24/37 2021-02-10T04:17:49.331 INFO:teuthology.orchestra.run.smithi164.stdout:(14/37): python3-pecan-1.3.2-9.el8.noarch.rpm 13 MB/s | 283 kB 00:00 2021-02-10T04:17:49.350 INFO:teuthology.orchestra.run.smithi164.stdout:(15/37): python3-portend-2.6-1.el8.noarch.rpm 795 kB/s | 16 kB 00:00 2021-02-10T04:17:49.371 INFO:teuthology.orchestra.run.smithi164.stdout:(16/37): python3-simplegeneric-0.8.1-17.el8.noa 942 kB/s | 19 kB 00:00 2021-02-10T04:17:49.384 INFO:teuthology.orchestra.run.smithi164.stdout:(17/37): ceph-mon-17.0.0-681.gc1ea6241.el8.x86_ 2.4 MB/s | 5.0 MB 00:02 2021-02-10T04:17:49.390 INFO:teuthology.orchestra.run.smithi164.stdout:(18/37): python3-singledispatch-3.4.0.3-18.el8. 1.2 MB/s | 24 kB 00:00 2021-02-10T04:17:49.410 INFO:teuthology.orchestra.run.smithi164.stdout:(19/37): python3-trustme-0.6.0-4.el8.noarch.rpm 1.4 MB/s | 27 kB 00:00 2021-02-10T04:17:49.432 INFO:teuthology.orchestra.run.smithi164.stdout:(20/37): python3-waitress-1.2.1-2.el8.1.noarch. 11 MB/s | 241 kB 00:00 2021-02-10T04:17:49.432 INFO:teuthology.orchestra.run.smithi164.stdout:(21/37): python3-tempora-1.14.1-5.el8.noarch.rp 605 kB/s | 29 kB 00:00 2021-02-10T04:17:49.453 INFO:teuthology.orchestra.run.smithi164.stdout:(22/37): python3-webob-1.8.5-1.el8.1.noarch.rpm 11 MB/s | 251 kB 00:00 2021-02-10T04:17:49.465 INFO:teuthology.orchestra.run.smithi164.stdout:(23/37): python3-webtest-2.0.33-1.el8.noarch.rp 2.6 MB/s | 85 kB 00:00 2021-02-10T04:17:49.474 INFO:teuthology.orchestra.run.smithi164.stdout:(24/37): python3-zc-lockfile-2.0-2.el8.noarch.r 1.1 MB/s | 23 kB 00:00 2021-02-10T04:17:49.505 INFO:teuthology.orchestra.run.smithi164.stdout:(25/37): python3-cffi-1.11.5-5.el8.x86_64.rpm 5.8 MB/s | 238 kB 00:00 2021-02-10T04:17:49.512 INFO:teuthology.orchestra.run.smithi164.stdout:(26/37): python3-asn1crypto-0.24.0-3.el8.noarch 4.6 MB/s | 181 kB 00:00 2021-02-10T04:17:49.529 INFO:teuthology.orchestra.run.smithi164.stdout:(27/37): python3-pycparser-2.14-14.el8.noarch.r 4.3 MB/s | 109 kB 00:00 2021-02-10T04:17:49.540 INFO:teuthology.orchestra.run.smithi164.stdout:(28/37): python3-cryptography-2.3-3.el8.x86_64. 18 MB/s | 511 kB 00:00 2021-02-10T04:17:49.552 INFO:teuthology.orchestra.run.smithi164.stdout:(29/37): python3-webencodings-0.5.1-6.el8.noarc 1.2 MB/s | 27 kB 00:00 2021-02-10T04:17:49.578 INFO:teuthology.orchestra.run.smithi164.stdout:(30/37): python3-lxml-4.2.3-1.el8.x86_64.rpm 45 MB/s | 1.5 MB 00:00 2021-02-10T04:17:49.579 INFO:teuthology.orchestra.run.smithi164.stdout:(31/37): python3-pyOpenSSL-18.0.0-1.el8.noarch. 3.7 MB/s | 103 kB 00:00 2021-02-10T04:17:49.604 INFO:teuthology.orchestra.run.smithi164.stdout:(32/37): python3-werkzeug-0.12.2-4.el8.noarch.r 17 MB/s | 457 kB 00:00 2021-02-10T04:17:49.607 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-portend-2.6-1.el8.noarch 25/37 2021-02-10T04:17:49.612 INFO:teuthology.orchestra.run.smithi164.stdout:(33/37): python3-mako-1.0.6-13.el8.noarch.rpm 4.7 MB/s | 157 kB 00:00 2021-02-10T04:17:49.630 INFO:teuthology.orchestra.run.smithi164.stdout:(34/37): python3-html5lib-0.999999999-6.el8.noa 8.1 MB/s | 214 kB 00:00 2021-02-10T04:17:49.636 INFO:teuthology.orchestra.run.smithi164.stdout:(35/37): python3-pytz-2017.2-9.el8.noarch.rpm 2.2 MB/s | 54 kB 00:00 2021-02-10T04:17:49.649 INFO:teuthology.orchestra.run.smithi164.stdout:(36/37): python3-markupsafe-0.23-19.el8.x86_64. 2.0 MB/s | 39 kB 00:00 2021-02-10T04:17:49.878 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 26/37 2021-02-10T04:17:50.108 INFO:teuthology.orchestra.run.smithi164.stdout:(37/37): ceph-osd-17.0.0-681.gc1ea6241.el8.x86_ 13 MB/s | 18 MB 00:01 2021-02-10T04:17:50.111 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2021-02-10T04:17:50.112 INFO:teuthology.orchestra.run.smithi164.stdout:Total 11 MB/s | 33 MB 00:03 2021-02-10T04:17:50.122 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2021-02-10T04:17:50.162 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2021-02-10T04:17:50.162 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2021-02-10T04:17:50.196 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 27/37 2021-02-10T04:17:50.425 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2021-02-10T04:17:50.429 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2021-02-10T04:17:50.499 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 28/37 2021-02-10T04:17:51.166 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 29/37 2021-02-10T04:17:51.542 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 30/37 2021-02-10T04:17:51.796 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2021-02-10T04:17:51.856 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 31/37 2021-02-10T04:17:52.034 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/37 2021-02-10T04:17:52.220 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/37 2021-02-10T04:17:52.220 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-681.gc1ea6241.el8.n 32/37 2021-02-10T04:17:52.435 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : ceph-mgr-2:17.0.0-681.gc1ea6241.el8.x86_64 33/37 2021-02-10T04:17:52.455 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 3/37 2021-02-10T04:17:52.658 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 4/37 2021-02-10T04:17:52.929 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 5/37 2021-02-10T04:17:53.159 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/37 2021-02-10T04:17:53.389 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 7/37 2021-02-10T04:17:53.645 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 8/37 2021-02-10T04:17:53.886 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 9/37 2021-02-10T04:17:53.959 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: ceph-mgr-2:17.0.0-681.gc1ea6241.el8.x86_64 33/37 2021-02-10T04:17:54.107 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 10/37 2021-02-10T04:17:54.177 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : ceph-osd-2:17.0.0-681.gc1ea6241.el8.x86_64 34/37 2021-02-10T04:17:54.335 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 11/37 2021-02-10T04:17:54.610 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: ceph-osd-2:17.0.0-681.gc1ea6241.el8.x86_64 34/37 2021-02-10T04:17:54.691 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-asn1crypto-0.24.0-3.el8.noarch 12/37 2021-02-10T04:17:54.793 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : ceph-mon-2:17.0.0-681.gc1ea6241.el8.x86_64 35/37 2021-02-10T04:17:54.944 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cryptography-2.3-3.el8.x86_64 13/37 2021-02-10T04:17:55.016 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: ceph-mon-2:17.0.0-681.gc1ea6241.el8.x86_64 35/37 2021-02-10T04:17:55.097 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyOpenSSL-18.0.0-1.el8.noarch 14/37 2021-02-10T04:17:55.165 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : ceph-mds-2:17.0.0-681.gc1ea6241.el8.x86_64 36/37 2021-02-10T04:17:55.185 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: ceph-mds-2:17.0.0-681.gc1ea6241.el8.x86_64 36/37 2021-02-10T04:17:55.298 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 15/37 2021-02-10T04:17:55.322 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : ceph-2:17.0.0-681.gc1ea6241.el8.x86_64 37/37 2021-02-10T04:17:55.508 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 16/37 2021-02-10T04:17:55.693 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: ceph-2:17.0.0-681.gc1ea6241.el8.x86_64 37/37 2021-02-10T04:17:55.693 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : ceph-2:17.0.0-681.gc1ea6241.el8.x86_64 1/37 2021-02-10T04:17:55.693 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : ceph-mds-2:17.0.0-681.gc1ea6241.el8.x86_64 2/37 2021-02-10T04:17:55.693 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : ceph-mgr-2:17.0.0-681.gc1ea6241.el8.x86_64 3/37 2021-02-10T04:17:55.693 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : ceph-mon-2:17.0.0-681.gc1ea6241.el8.x86_64 4/37 2021-02-10T04:17:55.693 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : ceph-osd-2:17.0.0-681.gc1ea6241.el8.x86_64 5/37 2021-02-10T04:17:55.693 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-681.gc1ea6241.el8.n 6/37 2021-02-10T04:17:55.694 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/37 2021-02-10T04:17:55.694 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/37 2021-02-10T04:17:55.694 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/37 2021-02-10T04:17:55.694 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/37 2021-02-10T04:17:55.694 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 11/37 2021-02-10T04:17:55.694 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 12/37 2021-02-10T04:17:55.694 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 13/37 2021-02-10T04:17:55.694 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 14/37 2021-02-10T04:17:55.694 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 15/37 2021-02-10T04:17:55.694 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 16/37 2021-02-10T04:17:55.694 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-portend-2.6-1.el8.noarch 17/37 2021-02-10T04:17:55.695 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 18/37 2021-02-10T04:17:55.695 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 19/37 2021-02-10T04:17:55.695 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 20/37 2021-02-10T04:17:55.695 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 21/37 2021-02-10T04:17:55.695 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 22/37 2021-02-10T04:17:55.695 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 23/37 2021-02-10T04:17:55.695 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 24/37 2021-02-10T04:17:55.695 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 25/37 2021-02-10T04:17:55.695 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 26/37 2021-02-10T04:17:55.695 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-asn1crypto-0.24.0-3.el8.noarch 27/37 2021-02-10T04:17:55.695 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 28/37 2021-02-10T04:17:55.695 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-cryptography-2.3-3.el8.x86_64 29/37 2021-02-10T04:17:55.695 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 30/37 2021-02-10T04:17:55.696 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 31/37 2021-02-10T04:17:55.696 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-pyOpenSSL-18.0.0-1.el8.noarch 32/37 2021-02-10T04:17:55.696 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 33/37 2021-02-10T04:17:55.696 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 34/37 2021-02-10T04:17:55.696 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 35/37 2021-02-10T04:17:55.696 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 36/37 2021-02-10T04:17:55.746 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 17/37 2021-02-10T04:17:55.893 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 18/37 2021-02-10T04:17:56.084 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 19/37 2021-02-10T04:17:56.259 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 20/37 2021-02-10T04:17:56.850 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 37/37 2021-02-10T04:17:56.850 INFO:teuthology.orchestra.run.smithi085.stdout:Installed products updated. 2021-02-10T04:17:56.930 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 21/37 2021-02-10T04:17:57.136 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 22/37 2021-02-10T04:17:57.428 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 23/37 2021-02-10T04:17:57.612 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 24/37 2021-02-10T04:17:57.881 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-portend-2.6-1.el8.noarch 25/37 2021-02-10T04:17:58.126 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 26/37 2021-02-10T04:17:58.444 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 27/37 2021-02-10T04:17:58.660 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:17:58.660 INFO:teuthology.orchestra.run.smithi085.stdout:Installed: 2021-02-10T04:17:58.660 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:58.660 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-mds-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:58.660 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-mgr-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:58.661 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-mgr-modules-core-2:17.0.0-681.gc1ea6241.el8.noarch 2021-02-10T04:17:58.661 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-mon-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:58.661 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-osd-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:17:58.661 INFO:teuthology.orchestra.run.smithi085.stdout: python3-asn1crypto-0.24.0-3.el8.noarch 2021-02-10T04:17:58.661 INFO:teuthology.orchestra.run.smithi085.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-02-10T04:17:58.661 INFO:teuthology.orchestra.run.smithi085.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-02-10T04:17:58.661 INFO:teuthology.orchestra.run.smithi085.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-02-10T04:17:58.661 INFO:teuthology.orchestra.run.smithi085.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-02-10T04:17:58.661 INFO:teuthology.orchestra.run.smithi085.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-02-10T04:17:58.661 INFO:teuthology.orchestra.run.smithi085.stdout: python3-cryptography-2.3-3.el8.x86_64 2021-02-10T04:17:58.661 INFO:teuthology.orchestra.run.smithi085.stdout: python3-cssselect-0.9.2-10.el8.noarch 2021-02-10T04:17:58.661 INFO:teuthology.orchestra.run.smithi085.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2021-02-10T04:17:58.662 INFO:teuthology.orchestra.run.smithi085.stdout: python3-jaraco-6.2-6.el8.noarch 2021-02-10T04:17:58.662 INFO:teuthology.orchestra.run.smithi085.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-02-10T04:17:58.662 INFO:teuthology.orchestra.run.smithi085.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-02-10T04:17:58.662 INFO:teuthology.orchestra.run.smithi085.stdout: python3-lxml-4.2.3-1.el8.x86_64 2021-02-10T04:17:58.662 INFO:teuthology.orchestra.run.smithi085.stdout: python3-mako-1.0.6-13.el8.noarch 2021-02-10T04:17:58.662 INFO:teuthology.orchestra.run.smithi085.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-02-10T04:17:58.663 INFO:teuthology.orchestra.run.smithi085.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-02-10T04:17:58.663 INFO:teuthology.orchestra.run.smithi085.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-02-10T04:17:58.663 INFO:teuthology.orchestra.run.smithi085.stdout: python3-portend-2.6-1.el8.noarch 2021-02-10T04:17:58.663 INFO:teuthology.orchestra.run.smithi085.stdout: python3-pyOpenSSL-18.0.0-1.el8.noarch 2021-02-10T04:17:58.663 INFO:teuthology.orchestra.run.smithi085.stdout: python3-pycparser-2.14-14.el8.noarch 2021-02-10T04:17:58.663 INFO:teuthology.orchestra.run.smithi085.stdout: python3-pytz-2017.2-9.el8.noarch 2021-02-10T04:17:58.663 INFO:teuthology.orchestra.run.smithi085.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-02-10T04:17:58.663 INFO:teuthology.orchestra.run.smithi085.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-02-10T04:17:58.663 INFO:teuthology.orchestra.run.smithi085.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-02-10T04:17:58.663 INFO:teuthology.orchestra.run.smithi085.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-02-10T04:17:58.663 INFO:teuthology.orchestra.run.smithi085.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-02-10T04:17:58.664 INFO:teuthology.orchestra.run.smithi085.stdout: python3-webencodings-0.5.1-6.el8.noarch 2021-02-10T04:17:58.664 INFO:teuthology.orchestra.run.smithi085.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-02-10T04:17:58.664 INFO:teuthology.orchestra.run.smithi085.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-02-10T04:17:58.664 INFO:teuthology.orchestra.run.smithi085.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-02-10T04:17:58.664 INFO:teuthology.orchestra.run.smithi085.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-02-10T04:17:58.664 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:17:58.664 INFO:teuthology.orchestra.run.smithi085.stdout:Complete! 2021-02-10T04:17:58.772 DEBUG:teuthology.orchestra.run.smithi085:> sudo yum -y install ceph-base 2021-02-10T04:17:58.806 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 28/37 2021-02-10T04:17:59.040 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 29/37 2021-02-10T04:17:59.052 INFO:teuthology.orchestra.run.smithi085.stdout:Updating Subscription Management repositories. 2021-02-10T04:17:59.328 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 30/37 2021-02-10T04:17:59.564 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 85 kB/s | 2.4 kB 00:00 2021-02-10T04:17:59.612 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 31/37 2021-02-10T04:17:59.673 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 99 kB/s | 2.8 kB 00:00 2021-02-10T04:17:59.993 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-681.gc1ea6241.el8.n 32/37 2021-02-10T04:18:00.191 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-2:17.0.0-681.gc1ea6241.el8.x86_64 33/37 2021-02-10T04:18:00.864 INFO:teuthology.orchestra.run.smithi085.stdout:Package ceph-base-2:17.0.0-681.gc1ea6241.el8.x86_64 is already installed. 2021-02-10T04:18:01.183 INFO:teuthology.orchestra.run.smithi085.stdout:Dependencies resolved. 2021-02-10T04:18:01.184 INFO:teuthology.orchestra.run.smithi085.stdout:Nothing to do. 2021-02-10T04:18:01.184 INFO:teuthology.orchestra.run.smithi085.stdout:Complete! 2021-02-10T04:18:01.271 DEBUG:teuthology.orchestra.run.smithi085:> sudo yum -y install cephadm 2021-02-10T04:18:01.550 INFO:teuthology.orchestra.run.smithi085.stdout:Updating Subscription Management repositories. 2021-02-10T04:18:01.725 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-2:17.0.0-681.gc1ea6241.el8.x86_64 33/37 2021-02-10T04:18:01.924 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-osd-2:17.0.0-681.gc1ea6241.el8.x86_64 34/37 2021-02-10T04:18:02.056 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 78 kB/s | 2.4 kB 00:00 2021-02-10T04:18:02.166 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 96 kB/s | 2.8 kB 00:00 2021-02-10T04:18:02.360 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-osd-2:17.0.0-681.gc1ea6241.el8.x86_64 34/37 2021-02-10T04:18:02.549 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mon-2:17.0.0-681.gc1ea6241.el8.x86_64 35/37 2021-02-10T04:18:02.775 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mon-2:17.0.0-681.gc1ea6241.el8.x86_64 35/37 2021-02-10T04:18:02.971 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mds-2:17.0.0-681.gc1ea6241.el8.x86_64 36/37 2021-02-10T04:18:02.992 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mds-2:17.0.0-681.gc1ea6241.el8.x86_64 36/37 2021-02-10T04:18:03.103 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-2:17.0.0-681.gc1ea6241.el8.x86_64 37/37 2021-02-10T04:18:03.449 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-2:17.0.0-681.gc1ea6241.el8.x86_64 37/37 2021-02-10T04:18:03.449 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-2:17.0.0-681.gc1ea6241.el8.x86_64 1/37 2021-02-10T04:18:03.449 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mds-2:17.0.0-681.gc1ea6241.el8.x86_64 2/37 2021-02-10T04:18:03.449 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-2:17.0.0-681.gc1ea6241.el8.x86_64 3/37 2021-02-10T04:18:03.449 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mon-2:17.0.0-681.gc1ea6241.el8.x86_64 4/37 2021-02-10T04:18:03.450 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-osd-2:17.0.0-681.gc1ea6241.el8.x86_64 5/37 2021-02-10T04:18:03.450 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-681.gc1ea6241.el8.n 6/37 2021-02-10T04:18:03.450 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/37 2021-02-10T04:18:03.450 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/37 2021-02-10T04:18:03.450 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/37 2021-02-10T04:18:03.450 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/37 2021-02-10T04:18:03.450 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 11/37 2021-02-10T04:18:03.450 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 12/37 2021-02-10T04:18:03.450 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 13/37 2021-02-10T04:18:03.450 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 14/37 2021-02-10T04:18:03.450 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 15/37 2021-02-10T04:18:03.450 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 16/37 2021-02-10T04:18:03.450 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-portend-2.6-1.el8.noarch 17/37 2021-02-10T04:18:03.451 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 18/37 2021-02-10T04:18:03.451 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 19/37 2021-02-10T04:18:03.451 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 20/37 2021-02-10T04:18:03.451 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 21/37 2021-02-10T04:18:03.451 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 22/37 2021-02-10T04:18:03.451 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 23/37 2021-02-10T04:18:03.451 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 24/37 2021-02-10T04:18:03.451 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 25/37 2021-02-10T04:18:03.451 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 26/37 2021-02-10T04:18:03.451 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-asn1crypto-0.24.0-3.el8.noarch 27/37 2021-02-10T04:18:03.451 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 28/37 2021-02-10T04:18:03.451 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cryptography-2.3-3.el8.x86_64 29/37 2021-02-10T04:18:03.451 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 30/37 2021-02-10T04:18:03.452 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 31/37 2021-02-10T04:18:03.452 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyOpenSSL-18.0.0-1.el8.noarch 32/37 2021-02-10T04:18:03.452 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 33/37 2021-02-10T04:18:03.452 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 34/37 2021-02-10T04:18:03.452 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 35/37 2021-02-10T04:18:03.452 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 36/37 2021-02-10T04:18:03.651 INFO:teuthology.orchestra.run.smithi085.stdout:Dependencies resolved. 2021-02-10T04:18:03.652 INFO:teuthology.orchestra.run.smithi085.stdout:================================================================================ 2021-02-10T04:18:03.652 INFO:teuthology.orchestra.run.smithi085.stdout: Package Arch Version Repository Size 2021-02-10T04:18:03.652 INFO:teuthology.orchestra.run.smithi085.stdout:================================================================================ 2021-02-10T04:18:03.652 INFO:teuthology.orchestra.run.smithi085.stdout:Installing: 2021-02-10T04:18:03.652 INFO:teuthology.orchestra.run.smithi085.stdout: cephadm noarch 2:17.0.0-681.gc1ea6241.el8 ceph-noarch 69 k 2021-02-10T04:18:03.652 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:18:03.652 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction Summary 2021-02-10T04:18:03.652 INFO:teuthology.orchestra.run.smithi085.stdout:================================================================================ 2021-02-10T04:18:03.652 INFO:teuthology.orchestra.run.smithi085.stdout:Install 1 Package 2021-02-10T04:18:03.652 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:18:03.652 INFO:teuthology.orchestra.run.smithi085.stdout:Total download size: 69 k 2021-02-10T04:18:03.652 INFO:teuthology.orchestra.run.smithi085.stdout:Installed size: 277 k 2021-02-10T04:18:03.653 INFO:teuthology.orchestra.run.smithi085.stdout:Downloading Packages: 2021-02-10T04:18:04.083 INFO:teuthology.orchestra.run.smithi085.stdout:cephadm-17.0.0-681.gc1ea6241.el8.noarch.rpm 159 kB/s | 69 kB 00:00 2021-02-10T04:18:04.084 INFO:teuthology.orchestra.run.smithi085.stdout:-------------------------------------------------------------------------------- 2021-02-10T04:18:04.084 INFO:teuthology.orchestra.run.smithi085.stdout:Total 159 kB/s | 69 kB 00:00 2021-02-10T04:18:04.085 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction check 2021-02-10T04:18:04.091 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction check succeeded. 2021-02-10T04:18:04.092 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction test 2021-02-10T04:18:04.093 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction test succeeded. 2021-02-10T04:18:04.098 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction 2021-02-10T04:18:04.139 INFO:teuthology.orchestra.run.smithi085.stdout: Preparing : 1/1 2021-02-10T04:18:04.476 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 37/37 2021-02-10T04:18:04.476 INFO:teuthology.orchestra.run.smithi164.stdout:Installed products updated. 2021-02-10T04:18:06.125 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: cephadm-2:17.0.0-681.gc1ea6241.el8.noarch 1/1 2021-02-10T04:18:06.328 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:18:06.328 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2021-02-10T04:18:06.328 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:18:06.328 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mds-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:18:06.328 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:18:06.328 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-modules-core-2:17.0.0-681.gc1ea6241.el8.noarch 2021-02-10T04:18:06.328 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mon-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:18:06.328 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-osd-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:18:06.328 INFO:teuthology.orchestra.run.smithi164.stdout: python3-asn1crypto-0.24.0-3.el8.noarch 2021-02-10T04:18:06.328 INFO:teuthology.orchestra.run.smithi164.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-02-10T04:18:06.329 INFO:teuthology.orchestra.run.smithi164.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-02-10T04:18:06.329 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-02-10T04:18:06.329 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-02-10T04:18:06.329 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-02-10T04:18:06.329 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cryptography-2.3-3.el8.x86_64 2021-02-10T04:18:06.329 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cssselect-0.9.2-10.el8.noarch 2021-02-10T04:18:06.329 INFO:teuthology.orchestra.run.smithi164.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2021-02-10T04:18:06.329 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-6.2-6.el8.noarch 2021-02-10T04:18:06.329 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-02-10T04:18:06.329 INFO:teuthology.orchestra.run.smithi164.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-02-10T04:18:06.329 INFO:teuthology.orchestra.run.smithi164.stdout: python3-lxml-4.2.3-1.el8.x86_64 2021-02-10T04:18:06.329 INFO:teuthology.orchestra.run.smithi164.stdout: python3-mako-1.0.6-13.el8.noarch 2021-02-10T04:18:06.330 INFO:teuthology.orchestra.run.smithi164.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-02-10T04:18:06.330 INFO:teuthology.orchestra.run.smithi164.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-02-10T04:18:06.330 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-02-10T04:18:06.330 INFO:teuthology.orchestra.run.smithi164.stdout: python3-portend-2.6-1.el8.noarch 2021-02-10T04:18:06.330 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyOpenSSL-18.0.0-1.el8.noarch 2021-02-10T04:18:06.330 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pycparser-2.14-14.el8.noarch 2021-02-10T04:18:06.330 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pytz-2017.2-9.el8.noarch 2021-02-10T04:18:06.330 INFO:teuthology.orchestra.run.smithi164.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-02-10T04:18:06.330 INFO:teuthology.orchestra.run.smithi164.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-02-10T04:18:06.330 INFO:teuthology.orchestra.run.smithi164.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-02-10T04:18:06.330 INFO:teuthology.orchestra.run.smithi164.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-02-10T04:18:06.330 INFO:teuthology.orchestra.run.smithi164.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-02-10T04:18:06.331 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webencodings-0.5.1-6.el8.noarch 2021-02-10T04:18:06.331 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-02-10T04:18:06.331 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-02-10T04:18:06.331 INFO:teuthology.orchestra.run.smithi164.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-02-10T04:18:06.331 INFO:teuthology.orchestra.run.smithi164.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-02-10T04:18:06.331 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:18:06.331 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2021-02-10T04:18:06.337 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : cephadm-2:17.0.0-681.gc1ea6241.el8.noarch 1/1 2021-02-10T04:18:06.433 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-base 2021-02-10T04:18:06.504 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: cephadm-2:17.0.0-681.gc1ea6241.el8.noarch 1/1 2021-02-10T04:18:06.715 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2021-02-10T04:18:06.956 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : cephadm-2:17.0.0-681.gc1ea6241.el8.noarch 1/1 2021-02-10T04:18:06.956 INFO:teuthology.orchestra.run.smithi085.stdout:Installed products updated. 2021-02-10T04:18:07.216 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 79 kB/s | 2.4 kB 00:00 2021-02-10T04:18:07.329 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 94 kB/s | 2.8 kB 00:00 2021-02-10T04:18:08.493 INFO:teuthology.orchestra.run.smithi164.stdout:Package ceph-base-2:17.0.0-681.gc1ea6241.el8.x86_64 is already installed. 2021-02-10T04:18:08.779 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:18:08.780 INFO:teuthology.orchestra.run.smithi085.stdout:Installed: 2021-02-10T04:18:08.780 INFO:teuthology.orchestra.run.smithi085.stdout: cephadm-2:17.0.0-681.gc1ea6241.el8.noarch 2021-02-10T04:18:08.780 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:18:08.780 INFO:teuthology.orchestra.run.smithi085.stdout:Complete! 2021-02-10T04:18:08.817 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2021-02-10T04:18:08.818 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2021-02-10T04:18:08.818 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2021-02-10T04:18:08.870 DEBUG:teuthology.orchestra.run.smithi085:> sudo yum -y install ceph-immutable-object-cache 2021-02-10T04:18:08.890 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install cephadm 2021-02-10T04:18:09.157 INFO:teuthology.orchestra.run.smithi085.stdout:Updating Subscription Management repositories. 2021-02-10T04:18:09.167 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2021-02-10T04:18:09.654 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 82 kB/s | 2.4 kB 00:00 2021-02-10T04:18:09.686 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 2021-02-10T04:18:09.765 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 95 kB/s | 2.8 kB 00:00 2021-02-10T04:18:09.798 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 96 kB/s | 2.8 kB 00:00 2021-02-10T04:18:11.239 INFO:teuthology.orchestra.run.smithi085.stdout:Dependencies resolved. 2021-02-10T04:18:11.239 INFO:teuthology.orchestra.run.smithi085.stdout:================================================================================ 2021-02-10T04:18:11.239 INFO:teuthology.orchestra.run.smithi085.stdout: Package Arch Version Repo Size 2021-02-10T04:18:11.240 INFO:teuthology.orchestra.run.smithi085.stdout:================================================================================ 2021-02-10T04:18:11.240 INFO:teuthology.orchestra.run.smithi085.stdout:Installing: 2021-02-10T04:18:11.240 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 163 k 2021-02-10T04:18:11.240 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:18:11.240 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction Summary 2021-02-10T04:18:11.240 INFO:teuthology.orchestra.run.smithi085.stdout:================================================================================ 2021-02-10T04:18:11.240 INFO:teuthology.orchestra.run.smithi085.stdout:Install 1 Package 2021-02-10T04:18:11.240 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:18:11.240 INFO:teuthology.orchestra.run.smithi085.stdout:Total download size: 163 k 2021-02-10T04:18:11.240 INFO:teuthology.orchestra.run.smithi085.stdout:Installed size: 587 k 2021-02-10T04:18:11.240 INFO:teuthology.orchestra.run.smithi085.stdout:Downloading Packages: 2021-02-10T04:18:11.291 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2021-02-10T04:18:11.292 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2021-02-10T04:18:11.292 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2021-02-10T04:18:11.292 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2021-02-10T04:18:11.292 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2021-02-10T04:18:11.292 INFO:teuthology.orchestra.run.smithi164.stdout: cephadm noarch 2:17.0.0-681.gc1ea6241.el8 ceph-noarch 69 k 2021-02-10T04:18:11.292 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:18:11.293 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2021-02-10T04:18:11.293 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2021-02-10T04:18:11.293 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2021-02-10T04:18:11.293 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:18:11.294 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 69 k 2021-02-10T04:18:11.294 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 277 k 2021-02-10T04:18:11.294 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2021-02-10T04:18:11.709 INFO:teuthology.orchestra.run.smithi085.stdout:ceph-immutable-object-cache-17.0.0-681.gc1ea624 348 kB/s | 163 kB 00:00 2021-02-10T04:18:11.710 INFO:teuthology.orchestra.run.smithi085.stdout:-------------------------------------------------------------------------------- 2021-02-10T04:18:11.710 INFO:teuthology.orchestra.run.smithi085.stdout:Total 346 kB/s | 163 kB 00:00 2021-02-10T04:18:11.710 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction check 2021-02-10T04:18:11.722 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction check succeeded. 2021-02-10T04:18:11.723 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction test 2021-02-10T04:18:11.747 INFO:teuthology.orchestra.run.smithi164.stdout:cephadm-17.0.0-681.gc1ea6241.el8.noarch.rpm 151 kB/s | 69 kB 00:00 2021-02-10T04:18:11.747 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2021-02-10T04:18:11.748 INFO:teuthology.orchestra.run.smithi164.stdout:Total 151 kB/s | 69 kB 00:00 2021-02-10T04:18:11.748 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2021-02-10T04:18:11.755 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2021-02-10T04:18:11.755 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2021-02-10T04:18:11.757 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2021-02-10T04:18:11.762 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2021-02-10T04:18:11.775 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction test succeeded. 2021-02-10T04:18:11.779 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction 2021-02-10T04:18:11.833 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2021-02-10T04:18:11.898 INFO:teuthology.orchestra.run.smithi085.stdout: Preparing : 1/1 2021-02-10T04:18:12.071 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-681.gc1ea6241.e 1/1 2021-02-10T04:18:12.412 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-681.gc1ea6241.e 1/1 2021-02-10T04:18:12.870 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-681.gc1ea6241.e 1/1 2021-02-10T04:18:12.870 INFO:teuthology.orchestra.run.smithi085.stdout:Installed products updated. 2021-02-10T04:18:14.698 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:18:14.698 INFO:teuthology.orchestra.run.smithi085.stdout:Installed: 2021-02-10T04:18:14.699 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-immutable-object-cache-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:18:14.699 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:18:14.699 INFO:teuthology.orchestra.run.smithi085.stdout:Complete! 2021-02-10T04:18:14.789 DEBUG:teuthology.orchestra.run.smithi085:> sudo yum -y install ceph-mgr 2021-02-10T04:18:15.076 INFO:teuthology.orchestra.run.smithi085.stdout:Updating Subscription Management repositories. 2021-02-10T04:18:15.516 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: cephadm-2:17.0.0-681.gc1ea6241.el8.noarch 1/1 2021-02-10T04:18:15.555 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 2021-02-10T04:18:15.664 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 100 kB/s | 2.8 kB 00:00 2021-02-10T04:18:15.726 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : cephadm-2:17.0.0-681.gc1ea6241.el8.noarch 1/1 2021-02-10T04:18:15.892 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: cephadm-2:17.0.0-681.gc1ea6241.el8.noarch 1/1 2021-02-10T04:18:16.327 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : cephadm-2:17.0.0-681.gc1ea6241.el8.noarch 1/1 2021-02-10T04:18:16.327 INFO:teuthology.orchestra.run.smithi164.stdout:Installed products updated. 2021-02-10T04:18:16.803 INFO:teuthology.orchestra.run.smithi085.stdout:Package ceph-mgr-2:17.0.0-681.gc1ea6241.el8.x86_64 is already installed. 2021-02-10T04:18:17.118 INFO:teuthology.orchestra.run.smithi085.stdout:Dependencies resolved. 2021-02-10T04:18:17.119 INFO:teuthology.orchestra.run.smithi085.stdout:Nothing to do. 2021-02-10T04:18:17.119 INFO:teuthology.orchestra.run.smithi085.stdout:Complete! 2021-02-10T04:18:17.223 DEBUG:teuthology.orchestra.run.smithi085:> sudo yum -y install ceph-mgr-dashboard 2021-02-10T04:18:17.500 INFO:teuthology.orchestra.run.smithi085.stdout:Updating Subscription Management repositories. 2021-02-10T04:18:17.980 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 82 kB/s | 2.4 kB 00:00 2021-02-10T04:18:18.089 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 96 kB/s | 2.8 kB 00:00 2021-02-10T04:18:18.187 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:18:18.187 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2021-02-10T04:18:18.187 INFO:teuthology.orchestra.run.smithi164.stdout: cephadm-2:17.0.0-681.gc1ea6241.el8.noarch 2021-02-10T04:18:18.187 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:18:18.187 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2021-02-10T04:18:18.267 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-immutable-object-cache 2021-02-10T04:18:18.556 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2021-02-10T04:18:19.036 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 80 kB/s | 2.4 kB 00:00 2021-02-10T04:18:19.145 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 96 kB/s | 2.8 kB 00:00 2021-02-10T04:18:19.628 INFO:teuthology.orchestra.run.smithi085.stdout:Dependencies resolved. 2021-02-10T04:18:19.628 INFO:teuthology.orchestra.run.smithi085.stdout:================================================================================================== 2021-02-10T04:18:19.629 INFO:teuthology.orchestra.run.smithi085.stdout: Package Arch Version Repository Size 2021-02-10T04:18:19.629 INFO:teuthology.orchestra.run.smithi085.stdout:================================================================================================== 2021-02-10T04:18:19.629 INFO:teuthology.orchestra.run.smithi085.stdout:Installing: 2021-02-10T04:18:19.629 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-mgr-dashboard noarch 2:17.0.0-681.gc1ea6241.el8 ceph-noarch 2.6 M 2021-02-10T04:18:19.629 INFO:teuthology.orchestra.run.smithi085.stdout:Installing dependencies: 2021-02-10T04:18:19.629 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-grafana-dashboards noarch 2:17.0.0-681.gc1ea6241.el8 ceph-noarch 24 k 2021-02-10T04:18:19.629 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-prometheus-alerts noarch 2:17.0.0-681.gc1ea6241.el8 ceph-noarch 8.9 k 2021-02-10T04:18:19.629 INFO:teuthology.orchestra.run.smithi085.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2021-02-10T04:18:19.629 INFO:teuthology.orchestra.run.smithi085.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-02-10T04:18:19.629 INFO:teuthology.orchestra.run.smithi085.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-02-10T04:18:19.629 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:18:19.629 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction Summary 2021-02-10T04:18:19.629 INFO:teuthology.orchestra.run.smithi085.stdout:================================================================================================== 2021-02-10T04:18:19.630 INFO:teuthology.orchestra.run.smithi085.stdout:Install 6 Packages 2021-02-10T04:18:19.630 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:18:19.630 INFO:teuthology.orchestra.run.smithi085.stdout:Total download size: 2.9 M 2021-02-10T04:18:19.630 INFO:teuthology.orchestra.run.smithi085.stdout:Installed size: 15 M 2021-02-10T04:18:19.630 INFO:teuthology.orchestra.run.smithi085.stdout:Downloading Packages: 2021-02-10T04:18:19.741 INFO:teuthology.orchestra.run.smithi085.stdout:(1/6): ceph-prometheus-alerts-17.0.0-681.gc1ea6 81 kB/s | 8.9 kB 00:00 2021-02-10T04:18:19.757 INFO:teuthology.orchestra.run.smithi085.stdout:(2/6): ceph-grafana-dashboards-17.0.0-681.gc1ea 191 kB/s | 24 kB 00:00 2021-02-10T04:18:19.791 INFO:teuthology.orchestra.run.smithi085.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 674 kB/s | 34 kB 00:00 2021-02-10T04:18:19.823 INFO:teuthology.orchestra.run.smithi085.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.3 MB/s | 43 kB 00:00 2021-02-10T04:18:19.837 INFO:teuthology.orchestra.run.smithi085.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 2.4 MB/s | 196 kB 00:00 2021-02-10T04:18:20.135 INFO:teuthology.orchestra.run.smithi085.stdout:(6/6): ceph-mgr-dashboard-17.0.0-681.gc1ea6241. 5.2 MB/s | 2.6 MB 00:00 2021-02-10T04:18:20.137 INFO:teuthology.orchestra.run.smithi085.stdout:-------------------------------------------------------------------------------- 2021-02-10T04:18:20.138 INFO:teuthology.orchestra.run.smithi085.stdout:Total 5.8 MB/s | 2.9 MB 00:00 2021-02-10T04:18:20.139 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction check 2021-02-10T04:18:20.148 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction check succeeded. 2021-02-10T04:18:20.148 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction test 2021-02-10T04:18:20.233 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction test succeeded. 2021-02-10T04:18:20.236 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction 2021-02-10T04:18:20.531 INFO:teuthology.orchestra.run.smithi085.stdout: Preparing : 1/1 2021-02-10T04:18:20.646 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2021-02-10T04:18:20.646 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2021-02-10T04:18:20.646 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2021-02-10T04:18:20.647 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2021-02-10T04:18:20.647 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2021-02-10T04:18:20.647 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 163 k 2021-02-10T04:18:20.647 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:18:20.647 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2021-02-10T04:18:20.647 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2021-02-10T04:18:20.647 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2021-02-10T04:18:20.647 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:18:20.647 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 163 k 2021-02-10T04:18:20.647 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 587 k 2021-02-10T04:18:20.647 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2021-02-10T04:18:20.764 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2021-02-10T04:18:20.818 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-immutable-object-cache-17.0.0-681.gc1ea624 957 kB/s | 163 kB 00:00 2021-02-10T04:18:20.818 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2021-02-10T04:18:20.819 INFO:teuthology.orchestra.run.smithi164.stdout:Total 948 kB/s | 163 kB 00:00 2021-02-10T04:18:20.819 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2021-02-10T04:18:20.831 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2021-02-10T04:18:20.831 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2021-02-10T04:18:20.883 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2021-02-10T04:18:20.887 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2021-02-10T04:18:20.962 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2021-02-10T04:18:21.002 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2021-02-10T04:18:21.193 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-681.gc1ea6241.e 1/1 2021-02-10T04:18:21.214 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2021-02-10T04:18:21.326 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-681.gc1ea6241.el8.no 4/6 2021-02-10T04:18:21.552 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-681.gc1ea6241.e 1/1 2021-02-10T04:18:21.990 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-681.gc1ea6241.e 1/1 2021-02-10T04:18:21.990 INFO:teuthology.orchestra.run.smithi164.stdout:Installed products updated. 2021-02-10T04:18:22.378 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-681.gc1ea6241.el8.n 5/6 2021-02-10T04:18:22.680 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-681.gc1ea6241.el8.noarch 6/6 2021-02-10T04:18:22.893 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-681.gc1ea6241.el8.noarch 6/6 2021-02-10T04:18:22.893 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-681.gc1ea6241.el8.n 1/6 2021-02-10T04:18:22.893 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-681.gc1ea6241.el8.noarch 2/6 2021-02-10T04:18:22.893 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-681.gc1ea6241.el8.no 3/6 2021-02-10T04:18:22.893 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2021-02-10T04:18:22.894 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2021-02-10T04:18:23.391 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2021-02-10T04:18:23.391 INFO:teuthology.orchestra.run.smithi085.stdout:Installed products updated. 2021-02-10T04:18:23.835 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:18:23.835 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2021-02-10T04:18:23.836 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-immutable-object-cache-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:18:23.836 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:18:23.836 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2021-02-10T04:18:23.943 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-mgr 2021-02-10T04:18:24.225 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2021-02-10T04:18:24.702 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 84 kB/s | 2.4 kB 00:00 2021-02-10T04:18:24.819 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 81 kB/s | 2.8 kB 00:00 2021-02-10T04:18:25.292 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:18:25.292 INFO:teuthology.orchestra.run.smithi085.stdout:Installed: 2021-02-10T04:18:25.292 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-grafana-dashboards-2:17.0.0-681.gc1ea6241.el8.noarch 2021-02-10T04:18:25.292 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-mgr-dashboard-2:17.0.0-681.gc1ea6241.el8.noarch 2021-02-10T04:18:25.292 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-prometheus-alerts-2:17.0.0-681.gc1ea6241.el8.noarch 2021-02-10T04:18:25.292 INFO:teuthology.orchestra.run.smithi085.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-02-10T04:18:25.292 INFO:teuthology.orchestra.run.smithi085.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-02-10T04:18:25.292 INFO:teuthology.orchestra.run.smithi085.stdout: python3-routes-2.4.1-12.el8.noarch 2021-02-10T04:18:25.292 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:18:25.292 INFO:teuthology.orchestra.run.smithi085.stdout:Complete! 2021-02-10T04:18:25.411 DEBUG:teuthology.orchestra.run.smithi085:> sudo yum -y install ceph-mgr-diskprediction-local 2021-02-10T04:18:25.692 INFO:teuthology.orchestra.run.smithi085.stdout:Updating Subscription Management repositories. 2021-02-10T04:18:26.000 INFO:teuthology.orchestra.run.smithi164.stdout:Package ceph-mgr-2:17.0.0-681.gc1ea6241.el8.x86_64 is already installed. 2021-02-10T04:18:26.171 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 82 kB/s | 2.4 kB 00:00 2021-02-10T04:18:27.301 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2021-02-10T04:18:27.301 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2021-02-10T04:18:27.301 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2021-02-10T04:18:27.301 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-mgr-dashboard 2021-02-10T04:18:27.301 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 98 kB/s | 2.8 kB 00:00 2021-02-10T04:18:27.594 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2021-02-10T04:18:27.784 INFO:teuthology.orchestra.run.smithi085.stdout:Dependencies resolved. 2021-02-10T04:18:27.785 INFO:teuthology.orchestra.run.smithi085.stdout:====================================================================================================================== 2021-02-10T04:18:27.785 INFO:teuthology.orchestra.run.smithi085.stdout: Package Arch Version Repository Size 2021-02-10T04:18:27.785 INFO:teuthology.orchestra.run.smithi085.stdout:====================================================================================================================== 2021-02-10T04:18:27.785 INFO:teuthology.orchestra.run.smithi085.stdout:Installing: 2021-02-10T04:18:27.785 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-681.gc1ea6241.el8 ceph-noarch 8.6 M 2021-02-10T04:18:27.786 INFO:teuthology.orchestra.run.smithi085.stdout:Installing dependencies: 2021-02-10T04:18:27.786 INFO:teuthology.orchestra.run.smithi085.stdout: libgfortran x86_64 8.3.1-5.1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2021-02-10T04:18:27.786 INFO:teuthology.orchestra.run.smithi085.stdout: libquadmath x86_64 8.3.1-5.1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2021-02-10T04:18:27.786 INFO:teuthology.orchestra.run.smithi085.stdout: openblas x86_64 0.3.3-5.el8 rhel-8-for-x86_64-appstream-rpms 4.3 M 2021-02-10T04:18:27.786 INFO:teuthology.orchestra.run.smithi085.stdout: openblas-threads x86_64 0.3.3-5.el8 rhel-8-for-x86_64-appstream-rpms 4.4 M 2021-02-10T04:18:27.786 INFO:teuthology.orchestra.run.smithi085.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2021-02-10T04:18:27.786 INFO:teuthology.orchestra.run.smithi085.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2021-02-10T04:18:27.786 INFO:teuthology.orchestra.run.smithi085.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2021-02-10T04:18:27.786 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:18:27.786 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction Summary 2021-02-10T04:18:27.786 INFO:teuthology.orchestra.run.smithi085.stdout:====================================================================================================================== 2021-02-10T04:18:27.786 INFO:teuthology.orchestra.run.smithi085.stdout:Install 8 Packages 2021-02-10T04:18:27.786 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:18:27.787 INFO:teuthology.orchestra.run.smithi085.stdout:Total download size: 37 M 2021-02-10T04:18:27.787 INFO:teuthology.orchestra.run.smithi085.stdout:Installed size: 196 M 2021-02-10T04:18:27.787 INFO:teuthology.orchestra.run.smithi085.stdout:Downloading Packages: 2021-02-10T04:18:27.833 INFO:teuthology.orchestra.run.smithi085.stdout:(1/8): libgfortran-8.3.1-5.1.el8.x86_64.rpm 14 MB/s | 639 kB 00:00 2021-02-10T04:18:27.834 INFO:teuthology.orchestra.run.smithi085.stdout:(2/8): libquadmath-8.3.1-5.1.el8.x86_64.rpm 3.6 MB/s | 169 kB 00:00 2021-02-10T04:18:27.862 INFO:teuthology.orchestra.run.smithi085.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 8.0 MB/s | 225 kB 00:00 2021-02-10T04:18:27.917 INFO:teuthology.orchestra.run.smithi085.stdout:(4/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 68 MB/s | 3.7 MB 00:00 2021-02-10T04:18:28.058 INFO:teuthology.orchestra.run.smithi085.stdout:(5/8): python3-scipy-1.0.0-20.module+el8.1.0+33 65 MB/s | 15 MB 00:00 2021-02-10T04:18:28.084 INFO:teuthology.orchestra.run.smithi085.stdout:(6/8): openblas-threads-0.3.3-5.el8.x86_64.rpm 26 MB/s | 4.4 MB 00:00 2021-02-10T04:18:28.092 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 2021-02-10T04:18:28.107 INFO:teuthology.orchestra.run.smithi085.stdout:(7/8): openblas-0.3.3-5.el8.x86_64.rpm 87 MB/s | 4.3 MB 00:00 2021-02-10T04:18:28.202 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 97 kB/s | 2.8 kB 00:00 2021-02-10T04:18:28.449 INFO:teuthology.orchestra.run.smithi085.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-681 13 MB/s | 8.6 MB 00:00 2021-02-10T04:18:28.450 INFO:teuthology.orchestra.run.smithi085.stdout:-------------------------------------------------------------------------------- 2021-02-10T04:18:28.451 INFO:teuthology.orchestra.run.smithi085.stdout:Total 55 MB/s | 37 MB 00:00 2021-02-10T04:18:28.459 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction check 2021-02-10T04:18:28.478 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction check succeeded. 2021-02-10T04:18:28.478 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction test 2021-02-10T04:18:28.794 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction test succeeded. 2021-02-10T04:18:28.798 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction 2021-02-10T04:18:29.356 INFO:teuthology.orchestra.run.smithi085.stdout: Preparing : 1/1 2021-02-10T04:18:29.526 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-02-10T04:18:29.641 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-02-10T04:18:29.668 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2021-02-10T04:18:29.669 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================================== 2021-02-10T04:18:29.669 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2021-02-10T04:18:29.669 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================================== 2021-02-10T04:18:29.669 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2021-02-10T04:18:29.669 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-dashboard noarch 2:17.0.0-681.gc1ea6241.el8 ceph-noarch 2.6 M 2021-02-10T04:18:29.669 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2021-02-10T04:18:29.669 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-grafana-dashboards noarch 2:17.0.0-681.gc1ea6241.el8 ceph-noarch 24 k 2021-02-10T04:18:29.669 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-prometheus-alerts noarch 2:17.0.0-681.gc1ea6241.el8 ceph-noarch 8.9 k 2021-02-10T04:18:29.669 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2021-02-10T04:18:29.669 INFO:teuthology.orchestra.run.smithi164.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-02-10T04:18:29.669 INFO:teuthology.orchestra.run.smithi164.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-02-10T04:18:29.669 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:18:29.670 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2021-02-10T04:18:29.670 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================================== 2021-02-10T04:18:29.670 INFO:teuthology.orchestra.run.smithi164.stdout:Install 6 Packages 2021-02-10T04:18:29.670 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:18:29.670 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 2.9 M 2021-02-10T04:18:29.670 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 15 M 2021-02-10T04:18:29.670 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2021-02-10T04:18:29.810 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-02-10T04:18:29.943 INFO:teuthology.orchestra.run.smithi164.stdout:(1/6): ceph-prometheus-alerts-17.0.0-681.gc1ea6 33 kB/s | 8.9 kB 00:00 2021-02-10T04:18:29.993 INFO:teuthology.orchestra.run.smithi164.stdout:(2/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 672 kB/s | 34 kB 00:00 2021-02-10T04:18:29.993 INFO:teuthology.orchestra.run.smithi164.stdout:(3/6): ceph-grafana-dashboards-17.0.0-681.gc1ea 75 kB/s | 24 kB 00:00 2021-02-10T04:18:30.026 INFO:teuthology.orchestra.run.smithi164.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.4 MB/s | 43 kB 00:00 2021-02-10T04:18:30.042 INFO:teuthology.orchestra.run.smithi164.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 3.9 MB/s | 196 kB 00:00 2021-02-10T04:18:30.302 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-02-10T04:18:30.559 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : openblas-threads-0.3.3-5.el8.x86_64 3/8 2021-02-10T04:18:31.062 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: openblas-threads-0.3.3-5.el8.x86_64 3/8 2021-02-10T04:18:31.236 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : openblas-0.3.3-5.el8.x86_64 4/8 2021-02-10T04:18:31.359 INFO:teuthology.orchestra.run.smithi164.stdout:(6/6): ceph-mgr-dashboard-17.0.0-681.gc1ea6241. 1.6 MB/s | 2.6 MB 00:01 2021-02-10T04:18:31.362 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2021-02-10T04:18:31.362 INFO:teuthology.orchestra.run.smithi164.stdout:Total 1.7 MB/s | 2.9 MB 00:01 2021-02-10T04:18:31.364 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2021-02-10T04:18:31.374 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2021-02-10T04:18:31.374 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2021-02-10T04:18:31.464 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2021-02-10T04:18:31.468 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2021-02-10T04:18:31.762 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2021-02-10T04:18:31.961 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: openblas-0.3.3-5.el8.x86_64 4/8 2021-02-10T04:18:31.986 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2021-02-10T04:18:32.186 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2021-02-10T04:18:32.335 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-02-10T04:18:32.402 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2021-02-10T04:18:32.499 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-681.gc1ea6241.el8.no 4/6 2021-02-10T04:18:32.551 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-02-10T04:18:33.527 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-681.gc1ea6241.el8.n 5/6 2021-02-10T04:18:33.777 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-681.gc1ea6241.el8.noarch 6/6 2021-02-10T04:18:33.999 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-681.gc1ea6241.el8.noarch 6/6 2021-02-10T04:18:34.000 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-681.gc1ea6241.el8.n 1/6 2021-02-10T04:18:34.000 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-681.gc1ea6241.el8.noarch 2/6 2021-02-10T04:18:34.000 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-681.gc1ea6241.el8.no 3/6 2021-02-10T04:18:34.000 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2021-02-10T04:18:34.000 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2021-02-10T04:18:34.449 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2021-02-10T04:18:34.449 INFO:teuthology.orchestra.run.smithi164.stdout:Installed products updated. 2021-02-10T04:18:34.572 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-02-10T04:18:35.947 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 7/8 2021-02-10T04:18:36.112 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-681.gc1ea6241 8/8 2021-02-10T04:18:36.315 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:18:36.316 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2021-02-10T04:18:36.316 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-grafana-dashboards-2:17.0.0-681.gc1ea6241.el8.noarch 2021-02-10T04:18:36.316 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-dashboard-2:17.0.0-681.gc1ea6241.el8.noarch 2021-02-10T04:18:36.316 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-prometheus-alerts-2:17.0.0-681.gc1ea6241.el8.noarch 2021-02-10T04:18:36.316 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-02-10T04:18:36.316 INFO:teuthology.orchestra.run.smithi164.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-02-10T04:18:36.316 INFO:teuthology.orchestra.run.smithi164.stdout: python3-routes-2.4.1-12.el8.noarch 2021-02-10T04:18:36.316 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:18:36.316 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2021-02-10T04:18:36.398 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-681.gc1ea6241 8/8 2021-02-10T04:18:36.398 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-681.gc1ea6241 1/8 2021-02-10T04:18:36.398 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : libquadmath-8.3.1-5.1.el8.x86_64 2/8 2021-02-10T04:18:36.398 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : libgfortran-8.3.1-5.1.el8.x86_64 3/8 2021-02-10T04:18:36.398 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 4/8 2021-02-10T04:18:36.398 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-02-10T04:18:36.398 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 6/8 2021-02-10T04:18:36.398 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : openblas-threads-0.3.3-5.el8.x86_64 7/8 2021-02-10T04:18:36.441 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-mgr-diskprediction-local 2021-02-10T04:18:36.722 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2021-02-10T04:18:36.890 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : openblas-0.3.3-5.el8.x86_64 8/8 2021-02-10T04:18:36.890 INFO:teuthology.orchestra.run.smithi085.stdout:Installed products updated. 2021-02-10T04:18:37.217 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 80 kB/s | 2.4 kB 00:00 2021-02-10T04:18:37.327 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 95 kB/s | 2.8 kB 00:00 2021-02-10T04:18:38.797 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2021-02-10T04:18:38.798 INFO:teuthology.orchestra.run.smithi164.stdout:====================================================================================================================== 2021-02-10T04:18:38.798 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2021-02-10T04:18:38.798 INFO:teuthology.orchestra.run.smithi164.stdout:====================================================================================================================== 2021-02-10T04:18:38.798 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2021-02-10T04:18:38.798 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-681.gc1ea6241.el8 ceph-noarch 8.6 M 2021-02-10T04:18:38.798 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2021-02-10T04:18:38.798 INFO:teuthology.orchestra.run.smithi164.stdout: libgfortran x86_64 8.3.1-5.1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2021-02-10T04:18:38.799 INFO:teuthology.orchestra.run.smithi164.stdout: libquadmath x86_64 8.3.1-5.1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2021-02-10T04:18:38.799 INFO:teuthology.orchestra.run.smithi164.stdout: openblas x86_64 0.3.3-5.el8 rhel-8-for-x86_64-appstream-rpms 4.3 M 2021-02-10T04:18:38.799 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-threads x86_64 0.3.3-5.el8 rhel-8-for-x86_64-appstream-rpms 4.4 M 2021-02-10T04:18:38.799 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2021-02-10T04:18:38.799 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2021-02-10T04:18:38.799 INFO:teuthology.orchestra.run.smithi164.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2021-02-10T04:18:38.799 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:18:38.799 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2021-02-10T04:18:38.799 INFO:teuthology.orchestra.run.smithi164.stdout:====================================================================================================================== 2021-02-10T04:18:38.799 INFO:teuthology.orchestra.run.smithi164.stdout:Install 8 Packages 2021-02-10T04:18:38.799 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:18:38.800 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 37 M 2021-02-10T04:18:38.800 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 196 M 2021-02-10T04:18:38.800 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2021-02-10T04:18:38.822 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:18:38.822 INFO:teuthology.orchestra.run.smithi085.stdout:Installed: 2021-02-10T04:18:38.822 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-mgr-diskprediction-local-2:17.0.0-681.gc1ea6241.el8.noarch 2021-02-10T04:18:38.822 INFO:teuthology.orchestra.run.smithi085.stdout: libgfortran-8.3.1-5.1.el8.x86_64 2021-02-10T04:18:38.822 INFO:teuthology.orchestra.run.smithi085.stdout: libquadmath-8.3.1-5.1.el8.x86_64 2021-02-10T04:18:38.822 INFO:teuthology.orchestra.run.smithi085.stdout: openblas-0.3.3-5.el8.x86_64 2021-02-10T04:18:38.822 INFO:teuthology.orchestra.run.smithi085.stdout: openblas-threads-0.3.3-5.el8.x86_64 2021-02-10T04:18:38.822 INFO:teuthology.orchestra.run.smithi085.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-02-10T04:18:38.823 INFO:teuthology.orchestra.run.smithi085.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-02-10T04:18:38.823 INFO:teuthology.orchestra.run.smithi085.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2021-02-10T04:18:38.823 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:18:38.823 INFO:teuthology.orchestra.run.smithi085.stdout:Complete! 2021-02-10T04:18:38.839 INFO:teuthology.orchestra.run.smithi164.stdout:(1/8): libquadmath-8.3.1-5.1.el8.x86_64.rpm 4.2 MB/s | 169 kB 00:00 2021-02-10T04:18:38.850 INFO:teuthology.orchestra.run.smithi164.stdout:(2/8): libgfortran-8.3.1-5.1.el8.x86_64.rpm 13 MB/s | 639 kB 00:00 2021-02-10T04:18:38.876 INFO:teuthology.orchestra.run.smithi164.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 10 MB/s | 225 kB 00:00 2021-02-10T04:18:38.920 DEBUG:teuthology.orchestra.run.smithi085:> sudo yum -y install ceph-mgr-rook 2021-02-10T04:18:38.924 INFO:teuthology.orchestra.run.smithi164.stdout:(4/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 78 MB/s | 3.7 MB 00:00 2021-02-10T04:18:39.065 INFO:teuthology.orchestra.run.smithi164.stdout:(5/8): python3-scipy-1.0.0-20.module+el8.1.0+33 64 MB/s | 15 MB 00:00 2021-02-10T04:18:39.078 INFO:teuthology.orchestra.run.smithi164.stdout:(6/8): openblas-threads-0.3.3-5.el8.x86_64.rpm 28 MB/s | 4.4 MB 00:00 2021-02-10T04:18:39.120 INFO:teuthology.orchestra.run.smithi164.stdout:(7/8): openblas-0.3.3-5.el8.x86_64.rpm 79 MB/s | 4.3 MB 00:00 2021-02-10T04:18:39.202 INFO:teuthology.orchestra.run.smithi085.stdout:Updating Subscription Management repositories. 2021-02-10T04:18:39.696 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 80 kB/s | 2.4 kB 00:00 2021-02-10T04:18:40.191 INFO:teuthology.orchestra.run.smithi164.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-681 7.5 MB/s | 8.6 MB 00:01 2021-02-10T04:18:40.191 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2021-02-10T04:18:40.191 INFO:teuthology.orchestra.run.smithi164.stdout:Total 32 MB/s | 37 MB 00:01 2021-02-10T04:18:40.191 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2021-02-10T04:18:40.191 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2021-02-10T04:18:40.191 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2021-02-10T04:18:40.191 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 93 kB/s | 2.8 kB 00:00 2021-02-10T04:18:40.292 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2021-02-10T04:18:40.296 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2021-02-10T04:18:40.871 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2021-02-10T04:18:41.023 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-02-10T04:18:41.139 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-02-10T04:18:41.332 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-02-10T04:18:41.494 INFO:teuthology.orchestra.run.smithi085.stdout:Dependencies resolved. 2021-02-10T04:18:41.496 INFO:teuthology.orchestra.run.smithi085.stdout:======================================================================================================= 2021-02-10T04:18:41.496 INFO:teuthology.orchestra.run.smithi085.stdout: Package Arch Version Repository Size 2021-02-10T04:18:41.496 INFO:teuthology.orchestra.run.smithi085.stdout:======================================================================================================= 2021-02-10T04:18:41.496 INFO:teuthology.orchestra.run.smithi085.stdout:Installing: 2021-02-10T04:18:41.497 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-mgr-rook noarch 2:17.0.0-681.gc1ea6241.el8 ceph-noarch 28 k 2021-02-10T04:18:41.497 INFO:teuthology.orchestra.run.smithi085.stdout:Installing dependencies: 2021-02-10T04:18:41.497 INFO:teuthology.orchestra.run.smithi085.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-02-10T04:18:41.497 INFO:teuthology.orchestra.run.smithi085.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-02-10T04:18:41.497 INFO:teuthology.orchestra.run.smithi085.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-02-10T04:18:41.497 INFO:teuthology.orchestra.run.smithi085.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2021-02-10T04:18:41.497 INFO:teuthology.orchestra.run.smithi085.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2021-02-10T04:18:41.497 INFO:teuthology.orchestra.run.smithi085.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-02-10T04:18:41.497 INFO:teuthology.orchestra.run.smithi085.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2021-02-10T04:18:41.497 INFO:teuthology.orchestra.run.smithi085.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2021-02-10T04:18:41.497 INFO:teuthology.orchestra.run.smithi085.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2021-02-10T04:18:41.497 INFO:teuthology.orchestra.run.smithi085.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2021-02-10T04:18:41.498 INFO:teuthology.orchestra.run.smithi085.stdout: python3-rsa noarch 3.4.2-16.el8 epel 73 k 2021-02-10T04:18:41.498 INFO:teuthology.orchestra.run.smithi085.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-02-10T04:18:41.498 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:18:41.498 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction Summary 2021-02-10T04:18:41.498 INFO:teuthology.orchestra.run.smithi085.stdout:======================================================================================================= 2021-02-10T04:18:41.498 INFO:teuthology.orchestra.run.smithi085.stdout:Install 13 Packages 2021-02-10T04:18:41.498 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:18:41.498 INFO:teuthology.orchestra.run.smithi085.stdout:Total download size: 1.9 M 2021-02-10T04:18:41.499 INFO:teuthology.orchestra.run.smithi085.stdout:Installed size: 24 M 2021-02-10T04:18:41.499 INFO:teuthology.orchestra.run.smithi085.stdout:Downloading Packages: 2021-02-10T04:18:41.559 INFO:teuthology.orchestra.run.smithi085.stdout:(1/13): python3-certifi-2018.10.15-7.el8.noarch 272 kB/s | 16 kB 00:00 2021-02-10T04:18:41.559 INFO:teuthology.orchestra.run.smithi085.stdout:(2/13): python3-cachetools-3.1.1-4.el8.noarch.r 547 kB/s | 33 kB 00:00 2021-02-10T04:18:41.598 INFO:teuthology.orchestra.run.smithi085.stdout:(3/13): python3-google-auth-1.1.1-10.el8.noarch 2.0 MB/s | 81 kB 00:00 2021-02-10T04:18:41.618 INFO:teuthology.orchestra.run.smithi085.stdout:(4/13): python3-rsa-3.4.2-16.el8.noarch.rpm 3.6 MB/s | 73 kB 00:00 2021-02-10T04:18:41.638 INFO:teuthology.orchestra.run.smithi085.stdout:(5/13): python3-websocket-client-0.56.0-5.el8.n 3.0 MB/s | 61 kB 00:00 2021-02-10T04:18:41.670 INFO:teuthology.orchestra.run.smithi085.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.3 MB/s | 43 kB 00:00 2021-02-10T04:18:41.691 INFO:teuthology.orchestra.run.smithi085.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 7.2 MB/s | 155 kB 00:00 2021-02-10T04:18:41.700 INFO:teuthology.orchestra.run.smithi085.stdout:(8/13): python3-kubernetes-11.0.0-6.el8.noarch. 8.3 MB/s | 1.2 MB 00:00 2021-02-10T04:18:41.714 INFO:teuthology.orchestra.run.smithi085.stdout:(9/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 1.1 MB/s | 27 kB 00:00 2021-02-10T04:18:41.738 INFO:teuthology.orchestra.run.smithi085.stdout:(10/13): python3-jsonpointer-1.10-11.el8.noarch 565 kB/s | 20 kB 00:00 2021-02-10T04:18:41.744 INFO:teuthology.orchestra.run.smithi085.stdout:(11/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 4.1 MB/s | 126 kB 00:00 2021-02-10T04:18:41.757 INFO:teuthology.orchestra.run.smithi085.stdout:(12/13): python3-pyasn1-modules-0.3.7-6.el8.noa 5.4 MB/s | 110 kB 00:00 2021-02-10T04:18:41.827 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-02-10T04:18:41.829 INFO:teuthology.orchestra.run.smithi085.stdout:(13/13): ceph-mgr-rook-17.0.0-681.gc1ea6241.el8 85 kB/s | 28 kB 00:00 2021-02-10T04:18:41.829 INFO:teuthology.orchestra.run.smithi085.stdout:-------------------------------------------------------------------------------- 2021-02-10T04:18:41.830 INFO:teuthology.orchestra.run.smithi085.stdout:Total 5.8 MB/s | 1.9 MB 00:00 2021-02-10T04:18:41.835 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction check 2021-02-10T04:18:41.848 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction check succeeded. 2021-02-10T04:18:41.848 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction test 2021-02-10T04:18:41.918 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction test succeeded. 2021-02-10T04:18:41.922 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction 2021-02-10T04:18:42.014 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : openblas-threads-0.3.3-5.el8.x86_64 3/8 2021-02-10T04:18:42.412 INFO:teuthology.orchestra.run.smithi085.stdout: Preparing : 1/1 2021-02-10T04:18:42.504 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: openblas-threads-0.3.3-5.el8.x86_64 3/8 2021-02-10T04:18:42.660 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-02-10T04:18:42.681 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : openblas-0.3.3-5.el8.x86_64 4/8 2021-02-10T04:18:42.940 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-rsa-3.4.2-16.el8.noarch 2/13 2021-02-10T04:18:43.190 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2021-02-10T04:18:43.382 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/13 2021-02-10T04:18:43.406 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: openblas-0.3.3-5.el8.x86_64 4/8 2021-02-10T04:18:43.646 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/13 2021-02-10T04:18:43.750 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-02-10T04:18:43.918 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-02-10T04:18:43.956 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-02-10T04:18:44.155 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-02-10T04:18:44.365 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2021-02-10T04:18:44.533 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-02-10T04:18:44.760 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2021-02-10T04:18:45.979 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2021-02-10T04:18:46.013 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-02-10T04:18:46.273 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2021-02-10T04:18:46.487 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : ceph-mgr-rook-2:17.0.0-681.gc1ea6241.el8.noarch 13/13 2021-02-10T04:18:46.750 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-681.gc1ea6241.el8.noarch 13/13 2021-02-10T04:18:46.750 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : ceph-mgr-rook-2:17.0.0-681.gc1ea6241.el8.noarch 1/13 2021-02-10T04:18:46.750 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2021-02-10T04:18:46.750 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2021-02-10T04:18:46.750 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2021-02-10T04:18:46.750 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2021-02-10T04:18:46.750 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-rsa-3.4.2-16.el8.noarch 6/13 2021-02-10T04:18:46.751 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2021-02-10T04:18:46.751 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 8/13 2021-02-10T04:18:46.751 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 9/13 2021-02-10T04:18:46.751 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 10/13 2021-02-10T04:18:46.751 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-02-10T04:18:46.751 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 12/13 2021-02-10T04:18:47.281 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 13/13 2021-02-10T04:18:47.281 INFO:teuthology.orchestra.run.smithi085.stdout:Installed products updated. 2021-02-10T04:18:47.560 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 7/8 2021-02-10T04:18:47.739 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-681.gc1ea6241 8/8 2021-02-10T04:18:47.972 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-681.gc1ea6241 8/8 2021-02-10T04:18:47.972 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-681.gc1ea6241 1/8 2021-02-10T04:18:47.972 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libquadmath-8.3.1-5.1.el8.x86_64 2/8 2021-02-10T04:18:47.972 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libgfortran-8.3.1-5.1.el8.x86_64 3/8 2021-02-10T04:18:47.972 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 4/8 2021-02-10T04:18:47.972 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-02-10T04:18:47.973 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 6/8 2021-02-10T04:18:47.973 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openblas-threads-0.3.3-5.el8.x86_64 7/8 2021-02-10T04:18:48.483 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openblas-0.3.3-5.el8.x86_64 8/8 2021-02-10T04:18:48.484 INFO:teuthology.orchestra.run.smithi164.stdout:Installed products updated. 2021-02-10T04:18:49.080 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:18:49.080 INFO:teuthology.orchestra.run.smithi085.stdout:Installed: 2021-02-10T04:18:49.080 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-mgr-rook-2:17.0.0-681.gc1ea6241.el8.noarch 2021-02-10T04:18:49.080 INFO:teuthology.orchestra.run.smithi085.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-02-10T04:18:49.081 INFO:teuthology.orchestra.run.smithi085.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-02-10T04:18:49.081 INFO:teuthology.orchestra.run.smithi085.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-02-10T04:18:49.081 INFO:teuthology.orchestra.run.smithi085.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-02-10T04:18:49.081 INFO:teuthology.orchestra.run.smithi085.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-02-10T04:18:49.081 INFO:teuthology.orchestra.run.smithi085.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-02-10T04:18:49.081 INFO:teuthology.orchestra.run.smithi085.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-02-10T04:18:49.081 INFO:teuthology.orchestra.run.smithi085.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-02-10T04:18:49.081 INFO:teuthology.orchestra.run.smithi085.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-02-10T04:18:49.081 INFO:teuthology.orchestra.run.smithi085.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-02-10T04:18:49.081 INFO:teuthology.orchestra.run.smithi085.stdout: python3-rsa-3.4.2-16.el8.noarch 2021-02-10T04:18:49.081 INFO:teuthology.orchestra.run.smithi085.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-02-10T04:18:49.081 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:18:49.081 INFO:teuthology.orchestra.run.smithi085.stdout:Complete! 2021-02-10T04:18:49.202 DEBUG:teuthology.orchestra.run.smithi085:> sudo yum -y install ceph-mgr-cephadm 2021-02-10T04:18:49.487 INFO:teuthology.orchestra.run.smithi085.stdout:Updating Subscription Management repositories. 2021-02-10T04:18:49.975 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 2021-02-10T04:18:50.087 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 92 kB/s | 2.8 kB 00:00 2021-02-10T04:18:50.294 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:18:50.294 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2021-02-10T04:18:50.294 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-diskprediction-local-2:17.0.0-681.gc1ea6241.el8.noarch 2021-02-10T04:18:50.294 INFO:teuthology.orchestra.run.smithi164.stdout: libgfortran-8.3.1-5.1.el8.x86_64 2021-02-10T04:18:50.295 INFO:teuthology.orchestra.run.smithi164.stdout: libquadmath-8.3.1-5.1.el8.x86_64 2021-02-10T04:18:50.295 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-0.3.3-5.el8.x86_64 2021-02-10T04:18:50.295 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-threads-0.3.3-5.el8.x86_64 2021-02-10T04:18:50.295 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-02-10T04:18:50.295 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-02-10T04:18:50.295 INFO:teuthology.orchestra.run.smithi164.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2021-02-10T04:18:50.295 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:18:50.295 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2021-02-10T04:18:50.372 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-mgr-rook 2021-02-10T04:18:50.654 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2021-02-10T04:18:51.125 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 85 kB/s | 2.4 kB 00:00 2021-02-10T04:18:51.237 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 95 kB/s | 2.8 kB 00:00 2021-02-10T04:18:51.568 INFO:teuthology.orchestra.run.smithi085.stdout:Dependencies resolved. 2021-02-10T04:18:51.569 INFO:teuthology.orchestra.run.smithi085.stdout:============================================================================================== 2021-02-10T04:18:51.569 INFO:teuthology.orchestra.run.smithi085.stdout: Package Arch Version Repository Size 2021-02-10T04:18:51.569 INFO:teuthology.orchestra.run.smithi085.stdout:============================================================================================== 2021-02-10T04:18:51.569 INFO:teuthology.orchestra.run.smithi085.stdout:Installing: 2021-02-10T04:18:51.569 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-mgr-cephadm noarch 2:17.0.0-681.gc1ea6241.el8 ceph-noarch 79 k 2021-02-10T04:18:51.569 INFO:teuthology.orchestra.run.smithi085.stdout:Installing dependencies: 2021-02-10T04:18:51.569 INFO:teuthology.orchestra.run.smithi085.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2021-02-10T04:18:51.569 INFO:teuthology.orchestra.run.smithi085.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2021-02-10T04:18:51.569 INFO:teuthology.orchestra.run.smithi085.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2021-02-10T04:18:51.569 INFO:teuthology.orchestra.run.smithi085.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2021-02-10T04:18:51.569 INFO:teuthology.orchestra.run.smithi085.stdout: python3-remoto noarch 1.1.4-4.el8 epel 48 k 2021-02-10T04:18:51.569 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:18:51.570 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction Summary 2021-02-10T04:18:51.570 INFO:teuthology.orchestra.run.smithi085.stdout:============================================================================================== 2021-02-10T04:18:51.570 INFO:teuthology.orchestra.run.smithi085.stdout:Install 6 Packages 2021-02-10T04:18:51.570 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:18:51.570 INFO:teuthology.orchestra.run.smithi085.stdout:Total download size: 5.8 M 2021-02-10T04:18:51.570 INFO:teuthology.orchestra.run.smithi085.stdout:Installed size: 24 M 2021-02-10T04:18:51.570 INFO:teuthology.orchestra.run.smithi085.stdout:Downloading Packages: 2021-02-10T04:18:51.630 INFO:teuthology.orchestra.run.smithi085.stdout:(1/6): python3-apipkg-1.5-6.el8.noarch.rpm 357 kB/s | 21 kB 00:00 2021-02-10T04:18:51.669 INFO:teuthology.orchestra.run.smithi085.stdout:(2/6): python3-remoto-1.1.4-4.el8.noarch.rpm 1.2 MB/s | 48 kB 00:00 2021-02-10T04:18:51.690 INFO:teuthology.orchestra.run.smithi085.stdout:(3/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.9 MB/s | 355 kB 00:00 2021-02-10T04:18:51.706 INFO:teuthology.orchestra.run.smithi085.stdout:(4/6): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 14 MB/s | 538 kB 00:00 2021-02-10T04:18:51.751 INFO:teuthology.orchestra.run.smithi085.stdout:(5/6): python3-babel-2.5.1-5.el8.noarch.rpm 78 MB/s | 4.8 MB 00:00 2021-02-10T04:18:51.804 INFO:teuthology.orchestra.run.smithi085.stdout:(6/6): ceph-mgr-cephadm-17.0.0-681.gc1ea6241.el 339 kB/s | 79 kB 00:00 2021-02-10T04:18:51.805 INFO:teuthology.orchestra.run.smithi085.stdout:-------------------------------------------------------------------------------- 2021-02-10T04:18:51.805 INFO:teuthology.orchestra.run.smithi085.stdout:Total 25 MB/s | 5.8 MB 00:00 2021-02-10T04:18:51.808 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction check 2021-02-10T04:18:51.817 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction check succeeded. 2021-02-10T04:18:51.817 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction test 2021-02-10T04:18:51.894 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction test succeeded. 2021-02-10T04:18:51.898 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction 2021-02-10T04:18:52.695 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2021-02-10T04:18:52.697 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================================= 2021-02-10T04:18:52.697 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2021-02-10T04:18:52.697 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================================= 2021-02-10T04:18:52.697 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2021-02-10T04:18:52.697 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-rook noarch 2:17.0.0-681.gc1ea6241.el8 ceph-noarch 28 k 2021-02-10T04:18:52.697 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2021-02-10T04:18:52.697 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-02-10T04:18:52.697 INFO:teuthology.orchestra.run.smithi164.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-02-10T04:18:52.697 INFO:teuthology.orchestra.run.smithi164.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-02-10T04:18:52.697 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2021-02-10T04:18:52.698 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2021-02-10T04:18:52.698 INFO:teuthology.orchestra.run.smithi164.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-02-10T04:18:52.698 INFO:teuthology.orchestra.run.smithi164.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2021-02-10T04:18:52.698 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2021-02-10T04:18:52.698 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2021-02-10T04:18:52.698 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2021-02-10T04:18:52.698 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rsa noarch 3.4.2-16.el8 epel 73 k 2021-02-10T04:18:52.698 INFO:teuthology.orchestra.run.smithi164.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-02-10T04:18:52.698 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:18:52.698 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2021-02-10T04:18:52.698 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================================= 2021-02-10T04:18:52.698 INFO:teuthology.orchestra.run.smithi164.stdout:Install 13 Packages 2021-02-10T04:18:52.698 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:18:52.699 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 1.9 M 2021-02-10T04:18:52.699 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 24 M 2021-02-10T04:18:52.699 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2021-02-10T04:18:52.749 INFO:teuthology.orchestra.run.smithi164.stdout:(1/13): python3-certifi-2018.10.15-7.el8.noarch 334 kB/s | 16 kB 00:00 2021-02-10T04:18:52.751 INFO:teuthology.orchestra.run.smithi164.stdout:(2/13): python3-cachetools-3.1.1-4.el8.noarch.r 653 kB/s | 33 kB 00:00 2021-02-10T04:18:52.781 INFO:teuthology.orchestra.run.smithi164.stdout:(3/13): python3-google-auth-1.1.1-10.el8.noarch 2.4 MB/s | 81 kB 00:00 2021-02-10T04:18:52.798 INFO:teuthology.orchestra.run.smithi164.stdout:(4/13): python3-rsa-3.4.2-16.el8.noarch.rpm 4.3 MB/s | 73 kB 00:00 2021-02-10T04:18:52.815 INFO:teuthology.orchestra.run.smithi164.stdout:(5/13): python3-websocket-client-0.56.0-5.el8.n 3.5 MB/s | 61 kB 00:00 2021-02-10T04:18:52.825 INFO:teuthology.orchestra.run.smithi164.stdout:(6/13): ceph-mgr-rook-17.0.0-681.gc1ea6241.el8. 223 kB/s | 28 kB 00:00 2021-02-10T04:18:52.852 INFO:teuthology.orchestra.run.smithi164.stdout:(7/13): python3-kubernetes-11.0.0-6.el8.noarch. 11 MB/s | 1.2 MB 00:00 2021-02-10T04:18:52.858 INFO:teuthology.orchestra.run.smithi164.stdout:(8/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.0 MB/s | 43 kB 00:00 2021-02-10T04:18:52.865 INFO:teuthology.orchestra.run.smithi164.stdout:(9/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 3.8 MB/s | 155 kB 00:00 2021-02-10T04:18:52.893 INFO:teuthology.orchestra.run.smithi164.stdout:(10/13): python3-jsonpointer-1.10-11.el8.noarch 573 kB/s | 20 kB 00:00 2021-02-10T04:18:52.895 INFO:teuthology.orchestra.run.smithi164.stdout:(11/13): python3-jsonpatch-1.21-2.el8.noarch.rp 688 kB/s | 27 kB 00:00 2021-02-10T04:18:52.900 INFO:teuthology.orchestra.run.smithi164.stdout:(12/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 4.0 MB/s | 126 kB 00:00 2021-02-10T04:18:52.914 INFO:teuthology.orchestra.run.smithi164.stdout:(13/13): python3-pyasn1-modules-0.3.7-6.el8.noa 5.0 MB/s | 110 kB 00:00 2021-02-10T04:18:52.915 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2021-02-10T04:18:52.915 INFO:teuthology.orchestra.run.smithi164.stdout:Total 8.9 MB/s | 1.9 MB 00:00 2021-02-10T04:18:52.920 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2021-02-10T04:18:52.934 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2021-02-10T04:18:52.934 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2021-02-10T04:18:52.952 INFO:teuthology.orchestra.run.smithi085.stdout: Preparing : 1/1 2021-02-10T04:18:53.004 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2021-02-10T04:18:53.008 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2021-02-10T04:18:53.358 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 1/6 2021-02-10T04:18:53.506 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2021-02-10T04:18:53.621 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 2/6 2021-02-10T04:18:53.764 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-02-10T04:18:53.907 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 3/6 2021-02-10T04:18:54.001 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rsa-3.4.2-16.el8.noarch 2/13 2021-02-10T04:18:54.230 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 4/6 2021-02-10T04:18:54.236 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2021-02-10T04:18:54.395 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/13 2021-02-10T04:18:54.496 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 5/6 2021-02-10T04:18:54.666 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/13 2021-02-10T04:18:54.706 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-681.gc1ea6241.el8.noarch 6/6 2021-02-10T04:18:54.903 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-02-10T04:18:54.921 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-681.gc1ea6241.el8.noarch 6/6 2021-02-10T04:18:54.922 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-681.gc1ea6241.el8.noarch 1/6 2021-02-10T04:18:54.922 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 2/6 2021-02-10T04:18:54.922 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 3/6 2021-02-10T04:18:54.922 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 4/6 2021-02-10T04:18:54.922 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 5/6 2021-02-10T04:18:55.117 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-02-10T04:18:55.320 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2021-02-10T04:18:55.440 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 6/6 2021-02-10T04:18:55.440 INFO:teuthology.orchestra.run.smithi085.stdout:Installed products updated. 2021-02-10T04:18:55.486 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-02-10T04:18:55.728 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2021-02-10T04:18:56.880 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2021-02-10T04:18:57.113 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2021-02-10T04:18:57.301 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:18:57.301 INFO:teuthology.orchestra.run.smithi085.stdout:Installed: 2021-02-10T04:18:57.301 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-mgr-cephadm-2:17.0.0-681.gc1ea6241.el8.noarch 2021-02-10T04:18:57.301 INFO:teuthology.orchestra.run.smithi085.stdout: python3-apipkg-1.5-6.el8.noarch 2021-02-10T04:18:57.301 INFO:teuthology.orchestra.run.smithi085.stdout: python3-babel-2.5.1-5.el8.noarch 2021-02-10T04:18:57.301 INFO:teuthology.orchestra.run.smithi085.stdout: python3-execnet-1.7.1-1.el8.noarch 2021-02-10T04:18:57.301 INFO:teuthology.orchestra.run.smithi085.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2021-02-10T04:18:57.301 INFO:teuthology.orchestra.run.smithi085.stdout: python3-remoto-1.1.4-4.el8.noarch 2021-02-10T04:18:57.301 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:18:57.302 INFO:teuthology.orchestra.run.smithi085.stdout:Complete! 2021-02-10T04:18:57.310 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-rook-2:17.0.0-681.gc1ea6241.el8.noarch 13/13 2021-02-10T04:18:57.387 DEBUG:teuthology.orchestra.run.smithi085:> sudo yum -y install ceph-fuse 2021-02-10T04:18:57.534 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-681.gc1ea6241.el8.noarch 13/13 2021-02-10T04:18:57.535 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-rook-2:17.0.0-681.gc1ea6241.el8.noarch 1/13 2021-02-10T04:18:57.535 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2021-02-10T04:18:57.535 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2021-02-10T04:18:57.535 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2021-02-10T04:18:57.535 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2021-02-10T04:18:57.535 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rsa-3.4.2-16.el8.noarch 6/13 2021-02-10T04:18:57.535 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2021-02-10T04:18:57.535 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 8/13 2021-02-10T04:18:57.535 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 9/13 2021-02-10T04:18:57.535 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 10/13 2021-02-10T04:18:57.535 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-02-10T04:18:57.536 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 12/13 2021-02-10T04:18:57.675 INFO:teuthology.orchestra.run.smithi085.stdout:Updating Subscription Management repositories. 2021-02-10T04:18:58.059 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 13/13 2021-02-10T04:18:58.059 INFO:teuthology.orchestra.run.smithi164.stdout:Installed products updated. 2021-02-10T04:18:58.162 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 79 kB/s | 2.4 kB 00:00 2021-02-10T04:18:58.271 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 94 kB/s | 2.8 kB 00:00 2021-02-10T04:18:59.770 INFO:teuthology.orchestra.run.smithi085.stdout:Dependencies resolved. 2021-02-10T04:18:59.771 INFO:teuthology.orchestra.run.smithi085.stdout:====================================================================================== 2021-02-10T04:18:59.771 INFO:teuthology.orchestra.run.smithi085.stdout: Package Arch Version Repository Size 2021-02-10T04:18:59.771 INFO:teuthology.orchestra.run.smithi085.stdout:====================================================================================== 2021-02-10T04:18:59.771 INFO:teuthology.orchestra.run.smithi085.stdout:Installing: 2021-02-10T04:18:59.771 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-fuse x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 882 k 2021-02-10T04:18:59.771 INFO:teuthology.orchestra.run.smithi085.stdout:Installing dependencies: 2021-02-10T04:18:59.771 INFO:teuthology.orchestra.run.smithi085.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2021-02-10T04:18:59.772 INFO:teuthology.orchestra.run.smithi085.stdout: fuse-common x86_64 3.2.1-12.el8 rhel-8-for-x86_64-baseos-rpms 21 k 2021-02-10T04:18:59.772 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:18:59.772 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction Summary 2021-02-10T04:18:59.772 INFO:teuthology.orchestra.run.smithi085.stdout:====================================================================================== 2021-02-10T04:18:59.772 INFO:teuthology.orchestra.run.smithi085.stdout:Install 3 Packages 2021-02-10T04:18:59.772 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:18:59.773 INFO:teuthology.orchestra.run.smithi085.stdout:Total download size: 985 k 2021-02-10T04:18:59.773 INFO:teuthology.orchestra.run.smithi085.stdout:Installed size: 3.2 M 2021-02-10T04:18:59.773 INFO:teuthology.orchestra.run.smithi085.stdout:Downloading Packages: 2021-02-10T04:18:59.812 INFO:teuthology.orchestra.run.smithi085.stdout:(1/3): fuse-2.9.7-12.el8.x86_64.rpm 2.1 MB/s | 83 kB 00:00 2021-02-10T04:18:59.813 INFO:teuthology.orchestra.run.smithi085.stdout:(2/3): fuse-common-3.2.1-12.el8.x86_64.rpm 519 kB/s | 21 kB 00:00 2021-02-10T04:18:59.913 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:18:59.914 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2021-02-10T04:18:59.914 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-rook-2:17.0.0-681.gc1ea6241.el8.noarch 2021-02-10T04:18:59.914 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-02-10T04:18:59.914 INFO:teuthology.orchestra.run.smithi164.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-02-10T04:18:59.914 INFO:teuthology.orchestra.run.smithi164.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-02-10T04:18:59.914 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-02-10T04:18:59.914 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-02-10T04:18:59.914 INFO:teuthology.orchestra.run.smithi164.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-02-10T04:18:59.914 INFO:teuthology.orchestra.run.smithi164.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-02-10T04:18:59.915 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-02-10T04:18:59.915 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-02-10T04:18:59.915 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-02-10T04:18:59.915 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rsa-3.4.2-16.el8.noarch 2021-02-10T04:18:59.915 INFO:teuthology.orchestra.run.smithi164.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-02-10T04:18:59.915 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:18:59.915 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2021-02-10T04:18:59.995 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-mgr-cephadm 2021-02-10T04:19:00.217 INFO:teuthology.orchestra.run.smithi085.stdout:(3/3): ceph-fuse-17.0.0-681.gc1ea6241.el8.x86_6 1.9 MB/s | 882 kB 00:00 2021-02-10T04:19:00.219 INFO:teuthology.orchestra.run.smithi085.stdout:-------------------------------------------------------------------------------- 2021-02-10T04:19:00.219 INFO:teuthology.orchestra.run.smithi085.stdout:Total 2.2 MB/s | 985 kB 00:00 2021-02-10T04:19:00.221 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction check 2021-02-10T04:19:00.242 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction check succeeded. 2021-02-10T04:19:00.242 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction test 2021-02-10T04:19:00.284 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2021-02-10T04:19:00.316 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction test succeeded. 2021-02-10T04:19:00.320 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction 2021-02-10T04:19:00.493 INFO:teuthology.orchestra.run.smithi085.stdout: Preparing : 1/1 2021-02-10T04:19:00.494 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 1/3 2021-02-10T04:19:00.494 INFO:teuthology.orchestra.run.smithi085.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2021-02-10T04:19:00.494 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:19:00.761 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 83 kB/s | 2.4 kB 00:00 2021-02-10T04:19:00.872 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 97 kB/s | 2.8 kB 00:00 2021-02-10T04:19:00.897 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : fuse-2.9.7-12.el8.x86_64 2/3 2021-02-10T04:19:01.087 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : ceph-fuse-2:17.0.0-681.gc1ea6241.el8.x86_64 3/3 2021-02-10T04:19:01.427 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: ceph-fuse-2:17.0.0-681.gc1ea6241.el8.x86_64 3/3 2021-02-10T04:19:01.427 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : ceph-fuse-2:17.0.0-681.gc1ea6241.el8.x86_64 1/3 2021-02-10T04:19:01.427 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/3 2021-02-10T04:19:02.370 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2021-02-10T04:19:02.371 INFO:teuthology.orchestra.run.smithi164.stdout:============================================================================================== 2021-02-10T04:19:02.371 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2021-02-10T04:19:02.371 INFO:teuthology.orchestra.run.smithi164.stdout:============================================================================================== 2021-02-10T04:19:02.372 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2021-02-10T04:19:02.372 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-cephadm noarch 2:17.0.0-681.gc1ea6241.el8 ceph-noarch 79 k 2021-02-10T04:19:02.372 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2021-02-10T04:19:02.372 INFO:teuthology.orchestra.run.smithi164.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2021-02-10T04:19:02.372 INFO:teuthology.orchestra.run.smithi164.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2021-02-10T04:19:02.372 INFO:teuthology.orchestra.run.smithi164.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2021-02-10T04:19:02.372 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2021-02-10T04:19:02.372 INFO:teuthology.orchestra.run.smithi164.stdout: python3-remoto noarch 1.1.4-4.el8 epel 48 k 2021-02-10T04:19:02.372 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:19:02.372 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2021-02-10T04:19:02.372 INFO:teuthology.orchestra.run.smithi164.stdout:============================================================================================== 2021-02-10T04:19:02.372 INFO:teuthology.orchestra.run.smithi164.stdout:Install 6 Packages 2021-02-10T04:19:02.372 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:19:02.373 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 5.8 M 2021-02-10T04:19:02.373 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 24 M 2021-02-10T04:19:02.373 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2021-02-10T04:19:02.423 INFO:teuthology.orchestra.run.smithi164.stdout:(1/6): python3-apipkg-1.5-6.el8.noarch.rpm 428 kB/s | 21 kB 00:00 2021-02-10T04:19:02.456 INFO:teuthology.orchestra.run.smithi164.stdout:(2/6): python3-remoto-1.1.4-4.el8.noarch.rpm 1.4 MB/s | 48 kB 00:00 2021-02-10T04:19:02.483 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/3 2021-02-10T04:19:02.484 INFO:teuthology.orchestra.run.smithi085.stdout:Installed products updated. 2021-02-10T04:19:02.490 INFO:teuthology.orchestra.run.smithi164.stdout:(3/6): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 15 MB/s | 538 kB 00:00 2021-02-10T04:19:02.492 INFO:teuthology.orchestra.run.smithi164.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.9 MB/s | 355 kB 00:00 2021-02-10T04:19:02.523 INFO:teuthology.orchestra.run.smithi164.stdout:(5/6): ceph-mgr-cephadm-17.0.0-681.gc1ea6241.el 530 kB/s | 79 kB 00:00 2021-02-10T04:19:02.538 INFO:teuthology.orchestra.run.smithi164.stdout:(6/6): python3-babel-2.5.1-5.el8.noarch.rpm 99 MB/s | 4.8 MB 00:00 2021-02-10T04:19:02.540 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2021-02-10T04:19:02.540 INFO:teuthology.orchestra.run.smithi164.stdout:Total 35 MB/s | 5.8 MB 00:00 2021-02-10T04:19:02.543 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2021-02-10T04:19:02.552 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2021-02-10T04:19:02.552 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2021-02-10T04:19:02.629 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2021-02-10T04:19:02.633 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2021-02-10T04:19:03.657 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2021-02-10T04:19:06.958 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:19:06.959 INFO:teuthology.orchestra.run.smithi085.stdout:Installed: 2021-02-10T04:19:06.959 INFO:teuthology.orchestra.run.smithi085.stdout: ceph-fuse-2:17.0.0-681.gc1ea6241.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-02-10T04:19:06.959 INFO:teuthology.orchestra.run.smithi085.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-02-10T04:19:06.959 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:19:06.959 INFO:teuthology.orchestra.run.smithi085.stdout:Complete! 2021-02-10T04:19:06.959 DEBUG:teuthology.orchestra.run.smithi085:> sudo yum -y install librados-devel 2021-02-10T04:19:06.960 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 1/6 2021-02-10T04:19:06.960 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 2/6 2021-02-10T04:19:06.960 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 3/6 2021-02-10T04:19:06.960 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 4/6 2021-02-10T04:19:06.960 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 5/6 2021-02-10T04:19:06.960 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-681.gc1ea6241.el8.noarch 6/6 2021-02-10T04:19:06.960 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-681.gc1ea6241.el8.noarch 6/6 2021-02-10T04:19:06.960 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-681.gc1ea6241.el8.noarch 1/6 2021-02-10T04:19:06.960 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 2/6 2021-02-10T04:19:06.960 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 3/6 2021-02-10T04:19:06.960 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 4/6 2021-02-10T04:19:06.960 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 5/6 2021-02-10T04:19:06.961 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 6/6 2021-02-10T04:19:06.961 INFO:teuthology.orchestra.run.smithi164.stdout:Installed products updated. 2021-02-10T04:19:07.265 INFO:teuthology.orchestra.run.smithi085.stdout:Updating Subscription Management repositories. 2021-02-10T04:19:07.698 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:19:07.699 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2021-02-10T04:19:07.699 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-cephadm-2:17.0.0-681.gc1ea6241.el8.noarch 2021-02-10T04:19:07.699 INFO:teuthology.orchestra.run.smithi164.stdout: python3-apipkg-1.5-6.el8.noarch 2021-02-10T04:19:07.699 INFO:teuthology.orchestra.run.smithi164.stdout: python3-babel-2.5.1-5.el8.noarch 2021-02-10T04:19:07.699 INFO:teuthology.orchestra.run.smithi164.stdout: python3-execnet-1.7.1-1.el8.noarch 2021-02-10T04:19:07.700 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2021-02-10T04:19:07.700 INFO:teuthology.orchestra.run.smithi164.stdout: python3-remoto-1.1.4-4.el8.noarch 2021-02-10T04:19:07.700 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:19:07.700 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2021-02-10T04:19:07.757 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 83 kB/s | 2.4 kB 00:00 2021-02-10T04:19:07.812 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-fuse 2021-02-10T04:19:07.861 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 95 kB/s | 2.8 kB 00:00 2021-02-10T04:19:08.105 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2021-02-10T04:19:08.598 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 2021-02-10T04:19:08.707 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 97 kB/s | 2.8 kB 00:00 2021-02-10T04:19:09.617 INFO:teuthology.orchestra.run.smithi085.stdout:Dependencies resolved. 2021-02-10T04:19:09.618 INFO:teuthology.orchestra.run.smithi085.stdout:================================================================================ 2021-02-10T04:19:09.618 INFO:teuthology.orchestra.run.smithi085.stdout: Package Arch Version Repo Size 2021-02-10T04:19:09.618 INFO:teuthology.orchestra.run.smithi085.stdout:================================================================================ 2021-02-10T04:19:09.618 INFO:teuthology.orchestra.run.smithi085.stdout:Installing: 2021-02-10T04:19:09.618 INFO:teuthology.orchestra.run.smithi085.stdout: librados-devel x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 131 k 2021-02-10T04:19:09.618 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:19:09.618 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction Summary 2021-02-10T04:19:09.618 INFO:teuthology.orchestra.run.smithi085.stdout:================================================================================ 2021-02-10T04:19:09.618 INFO:teuthology.orchestra.run.smithi085.stdout:Install 1 Package 2021-02-10T04:19:09.619 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:19:09.619 INFO:teuthology.orchestra.run.smithi085.stdout:Total download size: 131 k 2021-02-10T04:19:09.619 INFO:teuthology.orchestra.run.smithi085.stdout:Installed size: 437 k 2021-02-10T04:19:09.619 INFO:teuthology.orchestra.run.smithi085.stdout:Downloading Packages: 2021-02-10T04:19:09.790 INFO:teuthology.orchestra.run.smithi085.stdout:librados-devel-17.0.0-681.gc1ea6241.el8.x86_64. 769 kB/s | 131 kB 00:00 2021-02-10T04:19:09.790 INFO:teuthology.orchestra.run.smithi085.stdout:-------------------------------------------------------------------------------- 2021-02-10T04:19:09.790 INFO:teuthology.orchestra.run.smithi085.stdout:Total 763 kB/s | 131 kB 00:00 2021-02-10T04:19:09.791 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction check 2021-02-10T04:19:09.797 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction check succeeded. 2021-02-10T04:19:09.797 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction test 2021-02-10T04:19:09.851 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction test succeeded. 2021-02-10T04:19:09.856 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction 2021-02-10T04:19:09.976 INFO:teuthology.orchestra.run.smithi085.stdout: Preparing : 1/1 2021-02-10T04:19:10.135 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : librados-devel-2:17.0.0-681.gc1ea6241.el8.x86_64 1/1 2021-02-10T04:19:10.186 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2021-02-10T04:19:10.187 INFO:teuthology.orchestra.run.smithi164.stdout:====================================================================================== 2021-02-10T04:19:10.187 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2021-02-10T04:19:10.187 INFO:teuthology.orchestra.run.smithi164.stdout:====================================================================================== 2021-02-10T04:19:10.187 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2021-02-10T04:19:10.187 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-fuse x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 882 k 2021-02-10T04:19:10.187 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2021-02-10T04:19:10.187 INFO:teuthology.orchestra.run.smithi164.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2021-02-10T04:19:10.187 INFO:teuthology.orchestra.run.smithi164.stdout: fuse-common x86_64 3.2.1-12.el8 rhel-8-for-x86_64-baseos-rpms 21 k 2021-02-10T04:19:10.188 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:19:10.188 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2021-02-10T04:19:10.188 INFO:teuthology.orchestra.run.smithi164.stdout:====================================================================================== 2021-02-10T04:19:10.188 INFO:teuthology.orchestra.run.smithi164.stdout:Install 3 Packages 2021-02-10T04:19:10.188 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:19:10.188 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 985 k 2021-02-10T04:19:10.188 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 3.2 M 2021-02-10T04:19:10.188 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2021-02-10T04:19:10.229 INFO:teuthology.orchestra.run.smithi164.stdout:(1/3): fuse-2.9.7-12.el8.x86_64.rpm 2.0 MB/s | 83 kB 00:00 2021-02-10T04:19:10.234 INFO:teuthology.orchestra.run.smithi164.stdout:(2/3): fuse-common-3.2.1-12.el8.x86_64.rpm 470 kB/s | 21 kB 00:00 2021-02-10T04:19:10.350 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: librados-devel-2:17.0.0-681.gc1ea6241.el8.x86_64 1/1 2021-02-10T04:19:10.432 INFO:teuthology.orchestra.run.smithi164.stdout:(3/3): ceph-fuse-17.0.0-681.gc1ea6241.el8.x86_6 3.5 MB/s | 882 kB 00:00 2021-02-10T04:19:10.433 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2021-02-10T04:19:10.433 INFO:teuthology.orchestra.run.smithi164.stdout:Total 3.9 MB/s | 985 kB 00:00 2021-02-10T04:19:10.436 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2021-02-10T04:19:10.456 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2021-02-10T04:19:10.456 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2021-02-10T04:19:10.531 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2021-02-10T04:19:10.535 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2021-02-10T04:19:10.711 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2021-02-10T04:19:10.711 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 1/3 2021-02-10T04:19:10.711 INFO:teuthology.orchestra.run.smithi164.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2021-02-10T04:19:10.711 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:19:10.804 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : librados-devel-2:17.0.0-681.gc1ea6241.el8.x86_64 1/1 2021-02-10T04:19:10.804 INFO:teuthology.orchestra.run.smithi085.stdout:Installed products updated. 2021-02-10T04:19:11.110 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : fuse-2.9.7-12.el8.x86_64 2/3 2021-02-10T04:19:11.300 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-fuse-2:17.0.0-681.gc1ea6241.el8.x86_64 3/3 2021-02-10T04:19:11.646 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-fuse-2:17.0.0-681.gc1ea6241.el8.x86_64 3/3 2021-02-10T04:19:11.646 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-fuse-2:17.0.0-681.gc1ea6241.el8.x86_64 1/3 2021-02-10T04:19:11.646 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/3 2021-02-10T04:19:12.131 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/3 2021-02-10T04:19:12.131 INFO:teuthology.orchestra.run.smithi164.stdout:Installed products updated. 2021-02-10T04:19:12.643 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:19:12.643 INFO:teuthology.orchestra.run.smithi085.stdout:Installed: 2021-02-10T04:19:12.643 INFO:teuthology.orchestra.run.smithi085.stdout: librados-devel-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:19:12.644 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:19:12.644 INFO:teuthology.orchestra.run.smithi085.stdout:Complete! 2021-02-10T04:19:12.724 DEBUG:teuthology.orchestra.run.smithi085:> sudo yum -y install libcephfs2 2021-02-10T04:19:13.011 INFO:teuthology.orchestra.run.smithi085.stdout:Updating Subscription Management repositories. 2021-02-10T04:19:13.490 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 82 kB/s | 2.4 kB 00:00 2021-02-10T04:19:13.603 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 88 kB/s | 2.8 kB 00:00 2021-02-10T04:19:13.980 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:19:13.980 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2021-02-10T04:19:13.980 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-fuse-2:17.0.0-681.gc1ea6241.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-02-10T04:19:13.980 INFO:teuthology.orchestra.run.smithi164.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-02-10T04:19:13.980 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:19:13.980 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2021-02-10T04:19:14.053 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install librados-devel 2021-02-10T04:19:14.347 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2021-02-10T04:19:14.779 INFO:teuthology.orchestra.run.smithi085.stdout:Package libcephfs2-2:17.0.0-681.gc1ea6241.el8.x86_64 is already installed. 2021-02-10T04:19:14.819 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 87 kB/s | 2.4 kB 00:00 2021-02-10T04:19:14.931 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 99 kB/s | 2.8 kB 00:00 2021-02-10T04:19:15.097 INFO:teuthology.orchestra.run.smithi085.stdout:Dependencies resolved. 2021-02-10T04:19:15.098 INFO:teuthology.orchestra.run.smithi085.stdout:Nothing to do. 2021-02-10T04:19:15.098 INFO:teuthology.orchestra.run.smithi085.stdout:Complete! 2021-02-10T04:19:15.171 DEBUG:teuthology.orchestra.run.smithi085:> sudo yum -y install libcephfs-devel 2021-02-10T04:19:15.458 INFO:teuthology.orchestra.run.smithi085.stdout:Updating Subscription Management repositories. 2021-02-10T04:19:15.969 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 80 kB/s | 2.4 kB 00:00 2021-02-10T04:19:16.081 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 95 kB/s | 2.8 kB 00:00 2021-02-10T04:19:16.419 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2021-02-10T04:19:16.419 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2021-02-10T04:19:16.419 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2021-02-10T04:19:16.419 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2021-02-10T04:19:16.420 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2021-02-10T04:19:16.420 INFO:teuthology.orchestra.run.smithi164.stdout: librados-devel x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 131 k 2021-02-10T04:19:16.420 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:19:16.420 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2021-02-10T04:19:16.420 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2021-02-10T04:19:16.420 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2021-02-10T04:19:16.420 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:19:16.421 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 131 k 2021-02-10T04:19:16.421 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 437 k 2021-02-10T04:19:16.421 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2021-02-10T04:19:16.873 INFO:teuthology.orchestra.run.smithi164.stdout:librados-devel-17.0.0-681.gc1ea6241.el8.x86_64. 290 kB/s | 131 kB 00:00 2021-02-10T04:19:16.874 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2021-02-10T04:19:16.874 INFO:teuthology.orchestra.run.smithi164.stdout:Total 289 kB/s | 131 kB 00:00 2021-02-10T04:19:16.874 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2021-02-10T04:19:16.881 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2021-02-10T04:19:16.881 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2021-02-10T04:19:16.937 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2021-02-10T04:19:16.941 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2021-02-10T04:19:17.090 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2021-02-10T04:19:17.249 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librados-devel-2:17.0.0-681.gc1ea6241.el8.x86_64 1/1 2021-02-10T04:19:17.473 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados-devel-2:17.0.0-681.gc1ea6241.el8.x86_64 1/1 2021-02-10T04:19:17.549 INFO:teuthology.orchestra.run.smithi085.stdout:Dependencies resolved. 2021-02-10T04:19:17.550 INFO:teuthology.orchestra.run.smithi085.stdout:================================================================================ 2021-02-10T04:19:17.550 INFO:teuthology.orchestra.run.smithi085.stdout: Package Arch Version Repo Size 2021-02-10T04:19:17.550 INFO:teuthology.orchestra.run.smithi085.stdout:================================================================================ 2021-02-10T04:19:17.550 INFO:teuthology.orchestra.run.smithi085.stdout:Installing: 2021-02-10T04:19:17.550 INFO:teuthology.orchestra.run.smithi085.stdout: libcephfs-devel x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 24 k 2021-02-10T04:19:17.550 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:19:17.550 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction Summary 2021-02-10T04:19:17.551 INFO:teuthology.orchestra.run.smithi085.stdout:================================================================================ 2021-02-10T04:19:17.551 INFO:teuthology.orchestra.run.smithi085.stdout:Install 1 Package 2021-02-10T04:19:17.551 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:19:17.551 INFO:teuthology.orchestra.run.smithi085.stdout:Total download size: 24 k 2021-02-10T04:19:17.551 INFO:teuthology.orchestra.run.smithi085.stdout:Installed size: 89 k 2021-02-10T04:19:17.551 INFO:teuthology.orchestra.run.smithi085.stdout:Downloading Packages: 2021-02-10T04:19:17.908 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados-devel-2:17.0.0-681.gc1ea6241.el8.x86_64 1/1 2021-02-10T04:19:17.908 INFO:teuthology.orchestra.run.smithi164.stdout:Installed products updated. 2021-02-10T04:19:17.966 INFO:teuthology.orchestra.run.smithi085.stdout:libcephfs-devel-17.0.0-681.gc1ea6241.el8.x86_64 58 kB/s | 24 kB 00:00 2021-02-10T04:19:17.966 INFO:teuthology.orchestra.run.smithi085.stdout:-------------------------------------------------------------------------------- 2021-02-10T04:19:17.967 INFO:teuthology.orchestra.run.smithi085.stdout:Total 57 kB/s | 24 kB 00:00 2021-02-10T04:19:17.967 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction check 2021-02-10T04:19:17.971 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction check succeeded. 2021-02-10T04:19:17.971 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction test 2021-02-10T04:19:17.973 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction test succeeded. 2021-02-10T04:19:17.977 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction 2021-02-10T04:19:18.027 INFO:teuthology.orchestra.run.smithi085.stdout: Preparing : 1/1 2021-02-10T04:19:18.217 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : libcephfs-devel-2:17.0.0-681.gc1ea6241.el8.x86_64 1/1 2021-02-10T04:19:19.058 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-681.gc1ea6241.el8.x86_64 1/1 2021-02-10T04:19:19.058 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : libcephfs-devel-2:17.0.0-681.gc1ea6241.el8.x86_64 1/1 2021-02-10T04:19:19.058 INFO:teuthology.orchestra.run.smithi085.stdout:Installed products updated. 2021-02-10T04:19:19.756 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:19:19.757 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2021-02-10T04:19:19.757 INFO:teuthology.orchestra.run.smithi164.stdout: librados-devel-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:19:19.757 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:19:19.757 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2021-02-10T04:19:19.930 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install libcephfs2 2021-02-10T04:19:20.218 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2021-02-10T04:19:20.686 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:19:20.687 INFO:teuthology.orchestra.run.smithi085.stdout:Installed: 2021-02-10T04:19:20.687 INFO:teuthology.orchestra.run.smithi085.stdout: libcephfs-devel-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:19:20.687 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:19:20.687 INFO:teuthology.orchestra.run.smithi085.stdout:Complete! 2021-02-10T04:19:20.718 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 80 kB/s | 2.4 kB 00:00 2021-02-10T04:19:20.781 DEBUG:teuthology.orchestra.run.smithi085:> sudo yum -y install librados2 2021-02-10T04:19:20.829 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 96 kB/s | 2.8 kB 00:00 2021-02-10T04:19:21.068 INFO:teuthology.orchestra.run.smithi085.stdout:Updating Subscription Management repositories. 2021-02-10T04:19:21.545 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 80 kB/s | 2.4 kB 00:00 2021-02-10T04:19:21.656 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 95 kB/s | 2.8 kB 00:00 2021-02-10T04:19:21.991 INFO:teuthology.orchestra.run.smithi164.stdout:Package libcephfs2-2:17.0.0-681.gc1ea6241.el8.x86_64 is already installed. 2021-02-10T04:19:22.320 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2021-02-10T04:19:22.321 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2021-02-10T04:19:22.321 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2021-02-10T04:19:22.394 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install libcephfs-devel 2021-02-10T04:19:22.682 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2021-02-10T04:19:22.826 INFO:teuthology.orchestra.run.smithi085.stdout:Package librados2-2:17.0.0-681.gc1ea6241.el8.x86_64 is already installed. 2021-02-10T04:19:23.150 INFO:teuthology.orchestra.run.smithi085.stdout:Dependencies resolved. 2021-02-10T04:19:23.150 INFO:teuthology.orchestra.run.smithi085.stdout:Nothing to do. 2021-02-10T04:19:23.150 INFO:teuthology.orchestra.run.smithi085.stdout:Complete! 2021-02-10T04:19:23.178 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 2021-02-10T04:19:23.258 DEBUG:teuthology.orchestra.run.smithi085:> sudo yum -y install librbd1 2021-02-10T04:19:23.289 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 96 kB/s | 2.8 kB 00:00 2021-02-10T04:19:23.543 INFO:teuthology.orchestra.run.smithi085.stdout:Updating Subscription Management repositories. 2021-02-10T04:19:24.041 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 83 kB/s | 2.4 kB 00:00 2021-02-10T04:19:28.662 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2021-02-10T04:19:28.662 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2021-02-10T04:19:28.663 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2021-02-10T04:19:28.663 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2021-02-10T04:19:28.663 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2021-02-10T04:19:28.663 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs-devel x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 24 k 2021-02-10T04:19:28.663 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:19:28.663 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2021-02-10T04:19:28.663 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2021-02-10T04:19:28.663 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2021-02-10T04:19:28.663 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:19:28.663 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 24 k 2021-02-10T04:19:28.663 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 89 k 2021-02-10T04:19:28.663 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2021-02-10T04:19:28.664 INFO:teuthology.orchestra.run.smithi164.stdout:libcephfs-devel-17.0.0-681.gc1ea6241.el8.x86_64 57 kB/s | 24 kB 00:00 2021-02-10T04:19:28.664 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2021-02-10T04:19:28.664 INFO:teuthology.orchestra.run.smithi164.stdout:Total 57 kB/s | 24 kB 00:00 2021-02-10T04:19:28.664 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2021-02-10T04:19:28.664 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2021-02-10T04:19:28.664 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2021-02-10T04:19:28.664 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2021-02-10T04:19:28.664 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2021-02-10T04:19:28.664 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2021-02-10T04:19:28.664 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephfs-devel-2:17.0.0-681.gc1ea6241.el8.x86_64 1/1 2021-02-10T04:19:28.664 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-681.gc1ea6241.el8.x86_64 1/1 2021-02-10T04:19:28.664 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephfs-devel-2:17.0.0-681.gc1ea6241.el8.x86_64 1/1 2021-02-10T04:19:28.665 INFO:teuthology.orchestra.run.smithi164.stdout:Installed products updated. 2021-02-10T04:19:28.665 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:19:28.665 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2021-02-10T04:19:28.665 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs-devel-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:19:28.665 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:19:28.665 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2021-02-10T04:19:28.665 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 95 kB/s | 2.8 kB 00:00 2021-02-10T04:19:28.665 INFO:teuthology.orchestra.run.smithi085.stdout:Package librbd1-2:17.0.0-681.gc1ea6241.el8.x86_64 is already installed. 2021-02-10T04:19:28.665 INFO:teuthology.orchestra.run.smithi085.stdout:Dependencies resolved. 2021-02-10T04:19:28.665 INFO:teuthology.orchestra.run.smithi085.stdout:Nothing to do. 2021-02-10T04:19:28.666 INFO:teuthology.orchestra.run.smithi085.stdout:Complete! 2021-02-10T04:19:28.666 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install librados2 2021-02-10T04:19:28.666 DEBUG:teuthology.orchestra.run.smithi085:> sudo yum -y install python3-rados 2021-02-10T04:19:28.968 INFO:teuthology.orchestra.run.smithi085.stdout:Updating Subscription Management repositories. 2021-02-10T04:19:28.969 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2021-02-10T04:19:29.463 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 69 kB/s | 2.4 kB 00:00 2021-02-10T04:19:29.465 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 68 kB/s | 2.4 kB 00:00 2021-02-10T04:19:29.580 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 83 kB/s | 2.8 kB 00:00 2021-02-10T04:19:29.583 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 78 kB/s | 2.8 kB 00:00 2021-02-10T04:19:30.739 INFO:teuthology.orchestra.run.smithi164.stdout:Package librados2-2:17.0.0-681.gc1ea6241.el8.x86_64 is already installed. 2021-02-10T04:19:30.758 INFO:teuthology.orchestra.run.smithi085.stdout:Package python3-rados-2:17.0.0-681.gc1ea6241.el8.x86_64 is already installed. 2021-02-10T04:19:31.061 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2021-02-10T04:19:31.062 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2021-02-10T04:19:31.062 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2021-02-10T04:19:31.082 INFO:teuthology.orchestra.run.smithi085.stdout:Dependencies resolved. 2021-02-10T04:19:31.082 INFO:teuthology.orchestra.run.smithi085.stdout:Nothing to do. 2021-02-10T04:19:31.082 INFO:teuthology.orchestra.run.smithi085.stdout:Complete! 2021-02-10T04:19:31.139 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install librbd1 2021-02-10T04:19:31.164 DEBUG:teuthology.orchestra.run.smithi085:> sudo yum -y install python3-rgw 2021-02-10T04:19:31.426 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2021-02-10T04:19:31.449 INFO:teuthology.orchestra.run.smithi085.stdout:Updating Subscription Management repositories. 2021-02-10T04:19:31.922 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 82 kB/s | 2.4 kB 00:00 2021-02-10T04:19:31.943 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 83 kB/s | 2.4 kB 00:00 2021-02-10T04:19:32.033 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 95 kB/s | 2.8 kB 00:00 2021-02-10T04:19:32.054 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 97 kB/s | 2.8 kB 00:00 2021-02-10T04:19:33.214 INFO:teuthology.orchestra.run.smithi164.stdout:Package librbd1-2:17.0.0-681.gc1ea6241.el8.x86_64 is already installed. 2021-02-10T04:19:33.233 INFO:teuthology.orchestra.run.smithi085.stdout:Package python3-rgw-2:17.0.0-681.gc1ea6241.el8.x86_64 is already installed. 2021-02-10T04:19:33.540 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2021-02-10T04:19:33.541 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2021-02-10T04:19:33.541 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2021-02-10T04:19:33.552 INFO:teuthology.orchestra.run.smithi085.stdout:Dependencies resolved. 2021-02-10T04:19:33.553 INFO:teuthology.orchestra.run.smithi085.stdout:Nothing to do. 2021-02-10T04:19:33.554 INFO:teuthology.orchestra.run.smithi085.stdout:Complete! 2021-02-10T04:19:33.607 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install python3-rados 2021-02-10T04:19:33.635 DEBUG:teuthology.orchestra.run.smithi085:> sudo yum -y install python3-cephfs 2021-02-10T04:19:33.895 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2021-02-10T04:19:33.921 INFO:teuthology.orchestra.run.smithi085.stdout:Updating Subscription Management repositories. 2021-02-10T04:19:34.423 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 79 kB/s | 2.4 kB 00:00 2021-02-10T04:19:34.425 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 80 kB/s | 2.4 kB 00:00 2021-02-10T04:19:34.533 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 98 kB/s | 2.8 kB 00:00 2021-02-10T04:19:34.625 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 99 kB/s | 2.8 kB 00:00 2021-02-10T04:19:35.683 INFO:teuthology.orchestra.run.smithi085.stdout:Package python3-cephfs-2:17.0.0-681.gc1ea6241.el8.x86_64 is already installed. 2021-02-10T04:19:35.739 INFO:teuthology.orchestra.run.smithi164.stdout:Package python3-rados-2:17.0.0-681.gc1ea6241.el8.x86_64 is already installed. 2021-02-10T04:19:36.007 INFO:teuthology.orchestra.run.smithi085.stdout:Dependencies resolved. 2021-02-10T04:19:36.008 INFO:teuthology.orchestra.run.smithi085.stdout:Nothing to do. 2021-02-10T04:19:36.008 INFO:teuthology.orchestra.run.smithi085.stdout:Complete! 2021-02-10T04:19:36.061 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2021-02-10T04:19:36.062 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2021-02-10T04:19:36.063 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2021-02-10T04:19:36.132 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install python3-rgw 2021-02-10T04:19:36.148 DEBUG:teuthology.orchestra.run.smithi085:> sudo yum -y install python3-rbd 2021-02-10T04:19:36.419 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2021-02-10T04:19:36.434 INFO:teuthology.orchestra.run.smithi085.stdout:Updating Subscription Management repositories. 2021-02-10T04:19:36.907 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 80 kB/s | 2.4 kB 00:00 2021-02-10T04:19:36.917 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 80 kB/s | 2.4 kB 00:00 2021-02-10T04:19:37.019 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 96 kB/s | 2.8 kB 00:00 2021-02-10T04:19:37.027 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 95 kB/s | 2.8 kB 00:00 2021-02-10T04:19:38.197 INFO:teuthology.orchestra.run.smithi164.stdout:Package python3-rgw-2:17.0.0-681.gc1ea6241.el8.x86_64 is already installed. 2021-02-10T04:19:38.202 INFO:teuthology.orchestra.run.smithi085.stdout:Package python3-rbd-2:17.0.0-681.gc1ea6241.el8.x86_64 is already installed. 2021-02-10T04:19:38.518 INFO:teuthology.orchestra.run.smithi085.stdout:Dependencies resolved. 2021-02-10T04:19:38.519 INFO:teuthology.orchestra.run.smithi085.stdout:Nothing to do. 2021-02-10T04:19:38.519 INFO:teuthology.orchestra.run.smithi085.stdout:Complete! 2021-02-10T04:19:38.521 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2021-02-10T04:19:38.522 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2021-02-10T04:19:38.522 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2021-02-10T04:19:38.591 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install python3-cephfs 2021-02-10T04:19:38.593 DEBUG:teuthology.orchestra.run.smithi085:> sudo yum -y install rbd-fuse 2021-02-10T04:19:38.879 INFO:teuthology.orchestra.run.smithi085.stdout:Updating Subscription Management repositories. 2021-02-10T04:19:38.883 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2021-02-10T04:19:39.372 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 82 kB/s | 2.4 kB 00:00 2021-02-10T04:19:39.400 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 80 kB/s | 2.4 kB 00:00 2021-02-10T04:19:39.485 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 96 kB/s | 2.8 kB 00:00 2021-02-10T04:19:39.512 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 98 kB/s | 2.8 kB 00:00 2021-02-10T04:19:40.680 INFO:teuthology.orchestra.run.smithi164.stdout:Package python3-cephfs-2:17.0.0-681.gc1ea6241.el8.x86_64 is already installed. 2021-02-10T04:19:40.966 INFO:teuthology.orchestra.run.smithi085.stdout:Dependencies resolved. 2021-02-10T04:19:40.968 INFO:teuthology.orchestra.run.smithi085.stdout:================================================================================ 2021-02-10T04:19:40.968 INFO:teuthology.orchestra.run.smithi085.stdout: Package Architecture Version Repository Size 2021-02-10T04:19:40.968 INFO:teuthology.orchestra.run.smithi085.stdout:================================================================================ 2021-02-10T04:19:40.968 INFO:teuthology.orchestra.run.smithi085.stdout:Installing: 2021-02-10T04:19:40.968 INFO:teuthology.orchestra.run.smithi085.stdout: rbd-fuse x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 89 k 2021-02-10T04:19:40.969 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:19:40.969 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction Summary 2021-02-10T04:19:40.969 INFO:teuthology.orchestra.run.smithi085.stdout:================================================================================ 2021-02-10T04:19:40.969 INFO:teuthology.orchestra.run.smithi085.stdout:Install 1 Package 2021-02-10T04:19:40.969 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:19:40.969 INFO:teuthology.orchestra.run.smithi085.stdout:Total download size: 89 k 2021-02-10T04:19:40.970 INFO:teuthology.orchestra.run.smithi085.stdout:Installed size: 298 k 2021-02-10T04:19:40.970 INFO:teuthology.orchestra.run.smithi085.stdout:Downloading Packages: 2021-02-10T04:19:41.006 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2021-02-10T04:19:41.007 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2021-02-10T04:19:41.007 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2021-02-10T04:19:41.083 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install python3-rbd 2021-02-10T04:19:41.373 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2021-02-10T04:19:41.398 INFO:teuthology.orchestra.run.smithi085.stdout:rbd-fuse-17.0.0-681.gc1ea6241.el8.x86_64.rpm 207 kB/s | 89 kB 00:00 2021-02-10T04:19:41.398 INFO:teuthology.orchestra.run.smithi085.stdout:-------------------------------------------------------------------------------- 2021-02-10T04:19:41.399 INFO:teuthology.orchestra.run.smithi085.stdout:Total 206 kB/s | 89 kB 00:00 2021-02-10T04:19:41.399 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction check 2021-02-10T04:19:41.411 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction check succeeded. 2021-02-10T04:19:41.411 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction test 2021-02-10T04:19:41.469 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction test succeeded. 2021-02-10T04:19:41.473 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction 2021-02-10T04:19:41.618 INFO:teuthology.orchestra.run.smithi085.stdout: Preparing : 1/1 2021-02-10T04:19:41.817 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : rbd-fuse-2:17.0.0-681.gc1ea6241.el8.x86_64 1/1 2021-02-10T04:19:41.866 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 75 kB/s | 2.4 kB 00:00 2021-02-10T04:19:41.978 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 96 kB/s | 2.8 kB 00:00 2021-02-10T04:19:42.040 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: rbd-fuse-2:17.0.0-681.gc1ea6241.el8.x86_64 1/1 2021-02-10T04:19:42.505 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : rbd-fuse-2:17.0.0-681.gc1ea6241.el8.x86_64 1/1 2021-02-10T04:19:42.505 INFO:teuthology.orchestra.run.smithi085.stdout:Installed products updated. 2021-02-10T04:19:43.129 INFO:teuthology.orchestra.run.smithi164.stdout:Package python3-rbd-2:17.0.0-681.gc1ea6241.el8.x86_64 is already installed. 2021-02-10T04:19:43.450 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2021-02-10T04:19:43.451 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2021-02-10T04:19:43.451 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2021-02-10T04:19:43.523 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install rbd-fuse 2021-02-10T04:19:43.812 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2021-02-10T04:19:44.339 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 80 kB/s | 2.4 kB 00:00 2021-02-10T04:19:44.395 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:19:44.395 INFO:teuthology.orchestra.run.smithi085.stdout:Installed: 2021-02-10T04:19:44.395 INFO:teuthology.orchestra.run.smithi085.stdout: rbd-fuse-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:19:44.395 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:19:44.395 INFO:teuthology.orchestra.run.smithi085.stdout:Complete! 2021-02-10T04:19:44.450 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 98 kB/s | 2.8 kB 00:00 2021-02-10T04:19:44.516 DEBUG:teuthology.orchestra.run.smithi085:> sudo yum -y install rbd-mirror 2021-02-10T04:19:44.803 INFO:teuthology.orchestra.run.smithi085.stdout:Updating Subscription Management repositories. 2021-02-10T04:19:45.279 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 84 kB/s | 2.4 kB 00:00 2021-02-10T04:19:45.386 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 101 kB/s | 2.8 kB 00:00 2021-02-10T04:19:45.914 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2021-02-10T04:19:45.915 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2021-02-10T04:19:45.915 INFO:teuthology.orchestra.run.smithi164.stdout: Package Architecture Version Repository Size 2021-02-10T04:19:45.915 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2021-02-10T04:19:45.915 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2021-02-10T04:19:45.915 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-fuse x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 89 k 2021-02-10T04:19:45.915 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:19:45.915 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2021-02-10T04:19:45.915 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2021-02-10T04:19:45.916 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2021-02-10T04:19:45.916 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:19:45.916 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 89 k 2021-02-10T04:19:45.916 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 298 k 2021-02-10T04:19:45.916 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2021-02-10T04:19:46.344 INFO:teuthology.orchestra.run.smithi164.stdout:rbd-fuse-17.0.0-681.gc1ea6241.el8.x86_64.rpm 207 kB/s | 89 kB 00:00 2021-02-10T04:19:46.345 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2021-02-10T04:19:46.345 INFO:teuthology.orchestra.run.smithi164.stdout:Total 207 kB/s | 89 kB 00:00 2021-02-10T04:19:46.346 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2021-02-10T04:19:46.357 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2021-02-10T04:19:46.357 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2021-02-10T04:19:46.412 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2021-02-10T04:19:46.416 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2021-02-10T04:19:46.559 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2021-02-10T04:19:46.744 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-fuse-2:17.0.0-681.gc1ea6241.el8.x86_64 1/1 2021-02-10T04:19:46.888 INFO:teuthology.orchestra.run.smithi085.stdout:Dependencies resolved. 2021-02-10T04:19:46.888 INFO:teuthology.orchestra.run.smithi085.stdout:================================================================================ 2021-02-10T04:19:46.889 INFO:teuthology.orchestra.run.smithi085.stdout: Package Arch Version Repo Size 2021-02-10T04:19:46.889 INFO:teuthology.orchestra.run.smithi085.stdout:================================================================================ 2021-02-10T04:19:46.889 INFO:teuthology.orchestra.run.smithi085.stdout:Installing: 2021-02-10T04:19:46.889 INFO:teuthology.orchestra.run.smithi085.stdout: rbd-mirror x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 3.7 M 2021-02-10T04:19:46.889 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:19:46.889 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction Summary 2021-02-10T04:19:46.889 INFO:teuthology.orchestra.run.smithi085.stdout:================================================================================ 2021-02-10T04:19:46.889 INFO:teuthology.orchestra.run.smithi085.stdout:Install 1 Package 2021-02-10T04:19:46.889 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:19:46.889 INFO:teuthology.orchestra.run.smithi085.stdout:Total download size: 3.7 M 2021-02-10T04:19:46.889 INFO:teuthology.orchestra.run.smithi085.stdout:Installed size: 18 M 2021-02-10T04:19:46.889 INFO:teuthology.orchestra.run.smithi085.stdout:Downloading Packages: 2021-02-10T04:19:46.959 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: rbd-fuse-2:17.0.0-681.gc1ea6241.el8.x86_64 1/1 2021-02-10T04:19:47.421 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-fuse-2:17.0.0-681.gc1ea6241.el8.x86_64 1/1 2021-02-10T04:19:47.422 INFO:teuthology.orchestra.run.smithi164.stdout:Installed products updated. 2021-02-10T04:19:47.908 INFO:teuthology.orchestra.run.smithi085.stdout:rbd-mirror-17.0.0-681.gc1ea6241.el8.x86_64.rpm 3.6 MB/s | 3.7 MB 00:01 2021-02-10T04:19:47.909 INFO:teuthology.orchestra.run.smithi085.stdout:-------------------------------------------------------------------------------- 2021-02-10T04:19:47.910 INFO:teuthology.orchestra.run.smithi085.stdout:Total 3.6 MB/s | 3.7 MB 00:01 2021-02-10T04:19:47.910 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction check 2021-02-10T04:19:47.925 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction check succeeded. 2021-02-10T04:19:47.925 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction test 2021-02-10T04:19:47.996 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction test succeeded. 2021-02-10T04:19:48.000 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction 2021-02-10T04:19:48.471 INFO:teuthology.orchestra.run.smithi085.stdout: Preparing : 1/1 2021-02-10T04:19:48.651 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : rbd-mirror-2:17.0.0-681.gc1ea6241.el8.x86_64 1/1 2021-02-10T04:19:49.016 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: rbd-mirror-2:17.0.0-681.gc1ea6241.el8.x86_64 1/1 2021-02-10T04:19:49.266 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:19:49.266 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2021-02-10T04:19:49.266 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-fuse-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:19:49.266 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:19:49.266 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2021-02-10T04:19:49.354 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install rbd-mirror 2021-02-10T04:19:49.512 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : rbd-mirror-2:17.0.0-681.gc1ea6241.el8.x86_64 1/1 2021-02-10T04:19:49.513 INFO:teuthology.orchestra.run.smithi085.stdout:Installed products updated. 2021-02-10T04:19:49.643 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2021-02-10T04:19:50.131 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 83 kB/s | 2.4 kB 00:00 2021-02-10T04:19:50.244 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 97 kB/s | 2.8 kB 00:00 2021-02-10T04:19:51.456 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:19:51.456 INFO:teuthology.orchestra.run.smithi085.stdout:Installed: 2021-02-10T04:19:51.456 INFO:teuthology.orchestra.run.smithi085.stdout: rbd-mirror-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:19:51.456 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:19:51.456 INFO:teuthology.orchestra.run.smithi085.stdout:Complete! 2021-02-10T04:19:51.579 DEBUG:teuthology.orchestra.run.smithi085:> sudo yum -y install rbd-nbd 2021-02-10T04:19:51.711 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2021-02-10T04:19:51.711 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2021-02-10T04:19:51.711 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2021-02-10T04:19:51.712 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2021-02-10T04:19:51.712 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2021-02-10T04:19:51.712 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-mirror x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 3.7 M 2021-02-10T04:19:51.712 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:19:51.712 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2021-02-10T04:19:51.712 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2021-02-10T04:19:51.712 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2021-02-10T04:19:51.712 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:19:51.713 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 3.7 M 2021-02-10T04:19:51.713 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 18 M 2021-02-10T04:19:51.713 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2021-02-10T04:19:51.869 INFO:teuthology.orchestra.run.smithi085.stdout:Updating Subscription Management repositories. 2021-02-10T04:19:52.426 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 82 kB/s | 2.4 kB 00:00 2021-02-10T04:19:52.560 INFO:teuthology.orchestra.run.smithi085.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 93 kB/s | 2.8 kB 00:00 2021-02-10T04:19:53.650 INFO:teuthology.orchestra.run.smithi164.stdout:rbd-mirror-17.0.0-681.gc1ea6241.el8.x86_64.rpm 1.9 MB/s | 3.7 MB 00:01 2021-02-10T04:19:53.651 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2021-02-10T04:19:53.652 INFO:teuthology.orchestra.run.smithi164.stdout:Total 1.9 MB/s | 3.7 MB 00:01 2021-02-10T04:19:53.652 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2021-02-10T04:19:53.668 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2021-02-10T04:19:53.668 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2021-02-10T04:19:53.740 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2021-02-10T04:19:53.744 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2021-02-10T04:19:54.022 INFO:teuthology.orchestra.run.smithi085.stdout:Dependencies resolved. 2021-02-10T04:19:54.023 INFO:teuthology.orchestra.run.smithi085.stdout:================================================================================ 2021-02-10T04:19:54.023 INFO:teuthology.orchestra.run.smithi085.stdout: Package Architecture Version Repository Size 2021-02-10T04:19:54.023 INFO:teuthology.orchestra.run.smithi085.stdout:================================================================================ 2021-02-10T04:19:54.023 INFO:teuthology.orchestra.run.smithi085.stdout:Installing: 2021-02-10T04:19:54.023 INFO:teuthology.orchestra.run.smithi085.stdout: rbd-nbd x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 174 k 2021-02-10T04:19:54.023 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:19:54.023 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction Summary 2021-02-10T04:19:54.023 INFO:teuthology.orchestra.run.smithi085.stdout:================================================================================ 2021-02-10T04:19:54.024 INFO:teuthology.orchestra.run.smithi085.stdout:Install 1 Package 2021-02-10T04:19:54.024 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:19:54.024 INFO:teuthology.orchestra.run.smithi085.stdout:Total download size: 174 k 2021-02-10T04:19:54.024 INFO:teuthology.orchestra.run.smithi085.stdout:Installed size: 649 k 2021-02-10T04:19:54.024 INFO:teuthology.orchestra.run.smithi085.stdout:Downloading Packages: 2021-02-10T04:19:54.217 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2021-02-10T04:19:54.387 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-mirror-2:17.0.0-681.gc1ea6241.el8.x86_64 1/1 2021-02-10T04:19:54.528 INFO:teuthology.orchestra.run.smithi085.stdout:rbd-nbd-17.0.0-681.gc1ea6241.el8.x86_64.rpm 344 kB/s | 174 kB 00:00 2021-02-10T04:19:54.529 INFO:teuthology.orchestra.run.smithi085.stdout:-------------------------------------------------------------------------------- 2021-02-10T04:19:54.530 INFO:teuthology.orchestra.run.smithi085.stdout:Total 343 kB/s | 174 kB 00:00 2021-02-10T04:19:54.530 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction check 2021-02-10T04:19:54.542 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction check succeeded. 2021-02-10T04:19:54.542 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction test 2021-02-10T04:19:54.598 INFO:teuthology.orchestra.run.smithi085.stdout:Transaction test succeeded. 2021-02-10T04:19:54.602 INFO:teuthology.orchestra.run.smithi085.stdout:Running transaction 2021-02-10T04:19:54.729 INFO:teuthology.orchestra.run.smithi085.stdout: Preparing : 1/1 2021-02-10T04:19:54.748 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: rbd-mirror-2:17.0.0-681.gc1ea6241.el8.x86_64 1/1 2021-02-10T04:19:54.903 INFO:teuthology.orchestra.run.smithi085.stdout: Installing : rbd-nbd-2:17.0.0-681.gc1ea6241.el8.x86_64 1/1 2021-02-10T04:19:55.120 INFO:teuthology.orchestra.run.smithi085.stdout: Running scriptlet: rbd-nbd-2:17.0.0-681.gc1ea6241.el8.x86_64 1/1 2021-02-10T04:19:55.218 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-mirror-2:17.0.0-681.gc1ea6241.el8.x86_64 1/1 2021-02-10T04:19:55.218 INFO:teuthology.orchestra.run.smithi164.stdout:Installed products updated. 2021-02-10T04:19:55.633 INFO:teuthology.orchestra.run.smithi085.stdout: Verifying : rbd-nbd-2:17.0.0-681.gc1ea6241.el8.x86_64 1/1 2021-02-10T04:19:55.633 INFO:teuthology.orchestra.run.smithi085.stdout:Installed products updated. 2021-02-10T04:19:57.105 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:20:01.224 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2021-02-10T04:20:01.225 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-mirror-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:20:01.225 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:20:01.225 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2021-02-10T04:20:01.226 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:20:01.227 INFO:teuthology.orchestra.run.smithi085.stdout:Installed: 2021-02-10T04:20:01.227 INFO:teuthology.orchestra.run.smithi085.stdout: rbd-nbd-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:20:01.227 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:20:01.227 INFO:teuthology.orchestra.run.smithi085.stdout:Complete! 2021-02-10T04:20:01.227 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install rbd-nbd 2021-02-10T04:20:01.228 DEBUG:teuthology.parallel:result is None 2021-02-10T04:20:01.537 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2021-02-10T04:20:02.111 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 2021-02-10T04:20:02.245 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 93 kB/s | 2.8 kB 00:00 2021-02-10T04:20:03.736 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2021-02-10T04:20:03.736 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2021-02-10T04:20:03.736 INFO:teuthology.orchestra.run.smithi164.stdout: Package Architecture Version Repository Size 2021-02-10T04:20:03.737 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2021-02-10T04:20:03.737 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2021-02-10T04:20:03.737 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-nbd x86_64 2:17.0.0-681.gc1ea6241.el8 ceph 174 k 2021-02-10T04:20:03.737 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:20:03.737 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2021-02-10T04:20:03.737 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2021-02-10T04:20:03.737 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2021-02-10T04:20:03.737 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:20:03.737 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 174 k 2021-02-10T04:20:03.738 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 649 k 2021-02-10T04:20:03.738 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2021-02-10T04:20:03.906 INFO:teuthology.orchestra.run.smithi164.stdout:rbd-nbd-17.0.0-681.gc1ea6241.el8.x86_64.rpm 1.0 MB/s | 174 kB 00:00 2021-02-10T04:20:03.907 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2021-02-10T04:20:03.907 INFO:teuthology.orchestra.run.smithi164.stdout:Total 1.0 MB/s | 174 kB 00:00 2021-02-10T04:20:03.908 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2021-02-10T04:20:03.920 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2021-02-10T04:20:03.920 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2021-02-10T04:20:03.974 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2021-02-10T04:20:03.978 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2021-02-10T04:20:04.099 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2021-02-10T04:20:04.324 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-nbd-2:17.0.0-681.gc1ea6241.el8.x86_64 1/1 2021-02-10T04:20:04.546 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: rbd-nbd-2:17.0.0-681.gc1ea6241.el8.x86_64 1/1 2021-02-10T04:20:05.036 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-nbd-2:17.0.0-681.gc1ea6241.el8.x86_64 1/1 2021-02-10T04:20:05.037 INFO:teuthology.orchestra.run.smithi164.stdout:Installed products updated. 2021-02-10T04:20:06.932 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:20:06.932 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2021-02-10T04:20:06.933 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-nbd-2:17.0.0-681.gc1ea6241.el8.x86_64 2021-02-10T04:20:06.933 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:20:06.933 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2021-02-10T04:20:07.065 DEBUG:teuthology.parallel:result is None 2021-02-10T04:20:07.066 DEBUG:teuthology.orchestra.run.smithi085:> sudo lsb_release -is 2021-02-10T04:20:07.112 INFO:teuthology.orchestra.run.smithi085.stdout:RedHatEnterprise 2021-02-10T04:20:07.114 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-02-10T04:20:07.115 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c1ea624123d412aff8b9d1430e36cb45fcab76b8 2021-02-10T04:20:07.283 DEBUG:teuthology.orchestra.run.smithi085:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-02-10T04:20:07.304 INFO:teuthology.orchestra.run.smithi085.stdout:17.0.0-681.gc1ea6241.el8 2021-02-10T04:20:07.305 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-681.gc1ea6241.el8 2021-02-10T04:20:07.305 INFO:teuthology.task.install:The correct ceph version 17.0.0-681.gc1ea6241 is installed. 2021-02-10T04:20:07.305 DEBUG:teuthology.orchestra.run.smithi164:> sudo lsb_release -is 2021-02-10T04:20:07.345 INFO:teuthology.orchestra.run.smithi164.stdout:RedHatEnterprise 2021-02-10T04:20:07.346 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-02-10T04:20:07.346 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c1ea624123d412aff8b9d1430e36cb45fcab76b8 2021-02-10T04:20:07.516 DEBUG:teuthology.orchestra.run.smithi164:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-02-10T04:20:07.535 INFO:teuthology.orchestra.run.smithi164.stdout:17.0.0-681.gc1ea6241.el8 2021-02-10T04:20:07.535 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-681.gc1ea6241.el8 2021-02-10T04:20:07.535 INFO:teuthology.task.install:The correct ceph version 17.0.0-681.gc1ea6241 is installed. 2021-02-10T04:20:07.535 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-02-10T04:20:07.541 DEBUG:teuthology.orchestra.run.smithi085:> set -ex 2021-02-10T04:20:07.541 DEBUG:teuthology.orchestra.run.smithi085:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-02-10T04:20:07.570 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2021-02-10T04:20:07.570 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-02-10T04:20:07.600 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-02-10T04:20:07.600 DEBUG:teuthology.orchestra.run.smithi085:> set -ex 2021-02-10T04:20:07.600 DEBUG:teuthology.orchestra.run.smithi085:> sudo dd of=/usr/bin/daemon-helper 2021-02-10T04:20:07.638 DEBUG:teuthology.orchestra.run.smithi085:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-02-10T04:20:07.704 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2021-02-10T04:20:07.705 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/daemon-helper 2021-02-10T04:20:07.728 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-02-10T04:20:07.792 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-02-10T04:20:07.792 DEBUG:teuthology.orchestra.run.smithi085:> set -ex 2021-02-10T04:20:07.792 DEBUG:teuthology.orchestra.run.smithi085:> sudo dd of=/usr/bin/adjust-ulimits 2021-02-10T04:20:07.819 DEBUG:teuthology.orchestra.run.smithi085:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-02-10T04:20:07.886 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2021-02-10T04:20:07.886 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/adjust-ulimits 2021-02-10T04:20:07.909 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-02-10T04:20:07.973 INFO:teuthology.run_tasks:Running task cephadm... 2021-02-10T04:20:08.079 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'global': {'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'c1ea624123d412aff8b9d1430e36cb45fcab76b8', 'cephadm_mode': 'root'} 2021-02-10T04:20:08.079 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 2021-02-10T04:20:08.080 INFO:tasks.cephadm:Cluster fsid is 42224fce-6b57-11eb-8feb-001a4aab830c 2021-02-10T04:20:08.080 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2021-02-10T04:20:08.080 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.85', 'mon.c': '[v2:172.21.15.85:3301,v1:172.21.15.85:6790]', 'mon.b': '172.21.15.164'} 2021-02-10T04:20:08.080 INFO:tasks.cephadm:First mon is mon.a on smithi085 2021-02-10T04:20:08.081 INFO:tasks.cephadm:First mgr is y 2021-02-10T04:20:08.081 INFO:tasks.cephadm:Normalizing hostnames... 2021-02-10T04:20:08.081 DEBUG:teuthology.orchestra.run.smithi085:> sudo hostname $(hostname -s) 2021-02-10T04:20:08.109 DEBUG:teuthology.orchestra.run.smithi164:> sudo hostname $(hostname -s) 2021-02-10T04:20:08.133 INFO:tasks.cephadm:Downloading cephadm (repo git://git.ceph.com/ceph-ci.git ref c1ea624123d412aff8b9d1430e36cb45fcab76b8)... 2021-02-10T04:20:08.133 DEBUG:teuthology.orchestra.run.smithi085:> git archive --remote=git://git.ceph.com/ceph-ci.git c1ea624123d412aff8b9d1430e36cb45fcab76b8 src/cephadm/cephadm | tar -xO src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm 2021-02-10T04:20:08.275 DEBUG:teuthology.orchestra.run.smithi164:> git archive --remote=git://git.ceph.com/ceph-ci.git c1ea624123d412aff8b9d1430e36cb45fcab76b8 src/cephadm/cephadm | tar -xO src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm 2021-02-10T04:20:08.393 DEBUG:teuthology.orchestra.run.smithi085:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2021-02-10T04:20:08.410 DEBUG:teuthology.orchestra.run.smithi164:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2021-02-10T04:20:08.522 DEBUG:teuthology.orchestra.run.smithi085:> sudo mkdir -p /etc/ceph 2021-02-10T04:20:08.552 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /etc/ceph 2021-02-10T04:20:08.576 DEBUG:teuthology.orchestra.run.smithi085:> sudo chmod 777 /etc/ceph 2021-02-10T04:20:08.620 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 777 /etc/ceph 2021-02-10T04:20:08.643 INFO:tasks.cephadm:Adding local image mirror docker-mirror.front.sepia.ceph.com:5000 2021-02-10T04:20:08.643 DEBUG:teuthology.orchestra.run.smithi085:> set -ex 2021-02-10T04:20:08.643 DEBUG:teuthology.orchestra.run.smithi085:> dd if=/etc/containers/registries.conf of=/dev/stdout 2021-02-10T04:20:08.677 DEBUG:teuthology.orchestra.run.smithi085:> set -ex 2021-02-10T04:20:12.126 DEBUG:teuthology.orchestra.run.smithi085:> sudo dd of=/etc/containers/registries.conf 2021-02-10T04:20:12.155 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2021-02-10T04:20:12.155 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/containers/registries.conf of=/dev/stdout 2021-02-10T04:20:12.173 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2021-02-10T04:20:12.173 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/containers/registries.conf 2021-02-10T04:20:12.240 INFO:tasks.cephadm:Writing seed config... 2021-02-10T04:20:12.244 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2021-02-10T04:20:12.244 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2021-02-10T04:20:12.244 INFO:tasks.cephadm: override: [global] ms bind msgr1 = False 2021-02-10T04:20:12.244 INFO:tasks.cephadm: override: [global] ms bind msgr2 = True 2021-02-10T04:20:12.244 INFO:tasks.cephadm: override: [global] ms type = async 2021-02-10T04:20:12.245 INFO:tasks.cephadm: override: [mon] debug mon = 20 2021-02-10T04:20:12.245 INFO:tasks.cephadm: override: [mon] debug ms = 1 2021-02-10T04:20:12.245 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2021-02-10T04:20:12.245 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2021-02-10T04:20:12.245 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2021-02-10T04:20:12.245 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2021-02-10T04:20:12.245 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2021-02-10T04:20:12.245 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2021-02-10T04:20:12.245 INFO:tasks.cephadm: override: [osd] debug filestore = 20 2021-02-10T04:20:12.245 INFO:tasks.cephadm: override: [osd] debug journal = 20 2021-02-10T04:20:12.245 INFO:tasks.cephadm: override: [osd] debug ms = 20 2021-02-10T04:20:12.245 INFO:tasks.cephadm: override: [osd] debug osd = 25 2021-02-10T04:20:12.246 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2021-02-10T04:20:12.246 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2021-02-10T04:20:12.246 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2021-02-10T04:20:12.246 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2021-02-10T04:20:12.246 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2021-02-10T04:20:12.246 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2021-02-10T04:20:12.246 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2021-02-10T04:20:12.248 DEBUG:teuthology.orchestra.run.smithi085:> set -ex 2021-02-10T04:20:12.248 DEBUG:teuthology.orchestra.run.smithi085:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2021-02-10T04:20:12.265 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 42224fce-6b57-11eb-8feb-001a4aab830c ms bind msgr1 = False ms bind msgr2 = True ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug filestore = 20 debug journal = 20 debug ms = 20 debug osd = 25 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2021-02-10T04:20:12.265 DEBUG:teuthology.orchestra.run.smithi085:mon.a> sudo journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mon.a.service 2021-02-10T04:20:12.308 DEBUG:teuthology.orchestra.run.smithi085:mgr.y> sudo journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mgr.y.service 2021-02-10T04:20:12.336 INFO:journalctl@ceph.mon.a.smithi085.stdout:-- Logs begin at Wed 2021-02-10 04:09:23 UTC. -- 2021-02-10T04:20:12.336 INFO:tasks.cephadm:Bootstrapping... 2021-02-10T04:20:12.336 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -v bootstrap --fsid 42224fce-6b57-11eb-8feb-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.85 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2021-02-10T04:20:12.362 INFO:journalctl@ceph.mgr.y.smithi085.stdout:-- Logs begin at Wed 2021-02-10 04:09:23 UTC. -- 2021-02-10T04:20:12.616 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman --version 2021-02-10T04:20:13.193 INFO:teuthology.orchestra.run.smithi085.stderr:/bin/podman: podman version 2.0.5 2021-02-10T04:20:13.195 INFO:teuthology.orchestra.run.smithi085.stderr:Verifying podman|docker is present... 2021-02-10T04:20:13.195 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman --version 2021-02-10T04:20:13.229 INFO:teuthology.orchestra.run.smithi085.stderr:/bin/podman: podman version 2.0.5 2021-02-10T04:20:13.231 INFO:teuthology.orchestra.run.smithi085.stderr:Verifying lvm2 is present... 2021-02-10T04:20:13.232 INFO:teuthology.orchestra.run.smithi085.stderr:Verifying time synchronization is in place... 2021-02-10T04:20:13.232 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl is-enabled chrony.service 2021-02-10T04:20:13.239 INFO:teuthology.orchestra.run.smithi085.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2021-02-10T04:20:13.239 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl is-active chrony.service 2021-02-10T04:20:13.245 INFO:teuthology.orchestra.run.smithi085.stderr:systemctl: inactive 2021-02-10T04:20:13.245 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl is-enabled chronyd.service 2021-02-10T04:20:13.251 INFO:teuthology.orchestra.run.smithi085.stderr:systemctl: enabled 2021-02-10T04:20:13.252 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl is-active chronyd.service 2021-02-10T04:20:13.256 INFO:teuthology.orchestra.run.smithi085.stderr:systemctl: active 2021-02-10T04:20:13.257 INFO:teuthology.orchestra.run.smithi085.stderr:Unit chronyd.service is enabled and running 2021-02-10T04:20:13.257 INFO:teuthology.orchestra.run.smithi085.stderr:Repeating the final host check... 2021-02-10T04:20:13.257 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman --version 2021-02-10T04:20:13.288 INFO:teuthology.orchestra.run.smithi085.stderr:/bin/podman: podman version 2.0.5 2021-02-10T04:20:13.290 INFO:teuthology.orchestra.run.smithi085.stderr:podman|docker (/bin/podman) is present 2021-02-10T04:20:13.290 INFO:teuthology.orchestra.run.smithi085.stderr:systemctl is present 2021-02-10T04:20:13.290 INFO:teuthology.orchestra.run.smithi085.stderr:lvcreate is present 2021-02-10T04:20:13.290 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl is-enabled chrony.service 2021-02-10T04:20:13.296 INFO:teuthology.orchestra.run.smithi085.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2021-02-10T04:20:13.296 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl is-active chrony.service 2021-02-10T04:20:13.301 INFO:teuthology.orchestra.run.smithi085.stderr:systemctl: inactive 2021-02-10T04:20:13.302 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl is-enabled chronyd.service 2021-02-10T04:20:13.307 INFO:teuthology.orchestra.run.smithi085.stderr:systemctl: enabled 2021-02-10T04:20:13.307 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl is-active chronyd.service 2021-02-10T04:20:13.312 INFO:teuthology.orchestra.run.smithi085.stderr:systemctl: active 2021-02-10T04:20:13.313 INFO:teuthology.orchestra.run.smithi085.stderr:Unit chronyd.service is enabled and running 2021-02-10T04:20:13.313 INFO:teuthology.orchestra.run.smithi085.stderr:Host looks OK 2021-02-10T04:20:13.313 INFO:teuthology.orchestra.run.smithi085.stderr:Cluster fsid: 42224fce-6b57-11eb-8feb-001a4aab830c 2021-02-10T04:20:13.313 INFO:teuthology.orchestra.run.smithi085.stderr:Acquiring lock 140199189940656 on /run/cephadm/42224fce-6b57-11eb-8feb-001a4aab830c.lock 2021-02-10T04:20:13.320 INFO:teuthology.orchestra.run.smithi085.stderr:Lock 140199189940656 acquired on /run/cephadm/42224fce-6b57-11eb-8feb-001a4aab830c.lock 2021-02-10T04:20:13.321 INFO:teuthology.orchestra.run.smithi085.stderr:Verifying IP 172.21.15.85 port 3300 ... 2021-02-10T04:20:13.321 INFO:teuthology.orchestra.run.smithi085.stderr:Verifying IP 172.21.15.85 port 6789 ... 2021-02-10T04:20:13.321 INFO:teuthology.orchestra.run.smithi085.stderr:Base mon IP is 172.21.15.85, final addrv is [v2:172.21.15.85:3300,v1:172.21.15.85:6789] 2021-02-10T04:20:13.321 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /sbin/ip route ls 2021-02-10T04:20:13.321 INFO:teuthology.orchestra.run.smithi085.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp metric 100 2021-02-10T04:20:13.321 INFO:teuthology.orchestra.run.smithi085.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.85 metric 100 2021-02-10T04:20:13.321 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /sbin/ip -6 route ls 2021-02-10T04:20:13.321 INFO:teuthology.orchestra.run.smithi085.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2021-02-10T04:20:13.321 INFO:teuthology.orchestra.run.smithi085.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2021-02-10T04:20:13.321 INFO:teuthology.orchestra.run.smithi085.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1796sec hoplimit 64 pref medium 2021-02-10T04:20:13.321 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /sbin/ip -6 addr ls 2021-02-10T04:20:13.322 INFO:teuthology.orchestra.run.smithi085.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2021-02-10T04:20:13.323 INFO:teuthology.orchestra.run.smithi085.stderr:/sbin/ip: inet6 ::1/128 scope host 2021-02-10T04:20:13.323 INFO:teuthology.orchestra.run.smithi085.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2021-02-10T04:20:13.323 INFO:teuthology.orchestra.run.smithi085.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2021-02-10T04:20:13.323 INFO:teuthology.orchestra.run.smithi085.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:d7bb/64 scope link 2021-02-10T04:20:13.323 INFO:teuthology.orchestra.run.smithi085.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2021-02-10T04:20:13.323 INFO:teuthology.orchestra.run.smithi085.stderr:Mon IP 172.21.15.85 is in CIDR network 172.21.0.0/20 2021-02-10T04:20:13.323 INFO:teuthology.orchestra.run.smithi085.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2021-02-10T04:20:13.323 INFO:teuthology.orchestra.run.smithi085.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8... 2021-02-10T04:20:13.324 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman pull quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 2021-02-10T04:20:14.518 INFO:teuthology.orchestra.run.smithi085.stderr:/bin/podman: Trying to pull quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8... 2021-02-10T04:20:14.607 INFO:teuthology.orchestra.run.smithi085.stderr:/bin/podman: Getting image source signatures 2021-02-10T04:20:14.706 INFO:teuthology.orchestra.run.smithi085.stderr:/bin/podman: Copying blob sha256:889524391d554cc9bd269d2e47820d562c512d9d60fd7d4da881803a833083d7 2021-02-10T04:20:14.757 INFO:teuthology.orchestra.run.smithi085.stderr:/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-02-10T04:20:14.773 INFO:teuthology.orchestra.run.smithi085.stderr:/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-02-10T04:20:14.790 INFO:teuthology.orchestra.run.smithi085.stderr:/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-02-10T04:20:14.806 INFO:teuthology.orchestra.run.smithi085.stderr:/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-02-10T04:20:14.823 INFO:teuthology.orchestra.run.smithi085.stderr:/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-02-10T04:20:14.873 INFO:teuthology.orchestra.run.smithi085.stderr:/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-02-10T04:20:14.907 INFO:teuthology.orchestra.run.smithi085.stderr:/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-02-10T04:20:14.907 INFO:teuthology.orchestra.run.smithi085.stderr:/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-02-10T04:20:14.907 INFO:teuthology.orchestra.run.smithi085.stderr:/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-02-10T04:20:14.907 INFO:teuthology.orchestra.run.smithi085.stderr:/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-02-10T04:20:14.907 INFO:teuthology.orchestra.run.smithi085.stderr:/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-02-10T04:20:14.908 INFO:teuthology.orchestra.run.smithi085.stderr:/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-02-10T04:20:14.908 INFO:teuthology.orchestra.run.smithi085.stderr:/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-02-10T04:20:14.908 INFO:teuthology.orchestra.run.smithi085.stderr:/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-02-10T04:20:14.908 INFO:teuthology.orchestra.run.smithi085.stderr:/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-02-10T04:20:15.031 INFO:teuthology.orchestra.run.smithi085.stderr:/bin/podman: Copying blob sha256:881d8923ea9dd9278093c96f431aa02d928e2bb97f4c27b1868dc23608408b18 2021-02-10T04:20:32.112 INFO:teuthology.orchestra.run.smithi085.stderr:/bin/podman: Writing manifest to image destination 2021-02-10T04:20:32.113 INFO:teuthology.orchestra.run.smithi085.stderr:/bin/podman: Storing signatures 2021-02-10T04:20:41.417 INFO:teuthology.orchestra.run.smithi085.stderr:/bin/podman: 20674628c8f0a3d61c7a66ac0dc2b9a1a348791abf3854f4f410ac3e15e09fd5 2021-02-10T04:20:41.430 INFO:teuthology.orchestra.run.smithi085.stderr:Extracting ceph user uid/gid from container image... 2021-02-10T04:20:41.430 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -e NODE_NAME=smithi085 quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -c %u %g /var/lib/ceph 2021-02-10T04:20:42.336 INFO:teuthology.orchestra.run.smithi085.stderr:stat: 167 167 2021-02-10T04:20:52.169 INFO:teuthology.orchestra.run.smithi085.stderr:Creating initial keys... 2021-02-10T04:20:52.169 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -e NODE_NAME=smithi085 quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 --gen-print-key 2021-02-10T04:20:52.664 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-authtool: AQAkXyNgeB5IJxAA+Q+FBPj7xePPxqCz+/SQnw== 2021-02-10T04:20:53.097 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -e NODE_NAME=smithi085 quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 --gen-print-key 2021-02-10T04:20:53.597 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-authtool: AQAlXyNgNt5BIxAA3oqoiJQnGg2dkhygTn2Z7g== 2021-02-10T04:20:54.024 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -e NODE_NAME=smithi085 quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 --gen-print-key 2021-02-10T04:20:54.555 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-authtool: AQAmXyNg2DbDIBAAkzYpJIj6crks1PpC33CAgA== 2021-02-10T04:20:54.966 INFO:teuthology.orchestra.run.smithi085.stderr:Creating initial monmap... 2021-02-10T04:20:54.966 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -e NODE_NAME=smithi085 -v /tmp/ceph-tmpwhwgd624:/tmp/monmap:z quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 --create --clobber --fsid 42224fce-6b57-11eb-8feb-001a4aab830c --addv a [v2:172.21.15.85:3300,v1:172.21.15.85:6789] /tmp/monmap 2021-02-10T04:20:55.505 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2021-02-10T04:20:55.506 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 42224fce-6b57-11eb-8feb-001a4aab830c 2021-02-10T04:20:55.506 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2021-02-10T04:20:55.947 INFO:teuthology.orchestra.run.smithi085.stderr:monmaptool for a [v2:172.21.15.85:3300,v1:172.21.15.85:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2021-02-10T04:20:55.948 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/monmaptool: set fsid to 42224fce-6b57-11eb-8feb-001a4aab830c 2021-02-10T04:20:55.948 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2021-02-10T04:20:55.949 INFO:teuthology.orchestra.run.smithi085.stderr: 2021-02-10T04:20:55.949 INFO:teuthology.orchestra.run.smithi085.stderr:Creating mon... 2021-02-10T04:20:55.949 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -e NODE_NAME=smithi085 -v /var/log/ceph/42224fce-6b57-11eb-8feb-001a4aab830c:/var/log/ceph:z -v /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z -v /tmp/ceph-tmpo5c7dsrg:/tmp/keyring:z -v /tmp/ceph-tmpwhwgd624:/tmp/monmap:z quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 --mkfs -i a --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix="debug " --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2021-02-10T04:20:56.431 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.423+0000 7f12bddb1700 0 set uid:gid to 167:167 (ceph:ceph) 2021-02-10T04:20:56.431 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.424+0000 7f12bddb1700 1 imported monmap: 2021-02-10T04:20:56.432 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: epoch 0 2021-02-10T04:20:56.432 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: fsid 42224fce-6b57-11eb-8feb-001a4aab830c 2021-02-10T04:20:56.432 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: last_changed 2021-02-10T04:20:55.445955+0000 2021-02-10T04:20:56.432 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: created 2021-02-10T04:20:55.445955+0000 2021-02-10T04:20:56.432 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2021-02-10T04:20:56.432 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: election_strategy: 1 2021-02-10T04:20:56.432 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0] mon.a 2021-02-10T04:20:56.432 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: 2021-02-10T04:20:56.454 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.424+0000 7f12bddb1700 0 /usr/bin/ceph-mon: set fsid to 42224fce-6b57-11eb-8feb-001a4aab830c 2021-02-10T04:20:56.454 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.424+0000 7f12bddb1700 4 rocksdb: RocksDB version: 6.8.1 2021-02-10T04:20:56.454 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: 2021-02-10T04:20:56.454 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.424+0000 7f12bddb1700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-02-10T04:20:56.454 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.424+0000 7f12bddb1700 4 rocksdb: Compile date Feb 9 2021 2021-02-10T04:20:56.454 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.424+0000 7f12bddb1700 4 rocksdb: DB SUMMARY 2021-02-10T04:20:56.454 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: 2021-02-10T04:20:56.454 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.424+0000 7f12bddb1700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 0, files: 2021-02-10T04:20:56.454 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: 2021-02-10T04:20:56.454 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.424+0000 7f12bddb1700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 2021-02-10T04:20:56.454 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: 2021-02-10T04:20:56.455 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.424+0000 7f12bddb1700 4 rocksdb: Options.error_if_exists: 0 2021-02-10T04:20:56.455 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.424+0000 7f12bddb1700 4 rocksdb: Options.create_if_missing: 1 2021-02-10T04:20:56.455 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.424+0000 7f12bddb1700 4 rocksdb: Options.paranoid_checks: 1 2021-02-10T04:20:56.455 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.424+0000 7f12bddb1700 4 rocksdb: Options.env: 0x55e48cbb91c0 2021-02-10T04:20:56.455 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.424+0000 7f12bddb1700 4 rocksdb: Options.fs: Posix File System 2021-02-10T04:20:56.455 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.424+0000 7f12bddb1700 4 rocksdb: Options.info_log: 0x55e48f1dea40 2021-02-10T04:20:56.455 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.424+0000 7f12bddb1700 4 rocksdb: Options.max_file_opening_threads: 16 2021-02-10T04:20:56.455 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.statistics: (nil) 2021-02-10T04:20:56.455 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.use_fsync: 0 2021-02-10T04:20:56.455 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.max_log_file_size: 0 2021-02-10T04:20:56.455 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-02-10T04:20:56.455 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-02-10T04:20:56.455 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.keep_log_file_num: 1000 2021-02-10T04:20:56.456 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.recycle_log_file_num: 0 2021-02-10T04:20:56.456 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.allow_fallocate: 1 2021-02-10T04:20:56.456 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.allow_mmap_reads: 0 2021-02-10T04:20:56.456 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.allow_mmap_writes: 0 2021-02-10T04:20:56.456 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.use_direct_reads: 0 2021-02-10T04:20:56.456 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-02-10T04:20:56.456 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.create_missing_column_families: 0 2021-02-10T04:20:56.456 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.db_log_dir: 2021-02-10T04:20:56.456 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2021-02-10T04:20:56.456 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-02-10T04:20:56.456 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.max_subcompactions: 1 2021-02-10T04:20:56.456 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.max_background_flushes: -1 2021-02-10T04:20:56.456 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-02-10T04:20:56.457 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-02-10T04:20:56.457 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-02-10T04:20:56.457 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-02-10T04:20:56.457 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-02-10T04:20:56.457 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.advise_random_on_open: 1 2021-02-10T04:20:56.457 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.db_write_buffer_size: 0 2021-02-10T04:20:56.457 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.write_buffer_manager: 0x55e48f1a5230 2021-02-10T04:20:56.457 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-02-10T04:20:56.457 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-02-10T04:20:56.457 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-02-10T04:20:56.457 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-02-10T04:20:56.457 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.rate_limiter: (nil) 2021-02-10T04:20:56.457 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-02-10T04:20:56.458 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.wal_recovery_mode: 2 2021-02-10T04:20:56.458 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.enable_thread_tracking: 0 2021-02-10T04:20:56.458 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.enable_pipelined_write: 0 2021-02-10T04:20:56.458 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.unordered_write: 0 2021-02-10T04:20:56.458 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-02-10T04:20:56.458 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-02-10T04:20:56.458 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-02-10T04:20:56.458 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-02-10T04:20:56.458 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.row_cache: None 2021-02-10T04:20:56.458 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.wal_filter: None 2021-02-10T04:20:56.458 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-02-10T04:20:56.458 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.allow_ingest_behind: 0 2021-02-10T04:20:56.458 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.preserve_deletes: 0 2021-02-10T04:20:56.459 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.two_write_queues: 0 2021-02-10T04:20:56.459 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.manual_wal_flush: 0 2021-02-10T04:20:56.459 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.atomic_flush: 0 2021-02-10T04:20:56.459 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-02-10T04:20:56.459 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-02-10T04:20:56.459 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-02-10T04:20:56.459 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.log_readahead_size: 0 2021-02-10T04:20:56.459 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-02-10T04:20:56.459 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.max_background_jobs: 2 2021-02-10T04:20:56.459 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.max_background_compactions: -1 2021-02-10T04:20:56.459 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-02-10T04:20:56.459 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-02-10T04:20:56.460 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-02-10T04:20:56.460 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.max_total_wal_size: 0 2021-02-10T04:20:56.460 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-02-10T04:20:56.460 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-02-10T04:20:56.460 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-02-10T04:20:56.460 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-02-10T04:20:56.460 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.max_open_files: -1 2021-02-10T04:20:56.460 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.bytes_per_sync: 0 2021-02-10T04:20:56.460 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-02-10T04:20:56.460 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-02-10T04:20:56.460 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.compaction_readahead_size: 0 2021-02-10T04:20:56.460 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Compression algorithms supported: 2021-02-10T04:20:56.460 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-02-10T04:20:56.460 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: kZSTD supported: 0 2021-02-10T04:20:56.461 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: kXpressCompression supported: 0 2021-02-10T04:20:56.461 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: kLZ4HCCompression supported: 1 2021-02-10T04:20:56.461 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: kLZ4Compression supported: 1 2021-02-10T04:20:56.461 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: kBZip2Compression supported: 0 2021-02-10T04:20:56.461 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: kZlibCompression supported: 1 2021-02-10T04:20:56.461 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: kSnappyCompression supported: 1 2021-02-10T04:20:56.461 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-02-10T04:20:56.461 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2021-02-10T04:20:56.461 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: 2021-02-10T04:20:56.461 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 2021-02-10T04:20:56.461 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: 2021-02-10T04:20:56.461 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-02-10T04:20:56.461 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: 2021-02-10T04:20:56.462 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-02-10T04:20:56.462 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.merge_operator: 2021-02-10T04:20:56.462 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.compaction_filter: None 2021-02-10T04:20:56.462 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.compaction_filter_factory: None 2021-02-10T04:20:56.462 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-02-10T04:20:56.462 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-02-10T04:20:56.462 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e48f0f01c8) 2021-02-10T04:20:56.462 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2021-02-10T04:20:56.462 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2021-02-10T04:20:56.462 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2021-02-10T04:20:56.462 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2021-02-10T04:20:56.462 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: index_type: 0 2021-02-10T04:20:56.462 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2021-02-10T04:20:56.463 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: index_shortening: 1 2021-02-10T04:20:56.463 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2021-02-10T04:20:56.463 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2021-02-10T04:20:56.463 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: checksum: 1 2021-02-10T04:20:56.463 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2021-02-10T04:20:56.463 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: block_cache: 0x55e48f12a110 2021-02-10T04:20:56.463 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2021-02-10T04:20:56.463 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: block_cache_options: 2021-02-10T04:20:56.463 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: capacity : 536870912 2021-02-10T04:20:56.463 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2021-02-10T04:20:56.463 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2021-02-10T04:20:56.463 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2021-02-10T04:20:56.463 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2021-02-10T04:20:56.464 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2021-02-10T04:20:56.464 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: block_size: 4096 2021-02-10T04:20:56.464 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2021-02-10T04:20:56.464 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2021-02-10T04:20:56.464 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2021-02-10T04:20:56.464 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2021-02-10T04:20:56.464 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: partition_filters: 0 2021-02-10T04:20:56.464 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2021-02-10T04:20:56.464 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2021-02-10T04:20:56.464 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2021-02-10T04:20:56.464 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: verify_compression: 0 2021-02-10T04:20:56.464 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2021-02-10T04:20:56.464 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: format_version: 2 2021-02-10T04:20:56.465 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2021-02-10T04:20:56.465 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: block_align: 0 2021-02-10T04:20:56.465 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: 2021-02-10T04:20:56.465 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.write_buffer_size: 33554432 2021-02-10T04:20:56.465 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.max_write_buffer_number: 2 2021-02-10T04:20:56.465 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.compression: NoCompression 2021-02-10T04:20:56.465 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.bottommost_compression: Disabled 2021-02-10T04:20:56.465 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.prefix_extractor: nullptr 2021-02-10T04:20:56.465 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-02-10T04:20:56.465 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.num_levels: 7 2021-02-10T04:20:56.465 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-02-10T04:20:56.465 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-02-10T04:20:56.465 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-02-10T04:20:56.466 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-02-10T04:20:56.466 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-02-10T04:20:56.466 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-02-10T04:20:56.466 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-02-10T04:20:56.466 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-02-10T04:20:56.466 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-02-10T04:20:56.466 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-02-10T04:20:56.466 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.compression_opts.level: 32767 2021-02-10T04:20:56.466 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.compression_opts.strategy: 0 2021-02-10T04:20:56.466 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-02-10T04:20:56.466 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-02-10T04:20:56.466 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.compression_opts.enabled: false 2021-02-10T04:20:56.466 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-02-10T04:20:56.467 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-02-10T04:20:56.467 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-02-10T04:20:56.467 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.target_file_size_base: 67108864 2021-02-10T04:20:56.467 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-02-10T04:20:56.467 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-02-10T04:20:56.467 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-02-10T04:20:56.467 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-02-10T04:20:56.467 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-02-10T04:20:56.467 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-02-10T04:20:56.467 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-02-10T04:20:56.467 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-02-10T04:20:56.467 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-02-10T04:20:56.467 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-02-10T04:20:56.468 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-02-10T04:20:56.468 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-02-10T04:20:56.468 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-02-10T04:20:56.468 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.arena_block_size: 4194304 2021-02-10T04:20:56.468 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-02-10T04:20:56.468 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-02-10T04:20:56.468 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-02-10T04:20:56.468 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.disable_auto_compactions: 0 2021-02-10T04:20:56.468 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-02-10T04:20:56.468 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-02-10T04:20:56.468 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-02-10T04:20:56.468 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-02-10T04:20:56.468 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-02-10T04:20:56.469 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-02-10T04:20:56.469 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-02-10T04:20:56.469 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-02-10T04:20:56.469 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-02-10T04:20:56.469 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-02-10T04:20:56.469 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.table_properties_collectors: 2021-02-10T04:20:56.469 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.inplace_update_support: 0 2021-02-10T04:20:56.469 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-02-10T04:20:56.469 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-02-10T04:20:56.469 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-02-10T04:20:56.469 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-02-10T04:20:56.469 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.bloom_locality: 0 2021-02-10T04:20:56.469 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.max_successive_merges: 0 2021-02-10T04:20:56.469 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-02-10T04:20:56.470 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.paranoid_file_checks: 0 2021-02-10T04:20:56.470 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.force_consistency_checks: 0 2021-02-10T04:20:56.470 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.report_bg_io_stats: 0 2021-02-10T04:20:56.470 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.ttl: 2592000 2021-02-10T04:20:56.470 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.425+0000 7f12bddb1700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-02-10T04:20:56.470 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.426+0000 7f12bddb1700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-02-10T04:20:56.470 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: 2021-02-10T04:20:56.470 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.426+0000 7f12bddb1700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2021-02-10T04:20:56.470 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: 2021-02-10T04:20:56.470 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.426+0000 7f12bddb1700 4 rocksdb: DB pointer 0x55e48f1f1800 2021-02-10T04:20:56.470 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.426+0000 7f12a6c50700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-02-10T04:20:56.470 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.426+0000 7f12a6c50700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-02-10T04:20:56.471 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2021-02-10T04:20:56.471 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2021-02-10T04:20:56.471 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-02-10T04:20:56.471 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-02-10T04:20:56.471 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-02-10T04:20:56.471 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-02-10T04:20:56.471 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-02-10T04:20:56.471 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-02-10T04:20:56.471 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: 2021-02-10T04:20:56.471 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-02-10T04:20:56.471 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-02-10T04:20:56.471 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-02-10T04:20:56.471 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-02-10T04:20:56.471 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-02-10T04:20:56.472 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: 2021-02-10T04:20:56.472 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-02-10T04:20:56.472 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-02-10T04:20:56.472 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-02-10T04:20:56.472 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2021-02-10T04:20:56.472 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2021-02-10T04:20:56.472 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2021-02-10T04:20:56.472 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2021-02-10T04:20:56.472 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2021-02-10T04:20:56.472 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2021-02-10T04:20:56.472 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-02-10T04:20:56.472 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-02-10T04:20:56.472 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-02-10T04:20:56.473 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: 2021-02-10T04:20:56.473 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2021-02-10T04:20:56.473 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: 2021-02-10T04:20:56.473 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-02-10T04:20:56.473 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-02-10T04:20:56.473 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-02-10T04:20:56.473 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-02-10T04:20:56.473 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-02-10T04:20:56.473 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: 2021-02-10T04:20:56.473 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-02-10T04:20:56.473 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-02-10T04:20:56.473 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-02-10T04:20:56.473 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2021-02-10T04:20:56.473 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2021-02-10T04:20:56.474 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2021-02-10T04:20:56.474 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2021-02-10T04:20:56.474 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2021-02-10T04:20:56.474 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2021-02-10T04:20:56.474 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-02-10T04:20:56.474 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-02-10T04:20:56.474 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-02-10T04:20:56.474 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: 2021-02-10T04:20:56.474 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2021-02-10T04:20:56.474 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: 2021-02-10T04:20:56.474 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.428+0000 7f12bddb1700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2021-02-10T04:20:56.474 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.428+0000 7f12bddb1700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2021-02-10T04:20:56.474 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph-mon: "debug "2021-02-10T04:20:56.428+0000 7f12bddb1700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a 2021-02-10T04:20:57.019 INFO:teuthology.orchestra.run.smithi085.stderr:create mon.a on 2021-02-10T04:20:57.020 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/42224fce-6b57-11eb-8feb-001a4aab830c 2021-02-10T04:20:57.025 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl disable ceph.target 2021-02-10T04:20:57.032 INFO:teuthology.orchestra.run.smithi085.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2021-02-10T04:20:57.113 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl enable ceph.target 2021-02-10T04:20:57.121 INFO:teuthology.orchestra.run.smithi085.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2021-02-10T04:20:57.201 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl start ceph.target 2021-02-10T04:20:57.208 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl enable ceph-42224fce-6b57-11eb-8feb-001a4aab830c.target 2021-02-10T04:20:57.214 INFO:teuthology.orchestra.run.smithi085.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-42224fce-6b57-11eb-8feb-001a4aab830c.target → /etc/systemd/system/ceph-42224fce-6b57-11eb-8feb-001a4aab830c.target. 2021-02-10T04:20:57.214 INFO:teuthology.orchestra.run.smithi085.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-42224fce-6b57-11eb-8feb-001a4aab830c.target → /etc/systemd/system/ceph-42224fce-6b57-11eb-8feb-001a4aab830c.target. 2021-02-10T04:20:57.295 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl start ceph-42224fce-6b57-11eb-8feb-001a4aab830c.target 2021-02-10T04:20:57.302 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl daemon-reload 2021-02-10T04:20:57.387 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl stop ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mon.a 2021-02-10T04:20:57.407 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl reset-failed ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mon.a 2021-02-10T04:20:57.412 INFO:teuthology.orchestra.run.smithi085.stderr:systemctl: Failed to reset failed state of unit ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mon.a.service: Unit ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mon.a.service not loaded. 2021-02-10T04:20:57.413 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl enable ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mon.a 2021-02-10T04:20:57.419 INFO:teuthology.orchestra.run.smithi085.stderr:systemctl: Created symlink /etc/systemd/system/ceph-42224fce-6b57-11eb-8feb-001a4aab830c.target.wants/ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mon.a.service → /etc/systemd/system/ceph-42224fce-6b57-11eb-8feb-001a4aab830c@.service. 2021-02-10T04:20:57.496 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl start ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mon.a 2021-02-10T04:20:57.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:20:57 smithi085 systemd[1]: Starting Ceph mon.a for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T04:20:57.912 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl is-enabled firewalld.service 2021-02-10T04:20:57.922 INFO:teuthology.orchestra.run.smithi085.stderr:systemctl: disabled 2021-02-10T04:20:57.922 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl is-active firewalld.service 2021-02-10T04:20:57.929 INFO:teuthology.orchestra.run.smithi085.stderr:systemctl: inactive 2021-02-10T04:20:57.930 INFO:teuthology.orchestra.run.smithi085.stderr:firewalld.service is not enabled 2021-02-10T04:20:57.930 INFO:teuthology.orchestra.run.smithi085.stderr:Not possible to enable service . firewalld.service is not available 2021-02-10T04:20:57.930 INFO:teuthology.orchestra.run.smithi085.stderr:Not possible to open ports <[]>. firewalld.service is not available 2021-02-10T04:20:57.930 INFO:teuthology.orchestra.run.smithi085.stderr:Waiting for mon to start... 2021-02-10T04:20:57.930 INFO:teuthology.orchestra.run.smithi085.stderr:Waiting for mon... 2021-02-10T04:20:57.931 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -e NODE_NAME=smithi085 -v /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z -v /tmp/ceph-tmpqlcrjfio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpsf4d5t6m:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 status 2021-02-10T04:20:58.138 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:20:57 smithi085 bash[28501]: 72df038b1f52230981853929aa1050c60581ddb8c4bb1b318bfa7d99413e3cb3 2021-02-10T04:20:58.138 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:20:57 smithi085 systemd[1]: Started Ceph mon.a for 42224fce-6b57-11eb-8feb-001a4aab830c. 2021-02-10T04:20:58.138 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:20:57 smithi085 conmon[28613]: cluster 2021-02-10T04:20:57.933496+0000 mon.a (mon.0 2021-02-10T04:20:58.138 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:20:57 smithi085 conmon[28613]: ) 0 : [INF] mkfs 42224fce-6b57-11eb-8feb-001a4aab830c 2021-02-10T04:20:58.138 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:20:57 smithi085 conmon[28613]: cluster 2021-02-10T04:20:57.920758+0000 mon.a ( 2021-02-10T04:20:58.138 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:20:57 smithi085 conmon[28613]: mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2021-02-10T04:20:58.607 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: cluster: 2021-02-10T04:20:58.607 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: id: 42224fce-6b57-11eb-8feb-001a4aab830c 2021-02-10T04:20:58.607 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: health: HEALTH_OK 2021-02-10T04:20:58.607 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: 2021-02-10T04:20:58.607 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: services: 2021-02-10T04:20:58.607 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.675354s) 2021-02-10T04:20:58.607 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: mgr: no daemons active 2021-02-10T04:20:58.607 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2021-02-10T04:20:58.607 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: 2021-02-10T04:20:58.607 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: data: 2021-02-10T04:20:58.608 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2021-02-10T04:20:58.608 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2021-02-10T04:20:58.608 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2021-02-10T04:20:58.608 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: pgs: 2021-02-10T04:20:58.608 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: 2021-02-10T04:20:59.344 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:20:58 smithi085 conmon[28613]: cluster 2021-02-10T04:20:57.930795+0000 mon.a ( 2021-02-10T04:20:59.344 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:20:58 smithi085 conmon[28613]: mon.0) 2 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2021-02-10T04:20:59.344 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:20:58 smithi085 conmon[28613]: cluster 2021-02-10T04:20:57.932902+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {a=[v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0]} 2021-02-10T04:20:59.344 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:20:58 smithi085 conmon[28613]: cluster 2021-02-10T04:20:57.939703+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap 2021-02-10T04:20:59.344 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:20:58 smithi085 conmon[28613]: cluster 2021-02-10T04:20:57.945894+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2021-02-10T04:20:59.345 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:20:58 smithi085 conmon[28613]: cluster 2021-02-10T04:20:57.946379+0000 mon.a (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2021-02-10T04:20:59.345 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:20:58 smithi085 conmon[28613]: audit 2021-02-10T04:20:58.606117+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? 172.21.15.85:0/2271463435' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2021-02-10T04:20:59.345 INFO:teuthology.orchestra.run.smithi085.stderr:mon is available 2021-02-10T04:20:59.345 INFO:teuthology.orchestra.run.smithi085.stderr:Assimilating anything we can from ceph.conf... 2021-02-10T04:20:59.345 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -e NODE_NAME=smithi085 -v /var/log/ceph/42224fce-6b57-11eb-8feb-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpqlcrjfio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpsf4d5t6m:/etc/ceph/ceph.conf:z -v /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 config assimilate-conf -i /var/lib/ceph/mon/ceph-a/config 2021-02-10T04:21:00.005 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: 2021-02-10T04:21:00.005 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: [global] 2021-02-10T04:21:00.005 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: fsid = 42224fce-6b57-11eb-8feb-001a4aab830c 2021-02-10T04:21:00.005 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.85:3300,v1:172.21.15.85:6789] 2021-02-10T04:21:00.005 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2021-02-10T04:21:00.005 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2021-02-10T04:21:00.005 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2021-02-10T04:21:00.006 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2021-02-10T04:21:00.006 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: 2021-02-10T04:21:00.006 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: [mgr] 2021-02-10T04:21:00.006 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2021-02-10T04:21:00.006 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: 2021-02-10T04:21:00.006 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: [osd] 2021-02-10T04:21:00.006 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2021-02-10T04:21:00.006 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2021-02-10T04:21:00.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:00 smithi085 conmon[28613]: audit 2021-02-10T04:21:00.001950+0000 mon.a (mon.0) 8 : audit [INF] from='client.? 172.21.15.85:0/179137251' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2021-02-10T04:21:00.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:00 smithi085 conmon[28613]: audit 2021-02-10T04: 2021-02-10T04:21:00.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:00 smithi085 conmon[28613]: 21:00.004519+0000 mon.a (mon.0) 9 : audit [INF] from='client.? 172.21.15.85:0/179137251' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2021-02-10T04:21:00.557 INFO:teuthology.orchestra.run.smithi085.stderr:Generating new minimal ceph.conf... 2021-02-10T04:21:00.557 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -e NODE_NAME=smithi085 -v /var/log/ceph/42224fce-6b57-11eb-8feb-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpqlcrjfio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpsf4d5t6m:/etc/ceph/ceph.conf:z -v /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-a/config 2021-02-10T04:21:01.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:01 smithi085 conmon[28613]: audit 2021-02-10T04:21:01.217413 2021-02-10T04:21:01.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:01 smithi085 conmon[28613]: +0000 mon.a (mon.0) 10 : audit [DBG] from='client.? 172.21.15.85:0/227340929' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:21:01.766 INFO:teuthology.orchestra.run.smithi085.stderr:Restarting the monitor... 2021-02-10T04:21:01.766 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl restart ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mon.a 2021-02-10T04:21:02.080 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:01 smithi085 systemd[1]: Stopping Ceph mon.a for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T04:21:02.081 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:01 smithi085 conmon[28613]: debug 2021-02-10T04:21:01.834+0000 7faa8a85d700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-02-10T04:21:02.081 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:01 smithi085 conmon[28613]: debug 2021-02-10T04:21:01.834+0000 7faa8a85d700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2021-02-10T04:21:02.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 podman[29107]: 72df038b1f52230981853929aa1050c60581ddb8c4bb1b318bfa7d99413e3cb3 2021-02-10T04:21:02.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 systemd[1]: ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mon.a.service: Succeeded. 2021-02-10T04:21:02.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 systemd[1]: Stopped Ceph mon.a for 42224fce-6b57-11eb-8feb-001a4aab830c. 2021-02-10T04:21:02.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 systemd[1]: Starting Ceph mon.a for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T04:21:02.851 INFO:teuthology.orchestra.run.smithi085.stderr:Setting mon public_network to 172.21.0.0/20 2021-02-10T04:21:02.852 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -e NODE_NAME=smithi085 -v /var/log/ceph/42224fce-6b57-11eb-8feb-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpqlcrjfio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpsf4d5t6m:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 config set mon public_network 172.21.0.0/20 2021-02-10T04:21:03.062 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.839+0000 7f9e26fb3700 0 set uid:gid to 167:167 (ceph:ceph) 2021-02-10T04:21:03.062 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.839+0000 7f9e26fb3700 0 ceph version 17.0.0-681-gc1ea6241 (c1ea624123d412aff8b9d1430e36cb45fcab76b8) quincy (dev), process ceph-mon, pid 7 2021-02-10T04:21:03.062 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.839+0000 7f9e26fb3700 0 pidfile_write: ignore empty --pid-file 2021-02-10T04:21:03.062 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 0 load: jerasure load: lrc load: isa 2021-02-10T04:21:03.062 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: RocksDB version: 6.8.1 2021-02-10T04:21:03.062 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:03.062 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-02-10T04:21:03.062 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: Compile date Feb 9 2021 2021-02-10T04:21:03.063 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: DB SUMMARY 2021-02-10T04:21:03.063 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:03.063 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: CURRENT file: CURRENT 2021-02-10T04:21:03.063 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:03.063 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: IDENTITY file: IDENTITY 2021-02-10T04:21:03.063 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:03.063 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2021-02-10T04:21:03.063 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:03.063 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000004.sst 2021-02-10T04:21:03.063 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:03.063 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:03.063 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000006.log size: 73802 ; 2021-02-10T04:21:03.064 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:03.064 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: Options.error_if_exists: 0 2021-02-10T04:21:03.064 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: Options.create_if_missing: 0 2021-02-10T04:21:03.064 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: Options.paranoid_checks: 1 2021-02-10T04:21:03.064 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: Options.env: 0x55a5ed0941c0 2021-02-10T04:21:03.064 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: Options.fs: Posix File System 2021-02-10T04:21:03.064 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: Options.info_log: 0x55a5ef65bd60 2021-02-10T04:21:03.064 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: Options.max_file_opening_threads: 16 2021-02-10T04:21:03.064 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: Options.statistics: (nil) 2021-02-10T04:21:03.064 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:03.064 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: Options.use_fsync: 0 2021-02-10T04:21:03.064 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: Options.max_log_file_size: 0 2021-02-10T04:21:03.065 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-02-10T04:21:03.065 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-02-10T04:21:03.065 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: Options.keep_log_file_num: 1000 2021-02-10T04:21:03.065 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: Options.recycle_log_file_num: 0 2021-02-10T04:21:03.065 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: Options.allow_fallocate: 1 2021-02-10T04:21:03.065 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: Options.allow_mmap_reads: 0 2021-02-10T04:21:03.065 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: Options.allow_mmap_writes: 0 2021-02-10T04:21:03.065 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: Options.use_direct_reads: 0 2021-02-10T04:21:03.065 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-02-10T04:21:03.065 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: Options.create_missing_column_families: 0 2021-02-10T04:21:03.065 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: Options.db_log_dir: 2021-02-10T04:21:03.065 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2021-02-10T04:21:03.066 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-02-10T04:21:03.066 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: Options.max_subcompactions: 1 2021-02-10T04:21:03.066 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: Options.max_background_flushes: -1 2021-02-10T04:21:03.066 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.843+0000 7f9e26fb3700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-02-10T04:21:03.066 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-02-10T04:21:03.066 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-02-10T04:21:03.066 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-02-10T04:21:03.066 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-02-10T04:21:03.066 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.advise_random_on_open: 1 2021-02-10T04:21:03.066 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.db_write_buffer_size: 0 2021-02-10T04:21:03.066 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.write_buffer_manager: 0x55a5ef6b7c80 2021-02-10T04:21:03.066 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-02-10T04:21:03.066 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-02-10T04:21:03.067 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-02-10T04:21:03.067 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-02-10T04:21:03.067 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.rate_limiter: (nil) 2021-02-10T04:21:03.068 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-02-10T04:21:03.069 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.wal_recovery_mode: 2 2021-02-10T04:21:03.069 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.enable_thread_tracking: 0 2021-02-10T04:21:03.069 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.enable_pipelined_write: 0 2021-02-10T04:21:03.069 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:03.069 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.unordered_write: 0 2021-02-10T04:21:03.069 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-02-10T04:21:03.069 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-02-10T04:21:03.069 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-02-10T04:21:03.069 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-02-10T04:21:03.069 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.row_cache: None 2021-02-10T04:21:03.069 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.wal_filter: None 2021-02-10T04:21:03.069 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-02-10T04:21:03.069 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.allow_ingest_behind: 0 2021-02-10T04:21:03.070 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.preserve_deletes: 0 2021-02-10T04:21:03.070 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 bash[29206]: bf319d2ea0fda38c47212fd3d8128abd89c4488daf361d568515f1316c49db32 2021-02-10T04:21:03.070 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.two_write_queues: 0 2021-02-10T04:21:03.070 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.manual_wal_flush: 0 2021-02-10T04:21:03.070 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.atomic_flush: 0 2021-02-10T04:21:03.070 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-02-10T04:21:03.070 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-02-10T04:21:03.070 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-02-10T04:21:03.070 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.log_readahead_size: 0 2021-02-10T04:21:03.070 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-02-10T04:21:03.070 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.max_background_jobs: 2 2021-02-10T04:21:03.070 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.max_background_compactions: -1 2021-02-10T04:21:03.070 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-02-10T04:21:03.071 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-02-10T04:21:03.071 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-02-10T04:21:03.071 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.max_total_wal_size: 0 2021-02-10T04:21:03.071 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-02-10T04:21:03.071 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-02-10T04:21:03.071 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-02-10T04:21:03.071 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-02-10T04:21:03.071 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.max_open_files: -1 2021-02-10T04:21:03.071 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.bytes_per_sync: 0 2021-02-10T04:21:03.071 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-02-10T04:21:03.071 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-02-10T04:21:03.071 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.compaction_readahead_size: 0 2021-02-10T04:21:03.072 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:03.072 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Compression algorithms supported: 2021-02-10T04:21:03.072 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-02-10T04:21:03.072 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: kZSTD supported: 0 2021-02-10T04:21:03.072 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: kXpressCompression supported: 0 2021-02-10T04:21:03.072 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: kLZ4HCCompression supported: 1 2021-02-10T04:21:03.072 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: kLZ4Compression supported: 1 2021-02-10T04:21:03.072 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: kBZip2Compression supported: 0 2021-02-10T04:21:03.072 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: kZlibCompression supported: 1 2021-02-10T04:21:03.072 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: kSnappyCompression supported: 1 2021-02-10T04:21:03.073 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-02-10T04:21:03.073 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 2021-02-10T04:21:03.073 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:03.073 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:03.073 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-02-10T04:21:03.073 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:03.073 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-02-10T04:21:03.073 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.merge_operator: 2021-02-10T04:21:03.073 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.compaction_filter: None 2021-02-10T04:21:03.073 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.compaction_filter_factory: None 2021-02-10T04:21:03.073 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-02-10T04:21:03.074 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-02-10T04:21:03.074 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a5ef6021e0) 2021-02-10T04:21:03.074 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: cache_index_and_filter_blocks: 1 2021-02-10T04:21:03.074 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: cache_index_and_filter_blocks_with_high_priority: 0 2021-02-10T04:21:03.074 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-02-10T04:21:03.074 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: pin_top_level_index_and_filter: 1 2021-02-10T04:21:03.074 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: index_type: 0 2021-02-10T04:21:03.074 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: data_block_index_type: 0 2021-02-10T04:21:03.075 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: index_shortening: 1 2021-02-10T04:21:03.075 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: data_block_hash_table_util_ratio: 0.750000 2021-02-10T04:21:03.075 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: hash_index_allow_collision: 1 2021-02-10T04:21:03.075 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: checksum: 1 2021-02-10T04:21:03.075 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: no_block_cache: 0 2021-02-10T04:21:03.075 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: block_cache: 0x55a5ef63c510 2021-02-10T04:21:03.075 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: block_cache_name: BinnedLRUCache 2021-02-10T04:21:03.075 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: block_cache_options: 2021-02-10T04:21:03.075 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: capacity : 536870912 2021-02-10T04:21:03.076 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: num_shard_bits : 4 2021-02-10T04:21:03.076 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: strict_capacity_limit : 0 2021-02-10T04:21:03.076 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: high_pri_pool_ratio: 0.000 2021-02-10T04:21:03.076 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: block_cache_compressed: (nil) 2021-02-10T04:21:03.076 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: persistent_cache: (nil) 2021-02-10T04:21:03.076 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: block_size: 4096 2021-02-10T04:21:03.076 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: block_size_deviation: 10 2021-02-10T04:21:03.076 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: block_restart_interval: 16 2021-02-10T04:21:03.076 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: index_block_restart_interval: 1 2021-02-10T04:21:03.076 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: metadata_block_size: 4096 2021-02-10T04:21:03.076 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: partition_filters: 0 2021-02-10T04:21:03.076 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: use_delta_encoding: 1 2021-02-10T04:21:03.077 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: filter_policy: rocksdb.BuiltinBloomFilter 2021-02-10T04:21:03.077 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: whole_key_filtering: 1 2021-02-10T04:21:03.077 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: verify_compression: 0 2021-02-10T04:21:03.077 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: read_amp_bytes_per_bit: 0 2021-02-10T04:21:03.077 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: format_version: 2 2021-02-10T04:21:03.077 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: enable_index_compression: 1 2021-02-10T04:21:03.077 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: block_align: 0 2021-02-10T04:21:03.077 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:03.078 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.write_buffer_size: 33554432 2021-02-10T04:21:03.078 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.max_write_buffer_number: 2 2021-02-10T04:21:03.078 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.compression: NoCompression 2021-02-10T04:21:03.078 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.bottommost_compression: Disabled 2021-02-10T04:21:03.078 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.prefix_extractor: nullptr 2021-02-10T04:21:03.078 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-02-10T04:21:03.078 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:03.078 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.num_levels: 7 2021-02-10T04:21:03.078 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-02-10T04:21:03.078 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-02-10T04:21:03.078 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-02-10T04:21:03.079 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-02-10T04:21:03.079 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-02-10T04:21:03.079 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-02-10T04:21:03.079 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-02-10T04:21:03.080 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-02-10T04:21:03.080 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-02-10T04:21:03.080 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-02-10T04:21:03.080 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.compression_opts.level: 32767 2021-02-10T04:21:03.080 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.compression_opts.strategy: 0 2021-02-10T04:21:03.080 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-02-10T04:21:03.080 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-02-10T04:21:03.080 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.compression_opts.enabled: false 2021-02-10T04:21:03.080 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-02-10T04:21:03.081 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-02-10T04:21:03.081 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-02-10T04:21:03.081 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.target_file_size_base: 67108864 2021-02-10T04:21:03.081 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-02-10T04:21:03.081 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-02-10T04:21:03.081 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-02-10T04:21:03.081 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-02-10T04:21:03.081 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-02-10T04:21:03.081 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-02-10T04:21:03.081 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-02-10T04:21:03.081 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-02-10T04:21:03.081 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-02-10T04:21:03.081 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-02-10T04:21:03.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-02-10T04:21:03.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-02-10T04:21:03.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-02-10T04:21:03.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.arena_block_size: 4194304 2021-02-10T04:21:03.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-02-10T04:21:03.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-02-10T04:21:03.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-02-10T04:21:03.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.disable_auto_compactions: 0 2021-02-10T04:21:03.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-02-10T04:21:03.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-02-10T04:21:03.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-02-10T04:21:03.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-02-10T04:21:03.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-02-10T04:21:03.083 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-02-10T04:21:03.083 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-02-10T04:21:03.083 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-02-10T04:21:03.083 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-02-10T04:21:03.083 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-02-10T04:21:03.083 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.table_properties_collectors: 2021-02-10T04:21:03.083 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.inplace_update_support: 0 2021-02-10T04:21:03.083 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-02-10T04:21:03.083 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-02-10T04:21:03.083 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-02-10T04:21:03.083 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-02-10T04:21:03.083 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.bloom_locality: 0 2021-02-10T04:21:03.083 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.max_successive_merges: 0 2021-02-10T04:21:03.084 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-02-10T04:21:03.084 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.paranoid_file_checks: 0 2021-02-10T04:21:03.084 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.force_consistency_checks: 0 2021-02-10T04:21:03.084 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.report_bg_io_stats: 0 2021-02-10T04:21:03.084 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.ttl: 2592000 2021-02-10T04:21:03.084 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-02-10T04:21:03.084 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-02-10T04:21:03.084 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:03.084 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.844+0000 7f9e26fb3700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2021-02-10T04:21:03.084 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:03.084 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.845+0000 7f9e26fb3700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1612930862845607, "job": 1, "event": "recovery_started", "log_files": [6]} 2021-02-10T04:21:03.084 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.845+0000 7f9e26fb3700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2021-02-10T04:21:03.084 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:03.085 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.845+0000 7f9e26fb3700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-02-10T04:21:03.085 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.845+0000 7f9e26fb3700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1612930862846423, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 70008, "table_properties": {"data_size": 68305, "index_size": 308, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 581, "raw_key_size": 10243, "raw_average_key_size": 48, "raw_value_size": 62647, "raw_average_value_size": 294, "num_data_blocks": 10, "num_entries": 213, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1612930862, "oldest_key_time": 3, "file_creation_time": 0}} 2021-02-10T04:21:03.085 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.845+0000 7f9e26fb3700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2021-02-10T04:21:03.085 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:03.085 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.846+0000 7f9e26fb3700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1612930862846831, "job": 1, "event": "recovery_finished"} 2021-02-10T04:21:03.085 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.846+0000 7f9e26fb3700 4 rocksdb: DB pointer 0x55a5ef703800 2021-02-10T04:21:03.085 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.847+0000 7f9e0ddd1700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-02-10T04:21:03.085 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.847+0000 7f9e0ddd1700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-02-10T04:21:03.085 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: ** DB Stats ** 2021-02-10T04:21:03.085 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: Uptime(secs): 0.0 total, 0.0 interval 2021-02-10T04:21:03.085 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-02-10T04:21:03.086 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-02-10T04:21:03.086 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-02-10T04:21:03.086 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-02-10T04:21:03.086 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-02-10T04:21:03.086 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-02-10T04:21:03.086 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:03.086 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: ** Compaction Stats [default] ** 2021-02-10T04:21:03.086 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-02-10T04:21:03.086 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-02-10T04:21:03.086 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: L0 2/0 70.02 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.0 0.00 0.00 1 0.001 0 0 2021-02-10T04:21:03.086 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: Sum 2/0 70.02 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.0 0.00 0.00 1 0.001 0 0 2021-02-10T04:21:03.086 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.0 0.00 0.00 1 0.001 0 0 2021-02-10T04:21:03.087 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:03.087 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: ** Compaction Stats [default] ** 2021-02-10T04:21:03.087 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-02-10T04:21:03.087 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-02-10T04:21:03.087 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 125.0 0.00 0.00 1 0.001 0 0 2021-02-10T04:21:03.087 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: Uptime(secs): 0.0 total, 0.0 interval 2021-02-10T04:21:03.087 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: Flush(GB): cumulative 0.000, interval 0.000 2021-02-10T04:21:03.087 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: AddFile(GB): cumulative 0.000, interval 0.000 2021-02-10T04:21:03.087 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: AddFile(Total Files): cumulative 0, interval 0 2021-02-10T04:21:03.087 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: AddFile(L0 Files): cumulative 0, interval 0 2021-02-10T04:21:03.087 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: AddFile(Keys): cumulative 0, interval 0 2021-02-10T04:21:03.087 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: Cumulative compaction: 0.00 GB write, 23.85 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-02-10T04:21:03.088 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: Interval compaction: 0.00 GB write, 23.85 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-02-10T04:21:03.088 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-02-10T04:21:03.088 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:03.088 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: ** File Read Latency Histogram By Level [default] ** 2021-02-10T04:21:03.088 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:03.088 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: ** Compaction Stats [default] ** 2021-02-10T04:21:03.088 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-02-10T04:21:03.088 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-02-10T04:21:03.088 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: L0 2/0 70.02 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.0 0.00 0.00 1 0.001 0 0 2021-02-10T04:21:03.088 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: Sum 2/0 70.02 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.0 0.00 0.00 1 0.001 0 0 2021-02-10T04:21:03.088 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-02-10T04:21:03.088 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:03.088 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: ** Compaction Stats [default] ** 2021-02-10T04:21:03.089 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-02-10T04:21:03.089 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-02-10T04:21:03.089 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 125.0 0.00 0.00 1 0.001 0 0 2021-02-10T04:21:03.089 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: Uptime(secs): 0.0 total, 0.0 interval 2021-02-10T04:21:03.089 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: Flush(GB): cumulative 0.000, interval 0.000 2021-02-10T04:21:03.089 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: AddFile(GB): cumulative 0.000, interval 0.000 2021-02-10T04:21:03.089 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: AddFile(Total Files): cumulative 0, interval 0 2021-02-10T04:21:03.089 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: AddFile(L0 Files): cumulative 0, interval 0 2021-02-10T04:21:03.089 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: AddFile(Keys): cumulative 0, interval 0 2021-02-10T04:21:03.089 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: Cumulative compaction: 0.00 GB write, 23.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-02-10T04:21:03.089 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-02-10T04:21:03.089 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-02-10T04:21:03.089 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:03.090 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: ** File Read Latency Histogram By Level [default] ** 2021-02-10T04:21:03.090 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:03.090 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.847+0000 7f9e26fb3700 0 starting mon.a rank 0 at public addrs [v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0] at bind addrs [v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 42224fce-6b57-11eb-8feb-001a4aab830c 2021-02-10T04:21:03.090 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.848+0000 7f9e26fb3700 1 mon.a@-1(???) e1 preinit fsid 42224fce-6b57-11eb-8feb-001a4aab830c 2021-02-10T04:21:03.090 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.848+0000 7f9e26fb3700 0 mon.a@-1(???).mds e1 new map 2021-02-10T04:21:03.090 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.848+0000 7f9e26fb3700 0 mon.a@-1(???).mds e1 print_map 2021-02-10T04:21:03.090 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: e1 2021-02-10T04:21:03.090 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: enable_multiple, ever_enabled_multiple: 1,1 2021-02-10T04:21:03.090 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-02-10T04:21:03.090 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: legacy client fscid: -1 2021-02-10T04:21:03.090 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:03.090 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: No filesystems configured 2021-02-10T04:21:03.090 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:03.091 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.848+0000 7f9e26fb3700 0 mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2021-02-10T04:21:03.091 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.848+0000 7f9e26fb3700 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2021-02-10T04:21:03.091 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:03.091 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.848+0000 7f9e26fb3700 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2021-02-10T04:21:03.091 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.848+0000 7f9e26fb3700 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2021-02-10T04:21:03.091 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: audit 2021-02-10T04:21:03.091 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21: 2021-02-10T04:21:03.091 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 01.217413+0000 mon.a (mon.0 2021-02-10T04:21:03.091 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: ) 10 : audit [DBG] from='client.? 172.21.15.85:0/227340929' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:21:03.091 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.849+0000 7f9e26fb3700 1 mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2021-02-10T04:21:03.091 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.849+0000 7f9e26fb3700 20 expand_channel_meta expand map: {default=false} 2021-02-10T04:21:03.091 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.849+0000 7f9e26fb3700 20 expand_channel_meta from 'false' to 'false' 2021-02-10T04:21:03.091 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.849+0000 7f9e26fb3700 20 expand_channel_meta expanded map: {default=false} 2021-02-10T04:21:03.092 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.849+0000 7f9e26fb3700 20 expand_channel_meta expand map: {default=info} 2021-02-10T04:21:03.092 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.849+0000 7f9e26fb3700 20 expand_channel_meta from 'info' to 'info' 2021-02-10T04:21:03.092 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:03.092 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:02.849+0000 7f9e26fb3700 20 expand_channel_meta expanded map: {default=info} 2021-02-10T04:21:03.092 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.849+0000 7f9e26fb3700 20 expand_channel_meta expand map: {default=daemon} 2021-02-10T04:21:03.092 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.849+0000 7f9e26fb3700 20 expand_channel_meta from 'daemon' to 'daemon' 2021-02-10T04:21:03.092 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.849+0000 7f9e26fb3700 20 expand_channel_meta expanded map: {default=daemon} 2021-02-10T04:21:03.092 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.849+0000 7f9e26fb3700 20 expand_channel_meta expand map: {default=debug} 2021-02-10T04:21:03.092 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.849+0000 7f9e26fb3700 20 expand_channel_meta from 'debug' to 'debug' 2021-02-10T04:21:03.092 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: debug 2021-02-10T04:21:02.849+0000 7f9e26fb3700 20 expand_channel_meta expanded map: {default=debug} 2021-02-10T04:21:03.092 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 systemd[1]: Started Ceph mon.a for 42224fce-6b57-11eb-8feb-001a4aab830c. 2021-02-10T04:21:03.092 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: cluster 2021-02-10T04:21:02.852972+0000 mon.a ( 2021-02-10T04:21:03.092 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2021-02-10T04:21:03.093 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: cluster 2021-02-10T04:21:02.853021+0000 mon.a (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {a=[v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0]} 2021-02-10T04:21:03.093 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: cluster 2021-02-10T04:21:02.853052+0000 mon.a (mon.0) 3 : cluster [DBG] fsmap 2021-02-10T04:21:03.093 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: cluster 2021-02-10T04:21:02 2021-02-10T04:21:03.093 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: .853076+0000 mon.a (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2021-02-10T04:21:03.093 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: cluster 2021-02-10T04:21:03.093 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: 2021-02-10T04:21:02.853419+0000 mon.a (mon 2021-02-10T04:21:03.093 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:02 smithi085 conmon[29319]: .0) 5 : cluster [DBG] mgrmap e1: no daemons active 2021-02-10T04:21:04.108 INFO:teuthology.orchestra.run.smithi085.stderr:Wrote config to /etc/ceph/ceph.conf 2021-02-10T04:21:04.108 INFO:teuthology.orchestra.run.smithi085.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2021-02-10T04:21:04.108 INFO:teuthology.orchestra.run.smithi085.stderr:Creating mgr... 2021-02-10T04:21:04.109 INFO:teuthology.orchestra.run.smithi085.stderr:Verifying port 9283 ... 2021-02-10T04:21:04.110 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl daemon-reload 2021-02-10T04:21:04.199 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl stop ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mgr.y 2021-02-10T04:21:04.220 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl reset-failed ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mgr.y 2021-02-10T04:21:04.225 INFO:teuthology.orchestra.run.smithi085.stderr:systemctl: Failed to reset failed state of unit ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mgr.y.service: Unit ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mgr.y.service not loaded. 2021-02-10T04:21:04.226 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl enable ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mgr.y 2021-02-10T04:21:04.231 INFO:teuthology.orchestra.run.smithi085.stderr:systemctl: Created symlink /etc/systemd/system/ceph-42224fce-6b57-11eb-8feb-001a4aab830c.target.wants/ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mgr.y.service → /etc/systemd/system/ceph-42224fce-6b57-11eb-8feb-001a4aab830c@.service. 2021-02-10T04:21:04.313 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl start ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mgr.y 2021-02-10T04:21:04.440 INFO:journalctl@ceph.mgr.y.smithi085.stdout:Feb 10 04:21:04 smithi085 systemd[1]: Starting Ceph mgr.y for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T04:21:04.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:04 smithi085 conmon[29319]: audit 2021-02-10T04:21:03.541449+0000 mon.a ( 2021-02-10T04:21:04.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:04 smithi085 conmon[29319]: mon.0) 6 : audit [INF] from='client.? 172.21.15.85:0/181924930' entity='client.admin' 2021-02-10T04:21:04.989 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl is-enabled firewalld.service 2021-02-10T04:21:04.997 INFO:teuthology.orchestra.run.smithi085.stderr:systemctl: disabled 2021-02-10T04:21:04.997 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl is-active firewalld.service 2021-02-10T04:21:05.004 INFO:teuthology.orchestra.run.smithi085.stderr:systemctl: inactive 2021-02-10T04:21:05.005 INFO:teuthology.orchestra.run.smithi085.stderr:firewalld.service is not enabled 2021-02-10T04:21:05.005 INFO:teuthology.orchestra.run.smithi085.stderr:Not possible to enable service . firewalld.service is not available 2021-02-10T04:21:05.005 INFO:teuthology.orchestra.run.smithi085.stderr:Not possible to open ports <[]>. firewalld.service is not available 2021-02-10T04:21:05.005 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl is-enabled firewalld.service 2021-02-10T04:21:05.013 INFO:teuthology.orchestra.run.smithi085.stderr:systemctl: disabled 2021-02-10T04:21:05.013 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl is-active firewalld.service 2021-02-10T04:21:05.020 INFO:teuthology.orchestra.run.smithi085.stderr:systemctl: inactive 2021-02-10T04:21:05.020 INFO:teuthology.orchestra.run.smithi085.stderr:firewalld.service is not enabled 2021-02-10T04:21:05.021 INFO:teuthology.orchestra.run.smithi085.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2021-02-10T04:21:05.021 INFO:teuthology.orchestra.run.smithi085.stderr:Waiting for mgr to start... 2021-02-10T04:21:05.021 INFO:teuthology.orchestra.run.smithi085.stderr:Waiting for mgr... 2021-02-10T04:21:05.021 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -e NODE_NAME=smithi085 -v /var/log/ceph/42224fce-6b57-11eb-8feb-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpqlcrjfio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpsf4d5t6m:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 status -f json-pretty 2021-02-10T04:21:05.241 INFO:journalctl@ceph.mgr.y.smithi085.stdout:Feb 10 04:21:04 smithi085 bash[29579]: 66b86e945424642cc09467d5260466afbd318d6452043f2703fa1a0216437db3 2021-02-10T04:21:05.241 INFO:journalctl@ceph.mgr.y.smithi085.stdout:Feb 10 04:21:04 smithi085 systemd[1]: Started Ceph mgr.y for 42224fce-6b57-11eb-8feb-001a4aab830c. 2021-02-10T04:21:05.753 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: 2021-02-10T04:21:05.753 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: { 2021-02-10T04:21:05.753 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "fsid": "42224fce-6b57-11eb-8feb-001a4aab830c", 2021-02-10T04:21:05.753 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "health": { 2021-02-10T04:21:05.754 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-02-10T04:21:05.754 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "checks": {}, 2021-02-10T04:21:05.754 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "mutes": [] 2021-02-10T04:21:05.754 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: }, 2021-02-10T04:21:05.754 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-02-10T04:21:05.754 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "quorum": [ 2021-02-10T04:21:05.754 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: 0 2021-02-10T04:21:05.754 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: ], 2021-02-10T04:21:05.754 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "quorum_names": [ 2021-02-10T04:21:05.754 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "a" 2021-02-10T04:21:05.754 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: ], 2021-02-10T04:21:05.754 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "quorum_age": 2, 2021-02-10T04:21:05.754 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "monmap": { 2021-02-10T04:21:05.755 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "epoch": 1, 2021-02-10T04:21:05.755 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2021-02-10T04:21:05.755 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "num_mons": 1 2021-02-10T04:21:05.755 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: }, 2021-02-10T04:21:05.755 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "osdmap": { 2021-02-10T04:21:05.755 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "epoch": 1, 2021-02-10T04:21:05.755 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "num_osds": 0, 2021-02-10T04:21:05.755 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-02-10T04:21:05.755 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-02-10T04:21:05.755 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-02-10T04:21:05.758 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-02-10T04:21:05.758 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-02-10T04:21:05.758 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: }, 2021-02-10T04:21:05.758 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "pgmap": { 2021-02-10T04:21:05.759 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-02-10T04:21:05.759 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-02-10T04:21:05.759 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "num_pools": 0, 2021-02-10T04:21:05.759 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "num_objects": 0, 2021-02-10T04:21:05.759 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-02-10T04:21:05.759 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-02-10T04:21:05.759 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-02-10T04:21:05.759 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "bytes_total": 0 2021-02-10T04:21:05.759 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: }, 2021-02-10T04:21:05.759 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "fsmap": { 2021-02-10T04:21:05.759 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "epoch": 1, 2021-02-10T04:21:05.759 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "by_rank": [], 2021-02-10T04:21:05.759 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "up:standby": 0 2021-02-10T04:21:05.760 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: }, 2021-02-10T04:21:05.760 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "mgrmap": { 2021-02-10T04:21:05.760 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "available": false, 2021-02-10T04:21:05.760 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-02-10T04:21:05.760 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "modules": [ 2021-02-10T04:21:05.760 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "iostat", 2021-02-10T04:21:05.760 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "restful" 2021-02-10T04:21:05.760 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: ], 2021-02-10T04:21:05.760 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "services": {} 2021-02-10T04:21:05.760 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: }, 2021-02-10T04:21:05.760 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "servicemap": { 2021-02-10T04:21:05.760 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "epoch": 1, 2021-02-10T04:21:05.760 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "modified": "2021-02-10T04:20:57.934086+0000", 2021-02-10T04:21:05.761 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "services": {} 2021-02-10T04:21:05.761 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: }, 2021-02-10T04:21:05.761 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "progress_events": {} 2021-02-10T04:21:05.761 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: } 2021-02-10T04:21:06.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:05 smithi085 conmon[29319]: audit 2021-02-10T04:21:05.751599+0000 mon.a (mon.0) 7 : audit 2021-02-10T04:21:06.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:05 smithi085 conmon[29319]: [DBG] from='client.? 172.21.15.85:0/1929756552' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-02-10T04:21:06.383 INFO:teuthology.orchestra.run.smithi085.stderr:mgr not available, waiting (1/15)... 2021-02-10T04:21:08.386 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -e NODE_NAME=smithi085 -v /var/log/ceph/42224fce-6b57-11eb-8feb-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpqlcrjfio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpsf4d5t6m:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 status -f json-pretty 2021-02-10T04:21:09.090 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: 2021-02-10T04:21:09.090 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: { 2021-02-10T04:21:09.090 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "fsid": "42224fce-6b57-11eb-8feb-001a4aab830c", 2021-02-10T04:21:09.090 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "health": { 2021-02-10T04:21:09.090 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-02-10T04:21:09.090 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "checks": {}, 2021-02-10T04:21:09.090 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "mutes": [] 2021-02-10T04:21:09.090 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: }, 2021-02-10T04:21:09.091 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-02-10T04:21:09.091 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "quorum": [ 2021-02-10T04:21:09.091 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: 0 2021-02-10T04:21:09.091 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: ], 2021-02-10T04:21:09.091 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "quorum_names": [ 2021-02-10T04:21:09.091 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "a" 2021-02-10T04:21:09.091 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: ], 2021-02-10T04:21:09.091 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "quorum_age": 6, 2021-02-10T04:21:09.091 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "monmap": { 2021-02-10T04:21:09.091 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "epoch": 1, 2021-02-10T04:21:09.091 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2021-02-10T04:21:09.091 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "num_mons": 1 2021-02-10T04:21:09.091 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: }, 2021-02-10T04:21:09.091 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "osdmap": { 2021-02-10T04:21:09.092 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "epoch": 1, 2021-02-10T04:21:09.092 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "num_osds": 0, 2021-02-10T04:21:09.092 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-02-10T04:21:09.092 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-02-10T04:21:09.092 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-02-10T04:21:09.092 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-02-10T04:21:09.092 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-02-10T04:21:09.092 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: }, 2021-02-10T04:21:09.092 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "pgmap": { 2021-02-10T04:21:09.092 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-02-10T04:21:09.093 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-02-10T04:21:09.093 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "num_pools": 0, 2021-02-10T04:21:09.093 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "num_objects": 0, 2021-02-10T04:21:09.093 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-02-10T04:21:09.093 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-02-10T04:21:09.093 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-02-10T04:21:09.093 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "bytes_total": 0 2021-02-10T04:21:09.093 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: }, 2021-02-10T04:21:09.093 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "fsmap": { 2021-02-10T04:21:09.093 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "epoch": 1, 2021-02-10T04:21:09.093 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "by_rank": [], 2021-02-10T04:21:09.094 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "up:standby": 0 2021-02-10T04:21:09.094 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: }, 2021-02-10T04:21:09.094 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "mgrmap": { 2021-02-10T04:21:09.094 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "available": false, 2021-02-10T04:21:09.094 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-02-10T04:21:09.094 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "modules": [ 2021-02-10T04:21:09.094 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "iostat", 2021-02-10T04:21:09.094 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "restful" 2021-02-10T04:21:09.094 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: ], 2021-02-10T04:21:09.094 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "services": {} 2021-02-10T04:21:09.095 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: }, 2021-02-10T04:21:09.095 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "servicemap": { 2021-02-10T04:21:09.095 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "epoch": 1, 2021-02-10T04:21:09.095 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "modified": "2021-02-10T04:20:57.934086+0000", 2021-02-10T04:21:09.095 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "services": {} 2021-02-10T04:21:09.095 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: }, 2021-02-10T04:21:09.095 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "progress_events": {} 2021-02-10T04:21:09.095 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: } 2021-02-10T04:21:09.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:09 smithi085 conmon[29319]: audit 2021-02-10T04:21:09.085337+0000 mon.a (mon.0) 8 : audit [DBG] from='client.? 172.21.15.85:0/1701794443' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-02-10T04:21:09.639 INFO:teuthology.orchestra.run.smithi085.stderr:mgr not available, waiting (2/15)... 2021-02-10T04:21:10.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:10 smithi085 conmon[29319]: cluster 2021-02-10T04:21:09.789494+0000 mon.a (mon.0) 9 : cluster [INF] Activating manager daemon y 2021-02-10T04:21:10.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:10 smithi085 conmon[29319]: cluster 2021-02-10T04:21:09.792156+0000 mon.a (mon.0) 10 : cluster [DBG] mgrmap e2: y(active, starting, since 0.00275766s) 2021-02-10T04:21:10.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:10 smithi085 conmon[29319]: audit 2021-02-10T04:21:09.792687+0000 mon.a (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-02-10T04:21:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:10 smithi085 conmon[29319]: audit 2021-02-10T04:21:09.792805+0000 mon.a (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-02-10T04:21:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:10 smithi085 conmon[29319]: audit 2021-02-10T04:21:09.792918+0000 mon.a (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-02-10T04:21:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:10 smithi085 conmon[29319]: audit 2021-02-10T04:21:09.793529+0000 mon.a (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-02-10T04:21:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:10 smithi085 conmon[29319]: audit 2021-02-10T04:21:09.793813+0000 mon.a (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config-key ls"}]: dispatch 2021-02-10T04:21:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:10 smithi085 conmon[29319]: cluster 2021-02-10T04:21:09.795882+0000 mon.a (mon.0) 16 : cluster [INF] Manager daemon y is now available 2021-02-10T04:21:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:10 smithi085 conmon[29319]: audit 2021-02-10T04:21:09.805455+0000 mon.a (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:21:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:10 smithi085 conmon[29319]: audit 2021-02-10T04:21:09.807115+0000 mon.a (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:21:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:10 smithi085 conmon[29319]: audit 2021-02-10T04:21:09.808149+0000 mon.a (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:21:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:10 smithi085 conmon[29319]: audit 2021-02-10T04:21:09.815223+0000 mon.a (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:21:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:10 smithi085 conmon[29319]: audit 2021-02-10T04:21:09.819827+0000 mon.a (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:21:11.641 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -e NODE_NAME=smithi085 -v /var/log/ceph/42224fce-6b57-11eb-8feb-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpqlcrjfio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpsf4d5t6m:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 status -f json-pretty 2021-02-10T04:21:12.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:11 smithi085 conmon[29319]: cluster 2021-02-10T04:21:10.794722+0000 mon.a (mon.0) 22 : cluster [DBG] mgrmap e3: y(active, since 1.00532s) 2021-02-10T04:21:12.470 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: 2021-02-10T04:21:12.471 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: { 2021-02-10T04:21:12.471 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "fsid": "42224fce-6b57-11eb-8feb-001a4aab830c", 2021-02-10T04:21:12.471 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "health": { 2021-02-10T04:21:12.471 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-02-10T04:21:12.471 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "checks": {}, 2021-02-10T04:21:12.471 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "mutes": [] 2021-02-10T04:21:12.471 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: }, 2021-02-10T04:21:12.471 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-02-10T04:21:12.472 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "quorum": [ 2021-02-10T04:21:12.472 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: 0 2021-02-10T04:21:12.472 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: ], 2021-02-10T04:21:12.472 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "quorum_names": [ 2021-02-10T04:21:12.472 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "a" 2021-02-10T04:21:12.472 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: ], 2021-02-10T04:21:12.472 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "quorum_age": 9, 2021-02-10T04:21:12.472 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "monmap": { 2021-02-10T04:21:12.472 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "epoch": 1, 2021-02-10T04:21:12.472 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2021-02-10T04:21:12.472 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "num_mons": 1 2021-02-10T04:21:12.472 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: }, 2021-02-10T04:21:12.472 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "osdmap": { 2021-02-10T04:21:12.473 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "epoch": 1, 2021-02-10T04:21:12.473 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "num_osds": 0, 2021-02-10T04:21:12.473 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-02-10T04:21:12.473 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-02-10T04:21:12.476 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-02-10T04:21:12.476 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-02-10T04:21:12.476 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-02-10T04:21:12.476 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: }, 2021-02-10T04:21:12.476 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "pgmap": { 2021-02-10T04:21:12.477 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-02-10T04:21:12.477 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-02-10T04:21:12.477 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "num_pools": 0, 2021-02-10T04:21:12.477 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "num_objects": 0, 2021-02-10T04:21:12.477 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-02-10T04:21:12.477 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-02-10T04:21:12.477 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-02-10T04:21:12.477 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "bytes_total": 0 2021-02-10T04:21:12.477 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: }, 2021-02-10T04:21:12.477 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "fsmap": { 2021-02-10T04:21:12.477 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "epoch": 1, 2021-02-10T04:21:12.477 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "by_rank": [], 2021-02-10T04:21:12.478 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "up:standby": 0 2021-02-10T04:21:12.478 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: }, 2021-02-10T04:21:12.478 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "mgrmap": { 2021-02-10T04:21:12.478 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "available": true, 2021-02-10T04:21:12.478 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-02-10T04:21:12.478 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "modules": [ 2021-02-10T04:21:12.478 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "iostat", 2021-02-10T04:21:12.478 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "restful" 2021-02-10T04:21:12.478 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: ], 2021-02-10T04:21:12.478 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "services": {} 2021-02-10T04:21:12.478 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: }, 2021-02-10T04:21:12.478 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "servicemap": { 2021-02-10T04:21:12.478 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "epoch": 1, 2021-02-10T04:21:12.479 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "modified": "2021-02-10T04:20:57.934086+0000", 2021-02-10T04:21:12.479 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "services": {} 2021-02-10T04:21:12.479 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: }, 2021-02-10T04:21:12.479 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "progress_events": {} 2021-02-10T04:21:12.479 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: } 2021-02-10T04:21:13.031 INFO:teuthology.orchestra.run.smithi085.stderr:mgr is available 2021-02-10T04:21:13.031 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -e NODE_NAME=smithi085 -v /var/log/ceph/42224fce-6b57-11eb-8feb-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpqlcrjfio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpsf4d5t6m:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 config-key set mgr/cephadm/ssh_user root 2021-02-10T04:21:13.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:12 smithi085 conmon[29319]: audit 2021-02-10T04:21:12.469204+0000 mon.a (mon.0) 23 : audit 2021-02-10T04:21:13.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:12 smithi085 conmon[29319]: [DBG] from='client.? 172.21.15.85:0/4035077391' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-02-10T04:21:13.844 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: set mgr/cephadm/ssh_user 2021-02-10T04:21:14.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:13 smithi085 conmon[29319]: cluster 2021-02-10T04:21:12.794710+0000 mon.a (mon.0) 24 : cluster [DBG] 2021-02-10T04:21:14.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:13 smithi085 conmon[29319]: mgrmap e4: y(active, since 3s) 2021-02-10T04:21:14.388 INFO:teuthology.orchestra.run.smithi085.stderr:Enabling cephadm module... 2021-02-10T04:21:14.388 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -e NODE_NAME=smithi085 -v /var/log/ceph/42224fce-6b57-11eb-8feb-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpqlcrjfio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpsf4d5t6m:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 mgr module enable cephadm 2021-02-10T04:21:14.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:14 smithi085 conmon[29319]: audit 2021-02-10T04:21:13.842439+0000 mon.a (mon.0) 25 2021-02-10T04:21:14.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:14 smithi085 conmon[29319]: : audit [INF] from='client.? 172.21.15.85:0/3563974839' entity='client.admin' 2021-02-10T04:21:16.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:15 smithi085 conmon[29319]: audit 2021-02-10T04:21:15.219125+0000 mon.a (mon.0) 26 : audit 2021-02-10T04:21:16.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:15 smithi085 conmon[29319]: [INF] from='client.? 172.21.15.85:0/751632972' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2021-02-10T04:21:16.191 INFO:journalctl@ceph.mgr.y.smithi085.stdout:Feb 10 04:21:15 smithi085 conmon[29692]: ignoring --setuser ceph since I am not root 2021-02-10T04:21:16.191 INFO:journalctl@ceph.mgr.y.smithi085.stdout:Feb 10 04:21:15 smithi085 conmon[29692]: ignoring --setgroup ceph since I am not root 2021-02-10T04:21:16.363 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -e NODE_NAME=smithi085 -v /var/log/ceph/42224fce-6b57-11eb-8feb-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpqlcrjfio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpsf4d5t6m:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 mgr stat 2021-02-10T04:21:16.940 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:16 smithi085 conmon[29319]: audit 2021-02-10T04:21:15.847372+0000 mon.a ( 2021-02-10T04:21:16.940 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:16 smithi085 conmon[29319]: mon.0) 27 : audit [INF] from='client.? 172.21.15.85:0/751632972' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2021-02-10T04:21:16.940 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:16 smithi085 conmon[29319]: cluster 2021-02-10T04:21:15.847426+0000 mon.a (mon.0) 28 : cluster [DBG] mgrmap e5: y(active, since 6s) 2021-02-10T04:21:17.208 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: { 2021-02-10T04:21:17.208 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "epoch": 5, 2021-02-10T04:21:17.208 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "available": true, 2021-02-10T04:21:17.209 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "active_name": "y", 2021-02-10T04:21:17.209 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "num_standby": 0 2021-02-10T04:21:17.209 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: } 2021-02-10T04:21:17.775 INFO:teuthology.orchestra.run.smithi085.stderr:Waiting for the mgr to restart... 2021-02-10T04:21:17.775 INFO:teuthology.orchestra.run.smithi085.stderr:Waiting for mgr epoch 5... 2021-02-10T04:21:17.775 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -e NODE_NAME=smithi085 -v /var/log/ceph/42224fce-6b57-11eb-8feb-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpqlcrjfio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpsf4d5t6m:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 tell mgr mgr_status 2021-02-10T04:21:18.024 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:17 smithi085 conmon[29319]: audit 2021-02-10T04:21:17.208597+0000 mon.a ( 2021-02-10T04:21:18.025 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:17 smithi085 conmon[29319]: mon.0) 29 : audit [DBG] from='client.? 172.21.15.85:0/2404434838' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2021-02-10T04:21:21.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:20 smithi085 conmon[29319]: cluster 2021-02-10T04:21:20.700460+0000 mon.a (mon. 2021-02-10T04:21:21.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:20 smithi085 conmon[29319]: 0) 30 : cluster [INF] Active manager daemon y restarted 2021-02-10T04:21:21.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:20 smithi085 conmon[29319]: cluster 2021-02-10T04:21:20.700847+0000 mon.a (mon.0) 31 : cluster [INF] Activating manager daemon y 2021-02-10T04:21:21.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:20 smithi085 conmon[29319]: cluster 2021-02-10T04:21:20. 2021-02-10T04:21:21.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:20 smithi085 conmon[29319]: 702517+0000 mon.a (mon.0) 32 2021-02-10T04:21:21.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:20 smithi085 conmon[29319]: : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2021-02-10T04:21:21.771 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: { 2021-02-10T04:21:21.771 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2021-02-10T04:21:21.771 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "initialized": true 2021-02-10T04:21:21.772 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: } 2021-02-10T04:21:22.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: cluster 2021-02-10T04:21:20.753971+0000 mon.a (mon.0) 33 : cluster [DBG] 2021-02-10T04:21:22.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: mgrmap e6: y(active, starting, since 0.0531968s) 2021-02-10T04:21:22.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: audit 2021-02-10T04:21:20.756762+0000 mon.a (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-02-10T04:21:22.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: audit 2021-02-10T04:21:22.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: 2021-02-10T04:21:20.757312+0000 mon.a (mon.0) 35 : 2021-02-10T04:21:22.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: audit [DBG] from='mgr.14116 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-02-10T04:21:22.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:21:22.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: 02-10T04:21:20.757478+0000 2021-02-10T04:21:22.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: mon.a (mon.0) 36 : audit 2021-02-10T04:21:22.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: [DBG] from='mgr.14116 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-02-10T04:21:22.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:21:22.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: 02-10T04:21:20.757614 2021-02-10T04:21:22.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: +0000 mon.a (mon.0) 37 : 2021-02-10T04:21:22.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: audit [DBG] from='mgr.14116 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-02-10T04:21:22.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:21:22.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: 02-10T04:21:20.757863 2021-02-10T04:21:22.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: +0000 mon.a (mon.0) 38 : 2021-02-10T04:21:22.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: audit [DBG] from='mgr.14116 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config-key ls"}]: dispatch 2021-02-10T04:21:22.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: audit 2021 2021-02-10T04:21:22.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: -02-10T04:21:20.758536 2021-02-10T04:21:22.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: +0000 mon.a (mon.0) 39 : 2021-02-10T04:21:22.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: audit [DBG] from='mgr.14116 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_user"}]: dispatch 2021-02-10T04:21:22.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:21:22.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: 02-10T04:21:20.758813+0000 2021-02-10T04:21:22.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: mon.a (mon.0) 40 : audit 2021-02-10T04:21:22.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: [DBG] from='mgr.14116 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/devicehealth/last_scrape"}]: dispatch 2021-02-10T04:21:22.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: audit 2021 2021-02-10T04:21:22.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: -02-10T04:21:20. 2021-02-10T04:21:22.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: 759020+0000 mon.a (mon.0) 2021-02-10T04:21:22.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: 41 : audit [DBG] from='mgr.14116 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/report_id"}]: dispatch 2021-02-10T04:21:22.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:21:22.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: 02-10T04:21:20.759217 2021-02-10T04:21:22.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: +0000 mon.a (mon.0) 42 2021-02-10T04:21:22.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: : audit [DBG] from='mgr.14116 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/salt"}]: dispatch 2021-02-10T04:21:22.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: cluster 2021-02-10T04:21:22.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: 2021-02-10T04:21: 2021-02-10T04:21:22.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: 20.760920+0000 mon.a (mon.0 2021-02-10T04:21:22.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: ) 43 : cluster [INF] Manager daemon y is now available 2021-02-10T04:21:22.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:21:22.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: 02-10T04:21:20.772388+0000 2021-02-10T04:21:22.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: mon.a (mon.0) 44 : audit 2021-02-10T04:21:22.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: [INF] from='mgr.14116 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:21:22.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: audit 2021 2021-02-10T04:21:22.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: -02-10T04:21:20. 2021-02-10T04:21:22.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: 775051+0000 mon.a (mon.0) 2021-02-10T04:21:22.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: 45 : audit [INF] from='mgr.14116 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:21:22.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:21:22.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: 02-10T04:21:20.777454 2021-02-10T04:21:22.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: +0000 mon.a (mon.0) 46 : 2021-02-10T04:21:22.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: audit [INF] from='mgr.14116 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:21:22.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: audit 2021 2021-02-10T04:21:22.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: -02-10T04:21:20. 2021-02-10T04:21:22.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: 863593+0000 mon.a (mon.0 2021-02-10T04:21:22.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: ) 47 : audit [INF] from='mgr.14116 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:21:22.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: audit 2021 2021-02-10T04:21:22.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: -02-10T04:21:20. 2021-02-10T04:21:22.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: 869138+0000 mon.a (mon.0) 2021-02-10T04:21:22.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: 48 : audit [INF] from='mgr.14116 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:21:22.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: audit 2021 2021-02-10T04:21:22.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: -02-10T04:21:20. 2021-02-10T04:21:22.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: 873567+0000 mon.a (mon.0) 2021-02-10T04:21:22.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: 49 : audit [INF] from='mgr.14116 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:21:22.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: audit 2021 2021-02-10T04:21:22.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: -02-10T04:21:20 2021-02-10T04:21:22.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: .874592+0000 mon.a (mon.0) 2021-02-10T04:21:22.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:21 smithi085 conmon[29319]: 50 : audit [INF] from='mgr.14116 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:21:22.322 INFO:teuthology.orchestra.run.smithi085.stderr:mgr epoch 5 is available 2021-02-10T04:21:22.322 INFO:teuthology.orchestra.run.smithi085.stderr:Setting orchestrator backend to cephadm... 2021-02-10T04:21:22.322 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -e NODE_NAME=smithi085 -v /var/log/ceph/42224fce-6b57-11eb-8feb-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpqlcrjfio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpsf4d5t6m:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 orch set backend cephadm 2021-02-10T04:21:22.940 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:22 smithi085 conmon[29319]: audit 2021-02-10T04:21:21.764400+0000 2021-02-10T04:21:22.940 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:22 smithi085 conmon[29319]: mgr.y (mgr.14116) 1 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2021-02-10T04:21:22.940 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:22 smithi085 conmon[29319]: cluster 2021-02-10T04:21:21.765211+0000 mon.a (mon.0) 51 : cluster [DBG] mgrmap e7: y(active, since 1.06443s) 2021-02-10T04:21:22.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:22 smithi085 conmon[29319]: audit 2021-02-10T04:21:21.771243+0000 mgr.y (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2021-02-10T04:21:23.620 INFO:teuthology.orchestra.run.smithi085.stderr:Generating ssh key... 2021-02-10T04:21:23.621 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -e NODE_NAME=smithi085 -v /var/log/ceph/42224fce-6b57-11eb-8feb-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpqlcrjfio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpsf4d5t6m:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 cephadm generate-key 2021-02-10T04:21:23.891 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:23 smithi085 conmon[29319]: cluster 2021-02-10T04:21:22.769827+0000 mon.a (mon.0) 52 : cluster [DBG] mgrmap e8: y(active, since 2s) 2021-02-10T04:21:23.891 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:23 smithi085 conmon[29319]: audit 2021-02-10T04:21:23.117574+0000 mon.a (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:21:23.891 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:23 smithi085 conmon[29319]: audit 2021-02-10T04:21:23.122991+0000 mon.a (mon.0) 54 : audit [INF] from='mgr.14116 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:21:25.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:24 smithi085 conmon[29319]: audit 2021-02-10T04:21:23.111050+0000 mgr.y (mgr.14116) 3 : audit 2021-02-10T04:21:25.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:24 smithi085 conmon[29319]: [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:21:25.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:24 smithi085 conmon[29319]: audit 2021-02-10T04:21:24.702229+0000 mon.a (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:21:25.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:24 smithi085 conmon[29319]: audit 2021-02-10T04:21:24.703867+0000 mon.a (mon.0) 56 : audit [INF] from='mgr.14116 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:21:25.191 INFO:journalctl@ceph.mgr.y.smithi085.stdout:Feb 10 04:21:24 smithi085 conmon[29692]: Generating public/private rsa key pair. 2021-02-10T04:21:25.191 INFO:journalctl@ceph.mgr.y.smithi085.stdout:Feb 10 04:21:24 smithi085 conmon[29692]: Your identification has been saved in /tmp/tmp8v2g6m1n/key. 2021-02-10T04:21:25.191 INFO:journalctl@ceph.mgr.y.smithi085.stdout:Feb 10 04:21:24 smithi085 conmon[29692]: Your public key has been saved in /tmp/tmp8v2g6m1n/key.pub. 2021-02-10T04:21:25.191 INFO:journalctl@ceph.mgr.y.smithi085.stdout:Feb 10 04:21:24 smithi085 conmon[29692]: The key fingerprint is: 2021-02-10T04:21:25.191 INFO:journalctl@ceph.mgr.y.smithi085.stdout:Feb 10 04:21:24 smithi085 conmon[29692]: SHA256:s3ATmhnIVf21jwU9/Acc+YCwhYTheXufzYuZz/Bki6A ceph-42224fce-6b57-11eb-8feb-001a4aab830c 2021-02-10T04:21:25.191 INFO:journalctl@ceph.mgr.y.smithi085.stdout:Feb 10 04:21:24 smithi085 conmon[29692]: The key's randomart image is: 2021-02-10T04:21:25.192 INFO:journalctl@ceph.mgr.y.smithi085.stdout:Feb 10 04:21:24 smithi085 conmon[29692]: +---[RSA 3072]----+ 2021-02-10T04:21:25.192 INFO:journalctl@ceph.mgr.y.smithi085.stdout:Feb 10 04:21:24 smithi085 conmon[29692]: | ..o=oo.oo+ | 2021-02-10T04:21:25.192 INFO:journalctl@ceph.mgr.y.smithi085.stdout:Feb 10 04:21:24 smithi085 conmon[29692]: | . o ...oo. Bo.| 2021-02-10T04:21:25.192 INFO:journalctl@ceph.mgr.y.smithi085.stdout:Feb 10 04:21:24 smithi085 conmon[29692]: | o . + o. . *o| 2021-02-10T04:21:25.192 INFO:journalctl@ceph.mgr.y.smithi085.stdout:Feb 10 04:21:24 smithi085 conmon[29692]: | = o .. . =| 2021-02-10T04:21:25.192 INFO:journalctl@ceph.mgr.y.smithi085.stdout:Feb 10 04:21:24 smithi085 conmon[29692]: | = S . . +.| 2021-02-10T04:21:25.192 INFO:journalctl@ceph.mgr.y.smithi085.stdout:Feb 10 04:21:24 smithi085 conmon[29692]: | o + . ..+.| 2021-02-10T04:21:25.192 INFO:journalctl@ceph.mgr.y.smithi085.stdout:Feb 10 04:21:24 smithi085 conmon[29692]: | . . .ooo| 2021-02-10T04:21:25.192 INFO:journalctl@ceph.mgr.y.smithi085.stdout:Feb 10 04:21:24 smithi085 conmon[29692]: | . . @ o| 2021-02-10T04:21:25.192 INFO:journalctl@ceph.mgr.y.smithi085.stdout:Feb 10 04:21:24 smithi085 conmon[29692]: | E =.* | 2021-02-10T04:21:25.192 INFO:journalctl@ceph.mgr.y.smithi085.stdout:Feb 10 04:21:24 smithi085 conmon[29692]: +----[SHA256]-----+ 2021-02-10T04:21:25.286 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -e NODE_NAME=smithi085 -v /var/log/ceph/42224fce-6b57-11eb-8feb-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpqlcrjfio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpsf4d5t6m:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 cephadm get-pub-key 2021-02-10T04:21:26.026 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCb6TSreuF5f6ASPML9dm6XDTeQCU9UmQhnRaSfX7PWnA+G2cpWil0rgyOFWlvOPX4ImiPcZfIU9M6ktPvTESMcz04iaQ+7MryrH763L11nLsShmNVkglfbXXkEdDud0SpYqX5XUMGIxPwfoPoShc1OfQ5eN9mHkxy60skLXWVZwawlHIDJ6j9mTf/d20POmFcMtSZWtsnrv+AnHstqCQi55Ho+bCVgsN+3OanqJkYIcU1hRlT8odnDERj1qb5cnkqii+/BaHXoRZmeNnH5C5FR1WR4BnESWPOtPW1SLBoeSQCGf3ZNM/4w/Fs3qYXAvrIRSak1HjP3Q28ZOEzkYS6YDAG+8nKC3uVl8f6ydGXKVim/MMg4Ryrg5W0r/mLQ9f3vx+6QfDMbCNbZPvLKg+dfVFsJFg/e0Giw27pst3rfLuW84t6cQucaWCCRfecOkKVEmFaW8X56kD80jsl5nbvwTrWh4K2suVOSfCEnhxuiW8vsW9r+j9rEhrNW2OlCGpk= ceph-42224fce-6b57-11eb-8feb-001a4aab830c 2021-02-10T04:21:26.046 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:25 smithi085 conmon[29319]: audit 2021-02-10T04:21:24.391746+0000 mgr.y (mgr.14116) 4 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:21:26.145 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:25 smithi085 conmon[29319]: cephadm 2021-02-10T04:21:24.391959+0000 mgr.y (mgr.14116) 5 : cephadm [INF] Generating ssh key... 2021-02-10T04:21:26.520 INFO:teuthology.orchestra.run.smithi085.stderr:Wrote public SSH key to to /home/ubuntu/cephtest/ceph.pub 2021-02-10T04:21:26.520 INFO:teuthology.orchestra.run.smithi085.stderr:Adding key to root@localhost's authorized_keys... 2021-02-10T04:21:26.522 INFO:teuthology.orchestra.run.smithi085.stderr:Adding host smithi085... 2021-02-10T04:21:26.522 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -e NODE_NAME=smithi085 -v /var/log/ceph/42224fce-6b57-11eb-8feb-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpqlcrjfio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpsf4d5t6m:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 orch host add smithi085 2021-02-10T04:21:27.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:26 smithi085 conmon[29319]: audit 2021-02-10T04:21:26.026635+0000 mgr.y (mgr.14116 2021-02-10T04:21:27.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:26 smithi085 conmon[29319]: ) 6 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:21:27.690 INFO:journalctl@ceph.mgr.y.smithi085.stdout:Feb 10 04:21:27 smithi085 conmon[29692]: [89B blob data] 2021-02-10T04:21:27.980 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: Added host 'smithi085' 2021-02-10T04:21:28.578 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -e NODE_NAME=smithi085 -v /var/log/ceph/42224fce-6b57-11eb-8feb-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpqlcrjfio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpsf4d5t6m:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 config set mgr mgr/cephadm/container_init True --force 2021-02-10T04:21:29.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:28 smithi085 conmon[29319]: audit 2021-02-10T04:21:27.320846+0000 mgr.y (mgr.14116) 7 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi085", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:21:29.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:28 smithi085 conmon[29319]: audit 2021-02-10T04:21:27.978255+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:21:29.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:28 smithi085 conmon[29319]: audit 2021-02-10T04:21:28.328938+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:21:29.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:28 smithi085 conmon[29319]: audit 2021-02-10T04:21:28.329582+0000 mon.a (mon.0) 59 : audit [DBG] from='mgr.14116 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:21:29.915 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -e NODE_NAME=smithi085 -v /var/log/ceph/42224fce-6b57-11eb-8feb-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpqlcrjfio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpsf4d5t6m:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 config set mgr mgr/dashboard/ssl_server_port 8443 2021-02-10T04:21:30.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:29 smithi085 conmon[29319]: cephadm 2021-02-10T04:21:27.978612+0000 mgr.y (mgr.14116) 8 : cephadm [INF] Added host smithi085 2021-02-10T04:21:30.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:29 smithi085 conmon[29319]: audit 2021-02-10T04:21:29.304828+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.14116 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:21:30.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:29 smithi085 conmon[29319]: audit 2021-02-10T04:21:29.305518+0000 mon.a (mon.0) 61 : audit [DBG] from='mgr.14116 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:21:30.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:29 smithi085 conmon[29319]: audit 2021-02-10T04:21:29.372972+0000 mon.a (mon.0) 62 : audit [INF] from='client.? 172.21.15.85:0/383721833' entity='client.admin' 2021-02-10T04:21:31.481 INFO:teuthology.orchestra.run.smithi085.stderr:Enabling the dashboard module... 2021-02-10T04:21:31.482 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -e NODE_NAME=smithi085 -v /var/log/ceph/42224fce-6b57-11eb-8feb-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpqlcrjfio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpsf4d5t6m:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 mgr module enable dashboard 2021-02-10T04:21:32.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:31 smithi085 conmon[29319]: audit 2021-02-10T04:21:30.993788+0000 mon.a (mon.0) 63 : audit 2021-02-10T04:21:32.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:31 smithi085 conmon[29319]: [INF] from='client.? 172.21.15.85:0/3326043332' entity='client.admin' 2021-02-10T04:21:33.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:32 smithi085 conmon[29319]: audit 2021-02-10T04:21:32.245544+0000 mon.a (mon.0) 64 : audit [INF] from='client.? 172.21.15.85:0/1482775426' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2021-02-10T04:21:33.440 INFO:journalctl@ceph.mgr.y.smithi085.stdout:Feb 10 04:21:33 smithi085 conmon[29692]: ignoring --setuser ceph since I am not root 2021-02-10T04:21:33.441 INFO:journalctl@ceph.mgr.y.smithi085.stdout:Feb 10 04:21:33 smithi085 conmon[29692]: ignoring --setgroup ceph since I am not root 2021-02-10T04:21:33.540 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -e NODE_NAME=smithi085 -v /var/log/ceph/42224fce-6b57-11eb-8feb-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpqlcrjfio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpsf4d5t6m:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 mgr stat 2021-02-10T04:21:34.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:33 smithi085 conmon[29319]: audit 2021-02-10T04:21:32.998975+0000 mon.a (mon.0) 2021-02-10T04:21:34.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:33 smithi085 conmon[29319]: 65 : audit [INF] from='client.? 172.21.15.85:0/1482775426' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2021-02-10T04:21:34.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:33 smithi085 conmon[29319]: cluster 2021-02-10T04:21:32.999068+0000 mon.a (mon.0) 66 : cluster [DBG] mgrmap e9: y(active, since 12s) 2021-02-10T04:21:34.392 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: { 2021-02-10T04:21:34.392 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "epoch": 9, 2021-02-10T04:21:34.392 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "available": true, 2021-02-10T04:21:34.392 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "active_name": "y", 2021-02-10T04:21:34.392 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "num_standby": 0 2021-02-10T04:21:34.392 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: } 2021-02-10T04:21:34.920 INFO:teuthology.orchestra.run.smithi085.stderr:Waiting for the mgr to restart... 2021-02-10T04:21:34.920 INFO:teuthology.orchestra.run.smithi085.stderr:Waiting for mgr epoch 9... 2021-02-10T04:21:34.920 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -e NODE_NAME=smithi085 -v /var/log/ceph/42224fce-6b57-11eb-8feb-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpqlcrjfio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpsf4d5t6m:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 tell mgr mgr_status 2021-02-10T04:21:35.185 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:34 smithi085 conmon[29319]: audit 2021-02-10T04:21:34.391665+0000 mon.a 2021-02-10T04:21:35.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:34 smithi085 conmon[29319]: (mon.0) 67 : audit [DBG] from='client.? 172.21.15.85:0/2523640392' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2021-02-10T04:21:38.190 INFO:journalctl@ceph.mgr.y.smithi085.stdout:Feb 10 04:21:38 smithi085 conmon[29692]: [89B blob data] 2021-02-10T04:21:38.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:37 smithi085 conmon[29319]: cluster 2021-02-10T04:21:37.851587 2021-02-10T04:21:38.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:37 smithi085 conmon[29319]: +0000 mon.a (mon.0) 68 : cluster [INF] Active manager daemon y restarted 2021-02-10T04:21:38.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:37 smithi085 conmon[29319]: cluster 2021-02-10T04:21:37.851956+0000 mon.a (mon.0) 69 : cluster [INF] Activating manager daemon y 2021-02-10T04:21:38.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:37 smithi085 conmon[29319]: cluster 2021-02-10T04:21:37.853556+0000 mon.a (mon.0) 70 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2021-02-10T04:21:38.915 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: { 2021-02-10T04:21:39.070 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2021-02-10T04:21:39.070 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: "initialized": true 2021-02-10T04:21:39.070 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: } 2021-02-10T04:21:39.116 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:38 smithi085 conmon[29319]: cluster 2021-02-10T04:21:37.904827+0000 mon.a (mon.0) 71 : cluster [DBG] mgrmap e10: y(active, starting, since 0.0529535s) 2021-02-10T04:21:39.117 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:38 smithi085 conmon[29319]: audit 2021-02-10T04:21:37.907445+0000 mon.a (mon.0) 72 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-02-10T04:21:39.117 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:38 smithi085 conmon[29319]: audit 2021-02-10T04:21:37.907699+0000 mon.a (mon.0) 73 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-02-10T04:21:39.117 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:38 smithi085 conmon[29319]: audit 2021-02-10T04:21:37.907846+0000 mon.a (mon.0) 74 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-02-10T04:21:39.117 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:38 smithi085 conmon[29319]: audit 2021-02-10T04:21:37.907994+0000 mon.a (mon.0) 75 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-02-10T04:21:39.117 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:38 smithi085 conmon[29319]: audit 2021-02-10T04:21:37.908254+0000 mon.a (mon.0) 76 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config-key ls"}]: dispatch 2021-02-10T04:21:39.117 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:38 smithi085 conmon[29319]: audit 2021-02-10T04:21:37.908986+0000 mon.a (mon.0) 77 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/host.smithi085"}]: dispatch 2021-02-10T04:21:39.117 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:38 smithi085 conmon[29319]: audit 2021-02-10T04:21:37.909200+0000 mon.a (mon.0) 78 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/inventory"}]: dispatch 2021-02-10T04:21:39.117 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:38 smithi085 conmon[29319]: audit 2021-02-10T04:21:37.909403+0000 mon.a (mon.0) 79 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/osd_remove_queue"}]: dispatch 2021-02-10T04:21:39.117 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:38 smithi085 conmon[29319]: audit 2021-02-10T04:21:37.909605+0000 mon.a (mon.0) 80 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_key"}]: dispatch 2021-02-10T04:21:39.117 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:38 smithi085 conmon[29319]: audit 2021-02-10T04:21:37.909814+0000 mon.a (mon.0) 81 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_identity_pub"}]: dispatch 2021-02-10T04:21:39.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:38 smithi085 conmon[29319]: audit 2021-02-10T04:21:37.910006+0000 mon.a (mon.0) 82 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/cephadm/ssh_user"}]: dispatch 2021-02-10T04:21:39.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:38 smithi085 conmon[29319]: audit 2021-02-10T04:21:37.910201+0000 mon.a (mon.0) 83 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/devicehealth/last_scrape"}]: dispatch 2021-02-10T04:21:39.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:38 smithi085 conmon[29319]: audit 2021-02-10T04:21:37.910387+0000 mon.a (mon.0) 84 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/report_id"}]: dispatch 2021-02-10T04:21:39.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:38 smithi085 conmon[29319]: audit 2021-02-10T04:21:37.910584+0000 mon.a (mon.0) 85 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/telemetry/salt"}]: dispatch 2021-02-10T04:21:39.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:38 smithi085 conmon[29319]: cluster 2021-02-10T04:21:37.912219+0000 mon.a (mon.0) 86 : cluster [INF] Manager daemon y is now available 2021-02-10T04:21:39.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:38 smithi085 conmon[29319]: audit 2021-02-10T04:21:38.027813+0000 mon.a (mon.0) 87 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:21:39.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:38 smithi085 conmon[29319]: audit 2021-02-10T04:21:38.029341+0000 mon.a (mon.0) 88 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:21:39.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:38 smithi085 conmon[29319]: audit 2021-02-10T04:21:38.230367+0000 mon.a (mon.0) 89 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:21:39.526 INFO:teuthology.orchestra.run.smithi085.stderr:mgr epoch 9 is available 2021-02-10T04:21:39.526 INFO:teuthology.orchestra.run.smithi085.stderr:Generating a dashboard self-signed certificate... 2021-02-10T04:21:39.526 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -e NODE_NAME=smithi085 -v /var/log/ceph/42224fce-6b57-11eb-8feb-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpqlcrjfio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpsf4d5t6m:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 dashboard create-self-signed-cert 2021-02-10T04:21:40.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:39 smithi085 conmon[29319]: audit 2021-02-10T04:21:38.908883+0000 mgr.y (mgr.14140 2021-02-10T04:21:40.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:39 smithi085 conmon[29319]: ) 1 : audit [DBG] from='client.14144 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2021-02-10T04:21:40.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:39 smithi085 conmon[29319]: cluster 2021-02-10T04:21:38.909603+0000 mon.a (mon.0) 90 : cluster [DBG] mgrmap e11: y(active, since 1.05773s) 2021-02-10T04:21:40.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:39 smithi085 conmon[29319]: audit 2021-02-10T04:21:38.915369+0000 mgr.y (mgr.14140) 2 : audit [DBG] from='client.14144 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2021-02-10T04:21:40.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:39 smithi085 conmon[29319]: audit 2021-02-10T04:21:39.338424+0000 mon.a (mon.0) 91 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:21:40.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:39 smithi085 conmon[29319]: audit 2021-02-10T04:21:39.339119+0000 mon.a (mon.0) 92 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:21:40.432 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: Self-signed certificate created 2021-02-10T04:21:41.211 INFO:teuthology.orchestra.run.smithi085.stderr:Creating initial admin user... 2021-02-10T04:21:41.212 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -e NODE_NAME=smithi085 -v /var/log/ceph/42224fce-6b57-11eb-8feb-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpqlcrjfio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpsf4d5t6m:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp8w5qul82:/tmp/dashboard.pw:z quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2021-02-10T04:21:41.468 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:41 smithi085 conmon[29319]: cluster 2021-02-10T04:21:40.339899+0000 mon.a (mon.0) 93 : cluster [DBG] mgrmap e12: y(active, since 2s) 2021-02-10T04:21:41.468 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:41 smithi085 conmon[29319]: audit 2021-02-10T04:21:40.351168+0000 mgr.y (mgr.14140) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:21:41.468 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:41 smithi085 conmon[29319]: audit 2021-02-10T04:21:40.430967+0000 mon.a (mon.0) 94 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:21:41.468 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:41 smithi085 conmon[29319]: audit 2021-02-10T04:21:40.432658+0000 mon.a (mon.0) 95 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:21:42.197 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$rMkC.qSIkQ00eC2afrGN9e4Vbk9RgOkTqezbi0A/7p.I.0Xn/CVee", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1612930902, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2021-02-10T04:21:42.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:42 smithi085 conmon[29319]: audit 2021-02-10T04:21:41.981745+0000 mgr.y (mgr.14140) 2021-02-10T04:21:42.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:42 smithi085 conmon[29319]: 4 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:21:42.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:42 smithi085 conmon[29319]: audit 2021-02-10T04:21:42.196645+0000 mon.a (mon.0) 96 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:21:42.738 INFO:teuthology.orchestra.run.smithi085.stderr:Fetching dashboard port number... 2021-02-10T04:21:42.738 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 -e NODE_NAME=smithi085 -v /var/log/ceph/42224fce-6b57-11eb-8feb-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpqlcrjfio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpsf4d5t6m:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 config get mgr mgr/dashboard/ssl_server_port 2021-02-10T04:21:43.499 INFO:teuthology.orchestra.run.smithi085.stderr:/usr/bin/ceph: 8443 2021-02-10T04:21:44.059 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl is-enabled firewalld.service 2021-02-10T04:21:44.068 INFO:teuthology.orchestra.run.smithi085.stderr:systemctl: disabled 2021-02-10T04:21:44.068 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: systemctl is-active firewalld.service 2021-02-10T04:21:44.075 INFO:teuthology.orchestra.run.smithi085.stderr:systemctl: inactive 2021-02-10T04:21:44.075 INFO:teuthology.orchestra.run.smithi085.stderr:firewalld.service is not enabled 2021-02-10T04:21:44.076 INFO:teuthology.orchestra.run.smithi085.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2021-02-10T04:21:44.077 INFO:teuthology.orchestra.run.smithi085.stderr:Ceph Dashboard is now available at: 2021-02-10T04:21:44.078 INFO:teuthology.orchestra.run.smithi085.stderr: 2021-02-10T04:21:44.078 INFO:teuthology.orchestra.run.smithi085.stderr: URL: https://smithi085:8443/ 2021-02-10T04:21:44.078 INFO:teuthology.orchestra.run.smithi085.stderr: User: admin 2021-02-10T04:21:44.078 INFO:teuthology.orchestra.run.smithi085.stderr: Password: gduelq5in7 2021-02-10T04:21:44.078 INFO:teuthology.orchestra.run.smithi085.stderr: 2021-02-10T04:21:44.079 INFO:teuthology.orchestra.run.smithi085.stderr:You can access the Ceph CLI with: 2021-02-10T04:21:44.079 INFO:teuthology.orchestra.run.smithi085.stderr: 2021-02-10T04:21:44.079 INFO:teuthology.orchestra.run.smithi085.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2021-02-10T04:21:44.079 INFO:teuthology.orchestra.run.smithi085.stderr: 2021-02-10T04:21:44.079 INFO:teuthology.orchestra.run.smithi085.stderr:Please consider enabling telemetry to help improve Ceph: 2021-02-10T04:21:44.079 INFO:teuthology.orchestra.run.smithi085.stderr: 2021-02-10T04:21:44.079 INFO:teuthology.orchestra.run.smithi085.stderr: ceph telemetry on 2021-02-10T04:21:44.079 INFO:teuthology.orchestra.run.smithi085.stderr: 2021-02-10T04:21:44.079 INFO:teuthology.orchestra.run.smithi085.stderr:For more information see: 2021-02-10T04:21:44.079 INFO:teuthology.orchestra.run.smithi085.stderr: 2021-02-10T04:21:44.079 INFO:teuthology.orchestra.run.smithi085.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2021-02-10T04:21:44.079 INFO:teuthology.orchestra.run.smithi085.stderr: 2021-02-10T04:21:44.079 INFO:teuthology.orchestra.run.smithi085.stderr:Bootstrap complete. 2021-02-10T04:21:44.080 INFO:teuthology.orchestra.run.smithi085.stderr:Releasing lock 140199189940656 on /run/cephadm/42224fce-6b57-11eb-8feb-001a4aab830c.lock 2021-02-10T04:21:44.080 INFO:teuthology.orchestra.run.smithi085.stderr:Lock 140199189940656 released on /run/cephadm/42224fce-6b57-11eb-8feb-001a4aab830c.lock 2021-02-10T04:21:44.113 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:43 smithi085 conmon[29319]: audit 2021-02-10T04:21:43.499223+0000 mon.a (mon.0) 97 : audit [DBG] from='client.? 172.21.15.85:0/929365206' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2021-02-10T04:21:44.117 INFO:tasks.cephadm:Fetching config... 2021-02-10T04:21:44.117 DEBUG:teuthology.orchestra.run.smithi085:> set -ex 2021-02-10T04:21:44.118 DEBUG:teuthology.orchestra.run.smithi085:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2021-02-10T04:21:44.132 INFO:tasks.cephadm:Fetching client.admin keyring... 2021-02-10T04:21:44.133 DEBUG:teuthology.orchestra.run.smithi085:> set -ex 2021-02-10T04:21:44.133 DEBUG:teuthology.orchestra.run.smithi085:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2021-02-10T04:21:44.187 INFO:tasks.cephadm:Fetching mon keyring... 2021-02-10T04:21:44.187 DEBUG:teuthology.orchestra.run.smithi085:> set -ex 2021-02-10T04:21:44.187 DEBUG:teuthology.orchestra.run.smithi085:> sudo dd if=/var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/keyring of=/dev/stdout 2021-02-10T04:21:44.253 INFO:tasks.cephadm:Fetching pub ssh key... 2021-02-10T04:21:44.253 DEBUG:teuthology.orchestra.run.smithi085:> set -ex 2021-02-10T04:21:44.253 DEBUG:teuthology.orchestra.run.smithi085:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2021-02-10T04:21:44.308 INFO:tasks.cephadm:Installing pub ssh key for root users... 2021-02-10T04:21:44.308 DEBUG:teuthology.orchestra.run.smithi085:> sudo install -d -m 0700 /root/.ssh &&echo ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCb6TSreuF5f6ASPML9dm6XDTeQCU9UmQhnRaSfX7PWnA+G2cpWil0rgyOFWlvOPX4ImiPcZfIU9M6ktPvTESMcz04iaQ+7MryrH763L11nLsShmNVkglfbXXkEdDud0SpYqX5XUMGIxPwfoPoShc1OfQ5eN9mHkxy60skLXWVZwawlHIDJ6j9mTf/d20POmFcMtSZWtsnrv+AnHstqCQi55Ho+bCVgsN+3OanqJkYIcU1hRlT8odnDERj1qb5cnkqii+/BaHXoRZmeNnH5C5FR1WR4BnESWPOtPW1SLBoeSQCGf3ZNM/4w/Fs3qYXAvrIRSak1HjP3Q28ZOEzkYS6YDAG+8nKC3uVl8f6ydGXKVim/MMg4Ryrg5W0r/mLQ9f3vx+6QfDMbCNbZPvLKg+dfVFsJFg/e0Giw27pst3rfLuW84t6cQucaWCCRfecOkKVEmFaW8X56kD80jsl5nbvwTrWh4K2suVOSfCEnhxuiW8vsW9r+j9rEhrNW2OlCGpk= ceph-42224fce-6b57-11eb-8feb-001a4aab830c |sudo tee -a /root/.ssh/authorized_keys &&sudo chmod 0600 /root/.ssh/authorized_keys 2021-02-10T04:21:44.384 INFO:teuthology.orchestra.run.smithi085.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCb6TSreuF5f6ASPML9dm6XDTeQCU9UmQhnRaSfX7PWnA+G2cpWil0rgyOFWlvOPX4ImiPcZfIU9M6ktPvTESMcz04iaQ+7MryrH763L11nLsShmNVkglfbXXkEdDud0SpYqX5XUMGIxPwfoPoShc1OfQ5eN9mHkxy60skLXWVZwawlHIDJ6j9mTf/d20POmFcMtSZWtsnrv+AnHstqCQi55Ho+bCVgsN+3OanqJkYIcU1hRlT8odnDERj1qb5cnkqii+/BaHXoRZmeNnH5C5FR1WR4BnESWPOtPW1SLBoeSQCGf3ZNM/4w/Fs3qYXAvrIRSak1HjP3Q28ZOEzkYS6YDAG+8nKC3uVl8f6ydGXKVim/MMg4Ryrg5W0r/mLQ9f3vx+6QfDMbCNbZPvLKg+dfVFsJFg/e0Giw27pst3rfLuW84t6cQucaWCCRfecOkKVEmFaW8X56kD80jsl5nbvwTrWh4K2suVOSfCEnhxuiW8vsW9r+j9rEhrNW2OlCGpk= ceph-42224fce-6b57-11eb-8feb-001a4aab830c 2021-02-10T04:21:44.395 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -d -m 0700 /root/.ssh &&echo ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCb6TSreuF5f6ASPML9dm6XDTeQCU9UmQhnRaSfX7PWnA+G2cpWil0rgyOFWlvOPX4ImiPcZfIU9M6ktPvTESMcz04iaQ+7MryrH763L11nLsShmNVkglfbXXkEdDud0SpYqX5XUMGIxPwfoPoShc1OfQ5eN9mHkxy60skLXWVZwawlHIDJ6j9mTf/d20POmFcMtSZWtsnrv+AnHstqCQi55Ho+bCVgsN+3OanqJkYIcU1hRlT8odnDERj1qb5cnkqii+/BaHXoRZmeNnH5C5FR1WR4BnESWPOtPW1SLBoeSQCGf3ZNM/4w/Fs3qYXAvrIRSak1HjP3Q28ZOEzkYS6YDAG+8nKC3uVl8f6ydGXKVim/MMg4Ryrg5W0r/mLQ9f3vx+6QfDMbCNbZPvLKg+dfVFsJFg/e0Giw27pst3rfLuW84t6cQucaWCCRfecOkKVEmFaW8X56kD80jsl5nbvwTrWh4K2suVOSfCEnhxuiW8vsW9r+j9rEhrNW2OlCGpk= ceph-42224fce-6b57-11eb-8feb-001a4aab830c |sudo tee -a /root/.ssh/authorized_keys &&sudo chmod 0600 /root/.ssh/authorized_keys 2021-02-10T04:21:44.438 INFO:teuthology.orchestra.run.smithi164.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCb6TSreuF5f6ASPML9dm6XDTeQCU9UmQhnRaSfX7PWnA+G2cpWil0rgyOFWlvOPX4ImiPcZfIU9M6ktPvTESMcz04iaQ+7MryrH763L11nLsShmNVkglfbXXkEdDud0SpYqX5XUMGIxPwfoPoShc1OfQ5eN9mHkxy60skLXWVZwawlHIDJ6j9mTf/d20POmFcMtSZWtsnrv+AnHstqCQi55Ho+bCVgsN+3OanqJkYIcU1hRlT8odnDERj1qb5cnkqii+/BaHXoRZmeNnH5C5FR1WR4BnESWPOtPW1SLBoeSQCGf3ZNM/4w/Fs3qYXAvrIRSak1HjP3Q28ZOEzkYS6YDAG+8nKC3uVl8f6ydGXKVim/MMg4Ryrg5W0r/mLQ9f3vx+6QfDMbCNbZPvLKg+dfVFsJFg/e0Giw27pst3rfLuW84t6cQucaWCCRfecOkKVEmFaW8X56kD80jsl5nbvwTrWh4K2suVOSfCEnhxuiW8vsW9r+j9rEhrNW2OlCGpk= ceph-42224fce-6b57-11eb-8feb-001a4aab830c 2021-02-10T04:21:44.451 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2021-02-10T04:21:45.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:44 smithi085 conmon[29319]: cluster 2021-02-10T04:21:43.863879+0000 mon.a (mon.0) 98 : cluster [DBG] mgrmap e13: y(active, since 6s) 2021-02-10T04:21:47.403 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi164 2021-02-10T04:21:47.404 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2021-02-10T04:21:47.404 DEBUG:teuthology.orchestra.run.smithi164:> dd of=/etc/ceph/ceph.conf 2021-02-10T04:21:47.421 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2021-02-10T04:21:47.421 DEBUG:teuthology.orchestra.run.smithi164:> dd of=/etc/ceph/ceph.client.admin.keyring 2021-02-10T04:21:47.476 INFO:tasks.cephadm:Adding host smithi164 to orchestrator... 2021-02-10T04:21:47.477 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph orch host add smithi164 2021-02-10T04:21:47.940 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:47 smithi085 conmon[29319]: audit 2021-02-10T04:21:46.746071+0000 mon.a (mon.0) 99 : audit [INF] from='client.? 172.21.15.85:0/1730523459' entity='client.admin' 2021-02-10T04:21:49.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:48 smithi085 conmon[29319]: audit 2021-02-10T04:21:48.186161+0000 mon.a (mon.0) 100 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:21:49.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:48 smithi085 conmon[29319]: audit 2021-02-10T04:21:48.505697+0000 mon.a (mon.0) 101 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:21:50.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:49 smithi085 conmon[29319]: cephadm 2021-02-10T04:21:48.506021+0000 mgr.y ( 2021-02-10T04:21:50.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:49 smithi085 conmon[29319]: mgr.14140) 5 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:21:50.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:49 smithi085 conmon[29319]: audit 2021-02-10T04:21:48.882285+0000 mon.a (mon.0) 102 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:21:50.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:49 smithi085 conmon[29319]: cephadm 2021-02-10T04:21:48.882628+0000 mgr.y (mgr.14140) 6 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2021-02-10T04:21:50.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:49 smithi085 conmon[29319]: audit 2021-02-10T04:21:48.882863+0000 mon.a (mon.0) 103 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-02-10T04:21:50.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:49 smithi085 conmon[29319]: audit 2021-02-10T04:21:48.883390+0000 mon.a (mon.0) 104 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:21:50.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:49 smithi085 conmon[29319]: cephadm 2021-02-10T04:21:48.883840+0000 mgr.y (mgr.14140) 7 : cephadm [INF] Reconfiguring daemon mon.a on smithi085 2021-02-10T04:21:50.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:49 smithi085 conmon[29319]: audit 2021-02-10T04:21:48.884220+0000 mon.a (mon.0) 105 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon.a", "key": "container_image"}]: dispatch 2021-02-10T04:21:51.652 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:51 smithi085 conmon[29319]: audit 2021-02-10T04:21:50.347916+0000 mon.a (mon.0) 2021-02-10T04:21:51.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:51 smithi085 conmon[29319]: 106 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:21:51.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:51 smithi085 conmon[29319]: cephadm 2021-02-10T04:21:50.348348+0000 mgr.y (mgr.14140) 8 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2021-02-10T04:21:51.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:51 smithi085 conmon[29319]: audit 2021-02-10T04:21:51.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:51 smithi085 conmon[29319]: 2021-02-10T04:21:50.348582+0000 mon.a (mon.0) 107 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mgr.y"}]: dispatch 2021-02-10T04:21:51.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:51 smithi085 conmon[29319]: 2021-02-10T04:21:51.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:51 smithi085 conmon[29319]: audit 2021-02-10T04:21:51.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:51 smithi085 conmon[29319]: 2021-02-10T04:21:50.349157+0000 mon.a ( 2021-02-10T04:21:51.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:51 smithi085 conmon[29319]: mon.0) 108 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:21:51.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:51 smithi085 conmon[29319]: cephadm 2021 2021-02-10T04:21:51.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:51 smithi085 conmon[29319]: -02-10T04:21:50.349582 2021-02-10T04:21:51.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:51 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 9 : cephadm 2021-02-10T04:21:51.654 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:51 smithi085 conmon[29319]: [INF] Reconfiguring daemon mgr.y on smithi085 2021-02-10T04:21:51.654 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:51 smithi085 conmon[29319]: audit 2021-02-10T04: 2021-02-10T04:21:51.654 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:51 smithi085 conmon[29319]: 21:50.349898+0000 mon.a (mon.0) 2021-02-10T04:21:51.654 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:51 smithi085 conmon[29319]: 109 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mgr.y", "key": "container_image"}]: dispatch 2021-02-10T04:21:53.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:52 smithi085 conmon[29319]: audit 2021-02-10T04:21:51.698040+0000 mon.a ( 2021-02-10T04:21:53.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:52 smithi085 conmon[29319]: mon.0) 110 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:21:53.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:52 smithi085 conmon[29319]: audit 2021-02-10T04:21:51.699537+0000 mon.a (mon.0) 111 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:21:53.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:52 smithi085 conmon[29319]: audit 2021-02-10T04:21:52.564787+0000 mon.a (mon.0) 112 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:21:53.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:52 smithi085 conmon[29319]: audit 2021-02-10T04:21:52.569989+0000 mon.a (mon.0) 113 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:21:59.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:58 smithi085 conmon[29319]: cluster 2021-02-10T04:21:57.908304 2021-02-10T04:21:59.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:21:58 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 10 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:01.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:00 smithi085 conmon[29319]: cluster 2021-02-10T04:21:59.908618+0000 mgr.y (mgr 2021-02-10T04:22:01.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:00 smithi085 conmon[29319]: .14140) 11 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:03.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:02 smithi085 conmon[29319]: cluster 2021-02-10T04:22:01.908767+0000 mgr.y (mgr.14140) 12 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:05.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:04 smithi085 conmon[29319]: cluster 2021-02-10T04:22:03.909101+0000 mgr.y (mgr 2021-02-10T04:22:05.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:04 smithi085 conmon[29319]: .14140) 13 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:07.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:06 smithi085 conmon[29319]: cluster 2021-02-10T04:22:05.909374+0000 mgr.y (mgr.14140) 14 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:09.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:08 smithi085 conmon[29319]: cluster 2021-02-10T04:22:07.909630+0000 mgr.y (mgr.14140) 15 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:11.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:10 smithi085 conmon[29319]: cluster 2021-02-10T04:22:09.909939+0000 mgr.y (mgr.14140) 16 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:13.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:12 smithi085 conmon[29319]: cluster 2021-02-10T04:22:11.910237+0000 mgr.y (mgr.14140) 17 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:15.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:14 smithi085 conmon[29319]: cluster 2021-02-10T04:22:13.910513+0000 mgr.y (mgr.14140) 18 2021-02-10T04:22:15.541 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:14 smithi085 conmon[29319]: : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:17.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:16 smithi085 conmon[29319]: cluster 2021-02-10T04:22:15.910763+0000 mgr.y (mgr.14140) 19 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:19.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:18 smithi085 conmon[29319]: cluster 2021-02-10T04:22:17.910995+0000 mgr.y (mgr.14140) 20 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:21.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:20 smithi085 conmon[29319]: cluster 2021-02-10T04:22:19.911304+0000 mgr.y (mgr.14140) 2021-02-10T04:22:21.757 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:20 smithi085 conmon[29319]: 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:23.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:22 smithi085 conmon[29319]: cluster 2021-02-10T04:22:21.911542+0000 mgr.y (mgr.14140) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:25.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:24 smithi085 conmon[29319]: cluster 2021-02-10T04:22:23.911768+0000 mgr.y (mgr.14140) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:27.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:26 smithi085 conmon[29319]: cluster 2021-02-10T04:22:25.912011+0000 mgr.y (mgr. 2021-02-10T04:22:27.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:26 smithi085 conmon[29319]: 14140) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:27.940 INFO:journalctl@ceph.mgr.y.smithi085.stdout:Feb 10 04:22:27 smithi085 conmon[29692]: [90B blob data] 2021-02-10T04:22:28.092 INFO:teuthology.orchestra.run.smithi164.stdout:Added host 'smithi164' 2021-02-10T04:22:28.797 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph orch host ls --format=json 2021-02-10T04:22:28.940 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:28 smithi085 conmon[29319]: audit 2021-02-10T04:22:27.445930+0000 2021-02-10T04:22:28.940 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:28 smithi085 conmon[29319]: mgr.y (mgr.14140) 25 : audit [DBG] from='client.14158 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi164", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:22:28.940 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:28 smithi085 conmon[29319]: cluster 2021-02-10T04:22:27.912357+0000 mgr.y (mgr.14140) 26 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:28.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:28 smithi085 conmon[29319]: audit 2021-02-10T04:22:28.089773+0000 mon.a (mon.0) 114 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:22:28.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:28 smithi085 conmon[29319]: audit 2021-02-10T04:22:28.092505+0000 mon.a (mon.0) 115 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:22:28.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:28 smithi085 conmon[29319]: audit 2021-02-10T04:22:28.444597+0000 mon.a (mon.0) 116 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:22:28.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:28 smithi085 conmon[29319]: audit 2021-02-10T04:22:28.445379+0000 mon.a (mon.0) 117 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:22:28.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:28 smithi085 conmon[29319]: audit 2021-02-10T04:22:28.755800+0000 mon.a (mon.0) 118 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:22:28.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:28 smithi085 conmon[29319]: audit 2021-02-10T04:22:28.756355+0000 mon.a (mon.0) 119 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:22:30.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:29 smithi085 conmon[29319]: cephadm 2021-02-10T04:22:28.090260+0000 mgr.y (mgr.14140) 27 : cephadm [INF] 2021-02-10T04:22:30.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:29 smithi085 conmon[29319]: Added host smithi164 2021-02-10T04:22:30.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:29 smithi085 conmon[29319]: audit 2021-02-10T04:22:28.920778+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:22:31.067 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:22:31.067 INFO:teuthology.orchestra.run.smithi164.stdout:[{"addr": "smithi085", "hostname": "smithi085", "labels": [], "status": ""}, {"addr": "smithi164", "hostname": "smithi164", "labels": [], "status": ""}] 2021-02-10T04:22:31.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:30 smithi085 conmon[29319]: cluster 2021-02-10T04:22:29.912705+0000 mgr.y (mgr.14140) 28 : cluster 2021-02-10T04:22:31.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:30 smithi085 conmon[29319]: [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:31.616 INFO:tasks.cephadm:Setting crush tunables to default 2021-02-10T04:22:31.616 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd crush tunables default 2021-02-10T04:22:33.112 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:32 smithi085 conmon[29319]: audit 2021-02-10T04:22:31.065886+0000 mgr.y (mgr.14140) 29 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:22:33.112 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:32 smithi085 conmon[29319]: cluster 2021-02-10T04:22:31.913037+0000 mgr.y (mgr.14140) 30 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:33.112 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:32 smithi085 conmon[29319]: audit 2021-02-10T04:22:32.352581+0000 mon.a (mon.0) 121 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:22:33.112 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:32 smithi085 conmon[29319]: audit 2021-02-10T04:22:32.672670+0000 mon.a (mon.0) 122 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:22:34.055 INFO:teuthology.orchestra.run.smithi085.stderr:adjusted tunables profile to default 2021-02-10T04:22:34.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:33 smithi085 conmon[29319]: cephadm 2021-02-10T04:22:32.672976+0000 mgr.y (mgr.14140) 31 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:22:34.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:33 smithi085 conmon[29319]: audit 2021-02-10T04:22:33.048131+0000 mon.a (mon.0) 123 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:22:34.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:33 smithi085 conmon[29319]: audit 2021-02-10T04:22:33.050070+0000 mon.a (mon.0) 124 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:22:34.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:33 smithi085 conmon[29319]: audit 2021-02-10T04:22:33.394906+0000 mon.a (mon.0) 125 : audit [INF] from='client.? 172.21.15.85:0/3348952785' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2021-02-10T04:22:34.588 INFO:tasks.cephadm:Adding mon.c on smithi085 2021-02-10T04:22:34.588 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph orch daemon add mon 'smithi085:[v2:172.21.15.85:3301,v1:172.21.15.85:6790]=c' 2021-02-10T04:22:35.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:34 smithi085 conmon[29319]: cluster 2021-02-10T04:22:33.913285+0000 mgr.y (mgr 2021-02-10T04:22:35.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:34 smithi085 conmon[29319]: .14140) 32 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:35.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:34 smithi085 conmon[29319]: audit 2021-02-10T04:22:34.053660+0000 mon.a (mon.0) 126 : audit [INF] from='client.? 172.21.15.85:0/3348952785' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2021-02-10T04:22:35.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:34 smithi085 conmon[29319]: cluster 2021-02-10T04:22:34.053728+0000 mon.a (mon.0) 127 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-02-10T04:22:37.128 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:36 smithi085 conmon[29319]: cluster 2021-02-10T04:22: 2021-02-10T04:22:37.128 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:36 smithi085 conmon[29319]: 35.913627+0000 mgr.y (mgr.14140) 33 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:37.128 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:36 smithi085 conmon[29319]: audit 2021-02-10T04:22:36.319965+0000 mon.a (mon.0) 128 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-02-10T04:22:37.129 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:36 smithi085 conmon[29319]: audit 2021-02-10T04:22:36.320671+0000 mon.a (mon.0) 129 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:22:37.129 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:36 smithi085 conmon[29319]: audit 2021-02-10T04:22:36.321384+0000 mon.a (mon.0) 130 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon.c", "key": "container_image"}]: dispatch 2021-02-10T04:22:38.180 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:37 smithi085 conmon[29319]: audit 2021-02-10T04:22:36.318229+0000 mgr.y (mgr.14140) 34 : audit [DBG] from='client.14164 -' entity='client.admin' cmd=[{"prefix": "orch daemon add", "daemon_type": "mon", "placement": "smithi085:[v2:172.21.15.85:3301,v1:172.21.15.85:6790]=c", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:22:38.180 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:37 smithi085 conmon[29319]: cephadm 2021-02-10T04:22:36.321121+0000 mgr.y (mgr.14140) 35 : cephadm [INF] Deploying daemon mon.c on smithi085 2021-02-10T04:22:38.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:38 smithi085 conmon[29319]: cluster 2021-02-10T04:22:37.913958+0000 mgr.y (mgr.14140) 36 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:39.833 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:38 smithi085 conmon[29319]: audit 2021-02-10T04:22:38 2021-02-10T04:22:39.833 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:38 smithi085 conmon[29319]: .029074+0000 mon.a (mon.0) 131 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:22:39.833 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:38 smithi085 conmon[29319]: audit 2021-02-10T04:22:38.030143+0000 mon.a (mon.0) 132 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:22:44.363 INFO:teuthology.orchestra.run.smithi085.stdout:Deployed mon.c on host 'smithi085' 2021-02-10T04:22:44.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:44 smithi085 conmon[29319]: cluster 2021-02-10T04:22:39.335110+0000 mon.a (mon.0) 134 : 2021-02-10T04:22:44.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:44 smithi085 conmon[29319]: cluster [INF] mon.a calling monitor election 2021-02-10T04:22:44.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:44 smithi085 conmon[29319]: audit 2021-02-10T04:22:39.350784+0000 mon.a (mon.0) 135 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:22:44.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:44 smithi085 conmon[29319]: cluster 2021-02-10T04:22:39.914225+0000 mgr.y (mgr.14140) 37 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:44.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:44 smithi085 conmon[29319]: audit 2021-02-10T04:22:40.331795+0000 mon.a (mon.0) 136 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-02-10T04:22:44.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:44 smithi085 conmon[29319]: audit 2021-02-10T04:22:41.331835+0000 mon.a (mon.0) 137 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-02-10T04:22:44.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:44 smithi085 conmon[29319]: cluster 2021-02-10T04:22:41.333213+0000 mon.c (mon.1) 1 : cluster [INF] mon.c calling monitor election 2021-02-10T04:22:44.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:44 smithi085 conmon[29319]: cluster 2021-02-10T04:22:41.914453+0000 mgr.y (mgr.14140) 38 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:44.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:44 smithi085 conmon[29319]: audit 2021-02-10T04:22:42.331849+0000 mon.a (mon.0) 138 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-02-10T04:22:44.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:44 smithi085 conmon[29319]: audit 2021-02-10T04:22:43.332101+0000 mon.a (mon.0) 139 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-02-10T04:22:44.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:44 smithi085 conmon[29319]: cluster 2021-02-10T04:22:43.914787+0000 mgr.y (mgr.14140) 39 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:44.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:44 smithi085 conmon[29319]: audit 2021-02-10T04:22:44.332214+0000 mon.a (mon.0) 140 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-02-10T04:22:44.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:44 smithi085 conmon[29319]: cluster 2021-02-10T04:22:44.337969+0000 mon.a (mon.0) 141 : cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2021-02-10T04:22:44.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:44 smithi085 conmon[29319]: cluster 2021-02-10T04:22:44.348371+0000 mon.a (mon.0) 142 : cluster [DBG] monmap e2: 2 mons at {a=[v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0],c=[v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0]} 2021-02-10T04:22:44.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:44 smithi085 conmon[29319]: cluster 2021-02-10T04:22:44.348441+0000 mon.a (mon.0) 143 : cluster [DBG] fsmap 2021-02-10T04:22:44.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:44 smithi085 conmon[29319]: cluster 2021-02-10T04:22:44.348472+0000 mon.a (mon.0) 144 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-02-10T04:22:44.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:44 smithi085 conmon[29319]: cluster 2021-02-10T04:22:44.348672+0000 mon.a (mon.0) 145 : cluster [DBG] mgrmap e13: y(active, since 66s) 2021-02-10T04:22:44.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:44 smithi085 conmon[29319]: cluster 2021-02-10T04:22:44.355543+0000 mon.a (mon.0) 146 : cluster [INF] overall HEALTH_OK 2021-02-10T04:22:44.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:44 smithi085 conmon[29319]: audit 2021-02-10T04:22:44.360935+0000 mon.a (mon.0) 147 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:22:44.982 DEBUG:teuthology.orchestra.run.smithi085:mon.c> sudo journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mon.c.service 2021-02-10T04:22:44.985 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-02-10T04:22:44.986 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph mon dump -f json 2021-02-10T04:22:45.013 INFO:journalctl@ceph.mon.c.smithi085.stdout:-- Logs begin at Wed 2021-02-10 04:09:23 UTC. -- 2021-02-10T04:22:45.534 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:45 smithi085 conmon[29319]: audit 2021-02-10T04:22:45.332608+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-02-10T04:22:45.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:45 smithi085 conmon[35961]: audit 2021-02-10T04:22:45.332608+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-02-10T04:22:46.667 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:46 smithi085 conmon[29319]: audit 2021-02-10T04:22:45.603720+0000 mon.a (mon. 2021-02-10T04:22:46.667 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:46 smithi085 conmon[29319]: 0) 149 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:22:46.667 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:46 smithi085 conmon[29319]: audit 2021-02-10T04:22:45.606149+0000 mon.a (mon.0) 150 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:22:46.667 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:46 smithi085 conmon[29319]: cephadm 2021-02-10T04:22:45.606444+0000 mgr.y (mgr.14140) 40 : cephadm [INF] Reconfiguring mon.a (monmap changed)... 2021-02-10T04:22:46.667 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:46 smithi085 conmon[29319]: audit 2021-02-10T04:22:45.606667+0000 mon.a (mon.0) 151 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-02-10T04:22:46.667 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:46 smithi085 conmon[29319]: audit 2021-02-10T04:22:45.607237+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:22:46.667 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:46 smithi085 conmon[29319]: cephadm 2021-02-10T04:22:45.607669+0000 mgr.y (mgr.14140) 41 : cephadm [INF] Reconfiguring daemon mon.a on smithi085 2021-02-10T04:22:46.668 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:46 smithi085 conmon[29319]: audit 2021-02-10T04:22:45.608022+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon.a", "key": "container_image"}]: dispatch 2021-02-10T04:22:46.668 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:46 smithi085 conmon[29319]: cluster 2021-02-10T04:22:45.915059+0000 mgr.y (mgr.14140) 42 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:46.677 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:46 smithi085 conmon[35961]: audit 2021-02-10T04:22:45.603720+0000 mon.a (mon.0) 149 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:22:46.677 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:46 smithi085 conmon[35961]: audit 2021-02-10T04:22:45.606149+0000 mon.a (mon.0) 150 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:22:46.677 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:46 smithi085 conmon[35961]: cephadm 2021-02-10T04:22:45.606444+0000 mgr.y (mgr.14140) 40 : cephadm [INF] Reconfiguring mon.a (monmap changed)... 2021-02-10T04:22:46.678 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:46 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:22:46.678 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:46 smithi085 conmon[35961]: T04:22:45.606667+0000 mon.a (mon.0) 151 : audit [INF] 2021-02-10T04:22:46.678 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:46 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-02-10T04:22:46.678 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:46 smithi085 conmon[35961]: audit 2021-02-10T04:22:45.607237 2021-02-10T04:22:46.678 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:46 smithi085 conmon[35961]: +0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:22:46.678 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:46 smithi085 conmon[35961]: cephadm 2021- 2021-02-10T04:22:46.678 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:46 smithi085 conmon[35961]: 02-10T04:22:45.607669+0000 2021-02-10T04:22:46.678 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:46 smithi085 conmon[35961]: mgr.y (mgr.14140) 41 : cephadm 2021-02-10T04:22:46.678 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:46 smithi085 conmon[35961]: [INF] Reconfiguring daemon mon.a on smithi085 2021-02-10T04:22:46.678 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:46 smithi085 conmon[35961]: audit 2021 2021-02-10T04:22:46.678 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:46 smithi085 conmon[35961]: -02-10T04:22:45 2021-02-10T04:22:46.678 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:46 smithi085 conmon[35961]: .608022+0000 mon.a (mon.0) 2021-02-10T04:22:46.678 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:46 smithi085 conmon[35961]: 153 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon.a", "key": "container_image"}]: dispatch 2021-02-10T04:22:46.679 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:46 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:22:46.679 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:46 smithi085 conmon[35961]: -10T04:22:45.915059 2021-02-10T04:22:46.679 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:46 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 42 : 2021-02-10T04:22:46.679 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:46 smithi085 conmon[35961]: cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:47.061 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:22:47.061 INFO:teuthology.orchestra.run.smithi085.stdout:{"epoch":2,"fsid":"42224fce-6b57-11eb-8feb-001a4aab830c","modified":"2021-02-10T04:22:39.332328Z","created":"2021-02-10T04:20:55.445955Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:3300","nonce":0},{"type":"v1","addr":"172.21.15.85:6789","nonce":0}]},"addr":"172.21.15.85:6789/0","public_addr":"172.21.15.85:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:3301","nonce":0},{"type":"v1","addr":"172.21.15.85:6790","nonce":0}]},"addr":"172.21.15.85:6790/0","public_addr":"172.21.15.85:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2021-02-10T04:22:47.063 INFO:teuthology.orchestra.run.smithi085.stderr:dumped monmap epoch 2 2021-02-10T04:22:47.940 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:47 smithi085 conmon[29319]: audit 2021-02-10T04:22:47.062077+0000 mon.a (mon.0) 154 : 2021-02-10T04:22:47.940 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:47 smithi085 conmon[29319]: audit [DBG] from='client.? 172.21.15.85:0/1855194259' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-02-10T04:22:47.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:47 smithi085 conmon[29319]: audit 2021-02-10T04:22:47.578009+0000 mon.a (mon.0) 155 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:22:47.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:47 smithi085 conmon[29319]: audit 2021-02-10T04:22:47.578671+0000 mon.a (mon.0) 156 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mgr.y"}]: dispatch 2021-02-10T04:22:47.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:47 smithi085 conmon[29319]: audit 2021-02-10T04:22:47.579195+0000 mon.a (mon.0) 157 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:22:47.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:47 smithi085 conmon[29319]: audit 2021-02-10T04:22:47.580025+0000 mon.a (mon.0) 158 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mgr.y", "key": "container_image"}]: dispatch 2021-02-10T04:22:47.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:47 smithi085 conmon[35961]: audit 2021-02-10T04:22:47. 2021-02-10T04:22:47.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:47 smithi085 conmon[35961]: 062077+0000 mon.a (mon.0) 154 : audit [DBG] from='client.? 172.21.15.85:0/1855194259' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-02-10T04:22:47.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:47 smithi085 conmon[35961]: audit 2021-02-10T04:22:47.578009+0000 mon.a (mon.0) 155 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:22:47.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:47 smithi085 conmon[35961]: audit 2021-02-10T04:22:47.578671+0000 mon.a (mon.0) 156 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mgr.y"}]: dispatch 2021-02-10T04:22:47.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:47 smithi085 conmon[35961]: audit 2021-02-10T04:22:47.579195+0000 mon.a (mon.0) 157 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:22:47.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:47 smithi085 conmon[35961]: audit 2021-02-10T04:22:47.580025+0000 mon.a (mon.0) 158 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mgr.y", "key": "container_image"}]: dispatch 2021-02-10T04:22:47.962 INFO:tasks.cephadm:Adding mon.b on smithi164 2021-02-10T04:22:47.962 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph orch daemon add mon smithi164:172.21.15.164=b 2021-02-10T04:22:49.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:48 smithi085 conmon[29319]: cephadm 2021-02-10T04:22:47.578428+0000 mgr.y (mgr.14140) 43 : cephadm [INF] Reconfiguring mgr.y (monmap changed)... 2021-02-10T04:22:49.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:48 smithi085 conmon[29319]: cephadm 2021-02-10T04:22:47.579667+0000 mgr.y (mgr.14140) 44 : cephadm [INF] Reconfiguring daemon mgr.y on smithi085 2021-02-10T04:22:49.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:48 smithi085 conmon[29319]: cluster 2021-02-10T04:22:47.915314+0000 mgr.y (mgr.14140) 45 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:49.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:48 smithi085 conmon[35961]: cephadm 2021-02-10T04:22 2021-02-10T04:22:49.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:48 smithi085 conmon[35961]: :47.578428+0000 mgr.y (mgr.14140) 43 : cephadm [INF] Reconfiguring mgr.y (monmap changed)... 2021-02-10T04:22:49.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:48 smithi085 conmon[35961]: cephadm 2021-02-10T04:22:47.579667+0000 mgr.y (mgr.14140) 44 2021-02-10T04:22:49.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:48 smithi085 conmon[35961]: : cephadm [INF] Reconfiguring daemon mgr.y on smithi085 2021-02-10T04:22:49.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:48 smithi085 conmon[35961]: cluster 2021-02-10T04:22:49.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:48 smithi085 conmon[35961]: 2021-02-10T04:22:47 2021-02-10T04:22:49.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:48 smithi085 conmon[35961]: .915314+0000 mgr.y (mgr. 2021-02-10T04:22:49.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:48 smithi085 conmon[35961]: 14140) 45 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:48 smithi085 conmon[35961]: 2021-02-10T04:22:50.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[29319]: audit 2021-02-10T04:22:49.068324+0000 mon.a (mon.0) 159 2021-02-10T04:22:50.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[29319]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:22:50.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[29319]: cephadm 2021-02-10T04:22:49.068774+0000 mgr.y (mgr.14140) 46 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2021-02-10T04:22:50.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[29319]: audit 2021-02-10T04:22:49.069009+0000 mon.a (mon.0) 160 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-02-10T04:22:50.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[29319]: audit 2021-02-10T04:22:49.069524+0000 mon.a (mon.0) 161 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:22:50.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[29319]: cephadm 2021-02-10T04:22:49.070080+0000 mgr.y (mgr.14140) 47 : cephadm [INF] Reconfiguring daemon mon.c on smithi085 2021-02-10T04:22:50.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[29319]: audit 2021-02-10T04:22:49.070356+0000 mon.a (mon.0) 162 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon.c", "key": "container_image"}]: dispatch 2021-02-10T04:22:50.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[29319]: audit 2021-02-10T04:22:49.701277+0000 mgr.y (mgr.14140) 48 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add", "daemon_type": "mon", "placement": "smithi164:172.21.15.164=b", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:22:50.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[29319]: audit 2021-02-10T04:22:49.702513+0000 mon.a (mon.0) 163 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-02-10T04:22:50.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[29319]: audit 2021-02-10T04:22:49.703177+0000 mon.a (mon.0) 164 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:22:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[29319]: cephadm 2021-02-10T04:22:49.703589+0000 mgr.y (mgr.14140) 49 : cephadm [INF] Deploying daemon mon.b on smithi164 2021-02-10T04:22:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[29319]: audit 2021-02-10T04:22:49.703862+0000 mon.a (mon.0) 165 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon.b", "key": "container_image"}]: dispatch 2021-02-10T04:22:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[29319]: cluster 2021-02-10T04:22:49.915547+0000 mgr.y (mgr.14140) 50 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:50.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[35961]: audit 2021-02-10T04:22:49.068324 2021-02-10T04:22:50.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[35961]: +0000 mon.a (mon.0) 159 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:22:50.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[35961]: cephadm 2021-02-10T04:22:49.068774+0000 mgr.y (mgr.14140) 46 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2021-02-10T04:22:50.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[35961]: audit 2021-02-10T04:22:49.069009+0000 mon.a (mon.0) 160 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-02-10T04:22:50.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[35961]: audit 2021-02-10T04:22:49.069524+0000 mon.a (mon.0) 161 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:22:50.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[35961]: cephadm 2021-02-10T04:22:49.070080+0000 mgr.y (mgr.14140) 47 : cephadm [INF] Reconfiguring daemon mon.c on smithi085 2021-02-10T04:22:50.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[35961]: audit 2021-02-10T04:22:49.070356+0000 mon.a (mon.0) 162 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon.c", "key": "container_image"}]: dispatch 2021-02-10T04:22:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[35961]: audit 2021-02-10T04:22:49.701277+0000 mgr.y (mgr.14140) 48 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add", "daemon_type": "mon", "placement": "smithi164:172.21.15.164=b", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:22:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[35961]: audit 2021-02-10T04:22:49.702513+0000 mon.a (mon.0) 163 : audit 2021-02-10T04:22:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-02-10T04:22:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[35961]: audit 2021-02-10T04:22:49.703177+0000 mon.a (mon.0) 164 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:22:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[35961]: cephadm 2021-02-10T04:22:49.703589+0000 2021-02-10T04:22:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[35961]: mgr.y (mgr.14140) 49 : cephadm [INF] Deploying daemon mon.b on smithi164 2021-02-10T04:22:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[35961]: 2021-02-10T04:22:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:22:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[35961]: 22:49.703862+0000 mon.a (mon.0 2021-02-10T04:22:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[35961]: ) 165 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon.b", "key": "container_image"}]: dispatch 2021-02-10T04:22:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:22:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[35961]: 04:22:49.915547+0000 mgr.y (mgr. 2021-02-10T04:22:50.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:50 smithi085 conmon[35961]: 14140) 50 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:51.940 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:51 smithi085 conmon[29319]: audit 2021-02-10T04:22:50.514041+0000 mon.a (mon.0) 166 : 2021-02-10T04:22:51.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:51 smithi085 conmon[29319]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:22:51.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:51 smithi085 conmon[29319]: audit 2021-02-10T04:22:50.514978+0000 mon.a (mon.0) 167 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:22:51.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:51 smithi085 conmon[29319]: audit 2021-02-10T04:22:51.434403+0000 mon.a (mon.0) 168 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:22:51.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:51 smithi085 conmon[35961]: audit 2021-02-10T04:22:50.514041+0000 mon.a (mon.0) 2021-02-10T04:22:51.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:51 smithi085 conmon[35961]: 166 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:22:51.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:51 smithi085 conmon[35961]: audit 2021-02-10T04:22:50.514978+0000 2021-02-10T04:22:51.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:51 smithi085 conmon[35961]: mon.a (mon.0) 167 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:22:51.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:51 smithi085 conmon[35961]: audit 2021-02-10T04:22:51.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:51 smithi085 conmon[35961]: 2021-02-10T04:22 2021-02-10T04:22:51.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:51 smithi085 conmon[35961]: :51.434403+0000 mon.a (mon. 2021-02-10T04:22:51.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:51 smithi085 conmon[35961]: 0) 168 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:22:53.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:52 smithi085 conmon[29319]: cephadm 2021-02-10T04:22:51.434776+0000 mgr.y (mgr.14140) 51 : cephadm [INF] 2021-02-10T04:22:53.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:52 smithi085 conmon[29319]: Refreshing smithi085 facts 2021-02-10T04:22:53.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:52 smithi085 conmon[29319]: audit 2021-02-10T04:22:51.790903+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:22:53.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:52 smithi085 conmon[29319]: cluster 2021-02-10T04:22:51.915780+0000 mgr.y (mgr.14140) 52 : cluster [DBG] 2021-02-10T04:22:53.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:52 smithi085 conmon[29319]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:53.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:52 smithi085 conmon[35961]: cephadm 2021-02-10T04:22:51.434776+0000 mgr.y (mgr. 2021-02-10T04:22:53.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:52 smithi085 conmon[35961]: 14140) 51 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:22:53.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:52 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:22:53.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:52 smithi085 conmon[35961]: 02-10T04:22:51.790903+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:22:53.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:52 smithi085 conmon[35961]: cluster 2021-02-10T04:22:53.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:52 smithi085 conmon[35961]: 2021-02-10T04:22:51.915780+0000 mgr.y 2021-02-10T04:22:53.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:52 smithi085 conmon[35961]: (mgr.14140) 52 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:57.981 INFO:teuthology.orchestra.run.smithi164.stdout:Deployed mon.b on host 'smithi164' 2021-02-10T04:22:58.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: cluster 2021-02-10T04:22:52. 2021-02-10T04:22:58.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: 966635+0000 mon.a (mon.0) 171 : cluster [INF] mon.a calling monitor election 2021-02-10T04:22:58.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:22:58.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: 22:52.966697+0000 mon.c ( 2021-02-10T04:22:58.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: mon.1) 2 : cluster [INF] 2021-02-10T04:22:58.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: mon.c calling monitor election 2021-02-10T04:22:58.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:22:58.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: -10T04:22:52.981685+0000 2021-02-10T04:22:58.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: mon.a (mon.0) 172 : 2021-02-10T04:22:58.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:22:58.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:22:58.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: 02-10T04:22:52. 2021-02-10T04:22:58.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: 981889+0000 mon.a (mon.0) 2021-02-10T04:22:58.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: 173 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:22:58.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:22:58.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: 02-10T04:22:53. 2021-02-10T04:22:58.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: 915983+0000 mgr.y (mgr.14140) 2021-02-10T04:22:58.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: 53 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:58.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:22:58.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: -10T04:22:53.962281+0000 mon.a (mon.0) 2021-02-10T04:22:58.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: 174 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-02-10T04:22:58.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:22:58.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: 02-10T04:22:54.962043+0000 mon.a (mon 2021-02-10T04:22:58.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: .0) 175 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-02-10T04:22:58.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: cluster 2021-02-10T04:22:58.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: 2021-02-10T04:22:55.916166+0000 2021-02-10T04:22:58.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: mgr.y (mgr.14140) 54 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:58.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: 2021-02-10T04:22:58.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:22:58.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: :22:55.962165+0000 mon.a (mon.0) 2021-02-10T04:22:58.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: 176 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-02-10T04:22:58.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: audit 2021-02-10T04:22:58.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: 2021-02-10T04:22:56. 2021-02-10T04:22:58.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: 962659+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-02-10T04:22:58.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: audit 2021 2021-02-10T04:22:58.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: -02-10T04:22:57.962652+0000 mon.a ( 2021-02-10T04:22:58.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: mon.0) 178 : audit [DBG] 2021-02-10T04:22:58.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-02-10T04:22:58.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:22:58.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: 10T04:22:57.968078+0000 mon.a (mon. 2021-02-10T04:22:58.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: 0) 179 : cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2021-02-10T04:22:58.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:22:58.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: 02-10T04:22:57.970689+0000 2021-02-10T04:22:58.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: mon.a (mon.0) 180 : cluster 2021-02-10T04:22:58.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0],b=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0],c=[v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0]} 2021-02-10T04:22:58.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:22:58.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: 04:22:57.970734+0000 mon.a (mon.0) 181 : cluster [DBG] 2021-02-10T04:22:58.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: fsmap 2021-02-10T04:22:58.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: cluster 2021-02-10T04:22:58.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:22:58.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: 22:57.970765+0000 mon.a (mon.0) 182 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-02-10T04:22:58.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:22:58.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: 02-10T04:22:57.970968+0000 2021-02-10T04:22:58.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: mon.a (mon.0) 183 : cluster 2021-02-10T04:22:58.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: [DBG] mgrmap e13: y(active, since 80s) 2021-02-10T04:22:58.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: cluster 2021-02-10T04:22:58.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:22:58.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: 22:57.971143+0000 mon.a (mon. 2021-02-10T04:22:58.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: 0) 184 : cluster [WRN] 2021-02-10T04:22:58.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2021-02-10T04:22:58.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:22:58.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: -10T04:22:57.974917+0000 mon.a ( 2021-02-10T04:22:58.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: mon.0) 185 : cluster [WRN] 2021-02-10T04:22:58.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2021-02-10T04:22:58.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: cluster 2021-02-10T04:22:58.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: 2021-02-10T04:22 2021-02-10T04:22:58.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: :57.974955+0000 mon.a (mon.0 2021-02-10T04:22:58.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: ) 186 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2021-02-10T04:22:58.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: 2021-02-10T04:22:58.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:22:58.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: -10T04:22:57.974976+0000 mon.a (mon.0 2021-02-10T04:22:58.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: ) 187 : cluster [WRN] mon.b (rank 2) addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] is down (out of quorum) 2021-02-10T04:22:58.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: audit 2021 2021-02-10T04:22:58.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: -02-10T04:22:57.978350+0000 mon.a 2021-02-10T04:22:58.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: (mon.0) 188 : audit 2021-02-10T04:22:58.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:22:58.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: cluster 2021-02-10T04:22:52.966635+0000 mon.a (mon.0) 171 : cluster 2021-02-10T04:22:58.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: [INF] mon.a calling monitor election 2021-02-10T04:22:58.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: cluster 2021-02-10T04:22:52.966697+0000 mon.c (mon.1) 2 : cluster [INF] mon.c calling monitor election 2021-02-10T04:22:58.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: audit 2021-02-10T04:22:52.981685+0000 mon.a (mon 2021-02-10T04:22:58.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: .0) 172 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:22:58.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: audit 2021-02-10T04:22:58.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:22:58.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: 22:52.981889+0000 mon.a (mon.0) 173 : 2021-02-10T04:22:58.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:22:58.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: cluster 2021-02-10T04:22:58.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: 2021-02-10T04:22:53.915983 2021-02-10T04:22:58.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 53 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:58.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: 2021-02-10T04:22:58.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: audit 2021-02-10T04: 2021-02-10T04:22:58.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: 22:53.962281+0000 mon.a (mon.0) 2021-02-10T04:22:58.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: 174 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-02-10T04:22:58.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: audit 2021-02-10T04:22:58.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: 2021-02-10T04:22:54.962043+0000 2021-02-10T04:22:58.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: mon.a (mon.0) 175 : audit [DBG] 2021-02-10T04:22:58.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-02-10T04:22:58.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:22:58.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: -02-10T04:22:55.916166+0000 2021-02-10T04:22:58.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: mgr.y (mgr.14140) 54 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:58.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: 2021-02-10T04:22:58.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: audit 2021-02-10T 2021-02-10T04:22:58.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: 04:22:55.962165+0000 mon.a (mon. 2021-02-10T04:22:58.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: 0) 176 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-02-10T04:22:58.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: audit 2021 2021-02-10T04:22:58.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: -02-10T04:22:56.962659+0000 2021-02-10T04:22:58.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: mon.a (mon.0) 177 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-02-10T04:22:58.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: 2021-02-10T04:22:58.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: audit 2021 2021-02-10T04:22:58.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: -02-10T04:22:57.962652 2021-02-10T04:22:58.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: +0000 mon.a (mon.0) 178 : audit [DBG] 2021-02-10T04:22:58.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-02-10T04:22:58.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: cluster 2021-02-10T04:22:57.968078+0000 mon.a ( 2021-02-10T04:22:58.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: mon.0) 179 : cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2021-02-10T04:22:58.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: cluster 2021-02-10T04:22:58.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:22:58.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: T04:22:57.970689+0000 mon.a (mon.0) 2021-02-10T04:22:58.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: 180 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0],b=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0],c=[v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0]} 2021-02-10T04:22:58.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: cluster 2021-02-10T04:22:58.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: 2021-02-10T04:22:57.970734 2021-02-10T04:22:58.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: +0000 mon.a (mon.0) 181 : cluster [DBG] 2021-02-10T04:22:58.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: fsmap 2021-02-10T04:22:58.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:22:58.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: 04:22:57.970765+0000 mon.a (mon.0 2021-02-10T04:22:58.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: ) 182 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-02-10T04:22:58.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: cluster 2021-02-10T04:22:58.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: 2021-02-10T04:22:57.970968 2021-02-10T04:22:58.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: +0000 mon.a (mon.0) 183 : cluster [DBG] 2021-02-10T04:22:58.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: mgrmap e13: y(active, since 80s) 2021-02-10T04:22:58.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: cluster 2021-02-10T04:22 2021-02-10T04:22:58.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: :57.971143+0000 mon.a (mon.0) 184 : 2021-02-10T04:22:58.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2021-02-10T04:22:58.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:22:58.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: :22:57.974917+0000 mon.a (mon.0) 2021-02-10T04:22:58.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: 185 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2021-02-10T04:22:58.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:22:58.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: T04:22:57.974955+0000 mon.a (mon.0 2021-02-10T04:22:58.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: ) 186 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2021-02-10T04:22:58.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:22:58.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: 02-10T04:22:57.974976+0000 mon.a ( 2021-02-10T04:22:58.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: mon.0) 187 : cluster [WRN] mon.b (rank 2) addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] is down (out of quorum) 2021-02-10T04:22:58.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:22:58.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: 02-10T04:22:57.978350+0000 2021-02-10T04:22:58.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:58 smithi085 conmon[29319]: mon.a (mon.0) 188 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:22:58.575 DEBUG:teuthology.orchestra.run.smithi164:mon.b> sudo journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mon.b.service 2021-02-10T04:22:58.577 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2021-02-10T04:22:58.578 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph mon dump -f json 2021-02-10T04:22:58.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:-- Logs begin at Wed 2021-02-10 04:09:33 UTC. -- 2021-02-10T04:22:59.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[29319]: cluster 2021-02-10T04:22:57.916470+0000 mgr.y (mgr.14140) 55 : cluster 2021-02-10T04:22:59.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[29319]: [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:59.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[29319]: audit 2021-02-10T04:22:58.892881+0000 mon.a (mon.0) 189 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:22:59.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[29319]: audit 2021-02-10T04:22:58.961739+0000 mon.a (mon.0) 190 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:22:59.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[29319]: audit 2021-02-10T04:22:58.963296+0000 mon.a (mon.0) 191 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-02-10T04:22:59.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[29319]: audit 2021-02-10T04:22:58.965763+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:22:59.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[29319]: 2021-02-10T04:22:59.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[29319]: audit 2021-02-10T04:22:58.966290+0000 mon.a 2021-02-10T04:22:59.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[29319]: (mon.0) 193 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-02-10T04:22:59.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:22:59.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[29319]: 02-10T04:22:58.966854+0000 2021-02-10T04:22:59.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[29319]: mon.a (mon.0) 194 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:22:59.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[29319]: 2021-02-10T04:22:59.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[29319]: audit 2021-02-10T04:22:59.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[29319]: 2021-02-10T04:22:58.967671+0000 mon.a 2021-02-10T04:22:59.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[29319]: (mon.0) 195 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon.a", "key": "container_image"}]: dispatch 2021-02-10T04:22:59.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: cluster 2021-02-10T04:22:57. 2021-02-10T04:22:59.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: 916470+0000 mgr.y (mgr.14140) 55 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:22:59.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:22:59.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: :22:58.892881+0000 mon.a 2021-02-10T04:22:59.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: (mon.0) 189 : audit 2021-02-10T04:22:59.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:22:59.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: audit 2021 2021-02-10T04:22:59.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: -02-10T04:22: 2021-02-10T04:22:59.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: 58.961739+0000 mon.a (mon 2021-02-10T04:22:59.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: .0) 190 : audit [INF] 2021-02-10T04:22:59.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:22:59.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: audit 2021 2021-02-10T04:22:59.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: -02-10T04:22: 2021-02-10T04:22:59.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: 58.963296+0000 mon.a (mon. 2021-02-10T04:22:59.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: 0) 191 : audit [DBG] 2021-02-10T04:22:59.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-02-10T04:22:59.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: audit 2021 2021-02-10T04:22:59.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: -02-10T04:22 2021-02-10T04:22:59.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: :58.965763+0000 mon.a ( 2021-02-10T04:22:59.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: mon.0) 192 : audit [INF] 2021-02-10T04:22:59.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:22:59.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:22:59.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: 02-10T04:22:58 2021-02-10T04:22:59.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: .966290+0000 mon.a (mon. 2021-02-10T04:22:59.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: 0) 193 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-02-10T04:22:59.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: 2021-02-10T04:22:59.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:22:59.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: -10T04:22:58 2021-02-10T04:22:59.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: .966854+0000 mon.a (mon.0 2021-02-10T04:22:59.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: ) 194 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:22:59.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: audit 2021 2021-02-10T04:22:59.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: -02-10T04:22 2021-02-10T04:22:59.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: :58.967671+0000 mon.a ( 2021-02-10T04:22:59.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: mon.0) 195 : audit [DBG] 2021-02-10T04:22:59.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:22:59 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon.a", "key": "container_image"}]: dispatch 2021-02-10T04:23:00.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:22:59 smithi164 conmon[29288]: cluster 2021-02-10T04:22:57.916470+0000 mgr.y (mgr.14140) 55 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:00.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:22:59 smithi164 conmon[29288]: audit 2021-02-10T04:22:58.892881+0000 mon.a (mon.0) 189 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:00.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:22:59 smithi164 conmon[29288]: audit 2021-02-10T04:22:58.961739+0000 mon.a (mon.0) 190 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:00.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:22:59 smithi164 conmon[29288]: audit 2021-02-10T04:22:58.963296+0000 mon.a (mon.0) 191 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-02-10T04:23:00.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:22:59 smithi164 conmon[29288]: audit 2021-02-10T04:22:58.965763+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:00.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:22:59 smithi164 conmon[29288]: audit 2021-02-10T04:22:58.966290+0000 mon.a (mon.0) 193 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-02-10T04:23:00.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:22:59 smithi164 conmon[29288]: audit 2021-02-10T04:22:58.966854+0000 mon.a (mon.0) 194 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:22:59 smithi164 conmon[29288]: audit 2021-02-10T04:22:58.967671+0000 mon.a (mon.0) 195 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon.a", "key": "container_image"}]: dispatch 2021-02-10T04:23:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:00 smithi164 conmon[29288]: cluster 2021-02-10T04:22:54.963966+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2021-02-10T04:23:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:00 smithi164 conmon[29288]: cluster 2021-02-10T04:22:59.977144+0000 mon.b (mon.2) 2 : cluster [INF] mon.b calling monitor election 2021-02-10T04:23:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:00 smithi164 conmon[29288]: cluster 2021-02-10T04:22:59.977425+0000 mon.a (mon.0) 197 : cluster [INF] mon.a calling monitor election 2021-02-10T04:23:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:00 smithi164 conmon[29288]: cluster 2021-02-10T04:22:59.977426+0000 mon.c (mon.1) 3 : cluster [INF] mon.c calling monitor election 2021-02-10T04:23:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:00 smithi164 conmon[29288]: cluster 2021-02-10T04:22:59.979460+0000 mon.a (mon.0) 198 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2021-02-10T04:23:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:00 smithi164 conmon[29288]: cluster 2021-02-10T04:22:59.984008+0000 2021-02-10T04:23:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:00 smithi164 conmon[29288]: mon.a (mon.0) 199 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0],b=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0],c=[v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0]} 2021-02-10T04:23:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:00 smithi164 conmon[29288]: cluster 2021-02-10T04:22:59.984062+0000 mon.a (mon.0) 200 : cluster [DBG] fsmap 2021-02-10T04:23:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:00 smithi164 conmon[29288]: cluster 2021-02-10T04:22:59.984084+0000 mon.a 2021-02-10T04:23:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:00 smithi164 conmon[29288]: (mon.0) 201 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-02-10T04:23:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:00 smithi164 conmon[29288]: cluster 2021-02-10T04:23:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:00 smithi164 conmon[29288]: 2021-02-10T04:22:59.984227+0000 2021-02-10T04:23:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:00 smithi164 conmon[29288]: mon.a (mon.0) 202 : cluster [DBG] mgrmap e13: y(active, since 82s) 2021-02-10T04:23:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:00 smithi164 conmon[29288]: 2021-02-10T04:23:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:00 smithi164 conmon[29288]: cluster 2021-02-10T04:22: 2021-02-10T04:23:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:00 smithi164 conmon[29288]: 59.984340+0000 mon.a (mon.0) 203 : 2021-02-10T04:23:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:00 smithi164 conmon[29288]: cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2021-02-10T04:23:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:00 smithi164 conmon[29288]: cluster 2021-02-10T04:23:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:00 smithi164 conmon[29288]: 2021-02-10T04:22:59 2021-02-10T04:23:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:00 smithi164 conmon[29288]: .984357+0000 mon.a (mon.0) 204 : cluster 2021-02-10T04:23:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:00 smithi164 conmon[29288]: [INF] Cluster is now healthy 2021-02-10T04:23:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:00 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:23:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:00 smithi164 conmon[29288]: 10T04:22:59.989129+0000 mon.a ( 2021-02-10T04:23:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:00 smithi164 conmon[29288]: mon.0) 205 : cluster [INF] overall HEALTH_OK 2021-02-10T04:23:00.308 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[29319]: cluster 2021-02-10T04:22:54.963966+0000 mon.b (mon.2) 1 : 2021-02-10T04:23:00.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[29319]: cluster [INF] mon.b calling monitor election 2021-02-10T04:23:00.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[29319]: cluster 2021-02-10T04:22:59.977144+0000 mon.b (mon.2) 2 : cluster [INF] mon.b calling monitor election 2021-02-10T04:23:00.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[29319]: cluster 2021-02-10T04:22:59.977425+0000 mon.a (mon.0) 197 : cluster [INF] mon.a calling monitor election 2021-02-10T04:23:00.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[29319]: cluster 2021-02-10T04:22:59.977426+0000 mon.c (mon.1) 3 : cluster [INF] mon.c calling monitor election 2021-02-10T04:23:00.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[29319]: cluster 2021-02-10T04:22:59.979460+0000 mon.a (mon.0) 198 : cluster 2021-02-10T04:23:00.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[29319]: [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2021-02-10T04:23:00.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[29319]: cluster 2021-02-10T04:22:59.984008+0000 mon.a (mon.0) 199 : 2021-02-10T04:23:00.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[29319]: cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0],b=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0],c=[v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0]} 2021-02-10T04:23:00.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[29319]: cluster 2021-02-10T04:23:00.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[29319]: 2021-02-10T04:22:59.984062+0000 mon.a (mon. 2021-02-10T04:23:00.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[29319]: 0) 200 : cluster [DBG] fsmap 2021-02-10T04:23:00.310 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:23:00.310 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[29319]: 02-10T04:22:59.984084+0000 mon.a (mon.0 2021-02-10T04:23:00.310 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[29319]: ) 201 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-02-10T04:23:00.310 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[29319]: cluster 2021-02-10T04:22:59.984227+0000 2021-02-10T04:23:00.310 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[29319]: mon.a (mon.0) 202 : cluster [DBG] mgrmap e13: y(active, since 82s) 2021-02-10T04:23:00.310 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[29319]: cluster 2021-02-10T04:22:59. 2021-02-10T04:23:00.310 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[29319]: 984340+0000 mon.a (mon.0) 203 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2021-02-10T04:23:00.310 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[29319]: cluster 2021-02-10T04:22:59 2021-02-10T04:23:00.310 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[29319]: .984357+0000 mon.a (mon.0) 204 : cluster [INF] Cluster is now healthy 2021-02-10T04:23:00.310 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:23:00.310 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[29319]: 22:59.989129+0000 mon.a (mon.0) 205 : cluster [INF] 2021-02-10T04:23:00.310 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[29319]: overall HEALTH_OK 2021-02-10T04:23:00.311 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: cluster 2021-02-10T04:22:54.963966+0000 2021-02-10T04:23:00.311 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2021-02-10T04:23:00.311 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:23:00.311 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: 02-10T04:22:59.977144+0000 mon.b (mon.2) 2 : cluster 2021-02-10T04:23:00.311 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: [INF] mon.b calling monitor election 2021-02-10T04:23:00.311 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:23:00.311 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: 22:59.977425+0000 mon.a (mon.0 2021-02-10T04:23:00.311 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: ) 197 : cluster [INF] mon.a calling monitor election 2021-02-10T04:23:00.311 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:23:00.311 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: -02-10T04:22:59.977426+0000 2021-02-10T04:23:00.311 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: mon.c (mon.1) 3 : cluster [INF] mon.c calling monitor election 2021-02-10T04:23:00.311 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: 2021-02-10T04:23:00.311 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: cluster 2021-02-10T04:22 2021-02-10T04:23:00.311 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: :59.979460+0000 mon.a (mon.0) 2021-02-10T04:23:00.312 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: 198 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2021-02-10T04:23:00.312 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: cluster 2021-02-10T04:23:00.312 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: 2021-02-10T04:22:59.984008+0000 2021-02-10T04:23:00.312 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: mon.a (mon.0) 199 : cluster [DBG] 2021-02-10T04:23:00.312 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: monmap e3: 3 mons at {a=[v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0],b=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0],c=[v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0]} 2021-02-10T04:23:00.312 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:23:00.312 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: 10T04:22:59.984062+0000 mon.a ( 2021-02-10T04:23:00.312 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: mon.0) 200 : cluster [DBG] fsmap 2021-02-10T04:23:00.312 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: cluster 2021-02-10T04:23:00.312 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: 2021-02-10T04:22:59.984084 2021-02-10T04:23:00.312 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: +0000 mon.a (mon.0) 201 : cluster 2021-02-10T04:23:00.312 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-02-10T04:23:00.312 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: cluster 2021-02-10T04:23:00.312 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: 2021-02-10T04:22:59 2021-02-10T04:23:00.313 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: .984227+0000 mon.a (mon.0) 202 : 2021-02-10T04:23:00.313 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: cluster [DBG] mgrmap e13: y(active, since 82s) 2021-02-10T04:23:00.313 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:23:00.313 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: -10T04:22:59.984340+0000 2021-02-10T04:23:00.313 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: mon.a (mon.0) 203 : cluster [INF] 2021-02-10T04:23:00.313 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2021-02-10T04:23:00.313 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: cluster 2021-02-10T04:22 2021-02-10T04:23:00.313 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: :59.984357+0000 mon.a (mon.0) 204 2021-02-10T04:23:00.313 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: : cluster [INF] Cluster is now healthy 2021-02-10T04:23:00.313 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:23:00.313 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: 02-10T04:22:59.989129+0000 2021-02-10T04:23:00.313 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: mon.a (mon.0) 205 : cluster [INF] 2021-02-10T04:23:00.313 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:00 smithi085 conmon[35961]: overall HEALTH_OK 2021-02-10T04:23:00.419 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-02-10T04:23:00.420 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":3,"fsid":"42224fce-6b57-11eb-8feb-001a4aab830c","modified":"2021-02-10T04:22:52.962887Z","created":"2021-02-10T04:20:55.445955Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:3300","nonce":0},{"type":"v1","addr":"172.21.15.85:6789","nonce":0}]},"addr":"172.21.15.85:6789/0","public_addr":"172.21.15.85:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:3301","nonce":0},{"type":"v1","addr":"172.21.15.85:6790","nonce":0}]},"addr":"172.21.15.85:6790/0","public_addr":"172.21.15.85:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2021-02-10T04:23:00.421 INFO:teuthology.orchestra.run.smithi164.stderr:dumped monmap epoch 3 2021-02-10T04:23:00.945 INFO:tasks.cephadm:Generating final ceph.conf file... 2021-02-10T04:23:00.946 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph config generate-minimal-conf 2021-02-10T04:23:01.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:01 smithi085 conmon[29319]: cluster 2021-02-10T04:22:59.916681+0000 mgr.y (mgr.14140) 58 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:01.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:01 smithi085 conmon[29319]: audit 2021-02-10T04:23:00.353810+0000 mon.a (mon.0) 206 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:01.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:01 smithi085 conmon[29319]: audit 2021-02-10T04:23:00.354434+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mgr.y"}]: dispatch 2021-02-10T04:23:01.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:01 smithi085 conmon[29319]: audit 2021-02-10T04:23:00.354985+0000 mon.a (mon.0) 208 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:01.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:01 smithi085 conmon[29319]: audit 2021-02-10T04:23:00.355779+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mgr.y", "key": "container_image"}]: dispatch 2021-02-10T04:23:01.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:01 smithi085 conmon[29319]: audit 2021-02-10T04:23:00.420112+0000 mon.a (mon.0) 210 : audit [DBG] from='client.? 172.21.15.164:0/2890582834' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-02-10T04:23:01.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:01 smithi085 conmon[29319]: audit 2021-02-10T04:23:00.962810+0000 mon.a (mon.0) 211 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-02-10T04:23:01.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:01 smithi085 conmon[35961]: cluster 2021-02-10T04:22:59.916681+0000 mgr.y (mgr.14140) 58 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:01.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:01 smithi085 conmon[35961]: audit 2021-02-10T04:23:00.353810+0000 mon.a (mon.0) 206 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:01.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:01 smithi085 conmon[35961]: audit 2021-02-10T04:23:00.354434+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mgr.y"}]: dispatch 2021-02-10T04:23:01.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:01 smithi085 conmon[35961]: audit 2021-02-10T04:23:00.354985+0000 mon.a (mon.0) 208 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:01.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:01 smithi085 conmon[35961]: audit 2021-02-10T04:23:00.355779+0000 mon.a (mon.0 2021-02-10T04:23:01.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:01 smithi085 conmon[35961]: ) 209 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mgr.y", "key": "container_image"}]: dispatch 2021-02-10T04:23:01.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:01 smithi085 conmon[35961]: audit 2021-02-10T04:23:00.420112+0000 mon.a (mon.0) 210 : audit [DBG] from='client.? 172.21.15.164:0/2890582834' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-02-10T04:23:01.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:01 smithi085 conmon[35961]: audit 2021-02-10T04:23:00.962810+0000 mon.a (mon.0) 211 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-02-10T04:23:01.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:01 smithi164 conmon[29288]: cluster 2021-02-10T04:22:59.916681+0000 mgr.y (mgr.14140) 58 2021-02-10T04:23:01.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:01 smithi164 conmon[29288]: : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:01.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:01 smithi164 conmon[29288]: audit 2021-02-10T04:23:00.353810+0000 mon.a (mon.0) 206 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:01.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:01 smithi164 conmon[29288]: audit 2021-02-10T04:23:00.354434+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mgr.y"}]: dispatch 2021-02-10T04:23:01.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:01 smithi164 conmon[29288]: audit 2021-02-10T04:23:00.354985+0000 mon.a (mon.0) 208 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:01.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:01 smithi164 conmon[29288]: audit 2021-02-10T04:23:00.355779+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mgr.y", "key": "container_image"}]: dispatch 2021-02-10T04:23:01.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:01 smithi164 conmon[29288]: audit 2021-02-10T04:23:00.420112+0000 mon.a (mon.0) 210 : audit [DBG] from='client.? 172.21.15.164:0/2890582834' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-02-10T04:23:01.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:01 smithi164 conmon[29288]: audit 2021-02-10T04:23:01.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:01 smithi164 conmon[29288]: 2021-02-10T04:23:00.962810+0000 mon.a (mon.0) 211 : audit 2021-02-10T04:23:01.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:01 smithi164 conmon[29288]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-02-10T04:23:02.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:02 smithi085 conmon[29319]: cephadm 2021-02-10T04:23:00.354217+0000 mgr.y (mgr.14140 2021-02-10T04:23:02.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:02 smithi085 conmon[29319]: ) 59 : cephadm [INF] Reconfiguring mgr.y (monmap changed)... 2021-02-10T04:23:02.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:02 smithi085 conmon[29319]: cephadm 2021-02-10T04:23:00.355418+0000 mgr.y (mgr.14140) 60 : cephadm [INF] Reconfiguring daemon mgr.y on smithi085 2021-02-10T04:23:02.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:02 smithi085 conmon[29319]: audit 2021-02-10T04:23:01.813092+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:02.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:02 smithi085 conmon[29319]: audit 2021-02-10T04:23:01.813845+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-02-10T04:23:02.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:02 smithi085 conmon[29319]: audit 2021-02-10T04:23:01.814867+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:02.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:02 smithi085 conmon[29319]: audit 2021-02-10T04:23:01.815883+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon.c", "key": "container_image"}]: dispatch 2021-02-10T04:23:02.312 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:02 smithi085 conmon[35961]: cephadm 2021-02-10T04:23:02.312 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:02 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:23:02.312 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:02 smithi085 conmon[35961]: 23:00.354217+0000 mgr.y (mgr.14140) 59 : cephadm [INF] Reconfiguring mgr.y (monmap changed)... 2021-02-10T04:23:02.313 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:02 smithi085 conmon[35961]: cephadm 2021-02-10 2021-02-10T04:23:02.313 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:02 smithi085 conmon[35961]: T04:23:00.355418+0000 mgr.y (mgr.14140) 2021-02-10T04:23:02.313 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:02 smithi085 conmon[35961]: 60 : cephadm [INF] Reconfiguring daemon mgr.y on smithi085 2021-02-10T04:23:02.313 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:02 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:23:02.313 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:02 smithi085 conmon[35961]: T04:23:01.813092+0000 mon.a (mon. 2021-02-10T04:23:02.313 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:02 smithi085 conmon[35961]: 0) 212 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:02.313 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:02 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:23:02.313 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:02 smithi085 conmon[35961]: 10T04:23:01.813845+0000 mon.a ( 2021-02-10T04:23:02.313 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:02 smithi085 conmon[35961]: mon.0) 213 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-02-10T04:23:02.313 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:02 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:23:02.313 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:02 smithi085 conmon[35961]: 10T04:23:01.814867+0000 mon.a (mon. 2021-02-10T04:23:02.313 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:02 smithi085 conmon[35961]: 0) 214 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:02.313 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:02 smithi085 conmon[35961]: audit 2021-02-10T04:23:02.313 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:02 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:23:02.314 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:02 smithi085 conmon[35961]: T04:23:01.815883+0000 mon.a ( 2021-02-10T04:23:02.314 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:02 smithi085 conmon[35961]: mon.0) 215 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon.c", "key": "container_image"}]: dispatch 2021-02-10T04:23:02.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:02 smithi164 conmon[29288]: cephadm 2021-02-10T04:23:00.354217+0000 mgr.y 2021-02-10T04:23:02.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:02 smithi164 conmon[29288]: (mgr.14140) 59 : cephadm [INF] Reconfiguring mgr.y (monmap changed)... 2021-02-10T04:23:02.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:02 smithi164 conmon[29288]: cephadm 2021-02-10T04:23:00.355418+0000 mgr.y (mgr.14140) 60 : cephadm [INF] Reconfiguring daemon mgr.y on smithi085 2021-02-10T04:23:02.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:02 smithi164 conmon[29288]: audit 2021-02-10T04:23:01.813092+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:02.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:02 smithi164 conmon[29288]: audit 2021-02-10T04:23:01.813845+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-02-10T04:23:02.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:02 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:23:02.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:02 smithi164 conmon[29288]: T04:23:01.814867+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:02.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:02 smithi164 conmon[29288]: audit 2021-02-10T04:23:01.815883+0000 mon.a (mon.0) 215 : audit 2021-02-10T04:23:02.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:02 smithi164 conmon[29288]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon.c", "key": "container_image"}]: dispatch 2021-02-10T04:23:03.371 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:03 smithi085 conmon[29319]: cephadm 2021-02-10T04:23:01.813565+0000 2021-02-10T04:23:03.371 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:03 smithi085 conmon[29319]: mgr.y (mgr.14140) 61 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2021-02-10T04:23:03.371 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:03 smithi085 conmon[29319]: cephadm 2021-02-10T04:23:01.815400+0000 mgr.y (mgr.14140) 62 : cephadm [INF] Reconfiguring daemon mon.c on smithi085 2021-02-10T04:23:03.371 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:03 smithi085 conmon[29319]: cluster 2021-02-10T04:23:01.916939+0000 mgr.y (mgr.14140) 63 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:03.371 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:03 smithi085 conmon[35961]: cephadm 2021-02-10T04:23:03.371 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:03 smithi085 conmon[35961]: 2021-02-10T04:23 2021-02-10T04:23:03.371 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:03 smithi085 conmon[35961]: :01.813565+0000 mgr.y (mgr.14140) 61 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2021-02-10T04:23:03.371 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:03 smithi085 conmon[35961]: cephadm 2021-02-10T04:23:01.815400+0000 2021-02-10T04:23:03.371 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:03 smithi085 conmon[35961]: mgr.y (mgr.14140) 62 : cephadm [INF] 2021-02-10T04:23:03.371 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:03 smithi085 conmon[35961]: Reconfiguring daemon mon.c on smithi085 2021-02-10T04:23:03.371 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:03 smithi085 conmon[35961]: cluster 2021-02-10T04:23 2021-02-10T04:23:03.372 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:03 smithi085 conmon[35961]: :01.916939+0000 mgr.y (mgr.14140) 63 : 2021-02-10T04:23:03.372 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:03 smithi085 conmon[35961]: cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:03.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:03 smithi164 conmon[29288]: cephadm 2021-02-10T04:23:01.813565+0000 2021-02-10T04:23:03.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:03 smithi164 conmon[29288]: mgr.y (mgr.14140) 61 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2021-02-10T04:23:03.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:03 smithi164 conmon[29288]: cephadm 2021-02-10T04:23:01.815400+0000 mgr.y (mgr.14140) 62 : cephadm [INF] 2021-02-10T04:23:03.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:03 smithi164 conmon[29288]: Reconfiguring daemon mon.c on smithi085 2021-02-10T04:23:03.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:03 smithi164 conmon[29288]: cluster 2021-02-10T04:23:01.916939+0000 mgr.y (mgr. 2021-02-10T04:23:03.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:03 smithi164 conmon[29288]: 14140) 63 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:03.451 INFO:teuthology.orchestra.run.smithi085.stdout:# minimal ceph.conf for 42224fce-6b57-11eb-8feb-001a4aab830c 2021-02-10T04:23:03.451 INFO:teuthology.orchestra.run.smithi085.stdout:[global] 2021-02-10T04:23:03.451 INFO:teuthology.orchestra.run.smithi085.stdout: fsid = 42224fce-6b57-11eb-8feb-001a4aab830c 2021-02-10T04:23:03.451 INFO:teuthology.orchestra.run.smithi085.stdout: mon_host = [v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0] [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] 2021-02-10T04:23:04.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:04 smithi085 conmon[29319]: audit 2021-02-10T04:23:03.451916+0000 mon.a (mon.0) 2021-02-10T04:23:04.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:04 smithi085 conmon[29319]: 216 : audit [DBG] from='client.? 172.21.15.85:0/2741527386' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:04.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:04 smithi085 conmon[35961]: audit 2021-02-10T04:23:03. 2021-02-10T04:23:04.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:04 smithi085 conmon[35961]: 451916+0000 mon.a (mon.0) 216 : audit [DBG] from='client.? 172.21.15.85:0/2741527386' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:04.317 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2021-02-10T04:23:04.318 DEBUG:teuthology.orchestra.run.smithi085:> set -ex 2021-02-10T04:23:04.318 DEBUG:teuthology.orchestra.run.smithi085:> sudo dd of=/etc/ceph/ceph.conf 2021-02-10T04:23:04.345 DEBUG:teuthology.orchestra.run.smithi085:> set -ex 2021-02-10T04:23:04.346 DEBUG:teuthology.orchestra.run.smithi085:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2021-02-10T04:23:04.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:04 smithi164 conmon[29288]: audit 2021-02-10T04:23:03.451916+0000 2021-02-10T04:23:04.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:04 smithi164 conmon[29288]: mon.a (mon.0) 216 : audit [DBG] from='client.? 172.21.15.85:0/2741527386' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:04.409 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2021-02-10T04:23:04.410 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/ceph.conf 2021-02-10T04:23:04.439 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2021-02-10T04:23:04.439 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2021-02-10T04:23:04.503 INFO:tasks.cephadm:Adding mgr.y on smithi085 2021-02-10T04:23:04.504 INFO:tasks.cephadm:Adding mgr.x on smithi164 2021-02-10T04:23:04.504 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph orch apply mgr '3;smithi085=y;smithi164=x' 2021-02-10T04:23:05.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:05 smithi164 conmon[29288]: cluster 2021-02-10T04:23:03.917209+0000 mgr.y ( 2021-02-10T04:23:05.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:05 smithi164 conmon[29288]: mgr.14140) 64 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:05.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:05 smithi164 conmon[29288]: audit 2021-02-10T04:23:04.150490+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:05.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:05 smithi164 conmon[29288]: audit 2021-02-10T04:23:04.151105+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-02-10T04:23:05.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:05 smithi164 conmon[29288]: audit 2021-02-10T04:23:04.151632+0000 mon.a (mon.0) 219 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:05.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:05 smithi164 conmon[29288]: audit 2021-02-10T04:23:04.152414+0000 mon.a (mon.0) 220 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon.b", "key": "container_image"}]: dispatch 2021-02-10T04:23:05.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:05 smithi085 conmon[29319]: cluster 2021-02-10T04:23:03.917209+0000 mgr.y (mgr.14140) 64 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:05.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:05 smithi085 conmon[29319]: audit 2021-02-10T04:23:04.150490+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:05.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:05 smithi085 conmon[29319]: audit 2021-02-10T04:23:04.151105+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-02-10T04:23:05.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:05 smithi085 conmon[29319]: audit 2021-02-10T04:23:04.151632+0000 mon.a (mon.0) 219 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:05.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:05 smithi085 conmon[29319]: audit 2021-02-10T04:23:04.152414+0000 mon.a (mon.0) 220 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon.b", "key": "container_image"}]: dispatch 2021-02-10T04:23:05.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:05 smithi085 conmon[35961]: cluster 2021-02-10T04:23:03.917209+0000 mgr.y ( 2021-02-10T04:23:05.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:05 smithi085 conmon[35961]: mgr.14140) 64 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:05.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:05 smithi085 conmon[35961]: audit 2021-02-10T04:23:04.150490+0000 mon.a ( 2021-02-10T04:23:05.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:05 smithi085 conmon[35961]: mon.0) 217 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:05.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:05 smithi085 conmon[35961]: audit 2021-02-10T04:23 2021-02-10T04:23:05.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:05 smithi085 conmon[35961]: :04.151105+0000 mon.a (mon.0) 218 2021-02-10T04:23:05.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:05 smithi085 conmon[35961]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-02-10T04:23:05.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:05 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:23:05.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:05 smithi085 conmon[35961]: 23:04.151632+0000 mon.a (mon.0) 2021-02-10T04:23:05.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:05 smithi085 conmon[35961]: 219 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:05.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:05 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:23:05.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:05 smithi085 conmon[35961]: T04:23:04.152414+0000 mon.a (mon 2021-02-10T04:23:05.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:05 smithi085 conmon[35961]: .0) 220 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon.b", "key": "container_image"}]: dispatch 2021-02-10T04:23:06.295 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:06 smithi164 conmon[29288]: cephadm 2021-02-10T04:23:04.150892+0000 mgr.y (mgr.14140) 65 : cephadm 2021-02-10T04:23:06.295 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:06 smithi164 conmon[29288]: [INF] Reconfiguring mon.b (monmap changed)... 2021-02-10T04:23:06.295 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:06 smithi164 conmon[29288]: cephadm 2021-02-10T04:23:04.152055+0000 mgr.y (mgr.14140) 66 : cephadm [INF] Reconfiguring daemon mon.b on smithi164 2021-02-10T04:23:06.296 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:06 smithi164 conmon[29288]: cluster 2021-02-10T04:23:05.917480+0000 mgr.y (mgr.14140) 67 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:06.296 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:06 smithi164 conmon[29288]: audit 2021-02-10T04:23:05.921223+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:06.296 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:06 smithi164 conmon[29288]: audit 2021-02-10T04:23:05.922612+0000 2021-02-10T04:23:06.296 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:06 smithi164 conmon[29288]: mon.a (mon.0) 222 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:06.296 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:06 smithi164 conmon[29288]: audit 2021-02-10T04:23:05 2021-02-10T04:23:06.296 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:06 smithi164 conmon[29288]: .923034+0000 mon.a (mon.0) 223 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:06.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[29319]: cephadm 2021-02-10T04:23:04.150892+0000 mgr.y (mgr.14140) 2021-02-10T04:23:06.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[29319]: 65 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2021-02-10T04:23:06.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[29319]: cephadm 2021-02-10T04:23:04.152055+0000 mgr.y (mgr.14140) 66 : cephadm [INF] Reconfiguring daemon mon.b on smithi164 2021-02-10T04:23:06.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[29319]: cluster 2021-02-10T04:23:05.917480+0000 mgr.y (mgr.14140) 67 : cluster 2021-02-10T04:23:06.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[29319]: [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:06.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:23:06.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[29319]: 10T04:23:05.921223+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:06.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:23:06.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[29319]: T04:23:05.922612+0000 mon.a (mon.0) 2021-02-10T04:23:06.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[29319]: 222 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:06.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[29319]: audit 2021-02-10T 2021-02-10T04:23:06.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[29319]: 04:23:05.923034+0000 mon.a (mon.0) 223 : audit 2021-02-10T04:23:06.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[29319]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:06.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[35961]: cephadm 2021-02-10T04:23:04.150892+0000 2021-02-10T04:23:06.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[35961]: mgr.y (mgr.14140) 65 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2021-02-10T04:23:06.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[35961]: cephadm 2021-02 2021-02-10T04:23:06.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[35961]: -10T04:23:04.152055+0000 mgr.y (mgr.14140) 66 : cephadm 2021-02-10T04:23:06.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[35961]: [INF] Reconfiguring daemon mon.b on smithi164 2021-02-10T04:23:06.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:23:06.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[35961]: :23:05.917480+0000 mgr.y (mgr.14140 2021-02-10T04:23:06.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[35961]: ) 67 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:06.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:23:06.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[35961]: -10T04:23:05.921223+0000 mon.a 2021-02-10T04:23:06.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[35961]: (mon.0) 221 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:06.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[35961]: audit 2021 2021-02-10T04:23:06.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[35961]: -02-10T04:23:05.922612 2021-02-10T04:23:06.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[35961]: +0000 mon.a (mon.0) 222 : audit [DBG] 2021-02-10T04:23:06.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:06.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:23:06.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[35961]: -10T04:23:05.923034+0000 2021-02-10T04:23:06.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[35961]: mon.a (mon.0) 223 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:06.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:06 smithi085 conmon[35961]: 2021-02-10T04:23:06.634 INFO:teuthology.orchestra.run.smithi164.stdout:Scheduled mgr update... 2021-02-10T04:23:07.285 DEBUG:teuthology.orchestra.run.smithi164:mgr.x> sudo journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mgr.x.service 2021-02-10T04:23:07.293 INFO:tasks.cephadm:Deploying OSDs... 2021-02-10T04:23:07.293 DEBUG:teuthology.orchestra.run.smithi085:> set -ex 2021-02-10T04:23:07.293 DEBUG:teuthology.orchestra.run.smithi085:> dd if=/scratch_devs of=/dev/stdout 2021-02-10T04:23:07.311 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-02-10T04:23:07.311 DEBUG:teuthology.orchestra.run.smithi085:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-02-10T04:23:07.319 INFO:journalctl@ceph.mgr.x.smithi164.stdout:-- Logs begin at Wed 2021-02-10 04:09:33 UTC. -- 2021-02-10T04:23:07.368 INFO:teuthology.orchestra.run.smithi085.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-02-10T04:23:07.368 INFO:teuthology.orchestra.run.smithi085.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T04:23:07.369 INFO:teuthology.orchestra.run.smithi085.stdout:Device: 6h/6d Inode: 111068 Links: 1 2021-02-10T04:23:07.369 INFO:teuthology.orchestra.run.smithi085.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T04:23:07.369 INFO:teuthology.orchestra.run.smithi085.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T04:23:07.369 INFO:teuthology.orchestra.run.smithi085.stdout:Access: 2021-02-10 04:22:38.854678418 +0000 2021-02-10T04:23:07.369 INFO:teuthology.orchestra.run.smithi085.stdout:Modify: 2021-02-10 04:15:59.034816408 +0000 2021-02-10T04:23:07.369 INFO:teuthology.orchestra.run.smithi085.stdout:Change: 2021-02-10 04:15:59.034816408 +0000 2021-02-10T04:23:07.369 INFO:teuthology.orchestra.run.smithi085.stdout: Birth: - 2021-02-10T04:23:07.380 INFO:teuthology.orchestra.run.smithi085.stderr:1+0 records in 2021-02-10T04:23:07.380 INFO:teuthology.orchestra.run.smithi085.stderr:1+0 records out 2021-02-10T04:23:07.380 INFO:teuthology.orchestra.run.smithi085.stderr:512 bytes copied, 0.000239237 s, 2.1 MB/s 2021-02-10T04:23:07.382 DEBUG:teuthology.orchestra.run.smithi085:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-02-10T04:23:07.438 INFO:teuthology.orchestra.run.smithi085.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-02-10T04:23:07.438 INFO:teuthology.orchestra.run.smithi085.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T04:23:07.438 INFO:teuthology.orchestra.run.smithi085.stdout:Device: 6h/6d Inode: 109988 Links: 1 2021-02-10T04:23:07.438 INFO:teuthology.orchestra.run.smithi085.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T04:23:07.439 INFO:teuthology.orchestra.run.smithi085.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T04:23:07.439 INFO:teuthology.orchestra.run.smithi085.stdout:Access: 2021-02-10 04:22:38.854678418 +0000 2021-02-10T04:23:07.439 INFO:teuthology.orchestra.run.smithi085.stdout:Modify: 2021-02-10 04:15:59.356810873 +0000 2021-02-10T04:23:07.439 INFO:teuthology.orchestra.run.smithi085.stdout:Change: 2021-02-10 04:15:59.356810873 +0000 2021-02-10T04:23:07.439 INFO:teuthology.orchestra.run.smithi085.stdout: Birth: - 2021-02-10T04:23:07.448 INFO:teuthology.orchestra.run.smithi085.stderr:1+0 records in 2021-02-10T04:23:07.448 INFO:teuthology.orchestra.run.smithi085.stderr:1+0 records out 2021-02-10T04:23:07.448 INFO:teuthology.orchestra.run.smithi085.stderr:512 bytes copied, 0.000211644 s, 2.4 MB/s 2021-02-10T04:23:07.450 DEBUG:teuthology.orchestra.run.smithi085:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-02-10T04:23:07.505 INFO:teuthology.orchestra.run.smithi085.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-02-10T04:23:07.505 INFO:teuthology.orchestra.run.smithi085.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T04:23:07.505 INFO:teuthology.orchestra.run.smithi085.stdout:Device: 6h/6d Inode: 110054 Links: 1 2021-02-10T04:23:07.505 INFO:teuthology.orchestra.run.smithi085.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T04:23:07.505 INFO:teuthology.orchestra.run.smithi085.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T04:23:07.507 INFO:teuthology.orchestra.run.smithi085.stdout:Access: 2021-02-10 04:22:38.854678418 +0000 2021-02-10T04:23:07.507 INFO:teuthology.orchestra.run.smithi085.stdout:Modify: 2021-02-10 04:15:59.684805236 +0000 2021-02-10T04:23:07.507 INFO:teuthology.orchestra.run.smithi085.stdout:Change: 2021-02-10 04:15:59.684805236 +0000 2021-02-10T04:23:07.508 INFO:teuthology.orchestra.run.smithi085.stdout: Birth: - 2021-02-10T04:23:07.515 INFO:teuthology.orchestra.run.smithi085.stderr:1+0 records in 2021-02-10T04:23:07.515 INFO:teuthology.orchestra.run.smithi085.stderr:1+0 records out 2021-02-10T04:23:07.515 INFO:teuthology.orchestra.run.smithi085.stderr:512 bytes copied, 0.000212941 s, 2.4 MB/s 2021-02-10T04:23:07.517 DEBUG:teuthology.orchestra.run.smithi085:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-02-10T04:23:07.572 INFO:teuthology.orchestra.run.smithi085.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-02-10T04:23:07.572 INFO:teuthology.orchestra.run.smithi085.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T04:23:07.572 INFO:teuthology.orchestra.run.smithi085.stdout:Device: 6h/6d Inode: 111267 Links: 1 2021-02-10T04:23:07.572 INFO:teuthology.orchestra.run.smithi085.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T04:23:07.572 INFO:teuthology.orchestra.run.smithi085.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T04:23:07.572 INFO:teuthology.orchestra.run.smithi085.stdout:Access: 2021-02-10 04:22:38.854678418 +0000 2021-02-10T04:23:07.572 INFO:teuthology.orchestra.run.smithi085.stdout:Modify: 2021-02-10 04:15:59.992799943 +0000 2021-02-10T04:23:07.572 INFO:teuthology.orchestra.run.smithi085.stdout:Change: 2021-02-10 04:15:59.992799943 +0000 2021-02-10T04:23:07.572 INFO:teuthology.orchestra.run.smithi085.stdout: Birth: - 2021-02-10T04:23:07.581 INFO:teuthology.orchestra.run.smithi085.stderr:1+0 records in 2021-02-10T04:23:07.582 INFO:teuthology.orchestra.run.smithi085.stderr:1+0 records out 2021-02-10T04:23:07.582 INFO:teuthology.orchestra.run.smithi085.stderr:512 bytes copied, 0.000239523 s, 2.1 MB/s 2021-02-10T04:23:07.584 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2021-02-10T04:23:07.585 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/scratch_devs of=/dev/stdout 2021-02-10T04:23:07.600 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-02-10T04:23:07.600 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-02-10T04:23:07.654 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-02-10T04:23:07.654 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T04:23:07.654 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 6h/6d Inode: 107419 Links: 1 2021-02-10T04:23:07.654 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T04:23:07.654 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T04:23:07.654 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2021-02-10 04:22:52.467761472 +0000 2021-02-10T04:23:07.654 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2021-02-10 04:15:59.050842651 +0000 2021-02-10T04:23:07.654 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2021-02-10 04:15:59.050842651 +0000 2021-02-10T04:23:07.655 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: - 2021-02-10T04:23:07.664 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2021-02-10T04:23:07.664 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2021-02-10T04:23:07.665 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000230648 s, 2.2 MB/s 2021-02-10T04:23:07.668 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-02-10T04:23:07.723 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-02-10T04:23:07.723 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T04:23:07.723 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 6h/6d Inode: 110966 Links: 1 2021-02-10T04:23:07.723 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T04:23:07.723 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T04:23:07.723 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2021-02-10 04:22:52.467761472 +0000 2021-02-10T04:23:07.724 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2021-02-10 04:15:59.383836782 +0000 2021-02-10T04:23:07.724 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2021-02-10 04:15:59.383836782 +0000 2021-02-10T04:23:07.724 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: - 2021-02-10T04:23:07.734 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2021-02-10T04:23:07.735 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2021-02-10T04:23:07.735 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000223541 s, 2.3 MB/s 2021-02-10T04:23:07.737 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-02-10T04:23:07.792 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-02-10T04:23:07.793 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T04:23:07.793 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 6h/6d Inode: 107481 Links: 1 2021-02-10T04:23:07.793 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T04:23:07.793 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T04:23:07.793 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2021-02-10 04:22:52.468761454 +0000 2021-02-10T04:23:07.793 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2021-02-10 04:15:59.694831301 +0000 2021-02-10T04:23:07.793 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2021-02-10 04:15:59.694831301 +0000 2021-02-10T04:23:07.793 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: - 2021-02-10T04:23:07.804 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2021-02-10T04:23:07.804 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2021-02-10T04:23:07.804 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000195588 s, 2.6 MB/s 2021-02-10T04:23:07.807 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-02-10T04:23:07.861 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-02-10T04:23:07.861 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T04:23:07.862 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 6h/6d Inode: 112730 Links: 1 2021-02-10T04:23:07.862 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T04:23:07.862 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T04:23:07.862 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2021-02-10 04:22:52.468761454 +0000 2021-02-10T04:23:07.862 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2021-02-10 04:16:00.019825574 +0000 2021-02-10T04:23:07.862 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2021-02-10 04:16:00.019825574 +0000 2021-02-10T04:23:07.862 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: - 2021-02-10T04:23:07.871 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2021-02-10T04:23:07.871 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2021-02-10T04:23:07.871 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000177493 s, 2.9 MB/s 2021-02-10T04:23:07.874 INFO:tasks.cephadm:Deploying osd.0 on smithi085 with /dev/vg_nvme/lv_4... 2021-02-10T04:23:07.874 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2021-02-10T04:23:07.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:07 smithi164 conmon[29288]: audit 2021-02-10T04:23:06.630749+0000 mgr.y (mgr.14140) 68 : audit [DBG] from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi085=y;smithi164=x", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:23:07.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:07 smithi164 conmon[29288]: cephadm 2021-02-10T04:23:06.631504+0000 mgr.y (mgr.14140) 69 : cephadm [INF] Saving service mgr spec with placement smithi085=y;smithi164=x;count:3 2021-02-10T04:23:07.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:07 smithi164 conmon[29288]: audit 2021-02-10T04:23:06.634033+0000 mon.a (mon.0) 224 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:07.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:07 smithi164 conmon[29288]: audit 2021-02-10T04:23:06.815199+0000 mon.a (mon.0) 225 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:07.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:07 smithi164 conmon[29288]: audit 2021-02-10T04:23:06.856836+0000 mon.a (mon.0) 226 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:07.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:07 smithi164 conmon[29288]: audit 2021-02-10T04:23:06.861854+0000 mon.a (mon.0) 227 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:07.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:07 smithi164 conmon[29288]: audit 2021-02-10T04:23:06.862683+0000 mon.a (mon.0) 228 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-02-10T04:23:07.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:07 smithi164 conmon[29288]: audit 2021-02-10T04:23:07.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:07 smithi164 conmon[29288]: 2021-02-10T04:23:06.867604+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2021-02-10T04:23:07.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:07 smithi164 conmon[29288]: audit 2021-02-10T04:23:06.868127+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-02-10T04:23:07.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:07 smithi164 conmon[29288]: audit 2021-02-10T04:23:06.868802+0000 mon.a (mon.0) 231 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:07.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:07 smithi164 conmon[29288]: cephadm 2021-02-10T04:23:06.869405+0000 mgr.y (mgr.14140) 70 : cephadm [INF] Deploying daemon mgr.x on smithi164 2021-02-10T04:23:07.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:07 smithi164 conmon[29288]: audit 2021-02-10T04:23:06.869736+0000 mon.a (mon.0) 232 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mgr.x", "key": "container_image"}]: dispatch 2021-02-10T04:23:07.895 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[29319]: audit 2021-02-10T04:23:06.630749+0000 mgr.y ( 2021-02-10T04:23:07.895 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[29319]: mgr.14140) 68 : audit [DBG] from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi085=y;smithi164=x", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:23:07.895 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[29319]: cephadm 2021-02-10T04:23:06.631504+0000 mgr.y (mgr.14140) 69 : cephadm [INF] Saving service mgr spec with placement smithi085=y;smithi164=x;count:3 2021-02-10T04:23:07.895 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[29319]: audit 2021-02-10T04:23:06.634033+0000 mon.a (mon.0) 224 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:07.896 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[29319]: audit 2021-02-10T04:23:06.815199+0000 mon.a (mon.0) 225 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:07.896 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[29319]: audit 2021-02-10T04:23:06.856836+0000 mon.a (mon.0) 226 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:07.896 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[29319]: audit 2021-02-10T04:23:06.861854+0000 mon.a (mon.0) 227 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:07.896 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[29319]: audit 2021-02-10T04:23:06.862683+0000 mon.a (mon.0) 228 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-02-10T04:23:07.896 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[29319]: audit 2021-02-10T04:23:06.867604+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2021-02-10T04:23:07.896 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[29319]: audit 2021-02-10T04:23:06 2021-02-10T04:23:07.896 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[29319]: .868127+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-02-10T04:23:07.896 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[29319]: audit 2021-02-10T04:23:06.868802+0000 mon.a (mon.0) 231 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:07.896 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[29319]: 2021-02-10T04:23:07.896 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[29319]: cephadm 2021 2021-02-10T04:23:07.896 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[29319]: -02-10T04:23:06.869405+0000 mgr.y (mgr. 2021-02-10T04:23:07.896 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[29319]: 14140) 70 : cephadm [INF] Deploying daemon mgr.x on smithi164 2021-02-10T04:23:07.896 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[29319]: audit 2021-02-10T04:23:07.897 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:23:07.897 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[29319]: 04:23:06.869736+0000 mon.a (mon. 2021-02-10T04:23:07.897 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[29319]: 0) 232 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mgr.x", "key": "container_image"}]: dispatch 2021-02-10T04:23:07.897 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[35961]: audit 2021-02-10T04:23:06.630749+0000 2021-02-10T04:23:07.897 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[35961]: mgr.y (mgr.14140) 68 : audit [DBG] from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi085=y;smithi164=x", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:23:07.897 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[35961]: cephadm 2021-02-10T04:23:06.631504+0000 2021-02-10T04:23:07.897 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[35961]: mgr.y (mgr.14140) 69 : cephadm [INF] Saving service mgr spec with placement smithi085=y;smithi164=x;count:3 2021-02-10T04:23:07.897 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[35961]: audit 2021-02-10T04:23:06.634033 2021-02-10T04:23:07.897 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[35961]: +0000 mon.a (mon.0) 224 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:07.897 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[35961]: audit 2021-02-10T04:23:06. 2021-02-10T04:23:07.897 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[35961]: 815199+0000 mon.a (mon.0) 225 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:07.897 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[35961]: audit 2021-02-10T04:23:06.856836+0000 2021-02-10T04:23:07.897 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[35961]: mon.a (mon.0) 226 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:07.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[35961]: audit 2021-02-10T04:23:06.861854+0000 2021-02-10T04:23:07.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[35961]: mon.a (mon.0) 227 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:07.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[35961]: audit 2021-02-10T04:23:06.862683+0000 2021-02-10T04:23:07.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[35961]: mon.a (mon.0) 228 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-02-10T04:23:07.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[35961]: audit 2021-02-10T04:23:06.867604+0000 mon.a (mon 2021-02-10T04:23:07.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[35961]: .0) 229 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2021-02-10T04:23:07.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[35961]: audit 2021-02-10T04:23:07.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[35961]: 2021-02-10T04:23:06.868127+0000 mon.a (mon.0) 230 : audit [DBG] 2021-02-10T04:23:07.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-02-10T04:23:07.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[35961]: audit 2021-02-10T04:23:07.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[35961]: 2021-02-10T04:23:06.868802+0000 mon.a 2021-02-10T04:23:07.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[35961]: (mon.0) 231 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:07.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[35961]: 2021-02-10T04:23:07.899 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[35961]: cephadm 2021-02-10T04:23:07.899 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[35961]: 2021-02-10T04:23:06.869405+0000 mgr.y 2021-02-10T04:23:07.899 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[35961]: (mgr.14140) 70 : cephadm [INF] Deploying daemon mgr.x on smithi164 2021-02-10T04:23:07.899 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[35961]: 2021-02-10T04:23:07.899 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[35961]: audit 2021-02-10T04:23:07.899 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[35961]: 2021-02-10T04:23:06.869736+0000 mon.a (mon.0) 232 : audit 2021-02-10T04:23:07.899 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:07 smithi085 conmon[35961]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mgr.x", "key": "container_image"}]: dispatch 2021-02-10T04:23:08.643 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Feb 10 04:23:08 smithi164 systemd[1]: Starting Ceph mgr.x for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T04:23:09.138 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:08 smithi164 conmon[29288]: cluster 2021-02-10T04:23:07.917735+0000 mgr.y (mgr.14140) 2021-02-10T04:23:09.138 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:08 smithi164 conmon[29288]: 71 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:09.144 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:08 smithi085 conmon[29319]: cluster 2021-02-10T04:23:07.917735+0000 mgr.y (mgr.14140) 71 : cluster 2021-02-10T04:23:09.145 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:08 smithi085 conmon[29319]: [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:09.145 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:08 smithi085 conmon[35961]: cluster 2021-02-10T04:23:09.145 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:08 smithi085 conmon[35961]: 2021-02-10T04:23:07. 2021-02-10T04:23:09.145 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:08 smithi085 conmon[35961]: 917735+0000 mgr.y (mgr.14140) 71 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:09.393 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Feb 10 04:23:09 smithi164 bash[30788]: d2e6b5b121c998c03d5316f561dc6f4337773ef16b2ef8bd0673bc4fffb5a47e 2021-02-10T04:23:09.393 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Feb 10 04:23:09 smithi164 conmon[30902]: debug 2021-02-10T04:23:09.139+0000 7f931b7fc700 1 -- 172.21.15.164:0/1811481768 <== mon.2 v2:172.21.15.164:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555af35984e0 con 0x555af356f000 2021-02-10T04:23:09.393 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Feb 10 04:23:09 smithi164 systemd[1]: Started Ceph mgr.x for 42224fce-6b57-11eb-8feb-001a4aab830c. 2021-02-10T04:23:09.807 INFO:teuthology.orchestra.run.smithi085.stderr:--> Zapping: /dev/vg_nvme/lv_4 2021-02-10T04:23:09.807 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2021-02-10T04:23:09.807 INFO:teuthology.orchestra.run.smithi085.stderr: stderr: 10+0 records in 2021-02-10T04:23:09.807 INFO:teuthology.orchestra.run.smithi085.stderr:10+0 records out 2021-02-10T04:23:09.807 INFO:teuthology.orchestra.run.smithi085.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0137812 s, 761 MB/s 2021-02-10T04:23:09.807 INFO:teuthology.orchestra.run.smithi085.stderr:--> Zapping successful for: 2021-02-10T04:23:10.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:10 smithi085 conmon[29319]: audit 2021-02-10T04:23:09.212390+0000 mon.a (mon.0) 233 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:10 smithi085 conmon[29319]: audit 2021-02-10T04:23:09.213358+0000 mon.a (mon.0) 234 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:10 smithi085 conmon[29319]: audit 2021-02-10T04:23:09.213592+0000 mon.a (mon.0) 235 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:10 smithi085 conmon[29319]: cluster 2021-02-10T04:23:09.917956+0000 mgr.y (mgr.14140) 72 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:10 smithi085 conmon[29319]: audit 2021-02-10T04:23:10.115606+0000 mon.a (mon.0) 236 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:10.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:10 smithi085 conmon[35961]: audit 2021-02-10T04:23:09.212390+0000 mon.a (mon.0) 233 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:10.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:10 smithi085 conmon[35961]: audit 2021-02-10T04:23:09.213358+0000 mon.a (mon.0) 234 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:10.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:10 smithi085 conmon[35961]: audit 2021-02-10T04:23:09.213592+0000 mon.a (mon.0) 235 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:10.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:10 smithi085 conmon[35961]: cluster 2021-02-10T04:23:09.917956+0000 mgr.y (mgr.14140) 72 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:10.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:10 smithi085 conmon[35961]: audit 2021-02-10T04:23:10.115606+0000 mon.a (mon. 2021-02-10T04:23:10.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:10 smithi085 conmon[35961]: 0) 236 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:10.532 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph orch daemon add osd smithi085:vg_nvme/lv_4 2021-02-10T04:23:10.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:10 smithi164 conmon[29288]: audit 2021-02-10T04:23:09.212390+0000 mon.a (mon.0) 233 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:10.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:10 smithi164 conmon[29288]: audit 2021-02-10T04:23:09.213358+0000 mon.a (mon.0) 234 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:10.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:10 smithi164 conmon[29288]: audit 2021-02-10T04:23:09.213592+0000 mon.a (mon.0) 235 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:10.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:10 smithi164 conmon[29288]: cluster 2021-02-10T04:23:09.917956+0000 mgr.y (mgr.14140) 72 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:10.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:10 smithi164 conmon[29288]: audit 2021-02-10T04:23:10.115606+0000 mon.a (mon.0) 236 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:11 smithi164 conmon[29288]: audit 2021-02-10T04:23:10.580666+0000 mon.a (mon.0) 237 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:11 smithi164 conmon[29288]: audit 2021-02-10T04:23:10.583716+0000 mon.a (mon.0) 238 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:11 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:23:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:11 smithi164 conmon[29288]: -10T04:23:10.585017 2021-02-10T04:23:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:11 smithi164 conmon[29288]: +0000 mon.a (mon.0) 239 2021-02-10T04:23:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:11 smithi164 conmon[29288]: : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:11.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:11 smithi085 conmon[29319]: audit 2021-02-10T04:23:10.580666+0000 mon.a (mon.0) 237 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:11.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:11 smithi085 conmon[29319]: audit 2021-02-10T04:23:10.583716+0000 mon.a (mon.0) 238 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:11.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:11 smithi085 conmon[29319]: audit 2021-02-10T04:23:10.585017+0000 mon.a (mon.0) 239 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:11.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:11 smithi085 conmon[35961]: audit 2021-02-10T04:23:10.580666+0000 mon.a (mon.0) 237 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:11.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:11 smithi085 conmon[35961]: audit 2021-02-10T04:23:10.583716+0000 mon.a (mon.0) 238 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:11.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:11 smithi085 conmon[35961]: audit 2021-02-10T04:23:10.585017+0000 mon.a (mon.0) 239 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:12.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:12 smithi164 conmon[29288]: cluster 2021-02-10T04:23:11.918381+0000 mgr.y (mgr.14140) 73 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:12.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:12 smithi164 conmon[29288]: audit 2021-02-10T04:23:12.098237+0000 mon.a (mon.0) 240 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:12.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:12 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:23:12.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:12 smithi164 conmon[29288]: -10T04:23:12.101397 2021-02-10T04:23:12.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:12 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2021-02-10T04:23:12.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:12 smithi164 conmon[29288]: 241 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:12.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:12 smithi164 conmon[29288]: audit 2021-02-10T04:23:12.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:12 smithi164 conmon[29288]: 2021-02-10T04:23 2021-02-10T04:23:12.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:12 smithi164 conmon[29288]: :12.576438+0000 mon.a (mon 2021-02-10T04:23:12.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:12 smithi164 conmon[29288]: .0) 242 : audit [DBG] 2021-02-10T04:23:12.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:12 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-02-10T04:23:12.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:12 smithi164 conmon[29288]: audit 2021-02-10T04:23:12.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:12 smithi164 conmon[29288]: 2021-02-10T04:23 2021-02-10T04:23:12.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:12 smithi164 conmon[29288]: :12.577774+0000 mon.a ( 2021-02-10T04:23:12.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:12 smithi164 conmon[29288]: mon.0) 243 : audit [INF] 2021-02-10T04:23:12.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:12 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-02-10T04:23:12.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:12 smithi164 conmon[29288]: audit 2021-02-10T04:23:12.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:12 smithi164 conmon[29288]: 2021-02-10T04:23 2021-02-10T04:23:12.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:12 smithi164 conmon[29288]: :12.578261+0000 mon.a ( 2021-02-10T04:23:12.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:12 smithi164 conmon[29288]: mon.0) 244 : audit 2021-02-10T04:23:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:12 smithi164 conmon[29288]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:12 smithi164 conmon[29288]: audit 2021-02-10T04:23:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:12 smithi164 conmon[29288]: 2021-02-10T04:23 2021-02-10T04:23:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:12 smithi164 conmon[29288]: :12.578838+0000 mon.a ( 2021-02-10T04:23:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:12 smithi164 conmon[29288]: mon.0) 245 : audit 2021-02-10T04:23:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:12 smithi164 conmon[29288]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:23:12.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:12 smithi085 conmon[29319]: cluster 2021-02-10T04:23:11.918381+0000 mgr.y (mgr. 2021-02-10T04:23:12.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:12 smithi085 conmon[29319]: 14140) 73 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:12.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:12 smithi085 conmon[29319]: audit 2021-02-10T04:23:12.098237+0000 mon.a (mon.0) 240 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:12.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:12 smithi085 conmon[29319]: audit 2021-02-10T04:23:12.101397+0000 mon.a (mon.0) 241 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:12.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:12 smithi085 conmon[29319]: audit 2021-02-10T04:23:12.576438+0000 mon.a (mon.0) 242 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-02-10T04:23:12.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:12 smithi085 conmon[29319]: audit 2021-02-10T04:23:12.577774+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-02-10T04:23:12.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:12 smithi085 conmon[29319]: audit 2021-02-10T04:23:12.578261+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:12.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:12 smithi085 conmon[29319]: audit 2021-02-10T04:23:12.578838+0000 mon.a (mon.0) 245 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:23:12.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:12 smithi085 conmon[35961]: cluster 2021-02-10T04:23:11.918381+0000 2021-02-10T04:23:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:12 smithi085 conmon[35961]: mgr.y (mgr.14140) 73 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:12 smithi085 conmon[35961]: audit 2021-02-10T04:23:12.098237+0000 mon.a (mon.0) 240 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:12 smithi085 conmon[35961]: audit 2021-02-10T04:23:12.101397+0000 mon.a (mon.0) 241 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:12 smithi085 conmon[35961]: audit 2021-02-10T04:23:12.576438+0000 mon.a (mon.0) 242 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-02-10T04:23:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:12 smithi085 conmon[35961]: audit 2021-02-10T04:23:12.577774+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-02-10T04:23:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:12 smithi085 conmon[35961]: audit 2021-02-10T04:23:12.578261+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:12 smithi085 conmon[35961]: audit 2021-02-10T04:23:12.578838+0000 mon.a (mon.0) 245 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:23:14.142 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:13 smithi164 conmon[29288]: audit 2021-02-10T04:23:12.575506+0000 mgr.y (mgr.14140) 74 : audit [DBG] from='client.24124 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi085:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:23:14.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:13 smithi164 conmon[29288]: cephadm 2021-02-10T04:23:12.577548+0000 mgr.y (mgr.14140) 75 : cephadm [INF] Applying service osd.None on host smithi085... 2021-02-10T04:23:14.157 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:13 smithi085 conmon[29319]: audit 2021-02-10T04:23:12.575506+0000 mgr.y (mgr.14140) 74 2021-02-10T04:23:14.157 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:13 smithi085 conmon[29319]: : audit [DBG] from='client.24124 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi085:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:23:14.157 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:13 smithi085 conmon[29319]: cephadm 2021-02-10T04:23:12.577548+0000 mgr.y (mgr.14140) 75 : cephadm [INF] Applying service osd.None on host smithi085... 2021-02-10T04:23:14.157 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:13 smithi085 conmon[35961]: audit 2021-02-10T04:23:12.575506+0000 mgr.y 2021-02-10T04:23:14.157 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:13 smithi085 conmon[35961]: (mgr.14140) 74 : audit [DBG] from='client.24124 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi085:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:23:14.157 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:13 smithi085 conmon[35961]: cephadm 2021-02-10T04: 2021-02-10T04:23:14.157 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:13 smithi085 conmon[35961]: 23:12.577548+0000 mgr.y (mgr.14140) 75 : cephadm [INF] Applying service osd.None on host smithi085... 2021-02-10T04:23:15.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:14 smithi164 conmon[29288]: cluster 2021-02-10T04:23:13.918705+0000 mgr.y (mgr.14140) 76 : cluster 2021-02-10T04:23:15.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:14 smithi164 conmon[29288]: [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:15.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:14 smithi164 conmon[29288]: cluster 2021-02-10T04:23:13.982957+0000 mon.a (mon.0) 246 : cluster [DBG] Standby manager daemon x started 2021-02-10T04:23:15.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:14 smithi164 conmon[29288]: audit 2021-02-10T04:23:13.983762+0000 mon.b (mon.2) 3 : audit [DBG] from='mgr.? 172.21.15.164:0/7' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-02-10T04:23:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:14 smithi164 conmon[29288]: audit 2021-02-10T04:23:13.984738+0000 mon.b (mon.2) 4 : audit [DBG] from='mgr.? 172.21.15.164:0/7' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-02-10T04:23:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:14 smithi164 conmon[29288]: audit 2021-02-10T04:23:14.753040+0000 mon.c (mon.1) 4 : audit [INF] from='client.? 172.21.15.85:0/4029841392' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "45d0bb5f-35fa-4596-964f-75abcee53bee"}]: dispatch 2021-02-10T04:23:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:14 smithi164 conmon[29288]: audit 2021-02-10T04:23:14.753395+0000 mon.a (mon.0) 247 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "45d0bb5f-35fa-4596-964f-75abcee53bee"}]: dispatch 2021-02-10T04:23:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:14 smithi164 conmon[29288]: audit 2021-02-10T04:23:14.756526+0000 mon.a (mon.0) 248 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "45d0bb5f-35fa-4596-964f-75abcee53bee"}]': finished 2021-02-10T04:23:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:14 smithi164 conmon[29288]: cluster 2021-02-10T04:23:14.756607+0000 mon.a (mon.0) 249 : cluster [DBG] osdmap e5: 1 total, 0 up, 0 in 2021-02-10T04:23:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:14 smithi164 conmon[29288]: audit 2021-02-10T04:23:14.756703+0000 mon.a (mon.0) 250 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-02-10T04:23:15.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:14 smithi085 conmon[29319]: cluster 2021-02-10T04:23:13.918705+0000 mgr.y (mgr.14140) 76 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:15.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:14 smithi085 conmon[29319]: cluster 2021-02-10T04:23:13.982957+0000 mon.a (mon.0) 246 : cluster [DBG] Standby manager daemon x started 2021-02-10T04:23:15.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:14 smithi085 conmon[29319]: audit 2021-02-10T04:23:13.983762+0000 mon.b (mon.2) 3 : audit [DBG] from='mgr.? 172.21.15.164:0/7' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-02-10T04:23:15.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:14 smithi085 conmon[29319]: audit 2021-02-10T04:23:13.984738+0000 mon.b (mon.2) 4 : audit [DBG] from='mgr.? 172.21.15.164:0/7' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-02-10T04:23:15.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:14 smithi085 conmon[29319]: audit 2021-02-10T04:23:14.753040+0000 mon.c (mon.1) 4 : audit [INF] from='client.? 172.21.15.85:0/4029841392' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "45d0bb5f-35fa-4596-964f-75abcee53bee"}]: dispatch 2021-02-10T04:23:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:14 smithi085 conmon[29319]: audit 2021-02-10T04:23:14.753395+0000 mon.a (mon.0) 247 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "45d0bb5f-35fa-4596-964f-75abcee53bee"}]: dispatch 2021-02-10T04:23:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:14 smithi085 conmon[29319]: audit 2021-02-10T04:23:14.756526+0000 mon.a (mon.0) 248 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "45d0bb5f-35fa-4596-964f-75abcee53bee"}]': finished 2021-02-10T04:23:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:14 smithi085 conmon[29319]: cluster 2021-02-10T04:23:14.756607+0000 mon.a (mon.0) 249 : cluster [DBG] osdmap e5: 1 total, 0 up, 0 in 2021-02-10T04:23:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:14 smithi085 conmon[29319]: audit 2021-02-10T04:23:14.756703+0000 mon.a (mon.0) 250 : audit [DBG] 2021-02-10T04:23:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:14 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-02-10T04:23:15.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:14 smithi085 conmon[35961]: cluster 2021-02-10T04:23:13.918705+0000 mgr.y (mgr.14140) 76 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:15.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:14 smithi085 conmon[35961]: cluster 2021-02-10T04:23:13.982957+0000 mon.a (mon.0) 246 : cluster [DBG] Standby manager daemon x started 2021-02-10T04:23:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:14 smithi085 conmon[35961]: audit 2021-02-10T04:23:13.983762+0000 mon.b (mon.2) 3 : audit [DBG] from='mgr.? 172.21.15.164:0/7' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-02-10T04:23:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:14 smithi085 conmon[35961]: audit 2021-02-10T04:23:13.984738+0000 mon.b (mon.2) 4 : audit [DBG] from='mgr.? 172.21.15.164:0/7' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-02-10T04:23:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:14 smithi085 conmon[35961]: audit 2021-02-10T04:23:14.753040+0000 mon.c (mon.1) 4 : audit [INF] from='client.? 172.21.15.85:0/4029841392' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "45d0bb5f-35fa-4596-964f-75abcee53bee"}]: dispatch 2021-02-10T04:23:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:14 smithi085 conmon[35961]: audit 2021-02-10T04:23:14.753395+0000 mon.a (mon.0) 247 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "45d0bb5f-35fa-4596-964f-75abcee53bee"}]: dispatch 2021-02-10T04:23:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:14 smithi085 conmon[35961]: audit 2021-02-10T04:23:14.756526+0000 mon.a (mon.0) 248 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "45d0bb5f-35fa-4596-964f-75abcee53bee"}]': finished 2021-02-10T04:23:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:14 smithi085 conmon[35961]: cluster 2021-02-10T04:23:14.756607+0000 mon.a (mon.0) 249 : cluster 2021-02-10T04:23:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:14 smithi085 conmon[35961]: [DBG] osdmap e5: 1 total, 0 up, 0 in 2021-02-10T04:23:15.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:14 smithi085 conmon[35961]: audit 2021-02-10T04:23:14.756703+0000 mon.a (mon.0) 250 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-02-10T04:23:16.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:15 smithi164 conmon[29288]: cluster 2021-02-10T04:23:14.889640+0000 mon.a (mon.0) 251 : cluster 2021-02-10T04:23:16.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:15 smithi164 conmon[29288]: [DBG] mgrmap e14: y(active, since 97s), standbys: x 2021-02-10T04:23:16.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:15 smithi164 conmon[29288]: audit 2021-02-10T04:23:14.889761+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2021-02-10T04:23:16.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:15 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:23:16.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:15 smithi164 conmon[29288]: T04:23:15.232983+0000 mon.c (mon.1) 5 : audit [DBG] from='client.? 172.21.15.85:0/4162487907' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-02-10T04:23:16.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:15 smithi085 conmon[29319]: cluster 2021-02-10T04:23:14.889640+0000 mon.a (mon.0) 251 : 2021-02-10T04:23:16.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:15 smithi085 conmon[29319]: cluster [DBG] mgrmap e14: y(active, since 97s), standbys: x 2021-02-10T04:23:16.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:15 smithi085 conmon[29319]: audit 2021-02-10T04:23:14.889761+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2021-02-10T04:23:16.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:15 smithi085 conmon[29319]: audit 2021-02-10T04:23:15.232983+0000 mon.c (mon.1) 5 : audit [DBG] from='client.? 172.21.15.85:0/4162487907' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-02-10T04:23:16.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:15 smithi085 conmon[35961]: cluster 2021-02-10T04:23:14.889640+0000 mon.a 2021-02-10T04:23:16.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:15 smithi085 conmon[35961]: (mon.0) 251 : cluster [DBG] mgrmap e14: y(active, since 97s), standbys: x 2021-02-10T04:23:16.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:15 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:23:16.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:15 smithi085 conmon[35961]: 10T04:23:14.889761+0000 mon.a (mon.0) 2021-02-10T04:23:16.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:15 smithi085 conmon[35961]: 252 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2021-02-10T04:23:16.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:15 smithi085 conmon[35961]: audit 2021-02-10T04:23: 2021-02-10T04:23:16.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:15 smithi085 conmon[35961]: 15.232983+0000 mon.c (mon.1) 5 2021-02-10T04:23:16.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:15 smithi085 conmon[35961]: : audit [DBG] from='client.? 172.21.15.85:0/4162487907' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-02-10T04:23:17.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:16 smithi164 conmon[29288]: cluster 2021-02-10T04:23:15.919046+0000 mgr.y (mgr.14140) 77 2021-02-10T04:23:17.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:16 smithi164 conmon[29288]: : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:17.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:16 smithi085 conmon[35961]: cluster 2021-02-10T04:23:15.919046+0000 mgr.y (mgr.14140) 77 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:17.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:16 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:23:17.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:16 smithi085 conmon[29319]: 04:23:15.919046+0000 mgr.y (mgr.14140) 77 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:19.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:18 smithi164 conmon[29288]: cluster 2021-02-10T04:23:17.919282+0000 mgr.y (mgr. 2021-02-10T04:23:19.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:18 smithi164 conmon[29288]: 14140) 78 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:19.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:18 smithi164 conmon[29288]: audit 2021-02-10T04:23:18.007129+0000 mon.a (mon.0) 253 : audit 2021-02-10T04:23:19.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:18 smithi164 conmon[29288]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:23:19.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:18 smithi085 conmon[29319]: cluster 2021-02-10T04:23:17. 2021-02-10T04:23:19.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:18 smithi085 conmon[29319]: 919282+0000 mgr.y (mgr.14140) 78 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:19.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:18 smithi085 conmon[29319]: audit 2021-02-10T04:23:18.007129+0000 mon.a (mon.0) 253 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:23:19.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:18 smithi085 conmon[35961]: cluster 2021-02-10T04:23:17 2021-02-10T04:23:19.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:18 smithi085 conmon[35961]: .919282+0000 mgr.y (mgr.14140) 78 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:19.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:18 smithi085 conmon[35961]: audit 2021 2021-02-10T04:23:19.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:18 smithi085 conmon[35961]: -02-10T04:23:18.007129 2021-02-10T04:23:19.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:18 smithi085 conmon[35961]: +0000 mon.a (mon.0) 253 2021-02-10T04:23:19.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:18 smithi085 conmon[35961]: : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:23:21.298 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:20 smithi085 conmon[29319]: cluster 2021-02-10T04:23 2021-02-10T04:23:21.298 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:20 smithi085 conmon[29319]: :19.919562+0000 mgr.y (mgr.14140) 79 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:21.298 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:20 smithi085 conmon[29319]: audit 2021-02-10T04:23:20.448873+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-02-10T04:23:21.299 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:20 smithi085 conmon[29319]: audit 2021-02-10T04:23:20.449513+0000 mon.a (mon.0) 255 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:21.299 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:20 smithi085 conmon[29319]: audit 2021-02-10T04:23:20.450326+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.0", "key": "container_image"}]: dispatch 2021-02-10T04:23:21.299 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:20 smithi085 conmon[35961]: cluster 2021-02-10T04:23: 2021-02-10T04:23:21.299 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:20 smithi085 conmon[35961]: 19.919562+0000 mgr.y (mgr.14140) 79 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:21.299 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:20 smithi085 conmon[35961]: audit 2021-02-10T04:23:20.448873+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-02-10T04:23:21.299 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:20 smithi085 conmon[35961]: audit 2021-02-10T04:23:20.449513+0000 mon.a (mon.0) 255 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:21.299 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:20 smithi085 conmon[35961]: audit 2021-02-10T04:23:20.450326+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.0", "key": "container_image"}]: dispatch 2021-02-10T04:23:21.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:20 smithi164 conmon[29288]: cluster 2021-02-10T04:23:19.919562+0000 mgr.y (mgr. 2021-02-10T04:23:21.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:20 smithi164 conmon[29288]: 14140) 79 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:21.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:20 smithi164 conmon[29288]: audit 2021-02-10T04:23:20.448873+0000 mon.a (mon.0) 254 : audit 2021-02-10T04:23:21.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:20 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-02-10T04:23:21.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:20 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:23:21.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:20 smithi164 conmon[29288]: -10T04:23:20.449513+0000 mon.a (mon.0) 255 : audit 2021-02-10T04:23:21.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:20 smithi164 conmon[29288]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:21.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:20 smithi164 conmon[29288]: audit 2021-02-10T04:23: 2021-02-10T04:23:21.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:20 smithi164 conmon[29288]: 20.450326+0000 mon.a (mon.0) 256 : audit 2021-02-10T04:23:21.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:20 smithi164 conmon[29288]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.0", "key": "container_image"}]: dispatch 2021-02-10T04:23:22.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:21 smithi085 conmon[29319]: cephadm 2021-02-10T04:23:20.449962+0000 mgr.y (mgr.14140) 2021-02-10T04:23:22.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:21 smithi085 conmon[29319]: 80 : cephadm [INF] Deploying daemon osd.0 on smithi085 2021-02-10T04:23:22.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:21 smithi085 conmon[35961]: cephadm 2021-02-10T04:23 2021-02-10T04:23:22.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:21 smithi085 conmon[35961]: :20.449962+0000 mgr.y (mgr.14140) 80 : cephadm [INF] Deploying daemon osd.0 on smithi085 2021-02-10T04:23:22.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:21 smithi164 conmon[29288]: cephadm 2021-02-10T04:23:20.449962+0000 mgr.y 2021-02-10T04:23:22.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:21 smithi164 conmon[29288]: (mgr.14140) 80 : cephadm [INF] Deploying daemon osd.0 on smithi085 2021-02-10T04:23:23.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:22 smithi164 conmon[29288]: cluster 2021-02-10T04:23:21.920226+0000 mgr.y (mgr.14140) 81 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:23.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:22 smithi085 conmon[29319]: cluster 2021-02-10T04:23:21.920226+0000 2021-02-10T04:23:23.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:22 smithi085 conmon[29319]: mgr.y (mgr.14140) 81 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:23.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:22 smithi085 conmon[35961]: cluster 2021-02-10T04:23:21.920226+0000 2021-02-10T04:23:23.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:22 smithi085 conmon[35961]: mgr.y (mgr.14140) 81 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:23.624 INFO:teuthology.orchestra.run.smithi085.stdout:Created osd(s) 0 on host 'smithi085' 2021-02-10T04:23:24.088 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:23 smithi085 conmon[29319]: audit 2021-02-10T04:23:23.614624+0000 mon.a (mon.0) 257 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:24.088 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:23 smithi085 conmon[29319]: audit 2021-02-10T04:23:23.617006+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:24.088 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:23 smithi085 conmon[29319]: audit 2021-02-10T04:23:23.622172+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:24.088 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:23 smithi085 conmon[35961]: audit 2021-02-10T04:23:23.614624+0000 mon.a (mon.0) 257 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:24.088 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:23 smithi085 conmon[35961]: audit 2021-02-10T04:23:23.617006+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:24.089 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:23 smithi085 conmon[35961]: audit 2021-02-10T04:23:23.622172+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:24.188 DEBUG:teuthology.orchestra.run.smithi085:osd.0> sudo journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.0.service 2021-02-10T04:23:24.190 INFO:tasks.cephadm:Deploying osd.1 on smithi085 with /dev/vg_nvme/lv_3... 2021-02-10T04:23:24.191 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2021-02-10T04:23:24.218 INFO:journalctl@ceph.osd.0.smithi085.stdout:-- Logs begin at Wed 2021-02-10 04:09:23 UTC. -- 2021-02-10T04:23:24.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:23 smithi164 conmon[29288]: audit 2021-02-10T04:23:23.614624+0000 mon.a (mon.0) 257 : audit 2021-02-10T04:23:24.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:23 smithi164 conmon[29288]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:24.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:23 smithi164 conmon[29288]: audit 2021-02-10T04:23:23.617006+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:24.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:23 smithi164 conmon[29288]: audit 2021-02-10T04:23:24.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:23 smithi164 conmon[29288]: 2021-02-10T04:23:23.622172+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:25.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:24 smithi164 conmon[29288]: cluster 2021-02-10T04:23:23.920469+0000 mgr.y (mgr.14140) 82 2021-02-10T04:23:25.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:24 smithi164 conmon[29288]: : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:25.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:24 smithi164 conmon[29288]: audit 2021-02-10T04:23:24.932288+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:25.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:24 smithi164 conmon[29288]: audit 2021-02-10T04:23:24.932954+0000 mon.a (mon.0) 261 : audit 2021-02-10T04:23:25.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:24 smithi164 conmon[29288]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:23:25.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:24 smithi085 conmon[29319]: cluster 2021-02-10T04:23:23.920469+0000 mgr.y (mgr.14140) 82 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:25.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:24 smithi085 conmon[29319]: audit 2021-02-10T04:23:24.932288+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:25.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:24 smithi085 conmon[29319]: audit 2021-02-10T04:23:24.932954+0000 mon.a (mon.0) 261 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:23:25.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:24 smithi085 conmon[35961]: cluster 2021-02-10T04:23:23.920469+0000 mgr.y (mgr.14140) 82 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:25.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:24 smithi085 conmon[35961]: audit 2021-02-10T04:23:24.932288+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:25.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:24 smithi085 conmon[35961]: audit 2021-02-10T04:23:24.932954+0000 mon.a (mon.0) 261 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:23:26.525 INFO:teuthology.orchestra.run.smithi085.stderr:--> Zapping: /dev/vg_nvme/lv_3 2021-02-10T04:23:26.525 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2021-02-10T04:23:26.526 INFO:teuthology.orchestra.run.smithi085.stderr: stderr: 10+0 records in 2021-02-10T04:23:26.526 INFO:teuthology.orchestra.run.smithi085.stderr:10+0 records out 2021-02-10T04:23:26.526 INFO:teuthology.orchestra.run.smithi085.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122281 s, 858 MB/s 2021-02-10T04:23:26.526 INFO:teuthology.orchestra.run.smithi085.stderr:--> Zapping successful for: 2021-02-10T04:23:26.825 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:23:26 smithi085 conmon[41540]: debug 2021-02-10T04:23:26.796+0000 7fb970eeb200 -1 osd.0 0 log_to_monitors {default=true} 2021-02-10T04:23:27.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:26 smithi085 conmon[29319]: cluster 2021-02-10T04:23:25.920737+0000 mgr.y 2021-02-10T04:23:27.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:26 smithi085 conmon[29319]: (mgr.14140) 83 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:27.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:26 smithi085 conmon[29319]: audit 2021-02-10T04:23:26.800976+0000 mon.a (mon. 2021-02-10T04:23:27.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:26 smithi085 conmon[29319]: 0) 262 : audit [INF] from='osd.0 v2:172.21.15.85:6801/7' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-02-10T04:23:27.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:26 smithi085 conmon[35961]: cluster 2021-02-10T04:23:25 2021-02-10T04:23:27.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:26 smithi085 conmon[35961]: .920737+0000 mgr.y (mgr.14140) 83 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:27.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:26 smithi085 conmon[35961]: audit 2021-02-10T04:23:26 2021-02-10T04:23:27.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:26 smithi085 conmon[35961]: .800976+0000 mon.a (mon.0) 262 : audit 2021-02-10T04:23:27.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:26 smithi085 conmon[35961]: [INF] from='osd.0 v2:172.21.15.85:6801/7' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-02-10T04:23:27.213 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph orch daemon add osd smithi085:vg_nvme/lv_3 2021-02-10T04:23:27.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:26 smithi164 conmon[29288]: cluster 2021-02-10T04:23:25.920737+0000 mgr.y (mgr.14140) 83 : cluster 2021-02-10T04:23:27.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:26 smithi164 conmon[29288]: [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:27.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:26 smithi164 conmon[29288]: audit 2021-02-10T04:23:26.800976+0000 mon.a (mon.0) 262 : audit [INF] from='osd.0 v2:172.21.15.85:6801/7' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-02-10T04:23:27.997 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:27 smithi085 conmon[29319]: audit 2021-02-10T04:23:26.984940+0000 mon.a (mon.0) 263 : audit [INF] from='osd.0 v2:172.21.15.85:6801/7' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-02-10T04:23:27.997 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:27 smithi085 conmon[29319]: cluster 2021-02-10T04:23:26.984990+0000 mon.a (mon.0) 264 : cluster [DBG] osdmap e6: 1 total, 0 up, 0 in 2021-02-10T04:23:27.998 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:27 smithi085 conmon[29319]: audit 2021-02-10T04:23:26.985260+0000 mon.a (mon.0) 265 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-02-10T04:23:27.998 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:27 smithi085 conmon[29319]: audit 2021-02-10T04:23:26.985607+0000 mon.a (mon.0) 266 : audit [INF] from='osd.0 v2:172.21.15.85:6801/7' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:23:27.998 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:27 smithi085 conmon[35961]: audit 2021-02-10T04:23:27.998 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:27 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:23:27.998 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:27 smithi085 conmon[35961]: 23:26.984940+0000 mon.a (mon.0) 263 : audit [INF] from='osd.0 v2:172.21.15.85:6801/7' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-02-10T04:23:27.999 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:27 smithi085 conmon[35961]: cluster 2021-02-10T04:23:26.984990+0000 2021-02-10T04:23:27.999 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:27 smithi085 conmon[35961]: mon.a (mon.0) 264 : cluster [DBG] osdmap e6: 1 total, 0 up, 0 in 2021-02-10T04:23:27.999 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:27 smithi085 conmon[35961]: audit 2021-02-10T04:23:27.999 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:27 smithi085 conmon[35961]: 2021-02-10T04:23:27.999 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:27 smithi085 conmon[35961]: 2021 2021-02-10T04:23:28.000 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:27 smithi085 conmon[35961]: - 2021-02-10T04:23:28.000 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:27 smithi085 conmon[35961]: 02 2021-02-10T04:23:28.000 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:27 smithi085 conmon[35961]: -10T04:23:26 2021-02-10T04:23:28.000 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:27 smithi085 conmon[35961]: .985260+0000 mon.a (mon.0) 265 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-02-10T04:23:28.000 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:27 smithi085 conmon[35961]: audit 2021-02-10T04:23:26.985607+0000 mon.a (mon.0) 266 : audit [INF] from='osd.0 v2:172.21.15.85:6801/7' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:23:28.001 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:23:27 smithi085 conmon[41540]: debug 2021-02-10T04:23:27.994+0000 7fb96745c700 -1 osd.0 0 waiting for initial osdmap 2021-02-10T04:23:28.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:27 smithi164 conmon[29288]: audit 2021-02-10T04:23:26.984940+0000 mon.a (mon.0) 2021-02-10T04:23:31.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:27 smithi164 conmon[29288]: 263 : audit [INF] from='osd.0 v2:172.21.15.85:6801/7' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-02-10T04:23:31.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:27 smithi164 conmon[29288]: cluster 2021-02-10T04:23:26.984990+0000 mon.a (mon.0) 264 : cluster [DBG] osdmap e6: 1 total, 0 up, 0 in 2021-02-10T04:23:31.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:27 smithi164 conmon[29288]: audit 2021-02-10T04:23:26.985260+0000 mon.a (mon.0) 265 2021-02-10T04:23:31.962 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:27 smithi164 conmon[29288]: : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-02-10T04:23:31.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:27 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:23:31.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:27 smithi164 conmon[29288]: :23:26.985607+0000 mon.a (mon.0) 266 2021-02-10T04:23:31.963 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:27 smithi164 conmon[29288]: : audit [INF] from='osd.0 v2:172.21.15.85:6801/7' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:23:31.964 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:29 smithi085 conmon[35961]: cluster 2021-02-10T04:23:27.920940+0000 mgr.y ( 2021-02-10T04:23:31.964 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:29 smithi085 conmon[35961]: mgr.14140) 84 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:31.965 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:29 smithi085 conmon[35961]: audit 2021-02-10T04:23:27.992985+0000 mon.a (mon.0) 267 : audit [INF] from='osd.0 v2:172.21.15.85:6801/7' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]': finished 2021-02-10T04:23:31.965 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:29 smithi085 conmon[35961]: cluster 2021-02-10T04:23:27.993050+0000 mon.a (mon.0) 268 : cluster [DBG] osdmap e7: 1 total, 0 up, 0 in 2021-02-10T04:23:31.965 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:29 smithi085 conmon[35961]: audit 2021-02-10T04:23:27.993684+0000 mon.a (mon.0) 269 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-02-10T04:23:31.965 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:29 smithi085 conmon[35961]: audit 2021-02-10T04:23:27.997411+0000 mon.a (mon.0) 270 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-02-10T04:23:31.965 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:30 smithi085 conmon[35961]: cluster 2021-02-10T04:23:27.755677+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:23:31.965 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:30 smithi085 conmon[35961]: cluster 2021-02-10T04:23:27.755775+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:23:31.965 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:30 smithi085 conmon[35961]: cluster 2021-02-10T04:23:28.998730+0000 mon.a (mon.0) 271 : cluster [INF] osd.0 v2:172.21.15.85:6801/7 boot 2021-02-10T04:23:31.965 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:30 smithi085 conmon[35961]: cluster 2021-02-10T04:23:28.998760+0000 mon.a (mon.0) 272 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2021-02-10T04:23:31.965 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:30 smithi085 conmon[35961]: audit 2021-02-10T04:23:31.965 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:30 smithi085 conmon[35961]: 2021-02-10T04:23:28.999069+0000 mon.a (mon.0) 273 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-02-10T04:23:31.965 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:30 smithi085 conmon[35961]: audit 2021-02-10T04:23:29.080004+0000 mon.a (mon.0) 274 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:31.965 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:30 smithi085 conmon[35961]: audit 2021-02-10T04:23:29.429670+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:31.965 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:30 smithi085 conmon[35961]: audit 2021-02-10T04:23:29.437496+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:31.966 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:30 smithi085 conmon[35961]: audit 2021-02-10T04:23:29.439095+0000 mon.a (mon.0) 277 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:31.966 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:30 smithi085 conmon[35961]: audit 2021-02-10T04:23:29.615295+0000 mon.a (mon.0) 278 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-02-10T04:23:31.966 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:30 smithi085 conmon[35961]: audit 2021-02-10T04:23:29.616359+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-02-10T04:23:31.966 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:30 smithi085 conmon[35961]: audit 2021-02-10T04:23:29.616838+0000 mon.a (mon.0) 280 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:31.966 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:30 smithi085 conmon[35961]: audit 2021-02-10T04:23:29.617409+0000 mon.a (mon.0) 281 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:23:31.966 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:31 smithi085 conmon[35961]: cephadm 2021-02-10T04:23:29.425635+0000 mgr.y (mgr 2021-02-10T04:23:31.966 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:31 smithi085 conmon[35961]: .14140) 85 : cephadm [INF] Detected new or changed devices on smithi085 2021-02-10T04:23:31.966 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:31 smithi085 conmon[35961]: audit 2021-02-10T04:23:29.614454+0000 mgr.y (mgr.14140) 86 : audit [DBG] from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi085:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:23:31.966 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:31 smithi085 conmon[35961]: cephadm 2021-02-10T04:23:29.616185+0000 mgr.y (mgr.14140) 87 : cephadm [INF] Applying service osd.None on host smithi085... 2021-02-10T04:23:31.966 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:31 smithi085 conmon[35961]: cluster 2021-02-10T04:23:29.921135+0000 mgr.y (mgr.14140) 88 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:31.966 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:31 smithi085 conmon[35961]: cluster 2021-02-10T04:23:30.441260+0000 mon.a (mon.0) 282 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2021-02-10T04:23:31.966 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:31 smithi085 conmon[35961]: audit 2021-02-10T04:23:30.531380+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:31.966 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:31 smithi085 conmon[35961]: audit 2021-02-10T04:23:30.534680 2021-02-10T04:23:31.967 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:31 smithi085 conmon[35961]: +0000 mon.a (mon.0) 284 : audit 2021-02-10T04:23:31.967 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:31 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:31.967 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:29 smithi085 conmon[29319]: cluster 2021-02-10T04:23:27.920940+0000 mgr.y (mgr. 2021-02-10T04:23:31.967 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:29 smithi085 conmon[29319]: 14140) 84 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:31.967 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:29 smithi085 conmon[29319]: audit 2021-02-10T04:23:27.992985+0000 mon.a (mon.0) 267 : audit [INF] from='osd.0 v2:172.21.15.85:6801/7' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]': finished 2021-02-10T04:23:31.967 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:29 smithi085 conmon[29319]: cluster 2021-02-10T04:23:27.993050+0000 mon.a (mon.0) 268 : cluster [DBG] osdmap e7: 1 total, 0 up, 0 in 2021-02-10T04:23:31.967 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:29 smithi085 conmon[29319]: audit 2021-02-10T04:23:27.993684+0000 mon.a (mon.0) 269 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-02-10T04:23:31.967 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:29 smithi085 conmon[29319]: audit 2021-02-10T04:23:27.997411+0000 mon.a (mon.0) 270 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-02-10T04:23:31.967 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:30 smithi085 conmon[29319]: cluster 2021-02-10T04:23:27.755677+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:23:31.967 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:30 smithi085 conmon[29319]: cluster 2021-02-10T04:23:27.755775+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:23:31.967 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:30 smithi085 conmon[29319]: cluster 2021-02-10T04:23:28.998730+0000 mon.a (mon.0) 271 : cluster [INF] osd.0 v2:172.21.15.85:6801/7 boot 2021-02-10T04:23:31.967 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:30 smithi085 conmon[29319]: cluster 2021-02-10T04:23:28.998760+0000 mon.a (mon.0) 272 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2021-02-10T04:23:31.967 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:30 smithi085 conmon[29319]: audit 2021-02-10T04:23:28.999069+0000 mon.a (mon.0) 273 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-02-10T04:23:31.968 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:30 smithi085 conmon[29319]: audit 2021-02-10T04:23:29.080004+0000 mon.a 2021-02-10T04:23:31.968 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:30 smithi085 conmon[29319]: (mon.0) 274 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:31.968 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:30 smithi085 conmon[29319]: audit 2021-02-10T04:23:29.429670+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:31.968 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:30 smithi085 conmon[29319]: audit 2021-02-10T04:23:29.437496+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:31.968 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:30 smithi085 conmon[29319]: audit 2021-02-10T04:23:29.439095+0000 mon.a (mon.0) 277 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:31.968 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:30 smithi085 conmon[29319]: audit 2021-02-10T04:23:29.615295+0000 mon.a (mon.0) 278 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-02-10T04:23:31.968 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:30 smithi085 conmon[29319]: audit 2021-02-10T04:23:29.616359+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-02-10T04:23:31.968 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:30 smithi085 conmon[29319]: audit 2021-02-10T04:23:29.616838+0000 mon.a (mon.0) 280 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:31.968 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:30 smithi085 conmon[29319]: audit 2021-02-10T04:23:29.617409+0000 mon.a (mon.0) 281 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:23:31.968 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:31 smithi085 conmon[29319]: cephadm 2021-02-10T04:23:29.425635+0000 mgr.y (mgr.14140) 85 : 2021-02-10T04:23:31.968 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:31 smithi085 conmon[29319]: cephadm [INF] Detected new or changed devices on smithi085 2021-02-10T04:23:31.968 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:31 smithi085 conmon[29319]: audit 2021-02-10T04:23:29.614454+0000 mgr.y (mgr.14140) 86 : audit [DBG] from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi085:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:23:31.968 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:31 smithi085 conmon[29319]: cephadm 2021-02-10T04:23:29.616185+0000 mgr.y (mgr.14140) 87 : cephadm [INF] Applying service osd.None on host smithi085... 2021-02-10T04:23:31.969 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:31 smithi085 conmon[29319]: cluster 2021-02-10T04:23:29.921135+0000 mgr.y (mgr.14140) 88 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:31.969 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:31 smithi085 conmon[29319]: cluster 2021-02-10T04:23:30.441260+0000 mon.a (mon.0) 282 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2021-02-10T04:23:31.969 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:31 smithi085 conmon[29319]: audit 2021-02-10T04:23:30.531380+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:31.969 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:31 smithi085 conmon[29319]: audit 2021-02-10T04:23:30.534680+0000 mon.a (mon.0) 284 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:29 smithi164 conmon[29288]: cluster 2021-02-10T04:23:27.920940+0000 mgr.y (mgr.14140) 84 2021-02-10T04:23:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:29 smithi164 conmon[29288]: : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:29 smithi164 conmon[29288]: audit 2021-02-10T04:23:27.992985+0000 mon.a (mon.0) 267 : audit [INF] from='osd.0 v2:172.21.15.85:6801/7' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]': finished 2021-02-10T04:23:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:29 smithi164 conmon[29288]: cluster 2021-02-10T04:23:27.993050 2021-02-10T04:23:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:29 smithi164 conmon[29288]: +0000 mon.a (mon.0) 268 : cluster [DBG] osdmap e7: 1 total, 0 up, 0 in 2021-02-10T04:23:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:29 smithi164 conmon[29288]: 2021-02-10T04:23:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:29 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:23:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:29 smithi164 conmon[29288]: :23:27.993684+0000 mon.a (mon.0) 2021-02-10T04:23:31.969 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:29 smithi164 conmon[29288]: 269 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-02-10T04:23:31.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:29 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:23:31.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:29 smithi164 conmon[29288]: :23:27.997411+0000 mon.a (mon.0) 270 2021-02-10T04:23:31.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:29 smithi164 conmon[29288]: : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-02-10T04:23:31.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: cluster 2021-02-10T04:23:27.755677+0000 osd.0 (osd.0 2021-02-10T04:23:31.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: ) 1 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:23:31.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: cluster 2021-02-10T04:23:27.755775+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:23:31.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: cluster 2021-02-10T04:23:28.998730 2021-02-10T04:23:31.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: +0000 mon.a (mon.0) 271 : cluster [INF] osd.0 v2:172.21.15.85:6801/7 boot 2021-02-10T04:23:31.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: 2021-02-10T04:23:31.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: cluster 2021-02-10T04:23: 2021-02-10T04:23:31.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: 28.998760+0000 mon.a (mon.0) 272 : cluster 2021-02-10T04:23:31.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: [DBG] osdmap e8: 1 total, 1 up, 1 in 2021-02-10T04:23:31.970 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: audit 2021-02-10T04:23: 2021-02-10T04:23:31.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: 28.999069+0000 mon.a (mon.0) 273 : audit 2021-02-10T04:23:31.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-02-10T04:23:31.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: audit 2021-02-10T04: 2021-02-10T04:23:31.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: 23:29.080004+0000 mon.a (mon.0) 274 2021-02-10T04:23:31.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:31.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: audit 2021-02-10T04: 2021-02-10T04:23:31.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: 23:29.429670+0000 mon.a (mon.0) 275 : 2021-02-10T04:23:31.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:31.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: audit 2021-02-10T04:23 2021-02-10T04:23:31.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: :29.437496+0000 mon.a (mon.0) 276 : audit 2021-02-10T04:23:31.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:31.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: audit 2021-02-10T04:23:29. 2021-02-10T04:23:31.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: 439095+0000 mon.a (mon.0) 277 : audit [DBG] 2021-02-10T04:23:31.971 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:31.972 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: audit 2021-02-10T04:23:29.615295+0000 mon.a (mon.0) 278 : audit 2021-02-10T04:23:31.972 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-02-10T04:23:31.972 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:23:31.972 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: 10T04:23:29.616359+0000 mon.a ( 2021-02-10T04:23:31.972 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: mon.0) 279 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-02-10T04:23:31.972 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:23:31.972 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: -10T04:23:29.616838+0000 mon.a (mon. 2021-02-10T04:23:31.972 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: 0) 280 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:31.972 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:23:31.972 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: -10T04:23:29.617409+0000 mon.a ( 2021-02-10T04:23:31.972 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:30 smithi164 conmon[29288]: mon.0) 281 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:23:31.972 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:31 smithi164 conmon[29288]: cephadm 2021-02-10T04:23:29.425635+0000 mgr.y (mgr.14140) 85 : 2021-02-10T04:23:31.972 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:31 smithi164 conmon[29288]: cephadm [INF] Detected new or changed devices on smithi085 2021-02-10T04:23:31.972 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:31 smithi164 conmon[29288]: audit 2021-02-10T04:23:29.614454+0000 mgr.y (mgr.14140) 86 : audit [DBG] from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi085:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:23:31.973 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:31 smithi164 conmon[29288]: cephadm 2021-02-10T04:23:29.616185+0000 mgr.y (mgr.14140) 87 : cephadm [INF] Applying service osd.None on host smithi085... 2021-02-10T04:23:31.973 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:31 smithi164 conmon[29288]: cluster 2021-02-10T04:23:29.921135+0000 mgr.y (mgr.14140) 88 : cluster 2021-02-10T04:23:31.973 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:31 smithi164 conmon[29288]: [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-02-10T04:23:31.973 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:31 smithi164 conmon[29288]: cluster 2021-02-10T04:23:30.441260+0000 mon.a 2021-02-10T04:23:31.973 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:31 smithi164 conmon[29288]: (mon.0) 282 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2021-02-10T04:23:31.973 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:31 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:23:31.973 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:31 smithi164 conmon[29288]: T04:23:30.531380+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:31.973 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:31 smithi164 conmon[29288]: audit 2021-02-10T04:23:30.534680+0000 mon.a (mon.0 2021-02-10T04:23:31.973 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:31 smithi164 conmon[29288]: ) 284 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:33.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:33 smithi164 conmon[29288]: cluster 2021-02-10T04:23:31.921397+0000 mgr.y (mgr.14140) 89 : cluster [DBG] 2021-02-10T04:23:33.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:33 smithi164 conmon[29288]: pgmap v56: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-02-10T04:23:33.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:33 smithi164 conmon[29288]: audit 2021-02-10T04:23:32.187238+0000 mon.a (mon.0) 285 : audit [INF] from='client.? 172.21.15.85:0/1949314716' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3421a5ce-872e-4b52-adf7-6aaa595871a4"}]: dispatch 2021-02-10T04:23:33.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:33 smithi164 conmon[29288]: audit 2021-02-10T04:23:32.192376+0000 mon.a (mon.0) 286 2021-02-10T04:23:33.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:33 smithi164 conmon[29288]: : audit [INF] from='client.? 172.21.15.85:0/1949314716' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3421a5ce-872e-4b52-adf7-6aaa595871a4"}]': finished 2021-02-10T04:23:33.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:33 smithi164 conmon[29288]: cluster 2021-02-10T04:23:32.192422+0000 2021-02-10T04:23:33.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:33 smithi164 conmon[29288]: mon.a (mon.0) 287 : cluster [DBG] osdmap e10: 2 total, 1 up, 1 in 2021-02-10T04:23:33.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:33 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:23:33.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:33 smithi164 conmon[29288]: 10T04:23:32.192515+0000 mon.a (mon. 2021-02-10T04:23:33.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:33 smithi164 conmon[29288]: 0) 288 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-02-10T04:23:33.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:33 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:23:33.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:33 smithi164 conmon[29288]: 10T04:23:32.644481+0000 mon.a (mon 2021-02-10T04:23:33.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:33 smithi164 conmon[29288]: .0) 289 : audit [DBG] from='client.? 172.21.15.85:0/4056671231' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-02-10T04:23:33.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:33 smithi085 conmon[35961]: cluster 2021-02-10T04:23:31.921397+0000 mgr.y 2021-02-10T04:23:33.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:33 smithi085 conmon[35961]: (mgr.14140) 89 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-02-10T04:23:33.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:33 smithi085 conmon[35961]: audit 2021-02-10T04:23:32.187238+0000 mon.a (mon.0) 285 : audit [INF] from='client.? 172.21.15.85:0/1949314716' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3421a5ce-872e-4b52-adf7-6aaa595871a4"}]: dispatch 2021-02-10T04:23:33.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:33 smithi085 conmon[35961]: audit 2021-02-10T04:23:32.192376+0000 mon.a (mon.0) 286 : audit [INF] from='client.? 172.21.15.85:0/1949314716' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3421a5ce-872e-4b52-adf7-6aaa595871a4"}]': finished 2021-02-10T04:23:33.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:33 smithi085 conmon[35961]: cluster 2021-02-10T04:23:32.192422+0000 mon.a (mon.0) 287 : cluster [DBG] osdmap e10: 2 total, 1 up, 1 in 2021-02-10T04:23:33.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:33 smithi085 conmon[35961]: audit 2021-02-10T04:23:32.192515+0000 mon.a (mon.0) 288 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-02-10T04:23:33.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:33 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:23:33.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:33 smithi085 conmon[35961]: -10T04:23:32.644481+0000 mon.a (mon.0) 289 : audit [DBG] from='client.? 172.21.15.85:0/4056671231' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-02-10T04:23:33.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:33 smithi085 conmon[29319]: cluster 2021-02-10T04:23:31.921397+0000 mgr.y (mgr 2021-02-10T04:23:33.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:33 smithi085 conmon[29319]: .14140) 89 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-02-10T04:23:33.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:33 smithi085 conmon[29319]: audit 2021-02-10T04:23:32.187238+0000 mon.a (mon.0) 285 : audit [INF] from='client.? 172.21.15.85:0/1949314716' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3421a5ce-872e-4b52-adf7-6aaa595871a4"}]: dispatch 2021-02-10T04:23:33.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:33 smithi085 conmon[29319]: audit 2021-02-10T04:23:32.192376+0000 mon.a (mon.0) 286 : audit [INF] from='client.? 172.21.15.85:0/1949314716' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3421a5ce-872e-4b52-adf7-6aaa595871a4"}]': finished 2021-02-10T04:23:33.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:33 smithi085 conmon[29319]: cluster 2021-02-10T04:23:32.192422+0000 mon.a (mon.0) 287 : cluster [DBG] osdmap e10: 2 total, 1 up, 1 in 2021-02-10T04:23:33.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:33 smithi085 conmon[29319]: audit 2021-02-10T04:23:32.192515+0000 mon.a (mon.0) 288 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-02-10T04:23:33.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:33 smithi085 conmon[29319]: audit 2021-02-10T04:23:32.644481+0000 mon.a (mon.0) 289 : audit [DBG] from='client.? 172.21.15.85:0/4056671231' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-02-10T04:23:35.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:35 smithi085 conmon[29319]: cluster 2021-02-10T04:23:33.921683+0000 mgr.y (mgr.14140) 90 : cluster 2021-02-10T04:23:35.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:35 smithi085 conmon[29319]: [DBG] pgmap v58: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-02-10T04:23:35.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:35 smithi085 conmon[35961]: cluster 2021-02-10T04:23:33.921683+0000 mgr.y ( 2021-02-10T04:23:35.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:35 smithi085 conmon[35961]: mgr.14140) 90 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-02-10T04:23:35.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:35 smithi164 conmon[29288]: cluster 2021-02-10T04:23:33.921683+0000 mgr.y (mgr.14140) 90 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-02-10T04:23:36.303 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:36 smithi085 conmon[29319]: audit 2021-02-10T04:23:35.466525+0000 mon.a (mon.0) 290 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:23:36.303 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:36 smithi085 conmon[35961]: audit 2021-02-10T04:23:35.466525+0000 mon.a (mon.0) 290 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:23:36.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:36 smithi164 conmon[29288]: audit 2021-02-10T04:23:35.466525+0000 mon.a (mon.0) 290 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:23:37.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:37 smithi085 conmon[29319]: cluster 2021-02-10T04:23:35.922119+0000 mgr.y (mgr.14140 2021-02-10T04:23:37.083 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:37 smithi085 conmon[29319]: ) 91 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-02-10T04:23:37.083 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:37 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:23:37.083 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:37 smithi085 conmon[35961]: 23:35.922119+0000 mgr.y (mgr.14140) 91 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-02-10T04:23:37.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:37 smithi164 conmon[29288]: cluster 2021-02-10T04:23:35.922119+0000 mgr.y (mgr.14140) 91 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-02-10T04:23:39.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:38 smithi085 conmon[29319]: cluster 2021-02-10T04:23:37 2021-02-10T04:23:39.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:38 smithi085 conmon[29319]: .922430+0000 mgr.y (mgr.14140) 92 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-02-10T04:23:39.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:38 smithi085 conmon[29319]: audit 2021-02-10T04:23:37.930579+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-02-10T04:23:39.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:38 smithi085 conmon[29319]: audit 2021-02-10T04:23:37.931177+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:39.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:38 smithi085 conmon[29319]: cephadm 2021-02-10T04:23:37.931626+0000 mgr.y (mgr.14140) 93 : cephadm [INF] Deploying daemon osd.1 on smithi085 2021-02-10T04:23:39.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:38 smithi085 conmon[29319]: audit 2021-02-10T04:23:37.932017+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.1", "key": "container_image"}]: dispatch 2021-02-10T04:23:39.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:38 smithi085 conmon[29319]: audit 2021-02-10T04:23:38.029863+0000 mon.a (mon.0) 294 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:23:39.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:38 smithi085 conmon[29319]: audit 2021-02-10T04:23:38.030759+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:23:39.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:38 smithi085 conmon[35961]: cluster 2021-02-10T04:23:37. 2021-02-10T04:23:39.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:38 smithi085 conmon[35961]: 922430+0000 mgr.y (mgr.14140) 92 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-02-10T04:23:39.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:38 smithi085 conmon[35961]: audit 2021-02-10T04:23:37.930579+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-02-10T04:23:39.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:38 smithi085 conmon[35961]: audit 2021-02-10T04:23:37.931177+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:39.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:38 smithi085 conmon[35961]: cephadm 2021-02-10T04:23:37.931626+0000 mgr.y (mgr.14140) 93 : cephadm [INF] Deploying daemon osd.1 on smithi085 2021-02-10T04:23:39.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:38 smithi085 conmon[35961]: audit 2021-02-10T04:23:37.932017+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.1", "key": "container_image"}]: dispatch 2021-02-10T04:23:39.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:38 smithi085 conmon[35961]: audit 2021-02-10T04:23:38.029863+0000 mon.a (mon.0) 294 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:23:39.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:38 smithi085 conmon[35961]: audit 2021-02-10T04:23:38.030759+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:23:39.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:38 smithi164 conmon[29288]: cluster 2021-02-10T04:23:37.922430+0000 mgr.y (mgr.14140) 92 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-02-10T04:23:39.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:38 smithi164 conmon[29288]: audit 2021-02-10T04:23:37.930579+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-02-10T04:23:39.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:38 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:23:39.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:38 smithi164 conmon[29288]: :23:37.931177+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:39.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:38 smithi164 conmon[29288]: cephadm 2021-02-10T 2021-02-10T04:23:39.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:38 smithi164 conmon[29288]: 04:23:37.931626+0000 mgr.y (mgr.14140) 2021-02-10T04:23:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:38 smithi164 conmon[29288]: 93 : cephadm [INF] Deploying daemon osd.1 on smithi085 2021-02-10T04:23:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:38 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:23:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:38 smithi164 conmon[29288]: 04:23:37.932017+0000 mon.a (mon.0 2021-02-10T04:23:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:38 smithi164 conmon[29288]: ) 293 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.1", "key": "container_image"}]: dispatch 2021-02-10T04:23:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:38 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:23:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:38 smithi164 conmon[29288]: T04:23:38.029863+0000 mon.a (mon. 2021-02-10T04:23:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:38 smithi164 conmon[29288]: 0) 294 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:23:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:38 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:23:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:38 smithi164 conmon[29288]: -10T04:23:38.030759+0000 mon.a ( 2021-02-10T04:23:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:38 smithi164 conmon[29288]: mon.0) 295 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:23:41.114 INFO:teuthology.orchestra.run.smithi085.stdout:Created osd(s) 1 on host 'smithi085' 2021-02-10T04:23:41.265 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:40 smithi085 conmon[29319]: cluster 2021-02-10T04:23:39.922690+0000 mgr.y (mgr.14140) 94 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-02-10T04:23:41.265 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:40 smithi085 conmon[35961]: cluster 2021-02-10T04:23:39.922690+0000 mgr.y (mgr 2021-02-10T04:23:41.265 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:40 smithi085 conmon[35961]: .14140) 94 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-02-10T04:23:41.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:40 smithi164 conmon[29288]: cluster 2021-02-10T04:23:39.922690+0000 mgr.y (mgr.14140) 94 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-02-10T04:23:41.694 DEBUG:teuthology.orchestra.run.smithi085:osd.1> sudo journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.1.service 2021-02-10T04:23:41.696 INFO:tasks.cephadm:Deploying osd.2 on smithi085 with /dev/vg_nvme/lv_2... 2021-02-10T04:23:41.697 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2021-02-10T04:23:41.725 INFO:journalctl@ceph.osd.1.smithi085.stdout:-- Logs begin at Wed 2021-02-10 04:09:23 UTC. -- 2021-02-10T04:23:42.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:41 smithi085 conmon[29319]: audit 2021-02-10T04:23 2021-02-10T04:23:42.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:41 smithi085 conmon[29319]: :41.104113+0000 mon.a (mon.0) 296 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:42.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:41 smithi085 conmon[29319]: audit 2021-02-10T04:23:41.106550+0000 mon.a (mon.0) 297 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:42.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:41 smithi085 conmon[29319]: audit 2021-02-10T04:23:41.112644+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:42.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:41 smithi085 conmon[35961]: audit 2021-02-10T04:23 2021-02-10T04:23:42.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:41 smithi085 conmon[35961]: :41.104113+0000 mon.a (mon.0) 296 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:42.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:41 smithi085 conmon[35961]: audit 2021-02-10T04:23:42.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:41 smithi085 conmon[35961]: 2021-02-10T04:23:41. 2021-02-10T04:23:42.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:41 smithi085 conmon[35961]: 106550+0000 mon.a (mon.0) 297 : audit 2021-02-10T04:23:42.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:41 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:42.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:41 smithi085 conmon[35961]: audit 2021-02-10T04:23:42.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:41 smithi085 conmon[35961]: 2021-02-10T04:23:41.112644 2021-02-10T04:23:42.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:41 smithi085 conmon[35961]: +0000 mon.a (mon.0) 298 : audit 2021-02-10T04:23:42.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:41 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:42.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:41 smithi164 conmon[29288]: audit 2021-02-10T04:23:41.104113+0000 mon.a (mon.0) 296 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:42.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:41 smithi164 conmon[29288]: audit 2021-02-10T04:23:41.106550+0000 mon.a (mon.0) 297 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:42.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:41 smithi164 conmon[29288]: audit 2021-02-10T04:23:41.112644+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:43.271 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:42 smithi085 conmon[29319]: cephadm 2021-02-10T04:23:41.103732+0000 mgr.y (mgr.14140) 95 : 2021-02-10T04:23:43.271 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:42 smithi085 conmon[29319]: cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:23:43.271 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:42 smithi085 conmon[29319]: cluster 2021-02-10T04:23:41.922989+0000 mgr.y (mgr.14140) 96 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-02-10T04:23:43.271 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:42 smithi085 conmon[29319]: audit 2021-02-10T04:23:42.526535+0000 mon.a (mon.0) 299 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:43.271 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:42 smithi085 conmon[29319]: audit 2021-02-10T04:23:42.527255+0000 mon.a (mon.0) 300 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:23:43.271 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:42 smithi085 conmon[35961]: cephadm 2021-02-10T04:23:41.103732+0000 mgr.y 2021-02-10T04:23:43.271 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:42 smithi085 conmon[35961]: (mgr.14140) 95 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:23:43.271 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:42 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:23:43.272 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:42 smithi085 conmon[35961]: 04:23:41.922989+0000 mgr.y (mgr.14140) 96 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-02-10T04:23:43.272 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:42 smithi085 conmon[35961]: audit 2021-02-10T04:23:43.272 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:42 smithi085 conmon[35961]: 2021-02-10T04:23:42.526535 2021-02-10T04:23:43.272 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:42 smithi085 conmon[35961]: +0000 mon.a (mon.0) 299 : audit 2021-02-10T04:23:43.272 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:42 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:43.272 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:42 smithi085 conmon[35961]: audit 2021-02-10T04:23:43.272 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:42 smithi085 conmon[35961]: 2021-02-10T04:23:42. 2021-02-10T04:23:43.272 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:42 smithi085 conmon[35961]: 527255+0000 mon.a (mon.0) 300 : audit 2021-02-10T04:23:43.272 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:42 smithi085 conmon[35961]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:23:43.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:42 smithi164 conmon[29288]: cephadm 2021-02-10T04:23:41.103732+0000 mgr.y (mgr.14140) 95 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:23:43.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:42 smithi164 conmon[29288]: cluster 2021-02-10T04:23:41.922989+0000 mgr.y (mgr.14140) 96 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-02-10T04:23:43.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:42 smithi164 conmon[29288]: audit 2021-02-10T04:23:42.526535+0000 mon.a (mon.0) 299 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:43.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:42 smithi164 conmon[29288]: 2021-02-10T04:23:43.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:42 smithi164 conmon[29288]: audit 2021-02-10T04:23:42.527255+0000 mon.a (mon.0) 300 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:23:43.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:42 smithi164 conmon[29288]: 2021-02-10T04:23:44.054 INFO:teuthology.orchestra.run.smithi085.stderr:--> Zapping: /dev/vg_nvme/lv_2 2021-02-10T04:23:44.054 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2021-02-10T04:23:44.054 INFO:teuthology.orchestra.run.smithi085.stderr: stderr: 10+0 records in 2021-02-10T04:23:44.055 INFO:teuthology.orchestra.run.smithi085.stderr:10+0 records out 2021-02-10T04:23:44.055 INFO:teuthology.orchestra.run.smithi085.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0120689 s, 869 MB/s 2021-02-10T04:23:44.055 INFO:teuthology.orchestra.run.smithi085.stderr:--> Zapping successful for: 2021-02-10T04:23:44.440 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:23:44 smithi085 conmon[45443]: debug 2021-02-10T04:23:44.317+0000 7f714c082200 -1 osd.1 0 log_to_monitors {default=true} 2021-02-10T04:23:44.844 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph orch daemon add osd smithi085:vg_nvme/lv_2 2021-02-10T04:23:45.198 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:44 smithi085 conmon[29319]: cluster 2021-02-10T04:23:43.923559 2021-02-10T04:23:46.419 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:44 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 97 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-02-10T04:23:46.420 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:44 smithi085 conmon[29319]: audit 2021-02-10T04:23:44.320517+0000 mon.c (mon.1) 6 : audit [INF] from='osd.1 v2:172.21.15.85:6805/7' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-02-10T04:23:46.420 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:44 smithi085 conmon[29319]: audit 2021-02-10T04:23:44.320823+0000 mon.a (mon.0) 301 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-02-10T04:23:46.420 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:44 smithi085 conmon[35961]: cluster 2021-02-10T04:23:43.923559+0000 2021-02-10T04:23:46.420 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:44 smithi085 conmon[35961]: mgr.y (mgr.14140) 97 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-02-10T04:23:46.420 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:44 smithi085 conmon[35961]: audit 2021-02-10T04:23:44.320517+0000 mon.c (mon.1) 6 : audit [INF] from='osd.1 v2:172.21.15.85:6805/7' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-02-10T04:23:46.420 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:44 smithi085 conmon[35961]: audit 2021-02-10T04:23:44.320823+0000 mon.a (mon.0) 301 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-02-10T04:23:46.421 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:44 smithi164 conmon[29288]: cluster 2021-02-10T04:23:43.923559+0000 mgr.y (mgr.14140) 97 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-02-10T04:23:46.421 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:44 smithi164 conmon[29288]: audit 2021-02-10T04:23: 2021-02-10T04:23:46.421 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:44 smithi164 conmon[29288]: 44.320517+0000 mon.c (mon.1) 6 : audit [INF] from='osd.1 v2:172.21.15.85:6805/7' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-02-10T04:23:46.421 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:44 smithi164 conmon[29288]: audit 2021-02-10T04:23:44.320823+0000 mon.a (mon.0) 301 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-02-10T04:23:46.422 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:45 smithi164 conmon[29288]: audit 2021-02-10T04:23:44.986009+0000 mon.a (mon.0 2021-02-10T04:23:46.422 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:45 smithi164 conmon[29288]: ) 302 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-02-10T04:23:46.422 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:45 smithi164 conmon[29288]: cluster 2021-02-10T04:23:44.986076+0000 mon.a (mon.0) 303 : cluster [DBG] osdmap e11: 2 total, 1 up, 1 in 2021-02-10T04:23:46.422 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:45 smithi164 conmon[29288]: audit 2021-02-10T04:23:44.986540+0000 2021-02-10T04:23:46.422 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:45 smithi164 conmon[29288]: mon.a (mon.0) 304 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-02-10T04:23:46.422 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:45 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:23:46.422 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:45 smithi164 conmon[29288]: 02-10T04:23:44.989874+0000 mon.c ( 2021-02-10T04:23:46.422 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:45 smithi164 conmon[29288]: mon.1) 7 : audit [INF] from='osd.1 v2:172.21.15.85:6805/7' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:23:46.422 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:45 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:23:46.422 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:45 smithi164 conmon[29288]: -10T04:23:44.995256+0000 mon.a ( 2021-02-10T04:23:46.422 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:45 smithi164 conmon[29288]: mon.0) 305 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:23:46.423 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:23:46 smithi085 conmon[45443]: debug 2021-02-10T04:23:46.000+0000 7f71425f3700 -1 osd.1 0 waiting for initial osdmap 2021-02-10T04:23:46.423 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:45 smithi085 conmon[29319]: audit 2021-02-10T04:23:44.986009 2021-02-10T04:23:46.423 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:45 smithi085 conmon[29319]: +0000 mon.a (mon.0) 302 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-02-10T04:23:46.423 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:45 smithi085 conmon[29319]: cluster 2021-02-10T04:23:44.986076+0000 mon.a (mon.0) 303 : cluster [DBG] osdmap e11: 2 total, 1 up, 1 in 2021-02-10T04:23:46.423 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:45 smithi085 conmon[29319]: audit 2021-02-10T04:23:44.986540+0000 mon.a (mon.0) 304 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-02-10T04:23:46.423 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:45 smithi085 conmon[29319]: audit 2021-02-10T04:23:44.989874+0000 mon.c (mon.1) 7 : audit [INF] from='osd.1 v2:172.21.15.85:6805/7' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:23:46.423 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:45 smithi085 conmon[29319]: audit 2021-02-10T04:23:44.995256+0000 mon.a (mon.0) 305 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:23:46.423 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:45 smithi085 conmon[35961]: audit 2021-02-10T04:23:44.986009 2021-02-10T04:23:46.423 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:45 smithi085 conmon[35961]: +0000 mon.a (mon.0) 302 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-02-10T04:23:46.423 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:45 smithi085 conmon[35961]: cluster 2021-02-10T04:23:44.986076+0000 mon.a (mon.0) 303 : cluster [DBG] osdmap e11: 2 total, 1 up, 1 in 2021-02-10T04:23:46.424 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:45 smithi085 conmon[35961]: audit 2021-02-10T04:23:44.986540+0000 mon.a (mon.0) 304 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-02-10T04:23:46.424 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:45 smithi085 conmon[35961]: audit 2021-02-10T04:23:44.989874+0000 mon.c (mon.1) 7 : audit [INF] from='osd.1 v2:172.21.15.85:6805/7' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:23:46.424 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:45 smithi085 conmon[35961]: audit 2021-02-10T04:23:44.995256+0000 mon.a (mon.0) 305 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:23:47.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:47 smithi164 conmon[29288]: cluster 2021-02-10T04:23:45.923828+0000 mgr.y (mgr.14140) 98 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-02-10T04:23:47.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:47 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:23:47.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:47 smithi164 conmon[29288]: :23:45.997018+0000 mon.a (mon.0) 306 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]': finished 2021-02-10T04:23:47.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:47 smithi164 conmon[29288]: cluster 2021-02-10T04:23:45.997106+0000 mon.a (mon.0) 307 : cluster [DBG] osdmap e12: 2 total, 1 up, 1 in 2021-02-10T04:23:47.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:47 smithi164 conmon[29288]: audit 2021-02-10T04:23:45.997452+0000 2021-02-10T04:23:47.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:47 smithi164 conmon[29288]: mon.a (mon.0) 308 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-02-10T04:23:47.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:47 smithi164 conmon[29288]: audit 2021-02-10T04:23:46.001917 2021-02-10T04:23:47.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:47 smithi164 conmon[29288]: +0000 mon.a (mon.0) 309 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-02-10T04:23:47.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:47 smithi164 conmon[29288]: 2021-02-10T04:23:47.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:47 smithi164 conmon[29288]: audit 2021-02-10T04:23:47.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:47 smithi164 conmon[29288]: 2021-02-10T04:23:46.705654+0000 mon.a ( 2021-02-10T04:23:47.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:47 smithi164 conmon[29288]: mon.0) 310 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:47.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:47 smithi085 conmon[29319]: cluster 2021-02-10T04:23:45.923828+0000 mgr.y (mgr.14140) 98 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-02-10T04:23:47.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:47 smithi085 conmon[29319]: audit 2021-02-10T04:23:45.997018+0000 mon.a (mon.0) 306 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]': finished 2021-02-10T04:23:47.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:47 smithi085 conmon[29319]: cluster 2021-02-10T04:23:45.997106+0000 mon.a (mon.0) 307 : cluster [DBG] osdmap e12: 2 total, 1 up, 1 in 2021-02-10T04:23:47.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:47 smithi085 conmon[29319]: audit 2021-02-10T04:23:45.997452+0000 mon.a (mon.0) 308 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-02-10T04:23:47.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:47 smithi085 conmon[29319]: audit 2021-02-10T04:23:46.001917+0000 mon.a (mon.0) 309 : audit 2021-02-10T04:23:47.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:47 smithi085 conmon[29319]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-02-10T04:23:47.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:47 smithi085 conmon[29319]: audit 2021-02-10T04:23:46.705654+0000 mon.a (mon.0) 310 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:47.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:47 smithi085 conmon[35961]: cluster 2021-02-10T04:23:45.923828+0000 mgr.y (mgr.14140) 98 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-02-10T04:23:47.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:47 smithi085 conmon[35961]: audit 2021-02-10T04:23:45.997018+0000 mon.a (mon.0) 306 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]': finished 2021-02-10T04:23:47.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:47 smithi085 conmon[35961]: cluster 2021-02-10T04:23:45.997106+0000 mon.a (mon.0) 307 : cluster [DBG] osdmap e12: 2 total, 1 up, 1 in 2021-02-10T04:23:47.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:47 smithi085 conmon[35961]: audit 2021-02-10T04:23:45.997452+0000 mon.a (mon.0) 308 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-02-10T04:23:47.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:47 smithi085 conmon[35961]: audit 2021-02-10T04:23:46.001917+0000 mon.a (mon.0 2021-02-10T04:23:47.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:47 smithi085 conmon[35961]: ) 309 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-02-10T04:23:47.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:47 smithi085 conmon[35961]: audit 2021-02-10T04:23:46.705654+0000 mon.a (mon.0) 310 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:48.098 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[29319]: cluster 2021-02-10T04:23:45. 2021-02-10T04:23:48.099 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[29319]: 286605+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:23:48.099 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[29319]: cluster 2021-02-10T04:23:45.286696+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:23:48.099 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[29319]: cluster 2021-02-10T04:23:47.002448+0000 mon.a (mon.0) 311 : cluster [INF] osd.1 v2:172.21.15.85:6805/7 boot 2021-02-10T04:23:48.099 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[29319]: cluster 2021-02-10T04:23:47.002492+0000 mon.a (mon.0) 312 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2021-02-10T04:23:48.099 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[29319]: audit 2021-02-10T04:23:47.003751+0000 mon.a (mon.0) 313 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-02-10T04:23:48.099 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[29319]: cephadm 2021-02-10T04:23:47.050843+0000 mgr.y (mgr.14140) 99 : cephadm [INF] Detected new or changed devices on smithi085 2021-02-10T04:23:48.099 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[29319]: audit 2021-02-10T04:23:47.054257+0000 mon.a (mon.0) 314 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:48.099 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[29319]: audit 2021-02-10T04:23:47.059754+0000 mon.a (mon.0) 315 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:48.099 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[29319]: audit 2021-02-10T04:23:47.068147+0000 mon.a (mon.0) 316 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:48.099 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[29319]: audit 2021-02-10T04:23:47.233451+0000 mon.a (mon.0) 317 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-02-10T04:23:48.099 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[29319]: audit 2021-02-10T04:23:47.234620+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-02-10T04:23:48.099 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[29319]: audit 2021-02-10T04:23:47.235203+0000 mon.a (mon.0) 319 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:48.099 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[29319]: audit 2021-02-10T04:23:47.235841+0000 mon.a (mon.0) 320 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:23:48.100 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[35961]: cluster 2021-02-10T04:23:45.286605+0000 osd.1 (osd.1) 2021-02-10T04:23:48.100 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[35961]: 1 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:23:48.100 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[35961]: cluster 2021-02-10T04:23:45.286696+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:23:48.100 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[35961]: cluster 2021-02-10T04:23:47.002448+0000 mon.a (mon.0) 311 : cluster [INF] osd.1 v2:172.21.15.85:6805/7 boot 2021-02-10T04:23:48.100 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[35961]: cluster 2021-02-10T04:23:47.002492+0000 mon.a (mon.0) 312 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2021-02-10T04:23:48.100 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[35961]: audit 2021-02-10T04:23:47.003751+0000 mon.a (mon.0) 313 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-02-10T04:23:48.100 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[35961]: cephadm 2021-02-10T04:23:47.050843+0000 mgr.y (mgr.14140) 99 : cephadm [INF] Detected new or changed devices on smithi085 2021-02-10T04:23:48.100 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[35961]: audit 2021-02-10T04:23:47.054257+0000 mon.a (mon.0) 314 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:48.100 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[35961]: audit 2021-02-10T04:23:47.059754+0000 mon.a (mon.0) 315 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:48.100 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[35961]: audit 2021-02-10T04:23:47.068147+0000 mon.a (mon.0) 316 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:48.100 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[35961]: audit 2021-02-10T04:23:47.233451+0000 mon.a (mon.0) 317 : audit [DBG] 2021-02-10T04:23:48.100 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-02-10T04:23:48.101 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[35961]: audit 2021-02-10T04:23:47.234620+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-02-10T04:23:48.101 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:23:48.101 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[35961]: T04:23:47.235203+0000 mon.a (mon.0) 319 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:48.101 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:23:48.101 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[35961]: 02-10T04:23:47.235841+0000 mon.a (mon.0 2021-02-10T04:23:48.101 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:48 smithi085 conmon[35961]: ) 320 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:23:48.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: cluster 2021-02-10T04:23:45.286605+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:23:48.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:23:48.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: :23:45.286696+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:23:48.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: cluster 2021-02-10T04:23:47.002448+0000 mon.a (mon.0) 311 : cluster [INF] osd.1 v2:172.21.15.85:6805/7 boot 2021-02-10T04:23:48.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: cluster 2021-02-10T04:23:47. 2021-02-10T04:23:48.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: 002492+0000 mon.a (mon.0) 312 : cluster [DBG] 2021-02-10T04:23:48.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: osdmap e13: 2 total, 2 up, 2 in 2021-02-10T04:23:48.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: audit 2021-02-10T04:23:47. 2021-02-10T04:23:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: 003751+0000 mon.a (mon.0) 313 : audit [DBG] 2021-02-10T04:23:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-02-10T04:23:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: cephadm 2021-02-10T04:23:47 2021-02-10T04:23:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: .050843+0000 mgr.y (mgr.14140) 99 : cephadm 2021-02-10T04:23:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: [INF] Detected new or changed devices on smithi085 2021-02-10T04:23:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: audit 2021-02-10T04:23: 2021-02-10T04:23:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: 47.054257+0000 mon.a (mon.0) 314 : audit 2021-02-10T04:23:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: audit 2021-02-10T04:23 2021-02-10T04:23:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: :47.059754+0000 mon.a (mon.0) 315 : audit 2021-02-10T04:23:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: audit 2021-02-10T04:23 2021-02-10T04:23:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: :47.068147+0000 mon.a (mon.0) 316 : 2021-02-10T04:23:48.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:48.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:23:48.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: 04:23:47.233451+0000 mon.a (mon.0 2021-02-10T04:23:48.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: ) 317 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-02-10T04:23:48.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: audit 2021-02-10T04:23:48.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: 2021-02-10T04:23:47. 2021-02-10T04:23:48.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: 234620+0000 mon.a (mon.0) 318 : audit [INF] 2021-02-10T04:23:48.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-02-10T04:23:48.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: audit 2021-02-10T04:23:47 2021-02-10T04:23:48.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: .235203+0000 mon.a (mon.0) 319 : audit 2021-02-10T04:23:48.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:48.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: audit 2021-02-10T04:23 2021-02-10T04:23:48.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: :47.235841+0000 mon.a (mon.0) 320 2021-02-10T04:23:48.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:48 smithi164 conmon[29288]: : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:23:49.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:49 smithi164 conmon[29288]: audit 2021-02-10T04:23:47.232585+0000 mgr.y (mgr.14140) 100 : audit [DBG] from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi085:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:23:49.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:49 smithi164 conmon[29288]: cephadm 2021-02-10T04:23 2021-02-10T04:23:49.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:49 smithi164 conmon[29288]: :47.234381+0000 mgr.y (mgr.14140) 101 : cephadm [INF] Applying service osd.None on host smithi085... 2021-02-10T04:23:49.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:49 smithi164 conmon[29288]: cluster 2021-02-10T04:23:47.924142+0000 mgr.y (mgr.14140) 102 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-02-10T04:23:49.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:49 smithi164 conmon[29288]: cluster 2021-02-10T04:23:48.070026+0000 mon.a (mon.0 2021-02-10T04:23:49.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:49 smithi164 conmon[29288]: ) 321 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2021-02-10T04:23:49.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:49 smithi085 conmon[29319]: audit 2021-02-10T04:23:47.232585+0000 2021-02-10T04:23:49.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:49 smithi085 conmon[29319]: mgr.y (mgr.14140) 100 : audit [DBG] from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi085:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:23:49.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:49 smithi085 conmon[29319]: cephadm 2021-02-10T04:23:47.234381+0000 mgr.y (mgr.14140) 101 : cephadm [INF] Applying service osd.None on host smithi085... 2021-02-10T04:23:49.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:49 smithi085 conmon[29319]: cluster 2021-02-10T04:23:47.924142+0000 mgr.y (mgr.14140) 102 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-02-10T04:23:49.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:49 smithi085 conmon[29319]: cluster 2021-02-10T04:23:48.070026+0000 mon.a (mon.0) 321 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2021-02-10T04:23:49.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:49 smithi085 conmon[35961]: audit 2021-02-10T04:23:47. 2021-02-10T04:23:49.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:49 smithi085 conmon[35961]: 232585+0000 mgr.y (mgr.14140) 100 : audit [DBG] from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi085:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:23:49.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:49 smithi085 conmon[35961]: cephadm 2021 2021-02-10T04:23:49.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:49 smithi085 conmon[35961]: -02-10T04:23: 2021-02-10T04:23:49.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:49 smithi085 conmon[35961]: 47.234381+0000 mgr.y (mgr 2021-02-10T04:23:49.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:49 smithi085 conmon[35961]: .14140) 101 : cephadm [INF] 2021-02-10T04:23:49.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:49 smithi085 conmon[35961]: Applying service osd.None on host smithi085... 2021-02-10T04:23:49.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:49 smithi085 conmon[35961]: cluster 2021-02-10T04:23:49.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:49 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:23:49.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:49 smithi085 conmon[35961]: 04:23:47.924142+0000 2021-02-10T04:23:49.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:49 smithi085 conmon[35961]: mgr.y (mgr.14140) 102 2021-02-10T04:23:49.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:49 smithi085 conmon[35961]: : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-02-10T04:23:49.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:49 smithi085 conmon[35961]: cluster 2021-02-10T04:23:49.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:49 smithi085 conmon[35961]: 2021-02-10T04:23: 2021-02-10T04:23:49.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:49 smithi085 conmon[35961]: 48.070026+0000 mon.a (mon 2021-02-10T04:23:49.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:49 smithi085 conmon[35961]: .0) 321 : cluster [DBG] 2021-02-10T04:23:49.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:49 smithi085 conmon[35961]: osdmap e14: 2 total, 2 up, 2 in 2021-02-10T04:23:50.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:50 smithi164 conmon[29288]: audit 2021-02-10T04:23:49.570802+0000 mon.a (mon.0) 322 : audit [INF] from='client.? 172.21.15.85:0/1776578343' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96e1952a-1042-4316-9fc9-86813e88b912"}]: dispatch 2021-02-10T04:23:50.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:50 smithi164 conmon[29288]: audit 2021-02-10T04:23:49.574575+0000 mon.a (mon.0) 323 : audit [INF] from='client.? 172.21.15.85:0/1776578343' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "96e1952a-1042-4316-9fc9-86813e88b912"}]': finished 2021-02-10T04:23:50.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:50 smithi164 conmon[29288]: cluster 2021-02-10T04:23:49.574623+0000 2021-02-10T04:23:50.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:50 smithi164 conmon[29288]: mon.a (mon.0) 324 : cluster [DBG] osdmap e15: 3 total, 2 up, 2 in 2021-02-10T04:23:50.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:50 smithi164 conmon[29288]: audit 2021-02-10T04:23:49. 2021-02-10T04:23:50.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:50 smithi164 conmon[29288]: 574723+0000 mon.a (mon.0) 325 : audit [DBG] 2021-02-10T04:23:50.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:50 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-02-10T04:23:50.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:50 smithi085 conmon[29319]: audit 2021-02-10T04:23:49.570802+0000 mon.a (mon 2021-02-10T04:23:50.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:50 smithi085 conmon[29319]: .0) 322 : audit [INF] from='client.? 172.21.15.85:0/1776578343' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96e1952a-1042-4316-9fc9-86813e88b912"}]: dispatch 2021-02-10T04:23:50.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:50 smithi085 conmon[29319]: audit 2021-02-10T04:23:49.574575+0000 mon.a (mon.0) 323 : audit [INF] from='client.? 172.21.15.85:0/1776578343' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "96e1952a-1042-4316-9fc9-86813e88b912"}]': finished 2021-02-10T04:23:50.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:50 smithi085 conmon[29319]: cluster 2021-02-10T04:23:49.574623+0000 mon.a (mon.0) 324 : cluster [DBG] osdmap e15: 3 total, 2 up, 2 in 2021-02-10T04:23:50.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:50 smithi085 conmon[29319]: audit 2021-02-10T04:23:49.574723+0000 mon.a (mon.0) 325 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-02-10T04:23:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:50 smithi085 conmon[35961]: audit 2021-02-10T04:23:49.570802 2021-02-10T04:23:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:50 smithi085 conmon[35961]: +0000 mon.a (mon.0) 322 : audit [INF] from='client.? 172.21.15.85:0/1776578343' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96e1952a-1042-4316-9fc9-86813e88b912"}]: dispatch 2021-02-10T04:23:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:50 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:23:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:50 smithi085 conmon[35961]: 23:49.574575+0000 mon.a (mon.0) 323 : audit [INF] from='client.? 172.21.15.85:0/1776578343' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "96e1952a-1042-4316-9fc9-86813e88b912"}]': finished 2021-02-10T04:23:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:50 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:23:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:50 smithi085 conmon[35961]: 23:49.574623+0000 mon.a (mon.0) 324 : cluster [DBG] osdmap e15: 3 total, 2 up, 2 in 2021-02-10T04:23:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:50 smithi085 conmon[35961]: 2021-02-10T04:23:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:50 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:23:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:50 smithi085 conmon[35961]: 02-10T04:23:49.574723+0000 mon.a (mon.0) 325 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-02-10T04:23:51.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:51 smithi164 conmon[29288]: cluster 2021-02-10T04:23:49.924418+0000 mgr.y (mgr.14140) 103 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-02-10T04:23:51.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:51 smithi164 conmon[29288]: audit 2021-02-10T04: 2021-02-10T04:23:51.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:51 smithi164 conmon[29288]: 23:50.027625+0000 mon.c (mon.1) 8 : audit [DBG] from='client.? 172.21.15.85:0/4046134960' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-02-10T04:23:51.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:51 smithi085 conmon[29319]: cluster 2021-02-10T04:23:49.924418+0000 mgr.y (mgr.14140) 103 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-02-10T04:23:51.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:51 smithi085 conmon[29319]: 2021-02-10T04:23:51.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:51 smithi085 conmon[29319]: audit 2021-02-10T04:23:50.027625+0000 mon.c (mon.1) 8 : audit [DBG] from='client.? 172.21.15.85:0/4046134960' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-02-10T04:23:51.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:51 smithi085 conmon[35961]: cluster 2021-02-10T04:23:49.924418+0000 mgr.y (mgr.14140) 103 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-02-10T04:23:51.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:51 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:23:51.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:51 smithi085 conmon[35961]: 23:50.027625+0000 mon.c (mon.1) 8 : audit [DBG] from='client.? 172.21.15.85:0/4046134960' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-02-10T04:23:53.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:53 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:23:53.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:53 smithi085 conmon[29319]: :23:51.924811+0000 mgr.y (mgr.14140) 104 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-02-10T04:23:53.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:53 smithi085 conmon[29319]: audit 2021-02-10T04:23:52.911180+0000 mon.a (mon.0) 326 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:23:53.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:53 smithi085 conmon[35961]: cluster 2021-02-10T04:23:51 2021-02-10T04:23:53.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:53 smithi085 conmon[35961]: .924811+0000 mgr.y (mgr.14140) 104 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-02-10T04:23:53.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:53 smithi085 conmon[35961]: audit 2021-02-10T04:23:52.911180+0000 mon.a (mon.0) 326 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:23:53.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:53 smithi164 conmon[29288]: cluster 2021-02-10T04:23:51.924811+0000 mgr.y (mgr.14140) 104 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-02-10T04:23:53.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:53 smithi164 conmon[29288]: audit 2021-02-10T04: 2021-02-10T04:23:53.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:53 smithi164 conmon[29288]: 23:52.911180+0000 mon.a (mon.0) 326 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:23:55.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:55 smithi164 conmon[29288]: cluster 2021-02-10T04:23:53.925319+0000 mgr.y (mgr.14140) 105 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-02-10T04:23:55.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:55 smithi085 conmon[29319]: cluster 2021-02-10T04:23: 2021-02-10T04:23:55.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:55 smithi085 conmon[29319]: 53.925319+0000 mgr.y (mgr.14140) 105 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-02-10T04:23:55.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:55 smithi085 conmon[35961]: cluster 2021-02-10T04:23:53. 2021-02-10T04:23:55.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:55 smithi085 conmon[35961]: 925319+0000 mgr.y (mgr.14140) 105 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-02-10T04:23:56.341 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:56 smithi085 conmon[29319]: audit 2021-02-10T04: 2021-02-10T04:23:56.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:56 smithi085 conmon[29319]: 23:55.526696+0000 mon.a (mon.0) 327 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-02-10T04:23:56.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:56 smithi085 conmon[29319]: audit 2021-02-10T04:23:55.527349+0000 mon.a (mon.0) 328 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:56.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:56 smithi085 conmon[29319]: audit 2021-02-10T04:23:55.528105+0000 mon.a (mon.0) 329 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.2", "key": "container_image"}]: dispatch 2021-02-10T04:23:56.342 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:56 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:23:56.342 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:56 smithi085 conmon[35961]: 23:55.526696+0000 mon.a (mon.0) 327 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-02-10T04:23:56.342 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:56 smithi085 conmon[35961]: audit 2021-02-10T04:23:55.527349+0000 mon.a (mon.0) 328 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:56.342 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:56 smithi085 conmon[35961]: audit 2021-02-10T04:23:55.528105+0000 mon.a (mon.0) 329 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.2", "key": "container_image"}]: dispatch 2021-02-10T04:23:56.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:56 smithi164 conmon[29288]: audit 2021-02-10T04:23:55.526696+0000 mon.a (mon.0) 327 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-02-10T04:23:56.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:56 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:23:56.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:56 smithi164 conmon[29288]: -10T04:23:55.527349+0000 mon.a (mon.0) 328 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:23:56.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:56 smithi164 conmon[29288]: audit 2021-02-10T04:23:55.528105+0000 mon.a (mon.0) 329 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.2", "key": "container_image"}]: dispatch 2021-02-10T04:23:56.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:56 smithi164 conmon[29288]: 2021-02-10T04:23:57.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:57 smithi085 conmon[29319]: cephadm 2021-02-10T04:23:55.527788+0000 mgr.y (mgr.14140) 106 : cephadm [INF] Deploying daemon osd.2 on smithi085 2021-02-10T04:23:57.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:57 smithi085 conmon[29319]: cluster 2021-02-10T04:23:55.925651+0000 mgr.y (mgr.14140) 107 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-02-10T04:23:57.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:57 smithi085 conmon[35961]: cephadm 2021-02-10T04:23:55.527788+0000 mgr.y ( 2021-02-10T04:23:57.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:57 smithi085 conmon[35961]: mgr.14140) 106 : cephadm [INF] Deploying daemon osd.2 on smithi085 2021-02-10T04:23:57.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:57 smithi085 conmon[35961]: cluster 2021-02-10T04:23:55.925651+0000 mgr.y (mgr.14140 2021-02-10T04:23:57.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:57 smithi085 conmon[35961]: ) 107 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-02-10T04:23:57.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:57 smithi164 conmon[29288]: cephadm 2021-02-10T04:23:55.527788+0000 mgr.y (mgr.14140) 106 : cephadm [INF] Deploying daemon osd.2 on smithi085 2021-02-10T04:23:57.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:57 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:23:57.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:57 smithi164 conmon[29288]: 10T04:23:55.925651+0000 mgr.y (mgr.14140) 107 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-02-10T04:23:58.908 INFO:teuthology.orchestra.run.smithi085.stdout:Created osd(s) 2 on host 'smithi085' 2021-02-10T04:23:59.087 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:58 smithi085 conmon[29319]: cluster 2021-02-10T04:23:57.925914+0000 2021-02-10T04:23:59.088 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:58 smithi085 conmon[29319]: mgr.y (mgr.14140) 108 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-02-10T04:23:59.088 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:58 smithi085 conmon[35961]: cluster 2021-02-10T04:23:57.925914+0000 mgr.y 2021-02-10T04:23:59.088 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:58 smithi085 conmon[35961]: (mgr.14140) 108 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-02-10T04:23:59.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:58 smithi164 conmon[29288]: cluster 2021-02-10T04:23:57.925914+0000 mgr.y (mgr.14140) 108 2021-02-10T04:23:59.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:58 smithi164 conmon[29288]: : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-02-10T04:23:59.571 DEBUG:teuthology.orchestra.run.smithi085:osd.2> sudo journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.2.service 2021-02-10T04:23:59.574 INFO:tasks.cephadm:Deploying osd.3 on smithi085 with /dev/vg_nvme/lv_1... 2021-02-10T04:23:59.574 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2021-02-10T04:23:59.601 INFO:journalctl@ceph.osd.2.smithi085.stdout:-- Logs begin at Wed 2021-02-10 04:09:23 UTC. -- 2021-02-10T04:23:59.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:59 smithi085 conmon[35961]: audit 2021-02-10T04:23:58.893280+0000 mon.a ( 2021-02-10T04:23:59.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:59 smithi085 conmon[35961]: mon.0) 330 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:59.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:59 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:23:59.898 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:59 smithi085 conmon[29319]: audit 2021-02-10T04:23:58.893280+0000 mon.a (mon.0) 330 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:23:59.898 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:59 smithi085 conmon[29319]: audit 2021-02-10T04:23:58.902986+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:23:59.898 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:23:59 smithi085 conmon[29319]: audit 2021-02-10T04:23:58.906460+0000 mon.a (mon.0) 332 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:00.152 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:59 smithi085 conmon[35961]: 23:58.902986+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:00.153 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:59 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:24:00.153 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:59 smithi085 conmon[35961]: 04:23:58.906460+0000 mon.a (mon.0) 2021-02-10T04:24:00.153 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:23:59 smithi085 conmon[35961]: 332 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:00.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:59 smithi164 conmon[29288]: audit 2021-02-10T04:23:58.893280+0000 mon.a (mon 2021-02-10T04:24:00.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:59 smithi164 conmon[29288]: .0) 330 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:24:00.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:59 smithi164 conmon[29288]: audit 2021-02-10T04:23:58.902986+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:00.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:23:59 smithi164 conmon[29288]: audit 2021-02-10T04:23:58.906460+0000 mon.a (mon.0) 332 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:00.903 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:00 smithi085 conmon[29319]: cluster 2021-02-10T04:23:59.926184+0000 mgr.y 2021-02-10T04:24:00.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:00 smithi085 conmon[35961]: cluster 2021-02-10T04:23:59.926184 2021-02-10T04:24:00.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:00 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 109 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-02-10T04:24:00.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:00 smithi085 conmon[35961]: audit 2021-02-10T04:24:00.385002+0000 mon.a (mon.0) 333 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:00.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:00 smithi085 conmon[35961]: audit 2021-02-10T04:24:00.385994+0000 mon.a (mon.0) 334 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:24:01.170 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:00 smithi085 conmon[29319]: (mgr.14140) 109 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-02-10T04:24:01.170 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:00 smithi085 conmon[29319]: audit 2021-02-10T04:24:00.385002+0000 mon.a (mon.0) 333 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:01.170 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:00 smithi085 conmon[29319]: audit 2021-02-10T04:24:00.385994+0000 mon.a (mon.0) 334 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:24:01.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:00 smithi164 conmon[29288]: cluster 2021-02-10T04:23:59.926184+0000 mgr.y (mgr. 2021-02-10T04:24:01.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:00 smithi164 conmon[29288]: 14140) 109 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-02-10T04:24:01.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:00 smithi164 conmon[29288]: audit 2021-02-10T04:24:00.385002+0000 mon.a (mon.0) 333 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:01.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:00 smithi164 conmon[29288]: audit 2021-02-10T04:24:00.385994+0000 mon.a (mon.0) 334 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:24:02.284 INFO:teuthology.orchestra.run.smithi085.stderr:--> Zapping: /dev/vg_nvme/lv_1 2021-02-10T04:24:02.285 INFO:teuthology.orchestra.run.smithi085.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2021-02-10T04:24:02.285 INFO:teuthology.orchestra.run.smithi085.stderr: stderr: 10+0 records in 2021-02-10T04:24:02.285 INFO:teuthology.orchestra.run.smithi085.stderr:10+0 records out 2021-02-10T04:24:02.285 INFO:teuthology.orchestra.run.smithi085.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0120095 s, 873 MB/s 2021-02-10T04:24:02.285 INFO:teuthology.orchestra.run.smithi085.stderr:--> Zapping successful for: 2021-02-10T04:24:02.328 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:24:02 smithi085 conmon[49106]: debug 2021-02-10T04:24:02.051+0000 7f77b19a8200 -1 osd.2 0 log_to_monitors {default=true} 2021-02-10T04:24:02.959 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph orch daemon add osd smithi085:vg_nvme/lv_1 2021-02-10T04:24:03.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:02 smithi085 conmon[29319]: cluster 2021-02-10T04:24:01.926456+0000 mgr.y (mgr.14140) 110 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-02-10T04:24:03.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:02 smithi085 conmon[29319]: audit 2021-02-10T04:24:02.054893+0000 mon.a (mon.0) 335 : audit [INF] from='osd.2 v2:172.21.15.85:6809/7' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-02-10T04:24:03.386 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:02 smithi085 conmon[35961]: cluster 2021-02-10T04:24:01. 2021-02-10T04:24:03.386 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:02 smithi085 conmon[35961]: 926456+0000 mgr.y (mgr.14140) 110 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-02-10T04:24:03.386 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:02 smithi085 conmon[35961]: audit 2021-02-10T04:24:02.054893+0000 mon.a (mon.0) 335 : audit [INF] from='osd.2 v2:172.21.15.85:6809/7' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-02-10T04:24:03.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:02 smithi164 conmon[29288]: cluster 2021-02-10T04:24:01.926456+0000 mgr.y (mgr.14140) 110 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-02-10T04:24:03.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:02 smithi164 conmon[29288]: 2021-02-10T04:24:03.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:02 smithi164 conmon[29288]: audit 2021-02-10T04:24:02.054893+0000 mon.a (mon.0) 335 : audit [INF] from='osd.2 v2:172.21.15.85:6809/7' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-02-10T04:24:04.150 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:24:03 smithi085 conmon[49106]: debug 2021-02-10T04:24:03.990+0000 7f77a7f19700 -1 osd.2 0 waiting for initial osdmap 2021-02-10T04:24:04.151 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:03 smithi085 conmon[29319]: audit 2021-02-10T04:24:02.986766+0000 mon.a ( 2021-02-10T04:24:04.151 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:03 smithi085 conmon[29319]: mon.0) 336 : audit [INF] from='osd.2 v2:172.21.15.85:6809/7' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-02-10T04:24:04.151 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:03 smithi085 conmon[29319]: cluster 2021-02-10T04:24:02.986807+0000 mon.a ( 2021-02-10T04:24:04.151 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:03 smithi085 conmon[29319]: mon.0) 337 : cluster [DBG] osdmap e16: 3 total, 2 up, 2 in 2021-02-10T04:24:04.151 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:03 smithi085 conmon[29319]: audit 2021-02-10T04:24:04.151 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:03 smithi085 conmon[29319]: 2021-02-10T04:24:02.986911+0000 mon.a (mon.0) 338 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-02-10T04:24:04.151 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:03 smithi085 conmon[29319]: audit 2021-02-10T04:24:02 2021-02-10T04:24:04.151 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:03 smithi085 conmon[29319]: .987438+0000 mon.a (mon.0) 339 : audit [INF] from='osd.2 v2:172.21.15.85:6809/7' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:24:04.151 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:03 smithi085 conmon[35961]: audit 2021-02-10T04:24:02.986766+0000 mon.a ( 2021-02-10T04:24:04.151 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:03 smithi085 conmon[35961]: mon.0) 336 : audit [INF] from='osd.2 v2:172.21.15.85:6809/7' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-02-10T04:24:04.151 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:03 smithi085 conmon[35961]: cluster 2021-02-10T04:24:02.986807+0000 mon.a (mon.0) 2021-02-10T04:24:04.152 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:03 smithi085 conmon[35961]: 337 : cluster [DBG] osdmap e16: 3 total, 2 up, 2 in 2021-02-10T04:24:04.152 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:03 smithi085 conmon[35961]: audit 2021 2021-02-10T04:24:04.152 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:03 smithi085 conmon[35961]: -02-10T04:24:02.986911+0000 mon.a (mon.0) 2021-02-10T04:24:04.152 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:03 smithi085 conmon[35961]: 338 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-02-10T04:24:04.152 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:03 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:24:04.152 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:03 smithi085 conmon[35961]: 04:24:02.987438+0000 mon.a (mon. 2021-02-10T04:24:04.152 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:03 smithi085 conmon[35961]: 0) 339 : audit [INF] from='osd.2 v2:172.21.15.85:6809/7' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:24:04.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:03 smithi164 conmon[29288]: audit 2021-02-10T04:24:02.986766+0000 mon.a (mon. 2021-02-10T04:24:04.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:03 smithi164 conmon[29288]: 0) 336 : audit [INF] from='osd.2 v2:172.21.15.85:6809/7' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-02-10T04:24:04.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:03 smithi164 conmon[29288]: cluster 2021-02-10T04:24:02.986807+0000 mon.a (mon.0) 337 : cluster [DBG] osdmap e16: 3 total, 2 up, 2 in 2021-02-10T04:24:04.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:03 smithi164 conmon[29288]: audit 2021-02-10T04:24:02.986911+0000 mon.a (mon.0) 338 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-02-10T04:24:04.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:03 smithi164 conmon[29288]: audit 2021-02-10T04:24:02.987438+0000 mon.a (mon.0) 339 : 2021-02-10T04:24:04.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:03 smithi164 conmon[29288]: audit [INF] from='osd.2 v2:172.21.15.85:6809/7' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:24:05.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:04 smithi164 conmon[29288]: cluster 2021-02-10T04:24:03.926745+0000 mgr.y (mgr.14140 2021-02-10T04:24:05.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:04 smithi164 conmon[29288]: ) 111 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-02-10T04:24:05.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:04 smithi164 conmon[29288]: audit 2021-02-10T04:24:03.987914+0000 mon.a (mon.0) 340 : audit [INF] from='osd.2 v2:172.21.15.85:6809/7' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]': finished 2021-02-10T04:24:05.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:04 smithi164 conmon[29288]: cluster 2021-02-10T04:24:03.987974+0000 mon.a (mon.0) 341 : cluster [DBG] osdmap e17: 3 total, 2 up, 2 in 2021-02-10T04:24:05.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:04 smithi164 conmon[29288]: audit 2021-02-10T04:24:03.988334+0000 mon.a (mon.0) 342 : audit [DBG] 2021-02-10T04:24:05.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:04 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-02-10T04:24:05.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:04 smithi164 conmon[29288]: audit 2021-02-10T04:24:03. 2021-02-10T04:24:05.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:04 smithi164 conmon[29288]: 991205+0000 mon.a (mon.0) 343 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-02-10T04:24:05.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:04 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:24:05.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:04 smithi164 conmon[29288]: 02-10T04:24:04.667232+0000 mon.a (mon.0) 344 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:24:05.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:04 smithi164 conmon[29288]: audit 2021-02-10T04:24:04.991261+0000 mon.a (mon.0) 345 : 2021-02-10T04:24:05.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:04 smithi164 conmon[29288]: audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-02-10T04:24:05.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:04 smithi085 conmon[29319]: cluster 2021-02-10T04:24:03.926745+0000 mgr.y (mgr.14140) 111 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-02-10T04:24:05.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:04 smithi085 conmon[29319]: audit 2021-02-10T04:24:03.987914+0000 mon.a (mon.0) 340 : audit [INF] from='osd.2 v2:172.21.15.85:6809/7' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]': finished 2021-02-10T04:24:05.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:04 smithi085 conmon[29319]: cluster 2021-02-10T04:24:03.987974+0000 mon.a (mon.0) 341 : cluster [DBG] osdmap e17: 3 total, 2 up, 2 in 2021-02-10T04:24:05.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:04 smithi085 conmon[29319]: audit 2021-02-10T04:24:03.988334+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-02-10T04:24:05.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:04 smithi085 conmon[29319]: audit 2021-02-10T04:24:03.991205+0000 mon.a (mon.0) 343 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-02-10T04:24:05.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:04 smithi085 conmon[29319]: audit 2021-02-10T04:24:04.667232+0000 mon.a (mon.0) 344 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:24:05.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:04 smithi085 conmon[29319]: audit 2021-02-10T04:24:04.991261+0000 mon.a (mon.0) 345 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-02-10T04:24:05.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:04 smithi085 conmon[35961]: cluster 2021-02-10T04:24:03.926745+0000 mgr.y (mgr.14140 2021-02-10T04:24:05.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:04 smithi085 conmon[35961]: ) 111 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-02-10T04:24:05.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:04 smithi085 conmon[35961]: audit 2021-02-10T04:24:03.987914+0000 mon.a (mon.0) 340 : audit [INF] from='osd.2 v2:172.21.15.85:6809/7' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]': finished 2021-02-10T04:24:05.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:04 smithi085 conmon[35961]: cluster 2021-02-10T04:24:03.987974+0000 mon.a ( 2021-02-10T04:24:05.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:04 smithi085 conmon[35961]: mon.0) 341 : cluster [DBG] osdmap e17: 3 total, 2 up, 2 in 2021-02-10T04:24:05.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:04 smithi085 conmon[35961]: audit 2021 2021-02-10T04:24:05.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:04 smithi085 conmon[35961]: -02-10T04:24:03.988334 2021-02-10T04:24:05.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:04 smithi085 conmon[35961]: +0000 mon.a (mon.0) 342 2021-02-10T04:24:05.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:04 smithi085 conmon[35961]: : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-02-10T04:24:05.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:04 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:24:05.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:04 smithi085 conmon[35961]: 04:24:03.991205+0000 mon.a (mon.0) 343 : audit [DBG] 2021-02-10T04:24:05.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:04 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-02-10T04:24:05.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:04 smithi085 conmon[35961]: audit 2021-02-10T04:24 2021-02-10T04:24:05.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:04 smithi085 conmon[35961]: :04.667232+0000 mon.a (mon.0) 344 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:24:05.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:04 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:24:05.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:04 smithi085 conmon[35961]: 24:04.991261+0000 mon.a (mon.0) 2021-02-10T04:24:05.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:04 smithi085 conmon[35961]: 345 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-02-10T04:24:06.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:06 smithi164 conmon[29288]: cluster 2021-02-10T04:24:03.013103+0000 osd.2 (osd.2) 1 2021-02-10T04:24:06.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:06 smithi164 conmon[29288]: : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:24:06.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:06 smithi164 conmon[29288]: cluster 2021-02-10T04:24:03.013186+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:24:06.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:06 smithi164 conmon[29288]: cluster 2021-02-10T04:24:05.004300+0000 mon.a (mon.0) 346 : cluster [INF] osd.2 v2:172.21.15.85:6809/7 boot 2021-02-10T04:24:06.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:06 smithi164 conmon[29288]: cluster 2021-02-10T04:24:05.004333+0000 mon.a (mon.0) 347 : cluster 2021-02-10T04:24:06.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:06 smithi164 conmon[29288]: [DBG] osdmap e18: 3 total, 3 up, 3 in 2021-02-10T04:24:06.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:06 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:24:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:06 smithi164 conmon[29288]: 10T04:24:05.004659+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-02-10T04:24:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:06 smithi164 conmon[29288]: audit 2021-02-10T04:24:05.004965+0000 mon.a (mon 2021-02-10T04:24:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:06 smithi164 conmon[29288]: .0) 349 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2021-02-10T04:24:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:06 smithi164 conmon[29288]: cephadm 2021-02-10T04:24:05.018512+0000 mgr.y (mgr.14140) 112 : cephadm 2021-02-10T04:24:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:06 smithi164 conmon[29288]: [INF] Detected new or changed devices on smithi085 2021-02-10T04:24:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:06 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:24:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:06 smithi164 conmon[29288]: 02-10T04:24:05.261837+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:06 smithi164 conmon[29288]: audit 2021-02-10T04:24:05.614389+0000 2021-02-10T04:24:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:06 smithi164 conmon[29288]: mon.a (mon.0) 351 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:06 smithi164 conmon[29288]: audit 2021-02-10T04:24:05.618146+0000 2021-02-10T04:24:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:06 smithi164 conmon[29288]: mon.a (mon.0) 352 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:06.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[29319]: cluster 2021-02-10T04:24:03.013103+0000 2021-02-10T04:24:06.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[29319]: osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:24:06.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:24:06.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[29319]: -10T04:24:03.013186+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:24:06.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[29319]: cluster 2021-02-10T04:24:06.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[29319]: 2021-02-10T04:24:05.004300+0000 mon.a ( 2021-02-10T04:24:06.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[29319]: mon.0) 346 : cluster [INF] osd.2 v2:172.21.15.85:6809/7 boot 2021-02-10T04:24:06.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:24:06.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[29319]: -02-10T04:24:05. 2021-02-10T04:24:06.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[29319]: 004333+0000 2021-02-10T04:24:06.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[29319]: mon.a (mon.0) 347 : cluster 2021-02-10T04:24:06.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[29319]: [DBG] osdmap e18: 3 total, 3 up, 3 in 2021-02-10T04:24:06.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:24:06.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[29319]: 10T04:24:05.004659+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-02-10T04:24:06.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:24:06.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[29319]: 10T04:24:05.004965 2021-02-10T04:24:06.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[29319]: +0000 mon.a (mon.0) 349 2021-02-10T04:24:06.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[29319]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2021-02-10T04:24:06.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[29319]: cephadm 2021- 2021-02-10T04:24:06.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[29319]: 02-10T04:24:05. 2021-02-10T04:24:06.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[29319]: 018512+0000 mgr.y (mgr.14140 2021-02-10T04:24:06.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[29319]: ) 112 : cephadm [INF] Detected new or changed devices on smithi085 2021-02-10T04:24:06.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:24:06.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[29319]: 10T04:24:05.261837+0000 mon.a (mon.0) 350 : audit 2021-02-10T04:24:06.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[29319]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:06.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[29319]: audit 2021-02-10T04:24:06.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[29319]: 2021-02-10T04:24:05.614389+0000 mon.a (mon.0) 351 2021-02-10T04:24:06.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[29319]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:06.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[29319]: audit 2021-02-10T04:24:05 2021-02-10T04:24:06.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[29319]: .618146+0000 mon.a (mon.0) 352 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:06.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: cluster 2021-02-10T04:24:03.013103 2021-02-10T04:24:06.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:24:06.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: cluster 2021-02-10T04:24:03 2021-02-10T04:24:06.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: .013186+0000 osd.2 (osd.2) 2 : cluster 2021-02-10T04:24:06.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: [DBG] purged_snaps scrub ok 2021-02-10T04:24:06.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:24:06.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: -02-10T04:24:05.004300+0000 mon.a ( 2021-02-10T04:24:06.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: mon.0) 346 : cluster [INF] osd.2 v2:172.21.15.85:6809/7 boot 2021-02-10T04:24:06.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: 2021-02-10T04:24:06.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:24:06.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: -10T04:24:05.004333+0000 mon.a (mon.0 2021-02-10T04:24:06.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: ) 347 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2021-02-10T04:24:06.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: audit 2021-02-10T04:24:06.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: 2021-02 2021-02-10T04:24:06.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: -10T04:24:05.004659+0000 mon.a (mon 2021-02-10T04:24:06.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: .0) 348 : audit [DBG] 2021-02-10T04:24:06.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-02-10T04:24:06.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:24:06.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: -10T04:24:05.004965+0000 mon.a (mon.0) 2021-02-10T04:24:06.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: 349 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2021-02-10T04:24:06.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: cephadm 2021-02-10T04:24:05 2021-02-10T04:24:06.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: .018512+0000 mgr.y (mgr.14140) 2021-02-10T04:24:06.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: 112 : cephadm [INF] Detected new or changed devices on smithi085 2021-02-10T04:24:06.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:24:06.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: -10T04:24:05.261837+0000 mon.a ( 2021-02-10T04:24:06.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: mon.0) 350 : audit [INF] 2021-02-10T04:24:06.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:06.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: audit 2021-02-10T04:24:06.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: 2021-02-10T04:24: 2021-02-10T04:24:06.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: 05.614389+0000 mon.a (mon. 2021-02-10T04:24:06.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: 0) 351 : audit [INF] 2021-02-10T04:24:06.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:06.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:24:06.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: 02-10T04:24:05.618146 2021-02-10T04:24:06.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: +0000 mon.a (mon.0) 352 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:06.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:06 smithi085 conmon[35961]: 2021-02-10T04:24:07.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:07 smithi164 conmon[29288]: audit 2021-02-10T04:24:05.207772+0000 mgr.y (mgr.14140) 113 : audit 2021-02-10T04:24:07.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:07 smithi164 conmon[29288]: [DBG] from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi085:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:24:07.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:07 smithi164 conmon[29288]: cephadm 2021-02-10T04:24:05.262325+0000 mgr.y (mgr.14140) 114 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:24:07.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:07 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:24:07.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:07 smithi164 conmon[29288]: 10T04:24:05.927110+0000 mgr.y (mgr.14140) 115 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-02-10T04:24:07.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:07 smithi164 conmon[29288]: audit 2021-02-10T04:24: 2021-02-10T04:24:07.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:07 smithi164 conmon[29288]: 06.011654+0000 mon.a (mon.0) 353 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2021-02-10T04:24:07.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:07 smithi164 conmon[29288]: cluster 2021-02-10T04:24:07.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:07 smithi164 conmon[29288]: 2021-02-10T04:24:06.011777+0000 mon.a (mon.0) 354 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2021-02-10T04:24:07.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:07 smithi164 conmon[29288]: audit 2021-02-10T04:24:06.012418+0000 2021-02-10T04:24:07.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:07 smithi164 conmon[29288]: mon.a (mon.0) 355 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2021-02-10T04:24:07.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[29319]: audit 2021-02-10T04:24:05.207772+0000 mgr.y (mgr.14140) 113 : audit [DBG] 2021-02-10T04:24:07.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[29319]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi085:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:24:07.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[29319]: cephadm 2021-02-10T04:24:05.262325+0000 mgr.y (mgr.14140) 114 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:24:07.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[29319]: cluster 2021-02-10T04:24:05 2021-02-10T04:24:07.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[29319]: .927110+0000 mgr.y (mgr.14140) 115 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-02-10T04:24:07.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[29319]: audit 2021-02-10T 2021-02-10T04:24:07.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[29319]: 04:24:06.011654+0000 mon.a (mon.0) 353 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2021-02-10T04:24:07.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:24:07.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[29319]: 02-10T04:24:06.011777+0000 2021-02-10T04:24:07.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[29319]: mon.a (mon.0) 354 : cluster 2021-02-10T04:24:07.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[29319]: [DBG] osdmap e19: 3 total, 3 up, 3 in 2021-02-10T04:24:07.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[29319]: audit 2021-02-10T04:24:06.012418+0000 2021-02-10T04:24:07.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[29319]: mon.a (mon.0) 355 : audit [INF] 2021-02-10T04:24:07.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2021-02-10T04:24:07.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[35961]: audit 2021-02-10T04:24:05.207772+0000 2021-02-10T04:24:07.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[35961]: mgr.y (mgr.14140) 113 : audit [DBG] from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi085:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:24:07.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[35961]: cephadm 2021-02-10T04:24:07.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[35961]: 2021-02-10T04:24:05.262325+0000 mgr.y (mgr.14140 2021-02-10T04:24:07.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[35961]: ) 114 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:24:07.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:24:07.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[35961]: 02-10T04:24:05.927110+0000 2021-02-10T04:24:07.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[35961]: mgr.y (mgr.14140) 115 : cluster 2021-02-10T04:24:07.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[35961]: [DBG] pgmap v82: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-02-10T04:24:07.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:24:07.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[35961]: 02-10T04:24:06.011654 2021-02-10T04:24:07.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[35961]: +0000 mon.a (mon.0) 353 2021-02-10T04:24:07.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[35961]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2021-02-10T04:24:07.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:24:07.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[35961]: -02-10T04:24:06 2021-02-10T04:24:07.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[35961]: .011777+0000 mon.a (mon. 2021-02-10T04:24:07.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[35961]: 0) 354 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2021-02-10T04:24:07.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[35961]: 2021-02-10T04:24:07.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[35961]: audit 2021 2021-02-10T04:24:07.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[35961]: -02-10T04:24:06 2021-02-10T04:24:07.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[35961]: .012418+0000 mon.a (mon.0 2021-02-10T04:24:07.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:07 smithi085 conmon[35961]: ) 355 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2021-02-10T04:24:08.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[29319]: audit 2021-02-10T04:24:07 2021-02-10T04:24:08.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[29319]: .015978+0000 mon.a (mon.0) 356 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2021-02-10T04:24:08.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[29319]: cluster 2021-02-10T04:24:07.016079+0000 mon.a (mon.0) 357 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2021-02-10T04:24:08.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[29319]: audit 2021-02-10T04:24:07.018901+0000 mon.a (mon.0) 358 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-02-10T04:24:08.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[29319]: 2021-02-10T04:24:08.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[29319]: cephadm 2021 2021-02-10T04:24:08.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[29319]: -02-10T04:24:07.020414+0000 mgr.y (mgr.14140) 116 2021-02-10T04:24:08.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[29319]: : cephadm [INF] Applying service osd.None on host smithi085... 2021-02-10T04:24:08.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[29319]: audit 2021 2021-02-10T04:24:08.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[29319]: -02-10T04:24:07.023184+0000 mon.a 2021-02-10T04:24:08.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[29319]: (mon.0) 359 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-02-10T04:24:08.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[29319]: audit 2021-02-10T04:24 2021-02-10T04:24:08.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[29319]: :07.023946+0000 mon.a (mon.0) 360 : audit 2021-02-10T04:24:08.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[29319]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:24:08.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:24:08.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[29319]: 10T04:24:07.024822+0000 mon.a ( 2021-02-10T04:24:08.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[29319]: mon.0) 361 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:24:08.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[35961]: audit 2021-02-10T04:24: 2021-02-10T04:24:08.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[35961]: 07.015978+0000 mon.a (mon.0) 356 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2021-02-10T04:24:08.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:24:08.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[35961]: 02-10T04:24:07.016079+0000 mon.a 2021-02-10T04:24:08.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[35961]: (mon.0) 357 : cluster [DBG] 2021-02-10T04:24:08.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[35961]: osdmap e20: 3 total, 3 up, 3 in 2021-02-10T04:24:08.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:24:08.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[35961]: -10T04:24:07.018901+0000 mon.a (mon. 2021-02-10T04:24:08.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[35961]: 0) 358 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-02-10T04:24:08.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[35961]: cephadm 2021-02-10T04: 2021-02-10T04:24:08.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[35961]: 24:07.020414+0000 mgr.y (mgr.14140) 2021-02-10T04:24:08.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[35961]: 116 : cephadm [INF] Applying service osd.None on host smithi085... 2021-02-10T04:24:08.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:24:08.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[35961]: 02-10T04:24:07.023184 2021-02-10T04:24:08.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[35961]: +0000 mon.a (mon.0) 359 2021-02-10T04:24:08.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[35961]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-02-10T04:24:08.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:24:08.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[35961]: 02-10T04:24:07 2021-02-10T04:24:08.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[35961]: .023946+0000 mon.a (mon.0) 360 : audit 2021-02-10T04:24:08.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[35961]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:24:08.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:24:08.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[35961]: -10T04:24:07. 2021-02-10T04:24:08.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[35961]: 024822+0000 mon.a (mon.0 2021-02-10T04:24:08.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[35961]: ) 361 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:24:08.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:08 smithi085 conmon[35961]: 2021-02-10T04:24:08.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:08 smithi164 conmon[29288]: audit 2021-02-10T04:24:07.015978+0000 mon.a ( 2021-02-10T04:24:08.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:08 smithi164 conmon[29288]: mon.0) 356 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2021-02-10T04:24:08.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:08 smithi164 conmon[29288]: cluster 2021-02-10T04:24:07.016079+0000 mon.a (mon.0) 357 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2021-02-10T04:24:08.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:08 smithi164 conmon[29288]: audit 2021-02-10T04:24:07.018901+0000 mon.a (mon.0) 358 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-02-10T04:24:08.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:08 smithi164 conmon[29288]: cephadm 2021-02-10T04:24: 2021-02-10T04:24:08.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:08 smithi164 conmon[29288]: 07.020414+0000 mgr.y (mgr.14140) 116 : cephadm [INF] Applying service osd.None on host smithi085... 2021-02-10T04:24:08.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:08 smithi164 conmon[29288]: audit 2021-02-10T04:24:07.023184 2021-02-10T04:24:08.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:08 smithi164 conmon[29288]: +0000 mon.a (mon.0) 359 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-02-10T04:24:08.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:08 smithi164 conmon[29288]: audit 2021-02-10T04:24:08.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:08 smithi164 conmon[29288]: 2021-02-10T04:24:07.023946+0000 mon.a (mon.0 2021-02-10T04:24:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:08 smithi164 conmon[29288]: ) 360 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:24:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:08 smithi164 conmon[29288]: audit 2021-02-10T04:24:07.024822 2021-02-10T04:24:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:08 smithi164 conmon[29288]: +0000 mon.a (mon.0) 361 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:24:09.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:09 smithi164 conmon[29288]: cluster 2021-02-10T04:24:07.927429+0000 mgr.y (mgr.14140) 117 2021-02-10T04:24:09.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:09 smithi164 conmon[29288]: : cluster [DBG] pgmap v85: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-02-10T04:24:09.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:09 smithi164 conmon[29288]: cluster 2021-02-10T04:24:08.032118+0000 mon.a (mon.0) 362 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2021-02-10T04:24:09.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:09 smithi085 conmon[29319]: cluster 2021-02-10T04:24:07.927429+0000 mgr.y (mgr.14140) 117 : cluster [DBG] pgmap v85: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-02-10T04:24:09.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:09 smithi085 conmon[29319]: cluster 2021-02-10T04:24:08.032118+0000 mon.a (mon.0) 362 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2021-02-10T04:24:09.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:09 smithi085 conmon[35961]: cluster 2021-02-10T04:24:07.927429+0000 mgr.y (mgr.14140) 117 : cluster [DBG] pgmap v85: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-02-10T04:24:09.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:09 smithi085 conmon[35961]: cluster 2021-02-10T04:24:08.032118+0000 mon.a (mon.0) 362 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2021-02-10T04:24:10.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:10 smithi164 conmon[29288]: audit 2021-02-10T04:24:09.364446+0000 mon.a (mon. 2021-02-10T04:24:10.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:10 smithi164 conmon[29288]: 0) 363 : audit [INF] from='client.? 172.21.15.85:0/2054536809' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e019f49f-5dd4-4104-9ced-94c710616642"}]: dispatch 2021-02-10T04:24:10.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:10 smithi164 conmon[29288]: audit 2021-02-10T04:24:09.371036+0000 mon.a (mon.0) 364 : audit [INF] from='client.? 172.21.15.85:0/2054536809' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e019f49f-5dd4-4104-9ced-94c710616642"}]': finished 2021-02-10T04:24:10.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:10 smithi164 conmon[29288]: cluster 2021-02-10T04:24:09.371078+0000 mon.a (mon.0) 365 : cluster [DBG] osdmap e22: 4 total, 3 up, 3 in 2021-02-10T04:24:10.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:10 smithi164 conmon[29288]: audit 2021-02-10T04:24:09.371192+0000 mon.a (mon.0) 366 2021-02-10T04:24:10.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:10 smithi164 conmon[29288]: : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-02-10T04:24:10.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:10 smithi164 conmon[29288]: audit 2021-02-10T04: 2021-02-10T04:24:10.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:10 smithi164 conmon[29288]: 24:09.873183+0000 mon.c (mon.1) 9 : audit [DBG] from='client.? 172.21.15.85:0/4227748669' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-02-10T04:24:10.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:10 smithi085 conmon[29319]: audit 2021-02-10T04:24:09.364446 2021-02-10T04:24:10.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:10 smithi085 conmon[29319]: +0000 mon.a (mon.0) 363 : audit [INF] from='client.? 172.21.15.85:0/2054536809' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e019f49f-5dd4-4104-9ced-94c710616642"}]: dispatch 2021-02-10T04:24:10.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:10 smithi085 conmon[29319]: audit 2021-02-10T04:24:09.371036+0000 mon.a (mon.0) 364 : audit [INF] from='client.? 172.21.15.85:0/2054536809' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e019f49f-5dd4-4104-9ced-94c710616642"}]': finished 2021-02-10T04:24:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:10 smithi085 conmon[29319]: cluster 2021-02-10T04:24:09.371078+0000 mon.a (mon.0) 365 : cluster [DBG] osdmap e22: 4 total, 3 up, 3 in 2021-02-10T04:24:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:10 smithi085 conmon[29319]: audit 2021-02-10T04:24:09.371192+0000 mon.a (mon.0) 366 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-02-10T04:24:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:10 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:24:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:10 smithi085 conmon[29319]: -10T04:24:09.873183+0000 mon.c (mon.1) 9 : audit [DBG] from='client.? 172.21.15.85:0/4227748669' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-02-10T04:24:10.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:10 smithi085 conmon[35961]: audit 2021-02-10T04:24:09.364446+0000 2021-02-10T04:24:10.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:10 smithi085 conmon[35961]: mon.a (mon.0) 363 : audit [INF] from='client.? 172.21.15.85:0/2054536809' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e019f49f-5dd4-4104-9ced-94c710616642"}]: dispatch 2021-02-10T04:24:10.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:10 smithi085 conmon[35961]: audit 2021-02-10T04:24:09. 2021-02-10T04:24:10.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:10 smithi085 conmon[35961]: 371036+0000 mon.a (mon.0) 364 : audit [INF] from='client.? 172.21.15.85:0/2054536809' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e019f49f-5dd4-4104-9ced-94c710616642"}]': finished 2021-02-10T04:24:10.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:10 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:24:10.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:10 smithi085 conmon[35961]: 04:24:09.371078+0000 mon.a (mon.0 2021-02-10T04:24:10.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:10 smithi085 conmon[35961]: ) 365 : cluster [DBG] osdmap e22: 4 total, 3 up, 3 in 2021-02-10T04:24:10.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:10 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:24:10.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:10 smithi085 conmon[35961]: -10T04:24:09.371192+0000 mon.a 2021-02-10T04:24:10.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:10 smithi085 conmon[35961]: (mon.0) 366 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-02-10T04:24:10.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:10 smithi085 conmon[35961]: 2021-02-10T04:24:10.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:10 smithi085 conmon[35961]: audit 2021-02-10T04:24:10.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:10 smithi085 conmon[35961]: 2021-02-10T04:24:09.873183+0000 mon.c ( 2021-02-10T04:24:10.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:10 smithi085 conmon[35961]: mon.1) 9 : audit [DBG] from='client.? 172.21.15.85:0/4227748669' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-02-10T04:24:11.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:11 smithi164 conmon[29288]: cluster 2021-02-10T04:24:09.927701+0000 mgr.y 2021-02-10T04:24:11.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:11 smithi164 conmon[29288]: (mgr.14140) 118 : cluster [DBG] pgmap v88: 1 pgs: 1 active+clean; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-02-10T04:24:11.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:11 smithi085 conmon[29319]: cluster 2021-02-10T04:24:09.927701+0000 2021-02-10T04:24:11.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:11 smithi085 conmon[29319]: mgr.y (mgr.14140) 118 : cluster [DBG] pgmap v88: 1 pgs: 1 active+clean; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-02-10T04:24:11.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:11 smithi085 conmon[35961]: cluster 2021-02-10T04:24:09.927701 2021-02-10T04:24:11.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:11 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 118 : cluster [DBG] pgmap v88: 1 pgs: 1 active+clean; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-02-10T04:24:13.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:13 smithi085 conmon[35961]: cluster 2021-02-10T04:24:11.928069 2021-02-10T04:24:13.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:13 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 119 : cluster [DBG] pgmap v89: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-02-10T04:24:13.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:13 smithi085 conmon[35961]: audit 2021-02-10T04:24:12.769220+0000 mon.a (mon.0) 367 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:24:13.301 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:13 smithi085 conmon[29319]: cluster 2021-02-10T04:24:11.928069+0000 mgr.y (mgr.14140) 2021-02-10T04:24:13.301 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:13 smithi085 conmon[29319]: 119 : cluster [DBG] pgmap v89: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-02-10T04:24:13.302 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:13 smithi085 conmon[29319]: audit 2021-02-10T04:24:12.769220+0000 mon.a (mon.0) 367 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:24:13.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:13 smithi164 conmon[29288]: cluster 2021-02-10T04:24:11.928069+0000 mgr.y (mgr.14140 2021-02-10T04:24:13.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:13 smithi164 conmon[29288]: ) 119 : cluster [DBG] pgmap v89: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-02-10T04:24:13.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:13 smithi164 conmon[29288]: audit 2021-02-10T04:24:12.769220+0000 mon.a (mon.0) 367 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:24:15.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:15 smithi085 conmon[29319]: cluster 2021-02-10T04:24:13.928573+0000 mgr.y (mgr.14140) 120 : cluster [DBG] pgmap v90: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-02-10T04:24:15.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:15 smithi085 conmon[35961]: cluster 2021-02-10T04:24:13.928573+0000 mgr.y (mgr.14140) 120 : cluster [DBG] pgmap v90: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-02-10T04:24:15.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:15 smithi164 conmon[29288]: cluster 2021-02-10T04:24:13.928573+0000 mgr.y (mgr.14140 2021-02-10T04:24:15.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:15 smithi164 conmon[29288]: ) 120 : cluster [DBG] pgmap v90: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-02-10T04:24:16.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:16 smithi085 conmon[29319]: audit 2021-02-10T04:24:15.386353+0000 mon.a (mon. 2021-02-10T04:24:16.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:16 smithi085 conmon[29319]: 0) 368 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2021-02-10T04:24:16.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:16 smithi085 conmon[29319]: audit 2021-02-10T04:24:15.386909+0000 mon.a (mon.0) 369 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:24:16.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:16 smithi085 conmon[29319]: audit 2021-02-10T04:24:15.387622+0000 mon.a (mon.0) 370 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.3", "key": "container_image"}]: dispatch 2021-02-10T04:24:16.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:16 smithi085 conmon[35961]: audit 2021-02-10T04:24:15.386353+0000 2021-02-10T04:24:16.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:16 smithi085 conmon[35961]: mon.a (mon.0) 368 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2021-02-10T04:24:16.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:16 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:24:16.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:16 smithi085 conmon[35961]: 10T04:24:15.386909+0000 mon.a (mon.0) 369 : 2021-02-10T04:24:16.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:16 smithi085 conmon[35961]: audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:24:16.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:16 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:24:16.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:16 smithi085 conmon[35961]: -10T04:24:15.387622 2021-02-10T04:24:16.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:16 smithi085 conmon[35961]: +0000 mon.a (mon.0) 370 : audit 2021-02-10T04:24:16.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:16 smithi085 conmon[35961]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.3", "key": "container_image"}]: dispatch 2021-02-10T04:24:16.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:16 smithi164 conmon[29288]: audit 2021-02-10T04:24:15.386353+0000 2021-02-10T04:24:16.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:16 smithi164 conmon[29288]: mon.a (mon.0) 368 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2021-02-10T04:24:16.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:16 smithi164 conmon[29288]: audit 2021-02-10T04:24:15.386909+0000 mon.a (mon 2021-02-10T04:24:16.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:16 smithi164 conmon[29288]: .0) 369 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:24:16.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:16 smithi164 conmon[29288]: audit 2021-02-10T04:24:15. 2021-02-10T04:24:16.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:16 smithi164 conmon[29288]: 387622+0000 mon.a (mon.0) 370 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.3", "key": "container_image"}]: dispatch 2021-02-10T04:24:17.047 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:17 smithi085 conmon[35961]: cephadm 2021-02-10T04:24:15.387311 2021-02-10T04:24:17.362 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:17 smithi085 conmon[29319]: cephadm 2021-02-10T04:24:15.387311 2021-02-10T04:24:17.363 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:17 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 121 : cephadm [INF] Deploying daemon osd.3 on smithi085 2021-02-10T04:24:17.363 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:17 smithi085 conmon[29319]: cluster 2021-02-10T04:24:15.928870+0000 mgr.y (mgr.14140) 122 : cluster [DBG] pgmap v91: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-02-10T04:24:17.363 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:17 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 121 : cephadm [INF] Deploying daemon osd.3 on smithi085 2021-02-10T04:24:17.363 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:17 smithi085 conmon[35961]: cluster 2021-02-10T04:24:15.928870+0000 mgr.y (mgr.14140) 122 : cluster [DBG] pgmap v91: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-02-10T04:24:17.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:17 smithi164 conmon[29288]: cephadm 2021-02-10T04:24:15.387311+0000 mgr.y ( 2021-02-10T04:24:17.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:17 smithi164 conmon[29288]: mgr.14140) 121 : cephadm [INF] Deploying daemon osd.3 on smithi085 2021-02-10T04:24:17.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:17 smithi164 conmon[29288]: cluster 2021-02-10T04:24:15.928870+0000 mgr.y (mgr.14140) 122 : cluster [DBG] pgmap v91: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-02-10T04:24:18.815 INFO:teuthology.orchestra.run.smithi085.stdout:Created osd(s) 3 on host 'smithi085' 2021-02-10T04:24:19.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:18 smithi085 conmon[29319]: cluster 2021-02-10T04:24:19.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:18 smithi085 conmon[29319]: 2021-02-10T04:24:17.929130 2021-02-10T04:24:19.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:18 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 123 : cluster [DBG] pgmap v92: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-02-10T04:24:19.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:18 smithi085 conmon[29319]: audit 2021-02-10T04:24:18.806952 2021-02-10T04:24:19.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:18 smithi085 conmon[29319]: +0000 mon.a (mon.0) 371 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:24:19.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:18 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:24:19.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:18 smithi085 conmon[29319]: T04:24:18.809347+0000 mon.a (mon.0) 2021-02-10T04:24:19.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:18 smithi085 conmon[29319]: 372 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:19.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:18 smithi085 conmon[29319]: audit 2021-02-10T04:24:18.813463 2021-02-10T04:24:19.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:18 smithi085 conmon[29319]: +0000 mon.a (mon.0) 373 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:19.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:18 smithi085 conmon[35961]: cluster 2021-02-10T04:24:17.929130 2021-02-10T04:24:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:18 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 123 : cluster [DBG] pgmap v92: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-02-10T04:24:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:18 smithi085 conmon[35961]: audit 2021-02-10T04:24:18.806952 2021-02-10T04:24:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:18 smithi085 conmon[35961]: +0000 mon.a (mon.0) 371 : audit [DBG] 2021-02-10T04:24:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:18 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:24:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:18 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:24:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:18 smithi085 conmon[35961]: 04:24:18.809347+0000 mon.a (mon.0 2021-02-10T04:24:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:18 smithi085 conmon[35961]: ) 372 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:18 smithi085 conmon[35961]: audit 2021-02-10T04:24:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:18 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:24:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:18 smithi085 conmon[35961]: 24:18.813463+0000 mon.a (mon.0) 2021-02-10T04:24:19.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:18 smithi085 conmon[35961]: 373 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:19.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:18 smithi164 conmon[29288]: cluster 2021-02-10T04:24:17.929130+0000 mgr.y (mgr.14140 2021-02-10T04:24:19.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:18 smithi164 conmon[29288]: ) 123 : cluster [DBG] pgmap v92: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-02-10T04:24:19.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:18 smithi164 conmon[29288]: audit 2021-02-10T04:24:18.806952+0000 mon.a (mon.0) 371 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:24:19.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:18 smithi164 conmon[29288]: audit 2021-02-10T04:24:19.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:18 smithi164 conmon[29288]: 2021-02-10T04:24:18.809347+0000 mon.a (mon.0) 372 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:19.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:18 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:24:19.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:18 smithi164 conmon[29288]: 04:24:18.813463+0000 mon.a (mon.0) 373 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:19.510 DEBUG:teuthology.orchestra.run.smithi085:osd.3> sudo journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.3.service 2021-02-10T04:24:19.512 INFO:tasks.cephadm:Deploying osd.4 on smithi164 with /dev/vg_nvme/lv_4... 2021-02-10T04:24:19.512 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2021-02-10T04:24:19.540 INFO:journalctl@ceph.osd.3.smithi085.stdout:-- Logs begin at Wed 2021-02-10 04:09:23 UTC. -- 2021-02-10T04:24:20.984 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:20 smithi085 conmon[29319]: cluster 2021-02-10T04:24:19.929414+0000 mgr.y (mgr.14140) 124 : cluster [DBG] pgmap v93: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-02-10T04:24:20.984 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:20 smithi085 conmon[29319]: audit 2021-02-10T04:24:20.267057+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:20.984 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:20 smithi085 conmon[29319]: audit 2021-02-10T04:24:20.267828+0000 mon.a (mon.0) 375 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:24:21.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:20 smithi164 conmon[29288]: cluster 2021-02-10T04:24:19.929414+0000 mgr.y (mgr.14140) 124 : cluster [DBG] pgmap v93: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-02-10T04:24:21.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:20 smithi164 conmon[29288]: audit 2021-02-10T04:24:20.267057+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:21.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:20 smithi164 conmon[29288]: audit 2021-02-10T04:24:20.267828+0000 mon.a (mon.0) 375 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:24:21.243 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:20 smithi085 conmon[35961]: cluster 2021-02-10T04:24:19.929414+0000 mgr.y (mgr.14140) 124 : cluster [DBG] pgmap v93: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-02-10T04:24:21.244 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:20 smithi085 conmon[35961]: audit 2021-02-10T04:24:20.267057+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:21.244 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:20 smithi085 conmon[35961]: audit 2021-02-10T04:24:20.267828+0000 mon.a (mon.0) 375 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:24:21.375 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping: /dev/vg_nvme/lv_4 2021-02-10T04:24:21.375 INFO:teuthology.orchestra.run.smithi164.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2021-02-10T04:24:21.376 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10+0 records in 2021-02-10T04:24:21.376 INFO:teuthology.orchestra.run.smithi164.stderr:10+0 records out 2021-02-10T04:24:21.376 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109566 s, 957 MB/s 2021-02-10T04:24:21.376 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping successful for: 2021-02-10T04:24:21.932 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph orch daemon add osd smithi164:vg_nvme/lv_4 2021-02-10T04:24:22.309 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:21 smithi164 conmon[29288]: audit 2021-02-10T04:24:21.962066+0000 mon.c (mon.1) 10 : audit [INF] from='osd.3 v2:172.21.15.85:6813/7' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-02-10T04:24:22.309 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:21 smithi164 conmon[29288]: audit 2021-02-10T04:24:21.962400+0000 mon.a (mon.0) 376 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-02-10T04:24:22.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:21 smithi085 conmon[35961]: audit 2021-02-10T04:24:21.962066 2021-02-10T04:24:22.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:21 smithi085 conmon[35961]: +0000 mon.c (mon.1) 10 : audit [INF] from='osd.3 v2:172.21.15.85:6813/7' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-02-10T04:24:22.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:21 smithi085 conmon[35961]: audit 2021-02-10T04:24:21.962400+0000 2021-02-10T04:24:22.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:21 smithi085 conmon[35961]: mon.a (mon.0) 376 : audit [INF] 2021-02-10T04:24:22.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:21 smithi085 conmon[35961]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-02-10T04:24:22.441 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:24:21 smithi085 conmon[52836]: debug 2021-02-10T04:24:21.957+0000 7fba9c344200 -1 osd.3 0 log_to_monitors {default=true} 2021-02-10T04:24:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:21 smithi085 conmon[29319]: audit 2021-02-10T04:24:21.962066 2021-02-10T04:24:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:21 smithi085 conmon[29319]: +0000 mon.c (mon.1) 10 : audit [INF] from='osd.3 v2:172.21.15.85:6813/7' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-02-10T04:24:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:21 smithi085 conmon[29319]: audit 2021-02-10T04:24:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:21 smithi085 conmon[29319]: 2021-02-10T04:24:21.962400+0000 mon.a ( 2021-02-10T04:24:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:21 smithi085 conmon[29319]: mon.0) 376 : audit 2021-02-10T04:24:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:21 smithi085 conmon[29319]: [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-02-10T04:24:23.318 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:22 smithi164 conmon[29288]: cluster 2021-02-10T04:24:21.929638+0000 mgr.y (mgr.14140) 125 : cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-02-10T04:24:23.318 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:22 smithi164 conmon[29288]: audit 2021-02-10T04: 2021-02-10T04:24:23.318 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:22 smithi164 conmon[29288]: 24:21.988835+0000 mon.a (mon.0) 377 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2021-02-10T04:24:23.318 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:22 smithi164 conmon[29288]: cluster 2021-02-10T04:24:21.988894+0000 mon.a (mon.0) 378 : cluster [DBG] osdmap e23: 4 total, 3 up, 3 in 2021-02-10T04:24:23.318 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:22 smithi164 conmon[29288]: audit 2021-02-10T04:24:21.989121+0000 mon.a (mon.0) 379 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-02-10T04:24:23.318 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:22 smithi164 conmon[29288]: audit 2021-02-10T04:24:21.989762+0000 mon.c (mon.1) 11 : audit [INF] from='osd.3 v2:172.21.15.85:6813/7' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:24:23.318 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:22 smithi164 conmon[29288]: audit 2021-02-10T04:24:21.990125+0000 mon.a (mon.0) 380 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:24:23.440 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:24:22 smithi085 conmon[52836]: debug 2021-02-10T04:24:22.992+0000 7fba928b5700 -1 osd.3 0 waiting for initial osdmap 2021-02-10T04:24:23.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:22 smithi085 conmon[29319]: cluster 2021-02-10T04:24:21.929638+0000 mgr.y (mgr.14140) 125 : cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-02-10T04:24:23.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:22 smithi085 conmon[29319]: audit 2021-02-10T04:24:21.988835+0000 mon.a (mon.0) 377 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2021-02-10T04:24:23.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:22 smithi085 conmon[29319]: cluster 2021-02-10T04:24:21.988894+0000 mon.a (mon.0) 378 : cluster [DBG] osdmap e23: 4 total, 3 up, 3 in 2021-02-10T04:24:23.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:22 smithi085 conmon[29319]: audit 2021-02-10T04:24:21.989121+0000 mon.a (mon.0) 379 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-02-10T04:24:23.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:22 smithi085 conmon[29319]: audit 2021-02-10T04:24:21.989762+0000 mon.c (mon.1) 11 : audit [INF] from='osd.3 v2:172.21.15.85:6813/7' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:24:23.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:22 smithi085 conmon[29319]: audit 2021-02-10T04:24:21.990125+0000 mon.a (mon.0) 380 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:24:23.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:22 smithi085 conmon[35961]: cluster 2021-02-10T04:24:23.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:22 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:24:23.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:22 smithi085 conmon[35961]: 24:21.929638+0000 mgr.y (mgr.14140) 125 : cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-02-10T04:24:23.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:22 smithi085 conmon[35961]: audit 2021 2021-02-10T04:24:23.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:22 smithi085 conmon[35961]: -02-10T04:24:21.988835+0000 mon.a 2021-02-10T04:24:23.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:22 smithi085 conmon[35961]: (mon.0) 377 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2021-02-10T04:24:23.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:22 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:24:23.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:22 smithi085 conmon[35961]: T04:24:21.988894+0000 mon.a (mon.0) 378 : 2021-02-10T04:24:23.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:22 smithi085 conmon[35961]: cluster [DBG] osdmap e23: 4 total, 3 up, 3 in 2021-02-10T04:24:23.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:22 smithi085 conmon[35961]: audit 2021-02-10T04:24:23.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:22 smithi085 conmon[35961]: 2021-02-10T04:24:21.989121+0000 mon.a (mon.0 2021-02-10T04:24:23.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:22 smithi085 conmon[35961]: ) 379 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-02-10T04:24:23.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:22 smithi085 conmon[35961]: audit 2021-02-10T04:24:21 2021-02-10T04:24:23.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:22 smithi085 conmon[35961]: .989762+0000 mon.c (mon.1) 11 : audit [INF] from='osd.3 v2:172.21.15.85:6813/7' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:24:23.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:22 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:24:23.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:22 smithi085 conmon[35961]: 10T04:24:21.990125+0000 mon.a (mon.0) 2021-02-10T04:24:23.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:22 smithi085 conmon[35961]: 380 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:24:24.253 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:24 smithi164 conmon[29288]: audit 2021-02-10T04:24:22.990248+0000 2021-02-10T04:24:24.253 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:24 smithi164 conmon[29288]: mon.a (mon.0) 381 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]': finished 2021-02-10T04:24:24.253 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:24 smithi164 conmon[29288]: cluster 2021-02-10T04:24:22.990326+0000 mon.a (mon.0) 382 : cluster [DBG] osdmap e24: 4 total, 3 up, 3 in 2021-02-10T04:24:24.253 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:24 smithi164 conmon[29288]: audit 2021-02-10T04:24:22.990703+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-02-10T04:24:24.253 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:24 smithi164 conmon[29288]: audit 2021-02-10T04:24:22.994297+0000 mon.a (mon.0) 384 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-02-10T04:24:24.253 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:24 smithi164 conmon[29288]: audit 2021-02-10T04:24:23.188731+0000 mon.a (mon.0) 385 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:24:24.253 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:24 smithi164 conmon[29288]: audit 2021-02-10T04:24:23.515476+0000 mon.a (mon.0) 386 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:24.253 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:24 smithi164 conmon[29288]: audit 2021-02-10T04:24:23.518823+0000 mon.a (mon.0) 387 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:24.253 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:24 smithi164 conmon[29288]: audit 2021-02-10T04:24:23.523080+0000 mon.a (mon.0) 388 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:24.253 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:24 smithi164 conmon[29288]: audit 2021-02-10T04:24:23.679789+0000 mon.a (mon.0) 389 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-02-10T04:24:24.253 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:24 smithi164 conmon[29288]: audit 2021-02-10T04:24:23.681086+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-02-10T04:24:24.254 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:24 smithi164 conmon[29288]: audit 2021-02-10T04:24:23.681598+0000 mon.a (mon.0) 391 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:24:24.254 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:24 smithi164 conmon[29288]: audit 2021-02-10T04:24:23.682205+0000 mon.a (mon.0) 392 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:24:24.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:24 smithi085 conmon[29319]: audit 2021-02-10T04:24:22.990248+0000 mon.a (mon.0) 381 : audit 2021-02-10T04:24:24.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:24 smithi085 conmon[29319]: [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]': finished 2021-02-10T04:24:24.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:24 smithi085 conmon[29319]: cluster 2021-02-10T04:24:22.990326+0000 mon.a (mon.0) 382 : cluster [DBG] osdmap e24: 4 total, 3 up, 3 in 2021-02-10T04:24:24.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:24 smithi085 conmon[29319]: audit 2021-02-10T04:24:22.990703+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-02-10T04:24:24.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:24 smithi085 conmon[29319]: audit 2021-02-10T04:24:22.994297+0000 mon.a (mon.0) 384 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-02-10T04:24:24.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:24 smithi085 conmon[29319]: audit 2021-02-10T04:24:23.188731+0000 mon.a (mon.0) 385 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:24:24.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:24 smithi085 conmon[29319]: audit 2021-02-10T04:24:23.515476+0000 mon.a (mon.0) 386 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:24.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:24 smithi085 conmon[29319]: audit 2021-02-10T04:24:23.518823+0000 mon.a (mon.0) 387 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:24.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:24 smithi085 conmon[29319]: audit 2021-02-10T04:24:23.523080+0000 mon.a (mon.0) 388 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:24.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:24 smithi085 conmon[29319]: audit 2021-02-10T04:24:23.679789+0000 mon.a (mon.0) 389 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-02-10T04:24:24.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:24 smithi085 conmon[29319]: audit 2021-02-10T04:24:23.681086+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-02-10T04:24:24.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:24 smithi085 conmon[29319]: audit 2021-02-10T04:24:23.681598+0000 mon.a (mon.0) 391 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:24:24.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:24 smithi085 conmon[29319]: audit 2021-02-10T04:24:23.682205+0000 mon.a (mon.0) 392 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:24:24.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:24 smithi085 conmon[35961]: audit 2021-02-10T04:24:22.990248 2021-02-10T04:24:24.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:24 smithi085 conmon[35961]: +0000 mon.a (mon.0) 381 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]': finished 2021-02-10T04:24:24.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:24 smithi085 conmon[35961]: cluster 2021-02-10T04:24:22.990326+0000 mon.a (mon.0) 382 : cluster [DBG] osdmap e24: 4 total, 3 up, 3 in 2021-02-10T04:24:24.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:24 smithi085 conmon[35961]: audit 2021-02-10T04:24:22.990703+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-02-10T04:24:24.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:24 smithi085 conmon[35961]: audit 2021-02-10T04:24:22.994297+0000 mon.a (mon.0) 384 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-02-10T04:24:24.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:24 smithi085 conmon[35961]: audit 2021-02-10T04:24:23.188731+0000 mon.a (mon.0) 385 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:24:24.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:24 smithi085 conmon[35961]: audit 2021-02-10T04:24:23.515476+0000 mon.a (mon.0) 386 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:24.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:24 smithi085 conmon[35961]: audit 2021-02-10T04:24:23.518823+0000 mon.a (mon.0) 387 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:24.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:24 smithi085 conmon[35961]: audit 2021-02-10T04:24:23.523080+0000 mon.a (mon.0) 388 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:24.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:24 smithi085 conmon[35961]: audit 2021-02-10T04:24:23.679789+0000 mon.a (mon.0) 389 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-02-10T04:24:24.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:24 smithi085 conmon[35961]: audit 2021-02-10T04:24:23.681086+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-02-10T04:24:24.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:24 smithi085 conmon[35961]: audit 2021-02-10T04:24:23.681598+0000 mon.a (mon.0) 391 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:24:24.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:24 smithi085 conmon[35961]: audit 2021-02-10T04:24:23.682205+0000 2021-02-10T04:24:24.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:24 smithi085 conmon[35961]: mon.a (mon.0) 392 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:24:25.262 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:25 smithi164 conmon[29288]: cluster 2021-02-10T04:24:23.007806+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:24:25.262 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:25 smithi164 conmon[29288]: cluster 2021-02-10T04:24:23.007936+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:24:25.262 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:25 smithi164 conmon[29288]: cephadm 2021-02-10T04:24:23.511829+0000 mgr.y (mgr.14140) 126 : cephadm [INF] Detected new or changed devices on smithi085 2021-02-10T04:24:25.262 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:25 smithi164 conmon[29288]: audit 2021-02-10T04:24:23.678982+0000 mgr.y (mgr.14140) 127 : audit [DBG] from='client.34178 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:24:25.262 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:25 smithi164 conmon[29288]: cephadm 2021-02-10T04:24:23.680885+0000 mgr.y (mgr.14140) 128 : cephadm [INF] Applying service osd.None on host smithi164... 2021-02-10T04:24:25.262 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:25 smithi164 conmon[29288]: cluster 2021-02-10T04:24:23.929908+0000 mgr.y (mgr.14140) 129 : cluster [DBG] pgmap v97: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-02-10T04:24:25.262 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:25 smithi164 conmon[29288]: cluster 2021-02-10T04:24:23.994988+0000 mon.a (mon.0) 393 : cluster [INF] osd.3 v2:172.21.15.85:6813/7 boot 2021-02-10T04:24:25.263 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:25 smithi164 conmon[29288]: cluster 2021-02-10T04:24:23.995034+0000 mon.a (mon. 2021-02-10T04:24:25.263 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:25 smithi164 conmon[29288]: 0) 394 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2021-02-10T04:24:25.263 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:25 smithi164 conmon[29288]: audit 2021-02-10T04:24:23.995666+0000 mon.a (mon.0) 395 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-02-10T04:24:25.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[29319]: cluster 2021-02-10T04:24:23.007806+0000 osd.3 (osd.3) 1 : cluster 2021-02-10T04:24:25.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[29319]: [DBG] purged_snaps scrub starts 2021-02-10T04:24:25.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[29319]: cluster 2021-02-10T04:24:23.007936+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:24:25.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[29319]: cephadm 2021-02-10T04:24:23.511829+0000 mgr.y (mgr.14140) 126 : cephadm [INF] Detected new or changed devices on smithi085 2021-02-10T04:24:25.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[29319]: audit 2021-02-10T04:24:23.678982+0000 mgr.y (mgr.14140) 127 : audit [DBG] from='client.34178 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:24:25.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[29319]: cephadm 2021-02-10T04:24:23.680885+0000 mgr.y (mgr.14140) 128 : cephadm [INF] Applying service osd.None on host smithi164... 2021-02-10T04:24:25.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[29319]: cluster 2021-02-10T04:24:23.929908+0000 mgr.y (mgr.14140) 129 : cluster [DBG] pgmap v97: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-02-10T04:24:25.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:24:25.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[29319]: -10T04:24:23.994988+0000 mon.a ( 2021-02-10T04:24:25.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[29319]: mon.0) 393 : cluster [INF] osd.3 v2:172.21.15.85:6813/7 boot 2021-02-10T04:24:25.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[29319]: 2021-02-10T04:24:25.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:24:25.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[29319]: 02-10T04:24:23.995034+0000 2021-02-10T04:24:25.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[29319]: mon.a (mon.0) 394 : cluster 2021-02-10T04:24:25.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[29319]: [DBG] osdmap e25: 4 total, 4 up, 4 in 2021-02-10T04:24:25.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[29319]: audit 2021 2021-02-10T04:24:25.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[29319]: -02-10T04:24:23 2021-02-10T04:24:25.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[29319]: .995666+0000 mon.a (mon.0 2021-02-10T04:24:25.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[29319]: ) 395 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-02-10T04:24:25.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[35961]: cluster 2021-02-10T04:24:23.007806 2021-02-10T04:24:25.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[35961]: +0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:24:25.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[35961]: cluster 2021-02-10T04:24:25.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[35961]: 2021-02-10T04:24:23.007936+0000 osd.3 (osd. 2021-02-10T04:24:25.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[35961]: 3) 2 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:24:25.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[35961]: cephadm 2021-02-10T04:24:25.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[35961]: 2021-02-10T04:24:23.511829+0000 mgr.y (mgr.14140) 2021-02-10T04:24:25.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[35961]: 126 : cephadm [INF] Detected new or changed devices on smithi085 2021-02-10T04:24:25.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:24:25.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[35961]: 10T04:24:23.678982+0000 mgr.y (mgr. 2021-02-10T04:24:25.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[35961]: 14140) 127 : audit [DBG] from='client.34178 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:24:25.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[35961]: cephadm 2021-02 2021-02-10T04:24:25.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[35961]: -10T04:24:23.680885+0000 mgr.y 2021-02-10T04:24:25.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[35961]: (mgr.14140) 128 : cephadm [INF] Applying service osd.None on host smithi164... 2021-02-10T04:24:25.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:24:25.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[35961]: -02-10T04:24:23 2021-02-10T04:24:25.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[35961]: .929908+0000 mgr.y (mgr.14140) 129 : 2021-02-10T04:24:25.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[35961]: cluster [DBG] pgmap v97: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-02-10T04:24:25.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[35961]: cluster 2021-02-10T04:24:23.994988+0000 mon.a (mon.0 2021-02-10T04:24:25.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[35961]: ) 393 : cluster [INF] osd.3 v2:172.21.15.85:6813/7 boot 2021-02-10T04:24:25.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[35961]: cluster 2021-02-10T04:24:25.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[35961]: 2021-02-10T04:24:23.995034+0000 mon.a (mon.0) 394 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2021-02-10T04:24:25.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[35961]: audit 2021 2021-02-10T04:24:25.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[35961]: -02-10T04:24:23.995666+0000 2021-02-10T04:24:25.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[35961]: mon.a (mon.0) 395 : audit [DBG] 2021-02-10T04:24:25.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:25 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-02-10T04:24:26.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:26 smithi164 conmon[29288]: cluster 2021-02-10T04:24:25.010908+0000 mon.a (mon.0 2021-02-10T04:24:26.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:26 smithi164 conmon[29288]: ) 396 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2021-02-10T04:24:26.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:26 smithi164 conmon[29288]: audit 2021-02-10T04:24:25.859585+0000 mon.b (mon.2) 5 : audit [INF] from='client.? 172.21.15.164:0/868224073' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91c8a80a-37b6-4c30-a32a-647b2e6d801f"}]: dispatch 2021-02-10T04:24:26.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:26 smithi164 conmon[29288]: audit 2021-02-10T04:24:25.860233+0000 mon.a (mon.0) 397 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91c8a80a-37b6-4c30-a32a-647b2e6d801f"}]: dispatch 2021-02-10T04:24:26.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:26 smithi164 conmon[29288]: audit 2021-02-10T04:24:25.869731+0000 mon.a (mon.0) 398 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91c8a80a-37b6-4c30-a32a-647b2e6d801f"}]': finished 2021-02-10T04:24:26.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:26 smithi164 conmon[29288]: cluster 2021-02-10T04:24:25.869810+0000 mon.a (mon.0) 399 : cluster [DBG] osdmap e27: 5 total, 4 up, 4 in 2021-02-10T04:24:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:26 smithi164 conmon[29288]: audit 2021-02-10T04:24:25.869962+0000 mon.a (mon.0) 400 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-02-10T04:24:26.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:26 smithi085 conmon[29319]: cluster 2021-02-10T04:24:25.010908+0000 mon.a (mon.0) 396 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2021-02-10T04:24:26.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:26 smithi085 conmon[29319]: audit 2021-02-10T04:24:25.859585+0000 mon.b (mon.2) 5 : audit [INF] from='client.? 172.21.15.164:0/868224073' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91c8a80a-37b6-4c30-a32a-647b2e6d801f"}]: dispatch 2021-02-10T04:24:26.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:26 smithi085 conmon[29319]: audit 2021-02-10T 2021-02-10T04:24:26.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:26 smithi085 conmon[29319]: 04:24:25.860233+0000 mon.a (mon.0) 397 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91c8a80a-37b6-4c30-a32a-647b2e6d801f"}]: dispatch 2021-02-10T04:24:26.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:26 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:24:26.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:26 smithi085 conmon[29319]: 10T04:24:25.869731+0000 mon.a (mon.0) 398 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91c8a80a-37b6-4c30-a32a-647b2e6d801f"}]': finished 2021-02-10T04:24:26.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:26 smithi085 conmon[29319]: cluster 2021-02-10T04:24:25.869810+0000 mon.a (mon.0) 399 : cluster [DBG] 2021-02-10T04:24:26.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:26 smithi085 conmon[29319]: osdmap e27: 5 total, 4 up, 4 in 2021-02-10T04:24:26.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:26 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:24:26.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:26 smithi085 conmon[29319]: :24:25.869962+0000 mon.a (mon.0) 400 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-02-10T04:24:26.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:26 smithi085 conmon[35961]: cluster 2021-02-10T04:24:25.010908+0000 mon.a ( 2021-02-10T04:24:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:26 smithi085 conmon[35961]: mon.0) 396 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2021-02-10T04:24:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:26 smithi085 conmon[35961]: audit 2021-02-10T04:24:25 2021-02-10T04:24:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:26 smithi085 conmon[35961]: .859585+0000 mon.b (mon.2) 5 : audit [INF] from='client.? 172.21.15.164:0/868224073' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91c8a80a-37b6-4c30-a32a-647b2e6d801f"}]: dispatch 2021-02-10T04:24:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:26 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:24:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:26 smithi085 conmon[35961]: 02-10T04:24:25.860233+0000 mon.a (mon.0) 397 : audit 2021-02-10T04:24:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:26 smithi085 conmon[35961]: [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91c8a80a-37b6-4c30-a32a-647b2e6d801f"}]: dispatch 2021-02-10T04:24:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:26 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:24:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:26 smithi085 conmon[35961]: 02-10T04:24:25.869731+0000 mon.a (mon.0) 398 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91c8a80a-37b6-4c30-a32a-647b2e6d801f"}]': finished 2021-02-10T04:24:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:26 smithi085 conmon[35961]: cluster 2021-02-10T04:24:25.869810+0000 mon.a ( 2021-02-10T04:24:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:26 smithi085 conmon[35961]: mon.0) 399 : cluster [DBG] osdmap e27: 5 total, 4 up, 4 in 2021-02-10T04:24:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:26 smithi085 conmon[35961]: audit 2021-02-10T04:24:25.869962+0000 mon.a (mon. 2021-02-10T04:24:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:26 smithi085 conmon[35961]: 0) 400 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-02-10T04:24:27.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:27 smithi164 conmon[29288]: cluster 2021-02-10T04:24:25.930275+0000 mgr.y (mgr.14140) 130 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 19 MiB used, 358 GiB / 358 GiB avail 2021-02-10T04:24:27.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:27 smithi164 conmon[29288]: audit 2021-02-10T04:24:26.332507+0000 mon.b (mon.2) 6 : audit [DBG] from='client.? 172.21.15.164:0/3490645207' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-02-10T04:24:27.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:27 smithi085 conmon[29319]: cluster 2021-02-10T04:24:25.930275+0000 mgr.y 2021-02-10T04:24:27.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:27 smithi085 conmon[29319]: (mgr.14140) 130 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 19 MiB used, 358 GiB / 358 GiB avail 2021-02-10T04:24:27.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:27 smithi085 conmon[29319]: audit 2021-02-10T04:24:26.332507+0000 mon.b (mon.2) 6 : audit [DBG] from='client.? 172.21.15.164:0/3490645207' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-02-10T04:24:27.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:27 smithi085 conmon[35961]: cluster 2021-02-10T04:24:25.930275+0000 mgr.y (mgr.14140) 130 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 19 MiB used, 358 GiB / 358 GiB avail 2021-02-10T04:24:27.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:27 smithi085 conmon[35961]: 2021-02-10T04:24:27.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:27 smithi085 conmon[35961]: audit 2021-02-10T04:24:26.332507+0000 mon.b (mon.2) 6 : audit [DBG] from='client.? 172.21.15.164:0/3490645207' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-02-10T04:24:29.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:28 smithi085 conmon[29319]: cluster 2021-02-10T04:24:27.930737+0000 mgr.y (mgr.14140) 131 : cluster 2021-02-10T04:24:29.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:28 smithi085 conmon[29319]: [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-02-10T04:24:29.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:28 smithi085 conmon[29319]: cluster 2021-02-10T04:24:28.735494+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:24:29.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:28 smithi085 conmon[29319]: cluster 2021-02-10T04:24:28.735622+0000 osd.0 (osd.0) 4 : cluster 2021-02-10T04:24:29.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:28 smithi085 conmon[29319]: [DBG] purged_snaps scrub ok 2021-02-10T04:24:29.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:28 smithi085 conmon[35961]: cluster 2021-02-10T04:24:27.930737+0000 mgr.y (mgr. 2021-02-10T04:24:29.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:28 smithi085 conmon[35961]: 14140) 131 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-02-10T04:24:29.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:28 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:24:29.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:28 smithi085 conmon[35961]: 24:28.735494+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:24:29.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:28 smithi085 conmon[35961]: cluster 2021-02-10T04:24 2021-02-10T04:24:29.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:28 smithi085 conmon[35961]: :28.735622+0000 osd.0 (osd.0) 4 : cluster 2021-02-10T04:24:29.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:28 smithi085 conmon[35961]: [DBG] purged_snaps scrub ok 2021-02-10T04:24:29.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:28 smithi164 conmon[29288]: cluster 2021-02-10T04:24:27.930737+0000 mgr.y (mgr.14140) 131 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-02-10T04:24:29.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:28 smithi164 conmon[29288]: cluster 2021-02-10T04:24:28.735494+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:24:29.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:28 smithi164 conmon[29288]: cluster 2021-02-10T04:24:28.735622+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:24:30.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:29 smithi085 conmon[35961]: cluster 2021-02-10T04:24:28.906082+0000 mon.a (mon.0) 401 : cluster [DBG] osdmap e28: 5 total, 4 up, 4 in 2021-02-10T04:24:30.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:29 smithi085 conmon[35961]: 2021-02-10T04:24:30.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:29 smithi085 conmon[35961]: audit 2021-02-10T04:24:28.906281+0000 mon.a (mon.0) 402 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-02-10T04:24:30.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:29 smithi085 conmon[35961]: audit 2021-02-10T04:24:29.114853+0000 mon.a (mon.0) 403 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:24:30.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:29 smithi085 conmon[29319]: cluster 2021-02-10T04:24:28.906082+0000 mon.a (mon.0) 401 : cluster [DBG] 2021-02-10T04:24:30.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:29 smithi085 conmon[29319]: osdmap e28: 5 total, 4 up, 4 in 2021-02-10T04:24:30.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:29 smithi085 conmon[29319]: audit 2021-02-10T04:24:28.906281+0000 mon.a (mon.0) 402 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-02-10T04:24:30.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:29 smithi085 conmon[29319]: audit 2021-02-10T04:24:29.114853+0000 mon.a (mon.0) 2021-02-10T04:24:30.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:29 smithi085 conmon[29319]: 403 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:24:30.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:29 smithi164 conmon[29288]: cluster 2021-02-10T04:24:28.906082+0000 mon.a ( 2021-02-10T04:24:30.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:29 smithi164 conmon[29288]: mon.0) 401 : cluster [DBG] osdmap e28: 5 total, 4 up, 4 in 2021-02-10T04:24:30.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:29 smithi164 conmon[29288]: audit 2021-02-10T04:24:28.906281+0000 mon.a (mon.0) 402 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-02-10T04:24:30.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:29 smithi164 conmon[29288]: audit 2021-02-10T04:24:29.114853+0000 mon.a (mon.0) 403 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:24:31.168 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:30 smithi164 conmon[29288]: cluster 2021-02-10T04:24:29.931105+0000 mgr.y (mgr.14140) 132 : cluster 2021-02-10T04:24:31.168 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:30 smithi164 conmon[29288]: [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-02-10T04:24:31.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:30 smithi085 conmon[29319]: cluster 2021-02-10T04:24:29.931105+0000 mgr.y (mgr.14140) 132 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-02-10T04:24:31.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:30 smithi085 conmon[35961]: cluster 2021-02-10T04:24:29.931105+0000 mgr.y (mgr.14140) 132 : cluster 2021-02-10T04:24:31.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:30 smithi085 conmon[35961]: [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-02-10T04:24:32.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:31 smithi085 conmon[29319]: audit 2021-02-10T04:24:31.556357+0000 mon.a (mon.0) 2021-02-10T04:24:32.286 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:31 smithi085 conmon[29319]: 404 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-02-10T04:24:32.286 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:31 smithi085 conmon[29319]: audit 2021-02-10T04:24:31.557192+0000 mon.a (mon.0) 405 2021-02-10T04:24:32.286 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:31 smithi085 conmon[29319]: : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:24:32.286 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:31 smithi085 conmon[29319]: audit 2021-02-10T04:24:31 2021-02-10T04:24:32.286 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:31 smithi085 conmon[29319]: .558185+0000 mon.a (mon.0) 406 : audit 2021-02-10T04:24:32.287 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:31 smithi085 conmon[29319]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.4", "key": "container_image"}]: dispatch 2021-02-10T04:24:32.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:31 smithi085 conmon[35961]: audit 2021-02-10T04:24:31.556357+0000 mon.a ( 2021-02-10T04:24:32.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:31 smithi085 conmon[35961]: mon.0) 404 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-02-10T04:24:32.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:31 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:24:32.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:31 smithi085 conmon[35961]: T04:24:31.557192+0000 mon.a (mon.0) 405 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:24:32.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:31 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:24:32.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:31 smithi085 conmon[35961]: 02-10T04:24:31.558185+0000 mon.a ( 2021-02-10T04:24:32.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:31 smithi085 conmon[35961]: mon.0) 406 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.4", "key": "container_image"}]: dispatch 2021-02-10T04:24:32.371 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:31 smithi164 conmon[29288]: audit 2021-02-10T04:24:31.556357+0000 mon.a (mon.0) 404 : 2021-02-10T04:24:32.373 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:31 smithi164 conmon[29288]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-02-10T04:24:32.373 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:31 smithi164 conmon[29288]: audit 2021-02-10T04:24:31.557192+0000 mon.a (mon.0) 405 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:24:32.375 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:31 smithi164 conmon[29288]: audit 2021-02-10T04:24:31.558185+0000 mon.a (mon.0) 406 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.4", "key": "container_image"}]: dispatch 2021-02-10T04:24:33.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:32 smithi164 conmon[29288]: cephadm 2021-02-10T04:24:31.557774+0000 mgr.y (mgr.14140) 133 : cephadm [INF] Deploying daemon osd.4 on smithi164 2021-02-10T04:24:33.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:32 smithi164 conmon[29288]: cluster 2021-02-10T04:24:31.931485+0000 mgr.y (mgr.14140) 134 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-02-10T04:24:33.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:32 smithi085 conmon[29319]: cephadm 2021-02-10T04:24:31.557774+0000 mgr.y (mgr.14140) 133 : cephadm [INF] Deploying daemon osd.4 on smithi164 2021-02-10T04:24:33.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:32 smithi085 conmon[29319]: cluster 2021-02-10T04:24:31.931485+0000 mgr.y (mgr.14140) 134 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-02-10T04:24:33.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:32 smithi085 conmon[35961]: cephadm 2021-02-10T04:24:31.557774+0000 mgr.y (mgr.14140) 2021-02-10T04:24:33.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:32 smithi085 conmon[35961]: 133 : cephadm [INF] Deploying daemon osd.4 on smithi164 2021-02-10T04:24:33.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:32 smithi085 conmon[35961]: cluster 2021-02-10T04:24:31.931485+0000 mgr.y (mgr.14140) 134 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-02-10T04:24:34.793 INFO:teuthology.orchestra.run.smithi164.stdout:Created osd(s) 4 on host 'smithi164' 2021-02-10T04:24:35.256 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:34 smithi164 conmon[29288]: cluster 2021-02-10T04:24:33.931756+0000 mgr.y (mgr.14140 2021-02-10T04:24:35.256 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:34 smithi164 conmon[29288]: ) 135 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-02-10T04:24:35.256 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:34 smithi164 conmon[29288]: audit 2021-02-10T04:24:34.782091+0000 mon.a (mon.0) 407 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:24:35.256 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:34 smithi164 conmon[29288]: audit 2021-02-10T04:24:34.785907+0000 mon.a (mon.0) 408 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:35.256 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:34 smithi164 conmon[29288]: audit 2021-02-10T04:24:34.790718+0000 mon.a (mon.0) 409 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:35.369 DEBUG:teuthology.orchestra.run.smithi164:osd.4> sudo journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.4.service 2021-02-10T04:24:35.371 INFO:tasks.cephadm:Deploying osd.5 on smithi164 with /dev/vg_nvme/lv_3... 2021-02-10T04:24:35.372 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2021-02-10T04:24:35.400 INFO:journalctl@ceph.osd.4.smithi164.stdout:-- Logs begin at Wed 2021-02-10 04:09:33 UTC. -- 2021-02-10T04:24:35.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:34 smithi085 conmon[29319]: cluster 2021-02-10T04:24:33.931756+0000 mgr.y (mgr.14140) 135 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-02-10T04:24:35.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:34 smithi085 conmon[29319]: audit 2021-02-10T04:24:34.782091+0000 mon.a (mon.0) 407 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:24:35.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:34 smithi085 conmon[29319]: audit 2021-02-10T04:24:34.785907+0000 mon.a (mon.0) 408 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:35.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:34 smithi085 conmon[29319]: audit 2021-02-10T04:24:34.790718+0000 mon.a (mon.0) 409 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:35.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:34 smithi085 conmon[35961]: cluster 2021-02-10T04:24:33.931756+0000 mgr.y (mgr.14140) 135 : 2021-02-10T04:24:35.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:34 smithi085 conmon[35961]: cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-02-10T04:24:35.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:34 smithi085 conmon[35961]: audit 2021-02-10T04:24:34.782091+0000 mon.a (mon.0) 407 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:24:35.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:34 smithi085 conmon[35961]: audit 2021-02-10T04:24:34.785907+0000 mon.a (mon.0) 408 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:35.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:34 smithi085 conmon[35961]: 2021-02-10T04:24:35.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:34 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:24:35.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:34 smithi085 conmon[35961]: :24:34.790718+0000 mon.a (mon.0) 409 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:37.065 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:36 smithi164 conmon[29288]: audit 2021-02-10T04:24:35.925369+0000 2021-02-10T04:24:37.065 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:36 smithi164 conmon[29288]: mon.a (mon.0) 410 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:37.065 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:36 smithi164 conmon[29288]: audit 2021-02-10T04:24:35.926652+0000 mon.a (mon.0) 411 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:24:37.065 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:36 smithi164 conmon[29288]: cluster 2021-02-10T04:24:35.932171+0000 mgr.y (mgr.14140) 136 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-02-10T04:24:37.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:36 smithi085 conmon[29319]: audit 2021-02-10T04:24:35.925369+0000 mon.a (mon.0) 410 : audit [INF] 2021-02-10T04:24:37.205 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:36 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:37.205 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:36 smithi085 conmon[29319]: audit 2021-02-10T04:24:35.926652+0000 mon.a (mon.0) 411 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:24:37.206 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:36 smithi085 conmon[29319]: cluster 2021-02-10T04:24: 2021-02-10T04:24:37.206 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:36 smithi085 conmon[29319]: 35.932171+0000 mgr.y (mgr.14140) 136 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-02-10T04:24:37.207 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:36 smithi085 conmon[35961]: audit 2021-02-10T04:24:35.925369+0000 mon.a ( 2021-02-10T04:24:37.207 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:36 smithi085 conmon[35961]: mon.0) 410 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:37.208 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:36 smithi085 conmon[35961]: audit 2021-02-10T04:24:35.926652+0000 mon.a ( 2021-02-10T04:24:37.208 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:36 smithi085 conmon[35961]: mon.0) 411 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:24:37.208 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:36 smithi085 conmon[35961]: cluster 2021-02-10T04:24:35.932171+0000 mgr.y (mgr 2021-02-10T04:24:37.210 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:36 smithi085 conmon[35961]: .14140) 136 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-02-10T04:24:37.698 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping: /dev/vg_nvme/lv_3 2021-02-10T04:24:37.698 INFO:teuthology.orchestra.run.smithi164.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2021-02-10T04:24:37.698 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10+0 records in 2021-02-10T04:24:37.698 INFO:teuthology.orchestra.run.smithi164.stderr:10+0 records out 2021-02-10T04:24:37.698 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0117676 s, 891 MB/s 2021-02-10T04:24:37.698 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping successful for: 2021-02-10T04:24:38.372 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:24:37 smithi164 conmon[33677]: debug 2021-02-10T04:24:37.989+0000 7fbecf4b5200 -1 osd.4 0 log_to_monitors {default=true} 2021-02-10T04:24:38.390 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph orch daemon add osd smithi164:vg_nvme/lv_3 2021-02-10T04:24:39.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:38 smithi164 conmon[29288]: cluster 2021-02-10T04:24:37.932531+0000 mgr.y (mgr.14140) 137 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-02-10T04:24:39.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:38 smithi164 conmon[29288]: audit 2021-02-10T04:24:37.994000+0000 mon.b (mon.2) 7 : audit [INF] from='osd.4 v2:172.21.15.164:6800/7' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-02-10T04:24:39.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:38 smithi164 conmon[29288]: audit 2021-02-10T04:24:37.994323+0000 mon.a (mon.0) 412 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-02-10T04:24:39.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:38 smithi164 conmon[29288]: audit 2021-02-10T04:24:38.030862+0000 mon.a (mon.0) 413 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:24:39.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:38 smithi164 conmon[29288]: audit 2021-02-10T04:24:38.031826+0000 mon.a (mon.0) 414 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:24:39.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:38 smithi085 conmon[29319]: cluster 2021-02-10T04:24:37.932531+0000 mgr.y (mgr 2021-02-10T04:24:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:38 smithi085 conmon[29319]: .14140) 137 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-02-10T04:24:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:38 smithi085 conmon[29319]: audit 2021-02-10T04:24:37.994000+0000 mon.b (mon.2) 7 : audit [INF] from='osd.4 v2:172.21.15.164:6800/7' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-02-10T04:24:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:38 smithi085 conmon[29319]: audit 2021-02-10T04:24:37.994323+0000 mon.a (mon.0) 412 2021-02-10T04:24:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:38 smithi085 conmon[29319]: : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-02-10T04:24:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:38 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:24:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:38 smithi085 conmon[29319]: :24:38.030862+0000 mon.a (mon.0) 413 : audit 2021-02-10T04:24:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:38 smithi085 conmon[29319]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:24:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:38 smithi085 conmon[29319]: audit 2021-02-10T04:24:38.031826+0000 2021-02-10T04:24:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:38 smithi085 conmon[29319]: mon.a (mon.0) 414 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:24:39.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:38 smithi085 conmon[35961]: cluster 2021-02-10T04:24:37.932531+0000 mgr.y (mgr. 2021-02-10T04:24:39.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:38 smithi085 conmon[35961]: 14140) 137 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-02-10T04:24:39.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:38 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:24:39.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:38 smithi085 conmon[35961]: 04:24:37.994000+0000 mon.b (mon.2) 7 : audit [INF] from='osd.4 v2:172.21.15.164:6800/7' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-02-10T04:24:39.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:38 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:24:39.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:38 smithi085 conmon[35961]: 02-10T04:24:37.994323+0000 mon.a (mon.0) 412 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-02-10T04:24:39.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:38 smithi085 conmon[35961]: 2021-02-10T04:24:39.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:38 smithi085 conmon[35961]: audit 2021-02-10T04:24:39.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:38 smithi085 conmon[35961]: 2021-02-10T04:24:38.030862+0000 mon.a (mon. 2021-02-10T04:24:39.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:38 smithi085 conmon[35961]: 0) 413 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:24:39.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:38 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:24:39.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:38 smithi085 conmon[35961]: 02-10T04:24:38.031826+0000 mon.a ( 2021-02-10T04:24:39.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:38 smithi085 conmon[35961]: mon.0) 414 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:24:40.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:39 smithi085 conmon[35961]: audit 2021-02-10T04:24:38.912359+0000 mon.a (mon.0) 2021-02-10T04:24:40.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:39 smithi085 conmon[35961]: 415 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-02-10T04:24:40.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:39 smithi085 conmon[35961]: cluster 2021-02-10T04:24:38.912481+0000 mon.a (mon.0) 416 : cluster [DBG] osdmap e29: 5 total, 4 up, 4 in 2021-02-10T04:24:40.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:39 smithi085 conmon[35961]: audit 2021-02-10T04:24:38.912842+0000 mon.a (mon.0) 417 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-02-10T04:24:40.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:39 smithi085 conmon[35961]: audit 2021-02-10T04:24:38.917428+0000 mon.b (mon.2 2021-02-10T04:24:40.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:39 smithi085 conmon[35961]: ) 8 : audit [INF] from='osd.4 v2:172.21.15.164:6800/7' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2021-02-10T04:24:40.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:39 smithi085 conmon[35961]: audit 2021-02-10T04:24:38.917847+0000 mon.a (mon.0 2021-02-10T04:24:40.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:39 smithi085 conmon[35961]: ) 418 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2021-02-10T04:24:40.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:39 smithi085 conmon[29319]: audit 2021-02-10T04:24:38.912359+0000 mon.a (mon.0) 415 : audit 2021-02-10T04:24:40.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:39 smithi085 conmon[29319]: [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-02-10T04:24:40.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:39 smithi085 conmon[29319]: cluster 2021-02-10T04:24:38.912481+0000 mon.a (mon.0) 416 : cluster [DBG] osdmap e29: 5 total, 4 up, 4 in 2021-02-10T04:24:40.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:39 smithi085 conmon[29319]: audit 2021-02-10T04:24:38.912842+0000 mon.a (mon.0) 417 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-02-10T04:24:40.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:39 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:24:40.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:39 smithi085 conmon[29319]: 02-10T04:24:38.917428+0000 mon.b (mon.2) 8 : 2021-02-10T04:24:40.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:39 smithi085 conmon[29319]: audit [INF] from='osd.4 v2:172.21.15.164:6800/7' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2021-02-10T04:24:40.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:39 smithi085 conmon[29319]: audit 2021-02-10T04:24:40.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:39 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:24:40.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:39 smithi085 conmon[29319]: 04:24:38.917847+0000 mon.a (mon.0 2021-02-10T04:24:40.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:39 smithi085 conmon[29319]: ) 418 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2021-02-10T04:24:40.223 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:39 smithi164 conmon[29288]: audit 2021-02-10T04:24:38.912359+0000 mon.a (mon.0) 2021-02-10T04:24:40.224 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:39 smithi164 conmon[29288]: 415 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-02-10T04:24:40.224 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:39 smithi164 conmon[29288]: cluster 2021-02-10T04:24:38.912481+0000 mon.a (mon.0) 416 : cluster [DBG] osdmap e29: 5 total, 4 up, 4 in 2021-02-10T04:24:40.224 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:39 smithi164 conmon[29288]: audit 2021-02-10T04:24:38.912842+0000 mon.a (mon.0) 417 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-02-10T04:24:40.224 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:39 smithi164 conmon[29288]: audit 2021-02-10T04:24:38.917428+0000 mon.b (mon.2) 8 : audit [INF] from='osd.4 v2:172.21.15.164:6800/7' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2021-02-10T04:24:40.224 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:39 smithi164 conmon[29288]: audit 2021-02-10T04:24:38.917847+0000 mon.a (mon.0) 418 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2021-02-10T04:24:40.224 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:24:39 smithi164 conmon[33677]: debug 2021-02-10T04:24:39.918+0000 7fbec5a26700 -1 osd.4 0 waiting for initial osdmap 2021-02-10T04:24:41.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:40 smithi085 conmon[29319]: audit 2021-02-10T04:24:39.913962+0000 mon.a (mon.0) 2021-02-10T04:24:41.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:40 smithi085 conmon[29319]: 419 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2021-02-10T04:24:41.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:40 smithi085 conmon[29319]: cluster 2021-02-10T04:24:39.914082+0000 mon.a (mon.0) 420 : cluster [DBG] osdmap e30: 5 total, 4 up, 4 in 2021-02-10T04:24:41.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:40 smithi085 conmon[29319]: audit 2021-02-10T04:24 2021-02-10T04:24:41.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:40 smithi085 conmon[29319]: :39.914551+0000 mon.a (mon.0) 421 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-02-10T04:24:41.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:40 smithi085 conmon[29319]: audit 2021-02-10T04:24:41.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:40 smithi085 conmon[29319]: 2021-02-10T04:24:39.920537+0000 mon.a (mon.0) 422 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-02-10T04:24:41.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:40 smithi085 conmon[29319]: cluster 2021-02-10T04:24:41.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:40 smithi085 conmon[29319]: 2021-02-10T04:24:39.932959+0000 2021-02-10T04:24:41.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:40 smithi085 conmon[29319]: mgr.y (mgr.14140) 138 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-02-10T04:24:41.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:40 smithi085 conmon[29319]: audit 2021-02-10T04:24:40.671967+0000 mon.a 2021-02-10T04:24:41.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:40 smithi085 conmon[29319]: (mon.0) 423 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:24:41.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:40 smithi085 conmon[35961]: audit 2021-02-10T04:24:39.913962+0000 2021-02-10T04:24:41.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:40 smithi085 conmon[35961]: mon.a (mon.0) 419 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2021-02-10T04:24:41.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:40 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:24:41.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:40 smithi085 conmon[35961]: 02-10T04:24:39.914082+0000 mon.a (mon.0) 420 : cluster [DBG] osdmap e30: 5 total, 4 up, 4 in 2021-02-10T04:24:41.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:40 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:24:41.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:40 smithi085 conmon[35961]: -10T04:24:39.914551+0000 mon.a (mon.0) 421 : 2021-02-10T04:24:41.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:40 smithi085 conmon[35961]: audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-02-10T04:24:41.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:40 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:24:41.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:40 smithi085 conmon[35961]: T04:24:39.920537+0000 mon.a (mon.0) 2021-02-10T04:24:41.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:40 smithi085 conmon[35961]: 422 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-02-10T04:24:41.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:40 smithi085 conmon[35961]: cluster 2021-02-10T04:24:41.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:40 smithi085 conmon[35961]: 2021-02-10T04:24:39.932959+0000 mgr.y (mgr. 2021-02-10T04:24:41.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:40 smithi085 conmon[35961]: 14140) 138 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-02-10T04:24:41.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:40 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:24:41.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:40 smithi085 conmon[35961]: 04:24:40.671967+0000 mon.a (mon.0) 423 2021-02-10T04:24:41.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:40 smithi085 conmon[35961]: : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:24:41.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:40 smithi164 conmon[29288]: audit 2021-02-10T04:24:39.913962+0000 mon.a (mon.0) 419 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2021-02-10T04:24:41.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:40 smithi164 conmon[29288]: cluster 2021-02-10T04:24:39.914082+0000 mon.a (mon.0) 420 : cluster [DBG] osdmap e30: 5 total, 4 up, 4 in 2021-02-10T04:24:41.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:40 smithi164 conmon[29288]: audit 2021-02-10T04:24:39.914551+0000 mon.a (mon.0) 421 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-02-10T04:24:41.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:40 smithi164 conmon[29288]: audit 2021-02-10T04:24:39.920537+0000 mon.a (mon.0) 422 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-02-10T04:24:41.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:40 smithi164 conmon[29288]: cluster 2021-02-10T04:24:39.932959+0000 mgr.y (mgr.14140) 138 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-02-10T04:24:41.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:40 smithi164 conmon[29288]: audit 2021-02-10T04:24:40.671967+0000 mon.a (mon.0) 423 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:24:42.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: cluster 2021-02-10T04:24:39.034675+0000 osd.4 (osd. 2021-02-10T04:24:42.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: 4) 1 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:24:42.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: cluster 2021-02-10T04:24:39.034782+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:24:42.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: audit 2021-02-10T04:24:40.920747+0000 mon.a (mon.0) 424 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-02-10T04:24:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: cluster 2021-02-10T04:24:40.925931+0000 mon.a (mon.0) 425 : cluster [INF] osd.4 v2:172.21.15.164:6800/7 boot 2021-02-10T04:24:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: cluster 2021-02-10T04:24:40.925993+0000 mon.a (mon.0) 426 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2021-02-10T04:24:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: audit 2021-02-10T04:24:40.926269+0000 mon.a (mon.0) 427 : 2021-02-10T04:24:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-02-10T04:24:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: audit 2021-02-10T04:24:40.970932+0000 mgr.y (mgr.14140 2021-02-10T04:24:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: ) 139 : audit [DBG] from='client.34205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:24:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:24:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: 04:24:40.972116+0000 mon.a (mon 2021-02-10T04:24:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: .0) 428 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-02-10T04:24:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: cephadm 2021-02-10T04:24:40. 2021-02-10T04:24:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: 973471+0000 mgr.y (mgr.14140) 140 : 2021-02-10T04:24:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: cephadm [INF] Applying service osd.None on host smithi164... 2021-02-10T04:24:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:24:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: 02-10T04:24:40. 2021-02-10T04:24:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: 973754+0000 mon.a (mon.0) 429 : audit 2021-02-10T04:24:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-02-10T04:24:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:24:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: 10T04:24:40.974498+0000 2021-02-10T04:24:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: mon.a (mon.0) 430 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:24:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: 2021-02-10T04:24:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: audit 2021-02-10T04:24:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: 2021-02-10T04:24: 2021-02-10T04:24:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: 40.975386+0000 mon.a (mon.0) 431 : 2021-02-10T04:24:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:24:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: cephadm 2021-02 2021-02-10T04:24:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: -10T04:24:41.011560 2021-02-10T04:24:42.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 141 : cephadm 2021-02-10T04:24:42.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: [INF] Detected new or changed devices on smithi164 2021-02-10T04:24:42.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:24:42.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: 10T04:24:41.015445+0000 mon.a 2021-02-10T04:24:42.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: (mon.0) 432 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:42.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: 2021-02-10T04:24:42.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:24:42.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: 04:24:41.018766+0000 mon.a (mon 2021-02-10T04:24:42.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: .0) 433 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:42.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:24:42.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: T04:24:41.023041+0000 mon.a 2021-02-10T04:24:42.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:41 smithi164 conmon[29288]: (mon.0) 434 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:42.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: cluster 2021-02-10T04:24:39.034675+0000 osd.4 ( 2021-02-10T04:24:42.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:24:42.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: cluster 2021-02-10T04:24 2021-02-10T04:24:42.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: :39.034782+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:24:42.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: audit 2021-02-10T04:24:40 2021-02-10T04:24:42.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: .920747+0000 mon.a (mon.0) 424 : audit [DBG] 2021-02-10T04:24:42.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-02-10T04:24:42.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:24:42.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: 02-10T04:24:40.925931+0000 mon.a (mon.0) 425 : cluster [INF] osd.4 v2:172.21.15.164:6800/7 boot 2021-02-10T04:24:42.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:24:42.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: T04:24:40.925993+0000 mon.a (mon.0) 426 : cluster 2021-02-10T04:24:42.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: [DBG] osdmap e31: 5 total, 5 up, 5 in 2021-02-10T04:24:42.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:24:42.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: 02-10T04:24:40.926269+0000 mon.a (mon.0) 427 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-02-10T04:24:42.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:24:42.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: 10T04:24:40.970932+0000 mgr.y (mgr.14140) 139 2021-02-10T04:24:42.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: : audit [DBG] from='client.34205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:24:42.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: audit 2021-02-10T04:24:40.972116+0000 2021-02-10T04:24:42.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: mon.a (mon.0) 428 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-02-10T04:24:42.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: cephadm 2021-02-10T04:24: 2021-02-10T04:24:42.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: 40.973471+0000 mgr.y (mgr.14140) 140 : cephadm [INF] Applying service osd.None on host smithi164... 2021-02-10T04:24:42.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:24:42.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: 02-10T04:24:40.973754+0000 2021-02-10T04:24:42.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: mon.a (mon.0) 429 : audit [INF] 2021-02-10T04:24:42.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-02-10T04:24:42.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: audit 2021-02-10T04:24: 2021-02-10T04:24:42.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: 40.974498+0000 mon.a (mon.0) 430 : 2021-02-10T04:24:42.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:24:42.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:24:42.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: 02-10T04:24:40 2021-02-10T04:24:42.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: .975386+0000 mon.a (mon.0 2021-02-10T04:24:42.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: ) 431 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:24:42.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: cephadm 2021- 2021-02-10T04:24:42.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: 02-10T04:24:41.011560+0000 mgr.y (mgr.14140) 141 : cephadm [INF] Detected new or changed devices on smithi164 2021-02-10T04:24:42.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:24:42.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: 04:24:41.015445+0000 mon.a (mon.0) 432 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:42.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:24:42.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: :24:41.018766+0000 mon.a (mon.0) 433 : audit 2021-02-10T04:24:42.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:42.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:24:42.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: -10T04:24:41.023041+0000 mon.a (mon.0) 434 2021-02-10T04:24:42.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[35961]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:42.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: cluster 2021-02-10T04:24:39.034675+0000 osd.4 (osd.4) 2021-02-10T04:24:42.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: 1 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:24:42.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: cluster 2021-02-10T04:24:39.034782+0000 osd.4 (osd. 2021-02-10T04:24:42.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: 4) 2 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:24:42.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: audit 2021-02-10T04:24:40. 2021-02-10T04:24:42.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: 920747+0000 mon.a (mon.0) 424 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-02-10T04:24:42.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: cluster 2021-02-10T04:24: 2021-02-10T04:24:42.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: 40.925931+0000 mon.a (mon.0) 2021-02-10T04:24:42.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: 425 : cluster [INF] osd.4 v2:172.21.15.164:6800/7 boot 2021-02-10T04:24:42.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: cluster 2021-02-10T04:24:42.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: 2021-02-10T04:24: 2021-02-10T04:24:42.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: 40.925993+0000 mon.a (mon.0) 426 : 2021-02-10T04:24:42.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2021-02-10T04:24:42.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:24:42.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: 02-10T04:24:40.926269+0000 2021-02-10T04:24:42.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: mon.a (mon.0) 427 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-02-10T04:24:42.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: 2021-02-10T04:24:42.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: audit 2021-02-10T04:24:40 2021-02-10T04:24:42.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: .970932+0000 mgr.y (mgr.14140) 139 : 2021-02-10T04:24:42.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: audit [DBG] from='client.34205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:24:42.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:24:42.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: 02-10T04:24:40. 2021-02-10T04:24:42.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: 972116+0000 mon.a (mon.0) 2021-02-10T04:24:42.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: 428 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-02-10T04:24:42.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: cephadm 2021-02 2021-02-10T04:24:42.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: -10T04:24:40.973471+0000 mgr.y (mgr.14140) 140 : cephadm 2021-02-10T04:24:42.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: [INF] Applying service osd.None on host smithi164... 2021-02-10T04:24:42.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: audit 2021-02-10T04:24:40.973754 2021-02-10T04:24:42.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: +0000 mon.a (mon.0) 429 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-02-10T04:24:42.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: audit 2021-02-10T04:24:40 2021-02-10T04:24:42.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: .974498+0000 mon.a (mon.0) 430 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:24:42.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: audit 2021-02-10T04:24:40.975386+0000 2021-02-10T04:24:42.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: mon.a (mon.0) 431 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:24:42.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: cephadm 2021 2021-02-10T04:24:42.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: -02-10T04:24:41.011560+0000 mgr.y (mgr. 2021-02-10T04:24:42.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: 14140) 141 : cephadm [INF] Detected new or changed devices on smithi164 2021-02-10T04:24:42.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: audit 2021-02-10T04:24:42.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: 2021-02-10T04:24: 2021-02-10T04:24:42.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: 41.015445+0000 mon.a (mon.0) 432 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:42.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: 2021-02-10T04:24:42.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: audit 2021-02-10T04:24:41. 2021-02-10T04:24:42.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: 018766+0000 mon.a (mon.0) 433 : audit [INF] 2021-02-10T04:24:42.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:42.198 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: audit 2021-02-10T04:24:41. 2021-02-10T04:24:42.198 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:41 smithi085 conmon[29319]: 023041+0000 mon.a (mon.0) 434 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:43.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:42 smithi164 conmon[29288]: cluster 2021-02-10T04:24:41.929777+0000 mon.a (mon.0) 435 : cluster 2021-02-10T04:24:43.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:42 smithi164 conmon[29288]: [DBG] osdmap e32: 5 total, 5 up, 5 in 2021-02-10T04:24:43.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:42 smithi164 conmon[29288]: cluster 2021-02-10T04:24:41.933323+0000 mgr.y (mgr.14140) 142 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-02-10T04:24:43.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:42 smithi085 conmon[29319]: cluster 2021-02-10T04:24:41.929777+0000 mon.a (mon.0) 435 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2021-02-10T04:24:43.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:42 smithi085 conmon[29319]: cluster 2021-02-10T04:24:41.933323+0000 mgr.y (mgr.14140) 142 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-02-10T04:24:43.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:42 smithi085 conmon[35961]: cluster 2021-02-10T04:24:41.929777+0000 mon.a (mon.0) 435 2021-02-10T04:24:43.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:42 smithi085 conmon[35961]: : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2021-02-10T04:24:43.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:42 smithi085 conmon[35961]: cluster 2021-02-10T04:24:41.933323+0000 mgr.y (mgr.14140) 142 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-02-10T04:24:44.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[35961]: audit 2021-02-10T04:24:43.362339+0000 mon.b (mon.2) 2021-02-10T04:24:44.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[35961]: 9 : audit [INF] from='client.? 172.21.15.164:0/1845790735' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e"}]: dispatch 2021-02-10T04:24:44.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[35961]: audit 2021-02-10T04:24:43.362915+0000 mon.a (mon.0) 436 2021-02-10T04:24:44.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[35961]: : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e"}]: dispatch 2021-02-10T04:24:44.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[35961]: audit 2021-02-10T04:24:43 2021-02-10T04:24:44.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[35961]: .367120+0000 mon.a (mon.0) 437 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e"}]': finished 2021-02-10T04:24:44.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[35961]: cluster 2021-02-10T04:24:44.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[35961]: 2021-02-10T04:24:43.367207+0000 mon.a (mon.0) 438 : cluster [DBG] 2021-02-10T04:24:44.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[35961]: osdmap e33: 6 total, 5 up, 5 in 2021-02-10T04:24:44.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[35961]: audit 2021 2021-02-10T04:24:44.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[35961]: -02-10T04:24:43.367358+0000 mon.a (mon.0) 439 : audit 2021-02-10T04:24:44.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[35961]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-02-10T04:24:44.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[35961]: audit 2021 2021-02-10T04:24:44.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[35961]: -02-10T04:24:43.851520+0000 mon.b (mon.2) 10 : audit [DBG] from='client.? 172.21.15.164:0/2862433256' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-02-10T04:24:44.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[35961]: 2021-02-10T04:24:44.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[29319]: audit 2021-02-10T04:24:43.362339+0000 mon.b (mon.2) 9 : audit [INF] from='client.? 172.21.15.164:0/1845790735' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e"}]: dispatch 2021-02-10T04:24:44.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[29319]: 2021-02-10T04:24:44.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[29319]: audit 2021-02-10T04:24:43.362915+0000 mon.a (mon.0) 436 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e"}]: dispatch 2021-02-10T04:24:44.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:24:44.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[29319]: T04:24:43.367120+0000 2021-02-10T04:24:44.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[29319]: mon.a (mon.0) 437 : audit 2021-02-10T04:24:44.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[29319]: [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e"}]': finished 2021-02-10T04:24:44.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:24:44.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[29319]: 10T04:24:43.367207+0000 2021-02-10T04:24:44.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[29319]: mon.a (mon.0) 438 : cluster [DBG] osdmap e33: 6 total, 5 up, 5 in 2021-02-10T04:24:44.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[29319]: 2021-02-10T04:24:44.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:24:44.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[29319]: 10T04:24:43.367358 2021-02-10T04:24:44.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[29319]: +0000 mon.a (mon.0) 439 : audit [DBG] 2021-02-10T04:24:44.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-02-10T04:24:44.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:24:44.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[29319]: -10T04:24:43.851520 2021-02-10T04:24:44.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[29319]: +0000 mon.b (mon.2) 10 : audit [DBG] 2021-02-10T04:24:44.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:43 smithi085 conmon[29319]: from='client.? 172.21.15.164:0/2862433256' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-02-10T04:24:44.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:43 smithi164 conmon[29288]: audit 2021-02-10T04:24:43.362339+0000 mon.b (mon.2) 2021-02-10T04:24:44.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:43 smithi164 conmon[29288]: 9 : audit [INF] from='client.? 172.21.15.164:0/1845790735' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e"}]: dispatch 2021-02-10T04:24:44.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:43 smithi164 conmon[29288]: audit 2021-02-10T04:24:43.362915+0000 mon.a (mon.0) 436 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e"}]: dispatch 2021-02-10T04:24:44.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:43 smithi164 conmon[29288]: audit 2021-02-10T04:24:43.367120+0000 mon.a (mon.0) 437 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e"}]': finished 2021-02-10T04:24:44.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:43 smithi164 conmon[29288]: cluster 2021-02-10T04:24:43.367207+0000 mon.a (mon.0) 438 : cluster [DBG] osdmap e33: 6 total, 5 up, 5 in 2021-02-10T04:24:44.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:43 smithi164 conmon[29288]: audit 2021-02-10T04:24: 2021-02-10T04:24:44.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:43 smithi164 conmon[29288]: 43.367358+0000 mon.a (mon.0) 439 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-02-10T04:24:44.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:43 smithi164 conmon[29288]: audit 2021-02-10T04:24:43.851520+0000 mon.b ( 2021-02-10T04:24:44.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:43 smithi164 conmon[29288]: mon.2) 10 : audit [DBG] from='client.? 172.21.15.164:0/2862433256' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-02-10T04:24:45.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:44 smithi085 conmon[29319]: cluster 2021-02-10T04:24:43.933621+0000 mgr.y (mgr.14140) 143 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-02-10T04:24:45.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:44 smithi085 conmon[29319]: 2021-02-10T04:24:45.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:44 smithi085 conmon[35961]: cluster 2021-02-10T04:24:43.933621+0000 mgr.y (mgr.14140) 143 : cluster 2021-02-10T04:24:45.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:44 smithi085 conmon[35961]: [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-02-10T04:24:45.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:44 smithi164 conmon[29288]: cluster 2021-02-10T04:24:43.933621+0000 mgr.y (mgr.14140) 143 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-02-10T04:24:47.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:46 smithi164 conmon[29288]: cluster 2021-02-10T04:24:45.933999+0000 mgr.y (mgr.14140) 144 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-02-10T04:24:47.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:46 smithi164 conmon[29288]: cluster 2021-02-10T04:24:46.104768+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:24:47.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:46 smithi164 conmon[29288]: cluster 2021-02-10T04:24:46.104885+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:24:47.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:46 smithi164 conmon[29288]: audit 2021-02-10T04:24:46.656373+0000 mon.a (mon.0) 440 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:24:47.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:46 smithi085 conmon[35961]: cluster 2021-02-10T04:24:45.933999+0000 mgr.y (mgr 2021-02-10T04:24:47.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:46 smithi085 conmon[35961]: .14140) 144 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-02-10T04:24:47.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:46 smithi085 conmon[35961]: cluster 2021-02-10T04:24: 2021-02-10T04:24:47.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:46 smithi085 conmon[35961]: 46.104768+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:24:47.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:46 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:24:47.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:46 smithi085 conmon[35961]: 10T04:24:46.104885+0000 osd.1 (osd.1) 2021-02-10T04:24:47.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:46 smithi085 conmon[35961]: 4 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:24:47.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:46 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:24:47.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:46 smithi085 conmon[35961]: -10T04:24:46.656373+0000 mon.a ( 2021-02-10T04:24:47.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:46 smithi085 conmon[35961]: mon.0) 440 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:24:47.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:46 smithi085 conmon[29319]: cluster 2021-02-10T04:24:45.933999+0000 mgr.y (mgr.14140) 144 2021-02-10T04:24:47.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:46 smithi085 conmon[29319]: : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-02-10T04:24:47.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:46 smithi085 conmon[29319]: cluster 2021-02-10T04:24:46.104768+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:24:47.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:46 smithi085 conmon[29319]: 2021-02-10T04:24:47.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:46 smithi085 conmon[29319]: cluster 2021-02-10T04:24:47.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:46 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:24:47.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:46 smithi085 conmon[29319]: :24:46.104885+0000 2021-02-10T04:24:47.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:46 smithi085 conmon[29319]: osd.1 (osd.1) 4 : 2021-02-10T04:24:47.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:46 smithi085 conmon[29319]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:24:47.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:46 smithi085 conmon[29319]: audit 2021 2021-02-10T04:24:47.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:46 smithi085 conmon[29319]: -02-10T04:24:46.656373+0000 2021-02-10T04:24:47.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:46 smithi085 conmon[29319]: mon.a (mon.0) 440 : audit 2021-02-10T04:24:47.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:46 smithi085 conmon[29319]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:24:48.214 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:47 smithi164 conmon[29288]: cluster 2021-02-10T04:24:46.948056+0000 mon.a (mon.0) 2021-02-10T04:24:48.214 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:47 smithi164 conmon[29288]: 441 : cluster [DBG] osdmap e34: 6 total, 5 up, 5 in 2021-02-10T04:24:48.214 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:47 smithi164 conmon[29288]: audit 2021-02-10T04:24:46.948518+0000 mon.a (mon.0) 442 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-02-10T04:24:48.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:47 smithi085 conmon[29319]: cluster 2021-02-10T04:24:46.948056+0000 mon.a (mon.0) 441 : cluster [DBG] osdmap e34: 6 total, 5 up, 5 in 2021-02-10T04:24:48.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:47 smithi085 conmon[29319]: audit 2021-02-10T04:24:46.948518+0000 mon.a (mon.0) 442 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-02-10T04:24:48.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:47 smithi085 conmon[35961]: cluster 2021-02-10T04:24:46.948056+0000 mon.a (mon.0) 441 : cluster [DBG] 2021-02-10T04:24:48.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:47 smithi085 conmon[35961]: osdmap e34: 6 total, 5 up, 5 in 2021-02-10T04:24:48.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:47 smithi085 conmon[35961]: audit 2021-02-10T04:24:46.948518+0000 mon.a (mon.0) 442 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-02-10T04:24:49.219 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:48 smithi164 conmon[29288]: cluster 2021-02-10T04:24:47.934306+0000 mgr.y (mgr.14140) 145 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-02-10T04:24:49.976 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:48 smithi085 conmon[29319]: cluster 2021-02-10T04:24:47.934306+0000 mgr.y (mgr.14140) 145 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-02-10T04:24:49.976 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:48 smithi085 conmon[35961]: cluster 2021-02-10T04:24:47 2021-02-10T04:24:49.976 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:48 smithi085 conmon[35961]: .934306+0000 mgr.y (mgr.14140) 145 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-02-10T04:24:50.061 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:49 smithi164 conmon[29288]: audit 2021-02-10T04:24:49.241595+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-02-10T04:24:50.061 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:49 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:24:50.061 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:49 smithi164 conmon[29288]: 04:24:49.242464+0000 mon.a (mon.0) 444 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:24:50.061 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:49 smithi164 conmon[29288]: audit 2021-02-10T04:24:49.243612+0000 mon.a (mon.0) 445 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.5", "key": "container_image"}]: dispatch 2021-02-10T04:24:50.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:49 smithi085 conmon[29319]: audit 2021-02-10T04:24:49.241595+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-02-10T04:24:50.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:49 smithi085 conmon[29319]: audit 2021-02-10T04:24:49.242464+0000 mon.a (mon.0) 444 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:24:50.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:49 smithi085 conmon[29319]: audit 2021-02-10T04:24:49.243612+0000 mon.a (mon.0) 445 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.5", "key": "container_image"}]: dispatch 2021-02-10T04:24:50.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:49 smithi085 conmon[29319]: 2021-02-10T04:24:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:49 smithi085 conmon[35961]: audit 2021-02-10T04:24:49.241595+0000 mon.a (mon.0) 443 2021-02-10T04:24:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:49 smithi085 conmon[35961]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-02-10T04:24:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:49 smithi085 conmon[35961]: audit 2021-02-10T04:24:49.242464+0000 mon.a (mon.0 2021-02-10T04:24:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:49 smithi085 conmon[35961]: ) 444 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:24:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:49 smithi085 conmon[35961]: audit 2021-02-10T04:24:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:49 smithi085 conmon[35961]: 2021-02-10T04:24:49.243612+0000 mon.a (mon.0 2021-02-10T04:24:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:49 smithi085 conmon[35961]: ) 445 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.5", "key": "container_image"}]: dispatch 2021-02-10T04:24:51.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:50 smithi164 conmon[29288]: cephadm 2021-02-10T04:24:49.243153+0000 mgr.y (mgr 2021-02-10T04:24:56.561 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:50 smithi164 conmon[29288]: .14140) 146 : cephadm [INF] Deploying daemon osd.5 on smithi164 2021-02-10T04:24:56.562 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:50 smithi164 conmon[29288]: cluster 2021-02-10T04:24:49.934772+0000 mgr.y (mgr.14140) 147 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-02-10T04:24:56.564 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:50 smithi085 conmon[29319]: cephadm 2021-02-10T04:24:49.243153+0000 mgr.y (mgr.14140) 2021-02-10T04:24:56.564 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:50 smithi085 conmon[29319]: 146 : cephadm [INF] Deploying daemon osd.5 on smithi164 2021-02-10T04:24:56.564 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:50 smithi085 conmon[29319]: cluster 2021-02-10T04:24:49.934772+0000 mgr.y (mgr.14140) 147 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-02-10T04:24:56.564 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:52 smithi085 conmon[29319]: cluster 2021-02-10T04:24:51.935156+0000 mgr.y (mgr.14140) 148 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-02-10T04:24:56.564 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:52 smithi085 conmon[29319]: audit 2021-02-10T04:24:52.520882+0000 mon.a (mon.0) 446 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:24:56.564 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:52 smithi085 conmon[29319]: audit 2021-02-10T04:24:52.523724+0000 mon.a (mon.0) 2021-02-10T04:24:56.564 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:52 smithi085 conmon[29319]: 447 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:56.564 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:52 smithi085 conmon[29319]: audit 2021-02-10T04: 2021-02-10T04:24:56.565 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:52 smithi085 conmon[29319]: 24:52.530321+0000 mon.a (mon.0) 448 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:56.565 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:52 smithi085 conmon[29319]: 2021-02-10T04:24:56.565 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:54 smithi085 conmon[29319]: audit 2021-02-10T04:24:53.027486+0000 mon.a (mon.0) 449 : audit 2021-02-10T04:24:56.565 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:54 smithi085 conmon[29319]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:56.565 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:54 smithi085 conmon[29319]: audit 2021-02-10T04:24:53.682882+0000 mon.a (mon.0) 450 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:56.565 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:54 smithi085 conmon[29319]: audit 2021-02-10T04:24:56.565 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:54 smithi085 conmon[29319]: 2021-02-10T04:24:53.684029+0000 mon.a (mon.0) 451 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:24:56.565 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:55 smithi085 conmon[29319]: cluster 2021-02-10T04:24:53.935539+0000 mgr.y 2021-02-10T04:24:56.565 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:55 smithi085 conmon[29319]: (mgr.14140) 149 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-02-10T04:24:56.565 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:56 smithi085 conmon[29319]: audit 2021-02-10T04:24:55.694633+0000 mon.b 2021-02-10T04:24:56.565 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:56 smithi085 conmon[29319]: (mon.2) 11 : audit [INF] from='osd.5 v2:172.21.15.164:6804/7' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-02-10T04:24:56.565 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:56 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:24:56.565 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:56 smithi085 conmon[29319]: 10T04:24:55.695187+0000 mon.a (mon.0) 452 : audit 2021-02-10T04:24:56.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:56 smithi085 conmon[29319]: [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-02-10T04:24:56.566 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:50 smithi085 conmon[35961]: cephadm 2021-02-10T04:24:49.243153+0000 mgr.y (mgr.14140) 146 2021-02-10T04:24:56.566 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:50 smithi085 conmon[35961]: : cephadm [INF] Deploying daemon osd.5 on smithi164 2021-02-10T04:24:56.566 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:50 smithi085 conmon[35961]: cluster 2021-02-10T04:24:49.934772+0000 mgr.y (mgr.14140) 147 2021-02-10T04:24:56.566 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:50 smithi085 conmon[35961]: : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-02-10T04:24:56.566 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:52 smithi085 conmon[35961]: cluster 2021-02-10T04:24:51.935156+0000 mgr.y (mgr.14140) 148 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-02-10T04:24:56.566 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:52 smithi085 conmon[35961]: audit 2021-02-10T04:24:52.520882+0000 mon.a (mon.0) 446 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:24:56.566 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:52 smithi085 conmon[35961]: audit 2021-02-10T04:24:56.566 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:52 smithi085 conmon[35961]: 2021-02-10T04:24:52.523724+0000 mon.a (mon.0) 447 : audit [INF] 2021-02-10T04:24:56.566 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:52 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:56.566 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:52 smithi085 conmon[35961]: audit 2021-02-10T04:24 2021-02-10T04:24:56.566 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:52 smithi085 conmon[35961]: :52.530321+0000 mon.a (mon.0) 448 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:56.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:54 smithi085 conmon[35961]: audit 2021-02-10T04:24:53.027486+0000 2021-02-10T04:24:56.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:54 smithi085 conmon[35961]: mon.a (mon.0) 449 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:56.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:54 smithi085 conmon[35961]: audit 2021-02-10T04:24:56.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:54 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:24:56.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:54 smithi085 conmon[35961]: T04:24:53.682882+0000 2021-02-10T04:24:56.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:54 smithi085 conmon[35961]: mon.a (mon.0) 450 : audit 2021-02-10T04:24:56.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:54 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:56.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:54 smithi085 conmon[35961]: audit 2021 2021-02-10T04:24:56.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:54 smithi085 conmon[35961]: -02-10T04:24:53.684029+0000 2021-02-10T04:24:56.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:54 smithi085 conmon[35961]: mon.a (mon.0) 451 : 2021-02-10T04:24:56.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:54 smithi085 conmon[35961]: audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:24:56.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:55 smithi085 conmon[35961]: cluster 2021-02-10T04:24:53.935539+0000 mgr.y (mgr 2021-02-10T04:24:56.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:55 smithi085 conmon[35961]: .14140) 149 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-02-10T04:24:56.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:56 smithi085 conmon[35961]: audit 2021-02-10T04:24 2021-02-10T04:24:56.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:56 smithi085 conmon[35961]: :55.694633+0000 mon.b (mon.2) 11 : audit [INF] from='osd.5 v2:172.21.15.164:6804/7' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-02-10T04:24:56.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:56 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:24:56.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:56 smithi085 conmon[35961]: T04:24:55.695187+0000 2021-02-10T04:24:56.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:56 smithi085 conmon[35961]: mon.a (mon.0) 452 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-02-10T04:24:56.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:56 smithi085 conmon[35961]: 2021-02-10T04:24:56.568 INFO:teuthology.orchestra.run.smithi164.stdout:Created osd(s) 5 on host 'smithi164' 2021-02-10T04:24:56.569 DEBUG:teuthology.orchestra.run.smithi164:osd.5> sudo journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.5.service 2021-02-10T04:24:56.569 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:52 smithi164 conmon[29288]: cluster 2021-02-10T04:24:51.935156+0000 mgr.y (mgr.14140) 148 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-02-10T04:24:56.569 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:52 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:24:56.569 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:52 smithi164 conmon[29288]: :24:52.520882+0000 mon.a (mon.0) 446 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:24:56.569 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:52 smithi164 conmon[29288]: audit 2021-02-10T04:24:52.523724+0000 mon.a (mon.0) 447 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:56.569 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:52 smithi164 conmon[29288]: audit 2021-02-10T04:24:52.530321+0000 mon.a (mon.0) 448 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:56.569 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:54 smithi164 conmon[29288]: audit 2021-02-10T04:24:53.027486+0000 mon.a (mon.0) 449 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:56.569 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:54 smithi164 conmon[29288]: audit 2021-02-10T04:24:53.682882+0000 mon.a (mon.0) 450 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:56.570 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:54 smithi164 conmon[29288]: audit 2021-02-10T04:24:53.684029+0000 mon.a (mon.0) 451 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:24:56.570 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:55 smithi164 conmon[29288]: cluster 2021-02-10T04:24:53.935539+0000 mgr.y 2021-02-10T04:24:56.570 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:55 smithi164 conmon[29288]: (mgr.14140) 149 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-02-10T04:24:56.570 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:56 smithi164 conmon[29288]: audit 2021-02-10T04:24:55.694633+0000 mon.b ( 2021-02-10T04:24:56.570 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:56 smithi164 conmon[29288]: mon.2) 11 : audit [INF] from='osd.5 v2:172.21.15.164:6804/7' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-02-10T04:24:56.570 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:56 smithi164 conmon[29288]: audit 2021-02-10T04:24:55.695187+0000 mon.a (mon.0) 452 : 2021-02-10T04:24:56.570 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:56 smithi164 conmon[29288]: audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-02-10T04:24:56.572 INFO:tasks.cephadm:Deploying osd.6 on smithi164 with /dev/vg_nvme/lv_2... 2021-02-10T04:24:56.572 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2021-02-10T04:24:56.603 INFO:journalctl@ceph.osd.5.smithi164.stdout:-- Logs begin at Wed 2021-02-10 04:09:33 UTC. -- 2021-02-10T04:24:57.345 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:24:57 smithi164 conmon[37263]: debug 2021-02-10T04:24:57.044+0000 7f1c88ae7700 -1 osd.5 0 waiting for initial osdmap 2021-02-10T04:24:57.345 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:57 smithi164 conmon[29288]: cluster 2021-02-10T04:24:55.935958+0000 mgr.y (mgr. 2021-02-10T04:24:57.345 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:57 smithi164 conmon[29288]: 14140) 150 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-02-10T04:24:57.345 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:57 smithi164 conmon[29288]: audit 2021-02-10T04:24:56.037435+0000 mon.a (mon.0) 453 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-02-10T04:24:57.345 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:57 smithi164 conmon[29288]: cluster 2021-02-10T04:24:56.037542+0000 mon.a (mon.0) 454 : cluster [DBG] osdmap e35: 6 total, 5 up, 5 in 2021-02-10T04:24:57.345 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:57 smithi164 conmon[29288]: audit 2021-02-10T04:24:56.037696+0000 mon.a (mon.0) 455 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-02-10T04:24:57.345 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:57 smithi164 conmon[29288]: audit 2021-02-10T04:24:56.038215+0000 mon.b (mon.2) 12 : audit [INF] from='osd.5 v2:172.21.15.164:6804/7' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2021-02-10T04:24:57.345 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:57 smithi164 conmon[29288]: audit 2021-02-10T04:24:56.038530+0000 mon.a (mon.0) 456 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2021-02-10T04:24:57.346 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:57 smithi164 conmon[29288]: audit 2021-02-10T04:24:56.596689+0000 mon.a (mon.0) 457 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:24:57.346 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:57 smithi164 conmon[29288]: audit 2021-02-10T04:24:56.941132+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:57.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[35961]: cluster 2021-02-10T04:24:55.935958+0000 mgr.y ( 2021-02-10T04:24:57.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[35961]: mgr.14140) 150 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-02-10T04:24:57.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[35961]: audit 2021-02-10T04:24:57.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[35961]: 2021-02-10T04:24:56.037435+0000 mon.a (mon.0) 453 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-02-10T04:24:57.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[35961]: cluster 2021-02-10T04:24:57.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[35961]: 2021-02-10T04:24:56.037542+0000 mon.a (mon.0) 454 : cluster [DBG] osdmap e35: 6 total, 5 up, 5 in 2021-02-10T04:24:57.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[35961]: audit 2021-02-10T04:24:56. 2021-02-10T04:24:57.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[35961]: 037696+0000 mon.a (mon.0) 455 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-02-10T04:24:57.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[35961]: audit 2021-02-10T04:24:56.038215+0000 mon.b (mon.2) 12 : audit 2021-02-10T04:24:57.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[35961]: [INF] from='osd.5 v2:172.21.15.164:6804/7' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2021-02-10T04:24:57.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[35961]: audit 2021-02-10T04:24 2021-02-10T04:24:57.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[35961]: :56.038530+0000 mon.a (mon.0) 456 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2021-02-10T04:24:57.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:24:57.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[35961]: 02-10T04:24:56.596689+0000 mon.a (mon.0) 457 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:24:57.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[35961]: audit 2021-02-10T04:24:56.941132+0000 mon.a (mon.0) 458 : audit [INF] 2021-02-10T04:24:57.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:57.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[29319]: cluster 2021-02-10T04:24:55.935958+0000 mgr.y (mgr.14140) 150 2021-02-10T04:24:57.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[29319]: : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-02-10T04:24:57.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[29319]: audit 2021-02-10T04:24:56.037435+0000 mon.a (mon.0) 453 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-02-10T04:24:57.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[29319]: cluster 2021-02-10T04:24:57.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[29319]: 2021-02-10T04:24:56.037542 2021-02-10T04:24:57.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[29319]: +0000 mon.a (mon.0) 454 : cluster [DBG] osdmap e35: 6 total, 5 up, 5 in 2021-02-10T04:24:57.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[29319]: 2021-02-10T04:24:57.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[29319]: audit 2021-02-10T 2021-02-10T04:24:57.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[29319]: 04:24:56.037696+0000 mon.a (mon. 2021-02-10T04:24:57.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[29319]: 0) 455 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-02-10T04:24:57.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[29319]: audit 2021-02-10T04:24:57.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:24:57.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[29319]: 04:24:56.038215+0000 mon.b (mon.2 2021-02-10T04:24:57.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[29319]: ) 12 : audit [INF] from='osd.5 v2:172.21.15.164:6804/7' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2021-02-10T04:24:57.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[29319]: audit 2021 2021-02-10T04:24:57.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[29319]: -02-10T04:24:56.038530+0000 2021-02-10T04:24:57.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[29319]: mon.a (mon.0) 456 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2021-02-10T04:24:57.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[29319]: 2021-02-10T04:24:57.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:24:57.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[29319]: :24:56.596689+0000 mon.a (mon.0) 457 2021-02-10T04:24:57.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[29319]: : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:24:57.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:24:57.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[29319]: -10T04:24:56.941132+0000 mon.a 2021-02-10T04:24:57.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:57 smithi085 conmon[29319]: (mon.0) 458 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:58.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:58 smithi164 conmon[29288]: cephadm 2021-02-10T04:24:56.936545+0000 mgr.y (mgr.14140) 151 : cephadm [INF] Detected new or changed devices on smithi164 2021-02-10T04:24:58.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:58 smithi164 conmon[29288]: cephadm 2021-02-10T04:24:56.941840+0000 mgr.y (mgr.14140) 152 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:24:58.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:58 smithi164 conmon[29288]: audit 2021-02-10T04:24:57.039418+0000 mon.a (mon.0) 459 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2021-02-10T04:24:58.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:58 smithi164 conmon[29288]: cluster 2021-02-10T04:24:57.039524+0000 mon.a (mon.0) 460 : cluster [DBG] osdmap e36: 6 total, 5 up, 5 in 2021-02-10T04:24:58.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:58 smithi164 conmon[29288]: audit 2021-02-10T04:24:57.040094+0000 mon.a (mon.0) 461 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-02-10T04:24:58.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:58 smithi164 conmon[29288]: audit 2021-02-10T04:24:57.069829+0000 mon.a (mon.0) 462 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-02-10T04:24:58.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:58 smithi164 conmon[29288]: audit 2021-02-10T04:24:57.325449+0000 mon.a (mon.0) 463 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:58.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:58 smithi164 conmon[29288]: audit 2021-02-10T04:24 2021-02-10T04:24:58.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:58 smithi164 conmon[29288]: :57.331523+0000 mon.a (mon.0) 464 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:58.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[29319]: cephadm 2021-02-10T04:24:56.936545+0000 mgr.y (mgr.14140) 151 : cephadm [INF] Detected new or changed devices on smithi164 2021-02-10T04:24:58.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[29319]: cephadm 2021-02-10T04:24:56.941840+0000 mgr.y (mgr.14140) 152 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:24:58.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[29319]: audit 2021-02-10T04:24:57.039418+0000 mon.a (mon.0) 459 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2021-02-10T04:24:58.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[29319]: cluster 2021-02-10T04:24:57.039524+0000 mon.a (mon.0) 460 : cluster [DBG] osdmap e36: 6 total, 5 up, 5 in 2021-02-10T04:24:58.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[29319]: audit 2021-02-10T04:24:57. 2021-02-10T04:24:58.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[29319]: 040094+0000 mon.a (mon.0) 461 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-02-10T04:24:58.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[29319]: audit 2021-02-10T04:24:57. 2021-02-10T04:24:58.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[29319]: 069829+0000 mon.a (mon.0) 462 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-02-10T04:24:58.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[29319]: audit 2021-02-10T04:24:57 2021-02-10T04:24:58.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[29319]: .325449+0000 mon.a (mon.0) 463 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:58.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[29319]: audit 2021-02-10T04:24:58.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[29319]: 2021-02-10T04:24:57.331523+0000 mon.a (mon.0) 464 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:58.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[35961]: cephadm 2021-02-10T04:24:56.936545+0000 mgr.y (mgr.14140 2021-02-10T04:24:58.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[35961]: ) 151 : cephadm [INF] Detected new or changed devices on smithi164 2021-02-10T04:24:58.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[35961]: cephadm 2021-02-10T04:24:56.941840+0000 mgr.y ( 2021-02-10T04:24:58.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[35961]: mgr.14140) 152 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:24:58.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[35961]: audit 2021-02-10T04:24:58.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[35961]: 2021-02-10T04:24:57.039418 2021-02-10T04:24:58.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[35961]: +0000 mon.a (mon.0) 459 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2021-02-10T04:24:58.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:24:58.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[35961]: 02-10T04:24:57.039524+0000 mon.a (mon. 2021-02-10T04:24:58.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[35961]: 0) 460 : cluster [DBG] osdmap e36: 6 total, 5 up, 5 in 2021-02-10T04:24:58.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:24:58.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[35961]: 04:24:57.040094+0000 mon.a (mon.0) 461 : audit 2021-02-10T04:24:58.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[35961]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-02-10T04:24:58.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:24:58.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[35961]: 04:24:57.069829+0000 2021-02-10T04:24:58.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[35961]: mon.a (mon.0) 462 2021-02-10T04:24:58.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[35961]: : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-02-10T04:24:58.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[35961]: audit 2021-02-10T04:24:58.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[35961]: 2021-02-10T04:24:58.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[35961]: 2021-02-10T04:24 2021-02-10T04:24:58.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[35961]: :57.325449+0000 mon.a ( 2021-02-10T04:24:58.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[35961]: mon.0) 463 : audit [INF] 2021-02-10T04:24:58.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:58.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:24:58.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[35961]: 02-10T04:24:57.331523 2021-02-10T04:24:58.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[35961]: +0000 mon.a (mon.0) 464 2021-02-10T04:24:58.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:58 smithi085 conmon[35961]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:24:58.528 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping: /dev/vg_nvme/lv_2 2021-02-10T04:24:58.528 INFO:teuthology.orchestra.run.smithi164.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2021-02-10T04:24:58.528 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10+0 records in 2021-02-10T04:24:58.528 INFO:teuthology.orchestra.run.smithi164.stderr:10+0 records out 2021-02-10T04:24:58.528 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0107565 s, 975 MB/s 2021-02-10T04:24:58.528 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping successful for: 2021-02-10T04:24:59.210 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph orch daemon add osd smithi164:vg_nvme/lv_2 2021-02-10T04:24:59.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:59 smithi164 conmon[29288]: cluster 2021-02-10T04:24:56.700922+0000 osd.5 (osd.5) 1 2021-02-10T04:24:59.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:59 smithi164 conmon[29288]: : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:24:59.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:59 smithi164 conmon[29288]: cluster 2021-02-10T04:24:56.701045+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:24:59.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:59 smithi164 conmon[29288]: cluster 2021-02-10T04:24:57.936364+0000 mgr.y (mgr.14140) 153 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-02-10T04:24:59.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:59 smithi164 conmon[29288]: cluster 2021-02-10T04:24:58.041806+0000 mon.a (mon.0) 465 : cluster [INF] osd.5 v2:172.21.15.164:6804/7 boot 2021-02-10T04:24:59.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:59 smithi164 conmon[29288]: cluster 2021-02-10T04:24:58.041884+0000 mon.a (mon.0) 466 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2021-02-10T04:24:59.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:24:59 smithi164 conmon[29288]: audit 2021-02-10T04:24:58.042086+0000 mon.a (mon.0) 467 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-02-10T04:24:59.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[29319]: cluster 2021-02-10T04:24:56.700922+0000 osd.5 (osd.5) 1 : cluster 2021-02-10T04:24:59.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[29319]: [DBG] purged_snaps scrub starts 2021-02-10T04:24:59.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[29319]: cluster 2021-02-10T04:24:56.701045+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:24:59.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:24:59.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[29319]: 04:24:57.936364+0000 mgr.y (mgr.14140) 153 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-02-10T04:24:59.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[29319]: cluster 2021-02-10T04:24:59.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[29319]: 2021-02-10T04:24:58.041806+0000 2021-02-10T04:24:59.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[29319]: mon.a (mon.0) 465 : cluster [INF] osd.5 v2:172.21.15.164:6804/7 boot 2021-02-10T04:24:59.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:24:59.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[29319]: 02-10T04:24:58.041884+0000 mon.a (mon.0 2021-02-10T04:24:59.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[29319]: ) 466 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2021-02-10T04:24:59.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:24:59.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[29319]: 10T04:24:58.042086+0000 mon.a (mon 2021-02-10T04:24:59.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[29319]: .0) 467 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-02-10T04:24:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[35961]: cluster 2021-02-10T04:24:56. 2021-02-10T04:24:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[35961]: 700922+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:24:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[35961]: cluster 2021-02-10T04:24:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[35961]: 2021-02-10T04:24 2021-02-10T04:24:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[35961]: :56.701045+0000 osd.5 (osd. 2021-02-10T04:24:59.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[35961]: 5) 2 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:24:59.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[35961]: 2021-02-10T04:24:59.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:24:59.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[35961]: 02-10T04:24:57.936364+0000 mgr.y ( 2021-02-10T04:24:59.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[35961]: mgr.14140) 153 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-02-10T04:24:59.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[35961]: 2021-02-10T04:24:59.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:24:59.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[35961]: 10T04:24:58.041806+0000 mon.a (mon. 2021-02-10T04:24:59.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[35961]: 0) 465 : cluster [INF] osd.5 v2:172.21.15.164:6804/7 boot 2021-02-10T04:24:59.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:24:59.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[35961]: T04:24:58.041884+0000 mon.a (mon.0) 466 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2021-02-10T04:24:59.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[35961]: 2021-02-10T04:24:59.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:24:59.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:24:59 smithi085 conmon[35961]: 04:24:58.042086+0000 mon.a (mon.0) 467 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-02-10T04:25:00.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:00 smithi164 conmon[29288]: cluster 2021-02-10T04:24:59.052903+0000 mon.a (mon.0) 468 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2021-02-10T04:25:00.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:00 smithi085 conmon[29319]: cluster 2021-02-10T04:24:59.052903+0000 mon.a (mon.0) 468 : cluster 2021-02-10T04:25:00.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:00 smithi085 conmon[29319]: [DBG] osdmap e38: 6 total, 6 up, 6 in 2021-02-10T04:25:00.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:00 smithi085 conmon[35961]: cluster 2021-02-10T04:24:59. 2021-02-10T04:25:00.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:00 smithi085 conmon[35961]: 052903+0000 mon.a (mon.0) 468 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2021-02-10T04:25:01.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:01 smithi164 conmon[29288]: cluster 2021-02-10T04:24:59.936761+0000 mgr.y (mgr 2021-02-10T04:25:01.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:01 smithi164 conmon[29288]: .14140) 154 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-02-10T04:25:01.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:01 smithi164 conmon[29288]: audit 2021-02-10T04:25:00.989700+0000 mon.a (mon.0) 469 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-02-10T04:25:01.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:01 smithi164 conmon[29288]: audit 2021-02-10T04:25:00.991354+0000 mon.a (mon.0) 470 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-02-10T04:25:01.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:01 smithi164 conmon[29288]: audit 2021-02-10T04:25:00.992126+0000 mon.a (mon.0) 471 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:25:01.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:01 smithi164 conmon[29288]: audit 2021-02-10T04:25:00.993031+0000 mon.a (mon.0) 472 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:25:01.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:01 smithi085 conmon[29319]: cluster 2021-02-10T04:24:59.936761+0000 mgr.y (mgr.14140) 154 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-02-10T04:25:01.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:01 smithi085 conmon[29319]: 2021-02-10T04:25:01.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:01 smithi085 conmon[29319]: audit 2021-02-10T04:25:00.989700+0000 mon.a (mon.0) 469 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-02-10T04:25:01.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:01 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:25:01.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:01 smithi085 conmon[29319]: 02-10T04:25:00.991354+0000 mon.a (mon.0) 470 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-02-10T04:25:01.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:01 smithi085 conmon[29319]: 2021-02-10T04:25:01.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:01 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:25:01.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:01 smithi085 conmon[29319]: T04:25:00.992126+0000 mon.a (mon. 2021-02-10T04:25:01.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:01 smithi085 conmon[29319]: 0) 471 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:25:01.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:01 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:25:01.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:01 smithi085 conmon[29319]: -10T04:25:00.993031+0000 mon.a 2021-02-10T04:25:01.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:01 smithi085 conmon[29319]: (mon.0) 472 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:25:01.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:01 smithi085 conmon[35961]: cluster 2021-02-10T04:24:59.936761+0000 mgr.y (mgr.14140) 154 : cluster 2021-02-10T04:25:01.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:01 smithi085 conmon[35961]: [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-02-10T04:25:01.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:01 smithi085 conmon[35961]: audit 2021-02-10T04:25:00.989700+0000 mon.a (mon.0) 469 : 2021-02-10T04:25:01.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:01 smithi085 conmon[35961]: audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-02-10T04:25:01.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:01 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:25:01.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:01 smithi085 conmon[35961]: 02-10T04:25:00.991354+0000 mon.a (mon.0) 470 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-02-10T04:25:01.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:01 smithi085 conmon[35961]: audit 2021-02-10T04:25:01.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:01 smithi085 conmon[35961]: 2021-02-10T04:25 2021-02-10T04:25:01.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:01 smithi085 conmon[35961]: :00.992126+0000 mon.a (mon.0) 471 2021-02-10T04:25:01.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:01 smithi085 conmon[35961]: : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:25:01.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:01 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:25:01.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:01 smithi085 conmon[35961]: -10T04:25:00.993031 2021-02-10T04:25:01.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:01 smithi085 conmon[35961]: +0000 mon.a (mon.0) 472 : audit [DBG] 2021-02-10T04:25:01.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:01 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:25:02.367 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:02 smithi164 conmon[29288]: audit 2021-02-10T04:25:00.988638+0000 mgr.y (mgr.14140) 2021-02-10T04:25:02.367 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:02 smithi164 conmon[29288]: 155 : audit [DBG] from='client.34232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:25:02.367 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:02 smithi164 conmon[29288]: cephadm 2021-02-10T04:25:00.991067+0000 mgr.y (mgr.14140) 156 : cephadm [INF] Applying service osd.None on host smithi164... 2021-02-10T04:25:02.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:02 smithi085 conmon[29319]: audit 2021-02-10T04:25:00.988638+0000 mgr.y (mgr.14140) 155 : audit [DBG] from='client.34232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:25:02.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:02 smithi085 conmon[29319]: cephadm 2021-02-10T04:25:00.991067+0000 mgr.y (mgr.14140) 156 : cephadm [INF] Applying service osd.None on host smithi164... 2021-02-10T04:25:02.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:02 smithi085 conmon[35961]: audit 2021-02-10T04:25:00.988638+0000 mgr.y (mgr.14140) 155 2021-02-10T04:25:02.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:02 smithi085 conmon[35961]: : audit [DBG] from='client.34232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:25:02.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:02 smithi085 conmon[35961]: cephadm 2021-02-10T04:25:00.991067+0000 mgr.y (mgr.14140) 156 : cephadm [INF] Applying service osd.None on host smithi164... 2021-02-10T04:25:03.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:03 smithi164 conmon[29288]: cluster 2021-02-10T04:25:01.937179+0000 mgr.y (mgr.14140) 157 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-02-10T04:25:03.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:03 smithi085 conmon[29319]: cluster 2021-02-10T04:25:01.937179+0000 2021-02-10T04:25:03.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:03 smithi085 conmon[29319]: mgr.y (mgr.14140) 157 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-02-10T04:25:03.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:03 smithi085 conmon[35961]: cluster 2021-02-10T04:25:01.937179+0000 mgr.y (mgr.14140) 2021-02-10T04:25:03.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:03 smithi085 conmon[35961]: 157 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-02-10T04:25:04.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:04 smithi164 conmon[29288]: audit 2021-02-10T04:25:03.206930+0000 mon.b (mon.2) 13 : audit 2021-02-10T04:25:04.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:04 smithi164 conmon[29288]: [INF] from='client.? 172.21.15.164:0/812042960' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "adc40430-3eeb-4003-a824-343021243077"}]: dispatch 2021-02-10T04:25:04.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:04 smithi164 conmon[29288]: audit 2021-02-10T04:25:03.207533+0000 mon.a (mon.0) 473 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "adc40430-3eeb-4003-a824-343021243077"}]: dispatch 2021-02-10T04:25:04.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:04 smithi164 conmon[29288]: audit 2021-02-10T04:25:03.213672+0000 mon.a (mon.0) 474 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "adc40430-3eeb-4003-a824-343021243077"}]': finished 2021-02-10T04:25:04.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:04 smithi164 conmon[29288]: cluster 2021-02-10T04:25:03.213739+0000 mon.a (mon.0) 475 : cluster [DBG] osdmap e39: 7 total, 6 up, 6 in 2021-02-10T04:25:04.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:04 smithi164 conmon[29288]: audit 2021-02-10T04:25:03.213880+0000 mon.a (mon.0) 476 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-02-10T04:25:04.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:04 smithi164 conmon[29288]: audit 2021-02-10T04:25:03.703931+0000 mon.b (mon.2) 14 : audit [DBG] from='client.? 172.21.15.164:0/2365601597' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-02-10T04:25:04.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:04 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:25:04.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:04 smithi164 conmon[29288]: -02-10T04:25:03.898958+0000 2021-02-10T04:25:04.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:04 smithi164 conmon[29288]: osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:25:04.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:04 smithi164 conmon[29288]: cluster 2021-02-10T04:25:04.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:04 smithi164 conmon[29288]: 2021-02-10T04:25:03.899108 2021-02-10T04:25:04.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:04 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 4 : cluster [DBG] 2021-02-10T04:25:04.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:04 smithi164 conmon[29288]: purged_snaps scrub ok 2021-02-10T04:25:04.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:04 smithi164 conmon[29288]: cluster 2021-02-10T04:25 2021-02-10T04:25:04.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:04 smithi164 conmon[29288]: :03.937594+0000 mgr.y (mgr.14140) 158 2021-02-10T04:25:04.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:04 smithi164 conmon[29288]: : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-02-10T04:25:04.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[29319]: audit 2021-02-10T04:25:03.206930+0000 mon.b (mon.2) 13 2021-02-10T04:25:04.507 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[29319]: : audit [INF] from='client.? 172.21.15.164:0/812042960' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "adc40430-3eeb-4003-a824-343021243077"}]: dispatch 2021-02-10T04:25:04.507 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[29319]: audit 2021-02-10T04:25:03.207533+0000 mon.a (mon.0) 473 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "adc40430-3eeb-4003-a824-343021243077"}]: dispatch 2021-02-10T04:25:04.507 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[29319]: audit 2021-02-10T04:25:03.213672+0000 mon.a (mon.0) 474 : audit 2021-02-10T04:25:04.507 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[29319]: [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "adc40430-3eeb-4003-a824-343021243077"}]': finished 2021-02-10T04:25:04.507 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[29319]: cluster 2021-02-10T04:25 2021-02-10T04:25:04.507 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[29319]: :03.213739+0000 mon.a (mon.0) 475 : 2021-02-10T04:25:04.507 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[29319]: cluster [DBG] osdmap e39: 7 total, 6 up, 6 in 2021-02-10T04:25:04.508 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[29319]: audit 2021-02-10T 2021-02-10T04:25:04.508 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[29319]: 04:25:03.213880+0000 mon.a (mon. 2021-02-10T04:25:04.508 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[29319]: 0) 476 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-02-10T04:25:04.508 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:25:04.508 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[29319]: 02-10T04:25:03.703931+0000 2021-02-10T04:25:04.508 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[29319]: mon.b (mon.2) 14 : audit [DBG] from='client.? 172.21.15.164:0/2365601597' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-02-10T04:25:04.508 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[29319]: 2021-02-10T04:25:04.508 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[29319]: cluster 2021-02-10T04:25:04.508 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[29319]: 2021-02-10T04:25:03.898958+0000 osd.2 2021-02-10T04:25:04.508 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[29319]: (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:25:04.508 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[29319]: cluster 2021-02-10T04:25:04.508 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[29319]: 2021-02-10T04:25:03.899108 2021-02-10T04:25:04.508 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 4 : cluster [DBG] 2021-02-10T04:25:04.508 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[29319]: purged_snaps scrub ok 2021-02-10T04:25:04.509 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:25:04.509 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[29319]: :25:03.937594+0000 mgr.y (mgr.14140 2021-02-10T04:25:04.509 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[29319]: ) 158 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-02-10T04:25:04.509 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: audit 2021-02-10T04:25:03.206930+0000 mon.b (mon 2021-02-10T04:25:04.509 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: .2) 13 : audit [INF] from='client.? 172.21.15.164:0/812042960' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "adc40430-3eeb-4003-a824-343021243077"}]: dispatch 2021-02-10T04:25:04.509 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: audit 2021-02-10T04:25:04.509 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: 2021-02-10T04:25:03.207533+0000 mon.a (mon.0) 473 : 2021-02-10T04:25:04.509 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "adc40430-3eeb-4003-a824-343021243077"}]: dispatch 2021-02-10T04:25:04.509 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:25:04.509 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: :25:03.213672+0000 mon.a (mon.0) 474 : 2021-02-10T04:25:04.509 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "adc40430-3eeb-4003-a824-343021243077"}]': finished 2021-02-10T04:25:04.510 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:25:04.510 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: -02-10T04:25:03.213739 2021-02-10T04:25:04.510 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2021-02-10T04:25:04.510 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: 475 : cluster [DBG] osdmap e39: 7 total, 6 up, 6 in 2021-02-10T04:25:04.510 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: audit 2021 2021-02-10T04:25:04.510 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: -02-10T04:25:03.213880+0000 2021-02-10T04:25:04.510 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: mon.a (mon.0) 476 : 2021-02-10T04:25:04.510 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-02-10T04:25:04.510 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: audit 2021 2021-02-10T04:25:04.510 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: -02-10T04:25:03. 2021-02-10T04:25:04.510 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: 703931+0000 mon.b (mon.2) 2021-02-10T04:25:04.511 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: 14 : audit [DBG] from='client.? 172.21.15.164:0/2365601597' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-02-10T04:25:04.511 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:25:04.511 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: -02-10T04:25:03 2021-02-10T04:25:04.511 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: .898958+0000 osd.2 (osd. 2021-02-10T04:25:04.511 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: 2) 3 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:25:04.511 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: 2021-02-10T04:25:04.511 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: cluster 2021-02-10T04:25:04.511 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: 2021-02-10T04:25:03 2021-02-10T04:25:04.511 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: .899108+0000 osd.2 (osd.2 2021-02-10T04:25:04.511 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: ) 4 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:25:04.511 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: 2021-02-10T04:25:04.511 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:25:04.512 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: -02-10T04:25:03 2021-02-10T04:25:04.512 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: .937594+0000 mgr.y (mgr.14140 2021-02-10T04:25:04.512 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:04 smithi085 conmon[35961]: ) 158 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-02-10T04:25:05.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:05 smithi164 conmon[29288]: cluster 2021-02-10T04:25:04.217605+0000 mon.a (mon.0) 477 : cluster [DBG] osdmap e40: 7 total, 6 up, 6 in 2021-02-10T04:25:05.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:05 smithi164 conmon[29288]: 2021-02-10T04:25:05.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:05 smithi164 conmon[29288]: audit 2021-02-10T04:25:04.217777+0000 mon.a (mon.0) 478 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-02-10T04:25:05.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:05 smithi085 conmon[29319]: cluster 2021-02-10T04:25:04.217605+0000 mon.a (mon.0) 477 : cluster [DBG] osdmap e40: 7 total, 6 up, 6 in 2021-02-10T04:25:05.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:05 smithi085 conmon[29319]: 2021-02-10T04:25:05.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:05 smithi085 conmon[29319]: audit 2021-02-10T04:25:04.217777+0000 mon.a (mon.0) 478 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-02-10T04:25:05.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:05 smithi085 conmon[35961]: cluster 2021-02-10T04:25:04.217605+0000 mon.a (mon.0) 477 : cluster [DBG] osdmap e40: 7 total, 6 up, 6 in 2021-02-10T04:25:05.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:05 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:25:05.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:05 smithi085 conmon[35961]: -10T04:25:04.217777+0000 mon.a (mon.0) 478 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-02-10T04:25:06.550 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:06 smithi164 conmon[29288]: cluster 2021-02-10T04:25:05.937968+0000 mgr.y (mgr.14140) 159 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-02-10T04:25:06.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:06 smithi085 conmon[29319]: cluster 2021-02-10T04:25:05.937968+0000 mgr.y (mgr.14140) 159 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-02-10T04:25:06.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:06 smithi085 conmon[35961]: cluster 2021-02-10T04:25:05.937968 2021-02-10T04:25:06.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:06 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 159 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-02-10T04:25:07.363 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:07 smithi164 conmon[29288]: audit 2021-02-10T04:25:06.567446+0000 mon.a (mon. 2021-02-10T04:25:07.363 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:07 smithi164 conmon[29288]: 0) 479 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:25:07.690 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:07 smithi085 conmon[35961]: audit 2021-02-10T04:25:06.567446+0000 mon.a (mon.0) 479 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:25:07.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:07 smithi085 conmon[29319]: audit 2021-02-10T04:25:06.567446+0000 mon.a (mon.0) 479 : audit 2021-02-10T04:25:07.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:07 smithi085 conmon[29319]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:25:09.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:08 smithi085 conmon[35961]: cluster 2021-02-10T04:25:07.938331+0000 mgr.y (mgr.14140) 160 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-02-10T04:25:09.253 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:08 smithi085 conmon[29319]: cluster 2021-02-10T04:25:07.938331+0000 mgr.y (mgr.14140) 160 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-02-10T04:25:09.253 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:08 smithi085 conmon[29319]: 2021-02-10T04:25:09.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:08 smithi164 conmon[29288]: cluster 2021-02-10T04:25:07.938331+0000 mgr.y (mgr.14140) 160 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-02-10T04:25:10.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:09 smithi085 conmon[29319]: audit 2021-02-10T04:25:09.059884+0000 mon.a (mon.0) 480 : audit [INF] 2021-02-10T04:25:10.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:09 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2021-02-10T04:25:10.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:09 smithi085 conmon[29319]: audit 2021-02-10T04:25:09.060663+0000 mon.a (mon.0) 481 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:25:10.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:09 smithi085 conmon[29319]: cephadm 2021-02-10T04:25:09. 2021-02-10T04:25:10.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:09 smithi085 conmon[29319]: 061228+0000 mgr.y (mgr.14140) 161 : cephadm [INF] Deploying daemon osd.6 on smithi164 2021-02-10T04:25:10.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:09 smithi085 conmon[29319]: audit 2021-02-10T04:25:09.061622+0000 mon.a 2021-02-10T04:25:10.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:09 smithi085 conmon[29319]: (mon.0) 482 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.6", "key": "container_image"}]: dispatch 2021-02-10T04:25:10.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:09 smithi085 conmon[35961]: audit 2021-02-10T04:25:09.059884+0000 mon.a (mon. 2021-02-10T04:25:10.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:09 smithi085 conmon[35961]: 0) 480 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2021-02-10T04:25:10.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:09 smithi085 conmon[35961]: audit 2021-02-10T04:25: 2021-02-10T04:25:10.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:09 smithi085 conmon[35961]: 09.060663+0000 mon.a (mon.0) 481 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:25:10.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:09 smithi085 conmon[35961]: cephadm 2021 2021-02-10T04:25:10.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:09 smithi085 conmon[35961]: -02-10T04:25:09 2021-02-10T04:25:10.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:09 smithi085 conmon[35961]: .061228+0000 mgr.y (mgr.14140) 161 : cephadm 2021-02-10T04:25:10.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:09 smithi085 conmon[35961]: [INF] Deploying daemon osd.6 on smithi164 2021-02-10T04:25:10.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:09 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:25:10.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:09 smithi085 conmon[35961]: -10T04:25:09.061622+0000 2021-02-10T04:25:10.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:09 smithi085 conmon[35961]: mon.a (mon.0) 482 : audit [DBG] 2021-02-10T04:25:10.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:09 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.6", "key": "container_image"}]: dispatch 2021-02-10T04:25:10.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:09 smithi164 conmon[29288]: audit 2021-02-10T04:25:09.059884 2021-02-10T04:25:10.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:09 smithi164 conmon[29288]: +0000 mon.a (mon.0) 480 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2021-02-10T04:25:10.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:09 smithi164 conmon[29288]: audit 2021-02-10T04:25:09.060663+0000 mon.a (mon.0) 481 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:25:10.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:09 smithi164 conmon[29288]: cephadm 2021-02-10T04:25:09.061228+0000 mgr.y (mgr.14140) 161 : cephadm [INF] Deploying daemon osd.6 on smithi164 2021-02-10T04:25:10.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:09 smithi164 conmon[29288]: audit 2021-02-10T04:25:09.061622+0000 mon.a (mon.0) 482 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.6", "key": "container_image"}]: dispatch 2021-02-10T04:25:11.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:10 smithi085 conmon[29319]: cluster 2021-02-10T04:25:09.938721+0000 mgr.y (mgr.14140) 162 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-02-10T04:25:11.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:10 smithi085 conmon[35961]: cluster 2021-02-10T04:25:09.938721+0000 mgr.y (mgr.14140 2021-02-10T04:25:11.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:10 smithi085 conmon[35961]: ) 162 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-02-10T04:25:11.276 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:10 smithi164 conmon[29288]: cluster 2021-02-10T04:25:09.938721+0000 mgr.y (mgr 2021-02-10T04:25:11.276 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:10 smithi164 conmon[29288]: .14140) 162 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-02-10T04:25:12.340 INFO:teuthology.orchestra.run.smithi164.stdout:Created osd(s) 6 on host 'smithi164' 2021-02-10T04:25:12.940 DEBUG:teuthology.orchestra.run.smithi164:osd.6> sudo journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.6.service 2021-02-10T04:25:12.942 INFO:tasks.cephadm:Deploying osd.7 on smithi164 with /dev/vg_nvme/lv_1... 2021-02-10T04:25:12.942 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2021-02-10T04:25:12.972 INFO:journalctl@ceph.osd.6.smithi164.stdout:-- Logs begin at Wed 2021-02-10 04:09:33 UTC. -- 2021-02-10T04:25:13.141 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:12 smithi164 conmon[29288]: cluster 2021-02-10T04:25:11.939064+0000 mgr.y (mgr 2021-02-10T04:25:13.141 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:12 smithi164 conmon[29288]: .14140) 163 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-02-10T04:25:13.141 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:12 smithi164 conmon[29288]: audit 2021-02-10T04:25:12.325657+0000 mon.a (mon.0) 483 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:25:13.141 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:12 smithi164 conmon[29288]: audit 2021-02-10T04:25:12.330668+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:13.141 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:12 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:25:13.141 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:12 smithi164 conmon[29288]: 10T04:25:12.337682+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:13.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:12 smithi085 conmon[29319]: cluster 2021-02-10T04:25:11.939064+0000 mgr.y (mgr. 2021-02-10T04:25:13.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:12 smithi085 conmon[29319]: 14140) 163 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-02-10T04:25:13.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:12 smithi085 conmon[29319]: audit 2021-02-10T04:25:12.325657+0000 mon.a (mon.0) 483 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:25:13.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:12 smithi085 conmon[29319]: audit 2021-02-10T04:25:12.330668+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:13.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:12 smithi085 conmon[29319]: audit 2021-02-10T04:25:12.337682+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:13.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:12 smithi085 conmon[35961]: cluster 2021-02-10T04:25:11.939064+0000 mgr.y (mgr. 2021-02-10T04:25:13.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:12 smithi085 conmon[35961]: 14140) 163 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-02-10T04:25:13.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:12 smithi085 conmon[35961]: audit 2021-02-10T04:25:12.325657+0000 mon.a (mon.0) 2021-02-10T04:25:13.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:12 smithi085 conmon[35961]: 483 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:25:13.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:12 smithi085 conmon[35961]: audit 2021 2021-02-10T04:25:13.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:12 smithi085 conmon[35961]: -02-10T04:25:12.330668+0000 mon.a (mon.0 2021-02-10T04:25:13.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:12 smithi085 conmon[35961]: ) 484 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:13.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:12 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:25:13.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:12 smithi085 conmon[35961]: T04:25:12.337682+0000 mon.a ( 2021-02-10T04:25:13.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:12 smithi085 conmon[35961]: mon.0) 485 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:14.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:14 smithi164 conmon[29288]: cephadm 2021-02-10T04:25:12.325120+0000 mgr.y (mgr.14140) 164 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:25:14.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:14 smithi164 conmon[29288]: audit 2021-02-10T04:25:13.031468+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:14.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:14 smithi164 conmon[29288]: audit 2021-02-10T04:25:13.708896+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:14.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:14 smithi164 conmon[29288]: audit 2021-02-10T04:25:13.709969+0000 mon.a (mon.0) 488 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:25:14.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:14 smithi085 conmon[29319]: cephadm 2021-02-10T04:25:12.325120+0000 mgr.y (mgr.14140) 164 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:25:14.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:14 smithi085 conmon[29319]: audit 2021-02-10T04:25:13.031468+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:14.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:14 smithi085 conmon[29319]: audit 2021-02-10T04:25:13.708896+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:14.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:14 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:25:14.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:14 smithi085 conmon[29319]: T04:25:13.709969+0000 mon.a (mon.0) 488 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:25:14.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:14 smithi085 conmon[35961]: cephadm 2021-02-10T04:25:12.325120+0000 2021-02-10T04:25:14.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:14 smithi085 conmon[35961]: mgr.y (mgr.14140) 164 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:25:14.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:14 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:25:14.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:14 smithi085 conmon[35961]: T04:25:13.031468 2021-02-10T04:25:14.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:14 smithi085 conmon[35961]: +0000 mon.a (mon.0) 486 2021-02-10T04:25:14.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:14 smithi085 conmon[35961]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:14.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:14 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:25:14.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:14 smithi085 conmon[35961]: 02-10T04:25:13.708896+0000 mon.a ( 2021-02-10T04:25:14.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:14 smithi085 conmon[35961]: mon.0) 487 : audit [INF] 2021-02-10T04:25:14.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:14 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:14.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:14 smithi085 conmon[35961]: audit 2021 2021-02-10T04:25:14.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:14 smithi085 conmon[35961]: -02-10T04:25:13.709969+0000 mon.a ( 2021-02-10T04:25:14.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:14 smithi085 conmon[35961]: mon.0) 488 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:25:15.235 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping: /dev/vg_nvme/lv_1 2021-02-10T04:25:15.236 INFO:teuthology.orchestra.run.smithi164.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2021-02-10T04:25:15.236 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10+0 records in 2021-02-10T04:25:15.236 INFO:teuthology.orchestra.run.smithi164.stderr:10+0 records out 2021-02-10T04:25:15.236 INFO:teuthology.orchestra.run.smithi164.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.011851 s, 885 MB/s 2021-02-10T04:25:15.236 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping successful for: 2021-02-10T04:25:15.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:15 smithi164 conmon[29288]: cluster 2021-02-10T04:25:13.939477+0000 mgr.y (mgr 2021-02-10T04:25:15.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:15 smithi164 conmon[29288]: .14140) 165 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-02-10T04:25:15.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:15 smithi085 conmon[29319]: cluster 2021-02-10T04:25:13.939477+0000 mgr.y (mgr.14140) 165 : cluster [DBG] 2021-02-10T04:25:15.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:15 smithi085 conmon[29319]: pgmap v138: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-02-10T04:25:15.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:15 smithi085 conmon[35961]: cluster 2021-02-10T04:25:13.939477 2021-02-10T04:25:15.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:15 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 165 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-02-10T04:25:15.892 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:25:15 smithi164 conmon[40892]: debug 2021-02-10T04:25:15.533+0000 7f305d052200 -1 osd.6 0 log_to_monitors {default=true} 2021-02-10T04:25:15.988 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph orch daemon add osd smithi164:vg_nvme/lv_1 2021-02-10T04:25:16.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:16 smithi164 conmon[29288]: audit 2021-02-10T04:25:15.537294+0000 mon.b (mon.2) 15 : audit 2021-02-10T04:25:16.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:16 smithi164 conmon[29288]: [INF] from='osd.6 v2:172.21.15.164:6808/7' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-02-10T04:25:16.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:16 smithi164 conmon[29288]: audit 2021-02-10T04:25:15.537684+0000 mon.a (mon.0) 489 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-02-10T04:25:16.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:16 smithi085 conmon[29319]: audit 2021-02-10T04:25:15.537294+0000 mon.b (mon.2) 15 : audit [INF] from='osd.6 v2:172.21.15.164:6808/7' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-02-10T04:25:16.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:16 smithi085 conmon[29319]: audit 2021-02-10T04:25:15.537684+0000 mon.a (mon.0) 489 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-02-10T04:25:16.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:16 smithi085 conmon[35961]: audit 2021-02-10T04:25:15.537294+0000 mon.b 2021-02-10T04:25:16.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:16 smithi085 conmon[35961]: (mon.2) 15 : audit [INF] from='osd.6 v2:172.21.15.164:6808/7' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-02-10T04:25:16.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:16 smithi085 conmon[35961]: audit 2021-02-10T04:25:15. 2021-02-10T04:25:16.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:16 smithi085 conmon[35961]: 537684+0000 mon.a (mon.0) 489 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-02-10T04:25:17.143 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:25:17 smithi164 conmon[40892]: debug 2021-02-10T04:25:17.053+0000 7f30535c3700 -1 osd.6 0 waiting for initial osdmap 2021-02-10T04:25:17.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:17 smithi164 conmon[29288]: cluster 2021-02-10T04:25:15.939954+0000 mgr.y (mgr.14140) 166 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-02-10T04:25:17.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:17 smithi164 conmon[29288]: audit 2021-02-10T04:25:16.045959+0000 mon.a (mon.0) 490 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2021-02-10T04:25:17.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:17 smithi164 conmon[29288]: cluster 2021-02-10T04:25:16.046074+0000 mon.a (mon.0) 491 : cluster [DBG] osdmap e41: 7 total, 6 up, 6 in 2021-02-10T04:25:17.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:17 smithi164 conmon[29288]: audit 2021-02-10T04:25:16.046227+0000 mon.a (mon.0) 492 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-02-10T04:25:17.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:17 smithi164 conmon[29288]: audit 2021-02-10T04:25:16.046752+0000 mon.b (mon.2) 16 : audit [INF] from='osd.6 v2:172.21.15.164:6808/7' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2021-02-10T04:25:17.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:17 smithi164 conmon[29288]: audit 2021-02-10T04:25:16.047049+0000 mon.a (mon.0) 493 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2021-02-10T04:25:17.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[29319]: cluster 2021-02-10T04:25:15.939954+0000 mgr.y (mgr.14140) 166 : cluster [DBG] 2021-02-10T04:25:17.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[29319]: pgmap v139: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-02-10T04:25:17.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[29319]: audit 2021-02-10T04:25:16.045959+0000 mon.a (mon.0) 490 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2021-02-10T04:25:17.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[29319]: cluster 2021-02-10T04:25:17.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[29319]: 2021-02-10T04:25: 2021-02-10T04:25:17.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[29319]: 16.046074+0000 mon.a (mon.0) 491 : 2021-02-10T04:25:17.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[29319]: cluster [DBG] osdmap e41: 7 total, 6 up, 6 in 2021-02-10T04:25:17.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:25:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[29319]: 10T04:25:16.046227+0000 mon.a (mon.0) 492 : audit 2021-02-10T04:25:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[29319]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-02-10T04:25:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[29319]: audit 2021-02-10T04:25:16 2021-02-10T04:25:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[29319]: .046752+0000 mon.b (mon.2) 16 : audit 2021-02-10T04:25:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[29319]: [INF] from='osd.6 v2:172.21.15.164:6808/7' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2021-02-10T04:25:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:25:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[29319]: 10T04:25:16.047049+0000 mon.a (mon.0 2021-02-10T04:25:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[29319]: ) 493 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2021-02-10T04:25:17.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[35961]: cluster 2021-02-10T04:25:15.939954 2021-02-10T04:25:17.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 166 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-02-10T04:25:17.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[35961]: audit 2021-02-10T04:25:17.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[35961]: 2021-02-10T04:25:16.045959+0000 mon.a (mon.0) 490 : audit [INF] 2021-02-10T04:25:17.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[35961]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2021-02-10T04:25:17.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:25:17.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[35961]: -02-10T04:25:16.046074+0000 mon.a ( 2021-02-10T04:25:17.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[35961]: mon.0) 491 : cluster [DBG] osdmap e41: 7 total, 6 up, 6 in 2021-02-10T04:25:17.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:25:17.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[35961]: 10T04:25:16.046227+0000 mon.a (mon.0) 492 : audit [DBG] 2021-02-10T04:25:17.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-02-10T04:25:17.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[35961]: audit 2021 2021-02-10T04:25:17.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[35961]: -02-10T04:25:16.046752+0000 mon.b ( 2021-02-10T04:25:17.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[35961]: mon.2) 16 : audit [INF] from='osd.6 v2:172.21.15.164:6808/7' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2021-02-10T04:25:17.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[35961]: 2021-02-10T04:25:17.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[35961]: audit 2021 2021-02-10T04:25:17.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[35961]: -02-10T04:25:16.047049+0000 mon.a 2021-02-10T04:25:17.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[35961]: (mon.0) 493 : audit [INF] 2021-02-10T04:25:17.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:17 smithi085 conmon[35961]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2021-02-10T04:25:18.124 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:18 smithi164 conmon[29288]: audit 2021-02-10T04:25:17.047114+0000 mon.a (mon.0) 2021-02-10T04:25:18.124 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:18 smithi164 conmon[29288]: 494 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2021-02-10T04:25:18.124 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:18 smithi164 conmon[29288]: cluster 2021-02-10T04:25:17.047246+0000 mon.a (mon.0) 495 : cluster [DBG] osdmap e42: 7 total, 6 up, 6 in 2021-02-10T04:25:18.124 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:18 smithi164 conmon[29288]: audit 2021-02-10T04:25:17.047733+0000 mon.a (mon.0) 496 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-02-10T04:25:18.124 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:18 smithi164 conmon[29288]: audit 2021-02-10T04:25:17.053749+0000 mon.a (mon.0) 497 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-02-10T04:25:18.124 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:18 smithi164 conmon[29288]: audit 2021-02-10T04:25:17.883157+0000 mon.a (mon.0) 498 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:25:18.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:18 smithi085 conmon[29319]: audit 2021-02-10T04:25:17.047114+0000 mon.a (mon.0) 494 : audit [INF] 2021-02-10T04:25:18.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:18 smithi085 conmon[29319]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2021-02-10T04:25:18.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:18 smithi085 conmon[29319]: cluster 2021-02-10T04:25:17.047246+0000 mon.a (mon.0) 495 : cluster [DBG] osdmap e42: 7 total, 6 up, 6 in 2021-02-10T04:25:18.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:18 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:25:18.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:18 smithi085 conmon[29319]: :25:17.047733+0000 mon.a (mon.0) 496 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-02-10T04:25:18.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:18 smithi085 conmon[29319]: audit 2021-02-10T04:25:17.053749+0000 mon.a 2021-02-10T04:25:18.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:18 smithi085 conmon[29319]: (mon.0) 497 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-02-10T04:25:18.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:18 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:25:18.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:18 smithi085 conmon[29319]: :25:17.883157+0000 mon.a (mon.0) 498 2021-02-10T04:25:18.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:18 smithi085 conmon[29319]: : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:25:18.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:18 smithi085 conmon[35961]: audit 2021-02-10T04:25:17.047114+0000 mon.a (mon 2021-02-10T04:25:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:18 smithi085 conmon[35961]: .0) 494 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2021-02-10T04:25:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:18 smithi085 conmon[35961]: cluster 2021-02-10T04:25:17. 2021-02-10T04:25:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:18 smithi085 conmon[35961]: 047246+0000 mon.a (mon.0) 495 : cluster [DBG] osdmap e42: 7 total, 6 up, 6 in 2021-02-10T04:25:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:18 smithi085 conmon[35961]: audit 2021-02-10T04:25:17.047733+0000 2021-02-10T04:25:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:18 smithi085 conmon[35961]: mon.a (mon.0) 496 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-02-10T04:25:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:18 smithi085 conmon[35961]: 2021-02-10T04:25:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:18 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:25:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:18 smithi085 conmon[35961]: 25:17.053749+0000 mon.a (mon.0) 497 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-02-10T04:25:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:18 smithi085 conmon[35961]: audit 2021-02-10T04:25:17. 2021-02-10T04:25:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:18 smithi085 conmon[35961]: 883157+0000 mon.a (mon.0) 498 : 2021-02-10T04:25:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:18 smithi085 conmon[35961]: audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:25:19.295 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:19 smithi164 conmon[29288]: cluster 2021-02-10T04:25:16.520544+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:25:19.295 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:19 smithi164 conmon[29288]: cluster 2021-02-10T04:25:16.520766+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:25:19.295 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:19 smithi164 conmon[29288]: cluster 2021-02-10T04:25:17.940427+0000 mgr.y (mgr.14140) 167 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-02-10T04:25:19.295 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:19 smithi164 conmon[29288]: cluster 2021-02-10T04:25:18.054785+0000 mon.a (mon.0) 499 : cluster [INF] osd.6 v2:172.21.15.164:6808/7 boot 2021-02-10T04:25:19.295 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:19 smithi164 conmon[29288]: cluster 2021-02-10T04:25:18.054889+0000 mon.a (mon.0) 500 : cluster [DBG] osdmap e43: 7 total, 7 up, 7 in 2021-02-10T04:25:19.295 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:19 smithi164 conmon[29288]: audit 2021-02-10T04:25:18.055472+0000 mon.a (mon.0) 501 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-02-10T04:25:19.295 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:19 smithi164 conmon[29288]: audit 2021-02-10T04:25:18.237761+0000 mon.a (mon.0) 502 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:19.295 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:19 smithi164 conmon[29288]: audit 2021-02-10T04:25:19.295 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:19 smithi164 conmon[29288]: 2021-02-10T04:25:18.242070+0000 mon.a (mon.0) 503 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:19.296 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:19 smithi164 conmon[29288]: audit 2021-02-10T04:25:18.247491+0000 mon.a (mon.0) 504 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:19.296 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:19 smithi164 conmon[29288]: audit 2021-02-10T04:25:18.445417+0000 mon.a (mon.0) 505 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-02-10T04:25:19.296 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:19 smithi164 conmon[29288]: audit 2021-02-10T04:25:18.447177+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-02-10T04:25:19.296 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:19 smithi164 conmon[29288]: audit 2021-02-10T04:25:18.447928+0000 mon.a (mon.0) 507 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:25:19.296 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:19 smithi164 conmon[29288]: audit 2021-02-10T04:25:18.448850+0000 mon.a (mon.0) 508 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:25:19.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: cluster 2021-02-10T04:25:16.520544+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:25:19.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: cluster 2021-02-10T04:25:16.520766+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:25:19.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: cluster 2021-02-10T04:25:17.940427 2021-02-10T04:25:19.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 167 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-02-10T04:25:19.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:25:19.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: 10T04:25:18.054785+0000 mon.a (mon.0 2021-02-10T04:25:19.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: ) 499 : cluster [INF] osd.6 v2:172.21.15.164:6808/7 boot 2021-02-10T04:25:19.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: cluster 2021-02-10T04:25: 2021-02-10T04:25:19.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: 18.054889+0000 mon.a (mon.0) 500 : 2021-02-10T04:25:19.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: cluster [DBG] osdmap e43: 7 total, 7 up, 7 in 2021-02-10T04:25:19.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:25:19.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: -10T04:25:18.055472+0000 2021-02-10T04:25:19.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: mon.a (mon.0) 501 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-02-10T04:25:19.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:25:19.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: 10T04:25:18.237761+0000 2021-02-10T04:25:19.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: mon.a (mon.0) 502 : audit 2021-02-10T04:25:19.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:19.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: audit 2021-02-10T04:25:19.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: 2021-02-10T04:25 2021-02-10T04:25:19.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: :18.242070+0000 mon.a (mon.0) 2021-02-10T04:25:19.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: 503 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:19.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: audit 2021-02-10T04:25:19.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: 2021-02-10T04:25: 2021-02-10T04:25:19.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: 18.247491+0000 mon.a (mon.0) 504 2021-02-10T04:25:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: audit 2021 2021-02-10T04:25:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: -02-10T04:25:18.445417+0000 2021-02-10T04:25:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: mon.a (mon.0) 505 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-02-10T04:25:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:25:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: T04:25:18.447177+0000 2021-02-10T04:25:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: mon.a (mon.0) 506 : audit [INF] 2021-02-10T04:25:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-02-10T04:25:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: audit 2021 2021-02-10T04:25:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: -02-10T04:25:18.447928+0000 mon.a (mon.0) 507 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:25:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: audit 2021 2021-02-10T04:25:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: -02-10T04:25: 2021-02-10T04:25:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: 18.448850+0000 mon.a (mon.0) 508 : 2021-02-10T04:25:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[35961]: audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:25:19.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: cluster 2021-02-10T04:25:16.520544+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:25:19.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: cluster 2021-02-10T04:25:16.520766+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:25:19.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: cluster 2021-02-10T04:25:17.940427+0000 mgr.y (mgr.14140) 167 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-02-10T04:25:19.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: cluster 2021-02-10T04:25:18.054785+0000 mon.a ( 2021-02-10T04:25:19.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: mon.0) 499 : cluster [INF] osd.6 v2:172.21.15.164:6808/7 boot 2021-02-10T04:25:19.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:25:19.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: -10T04:25:18.054889+0000 mon.a ( 2021-02-10T04:25:19.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: mon.0) 500 : cluster [DBG] osdmap e43: 7 total, 7 up, 7 in 2021-02-10T04:25:19.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: 2021-02-10T04:25:19.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:25:19.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: 02-10T04:25:18.055472+0000 2021-02-10T04:25:19.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: mon.a (mon.0) 501 : audit 2021-02-10T04:25:19.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-02-10T04:25:19.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: audit 2021 2021-02-10T04:25:19.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: -02-10T04:25:18. 2021-02-10T04:25:19.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: 237761+0000 mon.a (mon.0) 2021-02-10T04:25:19.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: 502 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:19.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:25:19.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: 02-10T04:25:18.242070 2021-02-10T04:25:19.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: +0000 mon.a (mon.0) 503 : 2021-02-10T04:25:19.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:19.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:25:19.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: 02-10T04:25:18.247491 2021-02-10T04:25:19.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: +0000 mon.a (mon.0) 504 2021-02-10T04:25:19.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:19.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:25:19.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: 02-10T04:25:18.445417 2021-02-10T04:25:19.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: +0000 mon.a (mon.0) 505 2021-02-10T04:25:19.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-02-10T04:25:19.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:25:19.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: 02-10T04:25:18.447177 2021-02-10T04:25:19.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: +0000 mon.a (mon.0) 506 2021-02-10T04:25:19.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-02-10T04:25:19.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: audit 2021 2021-02-10T04:25:19.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: -02-10T04:25:18.447928+0000 2021-02-10T04:25:19.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: mon.a (mon.0) 507 : audit 2021-02-10T04:25:19.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:25:19.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:25:19.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: 02-10T04:25:18.448850+0000 2021-02-10T04:25:19.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: mon.a (mon.0) 508 : audit 2021-02-10T04:25:19.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:19 smithi085 conmon[29319]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:25:20.194 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:20 smithi164 conmon[29288]: cephadm 2021-02-10T04:25:18.232846+0000 mgr.y (mgr.14140) 168 : cephadm 2021-02-10T04:25:20.194 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:20 smithi164 conmon[29288]: [INF] Detected new or changed devices on smithi164 2021-02-10T04:25:20.194 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:20 smithi164 conmon[29288]: audit 2021-02-10T04:25:18.444254+0000 mgr.y (mgr.14140) 169 : audit [DBG] from='client.34259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:25:20.194 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:20 smithi164 conmon[29288]: cephadm 2021-02-10T04:25:18.446852+0000 mgr.y (mgr.14140) 170 : cephadm [INF] Applying service osd.None on host smithi164... 2021-02-10T04:25:20.194 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:20 smithi164 conmon[29288]: cluster 2021-02-10T04:25:19.060924+0000 mon.a (mon.0) 509 : cluster [DBG] osdmap e44: 7 total, 7 up, 7 in 2021-02-10T04:25:20.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:20 smithi085 conmon[29319]: cephadm 2021-02-10T04:25:18 2021-02-10T04:25:22.021 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:20 smithi085 conmon[29319]: .232846+0000 mgr.y (mgr.14140) 168 : cephadm [INF] Detected new or changed devices on smithi164 2021-02-10T04:25:22.021 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:20 smithi085 conmon[29319]: audit 2021-02-10T04:25:18.444254+0000 mgr.y 2021-02-10T04:25:22.021 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:20 smithi085 conmon[29319]: (mgr.14140) 169 : audit [DBG] from='client.34259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:25:22.021 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:20 smithi085 conmon[29319]: cephadm 2021-02-10T04:25:18.446852+0000 mgr.y (mgr. 2021-02-10T04:25:22.022 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:20 smithi085 conmon[29319]: 14140) 170 : cephadm [INF] Applying service osd.None on host smithi164... 2021-02-10T04:25:22.022 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:20 smithi085 conmon[29319]: cluster 2021-02-10T04:25:19.060924+0000 mon.a (mon. 2021-02-10T04:25:22.022 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:20 smithi085 conmon[29319]: 0) 509 : cluster [DBG] osdmap e44: 7 total, 7 up, 7 in 2021-02-10T04:25:22.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:20 smithi085 conmon[35961]: cephadm 2021-02-10T04:25:18.232846+0000 mgr.y (mgr.14140) 168 : cephadm [INF] Detected new or changed devices on smithi164 2021-02-10T04:25:22.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:20 smithi085 conmon[35961]: audit 2021-02-10T04:25:18.444254+0000 mgr.y (mgr.14140) 169 : audit [DBG] from='client.34259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:25:22.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:20 smithi085 conmon[35961]: cephadm 2021-02-10T 2021-02-10T04:25:22.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:20 smithi085 conmon[35961]: 04:25:18.446852+0000 mgr.y (mgr.14140) 170 : cephadm [INF] Applying service osd.None on host smithi164... 2021-02-10T04:25:22.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:20 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:25:22.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:20 smithi085 conmon[35961]: 02-10T04:25:19.060924+0000 mon.a (mon.0) 509 : cluster [DBG] osdmap e44: 7 total, 7 up, 7 in 2021-02-10T04:25:22.023 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:21 smithi164 conmon[29288]: cluster 2021-02-10T04:25:19.940796+0000 mgr.y (mgr.14140) 171 : cluster 2021-02-10T04:25:22.023 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:21 smithi164 conmon[29288]: [DBG] pgmap v145: 1 pgs: 1 peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-02-10T04:25:22.023 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:21 smithi164 conmon[29288]: audit 2021-02-10T04:25:20.804632+0000 mon.b (mon.2) 17 : audit [INF] from='client.? 172.21.15.164:0/2047942966' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37"}]: dispatch 2021-02-10T04:25:22.023 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:21 smithi164 conmon[29288]: audit 2021-02-10T04:25:20.805287+0000 mon.a (mon.0) 510 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37"}]: dispatch 2021-02-10T04:25:22.024 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:21 smithi164 conmon[29288]: audit 2021-02-10T04:25:20.809904+0000 mon.a (mon.0) 511 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37"}]': finished 2021-02-10T04:25:22.024 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:21 smithi164 conmon[29288]: cluster 2021-02-10T04:25:20.809988+0000 mon.a (mon.0) 512 : cluster [DBG] osdmap e45: 8 total, 7 up, 7 in 2021-02-10T04:25:22.024 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:21 smithi164 conmon[29288]: audit 2021-02-10T04:25:20.810130+0000 mon.a (mon.0) 513 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-02-10T04:25:22.024 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:21 smithi085 conmon[29319]: cluster 2021-02-10T04:25:19.940796+0000 mgr.y (mgr.14140) 171 : cluster 2021-02-10T04:25:22.024 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:21 smithi085 conmon[29319]: [DBG] pgmap v145: 1 pgs: 1 peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-02-10T04:25:22.024 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:21 smithi085 conmon[29319]: audit 2021-02-10T04:25:20.804632+0000 mon.b (mon.2) 17 : audit [INF] from='client.? 172.21.15.164:0/2047942966' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37"}]: dispatch 2021-02-10T04:25:22.024 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:21 smithi085 conmon[29319]: audit 2021-02-10T04:25:20.805287+0000 2021-02-10T04:25:22.024 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:21 smithi085 conmon[29319]: mon.a (mon.0) 510 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37"}]: dispatch 2021-02-10T04:25:22.024 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:21 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:25:22.024 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:21 smithi085 conmon[29319]: -10T04:25:20.809904+0000 mon.a (mon.0) 511 : audit 2021-02-10T04:25:22.025 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:21 smithi085 conmon[29319]: [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37"}]': finished 2021-02-10T04:25:22.025 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:21 smithi085 conmon[29319]: cluster 2021-02-10T04:25: 2021-02-10T04:25:22.025 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:21 smithi085 conmon[29319]: 20.809988+0000 mon.a (mon.0) 512 : 2021-02-10T04:25:22.025 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:21 smithi085 conmon[29319]: cluster [DBG] osdmap e45: 8 total, 7 up, 7 in 2021-02-10T04:25:22.025 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:21 smithi085 conmon[29319]: audit 2021-02-10T 2021-02-10T04:25:22.025 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:21 smithi085 conmon[29319]: 04:25:20.810130+0000 mon.a (mon.0 2021-02-10T04:25:22.025 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:21 smithi085 conmon[29319]: ) 513 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-02-10T04:25:22.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:21 smithi085 conmon[35961]: cluster 2021-02-10T04:25:19.940796+0000 mgr.y (mgr.14140) 171 2021-02-10T04:25:22.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:21 smithi085 conmon[35961]: : cluster [DBG] pgmap v145: 1 pgs: 1 peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-02-10T04:25:22.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:21 smithi085 conmon[35961]: audit 2021-02-10T04:25:20.804632+0000 mon.b (mon.2) 2021-02-10T04:25:22.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:21 smithi085 conmon[35961]: 17 : audit [INF] from='client.? 172.21.15.164:0/2047942966' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37"}]: dispatch 2021-02-10T04:25:22.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:21 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:25:22.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:21 smithi085 conmon[35961]: 02-10T04:25:20.805287+0000 mon.a (mon.0) 510 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37"}]: dispatch 2021-02-10T04:25:22.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:21 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:25:22.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:21 smithi085 conmon[35961]: 10T04:25:20.809904+0000 mon.a (mon 2021-02-10T04:25:22.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:21 smithi085 conmon[35961]: .0) 511 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37"}]': finished 2021-02-10T04:25:22.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:21 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:25:22.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:21 smithi085 conmon[35961]: T04:25:20.809988+0000 mon.a (mon.0 2021-02-10T04:25:22.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:21 smithi085 conmon[35961]: ) 512 : cluster [DBG] osdmap e45: 8 total, 7 up, 7 in 2021-02-10T04:25:22.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:21 smithi085 conmon[35961]: audit 2021-02-10T04:25:20.810130+0000 mon.a (mon.0) 513 : 2021-02-10T04:25:22.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:21 smithi085 conmon[35961]: audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-02-10T04:25:22.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:22 smithi164 conmon[29288]: audit 2021-02-10T04:25:21.294774+0000 mon.b (mon.2) 18 : audit [DBG] from='client.? 172.21.15.164:0/4124017977' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-02-10T04:25:22.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:22 smithi164 conmon[29288]: cluster 2021-02-10T04:25:21.941260+0000 mgr.y (mgr.14140) 172 : cluster [DBG] pgmap v147: 1 pgs: 1 peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-02-10T04:25:22.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:22 smithi085 conmon[29319]: audit 2021-02-10T04:25:21.294774+0000 mon.b (mon.2) 18 : audit [DBG] from='client.? 172.21.15.164:0/4124017977' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-02-10T04:25:22.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:22 smithi085 conmon[29319]: cluster 2021-02-10T04:25:21.941260+0000 mgr.y (mgr.14140) 172 : cluster [DBG] pgmap v147: 1 pgs: 1 peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-02-10T04:25:22.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:22 smithi085 conmon[35961]: audit 2021-02-10T04:25:21.294774+0000 mon.b (mon.2) 18 2021-02-10T04:25:22.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:22 smithi085 conmon[35961]: : audit [DBG] from='client.? 172.21.15.164:0/4124017977' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-02-10T04:25:22.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:22 smithi085 conmon[35961]: cluster 2021-02-10T04:25:21.941260+0000 mgr.y (mgr.14140) 172 : cluster [DBG] pgmap v147: 1 pgs: 1 peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-02-10T04:25:22.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:22 smithi085 conmon[35961]: 2021-02-10T04:25:25.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[35961]: cluster 2021-02-10T04:25:23.251650+0000 osd.3 (osd.3) 3 : 2021-02-10T04:25:25.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[35961]: cluster [DBG] purged_snaps scrub starts 2021-02-10T04:25:25.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[35961]: cluster 2021-02-10T04:25:23.251786+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:25:25.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[35961]: cluster 2021-02-10T04:25:23.917932+0000 2021-02-10T04:25:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[35961]: mon.a (mon.0) 514 : cluster [DBG] osdmap e46: 8 total, 7 up, 7 in 2021-02-10T04:25:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:25:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[35961]: :25:23.918064+0000 mon.a (mon.0) 515 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-02-10T04:25:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[35961]: 2021-02-10T04:25:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:25:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[35961]: -10T04:25:23.941818+0000 mgr.y (mgr.14140) 173 : cluster [DBG] pgmap v149: 1 pgs: 1 peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-02-10T04:25:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:25:25.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[35961]: 10T04:25:24.074817+0000 mon.a (mon.0) 516 : audit 2021-02-10T04:25:25.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[35961]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:25:25.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[29319]: cluster 2021-02-10T04:25:23.251650+0000 2021-02-10T04:25:25.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[29319]: osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:25:25.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[29319]: cluster 2021-02-10T04:25:23.251786+0000 osd.3 (osd 2021-02-10T04:25:25.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[29319]: .3) 4 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:25:25.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:25:25.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[29319]: 02-10T04:25:23. 2021-02-10T04:25:25.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[29319]: 917932+0000 mon.a (mon.0) 2021-02-10T04:25:25.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[29319]: 514 : cluster [DBG] osdmap e46: 8 total, 7 up, 7 in 2021-02-10T04:25:25.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:25:25.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[29319]: 02-10T04:25:23.918064+0000 2021-02-10T04:25:25.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[29319]: mon.a (mon.0) 515 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-02-10T04:25:25.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:25:25.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[29319]: -02-10T04:25:23.941818 2021-02-10T04:25:25.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 173 2021-02-10T04:25:25.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[29319]: : cluster [DBG] pgmap v149: 1 pgs: 1 peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-02-10T04:25:25.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[29319]: audit 2021 2021-02-10T04:25:25.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[29319]: -02-10T04:25:24. 2021-02-10T04:25:25.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[29319]: 074817+0000 mon.a (mon.0) 2021-02-10T04:25:25.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:24 smithi085 conmon[29319]: 516 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:25:25.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:24 smithi164 conmon[29288]: cluster 2021-02-10T04:25:23.251650+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:25:25.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:24 smithi164 conmon[29288]: cluster 2021-02-10T04:25:23.251786+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:25:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:24 smithi164 conmon[29288]: cluster 2021-02-10T04:25:23.917932+0000 mon.a (mon.0) 514 : cluster [DBG] osdmap e46: 8 total, 7 up, 7 in 2021-02-10T04:25:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:24 smithi164 conmon[29288]: audit 2021-02-10T04:25:23.918064+0000 mon.a (mon.0) 515 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-02-10T04:25:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:24 smithi164 conmon[29288]: cluster 2021-02-10T04:25:23.941818+0000 mgr.y (mgr.14140) 173 : cluster [DBG] pgmap v149: 1 pgs: 1 peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-02-10T04:25:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:24 smithi164 conmon[29288]: audit 2021-02-10T04:25:24.074817+0000 mon.a (mon.0) 516 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:25:26.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:25 smithi085 conmon[29319]: cluster 2021-02-10T04:25:24.604772+0000 2021-02-10T04:25:26.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:25 smithi085 conmon[29319]: osd.6 (osd.6) 3 : cluster [DBG] 1.0 scrub ok 2021-02-10T04:25:26.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:25 smithi085 conmon[35961]: cluster 2021-02-10T04:25:24.604772+0000 osd.6 (osd.6 2021-02-10T04:25:26.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:25 smithi085 conmon[35961]: ) 3 : cluster [DBG] 1.0 scrub ok 2021-02-10T04:25:26.267 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:25 smithi164 conmon[29288]: cluster 2021-02-10T04:25:24.604772+0000 osd.6 (osd.6) 3 : 2021-02-10T04:25:26.267 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:25 smithi164 conmon[29288]: cluster [DBG] 1.0 scrub ok 2021-02-10T04:25:27.187 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:26 smithi164 conmon[29288]: cluster 2021-02-10T04:25:25.942359+0000 mgr.y (mgr.14140) 2021-02-10T04:25:27.187 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:26 smithi164 conmon[29288]: 174 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-02-10T04:25:27.187 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:26 smithi164 conmon[29288]: audit 2021-02-10T04:25:26.625389+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2021-02-10T04:25:27.187 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:26 smithi164 conmon[29288]: audit 2021-02-10T04:25:26.626326+0000 mon.a (mon.0) 518 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:25:27.187 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:26 smithi164 conmon[29288]: audit 2021-02-10T04:25:26.627483+0000 mon.a (mon.0) 519 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.7", "key": "container_image"}]: dispatch 2021-02-10T04:25:27.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:26 smithi085 conmon[29319]: cluster 2021-02-10T04:25:25.942359+0000 mgr.y (mgr 2021-02-10T04:25:27.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:26 smithi085 conmon[29319]: .14140) 174 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-02-10T04:25:27.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:26 smithi085 conmon[29319]: audit 2021-02-10T04:25:26.625389+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2021-02-10T04:25:27.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:26 smithi085 conmon[29319]: audit 2021-02-10T04:25:26.626326+0000 2021-02-10T04:25:27.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:26 smithi085 conmon[29319]: mon.a (mon.0) 518 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:25:27.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:26 smithi085 conmon[29319]: audit 2021-02-10T04:25: 2021-02-10T04:25:27.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:26 smithi085 conmon[29319]: 26.627483+0000 mon.a (mon.0) 519 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.7", "key": "container_image"}]: dispatch 2021-02-10T04:25:27.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:26 smithi085 conmon[35961]: cluster 2021-02-10T04:25:25.942359+0000 mgr.y (mgr. 2021-02-10T04:25:27.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:26 smithi085 conmon[35961]: 14140) 174 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-02-10T04:25:27.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:26 smithi085 conmon[35961]: audit 2021-02-10T04:25:26.625389+0000 2021-02-10T04:25:27.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:26 smithi085 conmon[35961]: mon.a (mon.0) 517 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2021-02-10T04:25:27.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:26 smithi085 conmon[35961]: audit 2021-02-10T04:25 2021-02-10T04:25:27.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:26 smithi085 conmon[35961]: :26.626326+0000 mon.a (mon.0) 518 : 2021-02-10T04:25:27.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:26 smithi085 conmon[35961]: audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:25:27.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:26 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:25:27.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:26 smithi085 conmon[35961]: -10T04:25:26.627483+0000 mon.a (mon. 2021-02-10T04:25:27.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:26 smithi085 conmon[35961]: 0) 519 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd.7", "key": "container_image"}]: dispatch 2021-02-10T04:25:28.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:27 smithi164 conmon[29288]: cephadm 2021-02-10T04:25:26.626980+0000 mgr.y (mgr.14140) 175 : cephadm [INF] Deploying daemon osd.7 on smithi164 2021-02-10T04:25:28.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:27 smithi085 conmon[29319]: cephadm 2021-02-10T04:25:26.626980+0000 mgr.y (mgr.14140) 175 2021-02-10T04:25:28.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:27 smithi085 conmon[29319]: : cephadm [INF] Deploying daemon osd.7 on smithi164 2021-02-10T04:25:28.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:27 smithi085 conmon[35961]: cephadm 2021-02-10T04:25:26.626980+0000 mgr.y ( 2021-02-10T04:25:28.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:27 smithi085 conmon[35961]: mgr.14140) 175 : cephadm [INF] Deploying daemon osd.7 on smithi164 2021-02-10T04:25:29.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:28 smithi164 conmon[29288]: cluster 2021-02-10T04:25:27.942690+0000 mgr.y (mgr.14140) 176 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-02-10T04:25:29.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:28 smithi085 conmon[29319]: cluster 2021-02-10T04:25:27.942690+0000 mgr.y (mgr.14140) 176 : cluster 2021-02-10T04:25:29.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:28 smithi085 conmon[29319]: [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-02-10T04:25:29.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:28 smithi085 conmon[35961]: cluster 2021-02-10T04:25:27.942690+0000 mgr.y (mgr.14140) 176 2021-02-10T04:25:29.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:28 smithi085 conmon[35961]: : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-02-10T04:25:29.965 INFO:teuthology.orchestra.run.smithi164.stdout:Created osd(s) 7 on host 'smithi164' 2021-02-10T04:25:30.126 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:29 smithi164 conmon[29288]: cluster 2021-02-10T04:25:29.273422+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:25:30.126 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:29 smithi164 conmon[29288]: cluster 2021-02-10T04:25:29.273567+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:25:30.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:29 smithi085 conmon[35961]: cluster 2021-02-10T04:25:29.273422+0000 osd.0 (osd.0) 5 : cluster 2021-02-10T04:25:30.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:29 smithi085 conmon[35961]: [DBG] purged_snaps scrub starts 2021-02-10T04:25:30.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:29 smithi085 conmon[35961]: cluster 2021-02-10T04:25:29.273567+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:25:30.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:29 smithi085 conmon[29319]: cluster 2021-02-10T04:25:30.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:29 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:25:30.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:29 smithi085 conmon[29319]: T04:25:29.273422+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:25:30.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:29 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:25:30.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:29 smithi085 conmon[29319]: 04:25:29.273567+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:25:30.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:29 smithi085 conmon[29319]: 2021-02-10T04:25:30.586 DEBUG:teuthology.orchestra.run.smithi164:osd.7> sudo journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.7.service 2021-02-10T04:25:30.589 INFO:tasks.cephadm:Adding ceph.rgw.realm.zone.a on smithi085 2021-02-10T04:25:30.589 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- radosgw-admin realm create --rgw-realm realm --default 2021-02-10T04:25:30.617 INFO:journalctl@ceph.osd.7.smithi164.stdout:-- Logs begin at Wed 2021-02-10 04:09:33 UTC. -- 2021-02-10T04:25:31.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:30 smithi164 conmon[29288]: cluster 2021-02-10T04:25:29.943098+0000 mgr.y (mgr.14140) 177 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-02-10T04:25:31.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:30 smithi164 conmon[29288]: audit 2021-02-10T04:25:29.949475+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:25:31.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:30 smithi164 conmon[29288]: audit 2021-02-10T04:25:29.958518+0000 mon.a (mon.0) 521 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:31.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:30 smithi164 conmon[29288]: cluster 2021-02-10T04:25:29.958731+0000 mon.a (mon.0) 522 : cluster [DBG] osdmap e47: 8 total, 7 up, 7 in 2021-02-10T04:25:31.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:30 smithi164 conmon[29288]: audit 2021-02-10T04:25:29.959111+0000 mon.a (mon.0) 523 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-02-10T04:25:31.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:30 smithi164 conmon[29288]: audit 2021-02-10T04:25:29.963037+0000 mon.a (mon.0) 524 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:31.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[35961]: cluster 2021-02-10T04:25:29.943098 2021-02-10T04:25:31.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 177 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-02-10T04:25:31.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:25:31.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[35961]: 10T04:25:29.949475+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:25:31.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[35961]: audit 2021-02-10T04:25:31.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[35961]: 2021-02-10T04:25:29.958518+0000 2021-02-10T04:25:31.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[35961]: mon.a (mon.0) 521 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:31.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[35961]: 2021-02-10T04:25:31.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:25:31.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[35961]: 04:25:29.958731+0000 2021-02-10T04:25:31.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[35961]: mon.a (mon.0) 522 : cluster 2021-02-10T04:25:31.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[35961]: [DBG] osdmap e47: 8 total, 7 up, 7 in 2021-02-10T04:25:31.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:25:31.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[35961]: T04:25:29.959111+0000 mon.a ( 2021-02-10T04:25:31.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[35961]: mon.0) 523 : audit [DBG] 2021-02-10T04:25:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-02-10T04:25:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[35961]: audit 2021-02-10T04:25:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:25:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[35961]: 25:29.963037+0000 mon.a ( 2021-02-10T04:25:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[35961]: mon.0) 524 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[29319]: cluster 2021-02-10T04:25:29.943098+0000 2021-02-10T04:25:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[29319]: mgr.y (mgr.14140) 177 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-02-10T04:25:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[29319]: audit 2021-02-10T04:25:29. 2021-02-10T04:25:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[29319]: 949475+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:25:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[29319]: audit 2021 2021-02-10T04:25:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[29319]: -02-10T04:25:29.958518+0000 2021-02-10T04:25:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[29319]: mon.a (mon.0) 521 : audit 2021-02-10T04:25:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[29319]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:25:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[29319]: -10T04:25:29.958731+0000 mon.a 2021-02-10T04:25:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[29319]: (mon.0) 522 : cluster [DBG] 2021-02-10T04:25:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[29319]: osdmap e47: 8 total, 7 up, 7 in 2021-02-10T04:25:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:25:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[29319]: 02-10T04:25:29.959111+0000 2021-02-10T04:25:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[29319]: mon.a (mon.0) 523 : audit [DBG] 2021-02-10T04:25:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-02-10T04:25:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[29319]: audit 2021 2021-02-10T04:25:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[29319]: -02-10T04:25:29. 2021-02-10T04:25:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[29319]: 963037+0000 mon.a (mon.0) 2021-02-10T04:25:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:30 smithi085 conmon[29319]: 524 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:32.305 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:32 smithi164 conmon[29288]: audit 2021-02-10T04:25:31.301048+0000 mon.a (mon.0) 525 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:32.305 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:32 smithi164 conmon[29288]: audit 2021-02-10T04:25:31.302063+0000 mon.a (mon.0) 526 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:25:32.305 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:32 smithi164 conmon[29288]: cluster 2021-02-10T04:25:31.943680+0000 mgr.y (mgr.14140) 178 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-02-10T04:25:32.690 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:32 smithi085 conmon[35961]: audit 2021-02-10T04:25:31.301048+0000 mon.a (mon 2021-02-10T04:25:32.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:32 smithi085 conmon[35961]: .0) 525 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:32.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:32 smithi085 conmon[35961]: audit 2021-02-10T04:25:31.302063+0000 mon.a ( 2021-02-10T04:25:32.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:32 smithi085 conmon[35961]: mon.0) 526 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:25:32.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:32 smithi085 conmon[35961]: cluster 2021-02-10T04:25:31.943680+0000 2021-02-10T04:25:32.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:32 smithi085 conmon[35961]: mgr.y (mgr.14140) 178 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-02-10T04:25:32.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:32 smithi085 conmon[29319]: audit 2021-02-10T04:25:31.301048+0000 mon.a (mon.0) 2021-02-10T04:25:32.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:32 smithi085 conmon[29319]: 525 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:32.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:32 smithi085 conmon[29319]: audit 2021-02-10T04:25:31.302063+0000 mon.a (mon.0) 526 2021-02-10T04:25:32.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:32 smithi085 conmon[29319]: : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:25:32.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:32 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:25:32.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:32 smithi085 conmon[29319]: 02-10T04:25:31.943680+0000 mgr.y ( 2021-02-10T04:25:32.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:32 smithi085 conmon[29319]: mgr.14140) 178 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-02-10T04:25:33.393 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:25:33 smithi164 conmon[44544]: debug 2021-02-10T04:25:33.141+0000 7f64d4424200 -1 osd.7 0 log_to_monitors {default=true} 2021-02-10T04:25:34.316 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:34 smithi164 conmon[29288]: audit 2021-02-10T04:25:33.036366+0000 mon.a (mon.0) 527 2021-02-10T04:25:34.316 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:34 smithi164 conmon[29288]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:34.316 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:34 smithi164 conmon[29288]: audit 2021-02-10T04:25:33.146361+0000 mon.b (mon.2) 19 : audit [INF] from='osd.7 v2:172.21.15.164:6812/7' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2021-02-10T04:25:34.317 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:34 smithi164 conmon[29288]: audit 2021-02-10T04:25:33.146783+0000 mon.a (mon.0) 528 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2021-02-10T04:25:34.317 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:34 smithi164 conmon[29288]: audit 2021-02-10T04:25:33.309296+0000 mon.a (mon.0) 529 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2021-02-10T04:25:34.317 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:34 smithi164 conmon[29288]: cluster 2021-02-10T04:25:33.309384+0000 mon.a (mon.0) 530 : cluster [DBG] osdmap e48: 8 total, 7 up, 7 in 2021-02-10T04:25:34.317 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:34 smithi164 conmon[29288]: audit 2021-02-10T04:25:33.309557+0000 mon.a (mon.0) 531 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-02-10T04:25:34.317 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:34 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:25:34.317 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:34 smithi164 conmon[29288]: T04:25:33.311491+0000 mon.b (mon.2) 20 : audit [INF] from='osd.7 v2:172.21.15.164:6812/7' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2021-02-10T04:25:34.317 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:34 smithi164 conmon[29288]: audit 2021-02-10T04:25:34.317 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:34 smithi164 conmon[29288]: 2021-02-10T04:25:33.311902+0000 2021-02-10T04:25:34.317 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:34 smithi164 conmon[29288]: mon.a (mon.0) 532 : audit 2021-02-10T04:25:34.317 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:34 smithi164 conmon[29288]: [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2021-02-10T04:25:34.317 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:34 smithi164 conmon[29288]: audit 2021-02-10T04:25:34.317 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:34 smithi164 conmon[29288]: 2021-02-10T04:25:33 2021-02-10T04:25:34.318 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:34 smithi164 conmon[29288]: .312642+0000 mon.b (mon.2) 21 2021-02-10T04:25:34.318 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:34 smithi164 conmon[29288]: : audit [INF] from='client.? 172.21.15.164:0/3791211194' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2021-02-10T04:25:34.318 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:34 smithi164 conmon[29288]: audit 2021-02-10T04:25:34.318 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:34 smithi164 conmon[29288]: 2021-02-10T04:25:33 2021-02-10T04:25:34.318 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:34 smithi164 conmon[29288]: .316505+0000 mon.a (mon.0) 2021-02-10T04:25:34.318 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:34 smithi164 conmon[29288]: 533 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2021-02-10T04:25:34.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[29319]: audit 2021-02-10T04:25:33.036366+0000 mon.a ( 2021-02-10T04:25:34.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[29319]: mon.0) 527 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:34.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[29319]: audit 2021-02-10T04:25:33.146361+0000 mon.b (mon.2) 19 : audit 2021-02-10T04:25:34.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[29319]: [INF] from='osd.7 v2:172.21.15.164:6812/7' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2021-02-10T04:25:34.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[29319]: audit 2021-02-10T04:25:33.146783+0000 2021-02-10T04:25:34.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[29319]: mon.a (mon.0) 528 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2021-02-10T04:25:34.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:25:34.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[29319]: :25:33.309296+0000 mon.a (mon.0) 529 : audit 2021-02-10T04:25:34.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[29319]: [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2021-02-10T04:25:34.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:25:34.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[29319]: -10T04:25:33.309384+0000 mon.a (mon.0 2021-02-10T04:25:34.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[29319]: ) 530 : cluster [DBG] osdmap e48: 8 total, 7 up, 7 in 2021-02-10T04:25:34.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:25:34.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[29319]: 10T04:25:33.309557+0000 mon.a (mon.0) 531 : audit [DBG] 2021-02-10T04:25:34.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-02-10T04:25:34.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[29319]: audit 2021-02-10T04:25 2021-02-10T04:25:34.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[29319]: :33.311491+0000 mon.b (mon.2) 20 : audit [INF] from='osd.7 v2:172.21.15.164:6812/7' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2021-02-10T04:25:34.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:25:34.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[29319]: -10T04:25:33.311902+0000 mon.a (mon.0) 532 2021-02-10T04:25:34.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[29319]: : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2021-02-10T04:25:34.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[29319]: audit 2021-02-10T04:25:34.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[29319]: 2021-02-10T04:25:33.312642 2021-02-10T04:25:34.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[29319]: +0000 mon.b (mon.2) 21 : audit [INF] from='client.? 172.21.15.164:0/3791211194' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2021-02-10T04:25:34.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[29319]: audit 2021-02-10T04: 2021-02-10T04:25:34.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[29319]: 25:33.316505+0000 mon.a (mon.0) 533 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2021-02-10T04:25:34.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[35961]: audit 2021-02-10T04:25:33.036366+0000 mon.a 2021-02-10T04:25:34.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[35961]: (mon.0) 527 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:34.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[35961]: audit 2021-02-10T04:25 2021-02-10T04:25:34.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[35961]: :33.146361+0000 mon.b (mon.2) 19 : audit [INF] from='osd.7 v2:172.21.15.164:6812/7' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2021-02-10T04:25:34.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:25:34.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[35961]: T04:25:33.146783+0000 mon.a (mon.0) 528 : 2021-02-10T04:25:34.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[35961]: audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2021-02-10T04:25:34.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:25:34.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[35961]: 25:33.309296+0000 mon.a (mon 2021-02-10T04:25:34.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[35961]: .0) 529 : audit [INF] 2021-02-10T04:25:34.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[35961]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2021-02-10T04:25:34.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:25:34.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[35961]: -02-10T04:25:33.309384+0000 mon.a (mon 2021-02-10T04:25:34.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[35961]: .0) 530 : cluster [DBG] osdmap e48: 8 total, 7 up, 7 in 2021-02-10T04:25:34.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[35961]: audit 2021-02-10T04:25:34.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[35961]: 2021-02-10T04:25 2021-02-10T04:25:34.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[35961]: :33.309557+0000 mon.a (mon.0) 531 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-02-10T04:25:34.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:25:34.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[35961]: 10T04:25:33.311491+0000 mon.b (mon.2) 20 : audit 2021-02-10T04:25:34.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[35961]: [INF] from='osd.7 v2:172.21.15.164:6812/7' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2021-02-10T04:25:34.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:25:34.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[35961]: 04:25:33.311902+0000 mon.a (mon.0) 532 : 2021-02-10T04:25:34.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[35961]: audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2021-02-10T04:25:34.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[35961]: audit 2021-02-10T04:25: 2021-02-10T04:25:34.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[35961]: 33.312642+0000 mon.b (mon.2) 21 : audit [INF] 2021-02-10T04:25:34.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[35961]: from='client.? 172.21.15.164:0/3791211194' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2021-02-10T04:25:34.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:25:34.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[35961]: -10T04:25:33.316505+0000 mon.a (mon.0 2021-02-10T04:25:34.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:34 smithi085 conmon[35961]: ) 533 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2021-02-10T04:25:34.642 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:25:34 smithi164 conmon[44544]: debug 2021-02-10T04:25:34.326+0000 7f64ca995700 -1 osd.7 0 waiting for initial osdmap 2021-02-10T04:25:35.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:35 smithi164 conmon[29288]: cluster 2021-02-10T04:25:33.944076+0000 mgr.y 2021-02-10T04:25:35.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:35 smithi164 conmon[29288]: (mgr.14140) 179 : cluster [DBG] pgmap v156: 33 pgs: 32 unknown, 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-02-10T04:25:35.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:35 smithi164 conmon[29288]: audit 2021-02-10T04:25:34.316067+0000 mon.a (mon.0) 534 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2021-02-10T04:25:35.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:35 smithi164 conmon[29288]: audit 2021-02-10T04:25:34.316217+0000 mon.a (mon.0) 535 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2021-02-10T04:25:35.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:35 smithi164 conmon[29288]: cluster 2021-02-10T04:25:34.316352+0000 mon.a (mon.0) 536 : cluster [DBG] osdmap e49: 8 total, 7 up, 7 in 2021-02-10T04:25:35.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:35 smithi164 conmon[29288]: audit 2021-02-10T04:25:34.316697+0000 mon.a (mon.0) 537 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-02-10T04:25:35.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:35 smithi164 conmon[29288]: audit 2021-02-10T04:25:34.332071+0000 mon.a (mon.0) 538 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-02-10T04:25:35.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:35 smithi164 conmon[29288]: audit 2021-02-10T04:25:34.855644+0000 mon.a (mon.0) 539 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:25:35.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[29319]: cluster 2021-02-10T04:25:33.944076+0000 mgr.y (mgr.14140) 179 : cluster [DBG] 2021-02-10T04:25:35.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[29319]: pgmap v156: 33 pgs: 32 unknown, 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-02-10T04:25:35.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[29319]: audit 2021-02-10T04:25:34.316067+0000 mon.a (mon.0) 534 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2021-02-10T04:25:35.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[29319]: audit 2021-02-10T04:25:34.316217+0000 2021-02-10T04:25:35.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[29319]: mon.a (mon.0) 535 : audit [INF] 2021-02-10T04:25:35.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[29319]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2021-02-10T04:25:35.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:25:35.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[29319]: -02-10T04:25:34. 2021-02-10T04:25:35.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[29319]: 316352+0000 mon.a (mon.0) 2021-02-10T04:25:35.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[29319]: 536 : cluster [DBG] osdmap e49: 8 total, 7 up, 7 in 2021-02-10T04:25:35.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:25:35.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[29319]: 02-10T04:25:34.316697+0000 2021-02-10T04:25:35.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[29319]: mon.a (mon.0) 537 : audit [DBG] 2021-02-10T04:25:35.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-02-10T04:25:35.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[29319]: audit 2021-02-10T04:25: 2021-02-10T04:25:35.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[29319]: 34.332071+0000 mon.a (mon.0) 538 : audit 2021-02-10T04:25:35.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[29319]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-02-10T04:25:35.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:25:35.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[29319]: 02-10T04:25:34.855644+0000 2021-02-10T04:25:35.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[29319]: mon.a (mon.0) 539 : audit 2021-02-10T04:25:35.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[29319]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:25:35.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[35961]: cluster 2021-02-10T04:25:33.944076+0000 2021-02-10T04:25:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[35961]: mgr.y (mgr.14140) 179 : cluster [DBG] pgmap v156: 33 pgs: 32 unknown, 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-02-10T04:25:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[35961]: audit 2021-02-10T04:25:34.316067+0000 2021-02-10T04:25:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[35961]: mon.a (mon.0) 534 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2021-02-10T04:25:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[35961]: audit 2021-02-10T04:25:34 2021-02-10T04:25:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[35961]: .316217+0000 mon.a (mon.0) 535 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2021-02-10T04:25:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:25:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[35961]: 25:34.316352+0000 mon.a (mon 2021-02-10T04:25:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[35961]: .0) 536 : cluster [DBG] 2021-02-10T04:25:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[35961]: osdmap e49: 8 total, 7 up, 7 in 2021-02-10T04:25:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[35961]: audit 2021-02-10T04:25:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[35961]: 2021-02-10T04:25:34.316697+0000 mon.a 2021-02-10T04:25:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[35961]: (mon.0) 537 : audit [DBG] 2021-02-10T04:25:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-02-10T04:25:35.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[35961]: audit 2021-02-10T04:25:35.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[35961]: 2021-02-10T04:25:34.332071+0000 mon.a 2021-02-10T04:25:35.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[35961]: (mon.0) 538 : audit 2021-02-10T04:25:35.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[35961]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-02-10T04:25:35.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:25:35.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[35961]: -10T04:25:34.855644+0000 2021-02-10T04:25:35.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[35961]: mon.a (mon.0) 539 : audit 2021-02-10T04:25:35.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:35 smithi085 conmon[35961]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:25:36.356 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:25:36.358 INFO:teuthology.orchestra.run.smithi164.stdout: "id": "3c38ceb2-582a-4b86-86e9-359e50d32d97", 2021-02-10T04:25:36.359 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "realm", 2021-02-10T04:25:36.359 INFO:teuthology.orchestra.run.smithi164.stdout: "current_period": "e4377fc4-64de-4c31-b3e1-a5fdab383ec3", 2021-02-10T04:25:36.359 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1 2021-02-10T04:25:36.359 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:25:36.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[29319]: cluster 2021-02-10T04:25:34.164514+0000 osd.7 (osd. 2021-02-10T04:25:36.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[29319]: 7) 1 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:25:36.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[29319]: cluster 2021-02-10T04:25:34.164613+0000 osd.7 2021-02-10T04:25:36.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[29319]: (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:25:36.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[29319]: cephadm 2021 2021-02-10T04:25:36.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[29319]: -02-10T04:25:35.183300+0000 mgr.y (mgr.14140) 180 : 2021-02-10T04:25:36.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[29319]: cephadm [INF] Detected new or changed devices on smithi164 2021-02-10T04:25:36.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:25:36.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[29319]: -10T04:25:35.187659+0000 mon.a (mon.0) 540 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:36.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:25:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[29319]: :25:35.191241+0000 mon.a (mon.0) 2021-02-10T04:25:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[29319]: 541 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[29319]: audit 2021-02-10T 2021-02-10T04:25:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[29319]: 04:25:35.196507+0000 mon.a (mon.0 2021-02-10T04:25:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[29319]: ) 542 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:25:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[29319]: -10T04:25:35.317910+0000 mon.a 2021-02-10T04:25:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[29319]: (mon.0) 543 : cluster [INF] osd.7 v2:172.21.15.164:6812/7 boot 2021-02-10T04:25:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:25:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[29319]: -02-10T04:25:35 2021-02-10T04:25:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[29319]: .317959+0000 mon.a (mon.0) 544 : 2021-02-10T04:25:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[29319]: cluster [DBG] osdmap e50: 8 total, 8 up, 8 in 2021-02-10T04:25:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[29319]: audit 2021-02-10T04:25:36.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[29319]: 2021-02-10T04:25:35.318151+0000 2021-02-10T04:25:36.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[29319]: mon.a (mon.0) 545 : audit [DBG] 2021-02-10T04:25:36.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-02-10T04:25:36.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:25:36.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[29319]: 25:35.944618+0000 mgr.y (mgr.14140) 181 2021-02-10T04:25:36.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[29319]: : cluster [DBG] pgmap v159: 33 pgs: 4 active+clean, 1 peering, 11 creating+peering, 17 unknown; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-02-10T04:25:36.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[35961]: cluster 2021-02-10T04:25:34.164514+0000 osd.7 (osd. 2021-02-10T04:25:36.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[35961]: 7) 1 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:25:36.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[35961]: cluster 2021-02-10T04:25:34.164613+0000 osd.7 (osd.7) 2 2021-02-10T04:25:36.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[35961]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:25:36.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[35961]: cephadm 2021-02-10T04:25:36.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[35961]: 2021-02-10T04:25:35.183300+0000 mgr.y (mgr.14140) 180 2021-02-10T04:25:36.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[35961]: : cephadm [INF] Detected new or changed devices on smithi164 2021-02-10T04:25:36.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[35961]: audit 2021 2021-02-10T04:25:36.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[35961]: -02-10T04:25:35.187659+0000 mon.a ( 2021-02-10T04:25:36.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[35961]: mon.0) 540 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:36.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:25:36.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[35961]: 04:25:35.191241+0000 mon.a (mon.0) 541 : audit [INF] 2021-02-10T04:25:36.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:36.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:25:36.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[35961]: T04:25:35.196507+0000 mon.a (mon.0) 2021-02-10T04:25:36.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[35961]: 542 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:36.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:25:36.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[35961]: -02-10T04:25:35.317910+0000 mon.a 2021-02-10T04:25:36.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[35961]: (mon.0) 543 : cluster [INF] osd.7 v2:172.21.15.164:6812/7 boot 2021-02-10T04:25:36.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:25:36.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[35961]: 02-10T04:25:35.317959+0000 mon.a (mon.0) 544 2021-02-10T04:25:36.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[35961]: : cluster [DBG] osdmap e50: 8 total, 8 up, 8 in 2021-02-10T04:25:36.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[35961]: audit 2021-02-10T04:25:36.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[35961]: 2021-02-10T04:25:35. 2021-02-10T04:25:36.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[35961]: 318151+0000 mon.a (mon.0) 545 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-02-10T04:25:36.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:25:36.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[35961]: 10T04:25:35.944618+0000 mgr.y (mgr.14140) 181 : cluster [DBG] 2021-02-10T04:25:36.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:36 smithi085 conmon[35961]: pgmap v159: 33 pgs: 4 active+clean, 1 peering, 11 creating+peering, 17 unknown; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-02-10T04:25:36.584 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:36 smithi164 conmon[29288]: cluster 2021-02-10T04:25:34.164514+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:25:36.585 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:36 smithi164 conmon[29288]: cluster 2021-02-10T04:25:34.164613 2021-02-10T04:25:36.585 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:36 smithi164 conmon[29288]: +0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:25:36.586 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:36 smithi164 conmon[29288]: cephadm 2021-02-10T04:25:35.183300+0000 mgr.y (mgr.14140) 180 : cephadm [INF] Detected new or changed devices on smithi164 2021-02-10T04:25:36.586 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:36 smithi164 conmon[29288]: audit 2021-02-10T04:25:35.187659+0000 mon.a 2021-02-10T04:25:36.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:36 smithi164 conmon[29288]: (mon.0) 540 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:36.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:36 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:25:36.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:36 smithi164 conmon[29288]: -10T04:25:35.191241+0000 mon.a (mon.0) 541 2021-02-10T04:25:36.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:36 smithi164 conmon[29288]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:36.589 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:36 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:25:36.589 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:36 smithi164 conmon[29288]: :25:35.196507+0000 mon.a (mon.0) 2021-02-10T04:25:36.590 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:36 smithi164 conmon[29288]: 542 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:36.590 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:36 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:25:36.591 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:36 smithi164 conmon[29288]: T04:25:35.317910+0000 mon.a (mon 2021-02-10T04:25:36.591 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:36 smithi164 conmon[29288]: .0) 543 : cluster [INF] osd.7 v2:172.21.15.164:6812/7 boot 2021-02-10T04:25:36.591 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:36 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:25:36.592 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:36 smithi164 conmon[29288]: 02-10T04:25:35.317959+0000 mon.a 2021-02-10T04:25:36.592 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:36 smithi164 conmon[29288]: (mon.0) 544 : cluster [DBG] osdmap e50: 8 total, 8 up, 8 in 2021-02-10T04:25:36.592 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:36 smithi164 conmon[29288]: 2021-02-10T04:25:36.593 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:36 smithi164 conmon[29288]: audit 2021-02-10T04:25: 2021-02-10T04:25:36.593 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:36 smithi164 conmon[29288]: 35.318151+0000 mon.a (mon.0) 545 : audit 2021-02-10T04:25:36.594 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:36 smithi164 conmon[29288]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-02-10T04:25:36.594 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:36 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:25:36.594 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:36 smithi164 conmon[29288]: :25:35.944618+0000 mgr.y (mgr.14140) 2021-02-10T04:25:36.595 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:36 smithi164 conmon[29288]: 181 : cluster [DBG] pgmap v159: 33 pgs: 4 active+clean, 1 peering, 11 creating+peering, 17 unknown; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-02-10T04:25:37.056 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- radosgw-admin zonegroup create --rgw-zonegroup=default --master --default 2021-02-10T04:25:37.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:37 smithi164 conmon[29288]: cluster 2021-02-10T04:25:36.324169+0000 mon.a (mon.0) 546 : 2021-02-10T04:25:37.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:37 smithi164 conmon[29288]: cluster [DBG] osdmap e51: 8 total, 8 up, 8 in 2021-02-10T04:25:37.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:37 smithi085 conmon[29319]: cluster 2021-02-10T04:25:36.324169+0000 mon.a (mon.0) 546 : cluster [DBG] osdmap e51: 8 total, 8 up, 8 in 2021-02-10T04:25:37.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:37 smithi085 conmon[35961]: cluster 2021-02-10T04:25:36.324169+0000 2021-02-10T04:25:37.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:37 smithi085 conmon[35961]: mon.a (mon.0) 546 : cluster [DBG] osdmap e51: 8 total, 8 up, 8 in 2021-02-10T04:25:38.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:38 smithi164 conmon[29288]: cluster 2021-02-10T04:25:37.336609+0000 mon.a (mon.0) 547 : cluster 2021-02-10T04:25:38.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:38 smithi164 conmon[29288]: [DBG] osdmap e52: 8 total, 8 up, 8 in 2021-02-10T04:25:38.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:38 smithi164 conmon[29288]: cluster 2021-02-10T04:25:37.945124+0000 mgr.y (mgr.14140) 182 : cluster [DBG] pgmap v162: 33 pgs: 8 active+clean, 2 peering, 11 creating+peering, 12 unknown; 247 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-02-10T04:25:38.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:38 smithi164 conmon[29288]: audit 2021-02-10T04:25:38.032259+0000 mon.a (mon.0) 548 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:25:38.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:38 smithi164 conmon[29288]: audit 2021-02-10T04:25:38.033315+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:25:38.664 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:25:38.664 INFO:teuthology.orchestra.run.smithi164.stdout: "id": "a18611e9-0da4-4744-a658-9506d2033bed", 2021-02-10T04:25:38.664 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "default", 2021-02-10T04:25:38.664 INFO:teuthology.orchestra.run.smithi164.stdout: "api_name": "default", 2021-02-10T04:25:38.664 INFO:teuthology.orchestra.run.smithi164.stdout: "is_master": "true", 2021-02-10T04:25:38.664 INFO:teuthology.orchestra.run.smithi164.stdout: "endpoints": [], 2021-02-10T04:25:38.664 INFO:teuthology.orchestra.run.smithi164.stdout: "hostnames": [], 2021-02-10T04:25:38.664 INFO:teuthology.orchestra.run.smithi164.stdout: "hostnames_s3website": [], 2021-02-10T04:25:38.664 INFO:teuthology.orchestra.run.smithi164.stdout: "master_zone": "", 2021-02-10T04:25:38.665 INFO:teuthology.orchestra.run.smithi164.stdout: "zones": [], 2021-02-10T04:25:38.665 INFO:teuthology.orchestra.run.smithi164.stdout: "placement_targets": [], 2021-02-10T04:25:38.665 INFO:teuthology.orchestra.run.smithi164.stdout: "default_placement": "", 2021-02-10T04:25:38.665 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_id": "3c38ceb2-582a-4b86-86e9-359e50d32d97", 2021-02-10T04:25:38.665 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_policy": { 2021-02-10T04:25:38.665 INFO:teuthology.orchestra.run.smithi164.stdout: "groups": [] 2021-02-10T04:25:38.665 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:25:38.665 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:25:38.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:38 smithi085 conmon[29319]: cluster 2021-02-10T04:25:37.336609+0000 mon.a (mon.0) 547 : cluster [DBG] 2021-02-10T04:25:38.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:38 smithi085 conmon[29319]: osdmap e52: 8 total, 8 up, 8 in 2021-02-10T04:25:38.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:38 smithi085 conmon[29319]: cluster 2021-02-10T04:25:37.945124+0000 mgr.y (mgr.14140) 2021-02-10T04:25:38.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:38 smithi085 conmon[29319]: 182 : cluster [DBG] pgmap v162: 33 pgs: 8 active+clean, 2 peering, 11 creating+peering, 12 unknown; 247 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-02-10T04:25:38.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:38 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:25:38.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:38 smithi085 conmon[29319]: 02-10T04:25:38.032259+0000 2021-02-10T04:25:38.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:38 smithi085 conmon[29319]: mon.a (mon.0) 548 : audit 2021-02-10T04:25:38.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:38 smithi085 conmon[29319]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:25:38.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:38 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:25:38.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:38 smithi085 conmon[29319]: -10T04:25:38.033315+0000 mon.a ( 2021-02-10T04:25:38.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:38 smithi085 conmon[29319]: mon.0) 549 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:25:38.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:38 smithi085 conmon[29319]: 2021-02-10T04:25:38.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:38 smithi085 conmon[35961]: cluster 2021-02-10T04:25:37.336609+0000 2021-02-10T04:25:38.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:38 smithi085 conmon[35961]: mon.a (mon.0) 547 : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2021-02-10T04:25:38.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:38 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:25:38.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:38 smithi085 conmon[35961]: 25:37.945124+0000 mgr.y ( 2021-02-10T04:25:38.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:38 smithi085 conmon[35961]: mgr.14140) 182 : cluster [DBG] 2021-02-10T04:25:38.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:38 smithi085 conmon[35961]: pgmap v162: 33 pgs: 8 active+clean, 2 peering, 11 creating+peering, 12 unknown; 247 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-02-10T04:25:38.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:38 smithi085 conmon[35961]: audit 2021-02-10T04:25:38.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:38 smithi085 conmon[35961]: 2021-02-10T04:25 2021-02-10T04:25:38.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:38 smithi085 conmon[35961]: :38.032259+0000 mon.a (mon.0 2021-02-10T04:25:38.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:38 smithi085 conmon[35961]: ) 548 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:25:38.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:38 smithi085 conmon[35961]: 2021-02-10T04:25:38.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:38 smithi085 conmon[35961]: audit 2021-02-10T04:25:38.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:38 smithi085 conmon[35961]: 2021- 2021-02-10T04:25:38.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:38 smithi085 conmon[35961]: 02-10T04:25:38.033315+0000 2021-02-10T04:25:38.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:38 smithi085 conmon[35961]: mon.a (mon.0) 549 : audit 2021-02-10T04:25:38.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:38 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:25:39.262 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- radosgw-admin zone create --rgw-zonegroup=default --rgw-zone zone --master --default 2021-02-10T04:25:40.838 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:25:40.839 INFO:teuthology.orchestra.run.smithi164.stdout: "id": "31f351f5-a7bc-4340-9159-1339b03dc902", 2021-02-10T04:25:40.839 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "zone", 2021-02-10T04:25:40.839 INFO:teuthology.orchestra.run.smithi164.stdout: "domain_root": "zone.rgw.meta:root", 2021-02-10T04:25:40.839 INFO:teuthology.orchestra.run.smithi164.stdout: "control_pool": "zone.rgw.control", 2021-02-10T04:25:40.839 INFO:teuthology.orchestra.run.smithi164.stdout: "gc_pool": "zone.rgw.log:gc", 2021-02-10T04:25:40.839 INFO:teuthology.orchestra.run.smithi164.stdout: "lc_pool": "zone.rgw.log:lc", 2021-02-10T04:25:40.839 INFO:teuthology.orchestra.run.smithi164.stdout: "log_pool": "zone.rgw.log", 2021-02-10T04:25:40.839 INFO:teuthology.orchestra.run.smithi164.stdout: "intent_log_pool": "zone.rgw.log:intent", 2021-02-10T04:25:40.839 INFO:teuthology.orchestra.run.smithi164.stdout: "usage_log_pool": "zone.rgw.log:usage", 2021-02-10T04:25:40.839 INFO:teuthology.orchestra.run.smithi164.stdout: "roles_pool": "zone.rgw.meta:roles", 2021-02-10T04:25:40.839 INFO:teuthology.orchestra.run.smithi164.stdout: "reshard_pool": "zone.rgw.log:reshard", 2021-02-10T04:25:40.839 INFO:teuthology.orchestra.run.smithi164.stdout: "user_keys_pool": "zone.rgw.meta:users.keys", 2021-02-10T04:25:40.839 INFO:teuthology.orchestra.run.smithi164.stdout: "user_email_pool": "zone.rgw.meta:users.email", 2021-02-10T04:25:40.840 INFO:teuthology.orchestra.run.smithi164.stdout: "user_swift_pool": "zone.rgw.meta:users.swift", 2021-02-10T04:25:40.840 INFO:teuthology.orchestra.run.smithi164.stdout: "user_uid_pool": "zone.rgw.meta:users.uid", 2021-02-10T04:25:40.840 INFO:teuthology.orchestra.run.smithi164.stdout: "otp_pool": "zone.rgw.otp", 2021-02-10T04:25:40.840 INFO:teuthology.orchestra.run.smithi164.stdout: "system_key": { 2021-02-10T04:25:40.840 INFO:teuthology.orchestra.run.smithi164.stdout: "access_key": "", 2021-02-10T04:25:40.840 INFO:teuthology.orchestra.run.smithi164.stdout: "secret_key": "" 2021-02-10T04:25:40.840 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:25:40.840 INFO:teuthology.orchestra.run.smithi164.stdout: "placement_pools": [ 2021-02-10T04:25:40.840 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:25:40.840 INFO:teuthology.orchestra.run.smithi164.stdout: "key": "default-placement", 2021-02-10T04:25:40.840 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2021-02-10T04:25:40.840 INFO:teuthology.orchestra.run.smithi164.stdout: "index_pool": "zone.rgw.buckets.index", 2021-02-10T04:25:40.840 INFO:teuthology.orchestra.run.smithi164.stdout: "storage_classes": { 2021-02-10T04:25:40.841 INFO:teuthology.orchestra.run.smithi164.stdout: "STANDARD": { 2021-02-10T04:25:40.841 INFO:teuthology.orchestra.run.smithi164.stdout: "data_pool": "zone.rgw.buckets.data" 2021-02-10T04:25:40.841 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:25:40.841 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:25:40.841 INFO:teuthology.orchestra.run.smithi164.stdout: "data_extra_pool": "zone.rgw.buckets.non-ec", 2021-02-10T04:25:40.841 INFO:teuthology.orchestra.run.smithi164.stdout: "index_type": 0 2021-02-10T04:25:40.841 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:25:40.841 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:25:40.841 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2021-02-10T04:25:40.841 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_id": "3c38ceb2-582a-4b86-86e9-359e50d32d97", 2021-02-10T04:25:40.841 INFO:teuthology.orchestra.run.smithi164.stdout: "notif_pool": "zone.rgw.log:notif" 2021-02-10T04:25:40.841 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:25:41.142 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:40 smithi164 conmon[29288]: cluster 2021-02-10T04:25:39.867828+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:25:41.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:40 smithi164 conmon[29288]: cluster 2021-02-10T04:25:39.867917+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:25:41.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:40 smithi164 conmon[29288]: cluster 2021-02-10T04:25:39.915328+0000 mon.a (mon.0) 550 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2021-02-10T04:25:41.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:40 smithi164 conmon[29288]: cluster 2021-02-10T04:25:39.945668+0000 mgr.y (mgr.14140) 183 : cluster [DBG] pgmap v164: 33 pgs: 24 active+clean, 2 peering, 7 creating+peering; 524 B data, 42 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s wr, 2 op/s 2021-02-10T04:25:41.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:40 smithi085 conmon[29319]: cluster 2021-02-10T04:25:39.867828+0000 osd.4 (osd. 2021-02-10T04:25:41.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:40 smithi085 conmon[29319]: 4) 3 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:25:41.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:40 smithi085 conmon[29319]: cluster 2021-02-10T04:25:39.867917+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:25:41.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:40 smithi085 conmon[29319]: cluster 2021-02-10T04:25:39.915328+0000 mon.a (mon.0) 550 : cluster 2021-02-10T04:25:41.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:40 smithi085 conmon[29319]: [DBG] osdmap e53: 8 total, 8 up, 8 in 2021-02-10T04:25:41.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:40 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:25:41.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:40 smithi085 conmon[29319]: :25:39.945668+0000 mgr.y (mgr.14140 2021-02-10T04:25:41.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:40 smithi085 conmon[29319]: ) 183 : cluster [DBG] pgmap v164: 33 pgs: 24 active+clean, 2 peering, 7 creating+peering; 524 B data, 42 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s wr, 2 op/s 2021-02-10T04:25:41.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:40 smithi085 conmon[35961]: cluster 2021-02-10T04:25:39.867828+0000 2021-02-10T04:25:41.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:40 smithi085 conmon[35961]: osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:25:41.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:40 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:25:41.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:40 smithi085 conmon[35961]: 10T04:25:39.867917+0000 osd.4 (osd.4) 4 : cluster 2021-02-10T04:25:41.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:40 smithi085 conmon[35961]: [DBG] purged_snaps scrub ok 2021-02-10T04:25:41.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:40 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:25:41.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:40 smithi085 conmon[35961]: 02-10T04:25:39.915328+0000 2021-02-10T04:25:41.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:40 smithi085 conmon[35961]: mon.a (mon.0) 550 : 2021-02-10T04:25:41.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:40 smithi085 conmon[35961]: cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2021-02-10T04:25:41.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:40 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:25:41.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:40 smithi085 conmon[35961]: 02-10T04:25:39.945668+0000 mgr.y 2021-02-10T04:25:41.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:40 smithi085 conmon[35961]: (mgr.14140) 183 : cluster [DBG] pgmap v164: 33 pgs: 24 active+clean, 2 peering, 7 creating+peering; 524 B data, 42 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s wr, 2 op/s 2021-02-10T04:25:41.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:40 smithi085 conmon[35961]: 2021-02-10T04:25:41.453 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph orch apply rgw realm zone --placement '1;smithi085=realm.zone.a' 2021-02-10T04:25:43.050 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:41 smithi085 conmon[29319]: cluster 2021-02-10T04:25:40.912864+0000 mon.a (mon.0) 551 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-02-10T04:25:43.050 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:41 smithi085 conmon[35961]: cluster 2021-02-10T04:25:40.912864+0000 2021-02-10T04:25:43.050 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:41 smithi085 conmon[35961]: mon.a (mon.0) 551 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-02-10T04:25:43.050 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:41 smithi164 conmon[29288]: cluster 2021-02-10T04:25:40.912864+0000 mon.a (mon.0) 551 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-02-10T04:25:43.050 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:41 smithi164 conmon[29288]: 2021-02-10T04:25:43.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:42 smithi085 conmon[29319]: cluster 2021-02-10T04:25:41.946171+0000 mgr.y (mgr.14140) 184 : cluster [DBG] pgmap v165: 33 pgs: 33 active+clean; 961 B data, 43 MiB used, 715 GiB / 715 GiB avail; 364 B/s rd, 1.8 KiB/s wr, 2 op/s 2021-02-10T04:25:43.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:42 smithi085 conmon[35961]: cluster 2021-02-10T04:25:41.946171+0000 mgr.y (mgr.14140) 184 : cluster [DBG] 2021-02-10T04:25:43.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:42 smithi085 conmon[35961]: pgmap v165: 33 pgs: 33 active+clean; 961 B data, 43 MiB used, 715 GiB / 715 GiB avail; 364 B/s rd, 1.8 KiB/s wr, 2 op/s 2021-02-10T04:25:43.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:42 smithi164 conmon[29288]: cluster 2021-02-10T04:25:41.946171+0000 mgr.y (mgr.14140) 184 : cluster [DBG] pgmap v165: 33 pgs: 33 active+clean; 961 B data, 43 MiB used, 715 GiB / 715 GiB avail; 364 B/s rd, 1.8 KiB/s wr, 2 op/s 2021-02-10T04:25:44.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:43 smithi164 conmon[29288]: cluster 2021-02-10T04:25:42.921436+0000 mon.a (mon.0) 552 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2021-02-10T04:25:44.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:43 smithi164 conmon[29288]: cluster 2021-02-10T04:25:42.921496+0000 mon.a (mon.0) 553 : cluster [INF] Cluster is now healthy 2021-02-10T04:25:44.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:43 smithi164 conmon[29288]: audit 2021-02-10T04:25:43.040728+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:44.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:43 smithi085 conmon[29319]: cluster 2021-02-10T04:25:42.921436+0000 mon.a (mon.0) 552 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2021-02-10T04:25:44.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:43 smithi085 conmon[29319]: cluster 2021-02-10T04:25:42.921496+0000 mon.a (mon. 2021-02-10T04:25:44.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:43 smithi085 conmon[29319]: 0) 553 : cluster [INF] Cluster is now healthy 2021-02-10T04:25:44.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:43 smithi085 conmon[29319]: audit 2021-02-10T04:25:43.040728+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:44.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:43 smithi085 conmon[35961]: cluster 2021-02-10T04:25:42.921436+0000 mon.a (mon.0) 552 : 2021-02-10T04:25:44.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:43 smithi085 conmon[35961]: cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2021-02-10T04:25:44.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:43 smithi085 conmon[35961]: cluster 2021-02-10T04:25:42.921496+0000 mon.a (mon.0) 553 : cluster [INF] Cluster is now healthy 2021-02-10T04:25:44.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:43 smithi085 conmon[35961]: 2021-02-10T04:25:44.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:43 smithi085 conmon[35961]: audit 2021-02-10T04:25:43. 2021-02-10T04:25:44.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:43 smithi085 conmon[35961]: 040728+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:44.945 INFO:teuthology.orchestra.run.smithi164.stdout:Scheduled rgw.realm.zone update... 2021-02-10T04:25:45.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:44 smithi085 conmon[29319]: cluster 2021-02-10T04:25:43.946585+0000 mgr.y (mgr.14140) 185 : cluster [DBG] pgmap v166: 33 pgs: 33 active+clean; 1.8 KiB data, 43 MiB used, 715 GiB / 715 GiB avail; 671 B/s rd, 1.6 KiB/s wr, 2 op/s 2021-02-10T04:25:45.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:44 smithi085 conmon[35961]: cluster 2021-02-10T04:25:43.946585+0000 mgr.y (mgr.14140) 2021-02-10T04:25:45.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:44 smithi085 conmon[35961]: 185 : cluster [DBG] pgmap v166: 33 pgs: 33 active+clean; 1.8 KiB data, 43 MiB used, 715 GiB / 715 GiB avail; 671 B/s rd, 1.6 KiB/s wr, 2 op/s 2021-02-10T04:25:45.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:44 smithi164 conmon[29288]: cluster 2021-02-10T04:25:43.946585+0000 mgr.y (mgr.14140) 185 : cluster 2021-02-10T04:25:45.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:44 smithi164 conmon[29288]: [DBG] pgmap v166: 33 pgs: 33 active+clean; 1.8 KiB data, 43 MiB used, 715 GiB / 715 GiB avail; 671 B/s rd, 1.6 KiB/s wr, 2 op/s 2021-02-10T04:25:45.622 DEBUG:teuthology.orchestra.run.smithi085:rgw.realm.zone.a> sudo journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@rgw.realm.zone.a.service 2021-02-10T04:25:45.625 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi164 2021-02-10T04:25:45.625 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pool create iscsi 3 3 replicated 2021-02-10T04:25:45.655 INFO:journalctl@ceph.rgw.realm.zone.a.smithi085.stdout:-- Logs begin at Wed 2021-02-10 04:09:23 UTC. -- 2021-02-10T04:25:46.319 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:45 smithi164 conmon[29288]: audit 2021-02-10T04:25:44.934610+0000 mgr.y (mgr.14140) 186 : audit [DBG] 2021-02-10T04:25:46.320 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:45 smithi164 conmon[29288]: from='client.34331 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "realm_name": "realm", "zone_name": "zone", "placement": "1;smithi085=realm.zone.a", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:25:46.320 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:45 smithi164 conmon[29288]: cephadm 2021-02-10T04:25:44.935790+0000 mgr.y (mgr.14140) 187 : cephadm [INF] Saving service rgw.realm.zone spec with placement smithi085=realm.zone.a;count:1 2021-02-10T04:25:46.320 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:45 smithi164 conmon[29288]: audit 2021-02-10T04:25:44.944986+0000 mon.a (mon.0) 555 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:46.320 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:45 smithi164 conmon[29288]: audit 2021-02-10T04:25:44.950730+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:46.320 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:45 smithi164 conmon[29288]: audit 2021-02-10T04:25:44.952287+0000 mon.a (mon.0) 557 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2021-02-10T04:25:46.320 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:45 smithi164 conmon[29288]: audit 2021-02-10T04:25:44.953076+0000 mon.a (mon.0) 558 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.realm.zone.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2021-02-10T04:25:46.320 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:45 smithi164 conmon[29288]: audit 2021-02-10T04:25:44.956842+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.realm.zone.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2021-02-10T04:25:46.320 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:45 smithi164 conmon[29288]: audit 2021-02-10T04:25:45.103658+0000 mon.a (mon.0) 560 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:46.320 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:45 smithi164 conmon[29288]: audit 2021-02-10T04:25:45.107053+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:46.320 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:45 smithi164 conmon[29288]: audit 2021-02-10T04:25:45.110743+0000 mon.a (mon.0) 562 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:46.320 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:45 smithi164 conmon[29288]: audit 2021-02-10T04:25:45.114033+0000 mon.a (mon.0) 563 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:46.321 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:45 smithi164 conmon[29288]: audit 2021-02-10T04:25:45.114627+0000 mon.a (mon.0) 564 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.realm.zone.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2021-02-10T04:25:46.321 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:45 smithi164 conmon[29288]: audit 2021-02-10T04:25:45.115416+0000 mon.a (mon.0) 565 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:25:46.321 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:45 smithi164 conmon[29288]: audit 2021-02-10T04:25:45.116225+0000 mon.a (mon.0) 566 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "client.rgw.realm.zone.a", "key": "container_image"}]: dispatch 2021-02-10T04:25:46.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[29319]: audit 2021-02-10T04:25:44.934610+0000 mgr.y (mgr.14140) 186 : audit [DBG] from='client.34331 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "realm_name": "realm", "zone_name": "zone", "placement": "1;smithi085=realm.zone.a", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:25:46.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[29319]: cephadm 2021-02-10T04:25:44.935790+0000 mgr.y (mgr.14140) 187 : cephadm [INF] Saving service rgw.realm.zone spec with placement smithi085=realm.zone.a;count:1 2021-02-10T04:25:46.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[29319]: audit 2021-02-10T04:25:44.944986+0000 mon.a (mon.0) 555 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:46.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[29319]: audit 2021-02-10T04:25:44.950730+0000 mon.a (mon.0) 556 2021-02-10T04:25:46.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[29319]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:46.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[29319]: audit 2021-02-10T04:25:44.952287+0000 mon.a (mon.0) 557 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2021-02-10T04:25:46.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[29319]: audit 2021-02-10T04:25:44.953076+0000 mon.a (mon.0) 558 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.realm.zone.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2021-02-10T04:25:46.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[29319]: audit 2021-02-10T04:25:44.956842+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.realm.zone.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2021-02-10T04:25:46.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[29319]: audit 2021-02-10T04:25:45.103658+0000 mon.a (mon.0) 560 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:46.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[29319]: audit 2021-02-10T04:25:45.107053+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:46.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[29319]: audit 2021-02-10T04:25:45.110743+0000 mon.a (mon.0) 562 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:46.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[29319]: audit 2021-02-10T04:25:45.114033+0000 mon.a (mon.0) 563 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:46.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[29319]: audit 2021-02-10T04:25:45.114627+0000 mon.a (mon.0) 564 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.realm.zone.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2021-02-10T04:25:46.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[29319]: audit 2021-02-10T04:25:45.115416+0000 mon.a (mon.0) 565 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:25:46.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[29319]: audit 2021-02-10T04:25:45.116225+0000 mon.a (mon.0) 566 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "client.rgw.realm.zone.a", "key": "container_image"}]: dispatch 2021-02-10T04:25:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: audit 2021-02-10T04:25:44.934610+0000 mgr.y (mgr.14140) 186 : audit [DBG] from='client.34331 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "realm_name": "realm", "zone_name": "zone", "placement": "1;smithi085=realm.zone.a", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:25:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: cephadm 2021-02-10T04:25:44.935790+0000 mgr.y (mgr.14140) 187 : cephadm [INF] Saving service rgw.realm.zone spec with placement smithi085=realm.zone.a;count:1 2021-02-10T04:25:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: audit 2021-02-10T04:25:44.944986+0000 mon.a (mon.0) 555 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: audit 2021-02-10T04:25:44.950730+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: audit 2021-02-10T04:25:44.952287+0000 mon.a (mon. 2021-02-10T04:25:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: 0) 557 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2021-02-10T04:25:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:25:46.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: 10T04:25:44.953076+0000 mon.a (mon 2021-02-10T04:25:46.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: .0) 558 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.realm.zone.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2021-02-10T04:25:46.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:25:46.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: T04:25:44.956842+0000 mon.a (mon.0 2021-02-10T04:25:46.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: ) 559 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.realm.zone.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2021-02-10T04:25:46.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: audit 2021-02-10T04:25:46.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: 2021-02-10T04:25:45.103658+0000 mon.a (mon.0) 560 : audit [INF] 2021-02-10T04:25:46.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:46.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:25:46.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: 02-10T04:25:45.107053+0000 mon.a (mon.0 2021-02-10T04:25:46.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: ) 561 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:46.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: audit 2021-02-10T04:25:46.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: 2021-02-10T04:25: 2021-02-10T04:25:46.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: 45.110743+0000 mon.a (mon.0) 562 2021-02-10T04:25:46.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:46.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:25:46.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: 10T04:25:45.114033+0000 mon.a 2021-02-10T04:25:46.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: (mon.0) 563 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:46.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:25:46.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: 02-10T04:25:45.114627+0000 2021-02-10T04:25:46.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: mon.a (mon.0) 564 : audit [INF] 2021-02-10T04:25:46.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.realm.zone.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2021-02-10T04:25:46.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: audit 2021 2021-02-10T04:25:46.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: -02-10T04:25:45.115416+0000 mon.a (mon.0) 2021-02-10T04:25:46.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: 565 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:25:46.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: audit 2021-02-10T04:25:45. 2021-02-10T04:25:46.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: 116225+0000 mon.a (mon.0) 566 : audit [DBG] 2021-02-10T04:25:46.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:45 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "client.rgw.realm.zone.a", "key": "container_image"}]: dispatch 2021-02-10T04:25:47.117 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:46 smithi164 conmon[29288]: cephadm 2021-02-10T04:25:45.111355+0000 mgr.y (mgr.14140) 188 : cephadm [INF] Saving service rgw.realm.zone spec with placement smithi085=realm.zone.a;count:1 2021-02-10T04:25:47.117 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:46 smithi164 conmon[29288]: cephadm 2021-02-10T04:25:45.115891+0000 mgr.y (mgr.14140) 189 : cephadm [INF] Deploying daemon rgw.realm.zone.a on smithi085 2021-02-10T04:25:47.117 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:46 smithi164 conmon[29288]: cluster 2021-02-10T04:25:45.947062+0000 mgr.y (mgr.14140) 190 : cluster [DBG] pgmap v167: 33 pgs: 33 active+clean; 1.9 KiB data, 43 MiB used, 715 GiB / 715 GiB avail; 713 B/s rd, 1.7 KiB/s wr, 2 op/s 2021-02-10T04:25:47.117 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:46 smithi164 conmon[29288]: cluster 2021-02-10T04:25:46.116842+0000 osd.1 (osd.1) 5 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:25:47.117 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:46 smithi164 conmon[29288]: cluster 2021-02-10T04:25:46.116989+0000 osd.1 (osd.1) 6 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:25:47.191 INFO:journalctl@ceph.rgw.realm.zone.a.smithi085.stdout:Feb 10 04:25:47 smithi085 systemd[1]: Starting Ceph rgw.realm.zone.a for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T04:25:47.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:46 smithi085 conmon[29319]: cephadm 2021-02-10T04:25:45.111355+0000 mgr.y 2021-02-10T04:25:47.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:46 smithi085 conmon[29319]: (mgr.14140) 188 : cephadm [INF] Saving service rgw.realm.zone spec with placement smithi085=realm.zone.a;count:1 2021-02-10T04:25:47.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:46 smithi085 conmon[29319]: cephadm 2021-02-10T04:25:45.115891+0000 mgr.y (mgr.14140) 189 : cephadm [INF] Deploying daemon rgw.realm.zone.a on smithi085 2021-02-10T04:25:47.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:46 smithi085 conmon[29319]: cluster 2021-02-10T04:25:45.947062+0000 mgr.y (mgr.14140) 190 : cluster [DBG] pgmap v167: 33 pgs: 33 active+clean; 1.9 KiB data, 43 MiB used, 715 GiB / 715 GiB avail; 713 B/s rd, 1.7 KiB/s wr, 2 op/s 2021-02-10T04:25:47.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:46 smithi085 conmon[29319]: cluster 2021-02-10T04:25:46.116842+0000 osd.1 (osd.1) 5 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:25:47.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:46 smithi085 conmon[29319]: cluster 2021-02-10T04:25:46.116989+0000 osd.1 (osd.1) 6 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:25:47.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:46 smithi085 conmon[35961]: cephadm 2021-02-10T04:25:45.111355+0000 mgr.y (mgr 2021-02-10T04:25:47.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:46 smithi085 conmon[35961]: .14140) 188 : cephadm [INF] Saving service rgw.realm.zone spec with placement smithi085=realm.zone.a;count:1 2021-02-10T04:25:47.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:46 smithi085 conmon[35961]: cephadm 2021-02-10T04:25:45.115891+0000 mgr.y (mgr. 2021-02-10T04:25:47.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:46 smithi085 conmon[35961]: 14140) 189 : cephadm [INF] Deploying daemon rgw.realm.zone.a on smithi085 2021-02-10T04:25:47.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:46 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:25:47.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:46 smithi085 conmon[35961]: -10T04:25:45.947062+0000 mgr.y (mgr.14140) 190 : cluster 2021-02-10T04:25:47.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:46 smithi085 conmon[35961]: [DBG] pgmap v167: 33 pgs: 33 active+clean; 1.9 KiB data, 43 MiB used, 715 GiB / 715 GiB avail; 713 B/s rd, 1.7 KiB/s wr, 2 op/s 2021-02-10T04:25:47.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:46 smithi085 conmon[35961]: cluster 2021-02-10T04:25:47.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:46 smithi085 conmon[35961]: 2021-02-10T04:25:46.116842+0000 osd.1 (osd.1 2021-02-10T04:25:47.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:46 smithi085 conmon[35961]: ) 5 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:25:47.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:46 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:25:47.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:46 smithi085 conmon[35961]: :25:46.116989+0000 osd.1 (osd.1) 6 : 2021-02-10T04:25:47.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:46 smithi085 conmon[35961]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:25:47.980 INFO:journalctl@ceph.rgw.realm.zone.a.smithi085.stdout:Feb 10 04:25:47 smithi085 bash[54538]: e0cbee875f85c20b193b0c78eb310c391dbd682388494e701409bdf9134b43ff 2021-02-10T04:25:47.980 INFO:journalctl@ceph.rgw.realm.zone.a.smithi085.stdout:Feb 10 04:25:47 smithi085 systemd[1]: Started Ceph rgw.realm.zone.a for 42224fce-6b57-11eb-8feb-001a4aab830c. 2021-02-10T04:25:47.990 INFO:teuthology.orchestra.run.smithi164.stderr:pool 'iscsi' created 2021-02-10T04:25:48.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:47 smithi164 conmon[29288]: cluster 2021-02-10T04:25:48.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:47 smithi164 conmon[29288]: 2021-02-10T04:25:46.952209+0000 mon.a (mon.0) 567 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2021-02-10T04:25:48.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:47 smithi164 conmon[29288]: audit 2021-02-10T04:25:47.406727+0000 2021-02-10T04:25:48.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:47 smithi164 conmon[29288]: mon.b (mon.2) 22 : audit [INF] from='client.? 172.21.15.164:0/3190041417' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2021-02-10T04:25:48.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:47 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:25:48.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:47 smithi164 conmon[29288]: :25:47.407164+0000 mon.a (mon.0) 568 : audit [INF] 2021-02-10T04:25:48.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:47 smithi164 conmon[29288]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2021-02-10T04:25:48.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[35961]: cluster 2021-02-10T04:25:46. 2021-02-10T04:25:48.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[35961]: 952209+0000 mon.a (mon.0) 567 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2021-02-10T04:25:48.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:25:48.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[35961]: :25:47.406727+0000 mon.b (mon.2) 22 : audit [INF] from='client.? 172.21.15.164:0/3190041417' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2021-02-10T04:25:48.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[35961]: audit 2021 2021-02-10T04:25:48.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[35961]: -02-10T04:25:47.407164+0000 mon.a 2021-02-10T04:25:48.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[35961]: (mon.0) 568 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2021-02-10T04:25:48.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[35961]: 2021-02-10T04:25:48.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[29319]: cluster 2021-02-10T04:25:48.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[29319]: 2021-02-10T04:25:48.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[29319]: 2021 2021-02-10T04:25:48.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[29319]: - 2021-02-10T04:25:48.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[29319]: 02 2021-02-10T04:25:48.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[29319]: - 2021-02-10T04:25:48.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[29319]: 10 2021-02-10T04:25:48.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[29319]: T 2021-02-10T04:25:48.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[29319]: 04 2021-02-10T04:25:48.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[29319]: : 2021-02-10T04:25:48.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[29319]: 25 2021-02-10T04:25:48.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[29319]: : 2021-02-10T04:25:48.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[29319]: 46 2021-02-10T04:25:48.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[29319]: . 2021-02-10T04:25:48.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[29319]: 952209 2021-02-10T04:25:48.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[29319]: +0000 2021-02-10T04:25:48.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[29319]: 2021-02-10T04:25:48.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[29319]: mon.a (mon.0) 567 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2021-02-10T04:25:48.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[29319]: 2021-02-10T04:25:48.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:25:48.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[29319]: :25:47.406727+0000 mon.b (mon.2) 22 2021-02-10T04:25:48.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[29319]: : audit [INF] from='client.? 172.21.15.164:0/3190041417' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2021-02-10T04:25:48.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:25:48.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[29319]: -10T04:25:47.407164+0000 mon.a ( 2021-02-10T04:25:48.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:47 smithi085 conmon[29319]: mon.0) 568 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2021-02-10T04:25:48.560 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pool application enable iscsi rbd 2021-02-10T04:25:49.261 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:49 smithi164 conmon[29288]: cluster 2021-02-10T04:25:49.261 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:49 smithi164 conmon[29288]: 2021-02-10T04:25:49.261 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:49 smithi164 conmon[29288]: 2021 2021-02-10T04:25:49.261 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:49 smithi164 conmon[29288]: - 2021-02-10T04:25:49.261 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:49 smithi164 conmon[29288]: 02 2021-02-10T04:25:49.261 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:49 smithi164 conmon[29288]: - 2021-02-10T04:25:49.261 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:49 smithi164 conmon[29288]: 10T04:25: 2021-02-10T04:25:49.262 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:49 smithi164 conmon[29288]: 47.947379+0000 mgr.y (mgr.14140) 191 : cluster [DBG] pgmap v169: 33 pgs: 33 active+clean; 1.9 KiB data, 43 MiB used, 715 GiB / 715 GiB avail; 764 B/s rd, 1019 B/s wr, 1 op/s 2021-02-10T04:25:49.262 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:49 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:25:49.262 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:49 smithi164 conmon[29288]: 02-10T04:25:47.984057+0000 mon.a (mon. 2021-02-10T04:25:49.262 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:49 smithi164 conmon[29288]: 0) 569 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:49.262 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:49 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:25:49.262 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:49 smithi164 conmon[29288]: 02-10T04:25:47.984537+0000 2021-02-10T04:25:49.262 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:49 smithi164 conmon[29288]: mon.a (mon.0) 570 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2021-02-10T04:25:49.262 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:49 smithi164 conmon[29288]: 2021-02-10T04:25:49.262 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:49 smithi164 conmon[29288]: cluster 2021-02-10T04:25:49.262 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:49 smithi164 conmon[29288]: 2021-02-10T04:25:47.984651+0000 mon.a (mon 2021-02-10T04:25:49.262 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:49 smithi164 conmon[29288]: .0) 571 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2021-02-10T04:25:49.262 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:49 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:25:49.263 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:49 smithi164 conmon[29288]: 02-10T04:25:47.988704+0000 mon.a 2021-02-10T04:25:49.263 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:49 smithi164 conmon[29288]: (mon.0) 572 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:25:49.263 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:49 smithi164 conmon[29288]: 2021-02-10T04:25:49.263 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:49 smithi164 conmon[29288]: audit 2021-02-10T04:25:49.263 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:49 smithi164 conmon[29288]: 2021-02-10T04:25:47.990623+0000 mon.c ( 2021-02-10T04:25:49.263 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:49 smithi164 conmon[29288]: mon.1) 12 : audit [INF] from='client.? 172.21.15.85:0/1810546809' entity='client.rgw.realm.zone.a' cmd=[{"prefix": "osd pool application enable","pool": "zone.rgw.log","app": "rgw"}]: dispatch 2021-02-10T04:25:49.263 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:49 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:25:49.263 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:49 smithi164 conmon[29288]: -10T04:25:47.991718+0000 mon.a ( 2021-02-10T04:25:49.263 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:49 smithi164 conmon[29288]: mon.0) 573 : audit [INF] from='client.? ' entity='client.rgw.realm.zone.a' cmd=[{"prefix": "osd pool application enable","pool": "zone.rgw.log","app": "rgw"}]: dispatch 2021-02-10T04:25:49.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:49 smithi085 conmon[29319]: cluster 2021-02-10T04:25:47.947379+0000 mgr.y (mgr.14140 2021-02-10T04:25:49.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:49 smithi085 conmon[29319]: ) 191 : cluster [DBG] pgmap v169: 33 pgs: 33 active+clean; 1.9 KiB data, 43 MiB used, 715 GiB / 715 GiB avail; 764 B/s rd, 1019 B/s wr, 1 op/s 2021-02-10T04:25:49.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:49 smithi085 conmon[29319]: audit 2021-02-10T04:25:47.984057+0000 mon.a (mon.0) 569 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:49.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:49 smithi085 conmon[29319]: audit 2021-02-10T04:25:47.984537+0000 mon.a (mon.0) 570 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2021-02-10T04:25:49.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:49 smithi085 conmon[29319]: cluster 2021-02-10T04:25:47.984651+0000 mon.a (mon.0) 571 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2021-02-10T04:25:49.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:49 smithi085 conmon[29319]: audit 2021-02-10T04:25:47.988704+0000 mon.a (mon.0) 572 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:25:49.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:49 smithi085 conmon[29319]: audit 2021-02-10T04:25:47.990623+0000 mon.c (mon.1) 12 : audit [INF] from='client.? 172.21.15.85:0/1810546809' entity='client.rgw.realm.zone.a' cmd=[{"prefix": "osd pool application enable","pool": "zone.rgw.log","app": "rgw"}]: dispatch 2021-02-10T04:25:49.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:49 smithi085 conmon[29319]: audit 2021-02-10T04:25:47.991718+0000 mon.a (mon.0) 573 : audit [INF] from='client.? ' entity='client.rgw.realm.zone.a' cmd=[{"prefix": "osd pool application enable","pool": "zone.rgw.log","app": "rgw"}]: dispatch 2021-02-10T04:25:49.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:49 smithi085 conmon[35961]: cluster 2021-02-10T04:25:49.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:49 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:25:49.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:49 smithi085 conmon[35961]: :25:47.947379+0000 mgr.y (mgr.14140) 191 : cluster [DBG] pgmap v169: 33 pgs: 33 active+clean; 1.9 KiB data, 43 MiB used, 715 GiB / 715 GiB avail; 764 B/s rd, 1019 B/s wr, 1 op/s 2021-02-10T04:25:49.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:49 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:25:49.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:49 smithi085 conmon[35961]: 04:25:47.984057+0000 mon.a (mon.0) 2021-02-10T04:25:49.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:49 smithi085 conmon[35961]: 569 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:49.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:49 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:25:49.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:49 smithi085 conmon[35961]: T04:25:47.984537+0000 mon.a (mon. 2021-02-10T04:25:49.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:49 smithi085 conmon[35961]: 0) 570 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2021-02-10T04:25:49.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:49 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:25:49.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:49 smithi085 conmon[35961]: 25:47.984651+0000 mon.a (mon.0) 571 : 2021-02-10T04:25:49.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:49 smithi085 conmon[35961]: cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2021-02-10T04:25:49.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:49 smithi085 conmon[35961]: audit 2021-02-10T04:25:47.988704 2021-02-10T04:25:49.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:49 smithi085 conmon[35961]: +0000 mon.a (mon.0) 572 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:25:49.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:49 smithi085 conmon[35961]: 2021-02-10T04:25:49.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:49 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:25:49.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:49 smithi085 conmon[35961]: T04:25:47.990623+0000 mon.c (mon.1) 12 : audit 2021-02-10T04:25:49.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:49 smithi085 conmon[35961]: [INF] from='client.? 172.21.15.85:0/1810546809' entity='client.rgw.realm.zone.a' cmd=[{"prefix": "osd pool application enable","pool": "zone.rgw.log","app": "rgw"}]: dispatch 2021-02-10T04:25:49.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:49 smithi085 conmon[35961]: audit 2021-02-10T04:25:49.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:49 smithi085 conmon[35961]: 2021-02-10T04:25:47.991718+0000 mon.a 2021-02-10T04:25:49.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:49 smithi085 conmon[35961]: (mon.0) 573 : audit [INF] from='client.? ' entity='client.rgw.realm.zone.a' cmd=[{"prefix": "osd pool application enable","pool": "zone.rgw.log","app": "rgw"}]: dispatch 2021-02-10T04:25:49.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:49 smithi085 conmon[35961]: 2021-02-10T04:25:50.127 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:50 smithi164 conmon[29288]: audit 2021-02-10T04:25:50.127 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:50 smithi164 conmon[29288]: 2021-02-10T04:25:50.127 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:50 smithi164 conmon[29288]: 2021 2021-02-10T04:25:50.127 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:50 smithi164 conmon[29288]: - 2021-02-10T04:25:50.127 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:50 smithi164 conmon[29288]: 02 2021-02-10T04:25:50.127 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:50 smithi164 conmon[29288]: -10T04:25: 2021-02-10T04:25:50.128 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:50 smithi164 conmon[29288]: 48.988758+0000 mon.a (mon.0) 574 : audit [INF] from='client.? ' entity='client.rgw.realm.zone.a' cmd='[{"prefix": "osd pool application enable","pool": "zone.rgw.log","app": "rgw"}]': finished 2021-02-10T04:25:50.128 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:50 smithi164 conmon[29288]: cluster 2021-02-10T04:25:50.128 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:50 smithi164 conmon[29288]: 2021-02-10T04:25:50.128 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:50 smithi164 conmon[29288]: 2021 2021-02-10T04:25:50.128 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:50 smithi164 conmon[29288]: -02-10T04:25:48.988831 2021-02-10T04:25:50.128 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:50 smithi164 conmon[29288]: +0000 mon.a (mon.0) 575 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2021-02-10T04:25:50.128 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:50 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:25:50.128 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:50 smithi164 conmon[29288]: 02-10T04:25:49.285503+0000 mon.a ( 2021-02-10T04:25:50.128 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:50 smithi164 conmon[29288]: mon.0) 576 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:50.128 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:50 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:25:50.128 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:50 smithi164 conmon[29288]: -10T04:25:49.289040+0000 mon.a (mon. 2021-02-10T04:25:50.128 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:50 smithi164 conmon[29288]: 0) 577 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:50.128 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:50 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:25:50.129 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:50 smithi164 conmon[29288]: 04:25:49.294097+0000 mon.a (mon. 2021-02-10T04:25:50.129 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:50 smithi164 conmon[29288]: 0) 578 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[35961]: audit 2021-02-10T04:25:48.988758+0000 mon.a 2021-02-10T04:25:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[35961]: (mon.0) 574 : audit [INF] from='client.? ' entity='client.rgw.realm.zone.a' cmd='[{"prefix": "osd pool application enable","pool": "zone.rgw.log","app": "rgw"}]': finished 2021-02-10T04:25:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[35961]: cluster 2021-02-10T04:25 2021-02-10T04:25:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[35961]: :48.988831+0000 mon.a (mon.0) 575 : cluster 2021-02-10T04:25:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[35961]: [DBG] osdmap e56: 8 total, 8 up, 8 in 2021-02-10T04:25:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[35961]: audit 2021-02-10T04:25: 2021-02-10T04:25:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[35961]: 49.285503+0000 mon.a (mon.0) 576 : 2021-02-10T04:25:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[35961]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:25:50.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[35961]: 25:49.289040+0000 mon.a (mon.0) 577 2021-02-10T04:25:50.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[35961]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:50.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:25:50.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[35961]: :25:49.294097+0000 mon.a (mon.0) 578 2021-02-10T04:25:50.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[35961]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:50.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: audit 2021-02-10T04:25:48.988758+0000 mon.a ( 2021-02-10T04:25:50.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: mon.0) 574 : audit [INF] from='client.? ' entity='client.rgw.realm.zone.a' cmd='[{"prefix": "osd pool application enable","pool": "zone.rgw.log","app": "rgw"}]': finished 2021-02-10T04:25:50.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: cluster 2021-02-10T04:25:48.988831+0000 2021-02-10T04:25:50.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: mon.a (mon.0) 575 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2021-02-10T04:25:50.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: audit 2021-02-10T04:25:50.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: 2021-02-10T04:25:50.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: 2021 2021-02-10T04:25:50.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: - 2021-02-10T04:25:50.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: 02 2021-02-10T04:25:50.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: - 2021-02-10T04:25:50.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: 10 2021-02-10T04:25:50.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: T 2021-02-10T04:25:50.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: 04 2021-02-10T04:25:50.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: : 2021-02-10T04:25:50.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: 25 2021-02-10T04:25:50.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: : 2021-02-10T04:25:50.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: 49 2021-02-10T04:25:50.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: . 2021-02-10T04:25:50.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: 285503 2021-02-10T04:25:50.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: +0000 2021-02-10T04:25:50.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: 2021-02-10T04:25:50.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: mon.a 2021-02-10T04:25:50.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: ( 2021-02-10T04:25:50.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: mon 2021-02-10T04:25:50.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: . 2021-02-10T04:25:50.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: 0 2021-02-10T04:25:50.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: ) 2021-02-10T04:25:50.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: 576 2021-02-10T04:25:50.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: : 2021-02-10T04:25:50.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: audit 2021-02-10T04:25:50.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: 2021-02-10T04:25:50.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: [INF] 2021-02-10T04:25:50.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:50.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:25:50.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: 10T04:25:49.289040+0000 mon.a (mon.0 2021-02-10T04:25:50.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: ) 577 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:50.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: audit 2021-02-10T04:25:50.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: 2021-02-10T04:25:50.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: 2021-02-10T04:25:49 2021-02-10T04:25:50.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:50 smithi085 conmon[29319]: .294097+0000 mon.a (mon.0) 578 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:51.034 INFO:teuthology.orchestra.run.smithi164.stderr:enabled application 'rbd' on pool 'iscsi' 2021-02-10T04:25:51.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: cluster 2021-02-10T04:25:51.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: 2021-02-10T04:25:51.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: 2021 2021-02-10T04:25:51.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: - 2021-02-10T04:25:51.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: 02 2021-02-10T04:25:51.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: - 2021-02-10T04:25:51.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: 10 2021-02-10T04:25:51.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: T 2021-02-10T04:25:51.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: 04:25:49.947946+0000 mgr.y (mgr.14140 2021-02-10T04:25:51.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: ) 192 : cluster [DBG] pgmap v172: 68 pgs: 8 creating+peering, 24 unknown, 36 active+clean; 1.9 KiB data, 43 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 511 B/s wr, 3 op/s 2021-02-10T04:25:51.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: cluster 2021-02-10T04:25:51.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: 2021-02-10T04:25:50.004113+0000 mon.a (mon.0) 579 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2021-02-10T04:25:51.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: audit 2021 2021-02-10T04:25:51.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: -02-10T04:25:50.006330+0000 mon.c (mon.1) 13 : audit [INF] from='client.? 172.21.15.85:0/1810546809' entity='client.rgw.realm.zone.a' cmd=[{"prefix": "osd pool application enable","pool": "zone.rgw.control","app": "rgw"}]: dispatch 2021-02-10T04:25:51.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: audit 2021-02-10T04:25: 2021-02-10T04:25:51.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: 50.015377+0000 mon.a (mon.0) 580 : audit 2021-02-10T04:25:51.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: [INF] from='client.? ' entity='client.rgw.realm.zone.a' cmd=[{"prefix": "osd pool application enable","pool": "zone.rgw.control","app": "rgw"}]: dispatch 2021-02-10T04:25:51.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:25:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: :25:50.417175+0000 mon.b (mon.2) 23 : 2021-02-10T04:25:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: audit [INF] from='client.? 172.21.15.164:0/825882793' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2021-02-10T04:25:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: audit 2021-02-10T04:25 2021-02-10T04:25:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: :50.417728+0000 mon.a (mon.0) 581 : audit 2021-02-10T04:25:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2021-02-10T04:25:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: audit 2021-02-10T04:25:51. 2021-02-10T04:25:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: 008982+0000 mon.a (mon.0) 582 : audit [INF] 2021-02-10T04:25:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: from='client.? ' entity='client.rgw.realm.zone.a' cmd='[{"prefix": "osd pool application enable","pool": "zone.rgw.control","app": "rgw"}]': finished 2021-02-10T04:25:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: audit 2021-02-10T04:25:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: 2021-02-10T04:25:51.009133+0000 mon.a (mon 2021-02-10T04:25:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: .0) 583 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]': finished 2021-02-10T04:25:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:25:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: 04:25:51.009215+0000 mon.a (mon.0) 2021-02-10T04:25:51.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:51 smithi164 conmon[29288]: 584 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2021-02-10T04:25:51.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[35961]: cluster 2021-02-10T04:25:49.947946 2021-02-10T04:25:51.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 192 : cluster [DBG] pgmap v172: 68 pgs: 8 creating+peering, 24 unknown, 36 active+clean; 1.9 KiB data, 43 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 511 B/s wr, 3 op/s 2021-02-10T04:25:51.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[35961]: cluster 2021-02-10T04:25:50.004113 2021-02-10T04:25:51.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[35961]: +0000 mon.a (mon.0) 579 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2021-02-10T04:25:51.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:25:51.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[35961]: T04:25:50.006330+0000 mon.c (mon.1) 13 : audit 2021-02-10T04:25:51.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[35961]: [INF] from='client.? 172.21.15.85:0/1810546809' entity='client.rgw.realm.zone.a' cmd=[{"prefix": "osd pool application enable","pool": "zone.rgw.control","app": "rgw"}]: dispatch 2021-02-10T04:25:51.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[35961]: audit 2021-02-10T04:25:50.015377+0000 mon.a ( 2021-02-10T04:25:51.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[35961]: mon.0) 580 : audit [INF] from='client.? ' entity='client.rgw.realm.zone.a' cmd=[{"prefix": "osd pool application enable","pool": "zone.rgw.control","app": "rgw"}]: dispatch 2021-02-10T04:25:51.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[35961]: audit 2021-02-10T04:25:50.417175+0000 2021-02-10T04:25:51.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[35961]: mon.b (mon.2) 23 : audit [INF] from='client.? 172.21.15.164:0/825882793' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2021-02-10T04:25:51.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:25:51.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[35961]: T04:25:50.417728+0000 mon.a ( 2021-02-10T04:25:51.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[35961]: mon.0) 581 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2021-02-10T04:25:51.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:25:51.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[35961]: :25:51. 2021-02-10T04:25:51.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[35961]: 008982+0000 mon.a (mon.0) 582 : 2021-02-10T04:25:51.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[35961]: audit [INF] from='client.? ' entity='client.rgw.realm.zone.a' cmd='[{"prefix": "osd pool application enable","pool": "zone.rgw.control","app": "rgw"}]': finished 2021-02-10T04:25:51.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:25:51.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[35961]: T04:25:51.009133+0000 mon.a (mon. 2021-02-10T04:25:51.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[35961]: 0) 583 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]': finished 2021-02-10T04:25:51.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:25:51.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[35961]: 10T04:25:51.009215+0000 mon.a 2021-02-10T04:25:51.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[35961]: (mon.0) 584 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2021-02-10T04:25:51.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: cluster 2021-02-10T04:25:49.947946+0000 mgr.y (mgr.14140) 192 : cluster 2021-02-10T04:25:51.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: [DBG] pgmap v172: 68 pgs: 8 creating+peering, 24 unknown, 36 active+clean; 1.9 KiB data, 43 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 511 B/s wr, 3 op/s 2021-02-10T04:25:51.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: cluster 2021-02-10T04:25:50.004113+0000 mon.a (mon.0) 579 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2021-02-10T04:25:51.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:25:51.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: -10T04:25:50.006330+0000 mon.c (mon.1) 13 : audit [INF] from='client.? 172.21.15.85:0/1810546809' entity='client.rgw.realm.zone.a' cmd=[{"prefix": "osd pool application enable","pool": "zone.rgw.control","app": "rgw"}]: dispatch 2021-02-10T04:25:51.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: audit 2021-02-10T04:25:51.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: 2021-02-10T04:25:51.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: 2021 2021-02-10T04:25:51.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: - 2021-02-10T04:25:51.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: 02 2021-02-10T04:25:51.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: - 2021-02-10T04:25:51.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: 10 2021-02-10T04:25:51.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: T 2021-02-10T04:25:51.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: 04 2021-02-10T04:25:51.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: : 2021-02-10T04:25:51.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: 25 2021-02-10T04:25:51.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: : 2021-02-10T04:25:51.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: 50 2021-02-10T04:25:51.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: . 2021-02-10T04:25:51.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: 015377 2021-02-10T04:25:51.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: +0000 2021-02-10T04:25:51.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: 2021-02-10T04:25:51.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: mon.a 2021-02-10T04:25:51.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: ( 2021-02-10T04:25:51.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: mon 2021-02-10T04:25:51.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: . 2021-02-10T04:25:51.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: 0 2021-02-10T04:25:51.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: ) 2021-02-10T04:25:51.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: 580 2021-02-10T04:25:51.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: : 2021-02-10T04:25:51.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: audit 2021-02-10T04:25:51.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: 2021-02-10T04:25:51.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: [INF] 2021-02-10T04:25:51.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: 2021-02-10T04:25:51.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: from='client.? ' entity='client.rgw.realm.zone.a' cmd=[{"prefix": "osd pool application enable","pool": "zone.rgw.control","app": "rgw"}]: dispatch 2021-02-10T04:25:51.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: 2021-02-10T04:25:51.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: audit 2021-02-10T04:25:51.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: 2021-02-10T04:25:51.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: 2021 2021-02-10T04:25:51.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: - 2021-02-10T04:25:51.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: 02 2021-02-10T04:25:51.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: - 2021-02-10T04:25:51.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: 10 2021-02-10T04:25:51.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: T 2021-02-10T04:25:51.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: 04 2021-02-10T04:25:51.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: : 2021-02-10T04:25:51.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: 25 2021-02-10T04:25:51.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: : 2021-02-10T04:25:51.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: 50 2021-02-10T04:25:51.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: .417175+0000 mon.b (mon.2) 23 : audit [INF] 2021-02-10T04:25:51.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: from='client.? 172.21.15.164:0/825882793' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2021-02-10T04:25:51.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: audit 2021 2021-02-10T04:25:51.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: -02-10T04:25:50.417728+0000 mon.a (mon.0) 2021-02-10T04:25:51.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: 581 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2021-02-10T04:25:51.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: audit 2021-02-10T04:25:51.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:25:51.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: 25:51.008982+0000 mon.a (mon.0) 582 : audit [INF] from='client.? ' entity='client.rgw.realm.zone.a' cmd='[{"prefix": "osd pool application enable","pool": "zone.rgw.control","app": "rgw"}]': finished 2021-02-10T04:25:51.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: audit 2021-02-10T04:25:51.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:25:51.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: 25:51.009133+0000 mon.a (mon.0) 583 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]': finished 2021-02-10T04:25:51.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:25:51.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: -02-10T04:25:51.009215+0000 2021-02-10T04:25:51.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: mon.a (mon.0) 584 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2021-02-10T04:25:51.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:51 smithi085 conmon[29319]: 2021-02-10T04:25:51.776 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph orch apply iscsi iscsi user password --placement '1;smithi164=iscsi.a' 2021-02-10T04:25:53.362 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:53 smithi164 conmon[29288]: cluster 2021-02-10T04:25:51.948416+0000 mgr.y (mgr.14140) 193 : cluster [DBG] pgmap v175: 100 pgs: 20 creating+peering, 16 unknown, 64 active+clean; 7.2 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 8.7 KiB/s wr, 39 op/s 2021-02-10T04:25:53.363 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:53 smithi164 conmon[29288]: cluster 2021-02-10T04:25:52.023383+0000 mon.a (mon.0) 585 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2021-02-10T04:25:53.363 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:53 smithi164 conmon[29288]: audit 2021-02-10T04:25:52.029014+0000 mon.c (mon.1) 14 : audit [INF] from='client.? 172.21.15.85:0/3915233377' entity='client.rgw.realm.zone.a' cmd=[{"prefix": "osd pool application enable","pool": "zone.rgw.meta","app": "rgw"}]: dispatch 2021-02-10T04:25:53.363 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:53 smithi164 conmon[29288]: audit 2021-02-10T04:25:52.030247+0000 mon.a (mon.0) 586 : audit [INF] from='client.? ' entity='client.rgw.realm.zone.a' cmd=[{"prefix": "osd pool application enable","pool": "zone.rgw.meta","app": "rgw"}]: dispatch 2021-02-10T04:25:53.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: cluster 2021-02-10T04:25:53.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: 2021-02-10T04:25:53.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: 2021 2021-02-10T04:25:53.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: - 2021-02-10T04:25:53.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: 02 2021-02-10T04:25:53.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: - 2021-02-10T04:25:53.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: 10 2021-02-10T04:25:53.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: T 2021-02-10T04:25:53.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: 04 2021-02-10T04:25:53.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: : 2021-02-10T04:25:53.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: 25 2021-02-10T04:25:53.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: : 2021-02-10T04:25:53.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: 51 2021-02-10T04:25:53.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: . 2021-02-10T04:25:53.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: 948416 2021-02-10T04:25:53.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: +0000 2021-02-10T04:25:53.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: 2021-02-10T04:25:53.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: mgr.y 2021-02-10T04:25:53.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: ( 2021-02-10T04:25:53.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: mgr 2021-02-10T04:25:53.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: . 2021-02-10T04:25:53.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: 14140 2021-02-10T04:25:53.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: ) 2021-02-10T04:25:53.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: 193 2021-02-10T04:25:53.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: : 2021-02-10T04:25:53.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: cluster 2021-02-10T04:25:53.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: 2021-02-10T04:25:53.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: [DBG] 2021-02-10T04:25:53.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: 2021-02-10T04:25:53.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: pgmap v175: 100 pgs: 20 creating+peering, 16 unknown, 64 active+clean; 7.2 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 8.7 KiB/s wr, 39 op/s 2021-02-10T04:25:53.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: 2021-02-10T04:25:53.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: cluster 2021-02-10T04:25:53.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: 2021-02-10T04:25:53.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: 2021 2021-02-10T04:25:53.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: - 2021-02-10T04:25:53.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: 02 2021-02-10T04:25:53.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: - 2021-02-10T04:25:53.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: 10 2021-02-10T04:25:53.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: T 2021-02-10T04:25:53.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: 04 2021-02-10T04:25:53.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: :25:52.023383+0000 mon.a (mon.0) 585 : cluster [DBG] 2021-02-10T04:25:53.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: osdmap e59: 8 total, 8 up, 8 in 2021-02-10T04:25:53.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: audit 2021-02-10T04:25: 2021-02-10T04:25:53.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: 52.029014+0000 mon.c (mon.1) 14 : audit [INF] from='client.? 172.21.15.85:0/3915233377' entity='client.rgw.realm.zone.a' cmd=[{"prefix": "osd pool application enable","pool": "zone.rgw.meta","app": "rgw"}]: dispatch 2021-02-10T04:25:53.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:25:53.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: T04:25:52.030247+0000 mon.a (mon. 2021-02-10T04:25:53.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[29319]: 0) 586 : audit [INF] from='client.? ' entity='client.rgw.realm.zone.a' cmd=[{"prefix": "osd pool application enable","pool": "zone.rgw.meta","app": "rgw"}]: dispatch 2021-02-10T04:25:53.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: cluster 2021-02-10T04:25:53.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: 2021-02-10T04:25:53.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: 2021 2021-02-10T04:25:53.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: - 2021-02-10T04:25:53.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: 02 2021-02-10T04:25:53.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: - 2021-02-10T04:25:53.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: 10 2021-02-10T04:25:53.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: T 2021-02-10T04:25:53.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: 04 2021-02-10T04:25:53.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: : 2021-02-10T04:25:53.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: 25 2021-02-10T04:25:53.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: : 2021-02-10T04:25:53.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: 51 2021-02-10T04:25:53.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: . 2021-02-10T04:25:53.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: 948416 2021-02-10T04:25:53.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: +0000 2021-02-10T04:25:53.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: 2021-02-10T04:25:53.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: mgr.y 2021-02-10T04:25:53.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: ( 2021-02-10T04:25:53.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: mgr 2021-02-10T04:25:53.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: . 2021-02-10T04:25:53.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: 14140 2021-02-10T04:25:53.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: ) 2021-02-10T04:25:53.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: 193 2021-02-10T04:25:53.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: : 2021-02-10T04:25:53.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: cluster 2021-02-10T04:25:53.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: 2021-02-10T04:25:53.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: [DBG] 2021-02-10T04:25:53.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: 2021-02-10T04:25:53.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: pgmap v175: 100 pgs: 20 creating+peering, 16 unknown, 64 active+clean; 7.2 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 8.7 KiB/s wr, 39 op/s 2021-02-10T04:25:53.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: 2021-02-10T04:25:53.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: cluster 2021-02-10T04:25:53.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: 2021-02-10T04:25:53.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: 2021 2021-02-10T04:25:53.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: - 2021-02-10T04:25:53.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: 02 2021-02-10T04:25:53.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: - 2021-02-10T04:25:53.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: 10 2021-02-10T04:25:53.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: T 2021-02-10T04:25:53.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: 04 2021-02-10T04:25:53.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: :25:52.023383+0000 mon.a (mon. 2021-02-10T04:25:53.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: 0) 585 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2021-02-10T04:25:53.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:25:53.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: :25:52.029014+0000 mon.c ( 2021-02-10T04:25:53.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: mon.1) 14 : audit 2021-02-10T04:25:53.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: [INF] from='client.? 172.21.15.85:0/3915233377' entity='client.rgw.realm.zone.a' cmd=[{"prefix": "osd pool application enable","pool": "zone.rgw.meta","app": "rgw"}]: dispatch 2021-02-10T04:25:53.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: audit 2021 2021-02-10T04:25:53.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: -02-10T04:25: 2021-02-10T04:25:53.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: 52.030247+0000 mon.a (mon.0) 586 : audit 2021-02-10T04:25:53.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:53 smithi085 conmon[35961]: [INF] from='client.? ' entity='client.rgw.realm.zone.a' cmd=[{"prefix": "osd pool application enable","pool": "zone.rgw.meta","app": "rgw"}]: dispatch 2021-02-10T04:25:53.740 INFO:teuthology.orchestra.run.smithi164.stdout:Scheduled iscsi.iscsi update... 2021-02-10T04:25:54.328 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:54 smithi164 conmon[29288]: audit 2021-02-10T04:25:53.025184+0000 mon.a (mon.0) 587 : audit [INF] 2021-02-10T04:25:54.329 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:54 smithi164 conmon[29288]: from='client.? ' entity='client.rgw.realm.zone.a' cmd='[{"prefix": "osd pool application enable","pool": "zone.rgw.meta","app": "rgw"}]': finished 2021-02-10T04:25:54.329 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:54 smithi164 conmon[29288]: cluster 2021-02-10T04:25:53.025283+0000 mon.a (mon.0) 588 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2021-02-10T04:25:54.331 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:54 smithi164 conmon[29288]: audit 2021-02-10T04:25:53.043995+0000 mon.c (mon.1) 15 : audit [INF] from='client.? 172.21.15.85:0/3915233377' entity='client.rgw.realm.zone.a' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2021-02-10T04:25:54.331 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:54 smithi164 conmon[29288]: audit 2021-02-10T04:25:53.044825+0000 mon.a (mon.0) 589 : audit [INF] from='client.? ' entity='client.rgw.realm.zone.a' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2021-02-10T04:25:54.332 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:54 smithi164 conmon[29288]: audit 2021-02-10T04:25:53.739518+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:54.332 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:54 smithi164 conmon[29288]: audit 2021-02-10T04:25:53.744281+0000 mon.a (mon.0) 591 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:54.333 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:54 smithi164 conmon[29288]: audit 2021-02-10T04:25:53.755776+0000 mon.a (mon.0) 592 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:54.333 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:54 smithi164 conmon[29288]: audit 2021-02-10T04:25:53.756724+0000 mon.a (mon.0) 593 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2021-02-10T04:25:54.334 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:54 smithi164 conmon[29288]: audit 2021-02-10T04:25:53.760467+0000 mon.a (mon.0) 594 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2021-02-10T04:25:54.336 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:54 smithi164 conmon[29288]: audit 2021-02-10T04:25:53.767692+0000 mon.a (mon.0) 595 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:25:54.336 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:54 smithi164 conmon[29288]: audit 2021-02-10T04:25:53.768819+0000 mon.a (mon.0) 596 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "client.iscsi.iscsi.a", "key": "container_image"}]: dispatch 2021-02-10T04:25:54.357 DEBUG:teuthology.orchestra.run.smithi164:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@iscsi.iscsi.a.service 2021-02-10T04:25:54.827 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: audit 2021-02-10T04:25:53.025184 2021-02-10T04:25:54.827 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: +0000 mon.a (mon.0) 587 : audit [INF] from='client.? ' entity='client.rgw.realm.zone.a' cmd='[{"prefix": "osd pool application enable","pool": "zone.rgw.meta","app": "rgw"}]': finished 2021-02-10T04:25:54.827 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: cluster 2021-02-10T04:25: 2021-02-10T04:25:54.827 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: 53.025283+0000 mon.a (mon.0) 588 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2021-02-10T04:25:54.827 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:25:54.827 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: 10T04:25:53.043995+0000 mon.c (mon.1) 2021-02-10T04:25:54.827 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: 15 : audit [INF] from='client.? 172.21.15.85:0/3915233377' entity='client.rgw.realm.zone.a' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2021-02-10T04:25:54.827 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:25:54.827 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: T04:25:53.044825+0000 mon.a (mon 2021-02-10T04:25:54.827 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: .0) 589 : audit [INF] from='client.? ' entity='client.rgw.realm.zone.a' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2021-02-10T04:25:54.827 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:25:54.828 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: -10T04:25:53.739518+0000 mon.a 2021-02-10T04:25:54.828 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: (mon.0) 590 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:54.828 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:25:54.828 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: 02-10T04:25:53.744281+0000 2021-02-10T04:25:54.828 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: mon.a (mon.0) 591 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:54.828 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: 2021-02-10T04:25:54.828 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: audit 2021-02-10T04:25: 2021-02-10T04:25:54.828 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: 53.755776+0000 mon.a (mon.0) 592 : 2021-02-10T04:25:54.828 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:54.828 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: audit 2021-02-10T 2021-02-10T04:25:54.828 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: 04:25:53.756724+0000 mon.a (mon.0 2021-02-10T04:25:54.828 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: ) 593 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2021-02-10T04:25:54.828 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: audit 2021-02-10T 2021-02-10T04:25:54.828 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: 04:25:53.760467+0000 mon.a (mon. 2021-02-10T04:25:54.829 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: 0) 594 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2021-02-10T04:25:54.829 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:25:54.829 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: 02-10T04:25:53 2021-02-10T04:25:54.829 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: .767692+0000 mon.a (mon.0 2021-02-10T04:25:54.829 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: ) 595 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:25:54.829 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: audit 2021 2021-02-10T04:25:54.829 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: -02-10T04:25:53.768819 2021-02-10T04:25:54.829 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: +0000 mon.a (mon.0) 596 : audit 2021-02-10T04:25:54.829 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[29319]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "client.iscsi.iscsi.a", "key": "container_image"}]: dispatch 2021-02-10T04:25:54.829 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: audit 2021-02-10T04:25:53.025184+0000 mon.a (mon 2021-02-10T04:25:54.829 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: .0) 587 : audit [INF] from='client.? ' entity='client.rgw.realm.zone.a' cmd='[{"prefix": "osd pool application enable","pool": "zone.rgw.meta","app": "rgw"}]': finished 2021-02-10T04:25:54.829 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:25:54.830 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: 25:53.025283+0000 mon.a (mon.0) 588 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2021-02-10T04:25:54.830 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: audit 2021-02-10T04:25:54.830 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: 2021-02-10T04:25:53 2021-02-10T04:25:54.830 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: .043995+0000 mon.c (mon.1) 15 : audit [INF] 2021-02-10T04:25:54.830 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/3915233377' entity='client.rgw.realm.zone.a' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2021-02-10T04:25:54.830 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:25:54.830 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: 04:25:53.044825+0000 mon.a (mon.0) 2021-02-10T04:25:54.830 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: 589 : audit [INF] from='client.? ' entity='client.rgw.realm.zone.a' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2021-02-10T04:25:54.830 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: audit 2021-02-10T04:25:54.830 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: 2021-02-10T04:25:53.739518+0000 2021-02-10T04:25:54.830 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: mon.a (mon.0) 590 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:54.830 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:25:54.830 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: -10T04:25:53.744281+0000 mon.a (mon.0 2021-02-10T04:25:54.831 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: ) 591 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:54.831 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: audit 2021 2021-02-10T04:25:54.831 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: -02-10T04:25:53.755776+0000 2021-02-10T04:25:54.831 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: mon.a (mon.0) 592 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:54.831 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:25:54.831 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: 10T04:25:53.756724+0000 mon.a (mon.0) 593 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2021-02-10T04:25:54.831 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:25:54.831 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: 10T04:25:53.760467+0000 mon.a (mon.0) 594 : audit 2021-02-10T04:25:54.833 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2021-02-10T04:25:54.833 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: audit 2021-02-10T04:25:53.767692+0000 2021-02-10T04:25:54.833 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: mon.a (mon.0) 595 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-02-10T04:25:54.833 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: audit 2021 2021-02-10T04:25:54.833 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: -02-10T04:25: 2021-02-10T04:25:54.833 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: 53.768819+0000 mon.a (mon 2021-02-10T04:25:54.834 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: .0) 596 : audit [DBG] 2021-02-10T04:25:54.834 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:54 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "client.iscsi.iscsi.a", "key": "container_image"}]: dispatch 2021-02-10T04:25:54.836 INFO:tasks.cephadm:Adding prometheus.a on smithi164 2021-02-10T04:25:54.836 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph orch apply prometheus '1;smithi164=a' 2021-02-10T04:25:54.864 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:-- Logs begin at Wed 2021-02-10 04:09:33 UTC. -- 2021-02-10T04:25:55.052 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:55 smithi164 conmon[29288]: audit 2021-02-10T04:25:53.734693+0000 mgr.y 2021-02-10T04:25:55.052 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:55 smithi164 conmon[29288]: (mgr.14140) 194 : audit [DBG] from='client.34394 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "iscsi", "api_user": "user", "api_password": "password", "placement": "1;smithi164=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:25:55.052 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:55 smithi164 conmon[29288]: cephadm 2021-02-10T04:25:53.735916+0000 mgr.y (mgr.14140) 195 : cephadm [INF] Saving service iscsi.iscsi spec with placement smithi164=iscsi.a;count:1 2021-02-10T04:25:55.052 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:55 smithi164 conmon[29288]: cephadm 2021-02-10T04:25:53.745857+0000 mgr.y (mgr.14140) 196 : cephadm [INF] Checking pool "iscsi" exists for service iscsi.iscsi 2021-02-10T04:25:55.053 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:55 smithi164 conmon[29288]: cephadm 2021-02-10T04:25:53.752296+0000 mgr.y (mgr.14140) 197 : cephadm [INF] Saving service iscsi.iscsi spec with placement smithi164=iscsi.a;count:1 2021-02-10T04:25:55.053 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:55 smithi164 conmon[29288]: cephadm 2021-02-10T04:25:53.768445+0000 mgr.y (mgr.14140) 198 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi164 2021-02-10T04:25:55.053 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:55 smithi164 conmon[29288]: cluster 2021-02-10T04:25:55.053 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:55 smithi164 conmon[29288]: 2021-02-10T04:25:53.949044+0000 mgr.y (mgr.14140) 199 : cluster [DBG] pgmap v178: 132 pgs: 1 creating+activating, 15 creating+peering, 32 unknown, 84 active+clean; 11 KiB data, 45 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 15 KiB/s wr, 62 op/s 2021-02-10T04:25:55.053 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:55 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:25:55.053 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:55 smithi164 conmon[29288]: 10T04:25:54.040862+0000 mon.a (mon.0) 597 : audit 2021-02-10T04:25:55.053 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:55 smithi164 conmon[29288]: [INF] from='client.? ' entity='client.rgw.realm.zone.a' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2021-02-10T04:25:55.053 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:55 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:25:55.053 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:55 smithi164 conmon[29288]: :25:54.040961+0000 mon.a (mon.0 2021-02-10T04:25:55.053 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:55 smithi164 conmon[29288]: ) 598 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2021-02-10T04:25:55.053 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:55 smithi164 conmon[29288]: audit 2021 2021-02-10T04:25:55.054 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:55 smithi164 conmon[29288]: -02-10T04:25:54.042345+0000 2021-02-10T04:25:55.054 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:55 smithi164 conmon[29288]: mon.c (mon.1) 16 : audit [INF] 2021-02-10T04:25:55.054 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:55 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/3915233377' entity='client.rgw.realm.zone.a' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2021-02-10T04:25:55.054 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:55 smithi164 conmon[29288]: audit 2021-02-10T04: 2021-02-10T04:25:55.054 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:55 smithi164 conmon[29288]: 25:54.042752+0000 mon.a (mon.0 2021-02-10T04:25:55.054 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:55 smithi164 conmon[29288]: ) 599 : audit [INF] from='client.? ' entity='client.rgw.realm.zone.a' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2021-02-10T04:25:55.440 INFO:journalctl@ceph.rgw.realm.zone.a.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[54650]: debug 2021-02-10T04:25:55.440 INFO:journalctl@ceph.rgw.realm.zone.a.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[54650]: 2021-02-10T04:25:55.131+0000 7ffbac21c2c0 -1 rgw period pusher: The new period does not contain my zonegroup! 2021-02-10T04:25:55.441 INFO:journalctl@ceph.rgw.realm.zone.a.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[54650]: 2021-02-10T04:25:55.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[29319]: audit 2021-02-10T04:25:53.734693+0000 mgr.y (mgr.14140) 194 : audit [DBG] from='client.34394 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "iscsi", "api_user": "user", "api_password": "password", "placement": "1;smithi164=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:25:55.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[29319]: cephadm 2021-02-10T04:25:53.735916+0000 mgr.y (mgr.14140) 195 : cephadm [INF] Saving service iscsi.iscsi spec with placement smithi164=iscsi.a;count:1 2021-02-10T04:25:55.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[29319]: cephadm 2021-02-10T04:25:53.745857+0000 mgr.y (mgr.14140) 196 : cephadm 2021-02-10T04:25:55.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[29319]: [INF] Checking pool "iscsi" exists for service iscsi.iscsi 2021-02-10T04:25:55.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[29319]: cephadm 2021-02-10T04:25:53.752296+0000 mgr.y (mgr.14140) 197 : 2021-02-10T04:25:55.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[29319]: cephadm [INF] Saving service iscsi.iscsi spec with placement smithi164=iscsi.a;count:1 2021-02-10T04:25:55.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[29319]: cephadm 2021-02-10T04:25:53.768445+0000 mgr.y (mgr.14140 2021-02-10T04:25:55.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[29319]: ) 198 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi164 2021-02-10T04:25:55.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:25:55.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[29319]: T04:25:53.949044+0000 mgr.y (mgr.14140) 199 : cluster [DBG] pgmap v178: 132 pgs: 1 creating+activating, 15 creating+peering, 32 unknown, 84 active+clean; 11 KiB data, 45 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 15 KiB/s wr, 62 op/s 2021-02-10T04:25:55.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[29319]: audit 2021-02-10T04:25:54.040862 2021-02-10T04:25:55.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[29319]: +0000 mon.a (mon.0) 597 : audit [INF] from='client.? ' entity='client.rgw.realm.zone.a' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2021-02-10T04:25:55.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[29319]: cluster 2021-02-10T04:25: 2021-02-10T04:25:55.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[29319]: 54.040961+0000 mon.a (mon.0) 598 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2021-02-10T04:25:55.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:25:55.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[29319]: T04:25:54.042345+0000 mon.c (mon.1) 16 : audit [INF] 2021-02-10T04:25:55.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[29319]: from='client.? 172.21.15.85:0/3915233377' entity='client.rgw.realm.zone.a' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2021-02-10T04:25:55.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[29319]: audit 2021-02-10T04:25:55.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[29319]: 2021-02-10T04:25:54.042752+0000 mon.a (mon.0) 599 : audit [INF] from='client.? ' entity='client.rgw.realm.zone.a' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2021-02-10T04:25:55.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: audit 2021-02-10T04:25:53.734693 2021-02-10T04:25:55.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 194 : audit [DBG] from='client.34394 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "iscsi", "api_user": "user", "api_password": "password", "placement": "1;smithi164=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:25:55.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: cephadm 2021-02-10T04:25:55.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: 2021-02-10T04:25: 2021-02-10T04:25:55.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: 53.735916+0000 mgr.y (mgr.14140 2021-02-10T04:25:55.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: ) 195 : cephadm [INF] Saving service iscsi.iscsi spec with placement smithi164=iscsi.a;count:1 2021-02-10T04:25:55.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: cephadm 2021-02-10T04:25:55.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: 2021-02-10T04:25:53 2021-02-10T04:25:55.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: .745857+0000 mgr.y (mgr. 2021-02-10T04:25:55.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: 14140) 196 : cephadm [INF] Checking pool "iscsi" exists for service iscsi.iscsi 2021-02-10T04:25:55.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: 2021-02-10T04:25:55.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: cephadm 2021 2021-02-10T04:25:55.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: -02-10T04:25:53. 2021-02-10T04:25:55.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: 752296+0000 mgr.y (mgr.14140) 2021-02-10T04:25:55.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: 197 : cephadm [INF] Saving service iscsi.iscsi spec with placement smithi164=iscsi.a;count:1 2021-02-10T04:25:55.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: cephadm 2021-02-10T04:25:55.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: 2021-02-10T04:25:55.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: 2021-02-10T04:25: 2021-02-10T04:25:55.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: 53.768445+0000 mgr.y (mgr.14140) 198 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi164 2021-02-10T04:25:55.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: cluster 2021-02-10T04:25:55.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:25:55.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: T04:25:53.949044+0000 mgr.y (mgr.14140) 2021-02-10T04:25:55.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: 199 : cluster [DBG] pgmap v178: 132 pgs: 1 creating+activating, 15 creating+peering, 32 unknown, 84 active+clean; 11 KiB data, 45 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 15 KiB/s wr, 62 op/s 2021-02-10T04:25:55.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: audit 2021-02-10T04:25:55.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:25:55.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: 04:25:54.040862+0000 mon.a (mon.0) 597 : audit 2021-02-10T04:25:55.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: [INF] from='client.? ' entity='client.rgw.realm.zone.a' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2021-02-10T04:25:55.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: cluster 2021-02-10T04:25:55.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:25:55.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: T04:25:54.040961+0000 mon.a (mon.0 2021-02-10T04:25:55.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: ) 598 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2021-02-10T04:25:55.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: 2021-02-10T04:25:55.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: audit 2021-02-10T04:25:55.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: 2021-02-10T04:25:55.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: 2021-02-10T04:25:54. 2021-02-10T04:25:55.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: 042345+0000 mon.c (mon.1) 16 : audit [INF] from='client.? 172.21.15.85:0/3915233377' entity='client.rgw.realm.zone.a' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2021-02-10T04:25:55.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: audit 2021-02-10T04:25:54.042752+0000 2021-02-10T04:25:55.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:55 smithi085 conmon[35961]: mon.a (mon.0) 599 : audit [INF] from='client.? ' entity='client.rgw.realm.zone.a' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2021-02-10T04:25:56.181 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:56 smithi164 conmon[29288]: audit 2021-02-10T04:25:55.042728+0000 mon.a (mon.0) 600 : audit [INF] from='client.? ' entity='client.rgw.realm.zone.a' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2021-02-10T04:25:56.181 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:56 smithi164 conmon[29288]: cluster 2021-02-10T04:25:55.042844+0000 mon.a (mon.0) 601 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2021-02-10T04:25:56.181 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:56 smithi164 conmon[29288]: cluster 2021-02-10T04:25:55.766677+0000 mon.a (mon.0) 602 : cluster [DBG] mgrmap e15: y(active, since 4m), standbys: x 2021-02-10T04:25:56.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:56 smithi085 conmon[29319]: audit 2021-02-10T04:25:55.042728+0000 mon.a (mon.0) 600 : audit [INF] from='client.? ' entity='client.rgw.realm.zone.a' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2021-02-10T04:25:56.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:56 smithi085 conmon[29319]: cluster 2021-02-10T04:25:55.042844+0000 mon.a (mon.0) 601 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2021-02-10T04:25:56.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:56 smithi085 conmon[29319]: cluster 2021-02-10T04:25:55.766677+0000 mon.a (mon.0) 602 : cluster [DBG] mgrmap e15: y(active, since 4m), standbys: x 2021-02-10T04:25:56.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:56 smithi085 conmon[35961]: audit 2021-02-10T04:25:55 2021-02-10T04:25:56.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:56 smithi085 conmon[35961]: .042728+0000 mon.a (mon.0) 600 : audit [INF] from='client.? ' entity='client.rgw.realm.zone.a' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2021-02-10T04:25:56.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:56 smithi085 conmon[35961]: cluster 2021-02-10T04:25:56.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:56 smithi085 conmon[35961]: 2021-02-10T04:25: 2021-02-10T04:25:56.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:56 smithi085 conmon[35961]: 55.042844+0000 mon.a (mon 2021-02-10T04:25:56.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:56 smithi085 conmon[35961]: .0) 601 : cluster [DBG] 2021-02-10T04:25:56.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:56 smithi085 conmon[35961]: osdmap e62: 8 total, 8 up, 8 in 2021-02-10T04:25:56.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:56 smithi085 conmon[35961]: cluster 2021-02-10T04:25:55 2021-02-10T04:25:56.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:56 smithi085 conmon[35961]: .766677+0000 mon.a (mon.0) 602 : cluster [DBG] mgrmap e15: y(active, since 4m), standbys: x 2021-02-10T04:25:57.051 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:25:56 smithi164 bash[48232]: log file path now is '/var/log/tcmu-runner.log' 2021-02-10T04:25:57.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:57 smithi164 conmon[29288]: cluster 2021-02-10T04:25:55.949539+0000 mgr.y (mgr.14140) 200 : 2021-02-10T04:25:57.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:57 smithi164 conmon[29288]: cluster [DBG] pgmap v181: 132 pgs: 1 creating+activating, 8 creating+peering, 7 unknown, 116 active+clean; 21 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 23 KiB/s wr, 92 op/s 2021-02-10T04:25:57.393 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:25:57 smithi164 bash[48232]: 2ce80b592708bdba5a7095821258bc41f772ede7b045dbb91391cce6763cd122 2021-02-10T04:25:57.393 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:25:57 smithi164 systemd[1]: Started Ceph iscsi.iscsi.a for 42224fce-6b57-11eb-8feb-001a4aab830c. 2021-02-10T04:25:57.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:57 smithi085 conmon[35961]: cluster 2021-02-10T04:25:55.949539+0000 mgr.y (mgr.14140) 200 2021-02-10T04:25:57.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:57 smithi085 conmon[35961]: : cluster [DBG] pgmap v181: 132 pgs: 1 creating+activating, 8 creating+peering, 7 unknown, 116 active+clean; 21 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 23 KiB/s wr, 92 op/s 2021-02-10T04:25:57.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:57 smithi085 conmon[29319]: cluster 2021-02-10T04:25:55.949539+0000 mgr.y (mgr.14140) 200 : cluster [DBG] pgmap v181: 132 pgs: 1 creating+activating, 8 creating+peering, 7 unknown, 116 active+clean; 21 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 23 KiB/s wr, 92 op/s 2021-02-10T04:25:57.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:57 smithi085 conmon[29319]: 2021-02-10T04:25:57.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:25:57 smithi164 conmon[48560]: debug Started the configuration object watcher 2021-02-10T04:25:57.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:25:57 smithi164 conmon[48560]: debug Checking for config object changes every 1s 2021-02-10T04:25:57.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:25:57 smithi164 conmon[48560]: debug Processing osd blocklist entries for this node 2021-02-10T04:25:57.980 INFO:teuthology.orchestra.run.smithi164.stdout:Scheduled prometheus update... 2021-02-10T04:25:58.091 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:25:57 smithi164 conmon[48560]: debug Reading the configuration object to update local LIO configuration 2021-02-10T04:25:58.092 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:25:57 smithi164 conmon[48560]: debug Configuration does not have an entry for this host(smithi164) - nothing to define to LIO 2021-02-10T04:25:58.092 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:25:57 smithi164 conmon[48560]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2021-02-10T04:25:58.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:58 smithi164 conmon[29288]: audit 2021-02-10T04:25:57.317235+0000 2021-02-10T04:25:58.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:58 smithi164 conmon[29288]: mon.a (mon.0) 603 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:58.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:58 smithi164 conmon[29288]: audit 2021-02-10T04:25:57.318750+0000 mon.a (mon.0) 604 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:25:58.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:58 smithi164 conmon[29288]: cluster 2021-02-10T04:25:57.335366+0000 osd.5 (osd.5) 3 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:25:58.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:58 smithi164 conmon[29288]: cluster 2021-02-10T04:25:57.335460+0000 osd.5 (osd.5) 4 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:25:58.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:58 smithi164 conmon[29288]: audit 2021-02-10T04:25:57.805824+0000 mon.b (mon.2) 24 : audit [DBG] from='client.? 172.21.15.164:0/1786624706' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2021-02-10T04:25:58.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:58 smithi164 conmon[29288]: cluster 2021-02-10T04:25:57.950130+0000 mgr.y (mgr.14140) 201 : cluster [DBG] pgmap v182: 132 pgs: 1 creating+activating, 3 creating+peering, 128 active+clean; 22 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 17 KiB/s wr, 134 op/s 2021-02-10T04:25:58.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:58 smithi164 conmon[29288]: audit 2021-02-10T04:25:57.970803 2021-02-10T04:25:58.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:58 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 202 : audit [DBG] from='client.34433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi164=a", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:25:58.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:58 smithi164 conmon[29288]: cephadm 2021-02-10T04:25:57.971902+0000 mgr.y (mgr 2021-02-10T04:25:58.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:58 smithi164 conmon[29288]: .14140) 203 : cephadm [INF] Saving service prometheus spec with placement smithi164=a;count:1 2021-02-10T04:25:58.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:58 smithi164 conmon[29288]: audit 2021-02-10T04:25 2021-02-10T04:25:58.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:58 smithi164 conmon[29288]: :57.979878+0000 mon.a (mon.0) 605 2021-02-10T04:25:58.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:58 smithi164 conmon[29288]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[29319]: audit 2021-02-10T04:25:57.317235+0000 mon.a (mon.0) 603 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[29319]: audit 2021-02-10T04:25:57.318750+0000 mon.a (mon.0) 604 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:25:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[29319]: cluster 2021-02-10T04:25:57.335366+0000 osd.5 (osd.5) 3 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:25:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[29319]: cluster 2021-02-10T04:25:57.335460+0000 osd.5 (osd.5) 4 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:25:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[29319]: audit 2021-02-10T04:25:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[29319]: 2021-02-10T04:25:57.805824+0000 mon.b (mon.2) 24 : audit [DBG] from='client.? 172.21.15.164:0/1786624706' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2021-02-10T04:25:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[29319]: 2021-02-10T04:25:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:25:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[29319]: -02-10T04:25:57.950130+0000 mgr.y (mgr.14140) 201 : 2021-02-10T04:25:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[29319]: cluster [DBG] pgmap v182: 132 pgs: 1 creating+activating, 3 creating+peering, 128 active+clean; 22 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 17 KiB/s wr, 134 op/s 2021-02-10T04:25:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[29319]: audit 2021-02-10T04:25:58.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[29319]: 2021-02-10T04:25:57.970803+0000 mgr.y (mgr.14140) 202 : audit [DBG] from='client.34433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi164=a", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:25:58.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[29319]: cephadm 2021-02-10T04:25:57. 2021-02-10T04:25:58.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[29319]: 971902+0000 mgr.y (mgr.14140) 203 : cephadm [INF] Saving service prometheus spec with placement smithi164=a;count:1 2021-02-10T04:25:58.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:25:58.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[29319]: -10T04:25:57.979878 2021-02-10T04:25:58.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[29319]: +0000 mon.a (mon.0) 605 : audit 2021-02-10T04:25:58.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[29319]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[35961]: audit 2021-02-10T04:25:57.317235+0000 mon.a (mon.0) 603 : 2021-02-10T04:25:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[35961]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[35961]: audit 2021-02-10T04:25:57.318750+0000 mon.a (mon.0) 604 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:25:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[35961]: cluster 2021-02-10T04:25:57.335366+0000 2021-02-10T04:25:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[35961]: osd.5 (osd.5) 3 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:25:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[35961]: cluster 2021-02-10T04:25:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[35961]: 2021-02-10T04:25:57.335460+0000 2021-02-10T04:25:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[35961]: osd.5 (osd.5) 4 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:25:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[35961]: audit 2021-02-10T04:25:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[35961]: 2021-02-10T04:25:57.805824+0000 mon.b ( 2021-02-10T04:25:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[35961]: mon.2) 24 : audit [DBG] from='client.? 172.21.15.164:0/1786624706' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2021-02-10T04:25:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[35961]: cluster 2021-02-10T04:25:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[35961]: 2021-02-10T04:25:57.950130+0000 mgr.y ( 2021-02-10T04:25:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[35961]: mgr.14140) 201 : cluster [DBG] pgmap v182: 132 pgs: 1 creating+activating, 3 creating+peering, 128 active+clean; 22 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 17 KiB/s wr, 134 op/s 2021-02-10T04:25:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[35961]: audit 2021-02-10T04:25:57.970803+0000 mgr.y ( 2021-02-10T04:25:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[35961]: mgr.14140) 202 : audit [DBG] from='client.34433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi164=a", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:25:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[35961]: cephadm 2021-02-10T04:25:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[35961]: 2021-02-10T04:25:57.971902+0000 mgr.y (mgr.14140) 203 : cephadm [INF] 2021-02-10T04:25:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[35961]: Saving service prometheus spec with placement smithi164=a;count:1 2021-02-10T04:25:58.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[35961]: audit 2021-02-10T04:25:57 2021-02-10T04:25:58.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[35961]: .979878+0000 mon.a (mon.0) 605 : audit 2021-02-10T04:25:58.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:58 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:25:58.973 DEBUG:teuthology.orchestra.run.smithi164:prometheus.a> sudo journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@prometheus.a.service 2021-02-10T04:25:58.988 INFO:tasks.cephadm:Adding node-exporter.a on smithi085 2021-02-10T04:25:58.989 INFO:tasks.cephadm:Adding node-exporter.b on smithi164 2021-02-10T04:25:58.989 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph orch apply node-exporter '2;smithi085=a;smithi164=b' 2021-02-10T04:25:59.015 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:-- Logs begin at Wed 2021-02-10 04:09:33 UTC. -- 2021-02-10T04:25:59.591 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:59 smithi164 conmon[29288]: cluster 2021-02-10T04:25:58.330290+0000 mon.a ( 2021-02-10T04:25:59.592 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:25:59 smithi164 conmon[29288]: mon.0) 606 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2021-02-10T04:25:59.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:59 smithi085 conmon[29319]: cluster 2021-02-10T04:25:58.330290+0000 mon.a (mon 2021-02-10T04:25:59.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:25:59 smithi085 conmon[29319]: .0) 606 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2021-02-10T04:25:59.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:59 smithi085 conmon[35961]: cluster 2021-02-10T04:25:58.330290 2021-02-10T04:25:59.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:25:59 smithi085 conmon[35961]: +0000 mon.a (mon.0) 606 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2021-02-10T04:26:01.107 INFO:teuthology.orchestra.run.smithi164.stdout:Scheduled node-exporter update... 2021-02-10T04:26:01.131 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:00 smithi164 conmon[29288]: audit 2021-02-10T04:25:59.773797+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:01.132 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:00 smithi164 conmon[29288]: cephadm 2021-02-10T04:25:59.774427+0000 mgr.y (mgr.14140) 204 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:26:01.132 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:00 smithi164 conmon[29288]: cluster 2021-02-10T04:25:59.950763+0000 mgr.y (mgr.14140) 205 : cluster [DBG] pgmap v184: 132 pgs: 132 active+clean; 24 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 15 KiB/s wr, 199 op/s 2021-02-10T04:26:01.132 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:00 smithi164 conmon[29288]: audit 2021-02-10T04:26:00.155685+0000 mon.a (mon.0) 608 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:01.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:00 smithi085 conmon[29319]: audit 2021-02-10T04:25:59.773797+0000 mon.a (mon.0) 607 : audit 2021-02-10T04:26:01.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:00 smithi085 conmon[29319]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:01.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:00 smithi085 conmon[29319]: cephadm 2021-02-10T04:25:59.774427+0000 mgr.y (mgr.14140) 204 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:26:01.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:00 smithi085 conmon[29319]: cluster 2021-02-10T04:25:59.950763+0000 mgr.y (mgr.14140) 205 : cluster [DBG] pgmap v184: 132 pgs: 132 active+clean; 24 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 15 KiB/s wr, 199 op/s 2021-02-10T04:26:01.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:00 smithi085 conmon[29319]: audit 2021-02-10T04:26:00.155685+0000 mon.a (mon.0) 608 : audit [INF] 2021-02-10T04:26:01.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:00 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:01.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:00 smithi085 conmon[35961]: audit 2021-02-10T04:25:59.773797+0000 mon.a (mon.0) 2021-02-10T04:26:01.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:00 smithi085 conmon[35961]: 607 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:01.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:00 smithi085 conmon[35961]: cephadm 2021-02-10T04:25:59.774427+0000 mgr.y (mgr.14140) 204 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:26:01.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:00 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:26:01.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:00 smithi085 conmon[35961]: -10T04:25:59.950763+0000 mgr.y (mgr.14140) 205 : cluster [DBG] pgmap v184: 132 pgs: 132 active+clean; 24 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 15 KiB/s wr, 199 op/s 2021-02-10T04:26:01.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:00 smithi085 conmon[35961]: audit 2021-02-10T04:26: 2021-02-10T04:26:01.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:00 smithi085 conmon[35961]: 00.155685+0000 mon.a (mon.0) 608 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:01.778 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:01 smithi164 conmon[29288]: cephadm 2021-02-10T04:26:00.163701+0000 mgr.y (mgr.14140 2021-02-10T04:26:01.778 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:01 smithi164 conmon[29288]: ) 206 : cephadm [INF] Deploying daemon prometheus.a on smithi164 2021-02-10T04:26:01.778 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:01 smithi164 conmon[29288]: audit 2021-02-10T04:26:01.106768+0000 mon.a (mon.0) 609 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:02.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:01 smithi085 conmon[29319]: cephadm 2021-02-10T04:26:00.163701+0000 mgr.y (mgr.14140) 206 : cephadm [INF] Deploying daemon prometheus.a on smithi164 2021-02-10T04:26:02.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:01 smithi085 conmon[29319]: audit 2021-02-10T04:26:01.106768+0000 mon.a (mon.0) 609 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:02.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:01 smithi085 conmon[35961]: cephadm 2021-02-10T04:26:00.163701+0000 mgr.y (mgr.14140) 206 : cephadm [INF] Deploying daemon prometheus.a on smithi164 2021-02-10T04:26:02.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:01 smithi085 conmon[35961]: audit 2021-02-10T04:26:01.106768+0000 mon.a (mon.0) 609 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:02.572 DEBUG:teuthology.orchestra.run.smithi085:node-exporter.a> sudo journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@node-exporter.a.service 2021-02-10T04:26:02.575 DEBUG:teuthology.orchestra.run.smithi164:node-exporter.b> sudo journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@node-exporter.b.service 2021-02-10T04:26:02.578 INFO:tasks.cephadm:Adding alertmanager.a on smithi085 2021-02-10T04:26:02.578 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph orch apply alertmanager '1;smithi085=a' 2021-02-10T04:26:02.608 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:-- Logs begin at Wed 2021-02-10 04:09:33 UTC. -- 2021-02-10T04:26:02.609 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:-- Logs begin at Wed 2021-02-10 04:09:23 UTC. -- 2021-02-10T04:26:02.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:02 smithi164 conmon[29288]: audit 2021-02-10T04:26:01.102240+0000 mgr.y (mgr.14140) 207 : audit [DBG] from='client.34439 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi085=a;smithi164=b", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:26:02.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:02 smithi164 conmon[29288]: cephadm 2021-02-10T04:26:01.103266+0000 mgr.y (mgr.14140) 208 : cephadm [INF] Saving service node-exporter spec with placement smithi085=a;smithi164=b;count:2 2021-02-10T04:26:02.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:02 smithi164 conmon[29288]: cluster 2021-02-10T04:26:01.951548+0000 mgr.y (mgr.14140) 209 : cluster [DBG] pgmap v185: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 13 KiB/s wr, 231 op/s 2021-02-10T04:26:02.940 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:02 smithi085 conmon[29319]: audit 2021-02-10T04:26:01.102240+0000 mgr.y (mgr.14140) 2021-02-10T04:26:02.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:02 smithi085 conmon[29319]: 207 : audit [DBG] from='client.34439 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi085=a;smithi164=b", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:26:02.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:02 smithi085 conmon[29319]: cephadm 2021-02-10T04:26:01.103266+0000 mgr.y (mgr.14140) 208 : cephadm [INF] Saving service node-exporter spec with placement smithi085=a;smithi164=b;count:2 2021-02-10T04:26:02.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:02 smithi085 conmon[29319]: cluster 2021-02-10T04:26:01.951548+0000 mgr.y (mgr.14140) 209 : cluster [DBG] pgmap v185: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 13 KiB/s wr, 231 op/s 2021-02-10T04:26:02.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:02 smithi085 conmon[35961]: audit 2021-02-10T04:26:01.102240+0000 mgr.y (mgr.14140) 207 : audit [DBG] from='client.34439 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi085=a;smithi164=b", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:26:02.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:02 smithi085 conmon[35961]: cephadm 2021-02-10T04:26:01.103266+0000 mgr.y (mgr.14140) 208 : cephadm [INF] Saving service node-exporter spec with placement smithi085=a;smithi164=b;count:2 2021-02-10T04:26:02.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:02 smithi085 conmon[35961]: cluster 2021-02-10T04:26:01.951548+0000 mgr.y (mgr.14140) 209 : cluster [DBG] pgmap v185: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 13 KiB/s wr, 231 op/s 2021-02-10T04:26:04.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:04 smithi164 conmon[29288]: audit 2021-02-10T04:26:03.045390+0000 mon.a (mon.0 2021-02-10T04:26:04.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:04 smithi164 conmon[29288]: ) 610 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:04.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:04 smithi085 conmon[29319]: audit 2021-02-10T04:26:03.045390+0000 mon.a (mon.0) 610 2021-02-10T04:26:04.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:04 smithi085 conmon[29319]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:04.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:04 smithi085 conmon[35961]: audit 2021-02-10T04:26:03.045390+0000 mon.a (mon. 2021-02-10T04:26:04.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:04 smithi085 conmon[35961]: 0) 610 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:05.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:05 smithi164 conmon[29288]: cluster 2021-02-10T04:26:03.952090+0000 mgr.y (mgr.14140) 210 : cluster [DBG] pgmap v186: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 3.7 KiB/s wr, 179 op/s 2021-02-10T04:26:05.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:05 smithi085 conmon[29319]: cluster 2021-02-10T04:26:03.952090+0000 mgr.y (mgr.14140) 210 : cluster [DBG] pgmap v186: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 3.7 KiB/s wr, 179 op/s 2021-02-10T04:26:05.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:05 smithi085 conmon[35961]: cluster 2021-02-10T04:26:03.952090+0000 mgr.y (mgr.14140) 210 2021-02-10T04:26:05.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:05 smithi085 conmon[35961]: : cluster [DBG] pgmap v186: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 3.7 KiB/s wr, 179 op/s 2021-02-10T04:26:06.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:06 smithi164 conmon[29288]: cluster 2021-02-10T04:26:04.726179+0000 osd.2 (osd.2 2021-02-10T04:26:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:06 smithi164 conmon[29288]: ) 5 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:26:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:06 smithi164 conmon[29288]: cluster 2021-02-10T04:26:04.726304+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:26:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:06 smithi164 conmon[29288]: cluster 2021-02-10T04:26:05.056478+0000 mon.a (mon.0) 611 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2021-02-10T04:26:06.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:06 smithi085 conmon[29319]: cluster 2021-02-10T04:26:04.726179 2021-02-10T04:26:06.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:06 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:26:06.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:06 smithi085 conmon[29319]: cluster 2021-02-10T04:26 2021-02-10T04:26:06.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:06 smithi085 conmon[29319]: :04.726304+0000 osd.2 (osd.2) 6 : cluster [DBG] 2021-02-10T04:26:06.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:06 smithi085 conmon[29319]: purged_snaps scrub ok 2021-02-10T04:26:06.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:06 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:26:06.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:06 smithi085 conmon[29319]: -02-10T04:26:05.056478+0000 2021-02-10T04:26:06.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:06 smithi085 conmon[29319]: mon.a (mon.0) 611 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2021-02-10T04:26:06.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:06 smithi085 conmon[29319]: 2021-02-10T04:26:06.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:06 smithi085 conmon[35961]: cluster 2021-02-10T04:26:04.726179+0000 2021-02-10T04:26:06.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:06 smithi085 conmon[35961]: osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:26:06.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:06 smithi085 conmon[35961]: cluster 2021-02-10T04:26:06.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:06 smithi085 conmon[35961]: 2021-02-10T04:26:04.726304+0000 osd.2 2021-02-10T04:26:06.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:06 smithi085 conmon[35961]: (osd.2) 6 : cluster [DBG] 2021-02-10T04:26:06.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:06 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:26:06.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:06 smithi085 conmon[35961]: cluster 2021-02-10T04:26:06.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:06 smithi085 conmon[35961]: 2021-02 2021-02-10T04:26:06.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:06 smithi085 conmon[35961]: -10T04:26:05. 2021-02-10T04:26:06.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:06 smithi085 conmon[35961]: 056478+0000 mon.a (mon.0 2021-02-10T04:26:06.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:06 smithi085 conmon[35961]: ) 611 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2021-02-10T04:26:06.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:06 smithi085 conmon[35961]: 2021-02-10T04:26:07.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:07 smithi164 conmon[29288]: cluster 2021-02-10T04:26:05.952731+0000 mgr.y (mgr.14140) 211 : cluster [DBG] pgmap v188: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 3.0 KiB/s wr, 158 op/s 2021-02-10T04:26:07.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:07 smithi085 conmon[29319]: cluster 2021-02-10T04:26:05.952731+0000 mgr.y (mgr.14140) 2021-02-10T04:26:07.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:07 smithi085 conmon[29319]: 211 : cluster [DBG] pgmap v188: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 3.0 KiB/s wr, 158 op/s 2021-02-10T04:26:07.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:07 smithi085 conmon[35961]: cluster 2021-02-10T04:26:05.952731 2021-02-10T04:26:07.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:07 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 211 : cluster [DBG] pgmap v188: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 3.0 KiB/s wr, 158 op/s 2021-02-10T04:26:07.822 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:26:07 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:26:08.593 INFO:teuthology.orchestra.run.smithi164.stdout:Scheduled alertmanager update... 2021-02-10T04:26:09.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:09 smithi164 conmon[29288]: audit 2021-02-10T04:26:07.524415+0000 mgr.y (mgr. 2021-02-10T04:26:09.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:09 smithi164 conmon[29288]: 14140) 212 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:26:09.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:09 smithi164 conmon[29288]: cluster 2021-02-10T04:26:07.953246+0000 mgr.y (mgr.14140) 213 : cluster [DBG] pgmap v189: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 1.4 KiB/s wr, 93 op/s 2021-02-10T04:26:09.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:09 smithi164 conmon[29288]: audit 2021-02-10T04:26:08.592560+0000 mon.a (mon.0) 612 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:09.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:09 smithi085 conmon[29319]: audit 2021-02-10T04:26:07.524415+0000 mgr.y (mgr.14140) 212 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:26:09.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:09 smithi085 conmon[29319]: cluster 2021-02-10T04:26:07.953246+0000 mgr.y (mgr.14140) 213 : cluster [DBG] pgmap v189: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 1.4 KiB/s wr, 93 op/s 2021-02-10T04:26:09.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:09 smithi085 conmon[29319]: audit 2021-02-10T04:26:08.592560+0000 mon.a (mon.0) 612 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:09.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:09 smithi085 conmon[35961]: audit 2021-02-10T04:26:07.524415+0000 mgr.y (mgr. 2021-02-10T04:26:09.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:09 smithi085 conmon[35961]: 14140) 212 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:26:09.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:09 smithi085 conmon[35961]: cluster 2021-02-10T04:26:07.953246+0000 mgr.y 2021-02-10T04:26:09.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:09 smithi085 conmon[35961]: (mgr.14140) 213 : cluster [DBG] pgmap v189: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 1.4 KiB/s wr, 93 op/s 2021-02-10T04:26:09.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:09 smithi085 conmon[35961]: audit 2021-02-10T04:26:08. 2021-02-10T04:26:09.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:09 smithi085 conmon[35961]: 592560+0000 mon.a (mon.0) 612 : audit [INF] 2021-02-10T04:26:09.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:09 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:09.688 DEBUG:teuthology.orchestra.run.smithi085:alertmanager.a> sudo journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@alertmanager.a.service 2021-02-10T04:26:09.692 INFO:tasks.cephadm:Adding grafana.a on smithi164 2021-02-10T04:26:09.692 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph orch apply grafana '1;smithi164=a' 2021-02-10T04:26:09.721 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:-- Logs begin at Wed 2021-02-10 04:09:23 UTC. -- 2021-02-10T04:26:10.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:10 smithi164 conmon[29288]: audit 2021-02-10T04:26:08.587538+0000 mgr.y (mgr.14140) 214 : audit [DBG] from='client.34445 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi085=a", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:26:10.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:10 smithi164 conmon[29288]: cephadm 2021-02-10T04:26:08.588624+0000 mgr.y (mgr.14140) 215 : cephadm [INF] Saving service alertmanager spec with placement smithi085=a;count:1 2021-02-10T04:26:10.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:10 smithi085 conmon[29319]: audit 2021-02-10T04:26:08.587538+0000 mgr.y (mgr.14140) 214 2021-02-10T04:26:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:10 smithi085 conmon[29319]: : audit [DBG] from='client.34445 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi085=a", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:26:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:10 smithi085 conmon[29319]: cephadm 2021-02-10T04:26:08.588624+0000 mgr.y (mgr.14140) 215 : cephadm [INF] Saving service alertmanager spec with placement smithi085=a;count:1 2021-02-10T04:26:10.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:10 smithi085 conmon[35961]: audit 2021-02-10T04:26:08.587538+0000 mgr.y (mgr.14140) 214 2021-02-10T04:26:10.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:10 smithi085 conmon[35961]: : audit [DBG] from='client.34445 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi085=a", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:26:10.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:10 smithi085 conmon[35961]: cephadm 2021-02-10T04:26:08.588624+0000 mgr.y (mgr.14140) 215 : cephadm [INF] Saving service alertmanager spec with placement smithi085=a;count:1 2021-02-10T04:26:11.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:11 smithi164 conmon[29288]: cluster 2021-02-10T04:26:09.953830+0000 2021-02-10T04:26:11.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:11 smithi164 conmon[29288]: mgr.y (mgr.14140) 216 : cluster [DBG] pgmap v190: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 1.1 KiB/s wr, 76 op/s 2021-02-10T04:26:11.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:11 smithi085 conmon[29319]: cluster 2021-02-10T04:26:09.953830+0000 mgr.y (mgr.14140) 216 : cluster [DBG] pgmap v190: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 1.1 KiB/s wr, 76 op/s 2021-02-10T04:26:11.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:11 smithi085 conmon[35961]: cluster 2021-02-10T04:26:09.953830+0000 mgr.y (mgr. 2021-02-10T04:26:11.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:11 smithi085 conmon[35961]: 14140) 216 : cluster [DBG] pgmap v190: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 1.1 KiB/s wr, 76 op/s 2021-02-10T04:26:12.393 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:26:12 smithi164 conmon[50601]: level=info ts=2021-02-10T04:26:12.076Z caller=main.go:646 msg="Server is ready to receive web requests." 2021-02-10T04:26:12.393 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:26:12 smithi164 systemd[1]: Started Ceph prometheus.a for 42224fce-6b57-11eb-8feb-001a4aab830c. 2021-02-10T04:26:12.400 INFO:teuthology.orchestra.run.smithi164.stdout:Scheduled grafana update... 2021-02-10T04:26:13.031 DEBUG:teuthology.orchestra.run.smithi164:grafana.a> sudo journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@grafana.a.service 2021-02-10T04:26:16.207 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:13 smithi085 conmon[29319]: cluster 2021-02-10T04:26:11.954471+0000 mgr.y (mgr.14140 2021-02-10T04:26:16.207 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:13 smithi085 conmon[29319]: ) 217 : cluster [DBG] pgmap v191: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 9.0 KiB/s rd, 0 B/s wr, 12 op/s 2021-02-10T04:26:16.207 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:13 smithi085 conmon[29319]: audit 2021-02-10T04:26:12.142187+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:16.207 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:13 smithi085 conmon[29319]: audit 2021-02-10T04:26:16.207 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:13 smithi085 conmon[29319]: 2021-02-10T04:26:12.143655+0000 mon.a (mon.0) 614 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:26:16.207 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:13 smithi085 conmon[29319]: audit 2021-02-10T04:26:12.399415+0000 mon.a (mon.0) 615 : audit [INF] 2021-02-10T04:26:16.208 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:13 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:16.208 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:14 smithi085 conmon[29319]: audit 2021-02-10T04:26:12.392494+0000 mgr.y (mgr.14140) 218 2021-02-10T04:26:16.208 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:14 smithi085 conmon[29319]: : audit [DBG] from='client.34451 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi164=a", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:26:16.208 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:14 smithi085 conmon[29319]: cephadm 2021-02-10T04:26:12.393387+0000 mgr.y (mgr. 2021-02-10T04:26:16.208 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:14 smithi085 conmon[29319]: 14140) 219 : cephadm [INF] Saving service grafana spec with placement smithi164=a;count:1 2021-02-10T04:26:16.208 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:14 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:26:16.208 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:14 smithi085 conmon[29319]: 02-10T04:26:13.954964+0000 2021-02-10T04:26:16.208 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:14 smithi085 conmon[29319]: mgr.y (mgr.14140) 220 : cluster 2021-02-10T04:26:16.208 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:14 smithi085 conmon[29319]: [DBG] pgmap v192: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 0 B/s wr, 9 op/s 2021-02-10T04:26:16.208 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:15 smithi085 conmon[29319]: audit 2021-02-10T04:26:14.647089+0000 mon.a 2021-02-10T04:26:16.208 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:15 smithi085 conmon[29319]: (mon.0) 616 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:16.208 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:15 smithi085 conmon[29319]: audit 2021-02-10T04:26:14.653316+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:16.208 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:15 smithi085 conmon[29319]: cephadm 2021-02-10T04:26:14.655821+0000 mgr.y (mgr. 2021-02-10T04:26:16.209 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:15 smithi085 conmon[29319]: 14140) 221 : cephadm [INF] Deploying daemon node-exporter.b on smithi164 2021-02-10T04:26:16.209 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:13 smithi085 conmon[35961]: cluster 2021-02-10T04:26:11.954471+0000 mgr.y (mgr. 2021-02-10T04:26:16.209 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:13 smithi085 conmon[35961]: 14140) 217 : cluster [DBG] pgmap v191: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 9.0 KiB/s rd, 0 B/s wr, 12 op/s 2021-02-10T04:26:16.209 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:13 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:26:16.209 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:13 smithi085 conmon[35961]: 10T04:26:12.142187+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:16.209 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:13 smithi085 conmon[35961]: audit 2021-02-10T04:26:16.209 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:13 smithi085 conmon[35961]: 2021-02-10T04:26:12.143655+0000 mon.a ( 2021-02-10T04:26:16.209 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:13 smithi085 conmon[35961]: mon.0) 614 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:26:16.209 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:13 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:26:16.209 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:13 smithi085 conmon[35961]: 10T04:26:12.399415+0000 mon.a (mon.0) 615 : 2021-02-10T04:26:16.209 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:13 smithi085 conmon[35961]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:16.209 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:14 smithi085 conmon[35961]: audit 2021-02-10T04:26:12.392494 2021-02-10T04:26:16.210 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:14 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 218 : audit [DBG] from='client.34451 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi164=a", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:26:16.210 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:14 smithi085 conmon[35961]: cephadm 2021-02-10T04:26:16.210 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:14 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:26:16.210 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:14 smithi085 conmon[35961]: 10T04:26:12.393387 2021-02-10T04:26:16.210 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:14 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 2021-02-10T04:26:16.210 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:14 smithi085 conmon[35961]: 219 : cephadm [INF] Saving service grafana spec with placement smithi164=a;count:1 2021-02-10T04:26:16.210 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:14 smithi085 conmon[35961]: cluster 2021-02-10T04:26:16.210 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:14 smithi085 conmon[35961]: 2021-02-10T04:26:13.954964+0000 2021-02-10T04:26:16.210 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:14 smithi085 conmon[35961]: mgr.y (mgr.14140) 220 : cluster [DBG] pgmap v192: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 0 B/s wr, 9 op/s 2021-02-10T04:26:16.210 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:14 smithi085 conmon[35961]: 2021-02-10T04:26:16.210 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:15 smithi085 conmon[35961]: audit 2021-02-10T04:26:14.647089+0000 2021-02-10T04:26:16.210 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:15 smithi085 conmon[35961]: mon.a (mon.0) 616 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:16.210 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:15 smithi085 conmon[35961]: audit 2021 2021-02-10T04:26:16.210 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:15 smithi085 conmon[35961]: -02-10T04:26:14.653316+0000 mon.a (mon 2021-02-10T04:26:16.211 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:15 smithi085 conmon[35961]: .0) 617 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:16.211 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:15 smithi085 conmon[35961]: cephadm 2021-02-10T 2021-02-10T04:26:16.211 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:15 smithi085 conmon[35961]: 04:26:14.655821+0000 mgr.y (mgr.14140) 221 : cephadm [INF] Deploying daemon node-exporter.b on smithi164 2021-02-10T04:26:16.211 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:13 smithi164 conmon[29288]: cluster 2021-02-10T04:26:11.954471+0000 mgr.y (mgr.14140) 217 : cluster [DBG] pgmap v191: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 9.0 KiB/s rd, 0 B/s wr, 12 op/s 2021-02-10T04:26:16.211 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:13 smithi164 conmon[29288]: audit 2021-02-10T04:26:12.142187+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:16.211 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:13 smithi164 conmon[29288]: audit 2021-02-10T04:26:12.143655+0000 mon.a (mon.0) 614 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:26:16.211 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:13 smithi164 conmon[29288]: audit 2021-02-10T04:26:12.399415+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:16.211 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:14 smithi164 conmon[29288]: audit 2021-02-10T04:26:12.392494+0000 mgr.y (mgr.14140) 218 : audit 2021-02-10T04:26:16.211 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:14 smithi164 conmon[29288]: [DBG] from='client.34451 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi164=a", "target": ["mon-mgr", ""]}]: dispatch 2021-02-10T04:26:16.211 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:14 smithi164 conmon[29288]: cephadm 2021-02-10T04:26:12.393387+0000 mgr.y (mgr.14140) 219 : cephadm [INF] Saving service grafana spec with placement smithi164=a;count:1 2021-02-10T04:26:16.211 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:14 smithi164 conmon[29288]: cluster 2021-02-10T04:26:13.954964+0000 mgr.y (mgr.14140) 220 : cluster [DBG] pgmap v192: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 0 B/s wr, 9 op/s 2021-02-10T04:26:16.211 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:15 smithi164 conmon[29288]: audit 2021-02-10T04:26:14.647089+0000 mon.a (mon.0) 616 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:16.211 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:15 smithi164 conmon[29288]: audit 2021-02-10T04:26:14.653316+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:16.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:15 smithi164 conmon[29288]: cephadm 2021-02-10T04:26:14.655821+0000 mgr.y (mgr.14140) 221 : cephadm [INF] Deploying daemon node-exporter.b on smithi164 2021-02-10T04:26:16.212 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:15 smithi164 systemd[1]: Starting Ceph node-exporter.b for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T04:26:16.212 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:15 smithi164 bash[51459]: Trying to pull docker.io/prom/node-exporter:v0.18.1... 2021-02-10T04:26:16.217 INFO:tasks.cephadm:Setting up client nodes... 2021-02-10T04:26:16.217 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2021-02-10T04:26:16.252 INFO:journalctl@ceph.grafana.a.smithi164.stdout:-- Logs begin at Wed 2021-02-10 04:09:33 UTC. -- 2021-02-10T04:26:16.516 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:16 smithi164 bash[51459]: Getting image source signatures 2021-02-10T04:26:16.808 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:16 smithi164 conmon[29288]: cluster 2021-02-10T04:26:15.955790+0000 mgr.y (mgr.14140) 222 : cluster 2021-02-10T04:26:16.808 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:16 smithi164 conmon[29288]: [DBG] pgmap v193: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 939 B/s rd, 0 op/s 2021-02-10T04:26:16.809 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:16 smithi164 bash[51459]: Copying blob sha256:3589a6efd9ce96b2f2218a80bacb84f60d5e584d024023617b0f56b87fe798c3 2021-02-10T04:26:16.809 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:16 smithi164 bash[51459]: Copying blob sha256:49a2d53aa1af115e87f7e23a7fe4b7a3e12304d37982fb6528b035d0f68a3b56 2021-02-10T04:26:16.809 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:16 smithi164 bash[51459]: Copying blob sha256:190160031744b0810999dd6f293478be0b41524654c70782cf09437095430f2f 2021-02-10T04:26:16.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:16 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:26:16.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:16 smithi085 conmon[29319]: :26:15.955790+0000 mgr.y (mgr.14140) 222 : cluster [DBG] pgmap v193: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 939 B/s rd, 0 op/s 2021-02-10T04:26:16.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:16 smithi085 conmon[35961]: cluster 2021-02-10T04:26: 2021-02-10T04:26:16.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:16 smithi085 conmon[35961]: 15.955790+0000 mgr.y (mgr.14140) 222 : cluster [DBG] pgmap v193: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 939 B/s rd, 0 op/s 2021-02-10T04:26:17.142 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:16 smithi164 bash[51459]: Copying config sha256:e5a616e4b9cf68dfcad7782b78e118be4310022e874d52da85c55923fb615f87 2021-02-10T04:26:17.143 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:16 smithi164 bash[51459]: Writing manifest to image destination 2021-02-10T04:26:17.143 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:16 smithi164 bash[51459]: Storing signatures 2021-02-10T04:26:17.893 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:26:17 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:26:17.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:17 smithi164 conmon[29288]: cluster 2021-02-10T04:26:16.688455+0000 osd.6 (osd. 2021-02-10T04:26:17.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:17 smithi164 conmon[29288]: 6) 4 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:26:17.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:17 smithi164 conmon[29288]: cluster 2021-02-10T04:26:16.688555+0000 osd.6 (osd.6) 5 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:26:17.940 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:17 smithi085 conmon[29319]: cluster 2021-02-10T04:26:16.688455+0000 2021-02-10T04:26:17.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:17 smithi085 conmon[29319]: osd.6 (osd.6) 4 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:26:17.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:17 smithi085 conmon[29319]: cluster 2021-02-10T04:26:16.688555+0000 osd.6 (osd.6) 5 : cluster 2021-02-10T04:26:17.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:17 smithi085 conmon[29319]: [DBG] purged_snaps scrub ok 2021-02-10T04:26:17.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:17 smithi085 conmon[35961]: cluster 2021-02-10T04:26:16.688455+0000 osd.6 (osd.6 2021-02-10T04:26:17.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:17 smithi085 conmon[35961]: ) 4 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:26:17.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:17 smithi085 conmon[35961]: cluster 2021-02-10T04:26:16.688555+0000 osd.6 (osd.6) 5 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:26:17.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:17 smithi085 conmon[35961]: 2021-02-10T04:26:18.643 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156" 2021-02-10T04:26:18.643 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:41:18)" source="node_exporter.go:157" 2021-02-10T04:26:18.643 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg="Enabled collectors:" source="node_exporter.go:97" 2021-02-10T04:26:18.643 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - arp" source="node_exporter.go:104" 2021-02-10T04:26:18.644 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - bcache" source="node_exporter.go:104" 2021-02-10T04:26:18.644 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - bonding" source="node_exporter.go:104" 2021-02-10T04:26:18.644 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - conntrack" source="node_exporter.go:104" 2021-02-10T04:26:18.644 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - cpu" source="node_exporter.go:104" 2021-02-10T04:26:18.644 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - cpufreq" source="node_exporter.go:104" 2021-02-10T04:26:18.644 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - diskstats" source="node_exporter.go:104" 2021-02-10T04:26:18.644 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - edac" source="node_exporter.go:104" 2021-02-10T04:26:18.644 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - entropy" source="node_exporter.go:104" 2021-02-10T04:26:18.644 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - filefd" source="node_exporter.go:104" 2021-02-10T04:26:18.644 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - filesystem" source="node_exporter.go:104" 2021-02-10T04:26:18.644 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - hwmon" source="node_exporter.go:104" 2021-02-10T04:26:18.644 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - infiniband" source="node_exporter.go:104" 2021-02-10T04:26:18.645 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - ipvs" source="node_exporter.go:104" 2021-02-10T04:26:18.645 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - loadavg" source="node_exporter.go:104" 2021-02-10T04:26:18.645 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - mdadm" source="node_exporter.go:104" 2021-02-10T04:26:18.645 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - meminfo" source="node_exporter.go:104" 2021-02-10T04:26:18.645 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - netclass" source="node_exporter.go:104" 2021-02-10T04:26:18.645 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - netdev" source="node_exporter.go:104" 2021-02-10T04:26:18.645 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - netstat" source="node_exporter.go:104" 2021-02-10T04:26:18.645 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - nfs" source="node_exporter.go:104" 2021-02-10T04:26:18.645 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - nfsd" source="node_exporter.go:104" 2021-02-10T04:26:18.645 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - pressure" source="node_exporter.go:104" 2021-02-10T04:26:18.645 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - sockstat" source="node_exporter.go:104" 2021-02-10T04:26:18.645 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - stat" source="node_exporter.go:104" 2021-02-10T04:26:18.645 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - textfile" source="node_exporter.go:104" 2021-02-10T04:26:18.645 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - time" source="node_exporter.go:104" 2021-02-10T04:26:18.646 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - uname" source="node_exporter.go:104" 2021-02-10T04:26:18.646 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - vmstat" source="node_exporter.go:104" 2021-02-10T04:26:18.646 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - xfs" source="node_exporter.go:104" 2021-02-10T04:26:18.646 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg=" - zfs" source="node_exporter.go:104" 2021-02-10T04:26:18.646 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[51623]: time="2021-02-10T04:26:18Z" level=info msg="Listening on :9100" source="node_exporter.go:170" 2021-02-10T04:26:18.646 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 bash[51459]: e321ea0c60a542754dbd9e566d0f1b0ad9403f535811f3f308f4d33ed2472bc2 2021-02-10T04:26:18.646 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 04:26:18 smithi164 systemd[1]: Started Ceph node-exporter.b for 42224fce-6b57-11eb-8feb-001a4aab830c. 2021-02-10T04:26:18.921 INFO:teuthology.orchestra.run.smithi085.stdout:[client.0] 2021-02-10T04:26:18.921 INFO:teuthology.orchestra.run.smithi085.stdout: key = AQBqYCNg4AmNNhAAvcef33JM2yGdNJ9nPWEY1w== 2021-02-10T04:26:18.936 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:18 smithi085 conmon[29319]: audit 2021-02-10T04:26:17.535625+0000 mgr.y (mgr.14140 2021-02-10T04:26:18.936 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:18 smithi085 conmon[29319]: ) 223 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:26:18.937 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:18 smithi085 conmon[29319]: cluster 2021-02-10T04:26:17.654502+0000 mon.a (mon.0) 618 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2021-02-10T04:26:18.937 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:18 smithi085 conmon[29319]: cluster 2021-02-10T04:26:17.956166+0000 mgr.y (mgr.14140) 224 : cluster [DBG] pgmap v195: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:26:18.937 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:18 smithi085 conmon[29319]: audit 2021-02-10T04:26:18.376561+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:18.937 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:18 smithi085 conmon[35961]: audit 2021-02-10T04:26 2021-02-10T04:26:18.937 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:18 smithi085 conmon[35961]: :17.535625+0000 mgr.y (mgr.14140) 223 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:26:18.937 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:18 smithi085 conmon[35961]: cluster 2021-02-10T04:26:18.937 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:18 smithi085 conmon[35961]: 2021-02-10T04:26:17.654502+0000 2021-02-10T04:26:18.937 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:18 smithi085 conmon[35961]: mon.a (mon.0) 618 : 2021-02-10T04:26:18.937 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:18 smithi085 conmon[35961]: cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2021-02-10T04:26:18.937 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:18 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:26:18.937 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:18 smithi085 conmon[35961]: -02-10T04:26:17 2021-02-10T04:26:18.938 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:18 smithi085 conmon[35961]: .956166+0000 mgr.y (mgr. 2021-02-10T04:26:18.938 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:18 smithi085 conmon[35961]: 14140) 224 : cluster [DBG] 2021-02-10T04:26:18.938 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:18 smithi085 conmon[35961]: pgmap v195: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:26:18.938 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:18 smithi085 conmon[35961]: audit 2021 2021-02-10T04:26:18.938 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:18 smithi085 conmon[35961]: -02-10T04:26 2021-02-10T04:26:18.938 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:18 smithi085 conmon[35961]: :18.376561+0000 mon.a ( 2021-02-10T04:26:18.938 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:18 smithi085 conmon[35961]: mon.0) 619 : audit 2021-02-10T04:26:18.938 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:18 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:19.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[29288]: audit 2021-02-10T04:26:17.535625+0000 mgr.y (mgr.14140) 223 2021-02-10T04:26:19.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[29288]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:26:19.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[29288]: cluster 2021-02-10T04:26:17.654502+0000 mon.a (mon.0) 618 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2021-02-10T04:26:19.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:26:19.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[29288]: -10T04:26:17.956166+0000 mgr.y (mgr.14140) 224 : cluster [DBG] pgmap v195: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:26:19.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[29288]: audit 2021-02-10T04:26:19.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[29288]: 2021-02-10T04:26:18.376561+0000 mon.a (mon.0) 619 : audit [INF] 2021-02-10T04:26:19.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:18 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:19.511 DEBUG:teuthology.orchestra.run.smithi085:> set -ex 2021-02-10T04:26:19.511 DEBUG:teuthology.orchestra.run.smithi085:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2021-02-10T04:26:19.511 DEBUG:teuthology.orchestra.run.smithi085:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2021-02-10T04:26:19.549 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2021-02-10T04:26:19.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:19 smithi164 conmon[29288]: cephadm 2021-02-10T04:26:18.377444+0000 mgr.y (mgr.14140) 225 : cephadm 2021-02-10T04:26:19.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:19 smithi164 conmon[29288]: [INF] Deploying daemon node-exporter.a on smithi085 2021-02-10T04:26:19.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:19 smithi164 conmon[29288]: audit 2021-02-10T04:26:18.914669+0000 mon.c (mon.1) 17 : audit [INF] from='client.? 172.21.15.85:0/498335962' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-02-10T04:26:19.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:19 smithi164 conmon[29288]: audit 2021-02-10T04:26:18.915004+0000 mon.a (mon.0 2021-02-10T04:26:19.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:19 smithi164 conmon[29288]: ) 620 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-02-10T04:26:19.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:19 smithi164 conmon[29288]: audit 2021-02-10T04:26:18.918646+0000 mon.a (mon.0) 2021-02-10T04:26:19.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:19 smithi164 conmon[29288]: 621 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-02-10T04:26:19.941 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:19 smithi085 bash[56105]: Trying to pull docker.io/prom/node-exporter:v0.18.1... 2021-02-10T04:26:19.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:19 smithi085 conmon[29319]: cephadm 2021-02-10T04:26:18. 2021-02-10T04:26:19.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:19 smithi085 conmon[29319]: 377444+0000 mgr.y (mgr.14140) 225 : cephadm [INF] Deploying daemon node-exporter.a on smithi085 2021-02-10T04:26:19.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:19 smithi085 conmon[29319]: audit 2021-02-10T04:26:18.914669+0000 mon.c (mon.1) 17 : audit [INF] from='client.? 172.21.15.85:0/498335962' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-02-10T04:26:19.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:19 smithi085 conmon[29319]: audit 2021-02-10T04:26:18.915004+0000 mon.a (mon.0) 620 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-02-10T04:26:19.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:19 smithi085 conmon[29319]: audit 2021-02-10T04:26:18.918646+0000 mon.a (mon.0) 621 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-02-10T04:26:19.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:19 smithi085 conmon[35961]: cephadm 2021-02-10T04:26:18.377444+0000 mgr.y (mgr.14140) 225 : cephadm [INF] Deploying daemon node-exporter.a on smithi085 2021-02-10T04:26:19.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:19 smithi085 conmon[35961]: audit 2021-02-10T04:26:18.914669+0000 mon.c (mon.1) 17 : audit [INF] from='client.? 172.21.15.85:0/498335962' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-02-10T04:26:19.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:19 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:26:19.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:19 smithi085 conmon[35961]: 10T04:26:18.915004+0000 mon.a (mon.0) 620 : audit 2021-02-10T04:26:19.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:19 smithi085 conmon[35961]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-02-10T04:26:19.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:19 smithi085 conmon[35961]: audit 2021-02-10T04:26 2021-02-10T04:26:19.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:19 smithi085 conmon[35961]: :18.918646+0000 mon.a (mon.0) 621 : 2021-02-10T04:26:19.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:19 smithi085 conmon[35961]: audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-02-10T04:26:20.940 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:20 smithi085 conmon[29319]: cluster 2021-02-10T04:26:19.956995+0000 mgr.y (mgr.14140) 226 : cluster [DBG] 2021-02-10T04:26:20.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:20 smithi085 conmon[29319]: pgmap v196: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:26:20.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:20 smithi085 conmon[35961]: cluster 2021-02-10T04:26:19.956995+0000 mgr.y (mgr.14140) 2021-02-10T04:26:20.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:20 smithi085 conmon[35961]: 226 : cluster [DBG] pgmap v196: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:26:21.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:20 smithi164 conmon[29288]: cluster 2021-02-10T04:26:19.956995+0000 mgr.y (mgr.14140) 226 : cluster [DBG] pgmap v196: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:26:21.800 INFO:teuthology.orchestra.run.smithi164.stdout:[client.1] 2021-02-10T04:26:21.801 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBtYCNghOVuLxAAdw9sGwsQTgXSaRPk5C7yAw== 2021-02-10T04:26:22.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:21 smithi164 conmon[29288]: audit 2021-02-10T04:26:21.795088+0000 mon.b (mon.2) 25 : audit [INF] from='client.? 172.21.15.164:0/648238573' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-02-10T04:26:22.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:21 smithi164 conmon[29288]: audit 2021-02-10T04:26:21.795620+0000 mon.a (mon.0) 622 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-02-10T04:26:22.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:21 smithi164 conmon[29288]: audit 2021-02-10T04:26:21.800446+0000 mon.a (mon.0) 623 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-02-10T04:26:22.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:21 smithi085 conmon[35961]: audit 2021-02-10T04:26:21.795088 2021-02-10T04:26:22.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:21 smithi085 conmon[35961]: +0000 mon.b (mon.2) 25 : audit [INF] from='client.? 172.21.15.164:0/648238573' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-02-10T04:26:22.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:21 smithi085 conmon[35961]: audit 2021-02-10T04:26:21.795620+0000 mon.a (mon 2021-02-10T04:26:22.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:21 smithi085 conmon[35961]: .0) 622 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-02-10T04:26:22.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:21 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:26:22.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:21 smithi085 conmon[35961]: 10T04:26:21.800446+0000 mon.a (mon.0 2021-02-10T04:26:22.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:21 smithi085 conmon[35961]: ) 623 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-02-10T04:26:22.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:21 smithi085 conmon[29319]: audit 2021-02-10T04:26:21.795088+0000 mon.b (mon.2) 2021-02-10T04:26:22.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:21 smithi085 conmon[29319]: 25 : audit [INF] from='client.? 172.21.15.164:0/648238573' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-02-10T04:26:22.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:21 smithi085 conmon[29319]: audit 2021-02-10T04:26:21.795620+0000 mon.a (mon.0) 622 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-02-10T04:26:22.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:21 smithi085 conmon[29319]: 2021-02-10T04:26:22.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:21 smithi085 conmon[29319]: audit 2021-02-10T04:26:22.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:21 smithi085 conmon[29319]: 2021-02-10T04:26 2021-02-10T04:26:22.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:21 smithi085 conmon[29319]: :21.800446+0000 mon.a (mon 2021-02-10T04:26:22.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:21 smithi085 conmon[29319]: .0) 623 : audit [INF] 2021-02-10T04:26:22.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:21 smithi085 conmon[29319]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-02-10T04:26:22.397 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2021-02-10T04:26:22.397 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2021-02-10T04:26:22.397 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2021-02-10T04:26:22.436 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd dump --format=json 2021-02-10T04:26:22.643 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:26:23.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:22 smithi164 conmon[29288]: cluster 2021-02-10T04:26:21.957585+0000 mgr.y (mgr.14140) 227 : cluster [DBG] pgmap v197: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:26:23.188 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:22 smithi085 conmon[29319]: cluster 2021-02-10T04:26:21.957585+0000 mgr.y (mgr.14140 2021-02-10T04:26:23.189 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:22 smithi085 conmon[29319]: ) 227 : cluster [DBG] pgmap v197: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:26:23.189 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:22 smithi085 bash[56105]: Getting image source signatures 2021-02-10T04:26:23.189 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:22 smithi085 bash[56105]: Copying blob sha256:190160031744b0810999dd6f293478be0b41524654c70782cf09437095430f2f 2021-02-10T04:26:23.189 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:23 smithi085 bash[56105]: Copying blob sha256:3589a6efd9ce96b2f2218a80bacb84f60d5e584d024023617b0f56b87fe798c3 2021-02-10T04:26:23.189 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:23 smithi085 bash[56105]: Copying blob sha256:49a2d53aa1af115e87f7e23a7fe4b7a3e12304d37982fb6528b035d0f68a3b56 2021-02-10T04:26:23.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:22 smithi085 conmon[35961]: cluster 2021-02-10T04:26:21.957585+0000 2021-02-10T04:26:23.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:22 smithi085 conmon[35961]: mgr.y (mgr.14140) 227 : cluster [DBG] pgmap v197: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:26:23.441 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:23 smithi085 bash[56105]: Copying config sha256:e5a616e4b9cf68dfcad7782b78e118be4310022e874d52da85c55923fb615f87 2021-02-10T04:26:23.441 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:23 smithi085 bash[56105]: Writing manifest to image destination 2021-02-10T04:26:23.441 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:23 smithi085 bash[56105]: Storing signatures 2021-02-10T04:26:24.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:23 smithi164 conmon[29288]: cluster 2021-02-10T04:26:23.293941+0000 osd.3 (osd. 2021-02-10T04:26:24.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:23 smithi164 conmon[29288]: 3) 5 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:26:24.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:23 smithi164 conmon[29288]: cluster 2021-02-10T04:26:23.294051+0000 osd.3 (osd.3) 6 : cluster [DBG] 2021-02-10T04:26:24.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:23 smithi164 conmon[29288]: purged_snaps scrub ok 2021-02-10T04:26:24.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:23 smithi085 conmon[35961]: cluster 2021-02-10T04:26:23.293941+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:26:24.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:23 smithi085 conmon[35961]: cluster 2021-02-10T04:26:23.294051+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:26:24.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:23 smithi085 conmon[29319]: cluster 2021-02-10T04:26:23.293941+0000 osd.3 2021-02-10T04:26:24.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:23 smithi085 conmon[29319]: (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:26:24.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:23 smithi085 conmon[29319]: cluster 2021-02-10T04:26:23.294051+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:26:25.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:24 smithi085 conmon[29319]: cluster 2021-02-10T04:26:23.958035+0000 mgr.y (mgr.14140) 228 : cluster [DBG] pgmap v198: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:26:25.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:24 smithi085 conmon[29319]: cluster 2021-02-10T04:26:23.989244+0000 mon.a (mon.0) 624 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2021-02-10T04:26:25.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:24 smithi085 conmon[35961]: cluster 2021-02-10T04:26:23.958035+0000 mgr.y (mgr.14140) 228 : cluster [DBG] pgmap v198: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:26:25.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:24 smithi085 conmon[35961]: cluster 2021-02-10T04:26:23.989244+0000 mon.a (mon.0) 624 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2021-02-10T04:26:25.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:24 smithi164 conmon[29288]: cluster 2021-02-10T04:26:23.958035+0000 mgr.y (mgr.14140) 228 : cluster [DBG] pgmap v198: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:26:25.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:24 smithi164 conmon[29288]: cluster 2021-02-10T04:26:23.989244+0000 mon.a (mon.0) 624 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2021-02-10T04:26:25.709 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:26:25.710 INFO:teuthology.orchestra.run.smithi085.stdout:{"epoch":66,"fsid":"42224fce-6b57-11eb-8feb-001a4aab830c","created":"2021-02-10T04:20:57.933279+0000","modified":"2021-02-10T04:26:23.977916+0000","last_up_change":"2021-02-10T04:25:35.312604+0000","last_in_change":"2021-02-10T04:25:35.312604+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-10T04:24:05.258028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2021-02-10T04:25:32.688554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"iscsi","create_time":"2021-02-10T04:25:47.658613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"zone.rgw.log","create_time":"2021-02-10T04:25:47.968727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"zone.rgw.control","create_time":"2021-02-10T04:25:49.278260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"zone.rgw.meta","create_time":"2021-02-10T04:25:51.792514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"45d0bb5f-35fa-4596-964f-75abcee53bee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6801","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6802","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6804","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6803","nonce":7}]},"public_addr":"172.21.15.85:6801/7","cluster_addr":"172.21.15.85:6802/7","heartbeat_back_addr":"172.21.15.85:6804/7","heartbeat_front_addr":"172.21.15.85:6803/7","state":["exists","up"]},{"osd":1,"uuid":"3421a5ce-872e-4b52-adf7-6aaa595871a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6805","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6806","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6808","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6807","nonce":7}]},"public_addr":"172.21.15.85:6805/7","cluster_addr":"172.21.15.85:6806/7","heartbeat_back_addr":"172.21.15.85:6808/7","heartbeat_front_addr":"172.21.15.85:6807/7","state":["exists","up"]},{"osd":2,"uuid":"96e1952a-1042-4316-9fc9-86813e88b912","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6809","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6810","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6812","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6811","nonce":7}]},"public_addr":"172.21.15.85:6809/7","cluster_addr":"172.21.15.85:6810/7","heartbeat_back_addr":"172.21.15.85:6812/7","heartbeat_front_addr":"172.21.15.85:6811/7","state":["exists","up"]},{"osd":3,"uuid":"e019f49f-5dd4-4104-9ced-94c710616642","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6813","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6814","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6816","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6815","nonce":7}]},"public_addr":"172.21.15.85:6813/7","cluster_addr":"172.21.15.85:6814/7","heartbeat_back_addr":"172.21.15.85:6816/7","heartbeat_front_addr":"172.21.15.85:6815/7","state":["exists","up"]},{"osd":4,"uuid":"91c8a80a-37b6-4c30-a32a-647b2e6d801f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6801","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6803","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":7}]},"public_addr":"172.21.15.164:6800/7","cluster_addr":"172.21.15.164:6801/7","heartbeat_back_addr":"172.21.15.164:6803/7","heartbeat_front_addr":"172.21.15.164:6802/7","state":["exists","up"]},{"osd":5,"uuid":"cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6805","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":7}]},"public_addr":"172.21.15.164:6804/7","cluster_addr":"172.21.15.164:6805/7","heartbeat_back_addr":"172.21.15.164:6807/7","heartbeat_front_addr":"172.21.15.164:6806/7","state":["exists","up"]},{"osd":6,"uuid":"adc40430-3eeb-4003-a824-343021243077","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6809","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6811","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":7}]},"public_addr":"172.21.15.164:6808/7","cluster_addr":"172.21.15.164:6809/7","heartbeat_back_addr":"172.21.15.164:6811/7","heartbeat_front_addr":"172.21.15.164:6810/7","state":["exists","up"]},{"osd":7,"uuid":"a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6813","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":7}]},"public_addr":"172.21.15.164:6812/7","cluster_addr":"172.21.15.164:6813/7","heartbeat_back_addr":"172.21.15.164:6815/7","heartbeat_front_addr":"172.21.15.164:6814/7","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:25:29.273608+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:25:46.116995+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:26:04.726309+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:26:23.294056+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:25:39.867919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:25:57.335463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:26:16.688557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:25:34.164617+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.85:0/801393150":"2021-02-11T04:21:37.851607+0000","172.21.15.85:0/3049671696":"2021-02-11T04:21:37.851607+0000","172.21.15.85:0/4056505602":"2021-02-11T04:21:20.700478+0000","172.21.15.85:6800/7":"2021-02-11T04:21:20.700478+0000","172.21.15.85:0/1879026062":"2021-02-11T04:21:20.700478+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-10T04:26:25.898 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156" 2021-02-10T04:26:25.898 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:41:18)" source="node_exporter.go:157" 2021-02-10T04:26:25.898 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg="Enabled collectors:" source="node_exporter.go:97" 2021-02-10T04:26:25.898 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - arp" source="node_exporter.go:104" 2021-02-10T04:26:25.898 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - bcache" source="node_exporter.go:104" 2021-02-10T04:26:25.898 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - bonding" source="node_exporter.go:104" 2021-02-10T04:26:25.898 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - conntrack" source="node_exporter.go:104" 2021-02-10T04:26:25.899 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - cpu" source="node_exporter.go:104" 2021-02-10T04:26:25.899 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - cpufreq" source="node_exporter.go:104" 2021-02-10T04:26:25.899 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - diskstats" source="node_exporter.go:104" 2021-02-10T04:26:25.899 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - edac" source="node_exporter.go:104" 2021-02-10T04:26:25.899 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - entropy" source="node_exporter.go:104" 2021-02-10T04:26:25.899 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - filefd" source="node_exporter.go:104" 2021-02-10T04:26:25.899 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - filesystem" source="node_exporter.go:104" 2021-02-10T04:26:25.899 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - hwmon" source="node_exporter.go:104" 2021-02-10T04:26:25.899 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - infiniband" source="node_exporter.go:104" 2021-02-10T04:26:25.899 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - ipvs" source="node_exporter.go:104" 2021-02-10T04:26:25.899 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - loadavg" source="node_exporter.go:104" 2021-02-10T04:26:25.899 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - mdadm" source="node_exporter.go:104" 2021-02-10T04:26:25.899 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - meminfo" source="node_exporter.go:104" 2021-02-10T04:26:25.900 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - netclass" source="node_exporter.go:104" 2021-02-10T04:26:25.900 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - netdev" source="node_exporter.go:104" 2021-02-10T04:26:25.900 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - netstat" source="node_exporter.go:104" 2021-02-10T04:26:25.900 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - nfs" source="node_exporter.go:104" 2021-02-10T04:26:25.900 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - nfsd" source="node_exporter.go:104" 2021-02-10T04:26:25.900 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - pressure" source="node_exporter.go:104" 2021-02-10T04:26:25.900 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - sockstat" source="node_exporter.go:104" 2021-02-10T04:26:25.900 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - stat" source="node_exporter.go:104" 2021-02-10T04:26:25.900 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - textfile" source="node_exporter.go:104" 2021-02-10T04:26:25.900 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - time" source="node_exporter.go:104" 2021-02-10T04:26:25.900 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - uname" source="node_exporter.go:104" 2021-02-10T04:26:25.900 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - vmstat" source="node_exporter.go:104" 2021-02-10T04:26:25.900 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - xfs" source="node_exporter.go:104" 2021-02-10T04:26:25.900 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg=" - zfs" source="node_exporter.go:104" 2021-02-10T04:26:25.901 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 conmon[56512]: time="2021-02-10T04:26:25Z" level=info msg="Listening on :9100" source="node_exporter.go:170" 2021-02-10T04:26:25.901 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 bash[56105]: d7eb272b6262b1db7a15d101408d5d9f9125aa6495d73e8a65c61394e088d69d 2021-02-10T04:26:25.901 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 04:26:25 smithi085 systemd[1]: Started Ceph node-exporter.a for 42224fce-6b57-11eb-8feb-001a4aab830c. 2021-02-10T04:26:26.461 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-02-10T04:24:05.258028+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2021-02-10T04:25:32.688554+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2021-02-10T04:25:47.658613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'zone.rgw.log', 'create_time': '2021-02-10T04:25:47.968727+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'zone.rgw.control', 'create_time': '2021-02-10T04:25:49.278260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'zone.rgw.meta', 'create_time': '2021-02-10T04:25:51.792514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 8}, 'application_metadata': {'rgw': {}}}] 2021-02-10T04:26:26.461 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pool get device_health_metrics pg_num 2021-02-10T04:26:26.669 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:26:26.940 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:26 smithi085 conmon[29319]: audit 2021-02-10T04:26:25.647214+0000 mon.a (mon.0) 625 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:26.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:26 smithi085 conmon[29319]: cephadm 2021-02-10T04:26:25.651620+0000 mgr.y (mgr.14140) 229 : cephadm [INF] Deploying daemon alertmanager.a on smithi085 2021-02-10T04:26:26.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:26 smithi085 conmon[29319]: audit 2021-02-10T04:26:25.708215+0000 mon.a (mon.0) 626 : audit [DBG] from='client.? 172.21.15.85:0/2708867576' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-02-10T04:26:26.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:26 smithi085 conmon[29319]: cluster 2021-02-10T04:26:25.958509+0000 mgr.y (mgr.14140) 230 : cluster [DBG] pgmap v200: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:26:26.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:26 smithi085 conmon[35961]: audit 2021-02-10T04:26:25.647214+0000 mon.a (mon.0) 625 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:26.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:26 smithi085 conmon[35961]: cephadm 2021-02-10T04:26:25.651620+0000 mgr.y (mgr.14140) 229 : cephadm [INF] Deploying daemon alertmanager.a on smithi085 2021-02-10T04:26:26.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:26 smithi085 conmon[35961]: audit 2021-02-10T04:26:25.708215+0000 mon.a (mon.0) 626 : audit [DBG] from='client.? 172.21.15.85:0/2708867576' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-02-10T04:26:26.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:26 smithi085 conmon[35961]: cluster 2021-02-10T04:26:25.958509+0000 mgr.y (mgr.14140) 230 : cluster [DBG] pgmap v200: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:26:27.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:26 smithi164 conmon[29288]: audit 2021-02-10T04:26:25.647214+0000 mon.a (mon.0) 625 : 2021-02-10T04:26:27.209 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:26 smithi164 conmon[29288]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:27.209 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:26 smithi164 conmon[29288]: cephadm 2021-02-10T04:26:25.651620+0000 mgr.y (mgr.14140) 229 : cephadm [INF] Deploying daemon alertmanager.a on smithi085 2021-02-10T04:26:27.209 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:26 smithi164 conmon[29288]: audit 2021-02-10T04:26:25.708215+0000 mon.a (mon.0) 626 : audit [DBG] from='client.? 172.21.15.85:0/2708867576' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-02-10T04:26:27.209 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:26 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:26:27.209 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:26 smithi164 conmon[29288]: T04:26:25.958509+0000 mgr.y (mgr.14140) 230 : cluster [DBG] 2021-02-10T04:26:27.209 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:26 smithi164 conmon[29288]: pgmap v200: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:26:27.893 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:26:27 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:26:29.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:28 smithi164 conmon[29288]: audit 2021-02-10T04:26:27.546463+0000 mgr.y (mgr.14140) 231 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:26:32.412 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:28 smithi164 conmon[29288]: 2021-02-10T04:26:32.413 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:28 smithi164 conmon[29288]: cluster 2021-02-10T04:26:27.958901+0000 mgr.y (mgr.14140) 232 : cluster [DBG] pgmap v201: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:26:32.413 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:28 smithi164 conmon[29288]: cluster 2021-02-10T04:26:27.982887+0000 mon.a (mon.0) 627 : 2021-02-10T04:26:32.413 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:28 smithi164 conmon[29288]: cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2021-02-10T04:26:32.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:28 smithi085 conmon[29319]: audit 2021-02-10T04:26:32.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:28 smithi085 conmon[29319]: 2021-02-10T04:26:27.546463+0000 mgr.y 2021-02-10T04:26:32.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:28 smithi085 conmon[29319]: (mgr.14140) 231 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:26:32.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:28 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:26:32.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:28 smithi085 conmon[29319]: -02-10T04:26:27.958901+0000 mgr.y (mgr.14140) 232 : cluster [DBG] pgmap v201: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:26:32.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:28 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:26:32.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:28 smithi085 conmon[29319]: 04:26:27.982887+0000 mon.a (mon.0) 627 2021-02-10T04:26:32.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:28 smithi085 conmon[29319]: : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2021-02-10T04:26:32.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:29 smithi085 conmon[29319]: cluster 2021-02-10T04:26:29.762547+0000 osd.0 (osd.0) 7 2021-02-10T04:26:32.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:29 smithi085 conmon[29319]: : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:26:32.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:29 smithi085 conmon[29319]: cluster 2021-02-10T04:26:29.762634+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:26:32.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:30 smithi085 conmon[29319]: cluster 2021-02-10T04:26:29.959359 2021-02-10T04:26:32.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:30 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 233 : cluster [DBG] pgmap v203: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:26:32.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:30 smithi085 conmon[29319]: cluster 2021-02-10T04:26:29.991196+0000 mon.a (mon.0) 628 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2021-02-10T04:26:32.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:30 smithi085 conmon[29319]: audit 2021-02-10T04:26:30.663590+0000 mon.a (mon.0) 629 : audit [DBG] from='client.? 172.21.15.85:0/717078873' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-02-10T04:26:32.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:28 smithi085 conmon[35961]: audit 2021-02-10T04:26:27.546463+0000 mgr.y (mgr.14140 2021-02-10T04:26:32.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:28 smithi085 conmon[35961]: ) 231 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:26:32.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:28 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:26:32.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:28 smithi085 conmon[35961]: 04:26:27.958901+0000 mgr.y (mgr.14140) 232 : cluster [DBG] 2021-02-10T04:26:32.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:28 smithi085 conmon[35961]: pgmap v201: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:26:32.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:28 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:26:32.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:28 smithi085 conmon[35961]: -10T04:26:27.982887 2021-02-10T04:26:32.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:28 smithi085 conmon[35961]: +0000 mon.a (mon.0) 627 : cluster 2021-02-10T04:26:32.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:28 smithi085 conmon[35961]: [DBG] osdmap e67: 8 total, 8 up, 8 in 2021-02-10T04:26:32.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:29 smithi085 conmon[35961]: cluster 2021-02-10T04:26: 2021-02-10T04:26:32.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:29 smithi085 conmon[35961]: 29.762547+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:26:32.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:29 smithi085 conmon[35961]: cluster 2021-02-10T04:26:32.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:29 smithi085 conmon[35961]: 2021-02-10T04:26:29.762634+0000 2021-02-10T04:26:32.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:29 smithi085 conmon[35961]: osd.0 (osd.0) 8 : cluster 2021-02-10T04:26:32.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:29 smithi085 conmon[35961]: [DBG] purged_snaps scrub ok 2021-02-10T04:26:32.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:30 smithi085 conmon[35961]: cluster 2021-02-10T04:26:29.959359+0000 mgr.y (mgr 2021-02-10T04:26:32.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:30 smithi085 conmon[35961]: .14140) 233 : cluster [DBG] pgmap v203: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:26:32.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:30 smithi085 conmon[35961]: cluster 2021-02-10T04:26:29.991196+0000 mon.a (mon.0) 628 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2021-02-10T04:26:32.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:30 smithi085 conmon[35961]: audit 2021-02-10T04:26:30.663590+0000 mon.a (mon.0) 629 : audit [DBG] from='client.? 172.21.15.85:0/717078873' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-02-10T04:26:32.417 INFO:teuthology.orchestra.run.smithi085.stdout:pg_num: 1 2021-02-10T04:26:32.418 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pool get .rgw.root pg_num 2021-02-10T04:26:32.418 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:29 smithi164 conmon[29288]: cluster 2021-02-10T04:26:32.418 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:29 smithi164 conmon[29288]: 2021-02-10T04:26:29.762547+0000 osd.0 2021-02-10T04:26:32.419 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:29 smithi164 conmon[29288]: (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:26:32.419 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:29 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:26:32.419 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:29 smithi164 conmon[29288]: :26:29.762634+0000 osd.0 (osd.0) 2021-02-10T04:26:32.419 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:29 smithi164 conmon[29288]: 8 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:26:32.419 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:30 smithi164 conmon[29288]: cluster 2021-02-10T04:26:29.959359+0000 mgr.y (mgr.14140) 233 : cluster [DBG] pgmap v203: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:26:32.419 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:30 smithi164 conmon[29288]: 2021-02-10T04:26:32.419 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:30 smithi164 conmon[29288]: cluster 2021-02-10T04:26:29.991196+0000 mon.a (mon.0) 628 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2021-02-10T04:26:32.419 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:30 smithi164 conmon[29288]: audit 2021-02-10T04:26:30.663590+0000 mon.a (mon.0) 629 : audit 2021-02-10T04:26:32.419 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:30 smithi164 conmon[29288]: [DBG] from='client.? 172.21.15.85:0/717078873' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-02-10T04:26:32.629 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:26:33.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:32 smithi085 conmon[29319]: cluster 2021-02-10T04:26:31.960145+0000 mgr.y (mgr.14140) 234 : cluster [DBG] pgmap v205: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 642 B/s rd, 0 op/s 2021-02-10T04:26:33.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:32 smithi085 conmon[29319]: cluster 2021-02-10T04:26:32.982850+0000 mon.a (mon.0) 630 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2021-02-10T04:26:33.309 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:32 smithi085 conmon[35961]: cluster 2021-02-10T04:26:31.960145+0000 mgr.y (mgr.14140) 234 : cluster [DBG] pgmap v205: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 642 B/s rd, 0 op/s 2021-02-10T04:26:33.309 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:32 smithi085 conmon[35961]: cluster 2021-02-10T04:26:32.982850+0000 mon.a (mon.0) 630 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2021-02-10T04:26:33.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:32 smithi164 conmon[29288]: cluster 2021-02-10T04:26:31.960145+0000 mgr.y (mgr.14140) 234 : cluster [DBG] pgmap v205: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 642 B/s rd, 0 op/s 2021-02-10T04:26:33.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:32 smithi164 conmon[29288]: cluster 2021-02-10T04:26:32.982850+0000 mon.a (mon.0) 630 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2021-02-10T04:26:33.691 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 04:26:33 smithi085 systemd[1]: Starting Ceph alertmanager.a for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T04:26:34.940 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 04:26:34 smithi085 conmon[57535]: level=info ts=2021-02-10T04:26:34.516Z caller=main.go:231 msg="Starting Alertmanager" version="(version=0.20.0, branch=HEAD, revision=f74be0400a6243d10bb53812d6fa408ad71ff32d)" 2021-02-10T04:26:34.940 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 04:26:34 smithi085 conmon[57535]: level=info ts=2021-02-10T04:26:34.516Z caller=main.go:232 build_context="(go=go1.13.5, user=root@00c3106655f8, date=20191211-14:13:14)" 2021-02-10T04:26:34.941 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 04:26:34 smithi085 conmon[57535]: level=info ts=2021-02-10T04:26:34.517Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=172.21.15.85 port=9094 2021-02-10T04:26:34.941 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 04:26:34 smithi085 conmon[57535]: level=info ts=2021-02-10T04:26:34.519Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s 2021-02-10T04:26:34.941 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 04:26:34 smithi085 conmon[57535]: level=info ts=2021-02-10T04:26:34.537Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2021-02-10T04:26:34.941 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 04:26:34 smithi085 conmon[57535]: level=info ts=2021-02-10T04:26:34.537Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2021-02-10T04:26:34.941 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 04:26:34 smithi085 conmon[57535]: level=info ts=2021-02-10T04:26:34.540Z caller=main.go:497 msg=Listening address=:9093 2021-02-10T04:26:34.941 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 04:26:34 smithi085 bash[57401]: 3ea232dacd720787cb635c5023c5f355b7982d63b63abafd9e4cd91327b7fbdd 2021-02-10T04:26:34.941 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 04:26:34 smithi085 systemd[1]: Started Ceph alertmanager.a for 42224fce-6b57-11eb-8feb-001a4aab830c. 2021-02-10T04:26:35.308 INFO:teuthology.orchestra.run.smithi085.stdout:pg_num: 32 2021-02-10T04:26:35.330 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:34 smithi085 conmon[29319]: cluster 2021-02-10T04:26:33.960920+0000 mgr.y (mgr.14140) 235 : cluster [DBG] pgmap v207: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2021-02-10T04:26:35.330 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:34 smithi085 conmon[29319]: audit 2021-02-10T04:26:34.736809+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:35.330 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:34 smithi085 conmon[29319]: audit 2021-02-10T04:26:34.830963+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:35.330 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:34 smithi085 conmon[29319]: audit 2021-02-10T04:26:34.836183+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:35.330 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:34 smithi085 conmon[29319]: audit 2021-02-10T04:26:34.836746+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-02-10T04:26:35.330 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:34 smithi085 conmon[29319]: audit 2021-02-10T04:26:34.840330+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:35.330 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:34 smithi085 conmon[35961]: cluster 2021-02-10T04:26:33.960920+0000 mgr.y (mgr.14140) 235 : cluster [DBG] pgmap v207: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2021-02-10T04:26:35.330 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:34 smithi085 conmon[35961]: audit 2021-02-10T04:26:34.736809+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:35.331 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:34 smithi085 conmon[35961]: audit 2021-02-10T04:26:34.830963+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:35.331 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:34 smithi085 conmon[35961]: audit 2021-02-10T04:26:34.836183+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:35.331 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:34 smithi085 conmon[35961]: audit 2021-02-10T04:26:34.836746+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-02-10T04:26:35.331 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:34 smithi085 conmon[35961]: audit 2021-02-10T04:26:34.840330+0000 mon.a (mon.0) 635 : audit [INF] 2021-02-10T04:26:35.331 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:34 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:35.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:34 smithi164 conmon[29288]: cluster 2021-02-10T04:26:33.960920+0000 mgr.y (mgr.14140) 235 : cluster 2021-02-10T04:26:35.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:34 smithi164 conmon[29288]: [DBG] pgmap v207: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2021-02-10T04:26:35.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:34 smithi164 conmon[29288]: audit 2021-02-10T04:26:34.736809+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:35.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:34 smithi164 conmon[29288]: audit 2021-02-10T04:26:34.830963+0000 mon.a (mon.0) 632 : audit 2021-02-10T04:26:35.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:34 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:35.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:34 smithi164 conmon[29288]: audit 2021-02-10T04:26:34.836183 2021-02-10T04:26:35.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:34 smithi164 conmon[29288]: +0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:35.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:34 smithi164 conmon[29288]: audit 2021-02-10T04:26:35.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:34 smithi164 conmon[29288]: 2021-02-10T04:26:34.836746+0000 mon.a (mon.0) 634 : audit 2021-02-10T04:26:35.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:34 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-02-10T04:26:35.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:34 smithi164 conmon[29288]: audit 2021-02-10T04:26:35.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:34 smithi164 conmon[29288]: 2021-02-10T04:26:34.840330+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:35.997 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pool get iscsi pg_num 2021-02-10T04:26:36.200 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:26:36.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:36 smithi164 conmon[29288]: cluster 2021-02-10T04:26:34.066142+0000 osd.4 (osd.4) 5 : cluster [DBG] 2.f deep-scrub ok 2021-02-10T04:26:36.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:36 smithi164 conmon[29288]: audit 2021-02-10T04:26:34.837170+0000 2021-02-10T04:26:36.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:36 smithi164 conmon[29288]: mgr.y (mgr.14140) 236 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-02-10T04:26:36.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:36 smithi164 conmon[29288]: cephadm 2021-02-10T04:26:34.844394+0000 mgr.y (mgr.14140) 237 : cephadm [INF] Deploying daemon grafana.a on smithi164 2021-02-10T04:26:36.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:36 smithi164 conmon[29288]: cluster 2021-02-10T04:26:34.972945+0000 osd.7 (osd.7) 3 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:26:36.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:36 smithi164 conmon[29288]: cluster 2021-02-10T04:26:34.973032+0000 osd.7 (osd.7) 4 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:26:36.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:36 smithi164 conmon[29288]: audit 2021-02-10T04:26:35.307882+0000 mon.a (mon.0) 636 : audit [DBG] from='client.? 172.21.15.85:0/958246118' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]: dispatch 2021-02-10T04:26:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:36 smithi164 conmon[29288]: cluster 2021-02-10T04:26:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:36 smithi164 conmon[29288]: 2021-02-10T04:26:35.842604+0000 mon.a (mon.0) 637 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2021-02-10T04:26:36.435 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:35 smithi085 conmon[29319]: cluster 2021-02-10T04:26:34.066142+0000 osd.4 (osd.4) 5 : cluster 2021-02-10T04:26:36.435 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:35 smithi085 conmon[29319]: [DBG] 2.f deep-scrub ok 2021-02-10T04:26:36.436 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:36 smithi085 conmon[29319]: audit 2021-02-10T04:26:34.837170+0000 mgr.y (mgr.14140) 236 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-02-10T04:26:36.436 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:36 smithi085 conmon[29319]: cephadm 2021-02-10T04:26:34.844394+0000 mgr.y (mgr.14140) 237 : cephadm [INF] Deploying daemon grafana.a on smithi164 2021-02-10T04:26:36.436 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:36 smithi085 conmon[29319]: cluster 2021-02-10T04:26:34.972945+0000 osd.7 (osd.7) 3 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:26:36.436 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:36 smithi085 conmon[29319]: cluster 2021-02-10T04:26:34.973032+0000 osd.7 (osd.7) 4 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:26:36.436 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:36 smithi085 conmon[29319]: audit 2021-02-10T04:26:35.307882+0000 mon.a (mon.0) 636 : audit [DBG] from='client.? 172.21.15.85:0/958246118' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]: dispatch 2021-02-10T04:26:36.436 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:36 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:26:36.436 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:36 smithi085 conmon[29319]: -10T04:26:35.842604+0000 2021-02-10T04:26:36.436 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:36 smithi085 conmon[29319]: mon.a (mon.0) 637 : 2021-02-10T04:26:36.436 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:36 smithi085 conmon[29319]: cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2021-02-10T04:26:36.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:36 smithi085 conmon[35961]: cluster 2021-02-10T04:26:34.066142+0000 2021-02-10T04:26:36.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:36 smithi085 conmon[35961]: osd.4 (osd.4) 5 : cluster [DBG] 2.f deep-scrub ok 2021-02-10T04:26:36.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:36 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:26:36.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:36 smithi085 conmon[35961]: 04:26:34.837170+0000 mgr.y (mgr.14140) 236 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-02-10T04:26:36.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:36 smithi085 conmon[35961]: cephadm 2021-02-10T04:26:34. 2021-02-10T04:26:36.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:36 smithi085 conmon[35961]: 844394+0000 mgr.y (mgr.14140) 237 : cephadm [INF] Deploying daemon grafana.a on smithi164 2021-02-10T04:26:36.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:36 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:26:36.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:36 smithi085 conmon[35961]: 26:34.972945+0000 osd.7 (osd.7) 3 : cluster [DBG] 2021-02-10T04:26:36.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:36 smithi085 conmon[35961]: purged_snaps scrub starts 2021-02-10T04:26:36.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:36 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:26:36.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:36 smithi085 conmon[35961]: -10T04:26:34.973032+0000 osd.7 (osd.7) 4 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:26:36.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:36 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:26:36.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:36 smithi085 conmon[35961]: :26:35.307882+0000 mon.a (mon.0) 636 : audit 2021-02-10T04:26:36.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:36 smithi085 conmon[35961]: [DBG] from='client.? 172.21.15.85:0/958246118' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]: dispatch 2021-02-10T04:26:36.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:36 smithi085 conmon[35961]: cluster 2021-02-10T04:26:36.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:36 smithi085 conmon[35961]: 2021-02-10T04:26:35.842604+0000 mon.a (mon 2021-02-10T04:26:36.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:36 smithi085 conmon[35961]: .0) 637 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2021-02-10T04:26:36.690 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 04:26:36 smithi085 conmon[57535]: level=info ts=2021-02-10T04:26:36.519Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000093631s 2021-02-10T04:26:37.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:37 smithi164 conmon[29288]: cluster 2021-02-10T04:26:35.961321+0000 mgr.y (mgr.14140) 238 : cluster [DBG] pgmap v209: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:26:37.404 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:37 smithi085 conmon[29319]: cluster 2021-02-10T04:26:35.961321+0000 mgr.y (mgr.14140) 238 : cluster [DBG] 2021-02-10T04:26:37.405 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:37 smithi085 conmon[29319]: pgmap v209: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:26:37.405 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:37 smithi085 conmon[35961]: cluster 2021-02-10T04:26:35.961321+0000 mgr.y 2021-02-10T04:26:37.405 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:37 smithi085 conmon[35961]: (mgr.14140) 238 : cluster [DBG] pgmap v209: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:26:37.893 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:26:37 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:26:38.017 INFO:teuthology.orchestra.run.smithi085.stdout:pg_num: 3 2021-02-10T04:26:38.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: audit 2021-02-10T04:26:37.920433+0000 mon.a (mon.0) 2021-02-10T04:26:38.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: 638 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2021-02-10T04:26:38.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: audit 2021-02-10T04:26:37.920775+0000 mon.a (mon.0) 639 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]: dispatch 2021-02-10T04:26:38.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: audit 2021-02-10T04:26:38.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: 2021-02-10T04:26:37.921061+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2021-02-10T04:26:38.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:26:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: 02-10T04:26:37.921333+0000 mon.a 2021-02-10T04:26:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: (mon.0) 641 : audit [INF] 2021-02-10T04:26:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2021-02-10T04:26:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:26:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: -10T04:26:37.986330 2021-02-10T04:26:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: +0000 mon.a (mon.0) 642 2021-02-10T04:26:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]': finished 2021-02-10T04:26:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: audit 2021-02-10T04:26:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:26:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: :26:37.986399+0000 mon.a 2021-02-10T04:26:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: (mon.0) 643 : audit 2021-02-10T04:26:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]': finished 2021-02-10T04:26:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: audit 2021-02-10T04:26:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: 2021-02-10T04:26 2021-02-10T04:26:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: :37.986464+0000 mon.a (mon 2021-02-10T04:26:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: .0) 644 : audit [INF] 2021-02-10T04:26:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2021-02-10T04:26:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: audit 2021-02-10T04:26:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: 2021-02-10T04:26: 2021-02-10T04:26:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: 37.986536+0000 mon.a (mon.0) 645 : audit 2021-02-10T04:26:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2021-02-10T04:26:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:26:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: -02-10T04:26:37 2021-02-10T04:26:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: .986612+0000 mon.a (mon.0) 2021-02-10T04:26:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:38 smithi164 conmon[29288]: 646 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2021-02-10T04:26:38.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:38 smithi085 conmon[29319]: audit 2021-02-10T04:26:37.920433+0000 mon.a (mon.0) 638 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2021-02-10T04:26:38.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:38 smithi085 conmon[29319]: audit 2021-02-10T04:26:37.920775+0000 mon.a (mon.0) 639 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]: dispatch 2021-02-10T04:26:38.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:38 smithi085 conmon[29319]: audit 2021-02-10T04:26:37.921061+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2021-02-10T04:26:38.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:38 smithi085 conmon[29319]: audit 2021-02-10T04:26:37.921333+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2021-02-10T04:26:38.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:38 smithi085 conmon[29319]: audit 2021-02-10T04:26:37.986330+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]': finished 2021-02-10T04:26:38.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:38 smithi085 conmon[29319]: audit 2021-02-10T04:26:37.986399+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]': finished 2021-02-10T04:26:38.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:38 smithi085 conmon[29319]: audit 2021-02-10T04:26:37.986464+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2021-02-10T04:26:38.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:38 smithi085 conmon[29319]: audit 2021-02-10T04:26:37.986536+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2021-02-10T04:26:38.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:38 smithi085 conmon[29319]: cluster 2021-02-10T04:26:37.986612+0000 mon.a (mon.0) 646 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2021-02-10T04:26:38.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:38 smithi085 conmon[35961]: audit 2021-02-10T04:26:37.920433+0000 mon.a (mon.0) 638 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2021-02-10T04:26:38.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:38 smithi085 conmon[35961]: audit 2021-02-10T04:26:37.920775+0000 mon.a (mon.0) 639 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]: dispatch 2021-02-10T04:26:38.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:38 smithi085 conmon[35961]: audit 2021-02-10T04:26:37.921061+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2021-02-10T04:26:38.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:38 smithi085 conmon[35961]: audit 2021-02-10T04:26:37.921333+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2021-02-10T04:26:38.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:38 smithi085 conmon[35961]: audit 2021-02-10T04:26:37.986330+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]': finished 2021-02-10T04:26:38.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:38 smithi085 conmon[35961]: audit 2021-02-10T04:26:37.986399+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]': finished 2021-02-10T04:26:38.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:38 smithi085 conmon[35961]: audit 2021-02-10T04:26:37.986464+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2021-02-10T04:26:38.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:38 smithi085 conmon[35961]: audit 2021-02-10T04:26:37.986536+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2021-02-10T04:26:38.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:38 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:26:38.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:38 smithi085 conmon[35961]: 10T04:26:37.986612+0000 mon.a (mon.0) 646 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2021-02-10T04:26:38.699 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pool get zone.rgw.log pg_num 2021-02-10T04:26:38.895 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:26:39.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:39 smithi164 conmon[29288]: audit 2021-02-10T04:26:37.555602+0000 mgr.y (mgr.14140) 2021-02-10T04:26:39.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:39 smithi164 conmon[29288]: 239 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:26:39.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:39 smithi164 conmon[29288]: cluster 2021-02-10T04:26:37.961783+0000 mgr.y (mgr.14140) 240 : cluster [DBG] pgmap v210: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 642 B/s rd, 0 op/s 2021-02-10T04:26:39.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:39 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:26:39.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:39 smithi164 conmon[29288]: 26:38.013765+0000 osd.5 (osd.5) 5 : cluster [DBG] 2.3 scrub ok 2021-02-10T04:26:39.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:39 smithi164 conmon[29288]: audit 2021-02-10T04:26:38. 2021-02-10T04:26:39.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:39 smithi164 conmon[29288]: 018357+0000 mon.c (mon.1) 18 : audit [DBG] 2021-02-10T04:26:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:39 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/4176610361' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"}]: dispatch 2021-02-10T04:26:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:39 smithi164 conmon[29288]: audit 2021-02-10T04:26: 2021-02-10T04:26:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:39 smithi164 conmon[29288]: 38.033472+0000 mon.a (mon.0) 647 : 2021-02-10T04:26:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:39 smithi164 conmon[29288]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:26:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:39 smithi164 conmon[29288]: audit 2021-02-10T04:26 2021-02-10T04:26:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:39 smithi164 conmon[29288]: :38.034559+0000 mon.a (mon.0) 648 2021-02-10T04:26:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:39 smithi164 conmon[29288]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:26:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:39 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:26:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:39 smithi164 conmon[29288]: :26:38.432974+0000 osd.0 (osd.0 2021-02-10T04:26:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:39 smithi164 conmon[29288]: ) 9 : cluster [DBG] 2.13 scrub ok 2021-02-10T04:26:39.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:39 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:26:39.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:39 smithi164 conmon[29288]: T04:26:38.996914+0000 mon.a (mon 2021-02-10T04:26:39.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:39 smithi164 conmon[29288]: .0) 649 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2021-02-10T04:26:39.405 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[35961]: audit 2021-02-10T04:26:37.555602+0000 mgr.y 2021-02-10T04:26:39.405 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[35961]: (mgr.14140) 239 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:26:39.405 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[35961]: cluster 2021-02-10T04:26:37.961783+0000 mgr.y (mgr.14140) 240 : cluster [DBG] pgmap v210: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 642 B/s rd, 0 op/s 2021-02-10T04:26:39.405 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[35961]: cluster 2021-02-10T04:26:38.013765+0000 osd.5 (osd.5) 5 : cluster [DBG] 2.3 scrub ok 2021-02-10T04:26:39.405 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[35961]: audit 2021-02-10T04:26:38.018357+0000 mon.c (mon.1) 18 : audit [DBG] from='client.? 172.21.15.85:0/4176610361' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"}]: dispatch 2021-02-10T04:26:39.406 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[35961]: audit 2021-02-10T04:26:38.033472+0000 mon.a (mon.0) 647 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:26:39.406 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[35961]: audit 2021-02-10T04:26:38.034559+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:26:39.406 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[35961]: cluster 2021-02-10T04:26:38.432974+0000 osd.0 (osd.0) 9 : cluster [DBG] 2.13 scrub ok 2021-02-10T04:26:39.406 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[35961]: cluster 2021-02-10T04:26:38.996914+0000 mon.a (mon.0) 649 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2021-02-10T04:26:39.406 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[29319]: audit 2021-02-10T04:26:39.406 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[29319]: 2021-02-10T04:26 2021-02-10T04:26:39.406 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[29319]: :37.555602+0000 mgr.y (mgr.14140) 239 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:26:39.406 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:26:39.407 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[29319]: 10T04:26:37.961783+0000 mgr.y 2021-02-10T04:26:39.407 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[29319]: (mgr.14140) 240 : cluster [DBG] pgmap v210: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 642 B/s rd, 0 op/s 2021-02-10T04:26:39.407 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[29319]: 2021-02-10T04:26:39.407 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:26:39.407 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[29319]: -02-10T04:26:38.013765+0000 osd.5 (osd 2021-02-10T04:26:39.407 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[29319]: .5) 5 : cluster [DBG] 2.3 scrub ok 2021-02-10T04:26:39.407 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[29319]: audit 2021 2021-02-10T04:26:39.407 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[29319]: -02-10T04:26:38.018357 2021-02-10T04:26:39.407 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[29319]: +0000 mon.c (mon.1) 18 : audit 2021-02-10T04:26:39.407 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[29319]: [DBG] from='client.? 172.21.15.85:0/4176610361' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"}]: dispatch 2021-02-10T04:26:39.408 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[29319]: audit 2021-02-10T04:26:39.408 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[29319]: 2021-02-10T04:26:38.033472+0000 mon.a (mon. 2021-02-10T04:26:39.408 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[29319]: 0) 647 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:26:39.408 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:26:39.408 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[29319]: 10T04:26:38.034559+0000 2021-02-10T04:26:39.408 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[29319]: mon.a (mon.0) 648 : audit [INF] 2021-02-10T04:26:39.408 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:26:39.408 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[29319]: cluster 2021-02-10T04:26:39.408 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[29319]: 2021-02-10T04:26:38.432974+0000 2021-02-10T04:26:39.409 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[29319]: osd.0 (osd.0) 9 : cluster [DBG] 2021-02-10T04:26:39.409 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[29319]: 2.13 scrub ok 2021-02-10T04:26:39.409 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[29319]: cluster 2021-02-10T04:26:39.409 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[29319]: 2021-02-10T04:26:38.996914+0000 2021-02-10T04:26:39.409 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[29319]: mon.a (mon.0) 649 : cluster [DBG] 2021-02-10T04:26:39.409 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[29319]: osdmap e72: 8 total, 8 up, 8 in 2021-02-10T04:26:39.409 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:26:39 smithi085 conmon[41540]: debug 2021-02-10T04:26:39.403+0000 7fb94ab61700 -1 osd.0 pg_epoch: 72 pg[2.1f( empty local-lis/les=50/51 n=0 ec=48/48 lis/c=50/50 les/c/f=51/51/0 sis=50) [0,7,4] r=0 lpr=50 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.1f) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 4 (0x555d3284e000) 2021-02-10T04:26:40.748 INFO:teuthology.orchestra.run.smithi085.stdout:pg_num: 32 2021-02-10T04:26:41.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:41 smithi085 conmon[29319]: cluster 2021-02-10T04:26:39.962290+0000 mgr.y (mgr.14140) 241 : cluster [DBG] pgmap v213: 132 pgs: 1 peering, 131 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:26:41.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:41 smithi085 conmon[29319]: cluster 2021-02-10T04:26:40.302250+0000 osd.4 (osd.4) 6 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:26:41.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:41 smithi085 conmon[29319]: cluster 2021-02-10T04:26:40.302370+0000 osd.4 (osd.4) 7 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:26:41.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:41 smithi085 conmon[29319]: audit 2021-02-10T04:26:40.748979+0000 mon.c (mon.1) 19 : audit [DBG] from='client.? 172.21.15.85:0/3666568578' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "zone.rgw.log", "var": "pg_num"}]: dispatch 2021-02-10T04:26:41.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:41 smithi085 conmon[35961]: cluster 2021-02-10T04:26:39.962290+0000 mgr.y (mgr.14140) 241 : cluster [DBG] pgmap v213: 132 pgs: 1 peering, 131 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:26:41.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:41 smithi085 conmon[35961]: cluster 2021-02-10T04:26:40.302250+0000 osd.4 (osd.4) 6 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:26:41.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:41 smithi085 conmon[35961]: cluster 2021-02-10T04:26:40.302370+0000 osd.4 (osd.4) 7 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:26:41.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:41 smithi085 conmon[35961]: audit 2021-02-10T04:26:40.748979+0000 mon.c (mon.1) 19 : audit [DBG] from='client.? 172.21.15.85:0/3666568578' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "zone.rgw.log", "var": "pg_num"}]: dispatch 2021-02-10T04:26:41.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:41 smithi164 conmon[29288]: cluster 2021-02-10T04:26:39.962290+0000 mgr.y (mgr.14140) 2021-02-10T04:26:41.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:41 smithi164 conmon[29288]: 241 : cluster [DBG] pgmap v213: 132 pgs: 1 peering, 131 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:26:41.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:41 smithi164 conmon[29288]: cluster 2021-02-10T04:26:40.302250+0000 osd.4 (osd.4) 6 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:26:41.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:41 smithi164 conmon[29288]: cluster 2021-02-10T04:26: 2021-02-10T04:26:41.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:41 smithi164 conmon[29288]: 40.302370+0000 osd.4 (osd.4) 7 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:26:41.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:41 smithi164 conmon[29288]: audit 2021-02-10T04:26:40 2021-02-10T04:26:41.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:41 smithi164 conmon[29288]: .748979+0000 mon.c (mon.1) 19 : audit 2021-02-10T04:26:41.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:41 smithi164 conmon[29288]: [DBG] from='client.? 172.21.15.85:0/3666568578' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "zone.rgw.log", "var": "pg_num"}]: dispatch 2021-02-10T04:26:41.400 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pool get zone.rgw.control pg_num 2021-02-10T04:26:41.594 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:26:42.122 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:42 smithi085 conmon[29319]: cluster 2021-02-10T04:26:41.017790+0000 mon.a (mon. 2021-02-10T04:26:42.122 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:42 smithi085 conmon[29319]: 0) 650 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2021-02-10T04:26:42.122 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:42 smithi085 conmon[35961]: cluster 2021-02-10T04:26:41.017790+0000 mon.a ( 2021-02-10T04:26:42.123 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:42 smithi085 conmon[35961]: mon.0) 650 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2021-02-10T04:26:42.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:42 smithi164 conmon[29288]: cluster 2021-02-10T04:26:41.017790+0000 mon.a (mon. 2021-02-10T04:26:42.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:42 smithi164 conmon[29288]: 0) 650 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2021-02-10T04:26:43.019 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:43 smithi085 conmon[29319]: cluster 2021-02-10T04:26:41.962665+0000 mgr.y (mgr.14140) 242 : cluster [DBG] 2021-02-10T04:26:43.020 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:43 smithi085 conmon[29319]: pgmap v215: 132 pgs: 1 peering, 131 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 338 B/s, 4 objects/s recovering 2021-02-10T04:26:43.043 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:43 smithi085 conmon[29319]: cluster 2021-02-10T04:26:42.985412+0000 mon.a (mon.0) 651 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2021-02-10T04:26:43.044 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:43 smithi085 conmon[35961]: cluster 2021-02-10T04:26:41.962665+0000 mgr.y (mgr.14140) 242 2021-02-10T04:26:43.044 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:43 smithi085 conmon[35961]: : cluster [DBG] pgmap v215: 132 pgs: 1 peering, 131 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 338 B/s, 4 objects/s recovering 2021-02-10T04:26:43.044 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:43 smithi085 conmon[35961]: cluster 2021-02-10T04:26:42.985412+0000 mon.a (mon.0) 651 2021-02-10T04:26:43.044 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:43 smithi085 conmon[35961]: : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2021-02-10T04:26:43.369 INFO:teuthology.orchestra.run.smithi085.stdout:pg_num: 32 2021-02-10T04:26:43.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:43 smithi164 conmon[29288]: cluster 2021-02-10T04:26:41.962665+0000 mgr.y (mgr.14140) 2021-02-10T04:26:43.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:43 smithi164 conmon[29288]: 242 : cluster [DBG] pgmap v215: 132 pgs: 1 peering, 131 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 338 B/s, 4 objects/s recovering 2021-02-10T04:26:43.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:43 smithi164 conmon[29288]: cluster 2021-02-10T04:26:42.985412+0000 mon.a (mon.0) 651 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2021-02-10T04:26:44.067 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pool get zone.rgw.meta pg_num 2021-02-10T04:26:44.262 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:26:44.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:44 smithi164 conmon[29288]: cluster 2021-02-10T04:26:42.261915+0000 osd.7 (osd.7 2021-02-10T04:26:44.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:44 smithi164 conmon[29288]: ) 5 : cluster [DBG] 2.1d scrub ok 2021-02-10T04:26:44.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:44 smithi164 conmon[29288]: cluster 2021-02-10T04:26:42.627251+0000 osd.2 (osd.2) 7 : cluster [DBG] 2.1 scrub ok 2021-02-10T04:26:44.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:44 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:26:44.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:44 smithi164 conmon[29288]: 02-10T04:26:43.370102+0000 mon.a (mon.0) 652 : audit [DBG] from='client.? 172.21.15.85:0/2112207646' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "zone.rgw.control", "var": "pg_num"}]: dispatch 2021-02-10T04:26:44.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:44 smithi085 conmon[29319]: cluster 2021-02-10T04:26:42.261915+0000 osd.7 (osd.7 2021-02-10T04:26:44.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:44 smithi085 conmon[29319]: ) 5 : cluster [DBG] 2.1d scrub ok 2021-02-10T04:26:44.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:44 smithi085 conmon[29319]: cluster 2021-02-10T04:26:42.627251+0000 osd.2 (osd.2) 7 : cluster [DBG] 2.1 scrub ok 2021-02-10T04:26:44.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:44 smithi085 conmon[29319]: audit 2021-02-10T04:26:43.370102+0000 mon.a (mon.0) 652 : audit [DBG] from='client.? 172.21.15.85:0/2112207646' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "zone.rgw.control", "var": "pg_num"}]: dispatch 2021-02-10T04:26:44.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:44 smithi085 conmon[35961]: cluster 2021-02-10T04:26:42.261915+0000 osd.7 2021-02-10T04:26:44.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:44 smithi085 conmon[35961]: (osd.7) 5 : cluster [DBG] 2.1d scrub ok 2021-02-10T04:26:44.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:44 smithi085 conmon[35961]: cluster 2021-02-10T04:26:42. 2021-02-10T04:26:44.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:44 smithi085 conmon[35961]: 627251+0000 osd.2 (osd.2) 7 : cluster [DBG] 2.1 scrub ok 2021-02-10T04:26:44.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:44 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:26:44.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:44 smithi085 conmon[35961]: 02-10T04:26:43. 2021-02-10T04:26:44.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:44 smithi085 conmon[35961]: 370102+0000 mon.a (mon.0 2021-02-10T04:26:44.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:44 smithi085 conmon[35961]: ) 652 : audit [DBG] from='client.? 172.21.15.85:0/2112207646' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "zone.rgw.control", "var": "pg_num"}]: dispatch 2021-02-10T04:26:44.811 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 04:26:44 smithi085 conmon[57535]: level=info ts=2021-02-10T04:26:44.519Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.000641329s 2021-02-10T04:26:45.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:45 smithi085 conmon[29319]: cluster 2021-02-10T04:26:43.963195+0000 mgr.y (mgr.14140) 2021-02-10T04:26:45.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:45 smithi085 conmon[29319]: 243 : cluster [DBG] pgmap v217: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 340 B/s, 4 objects/s recovering 2021-02-10T04:26:45.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:45 smithi085 conmon[29319]: cluster 2021-02-10T04:26:44.264395+0000 osd.6 (osd.6) 6 : cluster [DBG] 2.7 scrub ok 2021-02-10T04:26:45.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:45 smithi085 conmon[29319]: cluster 2021-02-10T04:26:44.383069+0000 osd.0 (osd.0) 10 : cluster [DBG] 2.1f scrub ok 2021-02-10T04:26:45.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:45 smithi085 conmon[35961]: cluster 2021-02-10T04:26:43.963195+0000 mgr.y 2021-02-10T04:26:45.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:45 smithi085 conmon[35961]: (mgr.14140) 243 : cluster [DBG] pgmap v217: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 340 B/s, 4 objects/s recovering 2021-02-10T04:26:45.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:45 smithi085 conmon[35961]: cluster 2021-02-10T04:26:44.264395+0000 2021-02-10T04:26:45.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:45 smithi085 conmon[35961]: osd.6 (osd.6) 6 : cluster [DBG] 2.7 scrub ok 2021-02-10T04:26:45.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:45 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:26:45.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:45 smithi085 conmon[35961]: :26:44.383069+0000 osd.0 (osd.0 2021-02-10T04:26:45.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:45 smithi085 conmon[35961]: ) 10 : cluster [DBG] 2.1f scrub ok 2021-02-10T04:26:45.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:45 smithi164 conmon[29288]: cluster 2021-02-10T04:26:43.963195+0000 mgr.y (mgr.14140) 243 : cluster 2021-02-10T04:26:45.633 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:45 smithi164 conmon[29288]: [DBG] pgmap v217: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 340 B/s, 4 objects/s recovering 2021-02-10T04:26:45.634 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:45 smithi164 conmon[29288]: cluster 2021-02-10T04:26:44.264395+0000 osd.6 (osd.6) 6 : cluster [DBG] 2.7 scrub ok 2021-02-10T04:26:45.634 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:45 smithi164 conmon[29288]: cluster 2021-02-10T04:26:44.383069+0000 osd.0 (osd. 2021-02-10T04:26:45.634 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:45 smithi164 conmon[29288]: 0) 10 : cluster [DBG] 2.1f scrub ok 2021-02-10T04:26:46.050 INFO:teuthology.orchestra.run.smithi085.stdout:pg_num: 32 2021-02-10T04:26:46.287 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:46 smithi085 conmon[29319]: cluster 2021-02-10T04:26:44.252343+0000 osd.7 (osd.7 2021-02-10T04:26:46.287 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:46 smithi085 conmon[29319]: ) 6 : cluster [DBG] 1.0 scrub ok 2021-02-10T04:26:46.287 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:46 smithi085 conmon[29319]: cluster 2021-02-10T04:26:44.609918+0000 osd.2 (osd.2) 8 : cluster [DBG] 2.e scrub ok 2021-02-10T04:26:46.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:46 smithi085 conmon[35961]: cluster 2021-02-10T04:26:44.252343+0000 osd.7 (osd. 2021-02-10T04:26:46.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:46 smithi085 conmon[35961]: 7) 6 : cluster [DBG] 1.0 scrub ok 2021-02-10T04:26:46.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:46 smithi085 conmon[35961]: cluster 2021-02-10T04:26:44.609918+0000 osd.2 (osd.2) 8 : cluster [DBG] 2.e scrub ok 2021-02-10T04:26:46.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:46 smithi164 conmon[29288]: cluster 2021-02-10T04:26:44.252343+0000 osd.7 (osd.7) 6 : 2021-02-10T04:26:46.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:46 smithi164 conmon[29288]: cluster [DBG] 1.0 scrub ok 2021-02-10T04:26:46.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:46 smithi164 conmon[29288]: cluster 2021-02-10T04:26:44.609918+0000 osd.2 (osd.2) 8 : cluster [DBG] 2.e scrub ok 2021-02-10T04:26:46.716 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2021-02-10T04:26:46.716 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2021-02-10T04:26:46.716 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph mgr dump --format=json 2021-02-10T04:26:46.912 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:26:47.031 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[35961]: cluster 2021-02-10T04:26:45.223596+0000 2021-02-10T04:26:47.032 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[35961]: osd.7 (osd.7) 7 : cluster [DBG] 2.1c scrub ok 2021-02-10T04:26:47.032 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:26:47.032 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[35961]: 02-10T04:26:45.963711+0000 mgr.y (mgr.14140) 244 : cluster [DBG] 2021-02-10T04:26:47.032 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[35961]: pgmap v218: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 734 B/s rd, 0 op/s; 291 B/s, 3 objects/s recovering 2021-02-10T04:26:47.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:47 smithi164 conmon[29288]: cluster 2021-02-10T04:26:45.223596+0000 osd.7 (osd.7 2021-02-10T04:26:47.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:47 smithi164 conmon[29288]: ) 7 : cluster [DBG] 2.1c scrub ok 2021-02-10T04:26:47.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:47 smithi164 conmon[29288]: cluster 2021-02-10T04:26:45.963711+0000 mgr.y (mgr.14140) 244 : cluster [DBG] pgmap v218: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 734 B/s rd, 0 op/s; 291 B/s, 3 objects/s recovering 2021-02-10T04:26:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:47 smithi164 conmon[29288]: audit 2021-02-10T04:26:46.050219+0000 mon.a (mon.0) 653 : audit [DBG] from='client.? 172.21.15.85:0/226339202' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "zone.rgw.meta", "var": "pg_num"}]: dispatch 2021-02-10T04:26:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:47 smithi164 conmon[29288]: cluster 2021-02-10T04:26:46.092008+0000 osd.1 (osd.1) 7 : cluster [DBG] 2.9 scrub ok 2021-02-10T04:26:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:47 smithi164 conmon[29288]: cluster 2021-02-10T04:26:46.131125+0000 osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:26:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:47 smithi164 conmon[29288]: cluster 2021-02-10T04:26:46.131214+0000 osd.1 (osd.1) 9 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:26:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:47 smithi164 conmon[29288]: cluster 2021-02-10T04:26:46.425283+0000 osd.3 (osd.3) 7 : cluster [DBG] 2.1b deep-scrub ok 2021-02-10T04:26:47.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[29319]: cluster 2021-02-10T04:26:47.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[29319]: 2021-02 2021-02-10T04:26:47.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[29319]: -10T04:26:45.223596+0000 osd.7 (osd.7) 7 : cluster [DBG] 2.1c scrub ok 2021-02-10T04:26:47.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[29319]: cluster 2021-02-10T04:26:45.963711+0000 mgr.y (mgr.14140) 244 : cluster [DBG] pgmap v218: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 734 B/s rd, 0 op/s; 291 B/s, 3 objects/s recovering 2021-02-10T04:26:47.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[29319]: audit 2021-02-10T04:26:46.050219+0000 mon.a (mon.0) 653 : audit [DBG] from='client.? 172.21.15.85:0/226339202' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "zone.rgw.meta", "var": "pg_num"}]: dispatch 2021-02-10T04:26:47.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[29319]: cluster 2021-02-10T04:26:46.092008+0000 osd.1 ( 2021-02-10T04:26:47.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[29319]: osd.1) 7 : cluster [DBG] 2.9 scrub ok 2021-02-10T04:26:47.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:26:47.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[29319]: 26:46.131125+0000 osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:26:47.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[29319]: 2021-02-10T04:26:47.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[29319]: cluster 2021-02-10T04:26:47.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[29319]: 2021-02-10T04:26:46.131214+0000 osd.1 ( 2021-02-10T04:26:47.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[29319]: osd.1) 9 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:26:47.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:26:47.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[29319]: 02-10T04:26:46.425283+0000 2021-02-10T04:26:47.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[29319]: osd.3 (osd.3) 7 : cluster [DBG] 2.1b deep-scrub ok 2021-02-10T04:26:47.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[29319]: 2021-02-10T04:26:47.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[35961]: audit 2021-02-10T04:26:46.050219+0000 mon.a (mon. 2021-02-10T04:26:47.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[35961]: 0) 653 : audit [DBG] from='client.? 172.21.15.85:0/226339202' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "zone.rgw.meta", "var": "pg_num"}]: dispatch 2021-02-10T04:26:47.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:26:47.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[35961]: -10T04:26:46.092008+0000 osd.1 ( 2021-02-10T04:26:47.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[35961]: osd.1) 7 : cluster [DBG] 2.9 scrub ok 2021-02-10T04:26:47.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:26:47.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[35961]: -02-10T04:26:46.131125+0000 2021-02-10T04:26:47.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[35961]: osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:26:47.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:26:47.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[35961]: 02-10T04:26:46.131214+0000 2021-02-10T04:26:47.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[35961]: osd.1 (osd.1) 9 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:26:47.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[35961]: 2021-02-10T04:26:47.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[35961]: cluster 2021-02-10T04:26:47.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[35961]: 2021-02-10T04:26:46.425283+0000 osd.3 2021-02-10T04:26:47.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:47 smithi085 conmon[35961]: (osd.3) 7 : cluster [DBG] 2.1b deep-scrub ok 2021-02-10T04:26:47.894 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:26:47 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:26:48.335 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:48 smithi085 conmon[29319]: cluster 2021-02-10T04:26:47. 2021-02-10T04:26:48.336 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:48 smithi085 conmon[29319]: 039381+0000 mon.a (mon.0) 654 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2021-02-10T04:26:48.336 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:48 smithi085 conmon[29319]: cluster 2021-02-10T04:26:47.989554+0000 mon.a (mon.0) 655 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2021-02-10T04:26:48.336 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:48 smithi085 conmon[35961]: cluster 2021-02-10T04:26:47.039381+0000 mon.a ( 2021-02-10T04:26:48.336 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:48 smithi085 conmon[35961]: mon.0) 654 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2021-02-10T04:26:48.336 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:48 smithi085 conmon[35961]: cluster 2021-02-10T04:26:47.989554+0000 mon.a (mon.0) 655 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2021-02-10T04:26:48.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:48 smithi164 conmon[29288]: cluster 2021-02-10T04:26:47.039381+0000 mon.a (mon.0) 654 : cluster [DBG] 2021-02-10T04:26:48.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:48 smithi164 conmon[29288]: osdmap e75: 8 total, 8 up, 8 in 2021-02-10T04:26:48.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:48 smithi164 conmon[29288]: cluster 2021-02-10T04:26:47.989554+0000 mon.a (mon.0) 655 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2021-02-10T04:26:48.720 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:26:49.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:49 smithi164 conmon[29288]: audit 2021-02-10T04:26:47.566662+0000 mgr.y (mgr.14140) 245 : audit [DBG] 2021-02-10T04:26:49.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:49 smithi164 conmon[29288]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:26:49.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:49 smithi164 conmon[29288]: cluster 2021-02-10T04:26:47.964311+0000 mgr.y (mgr.14140) 246 : cluster [DBG] pgmap v220: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 736 B/s rd, 0 op/s 2021-02-10T04:26:49.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:49 smithi164 conmon[29288]: audit 2021-02-10T04:26: 2021-02-10T04:26:49.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:49 smithi164 conmon[29288]: 48.716995+0000 mon.a (mon.0) 656 : audit [DBG] from='client.? 172.21.15.85:0/2345195974' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2021-02-10T04:26:49.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:49 smithi085 conmon[29319]: audit 2021-02-10T04:26:47.566662+0000 mgr.y (mgr.14140) 245 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:26:49.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:49 smithi085 conmon[29319]: cluster 2021-02-10T04:26:47.964311+0000 mgr.y (mgr.14140) 246 : cluster [DBG] pgmap v220: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 736 B/s rd, 0 op/s 2021-02-10T04:26:49.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:49 smithi085 conmon[29319]: audit 2021-02-10T04:26:48.716995+0000 mon.a (mon.0) 656 : audit [DBG] from='client.? 172.21.15.85:0/2345195974' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2021-02-10T04:26:49.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:49 smithi085 conmon[35961]: audit 2021-02-10T04:26:47.566662+0000 mgr.y (mgr.14140) 245 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:26:49.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:49 smithi085 conmon[35961]: cluster 2021-02-10T04:26:47.964311+0000 mgr.y (mgr.14140) 246 : cluster [DBG] pgmap v220: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 736 B/s rd, 0 op/s 2021-02-10T04:26:49.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:49 smithi085 conmon[35961]: audit 2021-02-10T04:26:48.716995+0000 mon.a (mon.0) 656 : audit [DBG] from='client.? 172.21.15.85:0/2345195974' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2021-02-10T04:26:49.441 INFO:teuthology.orchestra.run.smithi085.stdout:{"epoch":15,"active_gid":14140,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6800","nonce":7}]},"active_addr":"172.21.15.85:6800/7","active_change":"2021-02-10T04:21:37.851865+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":34109,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://smithi085:8443/"},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":3,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.85:0","nonce":3042584991}]},{"addrvec":[{"type":"v2","addr":"172.21.15.85:0","nonce":1625077306}]},{"addrvec":[{"type":"v2","addr":"172.21.15.85:0","nonce":2852395118}]}]}} 2021-02-10T04:26:49.446 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2021-02-10T04:26:49.446 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2021-02-10T04:26:49.446 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd dump --format=json 2021-02-10T04:26:49.641 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:26:50.189 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:50 smithi085 conmon[29319]: cluster 2021-02-10T04:26:48.599248+0000 osd.2 (osd.2) 9 2021-02-10T04:26:50.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:50 smithi085 conmon[29319]: : cluster [DBG] 2.c scrub ok 2021-02-10T04:26:50.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:50 smithi085 conmon[29319]: cluster 2021-02-10T04:26:49.013677+0000 osd.5 (osd.5) 6 : cluster [DBG] 2.12 scrub ok 2021-02-10T04:26:50.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:50 smithi085 conmon[35961]: cluster 2021-02-10T04:26:48.599248+0000 2021-02-10T04:26:50.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:50 smithi085 conmon[35961]: osd.2 (osd.2) 9 : cluster [DBG] 2.c scrub ok 2021-02-10T04:26:50.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:50 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:26:50.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:50 smithi085 conmon[35961]: 02-10T04:26:49.013677+0000 osd.5 (osd.5) 6 : cluster [DBG] 2021-02-10T04:26:50.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:50 smithi085 conmon[35961]: 2.12 scrub ok 2021-02-10T04:26:50.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:50 smithi164 conmon[29288]: cluster 2021-02-10T04:26:48.599248+0000 osd.2 (osd.2) 9 : cluster [DBG] 2.c scrub ok 2021-02-10T04:26:50.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:50 smithi164 conmon[29288]: cluster 2021-02-10T04:26:49.013677+0000 osd.5 (osd.5) 6 : cluster [DBG] 2.12 scrub ok 2021-02-10T04:26:51.121 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:51 smithi085 conmon[29319]: cluster 2021-02-10T04:26:49.964788+0000 mgr.y (mgr.14140 2021-02-10T04:26:51.121 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:51 smithi085 conmon[29319]: ) 247 : cluster [DBG] pgmap v222: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-02-10T04:26:51.121 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:51 smithi085 conmon[29319]: cluster 2021-02-10T04:26:50.000946+0000 osd.1 (osd.1) 10 : cluster [DBG] 2.d scrub ok 2021-02-10T04:26:51.122 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:51 smithi085 conmon[29319]: cluster 2021-02-10T04:26:50.393860+0000 osd.3 (osd.3) 8 : cluster [DBG] 4.17 scrub ok 2021-02-10T04:26:51.122 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:51 smithi085 conmon[35961]: cluster 2021-02-10T04:26:49.964788+0000 2021-02-10T04:26:51.122 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:51 smithi085 conmon[35961]: mgr.y (mgr.14140) 247 : cluster [DBG] pgmap v222: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-02-10T04:26:51.122 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:51 smithi085 conmon[35961]: cluster 2021-02-10T04:26:50.000946+0000 osd.1 (osd.1) 10 : cluster [DBG] 2.d scrub ok 2021-02-10T04:26:51.122 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:51 smithi085 conmon[35961]: cluster 2021-02-10T04:26:50.393860+0000 osd.3 (osd.3) 8 : cluster [DBG] 4.17 scrub ok 2021-02-10T04:26:51.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:51 smithi164 conmon[29288]: cluster 2021-02-10T04:26:49.964788+0000 mgr.y (mgr.14140) 247 : cluster [DBG] pgmap v222: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-02-10T04:26:51.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:51 smithi164 conmon[29288]: 2021-02-10T04:26:51.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:51 smithi164 conmon[29288]: cluster 2021-02-10T04:26:50.000946+0000 osd.1 (osd.1) 10 : cluster [DBG] 2.d scrub ok 2021-02-10T04:26:51.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:51 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:26:51.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:51 smithi164 conmon[29288]: 04:26:50.393860+0000 osd.3 (osd.3) 8 : cluster [DBG] 4.17 scrub ok 2021-02-10T04:26:51.461 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:26:51.462 INFO:teuthology.orchestra.run.smithi085.stdout:{"epoch":76,"fsid":"42224fce-6b57-11eb-8feb-001a4aab830c","created":"2021-02-10T04:20:57.933279+0000","modified":"2021-02-10T04:26:47.982984+0000","last_up_change":"2021-02-10T04:25:35.312604+0000","last_in_change":"2021-02-10T04:25:35.312604+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-10T04:24:05.258028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2021-02-10T04:25:32.688554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"iscsi","create_time":"2021-02-10T04:25:47.658613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"zone.rgw.log","create_time":"2021-02-10T04:25:47.968727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"zone.rgw.control","create_time":"2021-02-10T04:25:49.278260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"zone.rgw.meta","create_time":"2021-02-10T04:25:51.792514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"45d0bb5f-35fa-4596-964f-75abcee53bee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6801","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6802","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6804","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6803","nonce":7}]},"public_addr":"172.21.15.85:6801/7","cluster_addr":"172.21.15.85:6802/7","heartbeat_back_addr":"172.21.15.85:6804/7","heartbeat_front_addr":"172.21.15.85:6803/7","state":["exists","up"]},{"osd":1,"uuid":"3421a5ce-872e-4b52-adf7-6aaa595871a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6805","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6806","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6808","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6807","nonce":7}]},"public_addr":"172.21.15.85:6805/7","cluster_addr":"172.21.15.85:6806/7","heartbeat_back_addr":"172.21.15.85:6808/7","heartbeat_front_addr":"172.21.15.85:6807/7","state":["exists","up"]},{"osd":2,"uuid":"96e1952a-1042-4316-9fc9-86813e88b912","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6809","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6810","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6812","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6811","nonce":7}]},"public_addr":"172.21.15.85:6809/7","cluster_addr":"172.21.15.85:6810/7","heartbeat_back_addr":"172.21.15.85:6812/7","heartbeat_front_addr":"172.21.15.85:6811/7","state":["exists","up"]},{"osd":3,"uuid":"e019f49f-5dd4-4104-9ced-94c710616642","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6813","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6814","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6816","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6815","nonce":7}]},"public_addr":"172.21.15.85:6813/7","cluster_addr":"172.21.15.85:6814/7","heartbeat_back_addr":"172.21.15.85:6816/7","heartbeat_front_addr":"172.21.15.85:6815/7","state":["exists","up"]},{"osd":4,"uuid":"91c8a80a-37b6-4c30-a32a-647b2e6d801f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6801","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6803","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":7}]},"public_addr":"172.21.15.164:6800/7","cluster_addr":"172.21.15.164:6801/7","heartbeat_back_addr":"172.21.15.164:6803/7","heartbeat_front_addr":"172.21.15.164:6802/7","state":["exists","up"]},{"osd":5,"uuid":"cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6805","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":7}]},"public_addr":"172.21.15.164:6804/7","cluster_addr":"172.21.15.164:6805/7","heartbeat_back_addr":"172.21.15.164:6807/7","heartbeat_front_addr":"172.21.15.164:6806/7","state":["exists","up"]},{"osd":6,"uuid":"adc40430-3eeb-4003-a824-343021243077","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6809","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6811","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":7}]},"public_addr":"172.21.15.164:6808/7","cluster_addr":"172.21.15.164:6809/7","heartbeat_back_addr":"172.21.15.164:6811/7","heartbeat_front_addr":"172.21.15.164:6810/7","state":["exists","up"]},{"osd":7,"uuid":"a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6813","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":7}]},"public_addr":"172.21.15.164:6812/7","cluster_addr":"172.21.15.164:6813/7","heartbeat_back_addr":"172.21.15.164:6815/7","heartbeat_front_addr":"172.21.15.164:6814/7","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:26:29.762636+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:26:46.131217+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:26:04.726309+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:26:23.294056+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:26:40.302373+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:25:57.335463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:26:16.688557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:26:34.973034+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":7,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.85:0/801393150":"2021-02-11T04:21:37.851607+0000","172.21.15.85:0/3049671696":"2021-02-11T04:21:37.851607+0000","172.21.15.85:0/4056505602":"2021-02-11T04:21:20.700478+0000","172.21.15.85:6800/7":"2021-02-11T04:21:20.700478+0000","172.21.15.85:0/1879026062":"2021-02-11T04:21:20.700478+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-10T04:26:52.165 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2021-02-10T04:26:52.165 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd dump --format=json 2021-02-10T04:26:52.360 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:26:52.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:52 smithi164 conmon[29288]: cluster 2021-02-10T04:26:50.624066+0000 osd.2 (osd.2) 10 : cluster [DBG] 5.d scrub ok 2021-02-10T04:26:52.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:52 smithi164 conmon[29288]: cluster 2021-02-10T04:26:51. 2021-02-10T04:26:52.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:52 smithi164 conmon[29288]: 409143+0000 osd.0 (osd.0) 11 : cluster [DBG] 4.1e scrub ok 2021-02-10T04:26:52.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:52 smithi164 conmon[29288]: audit 2021-02-10T04:26:51.461374+0000 mon.c (mon.1) 20 : audit [DBG] from='client.? 172.21.15.85:0/1585096637' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-02-10T04:26:52.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:52 smithi085 conmon[29319]: cluster 2021-02-10T04:26:50.624066+0000 osd.2 (osd.2) 10 : cluster [DBG] 5.d scrub ok 2021-02-10T04:26:52.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:52 smithi085 conmon[29319]: cluster 2021-02-10T04:26:51.409143+0000 osd.0 (osd.0) 11 : cluster [DBG] 4.1e scrub ok 2021-02-10T04:26:52.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:52 smithi085 conmon[29319]: audit 2021-02-10T04:26:51.461374+0000 mon.c (mon.1) 20 : audit [DBG] from='client.? 172.21.15.85:0/1585096637' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-02-10T04:26:52.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:52 smithi085 conmon[35961]: cluster 2021-02-10T04:26:50.624066+0000 osd.2 (osd.2) 10 : cluster [DBG] 5.d scrub ok 2021-02-10T04:26:52.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:52 smithi085 conmon[35961]: cluster 2021-02-10T04:26:51.409143+0000 osd.0 (osd.0) 11 : cluster [DBG] 4.1e scrub ok 2021-02-10T04:26:52.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:52 smithi085 conmon[35961]: audit 2021-02-10T04:26:51.461374+0000 mon.c (mon.1) 20 : audit [DBG] from='client.? 172.21.15.85:0/1585096637' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-02-10T04:26:52.893 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:52 smithi164 systemd[1]: Starting Ceph grafana.a for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T04:26:53.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:53 smithi085 conmon[29319]: cluster 2021-02-10T04:26:51.605221+0000 osd.2 ( 2021-02-10T04:26:53.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:53 smithi085 conmon[29319]: osd.2) 11 : cluster [DBG] 5.a scrub ok 2021-02-10T04:26:53.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:53 smithi085 conmon[29319]: cluster 2021-02-10T04:26:51.965439+0000 mgr.y (mgr.14140) 248 : cluster [DBG] pgmap v223: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:26:53.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:53 smithi085 conmon[29319]: cluster 2021-02-10T04:26:52.011092+0000 osd.1 (osd.1) 11 : cluster [DBG] 2.15 scrub ok 2021-02-10T04:26:53.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:53 smithi085 conmon[29319]: cluster 2021-02-10T04:26:52.256712+0000 osd.6 (osd.6) 7 : cluster [DBG] 2.17 scrub ok 2021-02-10T04:26:53.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:53 smithi085 conmon[35961]: cluster 2021-02-10T04:26:51.605221+0000 osd.2 ( 2021-02-10T04:26:53.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:53 smithi085 conmon[35961]: osd.2) 11 : cluster [DBG] 5.a scrub ok 2021-02-10T04:26:53.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:53 smithi085 conmon[35961]: cluster 2021-02-10T04:26:51.965439+0000 mgr.y (mgr.14140) 248 : cluster [DBG] pgmap v223: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:26:53.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:53 smithi085 conmon[35961]: cluster 2021-02-10T04:26:52.011092+0000 osd.1 (osd.1) 11 : cluster [DBG] 2.15 scrub ok 2021-02-10T04:26:53.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:53 smithi085 conmon[35961]: cluster 2021-02-10T04:26:52.256712+0000 osd.6 (osd.6) 7 : cluster [DBG] 2.17 scrub ok 2021-02-10T04:26:53.391 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[29288]: cluster 2021-02-10T04:26:51.605221+0000 osd.2 (osd.2) 11 : cluster [DBG] 5.a scrub ok 2021-02-10T04:26:53.391 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[29288]: cluster 2021-02-10T04:26:51.965439+0000 mgr.y (mgr.14140) 248 : cluster [DBG] pgmap v223: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:26:53.391 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[29288]: cluster 2021-02-10T04:26:52.011092+0000 osd.1 (osd.1) 11 : cluster [DBG] 2.15 scrub ok 2021-02-10T04:26:53.391 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[29288]: cluster 2021-02-10T04:26:52.256712+0000 osd.6 (osd 2021-02-10T04:26:53.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[29288]: .6) 7 : cluster [DBG] 2.17 scrub ok 2021-02-10T04:26:53.643 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[52406]: t=2021-02-10T04:26:53+0000 lvl=info msg="Starting Grafana" logger=server version=6.7.4 commit=8e44bbc5f5 branch=HEAD compiled=2020-05-26T17:35:38+0000 2021-02-10T04:26:53.643 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[52406]: t=2021-02-10T04:26:53+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2021-02-10T04:26:53.643 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[52406]: t=2021-02-10T04:26:53+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2021-02-10T04:26:53.643 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[52406]: t=2021-02-10T04:26:53+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2021-02-10T04:26:53.643 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[52406]: t=2021-02-10T04:26:53+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2021-02-10T04:26:53.643 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[52406]: t=2021-02-10T04:26:53+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2021-02-10T04:26:53.643 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[52406]: t=2021-02-10T04:26:53+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2021-02-10T04:26:53.643 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[52406]: t=2021-02-10T04:26:53+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2021-02-10T04:26:53.643 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[52406]: t=2021-02-10T04:26:53+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2021-02-10T04:26:53.644 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[52406]: t=2021-02-10T04:26:53+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2021-02-10T04:26:53.644 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[52406]: t=2021-02-10T04:26:53+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2021-02-10T04:26:53.644 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[52406]: t=2021-02-10T04:26:53+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2021-02-10T04:26:53.644 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[52406]: t=2021-02-10T04:26:53+0000 lvl=info msg="App mode production" logger=settings 2021-02-10T04:26:53.644 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[52406]: t=2021-02-10T04:26:53+0000 lvl=info msg="Initializing SqlStore" logger=server 2021-02-10T04:26:53.644 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[52406]: t=2021-02-10T04:26:53+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2021-02-10T04:26:53.644 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[52406]: t=2021-02-10T04:26:53+0000 lvl=info msg="Starting DB migration" logger=migrator 2021-02-10T04:26:53.644 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[52406]: t=2021-02-10T04:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2021-02-10T04:26:53.644 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:53 smithi164 bash[52293]: 0510ec24d30d8a6896ae8cc1991b7789a701c5ed9185e0334c264a4618fe3010 2021-02-10T04:26:53.644 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:53 smithi164 systemd[1]: Started Ceph grafana.a for 42224fce-6b57-11eb-8feb-001a4aab830c. 2021-02-10T04:26:53.644 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[52406]: t=2021-02-10T04:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2021-02-10T04:26:53.644 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[52406]: t=2021-02-10T04:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2021-02-10T04:26:53.644 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[52406]: t=2021-02-10T04:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2021-02-10T04:26:53.910 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[52406]: t=2021-02-10T04:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2021-02-10T04:26:53.910 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[52406]: t=2021-02-10T04:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2021-02-10T04:26:53.910 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[52406]: t=2021-02-10T04:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2021-02-10T04:26:53.910 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[52406]: t=2021-02-10T04:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2021-02-10T04:26:53.910 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[52406]: t=2021-02-10T04:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2021-02-10T04:26:54.179 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:53 smithi164 conmon[52406]: t=2021-02-10T04:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2021-02-10T04:26:54.179 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:54 smithi164 conmon[52406]: t=2021-02-10T04:26:54+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2021-02-10T04:26:54.179 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:54 smithi164 conmon[52406]: t=2021-02-10T04:26:54+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2021-02-10T04:26:54.199 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:26:54.199 INFO:teuthology.orchestra.run.smithi085.stdout:{"epoch":76,"fsid":"42224fce-6b57-11eb-8feb-001a4aab830c","created":"2021-02-10T04:20:57.933279+0000","modified":"2021-02-10T04:26:47.982984+0000","last_up_change":"2021-02-10T04:25:35.312604+0000","last_in_change":"2021-02-10T04:25:35.312604+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-10T04:24:05.258028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2021-02-10T04:25:32.688554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"iscsi","create_time":"2021-02-10T04:25:47.658613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"zone.rgw.log","create_time":"2021-02-10T04:25:47.968727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"zone.rgw.control","create_time":"2021-02-10T04:25:49.278260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"zone.rgw.meta","create_time":"2021-02-10T04:25:51.792514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"45d0bb5f-35fa-4596-964f-75abcee53bee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6801","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6802","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6804","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6803","nonce":7}]},"public_addr":"172.21.15.85:6801/7","cluster_addr":"172.21.15.85:6802/7","heartbeat_back_addr":"172.21.15.85:6804/7","heartbeat_front_addr":"172.21.15.85:6803/7","state":["exists","up"]},{"osd":1,"uuid":"3421a5ce-872e-4b52-adf7-6aaa595871a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6805","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6806","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6808","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6807","nonce":7}]},"public_addr":"172.21.15.85:6805/7","cluster_addr":"172.21.15.85:6806/7","heartbeat_back_addr":"172.21.15.85:6808/7","heartbeat_front_addr":"172.21.15.85:6807/7","state":["exists","up"]},{"osd":2,"uuid":"96e1952a-1042-4316-9fc9-86813e88b912","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6809","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6810","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6812","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6811","nonce":7}]},"public_addr":"172.21.15.85:6809/7","cluster_addr":"172.21.15.85:6810/7","heartbeat_back_addr":"172.21.15.85:6812/7","heartbeat_front_addr":"172.21.15.85:6811/7","state":["exists","up"]},{"osd":3,"uuid":"e019f49f-5dd4-4104-9ced-94c710616642","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6813","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6814","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6816","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6815","nonce":7}]},"public_addr":"172.21.15.85:6813/7","cluster_addr":"172.21.15.85:6814/7","heartbeat_back_addr":"172.21.15.85:6816/7","heartbeat_front_addr":"172.21.15.85:6815/7","state":["exists","up"]},{"osd":4,"uuid":"91c8a80a-37b6-4c30-a32a-647b2e6d801f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6801","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6803","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":7}]},"public_addr":"172.21.15.164:6800/7","cluster_addr":"172.21.15.164:6801/7","heartbeat_back_addr":"172.21.15.164:6803/7","heartbeat_front_addr":"172.21.15.164:6802/7","state":["exists","up"]},{"osd":5,"uuid":"cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6805","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":7}]},"public_addr":"172.21.15.164:6804/7","cluster_addr":"172.21.15.164:6805/7","heartbeat_back_addr":"172.21.15.164:6807/7","heartbeat_front_addr":"172.21.15.164:6806/7","state":["exists","up"]},{"osd":6,"uuid":"adc40430-3eeb-4003-a824-343021243077","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6809","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6811","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":7}]},"public_addr":"172.21.15.164:6808/7","cluster_addr":"172.21.15.164:6809/7","heartbeat_back_addr":"172.21.15.164:6811/7","heartbeat_front_addr":"172.21.15.164:6810/7","state":["exists","up"]},{"osd":7,"uuid":"a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6813","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":7}]},"public_addr":"172.21.15.164:6812/7","cluster_addr":"172.21.15.164:6813/7","heartbeat_back_addr":"172.21.15.164:6815/7","heartbeat_front_addr":"172.21.15.164:6814/7","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:26:29.762636+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:26:46.131217+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:26:04.726309+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:26:23.294056+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:26:40.302373+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:25:57.335463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:26:16.688557+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:26:34.973034+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":7,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.85:0/801393150":"2021-02-11T04:21:37.851607+0000","172.21.15.85:0/3049671696":"2021-02-11T04:21:37.851607+0000","172.21.15.85:0/4056505602":"2021-02-11T04:21:20.700478+0000","172.21.15.85:6800/7":"2021-02-11T04:21:20.700478+0000","172.21.15.85:0/1879026062":"2021-02-11T04:21:20.700478+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-10T04:26:54.458 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:54 smithi164 conmon[52406]: t=2021-02-10T04:26:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2021-02-10T04:26:54.458 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:54 smithi164 conmon[52406]: t=2021-02-10T04:26:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2021-02-10T04:26:54.715 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:54 smithi164 conmon[29288]: audit 2021-02-10T04:26:53.453687+0000 mon.a (mon.0) 657 : audit 2021-02-10T04:26:54.715 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:54 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:54.715 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:54 smithi164 conmon[29288]: audit 2021-02-10T04:26:53.454967+0000 mon.a (mon.0) 658 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:26:54.715 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:54 smithi164 conmon[29288]: audit 2021-02-10T04:26:53. 2021-02-10T04:26:54.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:54 smithi164 conmon[29288]: 455179+0000 mon.a (mon.0) 659 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:26:54.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:54 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:26:54.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:54 smithi164 conmon[29288]: 26:53.965905+0000 mgr.y (mgr.14140) 249 : cluster [DBG] pgmap v224: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:26:54.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:54 smithi164 conmon[29288]: cluster 2021-02-10T04:26:54 2021-02-10T04:26:54.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:54 smithi164 conmon[29288]: .007305+0000 osd.1 (osd.1) 12 : cluster [DBG] 2021-02-10T04:26:54.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:54 smithi164 conmon[29288]: 5.19 scrub ok 2021-02-10T04:26:54.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:54 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:26:54.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:54 smithi164 conmon[29288]: 02-10T04:26:54.198490+0000 mon.c (mon.1) 21 : audit [DBG] from='client.? 172.21.15.85:0/1209280263' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-02-10T04:26:54.716 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:54 smithi164 conmon[52406]: t=2021-02-10T04:26:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2021-02-10T04:26:54.716 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:54 smithi164 conmon[52406]: t=2021-02-10T04:26:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2021-02-10T04:26:54.716 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:54 smithi164 conmon[52406]: t=2021-02-10T04:26:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2021-02-10T04:26:54.716 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:54 smithi164 conmon[52406]: t=2021-02-10T04:26:54+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2021-02-10T04:26:54.717 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:54 smithi164 conmon[52406]: t=2021-02-10T04:26:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2021-02-10T04:26:54.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:54 smithi085 conmon[29319]: audit 2021-02-10T04:26:53.453687+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:54.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:54 smithi085 conmon[29319]: audit 2021-02-10T04:26:53.454967+0000 mon.a (mon.0) 658 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:26:54.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:54 smithi085 conmon[29319]: audit 2021-02-10T04:26:53.455179+0000 mon.a (mon.0) 659 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:26:54.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:54 smithi085 conmon[29319]: cluster 2021-02-10T04:26:53.965905+0000 mgr.y (mgr.14140) 249 : cluster [DBG] pgmap v224: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:26:54.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:54 smithi085 conmon[29319]: cluster 2021-02-10T04:26:54.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:54 smithi085 conmon[29319]: 2021-02-10T04:26:54.007305+0000 osd.1 (osd.1) 12 : cluster [DBG] 5.19 scrub ok 2021-02-10T04:26:54.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:54 smithi085 conmon[29319]: audit 2021-02-10T04:26:54.198490+0000 mon.c (mon.1) 21 : audit [DBG] from='client.? 172.21.15.85:0/1209280263' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-02-10T04:26:54.915 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:54 smithi085 conmon[35961]: audit 2021-02-10T04:26:54.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:54 smithi085 conmon[35961]: 2021-02-10T04:26:54.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:54 smithi085 conmon[35961]: 2021-02-10T04:26:53.453687 2021-02-10T04:26:54.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:54 smithi085 conmon[35961]: +0000 mon.a (mon.0) 657 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:54.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:54 smithi085 conmon[35961]: audit 2021-02-10T04:26:54.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:54 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:26:54.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:54 smithi085 conmon[35961]: :26:53.454967+0000 mon.a (mon.0 2021-02-10T04:26:54.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:54 smithi085 conmon[35961]: ) 658 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:26:54.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:54 smithi085 conmon[35961]: audit 2021-02-10T04:26:53.455179+0000 mon.a (mon.0) 659 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:26:54.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:54 smithi085 conmon[35961]: cluster 2021-02-10T04:26:53.965905+0000 mgr.y (mgr.14140) 249 : cluster [DBG] pgmap v224: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:26:54.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:54 smithi085 conmon[35961]: cluster 2021-02-10T04:26:54. 2021-02-10T04:26:54.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:54 smithi085 conmon[35961]: 007305+0000 osd.1 (osd.1) 12 : cluster [DBG] 5.19 scrub ok 2021-02-10T04:26:54.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:54 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:26:54.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:54 smithi085 conmon[35961]: 02-10T04:26:54.198490+0000 mon.c (mon.1) 21 2021-02-10T04:26:54.917 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:54 smithi085 conmon[35961]: : audit [DBG] from='client.? 172.21.15.85:0/1209280263' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-02-10T04:26:54.931 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph tell osd.0 flush_pg_stats 2021-02-10T04:26:55.041 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:54 smithi164 conmon[52406]: t=2021-02-10T04:26:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2021-02-10T04:26:55.042 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:54 smithi164 conmon[52406]: t=2021-02-10T04:26:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2021-02-10T04:26:55.164 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:26:55.327 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:55 smithi164 conmon[52406]: t=2021-02-10T04:26:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2021-02-10T04:26:55.327 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:55 smithi164 conmon[52406]: t=2021-02-10T04:26:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2021-02-10T04:26:55.327 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:55 smithi164 conmon[52406]: t=2021-02-10T04:26:55+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2021-02-10T04:26:55.327 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:55 smithi164 conmon[52406]: t=2021-02-10T04:26:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2021-02-10T04:26:55.595 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:55 smithi164 conmon[29288]: cluster 2021-02-10T04:26:54.256492+0000 osd.6 (osd.6) 8 : cluster 2021-02-10T04:26:55.595 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:55 smithi164 conmon[29288]: [DBG] 2.14 scrub ok 2021-02-10T04:26:55.595 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:55 smithi164 conmon[52406]: t=2021-02-10T04:26:55+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2021-02-10T04:26:55.595 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:55 smithi164 conmon[52406]: t=2021-02-10T04:26:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2021-02-10T04:26:55.595 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:55 smithi164 conmon[52406]: t=2021-02-10T04:26:55+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2021-02-10T04:26:55.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:55 smithi085 conmon[29319]: cluster 2021-02-10T04:26:54.256492+0000 osd.6 (osd.6) 2021-02-10T04:26:55.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:55 smithi085 conmon[29319]: 8 : cluster [DBG] 2.14 scrub ok 2021-02-10T04:26:55.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:55 smithi085 conmon[35961]: cluster 2021-02-10T04:26:54.256492+0000 osd.6 (osd.6 2021-02-10T04:26:55.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:55 smithi085 conmon[35961]: ) 8 : cluster [DBG] 2.14 scrub ok 2021-02-10T04:26:55.863 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:55 smithi164 conmon[52406]: t=2021-02-10T04:26:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2021-02-10T04:26:55.864 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:55 smithi164 conmon[52406]: t=2021-02-10T04:26:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2021-02-10T04:26:55.864 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:55 smithi164 conmon[52406]: t=2021-02-10T04:26:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2021-02-10T04:26:55.864 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:55 smithi164 conmon[52406]: t=2021-02-10T04:26:55+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2021-02-10T04:26:56.143 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:55 smithi164 conmon[52406]: t=2021-02-10T04:26:55+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2021-02-10T04:26:56.143 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:56 smithi164 conmon[52406]: t=2021-02-10T04:26:56+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2021-02-10T04:26:56.433 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:56 smithi164 conmon[52406]: t=2021-02-10T04:26:56+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2021-02-10T04:26:56.433 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:56 smithi164 conmon[52406]: t=2021-02-10T04:26:56+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2021-02-10T04:26:56.433 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:56 smithi164 conmon[52406]: t=2021-02-10T04:26:56+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2021-02-10T04:26:56.727 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:56 smithi164 conmon[29288]: cluster 2021-02-10T04:26:55.332811+0000 osd.3 (osd. 2021-02-10T04:26:56.727 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:56 smithi164 conmon[29288]: 3) 9 : cluster [DBG] 6.1f scrub ok 2021-02-10T04:26:56.727 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:56 smithi164 conmon[29288]: cluster 2021-02-10T04:26:55.626964+0000 osd.2 (osd.2) 12 : cluster [DBG] 6.f scrub ok 2021-02-10T04:26:56.727 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:56 smithi164 conmon[29288]: cluster 2021-02-10T04:26:55.966424+0000 mgr.y (mgr. 2021-02-10T04:26:56.727 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:56 smithi164 conmon[29288]: 14140) 250 : cluster [DBG] pgmap v225: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:26:56.727 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:56 smithi164 conmon[29288]: cluster 2021-02-10T04:26: 2021-02-10T04:26:56.727 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:56 smithi164 conmon[29288]: 56.059181+0000 osd.1 (osd.1) 13 : cluster 2021-02-10T04:26:56.727 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:56 smithi164 conmon[29288]: [DBG] 4.4 scrub ok 2021-02-10T04:26:56.728 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:56 smithi164 conmon[52406]: t=2021-02-10T04:26:56+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2021-02-10T04:26:56.728 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:56 smithi164 conmon[52406]: t=2021-02-10T04:26:56+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2021-02-10T04:26:56.728 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:56 smithi164 conmon[52406]: t=2021-02-10T04:26:56+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2021-02-10T04:26:56.850 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:56 smithi085 conmon[29319]: cluster 2021-02-10T04:26:55.332811+0000 osd.3 2021-02-10T04:26:56.851 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:56 smithi085 conmon[29319]: (osd.3) 9 : cluster [DBG] 6.1f scrub ok 2021-02-10T04:26:56.851 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:56 smithi085 conmon[29319]: cluster 2021-02-10T04:26:55.626964+0000 osd.2 (osd.2) 12 : cluster 2021-02-10T04:26:56.851 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:56 smithi085 conmon[29319]: [DBG] 6.f scrub ok 2021-02-10T04:26:56.851 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:56 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:26:56.851 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:56 smithi085 conmon[29319]: 26:55.966424+0000 mgr.y (mgr.14140 2021-02-10T04:26:56.851 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:56 smithi085 conmon[29319]: ) 250 : cluster [DBG] pgmap v225: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:26:56.851 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:56 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:26:56.851 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:56 smithi085 conmon[29319]: 02-10T04:26:56.059181 2021-02-10T04:26:56.851 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:56 smithi085 conmon[29319]: +0000 osd.1 (osd.1) 13 : 2021-02-10T04:26:56.851 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:56 smithi085 conmon[29319]: cluster [DBG] 4.4 scrub ok 2021-02-10T04:26:56.852 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:56 smithi085 conmon[35961]: cluster 2021-02-10T04:26: 2021-02-10T04:26:56.852 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:56 smithi085 conmon[35961]: 55.332811+0000 osd.3 (osd.3) 9 : cluster [DBG] 6.1f scrub ok 2021-02-10T04:26:56.852 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:56 smithi085 conmon[35961]: cluster 2021-02-10T04:26:56.852 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:56 smithi085 conmon[35961]: 2021-02-10T04:26:55.626964+0000 osd.2 (osd.2) 12 : cluster [DBG] 6.f scrub ok 2021-02-10T04:26:56.852 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:56 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:26:56.852 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:56 smithi085 conmon[35961]: -10T04:26:55.966424+0000 mgr.y (mgr.14140) 250 : 2021-02-10T04:26:56.852 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:56 smithi085 conmon[35961]: cluster [DBG] pgmap v225: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:26:56.852 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:56 smithi085 conmon[35961]: cluster 2021-02-10T04:26:56.059181+0000 2021-02-10T04:26:56.852 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:56 smithi085 conmon[35961]: osd.1 (osd.1) 13 : cluster [DBG] 4.4 scrub ok 2021-02-10T04:26:56.981 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:56 smithi164 conmon[52406]: t=2021-02-10T04:26:56+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2021-02-10T04:26:56.981 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:56 smithi164 conmon[52406]: t=2021-02-10T04:26:56+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2021-02-10T04:26:56.981 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:56 smithi164 conmon[52406]: t=2021-02-10T04:26:56+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2021-02-10T04:26:56.981 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:56 smithi164 conmon[52406]: t=2021-02-10T04:26:56+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2021-02-10T04:26:57.238 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:57 smithi164 conmon[52406]: t=2021-02-10T04:26:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2021-02-10T04:26:57.238 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:57 smithi164 conmon[52406]: t=2021-02-10T04:26:57+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2021-02-10T04:26:57.238 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:57 smithi164 conmon[52406]: t=2021-02-10T04:26:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2021-02-10T04:26:57.514 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:57 smithi164 conmon[52406]: t=2021-02-10T04:26:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2021-02-10T04:26:57.514 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:57 smithi164 conmon[52406]: t=2021-02-10T04:26:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2021-02-10T04:26:57.514 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:57 smithi164 conmon[52406]: t=2021-02-10T04:26:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2021-02-10T04:26:57.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:57 smithi164 conmon[29288]: cluster 2021-02-10T04:26:56.335730+0000 osd.3 (osd. 2021-02-10T04:26:57.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:57 smithi164 conmon[29288]: 3) 10 : cluster [DBG] 5.17 scrub ok 2021-02-10T04:26:57.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:57 smithi164 conmon[29288]: cluster 2021-02-10T04:26:56.623104+0000 osd.2 (osd.2) 13 : cluster [DBG] 4.1c scrub ok 2021-02-10T04:26:57.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:57 smithi164 conmon[29288]: cluster 2021-02-10T04:26:57.084858+0000 osd.7 (osd.7 2021-02-10T04:26:57.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:57 smithi164 conmon[29288]: ) 8 : cluster [DBG] 2.5 scrub ok 2021-02-10T04:26:57.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:57 smithi164 conmon[29288]: audit 2021-02-10T04:26:57.369238 2021-02-10T04:26:57.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:57 smithi164 conmon[29288]: +0000 mon.a (mon.0) 660 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:57.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:57 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:26:57.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:57 smithi164 conmon[29288]: -10T04:26:57.444040+0000 mon.a (mon.0) 661 : 2021-02-10T04:26:57.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:57 smithi164 conmon[29288]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:57.765 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:57 smithi164 conmon[52406]: t=2021-02-10T04:26:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2021-02-10T04:26:57.765 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:57 smithi164 conmon[52406]: t=2021-02-10T04:26:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2021-02-10T04:26:57.765 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:57 smithi164 conmon[52406]: t=2021-02-10T04:26:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2021-02-10T04:26:57.766 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:26:57 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:26:57.923 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:57 smithi085 conmon[29319]: cluster 2021-02-10T04:26:56.335730+0000 osd.3 (osd.3) 10 : cluster [DBG] 5.17 scrub ok 2021-02-10T04:26:57.923 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:57 smithi085 conmon[29319]: cluster 2021-02-10T04:26:56.623104+0000 osd.2 (osd.2) 13 : cluster [DBG] 4.1c scrub ok 2021-02-10T04:26:57.923 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:57 smithi085 conmon[29319]: cluster 2021-02-10T04:26:57.084858+0000 osd.7 (osd.7) 8 : cluster [DBG] 2.5 scrub ok 2021-02-10T04:26:57.923 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:57 smithi085 conmon[29319]: audit 2021-02-10T04:26:57.369238+0000 mon.a (mon.0) 660 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:57.923 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:57 smithi085 conmon[29319]: audit 2021-02-10T04:26:57.444040+0000 mon.a (mon.0) 661 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:57.923 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:57 smithi085 conmon[35961]: cluster 2021-02-10T04:26:57.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:57 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:26:57.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:57 smithi085 conmon[35961]: T04:26:56.335730+0000 osd.3 (osd.3) 10 : cluster [DBG] 5.17 scrub ok 2021-02-10T04:26:57.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:57 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:26:57.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:57 smithi085 conmon[35961]: 02-10T04:26:56.623104+0000 osd.2 (osd. 2021-02-10T04:26:57.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:57 smithi085 conmon[35961]: 2) 13 : cluster [DBG] 4.1c scrub ok 2021-02-10T04:26:57.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:57 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:26:57.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:57 smithi085 conmon[35961]: -10T04:26:57.084858+0000 osd.7 ( 2021-02-10T04:26:57.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:57 smithi085 conmon[35961]: osd.7) 8 : cluster [DBG] 2.5 scrub ok 2021-02-10T04:26:57.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:57 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:26:57.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:57 smithi085 conmon[35961]: -10T04:26:57.369238+0000 mon.a (mon 2021-02-10T04:26:57.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:57 smithi085 conmon[35961]: .0) 660 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:57.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:57 smithi085 conmon[35961]: audit 2021 2021-02-10T04:26:57.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:57 smithi085 conmon[35961]: -02-10T04:26:57.444040+0000 mon.a 2021-02-10T04:26:57.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:57 smithi085 conmon[35961]: (mon.0) 661 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:57.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:57 smithi085 conmon[35961]: 2021-02-10T04:26:57.943 INFO:teuthology.orchestra.run.smithi085.stdout:34359738413 2021-02-10T04:26:57.943 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph tell osd.1 flush_pg_stats 2021-02-10T04:26:58.033 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:57 smithi164 conmon[52406]: t=2021-02-10T04:26:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2021-02-10T04:26:58.033 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:57 smithi164 conmon[52406]: t=2021-02-10T04:26:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2021-02-10T04:26:58.033 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:57 smithi164 conmon[52406]: t=2021-02-10T04:26:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2021-02-10T04:26:58.154 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:26:58.334 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[52406]: t=2021-02-10T04:26:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2021-02-10T04:26:58.334 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[52406]: t=2021-02-10T04:26:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2021-02-10T04:26:58.335 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[52406]: t=2021-02-10T04:26:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2021-02-10T04:26:58.335 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[52406]: t=2021-02-10T04:26:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2021-02-10T04:26:58.585 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[52406]: t=2021-02-10T04:26:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2021-02-10T04:26:58.586 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[52406]: t=2021-02-10T04:26:58+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2021-02-10T04:26:58.586 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[52406]: t=2021-02-10T04:26:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2021-02-10T04:26:58.586 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[29288]: cluster 2021-02-10T04:26:57.309910+0000 osd.3 (osd.3) 11 2021-02-10T04:26:58.586 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[29288]: : cluster [DBG] 6.11 scrub ok 2021-02-10T04:26:58.586 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[29288]: cephadm 2021-02-10T04:26:57.369870+0000 mgr.y (mgr.14140) 251 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:26:58.586 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[29288]: cluster 2021-02-10T04:26:57.560890+0000 osd.5 (osd.5) 7 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:26:58.586 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[29288]: cluster 2021-02-10T04:26:57.561026+0000 osd.5 (osd.5) 8 : cluster 2021-02-10T04:26:58.586 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[29288]: [DBG] purged_snaps scrub ok 2021-02-10T04:26:58.586 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[29288]: audit 2021-02-10T04:26:58.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[29288]: 2021-02-10T04:26:57.571089+0000 mgr.y (mgr.14140) 252 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:26:58.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[29288]: 2021-02-10T04:26:58.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:26:58.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[29288]: 02-10T04:26:57.639276+0000 osd.2 (osd.2) 14 : cluster [DBG] 2.10 scrub ok 2021-02-10T04:26:58.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:26:58.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[29288]: -10T04:26:57.736302+0000 mon.a (mon.0) 662 : audit [INF] 2021-02-10T04:26:58.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:58.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[29288]: cephadm 2021-02-10T04:26:58.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[29288]: 2021-02-10T04:26:57.739628+0000 mgr.y (mgr.14140) 253 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2021-02-10T04:26:58.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[29288]: 2021-02-10T04:26:58.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[29288]: cephadm 2021-02- 2021-02-10T04:26:58.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[29288]: 10T04:26:57.742490+0000 mgr.y (mgr.14140) 254 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi085 2021-02-10T04:26:58.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:26:58.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[29288]: 10T04:26:57.966875+0000 mgr.y (mgr.14140) 255 : cluster [DBG] 2021-02-10T04:26:58.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[29288]: pgmap v226: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:26:58.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:58 smithi085 conmon[29319]: cluster 2021-02-10T04:26:57.309910+0000 osd.3 (osd.3) 11 : cluster 2021-02-10T04:26:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:58 smithi085 conmon[29319]: [DBG] 6.11 scrub ok 2021-02-10T04:26:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:58 smithi085 conmon[29319]: cephadm 2021-02-10T04:26:57.369870+0000 mgr.y (mgr.14140) 251 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:26:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:58 smithi085 conmon[29319]: cluster 2021-02-10T04:26:57.560890+0000 osd.5 (osd.5) 7 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:26:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:58 smithi085 conmon[29319]: cluster 2021-02-10T04:26:57.561026+0000 osd.5 (osd.5) 8 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:26:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:58 smithi085 conmon[29319]: audit 2021-02-10T04:26:57.571089+0000 mgr.y (mgr.14140) 252 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:26:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:58 smithi085 conmon[29319]: cluster 2021-02-10T04:26:57.639276+0000 osd.2 (osd.2) 14 : cluster [DBG] 2.10 scrub ok 2021-02-10T04:26:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:58 smithi085 conmon[29319]: audit 2021-02-10T04:26:57.736302+0000 mon.a (mon.0) 662 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:58 smithi085 conmon[29319]: cephadm 2021-02-10T04:26:57.739628+0000 mgr.y (mgr.14140) 253 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2021-02-10T04:26:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:58 smithi085 conmon[29319]: cephadm 2021-02-10T04:26:57.742490+0000 mgr.y (mgr.14140) 254 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi085 2021-02-10T04:26:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:58 smithi085 conmon[29319]: cluster 2021-02-10T04:26:57.966875+0000 mgr.y (mgr.14140) 255 : cluster [DBG] pgmap v226: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:26:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:58 smithi085 conmon[35961]: cluster 2021-02-10T04:26:57.309910 2021-02-10T04:26:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:58 smithi085 conmon[35961]: +0000 osd.3 (osd.3) 11 : cluster [DBG] 6.11 scrub ok 2021-02-10T04:26:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:58 smithi085 conmon[35961]: cephadm 2021-02-10T04:26:57.369870+0000 mgr.y (mgr.14140) 251 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:26:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:58 smithi085 conmon[35961]: cluster 2021-02-10T04:26:57.560890+0000 osd.5 (osd.5) 7 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:26:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:58 smithi085 conmon[35961]: cluster 2021-02-10T04:26:57.561026+0000 osd.5 (osd.5) 8 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:26:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:58 smithi085 conmon[35961]: audit 2021-02-10T04:26:57.571089+0000 mgr.y (mgr.14140) 252 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:26:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:58 smithi085 conmon[35961]: cluster 2021-02-10T04:26:57.639276+0000 osd.2 (osd.2) 14 : cluster [DBG] 2.10 scrub ok 2021-02-10T04:26:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:58 smithi085 conmon[35961]: audit 2021-02-10T04:26:57.736302+0000 mon.a (mon 2021-02-10T04:26:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:58 smithi085 conmon[35961]: .0) 662 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:26:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:58 smithi085 conmon[35961]: cephadm 2021-02-10T04:26:57.739628+0000 mgr.y (mgr.14140) 253 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2021-02-10T04:26:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:58 smithi085 conmon[35961]: cephadm 2021-02-10T04:26:57.742490+0000 mgr.y (mgr 2021-02-10T04:26:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:58 smithi085 conmon[35961]: .14140) 254 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi085 2021-02-10T04:26:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:58 smithi085 conmon[35961]: cluster 2021-02-10T04:26:57.966875+0000 mgr.y (mgr. 2021-02-10T04:26:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:58 smithi085 conmon[35961]: 14140) 255 : cluster [DBG] pgmap v226: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:26:58.893 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[52406]: t=2021-02-10T04:26:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2021-02-10T04:26:58.893 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[52406]: t=2021-02-10T04:26:58+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2021-02-10T04:26:58.893 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[52406]: t=2021-02-10T04:26:58+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2021-02-10T04:26:58.893 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[52406]: t=2021-02-10T04:26:58+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2021-02-10T04:26:59.180 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[52406]: t=2021-02-10T04:26:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2021-02-10T04:26:59.180 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:58 smithi164 conmon[52406]: t=2021-02-10T04:26:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2021-02-10T04:26:59.180 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:59 smithi164 conmon[52406]: t=2021-02-10T04:26:59+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2021-02-10T04:26:59.180 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:59 smithi164 conmon[52406]: t=2021-02-10T04:26:59+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2021-02-10T04:26:59.465 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:59 smithi164 conmon[52406]: t=2021-02-10T04:26:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2021-02-10T04:26:59.465 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:59 smithi164 conmon[52406]: t=2021-02-10T04:26:59+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2021-02-10T04:26:59.465 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:59 smithi164 conmon[52406]: t=2021-02-10T04:26:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2021-02-10T04:26:59.465 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:59 smithi164 conmon[52406]: t=2021-02-10T04:26:59+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2021-02-10T04:26:59.466 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:59 smithi164 conmon[52406]: t=2021-02-10T04:26:59+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2021-02-10T04:26:59.782 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:59 smithi164 conmon[52406]: t=2021-02-10T04:26:59+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2021-02-10T04:26:59.783 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:59 smithi164 conmon[52406]: t=2021-02-10T04:26:59+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2021-02-10T04:26:59.783 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:59 smithi164 conmon[52406]: t=2021-02-10T04:26:59+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2021-02-10T04:26:59.783 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:59 smithi164 conmon[52406]: t=2021-02-10T04:26:59+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2021-02-10T04:26:59.783 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:59 smithi164 conmon[29288]: cluster 2021-02-10T04:26:58.112678+0000 osd.5 (osd.5) 2021-02-10T04:26:59.783 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:59 smithi164 conmon[29288]: 9 : cluster [DBG] 5.f scrub ok 2021-02-10T04:26:59.784 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:59 smithi164 conmon[29288]: cluster 2021-02-10T04:26:58.198635+0000 osd.6 (osd.6 2021-02-10T04:26:59.784 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:59 smithi164 conmon[29288]: ) 9 : cluster [DBG] 4.1f deep-scrub ok 2021-02-10T04:26:59.784 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:59 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:26:59.784 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:59 smithi164 conmon[29288]: 26:58.478364+0000 mon.a (mon.0 2021-02-10T04:26:59.784 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:59 smithi164 conmon[29288]: ) 663 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2021-02-10T04:26:59.784 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:59 smithi164 conmon[29288]: cluster 2021-02-10T04:26:59.784 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:59 smithi164 conmon[29288]: 2021-02-10T04:26:58.989441+0000 osd.1 (osd.1 2021-02-10T04:26:59.784 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:26:59 smithi164 conmon[29288]: ) 14 : cluster [DBG] 3.0 scrub ok 2021-02-10T04:26:59.940 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:59 smithi085 conmon[29319]: cluster 2021-02-10T04:26:58.112678+0000 osd.5 (osd.5) 9 : 2021-02-10T04:26:59.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:59 smithi085 conmon[29319]: cluster [DBG] 5.f scrub ok 2021-02-10T04:26:59.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:59 smithi085 conmon[29319]: cluster 2021-02-10T04:26:58.198635+0000 osd.6 (osd.6) 9 : cluster [DBG] 4.1f deep-scrub ok 2021-02-10T04:26:59.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:59 smithi085 conmon[29319]: cluster 2021-02-10T04:26:58.478364+0000 mon.a (mon.0) 663 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2021-02-10T04:26:59.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:59 smithi085 conmon[29319]: cluster 2021-02-10T04:26:58.989441+0000 osd.1 (osd. 2021-02-10T04:26:59.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:26:59 smithi085 conmon[29319]: 1) 14 : cluster [DBG] 3.0 scrub ok 2021-02-10T04:26:59.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:59 smithi085 conmon[35961]: cluster 2021-02-10T04:26:58.112678+0000 osd.5 (osd.5) 9 : cluster [DBG] 5.f scrub ok 2021-02-10T04:26:59.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:59 smithi085 conmon[35961]: cluster 2021-02-10T04:26:58.198635+0000 osd.6 (osd.6) 9 : cluster [DBG] 4.1f deep-scrub ok 2021-02-10T04:26:59.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:59 smithi085 conmon[35961]: cluster 2021-02-10T04:26: 2021-02-10T04:26:59.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:59 smithi085 conmon[35961]: 58.478364+0000 mon.a (mon.0) 663 : cluster 2021-02-10T04:26:59.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:59 smithi085 conmon[35961]: [DBG] osdmap e77: 8 total, 8 up, 8 in 2021-02-10T04:26:59.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:59 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:26:59.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:59 smithi085 conmon[35961]: 10T04:26:58.989441+0000 2021-02-10T04:26:59.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:59 smithi085 conmon[35961]: osd.1 (osd.1) 14 : cluster [DBG] 2021-02-10T04:26:59.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:26:59 smithi085 conmon[35961]: 3.0 scrub ok 2021-02-10T04:27:00.084 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:59 smithi164 conmon[52406]: t=2021-02-10T04:26:59+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2021-02-10T04:27:00.084 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:59 smithi164 conmon[52406]: t=2021-02-10T04:26:59+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2021-02-10T04:27:00.084 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:26:59 smithi164 conmon[52406]: t=2021-02-10T04:26:59+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2021-02-10T04:27:00.084 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:00 smithi164 conmon[52406]: t=2021-02-10T04:27:00+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2021-02-10T04:27:00.084 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:00 smithi164 conmon[52406]: t=2021-02-10T04:27:00+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2021-02-10T04:27:00.351 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:00 smithi164 conmon[52406]: t=2021-02-10T04:27:00+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2021-02-10T04:27:00.352 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:00 smithi164 conmon[52406]: t=2021-02-10T04:27:00+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2021-02-10T04:27:00.352 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:00 smithi164 conmon[52406]: t=2021-02-10T04:27:00+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2021-02-10T04:27:00.602 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:00 smithi164 conmon[52406]: t=2021-02-10T04:27:00+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2021-02-10T04:27:00.603 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:00 smithi164 conmon[52406]: t=2021-02-10T04:27:00+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2021-02-10T04:27:00.603 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:00 smithi164 conmon[52406]: t=2021-02-10T04:27:00+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2021-02-10T04:27:00.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:00 smithi164 conmon[29288]: cluster 2021-02-10T04:26:59.174898+0000 osd.6 (osd.6) 10 : cluster 2021-02-10T04:27:00.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:00 smithi164 conmon[29288]: [DBG] 5.2 scrub ok 2021-02-10T04:27:00.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:00 smithi164 conmon[29288]: cluster 2021-02-10T04:26:59.205029+0000 osd.4 (osd.4) 8 : cluster [DBG] 2021-02-10T04:27:00.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:00 smithi164 conmon[29288]: 4.e scrub ok 2021-02-10T04:27:00.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:00 smithi164 conmon[29288]: cluster 2021-02-10T04:27:00.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:00 smithi164 conmon[29288]: 2021-02-10T04:26:59.967421+0000 2021-02-10T04:27:00.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:00 smithi164 conmon[29288]: mgr.y (mgr.14140) 256 : cluster [DBG] pgmap v228: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:00.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:00 smithi164 conmon[29288]: 2021-02-10T04:27:00.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:00 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:27:00.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:00 smithi164 conmon[29288]: 04:27:00.052023+0000 osd.7 (osd.7) 9 : 2021-02-10T04:27:00.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:00 smithi164 conmon[29288]: cluster [DBG] 2.b scrub ok 2021-02-10T04:27:00.760 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:00 smithi085 conmon[29319]: cluster 2021-02-10T04:26:59.174898+0000 2021-02-10T04:27:00.760 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:00 smithi085 conmon[29319]: osd.6 (osd.6) 10 : cluster [DBG] 5.2 scrub ok 2021-02-10T04:27:00.760 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:00 smithi085 conmon[29319]: cluster 2021-02-10T04:26:59.205029+0000 osd.4 (osd.4) 8 : cluster [DBG] 4.e scrub ok 2021-02-10T04:27:00.760 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:00 smithi085 conmon[29319]: cluster 2021-02-10T04:26:59.967421+0000 mgr.y (mgr.14140) 256 : cluster [DBG] pgmap v228: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:00.761 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:00 smithi085 conmon[29319]: cluster 2021-02-10T04:27:00.052023+0000 osd.7 (osd.7) 9 : cluster [DBG] 2.b scrub ok 2021-02-10T04:27:00.761 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:00 smithi085 conmon[35961]: cluster 2021-02-10T04:26:59.174898 2021-02-10T04:27:00.761 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:00 smithi085 conmon[35961]: +0000 osd.6 (osd.6) 10 : cluster [DBG] 5.2 scrub ok 2021-02-10T04:27:00.761 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:00 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:27:00.761 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:00 smithi085 conmon[35961]: 04:26:59.205029+0000 osd.4 (osd.4) 8 : cluster [DBG] 4.e scrub ok 2021-02-10T04:27:00.761 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:00 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:27:00.761 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:00 smithi085 conmon[35961]: 02-10T04:26:59.967421+0000 mgr.y ( 2021-02-10T04:27:00.761 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:00 smithi085 conmon[35961]: mgr.14140) 256 : cluster [DBG] pgmap v228: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:00.761 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:00 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:27:00.761 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:00 smithi085 conmon[35961]: -02-10T04:27:00.052023+0000 2021-02-10T04:27:00.761 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:00 smithi085 conmon[35961]: osd.7 (osd.7) 9 : cluster [DBG] 2.b scrub ok 2021-02-10T04:27:00.761 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:00 smithi085 conmon[35961]: 2021-02-10T04:27:00.879 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:00 smithi164 conmon[52406]: t=2021-02-10T04:27:00+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2021-02-10T04:27:00.879 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:00 smithi164 conmon[52406]: t=2021-02-10T04:27:00+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2021-02-10T04:27:00.880 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:00 smithi164 conmon[52406]: t=2021-02-10T04:27:00+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2021-02-10T04:27:00.880 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:00 smithi164 conmon[52406]: t=2021-02-10T04:27:00+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2021-02-10T04:27:01.138 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:00 smithi164 conmon[52406]: t=2021-02-10T04:27:00+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2021-02-10T04:27:01.139 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:01 smithi164 conmon[52406]: t=2021-02-10T04:27:01+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2021-02-10T04:27:01.139 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:01 smithi164 conmon[52406]: t=2021-02-10T04:27:01+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2021-02-10T04:27:01.393 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:01 smithi164 conmon[52406]: t=2021-02-10T04:27:01+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2021-02-10T04:27:01.393 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:01 smithi164 conmon[52406]: t=2021-02-10T04:27:01+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2021-02-10T04:27:01.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:01 smithi085 conmon[29319]: cluster 2021-02-10T04:27:00.215032+0000 osd.4 (osd.4) 9 : cluster [DBG] 4.d scrub ok 2021-02-10T04:27:01.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:01 smithi085 conmon[35961]: cluster 2021-02-10T04:27:00.215032+0000 osd.4 (osd.4) 9 : cluster [DBG] 4.d scrub ok 2021-02-10T04:27:01.699 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:01 smithi164 conmon[29288]: cluster 2021-02-10T04:27:00.215032+0000 osd.4 (osd.4) 9 : cluster [DBG] 2021-02-10T04:27:01.699 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:01 smithi164 conmon[29288]: 4.d scrub ok 2021-02-10T04:27:01.699 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:01 smithi164 conmon[52406]: t=2021-02-10T04:27:01+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2021-02-10T04:27:01.699 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:01 smithi164 conmon[52406]: t=2021-02-10T04:27:01+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2021-02-10T04:27:01.699 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:01 smithi164 conmon[52406]: t=2021-02-10T04:27:01+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2021-02-10T04:27:01.699 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:01 smithi164 conmon[52406]: t=2021-02-10T04:27:01+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2021-02-10T04:27:01.864 INFO:teuthology.orchestra.run.smithi085.stdout:55834574888 2021-02-10T04:27:01.864 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph tell osd.2 flush_pg_stats 2021-02-10T04:27:02.000 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:01 smithi164 conmon[52406]: t=2021-02-10T04:27:01+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2021-02-10T04:27:02.000 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:01 smithi164 conmon[52406]: t=2021-02-10T04:27:01+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2021-02-10T04:27:02.000 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:01 smithi164 conmon[52406]: t=2021-02-10T04:27:01+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2021-02-10T04:27:02.000 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:01 smithi164 conmon[52406]: t=2021-02-10T04:27:01+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2021-02-10T04:27:02.000 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:02 smithi164 conmon[52406]: t=2021-02-10T04:27:02+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2021-02-10T04:27:02.073 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:27:02.127 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 04:27:01 smithi085 systemd[1]: Stopping Ceph alertmanager.a for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T04:27:02.127 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 04:27:01 smithi085 conmon[57535]: level=info ts=2021-02-10T04:27:01.881Z caller=main.go:536 msg="Received SIGTERM, exiting gracefully..." 2021-02-10T04:27:02.268 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:02 smithi164 conmon[52406]: t=2021-02-10T04:27:02+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2021-02-10T04:27:02.268 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:02 smithi164 conmon[52406]: t=2021-02-10T04:27:02+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2021-02-10T04:27:02.268 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:02 smithi164 conmon[52406]: t=2021-02-10T04:27:02+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2021-02-10T04:27:02.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:02 smithi164 conmon[29288]: cluster 2021-02-10T04:27:01.184622+0000 osd.4 (osd 2021-02-10T04:27:02.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:02 smithi164 conmon[29288]: .4) 10 : cluster [DBG] 6.2 scrub ok 2021-02-10T04:27:02.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:02 smithi164 conmon[29288]: cluster 2021-02-10T04:27:01.968189+0000 mgr.y (mgr.14140) 257 : cluster [DBG] pgmap v229: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:27:02.603 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:02 smithi164 conmon[52406]: t=2021-02-10T04:27:02+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2021-02-10T04:27:02.603 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:02 smithi164 conmon[52406]: t=2021-02-10T04:27:02+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2021-02-10T04:27:02.603 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:02 smithi164 conmon[52406]: t=2021-02-10T04:27:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2021-02-10T04:27:02.603 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:02 smithi164 conmon[52406]: t=2021-02-10T04:27:02+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2021-02-10T04:27:02.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:02 smithi085 conmon[29319]: cluster 2021-02-10T04:27:01.184622+0000 osd.4 (osd.4) 10 : cluster 2021-02-10T04:27:02.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:02 smithi085 conmon[29319]: [DBG] 6.2 scrub ok 2021-02-10T04:27:02.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:02 smithi085 conmon[29319]: cluster 2021-02-10T04:27:01.968189+0000 mgr.y (mgr.14140) 257 : cluster [DBG] 2021-02-10T04:27:02.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:02 smithi085 conmon[29319]: pgmap v229: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:27:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:02 smithi085 conmon[35961]: cluster 2021-02-10T04:27:01.184622+0000 2021-02-10T04:27:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:02 smithi085 conmon[35961]: osd.4 (osd.4) 10 : cluster [DBG] 6.2 scrub ok 2021-02-10T04:27:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:02 smithi085 conmon[35961]: cluster 2021-02-10T04:27:01.968189+0000 mgr.y (mgr.14140) 257 : cluster [DBG] 2021-02-10T04:27:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:02 smithi085 conmon[35961]: pgmap v229: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:27:02.693 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 04:27:02 smithi085 podman[61354]: 3ea232dacd720787cb635c5023c5f355b7982d63b63abafd9e4cd91327b7fbdd 2021-02-10T04:27:02.693 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 04:27:02 smithi085 systemd[1]: ceph-42224fce-6b57-11eb-8feb-001a4aab830c@alertmanager.a.service: Succeeded. 2021-02-10T04:27:02.694 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 04:27:02 smithi085 systemd[1]: Stopped Ceph alertmanager.a for 42224fce-6b57-11eb-8feb-001a4aab830c. 2021-02-10T04:27:02.694 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 04:27:02 smithi085 systemd[1]: Starting Ceph alertmanager.a for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T04:27:02.879 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:02 smithi164 conmon[52406]: t=2021-02-10T04:27:02+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2021-02-10T04:27:02.879 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:02 smithi164 conmon[52406]: t=2021-02-10T04:27:02+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2021-02-10T04:27:02.879 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:02 smithi164 conmon[52406]: t=2021-02-10T04:27:02+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2021-02-10T04:27:02.879 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:02 smithi164 conmon[52406]: t=2021-02-10T04:27:02+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2021-02-10T04:27:03.143 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:02 smithi164 conmon[52406]: t=2021-02-10T04:27:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2021-02-10T04:27:03.143 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:03 smithi164 conmon[52406]: t=2021-02-10T04:27:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2021-02-10T04:27:03.143 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:03 smithi164 conmon[52406]: t=2021-02-10T04:27:03+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2021-02-10T04:27:03.465 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:03 smithi164 conmon[52406]: t=2021-02-10T04:27:03+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2021-02-10T04:27:03.465 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:03 smithi164 conmon[52406]: t=2021-02-10T04:27:03+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2021-02-10T04:27:03.465 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:03 smithi164 conmon[52406]: t=2021-02-10T04:27:03+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2021-02-10T04:27:03.465 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:03 smithi164 conmon[52406]: t=2021-02-10T04:27:03+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2021-02-10T04:27:03.466 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:03 smithi164 conmon[52406]: t=2021-02-10T04:27:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2021-02-10T04:27:03.745 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:03 smithi085 conmon[35961]: cluster 2021-02-10T04:27:02.039808+0000 2021-02-10T04:27:03.745 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:03 smithi085 conmon[35961]: osd.5 (osd.5) 10 : cluster [DBG] 4.8 scrub ok 2021-02-10T04:27:03.745 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:03 smithi085 conmon[35961]: cluster 2021-02-10T04:27:02.190047+0000 osd.6 (osd.6) 11 : cluster [DBG] 2.1a scrub ok 2021-02-10T04:27:03.745 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:03 smithi085 conmon[29319]: cluster 2021-02-10T04:27:02.039808+0000 2021-02-10T04:27:03.746 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:03 smithi085 conmon[29319]: osd.5 (osd.5) 10 : cluster [DBG] 4.8 scrub ok 2021-02-10T04:27:03.746 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:03 smithi085 conmon[29319]: cluster 2021-02-10T04:27:02.190047+0000 osd.6 (osd.6) 11 : cluster [DBG] 2.1a scrub ok 2021-02-10T04:27:03.758 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:03 smithi164 conmon[29288]: cluster 2021-02-10T04:27:02.039808+0000 osd.5 (osd.5) 10 : cluster 2021-02-10T04:27:03.758 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:03 smithi164 conmon[29288]: [DBG] 4.8 scrub ok 2021-02-10T04:27:03.758 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:03 smithi164 conmon[29288]: cluster 2021-02-10T04:27:02.190047+0000 osd.6 (osd.6) 11 : cluster [DBG] 2.1a scrub ok 2021-02-10T04:27:03.758 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:03 smithi164 conmon[52406]: t=2021-02-10T04:27:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2021-02-10T04:27:03.758 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:03 smithi164 conmon[52406]: t=2021-02-10T04:27:03+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2021-02-10T04:27:03.758 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:03 smithi164 conmon[52406]: t=2021-02-10T04:27:03+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2021-02-10T04:27:03.759 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:03 smithi164 conmon[52406]: t=2021-02-10T04:27:03+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2021-02-10T04:27:04.009 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:03 smithi164 conmon[52406]: t=2021-02-10T04:27:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2021-02-10T04:27:04.009 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:03 smithi164 conmon[52406]: t=2021-02-10T04:27:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2021-02-10T04:27:04.010 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:03 smithi164 conmon[52406]: t=2021-02-10T04:27:03+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2021-02-10T04:27:04.010 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[52406]: t=2021-02-10T04:27:04+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2021-02-10T04:27:04.193 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 04:27:03 smithi085 conmon[61668]: level=info ts=2021-02-10T04:27:03.745Z caller=main.go:231 msg="Starting Alertmanager" version="(version=0.20.0, branch=HEAD, revision=f74be0400a6243d10bb53812d6fa408ad71ff32d)" 2021-02-10T04:27:04.193 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 04:27:03 smithi085 conmon[61668]: level=info ts=2021-02-10T04:27:03.745Z caller=main.go:232 build_context="(go=go1.13.5, user=root@00c3106655f8, date=20191211-14:13:14)" 2021-02-10T04:27:04.193 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 04:27:03 smithi085 conmon[61668]: level=info ts=2021-02-10T04:27:03.746Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=172.21.15.85 port=9094 2021-02-10T04:27:04.193 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 04:27:03 smithi085 conmon[61668]: level=info ts=2021-02-10T04:27:03.747Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s 2021-02-10T04:27:04.193 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 04:27:03 smithi085 conmon[61668]: level=info ts=2021-02-10T04:27:03.765Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2021-02-10T04:27:04.193 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 04:27:03 smithi085 conmon[61668]: level=info ts=2021-02-10T04:27:03.766Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2021-02-10T04:27:04.193 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 04:27:03 smithi085 conmon[61668]: level=info ts=2021-02-10T04:27:03.768Z caller=main.go:497 msg=Listening address=:9093 2021-02-10T04:27:04.193 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 04:27:03 smithi085 bash[61534]: 6dd51e001cb902b709c2adf77ddc44b9a4beecc3f337400c5020afc4c2585882 2021-02-10T04:27:04.194 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 04:27:03 smithi085 systemd[1]: Started Ceph alertmanager.a for 42224fce-6b57-11eb-8feb-001a4aab830c. 2021-02-10T04:27:04.303 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[52406]: t=2021-02-10T04:27:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2021-02-10T04:27:04.303 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[52406]: t=2021-02-10T04:27:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2021-02-10T04:27:04.602 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[52406]: t=2021-02-10T04:27:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2021-02-10T04:27:04.602 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[52406]: t=2021-02-10T04:27:04+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2021-02-10T04:27:04.602 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[52406]: t=2021-02-10T04:27:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2021-02-10T04:27:04.602 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[29288]: cluster 2021-02-10T04:27:03.000814+0000 osd.5 (osd.5) 11 : cluster [DBG] 2021-02-10T04:27:04.602 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[29288]: 2.16 scrub ok 2021-02-10T04:27:04.602 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[29288]: cluster 2021-02-10T04:27:03.440776+0000 osd.0 (osd.0) 12 : cluster [DBG] 5.3 scrub ok 2021-02-10T04:27:04.602 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[29288]: 2021-02-10T04:27:04.602 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:27:04.602 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[29288]: 02-10T04:27:03.809979+0000 mon.a 2021-02-10T04:27:04.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[29288]: (mon.0) 664 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:04.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[29288]: cephadm 2021-02- 2021-02-10T04:27:04.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[29288]: 10T04:27:03.811188+0000 mgr.y 2021-02-10T04:27:04.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[29288]: (mgr.14140) 258 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2021-02-10T04:27:04.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[29288]: cephadm 2021-02-10T04:27:04.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[29288]: 2021-02-10T04:27 2021-02-10T04:27:04.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[29288]: :03.813354+0000 mgr.y (mgr.14140) 2021-02-10T04:27:04.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[29288]: 259 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi164 2021-02-10T04:27:04.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:27:04.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[29288]: -10T04:27:03.968651 2021-02-10T04:27:04.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 260 : cluster [DBG] 2021-02-10T04:27:04.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[29288]: pgmap v230: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:04.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:27:04.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[29288]: :27:04.024842+0000 osd.1 (osd. 2021-02-10T04:27:04.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[29288]: 1) 15 : cluster [DBG] 2.4 scrub ok 2021-02-10T04:27:04.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[29288]: cluster 2021-02-10T04:27:04.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[29288]: 2021-02-10T04:27 2021-02-10T04:27:04.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[29288]: :04.148324+0000 osd.7 ( 2021-02-10T04:27:04.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[29288]: osd.7) 10 : cluster [DBG] 2.8 scrub ok 2021-02-10T04:27:04.893 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[52406]: t=2021-02-10T04:27:04+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2021-02-10T04:27:04.893 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[52406]: t=2021-02-10T04:27:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2021-02-10T04:27:04.893 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[52406]: t=2021-02-10T04:27:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2021-02-10T04:27:04.893 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[52406]: t=2021-02-10T04:27:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2021-02-10T04:27:04.940 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:04 smithi085 conmon[29319]: cluster 2021-02-10T04:27:03.000814+0000 2021-02-10T04:27:04.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:04 smithi085 conmon[29319]: osd.5 (osd.5) 11 : cluster [DBG] 2.16 scrub ok 2021-02-10T04:27:04.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:04 smithi085 conmon[29319]: cluster 2021-02-10T04:27:03.440776+0000 osd.0 (osd.0) 12 : cluster [DBG] 5.3 scrub ok 2021-02-10T04:27:04.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:04 smithi085 conmon[29319]: audit 2021-02-10T04:27:03.809979+0000 mon.a (mon.0) 664 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:04.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:04 smithi085 conmon[29319]: cephadm 2021-02-10T04:27:03.811188+0000 mgr.y (mgr.14140) 258 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2021-02-10T04:27:04.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:04 smithi085 conmon[29319]: cephadm 2021-02-10T04:27:03.813354+0000 mgr.y (mgr.14140) 259 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi164 2021-02-10T04:27:04.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:04 smithi085 conmon[29319]: cluster 2021-02-10T04:27:03.968651+0000 mgr.y (mgr.14140) 260 : cluster [DBG] pgmap v230: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:04.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:04 smithi085 conmon[29319]: cluster 2021-02-10T04:27:04.024842+0000 osd.1 (osd.1) 15 : cluster [DBG] 2.4 scrub ok 2021-02-10T04:27:04.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:04 smithi085 conmon[29319]: cluster 2021-02-10T04:27:04.148324+0000 osd.7 (osd.7) 10 : cluster [DBG] 2.8 scrub ok 2021-02-10T04:27:04.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:04 smithi085 conmon[35961]: cluster 2021-02-10T04:27:03.000814+0000 osd.5 ( 2021-02-10T04:27:04.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:04 smithi085 conmon[35961]: osd.5) 11 : cluster [DBG] 2.16 scrub ok 2021-02-10T04:27:04.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:04 smithi085 conmon[35961]: cluster 2021-02-10T04:27:03.440776+0000 osd.0 (osd.0) 12 : cluster [DBG] 5.3 scrub ok 2021-02-10T04:27:04.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:04 smithi085 conmon[35961]: audit 2021-02-10T04:27:03.809979+0000 mon.a (mon.0) 664 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:04.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:04 smithi085 conmon[35961]: cephadm 2021-02-10T04:27:03.811188+0000 mgr.y (mgr.14140) 258 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2021-02-10T04:27:04.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:04 smithi085 conmon[35961]: cephadm 2021-02-10T04:27:03.813354+0000 mgr.y (mgr.14140) 259 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi164 2021-02-10T04:27:04.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:04 smithi085 conmon[35961]: cluster 2021-02-10T04:27:03.968651+0000 mgr.y (mgr.14140) 260 : cluster [DBG] pgmap v230: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:04.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:04 smithi085 conmon[35961]: cluster 2021-02-10T04:27:04.024842+0000 osd.1 (osd.1) 15 : cluster [DBG] 2.4 scrub ok 2021-02-10T04:27:04.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:04 smithi085 conmon[35961]: cluster 2021-02-10T04:27:04.148324+0000 osd.7 (osd.7) 10 : cluster [DBG] 2.8 scrub ok 2021-02-10T04:27:05.073 INFO:teuthology.orchestra.run.smithi085.stdout:77309411367 2021-02-10T04:27:05.074 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph tell osd.3 flush_pg_stats 2021-02-10T04:27:05.209 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:04 smithi164 conmon[52406]: t=2021-02-10T04:27:04+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2021-02-10T04:27:05.209 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:05 smithi164 conmon[52406]: t=2021-02-10T04:27:05+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2021-02-10T04:27:05.275 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:27:05.499 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:05 smithi164 conmon[52406]: t=2021-02-10T04:27:05+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2021-02-10T04:27:05.499 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:05 smithi164 conmon[52406]: t=2021-02-10T04:27:05+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2021-02-10T04:27:05.846 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:05 smithi085 conmon[29319]: cluster 2021-02-10T04:27:04.011046+0000 osd.5 (osd.5) 12 : cluster [DBG] 2.18 scrub ok 2021-02-10T04:27:05.846 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:05 smithi085 conmon[29319]: cluster 2021-02-10T04:27:04.947010+0000 osd.2 (osd.2) 15 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:27:05.846 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:05 smithi085 conmon[29319]: cluster 2021-02-10T04:27:04.947114+0000 osd.2 (osd.2) 16 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:27:05.846 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:05 smithi085 conmon[35961]: cluster 2021-02-10T04:27:04.011046+0000 osd.5 (osd.5) 12 : cluster [DBG] 2.18 scrub ok 2021-02-10T04:27:05.846 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:05 smithi085 conmon[35961]: cluster 2021-02-10T04:27:04.947010+0000 osd.2 (osd.2) 15 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:27:05.846 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:05 smithi085 conmon[35961]: cluster 2021-02-10T04:27:04.947114+0000 osd.2 (osd.2) 16 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:27:05.847 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 04:27:05 smithi085 conmon[61668]: level=info ts=2021-02-10T04:27:05.747Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000071472s 2021-02-10T04:27:05.889 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:05 smithi164 conmon[29288]: cluster 2021-02-10T04:27:04.011046+0000 osd.5 (osd. 2021-02-10T04:27:05.889 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:05 smithi164 conmon[29288]: 5) 12 : cluster [DBG] 2.18 scrub ok 2021-02-10T04:27:05.889 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:05 smithi164 conmon[29288]: cluster 2021-02-10T04:27:04.947010+0000 osd.2 (osd 2021-02-10T04:27:05.889 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:05 smithi164 conmon[29288]: .2) 15 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:27:05.889 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:05 smithi164 conmon[29288]: cluster 2021-02-10T04:27:04 2021-02-10T04:27:05.889 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:05 smithi164 conmon[29288]: .947114+0000 osd.2 (osd.2) 16 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:27:05.889 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:05 smithi164 conmon[29288]: 2021-02-10T04:27:06.143 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:05 smithi164 conmon[52406]: t=2021-02-10T04:27:05+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2021-02-10T04:27:06.429 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:06 smithi164 conmon[52406]: t=2021-02-10T04:27:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2021-02-10T04:27:06.429 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:06 smithi164 conmon[52406]: t=2021-02-10T04:27:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2021-02-10T04:27:06.429 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:06 smithi164 conmon[52406]: t=2021-02-10T04:27:06+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2021-02-10T04:27:06.429 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:06 smithi164 conmon[52406]: t=2021-02-10T04:27:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2021-02-10T04:27:06.756 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:06 smithi164 conmon[52406]: t=2021-02-10T04:27:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2021-02-10T04:27:06.756 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:06 smithi164 conmon[52406]: t=2021-02-10T04:27:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2021-02-10T04:27:06.756 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:06 smithi164 conmon[29288]: cluster 2021-02-10T04:27:04.977347+0000 osd.5 (osd.5) 13 : cluster [DBG] 5.1b scrub ok 2021-02-10T04:27:06.756 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:06 smithi164 conmon[29288]: cluster 2021-02-10T04:27:05.242749+0000 osd.4 (osd.4) 11 : cluster [DBG] 4.1a scrub ok 2021-02-10T04:27:06.756 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:06 smithi164 conmon[29288]: cluster 2021-02-10T04:27:05 2021-02-10T04:27:06.757 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:06 smithi164 conmon[29288]: .297641+0000 osd.3 (osd.3) 2021-02-10T04:27:06.757 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:06 smithi164 conmon[29288]: 12 : cluster [DBG] 5.13 deep-scrub ok 2021-02-10T04:27:06.757 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:06 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:27:06.757 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:06 smithi164 conmon[29288]: 02-10T04:27:05.812710+0000 2021-02-10T04:27:06.757 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:06 smithi164 conmon[29288]: mon.a (mon.0) 665 : 2021-02-10T04:27:06.757 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:06 smithi164 conmon[29288]: cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2021-02-10T04:27:06.757 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:06 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:27:06.757 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:06 smithi164 conmon[29288]: -02-10T04:27:05. 2021-02-10T04:27:06.757 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:06 smithi164 conmon[29288]: 968993+0000 mgr.y (mgr.14140) 261 2021-02-10T04:27:06.757 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:06 smithi164 conmon[29288]: : cluster [DBG] pgmap v232: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:27:06.760 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:06 smithi085 conmon[29319]: cluster 2021-02-10T04:27:04.977347+0000 osd.5 (osd.5) 13 2021-02-10T04:27:06.761 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:06 smithi085 conmon[29319]: : cluster [DBG] 5.1b scrub ok 2021-02-10T04:27:06.761 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:06 smithi085 conmon[29319]: cluster 2021-02-10T04:27:05.242749+0000 osd.4 (osd.4) 11 : cluster [DBG] 4.1a scrub ok 2021-02-10T04:27:06.761 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:06 smithi085 conmon[29319]: cluster 2021-02-10T04:27:05.297641+0000 osd.3 (osd.3) 12 : cluster [DBG] 5.13 deep-scrub ok 2021-02-10T04:27:06.761 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:06 smithi085 conmon[29319]: cluster 2021-02-10T04:27:05.812710+0000 mon.a (mon.0) 665 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2021-02-10T04:27:06.761 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:06 smithi085 conmon[29319]: cluster 2021-02-10T04:27:05.968993+0000 mgr.y (mgr.14140) 261 : cluster [DBG] 2021-02-10T04:27:06.761 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:06 smithi085 conmon[29319]: pgmap v232: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:27:06.761 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:06 smithi085 conmon[35961]: cluster 2021-02-10T04:27:04.977347+0000 osd.5 (osd.5) 13 2021-02-10T04:27:06.761 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:06 smithi085 conmon[35961]: : cluster [DBG] 5.1b scrub ok 2021-02-10T04:27:06.761 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:06 smithi085 conmon[35961]: cluster 2021-02-10T04:27:05.242749+0000 osd.4 (osd.4 2021-02-10T04:27:06.761 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:06 smithi085 conmon[35961]: ) 11 : cluster [DBG] 4.1a scrub ok 2021-02-10T04:27:06.761 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:06 smithi085 conmon[35961]: cluster 2021-02-10T04:27:06.761 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:06 smithi085 conmon[35961]: 2021-02-10T04:27:05.297641+0000 osd.3 (osd.3 2021-02-10T04:27:06.762 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:06 smithi085 conmon[35961]: ) 12 : cluster [DBG] 5.13 deep-scrub ok 2021-02-10T04:27:06.762 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:06 smithi085 conmon[35961]: cluster 2021-02-10T04:27:05 2021-02-10T04:27:06.762 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:06 smithi085 conmon[35961]: .812710+0000 mon.a (mon.0) 665 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2021-02-10T04:27:06.762 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:06 smithi085 conmon[35961]: 2021-02-10T04:27:06.762 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:06 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:27:06.762 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:06 smithi085 conmon[35961]: -02-10T04:27:05.968993+0000 mgr.y (mgr.14140) 261 : cluster 2021-02-10T04:27:06.762 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:06 smithi085 conmon[35961]: [DBG] pgmap v232: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:27:07.032 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:06 smithi164 conmon[52406]: t=2021-02-10T04:27:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2021-02-10T04:27:07.032 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:06 smithi164 conmon[52406]: t=2021-02-10T04:27:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2021-02-10T04:27:07.032 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:06 smithi164 conmon[52406]: t=2021-02-10T04:27:06+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2021-02-10T04:27:07.032 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:07 smithi164 conmon[52406]: t=2021-02-10T04:27:07+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2021-02-10T04:27:07.300 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:07 smithi164 conmon[52406]: t=2021-02-10T04:27:07+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2021-02-10T04:27:07.554 INFO:teuthology.orchestra.run.smithi085.stdout:107374182434 2021-02-10T04:27:07.555 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph tell osd.4 flush_pg_stats 2021-02-10T04:27:07.581 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:07 smithi164 conmon[52406]: t=2021-02-10T04:27:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2021-02-10T04:27:07.581 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:07 smithi164 conmon[52406]: t=2021-02-10T04:27:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2021-02-10T04:27:07.748 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:27:07.844 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:27:07 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:27:07.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:07 smithi164 conmon[29288]: cluster 2021-02-10T04:27:06.934278+0000 osd.5 (osd. 2021-02-10T04:27:07.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:07 smithi164 conmon[29288]: 5) 14 : cluster [DBG] 2.2 scrub ok 2021-02-10T04:27:07.844 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:07 smithi164 conmon[52406]: t=2021-02-10T04:27:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2021-02-10T04:27:07.844 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:07 smithi164 conmon[52406]: t=2021-02-10T04:27:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2021-02-10T04:27:08.129 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:07 smithi164 conmon[52406]: t=2021-02-10T04:27:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2021-02-10T04:27:08.130 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[52406]: t=2021-02-10T04:27:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2021-02-10T04:27:08.130 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[52406]: t=2021-02-10T04:27:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2021-02-10T04:27:08.130 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:07 smithi164 systemd[1]: Stopping Ceph prometheus.a for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T04:27:08.130 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[50601]: level=warn ts=2021-02-10T04:27:08.054Z caller=main.go:524 msg="Received SIGTERM, exiting gracefully..." 2021-02-10T04:27:08.130 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[50601]: level=info ts=2021-02-10T04:27:08.054Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2021-02-10T04:27:08.130 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[50601]: level=info ts=2021-02-10T04:27:08.054Z caller=main.go:561 msg="Stopping notify discovery manager..." 2021-02-10T04:27:08.130 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[50601]: level=info ts=2021-02-10T04:27:08.054Z caller=main.go:583 msg="Stopping scrape manager..." 2021-02-10T04:27:08.130 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[50601]: level=info ts=2021-02-10T04:27:08.054Z caller=main.go:543 msg="Scrape discovery manager stopped" 2021-02-10T04:27:08.130 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[50601]: level=info ts=2021-02-10T04:27:08.054Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2021-02-10T04:27:08.130 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[50601]: level=info ts=2021-02-10T04:27:08.054Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2021-02-10T04:27:08.131 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[50601]: level=info ts=2021-02-10T04:27:08.054Z caller=main.go:557 msg="Notify discovery manager stopped" 2021-02-10T04:27:08.131 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[50601]: level=info ts=2021-02-10T04:27:08.054Z caller=main.go:577 msg="Scrape manager stopped" 2021-02-10T04:27:08.131 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[50601]: level=info ts=2021-02-10T04:27:08.054Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2021-02-10T04:27:08.131 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[50601]: level=info ts=2021-02-10T04:27:08.054Z caller=main.go:749 msg="Notifier manager stopped" 2021-02-10T04:27:08.131 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[50601]: level=info ts=2021-02-10T04:27:08.054Z caller=main.go:761 msg="See you next time!" 2021-02-10T04:27:08.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:07 smithi085 conmon[29319]: cluster 2021-02-10T04:27:06.934278+0000 2021-02-10T04:27:08.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:07 smithi085 conmon[29319]: osd.5 (osd.5) 14 : cluster [DBG] 2.2 scrub ok 2021-02-10T04:27:08.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:07 smithi085 conmon[35961]: cluster 2021-02-10T04:27:06.934278 2021-02-10T04:27:08.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:07 smithi085 conmon[35961]: +0000 osd.5 (osd.5) 14 : cluster [DBG] 2.2 scrub ok 2021-02-10T04:27:08.393 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[52406]: t=2021-02-10T04:27:08+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2021-02-10T04:27:08.393 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[52406]: t=2021-02-10T04:27:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2021-02-10T04:27:08.781 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[52406]: t=2021-02-10T04:27:08+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2021-02-10T04:27:09.043 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[52406]: t=2021-02-10T04:27:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2021-02-10T04:27:09.043 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[52406]: t=2021-02-10T04:27:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2021-02-10T04:27:09.043 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:08 smithi164 podman[53619]: Error: container 8edab098c1831a11b049f1a6e6632273c3ffdd99543385935c3842c0ba97d32e does not exist in database: no such container 2021-02-10T04:27:09.043 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:08 smithi164 systemd[1]: ceph-42224fce-6b57-11eb-8feb-001a4aab830c@prometheus.a.service: Succeeded. 2021-02-10T04:27:09.043 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:08 smithi164 systemd[1]: Stopped Ceph prometheus.a for 42224fce-6b57-11eb-8feb-001a4aab830c. 2021-02-10T04:27:09.043 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:08 smithi164 systemd[1]: Starting Ceph prometheus.a for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T04:27:09.043 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[29288]: cluster 2021-02-10T04:27:07.181848+0000 osd.7 (osd.7) 11 : cluster [DBG] 5.10 scrub ok 2021-02-10T04:27:09.043 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[29288]: cluster 2021-02-10T04:27:07.339537+0000 osd.3 (osd.3) 13 : cluster [DBG] 6.1b scrub ok 2021-02-10T04:27:09.043 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:27:09.044 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[29288]: :27:07.581747+0000 mgr.y (mgr.14140) 262 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:27:09.044 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:27:09.044 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[29288]: 02-10T04:27:07.962106+0000 2021-02-10T04:27:09.044 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[29288]: osd.5 (osd.5) 15 : cluster [DBG] 6.19 scrub ok 2021-02-10T04:27:09.044 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[29288]: cluster 2021-02-10T04:27:09.044 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[29288]: 2021-02-10T04:27: 2021-02-10T04:27:09.044 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[29288]: 07.969581+0000 mgr.y (mgr.14140) 263 : 2021-02-10T04:27:09.044 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[29288]: cluster [DBG] pgmap v233: 132 pgs: 132 active+clean; 25 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 539 B/s rd, 0 op/s 2021-02-10T04:27:09.044 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:27:09.044 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[29288]: 04:27:08.069892+0000 osd.6 (osd. 2021-02-10T04:27:09.044 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:08 smithi164 conmon[29288]: 6) 12 : cluster [DBG] 4.5 scrub ok 2021-02-10T04:27:09.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:08 smithi085 conmon[29319]: cluster 2021-02-10T04:27:07.181848+0000 osd.7 (osd.7) 11 : cluster [DBG] 5.10 scrub ok 2021-02-10T04:27:09.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:08 smithi085 conmon[29319]: cluster 2021-02-10T04:27:07.339537+0000 osd.3 (osd.3) 13 : cluster [DBG] 6.1b scrub ok 2021-02-10T04:27:09.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:08 smithi085 conmon[29319]: audit 2021-02-10T04:27:07.581747+0000 mgr.y (mgr.14140) 262 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:27:09.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:08 smithi085 conmon[29319]: cluster 2021-02-10T04:27:07.962106+0000 osd.5 (osd.5) 15 : cluster [DBG] 6.19 scrub ok 2021-02-10T04:27:09.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:08 smithi085 conmon[29319]: cluster 2021-02-10T04:27:07.969581+0000 mgr.y (mgr.14140) 263 : cluster [DBG] pgmap v233: 132 pgs: 132 active+clean; 25 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 539 B/s rd, 0 op/s 2021-02-10T04:27:09.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:08 smithi085 conmon[29319]: cluster 2021-02-10T04:27:08.069892+0000 osd.6 (osd.6) 12 : cluster [DBG] 4.5 scrub ok 2021-02-10T04:27:09.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:08 smithi085 conmon[35961]: cluster 2021-02-10T04:27:07.181848+0000 osd.7 (osd.7) 11 : cluster [DBG] 5.10 scrub ok 2021-02-10T04:27:09.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:08 smithi085 conmon[35961]: cluster 2021-02-10T04:27:07.339537+0000 osd.3 (osd.3) 13 : cluster [DBG] 6.1b scrub ok 2021-02-10T04:27:09.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:08 smithi085 conmon[35961]: audit 2021-02-10T04:27:07.581747+0000 mgr.y (mgr.14140) 262 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:27:09.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:08 smithi085 conmon[35961]: cluster 2021-02-10T04:27:07.962106+0000 osd.5 (osd.5) 15 : cluster [DBG] 6.19 scrub ok 2021-02-10T04:27:09.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:08 smithi085 conmon[35961]: cluster 2021-02-10T04:27:07.969581+0000 mgr.y (mgr.14140) 263 : cluster [DBG] pgmap v233: 132 pgs: 132 active+clean; 25 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 539 B/s rd, 0 op/s 2021-02-10T04:27:09.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:08 smithi085 conmon[35961]: cluster 2021-02-10T04:27:08.069892+0000 osd.6 (osd.6) 12 : cluster [DBG] 4.5 scrub ok 2021-02-10T04:27:09.355 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[52406]: t=2021-02-10T04:27:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2021-02-10T04:27:09.355 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[52406]: t=2021-02-10T04:27:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2021-02-10T04:27:09.355 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[52406]: t=2021-02-10T04:27:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2021-02-10T04:27:09.631 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[52406]: t=2021-02-10T04:27:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2021-02-10T04:27:09.632 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[52406]: t=2021-02-10T04:27:09+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2021-02-10T04:27:09.632 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[52406]: t=2021-02-10T04:27:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2021-02-10T04:27:09.632 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[52406]: t=2021-02-10T04:27:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2021-02-10T04:27:09.632 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[52406]: t=2021-02-10T04:27:09+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2021-02-10T04:27:09.893 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[52406]: t=2021-02-10T04:27:09+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2021-02-10T04:27:09.893 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[52406]: t=2021-02-10T04:27:09+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2021-02-10T04:27:09.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[29288]: cluster 2021-02-10T04:27:08.192766+0000 osd.7 (osd.7) 12 : 2021-02-10T04:27:09.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[29288]: cluster [DBG] 2.0 scrub ok 2021-02-10T04:27:09.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[29288]: cluster 2021-02-10T04:27:08.486548+0000 osd.0 (osd.0) 13 : cluster [DBG] 6.16 scrub ok 2021-02-10T04:27:09.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[29288]: audit 2021-02-10T04:27:09.800468+0000 2021-02-10T04:27:09.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[29288]: mon.a (mon.0) 666 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:09.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[29288]: audit 2021-02-10T04:27: 2021-02-10T04:27:09.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[29288]: 09.802971+0000 mon.a (mon.0) 667 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-02-10T04:27:09.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[29288]: audit 2021-02-10T04:27:09 2021-02-10T04:27:09.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[29288]: .803968+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi085:9093"}]: dispatch 2021-02-10T04:27:09.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:27:09.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[29288]: T04:27:09.807856+0000 mon.a (mon.0) 669 : audit 2021-02-10T04:27:09.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:09.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[29288]: audit 2021-02-10T04:27:09.810593+0000 mon.a (mon 2021-02-10T04:27:09.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[29288]: .0) 670 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2021-02-10T04:27:09.895 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[53830]: level=info ts=2021-02-10T04:27:09.737Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2021-02-10T04:27:09.895 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[53830]: level=info ts=2021-02-10T04:27:09.737Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2021-02-10T04:27:09.895 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[53830]: level=info ts=2021-02-10T04:27:09.737Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2021-02-10T04:27:09.895 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[53830]: level=info ts=2021-02-10T04:27:09.737Z caller=main.go:339 host_details="(Linux 4.18.0-240.1.1.el8_3.x86_64 #1 SMP Fri Oct 16 13:36:46 EDT 2020 x86_64 smithi164 (none))" 2021-02-10T04:27:09.895 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[53830]: level=info ts=2021-02-10T04:27:09.737Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2021-02-10T04:27:09.895 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[53830]: level=info ts=2021-02-10T04:27:09.737Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2021-02-10T04:27:09.895 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[53830]: level=info ts=2021-02-10T04:27:09.738Z caller=main.go:678 msg="Starting TSDB ..." 2021-02-10T04:27:09.895 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[53830]: level=info ts=2021-02-10T04:27:09.738Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2021-02-10T04:27:09.895 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[53830]: level=info ts=2021-02-10T04:27:09.740Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2021-02-10T04:27:09.895 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[53830]: level=info ts=2021-02-10T04:27:09.741Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2021-02-10T04:27:09.895 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[53830]: level=info ts=2021-02-10T04:27:09.741Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2021-02-10T04:27:09.896 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[53830]: level=info ts=2021-02-10T04:27:09.741Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=1.027754ms 2021-02-10T04:27:09.896 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[53830]: level=info ts=2021-02-10T04:27:09.742Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2021-02-10T04:27:09.896 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[53830]: level=info ts=2021-02-10T04:27:09.742Z caller=main.go:695 msg="TSDB started" 2021-02-10T04:27:09.896 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[53830]: level=info ts=2021-02-10T04:27:09.742Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2021-02-10T04:27:09.896 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[53830]: level=info ts=2021-02-10T04:27:09.743Z caller=main.go:827 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml 2021-02-10T04:27:09.896 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[53830]: level=info ts=2021-02-10T04:27:09.743Z caller=main.go:646 msg="Server is ready to receive web requests." 2021-02-10T04:27:09.896 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:09 smithi164 bash[53720]: 1a5a35ea14bbab32595a820c00d6379968a3188057920160a4463fd9c3ce8785 2021-02-10T04:27:09.896 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 04:27:09 smithi164 systemd[1]: Started Ceph prometheus.a for 42224fce-6b57-11eb-8feb-001a4aab830c. 2021-02-10T04:27:09.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:09 smithi085 conmon[35961]: cluster 2021-02-10T04:27:08.192766+0000 osd.7 (osd.7) 2021-02-10T04:27:09.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:09 smithi085 conmon[35961]: 12 : cluster [DBG] 2.0 scrub ok 2021-02-10T04:27:09.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:09 smithi085 conmon[35961]: cluster 2021-02-10T04:27:08.486548+0000 osd.0 (osd.0) 13 : cluster [DBG] 6.16 scrub ok 2021-02-10T04:27:09.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:09 smithi085 conmon[35961]: audit 2021-02-10T04:27:09.800468+0000 mon.a (mon.0) 666 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:09.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:09 smithi085 conmon[35961]: audit 2021-02-10T04:27:09.802971+0000 mon.a (mon.0) 667 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-02-10T04:27:09.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:09 smithi085 conmon[35961]: audit 2021-02-10T04:27:09.803968+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi085:9093"}]: dispatch 2021-02-10T04:27:09.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:09 smithi085 conmon[35961]: audit 2021-02-10T04:27:09.807856+0000 mon.a (mon.0) 669 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:09.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:09 smithi085 conmon[35961]: audit 2021-02-10T04:27:09.810593+0000 mon.a (mon.0) 670 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2021-02-10T04:27:09.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:09 smithi085 conmon[29319]: cluster 2021-02-10T04:27:08.192766+0000 osd.7 (osd.7 2021-02-10T04:27:09.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:09 smithi085 conmon[29319]: ) 12 : cluster [DBG] 2.0 scrub ok 2021-02-10T04:27:09.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:09 smithi085 conmon[29319]: cluster 2021-02-10T04:27:08.486548+0000 osd.0 (osd.0) 13 : cluster [DBG] 6.16 scrub ok 2021-02-10T04:27:09.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:09 smithi085 conmon[29319]: audit 2021-02-10T04:27:09.800468+0000 mon.a (mon.0) 666 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:09.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:09 smithi085 conmon[29319]: audit 2021-02-10T04:27:09.802971+0000 mon.a (mon.0) 667 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-02-10T04:27:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:09 smithi085 conmon[29319]: audit 2021-02-10T04:27:09.803968+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi085:9093"}]: dispatch 2021-02-10T04:27:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:09 smithi085 conmon[29319]: audit 2021-02-10T04:27:09.807856+0000 mon.a (mon.0) 669 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:09 smithi085 conmon[29319]: audit 2021-02-10T04:27:09.810593+0000 mon.a (mon.0) 670 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2021-02-10T04:27:10.105 INFO:teuthology.orchestra.run.smithi085.stdout:133143986208 2021-02-10T04:27:10.105 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph tell osd.5 flush_pg_stats 2021-02-10T04:27:10.226 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[52406]: t=2021-02-10T04:27:09+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2021-02-10T04:27:10.461 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:09 smithi164 conmon[52406]: t=2021-02-10T04:27:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2021-02-10T04:27:10.461 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[52406]: t=2021-02-10T04:27:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2021-02-10T04:27:10.462 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[52406]: t=2021-02-10T04:27:10+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2021-02-10T04:27:10.462 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[52406]: t=2021-02-10T04:27:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2021-02-10T04:27:10.462 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:27:10.494 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[52406]: t=2021-02-10T04:27:10+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2021-02-10T04:27:10.494 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[52406]: t=2021-02-10T04:27:10+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2021-02-10T04:27:10.494 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[52406]: t=2021-02-10T04:27:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2021-02-10T04:27:10.750 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[52406]: t=2021-02-10T04:27:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2021-02-10T04:27:10.751 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[52406]: t=2021-02-10T04:27:10+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2021-02-10T04:27:10.991 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: cluster 2021-02-10T04:27:09.170421+0000 osd.4 (osd.4) 12 : cluster [DBG] 4.18 deep-scrub ok 2021-02-10T04:27:10.992 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: cluster 2021-02-10T04:27:09.483982+0000 osd.0 (osd.0) 14 : cluster [DBG] 4.6 scrub ok 2021-02-10T04:27:10.992 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: audit 2021-02-10T04:27:09.803250+0000 mgr.y (mgr.14140) 264 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-02-10T04:27:10.992 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: audit 2021-02-10T04:27:09.804184+0000 mgr.y (mgr.14140) 265 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi085:9093"}]: dispatch 2021-02-10T04:27:10.992 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: audit 2021-02-10T04:27:09.810998+0000 mgr.y (mgr.14140) 266 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2021-02-10T04:27:10.992 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: cephadm 2021-02-10T04:27:09.817267+0000 mgr.y (mgr.14140) 267 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.164:5000 to Dashboard 2021-02-10T04:27:10.992 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: audit 2021-02-10T04:27:09.819894+0000 mon.a (mon.0) 671 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2021-02-10T04:27:10.992 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: audit 2021-02-10T04:27:09.820219+0000 mgr.y (mgr.14140) 268 : audit 2021-02-10T04:27:10.992 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2021-02-10T04:27:10.993 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: audit 2021-02-10T04:27:09.823439+0000 mon.a (mon.0) 672 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:10.993 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: audit 2021-02-10T04:27:09.825384+0000 mon.a (mon.0) 673 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi164"}]: dispatch 2021-02-10T04:27:10.993 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: audit 2021-02-10T04:27:09.825798+0000 mgr.y (mgr.14140) 269 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi164"}]: dispatch 2021-02-10T04:27:10.993 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: audit 2021-02-10T04:27:09.832849+0000 mon.a (mon.0) 674 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:10.993 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: audit 2021-02-10T04:27:09.835300+0000 mon.a (mon.0) 675 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-02-10T04:27:10.993 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: audit 2021-02-10T04:27:09.835605+0000 mgr.y (mgr.14140) 270 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-02-10T04:27:10.993 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: audit 2021-02-10T04:27:09.836197+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi164:9095"}]: dispatch 2021-02-10T04:27:10.993 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: audit 2021-02-10T04:27:09.836449+0000 mgr.y (mgr.14140) 271 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi164:9095"}]: dispatch 2021-02-10T04:27:10.993 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: audit 2021-02-10T04:27:09.841330+0000 mon.a (mon.0) 677 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:10.993 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: audit 2021-02-10T04:27:09.845081+0000 mon.a (mon.0) 678 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-02-10T04:27:10.994 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: audit 2021-02-10T04:27:09.845502+0000 mgr.y (mgr.14140) 272 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-02-10T04:27:10.994 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: audit 2021-02-10T04:27:09.849083+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi164:3000"}]: dispatch 2021-02-10T04:27:10.994 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: audit 2021-02-10T04:27:09.849315+0000 mgr.y (mgr.14140) 273 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi164:3000"}]: dispatch 2021-02-10T04:27:10.994 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: audit 2021-02-10T04:27:10.994 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: 2021-02-10T04:27:09.854373+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:10.995 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: audit 2021-02-10T04:27:09.857006+0000 mon.a (mon.0) 681 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:27:10.995 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: audit 2021-02-10T04:27 2021-02-10T04:27:10.995 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: :09.857199+0000 mon.a (mon.0) 682 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:27:10.995 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:27:10.995 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: 27:09.970117+0000 mgr.y (mgr.14140) 274 : 2021-02-10T04:27:10.995 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: cluster [DBG] pgmap v234: 132 pgs: 132 active+clean; 25 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:10.995 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:27:10.995 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: 27:10.051796+0000 osd.1 (osd.1) 2021-02-10T04:27:10.995 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[29319]: 16 : cluster [DBG] 6.4 scrub ok 2021-02-10T04:27:10.995 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: cluster 2021-02-10T04:27:09.170421+0000 2021-02-10T04:27:10.996 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: osd.4 (osd.4) 12 : cluster [DBG] 4.18 deep-scrub ok 2021-02-10T04:27:10.996 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: cluster 2021-02-10T04:27:09.483982+0000 osd.0 (osd.0) 14 : cluster [DBG] 4.6 scrub ok 2021-02-10T04:27:10.996 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: audit 2021-02-10T04:27:09.803250+0000 mgr.y (mgr.14140) 264 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-02-10T04:27:10.996 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: audit 2021-02-10T04:27:09.804184+0000 mgr.y (mgr.14140) 265 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi085:9093"}]: dispatch 2021-02-10T04:27:10.996 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: audit 2021-02-10T04:27:09.810998+0000 mgr.y (mgr.14140) 266 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2021-02-10T04:27:10.996 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: cephadm 2021-02-10T04:27:09.817267+0000 mgr.y (mgr.14140) 267 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.164:5000 to Dashboard 2021-02-10T04:27:10.996 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: audit 2021-02-10T04:27:09.819894+0000 mon.a (mon.0) 671 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2021-02-10T04:27:10.996 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: audit 2021-02-10T04:27:09.820219+0000 mgr.y (mgr.14140) 268 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2021-02-10T04:27:10.996 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: audit 2021-02-10T04:27:09.823439+0000 mon.a (mon.0) 672 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:10.996 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:27:10.996 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: 02-10T04:27:09.825384+0000 mon.a (mon.0) 673 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi164"}]: dispatch 2021-02-10T04:27:10.996 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: audit 2021-02-10T04:27:09.825798+0000 mgr.y (mgr.14140) 269 : audit 2021-02-10T04:27:10.996 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi164"}]: dispatch 2021-02-10T04:27:10.997 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: audit 2021-02-10T04:27:10.997 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: 2021-02-10T04:27 2021-02-10T04:27:10.997 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: :09.832849+0000 mon.a (mon.0) 2021-02-10T04:27:10.997 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: 674 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:10.997 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:27:10.997 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: -10T04:27:09.835300+0000 2021-02-10T04:27:10.997 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: mon.a (mon.0) 675 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-02-10T04:27:10.997 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:27:10.997 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: 02-10T04:27:09.835605+0000 2021-02-10T04:27:10.997 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: mgr.y (mgr.14140) 270 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-02-10T04:27:10.997 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: 2021-02-10T04:27:10.997 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: audit 2021-02-10T04:27:09 2021-02-10T04:27:10.998 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: .836197+0000 mon.a (mon.0) 676 : 2021-02-10T04:27:10.999 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi164:9095"}]: dispatch 2021-02-10T04:27:10.999 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:27:10.999 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: 04:27:09.836449+0000 mgr.y (mgr.14140) 271 : audit [DBG] 2021-02-10T04:27:10.999 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi164:9095"}]: dispatch 2021-02-10T04:27:10.999 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:27:10.999 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: -10T04:27:09.841330+0000 mon.a (mon.0) 677 2021-02-10T04:27:10.999 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:10.999 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: audit 2021-02-10T04:27:09.845081+0000 mon.a 2021-02-10T04:27:10.999 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: (mon.0) 678 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-02-10T04:27:11.000 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: audit 2021 2021-02-10T04:27:11.000 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: -02-10T04:27:09.845502+0000 mgr.y 2021-02-10T04:27:11.000 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: (mgr.14140) 272 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-02-10T04:27:11.000 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:27:11.000 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: 27:09.849083+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi164:3000"}]: dispatch 2021-02-10T04:27:11.000 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: audit 2021-02-10T04:27:09 2021-02-10T04:27:11.000 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: .849315+0000 mgr.y (mgr.14140) 273 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi164:3000"}]: dispatch 2021-02-10T04:27:11.000 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: audit 2021-02-10T04:27:09. 2021-02-10T04:27:11.000 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: 854373+0000 mon.a (mon.0) 680 : audit [INF] 2021-02-10T04:27:11.000 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:11.000 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: audit 2021-02-10T04:27:11.000 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: 2021-02-10T04:27:09.857006+0000 mon.a ( 2021-02-10T04:27:11.000 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: mon.0) 681 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:27:11.001 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:27:11.001 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: -10T04:27:09.857199+0000 mon.a 2021-02-10T04:27:11.001 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: (mon.0) 682 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:27:11.001 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:27:11.001 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: -10T04:27:09.970117+0000 mgr.y 2021-02-10T04:27:11.001 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: (mgr.14140) 274 : cluster [DBG] pgmap v234: 132 pgs: 132 active+clean; 25 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:11.001 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:27:11.001 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: 02-10T04:27:10.051796+0000 2021-02-10T04:27:11.001 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: osd.1 (osd.1) 16 : cluster [DBG] 6.4 scrub ok 2021-02-10T04:27:11.001 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:10 smithi085 conmon[35961]: 2021-02-10T04:27:11.030 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[52406]: t=2021-02-10T04:27:10+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2021-02-10T04:27:11.031 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[52406]: t=2021-02-10T04:27:10+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2021-02-10T04:27:11.031 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: cluster 2021-02-10T04:27:09.170421+0000 osd.4 (osd.4) 12 : 2021-02-10T04:27:11.031 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: cluster [DBG] 4.18 deep-scrub ok 2021-02-10T04:27:11.032 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: cluster 2021-02-10T04:27:09.483982+0000 osd.0 (osd.0) 14 : cluster [DBG] 4.6 scrub ok 2021-02-10T04:27:11.032 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: audit 2021 2021-02-10T04:27:11.032 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: -02-10T04:27:09.803250+0000 mgr.y (mgr.14140) 264 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-02-10T04:27:11.032 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:27:11.032 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: 02-10T04:27:09.804184+0000 2021-02-10T04:27:11.032 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: mgr.y (mgr.14140) 265 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi085:9093"}]: dispatch 2021-02-10T04:27:11.032 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: 2021-02-10T04:27:11.032 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: audit 2021-02-10T04:27:11.032 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: 2021-02-10T04:27:09.810998+0000 2021-02-10T04:27:11.032 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: mgr.y (mgr.14140) 266 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2021-02-10T04:27:11.032 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: 2021-02-10T04:27:11.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: cephadm 2021-02-10T04:27:11.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: 2021-02-10T04:27:09.817267+0000 2021-02-10T04:27:11.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: mgr.y (mgr.14140) 267 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.164:5000 to Dashboard 2021-02-10T04:27:11.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: 2021-02-10T04:27:11.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: audit 2021-02-10T04:27:11.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: 2021-02-10T04:27:09.819894+0000 2021-02-10T04:27:11.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: mon.a (mon.0) 671 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2021-02-10T04:27:11.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: 2021-02-10T04:27:11.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: audit 2021-02-10T04:27:11.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: 2021-02-10T04:27:09.820219+0000 2021-02-10T04:27:11.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: mgr.y (mgr.14140) 268 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2021-02-10T04:27:11.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: 2021-02-10T04:27:11.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: audit 2021-02-10T04:27:11.034 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: 2021-02-10T04:27:09.823439+0000 2021-02-10T04:27:11.034 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: mon.a (mon.0) 672 : audit [INF] 2021-02-10T04:27:11.034 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:11.034 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: audit 2021-02-10T04:27 2021-02-10T04:27:11.034 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: :09.825384+0000 mon.a (mon.0) 673 2021-02-10T04:27:11.034 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi164"}]: dispatch 2021-02-10T04:27:11.034 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:27:11.034 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: T04:27:09.825798+0000 mgr.y ( 2021-02-10T04:27:11.034 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: mgr.14140) 269 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi164"}]: dispatch 2021-02-10T04:27:11.034 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: audit 2021-02-10T04:27:11.034 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: 2021-02-10T04:27:09.832849+0000 2021-02-10T04:27:11.034 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: mon.a (mon.0) 674 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:11.034 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: 2021-02-10T04:27:11.035 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: audit 2021-02-10T04:27:11.035 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: 2021-02-10T04:27:09.835300+0000 2021-02-10T04:27:11.035 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: mon.a (mon.0) 675 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-02-10T04:27:11.035 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: 2021-02-10T04:27:11.035 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: audit 2021-02-10T04:27:09.835605+0000 2021-02-10T04:27:11.035 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: mgr.y (mgr.14140) 270 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-02-10T04:27:11.035 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: audit 2021-02-10T04:27:09.836197+0000 2021-02-10T04:27:11.035 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: mon.a (mon.0) 676 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi164:9095"}]: dispatch 2021-02-10T04:27:11.035 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:27:11.035 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: T04:27:09.836449+0000 mgr.y (mgr. 2021-02-10T04:27:11.035 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: 14140) 271 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi164:9095"}]: dispatch 2021-02-10T04:27:11.035 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: 2021-02-10T04:27:11.035 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:27:11.036 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: 10T04:27:09.841330+0000 mon.a (mon.0) 677 : 2021-02-10T04:27:11.036 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:11.036 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: audit 2021-02-10T04:27:09.845081 2021-02-10T04:27:11.036 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: +0000 mon.a (mon.0) 678 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-02-10T04:27:11.036 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:27:11.036 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: T04:27:09.845502+0000 mgr.y (mgr.14140) 272 : audit 2021-02-10T04:27:11.036 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-02-10T04:27:11.036 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: audit 2021-02-10T04:27:09.849083+0000 mon.a (mon 2021-02-10T04:27:11.036 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: .0) 679 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi164:3000"}]: dispatch 2021-02-10T04:27:11.036 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: audit 2021-02-10T04:27:09 2021-02-10T04:27:11.036 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: .849315+0000 mgr.y (mgr.14140) 273 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi164:3000"}]: dispatch 2021-02-10T04:27:11.036 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: 2021-02-10T04:27:11.036 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:27:11.036 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: 02-10T04:27:09.854373+0000 mon.a (mon.0) 680 : 2021-02-10T04:27:11.037 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:11.037 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: audit 2021-02-10T04:27:11.037 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: 2021-02-10T04:27:09.857006+0000 mon.a (mon.0) 681 2021-02-10T04:27:11.037 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:27:11.037 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: audit 2021-02-10T04:27: 2021-02-10T04:27:11.037 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: 09.857199+0000 mon.a (mon.0) 682 : 2021-02-10T04:27:11.037 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:27:11.037 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:27:11.037 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: -02-10T04:27:09 2021-02-10T04:27:11.037 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: .970117+0000 mgr.y (mgr.14140) 274 : 2021-02-10T04:27:11.037 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: cluster [DBG] pgmap v234: 132 pgs: 132 active+clean; 25 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:11.037 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:27:11.038 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: -10T04:27:10. 2021-02-10T04:27:11.038 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: 051796+0000 osd.1 (osd.1) 2021-02-10T04:27:11.038 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:10 smithi164 conmon[29288]: 16 : cluster [DBG] 6.4 scrub ok 2021-02-10T04:27:11.295 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:11 smithi164 conmon[52406]: t=2021-02-10T04:27:11+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2021-02-10T04:27:11.295 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:11 smithi164 conmon[52406]: t=2021-02-10T04:27:11+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2021-02-10T04:27:11.295 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:11 smithi164 conmon[52406]: t=2021-02-10T04:27:11+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2021-02-10T04:27:11.295 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:11 smithi164 conmon[52406]: t=2021-02-10T04:27:11+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2021-02-10T04:27:11.617 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:11 smithi164 conmon[52406]: t=2021-02-10T04:27:11+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2021-02-10T04:27:12.838 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:11 smithi164 conmon[52406]: t=2021-02-10T04:27:11+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2021-02-10T04:27:12.840 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:11 smithi085 conmon[29319]: cluster 2021-02-10T04:27:10.456977+0000 osd.0 2021-02-10T04:27:12.840 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:11 smithi085 conmon[29319]: (osd.0) 15 : cluster [DBG] 4.3 scrub ok 2021-02-10T04:27:12.840 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:11 smithi085 conmon[29319]: cluster 2021-02-10T04:27:11.016644+0000 osd.5 (osd.5) 16 : cluster [DBG] 4.15 scrub ok 2021-02-10T04:27:12.841 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:11 smithi085 conmon[35961]: cluster 2021-02-10T04:27:10. 2021-02-10T04:27:12.841 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:11 smithi085 conmon[35961]: 456977+0000 osd.0 (osd.0) 15 : cluster [DBG] 4.3 scrub ok 2021-02-10T04:27:12.841 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:11 smithi085 conmon[35961]: cluster 2021-02-10T04:27:11.016644+0000 osd.5 (osd.5) 16 : cluster [DBG] 4.15 scrub ok 2021-02-10T04:27:12.841 INFO:teuthology.orchestra.run.smithi085.stdout:158913789981 2021-02-10T04:27:12.841 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:11 smithi164 conmon[29288]: cluster 2021-02-10T04:27:10.456977+0000 osd.0 (osd.0) 15 : cluster [DBG] 4.3 scrub ok 2021-02-10T04:27:12.841 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:11 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:27:12.841 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:11 smithi164 conmon[29288]: 27:11.016644+0000 osd.5 (osd.5) 16 : cluster [DBG] 4.15 scrub ok 2021-02-10T04:27:12.841 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:11 smithi164 conmon[52406]: t=2021-02-10T04:27:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2021-02-10T04:27:12.842 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:11 smithi164 conmon[52406]: t=2021-02-10T04:27:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2021-02-10T04:27:12.842 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:11 smithi164 conmon[52406]: t=2021-02-10T04:27:11+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2021-02-10T04:27:12.842 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:11 smithi164 conmon[52406]: t=2021-02-10T04:27:11+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2021-02-10T04:27:12.842 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:11 smithi164 conmon[52406]: t=2021-02-10T04:27:11+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2021-02-10T04:27:12.842 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:12 smithi164 conmon[52406]: t=2021-02-10T04:27:12+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2021-02-10T04:27:12.842 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:12 smithi164 conmon[52406]: t=2021-02-10T04:27:12+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2021-02-10T04:27:12.842 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:12 smithi164 conmon[52406]: t=2021-02-10T04:27:12+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2021-02-10T04:27:12.842 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:12 smithi164 conmon[52406]: t=2021-02-10T04:27:12+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2021-02-10T04:27:12.842 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:12 smithi164 conmon[52406]: t=2021-02-10T04:27:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2021-02-10T04:27:12.842 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:12 smithi164 conmon[52406]: t=2021-02-10T04:27:12+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2021-02-10T04:27:12.842 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:12 smithi164 conmon[52406]: t=2021-02-10T04:27:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2021-02-10T04:27:12.842 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:12 smithi164 conmon[52406]: t=2021-02-10T04:27:12+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2021-02-10T04:27:12.843 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:12 smithi164 conmon[52406]: t=2021-02-10T04:27:12+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2021-02-10T04:27:12.843 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:12 smithi164 conmon[52406]: t=2021-02-10T04:27:12+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2021-02-10T04:27:12.843 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph tell osd.6 flush_pg_stats 2021-02-10T04:27:13.071 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:27:13.086 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:12 smithi085 conmon[35961]: cluster 2021-02-10T04:27:11.606591+0000 osd.2 ( 2021-02-10T04:27:13.086 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:12 smithi085 conmon[35961]: osd.2) 17 : cluster [DBG] 5.b scrub ok 2021-02-10T04:27:13.087 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:12 smithi085 conmon[35961]: cluster 2021-02-10T04:27:11.970667+0000 mgr.y (mgr. 2021-02-10T04:27:13.087 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:12 smithi085 conmon[35961]: 14140) 275 : cluster [DBG] pgmap v235: 132 pgs: 132 active+clean; 25 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2021-02-10T04:27:13.087 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:12 smithi085 conmon[29319]: cluster 2021-02-10T04:27:11.606591+0000 2021-02-10T04:27:13.087 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:12 smithi085 conmon[29319]: osd.2 (osd.2) 17 : cluster [DBG] 5.b scrub ok 2021-02-10T04:27:13.087 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:12 smithi085 conmon[29319]: cluster 2021-02-10T04:27:11.970667+0000 mgr.y (mgr.14140) 275 : cluster [DBG] pgmap v235: 132 pgs: 132 active+clean; 25 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2021-02-10T04:27:13.142 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:12 smithi164 conmon[29288]: cluster 2021-02-10T04:27:11.606591+0000 osd.2 (osd. 2021-02-10T04:27:13.142 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:12 smithi164 conmon[29288]: 2) 17 : cluster [DBG] 5.b scrub ok 2021-02-10T04:27:13.142 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:12 smithi164 conmon[29288]: cluster 2021-02-10T04:27:11.970667+0000 mgr.y (mgr.14140) 275 : cluster [DBG] pgmap v235: 132 pgs: 132 active+clean; 25 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2021-02-10T04:27:13.142 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:12 smithi164 conmon[29288]: 2021-02-10T04:27:13.142 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:12 smithi164 conmon[52406]: t=2021-02-10T04:27:12+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2021-02-10T04:27:13.142 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:12 smithi164 conmon[52406]: t=2021-02-10T04:27:12+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2021-02-10T04:27:13.143 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:13 smithi164 conmon[52406]: t=2021-02-10T04:27:13+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2021-02-10T04:27:13.393 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:13 smithi164 conmon[52406]: t=2021-02-10T04:27:13+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2021-02-10T04:27:13.393 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:13 smithi164 conmon[52406]: t=2021-02-10T04:27:13+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2021-02-10T04:27:13.393 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:13 smithi164 conmon[52406]: t=2021-02-10T04:27:13+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2021-02-10T04:27:13.717 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:13 smithi164 conmon[52406]: t=2021-02-10T04:27:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2021-02-10T04:27:13.717 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:13 smithi164 conmon[52406]: t=2021-02-10T04:27:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2021-02-10T04:27:13.717 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:13 smithi164 conmon[52406]: t=2021-02-10T04:27:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2021-02-10T04:27:13.717 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:13 smithi164 conmon[52406]: t=2021-02-10T04:27:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2021-02-10T04:27:13.718 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:13 smithi164 conmon[52406]: t=2021-02-10T04:27:13+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2021-02-10T04:27:14.015 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:13 smithi164 conmon[29288]: cluster 2021-02-10T04:27:12.291790+0000 osd.3 (osd.3) 14 : cluster [DBG] 4.11 scrub ok 2021-02-10T04:27:14.015 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:13 smithi164 conmon[29288]: 2021-02-10T04:27:14.016 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:13 smithi164 conmon[29288]: cluster 2021-02-10T04:27:13.006707+0000 osd.5 (osd.5) 17 : cluster [DBG] 2021-02-10T04:27:14.016 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:13 smithi164 conmon[29288]: 6.a deep-scrub ok 2021-02-10T04:27:14.016 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:13 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:27:14.016 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:13 smithi164 conmon[29288]: T04:27:13.081393+0000 osd.6 (osd.6) 13 : cluster 2021-02-10T04:27:14.016 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:13 smithi164 conmon[29288]: [DBG] 4.a scrub ok 2021-02-10T04:27:14.016 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:13 smithi164 conmon[52406]: t=2021-02-10T04:27:13+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2021-02-10T04:27:14.016 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:13 smithi164 conmon[52406]: t=2021-02-10T04:27:13+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2021-02-10T04:27:14.016 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2021-02-10T04:27:14.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:13 smithi085 conmon[29319]: cluster 2021-02-10T04:27:12.291790+0000 osd.3 (osd.3) 14 : cluster [DBG] 4.11 scrub ok 2021-02-10T04:27:14.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:13 smithi085 conmon[29319]: cluster 2021-02-10T04:27:13.006707+0000 osd.5 (osd.5) 17 : cluster [DBG] 6.a deep-scrub ok 2021-02-10T04:27:14.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:13 smithi085 conmon[29319]: cluster 2021-02-10T04:27: 2021-02-10T04:27:14.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:13 smithi085 conmon[29319]: 13.081393+0000 osd.6 (osd.6) 13 : cluster [DBG] 4.a scrub ok 2021-02-10T04:27:14.195 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 04:27:13 smithi085 conmon[61668]: level=info ts=2021-02-10T04:27:13.748Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.000513704s 2021-02-10T04:27:14.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:13 smithi085 conmon[35961]: cluster 2021-02-10T04:27:12.291790+0000 osd.3 (osd.3) 14 : 2021-02-10T04:27:14.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:13 smithi085 conmon[35961]: cluster [DBG] 4.11 scrub ok 2021-02-10T04:27:14.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:13 smithi085 conmon[35961]: cluster 2021-02-10T04:27:13.006707+0000 osd.5 (osd.5) 17 : cluster [DBG] 2021-02-10T04:27:14.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:13 smithi085 conmon[35961]: 6.a deep-scrub ok 2021-02-10T04:27:14.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:13 smithi085 conmon[35961]: cluster 2021-02-10T04:27:13.081393+0000 2021-02-10T04:27:14.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:13 smithi085 conmon[35961]: osd.6 (osd.6) 13 : cluster [DBG] 4.a scrub ok 2021-02-10T04:27:14.300 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2021-02-10T04:27:14.300 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2021-02-10T04:27:14.300 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2021-02-10T04:27:14.643 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2021-02-10T04:27:14.643 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Created default admin" logger=sqlstore user=admin 2021-02-10T04:27:14.643 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Initializing HTTPServer" logger=server 2021-02-10T04:27:14.643 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Initializing BackendPluginManager" logger=server 2021-02-10T04:27:14.643 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Initializing PluginManager" logger=server 2021-02-10T04:27:14.643 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Starting plugin search" logger=plugins 2021-02-10T04:27:14.643 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Registering plugin" logger=plugins name="Pie Chart" 2021-02-10T04:27:14.644 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Registering plugin" logger=plugins name="Status Panel" 2021-02-10T04:27:14.644 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Initializing HooksService" logger=server 2021-02-10T04:27:14.644 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Initializing OSSLicensingService" logger=server 2021-02-10T04:27:14.644 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Initializing InternalMetricsService" logger=server 2021-02-10T04:27:14.644 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Initializing RemoteCache" logger=server 2021-02-10T04:27:14.644 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Initializing RenderingService" logger=server 2021-02-10T04:27:14.644 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Initializing AlertEngine" logger=server 2021-02-10T04:27:14.644 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Initializing QuotaService" logger=server 2021-02-10T04:27:14.644 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Initializing ServerLockService" logger=server 2021-02-10T04:27:14.644 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Initializing UserAuthTokenService" logger=server 2021-02-10T04:27:14.644 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Initializing DatasourceCacheService" logger=server 2021-02-10T04:27:14.644 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Initializing LoginService" logger=server 2021-02-10T04:27:14.644 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Initializing SearchService" logger=server 2021-02-10T04:27:14.645 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Initializing TracingService" logger=server 2021-02-10T04:27:14.645 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Initializing UsageStatsService" logger=server 2021-02-10T04:27:14.645 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Initializing CleanUpService" logger=server 2021-02-10T04:27:14.645 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Initializing NotificationService" logger=server 2021-02-10T04:27:14.645 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Initializing provisioningServiceImpl" logger=server 2021-02-10T04:27:14.645 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2021-02-10T04:27:14.645 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 2021-02-10T04:27:14.645 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Backend rendering via phantomJS" logger=rendering renderer=phantomJS 2021-02-10T04:27:14.645 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=warn msg="phantomJS is deprecated and will be removed in a future release. You should consider migrating from phantomJS to grafana-image-renderer plugin. Read more at https://grafana.com/docs/grafana/latest/administration/image_rendering/" logger=rendering renderer=phantomJS 2021-02-10T04:27:14.645 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="Initializing Stream Manager" 2021-02-10T04:27:14.645 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[52406]: t=2021-02-10T04:27:14+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2021-02-10T04:27:14.903 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:14 smithi085 conmon[29319]: cluster 2021-02-10T04:27:13.268406+0000 osd.4 (osd.4) 13 2021-02-10T04:27:14.903 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:14 smithi085 conmon[29319]: : cluster [DBG] 5.e deep-scrub ok 2021-02-10T04:27:14.903 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:14 smithi085 conmon[29319]: cluster 2021-02-10T04:27:13.332220+0000 osd.3 (osd.3) 15 : cluster [DBG] 2.1e deep-scrub ok 2021-02-10T04:27:14.903 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:14 smithi085 conmon[29319]: cluster 2021-02-10T04:27:13.573909+0000 osd.2 (osd.2) 18 : cluster [DBG] 6.14 scrub ok 2021-02-10T04:27:14.903 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:14 smithi085 conmon[29319]: audit 2021-02-10T04:27:13.874475+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:14.903 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:14 smithi085 conmon[29319]: cephadm 2021-02-10T04:27:13.875159+0000 mgr.y (mgr.14140) 276 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:27:14.903 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:14 smithi085 conmon[29319]: cluster 2021-02-10T04:27:13.971093+0000 mgr.y (mgr.14140) 277 : cluster [DBG] pgmap v236: 132 pgs: 132 active+clean; 25 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:14.903 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:14 smithi085 conmon[29319]: cluster 2021-02-10T04:27:14.083835+0000 osd.6 (osd.6) 14 : cluster [DBG] 2.11 scrub ok 2021-02-10T04:27:14.903 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:14 smithi085 conmon[29319]: audit 2021-02-10T04:27:14.492717+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:14.904 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:14 smithi085 conmon[29319]: audit 2021-02-10T04:27:14.498384+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:14.904 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:14 smithi085 conmon[29319]: audit 2021-02-10T04:27:14.502716+0000 mon.a (mon.0) 686 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:27:14.904 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:14 smithi085 conmon[29319]: audit 2021-02-10T04:27:14.502939+0000 mon.a (mon.0) 687 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:27:14.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:14 smithi085 conmon[35961]: cluster 2021-02-10T04:27:13.268406+0000 osd.4 ( 2021-02-10T04:27:14.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:14 smithi085 conmon[35961]: osd.4) 13 : cluster [DBG] 5.e deep-scrub ok 2021-02-10T04:27:14.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:14 smithi085 conmon[35961]: cluster 2021-02-10T04:27:13.332220+0000 osd.3 (osd.3) 15 : cluster [DBG] 2.1e deep-scrub ok 2021-02-10T04:27:14.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:14 smithi085 conmon[35961]: cluster 2021-02-10T04:27:13.573909+0000 osd.2 (osd.2) 18 : cluster [DBG] 6.14 scrub ok 2021-02-10T04:27:14.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:14 smithi085 conmon[35961]: audit 2021-02-10T04:27:13.874475+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:14.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:14 smithi085 conmon[35961]: cephadm 2021-02-10T04:27:13.875159+0000 mgr.y (mgr.14140) 276 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:27:14.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:14 smithi085 conmon[35961]: cluster 2021-02-10T04:27:13.971093+0000 mgr.y (mgr.14140) 277 : cluster [DBG] pgmap v236: 132 pgs: 132 active+clean; 25 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:14.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:14 smithi085 conmon[35961]: cluster 2021-02-10T04:27:14.083835+0000 osd.6 (osd.6) 14 : cluster [DBG] 2.11 scrub ok 2021-02-10T04:27:14.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:14 smithi085 conmon[35961]: audit 2021-02-10T04:27:14.492717+0000 mon.a (mon.0) 684 : audit 2021-02-10T04:27:14.905 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:14 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:14.905 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:14 smithi085 conmon[35961]: audit 2021-02-10T04:27:14.498384+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:14.905 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:14 smithi085 conmon[35961]: audit 2021-02-10T04:27:14.502716+0000 mon.a (mon. 2021-02-10T04:27:14.905 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:14 smithi085 conmon[35961]: 0) 686 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:27:14.905 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:14 smithi085 conmon[35961]: audit 2021-02-10T04:27:14.905 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:14 smithi085 conmon[35961]: 2021-02-10T04:27:14.502939+0000 mon.a (mon.0) 687 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:27:15.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: cluster 2021-02-10T04:27:13.268406+0000 osd.4 (osd.4 2021-02-10T04:27:15.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: ) 13 : cluster [DBG] 5.e deep-scrub ok 2021-02-10T04:27:15.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: cluster 2021-02-10T04:27:13.332220+0000 osd.3 (osd.3) 15 : cluster [DBG] 2.1e deep-scrub ok 2021-02-10T04:27:15.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: cluster 2021-02-10T04:27:13.573909+0000 osd.2 (osd.2) 18 : 2021-02-10T04:27:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: cluster [DBG] 6.14 scrub ok 2021-02-10T04:27:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: audit 2021 2021-02-10T04:27:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: -02-10T04:27:13.874475+0000 2021-02-10T04:27:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: mon.a (mon.0) 683 : audit [INF] 2021-02-10T04:27:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: cephadm 2021-02-10T04:27:13 2021-02-10T04:27:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: .875159+0000 mgr.y (mgr.14140) 276 : cephadm 2021-02-10T04:27:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: [INF] Refreshing smithi164 facts 2021-02-10T04:27:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: cluster 2021-02-10T04:27: 2021-02-10T04:27:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: 13.971093+0000 mgr.y (mgr.14140) 2021-02-10T04:27:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: 277 : cluster [DBG] pgmap v236: 132 pgs: 132 active+clean; 25 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:27:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: 10T04:27:14.083835+0000 osd.6 ( 2021-02-10T04:27:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: osd.6) 14 : cluster [DBG] 2.11 scrub ok 2021-02-10T04:27:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:27:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: 02-10T04:27:14.492717+0000 2021-02-10T04:27:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: mon.a (mon.0) 684 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: 2021-02-10T04:27:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: audit 2021-02-10T04:27:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: 2021-02-10T04:27:14.498384+0000 mon.a ( 2021-02-10T04:27:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: mon.0) 685 : audit [INF] 2021-02-10T04:27:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: audit 2021-02-10T04:27 2021-02-10T04:27:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: :14.502716+0000 mon.a (mon.0) 686 : 2021-02-10T04:27:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:27:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: audit 2021-02-10T04: 2021-02-10T04:27:15.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: 27:14.502939+0000 mon.a (mon.0 2021-02-10T04:27:15.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:14 smithi164 conmon[29288]: ) 687 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:27:16.083 INFO:teuthology.orchestra.run.smithi085.stdout:184683593754 2021-02-10T04:27:16.083 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph tell osd.7 flush_pg_stats 2021-02-10T04:27:16.132 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:15 smithi164 conmon[29288]: cluster 2021-02-10T04:27:14. 2021-02-10T04:27:16.132 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:15 smithi164 conmon[29288]: 256534+0000 osd.4 (osd.4) 14 : cluster [DBG] 4.c scrub ok 2021-02-10T04:27:16.132 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:15 smithi164 conmon[29288]: cluster 2021-02-10T04:27:14.530372+0000 osd.2 (osd.2) 19 : cluster [DBG] 5.8 scrub ok 2021-02-10T04:27:16.132 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:15 smithi164 conmon[29288]: cluster 2021-02-10T04:27:15.041320+0000 osd.6 (osd.6) 15 : cluster [DBG] 5.1f scrub ok 2021-02-10T04:27:16.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:15 smithi085 conmon[29319]: cluster 2021-02-10T04:27:14.256534+0000 osd.4 (osd.4) 2021-02-10T04:27:16.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:15 smithi085 conmon[29319]: 14 : cluster [DBG] 4.c scrub ok 2021-02-10T04:27:16.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:15 smithi085 conmon[29319]: cluster 2021-02-10T04:27:14.530372+0000 osd.2 (osd.2) 19 : cluster [DBG] 5.8 scrub ok 2021-02-10T04:27:16.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:15 smithi085 conmon[29319]: cluster 2021-02-10T04:27:15.041320+0000 osd.6 (osd.6) 15 : cluster [DBG] 5.1f scrub ok 2021-02-10T04:27:16.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:15 smithi085 conmon[35961]: cluster 2021-02-10T04:27:14.256534+0000 osd.4 (osd.4) 14 : 2021-02-10T04:27:16.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:15 smithi085 conmon[35961]: cluster [DBG] 4.c scrub ok 2021-02-10T04:27:16.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:15 smithi085 conmon[35961]: cluster 2021-02-10T04:27:14.530372+0000 osd.2 (osd.2) 19 : cluster [DBG] 5.8 scrub ok 2021-02-10T04:27:16.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:15 smithi085 conmon[35961]: cluster 2021-02-10T04:27:15.041320+0000 osd.6 (osd.6) 15 : cluster 2021-02-10T04:27:16.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:15 smithi085 conmon[35961]: [DBG] 5.1f scrub ok 2021-02-10T04:27:16.301 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:27:16.884 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:16 smithi085 conmon[29319]: cluster 2021-02-10T04:27:15.971657+0000 mgr.y (mgr.14140) 278 : cluster [DBG] pgmap v237: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1007 B/s rd, 0 op/s 2021-02-10T04:27:17.142 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:16 smithi164 conmon[29288]: cluster 2021-02-10T04:27:15.971657+0000 mgr.y (mgr.14140) 2021-02-10T04:27:17.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:16 smithi164 conmon[29288]: 278 : cluster [DBG] pgmap v237: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1007 B/s rd, 0 op/s 2021-02-10T04:27:17.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:16 smithi085 conmon[35961]: cluster 2021-02-10T04:27:15.971657+0000 mgr.y (mgr.14140) 278 : cluster [DBG] pgmap v237: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1007 B/s rd, 0 op/s 2021-02-10T04:27:17.882 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:27:17 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:27:18.024 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:17 smithi085 conmon[29319]: cluster 2021-02-10T04:27:16.340716+0000 osd.3 (osd.3) 2021-02-10T04:27:18.024 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:17 smithi085 conmon[29319]: 16 : cluster [DBG] 2.19 scrub ok 2021-02-10T04:27:18.024 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:17 smithi085 conmon[29319]: cluster 2021-02-10T04:27:16.564628+0000 osd.2 (osd.2) 20 : cluster [DBG] 5.6 deep-scrub ok 2021-02-10T04:27:18.024 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:17 smithi085 conmon[29319]: cluster 2021-02-10T04:27:16.916770+0000 osd.6 (osd.6) 16 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:27:18.024 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:17 smithi085 conmon[29319]: cluster 2021-02-10T04:27:16.916870+0000 osd.6 (osd.6) 17 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:27:18.024 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:17 smithi085 conmon[29319]: cluster 2021-02-10T04:27:17.003295+0000 osd.1 (osd.1) 17 : cluster [DBG] 2.a scrub ok 2021-02-10T04:27:18.024 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:17 smithi085 conmon[35961]: cluster 2021-02-10T04:27:16.340716+0000 osd.3 (osd.3) 16 : cluster [DBG] 2021-02-10T04:27:18.024 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:17 smithi085 conmon[35961]: 2.19 scrub ok 2021-02-10T04:27:18.024 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:17 smithi085 conmon[35961]: cluster 2021-02-10T04:27:16.564628+0000 osd.2 (osd.2) 20 : cluster [DBG] 5.6 deep-scrub ok 2021-02-10T04:27:18.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:17 smithi085 conmon[35961]: cluster 2021-02-10T04:27:16.916770+0000 osd.6 (osd.6) 16 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:27:18.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:17 smithi085 conmon[35961]: cluster 2021-02-10T04:27:16.916870+0000 osd.6 (osd.6) 17 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:27:18.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:17 smithi085 conmon[35961]: cluster 2021-02-10T04:27:17.003295+0000 osd.1 (osd.1) 17 : cluster [DBG] 2.a scrub ok 2021-02-10T04:27:18.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:17 smithi164 conmon[29288]: cluster 2021-02-10T04:27:16.340716+0000 osd.3 (osd.3) 16 : cluster [DBG] 2.19 scrub ok 2021-02-10T04:27:18.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:17 smithi164 conmon[29288]: cluster 2021-02-10T04:27 2021-02-10T04:27:18.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:17 smithi164 conmon[29288]: :16.564628+0000 osd.2 (osd.2) 20 : cluster [DBG] 5.6 deep-scrub ok 2021-02-10T04:27:18.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:17 smithi164 conmon[29288]: cluster 2021-02-10T04:27:16.916770+0000 osd.6 (osd.6) 16 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:27:18.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:17 smithi164 conmon[29288]: cluster 2021-02-10T04:27:16.916870+0000 osd.6 (osd.6) 17 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:27:18.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:17 smithi164 conmon[29288]: 2021-02-10T04:27:18.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:17 smithi164 conmon[29288]: cluster 2021-02-10T04:27:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:17 smithi164 conmon[29288]: 2021-02-10T04:27:17.003295+0000 osd.1 (osd. 2021-02-10T04:27:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:17 smithi164 conmon[29288]: 1) 17 : cluster [DBG] 2.a scrub ok 2021-02-10T04:27:19.082 INFO:teuthology.orchestra.run.smithi085.stdout:214748364822 2021-02-10T04:27:19.082 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd last-stat-seq osd.0 2021-02-10T04:27:19.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:18 smithi164 conmon[29288]: cluster 2021-02-10T04:27:17.352696+0000 osd.4 (osd.4) 15 : cluster [DBG] 4.9 scrub ok 2021-02-10T04:27:19.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:18 smithi164 conmon[29288]: audit 2021-02-10T04:27:17.592159+0000 mgr.y (mgr.14140) 279 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:27:19.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:18 smithi164 conmon[29288]: cluster 2021-02-10T04:27:17.889793+0000 mon.a (mon.0) 688 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2021-02-10T04:27:19.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:18 smithi164 conmon[29288]: audit 2021-02-10T04:27:17.944564+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:19.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:18 smithi164 conmon[29288]: cluster 2021-02-10T04:27:17.972053+0000 mgr.y (mgr.14140) 280 : cluster [DBG] pgmap v239: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:19.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:18 smithi164 conmon[29288]: cluster 2021-02-10T04:27:18.115737+0000 osd.5 (osd.5) 18 : cluster [DBG] 6.7 scrub ok 2021-02-10T04:27:19.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:18 smithi164 conmon[29288]: cluster 2021-02-10T04:27:18.521893+0000 osd.0 (osd.0) 16 : cluster [DBG] 6.0 scrub ok 2021-02-10T04:27:19.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:18 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:27:19.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:18 smithi164 conmon[29288]: 10T04:27:18.574950+0000 mon.a (mon.0) 690 : 2021-02-10T04:27:19.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:18 smithi164 conmon[29288]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:19.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:18 smithi164 conmon[29288]: audit 2021-02-10T04: 2021-02-10T04:27:19.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:18 smithi164 conmon[29288]: 27:18.578334+0000 mon.a (mon.0) 691 : audit [INF] 2021-02-10T04:27:19.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:18 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:19.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[29319]: cluster 2021-02-10T04:27:17.352696+0000 osd.4 (osd.4) 15 2021-02-10T04:27:19.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[29319]: : cluster [DBG] 4.9 scrub ok 2021-02-10T04:27:19.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[29319]: audit 2021-02-10T04:27:17.592159+0000 mgr.y (mgr.14140) 279 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:27:19.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[29319]: cluster 2021-02-10T04:27:17.889793+0000 mon.a (mon.0) 688 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2021-02-10T04:27:19.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[29319]: audit 2021-02-10T04:27:17.944564+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:19.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[29319]: cluster 2021-02-10T04:27:17.972053+0000 mgr.y (mgr.14140) 280 : cluster [DBG] pgmap v239: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:19.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[29319]: cluster 2021-02-10T04:27:18.115737+0000 osd.5 (osd.5) 18 : cluster [DBG] 6.7 scrub ok 2021-02-10T04:27:19.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[29319]: cluster 2021-02-10T04:27:18.521893+0000 osd.0 (osd.0) 16 : cluster [DBG] 6.0 scrub ok 2021-02-10T04:27:19.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[29319]: audit 2021-02-10T04:27:19.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[29319]: 2021-02-10T04:27:18.574950+0000 mon.a (mon.0) 690 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:19.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[29319]: audit 2021-02-10T04:27:18.578334+0000 mon.a (mon.0) 691 : 2021-02-10T04:27:19.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[29319]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[35961]: cluster 2021-02-10T04:27:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[35961]: 2021-02-10T04:27: 2021-02-10T04:27:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[35961]: 17.352696+0000 osd.4 (osd.4) 15 : cluster [DBG] 4.9 scrub ok 2021-02-10T04:27:19.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[35961]: audit 2021-02-10T04:27:17 2021-02-10T04:27:19.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[35961]: .592159+0000 mgr.y (mgr.14140) 279 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:27:19.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:27:19.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[35961]: -10T04:27:17.889793+0000 mon.a (mon. 2021-02-10T04:27:19.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[35961]: 0) 688 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2021-02-10T04:27:19.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:27:19.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[35961]: 27:17.944564+0000 mon.a (mon.0) 689 : audit [INF] 2021-02-10T04:27:19.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:19.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[35961]: cluster 2021-02-10T04:27:19.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[35961]: 2021-02-10T04:27:17.972053+0000 mgr.y (mgr.14140) 280 : cluster 2021-02-10T04:27:19.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[35961]: [DBG] pgmap v239: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:19.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[35961]: cluster 2021-02-10T04:27:19.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[35961]: 2021-02-10T04:27:18.115737+0000 osd.5 (osd.5) 18 2021-02-10T04:27:19.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[35961]: : cluster [DBG] 6.7 scrub ok 2021-02-10T04:27:19.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[35961]: cluster 2021-02-10T04:27:19.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[35961]: 2021-02-10T04:27:18.521893+0000 osd.0 (osd.0 2021-02-10T04:27:19.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[35961]: ) 16 : cluster [DBG] 6.0 scrub ok 2021-02-10T04:27:19.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[35961]: audit 2021-02-10T04:27: 2021-02-10T04:27:19.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[35961]: 18.574950+0000 mon.a (mon.0) 690 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:19.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[35961]: 2021-02-10T04:27:19.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:27:19.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[35961]: 02-10T04:27:18.578334+0000 mon.a (mon.0) 691 : audit [INF] 2021-02-10T04:27:19.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:18 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:19.280 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:27:20.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:19 smithi085 conmon[29319]: cluster 2021-02-10T04:27:18.990081+0000 osd.1 (osd.1) 18 : cluster [DBG] 4.2 deep-scrub ok 2021-02-10T04:27:20.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:19 smithi085 conmon[29319]: cluster 2021-02-10T04:27:19.077773+0000 osd.6 (osd.6) 18 : cluster [DBG] 5.11 deep-scrub ok 2021-02-10T04:27:20.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:19 smithi085 conmon[29319]: cluster 2021-02-10T04:27:19.134208+0000 osd.7 (osd.7) 13 : cluster [DBG] 6.12 deep-scrub ok 2021-02-10T04:27:20.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:19 smithi085 conmon[35961]: cluster 2021-02-10T04:27:18.990081+0000 osd.1 (osd.1) 18 : cluster [DBG] 4.2 deep-scrub ok 2021-02-10T04:27:20.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:19 smithi085 conmon[35961]: cluster 2021-02-10T04:27:19.077773+0000 osd.6 (osd.6) 18 : cluster [DBG] 5.11 deep-scrub ok 2021-02-10T04:27:20.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:19 smithi085 conmon[35961]: cluster 2021-02-10T04:27:19.134208+0000 osd.7 (osd.7) 13 : cluster [DBG] 6.12 deep-scrub ok 2021-02-10T04:27:20.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:19 smithi164 conmon[29288]: cluster 2021-02-10T04:27:18.990081+0000 osd.1 (osd.1) 18 : cluster [DBG] 4.2 deep-scrub ok 2021-02-10T04:27:20.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:19 smithi164 conmon[29288]: cluster 2021-02-10T04:27:19.077773+0000 osd.6 (osd.6) 18 : cluster [DBG] 5.11 deep-scrub ok 2021-02-10T04:27:20.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:19 smithi164 conmon[29288]: cluster 2021-02-10T04:27:19.134208+0000 osd.7 (osd.7) 13 : cluster [DBG] 6.12 deep-scrub ok 2021-02-10T04:27:21.010 INFO:teuthology.orchestra.run.smithi085.stdout:34359738417 2021-02-10T04:27:21.256 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:20 smithi085 conmon[29319]: cluster 2021-02-10T04:27:19.972672+0000 mgr.y (mgr.14140) 281 : cluster [DBG] pgmap v240: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:21.256 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:20 smithi085 conmon[29319]: cluster 2021-02-10T04:27:20.144848+0000 osd.7 (osd.7) 14 : cluster [DBG] 5.9 scrub ok 2021-02-10T04:27:21.256 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:20 smithi085 conmon[29319]: cluster 2021-02-10T04:27:20.157762+0000 osd.5 (osd.5) 19 : cluster [DBG] 5.15 scrub ok 2021-02-10T04:27:21.256 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:20 smithi085 conmon[29319]: cluster 2021-02-10T04:27:20.587598+0000 osd.0 (osd.0) 17 : cluster [DBG] 4.b scrub ok 2021-02-10T04:27:21.256 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:20 smithi085 conmon[35961]: cluster 2021-02-10T04:27:19 2021-02-10T04:27:21.256 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:20 smithi085 conmon[35961]: .972672+0000 mgr.y (mgr.14140) 281 : cluster [DBG] pgmap v240: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:21.257 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:20 smithi085 conmon[35961]: cluster 2021-02-10T04:27:20.144848+0000 osd.7 (osd.7) 14 : cluster [DBG] 5.9 scrub ok 2021-02-10T04:27:21.257 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:20 smithi085 conmon[35961]: cluster 2021-02-10T04:27:20.157762+0000 osd.5 (osd.5) 19 : cluster [DBG] 5.15 scrub ok 2021-02-10T04:27:21.257 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:20 smithi085 conmon[35961]: cluster 2021-02-10T04:27:20.587598+0000 osd.0 (osd.0) 17 : cluster [DBG] 4.b scrub ok 2021-02-10T04:27:21.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:20 smithi164 conmon[29288]: cluster 2021-02-10T04:27:19.972672+0000 mgr.y (mgr.14140) 281 : cluster [DBG] pgmap v240: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:21.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:20 smithi164 conmon[29288]: cluster 2021-02-10T04:27:20.144848+0000 osd.7 (osd.7) 14 : cluster [DBG] 5.9 scrub ok 2021-02-10T04:27:21.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:20 smithi164 conmon[29288]: cluster 2021-02-10T04:27:20.157762+0000 osd.5 (osd.5) 19 : cluster [DBG] 5.15 scrub ok 2021-02-10T04:27:21.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:20 smithi164 conmon[29288]: cluster 2021-02-10T04:27:20.587598+0000 osd.0 (osd. 2021-02-10T04:27:21.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:20 smithi164 conmon[29288]: 0) 17 : cluster [DBG] 4.b scrub ok 2021-02-10T04:27:21.711 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738413 got 34359738417 for osd.0 2021-02-10T04:27:21.711 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd last-stat-seq osd.1 2021-02-10T04:27:21.915 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:27:22.211 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:21 smithi085 conmon[29319]: cluster 2021-02-10T04:27:20.342352+0000 osd.3 (osd.3) 17 : cluster [DBG] 4.0 scrub ok 2021-02-10T04:27:22.211 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:21 smithi085 conmon[29319]: audit 2021-02-10T04:27:21.011342+0000 mon.c (mon.1) 22 : audit [DBG] from='client.? 172.21.15.85:0/2889869080' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2021-02-10T04:27:22.211 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:21 smithi085 conmon[29319]: cluster 2021-02-10T04:27:21.548376+0000 osd.0 (osd.0) 18 : cluster [DBG] 6.10 scrub ok 2021-02-10T04:27:22.211 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:21 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:27:22.211 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:21 smithi085 conmon[35961]: 27:20.342352+0000 osd.3 (osd.3) 17 : cluster [DBG] 4.0 scrub ok 2021-02-10T04:27:22.211 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:21 smithi085 conmon[35961]: audit 2021-02-10T04:27:22.211 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:21 smithi085 conmon[35961]: 2021-02-10T04:27 2021-02-10T04:27:22.211 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:21 smithi085 conmon[35961]: :21.011342+0000 mon.c (mon 2021-02-10T04:27:22.211 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:21 smithi085 conmon[35961]: .1) 22 : audit [DBG] 2021-02-10T04:27:22.212 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:21 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/2889869080' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2021-02-10T04:27:22.212 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:21 smithi085 conmon[35961]: cluster 2021-02-10T04:27:22.212 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:21 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:27:22.212 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:21 smithi085 conmon[35961]: 27:21.548376+0000 osd.0 ( 2021-02-10T04:27:22.212 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:21 smithi085 conmon[35961]: osd.0) 18 : cluster [DBG] 2021-02-10T04:27:22.212 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:21 smithi085 conmon[35961]: 6.10 scrub ok 2021-02-10T04:27:22.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:21 smithi164 conmon[29288]: cluster 2021-02-10T04:27:20.342352+0000 osd.3 (osd.3) 17 : cluster [DBG] 2021-02-10T04:27:22.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:21 smithi164 conmon[29288]: 4.0 scrub ok 2021-02-10T04:27:22.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:21 smithi164 conmon[29288]: audit 2021-02-10T04:27:21.011342+0000 mon.c (mon.1) 22 : audit [DBG] from='client.? 172.21.15.85:0/2889869080' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2021-02-10T04:27:22.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:21 smithi164 conmon[29288]: cluster 2021-02-10T04:27:21.548376+0000 osd.0 (osd.0) 2021-02-10T04:27:22.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:21 smithi164 conmon[29288]: 18 : cluster [DBG] 6.10 scrub ok 2021-02-10T04:27:23.346 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:22 smithi085 conmon[35961]: cluster 2021-02-10T04:27:21.973171+0000 mgr.y (mgr.14140) 282 : cluster [DBG] pgmap v241: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:23.346 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:22 smithi085 conmon[35961]: cluster 2021-02-10T04:27:22.512240+0000 osd.0 (osd.0) 19 : cluster [DBG] 6.9 scrub ok 2021-02-10T04:27:23.346 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:22 smithi085 conmon[29319]: cluster 2021-02-10T04:27:21.973171+0000 mgr.y (mgr.14140) 282 : cluster [DBG] pgmap v241: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:23.346 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:22 smithi085 conmon[29319]: cluster 2021-02-10T04:27:22.512240+0000 osd.0 (osd.0) 19 : cluster [DBG] 6.9 scrub ok 2021-02-10T04:27:23.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:22 smithi164 conmon[29288]: cluster 2021-02-10T04:27:21.973171+0000 mgr.y (mgr.14140) 282 : cluster [DBG] pgmap v241: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:23.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:22 smithi164 conmon[29288]: cluster 2021-02-10T04:27:22.512240+0000 osd.0 (osd. 2021-02-10T04:27:23.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:22 smithi164 conmon[29288]: 0) 19 : cluster [DBG] 6.9 scrub ok 2021-02-10T04:27:23.686 INFO:teuthology.orchestra.run.smithi085.stdout:55834574892 2021-02-10T04:27:24.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:24 smithi164 conmon[29288]: cluster 2021-02-10T04:27:23.236865+0000 osd.5 (osd.5) 20 : cluster [DBG] 2021-02-10T04:27:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:24 smithi164 conmon[29288]: 5.16 scrub ok 2021-02-10T04:27:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:24 smithi164 conmon[29288]: audit 2021-02-10T04:27:23.687388+0000 mon.c (mon.1) 23 : audit [DBG] from='client.? 172.21.15.85:0/2458953082' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2021-02-10T04:27:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:24 smithi164 conmon[29288]: cluster 2021-02-10T04:27 2021-02-10T04:27:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:24 smithi164 conmon[29288]: :23.918771+0000 osd.3 (osd.3) 18 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:27:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:24 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:27:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:24 smithi164 conmon[29288]: :27:23.918941+0000 osd.3 (osd.3) 19 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:27:24.395 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:24 smithi085 conmon[29319]: cluster 2021-02-10T04:27:23.236865+0000 osd.5 2021-02-10T04:27:24.395 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:24 smithi085 conmon[29319]: (osd.5) 20 : cluster [DBG] 5.16 scrub ok 2021-02-10T04:27:24.395 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:24 smithi085 conmon[29319]: audit 2021-02-10T04:27:23.687388+0000 mon.c (mon.1) 23 : audit [DBG] from='client.? 172.21.15.85:0/2458953082' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2021-02-10T04:27:24.395 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:24 smithi085 conmon[29319]: cluster 2021-02-10T04:27:23.918771+0000 osd.3 (osd.3) 18 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:27:24.395 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:24 smithi085 conmon[29319]: cluster 2021-02-10T04:27:23.918941+0000 osd.3 (osd.3) 19 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:27:24.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:24 smithi085 conmon[35961]: cluster 2021-02-10T04:27:24.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:24 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:27:24.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:24 smithi085 conmon[35961]: :27:23.236865+0000 osd.5 (osd.5) 20 : cluster [DBG] 5.16 scrub ok 2021-02-10T04:27:24.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:24 smithi085 conmon[35961]: audit 2021-02-10T04:27:24.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:24 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:27:24.396 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:24 smithi085 conmon[35961]: 10T04:27:23.687388+0000 mon.c 2021-02-10T04:27:24.396 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:24 smithi085 conmon[35961]: (mon.1) 23 : audit 2021-02-10T04:27:24.396 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:24 smithi085 conmon[35961]: [DBG] from='client.? 172.21.15.85:0/2458953082' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2021-02-10T04:27:24.396 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:24 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:27:24.396 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:24 smithi085 conmon[35961]: -02-10T04:27:23.918771 2021-02-10T04:27:24.396 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:24 smithi085 conmon[35961]: +0000 osd.3 (osd.3) 18 : cluster [DBG] 2021-02-10T04:27:24.396 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:24 smithi085 conmon[35961]: purged_snaps scrub starts 2021-02-10T04:27:24.396 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:24 smithi085 conmon[35961]: cluster 2021-02-10T04:27:24.396 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:24 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:27:24.396 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:24 smithi085 conmon[35961]: 27:23.918941+0000 osd.3 2021-02-10T04:27:24.396 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:24 smithi085 conmon[35961]: (osd.3) 19 : cluster 2021-02-10T04:27:24.396 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:24 smithi085 conmon[35961]: [DBG] purged_snaps scrub ok 2021-02-10T04:27:24.413 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574888 got 55834574892 for osd.1 2021-02-10T04:27:24.414 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd last-stat-seq osd.2 2021-02-10T04:27:24.613 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:27:25.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:25 smithi085 conmon[29319]: cluster 2021-02-10T04:27:23.973630+0000 2021-02-10T04:27:25.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:25 smithi085 conmon[29319]: mgr.y (mgr.14140) 283 : cluster [DBG] pgmap v242: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:25.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:25 smithi085 conmon[29319]: cluster 2021-02-10T04:27:24.012665+0000 mon.a (mon.0) 692 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2021-02-10T04:27:25.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:25 smithi085 conmon[29319]: cluster 2021-02-10T04:27:24.209453+0000 osd.5 (osd.5) 21 : cluster [DBG] 6.d deep-scrub ok 2021-02-10T04:27:25.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:25 smithi085 conmon[29319]: cluster 2021-02-10T04:27:24.523833+0000 osd.0 (osd.0) 20 : cluster [DBG] 4.16 scrub ok 2021-02-10T04:27:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:25 smithi085 conmon[35961]: cluster 2021-02-10T04:27:23.973630 2021-02-10T04:27:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:25 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 283 : cluster [DBG] pgmap v242: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:25.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:25 smithi085 conmon[35961]: cluster 2021-02-10T04:27:25.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:25 smithi085 conmon[35961]: 2021-02-10T04:27:24.012665+0000 mon.a (mon.0) 2021-02-10T04:27:25.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:25 smithi085 conmon[35961]: 692 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2021-02-10T04:27:25.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:25 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:27:25.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:25 smithi085 conmon[35961]: 10T04:27:24.209453+0000 osd.5 (osd 2021-02-10T04:27:25.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:25 smithi085 conmon[35961]: .5) 21 : cluster [DBG] 6.d deep-scrub ok 2021-02-10T04:27:25.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:25 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:27:25.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:25 smithi085 conmon[35961]: -02-10T04:27:24.523833+0000 2021-02-10T04:27:25.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:25 smithi085 conmon[35961]: osd.0 (osd.0) 20 : cluster [DBG] 4.16 scrub ok 2021-02-10T04:27:25.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:25 smithi085 conmon[35961]: 2021-02-10T04:27:25.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:25 smithi164 conmon[29288]: cluster 2021-02-10T04:27:23.973630+0000 mgr.y (mgr.14140) 283 : cluster [DBG] pgmap v242: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:25.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:25 smithi164 conmon[29288]: 2021-02-10T04:27:25.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:25 smithi164 conmon[29288]: cluster 2021-02-10T04:27:24.012665+0000 mon.a (mon.0) 692 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2021-02-10T04:27:25.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:25 smithi164 conmon[29288]: cluster 2021-02-10T04:27:24.209453+0000 osd.5 (osd.5) 21 : cluster [DBG] 6.d deep-scrub ok 2021-02-10T04:27:25.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:25 smithi164 conmon[29288]: cluster 2021-02-10T04:27:24.523833+0000 osd.0 (osd.0) 20 : cluster [DBG] 4.16 scrub ok 2021-02-10T04:27:26.102 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:26 smithi085 conmon[35961]: cluster 2021-02-10T04:27:25.026528+0000 osd.1 (osd.1) 19 : cluster [DBG] 2.6 scrub ok 2021-02-10T04:27:26.102 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:26 smithi085 conmon[35961]: cluster 2021-02-10T04:27:25.216253+0000 osd.5 (osd.5) 22 : cluster [DBG] 5.7 deep-scrub ok 2021-02-10T04:27:26.102 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:26 smithi085 conmon[29319]: cluster 2021-02-10T04:27:25.026528+0000 osd.1 (osd.1) 19 : cluster [DBG] 2.6 scrub ok 2021-02-10T04:27:26.102 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:26 smithi085 conmon[29319]: cluster 2021-02-10T04:27:25.216253+0000 osd.5 (osd.5) 22 : cluster [DBG] 5.7 deep-scrub ok 2021-02-10T04:27:26.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:26 smithi164 conmon[29288]: cluster 2021-02-10T04:27:25.026528+0000 osd.1 (osd.1) 19 : cluster [DBG] 2.6 scrub ok 2021-02-10T04:27:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:26 smithi164 conmon[29288]: cluster 2021-02-10T04:27:25.216253+0000 osd.5 (osd.5) 22 : cluster [DBG] 5.7 deep-scrub ok 2021-02-10T04:27:26.401 INFO:teuthology.orchestra.run.smithi085.stdout:77309411371 2021-02-10T04:27:27.101 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411367 got 77309411371 for osd.2 2021-02-10T04:27:27.101 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd last-stat-seq osd.3 2021-02-10T04:27:27.299 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:27:27.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:27 smithi164 conmon[29288]: cluster 2021-02-10T04:27:25.974201+0000 mgr.y (mgr.14140) 284 : cluster [DBG] pgmap v244: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:27:27.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:27 smithi164 conmon[29288]: cluster 2021-02-10T04:27:26.029852+0000 osd.1 (osd.1) 20 : cluster [DBG] 6.1d scrub ok 2021-02-10T04:27:27.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:27 smithi164 conmon[29288]: cluster 2021-02-10T04:27:26.264556+0000 osd.3 2021-02-10T04:27:27.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:27 smithi164 conmon[29288]: (osd.3) 20 : cluster [DBG] 5.14 scrub ok 2021-02-10T04:27:27.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:27 smithi164 conmon[29288]: audit 2021-02-10T04:27:26.401361 2021-02-10T04:27:27.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:27 smithi164 conmon[29288]: +0000 mon.a (mon.0) 693 : audit [DBG] from='client.? 172.21.15.85:0/542522470' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2021-02-10T04:27:27.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:27 smithi164 conmon[29288]: 2021-02-10T04:27:27.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:27 smithi164 conmon[29288]: cluster 2021-02-10T04:27:26.403744+0000 osd.4 (osd. 2021-02-10T04:27:27.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:27 smithi164 conmon[29288]: 4) 16 : cluster [DBG] 4.1 scrub ok 2021-02-10T04:27:27.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:27 smithi085 conmon[29319]: cluster 2021-02-10T04:27:25.974201+0000 2021-02-10T04:27:27.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:27 smithi085 conmon[29319]: mgr.y (mgr.14140) 284 : cluster [DBG] pgmap v244: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:27:27.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:27 smithi085 conmon[29319]: cluster 2021-02-10T04:27:26.029852+0000 osd.1 (osd.1) 20 : cluster [DBG] 6.1d scrub ok 2021-02-10T04:27:27.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:27 smithi085 conmon[29319]: cluster 2021-02-10T04:27:26.264556+0000 osd.3 (osd.3) 20 : cluster [DBG] 5.14 scrub ok 2021-02-10T04:27:27.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:27 smithi085 conmon[29319]: audit 2021-02-10T04:27:26.401361+0000 mon.a (mon.0) 693 : audit [DBG] from='client.? 172.21.15.85:0/542522470' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2021-02-10T04:27:27.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:27 smithi085 conmon[29319]: cluster 2021-02-10T04:27:26.403744+0000 osd.4 (osd.4) 16 : cluster [DBG] 4.1 scrub ok 2021-02-10T04:27:27.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:27 smithi085 conmon[35961]: cluster 2021-02-10T04:27:25. 2021-02-10T04:27:27.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:27 smithi085 conmon[35961]: 974201+0000 mgr.y (mgr.14140) 284 : cluster [DBG] pgmap v244: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:27:27.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:27 smithi085 conmon[35961]: cluster 2021-02-10T04:27:26.029852+0000 osd.1 (osd.1) 20 : cluster [DBG] 6.1d scrub ok 2021-02-10T04:27:27.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:27 smithi085 conmon[35961]: cluster 2021-02-10T04:27:26.264556+0000 osd.3 (osd.3) 20 : cluster [DBG] 5.14 scrub ok 2021-02-10T04:27:27.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:27 smithi085 conmon[35961]: audit 2021-02-10T04:27:26.401361+0000 mon.a (mon.0) 693 : audit [DBG] from='client.? 172.21.15.85:0/542522470' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2021-02-10T04:27:27.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:27 smithi085 conmon[35961]: cluster 2021-02-10T04:27:26.403744+0000 osd.4 (osd.4) 16 : cluster [DBG] 4.1 scrub ok 2021-02-10T04:27:27.893 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:27:27 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:27:28.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:28 smithi085 conmon[29319]: cluster 2021-02-10T04:27:26.234344+0000 2021-02-10T04:27:28.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:28 smithi085 conmon[29319]: osd.7 (osd.7) 15 : cluster [DBG] 6.3 scrub ok 2021-02-10T04:27:28.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:28 smithi085 conmon[29319]: cluster 2021-02-10T04:27:27.017033+0000 osd.1 (osd.1) 21 : cluster [DBG] 5.1d scrub ok 2021-02-10T04:27:28.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:28 smithi085 conmon[29319]: cluster 2021-02-10T04:27:27.259384+0000 osd.3 (osd.3) 21 : cluster [DBG] 6.b scrub ok 2021-02-10T04:27:28.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:28 smithi085 conmon[29319]: cluster 2021-02-10T04:27:27.564168+0000 osd.0 (osd.0) 21 : cluster [DBG] 5.1e scrub ok 2021-02-10T04:27:28.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:28 smithi085 conmon[35961]: cluster 2021-02-10T04:27:26.234344+0000 2021-02-10T04:27:28.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:28 smithi085 conmon[35961]: osd.7 (osd.7) 15 : cluster [DBG] 6.3 scrub ok 2021-02-10T04:27:28.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:28 smithi085 conmon[35961]: cluster 2021-02-10T04:27:27.017033+0000 osd.1 (osd.1) 21 : cluster [DBG] 5.1d scrub ok 2021-02-10T04:27:28.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:28 smithi085 conmon[35961]: cluster 2021-02-10T04:27:27.259384+0000 osd.3 (osd.3) 21 : cluster [DBG] 6.b scrub ok 2021-02-10T04:27:28.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:28 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:27:28.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:28 smithi085 conmon[35961]: :27:27.564168+0000 osd.0 (osd.0) 21 : cluster [DBG] 5.1e scrub ok 2021-02-10T04:27:28.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:28 smithi164 conmon[29288]: cluster 2021-02-10T04:27:26.234344+0000 osd.7 (osd.7) 15 : cluster [DBG] 6.3 scrub ok 2021-02-10T04:27:28.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:28 smithi164 conmon[29288]: cluster 2021-02-10T04:27:27.017033+0000 2021-02-10T04:27:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:28 smithi164 conmon[29288]: osd.1 (osd.1) 21 : cluster [DBG] 5.1d scrub ok 2021-02-10T04:27:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:28 smithi164 conmon[29288]: cluster 2021-02-10T04:27:27.259384+0000 osd.3 (osd.3) 21 : cluster [DBG] 6.b scrub ok 2021-02-10T04:27:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:28 smithi164 conmon[29288]: cluster 2021-02-10T04:27:27.564168+0000 osd.0 (osd. 2021-02-10T04:27:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:28 smithi164 conmon[29288]: 0) 21 : cluster [DBG] 5.1e scrub ok 2021-02-10T04:27:29.038 INFO:teuthology.orchestra.run.smithi085.stdout:107374182438 2021-02-10T04:27:29.050 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:29 smithi085 conmon[29319]: audit 2021-02-10T04:27:27.602912+0000 mgr.y (mgr.14140) 285 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:27:29.050 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:29 smithi085 conmon[29319]: cluster 2021-02-10T04:27:27.974678+0000 mgr.y (mgr.14140) 286 : cluster [DBG] pgmap v245: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:29.050 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:29 smithi085 conmon[29319]: cluster 2021-02-10T04:27:28.213449+0000 osd.3 (osd.3) 22 : cluster [DBG] 4.14 scrub ok 2021-02-10T04:27:29.050 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:29 smithi085 conmon[29319]: cluster 2021-02-10T04:27:28.373111+0000 osd.4 (osd.4) 17 : cluster [DBG] 6.17 scrub ok 2021-02-10T04:27:29.050 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:29 smithi085 conmon[29319]: audit 2021-02-10T04:27:29.039130+0000 mon.a (mon.0) 694 : audit [DBG] from='client.? 172.21.15.85:0/2648202345' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2021-02-10T04:27:29.050 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:29 smithi085 conmon[35961]: audit 2021-02-10T04:27:27.602912+0000 mgr.y (mgr.14140) 285 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:27:29.050 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:29 smithi085 conmon[35961]: cluster 2021-02-10T04:27:27.974678+0000 mgr.y (mgr.14140) 286 : cluster [DBG] pgmap v245: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:29.050 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:29 smithi085 conmon[35961]: cluster 2021-02-10T04:27:28.213449+0000 osd.3 (osd.3) 22 : cluster [DBG] 4.14 scrub ok 2021-02-10T04:27:29.050 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:29 smithi085 conmon[35961]: cluster 2021-02-10T04:27:28.373111+0000 osd.4 (osd.4) 17 : cluster [DBG] 6.17 scrub ok 2021-02-10T04:27:29.050 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:29 smithi085 conmon[35961]: audit 2021-02-10T04:27:29.039130+0000 mon.a (mon.0) 694 : audit [DBG] from='client.? 172.21.15.85:0/2648202345' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2021-02-10T04:27:29.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:29 smithi164 conmon[29288]: audit 2021-02-10T04:27:27.602912+0000 mgr.y (mgr.14140) 285 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:27:29.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:29 smithi164 conmon[29288]: cluster 2021-02-10T04:27:27.974678+0000 mgr.y ( 2021-02-10T04:27:29.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:29 smithi164 conmon[29288]: mgr.14140) 286 : cluster [DBG] pgmap v245: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:29.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:29 smithi164 conmon[29288]: cluster 2021-02-10T04:27:28.213449+0000 osd.3 (osd.3) 22 : cluster [DBG] 4.14 scrub ok 2021-02-10T04:27:29.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:29 smithi164 conmon[29288]: cluster 2021-02-10T04:27:28.373111+0000 osd.4 (osd.4) 17 : 2021-02-10T04:27:29.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:29 smithi164 conmon[29288]: cluster [DBG] 6.17 scrub ok 2021-02-10T04:27:29.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:29 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:27:29.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:29 smithi164 conmon[29288]: :27:29.039130+0000 mon.a (mon.0) 694 : audit [DBG] from='client.? 172.21.15.85:0/2648202345' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2021-02-10T04:27:29.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:29 smithi164 conmon[29288]: 2021-02-10T04:27:29.785 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182434 got 107374182438 for osd.3 2021-02-10T04:27:29.786 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd last-stat-seq osd.4 2021-02-10T04:27:29.981 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:27:30.054 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:30 smithi085 conmon[29319]: cluster 2021-02-10T04:27:28.264199+0000 osd.7 2021-02-10T04:27:30.054 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:30 smithi085 conmon[29319]: (osd.7) 16 : cluster [DBG] 6.5 scrub ok 2021-02-10T04:27:30.054 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:30 smithi085 conmon[29319]: cluster 2021-02-10T04:27:29.364578+0000 osd.4 (osd.4) 18 : cluster [DBG] 6.1e scrub ok 2021-02-10T04:27:30.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:30 smithi164 conmon[29288]: cluster 2021-02-10T04:27:28.264199+0000 osd.7 (osd.7) 16 2021-02-10T04:27:30.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:30 smithi164 conmon[29288]: : cluster [DBG] 6.5 scrub ok 2021-02-10T04:27:30.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:30 smithi164 conmon[29288]: cluster 2021-02-10T04:27:29.364578+0000 osd.4 (osd.4) 18 : cluster [DBG] 6.1e scrub ok 2021-02-10T04:27:30.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:30 smithi085 conmon[35961]: cluster 2021-02-10T04:27:28.264199+0000 osd.7 (osd.7) 16 : cluster [DBG] 6.5 scrub ok 2021-02-10T04:27:30.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:30 smithi085 conmon[35961]: cluster 2021-02-10T04:27:29.364578+0000 osd.4 (osd.4) 18 : cluster [DBG] 6.1e scrub ok 2021-02-10T04:27:31.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:31 smithi164 conmon[29288]: cluster 2021-02-10T04:27:29.250971+0000 osd.7 (osd.7) 17 : cluster [DBG] 5.1a scrub ok 2021-02-10T04:27:31.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:31 smithi164 conmon[29288]: cluster 2021-02-10T04:27:29.910413+0000 osd.0 (osd.0) 22 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:27:31.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:31 smithi164 conmon[29288]: cluster 2021-02-10T04:27:29.910491+0000 osd.0 (osd.0) 23 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:27:31.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:31 smithi164 conmon[29288]: 2021-02-10T04:27:31.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:31 smithi164 conmon[29288]: cluster 2021-02-10T04:27:29.975115 2021-02-10T04:27:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:31 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 287 : cluster [DBG] pgmap v246: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:31 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:27:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:31 smithi164 conmon[29288]: 10T04:27:30.012924+0000 osd.1 (osd 2021-02-10T04:27:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:31 smithi164 conmon[29288]: .1) 22 : cluster [DBG] 4.f scrub ok 2021-02-10T04:27:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:31 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:27:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:31 smithi164 conmon[29288]: -02-10T04:27:30.055787+0000 2021-02-10T04:27:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:31 smithi164 conmon[29288]: mon.a (mon.0) 695 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2021-02-10T04:27:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:31 smithi164 conmon[29288]: 2021-02-10T04:27:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:31 smithi164 conmon[29288]: cluster 2021-02-10T04:27: 2021-02-10T04:27:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:31 smithi164 conmon[29288]: 30.657778+0000 osd.0 (osd.0) 24 : 2021-02-10T04:27:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:31 smithi164 conmon[29288]: cluster [DBG] 3.1 scrub ok 2021-02-10T04:27:31.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[29319]: cluster 2021-02-10T04:27:29.250971+0000 osd.7 (osd.7) 17 : 2021-02-10T04:27:31.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[29319]: cluster [DBG] 5.1a scrub ok 2021-02-10T04:27:31.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[29319]: cluster 2021-02-10T04:27:29.910413+0000 osd.0 (osd.0) 22 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:27:31.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[29319]: cluster 2021-02-10T04:27:29.910491+0000 osd.0 (osd.0) 23 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:27:31.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[29319]: cluster 2021-02-10T04:27:29.975115+0000 mgr.y (mgr.14140) 287 : cluster [DBG] pgmap v246: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:31.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[29319]: cluster 2021-02-10T04:27:30.012924+0000 osd.1 (osd.1) 22 : cluster [DBG] 4.f scrub ok 2021-02-10T04:27:31.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[29319]: cluster 2021-02-10T04:27:30.055787+0000 mon.a (mon.0) 695 : cluster 2021-02-10T04:27:31.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[29319]: [DBG] osdmap e81: 8 total, 8 up, 8 in 2021-02-10T04:27:31.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[29319]: cluster 2021-02-10T04:27:30.657778+0000 osd.0 (osd.0) 24 : cluster [DBG] 3.1 scrub ok 2021-02-10T04:27:31.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[35961]: cluster 2021-02-10T04:27:29.250971 2021-02-10T04:27:31.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[35961]: +0000 osd.7 (osd.7) 17 : cluster [DBG] 5.1a scrub ok 2021-02-10T04:27:31.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:27:31.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[35961]: :27:29.910413+0000 osd.0 (osd.0 2021-02-10T04:27:31.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[35961]: ) 22 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:27:31.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:27:31.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[35961]: -10T04:27:29.910491+0000 osd.0 2021-02-10T04:27:31.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[35961]: (osd.0) 23 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:27:31.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[35961]: 2021-02-10T04:27:31.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[35961]: cluster 2021-02-10T04:27:31.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[35961]: 2021-02-10T04:27: 2021-02-10T04:27:31.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[35961]: 29.975115+0000 mgr.y (mgr.14140) 287 2021-02-10T04:27:31.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[35961]: : cluster [DBG] pgmap v246: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:31.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:27:31.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[35961]: 04:27:30.012924+0000 osd.1 (osd 2021-02-10T04:27:31.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[35961]: .1) 22 : cluster [DBG] 4.f scrub ok 2021-02-10T04:27:31.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:27:31.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[35961]: -10T04:27:30.055787+0000 2021-02-10T04:27:31.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[35961]: mon.a (mon.0) 695 : cluster [DBG] 2021-02-10T04:27:31.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[35961]: osdmap e81: 8 total, 8 up, 8 in 2021-02-10T04:27:31.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:27:31.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[35961]: 04:27:30.657778+0000 osd.0 (osd 2021-02-10T04:27:31.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:31 smithi085 conmon[35961]: .0) 24 : cluster [DBG] 3.1 scrub ok 2021-02-10T04:27:31.771 INFO:teuthology.orchestra.run.smithi085.stdout:133143986212 2021-02-10T04:27:32.059 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:32 smithi085 conmon[29319]: cluster 2021-02-10T04:27:31.650027+0000 osd.0 (osd.0) 25 : cluster [DBG] 5.5 deep-scrub ok 2021-02-10T04:27:32.060 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:32 smithi085 conmon[29319]: audit 2021-02-10T04:27:31.771377+0000 mon.c (mon.1) 24 : audit [DBG] from='client.? 172.21.15.85:0/3876424292' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2021-02-10T04:27:32.060 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:32 smithi085 conmon[35961]: cluster 2021-02-10T04:27:31.650027+0000 2021-02-10T04:27:32.060 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:32 smithi085 conmon[35961]: osd.0 (osd.0) 25 : cluster [DBG] 5.5 deep-scrub ok 2021-02-10T04:27:32.060 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:32 smithi085 conmon[35961]: audit 2021-02-10T04:27:31.771377+0000 mon.c (mon.1) 24 : audit [DBG] from='client.? 172.21.15.85:0/3876424292' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2021-02-10T04:27:32.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:32 smithi164 conmon[29288]: cluster 2021-02-10T04:27:31.650027+0000 osd.0 (osd.0) 25 : cluster [DBG] 5.5 deep-scrub ok 2021-02-10T04:27:32.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:32 smithi164 conmon[29288]: audit 2021-02-10T04:27:31.771377+0000 mon.c (mon.1) 24 : audit [DBG] from='client.? 172.21.15.85:0/3876424292' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2021-02-10T04:27:32.476 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986208 got 133143986212 for osd.4 2021-02-10T04:27:32.477 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd last-stat-seq osd.5 2021-02-10T04:27:32.672 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:27:33.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:33 smithi085 conmon[29319]: cluster 2021-02-10T04:27:31.975600+0000 mgr.y (mgr 2021-02-10T04:27:33.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:33 smithi085 conmon[29319]: .14140) 288 : cluster [DBG] pgmap v248: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:27:33.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:33 smithi085 conmon[35961]: cluster 2021-02-10T04:27:31.975600+0000 mgr.y 2021-02-10T04:27:33.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:33 smithi085 conmon[35961]: (mgr.14140) 288 : cluster [DBG] pgmap v248: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:27:33.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:33 smithi164 conmon[29288]: cluster 2021-02-10T04:27:31.975600+0000 mgr.y (mgr.14140) 288 : cluster [DBG] pgmap v248: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:27:33.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:33 smithi164 conmon[29288]: 2021-02-10T04:27:34.391 INFO:teuthology.orchestra.run.smithi085.stdout:158913789986 2021-02-10T04:27:34.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:34 smithi164 conmon[29288]: cluster 2021-02-10T04:27:33.029904+0000 osd.1 (osd.1) 23 : cluster [DBG] 5.12 deep-scrub ok 2021-02-10T04:27:34.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:34 smithi164 conmon[29288]: cluster 2021-02-10T04:27:33.210425+0000 osd.3 (osd.3) 23 : cluster [DBG] 5.0 scrub ok 2021-02-10T04:27:34.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:34 smithi164 conmon[29288]: cluster 2021-02-10T04:27:33.382315+0000 osd.4 (osd.4) 19 : cluster [DBG] 6.e scrub ok 2021-02-10T04:27:34.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:34 smithi164 conmon[29288]: cluster 2021-02-10T04:27:33.628033 2021-02-10T04:27:34.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:34 smithi164 conmon[29288]: +0000 osd.0 (osd.0) 26 : cluster 2021-02-10T04:27:34.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:34 smithi164 conmon[29288]: [DBG] 6.18 scrub ok 2021-02-10T04:27:34.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:34 smithi085 conmon[29319]: cluster 2021-02-10T04:27:33.029904+0000 osd.1 (osd.1) 23 : cluster [DBG] 2021-02-10T04:27:34.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:34 smithi085 conmon[29319]: 5.12 deep-scrub ok 2021-02-10T04:27:34.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:34 smithi085 conmon[29319]: cluster 2021-02-10T04:27:33.210425+0000 osd.3 (osd.3) 23 : cluster [DBG] 5.0 scrub ok 2021-02-10T04:27:34.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:34 smithi085 conmon[29319]: cluster 2021-02-10T04:27:33.382315+0000 osd.4 (osd.4) 19 : cluster [DBG] 6.e scrub ok 2021-02-10T04:27:34.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:34 smithi085 conmon[29319]: cluster 2021-02-10T04:27:33.628033+0000 osd.0 (osd.0) 26 : cluster [DBG] 6.18 scrub ok 2021-02-10T04:27:34.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:34 smithi085 conmon[35961]: cluster 2021-02-10T04:27:33.029904+0000 osd.1 (osd.1) 2021-02-10T04:27:34.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:34 smithi085 conmon[35961]: 23 : cluster [DBG] 5.12 deep-scrub ok 2021-02-10T04:27:34.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:34 smithi085 conmon[35961]: cluster 2021-02-10T04:27:33.210425+0000 2021-02-10T04:27:34.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:34 smithi085 conmon[35961]: osd.3 (osd.3) 23 : cluster [DBG] 5.0 scrub ok 2021-02-10T04:27:34.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:34 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:27:34.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:34 smithi085 conmon[35961]: 04:27:33.382315+0000 osd.4 (osd. 2021-02-10T04:27:34.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:34 smithi085 conmon[35961]: 4) 19 : cluster [DBG] 6.e scrub ok 2021-02-10T04:27:34.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:34 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:27:34.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:34 smithi085 conmon[35961]: -02-10T04:27:33.628033+0000 2021-02-10T04:27:34.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:34 smithi085 conmon[35961]: osd.0 (osd.0) 26 : cluster [DBG] 2021-02-10T04:27:34.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:34 smithi085 conmon[35961]: 6.18 scrub ok 2021-02-10T04:27:35.068 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913789981 got 158913789986 for osd.5 2021-02-10T04:27:35.068 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd last-stat-seq osd.6 2021-02-10T04:27:35.266 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:27:35.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:35 smithi164 conmon[29288]: cluster 2021-02-10T04:27:33.192282+0000 osd.7 (osd.7) 18 : cluster [DBG] 6.15 scrub ok 2021-02-10T04:27:35.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:35 smithi164 conmon[29288]: cluster 2021-02-10T04:27:33.976037+0000 mgr.y (mgr.14140) 289 : cluster [DBG] pgmap v249: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:27:35.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:35 smithi164 conmon[29288]: audit 2021-02-10T04:27:34.391737+0000 mon.c (mon.1) 25 : audit [DBG] from='client.? 172.21.15.85:0/4138786868' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2021-02-10T04:27:35.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:35 smithi085 conmon[29319]: cluster 2021-02-10T04:27:33.192282+0000 osd.7 (osd. 2021-02-10T04:27:35.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:35 smithi085 conmon[29319]: 7) 18 : cluster [DBG] 6.15 scrub ok 2021-02-10T04:27:35.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:35 smithi085 conmon[29319]: cluster 2021-02-10T04:27:33.976037+0000 mgr.y (mgr.14140) 289 : cluster [DBG] pgmap v249: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:27:35.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:35 smithi085 conmon[29319]: audit 2021-02-10T04:27:34.391737+0000 mon.c (mon.1) 25 : audit [DBG] from='client.? 172.21.15.85:0/4138786868' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2021-02-10T04:27:35.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:35 smithi085 conmon[35961]: cluster 2021-02-10T04:27:33.192282+0000 2021-02-10T04:27:35.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:35 smithi085 conmon[35961]: osd.7 (osd.7) 18 : cluster [DBG] 6.15 scrub ok 2021-02-10T04:27:35.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:35 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:27:35.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:35 smithi085 conmon[35961]: 02-10T04:27:33.976037+0000 mgr.y (mgr.14140) 289 : cluster [DBG] pgmap v249: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:27:35.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:35 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:27:35.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:35 smithi085 conmon[35961]: T04:27:34.391737+0000 mon.c (mon.1) 25 : audit 2021-02-10T04:27:35.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:35 smithi085 conmon[35961]: [DBG] from='client.? 172.21.15.85:0/4138786868' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2021-02-10T04:27:36.972 INFO:teuthology.orchestra.run.smithi085.stdout:184683593758 2021-02-10T04:27:37.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:37 smithi164 conmon[29288]: cluster 2021-02-10T04:27:35.166241+0000 osd.7 (osd.7) 19 : cluster [DBG] 6.8 scrub ok 2021-02-10T04:27:37.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:37 smithi164 conmon[29288]: cluster 2021-02-10T04:27:35.777542+0000 osd.7 (osd.7) 20 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:27:37.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:37 smithi164 conmon[29288]: cluster 2021-02-10T04:27:35.777712+0000 osd.7 (osd.7) 21 2021-02-10T04:27:37.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:37 smithi164 conmon[29288]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:27:37.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:37 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:27:37.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:37 smithi164 conmon[29288]: :27:35.976539+0000 mgr.y (mgr.14140) 290 : cluster [DBG] pgmap v250: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:37.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:37 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:27:37.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:37 smithi164 conmon[29288]: T04:27:36.077899+0000 2021-02-10T04:27:37.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:37 smithi164 conmon[29288]: mon.a (mon.0) 696 : cluster [DBG] 2021-02-10T04:27:37.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:37 smithi164 conmon[29288]: osdmap e82: 8 total, 8 up, 8 in 2021-02-10T04:27:37.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:37 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:27:37.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:37 smithi164 conmon[29288]: 02-10T04:27:36.972261 2021-02-10T04:27:37.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:37 smithi164 conmon[29288]: +0000 mon.c (mon.1) 26 : audit [DBG] 2021-02-10T04:27:37.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:37 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/1395129641' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2021-02-10T04:27:37.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:37 smithi085 conmon[35961]: cluster 2021-02-10T04:27:35.166241+0000 osd.7 (osd.7) 19 : cluster [DBG] 6.8 scrub ok 2021-02-10T04:27:37.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:37 smithi085 conmon[35961]: cluster 2021-02-10T04:27:35.777542+0000 osd.7 (osd.7) 20 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:27:37.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:37 smithi085 conmon[35961]: cluster 2021-02-10T04:27:35.777712+0000 osd.7 (osd.7) 21 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:27:37.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:37 smithi085 conmon[35961]: cluster 2021-02-10T04:27:35.976539+0000 mgr.y (mgr.14140) 290 : cluster [DBG] pgmap v250: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:37.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:37 smithi085 conmon[35961]: cluster 2021-02-10T04:27:36.077899+0000 mon.a (mon.0) 696 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2021-02-10T04:27:37.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:37 smithi085 conmon[35961]: audit 2021-02-10T04:27:36.972261+0000 mon.c (mon.1) 26 : audit [DBG] from='client.? 172.21.15.85:0/1395129641' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2021-02-10T04:27:37.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:37 smithi085 conmon[29319]: cluster 2021-02-10T04:27:35.166241+0000 osd.7 (osd.7) 19 : cluster [DBG] 6.8 scrub ok 2021-02-10T04:27:37.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:37 smithi085 conmon[29319]: cluster 2021-02-10T04:27:35.777542+0000 osd.7 (osd.7) 20 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:27:37.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:37 smithi085 conmon[29319]: cluster 2021-02-10T04:27:35.777712+0000 osd.7 (osd.7) 21 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:27:37.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:37 smithi085 conmon[29319]: cluster 2021-02-10T04:27:35.976539+0000 mgr.y (mgr.14140) 290 : cluster [DBG] pgmap v250: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:37.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:37 smithi085 conmon[29319]: cluster 2021-02-10T04:27:36.077899+0000 mon.a (mon.0) 696 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2021-02-10T04:27:37.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:37 smithi085 conmon[29319]: audit 2021-02-10T04:27:36.972261+0000 mon.c (mon.1) 26 : 2021-02-10T04:27:37.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:37 smithi085 conmon[29319]: audit [DBG] from='client.? 172.21.15.85:0/1395129641' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2021-02-10T04:27:37.618 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683593754 got 184683593758 for osd.6 2021-02-10T04:27:37.618 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd last-stat-seq osd.7 2021-02-10T04:27:37.817 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:27:37.893 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:27:37 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:27:38.321 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:38 smithi085 conmon[29319]: cluster 2021-02-10T04:27:37.160323+0000 osd.1 (osd.1) 24 : cluster [DBG] 2021-02-10T04:27:38.321 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:38 smithi085 conmon[29319]: 5.c scrub ok 2021-02-10T04:27:38.321 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:38 smithi085 conmon[29319]: cluster 2021-02-10T04:27:37.196840+0000 osd.3 (osd.3) 24 : cluster [DBG] 6.6 scrub ok 2021-02-10T04:27:38.321 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:38 smithi085 conmon[29319]: cluster 2021-02-10T04:27:37.441162+0000 osd.4 (osd.4) 20 : cluster [DBG] 4.1b scrub ok 2021-02-10T04:27:38.321 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:38 smithi085 conmon[29319]: audit 2021-02-10T04:27:38.037130+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:27:38.321 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:38 smithi085 conmon[29319]: audit 2021-02-10T04:27:38.037436+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:27:38.322 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:38 smithi085 conmon[35961]: cluster 2021-02-10T04:27:37.160323+0000 osd.1 ( 2021-02-10T04:27:38.322 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:38 smithi085 conmon[35961]: osd.1) 24 : cluster [DBG] 5.c scrub ok 2021-02-10T04:27:38.322 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:38 smithi085 conmon[35961]: cluster 2021-02-10T04:27:37.196840+0000 osd.3 (osd. 2021-02-10T04:27:38.322 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:38 smithi085 conmon[35961]: 3) 24 : cluster [DBG] 6.6 scrub ok 2021-02-10T04:27:38.322 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:38 smithi085 conmon[35961]: cluster 2021-02-10T04:27:38.322 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:38 smithi085 conmon[35961]: 2021-02-10T04:27:37.441162+0000 osd.4 2021-02-10T04:27:38.322 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:38 smithi085 conmon[35961]: (osd.4) 20 : cluster [DBG] 4.1b scrub ok 2021-02-10T04:27:38.322 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:38 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:27:38.322 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:38 smithi085 conmon[35961]: 02-10T04:27:38.037130+0000 mon.a 2021-02-10T04:27:38.322 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:38 smithi085 conmon[35961]: (mon.0) 697 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:27:38.322 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:38 smithi085 conmon[35961]: 2021-02-10T04:27:38.322 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:38 smithi085 conmon[35961]: audit 2021-02-10T04:27:38.323 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:38 smithi085 conmon[35961]: 2021-02-10T04:27:38.037436+0000 mon.a ( 2021-02-10T04:27:38.323 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:38 smithi085 conmon[35961]: mon.0) 698 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:27:38.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:38 smithi164 conmon[29288]: cluster 2021-02-10T04:27:37.160323+0000 osd.1 (osd.1) 24 : cluster [DBG] 2021-02-10T04:27:38.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:38 smithi164 conmon[29288]: 5.c scrub ok 2021-02-10T04:27:38.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:38 smithi164 conmon[29288]: cluster 2021-02-10T04:27:37.196840+0000 osd.3 (osd.3) 24 : cluster [DBG] 6.6 scrub ok 2021-02-10T04:27:38.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:38 smithi164 conmon[29288]: cluster 2021-02-10T04:27 2021-02-10T04:27:38.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:38 smithi164 conmon[29288]: :37.441162+0000 osd.4 (osd.4) 20 : cluster [DBG] 4.1b scrub ok 2021-02-10T04:27:38.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:38 smithi164 conmon[29288]: audit 2021-02-10T04:27 2021-02-10T04:27:38.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:38 smithi164 conmon[29288]: :38.037130+0000 mon.a (mon.0) 697 : 2021-02-10T04:27:38.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:38 smithi164 conmon[29288]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:27:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:38 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:27:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:38 smithi164 conmon[29288]: :27:38.037436+0000 mon.a (mon.0) 2021-02-10T04:27:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:38 smithi164 conmon[29288]: 698 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:27:39.249 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:39 smithi085 conmon[35961]: audit 2021-02-10T04:27:37.611633+0000 mgr.y (mgr.14140) 291 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:27:39.249 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:39 smithi085 conmon[35961]: cluster 2021-02-10T04:27:37.976935+0000 mgr.y (mgr.14140) 292 : cluster [DBG] pgmap v252: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:27:39.249 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:39 smithi085 conmon[35961]: cluster 2021-02-10T04:27:38.106190+0000 osd.7 (osd.7) 22 : cluster [DBG] 5.4 scrub ok 2021-02-10T04:27:39.249 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:39 smithi085 conmon[35961]: cluster 2021-02-10T04:27:38.176746+0000 osd.1 (osd.1) 25 : cluster [DBG] 4.7 deep-scrub ok 2021-02-10T04:27:39.249 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:39 smithi085 conmon[29319]: audit 2021-02-10T04:27:37.611633+0000 mgr.y (mgr.14140) 291 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:27:39.249 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:39 smithi085 conmon[29319]: cluster 2021-02-10T04:27:37.976935+0000 mgr.y (mgr.14140) 292 : cluster [DBG] pgmap v252: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:27:39.249 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:39 smithi085 conmon[29319]: cluster 2021-02-10T04:27:38.106190+0000 osd.7 (osd.7) 22 : cluster [DBG] 5.4 scrub ok 2021-02-10T04:27:39.249 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:39 smithi085 conmon[29319]: cluster 2021-02-10T04:27:38.176746+0000 osd.1 (osd.1) 25 : cluster [DBG] 4.7 deep-scrub ok 2021-02-10T04:27:39.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:39 smithi164 conmon[29288]: audit 2021-02-10T04:27:37.611633+0000 mgr.y (mgr.14140) 291 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:27:39.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:39 smithi164 conmon[29288]: cluster 2021-02-10T04:27:37.976935+0000 mgr.y (mgr.14140) 292 : cluster [DBG] pgmap v252: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:27:39.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:39 smithi164 conmon[29288]: cluster 2021-02-10T04:27:38.106190+0000 osd.7 (osd.7) 22 : cluster [DBG] 2021-02-10T04:27:39.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:39 smithi164 conmon[29288]: 5.4 scrub ok 2021-02-10T04:27:39.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:39 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:27:39.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:39 smithi164 conmon[29288]: :27:38.176746+0000 osd.1 (osd.1) 25 : cluster [DBG] 4.7 deep-scrub ok 2021-02-10T04:27:39.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:39 smithi164 conmon[29288]: 2021-02-10T04:27:39.569 INFO:teuthology.orchestra.run.smithi085.stdout:214748364826 2021-02-10T04:27:40.267 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748364822 got 214748364826 for osd.7 2021-02-10T04:27:40.267 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2021-02-10T04:27:40.267 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:27:40.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:40 smithi164 conmon[29288]: cluster 2021-02-10T04:27:39.138938+0000 osd.7 (osd.7) 23 : cluster [DBG] 6.1c scrub ok 2021-02-10T04:27:40.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:40 smithi164 conmon[29288]: cluster 2021-02-10T04:27:39.268109+0000 osd.3 (osd.3) 25 : cluster [DBG] 3.2 scrub ok 2021-02-10T04:27:40.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:40 smithi164 conmon[29288]: audit 2021-02-10T04:27:39.570356+0000 mon.a (mon.0) 699 : audit 2021-02-10T04:27:40.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:40 smithi164 conmon[29288]: [DBG] from='client.? 172.21.15.85:0/2195912212' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2021-02-10T04:27:40.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:40 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:27:40.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:40 smithi164 conmon[29288]: 10T04:27:39.616374+0000 osd.0 (osd.0) 27 : cluster [DBG] 2.13 deep-scrub ok 2021-02-10T04:27:40.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:40 smithi085 conmon[29319]: cluster 2021-02-10T04:27:39.138938+0000 2021-02-10T04:27:40.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:40 smithi085 conmon[29319]: osd.7 (osd.7) 23 : cluster [DBG] 6.1c scrub ok 2021-02-10T04:27:40.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:40 smithi085 conmon[29319]: cluster 2021-02-10T04:27:39.268109+0000 osd.3 (osd.3) 25 : cluster [DBG] 3.2 scrub ok 2021-02-10T04:27:40.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:40 smithi085 conmon[29319]: audit 2021-02-10T04:27:39.570356+0000 mon.a (mon.0) 699 : audit [DBG] from='client.? 172.21.15.85:0/2195912212' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2021-02-10T04:27:40.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:40 smithi085 conmon[29319]: cluster 2021-02-10T04:27:39.616374+0000 osd.0 (osd.0) 27 : cluster [DBG] 2.13 deep-scrub ok 2021-02-10T04:27:40.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:40 smithi085 conmon[35961]: cluster 2021-02-10T04:27:39.138938+0000 osd.7 (osd.7 2021-02-10T04:27:40.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:40 smithi085 conmon[35961]: ) 23 : cluster [DBG] 6.1c scrub ok 2021-02-10T04:27:40.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:40 smithi085 conmon[35961]: cluster 2021-02-10T04:27:39.268109+0000 osd.3 (osd.3) 25 : cluster [DBG] 3.2 scrub ok 2021-02-10T04:27:40.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:40 smithi085 conmon[35961]: audit 2021-02-10T04:27:39.570356+0000 mon.a (mon.0) 699 : audit [DBG] from='client.? 172.21.15.85:0/2195912212' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2021-02-10T04:27:40.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:40 smithi085 conmon[35961]: cluster 2021-02-10T04:27:39.616374+0000 osd.0 (osd.0) 27 : cluster [DBG] 2.13 deep-scrub ok 2021-02-10T04:27:40.465 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:27:41.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:41 smithi164 conmon[29288]: cluster 2021-02-10T04:27:39.977616+0000 mgr.y (mgr.14140) 293 : cluster [DBG] pgmap v253: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:27:41.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:41 smithi164 conmon[29288]: cluster 2021-02-10T04:27:40.218823+0000 osd.1 (osd.1) 26 : cluster [DBG] 4.12 scrub ok 2021-02-10T04:27:41.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:41 smithi164 conmon[29288]: cluster 2021-02-10T04:27:40.456438+0000 osd.4 (osd.4) 21 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:27:41.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:41 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:27:41.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:41 smithi164 conmon[29288]: 02-10T04:27:40.456575+0000 osd.4 (osd.4) 22 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:27:41.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:41 smithi085 conmon[29319]: cluster 2021-02-10T04:27:39.977616+0000 mgr.y (mgr.14140 2021-02-10T04:27:41.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:41 smithi085 conmon[29319]: ) 293 : cluster [DBG] pgmap v253: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:27:41.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:41 smithi085 conmon[29319]: cluster 2021-02-10T04:27:40.218823+0000 osd.1 (osd.1) 26 : cluster [DBG] 4.12 scrub ok 2021-02-10T04:27:41.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:41 smithi085 conmon[29319]: cluster 2021-02-10T04:27:40.456438+0000 osd.4 (osd.4) 21 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:27:41.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:41 smithi085 conmon[29319]: cluster 2021-02-10T04:27:40.456575+0000 osd.4 (osd.4) 22 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:27:41.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:41 smithi085 conmon[35961]: cluster 2021-02-10T04:27:39.977616+0000 mgr.y ( 2021-02-10T04:27:41.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:41 smithi085 conmon[35961]: mgr.14140) 293 : cluster [DBG] pgmap v253: 132 pgs: 132 active+clean; 25 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:27:41.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:41 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:27:41.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:41 smithi085 conmon[35961]: 04:27:40.218823+0000 osd.1 (osd.1) 26 : cluster [DBG] 4.12 scrub ok 2021-02-10T04:27:41.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:41 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:27:41.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:41 smithi085 conmon[35961]: 27:40.456438+0000 osd.4 (osd.4 2021-02-10T04:27:41.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:41 smithi085 conmon[35961]: ) 21 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:27:41.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:41 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:27:41.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:41 smithi085 conmon[35961]: -02-10T04:27:40.456575+0000 2021-02-10T04:27:41.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:41 smithi085 conmon[35961]: osd.4 (osd.4) 22 : cluster [DBG] 2021-02-10T04:27:41.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:41 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:27:42.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:42 smithi085 conmon[29319]: cluster 2021-02-10T04:27:41.099563+0000 mon.a 2021-02-10T04:27:42.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:42 smithi085 conmon[29319]: (mon.0) 700 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2021-02-10T04:27:42.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:42 smithi085 conmon[29319]: cluster 2021-02-10T04:27:41.977999+0000 mgr.y (mgr.14140) 294 : cluster [DBG] pgmap v255: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:27:42.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:42 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:27:42.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:42 smithi085 conmon[35961]: :27:41.099563+0000 mon.a (mon.0) 700 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2021-02-10T04:27:42.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:42 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:27:42.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:42 smithi085 conmon[35961]: -02-10T04:27: 2021-02-10T04:27:42.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:42 smithi085 conmon[35961]: 41.977999+0000 mgr.y ( 2021-02-10T04:27:42.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:42 smithi085 conmon[35961]: mgr.14140) 294 : cluster 2021-02-10T04:27:42.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:42 smithi085 conmon[35961]: [DBG] pgmap v255: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:27:42.224 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:27:42.226 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:27:42.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:42 smithi164 conmon[29288]: cluster 2021-02-10T04:27:41.099563+0000 mon.a (mon.0) 700 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2021-02-10T04:27:42.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:42 smithi164 conmon[29288]: cluster 2021-02-10T04:27:41.977999+0000 mgr.y (mgr.14140) 294 : cluster [DBG] pgmap v255: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:27:42.867 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2021-02-10T04:27:41.977764+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":25858,"num_objects":357,"num_object_clones":0,"num_object_copies":1071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":357,"num_whiteouts":0,"num_read":1586,"num_read_kb":1054,"num_write":759,"num_write_kb":177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":779,"ondisk_log_size":779,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":57864,"kb_used_data":6920,"kb_used_omap":0,"kb_used_meta":50880,"kb_avail":749903352,"statfs":{"total":767960285184,"available":767901032448,"internally_reserved":0,"allocated":7086080,"data_stored":2426044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52101120},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002014"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":"35","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.055645+0000","last_change":"2021-02-10T04:27:07.339624+0000","last_active":"2021-02-10T04:27:30.055645+0000","last_peered":"2021-02-10T04:27:30.055645+0000","last_clean":"2021-02-10T04:27:30.055645+0000","last_became_active":"2021-02-10T04:25:53.150917+0000","last_became_peered":"2021-02-10T04:25:53.150917+0000","last_unstale":"2021-02-10T04:27:30.055645+0000","last_undegraded":"2021-02-10T04:27:30.055645+0000","last_fullsized":"2021-02-10T04:27:30.055645+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:07.339551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:07.339551+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"34","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.077776+0000","last_change":"2021-02-10T04:25:51.029620+0000","last_active":"2021-02-10T04:27:36.077776+0000","last_peered":"2021-02-10T04:27:36.077776+0000","last_clean":"2021-02-10T04:27:36.077776+0000","last_became_active":"2021-02-10T04:25:51.029110+0000","last_became_peered":"2021-02-10T04:25:51.029110+0000","last_unstale":"2021-02-10T04:27:36.077776+0000","last_undegraded":"2021-02-10T04:27:36.077776+0000","last_fullsized":"2021-02-10T04:27:36.077776+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:25:49.999907+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"51","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.057588+0000","last_change":"2021-02-10T04:26:44.383142+0000","last_active":"2021-02-10T04:27:30.057588+0000","last_peered":"2021-02-10T04:27:30.057588+0000","last_clean":"2021-02-10T04:27:30.057588+0000","last_became_active":"2021-02-10T04:25:36.336112+0000","last_became_peered":"2021-02-10T04:25:36.336112+0000","last_unstale":"2021-02-10T04:27:30.057588+0000","last_undegraded":"2021-02-10T04:27:30.057588+0000","last_fullsized":"2021-02-10T04:27:30.057588+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:44.383073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:44.383073+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"63'44","reported_seq":"129","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.055532+0000","last_change":"2021-02-10T04:25:49.008766+0000","last_active":"2021-02-10T04:27:30.055532+0000","last_peered":"2021-02-10T04:27:30.055532+0000","last_clean":"2021-02-10T04:27:30.055532+0000","last_became_active":"2021-02-10T04:25:49.008451+0000","last_became_peered":"2021-02-10T04:25:49.008451+0000","last_unstale":"2021-02-10T04:27:30.055532+0000","last_undegraded":"2021-02-10T04:27:30.055532+0000","last_fullsized":"2021-02-10T04:27:30.055532+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:25:47.970100+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":89,"num_read_kb":66,"num_write":45,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"32","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.077977+0000","last_change":"2021-02-10T04:25:53.044249+0000","last_active":"2021-02-10T04:27:36.077977+0000","last_peered":"2021-02-10T04:27:36.077977+0000","last_clean":"2021-02-10T04:27:36.077977+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:27:36.077977+0000","last_undegraded":"2021-02-10T04:27:36.077977+0000","last_fullsized":"2021-02-10T04:27:36.077977+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:25:52.015519+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"38","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.747775+0000","last_change":"2021-02-10T04:26:54.007373+0000","last_active":"2021-02-10T04:27:30.747775+0000","last_peered":"2021-02-10T04:27:30.747775+0000","last_clean":"2021-02-10T04:27:30.747775+0000","last_became_active":"2021-02-10T04:25:51.030405+0000","last_became_peered":"2021-02-10T04:25:51.030405+0000","last_unstale":"2021-02-10T04:27:30.747775+0000","last_undegraded":"2021-02-10T04:27:30.747775+0000","last_fullsized":"2021-02-10T04:27:30.747775+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:54.007310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:26:54.007310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"46","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.056445+0000","last_change":"2021-02-10T04:27:13.332288+0000","last_active":"2021-02-10T04:27:30.056445+0000","last_peered":"2021-02-10T04:27:30.056445+0000","last_clean":"2021-02-10T04:27:30.056445+0000","last_became_active":"2021-02-10T04:25:34.327162+0000","last_became_peered":"2021-02-10T04:25:34.327162+0000","last_unstale":"2021-02-10T04:27:30.056445+0000","last_undegraded":"2021-02-10T04:27:30.056445+0000","last_fullsized":"2021-02-10T04:27:30.056445+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:27:13.332224+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"78'32","reported_seq":"95","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.077930+0000","last_change":"2021-02-10T04:27:09.170492+0000","last_active":"2021-02-10T04:27:36.077930+0000","last_peered":"2021-02-10T04:27:36.077930+0000","last_clean":"2021-02-10T04:27:36.077930+0000","last_became_active":"2021-02-10T04:25:49.010481+0000","last_became_peered":"2021-02-10T04:25:49.010481+0000","last_unstale":"2021-02-10T04:27:36.077930+0000","last_undegraded":"2021-02-10T04:27:36.077930+0000","last_fullsized":"2021-02-10T04:27:36.077930+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'32","last_scrub_stamp":"2021-02-10T04:27:09.170428+0000","last_deep_scrub":"78'32","last_deep_scrub_stamp":"2021-02-10T04:27:09.170428+0000","last_clean_scrub_stamp":"2021-02-10T04:27:09.170428+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":47,"num_read_kb":28,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"36","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748283+0000","last_change":"2021-02-10T04:27:07.962210+0000","last_active":"2021-02-10T04:27:30.748283+0000","last_peered":"2021-02-10T04:27:30.748283+0000","last_clean":"2021-02-10T04:27:30.748283+0000","last_became_active":"2021-02-10T04:25:53.042601+0000","last_became_peered":"2021-02-10T04:25:53.042601+0000","last_unstale":"2021-02-10T04:27:30.748283+0000","last_undegraded":"2021-02-10T04:27:30.748283+0000","last_fullsized":"2021-02-10T04:27:30.748283+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:07.962115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:07.962115+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"38","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748868+0000","last_change":"2021-02-10T04:27:29.251059+0000","last_active":"2021-02-10T04:27:30.748868+0000","last_peered":"2021-02-10T04:27:30.748868+0000","last_clean":"2021-02-10T04:27:30.748868+0000","last_became_active":"2021-02-10T04:25:51.029286+0000","last_became_peered":"2021-02-10T04:25:51.029286+0000","last_unstale":"2021-02-10T04:27:30.748868+0000","last_undegraded":"2021-02-10T04:27:30.748868+0000","last_fullsized":"2021-02-10T04:27:30.748868+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:29.250978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:29.250978+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"45","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.749528+0000","last_change":"2021-02-10T04:26:42.262025+0000","last_active":"2021-02-10T04:27:30.749528+0000","last_peered":"2021-02-10T04:27:30.749528+0000","last_clean":"2021-02-10T04:27:30.749528+0000","last_became_active":"2021-02-10T04:25:36.344477+0000","last_became_peered":"2021-02-10T04:25:36.344477+0000","last_unstale":"2021-02-10T04:27:30.749528+0000","last_undegraded":"2021-02-10T04:27:30.749528+0000","last_fullsized":"2021-02-10T04:27:30.749528+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:42.261921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:42.261921+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"63'12","reported_seq":"68","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:37.441319+0000","last_change":"2021-02-10T04:27:37.441260+0000","last_active":"2021-02-10T04:27:37.441319+0000","last_peered":"2021-02-10T04:27:37.441319+0000","last_clean":"2021-02-10T04:27:37.441319+0000","last_became_active":"2021-02-10T04:25:49.013196+0000","last_became_peered":"2021-02-10T04:25:49.013196+0000","last_unstale":"2021-02-10T04:27:37.441319+0000","last_undegraded":"2021-02-10T04:27:37.441319+0000","last_fullsized":"2021-02-10T04:27:37.441319+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2021-02-10T04:27:37.441168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:37.441168+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"36","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:33.628151+0000","last_change":"2021-02-10T04:27:33.628099+0000","last_active":"2021-02-10T04:27:33.628151+0000","last_peered":"2021-02-10T04:27:33.628151+0000","last_clean":"2021-02-10T04:27:33.628151+0000","last_became_active":"2021-02-10T04:25:53.039844+0000","last_became_peered":"2021-02-10T04:25:53.039844+0000","last_unstale":"2021-02-10T04:27:33.628151+0000","last_undegraded":"2021-02-10T04:27:33.628151+0000","last_fullsized":"2021-02-10T04:27:33.628151+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:33.628038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:33.628038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"38","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.749300+0000","last_change":"2021-02-10T04:27:04.977415+0000","last_active":"2021-02-10T04:27:30.749300+0000","last_peered":"2021-02-10T04:27:30.749300+0000","last_clean":"2021-02-10T04:27:30.749300+0000","last_became_active":"2021-02-10T04:25:51.025488+0000","last_became_peered":"2021-02-10T04:25:51.025488+0000","last_unstale":"2021-02-10T04:27:30.749300+0000","last_undegraded":"2021-02-10T04:27:30.749300+0000","last_fullsized":"2021-02-10T04:27:30.749300+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:04.977353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:04.977353+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"45","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.749155+0000","last_change":"2021-02-10T04:26:45.223669+0000","last_active":"2021-02-10T04:27:30.749155+0000","last_peered":"2021-02-10T04:27:30.749155+0000","last_clean":"2021-02-10T04:27:30.749155+0000","last_became_active":"2021-02-10T04:25:36.337260+0000","last_became_peered":"2021-02-10T04:25:36.337260+0000","last_unstale":"2021-02-10T04:27:30.749155+0000","last_undegraded":"2021-02-10T04:27:30.749155+0000","last_fullsized":"2021-02-10T04:27:30.749155+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:45.223603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:45.223603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"63'25","reported_seq":"95","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.078074+0000","last_change":"2021-02-10T04:27:05.242824+0000","last_active":"2021-02-10T04:27:36.078074+0000","last_peered":"2021-02-10T04:27:36.078074+0000","last_clean":"2021-02-10T04:27:36.078074+0000","last_became_active":"2021-02-10T04:25:49.000911+0000","last_became_peered":"2021-02-10T04:25:49.000911+0000","last_unstale":"2021-02-10T04:27:36.078074+0000","last_undegraded":"2021-02-10T04:27:36.078074+0000","last_fullsized":"2021-02-10T04:27:36.078074+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2021-02-10T04:27:05.242762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:05.242762+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":53,"num_read_kb":37,"num_write":26,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"35","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.056801+0000","last_change":"2021-02-10T04:26:55.332873+0000","last_active":"2021-02-10T04:27:30.056801+0000","last_peered":"2021-02-10T04:27:30.056801+0000","last_clean":"2021-02-10T04:27:30.056801+0000","last_became_active":"2021-02-10T04:25:53.147831+0000","last_became_peered":"2021-02-10T04:25:53.147831+0000","last_unstale":"2021-02-10T04:27:30.056801+0000","last_undegraded":"2021-02-10T04:27:30.056801+0000","last_fullsized":"2021-02-10T04:27:30.056801+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:55.332815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:26:55.332815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"34","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.078343+0000","last_change":"2021-02-10T04:25:51.019289+0000","last_active":"2021-02-10T04:27:36.078343+0000","last_peered":"2021-02-10T04:27:36.078343+0000","last_clean":"2021-02-10T04:27:36.078343+0000","last_became_active":"2021-02-10T04:25:51.018872+0000","last_became_peered":"2021-02-10T04:25:51.018872+0000","last_unstale":"2021-02-10T04:27:36.078343+0000","last_undegraded":"2021-02-10T04:27:36.078343+0000","last_fullsized":"2021-02-10T04:27:36.078343+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:25:49.999907+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"55","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.055716+0000","last_change":"2021-02-10T04:26:46.425375+0000","last_active":"2021-02-10T04:27:30.055716+0000","last_peered":"2021-02-10T04:27:30.055716+0000","last_clean":"2021-02-10T04:27:30.055716+0000","last_became_active":"2021-02-10T04:25:36.336893+0000","last_became_peered":"2021-02-10T04:25:36.336893+0000","last_unstale":"2021-02-10T04:27:30.055716+0000","last_undegraded":"2021-02-10T04:27:30.055716+0000","last_fullsized":"2021-02-10T04:27:30.055716+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:46.425288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:26:46.425288+0000","last_clean_scrub_stamp":"2021-02-10T04:26:46.425288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"62'22","reported_seq":"80","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.056178+0000","last_change":"2021-02-10T04:25:49.009749+0000","last_active":"2021-02-10T04:27:30.056178+0000","last_peered":"2021-02-10T04:27:30.056178+0000","last_clean":"2021-02-10T04:27:30.056178+0000","last_became_active":"2021-02-10T04:25:49.008997+0000","last_became_peered":"2021-02-10T04:25:49.008997+0000","last_unstale":"2021-02-10T04:27:30.056178+0000","last_undegraded":"2021-02-10T04:27:30.056178+0000","last_fullsized":"2021-02-10T04:27:30.056178+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:25:47.970100+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":49,"num_read_kb":31,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"37","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.078715+0000","last_change":"2021-02-10T04:27:29.364660+0000","last_active":"2021-02-10T04:27:36.078715+0000","last_peered":"2021-02-10T04:27:36.078715+0000","last_clean":"2021-02-10T04:27:36.078715+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:27:36.078715+0000","last_undegraded":"2021-02-10T04:27:36.078715+0000","last_fullsized":"2021-02-10T04:27:36.078715+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:29.364585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:29.364585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"38","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748355+0000","last_change":"2021-02-10T04:27:27.017142+0000","last_active":"2021-02-10T04:27:30.748355+0000","last_peered":"2021-02-10T04:27:30.748355+0000","last_clean":"2021-02-10T04:27:30.748355+0000","last_became_active":"2021-02-10T04:25:51.029399+0000","last_became_peered":"2021-02-10T04:25:51.029399+0000","last_unstale":"2021-02-10T04:27:30.748355+0000","last_undegraded":"2021-02-10T04:27:30.748355+0000","last_fullsized":"2021-02-10T04:27:30.748355+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:27.017040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:27.017040+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"50","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.272163+0000","last_change":"2021-02-10T04:27:02.190158+0000","last_active":"2021-02-10T04:27:30.272163+0000","last_peered":"2021-02-10T04:27:30.272163+0000","last_clean":"2021-02-10T04:27:30.272163+0000","last_became_active":"2021-02-10T04:25:36.336442+0000","last_became_peered":"2021-02-10T04:25:36.336442+0000","last_unstale":"2021-02-10T04:27:30.272163+0000","last_undegraded":"2021-02-10T04:27:30.272163+0000","last_fullsized":"2021-02-10T04:27:30.272163+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:27:02.190056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:02.190056+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"62'15","reported_seq":"64","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.164914+0000","last_change":"2021-02-10T04:26:56.623190+0000","last_active":"2021-02-10T04:27:30.164914+0000","last_peered":"2021-02-10T04:27:30.164914+0000","last_clean":"2021-02-10T04:27:30.164914+0000","last_became_active":"2021-02-10T04:25:49.010553+0000","last_became_peered":"2021-02-10T04:25:49.010553+0000","last_unstale":"2021-02-10T04:27:30.164914+0000","last_undegraded":"2021-02-10T04:27:30.164914+0000","last_fullsized":"2021-02-10T04:27:30.164914+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:26:56.623109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:26:56.623109+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"36","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748016+0000","last_change":"2021-02-10T04:27:26.029923+0000","last_active":"2021-02-10T04:27:30.748016+0000","last_peered":"2021-02-10T04:27:30.748016+0000","last_clean":"2021-02-10T04:27:30.748016+0000","last_became_active":"2021-02-10T04:25:53.041098+0000","last_became_peered":"2021-02-10T04:25:53.041098+0000","last_unstale":"2021-02-10T04:27:30.748016+0000","last_undegraded":"2021-02-10T04:27:30.748016+0000","last_fullsized":"2021-02-10T04:27:30.748016+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:26.029856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:26.029856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"38","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.057722+0000","last_change":"2021-02-10T04:27:27.564268+0000","last_active":"2021-02-10T04:27:30.057722+0000","last_peered":"2021-02-10T04:27:30.057722+0000","last_clean":"2021-02-10T04:27:30.057722+0000","last_became_active":"2021-02-10T04:25:51.022853+0000","last_became_peered":"2021-02-10T04:25:51.022853+0000","last_unstale":"2021-02-10T04:27:30.057722+0000","last_undegraded":"2021-02-10T04:27:30.057722+0000","last_fullsized":"2021-02-10T04:27:30.057722+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:27.564175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:27.564175+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"46","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.055338+0000","last_change":"2021-02-10T04:27:16.340830+0000","last_active":"2021-02-10T04:27:30.055338+0000","last_peered":"2021-02-10T04:27:30.055338+0000","last_clean":"2021-02-10T04:27:30.055338+0000","last_became_active":"2021-02-10T04:25:34.324981+0000","last_became_peered":"2021-02-10T04:25:34.324981+0000","last_unstale":"2021-02-10T04:27:30.055338+0000","last_undegraded":"2021-02-10T04:27:30.055338+0000","last_fullsized":"2021-02-10T04:27:30.055338+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:16.340724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:16.340724+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"76'35","reported_seq":"98","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.272675+0000","last_change":"2021-02-10T04:26:58.198706+0000","last_active":"2021-02-10T04:27:30.272675+0000","last_peered":"2021-02-10T04:27:30.272675+0000","last_clean":"2021-02-10T04:27:30.272675+0000","last_became_active":"2021-02-10T04:25:49.015760+0000","last_became_peered":"2021-02-10T04:25:49.015760+0000","last_unstale":"2021-02-10T04:27:30.272675+0000","last_undegraded":"2021-02-10T04:27:30.272675+0000","last_fullsized":"2021-02-10T04:27:30.272675+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'35","last_scrub_stamp":"2021-02-10T04:26:58.198641+0000","last_deep_scrub":"76'35","last_deep_scrub_stamp":"2021-02-10T04:26:58.198641+0000","last_clean_scrub_stamp":"2021-02-10T04:26:58.198641+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":50,"num_read_kb":30,"num_write":24,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"36","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:39.139028+0000","last_change":"2021-02-10T04:27:39.139028+0000","last_active":"2021-02-10T04:27:39.139028+0000","last_peered":"2021-02-10T04:27:39.139028+0000","last_clean":"2021-02-10T04:27:39.139028+0000","last_became_active":"2021-02-10T04:25:53.042279+0000","last_became_peered":"2021-02-10T04:25:53.042279+0000","last_unstale":"2021-02-10T04:27:39.139028+0000","last_undegraded":"2021-02-10T04:27:39.139028+0000","last_fullsized":"2021-02-10T04:27:39.139028+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:39.138947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:39.138947+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"58'2","reported_seq":"62","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.738481+0000","last_change":"2021-02-10T04:27:15.041424+0000","last_active":"2021-02-10T04:27:36.738481+0000","last_peered":"2021-02-10T04:27:36.738481+0000","last_clean":"2021-02-10T04:27:36.738481+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:27:36.738481+0000","last_undegraded":"2021-02-10T04:27:36.738481+0000","last_fullsized":"2021-02-10T04:27:36.738481+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:15.041329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:15.041329+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"56","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748148+0000","last_change":"2021-02-10T04:27:04.011149+0000","last_active":"2021-02-10T04:27:30.748148+0000","last_peered":"2021-02-10T04:27:30.748148+0000","last_clean":"2021-02-10T04:27:30.748148+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:27:30.748148+0000","last_undegraded":"2021-02-10T04:27:30.748148+0000","last_fullsized":"2021-02-10T04:27:30.748148+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:04.011053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:04.011053+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"63'22","reported_seq":"82","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.057739+0000","last_change":"2021-02-10T04:26:51.409218+0000","last_active":"2021-02-10T04:27:30.057739+0000","last_peered":"2021-02-10T04:27:30.057739+0000","last_clean":"2021-02-10T04:27:30.057739+0000","last_became_active":"2021-02-10T04:25:49.009177+0000","last_became_peered":"2021-02-10T04:25:49.009177+0000","last_unstale":"2021-02-10T04:27:30.057739+0000","last_undegraded":"2021-02-10T04:27:30.057739+0000","last_fullsized":"2021-02-10T04:27:30.057739+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2021-02-10T04:26:51.409147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:26:51.409147+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":22,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"30","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.056032+0000","last_change":"2021-02-10T04:25:53.151054+0000","last_active":"2021-02-10T04:27:30.056032+0000","last_peered":"2021-02-10T04:27:30.056032+0000","last_clean":"2021-02-10T04:27:30.056032+0000","last_became_active":"2021-02-10T04:25:53.150845+0000","last_became_peered":"2021-02-10T04:25:53.150845+0000","last_unstale":"2021-02-10T04:27:30.056032+0000","last_undegraded":"2021-02-10T04:27:30.056032+0000","last_fullsized":"2021-02-10T04:27:30.056032+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:25:52.015519+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"38","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748313+0000","last_change":"2021-02-10T04:27:07.181928+0000","last_active":"2021-02-10T04:27:30.748313+0000","last_peered":"2021-02-10T04:27:30.748313+0000","last_clean":"2021-02-10T04:27:30.748313+0000","last_became_active":"2021-02-10T04:25:51.018491+0000","last_became_peered":"2021-02-10T04:25:51.018491+0000","last_unstale":"2021-02-10T04:27:30.748313+0000","last_undegraded":"2021-02-10T04:27:30.748313+0000","last_fullsized":"2021-02-10T04:27:30.748313+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:07.181854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:07.181854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"50","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.272698+0000","last_change":"2021-02-10T04:26:52.256790+0000","last_active":"2021-02-10T04:27:30.272698+0000","last_peered":"2021-02-10T04:27:30.272698+0000","last_clean":"2021-02-10T04:27:30.272698+0000","last_became_active":"2021-02-10T04:25:35.328828+0000","last_became_peered":"2021-02-10T04:25:35.328828+0000","last_unstale":"2021-02-10T04:27:30.272698+0000","last_undegraded":"2021-02-10T04:27:30.272698+0000","last_fullsized":"2021-02-10T04:27:30.272698+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:26:52.256723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:52.256723+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"63'21","reported_seq":"80","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.055480+0000","last_change":"2021-02-10T04:27:12.291860+0000","last_active":"2021-02-10T04:27:30.055480+0000","last_peered":"2021-02-10T04:27:30.055480+0000","last_clean":"2021-02-10T04:27:30.055480+0000","last_became_active":"2021-02-10T04:25:49.007049+0000","last_became_peered":"2021-02-10T04:25:49.007049+0000","last_unstale":"2021-02-10T04:27:30.055480+0000","last_undegraded":"2021-02-10T04:27:30.055480+0000","last_fullsized":"2021-02-10T04:27:30.055480+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2021-02-10T04:27:12.291796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:12.291796+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":27,"num_write":21,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"36","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748346+0000","last_change":"2021-02-10T04:27:19.134286+0000","last_active":"2021-02-10T04:27:30.748346+0000","last_peered":"2021-02-10T04:27:30.748346+0000","last_clean":"2021-02-10T04:27:30.748346+0000","last_became_active":"2021-02-10T04:25:53.034771+0000","last_became_peered":"2021-02-10T04:25:53.034771+0000","last_unstale":"2021-02-10T04:27:30.748346+0000","last_undegraded":"2021-02-10T04:27:30.748346+0000","last_fullsized":"2021-02-10T04:27:30.748346+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:27:19.134212+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"38","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.272056+0000","last_change":"2021-02-10T04:27:19.077862+0000","last_active":"2021-02-10T04:27:30.272056+0000","last_peered":"2021-02-10T04:27:30.272056+0000","last_clean":"2021-02-10T04:27:30.272056+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:27:30.272056+0000","last_undegraded":"2021-02-10T04:27:30.272056+0000","last_fullsized":"2021-02-10T04:27:30.272056+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:27:19.077777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"47","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748651+0000","last_change":"2021-02-10T04:27:03.000902+0000","last_active":"2021-02-10T04:27:30.748651+0000","last_peered":"2021-02-10T04:27:30.748651+0000","last_clean":"2021-02-10T04:27:30.748651+0000","last_became_active":"2021-02-10T04:25:34.331787+0000","last_became_peered":"2021-02-10T04:25:34.331787+0000","last_unstale":"2021-02-10T04:27:30.748651+0000","last_undegraded":"2021-02-10T04:27:30.748651+0000","last_fullsized":"2021-02-10T04:27:30.748651+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:03.000821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:03.000821+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"62'12","reported_seq":"87","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.055246+0000","last_change":"2021-02-10T04:26:39.021685+0000","last_active":"2021-02-10T04:27:30.055246+0000","last_peered":"2021-02-10T04:27:30.055246+0000","last_clean":"2021-02-10T04:27:30.055246+0000","last_became_active":"2021-02-10T04:26:39.005616+0000","last_became_peered":"2021-02-10T04:26:39.005616+0000","last_unstale":"2021-02-10T04:27:30.055246+0000","last_undegraded":"2021-02-10T04:27:30.055246+0000","last_fullsized":"2021-02-10T04:27:30.055246+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:25:47.970100+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"35","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.055176+0000","last_change":"2021-02-10T04:26:57.309978+0000","last_active":"2021-02-10T04:27:30.055176+0000","last_peered":"2021-02-10T04:27:30.055176+0000","last_clean":"2021-02-10T04:27:30.055176+0000","last_became_active":"2021-02-10T04:25:53.043277+0000","last_became_peered":"2021-02-10T04:25:53.043277+0000","last_unstale":"2021-02-10T04:27:30.055176+0000","last_undegraded":"2021-02-10T04:27:30.055176+0000","last_fullsized":"2021-02-10T04:27:30.055176+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:57.309915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:26:57.309915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"38","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:33.030073+0000","last_change":"2021-02-10T04:27:33.030004+0000","last_active":"2021-02-10T04:27:33.030073+0000","last_peered":"2021-02-10T04:27:33.030073+0000","last_clean":"2021-02-10T04:27:33.030073+0000","last_became_active":"2021-02-10T04:25:51.030952+0000","last_became_peered":"2021-02-10T04:25:51.030952+0000","last_unstale":"2021-02-10T04:27:33.030073+0000","last_undegraded":"2021-02-10T04:27:33.030073+0000","last_fullsized":"2021-02-10T04:27:33.030073+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:27:33.029912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"56","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748175+0000","last_change":"2021-02-10T04:26:52.011165+0000","last_active":"2021-02-10T04:27:30.748175+0000","last_peered":"2021-02-10T04:27:30.748175+0000","last_clean":"2021-02-10T04:27:30.748175+0000","last_became_active":"2021-02-10T04:25:34.330242+0000","last_became_peered":"2021-02-10T04:25:34.330242+0000","last_unstale":"2021-02-10T04:27:30.748175+0000","last_undegraded":"2021-02-10T04:27:30.748175+0000","last_fullsized":"2021-02-10T04:27:30.748175+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:26:52.011096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:52.011096+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"62'15","reported_seq":"63","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.078209+0000","last_change":"2021-02-10T04:25:49.011596+0000","last_active":"2021-02-10T04:27:36.078209+0000","last_peered":"2021-02-10T04:27:36.078209+0000","last_clean":"2021-02-10T04:27:36.078209+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:27:36.078209+0000","last_undegraded":"2021-02-10T04:27:36.078209+0000","last_fullsized":"2021-02-10T04:27:36.078209+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:25:47.970100+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"36","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.056976+0000","last_change":"2021-02-10T04:27:21.548446+0000","last_active":"2021-02-10T04:27:30.056976+0000","last_peered":"2021-02-10T04:27:30.056976+0000","last_clean":"2021-02-10T04:27:30.056976+0000","last_became_active":"2021-02-10T04:25:53.039322+0000","last_became_peered":"2021-02-10T04:25:53.039322+0000","last_unstale":"2021-02-10T04:27:30.056976+0000","last_undegraded":"2021-02-10T04:27:30.056976+0000","last_fullsized":"2021-02-10T04:27:30.056976+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:21.548381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:21.548381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"37","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.055886+0000","last_change":"2021-02-10T04:27:05.297715+0000","last_active":"2021-02-10T04:27:30.055886+0000","last_peered":"2021-02-10T04:27:30.055886+0000","last_clean":"2021-02-10T04:27:30.055886+0000","last_became_active":"2021-02-10T04:25:51.029155+0000","last_became_peered":"2021-02-10T04:25:51.029155+0000","last_unstale":"2021-02-10T04:27:30.055886+0000","last_undegraded":"2021-02-10T04:27:30.055886+0000","last_fullsized":"2021-02-10T04:27:30.055886+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:27:05.297647+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"47","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.272247+0000","last_change":"2021-02-10T04:26:54.256587+0000","last_active":"2021-02-10T04:27:30.272247+0000","last_peered":"2021-02-10T04:27:30.272247+0000","last_clean":"2021-02-10T04:27:30.272247+0000","last_became_active":"2021-02-10T04:25:34.330070+0000","last_became_peered":"2021-02-10T04:25:34.330070+0000","last_unstale":"2021-02-10T04:27:30.272247+0000","last_undegraded":"2021-02-10T04:27:30.272247+0000","last_fullsized":"2021-02-10T04:27:30.272247+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:54.256498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:54.256498+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"62'15","reported_seq":"63","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748003+0000","last_change":"2021-02-10T04:25:49.015973+0000","last_active":"2021-02-10T04:27:30.748003+0000","last_peered":"2021-02-10T04:27:30.748003+0000","last_clean":"2021-02-10T04:27:30.748003+0000","last_became_active":"2021-02-10T04:25:49.015543+0000","last_became_peered":"2021-02-10T04:25:49.015543+0000","last_unstale":"2021-02-10T04:27:30.748003+0000","last_undegraded":"2021-02-10T04:27:30.748003+0000","last_fullsized":"2021-02-10T04:27:30.748003+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:25:47.970100+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"37","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.078878+0000","last_change":"2021-02-10T04:27:28.373194+0000","last_active":"2021-02-10T04:27:36.078878+0000","last_peered":"2021-02-10T04:27:36.078878+0000","last_clean":"2021-02-10T04:27:36.078878+0000","last_became_active":"2021-02-10T04:25:53.031782+0000","last_became_peered":"2021-02-10T04:25:53.031782+0000","last_unstale":"2021-02-10T04:27:36.078878+0000","last_undegraded":"2021-02-10T04:27:36.078878+0000","last_fullsized":"2021-02-10T04:27:36.078878+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:28.373117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:28.373117+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"58'2","reported_seq":"61","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.079389+0000","last_change":"2021-02-10T04:27:26.264628+0000","last_active":"2021-02-10T04:27:36.079389+0000","last_peered":"2021-02-10T04:27:36.079389+0000","last_clean":"2021-02-10T04:27:36.079389+0000","last_became_active":"2021-02-10T04:25:51.018300+0000","last_became_peered":"2021-02-10T04:25:51.018300+0000","last_unstale":"2021-02-10T04:27:36.079389+0000","last_undegraded":"2021-02-10T04:27:36.079389+0000","last_fullsized":"2021-02-10T04:27:36.079389+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:26.264560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:26.264560+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"52","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:39.616474+0000","last_change":"2021-02-10T04:27:39.616474+0000","last_active":"2021-02-10T04:27:39.616474+0000","last_peered":"2021-02-10T04:27:39.616474+0000","last_clean":"2021-02-10T04:27:39.616474+0000","last_became_active":"2021-02-10T04:25:34.321096+0000","last_became_peered":"2021-02-10T04:25:34.321096+0000","last_unstale":"2021-02-10T04:27:39.616474+0000","last_undegraded":"2021-02-10T04:27:39.616474+0000","last_fullsized":"2021-02-10T04:27:39.616474+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:27:39.616381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"63'19","reported_seq":"79","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.749622+0000","last_change":"2021-02-10T04:27:11.016733+0000","last_active":"2021-02-10T04:27:30.749622+0000","last_peered":"2021-02-10T04:27:30.749622+0000","last_clean":"2021-02-10T04:27:30.749622+0000","last_became_active":"2021-02-10T04:25:49.010890+0000","last_became_peered":"2021-02-10T04:25:49.010890+0000","last_unstale":"2021-02-10T04:27:30.749622+0000","last_undegraded":"2021-02-10T04:27:30.749622+0000","last_fullsized":"2021-02-10T04:27:30.749622+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2021-02-10T04:27:11.016650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:11.016650+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":26,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"36","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.057357+0000","last_change":"2021-02-10T04:27:08.486638+0000","last_active":"2021-02-10T04:27:30.057357+0000","last_peered":"2021-02-10T04:27:30.057357+0000","last_clean":"2021-02-10T04:27:30.057357+0000","last_became_active":"2021-02-10T04:25:53.045177+0000","last_became_peered":"2021-02-10T04:25:53.045177+0000","last_unstale":"2021-02-10T04:27:30.057357+0000","last_undegraded":"2021-02-10T04:27:30.057357+0000","last_fullsized":"2021-02-10T04:27:30.057357+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:08.486553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:08.486553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"58'2","reported_seq":"62","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.378260+0000","last_change":"2021-02-10T04:27:20.157871+0000","last_active":"2021-02-10T04:27:36.378260+0000","last_peered":"2021-02-10T04:27:36.378260+0000","last_clean":"2021-02-10T04:27:36.378260+0000","last_became_active":"2021-02-10T04:25:51.032328+0000","last_became_peered":"2021-02-10T04:25:51.032328+0000","last_unstale":"2021-02-10T04:27:36.378260+0000","last_undegraded":"2021-02-10T04:27:36.378260+0000","last_fullsized":"2021-02-10T04:27:36.378260+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:20.157772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:20.157772+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"52'4","reported_seq":"59","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748626+0000","last_change":"2021-02-10T04:26:49.013781+0000","last_active":"2021-02-10T04:27:30.748626+0000","last_peered":"2021-02-10T04:27:30.748626+0000","last_clean":"2021-02-10T04:27:30.748626+0000","last_became_active":"2021-02-10T04:25:36.334366+0000","last_became_peered":"2021-02-10T04:25:36.334366+0000","last_unstale":"2021-02-10T04:27:30.748626+0000","last_undegraded":"2021-02-10T04:27:30.748626+0000","last_fullsized":"2021-02-10T04:27:30.748626+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'4","last_scrub_stamp":"2021-02-10T04:26:49.013681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:49.013681+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"63'26","reported_seq":"125","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.055671+0000","last_change":"2021-02-10T04:27:28.213521+0000","last_active":"2021-02-10T04:27:30.055671+0000","last_peered":"2021-02-10T04:27:30.055671+0000","last_clean":"2021-02-10T04:27:30.055671+0000","last_became_active":"2021-02-10T04:26:39.003971+0000","last_became_peered":"2021-02-10T04:26:39.003971+0000","last_unstale":"2021-02-10T04:27:30.055671+0000","last_undegraded":"2021-02-10T04:27:30.055671+0000","last_fullsized":"2021-02-10T04:27:30.055671+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:27:28.213454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:28.213454+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"36","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:33.192435+0000","last_change":"2021-02-10T04:27:33.192375+0000","last_active":"2021-02-10T04:27:33.192435+0000","last_peered":"2021-02-10T04:27:33.192435+0000","last_clean":"2021-02-10T04:27:33.192435+0000","last_became_active":"2021-02-10T04:25:53.149820+0000","last_became_peered":"2021-02-10T04:25:53.149820+0000","last_unstale":"2021-02-10T04:27:33.192435+0000","last_undegraded":"2021-02-10T04:27:33.192435+0000","last_fullsized":"2021-02-10T04:27:33.192435+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:33.192287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:33.192287+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"38","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748573+0000","last_change":"2021-02-10T04:27:23.236953+0000","last_active":"2021-02-10T04:27:30.748573+0000","last_peered":"2021-02-10T04:27:30.748573+0000","last_clean":"2021-02-10T04:27:30.748573+0000","last_became_active":"2021-02-10T04:25:51.031864+0000","last_became_peered":"2021-02-10T04:25:51.031864+0000","last_unstale":"2021-02-10T04:27:30.748573+0000","last_undegraded":"2021-02-10T04:27:30.748573+0000","last_fullsized":"2021-02-10T04:27:30.748573+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:23.236870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:23.236870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"49","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.272081+0000","last_change":"2021-02-10T04:27:14.083910+0000","last_active":"2021-02-10T04:27:30.272081+0000","last_peered":"2021-02-10T04:27:30.272081+0000","last_clean":"2021-02-10T04:27:30.272081+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:27:30.272081+0000","last_undegraded":"2021-02-10T04:27:30.272081+0000","last_fullsized":"2021-02-10T04:27:30.272081+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:27:14.083841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:14.083841+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"63'13","reported_seq":"86","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.056756+0000","last_change":"2021-02-10T04:26:50.393926+0000","last_active":"2021-02-10T04:27:30.056756+0000","last_peered":"2021-02-10T04:27:30.056756+0000","last_clean":"2021-02-10T04:27:30.056756+0000","last_became_active":"2021-02-10T04:26:39.003832+0000","last_became_peered":"2021-02-10T04:26:39.003832+0000","last_unstale":"2021-02-10T04:27:30.056756+0000","last_undegraded":"2021-02-10T04:27:30.056756+0000","last_fullsized":"2021-02-10T04:27:30.056756+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2021-02-10T04:26:50.393865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:26:50.393865+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"35","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.164646+0000","last_change":"2021-02-10T04:27:13.573977+0000","last_active":"2021-02-10T04:27:30.164646+0000","last_peered":"2021-02-10T04:27:30.164646+0000","last_clean":"2021-02-10T04:27:30.164646+0000","last_became_active":"2021-02-10T04:25:53.035867+0000","last_became_peered":"2021-02-10T04:25:53.035867+0000","last_unstale":"2021-02-10T04:27:30.164646+0000","last_undegraded":"2021-02-10T04:27:30.164646+0000","last_fullsized":"2021-02-10T04:27:30.164646+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:13.573912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:13.573912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"37","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.056833+0000","last_change":"2021-02-10T04:26:56.335818+0000","last_active":"2021-02-10T04:27:30.056833+0000","last_peered":"2021-02-10T04:27:30.056833+0000","last_clean":"2021-02-10T04:27:30.056833+0000","last_became_active":"2021-02-10T04:25:51.030651+0000","last_became_peered":"2021-02-10T04:25:51.030651+0000","last_unstale":"2021-02-10T04:27:30.056833+0000","last_undegraded":"2021-02-10T04:27:30.056833+0000","last_fullsized":"2021-02-10T04:27:30.056833+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:56.335735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:26:56.335735+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"49","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.164398+0000","last_change":"2021-02-10T04:26:57.639364+0000","last_active":"2021-02-10T04:27:30.164398+0000","last_peered":"2021-02-10T04:27:30.164398+0000","last_clean":"2021-02-10T04:27:30.164398+0000","last_became_active":"2021-02-10T04:25:34.329694+0000","last_became_peered":"2021-02-10T04:25:34.329694+0000","last_unstale":"2021-02-10T04:27:30.164398+0000","last_undegraded":"2021-02-10T04:27:30.164398+0000","last_fullsized":"2021-02-10T04:27:30.164398+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:26:57.639281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:57.639281+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"63'28","reported_seq":"100","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.057492+0000","last_change":"2021-02-10T04:27:24.523914+0000","last_active":"2021-02-10T04:27:30.057492+0000","last_peered":"2021-02-10T04:27:30.057492+0000","last_clean":"2021-02-10T04:27:30.057492+0000","last_became_active":"2021-02-10T04:25:49.008942+0000","last_became_peered":"2021-02-10T04:25:49.008942+0000","last_unstale":"2021-02-10T04:27:30.057492+0000","last_undegraded":"2021-02-10T04:27:30.057492+0000","last_fullsized":"2021-02-10T04:27:30.057492+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'28","last_scrub_stamp":"2021-02-10T04:27:24.523837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:24.523837+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":62,"num_read_kb":40,"num_write":29,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"35","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.055959+0000","last_change":"2021-02-10T04:27:27.259500+0000","last_active":"2021-02-10T04:27:30.055959+0000","last_peered":"2021-02-10T04:27:30.055959+0000","last_clean":"2021-02-10T04:27:30.055959+0000","last_became_active":"2021-02-10T04:25:53.047736+0000","last_became_peered":"2021-02-10T04:25:53.047736+0000","last_unstale":"2021-02-10T04:27:30.055959+0000","last_undegraded":"2021-02-10T04:27:30.055959+0000","last_fullsized":"2021-02-10T04:27:30.055959+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:27.259393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:27.259393+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"61","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.078740+0000","last_change":"2021-02-10T04:26:34.066242+0000","last_active":"2021-02-10T04:27:36.078740+0000","last_peered":"2021-02-10T04:27:36.078740+0000","last_clean":"2021-02-10T04:27:36.078740+0000","last_became_active":"2021-02-10T04:25:36.337394+0000","last_became_peered":"2021-02-10T04:25:36.337394+0000","last_unstale":"2021-02-10T04:27:36.078740+0000","last_undegraded":"2021-02-10T04:27:36.078740+0000","last_fullsized":"2021-02-10T04:27:36.078740+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:26:34.066149+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"63'27","reported_seq":"100","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.077936+0000","last_change":"2021-02-10T04:27:17.352786+0000","last_active":"2021-02-10T04:27:36.077936+0000","last_peered":"2021-02-10T04:27:36.077936+0000","last_clean":"2021-02-10T04:27:36.077936+0000","last_became_active":"2021-02-10T04:25:49.013259+0000","last_became_peered":"2021-02-10T04:25:49.013259+0000","last_unstale":"2021-02-10T04:27:36.077936+0000","last_undegraded":"2021-02-10T04:27:36.077936+0000","last_fullsized":"2021-02-10T04:27:36.077936+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'27","last_scrub_stamp":"2021-02-10T04:27:17.352701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:17.352701+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":41,"num_write":29,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"37","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.164393+0000","last_change":"2021-02-10T04:27:14.530439+0000","last_active":"2021-02-10T04:27:30.164393+0000","last_peered":"2021-02-10T04:27:30.164393+0000","last_clean":"2021-02-10T04:27:30.164393+0000","last_became_active":"2021-02-10T04:25:51.023000+0000","last_became_peered":"2021-02-10T04:25:51.023000+0000","last_unstale":"2021-02-10T04:27:30.164393+0000","last_undegraded":"2021-02-10T04:27:30.164393+0000","last_fullsized":"2021-02-10T04:27:30.164393+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:14.530378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:14.530378+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"36","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.056976+0000","last_change":"2021-02-10T04:27:18.521962+0000","last_active":"2021-02-10T04:27:30.056976+0000","last_peered":"2021-02-10T04:27:30.056976+0000","last_clean":"2021-02-10T04:27:30.056976+0000","last_became_active":"2021-02-10T04:25:53.045215+0000","last_became_peered":"2021-02-10T04:25:53.045215+0000","last_unstale":"2021-02-10T04:27:30.056976+0000","last_undegraded":"2021-02-10T04:27:30.056976+0000","last_fullsized":"2021-02-10T04:27:30.056976+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:18.521897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:18.521897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"65","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748107+0000","last_change":"2021-02-10T04:27:04.024909+0000","last_active":"2021-02-10T04:27:30.748107+0000","last_peered":"2021-02-10T04:27:30.748107+0000","last_clean":"2021-02-10T04:27:30.748107+0000","last_became_active":"2021-02-10T04:26:39.004318+0000","last_became_peered":"2021-02-10T04:26:39.004318+0000","last_unstale":"2021-02-10T04:27:30.748107+0000","last_undegraded":"2021-02-10T04:27:30.748107+0000","last_fullsized":"2021-02-10T04:27:30.748107+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:04.024848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:04.024848+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"79'34","reported_seq":"94","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748008+0000","last_change":"2021-02-10T04:27:18.990156+0000","last_active":"2021-02-10T04:27:30.748008+0000","last_peered":"2021-02-10T04:27:30.748008+0000","last_clean":"2021-02-10T04:27:30.748008+0000","last_became_active":"2021-02-10T04:25:49.006147+0000","last_became_peered":"2021-02-10T04:25:49.006147+0000","last_unstale":"2021-02-10T04:27:30.748008+0000","last_undegraded":"2021-02-10T04:27:30.748008+0000","last_fullsized":"2021-02-10T04:27:30.748008+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'34","last_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:27:18.990088+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":22,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"58'2","reported_seq":"62","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.689167+0000","last_change":"2021-02-10T04:27:03.440844+0000","last_active":"2021-02-10T04:27:36.689167+0000","last_peered":"2021-02-10T04:27:36.689167+0000","last_clean":"2021-02-10T04:27:36.689167+0000","last_became_active":"2021-02-10T04:25:51.026251+0000","last_became_peered":"2021-02-10T04:25:51.026251+0000","last_unstale":"2021-02-10T04:27:36.689167+0000","last_undegraded":"2021-02-10T04:27:36.689167+0000","last_fullsized":"2021-02-10T04:27:36.689167+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:03.440780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:03.440780+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"35","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:37.196914+0000","last_change":"2021-02-10T04:27:37.196914+0000","last_active":"2021-02-10T04:27:37.196914+0000","last_peered":"2021-02-10T04:27:37.196914+0000","last_clean":"2021-02-10T04:27:37.196914+0000","last_became_active":"2021-02-10T04:25:53.049487+0000","last_became_peered":"2021-02-10T04:25:53.049487+0000","last_unstale":"2021-02-10T04:27:37.196914+0000","last_undegraded":"2021-02-10T04:27:37.196914+0000","last_fullsized":"2021-02-10T04:27:37.196914+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:37.196846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:37.196846+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"47","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748709+0000","last_change":"2021-02-10T04:27:06.934347+0000","last_active":"2021-02-10T04:27:30.748709+0000","last_peered":"2021-02-10T04:27:30.748709+0000","last_clean":"2021-02-10T04:27:30.748709+0000","last_became_active":"2021-02-10T04:25:34.331779+0000","last_became_peered":"2021-02-10T04:25:34.331779+0000","last_unstale":"2021-02-10T04:27:30.748709+0000","last_undegraded":"2021-02-10T04:27:30.748709+0000","last_fullsized":"2021-02-10T04:27:30.748709+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:06.934282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:06.934282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"63'31","reported_seq":"102","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748158+0000","last_change":"2021-02-10T04:26:56.059248+0000","last_active":"2021-02-10T04:27:30.748158+0000","last_peered":"2021-02-10T04:27:30.748158+0000","last_clean":"2021-02-10T04:27:30.748158+0000","last_became_active":"2021-02-10T04:25:49.007872+0000","last_became_peered":"2021-02-10T04:25:49.007872+0000","last_unstale":"2021-02-10T04:27:30.748158+0000","last_undegraded":"2021-02-10T04:27:30.748158+0000","last_fullsized":"2021-02-10T04:27:30.748158+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'31","last_scrub_stamp":"2021-02-10T04:26:56.059186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:26:56.059186+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":66,"num_read_kb":42,"num_write":34,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"38","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:31.650183+0000","last_change":"2021-02-10T04:27:31.650126+0000","last_active":"2021-02-10T04:27:31.650183+0000","last_peered":"2021-02-10T04:27:31.650183+0000","last_clean":"2021-02-10T04:27:31.650183+0000","last_became_active":"2021-02-10T04:25:51.027624+0000","last_became_peered":"2021-02-10T04:25:51.027624+0000","last_unstale":"2021-02-10T04:27:31.650183+0000","last_undegraded":"2021-02-10T04:27:31.650183+0000","last_fullsized":"2021-02-10T04:27:31.650183+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:27:31.650033+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"36","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.749660+0000","last_change":"2021-02-10T04:27:28.264288+0000","last_active":"2021-02-10T04:27:30.749660+0000","last_peered":"2021-02-10T04:27:30.749660+0000","last_clean":"2021-02-10T04:27:30.749660+0000","last_became_active":"2021-02-10T04:25:53.149677+0000","last_became_peered":"2021-02-10T04:25:53.149677+0000","last_unstale":"2021-02-10T04:27:30.749660+0000","last_undegraded":"2021-02-10T04:27:30.749660+0000","last_fullsized":"2021-02-10T04:27:30.749660+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:28.264204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:28.264204+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"149","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.690248+0000","last_change":"2021-02-10T04:26:58.989515+0000","last_active":"2021-02-10T04:27:36.690248+0000","last_peered":"2021-02-10T04:27:36.690248+0000","last_clean":"2021-02-10T04:27:36.690248+0000","last_became_active":"2021-02-10T04:25:49.013969+0000","last_became_peered":"2021-02-10T04:25:49.013969+0000","last_unstale":"2021-02-10T04:27:36.690248+0000","last_undegraded":"2021-02-10T04:27:36.690248+0000","last_fullsized":"2021-02-10T04:27:36.690248+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:26:58.989447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:26:58.989447+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":106,"num_read_kb":101,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"46","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.164411+0000","last_change":"2021-02-10T04:26:42.627320+0000","last_active":"2021-02-10T04:27:30.164411+0000","last_peered":"2021-02-10T04:27:30.164411+0000","last_clean":"2021-02-10T04:27:30.164411+0000","last_became_active":"2021-02-10T04:25:34.331972+0000","last_became_peered":"2021-02-10T04:25:34.331972+0000","last_unstale":"2021-02-10T04:27:30.164411+0000","last_undegraded":"2021-02-10T04:27:30.164411+0000","last_fullsized":"2021-02-10T04:27:30.164411+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:42.627255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:42.627255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'21","reported_seq":"81","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748507+0000","last_change":"2021-02-10T04:25:49.003171+0000","last_active":"2021-02-10T04:27:30.748507+0000","last_peered":"2021-02-10T04:27:30.748507+0000","last_clean":"2021-02-10T04:27:30.748507+0000","last_became_active":"2021-02-10T04:25:49.002843+0000","last_became_peered":"2021-02-10T04:25:49.002843+0000","last_unstale":"2021-02-10T04:27:30.748507+0000","last_undegraded":"2021-02-10T04:27:30.748507+0000","last_fullsized":"2021-02-10T04:27:30.748507+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:25:47.970100+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":32,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"37","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.164723+0000","last_change":"2021-02-10T04:27:16.564733+0000","last_active":"2021-02-10T04:27:30.164723+0000","last_peered":"2021-02-10T04:27:30.164723+0000","last_clean":"2021-02-10T04:27:30.164723+0000","last_became_active":"2021-02-10T04:25:51.025306+0000","last_became_peered":"2021-02-10T04:25:51.025306+0000","last_unstale":"2021-02-10T04:27:30.164723+0000","last_undegraded":"2021-02-10T04:27:30.164723+0000","last_fullsized":"2021-02-10T04:27:30.164723+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:27:16.564638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"36","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.749705+0000","last_change":"2021-02-10T04:27:18.115804+0000","last_active":"2021-02-10T04:27:30.749705+0000","last_peered":"2021-02-10T04:27:30.749705+0000","last_clean":"2021-02-10T04:27:30.749705+0000","last_became_active":"2021-02-10T04:25:53.037506+0000","last_became_peered":"2021-02-10T04:25:53.037506+0000","last_unstale":"2021-02-10T04:27:30.749705+0000","last_undegraded":"2021-02-10T04:27:30.749705+0000","last_fullsized":"2021-02-10T04:27:30.749705+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:18.115741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:18.115741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"39","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:39.268187+0000","last_change":"2021-02-10T04:27:39.268187+0000","last_active":"2021-02-10T04:27:39.268187+0000","last_peered":"2021-02-10T04:27:39.268187+0000","last_clean":"2021-02-10T04:27:39.268187+0000","last_became_active":"2021-02-10T04:25:49.010614+0000","last_became_peered":"2021-02-10T04:25:49.010614+0000","last_unstale":"2021-02-10T04:27:39.268187+0000","last_undegraded":"2021-02-10T04:27:39.268187+0000","last_fullsized":"2021-02-10T04:27:39.268187+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:39.268114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:39.268114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"49","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.749242+0000","last_change":"2021-02-10T04:26:38.013836+0000","last_active":"2021-02-10T04:27:30.749242+0000","last_peered":"2021-02-10T04:27:30.749242+0000","last_clean":"2021-02-10T04:27:30.749242+0000","last_became_active":"2021-02-10T04:25:36.334230+0000","last_became_peered":"2021-02-10T04:25:36.334230+0000","last_unstale":"2021-02-10T04:27:30.749242+0000","last_undegraded":"2021-02-10T04:27:30.749242+0000","last_fullsized":"2021-02-10T04:27:30.749242+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:38.013770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:38.013770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"46","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748194+0000","last_change":"2021-02-10T04:26:44.252455+0000","last_active":"2021-02-10T04:27:30.748194+0000","last_peered":"2021-02-10T04:27:30.748194+0000","last_clean":"2021-02-10T04:27:30.748194+0000","last_became_active":"2021-02-10T04:25:36.343397+0000","last_became_peered":"2021-02-10T04:25:36.343397+0000","last_unstale":"2021-02-10T04:27:30.748194+0000","last_undegraded":"2021-02-10T04:27:30.748194+0000","last_fullsized":"2021-02-10T04:27:30.748194+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:44.252351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:26:44.252351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"62'15","reported_seq":"64","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.272524+0000","last_change":"2021-02-10T04:27:08.069986+0000","last_active":"2021-02-10T04:27:30.272524+0000","last_peered":"2021-02-10T04:27:30.272524+0000","last_clean":"2021-02-10T04:27:30.272524+0000","last_became_active":"2021-02-10T04:25:49.002471+0000","last_became_peered":"2021-02-10T04:25:49.002471+0000","last_unstale":"2021-02-10T04:27:30.272524+0000","last_undegraded":"2021-02-10T04:27:30.272524+0000","last_fullsized":"2021-02-10T04:27:30.272524+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:27:08.069899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:08.069899+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":15,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"38","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:38.106291+0000","last_change":"2021-02-10T04:27:38.106291+0000","last_active":"2021-02-10T04:27:38.106291+0000","last_peered":"2021-02-10T04:27:38.106291+0000","last_clean":"2021-02-10T04:27:38.106291+0000","last_became_active":"2021-02-10T04:25:51.022821+0000","last_became_peered":"2021-02-10T04:25:51.022821+0000","last_unstale":"2021-02-10T04:27:38.106291+0000","last_undegraded":"2021-02-10T04:27:38.106291+0000","last_fullsized":"2021-02-10T04:27:38.106291+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:38.106209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:38.106209+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"36","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748234+0000","last_change":"2021-02-10T04:27:10.051857+0000","last_active":"2021-02-10T04:27:30.748234+0000","last_peered":"2021-02-10T04:27:30.748234+0000","last_clean":"2021-02-10T04:27:30.748234+0000","last_became_active":"2021-02-10T04:25:53.040390+0000","last_became_peered":"2021-02-10T04:25:53.040390+0000","last_unstale":"2021-02-10T04:27:30.748234+0000","last_undegraded":"2021-02-10T04:27:30.748234+0000","last_fullsized":"2021-02-10T04:27:30.748234+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:10.051800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:10.051800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"40","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.657900+0000","last_change":"2021-02-10T04:27:30.657844+0000","last_active":"2021-02-10T04:27:30.657900+0000","last_peered":"2021-02-10T04:27:30.657900+0000","last_clean":"2021-02-10T04:27:30.657900+0000","last_became_active":"2021-02-10T04:25:49.001782+0000","last_became_peered":"2021-02-10T04:25:49.001782+0000","last_unstale":"2021-02-10T04:27:30.657900+0000","last_undegraded":"2021-02-10T04:27:30.657900+0000","last_fullsized":"2021-02-10T04:27:30.657900+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:30.657781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:30.657781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"45","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748149+0000","last_change":"2021-02-10T04:27:08.192835+0000","last_active":"2021-02-10T04:27:30.748149+0000","last_peered":"2021-02-10T04:27:30.748149+0000","last_clean":"2021-02-10T04:27:30.748149+0000","last_became_active":"2021-02-10T04:25:36.340009+0000","last_became_peered":"2021-02-10T04:25:36.340009+0000","last_unstale":"2021-02-10T04:27:30.748149+0000","last_undegraded":"2021-02-10T04:27:30.748149+0000","last_fullsized":"2021-02-10T04:27:30.748149+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:08.192772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:08.192772+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"62'16","reported_seq":"66","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.057433+0000","last_change":"2021-02-10T04:27:09.484054+0000","last_active":"2021-02-10T04:27:30.057433+0000","last_peered":"2021-02-10T04:27:30.057433+0000","last_clean":"2021-02-10T04:27:30.057433+0000","last_became_active":"2021-02-10T04:25:49.003521+0000","last_became_peered":"2021-02-10T04:25:49.003521+0000","last_unstale":"2021-02-10T04:27:30.057433+0000","last_undegraded":"2021-02-10T04:27:30.057433+0000","last_fullsized":"2021-02-10T04:27:30.057433+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'16","last_scrub_stamp":"2021-02-10T04:27:09.483988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:09.483988+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":16,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"38","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748804+0000","last_change":"2021-02-10T04:27:25.216335+0000","last_active":"2021-02-10T04:27:30.748804+0000","last_peered":"2021-02-10T04:27:30.748804+0000","last_clean":"2021-02-10T04:27:30.748804+0000","last_became_active":"2021-02-10T04:25:51.032201+0000","last_became_peered":"2021-02-10T04:25:51.032201+0000","last_unstale":"2021-02-10T04:27:30.748804+0000","last_undegraded":"2021-02-10T04:27:30.748804+0000","last_fullsized":"2021-02-10T04:27:30.748804+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:27:25.216260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"31","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.747768+0000","last_change":"2021-02-10T04:25:53.150146+0000","last_active":"2021-02-10T04:27:30.747768+0000","last_peered":"2021-02-10T04:27:30.747768+0000","last_clean":"2021-02-10T04:27:30.747768+0000","last_became_active":"2021-02-10T04:25:53.149552+0000","last_became_peered":"2021-02-10T04:25:53.149552+0000","last_unstale":"2021-02-10T04:27:30.747768+0000","last_undegraded":"2021-02-10T04:27:30.747768+0000","last_fullsized":"2021-02-10T04:27:30.747768+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:25:52.015519+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"48","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.749704+0000","last_change":"2021-02-10T04:26:57.084942+0000","last_active":"2021-02-10T04:27:30.749704+0000","last_peered":"2021-02-10T04:27:30.749704+0000","last_clean":"2021-02-10T04:27:30.749704+0000","last_became_active":"2021-02-10T04:25:36.338600+0000","last_became_peered":"2021-02-10T04:25:36.338600+0000","last_unstale":"2021-02-10T04:27:30.749704+0000","last_undegraded":"2021-02-10T04:27:30.749704+0000","last_fullsized":"2021-02-10T04:27:30.749704+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:26:57.084864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:57.084864+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"63'27","reported_seq":"95","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.057422+0000","last_change":"2021-02-10T04:27:10.457048+0000","last_active":"2021-02-10T04:27:30.057422+0000","last_peered":"2021-02-10T04:27:30.057422+0000","last_clean":"2021-02-10T04:27:30.057422+0000","last_became_active":"2021-02-10T04:25:49.002937+0000","last_became_peered":"2021-02-10T04:25:49.002937+0000","last_unstale":"2021-02-10T04:27:30.057422+0000","last_undegraded":"2021-02-10T04:27:30.057422+0000","last_fullsized":"2021-02-10T04:27:30.057422+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'27","last_scrub_stamp":"2021-02-10T04:27:10.456981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:10.456981+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":38,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"38","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.272269+0000","last_change":"2021-02-10T04:26:59.175043+0000","last_active":"2021-02-10T04:27:30.272269+0000","last_peered":"2021-02-10T04:27:30.272269+0000","last_clean":"2021-02-10T04:27:30.272269+0000","last_became_active":"2021-02-10T04:25:51.025636+0000","last_became_peered":"2021-02-10T04:25:51.025636+0000","last_unstale":"2021-02-10T04:27:30.272269+0000","last_undegraded":"2021-02-10T04:27:30.272269+0000","last_fullsized":"2021-02-10T04:27:30.272269+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:59.174908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:26:59.174908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"37","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.078388+0000","last_change":"2021-02-10T04:27:01.184729+0000","last_active":"2021-02-10T04:27:36.078388+0000","last_peered":"2021-02-10T04:27:36.078388+0000","last_clean":"2021-02-10T04:27:36.078388+0000","last_became_active":"2021-02-10T04:25:53.045085+0000","last_became_peered":"2021-02-10T04:25:53.045085+0000","last_unstale":"2021-02-10T04:27:36.078388+0000","last_undegraded":"2021-02-10T04:27:36.078388+0000","last_fullsized":"2021-02-10T04:27:36.078388+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:01.184630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:01.184630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"51","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748471+0000","last_change":"2021-02-10T04:27:25.026606+0000","last_active":"2021-02-10T04:27:30.748471+0000","last_peered":"2021-02-10T04:27:30.748471+0000","last_clean":"2021-02-10T04:27:30.748471+0000","last_became_active":"2021-02-10T04:25:35.324960+0000","last_became_peered":"2021-02-10T04:25:35.324960+0000","last_unstale":"2021-02-10T04:27:30.748471+0000","last_undegraded":"2021-02-10T04:27:30.748471+0000","last_fullsized":"2021-02-10T04:27:30.748471+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:27:25.026534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:25.026534+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"63'34","reported_seq":"109","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.055150+0000","last_change":"2021-02-10T04:27:20.342445+0000","last_active":"2021-02-10T04:27:30.055150+0000","last_peered":"2021-02-10T04:27:30.055150+0000","last_clean":"2021-02-10T04:27:30.055150+0000","last_became_active":"2021-02-10T04:25:48.997481+0000","last_became_peered":"2021-02-10T04:25:48.997481+0000","last_unstale":"2021-02-10T04:27:30.055150+0000","last_undegraded":"2021-02-10T04:27:30.055150+0000","last_fullsized":"2021-02-10T04:27:30.055150+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'34","last_scrub_stamp":"2021-02-10T04:27:20.342358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:20.342358+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":70,"num_read_kb":48,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"34","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.078114+0000","last_change":"2021-02-10T04:25:51.019067+0000","last_active":"2021-02-10T04:27:36.078114+0000","last_peered":"2021-02-10T04:27:36.078114+0000","last_clean":"2021-02-10T04:27:36.078114+0000","last_became_active":"2021-02-10T04:25:51.018706+0000","last_became_peered":"2021-02-10T04:25:51.018706+0000","last_unstale":"2021-02-10T04:27:36.078114+0000","last_undegraded":"2021-02-10T04:27:36.078114+0000","last_fullsized":"2021-02-10T04:27:36.078114+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:25:49.999907+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"36","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748932+0000","last_change":"2021-02-10T04:27:26.234432+0000","last_active":"2021-02-10T04:27:30.748932+0000","last_peered":"2021-02-10T04:27:30.748932+0000","last_clean":"2021-02-10T04:27:30.748932+0000","last_became_active":"2021-02-10T04:25:53.150930+0000","last_became_peered":"2021-02-10T04:25:53.150930+0000","last_unstale":"2021-02-10T04:27:30.748932+0000","last_undegraded":"2021-02-10T04:27:30.748932+0000","last_fullsized":"2021-02-10T04:27:30.748932+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:26.234349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:26.234349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"56","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.272767+0000","last_change":"2021-02-10T04:26:44.264500+0000","last_active":"2021-02-10T04:27:30.272767+0000","last_peered":"2021-02-10T04:27:30.272767+0000","last_clean":"2021-02-10T04:27:30.272767+0000","last_became_active":"2021-02-10T04:25:36.339614+0000","last_became_peered":"2021-02-10T04:25:36.339614+0000","last_unstale":"2021-02-10T04:27:30.272767+0000","last_undegraded":"2021-02-10T04:27:30.272767+0000","last_fullsized":"2021-02-10T04:27:30.272767+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:44.264402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:44.264402+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"63'26","reported_seq":"89","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.077934+0000","last_change":"2021-02-10T04:27:26.403830+0000","last_active":"2021-02-10T04:27:36.077934+0000","last_peered":"2021-02-10T04:27:36.077934+0000","last_clean":"2021-02-10T04:27:36.077934+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:27:36.077934+0000","last_undegraded":"2021-02-10T04:27:36.077934+0000","last_fullsized":"2021-02-10T04:27:36.077934+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:27:26.403748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:26.403748+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":51,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"58'2","reported_seq":"61","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.078210+0000","last_change":"2021-02-10T04:27:33.210495+0000","last_active":"2021-02-10T04:27:36.078210+0000","last_peered":"2021-02-10T04:27:36.078210+0000","last_clean":"2021-02-10T04:27:36.078210+0000","last_became_active":"2021-02-10T04:25:51.031906+0000","last_became_peered":"2021-02-10T04:25:51.031906+0000","last_unstale":"2021-02-10T04:27:36.078210+0000","last_undegraded":"2021-02-10T04:27:36.078210+0000","last_fullsized":"2021-02-10T04:27:36.078210+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:33.210431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:33.210431+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"30","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.055551+0000","last_change":"2021-02-10T04:25:53.148449+0000","last_active":"2021-02-10T04:27:30.055551+0000","last_peered":"2021-02-10T04:27:30.055551+0000","last_clean":"2021-02-10T04:27:30.055551+0000","last_became_active":"2021-02-10T04:25:53.148054+0000","last_became_peered":"2021-02-10T04:25:53.148054+0000","last_unstale":"2021-02-10T04:27:30.055551+0000","last_undegraded":"2021-02-10T04:27:30.055551+0000","last_fullsized":"2021-02-10T04:27:30.055551+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:25:52.015519+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"38","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748716+0000","last_change":"2021-02-10T04:26:58.112766+0000","last_active":"2021-02-10T04:27:30.748716+0000","last_peered":"2021-02-10T04:27:30.748716+0000","last_clean":"2021-02-10T04:27:30.748716+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:27:30.748716+0000","last_undegraded":"2021-02-10T04:27:30.748716+0000","last_fullsized":"2021-02-10T04:27:30.748716+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:58.112683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:26:58.112683+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"45","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748100+0000","last_change":"2021-02-10T04:27:04.148397+0000","last_active":"2021-02-10T04:27:30.748100+0000","last_peered":"2021-02-10T04:27:30.748100+0000","last_clean":"2021-02-10T04:27:30.748100+0000","last_became_active":"2021-02-10T04:25:36.340895+0000","last_became_peered":"2021-02-10T04:25:36.340895+0000","last_unstale":"2021-02-10T04:27:30.748100+0000","last_undegraded":"2021-02-10T04:27:30.748100+0000","last_fullsized":"2021-02-10T04:27:30.748100+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:04.148329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:04.148329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"63'21","reported_seq":"83","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.078777+0000","last_change":"2021-02-10T04:26:59.205120+0000","last_active":"2021-02-10T04:27:36.078777+0000","last_peered":"2021-02-10T04:27:36.078777+0000","last_clean":"2021-02-10T04:27:36.078777+0000","last_became_active":"2021-02-10T04:25:49.011093+0000","last_became_peered":"2021-02-10T04:25:49.011093+0000","last_unstale":"2021-02-10T04:27:36.078777+0000","last_undegraded":"2021-02-10T04:27:36.078777+0000","last_fullsized":"2021-02-10T04:27:36.078777+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2021-02-10T04:26:59.205037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:26:59.205037+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":28,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"36","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.749370+0000","last_change":"2021-02-10T04:27:24.209544+0000","last_active":"2021-02-10T04:27:30.749370+0000","last_peered":"2021-02-10T04:27:30.749370+0000","last_clean":"2021-02-10T04:27:30.749370+0000","last_became_active":"2021-02-10T04:25:53.042697+0000","last_became_peered":"2021-02-10T04:25:53.042697+0000","last_unstale":"2021-02-10T04:27:30.749370+0000","last_undegraded":"2021-02-10T04:27:30.749370+0000","last_fullsized":"2021-02-10T04:27:30.749370+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:27:24.209460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"78'3","reported_seq":"64","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:37.723400+0000","last_change":"2021-02-10T04:27:13.268467+0000","last_active":"2021-02-10T04:27:37.723400+0000","last_peered":"2021-02-10T04:27:37.723400+0000","last_clean":"2021-02-10T04:27:37.723400+0000","last_became_active":"2021-02-10T04:25:51.025218+0000","last_became_peered":"2021-02-10T04:25:51.025218+0000","last_unstale":"2021-02-10T04:27:37.723400+0000","last_undegraded":"2021-02-10T04:27:37.723400+0000","last_fullsized":"2021-02-10T04:27:37.723400+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:27:13.268412+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"51","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.747932+0000","last_change":"2021-02-10T04:26:46.092082+0000","last_active":"2021-02-10T04:27:30.747932+0000","last_peered":"2021-02-10T04:27:30.747932+0000","last_clean":"2021-02-10T04:27:30.747932+0000","last_became_active":"2021-02-10T04:25:36.331437+0000","last_became_peered":"2021-02-10T04:25:36.331437+0000","last_unstale":"2021-02-10T04:27:30.747932+0000","last_undegraded":"2021-02-10T04:27:30.747932+0000","last_fullsized":"2021-02-10T04:27:30.747932+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:46.092013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:46.092013+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"62'17","reported_seq":"69","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748475+0000","last_change":"2021-02-10T04:27:30.012988+0000","last_active":"2021-02-10T04:27:30.748475+0000","last_peered":"2021-02-10T04:27:30.748475+0000","last_clean":"2021-02-10T04:27:30.748475+0000","last_became_active":"2021-02-10T04:25:49.004933+0000","last_became_peered":"2021-02-10T04:25:49.004933+0000","last_unstale":"2021-02-10T04:27:30.748475+0000","last_undegraded":"2021-02-10T04:27:30.748475+0000","last_fullsized":"2021-02-10T04:27:30.748475+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'17","last_scrub_stamp":"2021-02-10T04:27:30.012927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:30.012927+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"37","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.078569+0000","last_change":"2021-02-10T04:27:33.382403+0000","last_active":"2021-02-10T04:27:36.078569+0000","last_peered":"2021-02-10T04:27:36.078569+0000","last_clean":"2021-02-10T04:27:36.078569+0000","last_became_active":"2021-02-10T04:25:53.043027+0000","last_became_peered":"2021-02-10T04:25:53.043027+0000","last_unstale":"2021-02-10T04:27:36.078569+0000","last_undegraded":"2021-02-10T04:27:36.078569+0000","last_fullsized":"2021-02-10T04:27:36.078569+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:33.382319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:33.382319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"58'2","reported_seq":"61","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.077333+0000","last_change":"2021-02-10T04:26:50.624141+0000","last_active":"2021-02-10T04:27:36.077333+0000","last_peered":"2021-02-10T04:27:36.077333+0000","last_clean":"2021-02-10T04:27:36.077333+0000","last_became_active":"2021-02-10T04:25:51.025268+0000","last_became_peered":"2021-02-10T04:25:51.025268+0000","last_unstale":"2021-02-10T04:27:36.077333+0000","last_undegraded":"2021-02-10T04:27:36.077333+0000","last_fullsized":"2021-02-10T04:27:36.077333+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:26:50.624072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:26:50.624072+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"49","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.747908+0000","last_change":"2021-02-10T04:27:17.003396+0000","last_active":"2021-02-10T04:27:30.747908+0000","last_peered":"2021-02-10T04:27:30.747908+0000","last_clean":"2021-02-10T04:27:30.747908+0000","last_became_active":"2021-02-10T04:25:36.333883+0000","last_became_peered":"2021-02-10T04:25:36.333883+0000","last_unstale":"2021-02-10T04:27:30.747908+0000","last_undegraded":"2021-02-10T04:27:30.747908+0000","last_fullsized":"2021-02-10T04:27:30.747908+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:17.003304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:17.003304+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"62'13","reported_seq":"62","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.078308+0000","last_change":"2021-02-10T04:27:14.256603+0000","last_active":"2021-02-10T04:27:36.078308+0000","last_peered":"2021-02-10T04:27:36.078308+0000","last_clean":"2021-02-10T04:27:36.078308+0000","last_became_active":"2021-02-10T04:25:49.001725+0000","last_became_peered":"2021-02-10T04:25:49.001725+0000","last_unstale":"2021-02-10T04:27:36.078308+0000","last_undegraded":"2021-02-10T04:27:36.078308+0000","last_fullsized":"2021-02-10T04:27:36.078308+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'13","last_scrub_stamp":"2021-02-10T04:27:14.256539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:14.256539+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"35","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.164810+0000","last_change":"2021-02-10T04:26:55.627032+0000","last_active":"2021-02-10T04:27:30.164810+0000","last_peered":"2021-02-10T04:27:30.164810+0000","last_clean":"2021-02-10T04:27:30.164810+0000","last_became_active":"2021-02-10T04:25:53.038218+0000","last_became_peered":"2021-02-10T04:25:53.038218+0000","last_unstale":"2021-02-10T04:27:30.164810+0000","last_undegraded":"2021-02-10T04:27:30.164810+0000","last_fullsized":"2021-02-10T04:27:30.164810+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:55.626967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:26:55.626967+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"38","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:37.160395+0000","last_change":"2021-02-10T04:27:37.160395+0000","last_active":"2021-02-10T04:27:37.160395+0000","last_peered":"2021-02-10T04:27:37.160395+0000","last_clean":"2021-02-10T04:27:37.160395+0000","last_became_active":"2021-02-10T04:25:51.028302+0000","last_became_peered":"2021-02-10T04:25:51.028302+0000","last_unstale":"2021-02-10T04:27:37.160395+0000","last_undegraded":"2021-02-10T04:27:37.160395+0000","last_fullsized":"2021-02-10T04:27:37.160395+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:37.160328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:37.160328+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"45","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.749054+0000","last_change":"2021-02-10T04:27:00.052115+0000","last_active":"2021-02-10T04:27:30.749054+0000","last_peered":"2021-02-10T04:27:30.749054+0000","last_clean":"2021-02-10T04:27:30.749054+0000","last_became_active":"2021-02-10T04:25:36.344109+0000","last_became_peered":"2021-02-10T04:25:36.344109+0000","last_unstale":"2021-02-10T04:27:30.749054+0000","last_undegraded":"2021-02-10T04:27:30.749054+0000","last_fullsized":"2021-02-10T04:27:30.749054+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:00.052028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:00.052028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"62'21","reported_seq":"77","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.078564+0000","last_change":"2021-02-10T04:27:00.215115+0000","last_active":"2021-02-10T04:27:36.078564+0000","last_peered":"2021-02-10T04:27:36.078564+0000","last_clean":"2021-02-10T04:27:36.078564+0000","last_became_active":"2021-02-10T04:25:49.011699+0000","last_became_peered":"2021-02-10T04:25:49.011699+0000","last_unstale":"2021-02-10T04:27:36.078564+0000","last_undegraded":"2021-02-10T04:27:36.078564+0000","last_fullsized":"2021-02-10T04:27:36.078564+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'21","last_scrub_stamp":"2021-02-10T04:27:00.215037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:00.215037+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"36","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:35.166388+0000","last_change":"2021-02-10T04:27:35.166330+0000","last_active":"2021-02-10T04:27:35.166388+0000","last_peered":"2021-02-10T04:27:35.166388+0000","last_clean":"2021-02-10T04:27:35.166388+0000","last_became_active":"2021-02-10T04:25:53.037903+0000","last_became_peered":"2021-02-10T04:25:53.037903+0000","last_unstale":"2021-02-10T04:27:35.166388+0000","last_undegraded":"2021-02-10T04:27:35.166388+0000","last_fullsized":"2021-02-10T04:27:35.166388+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:35.166250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:35.166250+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"37","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.164601+0000","last_change":"2021-02-10T04:27:11.606664+0000","last_active":"2021-02-10T04:27:30.164601+0000","last_peered":"2021-02-10T04:27:30.164601+0000","last_clean":"2021-02-10T04:27:30.164601+0000","last_became_active":"2021-02-10T04:25:51.025882+0000","last_became_peered":"2021-02-10T04:25:51.025882+0000","last_unstale":"2021-02-10T04:27:30.164601+0000","last_undegraded":"2021-02-10T04:27:30.164601+0000","last_fullsized":"2021-02-10T04:27:30.164601+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:11.606603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:11.606603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"62'2","reported_seq":"69","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.077782+0000","last_change":"2021-02-10T04:26:48.599352+0000","last_active":"2021-02-10T04:27:36.077782+0000","last_peered":"2021-02-10T04:27:36.077782+0000","last_clean":"2021-02-10T04:27:36.077782+0000","last_became_active":"2021-02-10T04:25:34.331199+0000","last_became_peered":"2021-02-10T04:25:34.331199+0000","last_unstale":"2021-02-10T04:27:36.077782+0000","last_undegraded":"2021-02-10T04:27:36.077782+0000","last_fullsized":"2021-02-10T04:27:36.077782+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2021-02-10T04:26:48.599257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:48.599257+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"63'29","reported_seq":"96","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.272144+0000","last_change":"2021-02-10T04:27:13.081465+0000","last_active":"2021-02-10T04:27:30.272144+0000","last_peered":"2021-02-10T04:27:30.272144+0000","last_clean":"2021-02-10T04:27:30.272144+0000","last_became_active":"2021-02-10T04:25:49.015913+0000","last_became_peered":"2021-02-10T04:25:49.015913+0000","last_unstale":"2021-02-10T04:27:30.272144+0000","last_undegraded":"2021-02-10T04:27:30.272144+0000","last_fullsized":"2021-02-10T04:27:30.272144+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'29","last_scrub_stamp":"2021-02-10T04:27:13.081399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:13.081399+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":56,"num_read_kb":36,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"36","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.057020+0000","last_change":"2021-02-10T04:27:22.512316+0000","last_active":"2021-02-10T04:27:30.057020+0000","last_peered":"2021-02-10T04:27:30.057020+0000","last_clean":"2021-02-10T04:27:30.057020+0000","last_became_active":"2021-02-10T04:25:53.038249+0000","last_became_peered":"2021-02-10T04:25:53.038249+0000","last_unstale":"2021-02-10T04:27:30.057020+0000","last_undegraded":"2021-02-10T04:27:30.057020+0000","last_fullsized":"2021-02-10T04:27:30.057020+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:22.512245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:22.512245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"37","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.164529+0000","last_change":"2021-02-10T04:26:51.605291+0000","last_active":"2021-02-10T04:27:30.164529+0000","last_peered":"2021-02-10T04:27:30.164529+0000","last_clean":"2021-02-10T04:27:30.164529+0000","last_became_active":"2021-02-10T04:25:51.016118+0000","last_became_peered":"2021-02-10T04:25:51.016118+0000","last_unstale":"2021-02-10T04:27:30.164529+0000","last_undegraded":"2021-02-10T04:27:30.164529+0000","last_fullsized":"2021-02-10T04:27:30.164529+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:51.605227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:26:51.605227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"47","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748305+0000","last_change":"2021-02-10T04:26:50.001018+0000","last_active":"2021-02-10T04:27:30.748305+0000","last_peered":"2021-02-10T04:27:30.748305+0000","last_clean":"2021-02-10T04:27:30.748305+0000","last_became_active":"2021-02-10T04:25:34.334427+0000","last_became_peered":"2021-02-10T04:25:34.334427+0000","last_unstale":"2021-02-10T04:27:30.748305+0000","last_undegraded":"2021-02-10T04:27:30.748305+0000","last_fullsized":"2021-02-10T04:27:30.748305+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:50.000952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:50.000952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"63'20","reported_seq":"81","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.057517+0000","last_change":"2021-02-10T04:27:20.587671+0000","last_active":"2021-02-10T04:27:30.057517+0000","last_peered":"2021-02-10T04:27:30.057517+0000","last_clean":"2021-02-10T04:27:30.057517+0000","last_became_active":"2021-02-10T04:25:49.004346+0000","last_became_peered":"2021-02-10T04:25:49.004346+0000","last_unstale":"2021-02-10T04:27:30.057517+0000","last_undegraded":"2021-02-10T04:27:30.057517+0000","last_fullsized":"2021-02-10T04:27:30.057517+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'20","last_scrub_stamp":"2021-02-10T04:27:20.587602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:20.587602+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":27,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"36","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748520+0000","last_change":"2021-02-10T04:27:13.006848+0000","last_active":"2021-02-10T04:27:30.748520+0000","last_peered":"2021-02-10T04:27:30.748520+0000","last_clean":"2021-02-10T04:27:30.748520+0000","last_became_active":"2021-02-10T04:25:53.148090+0000","last_became_peered":"2021-02-10T04:25:53.148090+0000","last_unstale":"2021-02-10T04:27:30.748520+0000","last_undegraded":"2021-02-10T04:27:30.748520+0000","last_fullsized":"2021-02-10T04:27:30.748520+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:27:13.006716+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"60","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.164839+0000","last_change":"2021-02-10T04:26:44.609990+0000","last_active":"2021-02-10T04:27:30.164839+0000","last_peered":"2021-02-10T04:27:30.164839+0000","last_clean":"2021-02-10T04:27:30.164839+0000","last_became_active":"2021-02-10T04:25:36.344641+0000","last_became_peered":"2021-02-10T04:25:36.344641+0000","last_unstale":"2021-02-10T04:27:30.164839+0000","last_undegraded":"2021-02-10T04:27:30.164839+0000","last_fullsized":"2021-02-10T04:27:30.164839+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:26:44.609922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:44.609922+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"63'30","reported_seq":"101","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748200+0000","last_change":"2021-02-10T04:27:02.039902+0000","last_active":"2021-02-10T04:27:30.748200+0000","last_peered":"2021-02-10T04:27:30.748200+0000","last_clean":"2021-02-10T04:27:30.748200+0000","last_became_active":"2021-02-10T04:25:49.002058+0000","last_became_peered":"2021-02-10T04:25:49.002058+0000","last_unstale":"2021-02-10T04:27:30.748200+0000","last_undegraded":"2021-02-10T04:27:30.748200+0000","last_fullsized":"2021-02-10T04:27:30.748200+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2021-02-10T04:27:02.039813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:02.039813+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":59,"num_read_kb":42,"num_write":31,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"58'2","reported_seq":"62","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.378543+0000","last_change":"2021-02-10T04:27:20.144947+0000","last_active":"2021-02-10T04:27:36.378543+0000","last_peered":"2021-02-10T04:27:36.378543+0000","last_clean":"2021-02-10T04:27:36.378543+0000","last_became_active":"2021-02-10T04:25:51.019515+0000","last_became_peered":"2021-02-10T04:25:51.019515+0000","last_unstale":"2021-02-10T04:27:36.378543+0000","last_undegraded":"2021-02-10T04:27:36.378543+0000","last_fullsized":"2021-02-10T04:27:36.378543+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:20.144855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:20.144855+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":1453,"num_read_kb":926,"num_write":734,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":738,"ondisk_log_size":738,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":106,"num_read_kb":101,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1945,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":21,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":5835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748364827,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6540,"kb_used_data":836,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738612,"statfs":{"total":95995035648,"available":95988338688,"internally_reserved":0,"allocated":856064,"data_stored":300940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.876,"15min":0.88200000000000001},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":1.0529999999999999},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.85599999999999998,"15min":0.85899999999999999},"min":{"1min":0.57499999999999996,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.248,"5min":1.248,"15min":1.248},"last":1.1080000000000001}]},{"osd":1,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.88700000000000001,"15min":0.89100000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":1.139},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.0149999999999999,"15min":1.0209999999999999},"min":{"1min":0.64500000000000002,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.355,"5min":1.369,"15min":1.369},"last":1.173}]},{"osd":2,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.018,"15min":1.026},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":1.292},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.95799999999999996,"15min":0.95999999999999996},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.95699999999999996}]},{"osd":3,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.998,"15min":1.01},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.147,"15min":1.153},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.381,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":1.3779999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":0.97499999999999998,"15min":0.95099999999999996},"min":{"1min":0.77300000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.2530000000000001,"5min":2.2530000000000001,"15min":2.2530000000000001},"last":1.427},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.1060000000000001,"15min":1.1120000000000001},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.492,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":0.876}]},{"osd":5,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.161,"15min":1.1499999999999999},"min":{"1min":0.68899999999999995,"5min":0.624,"15min":0.624},"max":{"1min":1.714,"5min":1.714,"15min":1.714},"last":0.92600000000000005},{"interface":"front","average":{"1min":1.208,"5min":1.2,"15min":1.1990000000000001},"min":{"1min":0.90900000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.498,"5min":1.538,"15min":1.538},"last":1.2410000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.246,"15min":1.2430000000000001},"min":{"1min":0.94199999999999995,"5min":0.86899999999999999,"15min":0.86899999999999999},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.276,"5min":1.236,"15min":1.2290000000000001},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":1.268}]}]},{"osd":6,"up_from":43,"seq":184683593759,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6584,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738568,"statfs":{"total":95995035648,"available":95988293632,"internally_reserved":0,"allocated":901120,"data_stored":303810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:18 2021","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.3460000000000001,"5min":1.478,"15min":1.478},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.95499999999999996,"5min":1.008,"15min":1.0169999999999999},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.77000000000000002}]},{"osd":1,"last update":"Wed Feb 10 04:27:18 2021","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":1.0149999999999999,"15min":1.03},"min":{"1min":0.61599999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":2.121,"5min":2.121,"15min":2.121},"last":0.71799999999999997},{"interface":"front","average":{"1min":1.042,"5min":0.99099999999999999,"15min":0.98299999999999998},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":1.0409999999999999}]},{"osd":2,"last update":"Wed Feb 10 04:27:18 2021","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0620000000000001,"15min":1.0609999999999999},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.0289999999999999,"15min":1.014},"min":{"1min":0.73699999999999999,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.1920000000000002,"5min":2.1920000000000002,"15min":2.1920000000000002},"last":0.93300000000000005}]},{"osd":3,"last update":"Wed Feb 10 04:27:18 2021","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.1439999999999999,"15min":1.1379999999999999},"min":{"1min":0.85699999999999998,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":2,"5min":2,"15min":2},"last":0.91000000000000003},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.133,"15min":1.107},"min":{"1min":0.88,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":2.323,"5min":2.323,"15min":2.323},"last":1.0720000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:27:18 2021","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":0.99099999999999999,"15min":0.96099999999999997},"min":{"1min":0.61699999999999999,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.0800000000000001,"5min":2.0800000000000001,"15min":2.0800000000000001},"last":1.1830000000000001},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.90500000000000003,"15min":0.89200000000000002},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.997}]},{"osd":5,"last update":"Wed Feb 10 04:27:18 2021","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.0009999999999999,"15min":0.98699999999999999},"min":{"1min":0.67100000000000004,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.7130000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.419,"5min":1.179,"15min":1.139},"min":{"1min":0.83999999999999997,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.4550000000000001,"5min":2.4550000000000001,"15min":2.4550000000000001},"last":1.2170000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:26:37 2021","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.105},{"interface":"front","average":{"1min":1.385,"5min":1.385,"15min":1.385},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":1.925,"5min":1.925,"15min":1.925},"last":1.3}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":11216,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":10240,"kb_avail":93733936,"statfs":{"total":95995035648,"available":95983550464,"internally_reserved":0,"allocated":991232,"data_stored":306890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":10485760},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:26:53 2021","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.66600000000000004,"15min":0.626},"min":{"1min":0.56899999999999995,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.425,"5min":1.425,"15min":1.425},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.746,"15min":0.69199999999999995},"min":{"1min":0.57299999999999995,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.621}]},{"osd":2,"last update":"Wed Feb 10 04:27:14 2021","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.80800000000000005,"15min":0.76400000000000001},"min":{"1min":0.58199999999999996,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.83499999999999996,"15min":0.79900000000000004},"min":{"1min":0.51000000000000001,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":1.2310000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:27:31 2021","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.93100000000000005,"15min":0.91400000000000003},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.5860000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.026,"5min":1.018,"15min":1.014},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.226,"5min":2.226,"15min":2.226},"last":1.1060000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:26:45 2021","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.1599999999999999,"15min":1.147},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.004,"5min":2.004,"15min":2.004},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.2130000000000001,"15min":1.2090000000000001},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":2.2919999999999998,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":1.5069999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:27:08 2021","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.25,"15min":1.2769999999999999},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":2.7170000000000001,"5min":5.5860000000000003,"15min":5.5860000000000003},"last":1.024},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.6100000000000001,"15min":1.6559999999999999},"min":{"1min":0.97399999999999998,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":2.105,"5min":9.5310000000000006,"15min":9.5310000000000006},"last":1.736}]},{"osd":6,"last update":"Wed Feb 10 04:27:24 2021","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.2889999999999999,"15min":1.29},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.4449999999999998,"5min":2.4449999999999998,"15min":2.4449999999999998},"last":1.2609999999999999},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.264,"15min":1.292},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.905,"5min":3.653,"15min":3.653},"last":1.0880000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:27:36 2021","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.639,"15min":1.6890000000000001},"min":{"1min":1.0529999999999999,"5min":1.004,"15min":1.004},"max":{"1min":2.2090000000000001,"5min":7.5540000000000003,"15min":7.5540000000000003},"last":1.903},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.623,"15min":1.6619999999999999},"min":{"1min":1.006,"5min":1.006,"15min":1.006},"max":{"1min":2.556,"5min":8.1020000000000003,"15min":8.1020000000000003},"last":1.3939999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738422,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6784,"kb_used_data":824,"kb_used_omap":0,"kb_used_meta":5952,"kb_avail":93738368,"statfs":{"total":95995035648,"available":95988088832,"internally_reserved":0,"allocated":843776,"data_stored":300649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6094848},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 04:26:53 2021","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.64300000000000002,"15min":0.58699999999999997},"min":{"1min":0.53100000000000003,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.006,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.79100000000000004,"15min":0.73299999999999998},"min":{"1min":0.67400000000000004,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":1.0740000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:27:12 2021","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.79300000000000004,"15min":0.77300000000000002},"min":{"1min":0.52500000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.958,"5min":1.958,"15min":1.958},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.79700000000000004,"15min":0.79300000000000004},"min":{"1min":0.56100000000000005,"5min":0.495,"15min":0.495},"max":{"1min":1.4390000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.97799999999999998}]},{"osd":3,"last update":"Wed Feb 10 04:27:27 2021","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.80400000000000005,"15min":0.81200000000000006},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.224,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.89500000000000002,"5min":1.026,"15min":1.036},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.3049999999999999,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.59099999999999997}]},{"osd":4,"last update":"Wed Feb 10 04:26:44 2021","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.1659999999999999,"15min":1.1819999999999999},"min":{"1min":0.82699999999999996,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.3879999999999999,"5min":1.613,"15min":1.613},"last":0.85499999999999998},{"interface":"front","average":{"1min":1.137,"5min":1.1559999999999999,"15min":1.159},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.597,"5min":1.597,"15min":1.597},"last":0.76600000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:27:01 2021","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.2,"15min":1.2190000000000001},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.874},{"interface":"front","average":{"1min":1.218,"5min":1.2889999999999999,"15min":1.3009999999999999},"min":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"max":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":0.89600000000000002}]},{"osd":6,"last update":"Wed Feb 10 04:27:20 2021","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.2909999999999999,"15min":1.3029999999999999},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":1.601,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.351,"15min":1.363},"min":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":2.1019999999999999,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":1.0129999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.2330000000000001,"15min":1.252},"min":{"1min":0.94599999999999995,"5min":0.93500000000000005,"15min":0.93500000000000005},"max":{"1min":1.5900000000000001,"5min":1.76,"15min":1.76},"last":1.421},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3500000000000001,"15min":1.3640000000000001},"min":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"max":{"1min":1.611,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":1.036}]}]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6528,"kb_used_data":760,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738624,"statfs":{"total":95995035648,"available":95988350976,"internally_reserved":0,"allocated":778240,"data_stored":299008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:10 2021","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.70899999999999996,"15min":0.68300000000000005},"min":{"1min":0.54400000000000004,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.97999999999999998,"5min":1.349,"15min":1.349},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.73299999999999998,"15min":0.71099999999999997},"min":{"1min":0.59499999999999997,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.96399999999999997,"5min":1.274,"15min":1.274},"last":0.53900000000000003}]},{"osd":1,"last update":"Wed Feb 10 04:27:10 2021","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.81200000000000006,"15min":0.76900000000000002},"min":{"1min":0.57499999999999996,"5min":0.495,"15min":0.495},"max":{"1min":1.3069999999999999,"5min":1.726,"15min":1.726},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.71299999999999997,"15min":0.69099999999999995},"min":{"1min":0.54900000000000004,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.51200000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:27:31 2021","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.81499999999999995,"15min":0.80400000000000005},"min":{"1min":0.57999999999999996,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.90700000000000003,"5min":1.274,"15min":1.274},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.996,"15min":0.97099999999999997},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.2,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":0.96899999999999997}]},{"osd":4,"last update":"Wed Feb 10 04:26:43 2021","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.1990000000000001,"15min":1.1870000000000001},"min":{"1min":0.97799999999999998,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":1.5860000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":1.4359999999999999},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.1160000000000001,"15min":1.121},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.516,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":1.4630000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:26:59 2021","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.3180000000000001,"15min":1.3169999999999999},"min":{"1min":1.014,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":1.71,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.1879999999999999,"15min":1.1819999999999999},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":1.083}]},{"osd":6,"last update":"Wed Feb 10 04:27:23 2021","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.284,"15min":1.2989999999999999},"min":{"1min":0.875,"5min":0.875,"15min":0.875},"max":{"1min":1.504,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.26,"5min":1.409,"15min":1.4339999999999999},"min":{"1min":1.05,"5min":1.0489999999999999,"15min":1.0489999999999999},"max":{"1min":1.635,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.216}]},{"osd":7,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.4690000000000001,"15min":1.494},"min":{"1min":1.077,"5min":1.073,"15min":1.073},"max":{"1min":1.653,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":1.276},{"interface":"front","average":{"1min":1.171,"5min":1.252,"15min":1.266},"min":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"max":{"1min":1.52,"5min":1.6160000000000001,"15min":1.6160000000000001},"last":1.3879999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182441,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6900,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":5952,"kb_avail":93738252,"statfs":{"total":95995035648,"available":95987970048,"internally_reserved":0,"allocated":962560,"data_stored":304990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6094848},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:28 2021","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.74299999999999999,"15min":0.73499999999999999},"min":{"1min":0.50900000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.0600000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.88300000000000001,"15min":0.86699999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":4.2759999999999998,"5min":4.2759999999999998,"15min":4.2759999999999998},"last":0.81599999999999995}]},{"osd":1,"last update":"Wed Feb 10 04:27:28 2021","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.90300000000000002,"15min":0.877},"min":{"1min":0.64200000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":3.246,"5min":3.246,"15min":3.246},"last":0.71099999999999997},{"interface":"front","average":{"1min":1.0549999999999999,"5min":0.89700000000000002,"15min":0.86199999999999999},"min":{"1min":0.57999999999999996,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":5.4039999999999999,"5min":5.4039999999999999,"15min":5.4039999999999999},"last":0.59999999999999998}]},{"osd":2,"last update":"Wed Feb 10 04:27:28 2021","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":0.97199999999999998,"15min":0.95099999999999996},"min":{"1min":0.60299999999999998,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":4.2309999999999999,"5min":4.2309999999999999,"15min":4.2309999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.96099999999999997,"15min":0.97599999999999998},"min":{"1min":0.65900000000000003,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.2609999999999999,"5min":1.681,"15min":1.681},"last":1.038}]},{"osd":4,"last update":"Wed Feb 10 04:26:43 2021","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.2290000000000001,"15min":1.1970000000000001},"min":{"1min":0.879,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":3.371,"5min":3.371,"15min":3.371},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.254,"15min":1.2210000000000001},"min":{"1min":1.0469999999999999,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":3.5430000000000001,"5min":3.5430000000000001,"15min":3.5430000000000001},"last":1.0629999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:27:00 2021","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.2350000000000001,"15min":1.1930000000000001},"min":{"1min":0.94799999999999995,"5min":0.88500000000000001,"15min":0.88500000000000001},"max":{"1min":3.496,"5min":3.496,"15min":3.496},"last":1.169},{"interface":"front","average":{"1min":1.466,"5min":1.3,"15min":1.272},"min":{"1min":1.143,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":3.742,"5min":3.742,"15min":3.742},"last":1.218}]},{"osd":6,"last update":"Wed Feb 10 04:27:24 2021","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.429,"15min":1.4059999999999999},"min":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"max":{"1min":3.6859999999999999,"5min":3.6859999999999999,"15min":3.6859999999999999},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.514,"5min":1.47,"15min":1.462},"min":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"max":{"1min":3.8759999999999999,"5min":3.8759999999999999,"15min":3.8759999999999999},"last":0.999}]},{"osd":7,"last update":"Wed Feb 10 04:26:36 2021","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"min":{"1min":1.232,"5min":1.232,"15min":1.232},"max":{"1min":1.7809999999999999,"5min":1.7809999999999999,"15min":1.7809999999999999},"last":1.087},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"min":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":1.1539999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986215,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6800,"kb_used_data":904,"kb_used_omap":0,"kb_used_meta":5888,"kb_avail":93738352,"statfs":{"total":95995035648,"available":95988072448,"internally_reserved":0,"allocated":925696,"data_stored":309939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6029312},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:26:42 2021","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.99399999999999999,"15min":1.006},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.1930000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":1.042},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.97199999999999998,"15min":0.97999999999999998},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.633,"5min":1.633,"15min":1.633},"last":0.622}]},{"osd":1,"last update":"Wed Feb 10 04:26:42 2021","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.90700000000000003,"15min":0.92100000000000004},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.1619999999999999,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.97699999999999998,"5min":1.0129999999999999,"15min":1.0189999999999999},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.351,"5min":1.508,"15min":1.508},"last":1.1879999999999999}]},{"osd":2,"last update":"Wed Feb 10 04:26:42 2021","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.075,"15min":1.081},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.238,"5min":1.1719999999999999,"15min":1.161},"min":{"1min":0.90500000000000003,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.768,"5min":1.768,"15min":1.768},"last":0.84299999999999997}]},{"osd":3,"last update":"Wed Feb 10 04:26:42 2021","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.0629999999999999,"15min":1.0489999999999999},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":0.97299999999999998},{"interface":"front","average":{"1min":1.052,"5min":1.0880000000000001,"15min":1.0940000000000001},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.999}]},{"osd":5,"last update":"Wed Feb 10 04:27:00 2021","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":0.88,"15min":0.83599999999999997},"min":{"1min":0.63200000000000001,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.05,"5min":0.86599999999999999,"15min":0.83599999999999997},"min":{"1min":0.74099999999999999,"5min":0.624,"15min":0.624},"max":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.5740000000000001},"last":0.871}]},{"osd":6,"last update":"Wed Feb 10 04:27:19 2021","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.0840000000000001,"15min":1.0640000000000001},"min":{"1min":0.79200000000000004,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.27,"15min":1.252},"min":{"1min":1.139,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.714,"5min":1.74,"15min":1.74},"last":1.0880000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:27:40 2021","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.94899999999999995,"5min":0.88600000000000001,"15min":0.88600000000000001},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":1.4219999999999999},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3720000000000001,"15min":1.3740000000000001},"min":{"1min":1.073,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":1.6699999999999999,"5min":1.9450000000000001,"15min":1.9450000000000001},"last":1.161}]}]},{"osd":5,"up_from":37,"seq":158913789987,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6512,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738640,"statfs":{"total":95995035648,"available":95988367360,"internally_reserved":0,"allocated":827392,"data_stored":299818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:02 2021","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.98899999999999999,"15min":0.998},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.516,"5min":1.516,"15min":1.516},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.873,"5min":0.90400000000000003,"15min":0.90900000000000003},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.238,"5min":1.365,"15min":1.365},"last":1.034}]},{"osd":1,"last update":"Wed Feb 10 04:27:02 2021","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.92700000000000005,"15min":0.93100000000000005},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.502,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.998,"15min":1.002},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.419,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.81799999999999995}]},{"osd":2,"last update":"Wed Feb 10 04:27:02 2021","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.997,"15min":0.995},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":1.0760000000000001},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.831,"5min":1.831,"15min":1.831},"last":0.85799999999999998}]},{"osd":3,"last update":"Wed Feb 10 04:27:02 2021","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.0980000000000001,"15min":1.0920000000000001},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":1.2},{"interface":"front","average":{"1min":1.087,"5min":1.0720000000000001,"15min":1.0700000000000001},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.7030000000000001,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":0.75800000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:27:02 2021","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":0.92000000000000004,"15min":0.90500000000000003},"min":{"1min":0.65300000000000002,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.1699999999999999,"5min":0.98999999999999999,"15min":0.95999999999999996},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":1.29}]},{"osd":6,"last update":"Wed Feb 10 04:27:23 2021","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.149,"15min":1.119},"min":{"1min":0.88100000000000001,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":1.3720000000000001},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.085,"15min":1.0549999999999999},"min":{"1min":0.92100000000000004,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.3340000000000001,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":0.95299999999999996}]},{"osd":7,"last update":"Wed Feb 10 04:26:35 2021","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.171,"15min":1.171},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.4450000000000001,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.359,"5min":1.359,"15min":1.359},"min":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":1.673,"5min":1.673,"15min":1.673},"last":1.452}]}]}]}} 2021-02-10T04:27:42.873 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:27:43.077 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:27:43.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:43 smithi085 conmon[29319]: cluster 2021-02-10T04:27:42.181936+0000 osd.1 (osd.1) 27 : cluster [DBG] 6.1 scrub ok 2021-02-10T04:27:43.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:43 smithi085 conmon[29319]: audit 2021-02-10T04:27:42.211169+0000 mgr.y (mgr.14140) 295 : audit [DBG] from='client.24604 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:27:43.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:43 smithi085 conmon[29319]: cluster 2021-02-10T04:27:42.487146+0000 osd.4 (osd.4) 23 : cluster [DBG] 4.13 scrub ok 2021-02-10T04:27:43.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:43 smithi085 conmon[35961]: cluster 2021-02-10T04:27:42.181936+0000 osd.1 (osd.1) 2021-02-10T04:27:43.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:43 smithi085 conmon[35961]: 27 : cluster [DBG] 6.1 scrub ok 2021-02-10T04:27:43.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:43 smithi085 conmon[35961]: audit 2021-02-10T04:27:42.211169+0000 mgr.y (mgr.14140) 295 : audit 2021-02-10T04:27:43.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:43 smithi085 conmon[35961]: [DBG] from='client.24604 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:27:43.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:43 smithi085 conmon[35961]: cluster 2021-02-10T04:27:42.487146 2021-02-10T04:27:43.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:43 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 23 : cluster [DBG] 4.13 scrub ok 2021-02-10T04:27:43.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:43 smithi164 conmon[29288]: cluster 2021-02-10T04:27:42.181936+0000 osd.1 (osd.1) 27 : cluster [DBG] 6.1 scrub ok 2021-02-10T04:27:43.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:43 smithi164 conmon[29288]: audit 2021-02-10T04:27:42.211169+0000 mgr.y (mgr.14140) 295 : audit [DBG] from='client.24604 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:27:43.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:43 smithi164 conmon[29288]: cluster 2021-02-10T04:27:42.487146+0000 osd.4 (osd.4) 23 : cluster [DBG] 4.13 scrub ok 2021-02-10T04:27:44.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:44 smithi164 conmon[29288]: cluster 2021-02-10T04:27:43.978481+0000 mgr.y (mgr.14140) 296 : cluster [DBG] pgmap v256: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:27:44.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:44 smithi085 conmon[29319]: cluster 2021-02-10T04:27:43.978481+0000 mgr.y (mgr.14140) 296 : cluster [DBG] 2021-02-10T04:27:44.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:44 smithi085 conmon[29319]: pgmap v256: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:27:44.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:44 smithi085 conmon[35961]: cluster 2021-02-10T04:27:43.978481+0000 mgr.y (mgr 2021-02-10T04:27:44.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:44 smithi085 conmon[35961]: .14140) 296 : cluster [DBG] pgmap v256: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:27:44.852 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:27:44.855 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:27:45.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:45 smithi085 conmon[29319]: cluster 2021-02-10T04:27:44.501017+0000 osd.4 (osd.4) 2021-02-10T04:27:45.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:45 smithi085 conmon[29319]: 24 : cluster [DBG] 5.1 scrub ok 2021-02-10T04:27:45.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:45 smithi085 conmon[29319]: cluster 2021-02-10T04:27:44.556738+0000 osd.2 (osd.2) 21 : cluster [DBG] 2.1 scrub ok 2021-02-10T04:27:45.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:45 smithi085 conmon[29319]: audit 2021-02-10T04:27:44.840113+0000 mgr.y (mgr.14140) 297 : audit [DBG] from='client.24610 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:27:45.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:45 smithi085 conmon[35961]: cluster 2021-02-10T04:27:44.501017 2021-02-10T04:27:45.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:45 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 24 : cluster [DBG] 5.1 scrub ok 2021-02-10T04:27:45.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:45 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:27:45.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:45 smithi085 conmon[35961]: :27:44.556738+0000 osd.2 (osd 2021-02-10T04:27:45.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:45 smithi085 conmon[35961]: .2) 21 : cluster [DBG] 2.1 scrub ok 2021-02-10T04:27:45.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:45 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:27:45.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:45 smithi085 conmon[35961]: 02-10T04:27:44.840113+0000 mgr.y 2021-02-10T04:27:45.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:45 smithi085 conmon[35961]: (mgr.14140) 297 : audit [DBG] from='client.24610 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:27:45.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:45 smithi164 conmon[29288]: cluster 2021-02-10T04:27:44.501017+0000 osd.4 (osd.4) 24 : cluster [DBG] 5.1 scrub ok 2021-02-10T04:27:45.570 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:45 smithi164 conmon[29288]: cluster 2021-02-10T04:27:44.556738+0000 osd.2 (osd.2) 21 : cluster [DBG] 2.1 scrub ok 2021-02-10T04:27:45.571 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:45 smithi164 conmon[29288]: audit 2021-02-10T04:27:44.840113+0000 mgr.y (mgr.14140) 297 : audit [DBG] from='client.24610 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:27:45.592 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2021-02-10T04:27:43.978201+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":25858,"num_objects":357,"num_object_clones":0,"num_object_copies":1071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":357,"num_whiteouts":0,"num_read":1591,"num_read_kb":1059,"num_write":759,"num_write_kb":177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":787,"ondisk_log_size":787,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":57952,"kb_used_data":6944,"kb_used_omap":0,"kb_used_meta":50944,"kb_avail":749903264,"statfs":{"total":767960285184,"available":767900942336,"internally_reserved":0,"allocated":7110656,"data_stored":2438424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52166656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001979"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":"35","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.055645+0000","last_change":"2021-02-10T04:27:07.339624+0000","last_active":"2021-02-10T04:27:30.055645+0000","last_peered":"2021-02-10T04:27:30.055645+0000","last_clean":"2021-02-10T04:27:30.055645+0000","last_became_active":"2021-02-10T04:25:53.150917+0000","last_became_peered":"2021-02-10T04:25:53.150917+0000","last_unstale":"2021-02-10T04:27:30.055645+0000","last_undegraded":"2021-02-10T04:27:30.055645+0000","last_fullsized":"2021-02-10T04:27:30.055645+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:07.339551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:07.339551+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"34","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.077776+0000","last_change":"2021-02-10T04:25:51.029620+0000","last_active":"2021-02-10T04:27:36.077776+0000","last_peered":"2021-02-10T04:27:36.077776+0000","last_clean":"2021-02-10T04:27:36.077776+0000","last_became_active":"2021-02-10T04:25:51.029110+0000","last_became_peered":"2021-02-10T04:25:51.029110+0000","last_unstale":"2021-02-10T04:27:36.077776+0000","last_undegraded":"2021-02-10T04:27:36.077776+0000","last_fullsized":"2021-02-10T04:27:36.077776+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:25:49.999907+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"51","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.057588+0000","last_change":"2021-02-10T04:26:44.383142+0000","last_active":"2021-02-10T04:27:30.057588+0000","last_peered":"2021-02-10T04:27:30.057588+0000","last_clean":"2021-02-10T04:27:30.057588+0000","last_became_active":"2021-02-10T04:25:36.336112+0000","last_became_peered":"2021-02-10T04:25:36.336112+0000","last_unstale":"2021-02-10T04:27:30.057588+0000","last_undegraded":"2021-02-10T04:27:30.057588+0000","last_fullsized":"2021-02-10T04:27:30.057588+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:44.383073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:44.383073+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"63'44","reported_seq":"129","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.055532+0000","last_change":"2021-02-10T04:25:49.008766+0000","last_active":"2021-02-10T04:27:30.055532+0000","last_peered":"2021-02-10T04:27:30.055532+0000","last_clean":"2021-02-10T04:27:30.055532+0000","last_became_active":"2021-02-10T04:25:49.008451+0000","last_became_peered":"2021-02-10T04:25:49.008451+0000","last_unstale":"2021-02-10T04:27:30.055532+0000","last_undegraded":"2021-02-10T04:27:30.055532+0000","last_fullsized":"2021-02-10T04:27:30.055532+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:25:47.970100+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":89,"num_read_kb":66,"num_write":45,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"32","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.077977+0000","last_change":"2021-02-10T04:25:53.044249+0000","last_active":"2021-02-10T04:27:36.077977+0000","last_peered":"2021-02-10T04:27:36.077977+0000","last_clean":"2021-02-10T04:27:36.077977+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:27:36.077977+0000","last_undegraded":"2021-02-10T04:27:36.077977+0000","last_fullsized":"2021-02-10T04:27:36.077977+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:25:52.015519+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"39","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.376590+0000","last_change":"2021-02-10T04:26:54.007373+0000","last_active":"2021-02-10T04:27:36.376590+0000","last_peered":"2021-02-10T04:27:36.376590+0000","last_clean":"2021-02-10T04:27:36.376590+0000","last_became_active":"2021-02-10T04:25:51.030405+0000","last_became_peered":"2021-02-10T04:25:51.030405+0000","last_unstale":"2021-02-10T04:27:36.376590+0000","last_undegraded":"2021-02-10T04:27:36.376590+0000","last_fullsized":"2021-02-10T04:27:36.376590+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:54.007310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:26:54.007310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"46","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.056445+0000","last_change":"2021-02-10T04:27:13.332288+0000","last_active":"2021-02-10T04:27:30.056445+0000","last_peered":"2021-02-10T04:27:30.056445+0000","last_clean":"2021-02-10T04:27:30.056445+0000","last_became_active":"2021-02-10T04:25:34.327162+0000","last_became_peered":"2021-02-10T04:25:34.327162+0000","last_unstale":"2021-02-10T04:27:30.056445+0000","last_undegraded":"2021-02-10T04:27:30.056445+0000","last_fullsized":"2021-02-10T04:27:30.056445+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:27:13.332224+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"78'32","reported_seq":"95","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.077930+0000","last_change":"2021-02-10T04:27:09.170492+0000","last_active":"2021-02-10T04:27:36.077930+0000","last_peered":"2021-02-10T04:27:36.077930+0000","last_clean":"2021-02-10T04:27:36.077930+0000","last_became_active":"2021-02-10T04:25:49.010481+0000","last_became_peered":"2021-02-10T04:25:49.010481+0000","last_unstale":"2021-02-10T04:27:36.077930+0000","last_undegraded":"2021-02-10T04:27:36.077930+0000","last_fullsized":"2021-02-10T04:27:36.077930+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'32","last_scrub_stamp":"2021-02-10T04:27:09.170428+0000","last_deep_scrub":"78'32","last_deep_scrub_stamp":"2021-02-10T04:27:09.170428+0000","last_clean_scrub_stamp":"2021-02-10T04:27:09.170428+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":47,"num_read_kb":28,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"37","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.377466+0000","last_change":"2021-02-10T04:27:07.962210+0000","last_active":"2021-02-10T04:27:36.377466+0000","last_peered":"2021-02-10T04:27:36.377466+0000","last_clean":"2021-02-10T04:27:36.377466+0000","last_became_active":"2021-02-10T04:25:53.042601+0000","last_became_peered":"2021-02-10T04:25:53.042601+0000","last_unstale":"2021-02-10T04:27:36.377466+0000","last_undegraded":"2021-02-10T04:27:36.377466+0000","last_fullsized":"2021-02-10T04:27:36.377466+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:07.962115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:07.962115+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"38","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748868+0000","last_change":"2021-02-10T04:27:29.251059+0000","last_active":"2021-02-10T04:27:30.748868+0000","last_peered":"2021-02-10T04:27:30.748868+0000","last_clean":"2021-02-10T04:27:30.748868+0000","last_became_active":"2021-02-10T04:25:51.029286+0000","last_became_peered":"2021-02-10T04:25:51.029286+0000","last_unstale":"2021-02-10T04:27:30.748868+0000","last_undegraded":"2021-02-10T04:27:30.748868+0000","last_fullsized":"2021-02-10T04:27:30.748868+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:29.250978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:29.250978+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"45","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.749528+0000","last_change":"2021-02-10T04:26:42.262025+0000","last_active":"2021-02-10T04:27:30.749528+0000","last_peered":"2021-02-10T04:27:30.749528+0000","last_clean":"2021-02-10T04:27:30.749528+0000","last_became_active":"2021-02-10T04:25:36.344477+0000","last_became_peered":"2021-02-10T04:25:36.344477+0000","last_unstale":"2021-02-10T04:27:30.749528+0000","last_undegraded":"2021-02-10T04:27:30.749528+0000","last_fullsized":"2021-02-10T04:27:30.749528+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:42.261921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:42.261921+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"63'12","reported_seq":"68","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:37.441319+0000","last_change":"2021-02-10T04:27:37.441260+0000","last_active":"2021-02-10T04:27:37.441319+0000","last_peered":"2021-02-10T04:27:37.441319+0000","last_clean":"2021-02-10T04:27:37.441319+0000","last_became_active":"2021-02-10T04:25:49.013196+0000","last_became_peered":"2021-02-10T04:25:49.013196+0000","last_unstale":"2021-02-10T04:27:37.441319+0000","last_undegraded":"2021-02-10T04:27:37.441319+0000","last_fullsized":"2021-02-10T04:27:37.441319+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2021-02-10T04:27:37.441168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:37.441168+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"36","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:33.628151+0000","last_change":"2021-02-10T04:27:33.628099+0000","last_active":"2021-02-10T04:27:33.628151+0000","last_peered":"2021-02-10T04:27:33.628151+0000","last_clean":"2021-02-10T04:27:33.628151+0000","last_became_active":"2021-02-10T04:25:53.039844+0000","last_became_peered":"2021-02-10T04:25:53.039844+0000","last_unstale":"2021-02-10T04:27:33.628151+0000","last_undegraded":"2021-02-10T04:27:33.628151+0000","last_fullsized":"2021-02-10T04:27:33.628151+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:33.628038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:33.628038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"39","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.377988+0000","last_change":"2021-02-10T04:27:04.977415+0000","last_active":"2021-02-10T04:27:36.377988+0000","last_peered":"2021-02-10T04:27:36.377988+0000","last_clean":"2021-02-10T04:27:36.377988+0000","last_became_active":"2021-02-10T04:25:51.025488+0000","last_became_peered":"2021-02-10T04:25:51.025488+0000","last_unstale":"2021-02-10T04:27:36.377988+0000","last_undegraded":"2021-02-10T04:27:36.377988+0000","last_fullsized":"2021-02-10T04:27:36.377988+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:04.977353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:04.977353+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"45","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.749155+0000","last_change":"2021-02-10T04:26:45.223669+0000","last_active":"2021-02-10T04:27:30.749155+0000","last_peered":"2021-02-10T04:27:30.749155+0000","last_clean":"2021-02-10T04:27:30.749155+0000","last_became_active":"2021-02-10T04:25:36.337260+0000","last_became_peered":"2021-02-10T04:25:36.337260+0000","last_unstale":"2021-02-10T04:27:30.749155+0000","last_undegraded":"2021-02-10T04:27:30.749155+0000","last_fullsized":"2021-02-10T04:27:30.749155+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:45.223603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:45.223603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"63'25","reported_seq":"95","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.078074+0000","last_change":"2021-02-10T04:27:05.242824+0000","last_active":"2021-02-10T04:27:36.078074+0000","last_peered":"2021-02-10T04:27:36.078074+0000","last_clean":"2021-02-10T04:27:36.078074+0000","last_became_active":"2021-02-10T04:25:49.000911+0000","last_became_peered":"2021-02-10T04:25:49.000911+0000","last_unstale":"2021-02-10T04:27:36.078074+0000","last_undegraded":"2021-02-10T04:27:36.078074+0000","last_fullsized":"2021-02-10T04:27:36.078074+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2021-02-10T04:27:05.242762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:05.242762+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":53,"num_read_kb":37,"num_write":26,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"35","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.056801+0000","last_change":"2021-02-10T04:26:55.332873+0000","last_active":"2021-02-10T04:27:30.056801+0000","last_peered":"2021-02-10T04:27:30.056801+0000","last_clean":"2021-02-10T04:27:30.056801+0000","last_became_active":"2021-02-10T04:25:53.147831+0000","last_became_peered":"2021-02-10T04:25:53.147831+0000","last_unstale":"2021-02-10T04:27:30.056801+0000","last_undegraded":"2021-02-10T04:27:30.056801+0000","last_fullsized":"2021-02-10T04:27:30.056801+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:55.332815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:26:55.332815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"34","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.078343+0000","last_change":"2021-02-10T04:25:51.019289+0000","last_active":"2021-02-10T04:27:36.078343+0000","last_peered":"2021-02-10T04:27:36.078343+0000","last_clean":"2021-02-10T04:27:36.078343+0000","last_became_active":"2021-02-10T04:25:51.018872+0000","last_became_peered":"2021-02-10T04:25:51.018872+0000","last_unstale":"2021-02-10T04:27:36.078343+0000","last_undegraded":"2021-02-10T04:27:36.078343+0000","last_fullsized":"2021-02-10T04:27:36.078343+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:25:49.999907+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"55","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.055716+0000","last_change":"2021-02-10T04:26:46.425375+0000","last_active":"2021-02-10T04:27:30.055716+0000","last_peered":"2021-02-10T04:27:30.055716+0000","last_clean":"2021-02-10T04:27:30.055716+0000","last_became_active":"2021-02-10T04:25:36.336893+0000","last_became_peered":"2021-02-10T04:25:36.336893+0000","last_unstale":"2021-02-10T04:27:30.055716+0000","last_undegraded":"2021-02-10T04:27:30.055716+0000","last_fullsized":"2021-02-10T04:27:30.055716+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:46.425288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:26:46.425288+0000","last_clean_scrub_stamp":"2021-02-10T04:26:46.425288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"62'22","reported_seq":"80","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.056178+0000","last_change":"2021-02-10T04:25:49.009749+0000","last_active":"2021-02-10T04:27:30.056178+0000","last_peered":"2021-02-10T04:27:30.056178+0000","last_clean":"2021-02-10T04:27:30.056178+0000","last_became_active":"2021-02-10T04:25:49.008997+0000","last_became_peered":"2021-02-10T04:25:49.008997+0000","last_unstale":"2021-02-10T04:27:30.056178+0000","last_undegraded":"2021-02-10T04:27:30.056178+0000","last_fullsized":"2021-02-10T04:27:30.056178+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:25:47.970100+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":49,"num_read_kb":31,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"37","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.078715+0000","last_change":"2021-02-10T04:27:29.364660+0000","last_active":"2021-02-10T04:27:36.078715+0000","last_peered":"2021-02-10T04:27:36.078715+0000","last_clean":"2021-02-10T04:27:36.078715+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:27:36.078715+0000","last_undegraded":"2021-02-10T04:27:36.078715+0000","last_fullsized":"2021-02-10T04:27:36.078715+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:29.364585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:29.364585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"39","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.377296+0000","last_change":"2021-02-10T04:27:27.017142+0000","last_active":"2021-02-10T04:27:36.377296+0000","last_peered":"2021-02-10T04:27:36.377296+0000","last_clean":"2021-02-10T04:27:36.377296+0000","last_became_active":"2021-02-10T04:25:51.029399+0000","last_became_peered":"2021-02-10T04:25:51.029399+0000","last_unstale":"2021-02-10T04:27:36.377296+0000","last_undegraded":"2021-02-10T04:27:36.377296+0000","last_fullsized":"2021-02-10T04:27:36.377296+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:27.017040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:27.017040+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"50","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.272163+0000","last_change":"2021-02-10T04:27:02.190158+0000","last_active":"2021-02-10T04:27:30.272163+0000","last_peered":"2021-02-10T04:27:30.272163+0000","last_clean":"2021-02-10T04:27:30.272163+0000","last_became_active":"2021-02-10T04:25:36.336442+0000","last_became_peered":"2021-02-10T04:25:36.336442+0000","last_unstale":"2021-02-10T04:27:30.272163+0000","last_undegraded":"2021-02-10T04:27:30.272163+0000","last_fullsized":"2021-02-10T04:27:30.272163+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:27:02.190056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:02.190056+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"62'15","reported_seq":"64","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.164914+0000","last_change":"2021-02-10T04:26:56.623190+0000","last_active":"2021-02-10T04:27:30.164914+0000","last_peered":"2021-02-10T04:27:30.164914+0000","last_clean":"2021-02-10T04:27:30.164914+0000","last_became_active":"2021-02-10T04:25:49.010553+0000","last_became_peered":"2021-02-10T04:25:49.010553+0000","last_unstale":"2021-02-10T04:27:30.164914+0000","last_undegraded":"2021-02-10T04:27:30.164914+0000","last_fullsized":"2021-02-10T04:27:30.164914+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:26:56.623109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:26:56.623109+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"37","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.377081+0000","last_change":"2021-02-10T04:27:26.029923+0000","last_active":"2021-02-10T04:27:36.377081+0000","last_peered":"2021-02-10T04:27:36.377081+0000","last_clean":"2021-02-10T04:27:36.377081+0000","last_became_active":"2021-02-10T04:25:53.041098+0000","last_became_peered":"2021-02-10T04:25:53.041098+0000","last_unstale":"2021-02-10T04:27:36.377081+0000","last_undegraded":"2021-02-10T04:27:36.377081+0000","last_fullsized":"2021-02-10T04:27:36.377081+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:26.029856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:26.029856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"38","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.057722+0000","last_change":"2021-02-10T04:27:27.564268+0000","last_active":"2021-02-10T04:27:30.057722+0000","last_peered":"2021-02-10T04:27:30.057722+0000","last_clean":"2021-02-10T04:27:30.057722+0000","last_became_active":"2021-02-10T04:25:51.022853+0000","last_became_peered":"2021-02-10T04:25:51.022853+0000","last_unstale":"2021-02-10T04:27:30.057722+0000","last_undegraded":"2021-02-10T04:27:30.057722+0000","last_fullsized":"2021-02-10T04:27:30.057722+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:27.564175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:27.564175+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"46","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.055338+0000","last_change":"2021-02-10T04:27:16.340830+0000","last_active":"2021-02-10T04:27:30.055338+0000","last_peered":"2021-02-10T04:27:30.055338+0000","last_clean":"2021-02-10T04:27:30.055338+0000","last_became_active":"2021-02-10T04:25:34.324981+0000","last_became_peered":"2021-02-10T04:25:34.324981+0000","last_unstale":"2021-02-10T04:27:30.055338+0000","last_undegraded":"2021-02-10T04:27:30.055338+0000","last_fullsized":"2021-02-10T04:27:30.055338+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:16.340724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:16.340724+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"76'35","reported_seq":"98","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.272675+0000","last_change":"2021-02-10T04:26:58.198706+0000","last_active":"2021-02-10T04:27:30.272675+0000","last_peered":"2021-02-10T04:27:30.272675+0000","last_clean":"2021-02-10T04:27:30.272675+0000","last_became_active":"2021-02-10T04:25:49.015760+0000","last_became_peered":"2021-02-10T04:25:49.015760+0000","last_unstale":"2021-02-10T04:27:30.272675+0000","last_undegraded":"2021-02-10T04:27:30.272675+0000","last_fullsized":"2021-02-10T04:27:30.272675+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'35","last_scrub_stamp":"2021-02-10T04:26:58.198641+0000","last_deep_scrub":"76'35","last_deep_scrub_stamp":"2021-02-10T04:26:58.198641+0000","last_clean_scrub_stamp":"2021-02-10T04:26:58.198641+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":50,"num_read_kb":30,"num_write":24,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"36","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:39.139028+0000","last_change":"2021-02-10T04:27:39.139028+0000","last_active":"2021-02-10T04:27:39.139028+0000","last_peered":"2021-02-10T04:27:39.139028+0000","last_clean":"2021-02-10T04:27:39.139028+0000","last_became_active":"2021-02-10T04:25:53.042279+0000","last_became_peered":"2021-02-10T04:25:53.042279+0000","last_unstale":"2021-02-10T04:27:39.139028+0000","last_undegraded":"2021-02-10T04:27:39.139028+0000","last_fullsized":"2021-02-10T04:27:39.139028+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:39.138947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:39.138947+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"58'2","reported_seq":"62","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.738481+0000","last_change":"2021-02-10T04:27:15.041424+0000","last_active":"2021-02-10T04:27:36.738481+0000","last_peered":"2021-02-10T04:27:36.738481+0000","last_clean":"2021-02-10T04:27:36.738481+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:27:36.738481+0000","last_undegraded":"2021-02-10T04:27:36.738481+0000","last_fullsized":"2021-02-10T04:27:36.738481+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:15.041329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:15.041329+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"57","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.377253+0000","last_change":"2021-02-10T04:27:04.011149+0000","last_active":"2021-02-10T04:27:36.377253+0000","last_peered":"2021-02-10T04:27:36.377253+0000","last_clean":"2021-02-10T04:27:36.377253+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:27:36.377253+0000","last_undegraded":"2021-02-10T04:27:36.377253+0000","last_fullsized":"2021-02-10T04:27:36.377253+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:04.011053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:04.011053+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"63'22","reported_seq":"82","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.057739+0000","last_change":"2021-02-10T04:26:51.409218+0000","last_active":"2021-02-10T04:27:30.057739+0000","last_peered":"2021-02-10T04:27:30.057739+0000","last_clean":"2021-02-10T04:27:30.057739+0000","last_became_active":"2021-02-10T04:25:49.009177+0000","last_became_peered":"2021-02-10T04:25:49.009177+0000","last_unstale":"2021-02-10T04:27:30.057739+0000","last_undegraded":"2021-02-10T04:27:30.057739+0000","last_fullsized":"2021-02-10T04:27:30.057739+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2021-02-10T04:26:51.409147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:26:51.409147+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":22,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"30","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.056032+0000","last_change":"2021-02-10T04:25:53.151054+0000","last_active":"2021-02-10T04:27:30.056032+0000","last_peered":"2021-02-10T04:27:30.056032+0000","last_clean":"2021-02-10T04:27:30.056032+0000","last_became_active":"2021-02-10T04:25:53.150845+0000","last_became_peered":"2021-02-10T04:25:53.150845+0000","last_unstale":"2021-02-10T04:27:30.056032+0000","last_undegraded":"2021-02-10T04:27:30.056032+0000","last_fullsized":"2021-02-10T04:27:30.056032+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:25:52.015519+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"38","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748313+0000","last_change":"2021-02-10T04:27:07.181928+0000","last_active":"2021-02-10T04:27:30.748313+0000","last_peered":"2021-02-10T04:27:30.748313+0000","last_clean":"2021-02-10T04:27:30.748313+0000","last_became_active":"2021-02-10T04:25:51.018491+0000","last_became_peered":"2021-02-10T04:25:51.018491+0000","last_unstale":"2021-02-10T04:27:30.748313+0000","last_undegraded":"2021-02-10T04:27:30.748313+0000","last_fullsized":"2021-02-10T04:27:30.748313+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:07.181854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:07.181854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"50","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.272698+0000","last_change":"2021-02-10T04:26:52.256790+0000","last_active":"2021-02-10T04:27:30.272698+0000","last_peered":"2021-02-10T04:27:30.272698+0000","last_clean":"2021-02-10T04:27:30.272698+0000","last_became_active":"2021-02-10T04:25:35.328828+0000","last_became_peered":"2021-02-10T04:25:35.328828+0000","last_unstale":"2021-02-10T04:27:30.272698+0000","last_undegraded":"2021-02-10T04:27:30.272698+0000","last_fullsized":"2021-02-10T04:27:30.272698+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:26:52.256723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:52.256723+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"63'21","reported_seq":"80","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.055480+0000","last_change":"2021-02-10T04:27:12.291860+0000","last_active":"2021-02-10T04:27:30.055480+0000","last_peered":"2021-02-10T04:27:30.055480+0000","last_clean":"2021-02-10T04:27:30.055480+0000","last_became_active":"2021-02-10T04:25:49.007049+0000","last_became_peered":"2021-02-10T04:25:49.007049+0000","last_unstale":"2021-02-10T04:27:30.055480+0000","last_undegraded":"2021-02-10T04:27:30.055480+0000","last_fullsized":"2021-02-10T04:27:30.055480+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2021-02-10T04:27:12.291796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:12.291796+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":27,"num_write":21,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"36","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748346+0000","last_change":"2021-02-10T04:27:19.134286+0000","last_active":"2021-02-10T04:27:30.748346+0000","last_peered":"2021-02-10T04:27:30.748346+0000","last_clean":"2021-02-10T04:27:30.748346+0000","last_became_active":"2021-02-10T04:25:53.034771+0000","last_became_peered":"2021-02-10T04:25:53.034771+0000","last_unstale":"2021-02-10T04:27:30.748346+0000","last_undegraded":"2021-02-10T04:27:30.748346+0000","last_fullsized":"2021-02-10T04:27:30.748346+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:27:19.134212+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"38","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.272056+0000","last_change":"2021-02-10T04:27:19.077862+0000","last_active":"2021-02-10T04:27:30.272056+0000","last_peered":"2021-02-10T04:27:30.272056+0000","last_clean":"2021-02-10T04:27:30.272056+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:27:30.272056+0000","last_undegraded":"2021-02-10T04:27:30.272056+0000","last_fullsized":"2021-02-10T04:27:30.272056+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:27:19.077777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"48","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.378731+0000","last_change":"2021-02-10T04:27:03.000902+0000","last_active":"2021-02-10T04:27:36.378731+0000","last_peered":"2021-02-10T04:27:36.378731+0000","last_clean":"2021-02-10T04:27:36.378731+0000","last_became_active":"2021-02-10T04:25:34.331787+0000","last_became_peered":"2021-02-10T04:25:34.331787+0000","last_unstale":"2021-02-10T04:27:36.378731+0000","last_undegraded":"2021-02-10T04:27:36.378731+0000","last_fullsized":"2021-02-10T04:27:36.378731+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:03.000821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:03.000821+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"62'12","reported_seq":"87","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.055246+0000","last_change":"2021-02-10T04:26:39.021685+0000","last_active":"2021-02-10T04:27:30.055246+0000","last_peered":"2021-02-10T04:27:30.055246+0000","last_clean":"2021-02-10T04:27:30.055246+0000","last_became_active":"2021-02-10T04:26:39.005616+0000","last_became_peered":"2021-02-10T04:26:39.005616+0000","last_unstale":"2021-02-10T04:27:30.055246+0000","last_undegraded":"2021-02-10T04:27:30.055246+0000","last_fullsized":"2021-02-10T04:27:30.055246+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:25:47.970100+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"35","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.055176+0000","last_change":"2021-02-10T04:26:57.309978+0000","last_active":"2021-02-10T04:27:30.055176+0000","last_peered":"2021-02-10T04:27:30.055176+0000","last_clean":"2021-02-10T04:27:30.055176+0000","last_became_active":"2021-02-10T04:25:53.043277+0000","last_became_peered":"2021-02-10T04:25:53.043277+0000","last_unstale":"2021-02-10T04:27:30.055176+0000","last_undegraded":"2021-02-10T04:27:30.055176+0000","last_fullsized":"2021-02-10T04:27:30.055176+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:57.309915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:26:57.309915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"39","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.377071+0000","last_change":"2021-02-10T04:27:33.030004+0000","last_active":"2021-02-10T04:27:36.377071+0000","last_peered":"2021-02-10T04:27:36.377071+0000","last_clean":"2021-02-10T04:27:36.377071+0000","last_became_active":"2021-02-10T04:25:51.030952+0000","last_became_peered":"2021-02-10T04:25:51.030952+0000","last_unstale":"2021-02-10T04:27:36.377071+0000","last_undegraded":"2021-02-10T04:27:36.377071+0000","last_fullsized":"2021-02-10T04:27:36.377071+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:27:33.029912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"57","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.377155+0000","last_change":"2021-02-10T04:26:52.011165+0000","last_active":"2021-02-10T04:27:36.377155+0000","last_peered":"2021-02-10T04:27:36.377155+0000","last_clean":"2021-02-10T04:27:36.377155+0000","last_became_active":"2021-02-10T04:25:34.330242+0000","last_became_peered":"2021-02-10T04:25:34.330242+0000","last_unstale":"2021-02-10T04:27:36.377155+0000","last_undegraded":"2021-02-10T04:27:36.377155+0000","last_fullsized":"2021-02-10T04:27:36.377155+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:26:52.011096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:52.011096+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"62'15","reported_seq":"63","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.078209+0000","last_change":"2021-02-10T04:25:49.011596+0000","last_active":"2021-02-10T04:27:36.078209+0000","last_peered":"2021-02-10T04:27:36.078209+0000","last_clean":"2021-02-10T04:27:36.078209+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:27:36.078209+0000","last_undegraded":"2021-02-10T04:27:36.078209+0000","last_fullsized":"2021-02-10T04:27:36.078209+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:25:47.970100+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"36","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.056976+0000","last_change":"2021-02-10T04:27:21.548446+0000","last_active":"2021-02-10T04:27:30.056976+0000","last_peered":"2021-02-10T04:27:30.056976+0000","last_clean":"2021-02-10T04:27:30.056976+0000","last_became_active":"2021-02-10T04:25:53.039322+0000","last_became_peered":"2021-02-10T04:25:53.039322+0000","last_unstale":"2021-02-10T04:27:30.056976+0000","last_undegraded":"2021-02-10T04:27:30.056976+0000","last_fullsized":"2021-02-10T04:27:30.056976+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:21.548381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:21.548381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"37","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.055886+0000","last_change":"2021-02-10T04:27:05.297715+0000","last_active":"2021-02-10T04:27:30.055886+0000","last_peered":"2021-02-10T04:27:30.055886+0000","last_clean":"2021-02-10T04:27:30.055886+0000","last_became_active":"2021-02-10T04:25:51.029155+0000","last_became_peered":"2021-02-10T04:25:51.029155+0000","last_unstale":"2021-02-10T04:27:30.055886+0000","last_undegraded":"2021-02-10T04:27:30.055886+0000","last_fullsized":"2021-02-10T04:27:30.055886+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:27:05.297647+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"47","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.272247+0000","last_change":"2021-02-10T04:26:54.256587+0000","last_active":"2021-02-10T04:27:30.272247+0000","last_peered":"2021-02-10T04:27:30.272247+0000","last_clean":"2021-02-10T04:27:30.272247+0000","last_became_active":"2021-02-10T04:25:34.330070+0000","last_became_peered":"2021-02-10T04:25:34.330070+0000","last_unstale":"2021-02-10T04:27:30.272247+0000","last_undegraded":"2021-02-10T04:27:30.272247+0000","last_fullsized":"2021-02-10T04:27:30.272247+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:54.256498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:54.256498+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"62'15","reported_seq":"69","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:40.218960+0000","last_change":"2021-02-10T04:27:40.218910+0000","last_active":"2021-02-10T04:27:40.218960+0000","last_peered":"2021-02-10T04:27:40.218960+0000","last_clean":"2021-02-10T04:27:40.218960+0000","last_became_active":"2021-02-10T04:25:49.015543+0000","last_became_peered":"2021-02-10T04:25:49.015543+0000","last_unstale":"2021-02-10T04:27:40.218960+0000","last_undegraded":"2021-02-10T04:27:40.218960+0000","last_fullsized":"2021-02-10T04:27:40.218960+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:27:40.218828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:40.218828+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"37","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.078878+0000","last_change":"2021-02-10T04:27:28.373194+0000","last_active":"2021-02-10T04:27:36.078878+0000","last_peered":"2021-02-10T04:27:36.078878+0000","last_clean":"2021-02-10T04:27:36.078878+0000","last_became_active":"2021-02-10T04:25:53.031782+0000","last_became_peered":"2021-02-10T04:25:53.031782+0000","last_unstale":"2021-02-10T04:27:36.078878+0000","last_undegraded":"2021-02-10T04:27:36.078878+0000","last_fullsized":"2021-02-10T04:27:36.078878+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:28.373117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:28.373117+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"58'2","reported_seq":"61","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.079389+0000","last_change":"2021-02-10T04:27:26.264628+0000","last_active":"2021-02-10T04:27:36.079389+0000","last_peered":"2021-02-10T04:27:36.079389+0000","last_clean":"2021-02-10T04:27:36.079389+0000","last_became_active":"2021-02-10T04:25:51.018300+0000","last_became_peered":"2021-02-10T04:25:51.018300+0000","last_unstale":"2021-02-10T04:27:36.079389+0000","last_undegraded":"2021-02-10T04:27:36.079389+0000","last_fullsized":"2021-02-10T04:27:36.079389+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:26.264560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:26.264560+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"52","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:39.616474+0000","last_change":"2021-02-10T04:27:39.616474+0000","last_active":"2021-02-10T04:27:39.616474+0000","last_peered":"2021-02-10T04:27:39.616474+0000","last_clean":"2021-02-10T04:27:39.616474+0000","last_became_active":"2021-02-10T04:25:34.321096+0000","last_became_peered":"2021-02-10T04:25:34.321096+0000","last_unstale":"2021-02-10T04:27:39.616474+0000","last_undegraded":"2021-02-10T04:27:39.616474+0000","last_fullsized":"2021-02-10T04:27:39.616474+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:27:39.616381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"63'19","reported_seq":"80","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.378402+0000","last_change":"2021-02-10T04:27:11.016733+0000","last_active":"2021-02-10T04:27:36.378402+0000","last_peered":"2021-02-10T04:27:36.378402+0000","last_clean":"2021-02-10T04:27:36.378402+0000","last_became_active":"2021-02-10T04:25:49.010890+0000","last_became_peered":"2021-02-10T04:25:49.010890+0000","last_unstale":"2021-02-10T04:27:36.378402+0000","last_undegraded":"2021-02-10T04:27:36.378402+0000","last_fullsized":"2021-02-10T04:27:36.378402+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2021-02-10T04:27:11.016650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:11.016650+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":26,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"36","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.057357+0000","last_change":"2021-02-10T04:27:08.486638+0000","last_active":"2021-02-10T04:27:30.057357+0000","last_peered":"2021-02-10T04:27:30.057357+0000","last_clean":"2021-02-10T04:27:30.057357+0000","last_became_active":"2021-02-10T04:25:53.045177+0000","last_became_peered":"2021-02-10T04:25:53.045177+0000","last_unstale":"2021-02-10T04:27:30.057357+0000","last_undegraded":"2021-02-10T04:27:30.057357+0000","last_fullsized":"2021-02-10T04:27:30.057357+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:08.486553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:08.486553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"58'2","reported_seq":"64","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376770+0000","last_change":"2021-02-10T04:27:20.157871+0000","last_active":"2021-02-10T04:27:41.376770+0000","last_peered":"2021-02-10T04:27:41.376770+0000","last_clean":"2021-02-10T04:27:41.376770+0000","last_became_active":"2021-02-10T04:25:51.032328+0000","last_became_peered":"2021-02-10T04:25:51.032328+0000","last_unstale":"2021-02-10T04:27:41.376770+0000","last_undegraded":"2021-02-10T04:27:41.376770+0000","last_fullsized":"2021-02-10T04:27:41.376770+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:20.157772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:20.157772+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"52'4","reported_seq":"60","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.377255+0000","last_change":"2021-02-10T04:26:49.013781+0000","last_active":"2021-02-10T04:27:36.377255+0000","last_peered":"2021-02-10T04:27:36.377255+0000","last_clean":"2021-02-10T04:27:36.377255+0000","last_became_active":"2021-02-10T04:25:36.334366+0000","last_became_peered":"2021-02-10T04:25:36.334366+0000","last_unstale":"2021-02-10T04:27:36.377255+0000","last_undegraded":"2021-02-10T04:27:36.377255+0000","last_fullsized":"2021-02-10T04:27:36.377255+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'4","last_scrub_stamp":"2021-02-10T04:26:49.013681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:49.013681+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"63'26","reported_seq":"125","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.055671+0000","last_change":"2021-02-10T04:27:28.213521+0000","last_active":"2021-02-10T04:27:30.055671+0000","last_peered":"2021-02-10T04:27:30.055671+0000","last_clean":"2021-02-10T04:27:30.055671+0000","last_became_active":"2021-02-10T04:26:39.003971+0000","last_became_peered":"2021-02-10T04:26:39.003971+0000","last_unstale":"2021-02-10T04:27:30.055671+0000","last_undegraded":"2021-02-10T04:27:30.055671+0000","last_fullsized":"2021-02-10T04:27:30.055671+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:27:28.213454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:28.213454+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"36","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:33.192435+0000","last_change":"2021-02-10T04:27:33.192375+0000","last_active":"2021-02-10T04:27:33.192435+0000","last_peered":"2021-02-10T04:27:33.192435+0000","last_clean":"2021-02-10T04:27:33.192435+0000","last_became_active":"2021-02-10T04:25:53.149820+0000","last_became_peered":"2021-02-10T04:25:53.149820+0000","last_unstale":"2021-02-10T04:27:33.192435+0000","last_undegraded":"2021-02-10T04:27:33.192435+0000","last_fullsized":"2021-02-10T04:27:33.192435+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:33.192287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:33.192287+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"39","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.378664+0000","last_change":"2021-02-10T04:27:23.236953+0000","last_active":"2021-02-10T04:27:36.378664+0000","last_peered":"2021-02-10T04:27:36.378664+0000","last_clean":"2021-02-10T04:27:36.378664+0000","last_became_active":"2021-02-10T04:25:51.031864+0000","last_became_peered":"2021-02-10T04:25:51.031864+0000","last_unstale":"2021-02-10T04:27:36.378664+0000","last_undegraded":"2021-02-10T04:27:36.378664+0000","last_fullsized":"2021-02-10T04:27:36.378664+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:23.236870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:23.236870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"49","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.272081+0000","last_change":"2021-02-10T04:27:14.083910+0000","last_active":"2021-02-10T04:27:30.272081+0000","last_peered":"2021-02-10T04:27:30.272081+0000","last_clean":"2021-02-10T04:27:30.272081+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:27:30.272081+0000","last_undegraded":"2021-02-10T04:27:30.272081+0000","last_fullsized":"2021-02-10T04:27:30.272081+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:27:14.083841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:14.083841+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"63'13","reported_seq":"86","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.056756+0000","last_change":"2021-02-10T04:26:50.393926+0000","last_active":"2021-02-10T04:27:30.056756+0000","last_peered":"2021-02-10T04:27:30.056756+0000","last_clean":"2021-02-10T04:27:30.056756+0000","last_became_active":"2021-02-10T04:26:39.003832+0000","last_became_peered":"2021-02-10T04:26:39.003832+0000","last_unstale":"2021-02-10T04:27:30.056756+0000","last_undegraded":"2021-02-10T04:27:30.056756+0000","last_fullsized":"2021-02-10T04:27:30.056756+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2021-02-10T04:26:50.393865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:26:50.393865+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"35","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.164646+0000","last_change":"2021-02-10T04:27:13.573977+0000","last_active":"2021-02-10T04:27:30.164646+0000","last_peered":"2021-02-10T04:27:30.164646+0000","last_clean":"2021-02-10T04:27:30.164646+0000","last_became_active":"2021-02-10T04:25:53.035867+0000","last_became_peered":"2021-02-10T04:25:53.035867+0000","last_unstale":"2021-02-10T04:27:30.164646+0000","last_undegraded":"2021-02-10T04:27:30.164646+0000","last_fullsized":"2021-02-10T04:27:30.164646+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:13.573912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:13.573912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"37","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.056833+0000","last_change":"2021-02-10T04:26:56.335818+0000","last_active":"2021-02-10T04:27:30.056833+0000","last_peered":"2021-02-10T04:27:30.056833+0000","last_clean":"2021-02-10T04:27:30.056833+0000","last_became_active":"2021-02-10T04:25:51.030651+0000","last_became_peered":"2021-02-10T04:25:51.030651+0000","last_unstale":"2021-02-10T04:27:30.056833+0000","last_undegraded":"2021-02-10T04:27:30.056833+0000","last_fullsized":"2021-02-10T04:27:30.056833+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:56.335735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:26:56.335735+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"49","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.164398+0000","last_change":"2021-02-10T04:26:57.639364+0000","last_active":"2021-02-10T04:27:30.164398+0000","last_peered":"2021-02-10T04:27:30.164398+0000","last_clean":"2021-02-10T04:27:30.164398+0000","last_became_active":"2021-02-10T04:25:34.329694+0000","last_became_peered":"2021-02-10T04:25:34.329694+0000","last_unstale":"2021-02-10T04:27:30.164398+0000","last_undegraded":"2021-02-10T04:27:30.164398+0000","last_fullsized":"2021-02-10T04:27:30.164398+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:26:57.639281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:57.639281+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"63'28","reported_seq":"100","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.057492+0000","last_change":"2021-02-10T04:27:24.523914+0000","last_active":"2021-02-10T04:27:30.057492+0000","last_peered":"2021-02-10T04:27:30.057492+0000","last_clean":"2021-02-10T04:27:30.057492+0000","last_became_active":"2021-02-10T04:25:49.008942+0000","last_became_peered":"2021-02-10T04:25:49.008942+0000","last_unstale":"2021-02-10T04:27:30.057492+0000","last_undegraded":"2021-02-10T04:27:30.057492+0000","last_fullsized":"2021-02-10T04:27:30.057492+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'28","last_scrub_stamp":"2021-02-10T04:27:24.523837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:24.523837+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":62,"num_read_kb":40,"num_write":29,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"35","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.055959+0000","last_change":"2021-02-10T04:27:27.259500+0000","last_active":"2021-02-10T04:27:30.055959+0000","last_peered":"2021-02-10T04:27:30.055959+0000","last_clean":"2021-02-10T04:27:30.055959+0000","last_became_active":"2021-02-10T04:25:53.047736+0000","last_became_peered":"2021-02-10T04:25:53.047736+0000","last_unstale":"2021-02-10T04:27:30.055959+0000","last_undegraded":"2021-02-10T04:27:30.055959+0000","last_fullsized":"2021-02-10T04:27:30.055959+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:27.259393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:27.259393+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"61","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.078740+0000","last_change":"2021-02-10T04:26:34.066242+0000","last_active":"2021-02-10T04:27:36.078740+0000","last_peered":"2021-02-10T04:27:36.078740+0000","last_clean":"2021-02-10T04:27:36.078740+0000","last_became_active":"2021-02-10T04:25:36.337394+0000","last_became_peered":"2021-02-10T04:25:36.337394+0000","last_unstale":"2021-02-10T04:27:36.078740+0000","last_undegraded":"2021-02-10T04:27:36.078740+0000","last_fullsized":"2021-02-10T04:27:36.078740+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:26:34.066149+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"63'27","reported_seq":"100","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.077936+0000","last_change":"2021-02-10T04:27:17.352786+0000","last_active":"2021-02-10T04:27:36.077936+0000","last_peered":"2021-02-10T04:27:36.077936+0000","last_clean":"2021-02-10T04:27:36.077936+0000","last_became_active":"2021-02-10T04:25:49.013259+0000","last_became_peered":"2021-02-10T04:25:49.013259+0000","last_unstale":"2021-02-10T04:27:36.077936+0000","last_undegraded":"2021-02-10T04:27:36.077936+0000","last_fullsized":"2021-02-10T04:27:36.077936+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'27","last_scrub_stamp":"2021-02-10T04:27:17.352701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:17.352701+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":41,"num_write":29,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"37","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.164393+0000","last_change":"2021-02-10T04:27:14.530439+0000","last_active":"2021-02-10T04:27:30.164393+0000","last_peered":"2021-02-10T04:27:30.164393+0000","last_clean":"2021-02-10T04:27:30.164393+0000","last_became_active":"2021-02-10T04:25:51.023000+0000","last_became_peered":"2021-02-10T04:25:51.023000+0000","last_unstale":"2021-02-10T04:27:30.164393+0000","last_undegraded":"2021-02-10T04:27:30.164393+0000","last_fullsized":"2021-02-10T04:27:30.164393+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:14.530378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:14.530378+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"36","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.056976+0000","last_change":"2021-02-10T04:27:18.521962+0000","last_active":"2021-02-10T04:27:30.056976+0000","last_peered":"2021-02-10T04:27:30.056976+0000","last_clean":"2021-02-10T04:27:30.056976+0000","last_became_active":"2021-02-10T04:25:53.045215+0000","last_became_peered":"2021-02-10T04:25:53.045215+0000","last_unstale":"2021-02-10T04:27:30.056976+0000","last_undegraded":"2021-02-10T04:27:30.056976+0000","last_fullsized":"2021-02-10T04:27:30.056976+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:18.521897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:18.521897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"66","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.376954+0000","last_change":"2021-02-10T04:27:04.024909+0000","last_active":"2021-02-10T04:27:36.376954+0000","last_peered":"2021-02-10T04:27:36.376954+0000","last_clean":"2021-02-10T04:27:36.376954+0000","last_became_active":"2021-02-10T04:26:39.004318+0000","last_became_peered":"2021-02-10T04:26:39.004318+0000","last_unstale":"2021-02-10T04:27:36.376954+0000","last_undegraded":"2021-02-10T04:27:36.376954+0000","last_fullsized":"2021-02-10T04:27:36.376954+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:04.024848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:04.024848+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"79'34","reported_seq":"95","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.376988+0000","last_change":"2021-02-10T04:27:18.990156+0000","last_active":"2021-02-10T04:27:36.376988+0000","last_peered":"2021-02-10T04:27:36.376988+0000","last_clean":"2021-02-10T04:27:36.376988+0000","last_became_active":"2021-02-10T04:25:49.006147+0000","last_became_peered":"2021-02-10T04:25:49.006147+0000","last_unstale":"2021-02-10T04:27:36.376988+0000","last_undegraded":"2021-02-10T04:27:36.376988+0000","last_fullsized":"2021-02-10T04:27:36.376988+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'34","last_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:27:18.990088+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":22,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"58'2","reported_seq":"62","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.689167+0000","last_change":"2021-02-10T04:27:03.440844+0000","last_active":"2021-02-10T04:27:36.689167+0000","last_peered":"2021-02-10T04:27:36.689167+0000","last_clean":"2021-02-10T04:27:36.689167+0000","last_became_active":"2021-02-10T04:25:51.026251+0000","last_became_peered":"2021-02-10T04:25:51.026251+0000","last_unstale":"2021-02-10T04:27:36.689167+0000","last_undegraded":"2021-02-10T04:27:36.689167+0000","last_fullsized":"2021-02-10T04:27:36.689167+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:03.440780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:03.440780+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"35","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:37.196914+0000","last_change":"2021-02-10T04:27:37.196914+0000","last_active":"2021-02-10T04:27:37.196914+0000","last_peered":"2021-02-10T04:27:37.196914+0000","last_clean":"2021-02-10T04:27:37.196914+0000","last_became_active":"2021-02-10T04:25:53.049487+0000","last_became_peered":"2021-02-10T04:25:53.049487+0000","last_unstale":"2021-02-10T04:27:37.196914+0000","last_undegraded":"2021-02-10T04:27:37.196914+0000","last_fullsized":"2021-02-10T04:27:37.196914+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:37.196846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:37.196846+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"48","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.377691+0000","last_change":"2021-02-10T04:27:06.934347+0000","last_active":"2021-02-10T04:27:36.377691+0000","last_peered":"2021-02-10T04:27:36.377691+0000","last_clean":"2021-02-10T04:27:36.377691+0000","last_became_active":"2021-02-10T04:25:34.331779+0000","last_became_peered":"2021-02-10T04:25:34.331779+0000","last_unstale":"2021-02-10T04:27:36.377691+0000","last_undegraded":"2021-02-10T04:27:36.377691+0000","last_fullsized":"2021-02-10T04:27:36.377691+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:06.934282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:06.934282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"63'31","reported_seq":"103","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.376997+0000","last_change":"2021-02-10T04:26:56.059248+0000","last_active":"2021-02-10T04:27:36.376997+0000","last_peered":"2021-02-10T04:27:36.376997+0000","last_clean":"2021-02-10T04:27:36.376997+0000","last_became_active":"2021-02-10T04:25:49.007872+0000","last_became_peered":"2021-02-10T04:25:49.007872+0000","last_unstale":"2021-02-10T04:27:36.376997+0000","last_undegraded":"2021-02-10T04:27:36.376997+0000","last_fullsized":"2021-02-10T04:27:36.376997+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'31","last_scrub_stamp":"2021-02-10T04:26:56.059186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:26:56.059186+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":66,"num_read_kb":42,"num_write":34,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"38","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:31.650183+0000","last_change":"2021-02-10T04:27:31.650126+0000","last_active":"2021-02-10T04:27:31.650183+0000","last_peered":"2021-02-10T04:27:31.650183+0000","last_clean":"2021-02-10T04:27:31.650183+0000","last_became_active":"2021-02-10T04:25:51.027624+0000","last_became_peered":"2021-02-10T04:25:51.027624+0000","last_unstale":"2021-02-10T04:27:31.650183+0000","last_undegraded":"2021-02-10T04:27:31.650183+0000","last_fullsized":"2021-02-10T04:27:31.650183+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:27:31.650033+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"36","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.749660+0000","last_change":"2021-02-10T04:27:28.264288+0000","last_active":"2021-02-10T04:27:30.749660+0000","last_peered":"2021-02-10T04:27:30.749660+0000","last_clean":"2021-02-10T04:27:30.749660+0000","last_became_active":"2021-02-10T04:25:53.149677+0000","last_became_peered":"2021-02-10T04:25:53.149677+0000","last_unstale":"2021-02-10T04:27:30.749660+0000","last_undegraded":"2021-02-10T04:27:30.749660+0000","last_fullsized":"2021-02-10T04:27:30.749660+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:28.264204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:28.264204+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"155","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.700346+0000","last_change":"2021-02-10T04:26:58.989515+0000","last_active":"2021-02-10T04:27:41.700346+0000","last_peered":"2021-02-10T04:27:41.700346+0000","last_clean":"2021-02-10T04:27:41.700346+0000","last_became_active":"2021-02-10T04:25:49.013969+0000","last_became_peered":"2021-02-10T04:25:49.013969+0000","last_unstale":"2021-02-10T04:27:41.700346+0000","last_undegraded":"2021-02-10T04:27:41.700346+0000","last_fullsized":"2021-02-10T04:27:41.700346+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:26:58.989447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:26:58.989447+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":111,"num_read_kb":106,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"46","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.164411+0000","last_change":"2021-02-10T04:26:42.627320+0000","last_active":"2021-02-10T04:27:30.164411+0000","last_peered":"2021-02-10T04:27:30.164411+0000","last_clean":"2021-02-10T04:27:30.164411+0000","last_became_active":"2021-02-10T04:25:34.331972+0000","last_became_peered":"2021-02-10T04:25:34.331972+0000","last_unstale":"2021-02-10T04:27:30.164411+0000","last_undegraded":"2021-02-10T04:27:30.164411+0000","last_fullsized":"2021-02-10T04:27:30.164411+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:42.627255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:42.627255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"82'29","reported_seq":"95","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:38.176915+0000","last_change":"2021-02-10T04:27:38.176842+0000","last_active":"2021-02-10T04:27:38.176915+0000","last_peered":"2021-02-10T04:27:38.176915+0000","last_clean":"2021-02-10T04:27:38.176915+0000","last_became_active":"2021-02-10T04:25:49.002843+0000","last_became_peered":"2021-02-10T04:25:49.002843+0000","last_unstale":"2021-02-10T04:27:38.176915+0000","last_undegraded":"2021-02-10T04:27:38.176915+0000","last_fullsized":"2021-02-10T04:27:38.176915+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'29","last_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:27:38.176754+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":32,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"37","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.164723+0000","last_change":"2021-02-10T04:27:16.564733+0000","last_active":"2021-02-10T04:27:30.164723+0000","last_peered":"2021-02-10T04:27:30.164723+0000","last_clean":"2021-02-10T04:27:30.164723+0000","last_became_active":"2021-02-10T04:25:51.025306+0000","last_became_peered":"2021-02-10T04:25:51.025306+0000","last_unstale":"2021-02-10T04:27:30.164723+0000","last_undegraded":"2021-02-10T04:27:30.164723+0000","last_fullsized":"2021-02-10T04:27:30.164723+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:27:16.564638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"37","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.378899+0000","last_change":"2021-02-10T04:27:18.115804+0000","last_active":"2021-02-10T04:27:36.378899+0000","last_peered":"2021-02-10T04:27:36.378899+0000","last_clean":"2021-02-10T04:27:36.378899+0000","last_became_active":"2021-02-10T04:25:53.037506+0000","last_became_peered":"2021-02-10T04:25:53.037506+0000","last_unstale":"2021-02-10T04:27:36.378899+0000","last_undegraded":"2021-02-10T04:27:36.378899+0000","last_fullsized":"2021-02-10T04:27:36.378899+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:18.115741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:18.115741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"39","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:39.268187+0000","last_change":"2021-02-10T04:27:39.268187+0000","last_active":"2021-02-10T04:27:39.268187+0000","last_peered":"2021-02-10T04:27:39.268187+0000","last_clean":"2021-02-10T04:27:39.268187+0000","last_became_active":"2021-02-10T04:25:49.010614+0000","last_became_peered":"2021-02-10T04:25:49.010614+0000","last_unstale":"2021-02-10T04:27:39.268187+0000","last_undegraded":"2021-02-10T04:27:39.268187+0000","last_fullsized":"2021-02-10T04:27:39.268187+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:39.268114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:39.268114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"50","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.377265+0000","last_change":"2021-02-10T04:26:38.013836+0000","last_active":"2021-02-10T04:27:36.377265+0000","last_peered":"2021-02-10T04:27:36.377265+0000","last_clean":"2021-02-10T04:27:36.377265+0000","last_became_active":"2021-02-10T04:25:36.334230+0000","last_became_peered":"2021-02-10T04:25:36.334230+0000","last_unstale":"2021-02-10T04:27:36.377265+0000","last_undegraded":"2021-02-10T04:27:36.377265+0000","last_fullsized":"2021-02-10T04:27:36.377265+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:38.013770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:38.013770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"46","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748194+0000","last_change":"2021-02-10T04:26:44.252455+0000","last_active":"2021-02-10T04:27:30.748194+0000","last_peered":"2021-02-10T04:27:30.748194+0000","last_clean":"2021-02-10T04:27:30.748194+0000","last_became_active":"2021-02-10T04:25:36.343397+0000","last_became_peered":"2021-02-10T04:25:36.343397+0000","last_unstale":"2021-02-10T04:27:30.748194+0000","last_undegraded":"2021-02-10T04:27:30.748194+0000","last_fullsized":"2021-02-10T04:27:30.748194+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:44.252351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:26:44.252351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"62'15","reported_seq":"64","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.272524+0000","last_change":"2021-02-10T04:27:08.069986+0000","last_active":"2021-02-10T04:27:30.272524+0000","last_peered":"2021-02-10T04:27:30.272524+0000","last_clean":"2021-02-10T04:27:30.272524+0000","last_became_active":"2021-02-10T04:25:49.002471+0000","last_became_peered":"2021-02-10T04:25:49.002471+0000","last_unstale":"2021-02-10T04:27:30.272524+0000","last_undegraded":"2021-02-10T04:27:30.272524+0000","last_fullsized":"2021-02-10T04:27:30.272524+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:27:08.069899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:08.069899+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":15,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"38","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:38.106291+0000","last_change":"2021-02-10T04:27:38.106291+0000","last_active":"2021-02-10T04:27:38.106291+0000","last_peered":"2021-02-10T04:27:38.106291+0000","last_clean":"2021-02-10T04:27:38.106291+0000","last_became_active":"2021-02-10T04:25:51.022821+0000","last_became_peered":"2021-02-10T04:25:51.022821+0000","last_unstale":"2021-02-10T04:27:38.106291+0000","last_undegraded":"2021-02-10T04:27:38.106291+0000","last_fullsized":"2021-02-10T04:27:38.106291+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:38.106209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:38.106209+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"37","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.376837+0000","last_change":"2021-02-10T04:27:10.051857+0000","last_active":"2021-02-10T04:27:36.376837+0000","last_peered":"2021-02-10T04:27:36.376837+0000","last_clean":"2021-02-10T04:27:36.376837+0000","last_became_active":"2021-02-10T04:25:53.040390+0000","last_became_peered":"2021-02-10T04:25:53.040390+0000","last_unstale":"2021-02-10T04:27:36.376837+0000","last_undegraded":"2021-02-10T04:27:36.376837+0000","last_fullsized":"2021-02-10T04:27:36.376837+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:10.051800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:10.051800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"40","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.657900+0000","last_change":"2021-02-10T04:27:30.657844+0000","last_active":"2021-02-10T04:27:30.657900+0000","last_peered":"2021-02-10T04:27:30.657900+0000","last_clean":"2021-02-10T04:27:30.657900+0000","last_became_active":"2021-02-10T04:25:49.001782+0000","last_became_peered":"2021-02-10T04:25:49.001782+0000","last_unstale":"2021-02-10T04:27:30.657900+0000","last_undegraded":"2021-02-10T04:27:30.657900+0000","last_fullsized":"2021-02-10T04:27:30.657900+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:30.657781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:30.657781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"45","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748149+0000","last_change":"2021-02-10T04:27:08.192835+0000","last_active":"2021-02-10T04:27:30.748149+0000","last_peered":"2021-02-10T04:27:30.748149+0000","last_clean":"2021-02-10T04:27:30.748149+0000","last_became_active":"2021-02-10T04:25:36.340009+0000","last_became_peered":"2021-02-10T04:25:36.340009+0000","last_unstale":"2021-02-10T04:27:30.748149+0000","last_undegraded":"2021-02-10T04:27:30.748149+0000","last_fullsized":"2021-02-10T04:27:30.748149+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:08.192772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:08.192772+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"62'16","reported_seq":"66","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.057433+0000","last_change":"2021-02-10T04:27:09.484054+0000","last_active":"2021-02-10T04:27:30.057433+0000","last_peered":"2021-02-10T04:27:30.057433+0000","last_clean":"2021-02-10T04:27:30.057433+0000","last_became_active":"2021-02-10T04:25:49.003521+0000","last_became_peered":"2021-02-10T04:25:49.003521+0000","last_unstale":"2021-02-10T04:27:30.057433+0000","last_undegraded":"2021-02-10T04:27:30.057433+0000","last_fullsized":"2021-02-10T04:27:30.057433+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'16","last_scrub_stamp":"2021-02-10T04:27:09.483988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:09.483988+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":16,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"39","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.378881+0000","last_change":"2021-02-10T04:27:25.216335+0000","last_active":"2021-02-10T04:27:36.378881+0000","last_peered":"2021-02-10T04:27:36.378881+0000","last_clean":"2021-02-10T04:27:36.378881+0000","last_became_active":"2021-02-10T04:25:51.032201+0000","last_became_peered":"2021-02-10T04:25:51.032201+0000","last_unstale":"2021-02-10T04:27:36.378881+0000","last_undegraded":"2021-02-10T04:27:36.378881+0000","last_fullsized":"2021-02-10T04:27:36.378881+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:27:25.216260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"37","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:42.182019+0000","last_change":"2021-02-10T04:27:42.182019+0000","last_active":"2021-02-10T04:27:42.182019+0000","last_peered":"2021-02-10T04:27:42.182019+0000","last_clean":"2021-02-10T04:27:42.182019+0000","last_became_active":"2021-02-10T04:25:53.149552+0000","last_became_peered":"2021-02-10T04:25:53.149552+0000","last_unstale":"2021-02-10T04:27:42.182019+0000","last_undegraded":"2021-02-10T04:27:42.182019+0000","last_fullsized":"2021-02-10T04:27:42.182019+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:42.181940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:42.181940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"48","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.749704+0000","last_change":"2021-02-10T04:26:57.084942+0000","last_active":"2021-02-10T04:27:30.749704+0000","last_peered":"2021-02-10T04:27:30.749704+0000","last_clean":"2021-02-10T04:27:30.749704+0000","last_became_active":"2021-02-10T04:25:36.338600+0000","last_became_peered":"2021-02-10T04:25:36.338600+0000","last_unstale":"2021-02-10T04:27:30.749704+0000","last_undegraded":"2021-02-10T04:27:30.749704+0000","last_fullsized":"2021-02-10T04:27:30.749704+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:26:57.084864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:57.084864+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"63'27","reported_seq":"95","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.057422+0000","last_change":"2021-02-10T04:27:10.457048+0000","last_active":"2021-02-10T04:27:30.057422+0000","last_peered":"2021-02-10T04:27:30.057422+0000","last_clean":"2021-02-10T04:27:30.057422+0000","last_became_active":"2021-02-10T04:25:49.002937+0000","last_became_peered":"2021-02-10T04:25:49.002937+0000","last_unstale":"2021-02-10T04:27:30.057422+0000","last_undegraded":"2021-02-10T04:27:30.057422+0000","last_fullsized":"2021-02-10T04:27:30.057422+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'27","last_scrub_stamp":"2021-02-10T04:27:10.456981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:10.456981+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":38,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"38","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.272269+0000","last_change":"2021-02-10T04:26:59.175043+0000","last_active":"2021-02-10T04:27:30.272269+0000","last_peered":"2021-02-10T04:27:30.272269+0000","last_clean":"2021-02-10T04:27:30.272269+0000","last_became_active":"2021-02-10T04:25:51.025636+0000","last_became_peered":"2021-02-10T04:25:51.025636+0000","last_unstale":"2021-02-10T04:27:30.272269+0000","last_undegraded":"2021-02-10T04:27:30.272269+0000","last_fullsized":"2021-02-10T04:27:30.272269+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:59.174908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:26:59.174908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"37","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.078388+0000","last_change":"2021-02-10T04:27:01.184729+0000","last_active":"2021-02-10T04:27:36.078388+0000","last_peered":"2021-02-10T04:27:36.078388+0000","last_clean":"2021-02-10T04:27:36.078388+0000","last_became_active":"2021-02-10T04:25:53.045085+0000","last_became_peered":"2021-02-10T04:25:53.045085+0000","last_unstale":"2021-02-10T04:27:36.078388+0000","last_undegraded":"2021-02-10T04:27:36.078388+0000","last_fullsized":"2021-02-10T04:27:36.078388+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:01.184630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:01.184630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"52","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.377483+0000","last_change":"2021-02-10T04:27:25.026606+0000","last_active":"2021-02-10T04:27:36.377483+0000","last_peered":"2021-02-10T04:27:36.377483+0000","last_clean":"2021-02-10T04:27:36.377483+0000","last_became_active":"2021-02-10T04:25:35.324960+0000","last_became_peered":"2021-02-10T04:25:35.324960+0000","last_unstale":"2021-02-10T04:27:36.377483+0000","last_undegraded":"2021-02-10T04:27:36.377483+0000","last_fullsized":"2021-02-10T04:27:36.377483+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:27:25.026534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:25.026534+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"63'34","reported_seq":"109","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.055150+0000","last_change":"2021-02-10T04:27:20.342445+0000","last_active":"2021-02-10T04:27:30.055150+0000","last_peered":"2021-02-10T04:27:30.055150+0000","last_clean":"2021-02-10T04:27:30.055150+0000","last_became_active":"2021-02-10T04:25:48.997481+0000","last_became_peered":"2021-02-10T04:25:48.997481+0000","last_unstale":"2021-02-10T04:27:30.055150+0000","last_undegraded":"2021-02-10T04:27:30.055150+0000","last_fullsized":"2021-02-10T04:27:30.055150+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'34","last_scrub_stamp":"2021-02-10T04:27:20.342358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:20.342358+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":70,"num_read_kb":48,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"34","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.078114+0000","last_change":"2021-02-10T04:25:51.019067+0000","last_active":"2021-02-10T04:27:36.078114+0000","last_peered":"2021-02-10T04:27:36.078114+0000","last_clean":"2021-02-10T04:27:36.078114+0000","last_became_active":"2021-02-10T04:25:51.018706+0000","last_became_peered":"2021-02-10T04:25:51.018706+0000","last_unstale":"2021-02-10T04:27:36.078114+0000","last_undegraded":"2021-02-10T04:27:36.078114+0000","last_fullsized":"2021-02-10T04:27:36.078114+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:25:49.999907+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"36","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748932+0000","last_change":"2021-02-10T04:27:26.234432+0000","last_active":"2021-02-10T04:27:30.748932+0000","last_peered":"2021-02-10T04:27:30.748932+0000","last_clean":"2021-02-10T04:27:30.748932+0000","last_became_active":"2021-02-10T04:25:53.150930+0000","last_became_peered":"2021-02-10T04:25:53.150930+0000","last_unstale":"2021-02-10T04:27:30.748932+0000","last_undegraded":"2021-02-10T04:27:30.748932+0000","last_fullsized":"2021-02-10T04:27:30.748932+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:26.234349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:26.234349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"56","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.272767+0000","last_change":"2021-02-10T04:26:44.264500+0000","last_active":"2021-02-10T04:27:30.272767+0000","last_peered":"2021-02-10T04:27:30.272767+0000","last_clean":"2021-02-10T04:27:30.272767+0000","last_became_active":"2021-02-10T04:25:36.339614+0000","last_became_peered":"2021-02-10T04:25:36.339614+0000","last_unstale":"2021-02-10T04:27:30.272767+0000","last_undegraded":"2021-02-10T04:27:30.272767+0000","last_fullsized":"2021-02-10T04:27:30.272767+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:44.264402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:44.264402+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"63'26","reported_seq":"89","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.077934+0000","last_change":"2021-02-10T04:27:26.403830+0000","last_active":"2021-02-10T04:27:36.077934+0000","last_peered":"2021-02-10T04:27:36.077934+0000","last_clean":"2021-02-10T04:27:36.077934+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:27:36.077934+0000","last_undegraded":"2021-02-10T04:27:36.077934+0000","last_fullsized":"2021-02-10T04:27:36.077934+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:27:26.403748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:26.403748+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":51,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"58'2","reported_seq":"61","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.078210+0000","last_change":"2021-02-10T04:27:33.210495+0000","last_active":"2021-02-10T04:27:36.078210+0000","last_peered":"2021-02-10T04:27:36.078210+0000","last_clean":"2021-02-10T04:27:36.078210+0000","last_became_active":"2021-02-10T04:25:51.031906+0000","last_became_peered":"2021-02-10T04:25:51.031906+0000","last_unstale":"2021-02-10T04:27:36.078210+0000","last_undegraded":"2021-02-10T04:27:36.078210+0000","last_fullsized":"2021-02-10T04:27:36.078210+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:33.210431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:33.210431+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"30","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.055551+0000","last_change":"2021-02-10T04:25:53.148449+0000","last_active":"2021-02-10T04:27:30.055551+0000","last_peered":"2021-02-10T04:27:30.055551+0000","last_clean":"2021-02-10T04:27:30.055551+0000","last_became_active":"2021-02-10T04:25:53.148054+0000","last_became_peered":"2021-02-10T04:25:53.148054+0000","last_unstale":"2021-02-10T04:27:30.055551+0000","last_undegraded":"2021-02-10T04:27:30.055551+0000","last_fullsized":"2021-02-10T04:27:30.055551+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:25:52.015519+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"39","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.378788+0000","last_change":"2021-02-10T04:26:58.112766+0000","last_active":"2021-02-10T04:27:36.378788+0000","last_peered":"2021-02-10T04:27:36.378788+0000","last_clean":"2021-02-10T04:27:36.378788+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:27:36.378788+0000","last_undegraded":"2021-02-10T04:27:36.378788+0000","last_fullsized":"2021-02-10T04:27:36.378788+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:58.112683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:26:58.112683+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"45","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.748100+0000","last_change":"2021-02-10T04:27:04.148397+0000","last_active":"2021-02-10T04:27:30.748100+0000","last_peered":"2021-02-10T04:27:30.748100+0000","last_clean":"2021-02-10T04:27:30.748100+0000","last_became_active":"2021-02-10T04:25:36.340895+0000","last_became_peered":"2021-02-10T04:25:36.340895+0000","last_unstale":"2021-02-10T04:27:30.748100+0000","last_undegraded":"2021-02-10T04:27:30.748100+0000","last_fullsized":"2021-02-10T04:27:30.748100+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:04.148329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:04.148329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"63'21","reported_seq":"83","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.078777+0000","last_change":"2021-02-10T04:26:59.205120+0000","last_active":"2021-02-10T04:27:36.078777+0000","last_peered":"2021-02-10T04:27:36.078777+0000","last_clean":"2021-02-10T04:27:36.078777+0000","last_became_active":"2021-02-10T04:25:49.011093+0000","last_became_peered":"2021-02-10T04:25:49.011093+0000","last_unstale":"2021-02-10T04:27:36.078777+0000","last_undegraded":"2021-02-10T04:27:36.078777+0000","last_fullsized":"2021-02-10T04:27:36.078777+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2021-02-10T04:26:59.205037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:26:59.205037+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":28,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"37","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.378115+0000","last_change":"2021-02-10T04:27:24.209544+0000","last_active":"2021-02-10T04:27:36.378115+0000","last_peered":"2021-02-10T04:27:36.378115+0000","last_clean":"2021-02-10T04:27:36.378115+0000","last_became_active":"2021-02-10T04:25:53.042697+0000","last_became_peered":"2021-02-10T04:25:53.042697+0000","last_unstale":"2021-02-10T04:27:36.378115+0000","last_undegraded":"2021-02-10T04:27:36.378115+0000","last_fullsized":"2021-02-10T04:27:36.378115+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:27:24.209460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"78'3","reported_seq":"64","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:37.723400+0000","last_change":"2021-02-10T04:27:13.268467+0000","last_active":"2021-02-10T04:27:37.723400+0000","last_peered":"2021-02-10T04:27:37.723400+0000","last_clean":"2021-02-10T04:27:37.723400+0000","last_became_active":"2021-02-10T04:25:51.025218+0000","last_became_peered":"2021-02-10T04:25:51.025218+0000","last_unstale":"2021-02-10T04:27:37.723400+0000","last_undegraded":"2021-02-10T04:27:37.723400+0000","last_fullsized":"2021-02-10T04:27:37.723400+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:27:13.268412+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"52","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.376235+0000","last_change":"2021-02-10T04:26:46.092082+0000","last_active":"2021-02-10T04:27:36.376235+0000","last_peered":"2021-02-10T04:27:36.376235+0000","last_clean":"2021-02-10T04:27:36.376235+0000","last_became_active":"2021-02-10T04:25:36.331437+0000","last_became_peered":"2021-02-10T04:25:36.331437+0000","last_unstale":"2021-02-10T04:27:36.376235+0000","last_undegraded":"2021-02-10T04:27:36.376235+0000","last_fullsized":"2021-02-10T04:27:36.376235+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:46.092013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:46.092013+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"62'17","reported_seq":"70","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.377536+0000","last_change":"2021-02-10T04:27:30.012988+0000","last_active":"2021-02-10T04:27:36.377536+0000","last_peered":"2021-02-10T04:27:36.377536+0000","last_clean":"2021-02-10T04:27:36.377536+0000","last_became_active":"2021-02-10T04:25:49.004933+0000","last_became_peered":"2021-02-10T04:25:49.004933+0000","last_unstale":"2021-02-10T04:27:36.377536+0000","last_undegraded":"2021-02-10T04:27:36.377536+0000","last_fullsized":"2021-02-10T04:27:36.377536+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'17","last_scrub_stamp":"2021-02-10T04:27:30.012927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:30.012927+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"37","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.078569+0000","last_change":"2021-02-10T04:27:33.382403+0000","last_active":"2021-02-10T04:27:36.078569+0000","last_peered":"2021-02-10T04:27:36.078569+0000","last_clean":"2021-02-10T04:27:36.078569+0000","last_became_active":"2021-02-10T04:25:53.043027+0000","last_became_peered":"2021-02-10T04:25:53.043027+0000","last_unstale":"2021-02-10T04:27:36.078569+0000","last_undegraded":"2021-02-10T04:27:36.078569+0000","last_fullsized":"2021-02-10T04:27:36.078569+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:33.382319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:33.382319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"58'2","reported_seq":"61","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.077333+0000","last_change":"2021-02-10T04:26:50.624141+0000","last_active":"2021-02-10T04:27:36.077333+0000","last_peered":"2021-02-10T04:27:36.077333+0000","last_clean":"2021-02-10T04:27:36.077333+0000","last_became_active":"2021-02-10T04:25:51.025268+0000","last_became_peered":"2021-02-10T04:25:51.025268+0000","last_unstale":"2021-02-10T04:27:36.077333+0000","last_undegraded":"2021-02-10T04:27:36.077333+0000","last_fullsized":"2021-02-10T04:27:36.077333+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:26:50.624072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:26:50.624072+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"50","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.376246+0000","last_change":"2021-02-10T04:27:17.003396+0000","last_active":"2021-02-10T04:27:36.376246+0000","last_peered":"2021-02-10T04:27:36.376246+0000","last_clean":"2021-02-10T04:27:36.376246+0000","last_became_active":"2021-02-10T04:25:36.333883+0000","last_became_peered":"2021-02-10T04:25:36.333883+0000","last_unstale":"2021-02-10T04:27:36.376246+0000","last_undegraded":"2021-02-10T04:27:36.376246+0000","last_fullsized":"2021-02-10T04:27:36.376246+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:17.003304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:17.003304+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"62'13","reported_seq":"62","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.078308+0000","last_change":"2021-02-10T04:27:14.256603+0000","last_active":"2021-02-10T04:27:36.078308+0000","last_peered":"2021-02-10T04:27:36.078308+0000","last_clean":"2021-02-10T04:27:36.078308+0000","last_became_active":"2021-02-10T04:25:49.001725+0000","last_became_peered":"2021-02-10T04:25:49.001725+0000","last_unstale":"2021-02-10T04:27:36.078308+0000","last_undegraded":"2021-02-10T04:27:36.078308+0000","last_fullsized":"2021-02-10T04:27:36.078308+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'13","last_scrub_stamp":"2021-02-10T04:27:14.256539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:14.256539+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"35","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.164810+0000","last_change":"2021-02-10T04:26:55.627032+0000","last_active":"2021-02-10T04:27:30.164810+0000","last_peered":"2021-02-10T04:27:30.164810+0000","last_clean":"2021-02-10T04:27:30.164810+0000","last_became_active":"2021-02-10T04:25:53.038218+0000","last_became_peered":"2021-02-10T04:25:53.038218+0000","last_unstale":"2021-02-10T04:27:30.164810+0000","last_undegraded":"2021-02-10T04:27:30.164810+0000","last_fullsized":"2021-02-10T04:27:30.164810+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:55.626967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:26:55.626967+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"39","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:37.160464+0000","last_change":"2021-02-10T04:27:37.160395+0000","last_active":"2021-02-10T04:27:37.160464+0000","last_peered":"2021-02-10T04:27:37.160464+0000","last_clean":"2021-02-10T04:27:37.160464+0000","last_became_active":"2021-02-10T04:25:51.028302+0000","last_became_peered":"2021-02-10T04:25:51.028302+0000","last_unstale":"2021-02-10T04:27:37.160464+0000","last_undegraded":"2021-02-10T04:27:37.160464+0000","last_fullsized":"2021-02-10T04:27:37.160464+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:37.160328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:37.160328+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"45","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.749054+0000","last_change":"2021-02-10T04:27:00.052115+0000","last_active":"2021-02-10T04:27:30.749054+0000","last_peered":"2021-02-10T04:27:30.749054+0000","last_clean":"2021-02-10T04:27:30.749054+0000","last_became_active":"2021-02-10T04:25:36.344109+0000","last_became_peered":"2021-02-10T04:25:36.344109+0000","last_unstale":"2021-02-10T04:27:30.749054+0000","last_undegraded":"2021-02-10T04:27:30.749054+0000","last_fullsized":"2021-02-10T04:27:30.749054+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:00.052028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:00.052028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"62'21","reported_seq":"77","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.078564+0000","last_change":"2021-02-10T04:27:00.215115+0000","last_active":"2021-02-10T04:27:36.078564+0000","last_peered":"2021-02-10T04:27:36.078564+0000","last_clean":"2021-02-10T04:27:36.078564+0000","last_became_active":"2021-02-10T04:25:49.011699+0000","last_became_peered":"2021-02-10T04:25:49.011699+0000","last_unstale":"2021-02-10T04:27:36.078564+0000","last_undegraded":"2021-02-10T04:27:36.078564+0000","last_fullsized":"2021-02-10T04:27:36.078564+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'21","last_scrub_stamp":"2021-02-10T04:27:00.215037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:00.215037+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"36","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:35.166388+0000","last_change":"2021-02-10T04:27:35.166330+0000","last_active":"2021-02-10T04:27:35.166388+0000","last_peered":"2021-02-10T04:27:35.166388+0000","last_clean":"2021-02-10T04:27:35.166388+0000","last_became_active":"2021-02-10T04:25:53.037903+0000","last_became_peered":"2021-02-10T04:25:53.037903+0000","last_unstale":"2021-02-10T04:27:35.166388+0000","last_undegraded":"2021-02-10T04:27:35.166388+0000","last_fullsized":"2021-02-10T04:27:35.166388+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:35.166250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:35.166250+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"37","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.164601+0000","last_change":"2021-02-10T04:27:11.606664+0000","last_active":"2021-02-10T04:27:30.164601+0000","last_peered":"2021-02-10T04:27:30.164601+0000","last_clean":"2021-02-10T04:27:30.164601+0000","last_became_active":"2021-02-10T04:25:51.025882+0000","last_became_peered":"2021-02-10T04:25:51.025882+0000","last_unstale":"2021-02-10T04:27:30.164601+0000","last_undegraded":"2021-02-10T04:27:30.164601+0000","last_fullsized":"2021-02-10T04:27:30.164601+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:11.606603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:11.606603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"62'2","reported_seq":"69","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.077782+0000","last_change":"2021-02-10T04:26:48.599352+0000","last_active":"2021-02-10T04:27:36.077782+0000","last_peered":"2021-02-10T04:27:36.077782+0000","last_clean":"2021-02-10T04:27:36.077782+0000","last_became_active":"2021-02-10T04:25:34.331199+0000","last_became_peered":"2021-02-10T04:25:34.331199+0000","last_unstale":"2021-02-10T04:27:36.077782+0000","last_undegraded":"2021-02-10T04:27:36.077782+0000","last_fullsized":"2021-02-10T04:27:36.077782+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2021-02-10T04:26:48.599257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:48.599257+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"63'29","reported_seq":"96","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.272144+0000","last_change":"2021-02-10T04:27:13.081465+0000","last_active":"2021-02-10T04:27:30.272144+0000","last_peered":"2021-02-10T04:27:30.272144+0000","last_clean":"2021-02-10T04:27:30.272144+0000","last_became_active":"2021-02-10T04:25:49.015913+0000","last_became_peered":"2021-02-10T04:25:49.015913+0000","last_unstale":"2021-02-10T04:27:30.272144+0000","last_undegraded":"2021-02-10T04:27:30.272144+0000","last_fullsized":"2021-02-10T04:27:30.272144+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'29","last_scrub_stamp":"2021-02-10T04:27:13.081399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:13.081399+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":56,"num_read_kb":36,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"36","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.057020+0000","last_change":"2021-02-10T04:27:22.512316+0000","last_active":"2021-02-10T04:27:30.057020+0000","last_peered":"2021-02-10T04:27:30.057020+0000","last_clean":"2021-02-10T04:27:30.057020+0000","last_became_active":"2021-02-10T04:25:53.038249+0000","last_became_peered":"2021-02-10T04:25:53.038249+0000","last_unstale":"2021-02-10T04:27:30.057020+0000","last_undegraded":"2021-02-10T04:27:30.057020+0000","last_fullsized":"2021-02-10T04:27:30.057020+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:22.512245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:22.512245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"37","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.164529+0000","last_change":"2021-02-10T04:26:51.605291+0000","last_active":"2021-02-10T04:27:30.164529+0000","last_peered":"2021-02-10T04:27:30.164529+0000","last_clean":"2021-02-10T04:27:30.164529+0000","last_became_active":"2021-02-10T04:25:51.016118+0000","last_became_peered":"2021-02-10T04:25:51.016118+0000","last_unstale":"2021-02-10T04:27:30.164529+0000","last_undegraded":"2021-02-10T04:27:30.164529+0000","last_fullsized":"2021-02-10T04:27:30.164529+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:51.605227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:26:51.605227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"48","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.377225+0000","last_change":"2021-02-10T04:26:50.001018+0000","last_active":"2021-02-10T04:27:36.377225+0000","last_peered":"2021-02-10T04:27:36.377225+0000","last_clean":"2021-02-10T04:27:36.377225+0000","last_became_active":"2021-02-10T04:25:34.334427+0000","last_became_peered":"2021-02-10T04:25:34.334427+0000","last_unstale":"2021-02-10T04:27:36.377225+0000","last_undegraded":"2021-02-10T04:27:36.377225+0000","last_fullsized":"2021-02-10T04:27:36.377225+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:50.000952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:50.000952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"63'20","reported_seq":"81","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.057517+0000","last_change":"2021-02-10T04:27:20.587671+0000","last_active":"2021-02-10T04:27:30.057517+0000","last_peered":"2021-02-10T04:27:30.057517+0000","last_clean":"2021-02-10T04:27:30.057517+0000","last_became_active":"2021-02-10T04:25:49.004346+0000","last_became_peered":"2021-02-10T04:25:49.004346+0000","last_unstale":"2021-02-10T04:27:30.057517+0000","last_undegraded":"2021-02-10T04:27:30.057517+0000","last_fullsized":"2021-02-10T04:27:30.057517+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'20","last_scrub_stamp":"2021-02-10T04:27:20.587602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:20.587602+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":27,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"37","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.377673+0000","last_change":"2021-02-10T04:27:13.006848+0000","last_active":"2021-02-10T04:27:36.377673+0000","last_peered":"2021-02-10T04:27:36.377673+0000","last_clean":"2021-02-10T04:27:36.377673+0000","last_became_active":"2021-02-10T04:25:53.148090+0000","last_became_peered":"2021-02-10T04:25:53.148090+0000","last_unstale":"2021-02-10T04:27:36.377673+0000","last_undegraded":"2021-02-10T04:27:36.377673+0000","last_fullsized":"2021-02-10T04:27:36.377673+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:27:13.006716+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"60","reported_epoch":"81","state":"active+clean","last_fresh":"2021-02-10T04:27:30.164839+0000","last_change":"2021-02-10T04:26:44.609990+0000","last_active":"2021-02-10T04:27:30.164839+0000","last_peered":"2021-02-10T04:27:30.164839+0000","last_clean":"2021-02-10T04:27:30.164839+0000","last_became_active":"2021-02-10T04:25:36.344641+0000","last_became_peered":"2021-02-10T04:25:36.344641+0000","last_unstale":"2021-02-10T04:27:30.164839+0000","last_undegraded":"2021-02-10T04:27:30.164839+0000","last_fullsized":"2021-02-10T04:27:30.164839+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:26:44.609922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:44.609922+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"63'30","reported_seq":"102","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.377476+0000","last_change":"2021-02-10T04:27:02.039902+0000","last_active":"2021-02-10T04:27:36.377476+0000","last_peered":"2021-02-10T04:27:36.377476+0000","last_clean":"2021-02-10T04:27:36.377476+0000","last_became_active":"2021-02-10T04:25:49.002058+0000","last_became_peered":"2021-02-10T04:25:49.002058+0000","last_unstale":"2021-02-10T04:27:36.377476+0000","last_undegraded":"2021-02-10T04:27:36.377476+0000","last_fullsized":"2021-02-10T04:27:36.377476+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2021-02-10T04:27:02.039813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:02.039813+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":59,"num_read_kb":42,"num_write":31,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"58'2","reported_seq":"62","reported_epoch":"82","state":"active+clean","last_fresh":"2021-02-10T04:27:36.378543+0000","last_change":"2021-02-10T04:27:20.144947+0000","last_active":"2021-02-10T04:27:36.378543+0000","last_peered":"2021-02-10T04:27:36.378543+0000","last_clean":"2021-02-10T04:27:36.378543+0000","last_became_active":"2021-02-10T04:25:51.019515+0000","last_became_peered":"2021-02-10T04:25:51.019515+0000","last_unstale":"2021-02-10T04:27:36.378543+0000","last_undegraded":"2021-02-10T04:27:36.378543+0000","last_fullsized":"2021-02-10T04:27:36.378543+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:20.144855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:20.144855+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":1453,"num_read_kb":926,"num_write":734,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":746,"ondisk_log_size":746,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":111,"num_read_kb":106,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1945,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":21,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":5835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748364827,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6540,"kb_used_data":836,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738612,"statfs":{"total":95995035648,"available":95988338688,"internally_reserved":0,"allocated":856064,"data_stored":300940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.876,"15min":0.88200000000000001},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":1.0529999999999999},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.85599999999999998,"15min":0.85899999999999999},"min":{"1min":0.57499999999999996,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.248,"5min":1.248,"15min":1.248},"last":1.1080000000000001}]},{"osd":1,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.88700000000000001,"15min":0.89100000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":1.139},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.0149999999999999,"15min":1.0209999999999999},"min":{"1min":0.64500000000000002,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.355,"5min":1.369,"15min":1.369},"last":1.173}]},{"osd":2,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.018,"15min":1.026},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":1.292},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.95799999999999996,"15min":0.95999999999999996},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.95699999999999996}]},{"osd":3,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.998,"15min":1.01},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.147,"15min":1.153},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.381,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":1.3779999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":0.97499999999999998,"15min":0.95099999999999996},"min":{"1min":0.77300000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.2530000000000001,"5min":2.2530000000000001,"15min":2.2530000000000001},"last":1.427},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.1060000000000001,"15min":1.1120000000000001},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.492,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":0.876}]},{"osd":5,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.161,"15min":1.1499999999999999},"min":{"1min":0.68899999999999995,"5min":0.624,"15min":0.624},"max":{"1min":1.714,"5min":1.714,"15min":1.714},"last":0.92600000000000005},{"interface":"front","average":{"1min":1.208,"5min":1.2,"15min":1.1990000000000001},"min":{"1min":0.90900000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.498,"5min":1.538,"15min":1.538},"last":1.2410000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.246,"15min":1.2430000000000001},"min":{"1min":0.94199999999999995,"5min":0.86899999999999999,"15min":0.86899999999999999},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.276,"5min":1.236,"15min":1.2290000000000001},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":1.268}]}]},{"osd":6,"up_from":43,"seq":184683593759,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6584,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738568,"statfs":{"total":95995035648,"available":95988293632,"internally_reserved":0,"allocated":901120,"data_stored":303810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:18 2021","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.3460000000000001,"5min":1.478,"15min":1.478},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.95499999999999996,"5min":1.008,"15min":1.0169999999999999},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.77000000000000002}]},{"osd":1,"last update":"Wed Feb 10 04:27:18 2021","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":1.0149999999999999,"15min":1.03},"min":{"1min":0.61599999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":2.121,"5min":2.121,"15min":2.121},"last":0.71799999999999997},{"interface":"front","average":{"1min":1.042,"5min":0.99099999999999999,"15min":0.98299999999999998},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":1.0409999999999999}]},{"osd":2,"last update":"Wed Feb 10 04:27:18 2021","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0620000000000001,"15min":1.0609999999999999},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.0289999999999999,"15min":1.014},"min":{"1min":0.73699999999999999,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.1920000000000002,"5min":2.1920000000000002,"15min":2.1920000000000002},"last":0.93300000000000005}]},{"osd":3,"last update":"Wed Feb 10 04:27:18 2021","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.1439999999999999,"15min":1.1379999999999999},"min":{"1min":0.85699999999999998,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":2,"5min":2,"15min":2},"last":0.91000000000000003},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.133,"15min":1.107},"min":{"1min":0.88,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":2.323,"5min":2.323,"15min":2.323},"last":1.0720000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:27:18 2021","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":0.99099999999999999,"15min":0.96099999999999997},"min":{"1min":0.61699999999999999,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.0800000000000001,"5min":2.0800000000000001,"15min":2.0800000000000001},"last":1.1830000000000001},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.90500000000000003,"15min":0.89200000000000002},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.997}]},{"osd":5,"last update":"Wed Feb 10 04:27:18 2021","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.0009999999999999,"15min":0.98699999999999999},"min":{"1min":0.67100000000000004,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.7130000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.419,"5min":1.179,"15min":1.139},"min":{"1min":0.83999999999999997,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.4550000000000001,"5min":2.4550000000000001,"15min":2.4550000000000001},"last":1.2170000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:26:37 2021","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.105},{"interface":"front","average":{"1min":1.385,"5min":1.385,"15min":1.385},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":1.925,"5min":1.925,"15min":1.925},"last":1.3}]}]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":11292,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":10304,"kb_avail":93733860,"statfs":{"total":95995035648,"available":95983472640,"internally_reserved":0,"allocated":1003520,"data_stored":313080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":10551296},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:26:53 2021","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.66600000000000004,"15min":0.626},"min":{"1min":0.56899999999999995,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.425,"5min":1.425,"15min":1.425},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.746,"15min":0.69199999999999995},"min":{"1min":0.57299999999999995,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.79700000000000004}]},{"osd":2,"last update":"Wed Feb 10 04:27:14 2021","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.80800000000000005,"15min":0.76400000000000001},"min":{"1min":0.58199999999999996,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.83499999999999996,"15min":0.79900000000000004},"min":{"1min":0.51000000000000001,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":0.70099999999999996}]},{"osd":3,"last update":"Wed Feb 10 04:27:31 2021","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.93100000000000005,"15min":0.91400000000000003},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.5860000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":1.026,"5min":1.018,"15min":1.014},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.226,"5min":2.226,"15min":2.226},"last":0.86899999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:26:45 2021","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.1599999999999999,"15min":1.147},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.004,"5min":2.004,"15min":2.004},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.2130000000000001,"15min":1.2090000000000001},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":2.2919999999999998,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":1.0740000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:27:08 2021","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.25,"15min":1.2769999999999999},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":2.7170000000000001,"5min":5.5860000000000003,"15min":5.5860000000000003},"last":0.755},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.6100000000000001,"15min":1.6559999999999999},"min":{"1min":0.97399999999999998,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":2.105,"5min":9.5310000000000006,"15min":9.5310000000000006},"last":1.0549999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:27:24 2021","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.2889999999999999,"15min":1.29},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.4449999999999998,"5min":2.4449999999999998,"15min":2.4449999999999998},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.264,"15min":1.292},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.905,"5min":3.653,"15min":3.653},"last":0.996}]},{"osd":7,"last update":"Wed Feb 10 04:27:36 2021","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.639,"15min":1.6890000000000001},"min":{"1min":1.0529999999999999,"5min":1.004,"15min":1.004},"max":{"1min":2.2090000000000001,"5min":7.5540000000000003,"15min":7.5540000000000003},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.623,"15min":1.6619999999999999},"min":{"1min":1.006,"5min":1.006,"15min":1.006},"max":{"1min":2.556,"5min":8.1020000000000003,"15min":8.1020000000000003},"last":1.1659999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738422,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6784,"kb_used_data":824,"kb_used_omap":0,"kb_used_meta":5952,"kb_avail":93738368,"statfs":{"total":95995035648,"available":95988088832,"internally_reserved":0,"allocated":843776,"data_stored":300649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6094848},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 04:26:53 2021","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.64300000000000002,"15min":0.58699999999999997},"min":{"1min":0.53100000000000003,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.006,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.79100000000000004,"15min":0.73299999999999998},"min":{"1min":0.67400000000000004,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":1.0740000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:27:12 2021","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.79300000000000004,"15min":0.77300000000000002},"min":{"1min":0.52500000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.958,"5min":1.958,"15min":1.958},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.79700000000000004,"15min":0.79300000000000004},"min":{"1min":0.56100000000000005,"5min":0.495,"15min":0.495},"max":{"1min":1.4390000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.97799999999999998}]},{"osd":3,"last update":"Wed Feb 10 04:27:27 2021","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.80400000000000005,"15min":0.81200000000000006},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.224,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.89500000000000002,"5min":1.026,"15min":1.036},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.3049999999999999,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.59099999999999997}]},{"osd":4,"last update":"Wed Feb 10 04:26:44 2021","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.1659999999999999,"15min":1.1819999999999999},"min":{"1min":0.82699999999999996,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.3879999999999999,"5min":1.613,"15min":1.613},"last":0.85499999999999998},{"interface":"front","average":{"1min":1.137,"5min":1.1559999999999999,"15min":1.159},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.597,"5min":1.597,"15min":1.597},"last":0.76600000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:27:01 2021","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.2,"15min":1.2190000000000001},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.874},{"interface":"front","average":{"1min":1.218,"5min":1.2889999999999999,"15min":1.3009999999999999},"min":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"max":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":0.89600000000000002}]},{"osd":6,"last update":"Wed Feb 10 04:27:20 2021","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.2909999999999999,"15min":1.3029999999999999},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":1.601,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.351,"15min":1.363},"min":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":2.1019999999999999,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":1.0129999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.2330000000000001,"15min":1.252},"min":{"1min":0.94599999999999995,"5min":0.93500000000000005,"15min":0.93500000000000005},"max":{"1min":1.5900000000000001,"5min":1.76,"15min":1.76},"last":1.421},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3500000000000001,"15min":1.3640000000000001},"min":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"max":{"1min":1.611,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":1.036}]}]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6528,"kb_used_data":760,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738624,"statfs":{"total":95995035648,"available":95988350976,"internally_reserved":0,"allocated":778240,"data_stored":299008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:10 2021","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.70899999999999996,"15min":0.68300000000000005},"min":{"1min":0.54400000000000004,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.97999999999999998,"5min":1.349,"15min":1.349},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.73299999999999998,"15min":0.71099999999999997},"min":{"1min":0.59499999999999997,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.96399999999999997,"5min":1.274,"15min":1.274},"last":0.53900000000000003}]},{"osd":1,"last update":"Wed Feb 10 04:27:10 2021","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.81200000000000006,"15min":0.76900000000000002},"min":{"1min":0.57499999999999996,"5min":0.495,"15min":0.495},"max":{"1min":1.3069999999999999,"5min":1.726,"15min":1.726},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.71299999999999997,"15min":0.69099999999999995},"min":{"1min":0.54900000000000004,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.51200000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:27:31 2021","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.81499999999999995,"15min":0.80400000000000005},"min":{"1min":0.57999999999999996,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.90700000000000003,"5min":1.274,"15min":1.274},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.996,"15min":0.97099999999999997},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.2,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":0.96899999999999997}]},{"osd":4,"last update":"Wed Feb 10 04:26:43 2021","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.1990000000000001,"15min":1.1870000000000001},"min":{"1min":0.97799999999999998,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":1.5860000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":1.4359999999999999},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.1160000000000001,"15min":1.121},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.516,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":1.4630000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:26:59 2021","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.3180000000000001,"15min":1.3169999999999999},"min":{"1min":1.014,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":1.71,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.1879999999999999,"15min":1.1819999999999999},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":1.083}]},{"osd":6,"last update":"Wed Feb 10 04:27:23 2021","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.284,"15min":1.2989999999999999},"min":{"1min":0.875,"5min":0.875,"15min":0.875},"max":{"1min":1.504,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.26,"5min":1.409,"15min":1.4339999999999999},"min":{"1min":1.05,"5min":1.0489999999999999,"15min":1.0489999999999999},"max":{"1min":1.635,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.216}]},{"osd":7,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.4690000000000001,"15min":1.494},"min":{"1min":1.077,"5min":1.073,"15min":1.073},"max":{"1min":1.653,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":1.276},{"interface":"front","average":{"1min":1.171,"5min":1.252,"15min":1.266},"min":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"max":{"1min":1.52,"5min":1.6160000000000001,"15min":1.6160000000000001},"last":1.3879999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182441,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6900,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":5952,"kb_avail":93738252,"statfs":{"total":95995035648,"available":95987970048,"internally_reserved":0,"allocated":962560,"data_stored":304990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6094848},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:28 2021","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.74299999999999999,"15min":0.73499999999999999},"min":{"1min":0.50900000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.0600000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.88300000000000001,"15min":0.86699999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":4.2759999999999998,"5min":4.2759999999999998,"15min":4.2759999999999998},"last":0.81599999999999995}]},{"osd":1,"last update":"Wed Feb 10 04:27:28 2021","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.90300000000000002,"15min":0.877},"min":{"1min":0.64200000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":3.246,"5min":3.246,"15min":3.246},"last":0.71099999999999997},{"interface":"front","average":{"1min":1.0549999999999999,"5min":0.89700000000000002,"15min":0.86199999999999999},"min":{"1min":0.57999999999999996,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":5.4039999999999999,"5min":5.4039999999999999,"15min":5.4039999999999999},"last":0.59999999999999998}]},{"osd":2,"last update":"Wed Feb 10 04:27:28 2021","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":0.97199999999999998,"15min":0.95099999999999996},"min":{"1min":0.60299999999999998,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":4.2309999999999999,"5min":4.2309999999999999,"15min":4.2309999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.96099999999999997,"15min":0.97599999999999998},"min":{"1min":0.65900000000000003,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.2609999999999999,"5min":1.681,"15min":1.681},"last":1.038}]},{"osd":4,"last update":"Wed Feb 10 04:26:43 2021","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.2290000000000001,"15min":1.1970000000000001},"min":{"1min":0.879,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":3.371,"5min":3.371,"15min":3.371},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.254,"15min":1.2210000000000001},"min":{"1min":1.0469999999999999,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":3.5430000000000001,"5min":3.5430000000000001,"15min":3.5430000000000001},"last":1.0629999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:27:00 2021","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.2350000000000001,"15min":1.1930000000000001},"min":{"1min":0.94799999999999995,"5min":0.88500000000000001,"15min":0.88500000000000001},"max":{"1min":3.496,"5min":3.496,"15min":3.496},"last":1.169},{"interface":"front","average":{"1min":1.466,"5min":1.3,"15min":1.272},"min":{"1min":1.143,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":3.742,"5min":3.742,"15min":3.742},"last":1.218}]},{"osd":6,"last update":"Wed Feb 10 04:27:24 2021","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.429,"15min":1.4059999999999999},"min":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"max":{"1min":3.6859999999999999,"5min":3.6859999999999999,"15min":3.6859999999999999},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.514,"5min":1.47,"15min":1.462},"min":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"max":{"1min":3.8759999999999999,"5min":3.8759999999999999,"15min":3.8759999999999999},"last":0.999}]},{"osd":7,"last update":"Wed Feb 10 04:26:36 2021","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"min":{"1min":1.232,"5min":1.232,"15min":1.232},"max":{"1min":1.7809999999999999,"5min":1.7809999999999999,"15min":1.7809999999999999},"last":1.087},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"min":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":1.1539999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986215,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6800,"kb_used_data":904,"kb_used_omap":0,"kb_used_meta":5888,"kb_avail":93738352,"statfs":{"total":95995035648,"available":95988072448,"internally_reserved":0,"allocated":925696,"data_stored":309939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6029312},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:26:42 2021","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.99399999999999999,"15min":1.006},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.1930000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":1.042},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.97199999999999998,"15min":0.97999999999999998},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.633,"5min":1.633,"15min":1.633},"last":0.622}]},{"osd":1,"last update":"Wed Feb 10 04:26:42 2021","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.90700000000000003,"15min":0.92100000000000004},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.1619999999999999,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.97699999999999998,"5min":1.0129999999999999,"15min":1.0189999999999999},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.351,"5min":1.508,"15min":1.508},"last":1.1879999999999999}]},{"osd":2,"last update":"Wed Feb 10 04:26:42 2021","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.075,"15min":1.081},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.238,"5min":1.1719999999999999,"15min":1.161},"min":{"1min":0.90500000000000003,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.768,"5min":1.768,"15min":1.768},"last":0.84299999999999997}]},{"osd":3,"last update":"Wed Feb 10 04:26:42 2021","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.0629999999999999,"15min":1.0489999999999999},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":0.97299999999999998},{"interface":"front","average":{"1min":1.052,"5min":1.0880000000000001,"15min":1.0940000000000001},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.999}]},{"osd":5,"last update":"Wed Feb 10 04:27:00 2021","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":0.88,"15min":0.83599999999999997},"min":{"1min":0.63200000000000001,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.05,"5min":0.86599999999999999,"15min":0.83599999999999997},"min":{"1min":0.74099999999999999,"5min":0.624,"15min":0.624},"max":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.5740000000000001},"last":0.871}]},{"osd":6,"last update":"Wed Feb 10 04:27:19 2021","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.0840000000000001,"15min":1.0640000000000001},"min":{"1min":0.79200000000000004,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.27,"15min":1.252},"min":{"1min":1.139,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.714,"5min":1.74,"15min":1.74},"last":1.0880000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:27:40 2021","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.94899999999999995,"5min":0.88600000000000001,"15min":0.88600000000000001},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":1.4219999999999999},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3720000000000001,"15min":1.3740000000000001},"min":{"1min":1.073,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":1.6699999999999999,"5min":1.9450000000000001,"15min":1.9450000000000001},"last":1.161}]}]},{"osd":5,"up_from":37,"seq":158913789988,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6524,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738628,"statfs":{"total":95995035648,"available":95988355072,"internally_reserved":0,"allocated":839680,"data_stored":306008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:02 2021","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.98899999999999999,"15min":0.998},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.516,"5min":1.516,"15min":1.516},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.873,"5min":0.90400000000000003,"15min":0.90900000000000003},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.238,"5min":1.365,"15min":1.365},"last":0.91200000000000003}]},{"osd":1,"last update":"Wed Feb 10 04:27:02 2021","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.92700000000000005,"15min":0.93100000000000005},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.502,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.998,"15min":1.002},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.419,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.69999999999999996}]},{"osd":2,"last update":"Wed Feb 10 04:27:02 2021","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.997,"15min":0.995},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.831,"5min":1.831,"15min":1.831},"last":1.107}]},{"osd":3,"last update":"Wed Feb 10 04:27:02 2021","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.0980000000000001,"15min":1.0920000000000001},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.995},{"interface":"front","average":{"1min":1.087,"5min":1.0720000000000001,"15min":1.0700000000000001},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.7030000000000001,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.2490000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:27:02 2021","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":0.92000000000000004,"15min":0.90500000000000003},"min":{"1min":0.65300000000000002,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":1.032},{"interface":"front","average":{"1min":1.1699999999999999,"5min":0.98999999999999999,"15min":0.95999999999999996},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":1.0740000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:27:23 2021","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.149,"15min":1.119},"min":{"1min":0.88100000000000001,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.085,"15min":1.0549999999999999},"min":{"1min":0.92100000000000004,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.3340000000000001,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":1.345}]},{"osd":7,"last update":"Wed Feb 10 04:27:38 2021","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.1910000000000001,"15min":1.177},"min":{"1min":0.93400000000000005,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.2440000000000002,"5min":2.2440000000000002,"15min":2.2440000000000002},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.528,"5min":1.3919999999999999,"15min":1.3700000000000001},"min":{"1min":1.0169999999999999,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":2.2799999999999998,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.28}]}]}]}} 2021-02-10T04:27:45.598 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2021-02-10T04:27:45.598 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2021-02-10T04:27:45.598 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2021-02-10T04:27:45.598 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph health --format=json 2021-02-10T04:27:45.794 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:27:46.351 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:46 smithi085 conmon[29319]: cluster 2021-02-10T04:27:45.978981+0000 mgr.y (mgr.14140) 298 : cluster [DBG] pgmap v257: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:27:46.469 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:46 smithi085 conmon[35961]: cluster 2021-02-10T04:27:45.978981+0000 mgr.y (mgr.14140) 298 : cluster [DBG] pgmap v257: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:27:46.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:46 smithi164 conmon[29288]: cluster 2021-02-10T04:27:45.978981+0000 mgr.y (mgr.14140) 298 : cluster [DBG] pgmap v257: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:27:47.553 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:27:47.893 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:27:47 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:27:48.265 INFO:teuthology.orchestra.run.smithi085.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-02-10T04:27:48.266 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2021-02-10T04:27:48.266 INFO:tasks.cephadm:Setup complete, yielding 2021-02-10T04:27:48.266 INFO:teuthology.run_tasks:Running task thrashosds... 2021-02-10T04:27:48.292 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2021-02-10T04:27:48.292 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-sage2-testing-2021-02-09-1332'}, 'ceph': {'conf': {'global': {'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'c1ea624123d412aff8b9d1430e36cb45fcab76b8'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'sha1': 'c1ea624123d412aff8b9d1430e36cb45fcab76b8'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-sage2-testing-2021-02-09-1332', 'sha1': 'c1ea624123d412aff8b9d1430e36cb45fcab76b8'}} 2021-02-10T04:27:48.292 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2021-02-10T04:27:48.292 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2021-02-10T04:27:48.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:48 smithi164 conmon[29288]: cluster 2021-02-10T04:27:46.371646+0000 osd.1 (osd.1) 28 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:27:48.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:48 smithi164 conmon[29288]: 2021-02-10T04:27:48.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:48 smithi164 conmon[29288]: cluster 2021-02-10T04:27:46.371722+0000 osd.1 (osd.1) 29 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:27:48.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:48 smithi164 conmon[29288]: cluster 2021-02-10T04:27:47.117171+0000 mon.a (mon.0) 701 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2021-02-10T04:27:48.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:48 smithi164 conmon[29288]: audit 2021-02-10T04:27:47.551987+0000 mon.c (mon.1) 27 : audit 2021-02-10T04:27:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:48 smithi164 conmon[29288]: [DBG] from='client.? 172.21.15.85:0/1250316778' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2021-02-10T04:27:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:48 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:27:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:48 smithi164 conmon[29288]: -10T04:27:47.622567+0000 2021-02-10T04:27:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:48 smithi164 conmon[29288]: mgr.y (mgr.14140) 299 : audit 2021-02-10T04:27:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:48 smithi164 conmon[29288]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:27:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:48 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:27:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:48 smithi164 conmon[29288]: -02-10T04:27:47.979381 2021-02-10T04:27:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:48 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 300 : 2021-02-10T04:27:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:48 smithi164 conmon[29288]: cluster [DBG] pgmap v259: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:27:48.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:48 smithi085 conmon[29319]: cluster 2021-02-10T04:27:46.371646 2021-02-10T04:27:48.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:48 smithi085 conmon[29319]: +0000 osd.1 (osd.1) 28 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:27:48.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:48 smithi085 conmon[29319]: cluster 2021-02-10T04:27:46.371722+0000 osd.1 (osd.1) 29 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:27:48.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:48 smithi085 conmon[29319]: cluster 2021-02-10T04:27:47.117171+0000 mon.a (mon.0) 701 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2021-02-10T04:27:48.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:48 smithi085 conmon[29319]: audit 2021-02-10T04:27:47.551987+0000 mon.c (mon.1) 27 : audit [DBG] from='client.? 172.21.15.85:0/1250316778' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2021-02-10T04:27:48.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:48 smithi085 conmon[29319]: audit 2021-02-10T04:27:47.622567+0000 mgr.y (mgr.14140) 299 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:27:48.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:48 smithi085 conmon[29319]: cluster 2021-02-10T04:27:47.979381+0000 mgr.y (mgr.14140) 300 : cluster [DBG] pgmap v259: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:27:48.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:48 smithi085 conmon[35961]: cluster 2021-02-10T04:27:46.371646+0000 osd.1 (osd.1) 28 : cluster 2021-02-10T04:27:48.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:48 smithi085 conmon[35961]: [DBG] purged_snaps scrub starts 2021-02-10T04:27:48.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:48 smithi085 conmon[35961]: cluster 2021-02-10T04:27:46.371722+0000 osd.1 (osd.1) 29 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:27:48.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:48 smithi085 conmon[35961]: cluster 2021-02-10T04:27:47.117171+0000 mon.a (mon.0) 701 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2021-02-10T04:27:48.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:48 smithi085 conmon[35961]: audit 2021-02-10T04:27:47.551987+0000 mon.c (mon.1) 27 : audit [DBG] from='client.? 172.21.15.85:0/1250316778' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2021-02-10T04:27:48.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:48 smithi085 conmon[35961]: audit 2021-02-10T04:27:47.622567+0000 mgr.y (mgr.14140) 299 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:27:48.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:48 smithi085 conmon[35961]: cluster 2021-02-10T04:27:47.979381+0000 mgr.y (mgr.14140) 300 : cluster [DBG] pgmap v259: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:27:48.493 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:27:49.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:49 smithi164 conmon[29288]: cluster 2021-02-10T04:27:48.284243+0000 osd.3 (osd.3) 26 : cluster [DBG] 4.10 scrub ok 2021-02-10T04:27:49.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:49 smithi164 conmon[29288]: cluster 2021-02-10T04:27:48.598865+0000 osd.4 (osd. 2021-02-10T04:27:49.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:49 smithi164 conmon[29288]: 4) 25 : cluster [DBG] 5.1c scrub ok 2021-02-10T04:27:49.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:49 smithi164 conmon[29288]: cluster 2021-02-10T04:27:48.635762+0000 osd.0 (osd.0) 28 : cluster [DBG] 2.1f scrub ok 2021-02-10T04:27:49.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:49 smithi085 conmon[29319]: cluster 2021-02-10T04:27:48. 2021-02-10T04:27:49.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:49 smithi085 conmon[29319]: 284243+0000 osd.3 (osd.3) 26 : cluster [DBG] 4.10 scrub ok 2021-02-10T04:27:49.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:49 smithi085 conmon[29319]: cluster 2021-02-10T04:27:48.598865+0000 osd.4 (osd.4) 25 : cluster [DBG] 5.1c scrub ok 2021-02-10T04:27:49.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:49 smithi085 conmon[29319]: cluster 2021-02-10T04:27:48.635762+0000 osd.0 (osd.0) 28 : cluster [DBG] 2.1f scrub ok 2021-02-10T04:27:49.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:49 smithi085 conmon[35961]: cluster 2021-02-10T04:27:48. 2021-02-10T04:27:49.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:49 smithi085 conmon[35961]: 284243+0000 osd.3 (osd.3) 26 : cluster [DBG] 4.10 scrub ok 2021-02-10T04:27:49.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:49 smithi085 conmon[35961]: cluster 2021-02-10T04:27:48.598865+0000 osd.4 (osd.4) 25 : cluster [DBG] 5.1c scrub ok 2021-02-10T04:27:49.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:49 smithi085 conmon[35961]: cluster 2021-02-10T04:27:48.635762+0000 osd.0 (osd.0) 28 : cluster [DBG] 2021-02-10T04:27:49.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:49 smithi085 conmon[35961]: 2.1f scrub ok 2021-02-10T04:27:50.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:50 smithi164 conmon[29288]: cluster 2021-02-10T04:27:49.269534+0000 osd.3 (osd.3) 27 : cluster [DBG] 4.19 scrub ok 2021-02-10T04:27:50.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:50 smithi164 conmon[29288]: cluster 2021-02-10T04:27:49.628083+0000 2021-02-10T04:27:50.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:50 smithi164 conmon[29288]: osd.4 (osd.4) 26 : cluster [DBG] 5.18 scrub ok 2021-02-10T04:27:50.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:50 smithi164 conmon[29288]: cluster 2021-02-10T04:27:49.980065+0000 mgr.y (mgr.14140) 301 : cluster [DBG] pgmap v260: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:27:50.407 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:50 smithi085 conmon[29319]: cluster 2021-02-10T04:27:49.269534+0000 osd.3 (osd.3) 27 : cluster [DBG] 4.19 scrub ok 2021-02-10T04:27:50.407 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:50 smithi085 conmon[29319]: cluster 2021-02-10T04:27:49.628083+0000 osd.4 (osd.4) 26 : cluster [DBG] 5.18 scrub ok 2021-02-10T04:27:50.407 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:50 smithi085 conmon[29319]: cluster 2021-02-10T04:27:49.980065+0000 mgr.y (mgr.14140) 301 : cluster [DBG] pgmap v260: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:27:50.407 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:50 smithi085 conmon[35961]: cluster 2021-02-10T04:27:49.269534+0000 osd.3 2021-02-10T04:27:50.407 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:50 smithi085 conmon[35961]: (osd.3) 27 : cluster [DBG] 4.19 scrub ok 2021-02-10T04:27:50.407 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:50 smithi085 conmon[35961]: cluster 2021-02-10T04:27:49.628083+0000 osd.4 2021-02-10T04:27:50.407 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:50 smithi085 conmon[35961]: (osd.4) 26 : cluster [DBG] 5.18 scrub ok 2021-02-10T04:27:50.407 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:50 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:27:50.407 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:50 smithi085 conmon[35961]: -10T04:27:49.980065+0000 2021-02-10T04:27:50.408 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:50 smithi085 conmon[35961]: mgr.y (mgr.14140) 301 : cluster [DBG] pgmap v260: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:27:50.408 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:50 smithi085 conmon[35961]: 2021-02-10T04:27:51.090 INFO:tasks.thrashosds:Beginning thrashosds... 2021-02-10T04:27:51.090 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2021-02-10T04:27:51.090 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:27:51.286 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:27:51.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:51 smithi164 conmon[29288]: cluster 2021-02-10T04:27:50.297642+0000 osd.7 (osd.7) 24 : cluster [DBG] 2.1d deep-scrub ok 2021-02-10T04:27:51.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:51 smithi164 conmon[29288]: cluster 2021-02-10T04:27:50.299933+0000 osd.3 (osd.3) 2021-02-10T04:27:51.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:51 smithi164 conmon[29288]: 28 : cluster [DBG] 6.13 deep-scrub ok 2021-02-10T04:27:51.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:51 smithi164 conmon[29288]: audit 2021-02-10T04:27:50.377389+0000 mon.a (mon.0) 702 : audit [INF] from='client.? 172.21.15.85:0/3812610949' entity='client.admin' 2021-02-10T04:27:51.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:51 smithi164 conmon[29288]: audit 2021-02-10T04:27:50.386836+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:51.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:51 smithi164 conmon[29288]: 2021-02-10T04:27:51.690 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:51 smithi085 conmon[35961]: cluster 2021-02-10T04:27:50.297642+0000 osd.7 (osd. 2021-02-10T04:27:51.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:51 smithi085 conmon[35961]: 7) 24 : cluster [DBG] 2.1d deep-scrub ok 2021-02-10T04:27:51.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:51 smithi085 conmon[35961]: cluster 2021-02-10T04:27:50.299933+0000 osd.3 (osd.3) 28 : cluster [DBG] 6.13 deep-scrub ok 2021-02-10T04:27:51.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:51 smithi085 conmon[35961]: audit 2021-02-10T04:27:50.377389+0000 mon.a (mon.0) 702 : audit [INF] from='client.? 172.21.15.85:0/3812610949' entity='client.admin' 2021-02-10T04:27:51.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:51 smithi085 conmon[35961]: audit 2021-02-10T04:27:50.386836+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:51.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:51 smithi085 conmon[29319]: cluster 2021-02-10T04:27:50.297642+0000 osd.7 2021-02-10T04:27:51.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:51 smithi085 conmon[29319]: (osd.7) 24 : cluster [DBG] 2.1d deep-scrub ok 2021-02-10T04:27:51.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:51 smithi085 conmon[29319]: cluster 2021-02-10T04:27:50.299933+0000 osd.3 (osd.3) 28 : cluster [DBG] 6.13 deep-scrub ok 2021-02-10T04:27:51.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:51 smithi085 conmon[29319]: audit 2021-02-10T04:27:50.377389+0000 mon.a (mon.0) 702 : audit [INF] from='client.? 172.21.15.85:0/3812610949' entity='client.admin' 2021-02-10T04:27:51.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:51 smithi085 conmon[29319]: audit 2021-02-10T04:27:50.386836+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:27:52.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:52 smithi164 conmon[29288]: cluster 2021-02-10T04:27:51.679504+0000 osd.4 (osd.4) 27 : cluster [DBG] 6.1a scrub ok 2021-02-10T04:27:52.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:52 smithi164 conmon[29288]: 2021-02-10T04:27:52.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:52 smithi164 conmon[29288]: cluster 2021-02-10T04:27:51.980875+0000 mgr.y (mgr.14140) 302 : cluster [DBG] pgmap v261: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:52.666 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:52 smithi085 conmon[29319]: cluster 2021-02-10T04:27:51.679504+0000 osd.4 (osd. 2021-02-10T04:27:52.666 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:52 smithi085 conmon[29319]: 4) 27 : cluster [DBG] 6.1a scrub ok 2021-02-10T04:27:52.667 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:52 smithi085 conmon[29319]: cluster 2021-02-10T04:27:51.980875+0000 mgr.y (mgr.14140) 302 : cluster [DBG] pgmap v261: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:52.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:52 smithi085 conmon[35961]: cluster 2021-02-10T04:27:51.679504+0000 2021-02-10T04:27:52.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:52 smithi085 conmon[35961]: osd.4 (osd.4) 27 : cluster [DBG] 6.1a scrub ok 2021-02-10T04:27:52.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:52 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:27:52.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:52 smithi085 conmon[35961]: T04:27:51.980875+0000 mgr.y (mgr.14140) 302 : cluster [DBG] pgmap v261: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:52.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:52 smithi085 conmon[35961]: 2021-02-10T04:27:53.027 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:27:53.029 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:27:53.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:53 smithi164 conmon[29288]: cluster 2021-02-10T04:27:52.383182+0000 osd.3 (osd.3) 29 : cluster [DBG] 6.c scrub ok 2021-02-10T04:27:53.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:53 smithi164 conmon[29288]: audit 2021-02-10T04:27:53.013879+0000 mgr.y (mgr.14140) 303 : audit [DBG] from='client.24628 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:27:53.667 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:53 smithi085 conmon[29319]: cluster 2021-02-10T04:27:52.383182+0000 osd.3 (osd.3) 29 : cluster [DBG] 6.c scrub ok 2021-02-10T04:27:53.668 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:53 smithi085 conmon[29319]: audit 2021-02-10T04:27:53.013879+0000 mgr.y (mgr.14140) 303 : audit [DBG] from='client.24628 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:27:53.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:53 smithi085 conmon[35961]: cluster 2021-02-10T04:27:52.383182+0000 osd.3 (osd.3) 29 : cluster [DBG] 6.c scrub ok 2021-02-10T04:27:53.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:53 smithi085 conmon[35961]: audit 2021-02-10T04:27:53.013879+0000 mgr.y (mgr.14140) 303 : audit [DBG] from='client.24628 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:27:53.682 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":261,"stamp":"2021-02-10T04:27:51.980312+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":25858,"num_objects":357,"num_object_clones":0,"num_object_copies":1071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":357,"num_whiteouts":0,"num_read":1596,"num_read_kb":1064,"num_write":759,"num_write_kb":177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":787,"ondisk_log_size":787,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":58148,"kb_used_data":7076,"kb_used_omap":0,"kb_used_meta":51008,"kb_avail":749903068,"statfs":{"total":767960285184,"available":767900741632,"internally_reserved":0,"allocated":7245824,"data_stored":2506514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52232192},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002548"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":"37","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.374865+0000","last_change":"2021-02-10T04:27:07.339624+0000","last_active":"2021-02-10T04:27:41.374865+0000","last_peered":"2021-02-10T04:27:41.374865+0000","last_clean":"2021-02-10T04:27:41.374865+0000","last_became_active":"2021-02-10T04:25:53.150917+0000","last_became_peered":"2021-02-10T04:25:53.150917+0000","last_unstale":"2021-02-10T04:27:41.374865+0000","last_undegraded":"2021-02-10T04:27:41.374865+0000","last_fullsized":"2021-02-10T04:27:41.374865+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:07.339551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:07.339551+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"40","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:49.628172+0000","last_change":"2021-02-10T04:27:49.628172+0000","last_active":"2021-02-10T04:27:49.628172+0000","last_peered":"2021-02-10T04:27:49.628172+0000","last_clean":"2021-02-10T04:27:49.628172+0000","last_became_active":"2021-02-10T04:25:51.029110+0000","last_became_peered":"2021-02-10T04:25:51.029110+0000","last_unstale":"2021-02-10T04:27:49.628172+0000","last_undegraded":"2021-02-10T04:27:49.628172+0000","last_fullsized":"2021-02-10T04:27:49.628172+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:49.628090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:49.628090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"58","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:48.635827+0000","last_change":"2021-02-10T04:27:48.635827+0000","last_active":"2021-02-10T04:27:48.635827+0000","last_peered":"2021-02-10T04:27:48.635827+0000","last_clean":"2021-02-10T04:27:48.635827+0000","last_became_active":"2021-02-10T04:25:36.336112+0000","last_became_peered":"2021-02-10T04:25:36.336112+0000","last_unstale":"2021-02-10T04:27:48.635827+0000","last_undegraded":"2021-02-10T04:27:48.635827+0000","last_fullsized":"2021-02-10T04:27:48.635827+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:48.635765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:48.635765+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"63'44","reported_seq":"136","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:49.269619+0000","last_change":"2021-02-10T04:27:49.269619+0000","last_active":"2021-02-10T04:27:49.269619+0000","last_peered":"2021-02-10T04:27:49.269619+0000","last_clean":"2021-02-10T04:27:49.269619+0000","last_became_active":"2021-02-10T04:25:49.008451+0000","last_became_peered":"2021-02-10T04:25:49.008451+0000","last_unstale":"2021-02-10T04:27:49.269619+0000","last_undegraded":"2021-02-10T04:27:49.269619+0000","last_fullsized":"2021-02-10T04:27:49.269619+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'44","last_scrub_stamp":"2021-02-10T04:27:49.269539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:49.269539+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":89,"num_read_kb":66,"num_write":45,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"38","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:51.679613+0000","last_change":"2021-02-10T04:27:51.679613+0000","last_active":"2021-02-10T04:27:51.679613+0000","last_peered":"2021-02-10T04:27:51.679613+0000","last_clean":"2021-02-10T04:27:51.679613+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:27:51.679613+0000","last_undegraded":"2021-02-10T04:27:51.679613+0000","last_fullsized":"2021-02-10T04:27:51.679613+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:51.679515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:51.679515+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.099626+0000","last_change":"2021-02-10T04:26:54.007373+0000","last_active":"2021-02-10T04:27:41.099626+0000","last_peered":"2021-02-10T04:27:41.099626+0000","last_clean":"2021-02-10T04:27:41.099626+0000","last_became_active":"2021-02-10T04:25:51.030405+0000","last_became_peered":"2021-02-10T04:25:51.030405+0000","last_unstale":"2021-02-10T04:27:41.099626+0000","last_undegraded":"2021-02-10T04:27:41.099626+0000","last_fullsized":"2021-02-10T04:27:41.099626+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:54.007310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:26:54.007310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"48","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375707+0000","last_change":"2021-02-10T04:27:13.332288+0000","last_active":"2021-02-10T04:27:41.375707+0000","last_peered":"2021-02-10T04:27:41.375707+0000","last_clean":"2021-02-10T04:27:41.375707+0000","last_became_active":"2021-02-10T04:25:34.327162+0000","last_became_peered":"2021-02-10T04:25:34.327162+0000","last_unstale":"2021-02-10T04:27:41.375707+0000","last_undegraded":"2021-02-10T04:27:41.375707+0000","last_fullsized":"2021-02-10T04:27:41.375707+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:27:13.332224+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"78'32","reported_seq":"96","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375595+0000","last_change":"2021-02-10T04:27:09.170492+0000","last_active":"2021-02-10T04:27:41.375595+0000","last_peered":"2021-02-10T04:27:41.375595+0000","last_clean":"2021-02-10T04:27:41.375595+0000","last_became_active":"2021-02-10T04:25:49.010481+0000","last_became_peered":"2021-02-10T04:25:49.010481+0000","last_unstale":"2021-02-10T04:27:41.375595+0000","last_undegraded":"2021-02-10T04:27:41.375595+0000","last_fullsized":"2021-02-10T04:27:41.375595+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'32","last_scrub_stamp":"2021-02-10T04:27:09.170428+0000","last_deep_scrub":"78'32","last_deep_scrub_stamp":"2021-02-10T04:27:09.170428+0000","last_clean_scrub_stamp":"2021-02-10T04:27:09.170428+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":47,"num_read_kb":28,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375539+0000","last_change":"2021-02-10T04:27:07.962210+0000","last_active":"2021-02-10T04:27:41.375539+0000","last_peered":"2021-02-10T04:27:41.375539+0000","last_clean":"2021-02-10T04:27:41.375539+0000","last_became_active":"2021-02-10T04:25:53.042601+0000","last_became_peered":"2021-02-10T04:25:53.042601+0000","last_unstale":"2021-02-10T04:27:41.375539+0000","last_undegraded":"2021-02-10T04:27:41.375539+0000","last_fullsized":"2021-02-10T04:27:41.375539+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:07.962115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:07.962115+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376031+0000","last_change":"2021-02-10T04:27:29.251059+0000","last_active":"2021-02-10T04:27:41.376031+0000","last_peered":"2021-02-10T04:27:41.376031+0000","last_clean":"2021-02-10T04:27:41.376031+0000","last_became_active":"2021-02-10T04:25:51.029286+0000","last_became_peered":"2021-02-10T04:25:51.029286+0000","last_unstale":"2021-02-10T04:27:41.376031+0000","last_undegraded":"2021-02-10T04:27:41.376031+0000","last_fullsized":"2021-02-10T04:27:41.376031+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:29.250978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:29.250978+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"52","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:50.297737+0000","last_change":"2021-02-10T04:27:50.297737+0000","last_active":"2021-02-10T04:27:50.297737+0000","last_peered":"2021-02-10T04:27:50.297737+0000","last_clean":"2021-02-10T04:27:50.297737+0000","last_became_active":"2021-02-10T04:25:36.344477+0000","last_became_peered":"2021-02-10T04:25:36.344477+0000","last_unstale":"2021-02-10T04:27:50.297737+0000","last_undegraded":"2021-02-10T04:27:50.297737+0000","last_fullsized":"2021-02-10T04:27:50.297737+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:27:50.297650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"63'12","reported_seq":"69","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.378150+0000","last_change":"2021-02-10T04:27:37.441260+0000","last_active":"2021-02-10T04:27:41.378150+0000","last_peered":"2021-02-10T04:27:41.378150+0000","last_clean":"2021-02-10T04:27:41.378150+0000","last_became_active":"2021-02-10T04:25:49.013196+0000","last_became_peered":"2021-02-10T04:25:49.013196+0000","last_unstale":"2021-02-10T04:27:41.378150+0000","last_undegraded":"2021-02-10T04:27:41.378150+0000","last_fullsized":"2021-02-10T04:27:41.378150+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2021-02-10T04:27:37.441168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:37.441168+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.098695+0000","last_change":"2021-02-10T04:27:33.628099+0000","last_active":"2021-02-10T04:27:41.098695+0000","last_peered":"2021-02-10T04:27:41.098695+0000","last_clean":"2021-02-10T04:27:41.098695+0000","last_became_active":"2021-02-10T04:25:53.039844+0000","last_became_peered":"2021-02-10T04:25:53.039844+0000","last_unstale":"2021-02-10T04:27:41.098695+0000","last_undegraded":"2021-02-10T04:27:41.098695+0000","last_fullsized":"2021-02-10T04:27:41.098695+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:33.628038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:33.628038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375359+0000","last_change":"2021-02-10T04:27:04.977415+0000","last_active":"2021-02-10T04:27:41.375359+0000","last_peered":"2021-02-10T04:27:41.375359+0000","last_clean":"2021-02-10T04:27:41.375359+0000","last_became_active":"2021-02-10T04:25:51.025488+0000","last_became_peered":"2021-02-10T04:25:51.025488+0000","last_unstale":"2021-02-10T04:27:41.375359+0000","last_undegraded":"2021-02-10T04:27:41.375359+0000","last_fullsized":"2021-02-10T04:27:41.375359+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:04.977353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:04.977353+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"47","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375687+0000","last_change":"2021-02-10T04:26:45.223669+0000","last_active":"2021-02-10T04:27:41.375687+0000","last_peered":"2021-02-10T04:27:41.375687+0000","last_clean":"2021-02-10T04:27:41.375687+0000","last_became_active":"2021-02-10T04:25:36.337260+0000","last_became_peered":"2021-02-10T04:25:36.337260+0000","last_unstale":"2021-02-10T04:27:41.375687+0000","last_undegraded":"2021-02-10T04:27:41.375687+0000","last_fullsized":"2021-02-10T04:27:41.375687+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:45.223603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:45.223603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"63'25","reported_seq":"96","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376564+0000","last_change":"2021-02-10T04:27:05.242824+0000","last_active":"2021-02-10T04:27:41.376564+0000","last_peered":"2021-02-10T04:27:41.376564+0000","last_clean":"2021-02-10T04:27:41.376564+0000","last_became_active":"2021-02-10T04:25:49.000911+0000","last_became_peered":"2021-02-10T04:25:49.000911+0000","last_unstale":"2021-02-10T04:27:41.376564+0000","last_undegraded":"2021-02-10T04:27:41.376564+0000","last_fullsized":"2021-02-10T04:27:41.376564+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2021-02-10T04:27:05.242762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:05.242762+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":53,"num_read_kb":37,"num_write":26,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"37","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375909+0000","last_change":"2021-02-10T04:26:55.332873+0000","last_active":"2021-02-10T04:27:41.375909+0000","last_peered":"2021-02-10T04:27:41.375909+0000","last_clean":"2021-02-10T04:27:41.375909+0000","last_became_active":"2021-02-10T04:25:53.147831+0000","last_became_peered":"2021-02-10T04:25:53.147831+0000","last_unstale":"2021-02-10T04:27:41.375909+0000","last_undegraded":"2021-02-10T04:27:41.375909+0000","last_fullsized":"2021-02-10T04:27:41.375909+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:55.332815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:26:55.332815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"40","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:48.598953+0000","last_change":"2021-02-10T04:27:48.598953+0000","last_active":"2021-02-10T04:27:48.598953+0000","last_peered":"2021-02-10T04:27:48.598953+0000","last_clean":"2021-02-10T04:27:48.598953+0000","last_became_active":"2021-02-10T04:25:51.018872+0000","last_became_peered":"2021-02-10T04:25:51.018872+0000","last_unstale":"2021-02-10T04:27:48.598953+0000","last_undegraded":"2021-02-10T04:27:48.598953+0000","last_fullsized":"2021-02-10T04:27:48.598953+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:48.598870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:48.598870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"57","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.374821+0000","last_change":"2021-02-10T04:26:46.425375+0000","last_active":"2021-02-10T04:27:41.374821+0000","last_peered":"2021-02-10T04:27:41.374821+0000","last_clean":"2021-02-10T04:27:41.374821+0000","last_became_active":"2021-02-10T04:25:36.336893+0000","last_became_peered":"2021-02-10T04:25:36.336893+0000","last_unstale":"2021-02-10T04:27:41.374821+0000","last_undegraded":"2021-02-10T04:27:41.374821+0000","last_fullsized":"2021-02-10T04:27:41.374821+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:46.425288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:26:46.425288+0000","last_clean_scrub_stamp":"2021-02-10T04:26:46.425288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"62'22","reported_seq":"82","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375644+0000","last_change":"2021-02-10T04:25:49.009749+0000","last_active":"2021-02-10T04:27:41.375644+0000","last_peered":"2021-02-10T04:27:41.375644+0000","last_clean":"2021-02-10T04:27:41.375644+0000","last_became_active":"2021-02-10T04:25:49.008997+0000","last_became_peered":"2021-02-10T04:25:49.008997+0000","last_unstale":"2021-02-10T04:27:41.375644+0000","last_undegraded":"2021-02-10T04:27:41.375644+0000","last_fullsized":"2021-02-10T04:27:41.375644+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:25:47.970100+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":49,"num_read_kb":31,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.377998+0000","last_change":"2021-02-10T04:27:29.364660+0000","last_active":"2021-02-10T04:27:41.377998+0000","last_peered":"2021-02-10T04:27:41.377998+0000","last_clean":"2021-02-10T04:27:41.377998+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:27:41.377998+0000","last_undegraded":"2021-02-10T04:27:41.377998+0000","last_fullsized":"2021-02-10T04:27:41.377998+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:29.364585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:29.364585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.100986+0000","last_change":"2021-02-10T04:27:27.017142+0000","last_active":"2021-02-10T04:27:41.100986+0000","last_peered":"2021-02-10T04:27:41.100986+0000","last_clean":"2021-02-10T04:27:41.100986+0000","last_became_active":"2021-02-10T04:25:51.029399+0000","last_became_peered":"2021-02-10T04:25:51.029399+0000","last_unstale":"2021-02-10T04:27:41.100986+0000","last_undegraded":"2021-02-10T04:27:41.100986+0000","last_fullsized":"2021-02-10T04:27:41.100986+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:27.017040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:27.017040+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"52","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.097636+0000","last_change":"2021-02-10T04:27:02.190158+0000","last_active":"2021-02-10T04:27:41.097636+0000","last_peered":"2021-02-10T04:27:41.097636+0000","last_clean":"2021-02-10T04:27:41.097636+0000","last_became_active":"2021-02-10T04:25:36.336442+0000","last_became_peered":"2021-02-10T04:25:36.336442+0000","last_unstale":"2021-02-10T04:27:41.097636+0000","last_undegraded":"2021-02-10T04:27:41.097636+0000","last_fullsized":"2021-02-10T04:27:41.097636+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:27:02.190056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:02.190056+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"62'15","reported_seq":"66","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375583+0000","last_change":"2021-02-10T04:26:56.623190+0000","last_active":"2021-02-10T04:27:41.375583+0000","last_peered":"2021-02-10T04:27:41.375583+0000","last_clean":"2021-02-10T04:27:41.375583+0000","last_became_active":"2021-02-10T04:25:49.010553+0000","last_became_peered":"2021-02-10T04:25:49.010553+0000","last_unstale":"2021-02-10T04:27:41.375583+0000","last_undegraded":"2021-02-10T04:27:41.375583+0000","last_fullsized":"2021-02-10T04:27:41.375583+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:26:56.623109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:26:56.623109+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.100760+0000","last_change":"2021-02-10T04:27:26.029923+0000","last_active":"2021-02-10T04:27:41.100760+0000","last_peered":"2021-02-10T04:27:41.100760+0000","last_clean":"2021-02-10T04:27:41.100760+0000","last_became_active":"2021-02-10T04:25:53.041098+0000","last_became_peered":"2021-02-10T04:25:53.041098+0000","last_unstale":"2021-02-10T04:27:41.100760+0000","last_undegraded":"2021-02-10T04:27:41.100760+0000","last_fullsized":"2021-02-10T04:27:41.100760+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:26.029856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:26.029856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.100087+0000","last_change":"2021-02-10T04:27:27.564268+0000","last_active":"2021-02-10T04:27:41.100087+0000","last_peered":"2021-02-10T04:27:41.100087+0000","last_clean":"2021-02-10T04:27:41.100087+0000","last_became_active":"2021-02-10T04:25:51.022853+0000","last_became_peered":"2021-02-10T04:25:51.022853+0000","last_unstale":"2021-02-10T04:27:41.100087+0000","last_undegraded":"2021-02-10T04:27:41.100087+0000","last_fullsized":"2021-02-10T04:27:41.100087+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:27.564175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:27.564175+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"48","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.374806+0000","last_change":"2021-02-10T04:27:16.340830+0000","last_active":"2021-02-10T04:27:41.374806+0000","last_peered":"2021-02-10T04:27:41.374806+0000","last_clean":"2021-02-10T04:27:41.374806+0000","last_became_active":"2021-02-10T04:25:34.324981+0000","last_became_peered":"2021-02-10T04:25:34.324981+0000","last_unstale":"2021-02-10T04:27:41.374806+0000","last_undegraded":"2021-02-10T04:27:41.374806+0000","last_fullsized":"2021-02-10T04:27:41.374806+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:16.340724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:16.340724+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"76'35","reported_seq":"100","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.098180+0000","last_change":"2021-02-10T04:26:58.198706+0000","last_active":"2021-02-10T04:27:41.098180+0000","last_peered":"2021-02-10T04:27:41.098180+0000","last_clean":"2021-02-10T04:27:41.098180+0000","last_became_active":"2021-02-10T04:25:49.015760+0000","last_became_peered":"2021-02-10T04:25:49.015760+0000","last_unstale":"2021-02-10T04:27:41.098180+0000","last_undegraded":"2021-02-10T04:27:41.098180+0000","last_fullsized":"2021-02-10T04:27:41.098180+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'35","last_scrub_stamp":"2021-02-10T04:26:58.198641+0000","last_deep_scrub":"76'35","last_deep_scrub_stamp":"2021-02-10T04:26:58.198641+0000","last_clean_scrub_stamp":"2021-02-10T04:26:58.198641+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":50,"num_read_kb":30,"num_write":24,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376084+0000","last_change":"2021-02-10T04:27:39.139028+0000","last_active":"2021-02-10T04:27:41.376084+0000","last_peered":"2021-02-10T04:27:41.376084+0000","last_clean":"2021-02-10T04:27:41.376084+0000","last_became_active":"2021-02-10T04:25:53.042279+0000","last_became_peered":"2021-02-10T04:25:53.042279+0000","last_unstale":"2021-02-10T04:27:41.376084+0000","last_undegraded":"2021-02-10T04:27:41.376084+0000","last_fullsized":"2021-02-10T04:27:41.376084+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:39.138947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:39.138947+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"58'2","reported_seq":"66","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.727764+0000","last_change":"2021-02-10T04:27:15.041424+0000","last_active":"2021-02-10T04:27:47.727764+0000","last_peered":"2021-02-10T04:27:47.727764+0000","last_clean":"2021-02-10T04:27:47.727764+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:27:47.727764+0000","last_undegraded":"2021-02-10T04:27:47.727764+0000","last_fullsized":"2021-02-10T04:27:47.727764+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:15.041329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:15.041329+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"58","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375030+0000","last_change":"2021-02-10T04:27:04.011149+0000","last_active":"2021-02-10T04:27:41.375030+0000","last_peered":"2021-02-10T04:27:41.375030+0000","last_clean":"2021-02-10T04:27:41.375030+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:27:41.375030+0000","last_undegraded":"2021-02-10T04:27:41.375030+0000","last_fullsized":"2021-02-10T04:27:41.375030+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:04.011053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:04.011053+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"63'22","reported_seq":"84","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.100101+0000","last_change":"2021-02-10T04:26:51.409218+0000","last_active":"2021-02-10T04:27:41.100101+0000","last_peered":"2021-02-10T04:27:41.100101+0000","last_clean":"2021-02-10T04:27:41.100101+0000","last_became_active":"2021-02-10T04:25:49.009177+0000","last_became_peered":"2021-02-10T04:25:49.009177+0000","last_unstale":"2021-02-10T04:27:41.100101+0000","last_undegraded":"2021-02-10T04:27:41.100101+0000","last_fullsized":"2021-02-10T04:27:41.100101+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2021-02-10T04:26:51.409147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:26:51.409147+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":22,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"32","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375183+0000","last_change":"2021-02-10T04:25:53.151054+0000","last_active":"2021-02-10T04:27:41.375183+0000","last_peered":"2021-02-10T04:27:41.375183+0000","last_clean":"2021-02-10T04:27:41.375183+0000","last_became_active":"2021-02-10T04:25:53.150845+0000","last_became_peered":"2021-02-10T04:25:53.150845+0000","last_unstale":"2021-02-10T04:27:41.375183+0000","last_undegraded":"2021-02-10T04:27:41.375183+0000","last_fullsized":"2021-02-10T04:27:41.375183+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:25:52.015519+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375521+0000","last_change":"2021-02-10T04:27:07.181928+0000","last_active":"2021-02-10T04:27:41.375521+0000","last_peered":"2021-02-10T04:27:41.375521+0000","last_clean":"2021-02-10T04:27:41.375521+0000","last_became_active":"2021-02-10T04:25:51.018491+0000","last_became_peered":"2021-02-10T04:25:51.018491+0000","last_unstale":"2021-02-10T04:27:41.375521+0000","last_undegraded":"2021-02-10T04:27:41.375521+0000","last_fullsized":"2021-02-10T04:27:41.375521+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:07.181854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:07.181854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"52","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.098169+0000","last_change":"2021-02-10T04:26:52.256790+0000","last_active":"2021-02-10T04:27:41.098169+0000","last_peered":"2021-02-10T04:27:41.098169+0000","last_clean":"2021-02-10T04:27:41.098169+0000","last_became_active":"2021-02-10T04:25:35.328828+0000","last_became_peered":"2021-02-10T04:25:35.328828+0000","last_unstale":"2021-02-10T04:27:41.098169+0000","last_undegraded":"2021-02-10T04:27:41.098169+0000","last_fullsized":"2021-02-10T04:27:41.098169+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:26:52.256723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:52.256723+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"63'21","reported_seq":"82","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375014+0000","last_change":"2021-02-10T04:27:12.291860+0000","last_active":"2021-02-10T04:27:41.375014+0000","last_peered":"2021-02-10T04:27:41.375014+0000","last_clean":"2021-02-10T04:27:41.375014+0000","last_became_active":"2021-02-10T04:25:49.007049+0000","last_became_peered":"2021-02-10T04:25:49.007049+0000","last_unstale":"2021-02-10T04:27:41.375014+0000","last_undegraded":"2021-02-10T04:27:41.375014+0000","last_fullsized":"2021-02-10T04:27:41.375014+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2021-02-10T04:27:12.291796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:12.291796+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":27,"num_write":21,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375818+0000","last_change":"2021-02-10T04:27:19.134286+0000","last_active":"2021-02-10T04:27:41.375818+0000","last_peered":"2021-02-10T04:27:41.375818+0000","last_clean":"2021-02-10T04:27:41.375818+0000","last_became_active":"2021-02-10T04:25:53.034771+0000","last_became_peered":"2021-02-10T04:25:53.034771+0000","last_unstale":"2021-02-10T04:27:41.375818+0000","last_undegraded":"2021-02-10T04:27:41.375818+0000","last_fullsized":"2021-02-10T04:27:41.375818+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:27:19.134212+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.097597+0000","last_change":"2021-02-10T04:27:19.077862+0000","last_active":"2021-02-10T04:27:41.097597+0000","last_peered":"2021-02-10T04:27:41.097597+0000","last_clean":"2021-02-10T04:27:41.097597+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:27:41.097597+0000","last_undegraded":"2021-02-10T04:27:41.097597+0000","last_fullsized":"2021-02-10T04:27:41.097597+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:27:19.077777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"49","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.377580+0000","last_change":"2021-02-10T04:27:03.000902+0000","last_active":"2021-02-10T04:27:41.377580+0000","last_peered":"2021-02-10T04:27:41.377580+0000","last_clean":"2021-02-10T04:27:41.377580+0000","last_became_active":"2021-02-10T04:25:34.331787+0000","last_became_peered":"2021-02-10T04:25:34.331787+0000","last_unstale":"2021-02-10T04:27:41.377580+0000","last_undegraded":"2021-02-10T04:27:41.377580+0000","last_fullsized":"2021-02-10T04:27:41.377580+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:03.000821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:03.000821+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"62'12","reported_seq":"94","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:48.284315+0000","last_change":"2021-02-10T04:27:48.284315+0000","last_active":"2021-02-10T04:27:48.284315+0000","last_peered":"2021-02-10T04:27:48.284315+0000","last_clean":"2021-02-10T04:27:48.284315+0000","last_became_active":"2021-02-10T04:26:39.005616+0000","last_became_peered":"2021-02-10T04:26:39.005616+0000","last_unstale":"2021-02-10T04:27:48.284315+0000","last_undegraded":"2021-02-10T04:27:48.284315+0000","last_fullsized":"2021-02-10T04:27:48.284315+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2021-02-10T04:27:48.284248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:48.284248+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"37","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.374682+0000","last_change":"2021-02-10T04:26:57.309978+0000","last_active":"2021-02-10T04:27:41.374682+0000","last_peered":"2021-02-10T04:27:41.374682+0000","last_clean":"2021-02-10T04:27:41.374682+0000","last_became_active":"2021-02-10T04:25:53.043277+0000","last_became_peered":"2021-02-10T04:25:53.043277+0000","last_unstale":"2021-02-10T04:27:41.374682+0000","last_undegraded":"2021-02-10T04:27:41.374682+0000","last_fullsized":"2021-02-10T04:27:41.374682+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:57.309915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:26:57.309915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.100552+0000","last_change":"2021-02-10T04:27:33.030004+0000","last_active":"2021-02-10T04:27:41.100552+0000","last_peered":"2021-02-10T04:27:41.100552+0000","last_clean":"2021-02-10T04:27:41.100552+0000","last_became_active":"2021-02-10T04:25:51.030952+0000","last_became_peered":"2021-02-10T04:25:51.030952+0000","last_unstale":"2021-02-10T04:27:41.100552+0000","last_undegraded":"2021-02-10T04:27:41.100552+0000","last_fullsized":"2021-02-10T04:27:41.100552+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:27:33.029912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"58","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.100813+0000","last_change":"2021-02-10T04:26:52.011165+0000","last_active":"2021-02-10T04:27:41.100813+0000","last_peered":"2021-02-10T04:27:41.100813+0000","last_clean":"2021-02-10T04:27:41.100813+0000","last_became_active":"2021-02-10T04:25:34.330242+0000","last_became_peered":"2021-02-10T04:25:34.330242+0000","last_unstale":"2021-02-10T04:27:41.100813+0000","last_undegraded":"2021-02-10T04:27:41.100813+0000","last_fullsized":"2021-02-10T04:27:41.100813+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:26:52.011096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:52.011096+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"62'15","reported_seq":"69","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:42.487297+0000","last_change":"2021-02-10T04:27:42.487239+0000","last_active":"2021-02-10T04:27:42.487297+0000","last_peered":"2021-02-10T04:27:42.487297+0000","last_clean":"2021-02-10T04:27:42.487297+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:27:42.487297+0000","last_undegraded":"2021-02-10T04:27:42.487297+0000","last_fullsized":"2021-02-10T04:27:42.487297+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:27:42.487154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:42.487154+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.098426+0000","last_change":"2021-02-10T04:27:21.548446+0000","last_active":"2021-02-10T04:27:41.098426+0000","last_peered":"2021-02-10T04:27:41.098426+0000","last_clean":"2021-02-10T04:27:41.098426+0000","last_became_active":"2021-02-10T04:25:53.039322+0000","last_became_peered":"2021-02-10T04:25:53.039322+0000","last_unstale":"2021-02-10T04:27:41.098426+0000","last_undegraded":"2021-02-10T04:27:41.098426+0000","last_fullsized":"2021-02-10T04:27:41.098426+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:21.548381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:21.548381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"39","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375032+0000","last_change":"2021-02-10T04:27:05.297715+0000","last_active":"2021-02-10T04:27:41.375032+0000","last_peered":"2021-02-10T04:27:41.375032+0000","last_clean":"2021-02-10T04:27:41.375032+0000","last_became_active":"2021-02-10T04:25:51.029155+0000","last_became_peered":"2021-02-10T04:25:51.029155+0000","last_unstale":"2021-02-10T04:27:41.375032+0000","last_undegraded":"2021-02-10T04:27:41.375032+0000","last_fullsized":"2021-02-10T04:27:41.375032+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:27:05.297647+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"49","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.097864+0000","last_change":"2021-02-10T04:26:54.256587+0000","last_active":"2021-02-10T04:27:41.097864+0000","last_peered":"2021-02-10T04:27:41.097864+0000","last_clean":"2021-02-10T04:27:41.097864+0000","last_became_active":"2021-02-10T04:25:34.330070+0000","last_became_peered":"2021-02-10T04:25:34.330070+0000","last_unstale":"2021-02-10T04:27:41.097864+0000","last_undegraded":"2021-02-10T04:27:41.097864+0000","last_fullsized":"2021-02-10T04:27:41.097864+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:54.256498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:54.256498+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"62'15","reported_seq":"70","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.100505+0000","last_change":"2021-02-10T04:27:40.218910+0000","last_active":"2021-02-10T04:27:41.100505+0000","last_peered":"2021-02-10T04:27:41.100505+0000","last_clean":"2021-02-10T04:27:41.100505+0000","last_became_active":"2021-02-10T04:25:49.015543+0000","last_became_peered":"2021-02-10T04:25:49.015543+0000","last_unstale":"2021-02-10T04:27:41.100505+0000","last_undegraded":"2021-02-10T04:27:41.100505+0000","last_fullsized":"2021-02-10T04:27:41.100505+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:27:40.218828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:40.218828+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.378352+0000","last_change":"2021-02-10T04:27:28.373194+0000","last_active":"2021-02-10T04:27:41.378352+0000","last_peered":"2021-02-10T04:27:41.378352+0000","last_clean":"2021-02-10T04:27:41.378352+0000","last_became_active":"2021-02-10T04:25:53.031782+0000","last_became_peered":"2021-02-10T04:25:53.031782+0000","last_unstale":"2021-02-10T04:27:41.378352+0000","last_undegraded":"2021-02-10T04:27:41.378352+0000","last_fullsized":"2021-02-10T04:27:41.378352+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:28.373117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:28.373117+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"58'2","reported_seq":"65","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.117389+0000","last_change":"2021-02-10T04:27:26.264628+0000","last_active":"2021-02-10T04:27:47.117389+0000","last_peered":"2021-02-10T04:27:47.117389+0000","last_clean":"2021-02-10T04:27:47.117389+0000","last_became_active":"2021-02-10T04:25:51.018300+0000","last_became_peered":"2021-02-10T04:25:51.018300+0000","last_unstale":"2021-02-10T04:27:47.117389+0000","last_undegraded":"2021-02-10T04:27:47.117389+0000","last_fullsized":"2021-02-10T04:27:47.117389+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:26.264560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:26.264560+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"54","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.099451+0000","last_change":"2021-02-10T04:27:39.616474+0000","last_active":"2021-02-10T04:27:41.099451+0000","last_peered":"2021-02-10T04:27:41.099451+0000","last_clean":"2021-02-10T04:27:41.099451+0000","last_became_active":"2021-02-10T04:25:34.321096+0000","last_became_peered":"2021-02-10T04:25:34.321096+0000","last_unstale":"2021-02-10T04:27:41.099451+0000","last_undegraded":"2021-02-10T04:27:41.099451+0000","last_fullsized":"2021-02-10T04:27:41.099451+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:27:39.616381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"63'19","reported_seq":"81","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376932+0000","last_change":"2021-02-10T04:27:11.016733+0000","last_active":"2021-02-10T04:27:41.376932+0000","last_peered":"2021-02-10T04:27:41.376932+0000","last_clean":"2021-02-10T04:27:41.376932+0000","last_became_active":"2021-02-10T04:25:49.010890+0000","last_became_peered":"2021-02-10T04:25:49.010890+0000","last_unstale":"2021-02-10T04:27:41.376932+0000","last_undegraded":"2021-02-10T04:27:41.376932+0000","last_fullsized":"2021-02-10T04:27:41.376932+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2021-02-10T04:27:11.016650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:11.016650+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":26,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.099783+0000","last_change":"2021-02-10T04:27:08.486638+0000","last_active":"2021-02-10T04:27:41.099783+0000","last_peered":"2021-02-10T04:27:41.099783+0000","last_clean":"2021-02-10T04:27:41.099783+0000","last_became_active":"2021-02-10T04:25:53.045177+0000","last_became_peered":"2021-02-10T04:25:53.045177+0000","last_unstale":"2021-02-10T04:27:41.099783+0000","last_undegraded":"2021-02-10T04:27:41.099783+0000","last_fullsized":"2021-02-10T04:27:41.099783+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:08.486553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:08.486553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"58'2","reported_seq":"66","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.729828+0000","last_change":"2021-02-10T04:27:20.157871+0000","last_active":"2021-02-10T04:27:47.729828+0000","last_peered":"2021-02-10T04:27:47.729828+0000","last_clean":"2021-02-10T04:27:47.729828+0000","last_became_active":"2021-02-10T04:25:51.032328+0000","last_became_peered":"2021-02-10T04:25:51.032328+0000","last_unstale":"2021-02-10T04:27:47.729828+0000","last_undegraded":"2021-02-10T04:27:47.729828+0000","last_fullsized":"2021-02-10T04:27:47.729828+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:20.157772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:20.157772+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"52'4","reported_seq":"61","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376110+0000","last_change":"2021-02-10T04:26:49.013781+0000","last_active":"2021-02-10T04:27:41.376110+0000","last_peered":"2021-02-10T04:27:41.376110+0000","last_clean":"2021-02-10T04:27:41.376110+0000","last_became_active":"2021-02-10T04:25:36.334366+0000","last_became_peered":"2021-02-10T04:25:36.334366+0000","last_unstale":"2021-02-10T04:27:41.376110+0000","last_undegraded":"2021-02-10T04:27:41.376110+0000","last_fullsized":"2021-02-10T04:27:41.376110+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'4","last_scrub_stamp":"2021-02-10T04:26:49.013681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:49.013681+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"63'26","reported_seq":"127","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375225+0000","last_change":"2021-02-10T04:27:28.213521+0000","last_active":"2021-02-10T04:27:41.375225+0000","last_peered":"2021-02-10T04:27:41.375225+0000","last_clean":"2021-02-10T04:27:41.375225+0000","last_became_active":"2021-02-10T04:26:39.003971+0000","last_became_peered":"2021-02-10T04:26:39.003971+0000","last_unstale":"2021-02-10T04:27:41.375225+0000","last_undegraded":"2021-02-10T04:27:41.375225+0000","last_fullsized":"2021-02-10T04:27:41.375225+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:27:28.213454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:28.213454+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376351+0000","last_change":"2021-02-10T04:27:33.192375+0000","last_active":"2021-02-10T04:27:41.376351+0000","last_peered":"2021-02-10T04:27:41.376351+0000","last_clean":"2021-02-10T04:27:41.376351+0000","last_became_active":"2021-02-10T04:25:53.149820+0000","last_became_peered":"2021-02-10T04:25:53.149820+0000","last_unstale":"2021-02-10T04:27:41.376351+0000","last_undegraded":"2021-02-10T04:27:41.376351+0000","last_fullsized":"2021-02-10T04:27:41.376351+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:33.192287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:33.192287+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.377497+0000","last_change":"2021-02-10T04:27:23.236953+0000","last_active":"2021-02-10T04:27:41.377497+0000","last_peered":"2021-02-10T04:27:41.377497+0000","last_clean":"2021-02-10T04:27:41.377497+0000","last_became_active":"2021-02-10T04:25:51.031864+0000","last_became_peered":"2021-02-10T04:25:51.031864+0000","last_unstale":"2021-02-10T04:27:41.377497+0000","last_undegraded":"2021-02-10T04:27:41.377497+0000","last_fullsized":"2021-02-10T04:27:41.377497+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:23.236870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:23.236870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"51","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.098052+0000","last_change":"2021-02-10T04:27:14.083910+0000","last_active":"2021-02-10T04:27:41.098052+0000","last_peered":"2021-02-10T04:27:41.098052+0000","last_clean":"2021-02-10T04:27:41.098052+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:27:41.098052+0000","last_undegraded":"2021-02-10T04:27:41.098052+0000","last_fullsized":"2021-02-10T04:27:41.098052+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:27:14.083841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:14.083841+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"63'13","reported_seq":"88","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375913+0000","last_change":"2021-02-10T04:26:50.393926+0000","last_active":"2021-02-10T04:27:41.375913+0000","last_peered":"2021-02-10T04:27:41.375913+0000","last_clean":"2021-02-10T04:27:41.375913+0000","last_became_active":"2021-02-10T04:26:39.003832+0000","last_became_peered":"2021-02-10T04:26:39.003832+0000","last_unstale":"2021-02-10T04:27:41.375913+0000","last_undegraded":"2021-02-10T04:27:41.375913+0000","last_fullsized":"2021-02-10T04:27:41.375913+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2021-02-10T04:26:50.393865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:26:50.393865+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"37","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375351+0000","last_change":"2021-02-10T04:27:13.573977+0000","last_active":"2021-02-10T04:27:41.375351+0000","last_peered":"2021-02-10T04:27:41.375351+0000","last_clean":"2021-02-10T04:27:41.375351+0000","last_became_active":"2021-02-10T04:25:53.035867+0000","last_became_peered":"2021-02-10T04:25:53.035867+0000","last_unstale":"2021-02-10T04:27:41.375351+0000","last_undegraded":"2021-02-10T04:27:41.375351+0000","last_fullsized":"2021-02-10T04:27:41.375351+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:13.573912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:13.573912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"39","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375949+0000","last_change":"2021-02-10T04:26:56.335818+0000","last_active":"2021-02-10T04:27:41.375949+0000","last_peered":"2021-02-10T04:27:41.375949+0000","last_clean":"2021-02-10T04:27:41.375949+0000","last_became_active":"2021-02-10T04:25:51.030651+0000","last_became_peered":"2021-02-10T04:25:51.030651+0000","last_unstale":"2021-02-10T04:27:41.375949+0000","last_undegraded":"2021-02-10T04:27:41.375949+0000","last_fullsized":"2021-02-10T04:27:41.375949+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:56.335735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:26:56.335735+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"51","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375372+0000","last_change":"2021-02-10T04:26:57.639364+0000","last_active":"2021-02-10T04:27:41.375372+0000","last_peered":"2021-02-10T04:27:41.375372+0000","last_clean":"2021-02-10T04:27:41.375372+0000","last_became_active":"2021-02-10T04:25:34.329694+0000","last_became_peered":"2021-02-10T04:25:34.329694+0000","last_unstale":"2021-02-10T04:27:41.375372+0000","last_undegraded":"2021-02-10T04:27:41.375372+0000","last_fullsized":"2021-02-10T04:27:41.375372+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:26:57.639281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:57.639281+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"63'28","reported_seq":"102","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.099932+0000","last_change":"2021-02-10T04:27:24.523914+0000","last_active":"2021-02-10T04:27:41.099932+0000","last_peered":"2021-02-10T04:27:41.099932+0000","last_clean":"2021-02-10T04:27:41.099932+0000","last_became_active":"2021-02-10T04:25:49.008942+0000","last_became_peered":"2021-02-10T04:25:49.008942+0000","last_unstale":"2021-02-10T04:27:41.099932+0000","last_undegraded":"2021-02-10T04:27:41.099932+0000","last_fullsized":"2021-02-10T04:27:41.099932+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'28","last_scrub_stamp":"2021-02-10T04:27:24.523837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:24.523837+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":62,"num_read_kb":40,"num_write":29,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"37","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375110+0000","last_change":"2021-02-10T04:27:27.259500+0000","last_active":"2021-02-10T04:27:41.375110+0000","last_peered":"2021-02-10T04:27:41.375110+0000","last_clean":"2021-02-10T04:27:41.375110+0000","last_became_active":"2021-02-10T04:25:53.047736+0000","last_became_peered":"2021-02-10T04:25:53.047736+0000","last_unstale":"2021-02-10T04:27:41.375110+0000","last_undegraded":"2021-02-10T04:27:41.375110+0000","last_fullsized":"2021-02-10T04:27:41.375110+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:27.259393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:27.259393+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"62","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.378208+0000","last_change":"2021-02-10T04:26:34.066242+0000","last_active":"2021-02-10T04:27:41.378208+0000","last_peered":"2021-02-10T04:27:41.378208+0000","last_clean":"2021-02-10T04:27:41.378208+0000","last_became_active":"2021-02-10T04:25:36.337394+0000","last_became_peered":"2021-02-10T04:25:36.337394+0000","last_unstale":"2021-02-10T04:27:41.378208+0000","last_undegraded":"2021-02-10T04:27:41.378208+0000","last_fullsized":"2021-02-10T04:27:41.378208+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:26:34.066149+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"63'27","reported_seq":"101","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.378431+0000","last_change":"2021-02-10T04:27:17.352786+0000","last_active":"2021-02-10T04:27:41.378431+0000","last_peered":"2021-02-10T04:27:41.378431+0000","last_clean":"2021-02-10T04:27:41.378431+0000","last_became_active":"2021-02-10T04:25:49.013259+0000","last_became_peered":"2021-02-10T04:25:49.013259+0000","last_unstale":"2021-02-10T04:27:41.378431+0000","last_undegraded":"2021-02-10T04:27:41.378431+0000","last_fullsized":"2021-02-10T04:27:41.378431+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'27","last_scrub_stamp":"2021-02-10T04:27:17.352701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:17.352701+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":41,"num_write":29,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"39","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375286+0000","last_change":"2021-02-10T04:27:14.530439+0000","last_active":"2021-02-10T04:27:41.375286+0000","last_peered":"2021-02-10T04:27:41.375286+0000","last_clean":"2021-02-10T04:27:41.375286+0000","last_became_active":"2021-02-10T04:25:51.023000+0000","last_became_peered":"2021-02-10T04:25:51.023000+0000","last_unstale":"2021-02-10T04:27:41.375286+0000","last_undegraded":"2021-02-10T04:27:41.375286+0000","last_fullsized":"2021-02-10T04:27:41.375286+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:14.530378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:14.530378+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.098428+0000","last_change":"2021-02-10T04:27:18.521962+0000","last_active":"2021-02-10T04:27:41.098428+0000","last_peered":"2021-02-10T04:27:41.098428+0000","last_clean":"2021-02-10T04:27:41.098428+0000","last_became_active":"2021-02-10T04:25:53.045215+0000","last_became_peered":"2021-02-10T04:25:53.045215+0000","last_unstale":"2021-02-10T04:27:41.098428+0000","last_undegraded":"2021-02-10T04:27:41.098428+0000","last_fullsized":"2021-02-10T04:27:41.098428+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:18.521897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:18.521897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"67","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.100802+0000","last_change":"2021-02-10T04:27:04.024909+0000","last_active":"2021-02-10T04:27:41.100802+0000","last_peered":"2021-02-10T04:27:41.100802+0000","last_clean":"2021-02-10T04:27:41.100802+0000","last_became_active":"2021-02-10T04:26:39.004318+0000","last_became_peered":"2021-02-10T04:26:39.004318+0000","last_unstale":"2021-02-10T04:27:41.100802+0000","last_undegraded":"2021-02-10T04:27:41.100802+0000","last_fullsized":"2021-02-10T04:27:41.100802+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:04.024848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:04.024848+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"79'34","reported_seq":"96","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.100562+0000","last_change":"2021-02-10T04:27:18.990156+0000","last_active":"2021-02-10T04:27:41.100562+0000","last_peered":"2021-02-10T04:27:41.100562+0000","last_clean":"2021-02-10T04:27:41.100562+0000","last_became_active":"2021-02-10T04:25:49.006147+0000","last_became_peered":"2021-02-10T04:25:49.006147+0000","last_unstale":"2021-02-10T04:27:41.100562+0000","last_undegraded":"2021-02-10T04:27:41.100562+0000","last_fullsized":"2021-02-10T04:27:41.100562+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'34","last_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:27:18.990088+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":22,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"58'2","reported_seq":"66","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.726542+0000","last_change":"2021-02-10T04:27:03.440844+0000","last_active":"2021-02-10T04:27:47.726542+0000","last_peered":"2021-02-10T04:27:47.726542+0000","last_clean":"2021-02-10T04:27:47.726542+0000","last_became_active":"2021-02-10T04:25:51.026251+0000","last_became_peered":"2021-02-10T04:25:51.026251+0000","last_unstale":"2021-02-10T04:27:47.726542+0000","last_undegraded":"2021-02-10T04:27:47.726542+0000","last_fullsized":"2021-02-10T04:27:47.726542+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:03.440780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:03.440780+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"37","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375751+0000","last_change":"2021-02-10T04:27:37.196914+0000","last_active":"2021-02-10T04:27:41.375751+0000","last_peered":"2021-02-10T04:27:41.375751+0000","last_clean":"2021-02-10T04:27:41.375751+0000","last_became_active":"2021-02-10T04:25:53.049487+0000","last_became_peered":"2021-02-10T04:25:53.049487+0000","last_unstale":"2021-02-10T04:27:41.375751+0000","last_undegraded":"2021-02-10T04:27:41.375751+0000","last_fullsized":"2021-02-10T04:27:41.375751+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:37.196846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:37.196846+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"49","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376166+0000","last_change":"2021-02-10T04:27:06.934347+0000","last_active":"2021-02-10T04:27:41.376166+0000","last_peered":"2021-02-10T04:27:41.376166+0000","last_clean":"2021-02-10T04:27:41.376166+0000","last_became_active":"2021-02-10T04:25:34.331779+0000","last_became_peered":"2021-02-10T04:25:34.331779+0000","last_unstale":"2021-02-10T04:27:41.376166+0000","last_undegraded":"2021-02-10T04:27:41.376166+0000","last_fullsized":"2021-02-10T04:27:41.376166+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:06.934282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:06.934282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"63'31","reported_seq":"104","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.100871+0000","last_change":"2021-02-10T04:26:56.059248+0000","last_active":"2021-02-10T04:27:41.100871+0000","last_peered":"2021-02-10T04:27:41.100871+0000","last_clean":"2021-02-10T04:27:41.100871+0000","last_became_active":"2021-02-10T04:25:49.007872+0000","last_became_peered":"2021-02-10T04:25:49.007872+0000","last_unstale":"2021-02-10T04:27:41.100871+0000","last_undegraded":"2021-02-10T04:27:41.100871+0000","last_fullsized":"2021-02-10T04:27:41.100871+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'31","last_scrub_stamp":"2021-02-10T04:26:56.059186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:26:56.059186+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":66,"num_read_kb":42,"num_write":34,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.099631+0000","last_change":"2021-02-10T04:27:31.650126+0000","last_active":"2021-02-10T04:27:41.099631+0000","last_peered":"2021-02-10T04:27:41.099631+0000","last_clean":"2021-02-10T04:27:41.099631+0000","last_became_active":"2021-02-10T04:25:51.027624+0000","last_became_peered":"2021-02-10T04:25:51.027624+0000","last_unstale":"2021-02-10T04:27:41.099631+0000","last_undegraded":"2021-02-10T04:27:41.099631+0000","last_fullsized":"2021-02-10T04:27:41.099631+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:27:31.650033+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376568+0000","last_change":"2021-02-10T04:27:28.264288+0000","last_active":"2021-02-10T04:27:41.376568+0000","last_peered":"2021-02-10T04:27:41.376568+0000","last_clean":"2021-02-10T04:27:41.376568+0000","last_became_active":"2021-02-10T04:25:53.149677+0000","last_became_peered":"2021-02-10T04:25:53.149677+0000","last_unstale":"2021-02-10T04:27:41.376568+0000","last_undegraded":"2021-02-10T04:27:41.376568+0000","last_fullsized":"2021-02-10T04:27:41.376568+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:28.264204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:28.264204+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"161","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:47.710876+0000","last_change":"2021-02-10T04:26:58.989515+0000","last_active":"2021-02-10T04:27:47.710876+0000","last_peered":"2021-02-10T04:27:47.710876+0000","last_clean":"2021-02-10T04:27:47.710876+0000","last_became_active":"2021-02-10T04:25:49.013969+0000","last_became_peered":"2021-02-10T04:25:49.013969+0000","last_unstale":"2021-02-10T04:27:47.710876+0000","last_undegraded":"2021-02-10T04:27:47.710876+0000","last_fullsized":"2021-02-10T04:27:47.710876+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:26:58.989447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:26:58.989447+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":116,"num_read_kb":111,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"53","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:44.556846+0000","last_change":"2021-02-10T04:27:44.556805+0000","last_active":"2021-02-10T04:27:44.556846+0000","last_peered":"2021-02-10T04:27:44.556846+0000","last_clean":"2021-02-10T04:27:44.556846+0000","last_became_active":"2021-02-10T04:25:34.331972+0000","last_became_peered":"2021-02-10T04:25:34.331972+0000","last_unstale":"2021-02-10T04:27:44.556846+0000","last_undegraded":"2021-02-10T04:27:44.556846+0000","last_fullsized":"2021-02-10T04:27:44.556846+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:44.556741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:44.556741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"82'29","reported_seq":"96","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.101092+0000","last_change":"2021-02-10T04:27:38.176842+0000","last_active":"2021-02-10T04:27:41.101092+0000","last_peered":"2021-02-10T04:27:41.101092+0000","last_clean":"2021-02-10T04:27:41.101092+0000","last_became_active":"2021-02-10T04:25:49.002843+0000","last_became_peered":"2021-02-10T04:25:49.002843+0000","last_unstale":"2021-02-10T04:27:41.101092+0000","last_undegraded":"2021-02-10T04:27:41.101092+0000","last_fullsized":"2021-02-10T04:27:41.101092+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'29","last_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:27:38.176754+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":32,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"39","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375693+0000","last_change":"2021-02-10T04:27:16.564733+0000","last_active":"2021-02-10T04:27:41.375693+0000","last_peered":"2021-02-10T04:27:41.375693+0000","last_clean":"2021-02-10T04:27:41.375693+0000","last_became_active":"2021-02-10T04:25:51.025306+0000","last_became_peered":"2021-02-10T04:25:51.025306+0000","last_unstale":"2021-02-10T04:27:41.375693+0000","last_undegraded":"2021-02-10T04:27:41.375693+0000","last_fullsized":"2021-02-10T04:27:41.375693+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:27:16.564638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376851+0000","last_change":"2021-02-10T04:27:18.115804+0000","last_active":"2021-02-10T04:27:41.376851+0000","last_peered":"2021-02-10T04:27:41.376851+0000","last_clean":"2021-02-10T04:27:41.376851+0000","last_became_active":"2021-02-10T04:25:53.037506+0000","last_became_peered":"2021-02-10T04:25:53.037506+0000","last_unstale":"2021-02-10T04:27:41.376851+0000","last_undegraded":"2021-02-10T04:27:41.376851+0000","last_fullsized":"2021-02-10T04:27:41.376851+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:18.115741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:18.115741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"41","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375191+0000","last_change":"2021-02-10T04:27:39.268187+0000","last_active":"2021-02-10T04:27:41.375191+0000","last_peered":"2021-02-10T04:27:41.375191+0000","last_clean":"2021-02-10T04:27:41.375191+0000","last_became_active":"2021-02-10T04:25:49.010614+0000","last_became_peered":"2021-02-10T04:25:49.010614+0000","last_unstale":"2021-02-10T04:27:41.375191+0000","last_undegraded":"2021-02-10T04:27:41.375191+0000","last_fullsized":"2021-02-10T04:27:41.375191+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:39.268114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:39.268114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"51","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375282+0000","last_change":"2021-02-10T04:26:38.013836+0000","last_active":"2021-02-10T04:27:41.375282+0000","last_peered":"2021-02-10T04:27:41.375282+0000","last_clean":"2021-02-10T04:27:41.375282+0000","last_became_active":"2021-02-10T04:25:36.334230+0000","last_became_peered":"2021-02-10T04:25:36.334230+0000","last_unstale":"2021-02-10T04:27:41.375282+0000","last_undegraded":"2021-02-10T04:27:41.375282+0000","last_fullsized":"2021-02-10T04:27:41.375282+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:38.013770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:38.013770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"48","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375315+0000","last_change":"2021-02-10T04:26:44.252455+0000","last_active":"2021-02-10T04:27:41.375315+0000","last_peered":"2021-02-10T04:27:41.375315+0000","last_clean":"2021-02-10T04:27:41.375315+0000","last_became_active":"2021-02-10T04:25:36.343397+0000","last_became_peered":"2021-02-10T04:25:36.343397+0000","last_unstale":"2021-02-10T04:27:41.375315+0000","last_undegraded":"2021-02-10T04:27:41.375315+0000","last_fullsized":"2021-02-10T04:27:41.375315+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:44.252351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:26:44.252351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"62'15","reported_seq":"66","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.098203+0000","last_change":"2021-02-10T04:27:08.069986+0000","last_active":"2021-02-10T04:27:41.098203+0000","last_peered":"2021-02-10T04:27:41.098203+0000","last_clean":"2021-02-10T04:27:41.098203+0000","last_became_active":"2021-02-10T04:25:49.002471+0000","last_became_peered":"2021-02-10T04:25:49.002471+0000","last_unstale":"2021-02-10T04:27:41.098203+0000","last_undegraded":"2021-02-10T04:27:41.098203+0000","last_fullsized":"2021-02-10T04:27:41.098203+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:27:08.069899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:08.069899+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":15,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375911+0000","last_change":"2021-02-10T04:27:38.106291+0000","last_active":"2021-02-10T04:27:41.375911+0000","last_peered":"2021-02-10T04:27:41.375911+0000","last_clean":"2021-02-10T04:27:41.375911+0000","last_became_active":"2021-02-10T04:25:51.022821+0000","last_became_peered":"2021-02-10T04:25:51.022821+0000","last_unstale":"2021-02-10T04:27:41.375911+0000","last_undegraded":"2021-02-10T04:27:41.375911+0000","last_fullsized":"2021-02-10T04:27:41.375911+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:38.106209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:38.106209+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.100719+0000","last_change":"2021-02-10T04:27:10.051857+0000","last_active":"2021-02-10T04:27:41.100719+0000","last_peered":"2021-02-10T04:27:41.100719+0000","last_clean":"2021-02-10T04:27:41.100719+0000","last_became_active":"2021-02-10T04:25:53.040390+0000","last_became_peered":"2021-02-10T04:25:53.040390+0000","last_unstale":"2021-02-10T04:27:41.100719+0000","last_undegraded":"2021-02-10T04:27:41.100719+0000","last_fullsized":"2021-02-10T04:27:41.100719+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:10.051800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:10.051800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"42","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.098795+0000","last_change":"2021-02-10T04:27:30.657844+0000","last_active":"2021-02-10T04:27:41.098795+0000","last_peered":"2021-02-10T04:27:41.098795+0000","last_clean":"2021-02-10T04:27:41.098795+0000","last_became_active":"2021-02-10T04:25:49.001782+0000","last_became_peered":"2021-02-10T04:25:49.001782+0000","last_unstale":"2021-02-10T04:27:41.098795+0000","last_undegraded":"2021-02-10T04:27:41.098795+0000","last_fullsized":"2021-02-10T04:27:41.098795+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:30.657781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:30.657781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"47","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375221+0000","last_change":"2021-02-10T04:27:08.192835+0000","last_active":"2021-02-10T04:27:41.375221+0000","last_peered":"2021-02-10T04:27:41.375221+0000","last_clean":"2021-02-10T04:27:41.375221+0000","last_became_active":"2021-02-10T04:25:36.340009+0000","last_became_peered":"2021-02-10T04:25:36.340009+0000","last_unstale":"2021-02-10T04:27:41.375221+0000","last_undegraded":"2021-02-10T04:27:41.375221+0000","last_fullsized":"2021-02-10T04:27:41.375221+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:08.192772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:08.192772+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"62'16","reported_seq":"68","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.099853+0000","last_change":"2021-02-10T04:27:09.484054+0000","last_active":"2021-02-10T04:27:41.099853+0000","last_peered":"2021-02-10T04:27:41.099853+0000","last_clean":"2021-02-10T04:27:41.099853+0000","last_became_active":"2021-02-10T04:25:49.003521+0000","last_became_peered":"2021-02-10T04:25:49.003521+0000","last_unstale":"2021-02-10T04:27:41.099853+0000","last_undegraded":"2021-02-10T04:27:41.099853+0000","last_fullsized":"2021-02-10T04:27:41.099853+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'16","last_scrub_stamp":"2021-02-10T04:27:09.483988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:09.483988+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":16,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376842+0000","last_change":"2021-02-10T04:27:25.216335+0000","last_active":"2021-02-10T04:27:41.376842+0000","last_peered":"2021-02-10T04:27:41.376842+0000","last_clean":"2021-02-10T04:27:41.376842+0000","last_became_active":"2021-02-10T04:25:51.032201+0000","last_became_peered":"2021-02-10T04:25:51.032201+0000","last_unstale":"2021-02-10T04:27:41.376842+0000","last_undegraded":"2021-02-10T04:27:41.376842+0000","last_fullsized":"2021-02-10T04:27:41.376842+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:27:25.216260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:42.182071+0000","last_change":"2021-02-10T04:27:42.182019+0000","last_active":"2021-02-10T04:27:42.182071+0000","last_peered":"2021-02-10T04:27:42.182071+0000","last_clean":"2021-02-10T04:27:42.182071+0000","last_became_active":"2021-02-10T04:25:53.149552+0000","last_became_peered":"2021-02-10T04:25:53.149552+0000","last_unstale":"2021-02-10T04:27:42.182071+0000","last_undegraded":"2021-02-10T04:27:42.182071+0000","last_fullsized":"2021-02-10T04:27:42.182071+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:42.181940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:42.181940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"50","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376660+0000","last_change":"2021-02-10T04:26:57.084942+0000","last_active":"2021-02-10T04:27:41.376660+0000","last_peered":"2021-02-10T04:27:41.376660+0000","last_clean":"2021-02-10T04:27:41.376660+0000","last_became_active":"2021-02-10T04:25:36.338600+0000","last_became_peered":"2021-02-10T04:25:36.338600+0000","last_unstale":"2021-02-10T04:27:41.376660+0000","last_undegraded":"2021-02-10T04:27:41.376660+0000","last_fullsized":"2021-02-10T04:27:41.376660+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:26:57.084864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:57.084864+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"63'27","reported_seq":"97","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.099329+0000","last_change":"2021-02-10T04:27:10.457048+0000","last_active":"2021-02-10T04:27:41.099329+0000","last_peered":"2021-02-10T04:27:41.099329+0000","last_clean":"2021-02-10T04:27:41.099329+0000","last_became_active":"2021-02-10T04:25:49.002937+0000","last_became_peered":"2021-02-10T04:25:49.002937+0000","last_unstale":"2021-02-10T04:27:41.099329+0000","last_undegraded":"2021-02-10T04:27:41.099329+0000","last_fullsized":"2021-02-10T04:27:41.099329+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'27","last_scrub_stamp":"2021-02-10T04:27:10.456981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:10.456981+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":38,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.097747+0000","last_change":"2021-02-10T04:26:59.175043+0000","last_active":"2021-02-10T04:27:41.097747+0000","last_peered":"2021-02-10T04:27:41.097747+0000","last_clean":"2021-02-10T04:27:41.097747+0000","last_became_active":"2021-02-10T04:25:51.025636+0000","last_became_peered":"2021-02-10T04:25:51.025636+0000","last_unstale":"2021-02-10T04:27:41.097747+0000","last_undegraded":"2021-02-10T04:27:41.097747+0000","last_fullsized":"2021-02-10T04:27:41.097747+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:59.174908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:26:59.174908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376898+0000","last_change":"2021-02-10T04:27:01.184729+0000","last_active":"2021-02-10T04:27:41.376898+0000","last_peered":"2021-02-10T04:27:41.376898+0000","last_clean":"2021-02-10T04:27:41.376898+0000","last_became_active":"2021-02-10T04:25:53.045085+0000","last_became_peered":"2021-02-10T04:25:53.045085+0000","last_unstale":"2021-02-10T04:27:41.376898+0000","last_undegraded":"2021-02-10T04:27:41.376898+0000","last_fullsized":"2021-02-10T04:27:41.376898+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:01.184630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:01.184630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"53","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.100997+0000","last_change":"2021-02-10T04:27:25.026606+0000","last_active":"2021-02-10T04:27:41.100997+0000","last_peered":"2021-02-10T04:27:41.100997+0000","last_clean":"2021-02-10T04:27:41.100997+0000","last_became_active":"2021-02-10T04:25:35.324960+0000","last_became_peered":"2021-02-10T04:25:35.324960+0000","last_unstale":"2021-02-10T04:27:41.100997+0000","last_undegraded":"2021-02-10T04:27:41.100997+0000","last_fullsized":"2021-02-10T04:27:41.100997+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:27:25.026534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:25.026534+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"63'34","reported_seq":"111","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.374699+0000","last_change":"2021-02-10T04:27:20.342445+0000","last_active":"2021-02-10T04:27:41.374699+0000","last_peered":"2021-02-10T04:27:41.374699+0000","last_clean":"2021-02-10T04:27:41.374699+0000","last_became_active":"2021-02-10T04:25:48.997481+0000","last_became_peered":"2021-02-10T04:25:48.997481+0000","last_unstale":"2021-02-10T04:27:41.374699+0000","last_undegraded":"2021-02-10T04:27:41.374699+0000","last_fullsized":"2021-02-10T04:27:41.374699+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'34","last_scrub_stamp":"2021-02-10T04:27:20.342358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:20.342358+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":70,"num_read_kb":48,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:44.501196+0000","last_change":"2021-02-10T04:27:44.501111+0000","last_active":"2021-02-10T04:27:44.501196+0000","last_peered":"2021-02-10T04:27:44.501196+0000","last_clean":"2021-02-10T04:27:44.501196+0000","last_became_active":"2021-02-10T04:25:51.018706+0000","last_became_peered":"2021-02-10T04:25:51.018706+0000","last_unstale":"2021-02-10T04:27:44.501196+0000","last_undegraded":"2021-02-10T04:27:44.501196+0000","last_fullsized":"2021-02-10T04:27:44.501196+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:44.501024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:44.501024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375850+0000","last_change":"2021-02-10T04:27:26.234432+0000","last_active":"2021-02-10T04:27:41.375850+0000","last_peered":"2021-02-10T04:27:41.375850+0000","last_clean":"2021-02-10T04:27:41.375850+0000","last_became_active":"2021-02-10T04:25:53.150930+0000","last_became_peered":"2021-02-10T04:25:53.150930+0000","last_unstale":"2021-02-10T04:27:41.375850+0000","last_undegraded":"2021-02-10T04:27:41.375850+0000","last_fullsized":"2021-02-10T04:27:41.375850+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:26.234349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:26.234349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"58","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.098256+0000","last_change":"2021-02-10T04:26:44.264500+0000","last_active":"2021-02-10T04:27:41.098256+0000","last_peered":"2021-02-10T04:27:41.098256+0000","last_clean":"2021-02-10T04:27:41.098256+0000","last_became_active":"2021-02-10T04:25:36.339614+0000","last_became_peered":"2021-02-10T04:25:36.339614+0000","last_unstale":"2021-02-10T04:27:41.098256+0000","last_undegraded":"2021-02-10T04:27:41.098256+0000","last_fullsized":"2021-02-10T04:27:41.098256+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:44.264402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:44.264402+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"63'26","reported_seq":"90","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.378335+0000","last_change":"2021-02-10T04:27:26.403830+0000","last_active":"2021-02-10T04:27:41.378335+0000","last_peered":"2021-02-10T04:27:41.378335+0000","last_clean":"2021-02-10T04:27:41.378335+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:27:41.378335+0000","last_undegraded":"2021-02-10T04:27:41.378335+0000","last_fullsized":"2021-02-10T04:27:41.378335+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:27:26.403748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:26.403748+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":51,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"58'2","reported_seq":"65","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.116283+0000","last_change":"2021-02-10T04:27:33.210495+0000","last_active":"2021-02-10T04:27:47.116283+0000","last_peered":"2021-02-10T04:27:47.116283+0000","last_clean":"2021-02-10T04:27:47.116283+0000","last_became_active":"2021-02-10T04:25:51.031906+0000","last_became_peered":"2021-02-10T04:25:51.031906+0000","last_unstale":"2021-02-10T04:27:47.116283+0000","last_undegraded":"2021-02-10T04:27:47.116283+0000","last_fullsized":"2021-02-10T04:27:47.116283+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:33.210431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:33.210431+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"32","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375092+0000","last_change":"2021-02-10T04:25:53.148449+0000","last_active":"2021-02-10T04:27:41.375092+0000","last_peered":"2021-02-10T04:27:41.375092+0000","last_clean":"2021-02-10T04:27:41.375092+0000","last_became_active":"2021-02-10T04:25:53.148054+0000","last_became_peered":"2021-02-10T04:25:53.148054+0000","last_unstale":"2021-02-10T04:27:41.375092+0000","last_undegraded":"2021-02-10T04:27:41.375092+0000","last_fullsized":"2021-02-10T04:27:41.375092+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:25:52.015519+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376771+0000","last_change":"2021-02-10T04:26:58.112766+0000","last_active":"2021-02-10T04:27:41.376771+0000","last_peered":"2021-02-10T04:27:41.376771+0000","last_clean":"2021-02-10T04:27:41.376771+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:27:41.376771+0000","last_undegraded":"2021-02-10T04:27:41.376771+0000","last_fullsized":"2021-02-10T04:27:41.376771+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:58.112683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:26:58.112683+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"47","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375139+0000","last_change":"2021-02-10T04:27:04.148397+0000","last_active":"2021-02-10T04:27:41.375139+0000","last_peered":"2021-02-10T04:27:41.375139+0000","last_clean":"2021-02-10T04:27:41.375139+0000","last_became_active":"2021-02-10T04:25:36.340895+0000","last_became_peered":"2021-02-10T04:25:36.340895+0000","last_unstale":"2021-02-10T04:27:41.375139+0000","last_undegraded":"2021-02-10T04:27:41.375139+0000","last_fullsized":"2021-02-10T04:27:41.375139+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:04.148329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:04.148329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"63'21","reported_seq":"84","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.378009+0000","last_change":"2021-02-10T04:26:59.205120+0000","last_active":"2021-02-10T04:27:41.378009+0000","last_peered":"2021-02-10T04:27:41.378009+0000","last_clean":"2021-02-10T04:27:41.378009+0000","last_became_active":"2021-02-10T04:25:49.011093+0000","last_became_peered":"2021-02-10T04:25:49.011093+0000","last_unstale":"2021-02-10T04:27:41.378009+0000","last_undegraded":"2021-02-10T04:27:41.378009+0000","last_fullsized":"2021-02-10T04:27:41.378009+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2021-02-10T04:26:59.205037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:26:59.205037+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":28,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376564+0000","last_change":"2021-02-10T04:27:24.209544+0000","last_active":"2021-02-10T04:27:41.376564+0000","last_peered":"2021-02-10T04:27:41.376564+0000","last_clean":"2021-02-10T04:27:41.376564+0000","last_became_active":"2021-02-10T04:25:53.042697+0000","last_became_peered":"2021-02-10T04:25:53.042697+0000","last_unstale":"2021-02-10T04:27:41.376564+0000","last_undegraded":"2021-02-10T04:27:41.376564+0000","last_fullsized":"2021-02-10T04:27:41.376564+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:27:24.209460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"78'3","reported_seq":"67","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.117041+0000","last_change":"2021-02-10T04:27:13.268467+0000","last_active":"2021-02-10T04:27:47.117041+0000","last_peered":"2021-02-10T04:27:47.117041+0000","last_clean":"2021-02-10T04:27:47.117041+0000","last_became_active":"2021-02-10T04:25:51.025218+0000","last_became_peered":"2021-02-10T04:25:51.025218+0000","last_unstale":"2021-02-10T04:27:47.117041+0000","last_undegraded":"2021-02-10T04:27:47.117041+0000","last_fullsized":"2021-02-10T04:27:47.117041+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:27:13.268412+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"53","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.099760+0000","last_change":"2021-02-10T04:26:46.092082+0000","last_active":"2021-02-10T04:27:41.099760+0000","last_peered":"2021-02-10T04:27:41.099760+0000","last_clean":"2021-02-10T04:27:41.099760+0000","last_became_active":"2021-02-10T04:25:36.331437+0000","last_became_peered":"2021-02-10T04:25:36.331437+0000","last_unstale":"2021-02-10T04:27:41.099760+0000","last_undegraded":"2021-02-10T04:27:41.099760+0000","last_fullsized":"2021-02-10T04:27:41.099760+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:46.092013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:46.092013+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"62'17","reported_seq":"71","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.101049+0000","last_change":"2021-02-10T04:27:30.012988+0000","last_active":"2021-02-10T04:27:41.101049+0000","last_peered":"2021-02-10T04:27:41.101049+0000","last_clean":"2021-02-10T04:27:41.101049+0000","last_became_active":"2021-02-10T04:25:49.004933+0000","last_became_peered":"2021-02-10T04:25:49.004933+0000","last_unstale":"2021-02-10T04:27:41.101049+0000","last_undegraded":"2021-02-10T04:27:41.101049+0000","last_fullsized":"2021-02-10T04:27:41.101049+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'17","last_scrub_stamp":"2021-02-10T04:27:30.012927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:30.012927+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.377864+0000","last_change":"2021-02-10T04:27:33.382403+0000","last_active":"2021-02-10T04:27:41.377864+0000","last_peered":"2021-02-10T04:27:41.377864+0000","last_clean":"2021-02-10T04:27:41.377864+0000","last_became_active":"2021-02-10T04:25:53.043027+0000","last_became_peered":"2021-02-10T04:25:53.043027+0000","last_unstale":"2021-02-10T04:27:41.377864+0000","last_undegraded":"2021-02-10T04:27:41.377864+0000","last_fullsized":"2021-02-10T04:27:41.377864+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:33.382319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:33.382319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"58'2","reported_seq":"65","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.116957+0000","last_change":"2021-02-10T04:26:50.624141+0000","last_active":"2021-02-10T04:27:47.116957+0000","last_peered":"2021-02-10T04:27:47.116957+0000","last_clean":"2021-02-10T04:27:47.116957+0000","last_became_active":"2021-02-10T04:25:51.025268+0000","last_became_peered":"2021-02-10T04:25:51.025268+0000","last_unstale":"2021-02-10T04:27:47.116957+0000","last_undegraded":"2021-02-10T04:27:47.116957+0000","last_fullsized":"2021-02-10T04:27:47.116957+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:26:50.624072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:26:50.624072+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"51","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.100475+0000","last_change":"2021-02-10T04:27:17.003396+0000","last_active":"2021-02-10T04:27:41.100475+0000","last_peered":"2021-02-10T04:27:41.100475+0000","last_clean":"2021-02-10T04:27:41.100475+0000","last_became_active":"2021-02-10T04:25:36.333883+0000","last_became_peered":"2021-02-10T04:25:36.333883+0000","last_unstale":"2021-02-10T04:27:41.100475+0000","last_undegraded":"2021-02-10T04:27:41.100475+0000","last_fullsized":"2021-02-10T04:27:41.100475+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:17.003304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:17.003304+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"62'13","reported_seq":"63","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.377063+0000","last_change":"2021-02-10T04:27:14.256603+0000","last_active":"2021-02-10T04:27:41.377063+0000","last_peered":"2021-02-10T04:27:41.377063+0000","last_clean":"2021-02-10T04:27:41.377063+0000","last_became_active":"2021-02-10T04:25:49.001725+0000","last_became_peered":"2021-02-10T04:25:49.001725+0000","last_unstale":"2021-02-10T04:27:41.377063+0000","last_undegraded":"2021-02-10T04:27:41.377063+0000","last_fullsized":"2021-02-10T04:27:41.377063+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'13","last_scrub_stamp":"2021-02-10T04:27:14.256539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:14.256539+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"37","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375579+0000","last_change":"2021-02-10T04:26:55.627032+0000","last_active":"2021-02-10T04:27:41.375579+0000","last_peered":"2021-02-10T04:27:41.375579+0000","last_clean":"2021-02-10T04:27:41.375579+0000","last_became_active":"2021-02-10T04:25:53.038218+0000","last_became_peered":"2021-02-10T04:25:53.038218+0000","last_unstale":"2021-02-10T04:27:41.375579+0000","last_undegraded":"2021-02-10T04:27:41.375579+0000","last_fullsized":"2021-02-10T04:27:41.375579+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:55.626967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:26:55.626967+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.100713+0000","last_change":"2021-02-10T04:27:37.160395+0000","last_active":"2021-02-10T04:27:41.100713+0000","last_peered":"2021-02-10T04:27:41.100713+0000","last_clean":"2021-02-10T04:27:41.100713+0000","last_became_active":"2021-02-10T04:25:51.028302+0000","last_became_peered":"2021-02-10T04:25:51.028302+0000","last_unstale":"2021-02-10T04:27:41.100713+0000","last_undegraded":"2021-02-10T04:27:41.100713+0000","last_fullsized":"2021-02-10T04:27:41.100713+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:37.160328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:37.160328+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"47","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375998+0000","last_change":"2021-02-10T04:27:00.052115+0000","last_active":"2021-02-10T04:27:41.375998+0000","last_peered":"2021-02-10T04:27:41.375998+0000","last_clean":"2021-02-10T04:27:41.375998+0000","last_became_active":"2021-02-10T04:25:36.344109+0000","last_became_peered":"2021-02-10T04:25:36.344109+0000","last_unstale":"2021-02-10T04:27:41.375998+0000","last_undegraded":"2021-02-10T04:27:41.375998+0000","last_fullsized":"2021-02-10T04:27:41.375998+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:00.052028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:00.052028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"62'21","reported_seq":"78","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.377744+0000","last_change":"2021-02-10T04:27:00.215115+0000","last_active":"2021-02-10T04:27:41.377744+0000","last_peered":"2021-02-10T04:27:41.377744+0000","last_clean":"2021-02-10T04:27:41.377744+0000","last_became_active":"2021-02-10T04:25:49.011699+0000","last_became_peered":"2021-02-10T04:25:49.011699+0000","last_unstale":"2021-02-10T04:27:41.377744+0000","last_undegraded":"2021-02-10T04:27:41.377744+0000","last_fullsized":"2021-02-10T04:27:41.377744+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'21","last_scrub_stamp":"2021-02-10T04:27:00.215037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:00.215037+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375050+0000","last_change":"2021-02-10T04:27:35.166330+0000","last_active":"2021-02-10T04:27:41.375050+0000","last_peered":"2021-02-10T04:27:41.375050+0000","last_clean":"2021-02-10T04:27:41.375050+0000","last_became_active":"2021-02-10T04:25:53.037903+0000","last_became_peered":"2021-02-10T04:25:53.037903+0000","last_unstale":"2021-02-10T04:27:41.375050+0000","last_undegraded":"2021-02-10T04:27:41.375050+0000","last_fullsized":"2021-02-10T04:27:41.375050+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:35.166250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:35.166250+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"39","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.374929+0000","last_change":"2021-02-10T04:27:11.606664+0000","last_active":"2021-02-10T04:27:41.374929+0000","last_peered":"2021-02-10T04:27:41.374929+0000","last_clean":"2021-02-10T04:27:41.374929+0000","last_became_active":"2021-02-10T04:25:51.025882+0000","last_became_peered":"2021-02-10T04:25:51.025882+0000","last_unstale":"2021-02-10T04:27:41.374929+0000","last_undegraded":"2021-02-10T04:27:41.374929+0000","last_fullsized":"2021-02-10T04:27:41.374929+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:11.606603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:11.606603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"62'2","reported_seq":"73","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.116932+0000","last_change":"2021-02-10T04:26:48.599352+0000","last_active":"2021-02-10T04:27:47.116932+0000","last_peered":"2021-02-10T04:27:47.116932+0000","last_clean":"2021-02-10T04:27:47.116932+0000","last_became_active":"2021-02-10T04:25:34.331199+0000","last_became_peered":"2021-02-10T04:25:34.331199+0000","last_unstale":"2021-02-10T04:27:47.116932+0000","last_undegraded":"2021-02-10T04:27:47.116932+0000","last_fullsized":"2021-02-10T04:27:47.116932+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2021-02-10T04:26:48.599257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:48.599257+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"63'29","reported_seq":"98","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.097608+0000","last_change":"2021-02-10T04:27:13.081465+0000","last_active":"2021-02-10T04:27:41.097608+0000","last_peered":"2021-02-10T04:27:41.097608+0000","last_clean":"2021-02-10T04:27:41.097608+0000","last_became_active":"2021-02-10T04:25:49.015913+0000","last_became_peered":"2021-02-10T04:25:49.015913+0000","last_unstale":"2021-02-10T04:27:41.097608+0000","last_undegraded":"2021-02-10T04:27:41.097608+0000","last_fullsized":"2021-02-10T04:27:41.097608+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'29","last_scrub_stamp":"2021-02-10T04:27:13.081399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:13.081399+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":56,"num_read_kb":36,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.098541+0000","last_change":"2021-02-10T04:27:22.512316+0000","last_active":"2021-02-10T04:27:41.098541+0000","last_peered":"2021-02-10T04:27:41.098541+0000","last_clean":"2021-02-10T04:27:41.098541+0000","last_became_active":"2021-02-10T04:25:53.038249+0000","last_became_peered":"2021-02-10T04:25:53.038249+0000","last_unstale":"2021-02-10T04:27:41.098541+0000","last_undegraded":"2021-02-10T04:27:41.098541+0000","last_fullsized":"2021-02-10T04:27:41.098541+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:22.512245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:22.512245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"39","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375199+0000","last_change":"2021-02-10T04:26:51.605291+0000","last_active":"2021-02-10T04:27:41.375199+0000","last_peered":"2021-02-10T04:27:41.375199+0000","last_clean":"2021-02-10T04:27:41.375199+0000","last_became_active":"2021-02-10T04:25:51.016118+0000","last_became_peered":"2021-02-10T04:25:51.016118+0000","last_unstale":"2021-02-10T04:27:41.375199+0000","last_undegraded":"2021-02-10T04:27:41.375199+0000","last_fullsized":"2021-02-10T04:27:41.375199+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:51.605227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:26:51.605227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"49","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.100906+0000","last_change":"2021-02-10T04:26:50.001018+0000","last_active":"2021-02-10T04:27:41.100906+0000","last_peered":"2021-02-10T04:27:41.100906+0000","last_clean":"2021-02-10T04:27:41.100906+0000","last_became_active":"2021-02-10T04:25:34.334427+0000","last_became_peered":"2021-02-10T04:25:34.334427+0000","last_unstale":"2021-02-10T04:27:41.100906+0000","last_undegraded":"2021-02-10T04:27:41.100906+0000","last_fullsized":"2021-02-10T04:27:41.100906+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:50.000952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:50.000952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"63'20","reported_seq":"83","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.099533+0000","last_change":"2021-02-10T04:27:20.587671+0000","last_active":"2021-02-10T04:27:41.099533+0000","last_peered":"2021-02-10T04:27:41.099533+0000","last_clean":"2021-02-10T04:27:41.099533+0000","last_became_active":"2021-02-10T04:25:49.004346+0000","last_became_peered":"2021-02-10T04:25:49.004346+0000","last_unstale":"2021-02-10T04:27:41.099533+0000","last_undegraded":"2021-02-10T04:27:41.099533+0000","last_fullsized":"2021-02-10T04:27:41.099533+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'20","last_scrub_stamp":"2021-02-10T04:27:20.587602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:20.587602+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":27,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376021+0000","last_change":"2021-02-10T04:27:13.006848+0000","last_active":"2021-02-10T04:27:41.376021+0000","last_peered":"2021-02-10T04:27:41.376021+0000","last_clean":"2021-02-10T04:27:41.376021+0000","last_became_active":"2021-02-10T04:25:53.148090+0000","last_became_peered":"2021-02-10T04:25:53.148090+0000","last_unstale":"2021-02-10T04:27:41.376021+0000","last_undegraded":"2021-02-10T04:27:41.376021+0000","last_fullsized":"2021-02-10T04:27:41.376021+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:27:13.006716+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"62","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375797+0000","last_change":"2021-02-10T04:26:44.609990+0000","last_active":"2021-02-10T04:27:41.375797+0000","last_peered":"2021-02-10T04:27:41.375797+0000","last_clean":"2021-02-10T04:27:41.375797+0000","last_became_active":"2021-02-10T04:25:36.344641+0000","last_became_peered":"2021-02-10T04:25:36.344641+0000","last_unstale":"2021-02-10T04:27:41.375797+0000","last_undegraded":"2021-02-10T04:27:41.375797+0000","last_fullsized":"2021-02-10T04:27:41.375797+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:26:44.609922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:44.609922+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"63'30","reported_seq":"103","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375135+0000","last_change":"2021-02-10T04:27:02.039902+0000","last_active":"2021-02-10T04:27:41.375135+0000","last_peered":"2021-02-10T04:27:41.375135+0000","last_clean":"2021-02-10T04:27:41.375135+0000","last_became_active":"2021-02-10T04:25:49.002058+0000","last_became_peered":"2021-02-10T04:25:49.002058+0000","last_unstale":"2021-02-10T04:27:41.375135+0000","last_undegraded":"2021-02-10T04:27:41.375135+0000","last_fullsized":"2021-02-10T04:27:41.375135+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2021-02-10T04:27:02.039813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:02.039813+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":59,"num_read_kb":42,"num_write":31,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"58'2","reported_seq":"66","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.728076+0000","last_change":"2021-02-10T04:27:20.144947+0000","last_active":"2021-02-10T04:27:47.728076+0000","last_peered":"2021-02-10T04:27:47.728076+0000","last_clean":"2021-02-10T04:27:47.728076+0000","last_became_active":"2021-02-10T04:25:51.019515+0000","last_became_peered":"2021-02-10T04:25:51.019515+0000","last_unstale":"2021-02-10T04:27:47.728076+0000","last_undegraded":"2021-02-10T04:27:47.728076+0000","last_fullsized":"2021-02-10T04:27:47.728076+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:20.144855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:20.144855+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":1453,"num_read_kb":926,"num_write":734,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":746,"ondisk_log_size":746,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":116,"num_read_kb":111,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1945,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":21,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":5835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748364829,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6564,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738588,"statfs":{"total":95995035648,"available":95988314112,"internally_reserved":0,"allocated":880640,"data_stored":313320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.876,"15min":0.88200000000000001},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.85599999999999998,"15min":0.85899999999999999},"min":{"1min":0.57499999999999996,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.248,"5min":1.248,"15min":1.248},"last":1.0289999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.88700000000000001,"15min":0.89100000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":1.0049999999999999},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.0149999999999999,"15min":1.0209999999999999},"min":{"1min":0.64500000000000002,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.355,"5min":1.369,"15min":1.369},"last":0.78600000000000003}]},{"osd":2,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.018,"15min":1.026},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.95799999999999996,"15min":0.95999999999999996},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.93100000000000005}]},{"osd":3,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.998,"15min":1.01},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.147,"15min":1.153},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.381,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":1.0800000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":0.97499999999999998,"15min":0.95099999999999996},"min":{"1min":0.77300000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.2530000000000001,"5min":2.2530000000000001,"15min":2.2530000000000001},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.1060000000000001,"15min":1.1120000000000001},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.492,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":1.256}]},{"osd":5,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.161,"15min":1.1499999999999999},"min":{"1min":0.68899999999999995,"5min":0.624,"15min":0.624},"max":{"1min":1.714,"5min":1.714,"15min":1.714},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.2,"15min":1.1990000000000001},"min":{"1min":0.90900000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.498,"5min":1.538,"15min":1.538},"last":1.107}]},{"osd":6,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.246,"15min":1.2430000000000001},"min":{"1min":0.94199999999999995,"5min":0.86899999999999999,"15min":0.86899999999999999},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.276,"5min":1.236,"15min":1.2290000000000001},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":1.127}]}]},{"osd":6,"up_from":43,"seq":184683593761,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6608,"kb_used_data":904,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738544,"statfs":{"total":95995035648,"available":95988269056,"internally_reserved":0,"allocated":925696,"data_stored":316190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:18 2021","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.3460000000000001,"5min":1.478,"15min":1.478},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.95499999999999996,"5min":1.008,"15min":1.0169999999999999},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.626}]},{"osd":1,"last update":"Wed Feb 10 04:27:18 2021","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":1.0149999999999999,"15min":1.03},"min":{"1min":0.61599999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":2.121,"5min":2.121,"15min":2.121},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.042,"5min":0.99099999999999999,"15min":0.98299999999999998},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":0.78100000000000003}]},{"osd":2,"last update":"Wed Feb 10 04:27:18 2021","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0620000000000001,"15min":1.0609999999999999},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":0.90600000000000003},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.0289999999999999,"15min":1.014},"min":{"1min":0.73699999999999999,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.1920000000000002,"5min":2.1920000000000002,"15min":2.1920000000000002},"last":1.143}]},{"osd":3,"last update":"Wed Feb 10 04:27:18 2021","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.1439999999999999,"15min":1.1379999999999999},"min":{"1min":0.85699999999999998,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":2,"5min":2,"15min":2},"last":1.177},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.133,"15min":1.107},"min":{"1min":0.88,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":2.323,"5min":2.323,"15min":2.323},"last":0.876}]},{"osd":4,"last update":"Wed Feb 10 04:27:18 2021","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":0.99099999999999999,"15min":0.96099999999999997},"min":{"1min":0.61699999999999999,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.0800000000000001,"5min":2.0800000000000001,"15min":2.0800000000000001},"last":1.024},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.90500000000000003,"15min":0.89200000000000002},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.94299999999999995}]},{"osd":5,"last update":"Wed Feb 10 04:27:18 2021","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.0009999999999999,"15min":0.98699999999999999},"min":{"1min":0.67100000000000004,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.7130000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.419,"5min":1.179,"15min":1.139},"min":{"1min":0.83999999999999997,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.4550000000000001,"5min":2.4550000000000001,"15min":2.4550000000000001},"last":1.258}]},{"osd":7,"last update":"Wed Feb 10 04:27:40 2021","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.1060000000000001,"15min":1.0940000000000001},"min":{"1min":0.75800000000000001,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.901,"5min":1.901,"15min":1.901},"last":1.22},{"interface":"front","average":{"1min":1.506,"5min":1.409,"15min":1.393},"min":{"1min":0.90300000000000002,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":2.5339999999999998,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":1.333}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":11292,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":10304,"kb_avail":93733860,"statfs":{"total":95995035648,"available":95983472640,"internally_reserved":0,"allocated":1003520,"data_stored":313080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":10551296},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:26:53 2021","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.66600000000000004,"15min":0.626},"min":{"1min":0.56899999999999995,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.425,"5min":1.425,"15min":1.425},"last":1.179},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.746,"15min":0.69199999999999995},"min":{"1min":0.57299999999999995,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.68999999999999995}]},{"osd":2,"last update":"Wed Feb 10 04:27:14 2021","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.80800000000000005,"15min":0.76400000000000001},"min":{"1min":0.58199999999999996,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.83499999999999996,"15min":0.79900000000000004},"min":{"1min":0.51000000000000001,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":0.84399999999999997}]},{"osd":3,"last update":"Wed Feb 10 04:27:31 2021","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.93100000000000005,"15min":0.91400000000000003},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.5860000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.026,"5min":1.018,"15min":1.014},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.226,"5min":2.226,"15min":2.226},"last":1.0129999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:27:46 2021","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.157,"15min":1.1459999999999999},"min":{"1min":0.86699999999999999,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":1.736,"5min":2.004,"15min":2.004},"last":1.536},{"interface":"front","average":{"1min":1.23,"5min":1.218,"15min":1.21},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.6419999999999999,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":1.143}]},{"osd":5,"last update":"Wed Feb 10 04:27:08 2021","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.25,"15min":1.2769999999999999},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":2.7170000000000001,"5min":5.5860000000000003,"15min":5.5860000000000003},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.6100000000000001,"15min":1.6559999999999999},"min":{"1min":0.97399999999999998,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":2.105,"5min":9.5310000000000006,"15min":9.5310000000000006},"last":1.6659999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:27:24 2021","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.2889999999999999,"15min":1.29},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.4449999999999998,"5min":2.4449999999999998,"15min":2.4449999999999998},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.264,"15min":1.292},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.905,"5min":3.653,"15min":3.653},"last":1.274}]},{"osd":7,"last update":"Wed Feb 10 04:27:36 2021","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.639,"15min":1.6890000000000001},"min":{"1min":1.0529999999999999,"5min":1.004,"15min":1.004},"max":{"1min":2.2090000000000001,"5min":7.5540000000000003,"15min":7.5540000000000003},"last":1.7849999999999999},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.623,"15min":1.6619999999999999},"min":{"1min":1.006,"5min":1.006,"15min":1.006},"max":{"1min":2.556,"5min":8.1020000000000003,"15min":8.1020000000000003},"last":1.415}]}]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6808,"kb_used_data":848,"kb_used_omap":0,"kb_used_meta":5952,"kb_avail":93738344,"statfs":{"total":95995035648,"available":95988064256,"internally_reserved":0,"allocated":868352,"data_stored":313029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6094848},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 04:26:53 2021","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.64300000000000002,"15min":0.58699999999999997},"min":{"1min":0.53100000000000003,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.006,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.79100000000000004,"15min":0.73299999999999998},"min":{"1min":0.67400000000000004,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":0.65500000000000003}]},{"osd":2,"last update":"Wed Feb 10 04:27:12 2021","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.79300000000000004,"15min":0.77300000000000002},"min":{"1min":0.52500000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.958,"5min":1.958,"15min":1.958},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.79700000000000004,"15min":0.79300000000000004},"min":{"1min":0.56100000000000005,"5min":0.495,"15min":0.495},"max":{"1min":1.4390000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.92800000000000005}]},{"osd":3,"last update":"Wed Feb 10 04:27:27 2021","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.80400000000000005,"15min":0.81200000000000006},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.224,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":1.0680000000000001},{"interface":"front","average":{"1min":0.89500000000000002,"5min":1.026,"15min":1.036},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.3049999999999999,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":1.377}]},{"osd":4,"last update":"Wed Feb 10 04:27:45 2021","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.1259999999999999,"15min":1.169},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.3859999999999999,"5min":1.613,"15min":1.613},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.089,"5min":1.141,"15min":1.1539999999999999},"min":{"1min":0.76600000000000001,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.522,"5min":1.597,"15min":1.597},"last":1.2090000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:27:01 2021","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.2,"15min":1.2190000000000001},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":1.3149999999999999},{"interface":"front","average":{"1min":1.218,"5min":1.2889999999999999,"15min":1.3009999999999999},"min":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"max":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.51}]},{"osd":6,"last update":"Wed Feb 10 04:27:20 2021","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.2909999999999999,"15min":1.3029999999999999},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":1.601,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":1.405},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.351,"15min":1.363},"min":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":2.1019999999999999,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":1.6200000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.2330000000000001,"15min":1.252},"min":{"1min":0.94599999999999995,"5min":0.93500000000000005,"15min":0.93500000000000005},"max":{"1min":1.5900000000000001,"5min":1.76,"15min":1.76},"last":1.155},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3500000000000001,"15min":1.3640000000000001},"min":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"max":{"1min":1.611,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":1.4810000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411377,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6552,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738600,"statfs":{"total":95995035648,"available":95988326400,"internally_reserved":0,"allocated":802816,"data_stored":311388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:10 2021","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.70899999999999996,"15min":0.68300000000000005},"min":{"1min":0.54400000000000004,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.97999999999999998,"5min":1.349,"15min":1.349},"last":0.497},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.73299999999999998,"15min":0.71099999999999997},"min":{"1min":0.59499999999999997,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.96399999999999997,"5min":1.274,"15min":1.274},"last":0.54100000000000004}]},{"osd":1,"last update":"Wed Feb 10 04:27:10 2021","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.81200000000000006,"15min":0.76900000000000002},"min":{"1min":0.57499999999999996,"5min":0.495,"15min":0.495},"max":{"1min":1.3069999999999999,"5min":1.726,"15min":1.726},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.71299999999999997,"15min":0.69099999999999995},"min":{"1min":0.54900000000000004,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.58299999999999996}]},{"osd":3,"last update":"Wed Feb 10 04:27:31 2021","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.81499999999999995,"15min":0.80400000000000005},"min":{"1min":0.57999999999999996,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.90700000000000003,"5min":1.274,"15min":1.274},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.996,"15min":0.97099999999999997},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.2,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":0.877}]},{"osd":4,"last update":"Wed Feb 10 04:27:44 2021","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.181,"15min":1.181},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.4359999999999999,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":1.2609999999999999},{"interface":"front","average":{"1min":1.141,"5min":1.119,"15min":1.1220000000000001},"min":{"1min":0.80900000000000005,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":0.99399999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:26:59 2021","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.3180000000000001,"15min":1.3169999999999999},"min":{"1min":1.014,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":1.71,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":1.137},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.1879999999999999,"15min":1.1819999999999999},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":1.24}]},{"osd":6,"last update":"Wed Feb 10 04:27:23 2021","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.284,"15min":1.2989999999999999},"min":{"1min":0.875,"5min":0.875,"15min":0.875},"max":{"1min":1.504,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.26,"5min":1.409,"15min":1.4339999999999999},"min":{"1min":1.05,"5min":1.0489999999999999,"15min":1.0489999999999999},"max":{"1min":1.635,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.0629999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.4690000000000001,"15min":1.494},"min":{"1min":1.077,"5min":1.073,"15min":1.073},"max":{"1min":1.653,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":1.198},{"interface":"front","average":{"1min":1.171,"5min":1.252,"15min":1.266},"min":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"max":{"1min":1.52,"5min":1.6160000000000001,"15min":1.6160000000000001},"last":1.0820000000000001}]}]},{"osd":3,"up_from":25,"seq":107374182443,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6988,"kb_used_data":964,"kb_used_omap":0,"kb_used_meta":6016,"kb_avail":93738164,"statfs":{"total":95995035648,"available":95987879936,"internally_reserved":0,"allocated":987136,"data_stored":317370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6160384},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:28 2021","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.74299999999999999,"15min":0.73499999999999999},"min":{"1min":0.50900000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.0600000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.88300000000000001,"15min":0.86699999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":4.2759999999999998,"5min":4.2759999999999998,"15min":4.2759999999999998},"last":0.56200000000000006}]},{"osd":1,"last update":"Wed Feb 10 04:27:28 2021","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.90300000000000002,"15min":0.877},"min":{"1min":0.64200000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":3.246,"5min":3.246,"15min":3.246},"last":0.66900000000000004},{"interface":"front","average":{"1min":1.0549999999999999,"5min":0.89700000000000002,"15min":0.86199999999999999},"min":{"1min":0.57999999999999996,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":5.4039999999999999,"5min":5.4039999999999999,"15min":5.4039999999999999},"last":0.60799999999999998}]},{"osd":2,"last update":"Wed Feb 10 04:27:28 2021","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":0.97199999999999998,"15min":0.95099999999999996},"min":{"1min":0.60299999999999998,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":4.2309999999999999,"5min":4.2309999999999999,"15min":4.2309999999999999},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.96099999999999997,"15min":0.97599999999999998},"min":{"1min":0.65900000000000003,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.2609999999999999,"5min":1.681,"15min":1.681},"last":0.998}]},{"osd":4,"last update":"Wed Feb 10 04:27:43 2021","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.25,"15min":1.204},"min":{"1min":0.84099999999999997,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.9650000000000001,"5min":3.371,"15min":3.371},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.266,"5min":1.266,"15min":1.2250000000000001},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":1.8280000000000001,"5min":3.5430000000000001,"15min":3.5430000000000001},"last":1.236}]},{"osd":5,"last update":"Wed Feb 10 04:27:00 2021","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.2350000000000001,"15min":1.1930000000000001},"min":{"1min":0.94799999999999995,"5min":0.88500000000000001,"15min":0.88500000000000001},"max":{"1min":3.496,"5min":3.496,"15min":3.496},"last":1.069},{"interface":"front","average":{"1min":1.466,"5min":1.3,"15min":1.272},"min":{"1min":1.143,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":3.742,"5min":3.742,"15min":3.742},"last":1.1739999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:27:24 2021","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.429,"15min":1.4059999999999999},"min":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"max":{"1min":3.6859999999999999,"5min":3.6859999999999999,"15min":3.6859999999999999},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.514,"5min":1.47,"15min":1.462},"min":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"max":{"1min":3.8759999999999999,"5min":3.8759999999999999,"15min":3.8759999999999999},"last":1.0509999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:27:40 2021","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":1.5109999999999999,"15min":1.5049999999999999},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"max":{"1min":4.1319999999999997,"5min":4.1319999999999997,"15min":4.1319999999999997},"last":1.262},{"interface":"front","average":{"1min":1.587,"5min":1.5629999999999999,"15min":1.5589999999999999},"min":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"max":{"1min":4.0010000000000003,"5min":4.0010000000000003,"15min":4.0010000000000003},"last":1.1140000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986217,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6812,"kb_used_data":916,"kb_used_omap":0,"kb_used_meta":5888,"kb_avail":93738340,"statfs":{"total":95995035648,"available":95988060160,"internally_reserved":0,"allocated":937984,"data_stored":316129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6029312},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:43 2021","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.96599999999999997,"15min":0.997},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.95499999999999996,"15min":0.97499999999999998},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.4810000000000001,"5min":1.633,"15min":1.633},"last":0.85999999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:27:43 2021","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.89500000000000002,"15min":0.91700000000000004},"min":{"1min":0.58199999999999996,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":1.02,"5min":1.012,"15min":1.018},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.583,"5min":1.583,"15min":1.583},"last":1.0920000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:27:43 2021","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.0800000000000001,"15min":1.083},"min":{"1min":0.69399999999999995,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.766,"5min":1.766,"15min":1.766},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.1719999999999999,"15min":1.161},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.5469999999999999,"5min":1.768,"15min":1.768},"last":1.0129999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:27:43 2021","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.097,"15min":1.0609999999999999},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":5.1230000000000002,"5min":5.1230000000000002,"15min":5.1230000000000002},"last":1.147},{"interface":"front","average":{"1min":1.302,"5min":1.129,"15min":1.1080000000000001},"min":{"1min":0.79900000000000004,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":5.1710000000000003,"5min":5.1710000000000003,"15min":5.1710000000000003},"last":0.90700000000000003}]},{"osd":5,"last update":"Wed Feb 10 04:27:00 2021","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":0.88,"15min":0.83599999999999997},"min":{"1min":0.63200000000000001,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":1.389},{"interface":"front","average":{"1min":1.05,"5min":0.86599999999999999,"15min":0.83599999999999997},"min":{"1min":0.74099999999999999,"5min":0.624,"15min":0.624},"max":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.5740000000000001},"last":1.075}]},{"osd":6,"last update":"Wed Feb 10 04:27:19 2021","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.0840000000000001,"15min":1.0640000000000001},"min":{"1min":0.79200000000000004,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.27,"15min":1.252},"min":{"1min":1.139,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.714,"5min":1.74,"15min":1.74},"last":1.3080000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:27:40 2021","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.94899999999999995,"5min":0.88600000000000001,"15min":0.88600000000000001},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":1.258},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3720000000000001,"15min":1.3740000000000001},"min":{"1min":1.073,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":1.6699999999999999,"5min":1.9450000000000001,"15min":1.9450000000000001},"last":1.2330000000000001}]}]},{"osd":5,"up_from":37,"seq":158913789989,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6524,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738628,"statfs":{"total":95995035648,"available":95988355072,"internally_reserved":0,"allocated":839680,"data_stored":306008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:02 2021","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.98899999999999999,"15min":0.998},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.516,"5min":1.516,"15min":1.516},"last":0.98999999999999999},{"interface":"front","average":{"1min":0.873,"5min":0.90400000000000003,"15min":0.90900000000000003},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.238,"5min":1.365,"15min":1.365},"last":0.69399999999999995}]},{"osd":1,"last update":"Wed Feb 10 04:27:02 2021","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.92700000000000005,"15min":0.93100000000000005},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.502,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.998,"15min":1.002},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.419,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.83099999999999996}]},{"osd":2,"last update":"Wed Feb 10 04:27:02 2021","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.997,"15min":0.995},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.831,"5min":1.831,"15min":1.831},"last":1.048}]},{"osd":3,"last update":"Wed Feb 10 04:27:02 2021","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.0980000000000001,"15min":1.0920000000000001},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.871},{"interface":"front","average":{"1min":1.087,"5min":1.0720000000000001,"15min":1.0700000000000001},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.7030000000000001,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.155}]},{"osd":4,"last update":"Wed Feb 10 04:27:02 2021","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":0.92000000000000004,"15min":0.90500000000000003},"min":{"1min":0.65300000000000002,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.1699999999999999,"5min":0.98999999999999999,"15min":0.95999999999999996},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":1.232}]},{"osd":6,"last update":"Wed Feb 10 04:27:23 2021","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.149,"15min":1.119},"min":{"1min":0.88100000000000001,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.085,"15min":1.0549999999999999},"min":{"1min":0.92100000000000004,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.3340000000000001,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":1.2669999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:27:38 2021","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.1910000000000001,"15min":1.177},"min":{"1min":0.93400000000000005,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.2440000000000002,"5min":2.2440000000000002,"15min":2.2440000000000002},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.528,"5min":1.3919999999999999,"15min":1.3700000000000001},"min":{"1min":1.0169999999999999,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":2.2799999999999998,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.329}]}]}]}} 2021-02-10T04:27:53.687 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:27:53.882 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:27:54.387 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:54 smithi085 conmon[35961]: cluster 2021-02-10T04:27:53.635220+0000 osd.4 (osd. 2021-02-10T04:27:54.387 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:54 smithi085 conmon[35961]: 4) 28 : cluster [DBG] 2.f scrub ok 2021-02-10T04:27:54.387 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:54 smithi085 conmon[35961]: cluster 2021-02-10T04:27:53.981618+0000 mgr.y (mgr.14140) 304 : 2021-02-10T04:27:54.387 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:54 smithi085 conmon[35961]: cluster [DBG] pgmap v262: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:54.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:54 smithi164 conmon[29288]: cluster 2021-02-10T04:27:53.635220+0000 osd.4 (osd.4) 28 : cluster 2021-02-10T04:27:54.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:54 smithi164 conmon[29288]: [DBG] 2.f scrub ok 2021-02-10T04:27:54.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:54 smithi164 conmon[29288]: cluster 2021-02-10T04:27:53.981618+0000 mgr.y (mgr.14140) 304 : cluster [DBG] pgmap v262: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:54.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:54 smithi085 conmon[29319]: cluster 2021-02-10T04:27:53.635220+0000 osd.4 (osd. 2021-02-10T04:27:54.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:54 smithi085 conmon[29319]: 4) 28 : cluster [DBG] 2.f scrub ok 2021-02-10T04:27:54.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:54 smithi085 conmon[29319]: cluster 2021-02-10T04:27:53.981618+0000 mgr.y (mgr.14140) 304 : cluster [DBG] pgmap v262: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:55.632 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:27:55.634 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:27:55.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:55 smithi164 conmon[29288]: cluster 2021-02-10T04:27:54.402845+0000 osd.5 (osd.5) 23 : cluster [DBG] 2.12 deep-scrub ok 2021-02-10T04:27:55.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:55 smithi164 conmon[29288]: 2021-02-10T04:27:55.650 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:55 smithi085 conmon[29319]: cluster 2021-02-10T04:27:54.402845+0000 osd.5 (osd.5) 23 2021-02-10T04:27:55.650 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:55 smithi085 conmon[29319]: : cluster [DBG] 2.12 deep-scrub ok 2021-02-10T04:27:55.650 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:55 smithi085 conmon[35961]: cluster 2021-02-10T04:27:54.402845+0000 osd.5 (osd.5 2021-02-10T04:27:55.650 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:55 smithi085 conmon[35961]: ) 23 : cluster [DBG] 2.12 deep-scrub ok 2021-02-10T04:27:56.316 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":262,"stamp":"2021-02-10T04:27:53.981098+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":25858,"num_objects":357,"num_object_clones":0,"num_object_copies":1071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":357,"num_whiteouts":0,"num_read":1601,"num_read_kb":1069,"num_write":759,"num_write_kb":177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":787,"ondisk_log_size":787,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":58172,"kb_used_data":7100,"kb_used_omap":0,"kb_used_meta":51008,"kb_avail":749903044,"statfs":{"total":767960285184,"available":767900717056,"internally_reserved":0,"allocated":7270400,"data_stored":2518894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52232192},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002896"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":"37","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.374865+0000","last_change":"2021-02-10T04:27:07.339624+0000","last_active":"2021-02-10T04:27:41.374865+0000","last_peered":"2021-02-10T04:27:41.374865+0000","last_clean":"2021-02-10T04:27:41.374865+0000","last_became_active":"2021-02-10T04:25:53.150917+0000","last_became_peered":"2021-02-10T04:25:53.150917+0000","last_unstale":"2021-02-10T04:27:41.374865+0000","last_undegraded":"2021-02-10T04:27:41.374865+0000","last_fullsized":"2021-02-10T04:27:41.374865+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:07.339551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:07.339551+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"40","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:49.628172+0000","last_change":"2021-02-10T04:27:49.628172+0000","last_active":"2021-02-10T04:27:49.628172+0000","last_peered":"2021-02-10T04:27:49.628172+0000","last_clean":"2021-02-10T04:27:49.628172+0000","last_became_active":"2021-02-10T04:25:51.029110+0000","last_became_peered":"2021-02-10T04:25:51.029110+0000","last_unstale":"2021-02-10T04:27:49.628172+0000","last_undegraded":"2021-02-10T04:27:49.628172+0000","last_fullsized":"2021-02-10T04:27:49.628172+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:49.628090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:49.628090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"58","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:48.635827+0000","last_change":"2021-02-10T04:27:48.635827+0000","last_active":"2021-02-10T04:27:48.635827+0000","last_peered":"2021-02-10T04:27:48.635827+0000","last_clean":"2021-02-10T04:27:48.635827+0000","last_became_active":"2021-02-10T04:25:36.336112+0000","last_became_peered":"2021-02-10T04:25:36.336112+0000","last_unstale":"2021-02-10T04:27:48.635827+0000","last_undegraded":"2021-02-10T04:27:48.635827+0000","last_fullsized":"2021-02-10T04:27:48.635827+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:48.635765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:48.635765+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"63'44","reported_seq":"136","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:49.269619+0000","last_change":"2021-02-10T04:27:49.269619+0000","last_active":"2021-02-10T04:27:49.269619+0000","last_peered":"2021-02-10T04:27:49.269619+0000","last_clean":"2021-02-10T04:27:49.269619+0000","last_became_active":"2021-02-10T04:25:49.008451+0000","last_became_peered":"2021-02-10T04:25:49.008451+0000","last_unstale":"2021-02-10T04:27:49.269619+0000","last_undegraded":"2021-02-10T04:27:49.269619+0000","last_fullsized":"2021-02-10T04:27:49.269619+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'44","last_scrub_stamp":"2021-02-10T04:27:49.269539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:49.269539+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":89,"num_read_kb":66,"num_write":45,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"38","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:51.679613+0000","last_change":"2021-02-10T04:27:51.679613+0000","last_active":"2021-02-10T04:27:51.679613+0000","last_peered":"2021-02-10T04:27:51.679613+0000","last_clean":"2021-02-10T04:27:51.679613+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:27:51.679613+0000","last_undegraded":"2021-02-10T04:27:51.679613+0000","last_fullsized":"2021-02-10T04:27:51.679613+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:51.679515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:51.679515+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.099626+0000","last_change":"2021-02-10T04:26:54.007373+0000","last_active":"2021-02-10T04:27:41.099626+0000","last_peered":"2021-02-10T04:27:41.099626+0000","last_clean":"2021-02-10T04:27:41.099626+0000","last_became_active":"2021-02-10T04:25:51.030405+0000","last_became_peered":"2021-02-10T04:25:51.030405+0000","last_unstale":"2021-02-10T04:27:41.099626+0000","last_undegraded":"2021-02-10T04:27:41.099626+0000","last_fullsized":"2021-02-10T04:27:41.099626+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:54.007310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:26:54.007310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"48","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375707+0000","last_change":"2021-02-10T04:27:13.332288+0000","last_active":"2021-02-10T04:27:41.375707+0000","last_peered":"2021-02-10T04:27:41.375707+0000","last_clean":"2021-02-10T04:27:41.375707+0000","last_became_active":"2021-02-10T04:25:34.327162+0000","last_became_peered":"2021-02-10T04:25:34.327162+0000","last_unstale":"2021-02-10T04:27:41.375707+0000","last_undegraded":"2021-02-10T04:27:41.375707+0000","last_fullsized":"2021-02-10T04:27:41.375707+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:27:13.332224+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"78'32","reported_seq":"96","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375595+0000","last_change":"2021-02-10T04:27:09.170492+0000","last_active":"2021-02-10T04:27:41.375595+0000","last_peered":"2021-02-10T04:27:41.375595+0000","last_clean":"2021-02-10T04:27:41.375595+0000","last_became_active":"2021-02-10T04:25:49.010481+0000","last_became_peered":"2021-02-10T04:25:49.010481+0000","last_unstale":"2021-02-10T04:27:41.375595+0000","last_undegraded":"2021-02-10T04:27:41.375595+0000","last_fullsized":"2021-02-10T04:27:41.375595+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'32","last_scrub_stamp":"2021-02-10T04:27:09.170428+0000","last_deep_scrub":"78'32","last_deep_scrub_stamp":"2021-02-10T04:27:09.170428+0000","last_clean_scrub_stamp":"2021-02-10T04:27:09.170428+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":47,"num_read_kb":28,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375539+0000","last_change":"2021-02-10T04:27:07.962210+0000","last_active":"2021-02-10T04:27:41.375539+0000","last_peered":"2021-02-10T04:27:41.375539+0000","last_clean":"2021-02-10T04:27:41.375539+0000","last_became_active":"2021-02-10T04:25:53.042601+0000","last_became_peered":"2021-02-10T04:25:53.042601+0000","last_unstale":"2021-02-10T04:27:41.375539+0000","last_undegraded":"2021-02-10T04:27:41.375539+0000","last_fullsized":"2021-02-10T04:27:41.375539+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:07.962115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:07.962115+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376031+0000","last_change":"2021-02-10T04:27:29.251059+0000","last_active":"2021-02-10T04:27:41.376031+0000","last_peered":"2021-02-10T04:27:41.376031+0000","last_clean":"2021-02-10T04:27:41.376031+0000","last_became_active":"2021-02-10T04:25:51.029286+0000","last_became_peered":"2021-02-10T04:25:51.029286+0000","last_unstale":"2021-02-10T04:27:41.376031+0000","last_undegraded":"2021-02-10T04:27:41.376031+0000","last_fullsized":"2021-02-10T04:27:41.376031+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:29.250978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:29.250978+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"52","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:50.297737+0000","last_change":"2021-02-10T04:27:50.297737+0000","last_active":"2021-02-10T04:27:50.297737+0000","last_peered":"2021-02-10T04:27:50.297737+0000","last_clean":"2021-02-10T04:27:50.297737+0000","last_became_active":"2021-02-10T04:25:36.344477+0000","last_became_peered":"2021-02-10T04:25:36.344477+0000","last_unstale":"2021-02-10T04:27:50.297737+0000","last_undegraded":"2021-02-10T04:27:50.297737+0000","last_fullsized":"2021-02-10T04:27:50.297737+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:27:50.297650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"63'12","reported_seq":"69","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.378150+0000","last_change":"2021-02-10T04:27:37.441260+0000","last_active":"2021-02-10T04:27:41.378150+0000","last_peered":"2021-02-10T04:27:41.378150+0000","last_clean":"2021-02-10T04:27:41.378150+0000","last_became_active":"2021-02-10T04:25:49.013196+0000","last_became_peered":"2021-02-10T04:25:49.013196+0000","last_unstale":"2021-02-10T04:27:41.378150+0000","last_undegraded":"2021-02-10T04:27:41.378150+0000","last_fullsized":"2021-02-10T04:27:41.378150+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2021-02-10T04:27:37.441168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:37.441168+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.098695+0000","last_change":"2021-02-10T04:27:33.628099+0000","last_active":"2021-02-10T04:27:41.098695+0000","last_peered":"2021-02-10T04:27:41.098695+0000","last_clean":"2021-02-10T04:27:41.098695+0000","last_became_active":"2021-02-10T04:25:53.039844+0000","last_became_peered":"2021-02-10T04:25:53.039844+0000","last_unstale":"2021-02-10T04:27:41.098695+0000","last_undegraded":"2021-02-10T04:27:41.098695+0000","last_fullsized":"2021-02-10T04:27:41.098695+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:33.628038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:33.628038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375359+0000","last_change":"2021-02-10T04:27:04.977415+0000","last_active":"2021-02-10T04:27:41.375359+0000","last_peered":"2021-02-10T04:27:41.375359+0000","last_clean":"2021-02-10T04:27:41.375359+0000","last_became_active":"2021-02-10T04:25:51.025488+0000","last_became_peered":"2021-02-10T04:25:51.025488+0000","last_unstale":"2021-02-10T04:27:41.375359+0000","last_undegraded":"2021-02-10T04:27:41.375359+0000","last_fullsized":"2021-02-10T04:27:41.375359+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:04.977353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:04.977353+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"47","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375687+0000","last_change":"2021-02-10T04:26:45.223669+0000","last_active":"2021-02-10T04:27:41.375687+0000","last_peered":"2021-02-10T04:27:41.375687+0000","last_clean":"2021-02-10T04:27:41.375687+0000","last_became_active":"2021-02-10T04:25:36.337260+0000","last_became_peered":"2021-02-10T04:25:36.337260+0000","last_unstale":"2021-02-10T04:27:41.375687+0000","last_undegraded":"2021-02-10T04:27:41.375687+0000","last_fullsized":"2021-02-10T04:27:41.375687+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:45.223603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:45.223603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"63'25","reported_seq":"96","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376564+0000","last_change":"2021-02-10T04:27:05.242824+0000","last_active":"2021-02-10T04:27:41.376564+0000","last_peered":"2021-02-10T04:27:41.376564+0000","last_clean":"2021-02-10T04:27:41.376564+0000","last_became_active":"2021-02-10T04:25:49.000911+0000","last_became_peered":"2021-02-10T04:25:49.000911+0000","last_unstale":"2021-02-10T04:27:41.376564+0000","last_undegraded":"2021-02-10T04:27:41.376564+0000","last_fullsized":"2021-02-10T04:27:41.376564+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2021-02-10T04:27:05.242762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:05.242762+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":53,"num_read_kb":37,"num_write":26,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"37","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375909+0000","last_change":"2021-02-10T04:26:55.332873+0000","last_active":"2021-02-10T04:27:41.375909+0000","last_peered":"2021-02-10T04:27:41.375909+0000","last_clean":"2021-02-10T04:27:41.375909+0000","last_became_active":"2021-02-10T04:25:53.147831+0000","last_became_peered":"2021-02-10T04:25:53.147831+0000","last_unstale":"2021-02-10T04:27:41.375909+0000","last_undegraded":"2021-02-10T04:27:41.375909+0000","last_fullsized":"2021-02-10T04:27:41.375909+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:55.332815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:26:55.332815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"40","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:48.598953+0000","last_change":"2021-02-10T04:27:48.598953+0000","last_active":"2021-02-10T04:27:48.598953+0000","last_peered":"2021-02-10T04:27:48.598953+0000","last_clean":"2021-02-10T04:27:48.598953+0000","last_became_active":"2021-02-10T04:25:51.018872+0000","last_became_peered":"2021-02-10T04:25:51.018872+0000","last_unstale":"2021-02-10T04:27:48.598953+0000","last_undegraded":"2021-02-10T04:27:48.598953+0000","last_fullsized":"2021-02-10T04:27:48.598953+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:48.598870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:48.598870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"57","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.374821+0000","last_change":"2021-02-10T04:26:46.425375+0000","last_active":"2021-02-10T04:27:41.374821+0000","last_peered":"2021-02-10T04:27:41.374821+0000","last_clean":"2021-02-10T04:27:41.374821+0000","last_became_active":"2021-02-10T04:25:36.336893+0000","last_became_peered":"2021-02-10T04:25:36.336893+0000","last_unstale":"2021-02-10T04:27:41.374821+0000","last_undegraded":"2021-02-10T04:27:41.374821+0000","last_fullsized":"2021-02-10T04:27:41.374821+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:46.425288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:26:46.425288+0000","last_clean_scrub_stamp":"2021-02-10T04:26:46.425288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"62'22","reported_seq":"82","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375644+0000","last_change":"2021-02-10T04:25:49.009749+0000","last_active":"2021-02-10T04:27:41.375644+0000","last_peered":"2021-02-10T04:27:41.375644+0000","last_clean":"2021-02-10T04:27:41.375644+0000","last_became_active":"2021-02-10T04:25:49.008997+0000","last_became_peered":"2021-02-10T04:25:49.008997+0000","last_unstale":"2021-02-10T04:27:41.375644+0000","last_undegraded":"2021-02-10T04:27:41.375644+0000","last_fullsized":"2021-02-10T04:27:41.375644+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:25:47.970100+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":49,"num_read_kb":31,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.377998+0000","last_change":"2021-02-10T04:27:29.364660+0000","last_active":"2021-02-10T04:27:41.377998+0000","last_peered":"2021-02-10T04:27:41.377998+0000","last_clean":"2021-02-10T04:27:41.377998+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:27:41.377998+0000","last_undegraded":"2021-02-10T04:27:41.377998+0000","last_fullsized":"2021-02-10T04:27:41.377998+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:29.364585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:29.364585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.100986+0000","last_change":"2021-02-10T04:27:27.017142+0000","last_active":"2021-02-10T04:27:41.100986+0000","last_peered":"2021-02-10T04:27:41.100986+0000","last_clean":"2021-02-10T04:27:41.100986+0000","last_became_active":"2021-02-10T04:25:51.029399+0000","last_became_peered":"2021-02-10T04:25:51.029399+0000","last_unstale":"2021-02-10T04:27:41.100986+0000","last_undegraded":"2021-02-10T04:27:41.100986+0000","last_fullsized":"2021-02-10T04:27:41.100986+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:27.017040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:27.017040+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"52","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.097636+0000","last_change":"2021-02-10T04:27:02.190158+0000","last_active":"2021-02-10T04:27:41.097636+0000","last_peered":"2021-02-10T04:27:41.097636+0000","last_clean":"2021-02-10T04:27:41.097636+0000","last_became_active":"2021-02-10T04:25:36.336442+0000","last_became_peered":"2021-02-10T04:25:36.336442+0000","last_unstale":"2021-02-10T04:27:41.097636+0000","last_undegraded":"2021-02-10T04:27:41.097636+0000","last_fullsized":"2021-02-10T04:27:41.097636+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:27:02.190056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:02.190056+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"62'15","reported_seq":"66","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375583+0000","last_change":"2021-02-10T04:26:56.623190+0000","last_active":"2021-02-10T04:27:41.375583+0000","last_peered":"2021-02-10T04:27:41.375583+0000","last_clean":"2021-02-10T04:27:41.375583+0000","last_became_active":"2021-02-10T04:25:49.010553+0000","last_became_peered":"2021-02-10T04:25:49.010553+0000","last_unstale":"2021-02-10T04:27:41.375583+0000","last_undegraded":"2021-02-10T04:27:41.375583+0000","last_fullsized":"2021-02-10T04:27:41.375583+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:26:56.623109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:26:56.623109+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.100760+0000","last_change":"2021-02-10T04:27:26.029923+0000","last_active":"2021-02-10T04:27:41.100760+0000","last_peered":"2021-02-10T04:27:41.100760+0000","last_clean":"2021-02-10T04:27:41.100760+0000","last_became_active":"2021-02-10T04:25:53.041098+0000","last_became_peered":"2021-02-10T04:25:53.041098+0000","last_unstale":"2021-02-10T04:27:41.100760+0000","last_undegraded":"2021-02-10T04:27:41.100760+0000","last_fullsized":"2021-02-10T04:27:41.100760+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:26.029856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:26.029856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.100087+0000","last_change":"2021-02-10T04:27:27.564268+0000","last_active":"2021-02-10T04:27:41.100087+0000","last_peered":"2021-02-10T04:27:41.100087+0000","last_clean":"2021-02-10T04:27:41.100087+0000","last_became_active":"2021-02-10T04:25:51.022853+0000","last_became_peered":"2021-02-10T04:25:51.022853+0000","last_unstale":"2021-02-10T04:27:41.100087+0000","last_undegraded":"2021-02-10T04:27:41.100087+0000","last_fullsized":"2021-02-10T04:27:41.100087+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:27.564175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:27.564175+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"48","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.374806+0000","last_change":"2021-02-10T04:27:16.340830+0000","last_active":"2021-02-10T04:27:41.374806+0000","last_peered":"2021-02-10T04:27:41.374806+0000","last_clean":"2021-02-10T04:27:41.374806+0000","last_became_active":"2021-02-10T04:25:34.324981+0000","last_became_peered":"2021-02-10T04:25:34.324981+0000","last_unstale":"2021-02-10T04:27:41.374806+0000","last_undegraded":"2021-02-10T04:27:41.374806+0000","last_fullsized":"2021-02-10T04:27:41.374806+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:16.340724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:16.340724+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"76'35","reported_seq":"100","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.098180+0000","last_change":"2021-02-10T04:26:58.198706+0000","last_active":"2021-02-10T04:27:41.098180+0000","last_peered":"2021-02-10T04:27:41.098180+0000","last_clean":"2021-02-10T04:27:41.098180+0000","last_became_active":"2021-02-10T04:25:49.015760+0000","last_became_peered":"2021-02-10T04:25:49.015760+0000","last_unstale":"2021-02-10T04:27:41.098180+0000","last_undegraded":"2021-02-10T04:27:41.098180+0000","last_fullsized":"2021-02-10T04:27:41.098180+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'35","last_scrub_stamp":"2021-02-10T04:26:58.198641+0000","last_deep_scrub":"76'35","last_deep_scrub_stamp":"2021-02-10T04:26:58.198641+0000","last_clean_scrub_stamp":"2021-02-10T04:26:58.198641+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":50,"num_read_kb":30,"num_write":24,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376084+0000","last_change":"2021-02-10T04:27:39.139028+0000","last_active":"2021-02-10T04:27:41.376084+0000","last_peered":"2021-02-10T04:27:41.376084+0000","last_clean":"2021-02-10T04:27:41.376084+0000","last_became_active":"2021-02-10T04:25:53.042279+0000","last_became_peered":"2021-02-10T04:25:53.042279+0000","last_unstale":"2021-02-10T04:27:41.376084+0000","last_undegraded":"2021-02-10T04:27:41.376084+0000","last_fullsized":"2021-02-10T04:27:41.376084+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:39.138947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:39.138947+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"58'2","reported_seq":"66","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.727764+0000","last_change":"2021-02-10T04:27:15.041424+0000","last_active":"2021-02-10T04:27:47.727764+0000","last_peered":"2021-02-10T04:27:47.727764+0000","last_clean":"2021-02-10T04:27:47.727764+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:27:47.727764+0000","last_undegraded":"2021-02-10T04:27:47.727764+0000","last_fullsized":"2021-02-10T04:27:47.727764+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:15.041329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:15.041329+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"58","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375030+0000","last_change":"2021-02-10T04:27:04.011149+0000","last_active":"2021-02-10T04:27:41.375030+0000","last_peered":"2021-02-10T04:27:41.375030+0000","last_clean":"2021-02-10T04:27:41.375030+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:27:41.375030+0000","last_undegraded":"2021-02-10T04:27:41.375030+0000","last_fullsized":"2021-02-10T04:27:41.375030+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:04.011053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:04.011053+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"63'22","reported_seq":"84","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.100101+0000","last_change":"2021-02-10T04:26:51.409218+0000","last_active":"2021-02-10T04:27:41.100101+0000","last_peered":"2021-02-10T04:27:41.100101+0000","last_clean":"2021-02-10T04:27:41.100101+0000","last_became_active":"2021-02-10T04:25:49.009177+0000","last_became_peered":"2021-02-10T04:25:49.009177+0000","last_unstale":"2021-02-10T04:27:41.100101+0000","last_undegraded":"2021-02-10T04:27:41.100101+0000","last_fullsized":"2021-02-10T04:27:41.100101+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2021-02-10T04:26:51.409147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:26:51.409147+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":22,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"32","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375183+0000","last_change":"2021-02-10T04:25:53.151054+0000","last_active":"2021-02-10T04:27:41.375183+0000","last_peered":"2021-02-10T04:27:41.375183+0000","last_clean":"2021-02-10T04:27:41.375183+0000","last_became_active":"2021-02-10T04:25:53.150845+0000","last_became_peered":"2021-02-10T04:25:53.150845+0000","last_unstale":"2021-02-10T04:27:41.375183+0000","last_undegraded":"2021-02-10T04:27:41.375183+0000","last_fullsized":"2021-02-10T04:27:41.375183+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:25:52.015519+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375521+0000","last_change":"2021-02-10T04:27:07.181928+0000","last_active":"2021-02-10T04:27:41.375521+0000","last_peered":"2021-02-10T04:27:41.375521+0000","last_clean":"2021-02-10T04:27:41.375521+0000","last_became_active":"2021-02-10T04:25:51.018491+0000","last_became_peered":"2021-02-10T04:25:51.018491+0000","last_unstale":"2021-02-10T04:27:41.375521+0000","last_undegraded":"2021-02-10T04:27:41.375521+0000","last_fullsized":"2021-02-10T04:27:41.375521+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:07.181854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:07.181854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"52","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.098169+0000","last_change":"2021-02-10T04:26:52.256790+0000","last_active":"2021-02-10T04:27:41.098169+0000","last_peered":"2021-02-10T04:27:41.098169+0000","last_clean":"2021-02-10T04:27:41.098169+0000","last_became_active":"2021-02-10T04:25:35.328828+0000","last_became_peered":"2021-02-10T04:25:35.328828+0000","last_unstale":"2021-02-10T04:27:41.098169+0000","last_undegraded":"2021-02-10T04:27:41.098169+0000","last_fullsized":"2021-02-10T04:27:41.098169+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:26:52.256723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:52.256723+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"63'21","reported_seq":"82","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375014+0000","last_change":"2021-02-10T04:27:12.291860+0000","last_active":"2021-02-10T04:27:41.375014+0000","last_peered":"2021-02-10T04:27:41.375014+0000","last_clean":"2021-02-10T04:27:41.375014+0000","last_became_active":"2021-02-10T04:25:49.007049+0000","last_became_peered":"2021-02-10T04:25:49.007049+0000","last_unstale":"2021-02-10T04:27:41.375014+0000","last_undegraded":"2021-02-10T04:27:41.375014+0000","last_fullsized":"2021-02-10T04:27:41.375014+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2021-02-10T04:27:12.291796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:12.291796+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":27,"num_write":21,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375818+0000","last_change":"2021-02-10T04:27:19.134286+0000","last_active":"2021-02-10T04:27:41.375818+0000","last_peered":"2021-02-10T04:27:41.375818+0000","last_clean":"2021-02-10T04:27:41.375818+0000","last_became_active":"2021-02-10T04:25:53.034771+0000","last_became_peered":"2021-02-10T04:25:53.034771+0000","last_unstale":"2021-02-10T04:27:41.375818+0000","last_undegraded":"2021-02-10T04:27:41.375818+0000","last_fullsized":"2021-02-10T04:27:41.375818+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:27:19.134212+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.097597+0000","last_change":"2021-02-10T04:27:19.077862+0000","last_active":"2021-02-10T04:27:41.097597+0000","last_peered":"2021-02-10T04:27:41.097597+0000","last_clean":"2021-02-10T04:27:41.097597+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:27:41.097597+0000","last_undegraded":"2021-02-10T04:27:41.097597+0000","last_fullsized":"2021-02-10T04:27:41.097597+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:27:19.077777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"49","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.377580+0000","last_change":"2021-02-10T04:27:03.000902+0000","last_active":"2021-02-10T04:27:41.377580+0000","last_peered":"2021-02-10T04:27:41.377580+0000","last_clean":"2021-02-10T04:27:41.377580+0000","last_became_active":"2021-02-10T04:25:34.331787+0000","last_became_peered":"2021-02-10T04:25:34.331787+0000","last_unstale":"2021-02-10T04:27:41.377580+0000","last_undegraded":"2021-02-10T04:27:41.377580+0000","last_fullsized":"2021-02-10T04:27:41.377580+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:03.000821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:03.000821+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"62'12","reported_seq":"94","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:48.284315+0000","last_change":"2021-02-10T04:27:48.284315+0000","last_active":"2021-02-10T04:27:48.284315+0000","last_peered":"2021-02-10T04:27:48.284315+0000","last_clean":"2021-02-10T04:27:48.284315+0000","last_became_active":"2021-02-10T04:26:39.005616+0000","last_became_peered":"2021-02-10T04:26:39.005616+0000","last_unstale":"2021-02-10T04:27:48.284315+0000","last_undegraded":"2021-02-10T04:27:48.284315+0000","last_fullsized":"2021-02-10T04:27:48.284315+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2021-02-10T04:27:48.284248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:48.284248+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"37","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.374682+0000","last_change":"2021-02-10T04:26:57.309978+0000","last_active":"2021-02-10T04:27:41.374682+0000","last_peered":"2021-02-10T04:27:41.374682+0000","last_clean":"2021-02-10T04:27:41.374682+0000","last_became_active":"2021-02-10T04:25:53.043277+0000","last_became_peered":"2021-02-10T04:25:53.043277+0000","last_unstale":"2021-02-10T04:27:41.374682+0000","last_undegraded":"2021-02-10T04:27:41.374682+0000","last_fullsized":"2021-02-10T04:27:41.374682+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:57.309915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:26:57.309915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.100552+0000","last_change":"2021-02-10T04:27:33.030004+0000","last_active":"2021-02-10T04:27:41.100552+0000","last_peered":"2021-02-10T04:27:41.100552+0000","last_clean":"2021-02-10T04:27:41.100552+0000","last_became_active":"2021-02-10T04:25:51.030952+0000","last_became_peered":"2021-02-10T04:25:51.030952+0000","last_unstale":"2021-02-10T04:27:41.100552+0000","last_undegraded":"2021-02-10T04:27:41.100552+0000","last_fullsized":"2021-02-10T04:27:41.100552+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:27:33.029912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"58","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.100813+0000","last_change":"2021-02-10T04:26:52.011165+0000","last_active":"2021-02-10T04:27:41.100813+0000","last_peered":"2021-02-10T04:27:41.100813+0000","last_clean":"2021-02-10T04:27:41.100813+0000","last_became_active":"2021-02-10T04:25:34.330242+0000","last_became_peered":"2021-02-10T04:25:34.330242+0000","last_unstale":"2021-02-10T04:27:41.100813+0000","last_undegraded":"2021-02-10T04:27:41.100813+0000","last_fullsized":"2021-02-10T04:27:41.100813+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:26:52.011096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:52.011096+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"62'15","reported_seq":"69","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:42.487297+0000","last_change":"2021-02-10T04:27:42.487239+0000","last_active":"2021-02-10T04:27:42.487297+0000","last_peered":"2021-02-10T04:27:42.487297+0000","last_clean":"2021-02-10T04:27:42.487297+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:27:42.487297+0000","last_undegraded":"2021-02-10T04:27:42.487297+0000","last_fullsized":"2021-02-10T04:27:42.487297+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:27:42.487154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:42.487154+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.098426+0000","last_change":"2021-02-10T04:27:21.548446+0000","last_active":"2021-02-10T04:27:41.098426+0000","last_peered":"2021-02-10T04:27:41.098426+0000","last_clean":"2021-02-10T04:27:41.098426+0000","last_became_active":"2021-02-10T04:25:53.039322+0000","last_became_peered":"2021-02-10T04:25:53.039322+0000","last_unstale":"2021-02-10T04:27:41.098426+0000","last_undegraded":"2021-02-10T04:27:41.098426+0000","last_fullsized":"2021-02-10T04:27:41.098426+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:21.548381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:21.548381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"39","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375032+0000","last_change":"2021-02-10T04:27:05.297715+0000","last_active":"2021-02-10T04:27:41.375032+0000","last_peered":"2021-02-10T04:27:41.375032+0000","last_clean":"2021-02-10T04:27:41.375032+0000","last_became_active":"2021-02-10T04:25:51.029155+0000","last_became_peered":"2021-02-10T04:25:51.029155+0000","last_unstale":"2021-02-10T04:27:41.375032+0000","last_undegraded":"2021-02-10T04:27:41.375032+0000","last_fullsized":"2021-02-10T04:27:41.375032+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:27:05.297647+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"49","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.097864+0000","last_change":"2021-02-10T04:26:54.256587+0000","last_active":"2021-02-10T04:27:41.097864+0000","last_peered":"2021-02-10T04:27:41.097864+0000","last_clean":"2021-02-10T04:27:41.097864+0000","last_became_active":"2021-02-10T04:25:34.330070+0000","last_became_peered":"2021-02-10T04:25:34.330070+0000","last_unstale":"2021-02-10T04:27:41.097864+0000","last_undegraded":"2021-02-10T04:27:41.097864+0000","last_fullsized":"2021-02-10T04:27:41.097864+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:54.256498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:54.256498+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"62'15","reported_seq":"70","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.100505+0000","last_change":"2021-02-10T04:27:40.218910+0000","last_active":"2021-02-10T04:27:41.100505+0000","last_peered":"2021-02-10T04:27:41.100505+0000","last_clean":"2021-02-10T04:27:41.100505+0000","last_became_active":"2021-02-10T04:25:49.015543+0000","last_became_peered":"2021-02-10T04:25:49.015543+0000","last_unstale":"2021-02-10T04:27:41.100505+0000","last_undegraded":"2021-02-10T04:27:41.100505+0000","last_fullsized":"2021-02-10T04:27:41.100505+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:27:40.218828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:40.218828+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.378352+0000","last_change":"2021-02-10T04:27:28.373194+0000","last_active":"2021-02-10T04:27:41.378352+0000","last_peered":"2021-02-10T04:27:41.378352+0000","last_clean":"2021-02-10T04:27:41.378352+0000","last_became_active":"2021-02-10T04:25:53.031782+0000","last_became_peered":"2021-02-10T04:25:53.031782+0000","last_unstale":"2021-02-10T04:27:41.378352+0000","last_undegraded":"2021-02-10T04:27:41.378352+0000","last_fullsized":"2021-02-10T04:27:41.378352+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:28.373117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:28.373117+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"58'2","reported_seq":"65","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.117389+0000","last_change":"2021-02-10T04:27:26.264628+0000","last_active":"2021-02-10T04:27:47.117389+0000","last_peered":"2021-02-10T04:27:47.117389+0000","last_clean":"2021-02-10T04:27:47.117389+0000","last_became_active":"2021-02-10T04:25:51.018300+0000","last_became_peered":"2021-02-10T04:25:51.018300+0000","last_unstale":"2021-02-10T04:27:47.117389+0000","last_undegraded":"2021-02-10T04:27:47.117389+0000","last_fullsized":"2021-02-10T04:27:47.117389+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:26.264560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:26.264560+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"54","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.099451+0000","last_change":"2021-02-10T04:27:39.616474+0000","last_active":"2021-02-10T04:27:41.099451+0000","last_peered":"2021-02-10T04:27:41.099451+0000","last_clean":"2021-02-10T04:27:41.099451+0000","last_became_active":"2021-02-10T04:25:34.321096+0000","last_became_peered":"2021-02-10T04:25:34.321096+0000","last_unstale":"2021-02-10T04:27:41.099451+0000","last_undegraded":"2021-02-10T04:27:41.099451+0000","last_fullsized":"2021-02-10T04:27:41.099451+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:27:39.616381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"63'19","reported_seq":"81","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376932+0000","last_change":"2021-02-10T04:27:11.016733+0000","last_active":"2021-02-10T04:27:41.376932+0000","last_peered":"2021-02-10T04:27:41.376932+0000","last_clean":"2021-02-10T04:27:41.376932+0000","last_became_active":"2021-02-10T04:25:49.010890+0000","last_became_peered":"2021-02-10T04:25:49.010890+0000","last_unstale":"2021-02-10T04:27:41.376932+0000","last_undegraded":"2021-02-10T04:27:41.376932+0000","last_fullsized":"2021-02-10T04:27:41.376932+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2021-02-10T04:27:11.016650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:11.016650+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":26,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.099783+0000","last_change":"2021-02-10T04:27:08.486638+0000","last_active":"2021-02-10T04:27:41.099783+0000","last_peered":"2021-02-10T04:27:41.099783+0000","last_clean":"2021-02-10T04:27:41.099783+0000","last_became_active":"2021-02-10T04:25:53.045177+0000","last_became_peered":"2021-02-10T04:25:53.045177+0000","last_unstale":"2021-02-10T04:27:41.099783+0000","last_undegraded":"2021-02-10T04:27:41.099783+0000","last_fullsized":"2021-02-10T04:27:41.099783+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:08.486553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:08.486553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"58'2","reported_seq":"67","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.729992+0000","last_change":"2021-02-10T04:27:20.157871+0000","last_active":"2021-02-10T04:27:47.729992+0000","last_peered":"2021-02-10T04:27:47.729992+0000","last_clean":"2021-02-10T04:27:47.729992+0000","last_became_active":"2021-02-10T04:25:51.032328+0000","last_became_peered":"2021-02-10T04:25:51.032328+0000","last_unstale":"2021-02-10T04:27:47.729992+0000","last_undegraded":"2021-02-10T04:27:47.729992+0000","last_fullsized":"2021-02-10T04:27:47.729992+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:20.157772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:20.157772+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"52'4","reported_seq":"61","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376110+0000","last_change":"2021-02-10T04:26:49.013781+0000","last_active":"2021-02-10T04:27:41.376110+0000","last_peered":"2021-02-10T04:27:41.376110+0000","last_clean":"2021-02-10T04:27:41.376110+0000","last_became_active":"2021-02-10T04:25:36.334366+0000","last_became_peered":"2021-02-10T04:25:36.334366+0000","last_unstale":"2021-02-10T04:27:41.376110+0000","last_undegraded":"2021-02-10T04:27:41.376110+0000","last_fullsized":"2021-02-10T04:27:41.376110+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'4","last_scrub_stamp":"2021-02-10T04:26:49.013681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:49.013681+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"63'26","reported_seq":"127","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375225+0000","last_change":"2021-02-10T04:27:28.213521+0000","last_active":"2021-02-10T04:27:41.375225+0000","last_peered":"2021-02-10T04:27:41.375225+0000","last_clean":"2021-02-10T04:27:41.375225+0000","last_became_active":"2021-02-10T04:26:39.003971+0000","last_became_peered":"2021-02-10T04:26:39.003971+0000","last_unstale":"2021-02-10T04:27:41.375225+0000","last_undegraded":"2021-02-10T04:27:41.375225+0000","last_fullsized":"2021-02-10T04:27:41.375225+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:27:28.213454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:28.213454+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376351+0000","last_change":"2021-02-10T04:27:33.192375+0000","last_active":"2021-02-10T04:27:41.376351+0000","last_peered":"2021-02-10T04:27:41.376351+0000","last_clean":"2021-02-10T04:27:41.376351+0000","last_became_active":"2021-02-10T04:25:53.149820+0000","last_became_peered":"2021-02-10T04:25:53.149820+0000","last_unstale":"2021-02-10T04:27:41.376351+0000","last_undegraded":"2021-02-10T04:27:41.376351+0000","last_fullsized":"2021-02-10T04:27:41.376351+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:33.192287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:33.192287+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.377497+0000","last_change":"2021-02-10T04:27:23.236953+0000","last_active":"2021-02-10T04:27:41.377497+0000","last_peered":"2021-02-10T04:27:41.377497+0000","last_clean":"2021-02-10T04:27:41.377497+0000","last_became_active":"2021-02-10T04:25:51.031864+0000","last_became_peered":"2021-02-10T04:25:51.031864+0000","last_unstale":"2021-02-10T04:27:41.377497+0000","last_undegraded":"2021-02-10T04:27:41.377497+0000","last_fullsized":"2021-02-10T04:27:41.377497+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:23.236870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:23.236870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"51","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.098052+0000","last_change":"2021-02-10T04:27:14.083910+0000","last_active":"2021-02-10T04:27:41.098052+0000","last_peered":"2021-02-10T04:27:41.098052+0000","last_clean":"2021-02-10T04:27:41.098052+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:27:41.098052+0000","last_undegraded":"2021-02-10T04:27:41.098052+0000","last_fullsized":"2021-02-10T04:27:41.098052+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:27:14.083841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:14.083841+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"63'13","reported_seq":"88","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375913+0000","last_change":"2021-02-10T04:26:50.393926+0000","last_active":"2021-02-10T04:27:41.375913+0000","last_peered":"2021-02-10T04:27:41.375913+0000","last_clean":"2021-02-10T04:27:41.375913+0000","last_became_active":"2021-02-10T04:26:39.003832+0000","last_became_peered":"2021-02-10T04:26:39.003832+0000","last_unstale":"2021-02-10T04:27:41.375913+0000","last_undegraded":"2021-02-10T04:27:41.375913+0000","last_fullsized":"2021-02-10T04:27:41.375913+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2021-02-10T04:26:50.393865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:26:50.393865+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"37","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375351+0000","last_change":"2021-02-10T04:27:13.573977+0000","last_active":"2021-02-10T04:27:41.375351+0000","last_peered":"2021-02-10T04:27:41.375351+0000","last_clean":"2021-02-10T04:27:41.375351+0000","last_became_active":"2021-02-10T04:25:53.035867+0000","last_became_peered":"2021-02-10T04:25:53.035867+0000","last_unstale":"2021-02-10T04:27:41.375351+0000","last_undegraded":"2021-02-10T04:27:41.375351+0000","last_fullsized":"2021-02-10T04:27:41.375351+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:13.573912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:13.573912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"39","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375949+0000","last_change":"2021-02-10T04:26:56.335818+0000","last_active":"2021-02-10T04:27:41.375949+0000","last_peered":"2021-02-10T04:27:41.375949+0000","last_clean":"2021-02-10T04:27:41.375949+0000","last_became_active":"2021-02-10T04:25:51.030651+0000","last_became_peered":"2021-02-10T04:25:51.030651+0000","last_unstale":"2021-02-10T04:27:41.375949+0000","last_undegraded":"2021-02-10T04:27:41.375949+0000","last_fullsized":"2021-02-10T04:27:41.375949+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:56.335735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:26:56.335735+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"51","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375372+0000","last_change":"2021-02-10T04:26:57.639364+0000","last_active":"2021-02-10T04:27:41.375372+0000","last_peered":"2021-02-10T04:27:41.375372+0000","last_clean":"2021-02-10T04:27:41.375372+0000","last_became_active":"2021-02-10T04:25:34.329694+0000","last_became_peered":"2021-02-10T04:25:34.329694+0000","last_unstale":"2021-02-10T04:27:41.375372+0000","last_undegraded":"2021-02-10T04:27:41.375372+0000","last_fullsized":"2021-02-10T04:27:41.375372+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:26:57.639281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:57.639281+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"63'28","reported_seq":"102","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.099932+0000","last_change":"2021-02-10T04:27:24.523914+0000","last_active":"2021-02-10T04:27:41.099932+0000","last_peered":"2021-02-10T04:27:41.099932+0000","last_clean":"2021-02-10T04:27:41.099932+0000","last_became_active":"2021-02-10T04:25:49.008942+0000","last_became_peered":"2021-02-10T04:25:49.008942+0000","last_unstale":"2021-02-10T04:27:41.099932+0000","last_undegraded":"2021-02-10T04:27:41.099932+0000","last_fullsized":"2021-02-10T04:27:41.099932+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'28","last_scrub_stamp":"2021-02-10T04:27:24.523837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:24.523837+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":62,"num_read_kb":40,"num_write":29,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"37","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375110+0000","last_change":"2021-02-10T04:27:27.259500+0000","last_active":"2021-02-10T04:27:41.375110+0000","last_peered":"2021-02-10T04:27:41.375110+0000","last_clean":"2021-02-10T04:27:41.375110+0000","last_became_active":"2021-02-10T04:25:53.047736+0000","last_became_peered":"2021-02-10T04:25:53.047736+0000","last_unstale":"2021-02-10T04:27:41.375110+0000","last_undegraded":"2021-02-10T04:27:41.375110+0000","last_fullsized":"2021-02-10T04:27:41.375110+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:27.259393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:27.259393+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"62","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.378208+0000","last_change":"2021-02-10T04:26:34.066242+0000","last_active":"2021-02-10T04:27:41.378208+0000","last_peered":"2021-02-10T04:27:41.378208+0000","last_clean":"2021-02-10T04:27:41.378208+0000","last_became_active":"2021-02-10T04:25:36.337394+0000","last_became_peered":"2021-02-10T04:25:36.337394+0000","last_unstale":"2021-02-10T04:27:41.378208+0000","last_undegraded":"2021-02-10T04:27:41.378208+0000","last_fullsized":"2021-02-10T04:27:41.378208+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:26:34.066149+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"63'27","reported_seq":"101","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.378431+0000","last_change":"2021-02-10T04:27:17.352786+0000","last_active":"2021-02-10T04:27:41.378431+0000","last_peered":"2021-02-10T04:27:41.378431+0000","last_clean":"2021-02-10T04:27:41.378431+0000","last_became_active":"2021-02-10T04:25:49.013259+0000","last_became_peered":"2021-02-10T04:25:49.013259+0000","last_unstale":"2021-02-10T04:27:41.378431+0000","last_undegraded":"2021-02-10T04:27:41.378431+0000","last_fullsized":"2021-02-10T04:27:41.378431+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'27","last_scrub_stamp":"2021-02-10T04:27:17.352701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:17.352701+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":41,"num_write":29,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"39","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375286+0000","last_change":"2021-02-10T04:27:14.530439+0000","last_active":"2021-02-10T04:27:41.375286+0000","last_peered":"2021-02-10T04:27:41.375286+0000","last_clean":"2021-02-10T04:27:41.375286+0000","last_became_active":"2021-02-10T04:25:51.023000+0000","last_became_peered":"2021-02-10T04:25:51.023000+0000","last_unstale":"2021-02-10T04:27:41.375286+0000","last_undegraded":"2021-02-10T04:27:41.375286+0000","last_fullsized":"2021-02-10T04:27:41.375286+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:14.530378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:14.530378+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.098428+0000","last_change":"2021-02-10T04:27:18.521962+0000","last_active":"2021-02-10T04:27:41.098428+0000","last_peered":"2021-02-10T04:27:41.098428+0000","last_clean":"2021-02-10T04:27:41.098428+0000","last_became_active":"2021-02-10T04:25:53.045215+0000","last_became_peered":"2021-02-10T04:25:53.045215+0000","last_unstale":"2021-02-10T04:27:41.098428+0000","last_undegraded":"2021-02-10T04:27:41.098428+0000","last_fullsized":"2021-02-10T04:27:41.098428+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:18.521897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:18.521897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"67","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.100802+0000","last_change":"2021-02-10T04:27:04.024909+0000","last_active":"2021-02-10T04:27:41.100802+0000","last_peered":"2021-02-10T04:27:41.100802+0000","last_clean":"2021-02-10T04:27:41.100802+0000","last_became_active":"2021-02-10T04:26:39.004318+0000","last_became_peered":"2021-02-10T04:26:39.004318+0000","last_unstale":"2021-02-10T04:27:41.100802+0000","last_undegraded":"2021-02-10T04:27:41.100802+0000","last_fullsized":"2021-02-10T04:27:41.100802+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:04.024848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:04.024848+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"79'34","reported_seq":"96","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.100562+0000","last_change":"2021-02-10T04:27:18.990156+0000","last_active":"2021-02-10T04:27:41.100562+0000","last_peered":"2021-02-10T04:27:41.100562+0000","last_clean":"2021-02-10T04:27:41.100562+0000","last_became_active":"2021-02-10T04:25:49.006147+0000","last_became_peered":"2021-02-10T04:25:49.006147+0000","last_unstale":"2021-02-10T04:27:41.100562+0000","last_undegraded":"2021-02-10T04:27:41.100562+0000","last_fullsized":"2021-02-10T04:27:41.100562+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'34","last_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:27:18.990088+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":22,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"58'2","reported_seq":"66","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.726542+0000","last_change":"2021-02-10T04:27:03.440844+0000","last_active":"2021-02-10T04:27:47.726542+0000","last_peered":"2021-02-10T04:27:47.726542+0000","last_clean":"2021-02-10T04:27:47.726542+0000","last_became_active":"2021-02-10T04:25:51.026251+0000","last_became_peered":"2021-02-10T04:25:51.026251+0000","last_unstale":"2021-02-10T04:27:47.726542+0000","last_undegraded":"2021-02-10T04:27:47.726542+0000","last_fullsized":"2021-02-10T04:27:47.726542+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:03.440780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:03.440780+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"37","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375751+0000","last_change":"2021-02-10T04:27:37.196914+0000","last_active":"2021-02-10T04:27:41.375751+0000","last_peered":"2021-02-10T04:27:41.375751+0000","last_clean":"2021-02-10T04:27:41.375751+0000","last_became_active":"2021-02-10T04:25:53.049487+0000","last_became_peered":"2021-02-10T04:25:53.049487+0000","last_unstale":"2021-02-10T04:27:41.375751+0000","last_undegraded":"2021-02-10T04:27:41.375751+0000","last_fullsized":"2021-02-10T04:27:41.375751+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:37.196846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:37.196846+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"49","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376166+0000","last_change":"2021-02-10T04:27:06.934347+0000","last_active":"2021-02-10T04:27:41.376166+0000","last_peered":"2021-02-10T04:27:41.376166+0000","last_clean":"2021-02-10T04:27:41.376166+0000","last_became_active":"2021-02-10T04:25:34.331779+0000","last_became_peered":"2021-02-10T04:25:34.331779+0000","last_unstale":"2021-02-10T04:27:41.376166+0000","last_undegraded":"2021-02-10T04:27:41.376166+0000","last_fullsized":"2021-02-10T04:27:41.376166+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:06.934282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:06.934282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"63'31","reported_seq":"104","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.100871+0000","last_change":"2021-02-10T04:26:56.059248+0000","last_active":"2021-02-10T04:27:41.100871+0000","last_peered":"2021-02-10T04:27:41.100871+0000","last_clean":"2021-02-10T04:27:41.100871+0000","last_became_active":"2021-02-10T04:25:49.007872+0000","last_became_peered":"2021-02-10T04:25:49.007872+0000","last_unstale":"2021-02-10T04:27:41.100871+0000","last_undegraded":"2021-02-10T04:27:41.100871+0000","last_fullsized":"2021-02-10T04:27:41.100871+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'31","last_scrub_stamp":"2021-02-10T04:26:56.059186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:26:56.059186+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":66,"num_read_kb":42,"num_write":34,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.099631+0000","last_change":"2021-02-10T04:27:31.650126+0000","last_active":"2021-02-10T04:27:41.099631+0000","last_peered":"2021-02-10T04:27:41.099631+0000","last_clean":"2021-02-10T04:27:41.099631+0000","last_became_active":"2021-02-10T04:25:51.027624+0000","last_became_peered":"2021-02-10T04:25:51.027624+0000","last_unstale":"2021-02-10T04:27:41.099631+0000","last_undegraded":"2021-02-10T04:27:41.099631+0000","last_fullsized":"2021-02-10T04:27:41.099631+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:27:31.650033+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376568+0000","last_change":"2021-02-10T04:27:28.264288+0000","last_active":"2021-02-10T04:27:41.376568+0000","last_peered":"2021-02-10T04:27:41.376568+0000","last_clean":"2021-02-10T04:27:41.376568+0000","last_became_active":"2021-02-10T04:25:53.149677+0000","last_became_peered":"2021-02-10T04:25:53.149677+0000","last_unstale":"2021-02-10T04:27:41.376568+0000","last_undegraded":"2021-02-10T04:27:41.376568+0000","last_fullsized":"2021-02-10T04:27:41.376568+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:28.264204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:28.264204+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"166","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:51.718626+0000","last_change":"2021-02-10T04:26:58.989515+0000","last_active":"2021-02-10T04:27:51.718626+0000","last_peered":"2021-02-10T04:27:51.718626+0000","last_clean":"2021-02-10T04:27:51.718626+0000","last_became_active":"2021-02-10T04:25:49.013969+0000","last_became_peered":"2021-02-10T04:25:49.013969+0000","last_unstale":"2021-02-10T04:27:51.718626+0000","last_undegraded":"2021-02-10T04:27:51.718626+0000","last_fullsized":"2021-02-10T04:27:51.718626+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:26:58.989447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:26:58.989447+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":121,"num_read_kb":116,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"53","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:44.556846+0000","last_change":"2021-02-10T04:27:44.556805+0000","last_active":"2021-02-10T04:27:44.556846+0000","last_peered":"2021-02-10T04:27:44.556846+0000","last_clean":"2021-02-10T04:27:44.556846+0000","last_became_active":"2021-02-10T04:25:34.331972+0000","last_became_peered":"2021-02-10T04:25:34.331972+0000","last_unstale":"2021-02-10T04:27:44.556846+0000","last_undegraded":"2021-02-10T04:27:44.556846+0000","last_fullsized":"2021-02-10T04:27:44.556846+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:44.556741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:44.556741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"82'29","reported_seq":"96","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.101092+0000","last_change":"2021-02-10T04:27:38.176842+0000","last_active":"2021-02-10T04:27:41.101092+0000","last_peered":"2021-02-10T04:27:41.101092+0000","last_clean":"2021-02-10T04:27:41.101092+0000","last_became_active":"2021-02-10T04:25:49.002843+0000","last_became_peered":"2021-02-10T04:25:49.002843+0000","last_unstale":"2021-02-10T04:27:41.101092+0000","last_undegraded":"2021-02-10T04:27:41.101092+0000","last_fullsized":"2021-02-10T04:27:41.101092+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'29","last_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:27:38.176754+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":32,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"39","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375693+0000","last_change":"2021-02-10T04:27:16.564733+0000","last_active":"2021-02-10T04:27:41.375693+0000","last_peered":"2021-02-10T04:27:41.375693+0000","last_clean":"2021-02-10T04:27:41.375693+0000","last_became_active":"2021-02-10T04:25:51.025306+0000","last_became_peered":"2021-02-10T04:25:51.025306+0000","last_unstale":"2021-02-10T04:27:41.375693+0000","last_undegraded":"2021-02-10T04:27:41.375693+0000","last_fullsized":"2021-02-10T04:27:41.375693+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:27:16.564638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376851+0000","last_change":"2021-02-10T04:27:18.115804+0000","last_active":"2021-02-10T04:27:41.376851+0000","last_peered":"2021-02-10T04:27:41.376851+0000","last_clean":"2021-02-10T04:27:41.376851+0000","last_became_active":"2021-02-10T04:25:53.037506+0000","last_became_peered":"2021-02-10T04:25:53.037506+0000","last_unstale":"2021-02-10T04:27:41.376851+0000","last_undegraded":"2021-02-10T04:27:41.376851+0000","last_fullsized":"2021-02-10T04:27:41.376851+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:18.115741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:18.115741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"41","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375191+0000","last_change":"2021-02-10T04:27:39.268187+0000","last_active":"2021-02-10T04:27:41.375191+0000","last_peered":"2021-02-10T04:27:41.375191+0000","last_clean":"2021-02-10T04:27:41.375191+0000","last_became_active":"2021-02-10T04:25:49.010614+0000","last_became_peered":"2021-02-10T04:25:49.010614+0000","last_unstale":"2021-02-10T04:27:41.375191+0000","last_undegraded":"2021-02-10T04:27:41.375191+0000","last_fullsized":"2021-02-10T04:27:41.375191+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:39.268114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:39.268114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"51","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375282+0000","last_change":"2021-02-10T04:26:38.013836+0000","last_active":"2021-02-10T04:27:41.375282+0000","last_peered":"2021-02-10T04:27:41.375282+0000","last_clean":"2021-02-10T04:27:41.375282+0000","last_became_active":"2021-02-10T04:25:36.334230+0000","last_became_peered":"2021-02-10T04:25:36.334230+0000","last_unstale":"2021-02-10T04:27:41.375282+0000","last_undegraded":"2021-02-10T04:27:41.375282+0000","last_fullsized":"2021-02-10T04:27:41.375282+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:38.013770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:38.013770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"48","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375315+0000","last_change":"2021-02-10T04:26:44.252455+0000","last_active":"2021-02-10T04:27:41.375315+0000","last_peered":"2021-02-10T04:27:41.375315+0000","last_clean":"2021-02-10T04:27:41.375315+0000","last_became_active":"2021-02-10T04:25:36.343397+0000","last_became_peered":"2021-02-10T04:25:36.343397+0000","last_unstale":"2021-02-10T04:27:41.375315+0000","last_undegraded":"2021-02-10T04:27:41.375315+0000","last_fullsized":"2021-02-10T04:27:41.375315+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:44.252351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:26:44.252351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"62'15","reported_seq":"66","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.098203+0000","last_change":"2021-02-10T04:27:08.069986+0000","last_active":"2021-02-10T04:27:41.098203+0000","last_peered":"2021-02-10T04:27:41.098203+0000","last_clean":"2021-02-10T04:27:41.098203+0000","last_became_active":"2021-02-10T04:25:49.002471+0000","last_became_peered":"2021-02-10T04:25:49.002471+0000","last_unstale":"2021-02-10T04:27:41.098203+0000","last_undegraded":"2021-02-10T04:27:41.098203+0000","last_fullsized":"2021-02-10T04:27:41.098203+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:27:08.069899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:08.069899+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":15,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375911+0000","last_change":"2021-02-10T04:27:38.106291+0000","last_active":"2021-02-10T04:27:41.375911+0000","last_peered":"2021-02-10T04:27:41.375911+0000","last_clean":"2021-02-10T04:27:41.375911+0000","last_became_active":"2021-02-10T04:25:51.022821+0000","last_became_peered":"2021-02-10T04:25:51.022821+0000","last_unstale":"2021-02-10T04:27:41.375911+0000","last_undegraded":"2021-02-10T04:27:41.375911+0000","last_fullsized":"2021-02-10T04:27:41.375911+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:38.106209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:38.106209+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.100719+0000","last_change":"2021-02-10T04:27:10.051857+0000","last_active":"2021-02-10T04:27:41.100719+0000","last_peered":"2021-02-10T04:27:41.100719+0000","last_clean":"2021-02-10T04:27:41.100719+0000","last_became_active":"2021-02-10T04:25:53.040390+0000","last_became_peered":"2021-02-10T04:25:53.040390+0000","last_unstale":"2021-02-10T04:27:41.100719+0000","last_undegraded":"2021-02-10T04:27:41.100719+0000","last_fullsized":"2021-02-10T04:27:41.100719+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:10.051800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:10.051800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"42","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.098795+0000","last_change":"2021-02-10T04:27:30.657844+0000","last_active":"2021-02-10T04:27:41.098795+0000","last_peered":"2021-02-10T04:27:41.098795+0000","last_clean":"2021-02-10T04:27:41.098795+0000","last_became_active":"2021-02-10T04:25:49.001782+0000","last_became_peered":"2021-02-10T04:25:49.001782+0000","last_unstale":"2021-02-10T04:27:41.098795+0000","last_undegraded":"2021-02-10T04:27:41.098795+0000","last_fullsized":"2021-02-10T04:27:41.098795+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:30.657781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:30.657781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"47","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375221+0000","last_change":"2021-02-10T04:27:08.192835+0000","last_active":"2021-02-10T04:27:41.375221+0000","last_peered":"2021-02-10T04:27:41.375221+0000","last_clean":"2021-02-10T04:27:41.375221+0000","last_became_active":"2021-02-10T04:25:36.340009+0000","last_became_peered":"2021-02-10T04:25:36.340009+0000","last_unstale":"2021-02-10T04:27:41.375221+0000","last_undegraded":"2021-02-10T04:27:41.375221+0000","last_fullsized":"2021-02-10T04:27:41.375221+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:08.192772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:08.192772+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"62'16","reported_seq":"68","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.099853+0000","last_change":"2021-02-10T04:27:09.484054+0000","last_active":"2021-02-10T04:27:41.099853+0000","last_peered":"2021-02-10T04:27:41.099853+0000","last_clean":"2021-02-10T04:27:41.099853+0000","last_became_active":"2021-02-10T04:25:49.003521+0000","last_became_peered":"2021-02-10T04:25:49.003521+0000","last_unstale":"2021-02-10T04:27:41.099853+0000","last_undegraded":"2021-02-10T04:27:41.099853+0000","last_fullsized":"2021-02-10T04:27:41.099853+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'16","last_scrub_stamp":"2021-02-10T04:27:09.483988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:09.483988+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":16,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376842+0000","last_change":"2021-02-10T04:27:25.216335+0000","last_active":"2021-02-10T04:27:41.376842+0000","last_peered":"2021-02-10T04:27:41.376842+0000","last_clean":"2021-02-10T04:27:41.376842+0000","last_became_active":"2021-02-10T04:25:51.032201+0000","last_became_peered":"2021-02-10T04:25:51.032201+0000","last_unstale":"2021-02-10T04:27:41.376842+0000","last_undegraded":"2021-02-10T04:27:41.376842+0000","last_fullsized":"2021-02-10T04:27:41.376842+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:27:25.216260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:42.182071+0000","last_change":"2021-02-10T04:27:42.182019+0000","last_active":"2021-02-10T04:27:42.182071+0000","last_peered":"2021-02-10T04:27:42.182071+0000","last_clean":"2021-02-10T04:27:42.182071+0000","last_became_active":"2021-02-10T04:25:53.149552+0000","last_became_peered":"2021-02-10T04:25:53.149552+0000","last_unstale":"2021-02-10T04:27:42.182071+0000","last_undegraded":"2021-02-10T04:27:42.182071+0000","last_fullsized":"2021-02-10T04:27:42.182071+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:42.181940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:42.181940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"50","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376660+0000","last_change":"2021-02-10T04:26:57.084942+0000","last_active":"2021-02-10T04:27:41.376660+0000","last_peered":"2021-02-10T04:27:41.376660+0000","last_clean":"2021-02-10T04:27:41.376660+0000","last_became_active":"2021-02-10T04:25:36.338600+0000","last_became_peered":"2021-02-10T04:25:36.338600+0000","last_unstale":"2021-02-10T04:27:41.376660+0000","last_undegraded":"2021-02-10T04:27:41.376660+0000","last_fullsized":"2021-02-10T04:27:41.376660+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:26:57.084864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:57.084864+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"63'27","reported_seq":"97","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.099329+0000","last_change":"2021-02-10T04:27:10.457048+0000","last_active":"2021-02-10T04:27:41.099329+0000","last_peered":"2021-02-10T04:27:41.099329+0000","last_clean":"2021-02-10T04:27:41.099329+0000","last_became_active":"2021-02-10T04:25:49.002937+0000","last_became_peered":"2021-02-10T04:25:49.002937+0000","last_unstale":"2021-02-10T04:27:41.099329+0000","last_undegraded":"2021-02-10T04:27:41.099329+0000","last_fullsized":"2021-02-10T04:27:41.099329+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'27","last_scrub_stamp":"2021-02-10T04:27:10.456981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:10.456981+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":38,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.097747+0000","last_change":"2021-02-10T04:26:59.175043+0000","last_active":"2021-02-10T04:27:41.097747+0000","last_peered":"2021-02-10T04:27:41.097747+0000","last_clean":"2021-02-10T04:27:41.097747+0000","last_became_active":"2021-02-10T04:25:51.025636+0000","last_became_peered":"2021-02-10T04:25:51.025636+0000","last_unstale":"2021-02-10T04:27:41.097747+0000","last_undegraded":"2021-02-10T04:27:41.097747+0000","last_fullsized":"2021-02-10T04:27:41.097747+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:59.174908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:26:59.174908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376898+0000","last_change":"2021-02-10T04:27:01.184729+0000","last_active":"2021-02-10T04:27:41.376898+0000","last_peered":"2021-02-10T04:27:41.376898+0000","last_clean":"2021-02-10T04:27:41.376898+0000","last_became_active":"2021-02-10T04:25:53.045085+0000","last_became_peered":"2021-02-10T04:25:53.045085+0000","last_unstale":"2021-02-10T04:27:41.376898+0000","last_undegraded":"2021-02-10T04:27:41.376898+0000","last_fullsized":"2021-02-10T04:27:41.376898+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:01.184630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:01.184630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"53","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.100997+0000","last_change":"2021-02-10T04:27:25.026606+0000","last_active":"2021-02-10T04:27:41.100997+0000","last_peered":"2021-02-10T04:27:41.100997+0000","last_clean":"2021-02-10T04:27:41.100997+0000","last_became_active":"2021-02-10T04:25:35.324960+0000","last_became_peered":"2021-02-10T04:25:35.324960+0000","last_unstale":"2021-02-10T04:27:41.100997+0000","last_undegraded":"2021-02-10T04:27:41.100997+0000","last_fullsized":"2021-02-10T04:27:41.100997+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:27:25.026534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:25.026534+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"63'34","reported_seq":"111","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.374699+0000","last_change":"2021-02-10T04:27:20.342445+0000","last_active":"2021-02-10T04:27:41.374699+0000","last_peered":"2021-02-10T04:27:41.374699+0000","last_clean":"2021-02-10T04:27:41.374699+0000","last_became_active":"2021-02-10T04:25:48.997481+0000","last_became_peered":"2021-02-10T04:25:48.997481+0000","last_unstale":"2021-02-10T04:27:41.374699+0000","last_undegraded":"2021-02-10T04:27:41.374699+0000","last_fullsized":"2021-02-10T04:27:41.374699+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'34","last_scrub_stamp":"2021-02-10T04:27:20.342358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:20.342358+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":70,"num_read_kb":48,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:44.501196+0000","last_change":"2021-02-10T04:27:44.501111+0000","last_active":"2021-02-10T04:27:44.501196+0000","last_peered":"2021-02-10T04:27:44.501196+0000","last_clean":"2021-02-10T04:27:44.501196+0000","last_became_active":"2021-02-10T04:25:51.018706+0000","last_became_peered":"2021-02-10T04:25:51.018706+0000","last_unstale":"2021-02-10T04:27:44.501196+0000","last_undegraded":"2021-02-10T04:27:44.501196+0000","last_fullsized":"2021-02-10T04:27:44.501196+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:44.501024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:44.501024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375850+0000","last_change":"2021-02-10T04:27:26.234432+0000","last_active":"2021-02-10T04:27:41.375850+0000","last_peered":"2021-02-10T04:27:41.375850+0000","last_clean":"2021-02-10T04:27:41.375850+0000","last_became_active":"2021-02-10T04:25:53.150930+0000","last_became_peered":"2021-02-10T04:25:53.150930+0000","last_unstale":"2021-02-10T04:27:41.375850+0000","last_undegraded":"2021-02-10T04:27:41.375850+0000","last_fullsized":"2021-02-10T04:27:41.375850+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:26.234349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:26.234349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"58","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.098256+0000","last_change":"2021-02-10T04:26:44.264500+0000","last_active":"2021-02-10T04:27:41.098256+0000","last_peered":"2021-02-10T04:27:41.098256+0000","last_clean":"2021-02-10T04:27:41.098256+0000","last_became_active":"2021-02-10T04:25:36.339614+0000","last_became_peered":"2021-02-10T04:25:36.339614+0000","last_unstale":"2021-02-10T04:27:41.098256+0000","last_undegraded":"2021-02-10T04:27:41.098256+0000","last_fullsized":"2021-02-10T04:27:41.098256+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:44.264402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:44.264402+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"63'26","reported_seq":"90","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.378335+0000","last_change":"2021-02-10T04:27:26.403830+0000","last_active":"2021-02-10T04:27:41.378335+0000","last_peered":"2021-02-10T04:27:41.378335+0000","last_clean":"2021-02-10T04:27:41.378335+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:27:41.378335+0000","last_undegraded":"2021-02-10T04:27:41.378335+0000","last_fullsized":"2021-02-10T04:27:41.378335+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:27:26.403748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:26.403748+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":51,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"58'2","reported_seq":"65","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.116283+0000","last_change":"2021-02-10T04:27:33.210495+0000","last_active":"2021-02-10T04:27:47.116283+0000","last_peered":"2021-02-10T04:27:47.116283+0000","last_clean":"2021-02-10T04:27:47.116283+0000","last_became_active":"2021-02-10T04:25:51.031906+0000","last_became_peered":"2021-02-10T04:25:51.031906+0000","last_unstale":"2021-02-10T04:27:47.116283+0000","last_undegraded":"2021-02-10T04:27:47.116283+0000","last_fullsized":"2021-02-10T04:27:47.116283+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:33.210431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:33.210431+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"32","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375092+0000","last_change":"2021-02-10T04:25:53.148449+0000","last_active":"2021-02-10T04:27:41.375092+0000","last_peered":"2021-02-10T04:27:41.375092+0000","last_clean":"2021-02-10T04:27:41.375092+0000","last_became_active":"2021-02-10T04:25:53.148054+0000","last_became_peered":"2021-02-10T04:25:53.148054+0000","last_unstale":"2021-02-10T04:27:41.375092+0000","last_undegraded":"2021-02-10T04:27:41.375092+0000","last_fullsized":"2021-02-10T04:27:41.375092+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:25:52.015519+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376771+0000","last_change":"2021-02-10T04:26:58.112766+0000","last_active":"2021-02-10T04:27:41.376771+0000","last_peered":"2021-02-10T04:27:41.376771+0000","last_clean":"2021-02-10T04:27:41.376771+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:27:41.376771+0000","last_undegraded":"2021-02-10T04:27:41.376771+0000","last_fullsized":"2021-02-10T04:27:41.376771+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:58.112683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:26:58.112683+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"47","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375139+0000","last_change":"2021-02-10T04:27:04.148397+0000","last_active":"2021-02-10T04:27:41.375139+0000","last_peered":"2021-02-10T04:27:41.375139+0000","last_clean":"2021-02-10T04:27:41.375139+0000","last_became_active":"2021-02-10T04:25:36.340895+0000","last_became_peered":"2021-02-10T04:25:36.340895+0000","last_unstale":"2021-02-10T04:27:41.375139+0000","last_undegraded":"2021-02-10T04:27:41.375139+0000","last_fullsized":"2021-02-10T04:27:41.375139+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:04.148329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:04.148329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"63'21","reported_seq":"84","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.378009+0000","last_change":"2021-02-10T04:26:59.205120+0000","last_active":"2021-02-10T04:27:41.378009+0000","last_peered":"2021-02-10T04:27:41.378009+0000","last_clean":"2021-02-10T04:27:41.378009+0000","last_became_active":"2021-02-10T04:25:49.011093+0000","last_became_peered":"2021-02-10T04:25:49.011093+0000","last_unstale":"2021-02-10T04:27:41.378009+0000","last_undegraded":"2021-02-10T04:27:41.378009+0000","last_fullsized":"2021-02-10T04:27:41.378009+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2021-02-10T04:26:59.205037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:26:59.205037+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":28,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376564+0000","last_change":"2021-02-10T04:27:24.209544+0000","last_active":"2021-02-10T04:27:41.376564+0000","last_peered":"2021-02-10T04:27:41.376564+0000","last_clean":"2021-02-10T04:27:41.376564+0000","last_became_active":"2021-02-10T04:25:53.042697+0000","last_became_peered":"2021-02-10T04:25:53.042697+0000","last_unstale":"2021-02-10T04:27:41.376564+0000","last_undegraded":"2021-02-10T04:27:41.376564+0000","last_fullsized":"2021-02-10T04:27:41.376564+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:27:24.209460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"78'3","reported_seq":"67","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.117041+0000","last_change":"2021-02-10T04:27:13.268467+0000","last_active":"2021-02-10T04:27:47.117041+0000","last_peered":"2021-02-10T04:27:47.117041+0000","last_clean":"2021-02-10T04:27:47.117041+0000","last_became_active":"2021-02-10T04:25:51.025218+0000","last_became_peered":"2021-02-10T04:25:51.025218+0000","last_unstale":"2021-02-10T04:27:47.117041+0000","last_undegraded":"2021-02-10T04:27:47.117041+0000","last_fullsized":"2021-02-10T04:27:47.117041+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:27:13.268412+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"53","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.099760+0000","last_change":"2021-02-10T04:26:46.092082+0000","last_active":"2021-02-10T04:27:41.099760+0000","last_peered":"2021-02-10T04:27:41.099760+0000","last_clean":"2021-02-10T04:27:41.099760+0000","last_became_active":"2021-02-10T04:25:36.331437+0000","last_became_peered":"2021-02-10T04:25:36.331437+0000","last_unstale":"2021-02-10T04:27:41.099760+0000","last_undegraded":"2021-02-10T04:27:41.099760+0000","last_fullsized":"2021-02-10T04:27:41.099760+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:46.092013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:46.092013+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"62'17","reported_seq":"71","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.101049+0000","last_change":"2021-02-10T04:27:30.012988+0000","last_active":"2021-02-10T04:27:41.101049+0000","last_peered":"2021-02-10T04:27:41.101049+0000","last_clean":"2021-02-10T04:27:41.101049+0000","last_became_active":"2021-02-10T04:25:49.004933+0000","last_became_peered":"2021-02-10T04:25:49.004933+0000","last_unstale":"2021-02-10T04:27:41.101049+0000","last_undegraded":"2021-02-10T04:27:41.101049+0000","last_fullsized":"2021-02-10T04:27:41.101049+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'17","last_scrub_stamp":"2021-02-10T04:27:30.012927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:30.012927+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.377864+0000","last_change":"2021-02-10T04:27:33.382403+0000","last_active":"2021-02-10T04:27:41.377864+0000","last_peered":"2021-02-10T04:27:41.377864+0000","last_clean":"2021-02-10T04:27:41.377864+0000","last_became_active":"2021-02-10T04:25:53.043027+0000","last_became_peered":"2021-02-10T04:25:53.043027+0000","last_unstale":"2021-02-10T04:27:41.377864+0000","last_undegraded":"2021-02-10T04:27:41.377864+0000","last_fullsized":"2021-02-10T04:27:41.377864+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:33.382319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:33.382319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"58'2","reported_seq":"65","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.116957+0000","last_change":"2021-02-10T04:26:50.624141+0000","last_active":"2021-02-10T04:27:47.116957+0000","last_peered":"2021-02-10T04:27:47.116957+0000","last_clean":"2021-02-10T04:27:47.116957+0000","last_became_active":"2021-02-10T04:25:51.025268+0000","last_became_peered":"2021-02-10T04:25:51.025268+0000","last_unstale":"2021-02-10T04:27:47.116957+0000","last_undegraded":"2021-02-10T04:27:47.116957+0000","last_fullsized":"2021-02-10T04:27:47.116957+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:26:50.624072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:26:50.624072+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"51","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.100475+0000","last_change":"2021-02-10T04:27:17.003396+0000","last_active":"2021-02-10T04:27:41.100475+0000","last_peered":"2021-02-10T04:27:41.100475+0000","last_clean":"2021-02-10T04:27:41.100475+0000","last_became_active":"2021-02-10T04:25:36.333883+0000","last_became_peered":"2021-02-10T04:25:36.333883+0000","last_unstale":"2021-02-10T04:27:41.100475+0000","last_undegraded":"2021-02-10T04:27:41.100475+0000","last_fullsized":"2021-02-10T04:27:41.100475+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:17.003304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:17.003304+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"62'13","reported_seq":"63","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.377063+0000","last_change":"2021-02-10T04:27:14.256603+0000","last_active":"2021-02-10T04:27:41.377063+0000","last_peered":"2021-02-10T04:27:41.377063+0000","last_clean":"2021-02-10T04:27:41.377063+0000","last_became_active":"2021-02-10T04:25:49.001725+0000","last_became_peered":"2021-02-10T04:25:49.001725+0000","last_unstale":"2021-02-10T04:27:41.377063+0000","last_undegraded":"2021-02-10T04:27:41.377063+0000","last_fullsized":"2021-02-10T04:27:41.377063+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'13","last_scrub_stamp":"2021-02-10T04:27:14.256539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:14.256539+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"37","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375579+0000","last_change":"2021-02-10T04:26:55.627032+0000","last_active":"2021-02-10T04:27:41.375579+0000","last_peered":"2021-02-10T04:27:41.375579+0000","last_clean":"2021-02-10T04:27:41.375579+0000","last_became_active":"2021-02-10T04:25:53.038218+0000","last_became_peered":"2021-02-10T04:25:53.038218+0000","last_unstale":"2021-02-10T04:27:41.375579+0000","last_undegraded":"2021-02-10T04:27:41.375579+0000","last_fullsized":"2021-02-10T04:27:41.375579+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:55.626967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:26:55.626967+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"40","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.100713+0000","last_change":"2021-02-10T04:27:37.160395+0000","last_active":"2021-02-10T04:27:41.100713+0000","last_peered":"2021-02-10T04:27:41.100713+0000","last_clean":"2021-02-10T04:27:41.100713+0000","last_became_active":"2021-02-10T04:25:51.028302+0000","last_became_peered":"2021-02-10T04:25:51.028302+0000","last_unstale":"2021-02-10T04:27:41.100713+0000","last_undegraded":"2021-02-10T04:27:41.100713+0000","last_fullsized":"2021-02-10T04:27:41.100713+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:37.160328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:37.160328+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"47","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375998+0000","last_change":"2021-02-10T04:27:00.052115+0000","last_active":"2021-02-10T04:27:41.375998+0000","last_peered":"2021-02-10T04:27:41.375998+0000","last_clean":"2021-02-10T04:27:41.375998+0000","last_became_active":"2021-02-10T04:25:36.344109+0000","last_became_peered":"2021-02-10T04:25:36.344109+0000","last_unstale":"2021-02-10T04:27:41.375998+0000","last_undegraded":"2021-02-10T04:27:41.375998+0000","last_fullsized":"2021-02-10T04:27:41.375998+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:00.052028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:00.052028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"62'21","reported_seq":"78","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.377744+0000","last_change":"2021-02-10T04:27:00.215115+0000","last_active":"2021-02-10T04:27:41.377744+0000","last_peered":"2021-02-10T04:27:41.377744+0000","last_clean":"2021-02-10T04:27:41.377744+0000","last_became_active":"2021-02-10T04:25:49.011699+0000","last_became_peered":"2021-02-10T04:25:49.011699+0000","last_unstale":"2021-02-10T04:27:41.377744+0000","last_undegraded":"2021-02-10T04:27:41.377744+0000","last_fullsized":"2021-02-10T04:27:41.377744+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'21","last_scrub_stamp":"2021-02-10T04:27:00.215037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:00.215037+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375050+0000","last_change":"2021-02-10T04:27:35.166330+0000","last_active":"2021-02-10T04:27:41.375050+0000","last_peered":"2021-02-10T04:27:41.375050+0000","last_clean":"2021-02-10T04:27:41.375050+0000","last_became_active":"2021-02-10T04:25:53.037903+0000","last_became_peered":"2021-02-10T04:25:53.037903+0000","last_unstale":"2021-02-10T04:27:41.375050+0000","last_undegraded":"2021-02-10T04:27:41.375050+0000","last_fullsized":"2021-02-10T04:27:41.375050+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:35.166250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:35.166250+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"39","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.374929+0000","last_change":"2021-02-10T04:27:11.606664+0000","last_active":"2021-02-10T04:27:41.374929+0000","last_peered":"2021-02-10T04:27:41.374929+0000","last_clean":"2021-02-10T04:27:41.374929+0000","last_became_active":"2021-02-10T04:25:51.025882+0000","last_became_peered":"2021-02-10T04:25:51.025882+0000","last_unstale":"2021-02-10T04:27:41.374929+0000","last_undegraded":"2021-02-10T04:27:41.374929+0000","last_fullsized":"2021-02-10T04:27:41.374929+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:11.606603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:11.606603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"62'2","reported_seq":"73","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.116932+0000","last_change":"2021-02-10T04:26:48.599352+0000","last_active":"2021-02-10T04:27:47.116932+0000","last_peered":"2021-02-10T04:27:47.116932+0000","last_clean":"2021-02-10T04:27:47.116932+0000","last_became_active":"2021-02-10T04:25:34.331199+0000","last_became_peered":"2021-02-10T04:25:34.331199+0000","last_unstale":"2021-02-10T04:27:47.116932+0000","last_undegraded":"2021-02-10T04:27:47.116932+0000","last_fullsized":"2021-02-10T04:27:47.116932+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2021-02-10T04:26:48.599257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:48.599257+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"63'29","reported_seq":"98","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.097608+0000","last_change":"2021-02-10T04:27:13.081465+0000","last_active":"2021-02-10T04:27:41.097608+0000","last_peered":"2021-02-10T04:27:41.097608+0000","last_clean":"2021-02-10T04:27:41.097608+0000","last_became_active":"2021-02-10T04:25:49.015913+0000","last_became_peered":"2021-02-10T04:25:49.015913+0000","last_unstale":"2021-02-10T04:27:41.097608+0000","last_undegraded":"2021-02-10T04:27:41.097608+0000","last_fullsized":"2021-02-10T04:27:41.097608+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'29","last_scrub_stamp":"2021-02-10T04:27:13.081399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:13.081399+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":56,"num_read_kb":36,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.098541+0000","last_change":"2021-02-10T04:27:22.512316+0000","last_active":"2021-02-10T04:27:41.098541+0000","last_peered":"2021-02-10T04:27:41.098541+0000","last_clean":"2021-02-10T04:27:41.098541+0000","last_became_active":"2021-02-10T04:25:53.038249+0000","last_became_peered":"2021-02-10T04:25:53.038249+0000","last_unstale":"2021-02-10T04:27:41.098541+0000","last_undegraded":"2021-02-10T04:27:41.098541+0000","last_fullsized":"2021-02-10T04:27:41.098541+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:22.512245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:22.512245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"39","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375199+0000","last_change":"2021-02-10T04:26:51.605291+0000","last_active":"2021-02-10T04:27:41.375199+0000","last_peered":"2021-02-10T04:27:41.375199+0000","last_clean":"2021-02-10T04:27:41.375199+0000","last_became_active":"2021-02-10T04:25:51.016118+0000","last_became_peered":"2021-02-10T04:25:51.016118+0000","last_unstale":"2021-02-10T04:27:41.375199+0000","last_undegraded":"2021-02-10T04:27:41.375199+0000","last_fullsized":"2021-02-10T04:27:41.375199+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:51.605227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:26:51.605227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"49","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.100906+0000","last_change":"2021-02-10T04:26:50.001018+0000","last_active":"2021-02-10T04:27:41.100906+0000","last_peered":"2021-02-10T04:27:41.100906+0000","last_clean":"2021-02-10T04:27:41.100906+0000","last_became_active":"2021-02-10T04:25:34.334427+0000","last_became_peered":"2021-02-10T04:25:34.334427+0000","last_unstale":"2021-02-10T04:27:41.100906+0000","last_undegraded":"2021-02-10T04:27:41.100906+0000","last_fullsized":"2021-02-10T04:27:41.100906+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:50.000952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:50.000952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"63'20","reported_seq":"83","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.099533+0000","last_change":"2021-02-10T04:27:20.587671+0000","last_active":"2021-02-10T04:27:41.099533+0000","last_peered":"2021-02-10T04:27:41.099533+0000","last_clean":"2021-02-10T04:27:41.099533+0000","last_became_active":"2021-02-10T04:25:49.004346+0000","last_became_peered":"2021-02-10T04:25:49.004346+0000","last_unstale":"2021-02-10T04:27:41.099533+0000","last_undegraded":"2021-02-10T04:27:41.099533+0000","last_fullsized":"2021-02-10T04:27:41.099533+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'20","last_scrub_stamp":"2021-02-10T04:27:20.587602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:20.587602+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":27,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"38","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.376021+0000","last_change":"2021-02-10T04:27:13.006848+0000","last_active":"2021-02-10T04:27:41.376021+0000","last_peered":"2021-02-10T04:27:41.376021+0000","last_clean":"2021-02-10T04:27:41.376021+0000","last_became_active":"2021-02-10T04:25:53.148090+0000","last_became_peered":"2021-02-10T04:25:53.148090+0000","last_unstale":"2021-02-10T04:27:41.376021+0000","last_undegraded":"2021-02-10T04:27:41.376021+0000","last_fullsized":"2021-02-10T04:27:41.376021+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:27:13.006716+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"62","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375797+0000","last_change":"2021-02-10T04:26:44.609990+0000","last_active":"2021-02-10T04:27:41.375797+0000","last_peered":"2021-02-10T04:27:41.375797+0000","last_clean":"2021-02-10T04:27:41.375797+0000","last_became_active":"2021-02-10T04:25:36.344641+0000","last_became_peered":"2021-02-10T04:25:36.344641+0000","last_unstale":"2021-02-10T04:27:41.375797+0000","last_undegraded":"2021-02-10T04:27:41.375797+0000","last_fullsized":"2021-02-10T04:27:41.375797+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:26:44.609922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:44.609922+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"63'30","reported_seq":"103","reported_epoch":"83","state":"active+clean","last_fresh":"2021-02-10T04:27:41.375135+0000","last_change":"2021-02-10T04:27:02.039902+0000","last_active":"2021-02-10T04:27:41.375135+0000","last_peered":"2021-02-10T04:27:41.375135+0000","last_clean":"2021-02-10T04:27:41.375135+0000","last_became_active":"2021-02-10T04:25:49.002058+0000","last_became_peered":"2021-02-10T04:25:49.002058+0000","last_unstale":"2021-02-10T04:27:41.375135+0000","last_undegraded":"2021-02-10T04:27:41.375135+0000","last_fullsized":"2021-02-10T04:27:41.375135+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2021-02-10T04:27:02.039813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:02.039813+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":59,"num_read_kb":42,"num_write":31,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"58'2","reported_seq":"66","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.728076+0000","last_change":"2021-02-10T04:27:20.144947+0000","last_active":"2021-02-10T04:27:47.728076+0000","last_peered":"2021-02-10T04:27:47.728076+0000","last_clean":"2021-02-10T04:27:47.728076+0000","last_became_active":"2021-02-10T04:25:51.019515+0000","last_became_peered":"2021-02-10T04:25:51.019515+0000","last_unstale":"2021-02-10T04:27:47.728076+0000","last_undegraded":"2021-02-10T04:27:47.728076+0000","last_fullsized":"2021-02-10T04:27:47.728076+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:20.144855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:20.144855+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":1453,"num_read_kb":926,"num_write":734,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":746,"ondisk_log_size":746,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":121,"num_read_kb":116,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1945,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":21,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":5835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748364829,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6564,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738588,"statfs":{"total":95995035648,"available":95988314112,"internally_reserved":0,"allocated":880640,"data_stored":313320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.876,"15min":0.88200000000000001},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.85599999999999998,"15min":0.85899999999999999},"min":{"1min":0.57499999999999996,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.248,"5min":1.248,"15min":1.248},"last":1.0289999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.88700000000000001,"15min":0.89100000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":1.0049999999999999},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.0149999999999999,"15min":1.0209999999999999},"min":{"1min":0.64500000000000002,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.355,"5min":1.369,"15min":1.369},"last":0.78600000000000003}]},{"osd":2,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.018,"15min":1.026},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.95799999999999996,"15min":0.95999999999999996},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.93100000000000005}]},{"osd":3,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.998,"15min":1.01},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.147,"15min":1.153},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.381,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":1.0800000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":0.97499999999999998,"15min":0.95099999999999996},"min":{"1min":0.77300000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.2530000000000001,"5min":2.2530000000000001,"15min":2.2530000000000001},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.1060000000000001,"15min":1.1120000000000001},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.492,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":1.256}]},{"osd":5,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.161,"15min":1.1499999999999999},"min":{"1min":0.68899999999999995,"5min":0.624,"15min":0.624},"max":{"1min":1.714,"5min":1.714,"15min":1.714},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.2,"15min":1.1990000000000001},"min":{"1min":0.90900000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.498,"5min":1.538,"15min":1.538},"last":1.107}]},{"osd":6,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.246,"15min":1.2430000000000001},"min":{"1min":0.94199999999999995,"5min":0.86899999999999999,"15min":0.86899999999999999},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.276,"5min":1.236,"15min":1.2290000000000001},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":1.127}]}]},{"osd":6,"up_from":43,"seq":184683593761,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6608,"kb_used_data":904,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738544,"statfs":{"total":95995035648,"available":95988269056,"internally_reserved":0,"allocated":925696,"data_stored":316190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:18 2021","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.3460000000000001,"5min":1.478,"15min":1.478},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.95499999999999996,"5min":1.008,"15min":1.0169999999999999},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.626}]},{"osd":1,"last update":"Wed Feb 10 04:27:18 2021","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":1.0149999999999999,"15min":1.03},"min":{"1min":0.61599999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":2.121,"5min":2.121,"15min":2.121},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.042,"5min":0.99099999999999999,"15min":0.98299999999999998},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":0.78100000000000003}]},{"osd":2,"last update":"Wed Feb 10 04:27:18 2021","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0620000000000001,"15min":1.0609999999999999},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":0.90600000000000003},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.0289999999999999,"15min":1.014},"min":{"1min":0.73699999999999999,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.1920000000000002,"5min":2.1920000000000002,"15min":2.1920000000000002},"last":1.143}]},{"osd":3,"last update":"Wed Feb 10 04:27:18 2021","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.1439999999999999,"15min":1.1379999999999999},"min":{"1min":0.85699999999999998,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":2,"5min":2,"15min":2},"last":1.177},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.133,"15min":1.107},"min":{"1min":0.88,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":2.323,"5min":2.323,"15min":2.323},"last":0.876}]},{"osd":4,"last update":"Wed Feb 10 04:27:18 2021","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":0.99099999999999999,"15min":0.96099999999999997},"min":{"1min":0.61699999999999999,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.0800000000000001,"5min":2.0800000000000001,"15min":2.0800000000000001},"last":1.024},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.90500000000000003,"15min":0.89200000000000002},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.94299999999999995}]},{"osd":5,"last update":"Wed Feb 10 04:27:18 2021","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.0009999999999999,"15min":0.98699999999999999},"min":{"1min":0.67100000000000004,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.7130000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.419,"5min":1.179,"15min":1.139},"min":{"1min":0.83999999999999997,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.4550000000000001,"5min":2.4550000000000001,"15min":2.4550000000000001},"last":1.258}]},{"osd":7,"last update":"Wed Feb 10 04:27:40 2021","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.1060000000000001,"15min":1.0940000000000001},"min":{"1min":0.75800000000000001,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.901,"5min":1.901,"15min":1.901},"last":1.22},{"interface":"front","average":{"1min":1.506,"5min":1.409,"15min":1.393},"min":{"1min":0.90300000000000002,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":2.5339999999999998,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":1.333}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":11304,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":10304,"kb_avail":93733848,"statfs":{"total":95995035648,"available":95983460352,"internally_reserved":0,"allocated":1015808,"data_stored":319270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":10551296},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:26:53 2021","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.66600000000000004,"15min":0.626},"min":{"1min":0.56899999999999995,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.425,"5min":1.425,"15min":1.425},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.746,"15min":0.69199999999999995},"min":{"1min":0.57299999999999995,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.81899999999999995}]},{"osd":2,"last update":"Wed Feb 10 04:27:14 2021","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.80800000000000005,"15min":0.76400000000000001},"min":{"1min":0.58199999999999996,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.83499999999999996,"15min":0.79900000000000004},"min":{"1min":0.51000000000000001,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":0.79200000000000004}]},{"osd":3,"last update":"Wed Feb 10 04:27:31 2021","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.93100000000000005,"15min":0.91400000000000003},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.5860000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.026,"5min":1.018,"15min":1.014},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.226,"5min":2.226,"15min":2.226},"last":1.0489999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:27:46 2021","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.157,"15min":1.1459999999999999},"min":{"1min":0.86699999999999999,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":1.736,"5min":2.004,"15min":2.004},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.23,"5min":1.218,"15min":1.21},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.6419999999999999,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":0.94199999999999995}]},{"osd":5,"last update":"Wed Feb 10 04:27:08 2021","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.25,"15min":1.2769999999999999},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":2.7170000000000001,"5min":5.5860000000000003,"15min":5.5860000000000003},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.6100000000000001,"15min":1.6559999999999999},"min":{"1min":0.97399999999999998,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":2.105,"5min":9.5310000000000006,"15min":9.5310000000000006},"last":1.163}]},{"osd":6,"last update":"Wed Feb 10 04:27:24 2021","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.2889999999999999,"15min":1.29},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.4449999999999998,"5min":2.4449999999999998,"15min":2.4449999999999998},"last":1.131},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.264,"15min":1.292},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.905,"5min":3.653,"15min":3.653},"last":0.91900000000000004}]},{"osd":7,"last update":"Wed Feb 10 04:27:36 2021","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.639,"15min":1.6890000000000001},"min":{"1min":1.0529999999999999,"5min":1.004,"15min":1.004},"max":{"1min":2.2090000000000001,"5min":7.5540000000000003,"15min":7.5540000000000003},"last":1.038},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.623,"15min":1.6619999999999999},"min":{"1min":1.006,"5min":1.006,"15min":1.006},"max":{"1min":2.556,"5min":8.1020000000000003,"15min":8.1020000000000003},"last":1.2649999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6808,"kb_used_data":848,"kb_used_omap":0,"kb_used_meta":5952,"kb_avail":93738344,"statfs":{"total":95995035648,"available":95988064256,"internally_reserved":0,"allocated":868352,"data_stored":313029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6094848},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 04:26:53 2021","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.64300000000000002,"15min":0.58699999999999997},"min":{"1min":0.53100000000000003,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.006,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.79100000000000004,"15min":0.73299999999999998},"min":{"1min":0.67400000000000004,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":0.65500000000000003}]},{"osd":2,"last update":"Wed Feb 10 04:27:12 2021","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.79300000000000004,"15min":0.77300000000000002},"min":{"1min":0.52500000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.958,"5min":1.958,"15min":1.958},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.79700000000000004,"15min":0.79300000000000004},"min":{"1min":0.56100000000000005,"5min":0.495,"15min":0.495},"max":{"1min":1.4390000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.92800000000000005}]},{"osd":3,"last update":"Wed Feb 10 04:27:27 2021","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.80400000000000005,"15min":0.81200000000000006},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.224,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":1.0680000000000001},{"interface":"front","average":{"1min":0.89500000000000002,"5min":1.026,"15min":1.036},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.3049999999999999,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":1.377}]},{"osd":4,"last update":"Wed Feb 10 04:27:45 2021","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.1259999999999999,"15min":1.169},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.3859999999999999,"5min":1.613,"15min":1.613},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.089,"5min":1.141,"15min":1.1539999999999999},"min":{"1min":0.76600000000000001,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.522,"5min":1.597,"15min":1.597},"last":1.2090000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:27:01 2021","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.2,"15min":1.2190000000000001},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":1.3149999999999999},{"interface":"front","average":{"1min":1.218,"5min":1.2889999999999999,"15min":1.3009999999999999},"min":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"max":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.51}]},{"osd":6,"last update":"Wed Feb 10 04:27:20 2021","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.2909999999999999,"15min":1.3029999999999999},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":1.601,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":1.405},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.351,"15min":1.363},"min":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":2.1019999999999999,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":1.6200000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.2330000000000001,"15min":1.252},"min":{"1min":0.94599999999999995,"5min":0.93500000000000005,"15min":0.93500000000000005},"max":{"1min":1.5900000000000001,"5min":1.76,"15min":1.76},"last":1.155},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3500000000000001,"15min":1.3640000000000001},"min":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"max":{"1min":1.611,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":1.4810000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411377,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6552,"kb_used_data":784,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738600,"statfs":{"total":95995035648,"available":95988326400,"internally_reserved":0,"allocated":802816,"data_stored":311388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:10 2021","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.70899999999999996,"15min":0.68300000000000005},"min":{"1min":0.54400000000000004,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.97999999999999998,"5min":1.349,"15min":1.349},"last":0.497},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.73299999999999998,"15min":0.71099999999999997},"min":{"1min":0.59499999999999997,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.96399999999999997,"5min":1.274,"15min":1.274},"last":0.54100000000000004}]},{"osd":1,"last update":"Wed Feb 10 04:27:10 2021","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.81200000000000006,"15min":0.76900000000000002},"min":{"1min":0.57499999999999996,"5min":0.495,"15min":0.495},"max":{"1min":1.3069999999999999,"5min":1.726,"15min":1.726},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.71299999999999997,"15min":0.69099999999999995},"min":{"1min":0.54900000000000004,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.58299999999999996}]},{"osd":3,"last update":"Wed Feb 10 04:27:31 2021","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.81499999999999995,"15min":0.80400000000000005},"min":{"1min":0.57999999999999996,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.90700000000000003,"5min":1.274,"15min":1.274},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.996,"15min":0.97099999999999997},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.2,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":0.877}]},{"osd":4,"last update":"Wed Feb 10 04:27:44 2021","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.181,"15min":1.181},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.4359999999999999,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":1.2609999999999999},{"interface":"front","average":{"1min":1.141,"5min":1.119,"15min":1.1220000000000001},"min":{"1min":0.80900000000000005,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":0.99399999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:26:59 2021","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.3180000000000001,"15min":1.3169999999999999},"min":{"1min":1.014,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":1.71,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":1.137},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.1879999999999999,"15min":1.1819999999999999},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":1.24}]},{"osd":6,"last update":"Wed Feb 10 04:27:23 2021","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.284,"15min":1.2989999999999999},"min":{"1min":0.875,"5min":0.875,"15min":0.875},"max":{"1min":1.504,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.26,"5min":1.409,"15min":1.4339999999999999},"min":{"1min":1.05,"5min":1.0489999999999999,"15min":1.0489999999999999},"max":{"1min":1.635,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.0629999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.4690000000000001,"15min":1.494},"min":{"1min":1.077,"5min":1.073,"15min":1.073},"max":{"1min":1.653,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":1.198},{"interface":"front","average":{"1min":1.171,"5min":1.252,"15min":1.266},"min":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"max":{"1min":1.52,"5min":1.6160000000000001,"15min":1.6160000000000001},"last":1.0820000000000001}]}]},{"osd":3,"up_from":25,"seq":107374182443,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6988,"kb_used_data":964,"kb_used_omap":0,"kb_used_meta":6016,"kb_avail":93738164,"statfs":{"total":95995035648,"available":95987879936,"internally_reserved":0,"allocated":987136,"data_stored":317370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6160384},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:28 2021","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.74299999999999999,"15min":0.73499999999999999},"min":{"1min":0.50900000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.0600000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.88300000000000001,"15min":0.86699999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":4.2759999999999998,"5min":4.2759999999999998,"15min":4.2759999999999998},"last":0.56200000000000006}]},{"osd":1,"last update":"Wed Feb 10 04:27:28 2021","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.90300000000000002,"15min":0.877},"min":{"1min":0.64200000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":3.246,"5min":3.246,"15min":3.246},"last":0.66900000000000004},{"interface":"front","average":{"1min":1.0549999999999999,"5min":0.89700000000000002,"15min":0.86199999999999999},"min":{"1min":0.57999999999999996,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":5.4039999999999999,"5min":5.4039999999999999,"15min":5.4039999999999999},"last":0.60799999999999998}]},{"osd":2,"last update":"Wed Feb 10 04:27:28 2021","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":0.97199999999999998,"15min":0.95099999999999996},"min":{"1min":0.60299999999999998,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":4.2309999999999999,"5min":4.2309999999999999,"15min":4.2309999999999999},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.96099999999999997,"15min":0.97599999999999998},"min":{"1min":0.65900000000000003,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.2609999999999999,"5min":1.681,"15min":1.681},"last":0.998}]},{"osd":4,"last update":"Wed Feb 10 04:27:43 2021","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.25,"15min":1.204},"min":{"1min":0.84099999999999997,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.9650000000000001,"5min":3.371,"15min":3.371},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.266,"5min":1.266,"15min":1.2250000000000001},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":1.8280000000000001,"5min":3.5430000000000001,"15min":3.5430000000000001},"last":1.236}]},{"osd":5,"last update":"Wed Feb 10 04:27:00 2021","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.2350000000000001,"15min":1.1930000000000001},"min":{"1min":0.94799999999999995,"5min":0.88500000000000001,"15min":0.88500000000000001},"max":{"1min":3.496,"5min":3.496,"15min":3.496},"last":1.069},{"interface":"front","average":{"1min":1.466,"5min":1.3,"15min":1.272},"min":{"1min":1.143,"5min":0.97399999999999998,"15min":0.97399999999999998},"max":{"1min":3.742,"5min":3.742,"15min":3.742},"last":1.1739999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:27:24 2021","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.429,"15min":1.4059999999999999},"min":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"max":{"1min":3.6859999999999999,"5min":3.6859999999999999,"15min":3.6859999999999999},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.514,"5min":1.47,"15min":1.462},"min":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"max":{"1min":3.8759999999999999,"5min":3.8759999999999999,"15min":3.8759999999999999},"last":1.0509999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:27:40 2021","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":1.5109999999999999,"15min":1.5049999999999999},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"max":{"1min":4.1319999999999997,"5min":4.1319999999999997,"15min":4.1319999999999997},"last":1.262},{"interface":"front","average":{"1min":1.587,"5min":1.5629999999999999,"15min":1.5589999999999999},"min":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"max":{"1min":4.0010000000000003,"5min":4.0010000000000003,"15min":4.0010000000000003},"last":1.1140000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986217,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6812,"kb_used_data":916,"kb_used_omap":0,"kb_used_meta":5888,"kb_avail":93738340,"statfs":{"total":95995035648,"available":95988060160,"internally_reserved":0,"allocated":937984,"data_stored":316129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6029312},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:43 2021","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.96599999999999997,"15min":0.997},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.95499999999999996,"15min":0.97499999999999998},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.4810000000000001,"5min":1.633,"15min":1.633},"last":0.85999999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:27:43 2021","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.89500000000000002,"15min":0.91700000000000004},"min":{"1min":0.58199999999999996,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":1.02,"5min":1.012,"15min":1.018},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.583,"5min":1.583,"15min":1.583},"last":1.0920000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:27:43 2021","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.0800000000000001,"15min":1.083},"min":{"1min":0.69399999999999995,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.766,"5min":1.766,"15min":1.766},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.1719999999999999,"15min":1.161},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.5469999999999999,"5min":1.768,"15min":1.768},"last":1.0129999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:27:43 2021","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.097,"15min":1.0609999999999999},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":5.1230000000000002,"5min":5.1230000000000002,"15min":5.1230000000000002},"last":1.147},{"interface":"front","average":{"1min":1.302,"5min":1.129,"15min":1.1080000000000001},"min":{"1min":0.79900000000000004,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":5.1710000000000003,"5min":5.1710000000000003,"15min":5.1710000000000003},"last":0.90700000000000003}]},{"osd":5,"last update":"Wed Feb 10 04:27:00 2021","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":0.88,"15min":0.83599999999999997},"min":{"1min":0.63200000000000001,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":1.389},{"interface":"front","average":{"1min":1.05,"5min":0.86599999999999999,"15min":0.83599999999999997},"min":{"1min":0.74099999999999999,"5min":0.624,"15min":0.624},"max":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.5740000000000001},"last":1.075}]},{"osd":6,"last update":"Wed Feb 10 04:27:19 2021","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.0840000000000001,"15min":1.0640000000000001},"min":{"1min":0.79200000000000004,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.27,"15min":1.252},"min":{"1min":1.139,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.714,"5min":1.74,"15min":1.74},"last":1.3080000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:27:40 2021","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.94899999999999995,"5min":0.88600000000000001,"15min":0.88600000000000001},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":1.258},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3720000000000001,"15min":1.3740000000000001},"min":{"1min":1.073,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":1.6699999999999999,"5min":1.9450000000000001,"15min":1.9450000000000001},"last":1.2330000000000001}]}]},{"osd":5,"up_from":37,"seq":158913789990,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6536,"kb_used_data":832,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738616,"statfs":{"total":95995035648,"available":95988342784,"internally_reserved":0,"allocated":851968,"data_stored":312198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:02 2021","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.98899999999999999,"15min":0.998},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.516,"5min":1.516,"15min":1.516},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.873,"5min":0.90400000000000003,"15min":0.90900000000000003},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.238,"5min":1.365,"15min":1.365},"last":0.90200000000000002}]},{"osd":1,"last update":"Wed Feb 10 04:27:02 2021","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.92700000000000005,"15min":0.93100000000000005},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.502,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.998,"15min":1.002},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.419,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.69999999999999996}]},{"osd":2,"last update":"Wed Feb 10 04:27:02 2021","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.997,"15min":0.995},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":1.1859999999999999},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.831,"5min":1.831,"15min":1.831},"last":1.056}]},{"osd":3,"last update":"Wed Feb 10 04:27:02 2021","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.0980000000000001,"15min":1.0920000000000001},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.087,"5min":1.0720000000000001,"15min":1.0700000000000001},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.7030000000000001,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":0.93799999999999994}]},{"osd":4,"last update":"Wed Feb 10 04:27:02 2021","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":0.92000000000000004,"15min":0.90500000000000003},"min":{"1min":0.65300000000000002,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":1.01},{"interface":"front","average":{"1min":1.1699999999999999,"5min":0.98999999999999999,"15min":0.95999999999999996},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":1.341}]},{"osd":6,"last update":"Wed Feb 10 04:27:23 2021","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.149,"15min":1.119},"min":{"1min":0.88100000000000001,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.085,"15min":1.0549999999999999},"min":{"1min":0.92100000000000004,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.3340000000000001,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":1.1499999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:27:38 2021","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.1910000000000001,"15min":1.177},"min":{"1min":0.93400000000000005,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.2440000000000002,"5min":2.2440000000000002,"15min":2.2440000000000002},"last":1.095},{"interface":"front","average":{"1min":1.528,"5min":1.3919999999999999,"15min":1.3700000000000001},"min":{"1min":1.0169999999999999,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":2.2799999999999998,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.49}]}]}]}} 2021-02-10T04:27:56.323 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2021-02-10T04:27:56.323 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd dump 2021-02-10T04:27:56.553 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:27:56.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:56 smithi164 conmon[29288]: cluster 2021-02-10T04:27:55.385089+0000 osd.5 (osd.5) 24 : cluster 2021-02-10T04:27:56.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:56 smithi164 conmon[29288]: [DBG] 2.3 scrub ok 2021-02-10T04:27:56.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:56 smithi164 conmon[29288]: audit 2021-02-10T04:27:55.619126+0000 mgr.y (mgr.14140) 305 : audit [DBG] from='client.14733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:27:56.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:56 smithi164 conmon[29288]: cluster 2021-02-10T04:27:55.982223 2021-02-10T04:27:56.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:56 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 306 : 2021-02-10T04:27:56.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:56 smithi164 conmon[29288]: cluster [DBG] pgmap v263: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:56.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:56 smithi085 conmon[29319]: cluster 2021-02-10T04:27:55.385089+0000 osd.5 ( 2021-02-10T04:27:56.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:56 smithi085 conmon[29319]: osd.5) 24 : cluster [DBG] 2.3 scrub ok 2021-02-10T04:27:56.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:56 smithi085 conmon[29319]: audit 2021-02-10T04:27:55.619126+0000 mgr.y (mgr.14140) 305 : audit [DBG] from='client.14733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:27:56.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:56 smithi085 conmon[29319]: cluster 2021-02-10T04:27:55.982223+0000 mgr.y (mgr.14140) 306 : cluster [DBG] pgmap v263: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:56.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:56 smithi085 conmon[35961]: cluster 2021-02-10T04:27:55 2021-02-10T04:27:56.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:56 smithi085 conmon[35961]: .385089+0000 osd.5 (osd.5) 24 : cluster [DBG] 2.3 scrub ok 2021-02-10T04:27:56.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:56 smithi085 conmon[35961]: audit 2021-02-10T04:27:56.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:56 smithi085 conmon[35961]: 2021-02-10T04:27:55.619126+0000 mgr.y (mgr.14140) 305 2021-02-10T04:27:56.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:56 smithi085 conmon[35961]: : audit [DBG] from='client.14733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:27:56.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:56 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:27:56.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:56 smithi085 conmon[35961]: T04:27:55.982223+0000 mgr.y (mgr. 2021-02-10T04:27:56.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:56 smithi085 conmon[35961]: 14140) 306 : cluster [DBG] pgmap v263: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:27:57.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:57 smithi085 conmon[29319]: cluster 2021-02-10T04:27:56.492750+0000 osd.2 (osd.2) 22 : cluster [DBG] 5.a scrub ok 2021-02-10T04:27:57.914 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:57 smithi085 conmon[35961]: cluster 2021-02-10T04:27:56.492750+0000 osd.2 ( 2021-02-10T04:27:57.914 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:57 smithi085 conmon[35961]: osd.2) 22 : cluster [DBG] 5.a scrub ok 2021-02-10T04:27:57.915 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:27:57 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:27:57.915 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:57 smithi164 conmon[29288]: cluster 2021-02-10T04:27:56.492750+0000 osd.2 (osd.2) 22 : cluster [DBG] 5.a scrub ok 2021-02-10T04:27:57.915 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:57 smithi164 conmon[29288]: 2021-02-10T04:27:58.291 INFO:teuthology.orchestra.run.smithi085.stdout:epoch 84 2021-02-10T04:27:58.291 INFO:teuthology.orchestra.run.smithi085.stdout:fsid 42224fce-6b57-11eb-8feb-001a4aab830c 2021-02-10T04:27:58.291 INFO:teuthology.orchestra.run.smithi085.stdout:created 2021-02-10T04:20:57.933279+0000 2021-02-10T04:27:58.291 INFO:teuthology.orchestra.run.smithi085.stdout:modified 2021-02-10T04:27:47.112630+0000 2021-02-10T04:27:58.292 INFO:teuthology.orchestra.run.smithi085.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2021-02-10T04:27:58.292 INFO:teuthology.orchestra.run.smithi085.stdout:crush_version 18 2021-02-10T04:27:58.292 INFO:teuthology.orchestra.run.smithi085.stdout:full_ratio 0.95 2021-02-10T04:27:58.292 INFO:teuthology.orchestra.run.smithi085.stdout:backfillfull_ratio 0.9 2021-02-10T04:27:58.292 INFO:teuthology.orchestra.run.smithi085.stdout:nearfull_ratio 0.85 2021-02-10T04:27:58.292 INFO:teuthology.orchestra.run.smithi085.stdout:require_min_compat_client luminous 2021-02-10T04:27:58.292 INFO:teuthology.orchestra.run.smithi085.stdout:min_compat_client luminous 2021-02-10T04:27:58.292 INFO:teuthology.orchestra.run.smithi085.stdout:require_osd_release quincy 2021-02-10T04:27:58.292 INFO:teuthology.orchestra.run.smithi085.stdout:stretch_mode_enabled false 2021-02-10T04:27:58.292 INFO:teuthology.orchestra.run.smithi085.stdout:pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_min 1 application mgr_devicehealth 2021-02-10T04:27:58.292 INFO:teuthology.orchestra.run.smithi085.stdout:pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 52 flags hashpspool stripe_width 0 application rgw 2021-02-10T04:27:58.292 INFO:teuthology.orchestra.run.smithi085.stdout:pool 3 'iscsi' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 58 flags hashpspool stripe_width 0 application rbd 2021-02-10T04:27:58.293 INFO:teuthology.orchestra.run.smithi085.stdout:pool 4 'zone.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 57 flags hashpspool stripe_width 0 application rgw 2021-02-10T04:27:58.293 INFO:teuthology.orchestra.run.smithi085.stdout:pool 5 'zone.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 59 flags hashpspool stripe_width 0 application rgw 2021-02-10T04:27:58.293 INFO:teuthology.orchestra.run.smithi085.stdout:pool 6 'zone.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 62 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 8 application rgw 2021-02-10T04:27:58.293 INFO:teuthology.orchestra.run.smithi085.stdout:max_osd 8 2021-02-10T04:27:58.293 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0 up in weight 1 up_from 8 up_thru 59 down_at 0 last_clean_interval [0,0) v2:172.21.15.85:6801/7 v2:172.21.15.85:6802/7 exists,up 45d0bb5f-35fa-4596-964f-75abcee53bee 2021-02-10T04:27:58.293 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1 up in weight 1 up_from 13 up_thru 71 down_at 0 last_clean_interval [0,0) v2:172.21.15.85:6805/7 v2:172.21.15.85:6806/7 exists,up 3421a5ce-872e-4b52-adf7-6aaa595871a4 2021-02-10T04:27:58.293 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2 up in weight 1 up_from 18 up_thru 59 down_at 0 last_clean_interval [0,0) v2:172.21.15.85:6809/7 v2:172.21.15.85:6810/7 exists,up 96e1952a-1042-4316-9fc9-86813e88b912 2021-02-10T04:27:58.293 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3 up in weight 1 up_from 25 up_thru 71 down_at 0 last_clean_interval [0,0) v2:172.21.15.85:6813/7 v2:172.21.15.85:6814/7 exists,up e019f49f-5dd4-4104-9ced-94c710616642 2021-02-10T04:27:58.293 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4 up in weight 1 up_from 31 up_thru 59 down_at 0 last_clean_interval [0,0) v2:172.21.15.164:6800/7 v2:172.21.15.164:6801/7 exists,up 91c8a80a-37b6-4c30-a32a-647b2e6d801f 2021-02-10T04:27:58.293 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5 up in weight 1 up_from 37 up_thru 59 down_at 0 last_clean_interval [0,0) v2:172.21.15.164:6804/7 v2:172.21.15.164:6805/7 exists,up cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e 2021-02-10T04:27:58.293 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6 up in weight 1 up_from 43 up_thru 57 down_at 0 last_clean_interval [0,0) v2:172.21.15.164:6808/7 v2:172.21.15.164:6809/7 exists,up adc40430-3eeb-4003-a824-343021243077 2021-02-10T04:27:58.293 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7 up in weight 1 up_from 50 up_thru 59 down_at 0 last_clean_interval [0,0) v2:172.21.15.164:6812/7 v2:172.21.15.164:6813/7 exists,up a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37 2021-02-10T04:27:58.293 INFO:teuthology.orchestra.run.smithi085.stdout:pg_upmap_items 2.4 [7,2] 2021-02-10T04:27:58.294 INFO:teuthology.orchestra.run.smithi085.stdout:pg_upmap_items 4.10 [1,5] 2021-02-10T04:27:58.294 INFO:teuthology.orchestra.run.smithi085.stdout:pg_upmap_items 4.14 [1,2] 2021-02-10T04:27:58.294 INFO:teuthology.orchestra.run.smithi085.stdout:pg_upmap_items 4.17 [1,2] 2021-02-10T04:27:58.294 INFO:teuthology.orchestra.run.smithi085.stdout:blocklist 172.21.15.85:0/801393150 expires 2021-02-11T04:21:37.851607+0000 2021-02-10T04:27:58.294 INFO:teuthology.orchestra.run.smithi085.stdout:blocklist 172.21.15.85:0/3049671696 expires 2021-02-11T04:21:37.851607+0000 2021-02-10T04:27:58.294 INFO:teuthology.orchestra.run.smithi085.stdout:blocklist 172.21.15.85:0/4056505602 expires 2021-02-11T04:21:20.700478+0000 2021-02-10T04:27:58.294 INFO:teuthology.orchestra.run.smithi085.stdout:blocklist 172.21.15.85:6800/7 expires 2021-02-11T04:21:20.700478+0000 2021-02-10T04:27:58.294 INFO:teuthology.orchestra.run.smithi085.stdout:blocklist 172.21.15.85:0/1879026062 expires 2021-02-11T04:21:20.700478+0000 2021-02-10T04:27:59.004 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 59 down_at 0 last_clean_interval [0,0) v2:172.21.15.85:6801/7 v2:172.21.15.85:6802/7 exists,up 45d0bb5f-35fa-4596-964f-75abcee53bee', 'osd.1 up in weight 1 up_from 13 up_thru 71 down_at 0 last_clean_interval [0,0) v2:172.21.15.85:6805/7 v2:172.21.15.85:6806/7 exists,up 3421a5ce-872e-4b52-adf7-6aaa595871a4', 'osd.2 up in weight 1 up_from 18 up_thru 59 down_at 0 last_clean_interval [0,0) v2:172.21.15.85:6809/7 v2:172.21.15.85:6810/7 exists,up 96e1952a-1042-4316-9fc9-86813e88b912', 'osd.3 up in weight 1 up_from 25 up_thru 71 down_at 0 last_clean_interval [0,0) v2:172.21.15.85:6813/7 v2:172.21.15.85:6814/7 exists,up e019f49f-5dd4-4104-9ced-94c710616642', 'osd.4 up in weight 1 up_from 31 up_thru 59 down_at 0 last_clean_interval [0,0) v2:172.21.15.164:6800/7 v2:172.21.15.164:6801/7 exists,up 91c8a80a-37b6-4c30-a32a-647b2e6d801f', 'osd.5 up in weight 1 up_from 37 up_thru 59 down_at 0 last_clean_interval [0,0) v2:172.21.15.164:6804/7 v2:172.21.15.164:6805/7 exists,up cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e', 'osd.6 up in weight 1 up_from 43 up_thru 57 down_at 0 last_clean_interval [0,0) v2:172.21.15.164:6808/7 v2:172.21.15.164:6809/7 exists,up adc40430-3eeb-4003-a824-343021243077', 'osd.7 up in weight 1 up_from 50 up_thru 59 down_at 0 last_clean_interval [0,0) v2:172.21.15.164:6812/7 v2:172.21.15.164:6813/7 exists,up a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37'] 2021-02-10T04:27:59.005 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon mon.a config show 2021-02-10T04:27:59.212 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:27:59.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:59 smithi164 conmon[29288]: audit 2021-02-10T04:27:57.633437+0000 mgr.y (mgr 2021-02-10T04:27:59.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:59 smithi164 conmon[29288]: .14140) 307 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:27:59.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:59 smithi164 conmon[29288]: cluster 2021-02-10T04:27:57.982641+0000 mgr.y (mgr.14140) 308 : cluster [DBG] pgmap v264: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 942 B/s rd, 0 op/s 2021-02-10T04:27:59.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:59 smithi164 conmon[29288]: 2021-02-10T04:27:59.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:59 smithi164 conmon[29288]: cluster 2021-02-10T04:27:58.282703+0000 osd.5 (osd 2021-02-10T04:27:59.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:59 smithi164 conmon[29288]: .5) 25 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:27:59.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:59 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:27:59.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:59 smithi164 conmon[29288]: :27:58.282891+0000 osd.5 (osd.5) 26 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:27:59.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:59 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:27:59.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:59 smithi164 conmon[29288]: 10T04:27:58.291328+0000 mon.c (mon.1 2021-02-10T04:27:59.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:59 smithi164 conmon[29288]: ) 28 : audit [DBG] from='client.? 172.21.15.85:0/1989195275' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2021-02-10T04:27:59.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:59 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:27:59.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:59 smithi164 conmon[29288]: 02-10T04:27:58.363244+0000 osd.3 ( 2021-02-10T04:27:59.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:59 smithi164 conmon[29288]: osd.3) 30 : cluster [DBG] 4.1d scrub ok 2021-02-10T04:27:59.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:27:59 smithi164 conmon[29288]: 2021-02-10T04:27:59.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:59 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:27:59.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:59 smithi085 conmon[29319]: :27:57.633437+0000 mgr.y (mgr.14140) 307 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:27:59.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:59 smithi085 conmon[29319]: cluster 2021-02-10T04:27:57.982641+0000 mgr.y (mgr.14140) 308 : cluster [DBG] pgmap v264: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 942 B/s rd, 0 op/s 2021-02-10T04:27:59.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:59 smithi085 conmon[29319]: cluster 2021-02-10T04:27:58.282703+0000 osd.5 (osd.5) 25 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:27:59.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:59 smithi085 conmon[29319]: cluster 2021-02-10T04:27:58.282891+0000 osd.5 (osd.5) 26 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:27:59.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:59 smithi085 conmon[29319]: audit 2021-02-10T04:27:58.291328+0000 mon.c (mon.1) 28 : audit [DBG] from='client.? 172.21.15.85:0/1989195275' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2021-02-10T04:27:59.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:27:59 smithi085 conmon[29319]: cluster 2021-02-10T04:27:58.363244+0000 osd.3 (osd.3) 30 : cluster [DBG] 4.1d scrub ok 2021-02-10T04:27:59.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:59 smithi085 conmon[35961]: audit 2021-02-10T04:27:57.633437+0000 mgr.y 2021-02-10T04:27:59.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:59 smithi085 conmon[35961]: (mgr.14140) 307 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:27:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:59 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:27:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:59 smithi085 conmon[35961]: :27:57.982641+0000 mgr.y (mgr.14140) 308 : cluster [DBG] pgmap v264: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 942 B/s rd, 0 op/s 2021-02-10T04:27:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:59 smithi085 conmon[35961]: cluster 2021-02-10T04:27:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:59 smithi085 conmon[35961]: 2021-02-10T04:27:58.282703+0000 osd.5 2021-02-10T04:27:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:59 smithi085 conmon[35961]: (osd.5) 25 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:27:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:59 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:27:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:59 smithi085 conmon[35961]: 27:58.282891+0000 osd.5 (osd.5) 26 : cluster [DBG] 2021-02-10T04:27:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:59 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:27:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:59 smithi085 conmon[35961]: audit 2021 2021-02-10T04:27:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:59 smithi085 conmon[35961]: -02-10T04:27:58.291328+0000 mon.c (mon.1) 28 : audit 2021-02-10T04:27:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:59 smithi085 conmon[35961]: [DBG] from='client.? 172.21.15.85:0/1989195275' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2021-02-10T04:27:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:59 smithi085 conmon[35961]: cluster 2021-02-10T04:27:59.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:59 smithi085 conmon[35961]: 2021-02-10T04:27:58.363244+0000 osd.3 (osd.3) 30 2021-02-10T04:27:59.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:27:59 smithi085 conmon[35961]: : cluster [DBG] 4.1d scrub ok 2021-02-10T04:28:00.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:00 smithi085 conmon[29319]: cluster 2021-02-10T04:27:59.016115+0000 2021-02-10T04:28:00.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:00 smithi085 conmon[29319]: mon.a (mon.0) 704 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2021-02-10T04:28:00.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:00 smithi085 conmon[29319]: cluster 2021-02-10T04:27:59.739079+0000 osd.0 (osd.0) 29 : cluster [DBG] 4.1e scrub ok 2021-02-10T04:28:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:00 smithi085 conmon[35961]: cluster 2021-02-10T04:27:59.016115 2021-02-10T04:28:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:00 smithi085 conmon[35961]: +0000 mon.a (mon.0) 704 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2021-02-10T04:28:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:00 smithi085 conmon[35961]: cluster 2021-02-10T04:27:59.739079+0000 osd.0 (osd.0) 29 : cluster [DBG] 4.1e scrub ok 2021-02-10T04:28:00.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:00 smithi164 conmon[29288]: cluster 2021-02-10T04:27:59.016115+0000 mon.a (mon.0) 704 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2021-02-10T04:28:00.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:00 smithi164 conmon[29288]: cluster 2021-02-10T04:27:59.739079+0000 osd.0 (osd 2021-02-10T04:28:00.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:00 smithi164 conmon[29288]: .0) 29 : cluster [DBG] 4.1e scrub ok 2021-02-10T04:28:00.818 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:28:00.818 INFO:teuthology.orchestra.run.smithi085.stdout: "name": "mon.a", 2021-02-10T04:28:00.818 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster": "ceph", 2021-02-10T04:28:00.818 INFO:teuthology.orchestra.run.smithi085.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2021-02-10T04:28:00.818 INFO:teuthology.orchestra.run.smithi085.stdout: "admin_socket_mode": "", 2021-02-10T04:28:00.818 INFO:teuthology.orchestra.run.smithi085.stdout: "auth_client_required": "cephx, none", 2021-02-10T04:28:00.818 INFO:teuthology.orchestra.run.smithi085.stdout: "auth_cluster_required": "cephx", 2021-02-10T04:28:00.819 INFO:teuthology.orchestra.run.smithi085.stdout: "auth_debug": "true", 2021-02-10T04:28:00.819 INFO:teuthology.orchestra.run.smithi085.stdout: "auth_mon_ticket_ttl": "43200.000000", 2021-02-10T04:28:00.819 INFO:teuthology.orchestra.run.smithi085.stdout: "auth_service_required": "cephx", 2021-02-10T04:28:00.819 INFO:teuthology.orchestra.run.smithi085.stdout: "auth_service_ticket_ttl": "3600.000000", 2021-02-10T04:28:00.819 INFO:teuthology.orchestra.run.smithi085.stdout: "auth_supported": "", 2021-02-10T04:28:00.819 INFO:teuthology.orchestra.run.smithi085.stdout: "bdev_aio": "true", 2021-02-10T04:28:00.819 INFO:teuthology.orchestra.run.smithi085.stdout: "bdev_aio_max_queue_depth": "1024", 2021-02-10T04:28:00.819 INFO:teuthology.orchestra.run.smithi085.stdout: "bdev_aio_poll_ms": "250", 2021-02-10T04:28:00.819 INFO:teuthology.orchestra.run.smithi085.stdout: "bdev_aio_reap_max": "16", 2021-02-10T04:28:00.819 INFO:teuthology.orchestra.run.smithi085.stdout: "bdev_async_discard": "false", 2021-02-10T04:28:00.819 INFO:teuthology.orchestra.run.smithi085.stdout: "bdev_block_size": "4096", 2021-02-10T04:28:00.819 INFO:teuthology.orchestra.run.smithi085.stdout: "bdev_debug_aio": "false", 2021-02-10T04:28:00.819 INFO:teuthology.orchestra.run.smithi085.stdout: "bdev_debug_aio_log_age": "5.000000", 2021-02-10T04:28:00.820 INFO:teuthology.orchestra.run.smithi085.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2021-02-10T04:28:00.820 INFO:teuthology.orchestra.run.smithi085.stdout: "bdev_debug_inflight_ios": "false", 2021-02-10T04:28:00.820 INFO:teuthology.orchestra.run.smithi085.stdout: "bdev_enable_discard": "false", 2021-02-10T04:28:00.820 INFO:teuthology.orchestra.run.smithi085.stdout: "bdev_flock_retry": "3", 2021-02-10T04:28:00.820 INFO:teuthology.orchestra.run.smithi085.stdout: "bdev_flock_retry_interval": "0.100000", 2021-02-10T04:28:00.820 INFO:teuthology.orchestra.run.smithi085.stdout: "bdev_inject_crash": "0", 2021-02-10T04:28:00.820 INFO:teuthology.orchestra.run.smithi085.stdout: "bdev_inject_crash_flush_delay": "2", 2021-02-10T04:28:00.820 INFO:teuthology.orchestra.run.smithi085.stdout: "bdev_ioring": "false", 2021-02-10T04:28:00.820 INFO:teuthology.orchestra.run.smithi085.stdout: "bdev_ioring_hipri": "false", 2021-02-10T04:28:00.820 INFO:teuthology.orchestra.run.smithi085.stdout: "bdev_ioring_sqthread_poll": "false", 2021-02-10T04:28:00.820 INFO:teuthology.orchestra.run.smithi085.stdout: "bdev_nvme_unbind_from_kernel": "false", 2021-02-10T04:28:00.820 INFO:teuthology.orchestra.run.smithi085.stdout: "bdev_type": "", 2021-02-10T04:28:00.820 INFO:teuthology.orchestra.run.smithi085.stdout: "bluefs_alloc_size": "1048576", 2021-02-10T04:28:00.821 INFO:teuthology.orchestra.run.smithi085.stdout: "bluefs_allocator": "hybrid", 2021-02-10T04:28:00.821 INFO:teuthology.orchestra.run.smithi085.stdout: "bluefs_buffered_io": "false", 2021-02-10T04:28:00.821 INFO:teuthology.orchestra.run.smithi085.stdout: "bluefs_check_for_zeros": "false", 2021-02-10T04:28:00.821 INFO:teuthology.orchestra.run.smithi085.stdout: "bluefs_compact_log_sync": "false", 2021-02-10T04:28:00.821 INFO:teuthology.orchestra.run.smithi085.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2021-02-10T04:28:00.821 INFO:teuthology.orchestra.run.smithi085.stdout: "bluefs_log_compact_min_size": "16777216", 2021-02-10T04:28:00.821 INFO:teuthology.orchestra.run.smithi085.stdout: "bluefs_log_replay_check_allocations": "true", 2021-02-10T04:28:00.821 INFO:teuthology.orchestra.run.smithi085.stdout: "bluefs_max_log_runway": "4194304", 2021-02-10T04:28:00.821 INFO:teuthology.orchestra.run.smithi085.stdout: "bluefs_max_prefetch": "1048576", 2021-02-10T04:28:00.821 INFO:teuthology.orchestra.run.smithi085.stdout: "bluefs_min_flush_size": "524288", 2021-02-10T04:28:00.821 INFO:teuthology.orchestra.run.smithi085.stdout: "bluefs_min_log_runway": "1048576", 2021-02-10T04:28:00.821 INFO:teuthology.orchestra.run.smithi085.stdout: "bluefs_replay_recovery": "false", 2021-02-10T04:28:00.821 INFO:teuthology.orchestra.run.smithi085.stdout: "bluefs_replay_recovery_disable_compact": "false", 2021-02-10T04:28:00.822 INFO:teuthology.orchestra.run.smithi085.stdout: "bluefs_shared_alloc_size": "65536", 2021-02-10T04:28:00.822 INFO:teuthology.orchestra.run.smithi085.stdout: "bluefs_sync_write": "false", 2021-02-10T04:28:00.822 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2021-02-10T04:28:00.822 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2021-02-10T04:28:00.822 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2021-02-10T04:28:00.822 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_allocator": "hybrid", 2021-02-10T04:28:00.822 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2021-02-10T04:28:00.822 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2021-02-10T04:28:00.822 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2021-02-10T04:28:00.822 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_bitmapallocator_span_size": "1024", 2021-02-10T04:28:00.822 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_blobid_prealloc": "10240", 2021-02-10T04:28:00.822 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_block_create": "true", 2021-02-10T04:28:00.822 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_block_db_create": "false", 2021-02-10T04:28:00.823 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_block_db_path": "", 2021-02-10T04:28:00.823 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_block_db_size": "0", 2021-02-10T04:28:00.823 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_block_path": "", 2021-02-10T04:28:00.823 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_block_preallocate_file": "false", 2021-02-10T04:28:00.823 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_block_size": "107374182400", 2021-02-10T04:28:00.823 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_block_wal_create": "false", 2021-02-10T04:28:00.823 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_block_wal_path": "", 2021-02-10T04:28:00.823 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_block_wal_size": "100663296", 2021-02-10T04:28:00.823 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_bluefs": "true", 2021-02-10T04:28:00.823 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2021-02-10T04:28:00.823 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_bluefs_env_mirror": "false", 2021-02-10T04:28:00.823 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_cache_autotune": "true", 2021-02-10T04:28:00.823 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_cache_autotune_interval": "5.000000", 2021-02-10T04:28:00.824 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2021-02-10T04:28:00.824 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_cache_kv_ratio": "0.450000", 2021-02-10T04:28:00.824 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_cache_meta_ratio": "0.450000", 2021-02-10T04:28:00.824 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_cache_size": "0", 2021-02-10T04:28:00.824 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_cache_size_hdd": "1073741824", 2021-02-10T04:28:00.824 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_cache_size_ssd": "3221225472", 2021-02-10T04:28:00.824 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_cache_trim_interval": "0.050000", 2021-02-10T04:28:00.824 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_cache_trim_max_skip_pinned": "64", 2021-02-10T04:28:00.824 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_cache_type": "2q", 2021-02-10T04:28:00.824 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_clone_cow": "true", 2021-02-10T04:28:00.824 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_compression_algorithm": "snappy", 2021-02-10T04:28:00.824 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_compression_max_blob_size": "0", 2021-02-10T04:28:00.824 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2021-02-10T04:28:00.825 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2021-02-10T04:28:00.825 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_compression_min_blob_size": "0", 2021-02-10T04:28:00.825 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2021-02-10T04:28:00.825 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_compression_min_blob_size_ssd": "8192", 2021-02-10T04:28:00.825 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_compression_mode": "none", 2021-02-10T04:28:00.825 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_compression_required_ratio": "0.875000", 2021-02-10T04:28:00.825 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_csum_type": "crc32c", 2021-02-10T04:28:00.826 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_debug_enforce_settings": "default", 2021-02-10T04:28:00.826 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_debug_freelist": "false", 2021-02-10T04:28:00.826 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_debug_fsck_abort": "false", 2021-02-10T04:28:00.826 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_debug_inject_bug21040": "false", 2021-02-10T04:28:00.826 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2021-02-10T04:28:00.826 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_debug_inject_read_err": "false", 2021-02-10T04:28:00.826 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_debug_no_reuse_blocks": "false", 2021-02-10T04:28:00.826 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_debug_omit_block_device_write": "false", 2021-02-10T04:28:00.826 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_debug_omit_kv_commit": "false", 2021-02-10T04:28:00.826 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2021-02-10T04:28:00.826 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_debug_prefill": "0.000000", 2021-02-10T04:28:00.826 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_debug_prefragment_max": "1048576", 2021-02-10T04:28:00.827 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_debug_random_read_err": "0.000000", 2021-02-10T04:28:00.827 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2021-02-10T04:28:00.827 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_debug_small_allocations": "0", 2021-02-10T04:28:00.827 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2021-02-10T04:28:00.827 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_default_buffered_read": "true", 2021-02-10T04:28:00.827 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_default_buffered_write": "false", 2021-02-10T04:28:00.827 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_deferred_batch_ops": "0", 2021-02-10T04:28:00.827 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2021-02-10T04:28:00.827 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2021-02-10T04:28:00.827 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2021-02-10T04:28:00.827 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_extent_map_shard_max_size": "1200", 2021-02-10T04:28:00.827 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_extent_map_shard_min_size": "150", 2021-02-10T04:28:00.827 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_extent_map_shard_target_size": "500", 2021-02-10T04:28:00.827 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2021-02-10T04:28:00.828 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_freelist_blocks_per_key": "128", 2021-02-10T04:28:00.828 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2021-02-10T04:28:00.828 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2021-02-10T04:28:00.828 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_fsck_on_mkfs": "true", 2021-02-10T04:28:00.828 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2021-02-10T04:28:00.828 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_fsck_on_mount": "false", 2021-02-10T04:28:00.828 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_fsck_on_mount_deep": "false", 2021-02-10T04:28:00.828 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_fsck_on_umount": "false", 2021-02-10T04:28:00.828 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_fsck_on_umount_deep": "false", 2021-02-10T04:28:00.828 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_fsck_quick_fix_on_mount": "true", 2021-02-10T04:28:00.828 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_fsck_quick_fix_threads": "2", 2021-02-10T04:28:00.828 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2021-02-10T04:28:00.829 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_gc_enable_blob_threshold": "0", 2021-02-10T04:28:00.829 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_gc_enable_total_threshold": "0", 2021-02-10T04:28:00.829 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2021-02-10T04:28:00.829 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_ignore_data_csum": "false", 2021-02-10T04:28:00.829 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2021-02-10T04:28:00.829 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_kvbackend": "rocksdb", 2021-02-10T04:28:00.829 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_log_collection_list_age": "60.000000", 2021-02-10T04:28:00.829 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2021-02-10T04:28:00.829 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_log_op_age": "5.000000", 2021-02-10T04:28:00.829 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_max_alloc_size": "0", 2021-02-10T04:28:00.829 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_max_blob_size": "0", 2021-02-10T04:28:00.829 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_max_blob_size_hdd": "65536", 2021-02-10T04:28:00.829 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_max_blob_size_ssd": "65536", 2021-02-10T04:28:00.830 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_max_defer_interval": "3.000000", 2021-02-10T04:28:00.830 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_max_deferred_txc": "32", 2021-02-10T04:28:00.830 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_min_alloc_size": "0", 2021-02-10T04:28:00.830 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_min_alloc_size_hdd": "4096", 2021-02-10T04:28:00.830 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_min_alloc_size_ssd": "4096", 2021-02-10T04:28:00.830 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_nid_prealloc": "1024", 2021-02-10T04:28:00.830 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_prefer_deferred_size": "0", 2021-02-10T04:28:00.830 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2021-02-10T04:28:00.830 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2021-02-10T04:28:00.830 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_retry_disk_reads": "3", 2021-02-10T04:28:00.830 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_rocksdb_cf": "true", 2021-02-10T04:28:00.830 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P", 2021-02-10T04:28:00.831 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_rocksdb_options": "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824", 2021-02-10T04:28:00.831 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_rocksdb_options_annex": "", 2021-02-10T04:28:00.831 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_spdk_coremask": "0x1", 2021-02-10T04:28:00.831 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_spdk_io_sleep": "5", 2021-02-10T04:28:00.831 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_spdk_max_io_completion": "0", 2021-02-10T04:28:00.831 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_spdk_mem": "512", 2021-02-10T04:28:00.831 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_sync_submit_transaction": "false", 2021-02-10T04:28:00.831 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_throttle_bytes": "67108864", 2021-02-10T04:28:00.831 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_throttle_cost_per_io": "0", 2021-02-10T04:28:00.831 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2021-02-10T04:28:00.831 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2021-02-10T04:28:00.831 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2021-02-10T04:28:00.831 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_throttle_trace_rate": "0.000000", 2021-02-10T04:28:00.831 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_tracing": "false", 2021-02-10T04:28:00.832 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2021-02-10T04:28:00.832 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_volume_selection_reserved": "0", 2021-02-10T04:28:00.832 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2021-02-10T04:28:00.832 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2021-02-10T04:28:00.832 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_warn_on_legacy_statfs": "true", 2021-02-10T04:28:00.832 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2021-02-10T04:28:00.832 INFO:teuthology.orchestra.run.smithi085.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2021-02-10T04:28:00.832 INFO:teuthology.orchestra.run.smithi085.stdout: "cephadm_path": "/usr/sbin/cephadm", 2021-02-10T04:28:00.832 INFO:teuthology.orchestra.run.smithi085.stdout: "cephfs_mirror_action_update_interval": "2", 2021-02-10T04:28:00.832 INFO:teuthology.orchestra.run.smithi085.stdout: "cephfs_mirror_directory_scan_interval": "10", 2021-02-10T04:28:00.832 INFO:teuthology.orchestra.run.smithi085.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2021-02-10T04:28:00.832 INFO:teuthology.orchestra.run.smithi085.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2021-02-10T04:28:00.833 INFO:teuthology.orchestra.run.smithi085.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2021-02-10T04:28:00.833 INFO:teuthology.orchestra.run.smithi085.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2021-02-10T04:28:00.833 INFO:teuthology.orchestra.run.smithi085.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2021-02-10T04:28:00.833 INFO:teuthology.orchestra.run.smithi085.stdout: "cephx_cluster_require_signatures": "false", 2021-02-10T04:28:00.833 INFO:teuthology.orchestra.run.smithi085.stdout: "cephx_cluster_require_version": "1", 2021-02-10T04:28:00.833 INFO:teuthology.orchestra.run.smithi085.stdout: "cephx_require_signatures": "false", 2021-02-10T04:28:00.833 INFO:teuthology.orchestra.run.smithi085.stdout: "cephx_require_version": "1", 2021-02-10T04:28:00.833 INFO:teuthology.orchestra.run.smithi085.stdout: "cephx_service_require_signatures": "false", 2021-02-10T04:28:00.833 INFO:teuthology.orchestra.run.smithi085.stdout: "cephx_service_require_version": "1", 2021-02-10T04:28:00.833 INFO:teuthology.orchestra.run.smithi085.stdout: "cephx_sign_messages": "true", 2021-02-10T04:28:00.833 INFO:teuthology.orchestra.run.smithi085.stdout: "chdir": "", 2021-02-10T04:28:00.833 INFO:teuthology.orchestra.run.smithi085.stdout: "client_acl_type": "", 2021-02-10T04:28:00.833 INFO:teuthology.orchestra.run.smithi085.stdout: "client_asio_thread_count": "2", 2021-02-10T04:28:00.834 INFO:teuthology.orchestra.run.smithi085.stdout: "client_cache_mid": "0.750000", 2021-02-10T04:28:00.834 INFO:teuthology.orchestra.run.smithi085.stdout: "client_cache_size": "16384", 2021-02-10T04:28:00.834 INFO:teuthology.orchestra.run.smithi085.stdout: "client_caps_release_delay": "5", 2021-02-10T04:28:00.834 INFO:teuthology.orchestra.run.smithi085.stdout: "client_check_pool_perm": "true", 2021-02-10T04:28:00.834 INFO:teuthology.orchestra.run.smithi085.stdout: "client_debug_force_sync_read": "false", 2021-02-10T04:28:00.834 INFO:teuthology.orchestra.run.smithi085.stdout: "client_debug_getattr_caps": "false", 2021-02-10T04:28:00.834 INFO:teuthology.orchestra.run.smithi085.stdout: "client_debug_inject_tick_delay": "0", 2021-02-10T04:28:00.834 INFO:teuthology.orchestra.run.smithi085.stdout: "client_die_on_failed_dentry_invalidate": "true", 2021-02-10T04:28:00.834 INFO:teuthology.orchestra.run.smithi085.stdout: "client_die_on_failed_remount": "false", 2021-02-10T04:28:00.834 INFO:teuthology.orchestra.run.smithi085.stdout: "client_dirsize_rbytes": "true", 2021-02-10T04:28:00.834 INFO:teuthology.orchestra.run.smithi085.stdout: "client_force_lazyio": "false", 2021-02-10T04:28:00.834 INFO:teuthology.orchestra.run.smithi085.stdout: "client_fs": "", 2021-02-10T04:28:00.835 INFO:teuthology.orchestra.run.smithi085.stdout: "client_inject_fixed_oldest_tid": "false", 2021-02-10T04:28:00.835 INFO:teuthology.orchestra.run.smithi085.stdout: "client_inject_release_failure": "false", 2021-02-10T04:28:00.835 INFO:teuthology.orchestra.run.smithi085.stdout: "client_max_inline_size": "4096", 2021-02-10T04:28:00.835 INFO:teuthology.orchestra.run.smithi085.stdout: "client_mds_namespace": "", 2021-02-10T04:28:00.835 INFO:teuthology.orchestra.run.smithi085.stdout: "client_metadata": "", 2021-02-10T04:28:00.835 INFO:teuthology.orchestra.run.smithi085.stdout: "client_mount_gid": "-1", 2021-02-10T04:28:00.835 INFO:teuthology.orchestra.run.smithi085.stdout: "client_mount_timeout": "300.000000", 2021-02-10T04:28:00.835 INFO:teuthology.orchestra.run.smithi085.stdout: "client_mount_uid": "-1", 2021-02-10T04:28:00.835 INFO:teuthology.orchestra.run.smithi085.stdout: "client_mountpoint": "/", 2021-02-10T04:28:00.835 INFO:teuthology.orchestra.run.smithi085.stdout: "client_notify_timeout": "10", 2021-02-10T04:28:00.835 INFO:teuthology.orchestra.run.smithi085.stdout: "client_oc": "true", 2021-02-10T04:28:00.835 INFO:teuthology.orchestra.run.smithi085.stdout: "client_oc_max_dirty": "104857600", 2021-02-10T04:28:00.836 INFO:teuthology.orchestra.run.smithi085.stdout: "client_oc_max_dirty_age": "5.000000", 2021-02-10T04:28:00.836 INFO:teuthology.orchestra.run.smithi085.stdout: "client_oc_max_objects": "1000", 2021-02-10T04:28:00.836 INFO:teuthology.orchestra.run.smithi085.stdout: "client_oc_size": "209715200", 2021-02-10T04:28:00.836 INFO:teuthology.orchestra.run.smithi085.stdout: "client_oc_target_dirty": "8388608", 2021-02-10T04:28:00.836 INFO:teuthology.orchestra.run.smithi085.stdout: "client_permissions": "true", 2021-02-10T04:28:00.836 INFO:teuthology.orchestra.run.smithi085.stdout: "client_quota_df": "true", 2021-02-10T04:28:00.836 INFO:teuthology.orchestra.run.smithi085.stdout: "client_readahead_max_bytes": "0", 2021-02-10T04:28:00.836 INFO:teuthology.orchestra.run.smithi085.stdout: "client_readahead_max_periods": "4", 2021-02-10T04:28:00.836 INFO:teuthology.orchestra.run.smithi085.stdout: "client_readahead_min": "131072", 2021-02-10T04:28:00.836 INFO:teuthology.orchestra.run.smithi085.stdout: "client_reconnect_stale": "false", 2021-02-10T04:28:00.836 INFO:teuthology.orchestra.run.smithi085.stdout: "client_shutdown_timeout": "30", 2021-02-10T04:28:00.836 INFO:teuthology.orchestra.run.smithi085.stdout: "client_snapdir": ".snap", 2021-02-10T04:28:00.836 INFO:teuthology.orchestra.run.smithi085.stdout: "client_tick_interval": "1", 2021-02-10T04:28:00.837 INFO:teuthology.orchestra.run.smithi085.stdout: "client_trace": "", 2021-02-10T04:28:00.837 INFO:teuthology.orchestra.run.smithi085.stdout: "client_try_dentry_invalidate": "false", 2021-02-10T04:28:00.837 INFO:teuthology.orchestra.run.smithi085.stdout: "client_use_faked_inos": "false", 2021-02-10T04:28:00.837 INFO:teuthology.orchestra.run.smithi085.stdout: "client_use_random_mds": "false", 2021-02-10T04:28:00.837 INFO:teuthology.orchestra.run.smithi085.stdout: "clog_to_graylog": "false", 2021-02-10T04:28:00.837 INFO:teuthology.orchestra.run.smithi085.stdout: "clog_to_graylog_host": "127.0.0.1", 2021-02-10T04:28:00.837 INFO:teuthology.orchestra.run.smithi085.stdout: "clog_to_graylog_port": "12201", 2021-02-10T04:28:00.837 INFO:teuthology.orchestra.run.smithi085.stdout: "clog_to_monitors": "default=true", 2021-02-10T04:28:00.837 INFO:teuthology.orchestra.run.smithi085.stdout: "clog_to_syslog": "false", 2021-02-10T04:28:00.837 INFO:teuthology.orchestra.run.smithi085.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2021-02-10T04:28:00.837 INFO:teuthology.orchestra.run.smithi085.stdout: "clog_to_syslog_level": "info", 2021-02-10T04:28:00.837 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "-", 2021-02-10T04:28:00.837 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_network": "", 2021-02-10T04:28:00.838 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_network_interface": "", 2021-02-10T04:28:00.838 INFO:teuthology.orchestra.run.smithi085.stdout: "compressor_zlib_isal": "false", 2021-02-10T04:28:00.838 INFO:teuthology.orchestra.run.smithi085.stdout: "compressor_zlib_level": "5", 2021-02-10T04:28:00.838 INFO:teuthology.orchestra.run.smithi085.stdout: "compressor_zlib_winsize": "-15", 2021-02-10T04:28:00.838 INFO:teuthology.orchestra.run.smithi085.stdout: "compressor_zstd_level": "1", 2021-02-10T04:28:00.838 INFO:teuthology.orchestra.run.smithi085.stdout: "container_image": "quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8", 2021-02-10T04:28:00.838 INFO:teuthology.orchestra.run.smithi085.stdout: "crash_dir": "/var/lib/ceph/crash", 2021-02-10T04:28:00.838 INFO:teuthology.orchestra.run.smithi085.stdout: "crimson_alien_op_num_threads": "16", 2021-02-10T04:28:00.838 INFO:teuthology.orchestra.run.smithi085.stdout: "crimson_osd_obc_lru_size": "10", 2021-02-10T04:28:00.838 INFO:teuthology.orchestra.run.smithi085.stdout: "crimson_osd_scheduler_concurrency": "0", 2021-02-10T04:28:00.838 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_location": "", 2021-02-10T04:28:00.838 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_location_hook": "", 2021-02-10T04:28:00.839 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_location_hook_timeout": "10", 2021-02-10T04:28:00.839 INFO:teuthology.orchestra.run.smithi085.stdout: "daemonize": "false", 2021-02-10T04:28:00.839 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_allow_any_pool_priority": "false", 2021-02-10T04:28:00.839 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_asok": "1/5", 2021-02-10T04:28:00.839 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_asok_assert_abort": "false", 2021-02-10T04:28:00.839 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_asserts_on_shutdown": "true", 2021-02-10T04:28:00.839 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_auth": "1/5", 2021-02-10T04:28:00.839 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_bdev": "1/3", 2021-02-10T04:28:00.839 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_bluefs": "1/5", 2021-02-10T04:28:00.839 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_bluestore": "1/5", 2021-02-10T04:28:00.840 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_buffer": "0/1", 2021-02-10T04:28:00.840 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_cephfs_mirror": "0/5", 2021-02-10T04:28:00.840 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_civetweb": "1/10", 2021-02-10T04:28:00.840 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_client": "0/5", 2021-02-10T04:28:00.840 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_compressor": "1/5", 2021-02-10T04:28:00.840 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_context": "0/1", 2021-02-10T04:28:00.840 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_crush": "1/1", 2021-02-10T04:28:00.840 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_crypto": "1/5", 2021-02-10T04:28:00.840 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_deliberately_leak_memory": "false", 2021-02-10T04:28:00.840 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_disable_randomized_ping": "false", 2021-02-10T04:28:00.840 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_dpdk": "1/5", 2021-02-10T04:28:00.840 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_eventtrace": "1/5", 2021-02-10T04:28:00.841 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_filer": "0/1", 2021-02-10T04:28:00.841 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_filestore": "1/3", 2021-02-10T04:28:00.841 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_finisher": "1/1", 2021-02-10T04:28:00.841 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_fuse": "1/5", 2021-02-10T04:28:00.841 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_heartbeat_testing_span": "0", 2021-02-10T04:28:00.841 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_heartbeatmap": "1/5", 2021-02-10T04:28:00.841 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_immutable_obj_cache": "0/5", 2021-02-10T04:28:00.841 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_javaclient": "1/5", 2021-02-10T04:28:00.841 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_journal": "1/3", 2021-02-10T04:28:00.841 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_journaler": "0/5", 2021-02-10T04:28:00.842 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_kstore": "1/5", 2021-02-10T04:28:00.842 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_leveldb": "4/5", 2021-02-10T04:28:00.842 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_lockdep": "0/1", 2021-02-10T04:28:00.842 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_mds": "1/5", 2021-02-10T04:28:00.842 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_mds_balancer": "1/5", 2021-02-10T04:28:00.842 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_mds_locker": "1/5", 2021-02-10T04:28:00.842 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_mds_log": "1/5", 2021-02-10T04:28:00.842 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_mds_log_expire": "1/5", 2021-02-10T04:28:00.842 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_mds_migrator": "1/5", 2021-02-10T04:28:00.842 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_memdb": "4/5", 2021-02-10T04:28:00.843 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_mgr": "1/5", 2021-02-10T04:28:00.843 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_mgrc": "1/5", 2021-02-10T04:28:00.843 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_mon": "20/20", 2021-02-10T04:28:00.843 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_monc": "0/10", 2021-02-10T04:28:00.843 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_ms": "1/1", 2021-02-10T04:28:00.843 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_none": "0/5", 2021-02-10T04:28:00.843 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_objclass": "0/5", 2021-02-10T04:28:00.843 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_objectcacher": "0/5", 2021-02-10T04:28:00.843 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_objecter": "0/1", 2021-02-10T04:28:00.843 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_optracker": "0/5", 2021-02-10T04:28:00.843 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_osd": "1/5", 2021-02-10T04:28:00.843 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_paxos": "20/20", 2021-02-10T04:28:00.843 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_perfcounter": "1/5", 2021-02-10T04:28:00.844 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_prioritycache": "1/5", 2021-02-10T04:28:00.844 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_rados": "0/5", 2021-02-10T04:28:00.844 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_rbd": "0/5", 2021-02-10T04:28:00.844 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_rbd_mirror": "0/5", 2021-02-10T04:28:00.844 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_rbd_pwl": "0/5", 2021-02-10T04:28:00.844 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_rbd_replay": "0/5", 2021-02-10T04:28:00.844 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_refs": "0/0", 2021-02-10T04:28:00.844 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_reserver": "1/1", 2021-02-10T04:28:00.844 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_rgw": "1/5", 2021-02-10T04:28:00.844 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_rgw_sync": "1/5", 2021-02-10T04:28:00.844 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_rocksdb": "4/5", 2021-02-10T04:28:00.844 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_striper": "0/1", 2021-02-10T04:28:00.844 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_test": "0/5", 2021-02-10T04:28:00.844 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_throttle": "1/1", 2021-02-10T04:28:00.845 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_timer": "0/1", 2021-02-10T04:28:00.845 INFO:teuthology.orchestra.run.smithi085.stdout: "debug_tp": "0/5", 2021-02-10T04:28:00.845 INFO:teuthology.orchestra.run.smithi085.stdout: "device_failure_prediction_mode": "none", 2021-02-10T04:28:00.845 INFO:teuthology.orchestra.run.smithi085.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2021-02-10T04:28:00.845 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2021-02-10T04:28:00.845 INFO:teuthology.orchestra.run.smithi085.stdout: "err_to_graylog": "false", 2021-02-10T04:28:00.845 INFO:teuthology.orchestra.run.smithi085.stdout: "err_to_stderr": "true", 2021-02-10T04:28:00.845 INFO:teuthology.orchestra.run.smithi085.stdout: "err_to_syslog": "false", 2021-02-10T04:28:00.845 INFO:teuthology.orchestra.run.smithi085.stdout: "event_tracing": "false", 2021-02-10T04:28:00.845 INFO:teuthology.orchestra.run.smithi085.stdout: "fake_statfs_for_testing": "0", 2021-02-10T04:28:00.845 INFO:teuthology.orchestra.run.smithi085.stdout: "fatal_signal_handlers": "true", 2021-02-10T04:28:00.845 INFO:teuthology.orchestra.run.smithi085.stdout: "filer_max_purge_ops": "10", 2021-02-10T04:28:00.845 INFO:teuthology.orchestra.run.smithi085.stdout: "filer_max_truncate_ops": "128", 2021-02-10T04:28:00.846 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_apply_finisher_threads": "1", 2021-02-10T04:28:00.846 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_blackhole": "false", 2021-02-10T04:28:00.846 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_btrfs_clone_range": "true", 2021-02-10T04:28:00.846 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_btrfs_snap": "true", 2021-02-10T04:28:00.846 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_caller_concurrency": "10", 2021-02-10T04:28:00.846 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_collect_device_partition_information": "true", 2021-02-10T04:28:00.846 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_commit_timeout": "600.000000", 2021-02-10T04:28:00.846 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_debug_inject_read_err": "false", 2021-02-10T04:28:00.846 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_debug_omap_check": "false", 2021-02-10T04:28:00.846 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_debug_random_read_err": "0.000000", 2021-02-10T04:28:00.846 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_debug_verify_split": "false", 2021-02-10T04:28:00.846 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_dump_file": "", 2021-02-10T04:28:00.846 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2021-02-10T04:28:00.847 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_expected_throughput_ops": "200.000000", 2021-02-10T04:28:00.847 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_fadvise": "true", 2021-02-10T04:28:00.847 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_fail_eio": "true", 2021-02-10T04:28:00.847 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_fd_cache_shards": "16", 2021-02-10T04:28:00.847 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_fd_cache_size": "128", 2021-02-10T04:28:00.847 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_fiemap": "false", 2021-02-10T04:28:00.847 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_fiemap_threshold": "4096", 2021-02-10T04:28:00.847 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_fsync_flushes_journal_data": "false", 2021-02-10T04:28:00.847 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_index_retry_probability": "0.000000", 2021-02-10T04:28:00.847 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_inject_stall": "0", 2021-02-10T04:28:00.847 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_journal_parallel": "false", 2021-02-10T04:28:00.847 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_journal_trailing": "false", 2021-02-10T04:28:00.847 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_journal_writeahead": "false", 2021-02-10T04:28:00.847 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_kill_at": "0", 2021-02-10T04:28:00.848 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_max_alloc_hint_size": "1048576", 2021-02-10T04:28:00.848 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_max_inline_xattr_size": "0", 2021-02-10T04:28:00.848 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2021-02-10T04:28:00.848 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_max_inline_xattr_size_other": "512", 2021-02-10T04:28:00.848 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2021-02-10T04:28:00.848 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_max_inline_xattrs": "0", 2021-02-10T04:28:00.848 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2021-02-10T04:28:00.848 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_max_inline_xattrs_other": "2", 2021-02-10T04:28:00.848 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_max_inline_xattrs_xfs": "10", 2021-02-10T04:28:00.848 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_max_sync_interval": "5.000000", 2021-02-10T04:28:00.848 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_max_xattr_value_size": "0", 2021-02-10T04:28:00.848 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2021-02-10T04:28:00.848 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_max_xattr_value_size_other": "1024", 2021-02-10T04:28:00.849 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2021-02-10T04:28:00.849 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_merge_threshold": "-10", 2021-02-10T04:28:00.849 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_min_sync_interval": "0.010000", 2021-02-10T04:28:00.849 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_odsync_write": "false", 2021-02-10T04:28:00.849 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_omap_backend": "rocksdb", 2021-02-10T04:28:00.849 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_omap_backend_path": "", 2021-02-10T04:28:00.849 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_omap_header_cache_size": "1024", 2021-02-10T04:28:00.849 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_ondisk_finisher_threads": "1", 2021-02-10T04:28:00.849 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_op_thread_suicide_timeout": "180", 2021-02-10T04:28:00.849 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_op_thread_timeout": "60", 2021-02-10T04:28:00.849 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_op_threads": "2", 2021-02-10T04:28:00.849 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_punch_hole": "false", 2021-02-10T04:28:00.849 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2021-02-10T04:28:00.850 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2021-02-10T04:28:00.850 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2021-02-10T04:28:00.850 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_queue_high_threshhold": "0.900000", 2021-02-10T04:28:00.850 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_queue_low_threshhold": "0.300000", 2021-02-10T04:28:00.850 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_queue_max_bytes": "104857600", 2021-02-10T04:28:00.850 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2021-02-10T04:28:00.850 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2021-02-10T04:28:00.850 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2021-02-10T04:28:00.850 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_queue_max_ops": "50", 2021-02-10T04:28:00.850 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2021-02-10T04:28:00.850 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_seek_data_hole": "false", 2021-02-10T04:28:00.851 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_sloppy_crc": "false", 2021-02-10T04:28:00.851 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_sloppy_crc_block_size": "65536", 2021-02-10T04:28:00.851 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_splice": "false", 2021-02-10T04:28:00.851 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_split_multiple": "2", 2021-02-10T04:28:00.851 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_split_rand_factor": "20", 2021-02-10T04:28:00.851 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_update_to": "1000", 2021-02-10T04:28:00.851 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2021-02-10T04:28:00.851 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2021-02-10T04:28:00.851 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2021-02-10T04:28:00.851 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2021-02-10T04:28:00.851 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2021-02-10T04:28:00.851 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2021-02-10T04:28:00.851 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_wbthrottle_enable": "true", 2021-02-10T04:28:00.852 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2021-02-10T04:28:00.852 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2021-02-10T04:28:00.852 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2021-02-10T04:28:00.852 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2021-02-10T04:28:00.852 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2021-02-10T04:28:00.852 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2021-02-10T04:28:00.852 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_xfs_extsize": "false", 2021-02-10T04:28:00.852 INFO:teuthology.orchestra.run.smithi085.stdout: "filestore_zfs_snap": "false", 2021-02-10T04:28:00.852 INFO:teuthology.orchestra.run.smithi085.stdout: "fio_dir": "/tmp/fio", 2021-02-10T04:28:00.852 INFO:teuthology.orchestra.run.smithi085.stdout: "fsid": "42224fce-6b57-11eb-8feb-001a4aab830c", 2021-02-10T04:28:00.852 INFO:teuthology.orchestra.run.smithi085.stdout: "fuse_allow_other": "true", 2021-02-10T04:28:00.852 INFO:teuthology.orchestra.run.smithi085.stdout: "fuse_atomic_o_trunc": "true", 2021-02-10T04:28:00.852 INFO:teuthology.orchestra.run.smithi085.stdout: "fuse_big_writes": "true", 2021-02-10T04:28:00.853 INFO:teuthology.orchestra.run.smithi085.stdout: "fuse_debug": "false", 2021-02-10T04:28:00.853 INFO:teuthology.orchestra.run.smithi085.stdout: "fuse_default_permissions": "false", 2021-02-10T04:28:00.853 INFO:teuthology.orchestra.run.smithi085.stdout: "fuse_disable_pagecache": "false", 2021-02-10T04:28:00.853 INFO:teuthology.orchestra.run.smithi085.stdout: "fuse_max_write": "0", 2021-02-10T04:28:00.853 INFO:teuthology.orchestra.run.smithi085.stdout: "fuse_multithreaded": "true", 2021-02-10T04:28:00.853 INFO:teuthology.orchestra.run.smithi085.stdout: "fuse_require_active_mds": "true", 2021-02-10T04:28:00.853 INFO:teuthology.orchestra.run.smithi085.stdout: "fuse_set_user_groups": "true", 2021-02-10T04:28:00.853 INFO:teuthology.orchestra.run.smithi085.stdout: "fuse_splice_move": "true", 2021-02-10T04:28:00.853 INFO:teuthology.orchestra.run.smithi085.stdout: "fuse_splice_read": "true", 2021-02-10T04:28:00.853 INFO:teuthology.orchestra.run.smithi085.stdout: "fuse_splice_write": "true", 2021-02-10T04:28:00.853 INFO:teuthology.orchestra.run.smithi085.stdout: "fuse_syncfs_on_mksnap": "true", 2021-02-10T04:28:00.853 INFO:teuthology.orchestra.run.smithi085.stdout: "fuse_use_invalidate_cb": "true", 2021-02-10T04:28:00.853 INFO:teuthology.orchestra.run.smithi085.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2021-02-10T04:28:00.854 INFO:teuthology.orchestra.run.smithi085.stdout: "gss_target_name": "ceph", 2021-02-10T04:28:00.854 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_file": "", 2021-02-10T04:28:00.854 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_inject_failure": "0", 2021-02-10T04:28:00.854 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_interval": "5", 2021-02-10T04:28:00.854 INFO:teuthology.orchestra.run.smithi085.stdout: "host": "", 2021-02-10T04:28:00.854 INFO:teuthology.orchestra.run.smithi085.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2021-02-10T04:28:00.854 INFO:teuthology.orchestra.run.smithi085.stdout: "immutable_object_cache_max_inflight_ops": "128", 2021-02-10T04:28:00.854 INFO:teuthology.orchestra.run.smithi085.stdout: "immutable_object_cache_max_size": "1073741824", 2021-02-10T04:28:00.854 INFO:teuthology.orchestra.run.smithi085.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2021-02-10T04:28:00.854 INFO:teuthology.orchestra.run.smithi085.stdout: "immutable_object_cache_qos_bps_burst": "0", 2021-02-10T04:28:00.854 INFO:teuthology.orchestra.run.smithi085.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2021-02-10T04:28:00.854 INFO:teuthology.orchestra.run.smithi085.stdout: "immutable_object_cache_qos_bps_limit": "0", 2021-02-10T04:28:00.854 INFO:teuthology.orchestra.run.smithi085.stdout: "immutable_object_cache_qos_iops_burst": "0", 2021-02-10T04:28:00.855 INFO:teuthology.orchestra.run.smithi085.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2021-02-10T04:28:00.855 INFO:teuthology.orchestra.run.smithi085.stdout: "immutable_object_cache_qos_iops_limit": "0", 2021-02-10T04:28:00.855 INFO:teuthology.orchestra.run.smithi085.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2021-02-10T04:28:00.855 INFO:teuthology.orchestra.run.smithi085.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2021-02-10T04:28:00.855 INFO:teuthology.orchestra.run.smithi085.stdout: "immutable_object_cache_watermark": "0.900000", 2021-02-10T04:28:00.855 INFO:teuthology.orchestra.run.smithi085.stdout: "inject_early_sigterm": "false", 2021-02-10T04:28:00.855 INFO:teuthology.orchestra.run.smithi085.stdout: "journal_aio": "true", 2021-02-10T04:28:00.855 INFO:teuthology.orchestra.run.smithi085.stdout: "journal_align_min_size": "65536", 2021-02-10T04:28:00.855 INFO:teuthology.orchestra.run.smithi085.stdout: "journal_block_align": "true", 2021-02-10T04:28:00.855 INFO:teuthology.orchestra.run.smithi085.stdout: "journal_block_size": "4096", 2021-02-10T04:28:00.855 INFO:teuthology.orchestra.run.smithi085.stdout: "journal_dio": "true", 2021-02-10T04:28:00.855 INFO:teuthology.orchestra.run.smithi085.stdout: "journal_discard": "false", 2021-02-10T04:28:00.855 INFO:teuthology.orchestra.run.smithi085.stdout: "journal_force_aio": "false", 2021-02-10T04:28:00.856 INFO:teuthology.orchestra.run.smithi085.stdout: "journal_ignore_corruption": "false", 2021-02-10T04:28:00.856 INFO:teuthology.orchestra.run.smithi085.stdout: "journal_max_write_bytes": "10485760", 2021-02-10T04:28:00.856 INFO:teuthology.orchestra.run.smithi085.stdout: "journal_max_write_entries": "100", 2021-02-10T04:28:00.856 INFO:teuthology.orchestra.run.smithi085.stdout: "journal_replay_from": "0", 2021-02-10T04:28:00.856 INFO:teuthology.orchestra.run.smithi085.stdout: "journal_throttle_high_multiple": "0.000000", 2021-02-10T04:28:00.856 INFO:teuthology.orchestra.run.smithi085.stdout: "journal_throttle_high_threshhold": "0.900000", 2021-02-10T04:28:00.856 INFO:teuthology.orchestra.run.smithi085.stdout: "journal_throttle_low_threshhold": "0.600000", 2021-02-10T04:28:00.856 INFO:teuthology.orchestra.run.smithi085.stdout: "journal_throttle_max_multiple": "0.000000", 2021-02-10T04:28:00.856 INFO:teuthology.orchestra.run.smithi085.stdout: "journal_write_header_frequency": "0", 2021-02-10T04:28:00.856 INFO:teuthology.orchestra.run.smithi085.stdout: "journal_zero_on_create": "false", 2021-02-10T04:28:00.856 INFO:teuthology.orchestra.run.smithi085.stdout: "journaler_prefetch_periods": "10", 2021-02-10T04:28:00.856 INFO:teuthology.orchestra.run.smithi085.stdout: "journaler_prezero_periods": "5", 2021-02-10T04:28:00.856 INFO:teuthology.orchestra.run.smithi085.stdout: "journaler_write_head_interval": "15", 2021-02-10T04:28:00.857 INFO:teuthology.orchestra.run.smithi085.stdout: "key": "", 2021-02-10T04:28:00.857 INFO:teuthology.orchestra.run.smithi085.stdout: "keyfile": "", 2021-02-10T04:28:00.857 INFO:teuthology.orchestra.run.smithi085.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2021-02-10T04:28:00.857 INFO:teuthology.orchestra.run.smithi085.stdout: "kstore_backend": "rocksdb", 2021-02-10T04:28:00.857 INFO:teuthology.orchestra.run.smithi085.stdout: "kstore_default_stripe_size": "65536", 2021-02-10T04:28:00.857 INFO:teuthology.orchestra.run.smithi085.stdout: "kstore_fsck_on_mount": "false", 2021-02-10T04:28:00.857 INFO:teuthology.orchestra.run.smithi085.stdout: "kstore_fsck_on_mount_deep": "true", 2021-02-10T04:28:00.857 INFO:teuthology.orchestra.run.smithi085.stdout: "kstore_max_bytes": "67108864", 2021-02-10T04:28:00.857 INFO:teuthology.orchestra.run.smithi085.stdout: "kstore_max_ops": "512", 2021-02-10T04:28:00.857 INFO:teuthology.orchestra.run.smithi085.stdout: "kstore_nid_prealloc": "1024", 2021-02-10T04:28:00.857 INFO:teuthology.orchestra.run.smithi085.stdout: "kstore_onode_map_size": "1024", 2021-02-10T04:28:00.857 INFO:teuthology.orchestra.run.smithi085.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2021-02-10T04:28:00.857 INFO:teuthology.orchestra.run.smithi085.stdout: "kstore_sync_submit_transaction": "false", 2021-02-10T04:28:00.858 INFO:teuthology.orchestra.run.smithi085.stdout: "kstore_sync_transaction": "false", 2021-02-10T04:28:00.858 INFO:teuthology.orchestra.run.smithi085.stdout: "leveldb_block_size": "65536", 2021-02-10T04:28:00.858 INFO:teuthology.orchestra.run.smithi085.stdout: "leveldb_bloom_size": "0", 2021-02-10T04:28:00.858 INFO:teuthology.orchestra.run.smithi085.stdout: "leveldb_cache_size": "536870912", 2021-02-10T04:28:00.858 INFO:teuthology.orchestra.run.smithi085.stdout: "leveldb_compact_on_mount": "false", 2021-02-10T04:28:00.858 INFO:teuthology.orchestra.run.smithi085.stdout: "leveldb_compression": "false", 2021-02-10T04:28:00.858 INFO:teuthology.orchestra.run.smithi085.stdout: "leveldb_log": "", 2021-02-10T04:28:00.858 INFO:teuthology.orchestra.run.smithi085.stdout: "leveldb_log_to_ceph_log": "true", 2021-02-10T04:28:00.858 INFO:teuthology.orchestra.run.smithi085.stdout: "leveldb_max_open_files": "0", 2021-02-10T04:28:00.858 INFO:teuthology.orchestra.run.smithi085.stdout: "leveldb_paranoid": "false", 2021-02-10T04:28:00.858 INFO:teuthology.orchestra.run.smithi085.stdout: "leveldb_write_buffer_size": "33554432", 2021-02-10T04:28:00.858 INFO:teuthology.orchestra.run.smithi085.stdout: "librados_thread_count": "2", 2021-02-10T04:28:00.858 INFO:teuthology.orchestra.run.smithi085.stdout: "lockdep": "false", 2021-02-10T04:28:00.859 INFO:teuthology.orchestra.run.smithi085.stdout: "lockdep_force_backtrace": "false", 2021-02-10T04:28:00.859 INFO:teuthology.orchestra.run.smithi085.stdout: "log_coarse_timestamps": "true", 2021-02-10T04:28:00.859 INFO:teuthology.orchestra.run.smithi085.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2021-02-10T04:28:00.859 INFO:teuthology.orchestra.run.smithi085.stdout: "log_flush_on_exit": "false", 2021-02-10T04:28:00.859 INFO:teuthology.orchestra.run.smithi085.stdout: "log_graylog_host": "127.0.0.1", 2021-02-10T04:28:00.859 INFO:teuthology.orchestra.run.smithi085.stdout: "log_graylog_port": "12201", 2021-02-10T04:28:00.859 INFO:teuthology.orchestra.run.smithi085.stdout: "log_max_new": "1000", 2021-02-10T04:28:00.859 INFO:teuthology.orchestra.run.smithi085.stdout: "log_max_recent": "10000", 2021-02-10T04:28:00.859 INFO:teuthology.orchestra.run.smithi085.stdout: "log_stderr_prefix": "debug ", 2021-02-10T04:28:00.859 INFO:teuthology.orchestra.run.smithi085.stdout: "log_stop_at_utilization": "0.970000", 2021-02-10T04:28:00.859 INFO:teuthology.orchestra.run.smithi085.stdout: "log_to_file": "true", 2021-02-10T04:28:00.859 INFO:teuthology.orchestra.run.smithi085.stdout: "log_to_graylog": "false", 2021-02-10T04:28:00.859 INFO:teuthology.orchestra.run.smithi085.stdout: "log_to_stderr": "false", 2021-02-10T04:28:00.859 INFO:teuthology.orchestra.run.smithi085.stdout: "log_to_syslog": "false", 2021-02-10T04:28:00.860 INFO:teuthology.orchestra.run.smithi085.stdout: "max_rotating_auth_attempts": "10", 2021-02-10T04:28:00.860 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_action_on_write_error": "1", 2021-02-10T04:28:00.860 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_alternate_name_max": "8192", 2021-02-10T04:28:00.860 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_asio_thread_count": "2", 2021-02-10T04:28:00.860 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_bal_export_pin": "true", 2021-02-10T04:28:00.860 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_bal_fragment_dirs": "true", 2021-02-10T04:28:00.860 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2021-02-10T04:28:00.860 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_bal_fragment_interval": "5", 2021-02-10T04:28:00.860 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_bal_fragment_size_max": "100000", 2021-02-10T04:28:00.860 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_bal_idle_threshold": "0.000000", 2021-02-10T04:28:00.860 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_bal_interval": "10", 2021-02-10T04:28:00.860 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_bal_max": "-1", 2021-02-10T04:28:00.860 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_bal_max_until": "-1", 2021-02-10T04:28:00.861 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_bal_merge_size": "50", 2021-02-10T04:28:00.861 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_bal_midchunk": "0.300000", 2021-02-10T04:28:00.861 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_bal_min_rebalance": "0.100000", 2021-02-10T04:28:00.861 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_bal_min_start": "0.200000", 2021-02-10T04:28:00.861 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_bal_minchunk": "0.001000", 2021-02-10T04:28:00.861 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_bal_mode": "0", 2021-02-10T04:28:00.861 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_bal_need_max": "1.200000", 2021-02-10T04:28:00.861 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_bal_need_min": "0.800000", 2021-02-10T04:28:00.861 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_bal_replicate_threshold": "8000.000000", 2021-02-10T04:28:00.861 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_bal_sample_interval": "3.000000", 2021-02-10T04:28:00.861 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_bal_split_bits": "3", 2021-02-10T04:28:00.861 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_bal_split_rd": "25000.000000", 2021-02-10T04:28:00.861 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_bal_split_size": "10000", 2021-02-10T04:28:00.862 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_bal_split_wr": "10000.000000", 2021-02-10T04:28:00.862 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_bal_target_decay": "10.000000", 2021-02-10T04:28:00.862 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2021-02-10T04:28:00.862 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_beacon_grace": "15.000000", 2021-02-10T04:28:00.862 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_beacon_interval": "4.000000", 2021-02-10T04:28:00.862 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_cache_memory_limit": "4294967296", 2021-02-10T04:28:00.862 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_cache_mid": "0.700000", 2021-02-10T04:28:00.862 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_cache_release_free_interval": "10", 2021-02-10T04:28:00.862 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_cache_reservation": "0.050000", 2021-02-10T04:28:00.862 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_cache_trim_decay_rate": "1.000000", 2021-02-10T04:28:00.862 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_cache_trim_interval": "1", 2021-02-10T04:28:00.862 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_cache_trim_threshold": "262144", 2021-02-10T04:28:00.862 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2021-02-10T04:28:00.863 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2021-02-10T04:28:00.863 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_client_delegate_inos_pct": "50", 2021-02-10T04:28:00.863 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_client_prealloc_inos": "1000", 2021-02-10T04:28:00.863 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2021-02-10T04:28:00.863 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_damage_table_max_entries": "10000", 2021-02-10T04:28:00.863 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2021-02-10T04:28:00.863 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_debug_auth_pins": "false", 2021-02-10T04:28:00.863 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_debug_frag": "false", 2021-02-10T04:28:00.863 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_debug_scatterstat": "false", 2021-02-10T04:28:00.863 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_debug_subtrees": "false", 2021-02-10T04:28:00.863 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_decay_halflife": "5.000000", 2021-02-10T04:28:00.863 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_default_dir_hash": "2", 2021-02-10T04:28:00.863 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_defer_session_stale": "true", 2021-02-10T04:28:00.864 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_deny_all_reconnect": "false", 2021-02-10T04:28:00.864 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_dir_keys_per_op": "16384", 2021-02-10T04:28:00.864 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_dir_max_commit_size": "10", 2021-02-10T04:28:00.864 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_dirstat_min_interval": "1.000000", 2021-02-10T04:28:00.864 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_dump_cache_after_rejoin": "false", 2021-02-10T04:28:00.864 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_dump_cache_on_map": "false", 2021-02-10T04:28:00.864 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_dump_cache_threshold_file": "0", 2021-02-10T04:28:00.864 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2021-02-10T04:28:00.864 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_early_reply": "true", 2021-02-10T04:28:00.864 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_enable_op_tracker": "true", 2021-02-10T04:28:00.864 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_enforce_unique_name": "true", 2021-02-10T04:28:00.864 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_export_ephemeral_distributed": "true", 2021-02-10T04:28:00.864 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2021-02-10T04:28:00.865 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_export_ephemeral_random": "true", 2021-02-10T04:28:00.865 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_export_ephemeral_random_max": "0.010000", 2021-02-10T04:28:00.865 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_forward_all_requests_to_auth": "false", 2021-02-10T04:28:00.865 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_freeze_tree_timeout": "30.000000", 2021-02-10T04:28:00.865 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2021-02-10T04:28:00.865 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_health_cache_threshold": "1.500000", 2021-02-10T04:28:00.865 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_health_summarize_threshold": "10", 2021-02-10T04:28:00.865 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_heartbeat_grace": "15.000000", 2021-02-10T04:28:00.865 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_inject_migrator_session_race": "false", 2021-02-10T04:28:00.865 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2021-02-10T04:28:00.865 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_join_fs": "", 2021-02-10T04:28:00.865 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_journal_format": "1", 2021-02-10T04:28:00.865 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_kill_create_at": "0", 2021-02-10T04:28:00.866 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_kill_export_at": "0", 2021-02-10T04:28:00.866 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_kill_import_at": "0", 2021-02-10T04:28:00.866 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_kill_journal_at": "0", 2021-02-10T04:28:00.866 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_kill_journal_expire_at": "0", 2021-02-10T04:28:00.866 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_kill_journal_replay_at": "0", 2021-02-10T04:28:00.866 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_kill_link_at": "0", 2021-02-10T04:28:00.866 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_kill_mdstable_at": "0", 2021-02-10T04:28:00.866 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_kill_openc_at": "0", 2021-02-10T04:28:00.866 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_kill_rename_at": "0", 2021-02-10T04:28:00.866 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_log_events_per_segment": "1024", 2021-02-10T04:28:00.866 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_log_max_events": "-1", 2021-02-10T04:28:00.866 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_log_max_segments": "128", 2021-02-10T04:28:00.867 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_log_pause": "false", 2021-02-10T04:28:00.867 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_log_segment_size": "0", 2021-02-10T04:28:00.867 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_log_skip_corrupt_events": "false", 2021-02-10T04:28:00.867 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_log_warn_factor": "2.000000", 2021-02-10T04:28:00.867 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_max_caps_per_client": "1048576", 2021-02-10T04:28:00.867 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_max_completed_flushes": "100000", 2021-02-10T04:28:00.867 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_max_completed_requests": "100000", 2021-02-10T04:28:00.867 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_max_export_size": "20971520", 2021-02-10T04:28:00.867 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_max_file_recover": "32", 2021-02-10T04:28:00.867 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_max_purge_files": "64", 2021-02-10T04:28:00.867 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_max_purge_ops": "8192", 2021-02-10T04:28:00.867 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2021-02-10T04:28:00.867 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_max_retries_on_remount_failure": "5", 2021-02-10T04:28:00.868 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_max_scrub_ops_in_progress": "5", 2021-02-10T04:28:00.868 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_max_snaps_per_dir": "100", 2021-02-10T04:28:00.868 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_max_xattr_pairs_size": "65536", 2021-02-10T04:28:00.868 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_metrics_update_interval": "2", 2021-02-10T04:28:00.868 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_min_caps_per_client": "100", 2021-02-10T04:28:00.868 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_min_caps_working_set": "10000", 2021-02-10T04:28:00.868 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_mon_shutdown_timeout": "5.000000", 2021-02-10T04:28:00.868 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_numa_node": "-1", 2021-02-10T04:28:00.868 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_oft_prefetch_dirfrags": "true", 2021-02-10T04:28:00.868 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_op_complaint_time": "30.000000", 2021-02-10T04:28:00.868 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_op_history_duration": "600", 2021-02-10T04:28:00.868 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_op_history_size": "20", 2021-02-10T04:28:00.868 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_op_log_threshold": "5", 2021-02-10T04:28:00.869 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_ping_grace": "15", 2021-02-10T04:28:00.869 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_ping_interval": "5", 2021-02-10T04:28:00.869 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2021-02-10T04:28:00.869 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_recall_global_max_decay_threshold": "131072", 2021-02-10T04:28:00.869 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_recall_max_caps": "30000", 2021-02-10T04:28:00.869 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_recall_max_decay_rate": "1.500000", 2021-02-10T04:28:00.869 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_recall_max_decay_threshold": "131072", 2021-02-10T04:28:00.869 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_recall_warning_decay_rate": "60.000000", 2021-02-10T04:28:00.869 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_recall_warning_threshold": "262144", 2021-02-10T04:28:00.869 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_reconnect_timeout": "45.000000", 2021-02-10T04:28:00.869 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_replay_interval": "1.000000", 2021-02-10T04:28:00.869 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_replay_unsafe_with_closed_session": "false", 2021-02-10T04:28:00.869 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_request_load_average_decay_rate": "60.000000", 2021-02-10T04:28:00.869 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_root_ino_gid": "0", 2021-02-10T04:28:00.870 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_root_ino_uid": "0", 2021-02-10T04:28:00.870 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_scatter_nudge_interval": "5.000000", 2021-02-10T04:28:00.870 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_session_blocklist_on_evict": "true", 2021-02-10T04:28:00.870 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_session_blocklist_on_timeout": "true", 2021-02-10T04:28:00.870 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2021-02-10T04:28:00.870 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_session_cache_liveness_magnitude": "10", 2021-02-10T04:28:00.870 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_session_cap_acquisition_decay_rate": "10.000000", 2021-02-10T04:28:00.870 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_session_cap_acquisition_throttle": "500000", 2021-02-10T04:28:00.870 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2021-02-10T04:28:00.870 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_sessionmap_keys_per_op": "1024", 2021-02-10T04:28:00.870 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_shutdown_check": "0", 2021-02-10T04:28:00.870 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_skip_ino": "0", 2021-02-10T04:28:00.871 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_snap_max_uid": "4294967294", 2021-02-10T04:28:00.871 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_snap_min_uid": "0", 2021-02-10T04:28:00.871 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_snap_rstat": "false", 2021-02-10T04:28:00.871 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_task_status_update_interval": "2.000000", 2021-02-10T04:28:00.871 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_thrash_exports": "0", 2021-02-10T04:28:00.871 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_thrash_fragments": "0", 2021-02-10T04:28:00.871 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_tick_interval": "5.000000", 2021-02-10T04:28:00.871 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_verify_backtrace": "1", 2021-02-10T04:28:00.871 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_verify_scatter": "false", 2021-02-10T04:28:00.871 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_wipe_ino_prealloc": "false", 2021-02-10T04:28:00.871 INFO:teuthology.orchestra.run.smithi085.stdout: "mds_wipe_sessions": "false", 2021-02-10T04:28:00.871 INFO:teuthology.orchestra.run.smithi085.stdout: "mempool_debug": "false", 2021-02-10T04:28:00.871 INFO:teuthology.orchestra.run.smithi085.stdout: "memstore_debug_omit_block_device_write": "false", 2021-02-10T04:28:00.872 INFO:teuthology.orchestra.run.smithi085.stdout: "memstore_device_bytes": "1073741824", 2021-02-10T04:28:00.872 INFO:teuthology.orchestra.run.smithi085.stdout: "memstore_page_set": "false", 2021-02-10T04:28:00.872 INFO:teuthology.orchestra.run.smithi085.stdout: "memstore_page_size": "65536", 2021-02-10T04:28:00.872 INFO:teuthology.orchestra.run.smithi085.stdout: "mgr_client_bytes": "134217728", 2021-02-10T04:28:00.872 INFO:teuthology.orchestra.run.smithi085.stdout: "mgr_client_messages": "512", 2021-02-10T04:28:00.872 INFO:teuthology.orchestra.run.smithi085.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2021-02-10T04:28:00.872 INFO:teuthology.orchestra.run.smithi085.stdout: "mgr_connect_retry_interval": "1.000000", 2021-02-10T04:28:00.872 INFO:teuthology.orchestra.run.smithi085.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2021-02-10T04:28:00.872 INFO:teuthology.orchestra.run.smithi085.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2021-02-10T04:28:00.872 INFO:teuthology.orchestra.run.smithi085.stdout: "mgr_disabled_modules": "", 2021-02-10T04:28:00.872 INFO:teuthology.orchestra.run.smithi085.stdout: "mgr_initial_modules": "restful iostat", 2021-02-10T04:28:00.872 INFO:teuthology.orchestra.run.smithi085.stdout: "mgr_mds_bytes": "134217728", 2021-02-10T04:28:00.872 INFO:teuthology.orchestra.run.smithi085.stdout: "mgr_mds_messages": "128", 2021-02-10T04:28:00.873 INFO:teuthology.orchestra.run.smithi085.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2021-02-10T04:28:00.873 INFO:teuthology.orchestra.run.smithi085.stdout: "mgr_mon_bytes": "134217728", 2021-02-10T04:28:00.873 INFO:teuthology.orchestra.run.smithi085.stdout: "mgr_mon_messages": "128", 2021-02-10T04:28:00.873 INFO:teuthology.orchestra.run.smithi085.stdout: "mgr_osd_bytes": "536870912", 2021-02-10T04:28:00.873 INFO:teuthology.orchestra.run.smithi085.stdout: "mgr_osd_messages": "8192", 2021-02-10T04:28:00.873 INFO:teuthology.orchestra.run.smithi085.stdout: "mgr_service_beacon_grace": "60.000000", 2021-02-10T04:28:00.873 INFO:teuthology.orchestra.run.smithi085.stdout: "mgr_stats_period": "5", 2021-02-10T04:28:00.873 INFO:teuthology.orchestra.run.smithi085.stdout: "mgr_stats_threshold": "5", 2021-02-10T04:28:00.873 INFO:teuthology.orchestra.run.smithi085.stdout: "mgr_tick_period": "2", 2021-02-10T04:28:00.873 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_accept_timeout_factor": "2.000000", 2021-02-10T04:28:00.873 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_allow_pool_delete": "true", 2021-02-10T04:28:00.873 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_allow_pool_size_one": "false", 2021-02-10T04:28:00.873 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_auth_validate_all_caps": "true", 2021-02-10T04:28:00.874 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2021-02-10T04:28:00.874 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2021-02-10T04:28:00.874 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_client_bytes": "104857600", 2021-02-10T04:28:00.874 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_client_directed_command_retry": "2", 2021-02-10T04:28:00.874 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_client_hunt_interval": "3.000000", 2021-02-10T04:28:00.874 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2021-02-10T04:28:00.874 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2021-02-10T04:28:00.874 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2021-02-10T04:28:00.874 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_client_hunt_parallel": "3", 2021-02-10T04:28:00.874 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_client_log_interval": "1.000000", 2021-02-10T04:28:00.874 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_client_max_log_entries_per_message": "1000", 2021-02-10T04:28:00.874 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_client_ping_interval": "10.000000", 2021-02-10T04:28:00.875 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_client_ping_timeout": "30.000000", 2021-02-10T04:28:00.875 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_clock_drift_allowed": "1.000000", 2021-02-10T04:28:00.875 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2021-02-10T04:28:00.875 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2021-02-10T04:28:00.875 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_cluster_log_file_level": "debug", 2021-02-10T04:28:00.875 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_cluster_log_to_file": "false", 2021-02-10T04:28:00.875 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_cluster_log_to_graylog": "false", 2021-02-10T04:28:00.875 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2021-02-10T04:28:00.875 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_cluster_log_to_graylog_port": "12201", 2021-02-10T04:28:00.875 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_cluster_log_to_stderr": "true", 2021-02-10T04:28:00.875 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_cluster_log_to_syslog": "default=false", 2021-02-10T04:28:00.875 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2021-02-10T04:28:00.875 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_cluster_log_to_syslog_level": "info", 2021-02-10T04:28:00.876 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_compact_on_bootstrap": "false", 2021-02-10T04:28:00.876 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_compact_on_start": "false", 2021-02-10T04:28:00.876 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_compact_on_trim": "true", 2021-02-10T04:28:00.876 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_con_tracker_persist_interval": "10", 2021-02-10T04:28:00.876 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_con_tracker_score_halflife": "43200", 2021-02-10T04:28:00.876 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_config_key_max_entry_size": "65536", 2021-02-10T04:28:00.876 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_cpu_threads": "4", 2021-02-10T04:28:00.876 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_crush_min_required_version": "hammer", 2021-02-10T04:28:00.876 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_daemon_bytes": "419430400", 2021-02-10T04:28:00.876 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2021-02-10T04:28:00.876 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_data_avail_crit": "5", 2021-02-10T04:28:00.876 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_data_avail_warn": "5", 2021-02-10T04:28:00.876 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_data_size_warn": "16106127360", 2021-02-10T04:28:00.876 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_debug_block_osdmap_trim": "false", 2021-02-10T04:28:00.877 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_debug_deprecated_as_obsolete": "false", 2021-02-10T04:28:00.877 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_debug_dump_json": "false", 2021-02-10T04:28:00.877 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2021-02-10T04:28:00.877 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_debug_dump_transactions": "false", 2021-02-10T04:28:00.877 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_debug_extra_checks": "false", 2021-02-10T04:28:00.877 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_debug_no_initial_persistent_features": "false", 2021-02-10T04:28:00.877 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2021-02-10T04:28:00.877 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_debug_no_require_pacific": "false", 2021-02-10T04:28:00.877 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_debug_no_require_quincy": "false", 2021-02-10T04:28:00.877 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2021-02-10T04:28:00.877 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_delta_reset_interval": "10.000000", 2021-02-10T04:28:00.877 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_dns_srv_name": "ceph-mon", 2021-02-10T04:28:00.877 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_election_default_strategy": "1", 2021-02-10T04:28:00.878 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_election_timeout": "5.000000", 2021-02-10T04:28:00.878 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2021-02-10T04:28:00.878 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_elector_ping_divisor": "2", 2021-02-10T04:28:00.878 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_elector_ping_timeout": "2.000000", 2021-02-10T04:28:00.878 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_enable_op_tracker": "true", 2021-02-10T04:28:00.878 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_fake_pool_delete": "false", 2021-02-10T04:28:00.878 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_force_quorum_join": "false", 2021-02-10T04:28:00.878 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_globalid_prealloc": "10000", 2021-02-10T04:28:00.878 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_health_detail_to_clog": "true", 2021-02-10T04:28:00.878 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_health_log_update_period": "5", 2021-02-10T04:28:00.878 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_health_max_detail": "50", 2021-02-10T04:28:00.878 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_health_to_clog": "true", 2021-02-10T04:28:00.878 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_health_to_clog_interval": "600", 2021-02-10T04:28:00.879 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2021-02-10T04:28:00.879 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_host": "[v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0]", 2021-02-10T04:28:00.879 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_host_override": "", 2021-02-10T04:28:00.879 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_initial_members": "", 2021-02-10T04:28:00.879 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2021-02-10T04:28:00.879 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2021-02-10T04:28:00.879 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_inject_transaction_delay_max": "10.000000", 2021-02-10T04:28:00.879 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2021-02-10T04:28:00.879 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_keyvaluedb": "rocksdb", 2021-02-10T04:28:00.879 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_lease": "5.000000", 2021-02-10T04:28:00.879 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2021-02-10T04:28:00.879 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_lease_renew_interval_factor": "0.600000", 2021-02-10T04:28:00.879 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_log_max_summary": "50", 2021-02-10T04:28:00.880 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_max_log_entries_per_event": "4096", 2021-02-10T04:28:00.880 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_max_log_epochs": "500", 2021-02-10T04:28:00.880 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_max_mdsmap_epochs": "500", 2021-02-10T04:28:00.880 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_max_mgrmap_epochs": "500", 2021-02-10T04:28:00.880 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_max_osd": "10000", 2021-02-10T04:28:00.880 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_max_pg_per_osd": "10000", 2021-02-10T04:28:00.880 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_max_pool_pg_num": "65536", 2021-02-10T04:28:00.880 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_max_snap_prune_per_epoch": "100", 2021-02-10T04:28:00.880 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_mds_blocklist_interval": "86400.000000", 2021-02-10T04:28:00.880 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_mds_force_trim_to": "0", 2021-02-10T04:28:00.880 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_mds_skip_sanity": "false", 2021-02-10T04:28:00.880 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_memory_autotune": "true", 2021-02-10T04:28:00.880 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_memory_target": "2147483648", 2021-02-10T04:28:00.881 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_mgr_beacon_grace": "30", 2021-02-10T04:28:00.881 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2021-02-10T04:28:00.881 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_mgr_digest_period": "5", 2021-02-10T04:28:00.881 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_mgr_inactive_grace": "60", 2021-02-10T04:28:00.881 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_mgr_mkfs_grace": "240", 2021-02-10T04:28:00.881 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2021-02-10T04:28:00.881 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_min_osdmap_epochs": "50", 2021-02-10T04:28:00.881 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_op_complaint_time": "30", 2021-02-10T04:28:00.881 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_op_history_duration": "600", 2021-02-10T04:28:00.881 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_op_history_size": "20", 2021-02-10T04:28:00.881 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_op_history_slow_op_size": "20", 2021-02-10T04:28:00.881 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_op_history_slow_op_threshold": "10", 2021-02-10T04:28:00.881 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_op_log_threshold": "5", 2021-02-10T04:28:00.882 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_adjust_down_out_interval": "true", 2021-02-10T04:28:00.882 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2021-02-10T04:28:00.882 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2021-02-10T04:28:00.882 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_auto_mark_in": "false", 2021-02-10T04:28:00.882 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_auto_mark_new_in": "true", 2021-02-10T04:28:00.882 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2021-02-10T04:28:00.882 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2021-02-10T04:28:00.882 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_cache_size": "500", 2021-02-10T04:28:00.882 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_cache_size_min": "134217728", 2021-02-10T04:28:00.882 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_crush_smoke_test": "true", 2021-02-10T04:28:00.882 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_destroyed_out_interval": "600", 2021-02-10T04:28:00.882 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_down_out_interval": "600", 2021-02-10T04:28:00.882 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_down_out_subtree_limit": "rack", 2021-02-10T04:28:00.883 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2021-02-10T04:28:00.883 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_force_trim_to": "0", 2021-02-10T04:28:00.883 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_full_ratio": "0.950000", 2021-02-10T04:28:00.883 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2021-02-10T04:28:00.883 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_laggy_halflife": "3600", 2021-02-10T04:28:00.883 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_laggy_max_interval": "300", 2021-02-10T04:28:00.883 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_laggy_weight": "0.300000", 2021-02-10T04:28:00.883 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2021-02-10T04:28:00.883 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_max_creating_pgs": "1024", 2021-02-10T04:28:00.883 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_max_initial_pgs": "1024", 2021-02-10T04:28:00.883 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_min_down_reporters": "2", 2021-02-10T04:28:00.883 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_min_in_ratio": "0.750000", 2021-02-10T04:28:00.883 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_min_up_ratio": "0.300000", 2021-02-10T04:28:00.884 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_nearfull_ratio": "0.850000", 2021-02-10T04:28:00.884 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_prime_pg_temp": "true", 2021-02-10T04:28:00.884 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2021-02-10T04:28:00.884 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2021-02-10T04:28:00.884 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_report_timeout": "900", 2021-02-10T04:28:00.884 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_reporter_subtree_level": "osd", 2021-02-10T04:28:00.884 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2021-02-10T04:28:00.884 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_warn_num_repaired": "10", 2021-02-10T04:28:00.884 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osd_warn_op_age": "32.000000", 2021-02-10T04:28:00.884 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osdmap_full_prune_enabled": "true", 2021-02-10T04:28:00.884 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osdmap_full_prune_interval": "2", 2021-02-10T04:28:00.884 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osdmap_full_prune_min": "15", 2021-02-10T04:28:00.884 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_osdmap_full_prune_txsize": "2", 2021-02-10T04:28:00.885 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2021-02-10T04:28:00.885 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_pg_stuck_threshold": "60", 2021-02-10T04:28:00.885 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2021-02-10T04:28:00.885 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_pg_warn_min_objects": "10000", 2021-02-10T04:28:00.885 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_pg_warn_min_per_osd": "0", 2021-02-10T04:28:00.885 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_pg_warn_min_pool_objects": "1000", 2021-02-10T04:28:00.885 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_pool_quota_crit_threshold": "0", 2021-02-10T04:28:00.885 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_pool_quota_warn_threshold": "0", 2021-02-10T04:28:00.885 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_probe_timeout": "2.000000", 2021-02-10T04:28:00.885 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_reweight_max_change": "0.050000", 2021-02-10T04:28:00.885 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_reweight_max_osds": "4", 2021-02-10T04:28:00.885 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_reweight_min_bytes_per_osd": "10", 2021-02-10T04:28:00.885 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_reweight_min_pgs_per_osd": "4", 2021-02-10T04:28:00.886 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2021-02-10T04:28:00.886 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2021-02-10T04:28:00.886 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2021-02-10T04:28:00.886 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_scrub_interval": "86400", 2021-02-10T04:28:00.886 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_scrub_max_keys": "100", 2021-02-10T04:28:00.886 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_scrub_timeout": "300", 2021-02-10T04:28:00.886 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_session_timeout": "300", 2021-02-10T04:28:00.886 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_smart_report_timeout": "5", 2021-02-10T04:28:00.886 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_stat_smooth_intervals": "6", 2021-02-10T04:28:00.886 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2021-02-10T04:28:00.886 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_stretch_pool_min_size": "2", 2021-02-10T04:28:00.886 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_stretch_pool_size": "4", 2021-02-10T04:28:00.887 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2021-02-10T04:28:00.887 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_subscribe_interval": "86400.000000", 2021-02-10T04:28:00.887 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_sync_debug": "false", 2021-02-10T04:28:00.887 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_sync_max_payload_keys": "2000", 2021-02-10T04:28:00.887 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_sync_max_payload_size": "1048576", 2021-02-10T04:28:00.887 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_sync_provider_kill_at": "0", 2021-02-10T04:28:00.887 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_sync_requester_kill_at": "0", 2021-02-10T04:28:00.887 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_sync_timeout": "60.000000", 2021-02-10T04:28:00.887 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_target_pg_per_osd": "100", 2021-02-10T04:28:00.887 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_tick_interval": "5", 2021-02-10T04:28:00.887 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_timecheck_interval": "300.000000", 2021-02-10T04:28:00.887 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_timecheck_skew_interval": "30.000000", 2021-02-10T04:28:00.887 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_warn_older_version_delay": "604800", 2021-02-10T04:28:00.888 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2021-02-10T04:28:00.888 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2021-02-10T04:28:00.888 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2021-02-10T04:28:00.888 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2021-02-10T04:28:00.888 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_warn_on_misplaced": "false", 2021-02-10T04:28:00.888 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2021-02-10T04:28:00.888 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_warn_on_older_version": "true", 2021-02-10T04:28:00.888 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2021-02-10T04:28:00.888 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_warn_on_pool_no_app": "true", 2021-02-10T04:28:00.888 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_warn_on_pool_no_redundancy": "true", 2021-02-10T04:28:00.888 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2021-02-10T04:28:00.888 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2021-02-10T04:28:00.888 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2021-02-10T04:28:00.889 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_warn_on_too_few_osds": "false", 2021-02-10T04:28:00.889 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2021-02-10T04:28:00.889 INFO:teuthology.orchestra.run.smithi085.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2021-02-10T04:28:00.889 INFO:teuthology.orchestra.run.smithi085.stdout: "monmap": "", 2021-02-10T04:28:00.889 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_async_max_op_threads": "5", 2021-02-10T04:28:00.889 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_async_op_threads": "3", 2021-02-10T04:28:00.889 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_async_rdma_buffer_size": "131072", 2021-02-10T04:28:00.889 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_async_rdma_cm": "false", 2021-02-10T04:28:00.889 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_async_rdma_device_name": "", 2021-02-10T04:28:00.889 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_async_rdma_dscp": "96", 2021-02-10T04:28:00.889 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_async_rdma_enable_hugepage": "false", 2021-02-10T04:28:00.889 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_async_rdma_gid_idx": "0", 2021-02-10T04:28:00.889 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_async_rdma_local_gid": "", 2021-02-10T04:28:00.890 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_async_rdma_polling_us": "1000", 2021-02-10T04:28:00.890 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_async_rdma_port_num": "1", 2021-02-10T04:28:00.890 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_async_rdma_receive_buffers": "32768", 2021-02-10T04:28:00.890 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_async_rdma_receive_queue_len": "4096", 2021-02-10T04:28:00.890 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_async_rdma_roce_ver": "1", 2021-02-10T04:28:00.890 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_async_rdma_send_buffers": "1024", 2021-02-10T04:28:00.890 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_async_rdma_sl": "3", 2021-02-10T04:28:00.890 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_async_rdma_support_srq": "true", 2021-02-10T04:28:00.890 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_async_rdma_type": "ib", 2021-02-10T04:28:00.890 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_bind_before_connect": "false", 2021-02-10T04:28:00.890 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_bind_ipv4": "true", 2021-02-10T04:28:00.890 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_bind_ipv6": "false", 2021-02-10T04:28:00.890 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_bind_msgr1": "false", 2021-02-10T04:28:00.891 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_bind_msgr2": "true", 2021-02-10T04:28:00.891 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_bind_port_max": "7300", 2021-02-10T04:28:00.891 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_bind_port_min": "6800", 2021-02-10T04:28:00.891 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_bind_prefer_ipv4": "false", 2021-02-10T04:28:00.891 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_bind_retry_count": "3", 2021-02-10T04:28:00.891 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_bind_retry_delay": "5", 2021-02-10T04:28:00.891 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_blackhole_client": "false", 2021-02-10T04:28:00.891 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_blackhole_mds": "false", 2021-02-10T04:28:00.891 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_blackhole_mgr": "false", 2021-02-10T04:28:00.891 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_blackhole_mon": "false", 2021-02-10T04:28:00.891 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_blackhole_osd": "false", 2021-02-10T04:28:00.891 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_client_mode": "crc secure", 2021-02-10T04:28:00.891 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_cluster_mode": "crc secure", 2021-02-10T04:28:00.892 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_cluster_type": "", 2021-02-10T04:28:00.892 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_connection_idle_timeout": "900", 2021-02-10T04:28:00.892 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_connection_ready_timeout": "10", 2021-02-10T04:28:00.892 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_crc_data": "true", 2021-02-10T04:28:00.892 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_crc_header": "true", 2021-02-10T04:28:00.892 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_die_on_bad_msg": "false", 2021-02-10T04:28:00.892 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_die_on_bug": "true", 2021-02-10T04:28:00.892 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_die_on_old_message": "true", 2021-02-10T04:28:00.892 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_die_on_skipped_message": "false", 2021-02-10T04:28:00.892 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_die_on_unhandled_msg": "false", 2021-02-10T04:28:00.892 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_dispatch_throttle_bytes": "104857600", 2021-02-10T04:28:00.892 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_dpdk_coremask": "0xF", 2021-02-10T04:28:00.892 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_dpdk_debug_allow_loopback": "false", 2021-02-10T04:28:00.893 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2021-02-10T04:28:00.893 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_dpdk_host_ipv4_addr": "", 2021-02-10T04:28:00.893 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_dpdk_hugepages": "", 2021-02-10T04:28:00.893 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_dpdk_hw_flow_control": "true", 2021-02-10T04:28:00.893 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2021-02-10T04:28:00.893 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_dpdk_lro": "true", 2021-02-10T04:28:00.893 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_dpdk_memory_channel": "4", 2021-02-10T04:28:00.893 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2021-02-10T04:28:00.893 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_dpdk_pmd": "", 2021-02-10T04:28:00.893 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_dpdk_port_id": "0", 2021-02-10T04:28:00.893 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2021-02-10T04:28:00.893 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_dump_corrupt_message_level": "1", 2021-02-10T04:28:00.893 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_dump_on_send": "false", 2021-02-10T04:28:00.894 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_initial_backoff": "0.200000", 2021-02-10T04:28:00.894 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_inject_delay_max": "1.000000", 2021-02-10T04:28:00.894 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_inject_delay_probability": "0.000000", 2021-02-10T04:28:00.894 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_inject_delay_type": "", 2021-02-10T04:28:00.894 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_inject_internal_delays": "0.000000", 2021-02-10T04:28:00.894 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_inject_socket_failures": "0", 2021-02-10T04:28:00.894 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_learn_addr_from_peer": "true", 2021-02-10T04:28:00.894 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_max_accept_failures": "4", 2021-02-10T04:28:00.894 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_max_backoff": "15.000000", 2021-02-10T04:28:00.894 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_mon_client_mode": "secure crc", 2021-02-10T04:28:00.894 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_mon_cluster_mode": "secure crc", 2021-02-10T04:28:00.894 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_mon_service_mode": "secure crc", 2021-02-10T04:28:00.894 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2021-02-10T04:28:00.895 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_pq_min_cost": "65536", 2021-02-10T04:28:00.895 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_public_type": "", 2021-02-10T04:28:00.895 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_service_mode": "crc secure", 2021-02-10T04:28:00.895 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_tcp_listen_backlog": "512", 2021-02-10T04:28:00.895 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_tcp_nodelay": "true", 2021-02-10T04:28:00.895 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_tcp_prefetch_max_size": "4096", 2021-02-10T04:28:00.895 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_tcp_rcvbuf": "0", 2021-02-10T04:28:00.895 INFO:teuthology.orchestra.run.smithi085.stdout: "ms_type": "async+posix", 2021-02-10T04:28:00.895 INFO:teuthology.orchestra.run.smithi085.stdout: "no_config_file": "false", 2021-02-10T04:28:00.895 INFO:teuthology.orchestra.run.smithi085.stdout: "objecter_completion_locks_per_session": "32", 2021-02-10T04:28:00.895 INFO:teuthology.orchestra.run.smithi085.stdout: "objecter_debug_inject_relock_delay": "false", 2021-02-10T04:28:00.895 INFO:teuthology.orchestra.run.smithi085.stdout: "objecter_inflight_op_bytes": "104857600", 2021-02-10T04:28:00.895 INFO:teuthology.orchestra.run.smithi085.stdout: "objecter_inflight_ops": "1024", 2021-02-10T04:28:00.896 INFO:teuthology.orchestra.run.smithi085.stdout: "objecter_inject_no_watch_ping": "false", 2021-02-10T04:28:00.896 INFO:teuthology.orchestra.run.smithi085.stdout: "objecter_retry_writes_after_first_reply": "false", 2021-02-10T04:28:00.896 INFO:teuthology.orchestra.run.smithi085.stdout: "objecter_tick_interval": "5.000000", 2021-02-10T04:28:00.896 INFO:teuthology.orchestra.run.smithi085.stdout: "objecter_timeout": "10.000000", 2021-02-10T04:28:00.896 INFO:teuthology.orchestra.run.smithi085.stdout: "objectstore_blackhole": "false", 2021-02-10T04:28:00.896 INFO:teuthology.orchestra.run.smithi085.stdout: "openssl_engine_opts": "", 2021-02-10T04:28:00.896 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_agent_delay_time": "5.000000", 2021-02-10T04:28:00.896 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_agent_hist_halflife": "1000", 2021-02-10T04:28:00.896 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_agent_max_low_ops": "2", 2021-02-10T04:28:00.896 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_agent_max_ops": "4", 2021-02-10T04:28:00.896 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_agent_min_evict_effort": "0.100000", 2021-02-10T04:28:00.896 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_agent_quantize_effort": "0.100000", 2021-02-10T04:28:00.897 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_agent_slop": "0.020000", 2021-02-10T04:28:00.897 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_allow_recovery_below_min_size": "true", 2021-02-10T04:28:00.897 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_asio_thread_count": "2", 2021-02-10T04:28:00.897 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_async_recovery_min_cost": "100", 2021-02-10T04:28:00.897 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_auto_mark_unfound_lost": "false", 2021-02-10T04:28:00.897 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_backfill_retry_interval": "30.000000", 2021-02-10T04:28:00.897 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_backfill_scan_max": "512", 2021-02-10T04:28:00.897 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_backfill_scan_min": "64", 2021-02-10T04:28:00.897 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_backoff_on_degraded": "false", 2021-02-10T04:28:00.897 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_backoff_on_peering": "false", 2021-02-10T04:28:00.897 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_backoff_on_unfound": "true", 2021-02-10T04:28:00.897 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_beacon_report_interval": "300", 2021-02-10T04:28:00.897 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_bench_duration": "30", 2021-02-10T04:28:00.898 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_bench_large_size_max_throughput": "104857600", 2021-02-10T04:28:00.898 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_bench_max_block_size": "67108864", 2021-02-10T04:28:00.898 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_bench_small_size_max_iops": "100", 2021-02-10T04:28:00.898 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_blkin_trace_all": "false", 2021-02-10T04:28:00.898 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2021-02-10T04:28:00.898 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2021-02-10T04:28:00.898 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_check_for_log_corruption": "false", 2021-02-10T04:28:00.898 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_check_max_object_name_len_on_startup": "true", 2021-02-10T04:28:00.898 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2021-02-10T04:28:00.898 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2021-02-10T04:28:00.898 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2021-02-10T04:28:00.898 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_class_update_on_start": "true", 2021-02-10T04:28:00.898 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_client_message_cap": "0", 2021-02-10T04:28:00.899 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_client_message_size_cap": "524288000", 2021-02-10T04:28:00.899 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_client_op_priority": "63", 2021-02-10T04:28:00.899 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_client_watch_timeout": "30", 2021-02-10T04:28:00.899 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_command_max_records": "256", 2021-02-10T04:28:00.899 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_command_thread_suicide_timeout": "900", 2021-02-10T04:28:00.899 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_command_thread_timeout": "600", 2021-02-10T04:28:00.899 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_compact_on_start": "false", 2021-02-10T04:28:00.899 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_copyfrom_max_chunk": "8388608", 2021-02-10T04:28:00.899 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_crush_chooseleaf_type": "1", 2021-02-10T04:28:00.899 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_crush_initial_weight": "-1.000000", 2021-02-10T04:28:00.899 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_crush_update_on_start": "true", 2021-02-10T04:28:00.899 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_crush_update_weight_set": "true", 2021-02-10T04:28:00.899 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2021-02-10T04:28:00.900 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2021-02-10T04:28:00.900 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2021-02-10T04:28:00.900 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_debug_drop_ping_duration": "0", 2021-02-10T04:28:00.900 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_debug_drop_ping_probability": "0.000000", 2021-02-10T04:28:00.900 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_debug_feed_pullee": "-1", 2021-02-10T04:28:00.900 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_debug_inject_copyfrom_error": "false", 2021-02-10T04:28:00.900 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2021-02-10T04:28:00.900 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2021-02-10T04:28:00.900 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_debug_misdirected_ops": "false", 2021-02-10T04:28:00.900 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_debug_no_acting_change": "false", 2021-02-10T04:28:00.900 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_debug_no_purge_strays": "false", 2021-02-10T04:28:00.900 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_debug_op_order": "false", 2021-02-10T04:28:00.900 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_debug_pg_log_writeout": "false", 2021-02-10T04:28:00.901 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_debug_pretend_recovery_active": "false", 2021-02-10T04:28:00.901 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_debug_random_push_read_error": "0.000000", 2021-02-10T04:28:00.901 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2021-02-10T04:28:00.901 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_debug_shutdown": "false", 2021-02-10T04:28:00.901 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2021-02-10T04:28:00.901 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2021-02-10T04:28:00.901 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_debug_verify_cached_snaps": "false", 2021-02-10T04:28:00.901 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_debug_verify_missing_on_start": "false", 2021-02-10T04:28:00.901 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_debug_verify_snaps": "false", 2021-02-10T04:28:00.901 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_debug_verify_stray_on_activate": "false", 2021-02-10T04:28:00.901 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_deep_scrub_interval": "604800.000000", 2021-02-10T04:28:00.901 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_deep_scrub_keys": "1024", 2021-02-10T04:28:00.901 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2021-02-10T04:28:00.902 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2021-02-10T04:28:00.902 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2021-02-10T04:28:00.902 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_deep_scrub_stride": "524288", 2021-02-10T04:28:00.902 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2021-02-10T04:28:00.902 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_default_data_pool_replay_window": "45", 2021-02-10T04:28:00.902 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_default_notify_timeout": "30", 2021-02-10T04:28:00.902 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_delete_sleep": "0.000000", 2021-02-10T04:28:00.902 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_delete_sleep_hdd": "5.000000", 2021-02-10T04:28:00.902 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_delete_sleep_hybrid": "1.000000", 2021-02-10T04:28:00.902 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_delete_sleep_ssd": "1.000000", 2021-02-10T04:28:00.902 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_discard_disconnected_ops": "true", 2021-02-10T04:28:00.902 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_enable_op_tracker": "true", 2021-02-10T04:28:00.903 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_erasure_code_plugins": "jerasure lrc isa", 2021-02-10T04:28:00.903 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_failsafe_full_ratio": "0.970000", 2021-02-10T04:28:00.903 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_fast_fail_on_connection_refused": "true", 2021-02-10T04:28:00.903 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_fast_info": "true", 2021-02-10T04:28:00.903 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_fast_shutdown": "true", 2021-02-10T04:28:00.903 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_find_best_info_ignore_history_les": "false", 2021-02-10T04:28:00.903 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_force_auth_primary_missing_objects": "100", 2021-02-10T04:28:00.903 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2021-02-10T04:28:00.903 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_function_tracing": "false", 2021-02-10T04:28:00.903 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_heartbeat_grace": "20", 2021-02-10T04:28:00.903 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_heartbeat_interval": "6", 2021-02-10T04:28:00.903 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2021-02-10T04:28:00.903 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_heartbeat_min_peers": "10", 2021-02-10T04:28:00.903 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_heartbeat_min_size": "2000", 2021-02-10T04:28:00.904 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_heartbeat_stale": "600", 2021-02-10T04:28:00.904 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2021-02-10T04:28:00.904 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_hit_set_max_size": "100000", 2021-02-10T04:28:00.904 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_hit_set_min_size": "1000", 2021-02-10T04:28:00.904 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_hit_set_namespace": ".ceph-internal", 2021-02-10T04:28:00.904 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_ignore_stale_divergent_priors": "false", 2021-02-10T04:28:00.904 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2021-02-10T04:28:00.904 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_inject_failure_on_pg_removal": "false", 2021-02-10T04:28:00.904 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2021-02-10T04:28:00.904 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_journal_flush_on_shutdown": "true", 2021-02-10T04:28:00.904 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_journal_size": "5120", 2021-02-10T04:28:00.904 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_kill_backfill_at": "0", 2021-02-10T04:28:00.904 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_loop_before_reset_tphandle": "64", 2021-02-10T04:28:00.905 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_map_cache_size": "50", 2021-02-10T04:28:00.905 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_map_dedup": "true", 2021-02-10T04:28:00.905 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_map_message_max": "40", 2021-02-10T04:28:00.905 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_map_message_max_bytes": "10485760", 2021-02-10T04:28:00.905 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_map_share_max_epochs": "40", 2021-02-10T04:28:00.905 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_max_attr_name_len": "100", 2021-02-10T04:28:00.905 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_max_attr_size": "0", 2021-02-10T04:28:00.905 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_max_backfills": "1", 2021-02-10T04:28:00.905 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_max_markdown_count": "5", 2021-02-10T04:28:00.905 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_max_markdown_period": "600", 2021-02-10T04:28:00.905 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_max_object_name_len": "2048", 2021-02-10T04:28:00.905 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_max_object_namespace_len": "256", 2021-02-10T04:28:00.905 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_max_object_size": "134217728", 2021-02-10T04:28:00.906 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2021-02-10T04:28:00.906 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_max_omap_entries_per_request": "1024", 2021-02-10T04:28:00.906 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_max_pg_blocked_by": "16", 2021-02-10T04:28:00.906 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_max_pg_log_entries": "10000", 2021-02-10T04:28:00.906 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2021-02-10T04:28:00.906 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_max_pgls": "1024", 2021-02-10T04:28:00.906 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_max_push_cost": "8388608", 2021-02-10T04:28:00.906 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_max_push_objects": "10", 2021-02-10T04:28:00.906 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_max_scrubs": "1", 2021-02-10T04:28:00.906 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2021-02-10T04:28:00.906 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_max_trimming_pgs": "2", 2021-02-10T04:28:00.906 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_max_write_op_reply_len": "32", 2021-02-10T04:28:00.906 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_max_write_size": "90", 2021-02-10T04:28:00.907 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_mclock_cost_per_io_msec": "0", 2021-02-10T04:28:00.907 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_mclock_cost_per_io_msec_hdd": "0", 2021-02-10T04:28:00.907 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_mclock_cost_per_io_msec_ssd": "0", 2021-02-10T04:28:00.907 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_mclock_max_capacity_iops": "0.000000", 2021-02-10T04:28:00.907 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_mclock_max_capacity_iops_hdd": "10000.000000", 2021-02-10T04:28:00.907 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2021-02-10T04:28:00.907 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_mclock_profile": "balanced", 2021-02-10T04:28:00.907 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2021-02-10T04:28:00.907 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_mclock_scheduler_background_best_effort_lim": "999999", 2021-02-10T04:28:00.907 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_mclock_scheduler_background_best_effort_res": "1", 2021-02-10T04:28:00.907 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2021-02-10T04:28:00.907 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_mclock_scheduler_background_recovery_lim": "999999", 2021-02-10T04:28:00.907 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_mclock_scheduler_background_recovery_res": "1", 2021-02-10T04:28:00.908 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2021-02-10T04:28:00.908 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_mclock_scheduler_client_lim": "999999", 2021-02-10T04:28:00.908 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_mclock_scheduler_client_res": "1", 2021-02-10T04:28:00.908 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_mclock_scheduler_client_wgt": "1", 2021-02-10T04:28:00.908 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_memory_base": "805306368", 2021-02-10T04:28:00.908 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_memory_cache_min": "134217728", 2021-02-10T04:28:00.908 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_memory_cache_resize_interval": "1.000000", 2021-02-10T04:28:00.908 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_memory_expected_fragmentation": "0.150000", 2021-02-10T04:28:00.908 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_memory_target": "4294967296", 2021-02-10T04:28:00.908 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2021-02-10T04:28:00.908 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_min_pg_log_entries": "250", 2021-02-10T04:28:00.908 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_min_recovery_priority": "0", 2021-02-10T04:28:00.908 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_mon_heartbeat_interval": "30", 2021-02-10T04:28:00.908 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2021-02-10T04:28:00.909 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_mon_report_interval": "5", 2021-02-10T04:28:00.909 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_mon_report_max_in_flight": "2", 2021-02-10T04:28:00.909 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_mon_shutdown_timeout": "5.000000", 2021-02-10T04:28:00.909 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_num_cache_shards": "32", 2021-02-10T04:28:00.909 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_num_op_tracker_shard": "32", 2021-02-10T04:28:00.909 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_numa_auto_affinity": "true", 2021-02-10T04:28:00.909 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_numa_node": "-1", 2021-02-10T04:28:00.909 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_numa_prefer_iface": "true", 2021-02-10T04:28:00.909 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_object_clean_region_max_num_intervals": "10", 2021-02-10T04:28:00.909 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_objecter_finishers": "1", 2021-02-10T04:28:00.909 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_objectstore": "bluestore", 2021-02-10T04:28:00.909 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_objectstore_fuse": "false", 2021-02-10T04:28:00.909 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_objectstore_tracing": "false", 2021-02-10T04:28:00.909 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_op_complaint_time": "30.000000", 2021-02-10T04:28:00.910 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_op_history_duration": "600", 2021-02-10T04:28:00.910 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_op_history_size": "20", 2021-02-10T04:28:00.910 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_op_history_slow_op_size": "20", 2021-02-10T04:28:00.910 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2021-02-10T04:28:00.910 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_op_log_threshold": "5", 2021-02-10T04:28:00.910 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_op_num_shards": "0", 2021-02-10T04:28:00.910 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_op_num_shards_hdd": "5", 2021-02-10T04:28:00.910 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_op_num_shards_ssd": "8", 2021-02-10T04:28:00.910 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_op_num_threads_per_shard": "0", 2021-02-10T04:28:00.910 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2021-02-10T04:28:00.910 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2021-02-10T04:28:00.910 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2021-02-10T04:28:00.911 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_op_pq_min_cost": "65536", 2021-02-10T04:28:00.911 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_op_queue": "wpq", 2021-02-10T04:28:00.911 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_op_queue_cut_off": "high", 2021-02-10T04:28:00.911 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_op_thread_suicide_timeout": "150", 2021-02-10T04:28:00.911 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_op_thread_timeout": "15", 2021-02-10T04:28:00.911 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_open_classes_on_start": "true", 2021-02-10T04:28:00.911 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_os_flags": "0", 2021-02-10T04:28:00.911 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_peering_op_priority": "255", 2021-02-10T04:28:00.911 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pg_delete_cost": "1048576", 2021-02-10T04:28:00.911 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pg_delete_priority": "5", 2021-02-10T04:28:00.911 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2021-02-10T04:28:00.911 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2021-02-10T04:28:00.911 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pg_log_dups_tracked": "3000", 2021-02-10T04:28:00.912 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pg_log_trim_max": "10000", 2021-02-10T04:28:00.912 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pg_log_trim_min": "100", 2021-02-10T04:28:00.912 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2021-02-10T04:28:00.912 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pg_object_context_cache_count": "64", 2021-02-10T04:28:00.912 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pg_stat_report_interval_max": "500", 2021-02-10T04:28:00.912 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2021-02-10T04:28:00.912 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pool_default_cache_min_evict_age": "0", 2021-02-10T04:28:00.912 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pool_default_cache_min_flush_age": "0", 2021-02-10T04:28:00.912 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2021-02-10T04:28:00.912 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2021-02-10T04:28:00.912 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2021-02-10T04:28:00.912 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pool_default_crush_rule": "-1", 2021-02-10T04:28:00.912 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pool_default_ec_fast_read": "false", 2021-02-10T04:28:00.913 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd", 2021-02-10T04:28:00.913 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pool_default_flag_hashpspool": "true", 2021-02-10T04:28:00.913 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pool_default_flag_nodelete": "false", 2021-02-10T04:28:00.913 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pool_default_flag_nopgchange": "false", 2021-02-10T04:28:00.913 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pool_default_flag_nosizechange": "false", 2021-02-10T04:28:00.913 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pool_default_flags": "0", 2021-02-10T04:28:00.913 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2021-02-10T04:28:00.913 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pool_default_min_size": "0", 2021-02-10T04:28:00.913 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2021-02-10T04:28:00.913 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pool_default_pg_num": "32", 2021-02-10T04:28:00.913 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pool_default_pgp_num": "0", 2021-02-10T04:28:00.913 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2021-02-10T04:28:00.913 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pool_default_size": "3", 2021-02-10T04:28:00.914 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pool_default_type": "replicated", 2021-02-10T04:28:00.914 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2021-02-10T04:28:00.914 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_pool_use_gmt_hitset": "true", 2021-02-10T04:28:00.914 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_push_per_object_cost": "1000", 2021-02-10T04:28:00.914 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_read_ec_check_for_errors": "false", 2021-02-10T04:28:00.914 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_recover_clone_overlap": "true", 2021-02-10T04:28:00.914 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_recover_clone_overlap_limit": "10", 2021-02-10T04:28:00.914 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_recovery_cost": "20971520", 2021-02-10T04:28:00.914 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_recovery_delay_start": "0.000000", 2021-02-10T04:28:00.914 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_recovery_max_active": "0", 2021-02-10T04:28:00.914 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_recovery_max_active_hdd": "3", 2021-02-10T04:28:00.914 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_recovery_max_active_ssd": "10", 2021-02-10T04:28:00.914 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_recovery_max_chunk": "8388608", 2021-02-10T04:28:00.915 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2021-02-10T04:28:00.915 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_recovery_max_single_start": "1", 2021-02-10T04:28:00.915 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_recovery_op_priority": "3", 2021-02-10T04:28:00.915 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_recovery_op_warn_multiple": "16", 2021-02-10T04:28:00.915 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_recovery_priority": "5", 2021-02-10T04:28:00.915 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_recovery_retry_interval": "30.000000", 2021-02-10T04:28:00.915 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_recovery_sleep": "0.000000", 2021-02-10T04:28:00.915 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_recovery_sleep_hdd": "0.100000", 2021-02-10T04:28:00.915 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2021-02-10T04:28:00.915 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_recovery_sleep_ssd": "0.000000", 2021-02-10T04:28:00.915 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_repair_during_recovery": "false", 2021-02-10T04:28:00.915 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_requested_scrub_priority": "120", 2021-02-10T04:28:00.915 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_rollback_to_cluster_snap": "", 2021-02-10T04:28:00.916 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_scrub_auto_repair": "false", 2021-02-10T04:28:00.916 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_scrub_auto_repair_num_errors": "5", 2021-02-10T04:28:00.916 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_scrub_backoff_ratio": "0.660000", 2021-02-10T04:28:00.916 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_scrub_begin_hour": "0", 2021-02-10T04:28:00.916 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_scrub_begin_week_day": "0", 2021-02-10T04:28:00.916 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_scrub_chunk_max": "25", 2021-02-10T04:28:00.916 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_scrub_chunk_min": "5", 2021-02-10T04:28:00.916 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_scrub_cost": "52428800", 2021-02-10T04:28:00.916 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_scrub_during_recovery": "false", 2021-02-10T04:28:00.916 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_scrub_end_hour": "0", 2021-02-10T04:28:00.916 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_scrub_end_week_day": "0", 2021-02-10T04:28:00.916 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_scrub_extended_sleep": "0.000000", 2021-02-10T04:28:00.916 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2021-02-10T04:28:00.917 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_scrub_invalid_stats": "true", 2021-02-10T04:28:00.917 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_scrub_load_threshold": "0.500000", 2021-02-10T04:28:00.917 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_scrub_max_interval": "604800.000000", 2021-02-10T04:28:00.917 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_scrub_max_preemptions": "5", 2021-02-10T04:28:00.917 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_scrub_min_interval": "86400.000000", 2021-02-10T04:28:00.917 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_scrub_priority": "5", 2021-02-10T04:28:00.917 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_scrub_sleep": "0.000000", 2021-02-10T04:28:00.917 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_shutdown_pgref_assert": "false", 2021-02-10T04:28:00.917 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_skip_data_digest": "false", 2021-02-10T04:28:00.917 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_smart_report_timeout": "5", 2021-02-10T04:28:00.917 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_snap_trim_cost": "1048576", 2021-02-10T04:28:00.917 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_snap_trim_priority": "5", 2021-02-10T04:28:00.917 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_snap_trim_sleep": "0.000000", 2021-02-10T04:28:00.917 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2021-02-10T04:28:00.918 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2021-02-10T04:28:00.918 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2021-02-10T04:28:00.918 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2021-02-10T04:28:00.918 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_target_transaction_size": "30", 2021-02-10T04:28:00.918 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_tier_default_cache_hit_set_count": "4", 2021-02-10T04:28:00.918 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2021-02-10T04:28:00.918 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2021-02-10T04:28:00.918 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2021-02-10T04:28:00.918 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2021-02-10T04:28:00.918 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2021-02-10T04:28:00.918 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2021-02-10T04:28:00.918 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_tier_default_cache_mode": "writeback", 2021-02-10T04:28:00.919 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2021-02-10T04:28:00.919 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_tier_promote_max_objects_sec": "25", 2021-02-10T04:28:00.919 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_tracing": "false", 2021-02-10T04:28:00.919 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_use_stale_snap": "false", 2021-02-10T04:28:00.919 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2021-02-10T04:28:00.919 INFO:teuthology.orchestra.run.smithi085.stdout: "osdc_blkin_trace_all": "false", 2021-02-10T04:28:00.919 INFO:teuthology.orchestra.run.smithi085.stdout: "paxos_kill_at": "0", 2021-02-10T04:28:00.919 INFO:teuthology.orchestra.run.smithi085.stdout: "paxos_max_join_drift": "10", 2021-02-10T04:28:00.919 INFO:teuthology.orchestra.run.smithi085.stdout: "paxos_min": "500", 2021-02-10T04:28:00.919 INFO:teuthology.orchestra.run.smithi085.stdout: "paxos_min_wait": "0.050000", 2021-02-10T04:28:00.919 INFO:teuthology.orchestra.run.smithi085.stdout: "paxos_propose_interval": "1.000000", 2021-02-10T04:28:00.919 INFO:teuthology.orchestra.run.smithi085.stdout: "paxos_service_trim_max": "500", 2021-02-10T04:28:00.919 INFO:teuthology.orchestra.run.smithi085.stdout: "paxos_service_trim_min": "10", 2021-02-10T04:28:00.919 INFO:teuthology.orchestra.run.smithi085.stdout: "paxos_stash_full_interval": "25", 2021-02-10T04:28:00.920 INFO:teuthology.orchestra.run.smithi085.stdout: "paxos_trim_max": "500", 2021-02-10T04:28:00.920 INFO:teuthology.orchestra.run.smithi085.stdout: "paxos_trim_min": "250", 2021-02-10T04:28:00.920 INFO:teuthology.orchestra.run.smithi085.stdout: "perf": "true", 2021-02-10T04:28:00.920 INFO:teuthology.orchestra.run.smithi085.stdout: "pid_file": "", 2021-02-10T04:28:00.920 INFO:teuthology.orchestra.run.smithi085.stdout: "plugin_crypto_accelerator": "crypto_isal", 2021-02-10T04:28:00.920 INFO:teuthology.orchestra.run.smithi085.stdout: "plugin_dir": "/usr/lib64/ceph", 2021-02-10T04:28:00.920 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "-", 2021-02-10T04:28:00.920 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrv": "", 2021-02-10T04:28:00.920 INFO:teuthology.orchestra.run.smithi085.stdout: "public_bind_addr": "-", 2021-02-10T04:28:00.920 INFO:teuthology.orchestra.run.smithi085.stdout: "public_network": "", 2021-02-10T04:28:00.920 INFO:teuthology.orchestra.run.smithi085.stdout: "public_network_interface": "", 2021-02-10T04:28:00.920 INFO:teuthology.orchestra.run.smithi085.stdout: "qat_compressor_enabled": "false", 2021-02-10T04:28:00.920 INFO:teuthology.orchestra.run.smithi085.stdout: "rados_mon_op_timeout": "0", 2021-02-10T04:28:00.921 INFO:teuthology.orchestra.run.smithi085.stdout: "rados_osd_op_timeout": "0", 2021-02-10T04:28:00.921 INFO:teuthology.orchestra.run.smithi085.stdout: "rados_tracing": "false", 2021-02-10T04:28:00.921 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_atime_update_interval": "60", 2021-02-10T04:28:00.921 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2021-02-10T04:28:00.921 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_balance_parent_reads": "false", 2021-02-10T04:28:00.921 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_balance_snap_reads": "false", 2021-02-10T04:28:00.921 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_blkin_trace_all": "false", 2021-02-10T04:28:00.921 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_blocklist_expire_seconds": "0", 2021-02-10T04:28:00.921 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_blocklist_on_break_lock": "true", 2021-02-10T04:28:00.921 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_cache": "true", 2021-02-10T04:28:00.921 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_cache_block_writes_upfront": "false", 2021-02-10T04:28:00.921 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_cache_max_dirty": "25165824", 2021-02-10T04:28:00.921 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_cache_max_dirty_age": "1.000000", 2021-02-10T04:28:00.922 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_cache_max_dirty_object": "0", 2021-02-10T04:28:00.922 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_cache_policy": "writearound", 2021-02-10T04:28:00.922 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_cache_size": "33554432", 2021-02-10T04:28:00.922 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_cache_target_dirty": "16777216", 2021-02-10T04:28:00.922 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_cache_writethrough_until_flush": "true", 2021-02-10T04:28:00.922 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_clone_copy_on_read": "false", 2021-02-10T04:28:00.922 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_compression_hint": "none", 2021-02-10T04:28:00.922 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_concurrent_management_ops": "10", 2021-02-10T04:28:00.922 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_config_pool_override_update_timestamp": "0", 2021-02-10T04:28:00.922 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_default_clone_format": "auto", 2021-02-10T04:28:00.922 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_default_data_pool": "", 2021-02-10T04:28:00.922 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_default_features": "61", 2021-02-10T04:28:00.922 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_default_format": "2", 2021-02-10T04:28:00.923 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_default_map_options": "", 2021-02-10T04:28:00.923 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_default_order": "22", 2021-02-10T04:28:00.923 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_default_pool": "rbd", 2021-02-10T04:28:00.923 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2021-02-10T04:28:00.923 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_default_stripe_count": "0", 2021-02-10T04:28:00.923 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_default_stripe_unit": "0", 2021-02-10T04:28:00.923 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_disable_zero_copy_writes": "true", 2021-02-10T04:28:00.923 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_discard_granularity_bytes": "65536", 2021-02-10T04:28:00.923 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_discard_on_zeroed_write_same": "true", 2021-02-10T04:28:00.923 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_enable_alloc_hint": "true", 2021-02-10T04:28:00.923 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2021-02-10T04:28:00.923 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_io_scheduler": "simple", 2021-02-10T04:28:00.923 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2021-02-10T04:28:00.924 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_journal_commit_age": "5.000000", 2021-02-10T04:28:00.924 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2021-02-10T04:28:00.924 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_journal_max_payload_bytes": "16384", 2021-02-10T04:28:00.924 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_journal_object_flush_age": "0.000000", 2021-02-10T04:28:00.924 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_journal_object_flush_bytes": "1048576", 2021-02-10T04:28:00.924 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_journal_object_flush_interval": "0", 2021-02-10T04:28:00.924 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2021-02-10T04:28:00.924 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2021-02-10T04:28:00.924 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_journal_order": "24", 2021-02-10T04:28:00.924 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_journal_pool": "", 2021-02-10T04:28:00.924 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_journal_splay_width": "4", 2021-02-10T04:28:00.924 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_localize_parent_reads": "false", 2021-02-10T04:28:00.924 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_localize_snap_reads": "false", 2021-02-10T04:28:00.925 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2021-02-10T04:28:00.925 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2021-02-10T04:28:00.925 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2021-02-10T04:28:00.925 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2021-02-10T04:28:00.925 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2021-02-10T04:28:00.925 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2021-02-10T04:28:00.925 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_mirror_image_policy_type": "simple", 2021-02-10T04:28:00.925 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2021-02-10T04:28:00.925 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_mirror_image_state_check_interval": "30", 2021-02-10T04:28:00.925 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2021-02-10T04:28:00.925 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2021-02-10T04:28:00.925 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2021-02-10T04:28:00.925 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2021-02-10T04:28:00.926 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2021-02-10T04:28:00.926 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_mirror_memory_autotune": "true", 2021-02-10T04:28:00.926 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_mirror_memory_base": "805306368", 2021-02-10T04:28:00.926 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2021-02-10T04:28:00.926 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_mirror_memory_cache_min": "134217728", 2021-02-10T04:28:00.926 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2021-02-10T04:28:00.926 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2021-02-10T04:28:00.926 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_mirror_memory_target": "4294967296", 2021-02-10T04:28:00.926 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_mirror_perf_stats_prio": "5", 2021-02-10T04:28:00.926 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2021-02-10T04:28:00.926 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2021-02-10T04:28:00.926 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_mirroring_delete_delay": "0", 2021-02-10T04:28:00.927 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_mirroring_max_mirroring_snapshots": "3", 2021-02-10T04:28:00.927 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_mirroring_replay_delay": "0", 2021-02-10T04:28:00.927 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2021-02-10T04:28:00.927 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2021-02-10T04:28:00.927 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_move_to_trash_on_remove": "false", 2021-02-10T04:28:00.927 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2021-02-10T04:28:00.927 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_mtime_update_interval": "60", 2021-02-10T04:28:00.927 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_non_blocking_aio": "true", 2021-02-10T04:28:00.927 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_op_thread_timeout": "60", 2021-02-10T04:28:00.927 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_op_threads": "1", 2021-02-10T04:28:00.927 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_parent_cache_enabled": "false", 2021-02-10T04:28:00.927 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_persistent_cache_log_periodic_stats": "false", 2021-02-10T04:28:00.927 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_persistent_cache_mode": "disabled", 2021-02-10T04:28:00.928 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_persistent_cache_path": "/tmp", 2021-02-10T04:28:00.928 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_persistent_cache_size": "1073741824", 2021-02-10T04:28:00.928 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_plugins": "", 2021-02-10T04:28:00.928 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_qos_bps_burst": "0", 2021-02-10T04:28:00.928 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_qos_bps_burst_seconds": "1", 2021-02-10T04:28:00.928 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_qos_bps_limit": "0", 2021-02-10T04:28:00.928 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_qos_exclude_ops": "0", 2021-02-10T04:28:00.928 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_qos_iops_burst": "0", 2021-02-10T04:28:00.928 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_qos_iops_burst_seconds": "1", 2021-02-10T04:28:00.928 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_qos_iops_limit": "0", 2021-02-10T04:28:00.928 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_qos_read_bps_burst": "0", 2021-02-10T04:28:00.928 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2021-02-10T04:28:00.928 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_qos_read_bps_limit": "0", 2021-02-10T04:28:00.929 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_qos_read_iops_burst": "0", 2021-02-10T04:28:00.929 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2021-02-10T04:28:00.929 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_qos_read_iops_limit": "0", 2021-02-10T04:28:00.929 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_qos_schedule_tick_min": "50", 2021-02-10T04:28:00.929 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_qos_write_bps_burst": "0", 2021-02-10T04:28:00.929 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2021-02-10T04:28:00.929 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_qos_write_bps_limit": "0", 2021-02-10T04:28:00.929 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_qos_write_iops_burst": "0", 2021-02-10T04:28:00.929 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2021-02-10T04:28:00.929 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_qos_write_iops_limit": "0", 2021-02-10T04:28:00.929 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_quiesce_notification_attempts": "10", 2021-02-10T04:28:00.929 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_read_from_replica_policy": "default", 2021-02-10T04:28:00.929 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2021-02-10T04:28:00.930 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_readahead_max_bytes": "524288", 2021-02-10T04:28:00.930 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_readahead_trigger_requests": "10", 2021-02-10T04:28:00.930 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_request_timed_out_seconds": "30", 2021-02-10T04:28:00.930 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_skip_partial_discard": "true", 2021-02-10T04:28:00.930 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2021-02-10T04:28:00.930 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_tracing": "false", 2021-02-10T04:28:00.930 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_validate_names": "true", 2021-02-10T04:28:00.930 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd_validate_pool": "true", 2021-02-10T04:28:00.930 INFO:teuthology.orchestra.run.smithi085.stdout: "restapi_base_url": "", 2021-02-10T04:28:00.930 INFO:teuthology.orchestra.run.smithi085.stdout: "restapi_log_level": "", 2021-02-10T04:28:00.930 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_acl_grants_max_num": "100", 2021-02-10T04:28:00.930 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_admin_entry": "admin", 2021-02-10T04:28:00.930 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_barbican_url": "", 2021-02-10T04:28:00.931 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2021-02-10T04:28:00.931 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_bucket_default_quota_max_size": "-1", 2021-02-10T04:28:00.931 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_bucket_index_max_aio": "128", 2021-02-10T04:28:00.931 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_bucket_quota_cache_size": "10000", 2021-02-10T04:28:00.931 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_bucket_quota_soft_threshold": "0.950000", 2021-02-10T04:28:00.931 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_bucket_quota_ttl": "600", 2021-02-10T04:28:00.931 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_cache_enabled": "true", 2021-02-10T04:28:00.931 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_cache_expiry_interval": "900", 2021-02-10T04:28:00.931 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_cache_lru_size": "10000", 2021-02-10T04:28:00.931 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_content_length_compat": "false", 2021-02-10T04:28:00.931 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_copy_obj_progress": "true", 2021-02-10T04:28:00.931 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2021-02-10T04:28:00.931 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_cors_rules_max_num": "100", 2021-02-10T04:28:00.932 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_cross_domain_policy": "", 2021-02-10T04:28:00.932 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_crypt_default_encryption_key": "", 2021-02-10T04:28:00.932 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_crypt_require_ssl": "true", 2021-02-10T04:28:00.932 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2021-02-10T04:28:00.932 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2021-02-10T04:28:00.932 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_crypt_suppress_logs": "true", 2021-02-10T04:28:00.932 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_crypt_vault_addr": "", 2021-02-10T04:28:00.932 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_crypt_vault_auth": "token", 2021-02-10T04:28:00.932 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_crypt_vault_namespace": "", 2021-02-10T04:28:00.932 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_crypt_vault_prefix": "", 2021-02-10T04:28:00.932 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_crypt_vault_secret_engine": "transit", 2021-02-10T04:28:00.932 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_crypt_vault_token_file": "", 2021-02-10T04:28:00.932 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_curl_buffersize": "524288", 2021-02-10T04:28:00.933 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_curl_low_speed_limit": "1024", 2021-02-10T04:28:00.933 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_curl_low_speed_time": "300", 2021-02-10T04:28:00.933 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_curl_wait_timeout_ms": "1000", 2021-02-10T04:28:00.933 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2021-02-10T04:28:00.933 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_data_log_backing": "auto", 2021-02-10T04:28:00.933 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_data_log_changes_size": "1000", 2021-02-10T04:28:00.933 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_data_log_num_shards": "128", 2021-02-10T04:28:00.933 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_data_log_obj_prefix": "data_log", 2021-02-10T04:28:00.933 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_data_log_window": "30", 2021-02-10T04:28:00.933 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_data_notify_interval_msec": "200", 2021-02-10T04:28:00.933 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_default_realm_info_oid": "default.realm", 2021-02-10T04:28:00.933 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_default_region_info_oid": "default.region", 2021-02-10T04:28:00.933 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_default_zone_info_oid": "default.zone", 2021-02-10T04:28:00.934 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2021-02-10T04:28:00.934 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_defer_to_bucket_acls": "", 2021-02-10T04:28:00.934 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_delete_multi_obj_max_num": "1000", 2021-02-10T04:28:00.934 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_dmclock_admin_lim": "0.000000", 2021-02-10T04:28:00.934 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_dmclock_admin_res": "100.000000", 2021-02-10T04:28:00.934 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2021-02-10T04:28:00.934 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_dmclock_auth_lim": "0.000000", 2021-02-10T04:28:00.934 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_dmclock_auth_res": "200.000000", 2021-02-10T04:28:00.934 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2021-02-10T04:28:00.934 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_dmclock_data_lim": "0.000000", 2021-02-10T04:28:00.934 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_dmclock_data_res": "500.000000", 2021-02-10T04:28:00.934 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_dmclock_data_wgt": "500.000000", 2021-02-10T04:28:00.934 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2021-02-10T04:28:00.934 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_dmclock_metadata_res": "500.000000", 2021-02-10T04:28:00.935 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2021-02-10T04:28:00.935 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_dns_name": "", 2021-02-10T04:28:00.935 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_dns_s3website_name": "", 2021-02-10T04:28:00.935 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_dynamic_resharding": "true", 2021-02-10T04:28:00.935 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, pubsub", 2021-02-10T04:28:00.935 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_enable_gc_threads": "true", 2021-02-10T04:28:00.935 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_enable_lc_threads": "true", 2021-02-10T04:28:00.935 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_enable_ops_log": "false", 2021-02-10T04:28:00.935 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_enable_quota_threads": "true", 2021-02-10T04:28:00.935 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_enable_static_website": "false", 2021-02-10T04:28:00.935 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_enable_usage_log": "false", 2021-02-10T04:28:00.935 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_enforce_swift_acls": "true", 2021-02-10T04:28:00.935 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_exit_timeout_secs": "120", 2021-02-10T04:28:00.936 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_expose_bucket": "false", 2021-02-10T04:28:00.936 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_extended_http_attrs": "", 2021-02-10T04:28:00.936 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_fcgi_socket_backlog": "1024", 2021-02-10T04:28:00.936 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2021-02-10T04:28:00.936 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_frontends": "beast port=7480", 2021-02-10T04:28:00.936 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_gc_max_concurrent_io": "10", 2021-02-10T04:28:00.936 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_gc_max_deferred": "50", 2021-02-10T04:28:00.936 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2021-02-10T04:28:00.936 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_gc_max_objs": "32", 2021-02-10T04:28:00.936 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_gc_max_queue_size": "134213632", 2021-02-10T04:28:00.936 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_gc_max_trim_chunk": "16", 2021-02-10T04:28:00.936 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_gc_obj_min_wait": "7200", 2021-02-10T04:28:00.936 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_gc_processor_max_time": "3600", 2021-02-10T04:28:00.937 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_gc_processor_period": "3600", 2021-02-10T04:28:00.937 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_get_obj_max_req_size": "4194304", 2021-02-10T04:28:00.937 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_get_obj_window_size": "16777216", 2021-02-10T04:28:00.937 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_healthcheck_disabling_path": "", 2021-02-10T04:28:00.937 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_host": "", 2021-02-10T04:28:00.937 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_ignore_get_invalid_range": "false", 2021-02-10T04:28:00.937 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_init_timeout": "300", 2021-02-10T04:28:00.937 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2021-02-10T04:28:00.937 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_keystone_accepted_admin_roles": "", 2021-02-10T04:28:00.937 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2021-02-10T04:28:00.937 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_keystone_admin_domain": "", 2021-02-10T04:28:00.937 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_keystone_admin_password": "", 2021-02-10T04:28:00.937 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_keystone_admin_password_path": "", 2021-02-10T04:28:00.938 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_keystone_admin_project": "", 2021-02-10T04:28:00.938 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_keystone_admin_tenant": "", 2021-02-10T04:28:00.938 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_keystone_admin_token": "", 2021-02-10T04:28:00.938 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_keystone_admin_token_path": "", 2021-02-10T04:28:00.938 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_keystone_admin_user": "", 2021-02-10T04:28:00.938 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_keystone_api_version": "2", 2021-02-10T04:28:00.938 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_keystone_barbican_domain": "", 2021-02-10T04:28:00.938 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_keystone_barbican_password": "", 2021-02-10T04:28:00.938 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_keystone_barbican_project": "", 2021-02-10T04:28:00.938 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_keystone_barbican_tenant": "", 2021-02-10T04:28:00.938 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_keystone_barbican_user": "", 2021-02-10T04:28:00.938 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_keystone_implicit_tenants": "false", 2021-02-10T04:28:00.938 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_keystone_token_cache_size": "10000", 2021-02-10T04:28:00.939 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_keystone_url": "", 2021-02-10T04:28:00.939 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_keystone_verify_ssl": "true", 2021-02-10T04:28:00.939 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_lc_debug_interval": "-1", 2021-02-10T04:28:00.939 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_lc_lock_max_time": "90", 2021-02-10T04:28:00.939 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_lc_max_objs": "32", 2021-02-10T04:28:00.939 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_lc_max_rules": "1000", 2021-02-10T04:28:00.939 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_lc_max_worker": "3", 2021-02-10T04:28:00.939 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_lc_max_wp_worker": "3", 2021-02-10T04:28:00.939 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_lc_thread_delay": "0", 2021-02-10T04:28:00.939 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2021-02-10T04:28:00.939 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_ldap_dnattr": "uid", 2021-02-10T04:28:00.939 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2021-02-10T04:28:00.939 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_ldap_searchfilter": "", 2021-02-10T04:28:00.940 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2021-02-10T04:28:00.940 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_ldap_uri": "ldaps://", 2021-02-10T04:28:00.940 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2021-02-10T04:28:00.940 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_list_bucket_min_readahead": "1000", 2021-02-10T04:28:00.940 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_list_buckets_max_chunk": "1000", 2021-02-10T04:28:00.940 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_log_http_headers": "", 2021-02-10T04:28:00.940 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_log_nonexistent_bucket": "false", 2021-02-10T04:28:00.940 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2021-02-10T04:28:00.940 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_log_object_name_utc": "false", 2021-02-10T04:28:00.940 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2021-02-10T04:28:00.940 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_max_attr_name_len": "0", 2021-02-10T04:28:00.940 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_max_attr_size": "0", 2021-02-10T04:28:00.940 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_max_attrs_num_in_req": "0", 2021-02-10T04:28:00.941 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_max_chunk_size": "4194304", 2021-02-10T04:28:00.941 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_max_concurrent_requests": "1024", 2021-02-10T04:28:00.941 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_max_dynamic_shards": "1999", 2021-02-10T04:28:00.941 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_max_listing_results": "1000", 2021-02-10T04:28:00.941 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_max_notify_retries": "3", 2021-02-10T04:28:00.941 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_max_objs_per_shard": "100000", 2021-02-10T04:28:00.941 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_max_put_param_size": "1048576", 2021-02-10T04:28:00.941 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_max_put_size": "5368709120", 2021-02-10T04:28:00.941 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_max_slo_entries": "1000", 2021-02-10T04:28:00.941 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_md_log_max_shards": "64", 2021-02-10T04:28:00.941 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_md_notify_interval_msec": "200", 2021-02-10T04:28:00.941 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_mime_types_file": "/etc/mime.types", 2021-02-10T04:28:00.941 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_mp_lock_max_time": "600", 2021-02-10T04:28:00.941 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_multipart_min_part_size": "5242880", 2021-02-10T04:28:00.942 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_multipart_part_upload_limit": "10000", 2021-02-10T04:28:00.942 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_nfs_fhcache_partitions": "3", 2021-02-10T04:28:00.942 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_nfs_fhcache_size": "2017", 2021-02-10T04:28:00.942 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2021-02-10T04:28:00.942 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_nfs_lru_lanes": "5", 2021-02-10T04:28:00.942 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_nfs_max_gc": "300", 2021-02-10T04:28:00.942 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_nfs_namespace_expire_secs": "300", 2021-02-10T04:28:00.942 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_nfs_run_gc_threads": "false", 2021-02-10T04:28:00.942 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_nfs_run_lc_threads": "false", 2021-02-10T04:28:00.942 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_nfs_run_quota_threads": "false", 2021-02-10T04:28:00.942 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_nfs_run_sync_thread": "false", 2021-02-10T04:28:00.942 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_nfs_s3_fast_attrs": "false", 2021-02-10T04:28:00.942 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_nfs_write_completion_interval_s": "10", 2021-02-10T04:28:00.943 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_num_async_rados_threads": "32", 2021-02-10T04:28:00.943 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_num_control_oids": "8", 2021-02-10T04:28:00.943 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_num_rados_handles": "1", 2021-02-10T04:28:00.943 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_numa_node": "-1", 2021-02-10T04:28:00.943 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_obj_stripe_size": "4194304", 2021-02-10T04:28:00.943 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_obj_tombstone_cache_size": "1000", 2021-02-10T04:28:00.943 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_objexp_chunk_size": "100", 2021-02-10T04:28:00.943 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_objexp_gc_interval": "600", 2021-02-10T04:28:00.943 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_objexp_hints_num_shards": "127", 2021-02-10T04:28:00.943 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_olh_pending_timeout_sec": "3600", 2021-02-10T04:28:00.943 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_op_thread_suicide_timeout": "0", 2021-02-10T04:28:00.943 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_op_thread_timeout": "600", 2021-02-10T04:28:00.944 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_op_tracing": "false", 2021-02-10T04:28:00.944 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_opa_token": "", 2021-02-10T04:28:00.944 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_opa_url": "", 2021-02-10T04:28:00.944 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_opa_verify_ssl": "true", 2021-02-10T04:28:00.944 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_ops_log_data_backlog": "5242880", 2021-02-10T04:28:00.944 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_ops_log_rados": "true", 2021-02-10T04:28:00.944 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_ops_log_socket_path": "", 2021-02-10T04:28:00.944 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_override_bucket_index_max_shards": "0", 2021-02-10T04:28:00.944 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2021-02-10T04:28:00.944 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_period_push_interval": "2.000000", 2021-02-10T04:28:00.944 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_period_push_interval_max": "30.000000", 2021-02-10T04:28:00.944 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_period_root_pool": ".rgw.root", 2021-02-10T04:28:00.944 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_port": "", 2021-02-10T04:28:00.944 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_print_continue": "true", 2021-02-10T04:28:00.945 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_print_prohibited_content_length": "false", 2021-02-10T04:28:00.945 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_put_obj_max_window_size": "67108864", 2021-02-10T04:28:00.945 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_put_obj_min_window_size": "16777216", 2021-02-10T04:28:00.945 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2021-02-10T04:28:00.945 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_rados_pool_pg_num_min": "8", 2021-02-10T04:28:00.945 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_rados_pool_recovery_priority": "5", 2021-02-10T04:28:00.945 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_rados_tracing": "false", 2021-02-10T04:28:00.945 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_realm": "", 2021-02-10T04:28:00.945 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_realm_root_pool": ".rgw.root", 2021-02-10T04:28:00.945 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_region": "", 2021-02-10T04:28:00.945 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_region_root_pool": ".rgw.root", 2021-02-10T04:28:00.945 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_relaxed_region_enforcement": "false", 2021-02-10T04:28:00.945 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_relaxed_s3_bucket_names": "false", 2021-02-10T04:28:00.945 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2021-02-10T04:28:00.946 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_request_uri": "", 2021-02-10T04:28:00.946 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_reshard_batch_size": "64", 2021-02-10T04:28:00.946 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_reshard_bucket_lock_duration": "360", 2021-02-10T04:28:00.946 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_reshard_max_aio": "128", 2021-02-10T04:28:00.946 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_reshard_num_logs": "16", 2021-02-10T04:28:00.946 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_reshard_thread_interval": "600", 2021-02-10T04:28:00.946 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_resolve_cname": "false", 2021-02-10T04:28:00.946 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_rest_getusage_op_compat": "false", 2021-02-10T04:28:00.946 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_run_sync_thread": "true", 2021-02-10T04:28:00.946 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_s3_auth_order": "sts, external, local", 2021-02-10T04:28:00.946 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_s3_auth_use_keystone": "false", 2021-02-10T04:28:00.946 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_s3_auth_use_ldap": "false", 2021-02-10T04:28:00.947 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_s3_auth_use_rados": "true", 2021-02-10T04:28:00.947 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_s3_auth_use_sts": "false", 2021-02-10T04:28:00.947 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_s3_success_create_obj_status": "0", 2021-02-10T04:28:00.947 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_safe_max_objects_per_shard": "102400", 2021-02-10T04:28:00.947 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_scheduler_type": "throttler", 2021-02-10T04:28:00.947 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_script_uri": "", 2021-02-10T04:28:00.947 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_service_provider_name": "", 2021-02-10T04:28:00.947 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_shard_warning_threshold": "90.000000", 2021-02-10T04:28:00.947 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_socket_path": "", 2021-02-10T04:28:00.947 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_sts_client_id": "", 2021-02-10T04:28:00.947 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_sts_client_secret": "", 2021-02-10T04:28:00.947 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_sts_entry": "sts", 2021-02-10T04:28:00.947 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_sts_key": "sts", 2021-02-10T04:28:00.947 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_sts_max_session_duration": "43200", 2021-02-10T04:28:00.948 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_sts_min_session_duration": "900", 2021-02-10T04:28:00.948 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_sts_token_introspection_url": "", 2021-02-10T04:28:00.948 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_swift_account_in_url": "false", 2021-02-10T04:28:00.948 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_swift_auth_entry": "auth", 2021-02-10T04:28:00.948 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_swift_auth_url": "", 2021-02-10T04:28:00.948 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_swift_custom_header": "", 2021-02-10T04:28:00.948 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_swift_enforce_content_length": "false", 2021-02-10T04:28:00.948 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_swift_need_stats": "true", 2021-02-10T04:28:00.948 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_swift_tenant_name": "", 2021-02-10T04:28:00.948 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_swift_token_expiration": "86400", 2021-02-10T04:28:00.948 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_swift_url": "", 2021-02-10T04:28:00.948 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_swift_url_prefix": "swift", 2021-02-10T04:28:00.948 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_swift_versioning_enabled": "false", 2021-02-10T04:28:00.949 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2021-02-10T04:28:00.949 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_sync_lease_period": "120", 2021-02-10T04:28:00.949 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2021-02-10T04:28:00.949 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_sync_log_trim_interval": "1200", 2021-02-10T04:28:00.949 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_sync_log_trim_max_buckets": "16", 2021-02-10T04:28:00.949 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2021-02-10T04:28:00.949 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2021-02-10T04:28:00.949 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_sync_obj_etag_verify": "false", 2021-02-10T04:28:00.949 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_sync_trace_history_size": "4096", 2021-02-10T04:28:00.949 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_sync_trace_per_node_log_size": "32", 2021-02-10T04:28:00.949 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2021-02-10T04:28:00.949 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_thread_pool_size": "512", 2021-02-10T04:28:00.949 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_torrent_comment": "", 2021-02-10T04:28:00.950 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_torrent_createby": "", 2021-02-10T04:28:00.950 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_torrent_encoding": "", 2021-02-10T04:28:00.950 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_torrent_flag": "false", 2021-02-10T04:28:00.950 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_torrent_origin": "", 2021-02-10T04:28:00.950 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_torrent_sha_unit": "524288", 2021-02-10T04:28:00.950 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_torrent_tracker": "", 2021-02-10T04:28:00.950 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_trust_forwarded_https": "false", 2021-02-10T04:28:00.950 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_usage_log_flush_threshold": "1024", 2021-02-10T04:28:00.950 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_usage_log_tick_interval": "30", 2021-02-10T04:28:00.950 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_usage_max_shards": "32", 2021-02-10T04:28:00.950 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_usage_max_user_shards": "1", 2021-02-10T04:28:00.950 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_use_opa_authz": "false", 2021-02-10T04:28:00.950 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_user_default_quota_max_objects": "-1", 2021-02-10T04:28:00.950 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_user_default_quota_max_size": "-1", 2021-02-10T04:28:00.951 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_user_max_buckets": "1000", 2021-02-10T04:28:00.951 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2021-02-10T04:28:00.951 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_user_quota_sync_idle_users": "false", 2021-02-10T04:28:00.951 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_user_quota_sync_interval": "86400", 2021-02-10T04:28:00.951 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_user_quota_sync_wait_time": "86400", 2021-02-10T04:28:00.951 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_user_unique_email": "true", 2021-02-10T04:28:00.951 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_verify_ssl": "true", 2021-02-10T04:28:00.951 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_website_routing_rules_max_num": "50", 2021-02-10T04:28:00.951 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_zone": "", 2021-02-10T04:28:00.951 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_zone_root_pool": ".rgw.root", 2021-02-10T04:28:00.951 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_zonegroup": "", 2021-02-10T04:28:00.951 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2021-02-10T04:28:00.951 INFO:teuthology.orchestra.run.smithi085.stdout: "rocksdb_block_size": "4096", 2021-02-10T04:28:00.952 INFO:teuthology.orchestra.run.smithi085.stdout: "rocksdb_bloom_bits_per_key": "20", 2021-02-10T04:28:00.952 INFO:teuthology.orchestra.run.smithi085.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2021-02-10T04:28:00.952 INFO:teuthology.orchestra.run.smithi085.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2021-02-10T04:28:00.952 INFO:teuthology.orchestra.run.smithi085.stdout: "rocksdb_cache_row_ratio": "0.000000", 2021-02-10T04:28:00.952 INFO:teuthology.orchestra.run.smithi085.stdout: "rocksdb_cache_shard_bits": "4", 2021-02-10T04:28:00.952 INFO:teuthology.orchestra.run.smithi085.stdout: "rocksdb_cache_size": "536870912", 2021-02-10T04:28:00.952 INFO:teuthology.orchestra.run.smithi085.stdout: "rocksdb_cache_type": "binned_lru", 2021-02-10T04:28:00.952 INFO:teuthology.orchestra.run.smithi085.stdout: "rocksdb_collect_compaction_stats": "false", 2021-02-10T04:28:00.952 INFO:teuthology.orchestra.run.smithi085.stdout: "rocksdb_collect_extended_stats": "false", 2021-02-10T04:28:00.952 INFO:teuthology.orchestra.run.smithi085.stdout: "rocksdb_collect_memory_stats": "false", 2021-02-10T04:28:00.952 INFO:teuthology.orchestra.run.smithi085.stdout: "rocksdb_delete_range_threshold": "1048576", 2021-02-10T04:28:00.952 INFO:teuthology.orchestra.run.smithi085.stdout: "rocksdb_index_type": "binary_search", 2021-02-10T04:28:00.952 INFO:teuthology.orchestra.run.smithi085.stdout: "rocksdb_log_to_ceph_log": "true", 2021-02-10T04:28:00.953 INFO:teuthology.orchestra.run.smithi085.stdout: "rocksdb_metadata_block_size": "4096", 2021-02-10T04:28:00.953 INFO:teuthology.orchestra.run.smithi085.stdout: "rocksdb_partition_filters": "false", 2021-02-10T04:28:00.953 INFO:teuthology.orchestra.run.smithi085.stdout: "rocksdb_perf": "false", 2021-02-10T04:28:00.953 INFO:teuthology.orchestra.run.smithi085.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2021-02-10T04:28:00.953 INFO:teuthology.orchestra.run.smithi085.stdout: "rotating_keys_bootstrap_timeout": "30", 2021-02-10T04:28:00.953 INFO:teuthology.orchestra.run.smithi085.stdout: "rotating_keys_renewal_timeout": "10", 2021-02-10T04:28:00.953 INFO:teuthology.orchestra.run.smithi085.stdout: "run_dir": "/var/run/ceph", 2021-02-10T04:28:00.953 INFO:teuthology.orchestra.run.smithi085.stdout: "setgroup": "ceph", 2021-02-10T04:28:00.953 INFO:teuthology.orchestra.run.smithi085.stdout: "setuser": "ceph", 2021-02-10T04:28:00.953 INFO:teuthology.orchestra.run.smithi085.stdout: "setuser_match_path": "", 2021-02-10T04:28:00.953 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_misplaced_ratio": "0.050000", 2021-02-10T04:28:00.953 INFO:teuthology.orchestra.run.smithi085.stdout: "thp": "false", 2021-02-10T04:28:00.953 INFO:teuthology.orchestra.run.smithi085.stdout: "threadpool_default_timeout": "60", 2021-02-10T04:28:00.953 INFO:teuthology.orchestra.run.smithi085.stdout: "threadpool_empty_queue_max_wait": "2", 2021-02-10T04:28:00.954 INFO:teuthology.orchestra.run.smithi085.stdout: "throttler_perf_counter": "true" 2021-02-10T04:28:00.954 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:28:01.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:01 smithi164 conmon[29288]: cluster 2021-02-10T04:27:59.982990+0000 mgr.y (mgr.14140) 309 : cluster [DBG] pgmap v266: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:28:01.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:01 smithi164 conmon[29288]: 2021-02-10T04:28:01.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:01 smithi085 conmon[29319]: cluster 2021-02-10T04:27:59.982990+0000 mgr.y 2021-02-10T04:28:01.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:01 smithi085 conmon[29319]: (mgr.14140) 309 : cluster [DBG] pgmap v266: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:28:01.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:01 smithi085 conmon[35961]: cluster 2021-02-10T04:27:59.982990+0000 2021-02-10T04:28:01.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:01 smithi085 conmon[35961]: mgr.y (mgr.14140) 309 : cluster [DBG] pgmap v266: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:28:01.552 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2021-02-10T04:28:01.750 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:28:02.337 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:02 smithi085 conmon[29319]: cluster 2021-02-10T04:28:00.398472+0000 osd.7 (osd 2021-02-10T04:28:02.337 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:02 smithi085 conmon[29319]: .7) 25 : cluster [DBG] 2.1c scrub ok 2021-02-10T04:28:02.337 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:02 smithi085 conmon[35961]: cluster 2021-02-10T04:28:00.398472+0000 2021-02-10T04:28:02.337 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:02 smithi085 conmon[35961]: osd.7 (osd.7) 25 : cluster [DBG] 2.1c scrub ok 2021-02-10T04:28:02.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:02 smithi164 conmon[29288]: cluster 2021-02-10T04:28:00.398472+0000 osd.7 (osd.7) 25 : cluster [DBG] 2021-02-10T04:28:02.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:02 smithi164 conmon[29288]: 2.1c scrub ok 2021-02-10T04:28:03.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:03 smithi085 conmon[29319]: cluster 2021-02-10T04:28:01.983600+0000 mgr.y (mgr.14140 2021-02-10T04:28:03.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:03 smithi085 conmon[29319]: ) 310 : cluster [DBG] pgmap v267: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:28:03.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:03 smithi085 conmon[29319]: cluster 2021-02-10T04:28:02.288674+0000 osd.5 (osd.5) 27 : cluster 2021-02-10T04:28:03.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:03 smithi085 conmon[29319]: [DBG] 5.f scrub ok 2021-02-10T04:28:03.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:03 smithi085 conmon[35961]: cluster 2021-02-10T04:28:01.983600+0000 mgr.y (mgr 2021-02-10T04:28:03.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:03 smithi085 conmon[35961]: .14140) 310 : cluster [DBG] pgmap v267: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:28:03.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:03 smithi085 conmon[35961]: cluster 2021-02-10T04:28:02. 2021-02-10T04:28:03.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:03 smithi085 conmon[35961]: 288674+0000 osd.5 (osd.5) 27 : cluster [DBG] 5.f scrub ok 2021-02-10T04:28:03.348 INFO:teuthology.orchestra.run.smithi085.stdout:mon.a: {} 2021-02-10T04:28:03.357 INFO:teuthology.orchestra.run.smithi085.stdout:mon.c: {} 2021-02-10T04:28:03.367 INFO:teuthology.orchestra.run.smithi085.stdout:mon.b: {} 2021-02-10T04:28:03.368 INFO:teuthology.orchestra.run.smithi085.stderr:mon.a: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2021-02-10T04:28:03.368 INFO:teuthology.orchestra.run.smithi085.stderr:mon.c: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2021-02-10T04:28:03.368 INFO:teuthology.orchestra.run.smithi085.stderr:mon.b: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2021-02-10T04:28:03.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:03 smithi164 conmon[29288]: cluster 2021-02-10T04:28:01.983600+0000 mgr.y (mgr.14140) 310 : cluster [DBG] pgmap v267: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:28:03.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:03 smithi164 conmon[29288]: cluster 2021-02-10T04:28:02.288674+0000 osd.5 2021-02-10T04:28:03.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:03 smithi164 conmon[29288]: (osd.5) 27 : cluster [DBG] 5.f scrub ok 2021-02-10T04:28:04.016 INFO:teuthology.run_tasks:Running task rados... 2021-02-10T04:28:04.038 INFO:tasks.rados:Beginning rados... 2021-02-10T04:28:04.038 DEBUG:teuthology.run_tasks:Unwinding manager rados 2021-02-10T04:28:07.889 INFO:tasks.rados:joining rados 2021-02-10T04:28:07.890 INFO:tasks.thrashosds.thrasher:starting do_thrash 2021-02-10T04:28:07.890 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-02-10T04:28:07.890 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:28:07.890 INFO:tasks.thrashosds.thrasher:Growing pool .rgw.root 2021-02-10T04:28:07.891 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:28:07.891 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2021-02-10T04:28:07.891 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:28:07.892 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2021-02-10T04:28:07.892 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:28:07.892 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2021-02-10T04:28:07.892 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.0 dump_ops_in_flight 2021-02-10T04:28:07.893 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2021-02-10T04:28:07.893 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set noscrub 2021-02-10T04:28:07.893 INFO:tasks.rados:clients are ['client.0', 'client.1'] 2021-02-10T04:28:07.893 INFO:tasks.rados:starting run 0 out of 1 2021-02-10T04:28:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:04 smithi164 conmon[29288]: cluster 2021-02-10T04:28:02.435199+0000 osd.2 (osd.2) 23 : cluster [DBG] 2.e scrub ok 2021-02-10T04:28:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:04 smithi164 conmon[29288]: audit 2021-02-10T04:28:03.338400+0000 mon.c (mon.1) 29 : audit [DBG] from='client.? 172.21.15.85:0/1531272644' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-02-10T04:28:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:05 smithi164 conmon[29288]: cluster 2021-02-10T04:28:03.352772+0000 osd.1 (osd.1) 30 : cluster [DBG] 4.4 scrub ok 2021-02-10T04:28:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:05 smithi164 conmon[29288]: cluster 2021-02-10T04:28:03.984046+0000 mgr.y (mgr.14140) 311 : cluster [DBG] pgmap v268: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:28:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:06 smithi164 conmon[29288]: cluster 2021-02-10T04:28:05.073554+0000 osd.2 (osd.2) 24 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:28:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:06 smithi164 conmon[29288]: 2021-02-10T04:28:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:06 smithi164 conmon[29288]: cluster 2021-02-10T04:28:05.073705+0000 osd.2 (osd.2) 25 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:28:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:06 smithi164 conmon[29288]: cluster 2021-02-10T04:28 2021-02-10T04:28:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:06 smithi164 conmon[29288]: :05.641570+0000 osd.4 (osd.4) 29 : cluster [DBG] 6.2 scrub ok 2021-02-10T04:28:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:07 smithi164 conmon[29288]: cluster 2021-02-10T04:28:05.984760+0000 mgr.y (mgr.14140) 312 : cluster [DBG] pgmap v269: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:28:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:07 smithi164 conmon[29288]: cluster 2021-02-10T04:28:06.070348+0000 mon.a (mon.0) 705 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2021-02-10T04:28:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:07 smithi164 conmon[29288]: cluster 2021-02-10T04:28:06.118570+0000 osd.6 (osd.6) 19 : cluster [DBG] 4.1f scrub ok 2021-02-10T04:28:07.896 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:04 smithi085 conmon[29319]: cluster 2021-02-10T04:28:02.435199+0000 osd.2 (osd.2) 23 : cluster [DBG] 2.e scrub ok 2021-02-10T04:28:07.896 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:04 smithi085 conmon[29319]: audit 2021-02-10T04:28:03.338400+0000 mon.c (mon.1) 29 : audit [DBG] from='client.? 172.21.15.85:0/1531272644' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-02-10T04:28:07.896 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:05 smithi085 conmon[29319]: cluster 2021-02-10T04:28:03.352772+0000 osd.1 (osd.1) 30 : 2021-02-10T04:28:07.897 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:05 smithi085 conmon[29319]: cluster [DBG] 4.4 scrub ok 2021-02-10T04:28:07.897 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:05 smithi085 conmon[29319]: cluster 2021-02-10T04:28:03.984046+0000 mgr.y (mgr.14140) 311 : cluster [DBG] pgmap v268: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:28:07.897 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:06 smithi085 conmon[29319]: cluster 2021-02-10T04:28:05.073554+0000 osd.2 (osd.2) 24 : 2021-02-10T04:28:07.897 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:06 smithi085 conmon[29319]: cluster [DBG] purged_snaps scrub starts 2021-02-10T04:28:07.897 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:06 smithi085 conmon[29319]: cluster 2021-02-10T04:28:05.073705+0000 osd.2 (osd.2) 25 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:28:07.897 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:06 smithi085 conmon[29319]: cluster 2021-02-10T04:28:07.897 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:06 smithi085 conmon[29319]: 2021-02-10T04:28:05.641570+0000 osd.4 (osd.4) 29 : cluster [DBG] 6.2 scrub ok 2021-02-10T04:28:07.897 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:07 smithi085 conmon[29319]: cluster 2021-02-10T04:28:05.984760+0000 mgr.y (mgr.14140) 312 : cluster [DBG] pgmap v269: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:28:07.897 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:07 smithi085 conmon[29319]: cluster 2021-02-10T04:28:06.070348+0000 mon.a (mon.0) 705 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2021-02-10T04:28:07.897 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:07 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:28:07.897 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:07 smithi085 conmon[29319]: -10T04:28:06.118570+0000 osd.6 (osd.6) 19 2021-02-10T04:28:07.897 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:07 smithi085 conmon[29319]: : cluster [DBG] 4.1f scrub ok 2021-02-10T04:28:07.897 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:04 smithi085 conmon[35961]: cluster 2021-02-10T04:28:02.435199+0000 2021-02-10T04:28:07.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:04 smithi085 conmon[35961]: osd.2 (osd.2) 23 : cluster [DBG] 2.e scrub ok 2021-02-10T04:28:07.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:04 smithi085 conmon[35961]: audit 2021-02-10T04:28:03.338400+0000 2021-02-10T04:28:07.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:04 smithi085 conmon[35961]: mon.c (mon.1) 29 : audit [DBG] from='client.? 172.21.15.85:0/1531272644' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-02-10T04:28:07.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:05 smithi085 conmon[35961]: cluster 2021-02-10T04:28:03.352772+0000 osd.1 (osd.1) 30 : 2021-02-10T04:28:07.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:05 smithi085 conmon[35961]: cluster [DBG] 4.4 scrub ok 2021-02-10T04:28:07.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:05 smithi085 conmon[35961]: cluster 2021-02-10T04:28:03.984046+0000 mgr.y (mgr.14140) 311 : cluster [DBG] pgmap v268: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:28:07.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:06 smithi085 conmon[35961]: cluster 2021-02-10T04:28:05.073554+0000 osd.2 (osd.2) 24 2021-02-10T04:28:07.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:06 smithi085 conmon[35961]: : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:28:07.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:06 smithi085 conmon[35961]: cluster 2021-02-10T04:28:05.073705+0000 osd.2 (osd.2) 25 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:28:07.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:06 smithi085 conmon[35961]: cluster 2021-02-10T04:28:05.641570+0000 osd.4 (osd. 2021-02-10T04:28:07.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:06 smithi085 conmon[35961]: 4) 29 : cluster [DBG] 6.2 scrub ok 2021-02-10T04:28:07.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:07 smithi085 conmon[35961]: cluster 2021-02-10T04:28:05.984760+0000 mgr.y (mgr 2021-02-10T04:28:07.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:07 smithi085 conmon[35961]: .14140) 312 : cluster [DBG] pgmap v269: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:28:07.899 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:07 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:28:07.899 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:07 smithi085 conmon[35961]: 10T04:28:06.070348+0000 mon.a (mon.0) 705 : 2021-02-10T04:28:07.899 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:07 smithi085 conmon[35961]: cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2021-02-10T04:28:07.899 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:07 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:28:07.899 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:07 smithi085 conmon[35961]: -02-10T04:28: 2021-02-10T04:28:07.899 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:07 smithi085 conmon[35961]: 06.118570+0000 osd.6 (osd.6) 19 2021-02-10T04:28:07.899 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:07 smithi085 conmon[35961]: : cluster [DBG] 4.1f scrub ok 2021-02-10T04:28:07.967 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:08.097 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:28:08.143 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:28:07 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:28:08.148 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:28:08.162 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:28:08.192 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:28:08 smithi085 conmon[45443]: debug 2021-02-10T04:28:08.067+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:08.206 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:28:08.235 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:28:08.235 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:28:08.256 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:08.348 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:28:08.454 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:28:08.481 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:08.532 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:28:08 smithi164 conmon[37263]: debug 2021-02-10T04:28:08.330+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:08.556 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:28:08.660 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:28:08.683 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:08.773 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:28:08.877 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:28:08.893 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:28:08 smithi164 conmon[33677]: debug 2021-02-10T04:28:08.532+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:08.893 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:28:08 smithi164 conmon[33677]: debug 2021-02-10T04:28:08.738+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:08.904 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:09.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:09 smithi085 conmon[29319]: audit 2021-02-10T04:28 2021-02-10T04:28:09.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:09 smithi085 conmon[29319]: :07.643360+0000 mgr.y (mgr.14140) 313 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:28:09.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:09 smithi085 conmon[29319]: cluster 2021-02-10T04:28:07.985311+0000 mgr.y (mgr.14140) 314 : cluster [DBG] pgmap v271: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 570 B/s rd, 0 op/s 2021-02-10T04:28:09.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:09 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:28:09.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:09 smithi085 conmon[35961]: T04:28:07.643360+0000 mgr.y (mgr.14140) 313 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:28:09.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:09 smithi085 conmon[35961]: cluster 2021-02-10T04:28:07.985311+0000 mgr.y (mgr.14140) 314 : cluster [DBG] pgmap v271: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 570 B/s rd, 0 op/s 2021-02-10T04:28:09.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:09 smithi164 conmon[29288]: audit 2021-02-10T04:28:07.643360+0000 mgr.y (mgr.14140) 313 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:28:09.508 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:09 smithi164 conmon[29288]: cluster 2021-02-10T04:28:07.985311+0000 mgr.y (mgr.14140) 314 : cluster [DBG] pgmap v271: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 570 B/s rd, 0 op/s 2021-02-10T04:28:11.064 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:28:11.168 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:28:11.172 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:11 smithi085 conmon[29319]: cluster 2021-02-10T04:28:09.985858+0000 mgr.y (mgr.14140 2021-02-10T04:28:11.173 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:11 smithi085 conmon[29319]: ) 315 : cluster [DBG] pgmap v272: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:28:11.173 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:28:10 smithi085 conmon[41540]: debug 2021-02-10T04:28:10.872+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:11.173 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:11 smithi085 conmon[35961]: cluster 2021-02-10T04:28:09.985858+0000 mgr.y (mgr. 2021-02-10T04:28:11.173 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:11 smithi085 conmon[35961]: 14140) 315 : cluster [DBG] pgmap v272: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:28:11.200 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:11.298 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:28:11.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:11 smithi164 conmon[29288]: cluster 2021-02-10T04:28:09.985858+0000 mgr.y (mgr.14140) 315 : cluster [DBG] pgmap v272: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:28:11.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:28:11 smithi164 conmon[33677]: debug 2021-02-10T04:28:11.264+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:11.402 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:28:11.437 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:11.751 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:28:11.768 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:28:11.770 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:28:11.771 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:28:11.791 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:28:11.811 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:28:11.829 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:28:11.847 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:28:11.848 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:28:11.877 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:28:11.895 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:28:11.896 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:28:11.897 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:28:11.897 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:28:11.897 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:28:11.897 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:28:11.897 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:28:11.897 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:28:11.897 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:28:11.952 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:28:11.977 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:12.055 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:28:11 smithi085 conmon[41540]: debug 2021-02-10T04:28:11.797+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:12.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:12 smithi164 conmon[29288]: cluster 2021-02-10T04:28:11.234735+0000 osd.6 (osd.6) 20 : cluster [DBG] 2.7 scrub ok 2021-02-10T04:28:12.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:12 smithi164 conmon[29288]: cluster 2021-02-10T04:28:11.311176+0000 2021-02-10T04:28:12.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:12 smithi164 conmon[29288]: osd.3 (osd.3) 31 : cluster [DBG] 4.17 scrub ok 2021-02-10T04:28:12.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:12 smithi164 conmon[29288]: cluster 2021-02-10T04:28:11.693607+0000 osd.4 (osd.4) 30 : cluster [DBG] 4.e deep-scrub ok 2021-02-10T04:28:12.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:12 smithi164 conmon[29288]: audit 2021-02-10T04:28:11.727274+0000 mon.a (mon.0) 706 : audit [DBG] from='client.? 172.21.15.85:0/420272212' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:28:12.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:12 smithi164 conmon[29288]: 2021-02-10T04:28:12.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:12 smithi085 conmon[29319]: cluster 2021-02-10T04:28:11.234735+0000 osd.6 (osd.6) 20 : cluster [DBG] 2.7 scrub ok 2021-02-10T04:28:12.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:12 smithi085 conmon[29319]: cluster 2021-02-10T04:28:11.311176+0000 osd.3 (osd.3) 31 : cluster [DBG] 4.17 scrub ok 2021-02-10T04:28:12.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:12 smithi085 conmon[29319]: cluster 2021-02-10T04:28:11.693607+0000 osd.4 (osd.4) 30 : cluster [DBG] 4.e deep-scrub ok 2021-02-10T04:28:12.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:12 smithi085 conmon[29319]: audit 2021-02-10T04:28:11.727274+0000 mon.a (mon.0) 706 : audit [DBG] from='client.? 172.21.15.85:0/420272212' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:28:12.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:12 smithi085 conmon[35961]: cluster 2021-02-10T04:28:11.234735+0000 osd.6 (osd.6) 20 : cluster [DBG] 2.7 scrub ok 2021-02-10T04:28:12.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:12 smithi085 conmon[35961]: cluster 2021-02-10T04:28:11.311176+0000 osd.3 (osd.3) 31 : cluster [DBG] 4.17 scrub ok 2021-02-10T04:28:12.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:12 smithi085 conmon[35961]: cluster 2021-02-10T04:28:11.693607+0000 osd.4 (osd.4) 30 : cluster [DBG] 4.e deep-scrub ok 2021-02-10T04:28:12.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:12 smithi085 conmon[35961]: audit 2021-02-10T04:28:11.727274+0000 mon.a (mon.0) 706 : audit [DBG] from='client.? 172.21.15.85:0/420272212' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:28:13.159 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:28:13.160 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:28:13.178 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:13 smithi085 conmon[29319]: audit 2021-02-10T04:28:11.754168+0000 mgr.y (mgr.14140) 316 : audit [DBG] from='client.24658 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:28:13.178 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:13 smithi085 conmon[29319]: cluster 2021-02-10T04:28:11.986418+0000 mgr.y (mgr.14140) 317 : cluster [DBG] pgmap v273: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:28:13.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:13 smithi085 conmon[35961]: audit 2021-02-10T04:28:11.754168+0000 mgr.y (mgr 2021-02-10T04:28:13.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:13 smithi085 conmon[35961]: .14140) 316 : audit [DBG] from='client.24658 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:28:13.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:13 smithi085 conmon[35961]: cluster 2021-02-10T04:28:11.986418+0000 mgr.y (mgr.14140) 317 : cluster [DBG] pgmap v273: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:28:13.202 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2021-02-10T04:28:09.985490+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":25858,"num_objects":357,"num_object_clones":0,"num_object_copies":1071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":357,"num_whiteouts":0,"num_read":1616,"num_read_kb":1084,"num_write":759,"num_write_kb":177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":790,"ondisk_log_size":790,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":58432,"kb_used_data":7232,"kb_used_omap":0,"kb_used_meta":51136,"kb_avail":749902784,"statfs":{"total":767960285184,"available":767900450816,"internally_reserved":0,"allocated":7405568,"data_stored":2586984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52363264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002703"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":"38","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.116975+0000","last_change":"2021-02-10T04:27:07.339624+0000","last_active":"2021-02-10T04:27:47.116975+0000","last_peered":"2021-02-10T04:27:47.116975+0000","last_clean":"2021-02-10T04:27:47.116975+0000","last_became_active":"2021-02-10T04:25:53.150917+0000","last_became_peered":"2021-02-10T04:25:53.150917+0000","last_unstale":"2021-02-10T04:27:47.116975+0000","last_undegraded":"2021-02-10T04:27:47.116975+0000","last_fullsized":"2021-02-10T04:27:47.116975+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:07.339551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:07.339551+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"42","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.016060+0000","last_change":"2021-02-10T04:27:49.628172+0000","last_active":"2021-02-10T04:27:59.016060+0000","last_peered":"2021-02-10T04:27:59.016060+0000","last_clean":"2021-02-10T04:27:59.016060+0000","last_became_active":"2021-02-10T04:25:51.029110+0000","last_became_peered":"2021-02-10T04:25:51.029110+0000","last_unstale":"2021-02-10T04:27:59.016060+0000","last_undegraded":"2021-02-10T04:27:59.016060+0000","last_fullsized":"2021-02-10T04:27:59.016060+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:49.628090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:49.628090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"59","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:48.635871+0000","last_change":"2021-02-10T04:27:48.635827+0000","last_active":"2021-02-10T04:27:48.635871+0000","last_peered":"2021-02-10T04:27:48.635871+0000","last_clean":"2021-02-10T04:27:48.635871+0000","last_became_active":"2021-02-10T04:25:36.336112+0000","last_became_peered":"2021-02-10T04:25:36.336112+0000","last_unstale":"2021-02-10T04:27:48.635871+0000","last_undegraded":"2021-02-10T04:27:48.635871+0000","last_fullsized":"2021-02-10T04:27:48.635871+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:48.635765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:48.635765+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"63'44","reported_seq":"137","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:49.269673+0000","last_change":"2021-02-10T04:27:49.269619+0000","last_active":"2021-02-10T04:27:49.269673+0000","last_peered":"2021-02-10T04:27:49.269673+0000","last_clean":"2021-02-10T04:27:49.269673+0000","last_became_active":"2021-02-10T04:25:49.008451+0000","last_became_peered":"2021-02-10T04:25:49.008451+0000","last_unstale":"2021-02-10T04:27:49.269673+0000","last_undegraded":"2021-02-10T04:27:49.269673+0000","last_fullsized":"2021-02-10T04:27:49.269673+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'44","last_scrub_stamp":"2021-02-10T04:27:49.269539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:49.269539+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":89,"num_read_kb":66,"num_write":45,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"40","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.016270+0000","last_change":"2021-02-10T04:27:51.679613+0000","last_active":"2021-02-10T04:27:59.016270+0000","last_peered":"2021-02-10T04:27:59.016270+0000","last_clean":"2021-02-10T04:27:59.016270+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:27:59.016270+0000","last_undegraded":"2021-02-10T04:27:59.016270+0000","last_fullsized":"2021-02-10T04:27:59.016270+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:51.679515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:51.679515+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"42","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.210320+0000","last_change":"2021-02-10T04:26:54.007373+0000","last_active":"2021-02-10T04:27:59.210320+0000","last_peered":"2021-02-10T04:27:59.210320+0000","last_clean":"2021-02-10T04:27:59.210320+0000","last_became_active":"2021-02-10T04:25:51.030405+0000","last_became_peered":"2021-02-10T04:25:51.030405+0000","last_unstale":"2021-02-10T04:27:59.210320+0000","last_undegraded":"2021-02-10T04:27:59.210320+0000","last_fullsized":"2021-02-10T04:27:59.210320+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:54.007310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:26:54.007310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"49","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.116467+0000","last_change":"2021-02-10T04:27:13.332288+0000","last_active":"2021-02-10T04:27:47.116467+0000","last_peered":"2021-02-10T04:27:47.116467+0000","last_clean":"2021-02-10T04:27:47.116467+0000","last_became_active":"2021-02-10T04:25:34.327162+0000","last_became_peered":"2021-02-10T04:25:34.327162+0000","last_unstale":"2021-02-10T04:27:47.116467+0000","last_undegraded":"2021-02-10T04:27:47.116467+0000","last_fullsized":"2021-02-10T04:27:47.116467+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:27:13.332224+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"78'32","reported_seq":"98","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.016200+0000","last_change":"2021-02-10T04:27:09.170492+0000","last_active":"2021-02-10T04:27:59.016200+0000","last_peered":"2021-02-10T04:27:59.016200+0000","last_clean":"2021-02-10T04:27:59.016200+0000","last_became_active":"2021-02-10T04:25:49.010481+0000","last_became_peered":"2021-02-10T04:25:49.010481+0000","last_unstale":"2021-02-10T04:27:59.016200+0000","last_undegraded":"2021-02-10T04:27:59.016200+0000","last_fullsized":"2021-02-10T04:27:59.016200+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'32","last_scrub_stamp":"2021-02-10T04:27:09.170428+0000","last_deep_scrub":"78'32","last_deep_scrub_stamp":"2021-02-10T04:27:09.170428+0000","last_clean_scrub_stamp":"2021-02-10T04:27:09.170428+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":47,"num_read_kb":28,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"40","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.210476+0000","last_change":"2021-02-10T04:27:07.962210+0000","last_active":"2021-02-10T04:27:59.210476+0000","last_peered":"2021-02-10T04:27:59.210476+0000","last_clean":"2021-02-10T04:27:59.210476+0000","last_became_active":"2021-02-10T04:25:53.042601+0000","last_became_peered":"2021-02-10T04:25:53.042601+0000","last_unstale":"2021-02-10T04:27:59.210476+0000","last_undegraded":"2021-02-10T04:27:59.210476+0000","last_fullsized":"2021-02-10T04:27:59.210476+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:07.962115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:07.962115+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"41","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.727439+0000","last_change":"2021-02-10T04:27:29.251059+0000","last_active":"2021-02-10T04:27:47.727439+0000","last_peered":"2021-02-10T04:27:47.727439+0000","last_clean":"2021-02-10T04:27:47.727439+0000","last_became_active":"2021-02-10T04:25:51.029286+0000","last_became_peered":"2021-02-10T04:25:51.029286+0000","last_unstale":"2021-02-10T04:27:47.727439+0000","last_undegraded":"2021-02-10T04:27:47.727439+0000","last_fullsized":"2021-02-10T04:27:47.727439+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:29.250978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:29.250978+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"53","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:50.297789+0000","last_change":"2021-02-10T04:27:50.297737+0000","last_active":"2021-02-10T04:27:50.297789+0000","last_peered":"2021-02-10T04:27:50.297789+0000","last_clean":"2021-02-10T04:27:50.297789+0000","last_became_active":"2021-02-10T04:25:36.344477+0000","last_became_peered":"2021-02-10T04:25:36.344477+0000","last_unstale":"2021-02-10T04:27:50.297789+0000","last_undegraded":"2021-02-10T04:27:50.297789+0000","last_fullsized":"2021-02-10T04:27:50.297789+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:27:50.297650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"63'12","reported_seq":"71","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.016515+0000","last_change":"2021-02-10T04:27:37.441260+0000","last_active":"2021-02-10T04:27:59.016515+0000","last_peered":"2021-02-10T04:27:59.016515+0000","last_clean":"2021-02-10T04:27:59.016515+0000","last_became_active":"2021-02-10T04:25:49.013196+0000","last_became_peered":"2021-02-10T04:25:49.013196+0000","last_unstale":"2021-02-10T04:27:59.016515+0000","last_undegraded":"2021-02-10T04:27:59.016515+0000","last_fullsized":"2021-02-10T04:27:59.016515+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2021-02-10T04:27:37.441168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:37.441168+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"39","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.726371+0000","last_change":"2021-02-10T04:27:33.628099+0000","last_active":"2021-02-10T04:27:47.726371+0000","last_peered":"2021-02-10T04:27:47.726371+0000","last_clean":"2021-02-10T04:27:47.726371+0000","last_became_active":"2021-02-10T04:25:53.039844+0000","last_became_peered":"2021-02-10T04:25:53.039844+0000","last_unstale":"2021-02-10T04:27:47.726371+0000","last_undegraded":"2021-02-10T04:27:47.726371+0000","last_fullsized":"2021-02-10T04:27:47.726371+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:33.628038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:33.628038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"42","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.211764+0000","last_change":"2021-02-10T04:27:04.977415+0000","last_active":"2021-02-10T04:27:59.211764+0000","last_peered":"2021-02-10T04:27:59.211764+0000","last_clean":"2021-02-10T04:27:59.211764+0000","last_became_active":"2021-02-10T04:25:51.025488+0000","last_became_peered":"2021-02-10T04:25:51.025488+0000","last_unstale":"2021-02-10T04:27:59.211764+0000","last_undegraded":"2021-02-10T04:27:59.211764+0000","last_fullsized":"2021-02-10T04:27:59.211764+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:04.977353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:04.977353+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"53","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:28:00.398561+0000","last_change":"2021-02-10T04:28:00.398561+0000","last_active":"2021-02-10T04:28:00.398561+0000","last_peered":"2021-02-10T04:28:00.398561+0000","last_clean":"2021-02-10T04:28:00.398561+0000","last_became_active":"2021-02-10T04:25:36.337260+0000","last_became_peered":"2021-02-10T04:25:36.337260+0000","last_unstale":"2021-02-10T04:28:00.398561+0000","last_undegraded":"2021-02-10T04:28:00.398561+0000","last_fullsized":"2021-02-10T04:28:00.398561+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:00.398480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:00.398480+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"63'25","reported_seq":"98","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.016384+0000","last_change":"2021-02-10T04:27:05.242824+0000","last_active":"2021-02-10T04:27:59.016384+0000","last_peered":"2021-02-10T04:27:59.016384+0000","last_clean":"2021-02-10T04:27:59.016384+0000","last_became_active":"2021-02-10T04:25:49.000911+0000","last_became_peered":"2021-02-10T04:25:49.000911+0000","last_unstale":"2021-02-10T04:27:59.016384+0000","last_undegraded":"2021-02-10T04:27:59.016384+0000","last_fullsized":"2021-02-10T04:27:59.016384+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2021-02-10T04:27:05.242762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:05.242762+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":53,"num_read_kb":37,"num_write":26,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"38","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.117945+0000","last_change":"2021-02-10T04:26:55.332873+0000","last_active":"2021-02-10T04:27:47.117945+0000","last_peered":"2021-02-10T04:27:47.117945+0000","last_clean":"2021-02-10T04:27:47.117945+0000","last_became_active":"2021-02-10T04:25:53.147831+0000","last_became_peered":"2021-02-10T04:25:53.147831+0000","last_unstale":"2021-02-10T04:27:47.117945+0000","last_undegraded":"2021-02-10T04:27:47.117945+0000","last_fullsized":"2021-02-10T04:27:47.117945+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:55.332815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:26:55.332815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"42","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.016637+0000","last_change":"2021-02-10T04:27:48.598953+0000","last_active":"2021-02-10T04:27:59.016637+0000","last_peered":"2021-02-10T04:27:59.016637+0000","last_clean":"2021-02-10T04:27:59.016637+0000","last_became_active":"2021-02-10T04:25:51.018872+0000","last_became_peered":"2021-02-10T04:25:51.018872+0000","last_unstale":"2021-02-10T04:27:59.016637+0000","last_undegraded":"2021-02-10T04:27:59.016637+0000","last_fullsized":"2021-02-10T04:27:59.016637+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:48.598870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:48.598870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"58","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.117058+0000","last_change":"2021-02-10T04:26:46.425375+0000","last_active":"2021-02-10T04:27:47.117058+0000","last_peered":"2021-02-10T04:27:47.117058+0000","last_clean":"2021-02-10T04:27:47.117058+0000","last_became_active":"2021-02-10T04:25:36.336893+0000","last_became_peered":"2021-02-10T04:25:36.336893+0000","last_unstale":"2021-02-10T04:27:47.117058+0000","last_undegraded":"2021-02-10T04:27:47.117058+0000","last_fullsized":"2021-02-10T04:27:47.117058+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:46.425288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:26:46.425288+0000","last_clean_scrub_stamp":"2021-02-10T04:26:46.425288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"62'22","reported_seq":"88","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:58.363366+0000","last_change":"2021-02-10T04:27:58.363322+0000","last_active":"2021-02-10T04:27:58.363366+0000","last_peered":"2021-02-10T04:27:58.363366+0000","last_clean":"2021-02-10T04:27:58.363366+0000","last_became_active":"2021-02-10T04:25:49.008997+0000","last_became_peered":"2021-02-10T04:25:49.008997+0000","last_unstale":"2021-02-10T04:27:58.363366+0000","last_undegraded":"2021-02-10T04:27:58.363366+0000","last_fullsized":"2021-02-10T04:27:58.363366+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'22","last_scrub_stamp":"2021-02-10T04:27:58.363248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:58.363248+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":49,"num_read_kb":31,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"40","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.016899+0000","last_change":"2021-02-10T04:27:29.364660+0000","last_active":"2021-02-10T04:27:59.016899+0000","last_peered":"2021-02-10T04:27:59.016899+0000","last_clean":"2021-02-10T04:27:59.016899+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:27:59.016899+0000","last_undegraded":"2021-02-10T04:27:59.016899+0000","last_fullsized":"2021-02-10T04:27:59.016899+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:29.364585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:29.364585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"42","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.209911+0000","last_change":"2021-02-10T04:27:27.017142+0000","last_active":"2021-02-10T04:27:59.209911+0000","last_peered":"2021-02-10T04:27:59.209911+0000","last_clean":"2021-02-10T04:27:59.209911+0000","last_became_active":"2021-02-10T04:25:51.029399+0000","last_became_peered":"2021-02-10T04:25:51.029399+0000","last_unstale":"2021-02-10T04:27:59.209911+0000","last_undegraded":"2021-02-10T04:27:59.209911+0000","last_fullsized":"2021-02-10T04:27:59.209911+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:27.017040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:27.017040+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"54","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.066673+0000","last_change":"2021-02-10T04:27:02.190158+0000","last_active":"2021-02-10T04:27:59.066673+0000","last_peered":"2021-02-10T04:27:59.066673+0000","last_clean":"2021-02-10T04:27:59.066673+0000","last_became_active":"2021-02-10T04:25:36.336442+0000","last_became_peered":"2021-02-10T04:25:36.336442+0000","last_unstale":"2021-02-10T04:27:59.066673+0000","last_undegraded":"2021-02-10T04:27:59.066673+0000","last_fullsized":"2021-02-10T04:27:59.066673+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:27:02.190056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:02.190056+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"62'15","reported_seq":"67","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.117663+0000","last_change":"2021-02-10T04:26:56.623190+0000","last_active":"2021-02-10T04:27:47.117663+0000","last_peered":"2021-02-10T04:27:47.117663+0000","last_clean":"2021-02-10T04:27:47.117663+0000","last_became_active":"2021-02-10T04:25:49.010553+0000","last_became_peered":"2021-02-10T04:25:49.010553+0000","last_unstale":"2021-02-10T04:27:47.117663+0000","last_undegraded":"2021-02-10T04:27:47.117663+0000","last_fullsized":"2021-02-10T04:27:47.117663+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:26:56.623109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:26:56.623109+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"40","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.210984+0000","last_change":"2021-02-10T04:27:26.029923+0000","last_active":"2021-02-10T04:27:59.210984+0000","last_peered":"2021-02-10T04:27:59.210984+0000","last_clean":"2021-02-10T04:27:59.210984+0000","last_became_active":"2021-02-10T04:25:53.041098+0000","last_became_peered":"2021-02-10T04:25:53.041098+0000","last_unstale":"2021-02-10T04:27:59.210984+0000","last_undegraded":"2021-02-10T04:27:59.210984+0000","last_fullsized":"2021-02-10T04:27:59.210984+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:26.029856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:26.029856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"41","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.726855+0000","last_change":"2021-02-10T04:27:27.564268+0000","last_active":"2021-02-10T04:27:47.726855+0000","last_peered":"2021-02-10T04:27:47.726855+0000","last_clean":"2021-02-10T04:27:47.726855+0000","last_became_active":"2021-02-10T04:25:51.022853+0000","last_became_peered":"2021-02-10T04:25:51.022853+0000","last_unstale":"2021-02-10T04:27:47.726855+0000","last_undegraded":"2021-02-10T04:27:47.726855+0000","last_fullsized":"2021-02-10T04:27:47.726855+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:27.564175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:27.564175+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"49","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.117754+0000","last_change":"2021-02-10T04:27:16.340830+0000","last_active":"2021-02-10T04:27:47.117754+0000","last_peered":"2021-02-10T04:27:47.117754+0000","last_clean":"2021-02-10T04:27:47.117754+0000","last_became_active":"2021-02-10T04:25:34.324981+0000","last_became_peered":"2021-02-10T04:25:34.324981+0000","last_unstale":"2021-02-10T04:27:47.117754+0000","last_undegraded":"2021-02-10T04:27:47.117754+0000","last_fullsized":"2021-02-10T04:27:47.117754+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:16.340724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:16.340724+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"76'35","reported_seq":"107","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:28:06.118734+0000","last_change":"2021-02-10T04:28:06.118655+0000","last_active":"2021-02-10T04:28:06.118734+0000","last_peered":"2021-02-10T04:28:06.118734+0000","last_clean":"2021-02-10T04:28:06.118734+0000","last_became_active":"2021-02-10T04:25:49.015760+0000","last_became_peered":"2021-02-10T04:25:49.015760+0000","last_unstale":"2021-02-10T04:28:06.118734+0000","last_undegraded":"2021-02-10T04:28:06.118734+0000","last_fullsized":"2021-02-10T04:28:06.118734+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'35","last_scrub_stamp":"2021-02-10T04:28:06.118574+0000","last_deep_scrub":"76'35","last_deep_scrub_stamp":"2021-02-10T04:26:58.198641+0000","last_clean_scrub_stamp":"2021-02-10T04:28:06.118574+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":50,"num_read_kb":30,"num_write":24,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"39","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.728368+0000","last_change":"2021-02-10T04:27:39.139028+0000","last_active":"2021-02-10T04:27:47.728368+0000","last_peered":"2021-02-10T04:27:47.728368+0000","last_clean":"2021-02-10T04:27:47.728368+0000","last_became_active":"2021-02-10T04:25:53.042279+0000","last_became_peered":"2021-02-10T04:25:53.042279+0000","last_unstale":"2021-02-10T04:27:47.728368+0000","last_undegraded":"2021-02-10T04:27:47.728368+0000","last_fullsized":"2021-02-10T04:27:47.728368+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:39.138947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:39.138947+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"58'2","reported_seq":"72","reported_epoch":"86","state":"active+clean","last_fresh":"2021-02-10T04:28:07.159909+0000","last_change":"2021-02-10T04:27:15.041424+0000","last_active":"2021-02-10T04:28:07.159909+0000","last_peered":"2021-02-10T04:28:07.159909+0000","last_clean":"2021-02-10T04:28:07.159909+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:28:07.159909+0000","last_undegraded":"2021-02-10T04:28:07.159909+0000","last_fullsized":"2021-02-10T04:28:07.159909+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:15.041329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:15.041329+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"60","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.211004+0000","last_change":"2021-02-10T04:27:04.011149+0000","last_active":"2021-02-10T04:27:59.211004+0000","last_peered":"2021-02-10T04:27:59.211004+0000","last_clean":"2021-02-10T04:27:59.211004+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:27:59.211004+0000","last_undegraded":"2021-02-10T04:27:59.211004+0000","last_fullsized":"2021-02-10T04:27:59.211004+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:04.011053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:04.011053+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"63'22","reported_seq":"90","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.739174+0000","last_change":"2021-02-10T04:27:59.739174+0000","last_active":"2021-02-10T04:27:59.739174+0000","last_peered":"2021-02-10T04:27:59.739174+0000","last_clean":"2021-02-10T04:27:59.739174+0000","last_became_active":"2021-02-10T04:25:49.009177+0000","last_became_peered":"2021-02-10T04:25:49.009177+0000","last_unstale":"2021-02-10T04:27:59.739174+0000","last_undegraded":"2021-02-10T04:27:59.739174+0000","last_fullsized":"2021-02-10T04:27:59.739174+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2021-02-10T04:27:59.739086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:59.739086+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":22,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"38","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:50.300145+0000","last_change":"2021-02-10T04:27:50.300043+0000","last_active":"2021-02-10T04:27:50.300145+0000","last_peered":"2021-02-10T04:27:50.300145+0000","last_clean":"2021-02-10T04:27:50.300145+0000","last_became_active":"2021-02-10T04:25:53.150845+0000","last_became_peered":"2021-02-10T04:25:53.150845+0000","last_unstale":"2021-02-10T04:27:50.300145+0000","last_undegraded":"2021-02-10T04:27:50.300145+0000","last_fullsized":"2021-02-10T04:27:50.300145+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_clean_scrub_stamp":"2021-02-10T04:27:50.299942+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"41","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.728272+0000","last_change":"2021-02-10T04:27:07.181928+0000","last_active":"2021-02-10T04:27:47.728272+0000","last_peered":"2021-02-10T04:27:47.728272+0000","last_clean":"2021-02-10T04:27:47.728272+0000","last_became_active":"2021-02-10T04:25:51.018491+0000","last_became_peered":"2021-02-10T04:25:51.018491+0000","last_unstale":"2021-02-10T04:27:47.728272+0000","last_undegraded":"2021-02-10T04:27:47.728272+0000","last_fullsized":"2021-02-10T04:27:47.728272+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:07.181854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:07.181854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"54","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.066686+0000","last_change":"2021-02-10T04:26:52.256790+0000","last_active":"2021-02-10T04:27:59.066686+0000","last_peered":"2021-02-10T04:27:59.066686+0000","last_clean":"2021-02-10T04:27:59.066686+0000","last_became_active":"2021-02-10T04:25:35.328828+0000","last_became_peered":"2021-02-10T04:25:35.328828+0000","last_unstale":"2021-02-10T04:27:59.066686+0000","last_undegraded":"2021-02-10T04:27:59.066686+0000","last_fullsized":"2021-02-10T04:27:59.066686+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:26:52.256723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:52.256723+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"63'21","reported_seq":"83","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.117971+0000","last_change":"2021-02-10T04:27:12.291860+0000","last_active":"2021-02-10T04:27:47.117971+0000","last_peered":"2021-02-10T04:27:47.117971+0000","last_clean":"2021-02-10T04:27:47.117971+0000","last_became_active":"2021-02-10T04:25:49.007049+0000","last_became_peered":"2021-02-10T04:25:49.007049+0000","last_unstale":"2021-02-10T04:27:47.117971+0000","last_undegraded":"2021-02-10T04:27:47.117971+0000","last_fullsized":"2021-02-10T04:27:47.117971+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2021-02-10T04:27:12.291796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:12.291796+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":27,"num_write":21,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"39","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.727303+0000","last_change":"2021-02-10T04:27:19.134286+0000","last_active":"2021-02-10T04:27:47.727303+0000","last_peered":"2021-02-10T04:27:47.727303+0000","last_clean":"2021-02-10T04:27:47.727303+0000","last_became_active":"2021-02-10T04:25:53.034771+0000","last_became_peered":"2021-02-10T04:25:53.034771+0000","last_unstale":"2021-02-10T04:27:47.727303+0000","last_undegraded":"2021-02-10T04:27:47.727303+0000","last_fullsized":"2021-02-10T04:27:47.727303+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:27:19.134212+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"42","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.066081+0000","last_change":"2021-02-10T04:27:19.077862+0000","last_active":"2021-02-10T04:27:59.066081+0000","last_peered":"2021-02-10T04:27:59.066081+0000","last_clean":"2021-02-10T04:27:59.066081+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:27:59.066081+0000","last_undegraded":"2021-02-10T04:27:59.066081+0000","last_fullsized":"2021-02-10T04:27:59.066081+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:27:19.077777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"51","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.212214+0000","last_change":"2021-02-10T04:27:03.000902+0000","last_active":"2021-02-10T04:27:59.212214+0000","last_peered":"2021-02-10T04:27:59.212214+0000","last_clean":"2021-02-10T04:27:59.212214+0000","last_became_active":"2021-02-10T04:25:34.331787+0000","last_became_peered":"2021-02-10T04:25:34.331787+0000","last_unstale":"2021-02-10T04:27:59.212214+0000","last_undegraded":"2021-02-10T04:27:59.212214+0000","last_fullsized":"2021-02-10T04:27:59.212214+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:03.000821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:03.000821+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"62'12","reported_seq":"95","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:48.284400+0000","last_change":"2021-02-10T04:27:48.284315+0000","last_active":"2021-02-10T04:27:48.284400+0000","last_peered":"2021-02-10T04:27:48.284400+0000","last_clean":"2021-02-10T04:27:48.284400+0000","last_became_active":"2021-02-10T04:26:39.005616+0000","last_became_peered":"2021-02-10T04:26:39.005616+0000","last_unstale":"2021-02-10T04:27:48.284400+0000","last_undegraded":"2021-02-10T04:27:48.284400+0000","last_fullsized":"2021-02-10T04:27:48.284400+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2021-02-10T04:27:48.284248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:48.284248+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"38","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.117582+0000","last_change":"2021-02-10T04:26:57.309978+0000","last_active":"2021-02-10T04:27:47.117582+0000","last_peered":"2021-02-10T04:27:47.117582+0000","last_clean":"2021-02-10T04:27:47.117582+0000","last_became_active":"2021-02-10T04:25:53.043277+0000","last_became_peered":"2021-02-10T04:25:53.043277+0000","last_unstale":"2021-02-10T04:27:47.117582+0000","last_undegraded":"2021-02-10T04:27:47.117582+0000","last_fullsized":"2021-02-10T04:27:47.117582+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:57.309915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:26:57.309915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"42","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.209894+0000","last_change":"2021-02-10T04:27:33.030004+0000","last_active":"2021-02-10T04:27:59.209894+0000","last_peered":"2021-02-10T04:27:59.209894+0000","last_clean":"2021-02-10T04:27:59.209894+0000","last_became_active":"2021-02-10T04:25:51.030952+0000","last_became_peered":"2021-02-10T04:25:51.030952+0000","last_unstale":"2021-02-10T04:27:59.209894+0000","last_undegraded":"2021-02-10T04:27:59.209894+0000","last_fullsized":"2021-02-10T04:27:59.209894+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:27:33.029912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"60","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.211074+0000","last_change":"2021-02-10T04:26:52.011165+0000","last_active":"2021-02-10T04:27:59.211074+0000","last_peered":"2021-02-10T04:27:59.211074+0000","last_clean":"2021-02-10T04:27:59.211074+0000","last_became_active":"2021-02-10T04:25:34.330242+0000","last_became_peered":"2021-02-10T04:25:34.330242+0000","last_unstale":"2021-02-10T04:27:59.211074+0000","last_undegraded":"2021-02-10T04:27:59.211074+0000","last_fullsized":"2021-02-10T04:27:59.211074+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:26:52.011096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:52.011096+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"62'15","reported_seq":"71","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.016406+0000","last_change":"2021-02-10T04:27:42.487239+0000","last_active":"2021-02-10T04:27:59.016406+0000","last_peered":"2021-02-10T04:27:59.016406+0000","last_clean":"2021-02-10T04:27:59.016406+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:27:59.016406+0000","last_undegraded":"2021-02-10T04:27:59.016406+0000","last_fullsized":"2021-02-10T04:27:59.016406+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:27:42.487154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:42.487154+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"39","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.726157+0000","last_change":"2021-02-10T04:27:21.548446+0000","last_active":"2021-02-10T04:27:47.726157+0000","last_peered":"2021-02-10T04:27:47.726157+0000","last_clean":"2021-02-10T04:27:47.726157+0000","last_became_active":"2021-02-10T04:25:53.039322+0000","last_became_peered":"2021-02-10T04:25:53.039322+0000","last_unstale":"2021-02-10T04:27:47.726157+0000","last_undegraded":"2021-02-10T04:27:47.726157+0000","last_fullsized":"2021-02-10T04:27:47.726157+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:21.548381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:21.548381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"40","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.117245+0000","last_change":"2021-02-10T04:27:05.297715+0000","last_active":"2021-02-10T04:27:47.117245+0000","last_peered":"2021-02-10T04:27:47.117245+0000","last_clean":"2021-02-10T04:27:47.117245+0000","last_became_active":"2021-02-10T04:25:51.029155+0000","last_became_peered":"2021-02-10T04:25:51.029155+0000","last_unstale":"2021-02-10T04:27:47.117245+0000","last_undegraded":"2021-02-10T04:27:47.117245+0000","last_fullsized":"2021-02-10T04:27:47.117245+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:27:05.297647+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"51","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.066395+0000","last_change":"2021-02-10T04:26:54.256587+0000","last_active":"2021-02-10T04:27:59.066395+0000","last_peered":"2021-02-10T04:27:59.066395+0000","last_clean":"2021-02-10T04:27:59.066395+0000","last_became_active":"2021-02-10T04:25:34.330070+0000","last_became_peered":"2021-02-10T04:25:34.330070+0000","last_unstale":"2021-02-10T04:27:59.066395+0000","last_undegraded":"2021-02-10T04:27:59.066395+0000","last_fullsized":"2021-02-10T04:27:59.066395+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:54.256498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:54.256498+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"62'15","reported_seq":"72","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.211122+0000","last_change":"2021-02-10T04:27:40.218910+0000","last_active":"2021-02-10T04:27:59.211122+0000","last_peered":"2021-02-10T04:27:59.211122+0000","last_clean":"2021-02-10T04:27:59.211122+0000","last_became_active":"2021-02-10T04:25:49.015543+0000","last_became_peered":"2021-02-10T04:25:49.015543+0000","last_unstale":"2021-02-10T04:27:59.211122+0000","last_undegraded":"2021-02-10T04:27:59.211122+0000","last_fullsized":"2021-02-10T04:27:59.211122+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:27:40.218828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:40.218828+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"40","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.016896+0000","last_change":"2021-02-10T04:27:28.373194+0000","last_active":"2021-02-10T04:27:59.016896+0000","last_peered":"2021-02-10T04:27:59.016896+0000","last_clean":"2021-02-10T04:27:59.016896+0000","last_became_active":"2021-02-10T04:25:53.031782+0000","last_became_peered":"2021-02-10T04:25:53.031782+0000","last_unstale":"2021-02-10T04:27:59.016896+0000","last_undegraded":"2021-02-10T04:27:59.016896+0000","last_fullsized":"2021-02-10T04:27:59.016896+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:28.373117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:28.373117+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"58'2","reported_seq":"69","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.020594+0000","last_change":"2021-02-10T04:27:26.264628+0000","last_active":"2021-02-10T04:27:59.020594+0000","last_peered":"2021-02-10T04:27:59.020594+0000","last_clean":"2021-02-10T04:27:59.020594+0000","last_became_active":"2021-02-10T04:25:51.018300+0000","last_became_peered":"2021-02-10T04:25:51.018300+0000","last_unstale":"2021-02-10T04:27:59.020594+0000","last_undegraded":"2021-02-10T04:27:59.020594+0000","last_fullsized":"2021-02-10T04:27:59.020594+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:26.264560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:26.264560+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"55","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.726487+0000","last_change":"2021-02-10T04:27:39.616474+0000","last_active":"2021-02-10T04:27:47.726487+0000","last_peered":"2021-02-10T04:27:47.726487+0000","last_clean":"2021-02-10T04:27:47.726487+0000","last_became_active":"2021-02-10T04:25:34.321096+0000","last_became_peered":"2021-02-10T04:25:34.321096+0000","last_unstale":"2021-02-10T04:27:47.726487+0000","last_undegraded":"2021-02-10T04:27:47.726487+0000","last_fullsized":"2021-02-10T04:27:47.726487+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:27:39.616381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"63'19","reported_seq":"83","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.212215+0000","last_change":"2021-02-10T04:27:11.016733+0000","last_active":"2021-02-10T04:27:59.212215+0000","last_peered":"2021-02-10T04:27:59.212215+0000","last_clean":"2021-02-10T04:27:59.212215+0000","last_became_active":"2021-02-10T04:25:49.010890+0000","last_became_peered":"2021-02-10T04:25:49.010890+0000","last_unstale":"2021-02-10T04:27:59.212215+0000","last_undegraded":"2021-02-10T04:27:59.212215+0000","last_fullsized":"2021-02-10T04:27:59.212215+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2021-02-10T04:27:11.016650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:11.016650+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":26,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"39","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.726666+0000","last_change":"2021-02-10T04:27:08.486638+0000","last_active":"2021-02-10T04:27:47.726666+0000","last_peered":"2021-02-10T04:27:47.726666+0000","last_clean":"2021-02-10T04:27:47.726666+0000","last_became_active":"2021-02-10T04:25:53.045177+0000","last_became_peered":"2021-02-10T04:25:53.045177+0000","last_unstale":"2021-02-10T04:27:47.726666+0000","last_undegraded":"2021-02-10T04:27:47.726666+0000","last_fullsized":"2021-02-10T04:27:47.726666+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:08.486553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:08.486553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"58'2","reported_seq":"72","reported_epoch":"86","state":"active+clean","last_fresh":"2021-02-10T04:28:07.215906+0000","last_change":"2021-02-10T04:27:20.157871+0000","last_active":"2021-02-10T04:28:07.215906+0000","last_peered":"2021-02-10T04:28:07.215906+0000","last_clean":"2021-02-10T04:28:07.215906+0000","last_became_active":"2021-02-10T04:25:51.032328+0000","last_became_peered":"2021-02-10T04:25:51.032328+0000","last_unstale":"2021-02-10T04:28:07.215906+0000","last_undegraded":"2021-02-10T04:28:07.215906+0000","last_fullsized":"2021-02-10T04:28:07.215906+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:20.157772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:20.157772+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"84'7","reported_seq":"71","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.210714+0000","last_change":"2021-02-10T04:27:54.402957+0000","last_active":"2021-02-10T04:27:59.210714+0000","last_peered":"2021-02-10T04:27:59.210714+0000","last_clean":"2021-02-10T04:27:59.210714+0000","last_became_active":"2021-02-10T04:25:36.334366+0000","last_became_peered":"2021-02-10T04:25:36.334366+0000","last_unstale":"2021-02-10T04:27:59.210714+0000","last_undegraded":"2021-02-10T04:27:59.210714+0000","last_fullsized":"2021-02-10T04:27:59.210714+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'7","last_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_deep_scrub":"84'7","last_deep_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_clean_scrub_stamp":"2021-02-10T04:27:54.402855+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"63'26","reported_seq":"128","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.117247+0000","last_change":"2021-02-10T04:27:28.213521+0000","last_active":"2021-02-10T04:27:47.117247+0000","last_peered":"2021-02-10T04:27:47.117247+0000","last_clean":"2021-02-10T04:27:47.117247+0000","last_became_active":"2021-02-10T04:26:39.003971+0000","last_became_peered":"2021-02-10T04:26:39.003971+0000","last_unstale":"2021-02-10T04:27:47.117247+0000","last_undegraded":"2021-02-10T04:27:47.117247+0000","last_fullsized":"2021-02-10T04:27:47.117247+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:27:28.213454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:28.213454+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"39","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.727312+0000","last_change":"2021-02-10T04:27:33.192375+0000","last_active":"2021-02-10T04:27:47.727312+0000","last_peered":"2021-02-10T04:27:47.727312+0000","last_clean":"2021-02-10T04:27:47.727312+0000","last_became_active":"2021-02-10T04:25:53.149820+0000","last_became_peered":"2021-02-10T04:25:53.149820+0000","last_unstale":"2021-02-10T04:27:47.727312+0000","last_undegraded":"2021-02-10T04:27:47.727312+0000","last_fullsized":"2021-02-10T04:27:47.727312+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:33.192287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:33.192287+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"42","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.210395+0000","last_change":"2021-02-10T04:27:23.236953+0000","last_active":"2021-02-10T04:27:59.210395+0000","last_peered":"2021-02-10T04:27:59.210395+0000","last_clean":"2021-02-10T04:27:59.210395+0000","last_became_active":"2021-02-10T04:25:51.031864+0000","last_became_peered":"2021-02-10T04:25:51.031864+0000","last_unstale":"2021-02-10T04:27:59.210395+0000","last_undegraded":"2021-02-10T04:27:59.210395+0000","last_fullsized":"2021-02-10T04:27:59.210395+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:23.236870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:23.236870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"53","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.066281+0000","last_change":"2021-02-10T04:27:14.083910+0000","last_active":"2021-02-10T04:27:59.066281+0000","last_peered":"2021-02-10T04:27:59.066281+0000","last_clean":"2021-02-10T04:27:59.066281+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:27:59.066281+0000","last_undegraded":"2021-02-10T04:27:59.066281+0000","last_fullsized":"2021-02-10T04:27:59.066281+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:27:14.083841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:14.083841+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"63'13","reported_seq":"89","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.117873+0000","last_change":"2021-02-10T04:26:50.393926+0000","last_active":"2021-02-10T04:27:47.117873+0000","last_peered":"2021-02-10T04:27:47.117873+0000","last_clean":"2021-02-10T04:27:47.117873+0000","last_became_active":"2021-02-10T04:26:39.003832+0000","last_became_peered":"2021-02-10T04:26:39.003832+0000","last_unstale":"2021-02-10T04:27:47.117873+0000","last_undegraded":"2021-02-10T04:27:47.117873+0000","last_fullsized":"2021-02-10T04:27:47.117873+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2021-02-10T04:26:50.393865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:26:50.393865+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"38","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.116842+0000","last_change":"2021-02-10T04:27:13.573977+0000","last_active":"2021-02-10T04:27:47.116842+0000","last_peered":"2021-02-10T04:27:47.116842+0000","last_clean":"2021-02-10T04:27:47.116842+0000","last_became_active":"2021-02-10T04:25:53.035867+0000","last_became_peered":"2021-02-10T04:25:53.035867+0000","last_unstale":"2021-02-10T04:27:47.116842+0000","last_undegraded":"2021-02-10T04:27:47.116842+0000","last_fullsized":"2021-02-10T04:27:47.116842+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:13.573912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:13.573912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"40","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.118015+0000","last_change":"2021-02-10T04:26:56.335818+0000","last_active":"2021-02-10T04:27:47.118015+0000","last_peered":"2021-02-10T04:27:47.118015+0000","last_clean":"2021-02-10T04:27:47.118015+0000","last_became_active":"2021-02-10T04:25:51.030651+0000","last_became_peered":"2021-02-10T04:25:51.030651+0000","last_unstale":"2021-02-10T04:27:47.118015+0000","last_undegraded":"2021-02-10T04:27:47.118015+0000","last_fullsized":"2021-02-10T04:27:47.118015+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:56.335735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:26:56.335735+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"52","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.116154+0000","last_change":"2021-02-10T04:26:57.639364+0000","last_active":"2021-02-10T04:27:47.116154+0000","last_peered":"2021-02-10T04:27:47.116154+0000","last_clean":"2021-02-10T04:27:47.116154+0000","last_became_active":"2021-02-10T04:25:34.329694+0000","last_became_peered":"2021-02-10T04:25:34.329694+0000","last_unstale":"2021-02-10T04:27:47.116154+0000","last_undegraded":"2021-02-10T04:27:47.116154+0000","last_fullsized":"2021-02-10T04:27:47.116154+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:26:57.639281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:57.639281+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"63'28","reported_seq":"103","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.726758+0000","last_change":"2021-02-10T04:27:24.523914+0000","last_active":"2021-02-10T04:27:47.726758+0000","last_peered":"2021-02-10T04:27:47.726758+0000","last_clean":"2021-02-10T04:27:47.726758+0000","last_became_active":"2021-02-10T04:25:49.008942+0000","last_became_peered":"2021-02-10T04:25:49.008942+0000","last_unstale":"2021-02-10T04:27:47.726758+0000","last_undegraded":"2021-02-10T04:27:47.726758+0000","last_fullsized":"2021-02-10T04:27:47.726758+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'28","last_scrub_stamp":"2021-02-10T04:27:24.523837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:24.523837+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":62,"num_read_kb":40,"num_write":29,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"38","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.117331+0000","last_change":"2021-02-10T04:27:27.259500+0000","last_active":"2021-02-10T04:27:47.117331+0000","last_peered":"2021-02-10T04:27:47.117331+0000","last_clean":"2021-02-10T04:27:47.117331+0000","last_became_active":"2021-02-10T04:25:53.047736+0000","last_became_peered":"2021-02-10T04:25:53.047736+0000","last_unstale":"2021-02-10T04:27:47.117331+0000","last_undegraded":"2021-02-10T04:27:47.117331+0000","last_fullsized":"2021-02-10T04:27:47.117331+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:27.259393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:27.259393+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"69","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.016657+0000","last_change":"2021-02-10T04:27:53.635318+0000","last_active":"2021-02-10T04:27:59.016657+0000","last_peered":"2021-02-10T04:27:59.016657+0000","last_clean":"2021-02-10T04:27:59.016657+0000","last_became_active":"2021-02-10T04:25:36.337394+0000","last_became_peered":"2021-02-10T04:25:36.337394+0000","last_unstale":"2021-02-10T04:27:59.016657+0000","last_undegraded":"2021-02-10T04:27:59.016657+0000","last_fullsized":"2021-02-10T04:27:59.016657+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:27:53.635228+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:27:53.635228+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"63'27","reported_seq":"103","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.016223+0000","last_change":"2021-02-10T04:27:17.352786+0000","last_active":"2021-02-10T04:27:59.016223+0000","last_peered":"2021-02-10T04:27:59.016223+0000","last_clean":"2021-02-10T04:27:59.016223+0000","last_became_active":"2021-02-10T04:25:49.013259+0000","last_became_peered":"2021-02-10T04:25:49.013259+0000","last_unstale":"2021-02-10T04:27:59.016223+0000","last_undegraded":"2021-02-10T04:27:59.016223+0000","last_fullsized":"2021-02-10T04:27:59.016223+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'27","last_scrub_stamp":"2021-02-10T04:27:17.352701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:17.352701+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":41,"num_write":29,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"40","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.116057+0000","last_change":"2021-02-10T04:27:14.530439+0000","last_active":"2021-02-10T04:27:47.116057+0000","last_peered":"2021-02-10T04:27:47.116057+0000","last_clean":"2021-02-10T04:27:47.116057+0000","last_became_active":"2021-02-10T04:25:51.023000+0000","last_became_peered":"2021-02-10T04:25:51.023000+0000","last_unstale":"2021-02-10T04:27:47.116057+0000","last_undegraded":"2021-02-10T04:27:47.116057+0000","last_fullsized":"2021-02-10T04:27:47.116057+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:14.530378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:14.530378+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"39","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.726153+0000","last_change":"2021-02-10T04:27:18.521962+0000","last_active":"2021-02-10T04:27:47.726153+0000","last_peered":"2021-02-10T04:27:47.726153+0000","last_clean":"2021-02-10T04:27:47.726153+0000","last_became_active":"2021-02-10T04:25:53.045215+0000","last_became_peered":"2021-02-10T04:25:53.045215+0000","last_unstale":"2021-02-10T04:27:47.726153+0000","last_undegraded":"2021-02-10T04:27:47.726153+0000","last_fullsized":"2021-02-10T04:27:47.726153+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:18.521897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:18.521897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"69","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.211046+0000","last_change":"2021-02-10T04:27:04.024909+0000","last_active":"2021-02-10T04:27:59.211046+0000","last_peered":"2021-02-10T04:27:59.211046+0000","last_clean":"2021-02-10T04:27:59.211046+0000","last_became_active":"2021-02-10T04:26:39.004318+0000","last_became_peered":"2021-02-10T04:26:39.004318+0000","last_unstale":"2021-02-10T04:27:59.211046+0000","last_undegraded":"2021-02-10T04:27:59.211046+0000","last_fullsized":"2021-02-10T04:27:59.211046+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:04.024848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:04.024848+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"79'34","reported_seq":"98","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.211210+0000","last_change":"2021-02-10T04:27:18.990156+0000","last_active":"2021-02-10T04:27:59.211210+0000","last_peered":"2021-02-10T04:27:59.211210+0000","last_clean":"2021-02-10T04:27:59.211210+0000","last_became_active":"2021-02-10T04:25:49.006147+0000","last_became_peered":"2021-02-10T04:25:49.006147+0000","last_unstale":"2021-02-10T04:27:59.211210+0000","last_undegraded":"2021-02-10T04:27:59.211210+0000","last_fullsized":"2021-02-10T04:27:59.211210+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'34","last_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:27:18.990088+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":22,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"58'2","reported_seq":"70","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.212122+0000","last_change":"2021-02-10T04:27:03.440844+0000","last_active":"2021-02-10T04:27:59.212122+0000","last_peered":"2021-02-10T04:27:59.212122+0000","last_clean":"2021-02-10T04:27:59.212122+0000","last_became_active":"2021-02-10T04:25:51.026251+0000","last_became_peered":"2021-02-10T04:25:51.026251+0000","last_unstale":"2021-02-10T04:27:59.212122+0000","last_undegraded":"2021-02-10T04:27:59.212122+0000","last_fullsized":"2021-02-10T04:27:59.212122+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:03.440780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:03.440780+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"38","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.116515+0000","last_change":"2021-02-10T04:27:37.196914+0000","last_active":"2021-02-10T04:27:47.116515+0000","last_peered":"2021-02-10T04:27:47.116515+0000","last_clean":"2021-02-10T04:27:47.116515+0000","last_became_active":"2021-02-10T04:25:53.049487+0000","last_became_peered":"2021-02-10T04:25:53.049487+0000","last_unstale":"2021-02-10T04:27:47.116515+0000","last_undegraded":"2021-02-10T04:27:47.116515+0000","last_fullsized":"2021-02-10T04:27:47.116515+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:37.196846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:37.196846+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"51","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.210806+0000","last_change":"2021-02-10T04:27:06.934347+0000","last_active":"2021-02-10T04:27:59.210806+0000","last_peered":"2021-02-10T04:27:59.210806+0000","last_clean":"2021-02-10T04:27:59.210806+0000","last_became_active":"2021-02-10T04:25:34.331779+0000","last_became_peered":"2021-02-10T04:25:34.331779+0000","last_unstale":"2021-02-10T04:27:59.210806+0000","last_undegraded":"2021-02-10T04:27:59.210806+0000","last_fullsized":"2021-02-10T04:27:59.210806+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:06.934282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:06.934282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"63'31","reported_seq":"111","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:28:03.352890+0000","last_change":"2021-02-10T04:28:03.352844+0000","last_active":"2021-02-10T04:28:03.352890+0000","last_peered":"2021-02-10T04:28:03.352890+0000","last_clean":"2021-02-10T04:28:03.352890+0000","last_became_active":"2021-02-10T04:25:49.007872+0000","last_became_peered":"2021-02-10T04:25:49.007872+0000","last_unstale":"2021-02-10T04:28:03.352890+0000","last_undegraded":"2021-02-10T04:28:03.352890+0000","last_fullsized":"2021-02-10T04:28:03.352890+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'31","last_scrub_stamp":"2021-02-10T04:28:03.352778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:03.352778+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":66,"num_read_kb":42,"num_write":34,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"41","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.726677+0000","last_change":"2021-02-10T04:27:31.650126+0000","last_active":"2021-02-10T04:27:47.726677+0000","last_peered":"2021-02-10T04:27:47.726677+0000","last_clean":"2021-02-10T04:27:47.726677+0000","last_became_active":"2021-02-10T04:25:51.027624+0000","last_became_peered":"2021-02-10T04:25:51.027624+0000","last_unstale":"2021-02-10T04:27:47.726677+0000","last_undegraded":"2021-02-10T04:27:47.726677+0000","last_fullsized":"2021-02-10T04:27:47.726677+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:27:31.650033+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"39","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.728371+0000","last_change":"2021-02-10T04:27:28.264288+0000","last_active":"2021-02-10T04:27:47.728371+0000","last_peered":"2021-02-10T04:27:47.728371+0000","last_clean":"2021-02-10T04:27:47.728371+0000","last_became_active":"2021-02-10T04:25:53.149677+0000","last_became_peered":"2021-02-10T04:25:53.149677+0000","last_unstale":"2021-02-10T04:27:47.728371+0000","last_undegraded":"2021-02-10T04:27:47.728371+0000","last_fullsized":"2021-02-10T04:27:47.728371+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:28.264204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:28.264204+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"183","reported_epoch":"86","state":"active+clean","last_fresh":"2021-02-10T04:28:06.747664+0000","last_change":"2021-02-10T04:26:58.989515+0000","last_active":"2021-02-10T04:28:06.747664+0000","last_peered":"2021-02-10T04:28:06.747664+0000","last_clean":"2021-02-10T04:28:06.747664+0000","last_became_active":"2021-02-10T04:25:49.013969+0000","last_became_peered":"2021-02-10T04:25:49.013969+0000","last_unstale":"2021-02-10T04:28:06.747664+0000","last_undegraded":"2021-02-10T04:28:06.747664+0000","last_fullsized":"2021-02-10T04:28:06.747664+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:26:58.989447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:26:58.989447+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":136,"num_read_kb":131,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"54","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.116267+0000","last_change":"2021-02-10T04:27:44.556805+0000","last_active":"2021-02-10T04:27:47.116267+0000","last_peered":"2021-02-10T04:27:47.116267+0000","last_clean":"2021-02-10T04:27:47.116267+0000","last_became_active":"2021-02-10T04:25:34.331972+0000","last_became_peered":"2021-02-10T04:25:34.331972+0000","last_unstale":"2021-02-10T04:27:47.116267+0000","last_undegraded":"2021-02-10T04:27:47.116267+0000","last_fullsized":"2021-02-10T04:27:47.116267+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:44.556741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:44.556741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"82'29","reported_seq":"98","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.211506+0000","last_change":"2021-02-10T04:27:38.176842+0000","last_active":"2021-02-10T04:27:59.211506+0000","last_peered":"2021-02-10T04:27:59.211506+0000","last_clean":"2021-02-10T04:27:59.211506+0000","last_became_active":"2021-02-10T04:25:49.002843+0000","last_became_peered":"2021-02-10T04:25:49.002843+0000","last_unstale":"2021-02-10T04:27:59.211506+0000","last_undegraded":"2021-02-10T04:27:59.211506+0000","last_fullsized":"2021-02-10T04:27:59.211506+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'29","last_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:27:38.176754+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":32,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"40","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.117251+0000","last_change":"2021-02-10T04:27:16.564733+0000","last_active":"2021-02-10T04:27:47.117251+0000","last_peered":"2021-02-10T04:27:47.117251+0000","last_clean":"2021-02-10T04:27:47.117251+0000","last_became_active":"2021-02-10T04:25:51.025306+0000","last_became_peered":"2021-02-10T04:25:51.025306+0000","last_unstale":"2021-02-10T04:27:47.117251+0000","last_undegraded":"2021-02-10T04:27:47.117251+0000","last_fullsized":"2021-02-10T04:27:47.117251+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:27:16.564638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"40","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.212460+0000","last_change":"2021-02-10T04:27:18.115804+0000","last_active":"2021-02-10T04:27:59.212460+0000","last_peered":"2021-02-10T04:27:59.212460+0000","last_clean":"2021-02-10T04:27:59.212460+0000","last_became_active":"2021-02-10T04:25:53.037506+0000","last_became_peered":"2021-02-10T04:25:53.037506+0000","last_unstale":"2021-02-10T04:27:59.212460+0000","last_undegraded":"2021-02-10T04:27:59.212460+0000","last_fullsized":"2021-02-10T04:27:59.212460+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:18.115741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:18.115741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"42","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.116822+0000","last_change":"2021-02-10T04:27:39.268187+0000","last_active":"2021-02-10T04:27:47.116822+0000","last_peered":"2021-02-10T04:27:47.116822+0000","last_clean":"2021-02-10T04:27:47.116822+0000","last_became_active":"2021-02-10T04:25:49.010614+0000","last_became_peered":"2021-02-10T04:25:49.010614+0000","last_unstale":"2021-02-10T04:27:47.116822+0000","last_undegraded":"2021-02-10T04:27:47.116822+0000","last_fullsized":"2021-02-10T04:27:47.116822+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:39.268114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:39.268114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"58","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.211663+0000","last_change":"2021-02-10T04:27:55.385179+0000","last_active":"2021-02-10T04:27:59.211663+0000","last_peered":"2021-02-10T04:27:59.211663+0000","last_clean":"2021-02-10T04:27:59.211663+0000","last_became_active":"2021-02-10T04:25:36.334230+0000","last_became_peered":"2021-02-10T04:25:36.334230+0000","last_unstale":"2021-02-10T04:27:59.211663+0000","last_undegraded":"2021-02-10T04:27:59.211663+0000","last_fullsized":"2021-02-10T04:27:59.211663+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:55.385094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:55.385094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"49","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.728049+0000","last_change":"2021-02-10T04:26:44.252455+0000","last_active":"2021-02-10T04:27:47.728049+0000","last_peered":"2021-02-10T04:27:47.728049+0000","last_clean":"2021-02-10T04:27:47.728049+0000","last_became_active":"2021-02-10T04:25:36.343397+0000","last_became_peered":"2021-02-10T04:25:36.343397+0000","last_unstale":"2021-02-10T04:27:47.728049+0000","last_undegraded":"2021-02-10T04:27:47.728049+0000","last_fullsized":"2021-02-10T04:27:47.728049+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:44.252351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:26:44.252351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"62'15","reported_seq":"68","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.066694+0000","last_change":"2021-02-10T04:27:08.069986+0000","last_active":"2021-02-10T04:27:59.066694+0000","last_peered":"2021-02-10T04:27:59.066694+0000","last_clean":"2021-02-10T04:27:59.066694+0000","last_became_active":"2021-02-10T04:25:49.002471+0000","last_became_peered":"2021-02-10T04:25:49.002471+0000","last_unstale":"2021-02-10T04:27:59.066694+0000","last_undegraded":"2021-02-10T04:27:59.066694+0000","last_fullsized":"2021-02-10T04:27:59.066694+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:27:08.069899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:08.069899+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":15,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"41","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.728159+0000","last_change":"2021-02-10T04:27:38.106291+0000","last_active":"2021-02-10T04:27:47.728159+0000","last_peered":"2021-02-10T04:27:47.728159+0000","last_clean":"2021-02-10T04:27:47.728159+0000","last_became_active":"2021-02-10T04:25:51.022821+0000","last_became_peered":"2021-02-10T04:25:51.022821+0000","last_unstale":"2021-02-10T04:27:47.728159+0000","last_undegraded":"2021-02-10T04:27:47.728159+0000","last_fullsized":"2021-02-10T04:27:47.728159+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:38.106209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:38.106209+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"40","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.210879+0000","last_change":"2021-02-10T04:27:10.051857+0000","last_active":"2021-02-10T04:27:59.210879+0000","last_peered":"2021-02-10T04:27:59.210879+0000","last_clean":"2021-02-10T04:27:59.210879+0000","last_became_active":"2021-02-10T04:25:53.040390+0000","last_became_peered":"2021-02-10T04:25:53.040390+0000","last_unstale":"2021-02-10T04:27:59.210879+0000","last_undegraded":"2021-02-10T04:27:59.210879+0000","last_fullsized":"2021-02-10T04:27:59.210879+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:10.051800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:10.051800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"43","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.726328+0000","last_change":"2021-02-10T04:27:30.657844+0000","last_active":"2021-02-10T04:27:47.726328+0000","last_peered":"2021-02-10T04:27:47.726328+0000","last_clean":"2021-02-10T04:27:47.726328+0000","last_became_active":"2021-02-10T04:25:49.001782+0000","last_became_peered":"2021-02-10T04:25:49.001782+0000","last_unstale":"2021-02-10T04:27:47.726328+0000","last_undegraded":"2021-02-10T04:27:47.726328+0000","last_fullsized":"2021-02-10T04:27:47.726328+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:30.657781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:30.657781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"48","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.727931+0000","last_change":"2021-02-10T04:27:08.192835+0000","last_active":"2021-02-10T04:27:47.727931+0000","last_peered":"2021-02-10T04:27:47.727931+0000","last_clean":"2021-02-10T04:27:47.727931+0000","last_became_active":"2021-02-10T04:25:36.340009+0000","last_became_peered":"2021-02-10T04:25:36.340009+0000","last_unstale":"2021-02-10T04:27:47.727931+0000","last_undegraded":"2021-02-10T04:27:47.727931+0000","last_fullsized":"2021-02-10T04:27:47.727931+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:08.192772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:08.192772+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"62'16","reported_seq":"69","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.726676+0000","last_change":"2021-02-10T04:27:09.484054+0000","last_active":"2021-02-10T04:27:47.726676+0000","last_peered":"2021-02-10T04:27:47.726676+0000","last_clean":"2021-02-10T04:27:47.726676+0000","last_became_active":"2021-02-10T04:25:49.003521+0000","last_became_peered":"2021-02-10T04:25:49.003521+0000","last_unstale":"2021-02-10T04:27:47.726676+0000","last_undegraded":"2021-02-10T04:27:47.726676+0000","last_fullsized":"2021-02-10T04:27:47.726676+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'16","last_scrub_stamp":"2021-02-10T04:27:09.483988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:09.483988+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":16,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"42","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.212415+0000","last_change":"2021-02-10T04:27:25.216335+0000","last_active":"2021-02-10T04:27:59.212415+0000","last_peered":"2021-02-10T04:27:59.212415+0000","last_clean":"2021-02-10T04:27:59.212415+0000","last_became_active":"2021-02-10T04:25:51.032201+0000","last_became_peered":"2021-02-10T04:25:51.032201+0000","last_unstale":"2021-02-10T04:27:59.212415+0000","last_undegraded":"2021-02-10T04:27:59.212415+0000","last_fullsized":"2021-02-10T04:27:59.212415+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:27:25.216260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"40","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.210247+0000","last_change":"2021-02-10T04:27:42.182019+0000","last_active":"2021-02-10T04:27:59.210247+0000","last_peered":"2021-02-10T04:27:59.210247+0000","last_clean":"2021-02-10T04:27:59.210247+0000","last_became_active":"2021-02-10T04:25:53.149552+0000","last_became_peered":"2021-02-10T04:25:53.149552+0000","last_unstale":"2021-02-10T04:27:59.210247+0000","last_undegraded":"2021-02-10T04:27:59.210247+0000","last_fullsized":"2021-02-10T04:27:59.210247+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:42.181940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:42.181940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"51","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.728436+0000","last_change":"2021-02-10T04:26:57.084942+0000","last_active":"2021-02-10T04:27:47.728436+0000","last_peered":"2021-02-10T04:27:47.728436+0000","last_clean":"2021-02-10T04:27:47.728436+0000","last_became_active":"2021-02-10T04:25:36.338600+0000","last_became_peered":"2021-02-10T04:25:36.338600+0000","last_unstale":"2021-02-10T04:27:47.728436+0000","last_undegraded":"2021-02-10T04:27:47.728436+0000","last_fullsized":"2021-02-10T04:27:47.728436+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:26:57.084864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:57.084864+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"63'27","reported_seq":"98","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.726458+0000","last_change":"2021-02-10T04:27:10.457048+0000","last_active":"2021-02-10T04:27:47.726458+0000","last_peered":"2021-02-10T04:27:47.726458+0000","last_clean":"2021-02-10T04:27:47.726458+0000","last_became_active":"2021-02-10T04:25:49.002937+0000","last_became_peered":"2021-02-10T04:25:49.002937+0000","last_unstale":"2021-02-10T04:27:47.726458+0000","last_undegraded":"2021-02-10T04:27:47.726458+0000","last_fullsized":"2021-02-10T04:27:47.726458+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'27","last_scrub_stamp":"2021-02-10T04:27:10.456981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:10.456981+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":38,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"42","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.066688+0000","last_change":"2021-02-10T04:26:59.175043+0000","last_active":"2021-02-10T04:27:59.066688+0000","last_peered":"2021-02-10T04:27:59.066688+0000","last_clean":"2021-02-10T04:27:59.066688+0000","last_became_active":"2021-02-10T04:25:51.025636+0000","last_became_peered":"2021-02-10T04:25:51.025636+0000","last_unstale":"2021-02-10T04:27:59.066688+0000","last_undegraded":"2021-02-10T04:27:59.066688+0000","last_fullsized":"2021-02-10T04:27:59.066688+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:59.174908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:26:59.174908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"45","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:28:05.641741+0000","last_change":"2021-02-10T04:28:05.641668+0000","last_active":"2021-02-10T04:28:05.641741+0000","last_peered":"2021-02-10T04:28:05.641741+0000","last_clean":"2021-02-10T04:28:05.641741+0000","last_became_active":"2021-02-10T04:25:53.045085+0000","last_became_peered":"2021-02-10T04:25:53.045085+0000","last_unstale":"2021-02-10T04:28:05.641741+0000","last_undegraded":"2021-02-10T04:28:05.641741+0000","last_fullsized":"2021-02-10T04:28:05.641741+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:05.641577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:05.641577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"55","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.211399+0000","last_change":"2021-02-10T04:27:25.026606+0000","last_active":"2021-02-10T04:27:59.211399+0000","last_peered":"2021-02-10T04:27:59.211399+0000","last_clean":"2021-02-10T04:27:59.211399+0000","last_became_active":"2021-02-10T04:25:35.324960+0000","last_became_peered":"2021-02-10T04:25:35.324960+0000","last_unstale":"2021-02-10T04:27:59.211399+0000","last_undegraded":"2021-02-10T04:27:59.211399+0000","last_fullsized":"2021-02-10T04:27:59.211399+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:27:25.026534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:25.026534+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"63'34","reported_seq":"112","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.116356+0000","last_change":"2021-02-10T04:27:20.342445+0000","last_active":"2021-02-10T04:27:47.116356+0000","last_peered":"2021-02-10T04:27:47.116356+0000","last_clean":"2021-02-10T04:27:47.116356+0000","last_became_active":"2021-02-10T04:25:48.997481+0000","last_became_peered":"2021-02-10T04:25:48.997481+0000","last_unstale":"2021-02-10T04:27:47.116356+0000","last_undegraded":"2021-02-10T04:27:47.116356+0000","last_fullsized":"2021-02-10T04:27:47.116356+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'34","last_scrub_stamp":"2021-02-10T04:27:20.342358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:20.342358+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":70,"num_read_kb":48,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"42","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.016604+0000","last_change":"2021-02-10T04:27:44.501111+0000","last_active":"2021-02-10T04:27:59.016604+0000","last_peered":"2021-02-10T04:27:59.016604+0000","last_clean":"2021-02-10T04:27:59.016604+0000","last_became_active":"2021-02-10T04:25:51.018706+0000","last_became_peered":"2021-02-10T04:25:51.018706+0000","last_unstale":"2021-02-10T04:27:59.016604+0000","last_undegraded":"2021-02-10T04:27:59.016604+0000","last_fullsized":"2021-02-10T04:27:59.016604+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:44.501024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:44.501024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"39","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.727605+0000","last_change":"2021-02-10T04:27:26.234432+0000","last_active":"2021-02-10T04:27:47.727605+0000","last_peered":"2021-02-10T04:27:47.727605+0000","last_clean":"2021-02-10T04:27:47.727605+0000","last_became_active":"2021-02-10T04:25:53.150930+0000","last_became_peered":"2021-02-10T04:25:53.150930+0000","last_unstale":"2021-02-10T04:27:47.727605+0000","last_undegraded":"2021-02-10T04:27:47.727605+0000","last_fullsized":"2021-02-10T04:27:47.727605+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:26.234349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:26.234349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"60","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.066779+0000","last_change":"2021-02-10T04:26:44.264500+0000","last_active":"2021-02-10T04:27:59.066779+0000","last_peered":"2021-02-10T04:27:59.066779+0000","last_clean":"2021-02-10T04:27:59.066779+0000","last_became_active":"2021-02-10T04:25:36.339614+0000","last_became_peered":"2021-02-10T04:25:36.339614+0000","last_unstale":"2021-02-10T04:27:59.066779+0000","last_undegraded":"2021-02-10T04:27:59.066779+0000","last_fullsized":"2021-02-10T04:27:59.066779+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:44.264402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:44.264402+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"63'26","reported_seq":"92","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.016218+0000","last_change":"2021-02-10T04:27:26.403830+0000","last_active":"2021-02-10T04:27:59.016218+0000","last_peered":"2021-02-10T04:27:59.016218+0000","last_clean":"2021-02-10T04:27:59.016218+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:27:59.016218+0000","last_undegraded":"2021-02-10T04:27:59.016218+0000","last_fullsized":"2021-02-10T04:27:59.016218+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:27:26.403748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:26.403748+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":51,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"58'2","reported_seq":"69","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.018953+0000","last_change":"2021-02-10T04:27:33.210495+0000","last_active":"2021-02-10T04:27:59.018953+0000","last_peered":"2021-02-10T04:27:59.018953+0000","last_clean":"2021-02-10T04:27:59.018953+0000","last_became_active":"2021-02-10T04:25:51.031906+0000","last_became_peered":"2021-02-10T04:25:51.031906+0000","last_unstale":"2021-02-10T04:27:59.018953+0000","last_undegraded":"2021-02-10T04:27:59.018953+0000","last_fullsized":"2021-02-10T04:27:59.018953+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:33.210431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:33.210431+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"38","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:52.383310+0000","last_change":"2021-02-10T04:27:52.383262+0000","last_active":"2021-02-10T04:27:52.383310+0000","last_peered":"2021-02-10T04:27:52.383310+0000","last_clean":"2021-02-10T04:27:52.383310+0000","last_became_active":"2021-02-10T04:25:53.148054+0000","last_became_peered":"2021-02-10T04:25:53.148054+0000","last_unstale":"2021-02-10T04:27:52.383310+0000","last_undegraded":"2021-02-10T04:27:52.383310+0000","last_fullsized":"2021-02-10T04:27:52.383310+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:52.383188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:52.383188+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"47","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:28:02.288845+0000","last_change":"2021-02-10T04:28:02.288782+0000","last_active":"2021-02-10T04:28:02.288845+0000","last_peered":"2021-02-10T04:28:02.288845+0000","last_clean":"2021-02-10T04:28:02.288845+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:28:02.288845+0000","last_undegraded":"2021-02-10T04:28:02.288845+0000","last_fullsized":"2021-02-10T04:28:02.288845+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:02.288683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:02.288683+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"48","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.726741+0000","last_change":"2021-02-10T04:27:04.148397+0000","last_active":"2021-02-10T04:27:47.726741+0000","last_peered":"2021-02-10T04:27:47.726741+0000","last_clean":"2021-02-10T04:27:47.726741+0000","last_became_active":"2021-02-10T04:25:36.340895+0000","last_became_peered":"2021-02-10T04:25:36.340895+0000","last_unstale":"2021-02-10T04:27:47.726741+0000","last_undegraded":"2021-02-10T04:27:47.726741+0000","last_fullsized":"2021-02-10T04:27:47.726741+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:04.148329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:04.148329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"63'21","reported_seq":"86","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.016918+0000","last_change":"2021-02-10T04:26:59.205120+0000","last_active":"2021-02-10T04:27:59.016918+0000","last_peered":"2021-02-10T04:27:59.016918+0000","last_clean":"2021-02-10T04:27:59.016918+0000","last_became_active":"2021-02-10T04:25:49.011093+0000","last_became_peered":"2021-02-10T04:25:49.011093+0000","last_unstale":"2021-02-10T04:27:59.016918+0000","last_undegraded":"2021-02-10T04:27:59.016918+0000","last_fullsized":"2021-02-10T04:27:59.016918+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2021-02-10T04:26:59.205037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:26:59.205037+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":28,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"40","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.212064+0000","last_change":"2021-02-10T04:27:24.209544+0000","last_active":"2021-02-10T04:27:59.212064+0000","last_peered":"2021-02-10T04:27:59.212064+0000","last_clean":"2021-02-10T04:27:59.212064+0000","last_became_active":"2021-02-10T04:25:53.042697+0000","last_became_peered":"2021-02-10T04:25:53.042697+0000","last_unstale":"2021-02-10T04:27:59.212064+0000","last_undegraded":"2021-02-10T04:27:59.212064+0000","last_fullsized":"2021-02-10T04:27:59.212064+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:27:24.209460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"78'3","reported_seq":"72","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:28:02.728252+0000","last_change":"2021-02-10T04:27:13.268467+0000","last_active":"2021-02-10T04:28:02.728252+0000","last_peered":"2021-02-10T04:28:02.728252+0000","last_clean":"2021-02-10T04:28:02.728252+0000","last_became_active":"2021-02-10T04:25:51.025218+0000","last_became_peered":"2021-02-10T04:25:51.025218+0000","last_unstale":"2021-02-10T04:28:02.728252+0000","last_undegraded":"2021-02-10T04:28:02.728252+0000","last_fullsized":"2021-02-10T04:28:02.728252+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:27:13.268412+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"55","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.210611+0000","last_change":"2021-02-10T04:26:46.092082+0000","last_active":"2021-02-10T04:27:59.210611+0000","last_peered":"2021-02-10T04:27:59.210611+0000","last_clean":"2021-02-10T04:27:59.210611+0000","last_became_active":"2021-02-10T04:25:36.331437+0000","last_became_peered":"2021-02-10T04:25:36.331437+0000","last_unstale":"2021-02-10T04:27:59.210611+0000","last_undegraded":"2021-02-10T04:27:59.210611+0000","last_fullsized":"2021-02-10T04:27:59.210611+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:46.092013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:46.092013+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"62'17","reported_seq":"73","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.211422+0000","last_change":"2021-02-10T04:27:30.012988+0000","last_active":"2021-02-10T04:27:59.211422+0000","last_peered":"2021-02-10T04:27:59.211422+0000","last_clean":"2021-02-10T04:27:59.211422+0000","last_became_active":"2021-02-10T04:25:49.004933+0000","last_became_peered":"2021-02-10T04:25:49.004933+0000","last_unstale":"2021-02-10T04:27:59.211422+0000","last_undegraded":"2021-02-10T04:27:59.211422+0000","last_fullsized":"2021-02-10T04:27:59.211422+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'17","last_scrub_stamp":"2021-02-10T04:27:30.012927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:30.012927+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"40","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.016799+0000","last_change":"2021-02-10T04:27:33.382403+0000","last_active":"2021-02-10T04:27:59.016799+0000","last_peered":"2021-02-10T04:27:59.016799+0000","last_clean":"2021-02-10T04:27:59.016799+0000","last_became_active":"2021-02-10T04:25:53.043027+0000","last_became_peered":"2021-02-10T04:25:53.043027+0000","last_unstale":"2021-02-10T04:27:59.016799+0000","last_undegraded":"2021-02-10T04:27:59.016799+0000","last_fullsized":"2021-02-10T04:27:59.016799+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:33.382319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:33.382319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"58'2","reported_seq":"69","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.018759+0000","last_change":"2021-02-10T04:26:50.624141+0000","last_active":"2021-02-10T04:27:59.018759+0000","last_peered":"2021-02-10T04:27:59.018759+0000","last_clean":"2021-02-10T04:27:59.018759+0000","last_became_active":"2021-02-10T04:25:51.025268+0000","last_became_peered":"2021-02-10T04:25:51.025268+0000","last_unstale":"2021-02-10T04:27:59.018759+0000","last_undegraded":"2021-02-10T04:27:59.018759+0000","last_fullsized":"2021-02-10T04:27:59.018759+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:26:50.624072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:26:50.624072+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"53","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.211036+0000","last_change":"2021-02-10T04:27:17.003396+0000","last_active":"2021-02-10T04:27:59.211036+0000","last_peered":"2021-02-10T04:27:59.211036+0000","last_clean":"2021-02-10T04:27:59.211036+0000","last_became_active":"2021-02-10T04:25:36.333883+0000","last_became_peered":"2021-02-10T04:25:36.333883+0000","last_unstale":"2021-02-10T04:27:59.211036+0000","last_undegraded":"2021-02-10T04:27:59.211036+0000","last_fullsized":"2021-02-10T04:27:59.211036+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:17.003304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:17.003304+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"62'13","reported_seq":"65","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.016437+0000","last_change":"2021-02-10T04:27:14.256603+0000","last_active":"2021-02-10T04:27:59.016437+0000","last_peered":"2021-02-10T04:27:59.016437+0000","last_clean":"2021-02-10T04:27:59.016437+0000","last_became_active":"2021-02-10T04:25:49.001725+0000","last_became_peered":"2021-02-10T04:25:49.001725+0000","last_unstale":"2021-02-10T04:27:59.016437+0000","last_undegraded":"2021-02-10T04:27:59.016437+0000","last_fullsized":"2021-02-10T04:27:59.016437+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'13","last_scrub_stamp":"2021-02-10T04:27:14.256539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:14.256539+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"38","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.117572+0000","last_change":"2021-02-10T04:26:55.627032+0000","last_active":"2021-02-10T04:27:47.117572+0000","last_peered":"2021-02-10T04:27:47.117572+0000","last_clean":"2021-02-10T04:27:47.117572+0000","last_became_active":"2021-02-10T04:25:53.038218+0000","last_became_peered":"2021-02-10T04:25:53.038218+0000","last_unstale":"2021-02-10T04:27:47.117572+0000","last_undegraded":"2021-02-10T04:27:47.117572+0000","last_fullsized":"2021-02-10T04:27:47.117572+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:55.626967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:26:55.626967+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"42","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.209911+0000","last_change":"2021-02-10T04:27:37.160395+0000","last_active":"2021-02-10T04:27:59.209911+0000","last_peered":"2021-02-10T04:27:59.209911+0000","last_clean":"2021-02-10T04:27:59.209911+0000","last_became_active":"2021-02-10T04:25:51.028302+0000","last_became_peered":"2021-02-10T04:25:51.028302+0000","last_unstale":"2021-02-10T04:27:59.209911+0000","last_undegraded":"2021-02-10T04:27:59.209911+0000","last_fullsized":"2021-02-10T04:27:59.209911+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:37.160328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:37.160328+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"48","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.727676+0000","last_change":"2021-02-10T04:27:00.052115+0000","last_active":"2021-02-10T04:27:47.727676+0000","last_peered":"2021-02-10T04:27:47.727676+0000","last_clean":"2021-02-10T04:27:47.727676+0000","last_became_active":"2021-02-10T04:25:36.344109+0000","last_became_peered":"2021-02-10T04:25:36.344109+0000","last_unstale":"2021-02-10T04:27:47.727676+0000","last_undegraded":"2021-02-10T04:27:47.727676+0000","last_fullsized":"2021-02-10T04:27:47.727676+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:00.052028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:00.052028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"62'21","reported_seq":"80","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.017042+0000","last_change":"2021-02-10T04:27:00.215115+0000","last_active":"2021-02-10T04:27:59.017042+0000","last_peered":"2021-02-10T04:27:59.017042+0000","last_clean":"2021-02-10T04:27:59.017042+0000","last_became_active":"2021-02-10T04:25:49.011699+0000","last_became_peered":"2021-02-10T04:25:49.011699+0000","last_unstale":"2021-02-10T04:27:59.017042+0000","last_undegraded":"2021-02-10T04:27:59.017042+0000","last_fullsized":"2021-02-10T04:27:59.017042+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'21","last_scrub_stamp":"2021-02-10T04:27:00.215037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:00.215037+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"39","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.726605+0000","last_change":"2021-02-10T04:27:35.166330+0000","last_active":"2021-02-10T04:27:47.726605+0000","last_peered":"2021-02-10T04:27:47.726605+0000","last_clean":"2021-02-10T04:27:47.726605+0000","last_became_active":"2021-02-10T04:25:53.037903+0000","last_became_peered":"2021-02-10T04:25:53.037903+0000","last_unstale":"2021-02-10T04:27:47.726605+0000","last_undegraded":"2021-02-10T04:27:47.726605+0000","last_fullsized":"2021-02-10T04:27:47.726605+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:35.166250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:35.166250+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"40","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.116384+0000","last_change":"2021-02-10T04:27:11.606664+0000","last_active":"2021-02-10T04:27:47.116384+0000","last_peered":"2021-02-10T04:27:47.116384+0000","last_clean":"2021-02-10T04:27:47.116384+0000","last_became_active":"2021-02-10T04:25:51.025882+0000","last_became_peered":"2021-02-10T04:25:51.025882+0000","last_unstale":"2021-02-10T04:27:47.116384+0000","last_undegraded":"2021-02-10T04:27:47.116384+0000","last_fullsized":"2021-02-10T04:27:47.116384+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:11.606603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:11.606603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"62'2","reported_seq":"77","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:28:00.167937+0000","last_change":"2021-02-10T04:26:48.599352+0000","last_active":"2021-02-10T04:28:00.167937+0000","last_peered":"2021-02-10T04:28:00.167937+0000","last_clean":"2021-02-10T04:28:00.167937+0000","last_became_active":"2021-02-10T04:25:34.331199+0000","last_became_peered":"2021-02-10T04:25:34.331199+0000","last_unstale":"2021-02-10T04:28:00.167937+0000","last_undegraded":"2021-02-10T04:28:00.167937+0000","last_fullsized":"2021-02-10T04:28:00.167937+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2021-02-10T04:26:48.599257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:48.599257+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"63'29","reported_seq":"100","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.066583+0000","last_change":"2021-02-10T04:27:13.081465+0000","last_active":"2021-02-10T04:27:59.066583+0000","last_peered":"2021-02-10T04:27:59.066583+0000","last_clean":"2021-02-10T04:27:59.066583+0000","last_became_active":"2021-02-10T04:25:49.015913+0000","last_became_peered":"2021-02-10T04:25:49.015913+0000","last_unstale":"2021-02-10T04:27:59.066583+0000","last_undegraded":"2021-02-10T04:27:59.066583+0000","last_fullsized":"2021-02-10T04:27:59.066583+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'29","last_scrub_stamp":"2021-02-10T04:27:13.081399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:13.081399+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":56,"num_read_kb":36,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"39","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.726228+0000","last_change":"2021-02-10T04:27:22.512316+0000","last_active":"2021-02-10T04:27:47.726228+0000","last_peered":"2021-02-10T04:27:47.726228+0000","last_clean":"2021-02-10T04:27:47.726228+0000","last_became_active":"2021-02-10T04:25:53.038249+0000","last_became_peered":"2021-02-10T04:25:53.038249+0000","last_unstale":"2021-02-10T04:27:47.726228+0000","last_undegraded":"2021-02-10T04:27:47.726228+0000","last_fullsized":"2021-02-10T04:27:47.726228+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:22.512245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:22.512245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"45","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:56.492853+0000","last_change":"2021-02-10T04:27:56.492811+0000","last_active":"2021-02-10T04:27:56.492853+0000","last_peered":"2021-02-10T04:27:56.492853+0000","last_clean":"2021-02-10T04:27:56.492853+0000","last_became_active":"2021-02-10T04:25:51.016118+0000","last_became_peered":"2021-02-10T04:25:51.016118+0000","last_unstale":"2021-02-10T04:27:56.492853+0000","last_undegraded":"2021-02-10T04:27:56.492853+0000","last_fullsized":"2021-02-10T04:27:56.492853+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:56.492754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:56.492754+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"51","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.211136+0000","last_change":"2021-02-10T04:26:50.001018+0000","last_active":"2021-02-10T04:27:59.211136+0000","last_peered":"2021-02-10T04:27:59.211136+0000","last_clean":"2021-02-10T04:27:59.211136+0000","last_became_active":"2021-02-10T04:25:34.334427+0000","last_became_peered":"2021-02-10T04:25:34.334427+0000","last_unstale":"2021-02-10T04:27:59.211136+0000","last_undegraded":"2021-02-10T04:27:59.211136+0000","last_fullsized":"2021-02-10T04:27:59.211136+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:50.000952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:50.000952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"63'20","reported_seq":"84","reported_epoch":"84","state":"active+clean","last_fresh":"2021-02-10T04:27:47.726550+0000","last_change":"2021-02-10T04:27:20.587671+0000","last_active":"2021-02-10T04:27:47.726550+0000","last_peered":"2021-02-10T04:27:47.726550+0000","last_clean":"2021-02-10T04:27:47.726550+0000","last_became_active":"2021-02-10T04:25:49.004346+0000","last_became_peered":"2021-02-10T04:25:49.004346+0000","last_unstale":"2021-02-10T04:27:47.726550+0000","last_undegraded":"2021-02-10T04:27:47.726550+0000","last_fullsized":"2021-02-10T04:27:47.726550+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'20","last_scrub_stamp":"2021-02-10T04:27:20.587602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:20.587602+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":27,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"40","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.210618+0000","last_change":"2021-02-10T04:27:13.006848+0000","last_active":"2021-02-10T04:27:59.210618+0000","last_peered":"2021-02-10T04:27:59.210618+0000","last_clean":"2021-02-10T04:27:59.210618+0000","last_became_active":"2021-02-10T04:25:53.148090+0000","last_became_peered":"2021-02-10T04:25:53.148090+0000","last_unstale":"2021-02-10T04:27:59.210618+0000","last_undegraded":"2021-02-10T04:27:59.210618+0000","last_fullsized":"2021-02-10T04:27:59.210618+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:27:13.006716+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"68","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:28:02.435270+0000","last_change":"2021-02-10T04:28:02.435270+0000","last_active":"2021-02-10T04:28:02.435270+0000","last_peered":"2021-02-10T04:28:02.435270+0000","last_clean":"2021-02-10T04:28:02.435270+0000","last_became_active":"2021-02-10T04:25:36.344641+0000","last_became_peered":"2021-02-10T04:25:36.344641+0000","last_unstale":"2021-02-10T04:28:02.435270+0000","last_undegraded":"2021-02-10T04:28:02.435270+0000","last_fullsized":"2021-02-10T04:28:02.435270+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:28:02.435203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:02.435203+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"63'30","reported_seq":"105","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.211094+0000","last_change":"2021-02-10T04:27:02.039902+0000","last_active":"2021-02-10T04:27:59.211094+0000","last_peered":"2021-02-10T04:27:59.211094+0000","last_clean":"2021-02-10T04:27:59.211094+0000","last_became_active":"2021-02-10T04:25:49.002058+0000","last_became_peered":"2021-02-10T04:25:49.002058+0000","last_unstale":"2021-02-10T04:27:59.211094+0000","last_undegraded":"2021-02-10T04:27:59.211094+0000","last_fullsized":"2021-02-10T04:27:59.211094+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2021-02-10T04:27:02.039813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:02.039813+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":59,"num_read_kb":42,"num_write":31,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"58'2","reported_seq":"70","reported_epoch":"85","state":"active+clean","last_fresh":"2021-02-10T04:27:59.210252+0000","last_change":"2021-02-10T04:27:20.144947+0000","last_active":"2021-02-10T04:27:59.210252+0000","last_peered":"2021-02-10T04:27:59.210252+0000","last_clean":"2021-02-10T04:27:59.210252+0000","last_became_active":"2021-02-10T04:25:51.019515+0000","last_became_peered":"2021-02-10T04:25:51.019515+0000","last_unstale":"2021-02-10T04:27:59.210252+0000","last_undegraded":"2021-02-10T04:27:59.210252+0000","last_fullsized":"2021-02-10T04:27:59.210252+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:20.144855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:20.144855+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":1453,"num_read_kb":926,"num_write":734,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":746,"ondisk_log_size":746,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":136,"num_read_kb":131,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1945,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":21,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":5835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22,"ondisk_log_size":22,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748364832,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6576,"kb_used_data":872,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738576,"statfs":{"total":95995035648,"available":95988301824,"internally_reserved":0,"allocated":892928,"data_stored":319510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.876,"15min":0.88200000000000001},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":1.3149999999999999},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.85599999999999998,"15min":0.85899999999999999},"min":{"1min":0.57499999999999996,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.248,"5min":1.248,"15min":1.248},"last":0.98499999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.88700000000000001,"15min":0.89100000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.871},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.0149999999999999,"15min":1.0209999999999999},"min":{"1min":0.64500000000000002,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.355,"5min":1.369,"15min":1.369},"last":1.0609999999999999}]},{"osd":2,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.018,"15min":1.026},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.95799999999999996,"15min":0.95999999999999996},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":1.115}]},{"osd":3,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.998,"15min":1.01},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.147,"15min":1.153},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.381,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":1.198}]},{"osd":4,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":0.97499999999999998,"15min":0.95099999999999996},"min":{"1min":0.77300000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.2530000000000001,"5min":2.2530000000000001,"15min":2.2530000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.1060000000000001,"15min":1.1120000000000001},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.492,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":1.1479999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.161,"15min":1.1499999999999999},"min":{"1min":0.68899999999999995,"5min":0.624,"15min":0.624},"max":{"1min":1.714,"5min":1.714,"15min":1.714},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.2,"15min":1.1990000000000001},"min":{"1min":0.90900000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.498,"5min":1.538,"15min":1.538},"last":1.3799999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.246,"15min":1.2430000000000001},"min":{"1min":0.94199999999999995,"5min":0.86899999999999999,"15min":0.86899999999999999},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.276,"5min":1.236,"15min":1.2290000000000001},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":1.2909999999999999}]}]},{"osd":6,"up_from":43,"seq":184683593765,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6632,"kb_used_data":928,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738520,"statfs":{"total":95995035648,"available":95988244480,"internally_reserved":0,"allocated":950272,"data_stored":328570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:18 2021","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.3460000000000001,"5min":1.478,"15min":1.478},"last":1.262},{"interface":"front","average":{"1min":0.95499999999999996,"5min":1.008,"15min":1.0169999999999999},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.82899999999999996}]},{"osd":1,"last update":"Wed Feb 10 04:27:18 2021","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":1.0149999999999999,"15min":1.03},"min":{"1min":0.61599999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":2.121,"5min":2.121,"15min":2.121},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.042,"5min":0.99099999999999999,"15min":0.98299999999999998},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":1.1359999999999999}]},{"osd":2,"last update":"Wed Feb 10 04:27:18 2021","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0620000000000001,"15min":1.0609999999999999},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":1.052},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.0289999999999999,"15min":1.014},"min":{"1min":0.73699999999999999,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.1920000000000002,"5min":2.1920000000000002,"15min":2.1920000000000002},"last":1.103}]},{"osd":3,"last update":"Wed Feb 10 04:27:18 2021","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.1439999999999999,"15min":1.1379999999999999},"min":{"1min":0.85699999999999998,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":2,"5min":2,"15min":2},"last":0.97099999999999997},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.133,"15min":1.107},"min":{"1min":0.88,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":2.323,"5min":2.323,"15min":2.323},"last":1.1870000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:27:18 2021","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":0.99099999999999999,"15min":0.96099999999999997},"min":{"1min":0.61699999999999999,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.0800000000000001,"5min":2.0800000000000001,"15min":2.0800000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.90500000000000003,"15min":0.89200000000000002},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.76600000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:27:18 2021","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.0009999999999999,"15min":0.98699999999999999},"min":{"1min":0.67100000000000004,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.7130000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.419,"5min":1.179,"15min":1.139},"min":{"1min":0.83999999999999997,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.4550000000000001,"5min":2.4550000000000001,"15min":2.4550000000000001},"last":1.333}]},{"osd":7,"last update":"Wed Feb 10 04:27:40 2021","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.1060000000000001,"15min":1.0940000000000001},"min":{"1min":0.75800000000000001,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.901,"5min":1.901,"15min":1.901},"last":1.075},{"interface":"front","average":{"1min":1.506,"5min":1.409,"15min":1.393},"min":{"1min":0.90300000000000002,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":2.5339999999999998,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":1.401}]}]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":11328,"kb_used_data":1016,"kb_used_omap":0,"kb_used_meta":10304,"kb_avail":93733824,"statfs":{"total":95995035648,"available":95983435776,"internally_reserved":0,"allocated":1040384,"data_stored":331650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":10551296},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:56 2021","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.69099999999999995,"15min":0.63400000000000001},"min":{"1min":0.51800000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.179,"5min":1.425,"15min":1.425},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.78400000000000003,"15min":0.70499999999999996},"min":{"1min":0.54200000000000004,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":2.391,"5min":2.391,"15min":2.391},"last":0.89300000000000002}]},{"osd":2,"last update":"Wed Feb 10 04:27:14 2021","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.80800000000000005,"15min":0.76400000000000001},"min":{"1min":0.58199999999999996,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":1.1930000000000001},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.83499999999999996,"15min":0.79900000000000004},"min":{"1min":0.51000000000000001,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":0.82499999999999996}]},{"osd":3,"last update":"Wed Feb 10 04:27:31 2021","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.93100000000000005,"15min":0.91400000000000003},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.5860000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.026,"5min":1.018,"15min":1.014},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.226,"5min":2.226,"15min":2.226},"last":1.0469999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:27:46 2021","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.157,"15min":1.1459999999999999},"min":{"1min":0.86699999999999999,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":1.736,"5min":2.004,"15min":2.004},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.23,"5min":1.218,"15min":1.21},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.6419999999999999,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":1.409}]},{"osd":5,"last update":"Wed Feb 10 04:27:08 2021","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.25,"15min":1.2769999999999999},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":2.7170000000000001,"5min":5.5860000000000003,"15min":5.5860000000000003},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.6100000000000001,"15min":1.6559999999999999},"min":{"1min":0.97399999999999998,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":2.105,"5min":9.5310000000000006,"15min":9.5310000000000006},"last":1.246}]},{"osd":6,"last update":"Wed Feb 10 04:27:24 2021","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.2889999999999999,"15min":1.29},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.4449999999999998,"5min":2.4449999999999998,"15min":2.4449999999999998},"last":1.5900000000000001},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.264,"15min":1.292},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.905,"5min":3.653,"15min":3.653},"last":1.111}]},{"osd":7,"last update":"Wed Feb 10 04:27:36 2021","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.639,"15min":1.6890000000000001},"min":{"1min":1.0529999999999999,"5min":1.004,"15min":1.004},"max":{"1min":2.2090000000000001,"5min":7.5540000000000003,"15min":7.5540000000000003},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.623,"15min":1.6619999999999999},"min":{"1min":1.006,"5min":1.006,"15min":1.006},"max":{"1min":2.556,"5min":8.1020000000000003,"15min":8.1020000000000003},"last":1.716}]}]},{"osd":0,"up_from":8,"seq":34359738427,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6884,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":6016,"kb_avail":93738268,"statfs":{"total":95995035648,"available":95987986432,"internally_reserved":0,"allocated":880640,"data_stored":319219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6160384},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 04:27:55 2021","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.67800000000000005,"15min":0.59799999999999998},"min":{"1min":0.502,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.91600000000000004,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.81000000000000005,"15min":0.73899999999999999},"min":{"1min":0.63,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.0740000000000001,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":0.83899999999999997}]},{"osd":2,"last update":"Wed Feb 10 04:27:12 2021","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.79300000000000004,"15min":0.77300000000000002},"min":{"1min":0.52500000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.958,"5min":1.958,"15min":1.958},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.79700000000000004,"15min":0.79300000000000004},"min":{"1min":0.56100000000000005,"5min":0.495,"15min":0.495},"max":{"1min":1.4390000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.69099999999999995}]},{"osd":3,"last update":"Wed Feb 10 04:27:27 2021","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.80400000000000005,"15min":0.81200000000000006},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.224,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.89500000000000002,"5min":1.026,"15min":1.036},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.3049999999999999,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.95399999999999996}]},{"osd":4,"last update":"Wed Feb 10 04:27:45 2021","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.1259999999999999,"15min":1.169},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.3859999999999999,"5min":1.613,"15min":1.613},"last":1.448},{"interface":"front","average":{"1min":1.089,"5min":1.141,"15min":1.1539999999999999},"min":{"1min":0.76600000000000001,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.522,"5min":1.597,"15min":1.597},"last":1.296}]},{"osd":5,"last update":"Wed Feb 10 04:28:02 2021","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.1970000000000001,"15min":1.218},"min":{"1min":0.75800000000000001,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.593,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.216,"5min":1.2709999999999999,"15min":1.2949999999999999},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.51,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.609}]},{"osd":6,"last update":"Wed Feb 10 04:27:20 2021","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.2909999999999999,"15min":1.3029999999999999},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":1.601,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":1.498},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.351,"15min":1.363},"min":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":2.1019999999999999,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":1.5309999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.2330000000000001,"15min":1.252},"min":{"1min":0.94599999999999995,"5min":0.93500000000000005,"15min":0.93500000000000005},"max":{"1min":1.5900000000000001,"5min":1.76,"15min":1.76},"last":1.577},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3500000000000001,"15min":1.3640000000000001},"min":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"max":{"1min":1.611,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":1.4370000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411380,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6564,"kb_used_data":796,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738588,"statfs":{"total":95995035648,"available":95988314112,"internally_reserved":0,"allocated":815104,"data_stored":317578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:10 2021","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.70899999999999996,"15min":0.68300000000000005},"min":{"1min":0.54400000000000004,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.97999999999999998,"5min":1.349,"15min":1.349},"last":0.624},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.73299999999999998,"15min":0.71099999999999997},"min":{"1min":0.59499999999999997,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.96399999999999997,"5min":1.274,"15min":1.274},"last":1.0609999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:27:10 2021","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.81200000000000006,"15min":0.76900000000000002},"min":{"1min":0.57499999999999996,"5min":0.495,"15min":0.495},"max":{"1min":1.3069999999999999,"5min":1.726,"15min":1.726},"last":0.97999999999999998},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.71299999999999997,"15min":0.69099999999999995},"min":{"1min":0.54900000000000004,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.76100000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:27:31 2021","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.81499999999999995,"15min":0.80400000000000005},"min":{"1min":0.57999999999999996,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.90700000000000003,"5min":1.274,"15min":1.274},"last":1.361},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.996,"15min":0.97099999999999997},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.2,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":1.5149999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:27:44 2021","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.181,"15min":1.181},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.4359999999999999,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":1.4339999999999999},{"interface":"front","average":{"1min":1.141,"5min":1.119,"15min":1.1220000000000001},"min":{"1min":0.80900000000000005,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":1.1899999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:28:04 2021","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.2829999999999999,"15min":1.3049999999999999},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.508,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.137,"5min":1.1799999999999999,"15min":1.179},"min":{"1min":0.875,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.478,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":1.544}]},{"osd":6,"last update":"Wed Feb 10 04:27:23 2021","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.284,"15min":1.2989999999999999},"min":{"1min":0.875,"5min":0.875,"15min":0.875},"max":{"1min":1.504,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":1.6910000000000001},{"interface":"front","average":{"1min":1.26,"5min":1.409,"15min":1.4339999999999999},"min":{"1min":1.05,"5min":1.0489999999999999,"15min":1.0489999999999999},"max":{"1min":1.635,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.2989999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.4690000000000001,"15min":1.494},"min":{"1min":1.077,"5min":1.073,"15min":1.073},"max":{"1min":1.653,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":1.5960000000000001},{"interface":"front","average":{"1min":1.171,"5min":1.252,"15min":1.266},"min":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"max":{"1min":1.52,"5min":1.6160000000000001,"15min":1.6160000000000001},"last":1.0289999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182446,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7000,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":6016,"kb_avail":93738152,"statfs":{"total":95995035648,"available":95987867648,"internally_reserved":0,"allocated":999424,"data_stored":323560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6160384},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:28 2021","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.74299999999999999,"15min":0.73499999999999999},"min":{"1min":0.50900000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.0600000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.88300000000000001,"15min":0.86699999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":4.2759999999999998,"5min":4.2759999999999998,"15min":4.2759999999999998},"last":0.55400000000000005}]},{"osd":1,"last update":"Wed Feb 10 04:27:28 2021","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.90300000000000002,"15min":0.877},"min":{"1min":0.64200000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":3.246,"5min":3.246,"15min":3.246},"last":0.68600000000000005},{"interface":"front","average":{"1min":1.0549999999999999,"5min":0.89700000000000002,"15min":0.86199999999999999},"min":{"1min":0.57999999999999996,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":5.4039999999999999,"5min":5.4039999999999999,"15min":5.4039999999999999},"last":0.90800000000000003}]},{"osd":2,"last update":"Wed Feb 10 04:27:28 2021","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":0.97199999999999998,"15min":0.95099999999999996},"min":{"1min":0.60299999999999998,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":4.2309999999999999,"5min":4.2309999999999999,"15min":4.2309999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.96099999999999997,"15min":0.97599999999999998},"min":{"1min":0.65900000000000003,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.2609999999999999,"5min":1.681,"15min":1.681},"last":0.71399999999999997}]},{"osd":4,"last update":"Wed Feb 10 04:27:43 2021","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.25,"15min":1.204},"min":{"1min":0.84099999999999997,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.9650000000000001,"5min":3.371,"15min":3.371},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.266,"5min":1.266,"15min":1.2250000000000001},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":1.8280000000000001,"5min":3.5430000000000001,"15min":3.5430000000000001},"last":1.163}]},{"osd":5,"last update":"Wed Feb 10 04:28:01 2021","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.232,"15min":1.1919999999999999},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.474,"5min":3.496,"15min":3.496},"last":1.006},{"interface":"front","average":{"1min":1.254,"5min":1.2989999999999999,"15min":1.272},"min":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"max":{"1min":1.6819999999999999,"5min":3.742,"15min":3.742},"last":1.28}]},{"osd":6,"last update":"Wed Feb 10 04:27:24 2021","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.429,"15min":1.4059999999999999},"min":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"max":{"1min":3.6859999999999999,"5min":3.6859999999999999,"15min":3.6859999999999999},"last":1.306},{"interface":"front","average":{"1min":1.514,"5min":1.47,"15min":1.462},"min":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"max":{"1min":3.8759999999999999,"5min":3.8759999999999999,"15min":3.8759999999999999},"last":1.371}]},{"osd":7,"last update":"Wed Feb 10 04:27:40 2021","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":1.5109999999999999,"15min":1.5049999999999999},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"max":{"1min":4.1319999999999997,"5min":4.1319999999999997,"15min":4.1319999999999997},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.587,"5min":1.5629999999999999,"15min":1.5589999999999999},"min":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"max":{"1min":4.0010000000000003,"5min":4.0010000000000003,"15min":4.0010000000000003},"last":1.4359999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986220,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6836,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":5888,"kb_avail":93738316,"statfs":{"total":95995035648,"available":95988035584,"internally_reserved":0,"allocated":962560,"data_stored":328509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6029312},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:43 2021","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.96599999999999997,"15min":0.997},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":1.143},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.95499999999999996,"15min":0.97499999999999998},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.4810000000000001,"5min":1.633,"15min":1.633},"last":0.75900000000000001}]},{"osd":1,"last update":"Wed Feb 10 04:27:43 2021","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.89500000000000002,"15min":0.91700000000000004},"min":{"1min":0.58199999999999996,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":1.075},{"interface":"front","average":{"1min":1.02,"5min":1.012,"15min":1.018},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.583,"5min":1.583,"15min":1.583},"last":0.86899999999999999}]},{"osd":2,"last update":"Wed Feb 10 04:27:43 2021","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.0800000000000001,"15min":1.083},"min":{"1min":0.69399999999999995,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.766,"5min":1.766,"15min":1.766},"last":0.72699999999999998},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.1719999999999999,"15min":1.161},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.5469999999999999,"5min":1.768,"15min":1.768},"last":0.91700000000000004}]},{"osd":3,"last update":"Wed Feb 10 04:27:43 2021","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.097,"15min":1.0609999999999999},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":5.1230000000000002,"5min":5.1230000000000002,"15min":5.1230000000000002},"last":1.248},{"interface":"front","average":{"1min":1.302,"5min":1.129,"15min":1.1080000000000001},"min":{"1min":0.79900000000000004,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":5.1710000000000003,"5min":5.1710000000000003,"15min":5.1710000000000003},"last":0.82299999999999995}]},{"osd":5,"last update":"Wed Feb 10 04:28:05 2021","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":0.93500000000000005,"15min":0.85499999999999998},"min":{"1min":0.63500000000000001,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.5169999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":1.1080000000000001},{"interface":"front","average":{"1min":0.998,"5min":0.90200000000000002,"15min":0.84799999999999998},"min":{"1min":0.66800000000000004,"5min":0.624,"15min":0.624},"max":{"1min":1.3759999999999999,"5min":1.5740000000000001,"15min":1.5740000000000001},"last":0.97599999999999998}]},{"osd":6,"last update":"Wed Feb 10 04:27:19 2021","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.0840000000000001,"15min":1.0640000000000001},"min":{"1min":0.79200000000000004,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":1.008},{"interface":"front","average":{"1min":1.381,"5min":1.27,"15min":1.252},"min":{"1min":1.139,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.714,"5min":1.74,"15min":1.74},"last":1.363}]},{"osd":7,"last update":"Wed Feb 10 04:27:40 2021","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.94899999999999995,"5min":0.88600000000000001,"15min":0.88600000000000001},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3720000000000001,"15min":1.3740000000000001},"min":{"1min":1.073,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":1.6699999999999999,"5min":1.9450000000000001,"15min":1.9450000000000001},"last":1.4890000000000001}]}]},{"osd":5,"up_from":37,"seq":158913789993,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6612,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738540,"statfs":{"total":95995035648,"available":95988264960,"internally_reserved":0,"allocated":864256,"data_stored":318388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:28:04 2021","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.97999999999999998,"15min":0.995},"min":{"1min":0.61299999999999999,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.90800000000000003,"15min":0.91000000000000003},"min":{"1min":0.65300000000000002,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":1.377}]},{"osd":1,"last update":"Wed Feb 10 04:28:04 2021","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.92900000000000005,"15min":0.93200000000000005},"min":{"1min":0.63800000000000001,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":2.262,"5min":2.262,"15min":2.262},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.98699999999999999,"15min":0.998},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.53,"5min":1.53,"15min":1.53},"last":1.24}]},{"osd":2,"last update":"Wed Feb 10 04:28:04 2021","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.004,"15min":0.998},"min":{"1min":0.64100000000000001,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.129,"5min":1.01,"15min":0.98999999999999999},"min":{"1min":0.77800000000000002,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":2.383,"5min":2.383,"15min":2.383},"last":1.04}]},{"osd":3,"last update":"Wed Feb 10 04:28:04 2021","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.119,"15min":1.099},"min":{"1min":0.871,"5min":0.753,"15min":0.753},"max":{"1min":1.4430000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.1000000000000001,"15min":1.079},"min":{"1min":0.75800000000000001,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":2.516,"5min":2.516,"15min":2.516},"last":1.1870000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:28:04 2021","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":0.96499999999999997,"15min":0.92000000000000004},"min":{"1min":0.72799999999999998,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.718,"5min":1.718,"15min":1.718},"last":1.091},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.0820000000000001,"15min":0.99099999999999999},"min":{"1min":0.81999999999999995,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":2.6560000000000001,"5min":2.6560000000000001,"15min":2.6560000000000001},"last":1.4159999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:27:23 2021","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.149,"15min":1.119},"min":{"1min":0.88100000000000001,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":1.1220000000000001},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.085,"15min":1.0549999999999999},"min":{"1min":0.92100000000000004,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.3340000000000001,"5min":2.3340000000000001,"15min":2.3340000000000001},"last":1.274}]},{"osd":7,"last update":"Wed Feb 10 04:27:38 2021","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.1910000000000001,"15min":1.177},"min":{"1min":0.93400000000000005,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.2440000000000002,"5min":2.2440000000000002,"15min":2.2440000000000002},"last":1.506},{"interface":"front","average":{"1min":1.528,"5min":1.3919999999999999,"15min":1.3700000000000001},"min":{"1min":1.0169999999999999,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":2.2799999999999998,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.2150000000000001}]}]}]}} 2021-02-10T04:28:13.213 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2021-02-10T04:28:13.214 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pool set .rgw.root pg_num 42 2021-02-10T04:28:13.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:13 smithi164 conmon[29288]: audit 2021-02-10T04:28:11.754168+0000 mgr.y (mgr.14140) 316 : audit [DBG] from='client.24658 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:28:13.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:13 smithi164 conmon[29288]: cluster 2021-02-10T04:28:11.986418+0000 mgr.y (mgr. 2021-02-10T04:28:13.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:13 smithi164 conmon[29288]: 14140) 317 : cluster [DBG] pgmap v273: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:28:13.438 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:28:13.440 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:28:13 smithi085 conmon[52836]: debug 2021-02-10T04:28:13.380+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:13.467 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:28:13.543 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:28:13.568 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:14.097 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is set 2021-02-10T04:28:14.333 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: audit 2021-02-10T04:28:14.333 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: 2021-02-10T04:28:14.333 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: 2021 2021-02-10T04:28:14.333 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: - 2021-02-10T04:28:14.333 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: 02 2021-02-10T04:28:14.333 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: - 2021-02-10T04:28:14.334 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: 10 2021-02-10T04:28:14.334 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: T 2021-02-10T04:28:14.334 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: 04 2021-02-10T04:28:14.334 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: : 2021-02-10T04:28:14.334 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: 28 2021-02-10T04:28:14.334 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: : 2021-02-10T04:28:14.334 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: 13 2021-02-10T04:28:14.334 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: . 2021-02-10T04:28:14.334 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: 616902 2021-02-10T04:28:14.334 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: +0000 2021-02-10T04:28:14.334 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: 2021-02-10T04:28:14.334 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: mon.c 2021-02-10T04:28:14.334 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: ( 2021-02-10T04:28:14.335 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: mon 2021-02-10T04:28:14.335 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: . 2021-02-10T04:28:14.335 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: 1 2021-02-10T04:28:14.335 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: ) 2021-02-10T04:28:14.335 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: 30 2021-02-10T04:28:14.335 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: : 2021-02-10T04:28:14.335 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: audit 2021-02-10T04:28:14.335 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: 2021-02-10T04:28:14.335 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: [INF] 2021-02-10T04:28:14.335 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: 2021-02-10T04:28:14.335 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/1657753656' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:28:14.335 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: 2021-02-10T04:28:14.335 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: audit 2021-02-10T04:28:14.336 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: 2021-02-10T04:28:14.336 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: 2021 2021-02-10T04:28:14.336 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: - 2021-02-10T04:28:14.336 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: 02 2021-02-10T04:28:14.336 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: - 2021-02-10T04:28:14.336 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: 10 2021-02-10T04:28:14.336 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: T 2021-02-10T04:28:14.336 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: 04 2021-02-10T04:28:14.336 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: : 2021-02-10T04:28:14.336 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: 28 2021-02-10T04:28:14.336 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: : 2021-02-10T04:28:14.336 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: 13 2021-02-10T04:28:14.336 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: . 2021-02-10T04:28:14.337 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: 617206 2021-02-10T04:28:14.337 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: +0000 2021-02-10T04:28:14.337 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: 2021-02-10T04:28:14.337 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: mon.a 2021-02-10T04:28:14.337 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: ( 2021-02-10T04:28:14.337 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: mon 2021-02-10T04:28:14.337 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: . 2021-02-10T04:28:14.337 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: 0 2021-02-10T04:28:14.338 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: ) 2021-02-10T04:28:14.338 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: 707 2021-02-10T04:28:14.338 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: : 2021-02-10T04:28:14.338 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: audit 2021-02-10T04:28:14.338 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: 2021-02-10T04:28:14.338 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: [INF] 2021-02-10T04:28:14.338 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: 2021-02-10T04:28:14.338 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:28:14.338 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[35961]: 2021-02-10T04:28:14.338 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[29319]: audit 2021-02-10T04:28:13.616902+0000 mon.c 2021-02-10T04:28:14.338 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[29319]: (mon.1) 30 : audit [INF] from='client.? 172.21.15.85:0/1657753656' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:28:14.339 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[29319]: audit 2021-02-10T04:28:13.617206+0000 mon.a (mon.0) 707 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:28:14.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:14 smithi164 conmon[29288]: audit 2021-02-10T04:28:13.616902+0000 mon.c (mon.1) 30 : audit [INF] from='client.? 172.21.15.85:0/1657753656' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:28:14.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:14 smithi164 conmon[29288]: 2021-02-10T04:28:14.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:14 smithi164 conmon[29288]: audit 2021-02-10T04:28:13.617206+0000 mon.a (mon.0) 707 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:28:14.648 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:28:14.662 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:28:14 smithi085 conmon[41540]: debug 2021-02-10T04:28:14.575+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:14.681 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.0 dump_blocked_ops 2021-02-10T04:28:14.763 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:28:14.791 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:14.897 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:28:14.919 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:28:15.001 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:28:15.028 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:15.088 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:28:14 smithi164 conmon[33677]: debug 2021-02-10T04:28:14.865+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:15.089 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:15 smithi164 conmon[29288]: cluster 2021-02-10T04:28:13.388823 2021-02-10T04:28:15.089 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:15 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 31 : cluster [DBG] 2.9 scrub ok 2021-02-10T04:28:15.089 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:15 smithi164 conmon[29288]: cluster 2021-02-10T04:28:15.089 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:15 smithi164 conmon[29288]: 2021-02-10T04:28:13.486336+0000 osd.7 (osd. 2021-02-10T04:28:15.089 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:15 smithi164 conmon[29288]: 7) 26 : cluster [DBG] 2.8 deep-scrub ok 2021-02-10T04:28:15.089 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:15 smithi164 conmon[29288]: 2021-02-10T04:28:15.089 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:15 smithi164 conmon[29288]: cluster 2021-02-10T04:28:15.089 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:15 smithi164 conmon[29288]: 2021-02-10T04:28 2021-02-10T04:28:15.090 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:15 smithi164 conmon[29288]: :13.986859+0000 mgr.y (mgr.14140) 2021-02-10T04:28:15.090 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:15 smithi164 conmon[29288]: 318 : cluster [DBG] pgmap v274: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:28:15.090 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:15 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:28:15.090 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:15 smithi164 conmon[29288]: -02-10T04:28: 2021-02-10T04:28:15.090 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:15 smithi164 conmon[29288]: 14.082557+0000 mon.a (mon 2021-02-10T04:28:15.090 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:15 smithi164 conmon[29288]: .0) 708 : cluster [WRN] 2021-02-10T04:28:15.090 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:15 smithi164 conmon[29288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:28:15.090 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:15 smithi164 conmon[29288]: audit 2021-02-10T04:28:15.090 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:15 smithi164 conmon[29288]: 2021-02-10T04:28 2021-02-10T04:28:15.090 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:15 smithi164 conmon[29288]: :14.092018+0000 mon.a ( 2021-02-10T04:28:15.090 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:15 smithi164 conmon[29288]: mon.0) 709 : audit 2021-02-10T04:28:15.090 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:15 smithi164 conmon[29288]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:28:15.090 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:15 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:28:15.091 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:15 smithi164 conmon[29288]: T04:28:14.092272+0000 mon.a 2021-02-10T04:28:15.091 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:15 smithi164 conmon[29288]: (mon.0) 710 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2021-02-10T04:28:15.091 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:15 smithi164 conmon[29288]: 2021-02-10T04:28:15.114 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:28:15.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:15 smithi085 conmon[29319]: cluster 2021-02-10T04:28:13.388823+0000 osd.1 (osd.1) 31 : cluster [DBG] 2.9 scrub ok 2021-02-10T04:28:15.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:15 smithi085 conmon[29319]: cluster 2021-02-10T04:28:13.486336+0000 osd.7 (osd.7) 26 : cluster [DBG] 2.8 deep-scrub ok 2021-02-10T04:28:15.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:15 smithi085 conmon[29319]: cluster 2021-02-10T04:28:13.986859+0000 mgr.y (mgr.14140) 318 : cluster [DBG] pgmap v274: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:28:15.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:15 smithi085 conmon[29319]: cluster 2021-02-10T04:28:14.082557+0000 mon.a (mon.0) 708 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:28:15.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:15 smithi085 conmon[29319]: audit 2021-02-10T04:28:14.092018+0000 mon.a (mon.0) 709 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:28:15.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:15 smithi085 conmon[29319]: cluster 2021-02-10T04:28:14.092272+0000 mon.a (mon.0) 710 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2021-02-10T04:28:15.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:15 smithi085 conmon[35961]: cluster 2021-02-10T04:28:13.388823+0000 osd.1 (osd.1) 31 : cluster [DBG] 2.9 scrub ok 2021-02-10T04:28:15.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:15 smithi085 conmon[35961]: cluster 2021-02-10T04:28:13.486336+0000 osd.7 (osd.7) 26 : cluster [DBG] 2.8 deep-scrub ok 2021-02-10T04:28:15.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:15 smithi085 conmon[35961]: cluster 2021-02-10T04:28:13.986859+0000 mgr.y (mgr.14140) 318 : cluster [DBG] pgmap v274: 132 pgs: 132 active+clean; 25 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:28:15.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:15 smithi085 conmon[35961]: cluster 2021-02-10T04:28:14.082557+0000 mon.a (mon.0) 708 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:28:15.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:15 smithi085 conmon[35961]: audit 2021-02-10T04:28:14.092018+0000 mon.a (mon.0) 709 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:28:15.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:15 smithi085 conmon[35961]: cluster 2021-02-10T04:28:14.092272+0000 mon.a (mon.0) 710 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2021-02-10T04:28:15.218 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:28:15.242 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:15.322 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:28:15.369 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:28:15.393 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:28:15 smithi164 conmon[40892]: debug 2021-02-10T04:28:15.089+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:15.393 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:28:15 smithi164 conmon[40892]: debug 2021-02-10T04:28:15.297+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:15.426 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:28:15.450 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:15.530 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:28:15.567 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:28:15.634 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:28:15.659 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:15.712 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:28:15 smithi164 conmon[44544]: debug 2021-02-10T04:28:15.503+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:15.739 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:28:15.843 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:28:15.869 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:16.143 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:28:15 smithi164 conmon[44544]: debug 2021-02-10T04:28:15.711+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:16.755 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:28:16.860 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:28:16.883 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:28:16 smithi085 conmon[45443]: debug 2021-02-10T04:28:16.717+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:16.887 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:16.981 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set nodeep-scrub 2021-02-10T04:28:17.184 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:28:17.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:17 smithi085 conmon[29319]: cluster 2021-02-10T04:28:15.987282+0000 mgr.y (mgr.14140) 319 : cluster [DBG] pgmap v276: 132 pgs: 132 active+clean; 25 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:28:17.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:17 smithi085 conmon[35961]: cluster 2021-02-10T04:28:15.987282+0000 mgr.y (mgr.14140) 319 : cluster [DBG] pgmap v276: 132 pgs: 132 active+clean; 25 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:28:17.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:17 smithi164 conmon[29288]: cluster 2021-02-10T04:28:15.987282+0000 mgr.y (mgr.14140) 319 : cluster [DBG] pgmap v276: 132 pgs: 132 active+clean; 25 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:28:17.588 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:28:18.007 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:28:17 smithi085 conmon[49106]: debug 2021-02-10T04:28:17.548+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:18.094 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:28:17 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:28:18.108 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:28:18.110 INFO:teuthology.orchestra.run.smithi085.stderr:set pool 2 pg_num to 42 2021-02-10T04:28:18.138 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:18.218 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:28:18.230 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:28:18.244 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:28:18.259 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:28:18.276 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:28:18.295 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:28:18.314 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:28:18.332 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:28:18.335 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:28:18.352 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:28:18.354 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:28:18.354 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:28:18.354 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:28:18.354 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:28:18.354 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:28:18.354 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:28:18.354 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:28:18.355 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:28:18.357 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:28:18 smithi164 conmon[44544]: debug 2021-02-10T04:28:18.207+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:18.358 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:18 smithi164 conmon[29288]: audit 2021-02-10T04:28:17.094529+0000 mon.c (mon.1) 31 : audit [INF] from='client.? 172.21.15.85:0/3011290808' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]: dispatch 2021-02-10T04:28:18.358 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:18 smithi164 conmon[29288]: audit 2021-02-10T04:28:17.094806+0000 mon.a 2021-02-10T04:28:18.358 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:18 smithi164 conmon[29288]: (mon.0) 711 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]: dispatch 2021-02-10T04:28:18.359 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:18.375 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:18 smithi085 conmon[29319]: audit 2021-02-10T04:28:17.094529+0000 mon.c (mon.1) 2021-02-10T04:28:18.375 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:18 smithi085 conmon[29319]: 31 : audit [INF] from='client.? 172.21.15.85:0/3011290808' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]: dispatch 2021-02-10T04:28:18.375 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:18 smithi085 conmon[29319]: audit 2021-02-10T04:28:17.094806+0000 mon.a (mon.0) 711 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]: dispatch 2021-02-10T04:28:18.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:18 smithi085 conmon[35961]: audit 2021-02-10T04:28 2021-02-10T04:28:18.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:18 smithi085 conmon[35961]: :17.094529+0000 mon.c (mon.1) 31 : audit [INF] from='client.? 172.21.15.85:0/3011290808' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]: dispatch 2021-02-10T04:28:18.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:18 smithi085 conmon[35961]: audit 2021-02-10T04:28:17.094806+0000 mon.a (mon.0) 711 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]: dispatch 2021-02-10T04:28:18.430 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:28:18.454 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:28:18.454 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [], 2021-02-10T04:28:18.454 INFO:teuthology.orchestra.run.smithi085.stdout: "complaint_time": 30, 2021-02-10T04:28:18.454 INFO:teuthology.orchestra.run.smithi085.stdout: "num_blocked_ops": 0 2021-02-10T04:28:18.454 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:28:18.534 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:28:18.558 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:18.613 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:28:18 smithi164 conmon[33677]: debug 2021-02-10T04:28:18.413+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:18.646 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:28:18.751 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:28:18.774 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:18.893 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:28:18 smithi164 conmon[44544]: debug 2021-02-10T04:28:18.828+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:18.893 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:28:18 smithi164 conmon[37263]: debug 2021-02-10T04:28:18.611+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:18.905 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:28:19.010 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:28:19.429 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:19 smithi164 conmon[29288]: audit 2021-02-10T04:28:17.654363+0000 mgr.y (mgr.14140) 320 : 2021-02-10T04:28:19.429 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:19 smithi164 conmon[29288]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:28:19.429 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:19 smithi164 conmon[29288]: cluster 2021-02-10T04:28:17.726777+0000 osd.6 (osd.6) 21 : cluster 2021-02-10T04:28:19.430 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:19 smithi164 conmon[29288]: [DBG] purged_snaps scrub starts 2021-02-10T04:28:19.430 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:19 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:28:19.430 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:19 smithi164 conmon[29288]: -10T04:28:17.726891+0000 osd.6 (osd. 2021-02-10T04:28:19.430 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:19 smithi164 conmon[29288]: 6) 22 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:28:19.430 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:19 smithi164 conmon[29288]: cluster 2021-02-10T04:28:19.430 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:19 smithi164 conmon[29288]: 2021-02-10T04:28: 2021-02-10T04:28:19.430 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:19 smithi164 conmon[29288]: 17.987789+0000 mgr.y (mgr.14140) 321 2021-02-10T04:28:19.430 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:19 smithi164 conmon[29288]: : cluster [DBG] pgmap v277: 132 pgs: 132 active+clean; 25 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:28:19.430 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:19 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:28:19.430 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:19 smithi164 conmon[29288]: 02-10T04:28:18.105159 2021-02-10T04:28:19.430 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:19 smithi164 conmon[29288]: +0000 mon.a (mon.0) 712 : audit 2021-02-10T04:28:19.430 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:19 smithi164 conmon[29288]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]': finished 2021-02-10T04:28:19.431 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:19 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:28:19.431 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:19 smithi164 conmon[29288]: 28:18.105234+0000 mon.a (mon.0) 2021-02-10T04:28:19.431 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:19 smithi164 conmon[29288]: 713 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2021-02-10T04:28:19.431 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:19 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:28:19.431 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:19 smithi164 conmon[29288]: 02-10T04:28:18. 2021-02-10T04:28:19.431 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:19 smithi164 conmon[29288]: 195421+0000 mon.c (mon.1) 32 : audit 2021-02-10T04:28:19.431 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:19 smithi164 conmon[29288]: [DBG] from='client.? 172.21.15.85:0/2081925332' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:28:19.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:19 smithi085 conmon[29319]: audit 2021-02-10T04:28:17.654363+0000 2021-02-10T04:28:19.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:19 smithi085 conmon[29319]: mgr.y (mgr.14140) 320 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:28:19.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:19 smithi085 conmon[29319]: cluster 2021-02-10T04:28:17.726777+0000 osd.6 (osd.6) 21 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:28:19.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:19 smithi085 conmon[29319]: cluster 2021-02-10T04:28:17.726891+0000 osd.6 (osd.6) 22 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:28:19.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:19 smithi085 conmon[29319]: cluster 2021-02-10T04:28:17.987789+0000 mgr.y (mgr.14140) 321 : cluster [DBG] pgmap v277: 132 pgs: 132 active+clean; 25 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:28:19.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:19 smithi085 conmon[29319]: audit 2021-02-10T04:28 2021-02-10T04:28:19.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:19 smithi085 conmon[29319]: :18.105159+0000 mon.a (mon.0) 712 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]': finished 2021-02-10T04:28:19.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:19 smithi085 conmon[29319]: cluster 2021-02-10T04:28: 2021-02-10T04:28:19.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:19 smithi085 conmon[29319]: 18.105234+0000 mon.a (mon.0) 713 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2021-02-10T04:28:19.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:19 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:28:19.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:19 smithi085 conmon[29319]: 02-10T04:28:18.195421+0000 mon.c ( 2021-02-10T04:28:19.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:19 smithi085 conmon[29319]: mon.1) 32 : audit [DBG] from='client.? 172.21.15.85:0/2081925332' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:28:19.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:19 smithi085 conmon[35961]: audit 2021-02-10T04:28:17.654363+0000 mgr.y (mgr 2021-02-10T04:28:19.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:19 smithi085 conmon[35961]: .14140) 320 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:28:19.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:19 smithi085 conmon[35961]: cluster 2021-02-10T04:28:17.726777+0000 osd.6 (osd.6) 21 : cluster [DBG] 2021-02-10T04:28:19.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:19 smithi085 conmon[35961]: purged_snaps scrub starts 2021-02-10T04:28:19.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:19 smithi085 conmon[35961]: cluster 2021-02-10T04:28:17.726891+0000 osd.6 (osd. 2021-02-10T04:28:19.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:19 smithi085 conmon[35961]: 6) 22 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:28:19.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:19 smithi085 conmon[35961]: cluster 2021-02-10T04:28:19.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:19 smithi085 conmon[35961]: 2021-02-10T04:28:17.987789+0000 mgr.y ( 2021-02-10T04:28:19.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:19 smithi085 conmon[35961]: mgr.14140) 321 : cluster [DBG] pgmap v277: 132 pgs: 132 active+clean; 25 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:28:19.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:19 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:28:19.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:19 smithi085 conmon[35961]: T04:28:18.105159+0000 mon.a (mon 2021-02-10T04:28:19.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:19 smithi085 conmon[35961]: .0) 712 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]': finished 2021-02-10T04:28:19.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:19 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:28:19.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:19 smithi085 conmon[35961]: 10T04:28:18.105234+0000 mon.a 2021-02-10T04:28:19.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:19 smithi085 conmon[35961]: (mon.0) 713 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2021-02-10T04:28:19.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:19 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:28:19.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:19 smithi085 conmon[35961]: 10T04:28:18.195421+0000 mon.c ( 2021-02-10T04:28:19.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:19 smithi085 conmon[35961]: mon.1) 32 : audit [DBG] from='client.? 172.21.15.85:0/2081925332' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:28:19.455 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:19.546 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:28:19.651 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:28:19.674 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:19.729 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:28:19 smithi164 conmon[40892]: debug 2021-02-10T04:28:19.511+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:19.729 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:28:19 smithi164 conmon[33677]: debug 2021-02-10T04:28:19.727+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:19.755 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:28:19.859 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:28:19.893 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:20.043 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2021-02-10T04:28:20.043 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pool create unique_pool_0 16 2021-02-10T04:28:20.068 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.0 dump_historic_ops 2021-02-10T04:28:20.157 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:28:20.265 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:28:20.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:20 smithi085 conmon[35961]: cluster 2021-02-10T04:28:19.988299+0000 mgr.y (mgr.14140) 322 : cluster [DBG] pgmap v279: 132 pgs: 132 active+clean; 25 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:28:20.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:20 smithi085 conmon[35961]: audit 2021-02-10T04:28:19.989199+0000 mon.a (mon.0) 714 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "42"}]: dispatch 2021-02-10T04:28:20.288 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:20 smithi085 conmon[29319]: cluster 2021-02-10T04:28:19.988299+0000 mgr.y (mgr.14140) 322 : cluster [DBG] pgmap v279: 132 pgs: 132 active+clean; 25 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:28:20.289 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:20 smithi085 conmon[29319]: audit 2021-02-10T04:28:19.989199+0000 mon.a (mon.0) 714 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "42"}]: dispatch 2021-02-10T04:28:20.289 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:28:20 smithi085 conmon[45443]: debug 2021-02-10T04:28:20.289 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:28:20 smithi085 conmon[45443]: 2021-02-10T04:28:20.118+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:20.291 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:20.339 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:28:20.340 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:28:20.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:20 smithi164 conmon[29288]: cluster 2021-02-10T04:28:19.988299+0000 mgr.y (mgr.14140) 322 : cluster [DBG] 2021-02-10T04:28:20.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:20 smithi164 conmon[29288]: pgmap v279: 132 pgs: 132 active+clean; 25 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:28:20.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:20 smithi164 conmon[29288]: audit 2021-02-10T04:28:19.989199+0000 mon.a (mon.0) 714 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "42"}]: dispatch 2021-02-10T04:28:20.454 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:28:20.544 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:28:20 smithi085 conmon[52836]: debug 2021-02-10T04:28:20.418+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:20.558 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:28:20.592 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:20.688 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:28:20.792 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:28:20.819 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:20.876 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:28:20 smithi164 conmon[33677]: debug 2021-02-10T04:28:20.653+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:20.930 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:28:21.034 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:28:21.059 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:21.125 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is set 2021-02-10T04:28:21.138 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:28:21.140 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:28:21 smithi164 conmon[37263]: debug 2021-02-10T04:28:21.116+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:21.141 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:28:20 smithi164 conmon[44544]: debug 2021-02-10T04:28:20.874+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:21.141 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:21 smithi164 conmon[29288]: audit 2021-02-10T04:28:20.105907+0000 mon.a (mon.0 2021-02-10T04:28:21.141 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:21 smithi164 conmon[29288]: ) 715 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "42"}]': finished 2021-02-10T04:28:21.141 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:21 smithi164 conmon[29288]: cluster 2021-02-10T04:28:20.105986+0000 mon.a (mon.0) 716 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2021-02-10T04:28:21.141 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:21 smithi164 conmon[29288]: audit 2021-02-10T04:28:20.948600+0000 2021-02-10T04:28:21.141 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:21 smithi164 conmon[29288]: mon.c (mon.1) 33 : audit 2021-02-10T04:28:21.141 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:21 smithi164 conmon[29288]: [INF] from='client.? 172.21.15.85:0/15903667' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:28:21.141 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:21 smithi164 conmon[29288]: audit 2021-02-10T04:28:21.141 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:21 smithi164 conmon[29288]: 2021-02-10T04:28: 2021-02-10T04:28:21.142 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:21 smithi164 conmon[29288]: 20.948930+0000 mon.a (mon. 2021-02-10T04:28:21.142 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:21 smithi164 conmon[29288]: 0) 717 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:28:21.142 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:21 smithi164 conmon[29288]: 2021-02-10T04:28:21.243 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:28:21.269 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:21.355 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:28:21.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:28:21 smithi164 conmon[40892]: debug 2021-02-10T04:28:21.322+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:21.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:21 smithi085 conmon[29319]: audit 2021-02-10T04:28:20.105907+0000 mon.a (mon 2021-02-10T04:28:21.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:21 smithi085 conmon[29319]: .0) 715 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "42"}]': finished 2021-02-10T04:28:21.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:21 smithi085 conmon[29319]: cluster 2021-02-10T04:28:20.105986+0000 mon.a (mon.0) 716 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2021-02-10T04:28:21.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:21 smithi085 conmon[29319]: audit 2021-02-10T04:28:20.948600+0000 mon.c (mon.1) 33 : audit [INF] from='client.? 172.21.15.85:0/15903667' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:28:21.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:21 smithi085 conmon[29319]: audit 2021-02-10T04:28:20.948930+0000 mon.a (mon.0) 717 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:28:21.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:21 smithi085 conmon[35961]: audit 2021-02-10T04:28:20.105907+0000 mon.a (mon. 2021-02-10T04:28:21.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:21 smithi085 conmon[35961]: 0) 715 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "42"}]': finished 2021-02-10T04:28:21.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:21 smithi085 conmon[35961]: cluster 2021-02-10T04:28:20.105986+0000 mon.a (mon.0) 2021-02-10T04:28:21.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:21 smithi085 conmon[35961]: 716 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2021-02-10T04:28:21.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:21 smithi085 conmon[35961]: audit 2021-02-10T04:28: 2021-02-10T04:28:21.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:21 smithi085 conmon[35961]: 20.948600+0000 mon.c (mon.1) 33 : audit [INF] from='client.? 172.21.15.85:0/15903667' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:28:21.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:21 smithi085 conmon[35961]: audit 2021-02-10T04:28:20.948930+0000 2021-02-10T04:28:21.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:21 smithi085 conmon[35961]: mon.a (mon.0) 717 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:28:21.458 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:28:21.487 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:22.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:22 smithi164 conmon[29288]: cluster 2021-02-10T04:28:21.103926+0000 mon.a (mon.0) 718 : cluster 2021-02-10T04:28:22.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:22 smithi164 conmon[29288]: [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:28:22.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:22 smithi164 conmon[29288]: audit 2021-02-10T04:28:21.113002+0000 mon.a (mon.0) 719 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:28:22.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:22 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:28:22.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:22 smithi164 conmon[29288]: 10T04:28:21.113074+0000 mon.a (mon.0) 720 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2021-02-10T04:28:22.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:22 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:28:22.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:22 smithi164 conmon[29288]: T04:28:21.989101+0000 mgr.y (mgr 2021-02-10T04:28:22.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:22 smithi164 conmon[29288]: .14140) 323 : cluster [DBG] pgmap v282: 142 pgs: 4 peering, 10 unknown, 128 active+clean; 25 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2021-02-10T04:28:22.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:22 smithi164 conmon[29288]: audit 2021-02-10T04:28:21. 2021-02-10T04:28:22.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:22 smithi164 conmon[29288]: 989914+0000 mon.a (mon.0) 721 : audit 2021-02-10T04:28:22.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:22 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "42"}]: dispatch 2021-02-10T04:28:22.427 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:28:22.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:22 smithi085 conmon[29319]: cluster 2021-02-10T04:28:21.103926+0000 mon.a (mon.0) 718 2021-02-10T04:28:22.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:22 smithi085 conmon[29319]: : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:28:22.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:22 smithi085 conmon[29319]: audit 2021-02-10T04:28:21.113002+0000 mon.a (mon.0) 719 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:28:22.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:22 smithi085 conmon[29319]: cluster 2021-02-10T04:28:21.113074+0000 mon.a (mon.0) 720 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2021-02-10T04:28:22.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:22 smithi085 conmon[29319]: cluster 2021-02-10T04:28:21.989101+0000 mgr.y (mgr.14140) 323 : cluster [DBG] pgmap v282: 142 pgs: 4 peering, 10 unknown, 128 active+clean; 25 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2021-02-10T04:28:22.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:22 smithi085 conmon[29319]: audit 2021-02-10T04:28:21.989914+0000 mon.a (mon.0) 721 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "42"}]: dispatch 2021-02-10T04:28:22.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:22 smithi085 conmon[35961]: cluster 2021-02-10T04:28:21.103926 2021-02-10T04:28:22.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:22 smithi085 conmon[35961]: +0000 mon.a (mon.0) 718 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:28:22.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:22 smithi085 conmon[35961]: audit 2021-02-10T04:28:21.113002+0000 mon.a (mon.0) 719 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:28:22.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:22 smithi085 conmon[35961]: cluster 2021-02-10T04:28:21.113074+0000 mon.a (mon.0) 720 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2021-02-10T04:28:22.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:22 smithi085 conmon[35961]: cluster 2021-02-10T04:28:21.989101+0000 mgr.y (mgr.14140) 323 : cluster [DBG] pgmap v282: 142 pgs: 4 peering, 10 unknown, 128 active+clean; 25 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2021-02-10T04:28:22.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:22 smithi085 conmon[35961]: audit 2021-02-10T04:28:21.989914+0000 mon.a (mon.0) 721 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "42"}]: dispatch 2021-02-10T04:28:22.627 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:28:22.663 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:28:22.690 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:28:22 smithi085 conmon[49106]: debug 2021-02-10T04:28:22.448+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:22.767 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:28:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:22.929 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:28:23.033 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:28:23.057 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:23.129 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:28:22 smithi164 conmon[44544]: debug 2021-02-10T04:28:22.869+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:23.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[29319]: audit 2021-02-10T04:28:23.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[29319]: 2021-02-10T04:28:23.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[29319]: 2021 2021-02-10T04:28:23.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[29319]: -02-10T04:28:22 2021-02-10T04:28:23.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[29319]: .115135+0000 mon.a (mon.0) 722 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "42"}]': finished 2021-02-10T04:28:23.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:28:23.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[29319]: 02-10T04:28:22.115198 2021-02-10T04:28:23.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[29319]: +0000 mon.a (mon.0) 723 2021-02-10T04:28:23.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[29319]: : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2021-02-10T04:28:23.176 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[35961]: audit 2021-02-10T04:28:23.176 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[35961]: 2021-02-10T04:28:23.176 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[35961]: 2021 2021-02-10T04:28:23.177 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[35961]: - 2021-02-10T04:28:23.177 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[35961]: 02 2021-02-10T04:28:23.177 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[35961]: - 2021-02-10T04:28:23.177 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[35961]: 10 2021-02-10T04:28:23.177 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[35961]: T 2021-02-10T04:28:23.177 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[35961]: 04 2021-02-10T04:28:23.178 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[35961]: : 2021-02-10T04:28:23.178 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[35961]: 28 2021-02-10T04:28:23.178 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[35961]: : 2021-02-10T04:28:23.178 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[35961]: 22 2021-02-10T04:28:23.178 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[35961]: . 2021-02-10T04:28:23.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[35961]: 115135 2021-02-10T04:28:23.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[35961]: +0000 2021-02-10T04:28:23.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[35961]: 2021-02-10T04:28:23.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[35961]: mon.a 2021-02-10T04:28:23.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[35961]: ( 2021-02-10T04:28:23.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[35961]: mon 2021-02-10T04:28:23.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[35961]: . 2021-02-10T04:28:23.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[35961]: 0 2021-02-10T04:28:23.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[35961]: ) 2021-02-10T04:28:23.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[35961]: 722 2021-02-10T04:28:23.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[35961]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "42"}]': finished 2021-02-10T04:28:23.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[35961]: cluster 2021-02-10T04:28:22.115198+0000 mon.a (mon.0 2021-02-10T04:28:23.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[35961]: ) 723 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2021-02-10T04:28:23.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:23 smithi164 conmon[29288]: audit 2021-02-10T04:28:22.115135+0000 mon.a (mon.0) 722 : audit 2021-02-10T04:28:23.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:23 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "42"}]': finished 2021-02-10T04:28:23.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:23 smithi164 conmon[29288]: cluster 2021-02-10T04:28:22.115198+0000 mon.a (mon.0) 723 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2021-02-10T04:28:23.441 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:28:23 smithi085 conmon[52836]: debug 2021-02-10T04:28:23.173+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:23.602 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:28:23.602 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 20, 2021-02-10T04:28:23.602 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 600, 2021-02-10T04:28:23.602 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [ 2021-02-10T04:28:23.602 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.603 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:415 4.1e 4:7c9a00df:::data_log.43:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e58)", 2021-02-10T04:28:23.603 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:25:51.304437+0000", 2021-02-10T04:28:23.603 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 152.296190295, 2021-02-10T04:28:23.603 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0020714380000000001, 2021-02-10T04:28:23.603 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:23.603 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:28:23.603 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:23.603 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:28:23.603 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:28:23.603 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 415 2021-02-10T04:28:23.603 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.603 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:23.603 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.604 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:23.604 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.304437+0000", 2021-02-10T04:28:23.604 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:23.604 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.604 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.604 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:23.604 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.304437+0000", 2021-02-10T04:28:23.604 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.2857000000000002e-05 2021-02-10T04:28:23.604 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.604 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.605 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:23.605 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.304459+0000", 2021-02-10T04:28:23.605 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.3839999999999999e-06 2021-02-10T04:28:23.605 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.605 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.605 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:23.605 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.304464+0000", 2021-02-10T04:28:23.605 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.4849999999999998e-06 2021-02-10T04:28:23.606 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.606 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.606 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:23.606 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.304473+0000", 2021-02-10T04:28:23.606 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.8640000000000001e-06 2021-02-10T04:28:23.606 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.606 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.606 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:23.607 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.304479+0000", 2021-02-10T04:28:23.607 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.7225000000000001e-05 2021-02-10T04:28:23.607 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.607 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.607 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:23.607 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.304506+0000", 2021-02-10T04:28:23.607 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.5059000000000003e-05 2021-02-10T04:28:23.607 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.607 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.608 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:23.608 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.304561+0000", 2021-02-10T04:28:23.608 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001946564 2021-02-10T04:28:23.608 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.608 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.608 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:23.608 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.306508+0000", 2021-02-10T04:28:23.608 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0020714380000000001 2021-02-10T04:28:23.609 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.609 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:23.609 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.609 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.609 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.609 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:425 4.16 4:6b3358ad:::data_log.48:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e58)", 2021-02-10T04:28:23.609 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:25:51.319405+0000", 2021-02-10T04:28:23.610 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 152.281221617, 2021-02-10T04:28:23.610 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0019790369999999999, 2021-02-10T04:28:23.610 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:23.610 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:28:23.610 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:23.610 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:28:23.610 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:28:23.611 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 425 2021-02-10T04:28:23.611 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.611 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:23.611 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.612 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:23.612 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.319405+0000", 2021-02-10T04:28:23.612 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:23.612 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.612 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.612 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:23.612 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.319405+0000", 2021-02-10T04:28:23.613 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.4156e-05 2021-02-10T04:28:23.613 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.613 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.613 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:23.613 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.319429+0000", 2021-02-10T04:28:23.614 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4009999999999999e-06 2021-02-10T04:28:23.614 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.614 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.614 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:23.615 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.319434+0000", 2021-02-10T04:28:23.615 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.0010000000000007e-06 2021-02-10T04:28:23.615 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.615 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.616 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:23.616 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.319443+0000", 2021-02-10T04:28:23.616 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.4909999999999996e-06 2021-02-10T04:28:23.616 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.616 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.616 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:23.616 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.319448+0000", 2021-02-10T04:28:23.616 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.8730000000000001e-05 2021-02-10T04:28:23.616 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.616 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.616 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:23.616 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.319477+0000", 2021-02-10T04:28:23.617 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.3006e-05 2021-02-10T04:28:23.617 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.617 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.617 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:23.617 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.319530+0000", 2021-02-10T04:28:23.617 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001854252 2021-02-10T04:28:23.617 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.617 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.617 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:23.617 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.321384+0000", 2021-02-10T04:28:23.617 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0019790369999999999 2021-02-10T04:28:23.617 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.617 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:23.617 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.618 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.618 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.618 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:427 4.16 4:6e096a37:::data_log.49:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e58)", 2021-02-10T04:28:23.618 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:25:51.322959+0000", 2021-02-10T04:28:23.618 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 152.277668245, 2021-02-10T04:28:23.618 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0018977340000000001, 2021-02-10T04:28:23.618 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:23.618 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:28:23.618 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:23.618 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:28:23.618 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:28:23.618 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 427 2021-02-10T04:28:23.618 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.619 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:23.619 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.619 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:23.619 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.322959+0000", 2021-02-10T04:28:23.619 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:23.619 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.619 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.619 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:23.619 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.322959+0000", 2021-02-10T04:28:23.619 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.3118000000000001e-05 2021-02-10T04:28:23.619 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.619 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.619 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:23.620 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.322982+0000", 2021-02-10T04:28:23.620 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.5009999999999998e-06 2021-02-10T04:28:23.620 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.620 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.620 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:23.620 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.322986+0000", 2021-02-10T04:28:23.620 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.55e-06 2021-02-10T04:28:23.620 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.620 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.620 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:23.620 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.322996+0000", 2021-02-10T04:28:23.620 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.4309999999999997e-06 2021-02-10T04:28:23.620 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.620 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.621 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:23.621 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.323003+0000", 2021-02-10T04:28:23.621 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.2143e-05 2021-02-10T04:28:23.621 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.621 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.621 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:23.621 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.323035+0000", 2021-02-10T04:28:23.621 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.8495000000000002e-05 2021-02-10T04:28:23.621 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.621 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.621 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:23.621 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.323094+0000", 2021-02-10T04:28:23.621 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0017624959999999999 2021-02-10T04:28:23.622 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.622 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.622 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:23.622 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.324856+0000", 2021-02-10T04:28:23.622 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0018977340000000001 2021-02-10T04:28:23.622 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.622 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:23.622 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.622 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.622 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.622 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:501 4.b 4:d1061236:::data_log.86:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e58)", 2021-02-10T04:28:23.622 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:25:51.425472+0000", 2021-02-10T04:28:23.622 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 152.175155061, 2021-02-10T04:28:23.623 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0022142070000000002, 2021-02-10T04:28:23.623 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:23.623 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:28:23.623 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:23.623 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:28:23.623 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:28:23.623 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 501 2021-02-10T04:28:23.623 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.623 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:23.623 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.623 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:23.623 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.425472+0000", 2021-02-10T04:28:23.623 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:23.624 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.624 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.624 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:23.624 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.425472+0000", 2021-02-10T04:28:23.624 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.3238999999999999e-05 2021-02-10T04:28:23.624 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.624 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.624 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:23.624 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.425495+0000", 2021-02-10T04:28:23.624 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.2010000000000001e-06 2021-02-10T04:28:23.624 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.625 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.625 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:23.625 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.425499+0000", 2021-02-10T04:28:23.625 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.3809999999999999e-06 2021-02-10T04:28:23.625 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.625 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.625 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:23.625 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.425508+0000", 2021-02-10T04:28:23.625 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1895999999999999e-05 2021-02-10T04:28:23.625 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.625 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.625 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:23.625 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.425520+0000", 2021-02-10T04:28:23.625 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.6381999999999998e-05 2021-02-10T04:28:23.626 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.626 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.626 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:23.626 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.425556+0000", 2021-02-10T04:28:23.626 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.8251000000000002e-05 2021-02-10T04:28:23.626 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.626 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.626 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:23.626 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.425614+0000", 2021-02-10T04:28:23.626 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0020718569999999999 2021-02-10T04:28:23.626 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.626 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.626 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:23.627 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.427686+0000", 2021-02-10T04:28:23.627 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0022142070000000002 2021-02-10T04:28:23.627 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.627 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:23.627 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.627 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.627 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.627 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:507 4.16 4:6b9e929a:::data_log.89:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e58)", 2021-02-10T04:28:23.627 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:25:51.434371+0000", 2021-02-10T04:28:23.627 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 152.166256054, 2021-02-10T04:28:23.627 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0024555240000000002, 2021-02-10T04:28:23.628 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:23.628 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:28:23.628 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:23.628 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:28:23.628 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:28:23.628 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 507 2021-02-10T04:28:23.628 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.628 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:23.628 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.628 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:23.628 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.434371+0000", 2021-02-10T04:28:23.628 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:23.629 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.629 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.629 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:23.629 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.434371+0000", 2021-02-10T04:28:23.629 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.2972999999999998e-05 2021-02-10T04:28:23.629 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.629 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.629 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:23.629 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.434394+0000", 2021-02-10T04:28:23.629 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4429999999999996e-06 2021-02-10T04:28:23.629 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.629 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.629 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:23.629 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.434398+0000", 2021-02-10T04:28:23.630 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0512e-05 2021-02-10T04:28:23.630 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.630 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.630 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:23.630 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.434409+0000", 2021-02-10T04:28:23.630 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.6529999999999999e-06 2021-02-10T04:28:23.630 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.630 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.630 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:23.630 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.434414+0000", 2021-02-10T04:28:23.630 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.9056000000000001e-05 2021-02-10T04:28:23.630 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.631 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.631 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:23.631 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.434443+0000", 2021-02-10T04:28:23.631 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.4481999999999998e-05 2021-02-10T04:28:23.631 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.631 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.631 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:23.631 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.434498+0000", 2021-02-10T04:28:23.631 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.002328405 2021-02-10T04:28:23.631 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.631 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.631 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:23.632 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.436826+0000", 2021-02-10T04:28:23.632 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0024555240000000002 2021-02-10T04:28:23.632 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.632 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:23.632 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.632 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.632 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.632 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:509 4.1e 4:7dc404fa:::data_log.90:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e58)", 2021-02-10T04:28:23.632 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:25:51.438531+0000", 2021-02-10T04:28:23.632 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 152.162095454, 2021-02-10T04:28:23.632 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0021810559999999998, 2021-02-10T04:28:23.632 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:23.632 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:28:23.632 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:23.633 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:28:23.633 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:28:23.633 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 509 2021-02-10T04:28:23.633 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.633 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:23.633 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.633 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:23.633 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.438531+0000", 2021-02-10T04:28:23.633 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:23.633 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.633 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.633 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:23.633 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.438531+0000", 2021-02-10T04:28:23.634 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.2294e-05 2021-02-10T04:28:23.634 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.634 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.634 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:23.634 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.438554+0000", 2021-02-10T04:28:23.634 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.1450000000000001e-06 2021-02-10T04:28:23.634 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.634 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.634 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:23.634 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.438558+0000", 2021-02-10T04:28:23.634 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.0240000000000003e-06 2021-02-10T04:28:23.634 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.634 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.635 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:23.635 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.438567+0000", 2021-02-10T04:28:23.635 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.4679999999999992e-06 2021-02-10T04:28:23.635 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.635 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.635 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:23.635 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.438575+0000", 2021-02-10T04:28:23.635 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.1313999999999999e-05 2021-02-10T04:28:23.635 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.635 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.635 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:23.635 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.438607+0000", 2021-02-10T04:28:23.635 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0377999999999998e-05 2021-02-10T04:28:23.636 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.636 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.636 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:23.636 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.438657+0000", 2021-02-10T04:28:23.636 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0020554330000000002 2021-02-10T04:28:23.636 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.636 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.636 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:23.636 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.440712+0000", 2021-02-10T04:28:23.637 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0021810559999999998 2021-02-10T04:28:23.637 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.637 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:23.637 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.637 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.637 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.637 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:527 4.6 4:620d8b71:::data_log.99:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e58)", 2021-02-10T04:28:23.638 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:25:51.463746+0000", 2021-02-10T04:28:23.638 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 152.13688100900001, 2021-02-10T04:28:23.638 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0024016580000000001, 2021-02-10T04:28:23.638 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:23.638 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:28:23.639 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:23.639 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:28:23.639 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:28:23.639 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 527 2021-02-10T04:28:23.639 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.639 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:23.640 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.640 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:23.640 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.463746+0000", 2021-02-10T04:28:23.640 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:23.640 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.640 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.640 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:23.641 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.463746+0000", 2021-02-10T04:28:23.641 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.5360000000000001e-05 2021-02-10T04:28:23.641 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.641 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.641 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:23.641 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.463771+0000", 2021-02-10T04:28:23.642 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.1250000000000001e-06 2021-02-10T04:28:23.642 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.642 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.642 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:23.642 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.463776+0000", 2021-02-10T04:28:23.642 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.5859999999999994e-06 2021-02-10T04:28:23.643 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.643 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.643 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:23.643 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.463786+0000", 2021-02-10T04:28:23.643 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.75e-06 2021-02-10T04:28:23.643 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.643 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.644 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:23.644 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.463792+0000", 2021-02-10T04:28:23.644 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.7243e-05 2021-02-10T04:28:23.644 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.644 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.645 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:23.645 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.463819+0000", 2021-02-10T04:28:23.645 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.6882999999999997e-05 2021-02-10T04:28:23.645 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.645 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.646 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:23.646 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.463876+0000", 2021-02-10T04:28:23.646 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.002271711 2021-02-10T04:28:23.646 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.646 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.646 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:23.647 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.466148+0000", 2021-02-10T04:28:23.647 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0024016580000000001 2021-02-10T04:28:23.647 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.647 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:23.647 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.648 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.648 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.648 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:539 4.6 4:61dd9248:::data_log.105:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e58)", 2021-02-10T04:28:23.648 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:25:51.480013+0000", 2021-02-10T04:28:23.648 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 152.120613967, 2021-02-10T04:28:23.648 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.002350745, 2021-02-10T04:28:23.648 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:23.649 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:28:23.649 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:23.649 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:28:23.649 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:28:23.649 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 539 2021-02-10T04:28:23.649 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.649 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:23.649 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.649 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:23.649 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.480013+0000", 2021-02-10T04:28:23.649 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:23.649 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.650 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.650 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:23.650 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.480013+0000", 2021-02-10T04:28:23.650 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.4156e-05 2021-02-10T04:28:23.650 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.650 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.650 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:23.651 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.480037+0000", 2021-02-10T04:28:23.651 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.3939999999999998e-06 2021-02-10T04:28:23.651 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.651 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.651 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:23.651 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.480041+0000", 2021-02-10T04:28:23.652 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.8050000000000001e-06 2021-02-10T04:28:23.652 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.652 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.652 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:23.652 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.480051+0000", 2021-02-10T04:28:23.652 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.9619999999999998e-06 2021-02-10T04:28:23.653 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.653 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.653 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:23.653 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.480057+0000", 2021-02-10T04:28:23.653 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.8602000000000002e-05 2021-02-10T04:28:23.653 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.654 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.654 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:23.654 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.480086+0000", 2021-02-10T04:28:23.654 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.4866e-05 2021-02-10T04:28:23.654 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.655 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.655 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:23.655 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.480141+0000", 2021-02-10T04:28:23.655 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00222296 2021-02-10T04:28:23.655 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.656 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.656 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:23.656 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.482364+0000", 2021-02-10T04:28:23.656 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.002350745 2021-02-10T04:28:23.656 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.656 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:23.657 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.657 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.657 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.657 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:543 4.6 4:63b3b181:::data_log.107:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e58)", 2021-02-10T04:28:23.657 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:25:51.487728+0000", 2021-02-10T04:28:23.657 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 152.112899161, 2021-02-10T04:28:23.658 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0021194019999999998, 2021-02-10T04:28:23.658 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:23.658 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:28:23.658 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:23.658 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:28:23.658 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:28:23.659 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 543 2021-02-10T04:28:23.659 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.659 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:23.659 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.659 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:23.659 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.487728+0000", 2021-02-10T04:28:23.660 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:23.660 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.660 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.660 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:23.660 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.487728+0000", 2021-02-10T04:28:23.660 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.3569000000000001e-05 2021-02-10T04:28:23.661 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.661 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.661 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:23.661 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.487751+0000", 2021-02-10T04:28:23.661 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.2629999999999997e-06 2021-02-10T04:28:23.661 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.662 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.662 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:23.662 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.487756+0000", 2021-02-10T04:28:23.662 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.9539999999999993e-06 2021-02-10T04:28:23.662 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.662 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.663 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:23.663 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.487765+0000", 2021-02-10T04:28:23.663 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.2799999999999997e-06 2021-02-10T04:28:23.663 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.663 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.663 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:23.664 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.487769+0000", 2021-02-10T04:28:23.664 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.2811000000000001e-05 2021-02-10T04:28:23.664 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.664 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.664 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:23.664 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.487802+0000", 2021-02-10T04:28:23.665 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.2191000000000002e-05 2021-02-10T04:28:23.665 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.665 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.665 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:23.665 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.487854+0000", 2021-02-10T04:28:23.665 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0019933339999999998 2021-02-10T04:28:23.666 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.666 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.666 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:23.666 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.489847+0000", 2021-02-10T04:28:23.666 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0021194019999999998 2021-02-10T04:28:23.667 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.667 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:23.667 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.667 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.667 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.667 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24382.0:37 4.1e 4:7e82ab18:lc::lc.13:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2021-02-10T04:28:23.668 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:25:55.107956+0000", 2021-02-10T04:28:23.668 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 148.49267108800001, 2021-02-10T04:28:23.668 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0020291549999999999, 2021-02-10T04:28:23.668 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:23.668 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:28:23.668 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:23.668 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24382", 2021-02-10T04:28:23.669 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/3915233377", 2021-02-10T04:28:23.669 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 37 2021-02-10T04:28:23.669 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.669 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:23.669 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.669 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:23.670 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.107956+0000", 2021-02-10T04:28:23.670 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:23.670 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.670 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.670 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:23.671 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.107956+0000", 2021-02-10T04:28:23.671 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.5089999999999999e-05 2021-02-10T04:28:23.671 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.671 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.671 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:23.671 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.107981+0000", 2021-02-10T04:28:23.671 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.8499999999999994e-06 2021-02-10T04:28:23.672 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.672 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.672 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:23.672 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.107989+0000", 2021-02-10T04:28:23.672 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2734999999999999e-05 2021-02-10T04:28:23.673 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.673 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.673 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:23.673 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.108001+0000", 2021-02-10T04:28:23.673 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.4869999999999999e-06 2021-02-10T04:28:23.673 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.674 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.674 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:23.674 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.108008+0000", 2021-02-10T04:28:23.674 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.4948000000000003e-05 2021-02-10T04:28:23.674 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.674 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.675 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:23.675 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.108063+0000", 2021-02-10T04:28:23.675 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00014214499999999999 2021-02-10T04:28:23.675 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.675 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.675 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:23.676 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.108205+0000", 2021-02-10T04:28:23.676 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00022159999999999999 2021-02-10T04:28:23.676 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.676 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.676 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for subops from 3,7", 2021-02-10T04:28:23.676 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.108427+0000", 2021-02-10T04:28:23.677 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00090848999999999997 2021-02-10T04:28:23.677 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.677 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.677 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:23.677 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.109335+0000", 2021-02-10T04:28:23.677 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000494532 2021-02-10T04:28:23.678 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.678 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.678 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op_commit", 2021-02-10T04:28:23.678 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.109830+0000", 2021-02-10T04:28:23.678 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.3062e-05 2021-02-10T04:28:23.678 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.679 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.679 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:23.679 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.109893+0000", 2021-02-10T04:28:23.679 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.5627000000000001e-05 2021-02-10T04:28:23.679 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.680 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.680 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:28:23.680 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.109948+0000", 2021-02-10T04:28:23.680 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.6588999999999998e-05 2021-02-10T04:28:23.680 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.680 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.681 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:23.681 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.109985+0000", 2021-02-10T04:28:23.681 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0020291549999999999 2021-02-10T04:28:23.681 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.681 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:23.681 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.682 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.682 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.682 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24382.0:64 4.1e 4:7e82ab18:lc::lc.13:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2021-02-10T04:28:23.682 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:25:55.113381+0000", 2021-02-10T04:28:23.682 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 148.487245927, 2021-02-10T04:28:23.682 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001854865, 2021-02-10T04:28:23.682 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:23.682 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:28:23.683 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:23.683 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24382", 2021-02-10T04:28:23.683 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/3915233377", 2021-02-10T04:28:23.683 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 64 2021-02-10T04:28:23.683 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.683 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:23.683 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.684 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:23.684 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.113381+0000", 2021-02-10T04:28:23.684 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:23.684 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.684 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.685 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:23.685 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.113381+0000", 2021-02-10T04:28:23.685 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.1437999999999999e-05 2021-02-10T04:28:23.685 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.685 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.685 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:23.686 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.113422+0000", 2021-02-10T04:28:23.686 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0129000000000001e-05 2021-02-10T04:28:23.686 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.686 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.686 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:23.687 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.113433+0000", 2021-02-10T04:28:23.687 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2361000000000001e-05 2021-02-10T04:28:23.687 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.687 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.687 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:23.687 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.113445+0000", 2021-02-10T04:28:23.688 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.3779999999999998e-06 2021-02-10T04:28:23.688 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.688 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.688 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:23.688 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.113451+0000", 2021-02-10T04:28:23.688 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.2574999999999998e-05 2021-02-10T04:28:23.689 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.689 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.689 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:23.689 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.113494+0000", 2021-02-10T04:28:23.689 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00012247500000000001 2021-02-10T04:28:23.689 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.689 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.689 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:23.690 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.113616+0000", 2021-02-10T04:28:23.690 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000179216 2021-02-10T04:28:23.690 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.690 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.690 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for subops from 3,7", 2021-02-10T04:28:23.690 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.113796+0000", 2021-02-10T04:28:23.690 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00039559600000000001 2021-02-10T04:28:23.690 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.690 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.690 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op_commit", 2021-02-10T04:28:23.690 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.114191+0000", 2021-02-10T04:28:23.691 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00073538599999999998 2021-02-10T04:28:23.691 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.691 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.691 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:23.691 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.114927+0000", 2021-02-10T04:28:23.691 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00021909100000000001 2021-02-10T04:28:23.691 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.691 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.691 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:23.691 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.115146+0000", 2021-02-10T04:28:23.692 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.6616999999999997e-05 2021-02-10T04:28:23.692 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.692 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.692 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:28:23.692 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.115202+0000", 2021-02-10T04:28:23.692 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.3603000000000003e-05 2021-02-10T04:28:23.693 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.693 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.693 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:23.693 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.115236+0000", 2021-02-10T04:28:23.693 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001854865 2021-02-10T04:28:23.693 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.693 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:23.693 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.693 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.693 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.694 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24382.0:70 4.16 4:6d4431ed:lc::lc.7:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2021-02-10T04:28:23.694 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:25:55.115517+0000", 2021-02-10T04:28:23.694 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 148.48511027999999, 2021-02-10T04:28:23.694 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0020255680000000002, 2021-02-10T04:28:23.694 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:23.694 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:28:23.695 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:23.695 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24382", 2021-02-10T04:28:23.695 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/3915233377", 2021-02-10T04:28:23.695 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 70 2021-02-10T04:28:23.695 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.696 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:23.696 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.696 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:23.696 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.115517+0000", 2021-02-10T04:28:23.696 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:23.696 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.697 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.697 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:23.697 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.115517+0000", 2021-02-10T04:28:23.697 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.7299000000000001e-05 2021-02-10T04:28:23.697 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.697 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.697 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:23.697 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.115544+0000", 2021-02-10T04:28:23.698 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9089999999999999e-06 2021-02-10T04:28:23.698 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.698 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.698 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:23.698 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.115549+0000", 2021-02-10T04:28:23.698 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0453e-05 2021-02-10T04:28:23.699 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.699 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.699 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:23.699 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.115559+0000", 2021-02-10T04:28:23.699 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.549e-06 2021-02-10T04:28:23.699 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.699 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.699 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:23.699 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.115564+0000", 2021-02-10T04:28:23.700 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.1001000000000003e-05 2021-02-10T04:28:23.700 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.700 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.700 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:23.700 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.115605+0000", 2021-02-10T04:28:23.700 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000117854 2021-02-10T04:28:23.700 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.700 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.701 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:23.701 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.115723+0000", 2021-02-10T04:28:23.701 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00021387000000000001 2021-02-10T04:28:23.701 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.701 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.701 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for subops from 3,7", 2021-02-10T04:28:23.701 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.115937+0000", 2021-02-10T04:28:23.701 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00043025000000000002 2021-02-10T04:28:23.702 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.702 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.702 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op_commit", 2021-02-10T04:28:23.702 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.116367+0000", 2021-02-10T04:28:23.702 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00045105999999999998 2021-02-10T04:28:23.702 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.702 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.702 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:23.702 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.116818+0000", 2021-02-10T04:28:23.702 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00062451399999999995 2021-02-10T04:28:23.703 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.703 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.703 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:23.703 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.117442+0000", 2021-02-10T04:28:23.703 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.1966999999999998e-05 2021-02-10T04:28:23.704 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.704 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.704 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:28:23.704 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.117504+0000", 2021-02-10T04:28:23.704 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.7842000000000001e-05 2021-02-10T04:28:23.704 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.704 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.704 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:23.704 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.117542+0000", 2021-02-10T04:28:23.705 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0020255680000000002 2021-02-10T04:28:23.705 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.705 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:23.705 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.705 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.705 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.705 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24382.0:150 4.6 4:66924383:reshard::reshard.0000000005:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2021-02-10T04:28:23.705 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:25:55.140960+0000", 2021-02-10T04:28:23.705 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 148.45966668899999, 2021-02-10T04:28:23.705 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0021260369999999999, 2021-02-10T04:28:23.706 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:23.706 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:28:23.706 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:23.706 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24382", 2021-02-10T04:28:23.706 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/3915233377", 2021-02-10T04:28:23.706 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 150 2021-02-10T04:28:23.706 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.706 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:23.706 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.706 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:23.706 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.140960+0000", 2021-02-10T04:28:23.707 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:23.707 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.707 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.707 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:23.707 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.140960+0000", 2021-02-10T04:28:23.707 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.6331000000000001e-05 2021-02-10T04:28:23.708 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.708 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.708 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:23.708 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.141007+0000", 2021-02-10T04:28:23.708 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.3379999999999999e-06 2021-02-10T04:28:23.708 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.708 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.708 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:23.708 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.141014+0000", 2021-02-10T04:28:23.708 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.0392999999999999e-05 2021-02-10T04:28:23.708 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.708 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.708 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:23.709 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.141034+0000", 2021-02-10T04:28:23.709 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.8090000000000004e-06 2021-02-10T04:28:23.709 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.709 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.709 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:23.709 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.141041+0000", 2021-02-10T04:28:23.709 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.5018999999999995e-05 2021-02-10T04:28:23.709 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.709 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.709 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:23.709 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.141106+0000", 2021-02-10T04:28:23.709 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000138589 2021-02-10T04:28:23.709 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.710 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.710 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:23.710 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.141245+0000", 2021-02-10T04:28:23.710 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00020053399999999999 2021-02-10T04:28:23.710 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.710 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.710 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for subops from 1,2", 2021-02-10T04:28:23.710 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.141445+0000", 2021-02-10T04:28:23.710 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0012995789999999999 2021-02-10T04:28:23.710 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.710 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.710 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:23.710 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.142745+0000", 2021-02-10T04:28:23.711 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00011790000000000001 2021-02-10T04:28:23.711 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.711 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.711 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op_commit", 2021-02-10T04:28:23.711 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.142863+0000", 2021-02-10T04:28:23.711 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00014038599999999999 2021-02-10T04:28:23.711 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.711 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.711 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:23.711 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.143003+0000", 2021-02-10T04:28:23.711 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.6925999999999997e-05 2021-02-10T04:28:23.711 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.712 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.712 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:28:23.712 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.143060+0000", 2021-02-10T04:28:23.712 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.6233000000000001e-05 2021-02-10T04:28:23.712 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.712 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.713 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:23.713 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.143086+0000", 2021-02-10T04:28:23.713 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0021260369999999999 2021-02-10T04:28:23.713 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.713 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:23.713 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.714 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.714 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.714 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24382.0:160 4.6 4:66924383:reshard::reshard.0000000005:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2021-02-10T04:28:23.714 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:25:55.144131+0000", 2021-02-10T04:28:23.714 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 148.45649596300001, 2021-02-10T04:28:23.714 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0020016190000000001, 2021-02-10T04:28:23.714 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:23.714 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:28:23.714 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:23.715 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24382", 2021-02-10T04:28:23.715 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/3915233377", 2021-02-10T04:28:23.715 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 160 2021-02-10T04:28:23.715 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.715 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:23.715 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.715 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:23.715 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.144131+0000", 2021-02-10T04:28:23.715 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:23.715 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.715 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.715 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:23.715 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.144131+0000", 2021-02-10T04:28:23.716 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.9851000000000002e-05 2021-02-10T04:28:23.716 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.716 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.716 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:23.716 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.144171+0000", 2021-02-10T04:28:23.716 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.2949999999999996e-06 2021-02-10T04:28:23.716 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.716 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.716 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:23.716 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.144178+0000", 2021-02-10T04:28:23.716 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.6103000000000002e-05 2021-02-10T04:28:23.716 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.717 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.717 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:23.717 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.144194+0000", 2021-02-10T04:28:23.717 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.2349999999999996e-06 2021-02-10T04:28:23.717 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.717 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.717 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:23.717 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.144200+0000", 2021-02-10T04:28:23.717 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.9947000000000002e-05 2021-02-10T04:28:23.717 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.717 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.718 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:23.718 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.144240+0000", 2021-02-10T04:28:23.718 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.9501999999999999e-05 2021-02-10T04:28:23.718 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.718 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.718 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:23.718 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.144320+0000", 2021-02-10T04:28:23.718 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00018683600000000001 2021-02-10T04:28:23.718 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.718 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.718 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for subops from 1,2", 2021-02-10T04:28:23.718 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.144507+0000", 2021-02-10T04:28:23.719 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000440591 2021-02-10T04:28:23.719 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.719 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.719 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op_commit", 2021-02-10T04:28:23.719 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.144947+0000", 2021-02-10T04:28:23.719 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00083617999999999995 2021-02-10T04:28:23.719 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.719 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.719 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:23.719 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.145783+0000", 2021-02-10T04:28:23.719 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00025378199999999998 2021-02-10T04:28:23.719 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.719 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.720 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:23.720 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.146037+0000", 2021-02-10T04:28:23.720 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.8248999999999997e-05 2021-02-10T04:28:23.720 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.720 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.720 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:28:23.720 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.146095+0000", 2021-02-10T04:28:23.720 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.7048000000000002e-05 2021-02-10T04:28:23.720 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.720 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.720 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:23.720 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.146133+0000", 2021-02-10T04:28:23.720 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0020016190000000001 2021-02-10T04:28:23.721 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.721 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:23.721 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.721 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.721 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.721 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24382.0:164 4.16 4:6dc3b799:reshard::reshard.0000000006:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2021-02-10T04:28:23.721 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:25:55.146302+0000", 2021-02-10T04:28:23.721 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 148.45432474200001, 2021-02-10T04:28:23.721 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0019027359999999999, 2021-02-10T04:28:23.721 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:23.721 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:28:23.721 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:23.722 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24382", 2021-02-10T04:28:23.722 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/3915233377", 2021-02-10T04:28:23.722 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 164 2021-02-10T04:28:23.722 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.722 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:23.722 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.723 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:23.723 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.146302+0000", 2021-02-10T04:28:23.723 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:23.723 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.723 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.723 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:23.724 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.146302+0000", 2021-02-10T04:28:23.724 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.3733000000000001e-05 2021-02-10T04:28:23.724 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.724 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.724 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:23.724 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.146326+0000", 2021-02-10T04:28:23.724 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.8450000000000002e-06 2021-02-10T04:28:23.725 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.725 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.725 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:23.725 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.146331+0000", 2021-02-10T04:28:23.725 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2165e-05 2021-02-10T04:28:23.725 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.725 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.725 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:23.725 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.146343+0000", 2021-02-10T04:28:23.725 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.694e-06 2021-02-10T04:28:23.725 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.726 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.726 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:23.726 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.146349+0000", 2021-02-10T04:28:23.726 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.8909999999999998e-05 2021-02-10T04:28:23.726 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.726 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.726 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:23.726 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.146387+0000", 2021-02-10T04:28:23.726 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00013170400000000001 2021-02-10T04:28:23.726 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.726 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.726 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:23.727 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.146519+0000", 2021-02-10T04:28:23.727 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00016178699999999999 2021-02-10T04:28:23.727 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.727 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.727 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for subops from 3,7", 2021-02-10T04:28:23.727 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.146681+0000", 2021-02-10T04:28:23.727 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00037378699999999999 2021-02-10T04:28:23.727 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.727 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.727 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op_commit", 2021-02-10T04:28:23.727 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.147055+0000", 2021-02-10T04:28:23.728 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001050626 2021-02-10T04:28:23.728 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.728 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.728 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:23.728 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.148105+0000", 2021-02-10T04:28:23.728 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.3815e-05 2021-02-10T04:28:23.729 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.729 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.729 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:23.729 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.148149+0000", 2021-02-10T04:28:23.729 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.5179000000000003e-05 2021-02-10T04:28:23.729 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.729 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.729 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:28:23.729 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.148184+0000", 2021-02-10T04:28:23.730 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.0491000000000001e-05 2021-02-10T04:28:23.730 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.730 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.730 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:23.730 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.148205+0000", 2021-02-10T04:28:23.730 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0019027359999999999 2021-02-10T04:28:23.730 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.730 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:23.730 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.730 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.730 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.730 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24382.0:177 4.16 4:6dc3b799:reshard::reshard.0000000006:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2021-02-10T04:28:23.731 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:25:55.149301+0000", 2021-02-10T04:28:23.731 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 148.451325852, 2021-02-10T04:28:23.731 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0026554740000000001, 2021-02-10T04:28:23.731 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:23.731 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:28:23.731 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:23.731 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24382", 2021-02-10T04:28:23.731 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/3915233377", 2021-02-10T04:28:23.731 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 177 2021-02-10T04:28:23.731 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.731 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:23.731 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.731 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:23.732 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.149301+0000", 2021-02-10T04:28:23.732 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:23.732 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.732 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.732 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:23.732 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.149301+0000", 2021-02-10T04:28:23.732 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.2657e-05 2021-02-10T04:28:23.732 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.732 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.732 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:23.732 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.149324+0000", 2021-02-10T04:28:23.732 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.7350000000000004e-06 2021-02-10T04:28:23.732 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.733 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.733 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:23.733 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.149328+0000", 2021-02-10T04:28:23.733 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.0770000000000008e-06 2021-02-10T04:28:23.733 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.733 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.733 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:23.733 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.149336+0000", 2021-02-10T04:28:23.733 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.3220000000000003e-06 2021-02-10T04:28:23.733 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.733 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.733 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:23.733 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.149341+0000", 2021-02-10T04:28:23.733 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.1254000000000002e-05 2021-02-10T04:28:23.734 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.734 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.734 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:23.734 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.149382+0000", 2021-02-10T04:28:23.734 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.1033999999999997e-05 2021-02-10T04:28:23.734 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.734 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.734 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:23.734 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.149463+0000", 2021-02-10T04:28:23.734 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000202303 2021-02-10T04:28:23.734 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.734 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.735 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for subops from 3,7", 2021-02-10T04:28:23.735 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.149665+0000", 2021-02-10T04:28:23.735 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00041027099999999998 2021-02-10T04:28:23.735 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.735 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.735 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op_commit", 2021-02-10T04:28:23.735 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.150076+0000", 2021-02-10T04:28:23.735 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0010694260000000001 2021-02-10T04:28:23.735 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.735 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.735 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:23.735 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.151145+0000", 2021-02-10T04:28:23.735 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00073166699999999997 2021-02-10T04:28:23.736 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.736 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.736 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:23.736 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.151877+0000", 2021-02-10T04:28:23.736 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4861999999999998e-05 2021-02-10T04:28:23.736 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.736 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.736 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:28:23.736 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.151922+0000", 2021-02-10T04:28:23.736 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.4866000000000001e-05 2021-02-10T04:28:23.736 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.736 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.736 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:23.737 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.151956+0000", 2021-02-10T04:28:23.737 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0026554740000000001 2021-02-10T04:28:23.737 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.737 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:23.737 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.737 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.737 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.737 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24382.0:245 4.6 4:64760185:::obj_delete_at_hint.0000000010:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2021-02-10T04:28:23.737 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:25:55.166709+0000", 2021-02-10T04:28:23.737 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 148.433917845, 2021-02-10T04:28:23.737 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0021660189999999999, 2021-02-10T04:28:23.737 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:23.737 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:28:23.738 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:23.738 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24382", 2021-02-10T04:28:23.738 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/3915233377", 2021-02-10T04:28:23.738 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 245 2021-02-10T04:28:23.738 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.738 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:23.738 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.738 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:23.738 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.166709+0000", 2021-02-10T04:28:23.738 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:23.739 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.739 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.739 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:23.739 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.166709+0000", 2021-02-10T04:28:23.739 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4014999999999998e-05 2021-02-10T04:28:23.739 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.739 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.739 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:23.739 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.166753+0000", 2021-02-10T04:28:23.739 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.9099999999999999e-06 2021-02-10T04:28:23.740 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.740 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.740 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:23.740 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.166760+0000", 2021-02-10T04:28:23.740 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.4464000000000001e-05 2021-02-10T04:28:23.740 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.740 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.740 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:23.740 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.166774+0000", 2021-02-10T04:28:23.740 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.8819999999999998e-06 2021-02-10T04:28:23.740 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.740 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.740 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:23.741 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.166780+0000", 2021-02-10T04:28:23.741 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.5318e-05 2021-02-10T04:28:23.741 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.741 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.741 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:23.741 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.166846+0000", 2021-02-10T04:28:23.741 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.0482999999999994e-05 2021-02-10T04:28:23.741 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.741 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.741 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:23.741 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.166936+0000", 2021-02-10T04:28:23.741 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000191471 2021-02-10T04:28:23.742 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.742 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.742 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for subops from 1,2", 2021-02-10T04:28:23.742 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.167128+0000", 2021-02-10T04:28:23.742 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00087631700000000005 2021-02-10T04:28:23.742 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.742 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.742 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op_commit", 2021-02-10T04:28:23.742 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.168004+0000", 2021-02-10T04:28:23.742 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000436484 2021-02-10T04:28:23.742 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.742 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.743 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:23.743 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.168440+0000", 2021-02-10T04:28:23.743 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00034871300000000001 2021-02-10T04:28:23.743 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.743 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.743 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:23.743 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.168789+0000", 2021-02-10T04:28:23.744 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.2552000000000002e-05 2021-02-10T04:28:23.744 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.744 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.744 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:28:23.744 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.168832+0000", 2021-02-10T04:28:23.744 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.3409999999999999e-05 2021-02-10T04:28:23.745 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.745 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.745 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:23.745 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.168875+0000", 2021-02-10T04:28:23.745 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0021660189999999999 2021-02-10T04:28:23.745 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.745 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:23.746 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.746 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.746 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.746 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24382.0:653 4.3 4:c134263f:lc::lc.25:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2021-02-10T04:28:23.746 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:25:55.273715+0000", 2021-02-10T04:28:23.746 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 148.326911653, 2021-02-10T04:28:23.746 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0040237099999999998, 2021-02-10T04:28:23.746 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:23.746 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:28:23.747 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:23.747 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24382", 2021-02-10T04:28:23.747 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/3915233377", 2021-02-10T04:28:23.747 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 653 2021-02-10T04:28:23.747 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.747 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:23.747 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.747 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:23.747 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.273715+0000", 2021-02-10T04:28:23.747 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:23.747 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.747 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.747 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:23.747 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.273715+0000", 2021-02-10T04:28:23.748 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.3133e-05 2021-02-10T04:28:23.748 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.748 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.748 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:23.748 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.273738+0000", 2021-02-10T04:28:23.748 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.6839999999999998e-06 2021-02-10T04:28:23.748 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.748 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.748 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:23.748 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.273742+0000", 2021-02-10T04:28:23.748 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.4630000000000001e-06 2021-02-10T04:28:23.748 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.749 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.749 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:23.749 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.273751+0000", 2021-02-10T04:28:23.749 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.7389999999999999e-06 2021-02-10T04:28:23.749 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.749 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.749 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:23.749 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.273758+0000", 2021-02-10T04:28:23.749 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.8941999999999998e-05 2021-02-10T04:28:23.749 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.749 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.750 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:23.750 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.273787+0000", 2021-02-10T04:28:23.750 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.3495000000000002e-05 2021-02-10T04:28:23.750 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.750 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.750 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:23.750 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.273841+0000", 2021-02-10T04:28:23.750 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00012563 2021-02-10T04:28:23.750 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.750 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.750 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for subops from 5,7", 2021-02-10T04:28:23.750 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.273966+0000", 2021-02-10T04:28:23.751 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00071814299999999997 2021-02-10T04:28:23.751 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.751 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.751 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op_commit", 2021-02-10T04:28:23.751 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.274684+0000", 2021-02-10T04:28:23.751 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0028691070000000001 2021-02-10T04:28:23.751 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.751 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.751 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:23.751 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.277554+0000", 2021-02-10T04:28:23.751 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000101462 2021-02-10T04:28:23.752 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.752 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.752 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:23.752 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.277655+0000", 2021-02-10T04:28:23.752 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.6817999999999998e-05 2021-02-10T04:28:23.752 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.752 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.753 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:28:23.753 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.277702+0000", 2021-02-10T04:28:23.753 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.7094000000000001e-05 2021-02-10T04:28:23.753 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.753 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.753 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:23.753 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.277739+0000", 2021-02-10T04:28:23.753 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0040237099999999998 2021-02-10T04:28:23.753 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.754 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:23.754 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.754 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.754 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.754 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:786 5.3 5:c609908c:::notify.5:head [watch ping cookie 94187770079488] snapc 0=[] ondisk+write+known_if_redirected e65)", 2021-02-10T04:28:23.754 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:26:17.708779+0000", 2021-02-10T04:28:23.754 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 125.89184826499999, 2021-02-10T04:28:23.754 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0026955909999999998, 2021-02-10T04:28:23.754 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:23.754 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:28:23.754 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:23.755 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:28:23.755 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:28:23.755 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 786 2021-02-10T04:28:23.755 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.755 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:23.755 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.755 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:23.755 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:26:17.708779+0000", 2021-02-10T04:28:23.755 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:23.755 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.756 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.756 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:23.756 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:26:17.708779+0000", 2021-02-10T04:28:23.756 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.455e-05 2021-02-10T04:28:23.756 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.756 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.756 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:23.756 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:26:17.708813+0000", 2021-02-10T04:28:23.756 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.8139999999999995e-06 2021-02-10T04:28:23.756 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.756 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.756 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:23.757 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:26:17.708823+0000", 2021-02-10T04:28:23.757 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.8664999999999998e-05 2021-02-10T04:28:23.757 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.757 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.757 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:23.757 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:26:17.708842+0000", 2021-02-10T04:28:23.757 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0686999999999999e-05 2021-02-10T04:28:23.757 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.757 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.757 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:23.757 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:26:17.708852+0000", 2021-02-10T04:28:23.757 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.6649000000000003e-05 2021-02-10T04:28:23.757 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.757 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.758 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:23.758 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:26:17.708939+0000", 2021-02-10T04:28:23.758 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.3319999999999998e-06 2021-02-10T04:28:23.758 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.758 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.758 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op must wait for map", 2021-02-10T04:28:23.758 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:26:17.708948+0000", 2021-02-10T04:28:23.758 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0023250530000000001 2021-02-10T04:28:23.758 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.758 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.758 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:23.758 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:26:17.711273+0000", 2021-02-10T04:28:23.759 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.0395999999999994e-05 2021-02-10T04:28:23.759 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.759 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.759 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:23.759 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:26:17.711354+0000", 2021-02-10T04:28:23.759 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000120445 2021-02-10T04:28:23.759 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.759 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.759 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:23.759 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:26:17.711474+0000", 2021-02-10T04:28:23.759 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0026955909999999998 2021-02-10T04:28:23.759 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.759 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:23.759 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.760 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.760 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.760 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:930 5.3 5:c609908c:::notify.5:head [watch ping cookie 94187770079488] snapc 0=[] ondisk+write+known_if_redirected e84)", 2021-02-10T04:28:23.760 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:27:47.724836+0000", 2021-02-10T04:28:23.760 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 35.875791196000002, 2021-02-10T04:28:23.760 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0019910069999999999, 2021-02-10T04:28:23.760 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:23.760 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:28:23.760 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:23.760 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:28:23.760 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:28:23.760 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 930 2021-02-10T04:28:23.760 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.761 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:23.761 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.761 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:23.761 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:27:47.724836+0000", 2021-02-10T04:28:23.761 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:23.761 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.761 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.761 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:23.761 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:27:47.724836+0000", 2021-02-10T04:28:23.761 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.3785000000000003e-05 2021-02-10T04:28:23.761 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.761 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.761 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:23.762 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:27:47.724869+0000", 2021-02-10T04:28:23.762 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.6389999999999992e-06 2021-02-10T04:28:23.762 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.762 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.762 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:23.762 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:27:47.724877+0000", 2021-02-10T04:28:23.762 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.4209000000000001e-05 2021-02-10T04:28:23.762 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.762 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.762 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:23.762 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:27:47.724891+0000", 2021-02-10T04:28:23.762 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.9859999999999997e-06 2021-02-10T04:28:23.762 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.763 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.763 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:23.763 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:27:47.724898+0000", 2021-02-10T04:28:23.763 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.0419999999999993e-05 2021-02-10T04:28:23.763 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.763 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.763 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:23.763 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:27:47.724969+0000", 2021-02-10T04:28:23.763 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.1339999999999998e-06 2021-02-10T04:28:23.763 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.763 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.763 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op must wait for map", 2021-02-10T04:28:23.763 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:27:47.724976+0000", 2021-02-10T04:28:23.764 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001655288 2021-02-10T04:28:23.764 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.764 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.764 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:23.764 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:27:47.726631+0000", 2021-02-10T04:28:23.764 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.4847999999999998e-05 2021-02-10T04:28:23.764 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.764 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.764 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:23.764 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:27:47.726716+0000", 2021-02-10T04:28:23.764 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000110698 2021-02-10T04:28:23.764 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:23.765 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:23.765 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:23.765 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:27:47.726827+0000", 2021-02-10T04:28:23.765 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0019910069999999999 2021-02-10T04:28:23.765 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.765 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:23.765 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.765 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:23.765 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:23.766 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:28:23.767 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:28:23.868 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:28:23.894 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:24.145 INFO:teuthology.orchestra.run.smithi085.stderr:pool 'unique_pool_0' created 2021-02-10T04:28:24.167 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:24 smithi085 conmon[35961]: cluster 2021-02-10T04:28:23.122387+0000 mon.a (mon.0) 724 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2021-02-10T04:28:24.168 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:24 smithi085 conmon[35961]: audit 2021-02-10T04:28:23.673015+0000 mon.c (mon.1) 34 : audit [INF] from='client.? 172.21.15.85:0/1853331914' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2021-02-10T04:28:24.168 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:24 smithi085 conmon[35961]: audit 2021-02-10T04:28:23.673404+0000 mon.a (mon.0) 725 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2021-02-10T04:28:24.168 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:24 smithi085 conmon[35961]: cluster 2021-02-10T04:28:23.989512+0000 mgr.y (mgr.14140) 324 : cluster [DBG] pgmap v285: 142 pgs: 4 peering, 10 unknown, 128 active+clean; 25 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-02-10T04:28:24.168 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:24 smithi085 conmon[29319]: cluster 2021-02-10T04:28:23.122387+0000 mon.a (mon.0) 724 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2021-02-10T04:28:24.168 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:24 smithi085 conmon[29319]: audit 2021-02-10T04:28:23.673015+0000 mon.c (mon.1) 34 : audit [INF] from='client.? 172.21.15.85:0/1853331914' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2021-02-10T04:28:24.168 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:24 smithi085 conmon[29319]: audit 2021-02-10T04:28:23.673404+0000 mon.a (mon.0) 725 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2021-02-10T04:28:24.168 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:24 smithi085 conmon[29319]: cluster 2021-02-10T04:28:23.989512+0000 mgr.y (mgr.14140) 324 : cluster [DBG] pgmap v285: 142 pgs: 4 peering, 10 unknown, 128 active+clean; 25 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-02-10T04:28:24.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:24 smithi164 conmon[29288]: cluster 2021-02-10T04:28:23.122387+0000 mon.a 2021-02-10T04:28:24.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:24 smithi164 conmon[29288]: (mon.0) 724 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2021-02-10T04:28:24.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:24 smithi164 conmon[29288]: audit 2021-02-10T04:28:23.673015+0000 mon.c (mon.1) 34 : audit [INF] from='client.? 172.21.15.85:0/1853331914' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2021-02-10T04:28:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:24 smithi164 conmon[29288]: audit 2021-02-10T04:28:23.673404 2021-02-10T04:28:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:24 smithi164 conmon[29288]: +0000 mon.a (mon.0) 725 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2021-02-10T04:28:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:24 smithi164 conmon[29288]: cluster 2021-02-10T04:28:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:24 smithi164 conmon[29288]: 2021-02-10T04:28:23 2021-02-10T04:28:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:24 smithi164 conmon[29288]: .989512+0000 mgr.y (mgr.14140) 324 : cluster [DBG] pgmap v285: 142 pgs: 4 peering, 10 unknown, 128 active+clean; 25 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-02-10T04:28:24.638 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:28:24.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:28:24 smithi085 conmon[49106]: debug 2021-02-10T04:28:24.452+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:24.743 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:28:24.770 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:24.954 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset noscrub 2021-02-10T04:28:25.043 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-02-10T04:28:25.043 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:28:25.147 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:25 smithi085 conmon[35961]: cluster 2021-02-10T04:28:24.081562+0000 osd.3 (osd.3) 32 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:28:25.147 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:25 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:28:25.147 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:25 smithi085 conmon[35961]: -02-10T04:28:24.081648+0000 osd.3 (osd.3) 33 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:28:25.147 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:25 smithi085 conmon[35961]: audit 2021-02-10T04:28:24.135800+0000 mon.a ( 2021-02-10T04:28:25.147 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:25 smithi085 conmon[35961]: mon.0) 726 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2021-02-10T04:28:25.148 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:25 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:28:25.148 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:25 smithi085 conmon[35961]: 10T04:28:24.135864+0000 mon.a ( 2021-02-10T04:28:25.148 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:25 smithi085 conmon[35961]: mon.0) 727 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2021-02-10T04:28:25.172 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:28:25.239 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight 2021-02-10T04:28:25.261 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:28:25.366 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:28:25.390 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:25.439 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:28:25.440 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:28:25 smithi085 conmon[52836]: debug 2021-02-10T04:28:25.176+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:25.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:25 smithi085 conmon[29319]: cluster 2021-02-10T04:28:24.081562+0000 osd.3 (osd.3) 32 : cluster 2021-02-10T04:28:25.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:25 smithi085 conmon[29319]: [DBG] purged_snaps scrub starts 2021-02-10T04:28:25.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:25 smithi085 conmon[29319]: cluster 2021-02-10T04:28:24.081648+0000 osd.3 (osd.3) 33 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:28:25.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:25 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:28:25.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:25 smithi085 conmon[29319]: -10T04:28:24.135800+0000 mon.a (mon.0) 726 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2021-02-10T04:28:25.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:25 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:28:25.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:25 smithi085 conmon[29319]: 10T04:28:24.135864+0000 mon.a ( 2021-02-10T04:28:25.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:25 smithi085 conmon[29319]: mon.0) 727 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2021-02-10T04:28:25.478 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2021-02-10T04:28:25.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:25 smithi164 conmon[29288]: cluster 2021-02-10T04:28:25.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:25 smithi164 conmon[29288]: 2021-02-10T04:28:25.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:25 smithi164 conmon[29288]: 2021 2021-02-10T04:28:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:25 smithi164 conmon[29288]: - 2021-02-10T04:28:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:25 smithi164 conmon[29288]: 02-10T04:28:24.081562+0000 osd.3 (osd 2021-02-10T04:28:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:25 smithi164 conmon[29288]: .3) 32 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:28:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:25 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:28:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:25 smithi164 conmon[29288]: -10T04:28:24.081648+0000 osd.3 (osd.3) 33 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:28:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:25 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:28:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:25 smithi164 conmon[29288]: 04:28:24.135800+0000 mon.a (mon.0) 726 : audit 2021-02-10T04:28:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:25 smithi164 conmon[29288]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2021-02-10T04:28:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:25 smithi164 conmon[29288]: cluster 2021-02-10T04:28:24.135864+0000 2021-02-10T04:28:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:25 smithi164 conmon[29288]: mon.a (mon.0) 727 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2021-02-10T04:28:25.669 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:28:25.721 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:28:25.733 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:28:25 smithi085 conmon[49106]: debug 2021-02-10T04:28:25.578+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:25.836 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:28:25.868 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:26.064 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:28:26.170 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:28:26.200 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:26.280 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:28:26 smithi085 conmon[49106]: debug 2021-02-10T04:28:25.999+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:26.280 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:26 smithi085 conmon[29319]: cluster 2021-02-10T04:28:25.139110+0000 2021-02-10T04:28:26.281 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:26 smithi085 conmon[29319]: mon.a (mon.0) 728 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2021-02-10T04:28:26.281 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:26 smithi085 conmon[29319]: cluster 2021-02-10T04:28:25.989918+0000 mgr.y (mgr.14140) 325 : cluster [DBG] pgmap v288: 158 pgs: 7 creating+peering, 1 peering, 10 unknown, 140 active+clean; 25 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:28:26.281 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:26 smithi085 conmon[35961]: cluster 2021-02-10T04:28:25 2021-02-10T04:28:26.281 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:26 smithi085 conmon[35961]: .139110+0000 mon.a (mon.0) 728 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2021-02-10T04:28:26.281 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:26 smithi085 conmon[35961]: cluster 2021-02-10T04:28:25. 2021-02-10T04:28:26.281 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:26 smithi085 conmon[35961]: 989918+0000 mgr.y (mgr.14140) 325 : cluster [DBG] pgmap v288: 158 pgs: 7 creating+peering, 1 peering, 10 unknown, 140 active+clean; 25 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:28:26.304 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:28:26.410 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:28:26.433 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:26 smithi164 conmon[29288]: cluster 2021-02-10T04:28:25.139110+0000 mon.a (mon.0) 728 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2021-02-10T04:28:26.433 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:26 smithi164 conmon[29288]: cluster 2021-02-10T04:28:25.989918+0000 mgr.y (mgr.14140) 325 : cluster [DBG] pgmap v288: 158 pgs: 7 creating+peering, 1 peering, 10 unknown, 140 active+clean; 25 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:28:26.434 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:28:26 smithi164 conmon[44544]: debug 2021-02-10T04:28:26.275+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:26.435 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:26.546 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:28:26.651 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:28:26.675 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:26.730 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:28:26 smithi164 conmon[44544]: debug 2021-02-10T04:28:26.492+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:26.754 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:28:26.858 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:28:26.885 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:27.143 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:28:26 smithi164 conmon[44544]: debug 2021-02-10T04:28:26.729+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:27.325 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:28:27.341 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:28:27.358 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:28:27.375 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:28:27.394 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:28:27.412 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:28:27.429 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:28:27.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:27 smithi085 conmon[29319]: cluster 2021-02-10T04:28:26 2021-02-10T04:28:27.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:27 smithi085 conmon[29319]: .152514+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2021-02-10T04:28:27.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:27 smithi085 conmon[35961]: cluster 2021-02-10T04:28: 2021-02-10T04:28:27.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:27 smithi085 conmon[35961]: 26.152514+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2021-02-10T04:28:27.447 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:28:27.448 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:28:27.449 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:28:27.449 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:28:27.449 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:28:27.449 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:28:27.449 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:28:27.449 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:28:27.449 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:28:27.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:27 smithi164 conmon[29288]: cluster 2021-02-10T04:28:26.152514+0000 mon.a (mon.0) 2021-02-10T04:28:27.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:27 smithi164 conmon[29288]: 729 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2021-02-10T04:28:27.770 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:28:27.875 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:28:27.902 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:27.961 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:28:27 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:28:28.012 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:28:28.117 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:28:28.123 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:28:27 smithi085 conmon[41540]: debug 2021-02-10T04:28:27.747+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:28.143 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:28.393 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:28:27 smithi164 conmon[40892]: debug 2021-02-10T04:28:27.960+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:28.420 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:28 smithi164 conmon[29288]: audit 2021-02-10T04:28:27.308249+0000 mon.c (mon.1) 35 : audit [DBG] from='client.? 172.21.15.85:0/1154602398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:28:28.420 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:28 smithi164 conmon[29288]: audit 2021-02-10T04:28:27.665362+0000 mgr.y (mgr.14140) 326 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:28:28.420 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:28 smithi164 conmon[29288]: cluster 2021-02-10T04:28:27.990512+0000 mgr.y (mgr.14140 2021-02-10T04:28:28.420 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:28 smithi164 conmon[29288]: ) 327 : cluster [DBG] pgmap v290: 158 pgs: 7 creating+peering, 5 unknown, 146 active+clean; 25 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 210 B/s rd, 420 B/s wr, 0 op/s; 9 B/s, 0 objects/s recovering 2021-02-10T04:28:28.421 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:28 smithi085 conmon[29319]: audit 2021-02-10T04:28:27. 2021-02-10T04:28:28.421 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:28 smithi085 conmon[29319]: 308249+0000 mon.c (mon.1) 35 : audit [DBG] from='client.? 172.21.15.85:0/1154602398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:28:28.422 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:28 smithi085 conmon[29319]: audit 2021-02-10T04:28:27.665362+0000 mgr.y (mgr.14140) 326 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:28:28.422 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:28 smithi085 conmon[29319]: cluster 2021-02-10T04:28:27.990512+0000 mgr.y (mgr.14140) 327 : cluster [DBG] pgmap v290: 158 pgs: 7 creating+peering, 5 unknown, 146 active+clean; 25 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 210 B/s rd, 420 B/s wr, 0 op/s; 9 B/s, 0 objects/s recovering 2021-02-10T04:28:28.423 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:28 smithi085 conmon[35961]: audit 2021-02-10T04:28:27.308249 2021-02-10T04:28:28.423 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:28 smithi085 conmon[35961]: +0000 mon.c (mon.1) 35 : audit [DBG] from='client.? 172.21.15.85:0/1154602398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:28:28.423 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:28 smithi085 conmon[35961]: audit 2021-02-10T04:28:27.665362+0000 mgr.y (mgr.14140) 326 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:28:28.423 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:28 smithi085 conmon[35961]: cluster 2021-02-10T04:28:27.990512+0000 mgr.y (mgr.14140) 327 : cluster [DBG] pgmap v290: 158 pgs: 7 creating+peering, 5 unknown, 146 active+clean; 25 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 210 B/s rd, 420 B/s wr, 0 op/s; 9 B/s, 0 objects/s recovering 2021-02-10T04:28:28.913 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:28:28.913 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:28:28.930 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:28:28 smithi085 conmon[52836]: debug 2021-02-10T04:28:28.809+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:29.165 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is unset 2021-02-10T04:28:29.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:29 smithi085 conmon[35961]: audit 2021-02-10T04:28:28.787622+0000 2021-02-10T04:28:29.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:29 smithi085 conmon[35961]: mon.a (mon.0) 730 : audit [INF] from='client.? 172.21.15.85:0/1620366436' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:28:29.185 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:29 smithi085 conmon[29319]: audit 2021-02-10T04:28:28.787622+0000 mon.a (mon.0 2021-02-10T04:28:29.185 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:29 smithi085 conmon[29319]: ) 730 : audit [INF] from='client.? 172.21.15.85:0/1620366436' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:28:29.216 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:28:29.323 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:28:29.350 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:29.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:29 smithi164 conmon[29288]: audit 2021-02-10T04:28:28.787622+0000 mon.a (mon.0) 730 : audit [INF] from='client.? 172.21.15.85:0/1620366436' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:28:30.419 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:28:30.431 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:28:30 smithi085 conmon[52836]: debug 2021-02-10T04:28:30.363+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:30.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:30 smithi085 conmon[29319]: cluster 2021-02-10T04:28:29.154891+0000 mon.a (mon.0) 731 : 2021-02-10T04:28:30.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:30 smithi085 conmon[29319]: cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:28:30.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:30 smithi085 conmon[29319]: audit 2021-02-10T04:28:29.162594+0000 mon.a (mon.0) 732 : audit [INF] from='client.? 172.21.15.85:0/1620366436' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:28:30.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:30 smithi085 conmon[29319]: cluster 2021-02-10T04:28:29.162656+0000 mon.a (mon.0) 733 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2021-02-10T04:28:30.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:30 smithi085 conmon[29319]: cluster 2021-02-10T04:28:29.990995+0000 mgr.y (mgr.14140) 328 : cluster [DBG] 2021-02-10T04:28:30.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:30 smithi085 conmon[29319]: pgmap v292: 158 pgs: 7 creating+peering, 151 active+clean; 25 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 349 B/s wr, 1 op/s; 7 B/s, 0 objects/s recovering 2021-02-10T04:28:30.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:30 smithi085 conmon[35961]: cluster 2021-02-10T04:28:29.154891 2021-02-10T04:28:30.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:30 smithi085 conmon[35961]: +0000 mon.a (mon.0) 731 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:28:30.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:30 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:28:30.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:30 smithi085 conmon[35961]: T04:28:29.162594+0000 mon.a (mon.0) 2021-02-10T04:28:30.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:30 smithi085 conmon[35961]: 732 : audit [INF] from='client.? 172.21.15.85:0/1620366436' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:28:30.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:30 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:28:30.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:30 smithi085 conmon[35961]: 02-10T04:28:29. 2021-02-10T04:28:30.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:30 smithi085 conmon[35961]: 162656+0000 mon.a (mon.0) 2021-02-10T04:28:30.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:30 smithi085 conmon[35961]: 733 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2021-02-10T04:28:30.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:30 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:28:30.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:30 smithi085 conmon[35961]: -02-10T04:28: 2021-02-10T04:28:30.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:30 smithi085 conmon[35961]: 29.990995+0000 mgr.y (mgr 2021-02-10T04:28:30.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:30 smithi085 conmon[35961]: .14140) 328 : cluster [DBG] 2021-02-10T04:28:30.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:30 smithi085 conmon[35961]: pgmap v292: 158 pgs: 7 creating+peering, 151 active+clean; 25 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 349 B/s wr, 1 op/s; 7 B/s, 0 objects/s recovering 2021-02-10T04:28:30.434 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.1 dump_blocked_ops 2021-02-10T04:28:30.495 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:28:30.564 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:28:30.600 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:30.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:30 smithi164 conmon[29288]: cluster 2021-02-10T04:28:29.154891+0000 mon.a (mon.0) 731 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:28:30.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:30 smithi164 conmon[29288]: audit 2021-02-10T04:28:29.162594+0000 mon.a (mon.0) 732 : audit [INF] from='client.? 172.21.15.85:0/1620366436' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:28:30.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:30 smithi164 conmon[29288]: cluster 2021-02-10T04:28:29.162656+0000 mon.a (mon.0) 733 : cluster [DBG] 2021-02-10T04:28:30.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:30 smithi164 conmon[29288]: osdmap e96: 8 total, 8 up, 8 in 2021-02-10T04:28:30.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:30 smithi164 conmon[29288]: cluster 2021-02-10T04:28:30.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:30 smithi164 conmon[29288]: 2021-02-10T04:28:29.990995 2021-02-10T04:28:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:30 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 328 : 2021-02-10T04:28:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:30 smithi164 conmon[29288]: cluster [DBG] pgmap v292: 158 pgs: 7 creating+peering, 151 active+clean; 25 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 349 B/s wr, 1 op/s; 7 B/s, 0 objects/s recovering 2021-02-10T04:28:30.691 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:28:30 smithi085 conmon[41540]: debug 2021-02-10T04:28:30.664+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:30.711 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:28:30.757 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:28:30.794 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:28:30.900 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:28:30.924 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:31.054 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:28:31 smithi085 conmon[41540]: debug 2021-02-10T04:28:31.039+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:31.250 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:28:31.252 INFO:teuthology.orchestra.run.smithi085.stderr:enabled application 'rados' on pool 'unique_pool_0' 2021-02-10T04:28:31.339 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:31 smithi085 conmon[29319]: cluster 2021-02-10T04:28:30.229087+0000 osd.0 (osd.0) 30 : cluster 2021-02-10T04:28:31.339 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:31 smithi085 conmon[29319]: [DBG] purged_snaps scrub starts 2021-02-10T04:28:31.339 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:31 smithi085 conmon[29319]: cluster 2021-02-10T04:28:30.229216+0000 osd.0 (osd.0) 31 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:28:31.339 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:31 smithi085 conmon[29319]: cluster 2021-02-10T04:28:30.415130+0000 osd.3 (osd.3) 34 : cluster [DBG] 2.1b scrub ok 2021-02-10T04:28:31.339 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:31 smithi085 conmon[29319]: audit 2021-02-10T04:28:30.994360+0000 mon.a (mon.0) 734 : audit [INF] from='client.? 172.21.15.85:0/4028168328' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2021-02-10T04:28:31.339 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:31 smithi085 conmon[35961]: cluster 2021-02-10T04:28:30.229087+0000 osd.0 (osd.0 2021-02-10T04:28:31.340 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:31 smithi085 conmon[35961]: ) 30 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:28:31.340 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:31 smithi085 conmon[35961]: cluster 2021-02-10T04:28:30.229216+0000 osd.0 (osd 2021-02-10T04:28:31.340 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:31 smithi085 conmon[35961]: .0) 31 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:28:31.340 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:31 smithi085 conmon[35961]: cluster 2021-02-10T04:28:30. 2021-02-10T04:28:31.340 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:31 smithi085 conmon[35961]: 415130+0000 osd.3 (osd.3) 34 : cluster [DBG] 2.1b scrub ok 2021-02-10T04:28:31.340 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:31 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:28:31.340 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:31 smithi085 conmon[35961]: -10T04:28:30.994360+0000 mon.a (mon. 2021-02-10T04:28:31.340 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:31 smithi085 conmon[35961]: 0) 734 : audit [INF] from='client.? 172.21.15.85:0/4028168328' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2021-02-10T04:28:31.352 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:28:31.384 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:31.448 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:31 smithi164 conmon[29288]: cluster 2021-02-10T04:28:30.229087+0000 osd.0 (osd.0) 30 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:28:31.448 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:31 smithi164 conmon[29288]: cluster 2021-02-10T04:28:30.229216+0000 osd.0 (osd.0) 31 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:28:31.448 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:31 smithi164 conmon[29288]: cluster 2021-02-10T04:28:30.415130+0000 osd.3 (osd.3) 34 : cluster [DBG] 2.1b scrub ok 2021-02-10T04:28:31.448 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:31 smithi164 conmon[29288]: audit 2021-02-10T04:28:30.994360 2021-02-10T04:28:31.448 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:31 smithi164 conmon[29288]: +0000 mon.a (mon.0) 734 : audit 2021-02-10T04:28:31.448 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:31 smithi164 conmon[29288]: [INF] from='client.? 172.21.15.85:0/4028168328' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2021-02-10T04:28:31.478 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:28:31.582 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:28:31.609 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:31.893 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:28:31 smithi164 conmon[37263]: debug 2021-02-10T04:28:31.447+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:31.913 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:28:32.019 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:28:32.046 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:32.173 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:28:31 smithi085 conmon[52836]: debug 2021-02-10T04:28:31.865+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:32.441 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:28:32 smithi085 conmon[41540]: debug 2021-02-10T04:28:32.305+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:32.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:32 smithi085 conmon[35961]: cluster 2021-02-10T04:28:30.424319+0000 osd.7 (osd.7) 27 : cluster [DBG] 1.0 scrub ok 2021-02-10T04:28:32.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:32 smithi085 conmon[35961]: cluster 2021-02-10T04:28:31.102553+0000 osd.1 (osd.1) 32 : cluster [DBG] 2.d scrub ok 2021-02-10T04:28:32.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:32 smithi085 conmon[35961]: audit 2021-02-10T04:28:31.171700+0000 mon.a (mon.0) 735 : audit [INF] from='client.? 172.21.15.85:0/4028168328' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2021-02-10T04:28:32.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:32 smithi085 conmon[35961]: cluster 2021-02-10T04:28:31.171770+0000 mon.a (mon.0) 736 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2021-02-10T04:28:32.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:32 smithi085 conmon[35961]: cluster 2021-02-10T04:28:31.382665+0000 osd.3 (osd.3) 35 : cluster [DBG] 4.11 scrub ok 2021-02-10T04:28:32.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:32 smithi085 conmon[35961]: cluster 2021-02-10T04:28:31.991546+0000 mgr.y (mgr.14140) 329 : cluster [DBG] pgmap v294: 158 pgs: 158 active+clean; 25 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 170 B/s wr, 1 op/s; 7 B/s, 0 objects/s recovering 2021-02-10T04:28:32.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:32 smithi085 conmon[29319]: cluster 2021-02-10T04:28:30.424319+0000 osd.7 (osd.7) 2021-02-10T04:28:32.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:32 smithi085 conmon[29319]: 27 : cluster [DBG] 1.0 scrub ok 2021-02-10T04:28:32.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:32 smithi085 conmon[29319]: cluster 2021-02-10T04:28:31.102553+0000 osd.1 (osd.1) 32 : cluster [DBG] 2.d scrub ok 2021-02-10T04:28:32.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:32 smithi085 conmon[29319]: audit 2021-02-10T04:28:31.171700+0000 mon.a (mon.0) 735 : audit [INF] from='client.? 172.21.15.85:0/4028168328' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2021-02-10T04:28:32.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:32 smithi085 conmon[29319]: cluster 2021-02-10T04:28:31.171770+0000 mon.a (mon.0) 736 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2021-02-10T04:28:32.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:32 smithi085 conmon[29319]: cluster 2021-02-10T04:28:31.382665+0000 osd.3 ( 2021-02-10T04:28:32.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:32 smithi085 conmon[29319]: osd.3) 35 : cluster [DBG] 4.11 scrub ok 2021-02-10T04:28:32.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:32 smithi085 conmon[29319]: cluster 2021-02-10T04:28:31.991546+0000 mgr.y (mgr.14140) 329 : cluster [DBG] pgmap v294: 158 pgs: 158 active+clean; 25 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 170 B/s wr, 1 op/s; 7 B/s, 0 objects/s recovering 2021-02-10T04:28:32.473 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:28:32.491 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset nodeep-scrub 2021-02-10T04:28:32.593 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:28:32.621 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:32.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:32 smithi164 conmon[29288]: cluster 2021-02-10T04:28:30.424319+0000 osd.7 (osd.7) 27 : cluster [DBG] 1.0 scrub ok 2021-02-10T04:28:32.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:32 smithi164 conmon[29288]: 2021-02-10T04:28:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:32 smithi164 conmon[29288]: cluster 2021-02-10T04:28:31.102553+0000 osd.1 (osd.1) 32 : cluster [DBG] 2.d scrub ok 2021-02-10T04:28:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:32 smithi164 conmon[29288]: audit 2021-02-10T04: 2021-02-10T04:28:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:32 smithi164 conmon[29288]: 28:31.171700+0000 mon.a (mon.0) 735 : audit [INF] from='client.? 172.21.15.85:0/4028168328' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2021-02-10T04:28:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:32 smithi164 conmon[29288]: cluster 2021-02-10T04:28:31.171770 2021-02-10T04:28:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:32 smithi164 conmon[29288]: +0000 mon.a (mon.0) 736 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2021-02-10T04:28:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:32 smithi164 conmon[29288]: 2021-02-10T04:28:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:32 smithi164 conmon[29288]: cluster 2021-02-10T04:28 2021-02-10T04:28:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:32 smithi164 conmon[29288]: :31.382665+0000 osd.3 (osd.3) 35 : 2021-02-10T04:28:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:32 smithi164 conmon[29288]: cluster [DBG] 4.11 scrub ok 2021-02-10T04:28:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:32 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:28:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:32 smithi164 conmon[29288]: 04:28:31.991546+0000 mgr.y (mgr.14140 2021-02-10T04:28:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:32 smithi164 conmon[29288]: ) 329 : cluster [DBG] pgmap v294: 158 pgs: 158 active+clean; 25 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 170 B/s wr, 1 op/s; 7 B/s, 0 objects/s recovering 2021-02-10T04:28:32.738 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:28:33.027 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:28:33.088 DEBUG:teuthology.orchestra.run.smithi085:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0 2021-02-10T04:28:33.089 INFO:tasks.thrashosds.thrasher:Growing pool device_health_metrics 2021-02-10T04:28:33.089 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:28:33.111 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:28:32 smithi085 conmon[49106]: debug 2021-02-10T04:28:32.831+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:33.138 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:28:33.147 INFO:tasks.rados.rados.0.smithi085.stdout:adding op weight read -> 100 2021-02-10T04:28:33.147 INFO:tasks.rados.rados.0.smithi085.stdout:adding op weight write -> 50 2021-02-10T04:28:33.147 INFO:tasks.rados.rados.0.smithi085.stdout:adding op weight delete -> 50 2021-02-10T04:28:33.148 INFO:tasks.rados.rados.0.smithi085.stdout:adding op weight snap_create -> 50 2021-02-10T04:28:33.148 INFO:tasks.rados.rados.0.smithi085.stdout:adding op weight snap_remove -> 50 2021-02-10T04:28:33.148 INFO:tasks.rados.rados.0.smithi085.stdout:adding op weight rollback -> 50 2021-02-10T04:28:33.148 INFO:tasks.rados.rados.0.smithi085.stdout:adding op weight copy_from -> 50 2021-02-10T04:28:33.148 INFO:tasks.rados.rados.0.smithi085.stdout:adding op weight write_excl -> 50 2021-02-10T04:28:33.148 INFO:tasks.rados.rados.0.smithi085.stdout:ceph version 17.0.0-681-gc1ea6241 (c1ea624123d412aff8b9d1430e36cb45fcab76b8) quincy (dev) 2021-02-10T04:28:33.148 INFO:tasks.rados.rados.0.smithi085.stdout:Configuration: 2021-02-10T04:28:33.148 INFO:tasks.rados.rados.0.smithi085.stdout: Number of operations: 4000 2021-02-10T04:28:33.148 INFO:tasks.rados.rados.0.smithi085.stdout: Number of objects: 50 2021-02-10T04:28:33.148 INFO:tasks.rados.rados.0.smithi085.stdout: Max in flight operations: 16 2021-02-10T04:28:33.148 INFO:tasks.rados.rados.0.smithi085.stdout: Object size (in bytes): 4000000 2021-02-10T04:28:33.148 INFO:tasks.rados.rados.0.smithi085.stdout: Write stride min: 400000 2021-02-10T04:28:33.148 INFO:tasks.rados.rados.0.smithi085.stdout: Write stride max: 800000 2021-02-10T04:28:33.169 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:33.245 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:28:33.302 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:28:33.349 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:28:33.373 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:33.428 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:28:33 smithi164 conmon[40892]: debug 2021-02-10T04:28:33.229+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:33.428 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:33 smithi164 conmon[29288]: cluster 2021-02-10T04:28:31.453734+0000 osd.7 (osd.7) 28 : cluster [DBG] 2.5 scrub ok 2021-02-10T04:28:33.428 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:33 smithi164 conmon[29288]: cluster 2021-02-10T04:28:32.490239+0000 osd.5 (osd.5) 28 : cluster [DBG] 4.8 scrub ok 2021-02-10T04:28:33.428 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:33 smithi164 conmon[29288]: cluster 2021-02-10T04:28:32.565367+0000 osd.4 (osd.4) 31 : cluster [DBG] 4.1a scrub ok 2021-02-10T04:28:33.428 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:33 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:28:33.429 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:33 smithi164 conmon[29288]: 28:32.623786+0000 osd.2 (osd.2) 26 2021-02-10T04:28:33.429 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:33 smithi164 conmon[29288]: : cluster [DBG] 6.f scrub ok 2021-02-10T04:28:33.429 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:33 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:28:33.429 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:33 smithi164 conmon[29288]: -10T04:28:33.157128+0000 mon.a ( 2021-02-10T04:28:33.429 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:33 smithi164 conmon[29288]: mon.0) 737 : audit [INF] from='client.? 172.21.15.85:0/2026569549' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2021-02-10T04:28:33.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:33 smithi085 conmon[29319]: cluster 2021-02-10T04:28 2021-02-10T04:28:33.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:33 smithi085 conmon[29319]: :31.453734+0000 osd.7 (osd.7) 28 : cluster [DBG] 2.5 scrub ok 2021-02-10T04:28:33.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:33 smithi085 conmon[29319]: cluster 2021-02-10T04:28:32.490239+0000 osd.5 (osd.5) 28 : cluster [DBG] 4.8 scrub ok 2021-02-10T04:28:33.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:33 smithi085 conmon[29319]: cluster 2021-02-10T04:28:32.565367+0000 osd.4 (osd.4) 31 : cluster [DBG] 4.1a scrub ok 2021-02-10T04:28:33.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:33 smithi085 conmon[29319]: cluster 2021-02-10T04:28:32.623786+0000 osd.2 (osd.2) 26 : cluster [DBG] 6.f scrub ok 2021-02-10T04:28:33.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:33 smithi085 conmon[29319]: audit 2021-02-10T04:28:33.157128+0000 mon.a (mon.0) 737 : audit [INF] from='client.? 172.21.15.85:0/2026569549' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2021-02-10T04:28:33.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:33 smithi085 conmon[35961]: cluster 2021-02-10T04:28:31.453734 2021-02-10T04:28:33.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:33 smithi085 conmon[35961]: +0000 osd.7 (osd.7) 28 : cluster [DBG] 2.5 scrub ok 2021-02-10T04:28:33.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:33 smithi085 conmon[35961]: cluster 2021-02-10T04:28:32.490239+0000 osd.5 (osd.5) 28 : cluster [DBG] 4.8 scrub ok 2021-02-10T04:28:33.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:33 smithi085 conmon[35961]: cluster 2021-02-10T04:28:32.565367+0000 osd.4 (osd.4) 31 : cluster [DBG] 4.1a scrub ok 2021-02-10T04:28:33.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:33 smithi085 conmon[35961]: cluster 2021-02-10T04:28:32.623786+0000 osd.2 (osd.2) 26 : cluster [DBG] 6.f scrub ok 2021-02-10T04:28:33.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:33 smithi085 conmon[35961]: audit 2021-02-10T04:28:33.157128+0000 mon.a (mon.0) 737 : audit [INF] from='client.? 172.21.15.85:0/2026569549' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2021-02-10T04:28:33.453 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:28:33.557 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:28:33.590 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:33.734 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:28:33.734 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:28:33.893 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:28:33 smithi164 conmon[37263]: debug 2021-02-10T04:28:33.427+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:34.024 INFO:tasks.rados.rados.0.smithi085.stdout:1: write initial oid 1 2021-02-10T04:28:34.025 INFO:tasks.rados.rados.0.smithi085.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2021-02-10T04:28:34.029 INFO:tasks.rados.rados.0.smithi085.stdout:1: writing smithi08575397-1 from 416807 to 892056 tid 1 2021-02-10T04:28:34.035 INFO:tasks.rados.rados.0.smithi085.stdout:1: writing smithi08575397-1 from 1542129 to 2016807 tid 2 2021-02-10T04:28:34.035 INFO:tasks.rados.rados.0.smithi085.stdout:2: write initial oid 2 2021-02-10T04:28:34.035 INFO:tasks.rados.rados.0.smithi085.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2021-02-10T04:28:34.040 INFO:tasks.rados.rados.0.smithi085.stdout:2: writing smithi08575397-2 from 416807 to 892056 tid 1 2021-02-10T04:28:34.048 INFO:tasks.rados.rados.0.smithi085.stdout:2: writing smithi08575397-2 from 1542129 to 2016807 tid 2 2021-02-10T04:28:34.048 INFO:tasks.rados.rados.0.smithi085.stdout:1: finishing write tid 1 to smithi08575397-1 2021-02-10T04:28:34.048 INFO:tasks.rados.rados.0.smithi085.stdout:1: finishing write tid 2 to smithi08575397-1 2021-02-10T04:28:34.048 INFO:tasks.rados.rados.0.smithi085.stdout:1: finishing write tid 3 to smithi08575397-1 2021-02-10T04:28:34.048 INFO:tasks.rados.rados.0.smithi085.stdout:1: finishing write tid 4 to smithi08575397-1 2021-02-10T04:28:34.048 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2021-02-10T04:28:34.048 INFO:tasks.rados.rados.0.smithi085.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2021-02-10T04:28:34.048 INFO:tasks.rados.rados.0.smithi085.stdout:2: finishing write tid 1 to smithi08575397-2 2021-02-10T04:28:34.049 INFO:tasks.rados.rados.0.smithi085.stdout:1: done (1 left) 2021-02-10T04:28:34.049 INFO:tasks.rados.rados.0.smithi085.stdout:3: write initial oid 3 2021-02-10T04:28:34.049 INFO:tasks.rados.rados.0.smithi085.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2021-02-10T04:28:34.054 INFO:tasks.rados.rados.0.smithi085.stdout:3: writing smithi08575397-3 from 433614 to 984112 tid 1 2021-02-10T04:28:34.060 INFO:tasks.rados.rados.0.smithi085.stdout:3: writing smithi08575397-3 from 1600611 to 2033614 tid 2 2021-02-10T04:28:34.060 INFO:tasks.rados.rados.0.smithi085.stdout:4: write initial oid 4 2021-02-10T04:28:34.060 INFO:tasks.rados.rados.0.smithi085.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2021-02-10T04:28:34.066 INFO:tasks.rados.rados.0.smithi085.stdout:4: writing smithi08575397-4 from 450421 to 1076168 tid 1 2021-02-10T04:28:34.072 INFO:tasks.rados.rados.0.smithi085.stdout:4: writing smithi08575397-4 from 1659093 to 2050421 tid 2 2021-02-10T04:28:34.072 INFO:tasks.rados.rados.0.smithi085.stdout:5: write initial oid 5 2021-02-10T04:28:34.072 INFO:tasks.rados.rados.0.smithi085.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2021-02-10T04:28:34.079 INFO:tasks.rados.rados.0.smithi085.stdout:5: writing smithi08575397-5 from 467228 to 1168224 tid 1 2021-02-10T04:28:34.084 INFO:tasks.rados.rados.0.smithi085.stdout:5: writing smithi08575397-5 from 1717575 to 2067228 tid 2 2021-02-10T04:28:34.084 INFO:tasks.rados.rados.0.smithi085.stdout:6: write initial oid 6 2021-02-10T04:28:34.084 INFO:tasks.rados.rados.0.smithi085.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2021-02-10T04:28:34.091 INFO:tasks.rados.rados.0.smithi085.stdout:6: writing smithi08575397-6 from 484035 to 1260280 tid 1 2021-02-10T04:28:34.096 INFO:tasks.rados.rados.0.smithi085.stdout:6: writing smithi08575397-6 from 2059704 to 2084035 tid 2 2021-02-10T04:28:34.096 INFO:tasks.rados.rados.0.smithi085.stdout:7: write initial oid 7 2021-02-10T04:28:34.096 INFO:tasks.rados.rados.0.smithi085.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2021-02-10T04:28:34.101 INFO:tasks.rados.rados.0.smithi085.stdout:7: writing smithi08575397-7 from 500842 to 952336 tid 1 2021-02-10T04:28:34.108 INFO:tasks.rados.rados.0.smithi085.stdout:7: writing smithi08575397-7 from 1718186 to 2100842 tid 2 2021-02-10T04:28:34.108 INFO:tasks.rados.rados.0.smithi085.stdout:8: write initial oid 8 2021-02-10T04:28:34.108 INFO:tasks.rados.rados.0.smithi085.stdout:2: finishing write tid 2 to smithi08575397-2 2021-02-10T04:28:34.108 INFO:tasks.rados.rados.0.smithi085.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2021-02-10T04:28:34.114 INFO:tasks.rados.rados.0.smithi085.stdout:8: writing smithi08575397-8 from 517649 to 1044392 tid 1 2021-02-10T04:28:34.120 INFO:tasks.rados.rados.0.smithi085.stdout:8: writing smithi08575397-8 from 1776668 to 2117649 tid 2 2021-02-10T04:28:34.120 INFO:tasks.rados.rados.0.smithi085.stdout:9: write initial oid 9 2021-02-10T04:28:34.120 INFO:tasks.rados.rados.0.smithi085.stdout:2: finishing write tid 3 to smithi08575397-2 2021-02-10T04:28:34.120 INFO:tasks.rados.rados.0.smithi085.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2021-02-10T04:28:34.127 INFO:tasks.rados.rados.0.smithi085.stdout:9: writing smithi08575397-9 from 534456 to 1252801 tid 1 2021-02-10T04:28:34.132 INFO:tasks.rados.rados.0.smithi085.stdout:9: writing smithi08575397-9 from 1951503 to 2134456 tid 2 2021-02-10T04:28:34.132 INFO:tasks.rados.rados.0.smithi085.stdout:10: write initial oid 10 2021-02-10T04:28:34.133 INFO:tasks.rados.rados.0.smithi085.stdout:2: finishing write tid 4 to smithi08575397-2 2021-02-10T04:28:34.133 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2021-02-10T04:28:34.133 INFO:tasks.rados.rados.0.smithi085.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2021-02-10T04:28:34.133 INFO:tasks.rados.rados.0.smithi085.stdout:3: finishing write tid 1 to smithi08575397-3 2021-02-10T04:28:34.133 INFO:tasks.rados.rados.0.smithi085.stdout:3: finishing write tid 2 to smithi08575397-3 2021-02-10T04:28:34.133 INFO:tasks.rados.rados.0.smithi085.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2021-02-10T04:28:34.141 INFO:tasks.rados.rados.0.smithi085.stdout:10: writing smithi08575397-10 from 551263 to 1344857 tid 1 2021-02-10T04:28:34.146 INFO:tasks.rados.rados.0.smithi085.stdout:10: writing smithi08575397-10 from 1893632 to 2151263 tid 2 2021-02-10T04:28:34.146 INFO:tasks.rados.rados.0.smithi085.stdout:2: done (8 left) 2021-02-10T04:28:34.146 INFO:tasks.rados.rados.0.smithi085.stdout:11: write initial oid 11 2021-02-10T04:28:34.146 INFO:tasks.rados.rados.0.smithi085.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2021-02-10T04:28:34.152 INFO:tasks.rados.rados.0.smithi085.stdout:11: writing smithi08575397-11 from 568070 to 1036913 tid 1 2021-02-10T04:28:34.160 INFO:tasks.rados.rados.0.smithi085.stdout:11: writing smithi08575397-11 from 1552114 to 2168070 tid 2 2021-02-10T04:28:34.160 INFO:tasks.rados.rados.0.smithi085.stdout:12: write initial oid 12 2021-02-10T04:28:34.161 INFO:tasks.rados.rados.0.smithi085.stdout:3: finishing write tid 3 to smithi08575397-3 2021-02-10T04:28:34.161 INFO:tasks.rados.rados.0.smithi085.stdout:3: finishing write tid 4 to smithi08575397-3 2021-02-10T04:28:34.161 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2021-02-10T04:28:34.161 INFO:tasks.rados.rados.0.smithi085.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2021-02-10T04:28:34.161 INFO:tasks.rados.rados.0.smithi085.stdout:4: finishing write tid 1 to smithi08575397-4 2021-02-10T04:28:34.161 INFO:tasks.rados.rados.0.smithi085.stdout:4: finishing write tid 2 to smithi08575397-4 2021-02-10T04:28:34.161 INFO:tasks.rados.rados.0.smithi085.stdout:4: finishing write tid 3 to smithi08575397-4 2021-02-10T04:28:34.161 INFO:tasks.rados.rados.0.smithi085.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2021-02-10T04:28:34.167 INFO:tasks.rados.rados.0.smithi085.stdout:12: writing smithi08575397-12 from 584877 to 1128969 tid 1 2021-02-10T04:28:34.174 INFO:tasks.rados.rados.0.smithi085.stdout:12: writing smithi08575397-12 from 1610596 to 2172599 tid 2 2021-02-10T04:28:34.175 INFO:tasks.rados.rados.0.smithi085.stdout:3: done (9 left) 2021-02-10T04:28:34.175 INFO:tasks.rados.rados.0.smithi085.stdout:13: write initial oid 13 2021-02-10T04:28:34.175 INFO:tasks.rados.rados.0.smithi085.stdout:4: finishing write tid 4 to smithi08575397-4 2021-02-10T04:28:34.175 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2021-02-10T04:28:34.175 INFO:tasks.rados.rados.0.smithi085.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2021-02-10T04:28:34.175 INFO:tasks.rados.rados.0.smithi085.stdout:5: finishing write tid 1 to smithi08575397-5 2021-02-10T04:28:34.175 INFO:tasks.rados.rados.0.smithi085.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2021-02-10T04:28:34.182 INFO:tasks.rados.rados.0.smithi085.stdout:13: writing smithi08575397-13 from 601684 to 1221025 tid 1 2021-02-10T04:28:34.189 INFO:tasks.rados.rados.0.smithi085.stdout:13: writing smithi08575397-13 from 1669078 to 2201684 tid 2 2021-02-10T04:28:34.189 INFO:tasks.rados.rados.0.smithi085.stdout:4: done (9 left) 2021-02-10T04:28:34.190 INFO:tasks.rados.rados.0.smithi085.stdout:14: write initial oid 14 2021-02-10T04:28:34.190 INFO:tasks.rados.rados.0.smithi085.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2021-02-10T04:28:34.198 INFO:tasks.rados.rados.0.smithi085.stdout:14: writing smithi08575397-14 from 618491 to 1313081 tid 1 2021-02-10T04:28:34.203 INFO:tasks.rados.rados.0.smithi085.stdout:14: writing smithi08575397-14 from 2011207 to 2218491 tid 2 2021-02-10T04:28:34.204 INFO:tasks.rados.rados.0.smithi085.stdout:15: write initial oid 15 2021-02-10T04:28:34.204 INFO:tasks.rados.rados.0.smithi085.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2021-02-10T04:28:34.213 INFO:tasks.rados.rados.0.smithi085.stdout:15: writing smithi08575397-15 from 635298 to 1405137 tid 1 2021-02-10T04:28:34.217 INFO:tasks.rados.rados.0.smithi085.stdout:15: writing smithi08575397-15 from 2069689 to 2235298 tid 2 2021-02-10T04:28:34.218 INFO:tasks.rados.rados.0.smithi085.stdout:16: write initial oid 16 2021-02-10T04:28:34.218 INFO:tasks.rados.rados.0.smithi085.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2021-02-10T04:28:34.223 INFO:tasks.rados.rados.0.smithi085.stdout:16: writing smithi08575397-16 from 652105 to 1097193 tid 1 2021-02-10T04:28:34.229 INFO:tasks.rados.rados.0.smithi085.stdout:16: writing smithi08575397-16 from 1728171 to 2181159 tid 2 2021-02-10T04:28:34.230 INFO:tasks.rados.rados.0.smithi085.stdout:17: write initial oid 17 2021-02-10T04:28:34.230 INFO:tasks.rados.rados.0.smithi085.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2021-02-10T04:28:34.236 INFO:tasks.rados.rados.0.smithi085.stdout:17: writing smithi08575397-17 from 668912 to 1305602 tid 1 2021-02-10T04:28:34.242 INFO:tasks.rados.rados.0.smithi085.stdout:17: writing smithi08575397-17 from 1903006 to 2268912 tid 2 2021-02-10T04:28:34.242 INFO:tasks.rados.rados.0.smithi085.stdout:18: write initial oid 18 2021-02-10T04:28:34.242 INFO:tasks.rados.rados.0.smithi085.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2021-02-10T04:28:34.249 INFO:tasks.rados.rados.0.smithi085.stdout:18: writing smithi08575397-18 from 685719 to 1397658 tid 1 2021-02-10T04:28:34.256 INFO:tasks.rados.rados.0.smithi085.stdout:18: writing smithi08575397-18 from 1845135 to 2285719 tid 2 2021-02-10T04:28:34.256 INFO:tasks.rados.rados.0.smithi085.stdout:19: write initial oid 19 2021-02-10T04:28:34.256 INFO:tasks.rados.rados.0.smithi085.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2021-02-10T04:28:34.264 INFO:tasks.rados.rados.0.smithi085.stdout:19: writing smithi08575397-19 from 702526 to 1489714 tid 1 2021-02-10T04:28:34.269 INFO:tasks.rados.rados.0.smithi085.stdout:19: writing smithi08575397-19 from 1903617 to 2302526 tid 2 2021-02-10T04:28:34.269 INFO:tasks.rados.rados.0.smithi085.stdout:20: write initial oid 20 2021-02-10T04:28:34.269 INFO:tasks.rados.rados.0.smithi085.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2021-02-10T04:28:34.275 INFO:tasks.rados.rados.0.smithi085.stdout:20: writing smithi08575397-20 from 719333 to 1181770 tid 1 2021-02-10T04:28:34.282 INFO:tasks.rados.rados.0.smithi085.stdout:20: writing smithi08575397-20 from 1962099 to 2319333 tid 2 2021-02-10T04:28:34.282 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:28:34.282 INFO:tasks.rados.rados.0.smithi085.stdout:5: finishing write tid 2 to smithi08575397-5 2021-02-10T04:28:34.282 INFO:tasks.rados.rados.0.smithi085.stdout:5: finishing write tid 3 to smithi08575397-5 2021-02-10T04:28:34.282 INFO:tasks.rados.rados.0.smithi085.stdout:5: finishing write tid 4 to smithi08575397-5 2021-02-10T04:28:34.282 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2021-02-10T04:28:34.282 INFO:tasks.rados.rados.0.smithi085.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2021-02-10T04:28:34.282 INFO:tasks.rados.rados.0.smithi085.stdout:6: finishing write tid 1 to smithi08575397-6 2021-02-10T04:28:34.282 INFO:tasks.rados.rados.0.smithi085.stdout:5: done (15 left) 2021-02-10T04:28:34.282 INFO:tasks.rados.rados.0.smithi085.stdout:21: write initial oid 21 2021-02-10T04:28:34.283 INFO:tasks.rados.rados.0.smithi085.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2021-02-10T04:28:34.288 INFO:tasks.rados.rados.0.smithi085.stdout:21: writing smithi08575397-21 from 736140 to 1273826 tid 1 2021-02-10T04:28:34.295 INFO:tasks.rados.rados.0.smithi085.stdout:21: writing smithi08575397-21 from 2020581 to 2336140 tid 2 2021-02-10T04:28:34.295 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:28:34.295 INFO:tasks.rados.rados.0.smithi085.stdout:6: finishing write tid 2 to smithi08575397-6 2021-02-10T04:28:34.295 INFO:tasks.rados.rados.0.smithi085.stdout:6: finishing write tid 3 to smithi08575397-6 2021-02-10T04:28:34.295 INFO:tasks.rados.rados.0.smithi085.stdout:6: finishing write tid 4 to smithi08575397-6 2021-02-10T04:28:34.295 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2021-02-10T04:28:34.295 INFO:tasks.rados.rados.0.smithi085.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2021-02-10T04:28:34.295 INFO:tasks.rados.rados.0.smithi085.stdout:7: finishing write tid 1 to smithi08575397-7 2021-02-10T04:28:34.295 INFO:tasks.rados.rados.0.smithi085.stdout:6: done (15 left) 2021-02-10T04:28:34.295 INFO:tasks.rados.rados.0.smithi085.stdout:22: write initial oid 22 2021-02-10T04:28:34.296 INFO:tasks.rados.rados.0.smithi085.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2021-02-10T04:28:34.302 INFO:tasks.rados.rados.0.smithi085.stdout:22: writing smithi08575397-22 from 752947 to 1365882 tid 1 2021-02-10T04:28:34.308 INFO:tasks.rados.rados.0.smithi085.stdout:22: writing smithi08575397-22 from 1962710 to 2352947 tid 2 2021-02-10T04:28:34.309 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:28:34.309 INFO:tasks.rados.rados.0.smithi085.stdout:7: finishing write tid 2 to smithi08575397-7 2021-02-10T04:28:34.309 INFO:tasks.rados.rados.0.smithi085.stdout:7: finishing write tid 3 to smithi08575397-7 2021-02-10T04:28:34.309 INFO:tasks.rados.rados.0.smithi085.stdout:7: finishing write tid 4 to smithi08575397-7 2021-02-10T04:28:34.309 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 6 (ObjNum 6 snap 0 seq_num 6) dirty exists 2021-02-10T04:28:34.309 INFO:tasks.rados.rados.0.smithi085.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2021-02-10T04:28:34.309 INFO:tasks.rados.rados.0.smithi085.stdout:8: finishing write tid 1 to smithi08575397-8 2021-02-10T04:28:34.309 INFO:tasks.rados.rados.0.smithi085.stdout:8: finishing write tid 2 to smithi08575397-8 2021-02-10T04:28:34.309 INFO:tasks.rados.rados.0.smithi085.stdout:8: finishing write tid 3 to smithi08575397-8 2021-02-10T04:28:34.309 INFO:tasks.rados.rados.0.smithi085.stdout:8: finishing write tid 4 to smithi08575397-8 2021-02-10T04:28:34.309 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 6 (ObjNum 7 snap 0 seq_num 7) dirty exists 2021-02-10T04:28:34.309 INFO:tasks.rados.rados.0.smithi085.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2021-02-10T04:28:34.309 INFO:tasks.rados.rados.0.smithi085.stdout:9: finishing write tid 1 to smithi08575397-9 2021-02-10T04:28:34.309 INFO:tasks.rados.rados.0.smithi085.stdout:9: finishing write tid 2 to smithi08575397-9 2021-02-10T04:28:34.310 INFO:tasks.rados.rados.0.smithi085.stdout:9: finishing write tid 3 to smithi08575397-9 2021-02-10T04:28:34.310 INFO:tasks.rados.rados.0.smithi085.stdout:9: finishing write tid 4 to smithi08575397-9 2021-02-10T04:28:34.310 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2021-02-10T04:28:34.310 INFO:tasks.rados.rados.0.smithi085.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2021-02-10T04:28:34.310 INFO:tasks.rados.rados.0.smithi085.stdout:10: finishing write tid 1 to smithi08575397-10 2021-02-10T04:28:34.310 INFO:tasks.rados.rados.0.smithi085.stdout:10: finishing write tid 2 to smithi08575397-10 2021-02-10T04:28:34.310 INFO:tasks.rados.rados.0.smithi085.stdout:10: finishing write tid 3 to smithi08575397-10 2021-02-10T04:28:34.310 INFO:tasks.rados.rados.0.smithi085.stdout:7: done (15 left) 2021-02-10T04:28:34.310 INFO:tasks.rados.rados.0.smithi085.stdout:8: done (14 left) 2021-02-10T04:28:34.310 INFO:tasks.rados.rados.0.smithi085.stdout:9: done (13 left) 2021-02-10T04:28:34.310 INFO:tasks.rados.rados.0.smithi085.stdout:23: write initial oid 23 2021-02-10T04:28:34.310 INFO:tasks.rados.rados.0.smithi085.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2021-02-10T04:28:34.317 INFO:tasks.rados.rados.0.smithi085.stdout:23: writing smithi08575397-23 from 769754 to 1457938 tid 1 2021-02-10T04:28:34.321 INFO:tasks.rados.rados.0.smithi085.stdout:23: writing smithi08575397-23 from 2021192 to 2369754 tid 2 2021-02-10T04:28:34.321 INFO:tasks.rados.rados.0.smithi085.stdout:24: write initial oid 24 2021-02-10T04:28:34.321 INFO:tasks.rados.rados.0.smithi085.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2021-02-10T04:28:34.328 INFO:tasks.rados.rados.0.smithi085.stdout:24: writing smithi08575397-24 from 786561 to 1266347 tid 1 2021-02-10T04:28:34.334 INFO:tasks.rados.rados.0.smithi085.stdout:24: writing smithi08575397-24 from 1796027 to 2263691 tid 2 2021-02-10T04:28:34.334 INFO:tasks.rados.rados.0.smithi085.stdout:25: write initial oid 25 2021-02-10T04:28:34.334 INFO:tasks.rados.rados.0.smithi085.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2021-02-10T04:28:34.339 INFO:tasks.rados.rados.0.smithi085.stdout:25: writing smithi08575397-25 from 403368 to 958403 tid 1 2021-02-10T04:28:34.346 INFO:tasks.rados.rados.0.smithi085.stdout:25: writing smithi08575397-25 from 1454509 to 2182184 tid 2 2021-02-10T04:28:34.346 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:28:34.347 INFO:tasks.rados.rados.0.smithi085.stdout:10: finishing write tid 4 to smithi08575397-10 2021-02-10T04:28:34.347 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 10 v 6 (ObjNum 9 snap 0 seq_num 9) dirty exists 2021-02-10T04:28:34.347 INFO:tasks.rados.rados.0.smithi085.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2021-02-10T04:28:34.347 INFO:tasks.rados.rados.0.smithi085.stdout:10: done (15 left) 2021-02-10T04:28:34.347 INFO:tasks.rados.rados.0.smithi085.stdout:26: write initial oid 26 2021-02-10T04:28:34.347 INFO:tasks.rados.rados.0.smithi085.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2021-02-10T04:28:34.352 INFO:tasks.rados.rados.0.smithi085.stdout:26: writing smithi08575397-26 from 420175 to 1050459 tid 1 2021-02-10T04:28:34.360 INFO:tasks.rados.rados.0.smithi085.stdout:26: writing smithi08575397-26 from 1796638 to 2267971 tid 2 2021-02-10T04:28:34.360 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:28:34.360 INFO:tasks.rados.rados.0.smithi085.stdout:11: finishing write tid 1 to smithi08575397-11 2021-02-10T04:28:34.360 INFO:tasks.rados.rados.0.smithi085.stdout:11: finishing write tid 2 to smithi08575397-11 2021-02-10T04:28:34.360 INFO:tasks.rados.rados.0.smithi085.stdout:11: finishing write tid 3 to smithi08575397-11 2021-02-10T04:28:34.360 INFO:tasks.rados.rados.0.smithi085.stdout:11: finishing write tid 4 to smithi08575397-11 2021-02-10T04:28:34.360 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2021-02-10T04:28:34.360 INFO:tasks.rados.rados.0.smithi085.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2021-02-10T04:28:34.360 INFO:tasks.rados.rados.0.smithi085.stdout:12: finishing write tid 1 to smithi08575397-12 2021-02-10T04:28:34.360 INFO:tasks.rados.rados.0.smithi085.stdout:12: finishing write tid 2 to smithi08575397-12 2021-02-10T04:28:34.360 INFO:tasks.rados.rados.0.smithi085.stdout:12: finishing write tid 3 to smithi08575397-12 2021-02-10T04:28:34.360 INFO:tasks.rados.rados.0.smithi085.stdout:12: finishing write tid 4 to smithi08575397-12 2021-02-10T04:28:34.361 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 6 (ObjNum 11 snap 0 seq_num 11) dirty exists 2021-02-10T04:28:34.361 INFO:tasks.rados.rados.0.smithi085.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2021-02-10T04:28:34.361 INFO:tasks.rados.rados.0.smithi085.stdout:13: finishing write tid 1 to smithi08575397-13 2021-02-10T04:28:34.361 INFO:tasks.rados.rados.0.smithi085.stdout:15: finishing write tid 1 to smithi08575397-15 2021-02-10T04:28:34.361 INFO:tasks.rados.rados.0.smithi085.stdout:11: done (15 left) 2021-02-10T04:28:34.361 INFO:tasks.rados.rados.0.smithi085.stdout:12: done (14 left) 2021-02-10T04:28:34.361 INFO:tasks.rados.rados.0.smithi085.stdout:27: write initial oid 27 2021-02-10T04:28:34.361 INFO:tasks.rados.rados.0.smithi085.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2021-02-10T04:28:34.365 INFO:tasks.rados.rados.0.smithi085.stdout:27: writing smithi08575397-27 from 436982 to 1142515 tid 1 2021-02-10T04:28:34.373 INFO:tasks.rados.rados.0.smithi085.stdout:27: writing smithi08575397-27 from 1855120 to 2436982 tid 2 2021-02-10T04:28:34.373 INFO:tasks.rados.rados.0.smithi085.stdout:28: write initial oid 28 2021-02-10T04:28:34.373 INFO:tasks.rados.rados.0.smithi085.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2021-02-10T04:28:34.380 INFO:tasks.rados.rados.0.smithi085.stdout:28: writing smithi08575397-28 from 453789 to 1234571 tid 1 2021-02-10T04:28:34.387 INFO:tasks.rados.rados.0.smithi085.stdout:28: writing smithi08575397-28 from 1913602 to 2388604 tid 2 2021-02-10T04:28:34.387 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:28:34.387 INFO:tasks.rados.rados.0.smithi085.stdout:15: finishing write tid 2 to smithi08575397-15 2021-02-10T04:28:34.387 INFO:tasks.rados.rados.0.smithi085.stdout:15: finishing write tid 3 to smithi08575397-15 2021-02-10T04:28:34.387 INFO:tasks.rados.rados.0.smithi085.stdout:15: finishing write tid 4 to smithi08575397-15 2021-02-10T04:28:34.387 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 15 v 3 (ObjNum 14 snap 0 seq_num 14) dirty exists 2021-02-10T04:28:34.387 INFO:tasks.rados.rados.0.smithi085.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2021-02-10T04:28:34.387 INFO:tasks.rados.rados.0.smithi085.stdout:16: finishing write tid 1 to smithi08575397-16 2021-02-10T04:28:34.387 INFO:tasks.rados.rados.0.smithi085.stdout:15: done (15 left) 2021-02-10T04:28:34.387 INFO:tasks.rados.rados.0.smithi085.stdout:29: write initial oid 29 2021-02-10T04:28:34.387 INFO:tasks.rados.rados.0.smithi085.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2021-02-10T04:28:34.391 INFO:tasks.rados.rados.0.smithi085.stdout:29: writing smithi08575397-29 from 470596 to 926627 tid 1 2021-02-10T04:28:34.399 INFO:tasks.rados.rados.0.smithi085.stdout:29: writing smithi08575397-29 from 1572084 to 2190744 tid 2 2021-02-10T04:28:34.399 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:28:34.399 INFO:tasks.rados.rados.0.smithi085.stdout:16: finishing write tid 2 to smithi08575397-16 2021-02-10T04:28:34.399 INFO:tasks.rados.rados.0.smithi085.stdout:16: finishing write tid 3 to smithi08575397-16 2021-02-10T04:28:34.399 INFO:tasks.rados.rados.0.smithi085.stdout:16: finishing write tid 4 to smithi08575397-16 2021-02-10T04:28:34.399 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2021-02-10T04:28:34.399 INFO:tasks.rados.rados.0.smithi085.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2021-02-10T04:28:34.399 INFO:tasks.rados.rados.0.smithi085.stdout:17: finishing write tid 1 to smithi08575397-17 2021-02-10T04:28:34.400 INFO:tasks.rados.rados.0.smithi085.stdout:16: done (15 left) 2021-02-10T04:28:34.400 INFO:tasks.rados.rados.0.smithi085.stdout:30: write initial oid 30 2021-02-10T04:28:34.400 INFO:tasks.rados.rados.0.smithi085.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2021-02-10T04:28:34.404 INFO:tasks.rados.rados.0.smithi085.stdout:30: writing smithi08575397-30 from 487403 to 1018683 tid 1 2021-02-10T04:28:34.411 INFO:tasks.rados.rados.0.smithi085.stdout:30: writing smithi08575397-30 from 1514213 to 1992884 tid 2 2021-02-10T04:28:34.412 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:28:34.412 INFO:tasks.rados.rados.0.smithi085.stdout:17: finishing write tid 2 to smithi08575397-17 2021-02-10T04:28:34.412 INFO:tasks.rados.rados.0.smithi085.stdout:13: finishing write tid 2 to smithi08575397-13 2021-02-10T04:28:34.412 INFO:tasks.rados.rados.0.smithi085.stdout:13: finishing write tid 3 to smithi08575397-13 2021-02-10T04:28:34.412 INFO:tasks.rados.rados.0.smithi085.stdout:13: finishing write tid 4 to smithi08575397-13 2021-02-10T04:28:34.412 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2021-02-10T04:28:34.412 INFO:tasks.rados.rados.0.smithi085.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2021-02-10T04:28:34.412 INFO:tasks.rados.rados.0.smithi085.stdout:17: finishing write tid 3 to smithi08575397-17 2021-02-10T04:28:34.412 INFO:tasks.rados.rados.0.smithi085.stdout:17: finishing write tid 4 to smithi08575397-17 2021-02-10T04:28:34.413 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 17 v 3 (ObjNum 16 snap 0 seq_num 16) dirty exists 2021-02-10T04:28:34.413 INFO:tasks.rados.rados.0.smithi085.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2021-02-10T04:28:34.413 INFO:tasks.rados.rados.0.smithi085.stdout:18: finishing write tid 1 to smithi08575397-18 2021-02-10T04:28:34.413 INFO:tasks.rados.rados.0.smithi085.stdout:14: finishing write tid 1 to smithi08575397-14 2021-02-10T04:28:34.413 INFO:tasks.rados.rados.0.smithi085.stdout:14: finishing write tid 2 to smithi08575397-14 2021-02-10T04:28:34.413 INFO:tasks.rados.rados.0.smithi085.stdout:14: finishing write tid 3 to smithi08575397-14 2021-02-10T04:28:34.413 INFO:tasks.rados.rados.0.smithi085.stdout:14: finishing write tid 4 to smithi08575397-14 2021-02-10T04:28:34.413 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 9 (ObjNum 13 snap 0 seq_num 13) dirty exists 2021-02-10T04:28:34.414 INFO:tasks.rados.rados.0.smithi085.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2021-02-10T04:28:34.414 INFO:tasks.rados.rados.0.smithi085.stdout:18: finishing write tid 2 to smithi08575397-18 2021-02-10T04:28:34.414 INFO:tasks.rados.rados.0.smithi085.stdout:18: finishing write tid 3 to smithi08575397-18 2021-02-10T04:28:34.414 INFO:tasks.rados.rados.0.smithi085.stdout:18: finishing write tid 4 to smithi08575397-18 2021-02-10T04:28:34.414 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 18 v 3 (ObjNum 17 snap 0 seq_num 17) dirty exists 2021-02-10T04:28:34.414 INFO:tasks.rados.rados.0.smithi085.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2021-02-10T04:28:34.414 INFO:tasks.rados.rados.0.smithi085.stdout:19: finishing write tid 1 to smithi08575397-19 2021-02-10T04:28:34.414 INFO:tasks.rados.rados.0.smithi085.stdout:19: finishing write tid 2 to smithi08575397-19 2021-02-10T04:28:34.414 INFO:tasks.rados.rados.0.smithi085.stdout:19: finishing write tid 3 to smithi08575397-19 2021-02-10T04:28:34.414 INFO:tasks.rados.rados.0.smithi085.stdout:19: finishing write tid 4 to smithi08575397-19 2021-02-10T04:28:34.414 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 19 v 9 (ObjNum 18 snap 0 seq_num 18) dirty exists 2021-02-10T04:28:34.415 INFO:tasks.rados.rados.0.smithi085.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2021-02-10T04:28:34.415 INFO:tasks.rados.rados.0.smithi085.stdout:20: finishing write tid 1 to smithi08575397-20 2021-02-10T04:28:34.415 INFO:tasks.rados.rados.0.smithi085.stdout:20: finishing write tid 2 to smithi08575397-20 2021-02-10T04:28:34.415 INFO:tasks.rados.rados.0.smithi085.stdout:20: finishing write tid 3 to smithi08575397-20 2021-02-10T04:28:34.415 INFO:tasks.rados.rados.0.smithi085.stdout:20: finishing write tid 4 to smithi08575397-20 2021-02-10T04:28:34.415 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 20 v 3 (ObjNum 19 snap 0 seq_num 19) dirty exists 2021-02-10T04:28:34.415 INFO:tasks.rados.rados.0.smithi085.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2021-02-10T04:28:34.415 INFO:tasks.rados.rados.0.smithi085.stdout:21: finishing write tid 1 to smithi08575397-21 2021-02-10T04:28:34.415 INFO:tasks.rados.rados.0.smithi085.stdout:13: done (15 left) 2021-02-10T04:28:34.415 INFO:tasks.rados.rados.0.smithi085.stdout:14: done (14 left) 2021-02-10T04:28:34.415 INFO:tasks.rados.rados.0.smithi085.stdout:17: done (13 left) 2021-02-10T04:28:34.416 INFO:tasks.rados.rados.0.smithi085.stdout:18: done (12 left) 2021-02-10T04:28:34.416 INFO:tasks.rados.rados.0.smithi085.stdout:19: done (11 left) 2021-02-10T04:28:34.416 INFO:tasks.rados.rados.0.smithi085.stdout:20: done (10 left) 2021-02-10T04:28:34.416 INFO:tasks.rados.rados.0.smithi085.stdout:31: write initial oid 31 2021-02-10T04:28:34.417 INFO:tasks.rados.rados.0.smithi085.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2021-02-10T04:28:34.417 INFO:tasks.rados.rados.0.smithi085.stdout:31: writing smithi08575397-31 from 504210 to 1110739 tid 1 2021-02-10T04:28:34.421 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:34 smithi085 conmon[35961]: cluster 2021-02-10T04:28:33.070722+0000 osd.1 (osd.1) 33 : cluster [DBG] 2.15 scrub ok 2021-02-10T04:28:34.421 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:34 smithi085 conmon[35961]: cluster 2021-02-10T04:28:33.321781+0000 osd.3 (osd.3) 36 : cluster [DBG] 5.17 scrub ok 2021-02-10T04:28:34.422 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:34 smithi085 conmon[35961]: cluster 2021-02-10T04:28:33.477055+0000 osd.7 (osd.7) 29 : cluster [DBG] 2.0 scrub ok 2021-02-10T04:28:34.422 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:34 smithi085 conmon[35961]: cluster 2021-02-10T04:28:33.618160+0000 osd.2 (osd.2) 27 : cluster [DBG] 2.c scrub ok 2021-02-10T04:28:34.422 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:34 smithi085 conmon[35961]: cluster 2021-02-10T04:28:33.656504+0000 osd.0 (osd.0) 32 : cluster [DBG] 5.3 scrub ok 2021-02-10T04:28:34.422 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:34 smithi085 conmon[35961]: cluster 2021-02-10T04:28:33.991975+0000 mgr.y (mgr.14140) 330 : cluster [DBG] pgmap v295: 158 pgs: 158 active+clean; 25 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 130 B/s wr, 1 op/s; 5 B/s, 0 objects/s recovering 2021-02-10T04:28:34.422 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:34 smithi085 conmon[35961]: audit 2021-02-10T04:28:34.024107+0000 mon.a (mon.0) 738 : audit [INF] from='client.? 172.21.15.85:0/2026569549' entity='client.0' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2021-02-10T04:28:34.423 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:34 smithi085 conmon[35961]: cluster 2021-02-10T04:28:34.024169+0000 mon.a (mon.0) 739 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2021-02-10T04:28:34.423 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:34 smithi085 conmon[29319]: cluster 2021-02-10T04:28:33.070722+0000 osd.1 (osd.1) 33 : cluster [DBG] 2.15 scrub ok 2021-02-10T04:28:34.423 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:34 smithi085 conmon[29319]: cluster 2021-02-10T04:28:33.321781+0000 osd.3 (osd.3) 36 : cluster [DBG] 5.17 scrub ok 2021-02-10T04:28:34.423 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:34 smithi085 conmon[29319]: cluster 2021-02-10T04:28:33.477055+0000 osd.7 (osd.7) 29 : cluster [DBG] 2.0 scrub ok 2021-02-10T04:28:34.423 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:34 smithi085 conmon[29319]: cluster 2021-02-10T04:28:33.618160+0000 osd.2 (osd.2) 27 : cluster [DBG] 2.c scrub ok 2021-02-10T04:28:34.423 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:34 smithi085 conmon[29319]: cluster 2021-02-10T04:28:33.656504+0000 osd.0 (osd.0) 32 : cluster [DBG] 5.3 scrub ok 2021-02-10T04:28:34.423 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:34 smithi085 conmon[29319]: cluster 2021-02-10T04:28:33.991975+0000 mgr.y (mgr.14140) 330 : cluster [DBG] pgmap v295: 158 pgs: 158 active+clean; 25 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 130 B/s wr, 1 op/s; 5 B/s, 0 objects/s recovering 2021-02-10T04:28:34.423 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:34 smithi085 conmon[29319]: audit 2021-02-10T04:28:34.024107+0000 mon.a (mon.0) 738 : audit [INF] from='client.? 172.21.15.85:0/2026569549' entity='client.0' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2021-02-10T04:28:34.423 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:34 smithi085 conmon[29319]: cluster 2021-02-10T04:28:34.024169+0000 mon.a (mon.0) 739 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2021-02-10T04:28:34.424 INFO:tasks.rados.rados.0.smithi085.stdout:31: writing smithi08575397-31 from 1572695 to 2195024 tid 2 2021-02-10T04:28:34.424 INFO:tasks.rados.rados.0.smithi085.stdout:32: write initial oid 32 2021-02-10T04:28:34.424 INFO:tasks.rados.rados.0.smithi085.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2021-02-10T04:28:34.430 INFO:tasks.rados.rados.0.smithi085.stdout:32: writing smithi08575397-32 from 521017 to 1319148 tid 1 2021-02-10T04:28:34.435 INFO:tasks.rados.rados.0.smithi085.stdout:32: writing smithi08575397-32 from 1747530 to 2229870 tid 2 2021-02-10T04:28:34.436 INFO:tasks.rados.rados.0.smithi085.stdout:33: write initial oid 33 2021-02-10T04:28:34.436 INFO:tasks.rados.rados.0.smithi085.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2021-02-10T04:28:34.444 INFO:tasks.rados.rados.0.smithi085.stdout:33: writing smithi08575397-33 from 537824 to 1011204 tid 1 2021-02-10T04:28:34.449 INFO:tasks.rados.rados.0.smithi085.stdout:33: writing smithi08575397-33 from 1806012 to 2432010 tid 2 2021-02-10T04:28:34.449 INFO:tasks.rados.rados.0.smithi085.stdout:34: write initial oid 34 2021-02-10T04:28:34.449 INFO:tasks.rados.rados.0.smithi085.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2021-02-10T04:28:34.455 INFO:tasks.rados.rados.0.smithi085.stdout:34: writing smithi08575397-34 from 554631 to 1103260 tid 1 2021-02-10T04:28:34.461 INFO:tasks.rados.rados.0.smithi085.stdout:34: writing smithi08575397-34 from 1748141 to 2234150 tid 2 2021-02-10T04:28:34.461 INFO:tasks.rados.rados.0.smithi085.stdout:35: write initial oid 35 2021-02-10T04:28:34.462 INFO:tasks.rados.rados.0.smithi085.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2021-02-10T04:28:34.468 INFO:tasks.rados.rados.0.smithi085.stdout:35: writing smithi08575397-35 from 571438 to 1195316 tid 1 2021-02-10T04:28:34.475 INFO:tasks.rados.rados.0.smithi085.stdout:35: writing smithi08575397-35 from 1806623 to 2436290 tid 2 2021-02-10T04:28:34.475 INFO:tasks.rados.rados.0.smithi085.stdout:36: write initial oid 36 2021-02-10T04:28:34.475 INFO:tasks.rados.rados.0.smithi085.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2021-02-10T04:28:34.482 INFO:tasks.rados.rados.0.smithi085.stdout:36: writing smithi08575397-36 from 588245 to 1287372 tid 1 2021-02-10T04:28:34.488 INFO:tasks.rados.rados.0.smithi085.stdout:36: writing smithi08575397-36 from 1865105 to 2354783 tid 2 2021-02-10T04:28:34.488 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:28:34.488 INFO:tasks.rados.rados.0.smithi085.stdout:21: finishing write tid 2 to smithi08575397-21 2021-02-10T04:28:34.488 INFO:tasks.rados.rados.0.smithi085.stdout:21: finishing write tid 3 to smithi08575397-21 2021-02-10T04:28:34.489 INFO:tasks.rados.rados.0.smithi085.stdout:21: finishing write tid 4 to smithi08575397-21 2021-02-10T04:28:34.489 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 9 (ObjNum 20 snap 0 seq_num 20) dirty exists 2021-02-10T04:28:34.489 INFO:tasks.rados.rados.0.smithi085.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2021-02-10T04:28:34.489 INFO:tasks.rados.rados.0.smithi085.stdout:22: finishing write tid 1 to smithi08575397-22 2021-02-10T04:28:34.489 INFO:tasks.rados.rados.0.smithi085.stdout:21: done (15 left) 2021-02-10T04:28:34.489 INFO:tasks.rados.rados.0.smithi085.stdout:37: write initial oid 37 2021-02-10T04:28:34.489 INFO:tasks.rados.rados.0.smithi085.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2021-02-10T04:28:34.495 INFO:tasks.rados.rados.0.smithi085.stdout:37: writing smithi08575397-37 from 605052 to 1379428 tid 1 2021-02-10T04:28:34.503 INFO:tasks.rados.rados.0.smithi085.stdout:37: writing smithi08575397-37 from 1923587 to 2556923 tid 2 2021-02-10T04:28:34.503 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:28:34.503 INFO:tasks.rados.rados.0.smithi085.stdout:22: finishing write tid 2 to smithi08575397-22 2021-02-10T04:28:34.503 INFO:tasks.rados.rados.0.smithi085.stdout:22: finishing write tid 3 to smithi08575397-22 2021-02-10T04:28:34.503 INFO:tasks.rados.rados.0.smithi085.stdout:22: finishing write tid 4 to smithi08575397-22 2021-02-10T04:28:34.503 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 3 (ObjNum 21 snap 0 seq_num 21) dirty exists 2021-02-10T04:28:34.503 INFO:tasks.rados.rados.0.smithi085.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2021-02-10T04:28:34.503 INFO:tasks.rados.rados.0.smithi085.stdout:23: finishing write tid 1 to smithi08575397-23 2021-02-10T04:28:34.503 INFO:tasks.rados.rados.0.smithi085.stdout:22: done (15 left) 2021-02-10T04:28:34.504 INFO:tasks.rados.rados.0.smithi085.stdout:38: write initial oid 38 2021-02-10T04:28:34.504 INFO:tasks.rados.rados.0.smithi085.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2021-02-10T04:28:34.504 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:28:34.504 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:28:34.504 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:28:34.504 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:28:34.504 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:28:34.504 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:28:34.504 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:28:34.504 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:28:34.504 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:28:34.505 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:28:34.505 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:28:34.505 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:28:34.505 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:28:34.505 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:28:34.505 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:28:34.505 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:28:34.508 INFO:tasks.rados.rados.0.smithi085.stdout:38: writing smithi08575397-38 from 621859 to 1071484 tid 1 2021-02-10T04:28:34.518 INFO:tasks.rados.rados.0.smithi085.stdout:38: writing smithi08575397-38 from 1865716 to 2621859 tid 2 2021-02-10T04:28:34.518 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:28:34.518 INFO:tasks.rados.rados.0.smithi085.stdout:23: finishing write tid 2 to smithi08575397-23 2021-02-10T04:28:34.519 INFO:tasks.rados.rados.0.smithi085.stdout:23: finishing write tid 3 to smithi08575397-23 2021-02-10T04:28:34.519 INFO:tasks.rados.rados.0.smithi085.stdout:23: finishing write tid 4 to smithi08575397-23 2021-02-10T04:28:34.519 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 3 (ObjNum 22 snap 0 seq_num 22) dirty exists 2021-02-10T04:28:34.519 INFO:tasks.rados.rados.0.smithi085.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2021-02-10T04:28:34.519 INFO:tasks.rados.rados.0.smithi085.stdout:24: finishing write tid 1 to smithi08575397-24 2021-02-10T04:28:34.519 INFO:tasks.rados.rados.0.smithi085.stdout:24: finishing write tid 2 to smithi08575397-24 2021-02-10T04:28:34.519 INFO:tasks.rados.rados.0.smithi085.stdout:24: finishing write tid 3 to smithi08575397-24 2021-02-10T04:28:34.519 INFO:tasks.rados.rados.0.smithi085.stdout:24: finishing write tid 4 to smithi08575397-24 2021-02-10T04:28:34.519 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 24 v 3 (ObjNum 23 snap 0 seq_num 23) dirty exists 2021-02-10T04:28:34.519 INFO:tasks.rados.rados.0.smithi085.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2021-02-10T04:28:34.519 INFO:tasks.rados.rados.0.smithi085.stdout:25: finishing write tid 1 to smithi08575397-25 2021-02-10T04:28:34.519 INFO:tasks.rados.rados.0.smithi085.stdout:25: finishing write tid 2 to smithi08575397-25 2021-02-10T04:28:34.519 INFO:tasks.rados.rados.0.smithi085.stdout:25: finishing write tid 3 to smithi08575397-25 2021-02-10T04:28:34.520 INFO:tasks.rados.rados.0.smithi085.stdout:25: finishing write tid 4 to smithi08575397-25 2021-02-10T04:28:34.520 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 25 v 6 (ObjNum 24 snap 0 seq_num 24) dirty exists 2021-02-10T04:28:34.520 INFO:tasks.rados.rados.0.smithi085.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2021-02-10T04:28:34.520 INFO:tasks.rados.rados.0.smithi085.stdout:26: finishing write tid 1 to smithi08575397-26 2021-02-10T04:28:34.520 INFO:tasks.rados.rados.0.smithi085.stdout:26: finishing write tid 2 to smithi08575397-26 2021-02-10T04:28:34.520 INFO:tasks.rados.rados.0.smithi085.stdout:26: finishing write tid 3 to smithi08575397-26 2021-02-10T04:28:34.520 INFO:tasks.rados.rados.0.smithi085.stdout:26: finishing write tid 4 to smithi08575397-26 2021-02-10T04:28:34.520 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2021-02-10T04:28:34.520 INFO:tasks.rados.rados.0.smithi085.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2021-02-10T04:28:34.520 INFO:tasks.rados.rados.0.smithi085.stdout:27: finishing write tid 1 to smithi08575397-27 2021-02-10T04:28:34.520 INFO:tasks.rados.rados.0.smithi085.stdout:27: finishing write tid 2 to smithi08575397-27 2021-02-10T04:28:34.520 INFO:tasks.rados.rados.0.smithi085.stdout:27: finishing write tid 3 to smithi08575397-27 2021-02-10T04:28:34.520 INFO:tasks.rados.rados.0.smithi085.stdout:27: finishing write tid 4 to smithi08575397-27 2021-02-10T04:28:34.520 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 27 v 6 (ObjNum 26 snap 0 seq_num 26) dirty exists 2021-02-10T04:28:34.521 INFO:tasks.rados.rados.0.smithi085.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2021-02-10T04:28:34.521 INFO:tasks.rados.rados.0.smithi085.stdout:28: finishing write tid 1 to smithi08575397-28 2021-02-10T04:28:34.521 INFO:tasks.rados.rados.0.smithi085.stdout:28: finishing write tid 2 to smithi08575397-28 2021-02-10T04:28:34.521 INFO:tasks.rados.rados.0.smithi085.stdout:28: finishing write tid 3 to smithi08575397-28 2021-02-10T04:28:34.521 INFO:tasks.rados.rados.0.smithi085.stdout:28: finishing write tid 4 to smithi08575397-28 2021-02-10T04:28:34.521 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 6 (ObjNum 27 snap 0 seq_num 27) dirty exists 2021-02-10T04:28:34.521 INFO:tasks.rados.rados.0.smithi085.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2021-02-10T04:28:34.521 INFO:tasks.rados.rados.0.smithi085.stdout:29: finishing write tid 1 to smithi08575397-29 2021-02-10T04:28:34.521 INFO:tasks.rados.rados.0.smithi085.stdout:29: finishing write tid 2 to smithi08575397-29 2021-02-10T04:28:34.521 INFO:tasks.rados.rados.0.smithi085.stdout:29: finishing write tid 3 to smithi08575397-29 2021-02-10T04:28:34.521 INFO:tasks.rados.rados.0.smithi085.stdout:29: finishing write tid 4 to smithi08575397-29 2021-02-10T04:28:34.521 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 29 v 6 (ObjNum 28 snap 0 seq_num 28) dirty exists 2021-02-10T04:28:34.521 INFO:tasks.rados.rados.0.smithi085.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2021-02-10T04:28:34.521 INFO:tasks.rados.rados.0.smithi085.stdout:30: finishing write tid 1 to smithi08575397-30 2021-02-10T04:28:34.522 INFO:tasks.rados.rados.0.smithi085.stdout:30: finishing write tid 2 to smithi08575397-30 2021-02-10T04:28:34.522 INFO:tasks.rados.rados.0.smithi085.stdout:30: finishing write tid 3 to smithi08575397-30 2021-02-10T04:28:34.522 INFO:tasks.rados.rados.0.smithi085.stdout:30: finishing write tid 4 to smithi08575397-30 2021-02-10T04:28:34.522 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 6 (ObjNum 29 snap 0 seq_num 29) dirty exists 2021-02-10T04:28:34.522 INFO:tasks.rados.rados.0.smithi085.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2021-02-10T04:28:34.522 INFO:tasks.rados.rados.0.smithi085.stdout:31: finishing write tid 1 to smithi08575397-31 2021-02-10T04:28:34.522 INFO:tasks.rados.rados.0.smithi085.stdout:31: finishing write tid 2 to smithi08575397-31 2021-02-10T04:28:34.522 INFO:tasks.rados.rados.0.smithi085.stdout:23: done (15 left) 2021-02-10T04:28:34.522 INFO:tasks.rados.rados.0.smithi085.stdout:24: done (14 left) 2021-02-10T04:28:34.522 INFO:tasks.rados.rados.0.smithi085.stdout:25: done (13 left) 2021-02-10T04:28:34.522 INFO:tasks.rados.rados.0.smithi085.stdout:26: done (12 left) 2021-02-10T04:28:34.522 INFO:tasks.rados.rados.0.smithi085.stdout:27: done (11 left) 2021-02-10T04:28:34.522 INFO:tasks.rados.rados.0.smithi085.stdout:28: done (10 left) 2021-02-10T04:28:34.523 INFO:tasks.rados.rados.0.smithi085.stdout:29: done (9 left) 2021-02-10T04:28:34.523 INFO:tasks.rados.rados.0.smithi085.stdout:30: done (8 left) 2021-02-10T04:28:34.523 INFO:tasks.rados.rados.0.smithi085.stdout:39: write initial oid 39 2021-02-10T04:28:34.523 INFO:tasks.rados.rados.0.smithi085.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2021-02-10T04:28:34.524 INFO:tasks.rados.rados.0.smithi085.stdout:39: writing smithi08575397-39 from 638666 to 1163540 tid 1 2021-02-10T04:28:34.531 INFO:tasks.rados.rados.0.smithi085.stdout:39: writing smithi08575397-39 from 1924198 to 2561203 tid 2 2021-02-10T04:28:34.532 INFO:tasks.rados.rados.0.smithi085.stdout:40: write initial oid 40 2021-02-10T04:28:34.532 INFO:tasks.rados.rados.0.smithi085.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2021-02-10T04:28:34.539 INFO:tasks.rados.rados.0.smithi085.stdout:40: writing smithi08575397-40 from 655473 to 1371949 tid 1 2021-02-10T04:28:34.547 INFO:tasks.rados.rados.0.smithi085.stdout:40: writing smithi08575397-40 from 2099033 to 2655473 tid 2 2021-02-10T04:28:34.547 INFO:tasks.rados.rados.0.smithi085.stdout:41: write initial oid 41 2021-02-10T04:28:34.547 INFO:tasks.rados.rados.0.smithi085.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2021-02-10T04:28:34.554 INFO:tasks.rados.rados.0.smithi085.stdout:41: writing smithi08575397-41 from 672280 to 1464005 tid 1 2021-02-10T04:28:34.561 INFO:tasks.rados.rados.0.smithi085.stdout:41: writing smithi08575397-41 from 2157515 to 2672280 tid 2 2021-02-10T04:28:34.561 INFO:tasks.rados.rados.0.smithi085.stdout:42: write initial oid 42 2021-02-10T04:28:34.561 INFO:tasks.rados.rados.0.smithi085.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2021-02-10T04:28:34.567 INFO:tasks.rados.rados.0.smithi085.stdout:42: writing smithi08575397-42 from 689087 to 1156061 tid 1 2021-02-10T04:28:34.575 INFO:tasks.rados.rados.0.smithi085.stdout:42: writing smithi08575397-42 from 1699644 to 2483976 tid 2 2021-02-10T04:28:34.575 INFO:tasks.rados.rados.0.smithi085.stdout:43: write initial oid 43 2021-02-10T04:28:34.575 INFO:tasks.rados.rados.0.smithi085.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2021-02-10T04:28:34.582 INFO:tasks.rados.rados.0.smithi085.stdout:43: writing smithi08575397-43 from 705894 to 1248117 tid 1 2021-02-10T04:28:34.588 INFO:tasks.rados.rados.0.smithi085.stdout:43: writing smithi08575397-43 from 1758126 to 2402469 tid 2 2021-02-10T04:28:34.588 INFO:tasks.rados.rados.0.smithi085.stdout:44: write initial oid 44 2021-02-10T04:28:34.588 INFO:tasks.rados.rados.0.smithi085.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2021-02-10T04:28:34.596 INFO:tasks.rados.rados.0.smithi085.stdout:44: writing smithi08575397-44 from 722701 to 1340173 tid 1 2021-02-10T04:28:34.602 INFO:tasks.rados.rados.0.smithi085.stdout:44: writing smithi08575397-44 from 1816608 to 2604609 tid 2 2021-02-10T04:28:34.603 INFO:tasks.rados.rados.0.smithi085.stdout:45: write initial oid 45 2021-02-10T04:28:34.603 INFO:tasks.rados.rados.0.smithi085.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2021-02-10T04:28:34.610 INFO:tasks.rados.rados.0.smithi085.stdout:45: writing smithi08575397-45 from 739508 to 1432229 tid 1 2021-02-10T04:28:34.616 INFO:tasks.rados.rados.0.smithi085.stdout:45: writing smithi08575397-45 from 1875090 to 2523102 tid 2 2021-02-10T04:28:34.617 INFO:tasks.rados.rados.0.smithi085.stdout:46: write initial oid 46 2021-02-10T04:28:34.617 INFO:tasks.rados.rados.0.smithi085.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2021-02-10T04:28:34.624 INFO:tasks.rados.rados.0.smithi085.stdout:46: writing smithi08575397-46 from 756315 to 1524285 tid 1 2021-02-10T04:28:34.632 INFO:tasks.rados.rados.0.smithi085.stdout:46: writing smithi08575397-46 from 1933572 to 2725242 tid 2 2021-02-10T04:28:34.632 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:28:34.632 INFO:tasks.rados.rados.0.smithi085.stdout:31: finishing write tid 3 to smithi08575397-31 2021-02-10T04:28:34.632 INFO:tasks.rados.rados.0.smithi085.stdout:31: finishing write tid 4 to smithi08575397-31 2021-02-10T04:28:34.632 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2021-02-10T04:28:34.632 INFO:tasks.rados.rados.0.smithi085.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2021-02-10T04:28:34.632 INFO:tasks.rados.rados.0.smithi085.stdout:32: finishing write tid 1 to smithi08575397-32 2021-02-10T04:28:34.632 INFO:tasks.rados.rados.0.smithi085.stdout:32: finishing write tid 2 to smithi08575397-32 2021-02-10T04:28:34.632 INFO:tasks.rados.rados.0.smithi085.stdout:32: finishing write tid 3 to smithi08575397-32 2021-02-10T04:28:34.632 INFO:tasks.rados.rados.0.smithi085.stdout:32: finishing write tid 4 to smithi08575397-32 2021-02-10T04:28:34.632 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 32 v 9 (ObjNum 31 snap 0 seq_num 31) dirty exists 2021-02-10T04:28:34.632 INFO:tasks.rados.rados.0.smithi085.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2021-02-10T04:28:34.632 INFO:tasks.rados.rados.0.smithi085.stdout:33: finishing write tid 1 to smithi08575397-33 2021-02-10T04:28:34.633 INFO:tasks.rados.rados.0.smithi085.stdout:33: finishing write tid 2 to smithi08575397-33 2021-02-10T04:28:34.633 INFO:tasks.rados.rados.0.smithi085.stdout:33: finishing write tid 3 to smithi08575397-33 2021-02-10T04:28:34.633 INFO:tasks.rados.rados.0.smithi085.stdout:31: done (15 left) 2021-02-10T04:28:34.633 INFO:tasks.rados.rados.0.smithi085.stdout:32: done (14 left) 2021-02-10T04:28:34.633 INFO:tasks.rados.rados.0.smithi085.stdout:47: write initial oid 47 2021-02-10T04:28:34.633 INFO:tasks.rados.rados.0.smithi085.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2021-02-10T04:28:34.639 INFO:tasks.rados.rados.0.smithi085.stdout:47: writing smithi08575397-47 from 773122 to 1332694 tid 1 2021-02-10T04:28:34.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:34 smithi164 conmon[29288]: cluster 2021-02-10T04:28:33.070722+0000 osd.1 (osd.1) 33 : cluster 2021-02-10T04:28:34.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:34 smithi164 conmon[29288]: [DBG] 2.15 scrub ok 2021-02-10T04:28:34.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:34 smithi164 conmon[29288]: cluster 2021-02-10T04:28:33.321781+0000 osd.3 (osd.3) 36 : cluster [DBG] 5.17 scrub ok 2021-02-10T04:28:34.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:34 smithi164 conmon[29288]: cluster 2021-02-10T04:28:33. 2021-02-10T04:28:34.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:34 smithi164 conmon[29288]: 477055+0000 osd.7 (osd.7) 29 : cluster [DBG] 2.0 scrub ok 2021-02-10T04:28:34.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:34 smithi164 conmon[29288]: cluster 2021-02-10T04:28:34.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:34 smithi164 conmon[29288]: 2021-02-10T04:28:33.618160+0000 osd.2 2021-02-10T04:28:34.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:34 smithi164 conmon[29288]: (osd.2) 27 : cluster [DBG] 2.c scrub ok 2021-02-10T04:28:34.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:34 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:28:34.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:34 smithi164 conmon[29288]: -10T04:28:33.656504+0000 osd.0 (osd.0) 32 2021-02-10T04:28:34.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:34 smithi164 conmon[29288]: : cluster [DBG] 5.3 scrub ok 2021-02-10T04:28:34.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:34 smithi164 conmon[29288]: cluster 2021-02-10T04:28: 2021-02-10T04:28:34.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:34 smithi164 conmon[29288]: 33.991975+0000 mgr.y (mgr.14140) 330 : cluster [DBG] 2021-02-10T04:28:34.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:34 smithi164 conmon[29288]: pgmap v295: 158 pgs: 158 active+clean; 25 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 130 B/s wr, 1 op/s; 5 B/s, 0 objects/s recovering 2021-02-10T04:28:34.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:34 smithi164 conmon[29288]: audit 2021-02-10T04:28 2021-02-10T04:28:34.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:34 smithi164 conmon[29288]: :34.024107+0000 mon.a (mon.0) 2021-02-10T04:28:34.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:34 smithi164 conmon[29288]: 738 : audit [INF] from='client.? 172.21.15.85:0/2026569549' entity='client.0' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2021-02-10T04:28:34.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:34 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:28:34.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:34 smithi164 conmon[29288]: -10T04:28:34.024169+0000 mon.a 2021-02-10T04:28:34.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:34 smithi164 conmon[29288]: (mon.0) 739 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2021-02-10T04:28:34.648 INFO:tasks.rados.rados.0.smithi085.stdout:47: writing smithi08575397-47 from 1992054 to 2643735 tid 2 2021-02-10T04:28:34.649 INFO:tasks.rados.rados.0.smithi085.stdout:48: write initial oid 48 2021-02-10T04:28:34.649 INFO:tasks.rados.rados.0.smithi085.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2021-02-10T04:28:34.652 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:28:34.655 INFO:tasks.rados.rados.0.smithi085.stdout:48: writing smithi08575397-48 from 789929 to 1424750 tid 1 2021-02-10T04:28:34.662 INFO:tasks.rados.rados.0.smithi085.stdout:48: writing smithi08575397-48 from 2050536 to 2789929 tid 2 2021-02-10T04:28:34.663 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:28:34.663 INFO:tasks.rados.rados.0.smithi085.stdout:33: finishing write tid 4 to smithi08575397-33 2021-02-10T04:28:34.663 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 33 v 9 (ObjNum 32 snap 0 seq_num 32) dirty exists 2021-02-10T04:28:34.663 INFO:tasks.rados.rados.0.smithi085.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2021-02-10T04:28:34.663 INFO:tasks.rados.rados.0.smithi085.stdout:33: done (15 left) 2021-02-10T04:28:34.663 INFO:tasks.rados.rados.0.smithi085.stdout:49: write initial oid 49 2021-02-10T04:28:34.663 INFO:tasks.rados.rados.0.smithi085.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2021-02-10T04:28:34.669 INFO:tasks.rados.rados.0.smithi085.stdout:49: writing smithi08575397-49 from 406736 to 1116806 tid 1 2021-02-10T04:28:34.676 INFO:tasks.rados.rados.0.smithi085.stdout:49: writing smithi08575397-49 from 1709018 to 2364368 tid 2 2021-02-10T04:28:34.676 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:28:34.677 INFO:tasks.rados.rados.0.smithi085.stdout:34: finishing write tid 1 to smithi08575397-34 2021-02-10T04:28:34.677 INFO:tasks.rados.rados.0.smithi085.stdout:34: finishing write tid 2 to smithi08575397-34 2021-02-10T04:28:34.677 INFO:tasks.rados.rados.0.smithi085.stdout:34: finishing write tid 3 to smithi08575397-34 2021-02-10T04:28:34.677 INFO:tasks.rados.rados.0.smithi085.stdout:34: finishing write tid 4 to smithi08575397-34 2021-02-10T04:28:34.677 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 34 v 9 (ObjNum 33 snap 0 seq_num 33) dirty exists 2021-02-10T04:28:34.677 INFO:tasks.rados.rados.0.smithi085.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2021-02-10T04:28:34.677 INFO:tasks.rados.rados.0.smithi085.stdout:35: finishing write tid 1 to smithi08575397-35 2021-02-10T04:28:34.677 INFO:tasks.rados.rados.0.smithi085.stdout:34: done (15 left) 2021-02-10T04:28:34.677 INFO:tasks.rados.rados.0.smithi085.stdout:50: write initial oid 50 2021-02-10T04:28:34.677 INFO:tasks.rados.rados.0.smithi085.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2021-02-10T04:28:34.683 INFO:tasks.rados.rados.0.smithi085.stdout:50: writing smithi08575397-50 from 423543 to 1208862 tid 1 2021-02-10T04:28:34.690 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:28:34 smithi085 conmon[41540]: debug 2021-02-10T04:28:34.516+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:34.691 INFO:tasks.rados.rados.0.smithi085.stdout:50: writing smithi08575397-50 from 1767500 to 2566508 tid 2 2021-02-10T04:28:34.692 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:28:34.692 INFO:tasks.rados.rados.0.smithi085.stdout:35: finishing write tid 2 to smithi08575397-35 2021-02-10T04:28:34.692 INFO:tasks.rados.rados.0.smithi085.stdout:35: finishing write tid 3 to smithi08575397-35 2021-02-10T04:28:34.692 INFO:tasks.rados.rados.0.smithi085.stdout:35: finishing write tid 4 to smithi08575397-35 2021-02-10T04:28:34.692 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 35 v 9 (ObjNum 34 snap 0 seq_num 34) dirty exists 2021-02-10T04:28:34.692 INFO:tasks.rados.rados.0.smithi085.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2021-02-10T04:28:34.692 INFO:tasks.rados.rados.0.smithi085.stdout:36: finishing write tid 1 to smithi08575397-36 2021-02-10T04:28:34.692 INFO:tasks.rados.rados.0.smithi085.stdout:36: finishing write tid 2 to smithi08575397-36 2021-02-10T04:28:34.692 INFO:tasks.rados.rados.0.smithi085.stdout:35: done (15 left) 2021-02-10T04:28:34.692 INFO:tasks.rados.rados.0.smithi085.stdout:51: copy_from oid 2 from oid 11 current snap is 0 2021-02-10T04:28:34.692 INFO:tasks.rados.rados.0.smithi085.stdout:36: finishing write tid 3 to smithi08575397-36 2021-02-10T04:28:34.692 INFO:tasks.rados.rados.0.smithi085.stdout:36: finishing write tid 4 to smithi08575397-36 2021-02-10T04:28:34.693 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 6 (ObjNum 35 snap 0 seq_num 35) dirty exists 2021-02-10T04:28:34.693 INFO:tasks.rados.rados.0.smithi085.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2021-02-10T04:28:34.693 INFO:tasks.rados.rados.0.smithi085.stdout:37: finishing write tid 1 to smithi08575397-37 2021-02-10T04:28:34.693 INFO:tasks.rados.rados.0.smithi085.stdout:37: finishing write tid 2 to smithi08575397-37 2021-02-10T04:28:34.693 INFO:tasks.rados.rados.0.smithi085.stdout:37: finishing write tid 3 to smithi08575397-37 2021-02-10T04:28:34.693 INFO:tasks.rados.rados.0.smithi085.stdout:37: finishing write tid 4 to smithi08575397-37 2021-02-10T04:28:34.693 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 12 (ObjNum 36 snap 0 seq_num 36) dirty exists 2021-02-10T04:28:34.693 INFO:tasks.rados.rados.0.smithi085.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2021-02-10T04:28:34.693 INFO:tasks.rados.rados.0.smithi085.stdout:38: finishing write tid 1 to smithi08575397-38 2021-02-10T04:28:34.693 INFO:tasks.rados.rados.0.smithi085.stdout:38: finishing write tid 2 to smithi08575397-38 2021-02-10T04:28:34.693 INFO:tasks.rados.rados.0.smithi085.stdout:38: finishing write tid 3 to smithi08575397-38 2021-02-10T04:28:34.693 INFO:tasks.rados.rados.0.smithi085.stdout:38: finishing write tid 4 to smithi08575397-38 2021-02-10T04:28:34.693 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 9 (ObjNum 37 snap 0 seq_num 37) dirty exists 2021-02-10T04:28:34.693 INFO:tasks.rados.rados.0.smithi085.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2021-02-10T04:28:34.694 INFO:tasks.rados.rados.0.smithi085.stdout:39: finishing write tid 1 to smithi08575397-39 2021-02-10T04:28:34.694 INFO:tasks.rados.rados.0.smithi085.stdout:36: done (15 left) 2021-02-10T04:28:34.694 INFO:tasks.rados.rados.0.smithi085.stdout:37: done (14 left) 2021-02-10T04:28:34.694 INFO:tasks.rados.rados.0.smithi085.stdout:38: done (13 left) 2021-02-10T04:28:34.694 INFO:tasks.rados.rados.0.smithi085.stdout:52: write oid 15 current snap is 0 2021-02-10T04:28:34.694 INFO:tasks.rados.rados.0.smithi085.stdout:52: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2021-02-10T04:28:34.696 INFO:tasks.rados.rados.0.smithi085.stdout:52: writing smithi08575397-15 from 440350 to 900918 tid 1 2021-02-10T04:28:34.702 INFO:tasks.rados.rados.0.smithi085.stdout:52: writing smithi08575397-15 from 1309629 to 1852295 tid 2 2021-02-10T04:28:34.708 INFO:tasks.rados.rados.0.smithi085.stdout:52: writing smithi08575397-15 from 2323973 to 2840350 tid 3 2021-02-10T04:28:34.709 INFO:tasks.rados.rados.0.smithi085.stdout:53: rollback oid 6 current snap is 0 2021-02-10T04:28:34.709 INFO:tasks.rados.rados.0.smithi085.stdout:39: finishing write tid 2 to smithi08575397-39 2021-02-10T04:28:34.709 INFO:tasks.rados.rados.0.smithi085.stdout:39: finishing write tid 3 to smithi08575397-39 2021-02-10T04:28:34.709 INFO:tasks.rados.rados.0.smithi085.stdout:39: finishing write tid 4 to smithi08575397-39 2021-02-10T04:28:34.709 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 39 v 3 (ObjNum 38 snap 0 seq_num 38) dirty exists 2021-02-10T04:28:34.709 INFO:tasks.rados.rados.0.smithi085.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2021-02-10T04:28:34.709 INFO:tasks.rados.rados.0.smithi085.stdout:40: finishing write tid 1 to smithi08575397-40 2021-02-10T04:28:34.709 INFO:tasks.rados.rados.0.smithi085.stdout:40: finishing write tid 2 to smithi08575397-40 2021-02-10T04:28:34.709 INFO:tasks.rados.rados.0.smithi085.stdout:40: finishing write tid 3 to smithi08575397-40 2021-02-10T04:28:34.709 INFO:tasks.rados.rados.0.smithi085.stdout:40: finishing write tid 4 to smithi08575397-40 2021-02-10T04:28:34.709 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 15 (ObjNum 39 snap 0 seq_num 39) dirty exists 2021-02-10T04:28:34.709 INFO:tasks.rados.rados.0.smithi085.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2021-02-10T04:28:34.709 INFO:tasks.rados.rados.0.smithi085.stdout:41: finishing write tid 1 to smithi08575397-41 2021-02-10T04:28:34.710 INFO:tasks.rados.rados.0.smithi085.stdout:41: finishing write tid 2 to smithi08575397-41 2021-02-10T04:28:34.710 INFO:tasks.rados.rados.0.smithi085.stdout:41: finishing write tid 3 to smithi08575397-41 2021-02-10T04:28:34.710 INFO:tasks.rados.rados.0.smithi085.stdout:41: finishing write tid 4 to smithi08575397-41 2021-02-10T04:28:34.710 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 41 v 18 (ObjNum 40 snap 0 seq_num 40) dirty exists 2021-02-10T04:28:34.710 INFO:tasks.rados.rados.0.smithi085.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2021-02-10T04:28:34.710 INFO:tasks.rados.rados.0.smithi085.stdout:42: finishing write tid 1 to smithi08575397-42 2021-02-10T04:28:34.710 INFO:tasks.rados.rados.0.smithi085.stdout:42: finishing write tid 2 to smithi08575397-42 2021-02-10T04:28:34.710 INFO:tasks.rados.rados.0.smithi085.stdout:42: finishing write tid 3 to smithi08575397-42 2021-02-10T04:28:34.710 INFO:tasks.rados.rados.0.smithi085.stdout:39: done (14 left) 2021-02-10T04:28:34.710 INFO:tasks.rados.rados.0.smithi085.stdout:40: done (13 left) 2021-02-10T04:28:34.710 INFO:tasks.rados.rados.0.smithi085.stdout:41: done (12 left) 2021-02-10T04:28:34.710 INFO:tasks.rados.rados.0.smithi085.stdout:53: done (11 left) 2021-02-10T04:28:34.710 INFO:tasks.rados.rados.0.smithi085.stdout:54: write (excl) oid 31 current snap is 0 2021-02-10T04:28:34.711 INFO:tasks.rados.rados.0.smithi085.stdout:42: finishing write tid 4 to smithi08575397-42 2021-02-10T04:28:34.711 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 12 (ObjNum 41 snap 0 seq_num 41) dirty exists 2021-02-10T04:28:34.711 INFO:tasks.rados.rados.0.smithi085.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2021-02-10T04:28:34.711 INFO:tasks.rados.rados.0.smithi085.stdout:43: finishing write tid 1 to smithi08575397-43 2021-02-10T04:28:34.711 INFO:tasks.rados.rados.0.smithi085.stdout:43: finishing write tid 2 to smithi08575397-43 2021-02-10T04:28:34.711 INFO:tasks.rados.rados.0.smithi085.stdout:43: finishing write tid 3 to smithi08575397-43 2021-02-10T04:28:34.711 INFO:tasks.rados.rados.0.smithi085.stdout:43: finishing write tid 4 to smithi08575397-43 2021-02-10T04:28:34.711 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 9 (ObjNum 42 snap 0 seq_num 42) dirty exists 2021-02-10T04:28:34.711 INFO:tasks.rados.rados.0.smithi085.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2021-02-10T04:28:34.711 INFO:tasks.rados.rados.0.smithi085.stdout:44: finishing write tid 1 to smithi08575397-44 2021-02-10T04:28:34.711 INFO:tasks.rados.rados.0.smithi085.stdout:44: finishing write tid 2 to smithi08575397-44 2021-02-10T04:28:34.711 INFO:tasks.rados.rados.0.smithi085.stdout:54: seq_num 51 ranges {457157=535817,1768111=402677} 2021-02-10T04:28:34.713 INFO:tasks.rados.rados.0.smithi085.stdout:54: writing smithi08575397-31 from 457157 to 992974 tid 1 2021-02-10T04:28:34.722 INFO:tasks.rados.rados.0.smithi085.stdout:54: writing smithi08575397-31 from 1768111 to 2170788 tid 2 2021-02-10T04:28:34.722 INFO:tasks.rados.rados.0.smithi085.stdout:42: done (11 left) 2021-02-10T04:28:34.722 INFO:tasks.rados.rados.0.smithi085.stdout:43: done (10 left) 2021-02-10T04:28:34.722 INFO:tasks.rados.rados.0.smithi085.stdout:55: snap_create 2021-02-10T04:28:34.722 INFO:tasks.rados.rados.0.smithi085.stdout:44: finishing write tid 3 to smithi08575397-44 2021-02-10T04:28:34.722 INFO:tasks.rados.rados.0.smithi085.stdout:44: finishing write tid 4 to smithi08575397-44 2021-02-10T04:28:34.722 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 6 (ObjNum 43 snap 0 seq_num 43) dirty exists 2021-02-10T04:28:34.722 INFO:tasks.rados.rados.0.smithi085.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2021-02-10T04:28:34.722 INFO:tasks.rados.rados.0.smithi085.stdout:45: finishing write tid 1 to smithi08575397-45 2021-02-10T04:28:34.722 INFO:tasks.rados.rados.0.smithi085.stdout:45: finishing write tid 2 to smithi08575397-45 2021-02-10T04:28:34.722 INFO:tasks.rados.rados.0.smithi085.stdout:45: finishing write tid 3 to smithi08575397-45 2021-02-10T04:28:34.723 INFO:tasks.rados.rados.0.smithi085.stdout:45: finishing write tid 4 to smithi08575397-45 2021-02-10T04:28:34.723 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 45 v 9 (ObjNum 44 snap 0 seq_num 44) dirty exists 2021-02-10T04:28:34.723 INFO:tasks.rados.rados.0.smithi085.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2021-02-10T04:28:34.723 INFO:tasks.rados.rados.0.smithi085.stdout:46: finishing write tid 1 to smithi08575397-46 2021-02-10T04:28:34.723 INFO:tasks.rados.rados.0.smithi085.stdout:46: finishing write tid 2 to smithi08575397-46 2021-02-10T04:28:34.723 INFO:tasks.rados.rados.0.smithi085.stdout:46: finishing write tid 3 to smithi08575397-46 2021-02-10T04:28:34.723 INFO:tasks.rados.rados.0.smithi085.stdout:46: finishing write tid 4 to smithi08575397-46 2021-02-10T04:28:34.723 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 12 (ObjNum 45 snap 0 seq_num 45) dirty exists 2021-02-10T04:28:34.723 INFO:tasks.rados.rados.0.smithi085.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2021-02-10T04:28:34.723 INFO:tasks.rados.rados.0.smithi085.stdout:47: finishing write tid 1 to smithi08575397-47 2021-02-10T04:28:34.723 INFO:tasks.rados.rados.0.smithi085.stdout:47: finishing write tid 2 to smithi08575397-47 2021-02-10T04:28:34.723 INFO:tasks.rados.rados.0.smithi085.stdout:47: finishing write tid 3 to smithi08575397-47 2021-02-10T04:28:34.723 INFO:tasks.rados.rados.0.smithi085.stdout:47: finishing write tid 4 to smithi08575397-47 2021-02-10T04:28:34.723 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 47 v 21 (ObjNum 46 snap 0 seq_num 46) dirty exists 2021-02-10T04:28:34.724 INFO:tasks.rados.rados.0.smithi085.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2021-02-10T04:28:34.724 INFO:tasks.rados.rados.0.smithi085.stdout:48: finishing write tid 1 to smithi08575397-48 2021-02-10T04:28:34.724 INFO:tasks.rados.rados.0.smithi085.stdout:48: finishing write tid 2 to smithi08575397-48 2021-02-10T04:28:34.724 INFO:tasks.rados.rados.0.smithi085.stdout:48: finishing write tid 3 to smithi08575397-48 2021-02-10T04:28:34.724 INFO:tasks.rados.rados.0.smithi085.stdout:48: finishing write tid 4 to smithi08575397-48 2021-02-10T04:28:34.724 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 15 (ObjNum 47 snap 0 seq_num 47) dirty exists 2021-02-10T04:28:34.724 INFO:tasks.rados.rados.0.smithi085.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2021-02-10T04:28:34.724 INFO:tasks.rados.rados.0.smithi085.stdout:49: finishing write tid 1 to smithi08575397-49 2021-02-10T04:28:34.724 INFO:tasks.rados.rados.0.smithi085.stdout:49: finishing write tid 2 to smithi08575397-49 2021-02-10T04:28:34.724 INFO:tasks.rados.rados.0.smithi085.stdout:49: finishing write tid 3 to smithi08575397-49 2021-02-10T04:28:34.724 INFO:tasks.rados.rados.0.smithi085.stdout:49: finishing write tid 4 to smithi08575397-49 2021-02-10T04:28:34.724 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 49 v 12 (ObjNum 48 snap 0 seq_num 48) dirty exists 2021-02-10T04:28:34.724 INFO:tasks.rados.rados.0.smithi085.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2021-02-10T04:28:34.725 INFO:tasks.rados.rados.0.smithi085.stdout:50: finishing write tid 1 to smithi08575397-50 2021-02-10T04:28:34.725 INFO:tasks.rados.rados.0.smithi085.stdout:50: finishing write tid 2 to smithi08575397-50 2021-02-10T04:28:34.725 INFO:tasks.rados.rados.0.smithi085.stdout:50: finishing write tid 3 to smithi08575397-50 2021-02-10T04:28:34.725 INFO:tasks.rados.rados.0.smithi085.stdout:50: finishing write tid 4 to smithi08575397-50 2021-02-10T04:28:34.725 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 12 (ObjNum 49 snap 0 seq_num 49) dirty exists 2021-02-10T04:28:34.725 INFO:tasks.rados.rados.0.smithi085.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2021-02-10T04:28:34.725 INFO:tasks.rados.rados.0.smithi085.stdout:52: finishing write tid 1 to smithi08575397-15 2021-02-10T04:28:34.725 INFO:tasks.rados.rados.0.smithi085.stdout:51: finishing copy_from to smithi08575397-2 2021-02-10T04:28:34.725 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2021-02-10T04:28:34.725 INFO:tasks.rados.rados.0.smithi085.stdout:51: finishing copy_from racing read to smithi08575397-2 2021-02-10T04:28:34.725 INFO:tasks.rados.rados.0.smithi085.stdout:52: finishing write tid 2 to smithi08575397-15 2021-02-10T04:28:34.725 INFO:tasks.rados.rados.0.smithi085.stdout:52: finishing write tid 3 to smithi08575397-15 2021-02-10T04:28:34.725 INFO:tasks.rados.rados.0.smithi085.stdout:52: finishing write tid 4 to smithi08575397-15 2021-02-10T04:28:34.725 INFO:tasks.rados.rados.0.smithi085.stdout:52: finishing write tid 5 to smithi08575397-15 2021-02-10T04:28:34.726 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 15 v 13 (ObjNum 50 snap 0 seq_num 50) dirty exists 2021-02-10T04:28:34.726 INFO:tasks.rados.rados.0.smithi085.stdout:52: left oid 15 (ObjNum 50 snap 0 seq_num 50) 2021-02-10T04:28:34.726 INFO:tasks.rados.rados.0.smithi085.stdout:54: finishing write tid 1 to smithi08575397-31 2021-02-10T04:28:34.726 INFO:tasks.rados.rados.0.smithi085.stdout:54: finishing write tid 2 to smithi08575397-31 2021-02-10T04:28:34.726 INFO:tasks.rados.rados.0.smithi085.stdout:54: finishing write tid 3 to smithi08575397-31 2021-02-10T04:28:34.727 INFO:tasks.rados.rados.0.smithi085.stdout:54: finishing write tid 4 to smithi08575397-31 2021-02-10T04:28:34.727 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 12 (ObjNum 51 snap 0 seq_num 51) dirty exists 2021-02-10T04:28:34.727 INFO:tasks.rados.rados.0.smithi085.stdout:54: left oid 31 (ObjNum 51 snap 0 seq_num 51) 2021-02-10T04:28:34.763 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:28:34.788 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:35.032 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.1 dump_historic_ops 2021-02-10T04:28:35.192 INFO:tasks.rados.rados.0.smithi085.stdout:44: done (10 left) 2021-02-10T04:28:35.192 INFO:tasks.rados.rados.0.smithi085.stdout:45: done (9 left) 2021-02-10T04:28:35.192 INFO:tasks.rados.rados.0.smithi085.stdout:46: done (8 left) 2021-02-10T04:28:35.192 INFO:tasks.rados.rados.0.smithi085.stdout:47: done (7 left) 2021-02-10T04:28:35.192 INFO:tasks.rados.rados.0.smithi085.stdout:48: done (6 left) 2021-02-10T04:28:35.192 INFO:tasks.rados.rados.0.smithi085.stdout:49: done (5 left) 2021-02-10T04:28:35.192 INFO:tasks.rados.rados.0.smithi085.stdout:50: done (4 left) 2021-02-10T04:28:35.193 INFO:tasks.rados.rados.0.smithi085.stdout:51: done (3 left) 2021-02-10T04:28:35.193 INFO:tasks.rados.rados.0.smithi085.stdout:52: done (2 left) 2021-02-10T04:28:35.193 INFO:tasks.rados.rados.0.smithi085.stdout:54: done (1 left) 2021-02-10T04:28:35.193 INFO:tasks.rados.rados.0.smithi085.stdout:55: done (0 left) 2021-02-10T04:28:35.193 INFO:tasks.rados.rados.0.smithi085.stdout:56: delete oid 34 current snap is 1 2021-02-10T04:28:35.196 INFO:tasks.rados.rados.0.smithi085.stdout:56: done (0 left) 2021-02-10T04:28:35.196 INFO:tasks.rados.rados.0.smithi085.stdout:57: snap_create 2021-02-10T04:28:35.283 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:28:35.441 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:28:35 smithi085 conmon[52836]: debug 2021-02-10T04:28:35.305+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:35.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:35 smithi085 conmon[29319]: cluster 2021-02-10T04:28:34.053320+0000 2021-02-10T04:28:35.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:35 smithi085 conmon[29319]: osd.1 (osd.1) 34 : cluster [DBG] 2.4 scrub ok 2021-02-10T04:28:35.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:35 smithi085 conmon[29319]: audit 2021-02-10T04:28:34.189706+0000 mon.c (mon.1) 36 : audit [DBG] from='client.? 172.21.15.85:0/2559822517' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:28:35.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:35 smithi085 conmon[29319]: cluster 2021-02-10T04:28:34.444334+0000 osd.7 (osd.7) 30 : cluster [DBG] 2.8 scrub ok 2021-02-10T04:28:35.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:35 smithi085 conmon[29319]: cluster 2021-02-10T04:28:34.482900+0000 osd.5 (osd.5) 29 : cluster [DBG] 2.3 scrub ok 2021-02-10T04:28:35.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:35 smithi085 conmon[29319]: cluster 2021-02-10T04:28:34.694477+0000 osd.0 (osd.0) 33 : cluster [DBG] 4.6 scrub ok 2021-02-10T04:28:35.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:35 smithi085 conmon[35961]: cluster 2021-02-10T04:28:34.053320+0000 osd.1 (osd.1) 34 2021-02-10T04:28:35.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:35 smithi085 conmon[35961]: : cluster [DBG] 2.4 scrub ok 2021-02-10T04:28:35.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:35 smithi085 conmon[35961]: audit 2021-02-10T04:28:34.189706+0000 mon.c (mon.1) 36 : audit 2021-02-10T04:28:35.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:35 smithi085 conmon[35961]: [DBG] from='client.? 172.21.15.85:0/2559822517' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:28:35.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:35 smithi085 conmon[35961]: cluster 2021-02-10T04:28:34.444334+0000 osd.7 (osd.7) 30 2021-02-10T04:28:35.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:35 smithi085 conmon[35961]: : cluster [DBG] 2.8 scrub ok 2021-02-10T04:28:35.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:35 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:28:35.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:35 smithi085 conmon[35961]: 28:34.482900+0000 osd.5 (osd.5) 29 : cluster [DBG] 2.3 scrub ok 2021-02-10T04:28:35.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:35 smithi085 conmon[35961]: cluster 2021-02-10T04:28:34.694477 2021-02-10T04:28:35.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:35 smithi085 conmon[35961]: +0000 osd.0 (osd.0) 33 : cluster [DBG] 4.6 scrub ok 2021-02-10T04:28:35.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:35 smithi164 conmon[29288]: cluster 2021-02-10T04:28:34.053320+0000 osd.1 (osd.1) 34 : cluster [DBG] 2.4 scrub ok 2021-02-10T04:28:35.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:35 smithi164 conmon[29288]: 2021-02-10T04:28:35.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:35 smithi164 conmon[29288]: audit 2021-02-10T04:28:34.189706+0000 mon.c (mon.1) 36 : audit [DBG] from='client.? 172.21.15.85:0/2559822517' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:28:35.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:35 smithi164 conmon[29288]: cluster 2021-02-10T04:28:34. 2021-02-10T04:28:35.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:35 smithi164 conmon[29288]: 444334+0000 osd.7 (osd.7) 30 : cluster [DBG] 2.8 scrub ok 2021-02-10T04:28:35.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:35 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:28:35.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:35 smithi164 conmon[29288]: T04:28:34.482900+0000 osd.5 (osd.5) 29 : cluster 2021-02-10T04:28:35.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:35 smithi164 conmon[29288]: [DBG] 2.3 scrub ok 2021-02-10T04:28:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:35 smithi164 conmon[29288]: cluster 2021-02-10T04:28:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:35 smithi164 conmon[29288]: 2021-02-10T04:28:34.694477+0000 osd.0 (osd 2021-02-10T04:28:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:35 smithi164 conmon[29288]: .0) 33 : cluster [DBG] 4.6 scrub ok 2021-02-10T04:28:36.198 INFO:tasks.rados.rados.0.smithi085.stdout:57: done (0 left) 2021-02-10T04:28:36.199 INFO:tasks.rados.rados.0.smithi085.stdout:58: write oid 33 current snap is 2 2021-02-10T04:28:36.199 INFO:tasks.rados.rados.0.smithi085.stdout:58: seq_num 53 ranges {490771=686315,1885075=406346} 2021-02-10T04:28:36.205 INFO:tasks.rados.rados.0.smithi085.stdout:58: writing smithi08575397-33 from 490771 to 1177086 tid 1 2021-02-10T04:28:36.212 INFO:tasks.rados.rados.0.smithi085.stdout:58: writing smithi08575397-33 from 1885075 to 2291421 tid 2 2021-02-10T04:28:36.212 INFO:tasks.rados.rados.0.smithi085.stdout:59: delete oid 3 current snap is 2 2021-02-10T04:28:36.213 INFO:tasks.rados.rados.0.smithi085.stdout:58: finishing write tid 1 to smithi08575397-33 2021-02-10T04:28:36.214 INFO:tasks.rados.rados.0.smithi085.stdout:59: done (1 left) 2021-02-10T04:28:36.214 INFO:tasks.rados.rados.0.smithi085.stdout:60: write oid 12 current snap is 2 2021-02-10T04:28:36.214 INFO:tasks.rados.rados.0.smithi085.stdout:60: seq_num 55 ranges {524385=553166,1602039=410015,2577441=346944} 2021-02-10T04:28:36.220 INFO:tasks.rados.rados.0.smithi085.stdout:60: writing smithi08575397-12 from 524385 to 1077551 tid 1 2021-02-10T04:28:36.225 INFO:tasks.rados.rados.0.smithi085.stdout:60: writing smithi08575397-12 from 1602039 to 2012054 tid 2 2021-02-10T04:28:36.231 INFO:tasks.rados.rados.0.smithi085.stdout:60: writing smithi08575397-12 from 2577441 to 2924385 tid 3 2021-02-10T04:28:36.231 INFO:tasks.rados.rados.0.smithi085.stdout:61: rollback oid 47 current snap is 2 2021-02-10T04:28:36.231 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 47 to 1 2021-02-10T04:28:36.231 INFO:tasks.rados.rados.0.smithi085.stdout:58: finishing write tid 2 to smithi08575397-33 2021-02-10T04:28:36.231 INFO:tasks.rados.rados.0.smithi085.stdout:58: finishing write tid 3 to smithi08575397-33 2021-02-10T04:28:36.231 INFO:tasks.rados.rados.0.smithi085.stdout:58: finishing write tid 4 to smithi08575397-33 2021-02-10T04:28:36.231 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 33 v 13 (ObjNum 53 snap 2 seq_num 53) dirty exists 2021-02-10T04:28:36.231 INFO:tasks.rados.rados.0.smithi085.stdout:58: left oid 33 (ObjNum 53 snap 2 seq_num 53) 2021-02-10T04:28:36.231 INFO:tasks.rados.rados.0.smithi085.stdout:60: finishing write tid 1 to smithi08575397-12 2021-02-10T04:28:36.232 INFO:tasks.rados.rados.0.smithi085.stdout:60: finishing write tid 2 to smithi08575397-12 2021-02-10T04:28:36.232 INFO:tasks.rados.rados.0.smithi085.stdout:58: done (2 left) 2021-02-10T04:28:36.232 INFO:tasks.rados.rados.0.smithi085.stdout:62: snap_remove snap 0 2021-02-10T04:28:36.343 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:28:36.448 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:28:36.474 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:36 smithi164 conmon[29288]: cluster 2021-02-10T04:28:35.010703+0000 osd.1 (osd.1 2021-02-10T04:28:36.474 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:36 smithi164 conmon[29288]: ) 35 : cluster [DBG] 2.6 scrub ok 2021-02-10T04:28:36.475 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:36 smithi164 conmon[29288]: cluster 2021-02-10T04:28:35.190563+0000 mon.a (mon.0) 740 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2021-02-10T04:28:36.475 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:36 smithi164 conmon[29288]: cluster 2021-02-10T04:28:36.475 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:36 smithi164 conmon[29288]: 2021-02-10T04:28: 2021-02-10T04:28:36.475 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:36 smithi164 conmon[29288]: 35.196336+0000 osd.6 (osd.6) 23 : 2021-02-10T04:28:36.475 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:36 smithi164 conmon[29288]: cluster [DBG] 2.17 scrub ok 2021-02-10T04:28:36.475 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:36 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:28:36.475 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:36 smithi164 conmon[29288]: -10T04:28:35.416893+0000 osd.7 2021-02-10T04:28:36.475 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:36 smithi164 conmon[29288]: (osd.7) 31 : cluster [DBG] 2.26 scrub ok 2021-02-10T04:28:36.475 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:36 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:28:36.475 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:36 smithi164 conmon[29288]: -10T04:28:35.520403+0000 osd.4 ( 2021-02-10T04:28:36.475 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:36 smithi164 conmon[29288]: osd.4) 32 : cluster [DBG] 4.18 scrub ok 2021-02-10T04:28:36.475 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:36 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:28:36.475 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:36 smithi164 conmon[29288]: 02-10T04:28:35.648598+0000 2021-02-10T04:28:36.476 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:36 smithi164 conmon[29288]: osd.2 (osd.2) 28 : cluster [DBG] 5.d scrub ok 2021-02-10T04:28:36.476 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:36 smithi164 conmon[29288]: 2021-02-10T04:28:36.476 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:36 smithi164 conmon[29288]: cluster 2021-02-10T04:28:36.476 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:36 smithi164 conmon[29288]: 2021-02-10T04:28:35.992578+0000 mgr.y ( 2021-02-10T04:28:36.476 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:36 smithi164 conmon[29288]: mgr.14140) 331 : cluster [DBG] pgmap v298: 158 pgs: 158 active+clean; 50 MiB data, 114 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 4.2 MiB/s wr, 22 op/s 2021-02-10T04:28:36.477 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:36.565 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[29319]: cluster 2021-02-10T04:28:35.010703+0000 osd.1 (osd.1) 35 : cluster [DBG] 2.6 scrub ok 2021-02-10T04:28:36.565 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[29319]: cluster 2021-02-10T04:28:35.190563+0000 mon.a (mon.0) 740 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2021-02-10T04:28:36.565 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[29319]: cluster 2021-02-10T04:28:35.196336+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.17 scrub ok 2021-02-10T04:28:36.565 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[29319]: cluster 2021-02-10T04:28:35.416893+0000 osd.7 (osd.7) 31 : cluster [DBG] 2.26 scrub ok 2021-02-10T04:28:36.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[29319]: cluster 2021-02-10T04:28:35.520403+0000 osd.4 (osd.4) 32 : cluster [DBG] 4.18 scrub ok 2021-02-10T04:28:36.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[29319]: cluster 2021-02-10T04:28:35.648598+0000 osd.2 (osd.2) 28 : cluster [DBG] 5.d scrub ok 2021-02-10T04:28:36.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[29319]: cluster 2021-02-10T04:28:35.992578+0000 mgr.y (mgr.14140) 331 : cluster [DBG] pgmap v298: 158 pgs: 158 active+clean; 50 MiB data, 114 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 4.2 MiB/s wr, 22 op/s 2021-02-10T04:28:36.566 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[35961]: cluster 2021-02-10T04:28:35.010703 2021-02-10T04:28:36.566 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 35 : cluster [DBG] 2.6 scrub ok 2021-02-10T04:28:36.566 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:28:36.566 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[35961]: 10T04:28:35.190563+0000 2021-02-10T04:28:36.566 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[35961]: mon.a (mon.0) 740 : 2021-02-10T04:28:36.566 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[35961]: cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2021-02-10T04:28:36.566 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:28:36.566 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[35961]: -02-10T04:28:35. 2021-02-10T04:28:36.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[35961]: 196336+0000 osd.6 (osd.6 2021-02-10T04:28:36.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[35961]: ) 23 : cluster [DBG] 2.17 scrub ok 2021-02-10T04:28:36.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[35961]: 2021-02-10T04:28:36.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[35961]: cluster 2021-02-10T04:28:36.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:28:36.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[35961]: 04:28:35.416893+0000 2021-02-10T04:28:36.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[35961]: osd.7 (osd.7) 31 : 2021-02-10T04:28:36.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[35961]: cluster [DBG] 2.26 scrub ok 2021-02-10T04:28:36.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[35961]: cluster 2021-02-10T04:28:36.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[35961]: 2021-02-10T04:28:35 2021-02-10T04:28:36.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[35961]: .520403+0000 osd.4 (osd.4) 2021-02-10T04:28:36.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[35961]: 32 : cluster [DBG] 4.18 scrub ok 2021-02-10T04:28:36.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:28:36.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[35961]: -02-10T04:28:35. 2021-02-10T04:28:36.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[35961]: 648598+0000 osd.2 (osd.2) 28 2021-02-10T04:28:36.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[35961]: : cluster [DBG] 5.d scrub ok 2021-02-10T04:28:36.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[35961]: cluster 2021-02-10T04:28:36.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[35961]: 2021-02-10T04:28 2021-02-10T04:28:36.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[35961]: :35.992578+0000 mgr.y (mgr.14140) 331 : cluster [DBG] pgmap v298: 158 pgs: 158 active+clean; 50 MiB data, 114 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 4.2 MiB/s wr, 22 op/s 2021-02-10T04:28:36.577 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:28:36.681 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:28:36.705 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:36.893 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:28:36 smithi164 conmon[40892]: debug 2021-02-10T04:28:36.550+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:37.001 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:28:37.103 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:28:36 smithi085 conmon[49106]: debug 2021-02-10T04:28:36.959+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:37.307 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:28:37.308 INFO:tasks.rados.rados.0.smithi085.stdout:62: done (2 left) 2021-02-10T04:28:37.309 INFO:tasks.rados.rados.0.smithi085.stdout:63: read oid 28 snap -1 2021-02-10T04:28:37.309 INFO:tasks.rados.rados.0.smithi085.stdout:63: expect (ObjNum 27 snap 0 seq_num 27) 2021-02-10T04:28:37.309 INFO:tasks.rados.rados.0.smithi085.stdout:64: read oid 33 snap -1 2021-02-10T04:28:37.309 INFO:tasks.rados.rados.0.smithi085.stdout:64: expect (ObjNum 53 snap 2 seq_num 53) 2021-02-10T04:28:37.309 INFO:tasks.rados.rados.0.smithi085.stdout:65: delete oid 2 current snap is 2 2021-02-10T04:28:37.309 INFO:tasks.rados.rados.0.smithi085.stdout:61: finishing rollback tid 0 to smithi08575397-47 2021-02-10T04:28:37.309 INFO:tasks.rados.rados.0.smithi085.stdout:60: finishing write tid 3 to smithi08575397-12 2021-02-10T04:28:37.309 INFO:tasks.rados.rados.0.smithi085.stdout:60: finishing write tid 4 to smithi08575397-12 2021-02-10T04:28:37.309 INFO:tasks.rados.rados.0.smithi085.stdout:60: finishing write tid 5 to smithi08575397-12 2021-02-10T04:28:37.309 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 20 (ObjNum 55 snap 2 seq_num 55) dirty exists 2021-02-10T04:28:37.309 INFO:tasks.rados.rados.0.smithi085.stdout:60: left oid 12 (ObjNum 55 snap 2 seq_num 55) 2021-02-10T04:28:37.309 INFO:tasks.rados.rados.0.smithi085.stdout:61: finishing rollback tid 1 to smithi08575397-47 2021-02-10T04:28:37.309 INFO:tasks.rados.rados.0.smithi085.stdout:61: finishing rollback tid 2 to smithi08575397-47 2021-02-10T04:28:37.310 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 47 v 25 (ObjNum 46 snap 0 seq_num 46) dirty exists 2021-02-10T04:28:37.310 INFO:tasks.rados.rados.0.smithi085.stdout:60: done (4 left) 2021-02-10T04:28:37.310 INFO:tasks.rados.rados.0.smithi085.stdout:61: done (3 left) 2021-02-10T04:28:37.310 INFO:tasks.rados.rados.0.smithi085.stdout:65: done (2 left) 2021-02-10T04:28:37.310 INFO:tasks.rados.rados.0.smithi085.stdout:66: copy_from oid 3 from oid 31 current snap is 2 2021-02-10T04:28:37.310 INFO:tasks.rados.rados.0.smithi085.stdout:67: snap_create 2021-02-10T04:28:37.310 INFO:tasks.rados.rados.0.smithi085.stdout:63: expect (ObjNum 27 snap 0 seq_num 27) 2021-02-10T04:28:37.310 INFO:tasks.rados.rados.0.smithi085.stdout:64: expect (ObjNum 53 snap 2 seq_num 53) 2021-02-10T04:28:37.338 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:37.419 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:28:37.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[29319]: cluster 2021-02-10T04:28:37.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[29319]: 2021-02-10T04:28:36 2021-02-10T04:28:37.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[29319]: .197932+0000 mon.a (mon.0) 741 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2021-02-10T04:28:37.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:28:37.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[29319]: 04:28:36.204001+0000 osd.6 (osd 2021-02-10T04:28:37.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[29319]: .6) 24 : cluster [DBG] 2.14 scrub ok 2021-02-10T04:28:37.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:28:37.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[29319]: 02-10T04:28:36.311721+0000 osd.3 (osd 2021-02-10T04:28:37.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[29319]: .3) 37 : cluster [DBG] 5.13 scrub ok 2021-02-10T04:28:37.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[29319]: cluster 2021-02-10T04:28 2021-02-10T04:28:37.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[29319]: :36.316377+0000 osd.7 (osd.7) 32 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:28:37.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[29319]: 2021-02-10T04:28:37.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[29319]: cluster 2021-02-10T04:28: 2021-02-10T04:28:37.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[29319]: 36.316534+0000 osd.7 (osd.7) 33 : 2021-02-10T04:28:37.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[29319]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:28:37.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:28:37.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[29319]: 02-10T04:28:36. 2021-02-10T04:28:37.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[29319]: 490955+0000 osd.5 (osd.5) 30 : cluster [DBG] 2.2 scrub ok 2021-02-10T04:28:37.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:28:37.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[29319]: -02-10T04:28:36 2021-02-10T04:28:37.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[29319]: .753982+0000 osd.0 (osd.0) 34 : 2021-02-10T04:28:37.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[29319]: cluster [DBG] 4.3 scrub ok 2021-02-10T04:28:37.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[35961]: cluster 2021-02-10T04:28:36.197932+0000 2021-02-10T04:28:37.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[35961]: mon.a (mon.0) 741 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2021-02-10T04:28:37.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[35961]: cluster 2021-02-10T04:28:36.204001+0000 2021-02-10T04:28:37.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[35961]: osd.6 (osd.6) 24 : cluster [DBG] 2.14 scrub ok 2021-02-10T04:28:37.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[35961]: cluster 2021-02-10T04:28:37.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[35961]: 2021-02-10T04:28:36.311721+0000 osd.3 (osd. 2021-02-10T04:28:37.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[35961]: 3) 37 : cluster [DBG] 5.13 scrub ok 2021-02-10T04:28:37.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:28:37.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[35961]: T04:28:36.316377+0000 osd.7 (osd 2021-02-10T04:28:37.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[35961]: .7) 32 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:28:37.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:28:37.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[35961]: 02-10T04:28:36.316534+0000 2021-02-10T04:28:37.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[35961]: osd.7 (osd.7) 33 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:28:37.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[35961]: 2021-02-10T04:28:37.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[35961]: cluster 2021-02-10T04:28:37.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[35961]: 2021-02-10T04:28:36.490955+0000 osd.5 ( 2021-02-10T04:28:37.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[35961]: osd.5) 30 : cluster [DBG] 2.2 scrub ok 2021-02-10T04:28:37.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:28:37.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[35961]: -02-10T04:28: 2021-02-10T04:28:37.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[35961]: 36.753982+0000 osd.0 (osd. 2021-02-10T04:28:37.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:37 smithi085 conmon[35961]: 0) 34 : cluster [DBG] 4.3 scrub ok 2021-02-10T04:28:37.523 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:28:37.554 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:37.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:28:37 smithi164 conmon[37263]: debug 2021-02-10T04:28:37.211+0000 7f1c689e5700 -1 osd.5 pg_epoch: 101 pg[7.0( v 99'11 (0'0,99'11] local-lis/les=93/94 n=4 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [5,1,3] r=0 lpr=93 crt=99'11 lcod 98'9 mlcod 98'9 active+clean+snaptrim trimq=[2~1]] removing snap head 2021-02-10T04:28:37.644 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:28:37 smithi164 conmon[44544]: debug 2021-02-10T04:28:37.397+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:37.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:37 smithi164 conmon[29288]: cluster 2021-02-10T04:28:36.197932+0000 mon.a (mon.0) 741 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2021-02-10T04:28:37.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:37 smithi164 conmon[29288]: 2021-02-10T04:28:37.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:37 smithi164 conmon[29288]: cluster 2021-02-10T04:28:36.204001+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.14 scrub ok 2021-02-10T04:28:37.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:37 smithi164 conmon[29288]: cluster 2021-02-10T04:28:36.311721+0000 osd.3 (osd.3) 37 : cluster [DBG] 2021-02-10T04:28:37.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:37 smithi164 conmon[29288]: 5.13 scrub ok 2021-02-10T04:28:37.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:37 smithi164 conmon[29288]: cluster 2021-02-10T04:28:36.316377 2021-02-10T04:28:37.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:37 smithi164 conmon[29288]: +0000 osd.7 (osd.7) 32 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:28:37.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:37 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:28:37.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:37 smithi164 conmon[29288]: -02-10T04:28:36.316534+0000 2021-02-10T04:28:37.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:37 smithi164 conmon[29288]: osd.7 (osd.7) 33 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:28:37.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:37 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:28:37.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:37 smithi164 conmon[29288]: -02-10T04:28:36.490955+0000 osd.5 2021-02-10T04:28:37.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:37 smithi164 conmon[29288]: (osd.5) 30 : cluster [DBG] 2.2 scrub ok 2021-02-10T04:28:37.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:37 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:28:37.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:37 smithi164 conmon[29288]: 02-10T04:28:36.753982+0000 osd.0 ( 2021-02-10T04:28:37.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:37 smithi164 conmon[29288]: osd.0) 34 : cluster [DBG] 4.3 scrub ok 2021-02-10T04:28:38.025 INFO:tasks.rados.rados.0.smithi085.stdout:63: done (3 left) 2021-02-10T04:28:38.026 INFO:tasks.rados.rados.0.smithi085.stdout:64: done (2 left) 2021-02-10T04:28:38.026 INFO:tasks.rados.rados.0.smithi085.stdout:67: done (1 left) 2021-02-10T04:28:38.026 INFO:tasks.rados.rados.0.smithi085.stdout:68: snap_remove snap 1 2021-02-10T04:28:38.068 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:28:38.143 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:28:37 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:28:38.175 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:28:38.205 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:38.272 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[52836]: debug 2021-02-10T04:28:38.039+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:38.273 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: cluster 2021-02-10T04:28:37.201923+0000 mon.a (mon.0) 742 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2021-02-10T04:28:38.273 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: cluster 2021-02-10T04:28:37.215326+0000 osd.6 (osd.6) 25 : cluster [DBG] 5.2 scrub ok 2021-02-10T04:28:38.273 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: cluster 2021-02-10T04:28:37.514934+0000 osd.5 (osd.5) 31 : cluster [DBG] 2.18 scrub ok 2021-02-10T04:28:38.273 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: cluster 2021-02-10T04:28:37.645854+0000 osd.2 (osd.2) 29 : cluster [DBG] 5.b scrub ok 2021-02-10T04:28:38.273 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: audit 2021-02-10T04:28:37.675033+0000 mgr.y (mgr.14140) 332 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:28:38.273 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: audit 2021-02-10T04:28:37.992068+0000 mon.a (mon.0) 743 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 4]}]: dispatch 2021-02-10T04:28:38.273 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: audit 2021-02-10T04:28:37.992303+0000 mon.a (mon.0) 744 : audit 2021-02-10T04:28:38.273 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]: dispatch 2021-02-10T04:28:38.273 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: audit 2021-02-10T04:28 2021-02-10T04:28:38.274 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: :37.992554+0000 mon.a (mon.0) 745 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.21", "id": [7, 1]}]: dispatch 2021-02-10T04:28:38.274 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: audit 2021-02-10T04:28:37. 2021-02-10T04:28:38.274 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: 992760+0000 mon.a (mon.0) 746 : audit 2021-02-10T04:28:38.274 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.26", "id": [7, 2]}]: dispatch 2021-02-10T04:28:38.274 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: cluster 2021-02-10T04:28:37 2021-02-10T04:28:38.274 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: .992927+0000 mgr.y (mgr.14140) 333 : cluster 2021-02-10T04:28:38.274 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: [DBG] pgmap v301: 158 pgs: 158 active+clean; 68 MiB data, 150 MiB used, 715 GiB / 715 GiB avail; 7.7 KiB/s rd, 8.8 MiB/s wr, 46 op/s 2021-02-10T04:28:38.274 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: audit 2021-02-10T04:28 2021-02-10T04:28:38.274 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: :38.025190+0000 mon.a (mon.0) 747 2021-02-10T04:28:38.274 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 4]}]': finished 2021-02-10T04:28:38.274 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:28:38.274 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: :28:38.025243+0000 mon.a (mon.0 2021-02-10T04:28:38.274 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: ) 748 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]': finished 2021-02-10T04:28:38.275 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:28:38.275 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: 10T04:28:38.025272+0000 mon.a (mon 2021-02-10T04:28:38.275 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: .0) 749 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.21", "id": [7, 1]}]': finished 2021-02-10T04:28:38.275 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: audit 2021 2021-02-10T04:28:38.275 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: -02-10T04:28:38.025313+0000 2021-02-10T04:28:38.275 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: mon.a (mon.0 2021-02-10T04:28:38.275 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: ) 750 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.26", "id": [7, 2]}]': finished 2021-02-10T04:28:38.275 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:28:38.275 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: 02-10T04:28:38.025337+0000 2021-02-10T04:28:38.275 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: mon.a (mon.0) 751 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2021-02-10T04:28:38.275 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: 2021-02-10T04:28:38.275 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:28:38.275 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: :28:38.041978+0000 mon.a (mon.0) 752 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:28:38.275 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:28:38.276 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: 02-10T04:28:38.042888+0000 mon.a 2021-02-10T04:28:38.276 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: (mon.0) 753 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:28:38.276 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: audit 2021 2021-02-10T04:28:38.276 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: -02-10T04:28:38.059500+0000 2021-02-10T04:28:38.276 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: mon.c (mon.1) 37 : audit [INF] from='client.? 172.21.15.85:0/3811930449' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:28:38.276 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: 2021-02-10T04:28:38.276 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:28:38.276 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: 04:28:38.059837+0000 mon.a (mon 2021-02-10T04:28:38.276 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[35961]: .0) 754 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:28:38.277 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: cluster 2021-02-10T04:28:37.201923+0000 mon.a (mon.0) 742 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2021-02-10T04:28:38.277 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: cluster 2021-02-10T04:28:37.215326+0000 osd.6 (osd.6) 25 : cluster [DBG] 5.2 scrub ok 2021-02-10T04:28:38.277 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: cluster 2021-02-10T04:28:37.514934+0000 osd.5 (osd.5) 31 : cluster [DBG] 2.18 scrub ok 2021-02-10T04:28:38.277 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: cluster 2021-02-10T04:28:37.645854+0000 osd.2 (osd.2) 29 : cluster [DBG] 5.b scrub ok 2021-02-10T04:28:38.277 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: audit 2021-02-10T04:28:37.675033+0000 mgr.y (mgr.14140) 332 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:28:38.277 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: audit 2021-02-10T04:28:37.992068+0000 mon.a (mon.0) 743 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 4]}]: dispatch 2021-02-10T04:28:38.277 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: audit 2021-02-10T04:28:37.992303+0000 mon.a (mon.0) 744 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]: dispatch 2021-02-10T04:28:38.277 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: audit 2021-02-10T04:28:37.992554+0000 mon.a (mon.0) 745 2021-02-10T04:28:38.277 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.21", "id": [7, 1]}]: dispatch 2021-02-10T04:28:38.277 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: audit 2021-02-10T04:28:37.992760+0000 mon.a (mon.0) 746 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.26", "id": [7, 2]}]: dispatch 2021-02-10T04:28:38.277 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: cluster 2021-02-10T04:28:37.992927+0000 mgr.y (mgr.14140) 333 : cluster [DBG] pgmap v301: 158 pgs: 158 active+clean; 68 MiB data, 150 MiB used, 715 GiB / 715 GiB avail; 7.7 KiB/s rd, 8.8 MiB/s wr, 46 op/s 2021-02-10T04:28:38.277 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: audit 2021-02-10T04:28:38.025190+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 4]}]': finished 2021-02-10T04:28:38.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: audit 2021-02-10T04:28:38.025243+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]': finished 2021-02-10T04:28:38.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: audit 2021-02-10T04:28:38.025272+0000 mon.a (mon.0) 749 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.21", "id": [7, 1]}]': finished 2021-02-10T04:28:38.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:28:38.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: T04:28:38.025313+0000 mon.a (mon.0) 750 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.26", "id": [7, 2]}]': finished 2021-02-10T04:28:38.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: cluster 2021-02-10T04:28:38.025337+0000 mon.a (mon.0) 751 : cluster 2021-02-10T04:28:38.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: [DBG] osdmap e102: 8 total, 8 up, 8 in 2021-02-10T04:28:38.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:28:38.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: 02-10T04:28:38.041978+0000 mon.a (mon.0) 752 : audit [INF] 2021-02-10T04:28:38.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:28:38.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: audit 2021-02-10T04:28:38.042888 2021-02-10T04:28:38.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: +0000 mon.a (mon.0) 753 : audit [INF] 2021-02-10T04:28:38.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:28:38.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: audit 2021-02-10T04:28:38.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: 2021-02-10T04:28:38.059500+0000 mon.c (mon.1) 2021-02-10T04:28:38.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: 37 : audit [INF] from='client.? 172.21.15.85:0/3811930449' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:28:38.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: audit 2021-02-10T04:28:38.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: 2021-02-10T04:28 2021-02-10T04:28:38.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: :38.059837+0000 mon.a ( 2021-02-10T04:28:38.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: mon.0) 754 : audit [INF] 2021-02-10T04:28:38.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[29319]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:28:38.309 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:28:38.414 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:28:38.433 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:28:38.441 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:28:38.441 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:38.513 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: cluster 2021-02-10T04:28:37.201923+0000 mon.a (mon 2021-02-10T04:28:38.513 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: .0) 742 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2021-02-10T04:28:38.513 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: cluster 2021-02-10T04:28:37.215326+0000 osd.6 (osd.6) 25 : cluster [DBG] 5.2 scrub ok 2021-02-10T04:28:38.513 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: cluster 2021-02-10T04:28:37.514934+0000 osd.5 (osd.5) 31 : cluster [DBG] 2021-02-10T04:28:38.513 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: 2.18 scrub ok 2021-02-10T04:28:38.513 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: cluster 2021-02-10T04:28:37.645854+0000 osd.2 2021-02-10T04:28:38.513 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: (osd.2) 29 : cluster [DBG] 5.b scrub ok 2021-02-10T04:28:38.513 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: audit 2021-02-10T04:28:38.513 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: 2021-02-10T04:28:37.675033 2021-02-10T04:28:38.513 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 332 : audit [DBG] 2021-02-10T04:28:38.514 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:28:38.514 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: audit 2021-02-10T04:28:38.514 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: 2021-02-10T04:28:37.992068+0000 mon.a (mon.0) 743 : audit 2021-02-10T04:28:38.514 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 4]}]: dispatch 2021-02-10T04:28:38.514 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:28:38.514 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: :28:37.992303+0000 mon.a (mon.0) 744 : audit 2021-02-10T04:28:38.514 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]: dispatch 2021-02-10T04:28:38.514 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: audit 2021-02-10T04:28:38.514 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: 2021-02-10T04:28:37.992554+0000 mon.a ( 2021-02-10T04:28:38.514 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: mon.0) 745 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.21", "id": [7, 1]}]: dispatch 2021-02-10T04:28:38.514 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: audit 2021 2021-02-10T04:28:38.514 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: -02-10T04:28:37.992760 2021-02-10T04:28:38.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: +0000 mon.a (mon.0) 746 : audit [INF] 2021-02-10T04:28:38.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.26", "id": [7, 2]}]: dispatch 2021-02-10T04:28:38.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: cluster 2021-02-10T04:28:37.992927+0000 mgr.y (mgr.14140) 333 : cluster 2021-02-10T04:28:38.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: [DBG] pgmap v301: 158 pgs: 158 active+clean; 68 MiB data, 150 MiB used, 715 GiB / 715 GiB avail; 7.7 KiB/s rd, 8.8 MiB/s wr, 46 op/s 2021-02-10T04:28:38.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: audit 2021-02-10T04:28:38.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: 2021-02-10T04:28:38.025190+0000 mon.a 2021-02-10T04:28:38.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: (mon.0) 747 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 4]}]': finished 2021-02-10T04:28:38.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: audit 2021-02-10T04:28: 2021-02-10T04:28:38.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: 38.025243+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]': finished 2021-02-10T04:28:38.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: audit 2021-02-10T04:28: 2021-02-10T04:28:38.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: 38.025272+0000 mon.a (mon.0) 749 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.21", "id": [7, 1]}]': finished 2021-02-10T04:28:38.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:28:38.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: :28:38.025313+0000 mon.a (mon.0) 750 : audit 2021-02-10T04:28:38.516 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.26", "id": [7, 2]}]': finished 2021-02-10T04:28:38.516 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: cluster 2021-02-10T04:28:38.025337+0000 mon.a ( 2021-02-10T04:28:38.516 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: mon.0) 751 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2021-02-10T04:28:38.516 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:28:38.516 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: -10T04:28:38.041978+0000 mon.a ( 2021-02-10T04:28:38.516 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: mon.0) 752 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:28:38.516 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:28:38.516 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: 02-10T04:28:38.042888+0000 2021-02-10T04:28:38.516 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: mon.a (mon.0) 753 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:28:38.516 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: audit 2021-02-10T04:28:38. 2021-02-10T04:28:38.516 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: 059500+0000 mon.c (mon.1) 37 : audit 2021-02-10T04:28:38.516 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: [INF] from='client.? 172.21.15.85:0/3811930449' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:28:38.516 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: audit 2021-02-10T04:28: 2021-02-10T04:28:38.517 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[29288]: 38.059837+0000 mon.a (mon.0) 754 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:28:38.527 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:28:38.596 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:28:38.622 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:28:38 smithi085 conmon[52836]: debug 2021-02-10T04:28:38.272+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:38.631 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:28:38.657 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:38.752 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:28:38.834 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:28:38.835 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 20, 2021-02-10T04:28:38.835 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 600, 2021-02-10T04:28:38.835 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [ 2021-02-10T04:28:38.835 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.835 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.34319.0:39 2.6 2:60e91ebf:::zone_info.31f351f5-a7bc-4340-9159-1339b03dc902:head [read 0~846] snapc 0=[] ondisk+read+known_if_redirected e53)", 2021-02-10T04:28:38.835 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:25:40.829136+0000", 2021-02-10T04:28:38.835 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 177.994540065, 2021-02-10T04:28:38.835 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003342767, 2021-02-10T04:28:38.835 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:38.835 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:28:38.836 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:38.836 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.34319", 2021-02-10T04:28:38.836 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.164:0/1792421759", 2021-02-10T04:28:38.836 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 39 2021-02-10T04:28:38.836 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.836 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:38.836 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.836 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:38.836 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:40.829136+0000", 2021-02-10T04:28:38.836 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:38.836 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.836 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.836 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:38.837 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:40.829136+0000", 2021-02-10T04:28:38.837 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.8314000000000001e-05 2021-02-10T04:28:38.837 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.837 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.837 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:38.837 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:40.829154+0000", 2021-02-10T04:28:38.837 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.3460000000000001e-06 2021-02-10T04:28:38.837 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.837 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.837 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:38.837 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:40.829159+0000", 2021-02-10T04:28:38.837 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.8380000000000008e-06 2021-02-10T04:28:38.837 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.837 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.838 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:38.838 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:40.829168+0000", 2021-02-10T04:28:38.838 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.7238e-05 2021-02-10T04:28:38.838 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.838 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.838 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:38.838 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:40.829186+0000", 2021-02-10T04:28:38.838 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.8758999999999999e-05 2021-02-10T04:28:38.838 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.838 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.838 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:38.838 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:40.829234+0000", 2021-02-10T04:28:38.838 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.8459000000000003e-05 2021-02-10T04:28:38.839 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.839 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.839 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:38.839 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:40.829303+0000", 2021-02-10T04:28:38.839 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003175813 2021-02-10T04:28:38.839 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.839 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.839 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:38.839 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:40.832479+0000", 2021-02-10T04:28:38.839 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003342767 2021-02-10T04:28:38.839 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.839 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:38.839 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.840 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.840 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.840 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:53 4.2 4:4347d321:::bucket.sync-source-hints.:head [call,read 0~0] snapc 0=[] ondisk+read+known_if_redirected e56)", 2021-02-10T04:28:38.840 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:25:48.995080+0000", 2021-02-10T04:28:38.840 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 169.828595984, 2021-02-10T04:28:38.840 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.011610191000000001, 2021-02-10T04:28:38.840 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:38.840 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:28:38.840 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:38.840 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:28:38.840 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:28:38.840 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 53 2021-02-10T04:28:38.840 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.840 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:38.841 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.841 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:38.841 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:48.995080+0000", 2021-02-10T04:28:38.841 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:38.841 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.841 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.841 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:38.841 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:48.995080+0000", 2021-02-10T04:28:38.841 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.1201000000000001e-05 2021-02-10T04:28:38.841 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.841 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.841 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:38.841 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:48.995121+0000", 2021-02-10T04:28:38.842 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.5220000000000001e-06 2021-02-10T04:28:38.842 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.842 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.842 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:38.842 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:48.995129+0000", 2021-02-10T04:28:38.842 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.3692e-05 2021-02-10T04:28:38.842 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.842 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.842 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:38.842 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:48.995142+0000", 2021-02-10T04:28:38.842 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.9789999999999996e-06 2021-02-10T04:28:38.842 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.842 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.843 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:38.843 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:48.995149+0000", 2021-02-10T04:28:38.843 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00279948 2021-02-10T04:28:38.843 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.843 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.843 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:38.843 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:48.997949+0000", 2021-02-10T04:28:38.843 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.1929999999999993e-06 2021-02-10T04:28:38.843 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.843 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.843 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for peered", 2021-02-10T04:28:38.843 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:48.997957+0000", 2021-02-10T04:28:38.843 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0086204409999999995 2021-02-10T04:28:38.843 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.844 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.844 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:38.844 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:49.006577+0000", 2021-02-10T04:28:38.844 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00011268300000000001 2021-02-10T04:28:38.844 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.844 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.844 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:38.844 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:49.006690+0000", 2021-02-10T04:28:38.844 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.011610191000000001 2021-02-10T04:28:38.844 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.844 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:38.844 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.844 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.845 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.845 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:361 4.2 4:44ce71ea:::data_log.16:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e58)", 2021-02-10T04:28:38.845 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:25:51.208784+0000", 2021-02-10T04:28:38.845 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 167.61489228799999, 2021-02-10T04:28:38.845 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.002966879, 2021-02-10T04:28:38.845 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:38.845 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:28:38.845 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:38.845 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:28:38.845 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:28:38.845 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 361 2021-02-10T04:28:38.845 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.845 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:38.845 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.846 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:38.846 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.208784+0000", 2021-02-10T04:28:38.846 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:38.846 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.846 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.846 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:38.846 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.208784+0000", 2021-02-10T04:28:38.846 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.4419000000000001e-05 2021-02-10T04:28:38.846 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.846 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.846 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:38.846 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.208808+0000", 2021-02-10T04:28:38.846 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.3619999999999999e-06 2021-02-10T04:28:38.847 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.847 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.847 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:38.847 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.208812+0000", 2021-02-10T04:28:38.847 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.3840000000000003e-06 2021-02-10T04:28:38.847 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.847 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.847 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:38.847 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.208822+0000", 2021-02-10T04:28:38.847 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.7660000000000004e-06 2021-02-10T04:28:38.847 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.847 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.847 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:38.847 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.208828+0000", 2021-02-10T04:28:38.848 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.8626e-05 2021-02-10T04:28:38.848 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.848 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.848 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:38.848 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.208856+0000", 2021-02-10T04:28:38.848 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.8953999999999998e-05 2021-02-10T04:28:38.848 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.848 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.848 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:38.848 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.208915+0000", 2021-02-10T04:28:38.848 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.002835368 2021-02-10T04:28:38.848 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.848 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.849 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:38.849 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.211750+0000", 2021-02-10T04:28:38.849 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.002966879 2021-02-10T04:28:38.849 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.849 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:38.849 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.849 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.849 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.849 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:363 4.4 4:23fdf5ad:::data_log.17:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e58)", 2021-02-10T04:28:38.849 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:25:51.213150+0000", 2021-02-10T04:28:38.849 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 167.61052580800001, 2021-02-10T04:28:38.849 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003216099, 2021-02-10T04:28:38.849 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:38.850 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:28:38.850 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:38.850 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:28:38.850 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:28:38.850 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 363 2021-02-10T04:28:38.850 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.850 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:38.850 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.850 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:38.850 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.213150+0000", 2021-02-10T04:28:38.850 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:38.850 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.851 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.851 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:38.851 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.213150+0000", 2021-02-10T04:28:38.851 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.2656000000000001e-05 2021-02-10T04:28:38.851 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.851 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.851 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:38.851 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.213173+0000", 2021-02-10T04:28:38.851 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.0450000000000002e-06 2021-02-10T04:28:38.851 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.852 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.852 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:38.852 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.213177+0000", 2021-02-10T04:28:38.852 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.2010000000000005e-06 2021-02-10T04:28:38.852 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.852 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.852 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:38.852 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.213186+0000", 2021-02-10T04:28:38.852 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.4330000000000003e-06 2021-02-10T04:28:38.852 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.853 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.853 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:38.853 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.213191+0000", 2021-02-10T04:28:38.853 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.1112999999999998e-05 2021-02-10T04:28:38.853 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.853 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.853 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:38.853 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.213223+0000", 2021-02-10T04:28:38.853 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.6407999999999999e-05 2021-02-10T04:28:38.853 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.853 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.853 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:38.853 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.213279+0000", 2021-02-10T04:28:38.854 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0030872429999999999 2021-02-10T04:28:38.854 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.854 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.854 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:38.854 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.216366+0000", 2021-02-10T04:28:38.854 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003216099 2021-02-10T04:28:38.854 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.854 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:38.854 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.854 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.854 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.854 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:365 4.12 4:4cacdd72:::data_log.18:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e58)", 2021-02-10T04:28:38.854 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:25:51.217827+0000", 2021-02-10T04:28:38.855 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 167.605849183, 2021-02-10T04:28:38.855 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0033711119999999999, 2021-02-10T04:28:38.855 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:38.855 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:28:38.855 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:38.855 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:28:38.855 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:28:38.855 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 365 2021-02-10T04:28:38.855 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.855 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:38.855 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.855 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:38.855 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.217827+0000", 2021-02-10T04:28:38.856 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:38.856 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.856 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.856 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:38.856 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.217827+0000", 2021-02-10T04:28:38.856 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.4105e-05 2021-02-10T04:28:38.856 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.856 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.856 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:38.856 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.217851+0000", 2021-02-10T04:28:38.856 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.1309999999999999e-06 2021-02-10T04:28:38.856 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.856 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.856 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:38.857 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.217855+0000", 2021-02-10T04:28:38.857 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.1330000000000005e-06 2021-02-10T04:28:38.857 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.857 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.857 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:38.857 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.217864+0000", 2021-02-10T04:28:38.857 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.541e-06 2021-02-10T04:28:38.857 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.857 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.857 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:38.857 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.217870+0000", 2021-02-10T04:28:38.857 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.0925e-05 2021-02-10T04:28:38.857 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.857 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.858 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:38.858 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.217901+0000", 2021-02-10T04:28:38.858 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.5464999999999998e-05 2021-02-10T04:28:38.858 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.858 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.858 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:38.858 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.217956+0000", 2021-02-10T04:28:38.858 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0032418120000000002 2021-02-10T04:28:38.858 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.858 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.858 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:38.858 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.221198+0000", 2021-02-10T04:28:38.858 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0033711119999999999 2021-02-10T04:28:38.859 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.859 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:38.859 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.859 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.859 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.859 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:467 4.2 4:428e7f58:::data_log.69:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e58)", 2021-02-10T04:28:38.859 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:25:51.379047+0000", 2021-02-10T04:28:38.859 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 167.44462855399999, 2021-02-10T04:28:38.859 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0033495230000000001, 2021-02-10T04:28:38.859 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:38.859 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:28:38.859 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:38.859 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:28:38.860 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:28:38.860 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 467 2021-02-10T04:28:38.860 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.860 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:38.860 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.860 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:38.860 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.379047+0000", 2021-02-10T04:28:38.860 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:38.860 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.860 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.860 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:38.860 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.379047+0000", 2021-02-10T04:28:38.860 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.2711e-05 2021-02-10T04:28:38.860 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.861 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.861 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:38.861 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.379070+0000", 2021-02-10T04:28:38.861 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.6480000000000002e-06 2021-02-10T04:28:38.861 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.861 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.861 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:38.861 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.379075+0000", 2021-02-10T04:28:38.861 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.5029999999999997e-06 2021-02-10T04:28:38.861 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.861 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.861 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:38.861 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.379083+0000", 2021-02-10T04:28:38.861 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.8279999999999996e-06 2021-02-10T04:28:38.862 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.862 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.862 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:38.862 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.379087+0000", 2021-02-10T04:28:38.862 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.5537e-05 2021-02-10T04:28:38.862 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.862 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.862 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:38.862 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.379113+0000", 2021-02-10T04:28:38.862 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.4027999999999998e-05 2021-02-10T04:28:38.862 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.862 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.862 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:38.863 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.379167+0000", 2021-02-10T04:28:38.863 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0032302680000000001 2021-02-10T04:28:38.863 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.863 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.863 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:38.863 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:51.382397+0000", 2021-02-10T04:28:38.863 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0033495230000000001 2021-02-10T04:28:38.863 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.863 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:38.863 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.863 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.863 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.863 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24382.0:161 4.4 4:26625e58:lc::lc.26:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2021-02-10T04:28:38.864 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:25:55.144135+0000", 2021-02-10T04:28:38.864 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 163.679541366, 2021-02-10T04:28:38.864 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0032714440000000001, 2021-02-10T04:28:38.864 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:38.864 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:28:38.864 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:38.864 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24382", 2021-02-10T04:28:38.864 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/3915233377", 2021-02-10T04:28:38.864 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 161 2021-02-10T04:28:38.864 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.864 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:38.864 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.864 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:38.864 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.144135+0000", 2021-02-10T04:28:38.865 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:38.865 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.865 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.865 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:38.865 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.144135+0000", 2021-02-10T04:28:38.865 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.2123999999999999e-05 2021-02-10T04:28:38.865 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.865 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.865 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:38.865 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.144177+0000", 2021-02-10T04:28:38.865 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.0450000000000003e-06 2021-02-10T04:28:38.865 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.865 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.866 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:38.866 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.144184+0000", 2021-02-10T04:28:38.866 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2927e-05 2021-02-10T04:28:38.866 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.866 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.866 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:38.866 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.144197+0000", 2021-02-10T04:28:38.866 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.3839000000000001e-05 2021-02-10T04:28:38.866 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.866 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.866 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:38.866 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.144210+0000", 2021-02-10T04:28:38.867 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.6773000000000002e-05 2021-02-10T04:28:38.867 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.867 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.867 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:38.867 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.144247+0000", 2021-02-10T04:28:38.867 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000119376 2021-02-10T04:28:38.867 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.867 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.867 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:38.868 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.144367+0000", 2021-02-10T04:28:38.868 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00018880500000000001 2021-02-10T04:28:38.868 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.868 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.868 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for subops from 2,3", 2021-02-10T04:28:38.868 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.144555+0000", 2021-02-10T04:28:38.868 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00091727300000000002 2021-02-10T04:28:38.868 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.868 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.868 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op_commit", 2021-02-10T04:28:38.868 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.145473+0000", 2021-02-10T04:28:38.868 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00084821600000000005 2021-02-10T04:28:38.868 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.868 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.869 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:38.869 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.146321+0000", 2021-02-10T04:28:38.869 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001002768 2021-02-10T04:28:38.869 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.869 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.869 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:38.869 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.147324+0000", 2021-02-10T04:28:38.869 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.3621e-05 2021-02-10T04:28:38.869 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.869 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.869 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:28:38.869 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.147377+0000", 2021-02-10T04:28:38.869 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.8677e-05 2021-02-10T04:28:38.870 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.870 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.870 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:38.870 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.147406+0000", 2021-02-10T04:28:38.870 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0032714440000000001 2021-02-10T04:28:38.870 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.870 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:38.870 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.870 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.870 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.870 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24382.0:162 4.4 4:25ca39bb:::obj_delete_at_hint.0000000005:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2021-02-10T04:28:38.870 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:25:55.144251+0000", 2021-02-10T04:28:38.870 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 163.679424611, 2021-02-10T04:28:38.870 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0034329830000000001, 2021-02-10T04:28:38.871 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:38.871 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:28:38.871 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:38.871 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24382", 2021-02-10T04:28:38.871 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/3915233377", 2021-02-10T04:28:38.871 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 162 2021-02-10T04:28:38.871 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.871 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:38.871 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.871 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:38.871 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.144251+0000", 2021-02-10T04:28:38.871 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:38.871 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.872 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.872 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:38.872 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.144251+0000", 2021-02-10T04:28:38.872 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.2349999999999999e-05 2021-02-10T04:28:38.872 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.872 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.872 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:38.872 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.144294+0000", 2021-02-10T04:28:38.872 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.2779999999999999e-06 2021-02-10T04:28:38.872 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.872 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.872 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:38.872 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.144300+0000", 2021-02-10T04:28:38.872 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0732e-05 2021-02-10T04:28:38.873 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.873 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.873 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:38.873 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.144311+0000", 2021-02-10T04:28:38.873 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1371e-05 2021-02-10T04:28:38.873 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.873 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.873 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:38.873 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.144322+0000", 2021-02-10T04:28:38.873 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00066304999999999999 2021-02-10T04:28:38.873 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.873 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.873 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:38.873 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.144985+0000", 2021-02-10T04:28:38.874 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.1635e-05 2021-02-10T04:28:38.874 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.874 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.874 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:38.874 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.145067+0000", 2021-02-10T04:28:38.874 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00016985000000000001 2021-02-10T04:28:38.874 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.874 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.874 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for subops from 2,3", 2021-02-10T04:28:38.874 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.145237+0000", 2021-02-10T04:28:38.874 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00041447899999999999 2021-02-10T04:28:38.874 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.874 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.875 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op_commit", 2021-02-10T04:28:38.875 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.145651+0000", 2021-02-10T04:28:38.875 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0010244379999999999 2021-02-10T04:28:38.875 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.875 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.875 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:38.875 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.146675+0000", 2021-02-10T04:28:38.875 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00091148499999999996 2021-02-10T04:28:38.875 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.875 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.875 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:38.875 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.147587+0000", 2021-02-10T04:28:38.875 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.8878000000000001e-05 2021-02-10T04:28:38.875 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.876 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.876 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:28:38.876 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.147646+0000", 2021-02-10T04:28:38.876 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.8436999999999998e-05 2021-02-10T04:28:38.876 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.876 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.876 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:38.876 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.147684+0000", 2021-02-10T04:28:38.876 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0034329830000000001 2021-02-10T04:28:38.876 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.876 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:38.876 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.877 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.877 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.877 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24382.0:647 4.4 4:24c8e1ae:gc::gc.19:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2021-02-10T04:28:38.877 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:25:55.272560+0000", 2021-02-10T04:28:38.877 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 163.55111630900001, 2021-02-10T04:28:38.877 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0030608749999999998, 2021-02-10T04:28:38.877 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:38.877 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:28:38.877 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:38.877 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24382", 2021-02-10T04:28:38.877 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/3915233377", 2021-02-10T04:28:38.877 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 647 2021-02-10T04:28:38.877 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.878 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:38.878 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.878 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:38.878 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.272560+0000", 2021-02-10T04:28:38.878 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:38.878 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.878 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.878 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:38.878 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.272560+0000", 2021-02-10T04:28:38.878 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.7527000000000001e-05 2021-02-10T04:28:38.878 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.878 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.878 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:38.878 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.272607+0000", 2021-02-10T04:28:38.879 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.161e-06 2021-02-10T04:28:38.879 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.879 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.879 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:38.879 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.272613+0000", 2021-02-10T04:28:38.879 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2125e-05 2021-02-10T04:28:38.879 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.879 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.879 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:38.879 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.272625+0000", 2021-02-10T04:28:38.879 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9060000000000001e-06 2021-02-10T04:28:38.879 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.880 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.880 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:38.880 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.272630+0000", 2021-02-10T04:28:38.880 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0010458799999999999 2021-02-10T04:28:38.880 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.880 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.880 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:38.880 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.273676+0000", 2021-02-10T04:28:38.880 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.9535000000000003e-05 2021-02-10T04:28:38.880 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.880 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.880 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:38.880 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.273756+0000", 2021-02-10T04:28:38.881 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00018710800000000001 2021-02-10T04:28:38.881 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.881 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.881 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for subops from 2,3", 2021-02-10T04:28:38.881 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.273943+0000", 2021-02-10T04:28:38.881 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00036104399999999998 2021-02-10T04:28:38.881 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.881 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.881 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op_commit", 2021-02-10T04:28:38.881 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.274304+0000", 2021-02-10T04:28:38.881 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000340467 2021-02-10T04:28:38.881 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.881 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.881 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:38.882 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.274644+0000", 2021-02-10T04:28:38.882 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000872224 2021-02-10T04:28:38.882 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.882 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.882 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:38.882 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.275517+0000", 2021-02-10T04:28:38.882 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.6719999999999997e-05 2021-02-10T04:28:38.882 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.882 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.882 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:28:38.882 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.275593+0000", 2021-02-10T04:28:38.882 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.7178e-05 2021-02-10T04:28:38.882 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.883 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.883 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:38.883 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.275620+0000", 2021-02-10T04:28:38.883 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0030608749999999998 2021-02-10T04:28:38.883 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.883 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:38.883 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.883 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.883 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.883 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24382.0:863 4.7 4:e09767b9:::obj_delete_at_hint.0000000086:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2021-02-10T04:28:38.883 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:25:55.447886+0000", 2021-02-10T04:28:38.883 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 163.37578947899999, 2021-02-10T04:28:38.883 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0055700769999999997, 2021-02-10T04:28:38.883 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:38.884 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:28:38.884 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:38.884 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24382", 2021-02-10T04:28:38.884 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/3915233377", 2021-02-10T04:28:38.884 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 863 2021-02-10T04:28:38.884 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.884 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:38.884 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.884 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:38.884 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.447886+0000", 2021-02-10T04:28:38.884 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:38.884 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.884 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.885 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:38.885 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.447886+0000", 2021-02-10T04:28:38.885 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.2728e-05 2021-02-10T04:28:38.885 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.885 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.885 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:38.885 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.447909+0000", 2021-02-10T04:28:38.885 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.9400000000000004e-06 2021-02-10T04:28:38.885 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.885 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.885 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:38.885 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.447913+0000", 2021-02-10T04:28:38.885 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.9629999999999998e-06 2021-02-10T04:28:38.886 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.886 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.886 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:38.886 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.447921+0000", 2021-02-10T04:28:38.886 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.687e-06 2021-02-10T04:28:38.886 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.886 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.886 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:38.886 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.447925+0000", 2021-02-10T04:28:38.886 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.8694e-05 2021-02-10T04:28:38.886 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.886 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.886 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:38.887 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.447953+0000", 2021-02-10T04:28:38.887 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.4304e-05 2021-02-10T04:28:38.887 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.887 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.887 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:38.887 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.448008+0000", 2021-02-10T04:28:38.887 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000110139 2021-02-10T04:28:38.887 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.887 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.887 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for subops from 0,5", 2021-02-10T04:28:38.887 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.448118+0000", 2021-02-10T04:28:38.887 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00026411000000000002 2021-02-10T04:28:38.887 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.888 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.888 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op_commit", 2021-02-10T04:28:38.888 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.448382+0000", 2021-02-10T04:28:38.888 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000342188 2021-02-10T04:28:38.888 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.888 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.888 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:38.888 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.448724+0000", 2021-02-10T04:28:38.888 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0045584070000000004 2021-02-10T04:28:38.888 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.888 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.888 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:38.888 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.453283+0000", 2021-02-10T04:28:38.888 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00010706500000000001 2021-02-10T04:28:38.889 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.889 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.889 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:28:38.889 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.453390+0000", 2021-02-10T04:28:38.889 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.6852000000000003e-05 2021-02-10T04:28:38.889 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.889 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.889 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:38.889 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:55.453457+0000", 2021-02-10T04:28:38.889 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0055700769999999997 2021-02-10T04:28:38.889 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.889 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:38.889 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.889 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.890 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.890 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:166 7.b 7:d58f4058:::smithi08575397-42:head [write 1699644~784332] snapc 0=[] ondisk+write+known_if_redirected e98)", 2021-02-10T04:28:38.890 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:28:34.576281+0000", 2021-02-10T04:28:38.890 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 4.2473952580000001, 2021-02-10T04:28:38.890 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031718979999999998, 2021-02-10T04:28:38.890 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:38.890 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:28:38.890 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:38.890 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:28:38.890 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:28:38.890 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 166 2021-02-10T04:28:38.890 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.890 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:38.891 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.891 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:38.891 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.576281+0000", 2021-02-10T04:28:38.891 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:38.891 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.891 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.891 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:38.891 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.576281+0000", 2021-02-10T04:28:38.891 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00017653499999999999 2021-02-10T04:28:38.891 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.891 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.891 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:38.891 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.576457+0000", 2021-02-10T04:28:38.892 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.6109999999999996e-06 2021-02-10T04:28:38.892 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.892 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.892 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:38.892 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.576465+0000", 2021-02-10T04:28:38.892 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.4028e-05 2021-02-10T04:28:38.892 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.892 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.892 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:38.892 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.576479+0000", 2021-02-10T04:28:38.892 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.3980000000000002e-06 2021-02-10T04:28:38.892 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.892 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.892 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:38.893 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.576486+0000", 2021-02-10T04:28:38.893 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.1411000000000001e-05 2021-02-10T04:28:38.893 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.893 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.893 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:38.893 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.576518+0000", 2021-02-10T04:28:38.893 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9091999999999997e-05 2021-02-10T04:28:38.893 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.893 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.893 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:38.893 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.576567+0000", 2021-02-10T04:28:38.893 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.1756999999999996e-05 2021-02-10T04:28:38.893 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.893 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.894 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for subops from 5,6", 2021-02-10T04:28:38.894 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.576658+0000", 2021-02-10T04:28:38.894 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0011864410000000001 2021-02-10T04:28:38.894 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.894 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.894 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op_commit", 2021-02-10T04:28:38.894 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.577845+0000", 2021-02-10T04:28:38.894 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0012383260000000001 2021-02-10T04:28:38.894 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.894 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.894 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:38.894 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.579083+0000", 2021-02-10T04:28:38.894 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00030902600000000001 2021-02-10T04:28:38.895 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.895 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.895 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:38.895 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.579392+0000", 2021-02-10T04:28:38.895 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.7549000000000003e-05 2021-02-10T04:28:38.895 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.895 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.895 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:28:38.895 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.579430+0000", 2021-02-10T04:28:38.895 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.2724000000000001e-05 2021-02-10T04:28:38.895 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.895 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.895 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:38.896 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.579453+0000", 2021-02-10T04:28:38.896 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031718979999999998 2021-02-10T04:28:38.896 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.896 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:38.896 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.896 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.896 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.896 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:167 7.b 7:d58f4058:::smithi08575397-42:head [setxattr _header (56) in=7b,truncate 2689087] snapc 0=[] ondisk+write+known_if_redirected e98)", 2021-02-10T04:28:38.896 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:28:34.576526+0000", 2021-02-10T04:28:38.896 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 4.2471496479999997, 2021-02-10T04:28:38.896 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003018714, 2021-02-10T04:28:38.896 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:38.896 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:28:38.896 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:38.897 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:28:38.897 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:28:38.897 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 167 2021-02-10T04:28:38.897 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.897 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:38.897 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.897 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:38.897 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.576526+0000", 2021-02-10T04:28:38.897 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:38.897 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.897 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.897 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:38.897 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.576526+0000", 2021-02-10T04:28:38.897 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9119999999999997e-05 2021-02-10T04:28:38.898 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.898 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.898 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:38.898 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.576575+0000", 2021-02-10T04:28:38.898 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.114e-06 2021-02-10T04:28:38.898 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.898 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.898 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:38.898 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.576582+0000", 2021-02-10T04:28:38.898 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0276e-05 2021-02-10T04:28:38.898 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.898 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.898 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:38.899 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.576593+0000", 2021-02-10T04:28:38.899 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0880000000000002e-06 2021-02-10T04:28:38.899 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.899 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.899 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:38.899 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.576598+0000", 2021-02-10T04:28:38.899 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00082308299999999995 2021-02-10T04:28:38.899 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.899 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.899 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:38.899 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.577421+0000", 2021-02-10T04:28:38.899 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.3298000000000003e-05 2021-02-10T04:28:38.899 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.899 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.900 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:38.900 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.577484+0000", 2021-02-10T04:28:38.900 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000126606 2021-02-10T04:28:38.900 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.900 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.900 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for subops from 5,6", 2021-02-10T04:28:38.900 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.577611+0000", 2021-02-10T04:28:38.900 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00035811300000000002 2021-02-10T04:28:38.900 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.900 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.900 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op_commit", 2021-02-10T04:28:38.900 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.577969+0000", 2021-02-10T04:28:38.900 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0011838129999999999 2021-02-10T04:28:38.901 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.901 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.901 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:38.901 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.579153+0000", 2021-02-10T04:28:38.901 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00033073500000000002 2021-02-10T04:28:38.901 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.901 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.901 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:38.901 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.579484+0000", 2021-02-10T04:28:38.901 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.3204000000000003e-05 2021-02-10T04:28:38.901 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.901 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.901 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:28:38.901 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.579517+0000", 2021-02-10T04:28:38.902 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.8263999999999999e-05 2021-02-10T04:28:38.902 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.902 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.902 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:38.902 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.579545+0000", 2021-02-10T04:28:38.902 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003018714 2021-02-10T04:28:38.902 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.902 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:38.902 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.902 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.902 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.902 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:168 7.b 7:d58f4058:::smithi08575397-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e98)", 2021-02-10T04:28:38.902 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:28:34.576634+0000", 2021-02-10T04:28:38.903 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 4.2470414740000004, 2021-02-10T04:28:38.903 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003064916, 2021-02-10T04:28:38.903 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:38.903 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:28:38.903 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:38.903 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:28:38.903 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:28:38.903 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 168 2021-02-10T04:28:38.903 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.903 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:38.903 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.903 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:38.903 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.576634+0000", 2021-02-10T04:28:38.903 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:38.904 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.904 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.904 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:38.904 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.576634+0000", 2021-02-10T04:28:38.904 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.0663000000000001e-05 2021-02-10T04:28:38.904 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.904 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.904 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:38.904 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.576665+0000", 2021-02-10T04:28:38.904 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.1020000000000002e-06 2021-02-10T04:28:38.904 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.904 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.904 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:38.905 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.576671+0000", 2021-02-10T04:28:38.905 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1080000000000001e-05 2021-02-10T04:28:38.905 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.905 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.905 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:38.905 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.576682+0000", 2021-02-10T04:28:38.905 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4240000000000003e-06 2021-02-10T04:28:38.905 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.905 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.905 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:38.905 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.576687+0000", 2021-02-10T04:28:38.905 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001175059 2021-02-10T04:28:38.905 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.905 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.906 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:38.906 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.577862+0000", 2021-02-10T04:28:38.906 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.4877999999999998e-05 2021-02-10T04:28:38.906 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.906 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.906 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for rw locks", 2021-02-10T04:28:38.906 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.577927+0000", 2021-02-10T04:28:38.906 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0016504639999999999 2021-02-10T04:28:38.906 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.906 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.906 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:38.906 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.579577+0000", 2021-02-10T04:28:38.906 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.0105000000000001e-05 2021-02-10T04:28:38.907 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.907 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.907 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:38.907 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.579637+0000", 2021-02-10T04:28:38.907 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.2141e-05 2021-02-10T04:28:38.907 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.907 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.907 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:38.907 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.579699+0000", 2021-02-10T04:28:38.907 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003064916 2021-02-10T04:28:38.907 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.907 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:38.907 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.907 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.908 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.908 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:189 7.b 7:d422733f:::smithi08575397-48:head [write 789929~634821,stat] snapc 0=[] ondisk+write+known_if_redirected e98)", 2021-02-10T04:28:38.908 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:28:34.656030+0000", 2021-02-10T04:28:38.908 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 4.1676455729999997, 2021-02-10T04:28:38.908 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0028550530000000002, 2021-02-10T04:28:38.908 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:38.908 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:28:38.908 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:38.908 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:28:38.908 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:28:38.908 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 189 2021-02-10T04:28:38.908 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.908 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:38.909 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.909 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:38.909 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.656030+0000", 2021-02-10T04:28:38.909 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:38.909 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.909 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.909 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:38.909 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.656030+0000", 2021-02-10T04:28:38.909 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00017123799999999999 2021-02-10T04:28:38.909 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.909 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.909 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:38.909 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.656202+0000", 2021-02-10T04:28:38.910 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.696e-06 2021-02-10T04:28:38.910 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.910 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.910 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:38.910 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.656211+0000", 2021-02-10T04:28:38.910 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.6495e-05 2021-02-10T04:28:38.910 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.910 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.910 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:38.910 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.656228+0000", 2021-02-10T04:28:38.910 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.8090000000000004e-06 2021-02-10T04:28:38.910 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.910 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.910 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:38.911 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.656238+0000", 2021-02-10T04:28:38.911 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9641000000000001e-05 2021-02-10T04:28:38.911 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.911 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.911 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:38.911 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.656287+0000", 2021-02-10T04:28:38.911 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00011889599999999999 2021-02-10T04:28:38.911 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.911 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.911 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:38.911 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.656406+0000", 2021-02-10T04:28:38.911 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00012167000000000001 2021-02-10T04:28:38.911 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.912 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.912 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for subops from 5,6", 2021-02-10T04:28:38.912 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.656528+0000", 2021-02-10T04:28:38.912 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00099613400000000008 2021-02-10T04:28:38.912 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.912 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.912 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op_commit", 2021-02-10T04:28:38.912 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.657524+0000", 2021-02-10T04:28:38.912 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001245033 2021-02-10T04:28:38.912 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.912 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.912 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:38.912 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.658769+0000", 2021-02-10T04:28:38.912 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.7873000000000001e-05 2021-02-10T04:28:38.913 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.913 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.913 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:38.913 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.658807+0000", 2021-02-10T04:28:38.913 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0609999999999998e-05 2021-02-10T04:28:38.913 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.913 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.913 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:28:38.913 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.658857+0000", 2021-02-10T04:28:38.913 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.7957999999999999e-05 2021-02-10T04:28:38.913 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.913 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.913 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:38.913 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.658885+0000", 2021-02-10T04:28:38.914 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0028550530000000002 2021-02-10T04:28:38.914 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.914 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:38.914 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.914 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.914 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.914 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:190 7.b 7:d422733f:::smithi08575397-48:head [write 2050536~739393] snapc 0=[] ondisk+write+known_if_redirected e98)", 2021-02-10T04:28:38.914 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:28:34.663764+0000", 2021-02-10T04:28:38.914 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 4.1599118900000001, 2021-02-10T04:28:38.914 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003354849, 2021-02-10T04:28:38.914 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:38.914 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:28:38.914 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:38.915 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:28:38.915 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:28:38.915 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 190 2021-02-10T04:28:38.915 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.915 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:38.915 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.915 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:38.915 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.663764+0000", 2021-02-10T04:28:38.915 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:38.915 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.915 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.915 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:38.915 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.663764+0000", 2021-02-10T04:28:38.916 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00022767600000000001 2021-02-10T04:28:38.916 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.916 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.916 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:38.916 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.663992+0000", 2021-02-10T04:28:38.916 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.4371000000000001e-05 2021-02-10T04:28:38.916 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.916 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.916 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:38.916 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.664016+0000", 2021-02-10T04:28:38.916 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.1103000000000001e-05 2021-02-10T04:28:38.916 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.916 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.916 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:38.917 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.664037+0000", 2021-02-10T04:28:38.917 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.3725e-05 2021-02-10T04:28:38.917 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.917 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.917 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:38.917 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.664051+0000", 2021-02-10T04:28:38.917 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.0463e-05 2021-02-10T04:28:38.917 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.917 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.917 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:38.917 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.664111+0000", 2021-02-10T04:28:38.917 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000112291 2021-02-10T04:28:38.917 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.917 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.918 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:38.918 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.664224+0000", 2021-02-10T04:28:38.918 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000170008 2021-02-10T04:28:38.918 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.918 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.918 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for subops from 5,6", 2021-02-10T04:28:38.918 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.664394+0000", 2021-02-10T04:28:38.918 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0010865 2021-02-10T04:28:38.918 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.918 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.918 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op_commit", 2021-02-10T04:28:38.918 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.665480+0000", 2021-02-10T04:28:38.918 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0014837260000000001 2021-02-10T04:28:38.919 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.919 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.919 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:38.919 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.666964+0000", 2021-02-10T04:28:38.919 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.5918000000000003e-05 2021-02-10T04:28:38.919 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.919 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.919 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:38.919 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.667010+0000", 2021-02-10T04:28:38.919 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.1950000000000001e-05 2021-02-10T04:28:38.919 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.919 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.919 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:28:38.919 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.667072+0000", 2021-02-10T04:28:38.920 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.7117999999999998e-05 2021-02-10T04:28:38.920 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.920 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.920 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:38.920 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.667119+0000", 2021-02-10T04:28:38.920 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003354849 2021-02-10T04:28:38.920 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.920 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:38.920 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.920 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.920 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.920 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:191 7.b 7:d422733f:::smithi08575397-48:head [setxattr _header (56) in=7b,truncate 2789929] snapc 0=[] ondisk+write+known_if_redirected e98)", 2021-02-10T04:28:38.920 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:28:34.664099+0000", 2021-02-10T04:28:38.921 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 4.1595766940000001, 2021-02-10T04:28:38.921 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031779109999999998, 2021-02-10T04:28:38.921 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:38.921 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:28:38.921 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:38.921 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:28:38.921 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:28:38.921 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 191 2021-02-10T04:28:38.921 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.921 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:38.921 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.921 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:38.921 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.664099+0000", 2021-02-10T04:28:38.921 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:38.922 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.922 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.922 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:38.922 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.664099+0000", 2021-02-10T04:28:38.922 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.464e-05 2021-02-10T04:28:38.922 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.922 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.922 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:38.922 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.664144+0000", 2021-02-10T04:28:38.922 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.9510000000000003e-06 2021-02-10T04:28:38.922 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.922 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.922 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:38.923 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.664150+0000", 2021-02-10T04:28:38.923 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.3188e-05 2021-02-10T04:28:38.923 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.923 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.923 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:38.923 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.664163+0000", 2021-02-10T04:28:38.923 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.2830000000000001e-06 2021-02-10T04:28:38.923 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.923 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.923 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:38.923 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.664168+0000", 2021-02-10T04:28:38.923 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00088391700000000001 2021-02-10T04:28:38.923 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.924 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.924 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:38.924 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.665052+0000", 2021-02-10T04:28:38.924 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.8003999999999995e-05 2021-02-10T04:28:38.924 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.924 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.924 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:38.924 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.665120+0000", 2021-02-10T04:28:38.924 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000109202 2021-02-10T04:28:38.924 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.924 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.924 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for subops from 5,6", 2021-02-10T04:28:38.924 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.665229+0000", 2021-02-10T04:28:38.924 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00034329100000000002 2021-02-10T04:28:38.924 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.925 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.925 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op_commit", 2021-02-10T04:28:38.925 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.665573+0000", 2021-02-10T04:28:38.925 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0015959660000000001 2021-02-10T04:28:38.925 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.925 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.925 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:38.925 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.667169+0000", 2021-02-10T04:28:38.925 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.8940000000000003e-05 2021-02-10T04:28:38.925 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.925 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.925 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:38.925 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.667208+0000", 2021-02-10T04:28:38.925 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.7648000000000003e-05 2021-02-10T04:28:38.926 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.926 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.926 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:28:38.926 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.667245+0000", 2021-02-10T04:28:38.926 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.1881000000000002e-05 2021-02-10T04:28:38.926 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.926 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.926 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:38.926 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.667277+0000", 2021-02-10T04:28:38.926 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031779109999999998 2021-02-10T04:28:38.926 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.926 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:38.926 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.926 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.927 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.927 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:192 7.b 7:d422733f:::smithi08575397-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e98)", 2021-02-10T04:28:38.927 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:28:34.664203+0000", 2021-02-10T04:28:38.927 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 4.1594731559999998, 2021-02-10T04:28:38.927 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0032211290000000001, 2021-02-10T04:28:38.927 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:38.927 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:28:38.927 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:38.927 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:28:38.927 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:28:38.927 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 192 2021-02-10T04:28:38.927 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.927 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:38.928 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.928 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:38.928 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.664203+0000", 2021-02-10T04:28:38.928 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:38.928 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.928 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.928 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:38.928 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.664203+0000", 2021-02-10T04:28:38.928 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.9842999999999999e-05 2021-02-10T04:28:38.928 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.928 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.928 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:38.928 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.664233+0000", 2021-02-10T04:28:38.928 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.4020000000000001e-06 2021-02-10T04:28:38.929 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.929 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.929 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:38.929 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.664238+0000", 2021-02-10T04:28:38.929 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0383e-05 2021-02-10T04:28:38.929 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.929 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.929 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:38.929 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.664248+0000", 2021-02-10T04:28:38.929 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.164e-06 2021-02-10T04:28:38.929 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.929 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.929 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:38.930 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.664254+0000", 2021-02-10T04:28:38.930 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0011573499999999999 2021-02-10T04:28:38.930 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.930 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.930 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:38.930 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.665411+0000", 2021-02-10T04:28:38.930 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.6938000000000003e-05 2021-02-10T04:28:38.930 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.930 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.930 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for rw locks", 2021-02-10T04:28:38.930 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.665468+0000", 2021-02-10T04:28:38.930 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0018365300000000001 2021-02-10T04:28:38.930 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.930 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.931 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:38.931 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.667304+0000", 2021-02-10T04:28:38.931 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.2967999999999998e-05 2021-02-10T04:28:38.931 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.931 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.931 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:38.931 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.667357+0000", 2021-02-10T04:28:38.931 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.6550999999999994e-05 2021-02-10T04:28:38.931 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.931 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.931 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:38.931 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:34.667424+0000", 2021-02-10T04:28:38.931 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0032211290000000001 2021-02-10T04:28:38.932 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.932 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:38.932 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.932 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.932 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.932 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:220 7.b 7:db4ad6c0:::smithi08575397-12:head [write 524385~553166] snapc 3=[3,2] ondisk+write+known_if_redirected e100)", 2021-02-10T04:28:38.932 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:28:36.221465+0000", 2021-02-10T04:28:38.932 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 2.602210881, 2021-02-10T04:28:38.932 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0037070319999999999, 2021-02-10T04:28:38.932 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:38.932 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:28:38.932 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:38.932 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:28:38.933 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:28:38.933 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 220 2021-02-10T04:28:38.933 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.933 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:38.933 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.933 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:38.933 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:36.221465+0000", 2021-02-10T04:28:38.933 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:38.933 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.933 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.933 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:38.933 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:36.221465+0000", 2021-02-10T04:28:38.933 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00028765399999999998 2021-02-10T04:28:38.933 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.934 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.934 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:38.934 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:36.221753+0000", 2021-02-10T04:28:38.934 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.0730000000000005e-06 2021-02-10T04:28:38.934 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.934 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.934 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:38.934 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:36.221761+0000", 2021-02-10T04:28:38.934 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2957e-05 2021-02-10T04:28:38.934 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.934 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.934 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:38.934 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:36.221774+0000", 2021-02-10T04:28:38.934 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.5569999999999996e-06 2021-02-10T04:28:38.935 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.935 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.935 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:38.935 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:36.221782+0000", 2021-02-10T04:28:38.935 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.1413000000000003e-05 2021-02-10T04:28:38.935 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.935 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.935 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:38.935 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:36.221844+0000", 2021-02-10T04:28:38.935 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.5332000000000003e-05 2021-02-10T04:28:38.935 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.935 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.935 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:38.936 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:36.221919+0000", 2021-02-10T04:28:38.936 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00013710600000000001 2021-02-10T04:28:38.936 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.936 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.936 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for subops from 5,6", 2021-02-10T04:28:38.936 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:36.222056+0000", 2021-02-10T04:28:38.936 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0014214869999999999 2021-02-10T04:28:38.936 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.936 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.936 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op_commit", 2021-02-10T04:28:38.936 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:36.223478+0000", 2021-02-10T04:28:38.936 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00150975 2021-02-10T04:28:38.936 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.937 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.937 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:38.937 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:36.224987+0000", 2021-02-10T04:28:38.937 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.5654999999999994e-05 2021-02-10T04:28:38.937 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.937 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.937 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:28:38.937 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:36.225053+0000", 2021-02-10T04:28:38.937 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.5332000000000004e-05 2021-02-10T04:28:38.937 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.937 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.937 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:28:38.937 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:36.225118+0000", 2021-02-10T04:28:38.937 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.3715999999999999e-05 2021-02-10T04:28:38.938 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.938 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.938 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:38.938 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:36.225172+0000", 2021-02-10T04:28:38.938 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0037070319999999999 2021-02-10T04:28:38.938 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.938 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:38.938 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.938 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.938 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.938 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:224 7.b 7:db4ad6c0:::smithi08575397-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e100)", 2021-02-10T04:28:38.938 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:28:36.232106+0000", 2021-02-10T04:28:38.938 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 2.5915695460000001, 2021-02-10T04:28:38.939 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0028149529999999998, 2021-02-10T04:28:38.939 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:38.939 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:28:38.939 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:28:38.939 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:28:38.939 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:28:38.939 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 224 2021-02-10T04:28:38.939 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.939 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:38.939 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.939 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:38.939 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:36.232106+0000", 2021-02-10T04:28:38.939 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:38.940 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.940 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.940 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:38.940 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:36.232106+0000", 2021-02-10T04:28:38.940 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.1180000000000003e-05 2021-02-10T04:28:38.940 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.940 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.940 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:38.940 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:36.232138+0000", 2021-02-10T04:28:38.940 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.75e-06 2021-02-10T04:28:38.940 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.940 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.940 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:38.940 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:36.232143+0000", 2021-02-10T04:28:38.941 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1307e-05 2021-02-10T04:28:38.941 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.941 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.941 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:38.941 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:36.232155+0000", 2021-02-10T04:28:38.941 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.8020000000000003e-06 2021-02-10T04:28:38.941 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.941 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.941 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:38.941 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:36.232161+0000", 2021-02-10T04:28:38.941 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001017446 2021-02-10T04:28:38.941 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.941 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.942 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:38.942 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:36.233179+0000", 2021-02-10T04:28:38.942 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.6339000000000003e-05 2021-02-10T04:28:38.942 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.942 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.942 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for rw locks", 2021-02-10T04:28:38.942 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:36.233245+0000", 2021-02-10T04:28:38.942 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0015505950000000001 2021-02-10T04:28:38.942 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.942 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.942 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:38.942 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:36.234796+0000", 2021-02-10T04:28:38.942 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.7289e-05 2021-02-10T04:28:38.943 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.943 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.943 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:38.943 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:36.234853+0000", 2021-02-10T04:28:38.943 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.8244999999999995e-05 2021-02-10T04:28:38.943 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.943 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.943 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:38.943 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:36.234921+0000", 2021-02-10T04:28:38.943 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0028149529999999998 2021-02-10T04:28:38.943 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.943 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:38.943 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.944 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.944 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.944 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(osd.5.0:68 7.0 e101/93 7:03599452:::smithi08575397-34:2 v 101'13, mlcod=98'9)", 2021-02-10T04:28:38.944 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:28:37.213148+0000", 2021-02-10T04:28:38.944 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 1.610528044, 2021-02-10T04:28:38.944 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0035607450000000001, 2021-02-10T04:28:38.944 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:28:38.944 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:28:38.944 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:28:38.944 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.944 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:28:38.944 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:37.213148+0000", 2021-02-10T04:28:38.944 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:28:38.945 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.945 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.945 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:28:38.945 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:37.213148+0000", 2021-02-10T04:28:38.945 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.1705000000000001e-05 2021-02-10T04:28:38.945 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.945 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.945 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:28:38.945 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:37.213190+0000", 2021-02-10T04:28:38.945 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.4379999999999998e-06 2021-02-10T04:28:38.945 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.945 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.945 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:28:38.946 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:37.213197+0000", 2021-02-10T04:28:38.946 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.3427e-05 2021-02-10T04:28:38.946 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.946 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.946 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:28:38.946 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:37.213210+0000", 2021-02-10T04:28:38.946 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.1336e-05 2021-02-10T04:28:38.946 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.946 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.946 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:28:38.946 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:37.213232+0000", 2021-02-10T04:28:38.946 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.7994000000000002e-05 2021-02-10T04:28:38.947 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.947 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.947 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:28:38.947 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:37.213280+0000", 2021-02-10T04:28:38.947 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.7954000000000001e-05 2021-02-10T04:28:38.947 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.947 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.947 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:28:38.947 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:37.213298+0000", 2021-02-10T04:28:38.947 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0033613240000000002 2021-02-10T04:28:38.947 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.947 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.947 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:28:38.947 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:37.216659+0000", 2021-02-10T04:28:38.948 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9567000000000002e-05 2021-02-10T04:28:38.948 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:28:38.948 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:28:38.948 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:28:38.948 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:37.216709+0000", 2021-02-10T04:28:38.948 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0035607450000000001 2021-02-10T04:28:38.948 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.948 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:38.948 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.948 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:28:38.948 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:28:38.948 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:28:38.949 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:28:38.949 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:28:38.950 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[37263]: debug 2021-02-10T04:28:38.512+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:38.950 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:28:38 smithi164 conmon[33677]: debug 2021-02-10T04:28:38.715+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:38.975 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:39.034 INFO:tasks.rados.rados.0.smithi085.stdout:68: done (1 left) 2021-02-10T04:28:39.034 INFO:tasks.rados.rados.0.smithi085.stdout:69: read oid 2 snap -1 2021-02-10T04:28:39.034 INFO:tasks.rados.rados.0.smithi085.stdout:69: expect deleted 2021-02-10T04:28:39.034 INFO:tasks.rados.rados.0.smithi085.stdout:70: read oid 44 snap -1 2021-02-10T04:28:39.034 INFO:tasks.rados.rados.0.smithi085.stdout:70: expect (ObjNum 43 snap 0 seq_num 43) 2021-02-10T04:28:39.034 INFO:tasks.rados.rados.0.smithi085.stdout:71: read oid 16 snap -1 2021-02-10T04:28:39.034 INFO:tasks.rados.rados.0.smithi085.stdout:71: expect (ObjNum 15 snap 0 seq_num 15) 2021-02-10T04:28:39.034 INFO:tasks.rados.rados.0.smithi085.stdout:72: snap_remove snap 2 2021-02-10T04:28:39.045 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is unset 2021-02-10T04:28:39.441 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[41540]: debug 2021-02-10T04:28:39.441 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[41540]: 2021-02-10T04:28:39.039+0000 7fb94ab61700 -1 osd.0 pg_epoch: 103 pg[7.f( v 101'15 (0'0,101'15] local-lis/les=93/94 n=5 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [0,3,7] r=0 lpr=93 crt=101'15 lcod 98'13 mlcod 98'13 active+clean+snaptrim trimq=[3~1] ps=[2~1]] removing snap head 2021-02-10T04:28:39.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[29319]: cluster 2021-02-10T04:28:38.246764+0000 osd.6 (osd.6) 26 : cluster 2021-02-10T04:28:39.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[29319]: [DBG] 2.1a scrub ok 2021-02-10T04:28:39.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[29319]: audit 2021-02-10T04:28:38.419826+0000 mgr.y (mgr.14140) 334 : audit [DBG] from='client.34658 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:28:39.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:28:39.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[29319]: 28:38.503300+0000 osd.5 (osd.5) 32 : cluster [DBG] 6.7 scrub ok 2021-02-10T04:28:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:28:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[29319]: 02-10T04:28:38.788844+0000 osd.0 (osd.0) 35 : cluster 2021-02-10T04:28:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[29319]: [DBG] 6.10 scrub ok 2021-02-10T04:28:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:28:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[29319]: -10T04:28:39.022782+0000 mon.a (mon.0) 755 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:28:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[29319]: cluster 2021-02-10T04:28:39.022820+0000 mon.a ( 2021-02-10T04:28:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[29319]: mon.0) 756 : cluster [INF] Cluster is now healthy 2021-02-10T04:28:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[29319]: audit 2021-02-10T04:28:39.029437+0000 mon.a ( 2021-02-10T04:28:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[29319]: mon.0) 757 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:28:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:28:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[29319]: :28:39.029679+0000 mon.a (mon.0) 758 : cluster 2021-02-10T04:28:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[29319]: [DBG] osdmap e103: 8 total, 8 up, 8 in 2021-02-10T04:28:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[35961]: cluster 2021-02-10T04:28:38.246764 2021-02-10T04:28:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[35961]: +0000 osd.6 (osd.6) 26 : cluster [DBG] 2.1a scrub ok 2021-02-10T04:28:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[35961]: audit 2021-02-10T04:28:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[35961]: 2021-02-10T04:28:38.419826+0000 mgr.y (mgr.14140) 334 : audit 2021-02-10T04:28:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[35961]: [DBG] from='client.34658 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:28:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:28:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[35961]: 28:38.503300+0000 osd.5 (osd.5) 32 : cluster [DBG] 2021-02-10T04:28:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[35961]: 6.7 scrub ok 2021-02-10T04:28:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[35961]: cluster 2021-02-10T04:28:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[35961]: 2021-02-10T04:28:38.788844+0000 osd.0 (osd.0) 35 : cluster 2021-02-10T04:28:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[35961]: [DBG] 6.10 scrub ok 2021-02-10T04:28:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[35961]: cluster 2021-02-10T04:28:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[35961]: 2021-02-10T04:28 2021-02-10T04:28:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[35961]: :39.022782+0000 mon.a (mon.0) 755 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:28:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:28:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[35961]: 10T04:28:39.022820+0000 mon.a (mon.0) 756 : cluster 2021-02-10T04:28:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[35961]: [INF] Cluster is now healthy 2021-02-10T04:28:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[35961]: audit 2021-02-10T04:28:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[35961]: 2021-02-10T04:28:39.029437+0000 mon.a (mon.0) 757 : 2021-02-10T04:28:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[35961]: audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:28:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[35961]: cluster 2021-02-10T04:28:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[35961]: 2021-02-10T04:28:39.029679+0000 mon.a (mon.0) 758 : 2021-02-10T04:28:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[35961]: cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2021-02-10T04:28:39.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:39 smithi164 conmon[29288]: cluster 2021-02-10T04:28:38.246764+0000 osd.6 (osd.6) 26 : 2021-02-10T04:28:39.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:39 smithi164 conmon[29288]: cluster [DBG] 2.1a scrub ok 2021-02-10T04:28:39.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:39 smithi164 conmon[29288]: audit 2021-02-10T04:28:38.419826+0000 mgr.y (mgr.14140) 334 : audit [DBG] from='client.34658 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:28:39.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:39 smithi164 conmon[29288]: cluster 2021-02-10T04:28:38.503300 2021-02-10T04:28:39.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:39 smithi164 conmon[29288]: +0000 osd.5 (osd.5) 32 : cluster [DBG] 6.7 scrub ok 2021-02-10T04:28:39.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:39 smithi164 conmon[29288]: cluster 2021-02-10T04:28:38.788844+0000 osd.0 2021-02-10T04:28:39.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:39 smithi164 conmon[29288]: (osd.0) 35 : cluster [DBG] 6.10 scrub ok 2021-02-10T04:28:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:39 smithi164 conmon[29288]: cluster 2021-02-10T04:28:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:39 smithi164 conmon[29288]: 2021-02-10T04:28:39.022782+0000 mon.a (mon.0) 755 : cluster [INF] 2021-02-10T04:28:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:39 smithi164 conmon[29288]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:28:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:39 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:28:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:39 smithi164 conmon[29288]: 10T04:28:39.022820+0000 mon.a (mon.0) 756 : cluster [INF] Cluster is now healthy 2021-02-10T04:28:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:39 smithi164 conmon[29288]: audit 2021-02-10T04: 2021-02-10T04:28:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:39 smithi164 conmon[29288]: 28:39.029437+0000 mon.a (mon.0) 757 : audit [INF] 2021-02-10T04:28:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:39 smithi164 conmon[29288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:28:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:39 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:28:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:39 smithi164 conmon[29288]: 04:28:39.029679+0000 mon.a (mon.0) 758 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2021-02-10T04:28:39.940 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:28:39 smithi085 conmon[49106]: debug 2021-02-10T04:28:39.642+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:40.029 INFO:tasks.rados.rados.0.smithi085.stdout:72: done (4 left) 2021-02-10T04:28:40.029 INFO:tasks.rados.rados.0.smithi085.stdout:73: write oid 23 current snap is 3 2021-02-10T04:28:40.029 INFO:tasks.rados.rados.0.smithi085.stdout:66: finishing copy_from to smithi08575397-3 2021-02-10T04:28:40.030 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 12 (ObjNum 51 snap 0 seq_num 51) dirty exists 2021-02-10T04:28:40.030 INFO:tasks.rados.rados.0.smithi085.stdout:73: seq_num 57 ranges {557999=703664,1719003=413684,2632287=325712} 2021-02-10T04:28:40.035 INFO:tasks.rados.rados.0.smithi085.stdout:73: writing smithi08575397-23 from 557999 to 1261663 tid 1 2021-02-10T04:28:40.040 INFO:tasks.rados.rados.0.smithi085.stdout:73: writing smithi08575397-23 from 1719003 to 2132687 tid 2 2021-02-10T04:28:40.045 INFO:tasks.rados.rados.0.smithi085.stdout:73: writing smithi08575397-23 from 2632287 to 2957999 tid 3 2021-02-10T04:28:40.045 INFO:tasks.rados.rados.0.smithi085.stdout:74: copy_from oid 1 from oid 9 current snap is 3 2021-02-10T04:28:40.045 INFO:tasks.rados.rados.0.smithi085.stdout:66: finishing copy_from racing read to smithi08575397-3 2021-02-10T04:28:40.045 INFO:tasks.rados.rados.0.smithi085.stdout:70: expect (ObjNum 43 snap 0 seq_num 43) 2021-02-10T04:28:40.131 INFO:tasks.rados.rados.0.smithi085.stdout:71: expect (ObjNum 15 snap 0 seq_num 15) 2021-02-10T04:28:40.220 INFO:tasks.rados.rados.0.smithi085.stdout:74: finishing copy_from to smithi08575397-1 2021-02-10T04:28:40.220 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2021-02-10T04:28:40.220 INFO:tasks.rados.rados.0.smithi085.stdout:74: finishing copy_from racing read to smithi08575397-1 2021-02-10T04:28:40.220 INFO:tasks.rados.rados.0.smithi085.stdout:73: finishing write tid 1 to smithi08575397-23 2021-02-10T04:28:40.221 INFO:tasks.rados.rados.0.smithi085.stdout:73: finishing write tid 2 to smithi08575397-23 2021-02-10T04:28:40.221 INFO:tasks.rados.rados.0.smithi085.stdout:66: done (5 left) 2021-02-10T04:28:40.221 INFO:tasks.rados.rados.0.smithi085.stdout:69: done (4 left) 2021-02-10T04:28:40.221 INFO:tasks.rados.rados.0.smithi085.stdout:70: done (3 left) 2021-02-10T04:28:40.221 INFO:tasks.rados.rados.0.smithi085.stdout:71: done (2 left) 2021-02-10T04:28:40.221 INFO:tasks.rados.rados.0.smithi085.stdout:74: done (1 left) 2021-02-10T04:28:40.221 INFO:tasks.rados.rados.0.smithi085.stdout:75: copy_from oid 39 from oid 26 current snap is 3 2021-02-10T04:28:40.221 INFO:tasks.rados.rados.0.smithi085.stdout:73: finishing write tid 3 to smithi08575397-23 2021-02-10T04:28:40.221 INFO:tasks.rados.rados.0.smithi085.stdout:73: finishing write tid 4 to smithi08575397-23 2021-02-10T04:28:40.221 INFO:tasks.rados.rados.0.smithi085.stdout:73: finishing write tid 5 to smithi08575397-23 2021-02-10T04:28:40.221 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 10 (ObjNum 57 snap 3 seq_num 57) dirty exists 2021-02-10T04:28:40.221 INFO:tasks.rados.rados.0.smithi085.stdout:73: left oid 23 (ObjNum 57 snap 3 seq_num 57) 2021-02-10T04:28:40.221 INFO:tasks.rados.rados.0.smithi085.stdout:73: done (1 left) 2021-02-10T04:28:40.222 INFO:tasks.rados.rados.0.smithi085.stdout:76: snap_create 2021-02-10T04:28:40.235 INFO:tasks.rados.rados.0.smithi085.stdout:75: finishing copy_from to smithi08575397-39 2021-02-10T04:28:40.235 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 39 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2021-02-10T04:28:40.235 INFO:tasks.rados.rados.0.smithi085.stdout:75: finishing copy_from racing read to smithi08575397-39 2021-02-10T04:28:40.306 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:28:40.427 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:28:40.455 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:40.466 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:28:40.466 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[29319]: 28:39.128772+0000 osd.1 (osd.1) 36 : cluster [DBG] 2.9 deep-scrub ok 2021-02-10T04:28:40.467 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[29319]: cluster 2021-02-10T04:28:40.467 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[29319]: 2021-02-10T04:28:39.388878+0000 osd.7 2021-02-10T04:28:40.467 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[29319]: (osd.7) 34 : cluster [DBG] 2.27 scrub ok 2021-02-10T04:28:40.467 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:28:40.467 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[29319]: -02-10T04:28:39.690430+0000 2021-02-10T04:28:40.467 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[29319]: osd.2 (osd.2) 30 : cluster [DBG] 2.1 scrub ok 2021-02-10T04:28:40.467 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[29319]: 2021-02-10T04:28:40.467 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:28:40.467 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[29319]: -02-10T04:28:39.993330 2021-02-10T04:28:40.467 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 335 : cluster [DBG] 2021-02-10T04:28:40.467 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[29319]: pgmap v304: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 117 MiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.5 MiB/s rd, 8.8 MiB/s wr, 51 op/s 2021-02-10T04:28:40.468 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[29319]: cluster 2021-02-10T04:28:40 2021-02-10T04:28:40.468 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[29319]: .028968+0000 mon.a (mon.0) 759 : 2021-02-10T04:28:40.468 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[29319]: cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2021-02-10T04:28:40.469 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[35961]: cluster 2021-02-10T04:28:39. 2021-02-10T04:28:40.469 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[35961]: 128772+0000 osd.1 (osd.1) 36 : cluster [DBG] 2.9 deep-scrub ok 2021-02-10T04:28:40.469 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:28:40.469 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[35961]: -10T04:28:39.388878+0000 2021-02-10T04:28:40.469 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[35961]: osd.7 (osd.7) 34 : cluster [DBG] 2.27 scrub ok 2021-02-10T04:28:40.469 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[35961]: 2021-02-10T04:28:40.469 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:28:40.469 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[35961]: :28:39.690430+0000 osd.2 (osd. 2021-02-10T04:28:40.469 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[35961]: 2) 30 : cluster [DBG] 2.1 scrub ok 2021-02-10T04:28:40.469 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[35961]: cluster 2021-02-10T04:28:40.470 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[35961]: 2021-02-10T04:28:39. 2021-02-10T04:28:40.470 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[35961]: 993330+0000 mgr.y (mgr.14140) 335 : cluster 2021-02-10T04:28:40.470 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[35961]: [DBG] pgmap v304: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 117 MiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.5 MiB/s rd, 8.8 MiB/s wr, 51 op/s 2021-02-10T04:28:40.470 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:28:40.470 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[35961]: 10T04:28:40.028968+0000 mon.a 2021-02-10T04:28:40.471 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:40 smithi085 conmon[35961]: (mon.0) 759 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2021-02-10T04:28:40.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:40 smithi164 conmon[29288]: cluster 2021-02-10T04:28:39.128772+0000 osd.1 (osd.1 2021-02-10T04:28:40.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:40 smithi164 conmon[29288]: ) 36 : cluster [DBG] 2.9 deep-scrub ok 2021-02-10T04:28:40.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:40 smithi164 conmon[29288]: cluster 2021-02-10T04:28:39.388878+0000 osd.7 (osd.7) 34 : cluster [DBG] 2.27 scrub ok 2021-02-10T04:28:40.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:40 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:28:40.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:40 smithi164 conmon[29288]: T04:28:39.690430+0000 osd.2 (osd.2) 30 2021-02-10T04:28:40.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:40 smithi164 conmon[29288]: : cluster [DBG] 2.1 scrub ok 2021-02-10T04:28:40.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:40 smithi164 conmon[29288]: cluster 2021-02-10T04:28:39.993330 2021-02-10T04:28:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:40 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 335 : cluster [DBG] pgmap v304: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 117 MiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.5 MiB/s rd, 8.8 MiB/s wr, 51 op/s 2021-02-10T04:28:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:40 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:28:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:40 smithi164 conmon[29288]: T04:28:40.028968+0000 mon.a (mon 2021-02-10T04:28:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:40 smithi164 conmon[29288]: .0) 759 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2021-02-10T04:28:40.948 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2021-02-10T04:28:38.022929+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":71690132,"num_objects":387,"num_object_clones":1,"num_object_copies":1161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":387,"num_whiteouts":0,"num_read":1701,"num_read_kb":1139,"num_write":884,"num_write_kb":36322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":2080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":899,"ondisk_log_size":899,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":153412,"kb_used_data":87332,"kb_used_omap":0,"kb_used_meta":65664,"kb_avail":749807804,"statfs":{"total":767960285184,"available":767803191296,"internally_reserved":0,"allocated":89427968,"data_stored":84013892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":67239936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":71664274,"num_objects":30,"num_object_clones":1,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":61,"num_read_kb":31,"num_write":125,"num_write_kb":36145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000814"},"pg_stats":[{"pgid":"2.29","version":"0'0","reported_seq":"16","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.379078+0000","last_change":"2021-02-10T04:28:23.127331+0000","last_active":"2021-02-10T04:28:31.379078+0000","last_peered":"2021-02-10T04:28:31.379078+0000","last_clean":"2021-02-10T04:28:31.379078+0000","last_became_active":"2021-02-10T04:28:23.126795+0000","last_became_peered":"2021-02-10T04:28:23.126795+0000","last_unstale":"2021-02-10T04:28:31.379078+0000","last_undegraded":"2021-02-10T04:28:31.379078+0000","last_fullsized":"2021-02-10T04:28:31.379078+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:13.388827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:13.388827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":"16","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.170801+0000","last_change":"2021-02-10T04:28:23.129809+0000","last_active":"2021-02-10T04:28:31.170801+0000","last_peered":"2021-02-10T04:28:31.170801+0000","last_clean":"2021-02-10T04:28:31.170801+0000","last_became_active":"2021-02-10T04:28:23.129469+0000","last_became_peered":"2021-02-10T04:28:23.129469+0000","last_unstale":"2021-02-10T04:28:31.170801+0000","last_undegraded":"2021-02-10T04:28:31.170801+0000","last_fullsized":"2021-02-10T04:28:31.170801+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:28:13.486341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":"86","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.192741+0000","last_change":"2021-02-10T04:28:23.469669+0000","last_active":"2021-02-10T04:28:31.192741+0000","last_peered":"2021-02-10T04:28:31.192741+0000","last_clean":"2021-02-10T04:28:31.192741+0000","last_became_active":"2021-02-10T04:28:23.469025+0000","last_became_peered":"2021-02-10T04:28:23.469025+0000","last_unstale":"2021-02-10T04:28:31.192741+0000","last_undegraded":"2021-02-10T04:28:31.192741+0000","last_fullsized":"2021-02-10T04:28:31.192741+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:11.234741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:11.234741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.26","version":"53'1","reported_seq":"16","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.192457+0000","last_change":"2021-02-10T04:28:23.125805+0000","last_active":"2021-02-10T04:28:31.192457+0000","last_peered":"2021-02-10T04:28:31.192457+0000","last_clean":"2021-02-10T04:28:31.192457+0000","last_became_active":"2021-02-10T04:28:23.125344+0000","last_became_peered":"2021-02-10T04:28:23.125344+0000","last_unstale":"2021-02-10T04:28:31.192457+0000","last_undegraded":"2021-02-10T04:28:31.192457+0000","last_fullsized":"2021-02-10T04:28:31.192457+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:27:25.026534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:25.026534+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":423,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.25","version":"53'2","reported_seq":"36","reported_epoch":"99","state":"active+clean","last_fresh":"2021-02-10T04:28:35.419002+0000","last_change":"2021-02-10T04:28:23.518785+0000","last_active":"2021-02-10T04:28:35.419002+0000","last_peered":"2021-02-10T04:28:35.419002+0000","last_clean":"2021-02-10T04:28:35.419002+0000","last_became_active":"2021-02-10T04:28:23.467770+0000","last_became_peered":"2021-02-10T04:28:23.467770+0000","last_unstale":"2021-02-10T04:28:35.419002+0000","last_undegraded":"2021-02-10T04:28:35.419002+0000","last_fullsized":"2021-02-10T04:28:35.419002+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:26:57.084864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:57.084864+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":"16","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.379682+0000","last_change":"2021-02-10T04:28:23.126680+0000","last_active":"2021-02-10T04:28:31.379682+0000","last_peered":"2021-02-10T04:28:31.379682+0000","last_clean":"2021-02-10T04:28:31.379682+0000","last_became_active":"2021-02-10T04:28:23.126368+0000","last_became_peered":"2021-02-10T04:28:23.126368+0000","last_unstale":"2021-02-10T04:28:31.379682+0000","last_undegraded":"2021-02-10T04:28:31.379682+0000","last_fullsized":"2021-02-10T04:28:31.379682+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:04.024848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:04.024848+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":"16","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.170883+0000","last_change":"2021-02-10T04:28:23.129967+0000","last_active":"2021-02-10T04:28:31.170883+0000","last_peered":"2021-02-10T04:28:31.170883+0000","last_clean":"2021-02-10T04:28:31.170883+0000","last_became_active":"2021-02-10T04:28:23.129523+0000","last_became_peered":"2021-02-10T04:28:23.129523+0000","last_unstale":"2021-02-10T04:28:31.170883+0000","last_undegraded":"2021-02-10T04:28:31.170883+0000","last_fullsized":"2021-02-10T04:28:31.170883+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:55.385094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:55.385094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":"72","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.379289+0000","last_change":"2021-02-10T04:28:23.130371+0000","last_active":"2021-02-10T04:28:31.379289+0000","last_peered":"2021-02-10T04:28:31.379289+0000","last_clean":"2021-02-10T04:28:31.379289+0000","last_became_active":"2021-02-10T04:28:23.129832+0000","last_became_peered":"2021-02-10T04:28:23.129832+0000","last_unstale":"2021-02-10T04:28:31.379289+0000","last_undegraded":"2021-02-10T04:28:31.379289+0000","last_fullsized":"2021-02-10T04:28:31.379289+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:06.934282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:06.934282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":"85","reported_epoch":"98","state":"active+clean","last_fresh":"2021-02-10T04:28:34.050390+0000","last_change":"2021-02-10T04:28:23.126887+0000","last_active":"2021-02-10T04:28:34.050390+0000","last_peered":"2021-02-10T04:28:34.050390+0000","last_clean":"2021-02-10T04:28:34.050390+0000","last_became_active":"2021-02-10T04:28:23.126501+0000","last_became_peered":"2021-02-10T04:28:23.126501+0000","last_unstale":"2021-02-10T04:28:34.050390+0000","last_undegraded":"2021-02-10T04:28:34.050390+0000","last_fullsized":"2021-02-10T04:28:34.050390+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:44.556741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:44.556741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.20","version":"0'0","reported_seq":"78","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.190875+0000","last_change":"2021-02-10T04:28:23.467965+0000","last_active":"2021-02-10T04:28:31.190875+0000","last_peered":"2021-02-10T04:28:31.190875+0000","last_clean":"2021-02-10T04:28:31.190875+0000","last_became_active":"2021-02-10T04:28:23.467620+0000","last_became_peered":"2021-02-10T04:28:23.467620+0000","last_unstale":"2021-02-10T04:28:31.190875+0000","last_undegraded":"2021-02-10T04:28:31.190875+0000","last_fullsized":"2021-02-10T04:28:31.190875+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:08.192772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:08.192772+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"51","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.170963+0000","last_change":"2021-02-10T04:27:07.339624+0000","last_active":"2021-02-10T04:28:31.170963+0000","last_peered":"2021-02-10T04:28:31.170963+0000","last_clean":"2021-02-10T04:28:31.170963+0000","last_became_active":"2021-02-10T04:25:53.150917+0000","last_became_peered":"2021-02-10T04:25:53.150917+0000","last_unstale":"2021-02-10T04:28:31.170963+0000","last_undegraded":"2021-02-10T04:28:31.170963+0000","last_fullsized":"2021-02-10T04:28:31.170963+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:07.339551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:07.339551+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"56","reported_epoch":"99","state":"active+clean","last_fresh":"2021-02-10T04:28:35.418327+0000","last_change":"2021-02-10T04:27:49.628172+0000","last_active":"2021-02-10T04:28:35.418327+0000","last_peered":"2021-02-10T04:28:35.418327+0000","last_clean":"2021-02-10T04:28:35.418327+0000","last_became_active":"2021-02-10T04:25:51.029110+0000","last_became_peered":"2021-02-10T04:25:51.029110+0000","last_unstale":"2021-02-10T04:28:35.418327+0000","last_undegraded":"2021-02-10T04:28:35.418327+0000","last_fullsized":"2021-02-10T04:28:35.418327+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:49.628090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:49.628090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"72","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.451785+0000","last_change":"2021-02-10T04:27:48.635827+0000","last_active":"2021-02-10T04:28:31.451785+0000","last_peered":"2021-02-10T04:28:31.451785+0000","last_clean":"2021-02-10T04:28:31.451785+0000","last_became_active":"2021-02-10T04:25:36.336112+0000","last_became_peered":"2021-02-10T04:25:36.336112+0000","last_unstale":"2021-02-10T04:28:31.451785+0000","last_undegraded":"2021-02-10T04:28:31.451785+0000","last_fullsized":"2021-02-10T04:28:31.451785+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:48.635765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:48.635765+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"63'44","reported_seq":"150","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.171542+0000","last_change":"2021-02-10T04:27:49.269619+0000","last_active":"2021-02-10T04:28:31.171542+0000","last_peered":"2021-02-10T04:28:31.171542+0000","last_clean":"2021-02-10T04:28:31.171542+0000","last_became_active":"2021-02-10T04:25:49.008451+0000","last_became_peered":"2021-02-10T04:25:49.008451+0000","last_unstale":"2021-02-10T04:28:31.171542+0000","last_undegraded":"2021-02-10T04:28:31.171542+0000","last_fullsized":"2021-02-10T04:28:31.171542+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'44","last_scrub_stamp":"2021-02-10T04:27:49.269539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:49.269539+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":89,"num_read_kb":66,"num_write":45,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"54","reported_epoch":"99","state":"active+clean","last_fresh":"2021-02-10T04:28:35.418585+0000","last_change":"2021-02-10T04:27:51.679613+0000","last_active":"2021-02-10T04:28:35.418585+0000","last_peered":"2021-02-10T04:28:35.418585+0000","last_clean":"2021-02-10T04:28:35.418585+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:28:35.418585+0000","last_undegraded":"2021-02-10T04:28:35.418585+0000","last_fullsized":"2021-02-10T04:28:35.418585+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:51.679515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:51.679515+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"53","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:30.689863+0000","last_change":"2021-02-10T04:26:54.007373+0000","last_active":"2021-02-10T04:28:30.689863+0000","last_peered":"2021-02-10T04:28:30.689863+0000","last_clean":"2021-02-10T04:28:30.689863+0000","last_became_active":"2021-02-10T04:25:51.030405+0000","last_became_peered":"2021-02-10T04:25:51.030405+0000","last_unstale":"2021-02-10T04:28:30.689863+0000","last_undegraded":"2021-02-10T04:28:30.689863+0000","last_fullsized":"2021-02-10T04:28:30.689863+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:54.007310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:26:54.007310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"62","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.172700+0000","last_change":"2021-02-10T04:27:13.332288+0000","last_active":"2021-02-10T04:28:31.172700+0000","last_peered":"2021-02-10T04:28:31.172700+0000","last_clean":"2021-02-10T04:28:31.172700+0000","last_became_active":"2021-02-10T04:25:34.327162+0000","last_became_peered":"2021-02-10T04:25:34.327162+0000","last_unstale":"2021-02-10T04:28:31.172700+0000","last_undegraded":"2021-02-10T04:28:31.172700+0000","last_fullsized":"2021-02-10T04:28:31.172700+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:27:13.332224+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"78'32","reported_seq":"117","reported_epoch":"99","state":"active+clean","last_fresh":"2021-02-10T04:28:35.520600+0000","last_change":"2021-02-10T04:28:35.520489+0000","last_active":"2021-02-10T04:28:35.520600+0000","last_peered":"2021-02-10T04:28:35.520600+0000","last_clean":"2021-02-10T04:28:35.520600+0000","last_became_active":"2021-02-10T04:25:49.010481+0000","last_became_peered":"2021-02-10T04:25:49.010481+0000","last_unstale":"2021-02-10T04:28:35.520600+0000","last_undegraded":"2021-02-10T04:28:35.520600+0000","last_fullsized":"2021-02-10T04:28:35.520600+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'32","last_scrub_stamp":"2021-02-10T04:28:35.520410+0000","last_deep_scrub":"78'32","last_deep_scrub_stamp":"2021-02-10T04:27:09.170428+0000","last_clean_scrub_stamp":"2021-02-10T04:28:35.520410+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":47,"num_read_kb":28,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"51","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:29.865148+0000","last_change":"2021-02-10T04:27:07.962210+0000","last_active":"2021-02-10T04:28:29.865148+0000","last_peered":"2021-02-10T04:28:29.865148+0000","last_clean":"2021-02-10T04:28:29.865148+0000","last_became_active":"2021-02-10T04:25:53.042601+0000","last_became_peered":"2021-02-10T04:25:53.042601+0000","last_unstale":"2021-02-10T04:28:29.865148+0000","last_undegraded":"2021-02-10T04:28:29.865148+0000","last_fullsized":"2021-02-10T04:28:29.865148+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:07.962115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:07.962115+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"54","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.191347+0000","last_change":"2021-02-10T04:27:29.251059+0000","last_active":"2021-02-10T04:28:31.191347+0000","last_peered":"2021-02-10T04:28:31.191347+0000","last_clean":"2021-02-10T04:28:31.191347+0000","last_became_active":"2021-02-10T04:25:51.029286+0000","last_became_peered":"2021-02-10T04:25:51.029286+0000","last_unstale":"2021-02-10T04:28:31.191347+0000","last_undegraded":"2021-02-10T04:28:31.191347+0000","last_fullsized":"2021-02-10T04:28:31.191347+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:29.250978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:29.250978+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"66","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.191973+0000","last_change":"2021-02-10T04:27:50.297737+0000","last_active":"2021-02-10T04:28:31.191973+0000","last_peered":"2021-02-10T04:28:31.191973+0000","last_clean":"2021-02-10T04:28:31.191973+0000","last_became_active":"2021-02-10T04:25:36.344477+0000","last_became_peered":"2021-02-10T04:25:36.344477+0000","last_unstale":"2021-02-10T04:28:31.191973+0000","last_undegraded":"2021-02-10T04:28:31.191973+0000","last_fullsized":"2021-02-10T04:28:31.191973+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:27:50.297650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"63'12","reported_seq":"85","reported_epoch":"99","state":"active+clean","last_fresh":"2021-02-10T04:28:35.419162+0000","last_change":"2021-02-10T04:27:37.441260+0000","last_active":"2021-02-10T04:28:35.419162+0000","last_peered":"2021-02-10T04:28:35.419162+0000","last_clean":"2021-02-10T04:28:35.419162+0000","last_became_active":"2021-02-10T04:25:49.013196+0000","last_became_peered":"2021-02-10T04:25:49.013196+0000","last_unstale":"2021-02-10T04:28:35.419162+0000","last_undegraded":"2021-02-10T04:28:35.419162+0000","last_fullsized":"2021-02-10T04:28:35.419162+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2021-02-10T04:27:37.441168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:37.441168+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"52","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.451277+0000","last_change":"2021-02-10T04:27:33.628099+0000","last_active":"2021-02-10T04:28:31.451277+0000","last_peered":"2021-02-10T04:28:31.451277+0000","last_clean":"2021-02-10T04:28:31.451277+0000","last_became_active":"2021-02-10T04:25:53.039844+0000","last_became_peered":"2021-02-10T04:25:53.039844+0000","last_unstale":"2021-02-10T04:28:31.451277+0000","last_undegraded":"2021-02-10T04:28:31.451277+0000","last_fullsized":"2021-02-10T04:28:31.451277+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:33.628038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:33.628038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"53","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:29.866476+0000","last_change":"2021-02-10T04:27:04.977415+0000","last_active":"2021-02-10T04:28:29.866476+0000","last_peered":"2021-02-10T04:28:29.866476+0000","last_clean":"2021-02-10T04:28:29.866476+0000","last_became_active":"2021-02-10T04:25:51.025488+0000","last_became_peered":"2021-02-10T04:25:51.025488+0000","last_unstale":"2021-02-10T04:28:29.866476+0000","last_undegraded":"2021-02-10T04:28:29.866476+0000","last_fullsized":"2021-02-10T04:28:29.866476+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:04.977353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:04.977353+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"66","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.191784+0000","last_change":"2021-02-10T04:28:00.398561+0000","last_active":"2021-02-10T04:28:31.191784+0000","last_peered":"2021-02-10T04:28:31.191784+0000","last_clean":"2021-02-10T04:28:31.191784+0000","last_became_active":"2021-02-10T04:25:36.337260+0000","last_became_peered":"2021-02-10T04:25:36.337260+0000","last_unstale":"2021-02-10T04:28:31.191784+0000","last_undegraded":"2021-02-10T04:28:31.191784+0000","last_fullsized":"2021-02-10T04:28:31.191784+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:00.398480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:00.398480+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"63'25","reported_seq":"117","reported_epoch":"99","state":"active+clean","last_fresh":"2021-02-10T04:28:35.418645+0000","last_change":"2021-02-10T04:28:32.565457+0000","last_active":"2021-02-10T04:28:35.418645+0000","last_peered":"2021-02-10T04:28:35.418645+0000","last_clean":"2021-02-10T04:28:35.418645+0000","last_became_active":"2021-02-10T04:25:49.000911+0000","last_became_peered":"2021-02-10T04:25:49.000911+0000","last_unstale":"2021-02-10T04:28:35.418645+0000","last_undegraded":"2021-02-10T04:28:35.418645+0000","last_fullsized":"2021-02-10T04:28:35.418645+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2021-02-10T04:28:32.565374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:32.565374+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":53,"num_read_kb":37,"num_write":26,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"51","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.172734+0000","last_change":"2021-02-10T04:26:55.332873+0000","last_active":"2021-02-10T04:28:31.172734+0000","last_peered":"2021-02-10T04:28:31.172734+0000","last_clean":"2021-02-10T04:28:31.172734+0000","last_became_active":"2021-02-10T04:25:53.147831+0000","last_became_peered":"2021-02-10T04:25:53.147831+0000","last_unstale":"2021-02-10T04:28:31.172734+0000","last_undegraded":"2021-02-10T04:28:31.172734+0000","last_fullsized":"2021-02-10T04:28:31.172734+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:55.332815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:26:55.332815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"56","reported_epoch":"99","state":"active+clean","last_fresh":"2021-02-10T04:28:35.419232+0000","last_change":"2021-02-10T04:27:48.598953+0000","last_active":"2021-02-10T04:28:35.419232+0000","last_peered":"2021-02-10T04:28:35.419232+0000","last_clean":"2021-02-10T04:28:35.419232+0000","last_became_active":"2021-02-10T04:25:51.018872+0000","last_became_peered":"2021-02-10T04:25:51.018872+0000","last_unstale":"2021-02-10T04:28:35.419232+0000","last_undegraded":"2021-02-10T04:28:35.419232+0000","last_fullsized":"2021-02-10T04:28:35.419232+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:48.598870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:48.598870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"79","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.171029+0000","last_change":"2021-02-10T04:28:30.415204+0000","last_active":"2021-02-10T04:28:31.171029+0000","last_peered":"2021-02-10T04:28:31.171029+0000","last_clean":"2021-02-10T04:28:31.171029+0000","last_became_active":"2021-02-10T04:25:36.336893+0000","last_became_peered":"2021-02-10T04:25:36.336893+0000","last_unstale":"2021-02-10T04:28:31.171029+0000","last_undegraded":"2021-02-10T04:28:31.171029+0000","last_fullsized":"2021-02-10T04:28:31.171029+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:30.415134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:26:46.425288+0000","last_clean_scrub_stamp":"2021-02-10T04:28:30.415134+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"62'22","reported_seq":"101","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.173008+0000","last_change":"2021-02-10T04:27:58.363322+0000","last_active":"2021-02-10T04:28:31.173008+0000","last_peered":"2021-02-10T04:28:31.173008+0000","last_clean":"2021-02-10T04:28:31.173008+0000","last_became_active":"2021-02-10T04:25:49.008997+0000","last_became_peered":"2021-02-10T04:25:49.008997+0000","last_unstale":"2021-02-10T04:28:31.173008+0000","last_undegraded":"2021-02-10T04:28:31.173008+0000","last_fullsized":"2021-02-10T04:28:31.173008+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'22","last_scrub_stamp":"2021-02-10T04:27:58.363248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:58.363248+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":49,"num_read_kb":31,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"54","reported_epoch":"99","state":"active+clean","last_fresh":"2021-02-10T04:28:35.419352+0000","last_change":"2021-02-10T04:27:29.364660+0000","last_active":"2021-02-10T04:28:35.419352+0000","last_peered":"2021-02-10T04:28:35.419352+0000","last_clean":"2021-02-10T04:28:35.419352+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:28:35.419352+0000","last_undegraded":"2021-02-10T04:28:35.419352+0000","last_fullsized":"2021-02-10T04:28:35.419352+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:29.364585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:29.364585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"53","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:30.690861+0000","last_change":"2021-02-10T04:27:27.017142+0000","last_active":"2021-02-10T04:28:30.690861+0000","last_peered":"2021-02-10T04:28:30.690861+0000","last_clean":"2021-02-10T04:28:30.690861+0000","last_became_active":"2021-02-10T04:25:51.029399+0000","last_became_peered":"2021-02-10T04:25:51.029399+0000","last_unstale":"2021-02-10T04:28:30.690861+0000","last_undegraded":"2021-02-10T04:28:30.690861+0000","last_fullsized":"2021-02-10T04:28:30.690861+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:27.017040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:27.017040+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"66","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.379801+0000","last_change":"2021-02-10T04:27:02.190158+0000","last_active":"2021-02-10T04:28:31.379801+0000","last_peered":"2021-02-10T04:28:31.379801+0000","last_clean":"2021-02-10T04:28:31.379801+0000","last_became_active":"2021-02-10T04:25:36.336442+0000","last_became_peered":"2021-02-10T04:25:36.336442+0000","last_unstale":"2021-02-10T04:28:31.379801+0000","last_undegraded":"2021-02-10T04:28:31.379801+0000","last_fullsized":"2021-02-10T04:28:31.379801+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:27:02.190056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:02.190056+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"62'15","reported_seq":"81","reported_epoch":"98","state":"active+clean","last_fresh":"2021-02-10T04:28:34.051726+0000","last_change":"2021-02-10T04:26:56.623190+0000","last_active":"2021-02-10T04:28:34.051726+0000","last_peered":"2021-02-10T04:28:34.051726+0000","last_clean":"2021-02-10T04:28:34.051726+0000","last_became_active":"2021-02-10T04:25:49.010553+0000","last_became_peered":"2021-02-10T04:25:49.010553+0000","last_unstale":"2021-02-10T04:28:34.051726+0000","last_undegraded":"2021-02-10T04:28:34.051726+0000","last_fullsized":"2021-02-10T04:28:34.051726+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:26:56.623109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:26:56.623109+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"51","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:30.690718+0000","last_change":"2021-02-10T04:27:26.029923+0000","last_active":"2021-02-10T04:28:30.690718+0000","last_peered":"2021-02-10T04:28:30.690718+0000","last_clean":"2021-02-10T04:28:30.690718+0000","last_became_active":"2021-02-10T04:25:53.041098+0000","last_became_peered":"2021-02-10T04:25:53.041098+0000","last_unstale":"2021-02-10T04:28:30.690718+0000","last_undegraded":"2021-02-10T04:28:30.690718+0000","last_fullsized":"2021-02-10T04:28:30.690718+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:26.029856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:26.029856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"54","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.451859+0000","last_change":"2021-02-10T04:27:27.564268+0000","last_active":"2021-02-10T04:28:31.451859+0000","last_peered":"2021-02-10T04:28:31.451859+0000","last_clean":"2021-02-10T04:28:31.451859+0000","last_became_active":"2021-02-10T04:25:51.022853+0000","last_became_peered":"2021-02-10T04:25:51.022853+0000","last_unstale":"2021-02-10T04:28:31.451859+0000","last_undegraded":"2021-02-10T04:28:31.451859+0000","last_fullsized":"2021-02-10T04:28:31.451859+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:27.564175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:27.564175+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"62","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.171222+0000","last_change":"2021-02-10T04:27:16.340830+0000","last_active":"2021-02-10T04:28:31.171222+0000","last_peered":"2021-02-10T04:28:31.171222+0000","last_clean":"2021-02-10T04:28:31.171222+0000","last_became_active":"2021-02-10T04:25:34.324981+0000","last_became_peered":"2021-02-10T04:25:34.324981+0000","last_unstale":"2021-02-10T04:28:31.171222+0000","last_undegraded":"2021-02-10T04:28:31.171222+0000","last_fullsized":"2021-02-10T04:28:31.171222+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:16.340724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:16.340724+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"76'35","reported_seq":"119","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.380377+0000","last_change":"2021-02-10T04:28:06.118655+0000","last_active":"2021-02-10T04:28:31.380377+0000","last_peered":"2021-02-10T04:28:31.380377+0000","last_clean":"2021-02-10T04:28:31.380377+0000","last_became_active":"2021-02-10T04:25:49.015760+0000","last_became_peered":"2021-02-10T04:25:49.015760+0000","last_unstale":"2021-02-10T04:28:31.380377+0000","last_undegraded":"2021-02-10T04:28:31.380377+0000","last_fullsized":"2021-02-10T04:28:31.380377+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'35","last_scrub_stamp":"2021-02-10T04:28:06.118574+0000","last_deep_scrub":"76'35","last_deep_scrub_stamp":"2021-02-10T04:26:58.198641+0000","last_clean_scrub_stamp":"2021-02-10T04:28:06.118574+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":50,"num_read_kb":30,"num_write":24,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"52","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.191969+0000","last_change":"2021-02-10T04:27:39.139028+0000","last_active":"2021-02-10T04:28:31.191969+0000","last_peered":"2021-02-10T04:28:31.191969+0000","last_clean":"2021-02-10T04:28:31.191969+0000","last_became_active":"2021-02-10T04:25:53.042279+0000","last_became_peered":"2021-02-10T04:25:53.042279+0000","last_unstale":"2021-02-10T04:28:31.191969+0000","last_undegraded":"2021-02-10T04:28:31.191969+0000","last_fullsized":"2021-02-10T04:28:31.191969+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:39.138947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:39.138947+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"58'2","reported_seq":"89","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:32.734422+0000","last_change":"2021-02-10T04:27:15.041424+0000","last_active":"2021-02-10T04:28:32.734422+0000","last_peered":"2021-02-10T04:28:32.734422+0000","last_clean":"2021-02-10T04:28:32.734422+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:28:32.734422+0000","last_undegraded":"2021-02-10T04:28:32.734422+0000","last_fullsized":"2021-02-10T04:28:32.734422+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:15.041329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:15.041329+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"71","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:29.865230+0000","last_change":"2021-02-10T04:27:04.011149+0000","last_active":"2021-02-10T04:28:29.865230+0000","last_peered":"2021-02-10T04:28:29.865230+0000","last_clean":"2021-02-10T04:28:29.865230+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:28:29.865230+0000","last_undegraded":"2021-02-10T04:28:29.865230+0000","last_fullsized":"2021-02-10T04:28:29.865230+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:04.011053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:04.011053+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"63'22","reported_seq":"103","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.451887+0000","last_change":"2021-02-10T04:27:59.739174+0000","last_active":"2021-02-10T04:28:31.451887+0000","last_peered":"2021-02-10T04:28:31.451887+0000","last_clean":"2021-02-10T04:28:31.451887+0000","last_became_active":"2021-02-10T04:25:49.009177+0000","last_became_peered":"2021-02-10T04:25:49.009177+0000","last_unstale":"2021-02-10T04:28:31.451887+0000","last_undegraded":"2021-02-10T04:28:31.451887+0000","last_fullsized":"2021-02-10T04:28:31.451887+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2021-02-10T04:27:59.739086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:59.739086+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":22,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"51","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.171446+0000","last_change":"2021-02-10T04:27:50.300043+0000","last_active":"2021-02-10T04:28:31.171446+0000","last_peered":"2021-02-10T04:28:31.171446+0000","last_clean":"2021-02-10T04:28:31.171446+0000","last_became_active":"2021-02-10T04:25:53.150845+0000","last_became_peered":"2021-02-10T04:25:53.150845+0000","last_unstale":"2021-02-10T04:28:31.171446+0000","last_undegraded":"2021-02-10T04:28:31.171446+0000","last_fullsized":"2021-02-10T04:28:31.171446+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_clean_scrub_stamp":"2021-02-10T04:27:50.299942+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"54","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.191597+0000","last_change":"2021-02-10T04:27:07.181928+0000","last_active":"2021-02-10T04:28:31.191597+0000","last_peered":"2021-02-10T04:28:31.191597+0000","last_clean":"2021-02-10T04:28:31.191597+0000","last_became_active":"2021-02-10T04:25:51.018491+0000","last_became_peered":"2021-02-10T04:25:51.018491+0000","last_unstale":"2021-02-10T04:28:31.191597+0000","last_undegraded":"2021-02-10T04:28:31.191597+0000","last_fullsized":"2021-02-10T04:28:31.191597+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:07.181854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:07.181854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"66","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.378809+0000","last_change":"2021-02-10T04:26:52.256790+0000","last_active":"2021-02-10T04:28:31.378809+0000","last_peered":"2021-02-10T04:28:31.378809+0000","last_clean":"2021-02-10T04:28:31.378809+0000","last_became_active":"2021-02-10T04:25:35.328828+0000","last_became_peered":"2021-02-10T04:25:35.328828+0000","last_unstale":"2021-02-10T04:28:31.378809+0000","last_undegraded":"2021-02-10T04:28:31.378809+0000","last_fullsized":"2021-02-10T04:28:31.378809+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:26:52.256723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:52.256723+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"63'21","reported_seq":"101","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.382859+0000","last_change":"2021-02-10T04:28:31.382776+0000","last_active":"2021-02-10T04:28:31.382859+0000","last_peered":"2021-02-10T04:28:31.382859+0000","last_clean":"2021-02-10T04:28:31.382859+0000","last_became_active":"2021-02-10T04:25:49.007049+0000","last_became_peered":"2021-02-10T04:25:49.007049+0000","last_unstale":"2021-02-10T04:28:31.382859+0000","last_undegraded":"2021-02-10T04:28:31.382859+0000","last_fullsized":"2021-02-10T04:28:31.382859+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2021-02-10T04:28:31.382671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:31.382671+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":27,"num_write":21,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"52","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.191222+0000","last_change":"2021-02-10T04:27:19.134286+0000","last_active":"2021-02-10T04:28:31.191222+0000","last_peered":"2021-02-10T04:28:31.191222+0000","last_clean":"2021-02-10T04:28:31.191222+0000","last_became_active":"2021-02-10T04:25:53.034771+0000","last_became_peered":"2021-02-10T04:25:53.034771+0000","last_unstale":"2021-02-10T04:28:31.191222+0000","last_undegraded":"2021-02-10T04:28:31.191222+0000","last_fullsized":"2021-02-10T04:28:31.191222+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:27:19.134212+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"54","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.379390+0000","last_change":"2021-02-10T04:27:19.077862+0000","last_active":"2021-02-10T04:28:31.379390+0000","last_peered":"2021-02-10T04:28:31.379390+0000","last_clean":"2021-02-10T04:28:31.379390+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:28:31.379390+0000","last_undegraded":"2021-02-10T04:28:31.379390+0000","last_fullsized":"2021-02-10T04:28:31.379390+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:27:19.077777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"62","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:29.867439+0000","last_change":"2021-02-10T04:27:03.000902+0000","last_active":"2021-02-10T04:28:29.867439+0000","last_peered":"2021-02-10T04:28:29.867439+0000","last_clean":"2021-02-10T04:28:29.867439+0000","last_became_active":"2021-02-10T04:25:34.331787+0000","last_became_peered":"2021-02-10T04:25:34.331787+0000","last_unstale":"2021-02-10T04:28:29.867439+0000","last_undegraded":"2021-02-10T04:28:29.867439+0000","last_fullsized":"2021-02-10T04:28:29.867439+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:03.000821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:03.000821+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"62'12","reported_seq":"108","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.170585+0000","last_change":"2021-02-10T04:27:48.284315+0000","last_active":"2021-02-10T04:28:31.170585+0000","last_peered":"2021-02-10T04:28:31.170585+0000","last_clean":"2021-02-10T04:28:31.170585+0000","last_became_active":"2021-02-10T04:26:39.005616+0000","last_became_peered":"2021-02-10T04:26:39.005616+0000","last_unstale":"2021-02-10T04:28:31.170585+0000","last_undegraded":"2021-02-10T04:28:31.170585+0000","last_fullsized":"2021-02-10T04:28:31.170585+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2021-02-10T04:27:48.284248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:48.284248+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"51","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.171021+0000","last_change":"2021-02-10T04:26:57.309978+0000","last_active":"2021-02-10T04:28:31.171021+0000","last_peered":"2021-02-10T04:28:31.171021+0000","last_clean":"2021-02-10T04:28:31.171021+0000","last_became_active":"2021-02-10T04:25:53.043277+0000","last_became_peered":"2021-02-10T04:25:53.043277+0000","last_unstale":"2021-02-10T04:28:31.171021+0000","last_undegraded":"2021-02-10T04:28:31.171021+0000","last_fullsized":"2021-02-10T04:28:31.171021+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:57.309915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:26:57.309915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"53","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:30.690469+0000","last_change":"2021-02-10T04:27:33.030004+0000","last_active":"2021-02-10T04:28:30.690469+0000","last_peered":"2021-02-10T04:28:30.690469+0000","last_clean":"2021-02-10T04:28:30.690469+0000","last_became_active":"2021-02-10T04:25:51.030952+0000","last_became_peered":"2021-02-10T04:25:51.030952+0000","last_unstale":"2021-02-10T04:28:30.690469+0000","last_undegraded":"2021-02-10T04:28:30.690469+0000","last_fullsized":"2021-02-10T04:28:30.690469+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:27:33.029912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"71","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:30.689790+0000","last_change":"2021-02-10T04:26:52.011165+0000","last_active":"2021-02-10T04:28:30.689790+0000","last_peered":"2021-02-10T04:28:30.689790+0000","last_clean":"2021-02-10T04:28:30.689790+0000","last_became_active":"2021-02-10T04:25:34.330242+0000","last_became_peered":"2021-02-10T04:25:34.330242+0000","last_unstale":"2021-02-10T04:28:30.689790+0000","last_undegraded":"2021-02-10T04:28:30.689790+0000","last_fullsized":"2021-02-10T04:28:30.689790+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:26:52.011096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:52.011096+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"62'15","reported_seq":"85","reported_epoch":"99","state":"active+clean","last_fresh":"2021-02-10T04:28:35.418915+0000","last_change":"2021-02-10T04:27:42.487239+0000","last_active":"2021-02-10T04:28:35.418915+0000","last_peered":"2021-02-10T04:28:35.418915+0000","last_clean":"2021-02-10T04:28:35.418915+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:28:35.418915+0000","last_undegraded":"2021-02-10T04:28:35.418915+0000","last_fullsized":"2021-02-10T04:28:35.418915+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:27:42.487154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:42.487154+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"52","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.451053+0000","last_change":"2021-02-10T04:27:21.548446+0000","last_active":"2021-02-10T04:28:31.451053+0000","last_peered":"2021-02-10T04:28:31.451053+0000","last_clean":"2021-02-10T04:28:31.451053+0000","last_became_active":"2021-02-10T04:25:53.039322+0000","last_became_peered":"2021-02-10T04:25:53.039322+0000","last_unstale":"2021-02-10T04:28:31.451053+0000","last_undegraded":"2021-02-10T04:28:31.451053+0000","last_fullsized":"2021-02-10T04:28:31.451053+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:21.548381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:21.548381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"53","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.171208+0000","last_change":"2021-02-10T04:27:05.297715+0000","last_active":"2021-02-10T04:28:31.171208+0000","last_peered":"2021-02-10T04:28:31.171208+0000","last_clean":"2021-02-10T04:28:31.171208+0000","last_became_active":"2021-02-10T04:25:51.029155+0000","last_became_peered":"2021-02-10T04:25:51.029155+0000","last_unstale":"2021-02-10T04:28:31.171208+0000","last_undegraded":"2021-02-10T04:28:31.171208+0000","last_fullsized":"2021-02-10T04:28:31.171208+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:27:05.297647+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"63","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.379860+0000","last_change":"2021-02-10T04:26:54.256587+0000","last_active":"2021-02-10T04:28:31.379860+0000","last_peered":"2021-02-10T04:28:31.379860+0000","last_clean":"2021-02-10T04:28:31.379860+0000","last_became_active":"2021-02-10T04:25:34.330070+0000","last_became_peered":"2021-02-10T04:25:34.330070+0000","last_unstale":"2021-02-10T04:28:31.379860+0000","last_undegraded":"2021-02-10T04:28:31.379860+0000","last_fullsized":"2021-02-10T04:28:31.379860+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:54.256498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:54.256498+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"62'15","reported_seq":"83","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:30.690418+0000","last_change":"2021-02-10T04:27:40.218910+0000","last_active":"2021-02-10T04:28:30.690418+0000","last_peered":"2021-02-10T04:28:30.690418+0000","last_clean":"2021-02-10T04:28:30.690418+0000","last_became_active":"2021-02-10T04:25:49.015543+0000","last_became_peered":"2021-02-10T04:25:49.015543+0000","last_unstale":"2021-02-10T04:28:30.690418+0000","last_undegraded":"2021-02-10T04:28:30.690418+0000","last_fullsized":"2021-02-10T04:28:30.690418+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:27:40.218828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:40.218828+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"54","reported_epoch":"99","state":"active+clean","last_fresh":"2021-02-10T04:28:35.419772+0000","last_change":"2021-02-10T04:27:28.373194+0000","last_active":"2021-02-10T04:28:35.419772+0000","last_peered":"2021-02-10T04:28:35.419772+0000","last_clean":"2021-02-10T04:28:35.419772+0000","last_became_active":"2021-02-10T04:25:53.031782+0000","last_became_peered":"2021-02-10T04:25:53.031782+0000","last_unstale":"2021-02-10T04:28:35.419772+0000","last_undegraded":"2021-02-10T04:28:35.419772+0000","last_fullsized":"2021-02-10T04:28:35.419772+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:28.373117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:28.373117+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"58'2","reported_seq":"88","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:32.734250+0000","last_change":"2021-02-10T04:27:26.264628+0000","last_active":"2021-02-10T04:28:32.734250+0000","last_peered":"2021-02-10T04:28:32.734250+0000","last_clean":"2021-02-10T04:28:32.734250+0000","last_became_active":"2021-02-10T04:25:51.018300+0000","last_became_peered":"2021-02-10T04:25:51.018300+0000","last_unstale":"2021-02-10T04:28:32.734250+0000","last_undegraded":"2021-02-10T04:28:32.734250+0000","last_fullsized":"2021-02-10T04:28:32.734250+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:26.264560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:26.264560+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"68","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.451512+0000","last_change":"2021-02-10T04:27:39.616474+0000","last_active":"2021-02-10T04:28:31.451512+0000","last_peered":"2021-02-10T04:28:31.451512+0000","last_clean":"2021-02-10T04:28:31.451512+0000","last_became_active":"2021-02-10T04:25:34.321096+0000","last_became_peered":"2021-02-10T04:25:34.321096+0000","last_unstale":"2021-02-10T04:28:31.451512+0000","last_undegraded":"2021-02-10T04:28:31.451512+0000","last_fullsized":"2021-02-10T04:28:31.451512+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:27:39.616381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"63'19","reported_seq":"94","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:29.866121+0000","last_change":"2021-02-10T04:27:11.016733+0000","last_active":"2021-02-10T04:28:29.866121+0000","last_peered":"2021-02-10T04:28:29.866121+0000","last_clean":"2021-02-10T04:28:29.866121+0000","last_became_active":"2021-02-10T04:25:49.010890+0000","last_became_peered":"2021-02-10T04:25:49.010890+0000","last_unstale":"2021-02-10T04:28:29.866121+0000","last_undegraded":"2021-02-10T04:28:29.866121+0000","last_fullsized":"2021-02-10T04:28:29.866121+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2021-02-10T04:27:11.016650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:11.016650+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":26,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"52","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.451572+0000","last_change":"2021-02-10T04:27:08.486638+0000","last_active":"2021-02-10T04:28:31.451572+0000","last_peered":"2021-02-10T04:28:31.451572+0000","last_clean":"2021-02-10T04:28:31.451572+0000","last_became_active":"2021-02-10T04:25:53.045177+0000","last_became_peered":"2021-02-10T04:25:53.045177+0000","last_unstale":"2021-02-10T04:28:31.451572+0000","last_undegraded":"2021-02-10T04:28:31.451572+0000","last_fullsized":"2021-02-10T04:28:31.451572+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:08.486553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:08.486553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"58'2","reported_seq":"88","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.192460+0000","last_change":"2021-02-10T04:27:20.157871+0000","last_active":"2021-02-10T04:28:31.192460+0000","last_peered":"2021-02-10T04:28:31.192460+0000","last_clean":"2021-02-10T04:28:31.192460+0000","last_became_active":"2021-02-10T04:25:51.032328+0000","last_became_peered":"2021-02-10T04:25:51.032328+0000","last_unstale":"2021-02-10T04:28:31.192460+0000","last_undegraded":"2021-02-10T04:28:31.192460+0000","last_fullsized":"2021-02-10T04:28:31.192460+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:20.157772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:20.157772+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"84'7","reported_seq":"82","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:29.865564+0000","last_change":"2021-02-10T04:27:54.402957+0000","last_active":"2021-02-10T04:28:29.865564+0000","last_peered":"2021-02-10T04:28:29.865564+0000","last_clean":"2021-02-10T04:28:29.865564+0000","last_became_active":"2021-02-10T04:25:36.334366+0000","last_became_peered":"2021-02-10T04:25:36.334366+0000","last_unstale":"2021-02-10T04:28:29.865564+0000","last_undegraded":"2021-02-10T04:28:29.865564+0000","last_fullsized":"2021-02-10T04:28:29.865564+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'7","last_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_deep_scrub":"84'7","last_deep_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_clean_scrub_stamp":"2021-02-10T04:27:54.402855+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"63'26","reported_seq":"141","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.171806+0000","last_change":"2021-02-10T04:27:28.213521+0000","last_active":"2021-02-10T04:28:31.171806+0000","last_peered":"2021-02-10T04:28:31.171806+0000","last_clean":"2021-02-10T04:28:31.171806+0000","last_became_active":"2021-02-10T04:26:39.003971+0000","last_became_peered":"2021-02-10T04:26:39.003971+0000","last_unstale":"2021-02-10T04:28:31.171806+0000","last_undegraded":"2021-02-10T04:28:31.171806+0000","last_fullsized":"2021-02-10T04:28:31.171806+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:27:28.213454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:28.213454+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"52","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.191882+0000","last_change":"2021-02-10T04:27:33.192375+0000","last_active":"2021-02-10T04:28:31.191882+0000","last_peered":"2021-02-10T04:28:31.191882+0000","last_clean":"2021-02-10T04:28:31.191882+0000","last_became_active":"2021-02-10T04:25:53.149820+0000","last_became_peered":"2021-02-10T04:25:53.149820+0000","last_unstale":"2021-02-10T04:28:31.191882+0000","last_undegraded":"2021-02-10T04:28:31.191882+0000","last_fullsized":"2021-02-10T04:28:31.191882+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:33.192287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:33.192287+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"53","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:29.866997+0000","last_change":"2021-02-10T04:27:23.236953+0000","last_active":"2021-02-10T04:28:29.866997+0000","last_peered":"2021-02-10T04:28:29.866997+0000","last_clean":"2021-02-10T04:28:29.866997+0000","last_became_active":"2021-02-10T04:25:51.031864+0000","last_became_peered":"2021-02-10T04:25:51.031864+0000","last_unstale":"2021-02-10T04:28:29.866997+0000","last_undegraded":"2021-02-10T04:28:29.866997+0000","last_fullsized":"2021-02-10T04:28:29.866997+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:23.236870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:23.236870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"65","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.379459+0000","last_change":"2021-02-10T04:27:14.083910+0000","last_active":"2021-02-10T04:28:31.379459+0000","last_peered":"2021-02-10T04:28:31.379459+0000","last_clean":"2021-02-10T04:28:31.379459+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:28:31.379459+0000","last_undegraded":"2021-02-10T04:28:31.379459+0000","last_fullsized":"2021-02-10T04:28:31.379459+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:27:14.083841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:14.083841+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"63'13","reported_seq":"107","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.172701+0000","last_change":"2021-02-10T04:28:11.311247+0000","last_active":"2021-02-10T04:28:31.172701+0000","last_peered":"2021-02-10T04:28:31.172701+0000","last_clean":"2021-02-10T04:28:31.172701+0000","last_became_active":"2021-02-10T04:26:39.003832+0000","last_became_peered":"2021-02-10T04:26:39.003832+0000","last_unstale":"2021-02-10T04:28:31.172701+0000","last_undegraded":"2021-02-10T04:28:31.172701+0000","last_fullsized":"2021-02-10T04:28:31.172701+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2021-02-10T04:28:11.311182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:11.311182+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"52","reported_epoch":"98","state":"active+clean","last_fresh":"2021-02-10T04:28:34.051679+0000","last_change":"2021-02-10T04:27:13.573977+0000","last_active":"2021-02-10T04:28:34.051679+0000","last_peered":"2021-02-10T04:28:34.051679+0000","last_clean":"2021-02-10T04:28:34.051679+0000","last_became_active":"2021-02-10T04:25:53.035867+0000","last_became_peered":"2021-02-10T04:25:53.035867+0000","last_unstale":"2021-02-10T04:28:34.051679+0000","last_undegraded":"2021-02-10T04:28:34.051679+0000","last_fullsized":"2021-02-10T04:28:34.051679+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:13.573912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:13.573912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"58","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:33.321917+0000","last_change":"2021-02-10T04:28:33.321856+0000","last_active":"2021-02-10T04:28:33.321917+0000","last_peered":"2021-02-10T04:28:33.321917+0000","last_clean":"2021-02-10T04:28:33.321917+0000","last_became_active":"2021-02-10T04:25:51.030651+0000","last_became_peered":"2021-02-10T04:25:51.030651+0000","last_unstale":"2021-02-10T04:28:33.321917+0000","last_undegraded":"2021-02-10T04:28:33.321917+0000","last_fullsized":"2021-02-10T04:28:33.321917+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:33.321784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:33.321784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"66","reported_epoch":"98","state":"active+clean","last_fresh":"2021-02-10T04:28:34.050411+0000","last_change":"2021-02-10T04:26:57.639364+0000","last_active":"2021-02-10T04:28:34.050411+0000","last_peered":"2021-02-10T04:28:34.050411+0000","last_clean":"2021-02-10T04:28:34.050411+0000","last_became_active":"2021-02-10T04:25:34.329694+0000","last_became_peered":"2021-02-10T04:25:34.329694+0000","last_unstale":"2021-02-10T04:28:34.050411+0000","last_undegraded":"2021-02-10T04:28:34.050411+0000","last_fullsized":"2021-02-10T04:28:34.050411+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:26:57.639281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:26:57.639281+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"63'28","reported_seq":"116","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.451780+0000","last_change":"2021-02-10T04:27:24.523914+0000","last_active":"2021-02-10T04:28:31.451780+0000","last_peered":"2021-02-10T04:28:31.451780+0000","last_clean":"2021-02-10T04:28:31.451780+0000","last_became_active":"2021-02-10T04:25:49.008942+0000","last_became_peered":"2021-02-10T04:25:49.008942+0000","last_unstale":"2021-02-10T04:28:31.451780+0000","last_undegraded":"2021-02-10T04:28:31.451780+0000","last_fullsized":"2021-02-10T04:28:31.451780+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'28","last_scrub_stamp":"2021-02-10T04:27:24.523837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:24.523837+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":62,"num_read_kb":40,"num_write":29,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.a","version":"98'9","reported_seq":"25","reported_epoch":"98","state":"active+clean","last_fresh":"2021-02-10T04:28:34.591906+0000","last_change":"2021-02-10T04:28:25.158297+0000","last_active":"2021-02-10T04:28:34.591906+0000","last_peered":"2021-02-10T04:28:34.591906+0000","last_clean":"2021-02-10T04:28:34.591906+0000","last_became_active":"2021-02-10T04:28:25.157771+0000","last_became_peered":"2021-02-10T04:28:25.157771+0000","last_unstale":"2021-02-10T04:28:34.591906+0000","last_undegraded":"2021-02-10T04:28:34.591906+0000","last_fullsized":"2021-02-10T04:28:34.591906+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6840350,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":2936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"51","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.171281+0000","last_change":"2021-02-10T04:27:27.259500+0000","last_active":"2021-02-10T04:28:31.171281+0000","last_peered":"2021-02-10T04:28:31.171281+0000","last_clean":"2021-02-10T04:28:31.171281+0000","last_became_active":"2021-02-10T04:25:53.047736+0000","last_became_peered":"2021-02-10T04:25:53.047736+0000","last_unstale":"2021-02-10T04:28:31.171281+0000","last_undegraded":"2021-02-10T04:28:31.171281+0000","last_fullsized":"2021-02-10T04:28:31.171281+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:27.259393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:27.259393+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"83","reported_epoch":"99","state":"active+clean","last_fresh":"2021-02-10T04:28:35.419584+0000","last_change":"2021-02-10T04:27:53.635318+0000","last_active":"2021-02-10T04:28:35.419584+0000","last_peered":"2021-02-10T04:28:35.419584+0000","last_clean":"2021-02-10T04:28:35.419584+0000","last_became_active":"2021-02-10T04:25:36.337394+0000","last_became_peered":"2021-02-10T04:25:36.337394+0000","last_unstale":"2021-02-10T04:28:35.419584+0000","last_undegraded":"2021-02-10T04:28:35.419584+0000","last_fullsized":"2021-02-10T04:28:35.419584+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:27:53.635228+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:27:53.635228+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"63'27","reported_seq":"117","reported_epoch":"99","state":"active+clean","last_fresh":"2021-02-10T04:28:35.418797+0000","last_change":"2021-02-10T04:27:17.352786+0000","last_active":"2021-02-10T04:28:35.418797+0000","last_peered":"2021-02-10T04:28:35.418797+0000","last_clean":"2021-02-10T04:28:35.418797+0000","last_became_active":"2021-02-10T04:25:49.013259+0000","last_became_peered":"2021-02-10T04:25:49.013259+0000","last_unstale":"2021-02-10T04:28:35.418797+0000","last_undegraded":"2021-02-10T04:28:35.418797+0000","last_fullsized":"2021-02-10T04:28:35.418797+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'27","last_scrub_stamp":"2021-02-10T04:27:17.352701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:17.352701+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":41,"num_write":29,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"54","reported_epoch":"98","state":"active+clean","last_fresh":"2021-02-10T04:28:34.050440+0000","last_change":"2021-02-10T04:27:14.530439+0000","last_active":"2021-02-10T04:28:34.050440+0000","last_peered":"2021-02-10T04:28:34.050440+0000","last_clean":"2021-02-10T04:28:34.050440+0000","last_became_active":"2021-02-10T04:25:51.023000+0000","last_became_peered":"2021-02-10T04:25:51.023000+0000","last_unstale":"2021-02-10T04:28:34.050440+0000","last_undegraded":"2021-02-10T04:28:34.050440+0000","last_fullsized":"2021-02-10T04:28:34.050440+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:14.530378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:14.530378+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1","version":"100'13","reported_seq":"31","reported_epoch":"100","state":"active+clean","last_fresh":"2021-02-10T04:28:36.216873+0000","last_change":"2021-02-10T04:28:25.154812+0000","last_active":"2021-02-10T04:28:36.216873+0000","last_peered":"2021-02-10T04:28:36.216873+0000","last_clean":"2021-02-10T04:28:36.216873+0000","last_became_active":"2021-02-10T04:28:25.154434+0000","last_became_peered":"2021-02-10T04:28:25.154434+0000","last_unstale":"2021-02-10T04:28:36.216873+0000","last_undegraded":"2021-02-10T04:28:36.216873+0000","last_fullsized":"2021-02-10T04:28:36.216873+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8689326,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":16,"num_write_kb":4381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"52","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.451059+0000","last_change":"2021-02-10T04:27:18.521962+0000","last_active":"2021-02-10T04:28:31.451059+0000","last_peered":"2021-02-10T04:28:31.451059+0000","last_clean":"2021-02-10T04:28:31.451059+0000","last_became_active":"2021-02-10T04:25:53.045215+0000","last_became_peered":"2021-02-10T04:25:53.045215+0000","last_unstale":"2021-02-10T04:28:31.451059+0000","last_undegraded":"2021-02-10T04:28:31.451059+0000","last_fullsized":"2021-02-10T04:28:31.451059+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:18.521897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:18.521897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"89","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:30.690036+0000","last_change":"2021-02-10T04:28:21.134161+0000","last_active":"2021-02-10T04:28:30.690036+0000","last_peered":"2021-02-10T04:28:30.690036+0000","last_clean":"2021-02-10T04:28:30.690036+0000","last_became_active":"2021-02-10T04:28:21.133772+0000","last_became_peered":"2021-02-10T04:28:21.133772+0000","last_unstale":"2021-02-10T04:28:30.690036+0000","last_undegraded":"2021-02-10T04:28:30.690036+0000","last_fullsized":"2021-02-10T04:28:30.690036+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:04.024848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:04.024848+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"79'34","reported_seq":"109","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:30.690473+0000","last_change":"2021-02-10T04:27:18.990156+0000","last_active":"2021-02-10T04:28:30.690473+0000","last_peered":"2021-02-10T04:28:30.690473+0000","last_clean":"2021-02-10T04:28:30.690473+0000","last_became_active":"2021-02-10T04:25:49.006147+0000","last_became_peered":"2021-02-10T04:25:49.006147+0000","last_unstale":"2021-02-10T04:28:30.690473+0000","last_undegraded":"2021-02-10T04:28:30.690473+0000","last_fullsized":"2021-02-10T04:28:30.690473+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'34","last_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:27:18.990088+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":22,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"58'2","reported_seq":"94","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:33.656631+0000","last_change":"2021-02-10T04:28:33.656576+0000","last_active":"2021-02-10T04:28:33.656631+0000","last_peered":"2021-02-10T04:28:33.656631+0000","last_clean":"2021-02-10T04:28:33.656631+0000","last_became_active":"2021-02-10T04:25:51.026251+0000","last_became_peered":"2021-02-10T04:25:51.026251+0000","last_unstale":"2021-02-10T04:28:33.656631+0000","last_undegraded":"2021-02-10T04:28:33.656631+0000","last_fullsized":"2021-02-10T04:28:33.656631+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:28:33.656509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:33.656509+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":"13","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.380043+0000","last_change":"2021-02-10T04:28:25.148663+0000","last_active":"2021-02-10T04:28:31.380043+0000","last_peered":"2021-02-10T04:28:31.380043+0000","last_clean":"2021-02-10T04:28:31.380043+0000","last_became_active":"2021-02-10T04:28:25.147959+0000","last_became_peered":"2021-02-10T04:28:25.147959+0000","last_unstale":"2021-02-10T04:28:31.380043+0000","last_undegraded":"2021-02-10T04:28:31.380043+0000","last_fullsized":"2021-02-10T04:28:31.380043+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"51","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.172787+0000","last_change":"2021-02-10T04:27:37.196914+0000","last_active":"2021-02-10T04:28:31.172787+0000","last_peered":"2021-02-10T04:28:31.172787+0000","last_clean":"2021-02-10T04:28:31.172787+0000","last_became_active":"2021-02-10T04:25:53.049487+0000","last_became_peered":"2021-02-10T04:25:53.049487+0000","last_unstale":"2021-02-10T04:28:31.172787+0000","last_undegraded":"2021-02-10T04:28:31.172787+0000","last_fullsized":"2021-02-10T04:28:31.172787+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:37.196846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:37.196846+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"71","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:29.865599+0000","last_change":"2021-02-10T04:28:21.125956+0000","last_active":"2021-02-10T04:28:29.865599+0000","last_peered":"2021-02-10T04:28:29.865599+0000","last_clean":"2021-02-10T04:28:29.865599+0000","last_became_active":"2021-02-10T04:28:21.125541+0000","last_became_peered":"2021-02-10T04:28:21.125541+0000","last_unstale":"2021-02-10T04:28:29.865599+0000","last_undegraded":"2021-02-10T04:28:29.865599+0000","last_fullsized":"2021-02-10T04:28:29.865599+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:06.934282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:06.934282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"63'31","reported_seq":"122","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:30.689954+0000","last_change":"2021-02-10T04:28:03.352844+0000","last_active":"2021-02-10T04:28:30.689954+0000","last_peered":"2021-02-10T04:28:30.689954+0000","last_clean":"2021-02-10T04:28:30.689954+0000","last_became_active":"2021-02-10T04:25:49.007872+0000","last_became_peered":"2021-02-10T04:25:49.007872+0000","last_unstale":"2021-02-10T04:28:30.689954+0000","last_undegraded":"2021-02-10T04:28:30.689954+0000","last_fullsized":"2021-02-10T04:28:30.689954+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'31","last_scrub_stamp":"2021-02-10T04:28:03.352778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:03.352778+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":66,"num_read_kb":42,"num_write":34,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"54","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.451858+0000","last_change":"2021-02-10T04:27:31.650126+0000","last_active":"2021-02-10T04:28:31.451858+0000","last_peered":"2021-02-10T04:28:31.451858+0000","last_clean":"2021-02-10T04:28:31.451858+0000","last_became_active":"2021-02-10T04:25:51.027624+0000","last_became_peered":"2021-02-10T04:25:51.027624+0000","last_unstale":"2021-02-10T04:28:31.451858+0000","last_undegraded":"2021-02-10T04:28:31.451858+0000","last_fullsized":"2021-02-10T04:28:31.451858+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:27:31.650033+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"98'6","reported_seq":"21","reported_epoch":"98","state":"active+clean","last_fresh":"2021-02-10T04:28:34.390098+0000","last_change":"2021-02-10T04:28:25.151362+0000","last_active":"2021-02-10T04:28:34.390098+0000","last_peered":"2021-02-10T04:28:34.390098+0000","last_clean":"2021-02-10T04:28:34.390098+0000","last_became_active":"2021-02-10T04:28:25.151037+0000","last_became_peered":"2021-02-10T04:28:25.151037+0000","last_unstale":"2021-02-10T04:28:34.390098+0000","last_undegraded":"2021-02-10T04:28:34.390098+0000","last_fullsized":"2021-02-10T04:28:34.390098+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4840350,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"52","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.192144+0000","last_change":"2021-02-10T04:27:28.264288+0000","last_active":"2021-02-10T04:28:31.192144+0000","last_peered":"2021-02-10T04:28:31.192144+0000","last_clean":"2021-02-10T04:28:31.192144+0000","last_became_active":"2021-02-10T04:25:53.149677+0000","last_became_peered":"2021-02-10T04:25:53.149677+0000","last_unstale":"2021-02-10T04:28:31.192144+0000","last_undegraded":"2021-02-10T04:28:31.192144+0000","last_fullsized":"2021-02-10T04:28:31.192144+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:28.264204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:28.264204+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"219","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.793078+0000","last_change":"2021-02-10T04:26:58.989515+0000","last_active":"2021-02-10T04:28:31.793078+0000","last_peered":"2021-02-10T04:28:31.793078+0000","last_clean":"2021-02-10T04:28:31.793078+0000","last_became_active":"2021-02-10T04:25:49.013969+0000","last_became_peered":"2021-02-10T04:25:49.013969+0000","last_unstale":"2021-02-10T04:28:31.793078+0000","last_undegraded":"2021-02-10T04:28:31.793078+0000","last_fullsized":"2021-02-10T04:28:31.793078+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:26:58.989447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:26:58.989447+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":161,"num_read_kb":156,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"77","reported_epoch":"98","state":"active+clean","last_fresh":"2021-02-10T04:28:34.050403+0000","last_change":"2021-02-10T04:28:21.120629+0000","last_active":"2021-02-10T04:28:34.050403+0000","last_peered":"2021-02-10T04:28:34.050403+0000","last_clean":"2021-02-10T04:28:34.050403+0000","last_became_active":"2021-02-10T04:28:21.120083+0000","last_became_peered":"2021-02-10T04:28:21.120083+0000","last_unstale":"2021-02-10T04:28:34.050403+0000","last_undegraded":"2021-02-10T04:28:34.050403+0000","last_fullsized":"2021-02-10T04:28:34.050403+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:44.556741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:44.556741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"82'29","reported_seq":"109","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:30.691014+0000","last_change":"2021-02-10T04:27:38.176842+0000","last_active":"2021-02-10T04:28:30.691014+0000","last_peered":"2021-02-10T04:28:30.691014+0000","last_clean":"2021-02-10T04:28:30.691014+0000","last_became_active":"2021-02-10T04:25:49.002843+0000","last_became_peered":"2021-02-10T04:25:49.002843+0000","last_unstale":"2021-02-10T04:28:30.691014+0000","last_undegraded":"2021-02-10T04:28:30.691014+0000","last_fullsized":"2021-02-10T04:28:30.691014+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'29","last_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:27:38.176754+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":32,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"54","reported_epoch":"98","state":"active+clean","last_fresh":"2021-02-10T04:28:34.052050+0000","last_change":"2021-02-10T04:27:16.564733+0000","last_active":"2021-02-10T04:28:34.052050+0000","last_peered":"2021-02-10T04:28:34.052050+0000","last_clean":"2021-02-10T04:28:34.052050+0000","last_became_active":"2021-02-10T04:25:51.025306+0000","last_became_peered":"2021-02-10T04:25:51.025306+0000","last_unstale":"2021-02-10T04:28:34.052050+0000","last_undegraded":"2021-02-10T04:28:34.052050+0000","last_fullsized":"2021-02-10T04:28:34.052050+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:27:16.564638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"98'6","reported_seq":"21","reported_epoch":"98","state":"active+clean","last_fresh":"2021-02-10T04:28:34.362711+0000","last_change":"2021-02-10T04:28:25.150436+0000","last_active":"2021-02-10T04:28:34.362711+0000","last_peered":"2021-02-10T04:28:34.362711+0000","last_clean":"2021-02-10T04:28:34.362711+0000","last_became_active":"2021-02-10T04:28:25.150109+0000","last_became_peered":"2021-02-10T04:28:25.150109+0000","last_unstale":"2021-02-10T04:28:34.362711+0000","last_undegraded":"2021-02-10T04:28:34.362711+0000","last_fullsized":"2021-02-10T04:28:34.362711+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4470596,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"51","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:29.867628+0000","last_change":"2021-02-10T04:27:18.115804+0000","last_active":"2021-02-10T04:28:29.867628+0000","last_peered":"2021-02-10T04:28:29.867628+0000","last_clean":"2021-02-10T04:28:29.867628+0000","last_became_active":"2021-02-10T04:25:53.037506+0000","last_became_peered":"2021-02-10T04:25:53.037506+0000","last_unstale":"2021-02-10T04:28:29.867628+0000","last_undegraded":"2021-02-10T04:28:29.867628+0000","last_fullsized":"2021-02-10T04:28:29.867628+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:18.115741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:18.115741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"55","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.172664+0000","last_change":"2021-02-10T04:27:39.268187+0000","last_active":"2021-02-10T04:28:31.172664+0000","last_peered":"2021-02-10T04:28:31.172664+0000","last_clean":"2021-02-10T04:28:31.172664+0000","last_became_active":"2021-02-10T04:25:49.010614+0000","last_became_peered":"2021-02-10T04:25:49.010614+0000","last_unstale":"2021-02-10T04:28:31.172664+0000","last_undegraded":"2021-02-10T04:28:31.172664+0000","last_fullsized":"2021-02-10T04:28:31.172664+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:39.268114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:39.268114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"78","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:29.866553+0000","last_change":"2021-02-10T04:28:21.123391+0000","last_active":"2021-02-10T04:28:29.866553+0000","last_peered":"2021-02-10T04:28:29.866553+0000","last_clean":"2021-02-10T04:28:29.866553+0000","last_became_active":"2021-02-10T04:28:21.123027+0000","last_became_peered":"2021-02-10T04:28:21.123027+0000","last_unstale":"2021-02-10T04:28:29.866553+0000","last_undegraded":"2021-02-10T04:28:29.866553+0000","last_fullsized":"2021-02-10T04:28:29.866553+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:55.385094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:55.385094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"67","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.191253+0000","last_change":"2021-02-10T04:28:30.424410+0000","last_active":"2021-02-10T04:28:31.191253+0000","last_peered":"2021-02-10T04:28:31.191253+0000","last_clean":"2021-02-10T04:28:31.191253+0000","last_became_active":"2021-02-10T04:25:36.343397+0000","last_became_peered":"2021-02-10T04:25:36.343397+0000","last_unstale":"2021-02-10T04:28:31.191253+0000","last_undegraded":"2021-02-10T04:28:31.191253+0000","last_fullsized":"2021-02-10T04:28:31.191253+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:30.424326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:28:30.424326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"62'15","reported_seq":"80","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.380364+0000","last_change":"2021-02-10T04:27:08.069986+0000","last_active":"2021-02-10T04:28:31.380364+0000","last_peered":"2021-02-10T04:28:31.380364+0000","last_clean":"2021-02-10T04:28:31.380364+0000","last_became_active":"2021-02-10T04:25:49.002471+0000","last_became_peered":"2021-02-10T04:25:49.002471+0000","last_unstale":"2021-02-10T04:28:31.380364+0000","last_undegraded":"2021-02-10T04:28:31.380364+0000","last_fullsized":"2021-02-10T04:28:31.380364+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:27:08.069899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:08.069899+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":15,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"54","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.191884+0000","last_change":"2021-02-10T04:27:38.106291+0000","last_active":"2021-02-10T04:28:31.191884+0000","last_peered":"2021-02-10T04:28:31.191884+0000","last_clean":"2021-02-10T04:28:31.191884+0000","last_became_active":"2021-02-10T04:25:51.022821+0000","last_became_peered":"2021-02-10T04:25:51.022821+0000","last_unstale":"2021-02-10T04:28:31.191884+0000","last_undegraded":"2021-02-10T04:28:31.191884+0000","last_fullsized":"2021-02-10T04:28:31.191884+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:38.106209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:38.106209+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"98'6","reported_seq":"21","reported_epoch":"98","state":"active+clean","last_fresh":"2021-02-10T04:28:34.351728+0000","last_change":"2021-02-10T04:28:25.156802+0000","last_active":"2021-02-10T04:28:34.351728+0000","last_peered":"2021-02-10T04:28:34.351728+0000","last_clean":"2021-02-10T04:28:34.351728+0000","last_became_active":"2021-02-10T04:28:25.156013+0000","last_became_peered":"2021-02-10T04:28:25.156013+0000","last_unstale":"2021-02-10T04:28:34.351728+0000","last_undegraded":"2021-02-10T04:28:34.351728+0000","last_fullsized":"2021-02-10T04:28:34.351728+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4689087,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"51","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:30.690535+0000","last_change":"2021-02-10T04:27:10.051857+0000","last_active":"2021-02-10T04:28:30.690535+0000","last_peered":"2021-02-10T04:28:30.690535+0000","last_clean":"2021-02-10T04:28:30.690535+0000","last_became_active":"2021-02-10T04:25:53.040390+0000","last_became_peered":"2021-02-10T04:25:53.040390+0000","last_unstale":"2021-02-10T04:28:30.690535+0000","last_undegraded":"2021-02-10T04:28:30.690535+0000","last_fullsized":"2021-02-10T04:28:30.690535+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:10.051800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:10.051800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"56","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.451282+0000","last_change":"2021-02-10T04:27:30.657844+0000","last_active":"2021-02-10T04:28:31.451282+0000","last_peered":"2021-02-10T04:28:31.451282+0000","last_clean":"2021-02-10T04:28:31.451282+0000","last_became_active":"2021-02-10T04:25:49.001782+0000","last_became_peered":"2021-02-10T04:25:49.001782+0000","last_unstale":"2021-02-10T04:28:31.451282+0000","last_undegraded":"2021-02-10T04:28:31.451282+0000","last_fullsized":"2021-02-10T04:28:31.451282+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:30.657781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:30.657781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"75","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:33.477190+0000","last_change":"2021-02-10T04:28:33.477123+0000","last_active":"2021-02-10T04:28:33.477190+0000","last_peered":"2021-02-10T04:28:33.477190+0000","last_clean":"2021-02-10T04:28:33.477190+0000","last_became_active":"2021-02-10T04:28:21.126668+0000","last_became_peered":"2021-02-10T04:28:21.126668+0000","last_unstale":"2021-02-10T04:28:33.477190+0000","last_undegraded":"2021-02-10T04:28:33.477190+0000","last_fullsized":"2021-02-10T04:28:33.477190+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:33.477060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:33.477060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"62'16","reported_seq":"87","reported_epoch":"98","state":"active+clean","last_fresh":"2021-02-10T04:28:34.694585+0000","last_change":"2021-02-10T04:28:34.694585+0000","last_active":"2021-02-10T04:28:34.694585+0000","last_peered":"2021-02-10T04:28:34.694585+0000","last_clean":"2021-02-10T04:28:34.694585+0000","last_became_active":"2021-02-10T04:25:49.003521+0000","last_became_peered":"2021-02-10T04:25:49.003521+0000","last_unstale":"2021-02-10T04:28:34.694585+0000","last_undegraded":"2021-02-10T04:28:34.694585+0000","last_fullsized":"2021-02-10T04:28:34.694585+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'16","last_scrub_stamp":"2021-02-10T04:28:34.694486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:34.694486+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":16,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"53","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:29.867616+0000","last_change":"2021-02-10T04:27:25.216335+0000","last_active":"2021-02-10T04:28:29.867616+0000","last_peered":"2021-02-10T04:28:29.867616+0000","last_clean":"2021-02-10T04:28:29.867616+0000","last_became_active":"2021-02-10T04:25:51.032201+0000","last_became_peered":"2021-02-10T04:25:51.032201+0000","last_unstale":"2021-02-10T04:28:29.867616+0000","last_undegraded":"2021-02-10T04:28:29.867616+0000","last_fullsized":"2021-02-10T04:28:29.867616+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:27:25.216260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":"12","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:29.865023+0000","last_change":"2021-02-10T04:28:25.147122+0000","last_active":"2021-02-10T04:28:29.865023+0000","last_peered":"2021-02-10T04:28:29.865023+0000","last_clean":"2021-02-10T04:28:29.865023+0000","last_became_active":"2021-02-10T04:28:25.146766+0000","last_became_peered":"2021-02-10T04:28:25.146766+0000","last_unstale":"2021-02-10T04:28:29.865023+0000","last_undegraded":"2021-02-10T04:28:29.865023+0000","last_fullsized":"2021-02-10T04:28:29.865023+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"51","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:30.689952+0000","last_change":"2021-02-10T04:27:42.182019+0000","last_active":"2021-02-10T04:28:30.689952+0000","last_peered":"2021-02-10T04:28:30.689952+0000","last_clean":"2021-02-10T04:28:30.689952+0000","last_became_active":"2021-02-10T04:25:53.149552+0000","last_became_peered":"2021-02-10T04:25:53.149552+0000","last_unstale":"2021-02-10T04:28:30.689952+0000","last_undegraded":"2021-02-10T04:28:30.689952+0000","last_fullsized":"2021-02-10T04:28:30.689952+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:42.181940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:42.181940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"78","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.453915+0000","last_change":"2021-02-10T04:28:31.453821+0000","last_active":"2021-02-10T04:28:31.453915+0000","last_peered":"2021-02-10T04:28:31.453915+0000","last_clean":"2021-02-10T04:28:31.453915+0000","last_became_active":"2021-02-10T04:28:21.121979+0000","last_became_peered":"2021-02-10T04:28:21.121979+0000","last_unstale":"2021-02-10T04:28:31.453915+0000","last_undegraded":"2021-02-10T04:28:31.453915+0000","last_fullsized":"2021-02-10T04:28:31.453915+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:28:31.453740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:31.453740+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"63'27","reported_seq":"111","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.451423+0000","last_change":"2021-02-10T04:27:10.457048+0000","last_active":"2021-02-10T04:28:31.451423+0000","last_peered":"2021-02-10T04:28:31.451423+0000","last_clean":"2021-02-10T04:28:31.451423+0000","last_became_active":"2021-02-10T04:25:49.002937+0000","last_became_peered":"2021-02-10T04:25:49.002937+0000","last_unstale":"2021-02-10T04:28:31.451423+0000","last_undegraded":"2021-02-10T04:28:31.451423+0000","last_fullsized":"2021-02-10T04:28:31.451423+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'27","last_scrub_stamp":"2021-02-10T04:27:10.456981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:10.456981+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":38,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"54","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.379823+0000","last_change":"2021-02-10T04:26:59.175043+0000","last_active":"2021-02-10T04:28:31.379823+0000","last_peered":"2021-02-10T04:28:31.379823+0000","last_clean":"2021-02-10T04:28:31.379823+0000","last_became_active":"2021-02-10T04:25:51.025636+0000","last_became_peered":"2021-02-10T04:25:51.025636+0000","last_unstale":"2021-02-10T04:28:31.379823+0000","last_undegraded":"2021-02-10T04:28:31.379823+0000","last_fullsized":"2021-02-10T04:28:31.379823+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:26:59.174908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:26:59.174908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.3","version":"98'12","reported_seq":"29","reported_epoch":"98","state":"active+clean","last_fresh":"2021-02-10T04:28:34.637276+0000","last_change":"2021-02-10T04:28:25.161914+0000","last_active":"2021-02-10T04:28:34.637276+0000","last_peered":"2021-02-10T04:28:34.637276+0000","last_clean":"2021-02-10T04:28:34.637276+0000","last_became_active":"2021-02-10T04:28:25.161622+0000","last_became_peered":"2021-02-10T04:28:25.161622+0000","last_unstale":"2021-02-10T04:28:34.637276+0000","last_undegraded":"2021-02-10T04:28:34.637276+0000","last_fullsized":"2021-02-10T04:28:34.637276+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9949612,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"59","reported_epoch":"99","state":"active+clean","last_fresh":"2021-02-10T04:28:35.419265+0000","last_change":"2021-02-10T04:28:05.641668+0000","last_active":"2021-02-10T04:28:35.419265+0000","last_peered":"2021-02-10T04:28:35.419265+0000","last_clean":"2021-02-10T04:28:35.419265+0000","last_became_active":"2021-02-10T04:25:53.045085+0000","last_became_peered":"2021-02-10T04:25:53.045085+0000","last_unstale":"2021-02-10T04:28:35.419265+0000","last_undegraded":"2021-02-10T04:28:35.419265+0000","last_fullsized":"2021-02-10T04:28:35.419265+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:05.641577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:05.641577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"76","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:30.690861+0000","last_change":"2021-02-10T04:28:21.133617+0000","last_active":"2021-02-10T04:28:30.690861+0000","last_peered":"2021-02-10T04:28:30.690861+0000","last_clean":"2021-02-10T04:28:30.690861+0000","last_became_active":"2021-02-10T04:28:21.133248+0000","last_became_peered":"2021-02-10T04:28:21.133248+0000","last_unstale":"2021-02-10T04:28:30.690861+0000","last_undegraded":"2021-02-10T04:28:30.690861+0000","last_fullsized":"2021-02-10T04:28:30.690861+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:27:25.026534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:25.026534+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":423,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"63'34","reported_seq":"125","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.170680+0000","last_change":"2021-02-10T04:27:20.342445+0000","last_active":"2021-02-10T04:28:31.170680+0000","last_peered":"2021-02-10T04:28:31.170680+0000","last_clean":"2021-02-10T04:28:31.170680+0000","last_became_active":"2021-02-10T04:25:48.997481+0000","last_became_peered":"2021-02-10T04:25:48.997481+0000","last_unstale":"2021-02-10T04:28:31.170680+0000","last_undegraded":"2021-02-10T04:28:31.170680+0000","last_fullsized":"2021-02-10T04:28:31.170680+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'34","last_scrub_stamp":"2021-02-10T04:27:20.342358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:20.342358+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":70,"num_read_kb":48,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"56","reported_epoch":"99","state":"active+clean","last_fresh":"2021-02-10T04:28:35.419001+0000","last_change":"2021-02-10T04:27:44.501111+0000","last_active":"2021-02-10T04:28:35.419001+0000","last_peered":"2021-02-10T04:28:35.419001+0000","last_clean":"2021-02-10T04:28:35.419001+0000","last_became_active":"2021-02-10T04:25:51.018706+0000","last_became_peered":"2021-02-10T04:25:51.018706+0000","last_unstale":"2021-02-10T04:28:35.419001+0000","last_undegraded":"2021-02-10T04:28:35.419001+0000","last_fullsized":"2021-02-10T04:28:35.419001+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:44.501024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:44.501024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"98'6","reported_seq":"21","reported_epoch":"98","state":"active+clean","last_fresh":"2021-02-10T04:28:34.493275+0000","last_change":"2021-02-10T04:28:25.158098+0000","last_active":"2021-02-10T04:28:34.493275+0000","last_peered":"2021-02-10T04:28:34.493275+0000","last_clean":"2021-02-10T04:28:34.493275+0000","last_became_active":"2021-02-10T04:28:25.157306+0000","last_became_peered":"2021-02-10T04:28:25.157306+0000","last_unstale":"2021-02-10T04:28:34.493275+0000","last_undegraded":"2021-02-10T04:28:34.493275+0000","last_fullsized":"2021-02-10T04:28:34.493275+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4957999,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"52","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.191354+0000","last_change":"2021-02-10T04:27:26.234432+0000","last_active":"2021-02-10T04:28:31.191354+0000","last_peered":"2021-02-10T04:28:31.191354+0000","last_clean":"2021-02-10T04:28:31.191354+0000","last_became_active":"2021-02-10T04:25:53.150930+0000","last_became_peered":"2021-02-10T04:25:53.150930+0000","last_unstale":"2021-02-10T04:28:31.191354+0000","last_undegraded":"2021-02-10T04:28:31.191354+0000","last_fullsized":"2021-02-10T04:28:31.191354+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:26.234349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:26.234349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"86","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.380173+0000","last_change":"2021-02-10T04:28:21.125294+0000","last_active":"2021-02-10T04:28:31.380173+0000","last_peered":"2021-02-10T04:28:31.380173+0000","last_clean":"2021-02-10T04:28:31.380173+0000","last_became_active":"2021-02-10T04:28:21.124966+0000","last_became_peered":"2021-02-10T04:28:21.124966+0000","last_unstale":"2021-02-10T04:28:31.380173+0000","last_undegraded":"2021-02-10T04:28:31.380173+0000","last_fullsized":"2021-02-10T04:28:31.380173+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:11.234741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:11.234741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"63'26","reported_seq":"106","reported_epoch":"99","state":"active+clean","last_fresh":"2021-02-10T04:28:35.418644+0000","last_change":"2021-02-10T04:27:26.403830+0000","last_active":"2021-02-10T04:28:35.418644+0000","last_peered":"2021-02-10T04:28:35.418644+0000","last_clean":"2021-02-10T04:28:35.418644+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:28:35.418644+0000","last_undegraded":"2021-02-10T04:28:35.418644+0000","last_fullsized":"2021-02-10T04:28:35.418644+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:27:26.403748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:26.403748+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":51,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"58'2","reported_seq":"88","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:32.733971+0000","last_change":"2021-02-10T04:27:33.210495+0000","last_active":"2021-02-10T04:28:32.733971+0000","last_peered":"2021-02-10T04:28:32.733971+0000","last_clean":"2021-02-10T04:28:32.733971+0000","last_became_active":"2021-02-10T04:25:51.031906+0000","last_became_peered":"2021-02-10T04:25:51.031906+0000","last_unstale":"2021-02-10T04:28:32.733971+0000","last_undegraded":"2021-02-10T04:28:32.733971+0000","last_fullsized":"2021-02-10T04:28:32.733971+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:33.210431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:33.210431+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"97'3","reported_seq":"17","reported_epoch":"98","state":"active+clean","last_fresh":"2021-02-10T04:28:34.041557+0000","last_change":"2021-02-10T04:28:25.152401+0000","last_active":"2021-02-10T04:28:34.041557+0000","last_peered":"2021-02-10T04:28:34.041557+0000","last_clean":"2021-02-10T04:28:34.041557+0000","last_became_active":"2021-02-10T04:28:25.152030+0000","last_became_peered":"2021-02-10T04:28:25.152030+0000","last_unstale":"2021-02-10T04:28:34.041557+0000","last_undegraded":"2021-02-10T04:28:34.041557+0000","last_fullsized":"2021-02-10T04:28:34.041557+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2016807,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"51","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.171668+0000","last_change":"2021-02-10T04:27:52.383262+0000","last_active":"2021-02-10T04:28:31.171668+0000","last_peered":"2021-02-10T04:28:31.171668+0000","last_clean":"2021-02-10T04:28:31.171668+0000","last_became_active":"2021-02-10T04:25:53.148054+0000","last_became_peered":"2021-02-10T04:25:53.148054+0000","last_unstale":"2021-02-10T04:28:31.171668+0000","last_undegraded":"2021-02-10T04:28:31.171668+0000","last_fullsized":"2021-02-10T04:28:31.171668+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:52.383188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:52.383188+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"58","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:29.867528+0000","last_change":"2021-02-10T04:28:02.288782+0000","last_active":"2021-02-10T04:28:29.867528+0000","last_peered":"2021-02-10T04:28:29.867528+0000","last_clean":"2021-02-10T04:28:29.867528+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:28:29.867528+0000","last_undegraded":"2021-02-10T04:28:29.867528+0000","last_fullsized":"2021-02-10T04:28:29.867528+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:02.288683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:02.288683+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"80","reported_epoch":"98","state":"active+clean","last_fresh":"2021-02-10T04:28:34.444413+0000","last_change":"2021-02-10T04:28:34.444413+0000","last_active":"2021-02-10T04:28:34.444413+0000","last_peered":"2021-02-10T04:28:34.444413+0000","last_clean":"2021-02-10T04:28:34.444413+0000","last_became_active":"2021-02-10T04:28:21.126180+0000","last_became_peered":"2021-02-10T04:28:21.126180+0000","last_unstale":"2021-02-10T04:28:34.444413+0000","last_undegraded":"2021-02-10T04:28:34.444413+0000","last_fullsized":"2021-02-10T04:28:34.444413+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:34.444339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:28:34.444339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"86'31","reported_seq":"115","reported_epoch":"99","state":"active+clean","last_fresh":"2021-02-10T04:28:35.419204+0000","last_change":"2021-02-10T04:28:11.693702+0000","last_active":"2021-02-10T04:28:35.419204+0000","last_peered":"2021-02-10T04:28:35.419204+0000","last_clean":"2021-02-10T04:28:35.419204+0000","last_became_active":"2021-02-10T04:25:49.011093+0000","last_became_peered":"2021-02-10T04:25:49.011093+0000","last_unstale":"2021-02-10T04:28:35.419204+0000","last_undegraded":"2021-02-10T04:28:35.419204+0000","last_fullsized":"2021-02-10T04:28:35.419204+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'31","last_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:28:11.693616+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":28,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":"13","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.379683+0000","last_change":"2021-02-10T04:28:25.149207+0000","last_active":"2021-02-10T04:28:31.379683+0000","last_peered":"2021-02-10T04:28:31.379683+0000","last_clean":"2021-02-10T04:28:31.379683+0000","last_became_active":"2021-02-10T04:28:25.148500+0000","last_became_peered":"2021-02-10T04:28:25.148500+0000","last_unstale":"2021-02-10T04:28:31.379683+0000","last_undegraded":"2021-02-10T04:28:31.379683+0000","last_fullsized":"2021-02-10T04:28:31.379683+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"51","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:29.867031+0000","last_change":"2021-02-10T04:27:24.209544+0000","last_active":"2021-02-10T04:28:29.867031+0000","last_peered":"2021-02-10T04:28:29.867031+0000","last_clean":"2021-02-10T04:28:29.867031+0000","last_became_active":"2021-02-10T04:25:53.042697+0000","last_became_peered":"2021-02-10T04:25:53.042697+0000","last_unstale":"2021-02-10T04:28:29.867031+0000","last_undegraded":"2021-02-10T04:28:29.867031+0000","last_fullsized":"2021-02-10T04:28:29.867031+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:27:24.209460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"78'3","reported_seq":"92","reported_epoch":"99","state":"active+clean","last_fresh":"2021-02-10T04:28:35.419382+0000","last_change":"2021-02-10T04:27:13.268467+0000","last_active":"2021-02-10T04:28:35.419382+0000","last_peered":"2021-02-10T04:28:35.419382+0000","last_clean":"2021-02-10T04:28:35.419382+0000","last_became_active":"2021-02-10T04:25:51.025218+0000","last_became_peered":"2021-02-10T04:25:51.025218+0000","last_unstale":"2021-02-10T04:28:35.419382+0000","last_undegraded":"2021-02-10T04:28:35.419382+0000","last_fullsized":"2021-02-10T04:28:35.419382+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:27:13.268412+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"80","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:30.689778+0000","last_change":"2021-02-10T04:28:21.130555+0000","last_active":"2021-02-10T04:28:30.689778+0000","last_peered":"2021-02-10T04:28:30.689778+0000","last_clean":"2021-02-10T04:28:30.689778+0000","last_became_active":"2021-02-10T04:28:21.130021+0000","last_became_peered":"2021-02-10T04:28:21.130021+0000","last_unstale":"2021-02-10T04:28:30.689778+0000","last_undegraded":"2021-02-10T04:28:30.689778+0000","last_fullsized":"2021-02-10T04:28:30.689778+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:13.388827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:13.388827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"62'17","reported_seq":"84","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:30.690971+0000","last_change":"2021-02-10T04:27:30.012988+0000","last_active":"2021-02-10T04:28:30.690971+0000","last_peered":"2021-02-10T04:28:30.690971+0000","last_clean":"2021-02-10T04:28:30.690971+0000","last_became_active":"2021-02-10T04:25:49.004933+0000","last_became_peered":"2021-02-10T04:25:49.004933+0000","last_unstale":"2021-02-10T04:28:30.690971+0000","last_undegraded":"2021-02-10T04:28:30.690971+0000","last_fullsized":"2021-02-10T04:28:30.690971+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'17","last_scrub_stamp":"2021-02-10T04:27:30.012927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:30.012927+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.f","version":"98'13","reported_seq":"31","reported_epoch":"98","state":"active+clean","last_fresh":"2021-02-10T04:28:34.706988+0000","last_change":"2021-02-10T04:28:25.149633+0000","last_active":"2021-02-10T04:28:34.706988+0000","last_peered":"2021-02-10T04:28:34.706988+0000","last_clean":"2021-02-10T04:28:34.706988+0000","last_became_active":"2021-02-10T04:28:25.149226+0000","last_became_peered":"2021-02-10T04:28:25.149226+0000","last_unstale":"2021-02-10T04:28:34.706988+0000","last_undegraded":"2021-02-10T04:28:34.706988+0000","last_fullsized":"2021-02-10T04:28:34.706988+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9697507,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":17,"num_write_kb":6633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"54","reported_epoch":"99","state":"active+clean","last_fresh":"2021-02-10T04:28:35.419492+0000","last_change":"2021-02-10T04:27:33.382403+0000","last_active":"2021-02-10T04:28:35.419492+0000","last_peered":"2021-02-10T04:28:35.419492+0000","last_clean":"2021-02-10T04:28:35.419492+0000","last_became_active":"2021-02-10T04:25:53.043027+0000","last_became_peered":"2021-02-10T04:25:53.043027+0000","last_unstale":"2021-02-10T04:28:35.419492+0000","last_undegraded":"2021-02-10T04:28:35.419492+0000","last_fullsized":"2021-02-10T04:28:35.419492+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:33.382319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:33.382319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"58'2","reported_seq":"94","reported_epoch":"99","state":"active+clean","last_fresh":"2021-02-10T04:28:35.648670+0000","last_change":"2021-02-10T04:28:35.648670+0000","last_active":"2021-02-10T04:28:35.648670+0000","last_peered":"2021-02-10T04:28:35.648670+0000","last_clean":"2021-02-10T04:28:35.648670+0000","last_became_active":"2021-02-10T04:25:51.025268+0000","last_became_peered":"2021-02-10T04:25:51.025268+0000","last_unstale":"2021-02-10T04:28:35.648670+0000","last_undegraded":"2021-02-10T04:28:35.648670+0000","last_fullsized":"2021-02-10T04:28:35.648670+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:28:35.648602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:35.648602+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"64","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:30.690274+0000","last_change":"2021-02-10T04:27:17.003396+0000","last_active":"2021-02-10T04:28:30.690274+0000","last_peered":"2021-02-10T04:28:30.690274+0000","last_clean":"2021-02-10T04:28:30.690274+0000","last_became_active":"2021-02-10T04:25:36.333883+0000","last_became_peered":"2021-02-10T04:25:36.333883+0000","last_unstale":"2021-02-10T04:28:30.690274+0000","last_undegraded":"2021-02-10T04:28:30.690274+0000","last_fullsized":"2021-02-10T04:28:30.690274+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:17.003304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:17.003304+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"62'13","reported_seq":"79","reported_epoch":"99","state":"active+clean","last_fresh":"2021-02-10T04:28:35.419025+0000","last_change":"2021-02-10T04:27:14.256603+0000","last_active":"2021-02-10T04:28:35.419025+0000","last_peered":"2021-02-10T04:28:35.419025+0000","last_clean":"2021-02-10T04:28:35.419025+0000","last_became_active":"2021-02-10T04:25:49.001725+0000","last_became_peered":"2021-02-10T04:25:49.001725+0000","last_unstale":"2021-02-10T04:28:35.419025+0000","last_undegraded":"2021-02-10T04:28:35.419025+0000","last_fullsized":"2021-02-10T04:28:35.419025+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'13","last_scrub_stamp":"2021-02-10T04:27:14.256539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:14.256539+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"98'13","reported_seq":"31","reported_epoch":"98","state":"active+clean","last_fresh":"2021-02-10T04:28:34.711319+0000","last_change":"2021-02-10T04:28:25.158458+0000","last_active":"2021-02-10T04:28:34.711319+0000","last_peered":"2021-02-10T04:28:34.711319+0000","last_clean":"2021-02-10T04:28:34.711319+0000","last_became_active":"2021-02-10T04:28:25.157931+0000","last_became_peered":"2021-02-10T04:28:25.157931+0000","last_unstale":"2021-02-10T04:28:34.711319+0000","last_undegraded":"2021-02-10T04:28:34.711319+0000","last_fullsized":"2021-02-10T04:28:34.711319+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7831963,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":17,"num_write_kb":4702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"57","reported_epoch":"98","state":"active+clean","last_fresh":"2021-02-10T04:28:34.051977+0000","last_change":"2021-02-10T04:28:32.623869+0000","last_active":"2021-02-10T04:28:34.051977+0000","last_peered":"2021-02-10T04:28:34.051977+0000","last_clean":"2021-02-10T04:28:34.051977+0000","last_became_active":"2021-02-10T04:25:53.038218+0000","last_became_peered":"2021-02-10T04:25:53.038218+0000","last_unstale":"2021-02-10T04:28:34.051977+0000","last_undegraded":"2021-02-10T04:28:34.051977+0000","last_fullsized":"2021-02-10T04:28:34.051977+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:32.623791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:32.623791+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"53","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:30.690198+0000","last_change":"2021-02-10T04:27:37.160395+0000","last_active":"2021-02-10T04:28:30.690198+0000","last_peered":"2021-02-10T04:28:30.690198+0000","last_clean":"2021-02-10T04:28:30.690198+0000","last_became_active":"2021-02-10T04:25:51.028302+0000","last_became_peered":"2021-02-10T04:25:51.028302+0000","last_unstale":"2021-02-10T04:28:30.690198+0000","last_undegraded":"2021-02-10T04:28:30.690198+0000","last_fullsized":"2021-02-10T04:28:30.690198+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:37.160328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:37.160328+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"61","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.191640+0000","last_change":"2021-02-10T04:27:00.052115+0000","last_active":"2021-02-10T04:28:31.191640+0000","last_peered":"2021-02-10T04:28:31.191640+0000","last_clean":"2021-02-10T04:28:31.191640+0000","last_became_active":"2021-02-10T04:25:36.344109+0000","last_became_peered":"2021-02-10T04:25:36.344109+0000","last_unstale":"2021-02-10T04:28:31.191640+0000","last_undegraded":"2021-02-10T04:28:31.191640+0000","last_fullsized":"2021-02-10T04:28:31.191640+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:00.052028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:27:00.052028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"62'21","reported_seq":"94","reported_epoch":"99","state":"active+clean","last_fresh":"2021-02-10T04:28:35.419508+0000","last_change":"2021-02-10T04:27:00.215115+0000","last_active":"2021-02-10T04:28:35.419508+0000","last_peered":"2021-02-10T04:28:35.419508+0000","last_clean":"2021-02-10T04:28:35.419508+0000","last_became_active":"2021-02-10T04:25:49.011699+0000","last_became_peered":"2021-02-10T04:25:49.011699+0000","last_unstale":"2021-02-10T04:28:35.419508+0000","last_undegraded":"2021-02-10T04:28:35.419508+0000","last_fullsized":"2021-02-10T04:28:35.419508+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'21","last_scrub_stamp":"2021-02-10T04:27:00.215037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:00.215037+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"98'6","reported_seq":"21","reported_epoch":"98","state":"active+clean","last_fresh":"2021-02-10T04:28:34.606903+0000","last_change":"2021-02-10T04:28:25.150545+0000","last_active":"2021-02-10T04:28:34.606903+0000","last_peered":"2021-02-10T04:28:34.606903+0000","last_clean":"2021-02-10T04:28:34.606903+0000","last_became_active":"2021-02-10T04:28:25.150246+0000","last_became_peered":"2021-02-10T04:28:25.150246+0000","last_unstale":"2021-02-10T04:28:34.606903+0000","last_undegraded":"2021-02-10T04:28:34.606903+0000","last_fullsized":"2021-02-10T04:28:34.606903+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5042034,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"52","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.191040+0000","last_change":"2021-02-10T04:27:35.166330+0000","last_active":"2021-02-10T04:28:31.191040+0000","last_peered":"2021-02-10T04:28:31.191040+0000","last_clean":"2021-02-10T04:28:31.191040+0000","last_became_active":"2021-02-10T04:25:53.037903+0000","last_became_peered":"2021-02-10T04:25:53.037903+0000","last_unstale":"2021-02-10T04:28:31.191040+0000","last_undegraded":"2021-02-10T04:28:31.191040+0000","last_fullsized":"2021-02-10T04:28:31.191040+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:35.166250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:35.166250+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"54","reported_epoch":"98","state":"active+clean","last_fresh":"2021-02-10T04:28:34.051257+0000","last_change":"2021-02-10T04:27:11.606664+0000","last_active":"2021-02-10T04:28:34.051257+0000","last_peered":"2021-02-10T04:28:34.051257+0000","last_clean":"2021-02-10T04:28:34.051257+0000","last_became_active":"2021-02-10T04:25:51.025882+0000","last_became_peered":"2021-02-10T04:25:51.025882+0000","last_unstale":"2021-02-10T04:28:34.051257+0000","last_undegraded":"2021-02-10T04:28:34.051257+0000","last_fullsized":"2021-02-10T04:28:34.051257+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:11.606603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:11.606603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"62'2","reported_seq":"102","reported_epoch":"98","state":"active+clean","last_fresh":"2021-02-10T04:28:35.174151+0000","last_change":"2021-02-10T04:28:33.618291+0000","last_active":"2021-02-10T04:28:35.174151+0000","last_peered":"2021-02-10T04:28:35.174151+0000","last_clean":"2021-02-10T04:28:35.174151+0000","last_became_active":"2021-02-10T04:25:34.331199+0000","last_became_peered":"2021-02-10T04:25:34.331199+0000","last_unstale":"2021-02-10T04:28:35.174151+0000","last_undegraded":"2021-02-10T04:28:35.174151+0000","last_fullsized":"2021-02-10T04:28:35.174151+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2021-02-10T04:28:33.618170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:33.618170+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"63'29","reported_seq":"112","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.379498+0000","last_change":"2021-02-10T04:27:13.081465+0000","last_active":"2021-02-10T04:28:31.379498+0000","last_peered":"2021-02-10T04:28:31.379498+0000","last_clean":"2021-02-10T04:28:31.379498+0000","last_became_active":"2021-02-10T04:25:49.015913+0000","last_became_peered":"2021-02-10T04:25:49.015913+0000","last_unstale":"2021-02-10T04:28:31.379498+0000","last_undegraded":"2021-02-10T04:28:31.379498+0000","last_fullsized":"2021-02-10T04:28:31.379498+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'29","last_scrub_stamp":"2021-02-10T04:27:13.081399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:13.081399+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":56,"num_read_kb":36,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"98'3","reported_seq":"18","reported_epoch":"98","state":"active+clean","last_fresh":"2021-02-10T04:28:34.536023+0000","last_change":"2021-02-10T04:28:25.155945+0000","last_active":"2021-02-10T04:28:34.536023+0000","last_peered":"2021-02-10T04:28:34.536023+0000","last_clean":"2021-02-10T04:28:34.536023+0000","last_became_active":"2021-02-10T04:28:25.155557+0000","last_became_peered":"2021-02-10T04:28:25.155557+0000","last_unstale":"2021-02-10T04:28:34.536023+0000","last_undegraded":"2021-02-10T04:28:34.536023+0000","last_fullsized":"2021-02-10T04:28:34.536023+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2638666,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"52","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.451533+0000","last_change":"2021-02-10T04:27:22.512316+0000","last_active":"2021-02-10T04:28:31.451533+0000","last_peered":"2021-02-10T04:28:31.451533+0000","last_clean":"2021-02-10T04:28:31.451533+0000","last_became_active":"2021-02-10T04:25:53.038249+0000","last_became_peered":"2021-02-10T04:25:53.038249+0000","last_unstale":"2021-02-10T04:28:31.451533+0000","last_undegraded":"2021-02-10T04:28:31.451533+0000","last_fullsized":"2021-02-10T04:28:31.451533+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:22.512245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:22.512245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"59","reported_epoch":"98","state":"active+clean","last_fresh":"2021-02-10T04:28:34.050957+0000","last_change":"2021-02-10T04:27:56.492811+0000","last_active":"2021-02-10T04:28:34.050957+0000","last_peered":"2021-02-10T04:28:34.050957+0000","last_clean":"2021-02-10T04:28:34.050957+0000","last_became_active":"2021-02-10T04:25:51.016118+0000","last_became_peered":"2021-02-10T04:25:51.016118+0000","last_unstale":"2021-02-10T04:28:34.050957+0000","last_undegraded":"2021-02-10T04:28:34.050957+0000","last_fullsized":"2021-02-10T04:28:34.050957+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:56.492754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:56.492754+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"67","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:31.102664+0000","last_change":"2021-02-10T04:28:31.102622+0000","last_active":"2021-02-10T04:28:31.102664+0000","last_peered":"2021-02-10T04:28:31.102664+0000","last_clean":"2021-02-10T04:28:31.102664+0000","last_became_active":"2021-02-10T04:25:34.334427+0000","last_became_peered":"2021-02-10T04:25:34.334427+0000","last_unstale":"2021-02-10T04:28:31.102664+0000","last_undegraded":"2021-02-10T04:28:31.102664+0000","last_fullsized":"2021-02-10T04:28:31.102664+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:31.102558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:31.102558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"63'20","reported_seq":"97","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:31.451605+0000","last_change":"2021-02-10T04:27:20.587671+0000","last_active":"2021-02-10T04:28:31.451605+0000","last_peered":"2021-02-10T04:28:31.451605+0000","last_clean":"2021-02-10T04:28:31.451605+0000","last_became_active":"2021-02-10T04:25:49.004346+0000","last_became_peered":"2021-02-10T04:25:49.004346+0000","last_unstale":"2021-02-10T04:28:31.451605+0000","last_undegraded":"2021-02-10T04:28:31.451605+0000","last_fullsized":"2021-02-10T04:28:31.451605+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'20","last_scrub_stamp":"2021-02-10T04:27:20.587602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:20.587602+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":27,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":"12","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:30.690158+0000","last_change":"2021-02-10T04:28:25.149626+0000","last_active":"2021-02-10T04:28:30.690158+0000","last_peered":"2021-02-10T04:28:30.690158+0000","last_clean":"2021-02-10T04:28:30.690158+0000","last_became_active":"2021-02-10T04:28:25.149288+0000","last_became_peered":"2021-02-10T04:28:25.149288+0000","last_unstale":"2021-02-10T04:28:30.690158+0000","last_undegraded":"2021-02-10T04:28:30.690158+0000","last_fullsized":"2021-02-10T04:28:30.690158+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"51","reported_epoch":"96","state":"active+clean","last_fresh":"2021-02-10T04:28:29.865506+0000","last_change":"2021-02-10T04:27:13.006848+0000","last_active":"2021-02-10T04:28:29.865506+0000","last_peered":"2021-02-10T04:28:29.865506+0000","last_clean":"2021-02-10T04:28:29.865506+0000","last_became_active":"2021-02-10T04:25:53.148090+0000","last_became_peered":"2021-02-10T04:25:53.148090+0000","last_unstale":"2021-02-10T04:28:29.865506+0000","last_undegraded":"2021-02-10T04:28:29.865506+0000","last_fullsized":"2021-02-10T04:28:29.865506+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:27:13.006716+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"82","reported_epoch":"98","state":"active+clean","last_fresh":"2021-02-10T04:28:34.051609+0000","last_change":"2021-02-10T04:28:02.435270+0000","last_active":"2021-02-10T04:28:34.051609+0000","last_peered":"2021-02-10T04:28:34.051609+0000","last_clean":"2021-02-10T04:28:34.051609+0000","last_became_active":"2021-02-10T04:25:36.344641+0000","last_became_peered":"2021-02-10T04:25:36.344641+0000","last_unstale":"2021-02-10T04:28:34.051609+0000","last_undegraded":"2021-02-10T04:28:34.051609+0000","last_fullsized":"2021-02-10T04:28:34.051609+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:28:02.435203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:02.435203+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"63'30","reported_seq":"121","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:32.490336+0000","last_change":"2021-02-10T04:28:32.490336+0000","last_active":"2021-02-10T04:28:32.490336+0000","last_peered":"2021-02-10T04:28:32.490336+0000","last_clean":"2021-02-10T04:28:32.490336+0000","last_became_active":"2021-02-10T04:25:49.002058+0000","last_became_peered":"2021-02-10T04:25:49.002058+0000","last_unstale":"2021-02-10T04:28:32.490336+0000","last_undegraded":"2021-02-10T04:28:32.490336+0000","last_fullsized":"2021-02-10T04:28:32.490336+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2021-02-10T04:28:32.490246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:32.490246+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":59,"num_read_kb":42,"num_write":31,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"58'2","reported_seq":"89","reported_epoch":"97","state":"active+clean","last_fresh":"2021-02-10T04:28:32.734147+0000","last_change":"2021-02-10T04:27:20.144947+0000","last_active":"2021-02-10T04:28:32.734147+0000","last_peered":"2021-02-10T04:28:32.734147+0000","last_clean":"2021-02-10T04:28:32.734147+0000","last_became_active":"2021-02-10T04:25:51.019515+0000","last_became_peered":"2021-02-10T04:25:51.019515+0000","last_unstale":"2021-02-10T04:28:32.734147+0000","last_undegraded":"2021-02-10T04:28:32.734147+0000","last_fullsized":"2021-02-10T04:28:32.734147+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:27:20.144855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:20.144855+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":71664297,"num_objects":30,"num_object_clones":1,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":61,"num_read_kb":31,"num_write":126,"num_write_kb":36146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":117358592,"data_stored":116779376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":1922,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":26,"num_read_kb":26,"num_write":20,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":5835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25,"ondisk_log_size":25,"up":126,"acting":126,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":161,"num_read_kb":156,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":1453,"num_read_kb":926,"num_write":734,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":756,"ondisk_log_size":756,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748364838,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":15612,"kb_used_data":9588,"kb_used_omap":0,"kb_used_meta":6016,"kb_avail":93729540,"statfs":{"total":95995035648,"available":95979048960,"internally_reserved":0,"allocated":9818112,"data_stored":9156256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6160384},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.876,"15min":0.88200000000000001},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.85599999999999998,"15min":0.85899999999999999},"min":{"1min":0.57499999999999996,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.248,"5min":1.248,"15min":1.248},"last":0.86799999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.88700000000000001,"15min":0.89100000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.0149999999999999,"15min":1.0209999999999999},"min":{"1min":0.64500000000000002,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.355,"5min":1.369,"15min":1.369},"last":1.0660000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.018,"15min":1.026},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":1.2170000000000001},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.95799999999999996,"15min":0.95999999999999996},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":1.1619999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.998,"15min":1.01},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":1.248},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.147,"15min":1.153},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.381,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":1.4099999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":0.97499999999999998,"15min":0.95099999999999996},"min":{"1min":0.77300000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.2530000000000001,"5min":2.2530000000000001,"15min":2.2530000000000001},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.1060000000000001,"15min":1.1120000000000001},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.492,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":1.387}]},{"osd":5,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.161,"15min":1.1499999999999999},"min":{"1min":0.68899999999999995,"5min":0.624,"15min":0.624},"max":{"1min":1.714,"5min":1.714,"15min":1.714},"last":1.544},{"interface":"front","average":{"1min":1.208,"5min":1.2,"15min":1.1990000000000001},"min":{"1min":0.90900000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.498,"5min":1.538,"15min":1.538},"last":1.446}]},{"osd":6,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.246,"15min":1.2430000000000001},"min":{"1min":0.94199999999999995,"5min":0.86899999999999999,"15min":0.86899999999999999},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":1.5669999999999999},{"interface":"front","average":{"1min":1.276,"5min":1.236,"15min":1.2290000000000001},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":1.5229999999999999}]}]},{"osd":6,"up_from":43,"seq":184683593770,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6944,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":5952,"kb_avail":93738208,"statfs":{"total":95995035648,"available":95987924992,"internally_reserved":0,"allocated":1007616,"data_stored":365266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6094848},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:28:22 2021","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.95399999999999996,"15min":0.95099999999999996},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.6160000000000001,"5min":1.6160000000000001,"15min":1.6160000000000001},"last":1.4079999999999999},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.99099999999999999,"15min":1.0109999999999999},"min":{"1min":0.626,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.6950000000000001,"5min":1.6950000000000001,"15min":1.6950000000000001},"last":0.75900000000000001}]},{"osd":1,"last update":"Wed Feb 10 04:28:22 2021","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.0089999999999999,"15min":1.028},"min":{"1min":0.622,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.6919999999999999,"5min":2.121,"15min":2.121},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.0109999999999999,"15min":0.98899999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.8009999999999999,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":0.93600000000000005}]},{"osd":2,"last update":"Wed Feb 10 04:28:22 2021","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.0740000000000001,"15min":1.0649999999999999},"min":{"1min":0.64500000000000002,"5min":0.625,"15min":0.625},"max":{"1min":1.919,"5min":1.919,"15min":1.919},"last":1.143},{"interface":"front","average":{"1min":1.137,"5min":1.0549999999999999,"15min":1.0229999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.5700000000000001,"5min":2.1920000000000002,"15min":2.1920000000000002},"last":0.63}]},{"osd":3,"last update":"Wed Feb 10 04:28:22 2021","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.1639999999999999,"15min":1.145},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.3120000000000001},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.159,"15min":1.1160000000000001},"min":{"1min":0.876,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":2.0779999999999998,"5min":2.323,"15min":2.323},"last":1.2829999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:28:22 2021","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.0469999999999999,"15min":0.97899999999999998},"min":{"1min":0.66500000000000004,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.8899999999999999,"5min":2.0800000000000001,"15min":2.0800000000000001},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.0940000000000001,"5min":0.94699999999999995,"15min":0.90600000000000003},"min":{"1min":0.76600000000000001,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.536,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":1.244}]},{"osd":5,"last update":"Wed Feb 10 04:28:22 2021","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.0249999999999999,"15min":0.995},"min":{"1min":0.68600000000000005,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.5900000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":0.96999999999999997},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.23,"15min":1.1559999999999999},"min":{"1min":0.81799999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.8320000000000001,"5min":2.4550000000000001,"15min":2.4550000000000001},"last":1.1020000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:27:40 2021","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.1060000000000001,"15min":1.0940000000000001},"min":{"1min":0.75800000000000001,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.901,"5min":1.901,"15min":1.901},"last":1.4279999999999999},{"interface":"front","average":{"1min":1.506,"5min":1.409,"15min":1.393},"min":{"1min":0.90300000000000002,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":2.5339999999999998,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":1.5349999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":11636,"kb_used_data":1068,"kb_used_omap":0,"kb_used_meta":10560,"kb_avail":93733516,"statfs":{"total":95995035648,"available":95983120384,"internally_reserved":0,"allocated":1093632,"data_stored":368300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":10813440},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:56 2021","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.69099999999999995,"15min":0.63400000000000001},"min":{"1min":0.51800000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.179,"5min":1.425,"15min":1.425},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.78400000000000003,"15min":0.70499999999999996},"min":{"1min":0.54200000000000004,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":2.391,"5min":2.391,"15min":2.391},"last":0.71999999999999997}]},{"osd":2,"last update":"Wed Feb 10 04:28:15 2021","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.81699999999999995,"15min":0.76700000000000002},"min":{"1min":0.56699999999999995,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.1990000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.84099999999999997,"15min":0.80100000000000005},"min":{"1min":0.53800000000000003,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.2310000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":1.0589999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:28:31 2021","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.94099999999999995,"15min":0.91700000000000004},"min":{"1min":0.64900000000000002,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.4790000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.95299999999999996,"5min":1.006,"15min":1.01},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.1259999999999999,"5min":2.226,"15min":2.226},"last":1.0289999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:27:46 2021","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.157,"15min":1.1459999999999999},"min":{"1min":0.86699999999999999,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":1.736,"5min":2.004,"15min":2.004},"last":1.177},{"interface":"front","average":{"1min":1.23,"5min":1.218,"15min":1.21},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.6419999999999999,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":1.393}]},{"osd":5,"last update":"Wed Feb 10 04:28:09 2021","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.1930000000000001,"15min":1.258},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.401,"5min":5.5860000000000003,"15min":5.5860000000000003},"last":1.1359999999999999},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.524,"15min":1.627},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.736,"5min":9.5310000000000006,"15min":9.5310000000000006},"last":1.4099999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:28:25 2021","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.2689999999999999,"15min":1.2829999999999999},"min":{"1min":0.82499999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":1.5900000000000001,"5min":2.4449999999999998,"15min":2.4449999999999998},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.22,"15min":1.278},"min":{"1min":0.86099999999999999,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.4690000000000001,"5min":3.653,"15min":3.653},"last":1.218}]},{"osd":7,"last update":"Wed Feb 10 04:27:36 2021","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.639,"15min":1.6890000000000001},"min":{"1min":1.0529999999999999,"5min":1.004,"15min":1.004},"max":{"1min":2.2090000000000001,"5min":7.5540000000000003,"15min":7.5540000000000003},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.623,"15min":1.6619999999999999},"min":{"1min":1.006,"5min":1.006,"15min":1.006},"max":{"1min":2.556,"5min":8.1020000000000003,"15min":8.1020000000000003},"last":1.4570000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738433,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":39684,"kb_used_data":29180,"kb_used_omap":0,"kb_used_meta":10496,"kb_avail":93705468,"statfs":{"total":95995035648,"available":95954399232,"internally_reserved":0,"allocated":29880320,"data_stored":29085446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":10747904},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 04:27:55 2021","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.67800000000000005,"15min":0.59799999999999998},"min":{"1min":0.502,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.91600000000000004,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":3.4670000000000001},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.81000000000000005,"15min":0.73899999999999999},"min":{"1min":0.63,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.0740000000000001,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":7.5549999999999997}]},{"osd":2,"last update":"Wed Feb 10 04:28:17 2021","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.79300000000000004,"15min":0.77300000000000002},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.98499999999999999,"5min":1.958,"15min":1.958},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.78400000000000003,"15min":0.78900000000000003},"min":{"1min":0.51800000000000002,"5min":0.495,"15min":0.495},"max":{"1min":1.03,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.79500000000000004}]},{"osd":3,"last update":"Wed Feb 10 04:28:29 2021","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.78300000000000003,"15min":0.80500000000000005},"min":{"1min":0.53500000000000003,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0680000000000001,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.95599999999999996,"5min":1.0089999999999999,"15min":1.03},"min":{"1min":0.59099999999999997,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.377,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":1.4950000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:27:45 2021","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.1259999999999999,"15min":1.169},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.3859999999999999,"5min":1.613,"15min":1.613},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.089,"5min":1.141,"15min":1.1539999999999999},"min":{"1min":0.76600000000000001,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.522,"5min":1.597,"15min":1.597},"last":1.7529999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:28:02 2021","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.1970000000000001,"15min":1.218},"min":{"1min":0.75800000000000001,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.593,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":1.893},{"interface":"front","average":{"1min":1.216,"5min":1.2709999999999999,"15min":1.2949999999999999},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.51,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.3149999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:28:22 2021","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.2849999999999999,"15min":1.3009999999999999},"min":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":1.593,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":2.0409999999999999},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.343,"15min":1.361},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":1.704,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":1.6779999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.2330000000000001,"15min":1.252},"min":{"1min":0.94599999999999995,"5min":0.93500000000000005,"15min":0.93500000000000005},"max":{"1min":1.5900000000000001,"5min":1.76,"15min":1.76},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3500000000000001,"15min":1.3640000000000001},"min":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"max":{"1min":1.611,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":2.194}]}]},{"osd":2,"up_from":18,"seq":77309411386,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":24888,"kb_used_data":18448,"kb_used_omap":0,"kb_used_meta":6080,"kb_avail":93720264,"statfs":{"total":95995035648,"available":95969550336,"internally_reserved":0,"allocated":18890752,"data_stored":18226016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6225920},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:28:12 2021","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.71599999999999997,"15min":0.68500000000000005},"min":{"1min":0.497,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.2430000000000001,"5min":1.349,"15min":1.349},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.73299999999999998,"15min":0.71099999999999997},"min":{"1min":0.53800000000000003,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0609999999999999,"5min":1.274,"15min":1.274},"last":0.70999999999999996}]},{"osd":1,"last update":"Wed Feb 10 04:28:12 2021","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.81799999999999995,"15min":0.77100000000000002},"min":{"1min":0.51200000000000001,"5min":0.495,"15min":0.495},"max":{"1min":1.0029999999999999,"5min":1.726,"15min":1.726},"last":1.1200000000000001},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71899999999999997,"15min":0.69299999999999995},"min":{"1min":0.51200000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.95599999999999996,"5min":1.194,"15min":1.194},"last":0.58699999999999997}]},{"osd":3,"last update":"Wed Feb 10 04:28:33 2021","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.82199999999999995,"15min":0.80600000000000005},"min":{"1min":0.56100000000000005,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.361,"5min":1.361,"15min":1.361},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.008,"15min":0.97499999999999998},"min":{"1min":0.72699999999999998,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.76800000000000002}]},{"osd":4,"last update":"Wed Feb 10 04:27:44 2021","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.181,"15min":1.181},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.4359999999999999,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.877},{"interface":"front","average":{"1min":1.141,"5min":1.119,"15min":1.1220000000000001},"min":{"1min":0.80900000000000005,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":0.97399999999999998}]},{"osd":5,"last update":"Wed Feb 10 04:28:04 2021","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.2829999999999999,"15min":1.3049999999999999},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.508,"5min":1.9379999999999999,"15min":1.9379999999999999},"last":1},{"interface":"front","average":{"1min":1.137,"5min":1.1799999999999999,"15min":1.179},"min":{"1min":0.875,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.478,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":1.246}]},{"osd":6,"last update":"Wed Feb 10 04:28:26 2021","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.2649999999999999,"15min":1.2929999999999999},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.794,"5min":1.794,"15min":1.794},"last":1.3300000000000001},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.3919999999999999,"15min":1.4279999999999999},"min":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"max":{"1min":1.843,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.268}]},{"osd":7,"last update":"Wed Feb 10 04:27:37 2021","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.4690000000000001,"15min":1.494},"min":{"1min":1.077,"5min":1.073,"15min":1.073},"max":{"1min":1.653,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.171,"5min":1.252,"15min":1.266},"min":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"max":{"1min":1.52,"5min":1.6160000000000001,"15min":1.6160000000000001},"last":1.105}]}]},{"osd":3,"up_from":25,"seq":107374182452,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":22088,"kb_used_data":11712,"kb_used_omap":0,"kb_used_meta":10368,"kb_avail":93723064,"statfs":{"total":95995035648,"available":95972417536,"internally_reserved":0,"allocated":11993088,"data_stored":11197766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":10616832},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:28:33 2021","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.73699999999999999,"15min":0.73299999999999998},"min":{"1min":0.49199999999999999,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.85199999999999998,"15min":0.85699999999999998},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.006,"5min":4.2759999999999998,"15min":4.2759999999999998},"last":0.76000000000000001}]},{"osd":1,"last update":"Wed Feb 10 04:28:33 2021","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.88400000000000001,"15min":0.871},"min":{"1min":0.54400000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.069,"5min":3.246,"15min":3.246},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.873,"15min":0.85399999999999998},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.117,"5min":5.4039999999999999,"15min":5.4039999999999999},"last":0.79100000000000004}]},{"osd":2,"last update":"Wed Feb 10 04:28:33 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.59999999999999998,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.095,"5min":4.2309999999999999,"15min":4.2309999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.94599999999999995,"15min":0.97099999999999997},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.2190000000000001,"5min":1.681,"15min":1.681},"last":0.70899999999999996}]},{"osd":4,"last update":"Wed Feb 10 04:27:43 2021","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.25,"15min":1.204},"min":{"1min":0.84099999999999997,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.9650000000000001,"5min":3.371,"15min":3.371},"last":1.4399999999999999},{"interface":"front","average":{"1min":1.266,"5min":1.266,"15min":1.2250000000000001},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":1.8280000000000001,"5min":3.5430000000000001,"15min":3.5430000000000001},"last":1.946}]},{"osd":5,"last update":"Wed Feb 10 04:28:01 2021","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.232,"15min":1.1919999999999999},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.474,"5min":3.496,"15min":3.496},"last":1.575},{"interface":"front","average":{"1min":1.254,"5min":1.2989999999999999,"15min":1.272},"min":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"max":{"1min":1.6819999999999999,"5min":3.742,"15min":3.742},"last":1.536}]},{"osd":6,"last update":"Wed Feb 10 04:28:27 2021","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.397,"15min":1.395},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.726,"5min":3.6859999999999999,"15min":3.6859999999999999},"last":1.5},{"interface":"front","average":{"1min":1.268,"5min":1.431,"15min":1.4490000000000001},"min":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":1.7649999999999999,"5min":3.8759999999999999,"15min":3.8759999999999999},"last":1.667}]},{"osd":7,"last update":"Wed Feb 10 04:27:40 2021","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":1.5109999999999999,"15min":1.5049999999999999},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"max":{"1min":4.1319999999999997,"5min":4.1319999999999997,"15min":4.1319999999999997},"last":1.3720000000000001},{"interface":"front","average":{"1min":1.587,"5min":1.5629999999999999,"15min":1.5589999999999999},"min":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"max":{"1min":4.0010000000000003,"5min":4.0010000000000003,"15min":4.0010000000000003},"last":1.8009999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986226,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":25752,"kb_used_data":15440,"kb_used_omap":0,"kb_used_meta":10304,"kb_avail":93719400,"statfs":{"total":95995035648,"available":95968665600,"internally_reserved":0,"allocated":15810560,"data_stored":15253568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":10551296},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:27:43 2021","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.96599999999999997,"15min":0.997},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.95499999999999996,"15min":0.97499999999999998},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.4810000000000001,"5min":1.633,"15min":1.633},"last":0.996}]},{"osd":1,"last update":"Wed Feb 10 04:27:43 2021","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.89500000000000002,"15min":0.91700000000000004},"min":{"1min":0.58199999999999996,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.95399999999999996},{"interface":"front","average":{"1min":1.02,"5min":1.012,"15min":1.018},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.583,"5min":1.583,"15min":1.583},"last":1.3100000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:27:43 2021","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.0800000000000001,"15min":1.083},"min":{"1min":0.69399999999999995,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.766,"5min":1.766,"15min":1.766},"last":1.464},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.1719999999999999,"15min":1.161},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.5469999999999999,"5min":1.768,"15min":1.768},"last":1.6870000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:27:43 2021","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.097,"15min":1.0609999999999999},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":5.1230000000000002,"5min":5.1230000000000002,"15min":5.1230000000000002},"last":1.407},{"interface":"front","average":{"1min":1.302,"5min":1.129,"15min":1.1080000000000001},"min":{"1min":0.79900000000000004,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":5.1710000000000003,"5min":5.1710000000000003,"15min":5.1710000000000003},"last":1.343}]},{"osd":5,"last update":"Wed Feb 10 04:28:05 2021","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":0.93500000000000005,"15min":0.85499999999999998},"min":{"1min":0.63500000000000001,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.5169999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":1.4930000000000001},{"interface":"front","average":{"1min":0.998,"5min":0.90200000000000002,"15min":0.84799999999999998},"min":{"1min":0.66800000000000004,"5min":0.624,"15min":0.624},"max":{"1min":1.3759999999999999,"5min":1.5740000000000001,"15min":1.5740000000000001},"last":1.4470000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:28:21 2021","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.1399999999999999,"15min":1.083},"min":{"1min":0.878,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.9019999999999999,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":1.6000000000000001},{"interface":"front","average":{"1min":1.452,"5min":1.3120000000000001,"15min":1.266},"min":{"1min":1.0880000000000001,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":1.9350000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:27:40 2021","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.94899999999999995,"5min":0.88600000000000001,"15min":0.88600000000000001},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":1.8100000000000001},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3720000000000001,"15min":1.3740000000000001},"min":{"1min":1.073,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":1.6699999999999999,"5min":1.9450000000000001,"15min":1.9450000000000001},"last":2.0499999999999998}]}]},{"osd":5,"up_from":37,"seq":158913789998,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6808,"kb_used_data":912,"kb_used_omap":0,"kb_used_meta":5888,"kb_avail":93738344,"statfs":{"total":95995035648,"available":95988064256,"internally_reserved":0,"allocated":933888,"data_stored":361274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6029312},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:28:04 2021","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.97999999999999998,"15min":0.995},"min":{"1min":0.61299999999999999,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":1.081},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.90800000000000003,"15min":0.91000000000000003},"min":{"1min":0.65300000000000002,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":1.2909999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:28:04 2021","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.92900000000000005,"15min":0.93200000000000005},"min":{"1min":0.63800000000000001,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":2.262,"5min":2.262,"15min":2.262},"last":1.613},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.98699999999999999,"15min":0.998},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.53,"5min":1.53,"15min":1.53},"last":1.0509999999999999}]},{"osd":2,"last update":"Wed Feb 10 04:28:04 2021","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.004,"15min":0.998},"min":{"1min":0.64100000000000001,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":1.329},{"interface":"front","average":{"1min":1.129,"5min":1.01,"15min":0.98999999999999999},"min":{"1min":0.77800000000000002,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":2.383,"5min":2.383,"15min":2.383},"last":1.119}]},{"osd":3,"last update":"Wed Feb 10 04:28:04 2021","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.119,"15min":1.099},"min":{"1min":0.871,"5min":0.753,"15min":0.753},"max":{"1min":1.4430000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":1.669},{"interface":"front","average":{"1min":1.208,"5min":1.1000000000000001,"15min":1.079},"min":{"1min":0.75800000000000001,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":2.516,"5min":2.516,"15min":2.516},"last":1.391}]},{"osd":4,"last update":"Wed Feb 10 04:28:04 2021","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":0.96499999999999997,"15min":0.92000000000000004},"min":{"1min":0.72799999999999998,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.718,"5min":1.718,"15min":1.718},"last":1.456},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.0820000000000001,"15min":0.99099999999999999},"min":{"1min":0.81999999999999995,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":2.6560000000000001,"5min":2.6560000000000001,"15min":2.6560000000000001},"last":1.7729999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:28:24 2021","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.212,"15min":1.1399999999999999},"min":{"1min":0.85499999999999998,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.786,"5min":2.786,"15min":2.786},"last":1.8380000000000001},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.139,"15min":1.073},"min":{"1min":0.77800000000000002,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.4550000000000001,"5min":2.4550000000000001,"15min":2.4550000000000001},"last":1.736}]},{"osd":7,"last update":"Wed Feb 10 04:27:38 2021","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.1910000000000001,"15min":1.177},"min":{"1min":0.93400000000000005,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.2440000000000002,"5min":2.2440000000000002,"15min":2.2440000000000002},"last":1.6459999999999999},{"interface":"front","average":{"1min":1.528,"5min":1.3919999999999999,"15min":1.3700000000000001},"min":{"1min":1.0169999999999999,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":2.2799999999999998,"5min":2.2799999999999998,"15min":2.2799999999999998},"last":1.901}]}]}]}} 2021-02-10T04:28:40.991 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2021-02-10T04:28:40.991 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pool set device_health_metrics pg_num 11 2021-02-10T04:28:41.151 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:28:41.217 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[45443]: debug 2021-02-10T04:28:41.097+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:41.220 INFO:tasks.rados.rados.0.smithi085.stdout:75: done (1 left) 2021-02-10T04:28:41.220 INFO:tasks.rados.rados.0.smithi085.stdout:76: done (0 left) 2021-02-10T04:28:41.220 INFO:tasks.rados.rados.0.smithi085.stdout:77: copy_from oid 22 from oid 49 current snap is 4 2021-02-10T04:28:41.222 INFO:tasks.rados.rados.0.smithi085.stdout:78: read oid 39 snap 3 2021-02-10T04:28:41.222 INFO:tasks.rados.rados.0.smithi085.stdout:78: expect (ObjNum 25 snap 0 seq_num 25) 2021-02-10T04:28:41.222 INFO:tasks.rados.rados.0.smithi085.stdout:79: write oid 30 current snap is 4 2021-02-10T04:28:41.222 INFO:tasks.rados.rados.0.smithi085.stdout:79: seq_num 58 ranges {574806=778913,2061132=557342} 2021-02-10T04:28:41.230 INFO:tasks.rados.rados.0.smithi085.stdout:79: writing smithi08575397-30 from 574806 to 1353719 tid 1 2021-02-10T04:28:41.243 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:28:41.243 INFO:tasks.rados.rados.0.smithi085.stdout:79: writing smithi08575397-30 from 2061132 to 2618474 tid 2 2021-02-10T04:28:41.243 INFO:tasks.rados.rados.0.smithi085.stdout:80: write (excl) oid 29 current snap is 4 2021-02-10T04:28:41.244 INFO:tasks.rados.rados.0.smithi085.stdout:80: seq_num 59 ranges {591613=454162,1719614=417353,2570780=420833} 2021-02-10T04:28:41.249 INFO:tasks.rados.rados.0.smithi085.stdout:80: writing smithi08575397-29 from 591613 to 1045775 tid 1 2021-02-10T04:28:41.253 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.2 dump_ops_in_flight 2021-02-10T04:28:41.256 INFO:tasks.rados.rados.0.smithi085.stdout:80: writing smithi08575397-29 from 1719614 to 2136967 tid 2 2021-02-10T04:28:41.257 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:28:41.261 INFO:tasks.rados.rados.0.smithi085.stdout:80: writing smithi08575397-29 from 2570780 to 2991613 tid 3 2021-02-10T04:28:41.261 INFO:tasks.rados.rados.0.smithi085.stdout:81: read oid 36 snap 3 2021-02-10T04:28:41.261 INFO:tasks.rados.rados.0.smithi085.stdout:81: expect (ObjNum 35 snap 0 seq_num 35) 2021-02-10T04:28:41.261 INFO:tasks.rados.rados.0.smithi085.stdout:78: expect (ObjNum 25 snap 0 seq_num 25) 2021-02-10T04:28:41.311 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:41.366 INFO:tasks.rados.rados.0.smithi085.stdout:78: done (4 left) 2021-02-10T04:28:41.366 INFO:tasks.rados.rados.0.smithi085.stdout:82: copy_from oid 4 from oid 1 current snap is 4 2021-02-10T04:28:41.366 INFO:tasks.rados.rados.0.smithi085.stdout:79: finishing write tid 1 to smithi08575397-30 2021-02-10T04:28:41.366 INFO:tasks.rados.rados.0.smithi085.stdout:77: finishing copy_from to smithi08575397-22 2021-02-10T04:28:41.366 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 12 (ObjNum 48 snap 0 seq_num 48) dirty exists 2021-02-10T04:28:41.367 INFO:tasks.rados.rados.0.smithi085.stdout:77: finishing copy_from racing read to smithi08575397-22 2021-02-10T04:28:41.367 INFO:tasks.rados.rados.0.smithi085.stdout:79: finishing write tid 2 to smithi08575397-30 2021-02-10T04:28:41.367 INFO:tasks.rados.rados.0.smithi085.stdout:79: finishing write tid 3 to smithi08575397-30 2021-02-10T04:28:41.367 INFO:tasks.rados.rados.0.smithi085.stdout:79: finishing write tid 4 to smithi08575397-30 2021-02-10T04:28:41.367 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 16 (ObjNum 58 snap 4 seq_num 58) dirty exists 2021-02-10T04:28:41.367 INFO:tasks.rados.rados.0.smithi085.stdout:79: left oid 30 (ObjNum 58 snap 4 seq_num 58) 2021-02-10T04:28:41.367 INFO:tasks.rados.rados.0.smithi085.stdout:80: finishing write tid 1 to smithi08575397-29 2021-02-10T04:28:41.367 INFO:tasks.rados.rados.0.smithi085.stdout:80: finishing write tid 2 to smithi08575397-29 2021-02-10T04:28:41.367 INFO:tasks.rados.rados.0.smithi085.stdout:80: finishing write tid 3 to smithi08575397-29 2021-02-10T04:28:41.368 INFO:tasks.rados.rados.0.smithi085.stdout:80: finishing write tid 4 to smithi08575397-29 2021-02-10T04:28:41.368 INFO:tasks.rados.rados.0.smithi085.stdout:80: finishing write tid 5 to smithi08575397-29 2021-02-10T04:28:41.368 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 29 v 18 (ObjNum 59 snap 4 seq_num 59) dirty exists 2021-02-10T04:28:41.368 INFO:tasks.rados.rados.0.smithi085.stdout:80: left oid 29 (ObjNum 59 snap 4 seq_num 59) 2021-02-10T04:28:41.368 INFO:tasks.rados.rados.0.smithi085.stdout:81: expect (ObjNum 35 snap 0 seq_num 35) 2021-02-10T04:28:41.467 INFO:tasks.rados.rados.0.smithi085.stdout:77: done (4 left) 2021-02-10T04:28:41.467 INFO:tasks.rados.rados.0.smithi085.stdout:79: done (3 left) 2021-02-10T04:28:41.467 INFO:tasks.rados.rados.0.smithi085.stdout:80: done (2 left) 2021-02-10T04:28:41.467 INFO:tasks.rados.rados.0.smithi085.stdout:81: done (1 left) 2021-02-10T04:28:41.468 INFO:tasks.rados.rados.0.smithi085.stdout:83: rollback oid 34 current snap is 4 2021-02-10T04:28:41.468 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 34 to 3 2021-02-10T04:28:41.468 INFO:tasks.rados.rados.0.smithi085.stdout:84: read oid 19 snap -1 2021-02-10T04:28:41.468 INFO:tasks.rados.rados.0.smithi085.stdout:84: expect (ObjNum 18 snap 0 seq_num 18) 2021-02-10T04:28:41.468 INFO:tasks.rados.rados.0.smithi085.stdout:85: read oid 42 snap -1 2021-02-10T04:28:41.468 INFO:tasks.rados.rados.0.smithi085.stdout:85: expect (ObjNum 41 snap 0 seq_num 41) 2021-02-10T04:28:41.468 INFO:tasks.rados.rados.0.smithi085.stdout:86: read oid 24 snap 3 2021-02-10T04:28:41.468 INFO:tasks.rados.rados.0.smithi085.stdout:86: expect (ObjNum 23 snap 0 seq_num 23) 2021-02-10T04:28:41.468 INFO:tasks.rados.rados.0.smithi085.stdout:87: read oid 31 snap -1 2021-02-10T04:28:41.468 INFO:tasks.rados.rados.0.smithi085.stdout:87: expect (ObjNum 51 snap 0 seq_num 51) 2021-02-10T04:28:41.468 INFO:tasks.rados.rados.0.smithi085.stdout:88: copy_from oid 50 from oid 11 current snap is 4 2021-02-10T04:28:41.468 INFO:tasks.rados.rados.0.smithi085.stdout:89: read oid 18 snap -1 2021-02-10T04:28:41.468 INFO:tasks.rados.rados.0.smithi085.stdout:89: expect (ObjNum 17 snap 0 seq_num 17) 2021-02-10T04:28:41.469 INFO:tasks.rados.rados.0.smithi085.stdout:90: read oid 20 snap -1 2021-02-10T04:28:41.469 INFO:tasks.rados.rados.0.smithi085.stdout:90: expect (ObjNum 19 snap 0 seq_num 19) 2021-02-10T04:28:41.469 INFO:tasks.rados.rados.0.smithi085.stdout:91: write oid 22 current snap is 4 2021-02-10T04:28:41.469 INFO:tasks.rados.rados.0.smithi085.stdout:91: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2021-02-10T04:28:41.470 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[52836]: debug 2021-02-10T04:28:41.470 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[52836]: 2021-02-10T04:28:41.415+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:41.470 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: cluster 2021-02-10T04:28:41.470 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: 2021-02-10T04:28:41.470 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: 2021 2021-02-10T04:28:41.470 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: - 2021-02-10T04:28:41.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: 02 2021-02-10T04:28:41.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: - 2021-02-10T04:28:41.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: 10 2021-02-10T04:28:41.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: T 2021-02-10T04:28:41.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: 04 2021-02-10T04:28:41.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: : 2021-02-10T04:28:41.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: 28 2021-02-10T04:28:41.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: : 2021-02-10T04:28:41.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: 40 2021-02-10T04:28:41.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: . 2021-02-10T04:28:41.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: 098233 2021-02-10T04:28:41.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: +0000 2021-02-10T04:28:41.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: 2021-02-10T04:28:41.472 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: osd.1 (osd.1) 37 : cluster [DBG] 2021-02-10T04:28:41.472 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: 5.19 scrub ok 2021-02-10T04:28:41.472 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: cluster 2021-02-10T04:28:40. 2021-02-10T04:28:41.472 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: 322499+0000 osd.6 (osd.6) 27 : cluster [DBG] 2021-02-10T04:28:41.472 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: 4.5 scrub ok 2021-02-10T04:28:41.472 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: cluster 2021-02-10T04:28 2021-02-10T04:28:41.472 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: :40.422103+0000 osd.4 (osd.4) 33 2021-02-10T04:28:41.472 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: : cluster [DBG] 2.25 scrub ok 2021-02-10T04:28:41.472 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: cluster 2021-02-10T04:28:41.472 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: 2021-02-10T04:28:40. 2021-02-10T04:28:41.472 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: 480285+0000 osd.4 (osd.4) 34 : 2021-02-10T04:28:41.472 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: cluster [DBG] purged_snaps scrub starts 2021-02-10T04:28:41.472 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:28:41.472 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: 02-10T04:28:40.480323+0000 2021-02-10T04:28:41.473 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: osd.4 (osd.4) 35 : cluster [DBG] 2021-02-10T04:28:41.473 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[29319]: purged_snaps scrub ok 2021-02-10T04:28:41.473 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[35961]: cluster 2021-02-10T04:28:40.098233+0000 osd.1 ( 2021-02-10T04:28:41.473 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[35961]: osd.1) 37 : cluster [DBG] 5.19 scrub ok 2021-02-10T04:28:41.473 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[35961]: cluster 2021-02-10T04:28:40.322499+0000 osd.6 (osd.6) 27 : cluster 2021-02-10T04:28:41.473 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[35961]: [DBG] 4.5 scrub ok 2021-02-10T04:28:41.473 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:28:41.473 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[35961]: 10T04:28:40.422103+0000 2021-02-10T04:28:41.473 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[35961]: osd.4 (osd.4) 33 : cluster 2021-02-10T04:28:41.474 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[35961]: [DBG] 2.25 scrub ok 2021-02-10T04:28:41.474 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[35961]: cluster 2021-02-10T04:28:41.474 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[35961]: 2021-02-10T04:28 2021-02-10T04:28:41.474 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[35961]: :40.480285+0000 osd.4 (osd.4 2021-02-10T04:28:41.474 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[35961]: ) 34 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:28:41.474 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:28:41.474 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:41 smithi085 conmon[35961]: 10T04:28:40.480323+0000 osd.4 (osd.4) 35 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:28:41.474 INFO:tasks.rados.rados.0.smithi085.stdout:91: writing smithi08575397-22 from 608420 to 1137831 tid 1 2021-02-10T04:28:41.480 INFO:tasks.rados.rados.0.smithi085.stdout:91: writing smithi08575397-22 from 1778096 to 2339107 tid 2 2021-02-10T04:28:41.484 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:28:41.485 INFO:tasks.rados.rados.0.smithi085.stdout:91: writing smithi08575397-22 from 2881850 to 3008420 tid 3 2021-02-10T04:28:41.485 INFO:tasks.rados.rados.0.smithi085.stdout:92: read oid 49 snap -1 2021-02-10T04:28:41.485 INFO:tasks.rados.rados.0.smithi085.stdout:92: expect (ObjNum 48 snap 0 seq_num 48) 2021-02-10T04:28:41.486 INFO:tasks.rados.rados.0.smithi085.stdout:93: write (excl) oid 44 current snap is 4 2021-02-10T04:28:41.486 INFO:tasks.rados.rados.0.smithi085.stdout:93: seq_num 61 ranges {625227=721013,1952931=704669} 2021-02-10T04:28:41.493 INFO:tasks.rados.rados.0.smithi085.stdout:93: writing smithi08575397-44 from 625227 to 1346240 tid 1 2021-02-10T04:28:41.501 INFO:tasks.rados.rados.0.smithi085.stdout:93: writing smithi08575397-44 from 1952931 to 2657600 tid 2 2021-02-10T04:28:41.502 INFO:tasks.rados.rados.0.smithi085.stdout:94: write oid 13 current snap is 4 2021-02-10T04:28:41.502 INFO:tasks.rados.rados.0.smithi085.stdout:94: seq_num 62 ranges {642034=796262,1895060=564680} 2021-02-10T04:28:41.511 INFO:tasks.rados.rados.0.smithi085.stdout:94: writing smithi08575397-13 from 642034 to 1438296 tid 1 2021-02-10T04:28:41.517 INFO:tasks.rados.rados.0.smithi085.stdout:94: writing smithi08575397-13 from 1895060 to 2459740 tid 2 2021-02-10T04:28:41.517 INFO:tasks.rados.rados.0.smithi085.stdout:95: read oid 27 snap -1 2021-02-10T04:28:41.517 INFO:tasks.rados.rados.0.smithi085.stdout:95: expect (ObjNum 26 snap 0 seq_num 26) 2021-02-10T04:28:41.517 INFO:tasks.rados.rados.0.smithi085.stdout:96: copy_from oid 41 from oid 29 current snap is 4 2021-02-10T04:28:41.517 INFO:tasks.rados.rados.0.smithi085.stdout:83: finishing rollback tid 1 to smithi08575397-34 2021-02-10T04:28:41.518 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 34 v 0 (ObjNum 82237904 snap 22030 seq_num 82237936) dirty dne 2021-02-10T04:28:41.518 INFO:tasks.rados.rados.0.smithi085.stdout:85: expect (ObjNum 41 snap 0 seq_num 41) 2021-02-10T04:28:41.570 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:28:41.593 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:28:41.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:41 smithi164 conmon[29288]: cluster 2021-02-10T04:28:40.098233+0000 osd.1 (osd.1) 37 2021-02-10T04:28:41.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:41 smithi164 conmon[29288]: : cluster [DBG] 5.19 scrub ok 2021-02-10T04:28:41.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:41 smithi164 conmon[29288]: cluster 2021-02-10T04:28:40.322499+0000 osd.6 (osd.6) 27 : cluster [DBG] 4.5 scrub ok 2021-02-10T04:28:41.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:41 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:28:41.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:41 smithi164 conmon[29288]: -10T04:28:40.422103+0000 osd.4 (osd.4) 33 : cluster [DBG] 2.25 scrub ok 2021-02-10T04:28:41.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:41 smithi164 conmon[29288]: cluster 2021-02-10T04:28:40.480285 2021-02-10T04:28:41.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:41 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 34 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:28:41.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:41 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:28:41.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:41 smithi164 conmon[29288]: :28:40.480323+0000 osd.4 (osd.4) 35 : cluster 2021-02-10T04:28:41.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:41 smithi164 conmon[29288]: [DBG] purged_snaps scrub ok 2021-02-10T04:28:41.618 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:41.626 INFO:tasks.rados.rados.0.smithi085.stdout:86: expect (ObjNum 23 snap 0 seq_num 23) 2021-02-10T04:28:41.693 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:28:41.721 INFO:tasks.rados.rados.0.smithi085.stdout:90: expect (ObjNum 19 snap 0 seq_num 19) 2021-02-10T04:28:41.798 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:28:41.813 INFO:tasks.rados.rados.0.smithi085.stdout:83: done (14 left) 2021-02-10T04:28:41.813 INFO:tasks.rados.rados.0.smithi085.stdout:85: done (13 left) 2021-02-10T04:28:41.813 INFO:tasks.rados.rados.0.smithi085.stdout:86: done (12 left) 2021-02-10T04:28:41.813 INFO:tasks.rados.rados.0.smithi085.stdout:90: done (11 left) 2021-02-10T04:28:41.813 INFO:tasks.rados.rados.0.smithi085.stdout:97: snap_remove snap 3 2021-02-10T04:28:41.826 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:41.893 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:28:41 smithi164 conmon[33677]: debug 2021-02-10T04:28:41.677+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:42.228 INFO:tasks.rados.rados.0.smithi085.stdout:97: done (11 left) 2021-02-10T04:28:42.228 INFO:tasks.rados.rados.0.smithi085.stdout:98: write oid 16 current snap is 4 2021-02-10T04:28:42.228 INFO:tasks.rados.rados.0.smithi085.stdout:98: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2021-02-10T04:28:42.234 INFO:tasks.rados.rados.0.smithi085.stdout:98: writing smithi08575397-16 from 658841 to 1130352 tid 1 2021-02-10T04:28:42.240 INFO:tasks.rados.rados.0.smithi085.stdout:98: writing smithi08575397-16 from 1553542 to 2261880 tid 2 2021-02-10T04:28:42.245 INFO:tasks.rados.rados.0.smithi085.stdout:98: writing smithi08575397-16 from 2964119 to 3058841 tid 3 2021-02-10T04:28:42.245 INFO:tasks.rados.rados.0.smithi085.stdout:99: read oid 42 snap -1 2021-02-10T04:28:42.245 INFO:tasks.rados.rados.0.smithi085.stdout:99: expect (ObjNum 41 snap 0 seq_num 41) 2021-02-10T04:28:42.245 INFO:tasks.rados.rados.0.smithi085.stdout:100: rollback oid 47 current snap is 4 2021-02-10T04:28:42.245 INFO:tasks.rados.rados.0.smithi085.stdout:100: done (13 left) 2021-02-10T04:28:42.246 INFO:tasks.rados.rados.0.smithi085.stdout:101: read oid 17 snap -1 2021-02-10T04:28:42.246 INFO:tasks.rados.rados.0.smithi085.stdout:101: expect (ObjNum 16 snap 0 seq_num 16) 2021-02-10T04:28:42.246 INFO:tasks.rados.rados.0.smithi085.stdout:102: snap_create 2021-02-10T04:28:42.246 INFO:tasks.rados.rados.0.smithi085.stdout:92: expect (ObjNum 48 snap 0 seq_num 48) 2021-02-10T04:28:42.335 INFO:tasks.rados.rados.0.smithi085.stdout:84: expect (ObjNum 18 snap 0 seq_num 18) 2021-02-10T04:28:42.412 INFO:tasks.rados.rados.0.smithi085.stdout:89: expect (ObjNum 17 snap 0 seq_num 17) 2021-02-10T04:28:42.438 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set noscrub 2021-02-10T04:28:42.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:42 smithi085 conmon[29319]: cluster 2021-02-10T04:28:41.215967+0000 mon.a 2021-02-10T04:28:42.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:42 smithi085 conmon[29319]: (mon.0) 760 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2021-02-10T04:28:42.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:42 smithi085 conmon[29319]: cluster 2021-02-10T04:28:41.375268+0000 osd.7 (osd.7) 35 : cluster [DBG] 2.20 deep-scrub ok 2021-02-10T04:28:42.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:42 smithi085 conmon[29319]: cluster 2021-02-10T04:28:41.427044+0000 osd.4 (osd.4) 36 : cluster [DBG] 4.c scrub ok 2021-02-10T04:28:42.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:42 smithi085 conmon[29319]: cluster 2021-02-10T04:28:41. 2021-02-10T04:28:42.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:42 smithi085 conmon[29319]: 993723+0000 mgr.y (mgr.14140) 336 : cluster [DBG] pgmap v307: 158 pgs: 5 active+clean+snaptrim, 153 active+clean; 114 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 7.5 MiB/s wr, 47 op/s 2021-02-10T04:28:42.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:42 smithi085 conmon[35961]: cluster 2021-02-10T04:28:41.215967 2021-02-10T04:28:42.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:42 smithi085 conmon[35961]: +0000 mon.a (mon.0) 760 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2021-02-10T04:28:42.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:42 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:28:42.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:42 smithi085 conmon[35961]: T04:28:41.375268+0000 osd.7 (osd.7) 35 : cluster [DBG] 2021-02-10T04:28:42.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:42 smithi085 conmon[35961]: 2.20 deep-scrub ok 2021-02-10T04:28:42.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:42 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:28:42.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:42 smithi085 conmon[35961]: 02-10T04:28:41.427044+0000 osd.4 (osd.4) 36 : cluster [DBG] 4.c scrub ok 2021-02-10T04:28:42.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:42 smithi085 conmon[35961]: cluster 2021-02-10T04:28:41 2021-02-10T04:28:42.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:42 smithi085 conmon[35961]: .993723+0000 mgr.y (mgr.14140) 336 : cluster [DBG] pgmap v307: 158 pgs: 5 active+clean+snaptrim, 153 active+clean; 114 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 7.5 MiB/s wr, 47 op/s 2021-02-10T04:28:42.501 INFO:tasks.rados.rados.0.smithi085.stdout:87: expect (ObjNum 51 snap 0 seq_num 51) 2021-02-10T04:28:42.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:42 smithi164 conmon[29288]: cluster 2021-02-10T04:28:42.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:42 smithi164 conmon[29288]: 2021-02-10T04:28:42.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:42 smithi164 conmon[29288]: 2021-02-10T04:28:41.215967+0000 mon.a (mon.0) 760 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2021-02-10T04:28:42.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:42 smithi164 conmon[29288]: cluster 2021-02-10T04:28:41.375268+0000 osd.7 ( 2021-02-10T04:28:42.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:42 smithi164 conmon[29288]: osd.7) 35 : cluster [DBG] 2.20 deep-scrub ok 2021-02-10T04:28:42.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:42 smithi164 conmon[29288]: cluster 2021-02-10T04:28: 2021-02-10T04:28:42.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:42 smithi164 conmon[29288]: 41.427044+0000 osd.4 (osd.4) 36 : cluster [DBG] 4.c scrub ok 2021-02-10T04:28:42.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:42 smithi164 conmon[29288]: 2021-02-10T04:28:42.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:42 smithi164 conmon[29288]: cluster 2021-02-10T04:28:42.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:42 smithi164 conmon[29288]: 2021-02-10T04:28:41.993723+0000 mgr.y (mgr.14140) 336 : cluster [DBG] 2021-02-10T04:28:42.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:42 smithi164 conmon[29288]: pgmap v307: 158 pgs: 5 active+clean+snaptrim, 153 active+clean; 114 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 7.5 MiB/s wr, 47 op/s 2021-02-10T04:28:42.648 INFO:tasks.rados.rados.0.smithi085.stdout:91: finishing write tid 1 to smithi08575397-22 2021-02-10T04:28:42.648 INFO:tasks.rados.rados.0.smithi085.stdout:91: finishing write tid 2 to smithi08575397-22 2021-02-10T04:28:42.649 INFO:tasks.rados.rados.0.smithi085.stdout:91: finishing write tid 3 to smithi08575397-22 2021-02-10T04:28:42.649 INFO:tasks.rados.rados.0.smithi085.stdout:91: finishing write tid 4 to smithi08575397-22 2021-02-10T04:28:42.649 INFO:tasks.rados.rados.0.smithi085.stdout:91: finishing write tid 5 to smithi08575397-22 2021-02-10T04:28:42.649 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 18 (ObjNum 60 snap 4 seq_num 60) dirty exists 2021-02-10T04:28:42.649 INFO:tasks.rados.rados.0.smithi085.stdout:91: left oid 22 (ObjNum 60 snap 4 seq_num 60) 2021-02-10T04:28:42.649 INFO:tasks.rados.rados.0.smithi085.stdout:88: finishing copy_from to smithi08575397-50 2021-02-10T04:28:42.649 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2021-02-10T04:28:42.649 INFO:tasks.rados.rados.0.smithi085.stdout:88: finishing copy_from racing read to smithi08575397-50 2021-02-10T04:28:42.649 INFO:tasks.rados.rados.0.smithi085.stdout:93: finishing write tid 1 to smithi08575397-44 2021-02-10T04:28:42.649 INFO:tasks.rados.rados.0.smithi085.stdout:82: finishing copy_from to smithi08575397-4 2021-02-10T04:28:42.649 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 4 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2021-02-10T04:28:42.649 INFO:tasks.rados.rados.0.smithi085.stdout:82: finishing copy_from racing read to smithi08575397-4 2021-02-10T04:28:42.649 INFO:tasks.rados.rados.0.smithi085.stdout:93: finishing write tid 2 to smithi08575397-44 2021-02-10T04:28:42.650 INFO:tasks.rados.rados.0.smithi085.stdout:93: finishing write tid 3 to smithi08575397-44 2021-02-10T04:28:42.650 INFO:tasks.rados.rados.0.smithi085.stdout:93: finishing write tid 4 to smithi08575397-44 2021-02-10T04:28:42.650 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 10 (ObjNum 61 snap 4 seq_num 61) dirty exists 2021-02-10T04:28:42.650 INFO:tasks.rados.rados.0.smithi085.stdout:93: left oid 44 (ObjNum 61 snap 4 seq_num 61) 2021-02-10T04:28:42.650 INFO:tasks.rados.rados.0.smithi085.stdout:94: finishing write tid 1 to smithi08575397-13 2021-02-10T04:28:42.650 INFO:tasks.rados.rados.0.smithi085.stdout:94: finishing write tid 2 to smithi08575397-13 2021-02-10T04:28:42.650 INFO:tasks.rados.rados.0.smithi085.stdout:94: finishing write tid 3 to smithi08575397-13 2021-02-10T04:28:42.650 INFO:tasks.rados.rados.0.smithi085.stdout:94: finishing write tid 4 to smithi08575397-13 2021-02-10T04:28:42.650 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 17 (ObjNum 62 snap 4 seq_num 62) dirty exists 2021-02-10T04:28:42.650 INFO:tasks.rados.rados.0.smithi085.stdout:94: left oid 13 (ObjNum 62 snap 4 seq_num 62) 2021-02-10T04:28:42.650 INFO:tasks.rados.rados.0.smithi085.stdout:95: expect (ObjNum 26 snap 0 seq_num 26) 2021-02-10T04:28:42.687 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:28:42.743 INFO:tasks.rados.rados.0.smithi085.stdout:96: finishing copy_from to smithi08575397-41 2021-02-10T04:28:42.743 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 41 v 18 (ObjNum 59 snap 4 seq_num 59) dirty exists 2021-02-10T04:28:42.743 INFO:tasks.rados.rados.0.smithi085.stdout:96: finishing copy_from racing read to smithi08575397-41 2021-02-10T04:28:42.744 INFO:tasks.rados.rados.0.smithi085.stdout:101: expect (ObjNum 16 snap 0 seq_num 16) 2021-02-10T04:28:42.827 INFO:tasks.rados.rados.0.smithi085.stdout:99: expect (ObjNum 41 snap 0 seq_num 41) 2021-02-10T04:28:42.927 INFO:tasks.rados.rados.0.smithi085.stdout:98: finishing write tid 1 to smithi08575397-16 2021-02-10T04:28:42.927 INFO:tasks.rados.rados.0.smithi085.stdout:98: finishing write tid 2 to smithi08575397-16 2021-02-10T04:28:42.927 INFO:tasks.rados.rados.0.smithi085.stdout:98: finishing write tid 3 to smithi08575397-16 2021-02-10T04:28:42.927 INFO:tasks.rados.rados.0.smithi085.stdout:98: finishing write tid 4 to smithi08575397-16 2021-02-10T04:28:42.927 INFO:tasks.rados.rados.0.smithi085.stdout:98: finishing write tid 5 to smithi08575397-16 2021-02-10T04:28:42.927 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 35 (ObjNum 63 snap 4 seq_num 63) dirty exists 2021-02-10T04:28:42.927 INFO:tasks.rados.rados.0.smithi085.stdout:98: left oid 16 (ObjNum 63 snap 4 seq_num 63) 2021-02-10T04:28:43.026 INFO:tasks.rados.rados.0.smithi085.stdout:82: done (14 left) 2021-02-10T04:28:43.026 INFO:tasks.rados.rados.0.smithi085.stdout:84: done (13 left) 2021-02-10T04:28:43.026 INFO:tasks.rados.rados.0.smithi085.stdout:87: done (12 left) 2021-02-10T04:28:43.026 INFO:tasks.rados.rados.0.smithi085.stdout:88: done (11 left) 2021-02-10T04:28:43.026 INFO:tasks.rados.rados.0.smithi085.stdout:89: done (10 left) 2021-02-10T04:28:43.026 INFO:tasks.rados.rados.0.smithi085.stdout:91: done (9 left) 2021-02-10T04:28:43.026 INFO:tasks.rados.rados.0.smithi085.stdout:92: done (8 left) 2021-02-10T04:28:43.026 INFO:tasks.rados.rados.0.smithi085.stdout:93: done (7 left) 2021-02-10T04:28:43.026 INFO:tasks.rados.rados.0.smithi085.stdout:94: done (6 left) 2021-02-10T04:28:43.027 INFO:tasks.rados.rados.0.smithi085.stdout:95: done (5 left) 2021-02-10T04:28:43.027 INFO:tasks.rados.rados.0.smithi085.stdout:96: done (4 left) 2021-02-10T04:28:43.027 INFO:tasks.rados.rados.0.smithi085.stdout:98: done (3 left) 2021-02-10T04:28:43.027 INFO:tasks.rados.rados.0.smithi085.stdout:99: done (2 left) 2021-02-10T04:28:43.027 INFO:tasks.rados.rados.0.smithi085.stdout:101: done (1 left) 2021-02-10T04:28:43.027 INFO:tasks.rados.rados.0.smithi085.stdout:102: done (0 left) 2021-02-10T04:28:43.027 INFO:tasks.rados.rados.0.smithi085.stdout:103: copy_from oid 31 from oid 29 current snap is 5 2021-02-10T04:28:43.027 INFO:tasks.rados.rados.0.smithi085.stdout:104: read oid 50 snap -1 2021-02-10T04:28:43.027 INFO:tasks.rados.rados.0.smithi085.stdout:104: expect (ObjNum 10 snap 0 seq_num 10) 2021-02-10T04:28:43.027 INFO:tasks.rados.rados.0.smithi085.stdout:105: read oid 35 snap -1 2021-02-10T04:28:43.027 INFO:tasks.rados.rados.0.smithi085.stdout:105: expect (ObjNum 34 snap 0 seq_num 34) 2021-02-10T04:28:43.027 INFO:tasks.rados.rados.0.smithi085.stdout:106: snap_remove snap 4 2021-02-10T04:28:43.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:28:43 smithi085 conmon[52836]: debug 2021-02-10T04:28:43.051+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:43.409 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:28:43.513 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:28:43.535 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:43 smithi164 conmon[29288]: cluster 2021-02-10T04:28:42.228966+0000 mon.a (mon.0) 2021-02-10T04:28:43.535 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:43 smithi164 conmon[29288]: 761 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2021-02-10T04:28:43.535 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:43 smithi164 conmon[29288]: cluster 2021-02-10T04:28:42.382459+0000 osd.6 (osd.6) 28 : cluster [DBG] 2.22 scrub ok 2021-02-10T04:28:43.535 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:43 smithi164 conmon[29288]: cluster 2021-02-10T04:28: 2021-02-10T04:28:43.535 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:43 smithi164 conmon[29288]: 42.701920+0000 osd.0 (osd.0) 36 : cluster 2021-02-10T04:28:43.536 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:43 smithi164 conmon[29288]: [DBG] 6.0 scrub ok 2021-02-10T04:28:43.536 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:43 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:28:43.536 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:43 smithi164 conmon[29288]: -10T04:28:43.025782+0000 mon.a 2021-02-10T04:28:43.536 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:43 smithi164 conmon[29288]: (mon.0) 762 : cluster [DBG] 2021-02-10T04:28:43.536 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:43 smithi164 conmon[29288]: osdmap e107: 8 total, 8 up, 8 in 2021-02-10T04:28:43.537 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:43.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:43 smithi085 conmon[29319]: cluster 2021-02-10T04:28:42.228966+0000 mon.a (mon 2021-02-10T04:28:43.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:43 smithi085 conmon[29319]: .0) 761 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2021-02-10T04:28:43.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:43 smithi085 conmon[29319]: cluster 2021-02-10T04:28:42.382459+0000 osd.6 (osd.6) 28 : cluster [DBG] 2.22 scrub ok 2021-02-10T04:28:43.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:43 smithi085 conmon[29319]: cluster 2021-02-10T04:28:42.701920+0000 osd.0 (osd.0) 36 : cluster 2021-02-10T04:28:43.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:43 smithi085 conmon[29319]: [DBG] 6.0 scrub ok 2021-02-10T04:28:43.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:43 smithi085 conmon[29319]: cluster 2021-02-10T04:28:43.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:43 smithi085 conmon[29319]: 2021-02-10T04:28:43.025782+0000 mon.a (mon.0) 762 : cluster 2021-02-10T04:28:43.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:43 smithi085 conmon[29319]: [DBG] osdmap e107: 8 total, 8 up, 8 in 2021-02-10T04:28:43.594 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:43 smithi085 conmon[35961]: cluster 2021-02-10T04:28:42.228966+0000 2021-02-10T04:28:43.594 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:43 smithi085 conmon[35961]: mon.a (mon.0) 761 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2021-02-10T04:28:43.594 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:43 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:28:43.594 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:43 smithi085 conmon[35961]: 04:28:42.382459+0000 osd.6 (osd.6) 28 : cluster [DBG] 2.22 scrub ok 2021-02-10T04:28:43.594 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:43 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:28:43.594 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:43 smithi085 conmon[35961]: -02-10T04:28:42.701920+0000 osd.0 2021-02-10T04:28:43.594 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:43 smithi085 conmon[35961]: (osd.0) 36 : cluster [DBG] 6.0 scrub ok 2021-02-10T04:28:43.595 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:43 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:28:43.595 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:43 smithi085 conmon[35961]: 02-10T04:28:43.025782+0000 mon.a ( 2021-02-10T04:28:43.595 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:43 smithi085 conmon[35961]: mon.0) 762 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2021-02-10T04:28:43.617 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:28:43.722 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:28:43.750 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:43.893 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:28:43 smithi164 conmon[37263]: debug 2021-02-10T04:28:43.595+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:44.034 INFO:tasks.rados.rados.0.smithi085.stdout:106: done (3 left) 2021-02-10T04:28:44.034 INFO:tasks.rados.rados.0.smithi085.stdout:107: read oid 34 snap -1 2021-02-10T04:28:44.034 INFO:tasks.rados.rados.0.smithi085.stdout:107: expect deleted 2021-02-10T04:28:44.035 INFO:tasks.rados.rados.0.smithi085.stdout:108: copy_from oid 12 from oid 26 current snap is 5 2021-02-10T04:28:44.035 INFO:tasks.rados.rados.0.smithi085.stdout:109: read oid 7 snap -1 2021-02-10T04:28:44.035 INFO:tasks.rados.rados.0.smithi085.stdout:109: expect (ObjNum 6 snap 0 seq_num 6) 2021-02-10T04:28:44.035 INFO:tasks.rados.rados.0.smithi085.stdout:110: copy_from oid 41 from oid 23 current snap is 5 2021-02-10T04:28:44.035 INFO:tasks.rados.rados.0.smithi085.stdout:111: write oid 21 current snap is 5 2021-02-10T04:28:44.035 INFO:tasks.rados.rados.0.smithi085.stdout:111: seq_num 64 ranges {675648=546760,2012024=568349} 2021-02-10T04:28:44.041 INFO:tasks.rados.rados.0.smithi085.stdout:111: writing smithi08575397-21 from 675648 to 1222408 tid 1 2021-02-10T04:28:44.050 INFO:tasks.rados.rados.0.smithi085.stdout:111: writing smithi08575397-21 from 2012024 to 2580373 tid 2 2021-02-10T04:28:44.051 INFO:tasks.rados.rados.0.smithi085.stdout:112: copy_from oid 43 from oid 16 current snap is 5 2021-02-10T04:28:44.051 INFO:tasks.rados.rados.0.smithi085.stdout:113: write (excl) oid 37 current snap is 5 2021-02-10T04:28:44.051 INFO:tasks.rados.rados.0.smithi085.stdout:113: seq_num 65 ranges {692455=622009,2070506=712007} 2021-02-10T04:28:44.059 INFO:tasks.rados.rados.0.smithi085.stdout:113: writing smithi08575397-37 from 692455 to 1314464 tid 1 2021-02-10T04:28:44.068 INFO:tasks.rados.rados.0.smithi085.stdout:113: writing smithi08575397-37 from 2070506 to 2782513 tid 2 2021-02-10T04:28:44.068 INFO:tasks.rados.rados.0.smithi085.stdout:114: snap_create 2021-02-10T04:28:44.068 INFO:tasks.rados.rados.0.smithi085.stdout:104: expect (ObjNum 10 snap 0 seq_num 10) 2021-02-10T04:28:44.164 INFO:tasks.rados.rados.0.smithi085.stdout:105: expect (ObjNum 34 snap 0 seq_num 34) 2021-02-10T04:28:44.276 INFO:tasks.rados.rados.0.smithi085.stdout:103: finishing copy_from to smithi08575397-31 2021-02-10T04:28:44.276 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 18 (ObjNum 59 snap 4 seq_num 59) dirty exists 2021-02-10T04:28:44.277 INFO:tasks.rados.rados.0.smithi085.stdout:103: finishing copy_from racing read to smithi08575397-31 2021-02-10T04:28:44.277 INFO:tasks.rados.rados.0.smithi085.stdout:109: expect (ObjNum 6 snap 0 seq_num 6) 2021-02-10T04:28:44.278 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:28:44.313 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:28:44.347 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:28:44.375 INFO:tasks.rados.rados.0.smithi085.stdout:111: finishing write tid 1 to smithi08575397-21 2021-02-10T04:28:44.375 INFO:tasks.rados.rados.0.smithi085.stdout:108: finishing copy_from to smithi08575397-12 2021-02-10T04:28:44.375 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2021-02-10T04:28:44.375 INFO:tasks.rados.rados.0.smithi085.stdout:108: finishing copy_from racing read to smithi08575397-12 2021-02-10T04:28:44.375 INFO:tasks.rados.rados.0.smithi085.stdout:110: finishing copy_from to smithi08575397-41 2021-02-10T04:28:44.375 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 41 v 10 (ObjNum 57 snap 3 seq_num 57) dirty exists 2021-02-10T04:28:44.376 INFO:tasks.rados.rados.0.smithi085.stdout:110: finishing copy_from racing read to smithi08575397-41 2021-02-10T04:28:44.376 INFO:tasks.rados.rados.0.smithi085.stdout:111: finishing write tid 2 to smithi08575397-21 2021-02-10T04:28:44.376 INFO:tasks.rados.rados.0.smithi085.stdout:111: finishing write tid 3 to smithi08575397-21 2021-02-10T04:28:44.376 INFO:tasks.rados.rados.0.smithi085.stdout:111: finishing write tid 4 to smithi08575397-21 2021-02-10T04:28:44.376 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 39 (ObjNum 64 snap 5 seq_num 64) dirty exists 2021-02-10T04:28:44.376 INFO:tasks.rados.rados.0.smithi085.stdout:111: left oid 21 (ObjNum 64 snap 5 seq_num 64) 2021-02-10T04:28:44.376 INFO:tasks.rados.rados.0.smithi085.stdout:113: finishing write tid 1 to smithi08575397-37 2021-02-10T04:28:44.376 INFO:tasks.rados.rados.0.smithi085.stdout:113: finishing write tid 2 to smithi08575397-37 2021-02-10T04:28:44.376 INFO:tasks.rados.rados.0.smithi085.stdout:113: finishing write tid 3 to smithi08575397-37 2021-02-10T04:28:44.376 INFO:tasks.rados.rados.0.smithi085.stdout:113: finishing write tid 4 to smithi08575397-37 2021-02-10T04:28:44.376 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 42 (ObjNum 65 snap 5 seq_num 65) dirty exists 2021-02-10T04:28:44.376 INFO:tasks.rados.rados.0.smithi085.stdout:113: left oid 37 (ObjNum 65 snap 5 seq_num 65) 2021-02-10T04:28:44.376 INFO:tasks.rados.rados.0.smithi085.stdout:112: finishing copy_from to smithi08575397-43 2021-02-10T04:28:44.376 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 35 (ObjNum 63 snap 4 seq_num 63) dirty exists 2021-02-10T04:28:44.377 INFO:tasks.rados.rados.0.smithi085.stdout:112: finishing copy_from racing read to smithi08575397-43 2021-02-10T04:28:44.383 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:28:44.419 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:28:44.446 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:28:44.472 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:28:44.498 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:28:44.500 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:28:44.500 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:28:44.500 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:28:44.500 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:28:44.500 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:28:44.500 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:28:44.500 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:28:44.500 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:28:44.527 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[29319]: cluster 2021-02-10T04:28:44.527 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[29319]: 2021-02-10T04:28:44.527 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:28:44.527 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[29319]: 28:43.509341+0000 osd.4 (osd.4) 37 : cluster [DBG] 2021-02-10T04:28:44.527 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[29319]: 5.e scrub ok 2021-02-10T04:28:44.528 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:28:44.528 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[29319]: 02-10T04:28:43.740848+0000 2021-02-10T04:28:44.528 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[29319]: osd.0 (osd.0) 37 : cluster [DBG] 5.5 scrub ok 2021-02-10T04:28:44.528 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[29319]: 2021-02-10T04:28:44.528 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:28:44.528 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[29319]: :28:43.994221+0000 mgr.y (mgr. 2021-02-10T04:28:44.528 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[29319]: 14140) 337 : cluster [DBG] pgmap v310: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 114 MiB data, 266 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2021-02-10T04:28:44.528 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:28:44.528 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[29319]: 02-10T04:28:44.031278+0000 2021-02-10T04:28:44.528 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[29319]: mon.a (mon.0) 763 : cluster [DBG] 2021-02-10T04:28:44.528 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[29319]: osdmap e108: 8 total, 8 up, 8 in 2021-02-10T04:28:44.528 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[45443]: debug 2021-02-10T04:28:44.529 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[45443]: 2021-02-10T04:28:44.378+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:44.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[35961]: cluster 2021-02-10T04:28:44.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:28:44.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[35961]: 04:28:43.509341+0000 osd.4 (osd.4) 37 : cluster [DBG] 5.e scrub ok 2021-02-10T04:28:44.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:28:44.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[35961]: 10T04:28:43.740848+0000 osd.0 (osd. 2021-02-10T04:28:44.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[35961]: 0) 37 : cluster [DBG] 5.5 scrub ok 2021-02-10T04:28:44.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[35961]: cluster 2021-02-10T04:28:44.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[35961]: 2021 2021-02-10T04:28:44.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[35961]: -02-10T04:28:43. 2021-02-10T04:28:44.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[35961]: 994221+0000 mgr.y (mgr.14140) 337 : cluster [DBG] pgmap v310: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 114 MiB data, 266 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2021-02-10T04:28:44.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:28:44.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[35961]: 10T04:28:44.031278+0000 mon.a (mon. 2021-02-10T04:28:44.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:44 smithi085 conmon[35961]: 0) 763 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2021-02-10T04:28:44.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:44 smithi164 conmon[29288]: cluster 2021-02-10T04:28:43.509341+0000 osd.4 (osd.4) 2021-02-10T04:28:44.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:44 smithi164 conmon[29288]: 37 : cluster [DBG] 5.e scrub ok 2021-02-10T04:28:44.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:44 smithi164 conmon[29288]: cluster 2021-02-10T04:28:43.740848+0000 osd.0 (osd.0) 37 : cluster [DBG] 5.5 scrub ok 2021-02-10T04:28:44.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:44 smithi164 conmon[29288]: cluster 2021-02-10T04:28:43.994221+0000 2021-02-10T04:28:44.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:44 smithi164 conmon[29288]: mgr.y (mgr.14140) 337 : cluster [DBG] pgmap v310: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 114 MiB data, 266 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2021-02-10T04:28:44.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:44 smithi164 conmon[29288]: cluster 2021-02-10T04:28:44.031278+0000 mon.a (mon. 2021-02-10T04:28:44.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:44 smithi164 conmon[29288]: 0) 763 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2021-02-10T04:28:44.715 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:28:45.036 INFO:tasks.rados.rados.0.smithi085.stdout:103: done (10 left) 2021-02-10T04:28:45.036 INFO:tasks.rados.rados.0.smithi085.stdout:104: done (9 left) 2021-02-10T04:28:45.037 INFO:tasks.rados.rados.0.smithi085.stdout:105: done (8 left) 2021-02-10T04:28:45.037 INFO:tasks.rados.rados.0.smithi085.stdout:107: done (7 left) 2021-02-10T04:28:45.037 INFO:tasks.rados.rados.0.smithi085.stdout:108: done (6 left) 2021-02-10T04:28:45.037 INFO:tasks.rados.rados.0.smithi085.stdout:109: done (5 left) 2021-02-10T04:28:45.037 INFO:tasks.rados.rados.0.smithi085.stdout:110: done (4 left) 2021-02-10T04:28:45.037 INFO:tasks.rados.rados.0.smithi085.stdout:111: done (3 left) 2021-02-10T04:28:45.037 INFO:tasks.rados.rados.0.smithi085.stdout:112: done (2 left) 2021-02-10T04:28:45.037 INFO:tasks.rados.rados.0.smithi085.stdout:113: done (1 left) 2021-02-10T04:28:45.037 INFO:tasks.rados.rados.0.smithi085.stdout:114: done (0 left) 2021-02-10T04:28:45.037 INFO:tasks.rados.rados.0.smithi085.stdout:115: snap_create 2021-02-10T04:28:45.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[29319]: audit 2021-02-10T04:28:44.238729+0000 2021-02-10T04:28:45.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[29319]: mon.c (mon.1) 38 : audit [DBG] from='client.? 172.21.15.85:0/2187705746' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:28:45.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[29319]: cluster 2021-02-10T04:28:44.410811+0000 osd.7 2021-02-10T04:28:45.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[29319]: (osd.7) 36 : cluster [DBG] 2.b scrub ok 2021-02-10T04:28:45.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[29319]: audit 2021-02-10T04:28:45.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[29319]: 2021-02-10T04:28:44.640711+0000 mon.a (mon.0) 764 : audit 2021-02-10T04:28:45.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[29319]: [INF] from='client.? 172.21.15.85:0/2214915376' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num", "val": "11"}]: dispatch 2021-02-10T04:28:45.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:28:45.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[29319]: 10T04:28:44.719357+0000 osd.0 (osd.0) 38 : cluster [DBG] 2021-02-10T04:28:45.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[29319]: 6.18 scrub ok 2021-02-10T04:28:45.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[29319]: audit 2021-02-10T04:28:45.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[29319]: 2021-02-10T04:28:45.036358+0000 2021-02-10T04:28:45.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[29319]: mon.a (mon.0) 765 : audit [INF] from='client.? 172.21.15.85:0/2214915376' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num", "val": "11"}]': finished 2021-02-10T04:28:45.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[29319]: 2021-02-10T04:28:45.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[29319]: cluster 2021-02-10T04:28:45.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[29319]: 2021-02-10T04:28:45.036427+0000 2021-02-10T04:28:45.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[29319]: mon.a (mon.0) 766 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2021-02-10T04:28:45.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[29319]: 2021-02-10T04:28:45.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[35961]: audit 2021-02-10T04:28:44. 2021-02-10T04:28:45.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[35961]: 238729+0000 mon.c (mon.1) 38 : audit [DBG] from='client.? 172.21.15.85:0/2187705746' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:28:45.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:28:45.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[35961]: 10T04:28:44.410811+0000 osd.7 2021-02-10T04:28:45.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[35961]: (osd.7) 36 : cluster 2021-02-10T04:28:45.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[35961]: [DBG] 2.b scrub ok 2021-02-10T04:28:45.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:28:45.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[35961]: 02-10T04:28:44.640711+0000 mon.a (mon.0) 2021-02-10T04:28:45.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[35961]: 764 : audit [INF] from='client.? 172.21.15.85:0/2214915376' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num", "val": "11"}]: dispatch 2021-02-10T04:28:45.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[35961]: cluster 2021-02-10T04:28:44 2021-02-10T04:28:45.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[35961]: .719357+0000 osd.0 (osd.0) 38 : cluster [DBG] 6.18 scrub ok 2021-02-10T04:28:45.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[35961]: 2021-02-10T04:28:45.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[35961]: audit 2021-02-10T04:28:45.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[35961]: 2021-02-10T04:28:45.036358+0000 mon.a (mon.0 2021-02-10T04:28:45.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[35961]: ) 765 : audit [INF] from='client.? 172.21.15.85:0/2214915376' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num", "val": "11"}]': finished 2021-02-10T04:28:45.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[35961]: cluster 2021-02-10T04:28:45.036427 2021-02-10T04:28:45.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[35961]: +0000 mon.a (mon.0) 766 : cluster [DBG] 2021-02-10T04:28:45.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:45 smithi085 conmon[35961]: osdmap e109: 8 total, 8 up, 8 in 2021-02-10T04:28:45.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:45 smithi164 conmon[29288]: audit 2021-02-10T04:28:44.238729+0000 mon.c (mon.1) 2021-02-10T04:28:45.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:45 smithi164 conmon[29288]: 38 : audit [DBG] from='client.? 172.21.15.85:0/2187705746' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:28:45.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:45 smithi164 conmon[29288]: cluster 2021-02-10T04:28:44.410811+0000 osd.7 (osd.7) 36 : cluster [DBG] 2021-02-10T04:28:45.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:45 smithi164 conmon[29288]: 2.b scrub ok 2021-02-10T04:28:45.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:45 smithi164 conmon[29288]: audit 2021-02-10T04:28:45.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:45 smithi164 conmon[29288]: 2021-02-10T04:28: 2021-02-10T04:28:45.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:45 smithi164 conmon[29288]: 44.640711+0000 mon.a (mon.0) 2021-02-10T04:28:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:45 smithi164 conmon[29288]: 764 : audit [INF] from='client.? 172.21.15.85:0/2214915376' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num", "val": "11"}]: dispatch 2021-02-10T04:28:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:45 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:28:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:45 smithi164 conmon[29288]: 02-10T04:28:44.719357 2021-02-10T04:28:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:45 smithi164 conmon[29288]: +0000 osd.0 (osd.0) 38 : 2021-02-10T04:28:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:45 smithi164 conmon[29288]: cluster [DBG] 6.18 scrub ok 2021-02-10T04:28:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:45 smithi164 conmon[29288]: audit 2021 2021-02-10T04:28:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:45 smithi164 conmon[29288]: -02-10T04:28:45. 2021-02-10T04:28:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:45 smithi164 conmon[29288]: 036358+0000 mon.a (mon.0) 2021-02-10T04:28:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:45 smithi164 conmon[29288]: 765 : audit [INF] from='client.? 172.21.15.85:0/2214915376' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num", "val": "11"}]': finished 2021-02-10T04:28:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:45 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:28:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:45 smithi164 conmon[29288]: -02-10T04:28:45. 2021-02-10T04:28:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:45 smithi164 conmon[29288]: 036427+0000 mon.a (mon.0) 2021-02-10T04:28:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:45 smithi164 conmon[29288]: 766 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2021-02-10T04:28:45.873 INFO:teuthology.orchestra.run.smithi085.stderr:set pool 1 pg_num to 11 2021-02-10T04:28:46.034 INFO:tasks.rados.rados.0.smithi085.stdout:115: done (0 left) 2021-02-10T04:28:46.034 INFO:tasks.rados.rados.0.smithi085.stdout:116: delete oid 39 current snap is 7 2021-02-10T04:28:46.036 INFO:tasks.rados.rados.0.smithi085.stdout:116: done (0 left) 2021-02-10T04:28:46.036 INFO:tasks.rados.rados.0.smithi085.stdout:117: snap_remove snap 6 2021-02-10T04:28:46.077 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:28:46.112 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:46.246 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[29319]: cluster 2021-02-10T04:28:45.220250+0000 osd.6 2021-02-10T04:28:46.246 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[29319]: (osd.6) 29 : cluster [DBG] 2.29 deep-scrub ok 2021-02-10T04:28:46.246 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[29319]: cluster 2021-02-10T04:28:45.481467+0000 osd.5 (osd.5) 33 : cluster [DBG] 6.d scrub ok 2021-02-10T04:28:46.246 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[29319]: cluster 2021-02-10T04:28:45.647910+0000 osd.2 (osd.2) 31 : cluster [DBG] 2.10 scrub ok 2021-02-10T04:28:46.246 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[29319]: cluster 2021-02-10T04:28:45.697674+0000 osd.0 (osd.0) 39 : cluster [DBG] 2021-02-10T04:28:46.246 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[29319]: 6.16 deep-scrub ok 2021-02-10T04:28:46.247 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[29319]: cluster 2021-02-10T04:28:45.994715+0000 2021-02-10T04:28:46.247 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[29319]: mgr.y (mgr.14140) 338 : cluster [DBG] pgmap v313: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 123 MiB data, 307 MiB used, 715 GiB / 715 GiB avail; 28 MiB/s rd, 7.4 MiB/s wr, 41 op/s 2021-02-10T04:28:46.247 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:28:46.247 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[29319]: 02-10T04:28:45.995726+0000 mon.a 2021-02-10T04:28:46.247 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[29319]: (mon.0) 767 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num_actual", "val": "11"}]: dispatch 2021-02-10T04:28:46.247 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[29319]: 2021-02-10T04:28:46.247 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:28:46.247 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[29319]: :28:46.033340+0000 mon.a (mon.0) 768 : audit [INF] 2021-02-10T04:28:46.247 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num_actual", "val": "11"}]': finished 2021-02-10T04:28:46.247 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[29319]: cluster 2021-02-10T04:28: 2021-02-10T04:28:46.247 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[29319]: 46.033413+0000 mon.a (mon.0) 769 : cluster 2021-02-10T04:28:46.247 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[29319]: [DBG] osdmap e110: 8 total, 8 up, 8 in 2021-02-10T04:28:46.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:46 smithi164 conmon[29288]: cluster 2021-02-10T04:28:45.220250+0000 osd.6 (osd.6) 29 2021-02-10T04:28:46.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:46 smithi164 conmon[29288]: : cluster [DBG] 2.29 deep-scrub ok 2021-02-10T04:28:46.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:46 smithi164 conmon[29288]: cluster 2021-02-10T04:28:45.481467+0000 osd.5 (osd.5) 33 : cluster [DBG] 6.d scrub ok 2021-02-10T04:28:46.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:46 smithi164 conmon[29288]: cluster 2021-02-10T04:28:46.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:46 smithi164 conmon[29288]: 2021-02-10T04:28:45.647910+0000 osd.2 (osd.2 2021-02-10T04:28:46.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:46 smithi164 conmon[29288]: ) 31 : cluster [DBG] 2.10 scrub ok 2021-02-10T04:28:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:46 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:28:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:46 smithi164 conmon[29288]: 02-10T04:28:45.697674 2021-02-10T04:28:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:46 smithi164 conmon[29288]: +0000 osd.0 (osd.0) 39 2021-02-10T04:28:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:46 smithi164 conmon[29288]: : cluster [DBG] 6.16 deep-scrub ok 2021-02-10T04:28:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:46 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:28:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:46 smithi164 conmon[29288]: -02-10T04:28:45. 2021-02-10T04:28:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:46 smithi164 conmon[29288]: 994715+0000 mgr.y (mgr.14140) 2021-02-10T04:28:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:46 smithi164 conmon[29288]: 338 : cluster [DBG] pgmap v313: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 123 MiB data, 307 MiB used, 715 GiB / 715 GiB avail; 28 MiB/s rd, 7.4 MiB/s wr, 41 op/s 2021-02-10T04:28:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:46 smithi164 conmon[29288]: audit 2021 2021-02-10T04:28:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:46 smithi164 conmon[29288]: -02-10T04:28:45 2021-02-10T04:28:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:46 smithi164 conmon[29288]: .995726+0000 mon.a (mon.0 2021-02-10T04:28:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:46 smithi164 conmon[29288]: ) 767 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num_actual", "val": "11"}]: dispatch 2021-02-10T04:28:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:46 smithi164 conmon[29288]: audit 2021-02-10T04:28:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:46 smithi164 conmon[29288]: 2021-02- 2021-02-10T04:28:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:46 smithi164 conmon[29288]: 10T04:28:46.033340 2021-02-10T04:28:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:46 smithi164 conmon[29288]: +0000 mon.a (mon 2021-02-10T04:28:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:46 smithi164 conmon[29288]: .0) 768 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num_actual", "val": "11"}]': finished 2021-02-10T04:28:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:46 smithi164 conmon[29288]: cluster 2021-02-10T04:28:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:46 smithi164 conmon[29288]: 2021-02-10T04:28 2021-02-10T04:28:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:46 smithi164 conmon[29288]: :46.033413+0000 mon.a (mon 2021-02-10T04:28:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:46 smithi164 conmon[29288]: .0) 769 : cluster [DBG] 2021-02-10T04:28:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:46 smithi164 conmon[29288]: osdmap e110: 8 total, 8 up, 8 in 2021-02-10T04:28:46.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[35961]: cluster 2021-02-10T04:28:45.220250 2021-02-10T04:28:46.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[35961]: +0000 osd.6 (osd.6) 29 : cluster [DBG] 2.29 deep-scrub ok 2021-02-10T04:28:46.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[35961]: cluster 2021-02-10T04:28:45 2021-02-10T04:28:46.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[35961]: .481467+0000 osd.5 (osd.5) 33 : cluster [DBG] 6.d scrub ok 2021-02-10T04:28:46.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[35961]: cluster 2021-02-10T04:28:45.647910+0000 osd.2 (osd.2) 31 : cluster [DBG] 2.10 scrub ok 2021-02-10T04:28:46.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:28:46.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[35961]: :28:45.697674+0000 osd.0 (osd.0) 39 : 2021-02-10T04:28:46.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[35961]: cluster [DBG] 6.16 deep-scrub ok 2021-02-10T04:28:46.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[35961]: cluster 2021-02-10T04:28:45.994715+0000 mgr.y ( 2021-02-10T04:28:46.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[35961]: mgr.14140) 338 : cluster [DBG] pgmap v313: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 123 MiB data, 307 MiB used, 715 GiB / 715 GiB avail; 28 MiB/s rd, 7.4 MiB/s wr, 41 op/s 2021-02-10T04:28:46.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:28:46.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[35961]: 28:45.995726+0000 mon.a (mon.0) 767 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num_actual", "val": "11"}]: dispatch 2021-02-10T04:28:46.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[35961]: audit 2021-02-10T04:28:46.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[35961]: 2021-02-10T04:28:46.033340+0000 mon.a (mon.0) 768 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num_actual", "val": "11"}]': finished 2021-02-10T04:28:46.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:28:46.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[35961]: :28:46.033413+0000 mon.a (mon.0) 2021-02-10T04:28:46.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:46 smithi085 conmon[35961]: 769 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2021-02-10T04:28:47.042 INFO:tasks.rados.rados.0.smithi085.stdout:117: done (0 left) 2021-02-10T04:28:47.042 INFO:tasks.rados.rados.0.smithi085.stdout:118: read oid 14 snap 5 2021-02-10T04:28:47.042 INFO:tasks.rados.rados.0.smithi085.stdout:118: expect (ObjNum 13 snap 0 seq_num 13) 2021-02-10T04:28:47.042 INFO:tasks.rados.rados.0.smithi085.stdout:119: write oid 47 current snap is 7 2021-02-10T04:28:47.042 INFO:tasks.rados.rados.0.smithi085.stdout:119: seq_num 67 ranges {726069=772507,2071117=715676} 2021-02-10T04:28:47.045 INFO:tasks.rados.rados.0.smithi085.stdout:119: writing smithi08575397-47 from 726069 to 1498576 tid 1 2021-02-10T04:28:47.053 INFO:tasks.rados.rados.0.smithi085.stdout:119: writing smithi08575397-47 from 2071117 to 2786793 tid 2 2021-02-10T04:28:47.054 INFO:tasks.rados.rados.0.smithi085.stdout:120: read oid 13 snap -1 2021-02-10T04:28:47.054 INFO:tasks.rados.rados.0.smithi085.stdout:120: expect (ObjNum 62 snap 4 seq_num 62) 2021-02-10T04:28:47.054 INFO:tasks.rados.rados.0.smithi085.stdout:121: write (excl) oid 30 current snap is 7 2021-02-10T04:28:47.054 INFO:tasks.rados.rados.0.smithi085.stdout:118: expect (ObjNum 13 snap 0 seq_num 13) 2021-02-10T04:28:47.149 INFO:tasks.rados.rados.0.smithi085.stdout:119: finishing write tid 1 to smithi08575397-47 2021-02-10T04:28:47.149 INFO:tasks.rados.rados.0.smithi085.stdout:119: finishing write tid 2 to smithi08575397-47 2021-02-10T04:28:47.149 INFO:tasks.rados.rados.0.smithi085.stdout:119: finishing write tid 3 to smithi08575397-47 2021-02-10T04:28:47.149 INFO:tasks.rados.rados.0.smithi085.stdout:119: finishing write tid 4 to smithi08575397-47 2021-02-10T04:28:47.149 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 47 v 48 (ObjNum 67 snap 7 seq_num 67) dirty exists 2021-02-10T04:28:47.150 INFO:tasks.rados.rados.0.smithi085.stdout:119: left oid 47 (ObjNum 67 snap 7 seq_num 67) 2021-02-10T04:28:47.150 INFO:tasks.rados.rados.0.smithi085.stdout:121: seq_num 68 ranges {742876=447756,1729599=575687,3101234=41642} 2021-02-10T04:28:47.156 INFO:tasks.rados.rados.0.smithi085.stdout:121: writing smithi08575397-30 from 742876 to 1190632 tid 1 2021-02-10T04:28:47.163 INFO:tasks.rados.rados.0.smithi085.stdout:121: writing smithi08575397-30 from 1729599 to 2305286 tid 2 2021-02-10T04:28:47.168 INFO:tasks.rados.rados.0.smithi085.stdout:121: writing smithi08575397-30 from 3101234 to 3142876 tid 3 2021-02-10T04:28:47.168 INFO:tasks.rados.rados.0.smithi085.stdout:118: done (3 left) 2021-02-10T04:28:47.168 INFO:tasks.rados.rados.0.smithi085.stdout:119: done (2 left) 2021-02-10T04:28:47.168 INFO:tasks.rados.rados.0.smithi085.stdout:122: snap_remove snap 5 2021-02-10T04:28:47.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:47 smithi085 conmon[29319]: cluster 2021-02-10T04:28:46.202938+0000 2021-02-10T04:28:47.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:47 smithi085 conmon[29319]: osd.6 (osd.6) 30 : cluster [DBG] 2.24 scrub ok 2021-02-10T04:28:47.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:47 smithi085 conmon[29319]: cluster 2021-02-10T04:28:46.464103+0000 osd.5 (osd.5) 34 : cluster [DBG] 5.1b scrub ok 2021-02-10T04:28:47.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:47 smithi085 conmon[29319]: cluster 2021-02-10T04:28:46.493058+0000 osd.7 (osd.7) 37 : cluster [DBG] 5.1a scrub ok 2021-02-10T04:28:47.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:47 smithi085 conmon[29319]: cluster 2021-02-10T04:28:46.680172+0000 osd.2 (osd.2) 32 : cluster [DBG] 4.1c scrub ok 2021-02-10T04:28:47.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:47 smithi085 conmon[29319]: cluster 2021-02-10T04:28:47.035979+0000 mon.a (mon.0) 770 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2021-02-10T04:28:47.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:47 smithi085 conmon[35961]: cluster 2021-02-10T04:28:46.202938 2021-02-10T04:28:47.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:47 smithi085 conmon[35961]: +0000 osd.6 (osd.6) 30 : cluster [DBG] 2.24 scrub ok 2021-02-10T04:28:47.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:47 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:28:47.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:47 smithi085 conmon[35961]: T04:28:46.464103+0000 osd.5 (osd.5) 34 : cluster [DBG] 5.1b scrub ok 2021-02-10T04:28:47.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:47 smithi085 conmon[35961]: cluster 2021-02-10T04:28:47.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:47 smithi085 conmon[35961]: 2021-02-10T04:28:46.493058+0000 2021-02-10T04:28:47.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:47 smithi085 conmon[35961]: osd.7 (osd.7) 37 : cluster [DBG] 5.1a scrub ok 2021-02-10T04:28:47.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:47 smithi085 conmon[35961]: 2021-02-10T04:28:47.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:47 smithi085 conmon[35961]: cluster 2021-02-10T04:28:47.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:47 smithi085 conmon[35961]: 2021-02-10T04:28:46.680172+0000 osd.2 (osd 2021-02-10T04:28:47.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:47 smithi085 conmon[35961]: .2) 32 : cluster [DBG] 4.1c scrub ok 2021-02-10T04:28:47.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:47 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:28:47.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:47 smithi085 conmon[35961]: 28:47.035979+0000 mon.a (mon.0) 770 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2021-02-10T04:28:47.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:47 smithi164 conmon[29288]: cluster 2021-02-10T04:28:46.202938+0000 osd.6 (osd.6) 30 2021-02-10T04:28:47.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:47 smithi164 conmon[29288]: : cluster [DBG] 2.24 scrub ok 2021-02-10T04:28:47.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:47 smithi164 conmon[29288]: cluster 2021-02-10T04:28:46.464103+0000 osd.5 (osd.5) 34 : cluster [DBG] 5.1b scrub ok 2021-02-10T04:28:47.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:47 smithi164 conmon[29288]: cluster 2021-02-10T04:28:46.493058+0000 osd.7 (osd.7 2021-02-10T04:28:47.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:47 smithi164 conmon[29288]: ) 37 : cluster [DBG] 5.1a scrub ok 2021-02-10T04:28:47.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:47 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:28:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:47 smithi164 conmon[29288]: T04:28:46.680172+0000 osd.2 (osd. 2021-02-10T04:28:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:47 smithi164 conmon[29288]: 2) 32 : cluster [DBG] 4.1c scrub ok 2021-02-10T04:28:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:47 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:28:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:47 smithi164 conmon[29288]: -10T04:28:47.035979+0000 mon.a 2021-02-10T04:28:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:47 smithi164 conmon[29288]: (mon.0) 770 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2021-02-10T04:28:48.032 INFO:tasks.rados.rados.0.smithi085.stdout:122: done (2 left) 2021-02-10T04:28:48.032 INFO:tasks.rados.rados.0.smithi085.stdout:123: read oid 33 snap -1 2021-02-10T04:28:48.032 INFO:tasks.rados.rados.0.smithi085.stdout:123: expect (ObjNum 53 snap 2 seq_num 53) 2021-02-10T04:28:48.033 INFO:tasks.rados.rados.0.smithi085.stdout:124: delete oid 27 current snap is 7 2021-02-10T04:28:48.033 INFO:tasks.rados.rados.0.smithi085.stdout:120: expect (ObjNum 62 snap 4 seq_num 62) 2021-02-10T04:28:48.143 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:28:47 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:28:48.201 INFO:tasks.rados.rados.0.smithi085.stdout:120: done (3 left) 2021-02-10T04:28:48.201 INFO:tasks.rados.rados.0.smithi085.stdout:124: done (2 left) 2021-02-10T04:28:48.201 INFO:tasks.rados.rados.0.smithi085.stdout:125: write oid 35 current snap is 7 2021-02-10T04:28:48.201 INFO:tasks.rados.rados.0.smithi085.stdout:121: finishing write tid 1 to smithi08575397-30 2021-02-10T04:28:48.202 INFO:tasks.rados.rados.0.smithi085.stdout:121: finishing write tid 2 to smithi08575397-30 2021-02-10T04:28:48.202 INFO:tasks.rados.rados.0.smithi085.stdout:121: finishing write tid 3 to smithi08575397-30 2021-02-10T04:28:48.202 INFO:tasks.rados.rados.0.smithi085.stdout:121: finishing write tid 4 to smithi08575397-30 2021-02-10T04:28:48.202 INFO:tasks.rados.rados.0.smithi085.stdout:121: finishing write tid 5 to smithi08575397-30 2021-02-10T04:28:48.202 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 23 (ObjNum 68 snap 7 seq_num 68) dirty exists 2021-02-10T04:28:48.202 INFO:tasks.rados.rados.0.smithi085.stdout:121: left oid 30 (ObjNum 68 snap 7 seq_num 68) 2021-02-10T04:28:48.202 INFO:tasks.rados.rados.0.smithi085.stdout:125: seq_num 70 ranges {776490=714607,2246563=579356} 2021-02-10T04:28:48.209 INFO:tasks.rados.rados.0.smithi085.stdout:125: writing smithi08575397-35 from 776490 to 1491097 tid 1 2021-02-10T04:28:48.217 INFO:tasks.rados.rados.0.smithi085.stdout:125: writing smithi08575397-35 from 2246563 to 2825919 tid 2 2021-02-10T04:28:48.217 INFO:tasks.rados.rados.0.smithi085.stdout:121: done (2 left) 2021-02-10T04:28:48.217 INFO:tasks.rados.rados.0.smithi085.stdout:126: rollback oid 29 current snap is 7 2021-02-10T04:28:48.218 INFO:tasks.rados.rados.0.smithi085.stdout:123: expect (ObjNum 53 snap 2 seq_num 53) 2021-02-10T04:28:48.375 INFO:tasks.rados.rados.0.smithi085.stdout:123: done (2 left) 2021-02-10T04:28:48.376 INFO:tasks.rados.rados.0.smithi085.stdout:126: done (1 left) 2021-02-10T04:28:48.376 INFO:tasks.rados.rados.0.smithi085.stdout:127: rollback oid 44 current snap is 7 2021-02-10T04:28:48.376 INFO:tasks.rados.rados.0.smithi085.stdout:127: done (1 left) 2021-02-10T04:28:48.376 INFO:tasks.rados.rados.0.smithi085.stdout:128: write (excl) oid 10 current snap is 7 2021-02-10T04:28:48.376 INFO:tasks.rados.rados.0.smithi085.stdout:125: finishing write tid 1 to smithi08575397-35 2021-02-10T04:28:48.376 INFO:tasks.rados.rados.0.smithi085.stdout:125: finishing write tid 2 to smithi08575397-35 2021-02-10T04:28:48.376 INFO:tasks.rados.rados.0.smithi085.stdout:125: finishing write tid 3 to smithi08575397-35 2021-02-10T04:28:48.376 INFO:tasks.rados.rados.0.smithi085.stdout:125: finishing write tid 4 to smithi08575397-35 2021-02-10T04:28:48.376 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 35 v 20 (ObjNum 70 snap 7 seq_num 70) dirty exists 2021-02-10T04:28:48.376 INFO:tasks.rados.rados.0.smithi085.stdout:125: left oid 35 (ObjNum 70 snap 7 seq_num 70) 2021-02-10T04:28:48.376 INFO:tasks.rados.rados.0.smithi085.stdout:128: seq_num 71 ranges {793297=789856,2305045=723014} 2021-02-10T04:28:48.384 INFO:tasks.rados.rados.0.smithi085.stdout:128: writing smithi08575397-10 from 793297 to 1583153 tid 1 2021-02-10T04:28:48.393 INFO:tasks.rados.rados.0.smithi085.stdout:128: writing smithi08575397-10 from 2305045 to 3028059 tid 2 2021-02-10T04:28:48.393 INFO:tasks.rados.rados.0.smithi085.stdout:125: done (1 left) 2021-02-10T04:28:48.394 INFO:tasks.rados.rados.0.smithi085.stdout:129: copy_from oid 39 from oid 6 current snap is 7 2021-02-10T04:28:48.394 INFO:tasks.rados.rados.0.smithi085.stdout:128: finishing write tid 1 to smithi08575397-10 2021-02-10T04:28:48.394 INFO:tasks.rados.rados.0.smithi085.stdout:130: read oid 47 snap -1 2021-02-10T04:28:48.394 INFO:tasks.rados.rados.0.smithi085.stdout:130: expect (ObjNum 67 snap 7 seq_num 67) 2021-02-10T04:28:48.394 INFO:tasks.rados.rados.0.smithi085.stdout:131: write (excl) oid 20 current snap is 7 2021-02-10T04:28:48.394 INFO:tasks.rados.rados.0.smithi085.stdout:131: seq_num 72 ranges {410104=465105,1563527=583025,2810926=399178} 2021-02-10T04:28:48.398 INFO:tasks.rados.rados.0.smithi085.stdout:131: writing smithi08575397-20 from 410104 to 875209 tid 1 2021-02-10T04:28:48.406 INFO:tasks.rados.rados.0.smithi085.stdout:131: writing smithi08575397-20 from 1563527 to 2146552 tid 2 2021-02-10T04:28:48.412 INFO:tasks.rados.rados.0.smithi085.stdout:131: writing smithi08575397-20 from 2810926 to 3210104 tid 3 2021-02-10T04:28:48.412 INFO:tasks.rados.rados.0.smithi085.stdout:132: snap_create 2021-02-10T04:28:48.413 INFO:tasks.rados.rados.0.smithi085.stdout:128: finishing write tid 2 to smithi08575397-10 2021-02-10T04:28:48.413 INFO:tasks.rados.rados.0.smithi085.stdout:128: finishing write tid 3 to smithi08575397-10 2021-02-10T04:28:48.413 INFO:tasks.rados.rados.0.smithi085.stdout:128: finishing write tid 4 to smithi08575397-10 2021-02-10T04:28:48.413 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 10 v 23 (ObjNum 71 snap 7 seq_num 71) dirty exists 2021-02-10T04:28:48.413 INFO:tasks.rados.rados.0.smithi085.stdout:128: left oid 10 (ObjNum 71 snap 7 seq_num 71) 2021-02-10T04:28:48.413 INFO:tasks.rados.rados.0.smithi085.stdout:131: finishing write tid 1 to smithi08575397-20 2021-02-10T04:28:48.413 INFO:tasks.rados.rados.0.smithi085.stdout:130: expect (ObjNum 67 snap 7 seq_num 67) 2021-02-10T04:28:48.574 INFO:tasks.rados.rados.0.smithi085.stdout:129: finishing copy_from to smithi08575397-39 2021-02-10T04:28:48.574 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 39 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2021-02-10T04:28:48.574 INFO:tasks.rados.rados.0.smithi085.stdout:129: finishing copy_from racing read to smithi08575397-39 2021-02-10T04:28:48.574 INFO:tasks.rados.rados.0.smithi085.stdout:131: finishing write tid 2 to smithi08575397-20 2021-02-10T04:28:48.574 INFO:tasks.rados.rados.0.smithi085.stdout:131: finishing write tid 3 to smithi08575397-20 2021-02-10T04:28:48.574 INFO:tasks.rados.rados.0.smithi085.stdout:131: finishing write tid 4 to smithi08575397-20 2021-02-10T04:28:48.575 INFO:tasks.rados.rados.0.smithi085.stdout:131: finishing write tid 5 to smithi08575397-20 2021-02-10T04:28:48.575 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 20 v 16 (ObjNum 72 snap 7 seq_num 72) dirty exists 2021-02-10T04:28:48.575 INFO:tasks.rados.rados.0.smithi085.stdout:131: left oid 20 (ObjNum 72 snap 7 seq_num 72) 2021-02-10T04:28:48.610 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:28:48.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: cluster 2021-02-10T04:28:46.488678+0000 osd.1 (osd. 2021-02-10T04:28:48.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: 1) 38 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:28:48.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: cluster 2021-02-10T04:28:46.488725+0000 osd.1 (osd.1) 39 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:28:48.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: cluster 2021-02-10T04:28:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: 2021-02-10T04:28:47.473019+0000 osd.7 (osd.7) 38 : cluster [DBG] 5.10 scrub ok 2021-02-10T04:28:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:28:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: -02-10T04:28:47.476664+0000 2021-02-10T04:28:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: osd.4 (osd.4) 38 : cluster [DBG] 2021-02-10T04:28:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: 4.9 scrub ok 2021-02-10T04:28:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: cluster 2021-02-10T04:28:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: 2021-02-10T04:28:47.496463+0000 2021-02-10T04:28:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: osd.5 (osd.5) 35 : cluster [DBG] 5.15 scrub ok 2021-02-10T04:28:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: 2021-02-10T04:28:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: audit 2021-02-10T04:28:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: 2021-02-10T04:28:47.685026+0000 2021-02-10T04:28:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: mgr.y (mgr.14140) 339 : audit [DBG] 2021-02-10T04:28:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:28:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: cluster 2021-02-10T04:28 2021-02-10T04:28:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: :47.995140+0000 mgr.y (mgr.14140) 340 2021-02-10T04:28:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: : cluster [DBG] pgmap v316: 168 pgs: 10 unknown, 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 124 MiB data, 310 MiB used, 715 GiB / 715 GiB avail; 27 MiB/s rd, 7.4 MiB/s wr, 40 op/s 2021-02-10T04:28:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: audit 2021-02-10T04:28:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: 2021-02-10T04:28:47. 2021-02-10T04:28:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: 996028+0000 mon.a (mon.0) 771 : audit 2021-02-10T04:28:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num_actual", "val": "11"}]: dispatch 2021-02-10T04:28:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: audit 2021-02-10T04: 2021-02-10T04:28:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: 28:48.028084+0000 mon.a (mon.0 2021-02-10T04:28:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: ) 772 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num_actual", "val": "11"}]': finished 2021-02-10T04:28:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:28:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: -10T04:28:48.028240+0000 2021-02-10T04:28:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: mon.a (mon.0) 773 : cluster [DBG] 2021-02-10T04:28:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:48 smithi164 conmon[29288]: osdmap e112: 8 total, 8 up, 8 in 2021-02-10T04:28:48.690 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[52836]: debug 2021-02-10T04:28:48.289+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:48.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[29319]: cluster 2021-02-10T04:28:46.488678+0000 osd.1 2021-02-10T04:28:48.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[29319]: (osd.1) 38 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:28:48.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[29319]: cluster 2021-02-10T04:28:46.488725+0000 osd.1 (osd.1) 39 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:28:48.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[29319]: cluster 2021-02-10T04:28:47.473019+0000 osd.7 (osd 2021-02-10T04:28:48.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[29319]: .7) 38 : cluster [DBG] 5.10 scrub ok 2021-02-10T04:28:48.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:28:48.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[29319]: 10T04:28:47.476664+0000 osd.4 (osd.4) 38 2021-02-10T04:28:48.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[29319]: : cluster [DBG] 4.9 scrub ok 2021-02-10T04:28:48.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:28:48.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[29319]: T04:28:47.496463+0000 osd.5 (osd 2021-02-10T04:28:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[29319]: .5) 35 : cluster [DBG] 5.15 scrub ok 2021-02-10T04:28:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[29319]: audit 2021 2021-02-10T04:28:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[29319]: -02-10T04:28:47.685026+0000 2021-02-10T04:28:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[29319]: mgr.y (mgr.14140) 339 : audit [DBG] 2021-02-10T04:28:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[29319]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:28:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[29319]: cluster 2021-02-10T04:28 2021-02-10T04:28:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[29319]: :47.995140+0000 mgr.y (mgr.14140) 2021-02-10T04:28:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[29319]: 340 : cluster [DBG] pgmap v316: 168 pgs: 10 unknown, 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 124 MiB data, 310 MiB used, 715 GiB / 715 GiB avail; 27 MiB/s rd, 7.4 MiB/s wr, 40 op/s 2021-02-10T04:28:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[29319]: audit 2021-02-10T 2021-02-10T04:28:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[29319]: 04:28:47.996028+0000 mon.a (mon 2021-02-10T04:28:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[29319]: .0) 771 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num_actual", "val": "11"}]: dispatch 2021-02-10T04:28:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:28:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[29319]: -10T04:28:48.028084+0000 mon.a 2021-02-10T04:28:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[29319]: (mon.0) 772 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num_actual", "val": "11"}]': finished 2021-02-10T04:28:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[29319]: 2021-02-10T04:28:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[29319]: cluster 2021-02-10T04:28:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[29319]: 2021-02-10T04:28:48.028240+0000 mon.a ( 2021-02-10T04:28:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[29319]: mon.0) 773 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2021-02-10T04:28:48.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: cluster 2021-02-10T04:28:46.488678 2021-02-10T04:28:48.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 38 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:28:48.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:28:48.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: 04:28:46.488725+0000 osd.1 (osd.1) 39 2021-02-10T04:28:48.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:28:48.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: 2021-02-10T04:28:48.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: cluster 2021-02-10T04:28:48.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: 2021- 2021-02-10T04:28:48.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: 02-10T04:28:47.473019 2021-02-10T04:28:48.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: +0000 osd.7 (osd.7) 38 : cluster [DBG] 2021-02-10T04:28:48.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: 5.10 scrub ok 2021-02-10T04:28:48.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: cluster 2021-02-10T04:28:48.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:28:48.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: 04:28:47.476664+0000 osd.4 (osd.4) 38 : cluster [DBG] 4.9 scrub ok 2021-02-10T04:28:48.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: cluster 2021-02-10T04:28:48.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: 2021-02-10T04:28 2021-02-10T04:28:48.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: :47.496463+0000 osd.5 (osd.5) 35 : cluster [DBG] 5.15 scrub ok 2021-02-10T04:28:48.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:28:48.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: -10T04:28:47.685026+0000 mgr.y (mgr. 2021-02-10T04:28:48.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: 14140) 339 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:28:48.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: cluster 2021-02-10T04:28: 2021-02-10T04:28:48.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: 47.995140+0000 mgr.y (mgr.14140) 340 : 2021-02-10T04:28:48.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: cluster [DBG] pgmap v316: 168 pgs: 10 unknown, 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 124 MiB data, 310 MiB used, 715 GiB / 715 GiB avail; 27 MiB/s rd, 7.4 MiB/s wr, 40 op/s 2021-02-10T04:28:48.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: audit 2021 2021-02-10T04:28:48.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: -02-10T04:28:47.996028+0000 2021-02-10T04:28:48.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: mon.a (mon.0) 771 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num_actual", "val": "11"}]: dispatch 2021-02-10T04:28:48.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:28:48.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: 02-10T04:28:48.028084+0000 2021-02-10T04:28:48.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: mon.a (mon.0) 772 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num_actual", "val": "11"}]': finished 2021-02-10T04:28:48.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: 2021-02-10T04:28:48.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: cluster 2021-02-10T04:28:48.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: 2021-02-10T04:28:48.028240+0000 mon.a 2021-02-10T04:28:48.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: (mon.0) 773 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2021-02-10T04:28:48.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:48 smithi085 conmon[35961]: 2021-02-10T04:28:48.857 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:28:48.887 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:49.041 INFO:tasks.rados.rados.0.smithi085.stdout:128: done (4 left) 2021-02-10T04:28:49.041 INFO:tasks.rados.rados.0.smithi085.stdout:129: done (3 left) 2021-02-10T04:28:49.041 INFO:tasks.rados.rados.0.smithi085.stdout:130: done (2 left) 2021-02-10T04:28:49.041 INFO:tasks.rados.rados.0.smithi085.stdout:131: done (1 left) 2021-02-10T04:28:49.042 INFO:tasks.rados.rados.0.smithi085.stdout:132: done (0 left) 2021-02-10T04:28:49.042 INFO:tasks.rados.rados.0.smithi085.stdout:133: snap_create 2021-02-10T04:28:49.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:49 smithi164 conmon[29288]: cluster 2021-02-10T04:28:48.409805+0000 osd.3 (osd.3) 38 : cluster [DBG] 2.23 scrub ok 2021-02-10T04:28:49.974 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:49 smithi164 conmon[29288]: cluster 2021-02-10T04:28:48.500419+0000 osd.7 (osd.7) 39 : cluster [DBG] 5.9 scrub ok 2021-02-10T04:28:49.974 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:49 smithi164 conmon[29288]: cluster 2021-02-10T04:28:48.700413+0000 osd.0 (osd.0) 40 : cluster [DBG] 6.9 scrub ok 2021-02-10T04:28:49.974 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:49 smithi164 conmon[29288]: cluster 2021-02-10T04:28:49.041505+0000 mon.a (mon.0) 774 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2021-02-10T04:28:49.976 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:49 smithi085 conmon[29319]: cluster 2021-02-10T04:28:48.409805+0000 osd.3 (osd.3) 38 : cluster [DBG] 2.23 scrub ok 2021-02-10T04:28:49.976 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:49 smithi085 conmon[29319]: cluster 2021-02-10T04:28:48.500419+0000 osd.7 (osd.7) 39 : cluster [DBG] 5.9 scrub ok 2021-02-10T04:28:49.976 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:49 smithi085 conmon[29319]: cluster 2021-02-10T04:28:48.700413+0000 osd.0 (osd.0) 40 : cluster [DBG] 6.9 scrub ok 2021-02-10T04:28:49.976 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:49 smithi085 conmon[29319]: cluster 2021-02-10T04:28:49.041505+0000 mon.a (mon.0) 774 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2021-02-10T04:28:49.976 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:49 smithi085 conmon[35961]: cluster 2021-02-10T04:28:48.409805+0000 osd.3 (osd.3) 38 : cluster [DBG] 2.23 scrub ok 2021-02-10T04:28:49.976 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:49 smithi085 conmon[35961]: cluster 2021-02-10T04:28:48.500419+0000 osd.7 (osd.7) 39 : cluster [DBG] 5.9 scrub ok 2021-02-10T04:28:49.976 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:49 smithi085 conmon[35961]: cluster 2021-02-10T04:28:48.700413+0000 osd.0 (osd.0) 40 : cluster [DBG] 6.9 scrub ok 2021-02-10T04:28:49.977 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:49 smithi085 conmon[35961]: cluster 2021-02-10T04:28:49.041505+0000 mon.a (mon.0) 774 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2021-02-10T04:28:49.977 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:28:49.977 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:28:49 smithi085 conmon[49106]: debug 2021-02-10T04:28:49.844+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:50.042 INFO:tasks.rados.rados.0.smithi085.stdout:133: done (0 left) 2021-02-10T04:28:50.043 INFO:tasks.rados.rados.0.smithi085.stdout:134: snap_remove snap 8 2021-02-10T04:28:50.078 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:28:50.114 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:50.341 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:28:50 smithi085 conmon[52836]: debug 2021-02-10T04:28:50.339+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:50.341 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:50 smithi085 conmon[29319]: cluster 2021-02-10T04:28:49.514975+0000 osd.4 2021-02-10T04:28:50.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:50 smithi085 conmon[29319]: (osd.4) 39 : cluster [DBG] 4.d deep-scrub ok 2021-02-10T04:28:50.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:50 smithi085 conmon[29319]: cluster 2021-02-10T04:28:49.522121+0000 osd.5 (osd.5) 36 : cluster [DBG] 2.16 scrub ok 2021-02-10T04:28:50.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:50 smithi085 conmon[29319]: cluster 2021-02-10T04:28:49.995740+0000 mgr.y (mgr.14140) 341 : cluster [DBG] pgmap v319: 168 pgs: 3 peering, 5 unknown, 1 active+clean+snaptrim, 159 active+clean; 119 MiB data, 308 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 1.4 MiB/s wr, 16 op/s 2021-02-10T04:28:50.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:50 smithi085 conmon[29319]: cluster 2021-02-10T04:28:50.041726+0000 mon.a (mon.0) 775 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2021-02-10T04:28:50.342 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:50 smithi085 conmon[35961]: cluster 2021-02-10T04:28:49.514975+0000 2021-02-10T04:28:50.342 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:50 smithi085 conmon[35961]: osd.4 (osd.4) 39 : cluster [DBG] 4.d deep-scrub ok 2021-02-10T04:28:50.342 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:50 smithi085 conmon[35961]: cluster 2021-02-10T04:28:49. 2021-02-10T04:28:50.342 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:50 smithi085 conmon[35961]: 522121+0000 osd.5 (osd.5) 36 : cluster [DBG] 2.16 scrub ok 2021-02-10T04:28:50.342 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:50 smithi085 conmon[35961]: cluster 2021-02-10T04:28:50.342 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:50 smithi085 conmon[35961]: 2021-02-10T04:28:50.343 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:50 smithi085 conmon[35961]: 2021 2021-02-10T04:28:50.343 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:50 smithi085 conmon[35961]: - 2021-02-10T04:28:50.343 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:50 smithi085 conmon[35961]: 02 2021-02-10T04:28:50.343 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:50 smithi085 conmon[35961]: -10T04:28:49. 2021-02-10T04:28:50.343 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:50 smithi085 conmon[35961]: 995740+0000 mgr.y (mgr.14140) 341 : cluster [DBG] pgmap v319: 168 pgs: 3 peering, 5 unknown, 1 active+clean+snaptrim, 159 active+clean; 119 MiB data, 308 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 1.4 MiB/s wr, 16 op/s 2021-02-10T04:28:50.343 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:50 smithi085 conmon[35961]: cluster 2021-02-10T04:28:50.343 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:50 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:28:50.343 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:50 smithi085 conmon[35961]: :28:50.041726+0000 mon.a (mon.0) 775 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2021-02-10T04:28:50.350 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:28:50.351 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:28:50.396 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:28:50.502 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:28:50.524 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:50 smithi164 conmon[29288]: cluster 2021-02-10T04:28:49.514975+0000 osd.4 (osd.4 2021-02-10T04:28:50.524 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:50 smithi164 conmon[29288]: ) 39 : cluster [DBG] 4.d deep-scrub ok 2021-02-10T04:28:50.525 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:50 smithi164 conmon[29288]: cluster 2021-02-10T04:28:49.522121+0000 osd.5 (osd.5) 36 : cluster [DBG] 2.16 scrub ok 2021-02-10T04:28:50.525 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:50 smithi164 conmon[29288]: cluster 2021-02-10T04:28:50.525 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:50 smithi164 conmon[29288]: 2021-02-10T04:28:49.995740+0000 mgr.y 2021-02-10T04:28:50.525 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:50 smithi164 conmon[29288]: (mgr.14140) 341 : cluster [DBG] 2021-02-10T04:28:50.525 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:50 smithi164 conmon[29288]: pgmap v319: 168 pgs: 3 peering, 5 unknown, 1 active+clean+snaptrim, 159 active+clean; 119 MiB data, 308 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 1.4 MiB/s wr, 16 op/s 2021-02-10T04:28:50.525 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:50 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:28:50.525 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:50 smithi164 conmon[29288]: -10T04:28:50.041726+0000 mon.a (mon 2021-02-10T04:28:50.525 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:50 smithi164 conmon[29288]: .0) 775 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2021-02-10T04:28:50.527 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:50.625 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:28:50.729 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:28:50.760 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:50.822 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:28:50 smithi164 conmon[40892]: debug 2021-02-10T04:28:50.600+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:50.858 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:28:50.964 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:28:50.991 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:51.043 INFO:tasks.rados.rados.0.smithi085.stdout:134: done (0 left) 2021-02-10T04:28:51.043 INFO:tasks.rados.rados.0.smithi085.stdout:135: write oid 21 current snap is 9 2021-02-10T04:28:51.043 INFO:tasks.rados.rados.0.smithi085.stdout:135: seq_num 73 ranges {426911=540354,1622009=726683,3121996=104915} 2021-02-10T04:28:51.048 INFO:tasks.rados.rados.0.smithi085.stdout:135: writing smithi08575397-21 from 426911 to 967265 tid 1 2021-02-10T04:28:51.050 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is set 2021-02-10T04:28:51.057 INFO:tasks.rados.rados.0.smithi085.stdout:135: writing smithi08575397-21 from 1622009 to 2348692 tid 2 2021-02-10T04:28:51.063 INFO:tasks.rados.rados.0.smithi085.stdout:135: writing smithi08575397-21 from 3121996 to 3226911 tid 3 2021-02-10T04:28:51.063 INFO:tasks.rados.rados.0.smithi085.stdout:136: read oid 47 snap -1 2021-02-10T04:28:51.063 INFO:tasks.rados.rados.0.smithi085.stdout:136: expect (ObjNum 67 snap 7 seq_num 67) 2021-02-10T04:28:51.063 INFO:tasks.rados.rados.0.smithi085.stdout:137: delete oid 19 current snap is 9 2021-02-10T04:28:51.063 INFO:tasks.rados.rados.0.smithi085.stdout:135: finishing write tid 1 to smithi08575397-21 2021-02-10T04:28:51.063 INFO:tasks.rados.rados.0.smithi085.stdout:135: finishing write tid 2 to smithi08575397-21 2021-02-10T04:28:51.069 INFO:tasks.rados.rados.0.smithi085.stdout:136: expect (ObjNum 67 snap 7 seq_num 67) 2021-02-10T04:28:51.085 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:28:51.087 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:28:50 smithi164 conmon[37263]: debug 2021-02-10T04:28:50.822+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:51.088 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[33677]: debug 2021-02-10T04:28:51.051+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:51.189 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:28:51.225 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:51.247 INFO:tasks.rados.rados.0.smithi085.stdout:136: done (2 left) 2021-02-10T04:28:51.248 INFO:tasks.rados.rados.0.smithi085.stdout:137: done (1 left) 2021-02-10T04:28:51.248 INFO:tasks.rados.rados.0.smithi085.stdout:138: write oid 25 current snap is 9 2021-02-10T04:28:51.249 INFO:tasks.rados.rados.0.smithi085.stdout:138: seq_num 75 ranges {460525=690852,1622620=730352,3060489=200036} 2021-02-10T04:28:51.253 INFO:tasks.rados.rados.0.smithi085.stdout:138: writing smithi08575397-25 from 460525 to 1151377 tid 1 2021-02-10T04:28:51.260 INFO:tasks.rados.rados.0.smithi085.stdout:138: writing smithi08575397-25 from 1622620 to 2352972 tid 2 2021-02-10T04:28:51.265 INFO:tasks.rados.rados.0.smithi085.stdout:138: writing smithi08575397-25 from 3060489 to 3260525 tid 3 2021-02-10T04:28:51.266 INFO:tasks.rados.rados.0.smithi085.stdout:139: write (excl) oid 29 current snap is 9 2021-02-10T04:28:51.266 INFO:tasks.rados.rados.0.smithi085.stdout:139: seq_num 76 ranges {477332=766101,1681102=474010,2687912=589420} 2021-02-10T04:28:51.276 INFO:tasks.rados.rados.0.smithi085.stdout:139: writing smithi08575397-29 from 477332 to 1243433 tid 1 2021-02-10T04:28:51.281 INFO:tasks.rados.rados.0.smithi085.stdout:139: writing smithi08575397-29 from 1681102 to 2155112 tid 2 2021-02-10T04:28:51.289 INFO:tasks.rados.rados.0.smithi085.stdout:139: writing smithi08575397-29 from 2687912 to 3277332 tid 3 2021-02-10T04:28:51.290 INFO:tasks.rados.rados.0.smithi085.stdout:140: write (excl) oid 8 current snap is 9 2021-02-10T04:28:51.290 INFO:tasks.rados.rados.0.smithi085.stdout:135: finishing write tid 3 to smithi08575397-21 2021-02-10T04:28:51.290 INFO:tasks.rados.rados.0.smithi085.stdout:135: finishing write tid 4 to smithi08575397-21 2021-02-10T04:28:51.290 INFO:tasks.rados.rados.0.smithi085.stdout:135: finishing write tid 5 to smithi08575397-21 2021-02-10T04:28:51.290 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 55 (ObjNum 73 snap 9 seq_num 73) dirty exists 2021-02-10T04:28:51.290 INFO:tasks.rados.rados.0.smithi085.stdout:135: left oid 21 (ObjNum 73 snap 9 seq_num 73) 2021-02-10T04:28:51.290 INFO:tasks.rados.rados.0.smithi085.stdout:138: finishing write tid 1 to smithi08575397-25 2021-02-10T04:28:51.290 INFO:tasks.rados.rados.0.smithi085.stdout:138: finishing write tid 2 to smithi08575397-25 2021-02-10T04:28:51.290 INFO:tasks.rados.rados.0.smithi085.stdout:138: finishing write tid 3 to smithi08575397-25 2021-02-10T04:28:51.290 INFO:tasks.rados.rados.0.smithi085.stdout:138: finishing write tid 4 to smithi08575397-25 2021-02-10T04:28:51.291 INFO:tasks.rados.rados.0.smithi085.stdout:138: finishing write tid 5 to smithi08575397-25 2021-02-10T04:28:51.291 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 25 v 11 (ObjNum 75 snap 9 seq_num 75) dirty exists 2021-02-10T04:28:51.291 INFO:tasks.rados.rados.0.smithi085.stdout:138: left oid 25 (ObjNum 75 snap 9 seq_num 75) 2021-02-10T04:28:51.291 INFO:tasks.rados.rados.0.smithi085.stdout:139: finishing write tid 1 to smithi08575397-29 2021-02-10T04:28:51.291 INFO:tasks.rados.rados.0.smithi085.stdout:139: finishing write tid 2 to smithi08575397-29 2021-02-10T04:28:51.291 INFO:tasks.rados.rados.0.smithi085.stdout:140: seq_num 77 ranges {494139=557703,1455937=734021,2948041=346098} 2021-02-10T04:28:51.294 INFO:tasks.rados.rados.0.smithi085.stdout:140: writing smithi08575397-8 from 494139 to 1051842 tid 1 2021-02-10T04:28:51.303 INFO:tasks.rados.rados.0.smithi085.stdout:140: writing smithi08575397-8 from 1455937 to 2189958 tid 2 2021-02-10T04:28:51.308 INFO:tasks.rados.rados.0.smithi085.stdout:140: writing smithi08575397-8 from 2948041 to 3294139 tid 3 2021-02-10T04:28:51.308 INFO:tasks.rados.rados.0.smithi085.stdout:139: finishing write tid 3 to smithi08575397-29 2021-02-10T04:28:51.308 INFO:tasks.rados.rados.0.smithi085.stdout:139: finishing write tid 4 to smithi08575397-29 2021-02-10T04:28:51.308 INFO:tasks.rados.rados.0.smithi085.stdout:139: finishing write tid 5 to smithi08575397-29 2021-02-10T04:28:51.308 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 29 v 25 (ObjNum 76 snap 9 seq_num 76) dirty exists 2021-02-10T04:28:51.308 INFO:tasks.rados.rados.0.smithi085.stdout:139: left oid 29 (ObjNum 76 snap 9 seq_num 76) 2021-02-10T04:28:51.308 INFO:tasks.rados.rados.0.smithi085.stdout:140: finishing write tid 1 to smithi08575397-8 2021-02-10T04:28:51.309 INFO:tasks.rados.rados.0.smithi085.stdout:140: finishing write tid 2 to smithi08575397-8 2021-02-10T04:28:51.309 INFO:tasks.rados.rados.0.smithi085.stdout:135: done (3 left) 2021-02-10T04:28:51.309 INFO:tasks.rados.rados.0.smithi085.stdout:138: done (2 left) 2021-02-10T04:28:51.309 INFO:tasks.rados.rados.0.smithi085.stdout:139: done (1 left) 2021-02-10T04:28:51.309 INFO:tasks.rados.rados.0.smithi085.stdout:141: delete oid 41 current snap is 9 2021-02-10T04:28:51.312 INFO:tasks.rados.rados.0.smithi085.stdout:140: finishing write tid 3 to smithi08575397-8 2021-02-10T04:28:51.312 INFO:tasks.rados.rados.0.smithi085.stdout:140: finishing write tid 4 to smithi08575397-8 2021-02-10T04:28:51.312 INFO:tasks.rados.rados.0.smithi085.stdout:141: done (1 left) 2021-02-10T04:28:51.312 INFO:tasks.rados.rados.0.smithi085.stdout:142: rollback oid 10 current snap is 9 2021-02-10T04:28:51.312 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 10 to 7 2021-02-10T04:28:51.312 INFO:tasks.rados.rados.0.smithi085.stdout:143: read oid 34 snap -1 2021-02-10T04:28:51.312 INFO:tasks.rados.rados.0.smithi085.stdout:143: expect deleted 2021-02-10T04:28:51.312 INFO:tasks.rados.rados.0.smithi085.stdout:140: finishing write tid 5 to smithi08575397-8 2021-02-10T04:28:51.312 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 24 (ObjNum 77 snap 9 seq_num 77) dirty exists 2021-02-10T04:28:51.313 INFO:tasks.rados.rados.0.smithi085.stdout:140: left oid 8 (ObjNum 77 snap 9 seq_num 77) 2021-02-10T04:28:51.313 INFO:tasks.rados.rados.0.smithi085.stdout:140: done (2 left) 2021-02-10T04:28:51.313 INFO:tasks.rados.rados.0.smithi085.stdout:144: write (excl) oid 21 current snap is 9 2021-02-10T04:28:51.313 INFO:tasks.rados.rados.0.smithi085.stdout:144: seq_num 79 ranges {527753=708201,1856548=737690,3286534=41219} 2021-02-10T04:28:51.319 INFO:tasks.rados.rados.0.smithi085.stdout:144: writing smithi08575397-21 from 527753 to 1235954 tid 1 2021-02-10T04:28:51.327 INFO:tasks.rados.rados.0.smithi085.stdout:144: writing smithi08575397-21 from 1856548 to 2594238 tid 2 2021-02-10T04:28:51.332 INFO:tasks.rados.rados.0.smithi085.stdout:144: writing smithi08575397-21 from 3286534 to 3327753 tid 3 2021-02-10T04:28:51.332 INFO:tasks.rados.rados.0.smithi085.stdout:142: finishing rollback tid 0 to smithi08575397-10 2021-02-10T04:28:51.332 INFO:tasks.rados.rados.0.smithi085.stdout:142: finishing rollback tid 1 to smithi08575397-10 2021-02-10T04:28:51.332 INFO:tasks.rados.rados.0.smithi085.stdout:142: finishing rollback tid 2 to smithi08575397-10 2021-02-10T04:28:51.332 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 10 v 27 (ObjNum 71 snap 7 seq_num 71) dirty exists 2021-02-10T04:28:51.332 INFO:tasks.rados.rados.0.smithi085.stdout:144: finishing write tid 1 to smithi08575397-21 2021-02-10T04:28:51.332 INFO:tasks.rados.rados.0.smithi085.stdout:142: done (2 left) 2021-02-10T04:28:51.332 INFO:tasks.rados.rados.0.smithi085.stdout:143: done (1 left) 2021-02-10T04:28:51.332 INFO:tasks.rados.rados.0.smithi085.stdout:145: write oid 25 current snap is 9 2021-02-10T04:28:51.332 INFO:tasks.rados.rados.0.smithi085.stdout:145: seq_num 80 ranges {544560=783450,1915030=481348,2797604=479761} 2021-02-10T04:28:51.340 INFO:tasks.rados.rados.0.smithi085.stdout:145: writing smithi08575397-25 from 544560 to 1328010 tid 1 2021-02-10T04:28:51.347 INFO:tasks.rados.rados.0.smithi085.stdout:145: writing smithi08575397-25 from 1915030 to 2396378 tid 2 2021-02-10T04:28:51.352 INFO:tasks.rados.rados.0.smithi085.stdout:145: writing smithi08575397-25 from 2797604 to 3277365 tid 3 2021-02-10T04:28:51.352 INFO:tasks.rados.rados.0.smithi085.stdout:146: snap_create 2021-02-10T04:28:51.352 INFO:tasks.rados.rados.0.smithi085.stdout:144: finishing write tid 2 to smithi08575397-21 2021-02-10T04:28:51.353 INFO:tasks.rados.rados.0.smithi085.stdout:144: finishing write tid 3 to smithi08575397-21 2021-02-10T04:28:51.353 INFO:tasks.rados.rados.0.smithi085.stdout:144: finishing write tid 4 to smithi08575397-21 2021-02-10T04:28:51.353 INFO:tasks.rados.rados.0.smithi085.stdout:144: finishing write tid 5 to smithi08575397-21 2021-02-10T04:28:51.353 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 61 (ObjNum 79 snap 9 seq_num 79) dirty exists 2021-02-10T04:28:51.353 INFO:tasks.rados.rados.0.smithi085.stdout:144: left oid 21 (ObjNum 79 snap 9 seq_num 79) 2021-02-10T04:28:51.353 INFO:tasks.rados.rados.0.smithi085.stdout:145: finishing write tid 1 to smithi08575397-25 2021-02-10T04:28:51.353 INFO:tasks.rados.rados.0.smithi085.stdout:145: finishing write tid 2 to smithi08575397-25 2021-02-10T04:28:51.361 INFO:tasks.rados.rados.0.smithi085.stdout:145: finishing write tid 3 to smithi08575397-25 2021-02-10T04:28:51.361 INFO:tasks.rados.rados.0.smithi085.stdout:145: finishing write tid 4 to smithi08575397-25 2021-02-10T04:28:51.361 INFO:tasks.rados.rados.0.smithi085.stdout:145: finishing write tid 5 to smithi08575397-25 2021-02-10T04:28:51.361 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 25 v 15 (ObjNum 80 snap 9 seq_num 80) dirty exists 2021-02-10T04:28:51.361 INFO:tasks.rados.rados.0.smithi085.stdout:145: left oid 25 (ObjNum 80 snap 9 seq_num 80) 2021-02-10T04:28:51.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: cluster 2021-02-10T04:28:49.959157+0000 osd.1 (osd. 2021-02-10T04:28:51.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: 1) 40 : cluster [DBG] 4.2 scrub ok 2021-02-10T04:28:51.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: cephadm 2021-02-10T04:28:50.391295+0000 mgr.y (mgr.14140) 342 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:28:51.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: cephadm 2021-02-10T04:28:50.391773+0000 mgr.y (mgr.14140 2021-02-10T04:28:51.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: ) 343 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:28:51.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:28:51.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: -10T04:28:50.493635+0000 osd.5 (osd 2021-02-10T04:28:51.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: .5) 37 : cluster [DBG] 4.15 scrub ok 2021-02-10T04:28:51.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: cluster 2021-02-10T04:28:51.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: 2021-02-10T04:28:50.696117+0000 2021-02-10T04:28:51.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: osd.0 (osd.0) 41 : cluster [DBG] 2021-02-10T04:28:51.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: 4.b scrub ok 2021-02-10T04:28:51.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: audit 2021-02-10T04:28 2021-02-10T04:28:51.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: :50.849597+0000 mon.c (mon.1) 39 2021-02-10T04:28:51.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: : audit [INF] from='client.? 172.21.15.85:0/3328779771' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:28:51.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:28:51.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: -10T04:28:50.849970+0000 mon.a ( 2021-02-10T04:28:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: mon.0) 776 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:28:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:28:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: T04:28:50.945819+0000 mon.a (mon.0 2021-02-10T04:28:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: ) 777 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:28:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:28:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: 02-10T04:28:51.035951+0000 mon.a 2021-02-10T04:28:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: (mon.0) 778 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:28:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:28:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: -10T04:28:51.042755+0000 mon.a (mon. 2021-02-10T04:28:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: 0) 779 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:28:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: cluster 2021-02-10T04:28:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: 2021-02-10T04:28:51.042877+0000 2021-02-10T04:28:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: mon.a (mon.0) 780 : cluster [DBG] 2021-02-10T04:28:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:51 smithi164 conmon[29288]: osdmap e115: 8 total, 8 up, 8 in 2021-02-10T04:28:51.441 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[45443]: debug 2021-02-10T04:28:51.310+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:51.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[29319]: cluster 2021-02-10T04:28:49.959157+0000 osd.1 (osd.1) 40 : cluster [DBG] 4.2 scrub ok 2021-02-10T04:28:51.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[29319]: cephadm 2021-02-10T04:28:50.391295+0000 mgr.y (mgr.14140) 342 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:28:51.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[29319]: cephadm 2021-02-10T04:28:50.391773+0000 mgr.y (mgr.14140) 343 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:28:51.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[29319]: cluster 2021-02-10T04:28:50.493635+0000 osd.5 (osd.5) 37 : cluster [DBG] 4.15 scrub ok 2021-02-10T04:28:51.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[29319]: cluster 2021-02-10T04:28:50.696117+0000 osd.0 (osd.0) 41 : cluster [DBG] 4.b scrub ok 2021-02-10T04:28:51.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[29319]: audit 2021-02-10T04:28:50.849597+0000 mon.c (mon.1) 39 : audit [INF] from='client.? 172.21.15.85:0/3328779771' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:28:51.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:28:51.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[29319]: -10T04:28:50.849970+0000 mon.a (mon.0) 776 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:28:51.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[29319]: audit 2021-02-10T04:28:50.945819+0000 mon.a (mon.0) 777 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:28:51.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[29319]: cluster 2021-02-10T04:28:51.035951+0000 mon.a (mon.0) 778 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:28:51.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[29319]: audit 2021-02-10T04:28:51.042755+0000 mon.a (mon.0) 779 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:28:51.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[29319]: cluster 2021-02-10T04:28:51.042877+0000 mon.a (mon.0) 780 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2021-02-10T04:28:51.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: cluster 2021-02-10T04:28:51.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: 2021-02-10T04:28:51.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: 2021 2021-02-10T04:28:51.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: -02-10T04:28:49 2021-02-10T04:28:51.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: .959157+0000 osd.1 (osd.1) 40 : cluster [DBG] 4.2 scrub ok 2021-02-10T04:28:51.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: cephadm 2021-02-10T04:28:51.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: 2021-02-10T04:28:50.391295+0000 2021-02-10T04:28:51.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: mgr.y (mgr.14140) 342 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:28:51.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: cephadm 2021- 2021-02-10T04:28:51.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: 02-10T04:28:50.391773+0000 mgr.y (mgr.14140) 343 : 2021-02-10T04:28:51.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:28:51.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:28:51.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: 28:50.493635+0000 osd.5 (osd.5) 37 2021-02-10T04:28:51.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: : cluster [DBG] 4.15 scrub ok 2021-02-10T04:28:51.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:28:51.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: 04:28:50.696117+0000 osd.0 (osd.0 2021-02-10T04:28:51.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: ) 41 : cluster [DBG] 4.b scrub ok 2021-02-10T04:28:51.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:28:51.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: 10T04:28:50.849597 2021-02-10T04:28:51.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: +0000 mon.c (mon.1) 39 2021-02-10T04:28:51.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: : audit [INF] from='client.? 172.21.15.85:0/3328779771' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:28:51.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: audit 2021-02-10T04:28:51.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:28:51.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: 28:50.849970+0000 mon.a ( 2021-02-10T04:28:51.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: mon.0) 776 : audit 2021-02-10T04:28:51.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:28:51.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:28:51.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: 02-10T04:28:50 2021-02-10T04:28:51.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: .945819+0000 mon.a (mon. 2021-02-10T04:28:51.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: 0) 777 : audit [INF] 2021-02-10T04:28:51.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:28:51.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: cluster 2021-02-10T04:28:51.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: 2021-02-10T04:28 2021-02-10T04:28:51.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: :51.035951+0000 mon.a ( 2021-02-10T04:28:51.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: mon.0) 778 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:28:51.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: 2021-02-10T04:28:51.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: audit 2021-02-10T04:28:51.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: 2021-02-10T04:28:51.042755+0000 mon.a (mon.0 2021-02-10T04:28:51.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: ) 779 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:28:51.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:28:51.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: 28:51.042877+0000 mon.a (mon.0) 780 : 2021-02-10T04:28:51.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:51 smithi085 conmon[35961]: cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2021-02-10T04:28:51.566 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:28:51.978 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:28:52.211 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:28:52.274 INFO:tasks.rados.rados.0.smithi085.stdout:144: done (2 left) 2021-02-10T04:28:52.274 INFO:tasks.rados.rados.0.smithi085.stdout:145: done (1 left) 2021-02-10T04:28:52.274 INFO:tasks.rados.rados.0.smithi085.stdout:146: done (0 left) 2021-02-10T04:28:52.274 INFO:tasks.rados.rados.0.smithi085.stdout:147: read oid 29 snap 9 2021-02-10T04:28:52.274 INFO:tasks.rados.rados.0.smithi085.stdout:147: expect (ObjNum 76 snap 9 seq_num 76) 2021-02-10T04:28:52.275 INFO:tasks.rados.rados.0.smithi085.stdout:148: write (excl) oid 21 current snap is 10 2021-02-10T04:28:52.275 INFO:tasks.rados.rados.0.smithi085.stdout:148: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2021-02-10T04:28:52.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:52 smithi085 conmon[29319]: cluster 2021-02-10T04:28:50.992896+0000 osd.1 (osd. 2021-02-10T04:28:52.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:52 smithi085 conmon[29319]: 1) 41 : cluster [DBG] 3.0 scrub ok 2021-02-10T04:28:52.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:52 smithi085 conmon[29319]: cluster 2021-02-10T04:28:51.996222+0000 mgr.y (mgr.14140) 344 : cluster [DBG] pgmap v322: 168 pgs: 3 peering, 1 unknown, 3 active+clean+snaptrim, 161 active+clean; 118 MiB data, 302 MiB used, 715 GiB / 715 GiB avail; 9.1 MiB/s rd, 2.9 MiB/s wr, 19 op/s 2021-02-10T04:28:52.279 INFO:tasks.rados.rados.0.smithi085.stdout:148: writing smithi08575397-21 from 561367 to 1020066 tid 1 2021-02-10T04:28:52.288 INFO:tasks.rados.rados.0.smithi085.stdout:148: writing smithi08575397-21 from 1573512 to 2314871 tid 2 2021-02-10T04:28:52.295 INFO:tasks.rados.rados.0.smithi085.stdout:148: writing smithi08575397-21 from 2941380 to 3361367 tid 3 2021-02-10T04:28:52.296 INFO:tasks.rados.rados.0.smithi085.stdout:149: delete oid 23 current snap is 10 2021-02-10T04:28:52.296 INFO:tasks.rados.rados.0.smithi085.stdout:147: expect (ObjNum 76 snap 9 seq_num 76) 2021-02-10T04:28:52.512 INFO:tasks.rados.rados.0.smithi085.stdout:147: done (2 left) 2021-02-10T04:28:52.512 INFO:tasks.rados.rados.0.smithi085.stdout:149: done (1 left) 2021-02-10T04:28:52.512 INFO:tasks.rados.rados.0.smithi085.stdout:150: read oid 14 snap 7 2021-02-10T04:28:52.512 INFO:tasks.rados.rados.0.smithi085.stdout:150: expect (ObjNum 13 snap 0 seq_num 13) 2021-02-10T04:28:52.513 INFO:tasks.rados.rados.0.smithi085.stdout:151: snap_remove snap 9 2021-02-10T04:28:52.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:52 smithi164 conmon[29288]: cluster 2021-02-10T04:28:50.992896+0000 osd.1 2021-02-10T04:28:52.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:52 smithi164 conmon[29288]: (osd.1) 41 : cluster [DBG] 3.0 scrub ok 2021-02-10T04:28:52.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:52 smithi164 conmon[29288]: cluster 2021-02-10T04:28:51.996222+0000 mgr.y (mgr.14140) 344 : cluster [DBG] pgmap v322: 168 pgs: 3 peering, 1 unknown, 3 active+clean+snaptrim, 161 active+clean; 118 MiB data, 302 MiB used, 715 GiB / 715 GiB avail; 9.1 MiB/s rd, 2.9 MiB/s wr, 19 op/s 2021-02-10T04:28:52.690 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:52 smithi085 conmon[35961]: cluster 2021-02-10T04:28:52.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:52 smithi085 conmon[35961]: 2021-02-10T04:28:50.992896+0000 2021-02-10T04:28:52.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:52 smithi085 conmon[35961]: osd.1 (osd.1) 41 : cluster [DBG] 3.0 scrub ok 2021-02-10T04:28:52.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:52 smithi085 conmon[35961]: cluster 2021-02-10T04:28:52.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:52 smithi085 conmon[35961]: 2021-02-10T04:28:51.996222+0000 mgr.y (mgr.14140) 344 : cluster [DBG] pgmap v322: 168 pgs: 3 peering, 1 unknown, 3 active+clean+snaptrim, 161 active+clean; 118 MiB data, 302 MiB used, 715 GiB / 715 GiB avail; 9.1 MiB/s rd, 2.9 MiB/s wr, 19 op/s 2021-02-10T04:28:52.918 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:28:52.947 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:53.036 INFO:tasks.rados.rados.0.smithi085.stdout:151: done (2 left) 2021-02-10T04:28:53.037 INFO:tasks.rados.rados.0.smithi085.stdout:152: write oid 46 current snap is 10 2021-02-10T04:28:53.037 INFO:tasks.rados.rados.0.smithi085.stdout:152: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2021-02-10T04:28:53.043 INFO:tasks.rados.rados.0.smithi085.stdout:152: writing smithi08575397-46 from 594981 to 1204178 tid 1 2021-02-10T04:28:53.052 INFO:tasks.rados.rados.0.smithi085.stdout:152: writing smithi08575397-46 from 1974123 to 2719151 tid 2 2021-02-10T04:28:53.056 INFO:tasks.rados.rados.0.smithi085.stdout:152: writing smithi08575397-46 from 3279873 to 3394981 tid 3 2021-02-10T04:28:53.056 INFO:tasks.rados.rados.0.smithi085.stdout:153: read oid 35 snap -1 2021-02-10T04:28:53.056 INFO:tasks.rados.rados.0.smithi085.stdout:153: expect (ObjNum 70 snap 7 seq_num 70) 2021-02-10T04:28:53.056 INFO:tasks.rados.rados.0.smithi085.stdout:154: snap_create 2021-02-10T04:28:53.057 INFO:tasks.rados.rados.0.smithi085.stdout:148: finishing write tid 1 to smithi08575397-21 2021-02-10T04:28:53.057 INFO:tasks.rados.rados.0.smithi085.stdout:148: finishing write tid 2 to smithi08575397-21 2021-02-10T04:28:53.057 INFO:tasks.rados.rados.0.smithi085.stdout:148: finishing write tid 3 to smithi08575397-21 2021-02-10T04:28:53.057 INFO:tasks.rados.rados.0.smithi085.stdout:148: finishing write tid 4 to smithi08575397-21 2021-02-10T04:28:53.057 INFO:tasks.rados.rados.0.smithi085.stdout:148: finishing write tid 5 to smithi08575397-21 2021-02-10T04:28:53.057 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 66 (ObjNum 81 snap 10 seq_num 81) dirty exists 2021-02-10T04:28:53.057 INFO:tasks.rados.rados.0.smithi085.stdout:148: left oid 21 (ObjNum 81 snap 10 seq_num 81) 2021-02-10T04:28:53.057 INFO:tasks.rados.rados.0.smithi085.stdout:150: expect (ObjNum 13 snap 0 seq_num 13) 2021-02-10T04:28:53.151 INFO:tasks.rados.rados.0.smithi085.stdout:152: finishing write tid 1 to smithi08575397-46 2021-02-10T04:28:53.151 INFO:tasks.rados.rados.0.smithi085.stdout:152: finishing write tid 2 to smithi08575397-46 2021-02-10T04:28:53.152 INFO:tasks.rados.rados.0.smithi085.stdout:152: finishing write tid 3 to smithi08575397-46 2021-02-10T04:28:53.152 INFO:tasks.rados.rados.0.smithi085.stdout:152: finishing write tid 4 to smithi08575397-46 2021-02-10T04:28:53.152 INFO:tasks.rados.rados.0.smithi085.stdout:152: finishing write tid 5 to smithi08575397-46 2021-02-10T04:28:53.152 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 30 (ObjNum 83 snap 10 seq_num 83) dirty exists 2021-02-10T04:28:53.152 INFO:tasks.rados.rados.0.smithi085.stdout:152: left oid 46 (ObjNum 83 snap 10 seq_num 83) 2021-02-10T04:28:53.152 INFO:tasks.rados.rados.0.smithi085.stdout:153: expect (ObjNum 70 snap 7 seq_num 70) 2021-02-10T04:28:53.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:53 smithi164 conmon[29288]: cluster 2021-02-10T04:28:52.271859+0000 mon.a (mon. 2021-02-10T04:28:53.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:53 smithi164 conmon[29288]: 0) 781 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2021-02-10T04:28:53.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:53 smithi164 conmon[29288]: cluster 2021-02-10T04:28:53.029857+0000 mon.a (mon.0) 782 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2021-02-10T04:28:53.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:53 smithi085 conmon[29319]: cluster 2021-02-10T04:28:52.271859+0000 mon.a ( 2021-02-10T04:28:53.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:53 smithi085 conmon[29319]: mon.0) 781 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2021-02-10T04:28:53.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:53 smithi085 conmon[29319]: cluster 2021-02-10T04:28:53.029857+0000 mon.a (mon.0) 782 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2021-02-10T04:28:53.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:53 smithi085 conmon[35961]: cluster 2021-02-10T04:28:52.271859+0000 2021-02-10T04:28:53.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:53 smithi085 conmon[35961]: mon.a (mon.0) 781 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2021-02-10T04:28:53.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:53 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:28:53.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:53 smithi085 conmon[35961]: -10T04:28:53.029857+0000 mon.a (mon.0) 782 : cluster 2021-02-10T04:28:53.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:53 smithi085 conmon[35961]: [DBG] osdmap e117: 8 total, 8 up, 8 in 2021-02-10T04:28:54.044 INFO:tasks.rados.rados.0.smithi085.stdout:148: done (4 left) 2021-02-10T04:28:54.044 INFO:tasks.rados.rados.0.smithi085.stdout:150: done (3 left) 2021-02-10T04:28:54.044 INFO:tasks.rados.rados.0.smithi085.stdout:152: done (2 left) 2021-02-10T04:28:54.044 INFO:tasks.rados.rados.0.smithi085.stdout:153: done (1 left) 2021-02-10T04:28:54.044 INFO:tasks.rados.rados.0.smithi085.stdout:154: done (0 left) 2021-02-10T04:28:54.044 INFO:tasks.rados.rados.0.smithi085.stdout:155: rollback oid 38 current snap is 11 2021-02-10T04:28:54.045 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 38 to 10 2021-02-10T04:28:54.045 INFO:tasks.rados.rados.0.smithi085.stdout:156: write oid 33 current snap is 11 2021-02-10T04:28:54.045 INFO:tasks.rados.rados.0.smithi085.stdout:156: seq_num 84 ranges {611788=400799,1748958=488686,2907296=504492} 2021-02-10T04:28:54.050 INFO:tasks.rados.rados.0.smithi085.stdout:156: writing smithi08575397-33 from 611788 to 1012587 tid 1 2021-02-10T04:28:54.057 INFO:tasks.rados.rados.0.smithi085.stdout:156: writing smithi08575397-33 from 1748958 to 2237644 tid 2 2021-02-10T04:28:54.065 INFO:tasks.rados.rados.0.smithi085.stdout:156: writing smithi08575397-33 from 2907296 to 3411788 tid 3 2021-02-10T04:28:54.065 INFO:tasks.rados.rados.0.smithi085.stdout:157: read oid 34 snap 10 2021-02-10T04:28:54.065 INFO:tasks.rados.rados.0.smithi085.stdout:157: expect deleted 2021-02-10T04:28:54.065 INFO:tasks.rados.rados.0.smithi085.stdout:158: write (excl) oid 22 current snap is 11 2021-02-10T04:28:54.065 INFO:tasks.rados.rados.0.smithi085.stdout:156: finishing write tid 1 to smithi08575397-33 2021-02-10T04:28:54.065 INFO:tasks.rados.rados.0.smithi085.stdout:155: finishing rollback tid 0 to smithi08575397-38 2021-02-10T04:28:54.065 INFO:tasks.rados.rados.0.smithi085.stdout:155: finishing rollback tid 1 to smithi08575397-38 2021-02-10T04:28:54.065 INFO:tasks.rados.rados.0.smithi085.stdout:158: seq_num 85 ranges {628595=476048,1807440=632344,2934719=452474} 2021-02-10T04:28:54.071 INFO:tasks.rados.rados.0.smithi085.stdout:158: writing smithi08575397-22 from 628595 to 1104643 tid 1 2021-02-10T04:28:54.079 INFO:tasks.rados.rados.0.smithi085.stdout:158: writing smithi08575397-22 from 1807440 to 2439784 tid 2 2021-02-10T04:28:54.085 INFO:tasks.rados.rados.0.smithi085.stdout:158: writing smithi08575397-22 from 2934719 to 3387193 tid 3 2021-02-10T04:28:54.086 INFO:tasks.rados.rados.0.smithi085.stdout:155: finishing rollback tid 2 to smithi08575397-38 2021-02-10T04:28:54.086 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 34 (ObjNum 37 snap 0 seq_num 37) dirty exists 2021-02-10T04:28:54.086 INFO:tasks.rados.rados.0.smithi085.stdout:159: read oid 19 snap -1 2021-02-10T04:28:54.086 INFO:tasks.rados.rados.0.smithi085.stdout:159: expect deleted 2021-02-10T04:28:54.086 INFO:tasks.rados.rados.0.smithi085.stdout:155: done (4 left) 2021-02-10T04:28:54.086 INFO:tasks.rados.rados.0.smithi085.stdout:160: delete oid 49 current snap is 11 2021-02-10T04:28:54.086 INFO:tasks.rados.rados.0.smithi085.stdout:156: finishing write tid 2 to smithi08575397-33 2021-02-10T04:28:54.086 INFO:tasks.rados.rados.0.smithi085.stdout:156: finishing write tid 3 to smithi08575397-33 2021-02-10T04:28:54.086 INFO:tasks.rados.rados.0.smithi085.stdout:156: finishing write tid 4 to smithi08575397-33 2021-02-10T04:28:54.086 INFO:tasks.rados.rados.0.smithi085.stdout:156: finishing write tid 5 to smithi08575397-33 2021-02-10T04:28:54.086 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 33 v 23 (ObjNum 84 snap 11 seq_num 84) dirty exists 2021-02-10T04:28:54.086 INFO:tasks.rados.rados.0.smithi085.stdout:156: left oid 33 (ObjNum 84 snap 11 seq_num 84) 2021-02-10T04:28:54.087 INFO:tasks.rados.rados.0.smithi085.stdout:158: finishing write tid 1 to smithi08575397-22 2021-02-10T04:28:54.087 INFO:tasks.rados.rados.0.smithi085.stdout:158: finishing write tid 2 to smithi08575397-22 2021-02-10T04:28:54.087 INFO:tasks.rados.rados.0.smithi085.stdout:156: done (4 left) 2021-02-10T04:28:54.088 INFO:tasks.rados.rados.0.smithi085.stdout:157: done (3 left) 2021-02-10T04:28:54.088 INFO:tasks.rados.rados.0.smithi085.stdout:159: done (2 left) 2021-02-10T04:28:54.088 INFO:tasks.rados.rados.0.smithi085.stdout:160: done (1 left) 2021-02-10T04:28:54.088 INFO:tasks.rados.rados.0.smithi085.stdout:161: write (excl) oid 25 current snap is 11 2021-02-10T04:28:54.088 INFO:tasks.rados.rados.0.smithi085.stdout:161: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2021-02-10T04:28:54.095 INFO:tasks.rados.rados.0.smithi085.stdout:161: writing smithi08575397-25 from 662209 to 1288755 tid 1 2021-02-10T04:28:54.102 INFO:tasks.rados.rados.0.smithi085.stdout:161: writing smithi08575397-25 from 1808051 to 2444064 tid 2 2021-02-10T04:28:54.109 INFO:tasks.rados.rados.0.smithi085.stdout:161: writing smithi08575397-25 from 2873212 to 3462209 tid 3 2021-02-10T04:28:54.109 INFO:tasks.rados.rados.0.smithi085.stdout:162: snap_create 2021-02-10T04:28:54.109 INFO:tasks.rados.rados.0.smithi085.stdout:158: finishing write tid 3 to smithi08575397-22 2021-02-10T04:28:54.109 INFO:tasks.rados.rados.0.smithi085.stdout:158: finishing write tid 4 to smithi08575397-22 2021-02-10T04:28:54.109 INFO:tasks.rados.rados.0.smithi085.stdout:158: finishing write tid 5 to smithi08575397-22 2021-02-10T04:28:54.109 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 29 (ObjNum 85 snap 11 seq_num 85) dirty exists 2021-02-10T04:28:54.109 INFO:tasks.rados.rados.0.smithi085.stdout:158: left oid 22 (ObjNum 85 snap 11 seq_num 85) 2021-02-10T04:28:54.110 INFO:tasks.rados.rados.0.smithi085.stdout:161: finishing write tid 1 to smithi08575397-25 2021-02-10T04:28:54.110 INFO:tasks.rados.rados.0.smithi085.stdout:161: finishing write tid 2 to smithi08575397-25 2021-02-10T04:28:54.113 INFO:tasks.rados.rados.0.smithi085.stdout:161: finishing write tid 3 to smithi08575397-25 2021-02-10T04:28:54.113 INFO:tasks.rados.rados.0.smithi085.stdout:161: finishing write tid 4 to smithi08575397-25 2021-02-10T04:28:54.113 INFO:tasks.rados.rados.0.smithi085.stdout:161: finishing write tid 5 to smithi08575397-25 2021-02-10T04:28:54.113 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 25 v 20 (ObjNum 87 snap 11 seq_num 87) dirty exists 2021-02-10T04:28:54.114 INFO:tasks.rados.rados.0.smithi085.stdout:161: left oid 25 (ObjNum 87 snap 11 seq_num 87) 2021-02-10T04:28:54.301 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:28:54.406 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:28:54.430 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:54.431 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.2 dump_blocked_ops 2021-02-10T04:28:54.475 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:28:54 smithi085 conmon[49106]: debug 2021-02-10T04:28:54.206+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:54.516 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:28:54.620 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:28:54.644 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:54.701 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:28:54 smithi164 conmon[33677]: debug 2021-02-10T04:28:54.499+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:54.701 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:28:54 smithi164 conmon[33677]: debug 2021-02-10T04:28:54.700+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:54.732 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:28:54.754 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:28:54.837 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:28:54.860 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:54.941 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:28:54.978 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-02-10T04:28:54.978 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:28:54.978 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.000000 2021-02-10T04:28:54.979 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd primary-affinity 7 0 2021-02-10T04:28:55.043 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:28:54 smithi164 conmon[44544]: debug 2021-02-10T04:28:54.919+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:55.044 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:55 smithi164 conmon[29288]: cluster 2021-02-10T04:28:53.996712+0000 mgr.y (mgr. 2021-02-10T04:28:55.044 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:55 smithi164 conmon[29288]: 14140) 345 : cluster [DBG] pgmap v325: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 118 MiB data, 311 MiB used, 715 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.8 MiB/s wr, 12 op/s 2021-02-10T04:28:55.044 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:55 smithi164 conmon[29288]: cluster 2021-02-10T04:28:54.044107+0000 mon.a (mon.0) 783 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2021-02-10T04:28:55.046 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:28:55.054 INFO:tasks.rados.rados.0.smithi085.stdout:158: done (2 left) 2021-02-10T04:28:55.054 INFO:tasks.rados.rados.0.smithi085.stdout:161: done (1 left) 2021-02-10T04:28:55.055 INFO:tasks.rados.rados.0.smithi085.stdout:162: done (0 left) 2021-02-10T04:28:55.055 INFO:tasks.rados.rados.0.smithi085.stdout:163: snap_create 2021-02-10T04:28:55.084 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:55.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:55 smithi085 conmon[29319]: cluster 2021-02-10T04:28:53.996712+0000 mgr.y 2021-02-10T04:28:55.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:55 smithi085 conmon[29319]: (mgr.14140) 345 : cluster [DBG] pgmap v325: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 118 MiB data, 311 MiB used, 715 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.8 MiB/s wr, 12 op/s 2021-02-10T04:28:55.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:55 smithi085 conmon[29319]: cluster 2021-02-10T04:28:54.044107+0000 mon.a (mon.0) 783 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2021-02-10T04:28:55.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:55 smithi085 conmon[35961]: cluster 2021-02-10T04:28:53. 2021-02-10T04:28:55.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:55 smithi085 conmon[35961]: 996712+0000 mgr.y (mgr.14140) 345 : cluster [DBG] pgmap v325: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 118 MiB data, 311 MiB used, 715 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.8 MiB/s wr, 12 op/s 2021-02-10T04:28:55.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:55 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:28:55.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:55 smithi085 conmon[35961]: 02-10T04:28:54.044107+0000 mon.a 2021-02-10T04:28:55.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:55 smithi085 conmon[35961]: (mon.0) 783 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2021-02-10T04:28:55.270 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:28:56.060 INFO:tasks.rados.rados.0.smithi085.stdout:163: done (0 left) 2021-02-10T04:28:56.061 INFO:tasks.rados.rados.0.smithi085.stdout:164: write (excl) oid 5 current snap is 13 2021-02-10T04:28:56.061 INFO:tasks.rados.rados.0.smithi085.stdout:164: seq_num 88 ranges {679016=701795,1866533=496024,2900635=402643} 2021-02-10T04:28:56.064 INFO:tasks.rados.rados.0.smithi085.stdout:164: writing smithi08575397-5 from 679016 to 1380811 tid 1 2021-02-10T04:28:56.070 INFO:tasks.rados.rados.0.smithi085.stdout:164: writing smithi08575397-5 from 1866533 to 2362557 tid 2 2021-02-10T04:28:56.076 INFO:tasks.rados.rados.0.smithi085.stdout:164: writing smithi08575397-5 from 2900635 to 3303278 tid 3 2021-02-10T04:28:56.077 INFO:tasks.rados.rados.0.smithi085.stdout:165: read oid 22 snap -1 2021-02-10T04:28:56.077 INFO:tasks.rados.rados.0.smithi085.stdout:165: expect (ObjNum 85 snap 11 seq_num 85) 2021-02-10T04:28:56.077 INFO:tasks.rados.rados.0.smithi085.stdout:166: snap_remove snap 7 2021-02-10T04:28:56.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:56 smithi164 conmon[29288]: cluster 2021-02-10T04:28:55.049767+0000 mon.a (mon. 2021-02-10T04:28:56.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:56 smithi164 conmon[29288]: 0) 784 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2021-02-10T04:28:56.402 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set nodeep-scrub 2021-02-10T04:28:56.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:56 smithi085 conmon[29319]: cluster 2021-02-10T04:28:55.049767+0000 mon.a (mon. 2021-02-10T04:28:56.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:56 smithi085 conmon[29319]: 0) 784 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2021-02-10T04:28:56.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:56 smithi085 conmon[35961]: cluster 2021-02-10T04:28:56.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:56 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:28:56.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:56 smithi085 conmon[35961]: 28:55.049767+0000 mon.a (mon.0) 784 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2021-02-10T04:28:56.669 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:28:57.070 INFO:tasks.rados.rados.0.smithi085.stdout:166: done (2 left) 2021-02-10T04:28:57.071 INFO:tasks.rados.rados.0.smithi085.stdout:167: copy_from oid 47 from oid 43 current snap is 13 2021-02-10T04:28:57.071 INFO:tasks.rados.rados.0.smithi085.stdout:164: finishing write tid 1 to smithi08575397-5 2021-02-10T04:28:57.071 INFO:tasks.rados.rados.0.smithi085.stdout:164: finishing write tid 2 to smithi08575397-5 2021-02-10T04:28:57.071 INFO:tasks.rados.rados.0.smithi085.stdout:164: finishing write tid 3 to smithi08575397-5 2021-02-10T04:28:57.071 INFO:tasks.rados.rados.0.smithi085.stdout:164: finishing write tid 4 to smithi08575397-5 2021-02-10T04:28:57.071 INFO:tasks.rados.rados.0.smithi085.stdout:164: finishing write tid 5 to smithi08575397-5 2021-02-10T04:28:57.071 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 24 (ObjNum 88 snap 13 seq_num 88) dirty exists 2021-02-10T04:28:57.071 INFO:tasks.rados.rados.0.smithi085.stdout:164: left oid 5 (ObjNum 88 snap 13 seq_num 88) 2021-02-10T04:28:57.071 INFO:tasks.rados.rados.0.smithi085.stdout:165: expect (ObjNum 85 snap 11 seq_num 85) 2021-02-10T04:28:57.252 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:28:57.318 INFO:tasks.rados.rados.0.smithi085.stdout:164: done (2 left) 2021-02-10T04:28:57.319 INFO:tasks.rados.rados.0.smithi085.stdout:165: done (1 left) 2021-02-10T04:28:57.319 INFO:tasks.rados.rados.0.smithi085.stdout:168: snap_create 2021-02-10T04:28:57.319 INFO:tasks.rados.rados.0.smithi085.stdout:167: finishing copy_from to smithi08575397-47 2021-02-10T04:28:57.319 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 47 v 35 (ObjNum 63 snap 4 seq_num 63) dirty exists 2021-02-10T04:28:57.319 INFO:tasks.rados.rados.0.smithi085.stdout:167: finishing copy_from racing read to smithi08575397-47 2021-02-10T04:28:57.373 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:28:57.393 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:28:57 smithi164 conmon[40892]: debug 2021-02-10T04:28:57.064+0000 7f30364c7700 -1 osd.6 pg_epoch: 121 pg[7.d( v 121'26 (0'0,121'26] local-lis/les=93/94 n=5 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [6,1,0] r=0 lpr=93 luod=115'24 crt=121'26 lcod 115'23 mlcod 115'23 active+clean+snaptrim trimq=[10~1] ps=[c~1,11~1,13~1]] removing snap head 2021-02-10T04:28:57.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:57 smithi164 conmon[29288]: cluster 2021-02-10T04:28:55.997190+0000 mgr.y (mgr 2021-02-10T04:28:57.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:57 smithi164 conmon[29288]: .14140) 346 : cluster [DBG] pgmap v328: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 129 MiB data, 337 MiB used, 715 GiB / 715 GiB avail; 4.8 MiB/s rd, 3.1 MiB/s wr, 22 op/s 2021-02-10T04:28:57.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:57 smithi164 conmon[29288]: cluster 2021-02-10T04:28:56.054210+0000 mon.a (mon.0) 785 : cluster 2021-02-10T04:28:57.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:57 smithi164 conmon[29288]: [DBG] osdmap e120: 8 total, 8 up, 8 in 2021-02-10T04:28:57.402 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:28:57 smithi085 conmon[49106]: debug 2021-02-10T04:28:57.214+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:57.402 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:28:57 smithi085 conmon[52836]: debug 2021-02-10T04:28:57.402 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:28:57 smithi085 conmon[52836]: 2021-02-10T04:28:57.064+0000 7fba74fb8700 -1 osd.3 pg_epoch: 121 pg[7.2( v 117'14 (0'0,117'14] local-lis/les=93/94 n=3 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [3,1,2] r=0 lpr=93 crt=117'14 lcod 116'12 mlcod 116'12 active+clean+snaptrim trimq=[10~1] ps=[c~1,11~1,13~1]] removing snap head 2021-02-10T04:28:57.403 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:28:57 smithi085 conmon[52836]: 2021-02-10T04:28:57.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:57 smithi085 conmon[29319]: cluster 2021-02-10T04:28:57.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:57 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:28:57.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:57 smithi085 conmon[29319]: 10T04:28:55.997190+0000 mgr.y (mgr.14140) 346 : cluster [DBG] pgmap v328: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 129 MiB data, 337 MiB used, 715 GiB / 715 GiB avail; 4.8 MiB/s rd, 3.1 MiB/s wr, 22 op/s 2021-02-10T04:28:57.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:57 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:28:57.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:57 smithi085 conmon[29319]: -02-10T04:28:56. 2021-02-10T04:28:57.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:57 smithi085 conmon[29319]: 054210+0000 mon.a (mon.0 2021-02-10T04:28:57.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:57 smithi085 conmon[29319]: ) 785 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2021-02-10T04:28:57.404 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:57 smithi085 conmon[35961]: cluster 2021-02-10T04:28:55.997190 2021-02-10T04:28:57.404 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:57 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 346 : cluster [DBG] pgmap v328: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 129 MiB data, 337 MiB used, 715 GiB / 715 GiB avail; 4.8 MiB/s rd, 3.1 MiB/s wr, 22 op/s 2021-02-10T04:28:57.404 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:57 smithi085 conmon[35961]: cluster 2021-02-10T04:28:57.404 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:57 smithi085 conmon[35961]: 2021-02-10T04:28:56.054210+0000 mon.a ( 2021-02-10T04:28:57.404 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:57 smithi085 conmon[35961]: mon.0 2021-02-10T04:28:57.404 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:57 smithi085 conmon[35961]: ) 785 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2021-02-10T04:28:57.405 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:28:58.068 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:28:57 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:28:58.075 INFO:tasks.rados.rados.0.smithi085.stdout:167: done (1 left) 2021-02-10T04:28:58.075 INFO:tasks.rados.rados.0.smithi085.stdout:168: done (0 left) 2021-02-10T04:28:58.075 INFO:tasks.rados.rados.0.smithi085.stdout:169: write (excl) oid 42 current snap is 14 2021-02-10T04:28:58.075 INFO:tasks.rados.rados.0.smithi085.stdout:169: seq_num 89 ranges {695823=777044,1925015=639682,3328058=167765} 2021-02-10T04:28:58.084 INFO:tasks.rados.rados.0.smithi085.stdout:169: writing smithi08575397-42 from 695823 to 1472867 tid 1 2021-02-10T04:28:58.090 INFO:tasks.rados.rados.0.smithi085.stdout:169: writing smithi08575397-42 from 1925015 to 2564697 tid 2 2021-02-10T04:28:58.097 INFO:tasks.rados.rados.0.smithi085.stdout:169: writing smithi08575397-42 from 3328058 to 3495823 tid 3 2021-02-10T04:28:58.097 INFO:tasks.rados.rados.0.smithi085.stdout:170: copy_from oid 6 from oid 31 current snap is 14 2021-02-10T04:28:58.097 INFO:tasks.rados.rados.0.smithi085.stdout:169: finishing write tid 1 to smithi08575397-42 2021-02-10T04:28:58.097 INFO:tasks.rados.rados.0.smithi085.stdout:169: finishing write tid 2 to smithi08575397-42 2021-02-10T04:28:58.097 INFO:tasks.rados.rados.0.smithi085.stdout:171: snap_remove snap 11 2021-02-10T04:28:58.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:58 smithi085 conmon[29319]: cluster 2021-02-10T04:28:58.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:58 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:28:58.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:58 smithi085 conmon[29319]: 04:28:57.067064+0000 mon.a (mon.0) 786 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2021-02-10T04:28:58.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:58 smithi085 conmon[35961]: cluster 2021-02-10T04:28:57.067064+0000 2021-02-10T04:28:58.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:58 smithi085 conmon[35961]: mon.a (mon.0) 786 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2021-02-10T04:28:58.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:58 smithi164 conmon[29288]: cluster 2021-02-10T04:28:57.067064+0000 mon.a (mon.0) 786 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2021-02-10T04:28:58.477 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:28:58 smithi085 conmon[41540]: debug 2021-02-10T04:28:58.476+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:28:58.710 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:28:59.076 INFO:tasks.rados.rados.0.smithi085.stdout:169: finishing write tid 3 to smithi08575397-42 2021-02-10T04:28:59.076 INFO:tasks.rados.rados.0.smithi085.stdout:169: finishing write tid 4 to smithi08575397-42 2021-02-10T04:28:59.076 INFO:tasks.rados.rados.0.smithi085.stdout:169: finishing write tid 5 to smithi08575397-42 2021-02-10T04:28:59.076 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 30 (ObjNum 89 snap 14 seq_num 89) dirty exists 2021-02-10T04:28:59.076 INFO:tasks.rados.rados.0.smithi085.stdout:169: left oid 42 (ObjNum 89 snap 14 seq_num 89) 2021-02-10T04:28:59.076 INFO:tasks.rados.rados.0.smithi085.stdout:170: finishing copy_from to smithi08575397-6 2021-02-10T04:28:59.076 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 18 (ObjNum 59 snap 4 seq_num 59) dirty exists 2021-02-10T04:28:59.076 INFO:tasks.rados.rados.0.smithi085.stdout:170: finishing copy_from racing read to smithi08575397-6 2021-02-10T04:28:59.076 INFO:tasks.rados.rados.0.smithi085.stdout:169: done (2 left) 2021-02-10T04:28:59.076 INFO:tasks.rados.rados.0.smithi085.stdout:170: done (1 left) 2021-02-10T04:28:59.077 INFO:tasks.rados.rados.0.smithi085.stdout:171: done (0 left) 2021-02-10T04:28:59.077 INFO:tasks.rados.rados.0.smithi085.stdout:172: copy_from oid 8 from oid 46 current snap is 14 2021-02-10T04:28:59.077 INFO:tasks.rados.rados.0.smithi085.stdout:173: delete oid 38 current snap is 14 2021-02-10T04:28:59.084 INFO:tasks.rados.rados.0.smithi085.stdout:173: done (1 left) 2021-02-10T04:28:59.084 INFO:tasks.rados.rados.0.smithi085.stdout:174: rollback oid 18 current snap is 14 2021-02-10T04:28:59.084 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 18 to 13 2021-02-10T04:28:59.084 INFO:tasks.rados.rados.0.smithi085.stdout:175: delete oid 11 current snap is 14 2021-02-10T04:28:59.086 INFO:tasks.rados.rados.0.smithi085.stdout:175: done (2 left) 2021-02-10T04:28:59.086 INFO:tasks.rados.rados.0.smithi085.stdout:176: snap_create 2021-02-10T04:28:59.091 INFO:tasks.rados.rados.0.smithi085.stdout:174: finishing rollback tid 0 to smithi08575397-18 2021-02-10T04:28:59.091 INFO:tasks.rados.rados.0.smithi085.stdout:174: finishing rollback tid 1 to smithi08575397-18 2021-02-10T04:28:59.092 INFO:tasks.rados.rados.0.smithi085.stdout:174: finishing rollback tid 2 to smithi08575397-18 2021-02-10T04:28:59.092 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 18 v 26 (ObjNum 17 snap 0 seq_num 17) dirty exists 2021-02-10T04:28:59.101 INFO:tasks.rados.rados.0.smithi085.stdout:172: finishing copy_from to smithi08575397-8 2021-02-10T04:28:59.101 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 30 (ObjNum 83 snap 10 seq_num 83) dirty exists 2021-02-10T04:28:59.102 INFO:tasks.rados.rados.0.smithi085.stdout:172: finishing copy_from racing read to smithi08575397-8 2021-02-10T04:28:59.393 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:28:59 smithi164 conmon[40892]: debug 2021-02-10T04:28:59.075+0000 7f30314bd700 -1 osd.6 pg_epoch: 122 pg[7.7( v 122'72 (0'0,122'72] local-lis/les=93/94 n=9 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [6,0,3] r=0 lpr=93 luod=121'70 crt=122'72 lcod 117'68 mlcod 117'68 active+clean+snaptrim trimq=[10~1] ps=[c~1,11~1,13~1]] removing snap head 2021-02-10T04:28:59.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:59 smithi164 conmon[29288]: audit 2021-02-10T04:28:57.695047+0000 mgr.y (mgr.14140 2021-02-10T04:28:59.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:59 smithi164 conmon[29288]: ) 347 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:28:59.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:59 smithi164 conmon[29288]: cluster 2021-02-10T04:28:57.997541+0000 mgr.y (mgr.14140) 348 : cluster [DBG] pgmap v331: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 132 MiB data, 351 MiB used, 715 GiB / 715 GiB avail; 6.1 MiB/s rd, 3.8 MiB/s wr, 24 op/s 2021-02-10T04:28:59.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:59 smithi164 conmon[29288]: cluster 2021-02-10T04:28:58.074372+0000 2021-02-10T04:28:59.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:28:59 smithi164 conmon[29288]: mon.a (mon.0) 787 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2021-02-10T04:28:59.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:59 smithi085 conmon[29319]: audit 2021-02-10T04:28:57.695047+0000 mgr.y 2021-02-10T04:28:59.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:59 smithi085 conmon[29319]: (mgr.14140) 347 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:28:59.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:59 smithi085 conmon[29319]: cluster 2021-02-10T04:28:57.997541+0000 mgr.y (mgr.14140) 348 : cluster [DBG] pgmap v331: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 132 MiB data, 351 MiB used, 715 GiB / 715 GiB avail; 6.1 MiB/s rd, 3.8 MiB/s wr, 24 op/s 2021-02-10T04:28:59.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:59 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:28:59.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:28:59 smithi085 conmon[29319]: T04:28:58.074372+0000 mon.a (mon.0) 787 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2021-02-10T04:28:59.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:59 smithi085 conmon[35961]: audit 2021-02-10T04:28 2021-02-10T04:28:59.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:59 smithi085 conmon[35961]: :57.695047+0000 mgr.y (mgr.14140) 347 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:28:59.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:59 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:28:59.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:59 smithi085 conmon[35961]: T04:28:57.997541+0000 mgr.y (mgr. 2021-02-10T04:28:59.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:59 smithi085 conmon[35961]: 14140) 348 : cluster [DBG] pgmap v331: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 132 MiB data, 351 MiB used, 715 GiB / 715 GiB avail; 6.1 MiB/s rd, 3.8 MiB/s wr, 24 op/s 2021-02-10T04:28:59.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:59 smithi085 conmon[35961]: cluster 2021-02-10T04:28:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:59 smithi085 conmon[35961]: 2021-02-10T04:28:58.074372 2021-02-10T04:28:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:59 smithi085 conmon[35961]: +0000 mon.a (mon.0) 787 : cluster [DBG] 2021-02-10T04:28:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:28:59 smithi085 conmon[35961]: osdmap e122: 8 total, 8 up, 8 in 2021-02-10T04:29:00.096 INFO:tasks.rados.rados.0.smithi085.stdout:172: done (2 left) 2021-02-10T04:29:00.281 INFO:tasks.rados.rados.0.smithi085.stdout:174: done (1 left) 2021-02-10T04:29:00.281 INFO:tasks.rados.rados.0.smithi085.stdout:176: done (0 left) 2021-02-10T04:29:00.281 INFO:tasks.rados.rados.0.smithi085.stdout:177: write (excl) oid 12 current snap is 15 2021-02-10T04:29:00.281 INFO:tasks.rados.rados.0.smithi085.stdout:177: seq_num 92 ranges {746244=719144,2100461=503362,3126680=419564} 2021-02-10T04:29:00.282 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:29:00.283 INFO:tasks.rados.rados.0.smithi085.stdout:177: writing smithi08575397-12 from 746244 to 1465388 tid 1 2021-02-10T04:29:00.283 INFO:tasks.rados.rados.0.smithi085.stdout:177: writing smithi08575397-12 from 2100461 to 2603823 tid 2 2021-02-10T04:29:00.283 INFO:tasks.rados.rados.0.smithi085.stdout:177: writing smithi08575397-12 from 3126680 to 3546244 tid 3 2021-02-10T04:29:00.283 INFO:tasks.rados.rados.0.smithi085.stdout:177: finishing write tid 1 to smithi08575397-12 2021-02-10T04:29:00.284 INFO:tasks.rados.rados.0.smithi085.stdout:178: snap_create 2021-02-10T04:29:00.284 INFO:tasks.rados.rados.0.smithi085.stdout:177: finishing write tid 2 to smithi08575397-12 2021-02-10T04:29:00.284 INFO:tasks.rados.rados.0.smithi085.stdout:177: finishing write tid 3 to smithi08575397-12 2021-02-10T04:29:00.284 INFO:tasks.rados.rados.0.smithi085.stdout:177: finishing write tid 4 to smithi08575397-12 2021-02-10T04:29:00.284 INFO:tasks.rados.rados.0.smithi085.stdout:177: finishing write tid 5 to smithi08575397-12 2021-02-10T04:29:00.284 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 37 (ObjNum 92 snap 15 seq_num 92) dirty exists 2021-02-10T04:29:00.284 INFO:tasks.rados.rados.0.smithi085.stdout:177: left oid 12 (ObjNum 92 snap 15 seq_num 92) 2021-02-10T04:29:00.309 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:00.380 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:00 smithi164 conmon[29288]: cluster 2021-02-10T04:28:59.075359+0000 mon.a 2021-02-10T04:29:00.381 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:00 smithi164 conmon[29288]: (mon.0) 788 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2021-02-10T04:29:00.381 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:00 smithi164 conmon[29288]: cluster 2021-02-10T04:28:59.191985+0000 osd.5 (osd.5) 38 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:29:00.381 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:00 smithi164 conmon[29288]: cluster 2021-02-10T04:28:59.192067 2021-02-10T04:29:00.381 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:00 smithi164 conmon[29288]: +0000 osd.5 (osd.5) 39 : cluster 2021-02-10T04:29:00.381 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:00 smithi164 conmon[29288]: [DBG] purged_snaps scrub ok 2021-02-10T04:29:00.406 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:29:00.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:00 smithi085 conmon[29319]: cluster 2021-02-10T04:28:59.075359+0000 mon.a (mon 2021-02-10T04:29:00.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:00 smithi085 conmon[29319]: .0) 788 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2021-02-10T04:29:00.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:00 smithi085 conmon[29319]: cluster 2021-02-10T04:28:59.191985+0000 osd.5 (osd.5) 2021-02-10T04:29:00.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:00 smithi085 conmon[29319]: 38 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:29:00.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:00 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:29:00.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:00 smithi085 conmon[29319]: -10T04:28:59.192067+0000 osd.5 (osd.5) 39 : cluster [DBG] 2021-02-10T04:29:00.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:00 smithi085 conmon[29319]: purged_snaps scrub ok 2021-02-10T04:29:00.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:00 smithi085 conmon[35961]: cluster 2021-02-10T04:28:59.075359+0000 2021-02-10T04:29:00.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:00 smithi085 conmon[35961]: mon.a (mon.0) 788 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2021-02-10T04:29:00.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:00 smithi085 conmon[35961]: cluster 2021-02-10T04:28:59.191985+0000 osd.5 (osd.5) 38 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:29:00.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:00 smithi085 conmon[35961]: cluster 2021-02-10T04:28:59.192067+0000 osd.5 (osd.5) 39 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:29:00.512 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:29:00.546 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:00.643 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:29:00 smithi164 conmon[44544]: debug 2021-02-10T04:29:00.380+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:01.084 INFO:tasks.rados.rados.0.smithi085.stdout:177: done (1 left) 2021-02-10T04:29:01.084 INFO:tasks.rados.rados.0.smithi085.stdout:178: done (0 left) 2021-02-10T04:29:01.084 INFO:tasks.rados.rados.0.smithi085.stdout:179: delete oid 7 current snap is 16 2021-02-10T04:29:01.092 INFO:tasks.rados.rados.0.smithi085.stdout:179: done (0 left) 2021-02-10T04:29:01.092 INFO:tasks.rados.rados.0.smithi085.stdout:180: snap_create 2021-02-10T04:29:01.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:01 smithi164 conmon[29288]: cluster 2021-02-10T04:29:01.672 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:01 smithi164 conmon[29288]: 2021-02-10T04:29:01.672 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:01 smithi164 conmon[29288]: 2021-02-10T04:28:59 2021-02-10T04:29:01.672 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:01 smithi164 conmon[29288]: .997872+0000 mgr.y (mgr.14140) 349 : cluster [DBG] pgmap v334: 168 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 163 active+clean; 135 MiB data, 357 MiB used, 715 GiB / 715 GiB avail; 6.8 MiB/s rd, 2.3 MiB/s wr, 12 op/s 2021-02-10T04:29:01.672 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:01 smithi164 conmon[29288]: cluster 2021-02-10T04:29:01.672 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:01 smithi164 conmon[29288]: 2021-02-10T04:29:00 2021-02-10T04:29:01.672 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:01 smithi164 conmon[29288]: .085756+0000 mon.a (mon.0) 789 : 2021-02-10T04:29:01.672 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:01 smithi164 conmon[29288]: cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2021-02-10T04:29:01.673 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:01 smithi085 conmon[29319]: cluster 2021-02-10T04:28:59.997872+0000 mgr.y (mgr. 2021-02-10T04:29:01.674 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:01 smithi085 conmon[29319]: 14140) 349 : cluster [DBG] pgmap v334: 168 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 163 active+clean; 135 MiB data, 357 MiB used, 715 GiB / 715 GiB avail; 6.8 MiB/s rd, 2.3 MiB/s wr, 12 op/s 2021-02-10T04:29:01.674 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:01 smithi085 conmon[29319]: cluster 2021-02-10T04:29:00.085756+0000 mon.a (mon.0) 789 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2021-02-10T04:29:01.674 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:01 smithi085 conmon[35961]: cluster 2021-02-10T04:28:59.997872+0000 2021-02-10T04:29:01.674 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:01 smithi085 conmon[35961]: mgr.y (mgr.14140) 349 : cluster [DBG] pgmap v334: 168 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 163 active+clean; 135 MiB data, 357 MiB used, 715 GiB / 715 GiB avail; 6.8 MiB/s rd, 2.3 MiB/s wr, 12 op/s 2021-02-10T04:29:01.675 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:01 smithi085 conmon[35961]: cluster 2021-02-10T04:29:00.085756+0000 mon.a (mon.0) 789 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2021-02-10T04:29:01.675 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:29:01.675 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:29:01.696 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:29:01.723 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:29:01.750 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:29:01.779 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:29:01.803 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:29:01.830 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:29:01.832 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:29:01.832 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:29:01.832 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:29:01.832 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:29:01.832 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:29:01.833 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:29:01.833 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:29:01.833 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:29:02.100 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:29:01 smithi085 conmon[52836]: debug 2021-02-10T04:29:01.872+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:02.112 INFO:tasks.rados.rados.0.smithi085.stdout:180: done (0 left) 2021-02-10T04:29:02.112 INFO:tasks.rados.rados.0.smithi085.stdout:181: read oid 32 snap 15 2021-02-10T04:29:02.112 INFO:tasks.rados.rados.0.smithi085.stdout:181: expect (ObjNum 31 snap 0 seq_num 31) 2021-02-10T04:29:02.112 INFO:tasks.rados.rados.0.smithi085.stdout:182: snap_create 2021-02-10T04:29:02.116 INFO:tasks.rados.rados.0.smithi085.stdout:181: expect (ObjNum 31 snap 0 seq_num 31) 2021-02-10T04:29:02.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:02 smithi164 conmon[29288]: cluster 2021-02-10T04:29:01.083941+0000 mon.a ( 2021-02-10T04:29:02.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:02 smithi164 conmon[29288]: mon.0) 790 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2021-02-10T04:29:02.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:02 smithi164 conmon[29288]: audit 2021-02-10T04:29:01.617294+0000 mon.c (mon.1) 40 : audit [DBG] from='client.? 172.21.15.85:0/3255218041' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:29:02.415 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:29:02.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: cluster 2021-02-10T04:29:02.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: 2021-02-10T04:29:02.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: 2021 2021-02-10T04:29:02.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: - 2021-02-10T04:29:02.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: 02 2021-02-10T04:29:02.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: - 2021-02-10T04:29:02.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: 10 2021-02-10T04:29:02.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: T 2021-02-10T04:29:02.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: 04 2021-02-10T04:29:02.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: : 2021-02-10T04:29:02.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: 29 2021-02-10T04:29:02.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: : 2021-02-10T04:29:02.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: 01 2021-02-10T04:29:02.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: . 2021-02-10T04:29:02.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: 083941 2021-02-10T04:29:02.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: +0000 2021-02-10T04:29:02.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: 2021-02-10T04:29:02.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: mon.a 2021-02-10T04:29:02.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: ( 2021-02-10T04:29:02.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: mon 2021-02-10T04:29:02.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: . 2021-02-10T04:29:02.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: 0 2021-02-10T04:29:02.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: ) 2021-02-10T04:29:02.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: 790 2021-02-10T04:29:02.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: : 2021-02-10T04:29:02.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: cluster 2021-02-10T04:29:02.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: 2021-02-10T04:29:02.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: [DBG] 2021-02-10T04:29:02.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: 2021-02-10T04:29:02.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: osdmap e125: 8 total, 8 up, 8 in 2021-02-10T04:29:02.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: 2021-02-10T04:29:02.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: audit 2021-02-10T04:29:02.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: 2021-02-10T04:29:02.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: 2021 2021-02-10T04:29:02.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: - 2021-02-10T04:29:02.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: 02 2021-02-10T04:29:02.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: -10T04:29:01.617294 2021-02-10T04:29:02.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[35961]: +0000 mon.c (mon.1) 40 : audit [DBG] from='client.? 172.21.15.85:0/3255218041' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:29:02.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[29319]: cluster 2021-02-10T04:29:01.083941+0000 mon.a (mon.0) 790 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2021-02-10T04:29:02.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:02 smithi085 conmon[29319]: audit 2021-02-10T04:29:01.617294+0000 mon.c (mon.1) 40 : audit [DBG] from='client.? 172.21.15.85:0/3255218041' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:29:03.041 INFO:tasks.rados.rados.0.smithi085.stdout:181: done (1 left) 2021-02-10T04:29:03.042 INFO:tasks.rados.rados.0.smithi085.stdout:182: done (0 left) 2021-02-10T04:29:03.042 INFO:tasks.rados.rados.0.smithi085.stdout:183: copy_from oid 19 from oid 22 current snap is 18 2021-02-10T04:29:03.042 INFO:tasks.rados.rados.0.smithi085.stdout:184: read oid 21 snap 13 2021-02-10T04:29:03.042 INFO:tasks.rados.rados.0.smithi085.stdout:184: expect (ObjNum 81 snap 10 seq_num 81) 2021-02-10T04:29:03.042 INFO:tasks.rados.rados.0.smithi085.stdout:185: write oid 49 current snap is 18 2021-02-10T04:29:03.042 INFO:tasks.rados.rados.0.smithi085.stdout:185: seq_num 94 ranges {779858=469642,1817425=507031,2781526=586628} 2021-02-10T04:29:03.047 INFO:tasks.rados.rados.0.smithi085.stdout:185: writing smithi08575397-49 from 779858 to 1249500 tid 1 2021-02-10T04:29:03.054 INFO:tasks.rados.rados.0.smithi085.stdout:185: writing smithi08575397-49 from 1817425 to 2324456 tid 2 2021-02-10T04:29:03.062 INFO:tasks.rados.rados.0.smithi085.stdout:185: writing smithi08575397-49 from 2781526 to 3368154 tid 3 2021-02-10T04:29:03.062 INFO:tasks.rados.rados.0.smithi085.stdout:186: read oid 24 snap -1 2021-02-10T04:29:03.062 INFO:tasks.rados.rados.0.smithi085.stdout:186: expect (ObjNum 23 snap 0 seq_num 23) 2021-02-10T04:29:03.062 INFO:tasks.rados.rados.0.smithi085.stdout:187: delete oid 43 current snap is 18 2021-02-10T04:29:03.063 INFO:tasks.rados.rados.0.smithi085.stdout:185: finishing write tid 1 to smithi08575397-49 2021-02-10T04:29:03.064 INFO:tasks.rados.rados.0.smithi085.stdout:184: expect (ObjNum 81 snap 10 seq_num 81) 2021-02-10T04:29:03.299 INFO:tasks.rados.rados.0.smithi085.stdout:184: done (4 left) 2021-02-10T04:29:03.300 INFO:tasks.rados.rados.0.smithi085.stdout:187: done (3 left) 2021-02-10T04:29:03.300 INFO:tasks.rados.rados.0.smithi085.stdout:188: copy_from oid 33 from oid 13 current snap is 18 2021-02-10T04:29:03.301 INFO:tasks.rados.rados.0.smithi085.stdout:185: finishing write tid 2 to smithi08575397-49 2021-02-10T04:29:03.301 INFO:tasks.rados.rados.0.smithi085.stdout:186: expect (ObjNum 23 snap 0 seq_num 23) 2021-02-10T04:29:03.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:03 smithi164 conmon[29288]: cluster 2021-02-10T04:29:01.998375+0000 mgr.y (mgr.14140 2021-02-10T04:29:03.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:03 smithi164 conmon[29288]: ) 350 : cluster [DBG] pgmap v337: 168 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 163 active+clean; 131 MiB data, 355 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.2 MiB/s wr, 8 op/s 2021-02-10T04:29:03.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:03 smithi164 conmon[29288]: cluster 2021-02-10T04:29:02.110969+0000 mon.a (mon.0) 791 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2021-02-10T04:29:03.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:03 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:29:03.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:03 smithi164 conmon[29288]: 04:29:03.037992+0000 mon.a (mon.0) 792 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2021-02-10T04:29:03.400 INFO:tasks.rados.rados.0.smithi085.stdout:185: finishing write tid 3 to smithi08575397-49 2021-02-10T04:29:03.400 INFO:tasks.rados.rados.0.smithi085.stdout:185: finishing write tid 4 to smithi08575397-49 2021-02-10T04:29:03.400 INFO:tasks.rados.rados.0.smithi085.stdout:185: finishing write tid 5 to smithi08575397-49 2021-02-10T04:29:03.401 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 49 v 37 (ObjNum 94 snap 18 seq_num 94) dirty exists 2021-02-10T04:29:03.401 INFO:tasks.rados.rados.0.smithi085.stdout:185: left oid 49 (ObjNum 94 snap 18 seq_num 94) 2021-02-10T04:29:03.401 INFO:tasks.rados.rados.0.smithi085.stdout:183: finishing copy_from to smithi08575397-19 2021-02-10T04:29:03.401 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 19 v 29 (ObjNum 85 snap 11 seq_num 85) dirty exists 2021-02-10T04:29:03.401 INFO:tasks.rados.rados.0.smithi085.stdout:183: finishing copy_from racing read to smithi08575397-19 2021-02-10T04:29:03.401 INFO:tasks.rados.rados.0.smithi085.stdout:183: done (3 left) 2021-02-10T04:29:03.401 INFO:tasks.rados.rados.0.smithi085.stdout:185: done (2 left) 2021-02-10T04:29:03.401 INFO:tasks.rados.rados.0.smithi085.stdout:186: done (1 left) 2021-02-10T04:29:03.401 INFO:tasks.rados.rados.0.smithi085.stdout:189: snap_create 2021-02-10T04:29:03.401 INFO:tasks.rados.rados.0.smithi085.stdout:188: finishing copy_from to smithi08575397-33 2021-02-10T04:29:03.401 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 33 v 17 (ObjNum 62 snap 4 seq_num 62) dirty exists 2021-02-10T04:29:03.401 INFO:tasks.rados.rados.0.smithi085.stdout:188: finishing copy_from racing read to smithi08575397-33 2021-02-10T04:29:03.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:03 smithi085 conmon[29319]: cluster 2021-02-10T04:29:03.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:03 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:29:03.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:03 smithi085 conmon[29319]: :29:01.998375+0000 mgr.y (mgr.14140) 350 : cluster [DBG] pgmap v337: 168 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 163 active+clean; 131 MiB data, 355 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.2 MiB/s wr, 8 op/s 2021-02-10T04:29:03.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:03 smithi085 conmon[29319]: cluster 2021-02-10T04:29:02.110969+0000 mon.a (mon.0) 791 : cluster [DBG] 2021-02-10T04:29:03.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:03 smithi085 conmon[29319]: osdmap e126: 8 total, 8 up, 8 in 2021-02-10T04:29:03.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:03 smithi085 conmon[29319]: cluster 2021-02-10T04:29:03.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:03 smithi085 conmon[29319]: 2021-02-10T04:29:03.037992+0000 mon.a (mon.0 2021-02-10T04:29:03.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:03 smithi085 conmon[29319]: ) 792 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2021-02-10T04:29:03.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:03 smithi085 conmon[35961]: cluster 2021-02-10T04:29 2021-02-10T04:29:03.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:03 smithi085 conmon[35961]: :01.998375+0000 mgr.y (mgr.14140) 350 : cluster [DBG] pgmap v337: 168 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 163 active+clean; 131 MiB data, 355 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.2 MiB/s wr, 8 op/s 2021-02-10T04:29:03.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:03 smithi085 conmon[35961]: cluster 2021-02-10T04:29:03.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:03 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:29:03.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:03 smithi085 conmon[35961]: 10T04:29:02.110969 2021-02-10T04:29:03.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:03 smithi085 conmon[35961]: +0000 mon.a (mon.0 2021-02-10T04:29:03.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:03 smithi085 conmon[35961]: ) 791 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2021-02-10T04:29:03.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:03 smithi085 conmon[35961]: 2021-02-10T04:29:03.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:03 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:29:03.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:03 smithi085 conmon[35961]: T04:29:03.037992+0000 mon.a (mon 2021-02-10T04:29:03.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:03 smithi085 conmon[35961]: .0) 792 : cluster [DBG] 2021-02-10T04:29:03.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:03 smithi085 conmon[35961]: osdmap e127: 8 total, 8 up, 8 in 2021-02-10T04:29:03.613 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:29:03.642 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:03.715 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:29:03.819 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:29:03.845 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:03.893 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:29:03 smithi164 conmon[37263]: debug 2021-02-10T04:29:03.697+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:03.932 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:29:04.036 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:29:04.071 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:04.127 INFO:tasks.rados.rados.0.smithi085.stdout:188: done (1 left) 2021-02-10T04:29:04.127 INFO:tasks.rados.rados.0.smithi085.stdout:189: done (0 left) 2021-02-10T04:29:04.127 INFO:tasks.rados.rados.0.smithi085.stdout:190: copy_from oid 41 from oid 40 current snap is 19 2021-02-10T04:29:04.127 INFO:tasks.rados.rados.0.smithi085.stdout:191: copy_from oid 6 from oid 47 current snap is 19 2021-02-10T04:29:04.127 INFO:tasks.rados.rados.0.smithi085.stdout:192: read oid 3 snap -1 2021-02-10T04:29:04.128 INFO:tasks.rados.rados.0.smithi085.stdout:192: expect (ObjNum 51 snap 0 seq_num 51) 2021-02-10T04:29:04.128 INFO:tasks.rados.rados.0.smithi085.stdout:193: rollback oid 48 current snap is 19 2021-02-10T04:29:04.128 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 48 to 18 2021-02-10T04:29:04.128 INFO:tasks.rados.rados.0.smithi085.stdout:194: write (excl) oid 45 current snap is 19 2021-02-10T04:29:04.128 INFO:tasks.rados.rados.0.smithi085.stdout:194: seq_num 96 ranges {413472=620140,1818036=510700,3120019=493453} 2021-02-10T04:29:04.132 INFO:tasks.rados.rados.0.smithi085.stdout:194: writing smithi08575397-45 from 413472 to 1033612 tid 1 2021-02-10T04:29:04.141 INFO:tasks.rados.rados.0.smithi085.stdout:194: writing smithi08575397-45 from 1818036 to 2328736 tid 2 2021-02-10T04:29:04.149 INFO:tasks.rados.rados.0.smithi085.stdout:194: writing smithi08575397-45 from 3120019 to 3613472 tid 3 2021-02-10T04:29:04.150 INFO:tasks.rados.rados.0.smithi085.stdout:195: read oid 50 snap -1 2021-02-10T04:29:04.150 INFO:tasks.rados.rados.0.smithi085.stdout:195: expect (ObjNum 10 snap 0 seq_num 10) 2021-02-10T04:29:04.150 INFO:tasks.rados.rados.0.smithi085.stdout:196: write (excl) oid 44 current snap is 19 2021-02-10T04:29:04.150 INFO:tasks.rados.rados.0.smithi085.stdout:193: finishing rollback tid 0 to smithi08575397-48 2021-02-10T04:29:04.150 INFO:tasks.rados.rados.0.smithi085.stdout:196: seq_num 97 ranges {430279=695389,1876518=654358,3031089=536797} 2021-02-10T04:29:04.159 INFO:tasks.rados.rados.0.smithi085.stdout:196: writing smithi08575397-44 from 430279 to 1125668 tid 1 2021-02-10T04:29:04.166 INFO:tasks.rados.rados.0.smithi085.stdout:196: writing smithi08575397-44 from 1876518 to 2530876 tid 2 2021-02-10T04:29:04.172 INFO:tasks.rados.rados.0.smithi085.stdout:196: writing smithi08575397-44 from 3031089 to 3567886 tid 3 2021-02-10T04:29:04.172 INFO:tasks.rados.rados.0.smithi085.stdout:193: finishing rollback tid 1 to smithi08575397-48 2021-02-10T04:29:04.172 INFO:tasks.rados.rados.0.smithi085.stdout:193: finishing rollback tid 2 to smithi08575397-48 2021-02-10T04:29:04.172 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 41 (ObjNum 47 snap 0 seq_num 47) dirty exists 2021-02-10T04:29:04.172 INFO:tasks.rados.rados.0.smithi085.stdout:192: expect (ObjNum 51 snap 0 seq_num 51) 2021-02-10T04:29:04.335 INFO:tasks.rados.rados.0.smithi085.stdout:194: finishing write tid 1 to smithi08575397-45 2021-02-10T04:29:04.336 INFO:tasks.rados.rados.0.smithi085.stdout:190: finishing copy_from to smithi08575397-41 2021-02-10T04:29:04.336 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 41 v 15 (ObjNum 39 snap 0 seq_num 39) dirty exists 2021-02-10T04:29:04.336 INFO:tasks.rados.rados.0.smithi085.stdout:194: finishing write tid 2 to smithi08575397-45 2021-02-10T04:29:04.336 INFO:tasks.rados.rados.0.smithi085.stdout:192: done (6 left) 2021-02-10T04:29:04.336 INFO:tasks.rados.rados.0.smithi085.stdout:193: done (5 left) 2021-02-10T04:29:04.336 INFO:tasks.rados.rados.0.smithi085.stdout:197: copy_from oid 30 from oid 4 current snap is 19 2021-02-10T04:29:04.337 INFO:tasks.rados.rados.0.smithi085.stdout:190: finishing copy_from racing read to smithi08575397-41 2021-02-10T04:29:04.337 INFO:tasks.rados.rados.0.smithi085.stdout:195: expect (ObjNum 10 snap 0 seq_num 10) 2021-02-10T04:29:04.393 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:29:03 smithi164 conmon[44544]: debug 2021-02-10T04:29:03.905+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:04.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:04 smithi164 conmon[29288]: cluster 2021-02-10T04:29:03.998876+0000 2021-02-10T04:29:04.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:04 smithi164 conmon[29288]: mgr.y (mgr.14140) 351 : cluster [DBG] pgmap v340: 168 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 164 active+clean; 135 MiB data, 361 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 MiB/s wr, 5 op/s 2021-02-10T04:29:04.412 INFO:tasks.rados.rados.0.smithi085.stdout:194: finishing write tid 3 to smithi08575397-45 2021-02-10T04:29:04.412 INFO:tasks.rados.rados.0.smithi085.stdout:194: finishing write tid 4 to smithi08575397-45 2021-02-10T04:29:04.412 INFO:tasks.rados.rados.0.smithi085.stdout:194: finishing write tid 5 to smithi08575397-45 2021-02-10T04:29:04.412 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 45 v 36 (ObjNum 96 snap 19 seq_num 96) dirty exists 2021-02-10T04:29:04.412 INFO:tasks.rados.rados.0.smithi085.stdout:194: left oid 45 (ObjNum 96 snap 19 seq_num 96) 2021-02-10T04:29:04.412 INFO:tasks.rados.rados.0.smithi085.stdout:196: finishing write tid 1 to smithi08575397-44 2021-02-10T04:29:04.414 INFO:tasks.rados.rados.0.smithi085.stdout:196: finishing write tid 2 to smithi08575397-44 2021-02-10T04:29:04.414 INFO:tasks.rados.rados.0.smithi085.stdout:196: finishing write tid 3 to smithi08575397-44 2021-02-10T04:29:04.414 INFO:tasks.rados.rados.0.smithi085.stdout:190: done (5 left) 2021-02-10T04:29:04.414 INFO:tasks.rados.rados.0.smithi085.stdout:194: done (4 left) 2021-02-10T04:29:04.414 INFO:tasks.rados.rados.0.smithi085.stdout:195: done (3 left) 2021-02-10T04:29:04.414 INFO:tasks.rados.rados.0.smithi085.stdout:198: write (excl) oid 20 current snap is 19 2021-02-10T04:29:04.414 INFO:tasks.rados.rados.0.smithi085.stdout:196: finishing write tid 4 to smithi08575397-44 2021-02-10T04:29:04.414 INFO:tasks.rados.rados.0.smithi085.stdout:196: finishing write tid 5 to smithi08575397-44 2021-02-10T04:29:04.414 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 21 (ObjNum 97 snap 19 seq_num 97) dirty exists 2021-02-10T04:29:04.414 INFO:tasks.rados.rados.0.smithi085.stdout:196: left oid 44 (ObjNum 97 snap 19 seq_num 97) 2021-02-10T04:29:04.414 INFO:tasks.rados.rados.0.smithi085.stdout:198: seq_num 98 ranges {447086=770638,1935000=798016,3458512=188574} 2021-02-10T04:29:04.418 INFO:tasks.rados.rados.0.smithi085.stdout:198: writing smithi08575397-20 from 447086 to 1217724 tid 1 2021-02-10T04:29:04.427 INFO:tasks.rados.rados.0.smithi085.stdout:198: writing smithi08575397-20 from 1935000 to 2733016 tid 2 2021-02-10T04:29:04.432 INFO:tasks.rados.rados.0.smithi085.stdout:198: writing smithi08575397-20 from 3458512 to 3647086 tid 3 2021-02-10T04:29:04.432 INFO:tasks.rados.rados.0.smithi085.stdout:196: done (3 left) 2021-02-10T04:29:04.432 INFO:tasks.rados.rados.0.smithi085.stdout:199: snap_create 2021-02-10T04:29:04.432 INFO:tasks.rados.rados.0.smithi085.stdout:197: finishing copy_from to smithi08575397-30 2021-02-10T04:29:04.432 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2021-02-10T04:29:04.432 INFO:tasks.rados.rados.0.smithi085.stdout:197: finishing copy_from racing read to smithi08575397-30 2021-02-10T04:29:04.432 INFO:tasks.rados.rados.0.smithi085.stdout:198: finishing write tid 1 to smithi08575397-20 2021-02-10T04:29:04.433 INFO:tasks.rados.rados.0.smithi085.stdout:198: finishing write tid 2 to smithi08575397-20 2021-02-10T04:29:04.434 INFO:tasks.rados.rados.0.smithi085.stdout:198: finishing write tid 3 to smithi08575397-20 2021-02-10T04:29:04.434 INFO:tasks.rados.rados.0.smithi085.stdout:198: finishing write tid 4 to smithi08575397-20 2021-02-10T04:29:04.434 INFO:tasks.rados.rados.0.smithi085.stdout:198: finishing write tid 5 to smithi08575397-20 2021-02-10T04:29:04.434 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 20 v 26 (ObjNum 98 snap 19 seq_num 98) dirty exists 2021-02-10T04:29:04.435 INFO:tasks.rados.rados.0.smithi085.stdout:198: left oid 20 (ObjNum 98 snap 19 seq_num 98) 2021-02-10T04:29:04.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:04 smithi085 conmon[29319]: cluster 2021-02-10T04:29:03.998876+0000 2021-02-10T04:29:04.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:04 smithi085 conmon[29319]: mgr.y (mgr.14140) 351 : cluster [DBG] pgmap v340: 168 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 164 active+clean; 135 MiB data, 361 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 MiB/s wr, 5 op/s 2021-02-10T04:29:04.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:04 smithi085 conmon[35961]: cluster 2021-02-10T04:29: 2021-02-10T04:29:04.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:04 smithi085 conmon[35961]: 03.998876+0000 mgr.y (mgr.14140) 351 : cluster [DBG] pgmap v340: 168 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 164 active+clean; 135 MiB data, 361 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 MiB/s wr, 5 op/s 2021-02-10T04:29:05.129 INFO:tasks.rados.rados.0.smithi085.stdout:197: done (3 left) 2021-02-10T04:29:05.129 INFO:tasks.rados.rados.0.smithi085.stdout:198: done (2 left) 2021-02-10T04:29:05.129 INFO:tasks.rados.rados.0.smithi085.stdout:199: done (1 left) 2021-02-10T04:29:05.130 INFO:tasks.rados.rados.0.smithi085.stdout:200: write oid 18 current snap is 20 2021-02-10T04:29:05.130 INFO:tasks.rados.rados.0.smithi085.stdout:200: seq_num 99 ranges {463893=562240,1593482=658027,2685935=559341} 2021-02-10T04:29:05.133 INFO:tasks.rados.rados.0.smithi085.stdout:200: writing smithi08575397-18 from 463893 to 1026133 tid 1 2021-02-10T04:29:05.140 INFO:tasks.rados.rados.0.smithi085.stdout:200: writing smithi08575397-18 from 1593482 to 2251509 tid 2 2021-02-10T04:29:05.147 INFO:tasks.rados.rados.0.smithi085.stdout:200: writing smithi08575397-18 from 2685935 to 3245276 tid 3 2021-02-10T04:29:05.147 INFO:tasks.rados.rados.0.smithi085.stdout:201: write (excl) oid 12 current snap is 20 2021-02-10T04:29:05.147 INFO:tasks.rados.rados.0.smithi085.stdout:201: seq_num 100 ranges {480700=637489,1651964=401685,2713358=770613} 2021-02-10T04:29:05.154 INFO:tasks.rados.rados.0.smithi085.stdout:201: writing smithi08575397-12 from 480700 to 1118189 tid 1 2021-02-10T04:29:05.160 INFO:tasks.rados.rados.0.smithi085.stdout:201: writing smithi08575397-12 from 1651964 to 2053649 tid 2 2021-02-10T04:29:05.169 INFO:tasks.rados.rados.0.smithi085.stdout:201: writing smithi08575397-12 from 2713358 to 3483971 tid 3 2021-02-10T04:29:05.169 INFO:tasks.rados.rados.0.smithi085.stdout:202: write (excl) oid 39 current snap is 20 2021-02-10T04:29:05.169 INFO:tasks.rados.rados.0.smithi085.stdout:202: seq_num 101 ranges {497507=712738,1710446=661696,3140781=556726} 2021-02-10T04:29:05.176 INFO:tasks.rados.rados.0.smithi085.stdout:202: writing smithi08575397-39 from 497507 to 1210245 tid 1 2021-02-10T04:29:05.183 INFO:tasks.rados.rados.0.smithi085.stdout:202: writing smithi08575397-39 from 1710446 to 2372142 tid 2 2021-02-10T04:29:05.192 INFO:tasks.rados.rados.0.smithi085.stdout:202: writing smithi08575397-39 from 3140781 to 3697507 tid 3 2021-02-10T04:29:05.192 INFO:tasks.rados.rados.0.smithi085.stdout:203: delete oid 36 current snap is 20 2021-02-10T04:29:05.192 INFO:tasks.rados.rados.0.smithi085.stdout:200: finishing write tid 1 to smithi08575397-18 2021-02-10T04:29:05.192 INFO:tasks.rados.rados.0.smithi085.stdout:200: finishing write tid 2 to smithi08575397-18 2021-02-10T04:29:05.192 INFO:tasks.rados.rados.0.smithi085.stdout:200: finishing write tid 3 to smithi08575397-18 2021-02-10T04:29:05.192 INFO:tasks.rados.rados.0.smithi085.stdout:200: finishing write tid 4 to smithi08575397-18 2021-02-10T04:29:05.192 INFO:tasks.rados.rados.0.smithi085.stdout:200: finishing write tid 5 to smithi08575397-18 2021-02-10T04:29:05.192 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 18 v 31 (ObjNum 99 snap 20 seq_num 99) dirty exists 2021-02-10T04:29:05.192 INFO:tasks.rados.rados.0.smithi085.stdout:200: left oid 18 (ObjNum 99 snap 20 seq_num 99) 2021-02-10T04:29:05.192 INFO:tasks.rados.rados.0.smithi085.stdout:201: finishing write tid 1 to smithi08575397-12 2021-02-10T04:29:05.192 INFO:tasks.rados.rados.0.smithi085.stdout:201: finishing write tid 2 to smithi08575397-12 2021-02-10T04:29:05.193 INFO:tasks.rados.rados.0.smithi085.stdout:191: finishing copy_from to smithi08575397-6 2021-02-10T04:29:05.193 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 35 (ObjNum 63 snap 4 seq_num 63) dirty exists 2021-02-10T04:29:05.193 INFO:tasks.rados.rados.0.smithi085.stdout:191: finishing copy_from racing read to smithi08575397-6 2021-02-10T04:29:05.193 INFO:tasks.rados.rados.0.smithi085.stdout:201: finishing write tid 3 to smithi08575397-12 2021-02-10T04:29:05.193 INFO:tasks.rados.rados.0.smithi085.stdout:201: finishing write tid 4 to smithi08575397-12 2021-02-10T04:29:05.193 INFO:tasks.rados.rados.0.smithi085.stdout:201: finishing write tid 5 to smithi08575397-12 2021-02-10T04:29:05.193 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 46 (ObjNum 100 snap 20 seq_num 100) dirty exists 2021-02-10T04:29:05.193 INFO:tasks.rados.rados.0.smithi085.stdout:201: left oid 12 (ObjNum 100 snap 20 seq_num 100) 2021-02-10T04:29:05.193 INFO:tasks.rados.rados.0.smithi085.stdout:202: finishing write tid 1 to smithi08575397-39 2021-02-10T04:29:05.193 INFO:tasks.rados.rados.0.smithi085.stdout:202: finishing write tid 2 to smithi08575397-39 2021-02-10T04:29:05.197 INFO:tasks.rados.rados.0.smithi085.stdout:202: finishing write tid 3 to smithi08575397-39 2021-02-10T04:29:05.198 INFO:tasks.rados.rados.0.smithi085.stdout:202: finishing write tid 4 to smithi08575397-39 2021-02-10T04:29:05.198 INFO:tasks.rados.rados.0.smithi085.stdout:202: finishing write tid 5 to smithi08575397-39 2021-02-10T04:29:05.198 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 39 v 16 (ObjNum 101 snap 20 seq_num 101) dirty exists 2021-02-10T04:29:05.198 INFO:tasks.rados.rados.0.smithi085.stdout:202: left oid 39 (ObjNum 101 snap 20 seq_num 101) 2021-02-10T04:29:05.198 INFO:tasks.rados.rados.0.smithi085.stdout:191: done (4 left) 2021-02-10T04:29:05.198 INFO:tasks.rados.rados.0.smithi085.stdout:200: done (3 left) 2021-02-10T04:29:05.199 INFO:tasks.rados.rados.0.smithi085.stdout:201: done (2 left) 2021-02-10T04:29:05.199 INFO:tasks.rados.rados.0.smithi085.stdout:202: done (1 left) 2021-02-10T04:29:05.199 INFO:tasks.rados.rados.0.smithi085.stdout:203: done (0 left) 2021-02-10T04:29:05.200 INFO:tasks.rados.rados.0.smithi085.stdout:204: snap_remove snap 14 2021-02-10T04:29:05.239 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:29:05.242 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:05 smithi085 conmon[29319]: cluster 2021-02-10T04:29:04.122398+0000 mon.a ( 2021-02-10T04:29:05.242 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:05 smithi085 conmon[29319]: mon.0) 793 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2021-02-10T04:29:05.242 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:29:05 smithi085 conmon[45443]: debug 2021-02-10T04:29:05.065+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:05.242 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:05 smithi085 conmon[35961]: cluster 2021-02-10T04:29:04.122398+0000 mon.a 2021-02-10T04:29:05.242 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:05 smithi085 conmon[35961]: (mon.0) 793 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2021-02-10T04:29:05.344 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:29:05.375 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:05.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:05 smithi164 conmon[29288]: cluster 2021-02-10T04:29:04.122398+0000 mon.a (mon. 2021-02-10T04:29:05.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:05 smithi164 conmon[29288]: 0) 793 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2021-02-10T04:29:05.549 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:29:05.549 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [], 2021-02-10T04:29:05.549 INFO:teuthology.orchestra.run.smithi085.stdout: "complaint_time": 30, 2021-02-10T04:29:05.549 INFO:teuthology.orchestra.run.smithi085.stdout: "num_blocked_ops": 0 2021-02-10T04:29:05.549 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:29:05.549 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:29:05.651 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:29:05.676 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:05.756 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:29:05.782 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:29:05 smithi085 conmon[52836]: debug 2021-02-10T04:29:05.494+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:05.861 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:29:05.886 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:06.132 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:29:05 smithi164 conmon[33677]: debug 2021-02-10T04:29:05.732+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:06.133 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:06 smithi085 conmon[29319]: cluster 2021-02-10T04:29:06.133 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:06 smithi085 conmon[35961]: cluster 2021-02-10T04:29:05.126850+0000 mon.a (mon.0) 794 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2021-02-10T04:29:06.133 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:06 smithi085 conmon[35961]: audit 2021-02-10T04:29:05.413763+0000 mon.c (mon.1) 41 : audit [INF] from='client.? 172.21.15.85:0/830612390' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]: dispatch 2021-02-10T04:29:06.133 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:06 smithi085 conmon[35961]: audit 2021-02-10T04:29:05.414169+0000 mon.a (mon.0) 795 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]: dispatch 2021-02-10T04:29:06.133 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:06 smithi085 conmon[35961]: cluster 2021-02-10T04:29:05.999288+0000 mgr.y (mgr.14140) 352 : cluster [DBG] pgmap v343: 168 pgs: 168 active+clean; 149 MiB data, 393 MiB used, 715 GiB / 715 GiB avail; 8.6 MiB/s rd, 4.6 MiB/s wr, 20 op/s 2021-02-10T04:29:06.139 INFO:tasks.rados.rados.0.smithi085.stdout:204: done (0 left) 2021-02-10T04:29:06.139 INFO:tasks.rados.rados.0.smithi085.stdout:205: snap_remove snap 18 2021-02-10T04:29:06.147 INFO:teuthology.orchestra.run.smithi085.stderr:set osd.7 primary-affinity to 0 (802) 2021-02-10T04:29:06.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:06 smithi164 conmon[29288]: cluster 2021-02-10T04:29:05.126850+0000 mon.a (mon. 2021-02-10T04:29:06.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:06 smithi164 conmon[29288]: 0) 794 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2021-02-10T04:29:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:06 smithi164 conmon[29288]: audit 2021-02-10T04:29:05.413763+0000 mon.c (mon.1) 41 : audit [INF] from='client.? 172.21.15.85:0/830612390' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]: dispatch 2021-02-10T04:29:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:06 smithi164 conmon[29288]: audit 2021-02-10T04:29: 2021-02-10T04:29:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:06 smithi164 conmon[29288]: 05.414169+0000 mon.a (mon.0) 795 : audit [INF] 2021-02-10T04:29:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:06 smithi164 conmon[29288]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]: dispatch 2021-02-10T04:29:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:06 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:29:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:06 smithi164 conmon[29288]: 29:05.999288+0000 mgr.y (mgr.14140) 2021-02-10T04:29:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:06 smithi164 conmon[29288]: 352 : cluster [DBG] pgmap v343: 168 pgs: 168 active+clean; 149 MiB data, 393 MiB used, 715 GiB / 715 GiB avail; 8.6 MiB/s rd, 4.6 MiB/s wr, 20 op/s 2021-02-10T04:29:06.401 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:06 smithi085 conmon[29319]: 2021-02-10T04:29:05. 2021-02-10T04:29:06.401 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:06 smithi085 conmon[29319]: 126850+0000 mon.a (mon.0) 794 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2021-02-10T04:29:06.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:06 smithi085 conmon[29319]: audit 2021-02-10T04:29: 2021-02-10T04:29:06.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:06 smithi085 conmon[29319]: 05.413763+0000 mon.c (mon.1) 41 : audit [INF] 2021-02-10T04:29:06.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:06 smithi085 conmon[29319]: from='client.? 172.21.15.85:0/830612390' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]: dispatch 2021-02-10T04:29:06.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:06 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:29:06.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:06 smithi085 conmon[29319]: :29:05.414169+0000 mon.a (mon.0) 795 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]: dispatch 2021-02-10T04:29:06.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:06 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:29:06.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:06 smithi085 conmon[29319]: 10T04:29:05.999288+0000 mgr.y (mgr.14140) 352 2021-02-10T04:29:06.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:06 smithi085 conmon[29319]: : cluster [DBG] pgmap v343: 168 pgs: 168 active+clean; 149 MiB data, 393 MiB used, 715 GiB / 715 GiB avail; 8.6 MiB/s rd, 4.6 MiB/s wr, 20 op/s 2021-02-10T04:29:06.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:29:06 smithi085 conmon[49106]: debug 2021-02-10T04:29:06.400+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:06.700 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:29:06.923 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:29:06.950 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:07.135 INFO:tasks.rados.rados.0.smithi085.stdout:205: done (0 left) 2021-02-10T04:29:07.135 INFO:tasks.rados.rados.0.smithi085.stdout:206: rollback oid 6 current snap is 20 2021-02-10T04:29:07.135 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 6 to 15 2021-02-10T04:29:07.135 INFO:tasks.rados.rados.0.smithi085.stdout:207: read oid 4 snap -1 2021-02-10T04:29:07.135 INFO:tasks.rados.rados.0.smithi085.stdout:207: expect (ObjNum 8 snap 0 seq_num 8) 2021-02-10T04:29:07.135 INFO:tasks.rados.rados.0.smithi085.stdout:208: write (excl) oid 39 current snap is 20 2021-02-10T04:29:07.136 INFO:tasks.rados.rados.0.smithi085.stdout:208: seq_num 103 ranges {531121=463236,1711057=665365,3079274=651847} 2021-02-10T04:29:07.137 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is set 2021-02-10T04:29:07.140 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:07 smithi085 conmon[29319]: cluster 2021-02-10T04:29:05.479942+0000 osd.2 (osd.2) 33 : cluster [DBG] 2021-02-10T04:29:07.140 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:07 smithi085 conmon[29319]: purged_snaps scrub starts 2021-02-10T04:29:07.140 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:07 smithi085 conmon[29319]: cluster 2021-02-10T04:29:05.480032+0000 osd.2 (osd.2) 34 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:29:07.140 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:07 smithi085 conmon[29319]: audit 2021-02-10T04:29:06.138239+0000 mon.a (mon.0) 796 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]': finished 2021-02-10T04:29:07.140 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:07 smithi085 conmon[29319]: cluster 2021-02-10T04:29:06.138334+0000 mon.a (mon.0) 797 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2021-02-10T04:29:07.140 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:07 smithi085 conmon[29319]: audit 2021-02-10T04:29:06.433296+0000 mon.a (mon.0) 798 : audit [INF] from='client.? 172.21.15.85:0/1911724356' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:29:07.140 INFO:tasks.rados.rados.0.smithi085.stdout:208: writing smithi08575397-39 from 531121 to 994357 tid 1 2021-02-10T04:29:07.150 INFO:tasks.rados.rados.0.smithi085.stdout:208: writing smithi08575397-39 from 1711057 to 2376422 tid 2 2021-02-10T04:29:07.159 INFO:tasks.rados.rados.0.smithi085.stdout:208: writing smithi08575397-39 from 3079274 to 3731121 tid 3 2021-02-10T04:29:07.159 INFO:tasks.rados.rados.0.smithi085.stdout:209: copy_from oid 35 from oid 22 current snap is 20 2021-02-10T04:29:07.159 INFO:tasks.rados.rados.0.smithi085.stdout:208: finishing write tid 1 to smithi08575397-39 2021-02-10T04:29:07.159 INFO:tasks.rados.rados.0.smithi085.stdout:208: finishing write tid 2 to smithi08575397-39 2021-02-10T04:29:07.160 INFO:tasks.rados.rados.0.smithi085.stdout:210: copy_from oid 5 from oid 15 current snap is 20 2021-02-10T04:29:07.160 INFO:tasks.rados.rados.0.smithi085.stdout:211: snap_create 2021-02-10T04:29:07.163 INFO:tasks.rados.rados.0.smithi085.stdout:207: expect (ObjNum 8 snap 0 seq_num 8) 2021-02-10T04:29:07.198 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:29:07.241 INFO:tasks.rados.rados.0.smithi085.stdout:206: finishing rollback tid 0 to smithi08575397-6 2021-02-10T04:29:07.242 INFO:tasks.rados.rados.0.smithi085.stdout:206: finishing rollback tid 1 to smithi08575397-6 2021-02-10T04:29:07.242 INFO:tasks.rados.rados.0.smithi085.stdout:206: finishing rollback tid 2 to smithi08575397-6 2021-02-10T04:29:07.242 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 52 (ObjNum 59 snap 4 seq_num 59) dirty exists 2021-02-10T04:29:07.242 INFO:tasks.rados.rados.0.smithi085.stdout:208: finishing write tid 3 to smithi08575397-39 2021-02-10T04:29:07.242 INFO:tasks.rados.rados.0.smithi085.stdout:208: finishing write tid 4 to smithi08575397-39 2021-02-10T04:29:07.242 INFO:tasks.rados.rados.0.smithi085.stdout:208: finishing write tid 5 to smithi08575397-39 2021-02-10T04:29:07.242 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 39 v 22 (ObjNum 103 snap 20 seq_num 103) dirty exists 2021-02-10T04:29:07.242 INFO:tasks.rados.rados.0.smithi085.stdout:208: left oid 39 (ObjNum 103 snap 20 seq_num 103) 2021-02-10T04:29:07.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:07 smithi164 conmon[29288]: cluster 2021-02-10T04:29:05.479942+0000 osd.2 2021-02-10T04:29:07.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:07 smithi164 conmon[29288]: (osd.2) 33 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:29:07.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:07 smithi164 conmon[29288]: cluster 2021-02-10T04:29:05.480032+0000 osd.2 (osd.2) 34 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:29:07.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:07 smithi164 conmon[29288]: audit 2021-02-10T04:29:06.138239+0000 mon.a (mon.0) 2021-02-10T04:29:07.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:07 smithi164 conmon[29288]: 796 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]': finished 2021-02-10T04:29:07.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:07 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:29:07.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:07 smithi164 conmon[29288]: T04:29:06.138334+0000 mon.a (mon. 2021-02-10T04:29:07.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:07 smithi164 conmon[29288]: 0) 797 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2021-02-10T04:29:07.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:07 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:29:07.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:07 smithi164 conmon[29288]: 10T04:29:06.433296+0000 mon.a (mon 2021-02-10T04:29:07.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:07 smithi164 conmon[29288]: .0) 798 : audit [INF] from='client.? 172.21.15.85:0/1911724356' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:29:07.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:07 smithi085 conmon[35961]: cluster 2021-02-10T04:29:07.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:07 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:29:07.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:07 smithi085 conmon[35961]: 29:05.479942+0000 osd.2 (osd.2) 33 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:29:07.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:07 smithi085 conmon[35961]: cluster 2021-02-10T04:29: 2021-02-10T04:29:07.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:07 smithi085 conmon[35961]: 05.480032+0000 osd.2 (osd.2) 34 : cluster 2021-02-10T04:29:07.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:07 smithi085 conmon[35961]: [DBG] purged_snaps scrub ok 2021-02-10T04:29:07.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:07 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:29:07.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:07 smithi085 conmon[35961]: :29:06.138239+0000 mon.a (mon.0 2021-02-10T04:29:07.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:07 smithi085 conmon[35961]: ) 796 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]': finished 2021-02-10T04:29:07.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:07 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:29:07.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:07 smithi085 conmon[35961]: -02-10T04:29:06 2021-02-10T04:29:07.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:07 smithi085 conmon[35961]: .138334+0000 mon.a (mon.0) 797 : cluster 2021-02-10T04:29:07.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:07 smithi085 conmon[35961]: [DBG] osdmap e130: 8 total, 8 up, 8 in 2021-02-10T04:29:07.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:07 smithi085 conmon[35961]: audit 2021 2021-02-10T04:29:07.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:07 smithi085 conmon[35961]: -02-10T04:29: 2021-02-10T04:29:07.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:07 smithi085 conmon[35961]: 06.433296+0000 mon.a (mon 2021-02-10T04:29:07.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:07 smithi085 conmon[35961]: .0) 798 : audit [INF] 2021-02-10T04:29:07.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:07 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/1911724356' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:29:07.443 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:29:08.004 INFO:tasks.rados.rados.0.smithi085.stdout:209: finishing copy_from to smithi08575397-35 2021-02-10T04:29:08.004 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 35 v 29 (ObjNum 85 snap 11 seq_num 85) dirty exists 2021-02-10T04:29:08.004 INFO:tasks.rados.rados.0.smithi085.stdout:209: finishing copy_from racing read to smithi08575397-35 2021-02-10T04:29:08.034 INFO:tasks.rados.rados.0.smithi085.stdout:206: done (5 left) 2021-02-10T04:29:08.034 INFO:tasks.rados.rados.0.smithi085.stdout:207: done (4 left) 2021-02-10T04:29:08.034 INFO:tasks.rados.rados.0.smithi085.stdout:208: done (3 left) 2021-02-10T04:29:08.034 INFO:tasks.rados.rados.0.smithi085.stdout:209: done (2 left) 2021-02-10T04:29:08.034 INFO:tasks.rados.rados.0.smithi085.stdout:211: done (1 left) 2021-02-10T04:29:08.035 INFO:tasks.rados.rados.0.smithi085.stdout:212: write oid 13 current snap is 21 2021-02-10T04:29:08.035 INFO:tasks.rados.rados.0.smithi085.stdout:212: seq_num 104 ranges {547928=538485,1769539=409023,2706697=532054} 2021-02-10T04:29:08.041 INFO:tasks.rados.rados.0.smithi085.stdout:212: writing smithi08575397-13 from 547928 to 1086413 tid 1 2021-02-10T04:29:08.048 INFO:tasks.rados.rados.0.smithi085.stdout:212: writing smithi08575397-13 from 1769539 to 2178562 tid 2 2021-02-10T04:29:08.056 INFO:tasks.rados.rados.0.smithi085.stdout:212: writing smithi08575397-13 from 2706697 to 3238751 tid 3 2021-02-10T04:29:08.056 INFO:tasks.rados.rados.0.smithi085.stdout:213: read oid 2 snap -1 2021-02-10T04:29:08.056 INFO:tasks.rados.rados.0.smithi085.stdout:213: expect deleted 2021-02-10T04:29:08.056 INFO:tasks.rados.rados.0.smithi085.stdout:214: rollback oid 16 current snap is 21 2021-02-10T04:29:08.056 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 16 to 10 2021-02-10T04:29:08.056 INFO:tasks.rados.rados.0.smithi085.stdout:215: snap_create 2021-02-10T04:29:08.056 INFO:tasks.rados.rados.0.smithi085.stdout:212: finishing write tid 1 to smithi08575397-13 2021-02-10T04:29:08.056 INFO:tasks.rados.rados.0.smithi085.stdout:212: finishing write tid 2 to smithi08575397-13 2021-02-10T04:29:08.068 INFO:tasks.rados.rados.0.smithi085.stdout:214: finishing rollback tid 0 to smithi08575397-16 2021-02-10T04:29:08.092 INFO:tasks.rados.rados.0.smithi085.stdout:212: finishing write tid 3 to smithi08575397-13 2021-02-10T04:29:08.092 INFO:tasks.rados.rados.0.smithi085.stdout:214: finishing rollback tid 1 to smithi08575397-16 2021-02-10T04:29:08.092 INFO:tasks.rados.rados.0.smithi085.stdout:212: finishing write tid 4 to smithi08575397-13 2021-02-10T04:29:08.092 INFO:tasks.rados.rados.0.smithi085.stdout:214: finishing rollback tid 2 to smithi08575397-16 2021-02-10T04:29:08.092 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 83 (ObjNum 63 snap 4 seq_num 63) dirty exists 2021-02-10T04:29:08.092 INFO:tasks.rados.rados.0.smithi085.stdout:212: finishing write tid 5 to smithi08575397-13 2021-02-10T04:29:08.093 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 33 (ObjNum 104 snap 21 seq_num 104) dirty exists 2021-02-10T04:29:08.093 INFO:tasks.rados.rados.0.smithi085.stdout:212: left oid 13 (ObjNum 104 snap 21 seq_num 104) 2021-02-10T04:29:08.141 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:29:07 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:29:08.225 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.2 dump_historic_ops 2021-02-10T04:29:08.305 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:29:08.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:08 smithi164 conmon[29288]: cluster 2021-02-10T04:29:07.131187+0000 mon.a (mon.0) 799 : cluster 2021-02-10T04:29:08.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:08 smithi164 conmon[29288]: [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:29:08.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:08 smithi164 conmon[29288]: audit 2021-02-10T04:29:07.134913+0000 mon.a (mon.0) 800 : audit [INF] from='client.? 172.21.15.85:0/1911724356' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:29:08.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:08 smithi164 conmon[29288]: cluster 2021-02-10T04:29:07.134951+0000 2021-02-10T04:29:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:08 smithi164 conmon[29288]: mon.a (mon.0) 801 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2021-02-10T04:29:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:08 smithi164 conmon[29288]: audit 2021-02-10T04:29: 2021-02-10T04:29:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:08 smithi164 conmon[29288]: 07.705027+0000 mgr.y (mgr.14140) 353 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:29:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:08 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:29:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:08 smithi164 conmon[29288]: 02-10T04:29:07.999814+0000 2021-02-10T04:29:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:08 smithi164 conmon[29288]: mgr.y (mgr.14140) 354 : cluster [DBG] pgmap v346: 168 pgs: 1 active+clean+snaptrim, 4 peering, 163 active+clean; 154 MiB data, 416 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 5.0 MiB/s wr, 20 op/s 2021-02-10T04:29:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:08 smithi164 conmon[29288]: 2021-02-10T04:29:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:08 smithi164 conmon[29288]: cluster 2021-02-10T04:29:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:08 smithi164 conmon[29288]: 2021-02-10T04:29:08.033836+0000 mon.a ( 2021-02-10T04:29:08.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:08 smithi164 conmon[29288]: mon.0) 802 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2021-02-10T04:29:08.410 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:29:08.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:08 smithi085 conmon[35961]: cluster 2021-02-10T04:29:07.131187+0000 mon.a (mon.0) 799 : 2021-02-10T04:29:08.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:08 smithi085 conmon[35961]: cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:29:08.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:08 smithi085 conmon[35961]: audit 2021-02-10T04:29:07.134913+0000 mon.a (mon.0) 800 : audit [INF] from='client.? 172.21.15.85:0/1911724356' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:29:08.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:08 smithi085 conmon[35961]: cluster 2021-02-10T04:29:07.134951+0000 mon.a (mon.0) 801 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2021-02-10T04:29:08.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:08 smithi085 conmon[35961]: audit 2021-02-10T04:29:07 2021-02-10T04:29:08.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:08 smithi085 conmon[35961]: .705027+0000 mgr.y (mgr.14140) 353 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:29:08.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:08 smithi085 conmon[35961]: cluster 2021-02-10T04:29:07.999814+0000 2021-02-10T04:29:08.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:08 smithi085 conmon[35961]: mgr.y (mgr.14140) 354 : cluster [DBG] pgmap v346: 168 pgs: 1 active+clean+snaptrim, 4 peering, 163 active+clean; 154 MiB data, 416 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 5.0 MiB/s wr, 20 op/s 2021-02-10T04:29:08.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:08 smithi085 conmon[35961]: cluster 2021-02-10T04:29:08.033836+0000 mon.a ( 2021-02-10T04:29:08.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:08 smithi085 conmon[35961]: mon.0) 802 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2021-02-10T04:29:08.416 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:29:08 smithi085 conmon[45443]: debug 2021-02-10T04:29:08.416 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:29:08 smithi085 conmon[45443]: 2021-02-10T04:29:08.266+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:08.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:08 smithi085 conmon[29319]: cluster 2021-02-10T04:29:07.131187+0000 mon.a (mon.0) 799 : cluster 2021-02-10T04:29:08.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:08 smithi085 conmon[29319]: [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:29:08.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:08 smithi085 conmon[29319]: audit 2021-02-10T04:29:07.134913+0000 mon.a (mon.0) 800 : audit [INF] from='client.? 172.21.15.85:0/1911724356' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:29:08.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:08 smithi085 conmon[29319]: cluster 2021-02-10T04:29:07.134951+0000 mon.a (mon.0) 2021-02-10T04:29:08.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:08 smithi085 conmon[29319]: 801 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2021-02-10T04:29:08.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:08 smithi085 conmon[29319]: audit 2021-02-10T04:29:07.705027 2021-02-10T04:29:08.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:08 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 353 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:29:08.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:08 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:29:08.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:08 smithi085 conmon[29319]: :29:07.999814+0000 mgr.y (mgr.14140 2021-02-10T04:29:08.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:08 smithi085 conmon[29319]: ) 354 : cluster [DBG] pgmap v346: 168 pgs: 1 active+clean+snaptrim, 4 peering, 163 active+clean; 154 MiB data, 416 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 5.0 MiB/s wr, 20 op/s 2021-02-10T04:29:08.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:08 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:29:08.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:08 smithi085 conmon[29319]: :29:08.033836+0000 mon.a (mon. 2021-02-10T04:29:08.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:08 smithi085 conmon[29319]: 0) 802 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2021-02-10T04:29:08.437 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:08.547 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:29:08.562 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:29:08.652 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:29:08.677 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:08.744 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:29:08 smithi164 conmon[40892]: debug 2021-02-10T04:29:08.514+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:08.745 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:29:08 smithi164 conmon[33677]: debug 2021-02-10T04:29:08.744+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:08.772 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:29:08.857 INFO:tasks.rados.rados.0.smithi085.stdout:210: finishing copy_from to smithi08575397-5 2021-02-10T04:29:08.857 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 13 (ObjNum 50 snap 0 seq_num 50) dirty exists 2021-02-10T04:29:08.858 INFO:tasks.rados.rados.0.smithi085.stdout:210: finishing copy_from racing read to smithi08575397-5 2021-02-10T04:29:08.876 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:29:08.903 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:08.980 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:29:09.040 INFO:tasks.rados.rados.0.smithi085.stdout:210: done (4 left) 2021-02-10T04:29:09.041 INFO:tasks.rados.rados.0.smithi085.stdout:212: done (3 left) 2021-02-10T04:29:09.041 INFO:tasks.rados.rados.0.smithi085.stdout:213: done (2 left) 2021-02-10T04:29:09.041 INFO:tasks.rados.rados.0.smithi085.stdout:214: done (1 left) 2021-02-10T04:29:09.041 INFO:tasks.rados.rados.0.smithi085.stdout:215: done (0 left) 2021-02-10T04:29:09.041 INFO:tasks.rados.rados.0.smithi085.stdout:216: snap_remove snap 20 2021-02-10T04:29:09.084 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:29:09.116 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:09.142 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:29:08 smithi164 conmon[37263]: debug 2021-02-10T04:29:08.965+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:09.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:09 smithi164 conmon[29288]: cluster 2021-02-10T04:29:08.138421+0000 mon.a (mon. 2021-02-10T04:29:09.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:09 smithi164 conmon[29288]: 0) 803 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2021-02-10T04:29:09.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:09 smithi164 conmon[29288]: cluster 2021-02-10T04:29:09.040479+0000 mon.a (mon.0) 804 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2021-02-10T04:29:09.395 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:09 smithi085 conmon[29319]: cluster 2021-02-10T04:29:08.138421+0000 mon.a (mon.0) 803 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2021-02-10T04:29:09.395 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:09 smithi085 conmon[29319]: cluster 2021-02-10T04:29:09.040479+0000 mon.a (mon.0) 804 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2021-02-10T04:29:09.396 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:09 smithi085 conmon[35961]: cluster 2021-02-10T04:29:08.138421+0000 mon.a (mon.0) 803 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2021-02-10T04:29:09.396 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:09 smithi085 conmon[35961]: cluster 2021-02-10T04:29:09.040479+0000 mon.a (mon.0) 804 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2021-02-10T04:29:10.046 INFO:tasks.rados.rados.0.smithi085.stdout:216: done (0 left) 2021-02-10T04:29:10.047 INFO:tasks.rados.rados.0.smithi085.stdout:217: write oid 20 current snap is 22 2021-02-10T04:29:10.047 INFO:tasks.rados.rados.0.smithi085.stdout:217: seq_num 105 ranges {564735=613734,1828021=669034,3134120=630615} 2021-02-10T04:29:10.051 INFO:tasks.rados.rados.0.smithi085.stdout:217: writing smithi08575397-20 from 564735 to 1178469 tid 1 2021-02-10T04:29:10.058 INFO:tasks.rados.rados.0.smithi085.stdout:217: writing smithi08575397-20 from 1828021 to 2497055 tid 2 2021-02-10T04:29:10.065 INFO:tasks.rados.rados.0.smithi085.stdout:217: writing smithi08575397-20 from 3134120 to 3764735 tid 3 2021-02-10T04:29:10.065 INFO:tasks.rados.rados.0.smithi085.stdout:218: write oid 32 current snap is 22 2021-02-10T04:29:10.065 INFO:tasks.rados.rados.0.smithi085.stdout:218: seq_num 106 ranges {581542=688983,1886503=412692,2761543=670951} 2021-02-10T04:29:10.072 INFO:tasks.rados.rados.0.smithi085.stdout:218: writing smithi08575397-32 from 581542 to 1270525 tid 1 2021-02-10T04:29:10.078 INFO:tasks.rados.rados.0.smithi085.stdout:218: writing smithi08575397-32 from 1886503 to 2299195 tid 2 2021-02-10T04:29:10.085 INFO:tasks.rados.rados.0.smithi085.stdout:218: writing smithi08575397-32 from 2761543 to 3432494 tid 3 2021-02-10T04:29:10.085 INFO:tasks.rados.rados.0.smithi085.stdout:219: delete oid 17 current snap is 22 2021-02-10T04:29:10.089 INFO:tasks.rados.rados.0.smithi085.stdout:219: done (2 left) 2021-02-10T04:29:10.089 INFO:tasks.rados.rados.0.smithi085.stdout:220: write (excl) oid 47 current snap is 22 2021-02-10T04:29:10.089 INFO:tasks.rados.rados.0.smithi085.stdout:220: seq_num 108 ranges {615156=555834,1603467=416361,2816389=693495} 2021-02-10T04:29:10.096 INFO:tasks.rados.rados.0.smithi085.stdout:220: writing smithi08575397-47 from 615156 to 1170990 tid 1 2021-02-10T04:29:10.101 INFO:tasks.rados.rados.0.smithi085.stdout:220: writing smithi08575397-47 from 1603467 to 2019828 tid 2 2021-02-10T04:29:10.110 INFO:tasks.rados.rados.0.smithi085.stdout:220: writing smithi08575397-47 from 2816389 to 3509884 tid 3 2021-02-10T04:29:10.110 INFO:tasks.rados.rados.0.smithi085.stdout:221: copy_from oid 12 from oid 35 current snap is 22 2021-02-10T04:29:10.111 INFO:tasks.rados.rados.0.smithi085.stdout:222: delete oid 3 current snap is 22 2021-02-10T04:29:10.115 INFO:tasks.rados.rados.0.smithi085.stdout:218: finishing write tid 1 to smithi08575397-32 2021-02-10T04:29:10.115 INFO:tasks.rados.rados.0.smithi085.stdout:222: done (4 left) 2021-02-10T04:29:10.115 INFO:tasks.rados.rados.0.smithi085.stdout:223: snap_remove snap 12 2021-02-10T04:29:10.251 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset noscrub 2021-02-10T04:29:10.354 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:29:10.460 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:29:10.485 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:10.509 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:29:10.580 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:29:10.685 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:29:10.690 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:29:10 smithi085 conmon[52836]: debug 2021-02-10T04:29:10.291+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:10.709 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:10.893 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:29:10 smithi164 conmon[40892]: debug 2021-02-10T04:29:10.558+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:11.040 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:11 smithi085 conmon[35961]: cluster 2021-02-10T04:29:10 2021-02-10T04:29:11.041 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:11 smithi085 conmon[35961]: .000262+0000 mgr.y (mgr.14140) 355 : cluster [DBG] pgmap v349: 168 pgs: 3 active+clean+snaptrim, 4 peering, 161 active+clean; 162 MiB data, 432 MiB used, 715 GiB / 715 GiB avail; 5.3 MiB/s rd, 2.9 MiB/s wr, 13 op/s 2021-02-10T04:29:11.041 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:11 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:29:11.041 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:11 smithi085 conmon[35961]: -10T04:29:10. 2021-02-10T04:29:11.041 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:11 smithi085 conmon[35961]: 044461+0000 mon.a (mon.0 2021-02-10T04:29:11.041 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:11 smithi085 conmon[35961]: ) 805 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2021-02-10T04:29:11.041 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:11 smithi085 conmon[35961]: 2021-02-10T04:29:11.047 INFO:tasks.rados.rados.0.smithi085.stdout:223: done (4 left) 2021-02-10T04:29:11.047 INFO:tasks.rados.rados.0.smithi085.stdout:224: rollback oid 8 current snap is 22 2021-02-10T04:29:11.047 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 8 to 21 2021-02-10T04:29:11.048 INFO:tasks.rados.rados.0.smithi085.stdout:218: finishing write tid 2 to smithi08575397-32 2021-02-10T04:29:11.048 INFO:tasks.rados.rados.0.smithi085.stdout:218: finishing write tid 3 to smithi08575397-32 2021-02-10T04:29:11.048 INFO:tasks.rados.rados.0.smithi085.stdout:218: finishing write tid 4 to smithi08575397-32 2021-02-10T04:29:11.048 INFO:tasks.rados.rados.0.smithi085.stdout:218: finishing write tid 5 to smithi08575397-32 2021-02-10T04:29:11.048 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 32 v 32 (ObjNum 106 snap 22 seq_num 106) dirty exists 2021-02-10T04:29:11.048 INFO:tasks.rados.rados.0.smithi085.stdout:218: left oid 32 (ObjNum 106 snap 22 seq_num 106) 2021-02-10T04:29:11.049 INFO:tasks.rados.rados.0.smithi085.stdout:217: finishing write tid 1 to smithi08575397-20 2021-02-10T04:29:11.049 INFO:tasks.rados.rados.0.smithi085.stdout:220: finishing write tid 1 to smithi08575397-47 2021-02-10T04:29:11.049 INFO:tasks.rados.rados.0.smithi085.stdout:217: finishing write tid 2 to smithi08575397-20 2021-02-10T04:29:11.049 INFO:tasks.rados.rados.0.smithi085.stdout:217: finishing write tid 3 to smithi08575397-20 2021-02-10T04:29:11.049 INFO:tasks.rados.rados.0.smithi085.stdout:217: finishing write tid 4 to smithi08575397-20 2021-02-10T04:29:11.049 INFO:tasks.rados.rados.0.smithi085.stdout:217: finishing write tid 5 to smithi08575397-20 2021-02-10T04:29:11.050 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 20 v 39 (ObjNum 105 snap 22 seq_num 105) dirty exists 2021-02-10T04:29:11.050 INFO:tasks.rados.rados.0.smithi085.stdout:217: left oid 20 (ObjNum 105 snap 22 seq_num 105) 2021-02-10T04:29:11.050 INFO:tasks.rados.rados.0.smithi085.stdout:220: finishing write tid 2 to smithi08575397-47 2021-02-10T04:29:11.050 INFO:tasks.rados.rados.0.smithi085.stdout:220: finishing write tid 3 to smithi08575397-47 2021-02-10T04:29:11.050 INFO:tasks.rados.rados.0.smithi085.stdout:220: finishing write tid 4 to smithi08575397-47 2021-02-10T04:29:11.050 INFO:tasks.rados.rados.0.smithi085.stdout:220: finishing write tid 5 to smithi08575397-47 2021-02-10T04:29:11.051 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 47 v 90 (ObjNum 108 snap 22 seq_num 108) dirty exists 2021-02-10T04:29:11.051 INFO:tasks.rados.rados.0.smithi085.stdout:220: left oid 47 (ObjNum 108 snap 22 seq_num 108) 2021-02-10T04:29:11.051 INFO:tasks.rados.rados.0.smithi085.stdout:217: done (4 left) 2021-02-10T04:29:11.051 INFO:tasks.rados.rados.0.smithi085.stdout:218: done (3 left) 2021-02-10T04:29:11.051 INFO:tasks.rados.rados.0.smithi085.stdout:220: done (2 left) 2021-02-10T04:29:11.051 INFO:tasks.rados.rados.0.smithi085.stdout:225: snap_remove snap 19 2021-02-10T04:29:11.302 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:11 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:29:11.303 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:11 smithi085 conmon[29319]: -02-10T04:29:10.000262+0000 mgr.y (mgr.14140) 355 : cluster [DBG] pgmap v349: 168 pgs: 3 active+clean+snaptrim, 4 peering, 161 active+clean; 162 MiB data, 432 MiB used, 715 GiB / 715 GiB avail; 5.3 MiB/s rd, 2.9 MiB/s wr, 13 op/s 2021-02-10T04:29:11.303 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:11 smithi085 conmon[29319]: cluster 2021-02-10T04:29:10.044461+0000 mon.a (mon 2021-02-10T04:29:11.303 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:11 smithi085 conmon[29319]: .0) 805 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2021-02-10T04:29:11.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:11 smithi164 conmon[29288]: cluster 2021-02-10T04:29:10.000262+0000 mgr.y (mgr.14140) 2021-02-10T04:29:11.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:11 smithi164 conmon[29288]: 355 : cluster [DBG] pgmap v349: 168 pgs: 3 active+clean+snaptrim, 4 peering, 161 active+clean; 162 MiB data, 432 MiB used, 715 GiB / 715 GiB avail; 5.3 MiB/s rd, 2.9 MiB/s wr, 13 op/s 2021-02-10T04:29:11.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:11 smithi164 conmon[29288]: cluster 2021-02-10T04:29:10.044461+0000 mon.a (mon.0) 805 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2021-02-10T04:29:11.425 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:29:11.539 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:29:11.560 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:29:11 smithi085 conmon[41540]: debug 2021-02-10T04:29:11.345+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:11.574 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:12.012 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:29:12.035 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:29:12.051 INFO:tasks.rados.rados.0.smithi085.stdout:225: done (2 left) 2021-02-10T04:29:12.052 INFO:tasks.rados.rados.0.smithi085.stdout:226: read oid 4 snap -1 2021-02-10T04:29:12.052 INFO:tasks.rados.rados.0.smithi085.stdout:226: expect (ObjNum 8 snap 0 seq_num 8) 2021-02-10T04:29:12.052 INFO:tasks.rados.rados.0.smithi085.stdout:227: snap_create 2021-02-10T04:29:12.052 INFO:tasks.rados.rados.0.smithi085.stdout:224: finishing rollback tid 0 to smithi08575397-8 2021-02-10T04:29:12.052 INFO:tasks.rados.rados.0.smithi085.stdout:224: finishing rollback tid 1 to smithi08575397-8 2021-02-10T04:29:12.052 INFO:tasks.rados.rados.0.smithi085.stdout:224: finishing rollback tid 2 to smithi08575397-8 2021-02-10T04:29:12.052 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 35 (ObjNum 83 snap 10 seq_num 83) dirty exists 2021-02-10T04:29:12.060 INFO:tasks.rados.rados.0.smithi085.stdout:226: expect (ObjNum 8 snap 0 seq_num 8) 2021-02-10T04:29:12.065 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:29:12.089 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:29:12.117 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:29:12.119 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:29:12.119 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:29:12.124 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:12 smithi085 conmon[29319]: cluster 2021-02-10T04:29:11 2021-02-10T04:29:12.124 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:12 smithi085 conmon[29319]: .046412+0000 mon.a (mon.0) 806 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2021-02-10T04:29:12.124 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:12 smithi085 conmon[29319]: audit 2021-02-10T04:29:12.124 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:12 smithi085 conmon[29319]: 2021-02-10T04:29 2021-02-10T04:29:12.124 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:12 smithi085 conmon[29319]: :11.988190+0000 mon.a (mon.0) 807 : audit [DBG] from='client.? 172.21.15.85:0/171185585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:29:12.125 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:12 smithi085 conmon[35961]: cluster 2021-02-10T04:29:11 2021-02-10T04:29:12.125 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:12 smithi085 conmon[35961]: .046412+0000 mon.a (mon.0) 806 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2021-02-10T04:29:12.125 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:12 smithi085 conmon[35961]: audit 2021-02-10T04:29:11.988190+0000 mon.a (mon.0) 807 : audit [DBG] from='client.? 172.21.15.85:0/171185585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:29:12.148 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:29:12.174 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:29:12.201 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:29:12.203 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:29:12.203 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:29:12.203 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:29:12.203 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:29:12.203 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:29:12.203 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:29:12.203 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:29:12.203 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:29:12.204 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:29:12.312 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:29:12.335 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:12 smithi164 conmon[29288]: cluster 2021-02-10T04:29:11.046412+0000 mon.a (mon.0) 806 : cluster 2021-02-10T04:29:12.335 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:12 smithi164 conmon[29288]: [DBG] osdmap e135: 8 total, 8 up, 8 in 2021-02-10T04:29:12.335 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:12 smithi164 conmon[29288]: audit 2021-02-10T04:29:11.988190+0000 mon.a (mon.0) 807 : audit [DBG] from='client.? 172.21.15.85:0/171185585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:29:12.339 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:12.440 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:29:12 smithi085 conmon[49106]: debug 2021-02-10T04:29:12.162+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:12.447 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:29:12.552 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:29:12.587 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:12.645 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:29:12 smithi164 conmon[40892]: debug 2021-02-10T04:29:12.414+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:13.038 INFO:tasks.rados.rados.0.smithi085.stdout:224: done (3 left) 2021-02-10T04:29:13.038 INFO:tasks.rados.rados.0.smithi085.stdout:226: done (2 left) 2021-02-10T04:29:13.038 INFO:tasks.rados.rados.0.smithi085.stdout:227: done (1 left) 2021-02-10T04:29:13.038 INFO:tasks.rados.rados.0.smithi085.stdout:228: write oid 28 current snap is 23 2021-02-10T04:29:13.038 INFO:tasks.rados.rados.0.smithi085.stdout:228: seq_num 110 ranges {648770=706332,2120431=420030,3271235=432392} 2021-02-10T04:29:13.047 INFO:tasks.rados.rados.0.smithi085.stdout:228: writing smithi08575397-28 from 648770 to 1355102 tid 1 2021-02-10T04:29:13.054 INFO:tasks.rados.rados.0.smithi085.stdout:228: writing smithi08575397-28 from 2120431 to 2540461 tid 2 2021-02-10T04:29:13.061 INFO:tasks.rados.rados.0.smithi085.stdout:228: writing smithi08575397-28 from 3271235 to 3703627 tid 3 2021-02-10T04:29:13.061 INFO:tasks.rados.rados.0.smithi085.stdout:229: read oid 1 snap -1 2021-02-10T04:29:13.062 INFO:tasks.rados.rados.0.smithi085.stdout:229: expect (ObjNum 8 snap 0 seq_num 8) 2021-02-10T04:29:13.062 INFO:tasks.rados.rados.0.smithi085.stdout:230: snap_remove snap 21 2021-02-10T04:29:13.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:13 smithi164 conmon[29288]: cluster 2021-02-10T04:29:12.000722+0000 mgr.y 2021-02-10T04:29:13.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:13 smithi164 conmon[29288]: (mgr.14140) 356 : cluster [DBG] pgmap v352: 168 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 161 active+clean; 173 MiB data, 460 MiB used, 715 GiB / 715 GiB avail; 4.6 MiB/s rd, 4.1 MiB/s wr, 16 op/s 2021-02-10T04:29:13.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:13 smithi164 conmon[29288]: cluster 2021-02-10T04:29:12.044634+0000 mon.a (mon.0) 808 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2021-02-10T04:29:13.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:13 smithi164 conmon[29288]: 2021-02-10T04:29:13.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:13 smithi164 conmon[29288]: cluster 2021-02-10T04:29: 2021-02-10T04:29:13.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:13 smithi164 conmon[29288]: 12.048870+0000 mon.a (mon.0) 809 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2021-02-10T04:29:13.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:13 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:29:13.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:13 smithi164 conmon[29288]: 02-10T04:29:13.037639 2021-02-10T04:29:13.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:13 smithi164 conmon[29288]: +0000 mon.a (mon.0) 810 : cluster [DBG] 2021-02-10T04:29:13.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:13 smithi164 conmon[29288]: osdmap e137: 8 total, 8 up, 8 in 2021-02-10T04:29:13.415 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-02-10T04:29:13.415 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:29:13.416 INFO:tasks.thrashosds.thrasher:inject_pause on 2 2021-02-10T04:29:13.416 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2021-02-10T04:29:13.416 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2021-02-10T04:29:13.416 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.2 config set filestore_inject_stall 3 2021-02-10T04:29:13.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[35961]: cluster 2021-02-10T04:29:13.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[35961]: 2021-02-10T04:29:13.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[35961]: 2021 2021-02-10T04:29:13.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[35961]: -02-10T04:29:12.000722 2021-02-10T04:29:13.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 356 : cluster [DBG] pgmap v352: 168 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 161 active+clean; 173 MiB data, 460 MiB used, 715 GiB / 715 GiB avail; 4.6 MiB/s rd, 4.1 MiB/s wr, 16 op/s 2021-02-10T04:29:13.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[35961]: cluster 2021-02-10T04:29:12.044634+0000 2021-02-10T04:29:13.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[35961]: mon.a (mon.0) 808 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2021-02-10T04:29:13.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:29:13.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[35961]: -02-10T04:29:12.048870+0000 mon.a 2021-02-10T04:29:13.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[35961]: (mon.0) 809 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2021-02-10T04:29:13.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:29:13.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[35961]: 02-10T04:29:13.037639+0000 mon.a ( 2021-02-10T04:29:13.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[35961]: mon.0) 810 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2021-02-10T04:29:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: cluster 2021-02-10T04:29:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: 2021-02-10T04:29:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: 2021 2021-02-10T04:29:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: - 2021-02-10T04:29:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: 02 2021-02-10T04:29:13.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: -10T04:29: 2021-02-10T04:29:13.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: 12.000722+0000 mgr.y (mgr.14140) 356 : cluster [DBG] pgmap v352: 168 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 161 active+clean; 173 MiB data, 460 MiB used, 715 GiB / 715 GiB avail; 4.6 MiB/s rd, 4.1 MiB/s wr, 16 op/s 2021-02-10T04:29:13.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: 2021-02-10T04:29:13.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: cluster 2021-02-10T04:29:13.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: 2021-02-10T04:29:13.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: 2021 2021-02-10T04:29:13.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: - 2021-02-10T04:29:13.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: 02 2021-02-10T04:29:13.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: - 2021-02-10T04:29:13.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: 10 2021-02-10T04:29:13.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: T 2021-02-10T04:29:13.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: 04 2021-02-10T04:29:13.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: : 2021-02-10T04:29:13.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: 29 2021-02-10T04:29:13.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: : 2021-02-10T04:29:13.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: 12 2021-02-10T04:29:13.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: . 2021-02-10T04:29:13.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: 044634 2021-02-10T04:29:13.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: +0000 2021-02-10T04:29:13.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: 2021-02-10T04:29:13.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: mon.a 2021-02-10T04:29:13.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: ( 2021-02-10T04:29:13.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: mon 2021-02-10T04:29:13.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: . 2021-02-10T04:29:13.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: 0 2021-02-10T04:29:13.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: ) 2021-02-10T04:29:13.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: 808 2021-02-10T04:29:13.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: : 2021-02-10T04:29:13.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: cluster 2021-02-10T04:29:13.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: 2021-02-10T04:29:13.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: [INF] 2021-02-10T04:29:13.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: 2021-02-10T04:29:13.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2021-02-10T04:29:13.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: 2021-02-10T04:29:13.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: cluster 2021-02-10T04:29:13.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: 2021-02-10T04:29:13.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: 2021 2021-02-10T04:29:13.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: - 2021-02-10T04:29:13.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: 02 2021-02-10T04:29:13.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: - 2021-02-10T04:29:13.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: 10 2021-02-10T04:29:13.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: T 2021-02-10T04:29:13.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: 04:29:12.048870+0000 mon.a (mon 2021-02-10T04:29:13.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: .0) 809 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2021-02-10T04:29:13.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: cluster 2021-02-10T04:29:13.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: 2021-02-10T04:29:13.037639+0000 mon.a (mon.0) 810 2021-02-10T04:29:13.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[29319]: : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2021-02-10T04:29:13.685 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:29:14.010 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:29:14.066 INFO:tasks.rados.rados.0.smithi085.stdout:230: done (3 left) 2021-02-10T04:29:14.066 INFO:tasks.rados.rados.0.smithi085.stdout:231: write (excl) oid 7 current snap is 23 2021-02-10T04:29:14.066 INFO:tasks.rados.rados.0.smithi085.stdout:231: seq_num 111 ranges {665577=781581,2062560=563688,3182305=643664} 2021-02-10T04:29:14.075 INFO:tasks.rados.rados.0.smithi085.stdout:231: writing smithi08575397-7 from 665577 to 1447158 tid 1 2021-02-10T04:29:14.082 INFO:tasks.rados.rados.0.smithi085.stdout:231: writing smithi08575397-7 from 2062560 to 2626248 tid 2 2021-02-10T04:29:14.092 INFO:tasks.rados.rados.0.smithi085.stdout:231: writing smithi08575397-7 from 3182305 to 3825969 tid 3 2021-02-10T04:29:14.093 INFO:tasks.rados.rados.0.smithi085.stdout:232: write (excl) oid 29 current snap is 23 2021-02-10T04:29:14.093 INFO:tasks.rados.rados.0.smithi085.stdout:228: finishing write tid 1 to smithi08575397-28 2021-02-10T04:29:14.093 INFO:tasks.rados.rados.0.smithi085.stdout:232: seq_num 112 ranges {682384=456830,1721042=423699,2809728=454936,3731357=151027} 2021-02-10T04:29:14.096 INFO:tasks.rados.rados.0.smithi085.stdout:232: writing smithi08575397-29 from 682384 to 1139214 tid 1 2021-02-10T04:29:14.102 INFO:tasks.rados.rados.0.smithi085.stdout:232: writing smithi08575397-29 from 1721042 to 2144741 tid 2 2021-02-10T04:29:14.109 INFO:tasks.rados.rados.0.smithi085.stdout:232: writing smithi08575397-29 from 2809728 to 3264664 tid 3 2021-02-10T04:29:14.113 INFO:tasks.rados.rados.0.smithi085.stdout:232: writing smithi08575397-29 from 3731357 to 3882384 tid 4 2021-02-10T04:29:14.113 INFO:tasks.rados.rados.0.smithi085.stdout:233: read oid 41 snap -1 2021-02-10T04:29:14.113 INFO:tasks.rados.rados.0.smithi085.stdout:233: expect (ObjNum 39 snap 0 seq_num 39) 2021-02-10T04:29:14.113 INFO:tasks.rados.rados.0.smithi085.stdout:234: read oid 43 snap -1 2021-02-10T04:29:14.113 INFO:tasks.rados.rados.0.smithi085.stdout:234: expect deleted 2021-02-10T04:29:14.113 INFO:tasks.rados.rados.0.smithi085.stdout:235: snap_remove snap 13 2021-02-10T04:29:14.125 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:29:14.156 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:29:13 smithi085 conmon[41540]: debug 2021-02-10T04:29:13.806+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:14.159 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:14.904 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:29:14 smithi085 conmon[45443]: debug 2021-02-10T04:29:14.621+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:14.972 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.3 dump_ops_in_flight 2021-02-10T04:29:14.995 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:29:15.074 INFO:tasks.rados.rados.0.smithi085.stdout:235: done (7 left) 2021-02-10T04:29:15.075 INFO:tasks.rados.rados.0.smithi085.stdout:236: delete oid 42 current snap is 23 2021-02-10T04:29:15.075 INFO:tasks.rados.rados.0.smithi085.stdout:228: finishing write tid 2 to smithi08575397-28 2021-02-10T04:29:15.076 INFO:tasks.rados.rados.0.smithi085.stdout:229: expect (ObjNum 8 snap 0 seq_num 8) 2021-02-10T04:29:15.105 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:29:15.148 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:15.160 INFO:tasks.rados.rados.0.smithi085.stdout:229: done (7 left) 2021-02-10T04:29:15.161 INFO:tasks.rados.rados.0.smithi085.stdout:236: done (6 left) 2021-02-10T04:29:15.161 INFO:tasks.rados.rados.0.smithi085.stdout:237: snap_remove snap 16 2021-02-10T04:29:15.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:15 smithi085 conmon[29319]: cluster 2021-02-10T04:29:14.001187+0000 mgr.y ( 2021-02-10T04:29:15.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:15 smithi085 conmon[29319]: mgr.14140) 357 : cluster [DBG] pgmap v355: 168 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 160 active+clean; 177 MiB data, 472 MiB used, 715 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 10 op/s 2021-02-10T04:29:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:15 smithi085 conmon[29319]: cluster 2021-02-10T04:29:14.065014+0000 mon.a (mon.0) 811 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2021-02-10T04:29:15.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:15 smithi085 conmon[35961]: cluster 2021-02-10T04:29:15.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:15 smithi085 conmon[35961]: 2021-02-10T04:29:14.001187 2021-02-10T04:29:15.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:15 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 357 : cluster [DBG] pgmap v355: 168 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 160 active+clean; 177 MiB data, 472 MiB used, 715 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 10 op/s 2021-02-10T04:29:15.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:15 smithi085 conmon[35961]: cluster 2021-02-10T04:29:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:15 smithi085 conmon[35961]: 2021-02-10T04:29 2021-02-10T04:29:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:15 smithi085 conmon[35961]: :14.065014+0000 mon.a (mon.0) 811 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2021-02-10T04:29:15.345 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:29:15.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:15 smithi164 conmon[29288]: cluster 2021-02-10T04:29:14.001187+0000 mgr.y (mgr.14140) 357 : cluster [DBG] pgmap v355: 168 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 160 active+clean; 177 MiB data, 472 MiB used, 715 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 10 op/s 2021-02-10T04:29:15.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:15 smithi164 conmon[29288]: 2021-02-10T04:29:15.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:15 smithi164 conmon[29288]: cluster 2021-02-10T04:29:14.065014+0000 mon.a (mon.0) 811 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2021-02-10T04:29:15.483 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:29:15.636 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:29:15 smithi085 conmon[49106]: debug 2021-02-10T04:29:15.219+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:15.737 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:29:15.765 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:15.854 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:29:15.959 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:29:15.986 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:16.049 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:29:15 smithi164 conmon[37263]: debug 2021-02-10T04:29:15.826+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:16.049 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:29:16 smithi164 conmon[44544]: debug 2021-02-10T04:29:16.047+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:16.071 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:29:16.081 INFO:tasks.rados.rados.0.smithi085.stdout:237: done (6 left) 2021-02-10T04:29:16.082 INFO:tasks.rados.rados.0.smithi085.stdout:238: write oid 36 current snap is 23 2021-02-10T04:29:16.082 INFO:tasks.rados.rados.0.smithi085.stdout:238: seq_num 114 ranges {715998=607328,1838006=427368,2864574=477480,3863835=52163} 2021-02-10T04:29:16.087 INFO:tasks.rados.rados.0.smithi085.stdout:238: writing smithi08575397-36 from 715998 to 1323326 tid 1 2021-02-10T04:29:16.094 INFO:tasks.rados.rados.0.smithi085.stdout:238: writing smithi08575397-36 from 1838006 to 2265374 tid 2 2021-02-10T04:29:16.101 INFO:tasks.rados.rados.0.smithi085.stdout:238: writing smithi08575397-36 from 2864574 to 3342054 tid 3 2021-02-10T04:29:16.104 INFO:tasks.rados.rados.0.smithi085.stdout:238: writing smithi08575397-36 from 3863835 to 3915998 tid 4 2021-02-10T04:29:16.105 INFO:tasks.rados.rados.0.smithi085.stdout:228: finishing write tid 3 to smithi08575397-28 2021-02-10T04:29:16.105 INFO:tasks.rados.rados.0.smithi085.stdout:228: finishing write tid 4 to smithi08575397-28 2021-02-10T04:29:16.105 INFO:tasks.rados.rados.0.smithi085.stdout:228: finishing write tid 5 to smithi08575397-28 2021-02-10T04:29:16.105 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 11 (ObjNum 110 snap 23 seq_num 110) dirty exists 2021-02-10T04:29:16.105 INFO:tasks.rados.rados.0.smithi085.stdout:228: left oid 28 (ObjNum 110 snap 23 seq_num 110) 2021-02-10T04:29:16.105 INFO:tasks.rados.rados.0.smithi085.stdout:231: finishing write tid 1 to smithi08575397-7 2021-02-10T04:29:16.105 INFO:tasks.rados.rados.0.smithi085.stdout:231: finishing write tid 2 to smithi08575397-7 2021-02-10T04:29:16.105 INFO:tasks.rados.rados.0.smithi085.stdout:231: finishing write tid 3 to smithi08575397-7 2021-02-10T04:29:16.105 INFO:tasks.rados.rados.0.smithi085.stdout:231: finishing write tid 4 to smithi08575397-7 2021-02-10T04:29:16.105 INFO:tasks.rados.rados.0.smithi085.stdout:231: finishing write tid 5 to smithi08575397-7 2021-02-10T04:29:16.105 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 42 (ObjNum 111 snap 23 seq_num 111) dirty exists 2021-02-10T04:29:16.105 INFO:tasks.rados.rados.0.smithi085.stdout:231: left oid 7 (ObjNum 111 snap 23 seq_num 111) 2021-02-10T04:29:16.105 INFO:tasks.rados.rados.0.smithi085.stdout:232: finishing write tid 1 to smithi08575397-29 2021-02-10T04:29:16.106 INFO:tasks.rados.rados.0.smithi085.stdout:232: finishing write tid 2 to smithi08575397-29 2021-02-10T04:29:16.106 INFO:tasks.rados.rados.0.smithi085.stdout:232: finishing write tid 3 to smithi08575397-29 2021-02-10T04:29:16.106 INFO:tasks.rados.rados.0.smithi085.stdout:232: finishing write tid 4 to smithi08575397-29 2021-02-10T04:29:16.106 INFO:tasks.rados.rados.0.smithi085.stdout:232: finishing write tid 5 to smithi08575397-29 2021-02-10T04:29:16.106 INFO:tasks.rados.rados.0.smithi085.stdout:232: finishing write tid 6 to smithi08575397-29 2021-02-10T04:29:16.106 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 29 v 40 (ObjNum 112 snap 23 seq_num 112) dirty exists 2021-02-10T04:29:16.106 INFO:tasks.rados.rados.0.smithi085.stdout:232: left oid 29 (ObjNum 112 snap 23 seq_num 112) 2021-02-10T04:29:16.106 INFO:tasks.rados.rados.0.smithi085.stdout:233: expect (ObjNum 39 snap 0 seq_num 39) 2021-02-10T04:29:16.173 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:16 smithi085 conmon[29319]: cluster 2021-02-10T04:29:15.074234+0000 mon.a 2021-02-10T04:29:16.173 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:16 smithi085 conmon[29319]: (mon.0) 812 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2021-02-10T04:29:16.173 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:16 smithi085 conmon[35961]: cluster 2021-02-10T04:29:15.074234+0000 2021-02-10T04:29:16.173 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:16 smithi085 conmon[35961]: mon.a (mon.0) 812 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2021-02-10T04:29:16.176 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:29:16.201 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:16.235 INFO:tasks.rados.rados.0.smithi085.stdout:238: finishing write tid 1 to smithi08575397-36 2021-02-10T04:29:16.235 INFO:tasks.rados.rados.0.smithi085.stdout:238: finishing write tid 2 to smithi08575397-36 2021-02-10T04:29:16.235 INFO:tasks.rados.rados.0.smithi085.stdout:238: finishing write tid 3 to smithi08575397-36 2021-02-10T04:29:16.235 INFO:tasks.rados.rados.0.smithi085.stdout:228: done (6 left) 2021-02-10T04:29:16.236 INFO:tasks.rados.rados.0.smithi085.stdout:231: done (5 left) 2021-02-10T04:29:16.236 INFO:tasks.rados.rados.0.smithi085.stdout:232: done (4 left) 2021-02-10T04:29:16.236 INFO:tasks.rados.rados.0.smithi085.stdout:233: done (3 left) 2021-02-10T04:29:16.236 INFO:tasks.rados.rados.0.smithi085.stdout:234: done (2 left) 2021-02-10T04:29:16.236 INFO:tasks.rados.rados.0.smithi085.stdout:239: snap_remove snap 17 2021-02-10T04:29:16.296 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:29:16.393 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:29:16 smithi164 conmon[37263]: debug 2021-02-10T04:29:16.259+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:16.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:16 smithi164 conmon[29288]: cluster 2021-02-10T04:29:16.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:16 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:29:16.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:16 smithi164 conmon[29288]: 04:29:15.074234+0000 mon.a (mon.0) 812 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2021-02-10T04:29:16.401 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:29:16.436 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:16.923 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:29:17.082 INFO:tasks.rados.rados.0.smithi085.stdout:239: done (2 left) 2021-02-10T04:29:17.082 INFO:tasks.rados.rados.0.smithi085.stdout:240: delete oid 17 current snap is 23 2021-02-10T04:29:17.082 INFO:tasks.rados.rados.0.smithi085.stdout:238: finishing write tid 4 to smithi08575397-36 2021-02-10T04:29:17.083 INFO:tasks.rados.rados.0.smithi085.stdout:238: finishing write tid 5 to smithi08575397-36 2021-02-10T04:29:17.083 INFO:tasks.rados.rados.0.smithi085.stdout:238: finishing write tid 6 to smithi08575397-36 2021-02-10T04:29:17.083 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 31 (ObjNum 114 snap 23 seq_num 114) dirty exists 2021-02-10T04:29:17.083 INFO:tasks.rados.rados.0.smithi085.stdout:238: left oid 36 (ObjNum 114 snap 23 seq_num 114) 2021-02-10T04:29:17.086 INFO:tasks.rados.rados.0.smithi085.stdout:238: done (2 left) 2021-02-10T04:29:17.086 INFO:tasks.rados.rados.0.smithi085.stdout:240: done (1 left) 2021-02-10T04:29:17.087 INFO:tasks.rados.rados.0.smithi085.stdout:241: delete oid 30 current snap is 23 2021-02-10T04:29:17.127 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is unset 2021-02-10T04:29:17.140 INFO:tasks.rados.rados.0.smithi085.stdout:241: done (1 left) 2021-02-10T04:29:17.140 INFO:tasks.rados.rados.0.smithi085.stdout:242: copy_from oid 29 from oid 4 current snap is 23 2021-02-10T04:29:17.140 INFO:tasks.rados.rados.0.smithi085.stdout:243: copy_from oid 37 from oid 26 current snap is 23 2021-02-10T04:29:17.140 INFO:tasks.rados.rados.0.smithi085.stdout:244: copy_from oid 5 from oid 17 current snap is 23 2021-02-10T04:29:17.140 INFO:tasks.rados.rados.0.smithi085.stdout:245: read oid 23 snap -1 2021-02-10T04:29:17.140 INFO:tasks.rados.rados.0.smithi085.stdout:245: expect deleted 2021-02-10T04:29:17.140 INFO:tasks.rados.rados.0.smithi085.stdout:246: read oid 1 snap -1 2021-02-10T04:29:17.140 INFO:tasks.rados.rados.0.smithi085.stdout:246: expect (ObjNum 8 snap 0 seq_num 8) 2021-02-10T04:29:17.141 INFO:tasks.rados.rados.0.smithi085.stdout:247: snap_remove snap 15 2021-02-10T04:29:17.174 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:29:17.268 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:17 smithi085 conmon[29319]: cluster 2021-02-10T04:29:16.001558+0000 mgr.y (mgr.14140) 358 : cluster [DBG] pgmap v358: 168 pgs: 4 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 155 active+clean; 186 MiB data, 486 MiB used, 715 GiB / 715 GiB avail; 4.6 MiB/s rd, 2.5 MiB/s wr, 14 op/s 2021-02-10T04:29:17.269 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:17 smithi085 conmon[29319]: cluster 2021-02-10T04:29:16.077025+0000 mon.a (mon.0) 813 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2021-02-10T04:29:17.269 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:17 smithi085 conmon[29319]: audit 2021-02-10T04:29:16.773678+0000 mon.c (mon.1) 42 : audit [INF] from='client.? 172.21.15.85:0/252643868' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:29:17.269 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:17 smithi085 conmon[29319]: audit 2021-02-10T04:29:16.774087+0000 mon.a (mon.0) 814 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:29:17.269 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:17 smithi085 conmon[35961]: cluster 2021-02-10T04:29:16.001558+0000 mgr.y (mgr.14140) 2021-02-10T04:29:17.269 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:17 smithi085 conmon[35961]: 358 : cluster [DBG] pgmap v358: 168 pgs: 4 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 155 active+clean; 186 MiB data, 486 MiB used, 715 GiB / 715 GiB avail; 4.6 MiB/s rd, 2.5 MiB/s wr, 14 op/s 2021-02-10T04:29:17.269 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:17 smithi085 conmon[35961]: cluster 2021-02-10T04:29:16.077025+0000 mon.a (mon.0) 813 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2021-02-10T04:29:17.269 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:17 smithi085 conmon[35961]: audit 2021-02-10T04:29:16.773678+0000 mon.c (mon.1) 42 : audit [INF] from='client.? 172.21.15.85:0/252643868' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:29:17.269 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:17 smithi085 conmon[35961]: audit 2021-02-10T04:29:16.774087+0000 mon.a (mon.0) 814 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:29:17.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:17 smithi164 conmon[29288]: cluster 2021-02-10T04:29:16.001558+0000 mgr.y (mgr.14140) 358 : cluster 2021-02-10T04:29:17.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:17 smithi164 conmon[29288]: [DBG] pgmap v358: 168 pgs: 4 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 155 active+clean; 186 MiB data, 486 MiB used, 715 GiB / 715 GiB avail; 4.6 MiB/s rd, 2.5 MiB/s wr, 14 op/s 2021-02-10T04:29:17.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:17 smithi164 conmon[29288]: cluster 2021-02-10T04:29:16.077025+0000 mon.a (mon.0) 813 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2021-02-10T04:29:17.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:17 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:29:17.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:17 smithi164 conmon[29288]: T04:29:16.773678+0000 mon.c (mon.1) 42 : audit [INF] from='client.? 172.21.15.85:0/252643868' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:29:17.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:17 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:29:17.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:17 smithi164 conmon[29288]: :29:16.774087+0000 mon.a (mon.0 2021-02-10T04:29:17.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:17 smithi164 conmon[29288]: ) 814 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:29:17.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:29:17 smithi085 conmon[49106]: debug 2021-02-10T04:29:17.267+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:18.044 INFO:tasks.rados.rados.0.smithi085.stdout:247: done (6 left) 2021-02-10T04:29:18.045 INFO:tasks.rados.rados.0.smithi085.stdout:248: rollback oid 32 current snap is 23 2021-02-10T04:29:18.045 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 32 to 10 2021-02-10T04:29:18.045 INFO:tasks.rados.rados.0.smithi085.stdout:249: read oid 22 snap 10 2021-02-10T04:29:18.045 INFO:tasks.rados.rados.0.smithi085.stdout:249: expect (ObjNum 60 snap 4 seq_num 60) 2021-02-10T04:29:18.045 INFO:tasks.rados.rados.0.smithi085.stdout:250: copy_from oid 48 from oid 20 current snap is 23 2021-02-10T04:29:18.045 INFO:tasks.rados.rados.0.smithi085.stdout:251: delete oid 6 current snap is 23 2021-02-10T04:29:18.045 INFO:tasks.rados.rados.0.smithi085.stdout:244: finishing copy_from to smithi08575397-5 2021-02-10T04:29:18.045 INFO:tasks.rados.rados.0.smithi085.stdout:244: got expected ENOENT (src dne) 2021-02-10T04:29:18.045 INFO:tasks.rados.rados.0.smithi085.stdout:244: finishing copy_from racing read to smithi08575397-5 2021-02-10T04:29:18.045 INFO:tasks.rados.rados.0.smithi085.stdout:246: expect (ObjNum 8 snap 0 seq_num 8) 2021-02-10T04:29:18.075 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:29:17 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:29:18.126 INFO:tasks.rados.rados.0.smithi085.stdout:243: finishing copy_from to smithi08575397-37 2021-02-10T04:29:18.126 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2021-02-10T04:29:18.126 INFO:tasks.rados.rados.0.smithi085.stdout:243: finishing copy_from racing read to smithi08575397-37 2021-02-10T04:29:18.127 INFO:tasks.rados.rados.0.smithi085.stdout:243: done (9 left) 2021-02-10T04:29:18.127 INFO:tasks.rados.rados.0.smithi085.stdout:244: done (8 left) 2021-02-10T04:29:18.127 INFO:tasks.rados.rados.0.smithi085.stdout:245: done (7 left) 2021-02-10T04:29:18.127 INFO:tasks.rados.rados.0.smithi085.stdout:246: done (6 left) 2021-02-10T04:29:18.127 INFO:tasks.rados.rados.0.smithi085.stdout:251: done (5 left) 2021-02-10T04:29:18.127 INFO:tasks.rados.rados.0.smithi085.stdout:252: write oid 46 current snap is 23 2021-02-10T04:29:18.127 INFO:tasks.rados.rados.0.smithi085.stdout:249: expect (ObjNum 60 snap 4 seq_num 60) 2021-02-10T04:29:18.286 INFO:tasks.rados.rados.0.smithi085.stdout:248: finishing rollback tid 0 to smithi08575397-32 2021-02-10T04:29:18.287 INFO:tasks.rados.rados.0.smithi085.stdout:248: finishing rollback tid 1 to smithi08575397-32 2021-02-10T04:29:18.287 INFO:tasks.rados.rados.0.smithi085.stdout:248: finishing rollback tid 2 to smithi08575397-32 2021-02-10T04:29:18.287 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 32 v 46 (ObjNum 31 snap 0 seq_num 31) dirty exists 2021-02-10T04:29:18.287 INFO:tasks.rados.rados.0.smithi085.stdout:252: seq_num 118 ranges {783226=624677,2071934=434706,2974266=638921} 2021-02-10T04:29:18.295 INFO:tasks.rados.rados.0.smithi085.stdout:252: writing smithi08575397-46 from 783226 to 1407903 tid 1 2021-02-10T04:29:18.301 INFO:tasks.rados.rados.0.smithi085.stdout:252: writing smithi08575397-46 from 2071934 to 2506640 tid 2 2021-02-10T04:29:18.308 INFO:tasks.rados.rados.0.smithi085.stdout:252: writing smithi08575397-46 from 2974266 to 3613187 tid 3 2021-02-10T04:29:18.308 INFO:tasks.rados.rados.0.smithi085.stdout:248: done (5 left) 2021-02-10T04:29:18.308 INFO:tasks.rados.rados.0.smithi085.stdout:249: done (4 left) 2021-02-10T04:29:18.308 INFO:tasks.rados.rados.0.smithi085.stdout:253: write oid 47 current snap is 23 2021-02-10T04:29:18.308 INFO:tasks.rados.rados.0.smithi085.stdout:252: finishing write tid 1 to smithi08575397-46 2021-02-10T04:29:18.308 INFO:tasks.rados.rados.0.smithi085.stdout:252: finishing write tid 2 to smithi08575397-46 2021-02-10T04:29:18.308 INFO:tasks.rados.rados.0.smithi085.stdout:253: seq_num 119 ranges {400033=699926,1614063=385970} 2021-02-10T04:29:18.315 INFO:tasks.rados.rados.0.smithi085.stdout:253: writing smithi08575397-47 from 400033 to 1099959 tid 1 2021-02-10T04:29:18.320 INFO:tasks.rados.rados.0.smithi085.stdout:253: writing smithi08575397-47 from 1614063 to 2000033 tid 2 2021-02-10T04:29:18.321 INFO:tasks.rados.rados.0.smithi085.stdout:254: read oid 2 snap -1 2021-02-10T04:29:18.321 INFO:tasks.rados.rados.0.smithi085.stdout:254: expect deleted 2021-02-10T04:29:18.321 INFO:tasks.rados.rados.0.smithi085.stdout:255: delete oid 8 current snap is 23 2021-02-10T04:29:18.321 INFO:tasks.rados.rados.0.smithi085.stdout:252: finishing write tid 3 to smithi08575397-46 2021-02-10T04:29:18.321 INFO:tasks.rados.rados.0.smithi085.stdout:252: finishing write tid 4 to smithi08575397-46 2021-02-10T04:29:18.321 INFO:tasks.rados.rados.0.smithi085.stdout:252: finishing write tid 5 to smithi08575397-46 2021-02-10T04:29:18.321 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 75 (ObjNum 118 snap 23 seq_num 118) dirty exists 2021-02-10T04:29:18.321 INFO:tasks.rados.rados.0.smithi085.stdout:252: left oid 46 (ObjNum 118 snap 23 seq_num 118) 2021-02-10T04:29:18.321 INFO:tasks.rados.rados.0.smithi085.stdout:253: finishing write tid 1 to smithi08575397-47 2021-02-10T04:29:18.324 INFO:tasks.rados.rados.0.smithi085.stdout:252: done (6 left) 2021-02-10T04:29:18.324 INFO:tasks.rados.rados.0.smithi085.stdout:254: done (5 left) 2021-02-10T04:29:18.324 INFO:tasks.rados.rados.0.smithi085.stdout:255: done (4 left) 2021-02-10T04:29:18.324 INFO:tasks.rados.rados.0.smithi085.stdout:256: write (excl) oid 7 current snap is 23 2021-02-10T04:29:18.324 INFO:tasks.rados.rados.0.smithi085.stdout:256: seq_num 121 ranges {433647=450424,1331027=582033} 2021-02-10T04:29:18.329 INFO:tasks.rados.rados.0.smithi085.stdout:256: writing smithi08575397-7 from 433647 to 884071 tid 1 2021-02-10T04:29:18.335 INFO:tasks.rados.rados.0.smithi085.stdout:256: writing smithi08575397-7 from 1331027 to 1913060 tid 2 2021-02-10T04:29:18.336 INFO:tasks.rados.rados.0.smithi085.stdout:257: rollback oid 50 current snap is 23 2021-02-10T04:29:18.336 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 50 to 22 2021-02-10T04:29:18.336 INFO:tasks.rados.rados.0.smithi085.stdout:253: finishing write tid 2 to smithi08575397-47 2021-02-10T04:29:18.336 INFO:tasks.rados.rados.0.smithi085.stdout:253: finishing write tid 3 to smithi08575397-47 2021-02-10T04:29:18.336 INFO:tasks.rados.rados.0.smithi085.stdout:253: finishing write tid 4 to smithi08575397-47 2021-02-10T04:29:18.336 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 47 v 106 (ObjNum 119 snap 23 seq_num 119) dirty exists 2021-02-10T04:29:18.336 INFO:tasks.rados.rados.0.smithi085.stdout:253: left oid 47 (ObjNum 119 snap 23 seq_num 119) 2021-02-10T04:29:18.336 INFO:tasks.rados.rados.0.smithi085.stdout:256: finishing write tid 1 to smithi08575397-7 2021-02-10T04:29:18.336 INFO:tasks.rados.rados.0.smithi085.stdout:253: done (5 left) 2021-02-10T04:29:18.336 INFO:tasks.rados.rados.0.smithi085.stdout:258: read oid 30 snap -1 2021-02-10T04:29:18.336 INFO:tasks.rados.rados.0.smithi085.stdout:258: expect deleted 2021-02-10T04:29:18.336 INFO:tasks.rados.rados.0.smithi085.stdout:259: write oid 26 current snap is 23 2021-02-10T04:29:18.337 INFO:tasks.rados.rados.0.smithi085.stdout:259: seq_num 122 ranges {450454=642026,1505862=544592} 2021-02-10T04:29:18.342 INFO:tasks.rados.rados.0.smithi085.stdout:259: writing smithi08575397-26 from 450454 to 1092480 tid 1 2021-02-10T04:29:18.348 INFO:tasks.rados.rados.0.smithi085.stdout:259: writing smithi08575397-26 from 1505862 to 2050454 tid 2 2021-02-10T04:29:18.348 INFO:tasks.rados.rados.0.smithi085.stdout:260: read oid 8 snap -1 2021-02-10T04:29:18.348 INFO:tasks.rados.rados.0.smithi085.stdout:260: expect deleted 2021-02-10T04:29:18.348 INFO:tasks.rados.rados.0.smithi085.stdout:261: read oid 18 snap -1 2021-02-10T04:29:18.348 INFO:tasks.rados.rados.0.smithi085.stdout:261: expect (ObjNum 99 snap 20 seq_num 99) 2021-02-10T04:29:18.348 INFO:tasks.rados.rados.0.smithi085.stdout:262: write (excl) oid 24 current snap is 23 2021-02-10T04:29:18.348 INFO:tasks.rados.rados.0.smithi085.stdout:257: finishing rollback tid 0 to smithi08575397-50 2021-02-10T04:29:18.349 INFO:tasks.rados.rados.0.smithi085.stdout:257: finishing rollback tid 1 to smithi08575397-50 2021-02-10T04:29:18.349 INFO:tasks.rados.rados.0.smithi085.stdout:262: seq_num 123 ranges {467261=717275,1847991=219270} 2021-02-10T04:29:18.355 INFO:tasks.rados.rados.0.smithi085.stdout:262: writing smithi08575397-24 from 467261 to 1184536 tid 1 2021-02-10T04:29:18.360 INFO:tasks.rados.rados.0.smithi085.stdout:262: writing smithi08575397-24 from 1847991 to 2067261 tid 2 2021-02-10T04:29:18.360 INFO:tasks.rados.rados.0.smithi085.stdout:258: done (9 left) 2021-02-10T04:29:18.361 INFO:tasks.rados.rados.0.smithi085.stdout:263: write oid 23 current snap is 23 2021-02-10T04:29:18.361 INFO:tasks.rados.rados.0.smithi085.stdout:257: finishing rollback tid 2 to smithi08575397-50 2021-02-10T04:29:18.361 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 47 (ObjNum 10 snap 0 seq_num 10) dirty exists 2021-02-10T04:29:18.361 INFO:tasks.rados.rados.0.smithi085.stdout:256: finishing write tid 2 to smithi08575397-7 2021-02-10T04:29:18.361 INFO:tasks.rados.rados.0.smithi085.stdout:256: finishing write tid 3 to smithi08575397-7 2021-02-10T04:29:18.361 INFO:tasks.rados.rados.0.smithi085.stdout:256: finishing write tid 4 to smithi08575397-7 2021-02-10T04:29:18.361 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 45 (ObjNum 121 snap 23 seq_num 121) dirty exists 2021-02-10T04:29:18.361 INFO:tasks.rados.rados.0.smithi085.stdout:256: left oid 7 (ObjNum 121 snap 23 seq_num 121) 2021-02-10T04:29:18.361 INFO:tasks.rados.rados.0.smithi085.stdout:259: finishing write tid 1 to smithi08575397-26 2021-02-10T04:29:18.361 INFO:tasks.rados.rados.0.smithi085.stdout:263: seq_num 124 ranges {484068=792524,1906473=177595} 2021-02-10T04:29:18.368 INFO:tasks.rados.rados.0.smithi085.stdout:263: writing smithi08575397-23 from 484068 to 1276592 tid 1 2021-02-10T04:29:18.373 INFO:tasks.rados.rados.0.smithi085.stdout:263: writing smithi08575397-23 from 1906473 to 2084068 tid 2 2021-02-10T04:29:18.374 INFO:tasks.rados.rados.0.smithi085.stdout:256: done (9 left) 2021-02-10T04:29:18.374 INFO:tasks.rados.rados.0.smithi085.stdout:257: done (8 left) 2021-02-10T04:29:18.374 INFO:tasks.rados.rados.0.smithi085.stdout:264: write (excl) oid 36 current snap is 23 2021-02-10T04:29:18.374 INFO:tasks.rados.rados.0.smithi085.stdout:259: finishing write tid 2 to smithi08575397-26 2021-02-10T04:29:18.374 INFO:tasks.rados.rados.0.smithi085.stdout:259: finishing write tid 3 to smithi08575397-26 2021-02-10T04:29:18.374 INFO:tasks.rados.rados.0.smithi085.stdout:259: finishing write tid 4 to smithi08575397-26 2021-02-10T04:29:18.375 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 14 (ObjNum 122 snap 23 seq_num 122) dirty exists 2021-02-10T04:29:18.375 INFO:tasks.rados.rados.0.smithi085.stdout:259: left oid 26 (ObjNum 122 snap 23 seq_num 122) 2021-02-10T04:29:18.375 INFO:tasks.rados.rados.0.smithi085.stdout:261: expect (ObjNum 99 snap 20 seq_num 99) 2021-02-10T04:29:18.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:18 smithi164 conmon[29288]: cluster 2021-02-10T04:29:17.074498+0000 mon.a (mon 2021-02-10T04:29:18.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:18 smithi164 conmon[29288]: .0) 815 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:29:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:18 smithi164 conmon[29288]: audit 2021-02-10T04:29:17.082061+0000 mon.a (mon.0) 816 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:29:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:18 smithi164 conmon[29288]: cluster 2021-02-10T04:29:17.082127+0000 2021-02-10T04:29:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:18 smithi164 conmon[29288]: mon.a (mon.0) 817 : cluster [DBG] 2021-02-10T04:29:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:18 smithi164 conmon[29288]: osdmap e141: 8 total, 8 up, 8 in 2021-02-10T04:29:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:18 smithi164 conmon[29288]: cluster 2021-02-10T04:29:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:18 smithi164 conmon[29288]: 2021-02-10T04:29: 2021-02-10T04:29:18.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:18 smithi164 conmon[29288]: 18.038292+0000 mon.a (mon.0) 818 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2021-02-10T04:29:18.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:18 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:29:18.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:18 smithi085 conmon[35961]: -02-10T04:29:17.074498+0000 mon.a (mon.0) 815 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:29:18.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:18 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:29:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:18 smithi085 conmon[35961]: -10T04:29:17.082061+0000 mon.a 2021-02-10T04:29:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:18 smithi085 conmon[35961]: (mon.0) 816 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:29:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:18 smithi085 conmon[35961]: 2021-02-10T04:29:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:18 smithi085 conmon[35961]: cluster 2021-02-10T04:29:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:18 smithi085 conmon[35961]: 2021-02-10T04:29:17.082127+0000 mon.a (mon 2021-02-10T04:29:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:18 smithi085 conmon[35961]: .0) 817 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2021-02-10T04:29:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:18 smithi085 conmon[35961]: cluster 2021-02-10T04:29:18.038292 2021-02-10T04:29:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:18 smithi085 conmon[35961]: +0000 mon.a (mon.0) 818 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2021-02-10T04:29:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:18 smithi085 conmon[29319]: cluster 2021-02-10T04:29:17.074498+0000 mon.a ( 2021-02-10T04:29:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:18 smithi085 conmon[29319]: mon.0) 815 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:29:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:18 smithi085 conmon[29319]: audit 2021-02-10T04:29:17.082061+0000 mon.a (mon.0) 816 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:29:18.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:18 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:29:18.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:18 smithi085 conmon[29319]: 04:29:17.082127+0000 mon.a (mon.0) 817 2021-02-10T04:29:18.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:18 smithi085 conmon[29319]: : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2021-02-10T04:29:18.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:18 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:29:18.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:18 smithi085 conmon[29319]: -10T04:29:18.038292+0000 mon.a (mon 2021-02-10T04:29:18.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:18 smithi085 conmon[29319]: .0) 818 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2021-02-10T04:29:18.562 INFO:tasks.rados.rados.0.smithi085.stdout:262: finishing write tid 1 to smithi08575397-24 2021-02-10T04:29:18.562 INFO:tasks.rados.rados.0.smithi085.stdout:262: finishing write tid 2 to smithi08575397-24 2021-02-10T04:29:18.562 INFO:tasks.rados.rados.0.smithi085.stdout:262: finishing write tid 3 to smithi08575397-24 2021-02-10T04:29:18.563 INFO:tasks.rados.rados.0.smithi085.stdout:262: finishing write tid 4 to smithi08575397-24 2021-02-10T04:29:18.563 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 24 v 19 (ObjNum 123 snap 23 seq_num 123) dirty exists 2021-02-10T04:29:18.563 INFO:tasks.rados.rados.0.smithi085.stdout:262: left oid 24 (ObjNum 123 snap 23 seq_num 123) 2021-02-10T04:29:18.563 INFO:tasks.rados.rados.0.smithi085.stdout:263: finishing write tid 1 to smithi08575397-23 2021-02-10T04:29:18.563 INFO:tasks.rados.rados.0.smithi085.stdout:263: finishing write tid 2 to smithi08575397-23 2021-02-10T04:29:18.563 INFO:tasks.rados.rados.0.smithi085.stdout:263: finishing write tid 3 to smithi08575397-23 2021-02-10T04:29:18.563 INFO:tasks.rados.rados.0.smithi085.stdout:263: finishing write tid 4 to smithi08575397-23 2021-02-10T04:29:18.563 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 34 (ObjNum 124 snap 23 seq_num 124) dirty exists 2021-02-10T04:29:18.563 INFO:tasks.rados.rados.0.smithi085.stdout:263: left oid 23 (ObjNum 124 snap 23 seq_num 124) 2021-02-10T04:29:18.563 INFO:tasks.rados.rados.0.smithi085.stdout:264: seq_num 125 ranges {500875=467773,1564955=535920} 2021-02-10T04:29:18.568 INFO:tasks.rados.rados.0.smithi085.stdout:264: writing smithi08575397-36 from 500875 to 968648 tid 1 2021-02-10T04:29:18.574 INFO:tasks.rados.rados.0.smithi085.stdout:264: writing smithi08575397-36 from 1564955 to 2100875 tid 2 2021-02-10T04:29:18.575 INFO:tasks.rados.rados.0.smithi085.stdout:259: done (8 left) 2021-02-10T04:29:18.575 INFO:tasks.rados.rados.0.smithi085.stdout:260: done (7 left) 2021-02-10T04:29:18.575 INFO:tasks.rados.rados.0.smithi085.stdout:261: done (6 left) 2021-02-10T04:29:18.575 INFO:tasks.rados.rados.0.smithi085.stdout:262: done (5 left) 2021-02-10T04:29:18.575 INFO:tasks.rados.rados.0.smithi085.stdout:263: done (4 left) 2021-02-10T04:29:18.575 INFO:tasks.rados.rados.0.smithi085.stdout:265: write (excl) oid 16 current snap is 23 2021-02-10T04:29:18.575 INFO:tasks.rados.rados.0.smithi085.stdout:264: finishing write tid 1 to smithi08575397-36 2021-02-10T04:29:18.575 INFO:tasks.rados.rados.0.smithi085.stdout:265: seq_num 126 ranges {517682=543022,1623437=494245} 2021-02-10T04:29:18.581 INFO:tasks.rados.rados.0.smithi085.stdout:265: writing smithi08575397-16 from 517682 to 1060704 tid 1 2021-02-10T04:29:18.587 INFO:tasks.rados.rados.0.smithi085.stdout:265: writing smithi08575397-16 from 1623437 to 2117682 tid 2 2021-02-10T04:29:18.588 INFO:tasks.rados.rados.0.smithi085.stdout:266: write (excl) oid 22 current snap is 23 2021-02-10T04:29:18.588 INFO:tasks.rados.rados.0.smithi085.stdout:264: finishing write tid 2 to smithi08575397-36 2021-02-10T04:29:18.588 INFO:tasks.rados.rados.0.smithi085.stdout:264: finishing write tid 3 to smithi08575397-36 2021-02-10T04:29:18.588 INFO:tasks.rados.rados.0.smithi085.stdout:264: finishing write tid 4 to smithi08575397-36 2021-02-10T04:29:18.588 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 37 (ObjNum 125 snap 23 seq_num 125) dirty exists 2021-02-10T04:29:18.588 INFO:tasks.rados.rados.0.smithi085.stdout:264: left oid 36 (ObjNum 125 snap 23 seq_num 125) 2021-02-10T04:29:18.588 INFO:tasks.rados.rados.0.smithi085.stdout:265: finishing write tid 1 to smithi08575397-16 2021-02-10T04:29:18.588 INFO:tasks.rados.rados.0.smithi085.stdout:266: seq_num 127 ranges {534489=618271,1565566=568923} 2021-02-10T04:29:18.594 INFO:tasks.rados.rados.0.smithi085.stdout:266: writing smithi08575397-22 from 534489 to 1152760 tid 1 2021-02-10T04:29:18.600 INFO:tasks.rados.rados.0.smithi085.stdout:266: writing smithi08575397-22 from 1565566 to 2134489 tid 2 2021-02-10T04:29:18.600 INFO:tasks.rados.rados.0.smithi085.stdout:264: done (5 left) 2021-02-10T04:29:18.601 INFO:tasks.rados.rados.0.smithi085.stdout:267: read oid 46 snap -1 2021-02-10T04:29:18.601 INFO:tasks.rados.rados.0.smithi085.stdout:267: expect (ObjNum 118 snap 23 seq_num 118) 2021-02-10T04:29:18.601 INFO:tasks.rados.rados.0.smithi085.stdout:268: snap_create 2021-02-10T04:29:18.601 INFO:tasks.rados.rados.0.smithi085.stdout:265: finishing write tid 2 to smithi08575397-16 2021-02-10T04:29:18.601 INFO:tasks.rados.rados.0.smithi085.stdout:265: finishing write tid 3 to smithi08575397-16 2021-02-10T04:29:18.601 INFO:tasks.rados.rados.0.smithi085.stdout:265: finishing write tid 4 to smithi08575397-16 2021-02-10T04:29:18.601 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 110 (ObjNum 126 snap 23 seq_num 126) dirty exists 2021-02-10T04:29:18.601 INFO:tasks.rados.rados.0.smithi085.stdout:265: left oid 16 (ObjNum 126 snap 23 seq_num 126) 2021-02-10T04:29:18.601 INFO:tasks.rados.rados.0.smithi085.stdout:266: finishing write tid 1 to smithi08575397-22 2021-02-10T04:29:18.603 INFO:tasks.rados.rados.0.smithi085.stdout:266: finishing write tid 2 to smithi08575397-22 2021-02-10T04:29:18.605 INFO:tasks.rados.rados.0.smithi085.stdout:266: finishing write tid 3 to smithi08575397-22 2021-02-10T04:29:18.605 INFO:tasks.rados.rados.0.smithi085.stdout:266: finishing write tid 4 to smithi08575397-22 2021-02-10T04:29:18.605 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 48 (ObjNum 127 snap 23 seq_num 127) dirty exists 2021-02-10T04:29:18.605 INFO:tasks.rados.rados.0.smithi085.stdout:266: left oid 22 (ObjNum 127 snap 23 seq_num 127) 2021-02-10T04:29:18.605 INFO:tasks.rados.rados.0.smithi085.stdout:267: expect (ObjNum 118 snap 23 seq_num 118) 2021-02-10T04:29:19.088 INFO:tasks.rados.rados.0.smithi085.stdout:265: done (6 left) 2021-02-10T04:29:19.088 INFO:tasks.rados.rados.0.smithi085.stdout:266: done (5 left) 2021-02-10T04:29:19.088 INFO:tasks.rados.rados.0.smithi085.stdout:267: done (4 left) 2021-02-10T04:29:19.088 INFO:tasks.rados.rados.0.smithi085.stdout:268: done (3 left) 2021-02-10T04:29:19.088 INFO:tasks.rados.rados.0.smithi085.stdout:269: snap_remove snap 22 2021-02-10T04:29:19.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[35961]: audit 2021-02-10T04:29: 2021-02-10T04:29:19.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[35961]: 17.715022+0000 mgr.y (mgr.14140) 359 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:29:19.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:29:19.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[35961]: -10T04:29:18. 2021-02-10T04:29:19.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[35961]: 001929+0000 mgr.y (mgr.14140) 2021-02-10T04:29:19.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[35961]: 360 : cluster [DBG] pgmap v361: 168 pgs: 5 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 152 active+clean; 188 MiB data, 490 MiB used, 715 GiB / 715 GiB avail; 4.6 MiB/s rd, 2.1 MiB/s wr, 13 op/s 2021-02-10T04:29:19.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:29:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[35961]: 04:29:18.107864+0000 osd.1 (osd.1) 42 : cluster [DBG] 2.a scrub ok 2021-02-10T04:29:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:29:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[35961]: 04:29:18.193321+0000 osd.6 (osd.6) 2021-02-10T04:29:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[35961]: 31 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:29:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:29:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[35961]: -02-10T04:29: 2021-02-10T04:29:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[35961]: 18.193688+0000 osd.6 (osd. 2021-02-10T04:29:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[35961]: 6) 32 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:29:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[35961]: 2021-02-10T04:29:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[35961]: cluster 2021-02-10T04:29:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:29:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[35961]: 29:18.429642+0000 osd.5 2021-02-10T04:29:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[35961]: (osd.5) 40 : cluster 2021-02-10T04:29:19.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[35961]: [DBG] 5.16 scrub ok 2021-02-10T04:29:19.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[35961]: cluster 2021-02-10T04:29:19.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:29:19.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[35961]: 29:18.565406+0000 osd.6 2021-02-10T04:29:19.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[35961]: (osd.6) 33 : cluster 2021-02-10T04:29:19.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[35961]: [DBG] 2.11 scrub ok 2021-02-10T04:29:19.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[29319]: audit 2021-02-10T04:29:17.715022+0000 2021-02-10T04:29:19.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[29319]: mgr.y (mgr.14140) 359 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:29:19.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[29319]: cluster 2021-02-10T04:29:18.001929+0000 mgr.y (mgr.14140) 360 : cluster [DBG] pgmap v361: 168 pgs: 5 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 152 active+clean; 188 MiB data, 490 MiB used, 715 GiB / 715 GiB avail; 4.6 MiB/s rd, 2.1 MiB/s wr, 13 op/s 2021-02-10T04:29:19.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[29319]: cluster 2021-02-10T04:29:18.107864+0000 osd.1 (osd.1) 42 : cluster [DBG] 2.a scrub ok 2021-02-10T04:29:19.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[29319]: cluster 2021-02-10T04:29:18.193321+0000 osd.6 (osd.6) 31 : cluster 2021-02-10T04:29:19.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[29319]: [DBG] purged_snaps scrub starts 2021-02-10T04:29:19.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[29319]: cluster 2021-02-10T04:29:18.193688+0000 osd.6 (osd 2021-02-10T04:29:19.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[29319]: .6) 32 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:29:19.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:29:19.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[29319]: -02-10T04:29:18 2021-02-10T04:29:19.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[29319]: .429642+0000 osd.5 (osd.5 2021-02-10T04:29:19.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[29319]: ) 40 : cluster [DBG] 2021-02-10T04:29:19.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[29319]: 5.16 scrub ok 2021-02-10T04:29:19.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:29:19.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[29319]: -02-10T04:29: 2021-02-10T04:29:19.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[29319]: 18.565406+0000 osd.6 (osd.6 2021-02-10T04:29:19.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:19 smithi085 conmon[29319]: ) 33 : cluster [DBG] 2.11 scrub ok 2021-02-10T04:29:19.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:19 smithi164 conmon[29288]: audit 2021-02-10T04:29:17.715022+0000 2021-02-10T04:29:19.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:19 smithi164 conmon[29288]: mgr.y (mgr.14140) 359 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:29:19.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:19 smithi164 conmon[29288]: cluster 2021-02-10T04:29:18.001929+0000 mgr.y (mgr.14140) 360 : cluster [DBG] pgmap v361: 168 pgs: 5 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 152 active+clean; 188 MiB data, 490 MiB used, 715 GiB / 715 GiB avail; 4.6 MiB/s rd, 2.1 MiB/s wr, 13 op/s 2021-02-10T04:29:19.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:19 smithi164 conmon[29288]: cluster 2021-02-10T04:29:19.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:19 smithi164 conmon[29288]: 2021-02-10T04:29:18.107864+0000 osd.1 (osd.1) 42 : cluster [DBG] 2.a scrub ok 2021-02-10T04:29:19.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:19 smithi164 conmon[29288]: cluster 2021-02-10T04:29 2021-02-10T04:29:19.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:19 smithi164 conmon[29288]: :18.193321+0000 osd.6 (osd.6) 31 : cluster 2021-02-10T04:29:19.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:19 smithi164 conmon[29288]: [DBG] purged_snaps scrub starts 2021-02-10T04:29:19.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:19 smithi164 conmon[29288]: cluster 2021-02-10T04:29:19.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:19 smithi164 conmon[29288]: 2021-02-10T04:29:18.193688 2021-02-10T04:29:19.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:19 smithi164 conmon[29288]: +0000 osd.6 (osd.6) 32 : 2021-02-10T04:29:19.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:19 smithi164 conmon[29288]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:29:19.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:19 smithi164 conmon[29288]: cluster 2021-02-10T04:29:19.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:19 smithi164 conmon[29288]: 2021-02-10T04:29:18.429642+0000 2021-02-10T04:29:19.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:19 smithi164 conmon[29288]: osd.5 (osd.5) 40 : cluster [DBG] 5.16 scrub ok 2021-02-10T04:29:19.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:19 smithi164 conmon[29288]: 2021-02-10T04:29:19.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:19 smithi164 conmon[29288]: cluster 2021-02-10T04:29:19.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:19 smithi164 conmon[29288]: 2021-02-10T04:29:18.565406+0000 osd.6 (osd.6) 33 : cluster 2021-02-10T04:29:19.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:19 smithi164 conmon[29288]: [DBG] 2.11 scrub ok 2021-02-10T04:29:19.695 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:29:19.800 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:29:19.829 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:20.078 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:29:20.091 INFO:tasks.rados.rados.0.smithi085.stdout:269: done (3 left) 2021-02-10T04:29:20.091 INFO:tasks.rados.rados.0.smithi085.stdout:270: copy_from oid 19 from oid 42 current snap is 24 2021-02-10T04:29:20.091 INFO:tasks.rados.rados.0.smithi085.stdout:271: delete oid 6 current snap is 24 2021-02-10T04:29:20.096 INFO:tasks.rados.rados.0.smithi085.stdout:271: done (4 left) 2021-02-10T04:29:20.096 INFO:tasks.rados.rados.0.smithi085.stdout:272: delete oid 31 current snap is 24 2021-02-10T04:29:20.099 INFO:tasks.rados.rados.0.smithi085.stdout:270: finishing copy_from to smithi08575397-19 2021-02-10T04:29:20.100 INFO:tasks.rados.rados.0.smithi085.stdout:270: got expected ENOENT (src dne) 2021-02-10T04:29:20.100 INFO:tasks.rados.rados.0.smithi085.stdout:270: finishing copy_from racing read to smithi08575397-19 2021-02-10T04:29:20.101 INFO:tasks.rados.rados.0.smithi085.stdout:270: done (4 left) 2021-02-10T04:29:20.101 INFO:tasks.rados.rados.0.smithi085.stdout:272: done (3 left) 2021-02-10T04:29:20.101 INFO:tasks.rados.rados.0.smithi085.stdout:273: rollback oid 22 current snap is 24 2021-02-10T04:29:20.101 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 22 to 23 2021-02-10T04:29:20.101 INFO:tasks.rados.rados.0.smithi085.stdout:274: write (excl) oid 38 current snap is 24 2021-02-10T04:29:20.101 INFO:tasks.rados.rados.0.smithi085.stdout:274: seq_num 130 ranges {584910=560371,1857365=327545} 2021-02-10T04:29:20.107 INFO:tasks.rados.rados.0.smithi085.stdout:274: writing smithi08575397-38 from 584910 to 1145281 tid 1 2021-02-10T04:29:20.114 INFO:tasks.rados.rados.0.smithi085.stdout:274: writing smithi08575397-38 from 1857365 to 2184910 tid 2 2021-02-10T04:29:20.114 INFO:tasks.rados.rados.0.smithi085.stdout:275: read oid 23 snap 23 2021-02-10T04:29:20.114 INFO:tasks.rados.rados.0.smithi085.stdout:275: expect (ObjNum 124 snap 23 seq_num 124) 2021-02-10T04:29:20.114 INFO:tasks.rados.rados.0.smithi085.stdout:273: finishing rollback tid 0 to smithi08575397-22 2021-02-10T04:29:20.114 INFO:tasks.rados.rados.0.smithi085.stdout:273: finishing rollback tid 1 to smithi08575397-22 2021-02-10T04:29:20.114 INFO:tasks.rados.rados.0.smithi085.stdout:273: finishing rollback tid 2 to smithi08575397-22 2021-02-10T04:29:20.114 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 56 (ObjNum 127 snap 23 seq_num 127) dirty exists 2021-02-10T04:29:20.114 INFO:tasks.rados.rados.0.smithi085.stdout:273: done (5 left) 2021-02-10T04:29:20.115 INFO:tasks.rados.rados.0.smithi085.stdout:276: write oid 26 current snap is 24 2021-02-10T04:29:20.115 INFO:tasks.rados.rados.0.smithi085.stdout:276: seq_num 131 ranges {601717=635620,1799494=402223} 2021-02-10T04:29:20.121 INFO:tasks.rados.rados.0.smithi085.stdout:276: writing smithi08575397-26 from 601717 to 1237337 tid 1 2021-02-10T04:29:20.127 INFO:tasks.rados.rados.0.smithi085.stdout:276: writing smithi08575397-26 from 1799494 to 2201717 tid 2 2021-02-10T04:29:20.128 INFO:tasks.rados.rados.0.smithi085.stdout:277: delete oid 24 current snap is 24 2021-02-10T04:29:20.128 INFO:tasks.rados.rados.0.smithi085.stdout:274: finishing write tid 1 to smithi08575397-38 2021-02-10T04:29:20.128 INFO:tasks.rados.rados.0.smithi085.stdout:275: expect (ObjNum 124 snap 23 seq_num 124) 2021-02-10T04:29:20.184 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:29:20.190 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:29:20 smithi085 conmon[49106]: debug 2021-02-10T04:29:20.001+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:20.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:20 smithi085 conmon[29319]: cluster 2021-02-10T04:29:19.088143+0000 2021-02-10T04:29:20.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:20 smithi085 conmon[29319]: mon.a (mon.0) 819 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2021-02-10T04:29:20.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:20 smithi085 conmon[29319]: cluster 2021-02-10T04:29:19.405520+0000 osd.5 (osd.5) 41 2021-02-10T04:29:20.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:20 smithi085 conmon[29319]: : cluster [DBG] 6.a scrub ok 2021-02-10T04:29:20.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:20 smithi085 conmon[29319]: cluster 2021-02-10T04:29:20.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:20 smithi085 conmon[29319]: 2021-02-10T04:29:19.564713+0000 osd.0 (osd.0) 2021-02-10T04:29:20.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:20 smithi085 conmon[29319]: 42 : cluster [DBG] 2021-02-10T04:29:20.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:20 smithi085 conmon[29319]: 2.13 scrub ok 2021-02-10T04:29:20.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:20 smithi085 conmon[35961]: cluster 2021-02-10T04:29:20.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:20 smithi085 conmon[35961]: 2021-02-10T04:29:20.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:20 smithi085 conmon[35961]: 2021-02-10T04:29:19.088143 2021-02-10T04:29:20.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:20 smithi085 conmon[35961]: +0000 mon.a (mon.0) 819 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2021-02-10T04:29:20.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:20 smithi085 conmon[35961]: cluster 2021-02-10T04:29:20.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:20 smithi085 conmon[35961]: 2021-02-10T04:29:19.405520 2021-02-10T04:29:20.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:20 smithi085 conmon[35961]: +0000 osd.5 (osd.5) 41 : cluster [DBG] 2021-02-10T04:29:20.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:20 smithi085 conmon[35961]: 6.a scrub ok 2021-02-10T04:29:20.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:20 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:29:20.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:20 smithi085 conmon[35961]: 10T04:29:19.564713+0000 osd.0 (osd.0) 2021-02-10T04:29:20.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:20 smithi085 conmon[35961]: 42 : cluster [DBG] 2.13 scrub ok 2021-02-10T04:29:20.210 INFO:tasks.rados.rados.0.smithi085.stdout:274: finishing write tid 2 to smithi08575397-38 2021-02-10T04:29:20.210 INFO:tasks.rados.rados.0.smithi085.stdout:274: finishing write tid 3 to smithi08575397-38 2021-02-10T04:29:20.210 INFO:tasks.rados.rados.0.smithi085.stdout:274: finishing write tid 4 to smithi08575397-38 2021-02-10T04:29:20.210 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 81 (ObjNum 130 snap 24 seq_num 130) dirty exists 2021-02-10T04:29:20.211 INFO:tasks.rados.rados.0.smithi085.stdout:274: left oid 38 (ObjNum 130 snap 24 seq_num 130) 2021-02-10T04:29:20.211 INFO:tasks.rados.rados.0.smithi085.stdout:276: finishing write tid 1 to smithi08575397-26 2021-02-10T04:29:20.211 INFO:tasks.rados.rados.0.smithi085.stdout:276: finishing write tid 2 to smithi08575397-26 2021-02-10T04:29:20.211 INFO:tasks.rados.rados.0.smithi085.stdout:276: finishing write tid 3 to smithi08575397-26 2021-02-10T04:29:20.211 INFO:tasks.rados.rados.0.smithi085.stdout:276: finishing write tid 4 to smithi08575397-26 2021-02-10T04:29:20.211 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 18 (ObjNum 131 snap 24 seq_num 131) dirty exists 2021-02-10T04:29:20.211 INFO:tasks.rados.rados.0.smithi085.stdout:276: left oid 26 (ObjNum 131 snap 24 seq_num 131) 2021-02-10T04:29:20.211 INFO:tasks.rados.rados.0.smithi085.stdout:274: done (6 left) 2021-02-10T04:29:20.211 INFO:tasks.rados.rados.0.smithi085.stdout:275: done (5 left) 2021-02-10T04:29:20.211 INFO:tasks.rados.rados.0.smithi085.stdout:276: done (4 left) 2021-02-10T04:29:20.211 INFO:tasks.rados.rados.0.smithi085.stdout:277: done (3 left) 2021-02-10T04:29:20.211 INFO:tasks.rados.rados.0.smithi085.stdout:278: copy_from oid 47 from oid 46 current snap is 24 2021-02-10T04:29:20.211 INFO:tasks.rados.rados.0.smithi085.stdout:279: snap_create 2021-02-10T04:29:20.221 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:20.232 INFO:tasks.rados.rados.0.smithi085.stdout:278: finishing copy_from to smithi08575397-47 2021-02-10T04:29:20.232 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 47 v 75 (ObjNum 118 snap 23 seq_num 118) dirty exists 2021-02-10T04:29:20.233 INFO:tasks.rados.rados.0.smithi085.stdout:278: finishing copy_from racing read to smithi08575397-47 2021-02-10T04:29:20.303 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:29:20.339 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:29:20.339 INFO:teuthology.orchestra.run.smithi085.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2021-02-10T04:29:20.339 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:29:20.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:20 smithi164 conmon[29288]: cluster 2021-02-10T04:29:19.088143 2021-02-10T04:29:20.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:20 smithi164 conmon[29288]: +0000 mon.a (mon.0) 819 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2021-02-10T04:29:20.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:20 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:29:20.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:20 smithi164 conmon[29288]: 04:29:19.405520+0000 osd.5 (osd.5) 41 : cluster [DBG] 2021-02-10T04:29:20.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:20 smithi164 conmon[29288]: 6.a scrub ok 2021-02-10T04:29:20.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:20 smithi164 conmon[29288]: cluster 2021-02-10T04:29:19.564713+0000 2021-02-10T04:29:20.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:20 smithi164 conmon[29288]: osd.0 (osd.0) 42 : cluster [DBG] 2.13 scrub ok 2021-02-10T04:29:20.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:29:20 smithi164 conmon[33677]: debug 2021-02-10T04:29:20.283+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:20.408 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:29:20.444 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:21.103 INFO:tasks.rados.rados.0.smithi085.stdout:278: done (4 left) 2021-02-10T04:29:21.103 INFO:tasks.rados.rados.0.smithi085.stdout:279: done (3 left) 2021-02-10T04:29:21.103 INFO:tasks.rados.rados.0.smithi085.stdout:280: delete oid 21 current snap is 25 2021-02-10T04:29:21.110 INFO:tasks.rados.rados.0.smithi085.stdout:280: done (3 left) 2021-02-10T04:29:21.110 INFO:tasks.rados.rados.0.smithi085.stdout:281: write (excl) oid 30 current snap is 25 2021-02-10T04:29:21.110 INFO:tasks.rados.rados.0.smithi085.stdout:281: seq_num 134 ranges {652138=461367,1574940=677198} 2021-02-10T04:29:21.116 INFO:tasks.rados.rados.0.smithi085.stdout:281: writing smithi08575397-30 from 652138 to 1113505 tid 1 2021-02-10T04:29:21.123 INFO:tasks.rados.rados.0.smithi085.stdout:281: writing smithi08575397-30 from 1574940 to 2252138 tid 2 2021-02-10T04:29:21.124 INFO:tasks.rados.rados.0.smithi085.stdout:282: delete oid 34 current snap is 25 2021-02-10T04:29:21.124 INFO:tasks.rados.rados.0.smithi085.stdout:281: finishing write tid 1 to smithi08575397-30 2021-02-10T04:29:21.125 INFO:tasks.rados.rados.0.smithi085.stdout:282: done (4 left) 2021-02-10T04:29:21.125 INFO:tasks.rados.rados.0.smithi085.stdout:283: snap_remove snap 10 2021-02-10T04:29:21.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:21 smithi164 conmon[29288]: cluster 2021-02-10T04:29:20.002343 2021-02-10T04:29:21.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:21 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 361 : cluster [DBG] pgmap v364: 168 pgs: 5 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 152 active+clean; 192 MiB data, 498 MiB used, 715 GiB / 715 GiB avail; 8.5 MiB/s rd, 1.7 MiB/s wr, 15 op/s 2021-02-10T04:29:21.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:21 smithi164 conmon[29288]: cluster 2021-02-10T04:29:20.090596+0000 mon.a ( 2021-02-10T04:29:21.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:21 smithi164 conmon[29288]: mon.0) 820 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2021-02-10T04:29:21.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:21 smithi085 conmon[29319]: cluster 2021-02-10T04:29:20.002343+0000 mgr.y (mgr.14140) 361 : cluster [DBG] pgmap v364: 168 pgs: 5 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 152 active+clean; 192 MiB data, 498 MiB used, 715 GiB / 715 GiB avail; 8.5 MiB/s rd, 1.7 MiB/s wr, 15 op/s 2021-02-10T04:29:21.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:21 smithi085 conmon[29319]: cluster 2021-02-10T04:29:20.090596+0000 mon.a (mon.0) 820 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2021-02-10T04:29:21.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:21 smithi085 conmon[35961]: cluster 2021-02-10T04:29:20.002343 2021-02-10T04:29:21.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:21 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 361 : cluster [DBG] pgmap v364: 168 pgs: 5 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 152 active+clean; 192 MiB data, 498 MiB used, 715 GiB / 715 GiB avail; 8.5 MiB/s rd, 1.7 MiB/s wr, 15 op/s 2021-02-10T04:29:21.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:21 smithi085 conmon[35961]: cluster 2021-02-10T04:29:20.090596+0000 mon.a (mon.0) 820 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2021-02-10T04:29:21.658 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset nodeep-scrub 2021-02-10T04:29:21.902 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:29:22.045 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:29:22.046 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:29:22.108 INFO:tasks.rados.rados.0.smithi085.stdout:283: done (4 left) 2021-02-10T04:29:22.108 INFO:tasks.rados.rados.0.smithi085.stdout:284: read oid 14 snap 23 2021-02-10T04:29:22.108 INFO:tasks.rados.rados.0.smithi085.stdout:284: expect (ObjNum 13 snap 0 seq_num 13) 2021-02-10T04:29:22.108 INFO:tasks.rados.rados.0.smithi085.stdout:285: snap_create 2021-02-10T04:29:22.108 INFO:tasks.rados.rados.0.smithi085.stdout:281: finishing write tid 2 to smithi08575397-30 2021-02-10T04:29:22.108 INFO:tasks.rados.rados.0.smithi085.stdout:281: finishing write tid 3 to smithi08575397-30 2021-02-10T04:29:22.108 INFO:tasks.rados.rados.0.smithi085.stdout:281: finishing write tid 4 to smithi08575397-30 2021-02-10T04:29:22.108 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 84 (ObjNum 134 snap 25 seq_num 134) dirty exists 2021-02-10T04:29:22.108 INFO:tasks.rados.rados.0.smithi085.stdout:281: left oid 30 (ObjNum 134 snap 25 seq_num 134) 2021-02-10T04:29:22.118 INFO:tasks.rados.rados.0.smithi085.stdout:284: expect (ObjNum 13 snap 0 seq_num 13) 2021-02-10T04:29:22.348 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[29319]: cluster 2021-02-10T04:29:20.757188+0000 osd.2 (osd.2) 35 : cluster 2021-02-10T04:29:22.348 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[29319]: [DBG] 5.8 scrub ok 2021-02-10T04:29:22.348 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[29319]: cluster 2021-02-10T04:29:21.103024+0000 mon.a (mon.0) 821 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2021-02-10T04:29:22.348 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[29319]: cluster 2021-02-10T04:29:21.620479+0000 osd.0 (osd.0) 43 : cluster [DBG] 3.1 scrub ok 2021-02-10T04:29:22.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: cluster 2021-02-10T04:29:22.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: 2021-02-10T04:29:22.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: 2021 2021-02-10T04:29:22.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: - 2021-02-10T04:29:22.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: 02 2021-02-10T04:29:22.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: - 2021-02-10T04:29:22.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: 10 2021-02-10T04:29:22.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: T 2021-02-10T04:29:22.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: 04 2021-02-10T04:29:22.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: : 2021-02-10T04:29:22.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: 29 2021-02-10T04:29:22.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: : 2021-02-10T04:29:22.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: 20 2021-02-10T04:29:22.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: . 2021-02-10T04:29:22.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: 757188 2021-02-10T04:29:22.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: +0000 2021-02-10T04:29:22.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: 2021-02-10T04:29:22.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: osd.2 2021-02-10T04:29:22.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: ( 2021-02-10T04:29:22.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: osd 2021-02-10T04:29:22.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: . 2021-02-10T04:29:22.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: 2 2021-02-10T04:29:22.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: ) 2021-02-10T04:29:22.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: 35 2021-02-10T04:29:22.351 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: : 2021-02-10T04:29:22.351 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: cluster 2021-02-10T04:29:22.351 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: 2021-02-10T04:29:22.351 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: [DBG] 2021-02-10T04:29:22.351 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: 2021-02-10T04:29:22.351 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: 5.8 scrub ok 2021-02-10T04:29:22.351 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: 2021-02-10T04:29:22.351 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: cluster 2021-02-10T04:29:22.351 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: 2021-02-10T04:29:22.351 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: 2021 2021-02-10T04:29:22.351 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: -02-10T04: 2021-02-10T04:29:22.351 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: 29:21.103024+0000 mon.a (mon.0) 821 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2021-02-10T04:29:22.351 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:29:22.351 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: 02-10T04:29:21.620479+0000 osd.0 (osd 2021-02-10T04:29:22.352 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[35961]: .0) 43 : cluster [DBG] 3.1 scrub ok 2021-02-10T04:29:22.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:22 smithi164 conmon[29288]: cluster 2021-02-10T04:29:20.757188+0000 osd.2 (osd.2) 35 2021-02-10T04:29:22.573 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:22 smithi164 conmon[29288]: : cluster [DBG] 5.8 scrub ok 2021-02-10T04:29:22.573 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:22 smithi164 conmon[29288]: cluster 2021-02-10T04:29:21.103024+0000 mon.a (mon.0) 821 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2021-02-10T04:29:22.573 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:22 smithi164 conmon[29288]: cluster 2021-02-10T04:29: 2021-02-10T04:29:22.573 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:22 smithi164 conmon[29288]: 21.620479+0000 osd.0 (osd.0) 43 : cluster [DBG] 3.1 scrub ok 2021-02-10T04:29:22.573 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:29:22 smithi164 conmon[33677]: debug 2021-02-10T04:29:22.107+0000 7fbeaa12d700 -1 osd.4 pg_epoch: 146 pg[7.1( v 142'48 (0'0,142'48] local-lis/les=93/94 n=4 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [4,3,6] r=0 lpr=93 crt=142'48 lcod 142'46 mlcod 142'46 active+clean+snaptrim trimq=[15~1] ps=[17~1,19~1,1c~3,20~1,24~1,28~1]] removing snap head 2021-02-10T04:29:22.669 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:29:22.690 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:29:22 smithi085 conmon[49106]: debug 2021-02-10T04:29:22.371+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:23.040 INFO:tasks.rados.rados.0.smithi085.stdout:281: done (5 left) 2021-02-10T04:29:23.041 INFO:tasks.rados.rados.0.smithi085.stdout:284: done (4 left) 2021-02-10T04:29:23.041 INFO:tasks.rados.rados.0.smithi085.stdout:285: done (3 left) 2021-02-10T04:29:23.041 INFO:tasks.rados.rados.0.smithi085.stdout:286: read oid 43 snap 24 2021-02-10T04:29:23.041 INFO:tasks.rados.rados.0.smithi085.stdout:286: expect deleted 2021-02-10T04:29:23.041 INFO:tasks.rados.rados.0.smithi085.stdout:287: read oid 11 snap -1 2021-02-10T04:29:23.041 INFO:tasks.rados.rados.0.smithi085.stdout:287: expect deleted 2021-02-10T04:29:23.041 INFO:tasks.rados.rados.0.smithi085.stdout:288: copy_from oid 47 from oid 16 current snap is 26 2021-02-10T04:29:23.041 INFO:tasks.rados.rados.0.smithi085.stdout:289: write (excl) oid 44 current snap is 26 2021-02-10T04:29:23.041 INFO:tasks.rados.rados.0.smithi085.stdout:289: seq_num 136 ranges {685752=611865,1975551=310201} 2021-02-10T04:29:23.048 INFO:tasks.rados.rados.0.smithi085.stdout:289: writing smithi08575397-44 from 685752 to 1297617 tid 1 2021-02-10T04:29:23.055 INFO:tasks.rados.rados.0.smithi085.stdout:289: writing smithi08575397-44 from 1975551 to 2285752 tid 2 2021-02-10T04:29:23.055 INFO:tasks.rados.rados.0.smithi085.stdout:290: read oid 15 snap -1 2021-02-10T04:29:23.055 INFO:tasks.rados.rados.0.smithi085.stdout:290: expect (ObjNum 50 snap 0 seq_num 50) 2021-02-10T04:29:23.055 INFO:tasks.rados.rados.0.smithi085.stdout:291: rollback oid 25 current snap is 26 2021-02-10T04:29:23.056 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 25 to 23 2021-02-10T04:29:23.056 INFO:tasks.rados.rados.0.smithi085.stdout:292: copy_from oid 17 from oid 50 current snap is 26 2021-02-10T04:29:23.056 INFO:tasks.rados.rados.0.smithi085.stdout:293: delete oid 46 current snap is 26 2021-02-10T04:29:23.056 INFO:tasks.rados.rados.0.smithi085.stdout:289: finishing write tid 1 to smithi08575397-44 2021-02-10T04:29:23.056 INFO:tasks.rados.rados.0.smithi085.stdout:288: finishing copy_from to smithi08575397-47 2021-02-10T04:29:23.056 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 47 v 110 (ObjNum 126 snap 23 seq_num 126) dirty exists 2021-02-10T04:29:23.057 INFO:tasks.rados.rados.0.smithi085.stdout:288: finishing copy_from racing read to smithi08575397-47 2021-02-10T04:29:23.057 INFO:tasks.rados.rados.0.smithi085.stdout:286: done (10 left) 2021-02-10T04:29:23.057 INFO:tasks.rados.rados.0.smithi085.stdout:287: done (9 left) 2021-02-10T04:29:23.058 INFO:tasks.rados.rados.0.smithi085.stdout:288: done (8 left) 2021-02-10T04:29:23.058 INFO:tasks.rados.rados.0.smithi085.stdout:293: done (7 left) 2021-02-10T04:29:23.058 INFO:tasks.rados.rados.0.smithi085.stdout:294: snap_create 2021-02-10T04:29:23.058 INFO:tasks.rados.rados.0.smithi085.stdout:291: finishing rollback tid 0 to smithi08575397-25 2021-02-10T04:29:23.059 INFO:tasks.rados.rados.0.smithi085.stdout:289: finishing write tid 2 to smithi08575397-44 2021-02-10T04:29:23.060 INFO:tasks.rados.rados.0.smithi085.stdout:289: finishing write tid 3 to smithi08575397-44 2021-02-10T04:29:23.060 INFO:tasks.rados.rados.0.smithi085.stdout:289: finishing write tid 4 to smithi08575397-44 2021-02-10T04:29:23.060 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 55 (ObjNum 136 snap 26 seq_num 136) dirty exists 2021-02-10T04:29:23.060 INFO:tasks.rados.rados.0.smithi085.stdout:289: left oid 44 (ObjNum 136 snap 26 seq_num 136) 2021-02-10T04:29:23.061 INFO:tasks.rados.rados.0.smithi085.stdout:291: finishing rollback tid 1 to smithi08575397-25 2021-02-10T04:29:23.061 INFO:tasks.rados.rados.0.smithi085.stdout:291: finishing rollback tid 2 to smithi08575397-25 2021-02-10T04:29:23.061 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 25 v 45 (ObjNum 87 snap 11 seq_num 87) dirty exists 2021-02-10T04:29:23.063 INFO:tasks.rados.rados.0.smithi085.stdout:290: expect (ObjNum 50 snap 0 seq_num 50) 2021-02-10T04:29:23.240 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:29:23.266 INFO:tasks.rados.rados.0.smithi085.stdout:292: finishing copy_from to smithi08575397-17 2021-02-10T04:29:23.266 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 17 v 47 (ObjNum 10 snap 0 seq_num 10) dirty exists 2021-02-10T04:29:23.266 INFO:tasks.rados.rados.0.smithi085.stdout:292: finishing copy_from racing read to smithi08575397-17 2021-02-10T04:29:23.268 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:23.361 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:29:23.364 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:29:23 smithi164 conmon[44544]: debug 2021-02-10T04:29:23.331+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:23.364 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:23 smithi164 conmon[29288]: cluster 2021-02-10T04:29:21.708695 2021-02-10T04:29:23.364 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:23 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 36 : cluster [DBG] 2.21 scrub ok 2021-02-10T04:29:23.364 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:23 smithi164 conmon[29288]: cluster 2021-02-10T04:29:22.002807+0000 mgr.y ( 2021-02-10T04:29:23.364 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:23 smithi164 conmon[29288]: mgr.14140) 362 : cluster [DBG] pgmap v367: 168 pgs: 5 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 153 active+clean; 197 MiB data, 524 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 3.4 MiB/s wr, 28 op/s 2021-02-10T04:29:23.365 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:23 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:29:23.365 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:23 smithi164 conmon[29288]: -02-10T04:29:22.078243+0000 osd.1 (osd.1) 43 : cluster [DBG] 5.1d scrub ok 2021-02-10T04:29:23.365 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:23 smithi164 conmon[29288]: cluster 2021-02-10T04:29:22. 2021-02-10T04:29:23.365 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:23 smithi164 conmon[29288]: 108061+0000 mon.a (mon.0) 822 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2021-02-10T04:29:23.365 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:23 smithi164 conmon[29288]: cluster 2021-02-10T04:29: 2021-02-10T04:29:23.365 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:23 smithi164 conmon[29288]: 22.475497+0000 2021-02-10T04:29:23.365 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:23 smithi164 conmon[29288]: osd.5 (osd.5) 42 : cluster [DBG] 6.19 scrub ok 2021-02-10T04:29:23.365 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:23 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:29:23.365 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:23 smithi164 conmon[29288]: :29:23.040159+0000 mon.a (mon.0) 823 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2021-02-10T04:29:23.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[29319]: cluster 2021-02-10T04:29:23.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:29:23.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[29319]: 04:29:21.708695+0000 osd.2 (osd.2) 36 : cluster [DBG] 2.21 scrub ok 2021-02-10T04:29:23.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[29319]: cluster 2021-02-10T04:29:22.002807+0000 2021-02-10T04:29:23.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[29319]: mgr.y (mgr.14140) 362 : cluster [DBG] pgmap v367: 168 pgs: 5 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 153 active+clean; 197 MiB data, 524 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 3.4 MiB/s wr, 28 op/s 2021-02-10T04:29:23.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:29:23.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[29319]: -02-10T04:29:22.078243+0000 osd.1 ( 2021-02-10T04:29:23.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[29319]: osd.1) 43 : cluster [DBG] 2021-02-10T04:29:23.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[29319]: 5.1d scrub ok 2021-02-10T04:29:23.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:29:23.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[29319]: 02-10T04:29:22.108061+0000 mon.a (mon 2021-02-10T04:29:23.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[29319]: .0) 822 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2021-02-10T04:29:23.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:29:23.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[29319]: -02-10T04:29:22.475497+0000 2021-02-10T04:29:23.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[29319]: osd.5 (osd.5) 42 : cluster [DBG] 6.19 scrub ok 2021-02-10T04:29:23.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[29319]: 2021-02-10T04:29:23.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:29:23.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[29319]: :29:23.040159+0000 mon.a (mon.0) 2021-02-10T04:29:23.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[29319]: 823 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2021-02-10T04:29:23.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[35961]: cluster 2021-02-10T04:29:23.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[35961]: 2021-02-10T04:29:23.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[35961]: 2021 2021-02-10T04:29:23.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[35961]: -02-10T04:29:21. 2021-02-10T04:29:23.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[35961]: 708695+0000 osd.2 (osd.2) 36 : cluster [DBG] 2.21 scrub ok 2021-02-10T04:29:23.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:29:23.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[35961]: 02-10T04:29:22.002807+0000 mgr.y (mgr 2021-02-10T04:29:23.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[35961]: .14140) 362 : cluster [DBG] pgmap v367: 168 pgs: 5 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 153 active+clean; 197 MiB data, 524 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 3.4 MiB/s wr, 28 op/s 2021-02-10T04:29:23.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[35961]: cluster 2021-02-10T04:29:23.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:29:23.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[35961]: 04:29:22.078243+0000 osd.1 (osd. 2021-02-10T04:29:23.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[35961]: 1) 43 : cluster [DBG] 5.1d scrub ok 2021-02-10T04:29:23.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:29:23.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[35961]: 02-10T04:29:22.108061+0000 mon.a (mon.0) 822 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2021-02-10T04:29:23.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[35961]: cluster 2021-02-10T04:29:23.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[35961]: 2021 2021-02-10T04:29:23.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[35961]: -02-10T04:29:22. 2021-02-10T04:29:23.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[35961]: 475497+0000 osd.5 (osd.5) 42 : cluster [DBG] 6.19 scrub ok 2021-02-10T04:29:23.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[35961]: cluster 2021-02-10T04:29:23.040159+0000 2021-02-10T04:29:23.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[35961]: mon.a (mon.0) 823 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2021-02-10T04:29:23.467 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:29:23.495 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:23.959 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.3 dump_blocked_ops 2021-02-10T04:29:24.003 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:29:24.005 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:29:23 smithi085 conmon[52836]: debug 2021-02-10T04:29:23.976+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:24.047 INFO:tasks.rados.rados.0.smithi085.stdout:289: done (7 left) 2021-02-10T04:29:24.048 INFO:tasks.rados.rados.0.smithi085.stdout:290: done (6 left) 2021-02-10T04:29:24.048 INFO:tasks.rados.rados.0.smithi085.stdout:291: done (5 left) 2021-02-10T04:29:24.048 INFO:tasks.rados.rados.0.smithi085.stdout:292: done (4 left) 2021-02-10T04:29:24.048 INFO:tasks.rados.rados.0.smithi085.stdout:294: done (3 left) 2021-02-10T04:29:24.048 INFO:tasks.rados.rados.0.smithi085.stdout:295: snap_remove snap 26 2021-02-10T04:29:24.107 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:29:24.112 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:24 smithi164 conmon[29288]: cluster 2021-02-10T04:29:22.801925+0000 osd.2 (osd.2) 37 : cluster [DBG] 2021-02-10T04:29:24.112 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:24 smithi164 conmon[29288]: 2.26 scrub ok 2021-02-10T04:29:24.112 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:24 smithi164 conmon[29288]: cluster 2021-02-10T04:29:23.098672+0000 osd.1 (osd.1) 44 : cluster [DBG] 6.4 scrub ok 2021-02-10T04:29:24.112 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:24 smithi164 conmon[29288]: cluster 2021-02-10T04:29:23.440386+0000 osd.3 (osd.3) 39 : cluster [DBG] 2.28 scrub ok 2021-02-10T04:29:24.113 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:24 smithi164 conmon[29288]: cluster 2021-02-10T04:29:23.589155+0000 osd.0 (osd.0) 44 : cluster [DBG] 5.1e scrub ok 2021-02-10T04:29:24.113 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:24 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:29:24.113 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:24 smithi164 conmon[29288]: :29:23.934073+0000 osd.4 (osd.4) 40 : cluster [DBG] 6.17 scrub ok 2021-02-10T04:29:24.113 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:24 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:29:24.113 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:24 smithi164 conmon[29288]: 10T04:29:24.003270+0000 mgr.y (mgr.14140 2021-02-10T04:29:24.113 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:24 smithi164 conmon[29288]: ) 363 : cluster [DBG] pgmap v370: 168 pgs: 5 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 153 active+clean; 189 MiB data, 516 MiB used, 715 GiB / 715 GiB avail; 7.3 MiB/s rd, 2.0 MiB/s wr, 18 op/s 2021-02-10T04:29:24.113 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:24 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:29:24.113 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:24 smithi164 conmon[29288]: -10T04:29:24.047938+0000 mon.a (mon.0) 2021-02-10T04:29:24.113 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:24 smithi164 conmon[29288]: 824 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2021-02-10T04:29:24.135 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:24.220 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:29:24.266 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:29:24.323 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:24 smithi085 conmon[29319]: cluster 2021-02-10T04:29:22.801925+0000 osd.2 (osd. 2021-02-10T04:29:24.323 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:24 smithi085 conmon[29319]: 2) 37 : cluster [DBG] 2.26 scrub ok 2021-02-10T04:29:24.323 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:24 smithi085 conmon[29319]: cluster 2021-02-10T04:29:23.098672+0000 osd.1 (osd.1) 44 : cluster [DBG] 6.4 scrub ok 2021-02-10T04:29:24.323 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:24 smithi085 conmon[29319]: cluster 2021-02-10T04:29:23.440386+0000 osd.3 (osd.3) 39 : cluster [DBG] 2.28 scrub ok 2021-02-10T04:29:24.323 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:24 smithi085 conmon[29319]: cluster 2021-02-10T04:29:23.589155+0000 osd.0 (osd.0) 44 : cluster [DBG] 5.1e scrub ok 2021-02-10T04:29:24.323 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:24 smithi085 conmon[29319]: cluster 2021-02-10T04:29:23.934073+0000 osd.4 (osd.4) 40 : cluster [DBG] 6.17 scrub ok 2021-02-10T04:29:24.323 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:24 smithi085 conmon[29319]: cluster 2021-02-10T04:29:24.003270+0000 mgr.y (mgr.14140) 363 : cluster [DBG] pgmap v370: 168 pgs: 5 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 153 active+clean; 189 MiB data, 516 MiB used, 715 GiB / 715 GiB avail; 7.3 MiB/s rd, 2.0 MiB/s wr, 18 op/s 2021-02-10T04:29:24.323 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:24 smithi085 conmon[29319]: cluster 2021-02-10T04:29:24.047938+0000 mon.a (mon.0) 824 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2021-02-10T04:29:24.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:24 smithi085 conmon[35961]: cluster 2021-02-10T04:29:22 2021-02-10T04:29:24.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:24 smithi085 conmon[35961]: .801925+0000 osd.2 (osd.2) 37 : cluster [DBG] 2.26 scrub ok 2021-02-10T04:29:24.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:24 smithi085 conmon[35961]: cluster 2021-02-10T04:29:23.098672+0000 osd.1 (osd.1) 44 : cluster [DBG] 6.4 scrub ok 2021-02-10T04:29:24.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:24 smithi085 conmon[35961]: cluster 2021-02-10T04:29:23.440386+0000 osd.3 (osd.3) 39 : cluster [DBG] 2.28 scrub ok 2021-02-10T04:29:24.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:24 smithi085 conmon[35961]: cluster 2021-02-10T04:29:23.589155+0000 osd.0 (osd.0) 44 : cluster [DBG] 5.1e scrub ok 2021-02-10T04:29:24.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:24 smithi085 conmon[35961]: cluster 2021-02-10T04:29:23.934073+0000 osd.4 (osd.4) 40 : cluster [DBG] 6.17 scrub ok 2021-02-10T04:29:24.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:24 smithi085 conmon[35961]: cluster 2021-02-10T04:29:24.003270+0000 mgr.y (mgr.14140) 363 : cluster [DBG] pgmap v370: 168 pgs: 5 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 153 active+clean; 189 MiB data, 516 MiB used, 715 GiB / 715 GiB avail; 7.3 MiB/s rd, 2.0 MiB/s wr, 18 op/s 2021-02-10T04:29:24.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:24 smithi085 conmon[35961]: cluster 2021-02-10T04:29:24.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:24 smithi085 conmon[35961]: 2021-02-10T04:29:24.047938+0000 mon.a (mon.0) 824 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2021-02-10T04:29:24.326 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:29:24.354 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:24.393 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:29:24 smithi164 conmon[37263]: debug 2021-02-10T04:29:24.200+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:24.453 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:29:24.560 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:29:24.590 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:24.893 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:29:24 smithi164 conmon[40892]: debug 2021-02-10T04:29:24.417+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:25.050 INFO:tasks.rados.rados.0.smithi085.stdout:295: done (3 left) 2021-02-10T04:29:25.050 INFO:tasks.rados.rados.0.smithi085.stdout:296: delete oid 8 current snap is 27 2021-02-10T04:29:25.068 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:29:25.068 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:29:25.072 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:29:25.077 INFO:tasks.rados.rados.0.smithi085.stdout:296: done (3 left) 2021-02-10T04:29:25.077 INFO:tasks.rados.rados.0.smithi085.stdout:297: snap_remove snap 24 2021-02-10T04:29:25.104 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:29:25.130 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:29:25.159 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:29:25.184 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:29:25.206 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:29:25.207 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:29:25.207 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:29:25.207 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:29:25.207 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:29:25.207 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:29:25.207 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:29:25.207 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:29:25.207 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:29:25.393 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:29:25 smithi164 conmon[40892]: debug 2021-02-10T04:29:25.101+0000 7f30354c5700 -1 osd.6 pg_epoch: 149 pg[7.7( v 149'124 (0'0,149'124] local-lis/les=93/94 n=14 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [6,0,3] r=0 lpr=93 luod=147'122 crt=149'124 lcod 147'120 mlcod 147'120 active+clean+snaptrim trimq=[15~1,1c~3,20~1,24~1,28~1,33~1] ps=[19~1]] removing snap head 2021-02-10T04:29:25.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:25 smithi164 conmon[29288]: cluster 2021-02-10T04:29:23 2021-02-10T04:29:25.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:25 smithi164 conmon[29288]: .645687+0000 osd.6 (osd.6) 34 : cluster [DBG] 5.11 scrub ok 2021-02-10T04:29:25.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:25 smithi164 conmon[29288]: cluster 2021-02-10T04:29:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:25 smithi164 conmon[29288]: 2021-02-10T04:29:23.779764+0000 2021-02-10T04:29:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:25 smithi164 conmon[29288]: osd.2 (osd.2) 38 : 2021-02-10T04:29:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:25 smithi164 conmon[29288]: cluster [DBG] 5.6 scrub ok 2021-02-10T04:29:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:25 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:29:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:25 smithi164 conmon[29288]: 02-10T04:29:24. 2021-02-10T04:29:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:25 smithi164 conmon[29288]: 421706+0000 osd.3 (osd.3) 2021-02-10T04:29:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:25 smithi164 conmon[29288]: 40 : cluster [DBG] 6.11 scrub ok 2021-02-10T04:29:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:25 smithi164 conmon[29288]: audit 2021-02-10T04:29:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:25 smithi164 conmon[29288]: 2021-02-10T04:29:24.925261 2021-02-10T04:29:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:25 smithi164 conmon[29288]: +0000 mon.c (mon.1) 43 : audit [DBG] 2021-02-10T04:29:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:25 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/1803486755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:29:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:25 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:29:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:25 smithi164 conmon[29288]: 10T04:29:25.050135 2021-02-10T04:29:25.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:25 smithi164 conmon[29288]: +0000 mon.a (mon.0) 825 2021-02-10T04:29:25.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:25 smithi164 conmon[29288]: : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2021-02-10T04:29:25.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:25 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:29:25.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:25 smithi164 conmon[29288]: -02-10T04:29:25 2021-02-10T04:29:25.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:25 smithi164 conmon[29288]: .073368+0000 osd.3 (osd.3 2021-02-10T04:29:25.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:25 smithi164 conmon[29288]: ) 41 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:29:25.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:25 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:29:25.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:25 smithi164 conmon[29288]: -10T04:29:25.073824+0000 osd.3 ( 2021-02-10T04:29:25.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:25 smithi164 conmon[29288]: osd.3) 42 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:29:25.419 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:29:25.441 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[52836]: debug 2021-02-10T04:29:25.150+0000 7fba70fb0700 -1 osd.3 pg_epoch: 149 pg[7.a( v 145'50 (0'0,145'50] local-lis/les=93/94 n=6 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [3,2,5] r=0 lpr=93 crt=145'50 lcod 145'48 mlcod 145'48 active+clean+snaptrim trimq=[15~1,1d~2,20~1,24~1,28~1,33~1] ps=[19~1,1c~1]] removing snap head 2021-02-10T04:29:25.441 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[45443]: debug 2021-02-10T04:29:25.110+0000 7f71264f9700 -1 osd.1 pg_epoch: 149 pg[7.b( v 149'78 (0'0,149'78] local-lis/les=93/94 n=7 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [1,5,6] r=0 lpr=93 luod=147'76 crt=149'78 lcod 144'74 mlcod 144'74 active+clean+snaptrim trimq=[15~1,1d~2,24~1,28~1,33~1] ps=[19~1,1c~1,20~1]] removing snap head 2021-02-10T04:29:25.441 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[45443]: debug 2021-02-10T04:29:25.111+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:25.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[29319]: cluster 2021-02-10T04:29:23.645687+0000 osd.6 (osd.6 2021-02-10T04:29:25.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[29319]: ) 34 : cluster [DBG] 5.11 scrub ok 2021-02-10T04:29:25.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[29319]: cluster 2021-02-10T04:29:25.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[29319]: 2021-02-10T04:29 2021-02-10T04:29:25.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[29319]: :23.779764+0000 osd.2 (osd.2) 38 : 2021-02-10T04:29:25.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[29319]: cluster [DBG] 5.6 scrub ok 2021-02-10T04:29:25.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[29319]: cluster 2021-02-10T04:29:25.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[29319]: 2021-02-10T04:29:25.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[29319]: 2021 2021-02-10T04:29:25.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[29319]: - 2021-02-10T04:29:25.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[29319]: 02 2021-02-10T04:29:25.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[29319]: - 2021-02-10T04:29:25.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[29319]: 10T04:29:24.421706+0000 osd.3 (osd. 2021-02-10T04:29:25.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[29319]: 3) 40 : cluster [DBG] 6.11 scrub ok 2021-02-10T04:29:25.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:29:25.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[29319]: T04:29:24.925261+0000 mon.c (mon 2021-02-10T04:29:25.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[29319]: .1) 43 : audit [DBG] from='client.? 172.21.15.85:0/1803486755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:29:25.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[29319]: cluster 2021-02-10T04:29:25.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[29319]: 2021-02-10T04:29:25.050135+0000 mon.a (mon. 2021-02-10T04:29:25.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[29319]: 0) 825 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2021-02-10T04:29:25.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:29:25.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[29319]: -02-10T04:29:25.073368 2021-02-10T04:29:25.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[29319]: +0000 osd.3 (osd.3) 41 : cluster [DBG] 2021-02-10T04:29:25.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[29319]: purged_snaps scrub starts 2021-02-10T04:29:25.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:29:25.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[29319]: 10T04:29:25.073824+0000 2021-02-10T04:29:25.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[29319]: osd.3 (osd.3) 42 : cluster [DBG] 2021-02-10T04:29:25.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[29319]: purged_snaps scrub ok 2021-02-10T04:29:25.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[35961]: cluster 2021-02-10T04:29:25.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[35961]: 2021-02-10T04:29 2021-02-10T04:29:25.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[35961]: :23.645687+0000 osd.6 (osd.6) 34 : cluster [DBG] 5.11 scrub ok 2021-02-10T04:29:25.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[35961]: cluster 2021-02-10T04:29:25.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[35961]: 2021-02-10T04:29:23 2021-02-10T04:29:25.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[35961]: .779764+0000 osd.2 (osd.2) 38 : cluster 2021-02-10T04:29:25.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[35961]: [DBG] 5.6 scrub ok 2021-02-10T04:29:25.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:29:25.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[35961]: -10T04:29:24.421706+0000 osd.3 ( 2021-02-10T04:29:25.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[35961]: osd.3) 40 : cluster [DBG] 6.11 scrub ok 2021-02-10T04:29:25.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[35961]: audit 2021-02-10T04:29:25.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[35961]: 2021-02-10T04:29:25.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[35961]: 2021 2021-02-10T04:29:25.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[35961]: -02-10T04:29:24.925261+0000 mon.c 2021-02-10T04:29:25.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[35961]: (mon.1) 43 : audit [DBG] from='client.? 172.21.15.85:0/1803486755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:29:25.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[35961]: cluster 2021-02-10T04:29:25.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[35961]: 2021-02-10T04:29:25.050135+0000 2021-02-10T04:29:25.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[35961]: mon.a (mon.0) 825 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2021-02-10T04:29:25.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[35961]: 2021-02-10T04:29:25.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:29:25.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[35961]: :29:25.073368+0000 osd.3 (osd.3) 41 2021-02-10T04:29:25.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[35961]: : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:29:25.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:29:25.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[35961]: 10T04:29:25.073824+0000 osd.3 (osd. 2021-02-10T04:29:25.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:25 smithi085 conmon[35961]: 3) 42 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:29:25.704 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:29:25.734 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:26.055 INFO:tasks.rados.rados.0.smithi085.stdout:297: done (3 left) 2021-02-10T04:29:26.056 INFO:tasks.rados.rados.0.smithi085.stdout:298: rollback oid 6 current snap is 27 2021-02-10T04:29:26.056 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 6 to 23 2021-02-10T04:29:26.056 INFO:tasks.rados.rados.0.smithi085.stdout:299: rollback oid 40 current snap is 27 2021-02-10T04:29:26.056 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 40 to 25 2021-02-10T04:29:26.058 INFO:tasks.rados.rados.0.smithi085.stdout:300: write (excl) oid 15 current snap is 27 2021-02-10T04:29:26.058 INFO:tasks.rados.rados.0.smithi085.stdout:300: seq_num 139 ranges {736173=553965,1867350=468823} 2021-02-10T04:29:26.063 INFO:tasks.rados.rados.0.smithi085.stdout:300: writing smithi08575397-15 from 736173 to 1290138 tid 1 2021-02-10T04:29:26.069 INFO:tasks.rados.rados.0.smithi085.stdout:300: writing smithi08575397-15 from 1867350 to 2336173 tid 2 2021-02-10T04:29:26.069 INFO:tasks.rados.rados.0.smithi085.stdout:301: read oid 45 snap -1 2021-02-10T04:29:26.070 INFO:tasks.rados.rados.0.smithi085.stdout:301: expect (ObjNum 96 snap 19 seq_num 96) 2021-02-10T04:29:26.070 INFO:tasks.rados.rados.0.smithi085.stdout:302: copy_from oid 34 from oid 24 current snap is 27 2021-02-10T04:29:26.070 INFO:tasks.rados.rados.0.smithi085.stdout:303: rollback oid 39 current snap is 27 2021-02-10T04:29:26.070 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 39 to 25 2021-02-10T04:29:26.070 INFO:tasks.rados.rados.0.smithi085.stdout:304: delete oid 21 current snap is 27 2021-02-10T04:29:26.070 INFO:tasks.rados.rados.0.smithi085.stdout:299: finishing rollback tid 0 to smithi08575397-40 2021-02-10T04:29:26.070 INFO:tasks.rados.rados.0.smithi085.stdout:299: finishing rollback tid 1 to smithi08575397-40 2021-02-10T04:29:26.070 INFO:tasks.rados.rados.0.smithi085.stdout:299: finishing rollback tid 2 to smithi08575397-40 2021-02-10T04:29:26.070 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 130 (ObjNum 39 snap 0 seq_num 39) dirty exists 2021-02-10T04:29:26.070 INFO:tasks.rados.rados.0.smithi085.stdout:298: finishing rollback tid 1 to smithi08575397-6 2021-02-10T04:29:26.070 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 70 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-02-10T04:29:26.071 INFO:tasks.rados.rados.0.smithi085.stdout:300: finishing write tid 1 to smithi08575397-15 2021-02-10T04:29:26.072 INFO:tasks.rados.rados.0.smithi085.stdout:302: finishing copy_from to smithi08575397-34 2021-02-10T04:29:26.072 INFO:tasks.rados.rados.0.smithi085.stdout:302: got expected ENOENT (src dne) 2021-02-10T04:29:26.072 INFO:tasks.rados.rados.0.smithi085.stdout:302: finishing copy_from racing read to smithi08575397-34 2021-02-10T04:29:26.072 INFO:tasks.rados.rados.0.smithi085.stdout:300: finishing write tid 2 to smithi08575397-15 2021-02-10T04:29:26.072 INFO:tasks.rados.rados.0.smithi085.stdout:303: finishing rollback tid 0 to smithi08575397-39 2021-02-10T04:29:26.073 INFO:tasks.rados.rados.0.smithi085.stdout:300: finishing write tid 3 to smithi08575397-15 2021-02-10T04:29:26.075 INFO:tasks.rados.rados.0.smithi085.stdout:300: finishing write tid 4 to smithi08575397-15 2021-02-10T04:29:26.075 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 15 v 62 (ObjNum 139 snap 27 seq_num 139) dirty exists 2021-02-10T04:29:26.075 INFO:tasks.rados.rados.0.smithi085.stdout:300: left oid 15 (ObjNum 139 snap 27 seq_num 139) 2021-02-10T04:29:26.075 INFO:tasks.rados.rados.0.smithi085.stdout:303: finishing rollback tid 1 to smithi08575397-39 2021-02-10T04:29:26.075 INFO:tasks.rados.rados.0.smithi085.stdout:303: finishing rollback tid 2 to smithi08575397-39 2021-02-10T04:29:26.075 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 39 v 38 (ObjNum 103 snap 20 seq_num 103) dirty exists 2021-02-10T04:29:26.075 INFO:tasks.rados.rados.0.smithi085.stdout:298: done (9 left) 2021-02-10T04:29:26.075 INFO:tasks.rados.rados.0.smithi085.stdout:299: done (8 left) 2021-02-10T04:29:26.075 INFO:tasks.rados.rados.0.smithi085.stdout:300: done (7 left) 2021-02-10T04:29:26.075 INFO:tasks.rados.rados.0.smithi085.stdout:302: done (6 left) 2021-02-10T04:29:26.075 INFO:tasks.rados.rados.0.smithi085.stdout:303: done (5 left) 2021-02-10T04:29:26.075 INFO:tasks.rados.rados.0.smithi085.stdout:304: done (4 left) 2021-02-10T04:29:26.076 INFO:tasks.rados.rados.0.smithi085.stdout:305: write oid 13 current snap is 27 2021-02-10T04:29:26.076 INFO:tasks.rados.rados.0.smithi085.stdout:305: seq_num 141 ranges {769787=704463,2267961=101826} 2021-02-10T04:29:26.083 INFO:tasks.rados.rados.0.smithi085.stdout:305: writing smithi08575397-13 from 769787 to 1474250 tid 1 2021-02-10T04:29:26.088 INFO:tasks.rados.rados.0.smithi085.stdout:305: writing smithi08575397-13 from 2267961 to 2369787 tid 2 2021-02-10T04:29:26.089 INFO:tasks.rados.rados.0.smithi085.stdout:306: snap_create 2021-02-10T04:29:26.089 INFO:tasks.rados.rados.0.smithi085.stdout:301: expect (ObjNum 96 snap 19 seq_num 96) 2021-02-10T04:29:26.258 INFO:tasks.rados.rados.0.smithi085.stdout:305: finishing write tid 1 to smithi08575397-13 2021-02-10T04:29:26.258 INFO:tasks.rados.rados.0.smithi085.stdout:305: finishing write tid 2 to smithi08575397-13 2021-02-10T04:29:26.258 INFO:tasks.rados.rados.0.smithi085.stdout:305: finishing write tid 3 to smithi08575397-13 2021-02-10T04:29:26.258 INFO:tasks.rados.rados.0.smithi085.stdout:305: finishing write tid 4 to smithi08575397-13 2021-02-10T04:29:26.258 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 60 (ObjNum 141 snap 27 seq_num 141) dirty exists 2021-02-10T04:29:26.259 INFO:tasks.rados.rados.0.smithi085.stdout:305: left oid 13 (ObjNum 141 snap 27 seq_num 141) 2021-02-10T04:29:26.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:26 smithi164 conmon[29288]: cluster 2021-02-10T04:29:24.658356+0000 osd.6 (osd.6) 35 2021-02-10T04:29:26.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:26 smithi164 conmon[29288]: : cluster [DBG] 6.3 scrub ok 2021-02-10T04:29:26.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:26 smithi164 conmon[29288]: cluster 2021-02-10T04:29:24.731454+0000 osd.2 (osd.2) 39 : cluster [DBG] 6.8 scrub ok 2021-02-10T04:29:26.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:26 smithi164 conmon[29288]: cluster 2021-02-10T04:29:25.075932+0000 2021-02-10T04:29:26.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:26 smithi164 conmon[29288]: osd.1 (osd.1) 45 : cluster [DBG] 6.1d scrub ok 2021-02-10T04:29:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:26 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:29:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:26 smithi164 conmon[29288]: 02-10T04:29:25.382785+0000 osd.3 2021-02-10T04:29:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:26 smithi164 conmon[29288]: (osd.3) 43 : cluster 2021-02-10T04:29:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:26 smithi164 conmon[29288]: [DBG] 6.1b scrub ok 2021-02-10T04:29:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:26 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:29:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:26 smithi164 conmon[29288]: 02-10T04:29:25.645392+0000 osd.0 2021-02-10T04:29:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:26 smithi164 conmon[29288]: (osd.0) 45 : cluster [DBG] 4.16 scrub ok 2021-02-10T04:29:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:26 smithi164 conmon[29288]: cluster 2021-02-10T04:29:26.003761 2021-02-10T04:29:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:26 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 364 : cluster [DBG] pgmap v373: 168 pgs: 5 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 153 active+clean; 186 MiB data, 505 MiB used, 715 GiB / 715 GiB avail; 4.5 MiB/s rd, 2.4 MiB/s wr, 14 op/s 2021-02-10T04:29:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:26 smithi164 conmon[29288]: cluster 2021-02-10T04:29:26.055204+0000 mon.a ( 2021-02-10T04:29:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:26 smithi164 conmon[29288]: mon.0) 826 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2021-02-10T04:29:26.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[29319]: cluster 2021-02-10T04:29:24.658356+0000 2021-02-10T04:29:26.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[29319]: osd.6 (osd.6) 35 : cluster [DBG] 6.3 scrub ok 2021-02-10T04:29:26.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[29319]: cluster 2021-02-10T04:29:24.731454+0000 osd.2 (osd.2) 39 : cluster [DBG] 6.8 scrub ok 2021-02-10T04:29:26.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[29319]: cluster 2021-02-10T04:29:25.075932+0000 osd.1 (osd.1) 45 : cluster [DBG] 6.1d scrub ok 2021-02-10T04:29:26.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[29319]: cluster 2021-02-10T04:29:25.382785+0000 osd.3 2021-02-10T04:29:26.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[29319]: (osd.3) 43 : cluster [DBG] 6.1b scrub ok 2021-02-10T04:29:26.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[29319]: cluster 2021-02-10T04:29: 2021-02-10T04:29:26.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[29319]: 25.645392+0000 osd.0 (osd.0) 45 : cluster [DBG] 4.16 scrub ok 2021-02-10T04:29:26.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[29319]: cluster 2021-02-10T04:29:26.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[29319]: 2021-02-10T04:29:26.003761 2021-02-10T04:29:26.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 364 : cluster [DBG] 2021-02-10T04:29:26.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[29319]: pgmap v373: 168 pgs: 5 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 153 active+clean; 186 MiB data, 505 MiB used, 715 GiB / 715 GiB avail; 4.5 MiB/s rd, 2.4 MiB/s wr, 14 op/s 2021-02-10T04:29:26.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[29319]: cluster 2021-02-10T04:29 2021-02-10T04:29:26.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[29319]: :26.055204+0000 mon.a (mon.0) 826 : cluster [DBG] 2021-02-10T04:29:26.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[29319]: osdmap e150: 8 total, 8 up, 8 in 2021-02-10T04:29:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[35961]: cluster 2021-02-10T04:29:24.658356 2021-02-10T04:29:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[35961]: +0000 osd.6 (osd.6) 35 : cluster [DBG] 6.3 scrub ok 2021-02-10T04:29:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:29:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[35961]: 04:29:24.731454+0000 osd.2 (osd.2) 39 : cluster [DBG] 6.8 scrub ok 2021-02-10T04:29:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[35961]: cluster 2021-02-10T04:29:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[35961]: 2021-02-10T04:29:25.075932+0000 2021-02-10T04:29:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[35961]: osd.1 (osd.1) 45 : cluster [DBG] 6.1d scrub ok 2021-02-10T04:29:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[35961]: 2021-02-10T04:29:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:29:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[35961]: 29:25.382785+0000 osd.3 (osd.3) 43 : cluster [DBG] 2021-02-10T04:29:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[35961]: 6.1b scrub ok 2021-02-10T04:29:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[35961]: cluster 2021-02-10T04:29: 2021-02-10T04:29:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[35961]: 25.645392+0000 osd.0 ( 2021-02-10T04:29:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[35961]: osd.0) 45 : cluster [DBG] 4.16 scrub ok 2021-02-10T04:29:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:29:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[35961]: -02-10T04:29:26.003761+0000 2021-02-10T04:29:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[35961]: mgr.y (mgr.14140) 364 : cluster [DBG] 2021-02-10T04:29:26.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[35961]: pgmap v373: 168 pgs: 5 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 153 active+clean; 186 MiB data, 505 MiB used, 715 GiB / 715 GiB avail; 4.5 MiB/s rd, 2.4 MiB/s wr, 14 op/s 2021-02-10T04:29:26.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[35961]: cluster 2021-02-10T04:29: 2021-02-10T04:29:26.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[35961]: 26.055204+0000 mon.a (mon.0) 826 : 2021-02-10T04:29:26.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[35961]: cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2021-02-10T04:29:27.002 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:29:27.055 INFO:tasks.rados.rados.0.smithi085.stdout:301: done (5 left) 2021-02-10T04:29:27.055 INFO:tasks.rados.rados.0.smithi085.stdout:305: done (4 left) 2021-02-10T04:29:27.056 INFO:tasks.rados.rados.0.smithi085.stdout:306: done (3 left) 2021-02-10T04:29:27.056 INFO:tasks.rados.rados.0.smithi085.stdout:307: rollback oid 41 current snap is 28 2021-02-10T04:29:27.056 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 41 to 23 2021-02-10T04:29:27.056 INFO:tasks.rados.rados.0.smithi085.stdout:308: snap_create 2021-02-10T04:29:27.059 INFO:tasks.rados.rados.0.smithi085.stdout:307: finishing rollback tid 0 to smithi08575397-41 2021-02-10T04:29:27.060 INFO:tasks.rados.rados.0.smithi085.stdout:307: finishing rollback tid 1 to smithi08575397-41 2021-02-10T04:29:27.060 INFO:tasks.rados.rados.0.smithi085.stdout:307: finishing rollback tid 2 to smithi08575397-41 2021-02-10T04:29:27.060 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 41 v 135 (ObjNum 39 snap 0 seq_num 39) dirty exists 2021-02-10T04:29:27.107 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:29:27.120 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:29:26 smithi085 conmon[49106]: debug 2021-02-10T04:29:26.976+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:27.136 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:27.252 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:29:27.358 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:29:27.393 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:29:27 smithi085 conmon[52836]: debug 2021-02-10T04:29:27.153+0000 7fba74fb8700 -1 osd.3 pg_epoch: 151 pg[7.a( v 149'54 (0'0,149'54] local-lis/les=93/94 n=3 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [3,2,5] r=0 lpr=93 crt=149'54 lcod 149'52 mlcod 149'52 active+clean+snaptrim trimq=[15~1,1d~2,20~1,24~1,28~1,30~1,33~1] ps=[19~1,1c~1]] removing snap head 2021-02-10T04:29:27.393 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:27 smithi085 conmon[29319]: cluster 2021-02-10T04:29:25.974275+0000 osd.1 ( 2021-02-10T04:29:27.393 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:27 smithi085 conmon[29319]: osd.1) 46 : cluster [DBG] 4.7 scrub ok 2021-02-10T04:29:27.393 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:27 smithi085 conmon[29319]: cluster 2021-02-10T04:29:26.355647+0000 osd.3 (osd.3) 2021-02-10T04:29:27.393 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:27 smithi085 conmon[29319]: 44 : cluster [DBG] 6.1f scrub ok 2021-02-10T04:29:27.394 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:27 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:29:27.394 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:27 smithi085 conmon[29319]: 04:29:26.437583+0000 osd.5 (osd. 2021-02-10T04:29:27.394 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:27 smithi085 conmon[29319]: 5) 43 : cluster [DBG] 6.1c scrub ok 2021-02-10T04:29:27.394 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:27 smithi085 conmon[29319]: cluster 2021-02-10T04:29:27.394 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:27 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:29:27.394 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:27 smithi085 conmon[29319]: :29:27.052362+0000 mon.a (mon.0) 2021-02-10T04:29:27.394 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:27 smithi085 conmon[29319]: 827 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2021-02-10T04:29:27.394 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:27 smithi085 conmon[35961]: cluster 2021-02-10T04:29:25.974275 2021-02-10T04:29:27.394 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:27 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 46 : cluster [DBG] 4.7 scrub ok 2021-02-10T04:29:27.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:27 smithi085 conmon[35961]: cluster 2021-02-10T04:29:27.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:27 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:29:27.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:27 smithi085 conmon[35961]: T04:29:26.355647+0000 osd.3 (osd. 2021-02-10T04:29:27.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:27 smithi085 conmon[35961]: 3) 44 : cluster [DBG] 6.1f scrub ok 2021-02-10T04:29:27.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:27 smithi085 conmon[35961]: cluster 2021-02-10T04:29:27.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:27 smithi085 conmon[35961]: 2021-02-10T04:29 2021-02-10T04:29:27.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:27 smithi085 conmon[35961]: :26.437583+0000 osd.5 (osd.5) 43 : 2021-02-10T04:29:27.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:27 smithi085 conmon[35961]: cluster [DBG] 6.1c scrub ok 2021-02-10T04:29:27.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:27 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:29:27.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:27 smithi085 conmon[35961]: -10T04:29:27.052362+0000 mon.a (mon 2021-02-10T04:29:27.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:27 smithi085 conmon[35961]: .0) 827 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2021-02-10T04:29:27.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:27 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:29:27.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:27 smithi164 conmon[29288]: 04:29:25.974275+0000 osd.1 (osd.1) 46 : cluster [DBG] 4.7 scrub ok 2021-02-10T04:29:27.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:27 smithi164 conmon[29288]: cluster 2021-02-10T04:29:27.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:27 smithi164 conmon[29288]: 2021-02-10T04:29: 2021-02-10T04:29:27.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:27 smithi164 conmon[29288]: 26.355647+0000 osd.3 (osd.3) 2021-02-10T04:29:27.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:27 smithi164 conmon[29288]: 44 : cluster [DBG] 6.1f scrub ok 2021-02-10T04:29:27.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:27 smithi164 conmon[29288]: cluster 2021-02-10T04:29:27.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:27 smithi164 conmon[29288]: 2021-02-10T04:29:26.437583 2021-02-10T04:29:27.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:27 smithi164 conmon[29288]: +0000 osd.5 (osd.5) 43 : cluster 2021-02-10T04:29:27.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:27 smithi164 conmon[29288]: [DBG] 6.1c scrub ok 2021-02-10T04:29:27.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:27 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:29:27.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:27 smithi164 conmon[29288]: -10T04:29:27 2021-02-10T04:29:27.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:27 smithi164 conmon[29288]: .052362+0000 mon.a (mon.0) 827 : cluster [DBG] 2021-02-10T04:29:27.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:27 smithi164 conmon[29288]: osdmap e151: 8 total, 8 up, 8 in 2021-02-10T04:29:27.397 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:29:27 smithi164 conmon[44544]: debug 2021-02-10T04:29:27.217+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:27.398 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:27.691 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:29:27 smithi085 conmon[45443]: debug 2021-02-10T04:29:27.628+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:27.946 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:29:28.059 INFO:tasks.rados.rados.0.smithi085.stdout:307: done (4 left) 2021-02-10T04:29:28.060 INFO:tasks.rados.rados.0.smithi085.stdout:308: done (3 left) 2021-02-10T04:29:28.060 INFO:tasks.rados.rados.0.smithi085.stdout:309: snap_remove snap 28 2021-02-10T04:29:28.065 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is unset 2021-02-10T04:29:28.126 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:29:27 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:29:28.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:28 smithi164 conmon[29288]: cluster 2021-02-10T04:29:26.670940+0000 osd.6 (osd.6) 36 : cluster 2021-02-10T04:29:28.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:28 smithi164 conmon[29288]: [DBG] 4.a scrub ok 2021-02-10T04:29:28.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:28 smithi164 conmon[29288]: cluster 2021-02-10T04:29:26.978058+0000 osd.1 (osd.1) 47 : cluster [DBG] 6.1 scrub ok 2021-02-10T04:29:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:28 smithi164 conmon[29288]: audit 2021-02-10T04:29: 2021-02-10T04:29:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:28 smithi164 conmon[29288]: 27.719425+0000 mon.b (mon.2) 26 : audit 2021-02-10T04:29:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:28 smithi164 conmon[29288]: [INF] 2021-02-10T04:29:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:28 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/406058490' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:29:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:28 smithi164 conmon[29288]: audit 2021-02-10T04:29:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:28 smithi164 conmon[29288]: 2021-02-10T04:29:27 2021-02-10T04:29:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:28 smithi164 conmon[29288]: .719880+0000 mon.a (mon.0) 2021-02-10T04:29:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:28 smithi164 conmon[29288]: 828 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:29:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:28 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:29:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:28 smithi164 conmon[29288]: -10T04:29:27.725040+0000 mgr.y (mgr.14140 2021-02-10T04:29:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:28 smithi164 conmon[29288]: ) 365 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:29:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:28 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:29:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:28 smithi164 conmon[29288]: 29:28.004195+0000 mgr.y (mgr.14140) 366 : cluster [DBG] pgmap v376: 168 pgs: 4 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 155 active+clean; 174 MiB data, 479 MiB used, 715 GiB / 715 GiB avail; 6.2 MiB/s rd, 2.9 MiB/s wr, 14 op/s 2021-02-10T04:29:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:28 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:29:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:28 smithi164 conmon[29288]: 29:28.049426+0000 mon.a (mon.0 2021-02-10T04:29:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:28 smithi164 conmon[29288]: ) 829 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:29:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:28 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:29:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:28 smithi164 conmon[29288]: 02-10T04:29:28.049453+0000 mon.a (mon.0) 830 : cluster [INF] Cluster is now healthy 2021-02-10T04:29:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:28 smithi164 conmon[29288]: audit 2021-02-10T04:29:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:28 smithi164 conmon[29288]: 2021-02-10T04:29:28.058878+0000 2021-02-10T04:29:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:28 smithi164 conmon[29288]: mon.a (mon.0) 831 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:29:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:28 smithi164 conmon[29288]: cluster 2021-02-10T04:29:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:28 smithi164 conmon[29288]: 2021-02-10T04:29:28.059113+0000 2021-02-10T04:29:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:28 smithi164 conmon[29288]: mon.a (mon.0) 832 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2021-02-10T04:29:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:28 smithi164 conmon[29288]: 2021-02-10T04:29:28.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[29319]: cluster 2021-02-10T04:29:26.670940+0000 osd.6 (osd.6) 36 : cluster [DBG] 4.a scrub ok 2021-02-10T04:29:28.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[29319]: cluster 2021-02-10T04:29:26.978058+0000 osd.1 (osd.1) 47 : cluster [DBG] 6.1 scrub ok 2021-02-10T04:29:28.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[29319]: audit 2021-02-10T04:29:27.719425+0000 mon.b (mon.2) 26 : audit [INF] from='client.? 172.21.15.85:0/406058490' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:29:28.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[29319]: audit 2021-02-10T04:29:27.719880+0000 mon.a (mon.0) 828 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:29:28.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[29319]: audit 2021-02-10T04:29:27.725040+0000 mgr.y (mgr.14140) 365 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:29:28.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[29319]: cluster 2021-02-10T04:29:28.004195+0000 mgr.y (mgr.14140) 366 : cluster [DBG] pgmap v376: 168 pgs: 4 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 155 active+clean; 174 MiB data, 479 MiB used, 715 GiB / 715 GiB avail; 6.2 MiB/s rd, 2.9 MiB/s wr, 14 op/s 2021-02-10T04:29:28.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[29319]: cluster 2021-02-10T04:29:28.049426+0000 mon.a (mon.0) 829 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:29:28.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[29319]: cluster 2021-02-10T04:29:28.049453+0000 mon.a (mon.0) 830 : cluster [INF] Cluster is now healthy 2021-02-10T04:29:28.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[29319]: audit 2021-02-10T04:29:28.058878+0000 mon.a (mon.0) 831 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:29:28.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[29319]: cluster 2021-02-10T04:29:28.059113+0000 mon.a (mon.0) 832 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2021-02-10T04:29:28.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: cluster 2021-02-10T04:29:26.670940+0000 2021-02-10T04:29:28.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: osd.6 (osd.6) 36 : cluster [DBG] 4.a scrub ok 2021-02-10T04:29:28.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: cluster 2021-02-10T04:29:26.978058+0000 2021-02-10T04:29:28.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: osd.1 (osd.1) 47 : cluster [DBG] 6.1 scrub ok 2021-02-10T04:29:28.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:29:28.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: -10T04:29:27.719425+0000 mon.b ( 2021-02-10T04:29:28.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: mon.2) 26 : audit [INF] from='client.? 172.21.15.85:0/406058490' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:29:28.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: audit 2021-02-10T04:29:27.719880+0000 2021-02-10T04:29:28.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: mon.a (mon.0) 828 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:29:28.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: 2021-02-10T04:29:28.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: audit 2021-02-10T04:29:27 2021-02-10T04:29:28.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: .725040+0000 mgr.y (mgr.14140) 365 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:29:28.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: 2021-02-10T04:29:28.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:29:28.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: :29:28.004195+0000 mgr.y (mgr.14140 2021-02-10T04:29:28.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: ) 366 : cluster [DBG] pgmap v376: 168 pgs: 4 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 155 active+clean; 174 MiB data, 479 MiB used, 715 GiB / 715 GiB avail; 6.2 MiB/s rd, 2.9 MiB/s wr, 14 op/s 2021-02-10T04:29:28.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: cluster 2021-02-10T04:29:28.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:29:28.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: 10T04:29:28.049426+0000 mon.a (mon.0 2021-02-10T04:29:28.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: ) 829 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:29:28.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:29:28.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: -02-10T04:29:28.049453+0000 2021-02-10T04:29:28.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: mon.a (mon.0) 830 : cluster [INF] 2021-02-10T04:29:28.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: Cluster is now healthy 2021-02-10T04:29:28.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:29:28.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: :29:28.058878+0000 mon.a (mon. 2021-02-10T04:29:28.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: 0) 831 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:29:28.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: cluster 2021-02-10T04:29:28.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: 2021-02-10T04:29:28. 2021-02-10T04:29:28.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: 059113+0000 mon.a (mon.0) 832 : cluster 2021-02-10T04:29:28.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:28 smithi085 conmon[35961]: [DBG] osdmap e152: 8 total, 8 up, 8 in 2021-02-10T04:29:28.967 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:29:28.987 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-02-10T04:29:28.987 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 0 1 2021-02-10T04:29:29.000 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:29:29.028 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:29.060 INFO:tasks.rados.rados.0.smithi085.stdout:309: done (3 left) 2021-02-10T04:29:29.061 INFO:tasks.rados.rados.0.smithi085.stdout:310: copy_from oid 17 from oid 19 current snap is 29 2021-02-10T04:29:29.061 INFO:tasks.rados.rados.0.smithi085.stdout:311: rollback oid 45 current snap is 29 2021-02-10T04:29:29.061 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 45 to 23 2021-02-10T04:29:29.061 INFO:tasks.rados.rados.0.smithi085.stdout:312: read oid 49 snap -1 2021-02-10T04:29:29.061 INFO:tasks.rados.rados.0.smithi085.stdout:312: expect (ObjNum 94 snap 18 seq_num 94) 2021-02-10T04:29:29.061 INFO:tasks.rados.rados.0.smithi085.stdout:313: read oid 25 snap -1 2021-02-10T04:29:29.061 INFO:tasks.rados.rados.0.smithi085.stdout:313: expect (ObjNum 87 snap 11 seq_num 87) 2021-02-10T04:29:29.061 INFO:tasks.rados.rados.0.smithi085.stdout:314: write oid 14 current snap is 29 2021-02-10T04:29:29.061 INFO:tasks.rados.rados.0.smithi085.stdout:314: seq_num 142 ranges {786594=779712,2326443=60151} 2021-02-10T04:29:29.073 INFO:tasks.rados.rados.0.smithi085.stdout:314: writing smithi08575397-14 from 786594 to 1566306 tid 1 2021-02-10T04:29:29.079 INFO:tasks.rados.rados.0.smithi085.stdout:314: writing smithi08575397-14 from 2326443 to 2386594 tid 2 2021-02-10T04:29:29.080 INFO:tasks.rados.rados.0.smithi085.stdout:313: expect (ObjNum 87 snap 11 seq_num 87) 2021-02-10T04:29:29.135 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:29:29.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[35961]: cluster 2021-02-10T04:29:28.398416+0000 osd.3 (osd.3) 2021-02-10T04:29:29.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[35961]: 45 : cluster [DBG] 4.0 deep-scrub ok 2021-02-10T04:29:29.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[35961]: cluster 2021-02-10T04:29:28.719143+0000 osd.0 (osd.0) 46 : cluster [DBG] 2.1f scrub ok 2021-02-10T04:29:29.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[35961]: cluster 2021-02-10T04:29:29.059593+0000 mon.a (mon.0) 833 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2021-02-10T04:29:29.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: cluster 2021-02-10T04:29:29.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 2021-02-10T04:29:29.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 2021 2021-02-10T04:29:29.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: - 2021-02-10T04:29:29.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 02 2021-02-10T04:29:29.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: - 2021-02-10T04:29:29.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 10 2021-02-10T04:29:29.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: T 2021-02-10T04:29:29.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 04 2021-02-10T04:29:29.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: : 2021-02-10T04:29:29.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 29 2021-02-10T04:29:29.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: : 2021-02-10T04:29:29.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 28 2021-02-10T04:29:29.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: . 2021-02-10T04:29:29.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 398416 2021-02-10T04:29:29.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: +0000 2021-02-10T04:29:29.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 2021-02-10T04:29:29.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: osd.3 2021-02-10T04:29:29.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: ( 2021-02-10T04:29:29.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: osd 2021-02-10T04:29:29.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: . 2021-02-10T04:29:29.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 3 2021-02-10T04:29:29.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: ) 2021-02-10T04:29:29.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 45 2021-02-10T04:29:29.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: : 2021-02-10T04:29:29.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: cluster 2021-02-10T04:29:29.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 2021-02-10T04:29:29.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: [DBG] 2021-02-10T04:29:29.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 2021-02-10T04:29:29.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 4.0 deep-scrub ok 2021-02-10T04:29:29.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 2021-02-10T04:29:29.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: cluster 2021-02-10T04:29:29.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 2021-02-10T04:29:29.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 2021 2021-02-10T04:29:29.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: - 2021-02-10T04:29:29.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 02 2021-02-10T04:29:29.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: - 2021-02-10T04:29:29.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 10 2021-02-10T04:29:29.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: T 2021-02-10T04:29:29.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 04 2021-02-10T04:29:29.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: : 2021-02-10T04:29:29.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 29 2021-02-10T04:29:29.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: : 2021-02-10T04:29:29.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 28 2021-02-10T04:29:29.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: . 2021-02-10T04:29:29.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 719143 2021-02-10T04:29:29.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: +0000 2021-02-10T04:29:29.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 2021-02-10T04:29:29.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: osd.0 2021-02-10T04:29:29.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: ( 2021-02-10T04:29:29.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: osd 2021-02-10T04:29:29.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: . 2021-02-10T04:29:29.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 0 2021-02-10T04:29:29.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: ) 2021-02-10T04:29:29.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 46 2021-02-10T04:29:29.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: : 2021-02-10T04:29:29.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: cluster 2021-02-10T04:29:29.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 2021-02-10T04:29:29.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: [DBG] 2021-02-10T04:29:29.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 2021-02-10T04:29:29.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 2.1f scrub ok 2021-02-10T04:29:29.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 2021-02-10T04:29:29.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: cluster 2021-02-10T04:29:29.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 2021-02-10T04:29:29.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 2021 2021-02-10T04:29:29.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: - 2021-02-10T04:29:29.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 02 2021-02-10T04:29:29.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: - 2021-02-10T04:29:29.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 10 2021-02-10T04:29:29.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: T 2021-02-10T04:29:29.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 04 2021-02-10T04:29:29.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: : 2021-02-10T04:29:29.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 29 2021-02-10T04:29:29.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: : 2021-02-10T04:29:29.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 29 2021-02-10T04:29:29.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: . 2021-02-10T04:29:29.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 059593 2021-02-10T04:29:29.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: +0000 2021-02-10T04:29:29.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 2021-02-10T04:29:29.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: mon.a 2021-02-10T04:29:29.198 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: ( 2021-02-10T04:29:29.198 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: mon 2021-02-10T04:29:29.198 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: . 2021-02-10T04:29:29.198 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 0 2021-02-10T04:29:29.198 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: ) 2021-02-10T04:29:29.198 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 833 2021-02-10T04:29:29.198 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: : 2021-02-10T04:29:29.198 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: cluster 2021-02-10T04:29:29.198 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 2021-02-10T04:29:29.198 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: [DBG] 2021-02-10T04:29:29.198 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 2021-02-10T04:29:29.198 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: osdmap e153: 8 total, 8 up, 8 in 2021-02-10T04:29:29.198 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[29319]: 2021-02-10T04:29:29.240 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:29:29.277 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:29.288 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:29:29.319 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:29:29.348 INFO:tasks.rados.rados.0.smithi085.stdout:311: finishing rollback tid 0 to smithi08575397-45 2021-02-10T04:29:29.348 INFO:tasks.rados.rados.0.smithi085.stdout:313: done (7 left) 2021-02-10T04:29:29.348 INFO:tasks.rados.rados.0.smithi085.stdout:315: snap_remove snap 25 2021-02-10T04:29:29.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:29:29 smithi164 conmon[40892]: debug 2021-02-10T04:29:29.106+0000 7f30324bf700 -1 osd.6 pg_epoch: 153 pg[7.d( v 150'54 (0'0,150'54] local-lis/les=93/94 n=5 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [6,1,0] r=0 lpr=93 crt=150'54 lcod 150'52 mlcod 150'52 active+clean+snaptrim trimq=[28~1,30~1,33~1,37~1] ps=[15~1,19~1,1c~3,20~1,24~1]] removing snap head 2021-02-10T04:29:29.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:29:29 smithi164 conmon[44544]: debug 2021-02-10T04:29:29.112+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:29.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:29 smithi164 conmon[29288]: cluster 2021-02-10T04:29:28.398416+0000 osd.3 (osd. 2021-02-10T04:29:29.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:29 smithi164 conmon[29288]: 3) 45 : cluster [DBG] 4.0 deep-scrub ok 2021-02-10T04:29:29.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:29 smithi164 conmon[29288]: cluster 2021-02-10T04:29:28.719143+0000 osd.0 (osd.0) 46 : cluster [DBG] 2.1f scrub ok 2021-02-10T04:29:29.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:29 smithi164 conmon[29288]: cluster 2021-02-10T04:29:29.059593+0000 2021-02-10T04:29:29.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:29 smithi164 conmon[29288]: mon.a (mon.0) 833 : cluster [DBG] 2021-02-10T04:29:29.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:29 smithi164 conmon[29288]: osdmap e153: 8 total, 8 up, 8 in 2021-02-10T04:29:29.977 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:29:30.082 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:29:30.110 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:30.145 INFO:tasks.rados.rados.0.smithi085.stdout:315: done (7 left) 2021-02-10T04:29:30.145 INFO:tasks.rados.rados.0.smithi085.stdout:316: read oid 30 snap 23 2021-02-10T04:29:30.145 INFO:tasks.rados.rados.0.smithi085.stdout:316: expect deleted 2021-02-10T04:29:30.145 INFO:tasks.rados.rados.0.smithi085.stdout:317: delete oid 11 current snap is 29 2021-02-10T04:29:30.145 INFO:tasks.rados.rados.0.smithi085.stdout:311: finishing rollback tid 1 to smithi08575397-45 2021-02-10T04:29:30.146 INFO:tasks.rados.rados.0.smithi085.stdout:311: finishing rollback tid 2 to smithi08575397-45 2021-02-10T04:29:30.146 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 45 v 68 (ObjNum 96 snap 19 seq_num 96) dirty exists 2021-02-10T04:29:30.146 INFO:tasks.rados.rados.0.smithi085.stdout:312: expect (ObjNum 94 snap 18 seq_num 94) 2021-02-10T04:29:30.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:30 smithi085 conmon[29319]: cluster 2021-02-10T04:29:28.632078+0000 osd.6 (osd.6) 37 : cluster 2021-02-10T04:29:30.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:30 smithi085 conmon[29319]: [DBG] 2.7 scrub ok 2021-02-10T04:29:30.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:30 smithi085 conmon[29319]: cluster 2021-02-10T04:29:28.776334+0000 osd.2 (osd.2) 40 : cluster [DBG] 6.14 scrub ok 2021-02-10T04:29:30.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:30 smithi085 conmon[29319]: cluster 2021-02-10T04:29:28.928898+0000 osd.1 (osd.1) 48 : cluster [DBG] 1.3 scrub ok 2021-02-10T04:29:30.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:30 smithi085 conmon[29319]: cluster 2021-02-10T04:29:29.355250+0000 osd.3 (osd.3) 46 : cluster [DBG] 2.1e scrub ok 2021-02-10T04:29:30.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:30 smithi085 conmon[29319]: cluster 2021-02-10T04:29:29.384165+0000 osd.5 (osd.5) 44 : cluster [DBG] 1.a scrub ok 2021-02-10T04:29:30.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:30 smithi085 conmon[29319]: cluster 2021-02-10T04:29:29.683801+0000 osd.0 (osd.0) 47 : cluster [DBG] 1.2 scrub ok 2021-02-10T04:29:30.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:30 smithi085 conmon[29319]: cluster 2021-02-10T04:29:30.004770+0000 mgr.y (mgr.14140) 367 : cluster [DBG] pgmap v379: 168 pgs: 4 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 156 active+clean; 158 MiB data, 450 MiB used, 715 GiB / 715 GiB avail; 6.0 MiB/s rd, 726 KiB/s wr, 8 op/s 2021-02-10T04:29:30.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:30 smithi085 conmon[35961]: cluster 2021-02-10T04:29:28.632078+0000 osd.6 (osd.6) 37 : cluster [DBG] 2.7 scrub ok 2021-02-10T04:29:30.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:30 smithi085 conmon[35961]: cluster 2021-02-10T04:29:28.776334+0000 osd.2 (osd.2) 40 : cluster [DBG] 6.14 scrub ok 2021-02-10T04:29:30.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:30 smithi085 conmon[35961]: cluster 2021-02-10T04:29:28.928898+0000 osd.1 (osd.1) 48 : cluster [DBG] 1.3 scrub ok 2021-02-10T04:29:30.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:30 smithi085 conmon[35961]: cluster 2021-02-10T04:29:29.355250+0000 osd.3 (osd.3) 46 : cluster [DBG] 2.1e scrub ok 2021-02-10T04:29:30.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:30 smithi085 conmon[35961]: cluster 2021-02-10T04:29:29.384165+0000 osd.5 (osd.5) 44 : cluster [DBG] 1.a scrub ok 2021-02-10T04:29:30.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:30 smithi085 conmon[35961]: cluster 2021-02-10T04:29:29.683801+0000 osd.0 (osd.0) 47 : cluster [DBG] 1.2 scrub ok 2021-02-10T04:29:30.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:30 smithi085 conmon[35961]: cluster 2021-02-10T04:29:30.004770+0000 mgr.y (mgr.14140) 367 : cluster [DBG] pgmap v379: 168 pgs: 4 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 156 active+clean; 158 MiB data, 450 MiB used, 715 GiB / 715 GiB avail; 6.0 MiB/s rd, 726 KiB/s wr, 8 op/s 2021-02-10T04:29:30.192 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:29:29 smithi085 conmon[45443]: debug 2021-02-10T04:29:29.921+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:30.202 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:29:30.303 INFO:tasks.rados.rados.0.smithi085.stdout:314: finishing write tid 1 to smithi08575397-14 2021-02-10T04:29:30.304 INFO:tasks.rados.rados.0.smithi085.stdout:314: finishing write tid 2 to smithi08575397-14 2021-02-10T04:29:30.304 INFO:tasks.rados.rados.0.smithi085.stdout:314: finishing write tid 3 to smithi08575397-14 2021-02-10T04:29:30.304 INFO:tasks.rados.rados.0.smithi085.stdout:314: finishing write tid 4 to smithi08575397-14 2021-02-10T04:29:30.304 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 86 (ObjNum 142 snap 29 seq_num 142) dirty exists 2021-02-10T04:29:30.304 INFO:tasks.rados.rados.0.smithi085.stdout:314: left oid 14 (ObjNum 142 snap 29 seq_num 142) 2021-02-10T04:29:30.304 INFO:tasks.rados.rados.0.smithi085.stdout:311: done (8 left) 2021-02-10T04:29:30.304 INFO:tasks.rados.rados.0.smithi085.stdout:312: done (7 left) 2021-02-10T04:29:30.304 INFO:tasks.rados.rados.0.smithi085.stdout:314: done (6 left) 2021-02-10T04:29:30.304 INFO:tasks.rados.rados.0.smithi085.stdout:317: done (5 left) 2021-02-10T04:29:30.304 INFO:tasks.rados.rados.0.smithi085.stdout:318: snap_remove snap 27 2021-02-10T04:29:30.308 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:29:30.335 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:30.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:30 smithi164 conmon[29288]: cluster 2021-02-10T04:29:28.632078 2021-02-10T04:29:30.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:30 smithi164 conmon[29288]: +0000 osd.6 (osd.6) 37 : cluster [DBG] 2.7 scrub ok 2021-02-10T04:29:30.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:30 smithi164 conmon[29288]: cluster 2021-02-10T04:29:28.776334+0000 osd.2 (osd.2) 40 : cluster [DBG] 6.14 scrub ok 2021-02-10T04:29:30.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:30 smithi164 conmon[29288]: cluster 2021-02-10T04:29:28 2021-02-10T04:29:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:30 smithi164 conmon[29288]: .928898+0000 osd.1 (osd.1) 48 : cluster [DBG] 1.3 scrub ok 2021-02-10T04:29:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:30 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:29:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:30 smithi164 conmon[29288]: 02-10T04:29:29.355250+0000 2021-02-10T04:29:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:30 smithi164 conmon[29288]: osd.3 (osd.3) 46 : cluster [DBG] 2.1e scrub ok 2021-02-10T04:29:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:30 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:29:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:30 smithi164 conmon[29288]: -02-10T04:29:29.384165+0000 osd.5 ( 2021-02-10T04:29:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:30 smithi164 conmon[29288]: osd.5) 44 : cluster [DBG] 1.a scrub ok 2021-02-10T04:29:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:30 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:29:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:30 smithi164 conmon[29288]: :29:29.683801+0000 osd.0 (osd.0) 47 : cluster 2021-02-10T04:29:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:30 smithi164 conmon[29288]: [DBG] 1.2 scrub ok 2021-02-10T04:29:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:30 smithi164 conmon[29288]: cluster 2021-02-10T04:29:30. 2021-02-10T04:29:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:30 smithi164 conmon[29288]: 004770+0000 mgr.y (mgr.14140) 367 : cluster [DBG] pgmap v379: 168 pgs: 4 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 156 active+clean; 158 MiB data, 450 MiB used, 715 GiB / 715 GiB avail; 6.0 MiB/s rd, 726 KiB/s wr, 8 op/s 2021-02-10T04:29:30.395 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:29:30 smithi164 conmon[33677]: debug 2021-02-10T04:29:30.177+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:30.419 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:29:30.523 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:29:30.560 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:30.753 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:29:30.820 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:29:30 smithi085 conmon[52836]: debug 2021-02-10T04:29:30.820 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:29:30 smithi085 conmon[52836]: 2021-02-10T04:29:30.693+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:30.820 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:29:30 smithi085 conmon[52836]: 2021-02-10T04:29:30.858 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:29:30.893 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:29:30 smithi164 conmon[37263]: debug 2021-02-10T04:29:30.397+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:30.893 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:31.134 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:31 smithi085 conmon[29319]: cluster 2021-02-10T04:29:29.805428+0000 osd.2 (osd.2) 41 : cluster [DBG] 5.4 scrub ok 2021-02-10T04:29:31.134 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:31 smithi085 conmon[29319]: cluster 2021-02-10T04:29:30.138895+0000 mon.a (mon.0) 834 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2021-02-10T04:29:31.135 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:31 smithi085 conmon[29319]: cluster 2021-02-10T04:29:30.504202+0000 osd.0 (osd.0) 48 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:29:31.135 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:31 smithi085 conmon[29319]: cluster 2021-02-10T04:29:30.504328+0000 osd.0 (osd.0) 49 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:29:31.144 INFO:tasks.rados.rados.0.smithi085.stdout:318: done (5 left) 2021-02-10T04:29:31.144 INFO:tasks.rados.rados.0.smithi085.stdout:319: delete oid 5 current snap is 29 2021-02-10T04:29:31.150 INFO:tasks.rados.rados.0.smithi085.stdout:316: done (5 left) 2021-02-10T04:29:31.150 INFO:tasks.rados.rados.0.smithi085.stdout:319: done (4 left) 2021-02-10T04:29:31.150 INFO:tasks.rados.rados.0.smithi085.stdout:320: snap_create 2021-02-10T04:29:31.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:31 smithi164 conmon[29288]: cluster 2021-02-10T04:29 2021-02-10T04:29:31.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:31 smithi164 conmon[29288]: :29.805428+0000 osd.2 (osd.2) 41 : cluster [DBG] 5.4 scrub ok 2021-02-10T04:29:31.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:31 smithi164 conmon[29288]: cluster 2021-02-10T04:29:30.138895+0000 mon.a (mon.0) 834 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2021-02-10T04:29:31.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:31 smithi164 conmon[29288]: 2021-02-10T04:29:31.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:31 smithi164 conmon[29288]: cluster 2021-02-10T04:29:31.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:31 smithi164 conmon[29288]: 2021-02-10T04:29:30.504202+0000 osd.0 2021-02-10T04:29:31.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:31 smithi164 conmon[29288]: (osd.0) 48 : cluster 2021-02-10T04:29:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:31 smithi164 conmon[29288]: [DBG] purged_snaps scrub starts 2021-02-10T04:29:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:31 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:29:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:31 smithi164 conmon[29288]: 02-10T04:29:30.504328+0000 osd.0 (osd 2021-02-10T04:29:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:31 smithi164 conmon[29288]: .0) 49 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:29:31.441 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:29:31 smithi085 conmon[52836]: debug 2021-02-10T04:29:31.152+0000 7fba747b7700 -1 osd.3 pg_epoch: 155 pg[7.3( v 155'101 (0'0,155'101] local-lis/les=93/94 n=6 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [3,2,4] r=0 lpr=93 luod=153'99 crt=155'101 lcod 151'97 mlcod 151'97 active+clean+snaptrim trimq=[28~1,30~1,32~2,36~2] ps=[15~1,19~1,1c~3,20~1,24~1]] removing snap head 2021-02-10T04:29:31.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:31 smithi085 conmon[35961]: cluster 2021-02-10T04:29:29.805428+0000 osd.2 (osd 2021-02-10T04:29:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:31 smithi085 conmon[35961]: .2) 41 : cluster [DBG] 5.4 scrub ok 2021-02-10T04:29:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:31 smithi085 conmon[35961]: cluster 2021-02-10T04:29:30.138895+0000 mon.a (mon.0) 834 : cluster [DBG] 2021-02-10T04:29:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:31 smithi085 conmon[35961]: osdmap e154: 8 total, 8 up, 8 in 2021-02-10T04:29:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:31 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:29:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:31 smithi085 conmon[35961]: 02-10T04:29:30.504202+0000 osd.0 (osd.0) 2021-02-10T04:29:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:31 smithi085 conmon[35961]: 48 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:29:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:31 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:29:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:31 smithi085 conmon[35961]: -10T04:29:30.504328 2021-02-10T04:29:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:31 smithi085 conmon[35961]: +0000 osd.0 (osd.0) 49 : cluster [DBG] 2021-02-10T04:29:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:31 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:29:31.463 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:29:31.463 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [], 2021-02-10T04:29:31.463 INFO:teuthology.orchestra.run.smithi085.stdout: "complaint_time": 30, 2021-02-10T04:29:31.464 INFO:teuthology.orchestra.run.smithi085.stdout: "num_blocked_ops": 0 2021-02-10T04:29:31.464 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:29:32.150 INFO:tasks.rados.rados.0.smithi085.stdout:320: done (4 left) 2021-02-10T04:29:32.150 INFO:tasks.rados.rados.0.smithi085.stdout:321: read oid 27 snap 29 2021-02-10T04:29:32.150 INFO:tasks.rados.rados.0.smithi085.stdout:321: expect deleted 2021-02-10T04:29:32.150 INFO:tasks.rados.rados.0.smithi085.stdout:322: read oid 21 snap -1 2021-02-10T04:29:32.150 INFO:tasks.rados.rados.0.smithi085.stdout:322: expect deleted 2021-02-10T04:29:32.151 INFO:tasks.rados.rados.0.smithi085.stdout:323: read oid 31 snap 23 2021-02-10T04:29:32.151 INFO:tasks.rados.rados.0.smithi085.stdout:323: expect (ObjNum 59 snap 4 seq_num 59) 2021-02-10T04:29:32.151 INFO:tasks.rados.rados.0.smithi085.stdout:324: read oid 32 snap -1 2021-02-10T04:29:32.151 INFO:tasks.rados.rados.0.smithi085.stdout:324: expect (ObjNum 31 snap 0 seq_num 31) 2021-02-10T04:29:32.151 INFO:tasks.rados.rados.0.smithi085.stdout:325: read oid 11 snap -1 2021-02-10T04:29:32.151 INFO:tasks.rados.rados.0.smithi085.stdout:325: expect deleted 2021-02-10T04:29:32.151 INFO:tasks.rados.rados.0.smithi085.stdout:326: write oid 9 current snap is 30 2021-02-10T04:29:32.152 INFO:tasks.rados.rados.0.smithi085.stdout:326: seq_num 145 ranges {437015=721812,1701889=509708} 2021-02-10T04:29:32.154 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set noscrub 2021-02-10T04:29:32.156 INFO:tasks.rados.rados.0.smithi085.stdout:326: writing smithi08575397-9 from 437015 to 1158827 tid 1 2021-02-10T04:29:32.162 INFO:tasks.rados.rados.0.smithi085.stdout:326: writing smithi08575397-9 from 1701889 to 2211597 tid 2 2021-02-10T04:29:32.163 INFO:tasks.rados.rados.0.smithi085.stdout:327: read oid 2 snap 23 2021-02-10T04:29:32.163 INFO:tasks.rados.rados.0.smithi085.stdout:327: expect deleted 2021-02-10T04:29:32.163 INFO:tasks.rados.rados.0.smithi085.stdout:328: copy_from oid 33 from oid 47 current snap is 30 2021-02-10T04:29:32.163 INFO:tasks.rados.rados.0.smithi085.stdout:324: expect (ObjNum 31 snap 0 seq_num 31) 2021-02-10T04:29:32.270 INFO:tasks.rados.rados.0.smithi085.stdout:321: done (11 left) 2021-02-10T04:29:32.270 INFO:tasks.rados.rados.0.smithi085.stdout:322: done (10 left) 2021-02-10T04:29:32.270 INFO:tasks.rados.rados.0.smithi085.stdout:324: done (9 left) 2021-02-10T04:29:32.270 INFO:tasks.rados.rados.0.smithi085.stdout:329: write oid 15 current snap is 30 2021-02-10T04:29:32.270 INFO:tasks.rados.rados.0.smithi085.stdout:323: expect (ObjNum 59 snap 4 seq_num 59) 2021-02-10T04:29:32.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:32 smithi164 conmon[29288]: cluster 2021-02-10T04:29:31.144374+0000 2021-02-10T04:29:32.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:32 smithi164 conmon[29288]: mon.a (mon.0) 835 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2021-02-10T04:29:32.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:32 smithi164 conmon[29288]: cluster 2021-02-10T04:29:31.683206+0000 osd.0 (osd.0) 50 : cluster [DBG] 2.8 scrub ok 2021-02-10T04:29:32.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:32 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:29:32.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:32 smithi164 conmon[29288]: -02-10T04:29:31.902987+0000 osd.4 (osd.4) 41 : cluster [DBG] 6.1e scrub ok 2021-02-10T04:29:32.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:32 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:29:32.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:32 smithi164 conmon[29288]: -02-10T04:29:32.005268+0000 mgr.y 2021-02-10T04:29:32.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:32 smithi164 conmon[29288]: (mgr.14140) 368 : cluster 2021-02-10T04:29:32.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:32 smithi164 conmon[29288]: [DBG] pgmap v382: 168 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 160 active+clean; 134 MiB data, 404 MiB used, 715 GiB / 715 GiB avail; 8.1 MiB/s rd, 447 KiB/s wr, 11 op/s 2021-02-10T04:29:32.435 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:29:32.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:32 smithi085 conmon[29319]: cluster 2021-02-10T04:29:31.144374+0000 2021-02-10T04:29:32.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:32 smithi085 conmon[29319]: mon.a (mon.0) 835 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2021-02-10T04:29:32.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:32 smithi085 conmon[29319]: cluster 2021-02-10T04:29:31.683206+0000 osd.0 (osd.0) 50 : cluster [DBG] 2.8 scrub ok 2021-02-10T04:29:32.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:32 smithi085 conmon[29319]: cluster 2021-02-10T04:29:32.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:32 smithi085 conmon[29319]: 2021-02-10T04:29:31.902987+0000 osd.4 (osd.4) 41 : cluster [DBG] 6.1e scrub ok 2021-02-10T04:29:32.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:32 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:29:32.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:32 smithi085 conmon[29319]: 04:29:32.005268+0000 mgr.y (mgr.14140) 368 : cluster [DBG] pgmap v382: 168 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 160 active+clean; 134 MiB data, 404 MiB used, 715 GiB / 715 GiB avail; 8.1 MiB/s rd, 447 KiB/s wr, 11 op/s 2021-02-10T04:29:32.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:32 smithi085 conmon[35961]: cluster 2021-02-10T04:29:31.144374 2021-02-10T04:29:32.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:32 smithi085 conmon[35961]: +0000 mon.a (mon.0) 835 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2021-02-10T04:29:32.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:32 smithi085 conmon[35961]: cluster 2021-02-10T04:29:32.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:32 smithi085 conmon[35961]: 2021-02-10T04:29:31.683206+0000 osd.0 (osd.0) 50 : cluster [DBG] 2021-02-10T04:29:32.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:32 smithi085 conmon[35961]: 2.8 scrub ok 2021-02-10T04:29:32.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:32 smithi085 conmon[35961]: cluster 2021-02-10T04:29:32.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:32 smithi085 conmon[35961]: 2021-02-10T04:29:31.902987 2021-02-10T04:29:32.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:32 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 41 : cluster 2021-02-10T04:29:32.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:32 smithi085 conmon[35961]: [DBG] 6.1e scrub ok 2021-02-10T04:29:32.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:32 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:29:32.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:32 smithi085 conmon[35961]: 02-10T04:29:32.005268+0000 2021-02-10T04:29:32.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:32 smithi085 conmon[35961]: mgr.y (mgr.14140) 368 : cluster [DBG] 2021-02-10T04:29:32.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:32 smithi085 conmon[35961]: pgmap v382: 168 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 160 active+clean; 134 MiB data, 404 MiB used, 715 GiB / 715 GiB avail; 8.1 MiB/s rd, 447 KiB/s wr, 11 op/s 2021-02-10T04:29:32.443 INFO:tasks.rados.rados.0.smithi085.stdout:329: seq_num 146 ranges {453822=797061,1760371=653366} 2021-02-10T04:29:32.449 INFO:tasks.rados.rados.0.smithi085.stdout:329: writing smithi08575397-15 from 453822 to 1250883 tid 1 2021-02-10T04:29:32.456 INFO:tasks.rados.rados.0.smithi085.stdout:329: writing smithi08575397-15 from 1760371 to 2413737 tid 2 2021-02-10T04:29:32.457 INFO:tasks.rados.rados.0.smithi085.stdout:323: done (9 left) 2021-02-10T04:29:32.457 INFO:tasks.rados.rados.0.smithi085.stdout:330: write (excl) oid 16 current snap is 30 2021-02-10T04:29:32.457 INFO:tasks.rados.rados.0.smithi085.stdout:330: seq_num 147 ranges {470629=472310,1418853=513377} 2021-02-10T04:29:32.462 INFO:tasks.rados.rados.0.smithi085.stdout:330: writing smithi08575397-16 from 470629 to 942939 tid 1 2021-02-10T04:29:32.470 INFO:tasks.rados.rados.0.smithi085.stdout:330: writing smithi08575397-16 from 1418853 to 1932230 tid 2 2021-02-10T04:29:32.472 INFO:tasks.rados.rados.0.smithi085.stdout:331: read oid 31 snap 29 2021-02-10T04:29:32.472 INFO:tasks.rados.rados.0.smithi085.stdout:331: expect deleted 2021-02-10T04:29:32.472 INFO:tasks.rados.rados.0.smithi085.stdout:332: rollback oid 18 current snap is 30 2021-02-10T04:29:32.472 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 18 to 29 2021-02-10T04:29:32.472 INFO:tasks.rados.rados.0.smithi085.stdout:326: finishing write tid 1 to smithi08575397-9 2021-02-10T04:29:32.472 INFO:tasks.rados.rados.0.smithi085.stdout:326: finishing write tid 2 to smithi08575397-9 2021-02-10T04:29:32.472 INFO:tasks.rados.rados.0.smithi085.stdout:326: finishing write tid 3 to smithi08575397-9 2021-02-10T04:29:32.472 INFO:tasks.rados.rados.0.smithi085.stdout:326: finishing write tid 4 to smithi08575397-9 2021-02-10T04:29:32.472 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 9 v 94 (ObjNum 145 snap 30 seq_num 145) dirty exists 2021-02-10T04:29:32.472 INFO:tasks.rados.rados.0.smithi085.stdout:326: left oid 9 (ObjNum 145 snap 30 seq_num 145) 2021-02-10T04:29:32.472 INFO:tasks.rados.rados.0.smithi085.stdout:329: finishing write tid 1 to smithi08575397-15 2021-02-10T04:29:32.472 INFO:tasks.rados.rados.0.smithi085.stdout:329: finishing write tid 2 to smithi08575397-15 2021-02-10T04:29:32.473 INFO:tasks.rados.rados.0.smithi085.stdout:329: finishing write tid 3 to smithi08575397-15 2021-02-10T04:29:32.473 INFO:tasks.rados.rados.0.smithi085.stdout:329: finishing write tid 4 to smithi08575397-15 2021-02-10T04:29:32.473 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 15 v 68 (ObjNum 146 snap 30 seq_num 146) dirty exists 2021-02-10T04:29:32.473 INFO:tasks.rados.rados.0.smithi085.stdout:329: left oid 15 (ObjNum 146 snap 30 seq_num 146) 2021-02-10T04:29:32.473 INFO:tasks.rados.rados.0.smithi085.stdout:325: done (11 left) 2021-02-10T04:29:32.473 INFO:tasks.rados.rados.0.smithi085.stdout:326: done (10 left) 2021-02-10T04:29:32.473 INFO:tasks.rados.rados.0.smithi085.stdout:327: done (9 left) 2021-02-10T04:29:32.473 INFO:tasks.rados.rados.0.smithi085.stdout:329: done (8 left) 2021-02-10T04:29:32.473 INFO:tasks.rados.rados.0.smithi085.stdout:333: copy_from oid 21 from oid 6 current snap is 30 2021-02-10T04:29:32.473 INFO:tasks.rados.rados.0.smithi085.stdout:334: write (excl) oid 5 current snap is 30 2021-02-10T04:29:32.473 INFO:tasks.rados.rados.0.smithi085.stdout:334: seq_num 148 ranges {487436=547559,1760982=657035} 2021-02-10T04:29:32.478 INFO:tasks.rados.rados.0.smithi085.stdout:334: writing smithi08575397-5 from 487436 to 1034995 tid 1 2021-02-10T04:29:32.488 INFO:tasks.rados.rados.0.smithi085.stdout:334: writing smithi08575397-5 from 1760982 to 2418017 tid 2 2021-02-10T04:29:32.488 INFO:tasks.rados.rados.0.smithi085.stdout:335: read oid 14 snap -1 2021-02-10T04:29:32.489 INFO:tasks.rados.rados.0.smithi085.stdout:335: expect (ObjNum 142 snap 29 seq_num 142) 2021-02-10T04:29:32.489 INFO:tasks.rados.rados.0.smithi085.stdout:336: rollback oid 46 current snap is 30 2021-02-10T04:29:32.489 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 46 to 23 2021-02-10T04:29:32.489 INFO:tasks.rados.rados.0.smithi085.stdout:330: finishing write tid 1 to smithi08575397-16 2021-02-10T04:29:32.489 INFO:tasks.rados.rados.0.smithi085.stdout:332: finishing rollback tid 0 to smithi08575397-18 2021-02-10T04:29:32.489 INFO:tasks.rados.rados.0.smithi085.stdout:333: finishing copy_from to smithi08575397-21 2021-02-10T04:29:32.489 INFO:tasks.rados.rados.0.smithi085.stdout:333: got expected ENOENT (src dne) 2021-02-10T04:29:32.489 INFO:tasks.rados.rados.0.smithi085.stdout:333: finishing copy_from racing read to smithi08575397-21 2021-02-10T04:29:32.491 INFO:tasks.rados.rados.0.smithi085.stdout:332: finishing rollback tid 1 to smithi08575397-18 2021-02-10T04:29:32.491 INFO:tasks.rados.rados.0.smithi085.stdout:330: finishing write tid 2 to smithi08575397-16 2021-02-10T04:29:32.491 INFO:tasks.rados.rados.0.smithi085.stdout:330: finishing write tid 3 to smithi08575397-16 2021-02-10T04:29:32.491 INFO:tasks.rados.rados.0.smithi085.stdout:332: finishing rollback tid 2 to smithi08575397-18 2021-02-10T04:29:32.491 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 18 v 57 (ObjNum 99 snap 20 seq_num 99) dirty exists 2021-02-10T04:29:32.491 INFO:tasks.rados.rados.0.smithi085.stdout:330: finishing write tid 4 to smithi08575397-16 2021-02-10T04:29:32.491 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 146 (ObjNum 147 snap 30 seq_num 147) dirty exists 2021-02-10T04:29:32.491 INFO:tasks.rados.rados.0.smithi085.stdout:330: left oid 16 (ObjNum 147 snap 30 seq_num 147) 2021-02-10T04:29:32.491 INFO:tasks.rados.rados.0.smithi085.stdout:334: finishing write tid 1 to smithi08575397-5 2021-02-10T04:29:32.491 INFO:tasks.rados.rados.0.smithi085.stdout:330: done (11 left) 2021-02-10T04:29:32.491 INFO:tasks.rados.rados.0.smithi085.stdout:331: done (10 left) 2021-02-10T04:29:32.491 INFO:tasks.rados.rados.0.smithi085.stdout:332: done (9 left) 2021-02-10T04:29:32.491 INFO:tasks.rados.rados.0.smithi085.stdout:333: done (8 left) 2021-02-10T04:29:32.492 INFO:tasks.rados.rados.0.smithi085.stdout:337: snap_remove snap 29 2021-02-10T04:29:33.041 INFO:tasks.rados.rados.0.smithi085.stdout:335: expect (ObjNum 142 snap 29 seq_num 142) 2021-02-10T04:29:33.148 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:33 smithi085 conmon[35961]: cluster 2021-02-10T04:29:33.148 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:33 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:29:33.148 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:33 smithi085 conmon[35961]: T04:29:31.725928+0000 osd.6 (osd.6) 38 : cluster [DBG] 5.1f scrub ok 2021-02-10T04:29:33.148 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:33 smithi085 conmon[35961]: cluster 2021-02-10T04:29:33.149 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:33 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:29:33.149 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:33 smithi085 conmon[35961]: 10T04:29:31.824918+0000 osd.2 (osd.2 2021-02-10T04:29:33.149 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:33 smithi085 conmon[35961]: ) 42 : cluster [DBG] 6.12 scrub ok 2021-02-10T04:29:33.149 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:33 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:29:33.149 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:33 smithi085 conmon[35961]: -02-10T04:29:32.148031+0000 2021-02-10T04:29:33.149 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:33 smithi085 conmon[35961]: mon.a (mon.0) 836 : cluster [DBG] 2021-02-10T04:29:33.149 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:33 smithi085 conmon[35961]: osdmap e156: 8 total, 8 up, 8 in 2021-02-10T04:29:33.149 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:33 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:29:33.149 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:33 smithi085 conmon[35961]: 04:29:33.041306+0000 mon.a (mon. 2021-02-10T04:29:33.149 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:33 smithi085 conmon[35961]: 0) 837 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2021-02-10T04:29:33.178 INFO:tasks.rados.rados.0.smithi085.stdout:334: finishing write tid 2 to smithi08575397-5 2021-02-10T04:29:33.179 INFO:tasks.rados.rados.0.smithi085.stdout:334: finishing write tid 3 to smithi08575397-5 2021-02-10T04:29:33.179 INFO:tasks.rados.rados.0.smithi085.stdout:336: finishing rollback tid 1 to smithi08575397-46 2021-02-10T04:29:33.179 INFO:tasks.rados.rados.0.smithi085.stdout:334: finishing write tid 4 to smithi08575397-5 2021-02-10T04:29:33.179 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 67 (ObjNum 148 snap 30 seq_num 148) dirty exists 2021-02-10T04:29:33.179 INFO:tasks.rados.rados.0.smithi085.stdout:334: left oid 5 (ObjNum 148 snap 30 seq_num 148) 2021-02-10T04:29:33.179 INFO:tasks.rados.rados.0.smithi085.stdout:336: finishing rollback tid 2 to smithi08575397-46 2021-02-10T04:29:33.179 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 105 (ObjNum 118 snap 23 seq_num 118) dirty exists 2021-02-10T04:29:33.179 INFO:tasks.rados.rados.0.smithi085.stdout:334: done (8 left) 2021-02-10T04:29:33.179 INFO:tasks.rados.rados.0.smithi085.stdout:335: done (7 left) 2021-02-10T04:29:33.179 INFO:tasks.rados.rados.0.smithi085.stdout:336: done (6 left) 2021-02-10T04:29:33.179 INFO:tasks.rados.rados.0.smithi085.stdout:337: done (5 left) 2021-02-10T04:29:33.180 INFO:tasks.rados.rados.0.smithi085.stdout:338: copy_from oid 41 from oid 16 current snap is 30 2021-02-10T04:29:33.180 INFO:tasks.rados.rados.0.smithi085.stdout:339: snap_create 2021-02-10T04:29:33.193 INFO:tasks.rados.rados.0.smithi085.stdout:338: finishing copy_from to smithi08575397-41 2021-02-10T04:29:33.193 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 41 v 146 (ObjNum 147 snap 30 seq_num 147) dirty exists 2021-02-10T04:29:33.193 INFO:tasks.rados.rados.0.smithi085.stdout:338: finishing copy_from racing read to smithi08575397-41 2021-02-10T04:29:33.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:33 smithi085 conmon[29319]: cluster 2021-02-10T04:29:31.725928+0000 osd.6 (osd.6 2021-02-10T04:29:33.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:33 smithi085 conmon[29319]: ) 38 : cluster [DBG] 5.1f scrub ok 2021-02-10T04:29:33.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:33 smithi085 conmon[29319]: cluster 2021-02-10T04:29:31.824918+0000 osd.2 (osd.2) 42 : cluster [DBG] 6.12 scrub ok 2021-02-10T04:29:33.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:33 smithi085 conmon[29319]: cluster 2021-02-10T04:29:32.148031+0000 mon.a (mon.0) 836 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2021-02-10T04:29:33.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:33 smithi085 conmon[29319]: cluster 2021-02-10T04:29:33.041306+0000 mon.a (mon.0) 837 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2021-02-10T04:29:33.445 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:29:33 smithi085 conmon[45443]: debug 2021-02-10T04:29:33.290+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:33.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:33 smithi164 conmon[29288]: cluster 2021-02-10T04:29:31.725928+0000 osd.6 ( 2021-02-10T04:29:33.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:33 smithi164 conmon[29288]: osd.6) 38 : cluster [DBG] 5.1f scrub ok 2021-02-10T04:29:33.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:33 smithi164 conmon[29288]: cluster 2021-02-10T04:29:31.824918+0000 osd.2 (osd.2) 42 : cluster [DBG] 6.12 scrub ok 2021-02-10T04:29:33.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:33 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:29:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:33 smithi164 conmon[29288]: 02-10T04:29:32.148031+0000 mon.a ( 2021-02-10T04:29:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:33 smithi164 conmon[29288]: mon.0) 836 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2021-02-10T04:29:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:33 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:29:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:33 smithi164 conmon[29288]: -02-10T04:29:33.041306+0000 mon.a (mon.0) 837 : cluster 2021-02-10T04:29:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:33 smithi164 conmon[29288]: [DBG] osdmap e157: 8 total, 8 up, 8 in 2021-02-10T04:29:34.152 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:34 smithi085 conmon[29319]: cluster 2021-02-10T04:29:34.152 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:34 smithi085 conmon[29319]: 2021-02-10T04:29:34.152 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:34 smithi085 conmon[29319]: 2021 2021-02-10T04:29:34.153 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:34 smithi085 conmon[29319]: - 2021-02-10T04:29:34.153 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:34 smithi085 conmon[29319]: 02-10T04: 2021-02-10T04:29:34.153 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:34 smithi085 conmon[29319]: 29:33.026981+0000 osd.1 (osd.1) 49 : cluster 2021-02-10T04:29:34.153 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:34 smithi085 conmon[29319]: [DBG] 5.12 scrub ok 2021-02-10T04:29:34.153 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:34 smithi085 conmon[29319]: cluster 2021-02-10T04:29:33.839328 2021-02-10T04:29:34.153 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:34 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 42 : cluster [DBG] 1.5 scrub ok 2021-02-10T04:29:34.153 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:34 smithi085 conmon[29319]: cluster 2021-02-10T04:29:34 2021-02-10T04:29:34.153 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:34 smithi085 conmon[29319]: .005687+0000 mgr.y (mgr.14140) 369 : cluster [DBG] pgmap v385: 168 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 160 active+clean; 138 MiB data, 408 MiB used, 715 GiB / 715 GiB avail; 7.9 MiB/s rd, 1.0 MiB/s wr, 13 op/s 2021-02-10T04:29:34.154 INFO:tasks.rados.rados.0.smithi085.stdout:338: done (6 left) 2021-02-10T04:29:34.154 INFO:tasks.rados.rados.0.smithi085.stdout:339: done (5 left) 2021-02-10T04:29:34.154 INFO:tasks.rados.rados.0.smithi085.stdout:340: copy_from oid 23 from oid 43 current snap is 31 2021-02-10T04:29:34.154 INFO:tasks.rados.rados.0.smithi085.stdout:341: snap_create 2021-02-10T04:29:34.161 INFO:tasks.rados.rados.0.smithi085.stdout:340: finishing copy_from to smithi08575397-23 2021-02-10T04:29:34.161 INFO:tasks.rados.rados.0.smithi085.stdout:340: got expected ENOENT (src dne) 2021-02-10T04:29:34.161 INFO:tasks.rados.rados.0.smithi085.stdout:340: finishing copy_from racing read to smithi08575397-23 2021-02-10T04:29:34.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:34 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:29:34.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:34 smithi085 conmon[35961]: 04:29:33.026981+0000 osd.1 (osd.1) 49 : cluster [DBG] 5.12 scrub ok 2021-02-10T04:29:34.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:34 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:29:34.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:34 smithi085 conmon[35961]: :29:33.839328+0000 osd.4 (osd.4 2021-02-10T04:29:34.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:34 smithi085 conmon[35961]: ) 42 : cluster [DBG] 1.5 scrub ok 2021-02-10T04:29:34.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:34 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:29:34.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:34 smithi085 conmon[35961]: -02-10T04:29:34.005687 2021-02-10T04:29:34.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:34 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 369 : cluster 2021-02-10T04:29:34.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:34 smithi085 conmon[35961]: [DBG] pgmap v385: 168 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 160 active+clean; 138 MiB data, 408 MiB used, 715 GiB / 715 GiB avail; 7.9 MiB/s rd, 1.0 MiB/s wr, 13 op/s 2021-02-10T04:29:34.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:34 smithi164 conmon[29288]: cluster 2021-02-10T04:29:34.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:34 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:29:34.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:34 smithi164 conmon[29288]: 29:33.026981+0000 osd.1 (osd.1) 49 : cluster [DBG] 5.12 scrub ok 2021-02-10T04:29:34.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:34 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:29:34.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:34 smithi164 conmon[29288]: -10T04:29:33.839328 2021-02-10T04:29:34.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:34 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 42 2021-02-10T04:29:34.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:34 smithi164 conmon[29288]: : cluster [DBG] 1.5 scrub ok 2021-02-10T04:29:34.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:34 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:29:34.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:34 smithi164 conmon[29288]: -02-10T04:29:34.005687+0000 2021-02-10T04:29:34.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:34 smithi164 conmon[29288]: mgr.y (mgr.14140) 369 : cluster [DBG] pgmap v385: 168 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 160 active+clean; 138 MiB data, 408 MiB used, 715 GiB / 715 GiB avail; 7.9 MiB/s rd, 1.0 MiB/s wr, 13 op/s 2021-02-10T04:29:34.869 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:29:34.986 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.3 dump_historic_ops 2021-02-10T04:29:35.061 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:29:35.089 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:35.157 INFO:tasks.rados.rados.0.smithi085.stdout:340: done (6 left) 2021-02-10T04:29:35.157 INFO:tasks.rados.rados.0.smithi085.stdout:341: done (5 left) 2021-02-10T04:29:35.157 INFO:tasks.rados.rados.0.smithi085.stdout:342: delete oid 9 current snap is 32 2021-02-10T04:29:35.172 INFO:tasks.rados.rados.0.smithi085.stdout:342: done (5 left) 2021-02-10T04:29:35.173 INFO:tasks.rados.rados.0.smithi085.stdout:343: write oid 22 current snap is 32 2021-02-10T04:29:35.173 INFO:tasks.rados.rados.0.smithi085.stdout:343: seq_num 150 ranges {521050=698057,1877946=643104} 2021-02-10T04:29:35.179 INFO:tasks.rados.rados.0.smithi085.stdout:343: writing smithi08575397-22 from 521050 to 1219107 tid 1 2021-02-10T04:29:35.184 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:29:35.186 INFO:tasks.rados.rados.0.smithi085.stdout:343: writing smithi08575397-22 from 1877946 to 2521050 tid 2 2021-02-10T04:29:35.187 INFO:tasks.rados.rados.0.smithi085.stdout:344: snap_create 2021-02-10T04:29:35.206 INFO:tasks.rados.rados.0.smithi085.stdout:343: finishing write tid 1 to smithi08575397-22 2021-02-10T04:29:35.207 INFO:tasks.rados.rados.0.smithi085.stdout:343: finishing write tid 2 to smithi08575397-22 2021-02-10T04:29:35.207 INFO:tasks.rados.rados.0.smithi085.stdout:343: finishing write tid 3 to smithi08575397-22 2021-02-10T04:29:35.207 INFO:tasks.rados.rados.0.smithi085.stdout:343: finishing write tid 4 to smithi08575397-22 2021-02-10T04:29:35.207 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 76 (ObjNum 150 snap 32 seq_num 150) dirty exists 2021-02-10T04:29:35.208 INFO:tasks.rados.rados.0.smithi085.stdout:343: left oid 22 (ObjNum 150 snap 32 seq_num 150) 2021-02-10T04:29:35.211 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[29319]: cluster 2021-02-10T04:29:33 2021-02-10T04:29:35.211 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[29319]: .747441+0000 osd.6 (osd.6) 39 : cluster [DBG] 6.5 scrub ok 2021-02-10T04:29:35.211 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[29319]: cluster 2021-02-10T04:29:34.154794+0000 mon.a (mon.0) 838 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2021-02-10T04:29:35.211 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[29319]: cluster 2021-02-10T04:29:34.388908+0000 osd.3 (osd.3) 47 : cluster [DBG] 5.0 deep-scrub ok 2021-02-10T04:29:35.211 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: cluster 2021-02-10T04:29:35.211 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: 2021-02-10T04:29:35.211 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: 2021 2021-02-10T04:29:35.211 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: - 2021-02-10T04:29:35.211 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: 02 2021-02-10T04:29:35.212 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: - 2021-02-10T04:29:35.212 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: 10 2021-02-10T04:29:35.212 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: T 2021-02-10T04:29:35.212 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: 04 2021-02-10T04:29:35.212 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: : 2021-02-10T04:29:35.212 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: 29 2021-02-10T04:29:35.212 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: : 2021-02-10T04:29:35.212 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: 33 2021-02-10T04:29:35.212 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: . 2021-02-10T04:29:35.212 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: 747441 2021-02-10T04:29:35.212 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: +0000 2021-02-10T04:29:35.212 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: 2021-02-10T04:29:35.212 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: osd.6 2021-02-10T04:29:35.213 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: ( 2021-02-10T04:29:35.213 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: osd 2021-02-10T04:29:35.213 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: . 2021-02-10T04:29:35.213 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: 6 2021-02-10T04:29:35.213 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: ) 2021-02-10T04:29:35.213 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: 39 2021-02-10T04:29:35.213 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: : cluster [DBG] 6.5 scrub ok 2021-02-10T04:29:35.213 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: cluster 2021-02-10T04:29:35.213 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: 2021-02-10T04:29:34.154794+0000 mon.a (mon 2021-02-10T04:29:35.213 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: .0) 838 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2021-02-10T04:29:35.213 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:29:35.213 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: -02-10T04:29:34 2021-02-10T04:29:35.213 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: .388908+0000 osd.3 (osd. 2021-02-10T04:29:35.214 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: 3) 47 : cluster [DBG] 5.0 deep-scrub ok 2021-02-10T04:29:35.214 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[35961]: 2021-02-10T04:29:35.288 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:29:35.316 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:35.331 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:29:35.384 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:29:35 smithi164 conmon[37263]: debug 2021-02-10T04:29:35.166+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:35.384 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:35 smithi164 conmon[29288]: cluster 2021-02-10T04:29:33.747441+0000 osd.6 (osd.6 2021-02-10T04:29:35.385 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:35 smithi164 conmon[29288]: ) 39 : cluster [DBG] 6.5 scrub ok 2021-02-10T04:29:35.385 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:35 smithi164 conmon[29288]: cluster 2021-02-10T04:29:34.154794+0000 mon.a (mon.0) 838 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2021-02-10T04:29:35.385 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:35 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:29:35.385 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:35 smithi164 conmon[29288]: T04:29:34.388908+0000 osd.3 (osd.3) 47 : cluster [DBG] 5.0 deep-scrub ok 2021-02-10T04:29:35.409 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:29:35.515 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:29:35.542 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:35.643 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:29:35.643 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:29:35 smithi164 conmon[40892]: debug 2021-02-10T04:29:35.383+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:35.643 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:29:35 smithi164 conmon[44544]: debug 2021-02-10T04:29:35.608+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:35.747 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:29:35.784 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:35.927 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:29:35.954 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:29:35.984 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:29:36.010 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:29:36.039 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:29:36.064 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:29:36.091 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:29:36.117 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:29:36.118 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:29:36.118 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:29:36.118 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:29:36.119 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:29:36.119 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:29:36.119 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:29:36.119 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:29:36.119 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:29:36.174 INFO:tasks.rados.rados.0.smithi085.stdout:343: done (6 left) 2021-02-10T04:29:36.174 INFO:tasks.rados.rados.0.smithi085.stdout:344: done (5 left) 2021-02-10T04:29:36.174 INFO:tasks.rados.rados.0.smithi085.stdout:345: write (excl) oid 22 current snap is 33 2021-02-10T04:29:36.174 INFO:tasks.rados.rados.0.smithi085.stdout:345: seq_num 151 ranges {537857=773306,1936428=520715} 2021-02-10T04:29:36.181 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.0 to 1 (10000) 2021-02-10T04:29:36.181 INFO:tasks.rados.rados.0.smithi085.stdout:345: writing smithi08575397-22 from 537857 to 1311163 tid 1 2021-02-10T04:29:36.190 INFO:tasks.rados.rados.0.smithi085.stdout:345: writing smithi08575397-22 from 1936428 to 2457143 tid 2 2021-02-10T04:29:36.190 INFO:tasks.rados.rados.0.smithi085.stdout:345: finishing write tid 1 to smithi08575397-22 2021-02-10T04:29:36.190 INFO:tasks.rados.rados.0.smithi085.stdout:346: copy_from oid 40 from oid 3 current snap is 33 2021-02-10T04:29:36.190 INFO:tasks.rados.rados.0.smithi085.stdout:347: read oid 39 snap -1 2021-02-10T04:29:36.190 INFO:tasks.rados.rados.0.smithi085.stdout:347: expect (ObjNum 103 snap 20 seq_num 103) 2021-02-10T04:29:36.190 INFO:tasks.rados.rados.0.smithi085.stdout:348: rollback oid 43 current snap is 33 2021-02-10T04:29:36.191 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 43 to 30 2021-02-10T04:29:36.191 INFO:tasks.rados.rados.0.smithi085.stdout:349: delete oid 13 current snap is 33 2021-02-10T04:29:36.191 INFO:tasks.rados.rados.0.smithi085.stdout:348: finishing rollback tid 1 to smithi08575397-43 2021-02-10T04:29:36.191 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 0 (ObjNum 82608288 snap 22030 seq_num 1006635216) dirty dne 2021-02-10T04:29:36.192 INFO:tasks.rados.rados.0.smithi085.stdout:346: finishing copy_from to smithi08575397-40 2021-02-10T04:29:36.192 INFO:tasks.rados.rados.0.smithi085.stdout:346: got expected ENOENT (src dne) 2021-02-10T04:29:36.192 INFO:tasks.rados.rados.0.smithi085.stdout:346: finishing copy_from racing read to smithi08575397-40 2021-02-10T04:29:36.198 INFO:tasks.rados.rados.0.smithi085.stdout:346: done (9 left) 2021-02-10T04:29:36.198 INFO:tasks.rados.rados.0.smithi085.stdout:348: done (8 left) 2021-02-10T04:29:36.198 INFO:tasks.rados.rados.0.smithi085.stdout:349: done (7 left) 2021-02-10T04:29:36.198 INFO:tasks.rados.rados.0.smithi085.stdout:350: snap_create 2021-02-10T04:29:36.198 INFO:tasks.rados.rados.0.smithi085.stdout:345: finishing write tid 2 to smithi08575397-22 2021-02-10T04:29:36.199 INFO:tasks.rados.rados.0.smithi085.stdout:345: finishing write tid 3 to smithi08575397-22 2021-02-10T04:29:36.199 INFO:tasks.rados.rados.0.smithi085.stdout:345: finishing write tid 4 to smithi08575397-22 2021-02-10T04:29:36.199 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 80 (ObjNum 151 snap 33 seq_num 151) dirty exists 2021-02-10T04:29:36.199 INFO:tasks.rados.rados.0.smithi085.stdout:345: left oid 22 (ObjNum 151 snap 33 seq_num 151) 2021-02-10T04:29:36.200 INFO:tasks.rados.rados.0.smithi085.stdout:347: expect (ObjNum 103 snap 20 seq_num 103) 2021-02-10T04:29:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[29319]: cluster 2021-02-10T04:29:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[29319]: 2021-02-10T04:29:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[29319]: 2021 2021-02-10T04:29:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[29319]: - 2021-02-10T04:29:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[29319]: 02 2021-02-10T04:29:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[29319]: - 2021-02-10T04:29:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[29319]: 10T04:29:34.787262+0000 2021-02-10T04:29:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[29319]: osd.2 (osd.2) 43 : cluster [DBG] 2.e scrub ok 2021-02-10T04:29:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[29319]: cluster 2021-02-10T04:29:36.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[29319]: 2021-02-10T04:29:35 2021-02-10T04:29:36.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[29319]: .156004+0000 mon.a (mon.0 2021-02-10T04:29:36.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[29319]: ) 839 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2021-02-10T04:29:36.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:29:36.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[29319]: 10T04:29:35.382334+0000 osd.3 (osd.3) 48 : cluster [DBG] 3.2 scrub ok 2021-02-10T04:29:36.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[29319]: cluster 2021-02-10T04:29:35.711554 2021-02-10T04:29:36.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[29319]: +0000 osd.0 (osd.0) 51 : cluster [DBG] 2.5 deep-scrub ok 2021-02-10T04:29:36.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[29319]: audit 2021-02-10T04:29:36.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[29319]: 2021 2021-02-10T04:29:36.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[29319]: -02-10T04:29:35 2021-02-10T04:29:36.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[29319]: .764651+0000 mon.a (mon.0) 840 : audit [INF] from='client.? 172.21.15.85:0/2086686276' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T04:29:36.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:29:36.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[29319]: 02-10T04:29:35.901185+0000 mon.c (mon 2021-02-10T04:29:36.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[29319]: .1) 44 : audit [DBG] from='client.? 172.21.15.85:0/2918351861' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:29:36.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:29:36.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[29319]: T04:29:36.006047+0000 mgr.y (mgr.14140) 2021-02-10T04:29:36.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[29319]: 370 : cluster [DBG] pgmap v388: 168 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 161 active+clean; 129 MiB data, 393 MiB used, 715 GiB / 715 GiB avail; 6.1 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2021-02-10T04:29:36.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[35961]: cluster 2021-02-10T04:29:36.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[35961]: 2021-02 2021-02-10T04:29:36.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[35961]: -10T04:29:34.787262+0000 osd.2 (osd.2) 43 : cluster [DBG] 2.e scrub ok 2021-02-10T04:29:36.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[35961]: 2021-02-10T04:29:36.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:29:36.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[35961]: 10T04:29:35.156004+0000 mon.a (mon.0) 839 : 2021-02-10T04:29:36.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[35961]: cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2021-02-10T04:29:36.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:29:36.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[35961]: 29:35.382334+0000 osd.3 (osd.3) 48 2021-02-10T04:29:36.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[35961]: : cluster [DBG] 3.2 scrub ok 2021-02-10T04:29:36.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:29:36.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[35961]: 10T04:29:35.711554+0000 osd.0 ( 2021-02-10T04:29:36.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[35961]: osd.0) 51 : cluster [DBG] 2.5 deep-scrub ok 2021-02-10T04:29:36.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[35961]: audit 2021-02-10T04:29:35.764651 2021-02-10T04:29:36.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[35961]: +0000 mon.a (mon.0) 840 : audit [INF] 2021-02-10T04:29:36.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/2086686276' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T04:29:36.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[35961]: audit 2021-02-10T04:29:35. 2021-02-10T04:29:36.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[35961]: 901185+0000 mon.c (mon.1) 44 : audit 2021-02-10T04:29:36.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[35961]: [DBG] from='client.? 172.21.15.85:0/2918351861' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:29:36.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:29:36.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[35961]: :29:36.006047+0000 mgr.y (mgr. 2021-02-10T04:29:36.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:36 smithi085 conmon[35961]: 14140) 370 : cluster [DBG] pgmap v388: 168 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 161 active+clean; 129 MiB data, 393 MiB used, 715 GiB / 715 GiB avail; 6.1 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2021-02-10T04:29:36.445 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:29:35 smithi085 conmon[41540]: debug 2021-02-10T04:29:35.964+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:36.577 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:29:36.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:36 smithi164 conmon[29288]: cluster 2021-02-10T04:29:36.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:36 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:29:36.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:36 smithi164 conmon[29288]: 29:34.787262+0000 osd.2 (osd.2) 43 : cluster [DBG] 2.e scrub ok 2021-02-10T04:29:36.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:36 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:29:36.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:36 smithi164 conmon[29288]: -10T04:29:35.156004+0000 2021-02-10T04:29:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:36 smithi164 conmon[29288]: mon.a (mon.0) 839 : cluster [DBG] 2021-02-10T04:29:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:36 smithi164 conmon[29288]: osdmap e159: 8 total, 8 up, 8 in 2021-02-10T04:29:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:36 smithi164 conmon[29288]: cluster 2021-02-10T04:29 2021-02-10T04:29:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:36 smithi164 conmon[29288]: :35.382334+0000 osd.3 (osd.3) 48 : cluster [DBG] 3.2 scrub ok 2021-02-10T04:29:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:36 smithi164 conmon[29288]: 2021-02-10T04:29:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:36 smithi164 conmon[29288]: cluster 2021-02-10T04:29 2021-02-10T04:29:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:36 smithi164 conmon[29288]: :35.711554+0000 osd.0 (osd.0) 51 : cluster [DBG] 2.5 deep-scrub ok 2021-02-10T04:29:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:36 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:29:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:36 smithi164 conmon[29288]: T04:29:35.764651+0000 mon.a (mon.0) 840 : audit 2021-02-10T04:29:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:36 smithi164 conmon[29288]: [INF] from='client.? 172.21.15.85:0/2086686276' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T04:29:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:36 smithi164 conmon[29288]: audit 2021-02-10T04:29:35. 2021-02-10T04:29:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:36 smithi164 conmon[29288]: 901185+0000 mon.c (mon.1) 44 : audit [DBG] 2021-02-10T04:29:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:36 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/2918351861' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:29:36.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:36 smithi164 conmon[29288]: cluster 2021-02-10T04:29:36.006047 2021-02-10T04:29:36.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:36 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 370 : cluster [DBG] 2021-02-10T04:29:36.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:36 smithi164 conmon[29288]: pgmap v388: 168 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 161 active+clean; 129 MiB data, 393 MiB used, 715 GiB / 715 GiB avail; 6.1 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2021-02-10T04:29:37.181 INFO:tasks.rados.rados.0.smithi085.stdout:345: done (7 left) 2021-02-10T04:29:37.181 INFO:tasks.rados.rados.0.smithi085.stdout:347: done (6 left) 2021-02-10T04:29:37.182 INFO:tasks.rados.rados.0.smithi085.stdout:350: done (5 left) 2021-02-10T04:29:37.182 INFO:tasks.rados.rados.0.smithi085.stdout:351: read oid 28 snap -1 2021-02-10T04:29:37.182 INFO:tasks.rados.rados.0.smithi085.stdout:351: expect (ObjNum 110 snap 23 seq_num 110) 2021-02-10T04:29:37.184 INFO:tasks.rados.rados.0.smithi085.stdout:352: snap_remove snap 32 2021-02-10T04:29:37.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[29319]: cluster 2021-02-10T04:29: 2021-02-10T04:29:37.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[29319]: 35.805319+0000 osd.2 (osd.2) 44 : cluster [DBG] 5.a scrub ok 2021-02-10T04:29:37.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[29319]: cluster 2021-02-10T04:29:35.958247+0000 osd.1 (osd.1) 50 : cluster [DBG] 5.c scrub ok 2021-02-10T04:29:37.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[29319]: 2021-02-10T04:29:37.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[29319]: audit 2021-02-10T04: 2021-02-10T04:29:37.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[29319]: 29:36.173120+0000 mon.a (mon.0) 841 : audit 2021-02-10T04:29:37.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[29319]: [INF] from='client.? 172.21.15.85:0/2086686276' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2021-02-10T04:29:37.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[29319]: cluster 2021-02-10T04:29:36 2021-02-10T04:29:37.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[29319]: .173199+0000 mon.a (mon.0) 842 : cluster 2021-02-10T04:29:37.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[29319]: [DBG] 2021-02-10T04:29:37.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[29319]: osdmap e160: 8 total, 8 up, 8 in 2021-02-10T04:29:37.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[29319]: cluster 2021-02-10T04:29:37.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[29319]: 2021-02 2021-02-10T04:29:37.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[29319]: -10T04:29: 2021-02-10T04:29:37.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[29319]: 36.395937+0000 osd.3 (osd.3) 49 : cluster 2021-02-10T04:29:37.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[29319]: [DBG] 6.6 scrub ok 2021-02-10T04:29:37.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:29:37.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[29319]: 29:36.663835+0000 osd.0 (osd.0) 52 : 2021-02-10T04:29:37.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[29319]: cluster [DBG] 1.0 scrub ok 2021-02-10T04:29:37.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:29:37.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[29319]: :29:36.813074+0000 2021-02-10T04:29:37.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[29319]: osd.4 (osd.4) 43 : 2021-02-10T04:29:37.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[29319]: cluster [DBG] 4.1 scrub ok 2021-02-10T04:29:37.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[35961]: cluster 2021-02-10T04:29:35.805319+0000 2021-02-10T04:29:37.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[35961]: osd.2 (osd.2) 44 : cluster [DBG] 5.a scrub ok 2021-02-10T04:29:37.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:29:37.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[35961]: -02-10T04:29:35.958247+0000 osd.1 (osd.1) 50 : cluster [DBG] 5.c scrub ok 2021-02-10T04:29:37.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:29:37.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[35961]: T04:29:36.173120+0000 mon.a (mon.0) 841 : audit 2021-02-10T04:29:37.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[35961]: [INF] from='client.? 172.21.15.85:0/2086686276' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2021-02-10T04:29:37.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:29:37.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[35961]: 29:36.173199+0000 mon.a (mon.0) 842 2021-02-10T04:29:37.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[35961]: : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2021-02-10T04:29:37.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:29:37.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[35961]: 02-10T04:29:36.395937+0000 2021-02-10T04:29:37.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[35961]: osd.3 (osd.3) 49 : 2021-02-10T04:29:37.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[35961]: cluster [DBG] 6.6 scrub ok 2021-02-10T04:29:37.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[35961]: cluster 2021-02-10T04:29 2021-02-10T04:29:37.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[35961]: :36.663835+0000 osd.0 (osd.0) 52 : cluster [DBG] 1.0 scrub ok 2021-02-10T04:29:37.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:29:37.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[35961]: T04:29:36.813074+0000 osd.4 (osd.4) 43 2021-02-10T04:29:37.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:37 smithi085 conmon[35961]: : cluster [DBG] 4.1 scrub ok 2021-02-10T04:29:37.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:37 smithi164 conmon[29288]: cluster 2021-02-10T04:29:35.805319+0000 osd.2 (osd.2) 44 2021-02-10T04:29:37.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:37 smithi164 conmon[29288]: : cluster [DBG] 5.a scrub ok 2021-02-10T04:29:37.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:37 smithi164 conmon[29288]: cluster 2021-02-10T04:29:35.958247+0000 osd.1 (osd.1) 50 : cluster [DBG] 5.c scrub ok 2021-02-10T04:29:37.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:37 smithi164 conmon[29288]: audit 2021-02-10T04:29:36.173120+0000 2021-02-10T04:29:37.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:37 smithi164 conmon[29288]: mon.a (mon.0) 841 : audit 2021-02-10T04:29:37.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:37 smithi164 conmon[29288]: [INF] from='client.? 172.21.15.85:0/2086686276' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2021-02-10T04:29:37.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:37 smithi164 conmon[29288]: cluster 2021-02-10T04:29:37.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:37 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:29:37.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:37 smithi164 conmon[29288]: 29:36.173199+0000 mon.a (mon.0) 842 2021-02-10T04:29:37.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:37 smithi164 conmon[29288]: : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2021-02-10T04:29:37.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:37 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:29:37.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:37 smithi164 conmon[29288]: :29:36.395937+0000 osd.3 (osd.3) 49 : cluster 2021-02-10T04:29:37.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:37 smithi164 conmon[29288]: [DBG] 6.6 scrub ok 2021-02-10T04:29:37.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:37 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:29:37.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:37 smithi164 conmon[29288]: 10T04:29:36.663835+0000 osd.0 ( 2021-02-10T04:29:37.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:37 smithi164 conmon[29288]: osd.0) 52 : cluster [DBG] 1.0 scrub ok 2021-02-10T04:29:37.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:37 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:29:37.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:37 smithi164 conmon[29288]: -02-10T04:29:36.813074+0000 2021-02-10T04:29:37.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:37 smithi164 conmon[29288]: osd.4 (osd.4) 43 : cluster [DBG] 4.1 scrub ok 2021-02-10T04:29:37.912 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:29:37.945 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:38.015 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:29:37 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:29:38.042 INFO:tasks.rados.rados.0.smithi085.stdout:352: done (6 left) 2021-02-10T04:29:38.042 INFO:tasks.rados.rados.0.smithi085.stdout:353: copy_from oid 10 from oid 21 current snap is 34 2021-02-10T04:29:38.043 INFO:tasks.rados.rados.0.smithi085.stdout:354: read oid 13 snap -1 2021-02-10T04:29:38.043 INFO:tasks.rados.rados.0.smithi085.stdout:354: expect deleted 2021-02-10T04:29:38.043 INFO:tasks.rados.rados.0.smithi085.stdout:355: write (excl) oid 22 current snap is 34 2021-02-10T04:29:38.043 INFO:tasks.rados.rados.0.smithi085.stdout:355: seq_num 153 ranges {571471=640157,1653392=524384} 2021-02-10T04:29:38.043 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:29:38.050 INFO:tasks.rados.rados.0.smithi085.stdout:355: writing smithi08575397-22 from 571471 to 1211628 tid 1 2021-02-10T04:29:38.057 INFO:tasks.rados.rados.0.smithi085.stdout:355: writing smithi08575397-22 from 1653392 to 2177776 tid 2 2021-02-10T04:29:38.057 INFO:tasks.rados.rados.0.smithi085.stdout:356: write oid 46 current snap is 34 2021-02-10T04:29:38.057 INFO:tasks.rados.rados.0.smithi085.stdout:356: seq_num 154 ranges {588278=715406,1711874=668042} 2021-02-10T04:29:38.065 INFO:tasks.rados.rados.0.smithi085.stdout:356: writing smithi08575397-46 from 588278 to 1303684 tid 1 2021-02-10T04:29:38.072 INFO:tasks.rados.rados.0.smithi085.stdout:356: writing smithi08575397-46 from 1711874 to 2379916 tid 2 2021-02-10T04:29:38.072 INFO:tasks.rados.rados.0.smithi085.stdout:357: write oid 15 current snap is 34 2021-02-10T04:29:38.072 INFO:tasks.rados.rados.0.smithi085.stdout:351: expect (ObjNum 110 snap 23 seq_num 110) 2021-02-10T04:29:38.149 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:29:38.172 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[29319]: cluster 2021-02-10T04:29:36.476735+0000 osd.7 (osd.7) 40 : cluster 2021-02-10T04:29:38.172 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[29319]: [DBG] purged_snaps scrub starts 2021-02-10T04:29:38.172 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[29319]: cluster 2021-02-10T04:29:36.476850+0000 osd.7 (osd.7) 41 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:29:38.172 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[29319]: cluster 2021-02-10T04:29:36.866815+0000 osd.6 (osd.6) 40 : cluster [DBG] 6.15 scrub ok 2021-02-10T04:29:38.172 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:29:38.184 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:38.300 INFO:tasks.rados.rados.0.smithi085.stdout:357: seq_num 155 ranges {605085=790655,2170356=434729} 2021-02-10T04:29:38.308 INFO:tasks.rados.rados.0.smithi085.stdout:357: writing smithi08575397-15 from 605085 to 1395740 tid 1 2021-02-10T04:29:38.316 INFO:tasks.rados.rados.0.smithi085.stdout:357: writing smithi08575397-15 from 2170356 to 2605085 tid 2 2021-02-10T04:29:38.316 INFO:tasks.rados.rados.0.smithi085.stdout:351: done (10 left) 2021-02-10T04:29:38.316 INFO:tasks.rados.rados.0.smithi085.stdout:358: snap_create 2021-02-10T04:29:38.317 INFO:tasks.rados.rados.0.smithi085.stdout:355: finishing write tid 1 to smithi08575397-22 2021-02-10T04:29:38.317 INFO:tasks.rados.rados.0.smithi085.stdout:353: finishing copy_from to smithi08575397-10 2021-02-10T04:29:38.317 INFO:tasks.rados.rados.0.smithi085.stdout:353: got expected ENOENT (src dne) 2021-02-10T04:29:38.317 INFO:tasks.rados.rados.0.smithi085.stdout:353: finishing copy_from racing read to smithi08575397-10 2021-02-10T04:29:38.317 INFO:tasks.rados.rados.0.smithi085.stdout:355: finishing write tid 2 to smithi08575397-22 2021-02-10T04:29:38.317 INFO:tasks.rados.rados.0.smithi085.stdout:355: finishing write tid 3 to smithi08575397-22 2021-02-10T04:29:38.317 INFO:tasks.rados.rados.0.smithi085.stdout:355: finishing write tid 4 to smithi08575397-22 2021-02-10T04:29:38.317 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 86 (ObjNum 153 snap 34 seq_num 153) dirty exists 2021-02-10T04:29:38.317 INFO:tasks.rados.rados.0.smithi085.stdout:355: left oid 22 (ObjNum 153 snap 34 seq_num 153) 2021-02-10T04:29:38.317 INFO:tasks.rados.rados.0.smithi085.stdout:356: finishing write tid 1 to smithi08575397-46 2021-02-10T04:29:38.317 INFO:tasks.rados.rados.0.smithi085.stdout:356: finishing write tid 2 to smithi08575397-46 2021-02-10T04:29:38.317 INFO:tasks.rados.rados.0.smithi085.stdout:356: finishing write tid 3 to smithi08575397-46 2021-02-10T04:29:38.317 INFO:tasks.rados.rados.0.smithi085.stdout:356: finishing write tid 4 to smithi08575397-46 2021-02-10T04:29:38.318 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 111 (ObjNum 154 snap 34 seq_num 154) dirty exists 2021-02-10T04:29:38.318 INFO:tasks.rados.rados.0.smithi085.stdout:356: left oid 46 (ObjNum 154 snap 34 seq_num 154) 2021-02-10T04:29:38.318 INFO:tasks.rados.rados.0.smithi085.stdout:357: finishing write tid 1 to smithi08575397-15 2021-02-10T04:29:38.320 INFO:tasks.rados.rados.0.smithi085.stdout:357: finishing write tid 2 to smithi08575397-15 2021-02-10T04:29:38.320 INFO:tasks.rados.rados.0.smithi085.stdout:357: finishing write tid 3 to smithi08575397-15 2021-02-10T04:29:38.320 INFO:tasks.rados.rados.0.smithi085.stdout:357: finishing write tid 4 to smithi08575397-15 2021-02-10T04:29:38.321 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 15 v 74 (ObjNum 155 snap 34 seq_num 155) dirty exists 2021-02-10T04:29:38.321 INFO:tasks.rados.rados.0.smithi085.stdout:357: left oid 15 (ObjNum 155 snap 34 seq_num 155) 2021-02-10T04:29:38.393 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:29:38 smithi164 conmon[40892]: debug 2021-02-10T04:29:38.014+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:38.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:38 smithi164 conmon[29288]: cluster 2021-02-10T04:29:36.476735+0000 2021-02-10T04:29:38.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:38 smithi164 conmon[29288]: osd.7 (osd.7) 40 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:29:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:38 smithi164 conmon[29288]: cluster 2021-02-10T04:29:36.476850+0000 osd.7 (osd.7) 41 : cluster [DBG] 2021-02-10T04:29:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:38 smithi164 conmon[29288]: purged_snaps scrub ok 2021-02-10T04:29:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:38 smithi164 conmon[29288]: cluster 2021-02-10T04:29: 2021-02-10T04:29:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:38 smithi164 conmon[29288]: 36.866815+0000 osd.6 (osd.6) 40 : cluster [DBG] 6.15 scrub ok 2021-02-10T04:29:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:38 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:29:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:38 smithi164 conmon[29288]: -10T04:29:37.177556+0000 mon.a (mon.0) 2021-02-10T04:29:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:38 smithi164 conmon[29288]: 843 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2021-02-10T04:29:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:38 smithi164 conmon[29288]: cluster 2021-02-10T04:29:37.336085 2021-02-10T04:29:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:38 smithi164 conmon[29288]: +0000 osd.5 (osd.5) 45 : cluster [DBG] 1.8 scrub ok 2021-02-10T04:29:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:38 smithi164 conmon[29288]: cluster 2021-02-10T04:29:37.364692+0000 osd.3 (osd.3) 50 : cluster [DBG] 6.b scrub ok 2021-02-10T04:29:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:38 smithi164 conmon[29288]: audit 2021-02-10T04:29:37.735064+0000 mgr.y (mgr. 2021-02-10T04:29:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:38 smithi164 conmon[29288]: 14140) 371 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:29:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:38 smithi164 conmon[29288]: cluster 2021-02-10T04:29:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:38 smithi164 conmon[29288]: 2021-02-10T04:29:38.006501+0000 mgr.y (mgr.14140) 372 : cluster [DBG] pgmap v391: 168 pgs: 5 active+clean+snaptrim, 163 active+clean; 130 MiB data, 389 MiB used, 715 GiB / 715 GiB avail; 6.3 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2021-02-10T04:29:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:38 smithi164 conmon[29288]: cluster 2021-02-10T04:29: 2021-02-10T04:29:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:38 smithi164 conmon[29288]: 38.042221+0000 mon.a (mon.0) 844 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2021-02-10T04:29:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:38 smithi164 conmon[29288]: audit 2021-02-10T04:29 2021-02-10T04:29:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:38 smithi164 conmon[29288]: :38.048610+0000 mon.a (mon.0) 845 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:29:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:38 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:29:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:38 smithi164 conmon[29288]: T04:29:38.053200+0000 mon.a (mon 2021-02-10T04:29:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:38 smithi164 conmon[29288]: .0) 846 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:29:38.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[29319]: -10T04:29:37 2021-02-10T04:29:38.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[29319]: .177556+0000 mon.a (mon.0) 843 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2021-02-10T04:29:38.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:29:38.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[29319]: -02-10T04:29:37.336085+0000 osd.5 (osd. 2021-02-10T04:29:38.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[29319]: 5) 45 : cluster [DBG] 1.8 scrub ok 2021-02-10T04:29:38.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:29:38.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[29319]: -02-10T04:29:37.364692+0000 osd.3 (osd. 2021-02-10T04:29:38.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[29319]: 3) 50 : cluster [DBG] 6.b scrub ok 2021-02-10T04:29:38.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[29319]: audit 2021-02-10T04:29 2021-02-10T04:29:38.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[29319]: :37.735064+0000 mgr.y (mgr.14140) 371 2021-02-10T04:29:38.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[29319]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:29:38.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[29319]: cluster 2021-02-10T04:29:38 2021-02-10T04:29:38.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[29319]: .006501+0000 mgr.y (mgr.14140) 372 : cluster 2021-02-10T04:29:38.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[29319]: [DBG] pgmap v391: 168 pgs: 5 active+clean+snaptrim, 163 active+clean; 130 MiB data, 389 MiB used, 715 GiB / 715 GiB avail; 6.3 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2021-02-10T04:29:38.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[29319]: cluster 2021-02-10T04:29:38. 2021-02-10T04:29:38.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[29319]: 042221+0000 mon.a (mon.0) 844 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2021-02-10T04:29:38.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:29:38.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[29319]: T04:29:38.048610+0000 mon.a (mon.0) 845 2021-02-10T04:29:38.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[29319]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:29:38.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[29319]: audit 2021-02-10T04:29 2021-02-10T04:29:38.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[29319]: :38.053200+0000 mon.a (mon.0) 2021-02-10T04:29:38.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[29319]: 846 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:29:38.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: cluster 2021-02-10T04:29:36.476735+0000 osd.7 ( 2021-02-10T04:29:38.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: osd.7) 40 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:29:38.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:29:38.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: T04:29:36.476850+0000 osd.7 (osd.7) 41 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:29:38.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: cluster 2021-02-10T04:29:38.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: 2021-02-10T04:29:36.866815+0000 2021-02-10T04:29:38.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: osd.6 (osd.6) 40 : cluster [DBG] 6.15 scrub ok 2021-02-10T04:29:38.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:29:38.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: 02-10T04:29:37.177556+0000 mon.a (mon.0) 843 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2021-02-10T04:29:38.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: cluster 2021-02-10T04:29:37.336085+0000 2021-02-10T04:29:38.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: osd.5 (osd.5) 45 : cluster [DBG] 1.8 scrub ok 2021-02-10T04:29:38.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:29:38.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: -02-10T04:29:37.364692+0000 2021-02-10T04:29:38.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: osd.3 (osd.3) 50 : cluster [DBG] 6.b scrub ok 2021-02-10T04:29:38.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: 2021-02-10T04:29:38.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: audit 2021-02-10T04:29:38.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: 2021-02-10T04:29:37.735064+0000 mgr.y (mgr 2021-02-10T04:29:38.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: .14140) 371 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:29:38.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:29:38.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: -10T04:29:38.006501+0000 mgr.y (mgr 2021-02-10T04:29:38.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: .14140) 372 : cluster [DBG] pgmap v391: 168 pgs: 5 active+clean+snaptrim, 163 active+clean; 130 MiB data, 389 MiB used, 715 GiB / 715 GiB avail; 6.3 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2021-02-10T04:29:38.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:29:38.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: 02-10T04:29:38.042221 2021-02-10T04:29:38.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: +0000 mon.a (mon. 2021-02-10T04:29:38.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: 0) 844 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2021-02-10T04:29:38.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: audit 2021-02-10T04:29:38.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: 2021-02 2021-02-10T04:29:38.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: -10T04:29:38.048610+0000 mon.a ( 2021-02-10T04:29:38.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: mon.0) 845 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:29:38.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:29:38.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: T04:29:38.053200+0000 mon.a (mon 2021-02-10T04:29:38.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:38 smithi085 conmon[35961]: .0) 846 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:29:39.188 INFO:tasks.rados.rados.0.smithi085.stdout:353: done (10 left) 2021-02-10T04:29:39.188 INFO:tasks.rados.rados.0.smithi085.stdout:354: done (9 left) 2021-02-10T04:29:39.188 INFO:tasks.rados.rados.0.smithi085.stdout:355: done (8 left) 2021-02-10T04:29:39.188 INFO:tasks.rados.rados.0.smithi085.stdout:356: done (7 left) 2021-02-10T04:29:39.188 INFO:tasks.rados.rados.0.smithi085.stdout:357: done (6 left) 2021-02-10T04:29:39.188 INFO:tasks.rados.rados.0.smithi085.stdout:358: done (5 left) 2021-02-10T04:29:39.188 INFO:tasks.rados.rados.0.smithi085.stdout:359: read oid 9 snap -1 2021-02-10T04:29:39.188 INFO:tasks.rados.rados.0.smithi085.stdout:359: expect deleted 2021-02-10T04:29:39.188 INFO:tasks.rados.rados.0.smithi085.stdout:360: delete oid 50 current snap is 35 2021-02-10T04:29:39.192 INFO:tasks.rados.rados.0.smithi085.stdout:360: done (6 left) 2021-02-10T04:29:39.192 INFO:tasks.rados.rados.0.smithi085.stdout:361: rollback oid 22 current snap is 35 2021-02-10T04:29:39.192 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 22 to 30 2021-02-10T04:29:39.192 INFO:tasks.rados.rados.0.smithi085.stdout:362: write (excl) oid 16 current snap is 35 2021-02-10T04:29:39.192 INFO:tasks.rados.rados.0.smithi085.stdout:362: seq_num 157 ranges {638699=541153,1770967=531722} 2021-02-10T04:29:39.199 INFO:tasks.rados.rados.0.smithi085.stdout:362: writing smithi08575397-16 from 638699 to 1179852 tid 1 2021-02-10T04:29:39.207 INFO:tasks.rados.rados.0.smithi085.stdout:362: writing smithi08575397-16 from 1770967 to 2302689 tid 2 2021-02-10T04:29:39.207 INFO:tasks.rados.rados.0.smithi085.stdout:363: write oid 46 current snap is 35 2021-02-10T04:29:39.207 INFO:tasks.rados.rados.0.smithi085.stdout:361: finishing rollback tid 0 to smithi08575397-22 2021-02-10T04:29:39.207 INFO:tasks.rados.rados.0.smithi085.stdout:361: finishing rollback tid 1 to smithi08575397-22 2021-02-10T04:29:39.207 INFO:tasks.rados.rados.0.smithi085.stdout:361: finishing rollback tid 2 to smithi08575397-22 2021-02-10T04:29:39.207 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 90 (ObjNum 127 snap 23 seq_num 127) dirty exists 2021-02-10T04:29:39.207 INFO:tasks.rados.rados.0.smithi085.stdout:362: finishing write tid 1 to smithi08575397-16 2021-02-10T04:29:39.207 INFO:tasks.rados.rados.0.smithi085.stdout:363: seq_num 158 ranges {655506=616402,1829449=675380} 2021-02-10T04:29:39.214 INFO:tasks.rados.rados.0.smithi085.stdout:363: writing smithi08575397-46 from 655506 to 1271908 tid 1 2021-02-10T04:29:39.222 INFO:tasks.rados.rados.0.smithi085.stdout:363: writing smithi08575397-46 from 1829449 to 2504829 tid 2 2021-02-10T04:29:39.222 INFO:tasks.rados.rados.0.smithi085.stdout:359: done (8 left) 2021-02-10T04:29:39.222 INFO:tasks.rados.rados.0.smithi085.stdout:361: done (7 left) 2021-02-10T04:29:39.222 INFO:tasks.rados.rados.0.smithi085.stdout:364: write (excl) oid 40 current snap is 35 2021-02-10T04:29:39.223 INFO:tasks.rados.rados.0.smithi085.stdout:362: finishing write tid 2 to smithi08575397-16 2021-02-10T04:29:39.223 INFO:tasks.rados.rados.0.smithi085.stdout:364: seq_num 159 ranges {672313=691651,1887931=419038} 2021-02-10T04:29:39.230 INFO:tasks.rados.rados.0.smithi085.stdout:364: writing smithi08575397-40 from 672313 to 1363964 tid 1 2021-02-10T04:29:39.237 INFO:tasks.rados.rados.0.smithi085.stdout:364: writing smithi08575397-40 from 1887931 to 2306969 tid 2 2021-02-10T04:29:39.237 INFO:tasks.rados.rados.0.smithi085.stdout:365: snap_create 2021-02-10T04:29:39.238 INFO:tasks.rados.rados.0.smithi085.stdout:362: finishing write tid 3 to smithi08575397-16 2021-02-10T04:29:39.238 INFO:tasks.rados.rados.0.smithi085.stdout:362: finishing write tid 4 to smithi08575397-16 2021-02-10T04:29:39.238 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 161 (ObjNum 157 snap 35 seq_num 157) dirty exists 2021-02-10T04:29:39.238 INFO:tasks.rados.rados.0.smithi085.stdout:362: left oid 16 (ObjNum 157 snap 35 seq_num 157) 2021-02-10T04:29:39.238 INFO:tasks.rados.rados.0.smithi085.stdout:363: finishing write tid 1 to smithi08575397-46 2021-02-10T04:29:39.238 INFO:tasks.rados.rados.0.smithi085.stdout:363: finishing write tid 2 to smithi08575397-46 2021-02-10T04:29:39.238 INFO:tasks.rados.rados.0.smithi085.stdout:363: finishing write tid 3 to smithi08575397-46 2021-02-10T04:29:39.238 INFO:tasks.rados.rados.0.smithi085.stdout:363: finishing write tid 4 to smithi08575397-46 2021-02-10T04:29:39.238 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 115 (ObjNum 158 snap 35 seq_num 158) dirty exists 2021-02-10T04:29:39.238 INFO:tasks.rados.rados.0.smithi085.stdout:363: left oid 46 (ObjNum 158 snap 35 seq_num 158) 2021-02-10T04:29:39.238 INFO:tasks.rados.rados.0.smithi085.stdout:364: finishing write tid 1 to smithi08575397-40 2021-02-10T04:29:39.240 INFO:tasks.rados.rados.0.smithi085.stdout:364: finishing write tid 2 to smithi08575397-40 2021-02-10T04:29:39.240 INFO:tasks.rados.rados.0.smithi085.stdout:364: finishing write tid 3 to smithi08575397-40 2021-02-10T04:29:39.241 INFO:tasks.rados.rados.0.smithi085.stdout:364: finishing write tid 4 to smithi08575397-40 2021-02-10T04:29:39.241 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 165 (ObjNum 159 snap 35 seq_num 159) dirty exists 2021-02-10T04:29:39.241 INFO:tasks.rados.rados.0.smithi085.stdout:364: left oid 40 (ObjNum 159 snap 35 seq_num 159) 2021-02-10T04:29:39.327 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:29:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:39 smithi085 conmon[29319]: cluster 2021-02-10T04:29:37.933198 2021-02-10T04:29:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:39 smithi085 conmon[29319]: +0000 osd.1 (osd.1) 51 : cluster [DBG] 4.f scrub ok 2021-02-10T04:29:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:39 smithi085 conmon[29319]: cluster 2021-02-10T04:29:38.343828+0000 osd.3 (osd.3) 51 : cluster [DBG] 2021-02-10T04:29:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:39 smithi085 conmon[29319]: 5.14 scrub ok 2021-02-10T04:29:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:39 smithi085 conmon[29319]: cluster 2021-02-10T04:29:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:39 smithi085 conmon[29319]: 2021-02-10T04:29:38.663268+0000 2021-02-10T04:29:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:39 smithi085 conmon[29319]: osd.0 (osd. 2021-02-10T04:29:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:39 smithi085 conmon[29319]: 0) 53 : cluster [DBG] 4.1e scrub ok 2021-02-10T04:29:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:39 smithi085 conmon[29319]: 2021-02-10T04:29:39.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:39 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:29:39.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:39 smithi085 conmon[29319]: T04:29:38.873096+0000 osd.4 (osd.4) 2021-02-10T04:29:39.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:39 smithi085 conmon[29319]: 44 : cluster [DBG] 6.e scrub ok 2021-02-10T04:29:39.443 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:29:39 smithi085 conmon[45443]: debug 2021-02-10T04:29:39.061+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:39 smithi085 conmon[35961]: cluster 2021-02-10T04:29: 2021-02-10T04:29:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:39 smithi085 conmon[35961]: 37.933198+0000 osd.1 (osd.1) 51 : cluster [DBG] 4.f scrub ok 2021-02-10T04:29:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:39 smithi085 conmon[35961]: cluster 2021-02-10T04:29:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:39 smithi085 conmon[35961]: 2021 2021-02-10T04:29:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:39 smithi085 conmon[35961]: - 2021-02-10T04:29:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:39 smithi085 conmon[35961]: 02-10T 2021-02-10T04:29:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:39 smithi085 conmon[35961]: 04:29:38.343828+0000 osd.3 (osd.3) 51 : cluster [DBG] 5.14 scrub ok 2021-02-10T04:29:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:39 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:29:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:39 smithi085 conmon[35961]: -02-10T04:29:38.663268+0000 osd.0 ( 2021-02-10T04:29:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:39 smithi085 conmon[35961]: osd.0) 53 : cluster [DBG] 4.1e scrub ok 2021-02-10T04:29:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:39 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:29:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:39 smithi085 conmon[35961]: 10T04:29:38.873096+0000 osd.4 (osd. 2021-02-10T04:29:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:39 smithi085 conmon[35961]: 4) 44 : cluster [DBG] 6.e scrub ok 2021-02-10T04:29:39.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:39 smithi164 conmon[29288]: cluster 2021-02-10T04:29:37.933198+0000 osd.1 (osd.1) 51 : cluster [DBG] 2021-02-10T04:29:39.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:39 smithi164 conmon[29288]: 4.f scrub ok 2021-02-10T04:29:39.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:39 smithi164 conmon[29288]: cluster 2021-02-10T04:29:38.343828+0000 osd.3 (osd.3) 51 : cluster [DBG] 5.14 scrub ok 2021-02-10T04:29:39.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:39 smithi164 conmon[29288]: cluster 2021-02-10T04:29: 2021-02-10T04:29:39.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:39 smithi164 conmon[29288]: 38.663268+0000 osd.0 (osd.0) 53 : cluster [DBG] 4.1e scrub ok 2021-02-10T04:29:39.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:39 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:29:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:39 smithi164 conmon[29288]: -02-10T04:29:38.873096+0000 osd.4 2021-02-10T04:29:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:39 smithi164 conmon[29288]: (osd.4) 44 : cluster [DBG] 2021-02-10T04:29:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:39 smithi164 conmon[29288]: 6.e scrub ok 2021-02-10T04:29:39.848 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:29:39.879 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:40.192 INFO:tasks.rados.rados.0.smithi085.stdout:362: done (8 left) 2021-02-10T04:29:40.193 INFO:tasks.rados.rados.0.smithi085.stdout:363: done (7 left) 2021-02-10T04:29:40.193 INFO:tasks.rados.rados.0.smithi085.stdout:364: done (6 left) 2021-02-10T04:29:40.193 INFO:tasks.rados.rados.0.smithi085.stdout:365: done (5 left) 2021-02-10T04:29:40.193 INFO:tasks.rados.rados.0.smithi085.stdout:366: delete oid 25 current snap is 36 2021-02-10T04:29:40.199 INFO:tasks.rados.rados.0.smithi085.stdout:366: done (5 left) 2021-02-10T04:29:40.199 INFO:tasks.rados.rados.0.smithi085.stdout:367: snap_remove snap 33 2021-02-10T04:29:40.293 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 1 1 2021-02-10T04:29:40.352 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:29:40.356 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[29319]: cluster 2021-02-10T04:29:38.793756+0000 osd.6 (osd.6) 41 : cluster [DBG] 1.6 scrub ok 2021-02-10T04:29:40.356 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[29319]: cluster 2021-02-10T04:29:39.188029+0000 mon.a (mon.0) 847 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2021-02-10T04:29:40.356 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[29319]: cluster 2021-02-10T04:29:39.434203+0000 osd.5 (osd.5) 46 : cluster [DBG] 1.1 scrub ok 2021-02-10T04:29:40.356 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[29319]: cluster 2021-02-10T04:29:39.700111+0000 osd.0 (osd.0) 54 : cluster [DBG] 7.9 scrub ok 2021-02-10T04:29:40.356 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[29319]: cluster 2021-02-10T04:29:40.006880+0000 mgr.y (mgr.14140) 373 : cluster [DBG] pgmap v394: 168 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 162 active+clean; 132 MiB data, 393 MiB used, 715 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.3 MiB/s wr, 10 op/s 2021-02-10T04:29:40.356 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[35961]: cluster 2021-02-10T04:29:40.356 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[35961]: 2021-02-10T04:29:40.356 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[35961]: 2021 2021-02-10T04:29:40.357 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[35961]: - 2021-02-10T04:29:40.357 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[35961]: 02 2021-02-10T04:29:40.357 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[35961]: - 2021-02-10T04:29:40.357 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[35961]: 10 2021-02-10T04:29:40.357 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[35961]: T 2021-02-10T04:29:40.357 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[35961]: 04 2021-02-10T04:29:40.357 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[35961]: : 2021-02-10T04:29:40.357 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[35961]: 29 2021-02-10T04:29:40.357 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[35961]: : 2021-02-10T04:29:40.357 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[35961]: 38 2021-02-10T04:29:40.357 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[35961]: . 2021-02-10T04:29:40.357 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[35961]: 793756 2021-02-10T04:29:40.357 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[35961]: +0000 osd.6 (osd.6) 41 : 2021-02-10T04:29:40.358 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[35961]: cluster [DBG] 1.6 scrub ok 2021-02-10T04:29:40.358 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[35961]: cluster 2021-02-10T04:29 2021-02-10T04:29:40.358 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[35961]: :39.188029+0000 mon.a (mon.0) 847 : 2021-02-10T04:29:40.358 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[35961]: cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2021-02-10T04:29:40.358 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:29:40.358 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[35961]: 10T04:29:39.434203+0000 osd.5 (osd 2021-02-10T04:29:40.358 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[35961]: .5) 46 : cluster [DBG] 1.1 scrub ok 2021-02-10T04:29:40.358 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:29:40.358 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[35961]: 04:29:39.700111+0000 osd.0 (osd.0) 2021-02-10T04:29:40.358 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[35961]: 54 : cluster [DBG] 7.9 scrub ok 2021-02-10T04:29:40.358 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[35961]: cluster 2021-02-10T04:29:40.358 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[35961]: 2021-02-10T04:29:40.006880+0000 mgr.y (mgr 2021-02-10T04:29:40.358 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[35961]: .14140) 373 : cluster [DBG] pgmap v394: 168 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 162 active+clean; 132 MiB data, 393 MiB used, 715 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.3 MiB/s wr, 10 op/s 2021-02-10T04:29:40.359 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:29:40 smithi085 conmon[41540]: debug 2021-02-10T04:29:40.292+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:40.460 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:29:40.495 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:40.578 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:29:40.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:40 smithi164 conmon[29288]: cluster 2021-02-10T04:29:38.793756+0000 osd.6 (osd. 2021-02-10T04:29:40.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:40 smithi164 conmon[29288]: 6) 41 : cluster [DBG] 1.6 scrub ok 2021-02-10T04:29:40.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:40 smithi164 conmon[29288]: cluster 2021-02-10T04:29:39.188029+0000 mon.a (mon.0) 847 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2021-02-10T04:29:40.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:40 smithi164 conmon[29288]: cluster 2021-02-10T04:29:39.434203+0000 osd.5 (osd.5) 46 : 2021-02-10T04:29:40.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:40 smithi164 conmon[29288]: cluster [DBG] 1.1 scrub ok 2021-02-10T04:29:40.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:40 smithi164 conmon[29288]: cluster 2021-02-10T04:29:39.700111 2021-02-10T04:29:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:40 smithi164 conmon[29288]: +0000 osd.0 (osd.0) 54 : cluster [DBG] 7.9 scrub ok 2021-02-10T04:29:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:40 smithi164 conmon[29288]: cluster 2021-02-10T04:29:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:40 smithi164 conmon[29288]: 2021-02- 2021-02-10T04:29:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:40 smithi164 conmon[29288]: 10T04:29:40.006880+0000 mgr.y (mgr.14140) 373 : cluster [DBG] pgmap v394: 168 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 162 active+clean; 132 MiB data, 393 MiB used, 715 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.3 MiB/s wr, 10 op/s 2021-02-10T04:29:41.206 INFO:tasks.rados.rados.0.smithi085.stdout:367: done (5 left) 2021-02-10T04:29:41.206 INFO:tasks.rados.rados.0.smithi085.stdout:368: copy_from oid 44 from oid 37 current snap is 36 2021-02-10T04:29:41.206 INFO:tasks.rados.rados.0.smithi085.stdout:369: read oid 16 snap -1 2021-02-10T04:29:41.206 INFO:tasks.rados.rados.0.smithi085.stdout:369: expect (ObjNum 157 snap 35 seq_num 157) 2021-02-10T04:29:41.210 INFO:tasks.rados.rados.0.smithi085.stdout:370: write (excl) oid 39 current snap is 36 2021-02-10T04:29:41.210 INFO:tasks.rados.rados.0.smithi085.stdout:370: seq_num 161 ranges {705927=558502,2004895=422707} 2021-02-10T04:29:41.213 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is set 2021-02-10T04:29:41.213 INFO:tasks.rados.rados.0.smithi085.stdout:370: writing smithi08575397-39 from 705927 to 1264429 tid 1 2021-02-10T04:29:41.221 INFO:tasks.rados.rados.0.smithi085.stdout:370: writing smithi08575397-39 from 2004895 to 2427602 tid 2 2021-02-10T04:29:41.221 INFO:tasks.rados.rados.0.smithi085.stdout:371: read oid 21 snap -1 2021-02-10T04:29:41.221 INFO:tasks.rados.rados.0.smithi085.stdout:371: expect deleted 2021-02-10T04:29:41.221 INFO:tasks.rados.rados.0.smithi085.stdout:372: write oid 6 current snap is 36 2021-02-10T04:29:41.222 INFO:tasks.rados.rados.0.smithi085.stdout:372: seq_num 162 ranges {722734=633751,2063377=659357} 2021-02-10T04:29:41.230 INFO:tasks.rados.rados.0.smithi085.stdout:372: writing smithi08575397-6 from 722734 to 1356485 tid 1 2021-02-10T04:29:41.238 INFO:tasks.rados.rados.0.smithi085.stdout:372: writing smithi08575397-6 from 2063377 to 2722734 tid 2 2021-02-10T04:29:41.238 INFO:tasks.rados.rados.0.smithi085.stdout:373: copy_from oid 3 from oid 5 current snap is 36 2021-02-10T04:29:41.238 INFO:tasks.rados.rados.0.smithi085.stdout:370: finishing write tid 1 to smithi08575397-39 2021-02-10T04:29:41.238 INFO:tasks.rados.rados.0.smithi085.stdout:369: expect (ObjNum 157 snap 35 seq_num 157) 2021-02-10T04:29:41.435 INFO:tasks.rados.rados.0.smithi085.stdout:370: finishing write tid 2 to smithi08575397-39 2021-02-10T04:29:41.435 INFO:tasks.rados.rados.0.smithi085.stdout:370: finishing write tid 3 to smithi08575397-39 2021-02-10T04:29:41.435 INFO:tasks.rados.rados.0.smithi085.stdout:370: finishing write tid 4 to smithi08575397-39 2021-02-10T04:29:41.435 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 39 v 44 (ObjNum 161 snap 36 seq_num 161) dirty exists 2021-02-10T04:29:41.435 INFO:tasks.rados.rados.0.smithi085.stdout:370: left oid 39 (ObjNum 161 snap 36 seq_num 161) 2021-02-10T04:29:41.435 INFO:tasks.rados.rados.0.smithi085.stdout:372: finishing write tid 1 to smithi08575397-6 2021-02-10T04:29:41.435 INFO:tasks.rados.rados.0.smithi085.stdout:372: finishing write tid 2 to smithi08575397-6 2021-02-10T04:29:41.435 INFO:tasks.rados.rados.0.smithi085.stdout:372: finishing write tid 3 to smithi08575397-6 2021-02-10T04:29:41.435 INFO:tasks.rados.rados.0.smithi085.stdout:372: finishing write tid 4 to smithi08575397-6 2021-02-10T04:29:41.436 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 120 (ObjNum 162 snap 36 seq_num 162) dirty exists 2021-02-10T04:29:41.436 INFO:tasks.rados.rados.0.smithi085.stdout:372: left oid 6 (ObjNum 162 snap 36 seq_num 162) 2021-02-10T04:29:41.436 INFO:tasks.rados.rados.0.smithi085.stdout:369: done (10 left) 2021-02-10T04:29:41.436 INFO:tasks.rados.rados.0.smithi085.stdout:370: done (9 left) 2021-02-10T04:29:41.436 INFO:tasks.rados.rados.0.smithi085.stdout:371: done (8 left) 2021-02-10T04:29:41.436 INFO:tasks.rados.rados.0.smithi085.stdout:372: done (7 left) 2021-02-10T04:29:41.436 INFO:tasks.rados.rados.0.smithi085.stdout:374: delete oid 50 current snap is 36 2021-02-10T04:29:41.436 INFO:tasks.rados.rados.0.smithi085.stdout:374: done (7 left) 2021-02-10T04:29:41.436 INFO:tasks.rados.rados.0.smithi085.stdout:375: read oid 11 snap -1 2021-02-10T04:29:41.436 INFO:tasks.rados.rados.0.smithi085.stdout:375: expect deleted 2021-02-10T04:29:41.436 INFO:tasks.rados.rados.0.smithi085.stdout:376: delete oid 27 current snap is 36 2021-02-10T04:29:41.436 INFO:tasks.rados.rados.0.smithi085.stdout:375: done (8 left) 2021-02-10T04:29:41.437 INFO:tasks.rados.rados.0.smithi085.stdout:376: done (7 left) 2021-02-10T04:29:41.437 INFO:tasks.rados.rados.0.smithi085.stdout:377: rollback oid 13 current snap is 36 2021-02-10T04:29:41.437 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 13 to 30 2021-02-10T04:29:41.437 INFO:tasks.rados.rados.0.smithi085.stdout:378: rollback oid 14 current snap is 36 2021-02-10T04:29:41.437 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 14 to 35 2021-02-10T04:29:41.437 INFO:tasks.rados.rados.0.smithi085.stdout:379: write oid 34 current snap is 36 2021-02-10T04:29:41.437 INFO:tasks.rados.rados.0.smithi085.stdout:379: seq_num 165 ranges {773155=459498,1722470=430045,2648676=124479} 2021-02-10T04:29:41.438 INFO:tasks.rados.rados.0.smithi085.stdout:379: writing smithi08575397-34 from 773155 to 1232653 tid 1 2021-02-10T04:29:41.444 INFO:tasks.rados.rados.0.smithi085.stdout:379: writing smithi08575397-34 from 1722470 to 2152515 tid 2 2021-02-10T04:29:41.449 INFO:tasks.rados.rados.0.smithi085.stdout:379: writing smithi08575397-34 from 2648676 to 2773155 tid 3 2021-02-10T04:29:41.449 INFO:tasks.rados.rados.0.smithi085.stdout:380: read oid 24 snap 35 2021-02-10T04:29:41.449 INFO:tasks.rados.rados.0.smithi085.stdout:380: expect deleted 2021-02-10T04:29:41.449 INFO:tasks.rados.rados.0.smithi085.stdout:381: copy_from oid 8 from oid 43 current snap is 36 2021-02-10T04:29:41.449 INFO:tasks.rados.rados.0.smithi085.stdout:378: finishing rollback tid 0 to smithi08575397-14 2021-02-10T04:29:41.449 INFO:tasks.rados.rados.0.smithi085.stdout:377: finishing rollback tid 1 to smithi08575397-13 2021-02-10T04:29:41.449 INFO:tasks.rados.rados.0.smithi085.stdout:377: finishing rollback tid 2 to smithi08575397-13 2021-02-10T04:29:41.449 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 73 (ObjNum 141 snap 27 seq_num 141) dirty exists 2021-02-10T04:29:41.449 INFO:tasks.rados.rados.0.smithi085.stdout:378: finishing rollback tid 1 to smithi08575397-14 2021-02-10T04:29:41.449 INFO:tasks.rados.rados.0.smithi085.stdout:378: finishing rollback tid 2 to smithi08575397-14 2021-02-10T04:29:41.449 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 102 (ObjNum 142 snap 29 seq_num 142) dirty exists 2021-02-10T04:29:41.450 INFO:tasks.rados.rados.0.smithi085.stdout:373: finishing copy_from to smithi08575397-3 2021-02-10T04:29:41.450 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 67 (ObjNum 148 snap 30 seq_num 148) dirty exists 2021-02-10T04:29:41.450 INFO:tasks.rados.rados.0.smithi085.stdout:373: finishing copy_from racing read to smithi08575397-3 2021-02-10T04:29:41.450 INFO:tasks.rados.rados.0.smithi085.stdout:379: finishing write tid 1 to smithi08575397-34 2021-02-10T04:29:41.450 INFO:tasks.rados.rados.0.smithi085.stdout:379: finishing write tid 2 to smithi08575397-34 2021-02-10T04:29:41.450 INFO:tasks.rados.rados.0.smithi085.stdout:373: done (11 left) 2021-02-10T04:29:41.450 INFO:tasks.rados.rados.0.smithi085.stdout:377: done (10 left) 2021-02-10T04:29:41.450 INFO:tasks.rados.rados.0.smithi085.stdout:378: done (9 left) 2021-02-10T04:29:41.450 INFO:tasks.rados.rados.0.smithi085.stdout:382: snap_remove snap 23 2021-02-10T04:29:41.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:41 smithi164 conmon[29288]: cluster 2021-02-10T04:29:39.828971 2021-02-10T04:29:41.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:41 smithi164 conmon[29288]: +0000 osd.6 (osd.6) 42 : cluster [DBG] 1.7 scrub ok 2021-02-10T04:29:41.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:41 smithi164 conmon[29288]: cluster 2021-02-10T04:29:40.192650+0000 2021-02-10T04:29:41.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:41 smithi164 conmon[29288]: mon.a (mon.0) 848 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2021-02-10T04:29:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:41 smithi164 conmon[29288]: cluster 2021-02-10T04:29 2021-02-10T04:29:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:41 smithi164 conmon[29288]: :40.451825+0000 2021-02-10T04:29:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:41 smithi164 conmon[29288]: osd.5 (osd.5) 47 : cluster [DBG] 5.7 scrub ok 2021-02-10T04:29:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:41 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:29:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:41 smithi164 conmon[29288]: T04:29:40.686330+0000 osd.4 (osd.4) 45 2021-02-10T04:29:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:41 smithi164 conmon[29288]: : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:29:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:41 smithi164 conmon[29288]: cluster 2021-02-10T04:29: 2021-02-10T04:29:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:41 smithi164 conmon[29288]: 40.686496+0000 osd.4 (osd.4) 46 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:29:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:41 smithi164 conmon[29288]: 2021-02-10T04:29:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:41 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:29:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:41 smithi164 conmon[29288]: 02-10T04:29:40 2021-02-10T04:29:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:41 smithi164 conmon[29288]: .859269+0000 osd.4 (osd. 2021-02-10T04:29:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:41 smithi164 conmon[29288]: 4) 47 : cluster [DBG] 5.1 scrub ok 2021-02-10T04:29:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:41 smithi164 conmon[29288]: audit 2021 2021-02-10T04:29:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:41 smithi164 conmon[29288]: -02-10T04: 2021-02-10T04:29:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:41 smithi164 conmon[29288]: 29:41.112429+0000 mon.a 2021-02-10T04:29:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:41 smithi164 conmon[29288]: (mon.0) 849 : audit [INF] from='client.? 172.21.15.85:0/3459789144' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:29:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:41 smithi164 conmon[29288]: 2021-02-10T04:29:41.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[29319]: cluster 2021-02-10T04:29:39.828971+0000 osd.6 ( 2021-02-10T04:29:41.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[29319]: osd.6) 42 : cluster [DBG] 1.7 scrub ok 2021-02-10T04:29:41.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[29319]: cluster 2021-02-10T04:29:40.192650+0000 mon.a (mon.0) 848 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2021-02-10T04:29:41.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[29319]: cluster 2021-02-10T04:29:40.451825+0000 osd.5 (osd.5) 47 : cluster [DBG] 5.7 scrub ok 2021-02-10T04:29:41.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[29319]: cluster 2021-02-10T04:29:40.686330+0000 osd.4 (osd.4) 45 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:29:41.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[29319]: cluster 2021-02-10T04:29:40.686496+0000 osd.4 (osd.4) 46 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:29:41.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[29319]: cluster 2021-02-10T04:29:40.859269+0000 osd.4 (osd.4) 47 : cluster [DBG] 5.1 scrub ok 2021-02-10T04:29:41.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[29319]: audit 2021-02-10T04:29:41.112429+0000 mon.a (mon.0) 849 : audit [INF] from='client.? 172.21.15.85:0/3459789144' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:29:41.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[35961]: cluster 2021-02-10T04:29:39. 2021-02-10T04:29:41.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[35961]: 828971+0000 osd.6 (osd.6) 42 : cluster [DBG] 1.7 scrub ok 2021-02-10T04:29:41.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[35961]: cluster 2021-02-10T04:29:40 2021-02-10T04:29:41.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[35961]: .192650+0000 mon.a (mon.0) 848 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2021-02-10T04:29:41.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[35961]: cluster 2021-02-10T04:29:40.451825 2021-02-10T04:29:41.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[35961]: +0000 osd.5 (osd.5) 47 : cluster [DBG] 2021-02-10T04:29:41.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[35961]: 5.7 scrub ok 2021-02-10T04:29:41.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[35961]: cluster 2021-02-10T04:29:41.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:29:41.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[35961]: 29:40.686330+0000 osd.4 (osd.4) 45 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:29:41.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:29:41.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[35961]: -02-10T04:29:40 2021-02-10T04:29:41.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[35961]: .686496+0000 osd.4 (osd.4 2021-02-10T04:29:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[35961]: ) 46 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:29:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[35961]: 2021-02-10T04:29:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[35961]: cluster 2021-02-10T04:29:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:29:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[35961]: :29:40.859269+0000 2021-02-10T04:29:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[35961]: osd.4 (osd.4) 47 : cluster 2021-02-10T04:29:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[35961]: [DBG] 5.1 scrub ok 2021-02-10T04:29:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[35961]: audit 2021 2021-02-10T04:29:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[35961]: -02- 2021-02-10T04:29:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[35961]: 10T04:29 2021-02-10T04:29:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[35961]: :41.112429+0000 mon.a (mon.0) 849 : 2021-02-10T04:29:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:41 smithi085 conmon[35961]: audit [INF] from='client.? 172.21.15.85:0/3459789144' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:29:42.124 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:29:42.208 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:29:42.213 INFO:tasks.rados.rados.0.smithi085.stdout:382: done (9 left) 2021-02-10T04:29:42.213 INFO:tasks.rados.rados.0.smithi085.stdout:383: snap_create 2021-02-10T04:29:42.213 INFO:tasks.rados.rados.0.smithi085.stdout:381: finishing copy_from to smithi08575397-8 2021-02-10T04:29:42.214 INFO:tasks.rados.rados.0.smithi085.stdout:381: got expected ENOENT (src dne) 2021-02-10T04:29:42.214 INFO:tasks.rados.rados.0.smithi085.stdout:381: finishing copy_from racing read to smithi08575397-8 2021-02-10T04:29:42.214 INFO:tasks.rados.rados.0.smithi085.stdout:379: finishing write tid 3 to smithi08575397-34 2021-02-10T04:29:42.214 INFO:tasks.rados.rados.0.smithi085.stdout:379: finishing write tid 4 to smithi08575397-34 2021-02-10T04:29:42.214 INFO:tasks.rados.rados.0.smithi085.stdout:379: finishing write tid 5 to smithi08575397-34 2021-02-10T04:29:42.214 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 34 v 77 (ObjNum 165 snap 36 seq_num 165) dirty exists 2021-02-10T04:29:42.214 INFO:tasks.rados.rados.0.smithi085.stdout:379: left oid 34 (ObjNum 165 snap 36 seq_num 165) 2021-02-10T04:29:42.214 INFO:tasks.rados.rados.0.smithi085.stdout:368: finishing copy_from to smithi08575397-44 2021-02-10T04:29:42.214 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2021-02-10T04:29:42.215 INFO:tasks.rados.rados.0.smithi085.stdout:368: finishing copy_from racing read to smithi08575397-44 2021-02-10T04:29:42.314 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:29:42.341 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:42.410 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:29:42.440 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:29:42 smithi085 conmon[49106]: debug 2021-02-10T04:29:42.176+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:42.441 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:29:42.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:42 smithi085 conmon[29319]: cluster 2021-02-10T04:29:41.190317+0000 mon.a ( 2021-02-10T04:29:42.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:42 smithi085 conmon[29319]: mon.0) 850 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:29:42.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:42 smithi085 conmon[29319]: audit 2021-02-10T04:29:41.205052+0000 mon.a (mon.0) 851 2021-02-10T04:29:42.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:42 smithi085 conmon[29319]: : audit [INF] from='client.? 172.21.15.85:0/3459789144' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:29:42.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:42 smithi085 conmon[29319]: cluster 2021-02-10T04:29:42.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:42 smithi085 conmon[29319]: 2021-02-10T04:29:41.205263+0000 mon.a (mon.0) 2021-02-10T04:29:42.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:42 smithi085 conmon[29319]: 852 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2021-02-10T04:29:42.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:42 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:29:42.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:42 smithi085 conmon[29319]: -02-10T04:29:42.007415+0000 2021-02-10T04:29:42.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:42 smithi085 conmon[29319]: mgr.y (mgr.14140) 374 : cluster [DBG] pgmap v397: 168 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 164 active+clean; 136 MiB data, 414 MiB used, 715 GiB / 715 GiB avail; 5.7 MiB/s rd, 1.8 MiB/s wr, 14 op/s 2021-02-10T04:29:42.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:42 smithi085 conmon[35961]: cluster 2021-02-10T04:29:42.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:42 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:29:42.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:42 smithi085 conmon[35961]: 29:41.190317+0000 mon.a (mon.0) 850 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:29:42.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:42 smithi085 conmon[35961]: audit 2021-02-10T04:29:42.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:42 smithi085 conmon[35961]: 2021-02-10T04:29:41.205052+0000 2021-02-10T04:29:42.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:42 smithi085 conmon[35961]: mon.a (mon.0) 851 : audit [INF] from='client.? 172.21.15.85:0/3459789144' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:29:42.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:42 smithi085 conmon[35961]: 2021-02-10T04:29:42.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:42 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:29:42.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:42 smithi085 conmon[35961]: -02-10T04:29:41.205263+0000 mon.a ( 2021-02-10T04:29:42.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:42 smithi085 conmon[35961]: mon.0) 852 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2021-02-10T04:29:42.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:42 smithi085 conmon[35961]: cluster 2021-02-10T04:29:42.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:42 smithi085 conmon[35961]: 2021-02-10T04:29: 2021-02-10T04:29:42.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:42 smithi085 conmon[35961]: 42.007415+0000 mgr.y (mgr.14140) 374 : cluster [DBG] pgmap v397: 168 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 164 active+clean; 136 MiB data, 414 MiB used, 715 GiB / 715 GiB avail; 5.7 MiB/s rd, 1.8 MiB/s wr, 14 op/s 2021-02-10T04:29:42.444 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:29:42 smithi085 conmon[41540]: debug 2021-02-10T04:29:42.221+0000 7fb94835c700 -1 osd.0 pg_epoch: 166 pg[7.4( v 147'27 (0'0,147'27] local-lis/les=93/94 n=3 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [0,4,1] r=0 lpr=93 crt=147'27 lcod 147'25 mlcod 147'25 active+clean+snaptrim trimq=[2e~1] ps=[30~1,32~2,36~2,3b~1,3f~2]] removing snap head 2021-02-10T04:29:42.445 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:42 smithi164 conmon[29288]: cluster 2021-02-10T04:29:41.190317+0000 mon.a (mon.0) 850 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:29:42.445 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:42 smithi164 conmon[29288]: audit 2021-02-10T04:29:41.205052+0000 mon.a (mon.0) 851 : audit [INF] from='client.? 172.21.15.85:0/3459789144' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:29:42.445 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:42 smithi164 conmon[29288]: cluster 2021-02-10T04:29:42.445 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:42 smithi164 conmon[29288]: 2021-02-10T04:29:41.205263+0000 mon.a (mon.0) 852 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2021-02-10T04:29:42.445 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:42 smithi164 conmon[29288]: cluster 2021-02-10T04:29 2021-02-10T04:29:42.445 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:42 smithi164 conmon[29288]: :42.007415+0000 mgr.y (mgr.14140) 374 2021-02-10T04:29:42.445 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:42 smithi164 conmon[29288]: : cluster [DBG] pgmap v397: 168 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 164 active+clean; 136 MiB data, 414 MiB used, 715 GiB / 715 GiB avail; 5.7 MiB/s rd, 1.8 MiB/s wr, 14 op/s 2021-02-10T04:29:42.446 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:29:42 smithi164 conmon[44544]: debug 2021-02-10T04:29:42.421+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:42.548 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:29:42.577 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:42.683 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:29:42.788 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:29:42.818 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:42.893 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:29:42 smithi164 conmon[40892]: debug 2021-02-10T04:29:42.644+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:43.050 INFO:tasks.rados.rados.0.smithi085.stdout:368: done (9 left) 2021-02-10T04:29:43.050 INFO:tasks.rados.rados.0.smithi085.stdout:379: done (8 left) 2021-02-10T04:29:43.050 INFO:tasks.rados.rados.0.smithi085.stdout:380: done (7 left) 2021-02-10T04:29:43.051 INFO:tasks.rados.rados.0.smithi085.stdout:381: done (6 left) 2021-02-10T04:29:43.051 INFO:tasks.rados.rados.0.smithi085.stdout:383: done (5 left) 2021-02-10T04:29:43.051 INFO:tasks.rados.rados.0.smithi085.stdout:384: write oid 49 current snap is 37 2021-02-10T04:29:43.051 INFO:tasks.rados.rados.0.smithi085.stdout:384: seq_num 166 ranges {789962=534747,1780952=690056} 2021-02-10T04:29:43.058 INFO:tasks.rados.rados.0.smithi085.stdout:384: writing smithi08575397-49 from 789962 to 1324709 tid 1 2021-02-10T04:29:43.076 INFO:tasks.rados.rados.0.smithi085.stdout:384: writing smithi08575397-49 from 1780952 to 2471008 tid 2 2021-02-10T04:29:43.076 INFO:tasks.rados.rados.0.smithi085.stdout:385: delete oid 43 current snap is 37 2021-02-10T04:29:43.076 INFO:tasks.rados.rados.0.smithi085.stdout:384: finishing write tid 1 to smithi08575397-49 2021-02-10T04:29:43.076 INFO:tasks.rados.rados.0.smithi085.stdout:385: done (6 left) 2021-02-10T04:29:43.076 INFO:tasks.rados.rados.0.smithi085.stdout:386: snap_remove snap 35 2021-02-10T04:29:43.370 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:29:43.485 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:29:43 smithi085 conmon[49106]: debug 2021-02-10T04:29:43.326+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:43.485 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:43 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:29:43.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:43 smithi085 conmon[35961]: 29:42.211399+0000 mon.a (mon.0) 853 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2021-02-10T04:29:43.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:43 smithi085 conmon[35961]: cluster 2021-02-10T04:29:43.046354+0000 mon.a 2021-02-10T04:29:43.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:43 smithi085 conmon[35961]: (mon.0) 854 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2021-02-10T04:29:43.486 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:43 smithi085 conmon[29319]: cluster 2021-02-10T04:29:43.486 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:43 smithi085 conmon[29319]: 2021-02-10T04:29:43.486 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:43 smithi085 conmon[29319]: 2021-02-10T04:29 2021-02-10T04:29:43.486 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:43 smithi085 conmon[29319]: :42.211399+0000 mon.a (mon.0) 853 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2021-02-10T04:29:43.486 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:43 smithi085 conmon[29319]: cluster 2021-02-10T04:29:43.486 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:43 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:29:43.487 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:43 smithi085 conmon[29319]: 29:43.046354+0000 mon.a (mon.0) 854 2021-02-10T04:29:43.487 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:43 smithi085 conmon[29319]: : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2021-02-10T04:29:43.488 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:29:43.512 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:43 smithi164 conmon[29288]: cluster 2021-02-10T04:29:42.211399+0000 mon.a ( 2021-02-10T04:29:43.513 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:43 smithi164 conmon[29288]: mon.0) 853 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2021-02-10T04:29:43.513 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:43 smithi164 conmon[29288]: cluster 2021-02-10T04:29:43.046354+0000 mon.a (mon.0) 854 : cluster [DBG] 2021-02-10T04:29:43.513 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:43 smithi164 conmon[29288]: osdmap e167: 8 total, 8 up, 8 in 2021-02-10T04:29:43.513 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:29:43 smithi164 conmon[40892]: debug 2021-02-10T04:29:43.215+0000 7f3036cc8700 -1 osd.6 pg_epoch: 167 pg[7.c( v 165'92 (0'0,165'92] local-lis/les=93/94 n=8 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [6,3,7] r=0 lpr=93 crt=165'92 lcod 163'90 mlcod 163'90 active+clean+snaptrim trimq=[2e~1] ps=[3b~1,3f~2]] removing snap head 2021-02-10T04:29:43.515 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:43.608 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:29:43.713 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:29:43.740 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:43.808 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:29:43 smithi164 conmon[44544]: debug 2021-02-10T04:29:43.582+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:43.825 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:29:43.930 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:29:43.964 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:44.061 INFO:tasks.rados.rados.0.smithi085.stdout:386: done (6 left) 2021-02-10T04:29:44.061 INFO:tasks.rados.rados.0.smithi085.stdout:387: snap_remove snap 36 2021-02-10T04:29:44.067 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:29:44.067 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:29:44.144 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:29:43 smithi164 conmon[37263]: debug 2021-02-10T04:29:43.808+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:45.069 INFO:tasks.rados.rados.0.smithi085.stdout:387: done (6 left) 2021-02-10T04:29:45.069 INFO:tasks.rados.rados.0.smithi085.stdout:388: write oid 24 current snap is 37 2021-02-10T04:29:45.069 INFO:tasks.rados.rados.0.smithi085.stdout:388: seq_num 168 ranges {423576=401598,1497916=693725} 2021-02-10T04:29:45.069 INFO:tasks.rados.rados.0.smithi085.stdout:388: writing smithi08575397-24 from 423576 to 825174 tid 1 2021-02-10T04:29:45.076 INFO:tasks.rados.rados.0.smithi085.stdout:388: writing smithi08575397-24 from 1497916 to 2191641 tid 2 2021-02-10T04:29:45.077 INFO:tasks.rados.rados.0.smithi085.stdout:389: rollback oid 21 current snap is 37 2021-02-10T04:29:45.077 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 21 to 34 2021-02-10T04:29:45.077 INFO:tasks.rados.rados.0.smithi085.stdout:384: finishing write tid 2 to smithi08575397-49 2021-02-10T04:29:45.077 INFO:tasks.rados.rados.0.smithi085.stdout:384: finishing write tid 3 to smithi08575397-49 2021-02-10T04:29:45.077 INFO:tasks.rados.rados.0.smithi085.stdout:384: finishing write tid 4 to smithi08575397-49 2021-02-10T04:29:45.077 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 49 v 55 (ObjNum 166 snap 37 seq_num 166) dirty exists 2021-02-10T04:29:45.077 INFO:tasks.rados.rados.0.smithi085.stdout:384: left oid 49 (ObjNum 166 snap 37 seq_num 166) 2021-02-10T04:29:45.077 INFO:tasks.rados.rados.0.smithi085.stdout:384: done (7 left) 2021-02-10T04:29:45.077 INFO:tasks.rados.rados.0.smithi085.stdout:390: rollback oid 18 current snap is 37 2021-02-10T04:29:45.077 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 18 to 34 2021-02-10T04:29:45.077 INFO:tasks.rados.rados.0.smithi085.stdout:391: read oid 6 snap 34 2021-02-10T04:29:45.077 INFO:tasks.rados.rados.0.smithi085.stdout:391: expect deleted 2021-02-10T04:29:45.078 INFO:tasks.rados.rados.0.smithi085.stdout:392: copy_from oid 13 from oid 39 current snap is 37 2021-02-10T04:29:45.078 INFO:tasks.rados.rados.0.smithi085.stdout:393: rollback oid 36 current snap is 37 2021-02-10T04:29:45.078 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 36 to 30 2021-02-10T04:29:45.078 INFO:tasks.rados.rados.0.smithi085.stdout:394: delete oid 16 current snap is 37 2021-02-10T04:29:45.080 INFO:tasks.rados.rados.0.smithi085.stdout:388: finishing write tid 1 to smithi08575397-24 2021-02-10T04:29:45.082 INFO:tasks.rados.rados.0.smithi085.stdout:393: finishing rollback tid 0 to smithi08575397-36 2021-02-10T04:29:45.082 INFO:tasks.rados.rados.0.smithi085.stdout:393: finishing rollback tid 1 to smithi08575397-36 2021-02-10T04:29:45.082 INFO:tasks.rados.rados.0.smithi085.stdout:389: finishing rollback tid 1 to smithi08575397-21 2021-02-10T04:29:45.082 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 116 (ObjNum 82402976 snap 22030 seq_num 1215200480) dirty dne 2021-02-10T04:29:45.082 INFO:tasks.rados.rados.0.smithi085.stdout:393: finishing rollback tid 2 to smithi08575397-36 2021-02-10T04:29:45.082 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 45 (ObjNum 125 snap 23 seq_num 125) dirty exists 2021-02-10T04:29:45.084 INFO:tasks.rados.rados.0.smithi085.stdout:388: finishing write tid 2 to smithi08575397-24 2021-02-10T04:29:45.084 INFO:tasks.rados.rados.0.smithi085.stdout:389: done (11 left) 2021-02-10T04:29:45.084 INFO:tasks.rados.rados.0.smithi085.stdout:391: done (10 left) 2021-02-10T04:29:45.084 INFO:tasks.rados.rados.0.smithi085.stdout:393: done (9 left) 2021-02-10T04:29:45.084 INFO:tasks.rados.rados.0.smithi085.stdout:394: done (8 left) 2021-02-10T04:29:45.084 INFO:tasks.rados.rados.0.smithi085.stdout:395: read oid 25 snap 34 2021-02-10T04:29:45.084 INFO:tasks.rados.rados.0.smithi085.stdout:395: expect (ObjNum 87 snap 11 seq_num 87) 2021-02-10T04:29:45.084 INFO:tasks.rados.rados.0.smithi085.stdout:390: finishing rollback tid 0 to smithi08575397-18 2021-02-10T04:29:45.084 INFO:tasks.rados.rados.0.smithi085.stdout:388: finishing write tid 3 to smithi08575397-24 2021-02-10T04:29:45.084 INFO:tasks.rados.rados.0.smithi085.stdout:396: read oid 43 snap -1 2021-02-10T04:29:45.084 INFO:tasks.rados.rados.0.smithi085.stdout:396: expect deleted 2021-02-10T04:29:45.085 INFO:tasks.rados.rados.0.smithi085.stdout:397: read oid 10 snap 31 2021-02-10T04:29:45.085 INFO:tasks.rados.rados.0.smithi085.stdout:397: expect (ObjNum 71 snap 7 seq_num 71) 2021-02-10T04:29:45.085 INFO:tasks.rados.rados.0.smithi085.stdout:398: rollback oid 8 current snap is 37 2021-02-10T04:29:45.085 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 8 to 31 2021-02-10T04:29:45.085 INFO:tasks.rados.rados.0.smithi085.stdout:399: rollback oid 14 current snap is 37 2021-02-10T04:29:45.085 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 14 to 34 2021-02-10T04:29:45.085 INFO:tasks.rados.rados.0.smithi085.stdout:388: finishing write tid 4 to smithi08575397-24 2021-02-10T04:29:45.085 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 24 v 32 (ObjNum 168 snap 37 seq_num 168) dirty exists 2021-02-10T04:29:45.085 INFO:tasks.rados.rados.0.smithi085.stdout:388: left oid 24 (ObjNum 168 snap 37 seq_num 168) 2021-02-10T04:29:45.085 INFO:tasks.rados.rados.0.smithi085.stdout:390: finishing rollback tid 1 to smithi08575397-18 2021-02-10T04:29:45.085 INFO:tasks.rados.rados.0.smithi085.stdout:388: done (12 left) 2021-02-10T04:29:45.085 INFO:tasks.rados.rados.0.smithi085.stdout:400: read oid 1 snap 34 2021-02-10T04:29:45.085 INFO:tasks.rados.rados.0.smithi085.stdout:400: expect (ObjNum 8 snap 0 seq_num 8) 2021-02-10T04:29:45.086 INFO:tasks.rados.rados.0.smithi085.stdout:401: read oid 3 snap -1 2021-02-10T04:29:45.086 INFO:tasks.rados.rados.0.smithi085.stdout:401: expect (ObjNum 148 snap 30 seq_num 148) 2021-02-10T04:29:45.086 INFO:tasks.rados.rados.0.smithi085.stdout:402: delete oid 24 current snap is 37 2021-02-10T04:29:45.086 INFO:tasks.rados.rados.0.smithi085.stdout:398: finishing rollback tid 1 to smithi08575397-8 2021-02-10T04:29:45.086 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 0 (ObjNum 82249792 snap 22030 seq_num 82302512) dirty dne 2021-02-10T04:29:45.087 INFO:tasks.rados.rados.0.smithi085.stdout:399: finishing rollback tid 0 to smithi08575397-14 2021-02-10T04:29:45.087 INFO:tasks.rados.rados.0.smithi085.stdout:399: finishing rollback tid 1 to smithi08575397-14 2021-02-10T04:29:45.087 INFO:tasks.rados.rados.0.smithi085.stdout:390: finishing rollback tid 2 to smithi08575397-18 2021-02-10T04:29:45.087 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 18 v 67 (ObjNum 99 snap 20 seq_num 99) dirty exists 2021-02-10T04:29:45.087 INFO:tasks.rados.rados.0.smithi085.stdout:399: finishing rollback tid 2 to smithi08575397-14 2021-02-10T04:29:45.087 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 111 (ObjNum 142 snap 29 seq_num 142) dirty exists 2021-02-10T04:29:45.091 INFO:tasks.rados.rados.0.smithi085.stdout:395: expect (ObjNum 87 snap 11 seq_num 87) 2021-02-10T04:29:45.358 INFO:tasks.rados.rados.0.smithi085.stdout:400: expect (ObjNum 8 snap 0 seq_num 8) 2021-02-10T04:29:45.393 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:29:45 smithi164 conmon[40892]: debug 2021-02-10T04:29:45.393 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:29:45 smithi164 conmon[40892]: 2021-02-10T04:29:45.218+0000 7f30314bd700 -1 osd.6 pg_epoch: 169 pg[7.7( v 169'171 (0'0,169'171] local-lis/les=93/94 n=13 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [6,0,3] r=0 lpr=93 crt=169'171 lcod 169'170 mlcod 169'170 active+clean+snaptrim trimq=[2e~1,43~1,46~1] ps=[3b~1,3f~2]] removing snap head 2021-02-10T04:29:45.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:45 smithi164 conmon[29288]: cluster 2021-02-10T04:29:44.007982+0000 mgr.y (mgr.14140) 2021-02-10T04:29:45.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:45 smithi164 conmon[29288]: 375 : cluster [DBG] pgmap v400: 168 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 163 active+clean; 134 MiB data, 409 MiB used, 715 GiB / 715 GiB avail; 5.7 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2021-02-10T04:29:45.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:45 smithi164 conmon[29288]: cluster 2021-02-10T04:29:44.060334+0000 mon.a (mon.0) 855 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2021-02-10T04:29:45.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:45 smithi085 conmon[29319]: cluster 2021-02-10T04:29:45.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:45 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:29:45.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:45 smithi085 conmon[29319]: 04:29:44.007982+0000 mgr.y (mgr.14140) 375 : cluster [DBG] pgmap v400: 168 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 163 active+clean; 134 MiB data, 409 MiB used, 715 GiB / 715 GiB avail; 5.7 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2021-02-10T04:29:45.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:45 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:29:45.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:45 smithi085 conmon[29319]: -10T04:29:44.060334+0000 mon.a (mon.0 2021-02-10T04:29:45.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:45 smithi085 conmon[29319]: ) 855 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2021-02-10T04:29:45.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:45 smithi085 conmon[35961]: cluster 2021-02-10T04:29:45.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:45 smithi085 conmon[35961]: 2021-02-10T04:29:45.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:45 smithi085 conmon[35961]: 2021-02-10T04:29:44.007982 2021-02-10T04:29:45.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:45 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 375 : cluster [DBG] pgmap v400: 168 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 163 active+clean; 134 MiB data, 409 MiB used, 715 GiB / 715 GiB avail; 5.7 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2021-02-10T04:29:45.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:45 smithi085 conmon[35961]: cluster 2021-02-10T04:29:45.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:45 smithi085 conmon[35961]: 2021-02-10T04:29:44.060334+0000 2021-02-10T04:29:45.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:45 smithi085 conmon[35961]: mon.a (mon.0) 855 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2021-02-10T04:29:45.449 INFO:tasks.rados.rados.0.smithi085.stdout:397: expect (ObjNum 71 snap 7 seq_num 71) 2021-02-10T04:29:45.552 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set nodeep-scrub 2021-02-10T04:29:45.636 INFO:tasks.rados.rados.0.smithi085.stdout:401: expect (ObjNum 148 snap 30 seq_num 148) 2021-02-10T04:29:45.741 INFO:tasks.rados.rados.0.smithi085.stdout:392: finishing copy_from to smithi08575397-13 2021-02-10T04:29:45.741 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 44 (ObjNum 161 snap 36 seq_num 161) dirty exists 2021-02-10T04:29:45.741 INFO:tasks.rados.rados.0.smithi085.stdout:392: finishing copy_from racing read to smithi08575397-13 2021-02-10T04:29:45.742 INFO:tasks.rados.rados.0.smithi085.stdout:390: done (14 left) 2021-02-10T04:29:45.742 INFO:tasks.rados.rados.0.smithi085.stdout:392: done (13 left) 2021-02-10T04:29:45.742 INFO:tasks.rados.rados.0.smithi085.stdout:395: done (12 left) 2021-02-10T04:29:45.742 INFO:tasks.rados.rados.0.smithi085.stdout:396: done (11 left) 2021-02-10T04:29:45.742 INFO:tasks.rados.rados.0.smithi085.stdout:397: done (10 left) 2021-02-10T04:29:45.742 INFO:tasks.rados.rados.0.smithi085.stdout:398: done (9 left) 2021-02-10T04:29:45.742 INFO:tasks.rados.rados.0.smithi085.stdout:399: done (8 left) 2021-02-10T04:29:45.742 INFO:tasks.rados.rados.0.smithi085.stdout:400: done (7 left) 2021-02-10T04:29:45.742 INFO:tasks.rados.rados.0.smithi085.stdout:401: done (6 left) 2021-02-10T04:29:45.742 INFO:tasks.rados.rados.0.smithi085.stdout:402: done (5 left) 2021-02-10T04:29:45.742 INFO:tasks.rados.rados.0.smithi085.stdout:403: rollback oid 7 current snap is 37 2021-02-10T04:29:45.742 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 7 to 31 2021-02-10T04:29:45.743 INFO:tasks.rados.rados.0.smithi085.stdout:404: read oid 26 snap -1 2021-02-10T04:29:45.743 INFO:tasks.rados.rados.0.smithi085.stdout:404: expect (ObjNum 131 snap 24 seq_num 131) 2021-02-10T04:29:45.743 INFO:tasks.rados.rados.0.smithi085.stdout:405: delete oid 49 current snap is 37 2021-02-10T04:29:45.746 INFO:tasks.rados.rados.0.smithi085.stdout:403: finishing rollback tid 0 to smithi08575397-7 2021-02-10T04:29:45.746 INFO:tasks.rados.rados.0.smithi085.stdout:403: finishing rollback tid 1 to smithi08575397-7 2021-02-10T04:29:45.746 INFO:tasks.rados.rados.0.smithi085.stdout:403: finishing rollback tid 2 to smithi08575397-7 2021-02-10T04:29:45.746 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 70 (ObjNum 121 snap 23 seq_num 121) dirty exists 2021-02-10T04:29:45.747 INFO:tasks.rados.rados.0.smithi085.stdout:404: expect (ObjNum 131 snap 24 seq_num 131) 2021-02-10T04:29:45.852 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:29:45.883 INFO:tasks.rados.rados.0.smithi085.stdout:403: done (7 left) 2021-02-10T04:29:45.883 INFO:tasks.rados.rados.0.smithi085.stdout:404: done (6 left) 2021-02-10T04:29:45.883 INFO:tasks.rados.rados.0.smithi085.stdout:405: done (5 left) 2021-02-10T04:29:45.883 INFO:tasks.rados.rados.0.smithi085.stdout:406: write (excl) oid 49 current snap is 37 2021-02-10T04:29:45.883 INFO:tasks.rados.rados.0.smithi085.stdout:406: seq_num 172 ranges {490804=702594,1615491=584710,2724284=166520} 2021-02-10T04:29:45.890 INFO:tasks.rados.rados.0.smithi085.stdout:406: writing smithi08575397-49 from 490804 to 1193398 tid 1 2021-02-10T04:29:45.896 INFO:tasks.rados.rados.0.smithi085.stdout:406: writing smithi08575397-49 from 1615491 to 2200201 tid 2 2021-02-10T04:29:45.900 INFO:tasks.rados.rados.0.smithi085.stdout:406: writing smithi08575397-49 from 2724284 to 2890804 tid 3 2021-02-10T04:29:45.901 INFO:tasks.rados.rados.0.smithi085.stdout:407: delete oid 24 current snap is 37 2021-02-10T04:29:45.901 INFO:tasks.rados.rados.0.smithi085.stdout:406: finishing write tid 1 to smithi08575397-49 2021-02-10T04:29:45.901 INFO:tasks.rados.rados.0.smithi085.stdout:406: finishing write tid 2 to smithi08575397-49 2021-02-10T04:29:45.904 INFO:tasks.rados.rados.0.smithi085.stdout:406: finishing write tid 3 to smithi08575397-49 2021-02-10T04:29:45.904 INFO:tasks.rados.rados.0.smithi085.stdout:406: finishing write tid 4 to smithi08575397-49 2021-02-10T04:29:45.904 INFO:tasks.rados.rados.0.smithi085.stdout:407: done (6 left) 2021-02-10T04:29:45.904 INFO:tasks.rados.rados.0.smithi085.stdout:408: rollback oid 41 current snap is 37 2021-02-10T04:29:45.904 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 41 to 34 2021-02-10T04:29:45.904 INFO:tasks.rados.rados.0.smithi085.stdout:406: finishing write tid 5 to smithi08575397-49 2021-02-10T04:29:45.904 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 49 v 60 (ObjNum 172 snap 37 seq_num 172) dirty exists 2021-02-10T04:29:45.904 INFO:tasks.rados.rados.0.smithi085.stdout:406: left oid 49 (ObjNum 172 snap 37 seq_num 172) 2021-02-10T04:29:45.904 INFO:tasks.rados.rados.0.smithi085.stdout:406: done (6 left) 2021-02-10T04:29:45.904 INFO:tasks.rados.rados.0.smithi085.stdout:409: write oid 8 current snap is 37 2021-02-10T04:29:45.904 INFO:tasks.rados.rados.0.smithi085.stdout:409: seq_num 174 ranges {524418=453092,1732455=588379,2779130=145288} 2021-02-10T04:29:45.910 INFO:tasks.rados.rados.0.smithi085.stdout:409: writing smithi08575397-8 from 524418 to 977510 tid 1 2021-02-10T04:29:45.919 INFO:tasks.rados.rados.0.smithi085.stdout:409: writing smithi08575397-8 from 1732455 to 2320834 tid 2 2021-02-10T04:29:45.923 INFO:tasks.rados.rados.0.smithi085.stdout:409: writing smithi08575397-8 from 2779130 to 2924418 tid 3 2021-02-10T04:29:45.924 INFO:tasks.rados.rados.0.smithi085.stdout:410: read oid 45 snap -1 2021-02-10T04:29:45.924 INFO:tasks.rados.rados.0.smithi085.stdout:410: expect (ObjNum 96 snap 19 seq_num 96) 2021-02-10T04:29:45.924 INFO:tasks.rados.rados.0.smithi085.stdout:408: finishing rollback tid 0 to smithi08575397-41 2021-02-10T04:29:45.924 INFO:tasks.rados.rados.0.smithi085.stdout:408: finishing rollback tid 1 to smithi08575397-41 2021-02-10T04:29:45.924 INFO:tasks.rados.rados.0.smithi085.stdout:408: finishing rollback tid 2 to smithi08575397-41 2021-02-10T04:29:45.924 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 41 v 179 (ObjNum 147 snap 30 seq_num 147) dirty exists 2021-02-10T04:29:45.924 INFO:tasks.rados.rados.0.smithi085.stdout:409: finishing write tid 1 to smithi08575397-8 2021-02-10T04:29:45.924 INFO:tasks.rados.rados.0.smithi085.stdout:409: finishing write tid 2 to smithi08575397-8 2021-02-10T04:29:45.924 INFO:tasks.rados.rados.0.smithi085.stdout:411: read oid 43 snap -1 2021-02-10T04:29:45.924 INFO:tasks.rados.rados.0.smithi085.stdout:411: expect deleted 2021-02-10T04:29:45.924 INFO:tasks.rados.rados.0.smithi085.stdout:408: done (8 left) 2021-02-10T04:29:45.925 INFO:tasks.rados.rados.0.smithi085.stdout:412: read oid 50 snap -1 2021-02-10T04:29:45.925 INFO:tasks.rados.rados.0.smithi085.stdout:412: expect deleted 2021-02-10T04:29:45.925 INFO:tasks.rados.rados.0.smithi085.stdout:413: read oid 5 snap -1 2021-02-10T04:29:45.925 INFO:tasks.rados.rados.0.smithi085.stdout:413: expect (ObjNum 148 snap 30 seq_num 148) 2021-02-10T04:29:45.925 INFO:tasks.rados.rados.0.smithi085.stdout:414: copy_from oid 1 from oid 23 current snap is 37 2021-02-10T04:29:45.925 INFO:tasks.rados.rados.0.smithi085.stdout:415: copy_from oid 44 from oid 39 current snap is 37 2021-02-10T04:29:45.925 INFO:tasks.rados.rados.0.smithi085.stdout:416: snap_remove snap 30 2021-02-10T04:29:46.084 INFO:tasks.rados.rados.0.smithi085.stdout:416: done (12 left) 2021-02-10T04:29:46.084 INFO:tasks.rados.rados.0.smithi085.stdout:417: write (excl) oid 6 current snap is 37 2021-02-10T04:29:46.084 INFO:tasks.rados.rados.0.smithi085.stdout:417: seq_num 175 ranges {541225=644694,1907290=448390} 2021-02-10T04:29:46.091 INFO:tasks.rados.rados.0.smithi085.stdout:417: writing smithi08575397-6 from 541225 to 1185919 tid 1 2021-02-10T04:29:46.098 INFO:tasks.rados.rados.0.smithi085.stdout:417: writing smithi08575397-6 from 1907290 to 2355680 tid 2 2021-02-10T04:29:46.099 INFO:tasks.rados.rados.0.smithi085.stdout:418: snap_create 2021-02-10T04:29:46.099 INFO:tasks.rados.rados.0.smithi085.stdout:415: finishing copy_from to smithi08575397-44 2021-02-10T04:29:46.099 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 44 (ObjNum 161 snap 36 seq_num 161) dirty exists 2021-02-10T04:29:46.099 INFO:tasks.rados.rados.0.smithi085.stdout:415: finishing copy_from racing read to smithi08575397-44 2021-02-10T04:29:46.099 INFO:tasks.rados.rados.0.smithi085.stdout:410: expect (ObjNum 96 snap 19 seq_num 96) 2021-02-10T04:29:46.293 INFO:tasks.rados.rados.0.smithi085.stdout:409: finishing write tid 3 to smithi08575397-8 2021-02-10T04:29:46.294 INFO:tasks.rados.rados.0.smithi085.stdout:409: finishing write tid 4 to smithi08575397-8 2021-02-10T04:29:46.294 INFO:tasks.rados.rados.0.smithi085.stdout:409: finishing write tid 5 to smithi08575397-8 2021-02-10T04:29:46.294 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 65 (ObjNum 174 snap 37 seq_num 174) dirty exists 2021-02-10T04:29:46.294 INFO:tasks.rados.rados.0.smithi085.stdout:409: left oid 8 (ObjNum 174 snap 37 seq_num 174) 2021-02-10T04:29:46.294 INFO:tasks.rados.rados.0.smithi085.stdout:413: expect (ObjNum 148 snap 30 seq_num 148) 2021-02-10T04:29:46.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:46 smithi164 conmon[29288]: cluster 2021-02-10T04:29:46.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:46 smithi164 conmon[29288]: 2021-02 2021-02-10T04:29:46.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:46 smithi164 conmon[29288]: -10T04:29:45.063038+0000 mon.a (mon.0) 856 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2021-02-10T04:29:46.397 INFO:tasks.rados.rados.0.smithi085.stdout:414: finishing copy_from to smithi08575397-1 2021-02-10T04:29:46.397 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 34 (ObjNum 124 snap 23 seq_num 124) dirty exists 2021-02-10T04:29:46.397 INFO:tasks.rados.rados.0.smithi085.stdout:414: finishing copy_from racing read to smithi08575397-1 2021-02-10T04:29:46.397 INFO:tasks.rados.rados.0.smithi085.stdout:417: finishing write tid 1 to smithi08575397-6 2021-02-10T04:29:46.397 INFO:tasks.rados.rados.0.smithi085.stdout:417: finishing write tid 2 to smithi08575397-6 2021-02-10T04:29:46.397 INFO:tasks.rados.rados.0.smithi085.stdout:417: finishing write tid 3 to smithi08575397-6 2021-02-10T04:29:46.397 INFO:tasks.rados.rados.0.smithi085.stdout:417: finishing write tid 4 to smithi08575397-6 2021-02-10T04:29:46.397 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 125 (ObjNum 175 snap 37 seq_num 175) dirty exists 2021-02-10T04:29:46.398 INFO:tasks.rados.rados.0.smithi085.stdout:417: left oid 6 (ObjNum 175 snap 37 seq_num 175) 2021-02-10T04:29:46.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:46 smithi085 conmon[29319]: cluster 2021-02-10T04:29:45. 2021-02-10T04:29:46.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:46 smithi085 conmon[29319]: 063038+0000 mon.a (mon.0) 856 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2021-02-10T04:29:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:46 smithi085 conmon[35961]: cluster 2021-02-10T04:29:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:46 smithi085 conmon[35961]: 2021-02-10T04:29:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:46 smithi085 conmon[35961]: 2021 2021-02-10T04:29:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:46 smithi085 conmon[35961]: - 2021-02-10T04:29:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:46 smithi085 conmon[35961]: 02 2021-02-10T04:29:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:46 smithi085 conmon[35961]: -10T04:29:45.063038 2021-02-10T04:29:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:46 smithi085 conmon[35961]: +0000 mon.a (mon.0) 856 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2021-02-10T04:29:47.095 INFO:tasks.rados.rados.0.smithi085.stdout:409: done (13 left) 2021-02-10T04:29:47.096 INFO:tasks.rados.rados.0.smithi085.stdout:410: done (12 left) 2021-02-10T04:29:47.096 INFO:tasks.rados.rados.0.smithi085.stdout:411: done (11 left) 2021-02-10T04:29:47.096 INFO:tasks.rados.rados.0.smithi085.stdout:412: done (10 left) 2021-02-10T04:29:47.096 INFO:tasks.rados.rados.0.smithi085.stdout:413: done (9 left) 2021-02-10T04:29:47.096 INFO:tasks.rados.rados.0.smithi085.stdout:414: done (8 left) 2021-02-10T04:29:47.096 INFO:tasks.rados.rados.0.smithi085.stdout:415: done (7 left) 2021-02-10T04:29:47.096 INFO:tasks.rados.rados.0.smithi085.stdout:417: done (6 left) 2021-02-10T04:29:47.096 INFO:tasks.rados.rados.0.smithi085.stdout:418: done (5 left) 2021-02-10T04:29:47.096 INFO:tasks.rados.rados.0.smithi085.stdout:419: write oid 5 current snap is 38 2021-02-10T04:29:47.096 INFO:tasks.rados.rados.0.smithi085.stdout:419: seq_num 176 ranges {558032=719943,1849419=592048} 2021-02-10T04:29:47.097 INFO:tasks.rados.rados.0.smithi085.stdout:419: writing smithi08575397-5 from 558032 to 1277975 tid 1 2021-02-10T04:29:47.105 INFO:tasks.rados.rados.0.smithi085.stdout:419: writing smithi08575397-5 from 1849419 to 2441467 tid 2 2021-02-10T04:29:47.106 INFO:tasks.rados.rados.0.smithi085.stdout:420: snap_remove snap 37 2021-02-10T04:29:47.250 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:29:47.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:47 smithi164 conmon[29288]: cluster 2021-02-10T04:29:46.008600+0000 mgr.y (mgr.14140 2021-02-10T04:29:47.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:47 smithi164 conmon[29288]: ) 376 : cluster [DBG] pgmap v403: 168 pgs: 3 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 158 active+clean; 134 MiB data, 400 MiB used, 715 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2021-02-10T04:29:47.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:47 smithi164 conmon[29288]: cluster 2021-02-10T04:29:46.082831+0000 mon.a (mon.0) 857 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2021-02-10T04:29:47.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:29:47 smithi085 conmon[49106]: debug 2021-02-10T04:29:47.048+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:47.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:47 smithi085 conmon[29319]: cluster 2021-02-10T04:29:46.008600+0000 mgr.y (mgr.14140) 376 : cluster [DBG] 2021-02-10T04:29:47.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:47 smithi085 conmon[29319]: pgmap v403: 168 pgs: 3 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 158 active+clean; 134 MiB data, 400 MiB used, 715 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2021-02-10T04:29:47.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:47 smithi085 conmon[29319]: cluster 2021-02-10T04:29:46.082831+0000 mon.a (mon 2021-02-10T04:29:47.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:47 smithi085 conmon[29319]: .0) 857 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2021-02-10T04:29:47.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:47 smithi085 conmon[35961]: cluster 2021-02-10T04:29:46. 2021-02-10T04:29:47.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:47 smithi085 conmon[35961]: 008600+0000 mgr.y (mgr.14140) 376 : cluster [DBG] pgmap v403: 168 pgs: 3 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 158 active+clean; 134 MiB data, 400 MiB used, 715 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2021-02-10T04:29:47.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:47 smithi085 conmon[35961]: cluster 2021-02-10T04:29:47.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:47 smithi085 conmon[35961]: 2021-02-10T04:29:46.082831+0000 mon.a (mon.0) 857 : 2021-02-10T04:29:47.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:47 smithi085 conmon[35961]: cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2021-02-10T04:29:48.048 INFO:tasks.rados.rados.0.smithi085.stdout:420: done (6 left) 2021-02-10T04:29:48.048 INFO:tasks.rados.rados.0.smithi085.stdout:421: write oid 15 current snap is 38 2021-02-10T04:29:48.048 INFO:tasks.rados.rados.0.smithi085.stdout:421: seq_num 177 ranges {574839=795192,1907901=452059} 2021-02-10T04:29:48.056 INFO:tasks.rados.rados.0.smithi085.stdout:421: writing smithi08575397-15 from 574839 to 1370031 tid 1 2021-02-10T04:29:48.063 INFO:tasks.rados.rados.0.smithi085.stdout:421: writing smithi08575397-15 from 1907901 to 2359960 tid 2 2021-02-10T04:29:48.063 INFO:tasks.rados.rados.0.smithi085.stdout:422: snap_remove snap 34 2021-02-10T04:29:48.064 INFO:tasks.rados.rados.0.smithi085.stdout:419: finishing write tid 1 to smithi08575397-5 2021-02-10T04:29:48.064 INFO:tasks.rados.rados.0.smithi085.stdout:419: finishing write tid 2 to smithi08575397-5 2021-02-10T04:29:48.064 INFO:tasks.rados.rados.0.smithi085.stdout:419: finishing write tid 3 to smithi08575397-5 2021-02-10T04:29:48.064 INFO:tasks.rados.rados.0.smithi085.stdout:419: finishing write tid 4 to smithi08575397-5 2021-02-10T04:29:48.064 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 88 (ObjNum 176 snap 38 seq_num 176) dirty exists 2021-02-10T04:29:48.064 INFO:tasks.rados.rados.0.smithi085.stdout:419: left oid 5 (ObjNum 176 snap 38 seq_num 176) 2021-02-10T04:29:48.064 INFO:tasks.rados.rados.0.smithi085.stdout:421: finishing write tid 1 to smithi08575397-15 2021-02-10T04:29:48.122 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:29:48.130 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:29:47 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:29:48.132 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:48 smithi164 conmon[29288]: cluster 2021-02-10T04:29:46.589060+0000 osd.1 (osd.1) 2021-02-10T04:29:48.132 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:48 smithi164 conmon[29288]: 52 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:29:48.132 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:48 smithi164 conmon[29288]: cluster 2021-02-10T04:29:46.589177+0000 osd.1 (osd.1) 53 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:29:48.132 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:48 smithi164 conmon[29288]: cluster 2021-02-10T04:29:47 2021-02-10T04:29:48.132 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:48 smithi164 conmon[29288]: .089278+0000 mon.a (mon.0) 858 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2021-02-10T04:29:48.132 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:48 smithi164 conmon[29288]: cluster 2021-02-10T04:29:48.132 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:48 smithi164 conmon[29288]: 2021-02-10T04:29:48.047332+0000 mon.a (mon.0 2021-02-10T04:29:48.132 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:48 smithi164 conmon[29288]: ) 859 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2021-02-10T04:29:48.133 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.4 dump_ops_in_flight 2021-02-10T04:29:48.157 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:48.249 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:29:48.355 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:29:48.361 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:48 smithi085 conmon[29319]: cluster 2021-02-10T04:29:46.589060+0000 osd.1 (osd.1) 52 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:29:48.361 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:48 smithi085 conmon[29319]: cluster 2021-02-10T04:29:46.589177+0000 osd.1 (osd.1) 53 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:29:48.361 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:48 smithi085 conmon[29319]: cluster 2021-02-10T04:29:47.089278+0000 mon.a (mon.0) 858 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2021-02-10T04:29:48.362 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:48 smithi085 conmon[29319]: cluster 2021-02-10T04:29:48.047332+0000 mon.a (mon.0) 859 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2021-02-10T04:29:48.362 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:48 smithi085 conmon[35961]: cluster 2021-02-10T04:29:46.589060+0000 osd.1 (osd.1) 52 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:29:48.363 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:48 smithi085 conmon[35961]: cluster 2021-02-10T04:29:46.589177+0000 osd.1 (osd.1) 53 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:29:48.363 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:48 smithi085 conmon[35961]: cluster 2021-02-10T04:29:47.089278+0000 mon.a (mon.0) 858 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2021-02-10T04:29:48.363 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:48 smithi085 conmon[35961]: cluster 2021-02-10T04:29:48.047332+0000 mon.a (mon.0) 859 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2021-02-10T04:29:48.393 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:29:48 smithi164 conmon[40892]: debug 2021-02-10T04:29:48.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:29:48 smithi164 conmon[40892]: 2021-02-10T04:29:48.229+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:48.399 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:48.425 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:29:48.600 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:29:48.631 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:29:48 smithi085 conmon[45443]: debug 2021-02-10T04:29:48.578+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:48.704 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:29:48.741 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:48.825 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:29:48.853 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:29:48.882 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:29:48.920 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:29:48.960 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:29:48.996 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:29:49.024 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:29:49.053 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:29:49.056 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:29:49.056 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:29:49.056 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:29:49.056 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:29:49.056 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:29:49.056 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:29:49.056 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:29:49.056 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:29:49.067 INFO:tasks.rados.rados.0.smithi085.stdout:419: done (7 left) 2021-02-10T04:29:49.067 INFO:tasks.rados.rados.0.smithi085.stdout:422: done (6 left) 2021-02-10T04:29:49.067 INFO:tasks.rados.rados.0.smithi085.stdout:423: snap_remove snap 31 2021-02-10T04:29:49.191 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:29:48 smithi085 conmon[45443]: debug 2021-02-10T04:29:48.916+0000 7f7140d73700 -1 osd.1 172 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:29:49.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:49 smithi085 conmon[29319]: audit 2021-02-10T04:29:47.745026+0000 mgr.y (mgr.14140) 377 : audit 2021-02-10T04:29:49.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:49 smithi085 conmon[29319]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:29:49.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:49 smithi085 conmon[29319]: cluster 2021-02-10T04:29:48.008995+0000 mgr.y (mgr.14140) 378 : cluster [DBG] pgmap v406: 168 pgs: 4 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 155 active+clean; 133 MiB data, 403 MiB used, 715 GiB / 715 GiB avail; 5.8 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2021-02-10T04:29:49.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:49 smithi085 conmon[29319]: audit 2021-02-10T04:29:48.799777+0000 mon.a (mon.0) 860 : audit [DBG] from='client.? 172.21.15.85:0/2056611180' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:29:49.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:49 smithi085 conmon[29319]: cluster 2021-02-10T04:29:49.067536+0000 mon.a (mon.0) 861 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2021-02-10T04:29:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:49 smithi085 conmon[35961]: audit 2021-02-10T04:29:47. 2021-02-10T04:29:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:49 smithi085 conmon[35961]: 745026+0000 mgr.y (mgr.14140) 377 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:29:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:49 smithi085 conmon[35961]: cluster 2021-02-10T04:29:48.008995+0000 mgr.y (mgr.14140) 378 : cluster [DBG] pgmap v406: 168 pgs: 4 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 155 active+clean; 133 MiB data, 403 MiB used, 715 GiB / 715 GiB avail; 5.8 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2021-02-10T04:29:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:49 smithi085 conmon[35961]: audit 2021-02-10T04:29:48.799777+0000 mon.a (mon.0) 860 : audit [DBG] from='client.? 172.21.15.85:0/2056611180' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:29:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:49 smithi085 conmon[35961]: cluster 2021-02-10T04:29:49.067536+0000 mon.a (mon.0) 2021-02-10T04:29:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:49 smithi085 conmon[35961]: 861 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2021-02-10T04:29:49.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:49 smithi164 conmon[29288]: audit 2021-02-10T04:29:47.745026+0000 2021-02-10T04:29:49.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:49 smithi164 conmon[29288]: mgr.y (mgr.14140) 377 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:29:49.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:49 smithi164 conmon[29288]: cluster 2021-02-10T04:29:48.008995+0000 mgr.y (mgr.14140) 378 : cluster [DBG] pgmap v406: 168 pgs: 4 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 155 active+clean; 133 MiB data, 403 MiB used, 715 GiB / 715 GiB avail; 5.8 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2021-02-10T04:29:49.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:49 smithi164 conmon[29288]: audit 2021-02-10T04:29:48.799777+0000 mon.a (mon.0) 860 : audit [DBG] from='client.? 172.21.15.85:0/2056611180' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:29:49.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:49 smithi164 conmon[29288]: cluster 2021-02-10T04:29:49.067536+0000 mon.a (mon.0) 861 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2021-02-10T04:29:49.530 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:29:49.690 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:29:49 smithi085 conmon[52836]: debug 2021-02-10T04:29:49.691 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:29:49 smithi085 conmon[52836]: 2021-02-10T04:29:49.221+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:50.068 INFO:tasks.rados.rados.0.smithi085.stdout:423: done (6 left) 2021-02-10T04:29:50.069 INFO:tasks.rados.rados.0.smithi085.stdout:424: snap_create 2021-02-10T04:29:50.069 INFO:tasks.rados.rados.0.smithi085.stdout:421: finishing write tid 2 to smithi08575397-15 2021-02-10T04:29:50.069 INFO:tasks.rados.rados.0.smithi085.stdout:421: finishing write tid 3 to smithi08575397-15 2021-02-10T04:29:50.069 INFO:tasks.rados.rados.0.smithi085.stdout:421: finishing write tid 4 to smithi08575397-15 2021-02-10T04:29:50.069 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 15 v 84 (ObjNum 177 snap 38 seq_num 177) dirty exists 2021-02-10T04:29:50.069 INFO:tasks.rados.rados.0.smithi085.stdout:421: left oid 15 (ObjNum 177 snap 38 seq_num 177) 2021-02-10T04:29:50.072 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.1 to 1 (10000) 2021-02-10T04:29:50.129 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:29:50.130 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [ 2021-02-10T04:29:50.130 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.130 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156)", 2021-02-10T04:29:50.130 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:32.164063+0000", 2021-02-10T04:29:50.130 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 17.965306368, 2021-02-10T04:29:50.130 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 17.965346632999999, 2021-02-10T04:29:50.130 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:29:50.131 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:29:50.131 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:29:50.131 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:29:50.131 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:29:50.131 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 910 2021-02-10T04:29:50.131 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.131 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:29:50.131 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.131 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:29:50.131 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164063+0000", 2021-02-10T04:29:50.132 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:29:50.132 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.132 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.132 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:29:50.132 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164063+0000", 2021-02-10T04:29:50.132 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.8188e-05 2021-02-10T04:29:50.132 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.133 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.133 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:29:50.133 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164111+0000", 2021-02-10T04:29:50.133 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0791000000000001e-05 2021-02-10T04:29:50.133 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.133 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.133 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:29:50.133 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164122+0000", 2021-02-10T04:29:50.133 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5682999999999999e-05 2021-02-10T04:29:50.134 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.134 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.134 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:29:50.134 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164138+0000", 2021-02-10T04:29:50.134 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0522e-05 2021-02-10T04:29:50.134 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.134 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.134 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:29:50.134 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164148+0000", 2021-02-10T04:29:50.135 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.9583999999999997e-05 2021-02-10T04:29:50.135 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.135 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.135 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:50.135 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164208+0000", 2021-02-10T04:29:50.135 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00013592500000000001 2021-02-10T04:29:50.135 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.135 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.135 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:29:50.136 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164344+0000", 2021-02-10T04:29:50.136 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00028069299999999999 2021-02-10T04:29:50.136 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:50.136 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:29:50.136 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:50.136 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.136 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.136 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156)", 2021-02-10T04:29:50.137 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:32.164227+0000", 2021-02-10T04:29:50.137 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 17.965141936999999, 2021-02-10T04:29:50.137 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 17.965272195000001, 2021-02-10T04:29:50.137 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:29:50.137 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "delayed", 2021-02-10T04:29:50.137 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:29:50.137 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:29:50.137 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:29:50.137 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 911 2021-02-10T04:29:50.137 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.138 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:29:50.138 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.138 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:29:50.138 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164227+0000", 2021-02-10T04:29:50.138 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:29:50.138 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.138 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.138 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:29:50.139 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164227+0000", 2021-02-10T04:29:50.139 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.3152999999999997e-05 2021-02-10T04:29:50.139 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.139 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.139 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:29:50.139 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164291+0000", 2021-02-10T04:29:50.139 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.8859999999999993e-06 2021-02-10T04:29:50.139 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.139 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.140 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:29:50.140 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164299+0000", 2021-02-10T04:29:50.140 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5695000000000001e-05 2021-02-10T04:29:50.140 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.140 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.140 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:29:50.140 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164315+0000", 2021-02-10T04:29:50.140 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.3790000000000003e-06 2021-02-10T04:29:50.140 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.140 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.141 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:29:50.141 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164322+0000", 2021-02-10T04:29:50.141 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00017179399999999999 2021-02-10T04:29:50.141 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.141 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.141 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:50.141 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164493+0000", 2021-02-10T04:29:50.141 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.9685000000000002e-05 2021-02-10T04:29:50.141 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.142 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.142 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:29:50.142 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164553+0000", 2021-02-10T04:29:50.142 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00032559199999999998 2021-02-10T04:29:50.142 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:50.142 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:29:50.142 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:50.142 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.143 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.143 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153)", 2021-02-10T04:29:50.143 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:29.061753+0000", 2021-02-10T04:29:50.143 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 21.067616298000001, 2021-02-10T04:29:50.143 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 21.067812836000002, 2021-02-10T04:29:50.143 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:29:50.143 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:29:50.143 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:29:50.143 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:29:50.144 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:29:50.144 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 865 2021-02-10T04:29:50.144 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.144 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:29:50.144 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.144 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:29:50.144 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061753+0000", 2021-02-10T04:29:50.144 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:29:50.144 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.145 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.145 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:29:50.145 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061753+0000", 2021-02-10T04:29:50.145 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.3646999999999997e-05 2021-02-10T04:29:50.145 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.145 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.145 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:29:50.145 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061797+0000", 2021-02-10T04:29:50.145 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.4220000000000003e-06 2021-02-10T04:29:50.146 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.146 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.146 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:29:50.146 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061804+0000", 2021-02-10T04:29:50.146 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.9582000000000002e-05 2021-02-10T04:29:50.146 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.146 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.146 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:29:50.147 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061844+0000", 2021-02-10T04:29:50.147 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.9789999999999996e-06 2021-02-10T04:29:50.147 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.147 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.147 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:29:50.147 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061851+0000", 2021-02-10T04:29:50.147 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00012051 2021-02-10T04:29:50.147 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.147 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.148 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:50.148 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061971+0000", 2021-02-10T04:29:50.148 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.0452999999999998e-05 2021-02-10T04:29:50.148 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.148 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.148 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:29:50.148 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.062042+0000", 2021-02-10T04:29:50.148 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00028859300000000002 2021-02-10T04:29:50.148 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:50.149 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:29:50.149 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:50.149 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.149 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.149 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153)", 2021-02-10T04:29:50.149 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:29.061891+0000", 2021-02-10T04:29:50.149 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 21.067478556000001, 2021-02-10T04:29:50.149 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 21.067720631, 2021-02-10T04:29:50.150 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:29:50.150 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "delayed", 2021-02-10T04:29:50.150 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:29:50.150 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:29:50.150 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:29:50.150 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 866 2021-02-10T04:29:50.150 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.150 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:29:50.151 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.151 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:29:50.151 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061891+0000", 2021-02-10T04:29:50.151 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:29:50.151 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.151 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.151 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:29:50.152 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061891+0000", 2021-02-10T04:29:50.152 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1696000000000003e-05 2021-02-10T04:29:50.152 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.152 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.152 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:29:50.152 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061923+0000", 2021-02-10T04:29:50.152 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.7699999999999998e-06 2021-02-10T04:29:50.152 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.152 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.153 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:29:50.153 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061928+0000", 2021-02-10T04:29:50.153 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1918e-05 2021-02-10T04:29:50.153 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.153 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.153 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:29:50.153 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061940+0000", 2021-02-10T04:29:50.153 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.8640000000000004e-06 2021-02-10T04:29:50.154 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.154 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.154 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:29:50.154 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061945+0000", 2021-02-10T04:29:50.154 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00022406100000000001 2021-02-10T04:29:50.154 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.154 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.154 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:50.154 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.062169+0000", 2021-02-10T04:29:50.155 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.0158999999999999e-05 2021-02-10T04:29:50.155 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:50.155 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:50.155 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:29:50.155 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.062219+0000", 2021-02-10T04:29:50.155 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000328468 2021-02-10T04:29:50.155 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:50.155 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:29:50.155 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:50.155 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:50.156 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2021-02-10T04:29:50.156 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 4 2021-02-10T04:29:50.156 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:29:50.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:50 smithi164 conmon[29288]: cluster 2021-02-10T04:29:48.916965+0000 osd.1 (osd.1) 54 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:29:50.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:50 smithi164 conmon[29288]: cluster 2021-02-10T04:29:48.916979+0000 osd.1 (osd.1) 55 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:29:50.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:50 smithi164 conmon[29288]: audit 2021-02-10T04:29:49.278843+0000 mon.a (mon.0) 862 : audit [INF] from='client.? 172.21.15.85:0/513459424' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-02-10T04:29:50.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:50 smithi164 conmon[29288]: cluster 2021-02-10T04:29:50.009500+0000 mgr.y (mgr.14140) 379 : cluster [DBG] pgmap v409: 168 pgs: 4 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 155 active+clean; 131 MiB data, 409 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 2.0 MiB/s wr, 16 op/s 2021-02-10T04:29:50.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:50 smithi164 conmon[29288]: audit 2021-02-10T04:29:50.068177+0000 mon.a (mon.0) 863 : audit [INF] from='client.? 172.21.15.85:0/513459424' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-02-10T04:29:50.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:50 smithi164 conmon[29288]: cluster 2021-02-10T04:29:50.068245+0000 mon.a (mon.0) 864 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2021-02-10T04:29:50.441 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:29:49 smithi085 conmon[45443]: debug 2021-02-10T04:29:49.964+0000 7f7140d73700 -1 osd.1 173 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:29:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[35961]: cluster 2021-02-10T04:29:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:29:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[35961]: :29:48.916965+0000 osd.1 (osd.1) 54 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:29:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:29:50.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[35961]: -02-10T04:29:48.916979+0000 osd.1 2021-02-10T04:29:50.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[35961]: (osd.1) 55 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:29:50.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[35961]: audit 2021-02-10T04:29:49.278843+0000 mon.a (mon.0 2021-02-10T04:29:50.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[35961]: ) 862 : audit [INF] from='client.? 172.21.15.85:0/513459424' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-02-10T04:29:50.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:29:50.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[35961]: 02-10T04:29:50.009500 2021-02-10T04:29:50.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 2021-02-10T04:29:50.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[35961]: 379 : cluster [DBG] pgmap v409: 168 pgs: 4 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 155 active+clean; 131 MiB data, 409 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 2.0 MiB/s wr, 16 op/s 2021-02-10T04:29:50.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:29:50.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[35961]: 02-10T04:29:50.068177+0000 mon.a (mon.0 2021-02-10T04:29:50.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[35961]: ) 863 : audit [INF] from='client.? 172.21.15.85:0/513459424' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-02-10T04:29:50.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:29:50.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[35961]: 10T04:29:50.068245+0000 2021-02-10T04:29:50.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[35961]: mon.a (mon.0) 864 2021-02-10T04:29:50.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[35961]: : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2021-02-10T04:29:50.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[29319]: cluster 2021-02-10T04:29:48.916965+0000 osd.1 (osd.1) 2021-02-10T04:29:50.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[29319]: 54 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:29:50.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[29319]: cluster 2021-02-10T04:29:48.916979+0000 osd.1 (osd.1) 55 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:29:50.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[29319]: audit 2021-02-10T04:29:49.278843+0000 2021-02-10T04:29:50.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[29319]: mon.a (mon.0) 862 : audit [INF] from='client.? 172.21.15.85:0/513459424' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-02-10T04:29:50.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:29:50.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[29319]: T04:29:50.009500+0000 mgr.y (mgr.14140) 379 : cluster [DBG] 2021-02-10T04:29:50.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[29319]: pgmap v409: 168 pgs: 4 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 155 active+clean; 131 MiB data, 409 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 2.0 MiB/s wr, 16 op/s 2021-02-10T04:29:50.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[29319]: audit 2021-02-10T04:29:50.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[29319]: 2021-02-10T04:29:50.068177+0000 mon.a (mon.0) 2021-02-10T04:29:50.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[29319]: 863 : audit [INF] from='client.? 172.21.15.85:0/513459424' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-02-10T04:29:50.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[29319]: cluster 2021-02-10T04:29:50 2021-02-10T04:29:50.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[29319]: .068245+0000 mon.a (mon.0) 864 : cluster [DBG] 2021-02-10T04:29:50.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:50 smithi085 conmon[29319]: osdmap e174: 8 total, 8 up, 8 in 2021-02-10T04:29:50.727 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.4 dump_blocked_ops 2021-02-10T04:29:50.923 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:29:50.954 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:50.985 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:29:51.068 INFO:tasks.rados.rados.0.smithi085.stdout:421: done (6 left) 2021-02-10T04:29:51.069 INFO:tasks.rados.rados.0.smithi085.stdout:424: done (5 left) 2021-02-10T04:29:51.069 INFO:tasks.rados.rados.0.smithi085.stdout:425: write (excl) oid 9 current snap is 39 2021-02-10T04:29:51.069 INFO:tasks.rados.rados.0.smithi085.stdout:425: seq_num 178 ranges {591646=470441,1566383=595717,2888822=102824} 2021-02-10T04:29:51.072 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[45443]: debug 2021-02-10T04:29:51.073 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[45443]: 2021-02-10T04:29:51.011+0000 7f7140d73700 -1 osd.1 174 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:29:51.073 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[45443]: 2021-02-10T04:29:51.073 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[45443]: debug 2021-02-10T04:29:51.071+0000 7f71264f9700 -1 osd.1 pg_epoch: 174 pg[7.b( v 174'119 (0'0,174'119] local-lis/les=93/94 n=5 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [1,5,6] r=0 lpr=93 luod=173'117 crt=174'119 lcod 170'115 mlcod 170'115 active+clean+snaptrim trimq=[3e~1] ps=[2e~1,3d~1,40~1,42~2,46~1,4a~1]] removing snap head 2021-02-10T04:29:51.074 INFO:tasks.rados.rados.0.smithi085.stdout:425: writing smithi08575397-9 from 591646 to 1062087 tid 1 2021-02-10T04:29:51.085 INFO:tasks.rados.rados.0.smithi085.stdout:425: writing smithi08575397-9 from 1566383 to 2162100 tid 2 2021-02-10T04:29:51.090 INFO:tasks.rados.rados.0.smithi085.stdout:425: writing smithi08575397-9 from 2888822 to 2991646 tid 3 2021-02-10T04:29:51.090 INFO:tasks.rados.rados.0.smithi085.stdout:425: finishing write tid 1 to smithi08575397-9 2021-02-10T04:29:51.091 INFO:tasks.rados.rados.0.smithi085.stdout:426: write (excl) oid 3 current snap is 39 2021-02-10T04:29:51.091 INFO:tasks.rados.rados.0.smithi085.stdout:426: seq_num 179 ranges {608453=545690,1624865=455728,2632598=375855} 2021-02-10T04:29:51.098 INFO:tasks.rados.rados.0.smithi085.stdout:426: writing smithi08575397-3 from 608453 to 1154143 tid 1 2021-02-10T04:29:51.104 INFO:tasks.rados.rados.0.smithi085.stdout:426: writing smithi08575397-3 from 1624865 to 2080593 tid 2 2021-02-10T04:29:51.110 INFO:tasks.rados.rados.0.smithi085.stdout:426: writing smithi08575397-3 from 2632598 to 3008453 tid 3 2021-02-10T04:29:51.110 INFO:tasks.rados.rados.0.smithi085.stdout:427: snap_create 2021-02-10T04:29:51.110 INFO:tasks.rados.rados.0.smithi085.stdout:425: finishing write tid 2 to smithi08575397-9 2021-02-10T04:29:51.110 INFO:tasks.rados.rados.0.smithi085.stdout:425: finishing write tid 3 to smithi08575397-9 2021-02-10T04:29:51.110 INFO:tasks.rados.rados.0.smithi085.stdout:425: finishing write tid 4 to smithi08575397-9 2021-02-10T04:29:51.110 INFO:tasks.rados.rados.0.smithi085.stdout:425: finishing write tid 5 to smithi08575397-9 2021-02-10T04:29:51.110 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 9 v 125 (ObjNum 178 snap 39 seq_num 178) dirty exists 2021-02-10T04:29:51.110 INFO:tasks.rados.rados.0.smithi085.stdout:425: left oid 9 (ObjNum 178 snap 39 seq_num 178) 2021-02-10T04:29:51.110 INFO:tasks.rados.rados.0.smithi085.stdout:426: finishing write tid 1 to smithi08575397-3 2021-02-10T04:29:51.111 INFO:tasks.rados.rados.0.smithi085.stdout:426: finishing write tid 2 to smithi08575397-3 2021-02-10T04:29:51.113 INFO:tasks.rados.rados.0.smithi085.stdout:426: finishing write tid 3 to smithi08575397-3 2021-02-10T04:29:51.113 INFO:tasks.rados.rados.0.smithi085.stdout:426: finishing write tid 4 to smithi08575397-3 2021-02-10T04:29:51.114 INFO:tasks.rados.rados.0.smithi085.stdout:426: finishing write tid 5 to smithi08575397-3 2021-02-10T04:29:51.114 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 74 (ObjNum 179 snap 39 seq_num 179) dirty exists 2021-02-10T04:29:51.114 INFO:tasks.rados.rados.0.smithi085.stdout:426: left oid 3 (ObjNum 179 snap 39 seq_num 179) 2021-02-10T04:29:51.139 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:51 smithi164 conmon[29288]: cluster 2021-02-10T04:29:49.964825+0000 osd.1 ( 2021-02-10T04:29:51.139 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:51 smithi164 conmon[29288]: osd.1) 56 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:29:51.139 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:51 smithi164 conmon[29288]: cluster 2021-02-10T04:29:49.964832+0000 osd.1 (osd.1) 57 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:29:51.139 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:51 smithi164 conmon[29288]: cephadm 2021-02-10T04:29:50.952124+0000 mgr.y (mgr.14140) 380 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:29:51.139 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:51 smithi164 conmon[29288]: cephadm 2021-02-10T04:29:50.952758+0000 mgr.y (mgr.14140) 381 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:29:51.139 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:51 smithi164 conmon[29288]: cluster 2021-02-10T04:29:51.066350+0000 mon.a (mon.0) 865 : cluster [DBG] 2021-02-10T04:29:51.139 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:51 smithi164 conmon[29288]: osdmap e175: 8 total, 8 up, 8 in 2021-02-10T04:29:51.393 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:29:51 smithi164 conmon[40892]: debug 2021-02-10T04:29:51.226+0000 7f30324bf700 -1 osd.6 pg_epoch: 175 pg[7.d( v 175'73 (0'0,175'73] local-lis/les=93/94 n=5 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [6,1,0] r=0 lpr=93 luod=171'71 crt=175'73 lcod 171'69 mlcod 171'69 active+clean+snaptrim trimq=[3e~1,40~1,42~2,46~1,4a~1] ps=[2e~1,3d~1]] removing snap head 2021-02-10T04:29:51.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[29319]: cluster 2021-02-10T04:29:49.964825+0000 osd.1 ( 2021-02-10T04:29:51.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[29319]: osd.1) 56 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:29:51.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[29319]: cluster 2021-02-10T04:29:49.964832+0000 osd.1 (osd.1) 57 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:29:51.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[29319]: cephadm 2021-02-10T 2021-02-10T04:29:51.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[29319]: 04:29:50. 2021-02-10T04:29:51.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[29319]: 952124+0000 mgr.y (mgr.14140) 380 : cephadm 2021-02-10T04:29:51.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[29319]: [INF] Refreshing smithi085 facts 2021-02-10T04:29:51.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[29319]: cephadm 2021-02-10T04:29:50.952758+0000 mgr.y 2021-02-10T04:29:51.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[29319]: (mgr.14140) 381 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:29:51.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:29:51.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[29319]: T04:29:51.066350+0000 mon.a (mon.0 2021-02-10T04:29:51.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[29319]: ) 865 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2021-02-10T04:29:51.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[35961]: cluster 2021-02-10T04:29:51.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[35961]: 2021-02-10T04:29:51.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[35961]: 2021 2021-02-10T04:29:51.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[35961]: - 2021-02-10T04:29:51.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[35961]: 02 2021-02-10T04:29:51.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[35961]: - 2021-02-10T04:29:51.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[35961]: 10T04:29:49 2021-02-10T04:29:51.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[35961]: .964825+0000 osd.1 (osd.1) 56 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:29:51.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:29:51.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[35961]: -02-10T04:29 2021-02-10T04:29:51.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[35961]: :49.964832+0000 osd.1 (osd.1) 57 2021-02-10T04:29:51.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:29:51.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[35961]: cephadm 2021 2021-02-10T04:29:51.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[35961]: -02-10T04:29:50.952124+0000 mgr.y (mgr 2021-02-10T04:29:51.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[35961]: .14140) 380 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:29:51.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[35961]: cephadm 2021 2021-02-10T04:29:51.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[35961]: -02-10T04:29:50.952758+0000 2021-02-10T04:29:51.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[35961]: mgr.y (mgr.14140) 2021-02-10T04:29:51.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[35961]: 381 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:29:51.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:29:51.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[35961]: 10T04:29:51.066350+0000 mon.a ( 2021-02-10T04:29:51.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[35961]: mon.0) 865 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2021-02-10T04:29:52.068 INFO:tasks.rados.rados.0.smithi085.stdout:425: done (7 left) 2021-02-10T04:29:52.068 INFO:tasks.rados.rados.0.smithi085.stdout:426: done (6 left) 2021-02-10T04:29:52.068 INFO:tasks.rados.rados.0.smithi085.stdout:427: done (5 left) 2021-02-10T04:29:52.068 INFO:tasks.rados.rados.0.smithi085.stdout:428: write oid 31 current snap is 40 2021-02-10T04:29:52.068 INFO:tasks.rados.rados.0.smithi085.stdout:428: seq_num 180 ranges {625260=620939,1683347=599386,2943668=81592} 2021-02-10T04:29:52.075 INFO:tasks.rados.rados.0.smithi085.stdout:428: writing smithi08575397-31 from 625260 to 1246199 tid 1 2021-02-10T04:29:52.082 INFO:tasks.rados.rados.0.smithi085.stdout:428: writing smithi08575397-31 from 1683347 to 2282733 tid 2 2021-02-10T04:29:52.086 INFO:tasks.rados.rados.0.smithi085.stdout:428: writing smithi08575397-31 from 2943668 to 3025260 tid 3 2021-02-10T04:29:52.087 INFO:tasks.rados.rados.0.smithi085.stdout:429: read oid 1 snap -1 2021-02-10T04:29:52.087 INFO:tasks.rados.rados.0.smithi085.stdout:429: expect (ObjNum 124 snap 23 seq_num 124) 2021-02-10T04:29:52.087 INFO:tasks.rados.rados.0.smithi085.stdout:430: copy_from oid 25 from oid 49 current snap is 40 2021-02-10T04:29:52.087 INFO:tasks.rados.rados.0.smithi085.stdout:428: finishing write tid 1 to smithi08575397-31 2021-02-10T04:29:52.087 INFO:tasks.rados.rados.0.smithi085.stdout:428: finishing write tid 2 to smithi08575397-31 2021-02-10T04:29:52.087 INFO:tasks.rados.rados.0.smithi085.stdout:431: write oid 28 current snap is 40 2021-02-10T04:29:52.087 INFO:tasks.rados.rados.0.smithi085.stdout:431: seq_num 181 ranges {642067=696188,2025476=459397,2971091=70976} 2021-02-10T04:29:52.094 INFO:tasks.rados.rados.0.smithi085.stdout:431: writing smithi08575397-28 from 642067 to 1338255 tid 1 2021-02-10T04:29:52.101 INFO:tasks.rados.rados.0.smithi085.stdout:431: writing smithi08575397-28 from 2025476 to 2484873 tid 2 2021-02-10T04:29:52.105 INFO:tasks.rados.rados.0.smithi085.stdout:431: writing smithi08575397-28 from 2971091 to 3042067 tid 3 2021-02-10T04:29:52.105 INFO:tasks.rados.rados.0.smithi085.stdout:432: write (excl) oid 7 current snap is 40 2021-02-10T04:29:52.105 INFO:tasks.rados.rados.0.smithi085.stdout:428: finishing write tid 3 to smithi08575397-31 2021-02-10T04:29:52.105 INFO:tasks.rados.rados.0.smithi085.stdout:432: seq_num 182 ranges {658874=771437,2083958=603055} 2021-02-10T04:29:52.113 INFO:tasks.rados.rados.0.smithi085.stdout:432: writing smithi08575397-7 from 658874 to 1430311 tid 1 2021-02-10T04:29:52.120 INFO:tasks.rados.rados.0.smithi085.stdout:432: writing smithi08575397-7 from 2083958 to 2687013 tid 2 2021-02-10T04:29:52.121 INFO:tasks.rados.rados.0.smithi085.stdout:433: copy_from oid 6 from oid 45 current snap is 40 2021-02-10T04:29:52.121 INFO:tasks.rados.rados.0.smithi085.stdout:428: finishing write tid 4 to smithi08575397-31 2021-02-10T04:29:52.121 INFO:tasks.rados.rados.0.smithi085.stdout:428: finishing write tid 5 to smithi08575397-31 2021-02-10T04:29:52.121 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 104 (ObjNum 180 snap 40 seq_num 180) dirty exists 2021-02-10T04:29:52.121 INFO:tasks.rados.rados.0.smithi085.stdout:428: left oid 31 (ObjNum 180 snap 40 seq_num 180) 2021-02-10T04:29:52.121 INFO:tasks.rados.rados.0.smithi085.stdout:429: expect (ObjNum 124 snap 23 seq_num 124) 2021-02-10T04:29:52.186 INFO:tasks.rados.rados.0.smithi085.stdout:431: finishing write tid 1 to smithi08575397-28 2021-02-10T04:29:52.187 INFO:tasks.rados.rados.0.smithi085.stdout:431: finishing write tid 2 to smithi08575397-28 2021-02-10T04:29:52.187 INFO:tasks.rados.rados.0.smithi085.stdout:431: finishing write tid 3 to smithi08575397-28 2021-02-10T04:29:52.187 INFO:tasks.rados.rados.0.smithi085.stdout:431: finishing write tid 4 to smithi08575397-28 2021-02-10T04:29:52.187 INFO:tasks.rados.rados.0.smithi085.stdout:431: finishing write tid 5 to smithi08575397-28 2021-02-10T04:29:52.187 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 39 (ObjNum 181 snap 40 seq_num 181) dirty exists 2021-02-10T04:29:52.187 INFO:tasks.rados.rados.0.smithi085.stdout:431: left oid 28 (ObjNum 181 snap 40 seq_num 181) 2021-02-10T04:29:52.187 INFO:tasks.rados.rados.0.smithi085.stdout:432: finishing write tid 1 to smithi08575397-7 2021-02-10T04:29:52.187 INFO:tasks.rados.rados.0.smithi085.stdout:428: done (10 left) 2021-02-10T04:29:52.187 INFO:tasks.rados.rados.0.smithi085.stdout:429: done (9 left) 2021-02-10T04:29:52.187 INFO:tasks.rados.rados.0.smithi085.stdout:431: done (8 left) 2021-02-10T04:29:52.188 INFO:tasks.rados.rados.0.smithi085.stdout:434: write oid 39 current snap is 40 2021-02-10T04:29:52.188 INFO:tasks.rados.rados.0.smithi085.stdout:434: seq_num 183 ranges {675681=563039,1858793=746713,3025937=49744} 2021-02-10T04:29:52.190 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:29:51 smithi085 conmon[45443]: debug 2021-02-10T04:29:51.975+0000 7f7140d73700 -1 osd.1 175 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:29:52.193 INFO:tasks.rados.rados.0.smithi085.stdout:434: writing smithi08575397-39 from 675681 to 1238720 tid 1 2021-02-10T04:29:52.200 INFO:tasks.rados.rados.0.smithi085.stdout:434: writing smithi08575397-39 from 1858793 to 2605506 tid 2 2021-02-10T04:29:52.203 INFO:tasks.rados.rados.0.smithi085.stdout:434: writing smithi08575397-39 from 3025937 to 3075681 tid 3 2021-02-10T04:29:52.203 INFO:tasks.rados.rados.0.smithi085.stdout:435: write oid 42 current snap is 40 2021-02-10T04:29:52.203 INFO:tasks.rados.rados.0.smithi085.stdout:435: seq_num 184 ranges {692488=638288,1917275=606724} 2021-02-10T04:29:52.210 INFO:tasks.rados.rados.0.smithi085.stdout:435: writing smithi08575397-42 from 692488 to 1330776 tid 1 2021-02-10T04:29:52.217 INFO:tasks.rados.rados.0.smithi085.stdout:435: writing smithi08575397-42 from 1917275 to 2523999 tid 2 2021-02-10T04:29:52.217 INFO:tasks.rados.rados.0.smithi085.stdout:436: copy_from oid 5 from oid 41 current snap is 40 2021-02-10T04:29:52.217 INFO:tasks.rados.rados.0.smithi085.stdout:435: finishing write tid 1 to smithi08575397-42 2021-02-10T04:29:52.217 INFO:tasks.rados.rados.0.smithi085.stdout:437: snap_remove snap 39 2021-02-10T04:29:52.490 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:29:52.501 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:29:52 smithi085 conmon[49106]: debug 2021-02-10T04:29:52.279+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:52.501 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:52 smithi085 conmon[29319]: cluster 2021-02-10T04:29:51.011816+0000 osd.1 (osd.1) 58 : 2021-02-10T04:29:52.501 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:52 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:29:52.501 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:52 smithi085 conmon[29319]: cluster 2021-02-10T04:29:51.011828+0000 osd.1 (osd.1) 59 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:29:52.501 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:52 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:29:52.501 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:52 smithi085 conmon[29319]: 02-10T04:29:51.431818+0000 mon.a (mon.0) 866 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:29:52.501 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:52 smithi085 conmon[29319]: cluster 2021-02-10T04:29:52.501 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:52 smithi085 conmon[29319]: 2021-02-10T04:29:52.010057 2021-02-10T04:29:52.501 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:52 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 382 : cluster [DBG] 2021-02-10T04:29:52.501 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:52 smithi085 conmon[29319]: pgmap v412: 168 pgs: 5 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 153 active+clean; 133 MiB data, 406 MiB used, 715 GiB / 715 GiB avail; 16 MiB/s rd, 3.6 MiB/s wr, 26 op/s 2021-02-10T04:29:52.502 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:52 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:29:52.502 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:52 smithi085 conmon[29319]: -02-10T04:29:52.067750+0000 2021-02-10T04:29:52.502 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:52 smithi085 conmon[29319]: mon.a (mon.0) 867 : cluster [DBG] 2021-02-10T04:29:52.502 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:52 smithi085 conmon[29319]: osdmap e176: 8 total, 8 up, 8 in 2021-02-10T04:29:52.502 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:52 smithi085 conmon[35961]: cluster 2021-02-10T04:29:52.502 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:52 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:29:52.502 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:52 smithi085 conmon[35961]: 29:51.011816+0000 osd.1 (osd.1) 58 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:29:52.502 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:52 smithi085 conmon[35961]: cluster 2021-02-10T04:29:52.502 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:52 smithi085 conmon[35961]: 2021-02-10T04:29:51.011828 2021-02-10T04:29:52.502 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:52 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 59 : cluster [WRN] 2021-02-10T04:29:52.503 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:52 smithi085 conmon[35961]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:29:52.503 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:52 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:29:52.503 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:52 smithi085 conmon[35961]: 29:51.431818+0000 mon.a (mon.0) 866 : audit 2021-02-10T04:29:52.503 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:52 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:29:52.503 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:52 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:29:52.503 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:52 smithi085 conmon[35961]: -10T04:29:52.010057+0000 mgr.y (mgr.14140) 382 : cluster [DBG] 2021-02-10T04:29:52.503 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:52 smithi085 conmon[35961]: pgmap v412: 168 pgs: 5 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 153 active+clean; 133 MiB data, 406 MiB used, 715 GiB / 715 GiB avail; 16 MiB/s rd, 3.6 MiB/s wr, 26 op/s 2021-02-10T04:29:52.503 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:52 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:29:52.503 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:52 smithi085 conmon[35961]: :29:52.067750+0000 mon.a (mon. 2021-02-10T04:29:52.503 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:52 smithi085 conmon[35961]: 0) 867 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2021-02-10T04:29:52.585 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:29:52.586 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2021-02-10T04:29:52.586 INFO:teuthology.orchestra.run.smithi164.stdout: "complaint_time": 30, 2021-02-10T04:29:52.586 INFO:teuthology.orchestra.run.smithi164.stdout: "num_blocked_ops": 0 2021-02-10T04:29:52.586 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:29:52.605 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:29:52.644 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:52.733 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:52 smithi164 conmon[29288]: cluster 2021-02-10T04:29:51.011816+0000 osd.1 (osd 2021-02-10T04:29:52.734 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:52 smithi164 conmon[29288]: .1) 58 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:29:52.734 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:52 smithi164 conmon[29288]: cluster 2021-02-10T04:29:51.011828+0000 osd.1 (osd.1) 59 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:29:52.734 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:52 smithi164 conmon[29288]: audit 2021-02-10T04:29:51.431818+0000 mon.a (mon.0) 866 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:29:52.734 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:52 smithi164 conmon[29288]: cluster 2021-02-10T04:29:52.010057+0000 mgr.y (mgr.14140) 382 : cluster [DBG] pgmap v412: 168 pgs: 5 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 153 active+clean; 133 MiB data, 406 MiB used, 715 GiB / 715 GiB avail; 16 MiB/s rd, 3.6 MiB/s wr, 26 op/s 2021-02-10T04:29:52.734 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:52 smithi164 conmon[29288]: cluster 2021-02-10T04:29:52.067750+0000 mon.a (mon.0) 867 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2021-02-10T04:29:52.875 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 2 1 2021-02-10T04:29:53.022 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:29:53.027 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:29:52 smithi164 conmon[33677]: debug 2021-02-10T04:29:52.772+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:53.054 INFO:tasks.rados.rados.0.smithi085.stdout:437: done (11 left) 2021-02-10T04:29:53.054 INFO:tasks.rados.rados.0.smithi085.stdout:438: snap_create 2021-02-10T04:29:53.054 INFO:tasks.rados.rados.0.smithi085.stdout:435: finishing write tid 2 to smithi08575397-42 2021-02-10T04:29:53.054 INFO:tasks.rados.rados.0.smithi085.stdout:435: finishing write tid 3 to smithi08575397-42 2021-02-10T04:29:53.054 INFO:tasks.rados.rados.0.smithi085.stdout:435: finishing write tid 4 to smithi08575397-42 2021-02-10T04:29:53.054 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 128 (ObjNum 184 snap 40 seq_num 184) dirty exists 2021-02-10T04:29:53.054 INFO:tasks.rados.rados.0.smithi085.stdout:435: left oid 42 (ObjNum 184 snap 40 seq_num 184) 2021-02-10T04:29:53.054 INFO:tasks.rados.rados.0.smithi085.stdout:432: finishing write tid 2 to smithi08575397-7 2021-02-10T04:29:53.055 INFO:tasks.rados.rados.0.smithi085.stdout:432: finishing write tid 3 to smithi08575397-7 2021-02-10T04:29:53.055 INFO:tasks.rados.rados.0.smithi085.stdout:432: finishing write tid 4 to smithi08575397-7 2021-02-10T04:29:53.055 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 77 (ObjNum 182 snap 40 seq_num 182) dirty exists 2021-02-10T04:29:53.055 INFO:tasks.rados.rados.0.smithi085.stdout:432: left oid 7 (ObjNum 182 snap 40 seq_num 182) 2021-02-10T04:29:53.055 INFO:tasks.rados.rados.0.smithi085.stdout:434: finishing write tid 1 to smithi08575397-39 2021-02-10T04:29:53.055 INFO:tasks.rados.rados.0.smithi085.stdout:434: finishing write tid 2 to smithi08575397-39 2021-02-10T04:29:53.055 INFO:tasks.rados.rados.0.smithi085.stdout:434: finishing write tid 3 to smithi08575397-39 2021-02-10T04:29:53.055 INFO:tasks.rados.rados.0.smithi085.stdout:434: finishing write tid 4 to smithi08575397-39 2021-02-10T04:29:53.055 INFO:tasks.rados.rados.0.smithi085.stdout:434: finishing write tid 5 to smithi08575397-39 2021-02-10T04:29:53.055 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 39 v 57 (ObjNum 183 snap 40 seq_num 183) dirty exists 2021-02-10T04:29:53.056 INFO:tasks.rados.rados.0.smithi085.stdout:434: left oid 39 (ObjNum 183 snap 40 seq_num 183) 2021-02-10T04:29:53.056 INFO:tasks.rados.rados.0.smithi085.stdout:436: finishing copy_from to smithi08575397-5 2021-02-10T04:29:53.056 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 179 (ObjNum 147 snap 30 seq_num 147) dirty exists 2021-02-10T04:29:53.056 INFO:tasks.rados.rados.0.smithi085.stdout:436: finishing copy_from racing read to smithi08575397-5 2021-02-10T04:29:53.056 INFO:tasks.rados.rados.0.smithi085.stdout:433: finishing copy_from to smithi08575397-6 2021-02-10T04:29:53.056 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 68 (ObjNum 96 snap 19 seq_num 96) dirty exists 2021-02-10T04:29:53.056 INFO:tasks.rados.rados.0.smithi085.stdout:433: finishing copy_from racing read to smithi08575397-6 2021-02-10T04:29:53.117 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:29:53 smithi085 conmon[45443]: debug 2021-02-10T04:29:53.020+0000 7f7140d73700 -1 osd.1 176 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:29:53.128 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:29:53.149 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:29:53.161 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:53.200 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.4 dump_historic_ops 2021-02-10T04:29:53.297 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:29:53.301 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:29:53 smithi164 conmon[37263]: debug 2021-02-10T04:29:53.237+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:53.301 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:29:53 smithi164 conmon[40892]: debug 2021-02-10T04:29:53.301 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:29:53 smithi164 conmon[40892]: 2021-02-10T04:29:53.231+0000 7f30354c5700 -1 osd.6 pg_epoch: 177 pg[7.7( v 175'189 (0'0,175'189] local-lis/les=93/94 n=6 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [6,0,3] r=0 lpr=93 crt=175'189 lcod 175'187 mlcod 175'187 active+clean+snaptrim trimq=[3d~2,42~2,46~1,4a~1,4f~1] ps=[2e~1,40~1]] removing snap head 2021-02-10T04:29:53.403 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:29:53.432 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:29:53 smithi085 conmon[49106]: debug 2021-02-10T04:29:53.432 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:29:53 smithi085 conmon[49106]: 2021-02-10T04:29:53.237+0000 7f7786614700 -1 osd.2 pg_epoch: 177 pg[7.5( v 177'77 (0'0,177'77] local-lis/les=130/131 n=3 ec=93/93 lis/c=130/130 les/c/f=131/131/0 sis=130) [2,7,0] r=0 lpr=130 luod=173'75 crt=177'77 lcod 173'73 mlcod 173'73 active+clean+snaptrim trimq=[3e~1,42~2,46~1,4a~1,4f~1] ps=[2e~1,3d~1,40~1]] removing snap head 2021-02-10T04:29:53.432 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:29:53 smithi085 conmon[49106]: 2021-02-10T04:29:53.433 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:53.507 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:29:53.523 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:29:53.628 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:29:53.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:53 smithi164 conmon[29288]: cluster 2021-02-10T04:29:51.975713+0000 osd.1 (osd.1 2021-02-10T04:29:53.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:53 smithi164 conmon[29288]: ) 60 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:29:53.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:53 smithi164 conmon[29288]: cluster 2021-02-10T04:29:51.975721+0000 osd.1 (osd.1) 61 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:29:53.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:53 smithi164 conmon[29288]: cluster 2021-02-10T04:29:53.051035+0000 mon.a (mon.0) 868 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2021-02-10T04:29:53.643 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:29:53 smithi164 conmon[40892]: debug 2021-02-10T04:29:53.499+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:53.662 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:53.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:53 smithi085 conmon[29319]: cluster 2021-02-10T04:29:51.975713+0000 osd.1 ( 2021-02-10T04:29:53.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:53 smithi085 conmon[29319]: osd.1) 60 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:29:53.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:53 smithi085 conmon[29319]: cluster 2021-02-10T04:29:51.975721+0000 osd.1 (osd.1) 61 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:29:53.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:53 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:29:53.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:53 smithi085 conmon[29319]: 02-10T04:29:53.051035+0000 mon.a (mon.0 2021-02-10T04:29:53.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:53 smithi085 conmon[29319]: ) 868 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2021-02-10T04:29:53.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:53 smithi085 conmon[35961]: cluster 2021-02-10T04:29:53.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:53 smithi085 conmon[35961]: 2021-02 2021-02-10T04:29:53.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:53 smithi085 conmon[35961]: -10T04:29:51.975713+0000 osd.1 (osd.1) 60 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:29:53.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:53 smithi085 conmon[35961]: cluster 2021-02-10T04:29:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:53 smithi085 conmon[35961]: 2021-02-10T04:29:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:53 smithi085 conmon[35961]: 2021 2021-02-10T04:29:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:53 smithi085 conmon[35961]: - 2021-02-10T04:29:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:53 smithi085 conmon[35961]: 02-10T04:29 2021-02-10T04:29:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:53 smithi085 conmon[35961]: :51.975721+0000 osd.1 (osd.1) 61 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:29:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:53 smithi085 conmon[35961]: 2021-02-10T04:29:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:53 smithi085 conmon[35961]: cluster 2021-02-10T04:29: 2021-02-10T04:29:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:53 smithi085 conmon[35961]: 53.051035+0000 mon.a (mon.0) 868 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2021-02-10T04:29:53.773 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:29:53.878 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:29:53.910 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:54.023 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:29:54.025 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:29:53 smithi085 conmon[45443]: debug 2021-02-10T04:29:53.739+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:54.025 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:29:53 smithi085 conmon[45443]: debug 2021-02-10T04:29:53.981+0000 7f7140d73700 -1 osd.1 177 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:29:54.028 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:29:53 smithi164 conmon[33677]: debug 2021-02-10T04:29:54.028 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:29:53 smithi164 conmon[33677]: 2021-02-10T04:29:53.992+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:54.029 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:29:53 smithi164 conmon[33677]: 2021-02-10T04:29:54.055 INFO:tasks.rados.rados.0.smithi085.stdout:432: done (11 left) 2021-02-10T04:29:54.055 INFO:tasks.rados.rados.0.smithi085.stdout:433: done (10 left) 2021-02-10T04:29:54.055 INFO:tasks.rados.rados.0.smithi085.stdout:434: done (9 left) 2021-02-10T04:29:54.056 INFO:tasks.rados.rados.0.smithi085.stdout:435: done (8 left) 2021-02-10T04:29:54.056 INFO:tasks.rados.rados.0.smithi085.stdout:436: done (7 left) 2021-02-10T04:29:54.056 INFO:tasks.rados.rados.0.smithi085.stdout:438: done (6 left) 2021-02-10T04:29:54.056 INFO:tasks.rados.rados.0.smithi085.stdout:439: read oid 5 snap -1 2021-02-10T04:29:54.056 INFO:tasks.rados.rados.0.smithi085.stdout:439: expect (ObjNum 147 snap 30 seq_num 147) 2021-02-10T04:29:54.056 INFO:tasks.rados.rados.0.smithi085.stdout:440: rollback oid 43 current snap is 41 2021-02-10T04:29:54.056 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 43 to 40 2021-02-10T04:29:54.056 INFO:tasks.rados.rados.0.smithi085.stdout:441: delete oid 38 current snap is 41 2021-02-10T04:29:54.060 INFO:tasks.rados.rados.0.smithi085.stdout:440: finishing rollback tid 1 to smithi08575397-43 2021-02-10T04:29:54.060 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 0 (ObjNum 1 snap 32576 seq_num 82371600) dirty dne 2021-02-10T04:29:54.060 INFO:tasks.rados.rados.0.smithi085.stdout:440: done (8 left) 2021-02-10T04:29:54.060 INFO:tasks.rados.rados.0.smithi085.stdout:441: done (7 left) 2021-02-10T04:29:54.060 INFO:tasks.rados.rados.0.smithi085.stdout:442: snap_remove snap 40 2021-02-10T04:29:54.061 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is set 2021-02-10T04:29:54.130 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:29:54.158 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:54.440 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[45443]: debug 2021-02-10T04:29:54.220+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:54.548 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:29:54.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:54 smithi164 conmon[29288]: cluster 2021-02-10T04:29:53.020826+0000 2021-02-10T04:29:54.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:54 smithi164 conmon[29288]: osd.1 (osd.1) 62 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:29:54.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:54 smithi164 conmon[29288]: cluster 2021-02-10T04:29:54.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:54 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:29:54.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:54 smithi164 conmon[29288]: 29:53.020837+0000 osd.1 (osd.1) 63 : cluster [WRN] 2021-02-10T04:29:54.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:54 smithi164 conmon[29288]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:29:54.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:54 smithi164 conmon[29288]: audit 2021-02-10T04:29:54.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:54 smithi164 conmon[29288]: 2021-02-10T04:29:53.593779+0000 mon.c 2021-02-10T04:29:54.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:54 smithi164 conmon[29288]: (mon.1) 45 : audit [INF] from='client.? 172.21.15.85:0/532738541' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:29:54.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:54 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:29:54.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:54 smithi164 conmon[29288]: 02-10T04:29:53.594110+0000 mon.a ( 2021-02-10T04:29:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:54 smithi164 conmon[29288]: mon.0) 869 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:29:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:54 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:29:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:54 smithi164 conmon[29288]: 02-10T04:29:54.010536+0000 mgr.y (mgr.14140) 383 : cluster 2021-02-10T04:29:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:54 smithi164 conmon[29288]: [DBG] pgmap v415: 168 pgs: 5 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 153 active+clean; 136 MiB data, 406 MiB used, 715 GiB / 715 GiB avail; 8.7 MiB/s rd, 3.0 MiB/s wr, 18 op/s 2021-02-10T04:29:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:54 smithi164 conmon[29288]: cluster 2021-02-10T04:29:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:54 smithi164 conmon[29288]: 2021-02-10T04:29:54.045719+0000 mon.a (mon.0) 2021-02-10T04:29:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:54 smithi164 conmon[29288]: 870 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:29:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:54 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:29:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:54 smithi164 conmon[29288]: 04:29:54.055389+0000 mon.a (mon.0) 871 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:29:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:54 smithi164 conmon[29288]: cluster 2021-02-10T04:29:54.055458+0000 mon.a (mon 2021-02-10T04:29:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:54 smithi164 conmon[29288]: .0) 872 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2021-02-10T04:29:54.921 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:29:54.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[29319]: cluster 2021-02-10T04:29:53.020826+0000 osd.1 (osd.1) 62 : 2021-02-10T04:29:54.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:29:54.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[29319]: cluster 2021-02-10T04:29:53.020837+0000 osd.1 (osd.1) 63 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:29:54.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[29319]: audit 2021-02-10T04:29:53.593779+0000 mon.c (mon.1) 45 : audit [INF] from='client.? 172.21.15.85:0/532738541' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:29:54.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[29319]: audit 2021-02-10T04:29:53.594110+0000 mon.a (mon.0) 869 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:29:54.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[29319]: cluster 2021-02-10T04:29:54.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[29319]: 2021-02-10T04:29:54.010536+0000 mgr.y (mgr.14140) 383 : cluster [DBG] pgmap v415: 168 pgs: 5 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 153 active+clean; 136 MiB data, 406 MiB used, 715 GiB / 715 GiB avail; 8.7 MiB/s rd, 3.0 MiB/s wr, 18 op/s 2021-02-10T04:29:54.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[29319]: cluster 2021-02-10T04:29:54 2021-02-10T04:29:54.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[29319]: .045719+0000 mon.a ( 2021-02-10T04:29:54.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[29319]: mon.0) 870 : cluster [WRN] 2021-02-10T04:29:54.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[29319]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:29:54.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:29:54.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[29319]: 02-10T04:29:54.055389+0000 mon.a (mon.0) 871 : audit [INF] 2021-02-10T04:29:54.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[29319]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:29:54.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[29319]: cluster 2021-02-10T04:29:54.055458+0000 mon.a (mon 2021-02-10T04:29:54.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[29319]: .0) 872 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2021-02-10T04:29:54.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[35961]: cluster 2021-02-10T04:29:54.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:29:54.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[35961]: 29:53.020826+0000 osd.1 (osd.1) 62 : cluster [WRN] 2021-02-10T04:29:54.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[35961]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:29:54.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:29:54.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[35961]: 29:53.020837+0000 osd.1 (osd.1) 63 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:29:54.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:29:54.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[35961]: 04:29:53.593779+0000 mon.c (mon.1) 2021-02-10T04:29:54.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[35961]: 45 : audit [INF] from='client.? 172.21.15.85:0/532738541' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:29:54.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:29:54.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[35961]: 02-10T04:29:53.594110+0000 mon.a (mon. 2021-02-10T04:29:54.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[35961]: 0) 869 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:29:54.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[35961]: cluster 2021-02-10T04:29:54.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[35961]: 2021-02-10T04:29:54.010536+0000 2021-02-10T04:29:54.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[35961]: mgr.y (mgr.14140) 383 : cluster [DBG] pgmap v415: 168 pgs: 5 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 153 active+clean; 136 MiB data, 406 MiB used, 715 GiB / 715 GiB avail; 8.7 MiB/s rd, 3.0 MiB/s wr, 18 op/s 2021-02-10T04:29:54.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:29:54.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[35961]: -02-10T04:29:54.045719+0000 2021-02-10T04:29:54.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[35961]: mon.a (mon.0) 870 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:29:54.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:29:54.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[35961]: 02-10T04:29:54.055389+0000 mon.a 2021-02-10T04:29:54.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[35961]: (mon.0) 871 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:29:54.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:29:54.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[35961]: 02-10T04:29:54.055458+0000 2021-02-10T04:29:54.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:54 smithi085 conmon[35961]: mon.a (mon.0) 872 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2021-02-10T04:29:55.059 INFO:tasks.rados.rados.0.smithi085.stdout:442: done (7 left) 2021-02-10T04:29:55.060 INFO:tasks.rados.rados.0.smithi085.stdout:443: rollback oid 43 current snap is 41 2021-02-10T04:29:55.060 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 43 to 38 2021-02-10T04:29:55.060 INFO:tasks.rados.rados.0.smithi085.stdout:444: write (excl) oid 40 current snap is 41 2021-02-10T04:29:55.060 INFO:tasks.rados.rados.0.smithi085.stdout:444: seq_num 186 ranges {726102=788786,1917886=610393,3108206=17896} 2021-02-10T04:29:55.064 INFO:tasks.rados.rados.0.smithi085.stdout:444: writing smithi08575397-40 from 726102 to 1514888 tid 1 2021-02-10T04:29:55.070 INFO:tasks.rados.rados.0.smithi085.stdout:444: writing smithi08575397-40 from 1917886 to 2528279 tid 2 2021-02-10T04:29:55.074 INFO:tasks.rados.rados.0.smithi085.stdout:444: writing smithi08575397-40 from 3108206 to 3126102 tid 3 2021-02-10T04:29:55.081 INFO:tasks.rados.rados.0.smithi085.stdout:439: expect (ObjNum 147 snap 30 seq_num 147) 2021-02-10T04:29:55.195 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:29:55.244 INFO:tasks.rados.rados.0.smithi085.stdout:439: done (8 left) 2021-02-10T04:29:55.244 INFO:tasks.rados.rados.0.smithi085.stdout:445: delete oid 30 current snap is 41 2021-02-10T04:29:55.244 INFO:tasks.rados.rados.0.smithi085.stdout:443: finishing rollback tid 1 to smithi08575397-43 2021-02-10T04:29:55.244 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-02-10T04:29:55.244 INFO:tasks.rados.rados.0.smithi085.stdout:444: finishing write tid 1 to smithi08575397-40 2021-02-10T04:29:55.244 INFO:tasks.rados.rados.0.smithi085.stdout:444: finishing write tid 2 to smithi08575397-40 2021-02-10T04:29:55.244 INFO:tasks.rados.rados.0.smithi085.stdout:444: finishing write tid 3 to smithi08575397-40 2021-02-10T04:29:55.244 INFO:tasks.rados.rados.0.smithi085.stdout:444: finishing write tid 4 to smithi08575397-40 2021-02-10T04:29:55.244 INFO:tasks.rados.rados.0.smithi085.stdout:444: finishing write tid 5 to smithi08575397-40 2021-02-10T04:29:55.245 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 196 (ObjNum 186 snap 41 seq_num 186) dirty exists 2021-02-10T04:29:55.245 INFO:tasks.rados.rados.0.smithi085.stdout:444: left oid 40 (ObjNum 186 snap 41 seq_num 186) 2021-02-10T04:29:55.246 INFO:tasks.rados.rados.0.smithi085.stdout:443: done (8 left) 2021-02-10T04:29:55.247 INFO:tasks.rados.rados.0.smithi085.stdout:444: done (7 left) 2021-02-10T04:29:55.247 INFO:tasks.rados.rados.0.smithi085.stdout:445: done (6 left) 2021-02-10T04:29:55.247 INFO:tasks.rados.rados.0.smithi085.stdout:446: copy_from oid 44 from oid 42 current snap is 41 2021-02-10T04:29:55.247 INFO:tasks.rados.rados.0.smithi085.stdout:447: read oid 32 snap 38 2021-02-10T04:29:55.247 INFO:tasks.rados.rados.0.smithi085.stdout:447: expect (ObjNum 31 snap 0 seq_num 31) 2021-02-10T04:29:55.247 INFO:tasks.rados.rados.0.smithi085.stdout:448: snap_create 2021-02-10T04:29:55.251 INFO:tasks.rados.rados.0.smithi085.stdout:447: expect (ObjNum 31 snap 0 seq_num 31) 2021-02-10T04:29:55.335 INFO:tasks.rados.rados.0.smithi085.stdout:446: finishing copy_from to smithi08575397-44 2021-02-10T04:29:55.335 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 128 (ObjNum 184 snap 40 seq_num 184) dirty exists 2021-02-10T04:29:55.335 INFO:tasks.rados.rados.0.smithi085.stdout:446: finishing copy_from racing read to smithi08575397-44 2021-02-10T04:29:55.440 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:29:55 smithi085 conmon[45443]: debug 2021-02-10T04:29:55.008+0000 7f7140d73700 -1 osd.1 178 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:29:55.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:55 smithi085 conmon[29319]: cluster 2021-02-10T04:29:53.981737+0000 osd.1 (osd. 2021-02-10T04:29:55.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:55 smithi085 conmon[29319]: 1) 64 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:29:55.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:55 smithi085 conmon[29319]: cluster 2021-02-10T04:29:53.981743+0000 osd.1 (osd.1) 65 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:29:55.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:55 smithi085 conmon[29319]: cluster 2021-02-10T04:29:54.435689 2021-02-10T04:29:55.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:55 smithi085 conmon[29319]: +0000 mon.a (mon.0) 873 : cluster 2021-02-10T04:29:55.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:55 smithi085 conmon[29319]: [WRN] Health check failed: 2 slow ops, oldest one blocked for 42 sec, osd.1 has slow ops (SLOW_OPS) 2021-02-10T04:29:55.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:55 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:29:55.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:55 smithi085 conmon[29319]: -02-10T04:29:55.054207+0000 2021-02-10T04:29:55.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:55 smithi085 conmon[29319]: mon.a (mon.0) 874 : cluster 2021-02-10T04:29:55.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:55 smithi085 conmon[29319]: [DBG] osdmap e179: 8 total, 8 up, 8 in 2021-02-10T04:29:55.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:55 smithi085 conmon[35961]: cluster 2021-02-10T04:29:53.981737+0000 2021-02-10T04:29:55.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:55 smithi085 conmon[35961]: osd.1 (osd.1) 64 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:29:55.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:55 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:29:55.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:55 smithi085 conmon[35961]: -02-10T04:29:53.981743+0000 osd.1 (osd.1) 65 : cluster 2021-02-10T04:29:55.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:55 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:29:55.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:55 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:29:55.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:55 smithi085 conmon[35961]: 29:54.435689+0000 mon.a (mon.0) 873 : cluster 2021-02-10T04:29:55.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:55 smithi085 conmon[35961]: [WRN] Health check failed: 2 slow ops, oldest one blocked for 42 sec, osd.1 has slow ops (SLOW_OPS) 2021-02-10T04:29:55.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:55 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:29:55.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:55 smithi085 conmon[35961]: T04:29:55.054207+0000 mon.a (mon. 2021-02-10T04:29:55.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:55 smithi085 conmon[35961]: 0) 874 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2021-02-10T04:29:55.804 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:29:55.837 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:55.850 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:55 smithi164 conmon[29288]: cluster 2021-02-10T04:29:53.981737+0000 osd.1 (osd.1) 64 : cluster 2021-02-10T04:29:55.850 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:55 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:29:55.850 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:55 smithi164 conmon[29288]: cluster 2021-02-10T04:29:53.981743+0000 osd.1 (osd.1) 65 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:29:55.850 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:55 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:29:55.851 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:55 smithi164 conmon[29288]: T04:29:54.435689+0000 mon.a (mon.0) 873 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 42 sec, osd.1 has slow ops (SLOW_OPS) 2021-02-10T04:29:55.851 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:55 smithi164 conmon[29288]: cluster 2021-02-10T04:29:55 2021-02-10T04:29:55.851 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:55 smithi164 conmon[29288]: .054207+0000 mon.a (mon.0) 874 : cluster 2021-02-10T04:29:55.851 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:55 smithi164 conmon[29288]: [DBG] osdmap e179: 8 total, 8 up, 8 in 2021-02-10T04:29:56.055 INFO:tasks.rados.rados.0.smithi085.stdout:446: done (8 left) 2021-02-10T04:29:56.055 INFO:tasks.rados.rados.0.smithi085.stdout:447: done (7 left) 2021-02-10T04:29:56.055 INFO:tasks.rados.rados.0.smithi085.stdout:448: done (6 left) 2021-02-10T04:29:56.055 INFO:tasks.rados.rados.0.smithi085.stdout:449: snap_remove snap 41 2021-02-10T04:29:56.363 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:29:56.363 INFO:teuthology.orchestra.run.smithi164.stdout: "size": 20, 2021-02-10T04:29:56.363 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 600, 2021-02-10T04:29:56.363 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [ 2021-02-10T04:29:56.363 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.363 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.24373.0:408 4.1 4:805d27d8:::data_log.40:head [call fifo.create_meta in=15b] snapc 0=[] ondisk+write+known_if_redirected e58)", 2021-02-10T04:29:56.363 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:25:51.290485+0000", 2021-02-10T04:29:56.363 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 245.06961099, 2021-02-10T04:29:56.364 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0059660349999999997, 2021-02-10T04:29:56.364 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:29:56.364 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:29:56.364 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:29:56.364 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.24373", 2021-02-10T04:29:56.364 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:29:56.364 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 408 2021-02-10T04:29:56.364 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.364 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:29:56.364 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.364 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:29:56.364 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:51.290485+0000", 2021-02-10T04:29:56.364 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:29:56.365 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.365 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.365 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:29:56.365 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:51.290485+0000", 2021-02-10T04:29:56.365 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4249000000000001e-05 2021-02-10T04:29:56.365 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.365 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.365 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:29:56.365 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:51.290509+0000", 2021-02-10T04:29:56.365 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.5740000000000003e-06 2021-02-10T04:29:56.365 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.365 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.365 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:29:56.366 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:51.290515+0000", 2021-02-10T04:29:56.366 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.6240000000000002e-06 2021-02-10T04:29:56.366 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.366 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.366 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:29:56.366 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:51.290522+0000", 2021-02-10T04:29:56.366 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.0249999999999996e-06 2021-02-10T04:29:56.366 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.366 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.366 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:29:56.366 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:51.290526+0000", 2021-02-10T04:29:56.366 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.2156000000000005e-05 2021-02-10T04:29:56.366 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.367 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.367 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:56.367 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:51.290599+0000", 2021-02-10T04:29:56.367 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.2886999999999994e-05 2021-02-10T04:29:56.367 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.367 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.367 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:29:56.367 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:51.290681+0000", 2021-02-10T04:29:56.367 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00013886200000000001 2021-02-10T04:29:56.367 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.367 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.367 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 5,6", 2021-02-10T04:29:56.368 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:51.290820+0000", 2021-02-10T04:29:56.368 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0052736010000000002 2021-02-10T04:29:56.368 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.368 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.368 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:29:56.368 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:51.296094+0000", 2021-02-10T04:29:56.368 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000193747 2021-02-10T04:29:56.368 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.368 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.369 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:29:56.369 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:51.296288+0000", 2021-02-10T04:29:56.369 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.0723999999999996e-05 2021-02-10T04:29:56.369 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.369 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.369 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:29:56.369 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:51.296358+0000", 2021-02-10T04:29:56.369 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.5977000000000003e-05 2021-02-10T04:29:56.369 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.369 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.370 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:29:56.370 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:51.296414+0000", 2021-02-10T04:29:56.370 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.6609000000000001e-05 2021-02-10T04:29:56.370 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.370 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.370 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:29:56.370 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:51.296451+0000", 2021-02-10T04:29:56.371 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0059660349999999997 2021-02-10T04:29:56.371 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.371 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:29:56.371 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.371 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.371 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.371 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.24382.0:3 4.18 4:1e85421f:gc::gc.1:head [create,call version.check_conds in=18b,call rgw_gc.rgw_gc_queue_init in=23b,call version.set in=10b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2021-02-10T04:29:56.371 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:25:55.048764+0000", 2021-02-10T04:29:56.372 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 241.311331886, 2021-02-10T04:29:56.372 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0080698590000000004, 2021-02-10T04:29:56.372 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:29:56.372 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:29:56.372 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:29:56.372 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.24382", 2021-02-10T04:29:56.373 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/3915233377", 2021-02-10T04:29:56.373 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 3 2021-02-10T04:29:56.373 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.373 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:29:56.373 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.373 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:29:56.373 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.048764+0000", 2021-02-10T04:29:56.374 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:29:56.374 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.374 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.374 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:29:56.374 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.048764+0000", 2021-02-10T04:29:56.374 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.4209999999999999e-05 2021-02-10T04:29:56.374 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.374 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.374 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:29:56.374 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.048808+0000", 2021-02-10T04:29:56.374 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0336000000000001e-05 2021-02-10T04:29:56.374 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.375 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.375 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:29:56.375 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.048819+0000", 2021-02-10T04:29:56.375 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6092000000000001e-05 2021-02-10T04:29:56.375 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.375 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.375 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:29:56.375 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.048835+0000", 2021-02-10T04:29:56.375 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7380000000000006e-06 2021-02-10T04:29:56.376 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.376 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.376 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:29:56.376 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.048844+0000", 2021-02-10T04:29:56.376 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.1111000000000005e-05 2021-02-10T04:29:56.376 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.376 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.376 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:56.376 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.048905+0000", 2021-02-10T04:29:56.376 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9629999999999997e-06 2021-02-10T04:29:56.376 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.376 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.376 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2021-02-10T04:29:56.376 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.048910+0000", 2021-02-10T04:29:56.377 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0048408469999999997 2021-02-10T04:29:56.377 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.377 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.377 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:56.377 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.053751+0000", 2021-02-10T04:29:56.377 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00014035100000000001 2021-02-10T04:29:56.377 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.377 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.377 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:29:56.377 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.053892+0000", 2021-02-10T04:29:56.377 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00031588200000000002 2021-02-10T04:29:56.377 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.377 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.378 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 1,6", 2021-02-10T04:29:56.378 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.054208+0000", 2021-02-10T04:29:56.378 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00054399299999999999 2021-02-10T04:29:56.378 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.378 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.378 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:29:56.378 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.054752+0000", 2021-02-10T04:29:56.378 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00056921399999999996 2021-02-10T04:29:56.378 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.378 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.378 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:29:56.378 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.055321+0000", 2021-02-10T04:29:56.378 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001412068 2021-02-10T04:29:56.379 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.379 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.379 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:29:56.379 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.056733+0000", 2021-02-10T04:29:56.379 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.4084999999999994e-05 2021-02-10T04:29:56.379 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.379 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.379 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:29:56.379 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.056797+0000", 2021-02-10T04:29:56.379 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.6968999999999998e-05 2021-02-10T04:29:56.379 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.379 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.379 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:29:56.380 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.056834+0000", 2021-02-10T04:29:56.380 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0080698590000000004 2021-02-10T04:29:56.380 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.380 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:29:56.380 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.380 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.380 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.380 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.24382.0:722 4.1a 4:5f2e3d26:lc::lc.18:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2021-02-10T04:29:56.380 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:25:55.298373+0000", 2021-02-10T04:29:56.380 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 241.06172256299999, 2021-02-10T04:29:56.380 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0070891820000000003, 2021-02-10T04:29:56.380 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:29:56.381 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:29:56.381 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:29:56.381 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.24382", 2021-02-10T04:29:56.381 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/3915233377", 2021-02-10T04:29:56.381 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 722 2021-02-10T04:29:56.381 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.381 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:29:56.381 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.381 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:29:56.381 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.298373+0000", 2021-02-10T04:29:56.381 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:29:56.381 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.381 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.382 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:29:56.382 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.298373+0000", 2021-02-10T04:29:56.382 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4290999999999998e-05 2021-02-10T04:29:56.382 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.382 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.382 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:29:56.382 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.298398+0000", 2021-02-10T04:29:56.382 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.853e-06 2021-02-10T04:29:56.382 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.382 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.382 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:29:56.382 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.298402+0000", 2021-02-10T04:29:56.382 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2519e-05 2021-02-10T04:29:56.383 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.383 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.383 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:29:56.383 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.298415+0000", 2021-02-10T04:29:56.383 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.4790000000000001e-06 2021-02-10T04:29:56.383 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.383 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.383 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:29:56.384 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.298421+0000", 2021-02-10T04:29:56.384 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.8016000000000002e-05 2021-02-10T04:29:56.384 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.384 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.384 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:56.384 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.298469+0000", 2021-02-10T04:29:56.384 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000103084 2021-02-10T04:29:56.384 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.384 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.384 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:29:56.384 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.298573+0000", 2021-02-10T04:29:56.384 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00020350300000000001 2021-02-10T04:29:56.384 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.385 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.385 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 0,3", 2021-02-10T04:29:56.385 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.298776+0000", 2021-02-10T04:29:56.385 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0054843940000000001 2021-02-10T04:29:56.385 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.385 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.385 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:29:56.385 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.304260+0000", 2021-02-10T04:29:56.385 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0455999999999999e-05 2021-02-10T04:29:56.385 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.385 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.385 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:29:56.385 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.304281+0000", 2021-02-10T04:29:56.385 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001067903 2021-02-10T04:29:56.386 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.386 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.386 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:29:56.386 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.305349+0000", 2021-02-10T04:29:56.386 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.7640999999999998e-05 2021-02-10T04:29:56.386 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.386 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.386 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:29:56.386 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.305416+0000", 2021-02-10T04:29:56.386 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.6042999999999999e-05 2021-02-10T04:29:56.386 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.386 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.386 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:29:56.387 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.305462+0000", 2021-02-10T04:29:56.387 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0070891820000000003 2021-02-10T04:29:56.387 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.387 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:29:56.387 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.387 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.387 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.387 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop_reply(client.24382.0:722 4.1a e62/55 ondisk, result = 0)", 2021-02-10T04:29:56.387 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:25:55.300004+0000", 2021-02-10T04:29:56.387 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 241.06009186899999, 2021-02-10T04:29:56.387 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0042597809999999998, 2021-02-10T04:29:56.387 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:29:56.387 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:29:56.388 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:29:56.388 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.388 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:29:56.388 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.300004+0000", 2021-02-10T04:29:56.388 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:29:56.388 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.388 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.388 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:29:56.388 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.300004+0000", 2021-02-10T04:29:56.388 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1612000000000002e-05 2021-02-10T04:29:56.388 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.388 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.388 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:29:56.388 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.300036+0000", 2021-02-10T04:29:56.389 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.1085000000000003e-05 2021-02-10T04:29:56.389 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.389 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.389 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:29:56.389 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.300087+0000", 2021-02-10T04:29:56.389 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2135999999999999e-05 2021-02-10T04:29:56.389 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.389 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.389 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:29:56.389 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.300109+0000", 2021-02-10T04:29:56.389 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6654999999999999e-05 2021-02-10T04:29:56.389 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.389 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.390 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:29:56.390 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.300125+0000", 2021-02-10T04:29:56.390 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0041156400000000003 2021-02-10T04:29:56.390 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.390 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.390 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:56.390 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.304241+0000", 2021-02-10T04:29:56.390 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7850000000000003e-06 2021-02-10T04:29:56.390 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.390 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.390 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:29:56.390 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.304251+0000", 2021-02-10T04:29:56.391 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2867999999999999e-05 2021-02-10T04:29:56.391 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.391 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.391 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:29:56.391 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.304264+0000", 2021-02-10T04:29:56.391 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0042597809999999998 2021-02-10T04:29:56.391 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.391 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:29:56.391 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.391 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.392 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.392 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.24382.0:861 4.1 4:80d23011:::obj_delete_at_hint.0000000085:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2021-02-10T04:29:56.392 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:25:55.441486+0000", 2021-02-10T04:29:56.392 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 240.91860967599999, 2021-02-10T04:29:56.392 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0045208790000000002, 2021-02-10T04:29:56.392 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:29:56.392 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:29:56.392 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:29:56.392 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.24382", 2021-02-10T04:29:56.392 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/3915233377", 2021-02-10T04:29:56.392 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 861 2021-02-10T04:29:56.392 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.392 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:29:56.392 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.393 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:29:56.393 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.441486+0000", 2021-02-10T04:29:56.393 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:29:56.393 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.393 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.393 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:29:56.393 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.441486+0000", 2021-02-10T04:29:56.393 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6567000000000002e-05 2021-02-10T04:29:56.393 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.394 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.394 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:29:56.394 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.441543+0000", 2021-02-10T04:29:56.394 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.7770000000000008e-06 2021-02-10T04:29:56.394 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.395 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.395 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:29:56.395 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.441552+0000", 2021-02-10T04:29:56.395 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6990000000000002e-05 2021-02-10T04:29:56.395 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.395 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.395 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:29:56.395 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.441569+0000", 2021-02-10T04:29:56.396 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.0690000000000002e-06 2021-02-10T04:29:56.396 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.396 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.396 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:29:56.396 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.441577+0000", 2021-02-10T04:29:56.396 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.6322000000000002e-05 2021-02-10T04:29:56.396 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.396 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.396 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:56.396 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.441613+0000", 2021-02-10T04:29:56.396 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.5986999999999998e-05 2021-02-10T04:29:56.397 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.397 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.397 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:29:56.397 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.441669+0000", 2021-02-10T04:29:56.397 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000114152 2021-02-10T04:29:56.397 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.397 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.397 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 5,6", 2021-02-10T04:29:56.397 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.441783+0000", 2021-02-10T04:29:56.397 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00030620400000000002 2021-02-10T04:29:56.397 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.397 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.398 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:29:56.398 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.442089+0000", 2021-02-10T04:29:56.398 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00199271 2021-02-10T04:29:56.398 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.398 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.398 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:29:56.398 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.444082+0000", 2021-02-10T04:29:56.398 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001850729 2021-02-10T04:29:56.398 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.398 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.398 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:29:56.398 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.445933+0000", 2021-02-10T04:29:56.398 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.7020000000000001e-05 2021-02-10T04:29:56.399 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.399 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.399 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:29:56.399 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.445970+0000", 2021-02-10T04:29:56.399 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.7351999999999997e-05 2021-02-10T04:29:56.399 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.399 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.399 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:29:56.399 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.446007+0000", 2021-02-10T04:29:56.399 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0045208790000000002 2021-02-10T04:29:56.399 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.399 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:29:56.399 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.400 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.400 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.400 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:215 7.1 7:8132b738:::smithi08575397-33:head [write 490771~686315] snapc 3=[3,2] ondisk+write+known_if_redirected e100)", 2021-02-10T04:29:56.400 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:28:36.206470+0000", 2021-02-10T04:29:56.400 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 80.153625624, 2021-02-10T04:29:56.400 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0048899979999999996, 2021-02-10T04:29:56.400 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:29:56.400 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:29:56.400 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:29:56.400 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:29:56.400 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:29:56.400 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 215 2021-02-10T04:29:56.401 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.401 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:29:56.401 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.401 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:29:56.401 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:36.206470+0000", 2021-02-10T04:29:56.401 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:29:56.401 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.401 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.401 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:29:56.401 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:36.206470+0000", 2021-02-10T04:29:56.401 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0010350719999999999 2021-02-10T04:29:56.401 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.401 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.402 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:29:56.402 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:36.207505+0000", 2021-02-10T04:29:56.402 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0220999999999999e-05 2021-02-10T04:29:56.402 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.402 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.402 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:29:56.402 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:36.207516+0000", 2021-02-10T04:29:56.402 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4321999999999999e-05 2021-02-10T04:29:56.402 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.402 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.402 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:29:56.402 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:36.207530+0000", 2021-02-10T04:29:56.402 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.4209999999999998e-06 2021-02-10T04:29:56.403 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.403 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.403 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:29:56.403 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:36.207537+0000", 2021-02-10T04:29:56.403 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00011849999999999999 2021-02-10T04:29:56.403 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.403 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.403 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:56.403 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:36.207656+0000", 2021-02-10T04:29:56.403 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000101578 2021-02-10T04:29:56.403 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.403 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.403 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:29:56.404 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:36.207757+0000", 2021-02-10T04:29:56.404 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00018047899999999999 2021-02-10T04:29:56.404 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.404 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.404 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 3,6", 2021-02-10T04:29:56.404 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:36.207938+0000", 2021-02-10T04:29:56.404 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0020882710000000001 2021-02-10T04:29:56.404 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.404 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.404 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:29:56.404 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:36.210026+0000", 2021-02-10T04:29:56.404 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00057563999999999998 2021-02-10T04:29:56.404 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.405 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.405 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:29:56.405 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:36.210602+0000", 2021-02-10T04:29:56.405 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00058778599999999997 2021-02-10T04:29:56.405 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.405 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.405 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:29:56.405 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:36.211190+0000", 2021-02-10T04:29:56.405 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.8442999999999993e-05 2021-02-10T04:29:56.405 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.405 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.405 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:29:56.405 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:36.211278+0000", 2021-02-10T04:29:56.406 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.2264999999999999e-05 2021-02-10T04:29:56.406 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.406 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.406 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:29:56.406 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:36.211360+0000", 2021-02-10T04:29:56.406 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0048899979999999996 2021-02-10T04:29:56.406 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.406 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:29:56.406 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.406 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.406 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.406 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:232 7.1 7:8132b738:::smithi08575397-33:head [read 0~2890771,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e101)", 2021-02-10T04:29:56.407 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:28:37.203216+0000", 2021-02-10T04:29:56.407 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 79.156879634999996, 2021-02-10T04:29:56.407 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010334987, 2021-02-10T04:29:56.407 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:29:56.407 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:29:56.407 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:29:56.407 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:29:56.407 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:29:56.407 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 232 2021-02-10T04:29:56.407 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.407 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:29:56.407 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.407 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:29:56.407 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203216+0000", 2021-02-10T04:29:56.408 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:29:56.408 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.408 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.408 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:29:56.408 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203216+0000", 2021-02-10T04:29:56.408 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.4169999999999999e-05 2021-02-10T04:29:56.408 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.408 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.408 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:29:56.408 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203260+0000", 2021-02-10T04:29:56.408 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.064e-06 2021-02-10T04:29:56.408 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.408 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.409 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:29:56.409 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203269+0000", 2021-02-10T04:29:56.409 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5914999999999999e-05 2021-02-10T04:29:56.409 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.409 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.409 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:29:56.409 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203285+0000", 2021-02-10T04:29:56.409 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.4010000000000003e-06 2021-02-10T04:29:56.409 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.409 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.409 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:29:56.409 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203295+0000", 2021-02-10T04:29:56.409 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1817999999999995e-05 2021-02-10T04:29:56.410 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.410 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.410 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:56.410 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203377+0000", 2021-02-10T04:29:56.410 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.4599999999999992e-06 2021-02-10T04:29:56.410 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.410 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.410 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2021-02-10T04:29:56.410 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203386+0000", 2021-02-10T04:29:56.410 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0059418359999999998 2021-02-10T04:29:56.410 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.410 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.410 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:56.411 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.209328+0000", 2021-02-10T04:29:56.411 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000121338 2021-02-10T04:29:56.411 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.411 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.411 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:29:56.411 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.209449+0000", 2021-02-10T04:29:56.411 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0041019849999999998 2021-02-10T04:29:56.411 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.411 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.411 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:29:56.411 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.213551+0000", 2021-02-10T04:29:56.411 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010334987 2021-02-10T04:29:56.411 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.412 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:29:56.412 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.412 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.412 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.412 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:233 7.1 7:8132b738:::smithi08575397-33:head [read 0~2890771,checksum] snapc 0=[] ondisk+read+known_if_redirected e101)", 2021-02-10T04:29:56.412 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:28:37.203339+0000", 2021-02-10T04:29:56.412 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 79.156756646000005, 2021-02-10T04:29:56.412 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012015658, 2021-02-10T04:29:56.412 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:29:56.412 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:29:56.412 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:29:56.412 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:29:56.412 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:29:56.413 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 233 2021-02-10T04:29:56.413 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.413 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:29:56.413 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.413 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:29:56.413 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203339+0000", 2021-02-10T04:29:56.413 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:29:56.413 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.413 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.413 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:29:56.413 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203339+0000", 2021-02-10T04:29:56.413 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.3578999999999998e-05 2021-02-10T04:29:56.413 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.414 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.414 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:29:56.414 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203373+0000", 2021-02-10T04:29:56.414 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.9829999999999998e-06 2021-02-10T04:29:56.414 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.414 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.414 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:29:56.414 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203377+0000", 2021-02-10T04:29:56.414 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.499e-06 2021-02-10T04:29:56.414 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.414 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.414 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:29:56.415 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203386+0000", 2021-02-10T04:29:56.415 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.0490000000000004e-06 2021-02-10T04:29:56.415 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.415 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.415 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:29:56.415 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203394+0000", 2021-02-10T04:29:56.415 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000101165 2021-02-10T04:29:56.415 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.415 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.415 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:56.415 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203495+0000", 2021-02-10T04:29:56.415 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6439999999999997e-06 2021-02-10T04:29:56.415 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.415 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.416 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting_for_map not empty", 2021-02-10T04:29:56.416 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203501+0000", 2021-02-10T04:29:56.416 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010104657 2021-02-10T04:29:56.416 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.416 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.416 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:56.416 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.213606+0000", 2021-02-10T04:29:56.416 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00015320000000000001 2021-02-10T04:29:56.416 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.416 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.416 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:29:56.416 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.213759+0000", 2021-02-10T04:29:56.417 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0015958820000000001 2021-02-10T04:29:56.417 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.417 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.417 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:29:56.417 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.215355+0000", 2021-02-10T04:29:56.417 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012015658 2021-02-10T04:29:56.417 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.417 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:29:56.417 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.417 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.418 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.418 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:234 7.1 7:8132b738:::smithi08575397-33:head [sparse-read 0~1312313] snapc 0=[] ondisk+read+known_if_redirected e101)", 2021-02-10T04:29:56.418 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:28:37.203458+0000", 2021-02-10T04:29:56.418 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 79.156637704000005, 2021-02-10T04:29:56.418 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012303939, 2021-02-10T04:29:56.418 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:29:56.418 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:29:56.418 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:29:56.418 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:29:56.418 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:29:56.418 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 234 2021-02-10T04:29:56.418 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.418 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:29:56.419 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.419 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:29:56.419 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203458+0000", 2021-02-10T04:29:56.419 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:29:56.419 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.419 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.419 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:29:56.419 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203458+0000", 2021-02-10T04:29:56.419 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2258000000000001e-05 2021-02-10T04:29:56.419 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.419 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.419 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:29:56.419 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203500+0000", 2021-02-10T04:29:56.420 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.4709999999999998e-06 2021-02-10T04:29:56.420 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.420 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.420 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:29:56.420 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203506+0000", 2021-02-10T04:29:56.420 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0747e-05 2021-02-10T04:29:56.420 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.420 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.420 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:29:56.420 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203527+0000", 2021-02-10T04:29:56.420 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.7590000000000003e-06 2021-02-10T04:29:56.420 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.420 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.421 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:29:56.421 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203535+0000", 2021-02-10T04:29:56.421 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.5235999999999997e-05 2021-02-10T04:29:56.421 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.421 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.421 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:56.421 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203601+0000", 2021-02-10T04:29:56.421 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1539999999999996e-06 2021-02-10T04:29:56.421 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.421 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.421 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting_for_map not empty", 2021-02-10T04:29:56.422 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203608+0000", 2021-02-10T04:29:56.422 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011817237 2021-02-10T04:29:56.422 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.422 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.422 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:56.422 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.215425+0000", 2021-02-10T04:29:56.422 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00013556399999999999 2021-02-10T04:29:56.422 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.422 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.422 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:29:56.422 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.215561+0000", 2021-02-10T04:29:56.422 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00020151300000000001 2021-02-10T04:29:56.422 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.423 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.423 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:29:56.423 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.215762+0000", 2021-02-10T04:29:56.423 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012303939 2021-02-10T04:29:56.423 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.423 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:29:56.423 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.423 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.423 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.423 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:257 7.8 e104/93 7:10dac65a:::smithi08575397-39:head v 104'4, mlcod=98'2)", 2021-02-10T04:29:56.423 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:28:40.225588+0000", 2021-02-10T04:29:56.424 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 76.134508057999994, 2021-02-10T04:29:56.424 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0099257589999999993, 2021-02-10T04:29:56.424 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:29:56.424 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:29:56.424 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:29:56.424 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.424 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:29:56.424 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:40.225588+0000", 2021-02-10T04:29:56.424 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:29:56.424 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.425 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.425 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:29:56.425 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:40.225588+0000", 2021-02-10T04:29:56.425 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0075234569999999999 2021-02-10T04:29:56.425 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.425 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.425 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:29:56.425 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:40.233111+0000", 2021-02-10T04:29:56.425 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1969e-05 2021-02-10T04:29:56.425 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.425 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.425 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:29:56.425 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:40.233123+0000", 2021-02-10T04:29:56.426 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6528000000000002e-05 2021-02-10T04:29:56.426 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.426 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.426 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:29:56.426 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:40.233140+0000", 2021-02-10T04:29:56.426 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2013e-05 2021-02-10T04:29:56.426 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.426 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.426 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:29:56.426 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:40.233152+0000", 2021-02-10T04:29:56.426 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.4579e-05 2021-02-10T04:29:56.426 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.427 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.427 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:56.427 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:40.233206+0000", 2021-02-10T04:29:56.427 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7978999999999998e-05 2021-02-10T04:29:56.427 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.427 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.427 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:29:56.427 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:40.233224+0000", 2021-02-10T04:29:56.427 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0022181760000000001 2021-02-10T04:29:56.427 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.427 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.427 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:29:56.427 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:40.235443+0000", 2021-02-10T04:29:56.428 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1057999999999996e-05 2021-02-10T04:29:56.428 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.428 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.428 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:29:56.428 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:40.235514+0000", 2021-02-10T04:29:56.428 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0099257589999999993 2021-02-10T04:29:56.428 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.428 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:29:56.428 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.428 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.428 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.428 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:265 7.3 e105/93 7:c765b4cc:::smithi08575397-30:head v 105'14, mlcod=98'11)", 2021-02-10T04:29:56.429 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:28:41.238549+0000", 2021-02-10T04:29:56.429 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 75.121547042000003, 2021-02-10T04:29:56.429 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.009041561, 2021-02-10T04:29:56.429 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:29:56.429 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:29:56.429 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:29:56.429 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.429 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:29:56.429 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.238549+0000", 2021-02-10T04:29:56.429 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:29:56.429 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.429 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.429 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:29:56.429 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.238549+0000", 2021-02-10T04:29:56.430 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.003393877 2021-02-10T04:29:56.430 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.430 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.430 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:29:56.430 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.241943+0000", 2021-02-10T04:29:56.430 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2551999999999999e-05 2021-02-10T04:29:56.430 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.430 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.430 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:29:56.430 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.241955+0000", 2021-02-10T04:29:56.430 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7842999999999998e-05 2021-02-10T04:29:56.431 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.431 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.431 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:29:56.431 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.241973+0000", 2021-02-10T04:29:56.431 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0155e-05 2021-02-10T04:29:56.431 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.431 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.431 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:29:56.431 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.241983+0000", 2021-02-10T04:29:56.432 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.0531999999999994e-05 2021-02-10T04:29:56.432 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.432 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.432 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:56.432 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.242064+0000", 2021-02-10T04:29:56.432 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8025e-05 2021-02-10T04:29:56.432 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.432 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.432 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:29:56.432 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.242092+0000", 2021-02-10T04:29:56.432 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0054341579999999997 2021-02-10T04:29:56.432 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.432 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.433 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:29:56.433 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.247526+0000", 2021-02-10T04:29:56.433 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.4418999999999994e-05 2021-02-10T04:29:56.433 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.433 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.433 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:29:56.433 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.247590+0000", 2021-02-10T04:29:56.433 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.009041561 2021-02-10T04:29:56.433 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.433 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:29:56.433 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.433 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.433 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.434 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:330 7.1 7:8e9d3756:::smithi08575397-17:head [read 0~2268912,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e106)", 2021-02-10T04:29:56.434 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:28:42.246484+0000", 2021-02-10T04:29:56.434 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 74.113611423999998, 2021-02-10T04:29:56.434 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0048228409999999996, 2021-02-10T04:29:56.434 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:29:56.434 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:29:56.434 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:29:56.434 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:29:56.434 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:29:56.434 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 330 2021-02-10T04:29:56.434 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.434 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:29:56.435 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.435 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:29:56.435 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246484+0000", 2021-02-10T04:29:56.435 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:29:56.435 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.435 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.435 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:29:56.435 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246484+0000", 2021-02-10T04:29:56.435 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.5988000000000005e-05 2021-02-10T04:29:56.435 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.435 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.435 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:29:56.435 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246560+0000", 2021-02-10T04:29:56.435 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7100000000000002e-06 2021-02-10T04:29:56.436 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.436 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.436 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:29:56.436 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246570+0000", 2021-02-10T04:29:56.436 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8553e-05 2021-02-10T04:29:56.436 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.436 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.436 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:29:56.436 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246589+0000", 2021-02-10T04:29:56.436 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.6730000000000004e-06 2021-02-10T04:29:56.436 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.436 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.436 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:29:56.437 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246598+0000", 2021-02-10T04:29:56.437 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.2690999999999999e-05 2021-02-10T04:29:56.437 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.437 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.437 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:56.437 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246671+0000", 2021-02-10T04:29:56.437 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7110000000000007e-06 2021-02-10T04:29:56.437 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.437 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.437 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2021-02-10T04:29:56.437 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246681+0000", 2021-02-10T04:29:56.437 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.002088974 2021-02-10T04:29:56.437 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.438 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.438 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:56.438 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.248770+0000", 2021-02-10T04:29:56.438 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000114197 2021-02-10T04:29:56.438 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.438 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.438 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:29:56.438 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.248884+0000", 2021-02-10T04:29:56.438 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.002423344 2021-02-10T04:29:56.438 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.438 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.438 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:29:56.438 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.251307+0000", 2021-02-10T04:29:56.439 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0048228409999999996 2021-02-10T04:29:56.439 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.439 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:29:56.439 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.439 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.439 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.439 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:331 7.1 7:8e9d3756:::smithi08575397-17:head [read 0~2268912,checksum] snapc 0=[] ondisk+read+known_if_redirected e106)", 2021-02-10T04:29:56.439 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:28:42.246670+0000", 2021-02-10T04:29:56.439 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 74.113425655, 2021-02-10T04:29:56.439 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0054719130000000001, 2021-02-10T04:29:56.439 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:29:56.439 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:29:56.439 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:29:56.440 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:29:56.440 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:29:56.440 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 331 2021-02-10T04:29:56.440 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.440 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:29:56.440 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.440 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:29:56.440 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246670+0000", 2021-02-10T04:29:56.440 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:29:56.440 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.440 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.440 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:29:56.440 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246670+0000", 2021-02-10T04:29:56.441 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.2126999999999995e-05 2021-02-10T04:29:56.441 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.441 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.441 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:29:56.441 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246752+0000", 2021-02-10T04:29:56.441 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.4479999999999999e-06 2021-02-10T04:29:56.441 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.441 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.441 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:29:56.441 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246759+0000", 2021-02-10T04:29:56.441 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6818999999999999e-05 2021-02-10T04:29:56.441 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.441 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.442 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:29:56.442 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246776+0000", 2021-02-10T04:29:56.442 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.2739999999999998e-06 2021-02-10T04:29:56.442 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.442 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.442 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:29:56.442 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246781+0000", 2021-02-10T04:29:56.442 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.2923999999999997e-05 2021-02-10T04:29:56.442 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.442 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.442 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:56.442 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246834+0000", 2021-02-10T04:29:56.442 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0639999999999999e-06 2021-02-10T04:29:56.443 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.443 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.443 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting_for_map not empty", 2021-02-10T04:29:56.443 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246840+0000", 2021-02-10T04:29:56.443 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0044871240000000003 2021-02-10T04:29:56.443 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.443 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.443 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:56.443 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.251327+0000", 2021-02-10T04:29:56.443 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.7003999999999998e-05 2021-02-10T04:29:56.443 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.443 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:29:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.251384+0000", 2021-02-10T04:29:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00075812899999999996 2021-02-10T04:29:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:29:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.252142+0000", 2021-02-10T04:29:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0054719130000000001 2021-02-10T04:29:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:29:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:332 7.1 7:8e9d3756:::smithi08575397-17:head [sparse-read 0~1002596] snapc 0=[] ondisk+read+known_if_redirected e106)", 2021-02-10T04:29:56.445 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:28:42.246865+0000", 2021-02-10T04:29:56.445 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 74.113230845000004, 2021-02-10T04:29:56.445 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0054856089999999998, 2021-02-10T04:29:56.445 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:29:56.445 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:29:56.445 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:29:56.445 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:29:56.445 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:29:56.445 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 332 2021-02-10T04:29:56.445 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.445 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:29:56.445 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.445 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:29:56.446 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246865+0000", 2021-02-10T04:29:56.446 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:29:56.446 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.446 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.446 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:29:56.446 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246865+0000", 2021-02-10T04:29:56.446 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.7045000000000002e-05 2021-02-10T04:29:56.446 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.446 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.446 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:29:56.446 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246902+0000", 2021-02-10T04:29:56.446 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.525e-06 2021-02-10T04:29:56.447 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.447 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.447 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:29:56.447 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246910+0000", 2021-02-10T04:29:56.447 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4123e-05 2021-02-10T04:29:56.447 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.447 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.447 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:29:56.447 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246924+0000", 2021-02-10T04:29:56.447 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.9059999999999996e-06 2021-02-10T04:29:56.447 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.447 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.447 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:29:56.447 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246931+0000", 2021-02-10T04:29:56.448 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.0843000000000001e-05 2021-02-10T04:29:56.448 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.448 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.448 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:56.448 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246981+0000", 2021-02-10T04:29:56.448 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.353e-06 2021-02-10T04:29:56.448 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.448 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.448 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting_for_map not empty", 2021-02-10T04:29:56.448 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246988+0000", 2021-02-10T04:29:56.448 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0051933530000000004 2021-02-10T04:29:56.448 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.448 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.449 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:56.449 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.252181+0000", 2021-02-10T04:29:56.449 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0241000000000001e-05 2021-02-10T04:29:56.449 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.449 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.449 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:29:56.449 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.252241+0000", 2021-02-10T04:29:56.449 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00010922 2021-02-10T04:29:56.449 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.449 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.449 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:29:56.449 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.252351+0000", 2021-02-10T04:29:56.450 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0054856089999999998 2021-02-10T04:29:56.450 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.450 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:29:56.450 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.450 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.450 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.450 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:553 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 17] snapc 1e=[1e,1d,1c,1b,19,17,15] ondisk+write+known_if_redirected e127)", 2021-02-10T04:29:56.450 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:03.299307+0000", 2021-02-10T04:29:56.450 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 53.060789186000001, 2021-02-10T04:29:56.450 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012622955, 2021-02-10T04:29:56.450 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:29:56.450 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:29:56.450 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:29:56.451 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:29:56.451 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:29:56.451 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 553 2021-02-10T04:29:56.451 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.451 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:29:56.451 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.451 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:29:56.451 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299307+0000", 2021-02-10T04:29:56.451 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:29:56.451 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.451 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.451 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:29:56.452 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299307+0000", 2021-02-10T04:29:56.452 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.8406000000000003e-05 2021-02-10T04:29:56.452 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.452 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.452 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:29:56.452 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299365+0000", 2021-02-10T04:29:56.452 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7580000000000004e-06 2021-02-10T04:29:56.452 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.452 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.452 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:29:56.452 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299375+0000", 2021-02-10T04:29:56.452 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6067e-05 2021-02-10T04:29:56.452 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.452 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.453 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:29:56.453 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299391+0000", 2021-02-10T04:29:56.453 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.3809999999999999e-06 2021-02-10T04:29:56.453 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.453 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.453 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:29:56.453 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299399+0000", 2021-02-10T04:29:56.453 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6042999999999999e-05 2021-02-10T04:29:56.453 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.453 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.453 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:56.453 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299455+0000", 2021-02-10T04:29:56.453 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.2917999999999994e-05 2021-02-10T04:29:56.454 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.454 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.454 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:29:56.454 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299548+0000", 2021-02-10T04:29:56.454 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0037641939999999998 2021-02-10T04:29:56.454 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.454 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.454 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 3,6", 2021-02-10T04:29:56.454 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.303312+0000", 2021-02-10T04:29:56.454 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0044879940000000004 2021-02-10T04:29:56.454 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.454 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.455 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:29:56.455 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.307800+0000", 2021-02-10T04:29:56.455 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0023093699999999998 2021-02-10T04:29:56.455 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.455 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.455 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:29:56.455 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.310110+0000", 2021-02-10T04:29:56.455 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0017106650000000001 2021-02-10T04:29:56.455 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.455 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.455 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:29:56.455 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.311820+0000", 2021-02-10T04:29:56.455 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.2655000000000003e-05 2021-02-10T04:29:56.456 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.456 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.456 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:29:56.456 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.311873+0000", 2021-02-10T04:29:56.456 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6504e-05 2021-02-10T04:29:56.456 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.456 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.456 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:29:56.456 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.311930+0000", 2021-02-10T04:29:56.456 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012622955 2021-02-10T04:29:56.456 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.456 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:29:56.456 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.457 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.457 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.457 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:554 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e127)", 2021-02-10T04:29:56.457 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:03.299463+0000", 2021-02-10T04:29:56.457 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 53.060633142999997, 2021-02-10T04:29:56.457 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012735955, 2021-02-10T04:29:56.457 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:29:56.457 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:29:56.457 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:29:56.457 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:29:56.457 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:29:56.457 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 554 2021-02-10T04:29:56.457 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.458 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:29:56.458 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.458 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:29:56.458 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299463+0000", 2021-02-10T04:29:56.458 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:29:56.458 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.458 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.458 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:29:56.458 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299463+0000", 2021-02-10T04:29:56.458 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.3636e-05 2021-02-10T04:29:56.458 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.458 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.458 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:29:56.459 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299506+0000", 2021-02-10T04:29:56.459 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.7109999999999998e-06 2021-02-10T04:29:56.459 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.459 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.459 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:29:56.459 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299513+0000", 2021-02-10T04:29:56.459 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7858000000000001e-05 2021-02-10T04:29:56.459 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.459 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.459 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:29:56.459 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299531+0000", 2021-02-10T04:29:56.459 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9790000000000001e-06 2021-02-10T04:29:56.459 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.460 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.460 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:29:56.460 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299536+0000", 2021-02-10T04:29:56.460 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00019002799999999999 2021-02-10T04:29:56.460 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.460 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.460 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:56.460 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299726+0000", 2021-02-10T04:29:56.460 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5302000000000001e-05 2021-02-10T04:29:56.460 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.460 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.460 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:29:56.460 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299771+0000", 2021-02-10T04:29:56.461 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0078913030000000006 2021-02-10T04:29:56.461 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.461 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.461 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:56.461 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.307663+0000", 2021-02-10T04:29:56.461 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.8472000000000002e-05 2021-02-10T04:29:56.461 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.461 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.461 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2021-02-10T04:29:56.461 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.307761+0000", 2021-02-10T04:29:56.461 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0042153260000000001 2021-02-10T04:29:56.461 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.461 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.462 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:56.462 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.311976+0000", 2021-02-10T04:29:56.462 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.6063999999999994e-05 2021-02-10T04:29:56.462 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.462 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.462 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:29:56.462 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.312072+0000", 2021-02-10T04:29:56.462 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00012627600000000001 2021-02-10T04:29:56.462 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.462 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.462 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:29:56.462 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.312199+0000", 2021-02-10T04:29:56.462 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012735955 2021-02-10T04:29:56.463 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.463 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:29:56.463 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.463 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.463 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.463 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:579 7.3 e128/93 7:c765b4cc:::smithi08575397-30:head v 128'46, mlcod=123'42)", 2021-02-10T04:29:56.463 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:04.341743+0000", 2021-02-10T04:29:56.463 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 52.018352438999997, 2021-02-10T04:29:56.463 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0052680720000000004, 2021-02-10T04:29:56.463 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:29:56.463 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:29:56.463 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:29:56.463 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.464 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:29:56.464 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:04.341743+0000", 2021-02-10T04:29:56.464 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:29:56.464 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.464 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.464 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:29:56.464 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:04.341743+0000", 2021-02-10T04:29:56.464 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0031662869999999998 2021-02-10T04:29:56.464 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.464 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.464 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:29:56.464 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:04.344910+0000", 2021-02-10T04:29:56.464 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.8019999999999998e-06 2021-02-10T04:29:56.464 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.465 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.465 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:29:56.465 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:04.344919+0000", 2021-02-10T04:29:56.465 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4846e-05 2021-02-10T04:29:56.465 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.465 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.465 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:29:56.465 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:04.344933+0000", 2021-02-10T04:29:56.465 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.3240000000000003e-06 2021-02-10T04:29:56.465 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.465 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.465 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:29:56.465 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:04.344942+0000", 2021-02-10T04:29:56.466 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6986e-05 2021-02-10T04:29:56.466 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.466 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.466 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:56.466 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:04.344999+0000", 2021-02-10T04:29:56.466 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8238000000000001e-05 2021-02-10T04:29:56.466 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.466 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.466 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:29:56.466 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:04.345017+0000", 2021-02-10T04:29:56.466 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001945081 2021-02-10T04:29:56.466 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.467 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.467 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:29:56.467 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:04.346962+0000", 2021-02-10T04:29:56.467 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9508000000000001e-05 2021-02-10T04:29:56.467 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.467 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.467 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:29:56.467 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:04.347011+0000", 2021-02-10T04:29:56.467 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0052680720000000004 2021-02-10T04:29:56.467 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.467 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:29:56.467 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.467 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.467 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.468 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:558 7.3 e129/93 7:cf4c8c3b:::smithi08575397-6:head v 129'48, mlcod=123'44)", 2021-02-10T04:29:56.468 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:05.162978+0000", 2021-02-10T04:29:56.468 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 51.197118228999997, 2021-02-10T04:29:56.468 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0073914510000000003, 2021-02-10T04:29:56.468 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:29:56.468 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:29:56.468 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:29:56.468 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.468 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:29:56.468 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.162978+0000", 2021-02-10T04:29:56.468 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:29:56.468 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.468 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.469 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:29:56.469 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.162978+0000", 2021-02-10T04:29:56.469 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0044459269999999997 2021-02-10T04:29:56.469 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.469 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.469 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:29:56.469 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.167424+0000", 2021-02-10T04:29:56.469 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.3649999999999993e-06 2021-02-10T04:29:56.469 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.469 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.469 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:29:56.469 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.167433+0000", 2021-02-10T04:29:56.469 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1695999999999999e-05 2021-02-10T04:29:56.470 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.470 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.470 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:29:56.470 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.167445+0000", 2021-02-10T04:29:56.470 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.8929999999999999e-06 2021-02-10T04:29:56.470 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.470 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.470 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:29:56.470 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.167452+0000", 2021-02-10T04:29:56.470 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.0009000000000002e-05 2021-02-10T04:29:56.470 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.470 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.470 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:56.471 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.167502+0000", 2021-02-10T04:29:56.471 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0502000000000001e-05 2021-02-10T04:29:56.471 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.471 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.471 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:29:56.471 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.167522+0000", 2021-02-10T04:29:56.471 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0028030339999999998 2021-02-10T04:29:56.471 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.471 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.471 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:29:56.471 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.170325+0000", 2021-02-10T04:29:56.471 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.4024999999999999e-05 2021-02-10T04:29:56.471 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.472 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.472 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:29:56.472 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.170369+0000", 2021-02-10T04:29:56.472 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0073914510000000003 2021-02-10T04:29:56.472 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.472 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:29:56.472 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.472 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.472 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.472 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:1175 7.3 e176/93 7:cf4c8c3b:::smithi08575397-6:head v 176'131, mlcod=172'127)", 2021-02-10T04:29:56.472 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:52.233097+0000", 2021-02-10T04:29:56.472 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 4.1269986400000001, 2021-02-10T04:29:56.472 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011081865, 2021-02-10T04:29:56.473 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:29:56.473 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:29:56.473 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:29:56.473 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.473 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:29:56.473 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.233097+0000", 2021-02-10T04:29:56.473 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:29:56.473 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.473 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.473 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:29:56.473 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.233097+0000", 2021-02-10T04:29:56.473 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.005275587 2021-02-10T04:29:56.473 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.474 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.474 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:29:56.474 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.238373+0000", 2021-02-10T04:29:56.474 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.367e-05 2021-02-10T04:29:56.474 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.474 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.474 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:29:56.474 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.238386+0000", 2021-02-10T04:29:56.474 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5914e-05 2021-02-10T04:29:56.474 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.474 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.474 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:29:56.474 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.238402+0000", 2021-02-10T04:29:56.475 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.6599999999999995e-06 2021-02-10T04:29:56.475 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.475 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.475 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:29:56.475 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.238410+0000", 2021-02-10T04:29:56.475 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.1702000000000001e-05 2021-02-10T04:29:56.475 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.475 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.475 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:56.475 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.238462+0000", 2021-02-10T04:29:56.475 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.6571e-05 2021-02-10T04:29:56.475 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.475 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.476 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:29:56.476 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.238488+0000", 2021-02-10T04:29:56.476 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0056189509999999996 2021-02-10T04:29:56.476 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.476 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.476 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:29:56.476 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.244107+0000", 2021-02-10T04:29:56.476 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1810000000000005e-05 2021-02-10T04:29:56.476 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.476 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.476 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:29:56.476 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.244179+0000", 2021-02-10T04:29:56.476 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011081865 2021-02-10T04:29:56.476 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.477 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:29:56.477 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.477 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.477 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.477 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:1203 7.9 e179/93 7:93db45b2:::smithi08575397-44:head v 179'78, mlcod=173'74)", 2021-02-10T04:29:56.477 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:55.254800+0000", 2021-02-10T04:29:56.477 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 1.105295897, 2021-02-10T04:29:56.477 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.005866853, 2021-02-10T04:29:56.477 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:29:56.477 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:29:56.477 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:29:56.477 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.477 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:29:56.478 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:55.254800+0000", 2021-02-10T04:29:56.478 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:29:56.478 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.478 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.478 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:29:56.478 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:55.254800+0000", 2021-02-10T04:29:56.478 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0030262039999999998 2021-02-10T04:29:56.478 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.478 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.478 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:29:56.478 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:55.257826+0000", 2021-02-10T04:29:56.478 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.711e-06 2021-02-10T04:29:56.478 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.479 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.479 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:29:56.479 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:55.257832+0000", 2021-02-10T04:29:56.479 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1010000000000003e-06 2021-02-10T04:29:56.479 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.479 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.479 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:29:56.479 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:55.257839+0000", 2021-02-10T04:29:56.479 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.1680000000000001e-06 2021-02-10T04:29:56.479 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.479 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.479 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:29:56.479 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:55.257845+0000", 2021-02-10T04:29:56.480 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.3109999999999998e-05 2021-02-10T04:29:56.480 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.480 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.480 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:29:56.480 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:55.257898+0000", 2021-02-10T04:29:56.480 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.1696999999999999e-05 2021-02-10T04:29:56.480 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.480 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.480 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:29:56.480 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:55.257920+0000", 2021-02-10T04:29:56.480 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0027001820000000002 2021-02-10T04:29:56.480 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.481 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.481 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:29:56.481 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:55.260620+0000", 2021-02-10T04:29:56.481 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.668e-05 2021-02-10T04:29:56.481 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:29:56.481 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:29:56.481 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:29:56.481 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:55.260667+0000", 2021-02-10T04:29:56.481 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.005866853 2021-02-10T04:29:56.481 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.482 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:29:56.482 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.482 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:29:56.482 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:29:56.482 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:29:56.482 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:29:55 smithi085 conmon[45443]: debug 2021-02-10T04:29:56.483 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:29:55 smithi085 conmon[45443]: 2021-02-10T04:29:55.994+0000 7f7140d73700 -1 osd.1 179 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:29:56.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:56 smithi164 conmon[29288]: cluster 2021-02-10T04:29:55.008692+0000 osd.1 (osd.1) 66 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:29:56.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:56 smithi164 conmon[29288]: cluster 2021-02-10T04:29:55.008701+0000 osd.1 (osd.1) 67 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:29:56.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:56 smithi164 conmon[29288]: cluster 2021-02-10T04:29:55.994574+0000 osd.1 (osd.1) 68 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:29:56.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:56 smithi164 conmon[29288]: cluster 2021-02-10T04:29:55.994585+0000 osd.1 (osd.1) 69 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:29:56.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:56 smithi164 conmon[29288]: cluster 2021-02-10T04:29:56.011068+0000 mgr.y (mgr.14140) 384 : cluster [DBG] pgmap v418: 168 pgs: 3 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 156 active+clean; 127 MiB data, 386 MiB used, 715 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.3 MiB/s wr, 14 op/s 2021-02-10T04:29:56.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:56 smithi164 conmon[29288]: cluster 2021-02-10T04:29:56.054971+0000 mon.a (mon.0) 875 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2021-02-10T04:29:56.870 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[29319]: cluster 2021-02-10T04:29:55.008692+0000 osd.1 (osd.1) 66 2021-02-10T04:29:56.870 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:29:56.870 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[29319]: cluster 2021-02-10T04:29:55.008701+0000 osd.1 (osd.1) 67 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:29:56.870 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:29:56.870 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[29319]: 02-10T04:29:55.994574+0000 osd.1 (osd.1) 68 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:29:56.870 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[29319]: cluster 2021-02-10T04:29:56.870 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[29319]: 2021-02-10T04:29:55.994585 2021-02-10T04:29:56.870 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[29319]: +0000 osd.1 (osd.1) 69 : cluster 2021-02-10T04:29:56.871 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:29:56.871 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:29:56.871 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[29319]: 02-10T04:29:56.011068 2021-02-10T04:29:56.871 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 384 2021-02-10T04:29:56.871 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[29319]: : cluster [DBG] pgmap v418: 168 pgs: 3 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 156 active+clean; 127 MiB data, 386 MiB used, 715 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.3 MiB/s wr, 14 op/s 2021-02-10T04:29:56.871 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[29319]: cluster 2021-02-10T04:29:56.871 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[29319]: 2021-02-10T04:29:56. 2021-02-10T04:29:56.871 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[29319]: 054971+0000 mon.a (mon.0) 2021-02-10T04:29:56.871 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[29319]: 875 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2021-02-10T04:29:56.871 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[35961]: cluster 2021-02-10T04:29:55. 2021-02-10T04:29:56.871 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[35961]: 008692+0000 osd.1 (osd.1) 66 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:29:56.871 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[35961]: cluster 2021-02-10T04:29:56.872 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[35961]: 2021-02-10T04:29:55.008701+0000 2021-02-10T04:29:56.872 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[35961]: osd.1 (osd.1) 67 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:29:56.872 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[35961]: 2021-02-10T04:29:56.872 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:29:56.872 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[35961]: 02-10T04:29:55.994574+0000 osd.1 ( 2021-02-10T04:29:56.872 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[35961]: osd.1) 68 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:29:56.872 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[35961]: cluster 2021-02-10T04:29:56.872 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:29:56.872 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[35961]: 04:29:55.994585+0000 osd.1 (osd.1) 69 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:29:56.872 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[35961]: 2021-02-10T04:29:56.872 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:29:56.872 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[35961]: 10T04:29:56.011068+0000 mgr.y ( 2021-02-10T04:29:56.872 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[35961]: mgr.14140) 384 : cluster [DBG] pgmap v418: 168 pgs: 3 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 156 active+clean; 127 MiB data, 386 MiB used, 715 GiB / 715 GiB avail; 2.5 MiB/s rd, 3.3 MiB/s wr, 14 op/s 2021-02-10T04:29:56.873 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:29:56.873 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[35961]: 04:29:56.054971+0000 mon.a (mon.0) 875 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2021-02-10T04:29:57.011 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:29:57.058 INFO:tasks.rados.rados.0.smithi085.stdout:449: done (6 left) 2021-02-10T04:29:57.059 INFO:tasks.rados.rados.0.smithi085.stdout:450: read oid 43 snap -1 2021-02-10T04:29:57.059 INFO:tasks.rados.rados.0.smithi085.stdout:450: expect deleted 2021-02-10T04:29:57.059 INFO:tasks.rados.rados.0.smithi085.stdout:451: read oid 6 snap -1 2021-02-10T04:29:57.059 INFO:tasks.rados.rados.0.smithi085.stdout:451: expect (ObjNum 96 snap 19 seq_num 96) 2021-02-10T04:29:57.059 INFO:tasks.rados.rados.0.smithi085.stdout:452: snap_remove snap 38 2021-02-10T04:29:57.118 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:29:57.305 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:29:57 smithi085 conmon[45443]: debug 2021-02-10T04:29:57.040+0000 7f7140d73700 -1 osd.1 180 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:29:57.305 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:29:56 smithi085 conmon[41540]: debug 2021-02-10T04:29:56.965+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:57.345 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:57.547 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:29:57.653 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:29:57.686 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:57.940 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:29:57 smithi085 conmon[52836]: debug 2021-02-10T04:29:57.523+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:58.050 INFO:tasks.rados.rados.0.smithi085.stdout:452: done (8 left) 2021-02-10T04:29:58.050 INFO:tasks.rados.rados.0.smithi085.stdout:453: write oid 50 current snap is 42 2021-02-10T04:29:58.050 INFO:tasks.rados.rados.0.smithi085.stdout:453: seq_num 188 ranges {759716=539284,2034850=614062} 2021-02-10T04:29:58.058 INFO:tasks.rados.rados.0.smithi085.stdout:453: writing smithi08575397-50 from 759716 to 1299000 tid 1 2021-02-10T04:29:58.066 INFO:tasks.rados.rados.0.smithi085.stdout:453: writing smithi08575397-50 from 2034850 to 2648912 tid 2 2021-02-10T04:29:58.073 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.2 to 1 (10000) 2021-02-10T04:29:58.074 INFO:tasks.rados.rados.0.smithi085.stdout:451: expect (ObjNum 96 snap 19 seq_num 96) 2021-02-10T04:29:58.078 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:29:57 smithi164 conmon[40892]: debug 2021-02-10T04:29:57.753+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:58.079 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:29:57 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:29:58.240 INFO:tasks.rados.rados.0.smithi085.stdout:453: finishing write tid 1 to smithi08575397-50 2021-02-10T04:29:58.240 INFO:tasks.rados.rados.0.smithi085.stdout:450: done (8 left) 2021-02-10T04:29:58.240 INFO:tasks.rados.rados.0.smithi085.stdout:451: done (7 left) 2021-02-10T04:29:58.240 INFO:tasks.rados.rados.0.smithi085.stdout:454: write (excl) oid 3 current snap is 42 2021-02-10T04:29:58.240 INFO:tasks.rados.rados.0.smithi085.stdout:453: finishing write tid 2 to smithi08575397-50 2021-02-10T04:29:58.240 INFO:tasks.rados.rados.0.smithi085.stdout:453: finishing write tid 3 to smithi08575397-50 2021-02-10T04:29:58.240 INFO:tasks.rados.rados.0.smithi085.stdout:453: finishing write tid 4 to smithi08575397-50 2021-02-10T04:29:58.240 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 78 (ObjNum 188 snap 42 seq_num 188) dirty exists 2021-02-10T04:29:58.241 INFO:tasks.rados.rados.0.smithi085.stdout:453: left oid 50 (ObjNum 188 snap 42 seq_num 188) 2021-02-10T04:29:58.241 INFO:tasks.rados.rados.0.smithi085.stdout:454: seq_num 189 ranges {776523=614533,1976979=757720} 2021-02-10T04:29:58.248 INFO:tasks.rados.rados.0.smithi085.stdout:454: writing smithi08575397-3 from 776523 to 1391056 tid 1 2021-02-10T04:29:58.257 INFO:tasks.rados.rados.0.smithi085.stdout:454: writing smithi08575397-3 from 1976979 to 2734699 tid 2 2021-02-10T04:29:58.257 INFO:tasks.rados.rados.0.smithi085.stdout:453: done (7 left) 2021-02-10T04:29:58.257 INFO:tasks.rados.rados.0.smithi085.stdout:455: read oid 38 snap -1 2021-02-10T04:29:58.257 INFO:tasks.rados.rados.0.smithi085.stdout:455: expect deleted 2021-02-10T04:29:58.257 INFO:tasks.rados.rados.0.smithi085.stdout:454: finishing write tid 1 to smithi08575397-3 2021-02-10T04:29:58.257 INFO:tasks.rados.rados.0.smithi085.stdout:456: read oid 5 snap -1 2021-02-10T04:29:58.257 INFO:tasks.rados.rados.0.smithi085.stdout:456: expect (ObjNum 147 snap 30 seq_num 147) 2021-02-10T04:29:58.257 INFO:tasks.rados.rados.0.smithi085.stdout:457: write oid 11 current snap is 42 2021-02-10T04:29:58.257 INFO:tasks.rados.rados.0.smithi085.stdout:457: seq_num 190 ranges {793330=689782,2035461=617731,3101545=91785} 2021-02-10T04:29:58.265 INFO:tasks.rados.rados.0.smithi085.stdout:457: writing smithi08575397-11 from 793330 to 1483112 tid 1 2021-02-10T04:29:58.273 INFO:tasks.rados.rados.0.smithi085.stdout:457: writing smithi08575397-11 from 2035461 to 2653192 tid 2 2021-02-10T04:29:58.276 INFO:tasks.rados.rados.0.smithi085.stdout:457: writing smithi08575397-11 from 3101545 to 3193330 tid 3 2021-02-10T04:29:58.276 INFO:tasks.rados.rados.0.smithi085.stdout:458: rollback oid 43 current snap is 42 2021-02-10T04:29:58.276 INFO:tasks.rados.rados.0.smithi085.stdout:458: done (10 left) 2021-02-10T04:29:58.276 INFO:tasks.rados.rados.0.smithi085.stdout:459: snap_create 2021-02-10T04:29:58.276 INFO:tasks.rados.rados.0.smithi085.stdout:454: finishing write tid 2 to smithi08575397-3 2021-02-10T04:29:58.276 INFO:tasks.rados.rados.0.smithi085.stdout:454: finishing write tid 3 to smithi08575397-3 2021-02-10T04:29:58.276 INFO:tasks.rados.rados.0.smithi085.stdout:454: finishing write tid 4 to smithi08575397-3 2021-02-10T04:29:58.276 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 86 (ObjNum 189 snap 42 seq_num 189) dirty exists 2021-02-10T04:29:58.277 INFO:tasks.rados.rados.0.smithi085.stdout:454: left oid 3 (ObjNum 189 snap 42 seq_num 189) 2021-02-10T04:29:58.277 INFO:tasks.rados.rados.0.smithi085.stdout:456: expect (ObjNum 147 snap 30 seq_num 147) 2021-02-10T04:29:58.328 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:29:58.363 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:58 smithi164 conmon[29288]: cluster 2021-02-10T04:29:57.041282+0000 osd.1 (osd 2021-02-10T04:29:58.363 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:58 smithi164 conmon[29288]: .1) 70 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:29:58.363 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:58 smithi164 conmon[29288]: cluster 2021-02-10T04:29:57.041293+0000 osd.1 (osd.1) 71 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:29:58.363 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:58 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:29:58.363 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:58 smithi164 conmon[29288]: -10T04:29:57.057292+0000 mon.a (mon.0) 876 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2021-02-10T04:29:58.363 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:58 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:29:58.363 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:58 smithi164 conmon[29288]: 10T04:29:57.736657+0000 mon.c (mon.1) 46 : audit [INF] 2021-02-10T04:29:58.363 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:58 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/1365978016' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-02-10T04:29:58.363 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:58 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:29:58.364 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:58 smithi164 conmon[29288]: 02-10T04:29:57.737039+0000 mon.a (mon 2021-02-10T04:29:58.364 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:58 smithi164 conmon[29288]: .0) 877 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-02-10T04:29:58.364 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:58 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:29:58.365 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:58 smithi164 conmon[29288]: -10T04:29:58.049958+0000 mon.a 2021-02-10T04:29:58.365 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:58 smithi164 conmon[29288]: (mon.0) 878 : audit [INF] 2021-02-10T04:29:58.365 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:58 smithi164 conmon[29288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2021-02-10T04:29:58.365 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:58 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:29:58.365 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:58 smithi164 conmon[29288]: 04:29:58.050009+0000 mon.a (mon.0 2021-02-10T04:29:58.365 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:58 smithi164 conmon[29288]: ) 879 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2021-02-10T04:29:58.430 INFO:tasks.rados.rados.0.smithi085.stdout:457: finishing write tid 1 to smithi08575397-11 2021-02-10T04:29:58.431 INFO:tasks.rados.rados.0.smithi085.stdout:457: finishing write tid 2 to smithi08575397-11 2021-02-10T04:29:58.431 INFO:tasks.rados.rados.0.smithi085.stdout:457: finishing write tid 3 to smithi08575397-11 2021-02-10T04:29:58.431 INFO:tasks.rados.rados.0.smithi085.stdout:457: finishing write tid 4 to smithi08575397-11 2021-02-10T04:29:58.431 INFO:tasks.rados.rados.0.smithi085.stdout:457: finishing write tid 5 to smithi08575397-11 2021-02-10T04:29:58.431 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 202 (ObjNum 190 snap 42 seq_num 190) dirty exists 2021-02-10T04:29:58.431 INFO:tasks.rados.rados.0.smithi085.stdout:457: left oid 11 (ObjNum 190 snap 42 seq_num 190) 2021-02-10T04:29:58.441 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[45443]: debug 2021-02-10T04:29:58.040+0000 7f7140d73700 -1 osd.1 181 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:29:58.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: cluster 2021-02-10T04:29:58.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 2021-02-10T04:29:58.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 2021 2021-02-10T04:29:58.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: - 2021-02-10T04:29:58.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 02 2021-02-10T04:29:58.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: - 2021-02-10T04:29:58.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 10 2021-02-10T04:29:58.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: T 2021-02-10T04:29:58.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 04 2021-02-10T04:29:58.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: : 2021-02-10T04:29:58.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 29 2021-02-10T04:29:58.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: : 2021-02-10T04:29:58.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 57 2021-02-10T04:29:58.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: . 2021-02-10T04:29:58.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 041282 2021-02-10T04:29:58.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: +0000 2021-02-10T04:29:58.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 2021-02-10T04:29:58.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: osd.1 2021-02-10T04:29:58.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: ( 2021-02-10T04:29:58.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: osd 2021-02-10T04:29:58.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: . 2021-02-10T04:29:58.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 1 2021-02-10T04:29:58.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: ) 2021-02-10T04:29:58.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 70 2021-02-10T04:29:58.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: : 2021-02-10T04:29:58.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: cluster 2021-02-10T04:29:58.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 2021-02-10T04:29:58.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: [WRN] 2021-02-10T04:29:58.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 2021-02-10T04:29:58.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:29:58.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 2021-02-10T04:29:58.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: cluster 2021-02-10T04:29:58.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 2021-02-10T04:29:58.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 2021 2021-02-10T04:29:58.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: - 2021-02-10T04:29:58.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 02 2021-02-10T04:29:58.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: - 2021-02-10T04:29:58.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 10 2021-02-10T04:29:58.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: T 2021-02-10T04:29:58.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 04 2021-02-10T04:29:58.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: : 2021-02-10T04:29:58.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 29 2021-02-10T04:29:58.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: : 2021-02-10T04:29:58.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 57 2021-02-10T04:29:58.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: . 2021-02-10T04:29:58.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 041293 2021-02-10T04:29:58.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: +0000 2021-02-10T04:29:58.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 2021-02-10T04:29:58.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: osd.1 2021-02-10T04:29:58.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: ( 2021-02-10T04:29:58.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: osd 2021-02-10T04:29:58.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: . 2021-02-10T04:29:58.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 1 2021-02-10T04:29:58.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: ) 2021-02-10T04:29:58.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 71 2021-02-10T04:29:58.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: : 2021-02-10T04:29:58.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: cluster 2021-02-10T04:29:58.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 2021-02-10T04:29:58.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: [WRN] 2021-02-10T04:29:58.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 2021-02-10T04:29:58.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:29:58.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 2021-02-10T04:29:58.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: cluster 2021-02-10T04:29:58.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 2021-02-10T04:29:58.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 2021 2021-02-10T04:29:58.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: - 2021-02-10T04:29:58.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 02 2021-02-10T04:29:58.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: - 2021-02-10T04:29:58.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 10 2021-02-10T04:29:58.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: T 2021-02-10T04:29:58.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 04 2021-02-10T04:29:58.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: : 2021-02-10T04:29:58.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 29 2021-02-10T04:29:58.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: : 2021-02-10T04:29:58.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 57 2021-02-10T04:29:58.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: . 2021-02-10T04:29:58.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 057292 2021-02-10T04:29:58.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: +0000 2021-02-10T04:29:58.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 2021-02-10T04:29:58.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: mon.a (mon.0) 876 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2021-02-10T04:29:58.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 2021-02-10T04:29:58.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: audit 2021-02-10T04:29: 2021-02-10T04:29:58.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 57.736657+0000 mon.c (mon.1) 46 : audit [INF] from='client.? 172.21.15.85:0/1365978016' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-02-10T04:29:58.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: audit 2021-02-10T04:29:57. 2021-02-10T04:29:58.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 737039+0000 mon.a (mon.0) 877 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-02-10T04:29:58.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:29:58.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: :29:58.049958+0000 mon.a (mon.0) 878 : audit 2021-02-10T04:29:58.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2021-02-10T04:29:58.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:29:58.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 02-10T04:29:58.050009+0000 mon.a (mon.0) 879 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2021-02-10T04:29:58.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[29319]: 2021-02-10T04:29:58.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[35961]: cluster 2021-02-10T04:29:58.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[35961]: 2021-02-10T04:29:58.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[35961]: 2021 2021-02-10T04:29:58.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[35961]: -02-10T04:29:57.041282 2021-02-10T04:29:58.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 70 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:29:58.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[35961]: cluster 2021-02-10T04:29:57.041293+0000 osd.1 ( 2021-02-10T04:29:58.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[35961]: osd.1) 71 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:29:58.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[35961]: cluster 2021-02-10T04:29:58.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[35961]: 2021-02-10T04:29:57.057292+0000 mon.a (mon. 2021-02-10T04:29:58.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[35961]: 0) 876 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2021-02-10T04:29:58.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:29:58.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[35961]: 02-10T04:29:57.736657+0000 mon.c ( 2021-02-10T04:29:58.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[35961]: mon.1) 46 : audit [INF] from='client.? 172.21.15.85:0/1365978016' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-02-10T04:29:58.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:29:58.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[35961]: 02-10T04:29:57.737039+0000 2021-02-10T04:29:58.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[35961]: mon.a (mon.0) 877 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-02-10T04:29:58.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[35961]: 2021-02-10T04:29:58.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[35961]: audit 2021-02-10T04:29:58.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[35961]: 2021-02-10T04:29:58.049958+0000 2021-02-10T04:29:58.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[35961]: mon.a (mon.0) 878 : audit [INF] 2021-02-10T04:29:58.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[35961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2021-02-10T04:29:58.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:29:58.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[35961]: :29:58.050009+0000 mon.a (mon 2021-02-10T04:29:58.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:58 smithi085 conmon[35961]: .0) 879 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2021-02-10T04:29:58.466 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:29:58.498 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:58.595 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.5 dump_ops_in_flight 2021-02-10T04:29:58.655 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:29:58.761 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:29:58.790 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:29:58.893 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:29:58 smithi164 conmon[37263]: debug 2021-02-10T04:29:58.624+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:29:58.898 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:29:58.942 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset noscrub 2021-02-10T04:29:59.083 INFO:tasks.rados.rados.0.smithi085.stdout:454: done (10 left) 2021-02-10T04:29:59.083 INFO:tasks.rados.rados.0.smithi085.stdout:455: done (9 left) 2021-02-10T04:29:59.084 INFO:tasks.rados.rados.0.smithi085.stdout:456: done (8 left) 2021-02-10T04:29:59.084 INFO:tasks.rados.rados.0.smithi085.stdout:457: done (7 left) 2021-02-10T04:29:59.084 INFO:tasks.rados.rados.0.smithi085.stdout:459: done (6 left) 2021-02-10T04:29:59.085 INFO:tasks.rados.rados.0.smithi085.stdout:460: read oid 46 snap 42 2021-02-10T04:29:59.086 INFO:tasks.rados.rados.0.smithi085.stdout:460: expect (ObjNum 158 snap 35 seq_num 158) 2021-02-10T04:29:59.086 INFO:tasks.rados.rados.0.smithi085.stdout:461: write oid 3 current snap is 43 2021-02-10T04:29:59.087 INFO:tasks.rados.rados.0.smithi085.stdout:461: seq_num 191 ranges {410137=481384,1410296=761389,2728968=481169} 2021-02-10T04:29:59.088 INFO:tasks.rados.rados.0.smithi085.stdout:461: writing smithi08575397-3 from 410137 to 891521 tid 1 2021-02-10T04:29:59.096 INFO:tasks.rados.rados.0.smithi085.stdout:461: writing smithi08575397-3 from 1410296 to 2171685 tid 2 2021-02-10T04:29:59.102 INFO:tasks.rados.rados.0.smithi085.stdout:461: writing smithi08575397-3 from 2728968 to 3210137 tid 3 2021-02-10T04:29:59.102 INFO:tasks.rados.rados.0.smithi085.stdout:462: rollback oid 26 current snap is 43 2021-02-10T04:29:59.102 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 26 to 42 2021-02-10T04:29:59.103 INFO:tasks.rados.rados.0.smithi085.stdout:460: expect (ObjNum 158 snap 35 seq_num 158) 2021-02-10T04:29:59.229 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:29:59.371 INFO:tasks.rados.rados.0.smithi085.stdout:460: done (8 left) 2021-02-10T04:29:59.372 INFO:tasks.rados.rados.0.smithi085.stdout:463: write oid 6 current snap is 43 2021-02-10T04:29:59.372 INFO:tasks.rados.rados.0.smithi085.stdout:462: finishing rollback tid 0 to smithi08575397-26 2021-02-10T04:29:59.372 INFO:tasks.rados.rados.0.smithi085.stdout:463: seq_num 192 ranges {426944=556633,1468778=621400,2872744=354200} 2021-02-10T04:29:59.379 INFO:tasks.rados.rados.0.smithi085.stdout:463: writing smithi08575397-6 from 426944 to 983577 tid 1 2021-02-10T04:29:59.385 INFO:tasks.rados.rados.0.smithi085.stdout:463: writing smithi08575397-6 from 1468778 to 2090178 tid 2 2021-02-10T04:29:59.392 INFO:tasks.rados.rados.0.smithi085.stdout:463: writing smithi08575397-6 from 2872744 to 3226944 tid 3 2021-02-10T04:29:59.392 INFO:tasks.rados.rados.0.smithi085.stdout:464: read oid 43 snap 42 2021-02-10T04:29:59.392 INFO:tasks.rados.rados.0.smithi085.stdout:464: expect deleted 2021-02-10T04:29:59.392 INFO:tasks.rados.rados.0.smithi085.stdout:465: write (excl) oid 21 current snap is 43 2021-02-10T04:29:59.393 INFO:tasks.rados.rados.0.smithi085.stdout:465: seq_num 193 ranges {443751=631882,1810907=765058,3067461=176290} 2021-02-10T04:29:59.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:59 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:29:59.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:59 smithi164 conmon[29288]: T04:29:57.755014+0000 mgr.y (mgr.14140) 385 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:29:59.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:59 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:29:59.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:59 smithi164 conmon[29288]: 10T04:29:58.011443+0000 mgr.y 2021-02-10T04:29:59.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:59 smithi164 conmon[29288]: (mgr.14140) 386 : cluster [DBG] pgmap v421: 168 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 159 active+clean; 122 MiB data, 389 MiB used, 715 GiB / 715 GiB avail; 5.2 MiB/s rd, 2.4 MiB/s wr, 13 op/s 2021-02-10T04:29:59.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:59 smithi164 conmon[29288]: 2021-02-10T04:29:59.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:59 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:29:59.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:59 smithi164 conmon[29288]: T04:29:58.041083+0000 osd.1 (osd 2021-02-10T04:29:59.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:59 smithi164 conmon[29288]: .1) 72 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:29:59.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:59 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:29:59.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:59 smithi164 conmon[29288]: -02-10 2021-02-10T04:29:59.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:59 smithi164 conmon[29288]: T04:29:58.041094+0000 osd.1 2021-02-10T04:29:59.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:59 smithi164 conmon[29288]: (osd.1 2021-02-10T04:29:59.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:29:59 smithi164 conmon[29288]: ) 73 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:29:59.398 INFO:tasks.rados.rados.0.smithi085.stdout:465: writing smithi08575397-21 from 443751 to 1075633 tid 1 2021-02-10T04:29:59.405 INFO:tasks.rados.rados.0.smithi085.stdout:465: writing smithi08575397-21 from 1810907 to 2575965 tid 2 2021-02-10T04:29:59.410 INFO:tasks.rados.rados.0.smithi085.stdout:465: writing smithi08575397-21 from 3067461 to 3243751 tid 3 2021-02-10T04:29:59.410 INFO:tasks.rados.rados.0.smithi085.stdout:462: finishing rollback tid 1 to smithi08575397-26 2021-02-10T04:29:59.410 INFO:tasks.rados.rados.0.smithi085.stdout:462: finishing rollback tid 2 to smithi08575397-26 2021-02-10T04:29:59.410 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 28 (ObjNum 131 snap 24 seq_num 131) dirty exists 2021-02-10T04:29:59.411 INFO:tasks.rados.rados.0.smithi085.stdout:461: finishing write tid 1 to smithi08575397-3 2021-02-10T04:29:59.411 INFO:tasks.rados.rados.0.smithi085.stdout:462: done (10 left) 2021-02-10T04:29:59.411 INFO:tasks.rados.rados.0.smithi085.stdout:466: read oid 26 snap 42 2021-02-10T04:29:59.411 INFO:tasks.rados.rados.0.smithi085.stdout:466: expect (ObjNum 131 snap 24 seq_num 131) 2021-02-10T04:29:59.411 INFO:tasks.rados.rados.0.smithi085.stdout:467: write (excl) oid 16 current snap is 43 2021-02-10T04:29:59.411 INFO:tasks.rados.rados.0.smithi085.stdout:467: seq_num 194 ranges {460558=707131,1869389=508716,3094884=165674} 2021-02-10T04:29:59.416 INFO:tasks.rados.rados.0.smithi085.stdout:467: writing smithi08575397-16 from 460558 to 1167689 tid 1 2021-02-10T04:29:59.424 INFO:tasks.rados.rados.0.smithi085.stdout:467: writing smithi08575397-16 from 1869389 to 2378105 tid 2 2021-02-10T04:29:59.429 INFO:tasks.rados.rados.0.smithi085.stdout:467: writing smithi08575397-16 from 3094884 to 3260558 tid 3 2021-02-10T04:29:59.430 INFO:tasks.rados.rados.0.smithi085.stdout:468: delete oid 46 current snap is 43 2021-02-10T04:29:59.430 INFO:tasks.rados.rados.0.smithi085.stdout:461: finishing write tid 2 to smithi08575397-3 2021-02-10T04:29:59.430 INFO:tasks.rados.rados.0.smithi085.stdout:461: finishing write tid 3 to smithi08575397-3 2021-02-10T04:29:59.430 INFO:tasks.rados.rados.0.smithi085.stdout:461: finishing write tid 4 to smithi08575397-3 2021-02-10T04:29:59.430 INFO:tasks.rados.rados.0.smithi085.stdout:461: finishing write tid 5 to smithi08575397-3 2021-02-10T04:29:59.431 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 91 (ObjNum 191 snap 43 seq_num 191) dirty exists 2021-02-10T04:29:59.431 INFO:tasks.rados.rados.0.smithi085.stdout:461: left oid 3 (ObjNum 191 snap 43 seq_num 191) 2021-02-10T04:29:59.431 INFO:tasks.rados.rados.0.smithi085.stdout:463: finishing write tid 1 to smithi08575397-6 2021-02-10T04:29:59.431 INFO:tasks.rados.rados.0.smithi085.stdout:463: finishing write tid 2 to smithi08575397-6 2021-02-10T04:29:59.431 INFO:tasks.rados.rados.0.smithi085.stdout:463: finishing write tid 3 to smithi08575397-6 2021-02-10T04:29:59.431 INFO:tasks.rados.rados.0.smithi085.stdout:463: finishing write tid 4 to smithi08575397-6 2021-02-10T04:29:59.431 INFO:tasks.rados.rados.0.smithi085.stdout:463: finishing write tid 5 to smithi08575397-6 2021-02-10T04:29:59.431 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 150 (ObjNum 192 snap 43 seq_num 192) dirty exists 2021-02-10T04:29:59.431 INFO:tasks.rados.rados.0.smithi085.stdout:463: left oid 6 (ObjNum 192 snap 43 seq_num 192) 2021-02-10T04:29:59.431 INFO:tasks.rados.rados.0.smithi085.stdout:465: finishing write tid 1 to smithi08575397-21 2021-02-10T04:29:59.431 INFO:tasks.rados.rados.0.smithi085.stdout:465: finishing write tid 2 to smithi08575397-21 2021-02-10T04:29:59.431 INFO:tasks.rados.rados.0.smithi085.stdout:465: finishing write tid 3 to smithi08575397-21 2021-02-10T04:29:59.431 INFO:tasks.rados.rados.0.smithi085.stdout:466: expect (ObjNum 131 snap 24 seq_num 131) 2021-02-10T04:29:59.441 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:29:59 smithi085 conmon[45443]: debug 2021-02-10T04:29:59.064+0000 7f7140d73700 -1 osd.1 182 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:29:59.441 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:29:59 smithi085 conmon[52836]: debug 2021-02-10T04:29:59.441 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:29:59 smithi085 conmon[52836]: 2021-02-10T04:29:59.087+0000 7fba707af700 -1 osd.3 pg_epoch: 182 pg[7.3( v 181'141 (0'0,181'141] local-lis/les=93/94 n=7 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [3,2,4] r=0 lpr=93 crt=181'141 lcod 179'139 mlcod 179'139 active+clean+snaptrim trimq=[4e~1,53~1] ps=[3d~2,40~1,42~2,46~1,4a~1,4f~1,51~1]] removing snap head 2021-02-10T04:29:59.442 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:29:59 smithi085 conmon[52836]: 2021-02-10T04:29:59.442 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:29:59 smithi085 conmon[52836]: debug 2021-02-10T04:29:59.088+0000 7fba707af700 -1 osd.3 pg_epoch: 182 pg[7.3( v 182'143 (0'0,182'143] local-lis/les=93/94 n=5 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [3,2,4] r=0 lpr=93 luod=181'141 crt=182'143 lcod 179'139 mlcod 179'139 active+clean+snaptrim trimq=[4e~1,53~1] ps=[3d~2,40~1,42~2,46~1,4a~1,4f~1,51~1]] removing snap head 2021-02-10T04:29:59.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:59 smithi085 conmon[29319]: audit 2021-02-10T04:29:59.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:59 smithi085 conmon[29319]: 2021-02-10T04:29:57. 2021-02-10T04:29:59.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:59 smithi085 conmon[29319]: 755014+0000 mgr.y (mgr.14140) 385 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:29:59.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:59 smithi085 conmon[29319]: cluster 2021-02-10T04:29:58.011443+0000 mgr.y 2021-02-10T04:29:59.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:59 smithi085 conmon[29319]: (mgr.14140) 386 : cluster [DBG] pgmap v421: 168 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 159 active+clean; 122 MiB data, 389 MiB used, 715 GiB / 715 GiB avail; 5.2 MiB/s rd, 2.4 MiB/s wr, 13 op/s 2021-02-10T04:29:59.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:59 smithi085 conmon[29319]: cluster 2021-02-10T04:29 2021-02-10T04:29:59.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:59 smithi085 conmon[29319]: :58.041083+0000 osd.1 (osd.1) 72 : cluster [WRN] 2021-02-10T04:29:59.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:59 smithi085 conmon[29319]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:29:59.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:59 smithi085 conmon[29319]: cluster 2021-02-10T04:29:59.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:59 smithi085 conmon[29319]: 2021-02-10T04:29:58.041094+0000 osd.1 (osd.1) 73 : cluster [WRN] 2021-02-10T04:29:59.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:29:59 smithi085 conmon[29319]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:29:59.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:59 smithi085 conmon[35961]: audit 2021-02-10T04:29:57.755014+0000 mgr.y (mgr.14140) 385 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:29:59.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:59 smithi085 conmon[35961]: cluster 2021-02-10T04:29:58.011443+0000 mgr.y (mgr.14140) 386 : cluster [DBG] pgmap v421: 168 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 159 active+clean; 122 MiB data, 389 MiB used, 715 GiB / 715 GiB avail; 5.2 MiB/s rd, 2.4 MiB/s wr, 13 op/s 2021-02-10T04:29:59.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:59 smithi085 conmon[35961]: cluster 2021-02-10T04:29:58.041083+0000 osd.1 (osd.1) 72 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:29:59.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:29:59 smithi085 conmon[35961]: cluster 2021-02-10T04:29:58.041094+0000 osd.1 (osd.1) 73 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:29:59.542 INFO:tasks.rados.rados.0.smithi085.stdout:465: finishing write tid 4 to smithi08575397-21 2021-02-10T04:29:59.542 INFO:tasks.rados.rados.0.smithi085.stdout:465: finishing write tid 5 to smithi08575397-21 2021-02-10T04:29:59.542 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 206 (ObjNum 193 snap 43 seq_num 193) dirty exists 2021-02-10T04:29:59.542 INFO:tasks.rados.rados.0.smithi085.stdout:465: left oid 21 (ObjNum 193 snap 43 seq_num 193) 2021-02-10T04:29:59.542 INFO:tasks.rados.rados.0.smithi085.stdout:467: finishing write tid 1 to smithi08575397-16 2021-02-10T04:29:59.542 INFO:tasks.rados.rados.0.smithi085.stdout:467: finishing write tid 2 to smithi08575397-16 2021-02-10T04:29:59.542 INFO:tasks.rados.rados.0.smithi085.stdout:461: done (12 left) 2021-02-10T04:29:59.542 INFO:tasks.rados.rados.0.smithi085.stdout:463: done (11 left) 2021-02-10T04:29:59.543 INFO:tasks.rados.rados.0.smithi085.stdout:464: done (10 left) 2021-02-10T04:29:59.543 INFO:tasks.rados.rados.0.smithi085.stdout:465: done (9 left) 2021-02-10T04:29:59.543 INFO:tasks.rados.rados.0.smithi085.stdout:466: done (8 left) 2021-02-10T04:29:59.543 INFO:tasks.rados.rados.0.smithi085.stdout:468: done (7 left) 2021-02-10T04:29:59.543 INFO:tasks.rados.rados.0.smithi085.stdout:469: write oid 1 current snap is 43 2021-02-10T04:29:59.543 INFO:tasks.rados.rados.0.smithi085.stdout:467: finishing write tid 3 to smithi08575397-16 2021-02-10T04:29:59.543 INFO:tasks.rados.rados.0.smithi085.stdout:467: finishing write tid 4 to smithi08575397-16 2021-02-10T04:29:59.543 INFO:tasks.rados.rados.0.smithi085.stdout:467: finishing write tid 5 to smithi08575397-16 2021-02-10T04:29:59.543 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 210 (ObjNum 194 snap 43 seq_num 194) dirty exists 2021-02-10T04:29:59.543 INFO:tasks.rados.rados.0.smithi085.stdout:467: left oid 16 (ObjNum 194 snap 43 seq_num 194) 2021-02-10T04:29:59.543 INFO:tasks.rados.rados.0.smithi085.stdout:469: seq_num 196 ranges {494172=457629,1586353=512385,2749730=544442} 2021-02-10T04:29:59.547 INFO:tasks.rados.rados.0.smithi085.stdout:469: writing smithi08575397-1 from 494172 to 951801 tid 1 2021-02-10T04:29:59.554 INFO:tasks.rados.rados.0.smithi085.stdout:469: writing smithi08575397-1 from 1586353 to 2098738 tid 2 2021-02-10T04:29:59.561 INFO:tasks.rados.rados.0.smithi085.stdout:469: writing smithi08575397-1 from 2749730 to 3294172 tid 3 2021-02-10T04:29:59.562 INFO:tasks.rados.rados.0.smithi085.stdout:467: done (7 left) 2021-02-10T04:29:59.562 INFO:tasks.rados.rados.0.smithi085.stdout:470: copy_from oid 44 from oid 38 current snap is 43 2021-02-10T04:29:59.562 INFO:tasks.rados.rados.0.smithi085.stdout:469: finishing write tid 1 to smithi08575397-1 2021-02-10T04:29:59.562 INFO:tasks.rados.rados.0.smithi085.stdout:469: finishing write tid 2 to smithi08575397-1 2021-02-10T04:29:59.562 INFO:tasks.rados.rados.0.smithi085.stdout:471: rollback oid 21 current snap is 43 2021-02-10T04:29:59.562 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 21 to 42 2021-02-10T04:29:59.562 INFO:tasks.rados.rados.0.smithi085.stdout:472: read oid 42 snap -1 2021-02-10T04:29:59.562 INFO:tasks.rados.rados.0.smithi085.stdout:472: expect (ObjNum 184 snap 40 seq_num 184) 2021-02-10T04:29:59.562 INFO:tasks.rados.rados.0.smithi085.stdout:473: write (excl) oid 32 current snap is 43 2021-02-10T04:29:59.562 INFO:tasks.rados.rados.0.smithi085.stdout:473: seq_num 197 ranges {510979=532878,1528482=772396,3060800=250179} 2021-02-10T04:29:59.567 INFO:tasks.rados.rados.0.smithi085.stdout:473: writing smithi08575397-32 from 510979 to 1043857 tid 1 2021-02-10T04:29:59.575 INFO:tasks.rados.rados.0.smithi085.stdout:473: writing smithi08575397-32 from 1528482 to 2300878 tid 2 2021-02-10T04:29:59.581 INFO:tasks.rados.rados.0.smithi085.stdout:473: writing smithi08575397-32 from 3060800 to 3310979 tid 3 2021-02-10T04:29:59.581 INFO:tasks.rados.rados.0.smithi085.stdout:474: write oid 50 current snap is 43 2021-02-10T04:29:59.581 INFO:tasks.rados.rados.0.smithi085.stdout:474: seq_num 198 ranges {527786=724480,1703317=516054,2804576=435035} 2021-02-10T04:29:59.588 INFO:tasks.rados.rados.0.smithi085.stdout:474: writing smithi08575397-50 from 527786 to 1252266 tid 1 2021-02-10T04:29:59.593 INFO:tasks.rados.rados.0.smithi085.stdout:474: writing smithi08575397-50 from 1703317 to 2219371 tid 2 2021-02-10T04:29:59.600 INFO:tasks.rados.rados.0.smithi085.stdout:474: writing smithi08575397-50 from 2804576 to 3239611 tid 3 2021-02-10T04:29:59.600 INFO:tasks.rados.rados.0.smithi085.stdout:475: rollback oid 14 current snap is 43 2021-02-10T04:29:59.600 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 14 to 42 2021-02-10T04:29:59.601 INFO:tasks.rados.rados.0.smithi085.stdout:472: expect (ObjNum 184 snap 40 seq_num 184) 2021-02-10T04:29:59.725 INFO:tasks.rados.rados.0.smithi085.stdout:472: done (12 left) 2021-02-10T04:29:59.726 INFO:tasks.rados.rados.0.smithi085.stdout:476: write oid 2 current snap is 43 2021-02-10T04:29:59.726 INFO:tasks.rados.rados.0.smithi085.stdout:469: finishing write tid 3 to smithi08575397-1 2021-02-10T04:29:59.726 INFO:tasks.rados.rados.0.smithi085.stdout:469: finishing write tid 4 to smithi08575397-1 2021-02-10T04:29:59.726 INFO:tasks.rados.rados.0.smithi085.stdout:469: finishing write tid 5 to smithi08575397-1 2021-02-10T04:29:59.726 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 83 (ObjNum 196 snap 43 seq_num 196) dirty exists 2021-02-10T04:29:59.726 INFO:tasks.rados.rados.0.smithi085.stdout:469: left oid 1 (ObjNum 196 snap 43 seq_num 196) 2021-02-10T04:29:59.726 INFO:tasks.rados.rados.0.smithi085.stdout:473: finishing write tid 1 to smithi08575397-32 2021-02-10T04:29:59.726 INFO:tasks.rados.rados.0.smithi085.stdout:473: finishing write tid 2 to smithi08575397-32 2021-02-10T04:29:59.726 INFO:tasks.rados.rados.0.smithi085.stdout:476: seq_num 199 ranges {544593=799729,1761799=776065,2948352=396241} 2021-02-10T04:29:59.734 INFO:tasks.rados.rados.0.smithi085.stdout:476: writing smithi08575397-2 from 544593 to 1344322 tid 1 2021-02-10T04:29:59.742 INFO:tasks.rados.rados.0.smithi085.stdout:476: writing smithi08575397-2 from 1761799 to 2537864 tid 2 2021-02-10T04:29:59.748 INFO:tasks.rados.rados.0.smithi085.stdout:476: writing smithi08575397-2 from 2948352 to 3344593 tid 3 2021-02-10T04:29:59.748 INFO:tasks.rados.rados.0.smithi085.stdout:469: done (12 left) 2021-02-10T04:29:59.748 INFO:tasks.rados.rados.0.smithi085.stdout:477: read oid 13 snap -1 2021-02-10T04:29:59.748 INFO:tasks.rados.rados.0.smithi085.stdout:477: expect (ObjNum 161 snap 36 seq_num 161) 2021-02-10T04:29:59.748 INFO:tasks.rados.rados.0.smithi085.stdout:478: read oid 34 snap 42 2021-02-10T04:29:59.748 INFO:tasks.rados.rados.0.smithi085.stdout:478: expect (ObjNum 165 snap 36 seq_num 165) 2021-02-10T04:29:59.749 INFO:tasks.rados.rados.0.smithi085.stdout:479: write (excl) oid 7 current snap is 43 2021-02-10T04:29:59.749 INFO:tasks.rados.rados.0.smithi085.stdout:473: finishing write tid 3 to smithi08575397-32 2021-02-10T04:29:59.749 INFO:tasks.rados.rados.0.smithi085.stdout:473: finishing write tid 4 to smithi08575397-32 2021-02-10T04:29:59.749 INFO:tasks.rados.rados.0.smithi085.stdout:473: finishing write tid 5 to smithi08575397-32 2021-02-10T04:29:59.749 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 32 v 93 (ObjNum 197 snap 43 seq_num 197) dirty exists 2021-02-10T04:29:59.749 INFO:tasks.rados.rados.0.smithi085.stdout:473: left oid 32 (ObjNum 197 snap 43 seq_num 197) 2021-02-10T04:29:59.749 INFO:tasks.rados.rados.0.smithi085.stdout:474: finishing write tid 1 to smithi08575397-50 2021-02-10T04:29:59.749 INFO:tasks.rados.rados.0.smithi085.stdout:474: finishing write tid 2 to smithi08575397-50 2021-02-10T04:29:59.749 INFO:tasks.rados.rados.0.smithi085.stdout:475: finishing rollback tid 0 to smithi08575397-14 2021-02-10T04:29:59.749 INFO:tasks.rados.rados.0.smithi085.stdout:475: finishing rollback tid 1 to smithi08575397-14 2021-02-10T04:29:59.749 INFO:tasks.rados.rados.0.smithi085.stdout:475: finishing rollback tid 2 to smithi08575397-14 2021-02-10T04:29:59.749 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 132 (ObjNum 142 snap 29 seq_num 142) dirty exists 2021-02-10T04:29:59.749 INFO:tasks.rados.rados.0.smithi085.stdout:474: finishing write tid 3 to smithi08575397-50 2021-02-10T04:29:59.750 INFO:tasks.rados.rados.0.smithi085.stdout:474: finishing write tid 4 to smithi08575397-50 2021-02-10T04:29:59.750 INFO:tasks.rados.rados.0.smithi085.stdout:474: finishing write tid 5 to smithi08575397-50 2021-02-10T04:29:59.750 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 88 (ObjNum 198 snap 43 seq_num 198) dirty exists 2021-02-10T04:29:59.750 INFO:tasks.rados.rados.0.smithi085.stdout:474: left oid 50 (ObjNum 198 snap 43 seq_num 198) 2021-02-10T04:29:59.750 INFO:tasks.rados.rados.0.smithi085.stdout:471: finishing rollback tid 0 to smithi08575397-21 2021-02-10T04:29:59.750 INFO:tasks.rados.rados.0.smithi085.stdout:471: finishing rollback tid 1 to smithi08575397-21 2021-02-10T04:29:59.750 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 212 (ObjNum 71101512 snap 22030 seq_num 2) dirty dne 2021-02-10T04:29:59.750 INFO:tasks.rados.rados.0.smithi085.stdout:470: finishing copy_from to smithi08575397-44 2021-02-10T04:29:59.750 INFO:tasks.rados.rados.0.smithi085.stdout:470: got expected ENOENT (src dne) 2021-02-10T04:29:59.750 INFO:tasks.rados.rados.0.smithi085.stdout:470: finishing copy_from racing read to smithi08575397-44 2021-02-10T04:29:59.750 INFO:tasks.rados.rados.0.smithi085.stdout:476: finishing write tid 1 to smithi08575397-2 2021-02-10T04:29:59.750 INFO:tasks.rados.rados.0.smithi085.stdout:476: finishing write tid 2 to smithi08575397-2 2021-02-10T04:29:59.751 INFO:tasks.rados.rados.0.smithi085.stdout:479: seq_num 200 ranges {561400=474978,1820281=519723,2859422=457579} 2021-02-10T04:29:59.754 INFO:tasks.rados.rados.0.smithi085.stdout:479: writing smithi08575397-7 from 561400 to 1036378 tid 1 2021-02-10T04:29:59.764 INFO:tasks.rados.rados.0.smithi085.stdout:479: writing smithi08575397-7 from 1820281 to 2340004 tid 2 2021-02-10T04:29:59.771 INFO:tasks.rados.rados.0.smithi085.stdout:479: writing smithi08575397-7 from 2859422 to 3317001 tid 3 2021-02-10T04:29:59.771 INFO:tasks.rados.rados.0.smithi085.stdout:476: finishing write tid 3 to smithi08575397-2 2021-02-10T04:29:59.771 INFO:tasks.rados.rados.0.smithi085.stdout:476: finishing write tid 4 to smithi08575397-2 2021-02-10T04:29:59.772 INFO:tasks.rados.rados.0.smithi085.stdout:476: finishing write tid 5 to smithi08575397-2 2021-02-10T04:29:59.772 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 70 (ObjNum 199 snap 43 seq_num 199) dirty exists 2021-02-10T04:29:59.772 INFO:tasks.rados.rados.0.smithi085.stdout:476: left oid 2 (ObjNum 199 snap 43 seq_num 199) 2021-02-10T04:29:59.772 INFO:tasks.rados.rados.0.smithi085.stdout:477: expect (ObjNum 161 snap 36 seq_num 161) 2021-02-10T04:29:59.932 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:30:00.000 INFO:tasks.rados.rados.0.smithi085.stdout:479: finishing write tid 1 to smithi08575397-7 2021-02-10T04:30:00.000 INFO:tasks.rados.rados.0.smithi085.stdout:479: finishing write tid 2 to smithi08575397-7 2021-02-10T04:30:00.001 INFO:tasks.rados.rados.0.smithi085.stdout:478: expect (ObjNum 165 snap 36 seq_num 165) 2021-02-10T04:30:00.066 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:30:00.083 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:29:59 smithi085 conmon[49106]: debug 2021-02-10T04:30:00.083 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:29:59 smithi085 conmon[49106]: 2021-02-10T04:29:59.773+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:00.084 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:29:59 smithi085 conmon[49106]: 2021-02-10T04:30:00.084 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: cluster 2021-02-10T04:30:00.125 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:00.144 INFO:tasks.rados.rados.0.smithi085.stdout:470: done (14 left) 2021-02-10T04:30:00.144 INFO:tasks.rados.rados.0.smithi085.stdout:471: done (13 left) 2021-02-10T04:30:00.144 INFO:tasks.rados.rados.0.smithi085.stdout:473: done (12 left) 2021-02-10T04:30:00.144 INFO:tasks.rados.rados.0.smithi085.stdout:474: done (11 left) 2021-02-10T04:30:00.145 INFO:tasks.rados.rados.0.smithi085.stdout:475: done (10 left) 2021-02-10T04:30:00.145 INFO:tasks.rados.rados.0.smithi085.stdout:476: done (9 left) 2021-02-10T04:30:00.145 INFO:tasks.rados.rados.0.smithi085.stdout:477: done (8 left) 2021-02-10T04:30:00.145 INFO:tasks.rados.rados.0.smithi085.stdout:478: done (7 left) 2021-02-10T04:30:00.145 INFO:tasks.rados.rados.0.smithi085.stdout:480: snap_create 2021-02-10T04:30:00.145 INFO:tasks.rados.rados.0.smithi085.stdout:479: finishing write tid 3 to smithi08575397-7 2021-02-10T04:30:00.145 INFO:tasks.rados.rados.0.smithi085.stdout:479: finishing write tid 4 to smithi08575397-7 2021-02-10T04:30:00.145 INFO:tasks.rados.rados.0.smithi085.stdout:479: finishing write tid 5 to smithi08575397-7 2021-02-10T04:30:00.145 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 96 (ObjNum 200 snap 43 seq_num 200) dirty exists 2021-02-10T04:30:00.145 INFO:tasks.rados.rados.0.smithi085.stdout:479: left oid 7 (ObjNum 200 snap 43 seq_num 200) 2021-02-10T04:30:00.235 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 3 1 2021-02-10T04:30:00.312 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:30:00.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:00 smithi164 conmon[29288]: cluster 2021-02-10T04:30:00.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:00 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:30:00.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:00 smithi164 conmon[29288]: :29:59.065503+0000 osd.1 (osd.1) 74 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:30:00.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:00 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:30:00.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:00 smithi164 conmon[29288]: 02-10T04:29:59.065511+0000 osd.1 ( 2021-02-10T04:30:00.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:00 smithi164 conmon[29288]: osd.1) 75 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:30:00.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:00 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:30:00.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:00 smithi164 conmon[29288]: 10T04:29:59. 2021-02-10T04:30:00.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:00 smithi164 conmon[29288]: 081194+0000 mon.a (mon.0) 880 : cluster [DBG] 2021-02-10T04:30:00.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:00 smithi164 conmon[29288]: osdmap e183: 8 total, 8 up, 8 in 2021-02-10T04:30:00.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:00 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:30:00.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:00 smithi164 conmon[29288]: -02-10T04:30:00.000120+0000 mon.a (mon.0 2021-02-10T04:30:00.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:00 smithi164 conmon[29288]: ) 881 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 2 slow ops, oldest one blocked for 42 sec, osd.1 has slow ops 2021-02-10T04:30:00.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:00 smithi164 conmon[29288]: cluster 2021-02-10T04:30: 2021-02-10T04:30:00.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:00 smithi164 conmon[29288]: 00.000157+0000 mon.a (mon.0) 882 : cluster 2021-02-10T04:30:00.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:00 smithi164 conmon[29288]: [WRN] [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2021-02-10T04:30:00.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:00 smithi164 conmon[29288]: cluster 2021-02-10T04:30:00. 2021-02-10T04:30:00.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:00 smithi164 conmon[29288]: 000172+0000 mon.a (mon.0) 883 : cluster [WRN] [WRN] SLOW_OPS: 2 slow ops, oldest one blocked for 42 sec, osd.1 has slow ops 2021-02-10T04:30:00.420 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:30:00.445 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[45443]: debug 2021-02-10T04:30:00.089+0000 7f7140d73700 -1 osd.1 183 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:30:00.445 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[41540]: debug 2021-02-10T04:30:00.260+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:00.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[35961]: cluster 2021-02-10T04:30:00.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[35961]: 2021-02-10T04:30:00.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[35961]: 2021 2021-02-10T04:30:00.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[35961]: - 2021-02-10T04:30:00.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[35961]: 02 2021-02-10T04:30:00.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[35961]: - 2021-02-10T04:30:00.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[35961]: 10 2021-02-10T04:30:00.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[35961]: T 2021-02-10T04:30:00.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[35961]: 04 2021-02-10T04:30:00.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[35961]: : 2021-02-10T04:30:00.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[35961]: 29 2021-02-10T04:30:00.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[35961]: : 2021-02-10T04:30:00.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[35961]: 59 2021-02-10T04:30:00.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[35961]: .065503+0000 osd.1 (osd.1) 74 2021-02-10T04:30:00.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:30:00.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[35961]: cluster 2021-02-10T04:30:00.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[35961]: 2021-02-10T04:29:59.065511+0000 osd.1 (osd.1) 75 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:30:00.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:30:00.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[35961]: :29:59.081194+0000 mon.a (mon.0 2021-02-10T04:30:00.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[35961]: ) 880 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2021-02-10T04:30:00.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:30:00.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[35961]: -02-10T04:30:00.000120+0000 2021-02-10T04:30:00.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[35961]: mon.a (mon.0) 881 : cluster [WRN] 2021-02-10T04:30:00.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[35961]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 2 slow ops, oldest one blocked for 42 sec, osd.1 has slow ops 2021-02-10T04:30:00.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:30:00.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[35961]: T04:30:00.000157+0000 mon.a (mon 2021-02-10T04:30:00.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[35961]: .0) 882 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2021-02-10T04:30:00.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:30:00.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[35961]: :30:00.000172+0000 mon.a (mon.0) 2021-02-10T04:30:00.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[35961]: 883 : cluster [WRN] [WRN] SLOW_OPS: 2 slow ops, oldest one blocked for 42 sec, osd.1 has slow ops 2021-02-10T04:30:00.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: 2021-02-10T04:30:00.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: 2021 2021-02-10T04:30:00.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: - 2021-02-10T04:30:00.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: 02 2021-02-10T04:30:00.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: - 2021-02-10T04:30:00.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: 10 2021-02-10T04:30:00.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: T 2021-02-10T04:30:00.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: 04 2021-02-10T04:30:00.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: : 2021-02-10T04:30:00.454 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: 29 2021-02-10T04:30:00.454 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: : 2021-02-10T04:30:00.454 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: 59 2021-02-10T04:30:00.454 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: . 2021-02-10T04:30:00.454 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: 065503 2021-02-10T04:30:00.454 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: +0000 2021-02-10T04:30:00.455 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: 2021-02-10T04:30:00.455 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: osd.1 2021-02-10T04:30:00.455 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: ( 2021-02-10T04:30:00.455 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: osd 2021-02-10T04:30:00.455 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: . 2021-02-10T04:30:00.456 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: 1 2021-02-10T04:30:00.456 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: ) 2021-02-10T04:30:00.456 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: 74 2021-02-10T04:30:00.456 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: : 2021-02-10T04:30:00.456 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: cluster 2021-02-10T04:30:00.456 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: 2021-02-10T04:30:00.457 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: [WRN] 2021-02-10T04:30:00.457 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: 2021-02-10T04:30:00.457 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:30:00.457 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: 2021-02-10T04:30:00.457 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: cluster 2021-02-10T04:30:00.457 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: 2021-02-10T04:30:00.458 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: 2021 2021-02-10T04:30:00.458 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: - 2021-02-10T04:30:00.458 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: 02 2021-02-10T04:30:00.458 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: - 2021-02-10T04:30:00.458 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: 10 2021-02-10T04:30:00.459 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: T 2021-02-10T04:30:00.459 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: 04 2021-02-10T04:30:00.459 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: : 2021-02-10T04:30:00.459 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: 29 2021-02-10T04:30:00.459 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: : 2021-02-10T04:30:00.460 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: 59 2021-02-10T04:30:00.460 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: . 2021-02-10T04:30:00.460 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: 065511+0000 osd.1 (osd 2021-02-10T04:30:00.460 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: .1) 75 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:30:00.460 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:30:00.460 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: 02-10T04:29:59.081194+0000 2021-02-10T04:30:00.461 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: mon.a (mon.0) 880 : cluster [DBG] 2021-02-10T04:30:00.461 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: osdmap e183: 8 total, 8 up, 8 in 2021-02-10T04:30:00.461 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:30:00.461 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: 30:00.000120+0000 mon.a (mon.0 2021-02-10T04:30:00.461 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: ) 881 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 2 slow ops, oldest one blocked for 42 sec, osd.1 has slow ops 2021-02-10T04:30:00.461 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: cluster 2021-02-10T04:30:00.000157 2021-02-10T04:30:00.462 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: +0000 mon.a (mon.0) 882 : cluster [WRN] 2021-02-10T04:30:00.462 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2021-02-10T04:30:00.462 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: cluster 2021-02-10T04:30 2021-02-10T04:30:00.462 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: :00.000172+0000 mon.a (mon.0) 883 2021-02-10T04:30:00.462 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[29319]: : cluster [WRN] [WRN] SLOW_OPS: 2 slow ops, oldest one blocked for 42 sec, osd.1 has slow ops 2021-02-10T04:30:00.463 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:00.600 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:30:00.688 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:30:00.794 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:30:00.804 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:30:00.831 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:00.833 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:30:00.866 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:30:00.892 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:30:00 smithi085 conmon[52836]: debug 2021-02-10T04:30:00.671+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:00.895 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:30:00.928 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:30:00.963 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:30:00.991 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:30:01.020 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:30:01.021 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:30:01.021 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:30:01.021 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:30:01.021 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:30:01.022 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:30:01.022 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:30:01.022 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:30:01.022 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:30:01.086 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:01 smithi164 conmon[29288]: cluster 2021-02-10T04:29:59.926337+0000 osd.5 (osd.5) 48 : 2021-02-10T04:30:01.086 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:01 smithi164 conmon[29288]: cluster [DBG] purged_snaps scrub starts 2021-02-10T04:30:01.086 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:01 smithi164 conmon[29288]: cluster 2021-02-10T04:29:59.926469+0000 osd.5 (osd.5) 49 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:30:01.086 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:01 smithi164 conmon[29288]: cluster 2021-02-10T04:30:01.086 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:01 smithi164 conmon[29288]: 2021-02-10T04:30:00.011946+0000 mgr.y (mgr.14140) 387 : cluster [DBG] 2021-02-10T04:30:01.086 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:01 smithi164 conmon[29288]: pgmap v424: 168 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 162 active+clean; 124 MiB data, 393 MiB used, 715 GiB / 715 GiB avail; 5.7 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2021-02-10T04:30:01.086 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:01 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:30:01.087 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:01 smithi164 conmon[29288]: :30:00.077684+0000 mon.a (mon.0) 884 2021-02-10T04:30:01.087 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:01 smithi164 conmon[29288]: : cluster [WRN] Health check update: 2 slow ops, oldest one blocked for 46 sec, osd.1 has slow ops (SLOW_OPS) 2021-02-10T04:30:01.087 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:01 smithi164 conmon[29288]: cluster 2021-02-10T04:30:01.087 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:01 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:30:01.087 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:01 smithi164 conmon[29288]: 30:00.089146+0000 mon.a (mon.0) 885 : cluster 2021-02-10T04:30:01.087 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:01 smithi164 conmon[29288]: [DBG] osdmap e184: 8 total, 8 up, 8 in 2021-02-10T04:30:01.087 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:01 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:30:01.087 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:01 smithi164 conmon[29288]: 02-10T04:30:00.089786+0000 2021-02-10T04:30:01.087 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:01 smithi164 conmon[29288]: osd.1 (osd.1) 76 : cluster 2021-02-10T04:30:01.087 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:01 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:30:01.087 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:01 smithi164 conmon[29288]: cluster 2021-02-10T04:30: 2021-02-10T04:30:01.087 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:01 smithi164 conmon[29288]: 00.089796+0000 osd.1 (osd.1) 77 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:30:01.087 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:01 smithi164 conmon[29288]: audit 2021-02-10T04:30:00.772792+0000 mon.b 2021-02-10T04:30:01.088 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:01 smithi164 conmon[29288]: (mon.2) 27 : audit [DBG] from='client.? 172.21.15.85:0/2722444486' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:30:01.088 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:30:00 smithi164 conmon[44544]: debug 2021-02-10T04:30:00.906+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:01.097 INFO:tasks.rados.rados.0.smithi085.stdout:479: done (7 left) 2021-02-10T04:30:01.097 INFO:tasks.rados.rados.0.smithi085.stdout:480: done (6 left) 2021-02-10T04:30:01.098 INFO:tasks.rados.rados.0.smithi085.stdout:481: delete oid 5 current snap is 44 2021-02-10T04:30:01.100 INFO:tasks.rados.rados.0.smithi085.stdout:481: done (6 left) 2021-02-10T04:30:01.100 INFO:tasks.rados.rados.0.smithi085.stdout:482: write oid 1 current snap is 44 2021-02-10T04:30:01.100 INFO:tasks.rados.rados.0.smithi085.stdout:482: seq_num 202 ranges {595014=625476,1820892=523392,2797915=596476} 2021-02-10T04:30:01.108 INFO:tasks.rados.rados.0.smithi085.stdout:482: writing smithi08575397-1 from 595014 to 1220490 tid 1 2021-02-10T04:30:01.115 INFO:tasks.rados.rados.0.smithi085.stdout:482: writing smithi08575397-1 from 1820892 to 2344284 tid 2 2021-02-10T04:30:01.122 INFO:tasks.rados.rados.0.smithi085.stdout:482: writing smithi08575397-1 from 2797915 to 3394391 tid 3 2021-02-10T04:30:01.122 INFO:tasks.rados.rados.0.smithi085.stdout:482: finishing write tid 1 to smithi08575397-1 2021-02-10T04:30:01.122 INFO:tasks.rados.rados.0.smithi085.stdout:482: finishing write tid 2 to smithi08575397-1 2021-02-10T04:30:01.122 INFO:tasks.rados.rados.0.smithi085.stdout:483: read oid 42 snap -1 2021-02-10T04:30:01.122 INFO:tasks.rados.rados.0.smithi085.stdout:483: expect (ObjNum 184 snap 40 seq_num 184) 2021-02-10T04:30:01.122 INFO:tasks.rados.rados.0.smithi085.stdout:484: read oid 36 snap -1 2021-02-10T04:30:01.123 INFO:tasks.rados.rados.0.smithi085.stdout:484: expect (ObjNum 125 snap 23 seq_num 125) 2021-02-10T04:30:01.123 INFO:tasks.rados.rados.0.smithi085.stdout:485: read oid 37 snap -1 2021-02-10T04:30:01.123 INFO:tasks.rados.rados.0.smithi085.stdout:485: expect (ObjNum 25 snap 0 seq_num 25) 2021-02-10T04:30:01.123 INFO:tasks.rados.rados.0.smithi085.stdout:486: write oid 3 current snap is 44 2021-02-10T04:30:01.123 INFO:tasks.rados.rados.0.smithi085.stdout:486: seq_num 203 ranges {611821=700725,1879374=783403,3341691=70130} 2021-02-10T04:30:01.131 INFO:tasks.rados.rados.0.smithi085.stdout:486: writing smithi08575397-3 from 611821 to 1312546 tid 1 2021-02-10T04:30:01.139 INFO:tasks.rados.rados.0.smithi085.stdout:486: writing smithi08575397-3 from 1879374 to 2662777 tid 2 2021-02-10T04:30:01.144 INFO:tasks.rados.rados.0.smithi085.stdout:486: writing smithi08575397-3 from 3341691 to 3411821 tid 3 2021-02-10T04:30:01.144 INFO:tasks.rados.rados.0.smithi085.stdout:487: read oid 16 snap 42 2021-02-10T04:30:01.144 INFO:tasks.rados.rados.0.smithi085.stdout:487: expect deleted 2021-02-10T04:30:01.144 INFO:tasks.rados.rados.0.smithi085.stdout:484: expect (ObjNum 125 snap 23 seq_num 125) 2021-02-10T04:30:01.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[29319]: cluster 2021-02-10T04:29:59.926337+0000 osd.5 (osd 2021-02-10T04:30:01.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[29319]: .5) 48 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:30:01.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[29319]: cluster 2021-02-10T04:29:59.926469+0000 osd.5 (osd.5) 49 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:30:01.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[29319]: cluster 2021-02-10T04:30:00.011946+0000 mgr.y (mgr.14140) 387 : cluster [DBG] pgmap v424: 168 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 162 active+clean; 124 MiB data, 393 MiB used, 715 GiB / 715 GiB avail; 5.7 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2021-02-10T04:30:01.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[29319]: cluster 2021-02-10T04:30:00.077684+0000 mon.a (mon.0) 884 : cluster [WRN] Health check update: 2 slow ops, oldest one blocked for 46 sec, osd.1 has slow ops (SLOW_OPS) 2021-02-10T04:30:01.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[29319]: cluster 2021-02-10T04:30:00.089146+0000 mon.a (mon.0) 885 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2021-02-10T04:30:01.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[29319]: cluster 2021-02-10T04:30:00.089786+0000 osd.1 (osd.1) 76 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:30:01.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[29319]: cluster 2021-02-10T04:30:00.089796+0000 osd.1 (osd.1) 77 : cluster 2021-02-10T04:30:01.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:30:01.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[29319]: audit 2021-02-10T04:30:00.772792+0000 mon.b (mon.2) 27 : audit [DBG] from='client.? 172.21.15.85:0/2722444486' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:30:01.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: cluster 2021-02-10T04:29:59.926337+0000 2021-02-10T04:30:01.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: osd.5 (osd.5) 48 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:30:01.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:30:01.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: 10T04:29:59.926469+0000 osd.5 (osd.5) 49 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:30:01.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: cluster 2021-02-10T04:30:01.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: 2021-02-10T04:30:00.011946+0000 2021-02-10T04:30:01.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: mgr.y (mgr.14140) 387 : cluster [DBG] pgmap v424: 168 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 162 active+clean; 124 MiB data, 393 MiB used, 715 GiB / 715 GiB avail; 5.7 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2021-02-10T04:30:01.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: 2021-02-10T04:30:01.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: cluster 2021-02-10T04:30:01.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: 2021 2021-02-10T04:30:01.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: - 2021-02-10T04:30:01.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: 02-10T04:30 2021-02-10T04:30:01.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: :00.077684+0000 mon.a (mon.0) 884 : cluster [WRN] Health check update: 2 slow ops, oldest one blocked for 46 sec, osd.1 has slow ops (SLOW_OPS) 2021-02-10T04:30:01.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: cluster 2021-02-10T04:30:01.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: 2021-02-10T04:30:01.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: 2021 2021-02-10T04:30:01.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: - 2021-02-10T04:30:01.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: 02 2021-02-10T04:30:01.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: -10T04:30 2021-02-10T04:30:01.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: :00.089146+0000 mon.a (mon.0) 885 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2021-02-10T04:30:01.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: 2021-02-10T04:30:01.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:30:01.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: -02-10T04:30:00.089786+0000 osd.1 2021-02-10T04:30:01.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: (osd.1) 76 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:30:01.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: 2021-02-10T04:30:01.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:30:01.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: :30:00.089796+0000 osd.1 (osd. 2021-02-10T04:30:01.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: 1) 77 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:30:01.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: audit 2021-02-10T04:30:01.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:30:01.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: :30:00.772792+0000 mon.b (mon.2 2021-02-10T04:30:01.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:01 smithi085 conmon[35961]: ) 27 : audit [DBG] from='client.? 172.21.15.85:0/2722444486' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:30:01.262 INFO:tasks.rados.rados.0.smithi085.stdout:484: done (11 left) 2021-02-10T04:30:01.262 INFO:tasks.rados.rados.0.smithi085.stdout:488: read oid 26 snap -1 2021-02-10T04:30:01.262 INFO:tasks.rados.rados.0.smithi085.stdout:488: expect (ObjNum 131 snap 24 seq_num 131) 2021-02-10T04:30:01.262 INFO:tasks.rados.rados.0.smithi085.stdout:489: rollback oid 21 current snap is 44 2021-02-10T04:30:01.262 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 21 to 42 2021-02-10T04:30:01.262 INFO:tasks.rados.rados.0.smithi085.stdout:483: expect (ObjNum 184 snap 40 seq_num 184) 2021-02-10T04:30:01.340 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:30:01.403 INFO:tasks.rados.rados.0.smithi085.stdout:485: expect (ObjNum 25 snap 0 seq_num 25) 2021-02-10T04:30:01.467 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:30:01.495 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:01.512 INFO:tasks.rados.rados.0.smithi085.stdout:483: done (12 left) 2021-02-10T04:30:01.512 INFO:tasks.rados.rados.0.smithi085.stdout:485: done (11 left) 2021-02-10T04:30:01.512 INFO:tasks.rados.rados.0.smithi085.stdout:490: snap_remove snap 43 2021-02-10T04:30:01.688 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:30:01.794 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:30:01.829 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:01.893 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:30:01 smithi164 conmon[37263]: debug 2021-02-10T04:30:01.653+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:02.105 INFO:tasks.rados.rados.0.smithi085.stdout:490: done (11 left) 2021-02-10T04:30:02.106 INFO:tasks.rados.rados.0.smithi085.stdout:491: write (excl) oid 32 current snap is 44 2021-02-10T04:30:02.106 INFO:tasks.rados.rados.0.smithi085.stdout:482: finishing write tid 3 to smithi08575397-1 2021-02-10T04:30:02.106 INFO:tasks.rados.rados.0.smithi085.stdout:482: finishing write tid 4 to smithi08575397-1 2021-02-10T04:30:02.106 INFO:tasks.rados.rados.0.smithi085.stdout:482: finishing write tid 5 to smithi08575397-1 2021-02-10T04:30:02.106 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 93 (ObjNum 202 snap 44 seq_num 202) dirty exists 2021-02-10T04:30:02.106 INFO:tasks.rados.rados.0.smithi085.stdout:482: left oid 1 (ObjNum 202 snap 44 seq_num 202) 2021-02-10T04:30:02.106 INFO:tasks.rados.rados.0.smithi085.stdout:486: finishing write tid 1 to smithi08575397-3 2021-02-10T04:30:02.106 INFO:tasks.rados.rados.0.smithi085.stdout:486: finishing write tid 2 to smithi08575397-3 2021-02-10T04:30:02.106 INFO:tasks.rados.rados.0.smithi085.stdout:486: finishing write tid 3 to smithi08575397-3 2021-02-10T04:30:02.106 INFO:tasks.rados.rados.0.smithi085.stdout:486: finishing write tid 4 to smithi08575397-3 2021-02-10T04:30:02.107 INFO:tasks.rados.rados.0.smithi085.stdout:486: finishing write tid 5 to smithi08575397-3 2021-02-10T04:30:02.107 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 101 (ObjNum 203 snap 44 seq_num 203) dirty exists 2021-02-10T04:30:02.107 INFO:tasks.rados.rados.0.smithi085.stdout:486: left oid 3 (ObjNum 203 snap 44 seq_num 203) 2021-02-10T04:30:02.107 INFO:tasks.rados.rados.0.smithi085.stdout:488: expect (ObjNum 131 snap 24 seq_num 131) 2021-02-10T04:30:02.181 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:30:02.181 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:30:02.211 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:02 smithi164 conmon[29288]: cluster 2021-02-10T04:30:02.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:02 smithi164 conmon[29288]: 2021-02-10T04:30:02.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:02 smithi164 conmon[29288]: 2021 2021-02-10T04:30:02.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:02 smithi164 conmon[29288]: - 2021-02-10T04:30:02.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:02 smithi164 conmon[29288]: 02 2021-02-10T04:30:02.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:02 smithi164 conmon[29288]: - 2021-02-10T04:30:02.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:02 smithi164 conmon[29288]: 10 2021-02-10T04:30:02.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:02 smithi164 conmon[29288]: T 2021-02-10T04:30:02.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:02 smithi164 conmon[29288]: 04 2021-02-10T04:30:02.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:02 smithi164 conmon[29288]: : 2021-02-10T04:30:02.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:02 smithi164 conmon[29288]: 30 2021-02-10T04:30:02.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:02 smithi164 conmon[29288]: : 2021-02-10T04:30:02.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:02 smithi164 conmon[29288]: 01 2021-02-10T04:30:02.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:02 smithi164 conmon[29288]: . 2021-02-10T04:30:02.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:02 smithi164 conmon[29288]: 092194 2021-02-10T04:30:02.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:02 smithi164 conmon[29288]: +0000 2021-02-10T04:30:02.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:02 smithi164 conmon[29288]: 2021-02-10T04:30:02.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:02 smithi164 conmon[29288]: mon.a 2021-02-10T04:30:02.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:02 smithi164 conmon[29288]: (mon.0) 886 : cluster [DBG] 2021-02-10T04:30:02.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:02 smithi164 conmon[29288]: osdmap e185: 8 total, 8 up, 8 in 2021-02-10T04:30:02.236 INFO:tasks.rados.rados.0.smithi085.stdout:489: finishing rollback tid 1 to smithi08575397-21 2021-02-10T04:30:02.236 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 0 (ObjNum 82248880 snap 22030 seq_num 82372672) dirty dne 2021-02-10T04:30:02.236 INFO:tasks.rados.rados.0.smithi085.stdout:491: seq_num 204 ranges {628628=775974,1937856=527061,3252761=175867} 2021-02-10T04:30:02.244 INFO:tasks.rados.rados.0.smithi085.stdout:491: writing smithi08575397-32 from 628628 to 1404602 tid 1 2021-02-10T04:30:02.251 INFO:tasks.rados.rados.0.smithi085.stdout:491: writing smithi08575397-32 from 1937856 to 2464917 tid 2 2021-02-10T04:30:02.257 INFO:tasks.rados.rados.0.smithi085.stdout:491: writing smithi08575397-32 from 3252761 to 3428628 tid 3 2021-02-10T04:30:02.257 INFO:tasks.rados.rados.0.smithi085.stdout:491: finishing write tid 1 to smithi08575397-32 2021-02-10T04:30:02.257 INFO:tasks.rados.rados.0.smithi085.stdout:482: done (11 left) 2021-02-10T04:30:02.257 INFO:tasks.rados.rados.0.smithi085.stdout:486: done (10 left) 2021-02-10T04:30:02.258 INFO:tasks.rados.rados.0.smithi085.stdout:487: done (9 left) 2021-02-10T04:30:02.258 INFO:tasks.rados.rados.0.smithi085.stdout:488: done (8 left) 2021-02-10T04:30:02.258 INFO:tasks.rados.rados.0.smithi085.stdout:489: done (7 left) 2021-02-10T04:30:02.258 INFO:tasks.rados.rados.0.smithi085.stdout:492: delete oid 6 current snap is 44 2021-02-10T04:30:02.259 INFO:tasks.rados.rados.0.smithi085.stdout:492: done (7 left) 2021-02-10T04:30:02.259 INFO:tasks.rados.rados.0.smithi085.stdout:493: snap_remove snap 42 2021-02-10T04:30:02.363 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:02 smithi085 conmon[29319]: cluster 2021-02-10T04:30:02.363 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:02 smithi085 conmon[29319]: 2021-02-10T04:30:02.363 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:02 smithi085 conmon[29319]: 2021-02-10T04:30:01. 2021-02-10T04:30:02.363 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:02 smithi085 conmon[29319]: 092194+0000 mon.a (mon.0) 886 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2021-02-10T04:30:02.363 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:02 smithi085 conmon[35961]: cluster 2021-02-10T04:30:01.092194+0000 mon.a ( 2021-02-10T04:30:02.363 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:02 smithi085 conmon[35961]: mon.0) 886 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2021-02-10T04:30:02.582 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:30:02.690 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:30:02 smithi085 conmon[49106]: debug 2021-02-10T04:30:02.361+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:03.054 INFO:tasks.rados.rados.0.smithi085.stdout:493: done (7 left) 2021-02-10T04:30:03.055 INFO:tasks.rados.rados.0.smithi085.stdout:494: delete oid 27 current snap is 44 2021-02-10T04:30:03.055 INFO:tasks.rados.rados.0.smithi085.stdout:491: finishing write tid 2 to smithi08575397-32 2021-02-10T04:30:03.055 INFO:tasks.rados.rados.0.smithi085.stdout:491: finishing write tid 3 to smithi08575397-32 2021-02-10T04:30:03.055 INFO:tasks.rados.rados.0.smithi085.stdout:491: finishing write tid 4 to smithi08575397-32 2021-02-10T04:30:03.055 INFO:tasks.rados.rados.0.smithi085.stdout:491: finishing write tid 5 to smithi08575397-32 2021-02-10T04:30:03.055 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 32 v 97 (ObjNum 204 snap 44 seq_num 204) dirty exists 2021-02-10T04:30:03.055 INFO:tasks.rados.rados.0.smithi085.stdout:491: left oid 32 (ObjNum 204 snap 44 seq_num 204) 2021-02-10T04:30:03.056 INFO:tasks.rados.rados.0.smithi085.stdout:491: done (7 left) 2021-02-10T04:30:03.056 INFO:tasks.rados.rados.0.smithi085.stdout:494: done (6 left) 2021-02-10T04:30:03.057 INFO:tasks.rados.rados.0.smithi085.stdout:495: snap_create 2021-02-10T04:30:03.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:03 smithi164 conmon[29288]: cluster 2021-02-10T04:30:02.012521+0000 mgr.y (mgr.14140) 388 2021-02-10T04:30:03.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:03 smithi164 conmon[29288]: : cluster [DBG] pgmap v427: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 125 MiB data, 395 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 4.0 MiB/s wr, 25 op/s 2021-02-10T04:30:03.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:03 smithi164 conmon[29288]: cluster 2021-02-10T04:30:02.105115+0000 mon.a (mon.0) 887 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2021-02-10T04:30:03.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:03 smithi164 conmon[29288]: cluster 2021-02-10T04:30:03.053931+0000 mon.a (mon.0) 888 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2021-02-10T04:30:03.441 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:30:03 smithi085 conmon[52836]: debug 2021-02-10T04:30:03.101+0000 7fba747b7700 -1 osd.3 pg_epoch: 187 pg[7.3( v 186'155 (0'0,186'155] local-lis/les=93/94 n=4 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [3,2,4] r=0 lpr=93 crt=186'155 lcod 184'154 mlcod 184'154 active+clean+snaptrim trimq=[56~2] ps=[4e~1,53~1]] removing snap head 2021-02-10T04:30:03.441 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:30:03 smithi085 conmon[52836]: debug 2021-02-10T04:30:03.103+0000 7fba747b7700 -1 osd.3 pg_epoch: 187 pg[7.3( v 187'157 (0'0,187'157] local-lis/les=93/94 n=2 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [3,2,4] r=0 lpr=93 luod=186'155 crt=187'157 lcod 184'154 mlcod 184'154 active+clean+snaptrim trimq=[56~2] ps=[4e~1,53~1]] removing snap head 2021-02-10T04:30:03.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:03 smithi085 conmon[35961]: cluster 2021-02-10T04:30:03.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:03 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:30:03.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:03 smithi085 conmon[35961]: 10T04:30:02.012521+0000 mgr.y (mgr.14140) 388 : cluster [DBG] pgmap v427: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 125 MiB data, 395 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 4.0 MiB/s wr, 25 op/s 2021-02-10T04:30:03.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:03 smithi085 conmon[35961]: 2021-02-10T04:30:03.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:03 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:30:03.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:03 smithi085 conmon[35961]: -10T04:30:02.105115+0000 mon.a 2021-02-10T04:30:03.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:03 smithi085 conmon[35961]: (mon.0) 887 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2021-02-10T04:30:03.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:03 smithi085 conmon[35961]: cluster 2021-02-10T04:30:03.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:03 smithi085 conmon[35961]: 2021-02-10T04:30:03.053931+0000 2021-02-10T04:30:03.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:03 smithi085 conmon[35961]: mon.a (mon.0) 888 : cluster [DBG] 2021-02-10T04:30:03.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:03 smithi085 conmon[35961]: osdmap e187: 8 total, 8 up, 8 in 2021-02-10T04:30:03.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:03 smithi085 conmon[29319]: cluster 2021-02-10T04:30:02.012521+0000 mgr.y (mgr.14140) 388 : cluster 2021-02-10T04:30:03.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:03 smithi085 conmon[29319]: [DBG] pgmap v427: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 125 MiB data, 395 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 4.0 MiB/s wr, 25 op/s 2021-02-10T04:30:03.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:03 smithi085 conmon[29319]: cluster 2021-02-10T04:30:02.105115+0000 mon.a (mon.0) 887 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2021-02-10T04:30:03.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:03 smithi085 conmon[29319]: cluster 2021-02-10T04:30:03.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:03 smithi085 conmon[29319]: 2021-02-10T04:30:03.053931+0000 2021-02-10T04:30:03.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:03 smithi085 conmon[29319]: mon.a (mon.0) 888 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2021-02-10T04:30:03.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:03 smithi085 conmon[29319]: 2021-02-10T04:30:03.990 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.5 dump_blocked_ops 2021-02-10T04:30:03.994 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:30:04.027 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:04.055 INFO:tasks.rados.rados.0.smithi085.stdout:495: done (6 left) 2021-02-10T04:30:04.055 INFO:tasks.rados.rados.0.smithi085.stdout:496: snap_create 2021-02-10T04:30:04.273 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:30:04.393 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:30:04 smithi164 conmon[37263]: debug 2021-02-10T04:30:04.106+0000 7f1c6c9ed700 -1 osd.5 pg_epoch: 188 pg[7.0( v 186'104 (0'0,186'104] local-lis/les=93/94 n=4 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [5,1,3] r=0 lpr=93 crt=186'104 lcod 185'102 mlcod 185'102 active+clean+snaptrim trimq=[56~1] ps=[4e~1,53~1,57~1]] removing snap head 2021-02-10T04:30:05.064 INFO:tasks.rados.rados.0.smithi085.stdout:496: done (6 left) 2021-02-10T04:30:05.065 INFO:tasks.rados.rados.0.smithi085.stdout:497: write (excl) oid 46 current snap is 46 2021-02-10T04:30:05.065 INFO:tasks.rados.rados.0.smithi085.stdout:497: seq_num 207 ranges {679049=718074,2113302=674388,3335030=144019} 2021-02-10T04:30:05.066 INFO:tasks.rados.rados.0.smithi085.stdout:497: writing smithi08575397-46 from 679049 to 1397123 tid 1 2021-02-10T04:30:05.076 INFO:tasks.rados.rados.0.smithi085.stdout:497: writing smithi08575397-46 from 2113302 to 2787690 tid 2 2021-02-10T04:30:05.080 INFO:tasks.rados.rados.0.smithi085.stdout:497: writing smithi08575397-46 from 3335030 to 3479049 tid 3 2021-02-10T04:30:05.080 INFO:tasks.rados.rados.0.smithi085.stdout:498: write oid 14 current snap is 46 2021-02-10T04:30:05.080 INFO:tasks.rados.rados.0.smithi085.stdout:497: finishing write tid 1 to smithi08575397-46 2021-02-10T04:30:05.081 INFO:tasks.rados.rados.0.smithi085.stdout:497: finishing write tid 2 to smithi08575397-46 2021-02-10T04:30:05.081 INFO:tasks.rados.rados.0.smithi085.stdout:498: seq_num 208 ranges {695856=793323,2171784=534399,3362453=133403} 2021-02-10T04:30:05.089 INFO:tasks.rados.rados.0.smithi085.stdout:498: writing smithi08575397-14 from 695856 to 1489179 tid 1 2021-02-10T04:30:05.097 INFO:tasks.rados.rados.0.smithi085.stdout:498: writing smithi08575397-14 from 2171784 to 2706183 tid 2 2021-02-10T04:30:05.102 INFO:tasks.rados.rados.0.smithi085.stdout:498: writing smithi08575397-14 from 3362453 to 3495856 tid 3 2021-02-10T04:30:05.102 INFO:tasks.rados.rados.0.smithi085.stdout:497: finishing write tid 3 to smithi08575397-46 2021-02-10T04:30:05.102 INFO:tasks.rados.rados.0.smithi085.stdout:497: finishing write tid 4 to smithi08575397-46 2021-02-10T04:30:05.102 INFO:tasks.rados.rados.0.smithi085.stdout:497: finishing write tid 5 to smithi08575397-46 2021-02-10T04:30:05.103 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 163 (ObjNum 207 snap 46 seq_num 207) dirty exists 2021-02-10T04:30:05.103 INFO:tasks.rados.rados.0.smithi085.stdout:497: left oid 46 (ObjNum 207 snap 46 seq_num 207) 2021-02-10T04:30:05.103 INFO:tasks.rados.rados.0.smithi085.stdout:498: finishing write tid 1 to smithi08575397-14 2021-02-10T04:30:05.103 INFO:tasks.rados.rados.0.smithi085.stdout:497: done (7 left) 2021-02-10T04:30:05.103 INFO:tasks.rados.rados.0.smithi085.stdout:499: snap_remove snap 45 2021-02-10T04:30:05.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:05 smithi164 conmon[29288]: cluster 2021-02-10T04:30:04.013101+0000 mgr.y (mgr.14140) 389 : cluster [DBG] pgmap v430: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 125 MiB data, 403 MiB used, 715 GiB / 715 GiB avail; 26 MiB/s rd, 3.0 MiB/s wr, 31 op/s 2021-02-10T04:30:05.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:05 smithi164 conmon[29288]: cluster 2021-02-10T04:30:04.054045+0000 mon.a (mon.0) 889 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2021-02-10T04:30:05.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:05 smithi164 conmon[29288]: cluster 2021-02-10T04:30:04.095931+0000 mon.a (mon.0) 890 : cluster [INF] Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 46 sec, osd.1 has slow ops) 2021-02-10T04:30:05.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:05 smithi085 conmon[29319]: cluster 2021-02-10T04:30:04.013101+0000 mgr.y (mgr.14140 2021-02-10T04:30:05.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:05 smithi085 conmon[29319]: ) 389 : cluster [DBG] pgmap v430: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 125 MiB data, 403 MiB used, 715 GiB / 715 GiB avail; 26 MiB/s rd, 3.0 MiB/s wr, 31 op/s 2021-02-10T04:30:05.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:05 smithi085 conmon[29319]: cluster 2021-02-10T04:30:04.054045+0000 mon.a (mon.0) 889 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2021-02-10T04:30:05.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:05 smithi085 conmon[29319]: cluster 2021-02-10T04:30:04.095931+0000 mon.a (mon.0) 890 : cluster [INF] Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 46 sec, osd.1 has slow ops) 2021-02-10T04:30:05.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:05 smithi085 conmon[35961]: cluster 2021-02-10T04:30:04.013101+0000 2021-02-10T04:30:05.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:05 smithi085 conmon[35961]: mgr.y (mgr.14140) 389 : cluster [DBG] pgmap v430: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 125 MiB data, 403 MiB used, 715 GiB / 715 GiB avail; 26 MiB/s rd, 3.0 MiB/s wr, 31 op/s 2021-02-10T04:30:05.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:05 smithi085 conmon[35961]: cluster 2021-02-10T04:30:04.054045+0000 mon.a (mon.0) 889 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2021-02-10T04:30:05.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:05 smithi085 conmon[35961]: cluster 2021-02-10T04:30:04.095931+0000 mon.a (mon.0) 890 : cluster [INF] Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 46 sec, osd.1 has slow ops) 2021-02-10T04:30:05.442 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:30:05 smithi085 conmon[41540]: debug 2021-02-10T04:30:05.106+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:05.493 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:30:06.076 INFO:tasks.rados.rados.0.smithi085.stdout:498: finishing write tid 2 to smithi08575397-14 2021-02-10T04:30:06.076 INFO:tasks.rados.rados.0.smithi085.stdout:498: finishing write tid 3 to smithi08575397-14 2021-02-10T04:30:06.076 INFO:tasks.rados.rados.0.smithi085.stdout:498: finishing write tid 4 to smithi08575397-14 2021-02-10T04:30:06.077 INFO:tasks.rados.rados.0.smithi085.stdout:498: finishing write tid 5 to smithi08575397-14 2021-02-10T04:30:06.077 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 139 (ObjNum 208 snap 46 seq_num 208) dirty exists 2021-02-10T04:30:06.077 INFO:tasks.rados.rados.0.smithi085.stdout:498: left oid 14 (ObjNum 208 snap 46 seq_num 208) 2021-02-10T04:30:06.077 INFO:tasks.rados.rados.0.smithi085.stdout:498: done (7 left) 2021-02-10T04:30:06.077 INFO:tasks.rados.rados.0.smithi085.stdout:499: done (6 left) 2021-02-10T04:30:06.077 INFO:tasks.rados.rados.0.smithi085.stdout:500: snap_remove snap 44 2021-02-10T04:30:06.384 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:06 smithi085 conmon[35961]: cluster 2021-02-10T04:30:05.054665+0000 mon.a (mon.0) 891 2021-02-10T04:30:06.384 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:06 smithi085 conmon[35961]: : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2021-02-10T04:30:06.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:06 smithi085 conmon[29319]: cluster 2021-02-10T04:30:05.054665+0000 mon.a (mon. 2021-02-10T04:30:06.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:06 smithi085 conmon[29319]: 0) 891 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2021-02-10T04:30:06.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:06 smithi164 conmon[29288]: cluster 2021-02-10T04:30:05.054665+0000 2021-02-10T04:30:06.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:06 smithi164 conmon[29288]: mon.a (mon.0) 891 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2021-02-10T04:30:06.488 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:30:06.522 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:06.893 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:30:06 smithi164 conmon[37263]: debug 2021-02-10T04:30:06.893 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:30:06 smithi164 conmon[37263]: 2021-02-10T04:30:06.611+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:06.893 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:30:06 smithi164 conmon[37263]: 2021-02-10T04:30:06.929 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:30:07.058 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:30:07.079 INFO:tasks.rados.rados.0.smithi085.stdout:500: done (6 left) 2021-02-10T04:30:07.080 INFO:tasks.rados.rados.0.smithi085.stdout:501: copy_from oid 50 from oid 43 current snap is 46 2021-02-10T04:30:07.080 INFO:tasks.rados.rados.0.smithi085.stdout:502: write oid 38 current snap is 46 2021-02-10T04:30:07.080 INFO:tasks.rados.rados.0.smithi085.stdout:502: seq_num 209 ranges {712663=468572,1713913=678057,2873523=591733} 2021-02-10T04:30:07.087 INFO:tasks.rados.rados.0.smithi085.stdout:502: writing smithi08575397-38 from 712663 to 1181235 tid 1 2021-02-10T04:30:07.089 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:07 smithi085 conmon[29319]: cluster 2021-02-10T04:30:05.554074+0000 osd.2 2021-02-10T04:30:07.089 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:07 smithi085 conmon[29319]: (osd.2) 45 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:30:07.089 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:07 smithi085 conmon[29319]: cluster 2021-02-10T04:30:05.554250 2021-02-10T04:30:07.089 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:07 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 46 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:30:07.089 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:07 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:30:07.089 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:07 smithi085 conmon[29319]: -02-10T04:30:06.013697+0000 mgr.y (mgr 2021-02-10T04:30:07.089 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:07 smithi085 conmon[29319]: .14140) 390 : cluster [DBG] pgmap v433: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 121 MiB data, 407 MiB used, 715 GiB / 715 GiB avail; 22 MiB/s rd, 2.4 MiB/s wr, 30 op/s 2021-02-10T04:30:07.090 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:07 smithi085 conmon[29319]: cluster 2021-02-10T04:30:07.090 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:07 smithi085 conmon[29319]: 2021-02 2021-02-10T04:30:07.090 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:07 smithi085 conmon[29319]: -10T04:30:06.076209+0000 2021-02-10T04:30:07.090 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:07 smithi085 conmon[29319]: mon.a (mon.0) 892 : cluster [DBG] 2021-02-10T04:30:07.090 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:07 smithi085 conmon[29319]: osdmap e190: 8 total, 8 up, 8 in 2021-02-10T04:30:07.090 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:07.095 INFO:tasks.rados.rados.0.smithi085.stdout:502: writing smithi08575397-38 from 1713913 to 2391970 tid 2 2021-02-10T04:30:07.102 INFO:tasks.rados.rados.0.smithi085.stdout:502: writing smithi08575397-38 from 2873523 to 3465256 tid 3 2021-02-10T04:30:07.102 INFO:tasks.rados.rados.0.smithi085.stdout:503: write (excl) oid 26 current snap is 46 2021-02-10T04:30:07.102 INFO:tasks.rados.rados.0.smithi085.stdout:501: finishing copy_from to smithi08575397-50 2021-02-10T04:30:07.102 INFO:tasks.rados.rados.0.smithi085.stdout:501: got expected ENOENT (src dne) 2021-02-10T04:30:07.102 INFO:tasks.rados.rados.0.smithi085.stdout:503: seq_num 210 ranges {729470=543821,1772395=538068,2900946=403005} 2021-02-10T04:30:07.110 INFO:tasks.rados.rados.0.smithi085.stdout:503: writing smithi08575397-26 from 729470 to 1273291 tid 1 2021-02-10T04:30:07.116 INFO:tasks.rados.rados.0.smithi085.stdout:503: writing smithi08575397-26 from 1772395 to 2310463 tid 2 2021-02-10T04:30:07.123 INFO:tasks.rados.rados.0.smithi085.stdout:503: writing smithi08575397-26 from 2900946 to 3303951 tid 3 2021-02-10T04:30:07.123 INFO:tasks.rados.rados.0.smithi085.stdout:504: delete oid 18 current snap is 46 2021-02-10T04:30:07.123 INFO:tasks.rados.rados.0.smithi085.stdout:501: finishing copy_from racing read to smithi08575397-50 2021-02-10T04:30:07.123 INFO:tasks.rados.rados.0.smithi085.stdout:502: finishing write tid 1 to smithi08575397-38 2021-02-10T04:30:07.123 INFO:tasks.rados.rados.0.smithi085.stdout:502: finishing write tid 2 to smithi08575397-38 2021-02-10T04:30:07.123 INFO:tasks.rados.rados.0.smithi085.stdout:502: finishing write tid 3 to smithi08575397-38 2021-02-10T04:30:07.123 INFO:tasks.rados.rados.0.smithi085.stdout:502: finishing write tid 4 to smithi08575397-38 2021-02-10T04:30:07.123 INFO:tasks.rados.rados.0.smithi085.stdout:502: finishing write tid 5 to smithi08575397-38 2021-02-10T04:30:07.123 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 167 (ObjNum 209 snap 46 seq_num 209) dirty exists 2021-02-10T04:30:07.123 INFO:tasks.rados.rados.0.smithi085.stdout:502: left oid 38 (ObjNum 209 snap 46 seq_num 209) 2021-02-10T04:30:07.123 INFO:tasks.rados.rados.0.smithi085.stdout:503: finishing write tid 1 to smithi08575397-26 2021-02-10T04:30:07.124 INFO:tasks.rados.rados.0.smithi085.stdout:503: finishing write tid 2 to smithi08575397-26 2021-02-10T04:30:07.125 INFO:tasks.rados.rados.0.smithi085.stdout:503: finishing write tid 3 to smithi08575397-26 2021-02-10T04:30:07.125 INFO:tasks.rados.rados.0.smithi085.stdout:501: done (9 left) 2021-02-10T04:30:07.126 INFO:tasks.rados.rados.0.smithi085.stdout:502: done (8 left) 2021-02-10T04:30:07.126 INFO:tasks.rados.rados.0.smithi085.stdout:504: done (7 left) 2021-02-10T04:30:07.126 INFO:tasks.rados.rados.0.smithi085.stdout:505: delete oid 39 current snap is 46 2021-02-10T04:30:07.126 INFO:tasks.rados.rados.0.smithi085.stdout:503: finishing write tid 4 to smithi08575397-26 2021-02-10T04:30:07.126 INFO:tasks.rados.rados.0.smithi085.stdout:503: finishing write tid 5 to smithi08575397-26 2021-02-10T04:30:07.126 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 34 (ObjNum 210 snap 46 seq_num 210) dirty exists 2021-02-10T04:30:07.126 INFO:tasks.rados.rados.0.smithi085.stdout:503: left oid 26 (ObjNum 210 snap 46 seq_num 210) 2021-02-10T04:30:07.127 INFO:tasks.rados.rados.0.smithi085.stdout:503: done (7 left) 2021-02-10T04:30:07.127 INFO:tasks.rados.rados.0.smithi085.stdout:505: done (6 left) 2021-02-10T04:30:07.127 INFO:tasks.rados.rados.0.smithi085.stdout:506: copy_from oid 43 from oid 16 current snap is 46 2021-02-10T04:30:07.127 INFO:tasks.rados.rados.0.smithi085.stdout:507: snap_create 2021-02-10T04:30:07.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:07 smithi164 conmon[29288]: cluster 2021-02-10T04:30:05.554074 2021-02-10T04:30:07.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:07 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 45 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:30:07.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:07 smithi164 conmon[29288]: cluster 2021-02-10T04:30:05.554250+0000 osd.2 (osd.2) 46 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:30:07.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:07 smithi164 conmon[29288]: cluster 2021-02-10T04:30:06.013697+0000 mgr.y (mgr.14140) 390 2021-02-10T04:30:07.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:07 smithi164 conmon[29288]: : cluster [DBG] pgmap v433: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 121 MiB data, 407 MiB used, 715 GiB / 715 GiB avail; 22 MiB/s rd, 2.4 MiB/s wr, 30 op/s 2021-02-10T04:30:07.213 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:07 smithi164 conmon[29288]: cluster 2021-02-10T04:30:06.076209+0000 2021-02-10T04:30:07.214 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:07 smithi164 conmon[29288]: mon.a (mon.0) 892 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2021-02-10T04:30:07.246 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:30:07.351 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:30:07.385 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:07.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:07 smithi085 conmon[35961]: cluster 2021-02-10T04:30:05.554074+0000 osd.2 (osd.2 2021-02-10T04:30:07.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:07 smithi085 conmon[35961]: ) 45 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:30:07.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:07 smithi085 conmon[35961]: cluster 2021-02-10T04:30:05.554250+0000 osd.2 (osd.2) 46 2021-02-10T04:30:07.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:07 smithi085 conmon[35961]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:30:07.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:07 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:30:07.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:07 smithi085 conmon[35961]: T04:30:06.013697+0000 mgr.y (mgr.14140) 390 : cluster [DBG] 2021-02-10T04:30:07.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:07 smithi085 conmon[35961]: pgmap v433: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 121 MiB data, 407 MiB used, 715 GiB / 715 GiB avail; 22 MiB/s rd, 2.4 MiB/s wr, 30 op/s 2021-02-10T04:30:07.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:07 smithi085 conmon[35961]: cluster 2021-02-10T04:30:06.076209+0000 mon.a (mon.0) 892 2021-02-10T04:30:07.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:07 smithi085 conmon[35961]: : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2021-02-10T04:30:07.474 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:30:07 smithi164 conmon[33677]: debug 2021-02-10T04:30:07.211+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:07.546 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:30:07.650 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:30:07.683 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:07.765 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:30:07 smithi164 conmon[33677]: debug 2021-02-10T04:30:07.517+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:07.770 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:30:07.771 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:30:08.056 INFO:tasks.rados.rados.0.smithi085.stdout:507: done (7 left) 2021-02-10T04:30:08.056 INFO:tasks.rados.rados.0.smithi085.stdout:508: write (excl) oid 28 current snap is 47 2021-02-10T04:30:08.057 INFO:tasks.rados.rados.0.smithi085.stdout:508: seq_num 213 ranges {779891=485921,1947841=685395,3383215=196676} 2021-02-10T04:30:08.064 INFO:tasks.rados.rados.0.smithi085.stdout:508: writing smithi08575397-28 from 779891 to 1265812 tid 1 2021-02-10T04:30:08.072 INFO:tasks.rados.rados.0.smithi085.stdout:508: writing smithi08575397-28 from 1947841 to 2633236 tid 2 2021-02-10T04:30:08.078 INFO:tasks.rados.rados.0.smithi085.stdout:508: writing smithi08575397-28 from 3383215 to 3579891 tid 3 2021-02-10T04:30:08.079 INFO:tasks.rados.rados.0.smithi085.stdout:509: read oid 46 snap -1 2021-02-10T04:30:08.079 INFO:tasks.rados.rados.0.smithi085.stdout:509: expect (ObjNum 207 snap 46 seq_num 207) 2021-02-10T04:30:08.079 INFO:tasks.rados.rados.0.smithi085.stdout:510: read oid 11 snap -1 2021-02-10T04:30:08.079 INFO:tasks.rados.rados.0.smithi085.stdout:510: expect (ObjNum 190 snap 42 seq_num 190) 2021-02-10T04:30:08.079 INFO:tasks.rados.rados.0.smithi085.stdout:511: snap_remove snap 46 2021-02-10T04:30:08.139 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:30:07 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:30:08.140 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:08 smithi164 conmon[29288]: cluster 2021-02-10T04:30:07.078019+0000 mon.a (mon.0) 893 : cluster [DBG] 2021-02-10T04:30:08.140 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:08 smithi164 conmon[29288]: osdmap e191: 8 total, 8 up, 8 in 2021-02-10T04:30:08.140 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:08 smithi164 conmon[29288]: cluster 2021-02-10T04:30:08.055220+0000 mon.a (mon.0) 894 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2021-02-10T04:30:08.435 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:08 smithi085 conmon[29319]: cluster 2021-02-10T04:30:08.435 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:08 smithi085 conmon[29319]: 2021-02-10T04:30 2021-02-10T04:30:08.435 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:08 smithi085 conmon[29319]: :07.078019+0000 mon.a (mon.0) 893 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2021-02-10T04:30:08.435 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:08 smithi085 conmon[29319]: cluster 2021-02-10T04:30:08.435 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:08 smithi085 conmon[29319]: 2021-02-10T04:30:08.055220+0000 2021-02-10T04:30:08.435 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:08 smithi085 conmon[29319]: mon.a (mon.0) 894 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2021-02-10T04:30:08.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:08 smithi085 conmon[35961]: cluster 2021-02-10T04:30:07.078019+0000 2021-02-10T04:30:08.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:08 smithi085 conmon[35961]: mon.a (mon.0) 893 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2021-02-10T04:30:08.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:08 smithi085 conmon[35961]: cluster 2021-02-10T04:30:08.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:08 smithi085 conmon[35961]: 2021-02-10T04:30:08.055220+0000 mon.a (mon.0) 894 : cluster 2021-02-10T04:30:08.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:08 smithi085 conmon[35961]: [DBG] osdmap e192: 8 total, 8 up, 8 in 2021-02-10T04:30:08.596 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:30:08.691 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:30:08 smithi085 conmon[41540]: debug 2021-02-10T04:30:08.518+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:08.713 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:30:08.748 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:09.065 INFO:tasks.rados.rados.0.smithi085.stdout:511: done (10 left) 2021-02-10T04:30:09.065 INFO:tasks.rados.rados.0.smithi085.stdout:512: read oid 23 snap -1 2021-02-10T04:30:09.065 INFO:tasks.rados.rados.0.smithi085.stdout:512: expect (ObjNum 124 snap 23 seq_num 124) 2021-02-10T04:30:09.065 INFO:tasks.rados.rados.0.smithi085.stdout:513: delete oid 44 current snap is 47 2021-02-10T04:30:09.065 INFO:tasks.rados.rados.0.smithi085.stdout:508: finishing write tid 1 to smithi08575397-28 2021-02-10T04:30:09.065 INFO:tasks.rados.rados.0.smithi085.stdout:509: expect (ObjNum 207 snap 46 seq_num 207) 2021-02-10T04:30:09.143 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:30:08 smithi164 conmon[33677]: debug 2021-02-10T04:30:08.824+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:09.217 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:30:09.230 INFO:tasks.rados.rados.0.smithi085.stdout:510: expect (ObjNum 190 snap 42 seq_num 190) 2021-02-10T04:30:09.374 INFO:tasks.rados.rados.0.smithi085.stdout:508: finishing write tid 2 to smithi08575397-28 2021-02-10T04:30:09.374 INFO:tasks.rados.rados.0.smithi085.stdout:508: finishing write tid 3 to smithi08575397-28 2021-02-10T04:30:09.374 INFO:tasks.rados.rados.0.smithi085.stdout:508: finishing write tid 4 to smithi08575397-28 2021-02-10T04:30:09.374 INFO:tasks.rados.rados.0.smithi085.stdout:506: finishing copy_from to smithi08575397-43 2021-02-10T04:30:09.374 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 210 (ObjNum 194 snap 43 seq_num 194) dirty exists 2021-02-10T04:30:09.374 INFO:tasks.rados.rados.0.smithi085.stdout:508: finishing write tid 5 to smithi08575397-28 2021-02-10T04:30:09.375 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 48 (ObjNum 213 snap 47 seq_num 213) dirty exists 2021-02-10T04:30:09.375 INFO:tasks.rados.rados.0.smithi085.stdout:508: left oid 28 (ObjNum 213 snap 47 seq_num 213) 2021-02-10T04:30:09.375 INFO:tasks.rados.rados.0.smithi085.stdout:506: finishing copy_from racing read to smithi08575397-43 2021-02-10T04:30:09.375 INFO:tasks.rados.rados.0.smithi085.stdout:506: done (11 left) 2021-02-10T04:30:09.375 INFO:tasks.rados.rados.0.smithi085.stdout:508: done (10 left) 2021-02-10T04:30:09.375 INFO:tasks.rados.rados.0.smithi085.stdout:509: done (9 left) 2021-02-10T04:30:09.375 INFO:tasks.rados.rados.0.smithi085.stdout:510: done (8 left) 2021-02-10T04:30:09.375 INFO:tasks.rados.rados.0.smithi085.stdout:513: done (7 left) 2021-02-10T04:30:09.375 INFO:tasks.rados.rados.0.smithi085.stdout:514: rollback oid 26 current snap is 47 2021-02-10T04:30:09.375 INFO:tasks.rados.rados.0.smithi085.stdout:514: done (7 left) 2021-02-10T04:30:09.375 INFO:tasks.rados.rados.0.smithi085.stdout:515: read oid 14 snap -1 2021-02-10T04:30:09.375 INFO:tasks.rados.rados.0.smithi085.stdout:515: expect (ObjNum 208 snap 46 seq_num 208) 2021-02-10T04:30:09.375 INFO:tasks.rados.rados.0.smithi085.stdout:516: read oid 24 snap -1 2021-02-10T04:30:09.376 INFO:tasks.rados.rados.0.smithi085.stdout:516: expect deleted 2021-02-10T04:30:09.376 INFO:tasks.rados.rados.0.smithi085.stdout:512: expect (ObjNum 124 snap 23 seq_num 124) 2021-02-10T04:30:09.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[29319]: audit 2021-02-10T04:30:09.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[29319]: 2021-02-10T04:30:07 2021-02-10T04:30:09.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[29319]: .765066+0000 mgr.y (mgr.14140) 391 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:30:09.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:30:09.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[29319]: 02-10T04:30:08.014103+0000 mgr.y (mgr.14140) 2021-02-10T04:30:09.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[29319]: 392 : cluster [DBG] pgmap v436: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 120 MiB data, 391 MiB used, 715 GiB / 715 GiB avail; 5.6 MiB/s rd, 2.8 MiB/s wr, 18 op/s 2021-02-10T04:30:09.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:30:09.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[29319]: -02-10T04:30:09.065302 2021-02-10T04:30:09.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[29319]: +0000 mon.a (mon.0) 895 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2021-02-10T04:30:09.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[29319]: 2021-02-10T04:30:09.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: audit 2021-02-10T04:30:09.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 2021-02-10T04:30:09.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 2021 2021-02-10T04:30:09.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: - 2021-02-10T04:30:09.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 02 2021-02-10T04:30:09.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: - 2021-02-10T04:30:09.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 10 2021-02-10T04:30:09.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: T 2021-02-10T04:30:09.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 04 2021-02-10T04:30:09.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: : 2021-02-10T04:30:09.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 30 2021-02-10T04:30:09.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: : 2021-02-10T04:30:09.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 07 2021-02-10T04:30:09.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: . 2021-02-10T04:30:09.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 765066 2021-02-10T04:30:09.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: +0000 2021-02-10T04:30:09.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 2021-02-10T04:30:09.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: mgr.y 2021-02-10T04:30:09.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: ( 2021-02-10T04:30:09.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: mgr 2021-02-10T04:30:09.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: . 2021-02-10T04:30:09.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 14140 2021-02-10T04:30:09.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: ) 2021-02-10T04:30:09.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 391 2021-02-10T04:30:09.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: : 2021-02-10T04:30:09.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: audit 2021-02-10T04:30:09.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 2021-02-10T04:30:09.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: [DBG] 2021-02-10T04:30:09.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 2021-02-10T04:30:09.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:30:09.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 2021-02-10T04:30:09.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: cluster 2021-02-10T04:30:09.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 2021-02-10T04:30:09.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 2021 2021-02-10T04:30:09.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: - 2021-02-10T04:30:09.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 02 2021-02-10T04:30:09.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: - 2021-02-10T04:30:09.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 10 2021-02-10T04:30:09.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: T 2021-02-10T04:30:09.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 04 2021-02-10T04:30:09.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: : 2021-02-10T04:30:09.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 30 2021-02-10T04:30:09.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: : 2021-02-10T04:30:09.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 08 2021-02-10T04:30:09.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: . 2021-02-10T04:30:09.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 014103 2021-02-10T04:30:09.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: +0000 2021-02-10T04:30:09.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 2021-02-10T04:30:09.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: mgr.y 2021-02-10T04:30:09.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: ( 2021-02-10T04:30:09.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: mgr 2021-02-10T04:30:09.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: . 2021-02-10T04:30:09.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 14140 2021-02-10T04:30:09.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: ) 2021-02-10T04:30:09.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 392 2021-02-10T04:30:09.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: : 2021-02-10T04:30:09.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: cluster 2021-02-10T04:30:09.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 2021-02-10T04:30:09.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: [DBG] 2021-02-10T04:30:09.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 2021-02-10T04:30:09.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: pgmap v436: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 120 MiB data, 391 MiB used, 715 GiB / 715 GiB avail; 5.6 MiB/s rd, 2.8 MiB/s wr, 18 op/s 2021-02-10T04:30:09.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 2021-02-10T04:30:09.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: cluster 2021-02-10T04:30:09.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 2021-02-10T04:30:09.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 2021 2021-02-10T04:30:09.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: - 2021-02-10T04:30:09.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 02 2021-02-10T04:30:09.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: - 2021-02-10T04:30:09.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 10 2021-02-10T04:30:09.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: T 2021-02-10T04:30:09.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 04 2021-02-10T04:30:09.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: : 2021-02-10T04:30:09.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 30 2021-02-10T04:30:09.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: : 2021-02-10T04:30:09.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 09 2021-02-10T04:30:09.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: . 2021-02-10T04:30:09.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 065302 2021-02-10T04:30:09.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: +0000 2021-02-10T04:30:09.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 2021-02-10T04:30:09.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: mon.a 2021-02-10T04:30:09.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: ( 2021-02-10T04:30:09.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: mon 2021-02-10T04:30:09.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: . 2021-02-10T04:30:09.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 0 2021-02-10T04:30:09.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: ) 2021-02-10T04:30:09.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 895 2021-02-10T04:30:09.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: : 2021-02-10T04:30:09.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: cluster 2021-02-10T04:30:09.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 2021-02-10T04:30:09.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: [DBG] 2021-02-10T04:30:09.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 2021-02-10T04:30:09.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: osdmap e193: 8 total, 8 up, 8 in 2021-02-10T04:30:09.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:09 smithi085 conmon[35961]: 2021-02-10T04:30:09.482 INFO:tasks.rados.rados.0.smithi085.stdout:512: done (8 left) 2021-02-10T04:30:09.482 INFO:tasks.rados.rados.0.smithi085.stdout:517: read oid 44 snap -1 2021-02-10T04:30:09.483 INFO:tasks.rados.rados.0.smithi085.stdout:517: expect deleted 2021-02-10T04:30:09.483 INFO:tasks.rados.rados.0.smithi085.stdout:518: read oid 1 snap -1 2021-02-10T04:30:09.483 INFO:tasks.rados.rados.0.smithi085.stdout:518: expect (ObjNum 202 snap 44 seq_num 202) 2021-02-10T04:30:09.483 INFO:tasks.rados.rados.0.smithi085.stdout:519: snap_create 2021-02-10T04:30:09.483 INFO:tasks.rados.rados.0.smithi085.stdout:515: expect (ObjNum 208 snap 46 seq_num 208) 2021-02-10T04:30:09.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:09 smithi164 conmon[29288]: audit 2021-02-10T04:30:07.765066+0000 mgr.y (mgr.14140) 391 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:30:09.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:09 smithi164 conmon[29288]: 2021-02-10T04:30:09.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:09 smithi164 conmon[29288]: cluster 2021-02-10T04:30:08.014103+0000 mgr.y (mgr.14140) 392 : cluster [DBG] pgmap v436: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 120 MiB data, 391 MiB used, 715 GiB / 715 GiB avail; 5.6 MiB/s rd, 2.8 MiB/s wr, 18 op/s 2021-02-10T04:30:09.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:09 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:30:09.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:09 smithi164 conmon[29288]: :30:09.065302+0000 mon.a (mon.0) 895 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2021-02-10T04:30:09.727 INFO:tasks.rados.rados.0.smithi085.stdout:518: expect (ObjNum 202 snap 44 seq_num 202) 2021-02-10T04:30:09.828 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:30:09.858 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:09.979 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.5 dump_historic_ops 2021-02-10T04:30:10.012 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:30:10.117 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:30:10.149 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:10.163 INFO:tasks.rados.rados.0.smithi085.stdout:515: done (10 left) 2021-02-10T04:30:10.163 INFO:tasks.rados.rados.0.smithi085.stdout:516: done (9 left) 2021-02-10T04:30:10.163 INFO:tasks.rados.rados.0.smithi085.stdout:517: done (8 left) 2021-02-10T04:30:10.163 INFO:tasks.rados.rados.0.smithi085.stdout:518: done (7 left) 2021-02-10T04:30:10.163 INFO:tasks.rados.rados.0.smithi085.stdout:519: done (6 left) 2021-02-10T04:30:10.164 INFO:tasks.rados.rados.0.smithi085.stdout:520: read oid 18 snap 47 2021-02-10T04:30:10.164 INFO:tasks.rados.rados.0.smithi085.stdout:520: expect deleted 2021-02-10T04:30:10.164 INFO:tasks.rados.rados.0.smithi085.stdout:521: rollback oid 42 current snap is 48 2021-02-10T04:30:10.164 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 42 to 47 2021-02-10T04:30:10.164 INFO:tasks.rados.rados.0.smithi085.stdout:522: write (excl) oid 3 current snap is 48 2021-02-10T04:30:10.164 INFO:tasks.rados.rados.0.smithi085.stdout:522: seq_num 215 ranges {413505=636419,1664805=689064,3038061=492071} 2021-02-10T04:30:10.165 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.3 to 1 (10000) 2021-02-10T04:30:10.165 INFO:tasks.rados.rados.0.smithi085.stdout:522: writing smithi08575397-3 from 413505 to 1049924 tid 1 2021-02-10T04:30:10.167 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is unset 2021-02-10T04:30:10.172 INFO:tasks.rados.rados.0.smithi085.stdout:522: writing smithi08575397-3 from 1664805 to 2353869 tid 2 2021-02-10T04:30:10.181 INFO:tasks.rados.rados.0.smithi085.stdout:522: writing smithi08575397-3 from 3038061 to 3530132 tid 3 2021-02-10T04:30:10.181 INFO:tasks.rados.rados.0.smithi085.stdout:523: rollback oid 16 current snap is 48 2021-02-10T04:30:10.181 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 16 to 47 2021-02-10T04:30:10.181 INFO:tasks.rados.rados.0.smithi085.stdout:520: done (9 left) 2021-02-10T04:30:10.181 INFO:tasks.rados.rados.0.smithi085.stdout:524: delete oid 11 current snap is 48 2021-02-10T04:30:10.182 INFO:tasks.rados.rados.0.smithi085.stdout:522: finishing write tid 1 to smithi08575397-3 2021-02-10T04:30:10.183 INFO:tasks.rados.rados.0.smithi085.stdout:521: finishing rollback tid 0 to smithi08575397-42 2021-02-10T04:30:10.183 INFO:tasks.rados.rados.0.smithi085.stdout:521: finishing rollback tid 1 to smithi08575397-42 2021-02-10T04:30:10.183 INFO:tasks.rados.rados.0.smithi085.stdout:521: finishing rollback tid 2 to smithi08575397-42 2021-02-10T04:30:10.183 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 147 (ObjNum 184 snap 40 seq_num 184) dirty exists 2021-02-10T04:30:10.185 INFO:tasks.rados.rados.0.smithi085.stdout:522: finishing write tid 2 to smithi08575397-3 2021-02-10T04:30:10.185 INFO:tasks.rados.rados.0.smithi085.stdout:522: finishing write tid 3 to smithi08575397-3 2021-02-10T04:30:10.186 INFO:tasks.rados.rados.0.smithi085.stdout:522: finishing write tid 4 to smithi08575397-3 2021-02-10T04:30:10.187 INFO:tasks.rados.rados.0.smithi085.stdout:522: finishing write tid 5 to smithi08575397-3 2021-02-10T04:30:10.187 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 214 (ObjNum 215 snap 48 seq_num 215) dirty exists 2021-02-10T04:30:10.187 INFO:tasks.rados.rados.0.smithi085.stdout:522: left oid 3 (ObjNum 215 snap 48 seq_num 215) 2021-02-10T04:30:10.189 INFO:tasks.rados.rados.0.smithi085.stdout:523: finishing rollback tid 0 to smithi08575397-16 2021-02-10T04:30:10.189 INFO:tasks.rados.rados.0.smithi085.stdout:523: finishing rollback tid 1 to smithi08575397-16 2021-02-10T04:30:10.189 INFO:tasks.rados.rados.0.smithi085.stdout:523: finishing rollback tid 2 to smithi08575397-16 2021-02-10T04:30:10.190 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 217 (ObjNum 194 snap 43 seq_num 194) dirty exists 2021-02-10T04:30:10.190 INFO:tasks.rados.rados.0.smithi085.stdout:521: done (9 left) 2021-02-10T04:30:10.190 INFO:tasks.rados.rados.0.smithi085.stdout:522: done (8 left) 2021-02-10T04:30:10.190 INFO:tasks.rados.rados.0.smithi085.stdout:523: done (7 left) 2021-02-10T04:30:10.190 INFO:tasks.rados.rados.0.smithi085.stdout:524: done (6 left) 2021-02-10T04:30:10.190 INFO:tasks.rados.rados.0.smithi085.stdout:525: rollback oid 50 current snap is 48 2021-02-10T04:30:10.190 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 50 to 47 2021-02-10T04:30:10.190 INFO:tasks.rados.rados.0.smithi085.stdout:526: rollback oid 13 current snap is 48 2021-02-10T04:30:10.191 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 13 to 47 2021-02-10T04:30:10.191 INFO:tasks.rados.rados.0.smithi085.stdout:527: read oid 22 snap 47 2021-02-10T04:30:10.191 INFO:tasks.rados.rados.0.smithi085.stdout:527: expect (ObjNum 127 snap 23 seq_num 127) 2021-02-10T04:30:10.191 INFO:tasks.rados.rados.0.smithi085.stdout:528: snap_create 2021-02-10T04:30:10.192 INFO:tasks.rados.rados.0.smithi085.stdout:525: finishing rollback tid 0 to smithi08575397-50 2021-02-10T04:30:10.193 INFO:tasks.rados.rados.0.smithi085.stdout:525: finishing rollback tid 1 to smithi08575397-50 2021-02-10T04:30:10.193 INFO:tasks.rados.rados.0.smithi085.stdout:525: finishing rollback tid 2 to smithi08575397-50 2021-02-10T04:30:10.194 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 103 (ObjNum 198 snap 43 seq_num 198) dirty exists 2021-02-10T04:30:10.194 INFO:tasks.rados.rados.0.smithi085.stdout:526: finishing rollback tid 0 to smithi08575397-13 2021-02-10T04:30:10.194 INFO:tasks.rados.rados.0.smithi085.stdout:526: finishing rollback tid 1 to smithi08575397-13 2021-02-10T04:30:10.194 INFO:tasks.rados.rados.0.smithi085.stdout:526: finishing rollback tid 2 to smithi08575397-13 2021-02-10T04:30:10.195 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 183 (ObjNum 161 snap 36 seq_num 161) dirty exists 2021-02-10T04:30:10.202 INFO:tasks.rados.rados.0.smithi085.stdout:527: expect (ObjNum 127 snap 23 seq_num 127) 2021-02-10T04:30:10.254 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:30:10.257 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:30:10 smithi164 conmon[37263]: debug 2021-02-10T04:30:10.226+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:10.257 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:10 smithi164 conmon[29288]: audit 2021-02-10T04:30:09.455824 2021-02-10T04:30:10.258 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:10 smithi164 conmon[29288]: +0000 mon.a (mon.0) 896 : audit [INF] from='client.? 172.21.15.85:0/3158734121' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-02-10T04:30:10.258 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:10 smithi164 conmon[29288]: audit 2021-02-10T04:30:09.517387+0000 mon.a (mon.0) 2021-02-10T04:30:10.258 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:10 smithi164 conmon[29288]: 897 : audit [INF] from='client.? 172.21.15.85:0/2841810425' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:30:10.258 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:10 smithi164 conmon[29288]: cluster 2021-02-10T04:30:10.258 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:10 smithi164 conmon[29288]: 2021-02-10T04:30:10.014580+0000 mgr.y (mgr.14140) 393 : cluster [DBG] 2021-02-10T04:30:10.258 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:10 smithi164 conmon[29288]: pgmap v439: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 117 MiB data, 380 MiB used, 715 GiB / 715 GiB avail; 2.2 MiB/s rd, 718 KiB/s wr, 6 op/s 2021-02-10T04:30:10.258 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:30:09 smithi164 conmon[33677]: debug 2021-02-10T04:30:09.990+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:10.280 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:30:10.359 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:30:10.388 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:10.401 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:10 smithi085 conmon[35961]: audit 2021-02-10T04:30:10.401 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:10 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:30:10.401 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:10 smithi085 conmon[35961]: 10T04:30:09.455824+0000 mon.a (mon.0) 896 : audit [INF] from='client.? 172.21.15.85:0/3158734121' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-02-10T04:30:10.401 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:10 smithi085 conmon[35961]: audit 2021-02-10T04:30:10.401 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:10 smithi085 conmon[35961]: 2021-02 2021-02-10T04:30:10.402 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:10 smithi085 conmon[35961]: -10T04:30:09.517387+0000 mon.a (mon.0) 2021-02-10T04:30:10.402 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:10 smithi085 conmon[35961]: 897 : audit [INF] from='client.? 172.21.15.85:0/2841810425' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:30:10.402 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:10 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:30:10.402 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:10 smithi085 conmon[35961]: 02-10T04:30:10.014580+0000 2021-02-10T04:30:10.402 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:10 smithi085 conmon[35961]: mgr.y (mgr.14140) 393 : cluster [DBG] pgmap v439: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 117 MiB data, 380 MiB used, 715 GiB / 715 GiB avail; 2.2 MiB/s rd, 718 KiB/s wr, 6 op/s 2021-02-10T04:30:10.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:10 smithi085 conmon[29319]: audit 2021-02-10T04:30:09.455824+0000 mon.a (mon.0) 896 2021-02-10T04:30:10.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:10 smithi085 conmon[29319]: : audit [INF] from='client.? 172.21.15.85:0/3158734121' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-02-10T04:30:10.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:10 smithi085 conmon[29319]: audit 2021-02-10T04:30:09.517387+0000 mon.a (mon.0) 897 : audit [INF] from='client.? 172.21.15.85:0/2841810425' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:30:10.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:10 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:30:10.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:10 smithi085 conmon[29319]: 04:30:10.014580+0000 mgr.y (mgr.14140) 393 : cluster [DBG] pgmap v439: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 117 MiB data, 380 MiB used, 715 GiB / 715 GiB avail; 2.2 MiB/s rd, 718 KiB/s wr, 6 op/s 2021-02-10T04:30:10.457 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:30:10.612 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:30:10.719 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:30:10.750 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:30:10.754 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:10.825 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:30:10 smithi164 conmon[40892]: debug 2021-02-10T04:30:10.568+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:10.854 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:30:10.958 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:30:10.998 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:11.131 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:30:10 smithi164 conmon[44544]: debug 2021-02-10T04:30:10.823+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:11.166 INFO:tasks.rados.rados.0.smithi085.stdout:525: done (9 left) 2021-02-10T04:30:11.166 INFO:tasks.rados.rados.0.smithi085.stdout:526: done (8 left) 2021-02-10T04:30:11.166 INFO:tasks.rados.rados.0.smithi085.stdout:527: done (7 left) 2021-02-10T04:30:11.166 INFO:tasks.rados.rados.0.smithi085.stdout:528: done (6 left) 2021-02-10T04:30:11.166 INFO:tasks.rados.rados.0.smithi085.stdout:529: read oid 11 snap -1 2021-02-10T04:30:11.166 INFO:tasks.rados.rados.0.smithi085.stdout:529: expect deleted 2021-02-10T04:30:11.166 INFO:tasks.rados.rados.0.smithi085.stdout:530: write oid 15 current snap is 49 2021-02-10T04:30:11.166 INFO:tasks.rados.rados.0.smithi085.stdout:530: seq_num 217 ranges {447119=786917,1665416=692733,2976554=514615} 2021-02-10T04:30:11.169 INFO:tasks.rados.rados.0.smithi085.stdout:530: writing smithi08575397-15 from 447119 to 1234036 tid 1 2021-02-10T04:30:11.177 INFO:tasks.rados.rados.0.smithi085.stdout:530: writing smithi08575397-15 from 1665416 to 2358149 tid 2 2021-02-10T04:30:11.184 INFO:tasks.rados.rados.0.smithi085.stdout:530: writing smithi08575397-15 from 2976554 to 3491169 tid 3 2021-02-10T04:30:11.185 INFO:tasks.rados.rados.0.smithi085.stdout:531: snap_create 2021-02-10T04:30:11.185 INFO:tasks.rados.rados.0.smithi085.stdout:530: finishing write tid 1 to smithi08575397-15 2021-02-10T04:30:11.188 INFO:tasks.rados.rados.0.smithi085.stdout:530: finishing write tid 2 to smithi08575397-15 2021-02-10T04:30:11.190 INFO:tasks.rados.rados.0.smithi085.stdout:530: finishing write tid 3 to smithi08575397-15 2021-02-10T04:30:11.191 INFO:tasks.rados.rados.0.smithi085.stdout:530: finishing write tid 4 to smithi08575397-15 2021-02-10T04:30:11.191 INFO:tasks.rados.rados.0.smithi085.stdout:530: finishing write tid 5 to smithi08575397-15 2021-02-10T04:30:11.191 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 15 v 104 (ObjNum 217 snap 49 seq_num 217) dirty exists 2021-02-10T04:30:11.191 INFO:tasks.rados.rados.0.smithi085.stdout:530: left oid 15 (ObjNum 217 snap 49 seq_num 217) 2021-02-10T04:30:11.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:11 smithi164 conmon[29288]: cluster 2021-02-10T04:30:10.147169+0000 mon.a (mon.0) 898 : cluster 2021-02-10T04:30:11.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:11 smithi164 conmon[29288]: [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:30:11.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:11 smithi164 conmon[29288]: audit 2021-02-10T04:30:10.157234+0000 mon.a (mon.0) 899 : audit [INF] from='client.? 172.21.15.85:0/3158734121' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2021-02-10T04:30:11.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:11 smithi164 conmon[29288]: audit 2021-02-10T04:30:10.157368+0000 mon.a (mon.0) 900 : audit [INF] from='client.? 172.21.15.85:0/2841810425' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:30:11.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:11 smithi164 conmon[29288]: cluster 2021-02-10T04:30:10.157451+0000 mon.a (mon.0) 901 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2021-02-10T04:30:11.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:11 smithi164 conmon[29288]: cluster 2021-02-10T04:30:10.487522+0000 osd.5 (osd.5) 50 : cluster [DBG] 5.f scrub ok 2021-02-10T04:30:11.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:11 smithi085 conmon[29319]: cluster 2021-02-10T04:30:10.147169+0000 2021-02-10T04:30:11.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:11 smithi085 conmon[29319]: mon.a (mon.0) 898 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:30:11.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:11 smithi085 conmon[29319]: audit 2021-02-10T04:30:10.157234+0000 mon.a (mon.0) 899 : audit [INF] from='client.? 172.21.15.85:0/3158734121' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2021-02-10T04:30:11.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:11 smithi085 conmon[29319]: audit 2021-02-10T04:30:10.157368+0000 mon.a (mon.0) 900 : audit [INF] from='client.? 172.21.15.85:0/2841810425' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:30:11.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:11 smithi085 conmon[29319]: cluster 2021-02-10T04:30:10.157451+0000 mon.a (mon.0) 901 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2021-02-10T04:30:11.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:11 smithi085 conmon[29319]: cluster 2021-02-10T04:30:10.487522+0000 osd.5 (osd.5) 50 : cluster [DBG] 5.f scrub ok 2021-02-10T04:30:11.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:11 smithi085 conmon[35961]: cluster 2021-02-10T04:30:10. 2021-02-10T04:30:11.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:11 smithi085 conmon[35961]: 147169+0000 mon.a (mon.0) 898 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:30:11.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:11 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:30:11.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:11 smithi085 conmon[35961]: 04:30:10.157234+0000 mon.a (mon.0) 899 : audit [INF] 2021-02-10T04:30:11.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:11 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/3158734121' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2021-02-10T04:30:11.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:11 smithi085 conmon[35961]: audit 2021-02-10T04:30:11.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:11 smithi085 conmon[35961]: 2021-02-10T04:30:10.157368+0000 2021-02-10T04:30:11.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:11 smithi085 conmon[35961]: mon.a (mon.0) 900 : audit [INF] 2021-02-10T04:30:11.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:11 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/2841810425' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:30:11.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:11 smithi085 conmon[35961]: cluster 2021-02-10T04:30:11.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:11 smithi085 conmon[35961]: 2021 2021-02-10T04:30:11.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:11 smithi085 conmon[35961]: -02-10T04:30:10 2021-02-10T04:30:11.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:11 smithi085 conmon[35961]: .157451+0000 mon.a (mon.0) 901 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2021-02-10T04:30:11.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:11 smithi085 conmon[35961]: cluster 2021-02-10T04:30:10.487522+0000 osd.5 (osd.5) 50 : cluster [DBG] 2021-02-10T04:30:11.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:11 smithi085 conmon[35961]: 5.f scrub ok 2021-02-10T04:30:11.752 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:30:11.941 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:30:11 smithi085 conmon[41540]: debug 2021-02-10T04:30:11.941 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:30:11 smithi085 conmon[41540]: 2021-02-10T04:30:11.583+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:11.941 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:30:11 smithi085 conmon[41540]: 2021-02-10T04:30:12.170 INFO:tasks.rados.rados.0.smithi085.stdout:529: done (8 left) 2021-02-10T04:30:12.170 INFO:tasks.rados.rados.0.smithi085.stdout:530: done (7 left) 2021-02-10T04:30:12.170 INFO:tasks.rados.rados.0.smithi085.stdout:531: done (6 left) 2021-02-10T04:30:12.170 INFO:tasks.rados.rados.0.smithi085.stdout:532: snap_remove snap 48 2021-02-10T04:30:12.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[29319]: cluster 2021-02-10T04:30:12.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[29319]: 2021-02-10T04:30:12.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[29319]: 2021-02-10T04:30:10 2021-02-10T04:30:12.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[29319]: .905898+0000 osd.2 (osd.2) 47 : cluster [DBG] 7.8 scrub ok 2021-02-10T04:30:12.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:30:12.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[29319]: -10T04:30:11.158950+0000 mon.a ( 2021-02-10T04:30:12.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[29319]: mon.0) 902 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2021-02-10T04:30:12.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:30:12.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[29319]: 02-10T04:30:11.170030+0000 osd.1 (osd 2021-02-10T04:30:12.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[29319]: .1) 78 : cluster [DBG] 4.12 scrub ok 2021-02-10T04:30:12.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:30:12.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[29319]: 02-10T04:30:11.485001+0000 osd.3 2021-02-10T04:30:12.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[29319]: (osd.3) 52 : cluster [DBG] 2.19 scrub ok 2021-02-10T04:30:12.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:30:12.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[29319]: -10T04:30:11.492784+0000 osd.5 ( 2021-02-10T04:30:12.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[29319]: osd.5) 51 : cluster [DBG] 2.20 scrub ok 2021-02-10T04:30:12.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[29319]: cluster 2021-02-10T04:30:11.537130+0000 2021-02-10T04:30:12.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[29319]: osd.6 (osd.6) 43 : cluster [DBG] 2.1d scrub ok 2021-02-10T04:30:12.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:30:12.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[29319]: 30:12.015109+0000 mgr.y (mgr.14140) 394 : cluster [DBG] 2021-02-10T04:30:12.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[29319]: pgmap v442: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 118 MiB data, 378 MiB used, 715 GiB / 715 GiB avail; 7.3 MiB/s rd, 2.4 MiB/s wr, 17 op/s 2021-02-10T04:30:12.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[35961]: cluster 2021-02-10T04:30:10.905898+0000 osd.2 ( 2021-02-10T04:30:12.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[35961]: osd.2) 47 : cluster [DBG] 7.8 scrub ok 2021-02-10T04:30:12.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[35961]: cluster 2021-02-10T04:30:11.158950+0000 mon.a (mon.0) 902 : cluster 2021-02-10T04:30:12.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[35961]: [DBG] osdmap e195: 8 total, 8 up, 8 in 2021-02-10T04:30:12.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[35961]: cluster 2021-02-10T04:30:12.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[35961]: 2021-02-10T04:30:11.170030+0000 2021-02-10T04:30:12.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[35961]: osd.1 (osd.1) 78 : cluster [DBG] 4.12 scrub ok 2021-02-10T04:30:12.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:30:12.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[35961]: -10T04:30:11.485001+0000 osd.3 (osd 2021-02-10T04:30:12.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[35961]: .3) 52 : cluster [DBG] 2.19 scrub ok 2021-02-10T04:30:12.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:30:12.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[35961]: 02-10T04:30:11.492784+0000 2021-02-10T04:30:12.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[35961]: osd.5 (osd.5) 51 : cluster [DBG] 2.20 scrub ok 2021-02-10T04:30:12.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[35961]: 2021-02-10T04:30:12.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[35961]: cluster 2021-02-10T04:30: 2021-02-10T04:30:12.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[35961]: 11.537130+0000 osd.6 (osd.6) 43 : 2021-02-10T04:30:12.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[35961]: cluster [DBG] 2.1d scrub ok 2021-02-10T04:30:12.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:30:12.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[35961]: 02-10T04:30:12.015109+0000 2021-02-10T04:30:12.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[35961]: mgr.y (mgr.14140) 394 : cluster [DBG] 2021-02-10T04:30:12.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:12 smithi085 conmon[35961]: pgmap v442: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 118 MiB data, 378 MiB used, 715 GiB / 715 GiB avail; 7.3 MiB/s rd, 2.4 MiB/s wr, 17 op/s 2021-02-10T04:30:12.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:12 smithi164 conmon[29288]: cluster 2021-02-10T04:30:10.905898+0000 osd.2 (osd.2) 47 : cluster [DBG] 7.8 scrub ok 2021-02-10T04:30:12.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:12 smithi164 conmon[29288]: cluster 2021-02-10T04:30:11.158950+0000 mon.a (mon.0) 902 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2021-02-10T04:30:12.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:12 smithi164 conmon[29288]: cluster 2021-02-10T04:30:11.170030+0000 osd.1 (osd.1) 78 : cluster 2021-02-10T04:30:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:12 smithi164 conmon[29288]: [DBG] 4.12 scrub ok 2021-02-10T04:30:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:12 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:30:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:12 smithi164 conmon[29288]: 10T04:30:11.485001+0000 osd.3 (osd.3) 52 : cluster 2021-02-10T04:30:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:12 smithi164 conmon[29288]: [DBG] 2.19 scrub ok 2021-02-10T04:30:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:12 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:30:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:12 smithi164 conmon[29288]: :30:11.492784+0000 osd.5 (osd.5) 2021-02-10T04:30:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:12 smithi164 conmon[29288]: 51 : cluster [DBG] 2.20 scrub ok 2021-02-10T04:30:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:12 smithi164 conmon[29288]: cluster 2021-02-10T04:30:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:12 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:30:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:12 smithi164 conmon[29288]: 04:30:11.537130+0000 osd.6 (osd. 2021-02-10T04:30:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:12 smithi164 conmon[29288]: 6) 43 : cluster [DBG] 2.1d scrub ok 2021-02-10T04:30:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:12 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:30:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:12 smithi164 conmon[29288]: -02-10T04:30:12.015109+0000 2021-02-10T04:30:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:12 smithi164 conmon[29288]: mgr.y (mgr.14140) 394 : cluster [DBG] pgmap v442: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 118 MiB data, 378 MiB used, 715 GiB / 715 GiB avail; 7.3 MiB/s rd, 2.4 MiB/s wr, 17 op/s 2021-02-10T04:30:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:12 smithi164 conmon[29288]: 2021-02-10T04:30:13.060 INFO:tasks.rados.rados.0.smithi085.stdout:532: done (6 left) 2021-02-10T04:30:13.061 INFO:tasks.rados.rados.0.smithi085.stdout:533: write (excl) oid 39 current snap is 50 2021-02-10T04:30:13.061 INFO:tasks.rados.rados.0.smithi085.stdout:533: seq_num 218 ranges {463926=462166,1723898=436391,2603977=725887} 2021-02-10T04:30:13.061 INFO:tasks.rados.rados.0.smithi085.stdout:533: writing smithi08575397-39 from 463926 to 926092 tid 1 2021-02-10T04:30:13.069 INFO:tasks.rados.rados.0.smithi085.stdout:533: writing smithi08575397-39 from 1723898 to 2160289 tid 2 2021-02-10T04:30:13.077 INFO:tasks.rados.rados.0.smithi085.stdout:533: writing smithi08575397-39 from 2603977 to 3329864 tid 3 2021-02-10T04:30:13.077 INFO:tasks.rados.rados.0.smithi085.stdout:534: delete oid 26 current snap is 50 2021-02-10T04:30:13.077 INFO:tasks.rados.rados.0.smithi085.stdout:533: finishing write tid 1 to smithi08575397-39 2021-02-10T04:30:13.079 INFO:tasks.rados.rados.0.smithi085.stdout:534: done (7 left) 2021-02-10T04:30:13.079 INFO:tasks.rados.rados.0.smithi085.stdout:535: snap_remove snap 49 2021-02-10T04:30:13.324 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:13 smithi164 conmon[29288]: cluster 2021-02-10T04:30:12.168586+0000 mon.a ( 2021-02-10T04:30:13.324 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:13 smithi164 conmon[29288]: mon.0) 903 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2021-02-10T04:30:13.324 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:13 smithi164 conmon[29288]: cluster 2021-02-10T04:30:12.451994+0000 osd.3 (osd.3) 53 : cluster [DBG] 1.4 scrub ok 2021-02-10T04:30:13.324 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:13 smithi164 conmon[29288]: cluster 2021-02-10T04:30:12.538539+0000 osd.5 (osd.5) 52 : cluster [DBG] 2.12 scrub ok 2021-02-10T04:30:13.324 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:13 smithi164 conmon[29288]: cluster 2021-02-10T04:30:13.055496+0000 mon.a (mon.0) 904 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2021-02-10T04:30:13.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:13 smithi085 conmon[29319]: cluster 2021-02-10T04:30:12.168586+0000 mon.a (mon.0) 903 : cluster [DBG] 2021-02-10T04:30:13.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:13 smithi085 conmon[29319]: osdmap e196: 8 total, 8 up, 8 in 2021-02-10T04:30:13.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:13 smithi085 conmon[29319]: cluster 2021-02-10T04:30:12.451994+0000 osd.3 (osd.3) 53 : cluster [DBG] 1.4 scrub ok 2021-02-10T04:30:13.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:13 smithi085 conmon[29319]: cluster 2021-02-10T04:30:13.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:13 smithi085 conmon[29319]: 2021-02-10T04:30:12.538539+0000 osd.5 (osd.5) 52 : cluster [DBG] 2.12 scrub ok 2021-02-10T04:30:13.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:13 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:30:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:13 smithi085 conmon[29319]: T04:30:13.055496+0000 mon.a (mon.0) 904 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2021-02-10T04:30:13.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:13 smithi085 conmon[35961]: cluster 2021-02-10T04:30:13.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:13 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:30:13.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:13 smithi085 conmon[35961]: 04:30:12.168586+0000 mon.a (mon.0 2021-02-10T04:30:13.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:13 smithi085 conmon[35961]: ) 903 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2021-02-10T04:30:13.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:13 smithi085 conmon[35961]: cluster 2021-02-10T04:30:13.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:13 smithi085 conmon[35961]: 2021-02-10T04:30:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:13 smithi085 conmon[35961]: 2021 2021-02-10T04:30:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:13 smithi085 conmon[35961]: - 2021-02-10T04:30:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:13 smithi085 conmon[35961]: 02 2021-02-10T04:30:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:13 smithi085 conmon[35961]: - 2021-02-10T04:30:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:13 smithi085 conmon[35961]: 10 2021-02-10T04:30:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:13 smithi085 conmon[35961]: T04:30:12.451994+0000 2021-02-10T04:30:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:13 smithi085 conmon[35961]: osd.3 (osd.3) 53 : cluster [DBG] 1.4 scrub ok 2021-02-10T04:30:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:13 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:30:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:13 smithi085 conmon[35961]: 02-10T04: 2021-02-10T04:30:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:13 smithi085 conmon[35961]: 30:12.538539+0000 osd.5 (osd.5) 2021-02-10T04:30:13.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:13 smithi085 conmon[35961]: 52 : cluster [DBG] 2.12 scrub ok 2021-02-10T04:30:13.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:13 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:30:13.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:13 smithi085 conmon[35961]: -10T04:30:13.055496+0000 mon.a (mon. 2021-02-10T04:30:13.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:13 smithi085 conmon[35961]: 0) 904 : cluster [DBG] 2021-02-10T04:30:13.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:13 smithi085 conmon[35961]: osdmap e197: 8 total, 8 up, 8 in 2021-02-10T04:30:13.597 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:30:13.597 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:30:14.064 INFO:tasks.rados.rados.0.smithi085.stdout:533: finishing write tid 2 to smithi08575397-39 2021-02-10T04:30:14.065 INFO:tasks.rados.rados.0.smithi085.stdout:533: finishing write tid 3 to smithi08575397-39 2021-02-10T04:30:14.065 INFO:tasks.rados.rados.0.smithi085.stdout:533: finishing write tid 4 to smithi08575397-39 2021-02-10T04:30:14.065 INFO:tasks.rados.rados.0.smithi085.stdout:533: finishing write tid 5 to smithi08575397-39 2021-02-10T04:30:14.065 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 39 v 66 (ObjNum 218 snap 50 seq_num 218) dirty exists 2021-02-10T04:30:14.065 INFO:tasks.rados.rados.0.smithi085.stdout:533: left oid 39 (ObjNum 218 snap 50 seq_num 218) 2021-02-10T04:30:14.065 INFO:tasks.rados.rados.0.smithi085.stdout:533: done (7 left) 2021-02-10T04:30:14.066 INFO:tasks.rados.rados.0.smithi085.stdout:535: done (6 left) 2021-02-10T04:30:14.066 INFO:tasks.rados.rados.0.smithi085.stdout:536: write oid 24 current snap is 50 2021-02-10T04:30:14.066 INFO:tasks.rados.rados.0.smithi085.stdout:536: seq_num 220 ranges {497540=612664,1840862=440060,3058823=464784} 2021-02-10T04:30:14.071 INFO:tasks.rados.rados.0.smithi085.stdout:536: writing smithi08575397-24 from 497540 to 1110204 tid 1 2021-02-10T04:30:14.078 INFO:tasks.rados.rados.0.smithi085.stdout:536: writing smithi08575397-24 from 1840862 to 2280922 tid 2 2021-02-10T04:30:14.086 INFO:tasks.rados.rados.0.smithi085.stdout:536: writing smithi08575397-24 from 3058823 to 3523607 tid 3 2021-02-10T04:30:14.092 INFO:tasks.rados.rados.0.smithi085.stdout:536: finishing write tid 1 to smithi08575397-24 2021-02-10T04:30:14.092 INFO:tasks.rados.rados.0.smithi085.stdout:536: finishing write tid 2 to smithi08575397-24 2021-02-10T04:30:14.092 INFO:tasks.rados.rados.0.smithi085.stdout:537: rollback oid 46 current snap is 50 2021-02-10T04:30:14.092 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 46 to 47 2021-02-10T04:30:14.092 INFO:tasks.rados.rados.0.smithi085.stdout:538: copy_from oid 42 from oid 32 current snap is 50 2021-02-10T04:30:14.093 INFO:tasks.rados.rados.0.smithi085.stdout:539: write oid 11 current snap is 50 2021-02-10T04:30:14.093 INFO:tasks.rados.rados.0.smithi085.stdout:539: seq_num 221 ranges {514347=404266,1499344=700071,2686246=676056} 2021-02-10T04:30:14.093 INFO:tasks.rados.rados.0.smithi085.stdout:539: writing smithi08575397-11 from 514347 to 918613 tid 1 2021-02-10T04:30:14.100 INFO:tasks.rados.rados.0.smithi085.stdout:539: writing smithi08575397-11 from 1499344 to 2199415 tid 2 2021-02-10T04:30:14.108 INFO:tasks.rados.rados.0.smithi085.stdout:539: writing smithi08575397-11 from 2686246 to 3362302 tid 3 2021-02-10T04:30:14.108 INFO:tasks.rados.rados.0.smithi085.stdout:540: write (excl) oid 30 current snap is 50 2021-02-10T04:30:14.108 INFO:tasks.rados.rados.0.smithi085.stdout:537: finishing rollback tid 0 to smithi08575397-46 2021-02-10T04:30:14.108 INFO:tasks.rados.rados.0.smithi085.stdout:540: seq_num 222 ranges {531154=479515,1557826=443729,2713669=487328,3679295=51859} 2021-02-10T04:30:14.114 INFO:tasks.rados.rados.0.smithi085.stdout:540: writing smithi08575397-30 from 531154 to 1010669 tid 1 2021-02-10T04:30:14.120 INFO:tasks.rados.rados.0.smithi085.stdout:540: writing smithi08575397-30 from 1557826 to 2001555 tid 2 2021-02-10T04:30:14.128 INFO:tasks.rados.rados.0.smithi085.stdout:540: writing smithi08575397-30 from 2713669 to 3200997 tid 3 2021-02-10T04:30:14.132 INFO:tasks.rados.rados.0.smithi085.stdout:540: writing smithi08575397-30 from 3679295 to 3731154 tid 4 2021-02-10T04:30:14.132 INFO:tasks.rados.rados.0.smithi085.stdout:536: finishing write tid 3 to smithi08575397-24 2021-02-10T04:30:14.132 INFO:tasks.rados.rados.0.smithi085.stdout:536: finishing write tid 4 to smithi08575397-24 2021-02-10T04:30:14.132 INFO:tasks.rados.rados.0.smithi085.stdout:537: finishing rollback tid 1 to smithi08575397-46 2021-02-10T04:30:14.132 INFO:tasks.rados.rados.0.smithi085.stdout:536: finishing write tid 5 to smithi08575397-24 2021-02-10T04:30:14.133 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 24 v 54 (ObjNum 220 snap 50 seq_num 220) dirty exists 2021-02-10T04:30:14.133 INFO:tasks.rados.rados.0.smithi085.stdout:536: left oid 24 (ObjNum 220 snap 50 seq_num 220) 2021-02-10T04:30:14.133 INFO:tasks.rados.rados.0.smithi085.stdout:537: finishing rollback tid 2 to smithi08575397-46 2021-02-10T04:30:14.133 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 171 (ObjNum 207 snap 46 seq_num 207) dirty exists 2021-02-10T04:30:14.133 INFO:tasks.rados.rados.0.smithi085.stdout:539: finishing write tid 1 to smithi08575397-11 2021-02-10T04:30:14.133 INFO:tasks.rados.rados.0.smithi085.stdout:539: finishing write tid 2 to smithi08575397-11 2021-02-10T04:30:14.133 INFO:tasks.rados.rados.0.smithi085.stdout:536: done (10 left) 2021-02-10T04:30:14.133 INFO:tasks.rados.rados.0.smithi085.stdout:537: done (9 left) 2021-02-10T04:30:14.133 INFO:tasks.rados.rados.0.smithi085.stdout:541: snap_remove snap 47 2021-02-10T04:30:14.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:14 smithi085 conmon[29319]: cluster 2021-02-10T04:30:14.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:14 smithi085 conmon[29319]: 2021-02-10T04:30 2021-02-10T04:30:14.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:14 smithi085 conmon[29319]: :13.089484+0000 osd.4 (osd.4) 48 : cluster [DBG] 4.1b scrub ok 2021-02-10T04:30:14.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:14 smithi085 conmon[29319]: cluster 2021-02-10T04:30:14.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:14 smithi085 conmon[29319]: 2021-02-10T04:30 2021-02-10T04:30:14.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:14 smithi085 conmon[29319]: :13.526163+0000 osd.6 (osd 2021-02-10T04:30:14.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:14 smithi085 conmon[29319]: .6) 44 : cluster [DBG] 4.1f scrub ok 2021-02-10T04:30:14.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:14 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:30:14.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:14 smithi085 conmon[29319]: -10T04:30:13.863425+0000 osd.0 (osd.0 2021-02-10T04:30:14.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:14 smithi085 conmon[29319]: ) 55 : cluster [DBG] 7.6 scrub ok 2021-02-10T04:30:14.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:14 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:30:14.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:14 smithi085 conmon[29319]: -10T04:30:14.015560+0000 mgr.y ( 2021-02-10T04:30:14.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:14 smithi085 conmon[29319]: mgr.14140) 395 : cluster [DBG] pgmap v445: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 117 MiB data, 381 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 2.1 MiB/s wr, 16 op/s 2021-02-10T04:30:14.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:14 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:30:14.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:14 smithi085 conmon[29319]: -02-10T04:30:14.064821+0000 2021-02-10T04:30:14.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:14 smithi085 conmon[29319]: mon.a (mon.0) 905 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2021-02-10T04:30:14.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:14 smithi085 conmon[29319]: 2021-02-10T04:30:14.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:14 smithi085 conmon[35961]: cluster 2021-02-10T04:30:13.089484+0000 osd.4 (osd.4 2021-02-10T04:30:14.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:14 smithi085 conmon[35961]: ) 48 : cluster [DBG] 4.1b scrub ok 2021-02-10T04:30:14.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:14 smithi085 conmon[35961]: cluster 2021-02-10T04:30:13.526163+0000 osd.6 (osd.6) 44 : cluster [DBG] 4.1f scrub ok 2021-02-10T04:30:14.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:14 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:30:14.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:14 smithi085 conmon[35961]: T04:30:13.863425+0000 osd.0 (osd.0) 55 : cluster [DBG] 7.6 scrub ok 2021-02-10T04:30:14.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:14 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:30:14.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:14 smithi085 conmon[35961]: 02-10T04:30:14.015560+0000 mgr.y (mgr.14140) 2021-02-10T04:30:14.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:14 smithi085 conmon[35961]: 395 : cluster [DBG] pgmap v445: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 117 MiB data, 381 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 2.1 MiB/s wr, 16 op/s 2021-02-10T04:30:14.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:14 smithi085 conmon[35961]: cluster 2021-02-10T04:30 2021-02-10T04:30:14.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:14 smithi085 conmon[35961]: :14.064821+0000 mon.a (mon.0) 2021-02-10T04:30:14.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:14 smithi085 conmon[35961]: 905 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2021-02-10T04:30:14.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:14 smithi164 conmon[29288]: cluster 2021-02-10T04:30:13.089484+0000 osd.4 (osd 2021-02-10T04:30:14.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:14 smithi164 conmon[29288]: .4) 48 : cluster [DBG] 4.1b scrub ok 2021-02-10T04:30:14.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:14 smithi164 conmon[29288]: cluster 2021-02-10T04:30:13.526163+0000 osd.6 (osd.6) 44 : cluster [DBG] 4.1f scrub ok 2021-02-10T04:30:14.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:14 smithi164 conmon[29288]: cluster 2021-02-10T04:30:13.863425+0000 osd.0 (osd.0) 55 : cluster [DBG] 7.6 scrub ok 2021-02-10T04:30:14.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:14 smithi164 conmon[29288]: cluster 2021-02-10T04:30:14.015560+0000 mgr.y (mgr.14140) 395 : cluster [DBG] pgmap v445: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 117 MiB data, 381 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 2.1 MiB/s wr, 16 op/s 2021-02-10T04:30:14.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:14 smithi164 conmon[29288]: cluster 2021-02-10T04:30:14.064821+0000 mon.a (mon.0) 905 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2021-02-10T04:30:14.786 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:30:14.818 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:15.068 INFO:tasks.rados.rados.0.smithi085.stdout:541: done (9 left) 2021-02-10T04:30:15.069 INFO:tasks.rados.rados.0.smithi085.stdout:542: write (excl) oid 9 current snap is 50 2021-02-10T04:30:15.069 INFO:tasks.rados.rados.0.smithi085.stdout:542: seq_num 223 ranges {547961=554764,1616308=703740,2741092=698600} 2021-02-10T04:30:15.072 INFO:tasks.rados.rados.0.smithi085.stdout:542: writing smithi08575397-9 from 547961 to 1102725 tid 1 2021-02-10T04:30:15.080 INFO:tasks.rados.rados.0.smithi085.stdout:542: writing smithi08575397-9 from 1616308 to 2320048 tid 2 2021-02-10T04:30:15.088 INFO:tasks.rados.rados.0.smithi085.stdout:542: writing smithi08575397-9 from 2741092 to 3439692 tid 3 2021-02-10T04:30:15.088 INFO:tasks.rados.rados.0.smithi085.stdout:543: copy_from oid 50 from oid 44 current snap is 50 2021-02-10T04:30:15.088 INFO:tasks.rados.rados.0.smithi085.stdout:539: finishing write tid 3 to smithi08575397-11 2021-02-10T04:30:15.088 INFO:tasks.rados.rados.0.smithi085.stdout:539: finishing write tid 4 to smithi08575397-11 2021-02-10T04:30:15.088 INFO:tasks.rados.rados.0.smithi085.stdout:539: finishing write tid 5 to smithi08575397-11 2021-02-10T04:30:15.088 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 223 (ObjNum 221 snap 50 seq_num 221) dirty exists 2021-02-10T04:30:15.088 INFO:tasks.rados.rados.0.smithi085.stdout:539: left oid 11 (ObjNum 221 snap 50 seq_num 221) 2021-02-10T04:30:15.088 INFO:tasks.rados.rados.0.smithi085.stdout:540: finishing write tid 1 to smithi08575397-30 2021-02-10T04:30:15.088 INFO:tasks.rados.rados.0.smithi085.stdout:540: finishing write tid 2 to smithi08575397-30 2021-02-10T04:30:15.088 INFO:tasks.rados.rados.0.smithi085.stdout:540: finishing write tid 3 to smithi08575397-30 2021-02-10T04:30:15.088 INFO:tasks.rados.rados.0.smithi085.stdout:540: finishing write tid 4 to smithi08575397-30 2021-02-10T04:30:15.088 INFO:tasks.rados.rados.0.smithi085.stdout:540: finishing write tid 5 to smithi08575397-30 2021-02-10T04:30:15.089 INFO:tasks.rados.rados.0.smithi085.stdout:540: finishing write tid 6 to smithi08575397-30 2021-02-10T04:30:15.089 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 176 (ObjNum 222 snap 50 seq_num 222) dirty exists 2021-02-10T04:30:15.089 INFO:tasks.rados.rados.0.smithi085.stdout:540: left oid 30 (ObjNum 222 snap 50 seq_num 222) 2021-02-10T04:30:15.089 INFO:tasks.rados.rados.0.smithi085.stdout:542: finishing write tid 1 to smithi08575397-9 2021-02-10T04:30:15.089 INFO:tasks.rados.rados.0.smithi085.stdout:539: done (10 left) 2021-02-10T04:30:15.089 INFO:tasks.rados.rados.0.smithi085.stdout:540: done (9 left) 2021-02-10T04:30:15.089 INFO:tasks.rados.rados.0.smithi085.stdout:544: write oid 24 current snap is 50 2021-02-10T04:30:15.089 INFO:tasks.rados.rados.0.smithi085.stdout:544: seq_num 224 ranges {564768=630013,1674790=447398,2768515=626225} 2021-02-10T04:30:15.095 INFO:tasks.rados.rados.0.smithi085.stdout:544: writing smithi08575397-24 from 564768 to 1194781 tid 1 2021-02-10T04:30:15.101 INFO:tasks.rados.rados.0.smithi085.stdout:544: writing smithi08575397-24 from 1674790 to 2122188 tid 2 2021-02-10T04:30:15.109 INFO:tasks.rados.rados.0.smithi085.stdout:544: writing smithi08575397-24 from 2768515 to 3394740 tid 3 2021-02-10T04:30:15.110 INFO:tasks.rados.rados.0.smithi085.stdout:545: read oid 31 snap -1 2021-02-10T04:30:15.110 INFO:tasks.rados.rados.0.smithi085.stdout:545: expect (ObjNum 180 snap 40 seq_num 180) 2021-02-10T04:30:15.110 INFO:tasks.rados.rados.0.smithi085.stdout:543: finishing copy_from to smithi08575397-50 2021-02-10T04:30:15.110 INFO:tasks.rados.rados.0.smithi085.stdout:543: got expected ENOENT (src dne) 2021-02-10T04:30:15.110 INFO:tasks.rados.rados.0.smithi085.stdout:543: finishing copy_from racing read to smithi08575397-50 2021-02-10T04:30:15.110 INFO:tasks.rados.rados.0.smithi085.stdout:542: finishing write tid 2 to smithi08575397-9 2021-02-10T04:30:15.110 INFO:tasks.rados.rados.0.smithi085.stdout:542: finishing write tid 3 to smithi08575397-9 2021-02-10T04:30:15.110 INFO:tasks.rados.rados.0.smithi085.stdout:542: finishing write tid 4 to smithi08575397-9 2021-02-10T04:30:15.110 INFO:tasks.rados.rados.0.smithi085.stdout:542: finishing write tid 5 to smithi08575397-9 2021-02-10T04:30:15.110 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 9 v 153 (ObjNum 223 snap 50 seq_num 223) dirty exists 2021-02-10T04:30:15.110 INFO:tasks.rados.rados.0.smithi085.stdout:542: left oid 9 (ObjNum 223 snap 50 seq_num 223) 2021-02-10T04:30:15.111 INFO:tasks.rados.rados.0.smithi085.stdout:546: read oid 10 snap -1 2021-02-10T04:30:15.111 INFO:tasks.rados.rados.0.smithi085.stdout:546: expect (ObjNum 71 snap 7 seq_num 71) 2021-02-10T04:30:15.111 INFO:tasks.rados.rados.0.smithi085.stdout:542: done (11 left) 2021-02-10T04:30:15.111 INFO:tasks.rados.rados.0.smithi085.stdout:543: done (10 left) 2021-02-10T04:30:15.111 INFO:tasks.rados.rados.0.smithi085.stdout:547: snap_create 2021-02-10T04:30:15.111 INFO:tasks.rados.rados.0.smithi085.stdout:544: finishing write tid 1 to smithi08575397-24 2021-02-10T04:30:15.111 INFO:tasks.rados.rados.0.smithi085.stdout:544: finishing write tid 2 to smithi08575397-24 2021-02-10T04:30:15.113 INFO:tasks.rados.rados.0.smithi085.stdout:546: expect (ObjNum 71 snap 7 seq_num 71) 2021-02-10T04:30:15.307 INFO:tasks.rados.rados.0.smithi085.stdout:544: finishing write tid 3 to smithi08575397-24 2021-02-10T04:30:15.307 INFO:tasks.rados.rados.0.smithi085.stdout:544: finishing write tid 4 to smithi08575397-24 2021-02-10T04:30:15.307 INFO:tasks.rados.rados.0.smithi085.stdout:544: finishing write tid 5 to smithi08575397-24 2021-02-10T04:30:15.307 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 24 v 58 (ObjNum 224 snap 50 seq_num 224) dirty exists 2021-02-10T04:30:15.307 INFO:tasks.rados.rados.0.smithi085.stdout:544: left oid 24 (ObjNum 224 snap 50 seq_num 224) 2021-02-10T04:30:15.307 INFO:tasks.rados.rados.0.smithi085.stdout:545: expect (ObjNum 180 snap 40 seq_num 180) 2021-02-10T04:30:15.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:15 smithi085 conmon[29319]: cluster 2021-02-10T04:30:14.204525+0000 osd.1 (osd. 2021-02-10T04:30:15.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:15 smithi085 conmon[29319]: 1) 79 : cluster [DBG] 2.0 scrub ok 2021-02-10T04:30:15.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:15 smithi085 conmon[29319]: cluster 2021-02-10T04:30:14.478634+0000 osd.6 (osd.6) 45 : cluster [DBG] 7.7 scrub ok 2021-02-10T04:30:15.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:15 smithi085 conmon[29319]: cluster 2021-02-10T04:30:15.065759+0000 mon.a (mon.0) 906 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2021-02-10T04:30:15.441 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:30:15 smithi085 conmon[52836]: debug 2021-02-10T04:30:15.441 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:30:15 smithi085 conmon[52836]: 2021-02-10T04:30:15.300+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:15.442 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:30:15 smithi085 conmon[52836]: 2021-02-10T04:30:15.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:15 smithi085 conmon[35961]: cluster 2021-02-10T04:30:15.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:15 smithi085 conmon[35961]: 2021-02-10T04:30:15.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:15 smithi085 conmon[35961]: 2021 2021-02-10T04:30:15.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:15 smithi085 conmon[35961]: - 2021-02-10T04:30:15.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:15 smithi085 conmon[35961]: 02 2021-02-10T04:30:15.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:15 smithi085 conmon[35961]: - 2021-02-10T04:30:15.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:15 smithi085 conmon[35961]: 10T04:30:14.204525+0000 2021-02-10T04:30:15.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:15 smithi085 conmon[35961]: osd.1 (osd.1) 79 : cluster [DBG] 2.0 scrub ok 2021-02-10T04:30:15.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:15 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:30:15.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:15 smithi085 conmon[35961]: 02-10T04:30:14. 2021-02-10T04:30:15.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:15 smithi085 conmon[35961]: 478634+0000 osd.6 (osd.6 2021-02-10T04:30:15.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:15 smithi085 conmon[35961]: ) 45 : cluster [DBG] 7.7 scrub ok 2021-02-10T04:30:15.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:15 smithi085 conmon[35961]: cluster 2021-02-10T04:30:15.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:15 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:30:15.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:15 smithi085 conmon[35961]: :30:15.065759+0000 mon.a (mon.0) 906 : cluster [DBG] 2021-02-10T04:30:15.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:15 smithi085 conmon[35961]: osdmap e199: 8 total, 8 up, 8 in 2021-02-10T04:30:15.570 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:15 smithi164 conmon[29288]: cluster 2021-02-10T04:30:14.204525+0000 osd.1 ( 2021-02-10T04:30:15.570 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:15 smithi164 conmon[29288]: osd.1) 79 : cluster [DBG] 2.0 scrub ok 2021-02-10T04:30:15.570 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:15 smithi164 conmon[29288]: cluster 2021-02-10T04:30:14.478634+0000 osd.6 (osd.6) 45 : cluster [DBG] 7.7 scrub ok 2021-02-10T04:30:15.570 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:15 smithi164 conmon[29288]: cluster 2021-02-10T04:30:15.065759+0000 2021-02-10T04:30:15.570 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:15 smithi164 conmon[29288]: mon.a (mon.0) 906 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2021-02-10T04:30:15.578 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:30:15.591 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.6 dump_ops_in_flight 2021-02-10T04:30:15.846 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:30:16.068 INFO:tasks.rados.rados.0.smithi085.stdout:544: done (10 left) 2021-02-10T04:30:16.068 INFO:tasks.rados.rados.0.smithi085.stdout:545: done (9 left) 2021-02-10T04:30:16.068 INFO:tasks.rados.rados.0.smithi085.stdout:546: done (8 left) 2021-02-10T04:30:16.068 INFO:tasks.rados.rados.0.smithi085.stdout:547: done (7 left) 2021-02-10T04:30:16.068 INFO:tasks.rados.rados.0.smithi085.stdout:548: read oid 40 snap 50 2021-02-10T04:30:16.068 INFO:tasks.rados.rados.0.smithi085.stdout:548: expect (ObjNum 186 snap 41 seq_num 186) 2021-02-10T04:30:16.073 INFO:tasks.rados.rados.0.smithi085.stdout:549: read oid 23 snap -1 2021-02-10T04:30:16.073 INFO:tasks.rados.rados.0.smithi085.stdout:549: expect (ObjNum 124 snap 23 seq_num 124) 2021-02-10T04:30:16.074 INFO:tasks.rados.rados.0.smithi085.stdout:550: copy_from oid 36 from oid 22 current snap is 51 2021-02-10T04:30:16.074 INFO:tasks.rados.rados.0.smithi085.stdout:551: read oid 37 snap -1 2021-02-10T04:30:16.074 INFO:tasks.rados.rados.0.smithi085.stdout:551: expect (ObjNum 25 snap 0 seq_num 25) 2021-02-10T04:30:16.074 INFO:tasks.rados.rados.0.smithi085.stdout:552: copy_from oid 7 from oid 1 current snap is 51 2021-02-10T04:30:16.074 INFO:tasks.rados.rados.0.smithi085.stdout:553: read oid 45 snap -1 2021-02-10T04:30:16.074 INFO:tasks.rados.rados.0.smithi085.stdout:553: expect (ObjNum 96 snap 19 seq_num 96) 2021-02-10T04:30:16.074 INFO:tasks.rados.rados.0.smithi085.stdout:554: write oid 2 current snap is 51 2021-02-10T04:30:16.074 INFO:tasks.rados.rados.0.smithi085.stdout:554: seq_num 225 ranges {581575=705262,1733272=707409,3195938=437497} 2021-02-10T04:30:16.075 INFO:tasks.rados.rados.0.smithi085.stdout:554: writing smithi08575397-2 from 581575 to 1286837 tid 1 2021-02-10T04:30:16.083 INFO:tasks.rados.rados.0.smithi085.stdout:554: writing smithi08575397-2 from 1733272 to 2440681 tid 2 2021-02-10T04:30:16.092 INFO:tasks.rados.rados.0.smithi085.stdout:554: writing smithi08575397-2 from 3195938 to 3633435 tid 3 2021-02-10T04:30:16.092 INFO:tasks.rados.rados.0.smithi085.stdout:555: write oid 11 current snap is 51 2021-02-10T04:30:16.092 INFO:tasks.rados.rados.0.smithi085.stdout:555: seq_num 226 ranges {598382=780511,2075401=451067,3107008=648769} 2021-02-10T04:30:16.100 INFO:tasks.rados.rados.0.smithi085.stdout:555: writing smithi08575397-11 from 598382 to 1378893 tid 1 2021-02-10T04:30:16.108 INFO:tasks.rados.rados.0.smithi085.stdout:555: writing smithi08575397-11 from 2075401 to 2526468 tid 2 2021-02-10T04:30:16.115 INFO:tasks.rados.rados.0.smithi085.stdout:555: writing smithi08575397-11 from 3107008 to 3755777 tid 3 2021-02-10T04:30:16.115 INFO:tasks.rados.rados.0.smithi085.stdout:556: write (excl) oid 13 current snap is 51 2021-02-10T04:30:16.115 INFO:tasks.rados.rados.0.smithi085.stdout:556: seq_num 227 ranges {615189=455760,1733883=711078,3134431=460041} 2021-02-10T04:30:16.122 INFO:tasks.rados.rados.0.smithi085.stdout:556: writing smithi08575397-13 from 615189 to 1070949 tid 1 2021-02-10T04:30:16.130 INFO:tasks.rados.rados.0.smithi085.stdout:556: writing smithi08575397-13 from 1733883 to 2444961 tid 2 2021-02-10T04:30:16.137 INFO:tasks.rados.rados.0.smithi085.stdout:556: writing smithi08575397-13 from 3134431 to 3594472 tid 3 2021-02-10T04:30:16.137 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:30:16.137 INFO:tasks.rados.rados.0.smithi085.stdout:548: expect (ObjNum 186 snap 41 seq_num 186) 2021-02-10T04:30:16.360 INFO:tasks.rados.rados.0.smithi085.stdout:554: finishing write tid 1 to smithi08575397-2 2021-02-10T04:30:16.360 INFO:tasks.rados.rados.0.smithi085.stdout:553: expect (ObjNum 96 snap 19 seq_num 96) 2021-02-10T04:30:16.441 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:30:16 smithi085 conmon[41540]: debug 2021-02-10T04:30:16.441 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:30:16 smithi085 conmon[41540]: 2021-02-10T04:30:16.075+0000 7fb94735a700 -1 osd.0 pg_epoch: 200 pg[7.6( v 198'38 (0'0,198'38] local-lis/les=93/94 n=3 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [0,7,3] r=0 lpr=93 crt=198'38 lcod 197'36 mlcod 197'36 active+clean+snaptrim trimq=[60~1] ps=[5a~2,5e~1,61~2]] removing snap head 2021-02-10T04:30:16.441 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:30:16 smithi085 conmon[41540]: 2021-02-10T04:30:16.562 INFO:tasks.rados.rados.0.smithi085.stdout:549: expect (ObjNum 124 snap 23 seq_num 124) 2021-02-10T04:30:16.652 INFO:tasks.rados.rados.0.smithi085.stdout:554: finishing write tid 2 to smithi08575397-2 2021-02-10T04:30:16.652 INFO:tasks.rados.rados.0.smithi085.stdout:548: done (15 left) 2021-02-10T04:30:16.652 INFO:tasks.rados.rados.0.smithi085.stdout:549: done (14 left) 2021-02-10T04:30:16.652 INFO:tasks.rados.rados.0.smithi085.stdout:553: done (13 left) 2021-02-10T04:30:16.653 INFO:tasks.rados.rados.0.smithi085.stdout:557: delete oid 45 current snap is 51 2021-02-10T04:30:16.653 INFO:tasks.rados.rados.0.smithi085.stdout:551: expect (ObjNum 25 snap 0 seq_num 25) 2021-02-10T04:30:16.758 INFO:tasks.rados.rados.0.smithi085.stdout:554: finishing write tid 3 to smithi08575397-2 2021-02-10T04:30:16.758 INFO:tasks.rados.rados.0.smithi085.stdout:554: finishing write tid 4 to smithi08575397-2 2021-02-10T04:30:16.758 INFO:tasks.rados.rados.0.smithi085.stdout:554: finishing write tid 5 to smithi08575397-2 2021-02-10T04:30:16.758 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 75 (ObjNum 225 snap 51 seq_num 225) dirty exists 2021-02-10T04:30:16.759 INFO:tasks.rados.rados.0.smithi085.stdout:554: left oid 2 (ObjNum 225 snap 51 seq_num 225) 2021-02-10T04:30:16.759 INFO:tasks.rados.rados.0.smithi085.stdout:555: finishing write tid 1 to smithi08575397-11 2021-02-10T04:30:16.759 INFO:tasks.rados.rados.0.smithi085.stdout:555: finishing write tid 2 to smithi08575397-11 2021-02-10T04:30:16.759 INFO:tasks.rados.rados.0.smithi085.stdout:551: done (13 left) 2021-02-10T04:30:16.759 INFO:tasks.rados.rados.0.smithi085.stdout:554: done (12 left) 2021-02-10T04:30:16.759 INFO:tasks.rados.rados.0.smithi085.stdout:557: done (11 left) 2021-02-10T04:30:16.759 INFO:tasks.rados.rados.0.smithi085.stdout:558: write oid 9 current snap is 51 2021-02-10T04:30:16.759 INFO:tasks.rados.rados.0.smithi085.stdout:555: finishing write tid 3 to smithi08575397-11 2021-02-10T04:30:16.759 INFO:tasks.rados.rados.0.smithi085.stdout:558: seq_num 229 ranges {648803=722611,1967200=714747,3421983=426820} 2021-02-10T04:30:16.766 INFO:tasks.rados.rados.0.smithi085.stdout:558: writing smithi08575397-9 from 648803 to 1371414 tid 1 2021-02-10T04:30:16.774 INFO:tasks.rados.rados.0.smithi085.stdout:558: writing smithi08575397-9 from 1967200 to 2681947 tid 2 2021-02-10T04:30:16.781 INFO:tasks.rados.rados.0.smithi085.stdout:558: writing smithi08575397-9 from 3421983 to 3848803 tid 3 2021-02-10T04:30:16.782 INFO:tasks.rados.rados.0.smithi085.stdout:559: snap_create 2021-02-10T04:30:16.782 INFO:tasks.rados.rados.0.smithi085.stdout:555: finishing write tid 4 to smithi08575397-11 2021-02-10T04:30:16.782 INFO:tasks.rados.rados.0.smithi085.stdout:555: finishing write tid 5 to smithi08575397-11 2021-02-10T04:30:16.782 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 232 (ObjNum 226 snap 51 seq_num 226) dirty exists 2021-02-10T04:30:16.782 INFO:tasks.rados.rados.0.smithi085.stdout:555: left oid 11 (ObjNum 226 snap 51 seq_num 226) 2021-02-10T04:30:16.782 INFO:tasks.rados.rados.0.smithi085.stdout:550: finishing copy_from to smithi08575397-36 2021-02-10T04:30:16.782 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 90 (ObjNum 127 snap 23 seq_num 127) dirty exists 2021-02-10T04:30:16.782 INFO:tasks.rados.rados.0.smithi085.stdout:550: finishing copy_from racing read to smithi08575397-36 2021-02-10T04:30:16.782 INFO:tasks.rados.rados.0.smithi085.stdout:556: finishing write tid 1 to smithi08575397-13 2021-02-10T04:30:16.782 INFO:tasks.rados.rados.0.smithi085.stdout:552: finishing copy_from to smithi08575397-7 2021-02-10T04:30:16.783 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 93 (ObjNum 202 snap 44 seq_num 202) dirty exists 2021-02-10T04:30:16.783 INFO:tasks.rados.rados.0.smithi085.stdout:552: finishing copy_from racing read to smithi08575397-7 2021-02-10T04:30:16.783 INFO:tasks.rados.rados.0.smithi085.stdout:556: finishing write tid 2 to smithi08575397-13 2021-02-10T04:30:16.783 INFO:tasks.rados.rados.0.smithi085.stdout:556: finishing write tid 3 to smithi08575397-13 2021-02-10T04:30:16.783 INFO:tasks.rados.rados.0.smithi085.stdout:556: finishing write tid 4 to smithi08575397-13 2021-02-10T04:30:16.783 INFO:tasks.rados.rados.0.smithi085.stdout:556: finishing write tid 5 to smithi08575397-13 2021-02-10T04:30:16.783 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 187 (ObjNum 227 snap 51 seq_num 227) dirty exists 2021-02-10T04:30:16.783 INFO:tasks.rados.rados.0.smithi085.stdout:556: left oid 13 (ObjNum 227 snap 51 seq_num 227) 2021-02-10T04:30:16.783 INFO:tasks.rados.rados.0.smithi085.stdout:558: finishing write tid 1 to smithi08575397-9 2021-02-10T04:30:16.783 INFO:tasks.rados.rados.0.smithi085.stdout:558: finishing write tid 2 to smithi08575397-9 2021-02-10T04:30:16.784 INFO:tasks.rados.rados.0.smithi085.stdout:558: finishing write tid 3 to smithi08575397-9 2021-02-10T04:30:16.784 INFO:tasks.rados.rados.0.smithi085.stdout:558: finishing write tid 4 to smithi08575397-9 2021-02-10T04:30:16.784 INFO:tasks.rados.rados.0.smithi085.stdout:558: finishing write tid 5 to smithi08575397-9 2021-02-10T04:30:16.784 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 9 v 158 (ObjNum 229 snap 51 seq_num 229) dirty exists 2021-02-10T04:30:16.784 INFO:tasks.rados.rados.0.smithi085.stdout:558: left oid 9 (ObjNum 229 snap 51 seq_num 229) 2021-02-10T04:30:17.184 INFO:tasks.rados.rados.0.smithi085.stdout:550: done (12 left) 2021-02-10T04:30:17.184 INFO:tasks.rados.rados.0.smithi085.stdout:552: done (11 left) 2021-02-10T04:30:17.184 INFO:tasks.rados.rados.0.smithi085.stdout:555: done (10 left) 2021-02-10T04:30:17.184 INFO:tasks.rados.rados.0.smithi085.stdout:556: done (9 left) 2021-02-10T04:30:17.184 INFO:tasks.rados.rados.0.smithi085.stdout:558: done (8 left) 2021-02-10T04:30:17.184 INFO:tasks.rados.rados.0.smithi085.stdout:559: done (7 left) 2021-02-10T04:30:17.184 INFO:tasks.rados.rados.0.smithi085.stdout:560: read oid 31 snap 51 2021-02-10T04:30:17.184 INFO:tasks.rados.rados.0.smithi085.stdout:560: expect (ObjNum 180 snap 40 seq_num 180) 2021-02-10T04:30:17.184 INFO:tasks.rados.rados.0.smithi085.stdout:561: rollback oid 43 current snap is 52 2021-02-10T04:30:17.185 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 43 to 51 2021-02-10T04:30:17.185 INFO:tasks.rados.rados.0.smithi085.stdout:562: copy_from oid 46 from oid 36 current snap is 52 2021-02-10T04:30:17.185 INFO:tasks.rados.rados.0.smithi085.stdout:563: copy_from oid 21 from oid 9 current snap is 52 2021-02-10T04:30:17.185 INFO:tasks.rados.rados.0.smithi085.stdout:564: read oid 39 snap -1 2021-02-10T04:30:17.185 INFO:tasks.rados.rados.0.smithi085.stdout:564: expect (ObjNum 218 snap 50 seq_num 218) 2021-02-10T04:30:17.185 INFO:tasks.rados.rados.0.smithi085.stdout:565: rollback oid 27 current snap is 52 2021-02-10T04:30:17.185 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 27 to 51 2021-02-10T04:30:17.185 INFO:tasks.rados.rados.0.smithi085.stdout:566: write (excl) oid 5 current snap is 52 2021-02-10T04:30:17.185 INFO:tasks.rados.rados.0.smithi085.stdout:566: seq_num 230 ranges {665610=797860,1909329=458405,2816700=410210} 2021-02-10T04:30:17.192 INFO:tasks.rados.rados.0.smithi085.stdout:566: writing smithi08575397-5 from 665610 to 1463470 tid 1 2021-02-10T04:30:17.198 INFO:tasks.rados.rados.0.smithi085.stdout:566: writing smithi08575397-5 from 1909329 to 2367734 tid 2 2021-02-10T04:30:17.203 INFO:tasks.rados.rados.0.smithi085.stdout:566: writing smithi08575397-5 from 2816700 to 3226910 tid 3 2021-02-10T04:30:17.203 INFO:tasks.rados.rados.0.smithi085.stdout:567: snap_create 2021-02-10T04:30:17.208 INFO:tasks.rados.rados.0.smithi085.stdout:560: expect (ObjNum 180 snap 40 seq_num 180) 2021-02-10T04:30:17.340 INFO:tasks.rados.rados.0.smithi085.stdout:565: finishing rollback tid 1 to smithi08575397-27 2021-02-10T04:30:17.340 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 27 v 0 (ObjNum 82371856 snap 22030 seq_num 82338672) dirty dne 2021-02-10T04:30:17.340 INFO:tasks.rados.rados.0.smithi085.stdout:564: expect (ObjNum 218 snap 50 seq_num 218) 2021-02-10T04:30:17.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:17 smithi164 conmon[29288]: cluster 2021-02-10T04:30:16.016161+0000 mgr.y (mgr.14140) 396 : cluster [DBG] pgmap v448: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 126 MiB data, 393 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 1.1 MiB/s wr, 16 op/s 2021-02-10T04:30:17.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:17 smithi164 conmon[29288]: cluster 2021-02-10T04:30:16.067443+0000 mon.a (mon.0) 907 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2021-02-10T04:30:17.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:17 smithi085 conmon[29319]: cluster 2021-02-10T04:30:17.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:17 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:30:17.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:17 smithi085 conmon[29319]: 10T04:30:16.016161+0000 mgr.y (mgr.14140) 396 : cluster 2021-02-10T04:30:17.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:17 smithi085 conmon[29319]: [DBG] pgmap v448: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 126 MiB data, 393 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 1.1 MiB/s wr, 16 op/s 2021-02-10T04:30:17.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:17 smithi085 conmon[29319]: cluster 2021-02-10T04:30:17.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:17 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:30:17.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:17 smithi085 conmon[29319]: 30:16.067443+0000 mon.a (mon 2021-02-10T04:30:17.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:17 smithi085 conmon[29319]: .0) 907 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2021-02-10T04:30:17.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:17 smithi085 conmon[35961]: cluster 2021-02-10T04:30:16.016161+0000 2021-02-10T04:30:17.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:17 smithi085 conmon[35961]: mgr.y (mgr.14140) 396 : cluster [DBG] pgmap v448: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 126 MiB data, 393 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 1.1 MiB/s wr, 16 op/s 2021-02-10T04:30:17.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:17 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:30:17.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:17 smithi085 conmon[35961]: :30:16.067443+0000 mon.a (mon.0) 907 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2021-02-10T04:30:17.510 INFO:tasks.rados.rados.0.smithi085.stdout:561: finishing rollback tid 0 to smithi08575397-43 2021-02-10T04:30:17.510 INFO:tasks.rados.rados.0.smithi085.stdout:561: finishing rollback tid 1 to smithi08575397-43 2021-02-10T04:30:17.511 INFO:tasks.rados.rados.0.smithi085.stdout:561: finishing rollback tid 2 to smithi08575397-43 2021-02-10T04:30:17.511 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 217 (ObjNum 194 snap 43 seq_num 194) dirty exists 2021-02-10T04:30:17.511 INFO:tasks.rados.rados.0.smithi085.stdout:566: finishing write tid 1 to smithi08575397-5 2021-02-10T04:30:17.511 INFO:tasks.rados.rados.0.smithi085.stdout:566: finishing write tid 2 to smithi08575397-5 2021-02-10T04:30:17.511 INFO:tasks.rados.rados.0.smithi085.stdout:566: finishing write tid 3 to smithi08575397-5 2021-02-10T04:30:17.511 INFO:tasks.rados.rados.0.smithi085.stdout:566: finishing write tid 4 to smithi08575397-5 2021-02-10T04:30:17.511 INFO:tasks.rados.rados.0.smithi085.stdout:562: finishing copy_from to smithi08575397-46 2021-02-10T04:30:17.511 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 90 (ObjNum 127 snap 23 seq_num 127) dirty exists 2021-02-10T04:30:17.511 INFO:tasks.rados.rados.0.smithi085.stdout:562: finishing copy_from racing read to smithi08575397-46 2021-02-10T04:30:17.511 INFO:tasks.rados.rados.0.smithi085.stdout:566: finishing write tid 5 to smithi08575397-5 2021-02-10T04:30:17.511 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 191 (ObjNum 230 snap 52 seq_num 230) dirty exists 2021-02-10T04:30:17.511 INFO:tasks.rados.rados.0.smithi085.stdout:566: left oid 5 (ObjNum 230 snap 52 seq_num 230) 2021-02-10T04:30:17.511 INFO:tasks.rados.rados.0.smithi085.stdout:563: finishing copy_from to smithi08575397-21 2021-02-10T04:30:17.512 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 158 (ObjNum 229 snap 51 seq_num 229) dirty exists 2021-02-10T04:30:17.512 INFO:tasks.rados.rados.0.smithi085.stdout:563: finishing copy_from racing read to smithi08575397-21 2021-02-10T04:30:17.518 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:30:17.550 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:17.666 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 4 1 2021-02-10T04:30:17.787 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:30:17.891 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:30:17.919 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:17.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:30:17 smithi085 conmon[49106]: debug 2021-02-10T04:30:17.737+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:18.017 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:30:18.060 INFO:tasks.rados.rados.0.smithi085.stdout:560: done (14 left) 2021-02-10T04:30:18.060 INFO:tasks.rados.rados.0.smithi085.stdout:561: done (13 left) 2021-02-10T04:30:18.060 INFO:tasks.rados.rados.0.smithi085.stdout:562: done (12 left) 2021-02-10T04:30:18.060 INFO:tasks.rados.rados.0.smithi085.stdout:563: done (11 left) 2021-02-10T04:30:18.060 INFO:tasks.rados.rados.0.smithi085.stdout:564: done (10 left) 2021-02-10T04:30:18.060 INFO:tasks.rados.rados.0.smithi085.stdout:565: done (9 left) 2021-02-10T04:30:18.060 INFO:tasks.rados.rados.0.smithi085.stdout:566: done (8 left) 2021-02-10T04:30:18.060 INFO:tasks.rados.rados.0.smithi085.stdout:567: done (7 left) 2021-02-10T04:30:18.060 INFO:tasks.rados.rados.0.smithi085.stdout:568: rollback oid 18 current snap is 53 2021-02-10T04:30:18.060 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 18 to 51 2021-02-10T04:30:18.060 INFO:tasks.rados.rados.0.smithi085.stdout:569: copy_from oid 46 from oid 1 current snap is 53 2021-02-10T04:30:18.061 INFO:tasks.rados.rados.0.smithi085.stdout:570: delete oid 14 current snap is 53 2021-02-10T04:30:18.070 INFO:tasks.rados.rados.0.smithi085.stdout:568: finishing rollback tid 1 to smithi08575397-18 2021-02-10T04:30:18.070 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 18 v 0 (ObjNum 82383680 snap 22030 seq_num 82376000) dirty dne 2021-02-10T04:30:18.073 INFO:tasks.rados.rados.0.smithi085.stdout:568: done (9 left) 2021-02-10T04:30:18.073 INFO:tasks.rados.rados.0.smithi085.stdout:570: done (8 left) 2021-02-10T04:30:18.073 INFO:tasks.rados.rados.0.smithi085.stdout:571: write oid 27 current snap is 53 2021-02-10T04:30:18.074 INFO:tasks.rados.rados.0.smithi085.stdout:571: seq_num 232 ranges {699224=548358,2026293=462074,3271546=432754} 2021-02-10T04:30:18.081 INFO:tasks.rados.rados.0.smithi085.stdout:571: writing smithi08575397-27 from 699224 to 1247582 tid 1 2021-02-10T04:30:18.089 INFO:tasks.rados.rados.0.smithi085.stdout:571: writing smithi08575397-27 from 2026293 to 2488367 tid 2 2021-02-10T04:30:18.097 INFO:tasks.rados.rados.0.smithi085.stdout:571: writing smithi08575397-27 from 3271546 to 3704300 tid 3 2021-02-10T04:30:18.098 INFO:tasks.rados.rados.0.smithi085.stdout:572: copy_from oid 41 from oid 21 current snap is 53 2021-02-10T04:30:18.098 INFO:tasks.rados.rados.0.smithi085.stdout:571: finishing write tid 1 to smithi08575397-27 2021-02-10T04:30:18.098 INFO:tasks.rados.rados.0.smithi085.stdout:571: finishing write tid 2 to smithi08575397-27 2021-02-10T04:30:18.099 INFO:tasks.rados.rados.0.smithi085.stdout:573: rollback oid 43 current snap is 53 2021-02-10T04:30:18.099 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 43 to 51 2021-02-10T04:30:18.099 INFO:tasks.rados.rados.0.smithi085.stdout:574: read oid 39 snap 50 2021-02-10T04:30:18.099 INFO:tasks.rados.rados.0.smithi085.stdout:574: expect (ObjNum 218 snap 50 seq_num 218) 2021-02-10T04:30:18.099 INFO:tasks.rados.rados.0.smithi085.stdout:575: read oid 8 snap -1 2021-02-10T04:30:18.099 INFO:tasks.rados.rados.0.smithi085.stdout:575: expect (ObjNum 174 snap 37 seq_num 174) 2021-02-10T04:30:18.100 INFO:tasks.rados.rados.0.smithi085.stdout:576: read oid 10 snap -1 2021-02-10T04:30:18.100 INFO:tasks.rados.rados.0.smithi085.stdout:576: expect (ObjNum 71 snap 7 seq_num 71) 2021-02-10T04:30:18.100 INFO:tasks.rados.rados.0.smithi085.stdout:577: delete oid 37 current snap is 53 2021-02-10T04:30:18.104 INFO:tasks.rados.rados.0.smithi085.stdout:573: finishing rollback tid 0 to smithi08575397-43 2021-02-10T04:30:18.104 INFO:tasks.rados.rados.0.smithi085.stdout:573: finishing rollback tid 1 to smithi08575397-43 2021-02-10T04:30:18.105 INFO:tasks.rados.rados.0.smithi085.stdout:573: finishing rollback tid 2 to smithi08575397-43 2021-02-10T04:30:18.105 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 220 (ObjNum 194 snap 43 seq_num 194) dirty exists 2021-02-10T04:30:18.109 INFO:tasks.rados.rados.0.smithi085.stdout:576: expect (ObjNum 71 snap 7 seq_num 71) 2021-02-10T04:30:18.143 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:30:17 smithi164 conmon[37263]: debug 2021-02-10T04:30:17.983+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:18.144 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:30:17 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:30:18.247 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:18 smithi085 conmon[35961]: cluster 2021-02-10T04:30:17. 2021-02-10T04:30:18.247 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:18 smithi085 conmon[35961]: 182640+0000 mon.a (mon.0) 908 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2021-02-10T04:30:18.247 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:18 smithi085 conmon[35961]: audit 2021-02-10T04:30:17.775117+0000 mgr.y (mgr.14140) 397 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:30:18.247 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:18 smithi085 conmon[35961]: cluster 2021-02-10T04:30:18.016594+0000 mgr.y (mgr.14140) 398 : cluster [DBG] pgmap v451: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim, 165 active+clean; 131 MiB data, 403 MiB used, 715 GiB / 715 GiB avail; 6.5 MiB/s rd, 2.0 MiB/s wr, 15 op/s 2021-02-10T04:30:18.247 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:18 smithi085 conmon[35961]: cluster 2021-02-10T04:30:18.057179+0000 mon.a (mon.0) 909 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2021-02-10T04:30:18.248 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:18 smithi085 conmon[29319]: cluster 2021-02-10T04:30:17.182640+0000 mon.a (mon.0) 908 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2021-02-10T04:30:18.248 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:18 smithi085 conmon[29319]: audit 2021-02-10T04:30:17.775117+0000 mgr.y (mgr.14140) 397 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:30:18.248 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:18 smithi085 conmon[29319]: cluster 2021-02-10T04:30:18.016594+0000 mgr.y (mgr.14140) 398 : cluster [DBG] pgmap v451: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim, 165 active+clean; 131 MiB data, 403 MiB used, 715 GiB / 715 GiB avail; 6.5 MiB/s rd, 2.0 MiB/s wr, 15 op/s 2021-02-10T04:30:18.248 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:18 smithi085 conmon[29319]: cluster 2021-02-10T04:30:18.057179+0000 mon.a (mon.0) 909 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2021-02-10T04:30:18.314 INFO:tasks.rados.rados.0.smithi085.stdout:574: expect (ObjNum 218 snap 50 seq_num 218) 2021-02-10T04:30:18.428 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:30:18.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:18 smithi164 conmon[29288]: cluster 2021-02-10T04:30:17.182640+0000 mon.a (mon.0) 2021-02-10T04:30:18.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:18 smithi164 conmon[29288]: 908 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2021-02-10T04:30:18.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:18 smithi164 conmon[29288]: audit 2021-02-10T04:30:17.775117+0000 mgr.y (mgr.14140) 397 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:30:18.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:18 smithi164 conmon[29288]: cluster 2021-02-10T04:30:18.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:18 smithi164 conmon[29288]: 2021-02-10T04:30:18.016594+0000 mgr.y (mgr.14140) 398 : cluster [DBG] pgmap v451: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim, 165 active+clean; 131 MiB data, 403 MiB used, 715 GiB / 715 GiB avail; 6.5 MiB/s rd, 2.0 MiB/s wr, 15 op/s 2021-02-10T04:30:18.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:18 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:30:18.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:18 smithi164 conmon[29288]: -02-10T04:30:18.057179+0000 2021-02-10T04:30:18.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:18 smithi164 conmon[29288]: mon.a (mon.0) 909 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2021-02-10T04:30:18.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:18 smithi164 conmon[29288]: 2021-02-10T04:30:18.486 INFO:tasks.rados.rados.0.smithi085.stdout:573: done (14 left) 2021-02-10T04:30:18.486 INFO:tasks.rados.rados.0.smithi085.stdout:574: done (13 left) 2021-02-10T04:30:18.487 INFO:tasks.rados.rados.0.smithi085.stdout:576: done (12 left) 2021-02-10T04:30:18.487 INFO:tasks.rados.rados.0.smithi085.stdout:577: done (11 left) 2021-02-10T04:30:18.487 INFO:tasks.rados.rados.0.smithi085.stdout:578: read oid 37 snap 52 2021-02-10T04:30:18.487 INFO:tasks.rados.rados.0.smithi085.stdout:578: expect (ObjNum 25 snap 0 seq_num 25) 2021-02-10T04:30:18.487 INFO:tasks.rados.rados.0.smithi085.stdout:579: snap_remove snap 51 2021-02-10T04:30:18.560 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:30:18.595 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:19.186 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:19 smithi164 conmon[29288]: cluster 2021-02-10T04:30:18.326079 2021-02-10T04:30:19.186 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:19 smithi164 conmon[29288]: +0000 osd.6 (osd.6) 46 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:30:19.186 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:19 smithi164 conmon[29288]: cluster 2021-02-10T04:30:18.326280+0000 osd.6 (osd.6) 47 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:30:19.186 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:19 smithi164 conmon[29288]: cluster 2021-02-10T04:30:18.463777+0000 osd.6 (osd.6) 48 : cluster [DBG] 7.c scrub ok 2021-02-10T04:30:19.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:19 smithi085 conmon[35961]: cluster 2021-02-10T04:30:18.326079+0000 osd.6 (osd.6) 46 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:30:19.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:19 smithi085 conmon[35961]: cluster 2021-02-10T04:30:18.326280+0000 osd.6 (osd.6) 47 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:30:19.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:19 smithi085 conmon[35961]: cluster 2021-02-10T04:30:18.463777+0000 osd.6 (osd.6) 48 : cluster [DBG] 7.c scrub ok 2021-02-10T04:30:19.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:19 smithi085 conmon[29319]: cluster 2021-02-10T04:30:18.326079+0000 osd.6 (osd.6) 46 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:30:19.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:19 smithi085 conmon[29319]: cluster 2021-02-10T04:30:18.326280+0000 osd.6 (osd.6) 47 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:30:19.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:19 smithi085 conmon[29319]: cluster 2021-02-10T04:30:18.463777+0000 osd.6 (osd.6) 48 : cluster [DBG] 7.c scrub ok 2021-02-10T04:30:19.194 INFO:tasks.rados.rados.0.smithi085.stdout:579: done (12 left) 2021-02-10T04:30:19.194 INFO:tasks.rados.rados.0.smithi085.stdout:580: write oid 10 current snap is 53 2021-02-10T04:30:19.194 INFO:tasks.rados.rados.0.smithi085.stdout:580: seq_num 234 ranges {732838=698856,2026904=465743,3210039=571651} 2021-02-10T04:30:19.203 INFO:tasks.rados.rados.0.smithi085.stdout:580: writing smithi08575397-10 from 732838 to 1431694 tid 1 2021-02-10T04:30:19.210 INFO:tasks.rados.rados.0.smithi085.stdout:580: writing smithi08575397-10 from 2026904 to 2492647 tid 2 2021-02-10T04:30:19.214 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:30:19.214 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:30:19.217 INFO:tasks.rados.rados.0.smithi085.stdout:580: writing smithi08575397-10 from 3210039 to 3781690 tid 3 2021-02-10T04:30:19.218 INFO:tasks.rados.rados.0.smithi085.stdout:581: copy_from oid 11 from oid 9 current snap is 53 2021-02-10T04:30:19.218 INFO:tasks.rados.rados.0.smithi085.stdout:575: expect (ObjNum 174 snap 37 seq_num 174) 2021-02-10T04:30:19.352 INFO:tasks.rados.rados.0.smithi085.stdout:571: finishing write tid 3 to smithi08575397-27 2021-02-10T04:30:19.353 INFO:tasks.rados.rados.0.smithi085.stdout:571: finishing write tid 4 to smithi08575397-27 2021-02-10T04:30:19.353 INFO:tasks.rados.rados.0.smithi085.stdout:571: finishing write tid 5 to smithi08575397-27 2021-02-10T04:30:19.353 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 27 v 60 (ObjNum 232 snap 53 seq_num 232) dirty exists 2021-02-10T04:30:19.353 INFO:tasks.rados.rados.0.smithi085.stdout:571: left oid 27 (ObjNum 232 snap 53 seq_num 232) 2021-02-10T04:30:19.353 INFO:tasks.rados.rados.0.smithi085.stdout:572: finishing copy_from to smithi08575397-41 2021-02-10T04:30:19.353 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 41 v 158 (ObjNum 229 snap 51 seq_num 229) dirty exists 2021-02-10T04:30:19.353 INFO:tasks.rados.rados.0.smithi085.stdout:572: finishing copy_from racing read to smithi08575397-41 2021-02-10T04:30:19.353 INFO:tasks.rados.rados.0.smithi085.stdout:569: finishing copy_from to smithi08575397-46 2021-02-10T04:30:19.353 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 93 (ObjNum 202 snap 44 seq_num 202) dirty exists 2021-02-10T04:30:19.353 INFO:tasks.rados.rados.0.smithi085.stdout:569: finishing copy_from racing read to smithi08575397-46 2021-02-10T04:30:19.353 INFO:tasks.rados.rados.0.smithi085.stdout:569: done (13 left) 2021-02-10T04:30:19.354 INFO:tasks.rados.rados.0.smithi085.stdout:571: done (12 left) 2021-02-10T04:30:19.354 INFO:tasks.rados.rados.0.smithi085.stdout:572: done (11 left) 2021-02-10T04:30:19.354 INFO:tasks.rados.rados.0.smithi085.stdout:575: done (10 left) 2021-02-10T04:30:19.354 INFO:tasks.rados.rados.0.smithi085.stdout:582: snap_remove snap 50 2021-02-10T04:30:19.744 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset nodeep-scrub 2021-02-10T04:30:19.772 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:30:19 smithi085 conmon[52836]: debug 2021-02-10T04:30:19.467+0000 7fba6ffae700 -1 osd.3 pg_epoch: 203 pg[4.14( v 63'26 (0'0,63'26] local-lis/les=71/72 n=11 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,2,7] r=0 lpr=71 crt=63'26 lcod 63'25 mlcod 0'0 active+clean] scrubber pg(4.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55fec06b4160) 2021-02-10T04:30:20.049 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:30:20.203 INFO:tasks.rados.rados.0.smithi085.stdout:582: done (10 left) 2021-02-10T04:30:20.204 INFO:tasks.rados.rados.0.smithi085.stdout:583: write oid 43 current snap is 53 2021-02-10T04:30:20.204 INFO:tasks.rados.rados.0.smithi085.stdout:583: seq_num 235 ranges {749645=774105,2085386=609401,3237462=712183} 2021-02-10T04:30:20.209 INFO:tasks.rados.rados.0.smithi085.stdout:583: writing smithi08575397-43 from 749645 to 1523750 tid 1 2021-02-10T04:30:20.216 INFO:tasks.rados.rados.0.smithi085.stdout:583: writing smithi08575397-43 from 2085386 to 2694787 tid 2 2021-02-10T04:30:20.223 INFO:tasks.rados.rados.0.smithi085.stdout:583: writing smithi08575397-43 from 3237462 to 3949645 tid 3 2021-02-10T04:30:20.224 INFO:tasks.rados.rados.0.smithi085.stdout:584: write oid 21 current snap is 53 2021-02-10T04:30:20.224 INFO:tasks.rados.rados.0.smithi085.stdout:578: expect (ObjNum 25 snap 0 seq_num 25) 2021-02-10T04:30:20.338 INFO:tasks.rados.rados.0.smithi085.stdout:580: finishing write tid 1 to smithi08575397-10 2021-02-10T04:30:20.339 INFO:tasks.rados.rados.0.smithi085.stdout:580: finishing write tid 2 to smithi08575397-10 2021-02-10T04:30:20.339 INFO:tasks.rados.rados.0.smithi085.stdout:580: finishing write tid 3 to smithi08575397-10 2021-02-10T04:30:20.339 INFO:tasks.rados.rados.0.smithi085.stdout:580: finishing write tid 4 to smithi08575397-10 2021-02-10T04:30:20.339 INFO:tasks.rados.rados.0.smithi085.stdout:580: finishing write tid 5 to smithi08575397-10 2021-02-10T04:30:20.339 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 10 v 80 (ObjNum 234 snap 53 seq_num 234) dirty exists 2021-02-10T04:30:20.339 INFO:tasks.rados.rados.0.smithi085.stdout:580: left oid 10 (ObjNum 234 snap 53 seq_num 234) 2021-02-10T04:30:20.339 INFO:tasks.rados.rados.0.smithi085.stdout:581: finishing copy_from to smithi08575397-11 2021-02-10T04:30:20.339 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 158 (ObjNum 229 snap 51 seq_num 229) dirty exists 2021-02-10T04:30:20.339 INFO:tasks.rados.rados.0.smithi085.stdout:581: finishing copy_from racing read to smithi08575397-11 2021-02-10T04:30:20.339 INFO:tasks.rados.rados.0.smithi085.stdout:583: finishing write tid 1 to smithi08575397-43 2021-02-10T04:30:20.340 INFO:tasks.rados.rados.0.smithi085.stdout:583: finishing write tid 2 to smithi08575397-43 2021-02-10T04:30:20.340 INFO:tasks.rados.rados.0.smithi085.stdout:583: finishing write tid 3 to smithi08575397-43 2021-02-10T04:30:20.340 INFO:tasks.rados.rados.0.smithi085.stdout:583: finishing write tid 4 to smithi08575397-43 2021-02-10T04:30:20.340 INFO:tasks.rados.rados.0.smithi085.stdout:583: finishing write tid 5 to smithi08575397-43 2021-02-10T04:30:20.340 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 224 (ObjNum 235 snap 53 seq_num 235) dirty exists 2021-02-10T04:30:20.340 INFO:tasks.rados.rados.0.smithi085.stdout:583: left oid 43 (ObjNum 235 snap 53 seq_num 235) 2021-02-10T04:30:20.340 INFO:tasks.rados.rados.0.smithi085.stdout:584: seq_num 236 ranges {766452=565707,1860221=469412,2981238=594195} 2021-02-10T04:30:20.347 INFO:tasks.rados.rados.0.smithi085.stdout:584: writing smithi08575397-21 from 766452 to 1332159 tid 1 2021-02-10T04:30:20.353 INFO:tasks.rados.rados.0.smithi085.stdout:584: writing smithi08575397-21 from 1860221 to 2329633 tid 2 2021-02-10T04:30:20.362 INFO:tasks.rados.rados.0.smithi085.stdout:584: writing smithi08575397-21 from 2981238 to 3575433 tid 3 2021-02-10T04:30:20.362 INFO:tasks.rados.rados.0.smithi085.stdout:584: finishing write tid 1 to smithi08575397-21 2021-02-10T04:30:20.363 INFO:tasks.rados.rados.0.smithi085.stdout:584: finishing write tid 2 to smithi08575397-21 2021-02-10T04:30:20.363 INFO:tasks.rados.rados.0.smithi085.stdout:578: done (11 left) 2021-02-10T04:30:20.363 INFO:tasks.rados.rados.0.smithi085.stdout:580: done (10 left) 2021-02-10T04:30:20.363 INFO:tasks.rados.rados.0.smithi085.stdout:581: done (9 left) 2021-02-10T04:30:20.363 INFO:tasks.rados.rados.0.smithi085.stdout:583: done (8 left) 2021-02-10T04:30:20.363 INFO:tasks.rados.rados.0.smithi085.stdout:585: snap_create 2021-02-10T04:30:20.367 INFO:tasks.rados.rados.0.smithi085.stdout:584: finishing write tid 3 to smithi08575397-21 2021-02-10T04:30:20.368 INFO:tasks.rados.rados.0.smithi085.stdout:584: finishing write tid 4 to smithi08575397-21 2021-02-10T04:30:20.368 INFO:tasks.rados.rados.0.smithi085.stdout:584: finishing write tid 5 to smithi08575397-21 2021-02-10T04:30:20.368 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 248 (ObjNum 236 snap 53 seq_num 236) dirty exists 2021-02-10T04:30:20.368 INFO:tasks.rados.rados.0.smithi085.stdout:584: left oid 21 (ObjNum 236 snap 53 seq_num 236) 2021-02-10T04:30:20.643 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:30:20 smithi164 conmon[40892]: debug 2021-02-10T04:30:20.200+0000 7f3036cc8700 -1 osd.6 pg_epoch: 204 pg[7.c( v 200'110 (0'0,200'110] local-lis/les=93/94 n=4 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [6,3,7] r=0 lpr=93 crt=200'110 lcod 179'108 mlcod 179'108 active+clean+snaptrim trimq=[66~1] ps=[60~3,67~1]] removing snap head 2021-02-10T04:30:20.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:20 smithi164 conmon[29288]: cluster 2021-02-10T04:30:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:20 smithi164 conmon[29288]: 2021-02-10T04:30:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:20 smithi164 conmon[29288]: 2021 2021-02-10T04:30:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:20 smithi164 conmon[29288]: - 2021-02-10T04:30:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:20 smithi164 conmon[29288]: 02-10T04:30 2021-02-10T04:30:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:20 smithi164 conmon[29288]: :19.193963+0000 mon.a (mon.0) 910 : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2021-02-10T04:30:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:20 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:30:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:20 smithi164 conmon[29288]: -10T04:30:20.017007+0000 mgr.y (mgr.14140) 399 : cluster [DBG] pgmap v454: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim, 165 active+clean; 149 MiB data, 437 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 5.3 MiB/s wr, 31 op/s 2021-02-10T04:30:20.644 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:30:20.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:20 smithi085 conmon[29319]: cluster 2021-02-10T04:30:19.193963+0000 mon.a (mon.0) 910 : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2021-02-10T04:30:20.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:20 smithi085 conmon[29319]: 2021-02-10T04:30:20.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:20 smithi085 conmon[29319]: cluster 2021-02-10T04:30:20.017007+0000 mgr.y (mgr.14140) 399 : cluster [DBG] pgmap v454: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim, 165 active+clean; 149 MiB data, 437 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 5.3 MiB/s wr, 31 op/s 2021-02-10T04:30:20.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:20 smithi085 conmon[29319]: 2021-02-10T04:30:20.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:20 smithi085 conmon[35961]: cluster 2021-02-10T04:30:19.193963+0000 mon.a (mon.0) 910 2021-02-10T04:30:20.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:20 smithi085 conmon[35961]: : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2021-02-10T04:30:20.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:20 smithi085 conmon[35961]: cluster 2021-02-10T04:30:20.017007+0000 mgr.y (mgr.14140) 2021-02-10T04:30:20.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:20 smithi085 conmon[35961]: 399 : cluster [DBG] pgmap v454: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim, 165 active+clean; 149 MiB data, 437 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 5.3 MiB/s wr, 31 op/s 2021-02-10T04:30:20.692 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:30:20 smithi085 conmon[41540]: debug 2021-02-10T04:30:20.692 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:30:20 smithi085 conmon[41540]: 2021-02-10T04:30:20.377+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:20.991 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.6 dump_blocked_ops 2021-02-10T04:30:21.202 INFO:tasks.rados.rados.0.smithi085.stdout:584: done (8 left) 2021-02-10T04:30:21.203 INFO:tasks.rados.rados.0.smithi085.stdout:585: done (7 left) 2021-02-10T04:30:21.203 INFO:tasks.rados.rados.0.smithi085.stdout:586: rollback oid 11 current snap is 54 2021-02-10T04:30:21.203 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 11 to 52 2021-02-10T04:30:21.209 INFO:tasks.rados.rados.0.smithi085.stdout:587: copy_from oid 21 from oid 16 current snap is 54 2021-02-10T04:30:21.209 INFO:tasks.rados.rados.0.smithi085.stdout:588: read oid 44 snap -1 2021-02-10T04:30:21.209 INFO:tasks.rados.rados.0.smithi085.stdout:588: expect deleted 2021-02-10T04:30:21.209 INFO:tasks.rados.rados.0.smithi085.stdout:589: write (excl) oid 36 current snap is 54 2021-02-10T04:30:21.209 INFO:tasks.rados.rados.0.smithi085.stdout:589: seq_num 237 ranges {783259=640956,1918703=613070,3008661=405467,3821939=161320} 2021-02-10T04:30:21.211 INFO:tasks.rados.rados.0.smithi085.stdout:589: writing smithi08575397-36 from 783259 to 1424215 tid 1 2021-02-10T04:30:21.218 INFO:tasks.rados.rados.0.smithi085.stdout:589: writing smithi08575397-36 from 1918703 to 2531773 tid 2 2021-02-10T04:30:21.224 INFO:tasks.rados.rados.0.smithi085.stdout:589: writing smithi08575397-36 from 3008661 to 3414128 tid 3 2021-02-10T04:30:21.228 INFO:tasks.rados.rados.0.smithi085.stdout:589: writing smithi08575397-36 from 3821939 to 3983259 tid 4 2021-02-10T04:30:21.228 INFO:tasks.rados.rados.0.smithi085.stdout:590: delete oid 18 current snap is 54 2021-02-10T04:30:21.228 INFO:tasks.rados.rados.0.smithi085.stdout:586: finishing rollback tid 0 to smithi08575397-11 2021-02-10T04:30:21.228 INFO:tasks.rados.rados.0.smithi085.stdout:586: finishing rollback tid 1 to smithi08575397-11 2021-02-10T04:30:21.228 INFO:tasks.rados.rados.0.smithi085.stdout:586: finishing rollback tid 2 to smithi08575397-11 2021-02-10T04:30:21.228 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 252 (ObjNum 226 snap 51 seq_num 226) dirty exists 2021-02-10T04:30:21.228 INFO:tasks.rados.rados.0.smithi085.stdout:589: finishing write tid 1 to smithi08575397-36 2021-02-10T04:30:21.228 INFO:tasks.rados.rados.0.smithi085.stdout:589: finishing write tid 2 to smithi08575397-36 2021-02-10T04:30:21.230 INFO:tasks.rados.rados.0.smithi085.stdout:586: done (11 left) 2021-02-10T04:30:21.230 INFO:tasks.rados.rados.0.smithi085.stdout:588: done (10 left) 2021-02-10T04:30:21.230 INFO:tasks.rados.rados.0.smithi085.stdout:590: done (9 left) 2021-02-10T04:30:21.230 INFO:tasks.rados.rados.0.smithi085.stdout:591: write oid 34 current snap is 54 2021-02-10T04:30:21.231 INFO:tasks.rados.rados.0.smithi085.stdout:591: seq_num 239 ranges {416873=791454,1919314=97559} 2021-02-10T04:30:21.238 INFO:tasks.rados.rados.0.smithi085.stdout:591: writing smithi08575397-34 from 416873 to 1208327 tid 1 2021-02-10T04:30:21.243 INFO:tasks.rados.rados.0.smithi085.stdout:591: writing smithi08575397-34 from 1919314 to 2016873 tid 2 2021-02-10T04:30:21.243 INFO:tasks.rados.rados.0.smithi085.stdout:589: finishing write tid 3 to smithi08575397-36 2021-02-10T04:30:21.243 INFO:tasks.rados.rados.0.smithi085.stdout:589: finishing write tid 4 to smithi08575397-36 2021-02-10T04:30:21.243 INFO:tasks.rados.rados.0.smithi085.stdout:589: finishing write tid 5 to smithi08575397-36 2021-02-10T04:30:21.243 INFO:tasks.rados.rados.0.smithi085.stdout:589: finishing write tid 6 to smithi08575397-36 2021-02-10T04:30:21.244 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 95 (ObjNum 237 snap 54 seq_num 237) dirty exists 2021-02-10T04:30:21.244 INFO:tasks.rados.rados.0.smithi085.stdout:589: left oid 36 (ObjNum 237 snap 54 seq_num 237) 2021-02-10T04:30:21.244 INFO:tasks.rados.rados.0.smithi085.stdout:589: done (9 left) 2021-02-10T04:30:21.244 INFO:tasks.rados.rados.0.smithi085.stdout:592: rollback oid 7 current snap is 54 2021-02-10T04:30:21.244 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 7 to 52 2021-02-10T04:30:21.244 INFO:tasks.rados.rados.0.smithi085.stdout:593: snap_create 2021-02-10T04:30:21.244 INFO:tasks.rados.rados.0.smithi085.stdout:591: finishing write tid 1 to smithi08575397-34 2021-02-10T04:30:21.245 INFO:tasks.rados.rados.0.smithi085.stdout:591: finishing write tid 2 to smithi08575397-34 2021-02-10T04:30:21.245 INFO:tasks.rados.rados.0.smithi085.stdout:592: finishing rollback tid 0 to smithi08575397-7 2021-02-10T04:30:21.246 INFO:tasks.rados.rados.0.smithi085.stdout:591: finishing write tid 3 to smithi08575397-34 2021-02-10T04:30:21.246 INFO:tasks.rados.rados.0.smithi085.stdout:592: finishing rollback tid 1 to smithi08575397-7 2021-02-10T04:30:21.246 INFO:tasks.rados.rados.0.smithi085.stdout:591: finishing write tid 4 to smithi08575397-34 2021-02-10T04:30:21.246 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 34 v 194 (ObjNum 239 snap 54 seq_num 239) dirty exists 2021-02-10T04:30:21.246 INFO:tasks.rados.rados.0.smithi085.stdout:591: left oid 34 (ObjNum 239 snap 54 seq_num 239) 2021-02-10T04:30:21.247 INFO:tasks.rados.rados.0.smithi085.stdout:592: finishing rollback tid 2 to smithi08575397-7 2021-02-10T04:30:21.247 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 227 (ObjNum 202 snap 44 seq_num 202) dirty exists 2021-02-10T04:30:21.269 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:30:21.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:21 smithi164 conmon[29288]: cluster 2021-02-10T04:30:21.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:21 smithi164 conmon[29288]: 2021-02-10T04:30:21.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:21 smithi164 conmon[29288]: 2021-02-10T04:30:20 2021-02-10T04:30:21.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:21 smithi164 conmon[29288]: .197851+0000 mon.a (mon.0) 911 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2021-02-10T04:30:21.422 INFO:tasks.rados.rados.0.smithi085.stdout:587: finishing copy_from to smithi08575397-21 2021-02-10T04:30:21.422 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 217 (ObjNum 194 snap 43 seq_num 194) dirty exists 2021-02-10T04:30:21.422 INFO:tasks.rados.rados.0.smithi085.stdout:587: finishing copy_from racing read to smithi08575397-21 2021-02-10T04:30:21.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:21 smithi085 conmon[29319]: cluster 2021-02-10T04:30:20.197851+0000 mon.a ( 2021-02-10T04:30:21.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:21 smithi085 conmon[29319]: mon.0) 911 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2021-02-10T04:30:21.441 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:21 smithi085 conmon[45443]: debug 2021-02-10T04:30:21.441 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:21 smithi085 conmon[45443]: 2021-02-10T04:30:21.114+0000 7f7125cf8700 -1 osd.1 pg_epoch: 204 pg[4.4( v 63'31 (0'0,63'31] local-lis/les=55/56 n=14 ec=55/55 lis/c=55/55 les/c/f=56/56/0 sis=55) [1,2,3] r=0 lpr=55 crt=63'31 lcod 63'30 mlcod 63'30 active+clean] scrubber pg(4.4) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 3 (0x55dbc83b5080) 2021-02-10T04:30:21.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:21 smithi085 conmon[35961]: cluster 2021-02-10T04:30:20 2021-02-10T04:30:21.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:21 smithi085 conmon[35961]: .197851+0000 mon.a (mon.0) 911 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2021-02-10T04:30:22.206 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:22 smithi164 conmon[29288]: cluster 2021-02-10T04:30:21.202251+0000 mon.a (mon.0) 912 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2021-02-10T04:30:22.207 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:22 smithi164 conmon[29288]: cluster 2021-02-10T04:30:22.017505+0000 mgr.y (mgr.14140) 400 : cluster [DBG] pgmap v457: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim, 165 active+clean; 162 MiB data, 494 MiB used, 715 GiB / 715 GiB avail; 33 MiB/s rd, 8.6 MiB/s wr, 47 op/s 2021-02-10T04:30:22.209 INFO:tasks.rados.rados.0.smithi085.stdout:587: done (10 left) 2021-02-10T04:30:22.209 INFO:tasks.rados.rados.0.smithi085.stdout:591: done (9 left) 2021-02-10T04:30:22.209 INFO:tasks.rados.rados.0.smithi085.stdout:592: done (8 left) 2021-02-10T04:30:22.209 INFO:tasks.rados.rados.0.smithi085.stdout:593: done (7 left) 2021-02-10T04:30:22.210 INFO:tasks.rados.rados.0.smithi085.stdout:594: snap_create 2021-02-10T04:30:22.281 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:30:22.314 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:22.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[29319]: cluster 2021-02-10T04:30:21.202251+0000 mon.a ( 2021-02-10T04:30:22.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[29319]: mon.0) 912 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2021-02-10T04:30:22.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[29319]: cluster 2021-02-10T04:30:22.017505+0000 mgr.y (mgr.14140) 400 : cluster [DBG] 2021-02-10T04:30:22.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[29319]: pgmap v457: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim, 165 active+clean; 162 MiB data, 494 MiB used, 715 GiB / 715 GiB avail; 33 MiB/s rd, 8.6 MiB/s wr, 47 op/s 2021-02-10T04:30:22.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: cluster 2021-02-10T04:30:22.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: 2021-02-10T04:30:22.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: 2021 2021-02-10T04:30:22.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: - 2021-02-10T04:30:22.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: 02 2021-02-10T04:30:22.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: - 2021-02-10T04:30:22.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: 10 2021-02-10T04:30:22.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: T 2021-02-10T04:30:22.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: 04 2021-02-10T04:30:22.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: : 2021-02-10T04:30:22.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: 30 2021-02-10T04:30:22.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: : 2021-02-10T04:30:22.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: 21 2021-02-10T04:30:22.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: . 2021-02-10T04:30:22.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: 202251 2021-02-10T04:30:22.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: +0000 2021-02-10T04:30:22.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: 2021-02-10T04:30:22.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: mon.a 2021-02-10T04:30:22.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: ( 2021-02-10T04:30:22.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: mon 2021-02-10T04:30:22.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: . 2021-02-10T04:30:22.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: 0 2021-02-10T04:30:22.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: ) 2021-02-10T04:30:22.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: 912 2021-02-10T04:30:22.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: : 2021-02-10T04:30:22.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: cluster 2021-02-10T04:30:22.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: 2021-02-10T04:30:22.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: [DBG] 2021-02-10T04:30:22.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: 2021-02-10T04:30:22.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: osdmap e205: 8 total, 8 up, 8 in 2021-02-10T04:30:22.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: 2021-02-10T04:30:22.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: cluster 2021-02-10T04:30:22.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: 2021-02-10T04:30:22.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: 2021 2021-02-10T04:30:22.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: - 2021-02-10T04:30:22.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: 02 2021-02-10T04:30:22.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: - 2021-02-10T04:30:22.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: 10 2021-02-10T04:30:22.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: T 2021-02-10T04:30:22.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: 04 2021-02-10T04:30:22.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: : 2021-02-10T04:30:22.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: 30 2021-02-10T04:30:22.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: : 2021-02-10T04:30:22.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: 22 2021-02-10T04:30:22.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: . 2021-02-10T04:30:22.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: 017505 2021-02-10T04:30:22.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: +0000 2021-02-10T04:30:22.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: 2021-02-10T04:30:22.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: mgr.y 2021-02-10T04:30:22.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: ( 2021-02-10T04:30:22.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: mgr 2021-02-10T04:30:22.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: . 2021-02-10T04:30:22.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: 14140 2021-02-10T04:30:22.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: ) 2021-02-10T04:30:22.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: 400 2021-02-10T04:30:22.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: : 2021-02-10T04:30:22.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: cluster 2021-02-10T04:30:22.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: 2021-02-10T04:30:22.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: [DBG] 2021-02-10T04:30:22.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: 2021-02-10T04:30:22.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: pgmap v457: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim, 165 active+clean; 162 MiB data, 494 MiB used, 715 GiB / 715 GiB avail; 33 MiB/s rd, 8.6 MiB/s wr, 47 op/s 2021-02-10T04:30:22.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[35961]: 2021-02-10T04:30:23.061 INFO:tasks.rados.rados.0.smithi085.stdout:594: done (7 left) 2021-02-10T04:30:23.061 INFO:tasks.rados.rados.0.smithi085.stdout:595: rollback oid 9 current snap is 56 2021-02-10T04:30:23.061 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 9 to 54 2021-02-10T04:30:23.061 INFO:tasks.rados.rados.0.smithi085.stdout:596: write oid 34 current snap is 56 2021-02-10T04:30:23.061 INFO:tasks.rados.rados.0.smithi085.stdout:596: seq_num 240 ranges {433680=466703,1577796=455884} 2021-02-10T04:30:23.064 INFO:tasks.rados.rados.0.smithi085.stdout:596: writing smithi08575397-34 from 433680 to 900383 tid 1 2021-02-10T04:30:23.071 INFO:tasks.rados.rados.0.smithi085.stdout:596: writing smithi08575397-34 from 1577796 to 2033680 tid 2 2021-02-10T04:30:23.072 INFO:tasks.rados.rados.0.smithi085.stdout:595: finishing rollback tid 0 to smithi08575397-9 2021-02-10T04:30:23.072 INFO:tasks.rados.rados.0.smithi085.stdout:595: finishing rollback tid 1 to smithi08575397-9 2021-02-10T04:30:23.073 INFO:tasks.rados.rados.0.smithi085.stdout:595: finishing rollback tid 2 to smithi08575397-9 2021-02-10T04:30:23.073 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 9 v 170 (ObjNum 229 snap 51 seq_num 229) dirty exists 2021-02-10T04:30:23.073 INFO:tasks.rados.rados.0.smithi085.stdout:596: finishing write tid 1 to smithi08575397-34 2021-02-10T04:30:23.073 INFO:tasks.rados.rados.0.smithi085.stdout:595: done (8 left) 2021-02-10T04:30:23.073 INFO:tasks.rados.rados.0.smithi085.stdout:597: rollback oid 3 current snap is 56 2021-02-10T04:30:23.073 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 3 to 54 2021-02-10T04:30:23.073 INFO:tasks.rados.rados.0.smithi085.stdout:598: snap_remove snap 53 2021-02-10T04:30:23.124 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:30:23.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:30:22 smithi085 conmon[52836]: debug 2021-02-10T04:30:22.802+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:23.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:23 smithi164 conmon[29288]: cluster 2021-02-10T04:30:22.209730+0000 mon.a (mon.0) 913 2021-02-10T04:30:23.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:23 smithi164 conmon[29288]: : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2021-02-10T04:30:23.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:23 smithi164 conmon[29288]: cluster 2021-02-10T04:30:23.057708+0000 mon.a (mon.0) 914 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2021-02-10T04:30:23.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:23 smithi085 conmon[35961]: cluster 2021-02-10T04:30:22.209730+0000 mon.a 2021-02-10T04:30:23.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:23 smithi085 conmon[35961]: (mon.0) 913 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2021-02-10T04:30:23.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:23 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:30:23.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:23 smithi085 conmon[35961]: :30:23.057708+0000 mon.a (mon.0) 914 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2021-02-10T04:30:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:23 smithi085 conmon[29319]: cluster 2021-02-10T04:30:22.209730 2021-02-10T04:30:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:23 smithi085 conmon[29319]: +0000 mon.a (mon.0) 913 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2021-02-10T04:30:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:23 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:30:23.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:23 smithi085 conmon[29319]: 04:30:23.057708+0000 mon.a (mon.0) 914 : cluster 2021-02-10T04:30:23.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:23 smithi085 conmon[29319]: [DBG] osdmap e207: 8 total, 8 up, 8 in 2021-02-10T04:30:24.078 INFO:tasks.rados.rados.0.smithi085.stdout:596: finishing write tid 2 to smithi08575397-34 2021-02-10T04:30:24.078 INFO:tasks.rados.rados.0.smithi085.stdout:596: finishing write tid 3 to smithi08575397-34 2021-02-10T04:30:24.078 INFO:tasks.rados.rados.0.smithi085.stdout:596: finishing write tid 4 to smithi08575397-34 2021-02-10T04:30:24.078 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 34 v 197 (ObjNum 240 snap 56 seq_num 240) dirty exists 2021-02-10T04:30:24.078 INFO:tasks.rados.rados.0.smithi085.stdout:596: left oid 34 (ObjNum 240 snap 56 seq_num 240) 2021-02-10T04:30:24.078 INFO:tasks.rados.rados.0.smithi085.stdout:597: finishing rollback tid 0 to smithi08575397-3 2021-02-10T04:30:24.079 INFO:tasks.rados.rados.0.smithi085.stdout:597: finishing rollback tid 1 to smithi08575397-3 2021-02-10T04:30:24.079 INFO:tasks.rados.rados.0.smithi085.stdout:597: finishing rollback tid 2 to smithi08575397-3 2021-02-10T04:30:24.079 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 230 (ObjNum 215 snap 48 seq_num 215) dirty exists 2021-02-10T04:30:24.079 INFO:tasks.rados.rados.0.smithi085.stdout:596: done (9 left) 2021-02-10T04:30:24.079 INFO:tasks.rados.rados.0.smithi085.stdout:597: done (8 left) 2021-02-10T04:30:24.079 INFO:tasks.rados.rados.0.smithi085.stdout:598: done (7 left) 2021-02-10T04:30:24.080 INFO:tasks.rados.rados.0.smithi085.stdout:599: snap_create 2021-02-10T04:30:24.441 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:24 smithi085 conmon[45443]: debug 2021-02-10T04:30:24.046+0000 7f7125cf8700 -1 osd.1 pg_epoch: 207 pg[4.4( v 63'31 (0'0,63'31] local-lis/les=55/56 n=14 ec=55/55 lis/c=55/55 les/c/f=56/56/0 sis=55) [1,2,3] r=0 lpr=55 crt=63'31 lcod 63'30 mlcod 63'30 active+clean] scrubber pg(4.4) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 3 (0x55dbcb3c6580) 2021-02-10T04:30:24.604 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:30:24.644 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:24.647 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:30:24.647 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:30:24.893 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:30:24 smithi164 conmon[33677]: debug 2021-02-10T04:30:24.718+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:25.035 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:30:25.093 INFO:tasks.rados.rados.0.smithi085.stdout:599: done (7 left) 2021-02-10T04:30:25.094 INFO:tasks.rados.rados.0.smithi085.stdout:600: read oid 7 snap -1 2021-02-10T04:30:25.094 INFO:tasks.rados.rados.0.smithi085.stdout:600: expect (ObjNum 202 snap 44 seq_num 202) 2021-02-10T04:30:25.094 INFO:tasks.rados.rados.0.smithi085.stdout:601: write (excl) oid 16 current snap is 57 2021-02-10T04:30:25.094 INFO:tasks.rados.rados.0.smithi085.stdout:601: seq_num 241 ranges {450487=541952,1636278=414209} 2021-02-10T04:30:25.094 INFO:tasks.rados.rados.0.smithi085.stdout:601: writing smithi08575397-16 from 450487 to 992439 tid 1 2021-02-10T04:30:25.100 INFO:tasks.rados.rados.0.smithi085.stdout:601: writing smithi08575397-16 from 1636278 to 2050487 tid 2 2021-02-10T04:30:25.101 INFO:tasks.rados.rados.0.smithi085.stdout:602: snap_remove snap 52 2021-02-10T04:30:25.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:25 smithi164 conmon[29288]: cluster 2021-02-10T04:30:24.018091+0000 mgr.y (mgr.14140) 2021-02-10T04:30:25.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:25 smithi164 conmon[29288]: 401 : cluster [DBG] pgmap v460: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 164 active+clean; 161 MiB data, 494 MiB used, 715 GiB / 715 GiB avail; 15 MiB/s rd, 4.7 MiB/s wr, 23 op/s 2021-02-10T04:30:25.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:25 smithi164 conmon[29288]: cluster 2021-02-10T04:30:24.077430+0000 mon.a (mon.0) 915 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2021-02-10T04:30:25.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:25 smithi085 conmon[29319]: cluster 2021-02-10T04:30:24.018091+0000 mgr.y (mgr.14140) 401 : cluster [DBG] pgmap v460: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 164 active+clean; 161 MiB data, 494 MiB used, 715 GiB / 715 GiB avail; 15 MiB/s rd, 4.7 MiB/s wr, 23 op/s 2021-02-10T04:30:25.434 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:25 smithi085 conmon[29319]: cluster 2021-02-10T04:30:25.434 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:25 smithi085 conmon[29319]: 2021-02-10T04:30:24.077430+0000 mon.a (mon.0) 915 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2021-02-10T04:30:25.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:25 smithi085 conmon[35961]: cluster 2021-02-10T04:30:24.018091+0000 mgr.y (mgr.14140) 401 : cluster [DBG] pgmap v460: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 164 active+clean; 161 MiB data, 494 MiB used, 715 GiB / 715 GiB avail; 15 MiB/s rd, 4.7 MiB/s wr, 23 op/s 2021-02-10T04:30:25.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:25 smithi085 conmon[35961]: cluster 2021-02-10T04:30:24.077430+0000 mon.a (mon.0) 915 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2021-02-10T04:30:25.459 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:30:25.491 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:30:25.521 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:30:25.552 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:30:25.584 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:30:25.611 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:30:25.640 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:30:25.669 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:30:25.670 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:30:25.670 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:30:25.670 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:30:25.670 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:30:25.670 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:30:25.671 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:30:25.671 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:30:25.671 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:30:25.984 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:30:25 smithi085 conmon[49106]: debug 2021-02-10T04:30:25.870+0000 7f77a3e94700 -1 osd.2 209 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:30:25.985 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:25 smithi085 conmon[45443]: debug 2021-02-10T04:30:25.985 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:25 smithi085 conmon[45443]: 2021-02-10T04:30:25.983+0000 7f7140d73700 -1 osd.1 209 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:30:26.103 INFO:tasks.rados.rados.0.smithi085.stdout:602: done (9 left) 2021-02-10T04:30:26.103 INFO:tasks.rados.rados.0.smithi085.stdout:603: delete oid 40 current snap is 57 2021-02-10T04:30:26.103 INFO:tasks.rados.rados.0.smithi085.stdout:600: expect (ObjNum 202 snap 44 seq_num 202) 2021-02-10T04:30:26.355 INFO:tasks.rados.rados.0.smithi085.stdout:601: finishing write tid 1 to smithi08575397-16 2021-02-10T04:30:26.356 INFO:tasks.rados.rados.0.smithi085.stdout:601: finishing write tid 2 to smithi08575397-16 2021-02-10T04:30:26.356 INFO:tasks.rados.rados.0.smithi085.stdout:601: finishing write tid 3 to smithi08575397-16 2021-02-10T04:30:26.356 INFO:tasks.rados.rados.0.smithi085.stdout:601: finishing write tid 4 to smithi08575397-16 2021-02-10T04:30:26.356 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 264 (ObjNum 241 snap 57 seq_num 241) dirty exists 2021-02-10T04:30:26.356 INFO:tasks.rados.rados.0.smithi085.stdout:601: left oid 16 (ObjNum 241 snap 57 seq_num 241) 2021-02-10T04:30:26.356 INFO:tasks.rados.rados.0.smithi085.stdout:600: done (9 left) 2021-02-10T04:30:26.356 INFO:tasks.rados.rados.0.smithi085.stdout:601: done (8 left) 2021-02-10T04:30:26.356 INFO:tasks.rados.rados.0.smithi085.stdout:603: done (7 left) 2021-02-10T04:30:26.356 INFO:tasks.rados.rados.0.smithi085.stdout:604: rollback oid 36 current snap is 57 2021-02-10T04:30:26.357 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 36 to 56 2021-02-10T04:30:26.358 INFO:tasks.rados.rados.0.smithi085.stdout:605: read oid 45 snap 56 2021-02-10T04:30:26.358 INFO:tasks.rados.rados.0.smithi085.stdout:605: expect deleted 2021-02-10T04:30:26.359 INFO:tasks.rados.rados.0.smithi085.stdout:606: write oid 5 current snap is 57 2021-02-10T04:30:26.359 INFO:tasks.rados.rados.0.smithi085.stdout:606: seq_num 243 ranges {484101=692450,1636889=447212} 2021-02-10T04:30:26.362 INFO:tasks.rados.rados.0.smithi085.stdout:606: writing smithi08575397-5 from 484101 to 1176551 tid 1 2021-02-10T04:30:26.368 INFO:tasks.rados.rados.0.smithi085.stdout:606: writing smithi08575397-5 from 1636889 to 2084101 tid 2 2021-02-10T04:30:26.369 INFO:tasks.rados.rados.0.smithi085.stdout:607: snap_remove snap 55 2021-02-10T04:30:26.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:26 smithi164 conmon[29288]: cluster 2021-02-10T04:30:25.086614+0000 mon.a (mon.0) 916 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2021-02-10T04:30:26.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:26 smithi164 conmon[29288]: audit 2021-02-10T04:30:25.429205+0000 mon.a (mon.0) 917 : audit [DBG] from='client.? 172.21.15.85:0/657581864' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:30:26.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:30:26 smithi164 conmon[33677]: debug 2021-02-10T04:30:26.344+0000 7fbec19a1700 -1 osd.4 210 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:30:26.413 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[45443]: debug 2021-02-10T04:30:26.104+0000 7f71264f9700 -1 osd.1 pg_epoch: 210 pg[7.b( v 208'172 (0'0,208'172] local-lis/les=93/94 n=7 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [1,5,6] r=0 lpr=93 crt=208'172 lcod 207'170 mlcod 207'170 active+clean+snaptrim trimq=[68~1] ps=[60~3,66~2,6b~1]] removing snap head 2021-02-10T04:30:26.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[35961]: cluster 2021-02-10T04:30:26.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[35961]: 2021-02-10T04:30:26.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[35961]: 2021-02-10T04:30:25 2021-02-10T04:30:26.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[35961]: .086614+0000 mon.a (mon.0) 916 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2021-02-10T04:30:26.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[35961]: audit 2021-02-10T04:30:26.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[35961]: 2021-02-10T04:30:25. 2021-02-10T04:30:26.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[35961]: 429205+0000 mon.a (mon.0) 917 : audit [DBG] 2021-02-10T04:30:26.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/657581864' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:30:26.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: cluster 2021-02-10T04:30:26.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: 2021-02-10T04:30:26.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: 2021 2021-02-10T04:30:26.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: - 2021-02-10T04:30:26.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: 02 2021-02-10T04:30:26.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: - 2021-02-10T04:30:26.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: 10 2021-02-10T04:30:26.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: T 2021-02-10T04:30:26.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: 04 2021-02-10T04:30:26.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: : 2021-02-10T04:30:26.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: 30 2021-02-10T04:30:26.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: : 2021-02-10T04:30:26.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: 25 2021-02-10T04:30:26.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: . 2021-02-10T04:30:26.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: 086614 2021-02-10T04:30:26.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: +0000 2021-02-10T04:30:26.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: 2021-02-10T04:30:26.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: mon.a 2021-02-10T04:30:26.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: ( 2021-02-10T04:30:26.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: mon 2021-02-10T04:30:26.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: . 2021-02-10T04:30:26.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: 0 2021-02-10T04:30:26.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: ) 2021-02-10T04:30:26.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: 916 2021-02-10T04:30:26.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: : 2021-02-10T04:30:26.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: cluster 2021-02-10T04:30:26.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: 2021-02-10T04:30:26.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: [DBG] 2021-02-10T04:30:26.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: 2021-02-10T04:30:26.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: osdmap e209: 8 total, 8 up, 8 in 2021-02-10T04:30:26.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: 2021-02-10T04:30:26.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: audit 2021-02-10T04:30:26.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: 2021-02-10T04:30:26.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: 2021 2021-02-10T04:30:26.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: - 2021-02-10T04:30:26.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: 02 2021-02-10T04:30:26.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: - 2021-02-10T04:30:26.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: 10 2021-02-10T04:30:26.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: T 2021-02-10T04:30:26.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: 04:30:25.429205+0000 mon.a (mon 2021-02-10T04:30:26.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[29319]: .0) 917 : audit [DBG] from='client.? 172.21.15.85:0/657581864' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:30:26.437 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:30:26.464 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:26.610 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.6 dump_historic_ops 2021-02-10T04:30:26.643 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:30:26.690 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[52836]: debug 2021-02-10T04:30:26.411+0000 7fba73fb6700 -1 osd.3 pg_epoch: 210 pg[4.14( v 63'26 (0'0,63'26] local-lis/les=71/72 n=11 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,2,7] r=0 lpr=71 crt=63'26 lcod 63'25 mlcod 0'0 active+clean] scrubber pg(4.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55fec06b4000) 2021-02-10T04:30:26.748 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:30:26.786 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:26.893 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:30:26 smithi164 conmon[37263]: debug 2021-02-10T04:30:26.561+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:26.915 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:30:27.104 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[45443]: debug 2021-02-10T04:30:26.985+0000 7f7140d73700 -1 osd.1 210 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:30:27.104 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:30:26 smithi085 conmon[49106]: debug 2021-02-10T04:30:26.850+0000 7f77a3e94700 -1 osd.2 210 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:30:27.111 INFO:tasks.rados.rados.0.smithi085.stdout:607: done (10 left) 2021-02-10T04:30:27.112 INFO:tasks.rados.rados.0.smithi085.stdout:608: write (excl) oid 26 current snap is 57 2021-02-10T04:30:27.112 INFO:tasks.rados.rados.0.smithi085.stdout:608: seq_num 244 ranges {500908=484052,1411724=689184} 2021-02-10T04:30:27.114 INFO:tasks.rados.rados.0.smithi085.stdout:608: writing smithi08575397-26 from 500908 to 984960 tid 1 2021-02-10T04:30:27.121 INFO:tasks.rados.rados.0.smithi085.stdout:608: writing smithi08575397-26 from 1411724 to 2100908 tid 2 2021-02-10T04:30:27.121 INFO:tasks.rados.rados.0.smithi085.stdout:609: snap_remove snap 54 2021-02-10T04:30:27.365 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[40892]: debug 2021-02-10T04:30:27.222+0000 7f30314bd700 -1 osd.6 pg_epoch: 211 pg[7.7( v 210'270 (0'0,210'270] local-lis/les=93/94 n=13 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [6,0,3] r=0 lpr=93 crt=210'270 lcod 209'268 mlcod 209'268 active+clean+snaptrim trimq=[68~1,6d~1] ps=[60~3,66~2,6b~1]] removing snap head 2021-02-10T04:30:27.368 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[29288]: cluster 2021-02-10T04:30:25.864845+0000 osd.3 (osd. 2021-02-10T04:30:27.368 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[29288]: 3) 54 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:30:27.368 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[29288]: cluster 2021-02-10T04:30:25.865054+0000 osd.3 (osd.3) 55 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:30:27.368 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[29288]: cluster 2021-02-10T04:30:25.872534+0000 osd.2 (osd.2) 48 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:30:27.368 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[29288]: cluster 2021-02-10T04:30:25.872553+0000 osd.2 (osd.2) 49 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:30:27.368 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[29288]: cluster 2021-02-10T04:30:25.985039+0000 osd.1 (osd.1) 80 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:30:27.368 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:30:27.368 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[29288]: -02-10T04:30:25.985052+0000 osd.1 (osd.1) 81 : cluster 2021-02-10T04:30:27.369 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:30:27.369 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[29288]: cluster 2021-02-10T04:30:27.369 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[29288]: 2021-02-10T04:30:26.018650+0000 mgr.y (mgr.14140) 402 : cluster [DBG] 2021-02-10T04:30:27.369 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[29288]: pgmap v463: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 4 active+clean+snaptrim, 162 active+clean; 163 MiB data, 501 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 3.4 MiB/s wr, 11 op/s 2021-02-10T04:30:27.369 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[29288]: cluster 2021-02-10T04:30:26.101884 2021-02-10T04:30:27.369 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[29288]: +0000 mon.a (mon.0) 918 : cluster [DBG] osdmap e210: 8 total, 8 up, 8 in 2021-02-10T04:30:27.369 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[29288]: 2021-02-10T04:30:27.369 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:30:27.369 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[29288]: 30:26.345070+0000 osd.4 (osd.4) 49 2021-02-10T04:30:27.369 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:27.369 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:30:27.370 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[29288]: -02-10T04:30:26.345084+0000 2021-02-10T04:30:27.370 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[29288]: osd.4 (osd.4) 50 : cluster [WRN] 2021-02-10T04:30:27.370 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:27.370 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:30:27.370 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[29288]: 30:26.345088+0000 osd.4 (osd.4) 51 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:27.370 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[29288]: 2021-02-10T04:30:27.370 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:30:27.370 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[29288]: -10T04:30:26.345093+0000 osd.4 (osd 2021-02-10T04:30:27.370 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[29288]: .4) 52 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:27.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: cluster 2021-02-10T04:30:27.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: 2021-02-10T04:30:27.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: 2021 2021-02-10T04:30:27.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: - 2021-02-10T04:30:27.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: 02 2021-02-10T04:30:27.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: - 2021-02-10T04:30:27.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: 10 2021-02-10T04:30:27.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: T 2021-02-10T04:30:27.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: 04 2021-02-10T04:30:27.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: : 2021-02-10T04:30:27.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: 30 2021-02-10T04:30:27.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: : 2021-02-10T04:30:27.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: 25.864845+0000 osd.3 (osd.3) 2021-02-10T04:30:27.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: 54 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:30:27.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: cluster 2021-02-10T04:30:25.865054+0000 osd.3 (osd 2021-02-10T04:30:27.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: .3) 55 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:30:27.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: cluster 2021-02-10T04:30:27.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: 2021-02-10T04:30:25.872534+0000 2021-02-10T04:30:27.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: osd.2 (osd.2) 48 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:30:27.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: 2021-02-10T04:30:27.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: cluster 2021-02-10T04:30:25 2021-02-10T04:30:27.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: .872553+0000 osd.2 (osd.2) 49 : cluster 2021-02-10T04:30:27.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:30:27.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: cluster 2021-02-10T04:30 2021-02-10T04:30:27.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: :25.985039+0000 osd.1 (osd.1) 80 2021-02-10T04:30:27.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:30:27.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:30:27.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: 04:30:25.985052+0000 osd.1 (osd. 2021-02-10T04:30:27.416 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: 1) 81 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:30:27.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:30:27.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: 10T04:30:26.018650+0000 mgr.y ( 2021-02-10T04:30:27.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: mgr.14140) 402 : cluster [DBG] pgmap v463: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 4 active+clean+snaptrim, 162 active+clean; 163 MiB data, 501 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 3.4 MiB/s wr, 11 op/s 2021-02-10T04:30:27.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:30:27.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: 02-10T04:30:26.101884+0000 2021-02-10T04:30:27.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: mon.a (mon.0) 918 : cluster [DBG] osdmap e210: 8 total, 8 up, 8 in 2021-02-10T04:30:27.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: 2021-02-10T04:30:27.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: cluster 2021-02-10T04:30:27.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: 2021-02-10T04:30:26.345070+0000 osd.4 (osd 2021-02-10T04:30:27.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: .4) 49 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:27.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:30:27.418 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: 04:30:26.345084+0000 osd.4 (osd.4) 2021-02-10T04:30:27.418 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: 50 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:27.418 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: cluster 2021-02-10T04:30 2021-02-10T04:30:27.418 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: :26.345088+0000 osd.4 (osd.4) 51 2021-02-10T04:30:27.418 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:27.418 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[29319]: cluster 2021-02-10T04:30:26.345093+0000 osd.4 (osd.4) 52 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:27.419 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: cluster 2021-02-10T04:30:25.864845 2021-02-10T04:30:27.419 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: +0000 osd.3 (osd.3) 54 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:30:27.419 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:30:27.419 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: T04:30:25.865054+0000 osd.3 (osd. 2021-02-10T04:30:27.419 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: 3) 55 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:30:27.419 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:30:27.419 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: 02-10T 2021-02-10T04:30:27.419 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: 04:30:25.872534+0000 osd.2 ( 2021-02-10T04:30:27.419 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: osd.2) 48 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:30:27.419 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:30:27.419 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: -02-10T04:30:25.872553+0000 2021-02-10T04:30:27.420 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: osd.2 (osd.2) 49 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:30:27.420 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: 2021-02-10T04:30:27.420 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:30:27.420 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: -02-10T04:30:25.985039+0000 osd.1 (osd. 2021-02-10T04:30:27.420 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: 1) 80 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:30:27.420 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:30:27.420 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: 02-10T04:30:25. 2021-02-10T04:30:27.420 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: 985052+0000 osd.1 (osd.1) 81 : cluster [WRN] 2021-02-10T04:30:27.420 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:30:27.420 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: cluster 2021-02-10T04:30: 2021-02-10T04:30:27.420 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: 26.018650+0000 mgr.y (mgr.14140) 402 : 2021-02-10T04:30:27.420 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: cluster [DBG] pgmap v463: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 4 active+clean+snaptrim, 162 active+clean; 163 MiB data, 501 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 3.4 MiB/s wr, 11 op/s 2021-02-10T04:30:27.420 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: cluster 2021-02-10T04:30 2021-02-10T04:30:27.421 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: :26.101884+0000 mon.a (mon.0) 918 : cluster 2021-02-10T04:30:27.421 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: [DBG] osdmap e210: 8 total, 8 up, 8 in 2021-02-10T04:30:27.421 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:30:27.421 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: :30:26.345070+0000 osd.4 (osd.4 2021-02-10T04:30:27.421 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: ) 49 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:27.421 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:30:27.421 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: 10T04:30:26.345084+0000 osd.4 (osd.4) 50 : 2021-02-10T04:30:27.421 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:27.421 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: cluster 2021-02-10T04:30:26.345088+0000 osd.4 (osd 2021-02-10T04:30:27.421 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: .4) 51 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:27.421 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: cluster 2021-02-10T04:30:27.421 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: 2021-02-10T04:30:26.345093+0000 osd.4 (osd.4 2021-02-10T04:30:27.422 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[35961]: ) 52 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:27.630 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[33677]: debug 2021-02-10T04:30:27.365+0000 7fbec19a1700 -1 osd.4 211 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:30:27.691 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[52836]: debug 2021-02-10T04:30:27.413+0000 7fba6ffae700 -1 osd.3 pg_epoch: 211 pg[4.14( v 63'26 (0'0,63'26] local-lis/les=71/72 n=11 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,2,7] r=0 lpr=71 crt=63'26 lcod 63'25 mlcod 0'0 active+clean] scrubber pg(4.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55feb3a02c60) 2021-02-10T04:30:27.893 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:30:27 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:30:28.147 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[45443]: debug 2021-02-10T04:30:27.950+0000 7f7140d73700 -1 osd.1 211 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:30:28.147 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:30:27 smithi085 conmon[49106]: debug 2021-02-10T04:30:27.861+0000 7f77a3e94700 -1 osd.2 211 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:30:28.147 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[29319]: cluster 2021-02-10T04:30:26.852160+0000 osd.2 2021-02-10T04:30:28.147 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[29319]: (osd.2) 50 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:30:28.147 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[29319]: cluster 2021-02-10T04:30:26.852167+0000 osd.2 (osd.2) 51 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:30:28.148 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[29319]: cluster 2021-02-10T04:30:26. 2021-02-10T04:30:28.148 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[29319]: 986852+0000 osd.1 (osd.1) 82 : cluster 2021-02-10T04:30:28.148 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:30:28.148 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:30:28.148 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[29319]: -10T04:30:26.986863+0000 2021-02-10T04:30:28.148 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[29319]: osd.1 (osd.1) 83 : cluster [WRN] 2021-02-10T04:30:28.148 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[29319]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:30:28.148 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:30:28.148 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[29319]: T04:30:27.107825+0000 mon.a ( 2021-02-10T04:30:28.148 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[29319]: mon.0) 919 : cluster [DBG] osdmap e211: 8 total, 8 up, 8 in 2021-02-10T04:30:28.148 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:30:28.148 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[29319]: 04:30:27.365973+0000 osd.4 (osd.4) 53 : cluster 2021-02-10T04:30:28.148 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:28.149 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:30:28.149 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[29319]: 02-10T04:30:27.365983+0000 osd.4 (osd.4) 54 : cluster [WRN] 2021-02-10T04:30:28.149 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:28.149 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:30:28.149 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[29319]: :30:27.365991+0000 osd.4 (osd.4) 55 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:28.149 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[29319]: cluster 2021-02-10T04:30:27.365997+0000 osd.4 (osd. 2021-02-10T04:30:28.149 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[29319]: 4) 56 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:28.150 INFO:tasks.rados.rados.0.smithi085.stdout:604: finishing rollback tid 0 to smithi08575397-36 2021-02-10T04:30:28.150 INFO:tasks.rados.rados.0.smithi085.stdout:604: finishing rollback tid 1 to smithi08575397-36 2021-02-10T04:30:28.150 INFO:tasks.rados.rados.0.smithi085.stdout:604: finishing rollback tid 2 to smithi08575397-36 2021-02-10T04:30:28.150 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 98 (ObjNum 237 snap 54 seq_num 237) dirty exists 2021-02-10T04:30:28.150 INFO:tasks.rados.rados.0.smithi085.stdout:606: finishing write tid 1 to smithi08575397-5 2021-02-10T04:30:28.150 INFO:tasks.rados.rados.0.smithi085.stdout:606: finishing write tid 2 to smithi08575397-5 2021-02-10T04:30:28.150 INFO:tasks.rados.rados.0.smithi085.stdout:606: finishing write tid 3 to smithi08575397-5 2021-02-10T04:30:28.150 INFO:tasks.rados.rados.0.smithi085.stdout:606: finishing write tid 4 to smithi08575397-5 2021-02-10T04:30:28.150 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 200 (ObjNum 243 snap 57 seq_num 243) dirty exists 2021-02-10T04:30:28.150 INFO:tasks.rados.rados.0.smithi085.stdout:606: left oid 5 (ObjNum 243 snap 57 seq_num 243) 2021-02-10T04:30:28.151 INFO:tasks.rados.rados.0.smithi085.stdout:608: finishing write tid 1 to smithi08575397-26 2021-02-10T04:30:28.151 INFO:tasks.rados.rados.0.smithi085.stdout:608: finishing write tid 2 to smithi08575397-26 2021-02-10T04:30:28.151 INFO:tasks.rados.rados.0.smithi085.stdout:608: finishing write tid 3 to smithi08575397-26 2021-02-10T04:30:28.151 INFO:tasks.rados.rados.0.smithi085.stdout:608: finishing write tid 4 to smithi08575397-26 2021-02-10T04:30:28.151 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 43 (ObjNum 244 snap 57 seq_num 244) dirty exists 2021-02-10T04:30:28.151 INFO:tasks.rados.rados.0.smithi085.stdout:608: left oid 26 (ObjNum 244 snap 57 seq_num 244) 2021-02-10T04:30:28.151 INFO:tasks.rados.rados.0.smithi085.stdout:604: done (11 left) 2021-02-10T04:30:28.151 INFO:tasks.rados.rados.0.smithi085.stdout:605: done (10 left) 2021-02-10T04:30:28.151 INFO:tasks.rados.rados.0.smithi085.stdout:606: done (9 left) 2021-02-10T04:30:28.151 INFO:tasks.rados.rados.0.smithi085.stdout:608: done (8 left) 2021-02-10T04:30:28.151 INFO:tasks.rados.rados.0.smithi085.stdout:609: done (7 left) 2021-02-10T04:30:28.151 INFO:tasks.rados.rados.0.smithi085.stdout:610: delete oid 26 current snap is 57 2021-02-10T04:30:28.158 INFO:tasks.rados.rados.0.smithi085.stdout:610: done (7 left) 2021-02-10T04:30:28.158 INFO:tasks.rados.rados.0.smithi085.stdout:611: write oid 37 current snap is 57 2021-02-10T04:30:28.158 INFO:tasks.rados.rados.0.smithi085.stdout:611: seq_num 246 ranges {534522=634550,1812335=322187} 2021-02-10T04:30:28.166 INFO:tasks.rados.rados.0.smithi085.stdout:611: writing smithi08575397-37 from 534522 to 1169072 tid 1 2021-02-10T04:30:28.172 INFO:tasks.rados.rados.0.smithi085.stdout:611: writing smithi08575397-37 from 1812335 to 2134522 tid 2 2021-02-10T04:30:28.172 INFO:tasks.rados.rados.0.smithi085.stdout:611: finishing write tid 1 to smithi08575397-37 2021-02-10T04:30:28.172 INFO:tasks.rados.rados.0.smithi085.stdout:612: read oid 9 snap -1 2021-02-10T04:30:28.172 INFO:tasks.rados.rados.0.smithi085.stdout:612: expect (ObjNum 229 snap 51 seq_num 229) 2021-02-10T04:30:28.172 INFO:tasks.rados.rados.0.smithi085.stdout:613: delete oid 34 current snap is 57 2021-02-10T04:30:28.182 INFO:tasks.rados.rados.0.smithi085.stdout:612: expect (ObjNum 229 snap 51 seq_num 229) 2021-02-10T04:30:28.440 INFO:tasks.rados.rados.0.smithi085.stdout:612: done (9 left) 2021-02-10T04:30:28.440 INFO:tasks.rados.rados.0.smithi085.stdout:613: done (8 left) 2021-02-10T04:30:28.440 INFO:tasks.rados.rados.0.smithi085.stdout:614: read oid 45 snap -1 2021-02-10T04:30:28.440 INFO:tasks.rados.rados.0.smithi085.stdout:614: expect deleted 2021-02-10T04:30:28.441 INFO:tasks.rados.rados.0.smithi085.stdout:615: delete oid 10 current snap is 57 2021-02-10T04:30:28.441 INFO:tasks.rados.rados.0.smithi085.stdout:611: finishing write tid 2 to smithi08575397-37 2021-02-10T04:30:28.441 INFO:tasks.rados.rados.0.smithi085.stdout:611: finishing write tid 3 to smithi08575397-37 2021-02-10T04:30:28.441 INFO:tasks.rados.rados.0.smithi085.stdout:611: finishing write tid 4 to smithi08575397-37 2021-02-10T04:30:28.441 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 277 (ObjNum 246 snap 57 seq_num 246) dirty exists 2021-02-10T04:30:28.441 INFO:tasks.rados.rados.0.smithi085.stdout:611: left oid 37 (ObjNum 246 snap 57 seq_num 246) 2021-02-10T04:30:28.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: cluster 2021-02-10T04:30:28.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 2021-02-10T04:30:28.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 2021 2021-02-10T04:30:28.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: - 2021-02-10T04:30:28.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 02 2021-02-10T04:30:28.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: - 2021-02-10T04:30:28.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 10 2021-02-10T04:30:28.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: T 2021-02-10T04:30:28.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 04 2021-02-10T04:30:28.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: : 2021-02-10T04:30:28.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 30 2021-02-10T04:30:28.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: : 2021-02-10T04:30:28.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 26 2021-02-10T04:30:28.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: . 2021-02-10T04:30:28.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 852160 2021-02-10T04:30:28.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: +0000 2021-02-10T04:30:28.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 2021-02-10T04:30:28.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: osd.2 (osd.2) 50 : cluster 2021-02-10T04:30:28.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:30:28.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: cluster 2021-02-10T04:30:28.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 2021-02-10T04:30:28.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 2021 2021-02-10T04:30:28.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: - 2021-02-10T04:30:28.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 02 2021-02-10T04:30:28.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: - 2021-02-10T04:30:28.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 10 2021-02-10T04:30:28.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: T 2021-02-10T04:30:28.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 04 2021-02-10T04:30:28.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: : 2021-02-10T04:30:28.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 30 2021-02-10T04:30:28.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: : 2021-02-10T04:30:28.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 26 2021-02-10T04:30:28.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: . 2021-02-10T04:30:28.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 852167 2021-02-10T04:30:28.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: +0000 2021-02-10T04:30:28.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 2021-02-10T04:30:28.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: osd.2 2021-02-10T04:30:28.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: ( 2021-02-10T04:30:28.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: osd 2021-02-10T04:30:28.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: . 2021-02-10T04:30:28.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 2 2021-02-10T04:30:28.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: ) 2021-02-10T04:30:28.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 51 2021-02-10T04:30:28.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: : 2021-02-10T04:30:28.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: cluster 2021-02-10T04:30:28.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 2021-02-10T04:30:28.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: [WRN] 2021-02-10T04:30:28.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 2021-02-10T04:30:28.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:30:28.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 2021-02-10T04:30:28.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: cluster 2021-02-10T04:30:28.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 2021-02-10T04:30:28.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 2021 2021-02-10T04:30:28.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: - 2021-02-10T04:30:28.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 02 2021-02-10T04:30:28.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: - 2021-02-10T04:30:28.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 10 2021-02-10T04:30:28.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: T 2021-02-10T04:30:28.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 04 2021-02-10T04:30:28.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: : 2021-02-10T04:30:28.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 30 2021-02-10T04:30:28.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: : 2021-02-10T04:30:28.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 26 2021-02-10T04:30:28.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: . 2021-02-10T04:30:28.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 986852 2021-02-10T04:30:28.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: +0000 2021-02-10T04:30:28.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 2021-02-10T04:30:28.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: osd.1 2021-02-10T04:30:28.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: ( 2021-02-10T04:30:28.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: osd 2021-02-10T04:30:28.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: . 2021-02-10T04:30:28.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 1 2021-02-10T04:30:28.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: ) 2021-02-10T04:30:28.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 82 2021-02-10T04:30:28.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: : 2021-02-10T04:30:28.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: cluster 2021-02-10T04:30:28.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 2021-02-10T04:30:28.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: [WRN] 2021-02-10T04:30:28.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 2021-02-10T04:30:28.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:30:28.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 2021-02-10T04:30:28.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: cluster 2021-02-10T04:30:28.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 2021-02-10T04:30:28.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 2021 2021-02-10T04:30:28.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: - 2021-02-10T04:30:28.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 02 2021-02-10T04:30:28.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: - 2021-02-10T04:30:28.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 10 2021-02-10T04:30:28.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: T 2021-02-10T04:30:28.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 04 2021-02-10T04:30:28.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: : 2021-02-10T04:30:28.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 30 2021-02-10T04:30:28.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: :26.986863+0000 osd.1 (osd. 2021-02-10T04:30:28.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 1) 83 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:30:28.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:30:28.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 02-10T04:30:27.107825+0000 mon.a (mon 2021-02-10T04:30:28.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: .0) 919 : cluster [DBG] osdmap e211: 8 total, 8 up, 8 in 2021-02-10T04:30:28.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: cluster 2021-02-10T04:30:28.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:30:28.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: T04:30:27.365973+0000 osd.4 (osd.4) 53 : cluster 2021-02-10T04:30:28.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:28.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:30:28.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 02-10T04:30:27.365983+0000 2021-02-10T04:30:28.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: osd.4 (osd.4) 54 : cluster [WRN] 2021-02-10T04:30:28.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:28.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: cluster 2021-02-10T04:30:27. 2021-02-10T04:30:28.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 365991+0000 osd.4 (osd.4) 55 : cluster 2021-02-10T04:30:28.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:28.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:30:28.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 30:27.365997+0000 osd.4 (osd.4) 2021-02-10T04:30:28.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[35961]: 56 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:28.453 INFO:tasks.rados.rados.0.smithi085.stdout:611: done (9 left) 2021-02-10T04:30:28.453 INFO:tasks.rados.rados.0.smithi085.stdout:614: done (8 left) 2021-02-10T04:30:28.453 INFO:tasks.rados.rados.0.smithi085.stdout:615: done (7 left) 2021-02-10T04:30:28.453 INFO:tasks.rados.rados.0.smithi085.stdout:616: copy_from oid 16 from oid 15 current snap is 57 2021-02-10T04:30:28.453 INFO:tasks.rados.rados.0.smithi085.stdout:617: copy_from oid 44 from oid 21 current snap is 57 2021-02-10T04:30:28.453 INFO:tasks.rados.rados.0.smithi085.stdout:618: snap_remove snap 56 2021-02-10T04:30:28.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:30:28 smithi164 conmon[33677]: debug 2021-02-10T04:30:28.352+0000 7fbec19a1700 -1 osd.4 212 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:30:28.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:28 smithi164 conmon[29288]: cluster 2021-02-10T04:30:26.852160 2021-02-10T04:30:28.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:28 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 50 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:30:28.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:28 smithi164 conmon[29288]: cluster 2021-02-10T04:30:26.852167+0000 2021-02-10T04:30:28.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:28 smithi164 conmon[29288]: osd.2 (osd.2) 51 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:30:28.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:28 smithi164 conmon[29288]: cluster 2021-02-10T04:30:28.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:28 smithi164 conmon[29288]: 2021-02-10T04:30:26.986852+0000 2021-02-10T04:30:28.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:28 smithi164 conmon[29288]: osd.1 (osd.1) 82 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:30:28.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:28 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:30:28.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:28 smithi164 conmon[29288]: 04:30:26.986863+0000 osd.1 (osd.1) 83 2021-02-10T04:30:28.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:28 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:30:28.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:28 smithi164 conmon[29288]: cluster 2021-02-10T04:30:28.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:28 smithi164 conmon[29288]: 2021-02-10T04:30:27.107825+0000 mon.a (mon.0) 919 : cluster 2021-02-10T04:30:28.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:28 smithi164 conmon[29288]: [DBG] osdmap e211: 8 total, 8 up, 8 in 2021-02-10T04:30:28.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:28 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:30:28.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:28 smithi164 conmon[29288]: -10T04:30:27.365973+0000 osd.4 (osd.4) 2021-02-10T04:30:28.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:28 smithi164 conmon[29288]: 53 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:28.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:28 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:30:28.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:28 smithi164 conmon[29288]: 10T04:30:27.365983+0000 osd.4 ( 2021-02-10T04:30:28.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:28 smithi164 conmon[29288]: osd.4) 54 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:28.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:28 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:30:28.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:28 smithi164 conmon[29288]: 02-10T04:30:27.365991+0000 osd.4 (osd.4 2021-02-10T04:30:28.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:28 smithi164 conmon[29288]: ) 55 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:28.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:28 smithi164 conmon[29288]: cluster 2021-02-10T04:30:27.365997+0000 2021-02-10T04:30:28.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:28 smithi164 conmon[29288]: osd.4 (osd.4) 56 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:29.148 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[45443]: debug 2021-02-10T04:30:29.148 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[45443]: 2021-02-10T04:30:28.993+0000 7f7140d73700 -1 osd.1 212 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:30:29.148 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[45443]: 2021-02-10T04:30:29.151 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:30:28 smithi085 conmon[49106]: debug 2021-02-10T04:30:28.871+0000 7f77a3e94700 -1 osd.2 212 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:30:29.154 INFO:tasks.rados.rados.0.smithi085.stdout:618: done (9 left) 2021-02-10T04:30:29.154 INFO:tasks.rados.rados.0.smithi085.stdout:619: snap_create 2021-02-10T04:30:29.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: audit 2021-02-10T04:30:27.785023+0000 2021-02-10T04:30:29.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: mgr.y (mgr.14140) 403 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:30:29.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: cluster 2021-02-10T04:30:27.862447+0000 2021-02-10T04:30:29.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: osd.2 (osd.2) 52 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:30:29.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:30:29.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: 04:30:27.862455+0000 osd.2 (osd. 2021-02-10T04:30:29.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: 2) 53 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:30:29.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: cluster 2021-02-10T04:30:29.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: 2021-02-10T04:30:27 2021-02-10T04:30:29.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: .951450+0000 osd.1 (osd.1) 84 : cluster 2021-02-10T04:30:29.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:30:29.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: cluster 2021-02-10T04:30:29.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: 2021-02-10T04:30:27.951460 2021-02-10T04:30:29.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: +0000 osd.1 (osd.1) 85 : cluster [WRN] 2021-02-10T04:30:29.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:30:29.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:30:29.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: :30:28.019025+0000 mgr.y (mgr.14140 2021-02-10T04:30:29.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: ) 404 : cluster [DBG] pgmap v466: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 4 active+clean+snaptrim, 162 active+clean; 163 MiB data, 497 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 3.0 MiB/s wr, 7 op/s 2021-02-10T04:30:29.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:30:29.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: -10T04:30:28.104820+0000 mon.a ( 2021-02-10T04:30:29.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: mon.0) 920 : cluster [WRN] Health check failed: 6 slow ops, oldest one blocked for 57 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:30:29.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:30:29.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: -02-10T04:30:28.148800+0000 2021-02-10T04:30:29.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: mon.a (mon.0) 921 : cluster [DBG] 2021-02-10T04:30:29.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: osdmap e212: 8 total, 8 up, 8 in 2021-02-10T04:30:29.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: cluster 2021-02-10T04:30:29.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: 2021-02-10T04:30:28.353459+0000 osd.4 2021-02-10T04:30:29.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: (osd.4) 57 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:29.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:30:29.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: 02-10T04:30:28.353466+0000 osd.4 2021-02-10T04:30:29.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: (osd.4) 58 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:29.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: 2021-02-10T04:30:29.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:30:29.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: -02-10T04:30:28.353474+0000 osd.4 (osd.4 2021-02-10T04:30:29.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: ) 59 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:29.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:30:29.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: 04:30:28.353480+0000 osd.4 (osd.4 2021-02-10T04:30:29.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[29319]: ) 60 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:29.445 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[41540]: debug 2021-02-10T04:30:29.445 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[41540]: 2021-02-10T04:30:29.158+0000 7fb94ab61700 -1 osd.0 pg_epoch: 213 pg[7.f( v 212'88 (0'0,212'88] local-lis/les=93/94 n=5 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [0,3,7] r=0 lpr=93 crt=212'88 lcod 210'86 mlcod 210'86 active+clean+snaptrim trimq=[6f~1] ps=[60~3,66~3,6b~3]] removing snap head 2021-02-10T04:30:29.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: audit 2021-02-10T04:30: 2021-02-10T04:30:29.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: 27.785023+0000 mgr.y (mgr.14140) 403 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:30:29.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:30:29.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: -02-10T04:30:27.862447+0000 osd.2 2021-02-10T04:30:29.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: (osd.2) 52 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:30:29.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: cluster 2021-02-10T04:30 2021-02-10T04:30:29.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: :27.862455+0000 osd.2 (osd.2) 53 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:30:29.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: cluster 2021-02-10T04:30:27.951450+0000 osd.1 (osd. 2021-02-10T04:30:29.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: 1) 84 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:30:29.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:30:29.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: 10T04:30:27.951460+0000 osd.1 (osd 2021-02-10T04:30:29.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: .1) 85 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:30:29.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: cluster 2021-02-10T04:30:28.019025+0000 mgr.y (mgr. 2021-02-10T04:30:29.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: 14140) 404 : cluster [DBG] pgmap v466: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 4 active+clean+snaptrim, 162 active+clean; 163 MiB data, 497 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 3.0 MiB/s wr, 7 op/s 2021-02-10T04:30:29.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: cluster 2021-02-10T04:30 2021-02-10T04:30:29.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: :28.104820+0000 mon.a (mon.0) 920 : cluster [WRN] Health check failed: 6 slow ops, oldest one blocked for 57 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:30:29.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: cluster 2021-02-10T04:30:29.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: 2021-02-10T04:30:28.148800+0000 2021-02-10T04:30:29.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: mon.a (mon.0) 921 : cluster [DBG] osdmap e212: 8 total, 8 up, 8 in 2021-02-10T04:30:29.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: cluster 2021-02-10T04:30:29.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:30:29.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: :30:28.353459+0000 osd.4 (osd.4) 2021-02-10T04:30:29.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: 57 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:29.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: cluster 2021-02-10T04:30:29.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: 2021-02-10T04:30 2021-02-10T04:30:29.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: :28.353466+0000 osd.4 (osd.4) 2021-02-10T04:30:29.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: 58 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:29.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:30:29.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: -10T04:30:28.353474+0000 osd.4 ( 2021-02-10T04:30:29.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: osd.4) 59 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:29.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:30:29.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: -10T04:30:28.353480+0000 osd.4 ( 2021-02-10T04:30:29.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[35961]: osd.4) 60 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:29.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[33677]: debug 2021-02-10T04:30:29.311+0000 7fbec19a1700 -1 osd.4 213 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:30:29.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: audit 2021-02-10T04:30:27.785023+0000 mgr.y (mgr 2021-02-10T04:30:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: .14140) 403 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:30:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: cluster 2021-02-10T04:30:27.862447+0000 osd.2 (osd.2) 52 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:30:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: cluster 2021-02-10T04:30:27.862455+0000 osd.2 (osd.2) 53 2021-02-10T04:30:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:30:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:30:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: -02-10T04:30:27.951450 2021-02-10T04:30:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 84 : cluster [WRN] 2021-02-10T04:30:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:30:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:30:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: 30:27.951460+0000 osd.1 (osd.1) 2021-02-10T04:30:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: 85 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:30:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:30:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: 04:30:28.019025+0000 mgr.y (mgr 2021-02-10T04:30:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: .14140) 404 : cluster [DBG] pgmap v466: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 4 active+clean+snaptrim, 162 active+clean; 163 MiB data, 497 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 3.0 MiB/s wr, 7 op/s 2021-02-10T04:30:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:30:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: 02-10T04:30:28.104820+0000 2021-02-10T04:30:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: mon.a (mon.0) 920 : cluster [WRN] 2021-02-10T04:30:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: Health check failed: 6 slow ops, oldest one blocked for 57 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:30:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: cluster 2021-02-10T04:30: 2021-02-10T04:30:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: 28.148800+0000 mon.a (mon.0) 921 2021-02-10T04:30:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: : cluster [DBG] osdmap e212: 8 total, 8 up, 8 in 2021-02-10T04:30:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:30:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: 04:30:28.353459+0000 osd.4 (osd 2021-02-10T04:30:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: .4) 57 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: 2021-02-10T04:30:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: cluster 2021-02-10T04:30:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: 2021-02-10T04:30:28.353466 2021-02-10T04:30:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 58 : 2021-02-10T04:30:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:30:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: -02-10T04:30:28 2021-02-10T04:30:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: .353474+0000 osd.4 (osd. 2021-02-10T04:30:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: 4) 59 : cluster [WRN] 2021-02-10T04:30:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:30:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: -10T04:30:28.353480+0000 osd.4 (osd.4) 60 2021-02-10T04:30:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:29 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:29.648 INFO:tasks.rados.rados.0.smithi085.stdout:616: finishing copy_from to smithi08575397-16 2021-02-10T04:30:29.648 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 104 (ObjNum 217 snap 49 seq_num 217) dirty exists 2021-02-10T04:30:29.648 INFO:tasks.rados.rados.0.smithi085.stdout:616: finishing copy_from racing read to smithi08575397-16 2021-02-10T04:30:29.873 INFO:tasks.rados.rados.0.smithi085.stdout:617: finishing copy_from to smithi08575397-44 2021-02-10T04:30:29.873 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 217 (ObjNum 194 snap 43 seq_num 194) dirty exists 2021-02-10T04:30:29.873 INFO:tasks.rados.rados.0.smithi085.stdout:617: finishing copy_from racing read to smithi08575397-44 2021-02-10T04:30:30.151 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:30:30.157 INFO:tasks.rados.rados.0.smithi085.stdout:616: done (9 left) 2021-02-10T04:30:30.158 INFO:tasks.rados.rados.0.smithi085.stdout:617: done (8 left) 2021-02-10T04:30:30.158 INFO:tasks.rados.rados.0.smithi085.stdout:619: done (7 left) 2021-02-10T04:30:30.158 INFO:tasks.rados.rados.0.smithi085.stdout:620: delete oid 24 current snap is 58 2021-02-10T04:30:30.159 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[45443]: debug 2021-02-10T04:30:30.003+0000 7f7140d73700 -1 osd.1 213 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:30:30.163 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[49106]: debug 2021-02-10T04:30:29.860+0000 7f77a3e94700 -1 osd.2 213 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:30:30.163 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:30:29 smithi085 conmon[52836]: debug 2021-02-10T04:30:29.894+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:30.163 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: cluster 2021-02-10T04:30:28.871642+0000 2021-02-10T04:30:30.164 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: osd.2 (osd.2) 54 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:30:30.164 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:30:30.164 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: 30:28.871650+0000 osd.2 (osd.2) 55 : cluster 2021-02-10T04:30:30.164 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:30:30.164 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:30:30.164 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: -10T04:30:28.993928+0000 osd.1 (osd.1) 86 : 2021-02-10T04:30:30.164 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:30:30.164 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:30:30.164 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: -10T04:30:28.993937+0000 osd.1 (osd.1 2021-02-10T04:30:30.164 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: ) 87 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:30:30.165 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:30:30.165 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: -10T04:30:29.154437+0000 mon.a (mon.0) 2021-02-10T04:30:30.165 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: 922 : cluster [DBG] osdmap e213: 8 total, 8 up, 8 in 2021-02-10T04:30:30.165 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:30:30.165 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: -10T04:30:29.312009+0000 osd.4 (osd.4) 2021-02-10T04:30:30.165 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: 61 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:30.165 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:30:30.165 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: -10T04:30:29.312020+0000 osd.4 (osd.4 2021-02-10T04:30:30.165 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: ) 62 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:30.165 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: cluster 2021-02-10T04:30:30.166 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: 2021-02-10T04:30:29.312029+0000 2021-02-10T04:30:30.166 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: osd.4 (osd.4) 63 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:30.166 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: 2021-02-10T04:30:30.166 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: cluster 2021-02-10T04:30:30.166 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: 2021-02-10T04:30:30.166 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: 2021-02-10T04:30:29 2021-02-10T04:30:30.166 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: .312061+0000 osd.4 (osd.4) 64 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:30.166 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: cluster 2021-02-10T04:30:30.166 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:30:30.166 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[35961]: 04:30:30.019455+0000 mgr.y (mgr.14140) 405 : cluster [DBG] pgmap v469: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 5 active+clean+snaptrim, 161 active+clean; 151 MiB data, 484 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 746 KiB/s wr, 6 op/s 2021-02-10T04:30:30.167 INFO:tasks.rados.rados.0.smithi085.stdout:620: done (7 left) 2021-02-10T04:30:30.167 INFO:tasks.rados.rados.0.smithi085.stdout:621: snap_create 2021-02-10T04:30:30.309 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:30:30.310 INFO:teuthology.orchestra.run.smithi164.stdout: "size": 20, 2021-02-10T04:30:30.310 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 600, 2021-02-10T04:30:30.310 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [ 2021-02-10T04:30:30.310 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.310 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.34286.0:2 2.17 2:eacd3ea6:::realms_names.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected e49)", 2021-02-10T04:30:30.310 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:25:34.328322+0000", 2021-02-10T04:30:30.310 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 295.97835047900003, 2021-02-10T04:30:30.310 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0008720120000001, 2021-02-10T04:30:30.310 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:30.310 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T04:30:30.311 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:30:30.311 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.34286", 2021-02-10T04:30:30.311 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.164:0/3791211194", 2021-02-10T04:30:30.311 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 2 2021-02-10T04:30:30.311 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.311 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:30.311 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.311 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:30.311 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:34.328322+0000", 2021-02-10T04:30:30.311 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:30.311 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.311 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.312 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:30.312 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:34.328322+0000", 2021-02-10T04:30:30.312 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.0524000000000001e-05 2021-02-10T04:30:30.312 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.312 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.312 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:30.312 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:34.328352+0000", 2021-02-10T04:30:30.312 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.9410000000000007e-06 2021-02-10T04:30:30.312 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.312 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.312 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:30.312 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:34.328360+0000", 2021-02-10T04:30:30.312 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3811e-05 2021-02-10T04:30:30.312 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.313 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.313 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:30.313 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:34.328374+0000", 2021-02-10T04:30:30.313 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.4370000000000007e-06 2021-02-10T04:30:30.313 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.313 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.313 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:30.313 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:34.328382+0000", 2021-02-10T04:30:30.313 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.2986999999999999e-05 2021-02-10T04:30:30.313 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.313 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.313 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.313 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:34.328435+0000", 2021-02-10T04:30:30.314 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.7520000000000002e-06 2021-02-10T04:30:30.314 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.314 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.314 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for peered", 2021-02-10T04:30:30.314 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:34.328444+0000", 2021-02-10T04:30:30.314 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0006279760000001 2021-02-10T04:30:30.314 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.314 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.314 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.314 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:35.329072+0000", 2021-02-10T04:30:30.314 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00012158399999999999 2021-02-10T04:30:30.314 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.315 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.315 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:30.315 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:35.329194+0000", 2021-02-10T04:30:30.315 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0008720120000001 2021-02-10T04:30:30.315 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.315 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:30.315 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.315 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.315 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.315 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:292 7.d 7:b4cb5e6c:::smithi08575397-50:head [copy-from ver 3] snapc 8=[8] ondisk+write+known_if_redirected e105)", 2021-02-10T04:30:30.315 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:28:41.471573+0000", 2021-02-10T04:30:30.316 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 108.835099738, 2021-02-10T04:30:30.316 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.032617538000000001, 2021-02-10T04:30:30.316 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:30.316 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:30.316 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:30:30.316 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:30:30.316 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:30:30.316 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 292 2021-02-10T04:30:30.316 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.316 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:30.316 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.316 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:30.317 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.471573+0000", 2021-02-10T04:30:30.317 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:30.317 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.317 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.317 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:30.317 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.471573+0000", 2021-02-10T04:30:30.317 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.4827000000000006e-05 2021-02-10T04:30:30.317 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.317 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.317 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:30.317 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.471657+0000", 2021-02-10T04:30:30.317 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.4469999999999995e-06 2021-02-10T04:30:30.317 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.318 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.318 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:30.318 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.471667+0000", 2021-02-10T04:30:30.318 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7280999999999999e-05 2021-02-10T04:30:30.318 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.318 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.318 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:30.318 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.471684+0000", 2021-02-10T04:30:30.318 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.4889999999999995e-06 2021-02-10T04:30:30.318 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.318 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.318 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:30.318 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.471693+0000", 2021-02-10T04:30:30.319 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0015582949999999999 2021-02-10T04:30:30.319 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.319 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.319 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.319 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.473251+0000", 2021-02-10T04:30:30.319 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.1131999999999997e-05 2021-02-10T04:30:30.319 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.319 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.319 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:30.319 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.473312+0000", 2021-02-10T04:30:30.319 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0033112100000000002 2021-02-10T04:30:30.319 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.319 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.319 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 0,1", 2021-02-10T04:30:30.320 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.476623+0000", 2021-02-10T04:30:30.320 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0033264750000000002 2021-02-10T04:30:30.320 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.320 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.320 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:30:30.320 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.479950+0000", 2021-02-10T04:30:30.320 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011934696999999999 2021-02-10T04:30:30.321 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.321 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.321 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:30.321 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.491884+0000", 2021-02-10T04:30:30.321 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012156666999999999 2021-02-10T04:30:30.321 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.321 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.321 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:30.321 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.504041+0000", 2021-02-10T04:30:30.321 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.8189999999999994e-05 2021-02-10T04:30:30.322 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.322 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.322 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:30.322 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.504129+0000", 2021-02-10T04:30:30.322 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0828000000000001e-05 2021-02-10T04:30:30.322 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.322 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.322 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:30.322 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.504190+0000", 2021-02-10T04:30:30.322 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.032617538000000001 2021-02-10T04:30:30.322 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.322 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:30.322 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.323 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.323 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.323 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:293 7.d 7:b4cb5e6c:::smithi08575397-50:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e105)", 2021-02-10T04:30:30.323 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:28:41.471768+0000", 2021-02-10T04:30:30.323 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 108.834904533, 2021-02-10T04:30:30.323 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.032717686000000003, 2021-02-10T04:30:30.323 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:30.323 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:30:30.323 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:30:30.323 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:30:30.323 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:30:30.323 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 293 2021-02-10T04:30:30.323 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.324 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:30.324 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.324 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:30.324 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.471768+0000", 2021-02-10T04:30:30.324 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:30.324 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.324 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.324 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:30.324 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.471768+0000", 2021-02-10T04:30:30.324 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0587000000000001e-05 2021-02-10T04:30:30.324 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.324 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.324 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:30.324 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.471828+0000", 2021-02-10T04:30:30.325 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.3500000000000003e-06 2021-02-10T04:30:30.325 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.325 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.325 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:30.325 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.471838+0000", 2021-02-10T04:30:30.325 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0060000000000001e-05 2021-02-10T04:30:30.325 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.325 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.325 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:30.325 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.471858+0000", 2021-02-10T04:30:30.325 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.8189999999999998e-06 2021-02-10T04:30:30.325 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.325 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.326 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:30.326 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.471867+0000", 2021-02-10T04:30:30.326 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001611227 2021-02-10T04:30:30.326 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.326 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.326 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.326 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.473478+0000", 2021-02-10T04:30:30.326 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.5991999999999997e-05 2021-02-10T04:30:30.326 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.326 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.326 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:30:30.326 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.473514+0000", 2021-02-10T04:30:30.327 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0058399669999999997 2021-02-10T04:30:30.327 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.327 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.327 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.327 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.479354+0000", 2021-02-10T04:30:30.327 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.8769999999999994e-05 2021-02-10T04:30:30.327 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.327 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.327 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2021-02-10T04:30:30.327 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.479423+0000", 2021-02-10T04:30:30.327 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.024814082000000001 2021-02-10T04:30:30.327 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.327 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.328 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.328 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.504237+0000", 2021-02-10T04:30:30.328 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.8386000000000003e-05 2021-02-10T04:30:30.328 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.328 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.328 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:30.328 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.504335+0000", 2021-02-10T04:30:30.328 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00015044599999999999 2021-02-10T04:30:30.328 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.328 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.328 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:30.328 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.504485+0000", 2021-02-10T04:30:30.328 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.032717686000000003 2021-02-10T04:30:30.328 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.329 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:30.329 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.329 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.329 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.329 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:319 7.7 7:e3fec724:::smithi08575397-41:head [copy-from ver 18] snapc 8=[8] ondisk+write+known_if_redirected e105)", 2021-02-10T04:30:30.329 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:28:41.518302+0000", 2021-02-10T04:30:30.329 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 108.788369887, 2021-02-10T04:30:30.329 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.044002751999999999, 2021-02-10T04:30:30.329 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:30.329 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:30.329 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:30:30.329 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:30:30.329 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:30:30.329 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 319 2021-02-10T04:30:30.330 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.330 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:30.330 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.330 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:30.330 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.518302+0000", 2021-02-10T04:30:30.330 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:30.330 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.330 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.330 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:30.330 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.518302+0000", 2021-02-10T04:30:30.330 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5875999999999999e-05 2021-02-10T04:30:30.330 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.330 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.331 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:30.331 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.518348+0000", 2021-02-10T04:30:30.331 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7449999999999995e-06 2021-02-10T04:30:30.331 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.331 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.331 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:30.331 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.518356+0000", 2021-02-10T04:30:30.331 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4385e-05 2021-02-10T04:30:30.331 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.331 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.331 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:30.331 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.518370+0000", 2021-02-10T04:30:30.331 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.1500000000000005e-06 2021-02-10T04:30:30.331 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.332 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.332 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:30.332 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.518380+0000", 2021-02-10T04:30:30.332 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2805000000000001e-05 2021-02-10T04:30:30.332 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.332 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.332 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.332 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.518422+0000", 2021-02-10T04:30:30.332 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000101415 2021-02-10T04:30:30.332 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.332 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.332 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:30.332 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.518524+0000", 2021-02-10T04:30:30.332 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.028714389999999999 2021-02-10T04:30:30.333 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.333 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.333 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 0,3", 2021-02-10T04:30:30.333 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.547238+0000", 2021-02-10T04:30:30.333 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0037272690000000001 2021-02-10T04:30:30.333 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.333 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.333 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:30:30.333 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.550965+0000", 2021-02-10T04:30:30.333 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011100736999999999 2021-02-10T04:30:30.333 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.333 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.333 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:30.334 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.562066+0000", 2021-02-10T04:30:30.334 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000116366 2021-02-10T04:30:30.334 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.334 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.334 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:30.334 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.562183+0000", 2021-02-10T04:30:30.334 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.9454999999999999e-05 2021-02-10T04:30:30.334 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.334 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.334 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:30.334 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.562242+0000", 2021-02-10T04:30:30.334 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.3158999999999996e-05 2021-02-10T04:30:30.334 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.335 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.335 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:30.335 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.562305+0000", 2021-02-10T04:30:30.335 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.044002751999999999 2021-02-10T04:30:30.335 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.335 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:30.335 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.335 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.335 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.335 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:320 7.7 7:e3fec724:::smithi08575397-41:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e105)", 2021-02-10T04:30:30.335 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:28:41.518438+0000", 2021-02-10T04:30:30.335 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 108.788234751, 2021-02-10T04:30:30.335 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.044109844000000002, 2021-02-10T04:30:30.335 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:30.336 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:30:30.336 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:30:30.336 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:30:30.336 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:30:30.336 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 320 2021-02-10T04:30:30.336 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.336 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:30.336 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.336 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:30.336 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.518438+0000", 2021-02-10T04:30:30.336 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:30.336 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.336 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.337 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:30.337 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.518438+0000", 2021-02-10T04:30:30.337 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2179999999999999e-05 2021-02-10T04:30:30.337 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.337 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.337 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:30.337 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.518480+0000", 2021-02-10T04:30:30.337 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7810000000000005e-06 2021-02-10T04:30:30.337 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.337 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.337 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:30.337 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.518487+0000", 2021-02-10T04:30:30.337 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4786e-05 2021-02-10T04:30:30.337 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.338 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.338 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:30.338 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.518502+0000", 2021-02-10T04:30:30.338 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0630000000000002e-06 2021-02-10T04:30:30.338 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.338 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.338 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:30.338 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.518508+0000", 2021-02-10T04:30:30.338 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00016304500000000001 2021-02-10T04:30:30.338 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.338 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.338 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.338 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.518671+0000", 2021-02-10T04:30:30.338 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.2611000000000002e-05 2021-02-10T04:30:30.339 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.339 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.339 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:30:30.339 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.518724+0000", 2021-02-10T04:30:30.339 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.031391488000000002 2021-02-10T04:30:30.339 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.339 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.339 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.339 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.550115+0000", 2021-02-10T04:30:30.339 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.4478e-05 2021-02-10T04:30:30.339 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.339 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.339 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2021-02-10T04:30:30.340 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.550200+0000", 2021-02-10T04:30:30.340 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012150787 2021-02-10T04:30:30.340 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.340 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.340 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.340 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.562351+0000", 2021-02-10T04:30:30.340 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.8138000000000002e-05 2021-02-10T04:30:30.340 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.340 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.340 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:30.340 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.562439+0000", 2021-02-10T04:30:30.340 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00010848700000000001 2021-02-10T04:30:30.340 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.341 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.341 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:30.341 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.562547+0000", 2021-02-10T04:30:30.341 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.044109844000000002 2021-02-10T04:30:30.341 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.341 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:30.341 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.341 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.341 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.341 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:322 7.7 7:ef952473:::smithi08575397-16:head [write 658841~471511] snapc 8=[] ondisk+write+known_if_redirected e106)", 2021-02-10T04:30:30.341 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:28:42.236146+0000", 2021-02-10T04:30:30.341 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 108.070526464, 2021-02-10T04:30:30.341 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.144460106, 2021-02-10T04:30:30.342 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:30.342 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:30.342 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:30:30.342 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:30:30.342 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:30:30.342 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 322 2021-02-10T04:30:30.342 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.342 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:30.342 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.342 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:30.342 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.236146+0000", 2021-02-10T04:30:30.342 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:30.343 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.343 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.343 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:30.343 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.236146+0000", 2021-02-10T04:30:30.343 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000365029 2021-02-10T04:30:30.343 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.343 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.343 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:30.343 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.236511+0000", 2021-02-10T04:30:30.343 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0560000000000001e-05 2021-02-10T04:30:30.343 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.343 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.343 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:30.343 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.236521+0000", 2021-02-10T04:30:30.344 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2351000000000001e-05 2021-02-10T04:30:30.344 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.344 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.344 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:30.344 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.236534+0000", 2021-02-10T04:30:30.344 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6499999999999999e-06 2021-02-10T04:30:30.344 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.344 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.344 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:30.344 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.236540+0000", 2021-02-10T04:30:30.344 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.0032999999999998e-05 2021-02-10T04:30:30.344 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.344 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.344 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.345 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.236610+0000", 2021-02-10T04:30:30.345 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7854000000000004e-05 2021-02-10T04:30:30.345 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.345 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.345 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:30.345 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.236688+0000", 2021-02-10T04:30:30.345 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00012535899999999999 2021-02-10T04:30:30.345 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.345 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.345 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 0,3", 2021-02-10T04:30:30.345 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.236814+0000", 2021-02-10T04:30:30.345 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000946674 2021-02-10T04:30:30.345 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.346 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.346 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:30:30.346 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.237760+0000", 2021-02-10T04:30:30.346 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.041177099000000002 2021-02-10T04:30:30.346 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.346 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.346 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:30.346 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.278937+0000", 2021-02-10T04:30:30.346 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.10158276500000001 2021-02-10T04:30:30.346 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.346 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.346 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:30.346 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.380520+0000", 2021-02-10T04:30:30.346 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.2160000000000002e-05 2021-02-10T04:30:30.347 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.347 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.347 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:30.347 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.380572+0000", 2021-02-10T04:30:30.347 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.3572000000000003e-05 2021-02-10T04:30:30.347 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.347 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.347 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:30.347 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.380606+0000", 2021-02-10T04:30:30.347 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.144460106 2021-02-10T04:30:30.347 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.347 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:30.347 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.348 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.348 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.348 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:323 7.7 7:ef952473:::smithi08575397-16:head [write 1553542~708338] snapc 8=[] ondisk+write+known_if_redirected e106)", 2021-02-10T04:30:30.348 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:28:42.242224+0000", 2021-02-10T04:30:30.348 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 108.064448531, 2021-02-10T04:30:30.348 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.13851792700000001, 2021-02-10T04:30:30.348 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:30.348 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:30.348 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:30:30.348 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:30:30.348 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:30:30.348 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 323 2021-02-10T04:30:30.348 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.348 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:30.348 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.349 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:30.349 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.242224+0000", 2021-02-10T04:30:30.349 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:30.349 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.349 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.349 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:30.349 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.242224+0000", 2021-02-10T04:30:30.349 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00078808399999999999 2021-02-10T04:30:30.349 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.349 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.349 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:30.349 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.243012+0000", 2021-02-10T04:30:30.349 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.5869999999999998e-06 2021-02-10T04:30:30.350 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.350 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.350 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:30.350 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.243018+0000", 2021-02-10T04:30:30.350 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1462e-05 2021-02-10T04:30:30.350 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.350 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.350 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:30.350 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.243030+0000", 2021-02-10T04:30:30.350 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.5239999999999997e-06 2021-02-10T04:30:30.350 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.350 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.350 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:30.350 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.243036+0000", 2021-02-10T04:30:30.351 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.4359e-05 2021-02-10T04:30:30.351 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.351 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.351 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.351 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.243081+0000", 2021-02-10T04:30:30.351 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.8186000000000001e-05 2021-02-10T04:30:30.351 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.351 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.351 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:30.351 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.243149+0000", 2021-02-10T04:30:30.351 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000119175 2021-02-10T04:30:30.351 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.351 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.351 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 0,3", 2021-02-10T04:30:30.352 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.243268+0000", 2021-02-10T04:30:30.352 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001052575 2021-02-10T04:30:30.352 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.352 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.352 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:30:30.352 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.244321+0000", 2021-02-10T04:30:30.352 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.034795212999999998 2021-02-10T04:30:30.352 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.352 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.352 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:30.352 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.279116+0000", 2021-02-10T04:30:30.352 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.101537506 2021-02-10T04:30:30.352 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.353 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.353 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:30.353 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.380653+0000", 2021-02-10T04:30:30.353 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5629e-05 2021-02-10T04:30:30.353 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.353 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.353 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:30.353 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.380699+0000", 2021-02-10T04:30:30.353 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2626999999999997e-05 2021-02-10T04:30:30.353 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.353 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.353 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:30.353 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.380742+0000", 2021-02-10T04:30:30.354 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.13851792700000001 2021-02-10T04:30:30.354 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.354 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:30.354 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.354 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.354 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.354 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:324 7.7 7:ef952473:::smithi08575397-16:head [write 2964119~94722] snapc 8=[] ondisk+write+known_if_redirected e106)", 2021-02-10T04:30:30.354 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:28:42.246448+0000", 2021-02-10T04:30:30.354 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 108.060224455, 2021-02-10T04:30:30.354 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.13598811099999999, 2021-02-10T04:30:30.354 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:30.354 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:30.355 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:30:30.355 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:30:30.355 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:30:30.355 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 324 2021-02-10T04:30:30.355 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.355 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:30.355 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.355 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:30.355 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246448+0000", 2021-02-10T04:30:30.355 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:30.355 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.355 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.355 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:30.355 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246448+0000", 2021-02-10T04:30:30.356 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000102948 2021-02-10T04:30:30.356 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.356 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.356 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:30.356 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246551+0000", 2021-02-10T04:30:30.356 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.8510000000000005e-06 2021-02-10T04:30:30.356 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.356 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.356 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:30.356 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246560+0000", 2021-02-10T04:30:30.356 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6589e-05 2021-02-10T04:30:30.356 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.356 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.356 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:30.357 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246576+0000", 2021-02-10T04:30:30.357 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.371e-06 2021-02-10T04:30:30.357 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.357 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.357 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:30.357 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246585+0000", 2021-02-10T04:30:30.357 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.7770999999999999e-05 2021-02-10T04:30:30.357 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.357 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.357 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.357 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246652+0000", 2021-02-10T04:30:30.357 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000126255 2021-02-10T04:30:30.358 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.358 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.358 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:30.358 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246779+0000", 2021-02-10T04:30:30.358 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000210301 2021-02-10T04:30:30.358 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.358 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.358 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 0,3", 2021-02-10T04:30:30.358 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246989+0000", 2021-02-10T04:30:30.358 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00080194299999999995 2021-02-10T04:30:30.358 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.358 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.358 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:30:30.359 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.247791+0000", 2021-02-10T04:30:30.359 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.031425937000000001 2021-02-10T04:30:30.359 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.359 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.359 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:30.359 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.279217+0000", 2021-02-10T04:30:30.359 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.10310697100000001 2021-02-10T04:30:30.359 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.359 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.359 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:30.359 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.382324+0000", 2021-02-10T04:30:30.359 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.3995e-05 2021-02-10T04:30:30.359 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.359 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.360 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:30.360 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.382388+0000", 2021-02-10T04:30:30.360 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.8179000000000001e-05 2021-02-10T04:30:30.360 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.360 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.360 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:30.360 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.382436+0000", 2021-02-10T04:30:30.360 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.13598811099999999 2021-02-10T04:30:30.360 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.360 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:30.360 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.360 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.360 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.361 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:325 7.7 7:ef952473:::smithi08575397-16:head [setxattr _header (56) in=7b,truncate 3058841] snapc 8=[] ondisk+write+known_if_redirected e106)", 2021-02-10T04:30:30.361 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:28:42.246688+0000", 2021-02-10T04:30:30.361 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 108.059984314, 2021-02-10T04:30:30.361 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.13707872300000001, 2021-02-10T04:30:30.361 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:30.361 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:30.361 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:30:30.361 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:30:30.361 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:30:30.361 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 325 2021-02-10T04:30:30.361 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.361 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:30.361 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.362 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:30.362 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246688+0000", 2021-02-10T04:30:30.362 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:30.362 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.362 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.362 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:30.362 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246688+0000", 2021-02-10T04:30:30.362 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.2065999999999997e-05 2021-02-10T04:30:30.362 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.362 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.362 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:30.362 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246770+0000", 2021-02-10T04:30:30.362 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3417e-05 2021-02-10T04:30:30.363 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.363 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.363 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:30.363 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246783+0000", 2021-02-10T04:30:30.363 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8343e-05 2021-02-10T04:30:30.363 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.363 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.363 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:30.363 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246802+0000", 2021-02-10T04:30:30.363 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0129999999999999e-06 2021-02-10T04:30:30.363 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.363 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.363 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:30.363 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246808+0000", 2021-02-10T04:30:30.364 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00054755600000000004 2021-02-10T04:30:30.364 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.364 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.364 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.364 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.247355+0000", 2021-02-10T04:30:30.364 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.2678999999999999e-05 2021-02-10T04:30:30.364 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.364 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.364 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:30.364 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.247438+0000", 2021-02-10T04:30:30.364 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00011895700000000001 2021-02-10T04:30:30.364 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.364 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.365 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 0,3", 2021-02-10T04:30:30.365 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.247557+0000", 2021-02-10T04:30:30.365 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00035848400000000001 2021-02-10T04:30:30.365 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.365 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.365 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:30:30.365 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.247915+0000", 2021-02-10T04:30:30.365 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.031346306999999997 2021-02-10T04:30:30.365 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.365 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.365 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:30.365 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.279262+0000", 2021-02-10T04:30:30.365 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.104387568 2021-02-10T04:30:30.365 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.366 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.366 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:30.366 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.383649+0000", 2021-02-10T04:30:30.366 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.0586e-05 2021-02-10T04:30:30.366 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.366 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.366 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:30.366 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.383700+0000", 2021-02-10T04:30:30.366 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6747000000000003e-05 2021-02-10T04:30:30.366 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.366 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.366 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:30.366 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.383767+0000", 2021-02-10T04:30:30.366 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.13707872300000001 2021-02-10T04:30:30.367 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.367 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:30.367 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.367 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.367 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.367 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:326 7.7 7:ef952473:::smithi08575397-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e106)", 2021-02-10T04:30:30.367 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:28:42.246885+0000", 2021-02-10T04:30:30.367 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 108.059787183, 2021-02-10T04:30:30.367 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.13707413099999999, 2021-02-10T04:30:30.367 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:30.367 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:30:30.367 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:30:30.367 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:30:30.367 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:30:30.368 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 326 2021-02-10T04:30:30.368 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.368 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:30.368 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.368 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:30.368 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246885+0000", 2021-02-10T04:30:30.368 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:30.368 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.368 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.368 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:30.368 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246885+0000", 2021-02-10T04:30:30.368 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.5454000000000001e-05 2021-02-10T04:30:30.368 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.369 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.369 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:30.369 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246941+0000", 2021-02-10T04:30:30.369 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.7479999999999999e-06 2021-02-10T04:30:30.369 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.369 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.369 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:30.369 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246946+0000", 2021-02-10T04:30:30.369 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7218999999999999e-05 2021-02-10T04:30:30.369 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.369 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.369 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:30.369 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246963+0000", 2021-02-10T04:30:30.369 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.1819999999999997e-06 2021-02-10T04:30:30.370 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.370 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.370 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:30.370 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.246969+0000", 2021-02-10T04:30:30.370 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00085141700000000004 2021-02-10T04:30:30.370 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.370 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.370 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.370 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.247820+0000", 2021-02-10T04:30:30.370 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.8720999999999995e-05 2021-02-10T04:30:30.370 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.370 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.371 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2021-02-10T04:30:30.371 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.247889+0000", 2021-02-10T04:30:30.371 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.13591054 2021-02-10T04:30:30.371 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.371 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.371 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.371 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.383799+0000", 2021-02-10T04:30:30.371 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.2977000000000003e-05 2021-02-10T04:30:30.371 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.371 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.371 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:30.371 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.383872+0000", 2021-02-10T04:30:30.371 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.6873000000000007e-05 2021-02-10T04:30:30.371 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.372 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.372 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:30.372 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:42.383959+0000", 2021-02-10T04:30:30.372 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.13707413099999999 2021-02-10T04:30:30.372 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.372 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:30.372 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.372 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.372 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.372 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:351 7.7 7:e3fec724:::smithi08575397-41:head [copy-from ver 10] snapc a=[] ondisk+write+known_if_redirected e108)", 2021-02-10T04:30:30.372 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:28:44.035238+0000", 2021-02-10T04:30:30.372 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 106.271434391, 2021-02-10T04:30:30.372 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.031627293000000001, 2021-02-10T04:30:30.372 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:30.373 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:30.373 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:30:30.373 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:30:30.373 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:30:30.373 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 351 2021-02-10T04:30:30.373 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.373 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:30.373 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.373 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:30.373 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.035238+0000", 2021-02-10T04:30:30.373 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:30.373 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.373 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.373 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:30.374 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.035238+0000", 2021-02-10T04:30:30.374 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.3204000000000001e-05 2021-02-10T04:30:30.374 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.374 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.374 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:30.374 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.035291+0000", 2021-02-10T04:30:30.374 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.3000000000000002e-06 2021-02-10T04:30:30.374 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.374 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.374 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:30.374 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.035299+0000", 2021-02-10T04:30:30.374 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.414e-05 2021-02-10T04:30:30.374 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.375 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.375 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:30.375 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.035314+0000", 2021-02-10T04:30:30.375 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.5800000000000003e-06 2021-02-10T04:30:30.375 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.375 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.375 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:30.375 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.035321+0000", 2021-02-10T04:30:30.375 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.0084e-05 2021-02-10T04:30:30.375 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.375 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.375 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.375 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.035411+0000", 2021-02-10T04:30:30.375 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6302000000000001e-05 2021-02-10T04:30:30.376 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.376 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.376 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2021-02-10T04:30:30.376 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.035427+0000", 2021-02-10T04:30:30.376 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0032477740000000001 2021-02-10T04:30:30.376 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.376 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.376 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.376 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.038675+0000", 2021-02-10T04:30:30.376 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.1114999999999998e-05 2021-02-10T04:30:30.376 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.376 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.376 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:30.376 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.038766+0000", 2021-02-10T04:30:30.377 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.013720959 2021-02-10T04:30:30.377 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.377 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.377 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 0,3", 2021-02-10T04:30:30.377 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.052487+0000", 2021-02-10T04:30:30.377 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010917282 2021-02-10T04:30:30.377 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.377 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.377 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:30.377 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.063405+0000", 2021-02-10T04:30:30.377 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.7979999999999995e-06 2021-02-10T04:30:30.377 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.377 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.378 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:30:30.378 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.063413+0000", 2021-02-10T04:30:30.378 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0033396989999999998 2021-02-10T04:30:30.378 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.378 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.378 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:30.378 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.066753+0000", 2021-02-10T04:30:30.378 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.5998000000000002e-05 2021-02-10T04:30:30.378 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.378 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.378 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:30.378 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.066809+0000", 2021-02-10T04:30:30.378 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6057999999999998e-05 2021-02-10T04:30:30.379 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.379 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.379 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:30.379 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.066865+0000", 2021-02-10T04:30:30.379 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.031627293000000001 2021-02-10T04:30:30.379 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.379 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:30.379 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.379 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.379 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.379 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:352 7.7 7:e3fec724:::smithi08575397-41:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e108)", 2021-02-10T04:30:30.379 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:28:44.035388+0000", 2021-02-10T04:30:30.379 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 106.271283822, 2021-02-10T04:30:30.379 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.031710294, 2021-02-10T04:30:30.380 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:30.380 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:30:30.380 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:30:30.380 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:30:30.380 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:30:30.380 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 352 2021-02-10T04:30:30.380 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.380 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:30.380 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.380 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:30.380 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.035388+0000", 2021-02-10T04:30:30.380 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:30.380 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.380 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.381 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:30.381 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.035388+0000", 2021-02-10T04:30:30.381 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.1087999999999995e-05 2021-02-10T04:30:30.381 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.381 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.381 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:30.381 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.035450+0000", 2021-02-10T04:30:30.381 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.9609999999999994e-06 2021-02-10T04:30:30.381 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.381 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.381 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:30.381 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.035458+0000", 2021-02-10T04:30:30.381 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3903000000000001e-05 2021-02-10T04:30:30.382 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.382 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.382 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:30.382 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.035472+0000", 2021-02-10T04:30:30.382 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.5349999999999999e-06 2021-02-10T04:30:30.382 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.382 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.382 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:30.382 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.035480+0000", 2021-02-10T04:30:30.382 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.0043000000000006e-05 2021-02-10T04:30:30.382 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.382 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.382 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.383 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.035560+0000", 2021-02-10T04:30:30.383 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.6979999999999993e-06 2021-02-10T04:30:30.383 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.383 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.383 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting_for_map not empty", 2021-02-10T04:30:30.383 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.035570+0000", 2021-02-10T04:30:30.383 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0033434390000000001 2021-02-10T04:30:30.383 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.383 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.383 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.383 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.038913+0000", 2021-02-10T04:30:30.383 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2435000000000003e-05 2021-02-10T04:30:30.383 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.384 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.384 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:30:30.384 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.038956+0000", 2021-02-10T04:30:30.384 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.016729528 2021-02-10T04:30:30.384 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.384 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.384 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.384 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.055685+0000", 2021-02-10T04:30:30.384 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.8706e-05 2021-02-10T04:30:30.384 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.384 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.384 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2021-02-10T04:30:30.385 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.055784+0000", 2021-02-10T04:30:30.385 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011130578 2021-02-10T04:30:30.385 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.385 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.385 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.385 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.066914+0000", 2021-02-10T04:30:30.385 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.8273000000000001e-05 2021-02-10T04:30:30.385 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.385 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.385 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:30.385 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.066993+0000", 2021-02-10T04:30:30.385 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00010610700000000001 2021-02-10T04:30:30.385 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.386 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.386 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:30.386 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:44.067099+0000", 2021-02-10T04:30:30.386 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.031710294 2021-02-10T04:30:30.386 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.386 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:30.386 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.386 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.386 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.386 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:539 7.d 7:b1749140:::smithi08575397-19:head [copy-from ver 29] snapc 1e=[1e,1d,1c,1b,19,17,15] ondisk+write+known_if_redirected e127)", 2021-02-10T04:30:30.386 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:03.044727+0000", 2021-02-10T04:30:30.386 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 87.261945592000004, 2021-02-10T04:30:30.386 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.070068546999999995, 2021-02-10T04:30:30.387 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:30.387 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:30.387 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:30:30.387 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:30:30.387 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:30:30.387 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 539 2021-02-10T04:30:30.387 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.387 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:30.387 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.387 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:30.387 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.044727+0000", 2021-02-10T04:30:30.387 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:30.387 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.388 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.388 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:30.388 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.044727+0000", 2021-02-10T04:30:30.388 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.6217999999999997e-05 2021-02-10T04:30:30.388 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.388 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.388 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:30.388 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.044773+0000", 2021-02-10T04:30:30.388 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0399999999999998e-06 2021-02-10T04:30:30.388 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.388 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.388 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:30.388 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.044779+0000", 2021-02-10T04:30:30.389 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.944e-06 2021-02-10T04:30:30.389 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.389 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.389 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:30.389 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.044789+0000", 2021-02-10T04:30:30.389 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.824e-06 2021-02-10T04:30:30.389 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.389 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.389 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:30.389 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.044797+0000", 2021-02-10T04:30:30.389 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6033999999999998e-05 2021-02-10T04:30:30.389 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.389 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.389 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.390 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.044863+0000", 2021-02-10T04:30:30.390 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.4239999999999993e-06 2021-02-10T04:30:30.390 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.390 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.390 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2021-02-10T04:30:30.390 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.044871+0000", 2021-02-10T04:30:30.390 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0028711449999999999 2021-02-10T04:30:30.390 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.390 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.390 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.390 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.047742+0000", 2021-02-10T04:30:30.390 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000141211 2021-02-10T04:30:30.390 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.391 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.391 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:30.391 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.047884+0000", 2021-02-10T04:30:30.391 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0034499130000000002 2021-02-10T04:30:30.391 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.391 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.391 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 0,1", 2021-02-10T04:30:30.391 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.051333+0000", 2021-02-10T04:30:30.391 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0065834270000000002 2021-02-10T04:30:30.391 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.391 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.391 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:30:30.391 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.057917+0000", 2021-02-10T04:30:30.391 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0085505949999999994 2021-02-10T04:30:30.392 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.392 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.392 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:30.392 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.066467+0000", 2021-02-10T04:30:30.392 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.048180043999999998 2021-02-10T04:30:30.392 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.392 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.392 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:30.392 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.114647+0000", 2021-02-10T04:30:30.392 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.2442999999999998e-05 2021-02-10T04:30:30.392 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.392 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.392 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:30.393 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.114710+0000", 2021-02-10T04:30:30.393 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.5284999999999994e-05 2021-02-10T04:30:30.393 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.393 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.393 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:30.393 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.114795+0000", 2021-02-10T04:30:30.393 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.070068546999999995 2021-02-10T04:30:30.393 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.393 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:30.393 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.393 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.393 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.393 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:540 7.d 7:b1749140:::smithi08575397-19:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e127)", 2021-02-10T04:30:30.394 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:03.044909+0000", 2021-02-10T04:30:30.394 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 87.261763377999998, 2021-02-10T04:30:30.394 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.070174716999999998, 2021-02-10T04:30:30.394 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:30.394 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:30:30.394 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:30:30.394 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:30:30.394 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:30:30.394 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 540 2021-02-10T04:30:30.394 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.394 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:30.394 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.395 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:30.395 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.044909+0000", 2021-02-10T04:30:30.395 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:30.395 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.395 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.395 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:30.395 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.044909+0000", 2021-02-10T04:30:30.395 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.8542999999999999e-05 2021-02-10T04:30:30.395 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.395 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.395 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:30.395 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.044957+0000", 2021-02-10T04:30:30.395 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.3860000000000004e-06 2021-02-10T04:30:30.395 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.396 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.396 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:30.396 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.044964+0000", 2021-02-10T04:30:30.396 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3570000000000001e-05 2021-02-10T04:30:30.396 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.396 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.396 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:30.396 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.044977+0000", 2021-02-10T04:30:30.396 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.3890000000000002e-06 2021-02-10T04:30:30.396 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.396 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.396 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:30.396 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.044984+0000", 2021-02-10T04:30:30.396 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2202e-05 2021-02-10T04:30:30.397 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.397 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.397 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.397 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.045026+0000", 2021-02-10T04:30:30.397 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.2189999999999998e-06 2021-02-10T04:30:30.397 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.397 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.397 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting_for_map not empty", 2021-02-10T04:30:30.397 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.045033+0000", 2021-02-10T04:30:30.397 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0029873650000000001 2021-02-10T04:30:30.397 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.397 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.398 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.398 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.048021+0000", 2021-02-10T04:30:30.398 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0090999999999998e-05 2021-02-10T04:30:30.398 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.398 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.398 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:30:30.398 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.048081+0000", 2021-02-10T04:30:30.398 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0079513120000000003 2021-02-10T04:30:30.398 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.398 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.398 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.398 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.056032+0000", 2021-02-10T04:30:30.398 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.0277000000000005e-05 2021-02-10T04:30:30.399 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.399 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.399 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2021-02-10T04:30:30.399 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.056102+0000", 2021-02-10T04:30:30.399 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.058739560000000003 2021-02-10T04:30:30.399 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.399 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.399 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.399 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.114842+0000", 2021-02-10T04:30:30.399 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000103718 2021-02-10T04:30:30.399 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.399 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.399 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:30.400 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.114946+0000", 2021-02-10T04:30:30.400 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000138085 2021-02-10T04:30:30.400 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.400 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.400 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:30.400 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.115084+0000", 2021-02-10T04:30:30.400 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.070174716999999998 2021-02-10T04:30:30.400 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.400 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:30.400 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.400 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.400 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.400 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:630 7.7 7:ef952473:::smithi08575397-16:head [rollback 15] snapc 23=[23,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e132)", 2021-02-10T04:30:30.401 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:08.056922+0000", 2021-02-10T04:30:30.401 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 82.249750646999999, 2021-02-10T04:30:30.401 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.035817979, 2021-02-10T04:30:30.401 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:30.401 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:30.401 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:30:30.401 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:30:30.401 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:30:30.401 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 630 2021-02-10T04:30:30.401 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.401 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:30.401 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.401 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:30.402 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:08.056922+0000", 2021-02-10T04:30:30.402 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:30.402 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.402 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.402 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:30.402 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:08.056922+0000", 2021-02-10T04:30:30.402 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.7545000000000001e-05 2021-02-10T04:30:30.402 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.402 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.402 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:30.402 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:08.056959+0000", 2021-02-10T04:30:30.402 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.9559999999999992e-06 2021-02-10T04:30:30.402 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.403 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.403 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:30.403 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:08.056969+0000", 2021-02-10T04:30:30.403 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.551e-06 2021-02-10T04:30:30.403 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.403 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.403 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:30.403 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:08.056978+0000", 2021-02-10T04:30:30.403 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3737000000000001e-05 2021-02-10T04:30:30.403 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.403 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.403 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:30.403 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:08.056991+0000", 2021-02-10T04:30:30.404 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00085371500000000001 2021-02-10T04:30:30.404 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.404 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.404 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.404 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:08.057845+0000", 2021-02-10T04:30:30.404 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.2422999999999993e-05 2021-02-10T04:30:30.404 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.404 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.404 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:30.404 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:08.057928+0000", 2021-02-10T04:30:30.404 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000217328 2021-02-10T04:30:30.404 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.404 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.404 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 0,3", 2021-02-10T04:30:30.405 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:08.058145+0000", 2021-02-10T04:30:30.405 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0083778450000000001 2021-02-10T04:30:30.405 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.405 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.405 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:30.405 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:08.066523+0000", 2021-02-10T04:30:30.405 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0047915229999999998 2021-02-10T04:30:30.405 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.405 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.405 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:30:30.405 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:08.071314+0000", 2021-02-10T04:30:30.405 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.02130574 2021-02-10T04:30:30.405 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.414 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.414 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:30.414 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:08.092620+0000", 2021-02-10T04:30:30.414 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.5306000000000001e-05 2021-02-10T04:30:30.414 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.414 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.414 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:30.414 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:08.092685+0000", 2021-02-10T04:30:30.414 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.431e-05 2021-02-10T04:30:30.414 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.414 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.414 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:30.414 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:08.092740+0000", 2021-02-10T04:30:30.415 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.035817979 2021-02-10T04:30:30.415 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.415 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:30.415 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.415 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.415 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.415 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:631 7.7 7:ef952473:::smithi08575397-16:head [append 0~0] snapc 23=[23,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e132)", 2021-02-10T04:30:30.415 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:08.057050+0000", 2021-02-10T04:30:30.415 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 82.249621947999998, 2021-02-10T04:30:30.415 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.035869624000000003, 2021-02-10T04:30:30.415 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:30.415 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:30.415 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:30:30.415 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:30:30.416 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:30:30.416 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 631 2021-02-10T04:30:30.416 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.416 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:30.416 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.416 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:30.416 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:08.057050+0000", 2021-02-10T04:30:30.416 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:30.416 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.416 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.416 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:30.416 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:08.057050+0000", 2021-02-10T04:30:30.416 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1933000000000001e-05 2021-02-10T04:30:30.416 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.417 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.417 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:30.417 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:08.057082+0000", 2021-02-10T04:30:30.417 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2590000000000002e-06 2021-02-10T04:30:30.417 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.417 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.417 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:30.417 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:08.057087+0000", 2021-02-10T04:30:30.417 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7858000000000001e-05 2021-02-10T04:30:30.417 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.417 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.417 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:30.417 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:08.057104+0000", 2021-02-10T04:30:30.417 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.0979999999999996e-06 2021-02-10T04:30:30.418 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.418 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.418 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:30.418 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:08.057111+0000", 2021-02-10T04:30:30.418 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001426456 2021-02-10T04:30:30.418 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.418 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.418 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.418 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:08.058538+0000", 2021-02-10T04:30:30.418 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000107802 2021-02-10T04:30:30.418 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.418 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.418 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:30.419 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:08.058646+0000", 2021-02-10T04:30:30.419 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000144945 2021-02-10T04:30:30.419 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.419 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.419 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 0,3", 2021-02-10T04:30:30.419 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:08.058791+0000", 2021-02-10T04:30:30.419 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0078225380000000004 2021-02-10T04:30:30.419 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.419 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.419 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:30.419 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:08.066613+0000", 2021-02-10T04:30:30.419 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0047093769999999998 2021-02-10T04:30:30.419 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.420 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.420 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:30:30.420 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:08.071323+0000", 2021-02-10T04:30:30.420 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.021478721999999999 2021-02-10T04:30:30.420 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.420 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.420 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:30.420 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:08.092801+0000", 2021-02-10T04:30:30.420 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.4083000000000002e-05 2021-02-10T04:30:30.420 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.420 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.420 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:30.420 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:08.092875+0000", 2021-02-10T04:30:30.420 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.4552999999999998e-05 2021-02-10T04:30:30.421 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.421 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.421 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:30.421 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:08.092920+0000", 2021-02-10T04:30:30.421 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.035869624000000003 2021-02-10T04:30:30.421 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.421 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:30.421 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.421 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.421 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.421 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:1277 7.7 7:e498d07f:::smithi08575397-21:head [append 0~0] snapc 56=[56] ondisk+write+known_if_redirected e183)", 2021-02-10T04:30:30.421 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:59.562589+0000", 2021-02-10T04:30:30.421 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 30.744083331999999, 2021-02-10T04:30:30.421 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.172755138, 2021-02-10T04:30:30.422 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:30.422 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:30.422 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:30:30.422 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:30:30.422 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:30:30.422 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 1277 2021-02-10T04:30:30.422 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.422 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:30.422 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.422 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:30.422 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:59.562589+0000", 2021-02-10T04:30:30.422 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:30.422 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.423 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.423 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:30.423 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:59.562589+0000", 2021-02-10T04:30:30.423 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8561000000000001e-05 2021-02-10T04:30:30.423 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.423 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.423 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:30.423 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:59.562607+0000", 2021-02-10T04:30:30.423 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.7810000000000001e-06 2021-02-10T04:30:30.423 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.423 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.423 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:30.423 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:59.562610+0000", 2021-02-10T04:30:30.423 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.412e-06 2021-02-10T04:30:30.424 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.424 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.424 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:30.424 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:59.562616+0000", 2021-02-10T04:30:30.424 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.5390000000000002e-06 2021-02-10T04:30:30.424 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.424 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.424 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:30.424 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:59.562618+0000", 2021-02-10T04:30:30.424 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.9777000000000002e-05 2021-02-10T04:30:30.424 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.424 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.424 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.425 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:59.562658+0000", 2021-02-10T04:30:30.425 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1532999999999994e-05 2021-02-10T04:30:30.425 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.425 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.425 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:30.425 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:59.562730+0000", 2021-02-10T04:30:30.425 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000118999 2021-02-10T04:30:30.425 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.425 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.425 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 0,3", 2021-02-10T04:30:30.425 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:59.562849+0000", 2021-02-10T04:30:30.425 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00074997499999999995 2021-02-10T04:30:30.425 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.425 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.426 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:30:30.426 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:59.563599+0000", 2021-02-10T04:30:30.426 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7874999999999996e-05 2021-02-10T04:30:30.426 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.426 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.426 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:30.426 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:59.563676+0000", 2021-02-10T04:30:30.426 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.17152529799999999 2021-02-10T04:30:30.426 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.426 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.426 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:30.426 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:59.735202+0000", 2021-02-10T04:30:30.426 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.8019e-05 2021-02-10T04:30:30.426 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.427 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.427 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:30.427 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:59.735280+0000", 2021-02-10T04:30:30.427 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.4369000000000007e-05 2021-02-10T04:30:30.427 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.427 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.427 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:30.427 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:59.735344+0000", 2021-02-10T04:30:30.427 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.172755138 2021-02-10T04:30:30.427 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.427 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:30.427 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.427 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.428 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.428 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:1278 7.7 7:e498d07f:::smithi08575397-21:head [rollback 56] snapc 56=[56] ondisk+write+known_if_redirected e183)", 2021-02-10T04:30:30.428 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:59.562639+0000", 2021-02-10T04:30:30.428 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 30.744033486999999, 2021-02-10T04:30:30.428 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.17315222999999999, 2021-02-10T04:30:30.428 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:30.428 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:30.428 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:30:30.428 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:30:30.428 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:30:30.428 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 1278 2021-02-10T04:30:30.428 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.428 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:30.428 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.429 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:30.429 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:59.562639+0000", 2021-02-10T04:30:30.429 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:30.429 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.429 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.429 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:30.429 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:59.562639+0000", 2021-02-10T04:30:30.429 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5744999999999999e-05 2021-02-10T04:30:30.429 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.429 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.429 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:30.429 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:59.562655+0000", 2021-02-10T04:30:30.429 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.0299999999999998e-06 2021-02-10T04:30:30.429 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.430 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.430 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:30.430 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:59.562658+0000", 2021-02-10T04:30:30.430 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.0810000000000001e-06 2021-02-10T04:30:30.430 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.430 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.430 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:30.430 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:59.562663+0000", 2021-02-10T04:30:30.430 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4700000000000001e-06 2021-02-10T04:30:30.430 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.430 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.430 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:30.430 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:59.562665+0000", 2021-02-10T04:30:30.431 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00046103399999999998 2021-02-10T04:30:30.431 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.431 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.431 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.431 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:59.563126+0000", 2021-02-10T04:30:30.431 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6738999999999999e-05 2021-02-10T04:30:30.431 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.431 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.431 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:30.431 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:59.563193+0000", 2021-02-10T04:30:30.431 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000150657 2021-02-10T04:30:30.431 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.431 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.431 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 0,3", 2021-02-10T04:30:30.432 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:59.563344+0000", 2021-02-10T04:30:30.432 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000394683 2021-02-10T04:30:30.432 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.432 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.432 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:30:30.432 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:59.563738+0000", 2021-02-10T04:30:30.432 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0011501219999999999 2021-02-10T04:30:30.432 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.432 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.432 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:30.432 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:59.564888+0000", 2021-02-10T04:30:30.432 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.17078446799999999 2021-02-10T04:30:30.432 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.433 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.433 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:30.433 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:59.735673+0000", 2021-02-10T04:30:30.433 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.5995000000000007e-05 2021-02-10T04:30:30.433 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.433 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.433 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:30.433 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:59.735749+0000", 2021-02-10T04:30:30.433 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2206000000000002e-05 2021-02-10T04:30:30.433 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.433 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.433 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:30.433 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:59.735791+0000", 2021-02-10T04:30:30.433 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.17315222999999999 2021-02-10T04:30:30.434 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.434 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:30.434 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.434 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.434 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.434 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:1670 7.7 7:ef952473:::smithi08575397-16:head [copy-from ver 104] snapc 6f=[6f] ondisk+write+known_if_redirected e212)", 2021-02-10T04:30:30.434 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:30:28.445820+0000", 2021-02-10T04:30:30.434 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 1.860852714, 2021-02-10T04:30:30.434 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.203013114, 2021-02-10T04:30:30.434 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:30.434 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:30.434 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:30:30.434 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:30:30.435 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:30:30.435 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 1670 2021-02-10T04:30:30.435 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.435 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:30.435 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.435 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:30.435 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:28.445820+0000", 2021-02-10T04:30:30.435 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:30.435 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.435 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.435 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:30.435 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:28.445820+0000", 2021-02-10T04:30:30.435 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2370000000000001e-05 2021-02-10T04:30:30.436 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.436 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.436 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:30.436 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:28.445842+0000", 2021-02-10T04:30:30.436 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2780000000000004e-06 2021-02-10T04:30:30.436 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.436 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.436 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:30.436 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:28.445846+0000", 2021-02-10T04:30:30.436 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.4109999999999996e-06 2021-02-10T04:30:30.436 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.436 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.436 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:30.437 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:28.445855+0000", 2021-02-10T04:30:30.437 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.4010000000000002e-06 2021-02-10T04:30:30.437 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.437 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.437 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:30.437 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:28.445861+0000", 2021-02-10T04:30:30.437 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.1272000000000001e-05 2021-02-10T04:30:30.437 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.437 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.437 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.437 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:28.445902+0000", 2021-02-10T04:30:30.437 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.2515999999999994e-05 2021-02-10T04:30:30.437 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.438 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.438 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:30.438 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:28.445985+0000", 2021-02-10T04:30:30.438 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.186207255 2021-02-10T04:30:30.438 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.438 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.438 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 0,3", 2021-02-10T04:30:30.438 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:29.632192+0000", 2021-02-10T04:30:30.438 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0067697929999999996 2021-02-10T04:30:30.438 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.438 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.438 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:30:30.438 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:29.638962+0000", 2021-02-10T04:30:30.438 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0094740580000000005 2021-02-10T04:30:30.439 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.439 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.439 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:30.439 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:29.648436+0000", 2021-02-10T04:30:30.439 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00026857699999999999 2021-02-10T04:30:30.439 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.439 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.439 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:30.439 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:29.648704+0000", 2021-02-10T04:30:30.439 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.7704000000000001e-05 2021-02-10T04:30:30.439 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.439 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.439 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:30.440 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:29.648772+0000", 2021-02-10T04:30:30.440 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0479000000000002e-05 2021-02-10T04:30:30.440 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.440 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.440 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:30.440 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:29.648833+0000", 2021-02-10T04:30:30.440 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.203013114 2021-02-10T04:30:30.440 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.440 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:30.440 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.440 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.440 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.440 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:1671 7.7 7:ef952473:::smithi08575397-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e212)", 2021-02-10T04:30:30.440 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:30:28.445886+0000", 2021-02-10T04:30:30.441 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 1.860786222, 2021-02-10T04:30:30.441 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2031070230000001, 2021-02-10T04:30:30.441 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:30.441 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:30:30.441 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:30:30.441 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:30:30.441 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:30:30.441 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 1671 2021-02-10T04:30:30.441 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.441 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:30.441 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.441 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:30.441 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:28.445886+0000", 2021-02-10T04:30:30.441 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:30.442 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.442 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.442 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:30.442 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:28.445886+0000", 2021-02-10T04:30:30.442 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7921e-05 2021-02-10T04:30:30.442 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.442 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.442 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:30.442 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:28.445904+0000", 2021-02-10T04:30:30.442 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1200000000000002e-06 2021-02-10T04:30:30.442 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.442 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.442 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:30.442 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:28.445907+0000", 2021-02-10T04:30:30.443 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.5830000000000003e-06 2021-02-10T04:30:30.443 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.443 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.443 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:30.443 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:28.445914+0000", 2021-02-10T04:30:30.443 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.7949999999999999e-06 2021-02-10T04:30:30.443 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.443 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.443 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:30.443 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:28.445916+0000", 2021-02-10T04:30:30.443 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00024207100000000001 2021-02-10T04:30:30.443 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.443 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.444 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.444 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:28.446159+0000", 2021-02-10T04:30:30.444 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6351000000000003e-05 2021-02-10T04:30:30.444 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.444 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.444 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:30:30.444 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:28.446215+0000", 2021-02-10T04:30:30.444 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1901034340000001 2021-02-10T04:30:30.444 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.444 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.444 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.444 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:29.636318+0000", 2021-02-10T04:30:30.444 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1138999999999997e-05 2021-02-10T04:30:30.444 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.445 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.445 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2021-02-10T04:30:30.445 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:29.636399+0000", 2021-02-10T04:30:30.445 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012457947 2021-02-10T04:30:30.445 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.445 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.445 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:30.445 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:29.648857+0000", 2021-02-10T04:30:30.445 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6864000000000003e-05 2021-02-10T04:30:30.445 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.445 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.445 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:30.445 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:29.648914+0000", 2021-02-10T04:30:30.445 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.8798e-05 2021-02-10T04:30:30.446 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:30.446 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:30.446 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:30.446 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:29.648993+0000", 2021-02-10T04:30:30.446 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2031070230000001 2021-02-10T04:30:30.446 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.446 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:30.446 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.446 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:30.446 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:30.446 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:30:30.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[29319]: cluster 2021-02-10T04:30:28.871642+0000 osd.2 ( 2021-02-10T04:30:30.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[29319]: osd.2) 54 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:30:30.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[29319]: cluster 2021-02-10T04:30:28.871650+0000 osd.2 (osd.2) 55 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:30:30.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[29319]: cluster 2021-02-10T04:30:28.993928+0000 2021-02-10T04:30:30.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[29319]: osd.1 (osd.1) 86 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:30:30.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[29319]: 2021-02-10T04:30:30.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[29319]: cluster 2021-02-10T04:30:30.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[29319]: 2021-02-10T04:30:28.993937+0000 osd.1 ( 2021-02-10T04:30:30.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[29319]: osd.1) 87 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:30:30.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:30:30.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[29319]: 02-10T04:30:29.154437+0000 mon.a 2021-02-10T04:30:30.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[29319]: (mon.0) 922 : cluster [DBG] osdmap e213: 8 total, 8 up, 8 in 2021-02-10T04:30:30.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[29319]: 2021-02-10T04:30:30.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[29319]: cluster 2021-02-10T04:30:30.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[29319]: 2021-02-10T04:30:29.312009+0000 osd.4 ( 2021-02-10T04:30:30.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[29319]: osd.4) 61 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:30.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[29319]: cluster 2021-02-10T04:30:30.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[29319]: 2021-02-10T04:30:29.312020+0000 2021-02-10T04:30:30.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[29319]: osd.4 (osd.4) 62 : cluster [WRN] 2021-02-10T04:30:30.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:30.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[29319]: cluster 2021-02-10T04:30:29 2021-02-10T04:30:30.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[29319]: .312029+0000 osd.4 (osd.4) 63 : cluster 2021-02-10T04:30:30.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:30.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:30:30.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[29319]: 30:29.312061+0000 osd.4 (osd.4) 2021-02-10T04:30:30.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[29319]: 64 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:30.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:30:30.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[29319]: -02-10T04:30:30.019455+0000 2021-02-10T04:30:30.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[29319]: mgr.y (mgr.14140) 405 : cluster [DBG] 2021-02-10T04:30:30.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[29319]: pgmap v469: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 5 active+clean+snaptrim, 161 active+clean; 151 MiB data, 484 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 746 KiB/s wr, 6 op/s 2021-02-10T04:30:30.450 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: cluster 2021-02-10T04:30:30.450 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: 2021-02-10T04:30 2021-02-10T04:30:30.450 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: :28.871642+0000 osd.2 (osd.2) 54 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:30:30.450 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: cluster 2021-02-10T04:30:28.871650+0000 osd.2 (osd.2) 55 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:30:30.450 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: 2021-02-10T04:30:30.450 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:30:30.450 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: -02-10T04:30: 2021-02-10T04:30:30.450 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: 28.993928+0000 osd.1 ( 2021-02-10T04:30:30.450 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: osd.1) 86 : cluster 2021-02-10T04:30:30.450 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:30:30.450 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:30:30.451 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: -02-10T04:30: 2021-02-10T04:30:30.451 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: 28.993937+0000 osd.1 (osd 2021-02-10T04:30:30.451 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: .1) 87 : cluster [WRN] 2021-02-10T04:30:30.451 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:30:30.451 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: cluster 2021-02-10T04:30:30.451 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: 2021-02-10T04:30: 2021-02-10T04:30:30.451 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: 29.154437+0000 mon.a ( 2021-02-10T04:30:30.451 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: mon.0) 922 : cluster 2021-02-10T04:30:30.451 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: [DBG] osdmap e213: 8 total, 8 up, 8 in 2021-02-10T04:30:30.451 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: cluster 2021-02-10T04:30:30.451 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: 2021-02-10T04:30 2021-02-10T04:30:30.451 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: :29.312009+0000 osd.4 ( 2021-02-10T04:30:30.451 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: osd.4) 61 : cluster 2021-02-10T04:30:30.451 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:30.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: cluster 2021-02-10T04:30:30.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: 2021-02-10T04:30 2021-02-10T04:30:30.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: :29.312020+0000 osd.4 ( 2021-02-10T04:30:30.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: osd.4) 62 : cluster 2021-02-10T04:30:30.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:30.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:30:30.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: -02-10T04:30: 2021-02-10T04:30:30.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: 29.312029+0000 osd.4 ( 2021-02-10T04:30:30.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: osd.4) 63 : cluster 2021-02-10T04:30:30.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:30.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: cluster 2021-02-10T04:30:30.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: 2021-02-10T04:30: 2021-02-10T04:30:30.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: 29.312061+0000 osd.4 ( 2021-02-10T04:30:30.453 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: osd.4) 64 : cluster 2021-02-10T04:30:30.453 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:30.453 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: cluster 2021-02-10T04:30:30.453 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: 2021-02-10T04:30 2021-02-10T04:30:30.453 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: :30.019455+0000 mgr.y 2021-02-10T04:30:30.453 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: (mgr.14140) 405 : cluster 2021-02-10T04:30:30.453 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[29288]: [DBG] pgmap v469: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 5 active+clean+snaptrim, 161 active+clean; 151 MiB data, 484 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 746 KiB/s wr, 6 op/s 2021-02-10T04:30:30.453 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:30:30 smithi164 conmon[33677]: debug 2021-02-10T04:30:30.321+0000 7fbec19a1700 -1 osd.4 214 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:30:31.160 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[45443]: debug 2021-02-10T04:30:31.160 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[45443]: 2021-02-10T04:30:31.047+0000 7f7140d73700 -1 osd.1 214 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:30:31.161 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:30:30 smithi085 conmon[49106]: debug 2021-02-10T04:30:30.869+0000 7f77a3e94700 -1 osd.2 214 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:30:31.173 INFO:tasks.rados.rados.0.smithi085.stdout:621: done (7 left) 2021-02-10T04:30:31.173 INFO:tasks.rados.rados.0.smithi085.stdout:622: copy_from oid 40 from oid 22 current snap is 59 2021-02-10T04:30:31.173 INFO:tasks.rados.rados.0.smithi085.stdout:623: read oid 2 snap 58 2021-02-10T04:30:31.173 INFO:tasks.rados.rados.0.smithi085.stdout:623: expect (ObjNum 225 snap 51 seq_num 225) 2021-02-10T04:30:31.173 INFO:tasks.rados.rados.0.smithi085.stdout:624: copy_from oid 14 from oid 44 current snap is 59 2021-02-10T04:30:31.173 INFO:tasks.rados.rados.0.smithi085.stdout:625: rollback oid 13 current snap is 59 2021-02-10T04:30:31.174 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 13 to 57 2021-02-10T04:30:31.174 INFO:tasks.rados.rados.0.smithi085.stdout:626: copy_from oid 11 from oid 1 current snap is 59 2021-02-10T04:30:31.174 INFO:tasks.rados.rados.0.smithi085.stdout:627: read oid 5 snap -1 2021-02-10T04:30:31.174 INFO:tasks.rados.rados.0.smithi085.stdout:627: expect (ObjNum 243 snap 57 seq_num 243) 2021-02-10T04:30:31.174 INFO:tasks.rados.rados.0.smithi085.stdout:628: copy_from oid 15 from oid 34 current snap is 59 2021-02-10T04:30:31.174 INFO:tasks.rados.rados.0.smithi085.stdout:629: rollback oid 50 current snap is 59 2021-02-10T04:30:31.174 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 50 to 57 2021-02-10T04:30:31.174 INFO:tasks.rados.rados.0.smithi085.stdout:630: write oid 18 current snap is 59 2021-02-10T04:30:31.174 INFO:tasks.rados.rados.0.smithi085.stdout:630: seq_num 250 ranges {601750=535546,1929910=271840} 2021-02-10T04:30:31.180 INFO:tasks.rados.rados.0.smithi085.stdout:630: writing smithi08575397-18 from 601750 to 1137296 tid 1 2021-02-10T04:30:31.187 INFO:tasks.rados.rados.0.smithi085.stdout:630: writing smithi08575397-18 from 1929910 to 2201750 tid 2 2021-02-10T04:30:31.187 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:30:31.187 INFO:tasks.rados.rados.0.smithi085.stdout:627: expect (ObjNum 243 snap 57 seq_num 243) 2021-02-10T04:30:31.306 INFO:tasks.rados.rados.0.smithi085.stdout:625: finishing rollback tid 0 to smithi08575397-13 2021-02-10T04:30:31.382 INFO:tasks.rados.rados.0.smithi085.stdout:625: finishing rollback tid 1 to smithi08575397-13 2021-02-10T04:30:31.382 INFO:tasks.rados.rados.0.smithi085.stdout:625: finishing rollback tid 2 to smithi08575397-13 2021-02-10T04:30:31.382 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 204 (ObjNum 227 snap 51 seq_num 227) dirty exists 2021-02-10T04:30:31.382 INFO:tasks.rados.rados.0.smithi085.stdout:629: finishing rollback tid 0 to smithi08575397-50 2021-02-10T04:30:31.382 INFO:tasks.rados.rados.0.smithi085.stdout:629: finishing rollback tid 1 to smithi08575397-50 2021-02-10T04:30:31.382 INFO:tasks.rados.rados.0.smithi085.stdout:629: finishing rollback tid 2 to smithi08575397-50 2021-02-10T04:30:31.383 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 109 (ObjNum 198 snap 43 seq_num 198) dirty exists 2021-02-10T04:30:31.383 INFO:tasks.rados.rados.0.smithi085.stdout:623: expect (ObjNum 225 snap 51 seq_num 225) 2021-02-10T04:30:31.437 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: cluster 2021-02-10T04:30:31.437 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: 2021-02-10T04:30:31.438 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: 2021 2021-02-10T04:30:31.438 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: -02-10T04:30:29 2021-02-10T04:30:31.438 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: .861471+0000 osd.2 (osd.2) 56 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:30:31.438 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:30:31.438 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: 30:29.861482+0000 osd.2 (osd.2) 57 : cluster 2021-02-10T04:30:31.438 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:30:31.438 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: cluster 2021-02-10T04:30: 2021-02-10T04:30:31.438 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: 30.004056+0000 osd.1 (osd.1 2021-02-10T04:30:31.438 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: ) 88 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:30:31.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: 2021-02-10T04:30:31.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: cluster 2021-02-10T04:30:31.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:30:31.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: :30:30.004067+0000 osd.1 (osd.1) 2021-02-10T04:30:31.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: 89 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:30:31.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:30:31.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: -10T04:30:30.156669+0000 mon.a ( 2021-02-10T04:30:31.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: mon.0) 923 : cluster [DBG] osdmap e214: 8 total, 8 up, 8 in 2021-02-10T04:30:31.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:30:31.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: -02-10T04:30:30.322170+0000 osd.4 (osd.4 2021-02-10T04:30:31.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: ) 65 : cluster 2021-02-10T04:30:31.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:31.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: cluster 2021-02-10T04:30:31.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: 2021-02-10T04:30: 2021-02-10T04:30:31.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: 30.322177+0000 osd.4 (osd.4) 66 : cluster 2021-02-10T04:30:31.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:31.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:30:31.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: 02-10T04:30:30.322184+0000 osd.4 ( 2021-02-10T04:30:31.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: osd.4) 67 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:31.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: cluster 2021-02-10T04:30:31.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: 2021-02-10T04:30:31.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: 2021 2021-02-10T04:30:31.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: - 2021-02-10T04:30:31.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: 02 2021-02-10T04:30:31.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: - 2021-02-10T04:30:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: 10T04:30:30 2021-02-10T04:30:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: .322189+0000 osd.4 (osd.4) 68 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:30:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: -10T04:30:30.700352+0000 osd.0 (osd 2021-02-10T04:30:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: .0) 56 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:30:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:30:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: -10T04:30:30.700554+0000 osd.0 (osd 2021-02-10T04:30:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[29319]: .0) 57 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:30:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: cluster 2021-02-10T04:30:29.861471 2021-02-10T04:30:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 56 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:30:31.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: cluster 2021-02-10T04:30:31.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: 2021-02-10T04:30:29.861482+0000 osd.2 (osd.2) 57 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:30:31.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:30:31.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: -10T04:30:30.004056+0000 osd.1 ( 2021-02-10T04:30:31.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: osd.1) 88 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:30:31.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:30:31.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: -02-10T04:30:30.004067+0000 osd.1 (osd 2021-02-10T04:30:31.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: .1) 89 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:30:31.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: cluster 2021-02-10T04:30:30. 2021-02-10T04:30:31.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: 156669+0000 mon.a (mon.0) 923 : cluster [DBG] 2021-02-10T04:30:31.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: osdmap e214: 8 total, 8 up, 8 in 2021-02-10T04:30:31.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: cluster 2021-02-10T04:30:30.322170 2021-02-10T04:30:31.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 65 : cluster [WRN] 2021-02-10T04:30:31.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:31.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: cluster 2021-02-10T04:30:30 2021-02-10T04:30:31.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: .322177+0000 osd.4 (osd.4) 66 : cluster 2021-02-10T04:30:31.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:31.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: cluster 2021-02-10T04:30: 2021-02-10T04:30:31.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: 30.322184+0000 osd.4 (osd.4) 67 : 2021-02-10T04:30:31.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:31.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: cluster 2021-02-10T04:30 2021-02-10T04:30:31.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: :30.322189+0000 osd.4 (osd.4) 68 : cluster 2021-02-10T04:30:31.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:31.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: cluster 2021-02-10T04:30:30. 2021-02-10T04:30:31.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: 700352+0000 osd.0 (osd.0) 56 : cluster 2021-02-10T04:30:31.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: [DBG] purged_snaps scrub starts 2021-02-10T04:30:31.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:30:31.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: 10T04:30:30.700554+0000 osd.0 ( 2021-02-10T04:30:31.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[35961]: osd.0) 57 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:30:31.484 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:30:31.537 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:31.538 INFO:tasks.rados.rados.0.smithi085.stdout:630: finishing write tid 1 to smithi08575397-18 2021-02-10T04:30:31.538 INFO:tasks.rados.rados.0.smithi085.stdout:630: finishing write tid 2 to smithi08575397-18 2021-02-10T04:30:31.539 INFO:tasks.rados.rados.0.smithi085.stdout:630: finishing write tid 3 to smithi08575397-18 2021-02-10T04:30:31.539 INFO:tasks.rados.rados.0.smithi085.stdout:630: finishing write tid 4 to smithi08575397-18 2021-02-10T04:30:31.539 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 18 v 85 (ObjNum 250 snap 59 seq_num 250) dirty exists 2021-02-10T04:30:31.539 INFO:tasks.rados.rados.0.smithi085.stdout:630: left oid 18 (ObjNum 250 snap 59 seq_num 250) 2021-02-10T04:30:31.539 INFO:tasks.rados.rados.0.smithi085.stdout:622: finishing copy_from to smithi08575397-40 2021-02-10T04:30:31.539 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 90 (ObjNum 127 snap 23 seq_num 127) dirty exists 2021-02-10T04:30:31.539 INFO:tasks.rados.rados.0.smithi085.stdout:622: finishing copy_from racing read to smithi08575397-40 2021-02-10T04:30:31.539 INFO:tasks.rados.rados.0.smithi085.stdout:626: finishing copy_from to smithi08575397-11 2021-02-10T04:30:31.539 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 93 (ObjNum 202 snap 44 seq_num 202) dirty exists 2021-02-10T04:30:31.539 INFO:tasks.rados.rados.0.smithi085.stdout:626: finishing copy_from racing read to smithi08575397-11 2021-02-10T04:30:31.539 INFO:tasks.rados.rados.0.smithi085.stdout:622: done (15 left) 2021-02-10T04:30:31.539 INFO:tasks.rados.rados.0.smithi085.stdout:623: done (14 left) 2021-02-10T04:30:31.539 INFO:tasks.rados.rados.0.smithi085.stdout:625: done (13 left) 2021-02-10T04:30:31.539 INFO:tasks.rados.rados.0.smithi085.stdout:626: done (12 left) 2021-02-10T04:30:31.540 INFO:tasks.rados.rados.0.smithi085.stdout:627: done (11 left) 2021-02-10T04:30:31.540 INFO:tasks.rados.rados.0.smithi085.stdout:629: done (10 left) 2021-02-10T04:30:31.540 INFO:tasks.rados.rados.0.smithi085.stdout:630: done (9 left) 2021-02-10T04:30:31.540 INFO:tasks.rados.rados.0.smithi085.stdout:631: read oid 28 snap -1 2021-02-10T04:30:31.540 INFO:tasks.rados.rados.0.smithi085.stdout:631: expect (ObjNum 213 snap 47 seq_num 213) 2021-02-10T04:30:31.540 INFO:tasks.rados.rados.0.smithi085.stdout:632: write oid 11 current snap is 59 2021-02-10T04:30:31.540 INFO:tasks.rados.rados.0.smithi085.stdout:632: seq_num 251 ranges {618557=727148,2104745=113812} 2021-02-10T04:30:31.546 INFO:tasks.rados.rados.0.smithi085.stdout:632: writing smithi08575397-11 from 618557 to 1345705 tid 1 2021-02-10T04:30:31.551 INFO:tasks.rados.rados.0.smithi085.stdout:632: writing smithi08575397-11 from 2104745 to 2218557 tid 2 2021-02-10T04:30:31.552 INFO:tasks.rados.rados.0.smithi085.stdout:631: expect (ObjNum 213 snap 47 seq_num 213) 2021-02-10T04:30:31.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[33677]: debug 2021-02-10T04:30:31.299+0000 7fbec19a1700 -1 osd.4 215 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:30:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[29288]: cluster 2021-02-10T04:30:29.861471 2021-02-10T04:30:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 56 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:30:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[29288]: cluster 2021-02-10T04:30:29.861482+0000 osd.2 2021-02-10T04:30:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[29288]: (osd.2) 57 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:30:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[29288]: cluster 2021-02-10T04:30:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[29288]: 2021-02-10T04:30: 2021-02-10T04:30:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[29288]: 30.004056+0000 osd.1 (osd.1) 88 2021-02-10T04:30:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:30:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:30:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[29288]: 04:30:30.004067+0000 osd.1 (osd.1) 2021-02-10T04:30:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[29288]: 89 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:30:31.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[29288]: cluster 2021-02-10T04:30:30. 2021-02-10T04:30:31.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[29288]: 156669+0000 mon.a (mon.0) 923 : cluster [DBG] osdmap e214: 8 total, 8 up, 8 in 2021-02-10T04:30:31.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:30:31.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[29288]: -02-10T04:30:30.322170+0000 osd.4 (osd 2021-02-10T04:30:31.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[29288]: .4) 65 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:31.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[29288]: cluster 2021-02-10T04:30:30.322177 2021-02-10T04:30:31.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 66 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:31.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:30:31.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[29288]: T04:30:30.322184+0000 osd.4 (osd.4) 67 2021-02-10T04:30:31.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:31.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:30:31.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[29288]: 04:30:30.322189+0000 osd.4 (osd 2021-02-10T04:30:31.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[29288]: .4) 68 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:31.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[29288]: cluster 2021-02-10T04:30:31.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[29288]: 2021-02-10T04:30:30.700352+0000 2021-02-10T04:30:31.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[29288]: osd.0 (osd.0) 56 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:30:31.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[29288]: cluster 2021-02-10T04:30:31.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[29288]: 2021-02-10T04:30:30.700554+0000 2021-02-10T04:30:31.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:31 smithi164 conmon[29288]: osd.0 (osd.0) 57 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:30:31.691 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[52836]: debug 2021-02-10T04:30:31.691 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[52836]: 2021-02-10T04:30:31.649+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:31.692 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[52836]: 2021-02-10T04:30:31.726 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:30:31.830 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:30:31.842 INFO:tasks.rados.rados.0.smithi085.stdout:631: done (10 left) 2021-02-10T04:30:31.842 INFO:tasks.rados.rados.0.smithi085.stdout:633: delete oid 8 current snap is 59 2021-02-10T04:30:31.842 INFO:tasks.rados.rados.0.smithi085.stdout:632: finishing write tid 1 to smithi08575397-11 2021-02-10T04:30:31.842 INFO:tasks.rados.rados.0.smithi085.stdout:632: finishing write tid 2 to smithi08575397-11 2021-02-10T04:30:31.842 INFO:tasks.rados.rados.0.smithi085.stdout:632: finishing write tid 3 to smithi08575397-11 2021-02-10T04:30:31.843 INFO:tasks.rados.rados.0.smithi085.stdout:632: finishing write tid 4 to smithi08575397-11 2021-02-10T04:30:31.843 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 290 (ObjNum 251 snap 59 seq_num 251) dirty exists 2021-02-10T04:30:31.843 INFO:tasks.rados.rados.0.smithi085.stdout:632: left oid 11 (ObjNum 251 snap 59 seq_num 251) 2021-02-10T04:30:31.844 INFO:tasks.rados.rados.0.smithi085.stdout:632: done (10 left) 2021-02-10T04:30:31.845 INFO:tasks.rados.rados.0.smithi085.stdout:633: done (9 left) 2021-02-10T04:30:31.845 INFO:tasks.rados.rados.0.smithi085.stdout:634: copy_from oid 5 from oid 2 current snap is 59 2021-02-10T04:30:31.845 INFO:tasks.rados.rados.0.smithi085.stdout:635: copy_from oid 40 from oid 43 current snap is 59 2021-02-10T04:30:31.845 INFO:tasks.rados.rados.0.smithi085.stdout:636: read oid 45 snap 57 2021-02-10T04:30:31.845 INFO:tasks.rados.rados.0.smithi085.stdout:636: expect deleted 2021-02-10T04:30:31.845 INFO:tasks.rados.rados.0.smithi085.stdout:637: delete oid 6 current snap is 59 2021-02-10T04:30:31.846 INFO:tasks.rados.rados.0.smithi085.stdout:636: done (12 left) 2021-02-10T04:30:31.847 INFO:tasks.rados.rados.0.smithi085.stdout:637: done (11 left) 2021-02-10T04:30:31.847 INFO:tasks.rados.rados.0.smithi085.stdout:638: snap_remove snap 58 2021-02-10T04:30:31.874 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:31.992 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:30:31.997 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:31 smithi085 conmon[45443]: debug 2021-02-10T04:30:31.962+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:32.017 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.7 dump_ops_in_flight 2021-02-10T04:30:32.098 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:30:32.131 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:32.183 INFO:tasks.rados.rados.0.smithi085.stdout:638: done (11 left) 2021-02-10T04:30:32.183 INFO:tasks.rados.rados.0.smithi085.stdout:639: delete oid 3 current snap is 59 2021-02-10T04:30:32.183 INFO:tasks.rados.rados.0.smithi085.stdout:635: finishing copy_from to smithi08575397-40 2021-02-10T04:30:32.183 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 224 (ObjNum 235 snap 53 seq_num 235) dirty exists 2021-02-10T04:30:32.183 INFO:tasks.rados.rados.0.smithi085.stdout:635: finishing copy_from racing read to smithi08575397-40 2021-02-10T04:30:32.190 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.4 to 1 (10000) 2021-02-10T04:30:32.191 INFO:tasks.rados.rados.0.smithi085.stdout:635: done (11 left) 2021-02-10T04:30:32.191 INFO:tasks.rados.rados.0.smithi085.stdout:639: done (10 left) 2021-02-10T04:30:32.191 INFO:tasks.rados.rados.0.smithi085.stdout:640: write oid 43 current snap is 59 2021-02-10T04:30:32.191 INFO:tasks.rados.rados.0.smithi085.stdout:640: seq_num 255 ranges {685785=628144,1822320=463465} 2021-02-10T04:30:32.199 INFO:tasks.rados.rados.0.smithi085.stdout:640: writing smithi08575397-43 from 685785 to 1313929 tid 1 2021-02-10T04:30:32.205 INFO:tasks.rados.rados.0.smithi085.stdout:640: writing smithi08575397-43 from 1822320 to 2285785 tid 2 2021-02-10T04:30:32.205 INFO:tasks.rados.rados.0.smithi085.stdout:641: write oid 1 current snap is 59 2021-02-10T04:30:32.205 INFO:tasks.rados.rados.0.smithi085.stdout:641: seq_num 256 ranges {702592=703393,1880802=421790} 2021-02-10T04:30:32.214 INFO:tasks.rados.rados.0.smithi085.stdout:641: writing smithi08575397-1 from 702592 to 1405985 tid 1 2021-02-10T04:30:32.220 INFO:tasks.rados.rados.0.smithi085.stdout:641: writing smithi08575397-1 from 1880802 to 2302592 tid 2 2021-02-10T04:30:32.220 INFO:tasks.rados.rados.0.smithi085.stdout:640: finishing write tid 1 to smithi08575397-43 2021-02-10T04:30:32.220 INFO:tasks.rados.rados.0.smithi085.stdout:640: finishing write tid 2 to smithi08575397-43 2021-02-10T04:30:32.220 INFO:tasks.rados.rados.0.smithi085.stdout:640: finishing write tid 3 to smithi08575397-43 2021-02-10T04:30:32.220 INFO:tasks.rados.rados.0.smithi085.stdout:640: finishing write tid 4 to smithi08575397-43 2021-02-10T04:30:32.220 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 234 (ObjNum 255 snap 59 seq_num 255) dirty exists 2021-02-10T04:30:32.220 INFO:tasks.rados.rados.0.smithi085.stdout:640: left oid 43 (ObjNum 255 snap 59 seq_num 255) 2021-02-10T04:30:32.220 INFO:tasks.rados.rados.0.smithi085.stdout:641: finishing write tid 1 to smithi08575397-1 2021-02-10T04:30:32.221 INFO:tasks.rados.rados.0.smithi085.stdout:640: done (11 left) 2021-02-10T04:30:32.221 INFO:tasks.rados.rados.0.smithi085.stdout:642: snap_remove snap 57 2021-02-10T04:30:32.250 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:30:32.254 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:30:32 smithi164 conmon[37263]: debug 2021-02-10T04:30:32.255 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:30:32 smithi164 conmon[37263]: 2021-02-10T04:30:32.141+0000 7f1c689e5700 -1 osd.5 pg_epoch: 215 pg[7.0( v 215'153 (0'0,215'153] local-lis/les=93/94 n=5 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [5,1,3] r=0 lpr=93 crt=215'153 lcod 215'152 mlcod 215'152 active+clean+snaptrim trimq=[6f~1] ps=[60~3,66~3,6b~3]] removing snap head 2021-02-10T04:30:32.257 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:32 smithi164 conmon[29288]: cluster 2021-02-10T04:30:30.870548+0000 osd.2 (osd 2021-02-10T04:30:32.257 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:32 smithi164 conmon[29288]: .2) 58 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:30:32.257 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:32 smithi164 conmon[29288]: cluster 2021-02-10T04:30:30.870561+0000 osd.2 (osd.2) 59 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:30:32.257 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:32 smithi164 conmon[29288]: cluster 2021-02-10T04:30:31.047818+0000 osd.1 (osd.1) 90 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:30:32.257 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:32 smithi164 conmon[29288]: cluster 2021-02-10T04:30:31.047827+0000 osd.1 (osd.1) 91 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:30:32.258 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:32 smithi164 conmon[29288]: cluster 2021-02-10T04:30:31.172659+0000 mon.a (mon.0) 924 : cluster [DBG] osdmap e215: 8 total, 8 up, 8 in 2021-02-10T04:30:32.258 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:32 smithi164 conmon[29288]: cluster 2021-02-10T04:30:31.299863+0000 osd.4 (osd.4) 69 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:32.258 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:32 smithi164 conmon[29288]: cluster 2021-02-10T04:30:31.299881+0000 osd.4 (osd.4) 70 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:32.258 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:32 smithi164 conmon[29288]: cluster 2021-02-10T04:30:31.299891+0000 osd.4 (osd.4) 71 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:32.258 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:32 smithi164 conmon[29288]: cluster 2021-02-10T04:30:31.299897+0000 osd.4 (osd.4) 72 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:32.258 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:32 smithi164 conmon[29288]: audit 2021-02-10T04:30:31.987899+0000 mon.a (mon.0) 925 : audit [INF] from='client.? 172.21.15.85:0/299751928' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-02-10T04:30:32.259 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:32 smithi164 conmon[29288]: cluster 2021-02-10T04:30:32.019993+0000 mgr.y (mgr.14140) 406 : cluster [DBG] pgmap v472: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 160 active+clean; 145 MiB data, 445 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 1.8 MiB/s wr, 11 op/s 2021-02-10T04:30:32.259 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:30:32 smithi164 conmon[44544]: debug 2021-02-10T04:30:32.211+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:32.312 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:30:32.357 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:30:32.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[35961]: cluster 2021-02-10T04:30:30.870548+0000 2021-02-10T04:30:32.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[35961]: osd.2 (osd.2) 58 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:30:32.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[35961]: cluster 2021-02-10T04:30:30.870561+0000 osd.2 2021-02-10T04:30:32.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[35961]: (osd.2) 59 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:30:32.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[35961]: cluster 2021-02-10T04:30:32.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[35961]: 2021-02-10T04:30:31.047818+0000 osd.1 (osd.1) 2021-02-10T04:30:32.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[35961]: 90 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:30:32.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[35961]: cluster 2021-02-10T04:30:31.047827+0000 osd.1 ( 2021-02-10T04:30:32.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[35961]: osd.1) 91 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:30:32.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[35961]: cluster 2021-02-10T04:30:31.172659+0000 2021-02-10T04:30:32.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[35961]: mon.a (mon.0) 924 : cluster [DBG] osdmap e215: 8 total, 8 up, 8 in 2021-02-10T04:30:32.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[35961]: cluster 2021-02-10T04:30:31.299863+0000 2021-02-10T04:30:32.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[35961]: osd.4 (osd.4) 69 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:32.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[35961]: cluster 2021-02-10T04:30:31. 2021-02-10T04:30:32.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[35961]: 299881+0000 osd.4 (osd.4) 70 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:32.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[35961]: cluster 2021-02-10T04:30:31. 2021-02-10T04:30:32.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[35961]: 299891+0000 osd.4 (osd.4) 71 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:32.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[35961]: cluster 2021-02-10T04:30: 2021-02-10T04:30:32.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[35961]: 31.299897+0000 osd.4 (osd.4) 72 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:32.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:30:32.392 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[35961]: 30:31.987899+0000 mon.a (mon.0) 925 : audit [INF] from='client.? 172.21.15.85:0/299751928' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-02-10T04:30:32.392 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[35961]: cluster 2021-02-10T04:30:32.019993+0000 mgr.y (mgr.14140 2021-02-10T04:30:32.392 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[35961]: ) 406 : cluster [DBG] pgmap v472: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 160 active+clean; 145 MiB data, 445 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 1.8 MiB/s wr, 11 op/s 2021-02-10T04:30:32.392 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: cluster 2021-02-10T04:30:32.392 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: 2021-02-10T04:30:32.392 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: 2021 2021-02-10T04:30:32.393 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: - 2021-02-10T04:30:32.393 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: 02 2021-02-10T04:30:32.393 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: - 2021-02-10T04:30:32.393 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: 10 2021-02-10T04:30:32.393 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: T 2021-02-10T04:30:32.393 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: 04 2021-02-10T04:30:32.393 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: : 2021-02-10T04:30:32.393 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: 30 2021-02-10T04:30:32.393 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: : 2021-02-10T04:30:32.393 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: 30 2021-02-10T04:30:32.393 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: . 2021-02-10T04:30:32.393 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: 870548 2021-02-10T04:30:32.393 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: +0000 2021-02-10T04:30:32.394 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: 2021-02-10T04:30:32.394 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: osd.2 2021-02-10T04:30:32.394 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: ( 2021-02-10T04:30:32.394 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: osd 2021-02-10T04:30:32.394 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: . 2021-02-10T04:30:32.394 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: 2 2021-02-10T04:30:32.394 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: ) 2021-02-10T04:30:32.394 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: 58 2021-02-10T04:30:32.394 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: : 2021-02-10T04:30:32.394 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: cluster 2021-02-10T04:30:32.394 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: 2021-02-10T04:30:32.394 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: [WRN] 2021-02-10T04:30:32.394 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: 2021-02-10T04:30:32.395 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:30:32.395 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: 2021-02-10T04:30:32.395 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: cluster 2021-02-10T04:30:32.395 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: 2021-02-10T04:30:32.395 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: 2021 2021-02-10T04:30:32.395 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: - 2021-02-10T04:30:32.395 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: 02 2021-02-10T04:30:32.395 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: - 2021-02-10T04:30:32.395 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: 10 2021-02-10T04:30:32.395 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: T 2021-02-10T04:30:32.395 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: 04 2021-02-10T04:30:32.395 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: : 2021-02-10T04:30:32.395 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: 30 2021-02-10T04:30:32.395 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: : 2021-02-10T04:30:32.396 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: 30 2021-02-10T04:30:32.396 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: . 2021-02-10T04:30:32.396 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: 870561+0000 osd.2 (osd.2) 59 : 2021-02-10T04:30:32.396 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:30:32.396 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: cluster 2021-02-10T04:30:32.396 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: 2021-02-10T04:30:31.047818+0000 osd.1 (osd.1 2021-02-10T04:30:32.396 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: ) 90 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:30:32.396 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:30:32.396 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: 02-10T04:30:31.047827 2021-02-10T04:30:32.396 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: +0000 osd.1 (osd.1) 91 : cluster 2021-02-10T04:30:32.396 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:30:32.396 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:30:32.397 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: T04:30:31.172659+0000 mon.a 2021-02-10T04:30:32.397 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: (mon.0) 924 : cluster [DBG] osdmap e215: 8 total, 8 up, 8 in 2021-02-10T04:30:32.397 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: 2021-02-10T04:30:32.397 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: cluster 2021-02-10T04:30: 2021-02-10T04:30:32.397 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: 31.299863+0000 osd.4 (osd.4) 69 : 2021-02-10T04:30:32.397 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:32.397 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:30:32.397 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: 10T04:30:31.299881+0000 osd.4 ( 2021-02-10T04:30:32.397 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: osd.4) 70 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:32.397 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:30:32.397 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: -02-10T04:30:31.299891 2021-02-10T04:30:32.397 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 71 : cluster 2021-02-10T04:30:32.397 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:32.397 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: cluster 2021-02-10T04:30:32.398 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:30:32.398 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: T04:30:31.299897+0000 osd.4 (osd.4) 72 : cluster [WRN] 2021-02-10T04:30:32.398 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:32.398 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:30:32.398 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: 10T04:30:31.987899+0000 mon.a 2021-02-10T04:30:32.398 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: (mon.0) 925 : audit [INF] 2021-02-10T04:30:32.398 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: from='client.? 172.21.15.85:0/299751928' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-02-10T04:30:32.398 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:30:32.398 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: 10T04:30:32.019993+0000 2021-02-10T04:30:32.398 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: mgr.y (mgr.14140) 406 : cluster [DBG] 2021-02-10T04:30:32.398 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[29319]: pgmap v472: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 160 active+clean; 145 MiB data, 445 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 1.8 MiB/s wr, 11 op/s 2021-02-10T04:30:32.399 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:32.551 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:30:32 smithi164 conmon[33677]: debug 2021-02-10T04:30:32.264+0000 7fbec19a1700 -1 osd.4 216 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:30:32.690 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[52836]: debug 2021-02-10T04:30:32.691 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:30:32 smithi085 conmon[52836]: 2021-02-10T04:30:32.473+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:32.758 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:30:33.183 INFO:tasks.rados.rados.0.smithi085.stdout:642: done (11 left) 2021-02-10T04:30:33.183 INFO:tasks.rados.rados.0.smithi085.stdout:643: snap_create 2021-02-10T04:30:33.183 INFO:tasks.rados.rados.0.smithi085.stdout:641: finishing write tid 2 to smithi08575397-1 2021-02-10T04:30:33.184 INFO:tasks.rados.rados.0.smithi085.stdout:641: finishing write tid 3 to smithi08575397-1 2021-02-10T04:30:33.184 INFO:tasks.rados.rados.0.smithi085.stdout:641: finishing write tid 4 to smithi08575397-1 2021-02-10T04:30:33.184 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 119 (ObjNum 256 snap 59 seq_num 256) dirty exists 2021-02-10T04:30:33.184 INFO:tasks.rados.rados.0.smithi085.stdout:641: left oid 1 (ObjNum 256 snap 59 seq_num 256) 2021-02-10T04:30:33.190 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is unset 2021-02-10T04:30:33.286 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:33 smithi164 conmon[29288]: audit 2021-02-10T04:30:32.181127+0000 mon.a (mon.0) 926 : audit [INF] from='client.? 172.21.15.85:0/299751928' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-02-10T04:30:33.287 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:33 smithi164 conmon[29288]: cluster 2021-02-10T04:30:32.181272+0000 mon.a (mon.0) 927 : cluster [DBG] osdmap e216: 8 total, 8 up, 8 in 2021-02-10T04:30:33.287 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:33 smithi164 conmon[29288]: cluster 2021-02-10T04:30:32.265238+0000 osd.4 (osd.4) 73 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:33.287 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:33 smithi164 conmon[29288]: cluster 2021-02-10T04:30:32.265249+0000 osd.4 (osd.4) 74 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:33.287 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:33 smithi164 conmon[29288]: cluster 2021-02-10T04:30:32.265253+0000 osd.4 (osd.4) 75 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:33.287 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:33 smithi164 conmon[29288]: cluster 2021-02-10T04:30:32.265256+0000 osd.4 (osd.4) 76 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:33.287 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:33 smithi164 conmon[29288]: audit 2021-02-10T04:30:32.373243+0000 mon.b (mon.2) 28 : audit [INF] from='client.? 172.21.15.85:0/1642773864' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:30:33.287 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:33 smithi164 conmon[29288]: audit 2021-02-10T04:30:32.373702+0000 mon.a (mon.0) 928 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:30:33.288 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:30:33 smithi164 conmon[33677]: debug 2021-02-10T04:30:33.286+0000 7fbec19a1700 -1 osd.4 217 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:30:33.441 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[52836]: debug 2021-02-10T04:30:33.193+0000 7fba70fb0700 -1 osd.3 pg_epoch: 217 pg[7.a( v 217'161 (0'0,217'161] local-lis/les=93/94 n=4 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [3,2,5] r=0 lpr=93 luod=216'159 crt=217'161 lcod 216'158 mlcod 216'158 active+clean+snaptrim trimq=[74~1] ps=[60~3,66~3,6b~3,6f~1,75~1]] removing snap head 2021-02-10T04:30:33.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[29319]: audit 2021-02-10T04:30:32.181127+0000 mon.a (mon.0) 926 : 2021-02-10T04:30:33.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[29319]: audit [INF] from='client.? 172.21.15.85:0/299751928' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-02-10T04:30:33.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[29319]: cluster 2021-02-10T04:30:32.181272+0000 mon.a (mon.0) 927 : cluster [DBG] osdmap e216: 8 total, 8 up, 8 in 2021-02-10T04:30:33.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[29319]: cluster 2021-02-10T04:30:32.265238+0000 osd.4 (osd.4) 73 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:33.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:30:33.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[29319]: -02-10T04:30:32.265249+0000 2021-02-10T04:30:33.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[29319]: osd.4 (osd.4) 74 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:33.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:30:33.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[29319]: -02-10T04:30:32.265253+0000 2021-02-10T04:30:33.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[29319]: osd.4 (osd.4) 75 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:33.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:30:33.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[29319]: -10T04:30:32.265256+0000 osd.4 (osd 2021-02-10T04:30:33.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[29319]: .4) 76 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:33.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:30:33.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[29319]: -10T04:30:32.373243+0000 mon.b 2021-02-10T04:30:33.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[29319]: (mon.2) 28 : audit [INF] from='client.? 172.21.15.85:0/1642773864' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:30:33.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[29319]: audit 2021-02-10T04: 2021-02-10T04:30:33.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[29319]: 30:32.373702+0000 mon.a (mon.0) 928 : audit [INF] 2021-02-10T04:30:33.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[29319]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:30:33.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[35961]: audit 2021-02-10T04:30:32.181127 2021-02-10T04:30:33.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[35961]: +0000 mon.a (mon.0) 926 : audit [INF] from='client.? 172.21.15.85:0/299751928' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-02-10T04:30:33.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[35961]: cluster 2021-02-10T04:30:33.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[35961]: 2021-02-10T04:30:32.181272+0000 mon.a ( 2021-02-10T04:30:33.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[35961]: mon.0) 927 : cluster [DBG] osdmap e216: 8 total, 8 up, 8 in 2021-02-10T04:30:33.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[35961]: cluster 2021-02-10T04:30: 2021-02-10T04:30:33.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[35961]: 32.265238+0000 osd.4 (osd.4) 73 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:33.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:30:33.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[35961]: 02-10T04:30:32.265249+0000 osd.4 (osd.4) 74 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:33.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:30:33.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[35961]: 02-10T04:30:32.265253+0000 osd.4 (osd.4) 2021-02-10T04:30:33.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[35961]: 75 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:33.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[35961]: cluster 2021-02-10T04:30:32.265256+0000 osd.4 (osd.4) 76 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:33.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[35961]: audit 2021-02-10T04:30:32.373243+0000 mon.b (mon.2) 2021-02-10T04:30:33.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[35961]: 28 : audit [INF] from='client.? 172.21.15.85:0/1642773864' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:30:33.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:30:33.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[35961]: :30:32.373702+0000 mon.a (mon.0) 928 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:30:33.445 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:30:33 smithi085 conmon[41540]: debug 2021-02-10T04:30:33.187+0000 7fb94835c700 -1 osd.0 pg_epoch: 217 pg[7.4( v 214'60 (0'0,214'60] local-lis/les=93/94 n=3 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [0,4,1] r=0 lpr=93 crt=214'60 lcod 199'58 mlcod 199'58 active+clean+snaptrim trimq=[74~1] ps=[60~3,66~3,6b~3,6f~1,75~1]] removing snap head 2021-02-10T04:30:33.631 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:30:33.948 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:30:34.187 INFO:tasks.rados.rados.0.smithi085.stdout:641: done (11 left) 2021-02-10T04:30:34.187 INFO:tasks.rados.rados.0.smithi085.stdout:643: done (10 left) 2021-02-10T04:30:34.187 INFO:tasks.rados.rados.0.smithi085.stdout:644: read oid 41 snap 59 2021-02-10T04:30:34.187 INFO:tasks.rados.rados.0.smithi085.stdout:644: expect (ObjNum 229 snap 51 seq_num 229) 2021-02-10T04:30:34.188 INFO:tasks.rados.rados.0.smithi085.stdout:645: write (excl) oid 23 current snap is 60 2021-02-10T04:30:34.188 INFO:tasks.rados.rados.0.smithi085.stdout:645: seq_num 257 ranges {719399=778642,1939284=380115} 2021-02-10T04:30:34.196 INFO:tasks.rados.rados.0.smithi085.stdout:645: writing smithi08575397-23 from 719399 to 1498041 tid 1 2021-02-10T04:30:34.201 INFO:tasks.rados.rados.0.smithi085.stdout:645: writing smithi08575397-23 from 1939284 to 2319399 tid 2 2021-02-10T04:30:34.201 INFO:tasks.rados.rados.0.smithi085.stdout:646: delete oid 16 current snap is 60 2021-02-10T04:30:34.202 INFO:tasks.rados.rados.0.smithi085.stdout:645: finishing write tid 1 to smithi08575397-23 2021-02-10T04:30:34.203 INFO:tasks.rados.rados.0.smithi085.stdout:644: expect (ObjNum 229 snap 51 seq_num 229) 2021-02-10T04:30:34.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[35961]: cluster 2021-02-10T04:30:33.173714+0000 2021-02-10T04:30:34.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[35961]: mon.a (mon.0) 929 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:30:34.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[35961]: audit 2021-02-10T04:30:33.184005+0000 mon.a (mon.0) 930 2021-02-10T04:30:34.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[35961]: : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:30:34.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:30:34.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[35961]: 30:33.184116+0000 mon.a (mon.0) 931 2021-02-10T04:30:34.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[35961]: : cluster [DBG] osdmap e217: 8 total, 8 up, 8 in 2021-02-10T04:30:34.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:30:34.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[35961]: 02-10T04:30:33.287180+0000 osd.4 (osd.4) 2021-02-10T04:30:34.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[35961]: 77 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:34.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:30:34.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[35961]: T04:30:33.287193+0000 osd.4 (osd.4 2021-02-10T04:30:34.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[35961]: ) 78 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:34.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[35961]: cluster 2021-02-10T04:30:34.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[35961]: 2021-02-10T04:30:33. 2021-02-10T04:30:34.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[35961]: 287199+0000 osd.4 (osd.4) 79 : cluster [WRN] 2021-02-10T04:30:34.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:34.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:30:34.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[35961]: 04:30:33.287204+0000 osd.4 (osd.4) 80 : cluster [WRN] 2021-02-10T04:30:34.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:34.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[35961]: cluster 2021-02-10T04:30:34.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[35961]: 2021-02-10T04:30:34.020427+0000 mgr.y (mgr 2021-02-10T04:30:34.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[35961]: .14140) 407 : cluster [DBG] pgmap v475: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 5 active+clean+snaptrim, 161 active+clean; 139 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 7.7 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2021-02-10T04:30:34.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[29319]: cluster 2021-02-10T04:30:33.173714+0000 mon.a ( 2021-02-10T04:30:34.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[29319]: mon.0) 929 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:30:34.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[29319]: audit 2021-02-10T04:30:33.184005+0000 mon.a (mon.0) 930 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:30:34.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[29319]: cluster 2021-02-10T04:30:33.184116+0000 mon.a (mon.0) 931 : cluster [DBG] osdmap e217: 8 total, 8 up, 8 in 2021-02-10T04:30:34.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[29319]: cluster 2021-02-10T04:30:33.287180+0000 osd.4 (osd.4) 77 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:34.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[29319]: cluster 2021-02-10T04:30:33.287193+0000 osd.4 (osd.4) 78 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:34.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[29319]: cluster 2021-02-10T04:30:33.287199+0000 osd.4 (osd.4) 79 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:34.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[29319]: cluster 2021-02-10T04:30:33.287204+0000 osd.4 (osd.4) 80 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:34.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[29319]: cluster 2021-02-10T04:30:34.020427+0000 mgr.y (mgr.14140) 407 2021-02-10T04:30:34.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:34 smithi085 conmon[29319]: : cluster [DBG] pgmap v475: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 5 active+clean+snaptrim, 161 active+clean; 139 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 7.7 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2021-02-10T04:30:34.449 INFO:tasks.rados.rados.0.smithi085.stdout:644: done (12 left) 2021-02-10T04:30:34.449 INFO:tasks.rados.rados.0.smithi085.stdout:646: done (11 left) 2021-02-10T04:30:34.449 INFO:tasks.rados.rados.0.smithi085.stdout:647: write oid 45 current snap is 60 2021-02-10T04:30:34.450 INFO:tasks.rados.rados.0.smithi085.stdout:645: finishing write tid 2 to smithi08575397-23 2021-02-10T04:30:34.450 INFO:tasks.rados.rados.0.smithi085.stdout:645: finishing write tid 3 to smithi08575397-23 2021-02-10T04:30:34.450 INFO:tasks.rados.rados.0.smithi085.stdout:645: finishing write tid 4 to smithi08575397-23 2021-02-10T04:30:34.450 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 101 (ObjNum 257 snap 60 seq_num 257) dirty exists 2021-02-10T04:30:34.450 INFO:tasks.rados.rados.0.smithi085.stdout:645: left oid 23 (ObjNum 257 snap 60 seq_num 257) 2021-02-10T04:30:34.450 INFO:tasks.rados.rados.0.smithi085.stdout:647: seq_num 259 ranges {753013=645493,2056248=296765} 2021-02-10T04:30:34.457 INFO:tasks.rados.rados.0.smithi085.stdout:647: writing smithi08575397-45 from 753013 to 1398506 tid 1 2021-02-10T04:30:34.464 INFO:tasks.rados.rados.0.smithi085.stdout:647: writing smithi08575397-45 from 2056248 to 2353013 tid 2 2021-02-10T04:30:34.465 INFO:tasks.rados.rados.0.smithi085.stdout:645: done (11 left) 2021-02-10T04:30:34.465 INFO:tasks.rados.rados.0.smithi085.stdout:648: rollback oid 38 current snap is 60 2021-02-10T04:30:34.465 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 38 to 59 2021-02-10T04:30:34.465 INFO:tasks.rados.rados.0.smithi085.stdout:647: finishing write tid 1 to smithi08575397-45 2021-02-10T04:30:34.465 INFO:tasks.rados.rados.0.smithi085.stdout:649: read oid 46 snap -1 2021-02-10T04:30:34.465 INFO:tasks.rados.rados.0.smithi085.stdout:649: expect (ObjNum 202 snap 44 seq_num 202) 2021-02-10T04:30:34.465 INFO:tasks.rados.rados.0.smithi085.stdout:650: copy_from oid 1 from oid 7 current snap is 60 2021-02-10T04:30:34.465 INFO:tasks.rados.rados.0.smithi085.stdout:651: copy_from oid 16 from oid 9 current snap is 60 2021-02-10T04:30:34.465 INFO:tasks.rados.rados.0.smithi085.stdout:652: snap_create 2021-02-10T04:30:34.468 INFO:tasks.rados.rados.0.smithi085.stdout:648: finishing rollback tid 0 to smithi08575397-38 2021-02-10T04:30:34.468 INFO:tasks.rados.rados.0.smithi085.stdout:647: finishing write tid 2 to smithi08575397-45 2021-02-10T04:30:34.468 INFO:tasks.rados.rados.0.smithi085.stdout:647: finishing write tid 3 to smithi08575397-45 2021-02-10T04:30:34.468 INFO:tasks.rados.rados.0.smithi085.stdout:647: finishing write tid 4 to smithi08575397-45 2021-02-10T04:30:34.468 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 45 v 115 (ObjNum 259 snap 60 seq_num 259) dirty exists 2021-02-10T04:30:34.469 INFO:tasks.rados.rados.0.smithi085.stdout:647: left oid 45 (ObjNum 259 snap 60 seq_num 259) 2021-02-10T04:30:34.471 INFO:tasks.rados.rados.0.smithi085.stdout:648: finishing rollback tid 1 to smithi08575397-38 2021-02-10T04:30:34.471 INFO:tasks.rados.rados.0.smithi085.stdout:648: finishing rollback tid 2 to smithi08575397-38 2021-02-10T04:30:34.471 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 193 (ObjNum 209 snap 46 seq_num 209) dirty exists 2021-02-10T04:30:34.472 INFO:tasks.rados.rados.0.smithi085.stdout:649: expect (ObjNum 202 snap 44 seq_num 202) 2021-02-10T04:30:34.649 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:30:34 smithi164 conmon[40892]: debug 2021-02-10T04:30:34.191+0000 7f30324bf700 -1 osd.6 pg_epoch: 218 pg[7.d( v 218'121 (0'0,218'121] local-lis/les=93/94 n=6 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [6,1,0] r=0 lpr=93 luod=216'119 crt=218'121 lcod 216'118 mlcod 216'118 active+clean+snaptrim trimq=[74~1] ps=[60~3,66~3,6b~3,6f~1,75~1]] removing snap head 2021-02-10T04:30:34.650 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:30:34 smithi164 conmon[33677]: debug 2021-02-10T04:30:34.272+0000 7fbec19a1700 -1 osd.4 217 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:30:34.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:34 smithi164 conmon[29288]: cluster 2021-02-10T04:30:33.173714+0000 mon.a (mon.0) 929 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:30:34.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:34 smithi164 conmon[29288]: audit 2021-02-10T04:30:33.184005+0000 mon.a (mon.0) 930 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:30:34.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:34 smithi164 conmon[29288]: cluster 2021-02-10T04:30:33.184116+0000 mon.a (mon.0) 931 : cluster [DBG] osdmap e217: 8 total, 8 up, 8 in 2021-02-10T04:30:34.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:34 smithi164 conmon[29288]: cluster 2021-02-10T04:30:33.287180+0000 osd.4 (osd.4) 77 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:34.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:34 smithi164 conmon[29288]: cluster 2021-02-10T04:30:33.287193+0000 osd.4 (osd.4) 78 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:34.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:34 smithi164 conmon[29288]: cluster 2021-02-10T04:30:33.287199+0000 osd.4 2021-02-10T04:30:34.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:34 smithi164 conmon[29288]: (osd.4) 79 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:34.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:34 smithi164 conmon[29288]: cluster 2021-02-10T04:30:33.287204+0000 osd.4 (osd.4) 80 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:34.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:34 smithi164 conmon[29288]: 2021-02-10T04:30:34.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:34 smithi164 conmon[29288]: cluster 2021-02-10T04:30:34. 2021-02-10T04:30:34.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:34 smithi164 conmon[29288]: 020427+0000 mgr.y (mgr.14140) 407 : cluster [DBG] pgmap v475: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 5 active+clean+snaptrim, 161 active+clean; 139 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 7.7 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2021-02-10T04:30:34.730 INFO:tasks.rados.rados.0.smithi085.stdout:650: finishing copy_from to smithi08575397-1 2021-02-10T04:30:34.730 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 227 (ObjNum 202 snap 44 seq_num 202) dirty exists 2021-02-10T04:30:34.730 INFO:tasks.rados.rados.0.smithi085.stdout:650: finishing copy_from racing read to smithi08575397-1 2021-02-10T04:30:34.730 INFO:tasks.rados.rados.0.smithi085.stdout:651: finishing copy_from to smithi08575397-16 2021-02-10T04:30:34.730 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 170 (ObjNum 229 snap 51 seq_num 229) dirty exists 2021-02-10T04:30:34.730 INFO:tasks.rados.rados.0.smithi085.stdout:651: finishing copy_from racing read to smithi08575397-16 2021-02-10T04:30:35.199 INFO:tasks.rados.rados.0.smithi085.stdout:647: done (15 left) 2021-02-10T04:30:35.200 INFO:tasks.rados.rados.0.smithi085.stdout:648: done (14 left) 2021-02-10T04:30:35.200 INFO:tasks.rados.rados.0.smithi085.stdout:649: done (13 left) 2021-02-10T04:30:35.200 INFO:tasks.rados.rados.0.smithi085.stdout:650: done (12 left) 2021-02-10T04:30:35.200 INFO:tasks.rados.rados.0.smithi085.stdout:651: done (11 left) 2021-02-10T04:30:35.200 INFO:tasks.rados.rados.0.smithi085.stdout:652: done (10 left) 2021-02-10T04:30:35.201 INFO:tasks.rados.rados.0.smithi085.stdout:653: copy_from oid 26 from oid 6 current snap is 61 2021-02-10T04:30:35.201 INFO:tasks.rados.rados.0.smithi085.stdout:654: read oid 16 snap -1 2021-02-10T04:30:35.201 INFO:tasks.rados.rados.0.smithi085.stdout:654: expect (ObjNum 229 snap 51 seq_num 229) 2021-02-10T04:30:35.201 INFO:tasks.rados.rados.0.smithi085.stdout:655: snap_create 2021-02-10T04:30:35.202 INFO:tasks.rados.rados.0.smithi085.stdout:653: finishing copy_from to smithi08575397-26 2021-02-10T04:30:35.203 INFO:tasks.rados.rados.0.smithi085.stdout:653: got expected ENOENT (src dne) 2021-02-10T04:30:35.203 INFO:tasks.rados.rados.0.smithi085.stdout:653: finishing copy_from racing read to smithi08575397-26 2021-02-10T04:30:35.218 INFO:tasks.rados.rados.0.smithi085.stdout:654: expect (ObjNum 229 snap 51 seq_num 229) 2021-02-10T04:30:35.289 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:30:35.289 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2021-02-10T04:30:35.290 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 0 2021-02-10T04:30:35.290 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:30:35.311 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:35 smithi164 conmon[29288]: cluster 2021-02-10T04:30:34.180818+0000 mon.a (mon.0) 2021-02-10T04:30:35.311 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:35 smithi164 conmon[29288]: 932 : cluster [WRN] Health check update: 6 slow ops, oldest one blocked for 81 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:30:35.311 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:35 smithi164 conmon[29288]: cluster 2021-02-10T04:30:34.186216+0000 mon.a (mon.0) 933 : cluster [DBG] osdmap e218: 8 total, 8 up, 8 in 2021-02-10T04:30:35.312 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:35 smithi164 conmon[29288]: cluster 2021-02-10T04:30:34.273422+0000 osd.4 (osd.4) 81 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:35.312 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:35 smithi164 conmon[29288]: cluster 2021-02-10T04:30:34.273432+0000 osd.4 (osd.4) 82 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:35.312 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:35 smithi164 conmon[29288]: cluster 2021-02-10T04:30:34.273436+0000 osd.4 (osd.4) 83 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:35.312 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:35 smithi164 conmon[29288]: cluster 2021-02-10T04:30:34.273440+0000 osd.4 (osd.4) 84 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:35.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[29319]: cluster 2021-02-10T04:30:34.180818+0000 2021-02-10T04:30:35.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[29319]: mon.a (mon.0) 932 : cluster [WRN] Health check update: 6 slow ops, oldest one blocked for 81 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:30:35.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[29319]: cluster 2021-02-10T04:30:34 2021-02-10T04:30:35.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[29319]: .186216+0000 mon.a (mon.0) 933 : cluster [DBG] osdmap e218: 8 total, 8 up, 8 in 2021-02-10T04:30:35.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:30:35.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[29319]: -02-10T04:30:34.273422+0000 2021-02-10T04:30:35.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[29319]: osd.4 (osd.4) 81 : cluster [WRN] 2021-02-10T04:30:35.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:35.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:30:35.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[29319]: -02-10T04:30:34.273432 2021-02-10T04:30:35.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 82 : cluster 2021-02-10T04:30:35.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:35.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[29319]: cluster 2021-02-10T04:30:35.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[29319]: 2021-02-10T04:30:34. 2021-02-10T04:30:35.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[29319]: 273436+0000 osd.4 (osd.4) 83 2021-02-10T04:30:35.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:35.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[29319]: cluster 2021-02-10T04:30:35.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[29319]: 2021-02-10T04:30:34.273440 2021-02-10T04:30:35.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 84 : cluster [WRN] 2021-02-10T04:30:35.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:35.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[35961]: cluster 2021-02-10T04:30:35.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:30:35.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[35961]: T04:30:34.180818+0000 mon.a (mon.0) 932 : cluster [WRN] Health check update: 6 slow ops, oldest one blocked for 81 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:30:35.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[35961]: cluster 2021-02-10T04:30:35.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[35961]: 2021-02-10T04:30 2021-02-10T04:30:35.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[35961]: :34.186216+0000 mon.a (mon.0) 933 : cluster 2021-02-10T04:30:35.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[35961]: [DBG] osdmap e218: 8 total, 8 up, 8 in 2021-02-10T04:30:35.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:30:35.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[35961]: 02-10T04:30:34.273422 2021-02-10T04:30:35.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 81 : cluster [WRN] 2021-02-10T04:30:35.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:35.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:30:35.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[35961]: -02-10T04:30:34.273432+0000 osd.4 (osd.4) 82 : 2021-02-10T04:30:35.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:35.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[35961]: cluster 2021-02-10T04:30:34 2021-02-10T04:30:35.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[35961]: .273436+0000 osd.4 (osd.4) 83 : cluster 2021-02-10T04:30:35.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:35.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:30:35.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:35 smithi085 conmon[35961]: 10T04:30:34.273440+0000 osd.4 (osd.4) 84 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:35.498 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:30:35.531 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:35.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:30:35 smithi164 conmon[33677]: debug 2021-02-10T04:30:35.312+0000 7fbec19a1700 -1 osd.4 219 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:30:36.204 INFO:tasks.rados.rados.0.smithi085.stdout:653: done (12 left) 2021-02-10T04:30:36.204 INFO:tasks.rados.rados.0.smithi085.stdout:654: done (11 left) 2021-02-10T04:30:36.204 INFO:tasks.rados.rados.0.smithi085.stdout:655: done (10 left) 2021-02-10T04:30:36.204 INFO:tasks.rados.rados.0.smithi085.stdout:656: snap_create 2021-02-10T04:30:36.572 INFO:tasks.rados.rados.0.smithi085.stdout:634: finishing copy_from to smithi08575397-5 2021-02-10T04:30:36.572 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 75 (ObjNum 225 snap 51 seq_num 225) dirty exists 2021-02-10T04:30:36.572 INFO:tasks.rados.rados.0.smithi085.stdout:634: finishing copy_from racing read to smithi08575397-5 2021-02-10T04:30:36.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:30:36 smithi164 conmon[33677]: debug 2021-02-10T04:30:36.340+0000 7fbec19a1700 -1 osd.4 220 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:30:36.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:36 smithi164 conmon[29288]: cluster 2021-02-10T04:30:35.198716+0000 mon.a (mon. 2021-02-10T04:30:36.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:36 smithi164 conmon[29288]: 0) 934 : cluster [DBG] osdmap e219: 8 total, 8 up, 8 in 2021-02-10T04:30:36.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:36 smithi164 conmon[29288]: cluster 2021-02-10T04:30:35.312965+0000 osd.4 (osd.4) 85 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:36.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:36 smithi164 conmon[29288]: cluster 2021-02-10T04:30:35.312974+0000 osd.4 (osd.4) 86 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:36.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:36 smithi164 conmon[29288]: cluster 2021-02-10T04:30:35.312979+0000 osd.4 (osd.4) 87 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:36.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:36 smithi164 conmon[29288]: cluster 2021-02-10T04:30:35.312983+0000 osd.4 (osd.4) 88 : cluster 2021-02-10T04:30:36.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:36 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:36.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:36 smithi164 conmon[29288]: cluster 2021-02-10T04:30:36.020949+0000 mgr.y (mgr.14140) 408 : cluster 2021-02-10T04:30:36.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:36 smithi164 conmon[29288]: [DBG] pgmap v478: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim, 162 active+clean; 129 MiB data, 458 MiB used, 715 GiB / 715 GiB avail; 17 MiB/s rd, 4.1 MiB/s wr, 21 op/s 2021-02-10T04:30:36.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[35961]: cluster 2021-02-10T04:30:35.198716+0000 2021-02-10T04:30:36.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[35961]: mon.a (mon.0) 934 : cluster [DBG] osdmap e219: 8 total, 8 up, 8 in 2021-02-10T04:30:36.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[35961]: cluster 2021-02-10T04:30:35.312965+0000 2021-02-10T04:30:36.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[35961]: osd.4 (osd.4) 85 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:36.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[35961]: cluster 2021-02-10T04:30 2021-02-10T04:30:36.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[35961]: :35.312974+0000 osd.4 (osd.4) 2021-02-10T04:30:36.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[35961]: 86 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:36.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:30:36.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[35961]: T04:30:35.312979+0000 osd.4 (osd 2021-02-10T04:30:36.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[35961]: .4) 87 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:36.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:30:36.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[35961]: -02-10T04:30:35.312983+0000 2021-02-10T04:30:36.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[35961]: osd.4 (osd.4) 88 : cluster [WRN] 2021-02-10T04:30:36.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:36.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:30:36.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[35961]: 30:36.020949+0000 mgr.y (mgr.14140 2021-02-10T04:30:36.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[35961]: ) 408 : cluster [DBG] pgmap v478: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim, 162 active+clean; 129 MiB data, 458 MiB used, 715 GiB / 715 GiB avail; 17 MiB/s rd, 4.1 MiB/s wr, 21 op/s 2021-02-10T04:30:36.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[29319]: cluster 2021-02-10T04:30:36.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[29319]: 2021-02-10T04:30:36.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[29319]: 2021 2021-02-10T04:30:36.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[29319]: - 2021-02-10T04:30:36.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[29319]: 02 2021-02-10T04:30:36.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[29319]: -10T04:30:35.198716+0000 2021-02-10T04:30:36.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[29319]: mon.a (mon.0) 934 : cluster [DBG] osdmap e219: 8 total, 8 up, 8 in 2021-02-10T04:30:36.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[29319]: cluster 2021-02-10T04:30:35.312965+0000 osd.4 (osd.4) 85 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:36.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[29319]: cluster 2021-02-10T04:30:35.312974+0000 osd.4 (osd.4) 86 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:36.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[29319]: cluster 2021-02-10T04:30:35.312979+0000 osd.4 (osd.4) 87 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:36.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:30:36.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[29319]: 30:35.312983+0000 osd.4 (osd.4 2021-02-10T04:30:36.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[29319]: ) 88 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:36.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[29319]: cluster 2021-02-10T04:30:36.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[29319]: 2021-02-10T04:30:36.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[29319]: 2021-02-10T04:30:36.020949+0000 mgr.y ( 2021-02-10T04:30:36.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:36 smithi085 conmon[29319]: mgr.14140) 408 : cluster [DBG] pgmap v478: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim, 162 active+clean; 129 MiB data, 458 MiB used, 715 GiB / 715 GiB avail; 17 MiB/s rd, 4.1 MiB/s wr, 21 op/s 2021-02-10T04:30:36.873 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.7 dump_blocked_ops 2021-02-10T04:30:37.160 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:30:37.215 INFO:tasks.rados.rados.0.smithi085.stdout:634: done (10 left) 2021-02-10T04:30:37.215 INFO:tasks.rados.rados.0.smithi085.stdout:656: done (9 left) 2021-02-10T04:30:37.215 INFO:tasks.rados.rados.0.smithi085.stdout:657: read oid 10 snap -1 2021-02-10T04:30:37.215 INFO:tasks.rados.rados.0.smithi085.stdout:657: expect deleted 2021-02-10T04:30:37.215 INFO:tasks.rados.rados.0.smithi085.stdout:658: write oid 38 current snap is 63 2021-02-10T04:30:37.215 INFO:tasks.rados.rados.0.smithi085.stdout:658: seq_num 260 ranges {769820=720742,2114730=255090} 2021-02-10T04:30:37.215 INFO:tasks.rados.rados.0.smithi085.stdout:658: writing smithi08575397-38 from 769820 to 1490562 tid 1 2021-02-10T04:30:37.220 INFO:tasks.rados.rados.0.smithi085.stdout:658: writing smithi08575397-38 from 2114730 to 2369820 tid 2 2021-02-10T04:30:37.220 INFO:tasks.rados.rados.0.smithi085.stdout:659: delete oid 40 current snap is 63 2021-02-10T04:30:37.222 INFO:tasks.rados.rados.0.smithi085.stdout:658: finishing write tid 1 to smithi08575397-38 2021-02-10T04:30:37.224 INFO:tasks.rados.rados.0.smithi085.stdout:658: finishing write tid 2 to smithi08575397-38 2021-02-10T04:30:37.224 INFO:tasks.rados.rados.0.smithi085.stdout:657: done (11 left) 2021-02-10T04:30:37.224 INFO:tasks.rados.rados.0.smithi085.stdout:659: done (10 left) 2021-02-10T04:30:37.225 INFO:tasks.rados.rados.0.smithi085.stdout:660: rollback oid 24 current snap is 63 2021-02-10T04:30:37.225 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 24 to 60 2021-02-10T04:30:37.225 INFO:tasks.rados.rados.0.smithi085.stdout:661: snap_remove snap 62 2021-02-10T04:30:37.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:37 smithi164 conmon[29288]: cluster 2021-02-10T04:30:37.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:37 smithi164 conmon[29288]: 2021-02-10T04:30:37.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:37 smithi164 conmon[29288]: 2021-02-10T04:30 2021-02-10T04:30:37.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:37 smithi164 conmon[29288]: :36.203886+0000 mon.a (mon.0) 935 : cluster [DBG] osdmap e220: 8 total, 8 up, 8 in 2021-02-10T04:30:37.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:37 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:30:37.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:37 smithi164 conmon[29288]: :30:36.341070+0000 osd.4 (osd.4) 89 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:37.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:37 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:30:37.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:37 smithi164 conmon[29288]: -10T04:30:36.341076+0000 osd.4 2021-02-10T04:30:37.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:37 smithi164 conmon[29288]: (osd.4) 90 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:37.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:37 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:30:37.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:37 smithi164 conmon[29288]: T04:30:36.341080+0000 osd.4 (osd.4 2021-02-10T04:30:37.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:37 smithi164 conmon[29288]: ) 91 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:37.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:37 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:30:37.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:37 smithi164 conmon[29288]: 10T04:30:36.341084+0000 osd.4 ( 2021-02-10T04:30:37.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:37 smithi164 conmon[29288]: osd.4) 92 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:37.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:37 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:30:37.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:37 smithi164 conmon[29288]: 02-10T04:30:36.732530+0000 2021-02-10T04:30:37.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:37 smithi164 conmon[29288]: osd.7 (osd.7) 42 : cluster [DBG] 2021-02-10T04:30:37.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:37 smithi164 conmon[29288]: purged_snaps scrub starts 2021-02-10T04:30:37.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:37 smithi164 conmon[29288]: cluster 2021-02-10T04:30: 2021-02-10T04:30:37.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:37 smithi164 conmon[29288]: 36.732636+0000 osd.7 (osd.7) 43 : 2021-02-10T04:30:37.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:37 smithi164 conmon[29288]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:30:37.471 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:30:37 smithi164 conmon[33677]: debug 2021-02-10T04:30:37.342+0000 7fbec19a1700 -1 osd.4 221 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:30:37.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[29319]: cluster 2021-02-10T04:30:36.203886+0000 mon.a (mon.0 2021-02-10T04:30:37.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[29319]: ) 935 : cluster [DBG] osdmap e220: 8 total, 8 up, 8 in 2021-02-10T04:30:37.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[29319]: cluster 2021-02-10T04:30:36.341070+0000 osd.4 (osd.4) 89 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:37.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[29319]: cluster 2021-02-10T04:30:36.341076+0000 osd.4 (osd. 2021-02-10T04:30:37.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[29319]: 4) 90 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:37.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:30:37.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[29319]: -10T04:30:36.341080+0000 osd.4 ( 2021-02-10T04:30:37.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[29319]: osd.4) 91 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:37.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:30:37.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[29319]: 02-10T04:30:36.341084+0000 osd.4 2021-02-10T04:30:37.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[29319]: (osd.4) 92 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:37.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[29319]: 2021-02-10T04:30:37.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[29319]: cluster 2021-02-10T04:30: 2021-02-10T04:30:37.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[29319]: 36.732530+0000 osd.7 (osd.7) 42 : 2021-02-10T04:30:37.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[29319]: cluster [DBG] purged_snaps scrub starts 2021-02-10T04:30:37.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:30:37.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[29319]: -10T04:30:36.732636+0000 osd.7 2021-02-10T04:30:37.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[29319]: (osd.7) 43 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:30:37.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[29319]: 2021-02-10T04:30:37.685 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[35961]: cluster 2021-02-10T04:30:36.203886+0000 2021-02-10T04:30:37.685 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[35961]: mon.a (mon.0) 935 : cluster [DBG] osdmap e220: 8 total, 8 up, 8 in 2021-02-10T04:30:37.685 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:30:37.686 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[35961]: 04:30:36.341070+0000 osd.4 (osd.4) 89 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:37.686 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[35961]: 2021-02-10T04:30:37.686 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[35961]: cluster 2021-02-10T04:30:37.686 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[35961]: 2021-02-10T04:30:36.341076+0000 osd.4 (osd. 2021-02-10T04:30:37.686 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[35961]: 4) 90 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:37.686 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:30:37.686 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[35961]: 10T04:30:36.341080+0000 osd.4 ( 2021-02-10T04:30:37.686 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[35961]: osd.4) 91 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:37.686 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:30:37.686 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[35961]: 02-10T04:30:36.341084+0000 2021-02-10T04:30:37.686 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[35961]: osd.4 (osd.4) 92 : cluster [WRN] 2021-02-10T04:30:37.686 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:37.687 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[35961]: cluster 2021-02-10T04:30:37.687 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[35961]: 2021-02-10T04:30:36.732530+0000 osd.7 ( 2021-02-10T04:30:37.687 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[35961]: osd.7) 42 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:30:37.687 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:30:37.687 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[35961]: 02-10T04:30:36.732636+0000 osd.7 2021-02-10T04:30:37.687 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[35961]: (osd.7) 43 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:30:37.687 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[35961]: 2021-02-10T04:30:37.687 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:30:37 smithi085 conmon[41540]: debug 2021-02-10T04:30:37.683+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:37.741 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:30:37.849 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 5 1 2021-02-10T04:30:37.922 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:30:37.968 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:38.116 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:30:38.143 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:30:37 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:30:38.214 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:30:38.215 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[29319]: cluster 2021-02-10T04:30:37.205579+0000 mon.a (mon.0) 936 2021-02-10T04:30:38.215 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[29319]: : cluster [DBG] osdmap e221: 8 total, 8 up, 8 in 2021-02-10T04:30:38.215 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[29319]: cluster 2021-02-10T04:30:37.343594+0000 osd.4 (osd.4) 93 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:38.215 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[29319]: cluster 2021-02-10T04:30:37.343610+0000 osd.4 (osd.4) 94 2021-02-10T04:30:38.216 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:38.216 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[29319]: cluster 2021-02-10T04:30:37.343624+0000 osd.4 (osd.4) 95 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:38.216 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[29319]: 2021-02-10T04:30:38.216 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[29319]: cluster 2021-02-10T04:30:38.216 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[29319]: 2021-02-10T04:30:37.343630+0000 osd.4 (osd 2021-02-10T04:30:38.216 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[29319]: .4) 96 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:38.216 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:30:38.216 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[29319]: T04:30:37.795033+0000 mgr.y (mgr.14140) 409 : audit 2021-02-10T04:30:38.217 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[29319]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:30:38.217 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[29319]: cluster 2021-02-10T04:30:38 2021-02-10T04:30:38.217 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[29319]: .021897+0000 mgr.y (mgr.14140) 410 : cluster [DBG] pgmap v481: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim, 162 active+clean; 131 MiB data, 452 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 4.3 MiB/s wr, 17 op/s 2021-02-10T04:30:38.217 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[29319]: audit 2021-02-10T04:30:38. 2021-02-10T04:30:38.217 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[29319]: 056770+0000 mon.a (mon.0) 937 : audit 2021-02-10T04:30:38.217 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[29319]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:30:38.217 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[29319]: audit 2021-02-10T04:30 2021-02-10T04:30:38.217 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[29319]: :38.060044+0000 mon.a (mon.0) 938 : audit 2021-02-10T04:30:38.217 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[29319]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:30:38.217 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[45443]: debug 2021-02-10T04:30:38.217 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[45443]: 2021-02-10T04:30:38.078+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:38.218 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: cluster 2021-02-10T04:30:38.218 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: 2021-02-10T04:30:38.218 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: 2021 2021-02-10T04:30:38.218 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: - 2021-02-10T04:30:38.218 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: 02 2021-02-10T04:30:38.218 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: - 2021-02-10T04:30:38.218 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: 10 2021-02-10T04:30:38.218 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: T 2021-02-10T04:30:38.219 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: 04 2021-02-10T04:30:38.219 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: : 2021-02-10T04:30:38.219 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: 30 2021-02-10T04:30:38.219 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: : 2021-02-10T04:30:38.219 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: 37 2021-02-10T04:30:38.219 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: . 2021-02-10T04:30:38.219 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: 205579 2021-02-10T04:30:38.219 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: +0000 mon.a (mon. 2021-02-10T04:30:38.219 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: 0) 936 : cluster [DBG] osdmap e221: 8 total, 8 up, 8 in 2021-02-10T04:30:38.219 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:30:38.220 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: 02-10T04:30:37.343594+0000 osd.4 2021-02-10T04:30:38.220 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: (osd.4) 93 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:38.220 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: 2021-02-10T04:30:38.220 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: cluster 2021-02-10T04:30:38.220 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: 2021-02-10T04:30:37.343610+0000 osd.4 2021-02-10T04:30:38.220 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: (osd.4) 94 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:38.220 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: cluster 2021-02-10T04:30 2021-02-10T04:30:38.220 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: :37.343624+0000 osd.4 (osd.4) 95 : 2021-02-10T04:30:38.220 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:38.220 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:30:38.220 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: :30:37.343630+0000 osd.4 (osd.4 2021-02-10T04:30:38.220 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: ) 96 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:38.220 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:30:38.221 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: 10T04:30:37.795033+0000 mgr.y ( 2021-02-10T04:30:38.221 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: mgr.14140) 409 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:30:38.221 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:30:38.221 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: 02- 2021-02-10T04:30:38.221 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: 10T04:30:38.021897+0000 2021-02-10T04:30:38.221 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: mgr.y (mgr.14140) 410 : cluster [DBG] pgmap v481: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim, 162 active+clean; 131 MiB data, 452 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 4.3 MiB/s wr, 17 op/s 2021-02-10T04:30:38.221 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: 2021-02-10T04:30:38.221 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: audit 2021-02-10T04:30 2021-02-10T04:30:38.221 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: :38.056770+0000 2021-02-10T04:30:38.221 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: mon.a (mon.0) 937 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:30:38.221 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: 2021-02-10T04:30:38.221 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: audit 2021-02-10T04:30:38.221 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: 2021-02-10T04:30:38.060044+0000 mon.a 2021-02-10T04:30:38.222 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: (mon.0) 938 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:30:38.222 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[35961]: 2021-02-10T04:30:38.223 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:30:38.224 INFO:tasks.rados.rados.0.smithi085.stdout:661: done (11 left) 2021-02-10T04:30:38.224 INFO:tasks.rados.rados.0.smithi085.stdout:662: read oid 27 snap -1 2021-02-10T04:30:38.224 INFO:tasks.rados.rados.0.smithi085.stdout:662: expect (ObjNum 232 snap 53 seq_num 232) 2021-02-10T04:30:38.224 INFO:tasks.rados.rados.0.smithi085.stdout:663: snap_remove snap 61 2021-02-10T04:30:38.265 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:38.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:30:38 smithi164 conmon[33677]: debug 2021-02-10T04:30:38.316+0000 7fbec19a1700 -1 osd.4 222 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:30:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:38 smithi164 conmon[29288]: cluster 2021-02-10T04:30:37.205579+0000 mon.a (mon. 2021-02-10T04:30:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:38 smithi164 conmon[29288]: 0) 936 : cluster [DBG] osdmap e221: 8 total, 8 up, 8 in 2021-02-10T04:30:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:38 smithi164 conmon[29288]: cluster 2021-02-10T04:30:37.343594+0000 osd.4 (osd.4) 93 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:38 smithi164 conmon[29288]: cluster 2021-02-10T04:30:37.343610+0000 osd.4 (osd.4 2021-02-10T04:30:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:38 smithi164 conmon[29288]: ) 94 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:38 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:30:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:38 smithi164 conmon[29288]: 10T04:30:37.343624+0000 osd.4 (osd 2021-02-10T04:30:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:38 smithi164 conmon[29288]: .4) 95 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:38 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:30:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:38 smithi164 conmon[29288]: 02-10T04:30:37.343630+0000 osd.4 ( 2021-02-10T04:30:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:38 smithi164 conmon[29288]: osd.4) 96 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:38 smithi164 conmon[29288]: audit 2021-02-10T04:30:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:38 smithi164 conmon[29288]: 2021-02-10T04:30:37.795033+0000 2021-02-10T04:30:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:38 smithi164 conmon[29288]: mgr.y (mgr.14140) 409 : audit [DBG] 2021-02-10T04:30:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:38 smithi164 conmon[29288]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:30:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:38 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:30:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:38 smithi164 conmon[29288]: 30:38.021897+0000 mgr.y (mgr.14140) 2021-02-10T04:30:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:38 smithi164 conmon[29288]: 410 : cluster [DBG] pgmap v481: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim, 162 active+clean; 131 MiB data, 452 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 4.3 MiB/s wr, 17 op/s 2021-02-10T04:30:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:38 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:30:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:38 smithi164 conmon[29288]: -10T04:30:38.056770+0000 mon.a 2021-02-10T04:30:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:38 smithi164 conmon[29288]: (mon.0) 937 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:30:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:38 smithi164 conmon[29288]: audit 2021-02-10T04:30:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:38 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:30:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:38 smithi164 conmon[29288]: :30:38.060044+0000 mon.a (mon. 2021-02-10T04:30:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:38 smithi164 conmon[29288]: 0) 938 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:30:38.883 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:30:38.937 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[52836]: debug 2021-02-10T04:30:38.611+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:38.937 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:38 smithi085 conmon[45443]: debug 2021-02-10T04:30:38.937+0000 7f7121cf0700 -1 osd.1 pg_epoch: 222 pg[4.4( v 63'31 (0'0,63'31] local-lis/les=55/56 n=14 ec=55/55 lis/c=55/55 les/c/f=56/56/0 sis=55) [1,2,3] r=0 lpr=55 crt=63'31 lcod 63'30 mlcod 63'30 active+clean] scrubber pg(4.4) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 3 (0x55dbc85f22c0) 2021-02-10T04:30:39.222 INFO:tasks.rados.rados.0.smithi085.stdout:663: done (12 left) 2021-02-10T04:30:39.223 INFO:tasks.rados.rados.0.smithi085.stdout:664: read oid 31 snap -1 2021-02-10T04:30:39.223 INFO:tasks.rados.rados.0.smithi085.stdout:664: expect (ObjNum 180 snap 40 seq_num 180) 2021-02-10T04:30:39.223 INFO:tasks.rados.rados.0.smithi085.stdout:665: copy_from oid 9 from oid 10 current snap is 63 2021-02-10T04:30:39.223 INFO:tasks.rados.rados.0.smithi085.stdout:658: finishing write tid 3 to smithi08575397-38 2021-02-10T04:30:39.223 INFO:tasks.rados.rados.0.smithi085.stdout:658: finishing write tid 4 to smithi08575397-38 2021-02-10T04:30:39.223 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 197 (ObjNum 260 snap 63 seq_num 260) dirty exists 2021-02-10T04:30:39.223 INFO:tasks.rados.rados.0.smithi085.stdout:658: left oid 38 (ObjNum 260 snap 63 seq_num 260) 2021-02-10T04:30:39.223 INFO:tasks.rados.rados.0.smithi085.stdout:660: finishing rollback tid 1 to smithi08575397-24 2021-02-10T04:30:39.223 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 24 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-02-10T04:30:39.223 INFO:tasks.rados.rados.0.smithi085.stdout:662: expect (ObjNum 232 snap 53 seq_num 232) 2021-02-10T04:30:39.401 INFO:tasks.rados.rados.0.smithi085.stdout:658: done (13 left) 2021-02-10T04:30:39.401 INFO:tasks.rados.rados.0.smithi085.stdout:660: done (12 left) 2021-02-10T04:30:39.401 INFO:tasks.rados.rados.0.smithi085.stdout:662: done (11 left) 2021-02-10T04:30:39.402 INFO:tasks.rados.rados.0.smithi085.stdout:666: copy_from oid 13 from oid 3 current snap is 63 2021-02-10T04:30:39.402 INFO:tasks.rados.rados.0.smithi085.stdout:664: expect (ObjNum 180 snap 40 seq_num 180) 2021-02-10T04:30:39.541 INFO:tasks.rados.rados.0.smithi085.stdout:664: done (11 left) 2021-02-10T04:30:39.541 INFO:tasks.rados.rados.0.smithi085.stdout:667: copy_from oid 41 from oid 40 current snap is 63 2021-02-10T04:30:39.541 INFO:tasks.rados.rados.0.smithi085.stdout:666: finishing copy_from to smithi08575397-13 2021-02-10T04:30:39.541 INFO:tasks.rados.rados.0.smithi085.stdout:666: got expected ENOENT (src dne) 2021-02-10T04:30:39.541 INFO:tasks.rados.rados.0.smithi085.stdout:666: finishing copy_from racing read to smithi08575397-13 2021-02-10T04:30:39.541 INFO:tasks.rados.rados.0.smithi085.stdout:666: done (11 left) 2021-02-10T04:30:39.541 INFO:tasks.rados.rados.0.smithi085.stdout:668: write (excl) oid 36 current snap is 63 2021-02-10T04:30:39.541 INFO:tasks.rados.rados.0.smithi085.stdout:668: seq_num 262 ranges {403434=471240,1315341=400756} 2021-02-10T04:30:39.545 INFO:tasks.rados.rados.0.smithi085.stdout:668: writing smithi08575397-36 from 403434 to 874674 tid 1 2021-02-10T04:30:39.551 INFO:tasks.rados.rados.0.smithi085.stdout:668: writing smithi08575397-36 from 1315341 to 1716097 tid 2 2021-02-10T04:30:39.551 INFO:tasks.rados.rados.0.smithi085.stdout:669: snap_create 2021-02-10T04:30:39.552 INFO:tasks.rados.rados.0.smithi085.stdout:668: finishing write tid 1 to smithi08575397-36 2021-02-10T04:30:39.554 INFO:tasks.rados.rados.0.smithi085.stdout:668: finishing write tid 2 to smithi08575397-36 2021-02-10T04:30:39.554 INFO:tasks.rados.rados.0.smithi085.stdout:668: finishing write tid 3 to smithi08575397-36 2021-02-10T04:30:39.555 INFO:tasks.rados.rados.0.smithi085.stdout:668: finishing write tid 4 to smithi08575397-36 2021-02-10T04:30:39.555 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 104 (ObjNum 262 snap 63 seq_num 262) dirty exists 2021-02-10T04:30:39.555 INFO:tasks.rados.rados.0.smithi085.stdout:668: left oid 36 (ObjNum 262 snap 63 seq_num 262) 2021-02-10T04:30:39.636 INFO:tasks.rados.rados.0.smithi085.stdout:667: finishing copy_from to smithi08575397-41 2021-02-10T04:30:39.636 INFO:tasks.rados.rados.0.smithi085.stdout:667: got expected ENOENT (src dne) 2021-02-10T04:30:39.636 INFO:tasks.rados.rados.0.smithi085.stdout:667: finishing copy_from racing read to smithi08575397-41 2021-02-10T04:30:39.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:30:39 smithi164 conmon[33677]: debug 2021-02-10T04:30:39.309+0000 7fbec19a1700 -1 osd.4 223 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:30:39.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:39 smithi164 conmon[29288]: cluster 2021-02-10T04:30:38.223760+0000 mon.a (mon.0) 2021-02-10T04:30:39.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:39 smithi164 conmon[29288]: 939 : cluster [DBG] osdmap e222: 8 total, 8 up, 8 in 2021-02-10T04:30:39.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:39 smithi164 conmon[29288]: cluster 2021-02-10T04:30:38.317023+0000 osd.4 (osd.4) 97 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:39.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:39 smithi164 conmon[29288]: cluster 2021-02-10T04:30:38.317029+0000 osd.4 (osd.4) 98 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:39.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:39 smithi164 conmon[29288]: cluster 2021-02-10T04:30:38.317033+0000 osd.4 (osd.4) 99 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:39 smithi164 conmon[29288]: cluster 2021-02-10T04:30:38.317038+0000 osd.4 ( 2021-02-10T04:30:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:39 smithi164 conmon[29288]: osd.4) 100 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:39 smithi085 conmon[35961]: cluster 2021-02-10T04:30:38.223760+0000 mon.a (mon.0 2021-02-10T04:30:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:39 smithi085 conmon[35961]: ) 939 : cluster [DBG] osdmap e222: 8 total, 8 up, 8 in 2021-02-10T04:30:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:39 smithi085 conmon[35961]: cluster 2021-02-10T04:30:38. 2021-02-10T04:30:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:39 smithi085 conmon[35961]: 317023+0000 osd.4 (osd.4) 97 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:39 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:30:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:39 smithi085 conmon[35961]: :30:38.317029+0000 osd.4 (osd.4 2021-02-10T04:30:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:39 smithi085 conmon[35961]: ) 98 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:39 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:30:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:39 smithi085 conmon[35961]: 02-10T04:30:38.317033+0000 osd.4 ( 2021-02-10T04:30:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:39 smithi085 conmon[35961]: osd.4) 99 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:39 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:30:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:39 smithi085 conmon[35961]: 30:38.317038+0000 osd.4 (osd.4) 100 : cluster 2021-02-10T04:30:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:39 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:39.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:39 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:30:39.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:39 smithi085 conmon[29319]: -10T04:30:38.223760+0000 mon.a (mon.0) 939 : cluster [DBG] osdmap e222: 8 total, 8 up, 8 in 2021-02-10T04:30:39.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:39 smithi085 conmon[29319]: cluster 2021-02-10T04:30:39.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:39 smithi085 conmon[29319]: 2021-02-10T04:30:38.317023+0000 osd.4 (osd.4) 97 2021-02-10T04:30:39.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:39 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:39.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:39 smithi085 conmon[29319]: cluster 2021-02-10T04:30:39.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:39 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:30:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:39 smithi085 conmon[29319]: 30:38.317029+0000 osd.4 (osd.4) 98 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:39 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:30:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:39 smithi085 conmon[29319]: 04:30:38.317033+0000 osd.4 ( 2021-02-10T04:30:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:39 smithi085 conmon[29319]: osd.4) 99 : cluster [WRN] 2021-02-10T04:30:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:39 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:39 smithi085 conmon[29319]: cluster 2021-02-10T04:30:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:39 smithi085 conmon[29319]: 2021-02-10T04:30:38.317038 2021-02-10T04:30:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:39 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 100 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:39 smithi085 conmon[29319]: 2021-02-10T04:30:39.896 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set noscrub 2021-02-10T04:30:40.182 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:30:40.244 INFO:tasks.rados.rados.0.smithi085.stdout:667: done (12 left) 2021-02-10T04:30:40.244 INFO:tasks.rados.rados.0.smithi085.stdout:668: done (11 left) 2021-02-10T04:30:40.244 INFO:tasks.rados.rados.0.smithi085.stdout:669: done (10 left) 2021-02-10T04:30:40.244 INFO:tasks.rados.rados.0.smithi085.stdout:670: write (excl) oid 36 current snap is 64 2021-02-10T04:30:40.244 INFO:tasks.rados.rados.0.smithi085.stdout:670: seq_num 263 ranges {420241=546489,1373823=544414} 2021-02-10T04:30:40.247 INFO:tasks.rados.rados.0.smithi085.stdout:670: writing smithi08575397-36 from 420241 to 966730 tid 1 2021-02-10T04:30:40.255 INFO:tasks.rados.rados.0.smithi085.stdout:670: writing smithi08575397-36 from 1373823 to 1918237 tid 2 2021-02-10T04:30:40.255 INFO:tasks.rados.rados.0.smithi085.stdout:671: snap_remove snap 60 2021-02-10T04:30:40.276 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:30:40.276 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2021-02-10T04:30:40.277 INFO:teuthology.orchestra.run.smithi164.stdout: "complaint_time": 30, 2021-02-10T04:30:40.277 INFO:teuthology.orchestra.run.smithi164.stdout: "num_blocked_ops": 0 2021-02-10T04:30:40.277 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:30:40.300 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:40 smithi164 conmon[29288]: cluster 2021-02-10T04:30:40.300 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:40 smithi164 conmon[29288]: 2021-02-10T04:30:40.300 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:40 smithi164 conmon[29288]: 2021 2021-02-10T04:30:40.300 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:40 smithi164 conmon[29288]: -02-10T04:30:39.222011 2021-02-10T04:30:40.300 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:40 smithi164 conmon[29288]: +0000 mon.a (mon.0) 940 : cluster [DBG] osdmap e223: 8 total, 8 up, 8 in 2021-02-10T04:30:40.300 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:40 smithi164 conmon[29288]: cluster 2021-02-10T04:30:39. 2021-02-10T04:30:40.300 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:40 smithi164 conmon[29288]: 309762+0000 osd.4 (osd.4) 101 : cluster 2021-02-10T04:30:40.301 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:40 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:40.301 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:40 smithi164 conmon[29288]: cluster 2021-02-10T04:30:39. 2021-02-10T04:30:40.301 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:40 smithi164 conmon[29288]: 309771+0000 osd.4 (osd.4) 102 : cluster [WRN] 2021-02-10T04:30:40.301 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:40 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:40.301 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:40 smithi164 conmon[29288]: cluster 2021-02-10T04:30:39.309780 2021-02-10T04:30:40.301 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:40 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 103 : cluster [WRN] 2021-02-10T04:30:40.301 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:40 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:40.301 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:40 smithi164 conmon[29288]: cluster 2021-02-10T04:30:39.309786+0000 osd.4 (osd.4) 104 : 2021-02-10T04:30:40.301 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:40 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:40.301 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:40 smithi164 conmon[29288]: cluster 2021-02-10T04:30:40.301 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:40 smithi164 conmon[29288]: 2021-02-10T04:30:40.022360+0000 mgr.y ( 2021-02-10T04:30:40.301 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:40 smithi164 conmon[29288]: mgr.14140) 411 : cluster [DBG] pgmap v484: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 164 active+clean; 133 MiB data, 446 MiB used, 715 GiB / 715 GiB avail; 7.3 MiB/s rd, 3.0 MiB/s wr, 9 op/s 2021-02-10T04:30:40.302 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:30:40 smithi164 conmon[33677]: debug 2021-02-10T04:30:40.286+0000 7fbec19a1700 -1 osd.4 224 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:30:40.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[29319]: cluster 2021-02-10T04:30:39.222011+0000 mon.a 2021-02-10T04:30:40.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[29319]: (mon.0) 940 : cluster [DBG] osdmap e223: 8 total, 8 up, 8 in 2021-02-10T04:30:40.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[29319]: cluster 2021-02-10T04:30:39.309762+0000 osd.4 (osd.4) 101 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:40.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[29319]: cluster 2021-02-10T04:30:39.309771+0000 osd.4 (osd 2021-02-10T04:30:40.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[29319]: .4) 102 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:40.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:30:40.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[29319]: T04:30:39.309780+0000 osd.4 (osd 2021-02-10T04:30:40.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[29319]: .4) 103 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:40.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:30:40.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[29319]: -10T04:30:39.309786+0000 osd.4 (osd 2021-02-10T04:30:40.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[29319]: .4) 104 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:40.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:30:40.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[29319]: 30:40.022360+0000 mgr.y (mgr.14140) 411 : 2021-02-10T04:30:40.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[29319]: cluster [DBG] pgmap v484: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 164 active+clean; 133 MiB data, 446 MiB used, 715 GiB / 715 GiB avail; 7.3 MiB/s rd, 3.0 MiB/s wr, 9 op/s 2021-02-10T04:30:40.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[35961]: cluster 2021-02-10T04:30:40.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:30:40.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[35961]: 10T04:30:39.222011+0000 mon.a (mon.0) 940 : cluster [DBG] osdmap e223: 8 total, 8 up, 8 in 2021-02-10T04:30:40.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:30:40.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[35961]: -10T04:30:39.309762+0000 osd.4 (osd.4 2021-02-10T04:30:40.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[35961]: ) 101 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:40.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:30:40.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[35961]: -10T04:30:39.309771+0000 osd.4 ( 2021-02-10T04:30:40.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[35961]: osd.4) 102 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:40.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:30:40.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[35961]: -02-10T04:30:39.309780+0000 2021-02-10T04:30:40.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[35961]: osd.4 (osd.4) 103 : cluster [WRN] 2021-02-10T04:30:40.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:40.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[35961]: cluster 2021-02-10T04:30:39. 2021-02-10T04:30:40.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[35961]: 309786+0000 osd.4 (osd.4) 104 : cluster 2021-02-10T04:30:40.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:40.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[35961]: cluster 2021-02-10T04:30:40.022360+0000 mgr.y 2021-02-10T04:30:40.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:40 smithi085 conmon[35961]: (mgr.14140) 411 : cluster [DBG] pgmap v484: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 164 active+clean; 133 MiB data, 446 MiB used, 715 GiB / 715 GiB avail; 7.3 MiB/s rd, 3.0 MiB/s wr, 9 op/s 2021-02-10T04:30:41.244 INFO:tasks.rados.rados.0.smithi085.stdout:671: done (11 left) 2021-02-10T04:30:41.244 INFO:tasks.rados.rados.0.smithi085.stdout:672: write oid 39 current snap is 64 2021-02-10T04:30:41.244 INFO:tasks.rados.rados.0.smithi085.stdout:670: finishing write tid 1 to smithi08575397-36 2021-02-10T04:30:41.244 INFO:tasks.rados.rados.0.smithi085.stdout:672: seq_num 264 ranges {437048=621738,1832305=404425} 2021-02-10T04:30:41.250 INFO:tasks.rados.rados.0.smithi085.stdout:672: writing smithi08575397-39 from 437048 to 1058786 tid 1 2021-02-10T04:30:41.258 INFO:tasks.rados.rados.0.smithi085.stdout:672: writing smithi08575397-39 from 1832305 to 2236730 tid 2 2021-02-10T04:30:41.258 INFO:tasks.rados.rados.0.smithi085.stdout:673: delete oid 31 current snap is 64 2021-02-10T04:30:41.258 INFO:tasks.rados.rados.0.smithi085.stdout:670: finishing write tid 2 to smithi08575397-36 2021-02-10T04:30:41.258 INFO:tasks.rados.rados.0.smithi085.stdout:670: finishing write tid 3 to smithi08575397-36 2021-02-10T04:30:41.258 INFO:tasks.rados.rados.0.smithi085.stdout:670: finishing write tid 4 to smithi08575397-36 2021-02-10T04:30:41.258 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 107 (ObjNum 263 snap 64 seq_num 263) dirty exists 2021-02-10T04:30:41.259 INFO:tasks.rados.rados.0.smithi085.stdout:670: left oid 36 (ObjNum 263 snap 64 seq_num 263) 2021-02-10T04:30:41.261 INFO:tasks.rados.rados.0.smithi085.stdout:672: finishing write tid 1 to smithi08575397-39 2021-02-10T04:30:41.262 INFO:tasks.rados.rados.0.smithi085.stdout:670: done (12 left) 2021-02-10T04:30:41.262 INFO:tasks.rados.rados.0.smithi085.stdout:673: done (11 left) 2021-02-10T04:30:41.262 INFO:tasks.rados.rados.0.smithi085.stdout:674: rollback oid 23 current snap is 64 2021-02-10T04:30:41.262 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 23 to 59 2021-02-10T04:30:41.262 INFO:tasks.rados.rados.0.smithi085.stdout:672: finishing write tid 2 to smithi08575397-39 2021-02-10T04:30:41.262 INFO:tasks.rados.rados.0.smithi085.stdout:675: write oid 45 current snap is 64 2021-02-10T04:30:41.262 INFO:tasks.rados.rados.0.smithi085.stdout:675: seq_num 266 ranges {470662=772236,1832916=408094} 2021-02-10T04:30:41.270 INFO:tasks.rados.rados.0.smithi085.stdout:675: writing smithi08575397-45 from 470662 to 1242898 tid 1 2021-02-10T04:30:41.277 INFO:tasks.rados.rados.0.smithi085.stdout:675: writing smithi08575397-45 from 1832916 to 2241010 tid 2 2021-02-10T04:30:41.277 INFO:tasks.rados.rados.0.smithi085.stdout:676: snap_create 2021-02-10T04:30:41.277 INFO:tasks.rados.rados.0.smithi085.stdout:672: finishing write tid 3 to smithi08575397-39 2021-02-10T04:30:41.277 INFO:tasks.rados.rados.0.smithi085.stdout:672: finishing write tid 4 to smithi08575397-39 2021-02-10T04:30:41.277 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 39 v 70 (ObjNum 264 snap 64 seq_num 264) dirty exists 2021-02-10T04:30:41.277 INFO:tasks.rados.rados.0.smithi085.stdout:672: left oid 39 (ObjNum 264 snap 64 seq_num 264) 2021-02-10T04:30:41.278 INFO:tasks.rados.rados.0.smithi085.stdout:674: finishing rollback tid 0 to smithi08575397-23 2021-02-10T04:30:41.278 INFO:tasks.rados.rados.0.smithi085.stdout:674: finishing rollback tid 1 to smithi08575397-23 2021-02-10T04:30:41.278 INFO:tasks.rados.rados.0.smithi085.stdout:674: finishing rollback tid 2 to smithi08575397-23 2021-02-10T04:30:41.278 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 110 (ObjNum 124 snap 23 seq_num 124) dirty exists 2021-02-10T04:30:41.278 INFO:tasks.rados.rados.0.smithi085.stdout:675: finishing write tid 1 to smithi08575397-45 2021-02-10T04:30:41.280 INFO:tasks.rados.rados.0.smithi085.stdout:675: finishing write tid 2 to smithi08575397-45 2021-02-10T04:30:41.280 INFO:tasks.rados.rados.0.smithi085.stdout:675: finishing write tid 3 to smithi08575397-45 2021-02-10T04:30:41.281 INFO:tasks.rados.rados.0.smithi085.stdout:675: finishing write tid 4 to smithi08575397-45 2021-02-10T04:30:41.281 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 45 v 121 (ObjNum 266 snap 64 seq_num 266) dirty exists 2021-02-10T04:30:41.281 INFO:tasks.rados.rados.0.smithi085.stdout:675: left oid 45 (ObjNum 266 snap 64 seq_num 266) 2021-02-10T04:30:41.523 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[52836]: debug 2021-02-10T04:30:41.345+0000 7fba73fb6700 -1 osd.3 pg_epoch: 225 pg[4.14( v 63'26 (0'0,63'26] local-lis/les=71/72 n=11 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,2,7] r=0 lpr=71 crt=63'26 lcod 63'25 mlcod 0'0 active+clean] scrubber pg(4.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55fec4876f20) 2021-02-10T04:30:41.524 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[29319]: cluster 2021-02-10T04:30:40.229937+0000 mon.a (mon 2021-02-10T04:30:41.524 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[29319]: .0) 941 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 87 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:30:41.524 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[29319]: cluster 2021-02-10T04:30:40.239038+0000 mon.a (mon.0) 942 : cluster [DBG] osdmap e224: 8 total, 8 up, 8 in 2021-02-10T04:30:41.524 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[29319]: cluster 2021-02-10T04:30:40.287544+0000 osd.4 (osd.4 2021-02-10T04:30:41.524 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[29319]: ) 105 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:41.524 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[29319]: cluster 2021-02-10T04:30:41.524 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[29319]: 2021-02-10T04:30:40.287555+0000 2021-02-10T04:30:41.524 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[29319]: osd.4 (osd.4) 106 : cluster [WRN] 2021-02-10T04:30:41.524 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:41.524 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:30:41.525 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[29319]: 30:40.287561+0000 osd.4 (osd.4) 107 : cluster 2021-02-10T04:30:41.525 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:41.525 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:30:41.525 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[29319]: 30:40.287567+0000 osd.4 ( 2021-02-10T04:30:41.525 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[29319]: osd.4) 108 : cluster [WRN] 2021-02-10T04:30:41.525 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:41.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[35961]: cluster 2021-02-10T04:30:40.229937+0000 mon.a (mon.0) 941 : cluster 2021-02-10T04:30:41.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[35961]: [WRN] Health check update: 4 slow ops, oldest one blocked for 87 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:30:41.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:30:41.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[35961]: 30:40.239038+0000 mon.a (mon.0) 942 : cluster 2021-02-10T04:30:41.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[35961]: [DBG] osdmap e224: 8 total, 8 up, 8 in 2021-02-10T04:30:41.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[35961]: cluster 2021-02-10T04:30:40.287544 2021-02-10T04:30:41.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 105 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:41.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[35961]: 2021-02-10T04:30:41.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[35961]: cluster 2021-02-10T04:30:41.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[35961]: 2021-02-10T04:30:40.287555+0000 osd.4 (osd 2021-02-10T04:30:41.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[35961]: .4) 106 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:41.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:30:41.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[35961]: 10T04:30:40.287561+0000 osd.4 (osd 2021-02-10T04:30:41.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[35961]: .4) 107 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:41.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:30:41.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[35961]: T04:30:40.287567+0000 osd.4 (osd. 2021-02-10T04:30:41.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[35961]: 4) 108 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:41.624 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:30:41.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:30:41 smithi164 conmon[33677]: debug 2021-02-10T04:30:41.290+0000 7fbec19a1700 -1 osd.4 225 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:30:41.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:41 smithi164 conmon[29288]: cluster 2021-02-10T04:30:40.229937+0000 mon.a 2021-02-10T04:30:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:41 smithi164 conmon[29288]: (mon.0) 941 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 87 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:30:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:41 smithi164 conmon[29288]: cluster 2021-02-10T04:30:40.239038+0000 mon.a (mon.0) 942 : cluster [DBG] osdmap e224: 8 total, 8 up, 8 in 2021-02-10T04:30:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:41 smithi164 conmon[29288]: cluster 2021-02-10T04:30:40.287544+0000 osd.4 (osd.4) 105 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:41 smithi164 conmon[29288]: cluster 2021-02-10T04:30:40.287555+0000 osd.4 (osd.4) 106 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:41 smithi164 conmon[29288]: cluster 2021-02-10T04:30:40.287561+0000 osd.4 (osd.4) 107 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:41 smithi164 conmon[29288]: cluster 2021-02-10T04:30:40.287567+0000 osd.4 (osd.4) 108 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:41.655 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:41.814 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.7 dump_historic_ops 2021-02-10T04:30:42.041 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:30:42.044 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:41 smithi085 conmon[45443]: debug 2021-02-10T04:30:41.855+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:42.055 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:30:42.146 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:30:42.174 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:42.252 INFO:tasks.rados.rados.0.smithi085.stdout:672: done (13 left) 2021-02-10T04:30:42.252 INFO:tasks.rados.rados.0.smithi085.stdout:674: done (12 left) 2021-02-10T04:30:42.252 INFO:tasks.rados.rados.0.smithi085.stdout:675: done (11 left) 2021-02-10T04:30:42.252 INFO:tasks.rados.rados.0.smithi085.stdout:676: done (10 left) 2021-02-10T04:30:42.252 INFO:tasks.rados.rados.0.smithi085.stdout:677: read oid 1 snap -1 2021-02-10T04:30:42.252 INFO:tasks.rados.rados.0.smithi085.stdout:677: expect (ObjNum 202 snap 44 seq_num 202) 2021-02-10T04:30:42.253 INFO:tasks.rados.rados.0.smithi085.stdout:678: snap_create 2021-02-10T04:30:42.265 INFO:tasks.rados.rados.0.smithi085.stdout:677: expect (ObjNum 202 snap 44 seq_num 202) 2021-02-10T04:30:42.333 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[29319]: cluster 2021-02-10T04:30:41.243132+0000 mon.a (mon 2021-02-10T04:30:42.333 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[29319]: .0) 943 : cluster [DBG] osdmap e225: 8 total, 8 up, 8 in 2021-02-10T04:30:42.333 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[29319]: cluster 2021-02-10T04:30:41.290809+0000 osd.4 (osd.4) 109 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:42.334 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[29319]: cluster 2021-02-10T04:30:41.290819+0000 osd.4 (osd.4) 110 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:42.334 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[29319]: cluster 2021-02-10T04:30:41.290825+0000 osd.4 (osd.4) 111 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:42.334 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[29319]: cluster 2021-02-10T04:30:41.290831+0000 osd.4 (osd.4) 112 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:42.334 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[29319]: cluster 2021-02-10T04:30:41.488465+0000 osd.4 (osd.4) 113 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:30:42.334 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[29319]: cluster 2021-02-10T04:30:41.488618+0000 osd.4 (osd.4) 114 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:30:42.334 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[29319]: cluster 2021-02-10T04:30:42.022969+0000 mgr.y (mgr.14140) 412 : cluster [DBG] pgmap v487: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim, 165 active+clean; 135 MiB data, 457 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 3.3 MiB/s wr, 13 op/s 2021-02-10T04:30:42.334 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[35961]: cluster 2021-02-10T04:30:42.334 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[35961]: 2021-02-10T04:30:42.334 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[35961]: 2021 2021-02-10T04:30:42.334 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[35961]: - 2021-02-10T04:30:42.335 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[35961]: 02 2021-02-10T04:30:42.335 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[35961]: - 2021-02-10T04:30:42.335 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[35961]: 10 2021-02-10T04:30:42.335 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[35961]: T04:30:41.243132+0000 mon.a 2021-02-10T04:30:42.335 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[35961]: (mon.0) 943 : cluster [DBG] osdmap e225: 8 total, 8 up, 8 in 2021-02-10T04:30:42.335 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:30:42.335 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[35961]: 04:30:41.290809+0000 osd.4 (osd.4) 109 : 2021-02-10T04:30:42.335 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:42.335 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:30:42.335 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[35961]: :30:41.290819+0000 osd.4 (osd.4) 110 : cluster 2021-02-10T04:30:42.335 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:42.335 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[35961]: cluster 2021-02-10T04:30 2021-02-10T04:30:42.335 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[35961]: :41.290825+0000 osd.4 (osd.4) 111 : 2021-02-10T04:30:42.336 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:42.336 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[35961]: cluster 2021-02-10T04:30 2021-02-10T04:30:42.336 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[35961]: :41.290831+0000 osd.4 (osd.4) 112 : cluster 2021-02-10T04:30:42.336 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:42.336 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:30:42.336 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[35961]: 30:41.488465+0000 osd.4 (osd.4) 113 : 2021-02-10T04:30:42.336 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[35961]: cluster [DBG] purged_snaps scrub starts 2021-02-10T04:30:42.336 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:30:42.336 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[35961]: 04:30:41.488618+0000 osd.4 (osd.4) 2021-02-10T04:30:42.336 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[35961]: 114 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:30:42.336 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:30:42.336 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[35961]: -10T04:30:42.022969+0000 mgr.y (mgr.14140) 412 : 2021-02-10T04:30:42.336 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[35961]: cluster [DBG] pgmap v487: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim, 165 active+clean; 135 MiB data, 457 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 3.3 MiB/s wr, 13 op/s 2021-02-10T04:30:42.365 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:30:42.470 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:30:42.508 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:42 smithi164 conmon[29288]: cluster 2021-02-10T04:30:41.243132 2021-02-10T04:30:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:42 smithi164 conmon[29288]: +0000 mon.a (mon.0) 943 : cluster [DBG] osdmap e225: 8 total, 8 up, 8 in 2021-02-10T04:30:42.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:42 smithi164 conmon[29288]: cluster 2021-02-10T04:30:41.290809+0000 osd.4 (osd.4) 109 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:42.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:42 smithi164 conmon[29288]: cluster 2021-02-10T04:30:41.290819+0000 osd.4 (osd.4) 110 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:42.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:42 smithi164 conmon[29288]: cluster 2021-02-10T04:30:41.290825+0000 osd.4 (osd.4) 111 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:42.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:42 smithi164 conmon[29288]: cluster 2021-02-10T04:30:41.290831+0000 osd.4 (osd.4) 112 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:42.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:42 smithi164 conmon[29288]: cluster 2021-02-10T04:30:42.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:42 smithi164 conmon[29288]: 2021-02-10T04:30:41.488465 2021-02-10T04:30:42.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:42 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 113 : 2021-02-10T04:30:42.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:42 smithi164 conmon[29288]: cluster [DBG] purged_snaps scrub starts 2021-02-10T04:30:42.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:42 smithi164 conmon[29288]: cluster 2021-02-10T04:30:42.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:42 smithi164 conmon[29288]: 2021-02-10T04:30:41 2021-02-10T04:30:42.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:42 smithi164 conmon[29288]: .488618+0000 osd.4 (osd.4) 114 : 2021-02-10T04:30:42.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:42 smithi164 conmon[29288]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:30:42.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:42 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:30:42.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:42 smithi164 conmon[29288]: -02-10T04:30:42.022969+0000 2021-02-10T04:30:42.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:42 smithi164 conmon[29288]: mgr.y (mgr.14140) 412 : cluster 2021-02-10T04:30:42.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:42 smithi164 conmon[29288]: [DBG] pgmap v487: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim, 165 active+clean; 135 MiB data, 457 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 3.3 MiB/s wr, 13 op/s 2021-02-10T04:30:42.647 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:30:42 smithi164 conmon[44544]: debug 2021-02-10T04:30:42.344+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:42.647 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:30:42 smithi164 conmon[33677]: debug 2021-02-10T04:30:42.324+0000 7fbec19a1700 -1 osd.4 226 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:30:42.941 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[45443]: debug 2021-02-10T04:30:42.941 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[45443]: 2021-02-10T04:30:42.590+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:42.941 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:42 smithi085 conmon[45443]: 2021-02-10T04:30:42.974 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:30:43.068 INFO:tasks.rados.rados.0.smithi085.stdout:677: done (11 left) 2021-02-10T04:30:43.068 INFO:tasks.rados.rados.0.smithi085.stdout:678: done (10 left) 2021-02-10T04:30:43.068 INFO:tasks.rados.rados.0.smithi085.stdout:679: snap_create 2021-02-10T04:30:43.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:43 smithi164 conmon[29288]: cluster 2021-02-10T04:30:42.251284+0000 mon.a (mon.0) 944 : cluster [DBG] osdmap e226: 8 total, 8 up, 8 in 2021-02-10T04:30:43.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:43 smithi164 conmon[29288]: cluster 2021-02-10T04:30:42.325268+0000 osd.4 (osd.4) 115 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:43.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:43 smithi164 conmon[29288]: cluster 2021-02-10T04:30:42.325278+0000 osd.4 (osd.4) 116 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:43.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:43 smithi164 conmon[29288]: cluster 2021-02-10T04:30:42.325286+0000 osd.4 (osd.4) 117 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:43.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:43 smithi164 conmon[29288]: cluster 2021-02-10T04:30:42.325294+0000 osd.4 (osd.4) 118 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:43.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:43 smithi164 conmon[29288]: cluster 2021-02-10T04:30:43.066722+0000 mon.a (mon.0) 945 : cluster [DBG] osdmap e227: 8 total, 8 up, 8 in 2021-02-10T04:30:43.645 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:30:43 smithi164 conmon[33677]: debug 2021-02-10T04:30:43.305+0000 7fbec19a1700 -1 osd.4 227 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:30:43.691 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:30:43 smithi085 conmon[52836]: debug 2021-02-10T04:30:43.335+0000 7fba6ffae700 -1 osd.3 pg_epoch: 227 pg[4.14( v 63'26 (0'0,63'26] local-lis/les=71/72 n=11 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,2,7] r=0 lpr=71 crt=63'26 lcod 63'25 mlcod 0'0 active+clean] scrubber pg(4.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55fec1398c60) 2021-02-10T04:30:43.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:43 smithi085 conmon[29319]: cluster 2021-02-10T04:30:42.251284+0000 mon.a (mon.0) 944 : cluster 2021-02-10T04:30:43.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:43 smithi085 conmon[29319]: [DBG] osdmap e226: 8 total, 8 up, 8 in 2021-02-10T04:30:43.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:43 smithi085 conmon[29319]: cluster 2021-02-10T04:30:42.325268+0000 osd.4 (osd.4) 115 2021-02-10T04:30:43.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:43 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:43.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:43 smithi085 conmon[29319]: cluster 2021-02-10T04:30:42.325278+0000 osd.4 (osd.4) 116 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:43.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:43 smithi085 conmon[29319]: cluster 2021-02-10T04:30:42 2021-02-10T04:30:43.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:43 smithi085 conmon[29319]: .325286+0000 osd.4 (osd.4) 117 : cluster [WRN] 2021-02-10T04:30:43.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:43 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:43.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:43 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:30:43.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:43 smithi085 conmon[29319]: 02-10T04:30:42.325294+0000 osd.4 (osd.4) 118 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:43.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:43 smithi085 conmon[29319]: cluster 2021-02-10T04:30:43.066722+0000 mon.a (mon. 2021-02-10T04:30:43.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:43 smithi085 conmon[29319]: 0) 945 : cluster [DBG] osdmap e227: 8 total, 8 up, 8 in 2021-02-10T04:30:43.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:43 smithi085 conmon[35961]: cluster 2021-02-10T04:30:42. 2021-02-10T04:30:43.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:43 smithi085 conmon[35961]: 251284+0000 mon.a (mon.0) 944 : cluster [DBG] osdmap e226: 8 total, 8 up, 8 in 2021-02-10T04:30:43.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:43 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:30:43.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:43 smithi085 conmon[35961]: 10T04:30:42.325268+0000 osd.4 (osd.4) 115 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:43.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:43 smithi085 conmon[35961]: cluster 2021-02-10T04:30:42.325278+0000 osd.4 ( 2021-02-10T04:30:43.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:43 smithi085 conmon[35961]: osd.4) 116 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:43.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:43 smithi085 conmon[35961]: cluster 2021-02-10T04:30:42.325286+0000 osd.4 (osd. 2021-02-10T04:30:43.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:43 smithi085 conmon[35961]: 4) 117 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:43.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:43 smithi085 conmon[35961]: cluster 2021-02-10T04:30:42.325294+0000 osd.4 (osd.4) 118 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:43.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:43 smithi085 conmon[35961]: cluster 2021-02-10T04:30:43 2021-02-10T04:30:43.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:43 smithi085 conmon[35961]: .066722+0000 mon.a (mon.0) 945 : cluster [DBG] osdmap e227: 8 total, 8 up, 8 in 2021-02-10T04:30:44.065 INFO:tasks.rados.rados.0.smithi085.stdout:679: done (10 left) 2021-02-10T04:30:44.066 INFO:tasks.rados.rados.0.smithi085.stdout:680: snap_create 2021-02-10T04:30:44.258 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:43 smithi085 conmon[45443]: debug 2021-02-10T04:30:44.258 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:43 smithi085 conmon[45443]: 2021-02-10T04:30:43.966+0000 7f7125cf8700 -1 osd.1 pg_epoch: 227 pg[4.4( v 63'31 (0'0,63'31] local-lis/les=55/56 n=14 ec=55/55 lis/c=55/55 les/c/f=56/56/0 sis=55) [1,2,3] r=0 lpr=55 crt=63'31 lcod 63'30 mlcod 63'30 active+clean] scrubber pg(4.4) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 3 (0x55dbc85f2840) 2021-02-10T04:30:44.479 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:30:44 smithi164 conmon[33677]: debug 2021-02-10T04:30:44.349+0000 7fbec19a1700 -1 osd.4 228 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:30:44.479 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:44 smithi164 conmon[29288]: cluster 2021-02-10T04:30:43.306658+0000 osd.4 (osd. 2021-02-10T04:30:44.479 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:44 smithi164 conmon[29288]: 4) 119 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:44.479 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:44 smithi164 conmon[29288]: cluster 2021-02-10T04:30:43.306665+0000 osd.4 (osd.4) 120 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:44.479 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:44 smithi164 conmon[29288]: cluster 2021-02-10T04:30:43.306671+0000 osd.4 (osd.4) 121 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:44.479 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:44 smithi164 conmon[29288]: cluster 2021-02-10T04:30:43.306675+0000 osd.4 (osd.4) 122 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:44.479 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:44 smithi164 conmon[29288]: cluster 2021-02-10T04:30:44.023531+0000 mgr.y (mgr.14140) 413 : cluster [DBG] pgmap v490: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim, 165 active+clean; 135 MiB data, 459 MiB used, 715 GiB / 715 GiB avail; 4.1 MiB/s rd, 452 KiB/s wr, 6 op/s 2021-02-10T04:30:44.480 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:44 smithi164 conmon[29288]: cluster 2021-02-10T04:30:44.066644+0000 mon.a (mon.0) 946 : cluster [DBG] osdmap e228: 8 total, 8 up, 8 in 2021-02-10T04:30:44.684 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:30:44.685 INFO:teuthology.orchestra.run.smithi164.stdout: "size": 20, 2021-02-10T04:30:44.685 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 600, 2021-02-10T04:30:44.685 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [ 2021-02-10T04:30:44.685 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.685 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.24382.0:653 4.3 e62/55 4:c134263f:lc::lc.25:head v 62'19, mlcod=62'17)", 2021-02-10T04:30:44.685 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:25:55.274265+0000", 2021-02-10T04:30:44.686 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 289.40854445999997, 2021-02-10T04:30:44.686 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0030828539999999999, 2021-02-10T04:30:44.686 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:44.686 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:44.686 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:44.687 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.687 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:44.687 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.274265+0000", 2021-02-10T04:30:44.687 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:44.687 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.687 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.688 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:44.688 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.274265+0000", 2021-02-10T04:30:44.688 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4502e-05 2021-02-10T04:30:44.688 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.688 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.689 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:44.689 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.274290+0000", 2021-02-10T04:30:44.689 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.7899999999999999e-06 2021-02-10T04:30:44.689 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.689 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.689 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:44.690 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.274294+0000", 2021-02-10T04:30:44.690 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.8299999999999998e-06 2021-02-10T04:30:44.690 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.691 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.691 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:44.691 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.274301+0000", 2021-02-10T04:30:44.691 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.98e-06 2021-02-10T04:30:44.691 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.691 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.692 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:44.692 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.274305+0000", 2021-02-10T04:30:44.692 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.2799000000000002e-05 2021-02-10T04:30:44.692 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.692 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.693 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:44.693 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.274338+0000", 2021-02-10T04:30:44.693 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8970999999999998e-05 2021-02-10T04:30:44.693 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.693 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.693 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:44.694 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.274357+0000", 2021-02-10T04:30:44.694 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0029293000000000001 2021-02-10T04:30:44.694 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.694 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.695 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:44.695 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.277286+0000", 2021-02-10T04:30:44.695 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.1681999999999997e-05 2021-02-10T04:30:44.695 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.695 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.695 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:44.696 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.277348+0000", 2021-02-10T04:30:44.696 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0030828539999999999 2021-02-10T04:30:44.696 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.696 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:44.696 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.696 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.697 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.697 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.24382.0:710 4.a e62/55 4:5127e642:gc::gc.21:head v 62'14, mlcod=62'12)", 2021-02-10T04:30:44.697 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:25:55.294750+0000", 2021-02-10T04:30:44.697 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 289.38805950099999, 2021-02-10T04:30:44.697 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0030637189999999999, 2021-02-10T04:30:44.698 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:44.698 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:44.698 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:44.698 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.699 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:44.699 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.294750+0000", 2021-02-10T04:30:44.699 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:44.700 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.700 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.700 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:44.700 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.294750+0000", 2021-02-10T04:30:44.700 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.4215999999999998e-05 2021-02-10T04:30:44.700 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.701 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.701 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:44.701 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.294794+0000", 2021-02-10T04:30:44.701 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.9859999999999997e-06 2021-02-10T04:30:44.701 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.702 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.702 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:44.702 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.294801+0000", 2021-02-10T04:30:44.702 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1270000000000001e-05 2021-02-10T04:30:44.702 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.703 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.703 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:44.703 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.294812+0000", 2021-02-10T04:30:44.703 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.716e-06 2021-02-10T04:30:44.703 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.703 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.703 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:44.703 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.294818+0000", 2021-02-10T04:30:44.703 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9676999999999999e-05 2021-02-10T04:30:44.703 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.703 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.703 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:44.703 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.294868+0000", 2021-02-10T04:30:44.704 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8981999999999999e-05 2021-02-10T04:30:44.704 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.704 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.704 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:44.704 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.294887+0000", 2021-02-10T04:30:44.704 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0028739450000000001 2021-02-10T04:30:44.704 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.704 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.704 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:44.704 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.297761+0000", 2021-02-10T04:30:44.704 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.2927000000000003e-05 2021-02-10T04:30:44.704 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.704 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.705 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:44.705 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.297814+0000", 2021-02-10T04:30:44.705 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0030637189999999999 2021-02-10T04:30:44.705 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.705 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:44.705 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.705 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.705 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.705 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.24382.0:721 4.11 e62/55 4:8eb99c53:gc::gc.22:head v 62'12, mlcod=62'10)", 2021-02-10T04:30:44.705 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:25:55.299794+0000", 2021-02-10T04:30:44.706 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 289.38301598700002, 2021-02-10T04:30:44.706 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0049838299999999999, 2021-02-10T04:30:44.706 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:44.706 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:44.706 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:44.706 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.706 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:44.707 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.299794+0000", 2021-02-10T04:30:44.707 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:44.707 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.707 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.707 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:44.707 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.299794+0000", 2021-02-10T04:30:44.707 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.9113000000000001e-05 2021-02-10T04:30:44.707 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.707 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.707 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:44.707 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.299853+0000", 2021-02-10T04:30:44.707 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.679e-05 2021-02-10T04:30:44.708 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.708 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.708 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:44.708 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.299869+0000", 2021-02-10T04:30:44.708 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6226e-05 2021-02-10T04:30:44.708 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.708 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.708 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:44.709 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.299886+0000", 2021-02-10T04:30:44.709 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7292999999999999e-05 2021-02-10T04:30:44.709 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.709 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.709 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:44.709 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.299903+0000", 2021-02-10T04:30:44.709 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000162638 2021-02-10T04:30:44.709 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.709 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.709 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:44.709 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.300066+0000", 2021-02-10T04:30:44.710 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8708e-05 2021-02-10T04:30:44.710 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.710 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.710 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:44.710 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.300094+0000", 2021-02-10T04:30:44.710 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0046342359999999999 2021-02-10T04:30:44.710 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.710 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.710 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:44.710 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.304729+0000", 2021-02-10T04:30:44.710 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.8826000000000003e-05 2021-02-10T04:30:44.710 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.711 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.711 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:44.711 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:25:55.304777+0000", 2021-02-10T04:30:44.711 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0049838299999999999 2021-02-10T04:30:44.711 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.711 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:44.711 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.711 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.711 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.711 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.24373.0:937 5.9 5:93e5b521:::notify.7:head [watch ping cookie 94187770081792] snapc 0=[] ondisk+write+known_if_redirected e84)", 2021-02-10T04:30:44.711 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:27:47.724971+0000", 2021-02-10T04:30:44.711 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 176.95783825300001, 2021-02-10T04:30:44.711 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0033733579999999999, 2021-02-10T04:30:44.712 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:44.712 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:30:44.712 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:30:44.713 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:30:44.713 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[29319]: :30:43.306658+0000 osd.4 (osd.4) 119 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:44.713 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[29319]: cluster 2021-02-10T04:30:43.306665+0000 osd.4 (osd.4) 2021-02-10T04:30:44.713 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[29319]: 120 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:44.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:30:44.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[29319]: 04:30:43.306671+0000 osd.4 (osd.4) 121 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:44.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:30:44.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[29319]: -02-10T04:30:43.306675+0000 osd.4 2021-02-10T04:30:44.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[29319]: (osd.4) 122 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:44.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:30:44.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[29319]: -10T04:30:44.023531+0000 mgr.y ( 2021-02-10T04:30:44.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[29319]: mgr.14140) 413 : cluster [DBG] 2021-02-10T04:30:44.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[29319]: pgmap v490: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim, 165 active+clean; 135 MiB data, 459 MiB used, 715 GiB / 715 GiB avail; 4.1 MiB/s rd, 452 KiB/s wr, 6 op/s 2021-02-10T04:30:44.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[29319]: cluster 2021-02-10T04:30:44 2021-02-10T04:30:44.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[29319]: .066644+0000 mon.a (mon.0) 946 : cluster 2021-02-10T04:30:44.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[29319]: [DBG] osdmap e228: 8 total, 8 up, 8 in 2021-02-10T04:30:44.715 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[35961]: cluster 2021-02-10T04:30:43.306658 2021-02-10T04:30:44.715 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 119 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:44.715 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:30:44.715 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[35961]: 30:43.306665+0000 osd.4 (osd.4) 120 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:44.715 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:30:44.715 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[35961]: T04:30:43.306671+0000 osd.4 (osd 2021-02-10T04:30:44.715 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[35961]: .4) 121 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:44.715 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[35961]: cluster 2021-02-10T04:30:44.715 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[35961]: 2021-02-10T04:30:43.306675+0000 2021-02-10T04:30:44.715 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[35961]: osd.4 (osd.4) 122 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:44.715 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:30:44.715 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[35961]: :30:44.023531+0000 mgr.y (mgr.14140) 413 : cluster [DBG] 2021-02-10T04:30:44.716 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[35961]: pgmap v490: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim, 165 active+clean; 135 MiB data, 459 MiB used, 715 GiB / 715 GiB avail; 4.1 MiB/s rd, 452 KiB/s wr, 6 op/s 2021-02-10T04:30:44.716 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:30:44.716 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[35961]: -02-10T04:30:44.066644+0000 mon.a (mon.0) 946 : cluster [DBG] 2021-02-10T04:30:44.716 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:44 smithi085 conmon[35961]: osdmap e228: 8 total, 8 up, 8 in 2021-02-10T04:30:44.716 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.24373", 2021-02-10T04:30:44.716 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:30:44.716 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 937 2021-02-10T04:30:44.716 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.716 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:44.716 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.716 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:44.716 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:27:47.724971+0000", 2021-02-10T04:30:44.717 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:44.717 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.717 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.717 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:44.717 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:27:47.724971+0000", 2021-02-10T04:30:44.717 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.1644000000000001e-05 2021-02-10T04:30:44.717 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.717 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.717 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:44.717 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:27:47.724993+0000", 2021-02-10T04:30:44.717 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.5410000000000003e-06 2021-02-10T04:30:44.717 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.718 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.718 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:44.718 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:27:47.725000+0000", 2021-02-10T04:30:44.718 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6864e-05 2021-02-10T04:30:44.718 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.718 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.718 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:44.718 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:27:47.725017+0000", 2021-02-10T04:30:44.718 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7319999999999998e-06 2021-02-10T04:30:44.718 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.718 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.718 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:44.718 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:27:47.725025+0000", 2021-02-10T04:30:44.719 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7789999999999999e-05 2021-02-10T04:30:44.719 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.719 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.719 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:44.719 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:27:47.725103+0000", 2021-02-10T04:30:44.719 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0738e-05 2021-02-10T04:30:44.719 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.719 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.719 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2021-02-10T04:30:44.719 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:27:47.725114+0000", 2021-02-10T04:30:44.719 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0030191509999999999 2021-02-10T04:30:44.719 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.720 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.720 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:44.720 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:27:47.728133+0000", 2021-02-10T04:30:44.720 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.0023000000000003e-05 2021-02-10T04:30:44.720 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.720 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.720 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:44.720 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:27:47.728213+0000", 2021-02-10T04:30:44.720 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00013187499999999999 2021-02-10T04:30:44.720 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.721 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.721 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:44.721 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:27:47.728345+0000", 2021-02-10T04:30:44.721 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0033733579999999999 2021-02-10T04:30:44.721 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.721 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:44.721 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.721 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.721 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.721 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:201 7.f e98/93 7:fe141921:::smithi08575397-2:head v 98'13, mlcod=98'11)", 2021-02-10T04:30:44.721 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:28:34.702445+0000", 2021-02-10T04:30:44.721 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 129.98036415000001, 2021-02-10T04:30:44.721 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0042584390000000001, 2021-02-10T04:30:44.722 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:44.722 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:44.722 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:44.722 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.722 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:44.722 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:34.702445+0000", 2021-02-10T04:30:44.722 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:44.722 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.722 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.722 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:44.722 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:34.702445+0000", 2021-02-10T04:30:44.722 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.002347137 2021-02-10T04:30:44.723 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.723 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.723 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:44.723 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:34.704792+0000", 2021-02-10T04:30:44.723 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.5250000000000005e-06 2021-02-10T04:30:44.723 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.723 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.723 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:44.723 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:34.704801+0000", 2021-02-10T04:30:44.723 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1970000000000001e-05 2021-02-10T04:30:44.723 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.723 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.724 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:44.724 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:34.704813+0000", 2021-02-10T04:30:44.724 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7589999999999997e-06 2021-02-10T04:30:44.724 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.724 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.724 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:44.724 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:34.704821+0000", 2021-02-10T04:30:44.724 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6576000000000002e-05 2021-02-10T04:30:44.724 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.724 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.725 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:44.725 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:34.704877+0000", 2021-02-10T04:30:44.725 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.1880000000000001e-05 2021-02-10T04:30:44.725 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.725 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.725 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:44.725 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:34.704899+0000", 2021-02-10T04:30:44.725 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001739551 2021-02-10T04:30:44.725 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.725 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.725 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:44.725 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:34.706639+0000", 2021-02-10T04:30:44.725 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.5041000000000003e-05 2021-02-10T04:30:44.726 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.726 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.726 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:44.726 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:34.706704+0000", 2021-02-10T04:30:44.726 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0042584390000000001 2021-02-10T04:30:44.726 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.726 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:44.726 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.726 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.726 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.726 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:257 7.8 e104/93 7:10dac65a:::smithi08575397-39:head v 104'4, mlcod=98'2)", 2021-02-10T04:30:44.726 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:28:40.225547+0000", 2021-02-10T04:30:44.727 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 124.457262693, 2021-02-10T04:30:44.727 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.006945667, 2021-02-10T04:30:44.727 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:44.727 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:44.727 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:44.727 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.727 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:44.727 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:40.225547+0000", 2021-02-10T04:30:44.727 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:44.727 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.727 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.727 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:44.727 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:40.225547+0000", 2021-02-10T04:30:44.728 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0048111430000000004 2021-02-10T04:30:44.728 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.728 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.728 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:44.728 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:40.230358+0000", 2021-02-10T04:30:44.728 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9599999999999999e-05 2021-02-10T04:30:44.728 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.728 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.728 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:44.728 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:40.230378+0000", 2021-02-10T04:30:44.728 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2741999999999999e-05 2021-02-10T04:30:44.728 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.728 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.729 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:44.729 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:40.230390+0000", 2021-02-10T04:30:44.729 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.9219999999999997e-06 2021-02-10T04:30:44.729 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.729 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.729 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:44.729 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:40.230398+0000", 2021-02-10T04:30:44.729 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.8294999999999997e-05 2021-02-10T04:30:44.729 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.729 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.729 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:44.729 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:40.230457+0000", 2021-02-10T04:30:44.729 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2069000000000002e-05 2021-02-10T04:30:44.730 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.730 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.730 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:44.730 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:40.230479+0000", 2021-02-10T04:30:44.730 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0019592810000000002 2021-02-10T04:30:44.730 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.730 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.730 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:44.730 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:40.232438+0000", 2021-02-10T04:30:44.730 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.4614999999999998e-05 2021-02-10T04:30:44.730 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.730 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.730 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:44.731 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:40.232492+0000", 2021-02-10T04:30:44.731 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.006945667 2021-02-10T04:30:44.731 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.731 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:44.731 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.731 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.731 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.731 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:260 7.c e105/93 7:322cfa51:::smithi08575397-22:head v 105'14, mlcod=98'11)", 2021-02-10T04:30:44.731 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:28:41.237855+0000", 2021-02-10T04:30:44.731 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 123.444954282, 2021-02-10T04:30:44.731 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0094911179999999998, 2021-02-10T04:30:44.731 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:44.731 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:44.732 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:44.732 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.732 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:44.732 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.237855+0000", 2021-02-10T04:30:44.732 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:44.732 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.732 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.732 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:44.732 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.237855+0000", 2021-02-10T04:30:44.732 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0016589510000000001 2021-02-10T04:30:44.732 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.732 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.733 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:44.733 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.239514+0000", 2021-02-10T04:30:44.733 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2934e-05 2021-02-10T04:30:44.733 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.733 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.733 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:44.733 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.239527+0000", 2021-02-10T04:30:44.733 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3473e-05 2021-02-10T04:30:44.733 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.733 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.733 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:44.733 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.239541+0000", 2021-02-10T04:30:44.733 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0407999999999999e-05 2021-02-10T04:30:44.734 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.734 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.734 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:44.734 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.239551+0000", 2021-02-10T04:30:44.734 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00035611300000000002 2021-02-10T04:30:44.734 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.734 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.734 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:44.734 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.239907+0000", 2021-02-10T04:30:44.734 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6283e-05 2021-02-10T04:30:44.734 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.734 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.734 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:44.735 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.239923+0000", 2021-02-10T04:30:44.735 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0073575389999999997 2021-02-10T04:30:44.735 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.735 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.735 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:44.735 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.247281+0000", 2021-02-10T04:30:44.735 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.5417000000000001e-05 2021-02-10T04:30:44.735 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.735 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.735 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:44.735 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.247346+0000", 2021-02-10T04:30:44.735 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0094911179999999998 2021-02-10T04:30:44.736 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.736 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:44.736 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.736 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.736 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.736 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:277 7.6 e105/93 7:649ff6e5:::smithi08575397-4:head v 105'8, mlcod=98'5)", 2021-02-10T04:30:44.736 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:28:41.500707+0000", 2021-02-10T04:30:44.736 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 123.182102012, 2021-02-10T04:30:44.736 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.005635429, 2021-02-10T04:30:44.736 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:44.736 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:44.736 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:44.736 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.737 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:44.737 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.500707+0000", 2021-02-10T04:30:44.737 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:44.737 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.737 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.737 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:44.737 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.500707+0000", 2021-02-10T04:30:44.737 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.003855013 2021-02-10T04:30:44.737 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.737 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.737 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:44.737 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.504562+0000", 2021-02-10T04:30:44.737 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0930999999999999e-05 2021-02-10T04:30:44.738 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.738 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.738 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:44.738 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.504573+0000", 2021-02-10T04:30:44.738 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5299e-05 2021-02-10T04:30:44.738 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.738 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.738 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:44.738 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.504589+0000", 2021-02-10T04:30:44.738 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.8750000000000006e-06 2021-02-10T04:30:44.738 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.738 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.739 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:44.739 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.504598+0000", 2021-02-10T04:30:44.739 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.0171000000000003e-05 2021-02-10T04:30:44.739 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.739 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.739 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:44.739 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.504668+0000", 2021-02-10T04:30:44.739 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2299000000000001e-05 2021-02-10T04:30:44.739 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.739 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.739 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:44.739 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.504690+0000", 2021-02-10T04:30:44.739 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001599624 2021-02-10T04:30:44.740 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.740 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.740 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:44.740 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.506290+0000", 2021-02-10T04:30:44.740 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.3217000000000002e-05 2021-02-10T04:30:44.740 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.740 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.740 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:44.740 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.506343+0000", 2021-02-10T04:30:44.740 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.005635429 2021-02-10T04:30:44.740 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.740 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:44.740 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.741 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.741 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.741 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:334 7.c e107/93 7:30bca126:::smithi08575397-31:head v 107'22, mlcod=105'18)", 2021-02-10T04:30:44.741 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:28:43.043645+0000", 2021-02-10T04:30:44.741 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 121.639164415, 2021-02-10T04:30:44.741 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0073336349999999998, 2021-02-10T04:30:44.741 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:44.741 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:44.741 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:44.741 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.741 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:44.741 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:43.043645+0000", 2021-02-10T04:30:44.741 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:44.742 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.742 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.742 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:44.742 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:43.043645+0000", 2021-02-10T04:30:44.742 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001453717 2021-02-10T04:30:44.742 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.742 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.742 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:44.742 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:43.045099+0000", 2021-02-10T04:30:44.742 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2466000000000001e-05 2021-02-10T04:30:44.742 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.742 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.742 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:44.743 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:43.045111+0000", 2021-02-10T04:30:44.743 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5585000000000001e-05 2021-02-10T04:30:44.743 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.743 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.743 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:44.743 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:43.045127+0000", 2021-02-10T04:30:44.743 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3396e-05 2021-02-10T04:30:44.743 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.743 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.743 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:44.743 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:43.045140+0000", 2021-02-10T04:30:44.743 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.3097999999999996e-05 2021-02-10T04:30:44.743 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.744 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.744 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:44.744 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:43.045223+0000", 2021-02-10T04:30:44.744 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.2190999999999997e-05 2021-02-10T04:30:44.744 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.744 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.744 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:44.744 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:43.045256+0000", 2021-02-10T04:30:44.744 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0056550089999999999 2021-02-10T04:30:44.744 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.744 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.744 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:44.744 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:43.050911+0000", 2021-02-10T04:30:44.745 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.8172999999999999e-05 2021-02-10T04:30:44.745 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.745 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.745 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:44.745 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:43.050979+0000", 2021-02-10T04:30:44.745 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0073336349999999998 2021-02-10T04:30:44.745 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.745 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:44.745 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.745 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.745 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.745 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:546 7.f e127/93 7:f473c801:::smithi08575397-49:head v 127'36, mlcod=121'33)", 2021-02-10T04:30:44.745 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:03.068155+0000", 2021-02-10T04:30:44.746 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 101.614654479, 2021-02-10T04:30:44.746 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.015127623, 2021-02-10T04:30:44.746 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:44.746 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:44.746 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:44.746 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.746 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:44.746 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.068155+0000", 2021-02-10T04:30:44.746 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:44.746 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.746 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.746 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:44.746 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.068155+0000", 2021-02-10T04:30:44.747 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00050784699999999996 2021-02-10T04:30:44.747 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.747 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.747 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:44.747 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.068663+0000", 2021-02-10T04:30:44.747 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.6600000000000001e-06 2021-02-10T04:30:44.747 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.747 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.747 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:44.747 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.068672+0000", 2021-02-10T04:30:44.747 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2802e-05 2021-02-10T04:30:44.747 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.747 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.748 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:44.748 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.068684+0000", 2021-02-10T04:30:44.748 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.9689999999999994e-06 2021-02-10T04:30:44.748 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.748 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.748 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:44.748 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.068692+0000", 2021-02-10T04:30:44.748 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.5361000000000001e-05 2021-02-10T04:30:44.748 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.748 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.748 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:44.748 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.068748+0000", 2021-02-10T04:30:44.748 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.1291e-05 2021-02-10T04:30:44.749 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.749 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.749 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:44.749 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.068769+0000", 2021-02-10T04:30:44.749 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.014459856 2021-02-10T04:30:44.749 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.749 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.749 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:44.749 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.083229+0000", 2021-02-10T04:30:44.749 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.3837e-05 2021-02-10T04:30:44.749 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.749 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.749 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:44.750 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.083283+0000", 2021-02-10T04:30:44.750 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.015127623 2021-02-10T04:30:44.750 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.750 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:44.750 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.750 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.750 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.750 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:547 7.f e127/93 7:f473c801:::smithi08575397-49:head v 127'37, mlcod=121'33)", 2021-02-10T04:30:44.750 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:03.068728+0000", 2021-02-10T04:30:44.750 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 101.61408165, 2021-02-10T04:30:44.750 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.014605755, 2021-02-10T04:30:44.750 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:44.751 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:44.751 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:44.751 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.751 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:44.751 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.068728+0000", 2021-02-10T04:30:44.751 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:44.751 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.751 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.751 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:44.752 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.068728+0000", 2021-02-10T04:30:44.752 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.4261000000000003e-05 2021-02-10T04:30:44.752 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.752 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.752 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:44.752 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.068762+0000", 2021-02-10T04:30:44.752 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.2290000000000002e-06 2021-02-10T04:30:44.752 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.752 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.752 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:44.752 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.068767+0000", 2021-02-10T04:30:44.753 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.4290000000000001e-06 2021-02-10T04:30:44.753 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.753 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.753 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:44.753 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.068776+0000", 2021-02-10T04:30:44.753 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.07e-06 2021-02-10T04:30:44.753 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.753 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.753 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:44.754 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.068780+0000", 2021-02-10T04:30:44.754 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00043224600000000001 2021-02-10T04:30:44.754 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.754 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.754 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:44.754 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.069212+0000", 2021-02-10T04:30:44.754 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7577000000000001e-05 2021-02-10T04:30:44.754 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.754 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.754 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:44.755 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.069230+0000", 2021-02-10T04:30:44.755 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.014056879 2021-02-10T04:30:44.755 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.755 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.755 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:44.755 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.083287+0000", 2021-02-10T04:30:44.755 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.7064000000000002e-05 2021-02-10T04:30:44.755 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.755 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.755 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:44.755 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.083334+0000", 2021-02-10T04:30:44.755 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.014605755 2021-02-10T04:30:44.755 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.756 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:44.756 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.756 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.757 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.757 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:587 7.5 7:a5580f12:::smithi08575397-18:head [write 463893~562240] snapc 20=[20,1f,1e,1d,1c,1b,19,17,15] ondisk+write+known_if_redirected e129)", 2021-02-10T04:30:44.757 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:05.134384+0000", 2021-02-10T04:30:44.757 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 99.548425058000007, 2021-02-10T04:30:44.758 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.007617927, 2021-02-10T04:30:44.758 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:44.758 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:44.758 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:30:44.758 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:30:44.758 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:30:44.758 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 587 2021-02-10T04:30:44.759 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.759 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:44.759 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.759 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:44.759 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.134384+0000", 2021-02-10T04:30:44.759 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:44.759 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.759 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.759 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:44.759 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.134384+0000", 2021-02-10T04:30:44.759 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00067347900000000005 2021-02-10T04:30:44.759 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.759 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.760 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:44.760 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.135058+0000", 2021-02-10T04:30:44.760 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1919999999999999e-06 2021-02-10T04:30:44.760 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.760 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.760 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:44.760 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.135065+0000", 2021-02-10T04:30:44.760 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2726e-05 2021-02-10T04:30:44.760 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.760 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.760 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:44.760 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.135078+0000", 2021-02-10T04:30:44.760 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.6720000000000004e-06 2021-02-10T04:30:44.760 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.761 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.761 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:44.761 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.135086+0000", 2021-02-10T04:30:44.761 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1996000000000006e-05 2021-02-10T04:30:44.761 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.761 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.761 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:44.761 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.135168+0000", 2021-02-10T04:30:44.761 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.2620000000000001e-06 2021-02-10T04:30:44.761 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.761 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.761 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2021-02-10T04:30:44.761 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.135175+0000", 2021-02-10T04:30:44.762 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0021127770000000001 2021-02-10T04:30:44.762 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.762 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.762 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:44.762 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.137288+0000", 2021-02-10T04:30:44.762 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.3589000000000003e-05 2021-02-10T04:30:44.762 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.762 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.762 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:44.762 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.137361+0000", 2021-02-10T04:30:44.762 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000140686 2021-02-10T04:30:44.762 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.762 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.763 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 0,2", 2021-02-10T04:30:44.763 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.137502+0000", 2021-02-10T04:30:44.763 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001337237 2021-02-10T04:30:44.763 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.763 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.763 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:30:44.763 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.138839+0000", 2021-02-10T04:30:44.763 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001254788 2021-02-10T04:30:44.763 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.763 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.763 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:44.763 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.140094+0000", 2021-02-10T04:30:44.763 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0018065030000000001 2021-02-10T04:30:44.764 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.764 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.764 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:44.764 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.141900+0000", 2021-02-10T04:30:44.764 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.8050000000000002e-05 2021-02-10T04:30:44.764 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.764 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.764 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:44.764 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.141948+0000", 2021-02-10T04:30:44.764 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.397e-05 2021-02-10T04:30:44.764 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.764 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.764 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:44.764 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.142002+0000", 2021-02-10T04:30:44.765 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.007617927 2021-02-10T04:30:44.765 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.765 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:44.765 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.765 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.765 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.765 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:588 7.5 7:a5580f12:::smithi08575397-18:head [write 1593482~658027] snapc 20=[20,1f,1e,1d,1c,1b,19,17,15] ondisk+write+known_if_redirected e129)", 2021-02-10T04:30:44.765 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:05.141979+0000", 2021-02-10T04:30:44.765 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 99.540830303000007, 2021-02-10T04:30:44.765 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0042585729999999999, 2021-02-10T04:30:44.765 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:44.765 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:44.766 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:30:44.766 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:30:44.766 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:30:44.766 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 588 2021-02-10T04:30:44.766 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.766 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:44.766 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.766 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:44.766 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.141979+0000", 2021-02-10T04:30:44.766 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:44.766 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.766 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.766 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:44.767 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.141979+0000", 2021-02-10T04:30:44.767 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00072400899999999996 2021-02-10T04:30:44.767 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.767 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.767 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:44.767 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.142703+0000", 2021-02-10T04:30:44.767 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.3119999999999994e-06 2021-02-10T04:30:44.767 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.767 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.767 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:44.767 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.142712+0000", 2021-02-10T04:30:44.767 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5583999999999999e-05 2021-02-10T04:30:44.767 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.767 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.768 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:44.768 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.142727+0000", 2021-02-10T04:30:44.768 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.3000000000000004e-06 2021-02-10T04:30:44.768 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.768 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.768 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:44.768 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.142734+0000", 2021-02-10T04:30:44.768 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.9834999999999999e-05 2021-02-10T04:30:44.768 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.768 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.769 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:44.769 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.142794+0000", 2021-02-10T04:30:44.769 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.6446999999999995e-05 2021-02-10T04:30:44.769 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.769 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.769 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:44.769 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.142871+0000", 2021-02-10T04:30:44.769 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00014751099999999999 2021-02-10T04:30:44.769 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.769 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.769 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 0,2", 2021-02-10T04:30:44.770 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.143018+0000", 2021-02-10T04:30:44.770 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001105258 2021-02-10T04:30:44.770 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.770 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.770 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:30:44.770 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.144123+0000", 2021-02-10T04:30:44.770 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00179221 2021-02-10T04:30:44.770 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.770 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.770 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:44.770 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.145916+0000", 2021-02-10T04:30:44.770 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00021294799999999999 2021-02-10T04:30:44.770 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.771 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.771 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:44.771 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.146129+0000", 2021-02-10T04:30:44.771 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.0321999999999995e-05 2021-02-10T04:30:44.771 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.771 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.771 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:44.771 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.146199+0000", 2021-02-10T04:30:44.771 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.8837000000000001e-05 2021-02-10T04:30:44.771 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.771 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.771 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:44.771 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.146238+0000", 2021-02-10T04:30:44.772 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0042585729999999999 2021-02-10T04:30:44.772 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.772 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:44.772 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.772 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.772 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.772 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:589 7.5 7:a5580f12:::smithi08575397-18:head [write 2685935~559341] snapc 20=[20,1f,1e,1d,1c,1b,19,17,15] ondisk+write+known_if_redirected e129)", 2021-02-10T04:30:44.772 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:05.148302+0000", 2021-02-10T04:30:44.772 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 99.534507133000005, 2021-02-10T04:30:44.772 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0048484549999999998, 2021-02-10T04:30:44.772 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:44.772 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:44.772 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:30:44.772 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:30:44.773 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:30:44.773 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 589 2021-02-10T04:30:44.773 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.773 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:44.773 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.773 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:44.773 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.148302+0000", 2021-02-10T04:30:44.773 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:44.773 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.773 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.773 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:44.773 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.148302+0000", 2021-02-10T04:30:44.773 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00055083200000000001 2021-02-10T04:30:44.773 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.774 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.774 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:44.774 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.148853+0000", 2021-02-10T04:30:44.774 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.55e-06 2021-02-10T04:30:44.774 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.774 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.774 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:44.774 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.148860+0000", 2021-02-10T04:30:44.774 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1277999999999999e-05 2021-02-10T04:30:44.774 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.774 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.774 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:44.775 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.148871+0000", 2021-02-10T04:30:44.775 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.1229999999999997e-06 2021-02-10T04:30:44.775 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.775 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.775 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:44.775 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.148877+0000", 2021-02-10T04:30:44.775 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.0185000000000001e-05 2021-02-10T04:30:44.775 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.775 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.775 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:44.775 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.148927+0000", 2021-02-10T04:30:44.775 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.4608e-05 2021-02-10T04:30:44.775 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.775 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.776 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:44.776 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.149012+0000", 2021-02-10T04:30:44.776 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00021028999999999999 2021-02-10T04:30:44.776 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.776 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.776 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 0,2", 2021-02-10T04:30:44.776 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.149222+0000", 2021-02-10T04:30:44.776 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0010802120000000001 2021-02-10T04:30:44.776 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.776 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.776 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:30:44.776 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.150302+0000", 2021-02-10T04:30:44.776 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0025523099999999999 2021-02-10T04:30:44.777 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.777 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.777 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:44.777 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.152855+0000", 2021-02-10T04:30:44.777 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000170283 2021-02-10T04:30:44.777 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.777 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.777 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:44.777 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.153025+0000", 2021-02-10T04:30:44.777 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7354000000000005e-05 2021-02-10T04:30:44.777 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.777 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.777 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:44.778 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.153102+0000", 2021-02-10T04:30:44.778 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.8430000000000002e-05 2021-02-10T04:30:44.778 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.778 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.778 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:44.778 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.153151+0000", 2021-02-10T04:30:44.778 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0048484549999999998 2021-02-10T04:30:44.778 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.778 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:44.778 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.778 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.778 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.778 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:590 7.5 7:a5580f12:::smithi08575397-18:head [setxattr _header (57) in=7b,truncate 3663893] snapc 20=[20,1f,1e,1d,1c,1b,19,17,15] ondisk+write+known_if_redirected e129)", 2021-02-10T04:30:44.779 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:05.148919+0000", 2021-02-10T04:30:44.779 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 99.533890510999996, 2021-02-10T04:30:44.779 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.004420017, 2021-02-10T04:30:44.779 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:44.779 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:44.779 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:30:44.779 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:30:44.779 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:30:44.779 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 590 2021-02-10T04:30:44.779 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.779 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:44.779 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.779 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:44.780 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.148919+0000", 2021-02-10T04:30:44.780 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:44.780 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.780 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.780 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:44.780 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.148919+0000", 2021-02-10T04:30:44.780 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.7167000000000003e-05 2021-02-10T04:30:44.780 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.780 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.780 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:44.780 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.148986+0000", 2021-02-10T04:30:44.780 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2021999999999999e-05 2021-02-10T04:30:44.780 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.781 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.781 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:44.781 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.148998+0000", 2021-02-10T04:30:44.781 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8255999999999999e-05 2021-02-10T04:30:44.781 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.781 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.781 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:44.781 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.149016+0000", 2021-02-10T04:30:44.781 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4313e-05 2021-02-10T04:30:44.781 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.781 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.781 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:44.781 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.149031+0000", 2021-02-10T04:30:44.781 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00084521200000000002 2021-02-10T04:30:44.782 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.782 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.782 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:44.782 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.149876+0000", 2021-02-10T04:30:44.782 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.2990000000000003e-05 2021-02-10T04:30:44.782 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.782 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.782 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:44.782 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.149959+0000", 2021-02-10T04:30:44.782 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00013881899999999999 2021-02-10T04:30:44.782 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.782 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.782 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 0,2", 2021-02-10T04:30:44.783 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.150098+0000", 2021-02-10T04:30:44.783 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00040864200000000002 2021-02-10T04:30:44.783 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.783 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.783 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:30:44.783 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.150506+0000", 2021-02-10T04:30:44.783 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.002463584 2021-02-10T04:30:44.783 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.783 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.783 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:44.783 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.152970+0000", 2021-02-10T04:30:44.783 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000231902 2021-02-10T04:30:44.783 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.783 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.784 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:30:44.784 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.153202+0000", 2021-02-10T04:30:44.784 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.4593999999999999e-05 2021-02-10T04:30:44.784 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.784 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.784 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:44.784 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.153266+0000", 2021-02-10T04:30:44.784 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.2515999999999995e-05 2021-02-10T04:30:44.784 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.784 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.784 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:44.784 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.153339+0000", 2021-02-10T04:30:44.784 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.004420017 2021-02-10T04:30:44.785 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.785 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:44.785 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.785 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.785 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.785 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:591 7.5 7:a5580f12:::smithi08575397-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e129)", 2021-02-10T04:30:44.785 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:05.149099+0000", 2021-02-10T04:30:44.785 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 99.533710834999994, 2021-02-10T04:30:44.785 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0044523540000000004, 2021-02-10T04:30:44.785 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:44.785 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:30:44.785 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:30:44.785 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:30:44.786 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:30:44.786 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 591 2021-02-10T04:30:44.786 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.786 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:44.786 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.786 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:44.786 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.149099+0000", 2021-02-10T04:30:44.786 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:44.786 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.786 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.786 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:44.786 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.149099+0000", 2021-02-10T04:30:44.786 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.7129999999999998e-05 2021-02-10T04:30:44.786 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.787 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.787 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:44.787 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.149146+0000", 2021-02-10T04:30:44.787 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4625e-05 2021-02-10T04:30:44.787 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.787 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.787 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:44.787 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.149160+0000", 2021-02-10T04:30:44.787 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2909e-05 2021-02-10T04:30:44.787 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.787 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.787 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:44.787 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.149183+0000", 2021-02-10T04:30:44.788 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.8800000000000002e-06 2021-02-10T04:30:44.788 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.788 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.788 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:44.788 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.149190+0000", 2021-02-10T04:30:44.788 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0012141260000000001 2021-02-10T04:30:44.788 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.788 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.788 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:44.788 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.150404+0000", 2021-02-10T04:30:44.788 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.3104999999999997e-05 2021-02-10T04:30:44.788 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.788 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.788 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2021-02-10T04:30:44.789 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.150497+0000", 2021-02-10T04:30:44.789 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.002878542 2021-02-10T04:30:44.789 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.789 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.789 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:44.789 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.153376+0000", 2021-02-10T04:30:44.789 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.5690000000000003e-05 2021-02-10T04:30:44.789 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.789 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.789 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:44.789 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.153442+0000", 2021-02-10T04:30:44.789 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000109347 2021-02-10T04:30:44.789 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.789 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.790 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:44.790 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.153551+0000", 2021-02-10T04:30:44.790 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0044523540000000004 2021-02-10T04:30:44.790 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.790 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:44.790 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.790 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.790 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.790 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:616 7.f e131/93 7:fc15f6da:::smithi08575397-35:head v 131'39, mlcod=127'36)", 2021-02-10T04:30:44.790 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:07.917437+0000", 2021-02-10T04:30:44.790 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 96.765372653, 2021-02-10T04:30:44.790 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0064601019999999997, 2021-02-10T04:30:44.790 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:44.791 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:44.791 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:44.791 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.791 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:44.791 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:07.917437+0000", 2021-02-10T04:30:44.791 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:44.791 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.791 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.791 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:44.791 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:07.917437+0000", 2021-02-10T04:30:44.791 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0033244979999999999 2021-02-10T04:30:44.791 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.791 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.792 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:44.792 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:07.920761+0000", 2021-02-10T04:30:44.792 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0390999999999999e-05 2021-02-10T04:30:44.792 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.792 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.792 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:44.792 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:07.920772+0000", 2021-02-10T04:30:44.792 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3027000000000001e-05 2021-02-10T04:30:44.792 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.792 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.792 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:44.792 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:07.920785+0000", 2021-02-10T04:30:44.792 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.337e-06 2021-02-10T04:30:44.793 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.793 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.793 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:44.793 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:07.920793+0000", 2021-02-10T04:30:44.793 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6339000000000003e-05 2021-02-10T04:30:44.793 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.793 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.793 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:44.793 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:07.920859+0000", 2021-02-10T04:30:44.793 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2643999999999999e-05 2021-02-10T04:30:44.793 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.793 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.793 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:44.793 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:07.920882+0000", 2021-02-10T04:30:44.794 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0029601440000000001 2021-02-10T04:30:44.794 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.794 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.794 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:44.794 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:07.923842+0000", 2021-02-10T04:30:44.794 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.4722000000000002e-05 2021-02-10T04:30:44.794 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.794 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.794 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:44.794 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:07.923897+0000", 2021-02-10T04:30:44.794 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0064601019999999997 2021-02-10T04:30:44.794 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.794 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:44.795 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.795 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.795 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.795 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:1177 7.8 e176/93 7:10dac65a:::smithi08575397-39:head v 176'54, mlcod=171'50)", 2021-02-10T04:30:44.795 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:52.230237+0000", 2021-02-10T04:30:44.795 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 52.452572883999999, 2021-02-10T04:30:44.795 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0034100879999999999, 2021-02-10T04:30:44.795 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:44.795 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:44.795 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:44.795 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.795 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:44.795 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.230237+0000", 2021-02-10T04:30:44.796 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:44.796 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.796 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.796 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:44.796 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.230237+0000", 2021-02-10T04:30:44.796 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0020747159999999999 2021-02-10T04:30:44.796 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.796 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.796 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:44.796 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.232311+0000", 2021-02-10T04:30:44.796 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2109e-05 2021-02-10T04:30:44.796 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.796 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.796 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:44.797 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.232323+0000", 2021-02-10T04:30:44.797 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5651999999999999e-05 2021-02-10T04:30:44.797 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.797 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.797 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:44.797 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.232339+0000", 2021-02-10T04:30:44.797 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.1910000000000006e-06 2021-02-10T04:30:44.797 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.797 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.797 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:44.797 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.232348+0000", 2021-02-10T04:30:44.797 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.8849000000000004e-05 2021-02-10T04:30:44.797 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.798 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.798 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:44.798 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.232417+0000", 2021-02-10T04:30:44.798 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.5627999999999999e-05 2021-02-10T04:30:44.798 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.798 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.798 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:44.798 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.232443+0000", 2021-02-10T04:30:44.798 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0011409359999999999 2021-02-10T04:30:44.798 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.798 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.798 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:44.798 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.233584+0000", 2021-02-10T04:30:44.799 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.3007000000000002e-05 2021-02-10T04:30:44.799 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.799 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.799 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:44.799 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.233647+0000", 2021-02-10T04:30:44.799 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0034100879999999999 2021-02-10T04:30:44.799 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.799 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:44.799 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.799 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.799 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.799 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:1178 7.8 e176/93 7:10dac65a:::smithi08575397-39:head v 176'55, mlcod=171'50)", 2021-02-10T04:30:44.799 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:52.233847+0000", 2021-02-10T04:30:44.799 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 52.448962940999998, 2021-02-10T04:30:44.800 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0051555660000000003, 2021-02-10T04:30:44.800 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:44.800 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:44.800 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:44.800 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.800 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:44.800 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.233847+0000", 2021-02-10T04:30:44.800 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:44.800 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.800 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.800 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:44.800 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.233847+0000", 2021-02-10T04:30:44.800 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0039814339999999998 2021-02-10T04:30:44.801 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.801 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.801 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:44.801 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.237828+0000", 2021-02-10T04:30:44.801 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7360000000000001e-06 2021-02-10T04:30:44.801 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.801 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.801 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:44.801 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.237836+0000", 2021-02-10T04:30:44.801 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1529999999999997e-06 2021-02-10T04:30:44.801 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.801 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.801 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:44.801 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.237844+0000", 2021-02-10T04:30:44.802 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.2699999999999999e-06 2021-02-10T04:30:44.802 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.802 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.802 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:44.802 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.237851+0000", 2021-02-10T04:30:44.802 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6455000000000001e-05 2021-02-10T04:30:44.802 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.802 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.802 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:44.802 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.237908+0000", 2021-02-10T04:30:44.802 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2911000000000001e-05 2021-02-10T04:30:44.802 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.802 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.803 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:44.803 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.237931+0000", 2021-02-10T04:30:44.803 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001021781 2021-02-10T04:30:44.803 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.803 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.803 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:44.803 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.238952+0000", 2021-02-10T04:30:44.803 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9826e-05 2021-02-10T04:30:44.803 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.803 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.803 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:44.803 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.239002+0000", 2021-02-10T04:30:44.803 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0051555660000000003 2021-02-10T04:30:44.803 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.804 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:44.804 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.804 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.804 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.804 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:1793 7.8 e225/93 7:10dac65a:::smithi08575397-39:head v 225'68, mlcod=197'65)", 2021-02-10T04:30:44.804 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:30:41.257573+0000", 2021-02-10T04:30:44.804 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 3.4252366379999999, 2021-02-10T04:30:44.804 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0044024279999999999, 2021-02-10T04:30:44.804 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:30:44.804 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:30:44.804 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:30:44.804 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.804 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:30:44.805 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:41.257573+0000", 2021-02-10T04:30:44.805 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:30:44.805 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.805 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.805 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:30:44.805 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:41.257573+0000", 2021-02-10T04:30:44.805 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0028409540000000001 2021-02-10T04:30:44.805 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.805 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.805 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:30:44.805 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:41.260414+0000", 2021-02-10T04:30:44.805 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.9940000000000006e-06 2021-02-10T04:30:44.805 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.805 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.806 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:30:44.806 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:41.260423+0000", 2021-02-10T04:30:44.806 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1610000000000001e-05 2021-02-10T04:30:44.806 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.806 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.806 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:30:44.806 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:41.260434+0000", 2021-02-10T04:30:44.806 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.9789999999999998e-06 2021-02-10T04:30:44.806 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.806 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.806 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:30:44.806 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:41.260440+0000", 2021-02-10T04:30:44.806 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.7437e-05 2021-02-10T04:30:44.807 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.807 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.807 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:30:44.807 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:41.260488+0000", 2021-02-10T04:30:44.807 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9292000000000001e-05 2021-02-10T04:30:44.807 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.807 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.807 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:30:44.807 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:41.260507+0000", 2021-02-10T04:30:44.807 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0014192180000000001 2021-02-10T04:30:44.807 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.807 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.807 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:30:44.807 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:41.261926+0000", 2021-02-10T04:30:44.808 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.8943999999999998e-05 2021-02-10T04:30:44.808 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:30:44.808 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:30:44.808 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:30:44.808 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:41.261975+0000", 2021-02-10T04:30:44.808 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0044024279999999999 2021-02-10T04:30:44.808 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.808 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:44.808 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.808 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:30:44.808 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:30:44.808 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:30:45.067 INFO:tasks.rados.rados.0.smithi085.stdout:680: done (10 left) 2021-02-10T04:30:45.067 INFO:tasks.rados.rados.0.smithi085.stdout:681: snap_remove snap 65 2021-02-10T04:30:45.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:30:45 smithi164 conmon[33677]: debug 2021-02-10T04:30:45.317+0000 7fbec19a1700 -1 osd.4 229 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:30:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:45 smithi164 conmon[29288]: cluster 2021-02-10T04:30:44.350374+0000 osd.4 (osd.4) 123 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:45 smithi164 conmon[29288]: cluster 2021-02-10T04:30:44.350384+0000 osd.4 (osd.4) 124 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:45 smithi164 conmon[29288]: cluster 2021-02-10T04:30:44.350389+0000 osd.4 (osd.4) 125 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:45 smithi164 conmon[29288]: cluster 2021-02-10T04:30:44.350393+0000 osd.4 (osd.4) 126 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:45 smithi164 conmon[29288]: cluster 2021-02-10T04:30:45.067401+0000 mon.a (mon.0) 947 : cluster [DBG] osdmap e229: 8 total, 8 up, 8 in 2021-02-10T04:30:45.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:45 smithi085 conmon[29319]: cluster 2021-02-10T04:30:44.350374+0000 osd.4 (osd 2021-02-10T04:30:45.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:45 smithi085 conmon[29319]: .4) 123 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:45.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:45 smithi085 conmon[29319]: cluster 2021-02-10T04:30:44.350384+0000 osd.4 (osd.4) 124 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:45.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:45 smithi085 conmon[29319]: cluster 2021-02-10T04:30:44.350389+0000 osd.4 (osd.4) 125 : cluster [WRN] 2021-02-10T04:30:45.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:45 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:45.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:45 smithi085 conmon[29319]: cluster 2021-02-10T04:30:44 2021-02-10T04:30:45.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:45 smithi085 conmon[29319]: .350393+0000 osd.4 (osd.4) 126 : cluster 2021-02-10T04:30:45.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:45 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:45.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:45 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:30:45.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:45 smithi085 conmon[29319]: 10T04:30:45.067401+0000 mon.a ( 2021-02-10T04:30:45.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:45 smithi085 conmon[29319]: mon.0) 947 : cluster [DBG] osdmap e229: 8 total, 8 up, 8 in 2021-02-10T04:30:45.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:45 smithi085 conmon[35961]: cluster 2021-02-10T04:30:44. 2021-02-10T04:30:45.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:45 smithi085 conmon[35961]: 350374+0000 osd.4 (osd.4) 123 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:45.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:45 smithi085 conmon[35961]: cluster 2021-02-10T04:30:45.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:45 smithi085 conmon[35961]: 2021-02-10T04:30:44.350384 2021-02-10T04:30:45.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:45 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 124 : cluster 2021-02-10T04:30:45.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:45 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:45.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:45 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:30:45.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:45 smithi085 conmon[35961]: -10T04:30:44 2021-02-10T04:30:45.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:45 smithi085 conmon[35961]: .350389+0000 osd.4 2021-02-10T04:30:45.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:45 smithi085 conmon[35961]: (osd.4) 125 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:45.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:45 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:30:45.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:45 smithi085 conmon[35961]: 02-10T04:30:44.350393+0000 osd.4 ( 2021-02-10T04:30:45.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:45 smithi085 conmon[35961]: osd.4) 126 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:45.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:45 smithi085 conmon[35961]: 2021-02-10T04:30:45.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:45 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:30:45.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:45 smithi085 conmon[35961]: 30:45.067401+0000 mon.a (mon.0 2021-02-10T04:30:45.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:45 smithi085 conmon[35961]: ) 947 : cluster 2021-02-10T04:30:45.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:45 smithi085 conmon[35961]: [DBG] osdmap e229: 8 total, 8 up, 8 in 2021-02-10T04:30:46.028 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:30:46.060 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:46.071 INFO:tasks.rados.rados.0.smithi085.stdout:681: done (10 left) 2021-02-10T04:30:46.071 INFO:tasks.rados.rados.0.smithi085.stdout:682: rollback oid 21 current snap is 68 2021-02-10T04:30:46.071 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 21 to 67 2021-02-10T04:30:46.071 INFO:tasks.rados.rados.0.smithi085.stdout:683: copy_from oid 18 from oid 45 current snap is 68 2021-02-10T04:30:46.071 INFO:tasks.rados.rados.0.smithi085.stdout:684: write oid 24 current snap is 68 2021-02-10T04:30:46.071 INFO:tasks.rados.rados.0.smithi085.stdout:684: seq_num 267 ranges {487469=563838,1607751=551752} 2021-02-10T04:30:46.075 INFO:tasks.rados.rados.0.smithi085.stdout:684: writing smithi08575397-24 from 487469 to 1051307 tid 1 2021-02-10T04:30:46.082 INFO:tasks.rados.rados.0.smithi085.stdout:684: writing smithi08575397-24 from 1607751 to 2159503 tid 2 2021-02-10T04:30:46.082 INFO:tasks.rados.rados.0.smithi085.stdout:685: copy_from oid 39 from oid 41 current snap is 68 2021-02-10T04:30:46.082 INFO:tasks.rados.rados.0.smithi085.stdout:682: finishing rollback tid 0 to smithi08575397-21 2021-02-10T04:30:46.083 INFO:tasks.rados.rados.0.smithi085.stdout:682: finishing rollback tid 1 to smithi08575397-21 2021-02-10T04:30:46.083 INFO:tasks.rados.rados.0.smithi085.stdout:682: finishing rollback tid 2 to smithi08575397-21 2021-02-10T04:30:46.083 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 310 (ObjNum 194 snap 43 seq_num 194) dirty exists 2021-02-10T04:30:46.083 INFO:tasks.rados.rados.0.smithi085.stdout:684: finishing write tid 1 to smithi08575397-24 2021-02-10T04:30:46.083 INFO:tasks.rados.rados.0.smithi085.stdout:682: done (13 left) 2021-02-10T04:30:46.083 INFO:tasks.rados.rados.0.smithi085.stdout:686: write (excl) oid 1 current snap is 68 2021-02-10T04:30:46.083 INFO:tasks.rados.rados.0.smithi085.stdout:686: seq_num 268 ranges {504276=639087,1666233=695410} 2021-02-10T04:30:46.089 INFO:tasks.rados.rados.0.smithi085.stdout:686: writing smithi08575397-1 from 504276 to 1143363 tid 1 2021-02-10T04:30:46.097 INFO:tasks.rados.rados.0.smithi085.stdout:686: writing smithi08575397-1 from 1666233 to 2361643 tid 2 2021-02-10T04:30:46.097 INFO:tasks.rados.rados.0.smithi085.stdout:687: read oid 22 snap 67 2021-02-10T04:30:46.097 INFO:tasks.rados.rados.0.smithi085.stdout:687: expect (ObjNum 127 snap 23 seq_num 127) 2021-02-10T04:30:46.097 INFO:tasks.rados.rados.0.smithi085.stdout:688: rollback oid 38 current snap is 68 2021-02-10T04:30:46.097 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 38 to 64 2021-02-10T04:30:46.098 INFO:tasks.rados.rados.0.smithi085.stdout:684: finishing write tid 2 to smithi08575397-24 2021-02-10T04:30:46.098 INFO:tasks.rados.rados.0.smithi085.stdout:684: finishing write tid 3 to smithi08575397-24 2021-02-10T04:30:46.098 INFO:tasks.rados.rados.0.smithi085.stdout:684: finishing write tid 4 to smithi08575397-24 2021-02-10T04:30:46.098 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 24 v 65 (ObjNum 267 snap 68 seq_num 267) dirty exists 2021-02-10T04:30:46.098 INFO:tasks.rados.rados.0.smithi085.stdout:684: left oid 24 (ObjNum 267 snap 68 seq_num 267) 2021-02-10T04:30:46.098 INFO:tasks.rados.rados.0.smithi085.stdout:686: finishing write tid 1 to smithi08575397-1 2021-02-10T04:30:46.098 INFO:tasks.rados.rados.0.smithi085.stdout:684: done (15 left) 2021-02-10T04:30:46.098 INFO:tasks.rados.rados.0.smithi085.stdout:689: delete oid 43 current snap is 68 2021-02-10T04:30:46.100 INFO:tasks.rados.rados.0.smithi085.stdout:688: finishing rollback tid 0 to smithi08575397-38 2021-02-10T04:30:46.100 INFO:tasks.rados.rados.0.smithi085.stdout:688: finishing rollback tid 1 to smithi08575397-38 2021-02-10T04:30:46.101 INFO:tasks.rados.rados.0.smithi085.stdout:689: done (15 left) 2021-02-10T04:30:46.101 INFO:tasks.rados.rados.0.smithi085.stdout:690: read oid 50 snap -1 2021-02-10T04:30:46.101 INFO:tasks.rados.rados.0.smithi085.stdout:690: expect (ObjNum 198 snap 43 seq_num 198) 2021-02-10T04:30:46.101 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:30:46.102 INFO:tasks.rados.rados.0.smithi085.stdout:688: finishing rollback tid 2 to smithi08575397-38 2021-02-10T04:30:46.102 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 207 (ObjNum 260 snap 63 seq_num 260) dirty exists 2021-02-10T04:30:46.102 INFO:tasks.rados.rados.0.smithi085.stdout:688: done (15 left) 2021-02-10T04:30:46.102 INFO:tasks.rados.rados.0.smithi085.stdout:691: write oid 36 current snap is 68 2021-02-10T04:30:46.102 INFO:tasks.rados.rados.0.smithi085.stdout:691: seq_num 270 ranges {537890=789585,1783197=699079} 2021-02-10T04:30:46.110 INFO:tasks.rados.rados.0.smithi085.stdout:691: writing smithi08575397-36 from 537890 to 1327475 tid 1 2021-02-10T04:30:46.119 INFO:tasks.rados.rados.0.smithi085.stdout:691: writing smithi08575397-36 from 1783197 to 2482276 tid 2 2021-02-10T04:30:46.119 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:30:46.119 INFO:tasks.rados.rados.0.smithi085.stdout:687: expect (ObjNum 127 snap 23 seq_num 127) 2021-02-10T04:30:46.210 INFO:tasks.rados.rados.0.smithi085.stdout:686: finishing write tid 2 to smithi08575397-1 2021-02-10T04:30:46.210 INFO:tasks.rados.rados.0.smithi085.stdout:686: finishing write tid 3 to smithi08575397-1 2021-02-10T04:30:46.210 INFO:tasks.rados.rados.0.smithi085.stdout:687: done (15 left) 2021-02-10T04:30:46.210 INFO:tasks.rados.rados.0.smithi085.stdout:692: read oid 3 snap -1 2021-02-10T04:30:46.210 INFO:tasks.rados.rados.0.smithi085.stdout:692: expect deleted 2021-02-10T04:30:46.210 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:30:46.210 INFO:tasks.rados.rados.0.smithi085.stdout:686: finishing write tid 4 to smithi08575397-1 2021-02-10T04:30:46.210 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 230 (ObjNum 268 snap 68 seq_num 268) dirty exists 2021-02-10T04:30:46.210 INFO:tasks.rados.rados.0.smithi085.stdout:686: left oid 1 (ObjNum 268 snap 68 seq_num 268) 2021-02-10T04:30:46.211 INFO:tasks.rados.rados.0.smithi085.stdout:690: expect (ObjNum 198 snap 43 seq_num 198) 2021-02-10T04:30:46.268 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:30:46.325 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:46 smithi164 conmon[29288]: cluster 2021-02-10T04:30:45.317764+0000 osd.4 (osd.4 2021-02-10T04:30:46.325 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:46 smithi164 conmon[29288]: ) 127 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:46.326 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:46 smithi164 conmon[29288]: cluster 2021-02-10T04:30:45.317770+0000 osd.4 (osd.4) 128 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:46.326 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:46 smithi164 conmon[29288]: 2021-02-10T04:30:46.326 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:46 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:30:46.326 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:46 smithi164 conmon[29288]: -02-10T04:30:45.317776+0000 osd.4 (osd.4) 129 2021-02-10T04:30:46.326 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:46 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:46.326 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:46 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:30:46.326 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:46 smithi164 conmon[29288]: 02-10T04:30:45.317781+0000 osd.4 (osd.4) 130 2021-02-10T04:30:46.326 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:46 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:46.326 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:46 smithi164 conmon[29288]: cluster 2021-02-10T04:30:46.326 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:46 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:30:46.326 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:46 smithi164 conmon[29288]: 30:46.024228+0000 mgr.y ( 2021-02-10T04:30:46.326 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:46 smithi164 conmon[29288]: mgr.14140) 414 : cluster 2021-02-10T04:30:46.326 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:46 smithi164 conmon[29288]: [DBG] pgmap v493: 168 pgs: 1 active+clean+scrubbing, 1 active+clean+snaptrim, 166 active+clean; 139 MiB data, 467 MiB used, 715 GiB / 715 GiB avail; 5.6 MiB/s rd, 806 KiB/s wr, 10 op/s 2021-02-10T04:30:46.327 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:46 smithi164 conmon[29288]: cluster 2021-02-10T04:30:46.327 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:46 smithi164 conmon[29288]: 2021-02-10T04:30 2021-02-10T04:30:46.327 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:46 smithi164 conmon[29288]: :46.068038+0000 mon.a (mon.0) 948 2021-02-10T04:30:46.327 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:46 smithi164 conmon[29288]: : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2021-02-10T04:30:46.327 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:30:46 smithi164 conmon[44544]: debug 2021-02-10T04:30:46.134+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:46.327 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:30:46 smithi164 conmon[33677]: debug 2021-02-10T04:30:46.324+0000 7fbec19a1700 -1 osd.4 230 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:30:46.380 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:30:46.397 INFO:tasks.rados.rados.0.smithi085.stdout:691: finishing write tid 1 to smithi08575397-36 2021-02-10T04:30:46.397 INFO:tasks.rados.rados.0.smithi085.stdout:691: finishing write tid 2 to smithi08575397-36 2021-02-10T04:30:46.397 INFO:tasks.rados.rados.0.smithi085.stdout:691: finishing write tid 3 to smithi08575397-36 2021-02-10T04:30:46.397 INFO:tasks.rados.rados.0.smithi085.stdout:691: finishing write tid 4 to smithi08575397-36 2021-02-10T04:30:46.397 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 113 (ObjNum 270 snap 68 seq_num 270) dirty exists 2021-02-10T04:30:46.397 INFO:tasks.rados.rados.0.smithi085.stdout:691: left oid 36 (ObjNum 270 snap 68 seq_num 270) 2021-02-10T04:30:46.397 INFO:tasks.rados.rados.0.smithi085.stdout:686: done (15 left) 2021-02-10T04:30:46.397 INFO:tasks.rados.rados.0.smithi085.stdout:690: done (14 left) 2021-02-10T04:30:46.398 INFO:tasks.rados.rados.0.smithi085.stdout:691: done (13 left) 2021-02-10T04:30:46.398 INFO:tasks.rados.rados.0.smithi085.stdout:692: done (12 left) 2021-02-10T04:30:46.398 INFO:tasks.rados.rados.0.smithi085.stdout:693: read oid 36 snap -1 2021-02-10T04:30:46.398 INFO:tasks.rados.rados.0.smithi085.stdout:693: expect (ObjNum 270 snap 68 seq_num 270) 2021-02-10T04:30:46.398 INFO:tasks.rados.rados.0.smithi085.stdout:694: delete oid 30 current snap is 68 2021-02-10T04:30:46.400 INFO:tasks.rados.rados.0.smithi085.stdout:694: done (13 left) 2021-02-10T04:30:46.401 INFO:tasks.rados.rados.0.smithi085.stdout:695: snap_remove snap 67 2021-02-10T04:30:46.408 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:46.551 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.0 dump_ops_in_flight 2021-02-10T04:30:46.578 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[29319]: cluster 2021-02-10T04:30:45.317764+0000 osd.4 (osd.4) 127 : cluster 2021-02-10T04:30:46.579 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:46.579 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[29319]: cluster 2021-02-10T04:30:45.317770+0000 osd.4 (osd.4) 128 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:46.579 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[29319]: cluster 2021-02-10T04:30:45.317776+0000 2021-02-10T04:30:46.579 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[29319]: osd.4 (osd.4) 129 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:46.579 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[29319]: cluster 2021-02-10T04:30:45.317781+0000 osd.4 (osd.4) 130 : cluster [WRN] 2021-02-10T04:30:46.579 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:46.579 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[29319]: cluster 2021-02-10T04:30:46.024228 2021-02-10T04:30:46.579 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 414 : cluster [DBG] pgmap v493: 168 pgs: 1 active+clean+scrubbing, 1 active+clean+snaptrim, 166 active+clean; 139 MiB data, 467 MiB used, 715 GiB / 715 GiB avail; 5.6 MiB/s rd, 806 KiB/s wr, 10 op/s 2021-02-10T04:30:46.579 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:30:46.579 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[29319]: 10T04:30:46.068038+0000 mon.a (mon 2021-02-10T04:30:46.579 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[29319]: .0) 948 : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2021-02-10T04:30:46.580 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[35961]: cluster 2021-02-10T04:30:45. 2021-02-10T04:30:46.580 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[35961]: 317764+0000 osd.4 (osd.4) 127 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:46.580 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[35961]: cluster 2021-02-10T04:30:46.580 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[35961]: 2021-02-10T04:30:45.317770+0000 osd.4 (osd.4) 128 : cluster [WRN] 2021-02-10T04:30:46.580 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:46.580 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:30:46.580 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[35961]: 30:45.317776+0000 osd.4 2021-02-10T04:30:46.580 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[35961]: (osd.4) 129 : cluster 2021-02-10T04:30:46.580 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:46.581 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:30:46.581 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[35961]: -02-10T04:30: 2021-02-10T04:30:46.581 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[35961]: 45.317781+0000 osd.4 (osd. 2021-02-10T04:30:46.581 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[35961]: 4) 130 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:46.581 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[35961]: 2021-02-10T04:30:46.581 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:30:46.581 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[35961]: 10T04:30:46.024228+0000 mgr.y (mgr 2021-02-10T04:30:46.581 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[35961]: .14140) 414 : cluster [DBG] pgmap v493: 168 pgs: 1 active+clean+scrubbing, 1 active+clean+snaptrim, 166 active+clean; 139 MiB data, 467 MiB used, 715 GiB / 715 GiB avail; 5.6 MiB/s rd, 806 KiB/s wr, 10 op/s 2021-02-10T04:30:46.581 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[35961]: cluster 2021-02-10T04:30:46.068038+0000 mon.a (mon.0) 948 : cluster 2021-02-10T04:30:46.581 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[35961]: [DBG] osdmap e230: 8 total, 8 up, 8 in 2021-02-10T04:30:46.598 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:30:46.600 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:30:46 smithi164 conmon[40892]: debug 2021-02-10T04:30:46.568+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:46.706 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:30:46.750 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:46.857 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:30:47.106 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:30:47.191 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[45443]: debug 2021-02-10T04:30:46.845+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:47.191 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[45443]: debug 2021-02-10T04:30:47.191 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[45443]: 2021-02-10T04:30:46.932+0000 7f7125cf8700 -1 osd.1 pg_epoch: 230 pg[4.4( v 63'31 (0'0,63'31] local-lis/les=55/56 n=14 ec=55/55 lis/c=55/55 les/c/f=56/56/0 sis=55) [1,2,3] r=0 lpr=55 crt=63'31 lcod 63'30 mlcod 63'30 active+clean] scrubber pg(4.4) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 3 (0x55dbc827a000) 2021-02-10T04:30:47.191 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:46 smithi085 conmon[45443]: 2021-02-10T04:30:47.288 INFO:tasks.rados.rados.0.smithi085.stdout:695: done (13 left) 2021-02-10T04:30:47.288 INFO:tasks.rados.rados.0.smithi085.stdout:696: write (excl) oid 26 current snap is 68 2021-02-10T04:30:47.288 INFO:tasks.rados.rados.0.smithi085.stdout:696: seq_num 272 ranges {571504=540083,1783808=702748} 2021-02-10T04:30:47.290 INFO:tasks.rados.rados.0.smithi085.stdout:696: writing smithi08575397-26 from 571504 to 1111587 tid 1 2021-02-10T04:30:47.298 INFO:tasks.rados.rados.0.smithi085.stdout:696: writing smithi08575397-26 from 1783808 to 2486556 tid 2 2021-02-10T04:30:47.298 INFO:tasks.rados.rados.0.smithi085.stdout:697: delete oid 1 current snap is 68 2021-02-10T04:30:47.298 INFO:tasks.rados.rados.0.smithi085.stdout:693: expect (ObjNum 270 snap 68 seq_num 270) 2021-02-10T04:30:47.460 INFO:tasks.rados.rados.0.smithi085.stdout:696: finishing write tid 1 to smithi08575397-26 2021-02-10T04:30:47.460 INFO:tasks.rados.rados.0.smithi085.stdout:693: done (14 left) 2021-02-10T04:30:47.460 INFO:tasks.rados.rados.0.smithi085.stdout:697: done (13 left) 2021-02-10T04:30:47.460 INFO:tasks.rados.rados.0.smithi085.stdout:698: read oid 27 snap -1 2021-02-10T04:30:47.460 INFO:tasks.rados.rados.0.smithi085.stdout:698: expect (ObjNum 232 snap 53 seq_num 232) 2021-02-10T04:30:47.460 INFO:tasks.rados.rados.0.smithi085.stdout:699: write (excl) oid 5 current snap is 68 2021-02-10T04:30:47.460 INFO:tasks.rados.rados.0.smithi085.stdout:699: seq_num 274 ranges {605118=406934,1617125=706417} 2021-02-10T04:30:47.466 INFO:tasks.rados.rados.0.smithi085.stdout:699: writing smithi08575397-5 from 605118 to 1012052 tid 1 2021-02-10T04:30:47.474 INFO:tasks.rados.rados.0.smithi085.stdout:699: writing smithi08575397-5 from 1617125 to 2323542 tid 2 2021-02-10T04:30:47.474 INFO:tasks.rados.rados.0.smithi085.stdout:700: snap_remove snap 64 2021-02-10T04:30:47.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:30:47 smithi164 conmon[33677]: debug 2021-02-10T04:30:47.319+0000 7fbec19a1700 -1 osd.4 231 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:30:47.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:47 smithi164 conmon[29288]: cluster 2021-02-10T04:30:46.325386+0000 osd.4 (osd. 2021-02-10T04:30:47.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:47 smithi164 conmon[29288]: 4) 131 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:47.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:47 smithi164 conmon[29288]: cluster 2021-02-10T04:30:46.325396+0000 osd.4 (osd.4) 132 : cluster [WRN] 2021-02-10T04:30:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:47 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:47 smithi164 conmon[29288]: cluster 2021-02-10T04:30:46.325402+0000 osd.4 (osd.4 2021-02-10T04:30:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:47 smithi164 conmon[29288]: ) 133 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:47 smithi164 conmon[29288]: cluster 2021-02-10T04:30:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:47 smithi164 conmon[29288]: 2021-02-10T04:30:46.325407+0000 osd.4 (osd. 2021-02-10T04:30:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:47 smithi164 conmon[29288]: 4) 134 : cluster [WRN] 2021-02-10T04:30:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:47 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:47.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:47 smithi085 conmon[29319]: cluster 2021-02-10T04:30:46.325386+0000 osd.4 (osd.4 2021-02-10T04:30:47.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:47 smithi085 conmon[29319]: ) 131 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:47.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:47 smithi085 conmon[29319]: cluster 2021-02-10T04:30:46.325396+0000 osd.4 (osd.4) 132 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:47.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:47 smithi085 conmon[29319]: cluster 2021-02-10T04:30:47.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:47 smithi085 conmon[29319]: 2021-02-10T04:30:46.325402+0000 osd.4 (osd.4) 133 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:47.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:47 smithi085 conmon[29319]: cluster 2021-02-10T04:30:46.325407+0000 osd.4 (osd.4) 2021-02-10T04:30:47.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:47 smithi085 conmon[29319]: 134 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:47.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:47 smithi085 conmon[35961]: cluster 2021-02-10T04:30:47.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:47 smithi085 conmon[35961]: 2021-02-10T04:30:46.325386+0000 osd.4 (osd.4) 131 : cluster 2021-02-10T04:30:47.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:47 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:47.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:47 smithi085 conmon[35961]: cluster 2021-02-10T04:30:47.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:47 smithi085 conmon[35961]: 2021-02-10T04:30:46.325396+0000 2021-02-10T04:30:47.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:47 smithi085 conmon[35961]: osd.4 (osd.4) 132 : cluster [WRN] 2021-02-10T04:30:47.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:47 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:47.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:47 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:30:47.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:47 smithi085 conmon[35961]: :30:46.325402+0000 osd.4 (osd.4) 133 : cluster [WRN] 2021-02-10T04:30:47.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:47 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:47.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:47 smithi085 conmon[35961]: cluster 2021-02-10T04:30:46 2021-02-10T04:30:47.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:47 smithi085 conmon[35961]: .325407+0000 osd.4 (osd.4) 134 : cluster [WRN] 2021-02-10T04:30:47.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:47 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:48.143 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:30:47 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:30:48.191 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:47 smithi085 conmon[45443]: debug 2021-02-10T04:30:48.191 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:30:47 smithi085 conmon[45443]: 2021-02-10T04:30:47.883+0000 7f7125cf8700 -1 osd.1 pg_epoch: 231 pg[4.4( v 63'31 (0'0,63'31] local-lis/les=55/56 n=14 ec=55/55 lis/c=55/55 les/c/f=56/56/0 sis=55) [1,2,3] r=0 lpr=55 crt=63'31 lcod 63'30 mlcod 63'30 active+clean] scrubber pg(4.4) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 3 (0x55dbcb0c78c0) 2021-02-10T04:30:48.287 INFO:tasks.rados.rados.0.smithi085.stdout:700: done (15 left) 2021-02-10T04:30:48.287 INFO:tasks.rados.rados.0.smithi085.stdout:701: write (excl) oid 40 current snap is 68 2021-02-10T04:30:48.287 INFO:tasks.rados.rados.0.smithi085.stdout:701: seq_num 275 ranges {621925=482183,1559254=566428} 2021-02-10T04:30:48.293 INFO:tasks.rados.rados.0.smithi085.stdout:701: writing smithi08575397-40 from 621925 to 1104108 tid 1 2021-02-10T04:30:48.300 INFO:tasks.rados.rados.0.smithi085.stdout:701: writing smithi08575397-40 from 1559254 to 2125682 tid 2 2021-02-10T04:30:48.300 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:30:48.300 INFO:tasks.rados.rados.0.smithi085.stdout:696: finishing write tid 2 to smithi08575397-26 2021-02-10T04:30:48.301 INFO:tasks.rados.rados.0.smithi085.stdout:696: finishing write tid 3 to smithi08575397-26 2021-02-10T04:30:48.301 INFO:tasks.rados.rados.0.smithi085.stdout:696: finishing write tid 4 to smithi08575397-26 2021-02-10T04:30:48.301 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 47 (ObjNum 272 snap 68 seq_num 272) dirty exists 2021-02-10T04:30:48.301 INFO:tasks.rados.rados.0.smithi085.stdout:696: left oid 26 (ObjNum 272 snap 68 seq_num 272) 2021-02-10T04:30:48.301 INFO:tasks.rados.rados.0.smithi085.stdout:696: done (15 left) 2021-02-10T04:30:48.301 INFO:tasks.rados.rados.0.smithi085.stdout:702: read oid 43 snap 59 2021-02-10T04:30:48.301 INFO:tasks.rados.rados.0.smithi085.stdout:702: expect (ObjNum 255 snap 59 seq_num 255) 2021-02-10T04:30:48.301 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:30:48.301 INFO:tasks.rados.rados.0.smithi085.stdout:698: expect (ObjNum 232 snap 53 seq_num 232) 2021-02-10T04:30:48.473 INFO:tasks.rados.rados.0.smithi085.stdout:699: finishing write tid 1 to smithi08575397-5 2021-02-10T04:30:48.473 INFO:tasks.rados.rados.0.smithi085.stdout:699: finishing write tid 2 to smithi08575397-5 2021-02-10T04:30:48.473 INFO:tasks.rados.rados.0.smithi085.stdout:698: done (15 left) 2021-02-10T04:30:48.473 INFO:tasks.rados.rados.0.smithi085.stdout:703: write (excl) oid 24 current snap is 68 2021-02-10T04:30:48.473 INFO:tasks.rados.rados.0.smithi085.stdout:699: finishing write tid 3 to smithi08575397-5 2021-02-10T04:30:48.473 INFO:tasks.rados.rados.0.smithi085.stdout:699: finishing write tid 4 to smithi08575397-5 2021-02-10T04:30:48.473 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 207 (ObjNum 274 snap 68 seq_num 274) dirty exists 2021-02-10T04:30:48.473 INFO:tasks.rados.rados.0.smithi085.stdout:699: left oid 5 (ObjNum 274 snap 68 seq_num 274) 2021-02-10T04:30:48.473 INFO:tasks.rados.rados.0.smithi085.stdout:703: seq_num 276 ranges {638732=557432,1617736=710086} 2021-02-10T04:30:48.480 INFO:tasks.rados.rados.0.smithi085.stdout:703: writing smithi08575397-24 from 638732 to 1196164 tid 1 2021-02-10T04:30:48.487 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:30:48.488 INFO:tasks.rados.rados.0.smithi085.stdout:703: writing smithi08575397-24 from 1617736 to 2327822 tid 2 2021-02-10T04:30:48.488 INFO:tasks.rados.rados.0.smithi085.stdout:699: done (15 left) 2021-02-10T04:30:48.489 INFO:tasks.rados.rados.0.smithi085.stdout:704: write (excl) oid 13 current snap is 68 2021-02-10T04:30:48.489 INFO:tasks.rados.rados.0.smithi085.stdout:701: finishing write tid 1 to smithi08575397-40 2021-02-10T04:30:48.489 INFO:tasks.rados.rados.0.smithi085.stdout:701: finishing write tid 2 to smithi08575397-40 2021-02-10T04:30:48.489 INFO:tasks.rados.rados.0.smithi085.stdout:701: finishing write tid 3 to smithi08575397-40 2021-02-10T04:30:48.489 INFO:tasks.rados.rados.0.smithi085.stdout:701: finishing write tid 4 to smithi08575397-40 2021-02-10T04:30:48.489 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 315 (ObjNum 275 snap 68 seq_num 275) dirty exists 2021-02-10T04:30:48.489 INFO:tasks.rados.rados.0.smithi085.stdout:701: left oid 40 (ObjNum 275 snap 68 seq_num 275) 2021-02-10T04:30:48.489 INFO:tasks.rados.rados.0.smithi085.stdout:702: expect (ObjNum 255 snap 59 seq_num 255) 2021-02-10T04:30:48.507 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[52836]: debug 2021-02-10T04:30:48.266+0000 7fba73fb6700 -1 osd.3 pg_epoch: 231 pg[4.14( v 63'26 (0'0,63'26] local-lis/les=71/72 n=11 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,2,7] r=0 lpr=71 crt=63'26 lcod 63'25 mlcod 0'0 active+clean] scrubber pg(4.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55fec0846c60) 2021-02-10T04:30:48.509 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: cluster 2021-02-10T04:30:48.509 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: 2021-02-10T04:30:48.509 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: 2021 2021-02-10T04:30:48.509 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: - 2021-02-10T04:30:48.509 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: 02 2021-02-10T04:30:48.509 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: - 2021-02-10T04:30:48.509 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: 10 2021-02-10T04:30:48.510 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: T 2021-02-10T04:30:48.510 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: 04 2021-02-10T04:30:48.510 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: : 2021-02-10T04:30:48.510 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: 30 2021-02-10T04:30:48.510 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: : 2021-02-10T04:30:48.510 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: 46 2021-02-10T04:30:48.510 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: . 2021-02-10T04:30:48.510 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: 597414 2021-02-10T04:30:48.510 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: +0000 2021-02-10T04:30:48.510 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: 2021-02-10T04:30:48.510 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: osd.1 2021-02-10T04:30:48.510 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: ( 2021-02-10T04:30:48.510 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: osd 2021-02-10T04:30:48.511 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: . 2021-02-10T04:30:48.511 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: 1) 92 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:30:48.511 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:30:48.511 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: 02-10T04:30:46.597623+0000 osd.1 (osd.1) 93 2021-02-10T04:30:48.511 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:30:48.511 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:30:48.511 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: 02-10T04:30:47.278882+0000 2021-02-10T04:30:48.511 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: mon.a (mon.0) 949 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2021-02-10T04:30:48.511 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: 2021-02-10T04:30:48.511 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:30:48.511 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: 30:47.320013+0000 osd.4 (osd.4) 135 2021-02-10T04:30:48.511 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:48.511 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:30:48.512 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: T04:30:47.320021+0000 osd.4 (osd 2021-02-10T04:30:48.512 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: .4) 136 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:48.512 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:30:48.512 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: 10T04:30:47.320026+0000 osd.4 (osd. 2021-02-10T04:30:48.512 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: 4) 137 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:48.512 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:30:48.512 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: T04:30:47.320029+0000 osd.4 (osd. 2021-02-10T04:30:48.512 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: 4) 138 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:48.512 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:30:48.512 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: 10T04:30:47.805063+0000 mgr.y (mgr 2021-02-10T04:30:48.512 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: .14140) 415 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:30:48.512 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: cluster 2021-02-10T04:30:48. 2021-02-10T04:30:48.512 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: 024643+0000 mgr.y (mgr.14140) 416 : cluster [DBG] 2021-02-10T04:30:48.512 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: pgmap v496: 168 pgs: 1 active+clean+scrubbing, 1 active+clean+snaptrim, 166 active+clean; 139 MiB data, 468 MiB used, 715 GiB / 715 GiB avail; 5.6 MiB/s rd, 805 KiB/s wr, 9 op/s 2021-02-10T04:30:48.513 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: cluster 2021-02-10T04:30:48.513 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: 2021-02-10T04:30:48.060869+0000 mon.a (mon. 2021-02-10T04:30:48.513 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[29319]: 0) 950 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 92 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:30:48.513 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: cluster 2021-02-10T04:30:48.513 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: 2021-02-10T04:30:48.513 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: 2021 2021-02-10T04:30:48.513 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: - 2021-02-10T04:30:48.513 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: 02 2021-02-10T04:30:48.513 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: -10T04:30:46.597414+0000 osd.1 ( 2021-02-10T04:30:48.514 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: osd.1) 92 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:30:48.514 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:30:48.514 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: :30:46.597623+0000 osd.1 (osd.1) 93 2021-02-10T04:30:48.514 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:30:48.514 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:30:48.514 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: 10T04:30:47.278882+0000 mon.a ( 2021-02-10T04:30:48.514 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: mon.0) 949 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2021-02-10T04:30:48.514 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:30:48.514 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: 02-10T04:30:47.320013+0000 osd.4 2021-02-10T04:30:48.514 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: (osd.4) 135 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:48.514 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: 2021-02-10T04:30:48.514 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:30:48.514 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: -02-10T04:30:47.320021+0000 osd.4 (osd. 2021-02-10T04:30:48.515 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: 4) 136 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:48.515 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:30:48.515 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: 02-10T04:30:47.320026+0000 2021-02-10T04:30:48.515 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: osd.4 (osd.4) 137 : cluster [WRN] 2021-02-10T04:30:48.515 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:48.515 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: cluster 2021-02-10T04:30:48.515 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: 2021 2021-02-10T04:30:48.515 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: -02-10T04:30:47. 2021-02-10T04:30:48.515 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: 320029+0000 osd.4 (osd.4) 138 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:48.515 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: audit 2021-02-10T04:30:47.805063 2021-02-10T04:30:48.515 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 415 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:30:48.515 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: 2021-02-10T04:30:48.515 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:30:48.516 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: -02-10T04:30:48.024643 2021-02-10T04:30:48.516 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 416 : cluster [DBG] pgmap v496: 168 pgs: 1 active+clean+scrubbing, 1 active+clean+snaptrim, 166 active+clean; 139 MiB data, 468 MiB used, 715 GiB / 715 GiB avail; 5.6 MiB/s rd, 805 KiB/s wr, 9 op/s 2021-02-10T04:30:48.516 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:30:48.516 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: T04:30:48.060869+0000 mon.a (mon.0) 950 : cluster 2021-02-10T04:30:48.516 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[35961]: [WRN] Health check update: 4 slow ops, oldest one blocked for 92 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:30:48.536 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:48.632 INFO:tasks.rados.rados.0.smithi085.stdout:704: seq_num 277 ranges {655539=632681,2076218=570097} 2021-02-10T04:30:48.638 INFO:tasks.rados.rados.0.smithi085.stdout:704: writing smithi08575397-13 from 655539 to 1288220 tid 1 2021-02-10T04:30:48.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:30:48 smithi164 conmon[33677]: debug 2021-02-10T04:30:48.350+0000 7fbec19a1700 -1 osd.4 232 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:30:48.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:48 smithi164 conmon[29288]: cluster 2021-02-10T04:30:46.597414+0000 osd.1 (osd.1 2021-02-10T04:30:48.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:48 smithi164 conmon[29288]: ) 92 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:30:48.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:48 smithi164 conmon[29288]: cluster 2021-02-10T04:30:46.597623+0000 osd.1 (osd.1) 93 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:30:48.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:48 smithi164 conmon[29288]: cluster 2021-02-10T04:30:47.278882+0000 mon.a (mon.0) 2021-02-10T04:30:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:48 smithi164 conmon[29288]: 949 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2021-02-10T04:30:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:48 smithi164 conmon[29288]: cluster 2021-02-10T04:30: 2021-02-10T04:30:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:48 smithi164 conmon[29288]: 47.320013+0000 osd.4 (osd.4 2021-02-10T04:30:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:48 smithi164 conmon[29288]: ) 135 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:48 smithi164 conmon[29288]: cluster 2021-02-10T04:30:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:48 smithi164 conmon[29288]: 2021-02-10T04:30: 2021-02-10T04:30:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:48 smithi164 conmon[29288]: 47.320021+0000 osd.4 (osd. 2021-02-10T04:30:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:48 smithi164 conmon[29288]: 4) 136 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:48 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:30:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:48 smithi164 conmon[29288]: -10T04:30:47.320026+0000 2021-02-10T04:30:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:48 smithi164 conmon[29288]: osd.4 (osd.4) 137 2021-02-10T04:30:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:48 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:48 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:30:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:48 smithi164 conmon[29288]: 30:47.320029+0000 osd.4 (osd.4 2021-02-10T04:30:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:48 smithi164 conmon[29288]: ) 138 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:48 smithi164 conmon[29288]: audit 2021 2021-02-10T04:30:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:48 smithi164 conmon[29288]: -02-10T04:30:47. 2021-02-10T04:30:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:48 smithi164 conmon[29288]: 805063+0000 mgr.y (mgr.14140) 415 : audit [DBG] 2021-02-10T04:30:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:48 smithi164 conmon[29288]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:30:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:48 smithi164 conmon[29288]: cluster 2021-02-10T04:30:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:48 smithi164 conmon[29288]: 2021-02-10T04:30:48.024643+0000 mgr.y (mgr.14140 2021-02-10T04:30:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:48 smithi164 conmon[29288]: ) 416 : cluster [DBG] pgmap v496: 168 pgs: 1 active+clean+scrubbing, 1 active+clean+snaptrim, 166 active+clean; 139 MiB data, 468 MiB used, 715 GiB / 715 GiB avail; 5.6 MiB/s rd, 805 KiB/s wr, 9 op/s 2021-02-10T04:30:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:48 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:30:48.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:48 smithi164 conmon[29288]: 30:48.060869+0000 mon.a (mon.0) 950 2021-02-10T04:30:48.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:48 smithi164 conmon[29288]: : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 92 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:30:48.647 INFO:tasks.rados.rados.0.smithi085.stdout:704: writing smithi08575397-13 from 2076218 to 2646315 tid 2 2021-02-10T04:30:48.647 INFO:tasks.rados.rados.0.smithi085.stdout:701: done (15 left) 2021-02-10T04:30:48.647 INFO:tasks.rados.rados.0.smithi085.stdout:702: done (14 left) 2021-02-10T04:30:48.647 INFO:tasks.rados.rados.0.smithi085.stdout:705: snap_create 2021-02-10T04:30:48.648 INFO:tasks.rados.rados.0.smithi085.stdout:703: finishing write tid 1 to smithi08575397-24 2021-02-10T04:30:48.648 INFO:tasks.rados.rados.0.smithi085.stdout:703: finishing write tid 2 to smithi08575397-24 2021-02-10T04:30:48.648 INFO:tasks.rados.rados.0.smithi085.stdout:703: finishing write tid 3 to smithi08575397-24 2021-02-10T04:30:48.648 INFO:tasks.rados.rados.0.smithi085.stdout:703: finishing write tid 4 to smithi08575397-24 2021-02-10T04:30:48.648 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 24 v 68 (ObjNum 276 snap 68 seq_num 276) dirty exists 2021-02-10T04:30:48.648 INFO:tasks.rados.rados.0.smithi085.stdout:703: left oid 24 (ObjNum 276 snap 68 seq_num 276) 2021-02-10T04:30:48.648 INFO:tasks.rados.rados.0.smithi085.stdout:704: finishing write tid 1 to smithi08575397-13 2021-02-10T04:30:48.651 INFO:tasks.rados.rados.0.smithi085.stdout:704: finishing write tid 2 to smithi08575397-13 2021-02-10T04:30:48.651 INFO:tasks.rados.rados.0.smithi085.stdout:704: finishing write tid 3 to smithi08575397-13 2021-02-10T04:30:48.651 INFO:tasks.rados.rados.0.smithi085.stdout:704: finishing write tid 4 to smithi08575397-13 2021-02-10T04:30:48.652 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 210 (ObjNum 277 snap 68 seq_num 277) dirty exists 2021-02-10T04:30:48.652 INFO:tasks.rados.rados.0.smithi085.stdout:704: left oid 13 (ObjNum 277 snap 68 seq_num 277) 2021-02-10T04:30:48.676 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:30:48.707 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:30:48.741 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:30:48.778 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:30:48.820 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:30:48.840 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:30:48.843 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:30:48.869 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:30:48.902 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:30:48.905 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:30:48.905 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:30:48.905 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:30:48.905 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:30:48.905 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:30:48.906 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:30:48.906 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:30:48.906 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:30:48.977 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:30:49.016 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:30:48 smithi085 conmon[52836]: debug 2021-02-10T04:30:48.789+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:49.020 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:49.288 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:49 smithi085 conmon[29319]: cluster 2021-02-10T04:30:48.287720+0000 mon.a (mon 2021-02-10T04:30:49.288 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:49 smithi085 conmon[29319]: .0) 951 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2021-02-10T04:30:49.289 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:49 smithi085 conmon[29319]: cluster 2021-02-10T04:30:48.350783+0000 osd.4 (osd.4) 139 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:49.289 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:49 smithi085 conmon[29319]: cluster 2021-02-10T04:30:48.350793+0000 osd.4 (osd.4) 140 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:49.289 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:49 smithi085 conmon[29319]: cluster 2021-02-10T04:30:48.350798+0000 osd.4 (osd.4) 141 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:49.289 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:49 smithi085 conmon[29319]: cluster 2021-02-10T04:30:48.350804+0000 osd.4 (osd.4) 142 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:49.289 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:49 smithi085 conmon[29319]: audit 2021-02-10T04:30:48.642985+0000 mon.b (mon.2) 29 : audit [DBG] from='client.? 172.21.15.85:0/2471389252' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:30:49.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:49 smithi085 conmon[35961]: cluster 2021-02-10T04:30:48.287720+0000 2021-02-10T04:30:49.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:49 smithi085 conmon[35961]: mon.a (mon.0) 951 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2021-02-10T04:30:49.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:49 smithi085 conmon[35961]: cluster 2021-02-10T04:30:49.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:49 smithi085 conmon[35961]: 2021-02-10T04:30:48.350783+0000 osd.4 (osd.4) 139 2021-02-10T04:30:49.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:49 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:49.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:49 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:30:49.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:49 smithi085 conmon[35961]: 02-10T04:30:48.350793+0000 2021-02-10T04:30:49.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:49 smithi085 conmon[35961]: osd.4 (osd.4) 140 : cluster 2021-02-10T04:30:49.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:49 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:49.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:49 smithi085 conmon[35961]: cluster 2021-02-10T04:30 2021-02-10T04:30:49.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:49 smithi085 conmon[35961]: :48.350798+0000 osd.4 (osd.4) 141 2021-02-10T04:30:49.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:49 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:49.292 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:49 smithi085 conmon[35961]: cluster 2021-02-10T04:30:49.292 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:49 smithi085 conmon[35961]: 2021-02-10T04:30:48.350804+0000 osd.4 ( 2021-02-10T04:30:49.292 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:49 smithi085 conmon[35961]: osd.4) 142 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:49.292 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:49 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:30:49.292 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:49 smithi085 conmon[35961]: 04:30:48.642985+0000 mon.b (mon.2 2021-02-10T04:30:49.292 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:49 smithi085 conmon[35961]: ) 29 : audit [DBG] from='client.? 172.21.15.85:0/2471389252' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:30:49.308 INFO:tasks.rados.rados.0.smithi085.stdout:703: done (14 left) 2021-02-10T04:30:49.308 INFO:tasks.rados.rados.0.smithi085.stdout:704: done (13 left) 2021-02-10T04:30:49.308 INFO:tasks.rados.rados.0.smithi085.stdout:705: done (12 left) 2021-02-10T04:30:49.308 INFO:tasks.rados.rados.0.smithi085.stdout:706: write oid 11 current snap is 69 2021-02-10T04:30:49.308 INFO:tasks.rados.rados.0.smithi085.stdout:706: seq_num 278 ranges {672346=707930,2134700=537646} 2021-02-10T04:30:49.308 INFO:tasks.rados.rados.0.smithi085.stdout:706: writing smithi08575397-11 from 672346 to 1380276 tid 1 2021-02-10T04:30:49.313 INFO:tasks.rados.rados.0.smithi085.stdout:706: writing smithi08575397-11 from 2134700 to 2672346 tid 2 2021-02-10T04:30:49.314 INFO:tasks.rados.rados.0.smithi085.stdout:707: read oid 3 snap -1 2021-02-10T04:30:49.314 INFO:tasks.rados.rados.0.smithi085.stdout:707: expect deleted 2021-02-10T04:30:49.314 INFO:tasks.rados.rados.0.smithi085.stdout:708: read oid 23 snap -1 2021-02-10T04:30:49.314 INFO:tasks.rados.rados.0.smithi085.stdout:708: expect (ObjNum 124 snap 23 seq_num 124) 2021-02-10T04:30:49.314 INFO:tasks.rados.rados.0.smithi085.stdout:709: delete oid 27 current snap is 69 2021-02-10T04:30:49.315 INFO:tasks.rados.rados.0.smithi085.stdout:706: finishing write tid 1 to smithi08575397-11 2021-02-10T04:30:49.319 INFO:tasks.rados.rados.0.smithi085.stdout:706: finishing write tid 2 to smithi08575397-11 2021-02-10T04:30:49.319 INFO:tasks.rados.rados.0.smithi085.stdout:707: done (15 left) 2021-02-10T04:30:49.319 INFO:tasks.rados.rados.0.smithi085.stdout:709: done (14 left) 2021-02-10T04:30:49.319 INFO:tasks.rados.rados.0.smithi085.stdout:710: read oid 7 snap 68 2021-02-10T04:30:49.319 INFO:tasks.rados.rados.0.smithi085.stdout:710: expect (ObjNum 202 snap 44 seq_num 202) 2021-02-10T04:30:49.319 INFO:tasks.rados.rados.0.smithi085.stdout:711: read oid 2 snap -1 2021-02-10T04:30:49.319 INFO:tasks.rados.rados.0.smithi085.stdout:711: expect (ObjNum 225 snap 51 seq_num 225) 2021-02-10T04:30:49.319 INFO:tasks.rados.rados.0.smithi085.stdout:706: finishing write tid 3 to smithi08575397-11 2021-02-10T04:30:49.319 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:30:49.320 INFO:tasks.rados.rados.0.smithi085.stdout:706: finishing write tid 4 to smithi08575397-11 2021-02-10T04:30:49.320 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 321 (ObjNum 278 snap 69 seq_num 278) dirty exists 2021-02-10T04:30:49.320 INFO:tasks.rados.rados.0.smithi085.stdout:706: left oid 11 (ObjNum 278 snap 69 seq_num 278) 2021-02-10T04:30:49.320 INFO:tasks.rados.rados.0.smithi085.stdout:706: done (15 left) 2021-02-10T04:30:49.320 INFO:tasks.rados.rados.0.smithi085.stdout:712: snap_create 2021-02-10T04:30:49.321 INFO:tasks.rados.rados.0.smithi085.stdout:708: expect (ObjNum 124 snap 23 seq_num 124) 2021-02-10T04:30:49.422 INFO:tasks.rados.rados.0.smithi085.stdout:711: expect (ObjNum 225 snap 51 seq_num 225) 2021-02-10T04:30:49.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:49 smithi164 conmon[29288]: cluster 2021-02-10T04:30:48.287720+0000 2021-02-10T04:30:49.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:49 smithi164 conmon[29288]: mon.a (mon.0) 951 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2021-02-10T04:30:49.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:49 smithi164 conmon[29288]: cluster 2021-02-10T04:30:48.350783+0000 osd.4 (osd.4) 139 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:30:49.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:49 smithi164 conmon[29288]: cluster 2021-02-10T04:30:48.350793+0000 osd.4 (osd.4) 140 : cluster 2021-02-10T04:30:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:49 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:30:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:49 smithi164 conmon[29288]: cluster 2021-02-10T04:30:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:49 smithi164 conmon[29288]: 2021-02-10T04:30:48.350798+0000 2021-02-10T04:30:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:49 smithi164 conmon[29288]: osd.4 (osd.4) 141 : 2021-02-10T04:30:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:49 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:30:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:49 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:30:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:49 smithi164 conmon[29288]: 02-10T04:30:48.350804+0000 osd.4 ( 2021-02-10T04:30:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:49 smithi164 conmon[29288]: osd.4) 142 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:30:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:49 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:30:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:49 smithi164 conmon[29288]: -10T04:30:48.642985+0000 mon.b 2021-02-10T04:30:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:49 smithi164 conmon[29288]: (mon.2) 29 : audit [DBG] from='client.? 172.21.15.85:0/2471389252' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:30:49.645 INFO:tasks.rados.rados.0.smithi085.stdout:710: expect (ObjNum 202 snap 44 seq_num 202) 2021-02-10T04:30:49.690 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:30:49 smithi085 conmon[52836]: debug 2021-02-10T04:30:49.691 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:30:49 smithi085 conmon[52836]: 2021-02-10T04:30:49.306+0000 7fba73fb6700 -1 osd.3 pg_epoch: 233 pg[4.14( v 63'26 (0'0,63'26] local-lis/les=71/72 n=11 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,2,7] r=0 lpr=71 crt=63'26 lcod 63'25 mlcod 0'0 active+clean] scrubber pg(4.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55fec073d340) 2021-02-10T04:30:49.691 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:30:49 smithi085 conmon[52836]: 2021-02-10T04:30:50.307 INFO:tasks.rados.rados.0.smithi085.stdout:708: done (15 left) 2021-02-10T04:30:50.307 INFO:tasks.rados.rados.0.smithi085.stdout:710: done (14 left) 2021-02-10T04:30:50.307 INFO:tasks.rados.rados.0.smithi085.stdout:711: done (13 left) 2021-02-10T04:30:50.307 INFO:tasks.rados.rados.0.smithi085.stdout:712: done (12 left) 2021-02-10T04:30:50.307 INFO:tasks.rados.rados.0.smithi085.stdout:713: read oid 43 snap -1 2021-02-10T04:30:50.307 INFO:tasks.rados.rados.0.smithi085.stdout:713: expect deleted 2021-02-10T04:30:50.307 INFO:tasks.rados.rados.0.smithi085.stdout:714: write oid 40 current snap is 70 2021-02-10T04:30:50.307 INFO:tasks.rados.rados.0.smithi085.stdout:714: seq_num 280 ranges {705960=458428,1735311=717424} 2021-02-10T04:30:50.314 INFO:tasks.rados.rados.0.smithi085.stdout:714: writing smithi08575397-40 from 705960 to 1164388 tid 1 2021-02-10T04:30:50.323 INFO:tasks.rados.rados.0.smithi085.stdout:714: writing smithi08575397-40 from 1735311 to 2452735 tid 2 2021-02-10T04:30:50.323 INFO:tasks.rados.rados.0.smithi085.stdout:715: write oid 30 current snap is 70 2021-02-10T04:30:50.323 INFO:tasks.rados.rados.0.smithi085.stdout:715: seq_num 281 ranges {722767=533677,1793793=461082} 2021-02-10T04:30:50.331 INFO:tasks.rados.rados.0.smithi085.stdout:715: writing smithi08575397-30 from 722767 to 1256444 tid 1 2021-02-10T04:30:50.338 INFO:tasks.rados.rados.0.smithi085.stdout:715: writing smithi08575397-30 from 1793793 to 2254875 tid 2 2021-02-10T04:30:50.339 INFO:tasks.rados.rados.0.smithi085.stdout:716: delete oid 16 current snap is 70 2021-02-10T04:30:50.339 INFO:tasks.rados.rados.0.smithi085.stdout:714: finishing write tid 1 to smithi08575397-40 2021-02-10T04:30:50.339 INFO:tasks.rados.rados.0.smithi085.stdout:714: finishing write tid 2 to smithi08575397-40 2021-02-10T04:30:50.339 INFO:tasks.rados.rados.0.smithi085.stdout:714: finishing write tid 3 to smithi08575397-40 2021-02-10T04:30:50.339 INFO:tasks.rados.rados.0.smithi085.stdout:714: finishing write tid 4 to smithi08575397-40 2021-02-10T04:30:50.339 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 325 (ObjNum 280 snap 70 seq_num 280) dirty exists 2021-02-10T04:30:50.339 INFO:tasks.rados.rados.0.smithi085.stdout:714: left oid 40 (ObjNum 280 snap 70 seq_num 280) 2021-02-10T04:30:50.339 INFO:tasks.rados.rados.0.smithi085.stdout:715: finishing write tid 1 to smithi08575397-30 2021-02-10T04:30:50.342 INFO:tasks.rados.rados.0.smithi085.stdout:715: finishing write tid 2 to smithi08575397-30 2021-02-10T04:30:50.343 INFO:tasks.rados.rados.0.smithi085.stdout:715: finishing write tid 3 to smithi08575397-30 2021-02-10T04:30:50.343 INFO:tasks.rados.rados.0.smithi085.stdout:715: finishing write tid 4 to smithi08575397-30 2021-02-10T04:30:50.343 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 216 (ObjNum 281 snap 70 seq_num 281) dirty exists 2021-02-10T04:30:50.343 INFO:tasks.rados.rados.0.smithi085.stdout:715: left oid 30 (ObjNum 281 snap 70 seq_num 281) 2021-02-10T04:30:50.343 INFO:tasks.rados.rados.0.smithi085.stdout:713: done (15 left) 2021-02-10T04:30:50.344 INFO:tasks.rados.rados.0.smithi085.stdout:714: done (14 left) 2021-02-10T04:30:50.344 INFO:tasks.rados.rados.0.smithi085.stdout:715: done (13 left) 2021-02-10T04:30:50.344 INFO:tasks.rados.rados.0.smithi085.stdout:716: done (12 left) 2021-02-10T04:30:50.344 INFO:tasks.rados.rados.0.smithi085.stdout:717: read oid 46 snap -1 2021-02-10T04:30:50.344 INFO:tasks.rados.rados.0.smithi085.stdout:717: expect (ObjNum 202 snap 44 seq_num 202) 2021-02-10T04:30:50.344 INFO:tasks.rados.rados.0.smithi085.stdout:718: snap_remove snap 68 2021-02-10T04:30:50.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:50 smithi164 conmon[29288]: cluster 2021-02-10T04:30:49.296324 2021-02-10T04:30:50.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:50 smithi164 conmon[29288]: +0000 mon.a (mon.0) 952 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2021-02-10T04:30:50.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:50 smithi164 conmon[29288]: cluster 2021-02-10T04:30:50.025082+0000 2021-02-10T04:30:50.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:50 smithi164 conmon[29288]: mgr.y (mgr.14140) 417 : cluster [DBG] pgmap v499: 168 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 1 active+clean+scrubbing, 164 active+clean; 143 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 6.2 MiB/s rd, 854 KiB/s wr, 10 op/s 2021-02-10T04:30:50.691 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:30:50 smithi085 conmon[52836]: debug 2021-02-10T04:30:50.330+0000 7fba6ffae700 -1 osd.3 pg_epoch: 234 pg[4.14( v 63'26 (0'0,63'26] local-lis/les=71/72 n=11 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,2,7] r=0 lpr=71 crt=63'26 lcod 63'25 mlcod 0'0 active+clean] scrubber pg(4.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55fec073d760) 2021-02-10T04:30:50.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:50 smithi085 conmon[29319]: cluster 2021-02-10T04:30:49.296324+0000 mon.a ( 2021-02-10T04:30:50.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:50 smithi085 conmon[29319]: mon.0) 952 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2021-02-10T04:30:50.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:50 smithi085 conmon[29319]: cluster 2021-02-10T04:30:50.025082+0000 mgr.y (mgr.14140) 417 : cluster [DBG] pgmap v499: 168 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 1 active+clean+scrubbing, 164 active+clean; 143 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 6.2 MiB/s rd, 854 KiB/s wr, 10 op/s 2021-02-10T04:30:50.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:50 smithi085 conmon[35961]: cluster 2021-02-10T04:30:49.296324+0000 mon.a (mon.0) 952 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2021-02-10T04:30:50.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:50 smithi085 conmon[35961]: cluster 2021-02-10T04:30:50.025082+0000 mgr.y (mgr.14140) 417 : cluster [DBG] pgmap v499: 168 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 1 active+clean+scrubbing, 164 active+clean; 143 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 6.2 MiB/s rd, 854 KiB/s wr, 10 op/s 2021-02-10T04:30:50.692 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:30:50 smithi085 conmon[41540]: debug 2021-02-10T04:30:50.357+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:51.133 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:30:51.318 INFO:tasks.rados.rados.0.smithi085.stdout:718: done (13 left) 2021-02-10T04:30:51.318 INFO:tasks.rados.rados.0.smithi085.stdout:719: snap_remove snap 69 2021-02-10T04:30:51.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:51 smithi164 conmon[29288]: cluster 2021-02-10T04:30:50.306942+0000 mon.a 2021-02-10T04:30:51.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:51 smithi164 conmon[29288]: (mon.0) 953 : cluster [DBG] osdmap e234: 8 total, 8 up, 8 in 2021-02-10T04:30:51.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:51 smithi164 conmon[29288]: audit 2021-02-10T04:30:51.068417+0000 mon.b (mon.2 2021-02-10T04:30:51.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:51 smithi164 conmon[29288]: ) 30 : audit [INF] from='client.? 172.21.15.85:0/3479255852' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-02-10T04:30:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:51 smithi164 conmon[29288]: audit 2021-02-10T04:30:51.068871+0000 mon.a (mon 2021-02-10T04:30:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:51 smithi164 conmon[29288]: .0) 954 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-02-10T04:30:51.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:51 smithi085 conmon[29319]: cluster 2021-02-10T04:30:51.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:51 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:30:51.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:51 smithi085 conmon[29319]: 10T04:30:50.306942+0000 mon.a (mon.0) 953 : cluster [DBG] osdmap e234: 8 total, 8 up, 8 in 2021-02-10T04:30:51.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:51 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:30:51.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:51 smithi085 conmon[29319]: 02-10T04:30:51.068417+0000 mon.b (mon.2 2021-02-10T04:30:51.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:51 smithi085 conmon[29319]: ) 30 : audit [INF] from='client.? 172.21.15.85:0/3479255852' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-02-10T04:30:51.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:51 smithi085 conmon[29319]: audit 2021-02-10T 2021-02-10T04:30:51.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:51 smithi085 conmon[29319]: 04:30:51.068871+0000 mon.a (mon. 2021-02-10T04:30:51.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:51 smithi085 conmon[29319]: 0) 954 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-02-10T04:30:51.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:51 smithi085 conmon[35961]: cluster 2021-02-10T04:30:50.306942 2021-02-10T04:30:51.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:51 smithi085 conmon[35961]: +0000 mon.a (mon.0) 953 : cluster [DBG] osdmap e234: 8 total, 8 up, 8 in 2021-02-10T04:30:51.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:51 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:30:51.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:51 smithi085 conmon[35961]: -10T04:30:51.068417+0000 mon.b (mon. 2021-02-10T04:30:51.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:51 smithi085 conmon[35961]: 2) 30 : audit [INF] from='client.? 172.21.15.85:0/3479255852' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-02-10T04:30:51.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:51 smithi085 conmon[35961]: audit 2021-02-10T04:30:51.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:51 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:30:51.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:51 smithi085 conmon[35961]: 04:30:51.068871 2021-02-10T04:30:51.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:51 smithi085 conmon[35961]: +0000 mon.a (mon.0) 954 : audit 2021-02-10T04:30:51.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:51 smithi085 conmon[35961]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-02-10T04:30:51.838 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.5 to 1 (10000) 2021-02-10T04:30:51.953 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:30:51.981 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:52.072 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:30:52.177 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:30:52.210 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:52.320 INFO:tasks.rados.rados.0.smithi085.stdout:719: done (13 left) 2021-02-10T04:30:52.320 INFO:tasks.rados.rados.0.smithi085.stdout:720: rollback oid 27 current snap is 70 2021-02-10T04:30:52.320 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 27 to 59 2021-02-10T04:30:52.320 INFO:tasks.rados.rados.0.smithi085.stdout:717: expect (ObjNum 202 snap 44 seq_num 202) 2021-02-10T04:30:52.327 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:30:52 smithi164 conmon[33677]: debug 2021-02-10T04:30:52.045+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:52.572 INFO:tasks.rados.rados.0.smithi085.stdout:720: finishing rollback tid 1 to smithi08575397-27 2021-02-10T04:30:52.573 INFO:tasks.rados.rados.0.smithi085.stdout:720: finishing rollback tid 2 to smithi08575397-27 2021-02-10T04:30:52.573 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 27 v 66 (ObjNum 232 snap 53 seq_num 232) dirty exists 2021-02-10T04:30:52.573 INFO:tasks.rados.rados.0.smithi085.stdout:717: done (13 left) 2021-02-10T04:30:52.573 INFO:tasks.rados.rados.0.smithi085.stdout:720: done (12 left) 2021-02-10T04:30:52.573 INFO:tasks.rados.rados.0.smithi085.stdout:721: write oid 11 current snap is 70 2021-02-10T04:30:52.573 INFO:tasks.rados.rados.0.smithi085.stdout:721: seq_num 283 ranges {756381=400528,1910757=464751} 2021-02-10T04:30:52.579 INFO:tasks.rados.rados.0.smithi085.stdout:721: writing smithi08575397-11 from 756381 to 1156909 tid 1 2021-02-10T04:30:52.587 INFO:tasks.rados.rados.0.smithi085.stdout:721: writing smithi08575397-11 from 1910757 to 2375508 tid 2 2021-02-10T04:30:52.588 INFO:tasks.rados.rados.0.smithi085.stdout:722: write (excl) oid 5 current snap is 70 2021-02-10T04:30:52.588 INFO:tasks.rados.rados.0.smithi085.stdout:722: seq_num 284 ranges {773188=475777,1969239=724762} 2021-02-10T04:30:52.595 INFO:tasks.rados.rados.0.smithi085.stdout:722: writing smithi08575397-5 from 773188 to 1248965 tid 1 2021-02-10T04:30:52.605 INFO:tasks.rados.rados.0.smithi085.stdout:722: writing smithi08575397-5 from 1969239 to 2694001 tid 2 2021-02-10T04:30:52.606 INFO:tasks.rados.rados.0.smithi085.stdout:723: delete oid 27 current snap is 70 2021-02-10T04:30:52.606 INFO:tasks.rados.rados.0.smithi085.stdout:721: finishing write tid 1 to smithi08575397-11 2021-02-10T04:30:52.606 INFO:tasks.rados.rados.0.smithi085.stdout:721: finishing write tid 2 to smithi08575397-11 2021-02-10T04:30:52.606 INFO:tasks.rados.rados.0.smithi085.stdout:721: finishing write tid 3 to smithi08575397-11 2021-02-10T04:30:52.606 INFO:tasks.rados.rados.0.smithi085.stdout:721: finishing write tid 4 to smithi08575397-11 2021-02-10T04:30:52.606 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 334 (ObjNum 283 snap 70 seq_num 283) dirty exists 2021-02-10T04:30:52.606 INFO:tasks.rados.rados.0.smithi085.stdout:721: left oid 11 (ObjNum 283 snap 70 seq_num 283) 2021-02-10T04:30:52.606 INFO:tasks.rados.rados.0.smithi085.stdout:722: finishing write tid 1 to smithi08575397-5 2021-02-10T04:30:52.608 INFO:tasks.rados.rados.0.smithi085.stdout:721: done (14 left) 2021-02-10T04:30:52.608 INFO:tasks.rados.rados.0.smithi085.stdout:723: done (13 left) 2021-02-10T04:30:52.608 INFO:tasks.rados.rados.0.smithi085.stdout:724: write (excl) oid 28 current snap is 70 2021-02-10T04:30:52.608 INFO:tasks.rados.rados.0.smithi085.stdout:724: seq_num 286 ranges {406802=626275,1686203=728431} 2021-02-10T04:30:52.615 INFO:tasks.rados.rados.0.smithi085.stdout:724: writing smithi08575397-28 from 406802 to 1033077 tid 1 2021-02-10T04:30:52.626 INFO:tasks.rados.rados.0.smithi085.stdout:724: writing smithi08575397-28 from 1686203 to 2414634 tid 2 2021-02-10T04:30:52.627 INFO:tasks.rados.rados.0.smithi085.stdout:725: read oid 43 snap 59 2021-02-10T04:30:52.627 INFO:tasks.rados.rados.0.smithi085.stdout:725: expect (ObjNum 255 snap 59 seq_num 255) 2021-02-10T04:30:52.627 INFO:tasks.rados.rados.0.smithi085.stdout:726: read oid 27 snap -1 2021-02-10T04:30:52.627 INFO:tasks.rados.rados.0.smithi085.stdout:726: expect deleted 2021-02-10T04:30:52.627 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:30:52.627 INFO:tasks.rados.rados.0.smithi085.stdout:722: finishing write tid 2 to smithi08575397-5 2021-02-10T04:30:52.627 INFO:tasks.rados.rados.0.smithi085.stdout:722: finishing write tid 3 to smithi08575397-5 2021-02-10T04:30:52.627 INFO:tasks.rados.rados.0.smithi085.stdout:722: finishing write tid 4 to smithi08575397-5 2021-02-10T04:30:52.627 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 213 (ObjNum 284 snap 70 seq_num 284) dirty exists 2021-02-10T04:30:52.627 INFO:tasks.rados.rados.0.smithi085.stdout:722: left oid 5 (ObjNum 284 snap 70 seq_num 284) 2021-02-10T04:30:52.627 INFO:tasks.rados.rados.0.smithi085.stdout:724: finishing write tid 1 to smithi08575397-28 2021-02-10T04:30:52.628 INFO:tasks.rados.rados.0.smithi085.stdout:722: done (15 left) 2021-02-10T04:30:52.628 INFO:tasks.rados.rados.0.smithi085.stdout:727: snap_create 2021-02-10T04:30:52.628 INFO:tasks.rados.rados.0.smithi085.stdout:724: finishing write tid 2 to smithi08575397-28 2021-02-10T04:30:52.628 INFO:tasks.rados.rados.0.smithi085.stdout:724: finishing write tid 3 to smithi08575397-28 2021-02-10T04:30:52.629 INFO:tasks.rados.rados.0.smithi085.stdout:724: finishing write tid 4 to smithi08575397-28 2021-02-10T04:30:52.629 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 72 (ObjNum 286 snap 70 seq_num 286) dirty exists 2021-02-10T04:30:52.629 INFO:tasks.rados.rados.0.smithi085.stdout:724: left oid 28 (ObjNum 286 snap 70 seq_num 286) 2021-02-10T04:30:52.630 INFO:tasks.rados.rados.0.smithi085.stdout:725: expect (ObjNum 255 snap 59 seq_num 255) 2021-02-10T04:30:52.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:52 smithi164 conmon[29288]: audit 2021-02-10T04:30:51.317659+0000 mon.a (mon.0 2021-02-10T04:30:52.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:52 smithi164 conmon[29288]: ) 955 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-02-10T04:30:52.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:52 smithi164 conmon[29288]: cluster 2021-02-10T04:30:51.317748+0000 mon.a (mon.0) 956 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2021-02-10T04:30:52.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:52 smithi164 conmon[29288]: cephadm 2021-02-10T04:30:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:52 smithi164 conmon[29288]: 2021-02-10T04:30:51.438511+0000 mgr.y (mgr.14140) 418 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:30:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:52 smithi164 conmon[29288]: 2021-02-10T04:30:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:52 smithi164 conmon[29288]: cephadm 2021-02-10T04:30:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:52 smithi164 conmon[29288]: 2021-02-10T04:30:51.438983+0000 mgr.y ( 2021-02-10T04:30:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:52 smithi164 conmon[29288]: mgr.14140) 419 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:30:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:52 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:30:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:52 smithi164 conmon[29288]: 02-10T04:30:51.988631+0000 mon.a (mon. 2021-02-10T04:30:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:52 smithi164 conmon[29288]: 0) 957 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:30:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:52 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:30:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:52 smithi164 conmon[29288]: 10T04:30:52.025648+0000 mgr.y 2021-02-10T04:30:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:52 smithi164 conmon[29288]: (mgr.14140) 420 : cluster [DBG] pgmap v502: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 5 active+clean+snaptrim, 160 active+clean; 149 MiB data, 493 MiB used, 715 GiB / 715 GiB avail; 21 MiB/s rd, 2.1 MiB/s wr, 29 op/s 2021-02-10T04:30:52.695 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[52836]: debug 2021-02-10T04:30:52.287+0000 7fba73fb6700 -1 osd.3 pg_epoch: 235 pg[4.14( v 63'26 (0'0,63'26] local-lis/les=71/72 n=11 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,2,7] r=0 lpr=71 crt=63'26 lcod 63'25 mlcod 0'0 active+clean] scrubber pg(4.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55fec07f9e40) 2021-02-10T04:30:52.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[29319]: audit 2021-02-10T04:30:51.317659+0000 mon.a (mon.0) 955 : audit [INF] 2021-02-10T04:30:52.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[29319]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-02-10T04:30:52.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[29319]: cluster 2021-02-10T04:30:51.317748+0000 mon.a (mon.0) 956 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2021-02-10T04:30:52.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[29319]: cephadm 2021-02-10T 2021-02-10T04:30:52.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[29319]: 04:30:51.438511+0000 mgr.y (mgr.14140) 418 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:30:52.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[29319]: cephadm 2021-02-10T04:30:51 2021-02-10T04:30:52.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[29319]: .438983+0000 mgr.y (mgr.14140) 419 : cephadm 2021-02-10T04:30:52.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[29319]: [INF] Refreshing smithi164 facts 2021-02-10T04:30:52.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[29319]: audit 2021-02-10T04:30:51 2021-02-10T04:30:52.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[29319]: .988631+0000 mon.a (mon.0) 957 : audit 2021-02-10T04:30:52.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[29319]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:30:52.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:30:52.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[29319]: :30:52.025648+0000 mgr.y (mgr 2021-02-10T04:30:52.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[29319]: .14140) 420 : cluster [DBG] pgmap v502: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 5 active+clean+snaptrim, 160 active+clean; 149 MiB data, 493 MiB used, 715 GiB / 715 GiB avail; 21 MiB/s rd, 2.1 MiB/s wr, 29 op/s 2021-02-10T04:30:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: audit 2021-02-10T04:30:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: 2021-02-10T04:30:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: 2021 2021-02-10T04:30:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: - 2021-02-10T04:30:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: 02 2021-02-10T04:30:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: - 2021-02-10T04:30:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: 10 2021-02-10T04:30:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: T 2021-02-10T04:30:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: 04 2021-02-10T04:30:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: : 2021-02-10T04:30:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: 30 2021-02-10T04:30:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: : 2021-02-10T04:30:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: 51 2021-02-10T04:30:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: . 2021-02-10T04:30:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: 317659 2021-02-10T04:30:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: +0000 2021-02-10T04:30:52.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: 2021-02-10T04:30:52.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: mon.a 2021-02-10T04:30:52.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: ( 2021-02-10T04:30:52.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: mon 2021-02-10T04:30:52.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: . 2021-02-10T04:30:52.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: 0 2021-02-10T04:30:52.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: ) 2021-02-10T04:30:52.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: 955 2021-02-10T04:30:52.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: : 2021-02-10T04:30:52.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: audit 2021-02-10T04:30:52.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: 2021-02-10T04:30:52.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: [INF] 2021-02-10T04:30:52.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: 2021-02-10T04:30:52.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-02-10T04:30:52.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:30:52.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: -10T04:30:51.317748+0000 mon.a (mon.0) 956 : cluster 2021-02-10T04:30:52.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: [DBG] osdmap e235: 8 total, 8 up, 8 in 2021-02-10T04:30:52.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: cephadm 2021-02 2021-02-10T04:30:52.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: -10T04:30:51.438511+0000 mgr.y (mgr 2021-02-10T04:30:52.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: .14140) 418 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:30:52.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: cephadm 2021-02 2021-02-10T04:30:52.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: -10T04:30:51.438983+0000 mgr.y (mgr. 2021-02-10T04:30:52.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: 14140) 419 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:30:52.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:30:52.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: 02-10T04:30:51.988631+0000 mon.a (mon 2021-02-10T04:30:52.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: .0) 957 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:30:52.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:30:52.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: -10T04:30:52.025648+0000 mgr.y ( 2021-02-10T04:30:52.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:52 smithi085 conmon[35961]: mgr.14140) 420 : cluster [DBG] pgmap v502: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 5 active+clean+snaptrim, 160 active+clean; 149 MiB data, 493 MiB used, 715 GiB / 715 GiB avail; 21 MiB/s rd, 2.1 MiB/s wr, 29 op/s 2021-02-10T04:30:53.338 INFO:tasks.rados.rados.0.smithi085.stdout:724: done (15 left) 2021-02-10T04:30:53.338 INFO:tasks.rados.rados.0.smithi085.stdout:725: done (14 left) 2021-02-10T04:30:53.338 INFO:tasks.rados.rados.0.smithi085.stdout:726: done (13 left) 2021-02-10T04:30:53.338 INFO:tasks.rados.rados.0.smithi085.stdout:727: done (12 left) 2021-02-10T04:30:53.339 INFO:tasks.rados.rados.0.smithi085.stdout:728: write (excl) oid 38 current snap is 71 2021-02-10T04:30:53.339 INFO:tasks.rados.rados.0.smithi085.stdout:728: seq_num 287 ranges {423609=701524,1628332=472089,2648987=174622} 2021-02-10T04:30:53.345 INFO:tasks.rados.rados.0.smithi085.stdout:728: writing smithi08575397-38 from 423609 to 1125133 tid 1 2021-02-10T04:30:53.352 INFO:tasks.rados.rados.0.smithi085.stdout:728: writing smithi08575397-38 from 1628332 to 2100421 tid 2 2021-02-10T04:30:53.357 INFO:tasks.rados.rados.0.smithi085.stdout:728: writing smithi08575397-38 from 2648987 to 2823609 tid 3 2021-02-10T04:30:53.357 INFO:tasks.rados.rados.0.smithi085.stdout:729: write (excl) oid 26 current snap is 71 2021-02-10T04:30:53.357 INFO:tasks.rados.rados.0.smithi085.stdout:729: seq_num 288 ranges {440416=776773,1686814=732100} 2021-02-10T04:30:53.364 INFO:tasks.rados.rados.0.smithi085.stdout:729: writing smithi08575397-26 from 440416 to 1217189 tid 1 2021-02-10T04:30:53.373 INFO:tasks.rados.rados.0.smithi085.stdout:729: writing smithi08575397-26 from 1686814 to 2418914 tid 2 2021-02-10T04:30:53.373 INFO:tasks.rados.rados.0.smithi085.stdout:728: finishing write tid 1 to smithi08575397-38 2021-02-10T04:30:53.373 INFO:tasks.rados.rados.0.smithi085.stdout:730: delete oid 11 current snap is 71 2021-02-10T04:30:53.373 INFO:tasks.rados.rados.0.smithi085.stdout:728: finishing write tid 2 to smithi08575397-38 2021-02-10T04:30:53.373 INFO:tasks.rados.rados.0.smithi085.stdout:728: finishing write tid 3 to smithi08575397-38 2021-02-10T04:30:53.373 INFO:tasks.rados.rados.0.smithi085.stdout:728: finishing write tid 4 to smithi08575397-38 2021-02-10T04:30:53.373 INFO:tasks.rados.rados.0.smithi085.stdout:728: finishing write tid 5 to smithi08575397-38 2021-02-10T04:30:53.373 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 225 (ObjNum 287 snap 71 seq_num 287) dirty exists 2021-02-10T04:30:53.374 INFO:tasks.rados.rados.0.smithi085.stdout:728: left oid 38 (ObjNum 287 snap 71 seq_num 287) 2021-02-10T04:30:53.374 INFO:tasks.rados.rados.0.smithi085.stdout:729: finishing write tid 1 to smithi08575397-26 2021-02-10T04:30:53.376 INFO:tasks.rados.rados.0.smithi085.stdout:728: done (14 left) 2021-02-10T04:30:53.376 INFO:tasks.rados.rados.0.smithi085.stdout:730: done (13 left) 2021-02-10T04:30:53.376 INFO:tasks.rados.rados.0.smithi085.stdout:731: read oid 46 snap -1 2021-02-10T04:30:53.376 INFO:tasks.rados.rados.0.smithi085.stdout:731: expect (ObjNum 202 snap 44 seq_num 202) 2021-02-10T04:30:53.376 INFO:tasks.rados.rados.0.smithi085.stdout:732: copy_from oid 5 from oid 28 current snap is 71 2021-02-10T04:30:53.376 INFO:tasks.rados.rados.0.smithi085.stdout:733: rollback oid 8 current snap is 71 2021-02-10T04:30:53.376 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 8 to 59 2021-02-10T04:30:53.376 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:30:53.377 INFO:tasks.rados.rados.0.smithi085.stdout:729: finishing write tid 2 to smithi08575397-26 2021-02-10T04:30:53.377 INFO:tasks.rados.rados.0.smithi085.stdout:729: finishing write tid 3 to smithi08575397-26 2021-02-10T04:30:53.377 INFO:tasks.rados.rados.0.smithi085.stdout:729: finishing write tid 4 to smithi08575397-26 2021-02-10T04:30:53.377 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 51 (ObjNum 288 snap 71 seq_num 288) dirty exists 2021-02-10T04:30:53.377 INFO:tasks.rados.rados.0.smithi085.stdout:729: left oid 26 (ObjNum 288 snap 71 seq_num 288) 2021-02-10T04:30:53.378 INFO:tasks.rados.rados.0.smithi085.stdout:733: finishing rollback tid 1 to smithi08575397-8 2021-02-10T04:30:53.378 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 0 (ObjNum 83216224 snap 22030 seq_num 83276096) dirty dne 2021-02-10T04:30:53.378 INFO:tasks.rados.rados.0.smithi085.stdout:729: done (15 left) 2021-02-10T04:30:53.378 INFO:tasks.rados.rados.0.smithi085.stdout:733: done (14 left) 2021-02-10T04:30:53.378 INFO:tasks.rados.rados.0.smithi085.stdout:734: read oid 37 snap -1 2021-02-10T04:30:53.378 INFO:tasks.rados.rados.0.smithi085.stdout:734: expect (ObjNum 246 snap 57 seq_num 246) 2021-02-10T04:30:53.378 INFO:tasks.rados.rados.0.smithi085.stdout:735: snap_create 2021-02-10T04:30:53.380 INFO:tasks.rados.rados.0.smithi085.stdout:731: expect (ObjNum 202 snap 44 seq_num 202) 2021-02-10T04:30:53.624 INFO:tasks.rados.rados.0.smithi085.stdout:734: expect (ObjNum 246 snap 57 seq_num 246) 2021-02-10T04:30:53.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:53 smithi164 conmon[29288]: cluster 2021-02-10T04:30:52.318401+0000 mon.a (mon.0) 2021-02-10T04:30:53.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:53 smithi164 conmon[29288]: 958 : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2021-02-10T04:30:53.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:53 smithi164 conmon[29288]: cluster 2021-02-10T04:30:53.062520+0000 mon.a (mon.0) 959 : cluster [WRN] Health check update: 0 slow ops, oldest one blocked for 97 sec, osd.1 has slow ops (SLOW_OPS) 2021-02-10T04:30:53.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:53 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:30:53.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:53 smithi085 conmon[29319]: :30:52.318401+0000 mon.a (mon.0) 958 : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2021-02-10T04:30:53.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:53 smithi085 conmon[29319]: cluster 2021-02-10T04:30:53.062520+0000 mon.a (mon. 2021-02-10T04:30:53.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:53 smithi085 conmon[29319]: 0) 959 : cluster [WRN] Health check update: 0 slow ops, oldest one blocked for 97 sec, osd.1 has slow ops (SLOW_OPS) 2021-02-10T04:30:53.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:53 smithi085 conmon[35961]: cluster 2021-02-10T04:30:52.318401+0000 2021-02-10T04:30:53.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:53 smithi085 conmon[35961]: mon.a (mon.0) 958 : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2021-02-10T04:30:53.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:53 smithi085 conmon[35961]: cluster 2021-02-10T04:30:53.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:53 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:30:53.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:53 smithi085 conmon[35961]: :30:53.062520+0000 mon.a (mon. 2021-02-10T04:30:53.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:53 smithi085 conmon[35961]: 0) 959 : cluster [WRN] Health check update: 0 slow ops, oldest one blocked for 97 sec, osd.1 has slow ops (SLOW_OPS) 2021-02-10T04:30:53.722 INFO:tasks.rados.rados.0.smithi085.stdout:732: finishing copy_from to smithi08575397-5 2021-02-10T04:30:53.722 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 72 (ObjNum 286 snap 70 seq_num 286) dirty exists 2021-02-10T04:30:53.722 INFO:tasks.rados.rados.0.smithi085.stdout:732: finishing copy_from racing read to smithi08575397-5 2021-02-10T04:30:54.345 INFO:tasks.rados.rados.0.smithi085.stdout:731: done (15 left) 2021-02-10T04:30:54.345 INFO:tasks.rados.rados.0.smithi085.stdout:732: done (14 left) 2021-02-10T04:30:54.345 INFO:tasks.rados.rados.0.smithi085.stdout:734: done (13 left) 2021-02-10T04:30:54.345 INFO:tasks.rados.rados.0.smithi085.stdout:735: done (12 left) 2021-02-10T04:30:54.345 INFO:tasks.rados.rados.0.smithi085.stdout:736: copy_from oid 2 from oid 37 current snap is 72 2021-02-10T04:30:54.345 INFO:tasks.rados.rados.0.smithi085.stdout:737: snap_create 2021-02-10T04:30:54.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:54 smithi164 conmon[29288]: cluster 2021-02-10T04:30:53.337924+0000 mon.a (mon.0) 960 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2021-02-10T04:30:54.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:54 smithi164 conmon[29288]: 2021-02-10T04:30:54.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:54 smithi164 conmon[29288]: cluster 2021-02-10T04:30:54.026036+0000 mgr.y (mgr.14140) 421 : cluster [DBG] pgmap v505: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 5 active+clean+snaptrim, 160 active+clean; 151 MiB data, 499 MiB used, 715 GiB / 715 GiB avail; 15 MiB/s rd, 1.8 MiB/s wr, 23 op/s 2021-02-10T04:30:54.691 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:30:54 smithi085 conmon[52836]: debug 2021-02-10T04:30:54.361+0000 7fba73fb6700 -1 osd.3 pg_epoch: 238 pg[4.14( v 63'26 (0'0,63'26] local-lis/les=71/72 n=11 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,2,7] r=0 lpr=71 crt=63'26 lcod 63'25 mlcod 0'0 active+clean] scrubber pg(4.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55fec073cdc0) 2021-02-10T04:30:54.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:54 smithi085 conmon[35961]: cluster 2021-02-10T04:30:53.337924 2021-02-10T04:30:54.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:54 smithi085 conmon[35961]: +0000 mon.a (mon.0) 960 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2021-02-10T04:30:54.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:54 smithi085 conmon[35961]: cluster 2021-02-10T04:30 2021-02-10T04:30:54.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:54 smithi085 conmon[35961]: :54.026036+0000 mgr.y (mgr.14140) 421 : cluster [DBG] pgmap v505: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 5 active+clean+snaptrim, 160 active+clean; 151 MiB data, 499 MiB used, 715 GiB / 715 GiB avail; 15 MiB/s rd, 1.8 MiB/s wr, 23 op/s 2021-02-10T04:30:54.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:54 smithi085 conmon[29319]: cluster 2021-02-10T04:30:53.337924 2021-02-10T04:30:54.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:54 smithi085 conmon[29319]: +0000 mon.a (mon.0) 960 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2021-02-10T04:30:54.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:54 smithi085 conmon[29319]: cluster 2021-02-10T04:30:54.026036+0000 mgr.y (mgr. 2021-02-10T04:30:54.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:54 smithi085 conmon[29319]: 14140) 421 : cluster [DBG] pgmap v505: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 5 active+clean+snaptrim, 160 active+clean; 151 MiB data, 499 MiB used, 715 GiB / 715 GiB avail; 15 MiB/s rd, 1.8 MiB/s wr, 23 op/s 2021-02-10T04:30:54.695 INFO:tasks.rados.rados.0.smithi085.stdout:736: finishing copy_from to smithi08575397-2 2021-02-10T04:30:54.695 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 277 (ObjNum 246 snap 57 seq_num 246) dirty exists 2021-02-10T04:30:54.695 INFO:tasks.rados.rados.0.smithi085.stdout:736: finishing copy_from racing read to smithi08575397-2 2021-02-10T04:30:55.113 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:30:55.191 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:30:54 smithi085 conmon[41540]: debug 2021-02-10T04:30:54.861+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:55.353 INFO:tasks.rados.rados.0.smithi085.stdout:736: done (13 left) 2021-02-10T04:30:55.354 INFO:tasks.rados.rados.0.smithi085.stdout:737: done (12 left) 2021-02-10T04:30:55.354 INFO:tasks.rados.rados.0.smithi085.stdout:738: write (excl) oid 37 current snap is 73 2021-02-10T04:30:55.354 INFO:tasks.rados.rados.0.smithi085.stdout:738: seq_num 290 ranges {474030=643624,1520131=735769} 2021-02-10T04:30:55.356 INFO:tasks.rados.rados.0.smithi085.stdout:738: writing smithi08575397-37 from 474030 to 1117654 tid 1 2021-02-10T04:30:55.365 INFO:tasks.rados.rados.0.smithi085.stdout:738: writing smithi08575397-37 from 1520131 to 2255900 tid 2 2021-02-10T04:30:55.365 INFO:tasks.rados.rados.0.smithi085.stdout:739: rollback oid 24 current snap is 73 2021-02-10T04:30:55.365 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 24 to 66 2021-02-10T04:30:55.365 INFO:tasks.rados.rados.0.smithi085.stdout:738: finishing write tid 1 to smithi08575397-37 2021-02-10T04:30:55.366 INFO:tasks.rados.rados.0.smithi085.stdout:740: delete oid 43 current snap is 73 2021-02-10T04:30:55.368 INFO:tasks.rados.rados.0.smithi085.stdout:740: done (14 left) 2021-02-10T04:30:55.368 INFO:tasks.rados.rados.0.smithi085.stdout:741: delete oid 5 current snap is 73 2021-02-10T04:30:55.370 INFO:tasks.rados.rados.0.smithi085.stdout:739: finishing rollback tid 0 to smithi08575397-24 2021-02-10T04:30:55.370 INFO:tasks.rados.rados.0.smithi085.stdout:739: finishing rollback tid 1 to smithi08575397-24 2021-02-10T04:30:55.371 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 24 v 75 (ObjNum 1953066355 snap 942696808 seq_num 859125557) dirty dne 2021-02-10T04:30:55.372 INFO:tasks.rados.rados.0.smithi085.stdout:738: finishing write tid 2 to smithi08575397-37 2021-02-10T04:30:55.372 INFO:tasks.rados.rados.0.smithi085.stdout:738: finishing write tid 3 to smithi08575397-37 2021-02-10T04:30:55.372 INFO:tasks.rados.rados.0.smithi085.stdout:739: done (14 left) 2021-02-10T04:30:55.372 INFO:tasks.rados.rados.0.smithi085.stdout:741: done (13 left) 2021-02-10T04:30:55.372 INFO:tasks.rados.rados.0.smithi085.stdout:742: read oid 46 snap -1 2021-02-10T04:30:55.372 INFO:tasks.rados.rados.0.smithi085.stdout:742: expect (ObjNum 202 snap 44 seq_num 202) 2021-02-10T04:30:55.372 INFO:tasks.rados.rados.0.smithi085.stdout:743: read oid 36 snap 70 2021-02-10T04:30:55.372 INFO:tasks.rados.rados.0.smithi085.stdout:743: expect (ObjNum 270 snap 68 seq_num 270) 2021-02-10T04:30:55.372 INFO:tasks.rados.rados.0.smithi085.stdout:744: write oid 3 current snap is 73 2021-02-10T04:30:55.372 INFO:tasks.rados.rados.0.smithi085.stdout:738: finishing write tid 4 to smithi08575397-37 2021-02-10T04:30:55.373 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 344 (ObjNum 290 snap 73 seq_num 290) dirty exists 2021-02-10T04:30:55.373 INFO:tasks.rados.rados.0.smithi085.stdout:738: left oid 37 (ObjNum 290 snap 73 seq_num 290) 2021-02-10T04:30:55.373 INFO:tasks.rados.rados.0.smithi085.stdout:744: seq_num 293 ranges {524451=469371,1579224=483096,2529878=394573} 2021-02-10T04:30:55.378 INFO:tasks.rados.rados.0.smithi085.stdout:744: writing smithi08575397-3 from 524451 to 993822 tid 1 2021-02-10T04:30:55.385 INFO:tasks.rados.rados.0.smithi085.stdout:744: writing smithi08575397-3 from 1579224 to 2062320 tid 2 2021-02-10T04:30:55.391 INFO:tasks.rados.rados.0.smithi085.stdout:744: writing smithi08575397-3 from 2529878 to 2924451 tid 3 2021-02-10T04:30:55.391 INFO:tasks.rados.rados.0.smithi085.stdout:738: done (15 left) 2021-02-10T04:30:55.391 INFO:tasks.rados.rados.0.smithi085.stdout:745: snap_remove snap 59 2021-02-10T04:30:55.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:55 smithi164 conmon[29288]: cluster 2021-02-10T04:30:54.335020 2021-02-10T04:30:55.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:55 smithi164 conmon[29288]: +0000 mon.a (mon.0) 961 : cluster [INF] Health check cleared: SLOW_OPS (was: 0 slow ops, oldest one blocked for 97 sec, osd.1 has slow ops) 2021-02-10T04:30:55.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:55 smithi164 conmon[29288]: cluster 2021-02-10T04:30:54.335053+0000 mon.a (mon.0) 962 : cluster 2021-02-10T04:30:55.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:55 smithi164 conmon[29288]: [INF] Cluster is now healthy 2021-02-10T04:30:55.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:55 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:30:55.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:55 smithi164 conmon[29288]: - 2021-02-10T04:30:55.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:55 smithi164 conmon[29288]: 02-10T04:30:54.343857+0000 mon.a (mon 2021-02-10T04:30:55.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:55 smithi164 conmon[29288]: .0) 963 : cluster [DBG] osdmap e238: 8 total, 8 up, 8 in 2021-02-10T04:30:55.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:55 smithi085 conmon[35961]: cluster 2021-02-10T04:30:54.335020+0000 mon.a (mon.0) 961 : 2021-02-10T04:30:55.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:55 smithi085 conmon[35961]: cluster [INF] Health check cleared: SLOW_OPS (was: 0 slow ops, oldest one blocked for 97 sec, osd.1 has slow ops) 2021-02-10T04:30:55.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:55 smithi085 conmon[35961]: cluster 2021-02-10T04:30:54.335053+0000 mon.a (mon.0) 962 : cluster [INF] Cluster is now healthy 2021-02-10T04:30:55.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:55 smithi085 conmon[35961]: cluster 2021-02-10T04:30:55.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:55 smithi085 conmon[35961]: 2021-02-10T04:30:54.343857+0000 mon.a (mon.0) 963 : cluster [DBG] osdmap e238: 8 total, 8 up, 8 in 2021-02-10T04:30:55.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:55 smithi085 conmon[29319]: cluster 2021-02-10T04:30:54.335020 2021-02-10T04:30:55.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:55 smithi085 conmon[29319]: +0000 mon.a (mon.0) 961 : cluster [INF] Health check cleared: SLOW_OPS (was: 0 slow ops, oldest one blocked for 97 sec, osd.1 has slow ops) 2021-02-10T04:30:55.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:55 smithi085 conmon[29319]: cluster 2021-02-10T04:30:54.335053+0000 mon.a (mon.0) 962 : cluster [INF] Cluster is now healthy 2021-02-10T04:30:55.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:55 smithi085 conmon[29319]: cluster 2021-02-10T04:30:54.343857+0000 mon.a (mon.0) 963 : cluster [DBG] osdmap e238: 8 total, 8 up, 8 in 2021-02-10T04:30:56.358 INFO:tasks.rados.rados.0.smithi085.stdout:742: expect (ObjNum 202 snap 44 seq_num 202) 2021-02-10T04:30:56.610 INFO:tasks.rados.rados.0.smithi085.stdout:743: expect (ObjNum 270 snap 68 seq_num 270) 2021-02-10T04:30:56.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:56 smithi164 conmon[29288]: cluster 2021-02-10T04:30:55.350225+0000 mon.a (mon.0) 964 : cluster [DBG] 2021-02-10T04:30:56.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:56 smithi164 conmon[29288]: osdmap e239: 8 total, 8 up, 8 in 2021-02-10T04:30:56.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:56 smithi164 conmon[29288]: cluster 2021-02-10T04:30:56.026527+0000 mgr.y (mgr.14140) 422 : cluster [DBG] pgmap v508: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim, 165 active+clean; 163 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 3.1 MiB/s wr, 24 op/s 2021-02-10T04:30:56.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[29319]: cluster 2021-02-10T04:30:55.350225+0000 mon.a (mon.0) 964 2021-02-10T04:30:56.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[29319]: : cluster [DBG] osdmap e239: 8 total, 8 up, 8 in 2021-02-10T04:30:56.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[29319]: cluster 2021-02-10T04:30:56.026527+0000 mgr.y (mgr.14140) 422 : cluster [DBG] pgmap v508: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim, 165 active+clean; 163 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 3.1 MiB/s wr, 24 op/s 2021-02-10T04:30:56.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: cluster 2021-02-10T04:30:55.350225+0000 mon.a 2021-02-10T04:30:56.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: (mon.0) 964 : cluster [DBG] osdmap e239: 8 total, 8 up, 8 in 2021-02-10T04:30:56.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: cluster 2021-02-10T04:30:56.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: 2021-02-10T04:30:56.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: 2021 2021-02-10T04:30:56.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: - 2021-02-10T04:30:56.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: 02 2021-02-10T04:30:56.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: - 2021-02-10T04:30:56.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: 10 2021-02-10T04:30:56.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: T 2021-02-10T04:30:56.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: 04 2021-02-10T04:30:56.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: : 2021-02-10T04:30:56.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: 30 2021-02-10T04:30:56.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: : 2021-02-10T04:30:56.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: 56 2021-02-10T04:30:56.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: . 2021-02-10T04:30:56.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: 026527 2021-02-10T04:30:56.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: +0000 2021-02-10T04:30:56.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: 2021-02-10T04:30:56.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: mgr.y 2021-02-10T04:30:56.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: ( 2021-02-10T04:30:56.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: mgr 2021-02-10T04:30:56.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: . 2021-02-10T04:30:56.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: 14140 2021-02-10T04:30:56.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: ) 2021-02-10T04:30:56.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: 422 2021-02-10T04:30:56.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: : 2021-02-10T04:30:56.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: cluster 2021-02-10T04:30:56.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: 2021-02-10T04:30:56.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: [DBG] 2021-02-10T04:30:56.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: 2021-02-10T04:30:56.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: pgmap v508: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim, 165 active+clean; 163 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 3.1 MiB/s wr, 24 op/s 2021-02-10T04:30:56.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:56 smithi085 conmon[35961]: 2021-02-10T04:30:56.798 INFO:tasks.rados.rados.0.smithi085.stdout:744: finishing write tid 1 to smithi08575397-3 2021-02-10T04:30:56.879 INFO:tasks.rados.rados.0.smithi085.stdout:744: finishing write tid 2 to smithi08575397-3 2021-02-10T04:30:56.879 INFO:tasks.rados.rados.0.smithi085.stdout:744: finishing write tid 3 to smithi08575397-3 2021-02-10T04:30:56.879 INFO:tasks.rados.rados.0.smithi085.stdout:744: finishing write tid 4 to smithi08575397-3 2021-02-10T04:30:56.879 INFO:tasks.rados.rados.0.smithi085.stdout:744: finishing write tid 5 to smithi08575397-3 2021-02-10T04:30:56.879 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 239 (ObjNum 293 snap 73 seq_num 293) dirty exists 2021-02-10T04:30:56.880 INFO:tasks.rados.rados.0.smithi085.stdout:744: left oid 3 (ObjNum 293 snap 73 seq_num 293) 2021-02-10T04:30:56.880 INFO:tasks.rados.rados.0.smithi085.stdout:742: done (15 left) 2021-02-10T04:30:56.880 INFO:tasks.rados.rados.0.smithi085.stdout:743: done (14 left) 2021-02-10T04:30:56.880 INFO:tasks.rados.rados.0.smithi085.stdout:744: done (13 left) 2021-02-10T04:30:56.880 INFO:tasks.rados.rados.0.smithi085.stdout:745: done (12 left) 2021-02-10T04:30:56.880 INFO:tasks.rados.rados.0.smithi085.stdout:746: snap_create 2021-02-10T04:30:57.230 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:30:57.256 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:57.264 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 6 1 2021-02-10T04:30:57.373 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:30:57.379 INFO:tasks.rados.rados.0.smithi085.stdout:746: done (12 left) 2021-02-10T04:30:57.379 INFO:tasks.rados.rados.0.smithi085.stdout:747: read oid 36 snap 72 2021-02-10T04:30:57.380 INFO:tasks.rados.rados.0.smithi085.stdout:747: expect (ObjNum 270 snap 68 seq_num 270) 2021-02-10T04:30:57.380 INFO:tasks.rados.rados.0.smithi085.stdout:748: write oid 24 current snap is 74 2021-02-10T04:30:57.380 INFO:tasks.rados.rados.0.smithi085.stdout:748: seq_num 294 ranges {541258=544620,1637706=626754,2840948=100310} 2021-02-10T04:30:57.384 INFO:tasks.rados.rados.0.smithi085.stdout:748: writing smithi08575397-24 from 541258 to 1085878 tid 1 2021-02-10T04:30:57.391 INFO:tasks.rados.rados.0.smithi085.stdout:748: writing smithi08575397-24 from 1637706 to 2264460 tid 2 2021-02-10T04:30:57.396 INFO:tasks.rados.rados.0.smithi085.stdout:748: writing smithi08575397-24 from 2840948 to 2941258 tid 3 2021-02-10T04:30:57.396 INFO:tasks.rados.rados.0.smithi085.stdout:749: rollback oid 40 current snap is 74 2021-02-10T04:30:57.396 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 40 to 70 2021-02-10T04:30:57.396 INFO:tasks.rados.rados.0.smithi085.stdout:750: rollback oid 43 current snap is 74 2021-02-10T04:30:57.397 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 43 to 70 2021-02-10T04:30:57.397 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:30:57.397 INFO:tasks.rados.rados.0.smithi085.stdout:748: finishing write tid 1 to smithi08575397-24 2021-02-10T04:30:57.397 INFO:tasks.rados.rados.0.smithi085.stdout:747: expect (ObjNum 270 snap 68 seq_num 270) 2021-02-10T04:30:57.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:57 smithi085 conmon[29319]: cluster 2021-02-10T04:30:56.358118+0000 mon.a (mon. 2021-02-10T04:30:57.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:57 smithi085 conmon[29319]: 0) 965 : cluster [DBG] osdmap e240: 8 total, 8 up, 8 in 2021-02-10T04:30:57.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:57 smithi085 conmon[35961]: cluster 2021-02-10T04:30:56.358118 2021-02-10T04:30:57.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:57 smithi085 conmon[35961]: +0000 mon.a (mon.0) 965 : cluster [DBG] osdmap e240: 8 total, 8 up, 8 in 2021-02-10T04:30:57.476 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:30:57.516 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:57.583 INFO:tasks.rados.rados.0.smithi085.stdout:748: finishing write tid 2 to smithi08575397-24 2021-02-10T04:30:57.583 INFO:tasks.rados.rados.0.smithi085.stdout:748: finishing write tid 3 to smithi08575397-24 2021-02-10T04:30:57.583 INFO:tasks.rados.rados.0.smithi085.stdout:748: finishing write tid 4 to smithi08575397-24 2021-02-10T04:30:57.583 INFO:tasks.rados.rados.0.smithi085.stdout:748: finishing write tid 5 to smithi08575397-24 2021-02-10T04:30:57.583 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 24 v 81 (ObjNum 294 snap 74 seq_num 294) dirty exists 2021-02-10T04:30:57.583 INFO:tasks.rados.rados.0.smithi085.stdout:748: left oid 24 (ObjNum 294 snap 74 seq_num 294) 2021-02-10T04:30:57.583 INFO:tasks.rados.rados.0.smithi085.stdout:749: finishing rollback tid 0 to smithi08575397-40 2021-02-10T04:30:57.583 INFO:tasks.rados.rados.0.smithi085.stdout:749: finishing rollback tid 1 to smithi08575397-40 2021-02-10T04:30:57.583 INFO:tasks.rados.rados.0.smithi085.stdout:749: finishing rollback tid 2 to smithi08575397-40 2021-02-10T04:30:57.584 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 348 (ObjNum 280 snap 70 seq_num 280) dirty exists 2021-02-10T04:30:57.584 INFO:tasks.rados.rados.0.smithi085.stdout:750: finishing rollback tid 1 to smithi08575397-43 2021-02-10T04:30:57.584 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 235 (ObjNum 82682752 snap 22030 seq_num 82433488) dirty dne 2021-02-10T04:30:57.584 INFO:tasks.rados.rados.0.smithi085.stdout:747: done (15 left) 2021-02-10T04:30:57.584 INFO:tasks.rados.rados.0.smithi085.stdout:748: done (14 left) 2021-02-10T04:30:57.584 INFO:tasks.rados.rados.0.smithi085.stdout:749: done (13 left) 2021-02-10T04:30:57.584 INFO:tasks.rados.rados.0.smithi085.stdout:750: done (12 left) 2021-02-10T04:30:57.584 INFO:tasks.rados.rados.0.smithi085.stdout:751: delete oid 50 current snap is 74 2021-02-10T04:30:57.589 INFO:tasks.rados.rados.0.smithi085.stdout:751: done (12 left) 2021-02-10T04:30:57.589 INFO:tasks.rados.rados.0.smithi085.stdout:752: rollback oid 26 current snap is 74 2021-02-10T04:30:57.589 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 26 to 66 2021-02-10T04:30:57.589 INFO:tasks.rados.rados.0.smithi085.stdout:753: delete oid 8 current snap is 74 2021-02-10T04:30:57.594 INFO:tasks.rados.rados.0.smithi085.stdout:752: finishing rollback tid 0 to smithi08575397-26 2021-02-10T04:30:57.594 INFO:tasks.rados.rados.0.smithi085.stdout:752: finishing rollback tid 1 to smithi08575397-26 2021-02-10T04:30:57.594 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 54 (ObjNum 71101512 snap 22030 seq_num 3) dirty dne 2021-02-10T04:30:57.598 INFO:tasks.rados.rados.0.smithi085.stdout:752: done (13 left) 2021-02-10T04:30:57.599 INFO:tasks.rados.rados.0.smithi085.stdout:753: done (12 left) 2021-02-10T04:30:57.599 INFO:tasks.rados.rados.0.smithi085.stdout:754: write oid 6 current snap is 74 2021-02-10T04:30:57.599 INFO:tasks.rados.rados.0.smithi085.stdout:754: seq_num 297 ranges {591679=486720,1813152=490434} 2021-02-10T04:30:57.602 INFO:tasks.rados.rados.0.smithi085.stdout:754: writing smithi08575397-6 from 591679 to 1078399 tid 1 2021-02-10T04:30:57.609 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:30:57.612 INFO:tasks.rados.rados.0.smithi085.stdout:754: writing smithi08575397-6 from 1813152 to 2303586 tid 2 2021-02-10T04:30:57.613 INFO:tasks.rados.rados.0.smithi085.stdout:755: read oid 16 snap -1 2021-02-10T04:30:57.613 INFO:tasks.rados.rados.0.smithi085.stdout:755: expect deleted 2021-02-10T04:30:57.613 INFO:tasks.rados.rados.0.smithi085.stdout:756: write (excl) oid 37 current snap is 74 2021-02-10T04:30:57.613 INFO:tasks.rados.rados.0.smithi085.stdout:754: finishing write tid 1 to smithi08575397-6 2021-02-10T04:30:57.614 INFO:tasks.rados.rados.0.smithi085.stdout:756: seq_num 298 ranges {608486=561969,1871634=634092,2950640=57846} 2021-02-10T04:30:57.620 INFO:tasks.rados.rados.0.smithi085.stdout:756: writing smithi08575397-37 from 608486 to 1170455 tid 1 2021-02-10T04:30:57.628 INFO:tasks.rados.rados.0.smithi085.stdout:756: writing smithi08575397-37 from 1871634 to 2505726 tid 2 2021-02-10T04:30:57.632 INFO:tasks.rados.rados.0.smithi085.stdout:756: writing smithi08575397-37 from 2950640 to 3008486 tid 3 2021-02-10T04:30:57.632 INFO:tasks.rados.rados.0.smithi085.stdout:754: finishing write tid 2 to smithi08575397-6 2021-02-10T04:30:57.632 INFO:tasks.rados.rados.0.smithi085.stdout:754: finishing write tid 3 to smithi08575397-6 2021-02-10T04:30:57.632 INFO:tasks.rados.rados.0.smithi085.stdout:754: finishing write tid 4 to smithi08575397-6 2021-02-10T04:30:57.633 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 228 (ObjNum 297 snap 74 seq_num 297) dirty exists 2021-02-10T04:30:57.633 INFO:tasks.rados.rados.0.smithi085.stdout:754: left oid 6 (ObjNum 297 snap 74 seq_num 297) 2021-02-10T04:30:57.633 INFO:tasks.rados.rados.0.smithi085.stdout:756: finishing write tid 1 to smithi08575397-37 2021-02-10T04:30:57.633 INFO:tasks.rados.rados.0.smithi085.stdout:754: done (14 left) 2021-02-10T04:30:57.633 INFO:tasks.rados.rados.0.smithi085.stdout:755: done (13 left) 2021-02-10T04:30:57.633 INFO:tasks.rados.rados.0.smithi085.stdout:757: snap_remove snap 66 2021-02-10T04:30:57.643 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:30:57 smithi164 conmon[40892]: debug 2021-02-10T04:30:57.332+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:57.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:57 smithi164 conmon[29288]: cluster 2021-02-10T04:30:56.358118+0000 mon.a (mon. 2021-02-10T04:30:57.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:57 smithi164 conmon[29288]: 0) 965 : cluster [DBG] osdmap e240: 8 total, 8 up, 8 in 2021-02-10T04:30:57.646 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:30:57.753 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:30:57.778 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:57.871 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:30:57.885 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:30:57.885 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:30:57.913 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:30:57 smithi085 conmon[41540]: debug 2021-02-10T04:30:57.913 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:30:57 smithi085 conmon[41540]: 2021-02-10T04:30:57.607+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:57.913 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:30:57 smithi085 conmon[41540]: 2021-02-10T04:30:57.975 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:30:58.001 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:58.061 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:30:57 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:30:58.061 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:30:57 smithi164 conmon[44544]: debug 2021-02-10T04:30:57.842+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:58.096 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:30:58.201 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:30:58.235 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:30:58.375 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:30:58 smithi164 conmon[40892]: debug 2021-02-10T04:30:58.060+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:58.377 INFO:tasks.rados.rados.0.smithi085.stdout:757: done (13 left) 2021-02-10T04:30:58.377 INFO:tasks.rados.rados.0.smithi085.stdout:758: write oid 16 current snap is 74 2021-02-10T04:30:58.377 INFO:tasks.rados.rados.0.smithi085.stdout:758: seq_num 299 ranges {625293=637218,1813763=494103,2978063=47230} 2021-02-10T04:30:58.387 INFO:tasks.rados.rados.0.smithi085.stdout:758: writing smithi08575397-16 from 625293 to 1262511 tid 1 2021-02-10T04:30:58.393 INFO:tasks.rados.rados.0.smithi085.stdout:758: writing smithi08575397-16 from 1813763 to 2307866 tid 2 2021-02-10T04:30:58.394 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is set 2021-02-10T04:30:58.398 INFO:tasks.rados.rados.0.smithi085.stdout:758: writing smithi08575397-16 from 2978063 to 3025293 tid 3 2021-02-10T04:30:58.398 INFO:tasks.rados.rados.0.smithi085.stdout:759: delete oid 11 current snap is 74 2021-02-10T04:30:58.398 INFO:tasks.rados.rados.0.smithi085.stdout:756: finishing write tid 2 to smithi08575397-37 2021-02-10T04:30:58.398 INFO:tasks.rados.rados.0.smithi085.stdout:756: finishing write tid 3 to smithi08575397-37 2021-02-10T04:30:58.398 INFO:tasks.rados.rados.0.smithi085.stdout:756: finishing write tid 4 to smithi08575397-37 2021-02-10T04:30:58.398 INFO:tasks.rados.rados.0.smithi085.stdout:756: finishing write tid 5 to smithi08575397-37 2021-02-10T04:30:58.398 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 353 (ObjNum 298 snap 74 seq_num 298) dirty exists 2021-02-10T04:30:58.399 INFO:tasks.rados.rados.0.smithi085.stdout:756: left oid 37 (ObjNum 298 snap 74 seq_num 298) 2021-02-10T04:30:58.434 INFO:tasks.rados.rados.0.smithi085.stdout:758: finishing write tid 1 to smithi08575397-16 2021-02-10T04:30:58.434 INFO:tasks.rados.rados.0.smithi085.stdout:758: finishing write tid 2 to smithi08575397-16 2021-02-10T04:30:58.441 INFO:tasks.rados.rados.0.smithi085.stdout:758: finishing write tid 3 to smithi08575397-16 2021-02-10T04:30:58.441 INFO:tasks.rados.rados.0.smithi085.stdout:758: finishing write tid 4 to smithi08575397-16 2021-02-10T04:30:58.441 INFO:tasks.rados.rados.0.smithi085.stdout:758: finishing write tid 5 to smithi08575397-16 2021-02-10T04:30:58.442 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 361 (ObjNum 299 snap 74 seq_num 299) dirty exists 2021-02-10T04:30:58.442 INFO:tasks.rados.rados.0.smithi085.stdout:758: left oid 16 (ObjNum 299 snap 74 seq_num 299) 2021-02-10T04:30:58.442 INFO:tasks.rados.rados.0.smithi085.stdout:756: done (14 left) 2021-02-10T04:30:58.442 INFO:tasks.rados.rados.0.smithi085.stdout:758: done (13 left) 2021-02-10T04:30:58.443 INFO:tasks.rados.rados.0.smithi085.stdout:759: done (12 left) 2021-02-10T04:30:58.443 INFO:tasks.rados.rados.0.smithi085.stdout:760: write (excl) oid 37 current snap is 74 2021-02-10T04:30:58.443 INFO:tasks.rados.rados.0.smithi085.stdout:760: seq_num 301 ranges {658907=787716,1930727=497772,3032909=25998} 2021-02-10T04:30:58.444 INFO:tasks.rados.rados.0.smithi085.stdout:760: writing smithi08575397-37 from 658907 to 1446623 tid 1 2021-02-10T04:30:58.450 INFO:tasks.rados.rados.0.smithi085.stdout:760: writing smithi08575397-37 from 1930727 to 2428499 tid 2 2021-02-10T04:30:58.454 INFO:tasks.rados.rados.0.smithi085.stdout:760: writing smithi08575397-37 from 3032909 to 3058907 tid 3 2021-02-10T04:30:58.454 INFO:tasks.rados.rados.0.smithi085.stdout:761: delete oid 5 current snap is 74 2021-02-10T04:30:58.454 INFO:tasks.rados.rados.0.smithi085.stdout:760: finishing write tid 1 to smithi08575397-37 2021-02-10T04:30:58.455 INFO:tasks.rados.rados.0.smithi085.stdout:760: finishing write tid 2 to smithi08575397-37 2021-02-10T04:30:58.456 INFO:tasks.rados.rados.0.smithi085.stdout:761: done (13 left) 2021-02-10T04:30:58.456 INFO:tasks.rados.rados.0.smithi085.stdout:762: read oid 24 snap -1 2021-02-10T04:30:58.456 INFO:tasks.rados.rados.0.smithi085.stdout:762: expect (ObjNum 294 snap 74 seq_num 294) 2021-02-10T04:30:58.456 INFO:tasks.rados.rados.0.smithi085.stdout:763: delete oid 46 current snap is 74 2021-02-10T04:30:58.456 INFO:tasks.rados.rados.0.smithi085.stdout:760: finishing write tid 3 to smithi08575397-37 2021-02-10T04:30:58.457 INFO:tasks.rados.rados.0.smithi085.stdout:760: finishing write tid 4 to smithi08575397-37 2021-02-10T04:30:58.457 INFO:tasks.rados.rados.0.smithi085.stdout:760: finishing write tid 5 to smithi08575397-37 2021-02-10T04:30:58.457 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 366 (ObjNum 301 snap 74 seq_num 301) dirty exists 2021-02-10T04:30:58.457 INFO:tasks.rados.rados.0.smithi085.stdout:760: left oid 37 (ObjNum 301 snap 74 seq_num 301) 2021-02-10T04:30:58.460 INFO:tasks.rados.rados.0.smithi085.stdout:760: done (14 left) 2021-02-10T04:30:58.460 INFO:tasks.rados.rados.0.smithi085.stdout:763: done (13 left) 2021-02-10T04:30:58.460 INFO:tasks.rados.rados.0.smithi085.stdout:764: rollback oid 23 current snap is 74 2021-02-10T04:30:58.460 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 23 to 72 2021-02-10T04:30:58.460 INFO:tasks.rados.rados.0.smithi085.stdout:765: read oid 22 snap -1 2021-02-10T04:30:58.460 INFO:tasks.rados.rados.0.smithi085.stdout:765: expect (ObjNum 127 snap 23 seq_num 127) 2021-02-10T04:30:58.460 INFO:tasks.rados.rados.0.smithi085.stdout:766: delete oid 11 current snap is 74 2021-02-10T04:30:58.463 INFO:tasks.rados.rados.0.smithi085.stdout:762: expect (ObjNum 294 snap 74 seq_num 294) 2021-02-10T04:30:58.553 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[52836]: debug 2021-02-10T04:30:58.322+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:30:58.554 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[29319]: cluster 2021-02-10T04:30:58.554 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[29319]: 2021-02 2021-02-10T04:30:58.554 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[29319]: -10T04:30:57.376169+0000 mon.a (mon.0) 966 : cluster [DBG] osdmap e241: 8 total, 8 up, 8 in 2021-02-10T04:30:58.554 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[29319]: 2021-02-10T04:30:58.554 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[29319]: audit 2021 2021-02-10T04:30:58.554 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[29319]: -02-10T04:30:57.815050+0000 mgr.y ( 2021-02-10T04:30:58.554 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[29319]: mgr.14140) 423 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:30:58.554 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[29319]: audit 2021-02-10T04:30:58.554 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[29319]: 2021-02-10T04:30:57. 2021-02-10T04:30:58.554 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[29319]: 918694+0000 mon.c (mon.1) 47 : audit 2021-02-10T04:30:58.555 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[29319]: 2021-02-10T04:30:58.555 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[29319]: [INF] from='client.? 172.21.15.85:0/4063623053' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:30:58.555 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:30:58.555 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[29319]: 02-10T04:30:57.919083+0000 mon.a (mon. 2021-02-10T04:30:58.555 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[29319]: 0) 967 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:30:58.555 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:30:58.555 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[29319]: -02-10T04 2021-02-10T04:30:58.555 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[29319]: :30:58.027051+0000 mgr.y (mgr.14140 2021-02-10T04:30:58.555 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[29319]: ) 424 : cluster [DBG] 2021-02-10T04:30:58.555 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[29319]: pgmap v511: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim, 165 active+clean; 163 MiB data, 529 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 2.5 MiB/s wr, 20 op/s 2021-02-10T04:30:58.556 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: cluster 2021-02-10T04:30:58.556 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 2021-02-10T04:30:58.556 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 2021 2021-02-10T04:30:58.556 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: - 2021-02-10T04:30:58.556 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 02 2021-02-10T04:30:58.556 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: - 2021-02-10T04:30:58.556 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 10 2021-02-10T04:30:58.556 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: T 2021-02-10T04:30:58.556 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 04 2021-02-10T04:30:58.556 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: : 2021-02-10T04:30:58.556 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 30 2021-02-10T04:30:58.557 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: : 2021-02-10T04:30:58.557 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 57 2021-02-10T04:30:58.557 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: . 2021-02-10T04:30:58.557 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 376169 2021-02-10T04:30:58.557 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: +0000 2021-02-10T04:30:58.557 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 2021-02-10T04:30:58.557 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: mon.a 2021-02-10T04:30:58.557 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: ( 2021-02-10T04:30:58.557 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: mon 2021-02-10T04:30:58.557 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: . 2021-02-10T04:30:58.557 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 0 2021-02-10T04:30:58.557 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: ) 2021-02-10T04:30:58.557 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 966 2021-02-10T04:30:58.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: : 2021-02-10T04:30:58.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: cluster 2021-02-10T04:30:58.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 2021-02-10T04:30:58.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: [DBG] 2021-02-10T04:30:58.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 2021-02-10T04:30:58.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: osdmap e241: 8 total, 8 up, 8 in 2021-02-10T04:30:58.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 2021-02-10T04:30:58.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: audit 2021-02-10T04:30:58.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 2021-02-10T04:30:58.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 2021 2021-02-10T04:30:58.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: - 2021-02-10T04:30:58.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 02 2021-02-10T04:30:58.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: - 2021-02-10T04:30:58.559 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 10 2021-02-10T04:30:58.559 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: T 2021-02-10T04:30:58.559 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 04 2021-02-10T04:30:58.559 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: : 2021-02-10T04:30:58.559 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 30 2021-02-10T04:30:58.559 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: : 2021-02-10T04:30:58.559 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 57 2021-02-10T04:30:58.559 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: . 2021-02-10T04:30:58.559 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 815050 2021-02-10T04:30:58.559 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: +0000 2021-02-10T04:30:58.559 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 2021-02-10T04:30:58.559 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: mgr.y 2021-02-10T04:30:58.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: ( 2021-02-10T04:30:58.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: mgr 2021-02-10T04:30:58.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: . 2021-02-10T04:30:58.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 14140 2021-02-10T04:30:58.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: ) 2021-02-10T04:30:58.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 423 2021-02-10T04:30:58.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: : 2021-02-10T04:30:58.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: audit 2021-02-10T04:30:58.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 2021-02-10T04:30:58.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: [DBG] 2021-02-10T04:30:58.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 2021-02-10T04:30:58.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:30:58.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 2021-02-10T04:30:58.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: audit 2021-02-10T04:30:58.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 2021-02-10T04:30:58.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 2021 2021-02-10T04:30:58.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: - 2021-02-10T04:30:58.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 02 2021-02-10T04:30:58.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: - 2021-02-10T04:30:58.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 10 2021-02-10T04:30:58.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: T 2021-02-10T04:30:58.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 04 2021-02-10T04:30:58.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: : 2021-02-10T04:30:58.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: 30 2021-02-10T04:30:58.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: :57.918694+0000 mon.c (mon.1) 47 : 2021-02-10T04:30:58.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: audit [INF] from='client.? 172.21.15.85:0/4063623053' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:30:58.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: audit 2021-02-10T04:30:57 2021-02-10T04:30:58.562 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: .919083+0000 mon.a (mon.0) 967 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:30:58.562 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: cluster 2021-02-10T04:30 2021-02-10T04:30:58.562 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: :58.027051+0000 mgr.y (mgr.14140) 424 : 2021-02-10T04:30:58.562 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:58 smithi085 conmon[35961]: cluster [DBG] pgmap v511: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim, 165 active+clean; 163 MiB data, 529 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 2.5 MiB/s wr, 20 op/s 2021-02-10T04:30:58.589 INFO:tasks.rados.rados.0.smithi085.stdout:762: done (15 left) 2021-02-10T04:30:58.589 INFO:tasks.rados.rados.0.smithi085.stdout:766: done (14 left) 2021-02-10T04:30:58.589 INFO:tasks.rados.rados.0.smithi085.stdout:767: copy_from oid 13 from oid 37 current snap is 74 2021-02-10T04:30:58.589 INFO:tasks.rados.rados.0.smithi085.stdout:764: finishing rollback tid 0 to smithi08575397-23 2021-02-10T04:30:58.590 INFO:tasks.rados.rados.0.smithi085.stdout:764: finishing rollback tid 1 to smithi08575397-23 2021-02-10T04:30:58.590 INFO:tasks.rados.rados.0.smithi085.stdout:764: finishing rollback tid 2 to smithi08575397-23 2021-02-10T04:30:58.590 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 116 (ObjNum 124 snap 23 seq_num 124) dirty exists 2021-02-10T04:30:58.590 INFO:tasks.rados.rados.0.smithi085.stdout:765: expect (ObjNum 127 snap 23 seq_num 127) 2021-02-10T04:30:58.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:58 smithi164 conmon[29288]: cluster 2021-02-10T04:30:57.376169+0000 2021-02-10T04:30:58.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:58 smithi164 conmon[29288]: mon.a (mon.0) 966 : cluster [DBG] osdmap e241: 8 total, 8 up, 8 in 2021-02-10T04:30:58.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:58 smithi164 conmon[29288]: audit 2021-02-10T04:30:57.815050+0000 mgr.y (mgr 2021-02-10T04:30:58.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:58 smithi164 conmon[29288]: .14140) 423 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:30:58.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:58 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:30:58.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:58 smithi164 conmon[29288]: 02-10T04:30:57.918694+0000 mon.c (mon 2021-02-10T04:30:58.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:58 smithi164 conmon[29288]: .1) 47 : audit [INF] from='client.? 172.21.15.85:0/4063623053' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:30:58.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:58 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:30:58.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:58 smithi164 conmon[29288]: 02-10T04:30:57.919083+0000 2021-02-10T04:30:58.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:58 smithi164 conmon[29288]: mon.a (mon.0) 967 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:30:58.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:58 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:30:58.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:58 smithi164 conmon[29288]: -02-10T04:30:58.027051+0000 2021-02-10T04:30:58.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:58 smithi164 conmon[29288]: mgr.y (mgr.14140) 424 : cluster [DBG] 2021-02-10T04:30:58.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:58 smithi164 conmon[29288]: pgmap v511: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim, 165 active+clean; 163 MiB data, 529 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 2.5 MiB/s wr, 20 op/s 2021-02-10T04:30:58.686 INFO:tasks.rados.rados.0.smithi085.stdout:767: finishing copy_from to smithi08575397-13 2021-02-10T04:30:58.687 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 366 (ObjNum 301 snap 74 seq_num 301) dirty exists 2021-02-10T04:30:58.687 INFO:tasks.rados.rados.0.smithi085.stdout:764: done (14 left) 2021-02-10T04:30:58.687 INFO:tasks.rados.rados.0.smithi085.stdout:765: done (13 left) 2021-02-10T04:30:58.687 INFO:tasks.rados.rados.0.smithi085.stdout:768: rollback oid 30 current snap is 74 2021-02-10T04:30:58.687 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 30 to 63 2021-02-10T04:30:58.687 INFO:tasks.rados.rados.0.smithi085.stdout:767: finishing copy_from racing read to smithi08575397-13 2021-02-10T04:30:58.687 INFO:tasks.rados.rados.0.smithi085.stdout:767: done (13 left) 2021-02-10T04:30:58.687 INFO:tasks.rados.rados.0.smithi085.stdout:769: write (excl) oid 13 current snap is 74 2021-02-10T04:30:58.687 INFO:tasks.rados.rados.0.smithi085.stdout:769: seq_num 305 ranges {726135=405065,1764655=788757,3026248=99887} 2021-02-10T04:30:58.692 INFO:tasks.rados.rados.0.smithi085.stdout:769: writing smithi08575397-13 from 726135 to 1131200 tid 1 2021-02-10T04:30:58.702 INFO:tasks.rados.rados.0.smithi085.stdout:769: writing smithi08575397-13 from 1764655 to 2553412 tid 2 2021-02-10T04:30:58.705 INFO:tasks.rados.rados.0.smithi085.stdout:769: writing smithi08575397-13 from 3026248 to 3126135 tid 3 2021-02-10T04:30:58.705 INFO:tasks.rados.rados.0.smithi085.stdout:770: write (excl) oid 36 current snap is 74 2021-02-10T04:30:58.706 INFO:tasks.rados.rados.0.smithi085.stdout:770: seq_num 306 ranges {742942=480314,1823137=648768} 2021-02-10T04:30:58.712 INFO:tasks.rados.rados.0.smithi085.stdout:770: writing smithi08575397-36 from 742942 to 1223256 tid 1 2021-02-10T04:30:58.719 INFO:tasks.rados.rados.0.smithi085.stdout:770: writing smithi08575397-36 from 1823137 to 2471905 tid 2 2021-02-10T04:30:58.719 INFO:tasks.rados.rados.0.smithi085.stdout:771: copy_from oid 31 from oid 11 current snap is 74 2021-02-10T04:30:58.719 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:30:58.781 INFO:tasks.rados.rados.0.smithi085.stdout:769: finishing write tid 1 to smithi08575397-13 2021-02-10T04:30:58.783 INFO:tasks.rados.rados.0.smithi085.stdout:768: finishing rollback tid 0 to smithi08575397-30 2021-02-10T04:30:58.783 INFO:tasks.rados.rados.0.smithi085.stdout:770: finishing write tid 1 to smithi08575397-36 2021-02-10T04:30:58.783 INFO:tasks.rados.rados.0.smithi085.stdout:768: finishing rollback tid 1 to smithi08575397-30 2021-02-10T04:30:58.783 INFO:tasks.rados.rados.0.smithi085.stdout:768: finishing rollback tid 2 to smithi08575397-30 2021-02-10T04:30:58.783 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 238 (ObjNum 222 snap 50 seq_num 222) dirty exists 2021-02-10T04:30:58.783 INFO:tasks.rados.rados.0.smithi085.stdout:768: done (15 left) 2021-02-10T04:30:58.783 INFO:tasks.rados.rados.0.smithi085.stdout:772: rollback oid 21 current snap is 74 2021-02-10T04:30:58.783 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 21 to 71 2021-02-10T04:30:58.783 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:30:58.786 INFO:tasks.rados.rados.0.smithi085.stdout:770: finishing write tid 2 to smithi08575397-36 2021-02-10T04:30:58.786 INFO:tasks.rados.rados.0.smithi085.stdout:770: finishing write tid 3 to smithi08575397-36 2021-02-10T04:30:58.786 INFO:tasks.rados.rados.0.smithi085.stdout:769: finishing write tid 2 to smithi08575397-13 2021-02-10T04:30:58.786 INFO:tasks.rados.rados.0.smithi085.stdout:769: finishing write tid 3 to smithi08575397-13 2021-02-10T04:30:58.787 INFO:tasks.rados.rados.0.smithi085.stdout:769: finishing write tid 4 to smithi08575397-13 2021-02-10T04:30:58.788 INFO:tasks.rados.rados.0.smithi085.stdout:770: finishing write tid 4 to smithi08575397-36 2021-02-10T04:30:58.788 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 119 (ObjNum 306 snap 74 seq_num 306) dirty exists 2021-02-10T04:30:58.788 INFO:tasks.rados.rados.0.smithi085.stdout:770: left oid 36 (ObjNum 306 snap 74 seq_num 306) 2021-02-10T04:30:58.788 INFO:tasks.rados.rados.0.smithi085.stdout:770: done (15 left) 2021-02-10T04:30:58.788 INFO:tasks.rados.rados.0.smithi085.stdout:773: delete oid 24 current snap is 74 2021-02-10T04:30:58.788 INFO:tasks.rados.rados.0.smithi085.stdout:772: finishing rollback tid 0 to smithi08575397-21 2021-02-10T04:30:58.788 INFO:tasks.rados.rados.0.smithi085.stdout:769: finishing write tid 5 to smithi08575397-13 2021-02-10T04:30:58.788 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 370 (ObjNum 305 snap 74 seq_num 305) dirty exists 2021-02-10T04:30:58.789 INFO:tasks.rados.rados.0.smithi085.stdout:769: left oid 13 (ObjNum 305 snap 74 seq_num 305) 2021-02-10T04:30:58.789 INFO:tasks.rados.rados.0.smithi085.stdout:772: finishing rollback tid 1 to smithi08575397-21 2021-02-10T04:30:58.789 INFO:tasks.rados.rados.0.smithi085.stdout:772: finishing rollback tid 2 to smithi08575397-21 2021-02-10T04:30:58.789 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 371 (ObjNum 194 snap 43 seq_num 194) dirty exists 2021-02-10T04:30:58.789 INFO:tasks.rados.rados.0.smithi085.stdout:769: done (15 left) 2021-02-10T04:30:58.789 INFO:tasks.rados.rados.0.smithi085.stdout:772: done (14 left) 2021-02-10T04:30:58.789 INFO:tasks.rados.rados.0.smithi085.stdout:773: done (13 left) 2021-02-10T04:30:58.789 INFO:tasks.rados.rados.0.smithi085.stdout:774: write oid 1 current snap is 74 2021-02-10T04:30:58.790 INFO:tasks.rados.rados.0.smithi085.stdout:774: seq_num 308 ranges {776556=630812,1823748=652437,3108517=68039} 2021-02-10T04:30:58.796 INFO:tasks.rados.rados.0.smithi085.stdout:774: writing smithi08575397-1 from 776556 to 1407368 tid 1 2021-02-10T04:30:58.803 INFO:tasks.rados.rados.0.smithi085.stdout:774: writing smithi08575397-1 from 1823748 to 2476185 tid 2 2021-02-10T04:30:58.807 INFO:tasks.rados.rados.0.smithi085.stdout:774: writing smithi08575397-1 from 3108517 to 3176556 tid 3 2021-02-10T04:30:58.808 INFO:tasks.rados.rados.0.smithi085.stdout:775: read oid 37 snap -1 2021-02-10T04:30:58.808 INFO:tasks.rados.rados.0.smithi085.stdout:775: expect (ObjNum 301 snap 74 seq_num 301) 2021-02-10T04:30:58.808 INFO:tasks.rados.rados.0.smithi085.stdout:776: read oid 43 snap 72 2021-02-10T04:30:58.808 INFO:tasks.rados.rados.0.smithi085.stdout:776: expect deleted 2021-02-10T04:30:58.808 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:30:58.808 INFO:tasks.rados.rados.0.smithi085.stdout:774: finishing write tid 1 to smithi08575397-1 2021-02-10T04:30:58.808 INFO:tasks.rados.rados.0.smithi085.stdout:774: finishing write tid 2 to smithi08575397-1 2021-02-10T04:30:58.810 INFO:tasks.rados.rados.0.smithi085.stdout:776: done (15 left) 2021-02-10T04:30:58.810 INFO:tasks.rados.rados.0.smithi085.stdout:777: read oid 43 snap -1 2021-02-10T04:30:58.811 INFO:tasks.rados.rados.0.smithi085.stdout:777: expect deleted 2021-02-10T04:30:58.811 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:30:58.812 INFO:tasks.rados.rados.0.smithi085.stdout:777: done (15 left) 2021-02-10T04:30:58.812 INFO:tasks.rados.rados.0.smithi085.stdout:778: read oid 36 snap -1 2021-02-10T04:30:58.812 INFO:tasks.rados.rados.0.smithi085.stdout:778: expect (ObjNum 306 snap 74 seq_num 306) 2021-02-10T04:30:58.812 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:30:58.815 INFO:tasks.rados.rados.0.smithi085.stdout:775: expect (ObjNum 301 snap 74 seq_num 301) 2021-02-10T04:30:59.102 INFO:tasks.rados.rados.0.smithi085.stdout:775: done (15 left) 2021-02-10T04:30:59.103 INFO:tasks.rados.rados.0.smithi085.stdout:779: snap_create 2021-02-10T04:30:59.103 INFO:tasks.rados.rados.0.smithi085.stdout:774: finishing write tid 3 to smithi08575397-1 2021-02-10T04:30:59.103 INFO:tasks.rados.rados.0.smithi085.stdout:774: finishing write tid 4 to smithi08575397-1 2021-02-10T04:30:59.103 INFO:tasks.rados.rados.0.smithi085.stdout:774: finishing write tid 5 to smithi08575397-1 2021-02-10T04:30:59.103 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 236 (ObjNum 308 snap 74 seq_num 308) dirty exists 2021-02-10T04:30:59.103 INFO:tasks.rados.rados.0.smithi085.stdout:774: left oid 1 (ObjNum 308 snap 74 seq_num 308) 2021-02-10T04:30:59.104 INFO:tasks.rados.rados.0.smithi085.stdout:778: expect (ObjNum 306 snap 74 seq_num 306) 2021-02-10T04:30:59.156 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:30:59.348 INFO:tasks.rados.rados.0.smithi085.stdout:771: finishing copy_from to smithi08575397-31 2021-02-10T04:30:59.348 INFO:tasks.rados.rados.0.smithi085.stdout:771: got expected ENOENT (src dne) 2021-02-10T04:30:59.348 INFO:tasks.rados.rados.0.smithi085.stdout:771: finishing copy_from racing read to smithi08575397-31 2021-02-10T04:30:59.385 INFO:tasks.rados.rados.0.smithi085.stdout:771: done (15 left) 2021-02-10T04:30:59.385 INFO:tasks.rados.rados.0.smithi085.stdout:774: done (14 left) 2021-02-10T04:30:59.385 INFO:tasks.rados.rados.0.smithi085.stdout:778: done (13 left) 2021-02-10T04:30:59.385 INFO:tasks.rados.rados.0.smithi085.stdout:779: done (12 left) 2021-02-10T04:30:59.385 INFO:tasks.rados.rados.0.smithi085.stdout:780: delete oid 50 current snap is 75 2021-02-10T04:30:59.392 INFO:tasks.rados.rados.0.smithi085.stdout:780: done (12 left) 2021-02-10T04:30:59.392 INFO:tasks.rados.rados.0.smithi085.stdout:781: rollback oid 26 current snap is 75 2021-02-10T04:30:59.392 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 26 to 73 2021-02-10T04:30:59.392 INFO:tasks.rados.rados.0.smithi085.stdout:782: write (excl) oid 28 current snap is 75 2021-02-10T04:30:59.392 INFO:tasks.rados.rados.0.smithi085.stdout:782: seq_num 310 ranges {410170=781310,1940712=656106,3163363=46807} 2021-02-10T04:30:59.399 INFO:tasks.rados.rados.0.smithi085.stdout:782: writing smithi08575397-28 from 410170 to 1191480 tid 1 2021-02-10T04:30:59.408 INFO:tasks.rados.rados.0.smithi085.stdout:782: writing smithi08575397-28 from 1940712 to 2596818 tid 2 2021-02-10T04:30:59.411 INFO:tasks.rados.rados.0.smithi085.stdout:782: writing smithi08575397-28 from 3163363 to 3210170 tid 3 2021-02-10T04:30:59.411 INFO:tasks.rados.rados.0.smithi085.stdout:783: snap_create 2021-02-10T04:30:59.411 INFO:tasks.rados.rados.0.smithi085.stdout:781: finishing rollback tid 1 to smithi08575397-26 2021-02-10T04:30:59.411 INFO:tasks.rados.rados.0.smithi085.stdout:781: finishing rollback tid 2 to smithi08575397-26 2021-02-10T04:30:59.412 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 56 (ObjNum 288 snap 71 seq_num 288) dirty exists 2021-02-10T04:30:59.412 INFO:tasks.rados.rados.0.smithi085.stdout:782: finishing write tid 1 to smithi08575397-28 2021-02-10T04:30:59.412 INFO:tasks.rados.rados.0.smithi085.stdout:782: finishing write tid 2 to smithi08575397-28 2021-02-10T04:30:59.414 INFO:tasks.rados.rados.0.smithi085.stdout:782: finishing write tid 3 to smithi08575397-28 2021-02-10T04:30:59.414 INFO:tasks.rados.rados.0.smithi085.stdout:782: finishing write tid 4 to smithi08575397-28 2021-02-10T04:30:59.414 INFO:tasks.rados.rados.0.smithi085.stdout:782: finishing write tid 5 to smithi08575397-28 2021-02-10T04:30:59.414 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 89 (ObjNum 310 snap 75 seq_num 310) dirty exists 2021-02-10T04:30:59.414 INFO:tasks.rados.rados.0.smithi085.stdout:782: left oid 28 (ObjNum 310 snap 75 seq_num 310) 2021-02-10T04:30:59.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:59 smithi085 conmon[29319]: cluster 2021-02-10T04:30:59.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:59 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:30:59.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:59 smithi085 conmon[29319]: 30:58.365764+0000 mon.a (mon.0) 968 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:30:59.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:59 smithi085 conmon[29319]: audit 2021-02-10T04:30:59.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:59 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:30:59.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:59 smithi085 conmon[29319]: 04:30:58.378207+0000 mon.a (mon.0) 969 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:30:59.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:59 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:30:59.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:59 smithi085 conmon[29319]: :30:58.378271+0000 mon.a (mon.0) 2021-02-10T04:30:59.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:30:59 smithi085 conmon[29319]: 970 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2021-02-10T04:30:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:59 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:30:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:59 smithi085 conmon[35961]: 02-10T04:30:58.365764+0000 mon.a (mon.0) 968 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:30:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:59 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:30:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:59 smithi085 conmon[35961]: 30:58.378207+0000 mon.a (mon.0) 969 2021-02-10T04:30:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:59 smithi085 conmon[35961]: : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:30:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:59 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:30:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:59 smithi085 conmon[35961]: 10T04:30:58.378271+0000 mon.a ( 2021-02-10T04:30:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:30:59 smithi085 conmon[35961]: mon.0) 970 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2021-02-10T04:30:59.447 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:30:59.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:59 smithi164 conmon[29288]: cluster 2021-02-10T04:30:58.365764+0000 mon.a (mon. 2021-02-10T04:30:59.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:59 smithi164 conmon[29288]: 0) 968 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:30:59.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:59 smithi164 conmon[29288]: audit 2021-02-10T04:30:58.378207+0000 mon.a (mon.0) 969 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:30:59.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:59 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:30:59.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:30:59 smithi164 conmon[29288]: 10T04:30:58.378271+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2021-02-10T04:30:59.996 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:31:00.399 INFO:tasks.rados.rados.0.smithi085.stdout:781: done (14 left) 2021-02-10T04:31:00.399 INFO:tasks.rados.rados.0.smithi085.stdout:782: done (13 left) 2021-02-10T04:31:00.399 INFO:tasks.rados.rados.0.smithi085.stdout:783: done (12 left) 2021-02-10T04:31:00.399 INFO:tasks.rados.rados.0.smithi085.stdout:784: read oid 5 snap 71 2021-02-10T04:31:00.399 INFO:tasks.rados.rados.0.smithi085.stdout:784: expect (ObjNum 286 snap 70 seq_num 286) 2021-02-10T04:31:00.399 INFO:tasks.rados.rados.0.smithi085.stdout:785: write (excl) oid 7 current snap is 76 2021-02-10T04:31:00.399 INFO:tasks.rados.rados.0.smithi085.stdout:785: seq_num 311 ranges {426977=456559,1482841=799764,2958080=268897} 2021-02-10T04:31:00.402 INFO:tasks.rados.rados.0.smithi085.stdout:785: writing smithi08575397-7 from 426977 to 883536 tid 1 2021-02-10T04:31:00.412 INFO:tasks.rados.rados.0.smithi085.stdout:785: writing smithi08575397-7 from 1482841 to 2282605 tid 2 2021-02-10T04:31:00.418 INFO:tasks.rados.rados.0.smithi085.stdout:785: writing smithi08575397-7 from 2958080 to 3226977 tid 3 2021-02-10T04:31:00.418 INFO:tasks.rados.rados.0.smithi085.stdout:786: read oid 46 snap -1 2021-02-10T04:31:00.418 INFO:tasks.rados.rados.0.smithi085.stdout:786: expect deleted 2021-02-10T04:31:00.419 INFO:tasks.rados.rados.0.smithi085.stdout:787: read oid 2 snap 73 2021-02-10T04:31:00.419 INFO:tasks.rados.rados.0.smithi085.stdout:787: expect (ObjNum 246 snap 57 seq_num 246) 2021-02-10T04:31:00.419 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:00.419 INFO:tasks.rados.rados.0.smithi085.stdout:785: finishing write tid 1 to smithi08575397-7 2021-02-10T04:31:00.419 INFO:tasks.rados.rados.0.smithi085.stdout:784: expect (ObjNum 286 snap 70 seq_num 286) 2021-02-10T04:31:00.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:00 smithi164 conmon[29288]: cluster 2021-02-10T04:30:59.384792+0000 mon.a 2021-02-10T04:31:00.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:00 smithi164 conmon[29288]: (mon.0) 971 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2021-02-10T04:31:00.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:00 smithi164 conmon[29288]: cluster 2021-02-10T04:31:00.027558+0000 mgr.y (mgr.14140) 425 : cluster [DBG] pgmap v514: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 4 active+clean+snaptrim, 162 active+clean; 172 MiB data, 535 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 2.2 MiB/s wr, 16 op/s 2021-02-10T04:31:00.682 INFO:tasks.rados.rados.0.smithi085.stdout:785: finishing write tid 2 to smithi08575397-7 2021-02-10T04:31:00.682 INFO:tasks.rados.rados.0.smithi085.stdout:784: done (15 left) 2021-02-10T04:31:00.682 INFO:tasks.rados.rados.0.smithi085.stdout:788: snap_remove snap 71 2021-02-10T04:31:00.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:00 smithi085 conmon[29319]: cluster 2021-02-10T04:30:59.384792+0000 mon.a (mon.0) 971 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2021-02-10T04:31:00.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:00 smithi085 conmon[29319]: 2021-02-10T04:31:00.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:00 smithi085 conmon[29319]: cluster 2021-02-10T04:31:00.027558+0000 mgr.y (mgr.14140) 425 : cluster [DBG] pgmap v514: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 4 active+clean+snaptrim, 162 active+clean; 172 MiB data, 535 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 2.2 MiB/s wr, 16 op/s 2021-02-10T04:31:00.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:00 smithi085 conmon[35961]: cluster 2021-02-10T04:31:00.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:00 smithi085 conmon[35961]: 2021-02-10T04:31:00.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:00 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:31:00.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:00 smithi085 conmon[35961]: :30:59.384792+0000 mon.a (mon.0) 971 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2021-02-10T04:31:00.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:00 smithi085 conmon[35961]: cluster 2021-02-10T04:31:00.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:00 smithi085 conmon[35961]: 2021-02-10T04:31:00.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:00 smithi085 conmon[35961]: 2021 2021-02-10T04:31:00.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:00 smithi085 conmon[35961]: - 2021-02-10T04:31:00.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:00 smithi085 conmon[35961]: 02 2021-02-10T04:31:00.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:00 smithi085 conmon[35961]: - 2021-02-10T04:31:00.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:00 smithi085 conmon[35961]: 10 2021-02-10T04:31:00.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:00 smithi085 conmon[35961]: T04:31:00.027558+0000 mgr.y ( 2021-02-10T04:31:00.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:00 smithi085 conmon[35961]: mgr.14140) 425 : cluster [DBG] pgmap v514: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 4 active+clean+snaptrim, 162 active+clean; 172 MiB data, 535 MiB used, 715 GiB / 715 GiB avail; 5.0 MiB/s rd, 2.2 MiB/s wr, 16 op/s 2021-02-10T04:31:01.402 INFO:tasks.rados.rados.0.smithi085.stdout:788: done (15 left) 2021-02-10T04:31:01.402 INFO:tasks.rados.rados.0.smithi085.stdout:789: read oid 31 snap -1 2021-02-10T04:31:01.402 INFO:tasks.rados.rados.0.smithi085.stdout:789: expect deleted 2021-02-10T04:31:01.402 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:01.402 INFO:tasks.rados.rados.0.smithi085.stdout:785: finishing write tid 3 to smithi08575397-7 2021-02-10T04:31:01.402 INFO:tasks.rados.rados.0.smithi085.stdout:787: expect (ObjNum 246 snap 57 seq_num 246) 2021-02-10T04:31:01.505 INFO:tasks.rados.rados.0.smithi085.stdout:785: finishing write tid 4 to smithi08575397-7 2021-02-10T04:31:01.505 INFO:tasks.rados.rados.0.smithi085.stdout:785: finishing write tid 5 to smithi08575397-7 2021-02-10T04:31:01.505 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 243 (ObjNum 311 snap 76 seq_num 311) dirty exists 2021-02-10T04:31:01.505 INFO:tasks.rados.rados.0.smithi085.stdout:785: left oid 7 (ObjNum 311 snap 76 seq_num 311) 2021-02-10T04:31:01.505 INFO:tasks.rados.rados.0.smithi085.stdout:785: done (15 left) 2021-02-10T04:31:01.505 INFO:tasks.rados.rados.0.smithi085.stdout:786: done (14 left) 2021-02-10T04:31:01.505 INFO:tasks.rados.rados.0.smithi085.stdout:787: done (13 left) 2021-02-10T04:31:01.505 INFO:tasks.rados.rados.0.smithi085.stdout:790: snap_remove snap 73 2021-02-10T04:31:01.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:01 smithi085 conmon[29319]: cluster 2021-02-10T04:31:01.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:01 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:31:01.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:01 smithi085 conmon[29319]: 04:31:00.396156+0000 mon.a (mon.0) 972 : cluster [DBG] osdmap e244: 8 total, 8 up, 8 in 2021-02-10T04:31:01.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:01 smithi085 conmon[35961]: cluster 2021-02-10T04:31 2021-02-10T04:31:01.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:01 smithi085 conmon[35961]: :00.396156+0000 mon.a (mon.0) 972 : cluster [DBG] osdmap e244: 8 total, 8 up, 8 in 2021-02-10T04:31:01.875 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:31:01.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:01 smithi164 conmon[29288]: cluster 2021-02-10T04:31:00.396156+0000 mon.a (mon.0) 972 2021-02-10T04:31:01.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:01 smithi164 conmon[29288]: : cluster [DBG] osdmap e244: 8 total, 8 up, 8 in 2021-02-10T04:31:01.905 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:02.004 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:31:02.109 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:31:02.142 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:02.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:01 smithi164 conmon[33677]: debug 2021-02-10T04:31:01.976+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:02.411 INFO:tasks.rados.rados.0.smithi085.stdout:790: done (13 left) 2021-02-10T04:31:02.411 INFO:tasks.rados.rados.0.smithi085.stdout:791: read oid 8 snap -1 2021-02-10T04:31:02.411 INFO:tasks.rados.rados.0.smithi085.stdout:791: expect deleted 2021-02-10T04:31:02.411 INFO:tasks.rados.rados.0.smithi085.stdout:792: write oid 43 current snap is 76 2021-02-10T04:31:02.412 INFO:tasks.rados.rados.0.smithi085.stdout:792: seq_num 312 ranges {443784=648161,1657676=659775,2818209=425575} 2021-02-10T04:31:02.417 INFO:tasks.rados.rados.0.smithi085.stdout:792: writing smithi08575397-43 from 443784 to 1091945 tid 1 2021-02-10T04:31:02.425 INFO:tasks.rados.rados.0.smithi085.stdout:792: writing smithi08575397-43 from 1657676 to 2317451 tid 2 2021-02-10T04:31:02.431 INFO:tasks.rados.rados.0.smithi085.stdout:792: writing smithi08575397-43 from 2818209 to 3243784 tid 3 2021-02-10T04:31:02.431 INFO:tasks.rados.rados.0.smithi085.stdout:792: finishing write tid 1 to smithi08575397-43 2021-02-10T04:31:02.431 INFO:tasks.rados.rados.0.smithi085.stdout:792: finishing write tid 2 to smithi08575397-43 2021-02-10T04:31:02.432 INFO:tasks.rados.rados.0.smithi085.stdout:789: done (14 left) 2021-02-10T04:31:02.432 INFO:tasks.rados.rados.0.smithi085.stdout:791: done (13 left) 2021-02-10T04:31:02.432 INFO:tasks.rados.rados.0.smithi085.stdout:793: read oid 46 snap 74 2021-02-10T04:31:02.432 INFO:tasks.rados.rados.0.smithi085.stdout:793: expect deleted 2021-02-10T04:31:02.432 INFO:tasks.rados.rados.0.smithi085.stdout:794: read oid 36 snap -1 2021-02-10T04:31:02.432 INFO:tasks.rados.rados.0.smithi085.stdout:794: expect (ObjNum 306 snap 74 seq_num 306) 2021-02-10T04:31:02.432 INFO:tasks.rados.rados.0.smithi085.stdout:795: snap_remove snap 72 2021-02-10T04:31:02.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:02 smithi085 conmon[29319]: cluster 2021-02-10T04:31:00.747346+0000 osd.5 (osd 2021-02-10T04:31:02.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:02 smithi085 conmon[29319]: .5) 53 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:31:02.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:02 smithi085 conmon[29319]: cluster 2021-02-10T04:31:00.747533+0000 osd.5 (osd.5) 54 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:31:02.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:02 smithi085 conmon[29319]: cluster 2021-02-10T04:31:01.401558+0000 mon.a (mon.0) 973 2021-02-10T04:31:02.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:02 smithi085 conmon[29319]: : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2021-02-10T04:31:02.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:02 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:02.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:02 smithi085 conmon[29319]: T04:31:02. 2021-02-10T04:31:02.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:02 smithi085 conmon[29319]: 028147+0000 mgr.y (mgr.14140) 426 : 2021-02-10T04:31:02.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:02 smithi085 conmon[29319]: cluster [DBG] pgmap v517: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 158 active+clean; 181 MiB data, 552 MiB used, 715 GiB / 715 GiB avail; 22 MiB/s rd, 3.7 MiB/s wr, 37 op/s 2021-02-10T04:31:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:02 smithi085 conmon[35961]: cluster 2021-02-10T04:31:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:02 smithi085 conmon[35961]: 2021-02-10T04:31:00.747346 2021-02-10T04:31:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:02 smithi085 conmon[35961]: +0000 osd.5 (osd.5) 53 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:31:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:02 smithi085 conmon[35961]: cluster 2021-02-10T04:31:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:02 smithi085 conmon[35961]: 2021-02-10T04:31: 2021-02-10T04:31:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:02 smithi085 conmon[35961]: 00.747533+0000 osd.5 (osd.5) 54 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:31:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:02 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:02 smithi085 conmon[35961]: 10T04:31:01.401558+0000 mon.a 2021-02-10T04:31:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:02 smithi085 conmon[35961]: (mon.0) 973 : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2021-02-10T04:31:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:02 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:02 smithi085 conmon[35961]: T04:31:02.028147+0000 mgr.y (mgr.14140) 2021-02-10T04:31:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:02 smithi085 conmon[35961]: 426 : cluster [DBG] pgmap v517: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 158 active+clean; 181 MiB data, 552 MiB used, 715 GiB / 715 GiB avail; 22 MiB/s rd, 3.7 MiB/s wr, 37 op/s 2021-02-10T04:31:02.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:02 smithi164 conmon[29288]: cluster 2021-02-10T04:31:02.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:02 smithi164 conmon[29288]: 2021-02 2021-02-10T04:31:02.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:02 smithi164 conmon[29288]: -10T04:31:00.747346+0000 osd.5 (osd.5) 53 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:31:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:02 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:02 smithi164 conmon[29288]: -02-10T04:31:00.747533+0000 osd.5 (osd 2021-02-10T04:31:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:02 smithi164 conmon[29288]: .5) 54 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:31:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:02 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:02 smithi164 conmon[29288]: 02-10T04:31:01.401558+0000 mon.a ( 2021-02-10T04:31:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:02 smithi164 conmon[29288]: mon.0) 973 : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2021-02-10T04:31:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:02 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:02 smithi164 conmon[29288]: 02-10T04:31:02.028147+0000 mgr.y 2021-02-10T04:31:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:02 smithi164 conmon[29288]: (mgr.14140) 426 : cluster [DBG] pgmap v517: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 158 active+clean; 181 MiB data, 552 MiB used, 715 GiB / 715 GiB avail; 22 MiB/s rd, 3.7 MiB/s wr, 37 op/s 2021-02-10T04:31:03.078 INFO:tasks.rados.rados.0.smithi085.stdout:792: finishing write tid 3 to smithi08575397-43 2021-02-10T04:31:03.078 INFO:tasks.rados.rados.0.smithi085.stdout:792: finishing write tid 4 to smithi08575397-43 2021-02-10T04:31:03.078 INFO:tasks.rados.rados.0.smithi085.stdout:792: finishing write tid 5 to smithi08575397-43 2021-02-10T04:31:03.078 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 247 (ObjNum 312 snap 76 seq_num 312) dirty exists 2021-02-10T04:31:03.078 INFO:tasks.rados.rados.0.smithi085.stdout:792: left oid 43 (ObjNum 312 snap 76 seq_num 312) 2021-02-10T04:31:03.078 INFO:tasks.rados.rados.0.smithi085.stdout:794: expect (ObjNum 306 snap 74 seq_num 306) 2021-02-10T04:31:03.312 INFO:tasks.rados.rados.0.smithi085.stdout:792: done (15 left) 2021-02-10T04:31:03.312 INFO:tasks.rados.rados.0.smithi085.stdout:793: done (14 left) 2021-02-10T04:31:03.312 INFO:tasks.rados.rados.0.smithi085.stdout:794: done (13 left) 2021-02-10T04:31:03.312 INFO:tasks.rados.rados.0.smithi085.stdout:795: done (12 left) 2021-02-10T04:31:03.312 INFO:tasks.rados.rados.0.smithi085.stdout:796: read oid 28 snap 63 2021-02-10T04:31:03.312 INFO:tasks.rados.rados.0.smithi085.stdout:796: expect (ObjNum 213 snap 47 seq_num 213) 2021-02-10T04:31:03.313 INFO:tasks.rados.rados.0.smithi085.stdout:797: rollback oid 24 current snap is 76 2021-02-10T04:31:03.313 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 24 to 75 2021-02-10T04:31:03.313 INFO:tasks.rados.rados.0.smithi085.stdout:798: copy_from oid 21 from oid 43 current snap is 76 2021-02-10T04:31:03.313 INFO:tasks.rados.rados.0.smithi085.stdout:799: copy_from oid 16 from oid 22 current snap is 76 2021-02-10T04:31:03.313 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:03.318 INFO:tasks.rados.rados.0.smithi085.stdout:796: expect (ObjNum 213 snap 47 seq_num 213) 2021-02-10T04:31:03.601 INFO:tasks.rados.rados.0.smithi085.stdout:796: done (15 left) 2021-02-10T04:31:03.601 INFO:tasks.rados.rados.0.smithi085.stdout:800: write oid 3 current snap is 76 2021-02-10T04:31:03.601 INFO:tasks.rados.rados.0.smithi085.stdout:800: seq_num 313 ranges {460591=723410,1716158=403433,2729279=440140} 2021-02-10T04:31:03.608 INFO:tasks.rados.rados.0.smithi085.stdout:800: writing smithi08575397-3 from 460591 to 1184001 tid 1 2021-02-10T04:31:03.614 INFO:tasks.rados.rados.0.smithi085.stdout:800: writing smithi08575397-3 from 1716158 to 2119591 tid 2 2021-02-10T04:31:03.622 INFO:tasks.rados.rados.0.smithi085.stdout:800: writing smithi08575397-3 from 2729279 to 3169419 tid 3 2021-02-10T04:31:03.622 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:03.622 INFO:tasks.rados.rados.0.smithi085.stdout:797: finishing rollback tid 1 to smithi08575397-24 2021-02-10T04:31:03.622 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 24 v 84 (ObjNum 82251344 snap 22030 seq_num 80674832) dirty dne 2021-02-10T04:31:03.623 INFO:tasks.rados.rados.0.smithi085.stdout:799: finishing copy_from to smithi08575397-16 2021-02-10T04:31:03.623 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 90 (ObjNum 127 snap 23 seq_num 127) dirty exists 2021-02-10T04:31:03.623 INFO:tasks.rados.rados.0.smithi085.stdout:799: finishing copy_from racing read to smithi08575397-16 2021-02-10T04:31:03.623 INFO:tasks.rados.rados.0.smithi085.stdout:797: done (15 left) 2021-02-10T04:31:03.623 INFO:tasks.rados.rados.0.smithi085.stdout:799: done (14 left) 2021-02-10T04:31:03.623 INFO:tasks.rados.rados.0.smithi085.stdout:801: snap_remove snap 75 2021-02-10T04:31:03.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:03 smithi085 conmon[29319]: cluster 2021-02-10T04:31:03.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:03 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:31:03.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:03 smithi085 conmon[29319]: :31:02.407698+0000 mon.a (mon.0) 974 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2021-02-10T04:31:03.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:03 smithi085 conmon[29319]: cluster 2021-02-10T04:31:03.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:03 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:31:03.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:03 smithi085 conmon[29319]: 10T04:31:03.071434+0000 mon.a ( 2021-02-10T04:31:03.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:03 smithi085 conmon[29319]: mon.0) 975 : cluster [DBG] osdmap e247: 8 total, 8 up, 8 in 2021-02-10T04:31:03.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:03 smithi085 conmon[35961]: cluster 2021-02-10T04:31:02.407698+0000 mon.a (mon.0) 974 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2021-02-10T04:31:03.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:03 smithi085 conmon[35961]: cluster 2021-02-10T04:31:03.071434+0000 mon.a (mon.0) 975 : cluster 2021-02-10T04:31:03.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:03 smithi085 conmon[35961]: [DBG] osdmap e247: 8 total, 8 up, 8 in 2021-02-10T04:31:03.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:03 smithi164 conmon[29288]: cluster 2021-02-10T04:31:02.407698+0000 mon.a (mon.0) 2021-02-10T04:31:03.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:03 smithi164 conmon[29288]: 974 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2021-02-10T04:31:03.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:03 smithi164 conmon[29288]: cluster 2021-02-10T04:31:03.071434+0000 mon.a (mon.0) 975 : cluster [DBG] osdmap e247: 8 total, 8 up, 8 in 2021-02-10T04:31:04.415 INFO:tasks.rados.rados.0.smithi085.stdout:801: done (14 left) 2021-02-10T04:31:04.415 INFO:tasks.rados.rados.0.smithi085.stdout:802: read oid 37 snap -1 2021-02-10T04:31:04.416 INFO:tasks.rados.rados.0.smithi085.stdout:802: expect (ObjNum 301 snap 74 seq_num 301) 2021-02-10T04:31:04.416 INFO:tasks.rados.rados.0.smithi085.stdout:803: read oid 23 snap -1 2021-02-10T04:31:04.416 INFO:tasks.rados.rados.0.smithi085.stdout:803: expect (ObjNum 124 snap 23 seq_num 124) 2021-02-10T04:31:04.416 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:04.416 INFO:tasks.rados.rados.0.smithi085.stdout:800: finishing write tid 1 to smithi08575397-3 2021-02-10T04:31:04.416 INFO:tasks.rados.rados.0.smithi085.stdout:800: finishing write tid 2 to smithi08575397-3 2021-02-10T04:31:04.416 INFO:tasks.rados.rados.0.smithi085.stdout:800: finishing write tid 3 to smithi08575397-3 2021-02-10T04:31:04.416 INFO:tasks.rados.rados.0.smithi085.stdout:800: finishing write tid 4 to smithi08575397-3 2021-02-10T04:31:04.416 INFO:tasks.rados.rados.0.smithi085.stdout:800: finishing write tid 5 to smithi08575397-3 2021-02-10T04:31:04.416 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 251 (ObjNum 313 snap 76 seq_num 313) dirty exists 2021-02-10T04:31:04.416 INFO:tasks.rados.rados.0.smithi085.stdout:800: left oid 3 (ObjNum 313 snap 76 seq_num 313) 2021-02-10T04:31:04.416 INFO:tasks.rados.rados.0.smithi085.stdout:798: finishing copy_from to smithi08575397-21 2021-02-10T04:31:04.416 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 247 (ObjNum 312 snap 76 seq_num 312) dirty exists 2021-02-10T04:31:04.417 INFO:tasks.rados.rados.0.smithi085.stdout:798: finishing copy_from racing read to smithi08575397-21 2021-02-10T04:31:04.417 INFO:tasks.rados.rados.0.smithi085.stdout:798: done (15 left) 2021-02-10T04:31:04.417 INFO:tasks.rados.rados.0.smithi085.stdout:800: done (14 left) 2021-02-10T04:31:04.417 INFO:tasks.rados.rados.0.smithi085.stdout:804: snap_create 2021-02-10T04:31:04.425 INFO:tasks.rados.rados.0.smithi085.stdout:803: expect (ObjNum 124 snap 23 seq_num 124) 2021-02-10T04:31:04.525 INFO:tasks.rados.rados.0.smithi085.stdout:802: expect (ObjNum 301 snap 74 seq_num 301) 2021-02-10T04:31:04.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:04 smithi085 conmon[29319]: cluster 2021-02-10T04:31:04.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:04 smithi085 conmon[29319]: 2021-02-10T04:31:04.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:04 smithi085 conmon[29319]: 2021 2021-02-10T04:31:04.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:04 smithi085 conmon[29319]: - 2021-02-10T04:31:04.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:04 smithi085 conmon[29319]: 02 2021-02-10T04:31:04.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:04 smithi085 conmon[29319]: - 2021-02-10T04:31:04.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:04 smithi085 conmon[29319]: 10 2021-02-10T04:31:04.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:04 smithi085 conmon[29319]: T 2021-02-10T04:31:04.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:04 smithi085 conmon[29319]: 04 2021-02-10T04:31:04.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:04 smithi085 conmon[29319]: : 2021-02-10T04:31:04.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:04 smithi085 conmon[29319]: 31 2021-02-10T04:31:04.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:04 smithi085 conmon[29319]: : 2021-02-10T04:31:04.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:04 smithi085 conmon[29319]: 04 2021-02-10T04:31:04.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:04 smithi085 conmon[29319]: . 2021-02-10T04:31:04.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:04 smithi085 conmon[29319]: 028748+0000 mgr.y (mgr.14140) 427 2021-02-10T04:31:04.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:04 smithi085 conmon[29319]: : cluster [DBG] pgmap v520: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 158 active+clean; 184 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 20 MiB/s rd, 2.5 MiB/s wr, 27 op/s 2021-02-10T04:31:04.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:04 smithi085 conmon[35961]: cluster 2021-02-10T04:31:04.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:04 smithi085 conmon[35961]: 2021-02-10T04:31:04.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:04 smithi085 conmon[35961]: 2021 2021-02-10T04:31:04.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:04 smithi085 conmon[35961]: - 2021-02-10T04:31:04.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:04 smithi085 conmon[35961]: 02 2021-02-10T04:31:04.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:04 smithi085 conmon[35961]: - 2021-02-10T04:31:04.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:04 smithi085 conmon[35961]: 10T04:31:04.028748+0000 2021-02-10T04:31:04.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:04 smithi085 conmon[35961]: mgr.y (mgr.14140) 427 : cluster [DBG] pgmap v520: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 158 active+clean; 184 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 20 MiB/s rd, 2.5 MiB/s wr, 27 op/s 2021-02-10T04:31:04.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:04 smithi164 conmon[29288]: cluster 2021-02-10T04:31:04.028748+0000 mgr.y (mgr.14140) 427 : cluster 2021-02-10T04:31:04.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:04 smithi164 conmon[29288]: [DBG] pgmap v520: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 158 active+clean; 184 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 20 MiB/s rd, 2.5 MiB/s wr, 27 op/s 2021-02-10T04:31:05.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:31:04 smithi085 conmon[52836]: debug 2021-02-10T04:31:04.880+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:05.270 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:31:05.417 INFO:tasks.rados.rados.0.smithi085.stdout:802: done (14 left) 2021-02-10T04:31:05.417 INFO:tasks.rados.rados.0.smithi085.stdout:803: done (13 left) 2021-02-10T04:31:05.418 INFO:tasks.rados.rados.0.smithi085.stdout:804: done (12 left) 2021-02-10T04:31:05.418 INFO:tasks.rados.rados.0.smithi085.stdout:805: read oid 8 snap -1 2021-02-10T04:31:05.418 INFO:tasks.rados.rados.0.smithi085.stdout:805: expect deleted 2021-02-10T04:31:05.418 INFO:tasks.rados.rados.0.smithi085.stdout:806: snap_remove snap 70 2021-02-10T04:31:05.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:05 smithi085 conmon[35961]: cluster 2021-02-10T04:31:04.415779+0000 2021-02-10T04:31:05.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:05 smithi085 conmon[35961]: mon.a (mon.0) 976 : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2021-02-10T04:31:05.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:05 smithi085 conmon[29319]: cluster 2021-02-10T04:31:04.415779+0000 2021-02-10T04:31:05.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:05 smithi085 conmon[29319]: mon.a (mon.0) 976 : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2021-02-10T04:31:05.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:05 smithi164 conmon[29288]: cluster 2021-02-10T04:31:04.415779+0000 mon.a (mon 2021-02-10T04:31:05.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:05 smithi164 conmon[29288]: .0) 976 : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2021-02-10T04:31:06.420 INFO:tasks.rados.rados.0.smithi085.stdout:806: done (13 left) 2021-02-10T04:31:06.420 INFO:tasks.rados.rados.0.smithi085.stdout:807: read oid 24 snap -1 2021-02-10T04:31:06.420 INFO:tasks.rados.rados.0.smithi085.stdout:807: expect deleted 2021-02-10T04:31:06.420 INFO:tasks.rados.rados.0.smithi085.stdout:808: snap_create 2021-02-10T04:31:06.657 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:31:06.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:06 smithi085 conmon[29319]: cluster 2021-02-10T04:31:06.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:06 smithi085 conmon[29319]: 2021-02-10T04:31: 2021-02-10T04:31:06.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:06 smithi085 conmon[29319]: 05.416584+0000 mon.a (mon.0) 977 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2021-02-10T04:31:06.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:06 smithi085 conmon[29319]: cluster 2021-02-10T04:31:05. 2021-02-10T04:31:06.687 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:06 smithi085 conmon[29319]: 612083+0000 osd.2 (osd.2) 60 : cluster [DBG] 2021-02-10T04:31:06.687 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:06 smithi085 conmon[29319]: purged_snaps scrub starts 2021-02-10T04:31:06.687 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:06 smithi085 conmon[29319]: cluster 2021-02-10T04:31:05. 2021-02-10T04:31:06.687 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:06 smithi085 conmon[29319]: 612284+0000 osd.2 (osd.2) 61 : cluster [DBG] 2021-02-10T04:31:06.687 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:06 smithi085 conmon[29319]: purged_snaps scrub ok 2021-02-10T04:31:06.687 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:06 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:31:06.687 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:06 smithi085 conmon[29319]: 04:31:06.029288+0000 mgr.y (mgr. 2021-02-10T04:31:06.687 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:06 smithi085 conmon[29319]: 14140) 428 : cluster [DBG] pgmap v523: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 185 MiB data, 560 MiB used, 715 GiB / 715 GiB avail; 15 MiB/s rd, 2.7 MiB/s wr, 19 op/s 2021-02-10T04:31:06.688 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:06 smithi085 conmon[35961]: cluster 2021-02-10T04:31:06.688 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:06 smithi085 conmon[35961]: 2021-02-10T04:31:06.688 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:06 smithi085 conmon[35961]: 2021 2021-02-10T04:31:06.688 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:06 smithi085 conmon[35961]: - 2021-02-10T04:31:06.688 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:06 smithi085 conmon[35961]: 02 2021-02-10T04:31:06.688 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:06 smithi085 conmon[35961]: -10T04:31:05.416584 2021-02-10T04:31:06.688 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:06 smithi085 conmon[35961]: +0000 mon.a (mon.0) 977 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2021-02-10T04:31:06.688 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:06 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:06.688 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:06 smithi085 conmon[35961]: 31:05.612083+0000 osd.2 (osd.2) 60 : cluster 2021-02-10T04:31:06.688 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:06 smithi085 conmon[35961]: [DBG] purged_snaps scrub starts 2021-02-10T04:31:06.688 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:06 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:06.689 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:06 smithi085 conmon[35961]: T04:31:05.612284+0000 osd.2 (osd. 2021-02-10T04:31:06.689 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:06 smithi085 conmon[35961]: 2) 61 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:31:06.689 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:06 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:06.689 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:06 smithi085 conmon[35961]: T04:31:06.029288+0000 mgr.y (mgr.14140 2021-02-10T04:31:06.689 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:06 smithi085 conmon[35961]: ) 428 : cluster [DBG] pgmap v523: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 185 MiB data, 560 MiB used, 715 GiB / 715 GiB avail; 15 MiB/s rd, 2.7 MiB/s wr, 19 op/s 2021-02-10T04:31:06.689 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:31:06 smithi085 conmon[41540]: debug 2021-02-10T04:31:06.416+0000 7fb94c364700 -1 osd.0 pg_epoch: 250 pg[7.4( v 247'98 (0'0,247'98] local-lis/les=93/94 n=5 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [0,4,1] r=0 lpr=93 crt=247'98 lcod 247'96 mlcod 247'96 active+clean+snaptrim trimq=[8b~1] ps=[60~3,66~3,6b~3,6f~1,74~2,78~4,80~4,87~2,8c~2,8f~1,92~1]] removing snap head 2021-02-10T04:31:06.694 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:06.807 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.0 dump_blocked_ops 2021-02-10T04:31:06.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:06 smithi164 conmon[29288]: cluster 2021-02-10T04:31:05.416584+0000 mon.a 2021-02-10T04:31:06.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:06 smithi164 conmon[29288]: (mon.0) 977 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2021-02-10T04:31:06.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:06 smithi164 conmon[29288]: cluster 2021-02-10T04:31:05.612083+0000 osd.2 (osd.2) 60 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:31:06.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:06 smithi164 conmon[29288]: 2021-02-10T04:31:06.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:06 smithi164 conmon[29288]: cluster 2021-02-10T04:31:05.612284+0000 2021-02-10T04:31:06.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:06 smithi164 conmon[29288]: osd.2 (osd.2) 61 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:31:06.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:06 smithi164 conmon[29288]: cluster 2021-02-10T04:31:06.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:06 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:31:06.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:06 smithi164 conmon[29288]: 04:31:06.029288+0000 mgr.y ( 2021-02-10T04:31:06.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:06 smithi164 conmon[29288]: mgr.14140) 428 : cluster [DBG] pgmap v523: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 185 MiB data, 560 MiB used, 715 GiB / 715 GiB avail; 15 MiB/s rd, 2.7 MiB/s wr, 19 op/s 2021-02-10T04:31:06.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:06 smithi164 conmon[29288]: 2021-02-10T04:31:06.938 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:31:06 smithi085 conmon[52836]: debug 2021-02-10T04:31:06.923+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:06.987 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:31:07.092 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:31:07.105 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:31:07.122 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:07.245 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:31:07.349 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:31:07.385 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:07.393 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:31:07 smithi164 conmon[37263]: debug 2021-02-10T04:31:07.204+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:07.426 INFO:tasks.rados.rados.0.smithi085.stdout:805: done (14 left) 2021-02-10T04:31:07.426 INFO:tasks.rados.rados.0.smithi085.stdout:807: done (13 left) 2021-02-10T04:31:07.426 INFO:tasks.rados.rados.0.smithi085.stdout:808: done (12 left) 2021-02-10T04:31:07.426 INFO:tasks.rados.rados.0.smithi085.stdout:809: rollback oid 26 current snap is 78 2021-02-10T04:31:07.426 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 26 to 76 2021-02-10T04:31:07.426 INFO:tasks.rados.rados.0.smithi085.stdout:810: rollback oid 21 current snap is 78 2021-02-10T04:31:07.427 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 21 to 63 2021-02-10T04:31:07.427 INFO:tasks.rados.rados.0.smithi085.stdout:811: read oid 23 snap 77 2021-02-10T04:31:07.427 INFO:tasks.rados.rados.0.smithi085.stdout:811: expect (ObjNum 124 snap 23 seq_num 124) 2021-02-10T04:31:07.427 INFO:tasks.rados.rados.0.smithi085.stdout:812: rollback oid 3 current snap is 78 2021-02-10T04:31:07.427 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 3 to 74 2021-02-10T04:31:07.427 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:07.435 INFO:tasks.rados.rados.0.smithi085.stdout:811: expect (ObjNum 124 snap 23 seq_num 124) 2021-02-10T04:31:07.527 INFO:tasks.rados.rados.0.smithi085.stdout:809: finishing rollback tid 0 to smithi08575397-26 2021-02-10T04:31:07.527 INFO:tasks.rados.rados.0.smithi085.stdout:810: finishing rollback tid 0 to smithi08575397-21 2021-02-10T04:31:07.527 INFO:tasks.rados.rados.0.smithi085.stdout:811: done (15 left) 2021-02-10T04:31:07.527 INFO:tasks.rados.rados.0.smithi085.stdout:813: read oid 24 snap 63 2021-02-10T04:31:07.527 INFO:tasks.rados.rados.0.smithi085.stdout:813: expect deleted 2021-02-10T04:31:07.527 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:07.527 INFO:tasks.rados.rados.0.smithi085.stdout:810: finishing rollback tid 1 to smithi08575397-21 2021-02-10T04:31:07.528 INFO:tasks.rados.rados.0.smithi085.stdout:810: finishing rollback tid 2 to smithi08575397-21 2021-02-10T04:31:07.528 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 391 (ObjNum 194 snap 43 seq_num 194) dirty exists 2021-02-10T04:31:07.528 INFO:tasks.rados.rados.0.smithi085.stdout:809: finishing rollback tid 1 to smithi08575397-26 2021-02-10T04:31:07.528 INFO:tasks.rados.rados.0.smithi085.stdout:809: finishing rollback tid 2 to smithi08575397-26 2021-02-10T04:31:07.528 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 64 (ObjNum 288 snap 71 seq_num 288) dirty exists 2021-02-10T04:31:07.528 INFO:tasks.rados.rados.0.smithi085.stdout:812: finishing rollback tid 0 to smithi08575397-3 2021-02-10T04:31:07.528 INFO:tasks.rados.rados.0.smithi085.stdout:812: finishing rollback tid 1 to smithi08575397-3 2021-02-10T04:31:07.528 INFO:tasks.rados.rados.0.smithi085.stdout:809: done (15 left) 2021-02-10T04:31:07.528 INFO:tasks.rados.rados.0.smithi085.stdout:810: done (14 left) 2021-02-10T04:31:07.528 INFO:tasks.rados.rados.0.smithi085.stdout:814: snap_remove snap 76 2021-02-10T04:31:07.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:07 smithi085 conmon[29319]: cluster 2021-02-10T04:31:06.418690+0000 mon.a (mon.0) 978 : cluster [DBG] osdmap e250: 8 total, 8 up, 8 in 2021-02-10T04:31:07.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:07 smithi085 conmon[29319]: cluster 2021-02-10T04:31:07.420347+0000 mon.a (mon.0) 979 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2021-02-10T04:31:07.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:07 smithi085 conmon[35961]: cluster 2021-02-10T04:31:06.418690+0000 mon.a (mon.0) 978 : cluster [DBG] osdmap e250: 8 total, 8 up, 8 in 2021-02-10T04:31:07.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:07 smithi085 conmon[35961]: cluster 2021-02-10T04:31:07.420347+0000 mon.a (mon.0) 979 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2021-02-10T04:31:07.825 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:07 smithi164 conmon[29288]: cluster 2021-02-10T04:31:06.418690+0000 mon.a (mon. 2021-02-10T04:31:07.825 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:07 smithi164 conmon[29288]: 0) 978 : cluster [DBG] osdmap e250: 8 total, 8 up, 8 in 2021-02-10T04:31:07.825 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:07 smithi164 conmon[29288]: cluster 2021-02-10T04:31:07.420347+0000 mon.a (mon.0) 979 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2021-02-10T04:31:08.076 INFO:tasks.rados.rados.0.smithi085.stdout:814: done (14 left) 2021-02-10T04:31:08.076 INFO:tasks.rados.rados.0.smithi085.stdout:815: snap_create 2021-02-10T04:31:08.076 INFO:tasks.rados.rados.0.smithi085.stdout:812: finishing rollback tid 2 to smithi08575397-3 2021-02-10T04:31:08.076 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 254 (ObjNum 293 snap 73 seq_num 293) dirty exists 2021-02-10T04:31:08.081 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.6 to 1 (10000) 2021-02-10T04:31:08.143 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:31:07 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:31:08.164 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:31:08.191 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:07 smithi085 conmon[45443]: debug 2021-02-10T04:31:07.877+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:08.742 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set nodeep-scrub 2021-02-10T04:31:08.770 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:08 smithi085 conmon[29319]: audit 2021-02-10T04:31:07.692574+0000 mon.b (mon.2) 2021-02-10T04:31:08.770 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:08 smithi085 conmon[29319]: 31 : audit [INF] from='client.? 172.21.15.85:0/4124935377' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T04:31:08.770 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:08 smithi085 conmon[29319]: audit 2021-02-10T04:31:07.693036+0000 mon.a (mon.0) 980 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T04:31:08.770 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:08 smithi085 conmon[29319]: audit 2021-02-10T04:31:07.825072+0000 mgr.y (mgr.14140) 429 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:31:08.770 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:08 smithi085 conmon[29319]: cluster 2021-02-10T04:31:08.029704+0000 mgr.y (mgr.14140) 430 : cluster [DBG] pgmap v526: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 185 MiB data, 560 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 1.7 MiB/s wr, 13 op/s 2021-02-10T04:31:08.770 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:08 smithi085 conmon[29319]: audit 2021-02-10T04:31:08.074832+0000 mon.a (mon.0) 981 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2021-02-10T04:31:08.770 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:08 smithi085 conmon[29319]: cluster 2021-02-10T04:31:08.074906+0000 mon.a ( 2021-02-10T04:31:08.770 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:08 smithi085 conmon[29319]: mon.0) 982 : cluster [DBG] osdmap e252: 8 total, 8 up, 8 in 2021-02-10T04:31:08.771 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:08 smithi085 conmon[35961]: audit 2021-02-10T04:31:07.692574 2021-02-10T04:31:08.771 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:08 smithi085 conmon[35961]: +0000 mon.b (mon.2) 31 : audit [INF] from='client.? 172.21.15.85:0/4124935377' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T04:31:08.771 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:08 smithi085 conmon[35961]: audit 2021-02-10T04:31 2021-02-10T04:31:08.771 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:08 smithi085 conmon[35961]: :07.693036+0000 mon.a (mon.0) 980 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T04:31:08.771 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:08 smithi085 conmon[35961]: audit 2021-02-10T04:31:07.825072 2021-02-10T04:31:08.771 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:08 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 429 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:31:08.771 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:08 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:08.771 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:08 smithi085 conmon[35961]: 02-10T04:31:08.029704+0000 mgr.y (mgr.14140) 2021-02-10T04:31:08.771 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:08 smithi085 conmon[35961]: 430 : cluster [DBG] pgmap v526: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 185 MiB data, 560 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 1.7 MiB/s wr, 13 op/s 2021-02-10T04:31:08.772 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:08 smithi085 conmon[35961]: audit 2021-02-10T04:31:08.074832 2021-02-10T04:31:08.772 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:08 smithi085 conmon[35961]: +0000 mon.a (mon.0 2021-02-10T04:31:08.772 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:08 smithi085 conmon[35961]: ) 981 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2021-02-10T04:31:08.772 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:08 smithi085 conmon[35961]: cluster 2021-02-10T04:31: 2021-02-10T04:31:08.772 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:08 smithi085 conmon[35961]: 08.074906+0000 mon.a (mon.0) 982 : cluster [DBG] 2021-02-10T04:31:08.772 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:08 smithi085 conmon[35961]: osdmap e252: 8 total, 8 up, 8 in 2021-02-10T04:31:08.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:08 smithi164 conmon[29288]: audit 2021-02-10T04:31:07.692574+0000 mon.b 2021-02-10T04:31:09.508 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:08 smithi164 conmon[29288]: (mon.2) 31 : audit [INF] from='client.? 172.21.15.85:0/4124935377' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T04:31:09.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:08 smithi164 conmon[29288]: audit 2021-02-10T04:31: 2021-02-10T04:31:09.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:08 smithi164 conmon[29288]: 07.693036+0000 mon.a (mon.0) 980 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T04:31:09.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:08 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:31:09.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:08 smithi164 conmon[29288]: -10T04:31:07.825072+0000 mgr.y (mgr. 2021-02-10T04:31:09.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:08 smithi164 conmon[29288]: 14140) 429 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:31:09.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:08 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:09.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:08 smithi164 conmon[29288]: -10T04:31:08.029704+0000 mgr.y ( 2021-02-10T04:31:09.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:08 smithi164 conmon[29288]: mgr.14140) 430 : cluster [DBG] pgmap v526: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 185 MiB data, 560 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 1.7 MiB/s wr, 13 op/s 2021-02-10T04:31:09.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:08 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:31:09.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:08 smithi164 conmon[29288]: -10T04:31:08.074832+0000 mon.a (mon. 2021-02-10T04:31:09.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:08 smithi164 conmon[29288]: 0) 981 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2021-02-10T04:31:09.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:08 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:09.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:08 smithi164 conmon[29288]: 02-10T04:31:08.074906+0000 mon.a (mon.0) 982 : cluster [DBG] 2021-02-10T04:31:09.510 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:08 smithi164 conmon[29288]: osdmap e252: 8 total, 8 up, 8 in 2021-02-10T04:31:09.511 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:31:09.511 INFO:tasks.rados.rados.0.smithi085.stdout:812: done (14 left) 2021-02-10T04:31:09.511 INFO:tasks.rados.rados.0.smithi085.stdout:813: done (13 left) 2021-02-10T04:31:09.511 INFO:tasks.rados.rados.0.smithi085.stdout:815: done (12 left) 2021-02-10T04:31:09.511 INFO:tasks.rados.rados.0.smithi085.stdout:816: write oid 6 current snap is 79 2021-02-10T04:31:09.511 INFO:tasks.rados.rados.0.smithi085.stdout:816: seq_num 314 ranges {477398=798659,1774640=663444,2873055=404343} 2021-02-10T04:31:09.512 INFO:tasks.rados.rados.0.smithi085.stdout:816: writing smithi08575397-6 from 477398 to 1276057 tid 1 2021-02-10T04:31:09.512 INFO:tasks.rados.rados.0.smithi085.stdout:816: writing smithi08575397-6 from 1774640 to 2438084 tid 2 2021-02-10T04:31:09.512 INFO:tasks.rados.rados.0.smithi085.stdout:816: writing smithi08575397-6 from 2873055 to 3277398 tid 3 2021-02-10T04:31:09.512 INFO:tasks.rados.rados.0.smithi085.stdout:817: snap_create 2021-02-10T04:31:09.512 INFO:tasks.rados.rados.0.smithi085.stdout:816: finishing write tid 1 to smithi08575397-6 2021-02-10T04:31:09.512 INFO:tasks.rados.rados.0.smithi085.stdout:816: finishing write tid 2 to smithi08575397-6 2021-02-10T04:31:09.512 INFO:tasks.rados.rados.0.smithi085.stdout:816: finishing write tid 3 to smithi08575397-6 2021-02-10T04:31:09.512 INFO:tasks.rados.rados.0.smithi085.stdout:816: finishing write tid 4 to smithi08575397-6 2021-02-10T04:31:09.512 INFO:tasks.rados.rados.0.smithi085.stdout:816: finishing write tid 5 to smithi08575397-6 2021-02-10T04:31:09.512 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 255 (ObjNum 314 snap 79 seq_num 314) dirty exists 2021-02-10T04:31:09.512 INFO:tasks.rados.rados.0.smithi085.stdout:816: left oid 6 (ObjNum 314 snap 79 seq_num 314) 2021-02-10T04:31:10.085 INFO:tasks.rados.rados.0.smithi085.stdout:816: done (13 left) 2021-02-10T04:31:10.085 INFO:tasks.rados.rados.0.smithi085.stdout:817: done (12 left) 2021-02-10T04:31:10.085 INFO:tasks.rados.rados.0.smithi085.stdout:818: rollback oid 3 current snap is 80 2021-02-10T04:31:10.085 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 3 to 78 2021-02-10T04:31:10.085 INFO:tasks.rados.rados.0.smithi085.stdout:819: write oid 5 current snap is 80 2021-02-10T04:31:10.085 INFO:tasks.rados.rados.0.smithi085.stdout:819: seq_num 315 ranges {494205=473908,1433122=407102,2384125=462684} 2021-02-10T04:31:10.086 INFO:tasks.rados.rados.0.smithi085.stdout:819: writing smithi08575397-5 from 494205 to 968113 tid 1 2021-02-10T04:31:10.090 INFO:tasks.rados.rados.0.smithi085.stdout:819: writing smithi08575397-5 from 1433122 to 1840224 tid 2 2021-02-10T04:31:10.097 INFO:tasks.rados.rados.0.smithi085.stdout:819: writing smithi08575397-5 from 2384125 to 2846809 tid 3 2021-02-10T04:31:10.097 INFO:tasks.rados.rados.0.smithi085.stdout:819: finishing write tid 1 to smithi08575397-5 2021-02-10T04:31:10.097 INFO:tasks.rados.rados.0.smithi085.stdout:818: finishing rollback tid 0 to smithi08575397-3 2021-02-10T04:31:10.097 INFO:tasks.rados.rados.0.smithi085.stdout:818: finishing rollback tid 1 to smithi08575397-3 2021-02-10T04:31:10.097 INFO:tasks.rados.rados.0.smithi085.stdout:818: finishing rollback tid 2 to smithi08575397-3 2021-02-10T04:31:10.097 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 257 (ObjNum 293 snap 73 seq_num 293) dirty exists 2021-02-10T04:31:10.098 INFO:tasks.rados.rados.0.smithi085.stdout:819: finishing write tid 2 to smithi08575397-5 2021-02-10T04:31:10.098 INFO:tasks.rados.rados.0.smithi085.stdout:818: done (13 left) 2021-02-10T04:31:10.098 INFO:tasks.rados.rados.0.smithi085.stdout:820: write (excl) oid 37 current snap is 80 2021-02-10T04:31:10.098 INFO:tasks.rados.rados.0.smithi085.stdout:820: seq_num 316 ranges {511012=549157,1775251=550760,3095195=215817} 2021-02-10T04:31:10.103 INFO:tasks.rados.rados.0.smithi085.stdout:820: writing smithi08575397-37 from 511012 to 1060169 tid 1 2021-02-10T04:31:10.111 INFO:tasks.rados.rados.0.smithi085.stdout:820: writing smithi08575397-37 from 1775251 to 2326011 tid 2 2021-02-10T04:31:10.117 INFO:tasks.rados.rados.0.smithi085.stdout:820: writing smithi08575397-37 from 3095195 to 3311012 tid 3 2021-02-10T04:31:10.117 INFO:tasks.rados.rados.0.smithi085.stdout:821: rollback oid 16 current snap is 80 2021-02-10T04:31:10.117 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 16 to 78 2021-02-10T04:31:10.118 INFO:tasks.rados.rados.0.smithi085.stdout:819: finishing write tid 3 to smithi08575397-5 2021-02-10T04:31:10.118 INFO:tasks.rados.rados.0.smithi085.stdout:819: finishing write tid 4 to smithi08575397-5 2021-02-10T04:31:10.118 INFO:tasks.rados.rados.0.smithi085.stdout:819: finishing write tid 5 to smithi08575397-5 2021-02-10T04:31:10.118 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 374 (ObjNum 315 snap 80 seq_num 315) dirty exists 2021-02-10T04:31:10.118 INFO:tasks.rados.rados.0.smithi085.stdout:819: left oid 5 (ObjNum 315 snap 80 seq_num 315) 2021-02-10T04:31:10.118 INFO:tasks.rados.rados.0.smithi085.stdout:820: finishing write tid 1 to smithi08575397-37 2021-02-10T04:31:10.118 INFO:tasks.rados.rados.0.smithi085.stdout:820: finishing write tid 2 to smithi08575397-37 2021-02-10T04:31:10.118 INFO:tasks.rados.rados.0.smithi085.stdout:819: done (14 left) 2021-02-10T04:31:10.118 INFO:tasks.rados.rados.0.smithi085.stdout:822: read oid 23 snap -1 2021-02-10T04:31:10.118 INFO:tasks.rados.rados.0.smithi085.stdout:822: expect (ObjNum 124 snap 23 seq_num 124) 2021-02-10T04:31:10.118 INFO:tasks.rados.rados.0.smithi085.stdout:823: read oid 40 snap 74 2021-02-10T04:31:10.118 INFO:tasks.rados.rados.0.smithi085.stdout:823: expect (ObjNum 280 snap 70 seq_num 280) 2021-02-10T04:31:10.119 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:10.119 INFO:tasks.rados.rados.0.smithi085.stdout:822: expect (ObjNum 124 snap 23 seq_num 124) 2021-02-10T04:31:10.211 INFO:tasks.rados.rados.0.smithi085.stdout:822: done (15 left) 2021-02-10T04:31:10.212 INFO:tasks.rados.rados.0.smithi085.stdout:824: delete oid 3 current snap is 80 2021-02-10T04:31:10.212 INFO:tasks.rados.rados.0.smithi085.stdout:820: finishing write tid 3 to smithi08575397-37 2021-02-10T04:31:10.212 INFO:tasks.rados.rados.0.smithi085.stdout:820: finishing write tid 4 to smithi08575397-37 2021-02-10T04:31:10.212 INFO:tasks.rados.rados.0.smithi085.stdout:820: finishing write tid 5 to smithi08575397-37 2021-02-10T04:31:10.212 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 400 (ObjNum 316 snap 80 seq_num 316) dirty exists 2021-02-10T04:31:10.212 INFO:tasks.rados.rados.0.smithi085.stdout:820: left oid 37 (ObjNum 316 snap 80 seq_num 316) 2021-02-10T04:31:10.212 INFO:tasks.rados.rados.0.smithi085.stdout:821: finishing rollback tid 0 to smithi08575397-16 2021-02-10T04:31:10.212 INFO:tasks.rados.rados.0.smithi085.stdout:821: finishing rollback tid 1 to smithi08575397-16 2021-02-10T04:31:10.212 INFO:tasks.rados.rados.0.smithi085.stdout:821: finishing rollback tid 2 to smithi08575397-16 2021-02-10T04:31:10.212 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 404 (ObjNum 127 snap 23 seq_num 127) dirty exists 2021-02-10T04:31:10.213 INFO:tasks.rados.rados.0.smithi085.stdout:823: expect (ObjNum 280 snap 70 seq_num 280) 2021-02-10T04:31:10.366 INFO:tasks.rados.rados.0.smithi085.stdout:820: done (15 left) 2021-02-10T04:31:10.366 INFO:tasks.rados.rados.0.smithi085.stdout:821: done (14 left) 2021-02-10T04:31:10.366 INFO:tasks.rados.rados.0.smithi085.stdout:823: done (13 left) 2021-02-10T04:31:10.366 INFO:tasks.rados.rados.0.smithi085.stdout:824: done (12 left) 2021-02-10T04:31:10.366 INFO:tasks.rados.rados.0.smithi085.stdout:825: read oid 38 snap -1 2021-02-10T04:31:10.367 INFO:tasks.rados.rados.0.smithi085.stdout:825: expect (ObjNum 287 snap 71 seq_num 287) 2021-02-10T04:31:10.367 INFO:tasks.rados.rados.0.smithi085.stdout:826: snap_remove snap 77 2021-02-10T04:31:10.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:10 smithi164 conmon[29288]: cluster 2021-02-10T04:31:09.072811+0000 mon.a (mon.0 2021-02-10T04:31:10.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:10 smithi164 conmon[29288]: ) 983 : cluster [DBG] osdmap e253: 8 total, 8 up, 8 in 2021-02-10T04:31:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[29319]: cluster 2021-02-10T04:31:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[29319]: 2021-02-10T04:31:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[29319]: 2021 2021-02-10T04:31:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[29319]: - 2021-02-10T04:31:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[29319]: 02 2021-02-10T04:31:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[29319]: - 2021-02-10T04:31:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[29319]: 10 2021-02-10T04:31:10.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[29319]: T04:31:09.072811+0000 mon.a 2021-02-10T04:31:10.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[29319]: (mon.0) 983 : cluster [DBG] osdmap e253: 8 total, 8 up, 8 in 2021-02-10T04:31:10.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: cluster 2021-02-10T04:31:10.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: 2021-02-10T04:31:10.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: 2021 2021-02-10T04:31:10.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: - 2021-02-10T04:31:10.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: 02 2021-02-10T04:31:10.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: - 2021-02-10T04:31:10.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: 10 2021-02-10T04:31:10.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: T 2021-02-10T04:31:10.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: 04 2021-02-10T04:31:10.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: : 2021-02-10T04:31:10.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: 31 2021-02-10T04:31:10.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: : 2021-02-10T04:31:10.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: 09 2021-02-10T04:31:10.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: . 2021-02-10T04:31:10.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: 072811 2021-02-10T04:31:10.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: +0000 2021-02-10T04:31:10.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: 2021-02-10T04:31:10.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: mon.a 2021-02-10T04:31:10.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: ( 2021-02-10T04:31:10.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: mon 2021-02-10T04:31:10.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: . 2021-02-10T04:31:10.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: 0 2021-02-10T04:31:10.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: ) 2021-02-10T04:31:10.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: 983 2021-02-10T04:31:10.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: : 2021-02-10T04:31:10.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: cluster 2021-02-10T04:31:10.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: 2021-02-10T04:31:10.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: [DBG] 2021-02-10T04:31:10.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: 2021-02-10T04:31:10.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: osdmap e253: 8 total, 8 up, 8 in 2021-02-10T04:31:10.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:10 smithi085 conmon[35961]: 2021-02-10T04:31:11.092 INFO:tasks.rados.rados.0.smithi085.stdout:825: expect (ObjNum 287 snap 71 seq_num 287) 2021-02-10T04:31:11.303 INFO:tasks.rados.rados.0.smithi085.stdout:825: done (13 left) 2021-02-10T04:31:11.303 INFO:tasks.rados.rados.0.smithi085.stdout:826: done (12 left) 2021-02-10T04:31:11.303 INFO:tasks.rados.rados.0.smithi085.stdout:827: snap_create 2021-02-10T04:31:11.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:11 smithi164 conmon[29288]: cluster 2021-02-10T04:31:11.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:11 smithi164 conmon[29288]: 2021-02-10T04:31:10 2021-02-10T04:31:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:11 smithi164 conmon[29288]: .030194+0000 mgr.y (mgr.14140) 431 : cluster [DBG] pgmap v529: 168 pgs: 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 158 active+clean; 186 MiB data, 551 MiB used, 715 GiB / 715 GiB avail; 2.4 MiB/s rd, 792 KiB/s wr, 4 op/s 2021-02-10T04:31:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:11 smithi164 conmon[29288]: cluster 2021-02-10T04:31:10.078453+0000 mon.a (mon.0 2021-02-10T04:31:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:11 smithi164 conmon[29288]: ) 984 : cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2021-02-10T04:31:11.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:11 smithi085 conmon[29319]: cluster 2021-02-10T04:31:11.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:11 smithi085 conmon[29319]: 2021-02-10T04:31:11.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:11 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:31:11.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:11 smithi085 conmon[29319]: 31:10.030194+0000 mgr.y (mgr.14140) 431 : cluster [DBG] pgmap v529: 168 pgs: 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 158 active+clean; 186 MiB data, 551 MiB used, 715 GiB / 715 GiB avail; 2.4 MiB/s rd, 792 KiB/s wr, 4 op/s 2021-02-10T04:31:11.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:11 smithi085 conmon[29319]: cluster 2021-02-10T04:31:11.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:11 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:31:11.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:11 smithi085 conmon[29319]: 31:10.078453+0000 mon.a (mon.0) 984 : cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2021-02-10T04:31:11.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:11 smithi085 conmon[35961]: cluster 2021-02-10T04:31:10 2021-02-10T04:31:11.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:11 smithi085 conmon[35961]: .030194+0000 mgr.y (mgr.14140) 431 : cluster [DBG] pgmap v529: 168 pgs: 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 158 active+clean; 186 MiB data, 551 MiB used, 715 GiB / 715 GiB avail; 2.4 MiB/s rd, 792 KiB/s wr, 4 op/s 2021-02-10T04:31:11.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:11 smithi085 conmon[35961]: cluster 2021-02-10T04:31:10.078453 2021-02-10T04:31:11.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:11 smithi085 conmon[35961]: +0000 mon.a (mon.0) 984 : cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2021-02-10T04:31:11.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:11 smithi085 conmon[35961]: 2021-02-10T04:31:12.097 INFO:tasks.rados.rados.0.smithi085.stdout:827: done (12 left) 2021-02-10T04:31:12.097 INFO:tasks.rados.rados.0.smithi085.stdout:828: write (excl) oid 37 current snap is 81 2021-02-10T04:31:12.097 INFO:tasks.rados.rados.0.smithi085.stdout:828: seq_num 318 ranges {544626=699655,1892215=554429,3150041=194585} 2021-02-10T04:31:12.104 INFO:tasks.rados.rados.0.smithi085.stdout:828: writing smithi08575397-37 from 544626 to 1244281 tid 1 2021-02-10T04:31:12.111 INFO:tasks.rados.rados.0.smithi085.stdout:828: writing smithi08575397-37 from 1892215 to 2446644 tid 2 2021-02-10T04:31:12.117 INFO:tasks.rados.rados.0.smithi085.stdout:828: writing smithi08575397-37 from 3150041 to 3344626 tid 3 2021-02-10T04:31:12.117 INFO:tasks.rados.rados.0.smithi085.stdout:829: copy_from oid 28 from oid 50 current snap is 81 2021-02-10T04:31:12.117 INFO:tasks.rados.rados.0.smithi085.stdout:828: finishing write tid 1 to smithi08575397-37 2021-02-10T04:31:12.117 INFO:tasks.rados.rados.0.smithi085.stdout:828: finishing write tid 2 to smithi08575397-37 2021-02-10T04:31:12.117 INFO:tasks.rados.rados.0.smithi085.stdout:830: snap_remove snap 80 2021-02-10T04:31:12.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:12 smithi164 conmon[29288]: cluster 2021-02-10T04:31:11.091948+0000 mon.a (mon 2021-02-10T04:31:12.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:12 smithi164 conmon[29288]: .0) 985 : cluster [DBG] osdmap e255: 8 total, 8 up, 8 in 2021-02-10T04:31:12.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[29319]: cluster 2021-02-10T04:31:11.091948+0000 2021-02-10T04:31:12.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[29319]: mon.a (mon.0) 985 : cluster [DBG] osdmap e255: 8 total, 8 up, 8 in 2021-02-10T04:31:12.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: cluster 2021-02-10T04:31:12.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: 2021-02-10T04:31:12.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: 2021 2021-02-10T04:31:12.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: - 2021-02-10T04:31:12.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: 02 2021-02-10T04:31:12.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: - 2021-02-10T04:31:12.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: 10 2021-02-10T04:31:12.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: T 2021-02-10T04:31:12.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: 04 2021-02-10T04:31:12.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: : 2021-02-10T04:31:12.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: 31 2021-02-10T04:31:12.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: : 2021-02-10T04:31:12.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: 11 2021-02-10T04:31:12.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: . 2021-02-10T04:31:12.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: 091948 2021-02-10T04:31:12.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: +0000 2021-02-10T04:31:12.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: 2021-02-10T04:31:12.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: mon.a 2021-02-10T04:31:12.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: ( 2021-02-10T04:31:12.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: mon 2021-02-10T04:31:12.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: . 2021-02-10T04:31:12.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: 0 2021-02-10T04:31:12.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: ) 2021-02-10T04:31:12.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: 985 2021-02-10T04:31:12.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: : 2021-02-10T04:31:12.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: cluster 2021-02-10T04:31:12.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: 2021-02-10T04:31:12.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: [DBG] 2021-02-10T04:31:12.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: 2021-02-10T04:31:12.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: osdmap e255: 8 total, 8 up, 8 in 2021-02-10T04:31:12.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:12 smithi085 conmon[35961]: 2021-02-10T04:31:13.156 INFO:tasks.rados.rados.0.smithi085.stdout:829: finishing copy_from to smithi08575397-28 2021-02-10T04:31:13.156 INFO:tasks.rados.rados.0.smithi085.stdout:829: got expected ENOENT (src dne) 2021-02-10T04:31:13.156 INFO:tasks.rados.rados.0.smithi085.stdout:829: finishing copy_from racing read to smithi08575397-28 2021-02-10T04:31:13.157 INFO:tasks.rados.rados.0.smithi085.stdout:828: finishing write tid 3 to smithi08575397-37 2021-02-10T04:31:13.157 INFO:tasks.rados.rados.0.smithi085.stdout:828: finishing write tid 4 to smithi08575397-37 2021-02-10T04:31:13.157 INFO:tasks.rados.rados.0.smithi085.stdout:828: finishing write tid 5 to smithi08575397-37 2021-02-10T04:31:13.157 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 413 (ObjNum 318 snap 81 seq_num 318) dirty exists 2021-02-10T04:31:13.157 INFO:tasks.rados.rados.0.smithi085.stdout:828: left oid 37 (ObjNum 318 snap 81 seq_num 318) 2021-02-10T04:31:13.157 INFO:tasks.rados.rados.0.smithi085.stdout:828: done (14 left) 2021-02-10T04:31:13.157 INFO:tasks.rados.rados.0.smithi085.stdout:829: done (13 left) 2021-02-10T04:31:13.157 INFO:tasks.rados.rados.0.smithi085.stdout:830: done (12 left) 2021-02-10T04:31:13.157 INFO:tasks.rados.rados.0.smithi085.stdout:831: rollback oid 36 current snap is 81 2021-02-10T04:31:13.158 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 36 to 78 2021-02-10T04:31:13.158 INFO:tasks.rados.rados.0.smithi085.stdout:832: write (excl) oid 26 current snap is 81 2021-02-10T04:31:13.158 INFO:tasks.rados.rados.0.smithi085.stdout:832: seq_num 319 ranges {561433=774904,1950697=414440,2777464=583969} 2021-02-10T04:31:13.164 INFO:tasks.rados.rados.0.smithi085.stdout:832: writing smithi08575397-26 from 561433 to 1336337 tid 1 2021-02-10T04:31:13.170 INFO:tasks.rados.rados.0.smithi085.stdout:832: writing smithi08575397-26 from 1950697 to 2365137 tid 2 2021-02-10T04:31:13.177 INFO:tasks.rados.rados.0.smithi085.stdout:832: writing smithi08575397-26 from 2777464 to 3361433 tid 3 2021-02-10T04:31:13.177 INFO:tasks.rados.rados.0.smithi085.stdout:833: copy_from oid 13 from oid 1 current snap is 81 2021-02-10T04:31:13.177 INFO:tasks.rados.rados.0.smithi085.stdout:831: finishing rollback tid 0 to smithi08575397-36 2021-02-10T04:31:13.177 INFO:tasks.rados.rados.0.smithi085.stdout:831: finishing rollback tid 1 to smithi08575397-36 2021-02-10T04:31:13.177 INFO:tasks.rados.rados.0.smithi085.stdout:831: finishing rollback tid 2 to smithi08575397-36 2021-02-10T04:31:13.177 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 127 (ObjNum 306 snap 74 seq_num 306) dirty exists 2021-02-10T04:31:13.177 INFO:tasks.rados.rados.0.smithi085.stdout:831: done (14 left) 2021-02-10T04:31:13.177 INFO:tasks.rados.rados.0.smithi085.stdout:834: read oid 7 snap 79 2021-02-10T04:31:13.178 INFO:tasks.rados.rados.0.smithi085.stdout:834: expect (ObjNum 311 snap 76 seq_num 311) 2021-02-10T04:31:13.178 INFO:tasks.rados.rados.0.smithi085.stdout:835: rollback oid 40 current snap is 81 2021-02-10T04:31:13.178 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 40 to 79 2021-02-10T04:31:13.178 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:13.178 INFO:tasks.rados.rados.0.smithi085.stdout:832: finishing write tid 1 to smithi08575397-26 2021-02-10T04:31:13.184 INFO:tasks.rados.rados.0.smithi085.stdout:835: finishing rollback tid 0 to smithi08575397-40 2021-02-10T04:31:13.184 INFO:tasks.rados.rados.0.smithi085.stdout:835: finishing rollback tid 1 to smithi08575397-40 2021-02-10T04:31:13.184 INFO:tasks.rados.rados.0.smithi085.stdout:835: finishing rollback tid 2 to smithi08575397-40 2021-02-10T04:31:13.184 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 419 (ObjNum 280 snap 70 seq_num 280) dirty exists 2021-02-10T04:31:13.184 INFO:tasks.rados.rados.0.smithi085.stdout:835: done (15 left) 2021-02-10T04:31:13.184 INFO:tasks.rados.rados.0.smithi085.stdout:836: read oid 27 snap -1 2021-02-10T04:31:13.185 INFO:tasks.rados.rados.0.smithi085.stdout:836: expect deleted 2021-02-10T04:31:13.185 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:13.185 INFO:tasks.rados.rados.0.smithi085.stdout:832: finishing write tid 2 to smithi08575397-26 2021-02-10T04:31:13.186 INFO:tasks.rados.rados.0.smithi085.stdout:832: finishing write tid 3 to smithi08575397-26 2021-02-10T04:31:13.186 INFO:tasks.rados.rados.0.smithi085.stdout:832: finishing write tid 4 to smithi08575397-26 2021-02-10T04:31:13.186 INFO:tasks.rados.rados.0.smithi085.stdout:832: finishing write tid 5 to smithi08575397-26 2021-02-10T04:31:13.186 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 75 (ObjNum 319 snap 81 seq_num 319) dirty exists 2021-02-10T04:31:13.186 INFO:tasks.rados.rados.0.smithi085.stdout:832: left oid 26 (ObjNum 319 snap 81 seq_num 319) 2021-02-10T04:31:13.186 INFO:tasks.rados.rados.0.smithi085.stdout:832: done (15 left) 2021-02-10T04:31:13.186 INFO:tasks.rados.rados.0.smithi085.stdout:837: write (excl) oid 21 current snap is 81 2021-02-10T04:31:13.186 INFO:tasks.rados.rados.0.smithi085.stdout:837: seq_num 320 ranges {578240=566506,1609179=558098,2804887=551750} 2021-02-10T04:31:13.193 INFO:tasks.rados.rados.0.smithi085.stdout:837: writing smithi08575397-21 from 578240 to 1144746 tid 1 2021-02-10T04:31:13.199 INFO:tasks.rados.rados.0.smithi085.stdout:837: writing smithi08575397-21 from 1609179 to 2167277 tid 2 2021-02-10T04:31:13.208 INFO:tasks.rados.rados.0.smithi085.stdout:837: writing smithi08575397-21 from 2804887 to 3356637 tid 3 2021-02-10T04:31:13.208 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:13.208 INFO:tasks.rados.rados.0.smithi085.stdout:834: expect (ObjNum 311 snap 76 seq_num 311) 2021-02-10T04:31:13.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:13 smithi085 conmon[29319]: cluster 2021-02-10T04:31:13.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:13 smithi085 conmon[29319]: 2021-02-10T04:31:12.030828 2021-02-10T04:31:13.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:13 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 432 : cluster [DBG] pgmap v532: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 5 active+clean+snaptrim, 161 active+clean; 181 MiB data, 549 MiB used, 715 GiB / 715 GiB avail; 4.7 MiB/s rd, 1.2 MiB/s wr, 8 op/s 2021-02-10T04:31:13.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:13 smithi085 conmon[29319]: cluster 2021-02-10T04:31:12.096988+0000 2021-02-10T04:31:13.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:13 smithi085 conmon[29319]: mon.a (mon.0) 986 : cluster [DBG] osdmap e256: 8 total, 8 up, 8 in 2021-02-10T04:31:13.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:13 smithi085 conmon[35961]: cluster 2021-02-10T04:31:13.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:13 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:31:13.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:13 smithi085 conmon[35961]: 10T04:31:12.030828+0000 mgr.y (mgr.14140) 432 : cluster [DBG] pgmap v532: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 5 active+clean+snaptrim, 161 active+clean; 181 MiB data, 549 MiB used, 715 GiB / 715 GiB avail; 4.7 MiB/s rd, 1.2 MiB/s wr, 8 op/s 2021-02-10T04:31:13.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:13 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:13.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:13 smithi085 conmon[35961]: :31:12.096988+0000 mon.a (mon.0) 2021-02-10T04:31:13.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:13 smithi085 conmon[35961]: 986 : cluster [DBG] osdmap e256: 8 total, 8 up, 8 in 2021-02-10T04:31:13.489 INFO:tasks.rados.rados.0.smithi085.stdout:834: done (15 left) 2021-02-10T04:31:13.489 INFO:tasks.rados.rados.0.smithi085.stdout:836: done (14 left) 2021-02-10T04:31:13.489 INFO:tasks.rados.rados.0.smithi085.stdout:838: write oid 26 current snap is 81 2021-02-10T04:31:13.489 INFO:tasks.rados.rados.0.smithi085.stdout:837: finishing write tid 1 to smithi08575397-21 2021-02-10T04:31:13.489 INFO:tasks.rados.rados.0.smithi085.stdout:837: finishing write tid 2 to smithi08575397-21 2021-02-10T04:31:13.489 INFO:tasks.rados.rados.0.smithi085.stdout:837: finishing write tid 3 to smithi08575397-21 2021-02-10T04:31:13.489 INFO:tasks.rados.rados.0.smithi085.stdout:837: finishing write tid 4 to smithi08575397-21 2021-02-10T04:31:13.490 INFO:tasks.rados.rados.0.smithi085.stdout:838: seq_num 321 ranges {595047=641755,1667661=418109,2548663=763022} 2021-02-10T04:31:13.496 INFO:tasks.rados.rados.0.smithi085.stdout:838: writing smithi08575397-26 from 595047 to 1236802 tid 1 2021-02-10T04:31:13.501 INFO:tasks.rados.rados.0.smithi085.stdout:838: writing smithi08575397-26 from 1667661 to 2085770 tid 2 2021-02-10T04:31:13.509 INFO:tasks.rados.rados.0.smithi085.stdout:838: writing smithi08575397-26 from 2548663 to 3311685 tid 3 2021-02-10T04:31:13.509 INFO:tasks.rados.rados.0.smithi085.stdout:839: snap_create 2021-02-10T04:31:13.509 INFO:tasks.rados.rados.0.smithi085.stdout:837: finishing write tid 5 to smithi08575397-21 2021-02-10T04:31:13.509 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 424 (ObjNum 320 snap 81 seq_num 320) dirty exists 2021-02-10T04:31:13.509 INFO:tasks.rados.rados.0.smithi085.stdout:837: left oid 21 (ObjNum 320 snap 81 seq_num 320) 2021-02-10T04:31:13.509 INFO:tasks.rados.rados.0.smithi085.stdout:833: finishing copy_from to smithi08575397-13 2021-02-10T04:31:13.509 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 236 (ObjNum 308 snap 74 seq_num 308) dirty exists 2021-02-10T04:31:13.509 INFO:tasks.rados.rados.0.smithi085.stdout:833: finishing copy_from racing read to smithi08575397-13 2021-02-10T04:31:13.509 INFO:tasks.rados.rados.0.smithi085.stdout:838: finishing write tid 1 to smithi08575397-26 2021-02-10T04:31:13.510 INFO:tasks.rados.rados.0.smithi085.stdout:838: finishing write tid 2 to smithi08575397-26 2021-02-10T04:31:13.512 INFO:tasks.rados.rados.0.smithi085.stdout:838: finishing write tid 3 to smithi08575397-26 2021-02-10T04:31:13.512 INFO:tasks.rados.rados.0.smithi085.stdout:838: finishing write tid 4 to smithi08575397-26 2021-02-10T04:31:13.513 INFO:tasks.rados.rados.0.smithi085.stdout:838: finishing write tid 5 to smithi08575397-26 2021-02-10T04:31:13.513 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 79 (ObjNum 321 snap 81 seq_num 321) dirty exists 2021-02-10T04:31:13.513 INFO:tasks.rados.rados.0.smithi085.stdout:838: left oid 26 (ObjNum 321 snap 81 seq_num 321) 2021-02-10T04:31:13.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:13 smithi164 conmon[29288]: cluster 2021-02-10T04:31:13.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:13 smithi164 conmon[29288]: 2021-02-10T04:31:13.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:13 smithi164 conmon[29288]: 2021-02-10 2021-02-10T04:31:13.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:13 smithi164 conmon[29288]: T04:31:12.030828+0000 mgr.y (mgr.14140) 432 : cluster [DBG] pgmap v532: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 5 active+clean+snaptrim, 161 active+clean; 181 MiB data, 549 MiB used, 715 GiB / 715 GiB avail; 4.7 MiB/s rd, 1.2 MiB/s wr, 8 op/s 2021-02-10T04:31:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:13 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:13 smithi164 conmon[29288]: 02-10T04:31:12.096988+0000 mon.a (mon.0 2021-02-10T04:31:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:13 smithi164 conmon[29288]: ) 986 : cluster [DBG] osdmap e256: 8 total, 8 up, 8 in 2021-02-10T04:31:13.818 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:31:13.845 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:13.944 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:31:14.048 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:31:14.052 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 7 1 2021-02-10T04:31:14.079 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:14.143 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:31:13 smithi164 conmon[37263]: debug 2021-02-10T04:31:13.907+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:14.160 INFO:tasks.rados.rados.0.smithi085.stdout:833: done (15 left) 2021-02-10T04:31:14.160 INFO:tasks.rados.rados.0.smithi085.stdout:837: done (14 left) 2021-02-10T04:31:14.160 INFO:tasks.rados.rados.0.smithi085.stdout:838: done (13 left) 2021-02-10T04:31:14.160 INFO:tasks.rados.rados.0.smithi085.stdout:839: done (12 left) 2021-02-10T04:31:14.160 INFO:tasks.rados.rados.0.smithi085.stdout:840: read oid 7 snap -1 2021-02-10T04:31:14.160 INFO:tasks.rados.rados.0.smithi085.stdout:840: expect (ObjNum 311 snap 76 seq_num 311) 2021-02-10T04:31:14.161 INFO:tasks.rados.rados.0.smithi085.stdout:841: read oid 2 snap 79 2021-02-10T04:31:14.161 INFO:tasks.rados.rados.0.smithi085.stdout:841: expect (ObjNum 246 snap 57 seq_num 246) 2021-02-10T04:31:14.161 INFO:tasks.rados.rados.0.smithi085.stdout:842: read oid 28 snap -1 2021-02-10T04:31:14.161 INFO:tasks.rados.rados.0.smithi085.stdout:842: expect (ObjNum 310 snap 75 seq_num 310) 2021-02-10T04:31:14.161 INFO:tasks.rados.rados.0.smithi085.stdout:843: copy_from oid 6 from oid 43 current snap is 82 2021-02-10T04:31:14.161 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:14.168 INFO:tasks.rados.rados.0.smithi085.stdout:842: expect (ObjNum 310 snap 75 seq_num 310) 2021-02-10T04:31:14.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:14 smithi085 conmon[35961]: cluster 2021-02-10T04:31:13.154621+0000 2021-02-10T04:31:14.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:14 smithi085 conmon[35961]: mon.a (mon.0) 987 : cluster [DBG] osdmap e257: 8 total, 8 up, 8 in 2021-02-10T04:31:14.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:14 smithi085 conmon[35961]: cluster 2021-02-10T04:31:14.031254+0000 mgr.y (mgr.14140) 433 : cluster [DBG] pgmap v535: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 4 active+clean+snaptrim, 162 active+clean; 184 MiB data, 556 MiB used, 715 GiB / 715 GiB avail; 2.2 MiB/s rd, 784 KiB/s wr, 7 op/s 2021-02-10T04:31:14.289 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:14 smithi085 conmon[29319]: cluster 2021-02-10T04:31:13.154621 2021-02-10T04:31:14.289 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:14 smithi085 conmon[29319]: +0000 mon.a (mon.0) 987 : cluster [DBG] osdmap e257: 8 total, 8 up, 8 in 2021-02-10T04:31:14.289 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:14 smithi085 conmon[29319]: cluster 2021-02-10T04:31:14.031254+0000 mgr.y (mgr.14140) 433 : cluster [DBG] pgmap v535: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 4 active+clean+snaptrim, 162 active+clean; 184 MiB data, 556 MiB used, 715 GiB / 715 GiB avail; 2.2 MiB/s rd, 784 KiB/s wr, 7 op/s 2021-02-10T04:31:14.399 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:31:14.497 INFO:tasks.rados.rados.0.smithi085.stdout:842: done (15 left) 2021-02-10T04:31:14.497 INFO:tasks.rados.rados.0.smithi085.stdout:844: rollback oid 16 current snap is 82 2021-02-10T04:31:14.497 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 16 to 81 2021-02-10T04:31:14.497 INFO:tasks.rados.rados.0.smithi085.stdout:841: expect (ObjNum 246 snap 57 seq_num 246) 2021-02-10T04:31:14.595 INFO:tasks.rados.rados.0.smithi085.stdout:840: expect (ObjNum 311 snap 76 seq_num 311) 2021-02-10T04:31:14.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:14 smithi164 conmon[29288]: cluster 2021-02-10T04:31:13.154621+0000 mon.a (mon.0) 987 2021-02-10T04:31:14.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:14 smithi164 conmon[29288]: : cluster [DBG] osdmap e257: 8 total, 8 up, 8 in 2021-02-10T04:31:14.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:14 smithi164 conmon[29288]: cluster 2021-02-10T04:31:14.031254+0000 mgr.y (mgr.14140) 433 : cluster [DBG] 2021-02-10T04:31:14.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:14 smithi164 conmon[29288]: pgmap v535: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 4 active+clean+snaptrim, 162 active+clean; 184 MiB data, 556 MiB used, 715 GiB / 715 GiB avail; 2.2 MiB/s rd, 784 KiB/s wr, 7 op/s 2021-02-10T04:31:14.876 INFO:tasks.rados.rados.0.smithi085.stdout:843: finishing copy_from to smithi08575397-6 2021-02-10T04:31:14.876 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 247 (ObjNum 312 snap 76 seq_num 312) dirty exists 2021-02-10T04:31:14.876 INFO:tasks.rados.rados.0.smithi085.stdout:843: finishing copy_from racing read to smithi08575397-6 2021-02-10T04:31:14.876 INFO:tasks.rados.rados.0.smithi085.stdout:840: done (15 left) 2021-02-10T04:31:14.876 INFO:tasks.rados.rados.0.smithi085.stdout:841: done (14 left) 2021-02-10T04:31:14.876 INFO:tasks.rados.rados.0.smithi085.stdout:843: done (13 left) 2021-02-10T04:31:14.877 INFO:tasks.rados.rados.0.smithi085.stdout:845: read oid 21 snap -1 2021-02-10T04:31:14.877 INFO:tasks.rados.rados.0.smithi085.stdout:845: expect (ObjNum 320 snap 81 seq_num 320) 2021-02-10T04:31:14.877 INFO:tasks.rados.rados.0.smithi085.stdout:846: snap_remove snap 63 2021-02-10T04:31:15.167 INFO:tasks.rados.rados.0.smithi085.stdout:846: done (14 left) 2021-02-10T04:31:15.167 INFO:tasks.rados.rados.0.smithi085.stdout:847: rollback oid 38 current snap is 82 2021-02-10T04:31:15.168 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 38 to 74 2021-02-10T04:31:15.168 INFO:tasks.rados.rados.0.smithi085.stdout:848: rollback oid 43 current snap is 82 2021-02-10T04:31:15.168 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 43 to 78 2021-02-10T04:31:15.168 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:15.168 INFO:tasks.rados.rados.0.smithi085.stdout:844: finishing rollback tid 0 to smithi08575397-16 2021-02-10T04:31:15.168 INFO:tasks.rados.rados.0.smithi085.stdout:844: finishing rollback tid 1 to smithi08575397-16 2021-02-10T04:31:15.168 INFO:tasks.rados.rados.0.smithi085.stdout:844: finishing rollback tid 2 to smithi08575397-16 2021-02-10T04:31:15.168 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 432 (ObjNum 127 snap 23 seq_num 127) dirty exists 2021-02-10T04:31:15.168 INFO:tasks.rados.rados.0.smithi085.stdout:844: done (15 left) 2021-02-10T04:31:15.168 INFO:tasks.rados.rados.0.smithi085.stdout:849: read oid 50 snap -1 2021-02-10T04:31:15.168 INFO:tasks.rados.rados.0.smithi085.stdout:849: expect deleted 2021-02-10T04:31:15.168 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:15.168 INFO:tasks.rados.rados.0.smithi085.stdout:845: expect (ObjNum 320 snap 81 seq_num 320) 2021-02-10T04:31:15.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:15 smithi085 conmon[29319]: cluster 2021-02-10T04:31:14.160680+0000 mon.a (mon.0 2021-02-10T04:31:15.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:15 smithi085 conmon[29319]: ) 988 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2021-02-10T04:31:15.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:15 smithi085 conmon[35961]: cluster 2021-02-10T04:31:14.160680 2021-02-10T04:31:15.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:15 smithi085 conmon[35961]: +0000 mon.a (mon.0) 988 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2021-02-10T04:31:15.372 INFO:tasks.rados.rados.0.smithi085.stdout:845: done (15 left) 2021-02-10T04:31:15.372 INFO:tasks.rados.rados.0.smithi085.stdout:850: write oid 23 current snap is 82 2021-02-10T04:31:15.372 INFO:tasks.rados.rados.0.smithi085.stdout:850: seq_num 322 ranges {611854=717004,2126143=561767,3259733=152121} 2021-02-10T04:31:15.379 INFO:tasks.rados.rados.0.smithi085.stdout:850: writing smithi08575397-23 from 611854 to 1328858 tid 1 2021-02-10T04:31:15.388 INFO:tasks.rados.rados.0.smithi085.stdout:850: writing smithi08575397-23 from 2126143 to 2687910 tid 2 2021-02-10T04:31:15.394 INFO:tasks.rados.rados.0.smithi085.stdout:850: writing smithi08575397-23 from 3259733 to 3411854 tid 3 2021-02-10T04:31:15.394 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:15.394 INFO:tasks.rados.rados.0.smithi085.stdout:849: done (15 left) 2021-02-10T04:31:15.394 INFO:tasks.rados.rados.0.smithi085.stdout:851: read oid 21 snap -1 2021-02-10T04:31:15.394 INFO:tasks.rados.rados.0.smithi085.stdout:851: expect (ObjNum 320 snap 81 seq_num 320) 2021-02-10T04:31:15.394 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:15.394 INFO:tasks.rados.rados.0.smithi085.stdout:847: finishing rollback tid 0 to smithi08575397-38 2021-02-10T04:31:15.394 INFO:tasks.rados.rados.0.smithi085.stdout:847: finishing rollback tid 1 to smithi08575397-38 2021-02-10T04:31:15.394 INFO:tasks.rados.rados.0.smithi085.stdout:848: finishing rollback tid 0 to smithi08575397-43 2021-02-10T04:31:15.394 INFO:tasks.rados.rados.0.smithi085.stdout:848: finishing rollback tid 1 to smithi08575397-43 2021-02-10T04:31:15.395 INFO:tasks.rados.rados.0.smithi085.stdout:848: finishing rollback tid 2 to smithi08575397-43 2021-02-10T04:31:15.395 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 261 (ObjNum 312 snap 76 seq_num 312) dirty exists 2021-02-10T04:31:15.395 INFO:tasks.rados.rados.0.smithi085.stdout:847: finishing rollback tid 2 to smithi08575397-38 2021-02-10T04:31:15.395 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 265 (ObjNum 287 snap 71 seq_num 287) dirty exists 2021-02-10T04:31:15.395 INFO:tasks.rados.rados.0.smithi085.stdout:850: finishing write tid 1 to smithi08575397-23 2021-02-10T04:31:15.395 INFO:tasks.rados.rados.0.smithi085.stdout:850: finishing write tid 2 to smithi08575397-23 2021-02-10T04:31:15.395 INFO:tasks.rados.rados.0.smithi085.stdout:847: done (15 left) 2021-02-10T04:31:15.395 INFO:tasks.rados.rados.0.smithi085.stdout:848: done (14 left) 2021-02-10T04:31:15.395 INFO:tasks.rados.rados.0.smithi085.stdout:852: read oid 38 snap 74 2021-02-10T04:31:15.395 INFO:tasks.rados.rados.0.smithi085.stdout:852: expect (ObjNum 287 snap 71 seq_num 287) 2021-02-10T04:31:15.395 INFO:tasks.rados.rados.0.smithi085.stdout:853: snap_create 2021-02-10T04:31:15.397 INFO:tasks.rados.rados.0.smithi085.stdout:852: expect (ObjNum 287 snap 71 seq_num 287) 2021-02-10T04:31:15.590 INFO:tasks.rados.rados.0.smithi085.stdout:850: finishing write tid 3 to smithi08575397-23 2021-02-10T04:31:15.590 INFO:tasks.rados.rados.0.smithi085.stdout:850: finishing write tid 4 to smithi08575397-23 2021-02-10T04:31:15.590 INFO:tasks.rados.rados.0.smithi085.stdout:850: finishing write tid 5 to smithi08575397-23 2021-02-10T04:31:15.590 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 136 (ObjNum 322 snap 82 seq_num 322) dirty exists 2021-02-10T04:31:15.590 INFO:tasks.rados.rados.0.smithi085.stdout:850: left oid 23 (ObjNum 322 snap 82 seq_num 322) 2021-02-10T04:31:15.590 INFO:tasks.rados.rados.0.smithi085.stdout:851: expect (ObjNum 320 snap 81 seq_num 320) 2021-02-10T04:31:15.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:15 smithi164 conmon[29288]: cluster 2021-02-10T04:31:14.160680+0000 mon.a (mon. 2021-02-10T04:31:15.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:15 smithi164 conmon[29288]: 0) 988 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2021-02-10T04:31:15.644 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:31:15 smithi164 conmon[40892]: debug 2021-02-10T04:31:15.644 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:31:15 smithi164 conmon[40892]: 2021-02-10T04:31:15.170+0000 7f3032cc0700 -1 osd.6 pg_epoch: 259 pg[7.c( v 240'123 (0'0,240'123] local-lis/les=93/94 n=5 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [6,3,7] r=0 lpr=93 crt=240'123 lcod 225'121 mlcod 225'121 active+clean+snaptrim trimq=[7e~1] ps=[60~3,66~3,6b~3,6f~1,74~2,78~4,80~4,87~2,8b~3,8f~1,92~1,97~1,99~1,9e~1]] removing snap head 2021-02-10T04:31:15.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:15 smithi164 conmon[33677]: debug 2021-02-10T04:31:15.166+0000 7fbea6125700 -1 osd.4 pg_epoch: 259 pg[7.1( v 242'71 (0'0,242'71] local-lis/les=93/94 n=4 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [4,3,6] r=0 lpr=93 crt=242'71 lcod 240'69 mlcod 240'69 active+clean+snaptrim trimq=[7e~1] ps=[60~3,66~3,6b~3,6f~1,74~2,78~4,80~4,87~2,8b~3,8f~1,92~1,97~1,99~1,9e~1]] removing snap head 2021-02-10T04:31:16.172 INFO:tasks.rados.rados.0.smithi085.stdout:850: done (15 left) 2021-02-10T04:31:16.172 INFO:tasks.rados.rados.0.smithi085.stdout:851: done (14 left) 2021-02-10T04:31:16.172 INFO:tasks.rados.rados.0.smithi085.stdout:852: done (13 left) 2021-02-10T04:31:16.172 INFO:tasks.rados.rados.0.smithi085.stdout:853: done (12 left) 2021-02-10T04:31:16.172 INFO:tasks.rados.rados.0.smithi085.stdout:854: read oid 38 snap -1 2021-02-10T04:31:16.172 INFO:tasks.rados.rados.0.smithi085.stdout:854: expect (ObjNum 287 snap 71 seq_num 287) 2021-02-10T04:31:16.172 INFO:tasks.rados.rados.0.smithi085.stdout:855: read oid 8 snap 81 2021-02-10T04:31:16.172 INFO:tasks.rados.rados.0.smithi085.stdout:855: expect deleted 2021-02-10T04:31:16.172 INFO:tasks.rados.rados.0.smithi085.stdout:856: delete oid 37 current snap is 83 2021-02-10T04:31:16.179 INFO:tasks.rados.rados.0.smithi085.stdout:854: expect (ObjNum 287 snap 71 seq_num 287) 2021-02-10T04:31:16.379 INFO:tasks.rados.rados.0.smithi085.stdout:854: done (14 left) 2021-02-10T04:31:16.379 INFO:tasks.rados.rados.0.smithi085.stdout:855: done (13 left) 2021-02-10T04:31:16.379 INFO:tasks.rados.rados.0.smithi085.stdout:856: done (12 left) 2021-02-10T04:31:16.379 INFO:tasks.rados.rados.0.smithi085.stdout:857: rollback oid 27 current snap is 83 2021-02-10T04:31:16.379 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 27 to 82 2021-02-10T04:31:16.379 INFO:tasks.rados.rados.0.smithi085.stdout:858: read oid 11 snap 79 2021-02-10T04:31:16.379 INFO:tasks.rados.rados.0.smithi085.stdout:858: expect deleted 2021-02-10T04:31:16.379 INFO:tasks.rados.rados.0.smithi085.stdout:859: read oid 16 snap -1 2021-02-10T04:31:16.379 INFO:tasks.rados.rados.0.smithi085.stdout:859: expect (ObjNum 127 snap 23 seq_num 127) 2021-02-10T04:31:16.380 INFO:tasks.rados.rados.0.smithi085.stdout:860: copy_from oid 7 from oid 36 current snap is 83 2021-02-10T04:31:16.380 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:16.380 INFO:tasks.rados.rados.0.smithi085.stdout:857: finishing rollback tid 1 to smithi08575397-27 2021-02-10T04:31:16.380 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 27 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-02-10T04:31:16.381 INFO:tasks.rados.rados.0.smithi085.stdout:857: done (15 left) 2021-02-10T04:31:16.381 INFO:tasks.rados.rados.0.smithi085.stdout:861: rollback oid 38 current snap is 83 2021-02-10T04:31:16.381 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 38 to 78 2021-02-10T04:31:16.381 INFO:tasks.rados.rados.0.smithi085.stdout:858: done (15 left) 2021-02-10T04:31:16.381 INFO:tasks.rados.rados.0.smithi085.stdout:862: read oid 5 snap -1 2021-02-10T04:31:16.381 INFO:tasks.rados.rados.0.smithi085.stdout:862: expect (ObjNum 315 snap 80 seq_num 315) 2021-02-10T04:31:16.381 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:16.383 INFO:tasks.rados.rados.0.smithi085.stdout:861: finishing rollback tid 0 to smithi08575397-38 2021-02-10T04:31:16.385 INFO:tasks.rados.rados.0.smithi085.stdout:861: finishing rollback tid 1 to smithi08575397-38 2021-02-10T04:31:16.385 INFO:tasks.rados.rados.0.smithi085.stdout:861: finishing rollback tid 2 to smithi08575397-38 2021-02-10T04:31:16.385 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 269 (ObjNum 287 snap 71 seq_num 287) dirty exists 2021-02-10T04:31:16.385 INFO:tasks.rados.rados.0.smithi085.stdout:861: done (15 left) 2021-02-10T04:31:16.385 INFO:tasks.rados.rados.0.smithi085.stdout:863: read oid 46 snap 82 2021-02-10T04:31:16.385 INFO:tasks.rados.rados.0.smithi085.stdout:863: expect deleted 2021-02-10T04:31:16.385 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:16.387 INFO:tasks.rados.rados.0.smithi085.stdout:863: done (15 left) 2021-02-10T04:31:16.387 INFO:tasks.rados.rados.0.smithi085.stdout:864: write oid 22 current snap is 83 2021-02-10T04:31:16.387 INFO:tasks.rados.rados.0.smithi085.stdout:864: seq_num 324 ranges {645468=467502,1726754=565436,2798226=596838} 2021-02-10T04:31:16.394 INFO:tasks.rados.rados.0.smithi085.stdout:864: writing smithi08575397-22 from 645468 to 1112970 tid 1 2021-02-10T04:31:16.402 INFO:tasks.rados.rados.0.smithi085.stdout:864: writing smithi08575397-22 from 1726754 to 2292190 tid 2 2021-02-10T04:31:16.409 INFO:tasks.rados.rados.0.smithi085.stdout:864: writing smithi08575397-22 from 2798226 to 3395064 tid 3 2021-02-10T04:31:16.409 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:16.409 INFO:tasks.rados.rados.0.smithi085.stdout:859: expect (ObjNum 127 snap 23 seq_num 127) 2021-02-10T04:31:16.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:16 smithi085 conmon[29319]: cluster 2021-02-10T04:31:15.166781+0000 mon.a ( 2021-02-10T04:31:16.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:16 smithi085 conmon[29319]: mon.0) 989 : cluster [DBG] osdmap e259: 8 total, 8 up, 8 in 2021-02-10T04:31:16.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:16 smithi085 conmon[29319]: cluster 2021-02-10T04:31:16.031703+0000 mgr.y (mgr.14140) 434 : cluster [DBG] pgmap v538: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim, 165 active+clean; 185 MiB data, 573 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 3.0 MiB/s wr, 26 op/s 2021-02-10T04:31:16.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:16 smithi085 conmon[35961]: cluster 2021-02-10T04:31:16.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:16 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:31:16.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:16 smithi085 conmon[35961]: :31:15.166781+0000 mon.a (mon.0) 989 : cluster [DBG] osdmap e259: 8 total, 8 up, 8 in 2021-02-10T04:31:16.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:16 smithi085 conmon[35961]: cluster 2021-02-10T04:31:16.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:16 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:31:16.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:16 smithi085 conmon[35961]: 10T04:31:16. 2021-02-10T04:31:16.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:16 smithi085 conmon[35961]: 031703+0000 mgr.y (mgr.14140 2021-02-10T04:31:16.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:16 smithi085 conmon[35961]: ) 434 : cluster [DBG] pgmap v538: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim, 165 active+clean; 185 MiB data, 573 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 3.0 MiB/s wr, 26 op/s 2021-02-10T04:31:16.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:16 smithi085 conmon[35961]: 2021-02-10T04:31:16.521 INFO:tasks.rados.rados.0.smithi085.stdout:859: done (15 left) 2021-02-10T04:31:16.521 INFO:tasks.rados.rados.0.smithi085.stdout:865: write (excl) oid 2 current snap is 83 2021-02-10T04:31:16.521 INFO:tasks.rados.rados.0.smithi085.stdout:865: seq_num 325 ranges {662275=542751,1785236=425447,2942002=520273} 2021-02-10T04:31:16.522 INFO:tasks.rados.rados.0.smithi085.stdout:865: writing smithi08575397-2 from 662275 to 1205026 tid 1 2021-02-10T04:31:16.529 INFO:tasks.rados.rados.0.smithi085.stdout:865: writing smithi08575397-2 from 1785236 to 2210683 tid 2 2021-02-10T04:31:16.536 INFO:tasks.rados.rados.0.smithi085.stdout:865: writing smithi08575397-2 from 2942002 to 3462275 tid 3 2021-02-10T04:31:16.537 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:16.537 INFO:tasks.rados.rados.0.smithi085.stdout:860: finishing copy_from to smithi08575397-7 2021-02-10T04:31:16.537 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 127 (ObjNum 306 snap 74 seq_num 306) dirty exists 2021-02-10T04:31:16.537 INFO:tasks.rados.rados.0.smithi085.stdout:860: finishing copy_from racing read to smithi08575397-7 2021-02-10T04:31:16.537 INFO:tasks.rados.rados.0.smithi085.stdout:864: finishing write tid 1 to smithi08575397-22 2021-02-10T04:31:16.537 INFO:tasks.rados.rados.0.smithi085.stdout:862: expect (ObjNum 315 snap 80 seq_num 315) 2021-02-10T04:31:16.568 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:31:16.643 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:31:16 smithi164 conmon[40892]: debug 2021-02-10T04:31:16.388+0000 7f30324bf700 -1 osd.6 pg_epoch: 260 pg[7.d( v 260'158 (0'0,260'158] local-lis/les=93/94 n=4 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [6,1,0] r=0 lpr=93 luod=250'156 crt=260'158 lcod 247'154 mlcod 247'154 active+clean+snaptrim trimq=[7e~1] ps=[60~3,66~3,6b~3,6f~1,74~2,78~4,80~4,87~2,8b~3,8f~1,92~1,97~1,99~1,9e~1]] removing snap head 2021-02-10T04:31:16.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:16 smithi164 conmon[29288]: cluster 2021-02-10T04:31:15.166781+0000 mon.a (mon 2021-02-10T04:31:16.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:16 smithi164 conmon[29288]: .0) 989 : cluster [DBG] osdmap e259: 8 total, 8 up, 8 in 2021-02-10T04:31:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:16 smithi164 conmon[29288]: cluster 2021-02-10T04:31:16.031703+0000 mgr.y (mgr.14140) 434 : cluster [DBG] pgmap v538: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim, 165 active+clean; 185 MiB data, 573 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 3.0 MiB/s wr, 26 op/s 2021-02-10T04:31:16.676 INFO:tasks.rados.rados.0.smithi085.stdout:864: finishing write tid 2 to smithi08575397-22 2021-02-10T04:31:16.676 INFO:tasks.rados.rados.0.smithi085.stdout:864: finishing write tid 3 to smithi08575397-22 2021-02-10T04:31:16.676 INFO:tasks.rados.rados.0.smithi085.stdout:864: finishing write tid 4 to smithi08575397-22 2021-02-10T04:31:16.676 INFO:tasks.rados.rados.0.smithi085.stdout:864: finishing write tid 5 to smithi08575397-22 2021-02-10T04:31:16.677 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 132 (ObjNum 324 snap 83 seq_num 324) dirty exists 2021-02-10T04:31:16.677 INFO:tasks.rados.rados.0.smithi085.stdout:864: left oid 22 (ObjNum 324 snap 83 seq_num 324) 2021-02-10T04:31:16.677 INFO:tasks.rados.rados.0.smithi085.stdout:860: done (15 left) 2021-02-10T04:31:16.677 INFO:tasks.rados.rados.0.smithi085.stdout:862: done (14 left) 2021-02-10T04:31:16.677 INFO:tasks.rados.rados.0.smithi085.stdout:864: done (13 left) 2021-02-10T04:31:16.677 INFO:tasks.rados.rados.0.smithi085.stdout:866: write oid 37 current snap is 83 2021-02-10T04:31:16.677 INFO:tasks.rados.rados.0.smithi085.stdout:865: finishing write tid 1 to smithi08575397-2 2021-02-10T04:31:16.677 INFO:tasks.rados.rados.0.smithi085.stdout:866: seq_num 326 ranges {679082=618000,1843718=569105,2853072=626010} 2021-02-10T04:31:16.684 INFO:tasks.rados.rados.0.smithi085.stdout:866: writing smithi08575397-37 from 679082 to 1297082 tid 1 2021-02-10T04:31:16.691 INFO:tasks.rados.rados.0.smithi085.stdout:866: writing smithi08575397-37 from 1843718 to 2412823 tid 2 2021-02-10T04:31:16.693 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:31:16.699 INFO:tasks.rados.rados.0.smithi085.stdout:866: writing smithi08575397-37 from 2853072 to 3479082 tid 3 2021-02-10T04:31:16.699 INFO:tasks.rados.rados.0.smithi085.stdout:867: copy_from oid 43 from oid 26 current snap is 83 2021-02-10T04:31:16.699 INFO:tasks.rados.rados.0.smithi085.stdout:865: finishing write tid 2 to smithi08575397-2 2021-02-10T04:31:16.699 INFO:tasks.rados.rados.0.smithi085.stdout:865: finishing write tid 3 to smithi08575397-2 2021-02-10T04:31:16.699 INFO:tasks.rados.rados.0.smithi085.stdout:865: finishing write tid 4 to smithi08575397-2 2021-02-10T04:31:16.700 INFO:tasks.rados.rados.0.smithi085.stdout:865: finishing write tid 5 to smithi08575397-2 2021-02-10T04:31:16.700 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 281 (ObjNum 325 snap 83 seq_num 325) dirty exists 2021-02-10T04:31:16.700 INFO:tasks.rados.rados.0.smithi085.stdout:865: left oid 2 (ObjNum 325 snap 83 seq_num 325) 2021-02-10T04:31:16.700 INFO:tasks.rados.rados.0.smithi085.stdout:865: done (14 left) 2021-02-10T04:31:16.700 INFO:tasks.rados.rados.0.smithi085.stdout:868: read oid 6 snap -1 2021-02-10T04:31:16.700 INFO:tasks.rados.rados.0.smithi085.stdout:868: expect (ObjNum 312 snap 76 seq_num 312) 2021-02-10T04:31:16.700 INFO:tasks.rados.rados.0.smithi085.stdout:869: read oid 27 snap -1 2021-02-10T04:31:16.700 INFO:tasks.rados.rados.0.smithi085.stdout:869: expect deleted 2021-02-10T04:31:16.700 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:16.700 INFO:tasks.rados.rados.0.smithi085.stdout:866: finishing write tid 1 to smithi08575397-37 2021-02-10T04:31:16.700 INFO:tasks.rados.rados.0.smithi085.stdout:866: finishing write tid 2 to smithi08575397-37 2021-02-10T04:31:16.701 INFO:tasks.rados.rados.0.smithi085.stdout:869: done (15 left) 2021-02-10T04:31:16.701 INFO:tasks.rados.rados.0.smithi085.stdout:870: read oid 2 snap 78 2021-02-10T04:31:16.701 INFO:tasks.rados.rados.0.smithi085.stdout:870: expect (ObjNum 246 snap 57 seq_num 246) 2021-02-10T04:31:16.701 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:16.704 INFO:tasks.rados.rados.0.smithi085.stdout:870: expect (ObjNum 246 snap 57 seq_num 246) 2021-02-10T04:31:16.720 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:16.739 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:31:16 smithi085 conmon[41540]: debug 2021-02-10T04:31:16.739 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:31:16 smithi085 conmon[41540]: 2021-02-10T04:31:16.453+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:16.739 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:31:16 smithi085 conmon[41540]: 2021-02-10T04:31:16.797 INFO:tasks.rados.rados.0.smithi085.stdout:870: done (15 left) 2021-02-10T04:31:16.798 INFO:tasks.rados.rados.0.smithi085.stdout:871: read oid 11 snap -1 2021-02-10T04:31:16.821 INFO:tasks.rados.rados.0.smithi085.stdout:871: expect deleted 2021-02-10T04:31:16.821 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:16.821 INFO:tasks.rados.rados.0.smithi085.stdout:866: finishing write tid 3 to smithi08575397-37 2021-02-10T04:31:16.821 INFO:tasks.rados.rados.0.smithi085.stdout:866: finishing write tid 4 to smithi08575397-37 2021-02-10T04:31:16.821 INFO:tasks.rados.rados.0.smithi085.stdout:866: finishing write tid 5 to smithi08575397-37 2021-02-10T04:31:16.821 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 438 (ObjNum 326 snap 83 seq_num 326) dirty exists 2021-02-10T04:31:16.821 INFO:tasks.rados.rados.0.smithi085.stdout:866: left oid 37 (ObjNum 326 snap 83 seq_num 326) 2021-02-10T04:31:16.821 INFO:tasks.rados.rados.0.smithi085.stdout:866: done (15 left) 2021-02-10T04:31:16.821 INFO:tasks.rados.rados.0.smithi085.stdout:872: read oid 38 snap -1 2021-02-10T04:31:16.821 INFO:tasks.rados.rados.0.smithi085.stdout:872: expect (ObjNum 287 snap 71 seq_num 287) 2021-02-10T04:31:16.821 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:16.822 INFO:tasks.rados.rados.0.smithi085.stdout:868: expect (ObjNum 312 snap 76 seq_num 312) 2021-02-10T04:31:16.843 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:31:16.943 INFO:tasks.rados.rados.0.smithi085.stdout:868: done (15 left) 2021-02-10T04:31:16.943 INFO:tasks.rados.rados.0.smithi085.stdout:873: write (excl) oid 28 current snap is 83 2021-02-10T04:31:16.943 INFO:tasks.rados.rados.0.smithi085.stdout:873: seq_num 327 ranges {695889=409602,1618553=712763,2996848=499041} 2021-02-10T04:31:16.948 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:31:16.950 INFO:tasks.rados.rados.0.smithi085.stdout:873: writing smithi08575397-28 from 695889 to 1105491 tid 1 2021-02-10T04:31:16.957 INFO:tasks.rados.rados.0.smithi085.stdout:873: writing smithi08575397-28 from 1618553 to 2331316 tid 2 2021-02-10T04:31:16.964 INFO:tasks.rados.rados.0.smithi085.stdout:873: writing smithi08575397-28 from 2996848 to 3495889 tid 3 2021-02-10T04:31:16.964 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:16.964 INFO:tasks.rados.rados.0.smithi085.stdout:867: finishing copy_from to smithi08575397-43 2021-02-10T04:31:16.964 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 79 (ObjNum 321 snap 81 seq_num 321) dirty exists 2021-02-10T04:31:16.965 INFO:tasks.rados.rados.0.smithi085.stdout:867: finishing copy_from racing read to smithi08575397-43 2021-02-10T04:31:16.965 INFO:tasks.rados.rados.0.smithi085.stdout:867: done (15 left) 2021-02-10T04:31:16.965 INFO:tasks.rados.rados.0.smithi085.stdout:874: write oid 31 current snap is 83 2021-02-10T04:31:16.965 INFO:tasks.rados.rados.0.smithi085.stdout:872: expect (ObjNum 287 snap 71 seq_num 287) 2021-02-10T04:31:16.998 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:17.097 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:31:17.124 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:31:17.143 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:31:16 smithi164 conmon[40892]: debug 2021-02-10T04:31:16.793+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:17.152 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:31:17.158 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:31:17.182 INFO:tasks.rados.rados.0.smithi085.stdout:874: seq_num 328 ranges {712696=484851,1960682=572774,3307918=204778} 2021-02-10T04:31:17.188 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:31:17.190 INFO:tasks.rados.rados.0.smithi085.stdout:874: writing smithi08575397-31 from 712696 to 1197547 tid 1 2021-02-10T04:31:17.198 INFO:tasks.rados.rados.0.smithi085.stdout:874: writing smithi08575397-31 from 1960682 to 2533456 tid 2 2021-02-10T04:31:17.204 INFO:tasks.rados.rados.0.smithi085.stdout:874: writing smithi08575397-31 from 3307918 to 3512696 tid 3 2021-02-10T04:31:17.205 INFO:tasks.rados.rados.0.smithi085.stdout:872: done (15 left) 2021-02-10T04:31:17.205 INFO:tasks.rados.rados.0.smithi085.stdout:875: copy_from oid 8 from oid 6 current snap is 83 2021-02-10T04:31:17.206 INFO:tasks.rados.rados.0.smithi085.stdout:873: finishing write tid 1 to smithi08575397-28 2021-02-10T04:31:17.206 INFO:tasks.rados.rados.0.smithi085.stdout:873: finishing write tid 2 to smithi08575397-28 2021-02-10T04:31:17.206 INFO:tasks.rados.rados.0.smithi085.stdout:873: finishing write tid 3 to smithi08575397-28 2021-02-10T04:31:17.206 INFO:tasks.rados.rados.0.smithi085.stdout:873: finishing write tid 4 to smithi08575397-28 2021-02-10T04:31:17.206 INFO:tasks.rados.rados.0.smithi085.stdout:873: finishing write tid 5 to smithi08575397-28 2021-02-10T04:31:17.206 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 109 (ObjNum 327 snap 83 seq_num 327) dirty exists 2021-02-10T04:31:17.206 INFO:tasks.rados.rados.0.smithi085.stdout:873: left oid 28 (ObjNum 327 snap 83 seq_num 327) 2021-02-10T04:31:17.206 INFO:tasks.rados.rados.0.smithi085.stdout:874: finishing write tid 1 to smithi08575397-31 2021-02-10T04:31:17.206 INFO:tasks.rados.rados.0.smithi085.stdout:874: finishing write tid 2 to smithi08575397-31 2021-02-10T04:31:17.206 INFO:tasks.rados.rados.0.smithi085.stdout:871: done (15 left) 2021-02-10T04:31:17.206 INFO:tasks.rados.rados.0.smithi085.stdout:873: done (14 left) 2021-02-10T04:31:17.206 INFO:tasks.rados.rados.0.smithi085.stdout:876: copy_from oid 16 from oid 27 current snap is 83 2021-02-10T04:31:17.206 INFO:tasks.rados.rados.0.smithi085.stdout:877: snap_remove snap 81 2021-02-10T04:31:17.221 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:31:17.248 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:31:17.259 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:31:17.279 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:31:17.290 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:17.311 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:31:17.314 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:31:17.314 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:31:17.314 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:31:17.314 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:31:17.314 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:31:17.315 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:31:17.315 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:31:17.315 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:31:17.342 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[52836]: debug 2021-02-10T04:31:17.119+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:17.342 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[52836]: debug 2021-02-10T04:31:17.184+0000 7fba707af700 -1 osd.3 pg_epoch: 260 pg[7.3( v 260'271 (0'0,260'271] local-lis/les=93/94 n=10 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [3,2,4] r=0 lpr=93 luod=260'269 crt=260'271 lcod 260'268 mlcod 260'268 active+clean+snaptrim trimq=[7e~1] ps=[60~3,66~3,6b~3,6f~1,74~2,78~4,80~4,87~2,8b~3,8f~1,92~1,97~1,99~1,9e~1]] removing snap head 2021-02-10T04:31:17.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[29319]: cluster 2021-02-10T04:31:16.171859+0000 mon.a 2021-02-10T04:31:17.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[29319]: (mon.0) 990 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2021-02-10T04:31:17.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[29319]: audit 2021-02-10T04:31:17.059601+0000 mon.a (mon 2021-02-10T04:31:17.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[29319]: .0) 991 : audit [DBG] from='client.? 172.21.15.85:0/2621784662' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:31:17.343 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: cluster 2021-02-10T04:31:17.343 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: 2021-02-10T04:31:17.343 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: 2021 2021-02-10T04:31:17.343 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: - 2021-02-10T04:31:17.343 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: 02 2021-02-10T04:31:17.343 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: - 2021-02-10T04:31:17.343 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: 10 2021-02-10T04:31:17.343 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: T 2021-02-10T04:31:17.343 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: 04 2021-02-10T04:31:17.344 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: : 2021-02-10T04:31:17.344 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: 31 2021-02-10T04:31:17.344 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: : 2021-02-10T04:31:17.344 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: 16 2021-02-10T04:31:17.344 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: . 2021-02-10T04:31:17.344 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: 171859 2021-02-10T04:31:17.344 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: +0000 2021-02-10T04:31:17.344 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: 2021-02-10T04:31:17.344 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: mon.a 2021-02-10T04:31:17.344 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: ( 2021-02-10T04:31:17.344 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: mon 2021-02-10T04:31:17.344 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: . 2021-02-10T04:31:17.344 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: 0 2021-02-10T04:31:17.345 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: ) 2021-02-10T04:31:17.345 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: 990 2021-02-10T04:31:17.345 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: : 2021-02-10T04:31:17.345 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: cluster 2021-02-10T04:31:17.345 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: 2021-02-10T04:31:17.345 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: [DBG] 2021-02-10T04:31:17.345 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: 2021-02-10T04:31:17.345 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: osdmap e260: 8 total, 8 up, 8 in 2021-02-10T04:31:17.345 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: 2021-02-10T04:31:17.345 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: audit 2021-02-10T04:31:17.345 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: 2021-02-10T04:31:17.345 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: 2021 2021-02-10T04:31:17.345 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: - 2021-02-10T04:31:17.345 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: 02 2021-02-10T04:31:17.346 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: - 2021-02-10T04:31:17.346 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: 10 2021-02-10T04:31:17.346 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: T 2021-02-10T04:31:17.346 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: 04 2021-02-10T04:31:17.346 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: : 2021-02-10T04:31:17.346 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: 31 2021-02-10T04:31:17.346 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: : 2021-02-10T04:31:17.346 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: 17 2021-02-10T04:31:17.346 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: . 2021-02-10T04:31:17.346 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: 059601 2021-02-10T04:31:17.346 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: +0000 2021-02-10T04:31:17.346 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: 2021-02-10T04:31:17.346 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: mon.a 2021-02-10T04:31:17.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: ( 2021-02-10T04:31:17.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: mon 2021-02-10T04:31:17.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: . 2021-02-10T04:31:17.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: 0 2021-02-10T04:31:17.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: ) 2021-02-10T04:31:17.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: 991 2021-02-10T04:31:17.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: : 2021-02-10T04:31:17.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: audit 2021-02-10T04:31:17.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: 2021-02-10T04:31:17.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: [DBG] 2021-02-10T04:31:17.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: 2021-02-10T04:31:17.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/2621784662' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:31:17.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[35961]: 2021-02-10T04:31:17.393 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:31:17.444 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:17 smithi164 conmon[29288]: cluster 2021-02-10T04:31:16.171859+0000 mon.a (mon.0) 2021-02-10T04:31:17.444 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:17 smithi164 conmon[29288]: 990 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2021-02-10T04:31:17.444 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:17 smithi164 conmon[29288]: audit 2021-02-10T04:31:17.059601+0000 mon.a (mon.0) 991 : audit [DBG] from='client.? 172.21.15.85:0/2621784662' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:31:17.445 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:31:17 smithi164 conmon[44544]: debug 2021-02-10T04:31:17.363+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:17.445 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:17 smithi164 conmon[33677]: debug 2021-02-10T04:31:17.443+0000 7fbec19a1700 -1 osd.4 260 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:31:17.498 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:31:17.529 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:17.635 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:31:17.740 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:31:17.772 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:17.835 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:17 smithi164 conmon[33677]: debug 2021-02-10T04:31:17.596+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:17.835 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:31:17 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:31:17.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:17 smithi085 conmon[49106]: debug 2021-02-10T04:31:17.847+0000 7f77a3e94700 -1 osd.2 260 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:31:18.181 INFO:tasks.rados.rados.0.smithi085.stdout:877: done (15 left) 2021-02-10T04:31:18.182 INFO:tasks.rados.rados.0.smithi085.stdout:878: read oid 24 snap -1 2021-02-10T04:31:18.182 INFO:tasks.rados.rados.0.smithi085.stdout:878: expect deleted 2021-02-10T04:31:18.182 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:18.182 INFO:tasks.rados.rados.0.smithi085.stdout:874: finishing write tid 3 to smithi08575397-31 2021-02-10T04:31:18.182 INFO:tasks.rados.rados.0.smithi085.stdout:874: finishing write tid 4 to smithi08575397-31 2021-02-10T04:31:18.182 INFO:tasks.rados.rados.0.smithi085.stdout:876: finishing copy_from to smithi08575397-16 2021-02-10T04:31:18.182 INFO:tasks.rados.rados.0.smithi085.stdout:876: got expected ENOENT (src dne) 2021-02-10T04:31:18.182 INFO:tasks.rados.rados.0.smithi085.stdout:874: finishing write tid 5 to smithi08575397-31 2021-02-10T04:31:18.182 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 136 (ObjNum 328 snap 83 seq_num 328) dirty exists 2021-02-10T04:31:18.182 INFO:tasks.rados.rados.0.smithi085.stdout:874: left oid 31 (ObjNum 328 snap 83 seq_num 328) 2021-02-10T04:31:18.182 INFO:tasks.rados.rados.0.smithi085.stdout:876: finishing copy_from racing read to smithi08575397-16 2021-02-10T04:31:18.183 INFO:tasks.rados.rados.0.smithi085.stdout:875: finishing copy_from to smithi08575397-8 2021-02-10T04:31:18.183 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 247 (ObjNum 312 snap 76 seq_num 312) dirty exists 2021-02-10T04:31:18.183 INFO:tasks.rados.rados.0.smithi085.stdout:875: finishing copy_from racing read to smithi08575397-8 2021-02-10T04:31:18.183 INFO:tasks.rados.rados.0.smithi085.stdout:874: done (15 left) 2021-02-10T04:31:18.183 INFO:tasks.rados.rados.0.smithi085.stdout:875: done (14 left) 2021-02-10T04:31:18.183 INFO:tasks.rados.rados.0.smithi085.stdout:876: done (13 left) 2021-02-10T04:31:18.183 INFO:tasks.rados.rados.0.smithi085.stdout:879: copy_from oid 5 from oid 13 current snap is 83 2021-02-10T04:31:18.183 INFO:tasks.rados.rados.0.smithi085.stdout:880: rollback oid 7 current snap is 83 2021-02-10T04:31:18.183 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 7 to 74 2021-02-10T04:31:18.183 INFO:tasks.rados.rados.0.smithi085.stdout:881: read oid 37 snap -1 2021-02-10T04:31:18.183 INFO:tasks.rados.rados.0.smithi085.stdout:881: expect (ObjNum 326 snap 83 seq_num 326) 2021-02-10T04:31:18.183 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:18.184 INFO:tasks.rados.rados.0.smithi085.stdout:878: done (15 left) 2021-02-10T04:31:18.184 INFO:tasks.rados.rados.0.smithi085.stdout:882: snap_create 2021-02-10T04:31:18.193 INFO:tasks.rados.rados.0.smithi085.stdout:880: finishing rollback tid 0 to smithi08575397-7 2021-02-10T04:31:18.193 INFO:tasks.rados.rados.0.smithi085.stdout:880: finishing rollback tid 1 to smithi08575397-7 2021-02-10T04:31:18.198 INFO:tasks.rados.rados.0.smithi085.stdout:881: expect (ObjNum 326 snap 83 seq_num 326) 2021-02-10T04:31:18.356 INFO:tasks.rados.rados.0.smithi085.stdout:880: finishing rollback tid 2 to smithi08575397-7 2021-02-10T04:31:18.356 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 264 (ObjNum 202 snap 44 seq_num 202) dirty exists 2021-02-10T04:31:18.356 INFO:tasks.rados.rados.0.smithi085.stdout:879: finishing copy_from to smithi08575397-5 2021-02-10T04:31:18.356 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 236 (ObjNum 308 snap 74 seq_num 308) dirty exists 2021-02-10T04:31:18.356 INFO:tasks.rados.rados.0.smithi085.stdout:879: finishing copy_from racing read to smithi08575397-5 2021-02-10T04:31:18.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[29319]: cluster 2021-02-10T04:31:17.444518+0000 2021-02-10T04:31:18.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[29319]: osd.4 (osd.4) 143 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:18.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:31:18.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[29319]: 04:31:17.444524+0000 osd.4 (osd.4) 144 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:18.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:31:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[29319]: 31:17.444528+0000 osd.4 (osd.4) 145 2021-02-10T04:31:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[29319]: cluster 2021-02-10T04:31:17.444532 2021-02-10T04:31:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 146 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[29319]: audit 2021-02-10T04:31:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[29319]: 2021-02-10T04:31 2021-02-10T04:31:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[29319]: :17.835061+0000 mgr.y ( 2021-02-10T04:31:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[29319]: mgr.14140) 435 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:31:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[29319]: 2021-02-10T04:31:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[29319]: 10T04:31:18.032222+0000 mgr.y (mgr.14140) 436 : cluster [DBG] pgmap v540: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 164 active+clean; 181 MiB data, 572 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 2.2 MiB/s wr, 19 op/s 2021-02-10T04:31:18.442 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[45443]: debug 2021-02-10T04:31:18.080+0000 7f7140d73700 -1 osd.1 260 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:31:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: cluster 2021-02-10T04:31:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 2021-02-10T04:31:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 2021 2021-02-10T04:31:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: - 2021-02-10T04:31:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 02 2021-02-10T04:31:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: - 2021-02-10T04:31:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 10 2021-02-10T04:31:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: T 2021-02-10T04:31:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 04 2021-02-10T04:31:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: : 2021-02-10T04:31:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 31 2021-02-10T04:31:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: : 2021-02-10T04:31:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 17 2021-02-10T04:31:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: . 2021-02-10T04:31:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 444518 2021-02-10T04:31:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: +0000 2021-02-10T04:31:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 2021-02-10T04:31:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: osd.4 2021-02-10T04:31:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: ( 2021-02-10T04:31:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: osd 2021-02-10T04:31:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: . 2021-02-10T04:31:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 4 2021-02-10T04:31:18.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: ) 2021-02-10T04:31:18.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 143 2021-02-10T04:31:18.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: : 2021-02-10T04:31:18.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: cluster 2021-02-10T04:31:18.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 2021-02-10T04:31:18.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: [WRN] 2021-02-10T04:31:18.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 2021-02-10T04:31:18.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:18.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 2021-02-10T04:31:18.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: cluster 2021-02-10T04:31:18.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 2021-02-10T04:31:18.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 2021 2021-02-10T04:31:18.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: - 2021-02-10T04:31:18.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 02 2021-02-10T04:31:18.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: - 2021-02-10T04:31:18.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 10 2021-02-10T04:31:18.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: T 2021-02-10T04:31:18.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 04 2021-02-10T04:31:18.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: : 2021-02-10T04:31:18.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 31 2021-02-10T04:31:18.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: : 2021-02-10T04:31:18.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 17 2021-02-10T04:31:18.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: . 2021-02-10T04:31:18.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 444524 2021-02-10T04:31:18.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: +0000 2021-02-10T04:31:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 2021-02-10T04:31:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: osd.4 2021-02-10T04:31:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: ( 2021-02-10T04:31:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: osd 2021-02-10T04:31:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: . 2021-02-10T04:31:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 4 2021-02-10T04:31:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: ) 2021-02-10T04:31:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 144 2021-02-10T04:31:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: : 2021-02-10T04:31:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: cluster 2021-02-10T04:31:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 2021-02-10T04:31:18.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: [WRN] 2021-02-10T04:31:18.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 2021-02-10T04:31:18.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:18.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 2021-02-10T04:31:18.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: cluster 2021-02-10T04:31:18.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 2021-02-10T04:31:18.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 2021 2021-02-10T04:31:18.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: - 2021-02-10T04:31:18.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 02 2021-02-10T04:31:18.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: - 2021-02-10T04:31:18.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 10 2021-02-10T04:31:18.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: T 2021-02-10T04:31:18.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 04 2021-02-10T04:31:18.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: : 2021-02-10T04:31:18.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 31 2021-02-10T04:31:18.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: : 2021-02-10T04:31:18.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 17 2021-02-10T04:31:18.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: . 2021-02-10T04:31:18.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 444528 2021-02-10T04:31:18.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: +0000 2021-02-10T04:31:18.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 2021-02-10T04:31:18.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: osd.4 2021-02-10T04:31:18.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: ( 2021-02-10T04:31:18.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: osd 2021-02-10T04:31:18.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: . 2021-02-10T04:31:18.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 4 2021-02-10T04:31:18.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: ) 2021-02-10T04:31:18.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 145 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:18.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:18.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 02-10T04:31:17.444532+0000 osd.4 (osd.4) 2021-02-10T04:31:18.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 146 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:18.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:31:18.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: -10T04:31:17.835061+0000 mgr.y (mgr 2021-02-10T04:31:18.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: .14140) 435 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:31:18.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:18.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: 02-10T04:31:18.032222+0000 mgr.y (mgr 2021-02-10T04:31:18.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[35961]: .14140) 436 : cluster [DBG] pgmap v540: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 164 active+clean; 181 MiB data, 572 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 2.2 MiB/s wr, 19 op/s 2021-02-10T04:31:18.490 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:18 smithi164 conmon[29288]: cluster 2021-02-10T04:31:17.444518+0000 osd.4 (osd 2021-02-10T04:31:18.490 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:18 smithi164 conmon[29288]: .4) 143 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:18.490 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:18 smithi164 conmon[29288]: cluster 2021-02-10T04:31:17.444524+0000 osd.4 (osd.4) 144 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:18.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:18 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:31:18.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:18 smithi164 conmon[29288]: 04:31:17.444528+0000 osd.4 (osd.4) 145 : 2021-02-10T04:31:18.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:18 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:18.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:18 smithi164 conmon[29288]: cluster 2021-02-10T04:31:17.444532+0000 osd.4 2021-02-10T04:31:18.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:18 smithi164 conmon[29288]: (osd.4) 146 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:18.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:18 smithi164 conmon[29288]: audit 2021-02-10T04:31:18.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:18 smithi164 conmon[29288]: 2021-02-10T04:31:17.835061+0000 2021-02-10T04:31:18.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:18 smithi164 conmon[29288]: mgr.y (mgr.14140) 435 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:31:18.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:18 smithi164 conmon[29288]: 2021-02-10T04:31:18.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:18 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:18.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:18 smithi164 conmon[29288]: 10T04:31:18.032222+0000 mgr.y 2021-02-10T04:31:18.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:18 smithi164 conmon[29288]: (mgr.14140) 436 : cluster [DBG] 2021-02-10T04:31:18.492 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:18 smithi164 conmon[29288]: pgmap v540: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 164 active+clean; 181 MiB data, 572 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 2.2 MiB/s wr, 19 op/s 2021-02-10T04:31:18.492 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:18 smithi164 conmon[33677]: debug 2021-02-10T04:31:18.489+0000 7fbec19a1700 -1 osd.4 261 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:31:19.177 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[45443]: debug 2021-02-10T04:31:19.112+0000 7f7140d73700 -1 osd.1 261 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:31:19.177 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:18 smithi085 conmon[49106]: debug 2021-02-10T04:31:18.873+0000 7f77a3e94700 -1 osd.2 261 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:31:19.178 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: cluster 2021-02-10T04:31:19.178 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: cluster 2021-02-10T04:31:17.848522 2021-02-10T04:31:19.178 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 62 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:19.178 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: cluster 2021-02-10T04:31:17.848532+0000 osd.2 (osd.2) 63 2021-02-10T04:31:19.190 INFO:tasks.rados.rados.0.smithi085.stdout:879: done (15 left) 2021-02-10T04:31:19.190 INFO:tasks.rados.rados.0.smithi085.stdout:880: done (14 left) 2021-02-10T04:31:19.190 INFO:tasks.rados.rados.0.smithi085.stdout:881: done (13 left) 2021-02-10T04:31:19.190 INFO:tasks.rados.rados.0.smithi085.stdout:882: done (12 left) 2021-02-10T04:31:19.190 INFO:tasks.rados.rados.0.smithi085.stdout:883: rollback oid 1 current snap is 84 2021-02-10T04:31:19.190 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 1 to 82 2021-02-10T04:31:19.190 INFO:tasks.rados.rados.0.smithi085.stdout:884: delete oid 27 current snap is 84 2021-02-10T04:31:19.201 INFO:tasks.rados.rados.0.smithi085.stdout:884: done (13 left) 2021-02-10T04:31:19.202 INFO:tasks.rados.rados.0.smithi085.stdout:885: write oid 16 current snap is 84 2021-02-10T04:31:19.202 INFO:tasks.rados.rados.0.smithi085.stdout:885: seq_num 330 ranges {746310=635349,2077646=576443,3362764=183546} 2021-02-10T04:31:19.204 INFO:tasks.rados.rados.0.smithi085.stdout:885: writing smithi08575397-16 from 746310 to 1381659 tid 1 2021-02-10T04:31:19.212 INFO:tasks.rados.rados.0.smithi085.stdout:885: writing smithi08575397-16 from 2077646 to 2654089 tid 2 2021-02-10T04:31:19.219 INFO:tasks.rados.rados.0.smithi085.stdout:885: writing smithi08575397-16 from 3362764 to 3546310 tid 3 2021-02-10T04:31:19.219 INFO:tasks.rados.rados.0.smithi085.stdout:886: read oid 36 snap -1 2021-02-10T04:31:19.219 INFO:tasks.rados.rados.0.smithi085.stdout:886: expect (ObjNum 306 snap 74 seq_num 306) 2021-02-10T04:31:19.219 INFO:tasks.rados.rados.0.smithi085.stdout:887: write (excl) oid 31 current snap is 84 2021-02-10T04:31:19.219 INFO:tasks.rados.rados.0.smithi085.stdout:883: finishing rollback tid 0 to smithi08575397-1 2021-02-10T04:31:19.219 INFO:tasks.rados.rados.0.smithi085.stdout:883: finishing rollback tid 1 to smithi08575397-1 2021-02-10T04:31:19.219 INFO:tasks.rados.rados.0.smithi085.stdout:883: finishing rollback tid 2 to smithi08575397-1 2021-02-10T04:31:19.219 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 250 (ObjNum 308 snap 74 seq_num 308) dirty exists 2021-02-10T04:31:19.219 INFO:tasks.rados.rados.0.smithi085.stdout:885: finishing write tid 1 to smithi08575397-16 2021-02-10T04:31:19.219 INFO:tasks.rados.rados.0.smithi085.stdout:887: seq_num 331 ranges {763117=710598,2136128=720101,3390187=172930} 2021-02-10T04:31:19.229 INFO:tasks.rados.rados.0.smithi085.stdout:887: writing smithi08575397-31 from 763117 to 1473715 tid 1 2021-02-10T04:31:19.238 INFO:tasks.rados.rados.0.smithi085.stdout:887: writing smithi08575397-31 from 2136128 to 2856229 tid 2 2021-02-10T04:31:19.242 INFO:tasks.rados.rados.0.smithi085.stdout:887: writing smithi08575397-31 from 3390187 to 3563117 tid 3 2021-02-10T04:31:19.242 INFO:tasks.rados.rados.0.smithi085.stdout:883: done (15 left) 2021-02-10T04:31:19.242 INFO:tasks.rados.rados.0.smithi085.stdout:888: snap_remove snap 79 2021-02-10T04:31:19.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: 2021-02-10T04:31:19.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: 2021 2021-02-10T04:31:19.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: - 2021-02-10T04:31:19.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: 02 2021-02-10T04:31:19.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: - 2021-02-10T04:31:19.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: 10 2021-02-10T04:31:19.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: T 2021-02-10T04:31:19.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: 04:31:17. 2021-02-10T04:31:19.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: 848522+0000 osd.2 (osd.2) 62 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:19.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: cluster 2021-02-10T04:31:19.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: 2021-02-10T04:31 2021-02-10T04:31:19.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: :17.848532+0000 osd.2 (osd.2) 63 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:19.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: cluster 2021-02-10T04:31 2021-02-10T04:31:19.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: :17.848535+0000 osd.2 (osd.2) 64 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:19.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: 2021-02-10T04:31:19.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:31:19.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: 04:31:17.848540+0000 osd.2 (osd.2) 65 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:19.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: cluster 2021-02-10T04:31:17. 2021-02-10T04:31:19.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: 848547+0000 osd.2 (osd.2) 66 : cluster 2021-02-10T04:31:19.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:19.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:31:19.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: 31:17.848553+0000 osd.2 (osd.2) 2021-02-10T04:31:19.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: 67 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:19.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:31:19.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: :31:17.848559+0000 osd.2 (osd.2) 2021-02-10T04:31:19.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: 68 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:19.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:31:19.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: :31:17.848567+0000 osd.2 (osd.2 2021-02-10T04:31:19.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: ) 69 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:19.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:19.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: -10T04:31:18.081649+0000 osd.1 2021-02-10T04:31:19.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: (osd.1) 94 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:19.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:19.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: 02-10T04:31:18.081683 2021-02-10T04:31:19.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: +0000 osd.1 (osd.1) 95 : cluster [WRN] 2021-02-10T04:31:19.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:19.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: cluster 2021-02-10T04:31 2021-02-10T04:31:19.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: :18.081696+0000 osd.1 (osd.1) 2021-02-10T04:31:19.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: 96 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:19.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: cluster 2021-02-10T04:31:18.081702 2021-02-10T04:31:19.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: +0000 osd.1 (osd.1) 97 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:19.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: cluster 2021-02-10T04:31:18. 2021-02-10T04:31:19.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: 081707+0000 osd.1 (osd.1) 98 : cluster [WRN] 2021-02-10T04:31:19.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:19.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: cluster 2021-02-10T04:31:19.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: 2021-02-10T04:31:18.081711+0000 osd.1 (osd.1) 2021-02-10T04:31:19.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: 99 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:19.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: cluster 2021-02-10T04:31 2021-02-10T04:31:19.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: :18.181148+0000 mon.a (mon.0) 992 2021-02-10T04:31:19.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2021-02-10T04:31:19.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:19.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: T04:31:18.490417+0000 osd.4 2021-02-10T04:31:19.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: (osd.4) 147 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:19.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: 2021-02-10T04:31:19.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: cluster 2021-02-10T04:31:19.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: 2021-02-10T04:31:18.490426+0000 osd.4 ( 2021-02-10T04:31:19.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: osd.4) 148 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:19.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:19.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: -02-10T04:31:18.490432+0000 2021-02-10T04:31:19.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: osd.4 (osd.4) 149 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:19.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: cluster 2021-02-10T04:31:18 2021-02-10T04:31:19.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: .490438+0000 osd.4 (osd.4) 150 : cluster 2021-02-10T04:31:19.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:19.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:19.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: cluster 2021-02-10T04:31:19.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: 2021-02-10T04:31:17.848535+0000 osd.2 (osd.2) 2021-02-10T04:31:19.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: 64 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:19.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:19.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: -02-10T04:31:17.848540+0000 2021-02-10T04:31:19.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: osd.2 (osd.2) 65 : cluster [WRN] 2021-02-10T04:31:19.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:19.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: cluster 2021-02-10T04:31 2021-02-10T04:31:19.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: :17.848547+0000 osd.2 (osd.2) 2021-02-10T04:31:19.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: 66 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:19.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: cluster 2021-02-10T04:31:19.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: 2021 2021-02-10T04:31:19.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: -02-10T04:31:17.848553+0000 2021-02-10T04:31:19.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: osd.2 (osd.2) 67 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:19.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:19.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: -10T04:31:17.848559+0000 osd.2 (osd. 2021-02-10T04:31:19.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: 2) 68 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:19.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: cluster 2021-02-10T04:31:17.848567 2021-02-10T04:31:19.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 69 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:19.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: 2021-02-10T04:31:19.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:19.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: 04:31:18.081649+0000 osd.1 (osd.1) 94 : cluster [WRN] 2021-02-10T04:31:19.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:19.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: cluster 2021-02-10T04:31: 2021-02-10T04:31:19.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: 18.081683+0000 osd.1 (osd.1) 95 : cluster 2021-02-10T04:31:19.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:19.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:19.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: -02-10T04:31:18.081696+0000 2021-02-10T04:31:19.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: osd.1 (osd.1) 96 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:19.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: cluster 2021-02-10T04:31:19.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: 2021-02-10T04:31:18.081702+0000 2021-02-10T04:31:19.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: osd.1 (osd.1) 97 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:19.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: 2021-02-10T04:31:19.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:19.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: 02-10T04:31:18.081707+0000 osd.1 (osd.1 2021-02-10T04:31:19.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: ) 98 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:19.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:19.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: 10T04:31:18.081711+0000 osd.1 (osd.1) 99 : cluster 2021-02-10T04:31:19.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:19.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:19.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: 31:18.181148+0000 mon.a (mon.0) 992 : cluster 2021-02-10T04:31:19.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: [DBG] osdmap e261: 8 total, 8 up, 8 in 2021-02-10T04:31:19.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:19.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: T04:31:18.490417+0000 osd.4 ( 2021-02-10T04:31:19.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: osd.4) 147 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:19.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:19.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: -10T04:31:18.490426+0000 osd.4 ( 2021-02-10T04:31:19.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: osd.4) 148 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:19.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: cluster 2021-02-10T04:31:19.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: 2021-02-10T04:31 2021-02-10T04:31:19.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: :18.490432+0000 osd.4 ( 2021-02-10T04:31:19.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: osd.4) 149 : cluster [WRN] 2021-02-10T04:31:19.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:19.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: cluster 2021-02-10T04:31:19.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:31:19.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: 31:18.490438+0000 osd.4 2021-02-10T04:31:19.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: (osd.4) 150 : cluster 2021-02-10T04:31:19.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:19.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: cluster 2021-02-10T04:31:17.848522+0000 osd.2 (osd.2 2021-02-10T04:31:19.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: ) 62 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:19.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: cluster 2021-02-10T04:31:17.848532+0000 osd.2 (osd.2) 63 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:19.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: cluster 2021-02-10T04:31:17.848535+0000 osd.2 2021-02-10T04:31:19.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: (osd.2) 64 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:19.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:31:19.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: 04:31:17.848540+0000 osd.2 (osd.2) 65 : 2021-02-10T04:31:19.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:19.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: cluster 2021-02-10T04:31 2021-02-10T04:31:19.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: :17.848547+0000 osd.2 (osd.2) 66 2021-02-10T04:31:19.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:19.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:19.460 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: T04:31:17.848553+0000 osd.2 (osd 2021-02-10T04:31:19.460 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: .2) 67 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:19.460 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:19.460 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: -10T04:31:17.848559+0000 osd.2 (osd 2021-02-10T04:31:19.460 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: .2) 68 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:19.460 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:19.460 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: -10T04:31:17.848567+0000 osd.2 2021-02-10T04:31:19.460 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: (osd.2) 69 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:19.460 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:19.460 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: -10T04:31:18.081649+0000 2021-02-10T04:31:19.460 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: osd.1 (osd.1) 94 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:19.460 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: 2021-02-10T04:31:19.461 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: cluster 2021-02-10T04:31:19.461 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: 2021-02-10T04:31:18.081683+0000 osd.1 (osd.1) 95 : cluster 2021-02-10T04:31:19.461 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:19.461 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: cluster 2021-02-10T04:31: 2021-02-10T04:31:19.461 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: 18.081696+0000 osd.1 (osd.1) 96 : 2021-02-10T04:31:19.461 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:19.461 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:31:19.461 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: 31:18.081702+0000 osd.1 (osd.1 2021-02-10T04:31:19.461 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: ) 97 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:19.461 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:19.461 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: T04:31:18.081707+0000 osd.1 (osd.1) 98 : 2021-02-10T04:31:19.461 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:19.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:31:19.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: 31:18.081711+0000 osd.1 (osd.1 2021-02-10T04:31:19.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: ) 99 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:19.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:19.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: 10T04:31:18.181148+0000 mon.a 2021-02-10T04:31:19.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: (mon.0) 992 : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2021-02-10T04:31:19.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: 2021-02-10T04:31:19.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: cluster 2021-02-10T04:31:19.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: 2021-02-10T04:31:18.490417+0000 osd.4 ( 2021-02-10T04:31:19.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: osd.4) 147 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:19.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:19.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: -02-10T04:31:18.490426 2021-02-10T04:31:19.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 148 : cluster [WRN] 2021-02-10T04:31:19.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:19.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: cluster 2021-02-10T04:31: 2021-02-10T04:31:19.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: 18.490432+0000 osd.4 (osd.4) 149 : 2021-02-10T04:31:19.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:19.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:31:19.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: :31:18.490438+0000 osd.4 (osd. 2021-02-10T04:31:19.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[29288]: 4) 150 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:19.465 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[40892]: debug 2021-02-10T04:31:19.184+0000 7f30314bd700 -1 osd.6 pg_epoch: 261 pg[7.7( v 260'442 (0'0,260'442] local-lis/les=93/94 n=19 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [6,0,3] r=0 lpr=93 crt=260'442 lcod 260'440 mlcod 260'440 active+clean+snaptrim trimq=[7e~1,97~1,99~1,9e~1,a0~1] ps=[60~3,66~3,6b~3,6f~1,74~2,78~4,80~4,87~2,8b~3,8f~1,92~1]] removing snap head 2021-02-10T04:31:19.893 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:19 smithi164 conmon[33677]: debug 2021-02-10T04:31:19.457+0000 7fbec19a1700 -1 osd.4 262 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:31:20.134 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:19 smithi085 conmon[49106]: debug 2021-02-10T04:31:19.852+0000 7f77a3e94700 -1 osd.2 262 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:31:20.134 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[45443]: debug 2021-02-10T04:31:20.132+0000 7f7140d73700 -1 osd.1 262 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:31:20.205 INFO:tasks.rados.rados.0.smithi085.stdout:888: done (15 left) 2021-02-10T04:31:20.205 INFO:tasks.rados.rados.0.smithi085.stdout:889: read oid 46 snap 78 2021-02-10T04:31:20.205 INFO:tasks.rados.rados.0.smithi085.stdout:889: expect deleted 2021-02-10T04:31:20.205 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:20.205 INFO:tasks.rados.rados.0.smithi085.stdout:885: finishing write tid 2 to smithi08575397-16 2021-02-10T04:31:20.205 INFO:tasks.rados.rados.0.smithi085.stdout:886: expect (ObjNum 306 snap 74 seq_num 306) 2021-02-10T04:31:20.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: cluster 2021-02-10T04:31:20.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: 2021-02-10T04:31:20.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: 2021 2021-02-10T04:31:20.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: - 2021-02-10T04:31:20.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: 02 2021-02-10T04:31:20.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: -10T04:31: 2021-02-10T04:31:20.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: 18.615195+0000 osd.6 (osd.6) 49 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:31:20.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: cluster 2021-02-10T04:31:20.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: 2021- 2021-02-10T04:31:20.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: 02-10T04 2021-02-10T04:31:20.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: :31:18.615350+0000 osd.6 (osd 2021-02-10T04:31:20.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: .6) 50 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:31:20.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: cluster 2021-02-10T04:31:20.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: 2021-02 2021-02-10T04:31:20.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: -10T04:31:18 2021-02-10T04:31:20.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: .875426+0000 2021-02-10T04:31:20.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: osd.2 (osd. 2021-02-10T04:31:20.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: 2) 70 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:20.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: cluster 2021-02-10T04:31:20.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: 2021-02-10T04:31:18.875434+0000 2021-02-10T04:31:20.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: osd.2 (osd.2) 71 : 2021-02-10T04:31:20.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:20.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:31:20.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: 31:18.875437 2021-02-10T04:31:20.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 72 : cluster 2021-02-10T04:31:20.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:20.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:20.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: 10T04:31:18.875441+0000 osd.2 (osd.2 2021-02-10T04:31:20.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: ) 73 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:20.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: cluster 2021-02-10T04:31:20.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:31:20.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: 04:31:18.875445+0000 osd.2 (osd. 2021-02-10T04:31:20.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: 2) 74 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:20.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:20.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: 02-10T04:31:18.875449+0000 2021-02-10T04:31:20.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: osd.2 (osd.2) 75 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:20.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: 2021-02-10T04:31:20.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: cluster 2021-02-10T04:31:20.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: 2021-02-10T04:31:18.875453+0000 osd.2 2021-02-10T04:31:20.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: (osd.2) 76 : cluster [WRN] 2021-02-10T04:31:20.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:20.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: cluster 2021-02-10T04:31:18 2021-02-10T04:31:20.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: .875460+0000 osd.2 2021-02-10T04:31:20.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: (osd.2) 77 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:20.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:20.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: 02-10T04:31:19 2021-02-10T04:31:20.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: .114424+0000 osd.1 (osd.1) 100 : cluster 2021-02-10T04:31:20.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:20.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:31:20.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: 31:19.114430+0000 osd.1 (osd.1 2021-02-10T04:31:20.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: ) 101 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:20.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: cluster 2021-02-10T04:31:20.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: 2021-02-10T04:31:19.114436 2021-02-10T04:31:20.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: +0000 osd.1 (osd.1) 102 : 2021-02-10T04:31:20.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:20.456 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: cluster 2021-02-10T04:31:20.456 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: 2021-02-10T04:31:19 2021-02-10T04:31:20.456 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: .114440+0000 osd.1 (osd.1) 103 : cluster 2021-02-10T04:31:20.456 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:20.456 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:20.457 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: 10T04:31:19.114444+0000 osd.1 ( 2021-02-10T04:31:20.457 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: osd.1) 104 : cluster [WRN] 2021-02-10T04:31:20.457 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:20.457 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: cluster 2021-02-10T04:31 2021-02-10T04:31:20.457 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: :19.114448+0000 osd.1 (osd.1) 105 2021-02-10T04:31:20.457 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:20.457 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:31:20.457 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: :31:19.190388+0000 mon.a (mon.0 2021-02-10T04:31:20.457 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: ) 993 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2021-02-10T04:31:20.457 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:20.457 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: 10T04:31:19.458627+0000 osd.4 (osd 2021-02-10T04:31:20.457 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: .4) 151 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:20.457 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: 2021-02-10T04:31:20.458 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: cluster 2021-02-10T04:31:20.458 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: 2021-02-10T04:31:19.458637+0000 osd.4 (osd.4) 152 : cluster [WRN] 2021-02-10T04:31:20.458 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:20.458 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:31:20.458 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: 04:31:19.458644+0000 osd.4 (osd 2021-02-10T04:31:20.458 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: .4) 153 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:20.458 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:20.458 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: -02-10T04:31:19.458650+0000 2021-02-10T04:31:20.458 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: osd.4 (osd.4) 154 : cluster [WRN] 2021-02-10T04:31:20.458 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:20.458 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: cluster 2021-02-10T04:31:20.458 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: 2021-02-10T04:31:20.032628+0000 mgr.y 2021-02-10T04:31:20.458 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[29319]: (mgr.14140) 437 : cluster [DBG] pgmap v543: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 4 active+clean+snaptrim, 162 active+clean; 177 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 2.2 MiB/s wr, 15 op/s 2021-02-10T04:31:20.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: cluster 2021-02-10T04:31:18.615195+0000 2021-02-10T04:31:20.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: osd.6 (osd.6) 49 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:31:20.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: cluster 2021-02-10T04:31 2021-02-10T04:31:20.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: :18.615350+0000 osd.6 (osd.6) 50 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:31:20.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:20.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: 10T04:31:18.875426+0000 osd.2 (osd.2) 70 2021-02-10T04:31:20.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:20.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:20.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: 10T04:31:18.875434+0000 osd.2 ( 2021-02-10T04:31:20.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: osd.2) 71 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:20.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:20.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: 31:18.875437+0000 osd.2 (osd.2) 2021-02-10T04:31:20.460 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: 72 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:20.460 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:20.460 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: 02-10T04:31:18.875441+0000 osd.2 2021-02-10T04:31:20.460 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: (osd.2) 73 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:20.460 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:20.460 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: -10T04:31:18.875445+0000 osd.2 (osd 2021-02-10T04:31:20.460 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: .2) 74 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:20.460 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:20.460 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: 02-10T04:31:18.875449+0000 2021-02-10T04:31:20.460 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: osd.2 (osd.2) 75 : cluster [WRN] 2021-02-10T04:31:20.460 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:20.460 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: cluster 2021-02-10T04:31:20.460 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: 2021-02-10T04:31:18.875453+0000 osd.2 (osd.2) 2021-02-10T04:31:20.461 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: 76 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:20.461 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: cluster 2021-02-10T04:31:20.461 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:31:20.461 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: T04:31:18.875460+0000 osd.2 (osd 2021-02-10T04:31:20.461 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: .2) 77 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:20.461 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: cluster 2021-02-10T04:31:20.461 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: 2021-02-10T04:31:20.461 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:31:20.462 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: 04:31:19.114424+0000 osd.1 (osd.1) 100 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:20.462 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:20.462 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: -10T04:31:19.114430+0000 osd.1 ( 2021-02-10T04:31:20.462 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: osd.1) 101 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:20.462 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:20.462 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: -02-10T04:31:19.114436+0000 osd.1 2021-02-10T04:31:20.462 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: (osd.1) 102 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:20.462 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: cluster 2021-02-10T04:31:20.462 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: 2021-02-10T04:31:19.114440+0000 2021-02-10T04:31:20.462 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: osd.1 (osd.1) 103 : cluster [WRN] 2021-02-10T04:31:20.462 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:20.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: cluster 2021-02-10T04:31: 2021-02-10T04:31:20.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: 19.114444+0000 osd.1 (osd.1) 104 2021-02-10T04:31:20.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:20.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:20.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: -02-10T04:31:19.114448+0000 2021-02-10T04:31:20.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: osd.1 (osd.1) 105 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:20.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: cluster 2021-02-10T04:31:20.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: 2021-02-10T04:31:20.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: 2021-02-10T04:31 2021-02-10T04:31:20.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: :19.190388+0000 mon.a (mon.0) 993 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2021-02-10T04:31:20.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: cluster 2021-02-10T04:31:20.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: 2021-02-10T04:31 2021-02-10T04:31:20.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: :19.458627+0000 osd.4 (osd.4) 151 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:20.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:20.464 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: :31:19.458637+0000 osd.4 (osd.4) 152 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:20.464 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:20.464 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: T04:31:19.458644+0000 osd.4 (osd.4 2021-02-10T04:31:20.464 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: ) 153 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:20.464 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: cluster 2021-02-10T04:31 2021-02-10T04:31:20.464 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: :19.458650+0000 osd.4 (osd.4) 154 : cluster [WRN] 2021-02-10T04:31:20.464 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:20.464 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: cluster 2021-02-10T04:31:20. 2021-02-10T04:31:20.464 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: 032628+0000 mgr.y (mgr.14140) 437 2021-02-10T04:31:20.464 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[35961]: : cluster [DBG] pgmap v543: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 4 active+clean+snaptrim, 162 active+clean; 177 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 2.2 MiB/s wr, 15 op/s 2021-02-10T04:31:20.465 INFO:tasks.rados.rados.0.smithi085.stdout:885: finishing write tid 3 to smithi08575397-16 2021-02-10T04:31:20.465 INFO:tasks.rados.rados.0.smithi085.stdout:885: finishing write tid 4 to smithi08575397-16 2021-02-10T04:31:20.465 INFO:tasks.rados.rados.0.smithi085.stdout:886: done (15 left) 2021-02-10T04:31:20.465 INFO:tasks.rados.rados.0.smithi085.stdout:890: read oid 27 snap -1 2021-02-10T04:31:20.465 INFO:tasks.rados.rados.0.smithi085.stdout:890: expect deleted 2021-02-10T04:31:20.465 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:20.465 INFO:tasks.rados.rados.0.smithi085.stdout:885: finishing write tid 5 to smithi08575397-16 2021-02-10T04:31:20.466 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 451 (ObjNum 330 snap 84 seq_num 330) dirty exists 2021-02-10T04:31:20.466 INFO:tasks.rados.rados.0.smithi085.stdout:885: left oid 16 (ObjNum 330 snap 84 seq_num 330) 2021-02-10T04:31:20.466 INFO:tasks.rados.rados.0.smithi085.stdout:887: finishing write tid 1 to smithi08575397-31 2021-02-10T04:31:20.466 INFO:tasks.rados.rados.0.smithi085.stdout:887: finishing write tid 2 to smithi08575397-31 2021-02-10T04:31:20.466 INFO:tasks.rados.rados.0.smithi085.stdout:887: finishing write tid 3 to smithi08575397-31 2021-02-10T04:31:20.466 INFO:tasks.rados.rados.0.smithi085.stdout:887: finishing write tid 4 to smithi08575397-31 2021-02-10T04:31:20.466 INFO:tasks.rados.rados.0.smithi085.stdout:885: done (15 left) 2021-02-10T04:31:20.466 INFO:tasks.rados.rados.0.smithi085.stdout:891: delete oid 1 current snap is 84 2021-02-10T04:31:20.466 INFO:tasks.rados.rados.0.smithi085.stdout:887: finishing write tid 5 to smithi08575397-31 2021-02-10T04:31:20.466 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 143 (ObjNum 331 snap 84 seq_num 331) dirty exists 2021-02-10T04:31:20.466 INFO:tasks.rados.rados.0.smithi085.stdout:887: left oid 31 (ObjNum 331 snap 84 seq_num 331) 2021-02-10T04:31:20.467 INFO:tasks.rados.rados.0.smithi085.stdout:887: done (15 left) 2021-02-10T04:31:20.467 INFO:tasks.rados.rados.0.smithi085.stdout:889: done (14 left) 2021-02-10T04:31:20.467 INFO:tasks.rados.rados.0.smithi085.stdout:890: done (13 left) 2021-02-10T04:31:20.467 INFO:tasks.rados.rados.0.smithi085.stdout:891: done (12 left) 2021-02-10T04:31:20.467 INFO:tasks.rados.rados.0.smithi085.stdout:892: copy_from oid 3 from oid 37 current snap is 84 2021-02-10T04:31:20.467 INFO:tasks.rados.rados.0.smithi085.stdout:893: write (excl) oid 40 current snap is 84 2021-02-10T04:31:20.467 INFO:tasks.rados.rados.0.smithi085.stdout:893: seq_num 333 ranges {796731=461096,1736739=723770,2928680=668051} 2021-02-10T04:31:20.473 INFO:tasks.rados.rados.0.smithi085.stdout:893: writing smithi08575397-40 from 796731 to 1257827 tid 1 2021-02-10T04:31:20.481 INFO:tasks.rados.rados.0.smithi085.stdout:893: writing smithi08575397-40 from 1736739 to 2460509 tid 2 2021-02-10T04:31:20.489 INFO:tasks.rados.rados.0.smithi085.stdout:893: writing smithi08575397-40 from 2928680 to 3596731 tid 3 2021-02-10T04:31:20.489 INFO:tasks.rados.rados.0.smithi085.stdout:894: copy_from oid 8 from oid 11 current snap is 84 2021-02-10T04:31:20.489 INFO:tasks.rados.rados.0.smithi085.stdout:895: rollback oid 50 current snap is 84 2021-02-10T04:31:20.489 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 50 to 74 2021-02-10T04:31:20.489 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:20.489 INFO:tasks.rados.rados.0.smithi085.stdout:893: finishing write tid 1 to smithi08575397-40 2021-02-10T04:31:20.489 INFO:tasks.rados.rados.0.smithi085.stdout:893: finishing write tid 2 to smithi08575397-40 2021-02-10T04:31:20.491 INFO:tasks.rados.rados.0.smithi085.stdout:895: finishing rollback tid 1 to smithi08575397-50 2021-02-10T04:31:20.491 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-02-10T04:31:20.491 INFO:tasks.rados.rados.0.smithi085.stdout:895: done (15 left) 2021-02-10T04:31:20.491 INFO:tasks.rados.rados.0.smithi085.stdout:896: write oid 6 current snap is 84 2021-02-10T04:31:20.491 INFO:tasks.rados.rados.0.smithi085.stdout:896: seq_num 334 ranges {413538=536345,1395221=583781,2556103=658617} 2021-02-10T04:31:20.497 INFO:tasks.rados.rados.0.smithi085.stdout:896: writing smithi08575397-6 from 413538 to 949883 tid 1 2021-02-10T04:31:20.503 INFO:tasks.rados.rados.0.smithi085.stdout:896: writing smithi08575397-6 from 1395221 to 1979002 tid 2 2021-02-10T04:31:20.511 INFO:tasks.rados.rados.0.smithi085.stdout:896: writing smithi08575397-6 from 2556103 to 3214720 tid 3 2021-02-10T04:31:20.511 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:20.511 INFO:tasks.rados.rados.0.smithi085.stdout:894: finishing copy_from to smithi08575397-8 2021-02-10T04:31:20.511 INFO:tasks.rados.rados.0.smithi085.stdout:894: got expected ENOENT (src dne) 2021-02-10T04:31:20.511 INFO:tasks.rados.rados.0.smithi085.stdout:894: finishing copy_from racing read to smithi08575397-8 2021-02-10T04:31:20.511 INFO:tasks.rados.rados.0.smithi085.stdout:893: finishing write tid 3 to smithi08575397-40 2021-02-10T04:31:20.511 INFO:tasks.rados.rados.0.smithi085.stdout:893: finishing write tid 4 to smithi08575397-40 2021-02-10T04:31:20.511 INFO:tasks.rados.rados.0.smithi085.stdout:894: done (15 left) 2021-02-10T04:31:20.511 INFO:tasks.rados.rados.0.smithi085.stdout:897: delete oid 46 current snap is 84 2021-02-10T04:31:20.511 INFO:tasks.rados.rados.0.smithi085.stdout:893: finishing write tid 5 to smithi08575397-40 2021-02-10T04:31:20.511 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 456 (ObjNum 333 snap 84 seq_num 333) dirty exists 2021-02-10T04:31:20.511 INFO:tasks.rados.rados.0.smithi085.stdout:893: left oid 40 (ObjNum 333 snap 84 seq_num 333) 2021-02-10T04:31:20.512 INFO:tasks.rados.rados.0.smithi085.stdout:896: finishing write tid 1 to smithi08575397-6 2021-02-10T04:31:20.512 INFO:tasks.rados.rados.0.smithi085.stdout:896: finishing write tid 2 to smithi08575397-6 2021-02-10T04:31:20.514 INFO:tasks.rados.rados.0.smithi085.stdout:896: finishing write tid 3 to smithi08575397-6 2021-02-10T04:31:20.514 INFO:tasks.rados.rados.0.smithi085.stdout:896: finishing write tid 4 to smithi08575397-6 2021-02-10T04:31:20.515 INFO:tasks.rados.rados.0.smithi085.stdout:896: finishing write tid 5 to smithi08575397-6 2021-02-10T04:31:20.515 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 280 (ObjNum 334 snap 84 seq_num 334) dirty exists 2021-02-10T04:31:20.515 INFO:tasks.rados.rados.0.smithi085.stdout:896: left oid 6 (ObjNum 334 snap 84 seq_num 334) 2021-02-10T04:31:20.515 INFO:tasks.rados.rados.0.smithi085.stdout:893: done (15 left) 2021-02-10T04:31:20.515 INFO:tasks.rados.rados.0.smithi085.stdout:896: done (14 left) 2021-02-10T04:31:20.515 INFO:tasks.rados.rados.0.smithi085.stdout:897: done (13 left) 2021-02-10T04:31:20.515 INFO:tasks.rados.rados.0.smithi085.stdout:898: snap_remove snap 78 2021-02-10T04:31:20.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[33677]: debug 2021-02-10T04:31:20.409+0000 7fbec19a1700 -1 osd.4 263 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:31:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: cluster 2021-02-10T04:31:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:31:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: 31:18.615195+0000 osd.6 (osd.6) 49 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:31:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:31:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: 31:18.615350+0000 osd.6 (osd.6) 50 : 2021-02-10T04:31:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:31:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: -02-10T04:31:18. 2021-02-10T04:31:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: 875426+0000 osd.2 (osd.2 2021-02-10T04:31:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: ) 70 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: cluster 2021-02-10T04:31:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: 2021-02-10T04:31:18.875434+0000 osd.2 2021-02-10T04:31:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: (osd.2) 71 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: cluster 2021-02-10T04:31 2021-02-10T04:31:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: :18.875437+0000 osd.2 (osd.2) 72 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: 2021-02-10T04:31:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: 10T04:31:18.875441+0000 osd.2 (osd.2) 73 : cluster 2021-02-10T04:31:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: cluster 2021-02-10T04:31 2021-02-10T04:31:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: :18.875445+0000 osd.2 (osd.2) 74 2021-02-10T04:31:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: cluster 2021-02-10T04:31:18 2021-02-10T04:31:20.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: .875449+0000 osd.2 (osd.2) 75 : cluster [WRN] 2021-02-10T04:31:20.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:20.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:20.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: T04:31:18.875453+0000 osd.2 (osd.2) 76 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:20.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:20.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: 02-10T04:31:18.875460+0000 osd.2 2021-02-10T04:31:20.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: (osd.2) 77 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:20.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: cluster 2021-02-10T04:31: 2021-02-10T04:31:20.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: 19.114424+0000 osd.1 (osd.1) 100 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:20.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:20.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: 10T04:31:19.114430+0000 osd.1 (osd.1) 101 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:20.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:31:20.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: :31:19.114436+0000 osd.1 (osd.1 2021-02-10T04:31:20.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: ) 102 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:20.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: cluster 2021-02-10T04:31:19.114440+0000 osd.1 ( 2021-02-10T04:31:20.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: osd.1) 103 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:20.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: cluster 2021-02-10T04:31:19.114444+0000 osd.1 (osd.1) 104 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:20.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: 2021-02-10T04:31:20.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:31:20.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: 04:31:19.114448+0000 osd.1 (osd.1) 105 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:20.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: cluster 2021-02-10T04:31:19.190388+0000 2021-02-10T04:31:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: mon.a (mon.0) 993 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2021-02-10T04:31:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: 2021-02-10T04:31:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: -02-10T04:31:19.458627+0000 2021-02-10T04:31:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: osd.4 (osd.4) 151 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: 2021-02-10T04:31:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: cluster 2021-02-10T04:31:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: 2021-02-10T04:31:19.458637 2021-02-10T04:31:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 152 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: 2021-02-10T04:31:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: cluster 2021-02-10T04:31:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: 2021-02-10T04:31 2021-02-10T04:31:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: :19.458644+0000 osd.4 (osd.4) 153 2021-02-10T04:31:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:20.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:20.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: -10T04:31:19.458650+0000 osd.4 2021-02-10T04:31:20.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: (osd.4) 154 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:20.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:20.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: -02-10T04:31:20.032628 2021-02-10T04:31:20.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:20 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 437 : cluster [DBG] pgmap v543: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 4 active+clean+snaptrim, 162 active+clean; 177 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 2.2 MiB/s wr, 15 op/s 2021-02-10T04:31:21.146 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:20 smithi085 conmon[49106]: debug 2021-02-10T04:31:20.864+0000 7f77a3e94700 -1 osd.2 263 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:31:21.305 INFO:tasks.rados.rados.0.smithi085.stdout:898: done (13 left) 2021-02-10T04:31:21.306 INFO:tasks.rados.rados.0.smithi085.stdout:899: write oid 1 current snap is 84 2021-02-10T04:31:21.306 INFO:tasks.rados.rados.0.smithi085.stdout:892: finishing copy_from to smithi08575397-3 2021-02-10T04:31:21.306 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 438 (ObjNum 326 snap 83 seq_num 326) dirty exists 2021-02-10T04:31:21.306 INFO:tasks.rados.rados.0.smithi085.stdout:899: seq_num 336 ranges {447152=403196,1512185=587450,2727302=681161} 2021-02-10T04:31:21.306 INFO:tasks.rados.rados.0.smithi085.stdout:899: writing smithi08575397-1 from 447152 to 850348 tid 1 2021-02-10T04:31:21.307 INFO:tasks.rados.rados.0.smithi085.stdout:899: writing smithi08575397-1 from 1512185 to 2099635 tid 2 2021-02-10T04:31:21.307 INFO:tasks.rados.rados.0.smithi085.stdout:899: writing smithi08575397-1 from 2727302 to 3408463 tid 3 2021-02-10T04:31:21.307 INFO:tasks.rados.rados.0.smithi085.stdout:900: read oid 40 snap 82 2021-02-10T04:31:21.307 INFO:tasks.rados.rados.0.smithi085.stdout:900: expect (ObjNum 280 snap 70 seq_num 280) 2021-02-10T04:31:21.307 INFO:tasks.rados.rados.0.smithi085.stdout:901: snap_remove snap 83 2021-02-10T04:31:21.307 INFO:tasks.rados.rados.0.smithi085.stdout:892: finishing copy_from racing read to smithi08575397-3 2021-02-10T04:31:21.307 INFO:tasks.rados.rados.0.smithi085.stdout:899: finishing write tid 1 to smithi08575397-1 2021-02-10T04:31:21.308 INFO:tasks.rados.rados.0.smithi085.stdout:899: finishing write tid 2 to smithi08575397-1 2021-02-10T04:31:21.441 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[52836]: debug 2021-02-10T04:31:21.258+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:21.441 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[45443]: debug 2021-02-10T04:31:21.441 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[45443]: 2021-02-10T04:31:21.144+0000 7f7140d73700 -1 osd.1 263 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:31:21.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: cluster 2021-02-10T04:31:19.853962+0000 osd.2 2021-02-10T04:31:21.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: (osd.2) 78 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:21.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: cluster 2021-02-10T04:31:19.853970+0000 osd.2 (osd.2) 79 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:21.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: cluster 2021-02-10T04:31:19.853981+0000 osd.2 (osd.2) 80 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:21.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: cluster 2021-02-10T04:31:19.853988+0000 osd.2 (osd.2) 81 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:21.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: cluster 2021-02-10T04:31 2021-02-10T04:31:21.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: :19.853995+0000 osd.2 (osd.2) 82 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:21.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: cluster 2021-02-10T04:31 2021-02-10T04:31:21.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: :19.854001+0000 osd.2 (osd.2) 83 : cluster [WRN] 2021-02-10T04:31:21.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:21.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:21.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: -02-10T04:31:19. 2021-02-10T04:31:21.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: 854007+0000 osd.2 (osd.2 2021-02-10T04:31:21.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: ) 84 : cluster 2021-02-10T04:31:21.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:21.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:21.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: 10T04:31:19.854018+0000 osd.2 (osd.2) 85 : cluster [WRN] 2021-02-10T04:31:21.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:21.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:21.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: -10T04:31:20.133857+0000 2021-02-10T04:31:21.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: osd.1 (osd.1) 106 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:21.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: 2021-02-10T04:31:21.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:21.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: 02-10T04:31:20.133869+0000 osd.1 ( 2021-02-10T04:31:21.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: osd.1) 107 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:21.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:21.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: 02-10T04:31:20 2021-02-10T04:31:21.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: .133899+0000 osd.1 (osd.1) 108 : cluster 2021-02-10T04:31:21.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:21.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:21.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: 02-10T04:31:20.133906+0000 2021-02-10T04:31:21.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: osd.1 (osd.1) 109 : cluster [WRN] 2021-02-10T04:31:21.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:21.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: cluster 2021-02-10T04:31:21.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: 2021-02-10T04:31:20.133912+0000 osd.1 ( 2021-02-10T04:31:21.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: osd.1) 110 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:21.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:21.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: -10T04:31:20.133918+0000 osd.1 2021-02-10T04:31:21.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: (osd.1) 111 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:21.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:21.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: -02-10T04:31:20.201200+0000 mon.a (mon.0) 994 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2021-02-10T04:31:21.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: cluster 2021-02-10T04:31:20.410213 2021-02-10T04:31:21.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 155 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:21.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: 2021-02-10T04:31:21.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:31:21.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: 04:31:20.410223+0000 osd.4 (osd.4) 156 : 2021-02-10T04:31:21.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:21.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:21.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: 02-10T04:31:20.410229+0000 2021-02-10T04:31:21.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: osd.4 (osd.4) 157 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:21.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:31:21.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[29319]: 04:31:20.410235+0000 osd.4 (osd.4) 158 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:21.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: cluster 2021-02-10T04:31:21.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:31:21.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 31:19.853962+0000 osd.2 (osd.2) 78 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:21.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: cluster 2021-02-10T04:31:21.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 2021-02-10T04:31:21.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 2021 2021-02-10T04:31:21.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: - 2021-02-10T04:31:21.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 02 2021-02-10T04:31:21.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: - 2021-02-10T04:31:21.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 10 2021-02-10T04:31:21.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: T 2021-02-10T04:31:21.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 04 2021-02-10T04:31:21.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: : 2021-02-10T04:31:21.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 31 2021-02-10T04:31:21.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: : 2021-02-10T04:31:21.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 19 2021-02-10T04:31:21.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: . 2021-02-10T04:31:21.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 853970 2021-02-10T04:31:21.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: +0000 2021-02-10T04:31:21.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 2021-02-10T04:31:21.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: osd.2 2021-02-10T04:31:21.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: ( 2021-02-10T04:31:21.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: osd 2021-02-10T04:31:21.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: . 2021-02-10T04:31:21.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 2 2021-02-10T04:31:21.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: ) 2021-02-10T04:31:21.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 79 2021-02-10T04:31:21.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: : 2021-02-10T04:31:21.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: cluster 2021-02-10T04:31:21.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 2021-02-10T04:31:21.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: [WRN] 2021-02-10T04:31:21.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 2021-02-10T04:31:21.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:21.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 2021-02-10T04:31:21.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: cluster 2021-02-10T04:31:21.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 2021-02-10T04:31:21.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 2021 2021-02-10T04:31:21.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: - 2021-02-10T04:31:21.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 02 2021-02-10T04:31:21.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: - 2021-02-10T04:31:21.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 10 2021-02-10T04:31:21.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: T 2021-02-10T04:31:21.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 04 2021-02-10T04:31:21.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: : 2021-02-10T04:31:21.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 31 2021-02-10T04:31:21.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: : 2021-02-10T04:31:21.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 19 2021-02-10T04:31:21.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: . 2021-02-10T04:31:21.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 853981 2021-02-10T04:31:21.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: +0000 2021-02-10T04:31:21.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 2021-02-10T04:31:21.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: osd.2 2021-02-10T04:31:21.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: ( 2021-02-10T04:31:21.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: osd 2021-02-10T04:31:21.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: . 2021-02-10T04:31:21.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 2 2021-02-10T04:31:21.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: ) 2021-02-10T04:31:21.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 80 2021-02-10T04:31:21.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: : 2021-02-10T04:31:21.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: cluster 2021-02-10T04:31:21.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 2021-02-10T04:31:21.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: [WRN] 2021-02-10T04:31:21.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 2021-02-10T04:31:21.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:21.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 2021-02-10T04:31:21.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: cluster 2021-02-10T04:31:21.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 2021-02-10T04:31:21.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 2021 2021-02-10T04:31:21.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: - 2021-02-10T04:31:21.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 02 2021-02-10T04:31:21.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: - 2021-02-10T04:31:21.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 10 2021-02-10T04:31:21.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: T 2021-02-10T04:31:21.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 04 2021-02-10T04:31:21.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: : 2021-02-10T04:31:21.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 31:19.853988+0000 osd.2 (osd 2021-02-10T04:31:21.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: .2) 81 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:21.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: cluster 2021-02-10T04:31 2021-02-10T04:31:21.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: :19.853995+0000 osd.2 (osd.2) 82 : cluster [WRN] 2021-02-10T04:31:21.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:21.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: cluster 2021-02-10T04:31:21.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 2021-02-10T04:31:19.854001+0000 osd.2 (osd. 2021-02-10T04:31:21.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 2) 83 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:21.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:21.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 31:19.854007+0000 osd.2 (osd. 2021-02-10T04:31:21.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 2) 84 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:21.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 2021-02-10T04:31:21.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:21.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: -10T04:31:19. 2021-02-10T04:31:21.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 854018+0000 osd.2 (osd.2) 85 : 2021-02-10T04:31:21.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:21.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:21.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: :31:20.133857+0000 osd.1 (osd.1 2021-02-10T04:31:21.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: ) 106 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:21.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:21.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 10T04:31:20.133869+0000 osd.1 ( 2021-02-10T04:31:21.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: osd.1) 107 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:21.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: cluster 2021-02-10T04:31:21.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:31:21.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: :31:20.133899+0000 2021-02-10T04:31:21.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: osd.1 (osd.1) 108 : 2021-02-10T04:31:21.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:21.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:21.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: T04:31:20.133906+0000 osd.1 ( 2021-02-10T04:31:21.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: osd.1) 109 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:21.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: cluster 2021-02-10T04:31:20. 2021-02-10T04:31:21.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 133912+0000 osd.1 (osd.1) 110 : cluster [WRN] 2021-02-10T04:31:21.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:21.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: cluster 2021-02-10T04:31:20.133918 2021-02-10T04:31:21.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 111 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:21.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 2021-02-10T04:31:21.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: cluster 2021-02-10T04:31:20.201200+0000 2021-02-10T04:31:21.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: mon.a (mon.0) 994 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2021-02-10T04:31:21.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: cluster 2021-02-10T04:31:21.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 2021-02-10T04:31:21.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 2021 2021-02-10T04:31:21.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: - 2021-02-10T04:31:21.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 02 2021-02-10T04:31:21.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: - 2021-02-10T04:31:21.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 10 2021-02-10T04:31:21.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: T 2021-02-10T04:31:21.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 04 2021-02-10T04:31:21.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: : 2021-02-10T04:31:21.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 31 2021-02-10T04:31:21.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: : 2021-02-10T04:31:21.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 20 2021-02-10T04:31:21.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: . 2021-02-10T04:31:21.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 410213 2021-02-10T04:31:21.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: +0000 osd.4 (osd.4 2021-02-10T04:31:21.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: ) 155 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:21.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:21.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: -10T04:31:20.410223+0000 osd.4 ( 2021-02-10T04:31:21.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: osd.4) 156 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:21.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:21.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: -02-10T04:31:20.410229+0000 osd.4 2021-02-10T04:31:21.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: (osd.4) 157 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:21.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 2021-02-10T04:31:21.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:21.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: 31:20.410235+0000 osd.4 (osd.4 2021-02-10T04:31:21.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[35961]: ) 158 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:21.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[33677]: debug 2021-02-10T04:31:21.454+0000 7fbec19a1700 -1 osd.4 264 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:31:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: cluster 2021-02-10T04:31:19.853962+0000 osd.2 (osd.2) 78 2021-02-10T04:31:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: cluster 2021-02-10T04:31:19.853970+0000 osd.2 (osd.2) 79 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: cluster 2021-02-10T04:31:19.853981+0000 osd.2 (osd.2) 80 : cluster 2021-02-10T04:31:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:31:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: 31:19.853988+0000 osd.2 (osd.2) 81 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: cluster 2021-02-10T04:31:19 2021-02-10T04:31:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: .853995+0000 osd.2 (osd.2) 82 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:31:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: :31:19.854001+0000 osd.2 (osd.2) 2021-02-10T04:31:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: 83 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:31:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: :31:19.854007+0000 osd.2 (osd.2) 84 : 2021-02-10T04:31:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: 02-10T04:31:19.854018+0000 osd.2 (osd.2) 85 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: cluster 2021-02-10T04:31:20.133857+0000 osd.1 ( 2021-02-10T04:31:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: osd.1) 106 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: cluster 2021-02-10T04:31:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: 2021-02-10T04:31:20.133869+0000 osd.1 (osd.1) 107 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: 2021-02-10T04:31:21.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: cluster 2021-02-10T04:31:20 2021-02-10T04:31:21.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: .133899+0000 osd.1 (osd.1) 108 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:21.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: cluster 2021-02-10T04:31:21.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: 2021-02-10T04:31:20.133906+0000 osd.1 (osd.1) 109 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:21.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: cluster 2021-02-10T04:31 2021-02-10T04:31:21.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: :20.133912+0000 osd.1 (osd.1) 110 : cluster 2021-02-10T04:31:21.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:21.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: cluster 2021-02-10T04:31:20.133918 2021-02-10T04:31:21.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 111 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:21.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: 2021-02-10T04:31:21.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: cluster 2021-02-10T04:31:20. 2021-02-10T04:31:21.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: 201200+0000 mon.a (mon.0) 994 : cluster [DBG] 2021-02-10T04:31:21.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: osdmap e263: 8 total, 8 up, 8 in 2021-02-10T04:31:21.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: cluster 2021-02-10T04:31:21.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: 2021-02-10T04:31:20.410213+0000 osd.4 (osd 2021-02-10T04:31:21.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: .4) 155 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:21.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:31:21.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: 31:20.410223+0000 osd.4 (osd.4) 156 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:21.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:21.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: T04:31:20.410229+0000 osd.4 (osd 2021-02-10T04:31:21.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: .4) 157 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:21.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: cluster 2021-02-10T04:31:21.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: 2021-02-10T04:31:20. 2021-02-10T04:31:21.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: 410235+0000 osd.4 (osd.4) 158 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:21.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:21 smithi164 conmon[29288]: 2021-02-10T04:31:21.935 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:31:22.168 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:21 smithi085 conmon[49106]: debug 2021-02-10T04:31:21.909+0000 7f77a3e94700 -1 osd.2 264 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:31:22.169 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[45443]: debug 2021-02-10T04:31:22.167+0000 7f7140d73700 -1 osd.1 264 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:31:22.220 INFO:tasks.rados.rados.0.smithi085.stdout:892: done (15 left) 2021-02-10T04:31:22.220 INFO:tasks.rados.rados.0.smithi085.stdout:901: done (14 left) 2021-02-10T04:31:22.221 INFO:tasks.rados.rados.0.smithi085.stdout:902: snap_remove snap 74 2021-02-10T04:31:22.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: cluster 2021-02-10T04:31:20.865659+0000 osd.2 2021-02-10T04:31:22.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: (osd.2) 86 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:22.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: cluster 2021-02-10T04:31:22.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: 2021-02-10T04:31:20.865672+0000 osd.2 (osd.2) 2021-02-10T04:31:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: 87 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: -10T04:31:20.865678+0000 osd.2 (osd.2 2021-02-10T04:31:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: ) 88 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: 10T04:31:20.865684+0000 osd.2 (osd 2021-02-10T04:31:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: .2) 89 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: -10T04:31:20.865691+0000 osd.2 ( 2021-02-10T04:31:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: osd.2) 90 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: cluster 2021-02-10T04:31:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: 2021-02-10T04:31:20.865696+0000 2021-02-10T04:31:22.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: osd.2 (osd.2) 91 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:22.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: 2021-02-10T04:31:22.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:22.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: -02-10T04:31:20.865699+0000 osd.2 (osd. 2021-02-10T04:31:22.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: 2) 92 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:22.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:31:22.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: 04:31:20.865718+0000 osd.2 (osd.2) 2021-02-10T04:31:22.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: 93 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:22.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:31:22.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: 31:21.146165+0000 osd.1 (osd.1) 2021-02-10T04:31:22.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: 112 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:22.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:31:22.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: :31:21.146176+0000 osd.1 (osd.1) 113 2021-02-10T04:31:22.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:22.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:22.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: 02-10T04:31:21 2021-02-10T04:31:22.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: .146184+0000 osd.1 (osd.1 2021-02-10T04:31:22.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: ) 114 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:22.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:22.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: T04:31:21.146191+0000 osd.1 (osd.1) 115 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:22.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: cluster 2021-02-10T04:31:21.146197+0000 osd.1 2021-02-10T04:31:22.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: (osd.1) 116 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:22.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:22.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: 02-10T04:31:21.146203+0000 2021-02-10T04:31:22.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: osd.1 (osd.1) 117 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:22.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: 2021-02-10T04:31:22.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: cluster 2021-02-10T04:31: 2021-02-10T04:31:22.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: 21.205641+0000 mon.a (mon.0) 995 : 2021-02-10T04:31:22.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2021-02-10T04:31:22.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:22.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: T04:31:21.455232+0000 osd.4 (osd 2021-02-10T04:31:22.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: .4) 159 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:22.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:22.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: 02-10T04:31:21.455240+0000 osd.4 ( 2021-02-10T04:31:22.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: osd.4) 160 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:22.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: cluster 2021-02-10T04:31:22.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: 2021-02-10T04:31:21.455246 2021-02-10T04:31:22.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 161 : cluster [WRN] 2021-02-10T04:31:22.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:22.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: cluster 2021-02-10T04:31: 2021-02-10T04:31:22.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: 21.455250+0000 osd.4 (osd.4) 162 : 2021-02-10T04:31:22.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:22.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:31:22.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: :31:22.033144+0000 mgr.y (mgr.14140 2021-02-10T04:31:22.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[29319]: ) 438 : cluster [DBG] pgmap v546: 168 pgs: 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 158 active+clean; 170 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 29 MiB/s rd, 5.3 MiB/s wr, 38 op/s 2021-02-10T04:31:22.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: cluster 2021-02-10T04:31:20.865659 2021-02-10T04:31:22.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 86 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:22.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:22.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: 02-10T04:31:20.865672+0000 osd.2 (osd.2) 87 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:22.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:22.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: :31:20.865678+0000 osd.2 (osd.2 2021-02-10T04:31:22.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: ) 88 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:22.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:22.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: 10T04:31:20.865684+0000 2021-02-10T04:31:22.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: osd.2 (osd 2021-02-10T04:31:22.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: .2) 89 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:22.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:22.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: 02-10T04:31:20.865691+0000 2021-02-10T04:31:22.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: osd.2 (osd.2) 90 : cluster [WRN] 2021-02-10T04:31:22.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:22.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: cluster 2021-02-10T04:31:22.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: 2021-02-10T04:31:20.865696+0000 osd.2 ( 2021-02-10T04:31:22.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: osd.2) 91 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:22.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: cluster 2021-02-10T04:31:20.865699+0000 osd.2 2021-02-10T04:31:22.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: (osd.2) 92 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:22.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:22.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: 10T04:31:20.865718+0000 osd.2 2021-02-10T04:31:22.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: (osd.2) 93 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:22.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: 2021-02-10T04:31:22.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:22.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: -02-10T04:31:21.146165+0000 osd.1 (osd 2021-02-10T04:31:22.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: .1) 112 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:22.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:22.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: 10T04:31:21.146176+0000 2021-02-10T04:31:22.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: osd.1 (osd.1) 113 : cluster [WRN] 2021-02-10T04:31:22.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:22.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: cluster 2021-02-10T04:31:22.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: 2021-02-10T04:31:21.146184+0000 osd.1 (osd 2021-02-10T04:31:22.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: .1) 114 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:22.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:22.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: -10T04:31:21.146191+0000 osd.1 2021-02-10T04:31:22.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: (osd.1) 115 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:22.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:22.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: -10T04:31:21.146197+0000 2021-02-10T04:31:22.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: osd.1 (osd.1) 116 : cluster [WRN] 2021-02-10T04:31:22.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:22.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: cluster 2021-02-10T04:31:22.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: 2021-02-10T04:31:21.146203+0000 osd.1 (osd. 2021-02-10T04:31:22.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: 1) 117 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:22.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:22.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: T04:31:21.205641+0000 mon.a ( 2021-02-10T04:31:22.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: mon.0) 995 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2021-02-10T04:31:22.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:22.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: -10T04:31:21.455232+0000 2021-02-10T04:31:22.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: osd.4 (osd.4) 159 : cluster [WRN] 2021-02-10T04:31:22.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:22.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: cluster 2021-02-10T04:31:22.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: 2021-02-10T04:31:21.455240+0000 osd.4 ( 2021-02-10T04:31:22.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: osd.4) 160 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:22.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:22.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: -10T04:31:21.455246+0000 2021-02-10T04:31:22.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: osd.4 (osd.4) 161 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:22.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: 2021-02-10T04:31:22.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:22.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: -02-10T04:31:21.455250+0000 osd.4 (osd 2021-02-10T04:31:22.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: .4) 162 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:22.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:22.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: 10T04:31:22.033144+0000 mgr.y 2021-02-10T04:31:22.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[35961]: (mgr.14140) 438 : cluster [DBG] pgmap v546: 168 pgs: 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 158 active+clean; 170 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 29 MiB/s rd, 5.3 MiB/s wr, 38 op/s 2021-02-10T04:31:22.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[33677]: debug 2021-02-10T04:31:22.413+0000 7fbec19a1700 -1 osd.4 265 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:31:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: cluster 2021-02-10T04:31:20.865659+0000 osd.2 (osd.2 2021-02-10T04:31:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: ) 86 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: cluster 2021-02-10T04:31:20.865672+0000 osd.2 (osd.2) 87 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: cluster 2021-02-10T04:31:20.865678+0000 2021-02-10T04:31:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: osd.2 (osd.2) 88 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:31:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: :31:20.865684+0000 osd.2 (osd.2) 89 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: cluster 2021-02-10T04:31:20.865691+0000 2021-02-10T04:31:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: osd.2 (osd.2) 90 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: 02-10T04:31:20.865696+0000 osd.2 (osd 2021-02-10T04:31:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: .2) 91 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: cluster 2021-02-10T04:31:20.865699+0000 osd.2 (osd.2 2021-02-10T04:31:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: ) 92 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: cluster 2021-02-10T04:31:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: 2021-02-10T04:31:20.865718+0000 osd.2 (osd.2) 93 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: cluster 2021-02-10T04:31: 2021-02-10T04:31:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: 21.146165+0000 osd.1 (osd.1) 112 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: cluster 2021-02-10T04:31:21 2021-02-10T04:31:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: .146176+0000 osd.1 (osd.1) 113 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: cluster 2021-02-10T04:31:21.146184 2021-02-10T04:31:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 114 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: cluster 2021-02-10T04:31 2021-02-10T04:31:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: :21.146191+0000 osd.1 (osd.1) 115 : cluster [WRN] 2021-02-10T04:31:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: cluster 2021-02-10T04:31:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: 2021-02-10T04:31:21.146197+0000 osd.1 (osd.1) 116 : cluster [WRN] 2021-02-10T04:31:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: cluster 2021-02-10T04:31:21 2021-02-10T04:31:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: .146203+0000 osd.1 (osd.1) 117 : cluster 2021-02-10T04:31:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:31:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: 04:31:21.205641+0000 mon.a (mon. 2021-02-10T04:31:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: 0) 995 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2021-02-10T04:31:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: -02-10T04:31:21.455232+0000 osd.4 (osd 2021-02-10T04:31:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: .4) 159 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: cluster 2021-02-10T04:31:21.455240+0000 osd.4 (osd 2021-02-10T04:31:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: .4) 160 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: cluster 2021-02-10T04:31:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: 2021-02-10T04:31:21.455246+0000 osd.4 (osd.4) 161 : cluster [WRN] 2021-02-10T04:31:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:31:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: 31:21.455250+0000 osd.4 (osd.4) 162 : cluster [WRN] 2021-02-10T04:31:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: cluster 2021-02-10T04:31:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: 2021-02-10T04:31:22.033144+0000 mgr.y ( 2021-02-10T04:31:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:22 smithi164 conmon[29288]: mgr.14140) 438 : cluster [DBG] pgmap v546: 168 pgs: 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 158 active+clean; 170 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 29 MiB/s rd, 5.3 MiB/s wr, 38 op/s 2021-02-10T04:31:23.079 INFO:tasks.rados.rados.0.smithi085.stdout:902: done (14 left) 2021-02-10T04:31:23.079 INFO:tasks.rados.rados.0.smithi085.stdout:903: snap_create 2021-02-10T04:31:23.079 INFO:tasks.rados.rados.0.smithi085.stdout:899: finishing write tid 3 to smithi08575397-1 2021-02-10T04:31:23.079 INFO:tasks.rados.rados.0.smithi085.stdout:899: finishing write tid 4 to smithi08575397-1 2021-02-10T04:31:23.079 INFO:tasks.rados.rados.0.smithi085.stdout:899: finishing write tid 5 to smithi08575397-1 2021-02-10T04:31:23.079 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 255 (ObjNum 336 snap 84 seq_num 336) dirty exists 2021-02-10T04:31:23.079 INFO:tasks.rados.rados.0.smithi085.stdout:899: left oid 1 (ObjNum 336 snap 84 seq_num 336) 2021-02-10T04:31:23.079 INFO:tasks.rados.rados.0.smithi085.stdout:900: expect (ObjNum 280 snap 70 seq_num 280) 2021-02-10T04:31:23.191 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[45443]: debug 2021-02-10T04:31:23.191 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[45443]: 2021-02-10T04:31:23.169+0000 7f7140d73700 -1 osd.1 266 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:31:23.192 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:22 smithi085 conmon[49106]: debug 2021-02-10T04:31:22.923+0000 7f77a3e94700 -1 osd.2 265 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:31:23.517 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: cluster 2021-02-10T04:31:21.911409+0000 osd.2 (osd 2021-02-10T04:31:23.517 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: .2) 94 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:23.517 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: cluster 2021-02-10T04:31:21.911414+0000 osd.2 (osd.2) 95 : 2021-02-10T04:31:23.517 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:23.517 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:23.517 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: 10T04:31:21.911419+0000 osd.2 (osd.2) 96 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:23.517 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: 2021-02-10T04:31:23.517 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:23.517 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: T04:31:21.911422+0000 osd.2 (osd.2) 97 : cluster [WRN] 2021-02-10T04:31:23.517 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:23.518 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: cluster 2021-02-10T04:31:23.518 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: 2021-02-10T04:31:21 2021-02-10T04:31:23.518 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: .911426+0000 osd.2 (osd.2) 98 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:23.518 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:31:23.518 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: :31:21.911430+0000 osd.2 (osd.2) 2021-02-10T04:31:23.518 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: 99 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:23.518 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:23.518 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: -02-10T04:31:21.911433+0000 osd.2 ( 2021-02-10T04:31:23.518 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: osd.2) 100 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:23.518 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: 2021-02-10T04:31:23.518 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: cluster 2021-02-10T04:31:23.518 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: 2021-02-10T04:31:21. 2021-02-10T04:31:23.518 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: 911448+0000 osd.2 (osd.2) 101 : cluster [WRN] 2021-02-10T04:31:23.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:23.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: cluster 2021-02-10T04:31: 2021-02-10T04:31:23.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: 22.168991+0000 osd.1 (osd.1) 118 : 2021-02-10T04:31:23.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:23.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: cluster 2021-02-10T04:31:22.169000 2021-02-10T04:31:23.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: +0000 osd.1 (osd.1) 119 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:23.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:31:23.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: 04:31 2021-02-10T04:31:23.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: :22.169014+0000 osd.1 (osd.1) 120 2021-02-10T04:31:23.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:23.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:31:23.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: 31:22.169020+0000 osd.1 (osd.1) 121 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:23.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: cluster 2021-02-10T04:31:23.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: 2021-02-10T04:31:22.169027+0000 osd.1 (osd.1) 122 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:23.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:23.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: 10T04:31:22.169033+0000 osd.1 ( 2021-02-10T04:31:23.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: osd.1) 123 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:23.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:23.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: 02-10T04:31:22.209191+0000 mon.a (mon. 2021-02-10T04:31:23.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: 0) 996 : cluster [WRN] Health check failed: 12 slow ops, oldest one blocked for 112 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:31:23.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:23.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: 02-10T04:31:22.220599+0000 2021-02-10T04:31:23.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: mon.a (mon.0) 997 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2021-02-10T04:31:23.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: 2021-02-10T04:31:23.521 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: cluster 2021-02-10T04:31:23.521 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: 2021-02-10T04:31:22.414081+0000 osd.4 ( 2021-02-10T04:31:23.521 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: osd.4) 163 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:23.521 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:23.521 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: 02-10T04:31:22.414087+0000 2021-02-10T04:31:23.521 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: osd.4 (osd.4) 164 : cluster [WRN] 2021-02-10T04:31:23.521 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:23.521 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: cluster 2021-02-10T04:31:23.521 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: 2021-02-10T04:31:22.414091+0000 osd.4 ( 2021-02-10T04:31:23.521 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: osd.4) 165 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:23.521 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:31:23.521 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: :31:22.414106+0000 osd.4 (osd.4) 2021-02-10T04:31:23.521 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: 166 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:23.521 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: cluster 2021-02-10T04:31:23.522 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: 2021-02-10T04:31:23.078336+0000 mon.a (mon.0) 998 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2021-02-10T04:31:23.522 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[29319]: 2021-02-10T04:31:23.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: cluster 2021-02-10T04:31:23.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 2021-02-10T04:31:23.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 2021 2021-02-10T04:31:23.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: -02-10T04:31:21.911409+0000 osd.2 2021-02-10T04:31:23.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: (osd.2) 94 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:23.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: cluster 2021-02-10T04:31:21. 2021-02-10T04:31:23.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 911414+0000 osd.2 (osd.2) 95 : cluster [WRN] 2021-02-10T04:31:23.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:23.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:23.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 31:21.911419+0000 osd.2 (osd.2) 2021-02-10T04:31:23.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 96 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:23.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:23.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: -02-10T04:31:21.911422 2021-02-10T04:31:23.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 97 : cluster 2021-02-10T04:31:23.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:23.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:23.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: -10T04:31:21.911426+0000 2021-02-10T04:31:23.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: osd.2 (osd.2) 98 : cluster [WRN] 2021-02-10T04:31:23.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:23.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:23.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: :31:21.911430+0000 osd.2 (osd 2021-02-10T04:31:23.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: .2) 99 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:23.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: cluster 2021-02-10T04:31:23.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 2021-02-10T04:31:23.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 2021 2021-02-10T04:31:23.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: - 2021-02-10T04:31:23.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 02-10T04:31: 2021-02-10T04:31:23.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 21.911433+0000 osd.2 (osd.2) 100 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:23.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:23.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 02-10T04:31 2021-02-10T04:31:23.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: :21.911448+0000 osd.2 (osd.2) 101 2021-02-10T04:31:23.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:23.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:23.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 02-10T04:31:22.168991+0000 2021-02-10T04:31:23.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: osd.1 (osd.1) 118 : cluster [WRN] 2021-02-10T04:31:23.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:23.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: cluster 2021-02-10T04:31: 2021-02-10T04:31:23.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 22.169000+0000 osd.1 (osd 2021-02-10T04:31:23.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: .1) 119 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:23.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 2021-02-10T04:31:23.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: cluster 2021-02-10T04:31:23.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 2021-02-10T04:31:23.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 2021 2021-02-10T04:31:23.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: - 2021-02-10T04:31:23.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 02 2021-02-10T04:31:23.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: - 2021-02-10T04:31:23.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 10 2021-02-10T04:31:23.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: T 2021-02-10T04:31:23.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 04 2021-02-10T04:31:23.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: : 2021-02-10T04:31:23.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 31 2021-02-10T04:31:23.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: : 2021-02-10T04:31:23.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 22 2021-02-10T04:31:23.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: .169014+0000 osd.1 (osd 2021-02-10T04:31:23.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: .1) 120 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:23.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:23.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: -02-10T04:31:22.169020+0000 2021-02-10T04:31:23.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: osd.1 (osd.1) 121 : cluster [WRN] 2021-02-10T04:31:23.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:23.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: cluster 2021-02-10T04:31:23.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 2021-02-10T04:31 2021-02-10T04:31:23.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: :22.169027+0000 osd.1 (osd.1) 2021-02-10T04:31:23.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 122 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:23.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:23.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: -02-10T04:31:22. 2021-02-10T04:31:23.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 169033+0000 osd.1 (osd.1) 123 : 2021-02-10T04:31:23.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:23.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: cluster 2021-02-10T04:31: 2021-02-10T04:31:23.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 22.209191+0000 mon.a (mon.0) 2021-02-10T04:31:23.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 996 : cluster [WRN] Health check failed: 12 slow ops, oldest one blocked for 112 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:31:23.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: cluster 2021-02-10T04:31:23.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 2021-02-10T04:31:22 2021-02-10T04:31:23.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: .220599+0000 mon.a (mon.0) 997 2021-02-10T04:31:23.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2021-02-10T04:31:23.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: cluster 2021-02-10T04:31:23.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 2021-02-10T04:31:22 2021-02-10T04:31:23.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: .414081+0000 osd.4 (osd.4) 163 2021-02-10T04:31:23.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:23.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: cluster 2021-02-10T04:31:23.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 2021-02-10T04:31:22. 2021-02-10T04:31:23.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 414087+0000 osd.4 (osd.4) 164 : cluster 2021-02-10T04:31:23.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:23.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:23.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 02-10T04:31:22.414091 2021-02-10T04:31:23.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 165 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:23.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: cluster 2021-02-10T04:31:23.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: 2021-02-10T04:31:22 2021-02-10T04:31:23.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: .414106+0000 osd.4 (osd.4) 166 2021-02-10T04:31:23.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:23.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: cluster 2021-02-10T04:31:23 2021-02-10T04:31:23.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: .078336+0000 mon.a (mon.0) 998 : cluster 2021-02-10T04:31:23.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[35961]: [DBG] osdmap e266: 8 total, 8 up, 8 in 2021-02-10T04:31:23.633 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:31:23.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[33677]: debug 2021-02-10T04:31:23.413+0000 7fbec19a1700 -1 osd.4 266 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:31:23.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: cluster 2021-02-10T04:31:21.911409+0000 osd.2 (osd.2) 2021-02-10T04:31:23.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: 94 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:23.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: cluster 2021-02-10T04:31:21.911414+0000 osd.2 (osd.2) 95 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:23.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: cluster 2021-02-10T04:31:21.911419+0000 osd.2 (osd.2) 96 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: cluster 2021-02-10T04:31:21.911422 2021-02-10T04:31:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 97 : cluster [WRN] 2021-02-10T04:31:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: cluster 2021-02-10T04:31: 2021-02-10T04:31:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: 21.911426+0000 osd.2 (osd.2) 98 : 2021-02-10T04:31:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: cluster 2021-02-10T04:31: 2021-02-10T04:31:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: 21.911430+0000 osd.2 (osd.2) 99 : 2021-02-10T04:31:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: cluster 2021-02-10T04:31:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: 2021-02-10T04:31:21.911433+0000 osd.2 (osd 2021-02-10T04:31:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: .2) 100 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: cluster 2021-02-10T04:31:21.911448+0000 2021-02-10T04:31:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: osd.2 (osd.2) 101 : cluster [WRN] 2021-02-10T04:31:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: cluster 2021-02-10T04:31:22.168991+0000 2021-02-10T04:31:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: osd.1 (osd.1) 118 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: cluster 2021-02-10T04:31: 2021-02-10T04:31:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: 22.169000+0000 osd.1 (osd.1) 119 : 2021-02-10T04:31:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: cluster 2021-02-10T04:31:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: 2021-02-10T04:31:22.169014+0000 osd.1 ( 2021-02-10T04:31:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: osd.1) 120 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: cluster 2021-02-10T04:31:22. 2021-02-10T04:31:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: 169020+0000 osd.1 (osd.1) 121 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: -10T04:31:22.169027+0000 osd.1 (osd.1) 122 2021-02-10T04:31:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: -02-10T04:31:22.169033+0000 osd.1 (osd.1) 123 : 2021-02-10T04:31:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: cluster 2021-02-10T04:31: 2021-02-10T04:31:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: 22.209191+0000 mon.a (mon.0) 996 : cluster [WRN] Health check failed: 12 slow ops, oldest one blocked for 112 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:31:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: -10T04:31:22.220599+0000 mon.a 2021-02-10T04:31:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: (mon.0) 997 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2021-02-10T04:31:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: 2021-02-10T04:31:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: cluster 2021-02-10T04:31:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: 2021-02-10T04:31:22.414081+0000 osd.4 2021-02-10T04:31:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: (osd.4) 163 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: 2021-02-10T04:31:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: cluster 2021-02-10T04:31:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: 2021-02-10T04:31:22.414087+0000 osd.4 ( 2021-02-10T04:31:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: osd.4) 164 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: T04:31:22.414091+0000 osd.4 (osd.4) 165 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: cluster 2021-02-10T04:31:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: 2021-02-10T04:31:22.414106+0000 osd.4 2021-02-10T04:31:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: (osd.4) 166 : cluster [WRN] 2021-02-10T04:31:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: -02-10T04:31:23 2021-02-10T04:31:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: .078336+0000 mon.a (mon. 2021-02-10T04:31:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: 0) 998 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2021-02-10T04:31:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[29288]: 2021-02-10T04:31:23.665 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:23.954 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:31:23.960 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:31:23 smithi164 conmon[37263]: debug 2021-02-10T04:31:23.742+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:24.061 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:31:24.084 INFO:tasks.rados.rados.0.smithi085.stdout:899: done (14 left) 2021-02-10T04:31:24.084 INFO:tasks.rados.rados.0.smithi085.stdout:900: done (13 left) 2021-02-10T04:31:24.084 INFO:tasks.rados.rados.0.smithi085.stdout:903: done (12 left) 2021-02-10T04:31:24.084 INFO:tasks.rados.rados.0.smithi085.stdout:904: read oid 27 snap -1 2021-02-10T04:31:24.084 INFO:tasks.rados.rados.0.smithi085.stdout:904: expect deleted 2021-02-10T04:31:24.084 INFO:tasks.rados.rados.0.smithi085.stdout:905: read oid 1 snap -1 2021-02-10T04:31:24.084 INFO:tasks.rados.rados.0.smithi085.stdout:905: expect (ObjNum 336 snap 84 seq_num 336) 2021-02-10T04:31:24.084 INFO:tasks.rados.rados.0.smithi085.stdout:906: write (excl) oid 31 current snap is 85 2021-02-10T04:31:24.084 INFO:tasks.rados.rados.0.smithi085.stdout:906: seq_num 337 ranges {463959=478445,1570667=731108,3038372=492433} 2021-02-10T04:31:24.087 INFO:tasks.rados.rados.0.smithi085.stdout:906: writing smithi08575397-31 from 463959 to 942404 tid 1 2021-02-10T04:31:24.092 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:24.096 INFO:tasks.rados.rados.0.smithi085.stdout:906: writing smithi08575397-31 from 1570667 to 2301775 tid 2 2021-02-10T04:31:24.103 INFO:tasks.rados.rados.0.smithi085.stdout:906: writing smithi08575397-31 from 3038372 to 3530805 tid 3 2021-02-10T04:31:24.103 INFO:tasks.rados.rados.0.smithi085.stdout:907: rollback oid 30 current snap is 85 2021-02-10T04:31:24.103 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 30 to 82 2021-02-10T04:31:24.103 INFO:tasks.rados.rados.0.smithi085.stdout:905: expect (ObjNum 336 snap 84 seq_num 336) 2021-02-10T04:31:24.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:23 smithi085 conmon[49106]: debug 2021-02-10T04:31:23.904+0000 7f77a3e94700 -1 osd.2 266 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:31:24.226 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[33677]: debug 2021-02-10T04:31:24.164+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:24.270 INFO:tasks.rados.rados.0.smithi085.stdout:906: finishing write tid 1 to smithi08575397-31 2021-02-10T04:31:24.270 INFO:tasks.rados.rados.0.smithi085.stdout:906: finishing write tid 2 to smithi08575397-31 2021-02-10T04:31:24.270 INFO:tasks.rados.rados.0.smithi085.stdout:904: done (15 left) 2021-02-10T04:31:24.270 INFO:tasks.rados.rados.0.smithi085.stdout:905: done (14 left) 2021-02-10T04:31:24.270 INFO:tasks.rados.rados.0.smithi085.stdout:908: read oid 46 snap 84 2021-02-10T04:31:24.270 INFO:tasks.rados.rados.0.smithi085.stdout:908: expect deleted 2021-02-10T04:31:24.270 INFO:tasks.rados.rados.0.smithi085.stdout:909: rollback oid 2 current snap is 85 2021-02-10T04:31:24.270 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 2 to 84 2021-02-10T04:31:24.270 INFO:tasks.rados.rados.0.smithi085.stdout:906: finishing write tid 3 to smithi08575397-31 2021-02-10T04:31:24.270 INFO:tasks.rados.rados.0.smithi085.stdout:906: finishing write tid 4 to smithi08575397-31 2021-02-10T04:31:24.271 INFO:tasks.rados.rados.0.smithi085.stdout:906: finishing write tid 5 to smithi08575397-31 2021-02-10T04:31:24.271 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 150 (ObjNum 337 snap 85 seq_num 337) dirty exists 2021-02-10T04:31:24.271 INFO:tasks.rados.rados.0.smithi085.stdout:906: left oid 31 (ObjNum 337 snap 85 seq_num 337) 2021-02-10T04:31:24.271 INFO:tasks.rados.rados.0.smithi085.stdout:907: finishing rollback tid 0 to smithi08575397-30 2021-02-10T04:31:24.271 INFO:tasks.rados.rados.0.smithi085.stdout:907: finishing rollback tid 1 to smithi08575397-30 2021-02-10T04:31:24.271 INFO:tasks.rados.rados.0.smithi085.stdout:907: finishing rollback tid 2 to smithi08575397-30 2021-02-10T04:31:24.271 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 289 (ObjNum 222 snap 50 seq_num 222) dirty exists 2021-02-10T04:31:24.271 INFO:tasks.rados.rados.0.smithi085.stdout:906: done (15 left) 2021-02-10T04:31:24.271 INFO:tasks.rados.rados.0.smithi085.stdout:907: done (14 left) 2021-02-10T04:31:24.271 INFO:tasks.rados.rados.0.smithi085.stdout:910: snap_create 2021-02-10T04:31:24.272 INFO:tasks.rados.rados.0.smithi085.stdout:909: finishing rollback tid 0 to smithi08575397-2 2021-02-10T04:31:24.272 INFO:tasks.rados.rados.0.smithi085.stdout:909: finishing rollback tid 1 to smithi08575397-2 2021-02-10T04:31:24.272 INFO:tasks.rados.rados.0.smithi085.stdout:909: finishing rollback tid 2 to smithi08575397-2 2021-02-10T04:31:24.272 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 284 (ObjNum 325 snap 83 seq_num 325) dirty exists 2021-02-10T04:31:24.442 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:31:24.443 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [], 2021-02-10T04:31:24.443 INFO:teuthology.orchestra.run.smithi085.stdout: "complaint_time": 30, 2021-02-10T04:31:24.443 INFO:teuthology.orchestra.run.smithi085.stdout: "num_blocked_ops": 0 2021-02-10T04:31:24.443 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:31:24.450 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:31:24.556 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:31:24.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: cluster 2021-02-10T04:31:24.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:31:24.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 10T04:31:22. 2021-02-10T04:31:24.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 925147+0000 osd.2 ( 2021-02-10T04:31:24.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: osd 2021-02-10T04:31:24.597 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: .2) 102 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:24.597 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:24.597 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: -02-10T04:31:22.925156+0000 osd.2 2021-02-10T04:31:24.597 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: (osd.2) 103 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:24.597 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 2021-02-10T04:31:24.597 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: cluster 2021-02-10T04:31:24.597 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 2021-02-10T04:31:22.925163+0000 osd.2 2021-02-10T04:31:24.597 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: (osd.2) 104 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:24.598 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 2021-02-10T04:31:24.598 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: cluster 2021-02-10T04:31:24.598 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 2021-02-10T04:31:22.925169+0000 osd.2 ( 2021-02-10T04:31:24.598 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: osd.2) 105 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:24.598 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:24.598 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: -02-10T04:31:22.925174+0000 2021-02-10T04:31:24.598 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: osd.2 (osd.2) 106 : cluster [WRN] 2021-02-10T04:31:24.598 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:24.598 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: cluster 2021-02-10T04:31: 2021-02-10T04:31:24.598 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 22.925182+0000 osd.2 (osd.2) 107 2021-02-10T04:31:24.599 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:24.599 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:31:24.599 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 04:31:22.925189+0000 osd.2 (osd. 2021-02-10T04:31:24.599 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 2) 108 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:24.599 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:24.599 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: -10T04:31:22.925196+0000 osd.2 2021-02-10T04:31:24.599 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: (osd.2) 109 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:24.599 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 2021-02-10T04:31:24.599 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: cluster 2021-02-10T04:31:23.171414 2021-02-10T04:31:24.599 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: +0000 osd.1 (osd.1) 124 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:24.600 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: cluster 2021-02-10T04:31:24.600 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 2021-02-10T04:31:24.600 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 2021 2021-02-10T04:31:24.600 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: - 2021-02-10T04:31:24.600 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 02 2021-02-10T04:31:24.600 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: - 2021-02-10T04:31:24.600 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 10 2021-02-10T04:31:24.600 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: T 2021-02-10T04:31:24.600 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 04 2021-02-10T04:31:24.601 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: : 2021-02-10T04:31:24.601 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 31 2021-02-10T04:31:24.601 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: : 2021-02-10T04:31:24.601 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 23 2021-02-10T04:31:24.601 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: . 2021-02-10T04:31:24.601 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 171435 2021-02-10T04:31:24.601 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: +0000 2021-02-10T04:31:24.601 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 2021-02-10T04:31:24.601 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: osd.1 2021-02-10T04:31:24.601 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: ( 2021-02-10T04:31:24.601 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: osd 2021-02-10T04:31:24.602 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: . 2021-02-10T04:31:24.602 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 1 2021-02-10T04:31:24.602 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: ) 2021-02-10T04:31:24.602 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 125 2021-02-10T04:31:24.602 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: : 2021-02-10T04:31:24.602 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: cluster 2021-02-10T04:31:24.602 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 2021-02-10T04:31:24.602 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: [WRN] 2021-02-10T04:31:24.602 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 2021-02-10T04:31:24.602 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:24.602 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 2021-02-10T04:31:24.603 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: cluster 2021-02-10T04:31:24.603 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 2021-02-10T04:31:24.603 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 2021 2021-02-10T04:31:24.603 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: - 2021-02-10T04:31:24.603 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 02 2021-02-10T04:31:24.603 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: - 2021-02-10T04:31:24.603 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 10 2021-02-10T04:31:24.603 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: T 2021-02-10T04:31:24.603 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 04 2021-02-10T04:31:24.604 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: : 2021-02-10T04:31:24.604 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 31 2021-02-10T04:31:24.604 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: : 2021-02-10T04:31:24.604 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 23 2021-02-10T04:31:24.604 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: . 2021-02-10T04:31:24.604 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 171442 2021-02-10T04:31:24.604 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: +0000 2021-02-10T04:31:24.604 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 2021-02-10T04:31:24.604 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: osd.1 (osd.1) 126 : cluster 2021-02-10T04:31:24.604 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:24.605 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:31:24.605 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 04:31:23.171447+0000 osd.1 (osd. 2021-02-10T04:31:24.605 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 1) 127 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:24.605 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: cluster 2021-02-10T04:31:24.605 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 2021-02-10T04:31:23. 2021-02-10T04:31:24.605 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 171454+0000 osd.1 (osd.1) 128 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:24.605 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:24.605 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: -02-10T04:31:23.171460+0000 osd.1 (osd.1) 129 : 2021-02-10T04:31:24.605 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:24.605 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: cluster 2021-02-10T04:31:23.414482+0000 osd.4 (osd 2021-02-10T04:31:24.606 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: .4) 167 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:24.606 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:31:24.606 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 31:23.414492+0000 osd.4 (osd.4) 2021-02-10T04:31:24.606 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 168 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:24.606 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:31:24.606 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 04:31:23.414499+0000 osd.4 (osd.4) 169 : cluster [WRN] 2021-02-10T04:31:24.606 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:24.606 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: cluster 2021-02-10T04:31:24.606 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: 2021-02-10T04:31:23.414504+0000 osd.4 (osd.4) 170 : cluster [WRN] 2021-02-10T04:31:24.606 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:24.606 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:24.607 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: T04:31:24.033767+0000 mgr.y (mgr.14140) 439 : 2021-02-10T04:31:24.607 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: cluster [DBG] pgmap v549: 168 pgs: 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 157 active+clean; 173 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 15 MiB/s rd, 3.4 MiB/s wr, 21 op/s 2021-02-10T04:31:24.607 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:24.607 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: T04:31:24.081981+0000 mon.a (mon 2021-02-10T04:31:24.607 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[29319]: .0) 999 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2021-02-10T04:31:24.607 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: cluster 2021-02-10T04:31:24.607 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 2021-02-10T04:31:24.607 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 2021 2021-02-10T04:31:24.607 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: - 2021-02-10T04:31:24.608 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 02 2021-02-10T04:31:24.608 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: - 2021-02-10T04:31:24.608 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 10 2021-02-10T04:31:24.608 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: T 2021-02-10T04:31:24.608 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 04 2021-02-10T04:31:24.608 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: : 2021-02-10T04:31:24.608 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 31 2021-02-10T04:31:24.608 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: : 2021-02-10T04:31:24.608 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 22 2021-02-10T04:31:24.609 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: . 2021-02-10T04:31:24.609 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 925147 2021-02-10T04:31:24.609 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 102 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:24.609 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:24.609 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 10T04:31:22.925156+0000 osd.2 (osd 2021-02-10T04:31:24.609 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: .2) 103 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:24.609 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:24.609 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: -10T04:31:22.925163+0000 osd.2 (osd.2) 104 : cluster 2021-02-10T04:31:24.609 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:24.610 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:24.610 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 04:31:22.925169+0000 osd.2 (osd. 2021-02-10T04:31:24.610 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 2) 105 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:24.610 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:24.610 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: T04:31:22.925174+0000 osd.2 (osd.2) 2021-02-10T04:31:24.610 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 106 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:24.610 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:24.610 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 02-10T04:31:22.925182+0000 osd.2 ( 2021-02-10T04:31:24.611 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: osd.2) 107 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:24.611 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:24.611 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 02-10T04:31:22.925189+0000 osd.2 ( 2021-02-10T04:31:24.611 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: osd.2) 108 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:24.611 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:24.611 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: -10T04:31:22.925196+0000 osd.2 (osd 2021-02-10T04:31:24.611 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: .2) 109 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:24.611 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:24.611 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 02-10T04:31:23.171414+0000 osd.1 (osd 2021-02-10T04:31:24.611 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: .1) 124 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:24.611 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:24.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: -10T04:31:23.171435+0000 osd.1 (osd. 2021-02-10T04:31:24.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 1) 125 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:24.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:24.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 04:31:23.171442+0000 osd.1 (osd.1) 126 2021-02-10T04:31:24.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:24.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:24.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: -10T04:31:23.171447+0000 osd.1 ( 2021-02-10T04:31:24.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: osd.1) 127 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:24.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:24.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: -10T04:31:23.171454+0000 osd.1 (osd 2021-02-10T04:31:24.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: .1) 128 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:24.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:24.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: :31:23.171460+0000 osd.1 (osd.1) 129 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:24.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:24.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: -10T04:31:23.414482+0000 osd.4 (osd.4) 167 : 2021-02-10T04:31:24.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:24.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: cluster 2021-02-10T04:31:24.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 2021-02-10T04:31:24.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 2021 2021-02-10T04:31:24.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: - 2021-02-10T04:31:24.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 02 2021-02-10T04:31:24.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: - 2021-02-10T04:31:24.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 10 2021-02-10T04:31:24.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: T 2021-02-10T04:31:24.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 04 2021-02-10T04:31:24.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: : 2021-02-10T04:31:24.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 31 2021-02-10T04:31:24.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: : 2021-02-10T04:31:24.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 23 2021-02-10T04:31:24.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: . 2021-02-10T04:31:24.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 414492 2021-02-10T04:31:24.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: +0000 2021-02-10T04:31:24.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 2021-02-10T04:31:24.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: osd.4 2021-02-10T04:31:24.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: ( 2021-02-10T04:31:24.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: osd 2021-02-10T04:31:24.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: . 2021-02-10T04:31:24.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 4 2021-02-10T04:31:24.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: ) 2021-02-10T04:31:24.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 168 2021-02-10T04:31:24.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: : 2021-02-10T04:31:24.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: cluster 2021-02-10T04:31:24.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 2021-02-10T04:31:24.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: [WRN] 2021-02-10T04:31:24.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 2021-02-10T04:31:24.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:24.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:24.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 02-10T04:31:23.414499+0000 osd.4 (osd.4) 169 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:24.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: cluster 2021-02-10T04:31:24.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:31:24.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 04:31:23.414504+0000 osd.4 (osd.4 2021-02-10T04:31:24.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: ) 170 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:24.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: cluster 2021-02-10T04:31:24.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: 2021-02-10T04:31 2021-02-10T04:31:24.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: :24.033767+0000 mgr.y (mgr.14140) 439 : cluster [DBG] pgmap v549: 168 pgs: 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 157 active+clean; 173 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 15 MiB/s rd, 3.4 MiB/s wr, 21 op/s 2021-02-10T04:31:24.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:24.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: -02-10T04:31:24.081981+0000 2021-02-10T04:31:24.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: mon.a (mon.0) 999 : cluster [DBG] 2021-02-10T04:31:24.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[35961]: osdmap e267: 8 total, 8 up, 8 in 2021-02-10T04:31:24.618 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[45443]: debug 2021-02-10T04:31:24.200+0000 7f7140d73700 -1 osd.1 267 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:31:24.618 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:24.649 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[33677]: debug 2021-02-10T04:31:24.393+0000 7fbec19a1700 -1 osd.4 267 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:31:24.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: cluster 2021-02-10T04:31:22.925147+0000 osd.2 (osd.2) 102 2021-02-10T04:31:24.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:24.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: cluster 2021-02-10T04:31:22.925156+0000 osd.2 (osd.2) 103 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:24.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:31:24.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: :31:22.925163+0000 osd.2 (osd.2) 104 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:24.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:24.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: 02-10T04:31:22.925169+0000 2021-02-10T04:31:24.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: osd.2 (osd.2) 105 : cluster [WRN] 2021-02-10T04:31:24.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:24.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:24.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: T04:31:22.925174+0000 osd.2 (osd. 2021-02-10T04:31:24.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: 2) 106 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:24.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: cluster 2021-02-10T04:31:22.925182+0000 osd.2 (osd. 2021-02-10T04:31:24.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: 2) 107 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:24.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: cluster 2021-02-10T04:31 2021-02-10T04:31:24.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: :22.925189+0000 osd.2 (osd.2) 108 : 2021-02-10T04:31:24.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:24.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:24.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: -10T04:31:22.925196+0000 osd.2 (osd. 2021-02-10T04:31:24.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: 2) 109 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:24.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: cluster 2021-02-10T04:31 2021-02-10T04:31:24.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: :23.171414+0000 osd.1 (osd.1) 124 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:24.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:31:24.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: :31:23.171435+0000 osd.1 (osd.1) 125 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:24.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: 2021-02-10T04:31:24.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:24.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: 02-10T04:31:23.171442+0000 osd.1 2021-02-10T04:31:24.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: (osd.1) 126 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:24.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:24.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: -02-10T04:31:23 2021-02-10T04:31:24.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: .171447+0000 osd.1 (osd.1) 127 : cluster 2021-02-10T04:31:24.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:24.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:24.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: T04:31:23.171454+0000 osd.1 2021-02-10T04:31:24.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: (osd.1) 128 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:24.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:24.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: -02-10T04:31:23.171460 2021-02-10T04:31:24.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 129 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:24.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:31:24.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: :31:23.414482+0000 osd.4 (osd.4) 167 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:24.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:24.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: T04:31:23.414492+0000 osd.4 (osd. 2021-02-10T04:31:24.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: 4) 168 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:24.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:31:24.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: 04:31:23.414499+0000 osd.4 (osd.4) 169 : cluster 2021-02-10T04:31:24.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:24.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: cluster 2021-02-10T04:31:24.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: 2021-02-10T04:31:23.414504+0000 osd.4 (osd.4) 170 : cluster [WRN] 2021-02-10T04:31:24.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:24.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:24.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: 02-10T04:31:24.033767+0000 mgr.y (mgr.14140) 439 : cluster 2021-02-10T04:31:24.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: [DBG] pgmap v549: 168 pgs: 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 157 active+clean; 173 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 15 MiB/s rd, 3.4 MiB/s wr, 21 op/s 2021-02-10T04:31:24.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:24.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: 10T04:31:24.081981 2021-02-10T04:31:24.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: +0000 mon.a (mon.0) 999 : cluster 2021-02-10T04:31:24.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:24 smithi164 conmon[29288]: [DBG] osdmap e267: 8 total, 8 up, 8 in 2021-02-10T04:31:24.908 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:24 smithi085 conmon[49106]: debug 2021-02-10T04:31:24.907+0000 7f77a3e94700 -1 osd.2 267 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:31:25.232 INFO:tasks.rados.rados.0.smithi085.stdout:908: done (14 left) 2021-02-10T04:31:25.232 INFO:tasks.rados.rados.0.smithi085.stdout:909: done (13 left) 2021-02-10T04:31:25.232 INFO:tasks.rados.rados.0.smithi085.stdout:910: done (12 left) 2021-02-10T04:31:25.232 INFO:tasks.rados.rados.0.smithi085.stdout:911: snap_create 2021-02-10T04:31:25.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[33677]: debug 2021-02-10T04:31:25.396+0000 7fbec19a1700 -1 osd.4 268 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:31:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: cluster 2021-02-10T04:31:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: 2021-02-10T04:31:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: 2021-02-10T04:31:23. 2021-02-10T04:31:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: 905889+0000 osd.2 (osd.2) 110 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: cluster 2021-02-10T04:31:23.905898+0000 osd.2 (osd.2) 2021-02-10T04:31:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: 111 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: T04:31:23.905904+0000 osd.2 (osd.2 2021-02-10T04:31:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: ) 112 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:31:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: 04:31:23.905911+0000 osd.2 (osd. 2021-02-10T04:31:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: 2) 113 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: 10T04:31:23.905917+0000 osd.2 (osd.2) 2021-02-10T04:31:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: 114 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: cluster 2021-02-10T04:31:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: 2021-02-10T04:31:23.905923+0000 osd.2 (osd.2) 115 : cluster [WRN] 2021-02-10T04:31:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: cluster 2021-02-10T04:31:23 2021-02-10T04:31:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: .905927+0000 osd.2 (osd.2) 116 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: 02-10T04:31:23.905936+0000 osd.2 (osd.2) 117 : 2021-02-10T04:31:25.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:25.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: cluster 2021-02-10T04:31:24.201684+0000 osd.1 (osd 2021-02-10T04:31:25.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: .1) 130 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:25.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:25.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: -02-10T04:31:24.201690+0000 osd.1 (osd.1) 131 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:25.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: cluster 2021-02-10T04:31:24.201695 2021-02-10T04:31:25.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 132 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:25.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: cluster 2021-02-10T04:31:24.201698+0000 osd.1 (osd.1 2021-02-10T04:31:25.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: ) 133 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:25.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:25.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: T04:31:24.201702+0000 osd.1 (osd.1) 134 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:25.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: cluster 2021-02-10T04:31:24.201705+0000 osd.1 (osd.1) 2021-02-10T04:31:25.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: 135 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: -10T04:31:24.394162+0000 osd.4 (osd.4) 171 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: cluster 2021-02-10T04:31:24.394181+0000 osd.4 (osd.4) 172 : cluster [WRN] 2021-02-10T04:31:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: cluster 2021-02-10T04:31: 2021-02-10T04:31:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: 24.394191+0000 osd.4 (osd.4) 173 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: cluster 2021-02-10T04:31:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: 2021-02-10T04:31:24.394198+0000 osd.4 (osd.4) 174 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:25 smithi164 conmon[29288]: 2021-02-10T04:31:25.691 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[45443]: debug 2021-02-10T04:31:25.691 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[45443]: 2021-02-10T04:31:25.211+0000 7f7140d73700 -1 osd.1 267 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:31:25.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: cluster 2021-02-10T04:31:25.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:31:25.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: 10T04:31:23.905889+0000 osd.2 (osd.2) 110 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:25.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:25.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: 02-10T04:31:23.905898+0000 osd.2 (osd.2 2021-02-10T04:31:25.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: ) 111 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:25.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:25.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: 02-10T04:31:23.905904+0000 2021-02-10T04:31:25.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: osd.2 (osd.2) 112 : cluster [WRN] 2021-02-10T04:31:25.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:25.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: cluster 2021-02-10T04:31:23. 2021-02-10T04:31:25.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: 905911+0000 osd.2 (osd.2) 113 : cluster 2021-02-10T04:31:25.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:25.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:31:25.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: 31:23.905917+0000 osd.2 (osd.2 2021-02-10T04:31:25.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: ) 114 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:25.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:25.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: T04:31:23.905923+0000 osd.2 ( 2021-02-10T04:31:25.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: osd.2) 115 : cluster 2021-02-10T04:31:25.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:25.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: cluster 2021-02-10T04:31:23.905927+0000 osd.2 2021-02-10T04:31:25.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: (osd.2) 116 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:25.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: cluster 2021-02-10T04:31:25.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: 2021-02-10T04:31:23.905936+0000 osd.2 (osd.2 2021-02-10T04:31:25.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: ) 117 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:25.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:25.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: 10T04:31:24.201684+0000 osd.1 ( 2021-02-10T04:31:25.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: osd.1) 130 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:25.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:25.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: 02-10T04:31:24.201690+0000 osd.1 (osd.1) 131 : 2021-02-10T04:31:25.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:25.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: cluster 2021-02-10T04:31 2021-02-10T04:31:25.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: :24.201695+0000 osd.1 (osd.1) 132 2021-02-10T04:31:25.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:25.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:31:25.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: 31:24.201698+0000 osd.1 (osd.1) 133 : 2021-02-10T04:31:25.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:25.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: cluster 2021-02-10T04:31 2021-02-10T04:31:25.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: :24.201702+0000 osd.1 (osd.1) 2021-02-10T04:31:25.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: 134 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:25.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:31:25.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: 04:31:24.201705+0000 osd.1 (osd 2021-02-10T04:31:25.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: .1) 135 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:25.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:25.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: T04:31:24.394162+0000 osd.4 (osd 2021-02-10T04:31:25.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: .4) 171 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:25.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:25.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: 02-10T04:31:24.394181+0000 2021-02-10T04:31:25.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: osd.4 (osd.4) 172 : cluster [WRN] 2021-02-10T04:31:25.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:25.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: cluster 2021-02-10T04:31:25.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: 2021-02-10T04:31:24.394191+0000 osd.4 ( 2021-02-10T04:31:25.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: osd.4) 173 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:25.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:25.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: 02-10T04:31:24.394198+0000 2021-02-10T04:31:25.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: osd.4 (osd.4) 174 : cluster [WRN] 2021-02-10T04:31:25.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:25.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: cluster 2021-02-10T04:31:23. 2021-02-10T04:31:25.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: 905889+0000 osd.2 (osd.2) 110 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:25.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: cluster 2021-02-10T04:31:25.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: 2021 2021-02-10T04:31:25.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: - 2021-02-10T04:31:25.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: 02 2021-02-10T04:31:25.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: - 2021-02-10T04:31:25.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: 10 2021-02-10T04:31:25.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: T 2021-02-10T04:31:25.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: 04:31:23.905898+0000 osd.2 2021-02-10T04:31:25.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: (osd.2) 111 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:25.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:25.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: -02-10T04:31:23.905904+0000 osd.2 (osd 2021-02-10T04:31:25.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: .2) 112 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:25.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: cluster 2021-02-10T04:31:25.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: 2021-02-10T04:31:23.905911+0000 2021-02-10T04:31:25.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: osd.2 (osd.2) 113 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:25.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: cluster 2021-02-10T04:31:23.905917+0000 osd.2 (osd.2) 114 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:25.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:25.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: -10T04:31:23.905923+0000 osd.2 (osd 2021-02-10T04:31:25.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: .2) 115 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:25.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: cluster 2021-02-10T04:31:25.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:31:25.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: :31:23.905927+0000 osd.2 (osd.2) 116 : cluster 2021-02-10T04:31:25.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:25.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:25.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: -02-10T04:31:23.905936+0000 2021-02-10T04:31:25.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: osd.2 (osd.2) 117 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:25.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: 2021-02-10T04:31:25.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: cluster 2021-02-10T04:31:25.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: 2021-02-10T04:31: 2021-02-10T04:31:25.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: 24.201684+0000 osd.1 (osd.1) 130 : 2021-02-10T04:31:25.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:25.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: cluster 2021-02-10T04:31:25.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: 2021-02-10T04:31:24. 2021-02-10T04:31:25.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: 201690+0000 osd.1 (osd.1) 131 : cluster [WRN] 2021-02-10T04:31:25.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:25.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:25.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: T04:31:24.201695+0000 osd.1 (osd. 2021-02-10T04:31:25.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: 1) 132 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:25.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:25.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: -02-10T04:31:24.201698+0000 osd.1 2021-02-10T04:31:25.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: (osd.1) 133 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:25.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: cluster 2021-02-10T04:31:25.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: 2021-02-10T04:31:24.201702+0000 osd.1 2021-02-10T04:31:25.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: (osd.1) 134 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:25.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:25.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: 04:31:24.201705+0000 osd.1 (osd.1) 135 2021-02-10T04:31:25.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:25.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:25.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: 10T04:31:24.394162+0000 osd.4 (osd.4) 171 : cluster 2021-02-10T04:31:25.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:25.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: cluster 2021-02-10T04:31:25.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: 2021-02-10T04:31:24 2021-02-10T04:31:25.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: .394181+0000 osd.4 (osd.4) 172 : cluster 2021-02-10T04:31:25.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:25.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: cluster 2021-02-10T04:31:25.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: 2021-02-10T04:31:24.394191+0000 2021-02-10T04:31:25.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: osd.4 (osd.4) 173 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:25.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:25.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: 10T04:31:24.394198+0000 osd.4 (osd 2021-02-10T04:31:25.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[35961]: .4) 174 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:26.100 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:31:26.178 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[49106]: debug 2021-02-10T04:31:25.938+0000 7f77a3e94700 -1 osd.2 268 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:31:26.178 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[52836]: debug 2021-02-10T04:31:26.178 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:31:25 smithi085 conmon[52836]: 2021-02-10T04:31:25.822+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:26.178 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[45443]: debug 2021-02-10T04:31:26.177+0000 7f7140d73700 -1 osd.1 268 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:31:26.239 INFO:tasks.rados.rados.0.smithi085.stdout:911: done (12 left) 2021-02-10T04:31:26.239 INFO:tasks.rados.rados.0.smithi085.stdout:912: write oid 24 current snap is 87 2021-02-10T04:31:26.239 INFO:tasks.rados.rados.0.smithi085.stdout:912: seq_num 338 ranges {480766=553694,1629149=591119,2782148=420058} 2021-02-10T04:31:26.243 INFO:tasks.rados.rados.0.smithi085.stdout:912: writing smithi08575397-24 from 480766 to 1034460 tid 1 2021-02-10T04:31:26.251 INFO:tasks.rados.rados.0.smithi085.stdout:912: writing smithi08575397-24 from 1629149 to 2220268 tid 2 2021-02-10T04:31:26.263 INFO:tasks.rados.rados.0.smithi085.stdout:912: writing smithi08575397-24 from 2782148 to 3202206 tid 3 2021-02-10T04:31:26.263 INFO:tasks.rados.rados.0.smithi085.stdout:913: snap_remove snap 86 2021-02-10T04:31:26.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: cluster 2021-02-10T04:31 2021-02-10T04:31:26.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: :24.907887+0000 osd.2 (osd.2) 118 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:26.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: cluster 2021-02-10T04:31:24.907895+0000 osd.2 (osd.2) 119 2021-02-10T04:31:26.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:26.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: cluster 2021-02-10T04:31:26.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: 2021- 2021-02-10T04:31:26.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: 02-10T04:31:24.907905+0000 osd.2 2021-02-10T04:31:26.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: (osd.2) 120 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:26.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:26.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: 02-10T04:31:24.907911+0000 osd.2 (osd.2) 121 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:26.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: cluster 2021-02-10T04:31:24.907916+0000 2021-02-10T04:31:26.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: osd.2 (osd.2) 122 : cluster 2021-02-10T04:31:26.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:26.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:26.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: T04:31:24.907922+0000 osd.2 (osd.2) 123 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:26.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:26.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: -10T04:31:24.907928+0000 osd.2 (osd 2021-02-10T04:31:26.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: . 2021-02-10T04:31:26.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: 2) 124 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:26.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:26.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: 02-10T04:31:24.907933+0000 osd.2 (osd. 2021-02-10T04:31:26.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: 2) 125 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:26.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:26.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: 02-10T04:31:25.211533+0000 osd.1 2021-02-10T04:31:26.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: (osd.1) 136 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:26.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:26.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: -02-10T04:31:25.211542+0000 2021-02-10T04:31:26.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: osd.1 (osd.1) 137 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:26.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: 2021-02-10T04:31:26.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: cluster 2021-02-10T04:31:26.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: 2021-02-10T04:31:25.211550+0000 osd.1 (osd.1) 138 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:26.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:26.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: 02-10T04:31:25.211555+0000 osd.1 2021-02-10T04:31:26.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: (osd.1) 139 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:26.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: cluster 2021-02-10T04:31:26.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: 2021-02-10T04:31:25.211561+0000 2021-02-10T04:31:26.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: osd.1 (osd.1) 140 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:26.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: 2021-02-10T04:31:26.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: cluster 2021-02-10T04:31:26.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: 2021-02-10T04:31:25.211573 2021-02-10T04:31:26.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: +0000 osd.1 (osd.1) 141 : cluster [WRN] 2021-02-10T04:31:26.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:26.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: cluster 2021-02-10T04:31:26.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: 2021-02-10T04:31:25.230756 2021-02-10T04:31:26.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1000 : cluster [DBG] 2021-02-10T04:31:26.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: osdmap e268: 8 total, 8 up, 8 in 2021-02-10T04:31:26.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: cluster 2021-02-10T04:31:25.396993 2021-02-10T04:31:26.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 175 : cluster [WRN] 2021-02-10T04:31:26.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:26.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:26.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: -10T04:31:25.397000+0000 osd.4 2021-02-10T04:31:26.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: (osd.4) 176 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:26.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: 2021-02-10T04:31:26.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:26.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: T04:31:25.397005+0000 osd.4 (osd 2021-02-10T04:31:26.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: .4) 177 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:26.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:26.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: 02-10T04:31:25.397010+0000 2021-02-10T04:31:26.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: osd.4 (osd.4) 178 : cluster [WRN] 2021-02-10T04:31:26.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:26.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:26.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: -10T04:31:26.034236+0000 2021-02-10T04:31:26.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: mgr.y (mgr.14140) 440 : cluster [DBG] 2021-02-10T04:31:26.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[29319]: pgmap v552: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 172 MiB data, 550 MiB used, 715 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.6 MiB/s wr, 14 op/s 2021-02-10T04:31:26.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: cluster 2021-02-10T04:31:26.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: 2021-02-10T04:31:26.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: 2021-02-10T04:31:24. 2021-02-10T04:31:26.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: 907887+0000 osd.2 (osd.2) 118 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:26.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:26.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: -10T04:31:24.907895+0000 osd.2 ( 2021-02-10T04:31:26.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: osd.2) 119 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:26.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:26.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: -02-10T04:31:24.907905+0000 2021-02-10T04:31:26.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: osd.2 (osd.2) 120 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:26.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: 2021-02-10T04:31:26.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: cluster 2021-02-10T04:31:26.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: 2021-02-10T04:31:24.907911+0000 osd.2 (osd. 2021-02-10T04:31:26.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: 2) 121 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:26.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:26.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: -10T04:31:24.907916+0000 osd.2 2021-02-10T04:31:26.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: (osd.2) 122 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:26.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:26.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: 04:31:24.907922+0000 osd.2 (osd 2021-02-10T04:31:26.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: .2) 123 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:26.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:26.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: T04:31:24.907928+0000 osd.2 (osd.2) 124 2021-02-10T04:31:26.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:26.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:26.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: 10T04:31:24.907933+0000 osd.2 (osd 2021-02-10T04:31:26.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: .2) 125 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:26.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: cluster 2021-02-10T04:31: 2021-02-10T04:31:26.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: 25.211533+0000 osd.1 (osd.1) 136 : cluster [WRN] 2021-02-10T04:31:26.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:26.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: cluster 2021-02-10T04:31:26.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: 2021-02-10T04:31:25.211542+0000 osd.1 (osd.1) 137 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:26.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: cluster 2021-02-10T04:31 2021-02-10T04:31:26.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: :25.211550+0000 osd.1 (osd.1) 138 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:26.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: 2021-02-10T04:31:26.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: cluster 2021-02-10T04:31:26.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: 2021-02-10T04:31:25.211555+0000 osd.1 (osd.1) 2021-02-10T04:31:26.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: 139 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:26.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: cluster 2021-02-10T04:31:26.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: 2021-02-10T04:31:25.211561+0000 osd.1 (osd.1) 140 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:26.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: cluster 2021-02-10T04:31:25.211573+0000 osd.1 2021-02-10T04:31:26.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: (osd.1) 141 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:26.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:26.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: 10T04:31:25.230756+0000 mon.a (mon.0) 1000 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2021-02-10T04:31:26.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:26.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: 02-10T04:31:25.396993+0000 osd.4 (osd.4) 175 2021-02-10T04:31:26.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:26.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:26.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: 10T04:31:25.397000+0000 osd.4 ( 2021-02-10T04:31:26.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: osd.4) 176 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:26.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:26.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: -02-10T04:31:25 2021-02-10T04:31:26.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: .397005+0000 osd.4 (osd.4) 177 : cluster 2021-02-10T04:31:26.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:26.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:26.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: 31:25.397010+0000 osd.4 (osd 2021-02-10T04:31:26.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: .4) 178 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:26.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: cluster 2021-02-10T04:31:26.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: 2021-02-10T04:31:26. 2021-02-10T04:31:26.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: 034236+0000 mgr.y (mgr.14140) 440 : cluster 2021-02-10T04:31:26.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[35961]: [DBG] pgmap v552: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 172 MiB data, 550 MiB used, 715 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.6 MiB/s wr, 14 op/s 2021-02-10T04:31:26.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[33677]: debug 2021-02-10T04:31:26.425+0000 7fbec19a1700 -1 osd.4 269 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:31:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: cluster 2021-02-10T04:31:24.907887+0000 2021-02-10T04:31:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: osd.2 (osd.2) 118 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: cluster 2021-02-10T04:31:24.907895+0000 osd.2 (osd.2) 119 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: cluster 2021-02-10T04:31:24.907905+0000 osd.2 (osd.2) 120 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: cluster 2021-02-10T04:31:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: 2021-02-10T04:31:24.907911+0000 2021-02-10T04:31:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: osd.2 (osd.2) 121 : cluster [WRN] 2021-02-10T04:31:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: cluster 2021-02-10T04:31: 2021-02-10T04:31:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: 24.907916+0000 osd.2 (osd.2) 122 : 2021-02-10T04:31:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: cluster 2021-02-10T04:31:24.907922+0000 2021-02-10T04:31:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: osd.2 (osd.2) 123 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: cluster 2021-02-10T04:31:24.907928 2021-02-10T04:31:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 124 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: cluster 2021-02-10T04:31:24.907933+0000 osd.2 2021-02-10T04:31:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: (osd.2) 125 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: cluster 2021-02-10T04:31:25.211533+0000 osd.1 (osd.1 2021-02-10T04:31:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: ) 136 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: 02-10T04:31:25.211542+0000 osd.1 (osd.1) 137 : cluster [WRN] 2021-02-10T04:31:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: cluster 2021-02-10T04:31:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: 2021-02-10T04:31:25.211550+0000 osd.1 (osd.1) 2021-02-10T04:31:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: 138 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: -02-10T04:31:25.211555 2021-02-10T04:31:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 139 : cluster [WRN] 2021-02-10T04:31:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: cluster 2021-02-10T04:31:25.211561+0000 2021-02-10T04:31:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: osd.1 (osd.1) 140 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: cluster 2021-02-10T04:31:25.211573+0000 osd.1 ( 2021-02-10T04:31:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: osd.1) 141 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: cluster 2021-02-10T04:31:25.230756+0000 mon.a (mon. 2021-02-10T04:31:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: 0) 1000 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2021-02-10T04:31:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: 10T04:31:25.396993+0000 osd.4 (osd.4) 175 : cluster 2021-02-10T04:31:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: cluster 2021-02-10T04:31 2021-02-10T04:31:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: :25.397000+0000 osd.4 (osd.4) 176 2021-02-10T04:31:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: cluster 2021-02-10T04:31:25. 2021-02-10T04:31:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: 397005+0000 osd.4 (osd.4) 177 : cluster [WRN] 2021-02-10T04:31:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: 10T04:31:25.397010+0000 osd.4 (osd.4) 178 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: cluster 2021-02-10T04:31:26.034236+0000 2021-02-10T04:31:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:26 smithi164 conmon[29288]: mgr.y (mgr.14140) 440 : cluster [DBG] pgmap v552: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 172 MiB data, 550 MiB used, 715 GiB / 715 GiB avail; 2.9 MiB/s rd, 3.6 MiB/s wr, 14 op/s 2021-02-10T04:31:27.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[49106]: debug 2021-02-10T04:31:27.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[49106]: 2021-02-10T04:31:26.913+0000 7f77a3e94700 -1 osd.2 269 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:31:27.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:26 smithi085 conmon[49106]: 2021-02-10T04:31:27.261 INFO:tasks.rados.rados.0.smithi085.stdout:912: finishing write tid 1 to smithi08575397-24 2021-02-10T04:31:27.261 INFO:tasks.rados.rados.0.smithi085.stdout:912: finishing write tid 2 to smithi08575397-24 2021-02-10T04:31:27.261 INFO:tasks.rados.rados.0.smithi085.stdout:912: finishing write tid 3 to smithi08575397-24 2021-02-10T04:31:27.261 INFO:tasks.rados.rados.0.smithi085.stdout:912: finishing write tid 4 to smithi08575397-24 2021-02-10T04:31:27.261 INFO:tasks.rados.rados.0.smithi085.stdout:912: finishing write tid 5 to smithi08575397-24 2021-02-10T04:31:27.261 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 24 v 121 (ObjNum 338 snap 87 seq_num 338) dirty exists 2021-02-10T04:31:27.261 INFO:tasks.rados.rados.0.smithi085.stdout:912: left oid 24 (ObjNum 338 snap 87 seq_num 338) 2021-02-10T04:31:27.261 INFO:tasks.rados.rados.0.smithi085.stdout:912: done (13 left) 2021-02-10T04:31:27.261 INFO:tasks.rados.rados.0.smithi085.stdout:913: done (12 left) 2021-02-10T04:31:27.261 INFO:tasks.rados.rados.0.smithi085.stdout:914: write (excl) oid 8 current snap is 87 2021-02-10T04:31:27.262 INFO:tasks.rados.rados.0.smithi085.stdout:914: seq_num 339 ranges {497573=628943,1687631=734777,3093218=604355} 2021-02-10T04:31:27.273 INFO:tasks.rados.rados.0.smithi085.stdout:914: writing smithi08575397-8 from 497573 to 1126516 tid 1 2021-02-10T04:31:27.280 INFO:tasks.rados.rados.0.smithi085.stdout:914: writing smithi08575397-8 from 1687631 to 2422408 tid 2 2021-02-10T04:31:27.289 INFO:tasks.rados.rados.0.smithi085.stdout:914: writing smithi08575397-8 from 3093218 to 3697573 tid 3 2021-02-10T04:31:27.289 INFO:tasks.rados.rados.0.smithi085.stdout:915: write (excl) oid 7 current snap is 87 2021-02-10T04:31:27.289 INFO:tasks.rados.rados.0.smithi085.stdout:914: finishing write tid 1 to smithi08575397-8 2021-02-10T04:31:27.289 INFO:tasks.rados.rados.0.smithi085.stdout:914: finishing write tid 2 to smithi08575397-8 2021-02-10T04:31:27.289 INFO:tasks.rados.rados.0.smithi085.stdout:915: seq_num 340 ranges {514380=704192,1629760=478435,2604288=442602} 2021-02-10T04:31:27.296 INFO:tasks.rados.rados.0.smithi085.stdout:915: writing smithi08575397-7 from 514380 to 1218572 tid 1 2021-02-10T04:31:27.301 INFO:tasks.rados.rados.0.smithi085.stdout:915: writing smithi08575397-7 from 1629760 to 2108195 tid 2 2021-02-10T04:31:27.308 INFO:tasks.rados.rados.0.smithi085.stdout:915: writing smithi08575397-7 from 2604288 to 3046890 tid 3 2021-02-10T04:31:27.308 INFO:tasks.rados.rados.0.smithi085.stdout:916: snap_remove snap 82 2021-02-10T04:31:27.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[33677]: debug 2021-02-10T04:31:27.405+0000 7fbec19a1700 -1 osd.4 269 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:31:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: cluster 2021-02-10T04:31:25.939204+0000 osd.2 (osd 2021-02-10T04:31:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: .2) 126 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: cluster 2021-02-10T04:31:25.939210+0000 osd.2 (osd.2) 127 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: cluster 2021-02-10T04:31:25 2021-02-10T04:31:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: .939213+0000 osd.2 (osd.2) 128 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: cluster 2021-02-10T04:31:25. 2021-02-10T04:31:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: 939219+0000 osd.2 (osd.2) 129 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: cluster 2021-02-10T04:31 2021-02-10T04:31:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: :25.939223+0000 osd.2 (osd.2) 130 : cluster 2021-02-10T04:31:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: cluster 2021-02-10T04:31:25. 2021-02-10T04:31:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: 939226+0000 osd.2 (osd.2) 131 : cluster [WRN] 2021-02-10T04:31:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: cluster 2021-02-10T04:31:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: 2021-02-10T04:31:25.939229+0000 osd.2 (osd 2021-02-10T04:31:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: .2) 132 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:31:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: 31:25.939232+0000 osd.2 (osd.2) 133 : cluster 2021-02-10T04:31:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: cluster 2021-02-10T04:31: 2021-02-10T04:31:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: 26.177812+0000 osd.1 (osd.1) 142 : cluster 2021-02-10T04:31:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: cluster 2021-02-10T04:31:26. 2021-02-10T04:31:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: 177823+0000 osd.1 (osd.1) 143 : cluster [WRN] 2021-02-10T04:31:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: cluster 2021-02-10T04:31:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: 2021-02-10T04:31:26.177832+0000 osd.1 (osd 2021-02-10T04:31:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: .1) 144 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: cluster 2021-02-10T04:31:26.177838 2021-02-10T04:31:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 145 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: 2021-02-10T04:31:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: 02-10T04:31:26.177847+0000 osd.1 (osd.1) 2021-02-10T04:31:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: 146 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:27.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: cluster 2021-02-10T04:31:26.177853+0000 osd.1 (osd.1) 2021-02-10T04:31:27.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: 147 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:27.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:27.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: 10T04:31: 2021-02-10T04:31:27.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: 26.235489+0000 mon.a (mon.0) 1001 : cluster 2021-02-10T04:31:27.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: [DBG] osdmap e269: 8 total, 8 up, 8 in 2021-02-10T04:31:27.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:31:27.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: 04:31:26.426546+0000 osd.4 (osd. 2021-02-10T04:31:27.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: 4) 179 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:27.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:27.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: -10T04:31:26.426554+0000 osd.4 ( 2021-02-10T04:31:27.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: osd.4) 180 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:27.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:27.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: -02-10T04:31:26.426561 2021-02-10T04:31:27.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 181 : cluster [WRN] 2021-02-10T04:31:27.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:27.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: cluster 2021-02-10T04:31:27.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: 2021-02-10T04:31:26.426565+0000 osd.4 ( 2021-02-10T04:31:27.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: osd.4) 182 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:27.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:27.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: -02-10T04:31:26.550495+0000 2021-02-10T04:31:27.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: osd.3 (osd.3) 56 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:31:27.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: 2021-02-10T04:31:27.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: cluster 2021-02-10T04:31:27.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: 2021-02-10T04:31:26.550762+0000 osd.3 (osd.3 2021-02-10T04:31:27.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[29288]: ) 57 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:31:27.691 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[45443]: debug 2021-02-10T04:31:27.691 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[45443]: 2021-02-10T04:31:27.228+0000 7f7140d73700 -1 osd.1 269 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:31:27.691 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[45443]: 2021-02-10T04:31:27.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: cluster 2021-02-10T04:31:27.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: 2021-02-10T04:31:27.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: 2021 2021-02-10T04:31:27.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: - 2021-02-10T04:31:27.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: 02 2021-02-10T04:31:27.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: - 2021-02-10T04:31:27.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: 10T04:31:25.939204+0000 2021-02-10T04:31:27.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: osd.2 (osd.2) 126 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:27.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:27.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: :31:25.939210+0000 osd.2 (osd.2) 127 : 2021-02-10T04:31:27.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:27.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: cluster 2021-02-10T04:31:27.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:31:27.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: 31:25.939213+0000 osd.2 (osd.2) 128 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:27.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:27.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: -10T04:31:25.939219+0000 osd.2 (osd.2) 129 2021-02-10T04:31:27.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:27.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:27.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: -02-10T04:31:25.939223+0000 osd.2 2021-02-10T04:31:27.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: (osd.2) 130 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:27.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:27.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: -10T04:31:25.939226+0000 osd.2 2021-02-10T04:31:27.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: (osd.2) 131 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:27.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:27.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: 10T04:31:25.939229+0000 osd.2 ( 2021-02-10T04:31:27.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: osd.2) 132 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:27.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:27.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: -10T04:31:25.939232+0000 osd.2 ( 2021-02-10T04:31:27.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: osd.2) 133 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:27.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:27.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: 10T04:31:26.177812+0000 osd.1 (osd 2021-02-10T04:31:27.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: .1) 142 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:27.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:27.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: 04:31:26.177823+0000 osd.1 (osd. 2021-02-10T04:31:27.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: 1) 143 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:27.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: cluster 2021-02-10T04:31:27.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: 2021-02-10T04:31:26. 2021-02-10T04:31:27.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: 177832+0000 osd.1 (osd.1) 144 : cluster 2021-02-10T04:31:27.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:27.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:27.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: T04:31:26.177838+0000 osd.1 (osd 2021-02-10T04:31:27.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: .1) 145 : cluster [WRN] 2021-02-10T04:31:27.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:27.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:27.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: 02-10T04:31:26.177847+0000 osd.1 2021-02-10T04:31:27.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: (osd.1) 146 : cluster [WRN] 2021-02-10T04:31:27.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:27.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: cluster 2021-02-10T04:31:27.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: 2021-02-10T04:31:26.177853+0000 2021-02-10T04:31:27.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: osd.1 (osd.1) 147 : cluster [WRN] 2021-02-10T04:31:27.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:27.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: cluster 2021-02-10T04:31:27.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:31:27.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: T04:31:26.235489+0000 2021-02-10T04:31:27.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: mon.a (mon.0) 1001 : 2021-02-10T04:31:27.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2021-02-10T04:31:27.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:27.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: 10T04:31:26.426546+0000 osd.4 2021-02-10T04:31:27.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: (osd.4) 179 : cluster [WRN] 2021-02-10T04:31:27.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:27.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: cluster 2021-02-10T04:31 2021-02-10T04:31:27.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: :26.426554+0000 osd.4 (osd.4) 180 : 2021-02-10T04:31:27.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:27.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:27.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: T04:31:26.426561+0000 osd.4 (osd.4 2021-02-10T04:31:27.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: ) 181 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:27.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:27.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: 02-10T04:31:26.426565+0000 osd.4 ( 2021-02-10T04:31:27.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: osd.4) 182 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:27.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:27.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: -02-10T04:31:26.550495+0000 2021-02-10T04:31:27.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: osd.3 (osd.3) 56 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:31:27.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:27.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: -10T04:31:26.550762+0000 osd.3 2021-02-10T04:31:27.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: (osd.3) 57 : cluster 2021-02-10T04:31:27.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[35961]: [DBG] purged_snaps scrub ok 2021-02-10T04:31:27.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: cluster 2021-02-10T04:31:25.939204+0000 osd.2 (osd.2) 2021-02-10T04:31:27.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: 126 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:27.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: cluster 2021-02-10T04:31:25.939210+0000 osd.2 (osd.2) 127 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:27.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: cluster 2021-02-10T04:31:25 2021-02-10T04:31:27.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: .939213+0000 osd.2 (osd.2) 128 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:27.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:27.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: 10T04:31:25.939219+0000 osd.2 (osd. 2021-02-10T04:31:27.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: 2) 129 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:27.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:31:27.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: :31:25.939223+0000 osd.2 (osd.2) 130 : 2021-02-10T04:31:27.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:27.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:31:27.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: :31:25.939226+0000 osd.2 2021-02-10T04:31:27.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: (osd.2) 131 : cluster [WRN] 2021-02-10T04:31:27.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:27.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:27.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: 10T04:31:25.939229+0000 osd.2 ( 2021-02-10T04:31:27.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: osd.2) 132 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:27.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:27.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: T04:31:25.939232+0000 osd.2 (osd.2 2021-02-10T04:31:27.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: ) 133 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:27.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: cluster 2021-02-10T04:31:26 2021-02-10T04:31:27.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: .177812+0000 osd.1 (osd.1) 142 : cluster [WRN] 2021-02-10T04:31:27.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:27.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: cluster 2021-02-10T04:31:27.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: 2021-02-10T04:31:26.177823+0000 2021-02-10T04:31:27.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: osd.1 (osd.1) 143 : 2021-02-10T04:31:27.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:27.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:31:27.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: 04:31:26.177832+0000 osd.1 (osd. 2021-02-10T04:31:27.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: 1) 144 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:27.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:27.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: 02-10T04:31:26 2021-02-10T04:31:27.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: .177838+0000 osd.1 (osd.1) 145 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:27.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:27.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: T04:31:26.177847+0000 osd.1 (osd.1) 146 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:27.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:31:27.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: :31:26.177853+0000 osd.1 (osd.1) 147 2021-02-10T04:31:27.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:27.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: cluster 2021-02-10T04:31:26. 2021-02-10T04:31:27.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: 235489+0000 mon.a (mon.0) 1001 2021-02-10T04:31:27.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2021-02-10T04:31:27.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: cluster 2021-02-10T04:31:27.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: 2021-02-10T04:31:26 2021-02-10T04:31:27.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: .426546+0000 osd.4 (osd.4) 179 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:27.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:27.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: -02-10T04:31:26.426554+0000 2021-02-10T04:31:27.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: osd.4 (osd.4) 180 : cluster 2021-02-10T04:31:27.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:27.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:27.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: 02-10T04:31:26.426561+0000 2021-02-10T04:31:27.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: osd.4 (osd.4) 181 : 2021-02-10T04:31:27.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:27.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: cluster 2021-02-10T04:31:27.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:31:27.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: 04:31:26.426565+0000 osd.4 (osd.4) 182 : cluster [WRN] 2021-02-10T04:31:27.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:27.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: cluster 2021-02-10T04:31 2021-02-10T04:31:27.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: :26.550495+0000 osd.3 (osd.3) 56 : 2021-02-10T04:31:27.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: cluster [DBG] purged_snaps scrub starts 2021-02-10T04:31:27.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:27.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: T04:31:26.550762+0000 osd.3 (osd. 2021-02-10T04:31:27.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[29319]: 3) 57 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:31:28.143 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:31:27 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:31:28.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:27 smithi085 conmon[49106]: debug 2021-02-10T04:31:27.922+0000 7f77a3e94700 -1 osd.2 270 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:31:28.271 INFO:tasks.rados.rados.0.smithi085.stdout:914: finishing write tid 3 to smithi08575397-8 2021-02-10T04:31:28.271 INFO:tasks.rados.rados.0.smithi085.stdout:914: finishing write tid 4 to smithi08575397-8 2021-02-10T04:31:28.271 INFO:tasks.rados.rados.0.smithi085.stdout:914: finishing write tid 5 to smithi08575397-8 2021-02-10T04:31:28.271 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 259 (ObjNum 339 snap 87 seq_num 339) dirty exists 2021-02-10T04:31:28.272 INFO:tasks.rados.rados.0.smithi085.stdout:914: left oid 8 (ObjNum 339 snap 87 seq_num 339) 2021-02-10T04:31:28.272 INFO:tasks.rados.rados.0.smithi085.stdout:915: finishing write tid 1 to smithi08575397-7 2021-02-10T04:31:28.272 INFO:tasks.rados.rados.0.smithi085.stdout:915: finishing write tid 2 to smithi08575397-7 2021-02-10T04:31:28.272 INFO:tasks.rados.rados.0.smithi085.stdout:915: finishing write tid 3 to smithi08575397-7 2021-02-10T04:31:28.272 INFO:tasks.rados.rados.0.smithi085.stdout:915: finishing write tid 4 to smithi08575397-7 2021-02-10T04:31:28.272 INFO:tasks.rados.rados.0.smithi085.stdout:915: finishing write tid 5 to smithi08575397-7 2021-02-10T04:31:28.272 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 442 (ObjNum 340 snap 87 seq_num 340) dirty exists 2021-02-10T04:31:28.272 INFO:tasks.rados.rados.0.smithi085.stdout:915: left oid 7 (ObjNum 340 snap 87 seq_num 340) 2021-02-10T04:31:28.272 INFO:tasks.rados.rados.0.smithi085.stdout:914: done (14 left) 2021-02-10T04:31:28.272 INFO:tasks.rados.rados.0.smithi085.stdout:915: done (13 left) 2021-02-10T04:31:28.272 INFO:tasks.rados.rados.0.smithi085.stdout:916: done (12 left) 2021-02-10T04:31:28.272 INFO:tasks.rados.rados.0.smithi085.stdout:917: read oid 38 snap -1 2021-02-10T04:31:28.272 INFO:tasks.rados.rados.0.smithi085.stdout:917: expect (ObjNum 287 snap 71 seq_num 287) 2021-02-10T04:31:28.272 INFO:tasks.rados.rados.0.smithi085.stdout:918: snap_remove snap 84 2021-02-10T04:31:28.469 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: cluster 2021-02-10T04:31:28.469 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: 2021-02-10T04:31:28.469 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: 2021 2021-02-10T04:31:28.469 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: -02-10T04:31:26. 2021-02-10T04:31:28.469 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: 914005+0000 osd.2 (osd.2) 134 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:28.470 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:28.470 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: T04:31:26.914016+0000 osd.2 (osd.2) 135 2021-02-10T04:31:28.470 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:28.470 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: cluster 2021-02-10T04:31 2021-02-10T04:31:28.470 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: :26.914022+0000 osd.2 (osd.2) 136 2021-02-10T04:31:28.470 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:28.470 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:28.470 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: 31:26.914031+0000 osd.2 (osd.2) 137 2021-02-10T04:31:28.470 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:28.470 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:28.470 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: :31:26.914041+0000 osd.2 (osd.2 2021-02-10T04:31:28.470 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: ) 138 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:28.471 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:28.471 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: 02-10T04:31:26.914047+0000 2021-02-10T04:31:28.471 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: osd.2 (osd.2) 139 : cluster [WRN] 2021-02-10T04:31:28.471 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:28.471 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: cluster 2021-02-10T04:31:28.471 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: 2021-02-10T04:31:26.914054+0000 osd.2 (osd 2021-02-10T04:31:28.471 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: .2) 140 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:28.471 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:28.471 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: 10T04:31:26.914073+0000 osd.2 ( 2021-02-10T04:31:28.471 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: osd.2) 141 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:28.471 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:28.471 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: -02-10T04:31:27.228539+0000 2021-02-10T04:31:28.471 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: osd.1 (osd.1) 148 : cluster [WRN] 2021-02-10T04:31:28.471 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:28.472 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: cluster 2021-02-10T04:31:28.472 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: 2021-02-10T04:31:27.228555+0000 osd.1 (osd. 2021-02-10T04:31:28.472 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: 1) 149 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:28.472 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:28.472 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: 10T04:31:27.228561 2021-02-10T04:31:28.472 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 150 : cluster [WRN] 2021-02-10T04:31:28.472 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:28.472 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: cluster 2021-02-10T04:31 2021-02-10T04:31:28.472 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: :27.228565+0000 osd.1 (osd.1) 2021-02-10T04:31:28.472 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: 151 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:28.472 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: cluster 2021-02-10T04:31:27.228582+0000 osd.1 2021-02-10T04:31:28.472 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: (osd.1) 152 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:28.472 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:28.473 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: 10T04:31:27.228588+0000 osd.1 (osd 2021-02-10T04:31:28.473 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: .1) 153 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:28.473 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:28.473 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: -10T04:31:27.257413+0000 mon.a ( 2021-02-10T04:31:28.473 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: mon.0) 1002 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2021-02-10T04:31:28.473 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:28.473 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: 02-10T04:31:27.405829+0000 osd.4 ( 2021-02-10T04:31:28.473 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: osd.4) 183 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:28.473 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:28.473 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: T04:31:27.405836+0000 osd.4 (osd. 2021-02-10T04:31:28.473 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: 4) 184 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:28.473 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:28.473 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: 04:31:27.405841+0000 osd.4 (osd. 2021-02-10T04:31:28.474 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: 4) 185 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:28.474 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:28.474 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: T04:31:27.405844+0000 osd.4 (osd. 2021-02-10T04:31:28.474 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: 4) 186 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:28.474 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:31:28.474 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: T04:31:27.845072+0000 mgr.y (mgr 2021-02-10T04:31:28.474 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: .14140) 441 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:31:28.474 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:28.474 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: 02-10T04:31:28.034716+0000 2021-02-10T04:31:28.474 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: mgr.y (mgr.14140) 442 : cluster [DBG] pgmap v555: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 172 MiB data, 549 MiB used, 715 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 12 op/s 2021-02-10T04:31:28.474 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: 2021-02-10T04:31:28.474 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:28.474 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: T04:31:28.073234+0000 mon.a ( 2021-02-10T04:31:28.475 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[35961]: mon.0) 1003 : cluster [WRN] Health check update: 18 slow ops, oldest one blocked for 132 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:31:28.475 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[45443]: debug 2021-02-10T04:31:28.196+0000 7f7140d73700 -1 osd.1 270 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:31:28.475 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: cluster 2021-02-10T04:31:26.914005+0000 osd.2 (osd.2 2021-02-10T04:31:28.476 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: ) 134 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:28.476 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: cluster 2021-02-10T04:31:26.914016+0000 osd.2 (osd.2) 135 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:28.476 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: cluster 2021-02-10T04:31:26.914022+0000 osd.2 (osd.2) 136 : cluster 2021-02-10T04:31:28.476 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:28.476 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: cluster 2021-02-10T04:31:28.476 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:31:28.476 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: 31:26.914031+0000 osd.2 (osd.2) 137 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:28.476 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:28.476 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: -02-10T04:31:26.914041 2021-02-10T04:31:28.476 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 138 : cluster 2021-02-10T04:31:28.476 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:28.476 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: cluster 2021-02-10T04:31: 2021-02-10T04:31:28.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: 26.914047+0000 osd.2 (osd.2) 139 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:28.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:28.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: 10T04:31:26.914054+0000 osd.2 (osd.2 2021-02-10T04:31:28.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: ) 140 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:28.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:28.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: T04:31:26.914073+0000 osd.2 (osd.2) 141 2021-02-10T04:31:28.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:28.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: cluster 2021-02-10T04:31:27. 2021-02-10T04:31:28.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: 228539+0000 osd.1 (osd.1) 148 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:28.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:28.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: 10T04:31:27.228555+0000 osd.1 (osd.1) 149 2021-02-10T04:31:28.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:28.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: cluster 2021-02-10T04:31 2021-02-10T04:31:28.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: :27.228561+0000 osd.1 (osd.1) 150 2021-02-10T04:31:28.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:28.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:28.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: -10T04:31 2021-02-10T04:31:28.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: :27.228565+0000 osd.1 2021-02-10T04:31:28.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: (osd.1) 151 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:28.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: 2021-02-10T04:31:28.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: cluster 2021-02-10T04:31:28.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:31:28.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: :31:27.228582+0000 osd.1 (osd. 2021-02-10T04:31:28.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: 1) 152 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:28.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: cluster 2021-02-10T04:31:28.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: 2021-02-10T04:31:28.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: 2021-02-10T04:31 2021-02-10T04:31:28.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: :27.228588+0000 osd.1 (osd.1) 153 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:28.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: cluster 2021-02-10T04:31:28.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: 2021-02-10T04:31:28.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: 2021 2021-02-10T04:31:28.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: - 2021-02-10T04:31:28.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: 02 2021-02-10T04:31:28.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: - 2021-02-10T04:31:28.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: 10 2021-02-10T04:31:28.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: T 2021-02-10T04:31:28.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: 04 2021-02-10T04:31:28.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: :31:27.257413+0000 mon.a (mon.0 2021-02-10T04:31:28.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: ) 1002 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2021-02-10T04:31:28.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:31:28.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: 04:31:27.405829+0000 osd.4 (osd. 2021-02-10T04:31:28.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: 4) 183 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:28.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: 2021-02-10T04:31:28.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: cluster 2021-02-10T04:31 2021-02-10T04:31:28.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: :27.405836+0000 osd.4 (osd.4) 184 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:28.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:28.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: -02-10T04:31:27.405841+0000 2021-02-10T04:31:28.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: osd.4 (osd.4) 185 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:28.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: 2021-02-10T04:31:28.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: cluster 2021-02-10T04:31:28.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: 2021-02-10T04:31:27.405844+0000 osd.4 ( 2021-02-10T04:31:28.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: osd.4) 186 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:28.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: audit 2021 2021-02-10T04:31:28.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: -02-10T04:31:27.845072+0000 2021-02-10T04:31:28.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: mgr.y (mgr.14140) 441 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:31:28.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: 2021-02-10T04:31:28.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: cluster 2021-02-10T04:31:28.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: 2021-02-10T04:31:28.034716+0000 mgr.y 2021-02-10T04:31:28.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: (mgr.14140) 442 : cluster [DBG] pgmap v555: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 172 MiB data, 549 MiB used, 715 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 12 op/s 2021-02-10T04:31:28.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:28.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: 02-10T04:31:28.073234+0000 2021-02-10T04:31:28.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: mon.a (mon.0) 1003 : cluster [WRN] 2021-02-10T04:31:28.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[29319]: Health check update: 18 slow ops, oldest one blocked for 132 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:31:28.490 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:31:28.494 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.0 dump_historic_ops 2021-02-10T04:31:28.519 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: cluster 2021-02-10T04:31:28.519 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: 2021-02-10T04:31:28.519 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: 2021-02-10T04:31:26 2021-02-10T04:31:28.519 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: .914005+0000 osd.2 (osd.2) 134 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:28.520 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:31:28.520 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: 31:26.914016+0000 osd.2 (osd.2) 135 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:28.520 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: 2021-02-10T04:31:28.520 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: cluster 2021-02-10T04:31:28.520 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: 2021-02-10T04:31:26.914022+0000 osd.2 (osd 2021-02-10T04:31:28.520 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: .2) 136 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:28.520 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:28.520 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: -10T04:31:26.914031+0000 osd.2 (osd.2) 2021-02-10T04:31:28.520 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: 137 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:28.520 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: cluster 2021-02-10T04:31:26.914041+0000 osd.2 ( 2021-02-10T04:31:28.520 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: osd.2) 138 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:28.521 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: cluster 2021-02-10T04:31:26.914047+0000 osd.2 ( 2021-02-10T04:31:28.521 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: osd.2) 139 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:28.521 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: cluster 2021-02-10T04:31:26.914054+0000 osd.2 (osd.2) 140 : cluster [WRN] 2021-02-10T04:31:28.521 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:28.521 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:28.521 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: 10T04:31:26.914073+0000 osd.2 (osd.2) 141 : cluster [WRN] 2021-02-10T04:31:28.521 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:28.521 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: cluster 2021-02-10T04:31:27. 2021-02-10T04:31:28.521 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: 228539+0000 osd.1 (osd.1) 148 : cluster [WRN] 2021-02-10T04:31:28.521 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:28.521 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:31:28.521 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: :31:27.228555+0000 osd.1 (osd.1) 149 : cluster 2021-02-10T04:31:28.521 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:28.522 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: cluster 2021-02-10T04:31:28.522 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: 2021-02-10T04:31:27.228561+0000 osd.1 (osd.1) 150 : cluster 2021-02-10T04:31:28.522 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:28.522 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:28.522 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: 10T04:31:27. 2021-02-10T04:31:28.522 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: 228565+0000 osd.1 (osd.1) 151 : cluster 2021-02-10T04:31:28.522 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:28.522 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:28.522 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: -10T04:31:27.228582 2021-02-10T04:31:28.522 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 152 : 2021-02-10T04:31:28.522 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:28.522 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: cluster 2021-02-10T04:31:28.523 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: 2021-02 2021-02-10T04:31:28.523 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: -10T04:31:27.228588+0000 osd.1 (osd. 2021-02-10T04:31:28.523 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: 1) 153 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:28.523 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: 2021-02-10T04:31:28.523 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:28.523 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: -10T04:31:27.257413+0000 mon.a ( 2021-02-10T04:31:28.523 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: mon.0) 1002 : cluster 2021-02-10T04:31:28.523 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: [DBG] osdmap e270: 8 total, 8 up, 8 in 2021-02-10T04:31:28.523 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: 2021-02-10T04:31:28.523 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:28.523 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: -02-10T04:31:27.405829+0000 2021-02-10T04:31:28.523 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: osd.4 (osd.4) 183 : cluster [WRN] 2021-02-10T04:31:28.524 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:28.524 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: cluster 2021-02-10T04:31:28.524 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: 2021-02-10T04:31 2021-02-10T04:31:28.524 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: :27.405836+0000 osd.4 (osd 2021-02-10T04:31:28.524 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: .4) 184 : cluster [WRN] 2021-02-10T04:31:28.524 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:28.524 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:28.524 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: 10T04:31:27.405841+0000 osd.4 (osd.4) 185 : 2021-02-10T04:31:28.524 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:28.524 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: cluster 2021-02-10T04:31:27.405844+0000 2021-02-10T04:31:28.524 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: osd.4 (osd.4) 186 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:28.524 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: audit 2021-02-10T04:31:27.845072+0000 2021-02-10T04:31:28.525 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: mgr.y (mgr.14140) 441 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:31:28.525 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: cluster 2021-02-10T04:31:28.034716 2021-02-10T04:31:28.525 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 442 : cluster [DBG] pgmap v555: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 172 MiB data, 549 MiB used, 715 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 12 op/s 2021-02-10T04:31:28.525 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:28.525 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: -10T04:31:28.073234+0000 mon.a 2021-02-10T04:31:28.525 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: (mon.0) 1003 : cluster 2021-02-10T04:31:28.525 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[29288]: [WRN] Health check update: 18 slow ops, oldest one blocked for 132 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:31:28.525 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[33677]: debug 2021-02-10T04:31:28.451+0000 7fbec19a1700 -1 osd.4 271 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:31:28.526 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:28.834 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:31:28.863 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:31:28.866 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:31:28 smithi164 conmon[40892]: debug 2021-02-10T04:31:28.590+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:28.968 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:31:29.001 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[49106]: debug 2021-02-10T04:31:29.001 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[49106]: 2021-02-10T04:31:28.972+0000 7f77a3e94700 -1 osd.2 271 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:31:29.001 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:28 smithi085 conmon[49106]: 2021-02-10T04:31:29.004 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:29.134 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:31:29.240 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:31:29.273 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[52836]: debug 2021-02-10T04:31:29.097+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:29.273 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[45443]: debug 2021-02-10T04:31:29.167+0000 7f7140d73700 -1 osd.1 271 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:31:29.277 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:29.279 INFO:tasks.rados.rados.0.smithi085.stdout:918: done (13 left) 2021-02-10T04:31:29.279 INFO:tasks.rados.rados.0.smithi085.stdout:919: write oid 50 current snap is 87 2021-02-10T04:31:29.279 INFO:tasks.rados.rados.0.smithi085.stdout:919: seq_num 341 ranges {531187=779441,2088242=738446,3431711=299476} 2021-02-10T04:31:29.290 INFO:tasks.rados.rados.0.smithi085.stdout:919: writing smithi08575397-50 from 531187 to 1310628 tid 1 2021-02-10T04:31:29.305 INFO:tasks.rados.rados.0.smithi085.stdout:919: writing smithi08575397-50 from 2088242 to 2826688 tid 2 2021-02-10T04:31:29.312 INFO:tasks.rados.rados.0.smithi085.stdout:919: writing smithi08575397-50 from 3431711 to 3731187 tid 3 2021-02-10T04:31:29.312 INFO:tasks.rados.rados.0.smithi085.stdout:920: snap_remove snap 85 2021-02-10T04:31:29.442 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:31:29.528 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: cluster 2021-02-10T04:31:29.528 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 2021-02-10T04:31:29.528 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 2021 2021-02-10T04:31:29.528 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: - 2021-02-10T04:31:29.528 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 02 2021-02-10T04:31:29.528 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: - 2021-02-10T04:31:29.528 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 10 2021-02-10T04:31:29.529 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: T 2021-02-10T04:31:29.529 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 04 2021-02-10T04:31:29.529 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: : 2021-02-10T04:31:29.529 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 31 2021-02-10T04:31:29.529 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: : 2021-02-10T04:31:29.529 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 27 2021-02-10T04:31:29.529 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: . 2021-02-10T04:31:29.529 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 923294 2021-02-10T04:31:29.529 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: +0000 2021-02-10T04:31:29.529 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 2021-02-10T04:31:29.529 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: osd.2 2021-02-10T04:31:29.529 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: ( 2021-02-10T04:31:29.529 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: osd 2021-02-10T04:31:29.530 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: . 2021-02-10T04:31:29.530 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 2 2021-02-10T04:31:29.530 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: ) 2021-02-10T04:31:29.530 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 142 2021-02-10T04:31:29.530 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: : 2021-02-10T04:31:29.530 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: cluster 2021-02-10T04:31:29.530 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 2021-02-10T04:31:29.530 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: [WRN] 2021-02-10T04:31:29.530 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 2021-02-10T04:31:29.530 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:29.530 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 2021-02-10T04:31:29.530 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: cluster 2021-02-10T04:31:29.530 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 2021-02-10T04:31:29.531 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 2021 2021-02-10T04:31:29.531 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: - 2021-02-10T04:31:29.531 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 02 2021-02-10T04:31:29.531 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: - 2021-02-10T04:31:29.531 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 10 2021-02-10T04:31:29.531 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: T 2021-02-10T04:31:29.531 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 04 2021-02-10T04:31:29.531 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: : 2021-02-10T04:31:29.531 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 31 2021-02-10T04:31:29.531 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: : 2021-02-10T04:31:29.531 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 27 2021-02-10T04:31:29.531 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: . 2021-02-10T04:31:29.531 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 923317 2021-02-10T04:31:29.531 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: +0000 2021-02-10T04:31:29.532 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 2021-02-10T04:31:29.532 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: osd.2 (osd.2) 143 : cluster [WRN] 2021-02-10T04:31:29.532 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:29.532 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:29.532 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: T04:31:27.923326+0000 osd.2 (osd.2) 144 : cluster 2021-02-10T04:31:29.532 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:29.532 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: cluster 2021-02-10T04:31:27 2021-02-10T04:31:29.532 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: .923332+0000 osd.2 (osd.2) 145 : cluster [WRN] 2021-02-10T04:31:29.532 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:29.532 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: cluster 2021-02-10T04:31:27.923338+0000 osd.2 2021-02-10T04:31:29.532 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: (osd.2) 146 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:29.532 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: cluster 2021-02-10T04:31:27.923348+0000 2021-02-10T04:31:29.533 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: osd.2 (osd.2) 147 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:29.533 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:29.533 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: -10T04:31:27.923354+0000 osd.2 (osd 2021-02-10T04:31:29.533 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: .2) 148 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:29.534 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:29.534 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 10T04:31:27.923359+0000 osd.2 (osd.2 2021-02-10T04:31:29.534 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: ) 149 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:29.534 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:31:29.534 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 31:28.196873+0000 osd.1 (osd.1) 154 : cluster 2021-02-10T04:31:29.534 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:29.534 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: cluster 2021-02-10T04:31 2021-02-10T04:31:29.534 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: :28.196883+0000 osd.1 (osd.1) 155 : 2021-02-10T04:31:29.534 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:29.534 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:29.534 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: T04:31:28.196890+0000 osd.1 (osd.1) 156 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:29.534 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:29.535 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: -02-10T04:31:28.196895+0000 2021-02-10T04:31:29.535 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: osd.1 (osd.1) 157 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:29.535 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 2021-02-10T04:31:29.535 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: cluster 2021-02-10T04:31:29.535 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 2021-02-10T04:31:28.196916+0000 osd.1 2021-02-10T04:31:29.535 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: (osd.1) 158 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:29.535 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:29.535 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: -10T04:31:28.196923+0000 osd.1 (osd 2021-02-10T04:31:29.535 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: .1) 159 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:29.535 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:29.536 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 02-10T04:31:28.271451+0000 2021-02-10T04:31:29.536 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: mon.a (mon.0) 1004 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2021-02-10T04:31:29.536 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:31:29.536 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 04:31:28.451713+0000 osd.4 (osd.4) 187 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:29.536 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:29.536 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: -02-10T04:31:28.451727+0000 2021-02-10T04:31:29.536 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: osd.4 (osd.4) 188 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:29.537 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:29.537 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 02-10T04:31:28.451733+0000 osd.4 ( 2021-02-10T04:31:29.537 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: osd.4) 189 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:29.537 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: cluster 2021-02-10T04:31:29.537 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 2021-02-10T04:31:28.451739+0000 2021-02-10T04:31:29.537 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: osd.4 (osd.4) 190 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:29.537 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[29319]: 2021-02-10T04:31:29.537 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: cluster 2021-02-10T04:31:29.537 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 2021-02-10T04:31:29.537 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 2021 2021-02-10T04:31:29.537 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: - 2021-02-10T04:31:29.537 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 02 2021-02-10T04:31:29.538 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: - 2021-02-10T04:31:29.538 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 10 2021-02-10T04:31:29.538 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: T 2021-02-10T04:31:29.538 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 04 2021-02-10T04:31:29.538 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: : 2021-02-10T04:31:29.538 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 31 2021-02-10T04:31:29.538 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: : 2021-02-10T04:31:29.538 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 27 2021-02-10T04:31:29.538 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: . 2021-02-10T04:31:29.538 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 923294 2021-02-10T04:31:29.538 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: +0000 2021-02-10T04:31:29.538 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 2021-02-10T04:31:29.539 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: osd.2 2021-02-10T04:31:29.539 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: ( 2021-02-10T04:31:29.539 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: osd 2021-02-10T04:31:29.539 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: . 2021-02-10T04:31:29.539 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 2 2021-02-10T04:31:29.539 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: ) 2021-02-10T04:31:29.539 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 142 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:29.539 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:29.539 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: :31:27.923317+0000 osd.2 (osd.2) 143 : cluster 2021-02-10T04:31:29.539 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:29.539 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: cluster 2021-02-10T04:31:27.923326+0000 osd.2 2021-02-10T04:31:29.540 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: (osd.2) 144 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:29.540 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:29.540 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: -10T04:31:27.923332+0000 osd.2 ( 2021-02-10T04:31:29.540 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: osd.2) 145 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:29.540 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:29.540 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: -10T04:31:27.923338+0000 osd.2 ( 2021-02-10T04:31:29.540 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: osd.2) 146 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:29.540 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:29.540 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: T04:31:27.923348+0000 osd.2 (osd. 2021-02-10T04:31:29.540 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 2) 147 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:29.540 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:29.540 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: T04:31:27.923354+0000 osd.2 (osd.2) 2021-02-10T04:31:29.540 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 148 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:29.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: cluster 2021-02-10T04:31:27.923359 2021-02-10T04:31:29.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 149 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:29.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 2021-02-10T04:31:29.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:29.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: -02-10T04:31:28.196873+0000 osd.1 (osd.1 2021-02-10T04:31:29.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: ) 154 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:29.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: cluster 2021-02-10T04:31:28 2021-02-10T04:31:29.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: .196883+0000 osd.1 (osd.1) 155 : cluster 2021-02-10T04:31:29.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:29.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: cluster 2021-02-10T04:31:28. 2021-02-10T04:31:29.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 196890+0000 osd.1 (osd.1) 156 : cluster 2021-02-10T04:31:29.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:29.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: cluster 2021-02-10T04:31: 2021-02-10T04:31:29.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 28.196895+0000 osd.1 (osd.1) 157 : cluster 2021-02-10T04:31:29.542 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:29.542 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: cluster 2021-02-10T04:31:28. 2021-02-10T04:31:29.542 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 196916+0000 osd.1 (osd.1) 158 : cluster [WRN] 2021-02-10T04:31:29.542 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:29.542 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: cluster 2021-02-10T04:31:28 2021-02-10T04:31:29.542 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: .196923+0000 osd.1 (osd.1) 159 : cluster 2021-02-10T04:31:29.542 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:29.542 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: cluster 2021-02-10T04:31: 2021-02-10T04:31:29.542 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 28.271451+0000 mon.a (mon.0 2021-02-10T04:31:29.542 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: ) 2021-02-10T04:31:29.542 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 1004 2021-02-10T04:31:29.542 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: : 2021-02-10T04:31:29.542 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: cluster 2021-02-10T04:31:29.543 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 2021-02-10T04:31:29.543 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: [DBG] 2021-02-10T04:31:29.543 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 2021-02-10T04:31:29.543 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: osdmap e271: 8 total, 8 up, 8 in 2021-02-10T04:31:29.543 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 2021-02-10T04:31:29.543 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: cluster 2021-02-10T04:31:29.543 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 2021-02-10T04:31:29.543 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 2021 2021-02-10T04:31:29.543 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: - 2021-02-10T04:31:29.543 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 02 2021-02-10T04:31:29.543 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: - 2021-02-10T04:31:29.543 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 10 2021-02-10T04:31:29.543 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: T 2021-02-10T04:31:29.544 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 04 2021-02-10T04:31:29.544 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: : 2021-02-10T04:31:29.544 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 31 2021-02-10T04:31:29.544 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: : 2021-02-10T04:31:29.544 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 28 2021-02-10T04:31:29.544 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: . 2021-02-10T04:31:29.544 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 451713 2021-02-10T04:31:29.544 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 187 : cluster 2021-02-10T04:31:29.544 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:29.544 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: cluster 2021-02-10T04:31:29.544 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 2021-02-10T04:31:28.451727+0000 osd.4 (osd.4) 188 : cluster [WRN] 2021-02-10T04:31:29.544 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:29.544 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:29.544 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 04:31:28.451733+0000 osd.4 (osd.4) 189 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:29.545 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:29.545 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: 04:31:28.451739+0000 osd.4 (osd.4) 190 2021-02-10T04:31:29.545 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:29.545 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[45443]: debug 2021-02-10T04:31:29.545 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[45443]: 2021-02-10T04:31:29.417+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:29.545 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[45443]: 2021-02-10T04:31:29.547 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:31:29.581 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:29.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[33677]: debug 2021-02-10T04:31:29.477+0000 7fbec19a1700 -1 osd.4 272 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:31:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: cluster 2021-02-10T04:31:27.923294+0000 osd.2 (osd 2021-02-10T04:31:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: .2) 142 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: cluster 2021-02-10T04:31:27.923317+0000 osd.2 (osd.2) 143 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: cluster 2021-02-10T04:31:27.923326+0000 osd.2 (osd 2021-02-10T04:31:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: .2) 144 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: cluster 2021-02-10T04:31:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: 2021-02-10T04:31:27.923332+0000 2021-02-10T04:31:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: osd.2 (osd.2) 145 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: 2021-02-10T04:31:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:31:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: 04:31:27.923338+0000 osd.2 (osd. 2021-02-10T04:31:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: 2) 146 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: cluster 2021-02-10T04:31:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:31:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: 31:27.923348+0000 osd.2 (osd.2 2021-02-10T04:31:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: ) 147 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: 02-10T04:31:27.923354+0000 2021-02-10T04:31:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: osd.2 (osd.2) 148 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: 2021-02-10T04:31:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: cluster 2021-02-10T04:31:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: 2021-02-10T04:31:27.923359+0000 osd.2 (osd 2021-02-10T04:31:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: .2) 149 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: cluster 2021-02-10T04:31:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: 2021-02-10T04:31:28.196873+0000 2021-02-10T04:31:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: osd.1 (osd.1) 154 : cluster [WRN] 2021-02-10T04:31:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: cluster 2021-02-10T04:31:28 2021-02-10T04:31:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: .196883+0000 osd.1 (osd.1) 155 : cluster 2021-02-10T04:31:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: T04:31:28.196890+0000 osd.1 (osd 2021-02-10T04:31:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: .1) 156 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: cluster 2021-02-10T04:31:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: 2021-02-10T04:31:28.196895+0000 2021-02-10T04:31:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: osd.1 (osd.1) 157 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: -02-10T04:31:28.196916+0000 2021-02-10T04:31:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: osd.1 (osd.1) 158 : cluster [WRN] 2021-02-10T04:31:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: cluster 2021-02-10T04:31:28.196923 2021-02-10T04:31:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 159 : cluster [WRN] 2021-02-10T04:31:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:31:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: 31:28.271451+0000 mon.a (mon.0) 2021-02-10T04:31:29.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: 1004 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2021-02-10T04:31:29.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: cluster 2021-02-10T04:31:29.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: 2021-02-10T04:31:28.451713+0000 2021-02-10T04:31:29.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: osd.4 (osd.4) 187 : cluster [WRN] 2021-02-10T04:31:29.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:29.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: cluster 2021-02-10T04:31 2021-02-10T04:31:29.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: :28.451727+0000 osd.4 (osd.4) 188 2021-02-10T04:31:29.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:29.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:29.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: -02-10T04:31:28.451733+0000 2021-02-10T04:31:29.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: osd.4 (osd.4) 189 : cluster [WRN] 2021-02-10T04:31:29.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:29.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:31:29.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: 31:28.451739+0000 osd.4 (osd.4) 2021-02-10T04:31:29.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[29288]: 190 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:29.701 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:31:29.805 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:31:29.837 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:29.930 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[49106]: debug 2021-02-10T04:31:29.929+0000 7f77a3e94700 -1 osd.2 272 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:31:29.930 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:31:29 smithi085 conmon[41540]: debug 2021-02-10T04:31:29.670+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:30.143 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:29 smithi164 conmon[33677]: debug 2021-02-10T04:31:29.917+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:30.220 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:31:30.302 INFO:tasks.rados.rados.0.smithi085.stdout:920: done (14 left) 2021-02-10T04:31:30.302 INFO:tasks.rados.rados.0.smithi085.stdout:921: rollback oid 21 current snap is 87 2021-02-10T04:31:30.302 INFO:tasks.rados.rados.0.smithi085.stdout:921: done (14 left) 2021-02-10T04:31:30.302 INFO:tasks.rados.rados.0.smithi085.stdout:922: write oid 30 current snap is 87 2021-02-10T04:31:30.302 INFO:tasks.rados.rados.0.smithi085.stdout:922: seq_num 342 ranges {547994=454690,1746724=482104,2659134=465146,3605976=142018} 2021-02-10T04:31:30.302 INFO:tasks.rados.rados.0.smithi085.stdout:922: writing smithi08575397-30 from 547994 to 1002684 tid 1 2021-02-10T04:31:30.305 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is set 2021-02-10T04:31:30.309 INFO:tasks.rados.rados.0.smithi085.stdout:922: writing smithi08575397-30 from 1746724 to 2228828 tid 2 2021-02-10T04:31:30.314 INFO:tasks.rados.rados.0.smithi085.stdout:922: writing smithi08575397-30 from 2659134 to 3124280 tid 3 2021-02-10T04:31:30.318 INFO:tasks.rados.rados.0.smithi085.stdout:922: writing smithi08575397-30 from 3605976 to 3747994 tid 4 2021-02-10T04:31:30.318 INFO:tasks.rados.rados.0.smithi085.stdout:923: snap_create 2021-02-10T04:31:30.318 INFO:tasks.rados.rados.0.smithi085.stdout:917: expect (ObjNum 287 snap 71 seq_num 287) 2021-02-10T04:31:30.322 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.7 to 1 (10000) 2021-02-10T04:31:30.324 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:31:30.355 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:30.422 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:31:30.516 INFO:tasks.rados.rados.0.smithi085.stdout:919: finishing write tid 1 to smithi08575397-50 2021-02-10T04:31:30.516 INFO:tasks.rados.rados.0.smithi085.stdout:919: finishing write tid 2 to smithi08575397-50 2021-02-10T04:31:30.516 INFO:tasks.rados.rados.0.smithi085.stdout:919: finishing write tid 3 to smithi08575397-50 2021-02-10T04:31:30.516 INFO:tasks.rados.rados.0.smithi085.stdout:919: finishing write tid 4 to smithi08575397-50 2021-02-10T04:31:30.516 INFO:tasks.rados.rados.0.smithi085.stdout:919: finishing write tid 5 to smithi08575397-50 2021-02-10T04:31:30.516 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 263 (ObjNum 341 snap 87 seq_num 341) dirty exists 2021-02-10T04:31:30.516 INFO:tasks.rados.rados.0.smithi085.stdout:919: left oid 50 (ObjNum 341 snap 87 seq_num 341) 2021-02-10T04:31:30.516 INFO:tasks.rados.rados.0.smithi085.stdout:922: finishing write tid 1 to smithi08575397-30 2021-02-10T04:31:30.516 INFO:tasks.rados.rados.0.smithi085.stdout:922: finishing write tid 2 to smithi08575397-30 2021-02-10T04:31:30.516 INFO:tasks.rados.rados.0.smithi085.stdout:922: finishing write tid 3 to smithi08575397-30 2021-02-10T04:31:30.516 INFO:tasks.rados.rados.0.smithi085.stdout:922: finishing write tid 4 to smithi08575397-30 2021-02-10T04:31:30.516 INFO:tasks.rados.rados.0.smithi085.stdout:922: finishing write tid 5 to smithi08575397-30 2021-02-10T04:31:30.517 INFO:tasks.rados.rados.0.smithi085.stdout:922: finishing write tid 6 to smithi08575397-30 2021-02-10T04:31:30.517 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 304 (ObjNum 342 snap 87 seq_num 342) dirty exists 2021-02-10T04:31:30.517 INFO:tasks.rados.rados.0.smithi085.stdout:922: left oid 30 (ObjNum 342 snap 87 seq_num 342) 2021-02-10T04:31:30.643 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[37263]: debug 2021-02-10T04:31:30.426+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:30.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[33677]: debug 2021-02-10T04:31:30.429+0000 7fbec19a1700 -1 osd.4 273 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:31:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: cluster 2021-02-10T04:31:28.973054+0000 osd.2 (osd.2) 2021-02-10T04:31:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: 150 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: cluster 2021-02-10T04:31:28.973064+0000 osd.2 (osd. 2021-02-10T04:31:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: 2) 151 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: cluster 2021-02-10T04:31:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: 2021-02-10T04:31:28.973069+0000 osd.2 (osd.2) 152 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: -02-10T04:31:28.973075+0000 osd.2 ( 2021-02-10T04:31:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: osd.2) 153 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: cluster 2021-02-10T04:31:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: 2021-02-10T04:31:28 2021-02-10T04:31:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: .973080+0000 osd.2 (osd.2) 154 2021-02-10T04:31:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: -02-10T04:31:28. 2021-02-10T04:31:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: 973086+0000 osd.2 (osd.2 2021-02-10T04:31:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: ) 155 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: cluster 2021-02-10T04:31:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: 2021-02-10T04:31: 2021-02-10T04:31:30.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: 28.973092+0000 osd.2 (osd. 2021-02-10T04:31:30.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: 2) 156 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:30.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: 2021-02-10T04:31:30.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:30.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: -10T04:31:28.973097+0000 osd.2 (osd. 2021-02-10T04:31:30.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: 2) 157 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:30.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: cluster 2021-02-10T04:31:30.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: 2021-02-10T04:31:29.167604+0000 osd.1 2021-02-10T04:31:30.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: (osd.1) 160 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:30.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:30.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: -02-10T04:31: 2021-02-10T04:31:30.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: 29.167612+0000 osd.1 (osd. 2021-02-10T04:31:30.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: 1) 161 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:30.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: 2021-02-10T04:31:30.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: cluster 2021-02-10T04:31:30.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: 2021-02-10T04:31 2021-02-10T04:31:30.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: :29.167620+0000 osd.1 (osd.1) 162 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:30.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:30.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: 10T04:31:29.167625+0000 osd.1 (osd. 2021-02-10T04:31:30.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: 1) 163 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:30.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:30.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: T04:31:29.167631+0000 osd.1 (osd. 2021-02-10T04:31:30.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: 1) 164 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:30.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: cluster 2021-02-10T04:31:29.167636+0000 osd.1 (osd.1) 165 2021-02-10T04:31:30.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:30.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:31:30.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: :31:29.275170+0000 mon.a (mon.0) 2021-02-10T04:31:30.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: 1005 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2021-02-10T04:31:30.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:31:30.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: -10T04:31:29.350942+0000 mon.a 2021-02-10T04:31:30.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: (mon.0) 1006 : audit 2021-02-10T04:31:30.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: [INF] from='client.? 172.21.15.85:0/444861482' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:31:30.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: cluster 2021-02-10T04:31:30.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: 2021-02-10T04:31: 2021-02-10T04:31:30.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: 29.477722+0000 osd.4 (osd. 2021-02-10T04:31:30.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: 4) 191 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:30.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: 2021-02-10T04:31:30.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: cluster 2021-02-10T04:31:30.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:31:30.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: :31:29.477731+0000 2021-02-10T04:31:30.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: osd.4 (osd.4) 192 : cluster 2021-02-10T04:31:30.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:30.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: cluster 2021-02-10T04:31:30.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: 2021-02-10T04:31:29.477739+0000 2021-02-10T04:31:30.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: osd.4 (osd.4) 193 : cluster 2021-02-10T04:31:30.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:30.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: cluster 2021-02-10T04:31:30.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: 2021-02-10T04:31:29.477745 2021-02-10T04:31:30.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 194 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:30.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: audit 2021-02-10T04:31:30.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: 2021-02-10T04:31:29.677712+0000 2021-02-10T04:31:30.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: mon.b (mon.2) 32 : audit [INF] from='client.? 172.21.15.85:0/3774441808' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T04:31:30.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: audit 2021-02-10T04:31:30.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: 2021-02-10T04:31:29.678057+0000 2021-02-10T04:31:30.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: mon.a (mon.0) 1007 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T04:31:30.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: 2021-02-10T04:31:30.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: cluster 2021-02-10T04:31: 2021-02-10T04:31:30.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: 30.035280+0000 mgr.y (mgr.14140) 443 : cluster 2021-02-10T04:31:30.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:30 smithi164 conmon[29288]: [DBG] pgmap v558: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 161 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 2.5 MiB/s rd, 897 KiB/s wr, 6 op/s 2021-02-10T04:31:30.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: cluster 2021-02-10T04:31:28.973054+0000 osd.2 (osd 2021-02-10T04:31:30.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: .2) 150 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:30.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: cluster 2021-02-10T04:31:28.973064+0000 osd.2 (osd.2) 151 2021-02-10T04:31:30.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:30.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: cluster 2021-02-10T04:31:28 2021-02-10T04:31:30.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: .973069+0000 osd.2 (osd.2) 152 : cluster 2021-02-10T04:31:30.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:30.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:30.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: -10T04:31:28.973075+0000 osd.2 (osd. 2021-02-10T04:31:30.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: 2) 153 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:30.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:31:30.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: 31:28.973080+0000 osd.2 (osd.2) 154 2021-02-10T04:31:30.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:30.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: cluster 2021-02-10T04:31:30.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: 2021-02-10T04:31: 2021-02-10T04:31:30.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: 28.973086+0000 osd.2 (osd.2) 155 : cluster [WRN] 2021-02-10T04:31:30.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:30.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:31:30.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: 04:31:28.973092+0000 osd.2 (osd.2) 156 : cluster [WRN] 2021-02-10T04:31:30.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:30.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: cluster 2021-02-10T04:31:28.973097 2021-02-10T04:31:30.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 157 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:30.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:30.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: -10T04:31:29.167604+0000 2021-02-10T04:31:30.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: osd.1 (osd.1) 160 : 2021-02-10T04:31:30.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:30.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:30.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: 02-10T04:31:29.167612+0000 osd.1 2021-02-10T04:31:30.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: (osd.1) 161 : cluster 2021-02-10T04:31:30.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:30.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:31:30.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: :31:29.167620+0000 osd.1 (osd.1 2021-02-10T04:31:30.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: ) 162 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:30.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:31:30.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: 04:31:29.167625 2021-02-10T04:31:30.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: +0000 osd.1 (osd.1) 163 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:30.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:30.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: -10T04:31:29.167631+0000 osd.1 2021-02-10T04:31:30.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: (osd.1) 164 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:30.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:30.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: 02-10T04:31:29.167636+0000 2021-02-10T04:31:30.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: osd.1 (osd.1) 165 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:30.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:30.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: 02-10T04:31:29.275170+0000 mon.a 2021-02-10T04:31:30.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: (mon.0) 1005 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2021-02-10T04:31:30.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: audit 2021-02-10T04:31:30.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: 2021-02-10T04:31:29.350942+0000 mon.a 2021-02-10T04:31:30.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: (mon.0) 1006 : audit [INF] 2021-02-10T04:31:30.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: from='client.? 172.21.15.85:0/444861482' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:31:30.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:30.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: 02-10T04:31:29.477722+0000 osd.4 2021-02-10T04:31:30.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: (osd.4) 191 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:30.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:30.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: -02-10T04:31:29.477731+0000 2021-02-10T04:31:30.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: osd.4 (osd.4) 192 : cluster [WRN] 2021-02-10T04:31:30.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:30.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: cluster 2021-02-10T04:31:29 2021-02-10T04:31:30.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: .477739+0000 osd.4 (osd.4) 193 : cluster 2021-02-10T04:31:30.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:30.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:30.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: 10T04:31:29.477745+0000 osd.4 (osd 2021-02-10T04:31:30.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: .4) 194 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:30.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:31:30.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: 10T04:31:29.677712+0000 mon.b (mon.2 2021-02-10T04:31:30.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: ) 32 : audit [INF] from='client.? 172.21.15.85:0/3774441808' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T04:31:30.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:31:30.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: 02-10T04:31:29.678057+0000 mon.a ( 2021-02-10T04:31:30.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: mon.0) 1007 : 2021-02-10T04:31:30.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T04:31:30.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: cluster 2021-02-10T04:31:30.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: 2021-02-10T04:31:30 2021-02-10T04:31:30.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: .035280+0000 mgr.y (mgr.14140) 443 : cluster 2021-02-10T04:31:30.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[29319]: [DBG] pgmap v558: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 161 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 2.5 MiB/s rd, 897 KiB/s wr, 6 op/s 2021-02-10T04:31:30.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: cluster 2021-02-10T04:31:28.973054 2021-02-10T04:31:30.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 150 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:30.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: cluster 2021-02-10T04:31:30.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: 2021-02-10T04:31:28.973064+0000 osd.2 (osd.2) 151 2021-02-10T04:31:30.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:30.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:30.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: 02-10T04:31:28.973069+0000 osd.2 ( 2021-02-10T04:31:30.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: osd.2) 152 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:30.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:30.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: 02-10T04:31:28.973075+0000 2021-02-10T04:31:30.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: osd.2 (osd.2) 153 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:30.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: cluster 2021-02-10T04:31:30.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: 2021-02-10T04:31:28.973080 2021-02-10T04:31:30.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 154 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:30.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: 2021-02-10T04:31:30.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:30.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: :31:28.973086+0000 osd.2 (osd.2) 2021-02-10T04:31:30.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: 155 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:30.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: cluster 2021-02-10T04:31:30.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:31:30.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: T04:31:28.973092+0000 2021-02-10T04:31:30.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: osd.2 (osd.2) 156 : cluster 2021-02-10T04:31:30.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:30.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:30.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: 04:31:28.973097+0000 osd.2 (osd.2 2021-02-10T04:31:30.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: ) 157 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:30.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:30.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: T04:31:29.167604+0000 osd.1 (osd. 2021-02-10T04:31:30.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: 1) 160 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:30.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:30.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: 04:31:29.167612+0000 osd.1 (osd.1) 2021-02-10T04:31:30.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: 161 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:30.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: cluster 2021-02-10T04:31:30.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: 2021-02-10T04:31 2021-02-10T04:31:30.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: :29.167620+0000 osd.1 (osd 2021-02-10T04:31:30.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: .1) 162 : cluster [WRN] 2021-02-10T04:31:30.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:30.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:30.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: -10T04:31:29.167625+0000 2021-02-10T04:31:30.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: osd.1 (osd.1) 163 2021-02-10T04:31:30.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:30.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: cluster 2021-02-10T04:31: 2021-02-10T04:31:30.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: 29.167631+0000 osd.1 (osd.1) 164 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:30.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: cluster 2021-02-10T04:31:30.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: 2021-02-10T04:31:29.167636+0000 osd.1 (osd.1) 165 : 2021-02-10T04:31:30.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:30.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: cluster 2021-02-10T04:31:29 2021-02-10T04:31:30.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: .275170+0000 mon.a (mon.0) 1005 : cluster [DBG] 2021-02-10T04:31:30.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: osdmap e272: 8 total, 8 up, 8 in 2021-02-10T04:31:30.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: audit 2021-02-10T04:31:29 2021-02-10T04:31:30.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: .350942+0000 mon.a (mon.0) 1006 : audit 2021-02-10T04:31:30.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: [INF] from='client.? 172.21.15.85:0/444861482' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:31:30.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: cluster 2021-02-10T04:31:29 2021-02-10T04:31:30.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: .477722+0000 osd.4 (osd.4) 191 : cluster [WRN] 2021-02-10T04:31:30.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:30.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: cluster 2021-02-10T04:31:29 2021-02-10T04:31:30.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: .477731+0000 osd.4 (osd.4) 192 : cluster [WRN] 2021-02-10T04:31:30.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:30.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: cluster 2021-02-10T04:31:29. 2021-02-10T04:31:30.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: 477739+0000 osd.4 (osd.4) 193 : cluster 2021-02-10T04:31:30.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:30.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:30.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: 31:29.477745+0000 osd.4 (osd.4) 2021-02-10T04:31:30.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: 194 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:30.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:31:30.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: 04:31:29.677712+0000 mon.b (mon.2) 2021-02-10T04:31:30.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: 32 : audit [INF] from='client.? 172.21.15.85:0/3774441808' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T04:31:30.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: audit 2021-02-10T04:31:30.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:31:30.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: T04:31:29.678057+0000 2021-02-10T04:31:30.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: mon.a (mon.0) 1007 : audit 2021-02-10T04:31:30.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T04:31:30.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:30.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: 02-10T04:31:30.035280+0000 mgr.y 2021-02-10T04:31:30.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[35961]: (mgr.14140) 443 : cluster [DBG] pgmap v558: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 161 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 2.5 MiB/s rd, 897 KiB/s wr, 6 op/s 2021-02-10T04:31:30.710 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[45443]: debug 2021-02-10T04:31:30.206+0000 7f7140d73700 -1 osd.1 272 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:31:30.730 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:31:30.741 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:31:30.837 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:31:30.867 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:31.188 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[49106]: debug 2021-02-10T04:31:31.188 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[49106]: 2021-02-10T04:31:30.969+0000 7f77a3e94700 -1 osd.2 273 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:31:31.188 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:30 smithi085 conmon[49106]: 2021-02-10T04:31:31.188 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[45443]: debug 2021-02-10T04:31:31.188 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[45443]: 2021-02-10T04:31:31.187+0000 7f7140d73700 -1 osd.1 273 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:31:31.297 INFO:tasks.rados.rados.0.smithi085.stdout:917: done (15 left) 2021-02-10T04:31:31.297 INFO:tasks.rados.rados.0.smithi085.stdout:919: done (14 left) 2021-02-10T04:31:31.297 INFO:tasks.rados.rados.0.smithi085.stdout:922: done (13 left) 2021-02-10T04:31:31.297 INFO:tasks.rados.rados.0.smithi085.stdout:923: done (12 left) 2021-02-10T04:31:31.297 INFO:tasks.rados.rados.0.smithi085.stdout:924: read oid 1 snap -1 2021-02-10T04:31:31.297 INFO:tasks.rados.rados.0.smithi085.stdout:924: expect (ObjNum 336 snap 84 seq_num 336) 2021-02-10T04:31:31.297 INFO:tasks.rados.rados.0.smithi085.stdout:925: snap_remove snap 87 2021-02-10T04:31:31.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: cluster 2021-02-10T04:31:29.929760+0000 osd.2 (osd.2) 2021-02-10T04:31:31.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: 158 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:31.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: cluster 2021-02-10T04:31:29.929767+0000 osd.2 (osd.2) 159 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:31.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: cluster 2021-02-10T04:31:29.929774+0000 osd.2 ( 2021-02-10T04:31:31.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: osd.2) 160 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: 10T04:31:29.929782+0000 osd.2 (osd. 2021-02-10T04:31:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: 2) 161 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: cluster 2021-02-10T04:31:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:31:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: 31:29.929788+0000 osd.2 2021-02-10T04:31:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: (osd.2) 162 : cluster 2021-02-10T04:31:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: T04:31:29.929805+0000 osd.2 (osd.2) 163 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: 2021-02-10T04:31:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: cluster 2021-02-10T04:31:29 2021-02-10T04:31:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: .929818+0000 osd.2 (osd.2) 164 : 2021-02-10T04:31:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: cluster 2021-02-10T04:31:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:31:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: 04:31:29.929824+0000 osd.2 ( 2021-02-10T04:31:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: osd.2) 165 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: -10T04:31:30.207404+0000 osd.1 2021-02-10T04:31:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: (osd.1) 166 : cluster [WRN] 2021-02-10T04:31:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: cluster 2021-02-10T04:31:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: 2021-02-10T04:31:30.207413 2021-02-10T04:31:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: +0000 osd.1 (osd.1) 167 : cluster 2021-02-10T04:31:31.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:31.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:31.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: -02-10T04:31:30. 2021-02-10T04:31:31.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: 207420+0000 osd.1 (osd.1) 168 : cluster 2021-02-10T04:31:31.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:31.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: cluster 2021-02-10T04:31:30. 2021-02-10T04:31:31.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: 207429+0000 osd.1 2021-02-10T04:31:31.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: (osd.1) 169 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:31.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: 2021-02-10T04:31:31.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: cluster 2021-02-10T04:31:30.207435+0000 osd.1 (osd. 2021-02-10T04:31:31.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: 1) 170 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:31.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: cluster 2021-02-10T04:31:30. 2021-02-10T04:31:31.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: 207441+0000 osd.1 (osd.1) 171 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:31.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:31.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: -10T04:31:30.281383+0000 mon.a (mon.0) 1008 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:31:31.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: audit 2021-02-10T04:31:30.294411+0000 2021-02-10T04:31:31.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: mon.a (mon.0) 1009 : audit [INF] from='client.? 172.21.15.85:0/444861482' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:31:31.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: audit 2021-02-10T04:31:30.294484+0000 mon.a 2021-02-10T04:31:31.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: (mon.0) 1010 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-02-10T04:31:31.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: cluster 2021-02-10T04:31:30.294541+0000 mon.a (mon 2021-02-10T04:31:31.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: .0) 1011 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2021-02-10T04:31:31.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: cluster 2021-02-10T04:31:31.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: 2021-02-10T04:31:30.429830+0000 osd.4 (osd.4) 195 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:31.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: 2021-02-10T04:31:31.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: cluster 2021-02-10T04:31:30. 2021-02-10T04:31:31.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: 429838+0000 osd.4 (osd.4) 196 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:31.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: cluster 2021-02-10T04:31:31.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: 2021-02-10T04:31:30.429843+0000 osd.4 (osd.4) 197 : cluster [WRN] 2021-02-10T04:31:31.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:31.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: cluster 2021-02-10T04:31 2021-02-10T04:31:31.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[29319]: :30.429848+0000 osd.4 (osd.4) 198 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:31.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: cluster 2021-02-10T04:31:29.929760+0000 osd.2 (osd.2) 158 : cluster 2021-02-10T04:31:31.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:31.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: cluster 2021-02-10T04:31:29.929767 2021-02-10T04:31:31.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 159 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:31.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:31.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: -10T04:31:29.929774+0000 osd.2 (osd.2) 160 : cluster 2021-02-10T04:31:31.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:31.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:31.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: :31:29.929782+0000 osd.2 (osd. 2021-02-10T04:31:31.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: 2) 161 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:31.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:31.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: 02-10T04:31:29.929788+0000 2021-02-10T04:31:31.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: osd.2 (osd.2) 162 : cluster [WRN] 2021-02-10T04:31:31.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:31.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: cluster 2021-02-10T04:31: 2021-02-10T04:31:31.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: 29.929805+0000 osd.2 (osd.2) 163 2021-02-10T04:31:31.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:31.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:31.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: 04:31:29.929818+0000 osd.2 (osd 2021-02-10T04:31:31.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: .2) 164 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:31.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:31.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: -10T04:31:29.929824+0000 osd.2 2021-02-10T04:31:31.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: (osd.2) 165 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:31.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: 2021-02-10T04:31:31.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: cluster 2021-02-10T04:31:31.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: 2021-02-10T04:31:30.207404+0000 osd.1 ( 2021-02-10T04:31:31.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: osd.1) 166 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:31.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:31.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: 02-10T04:31:30.207413+0000 2021-02-10T04:31:31.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: osd.1 (osd.1) 167 : cluster [WRN] 2021-02-10T04:31:31.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:31.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: cluster 2021-02-10T04:31: 2021-02-10T04:31:31.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: 30.207420+0000 osd.1 (osd.1) 168 : 2021-02-10T04:31:31.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:31.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: cluster 2021-02-10T04:31: 2021-02-10T04:31:31.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: 30.207429+0000 osd.1 (osd.1) 169 : 2021-02-10T04:31:31.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:31.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: cluster 2021-02-10T04:31:30.207435 2021-02-10T04:31:31.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 170 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:31.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: cluster 2021-02-10T04:31:30 2021-02-10T04:31:31.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: .207441+0000 osd.1 (osd.1) 171 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:31.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: cluster 2021-02-10T04:31 2021-02-10T04:31:31.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: :30.281383+0000 mon.a (mon.0) 1008 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:31:31.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: 2021-02-10T04:31:31.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: audit 2021-02-10T04:31 2021-02-10T04:31:31.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: :30.294411+0000 mon.a (mon.0) 1009 : audit [INF] from='client.? 172.21.15.85:0/444861482' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:31:31.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: audit 2021-02-10T04:31:30.294484+0000 mon.a ( 2021-02-10T04:31:31.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: mon.0) 1010 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-02-10T04:31:31.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:31.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: 31:30.294541+0000 mon.a (mon.0) 2021-02-10T04:31:31.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: 1011 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2021-02-10T04:31:31.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:31.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: 10T04:31:30.429830+0000 osd.4 (osd. 2021-02-10T04:31:31.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: 4) 195 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:31.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:31.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: -10T04:31:30.429838+0000 osd.4 2021-02-10T04:31:31.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: (osd.4) 196 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:31.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:31.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: 10T04:31:30.429843+0000 osd.4 ( 2021-02-10T04:31:31.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: osd.4) 197 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:31.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:31.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: 04:31:30.429848+0000 osd.4 (osd. 2021-02-10T04:31:31.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[35961]: 4) 198 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:31.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[33677]: debug 2021-02-10T04:31:31.418+0000 7fbec19a1700 -1 osd.4 273 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:31:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: cluster 2021-02-10T04:31:29.929760+0000 osd.2 (osd.2) 158 : cluster [WRN] 2021-02-10T04:31:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: cluster 2021-02-10T04:31:29.929767+0000 osd.2 (osd.2) 159 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: cluster 2021-02-10T04:31:29.929774+0000 osd.2 (osd. 2021-02-10T04:31:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: 2) 160 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: cluster 2021-02-10T04:31:29.929782+0000 osd.2 (osd.2) 161 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: -02-10T04:31:29.929788+0000 osd.2 (osd.2) 162 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: -10T04:31:29.929805+0000 osd.2 2021-02-10T04:31:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: (osd.2) 163 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:31.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: 02-10T04:31:29.929818+0000 2021-02-10T04:31:31.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: osd.2 (osd.2) 164 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:31.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: 2021-02-10T04:31:31.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: cluster 2021-02-10T04:31:31.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: 2021-02-10T04:31:29.929824+0000 osd.2 (osd 2021-02-10T04:31:31.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: .2) 165 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:31.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:31.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: 10T04:31:30.207404+0000 osd.1 ( 2021-02-10T04:31:31.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: osd.1) 166 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:31.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:31.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: -10T04:31:30.207413+0000 osd.1 ( 2021-02-10T04:31:31.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: osd.1) 167 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:31.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:31.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: -10T04:31:30.207420+0000 osd.1 2021-02-10T04:31:31.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: (osd.1) 168 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:31.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:31.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: -10T04:31:30.207429+0000 osd.1 ( 2021-02-10T04:31:31.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: osd.1) 169 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:31.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:31:31.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: 31:30.207435+0000 osd.1 (osd.1) 2021-02-10T04:31:31.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: 170 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:31.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:31.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: T04:31:30.207441+0000 osd.1 (osd.1) 2021-02-10T04:31:31.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: 171 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:31.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:31.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: T04:31:30.281383+0000 mon.a (mon 2021-02-10T04:31:31.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: .0) 1008 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:31:31.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:31:31.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: 02-10T04:31:30.294411+0000 2021-02-10T04:31:31.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: mon.a (mon.0) 1009 : audit [INF] from='client.? 172.21.15.85:0/444861482' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:31:31.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: 2021-02-10T04:31:31.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: audit 2021-02-10T04:31:31.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: 2021-02-10T04:31:30.294484+0000 mon.a 2021-02-10T04:31:31.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: (mon.0) 1010 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-02-10T04:31:31.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:31.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: 02-10T04:31:30.294541+0000 2021-02-10T04:31:31.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: mon.a (mon.0) 1011 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2021-02-10T04:31:31.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: 2021-02-10T04:31:31.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: cluster 2021-02-10T04:31:31.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: 2021-02-10T04:31:30.429830+0000 osd.4 2021-02-10T04:31:31.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: (osd.4) 195 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:31.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:31.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: -10T04:31:30.429838+0000 osd.4 2021-02-10T04:31:31.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: (osd.4) 196 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:31.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:31.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: -10T04:31:30.429843+0000 2021-02-10T04:31:31.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: osd.4 (osd.4) 197 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:31.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: 2021-02-10T04:31:31.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: cluster 2021-02-10T04:31:31.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: 2021-02-10T04:31:30.429848+0000 osd.4 ( 2021-02-10T04:31:31.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:31 smithi164 conmon[29288]: osd.4) 198 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:32.298 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:31 smithi085 conmon[49106]: debug 2021-02-10T04:31:31.988+0000 7f77a3e94700 -1 osd.2 274 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:31:32.299 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[45443]: debug 2021-02-10T04:31:32.299 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[45443]: 2021-02-10T04:31:32.222+0000 7f7140d73700 -1 osd.1 274 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:31:32.299 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[45443]: 2021-02-10T04:31:32.304 INFO:tasks.rados.rados.0.smithi085.stdout:925: done (13 left) 2021-02-10T04:31:32.304 INFO:tasks.rados.rados.0.smithi085.stdout:926: copy_from oid 2 from oid 43 current snap is 88 2021-02-10T04:31:32.304 INFO:tasks.rados.rados.0.smithi085.stdout:924: expect (ObjNum 336 snap 84 seq_num 336) 2021-02-10T04:31:32.306 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: cluster 2021-02-10T04:31:30.970446+0000 2021-02-10T04:31:32.306 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: osd.2 (osd.2) 166 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:32.306 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:32.306 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: 10T04:31:30.970454+0000 osd.2 (osd.2) 167 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:32.306 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: 2021-02-10T04:31:32.307 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: cluster 2021-02-10T04:31:32.307 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: 2021-02-10T04:31:30.970458+0000 osd.2 (osd 2021-02-10T04:31:32.307 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: .2) 168 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:32.307 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:32.307 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: 02-10T04:31:30.970462+0000 2021-02-10T04:31:32.307 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: osd.2 (osd.2) 169 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:32.307 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: 2021-02-10T04:31:32.307 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: cluster 2021-02-10T04:31:32.307 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: 2021-02-10T04:31:30.970465+0000 osd.2 (osd.2) 170 : cluster [WRN] 2021-02-10T04:31:32.307 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:32.307 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: cluster 2021-02-10T04:31:32.307 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: 2021-02-10T04:31:30.970468+0000 osd.2 (osd.2) 171 2021-02-10T04:31:32.308 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:32.308 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:32.308 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: 31:30.970471+0000 osd.2 (osd.2) 2021-02-10T04:31:32.308 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: 172 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:32.308 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:32.308 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: 04:31:30.970474+0000 osd.2 (osd.2) 173 : cluster [WRN] 2021-02-10T04:31:32.308 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:32.308 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:32.308 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: -02-10T04:31:31.188470+0000 osd.1 (osd.1) 172 2021-02-10T04:31:32.308 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:32.308 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: cluster 2021-02-10T04:31:31 2021-02-10T04:31:32.308 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: .188479+0000 osd.1 (osd.1) 173 : cluster [WRN] 2021-02-10T04:31:32.308 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:32.309 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: cluster 2021-02-10T04:31:31 2021-02-10T04:31:32.309 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: .188486+0000 osd.1 (osd.1) 174 : cluster [WRN] 2021-02-10T04:31:32.309 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:32.309 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: cluster 2021-02-10T04:31:32.309 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: 2021-02-10T04:31:31.188489+0000 osd.1 (osd. 2021-02-10T04:31:32.309 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: 1) 175 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:32.309 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:32.309 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: 04:31:31.188494+0000 osd.1 (osd.1) 2021-02-10T04:31:32.309 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: 176 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:32.309 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: cluster 2021-02-10T04:31 2021-02-10T04:31:32.310 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: :31.188500+0000 osd.1 (osd.1) 177 2021-02-10T04:31:32.310 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:32.310 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: cluster 2021-02-10T04:31 2021-02-10T04:31:32.310 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: :31.296459+0000 mon.a (mon.0) 1012 2021-02-10T04:31:32.310 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2021-02-10T04:31:32.310 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:32.310 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: :31:31.399574+0000 osd.0 (osd.0) 58 : 2021-02-10T04:31:32.310 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: cluster [DBG] purged_snaps scrub starts 2021-02-10T04:31:32.311 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: cluster 2021-02-10T04:31:30.970446+0000 osd.2 ( 2021-02-10T04:31:32.311 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: osd.2) 166 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:32.311 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: cluster 2021-02-10T04:31:30.970454 2021-02-10T04:31:32.311 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 167 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:32.311 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:32.311 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: 02-10T04:31:30.970458+0000 osd.2 (osd.2 2021-02-10T04:31:32.311 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: ) 168 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:32.311 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:32.311 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: T04:31:30.970462+0000 osd.2 (osd.2) 169 2021-02-10T04:31:32.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:32.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:31:32.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: 31:30.970465 2021-02-10T04:31:32.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 170 : cluster [WRN] 2021-02-10T04:31:32.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:32.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: cluster 2021-02-10T04:31:32.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: 2021-02-10T04:31:30.970468+0000 osd.2 (osd. 2021-02-10T04:31:32.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: 2) 171 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:32.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:32.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: 02-10T04:31:30. 2021-02-10T04:31:32.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: 970471+0000 osd.2 (osd.2) 172 : cluster [WRN] 2021-02-10T04:31:32.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:32.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: cluster 2021-02-10T04:31:30. 2021-02-10T04:31:32.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: 970474+0000 osd.2 (osd.2) 173 : cluster [WRN] 2021-02-10T04:31:32.313 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:32.313 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: cluster 2021-02-10T04:31:32.313 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: 2021-02-10T04:31:31.188470+0000 osd.1 (osd.1 2021-02-10T04:31:32.313 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: ) 172 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:32.313 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:31:32.313 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: :31:31.188479+0000 2021-02-10T04:31:32.313 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: osd.1 (osd.1) 173 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:32.313 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: 2021-02-10T04:31:32.313 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:31:32.313 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: 31:31.188486+0000 osd.1 2021-02-10T04:31:32.313 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: (osd.1) 174 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:32.313 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: cluster 2021-02-10T04:31:32.313 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: 2021-02-10T04:31:31.188489+0000 osd.1 (osd.1) 2021-02-10T04:31:32.314 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: 175 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:32.314 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:32.314 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: 10T04:31:31.188494+0000 osd.1 (osd 2021-02-10T04:31:32.314 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: .1) 176 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:32.314 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:32.314 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: -10T04:31:31.188500+0000 2021-02-10T04:31:32.314 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: osd.1 (osd.1) 177 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:32.314 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: 2021-02-10T04:31:32.314 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: cluster 2021-02-10T04:31:32.314 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:31:32.314 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: 31:31.296459+0000 mon.a (mon.0) 2021-02-10T04:31:32.314 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: 1012 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2021-02-10T04:31:32.314 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:32.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: -10T04:31:31.399574+0000 osd.0 (osd. 2021-02-10T04:31:32.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: 0) 58 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:31:32.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:32.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: T04:31:31.399843+0000 osd.0 (osd.0 2021-02-10T04:31:32.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: ) 59 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:31:32.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:31:32.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: 31:31.419304+0000 osd.4 (osd.4) 2021-02-10T04:31:32.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: 199 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:32.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:31:32.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: 31:31.419309+0000 osd.4 (osd.4) 2021-02-10T04:31:32.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: 200 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:32.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: cluster 2021-02-10T04:31: 2021-02-10T04:31:32.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: 31.419313+0000 osd.4 (osd.4) 201 : cluster 2021-02-10T04:31:32.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:32.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: cluster 2021-02-10T04:31:32.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: 2021-02-10T04:31:31.419318+0000 osd.4 (osd 2021-02-10T04:31:32.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: .4) 202 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:32.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: cluster 2021-02-10T04:31:32.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: 2021-02-10T04:31:32 2021-02-10T04:31:32.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: .035809+0000 mgr.y (mgr.14140) 2021-02-10T04:31:32.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: 444 : cluster 2021-02-10T04:31:32.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[29319]: [DBG] pgmap v561: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 157 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 5.7 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2021-02-10T04:31:32.467 INFO:tasks.rados.rados.0.smithi085.stdout:924: done (13 left) 2021-02-10T04:31:32.467 INFO:tasks.rados.rados.0.smithi085.stdout:927: write oid 26 current snap is 88 2021-02-10T04:31:32.468 INFO:tasks.rados.rados.0.smithi085.stdout:926: finishing copy_from to smithi08575397-2 2021-02-10T04:31:32.468 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 79 (ObjNum 321 snap 81 seq_num 321) dirty exists 2021-02-10T04:31:32.468 INFO:tasks.rados.rados.0.smithi085.stdout:926: finishing copy_from racing read to smithi08575397-2 2021-02-10T04:31:32.468 INFO:tasks.rados.rados.0.smithi085.stdout:927: seq_num 343 ranges {564801=646292,1921559=742115,3202910=561891} 2021-02-10T04:31:32.475 INFO:tasks.rados.rados.0.smithi085.stdout:927: writing smithi08575397-26 from 564801 to 1211093 tid 1 2021-02-10T04:31:32.485 INFO:tasks.rados.rados.0.smithi085.stdout:927: writing smithi08575397-26 from 1921559 to 2663674 tid 2 2021-02-10T04:31:32.492 INFO:tasks.rados.rados.0.smithi085.stdout:927: writing smithi08575397-26 from 3202910 to 3764801 tid 3 2021-02-10T04:31:32.492 INFO:tasks.rados.rados.0.smithi085.stdout:926: done (13 left) 2021-02-10T04:31:32.492 INFO:tasks.rados.rados.0.smithi085.stdout:928: delete oid 21 current snap is 88 2021-02-10T04:31:32.492 INFO:tasks.rados.rados.0.smithi085.stdout:927: finishing write tid 1 to smithi08575397-26 2021-02-10T04:31:32.492 INFO:tasks.rados.rados.0.smithi085.stdout:927: finishing write tid 2 to smithi08575397-26 2021-02-10T04:31:32.495 INFO:tasks.rados.rados.0.smithi085.stdout:927: finishing write tid 3 to smithi08575397-26 2021-02-10T04:31:32.496 INFO:tasks.rados.rados.0.smithi085.stdout:928: done (13 left) 2021-02-10T04:31:32.496 INFO:tasks.rados.rados.0.smithi085.stdout:929: read oid 30 snap -1 2021-02-10T04:31:32.496 INFO:tasks.rados.rados.0.smithi085.stdout:929: expect (ObjNum 342 snap 87 seq_num 342) 2021-02-10T04:31:32.496 INFO:tasks.rados.rados.0.smithi085.stdout:930: write (excl) oid 43 current snap is 88 2021-02-10T04:31:32.496 INFO:tasks.rados.rados.0.smithi085.stdout:930: seq_num 345 ranges {598415=796790,1922170=745784,3141403=415315} 2021-02-10T04:31:32.505 INFO:tasks.rados.rados.0.smithi085.stdout:930: writing smithi08575397-43 from 598415 to 1395205 tid 1 2021-02-10T04:31:32.513 INFO:tasks.rados.rados.0.smithi085.stdout:930: writing smithi08575397-43 from 1922170 to 2667954 tid 2 2021-02-10T04:31:32.519 INFO:tasks.rados.rados.0.smithi085.stdout:930: writing smithi08575397-43 from 3141403 to 3556718 tid 3 2021-02-10T04:31:32.520 INFO:tasks.rados.rados.0.smithi085.stdout:931: write (excl) oid 23 current snap is 88 2021-02-10T04:31:32.520 INFO:tasks.rados.rados.0.smithi085.stdout:931: seq_num 346 ranges {615222=472039,1580652=489442,2768826=626587} 2021-02-10T04:31:32.526 INFO:tasks.rados.rados.0.smithi085.stdout:931: writing smithi08575397-23 from 615222 to 1087261 tid 1 2021-02-10T04:31:32.533 INFO:tasks.rados.rados.0.smithi085.stdout:931: writing smithi08575397-23 from 1580652 to 2070094 tid 2 2021-02-10T04:31:32.542 INFO:tasks.rados.rados.0.smithi085.stdout:931: writing smithi08575397-23 from 2768826 to 3395413 tid 3 2021-02-10T04:31:32.542 INFO:tasks.rados.rados.0.smithi085.stdout:927: finishing write tid 4 to smithi08575397-26 2021-02-10T04:31:32.542 INFO:tasks.rados.rados.0.smithi085.stdout:927: finishing write tid 5 to smithi08575397-26 2021-02-10T04:31:32.542 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 85 (ObjNum 343 snap 88 seq_num 343) dirty exists 2021-02-10T04:31:32.542 INFO:tasks.rados.rados.0.smithi085.stdout:927: left oid 26 (ObjNum 343 snap 88 seq_num 343) 2021-02-10T04:31:32.542 INFO:tasks.rados.rados.0.smithi085.stdout:929: expect (ObjNum 342 snap 87 seq_num 342) 2021-02-10T04:31:32.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[33677]: debug 2021-02-10T04:31:32.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[33677]: 2021-02-10T04:31:32.430+0000 7fbec19a1700 -1 osd.4 275 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:31:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: cluster 2021-02-10T04:31:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:31:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: 31:30.970446+0000 osd.2 (osd.2) 166 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: cluster 2021-02-10T04:31:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: 2021-02-10T04:31:30.970454+0000 2021-02-10T04:31:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: osd.2 (osd.2) 167 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: -10T04:31:30.970458+0000 osd.2 ( 2021-02-10T04:31:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: osd.2) 168 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: 2021-02-10T04:31:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: cluster 2021-02-10T04:31:30. 2021-02-10T04:31:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: 970462+0000 osd.2 (osd.2) 169 : cluster [WRN] 2021-02-10T04:31:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:31:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: :31:30.970465+0000 osd.2 (osd.2 2021-02-10T04:31:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: ) 170 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: T04:31:30.970468+0000 osd.2 ( 2021-02-10T04:31:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: osd.2) 171 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: -02-10T04:31:30.970471+0000 2021-02-10T04:31:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: osd.2 (osd.2) 172 : cluster [WRN] 2021-02-10T04:31:32.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:32.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: cluster 2021-02-10T04:31:32.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: 2021-02-10T04:31:30.970474+0000 2021-02-10T04:31:32.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: osd.2 (osd.2) 173 : cluster [WRN] 2021-02-10T04:31:32.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:32.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: cluster 2021-02-10T04:31:32.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: 2021-02-10T04:31:31.188470+0000 osd.1 2021-02-10T04:31:32.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: (osd.1) 172 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:32.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:32.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: -02-10T04:31:31.188479+0000 2021-02-10T04:31:32.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: osd.1 (osd.1) 173 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:32.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: 2021-02-10T04:31:32.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:32.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: 02-10T04:31:31.188486+0000 osd.1 (osd. 2021-02-10T04:31:32.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: 1) 174 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:32.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:31:32.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: :31:31.188489+0000 osd.1 (osd.1) 2021-02-10T04:31:32.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: 175 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:32.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:31:32.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: :31:31.188494+0000 osd.1 (osd. 2021-02-10T04:31:32.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: 1) 176 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:32.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:31:32.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: :31:31.188500+0000 osd.1 (osd.1 2021-02-10T04:31:32.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: ) 177 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:32.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: cluster 2021-02-10T04:31:31.296459 2021-02-10T04:31:32.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1012 : cluster [DBG] 2021-02-10T04:31:32.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: osdmap e274: 8 total, 8 up, 8 in 2021-02-10T04:31:32.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: cluster 2021-02-10T04:31:32.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: 2021-02-10T04:31:31.399574+0000 osd.0 2021-02-10T04:31:32.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: (osd.0) 58 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:31:32.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:32.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: 02-10T04:31:31.399843+0000 2021-02-10T04:31:32.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: osd.0 (osd.0) 59 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:31:32.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: 2021-02-10T04:31:32.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:32.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: -02-10T04:31:31.419304+0000 osd.4 (osd.4 2021-02-10T04:31:32.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: ) 199 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:32.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:31:32.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: 04:31:31.419309+0000 osd.4 (osd. 2021-02-10T04:31:32.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: 4) 200 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:32.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:31:32.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: 04:31:31.419313+0000 osd.4 (osd. 2021-02-10T04:31:32.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: 4) 201 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:32.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:32.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: T04:31:31.419318+0000 osd.4 (osd 2021-02-10T04:31:32.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: .4) 202 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:32.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:32.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: T04:31:32.035809+0000 mgr.y (mgr 2021-02-10T04:31:32.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:32 smithi164 conmon[29288]: .14140) 444 : cluster [DBG] pgmap v561: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 157 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 5.7 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2021-02-10T04:31:32.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: 31:31.399843+0000 osd.0 (osd.0) 59 : 2021-02-10T04:31:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:31:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: 31:31.419304+0000 osd.4 (osd.4 2021-02-10T04:31:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: ) 199 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: :31:31.419309+0000 osd.4 (osd.4) 2021-02-10T04:31:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: 200 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: 04:31:31.419313+0000 osd.4 (osd 2021-02-10T04:31:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: .4) 201 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: 02-10T04:31:31.419318+0000 osd.4 ( 2021-02-10T04:31:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: osd.4) 202 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: -02-10T04:31:32.035809+0000 2021-02-10T04:31:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[35961]: mgr.y (mgr.14140) 444 : cluster [DBG] pgmap v561: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 157 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 5.7 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2021-02-10T04:31:32.747 INFO:tasks.rados.rados.0.smithi085.stdout:930: finishing write tid 1 to smithi08575397-43 2021-02-10T04:31:32.747 INFO:tasks.rados.rados.0.smithi085.stdout:930: finishing write tid 2 to smithi08575397-43 2021-02-10T04:31:32.747 INFO:tasks.rados.rados.0.smithi085.stdout:927: done (15 left) 2021-02-10T04:31:32.747 INFO:tasks.rados.rados.0.smithi085.stdout:929: done (14 left) 2021-02-10T04:31:32.747 INFO:tasks.rados.rados.0.smithi085.stdout:932: copy_from oid 1 from oid 5 current snap is 88 2021-02-10T04:31:32.747 INFO:tasks.rados.rados.0.smithi085.stdout:930: finishing write tid 3 to smithi08575397-43 2021-02-10T04:31:32.748 INFO:tasks.rados.rados.0.smithi085.stdout:930: finishing write tid 4 to smithi08575397-43 2021-02-10T04:31:32.748 INFO:tasks.rados.rados.0.smithi085.stdout:930: finishing write tid 5 to smithi08575397-43 2021-02-10T04:31:32.748 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 446 (ObjNum 345 snap 88 seq_num 345) dirty exists 2021-02-10T04:31:32.748 INFO:tasks.rados.rados.0.smithi085.stdout:930: left oid 43 (ObjNum 345 snap 88 seq_num 345) 2021-02-10T04:31:32.748 INFO:tasks.rados.rados.0.smithi085.stdout:931: finishing write tid 1 to smithi08575397-23 2021-02-10T04:31:32.748 INFO:tasks.rados.rados.0.smithi085.stdout:931: finishing write tid 2 to smithi08575397-23 2021-02-10T04:31:32.748 INFO:tasks.rados.rados.0.smithi085.stdout:931: finishing write tid 3 to smithi08575397-23 2021-02-10T04:31:32.748 INFO:tasks.rados.rados.0.smithi085.stdout:931: finishing write tid 4 to smithi08575397-23 2021-02-10T04:31:32.748 INFO:tasks.rados.rados.0.smithi085.stdout:931: finishing write tid 5 to smithi08575397-23 2021-02-10T04:31:32.748 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 150 (ObjNum 346 snap 88 seq_num 346) dirty exists 2021-02-10T04:31:32.748 INFO:tasks.rados.rados.0.smithi085.stdout:931: left oid 23 (ObjNum 346 snap 88 seq_num 346) 2021-02-10T04:31:32.748 INFO:tasks.rados.rados.0.smithi085.stdout:930: done (14 left) 2021-02-10T04:31:32.748 INFO:tasks.rados.rados.0.smithi085.stdout:931: done (13 left) 2021-02-10T04:31:32.749 INFO:tasks.rados.rados.0.smithi085.stdout:933: read oid 6 snap -1 2021-02-10T04:31:32.749 INFO:tasks.rados.rados.0.smithi085.stdout:933: expect (ObjNum 334 snap 84 seq_num 334) 2021-02-10T04:31:32.749 INFO:tasks.rados.rados.0.smithi085.stdout:934: snap_create 2021-02-10T04:31:32.751 INFO:tasks.rados.rados.0.smithi085.stdout:933: expect (ObjNum 334 snap 84 seq_num 334) 2021-02-10T04:31:32.956 INFO:tasks.rados.rados.0.smithi085.stdout:932: finishing copy_from to smithi08575397-1 2021-02-10T04:31:32.956 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 236 (ObjNum 308 snap 74 seq_num 308) dirty exists 2021-02-10T04:31:32.956 INFO:tasks.rados.rados.0.smithi085.stdout:932: finishing copy_from racing read to smithi08575397-1 2021-02-10T04:31:33.002 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:31:32 smithi085 conmon[52836]: debug 2021-02-10T04:31:32.703+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:33.217 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:31:38.437 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[40892]: debug 2021-02-10T04:31:33.208+0000 7f30354c5700 -1 osd.6 pg_epoch: 275 pg[7.7( v 275'469 (0'0,275'469] local-lis/les=93/94 n=12 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [6,0,3] r=0 lpr=93 crt=275'469 lcod 273'468 mlcod 273'468 active+clean+snaptrim trimq=[9b~2,9e~1,a0~1,a2~1,a4~1,a9~3,b0~1] ps=[60~3,66~3,6b~3,6f~1,74~2,78~4,7e~1,80~4,87~2,8b~3,8f~1,91~2,97~1,99~1]] removing snap head 2021-02-10T04:31:38.437 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[33677]: debug 2021-02-10T04:31:33.381+0000 7fbec19a1700 -1 osd.4 276 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:31:38.437 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[33677]: debug 2021-02-10T04:31:38.438 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[33677]: 2021-02-10T04:31:34.385+0000 7fbec19a1700 -1 osd.4 276 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:31:38.438 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[33677]: debug 2021-02-10T04:31:35.362+0000 7fbec19a1700 -1 osd.4 278 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:31:38.438 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[33677]: debug 2021-02-10T04:31:38.438 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[33677]: 2021-02-10T04:31:36.351+0000 7fbec19a1700 -1 osd.4 278 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:31:38.438 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[33677]: 2021-02-10T04:31:38.438 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[33677]: debug 2021-02-10T04:31:37.366+0000 7fbec19a1700 -1 osd.4 280 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:31:38.438 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: cluster 2021-02-10T04:31:31.988626+0000 2021-02-10T04:31:38.438 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: osd.2 (osd.2) 174 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:38.438 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: cluster 2021-02-10T04:31:31.988636+0000 osd.2 ( 2021-02-10T04:31:38.438 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: osd.2) 175 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:38.439 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38.439 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: 2021-02-10T04:31:31.988650+0000 osd.2 (osd.2 2021-02-10T04:31:38.439 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: ) 176 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:38.439 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:38.439 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: 10T04:31:31.988656+0000 osd.2 (osd. 2021-02-10T04:31:38.439 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: 2) 177 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:38.439 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:38.439 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: 02-10T04:31:31.988659+0000 osd.2 ( 2021-02-10T04:31:38.439 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: osd.2) 178 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:38.439 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:38.439 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: 02-10T04:31:31.988664+0000 osd.2 ( 2021-02-10T04:31:38.439 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: osd.2) 179 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:38.439 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: 2021-02-10T04:31:38.440 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:38.440 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: -02-10T04:31:31.988679+0000 2021-02-10T04:31:38.440 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: osd.2 (osd.2) 180 2021-02-10T04:31:38.440 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:38.440 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38.440 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: 2021-02-10T04:31:31.988683 2021-02-10T04:31:38.440 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 181 2021-02-10T04:31:38.440 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:38.441 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:38.441 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: 02-10T04:31:32.223293+0000 2021-02-10T04:31:38.441 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: osd.1 (osd.1) 178 : cluster 2021-02-10T04:31:38.441 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:38.441 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:31:38.441 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: :31:32.223303+0000 osd.1 (osd.1) 179 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:38.441 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:38.441 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: -02-10T04:31:32.223312+0000 osd.1 2021-02-10T04:31:38.441 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: (osd.1) 180 : cluster [WRN] 2021-02-10T04:31:38.441 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:38.441 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38.441 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: 2021-02-10T04:31:32 2021-02-10T04:31:38.441 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: .223318+0000 osd.1 (osd.1) 181 : cluster 2021-02-10T04:31:38.442 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:38.442 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: cluster 2021-02-10T04:31:32. 2021-02-10T04:31:38.442 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: 223323+0000 osd.1 (osd.1) 182 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:38.442 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: 2021-02-10T04:31:38.442 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: cluster 2021-02-10T04:31:32 2021-02-10T04:31:38.442 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: .223331+0000 osd.1 (osd.1) 183 : cluster [WRN] 2021-02-10T04:31:38.442 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:38.442 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:31:38.442 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: 31:32.303740+0000 mon.a (mon.0) 1013 2021-02-10T04:31:38.442 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2021-02-10T04:31:38.443 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: cluster 2021-02-10T04:31:32.431618 2021-02-10T04:31:38.443 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 203 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:38.443 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: 2021-02-10T04:31:38.443 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:38.443 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: 02-10T04:31:32.431624 2021-02-10T04:31:38.443 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 204 : 2021-02-10T04:31:38.443 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:38.443 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:38.443 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: 10T04:31:32.431639+0000 2021-02-10T04:31:38.443 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: osd.4 (osd.4) 205 : cluster 2021-02-10T04:31:38.443 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:38.443 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:38.443 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: 02-10T04:31:32.431647+0000 osd.4 ( 2021-02-10T04:31:38.443 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: osd.4) 206 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:38.444 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:38.444 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: -02-10T04:31:33.074918+0000 2021-02-10T04:31:38.444 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: mon.a (mon.0) 1014 : cluster [WRN] Health check update: 18 slow ops, oldest one blocked for 137 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:31:38.444 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:33 smithi164 conmon[29288]: 2021-02-10T04:31:38.444 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: cluster 2021-02-10T04:31:33.002211+0000 osd.2 ( 2021-02-10T04:31:38.444 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: osd.2) 182 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:38.444 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: cluster 2021-02-10T04:31:33.002219+0000 osd.2 (osd.2) 183 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:38.444 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:38.444 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: -02-10T04:31:33.002225+0000 osd.2 (osd.2) 184 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:38.444 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38.444 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: 2021-02-10T04:31:33.002231+0000 osd.2 (osd.2) 185 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:38.444 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: cluster 2021-02-10T04:31:33.002237+0000 osd.2 (osd 2021-02-10T04:31:38.445 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: .2) 186 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:38.445 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:38.445 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: -02-10T04:31:33.002244+0000 osd.2 (osd.2) 187 2021-02-10T04:31:38.445 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:38.445 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:38.445 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: 02-10T04:31:33.002249+0000 osd.2 (osd.2) 188 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:38.445 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: cluster 2021-02-10T04:31:33. 2021-02-10T04:31:38.445 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: 002266+0000 osd.2 (osd.2) 189 : cluster [WRN] 2021-02-10T04:31:38.445 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:38.445 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38.445 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: 2021-02-10T04:31:33.221219+0000 osd.1 ( 2021-02-10T04:31:38.445 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: osd.1) 184 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:38.445 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:31:38.446 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: 04:31:33.221225+0000 osd.1 (osd. 2021-02-10T04:31:38.446 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: 1) 185 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:38.446 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:38.446 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: T04:31:33.221229+0000 2021-02-10T04:31:38.446 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: osd.1 (osd.1) 186 : cluster [WRN] 2021-02-10T04:31:38.446 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:38.446 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:31:38.446 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: :31:33.221232+0000 osd.1 (osd.1 2021-02-10T04:31:38.446 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: ) 187 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:38.446 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:38.446 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: T04:31:33.221235+0000 osd.1 (osd 2021-02-10T04:31:38.446 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: .1) 188 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:38.447 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:38.447 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: -02-10T04:31:33.221240+0000 2021-02-10T04:31:38.447 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: osd.1 (osd.1) 189 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:38.447 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: 2021-02-10T04:31:38.447 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38.447 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: 2021-02-10T04:31:33.319260+0000 2021-02-10T04:31:38.447 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: mon.a (mon.0) 1015 : cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2021-02-10T04:31:38.447 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: 2021-02-10T04:31:38.447 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38.448 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: 2021-02-10T04:31:33.382481+0000 2021-02-10T04:31:38.448 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: osd.4 (osd.4) 207 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:38.448 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: 2021-02-10T04:31:38.448 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38.448 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: 2021-02-10T04:31:33.382487+0000 2021-02-10T04:31:38.448 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: osd.4 (osd.4) 208 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:38.448 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: 2021-02-10T04:31:38.448 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38.448 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: 2021-02-10T04:31:33.382491+0000 2021-02-10T04:31:38.448 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: osd.4 (osd.4) 209 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:38.448 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: 2021-02-10T04:31:38.448 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38.448 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: 2021-02-10T04:31:33.382494+0000 2021-02-10T04:31:38.449 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: osd.4 (osd.4) 210 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:38.449 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: 2021-02-10T04:31:38.449 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38.449 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: 2021-02-10T04:31:34.036251+0000 2021-02-10T04:31:38.449 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: mgr.y (mgr.14140) 445 : cluster [DBG] pgmap v564: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 154 MiB data, 521 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 780 KiB/s wr, 6 op/s 2021-02-10T04:31:38.449 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:34 smithi164 conmon[29288]: 2021-02-10T04:31:38.449 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: cluster 2021-02-10T04:31:33.954238+0000 2021-02-10T04:31:38.449 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: osd.2 (osd.2) 190 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:38.449 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: cluster 2021-02-10T04:31:33.954245+0000 osd.2 (osd.2) 191 2021-02-10T04:31:38.449 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:38.449 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: cluster 2021-02-10T04:31:33.954249+0000 osd.2 (osd.2) 192 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:38.449 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: cluster 2021-02-10T04:31:33.954252+0000 osd.2 (osd.2 2021-02-10T04:31:38.449 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: ) 193 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:38.450 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38.450 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: 2021-02-10T04:31 2021-02-10T04:31:38.450 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: :33.954258+0000 osd.2 (osd.2) 194 2021-02-10T04:31:38.450 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:38.450 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:38.450 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: 10T04:31:33.954261+0000 osd.2 (osd 2021-02-10T04:31:38.450 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: .2) 195 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:38.450 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:38.450 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: 02-10T04:31:33.954264+0000 osd.2 ( 2021-02-10T04:31:38.450 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: osd.2) 196 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:38.450 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:38.450 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: -02-10T04:31:33.954267+0000 2021-02-10T04:31:38.451 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: osd.2 (osd.2) 197 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:38.451 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: 2021-02-10T04:31:38.451 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:31:38.451 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: 31:34.251037+0000 osd.1 (osd.1) 2021-02-10T04:31:38.451 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: 190 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:38.451 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38.451 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: 2021-02- 2021-02-10T04:31:38.451 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: 10T04:31:34.251046+0000 osd.1 ( 2021-02-10T04:31:38.451 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: osd.1) 191 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:38.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: 2021-02-10T04:31: 2021-02-10T04:31:38.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: 34.251058+0000 osd.1 (osd.1) 192 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:38.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: 2021-02 2021-02-10T04:31:38.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: -10T04:31:34.251068 2021-02-10T04:31:38.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 2021-02-10T04:31:38.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: 193 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:38.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:38.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: 02-10T04:31:34.251076 2021-02-10T04:31:38.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 194 2021-02-10T04:31:38.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:38.452 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:38.453 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: 02-10T04:31:34.251082+0000 2021-02-10T04:31:38.453 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: osd.1 (osd.1) 195 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:38.453 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:38.453 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: -02-10T04:31:34.324099+0000 mon.a (mon 2021-02-10T04:31:38.453 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: .0) 1016 : cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2021-02-10T04:31:38.453 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:38.453 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: -02-10T04:31:34.386301+0000 osd.4 ( 2021-02-10T04:31:38.453 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: osd.4) 211 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:38.453 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:38.453 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: -02-10T04:31:34.386309+0000 2021-02-10T04:31:38.453 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: osd.4 (osd.4) 212 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:38.453 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:38.453 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: T04:31:34.386315+0000 osd.4 (osd.4) 213 : cluster [WRN] 2021-02-10T04:31:38.453 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:38.454 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:38.454 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: 02-10T04:31:34.386321+0000 osd.4 ( 2021-02-10T04:31:38.454 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[29288]: osd.4) 214 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:38.454 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: cluster 2021-02-10T04:31:34.970725+0000 osd.2 ( 2021-02-10T04:31:38.454 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: osd.2) 198 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:38.454 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: cluster 2021-02-10T04:31:34.970734+0000 osd.2 (osd.2) 199 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:38.454 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: cluster 2021-02-10T04:31:34.970740+0000 osd.2 (osd.2) 200 : cluster 2021-02-10T04:31:38.454 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:38.454 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:38.454 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: -10T04:31:34.970746+0000 osd.2 (osd.2) 201 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:38.454 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: cluster 2021-02-10T04:31:34.970753 2021-02-10T04:31:38.455 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 202 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:38.455 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38.455 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: 2021-02-10T04:31:34.970758+0000 osd.2 (osd.2) 203 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:38.455 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:38.455 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: T04:31:34.970770+0000 osd.2 (osd. 2021-02-10T04:31:38.455 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: 2) 204 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:38.455 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:31:38.455 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: :31:34.970793+0000 osd.2 (osd.2) 205 : cluster 2021-02-10T04:31:38.455 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:38.456 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:38.456 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: 10T04:31:35.204506+0000 osd.1 (osd.1) 196 : cluster [WRN] 2021-02-10T04:31:38.456 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:38.456 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38.456 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: 2021-02-10T04:31:35.204512+0000 osd.1 (osd.1) 2021-02-10T04:31:38.456 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: 197 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:38.456 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: cluster 2021-02-10T04:31:35.204517 2021-02-10T04:31:38.456 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 198 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:38.456 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:31:38.457 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: 04:31:35.204520+0000 osd.1 (osd. 2021-02-10T04:31:38.457 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: 1) 199 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:38.457 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:38.457 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: T04:31:35.204524+0000 2021-02-10T04:31:38.457 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: osd.1 (osd.1) 200 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:38.457 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:38.457 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: 10T04:31:35.204528+0000 osd.1 (osd. 2021-02-10T04:31:38.457 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: 1) 201 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:38.457 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38.457 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: 2021-02-10T04:31:35.324488+0000 mon.a (mon.0) 1017 : 2021-02-10T04:31:38.457 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2021-02-10T04:31:38.457 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:38.457 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: -02-10T04:31:35.363160+0000 osd.4 (osd.4) 2021-02-10T04:31:38.457 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: 215 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:38.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: 2021-02-10T04:31:38.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:38.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: T04:31:35.363171 2021-02-10T04:31:38.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 216 2021-02-10T04:31:38.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:38.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:38.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: 02-10T04:31:35.363181+0000 2021-02-10T04:31:38.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: osd.4 (osd.4) 217 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:38.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:38.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: 10T04:31:35.363186+0000 osd.4 (osd 2021-02-10T04:31:38.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: .4) 218 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:38.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:31:38.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: :31:36.020100+0000 osd.2 (osd.2) 2021-02-10T04:31:38.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: 206 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:38.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: cluster 2021-02-10T04:31: 2021-02-10T04:31:38.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: 36.020120+0000 osd.2 (osd.2) 207 : cluster 2021-02-10T04:31:38.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:38.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: 2021-02-10T04:31:36.020129+0000 osd.2 (osd.2) 2021-02-10T04:31:38.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: 208 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:38.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:31:38.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: :31:36.020135+0000 osd.2 (osd.2) 209 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:38.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: 2021-02-10T04:31:38.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:38.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: 02-10T04:31:36.020143+0000 osd.2 (osd.2 2021-02-10T04:31:38.460 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: ) 210 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:38.460 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: cluster 2021-02-10T04:31:36. 2021-02-10T04:31:38.460 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: 020149+0000 osd.2 (osd.2) 211 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:38.460 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: cluster 2021-02-10T04:31:36.020155+0000 2021-02-10T04:31:38.460 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: osd.2 (osd.2) 212 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:38.460 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:31:38.460 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: :31:36.020162+0000 osd.2 (osd.2 2021-02-10T04:31:38.460 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: ) 213 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:38.460 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: cluster 2021-02-10T04:31:36 2021-02-10T04:31:38.460 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:36 smithi164 conmon[29288]: .036802+0000 mgr.y (mgr.14140) 446 : cluster [DBG] pgmap v567: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 3 active+clean+snaptrim, 163 active+clean; 145 MiB data, 520 MiB used, 715 GiB / 715 GiB avail; 19 MiB/s rd, 4.6 MiB/s wr, 26 op/s 2021-02-10T04:31:38.460 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: cluster 2021-02-10T04:31:36.214301+0000 osd.1 2021-02-10T04:31:38.460 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: (osd.1) 202 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:38.460 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: cluster 2021-02-10T04:31:36.214319+0000 osd.1 (osd.1) 203 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:38.461 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: cluster 2021-02-10T04:31:36.214346+0000 osd.1 (osd.1) 2021-02-10T04:31:38.461 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: 204 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:38.461 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:38.461 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: -10T04:31:36.214352+0000 osd.1 (osd 2021-02-10T04:31:38.461 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: .1) 205 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:38.461 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:38.461 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: -02-10T04:31:36.214358+0000 2021-02-10T04:31:38.461 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: osd.1 (osd.1) 206 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:38.461 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: 2021-02-10T04:31:38.461 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: cluster 2021-02-10T04:31:36 2021-02-10T04:31:38.461 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: .214364+0000 osd.1 (osd.1) 207 : cluster 2021-02-10T04:31:38.461 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:38.461 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:38.461 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: T04:31:36.343193+0000 mon.a (mon 2021-02-10T04:31:38.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: .0) 1018 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2021-02-10T04:31:38.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:38.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: -02-10T04:31:36.352614+0000 osd.4 2021-02-10T04:31:38.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: (osd.4) 219 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:38.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: 2021-02-10T04:31:36.352623+0000 2021-02-10T04:31:38.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: osd.4 (osd.4) 220 : cluster [WRN] 2021-02-10T04:31:38.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:38.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: cluster 2021-02-10T04:31 2021-02-10T04:31:38.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: :36.352631+0000 osd.4 (osd.4) 221 2021-02-10T04:31:38.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:38.462 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:38.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: 10T04:31:36.352637+0000 osd.4 ( 2021-02-10T04:31:38.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: osd.4) 222 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:38.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:38.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: -02-10T04:31:36.993293+0000 osd.2 ( 2021-02-10T04:31:38.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: osd.2) 214 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:38.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:38.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: 02-10T04:31:36.993302+0000 osd.2 ( 2021-02-10T04:31:38.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: osd.2) 215 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:38.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:38.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: 02-10T04:31:36.993308+0000 osd.2 2021-02-10T04:31:38.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: (osd.2) 216 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:38.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: 2021-02-10T04:31:38.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:38.463 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: -02-10T04:31:36.993314+0000 osd.2 (osd. 2021-02-10T04:31:38.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: 2) 217 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:38.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:38.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: T04:31:36.993335+0000 osd.2 (osd. 2021-02-10T04:31:38.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: 2) 218 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:38.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:38.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: 10T04:31:36.993340+0000 osd.2 ( 2021-02-10T04:31:38.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: osd.2) 2021-02-10T04:31:38.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: 219 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:38.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:31:38.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: :31:36.993345+0000 osd.2 (osd.2) 2021-02-10T04:31:38.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: 220 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:38.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:31:38.465 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: 04:31:36.993350+0000 osd.2 (osd. 2021-02-10T04:31:38.465 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[29288]: 2) 221 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:38.465 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:31:35 smithi164 conmon[37263]: debug 2021-02-10T04:31:35.322+0000 7f1c689e5700 -1 osd.5 pg_epoch: 278 pg[7.0( v 276'224 (0'0,276'224] local-lis/les=93/94 n=3 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [5,1,3] r=0 lpr=93 crt=276'224 lcod 271'222 mlcod 271'222 active+clean+snaptrim trimq=[b2~1] ps=41] removing snap head 2021-02-10T04:31:38.465 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:31:37 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:31:38.465 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: cluster 2021-02-10T04:31:38.466 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: 2021-02-10T04:31:38.466 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: 2021 2021-02-10T04:31:38.466 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: -02-10T04:31:31.988626 2021-02-10T04:31:38.466 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 174 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:38.466 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:38.466 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: 02-10T04:31:31.988636+0000 osd.2 ( 2021-02-10T04:31:38.466 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: osd.2) 175 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:38.466 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:31:38.466 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: 04:31:31.988650+0000 osd.2 (osd. 2021-02-10T04:31:38.466 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: 2) 176 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:38.466 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: cluster 2021-02-10T04:31:31.988656+0000 osd.2 (osd.2 2021-02-10T04:31:38.466 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: ) 177 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:38.466 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: cluster 2021-02-10T04:31:38.467 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: 2021-02-10T04:31:31.988659+0000 2021-02-10T04:31:38.467 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: osd.2 (osd.2) 178 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:38.467 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: cluster 2021-02-10T04:31:38.467 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: 2021-02-10T04:31:31.988664+0000 osd.2 2021-02-10T04:31:38.467 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: (osd.2) 179 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:38.467 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:38.467 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: 02-10T04:31:31.988679+0000 osd.2 2021-02-10T04:31:38.467 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: (osd.2) 180 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:38.467 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:38.467 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: 02-10T04:31:31.988683+0000 osd.2 2021-02-10T04:31:38.467 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: (osd.2) 181 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:38.467 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:38.467 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: 02-10T04:31:32.223293+0000 2021-02-10T04:31:38.468 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: osd.1 (osd.1) 178 : cluster [WRN] 2021-02-10T04:31:38.468 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:38.468 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: cluster 2021-02-10T04:31:32 2021-02-10T04:31:38.468 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: .223303+0000 osd.1 (osd.1) 179 : cluster 2021-02-10T04:31:38.468 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:38.468 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: cluster 2021-02-10T04:31:32 2021-02-10T04:31:38.468 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: .223312+0000 osd.1 (osd.1) 180 : cluster [WRN] 2021-02-10T04:31:38.468 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:38.468 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: cluster 2021-02-10T04:31 2021-02-10T04:31:38.468 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: :32.223318+0000 osd.1 (osd.1) 181 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:38.468 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: cluster 2021-02-10T04:31:32 2021-02-10T04:31:38.468 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: .223323+0000 osd.1 (osd.1) 182 : cluster 2021-02-10T04:31:38.468 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:38.468 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: cluster 2021-02-10T04:31:32 2021-02-10T04:31:38.469 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: .223331+0000 osd.1 (osd.1) 183 : cluster 2021-02-10T04:31:38.469 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:38.469 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:38.469 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: -02-10T04:31:32.303740 2021-02-10T04:31:38.469 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1013 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2021-02-10T04:31:38.469 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:38.469 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: -10T04:31:32.431618+0000 osd.4 (osd.4) 203 2021-02-10T04:31:38.469 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:38.469 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:38.469 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: 02-10T04:31:32.431624+0000 osd.4 (osd.4) 204 : cluster 2021-02-10T04:31:38.469 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:38.469 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: cluster 2021-02-10T04:31:32.431639 2021-02-10T04:31:38.469 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 205 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:38.470 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: cluster 2021-02-10T04:31:32.431647 2021-02-10T04:31:38.470 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 206 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:38.470 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: 2021-02-10T04:31:38.470 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:38.470 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: -02-10T04:31:33.074918+0000 mon.a (mon.0 2021-02-10T04:31:38.470 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[29319]: ) 1014 : cluster [WRN] Health check update: 18 slow ops, oldest one blocked for 137 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:31:38.470 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: cluster 2021-02-10T04:31:33.002211+0000 osd.2 (osd.2) 182 2021-02-10T04:31:38.470 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:38.470 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: cluster 2021-02-10T04:31:33.002219+0000 osd.2 (osd.2) 183 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:38.470 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:38.470 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: 10T04:31:33.002225+0000 osd.2 (osd.2) 184 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:38.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: cluster 2021-02-10T04:31:33 2021-02-10T04:31:38.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: .002231+0000 osd.2 (osd.2) 185 : 2021-02-10T04:31:38.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:38.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: cluster 2021-02-10T04:31 2021-02-10T04:31:38.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: :33.002237+0000 osd.2 (osd.2) 186 : cluster [WRN] 2021-02-10T04:31:38.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:38.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: cluster 2021-02-10T04:31:38.472 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: 2021-02-10T04:31:33.002244+0000 osd.2 (osd 2021-02-10T04:31:38.472 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: .2) 187 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:38.472 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:38.472 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: -10T04:31:33.002249+0000 osd.2 2021-02-10T04:31:38.472 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: (osd. 2021-02-10T04:31:38.472 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: 2) 188 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:38.472 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:38.472 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: -10T04:31:33.002266 2021-02-10T04:31:38.472 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: +0000 osd.2 (osd.2 2021-02-10T04:31:38.472 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: ) 189 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:38.472 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:38.473 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: 10T04:31:33.221219+0000 osd.1 2021-02-10T04:31:38.473 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: (osd.1) 184 : cluster 2021-02-10T04:31:38.473 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:38.473 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: cluster 2021-02-10T04:31 2021-02-10T04:31:38.473 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: :33.221225+0000 osd.1 (osd.1) 185 2021-02-10T04:31:38.473 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:38.473 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:31:38.473 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: :31:33.221229+0000 osd.1 (osd. 2021-02-10T04:31:38.473 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: 1) 186 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:38.474 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:38.474 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: 10T04:31:33.221232+0000 osd.1 (osd 2021-02-10T04:31:38.474 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: .1) 187 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:38.474 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:38.474 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: -10T04:31:33.221235+0000 osd.1 ( 2021-02-10T04:31:38.474 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: osd.1) 188 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:38.474 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:38.474 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: -10T04:31:33.221240+0000 osd.1 2021-02-10T04:31:38.474 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: (osd.1) 189 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:38.475 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: 2021-02-10T04:31:38.475 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: cluster 2021-02-10T04:31:38.475 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: 2021-02-10T04:31:33.319260+0000 mon.a ( 2021-02-10T04:31:38.475 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: mon.0) 1015 : cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2021-02-10T04:31:38.475 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:38.475 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: -02-10T04:31:33.382481+0000 2021-02-10T04:31:38.475 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: osd.4 (osd.4) 207 : cluster [WRN] 2021-02-10T04:31:38.475 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:38.475 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: cluster 2021-02-10T04:31:33.382487 2021-02-10T04:31:38.475 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 208 : cluster 2021-02-10T04:31:38.475 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:38.475 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: cluster 2021-02-10T04:31:33. 2021-02-10T04:31:38.475 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: 382491+0000 osd.4 (osd.4) 209 : cluster 2021-02-10T04:31:38.476 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:38.476 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: cluster 2021-02-10T04:31:33 2021-02-10T04:31:38.476 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: .382494+0000 osd.4 (osd.4) 210 : cluster 2021-02-10T04:31:38.476 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:38.476 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[29319]: cluster 2021-02-10T04:31:34.036251+0000 mgr.y (mgr.14140) 445 : cluster [DBG] pgmap v564: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 154 MiB data, 521 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 780 KiB/s wr, 6 op/s 2021-02-10T04:31:38.476 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: cluster 2021-02-10T04:31:38.476 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: 2021-02-10T04:31:38.476 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: 2021 2021-02-10T04:31:38.476 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: - 2021-02-10T04:31:38.476 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: 02 2021-02-10T04:31:38.476 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: - 2021-02-10T04:31:38.476 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: 10 2021-02-10T04:31:38.476 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: T 2021-02-10T04:31:38.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: 04 2021-02-10T04:31:38.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: : 2021-02-10T04:31:38.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: 31:33.954238+0000 2021-02-10T04:31:38.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: osd.2 (osd.2) 190 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:38.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: cluster 2021-02-10T04:31:38.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: 2021-02 2021-02-10T04:31:38.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: -10T04:31:33.954245+0000 osd.2 (osd.2 2021-02-10T04:31:38.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: ) 191 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:38.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: cluster 2021-02-10T04:31:38.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: 2021-02 2021-02-10T04:31:38.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: -10T04:31:33.954249+0000 osd.2 (osd.2 2021-02-10T04:31:38.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: ) 192 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:38.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: cluster 2021-02-10T04:31:38.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: 2021-02-10T04:31:33.954252+0000 2021-02-10T04:31:38.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: osd.2 (osd.2) 193 : cluster 2021-02-10T04:31:38.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:38.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:31:38.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: 31:33.954258+0000 osd.2 (osd.2 2021-02-10T04:31:38.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: ) 194 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:38.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:31:38.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: 31:33.954261+0000 osd.2 (osd.2 2021-02-10T04:31:38.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: ) 195 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:38.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:38.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: 10T04:31:33.954264+0000 osd.2 ( 2021-02-10T04:31:38.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: osd.2) 196 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:38.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:38.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: -02-10T04:31:33.954267+0000 2021-02-10T04:31:38.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: osd.2 (osd.2) 197 : cluster [WRN] 2021-02-10T04:31:38.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:38.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: cluster 2021-02-10T04:31:34 2021-02-10T04:31:38.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: .251037+0000 osd.1 (osd.1) 190 : 2021-02-10T04:31:38.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:38.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:31:38.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: :31:34.251046+0000 osd.1 (osd. 2021-02-10T04:31:38.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: 1) 191 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:38.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:38.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: 10T04:31:34.251058+0000 osd.1 2021-02-10T04:31:38.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: (osd.1) 192 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:38.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:38.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: -02-10T04:31:34.251068+0000 2021-02-10T04:31:38.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: osd.1 (osd.1) 193 : cluster [WRN] 2021-02-10T04:31:38.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:38.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: cluster 2021-02-10T04:31:34 2021-02-10T04:31:38.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: .251076+0000 osd.1 (osd.1) 194 : 2021-02-10T04:31:38.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:38.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:31:38.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: :31:34.251082+0000 osd.1 (osd.1) 195 : cluster 2021-02-10T04:31:38.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:38.483 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: cluster 2021-02-10T04:31:34 2021-02-10T04:31:38.483 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: .324099+0000 mon.a (mon.0) 1016 : cluster 2021-02-10T04:31:38.483 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: [DBG] osdmap e277: 8 total, 8 up, 8 in 2021-02-10T04:31:38.483 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: cluster 2021-02-10T04:31:34.386301 2021-02-10T04:31:38.483 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 211 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:38.483 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: 2021-02-10T04:31:38.483 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: cluster 2021-02-10T04:31:38.483 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:31:38.483 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: 10T04:31:34.386309+0000 osd.4 ( 2021-02-10T04:31:38.483 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: osd.4) 212 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:38.483 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:38.483 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: -02-10T04:31:34.386315+0000 2021-02-10T04:31:38.483 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: osd.4 (osd.4) 213 : cluster [WRN] 2021-02-10T04:31:38.484 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:38.484 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: cluster 2021-02-10T04:31:34. 2021-02-10T04:31:38.484 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: 386321+0000 osd.4 (osd.4) 214 : 2021-02-10T04:31:38.484 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:38.484 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: cluster 2021-02-10T04:31:34.970725+0000 osd.2 (osd.2 2021-02-10T04:31:38.484 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: ) 198 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:38.484 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: cluster 2021-02-10T04:31:34.970734+0000 osd.2 (osd.2) 199 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:38.485 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:38.486 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: 02-10T04:31:34.970740+0000 osd.2 (osd.2) 200 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:38.486 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: cluster 2021-02-10T04:31:34. 2021-02-10T04:31:38.486 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: 970746+0000 osd.2 (osd.2) 201 : cluster 2021-02-10T04:31:38.486 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:38.486 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: cluster 2021-02-10T04:31 2021-02-10T04:31:38.486 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: :34.970753+0000 osd.2 (osd.2) 202 2021-02-10T04:31:38.486 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:38.486 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:31:38.486 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: :31:34.970758+0000 osd.2 (osd. 2021-02-10T04:31:38.486 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: 2) 203 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:38.486 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:38.487 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: -10T04:31:34.970770+0000 osd.2 2021-02-10T04:31:38.487 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: (osd.2) 204 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:38.487 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: 2021-02-10T04:31:38.487 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: cluster 2021-02-10T04:31:38.487 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: 2021-02-10T04:31:34.970793+0000 osd.2 (osd 2021-02-10T04:31:38.487 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: .2) 205 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:38.487 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:38.487 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: 10T04:31:35.204506+0000 osd.1 (osd 2021-02-10T04:31:38.487 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: .1) 196 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:38.487 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:38.487 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: 02-10T04:31:35.204512+0000 2021-02-10T04:31:38.487 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: osd.1 (osd.1) 197 : cluster [WRN] 2021-02-10T04:31:38.487 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:38.488 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: cluster 2021-02-10T04:31: 2021-02-10T04:31:38.488 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: 35.204517+0000 osd.1 (osd.1) 198 2021-02-10T04:31:38.488 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:38.488 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: cluster 2021-02-10T04:31:38.488 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: 2021-02-10T04:31:35.204520 2021-02-10T04:31:38.488 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: +0000 osd.1 (osd.1) 199 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:38.488 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:38.488 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: 02-10T04:31:35.204524+0000 2021-02-10T04:31:38.488 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: osd.1 (osd.1) 200 : cluster [WRN] 2021-02-10T04:31:38.488 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:38.488 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:31:38.488 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: 31:35.204528+0000 osd.1 (osd.1 2021-02-10T04:31:38.488 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: ) 201 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:38.489 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:38.489 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: -02-10T04:31:35.324488+0000 2021-02-10T04:31:38.489 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: mon.a (mon.0) 1017 : cluster [DBG] 2021-02-10T04:31:38.489 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: osdmap e278: 8 total, 8 up, 8 in 2021-02-10T04:31:38.489 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:31:38.489 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: 04:31:35.363160+0000 osd.4 (osd 2021-02-10T04:31:38.489 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: .4) 215 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:38.489 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:38.489 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: -02-10T04:31:35.363171+0000 2021-02-10T04:31:38.489 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: osd.4 (osd.4) 216 : cluster [WRN] 2021-02-10T04:31:38.490 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:38.490 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:31:38.490 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: 31:35.363181+0000 osd.4 (osd.4 2021-02-10T04:31:38.490 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: ) 217 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:38.490 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:38.490 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: -02-10T04:31:35.363186+0000 2021-02-10T04:31:38.490 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: osd.4 (osd.4) 218 : cluster [WRN] 2021-02-10T04:31:38.490 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:38.490 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:31:38.490 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: 31:36.020100+0000 osd.2 (osd.2 2021-02-10T04:31:38.490 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: ) 206 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:38.490 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: cluster 2021-02-10T04:31:38.490 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: 2021-02-10T04:31:36.020120 2021-02-10T04:31:38.491 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 207 : cluster 2021-02-10T04:31:38.491 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:38.491 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: cluster 2021-02-10T04:31:38.491 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: 2021-02-10T04:31:36.020129+0000 osd.2 (osd 2021-02-10T04:31:38.491 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: .2) 208 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:38.491 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:38.491 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: -10T04:31:36.020135+0000 osd.2 ( 2021-02-10T04:31:38.491 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: osd.2) 209 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:38.491 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: cluster 2021-02-10T04:31:36.020143+0000 osd.2 (osd. 2021-02-10T04:31:38.491 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: 2) 210 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:38.491 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: cluster 2021-02-10T04:31:38.491 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: 2021- 2021-02-10T04:31:38.491 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: 02-10T04:31:36.020149+0000 2021-02-10T04:31:38.492 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: osd.2 (osd.2) 211 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:38.492 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: 2021-02-10T04:31:38.492 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:38.492 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: 02-10T04:31:36.020155+0000 2021-02-10T04:31:38.492 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: osd.2 (osd.2) 212 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:38.492 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: 2021-02-10T04:31:38.492 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:38.492 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: -10T04:31:36.020162+0000 osd.2 (osd 2021-02-10T04:31:38.492 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: .2) 213 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:38.492 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: cluster 2021-02-10T04:31:38.492 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: 2021-02-10T04:31:36.036802 2021-02-10T04:31:38.492 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 446 : cluster [DBG] 2021-02-10T04:31:38.492 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[29319]: pgmap v567: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 3 active+clean+snaptrim, 163 active+clean; 145 MiB data, 520 MiB used, 715 GiB / 715 GiB avail; 19 MiB/s rd, 4.6 MiB/s wr, 26 op/s 2021-02-10T04:31:38.493 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:31:38.493 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: 31:36.214301+0000 osd.1 (osd.1) 202 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:38.493 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: cluster 2021-02-10T04:31:36.214319+0000 2021-02-10T04:31:38.493 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: osd.1 (osd.1) 203 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:38.493 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: cluster 2021-02-10T04:31:36 2021-02-10T04:31:38.493 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: .214346+0000 osd.1 (osd.1) 204 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:38.493 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:38.493 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: -02-10T04:31:36.214352+0000 2021-02-10T04:31:38.493 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: osd.1 (osd.1) 205 : cluster [WRN] 2021-02-10T04:31:38.493 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:38.493 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:31:38.493 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: :31:36.214358+0000 osd.1 (osd.1) 206 : cluster 2021-02-10T04:31:38.493 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:38.494 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:38.494 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: T04:31:36.214364+0000 osd.1 (osd.1) 207 : cluster [WRN] 2021-02-10T04:31:38.494 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:38.494 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: cluster 2021-02-10T04:31:38.494 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: 2021-02-10T04:31:36.343193+0000 2021-02-10T04:31:38.494 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: mon.a (mon.0) 1018 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2021-02-10T04:31:38.494 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:38.494 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: 02-10T04:31: 2021-02-10T04:31:38.494 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: 36.352614+0000 osd.4 (osd.4) 219 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:38.494 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:38.494 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: 10T04:31:36 2021-02-10T04:31:38.494 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: .352623+0000 osd.4 (osd.4) 220 2021-02-10T04:31:38.495 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:38.495 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:38.495 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: -02-10T04:31:36.352631+0000 osd.4 (osd. 2021-02-10T04:31:38.495 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: 4) 221 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:38.495 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: cluster 2021-02-10T04:31: 2021-02-10T04:31:38.495 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: 36.352637+0000 osd.4 (osd.4) 222 : cluster [WRN] 2021-02-10T04:31:38.495 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:38.495 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: cluster 2021-02-10T04:31:38.495 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: 2021-02-10T04:31:36.993293+0000 osd.2 (osd.2) 214 : cluster 2021-02-10T04:31:38.495 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:38.495 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:31:38.495 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: 04:31:36.993302+0000 osd.2 (osd.2) 215 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:38.495 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: cluster 2021-02-10T04:31:36.993308+0000 osd.2 ( 2021-02-10T04:31:38.496 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: osd.2) 216 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:38.496 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: cluster 2021-02-10T04:31 2021-02-10T04:31:38.496 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: :36.993314+0000 osd.2 (osd.2) 217 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:38.496 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: cluster 2021-02-10T04:31: 2021-02-10T04:31:38.496 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: 36.993335+0000 osd.2 (osd.2) 218 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:38.496 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: cluster 2021-02-10T04:31:36 2021-02-10T04:31:38.496 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: .993340+0000 osd.2 (osd.2) 219 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:38.496 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: cluster 2021-02-10T04:31 2021-02-10T04:31:38.496 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: :36.993345+0000 osd.2 (osd.2) 220 : cluster 2021-02-10T04:31:38.496 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:38.496 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:38.496 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: -02-10T04:31:36.993350+0000 osd.2 (osd.2) 221 : cluster [WRN] 2021-02-10T04:31:38.496 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:38.496 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: cluster 2021-02-10T04:31:37.167085+0000 osd.1 2021-02-10T04:31:38.497 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: (osd.1) 208 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:38.497 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: cluster 2021-02-10T04:31:37.167092+0000 osd.1 (osd.1) 209 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:38.497 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: cluster 2021-02-10T04:31:37.167102+0000 osd.1 (osd.1) 210 2021-02-10T04:31:38.497 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:38.497 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: cluster 2021-02-10T04:31:37.167107+0000 osd.1 2021-02-10T04:31:38.497 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: (osd.1) 211 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:38.497 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: cluster 2021-02-10T04:31:37.167114+0000 osd.1 (osd. 2021-02-10T04:31:38.497 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: 1) 212 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:38.497 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: cluster 2021-02-10T04:31:37 2021-02-10T04:31:38.497 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: .167120+0000 osd.1 (osd.1) 213 : cluster [WRN] 2021-02-10T04:31:38.497 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:38.497 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: cluster 2021-02-10T04:31:38.497 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: 2021-02-10T04:31:37.353794+0000 2021-02-10T04:31:38.498 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: mon.a (mon.0) 1019 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2021-02-10T04:31:38.498 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: 2021-02-10T04:31:38.498 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: cluster 2021-02-10T04:31:38.498 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: 2021-02-10T04:31:37.367331+0000 osd.4 (osd. 2021-02-10T04:31:38.498 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: 4) 223 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:38.498 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:38.498 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: 02-10T04:31:37.367338 2021-02-10T04:31:38.498 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 224 2021-02-10T04:31:38.498 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:38.498 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:38.498 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: 02-10T04:31:37.367351 2021-02-10T04:31:38.498 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 225 : cluster [WRN] 2021-02-10T04:31:38.498 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:38.499 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: cluster 2021-02-10T04:31:38.499 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: 2021-02-10T04:31:37.367358+0000 2021-02-10T04:31:38.499 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: osd.4 (osd.4) 226 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:38.499 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: cluster 2021-02-10T04:31 2021-02-10T04:31:38.499 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: :37.684195+0000 osd.7 (osd.7) 44 : cluster [DBG] 2021-02-10T04:31:38.499 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: purged_snaps scrub starts 2021-02-10T04:31:38.499 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:38.499 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: 10T04:31:37.684403+0000 osd.7 (osd.7) 45 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:31:38.499 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: audit 2021-02-10T04:31:37 2021-02-10T04:31:38.499 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: .855048+0000 mgr.y (mgr.14140) 447 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:31:38.499 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: cluster 2021-02-10T04:31: 2021-02-10T04:31:38.499 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: 38.037227+0000 mgr.y (mgr.14140) 448 : cluster [DBG] pgmap v570: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 3 active+clean+snaptrim, 163 active+clean; 145 MiB data, 524 MiB used, 715 GiB / 715 GiB avail; 19 MiB/s rd, 4.6 MiB/s wr, 24 op/s 2021-02-10T04:31:38.499 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: 2021-02-10T04:31:38.500 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:38.500 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: -10T04:31:38.042397+0000 osd.2 (osd.2) 222 2021-02-10T04:31:38.500 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:38.500 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: cluster 2021-02-10T04:31:38.042402+0000 osd.2 ( 2021-02-10T04:31:38.500 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: osd.2) 223 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:38.500 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: cluster 2021-02-10T04:31:38.042407+0000 2021-02-10T04:31:38.500 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: osd.2 (osd.2) 224 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:38.500 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: cluster 2021-02-10T04:31:38.042411 2021-02-10T04:31:38.500 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 225 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:38.500 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: cluster 2021-02-10T04:31:38.042414+0000 osd.2 (osd.2) 226 2021-02-10T04:31:38.500 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:38.500 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:38.500 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: -10T04:31:38.042418+0000 osd.2 (osd.2) 227 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:38.500 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: 2021-02-10T04:31:38.501 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:38.501 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: T04:31:38.042421+0000 osd.2 (osd.2) 228 : cluster 2021-02-10T04:31:38.501 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:38.501 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: cluster 2021-02-10T04:31:38.042426+0000 osd.2 ( 2021-02-10T04:31:38.501 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: osd.2) 229 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:38.501 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: audit 2021-02-10T04:31:38.063845+0000 2021-02-10T04:31:38.501 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: mon.a (mon.0) 1020 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:31:38.501 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: audit 2021-02-10T04:31 2021-02-10T04:31:38.501 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: :38.064231+0000 mon.a (mon.0) 2021-02-10T04:31:38.501 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: 1021 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:31:38.501 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: cluster 2021-02-10T04:31:38.501 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:31:38.501 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: 04:31:38.076514+0000 mon.a (mon.0) 1022 : 2021-02-10T04:31:38.502 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[29319]: cluster [WRN] Health check update: 18 slow ops, oldest one blocked for 142 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:31:38.502 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[45443]: debug 2021-02-10T04:31:38.502 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[45443]: 2021-02-10T04:31:33.220+0000 7f7140d73700 -1 osd.1 275 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:31:38.502 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[45443]: 2021-02-10T04:31:38.502 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[45443]: debug 2021-02-10T04:31:38.502 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[45443]: 2021-02-10T04:31:34.250+0000 7f7140d73700 -1 osd.1 276 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:31:38.502 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[45443]: debug 2021-02-10T04:31:35.203+0000 7f7140d73700 -1 osd.1 277 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:31:38.502 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[45443]: debug 2021-02-10T04:31:36.213+0000 7f7140d73700 -1 osd.1 278 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:31:38.502 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[45443]: debug 2021-02-10T04:31:38.502 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[45443]: 2021-02-10T04:31:37.166+0000 7f7140d73700 -1 osd.1 279 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:31:38.503 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[45443]: 2021-02-10T04:31:38.503 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[45443]: debug 2021-02-10T04:31:38.201+0000 7f7140d73700 -1 osd.1 280 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:31:38.503 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[49106]: debug 2021-02-10T04:31:33.001+0000 7f77a3e94700 -1 osd.2 275 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:31:38.503 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[49106]: debug 2021-02-10T04:31:33.953+0000 7f77a3e94700 -1 osd.2 276 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:31:38.503 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[49106]: debug 2021-02-10T04:31:34.970+0000 7f77a3e94700 -1 osd.2 277 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:31:38.503 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[49106]: debug 2021-02-10T04:31:36.019+0000 7f77a3e94700 -1 osd.2 278 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:31:38.503 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[49106]: debug 2021-02-10T04:31:36.992+0000 7f77a3e94700 -1 osd.2 279 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:31:38.503 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[49106]: debug 2021-02-10T04:31:38.041+0000 7f77a3e94700 -1 osd.2 280 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:31:38.504 INFO:tasks.rados.rados.0.smithi085.stdout:932: done (14 left) 2021-02-10T04:31:38.504 INFO:tasks.rados.rados.0.smithi085.stdout:933: done (13 left) 2021-02-10T04:31:38.504 INFO:tasks.rados.rados.0.smithi085.stdout:934: done (12 left) 2021-02-10T04:31:38.504 INFO:tasks.rados.rados.0.smithi085.stdout:935: write (excl) oid 11 current snap is 89 2021-02-10T04:31:38.504 INFO:tasks.rados.rados.0.smithi085.stdout:935: seq_num 347 ranges {632029=547288,1639134=749453,2796249=437859,3653524=178505} 2021-02-10T04:31:38.504 INFO:tasks.rados.rados.0.smithi085.stdout:935: writing smithi08575397-11 from 632029 to 1179317 tid 1 2021-02-10T04:31:38.504 INFO:tasks.rados.rados.0.smithi085.stdout:935: writing smithi08575397-11 from 1639134 to 2388587 tid 2 2021-02-10T04:31:38.504 INFO:tasks.rados.rados.0.smithi085.stdout:935: writing smithi08575397-11 from 2796249 to 3234108 tid 3 2021-02-10T04:31:38.504 INFO:tasks.rados.rados.0.smithi085.stdout:935: writing smithi08575397-11 from 3653524 to 3832029 tid 4 2021-02-10T04:31:38.504 INFO:tasks.rados.rados.0.smithi085.stdout:936: write (excl) oid 16 current snap is 89 2021-02-10T04:31:38.504 INFO:tasks.rados.rados.0.smithi085.stdout:936: seq_num 348 ranges {648836=622537,1981263=493111,3107319=741517} 2021-02-10T04:31:38.504 INFO:tasks.rados.rados.0.smithi085.stdout:936: writing smithi08575397-16 from 648836 to 1271373 tid 1 2021-02-10T04:31:38.505 INFO:tasks.rados.rados.0.smithi085.stdout:936: writing smithi08575397-16 from 1981263 to 2474374 tid 2 2021-02-10T04:31:38.505 INFO:tasks.rados.rados.0.smithi085.stdout:936: writing smithi08575397-16 from 3107319 to 3848836 tid 3 2021-02-10T04:31:38.505 INFO:tasks.rados.rados.0.smithi085.stdout:937: delete oid 5 current snap is 89 2021-02-10T04:31:38.505 INFO:tasks.rados.rados.0.smithi085.stdout:935: finishing write tid 1 to smithi08575397-11 2021-02-10T04:31:38.505 INFO:tasks.rados.rados.0.smithi085.stdout:935: finishing write tid 2 to smithi08575397-11 2021-02-10T04:31:38.505 INFO:tasks.rados.rados.0.smithi085.stdout:935: finishing write tid 3 to smithi08575397-11 2021-02-10T04:31:38.505 INFO:tasks.rados.rados.0.smithi085.stdout:935: finishing write tid 4 to smithi08575397-11 2021-02-10T04:31:38.505 INFO:tasks.rados.rados.0.smithi085.stdout:935: finishing write tid 5 to smithi08575397-11 2021-02-10T04:31:38.505 INFO:tasks.rados.rados.0.smithi085.stdout:935: finishing write tid 6 to smithi08575397-11 2021-02-10T04:31:38.505 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 476 (ObjNum 347 snap 89 seq_num 347) dirty exists 2021-02-10T04:31:38.505 INFO:tasks.rados.rados.0.smithi085.stdout:935: left oid 11 (ObjNum 347 snap 89 seq_num 347) 2021-02-10T04:31:38.505 INFO:tasks.rados.rados.0.smithi085.stdout:936: finishing write tid 1 to smithi08575397-16 2021-02-10T04:31:38.505 INFO:tasks.rados.rados.0.smithi085.stdout:936: finishing write tid 2 to smithi08575397-16 2021-02-10T04:31:38.506 INFO:tasks.rados.rados.0.smithi085.stdout:935: done (14 left) 2021-02-10T04:31:38.506 INFO:tasks.rados.rados.0.smithi085.stdout:937: done (13 left) 2021-02-10T04:31:38.506 INFO:tasks.rados.rados.0.smithi085.stdout:938: snap_create 2021-02-10T04:31:38.506 INFO:tasks.rados.rados.0.smithi085.stdout:936: finishing write tid 3 to smithi08575397-16 2021-02-10T04:31:38.506 INFO:tasks.rados.rados.0.smithi085.stdout:936: finishing write tid 4 to smithi08575397-16 2021-02-10T04:31:38.506 INFO:tasks.rados.rados.0.smithi085.stdout:936: finishing write tid 5 to smithi08575397-16 2021-02-10T04:31:38.506 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 481 (ObjNum 348 snap 89 seq_num 348) dirty exists 2021-02-10T04:31:38.506 INFO:tasks.rados.rados.0.smithi085.stdout:936: left oid 16 (ObjNum 348 snap 89 seq_num 348) 2021-02-10T04:31:38.506 INFO:tasks.rados.rados.0.smithi085.stdout:936: done (13 left) 2021-02-10T04:31:38.506 INFO:tasks.rados.rados.0.smithi085.stdout:938: done (12 left) 2021-02-10T04:31:38.506 INFO:tasks.rados.rados.0.smithi085.stdout:939: read oid 38 snap -1 2021-02-10T04:31:38.506 INFO:tasks.rados.rados.0.smithi085.stdout:939: expect (ObjNum 287 snap 71 seq_num 287) 2021-02-10T04:31:38.507 INFO:tasks.rados.rados.0.smithi085.stdout:940: delete oid 50 current snap is 90 2021-02-10T04:31:38.507 INFO:tasks.rados.rados.0.smithi085.stdout:939: expect (ObjNum 287 snap 71 seq_num 287) 2021-02-10T04:31:38.507 INFO:tasks.rados.rados.0.smithi085.stdout:939: done (13 left) 2021-02-10T04:31:38.507 INFO:tasks.rados.rados.0.smithi085.stdout:940: done (12 left) 2021-02-10T04:31:38.507 INFO:tasks.rados.rados.0.smithi085.stdout:941: copy_from oid 22 from oid 23 current snap is 90 2021-02-10T04:31:38.507 INFO:tasks.rados.rados.0.smithi085.stdout:942: read oid 50 snap -1 2021-02-10T04:31:38.507 INFO:tasks.rados.rados.0.smithi085.stdout:942: expect deleted 2021-02-10T04:31:38.507 INFO:tasks.rados.rados.0.smithi085.stdout:943: read oid 5 snap 89 2021-02-10T04:31:38.507 INFO:tasks.rados.rados.0.smithi085.stdout:943: expect deleted 2021-02-10T04:31:38.507 INFO:tasks.rados.rados.0.smithi085.stdout:944: delete oid 43 current snap is 90 2021-02-10T04:31:38.507 INFO:tasks.rados.rados.0.smithi085.stdout:942: done (15 left) 2021-02-10T04:31:38.507 INFO:tasks.rados.rados.0.smithi085.stdout:943: done (14 left) 2021-02-10T04:31:38.507 INFO:tasks.rados.rados.0.smithi085.stdout:944: done (13 left) 2021-02-10T04:31:38.507 INFO:tasks.rados.rados.0.smithi085.stdout:945: read oid 2 snap -1 2021-02-10T04:31:38.508 INFO:tasks.rados.rados.0.smithi085.stdout:945: expect (ObjNum 321 snap 81 seq_num 321) 2021-02-10T04:31:38.508 INFO:tasks.rados.rados.0.smithi085.stdout:946: snap_remove snap 88 2021-02-10T04:31:38.508 INFO:tasks.rados.rados.0.smithi085.stdout:946: done (14 left) 2021-02-10T04:31:38.508 INFO:tasks.rados.rados.0.smithi085.stdout:947: write oid 7 current snap is 90 2021-02-10T04:31:38.508 INFO:tasks.rados.rados.0.smithi085.stdout:947: seq_num 352 ranges {716064=639886,1815191=500449,2817011=526925} 2021-02-10T04:31:38.508 INFO:tasks.rados.rados.0.smithi085.stdout:947: writing smithi08575397-7 from 716064 to 1355950 tid 1 2021-02-10T04:31:38.508 INFO:tasks.rados.rados.0.smithi085.stdout:947: writing smithi08575397-7 from 1815191 to 2315640 tid 2 2021-02-10T04:31:38.508 INFO:tasks.rados.rados.0.smithi085.stdout:947: writing smithi08575397-7 from 2817011 to 3343936 tid 3 2021-02-10T04:31:38.508 INFO:tasks.rados.rados.0.smithi085.stdout:945: expect (ObjNum 321 snap 81 seq_num 321) 2021-02-10T04:31:38.508 INFO:tasks.rados.rados.0.smithi085.stdout:945: done (14 left) 2021-02-10T04:31:38.508 INFO:tasks.rados.rados.0.smithi085.stdout:948: read oid 46 snap -1 2021-02-10T04:31:38.508 INFO:tasks.rados.rados.0.smithi085.stdout:948: expect deleted 2021-02-10T04:31:38.508 INFO:tasks.rados.rados.0.smithi085.stdout:949: rollback oid 40 current snap is 90 2021-02-10T04:31:38.509 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 40 to 89 2021-02-10T04:31:38.509 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:38.509 INFO:tasks.rados.rados.0.smithi085.stdout:947: finishing write tid 1 to smithi08575397-7 2021-02-10T04:31:38.509 INFO:tasks.rados.rados.0.smithi085.stdout:947: finishing write tid 2 to smithi08575397-7 2021-02-10T04:31:38.509 INFO:tasks.rados.rados.0.smithi085.stdout:947: finishing write tid 3 to smithi08575397-7 2021-02-10T04:31:38.509 INFO:tasks.rados.rados.0.smithi085.stdout:947: finishing write tid 4 to smithi08575397-7 2021-02-10T04:31:38.509 INFO:tasks.rados.rados.0.smithi085.stdout:947: finishing write tid 5 to smithi08575397-7 2021-02-10T04:31:38.509 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 451 (ObjNum 352 snap 90 seq_num 352) dirty exists 2021-02-10T04:31:38.509 INFO:tasks.rados.rados.0.smithi085.stdout:947: left oid 7 (ObjNum 352 snap 90 seq_num 352) 2021-02-10T04:31:38.509 INFO:tasks.rados.rados.0.smithi085.stdout:947: done (15 left) 2021-02-10T04:31:38.509 INFO:tasks.rados.rados.0.smithi085.stdout:950: read oid 26 snap -1 2021-02-10T04:31:38.509 INFO:tasks.rados.rados.0.smithi085.stdout:950: expect (ObjNum 343 snap 88 seq_num 343) 2021-02-10T04:31:38.509 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:38.509 INFO:tasks.rados.rados.0.smithi085.stdout:948: done (15 left) 2021-02-10T04:31:38.510 INFO:tasks.rados.rados.0.smithi085.stdout:951: write (excl) oid 28 current snap is 90 2021-02-10T04:31:38.510 INFO:tasks.rados.rados.0.smithi085.stdout:951: seq_num 353 ranges {732871=715135,1873673=644107,3244434=688437} 2021-02-10T04:31:38.510 INFO:tasks.rados.rados.0.smithi085.stdout:951: writing smithi08575397-28 from 732871 to 1448006 tid 1 2021-02-10T04:31:38.510 INFO:tasks.rados.rados.0.smithi085.stdout:951: writing smithi08575397-28 from 1873673 to 2517780 tid 2 2021-02-10T04:31:38.510 INFO:tasks.rados.rados.0.smithi085.stdout:951: writing smithi08575397-28 from 3244434 to 3932871 tid 3 2021-02-10T04:31:38.510 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:38.510 INFO:tasks.rados.rados.0.smithi085.stdout:950: expect (ObjNum 343 snap 88 seq_num 343) 2021-02-10T04:31:38.510 INFO:tasks.rados.rados.0.smithi085.stdout:950: done (15 left) 2021-02-10T04:31:38.510 INFO:tasks.rados.rados.0.smithi085.stdout:952: delete oid 13 current snap is 90 2021-02-10T04:31:38.510 INFO:tasks.rados.rados.0.smithi085.stdout:949: finishing rollback tid 0 to smithi08575397-40 2021-02-10T04:31:38.510 INFO:tasks.rados.rados.0.smithi085.stdout:949: finishing rollback tid 1 to smithi08575397-40 2021-02-10T04:31:38.510 INFO:tasks.rados.rados.0.smithi085.stdout:949: finishing rollback tid 2 to smithi08575397-40 2021-02-10T04:31:38.510 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 487 (ObjNum 333 snap 84 seq_num 333) dirty exists 2021-02-10T04:31:38.511 INFO:tasks.rados.rados.0.smithi085.stdout:951: finishing write tid 1 to smithi08575397-28 2021-02-10T04:31:38.511 INFO:tasks.rados.rados.0.smithi085.stdout:951: finishing write tid 2 to smithi08575397-28 2021-02-10T04:31:38.511 INFO:tasks.rados.rados.0.smithi085.stdout:951: finishing write tid 3 to smithi08575397-28 2021-02-10T04:31:38.511 INFO:tasks.rados.rados.0.smithi085.stdout:951: finishing write tid 4 to smithi08575397-28 2021-02-10T04:31:38.511 INFO:tasks.rados.rados.0.smithi085.stdout:951: finishing write tid 5 to smithi08575397-28 2021-02-10T04:31:38.511 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 128 (ObjNum 353 snap 90 seq_num 353) dirty exists 2021-02-10T04:31:38.511 INFO:tasks.rados.rados.0.smithi085.stdout:951: left oid 28 (ObjNum 353 snap 90 seq_num 353) 2021-02-10T04:31:38.511 INFO:tasks.rados.rados.0.smithi085.stdout:949: done (15 left) 2021-02-10T04:31:38.511 INFO:tasks.rados.rados.0.smithi085.stdout:951: done (14 left) 2021-02-10T04:31:38.511 INFO:tasks.rados.rados.0.smithi085.stdout:952: done (13 left) 2021-02-10T04:31:38.511 INFO:tasks.rados.rados.0.smithi085.stdout:953: rollback oid 37 current snap is 90 2021-02-10T04:31:38.511 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 37 to 89 2021-02-10T04:31:38.511 INFO:tasks.rados.rados.0.smithi085.stdout:954: write (excl) oid 24 current snap is 90 2021-02-10T04:31:38.511 INFO:tasks.rados.rados.0.smithi085.stdout:954: seq_num 355 ranges {766485=465633,1990637=647776,3299280=667205} 2021-02-10T04:31:38.512 INFO:tasks.rados.rados.0.smithi085.stdout:954: writing smithi08575397-24 from 766485 to 1232118 tid 1 2021-02-10T04:31:38.512 INFO:tasks.rados.rados.0.smithi085.stdout:954: writing smithi08575397-24 from 1990637 to 2638413 tid 2 2021-02-10T04:31:38.512 INFO:tasks.rados.rados.0.smithi085.stdout:954: writing smithi08575397-24 from 3299280 to 3966485 tid 3 2021-02-10T04:31:38.512 INFO:tasks.rados.rados.0.smithi085.stdout:955: write (excl) oid 11 current snap is 90 2021-02-10T04:31:38.512 INFO:tasks.rados.rados.0.smithi085.stdout:953: finishing rollback tid 0 to smithi08575397-37 2021-02-10T04:31:38.512 INFO:tasks.rados.rados.0.smithi085.stdout:955: seq_num 356 ranges {783292=540882,1932766=507787,3210350=572013} 2021-02-10T04:31:38.512 INFO:tasks.rados.rados.0.smithi085.stdout:955: writing smithi08575397-11 from 783292 to 1324174 tid 1 2021-02-10T04:31:38.512 INFO:tasks.rados.rados.0.smithi085.stdout:955: writing smithi08575397-11 from 1932766 to 2440553 tid 2 2021-02-10T04:31:38.512 INFO:tasks.rados.rados.0.smithi085.stdout:955: writing smithi08575397-11 from 3210350 to 3782363 tid 3 2021-02-10T04:31:38.512 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:38.512 INFO:tasks.rados.rados.0.smithi085.stdout:954: finishing write tid 1 to smithi08575397-24 2021-02-10T04:31:38.512 INFO:tasks.rados.rados.0.smithi085.stdout:953: finishing rollback tid 1 to smithi08575397-37 2021-02-10T04:31:38.512 INFO:tasks.rados.rados.0.smithi085.stdout:953: finishing rollback tid 2 to smithi08575397-37 2021-02-10T04:31:38.513 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 491 (ObjNum 326 snap 83 seq_num 326) dirty exists 2021-02-10T04:31:38.513 INFO:tasks.rados.rados.0.smithi085.stdout:954: finishing write tid 2 to smithi08575397-24 2021-02-10T04:31:38.513 INFO:tasks.rados.rados.0.smithi085.stdout:954: finishing write tid 3 to smithi08575397-24 2021-02-10T04:31:38.513 INFO:tasks.rados.rados.0.smithi085.stdout:954: finishing write tid 4 to smithi08575397-24 2021-02-10T04:31:38.513 INFO:tasks.rados.rados.0.smithi085.stdout:954: finishing write tid 5 to smithi08575397-24 2021-02-10T04:31:38.513 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 24 v 133 (ObjNum 355 snap 90 seq_num 355) dirty exists 2021-02-10T04:31:38.513 INFO:tasks.rados.rados.0.smithi085.stdout:954: left oid 24 (ObjNum 355 snap 90 seq_num 355) 2021-02-10T04:31:38.513 INFO:tasks.rados.rados.0.smithi085.stdout:955: finishing write tid 1 to smithi08575397-11 2021-02-10T04:31:38.513 INFO:tasks.rados.rados.0.smithi085.stdout:955: finishing write tid 2 to smithi08575397-11 2021-02-10T04:31:38.513 INFO:tasks.rados.rados.0.smithi085.stdout:953: done (15 left) 2021-02-10T04:31:38.513 INFO:tasks.rados.rados.0.smithi085.stdout:954: done (14 left) 2021-02-10T04:31:38.513 INFO:tasks.rados.rados.0.smithi085.stdout:956: write oid 6 current snap is 90 2021-02-10T04:31:38.513 INFO:tasks.rados.rados.0.smithi085.stdout:956: seq_num 357 ranges {400099=616131,1591248=408851} 2021-02-10T04:31:38.513 INFO:tasks.rados.rados.0.smithi085.stdout:956: writing smithi08575397-6 from 400099 to 1016230 tid 1 2021-02-10T04:31:38.514 INFO:tasks.rados.rados.0.smithi085.stdout:956: writing smithi08575397-6 from 1591248 to 2000099 tid 2 2021-02-10T04:31:38.514 INFO:tasks.rados.rados.0.smithi085.stdout:957: snap_create 2021-02-10T04:31:38.514 INFO:tasks.rados.rados.0.smithi085.stdout:955: finishing write tid 3 to smithi08575397-11 2021-02-10T04:31:38.514 INFO:tasks.rados.rados.0.smithi085.stdout:955: finishing write tid 4 to smithi08575397-11 2021-02-10T04:31:38.514 INFO:tasks.rados.rados.0.smithi085.stdout:955: finishing write tid 5 to smithi08575397-11 2021-02-10T04:31:38.514 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 496 (ObjNum 356 snap 90 seq_num 356) dirty exists 2021-02-10T04:31:38.514 INFO:tasks.rados.rados.0.smithi085.stdout:955: left oid 11 (ObjNum 356 snap 90 seq_num 356) 2021-02-10T04:31:38.514 INFO:tasks.rados.rados.0.smithi085.stdout:956: finishing write tid 1 to smithi08575397-6 2021-02-10T04:31:38.514 INFO:tasks.rados.rados.0.smithi085.stdout:956: finishing write tid 2 to smithi08575397-6 2021-02-10T04:31:38.514 INFO:tasks.rados.rados.0.smithi085.stdout:956: finishing write tid 3 to smithi08575397-6 2021-02-10T04:31:38.514 INFO:tasks.rados.rados.0.smithi085.stdout:956: finishing write tid 4 to smithi08575397-6 2021-02-10T04:31:38.514 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 312 (ObjNum 357 snap 90 seq_num 357) dirty exists 2021-02-10T04:31:38.514 INFO:tasks.rados.rados.0.smithi085.stdout:956: left oid 6 (ObjNum 357 snap 90 seq_num 357) 2021-02-10T04:31:38.515 INFO:tasks.rados.rados.0.smithi085.stdout:955: done (15 left) 2021-02-10T04:31:38.515 INFO:tasks.rados.rados.0.smithi085.stdout:956: done (14 left) 2021-02-10T04:31:38.515 INFO:tasks.rados.rados.0.smithi085.stdout:957: done (13 left) 2021-02-10T04:31:38.515 INFO:tasks.rados.rados.0.smithi085.stdout:958: delete oid 3 current snap is 91 2021-02-10T04:31:38.515 INFO:tasks.rados.rados.0.smithi085.stdout:958: done (13 left) 2021-02-10T04:31:38.515 INFO:tasks.rados.rados.0.smithi085.stdout:959: delete oid 43 current snap is 91 2021-02-10T04:31:38.515 INFO:tasks.rados.rados.0.smithi085.stdout:959: done (13 left) 2021-02-10T04:31:38.515 INFO:tasks.rados.rados.0.smithi085.stdout:960: delete oid 8 current snap is 91 2021-02-10T04:31:38.515 INFO:tasks.rados.rados.0.smithi085.stdout:941: finishing copy_from to smithi08575397-22 2021-02-10T04:31:38.515 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 150 (ObjNum 346 snap 88 seq_num 346) dirty exists 2021-02-10T04:31:38.515 INFO:tasks.rados.rados.0.smithi085.stdout:941: finishing copy_from racing read to smithi08575397-22 2021-02-10T04:31:38.515 INFO:tasks.rados.rados.0.smithi085.stdout:941: done (13 left) 2021-02-10T04:31:38.515 INFO:tasks.rados.rados.0.smithi085.stdout:960: done (12 left) 2021-02-10T04:31:38.516 INFO:tasks.rados.rados.0.smithi085.stdout:961: snap_create 2021-02-10T04:31:38.516 INFO:tasks.rados.rados.0.smithi085.stdout:961: done (12 left) 2021-02-10T04:31:38.516 INFO:tasks.rados.rados.0.smithi085.stdout:962: copy_from oid 37 from oid 5 current snap is 92 2021-02-10T04:31:38.516 INFO:tasks.rados.rados.0.smithi085.stdout:963: read oid 43 snap 91 2021-02-10T04:31:38.516 INFO:tasks.rados.rados.0.smithi085.stdout:963: expect deleted 2021-02-10T04:31:38.516 INFO:tasks.rados.rados.0.smithi085.stdout:964: write (excl) oid 2 current snap is 92 2021-02-10T04:31:38.516 INFO:tasks.rados.rados.0.smithi085.stdout:964: seq_num 361 ranges {467327=633480,1825176=242151} 2021-02-10T04:31:38.516 INFO:tasks.rados.rados.0.smithi085.stdout:964: writing smithi08575397-2 from 467327 to 1100807 tid 1 2021-02-10T04:31:38.516 INFO:tasks.rados.rados.0.smithi085.stdout:964: writing smithi08575397-2 from 1825176 to 2067327 tid 2 2021-02-10T04:31:38.516 INFO:tasks.rados.rados.0.smithi085.stdout:965: write oid 24 current snap is 92 2021-02-10T04:31:38.516 INFO:tasks.rados.rados.0.smithi085.stdout:965: seq_num 362 ranges {484134=708729,1883658=200476} 2021-02-10T04:31:38.516 INFO:tasks.rados.rados.0.smithi085.stdout:965: writing smithi08575397-24 from 484134 to 1192863 tid 1 2021-02-10T04:31:38.516 INFO:tasks.rados.rados.0.smithi085.stdout:965: writing smithi08575397-24 from 1883658 to 2084134 tid 2 2021-02-10T04:31:38.517 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:38.517 INFO:tasks.rados.rados.0.smithi085.stdout:962: finishing copy_from to smithi08575397-37 2021-02-10T04:31:38.517 INFO:tasks.rados.rados.0.smithi085.stdout:962: got expected ENOENT (src dne) 2021-02-10T04:31:38.517 INFO:tasks.rados.rados.0.smithi085.stdout:962: finishing copy_from racing read to smithi08575397-37 2021-02-10T04:31:38.517 INFO:tasks.rados.rados.0.smithi085.stdout:962: done (15 left) 2021-02-10T04:31:38.517 INFO:tasks.rados.rados.0.smithi085.stdout:966: snap_remove snap 89 2021-02-10T04:31:38.517 INFO:tasks.rados.rados.0.smithi085.stdout:966: done (15 left) 2021-02-10T04:31:38.517 INFO:tasks.rados.rados.0.smithi085.stdout:967: snap_remove snap 90 2021-02-10T04:31:38.517 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: cluster 2021-02-10T04:31:31.988626+0000 osd.2 (osd.2) 174 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:38.517 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: cluster 2021-02-10T04:31:31.988636+0000 osd.2 (osd.2) 175 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:38.517 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.517 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: 2021-02-10T04:31:31.988650+0000 osd.2 (osd.2) 176 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:38.518 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: cluster 2021-02-10T04:31:31.988656+0000 2021-02-10T04:31:38.518 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: osd.2 (osd.2) 177 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:38.518 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: cluster 2021-02-10T04:31 2021-02-10T04:31:38.518 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: :31.988659+0000 osd.2 (osd.2) 178 : 2021-02-10T04:31:38.518 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:38.518 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:38.518 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: :31:31.988664+0000 osd.2 (osd.2) 2021-02-10T04:31:38.518 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: 179 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:38.518 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:38.518 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: 04:31:31.988679+0000 osd.2 (osd. 2021-02-10T04:31:38.518 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: 2) 180 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:38.518 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:38.518 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: 10T 2021-02-10T04:31:38.519 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: 04:31:31.988683 2021-02-10T04:31:38.519 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 181 : cluster [WRN] 2021-02-10T04:31:38.519 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:38.519 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: cluster 2021-02-10T04:31 2021-02-10T04:31:38.519 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: :32.223293+0000 osd.1 (osd.1) 178 2021-02-10T04:31:38.519 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:38.519 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:38.519 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: T04:31:32.223303+0000 osd.1 (osd 2021-02-10T04:31:38.519 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: .1) 179 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:38.519 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:38.519 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: - 2021-02-10T04:31:38.519 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: 02- 2021-02-10T04:31:38.519 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: 10T04:31:32.223312+0000 osd.1 2021-02-10T04:31:38.519 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: (osd.1) 180 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:38.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: 2021-02-10T04:31:38.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: 2021 2021-02-10T04:31:38.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: - 2021-02-10T04:31:38.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: 02 2021-02-10T04:31:38.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: -10T04:31:32.223318 2021-02-10T04:31:38.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 181 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:38.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:38.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: -02-10T04:31:32.223323+0000 2021-02-10T04:31:38.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: osd.1 (osd.1) 182 : cluster [WRN] 2021-02-10T04:31:38.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:38.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: cluster 2021-02-10T04:31 2021-02-10T04:31:38.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: :32.223331+0000 osd.1 (osd.1) 183 2021-02-10T04:31:38.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:38.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:38.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: :31:32.303740+0000 mon.a (mon. 2021-02-10T04:31:38.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: 0) 1013 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2021-02-10T04:31:38.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: 2021-02-10T04:31:38.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: 2021-02-10T04:31: 2021-02-10T04:31:38.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: 32.431618+0000 osd.4 (osd.4) 203 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:38.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: 2021-02-10T04:31:32.431624 2021-02-10T04:31:38.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 204 : cluster 2021-02-10T04:31:38.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:38.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:38.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: T04:31:32.431639+0000 osd.4 (osd. 2021-02-10T04:31:38.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: 4) 205 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:38.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:38.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: 02-10T04:31:32.431647+0000 2021-02-10T04:31:38.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: osd.4 (osd.4) 206 : cluster [WRN] 2021-02-10T04:31:38.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:38.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: cluster 2021-02-10T04:31:33. 2021-02-10T04:31:38.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: 074918+0000 mon.a (mon.0) 1014 : cluster 2021-02-10T04:31:38.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:33 smithi085 conmon[35961]: [WRN] Health check update: 18 slow ops, oldest one blocked for 137 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:31:38.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: cluster 2021-02-10T04:31: 2021-02-10T04:31:38.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: 33.002211+0000 osd.2 (osd.2) 182 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:38.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: cluster 2021-02-10T04:31: 2021-02-10T04:31:38.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: 33.002219+0000 osd.2 (osd.2) 183 : cluster 2021-02-10T04:31:38.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:38.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:38.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: 31:33.002225+0000 osd.2 (osd.2) 2021-02-10T04:31:38.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: 184 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:38.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:38.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: 04:31:33.002231+0000 osd.2 (osd.2 2021-02-10T04:31:38.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: ) 185 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:38.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:38.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: :31:33.002237+0000 osd.2 (osd.2) 186 2021-02-10T04:31:38.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:38.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: cluster 2021-02-10T04:31 2021-02-10T04:31:38.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: :33.002244+0000 osd.2 (osd.2) 187 : 2021-02-10T04:31:38.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:38.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: cluster 2021-02-10T04:31:33 2021-02-10T04:31:38.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: .002249+0000 osd.2 (osd.2) 188 : cluster 2021-02-10T04:31:38.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:38.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: cluster 2021-02-10T04:31:33.002266+0000 osd.2 (osd 2021-02-10T04:31:38.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: .2) 189 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:38.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: 2021-02-10T04:31:33.221219+0000 osd.1 (osd.1) 184 2021-02-10T04:31:38.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:38.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: cluster 2021-02-10T04:31:33 2021-02-10T04:31:38.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: .221225+0000 osd.1 (osd.1) 185 : cluster [WRN] 2021-02-10T04:31:38.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:38.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: 2021-02-10T04:31:33.221229+0000 osd.1 (osd.1 2021-02-10T04:31:38.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: ) 186 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:38.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:38.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: T04:31:33.221232+0000 osd.1 ( 2021-02-10T04:31:38.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: osd.1) 187 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:38.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:38.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: 04:31:33.221235+0000 osd.1 (osd. 2021-02-10T04:31:38.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: 1) 188 : cluster [WRN] 2021-02-10T04:31:38.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:38.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: cluster 2021-02-10T04:31:33 2021-02-10T04:31:38.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: .221240+0000 osd.1 (osd.1) 189 : cluster 2021-02-10T04:31:38.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:38.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: cluster 2021-02-10T04:31:33 2021-02-10T04:31:38.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: .319260+0000 mon.a (mon.0) 1015 : cluster 2021-02-10T04:31:38.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: [DBG] osdmap e276: 8 total, 8 up, 8 in 2021-02-10T04:31:38.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: cluster 2021-02-10T04:31 2021-02-10T04:31:38.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: :33.382481+0000 osd.4 (osd.4) 207 : 2021-02-10T04:31:38.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:38.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: cluster 2021-02-10T04:31: 2021-02-10T04:31:38.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: 33.382487+0000 osd.4 (osd.4 2021-02-10T04:31:38.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: ) 208 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:38.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: 2021-02-10T04:31:38.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:38.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: T04:31:33.382491 2021-02-10T04:31:38.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 209 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:38.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: 2021-02-10T04:31:38.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:38.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: 10T04:31:33. 2021-02-10T04:31:38.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: 382494+0000 osd.4 (osd.4) 2021-02-10T04:31:38.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: 210 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:38.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: 2021-02-10T04:31:38.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: 2021 2021-02-10T04:31:38.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: - 2021-02-10T04:31:38.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: 02 2021-02-10T04:31:38.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: - 2021-02-10T04:31:38.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: 10 2021-02-10T04:31:38.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: T 2021-02-10T04:31:38.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: 04 2021-02-10T04:31:38.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: : 2021-02-10T04:31:38.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: 31 2021-02-10T04:31:38.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: : 2021-02-10T04:31:38.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: 34 2021-02-10T04:31:38.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: . 2021-02-10T04:31:38.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: 036251 2021-02-10T04:31:38.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: +0000 2021-02-10T04:31:38.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: 2021-02-10T04:31:38.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: mgr.y (mgr.14140) 445 : cluster 2021-02-10T04:31:38.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:34 smithi085 conmon[35961]: [DBG] pgmap v564: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 154 MiB data, 521 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 780 KiB/s wr, 6 op/s 2021-02-10T04:31:38.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: 2021-02-10T04:31:38.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: 2021 2021-02-10T04:31:38.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: -02-10T04:31:33.954238+0000 2021-02-10T04:31:38.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: osd.2 (osd.2) 190 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:38.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:38.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: 02-10T04:31:33.954245+0000 osd.2 (osd. 2021-02-10T04:31:38.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: 2) 191 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:38.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: 2021-02-10T04:31:38.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: 2021 2021-02-10T04:31:38.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: - 2021-02-10T04:31:38.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: 02 2021-02-10T04:31:38.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: - 2021-02-10T04:31:38.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: 10 2021-02-10T04:31:38.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: T 2021-02-10T04:31:38.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: 04:31:33.954249+0000 2021-02-10T04:31:38.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: osd.2 (osd.2) 192 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:38.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: cluster 2021-02-10T04:31:33.954252 2021-02-10T04:31:38.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 193 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:38.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: cluster 2021-02-10T04:31:33.954258 2021-02-10T04:31:38.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 194 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:38.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: 2021-02-10T04:31:38.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:38.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: -02-10T04:31:33.954261+0000 osd.2 ( 2021-02-10T04:31:38.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: osd.2) 195 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:38.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: cluster 2021-02-10T04:31:33. 2021-02-10T04:31:38.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: 954264+0000 osd.2 (osd.2) 196 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:38.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: cluster 2021-02-10T04:31:33. 2021-02-10T04:31:38.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: 954267+0000 osd.2 (osd.2) 197 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:38.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: 2021-02-10T04:31:38.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:38.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: -02-10T04:31:34.251037+0000 osd.1 (osd.1) 2021-02-10T04:31:38.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: 190 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:38.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:38.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: 04:31:34.251046+0000 osd.1 (osd. 2021-02-10T04:31:38.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: 1) 191 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:38.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:38.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: 31:34.251058+0000 osd.1 (osd.1) 192 2021-02-10T04:31:38.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:38.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: 2021-02-10T04:31:34.251068 2021-02-10T04:31:38.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 193 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:38.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:38.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: 02-10T04:31:34.251076+0000 osd.1 2021-02-10T04:31:38.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: (osd.1) 194 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:38.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:38.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: 02-10T04:31:34.251082+0000 osd.1 (osd.1 2021-02-10T04:31:38.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: ) 195 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:38.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:38.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: 31:34.324099+0000 mon.a (mon.0) 1016 : cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2021-02-10T04:31:38.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:38.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: :31:34.386301+0000 osd.4 (osd.4) 211 2021-02-10T04:31:38.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:38.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:38.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: 04:31:34.386309+0000 osd.4 (osd. 2021-02-10T04:31:38.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: 4) 212 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:38.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:38.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: -02-10T04:31:34.386315+0000 2021-02-10T04:31:38.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: osd.4 (osd.4) 213 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:38.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:38.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: -02-10T04:31:34.386321+0000 2021-02-10T04:31:38.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:35 smithi085 conmon[35961]: osd.4 (osd.4) 214 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:38.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:31:38.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: 04:31:34.970725+0000 osd.2 (osd.2) 198 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:38.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:38.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: -10T04:31:34.970734+0000 osd.2 (osd.2) 2021-02-10T04:31:38.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: 199 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:38.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:38.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: T04:31:34. 2021-02-10T04:31:38.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: 970740+0000 osd.2 (osd.2) 200 : cluster [WRN] 2021-02-10T04:31:38.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:38.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:38.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: :31:34.970746+0000 osd.2 (osd.2 2021-02-10T04:31:38.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: ) 201 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:38.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:38.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: 10T04:31:34.970753+0000 osd.2 ( 2021-02-10T04:31:38.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: osd.2) 202 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:38.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:38.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: -02-10T04:31:34.970758+0000 2021-02-10T04:31:38.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: osd.2 (osd.2) 203 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:38.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: 2021-02-10T04:31:38.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: 2021-02-10T04:31:34.970770+0000 osd.2 (osd 2021-02-10T04:31:38.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: .2) 204 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:38.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:38.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: -10T04:31:34.970793+0000 osd.2 2021-02-10T04:31:38.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: (osd.2) 205 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:38.536 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: 2021-02-10T04:31:38.536 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:38.536 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: :31:35.204506+0000 osd.1 (osd. 2021-02-10T04:31:38.536 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: 1) 196 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:38.536 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:38.536 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: 02-10T04:31:35.204512+0000 osd.1 ( 2021-02-10T04:31:38.536 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: osd.1) 197 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:38.536 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:38.536 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: 02-10T04:31:35.204517+0000 2021-02-10T04:31:38.536 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: osd.1 (osd.1) 198 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:38.536 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:38.536 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: -02-10T04:31:35.204520+0000 2021-02-10T04:31:38.536 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: osd.1 (osd.1) 199 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:38.537 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: 2021-02-10T04:31:38.537 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.537 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: 2021-02-10T04:31:35.204524+0000 osd.1 2021-02-10T04:31:38.537 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: (osd.1) 200 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:38.537 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:38.537 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: 02-10T04:31:35.204528+0000 osd.1 2021-02-10T04:31:38.537 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: (osd.1) 201 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:38.537 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.537 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: 2021-02-10T04:31:35.324488+0000 2021-02-10T04:31:38.537 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: mon.a (mon.0) 1017 : cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2021-02-10T04:31:38.537 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: 2021-02-10T04:31:38.537 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: cluster 2021-02-10T04:31:35 2021-02-10T04:31:38.537 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: .363160+0000 osd.4 (osd.4) 215 : cluster 2021-02-10T04:31:38.538 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:38.538 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: cluster 2021-02-10T04:31: 2021-02-10T04:31:38.538 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: 35.363171+0000 osd.4 (osd.4) 216 : cluster 2021-02-10T04:31:38.538 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:38.538 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: cluster 2021-02-10T04:31: 2021-02-10T04:31:38.538 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: 35.363181+0000 osd.4 (osd.4) 217 : cluster 2021-02-10T04:31:38.538 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:38.538 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:38.538 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: 31:35.363186+0000 osd.4 (osd.4) 218 : 2021-02-10T04:31:38.538 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:38.538 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:38.538 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: 31:36.020100+0000 osd.2 (osd.2) 206 2021-02-10T04:31:38.538 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:38.539 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: cluster 2021-02-10T04:31: 2021-02-10T04:31:38.539 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: 36.020120+0000 osd.2 (osd.2) 207 : cluster 2021-02-10T04:31:38.539 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:38.539 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: cluster 2021-02-10T04:31:36.020129 2021-02-10T04:31:38.539 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 208 : cluster [WRN] 2021-02-10T04:31:38.539 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:38.539 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.539 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: 2021-02-10T04:31:36.020135+0000 osd.2 (osd. 2021-02-10T04:31:38.539 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: 2) 209 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:38.539 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:38.539 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: 31:36.020143+0000 osd.2 (osd.2) 210 2021-02-10T04:31:38.539 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:38.539 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: cluster 2021-02-10T04:31 2021-02-10T04:31:38.540 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: :36.020149+0000 osd.2 (osd.2) 211 : 2021-02-10T04:31:38.540 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:38.540 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: cluster 2021-02-10T04:31:36 2021-02-10T04:31:38.540 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: .020155+0000 osd.2 (osd.2) 212 : cluster 2021-02-10T04:31:38.540 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:38.540 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: cluster 2021-02-10T04:31:36. 2021-02-10T04:31:38.540 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: 020162+0000 osd.2 (osd.2) 213 : cluster [WRN] 2021-02-10T04:31:38.540 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:38.540 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.540 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: 2021-02-10T04:31:36.036802+0000 mgr.y (mgr. 2021-02-10T04:31:38.540 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:36 smithi085 conmon[35961]: 14140) 446 : cluster [DBG] pgmap v567: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 3 active+clean+snaptrim, 163 active+clean; 145 MiB data, 520 MiB used, 715 GiB / 715 GiB avail; 19 MiB/s rd, 4.6 MiB/s wr, 26 op/s 2021-02-10T04:31:38.540 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: cluster 2021-02-10T04:31:36.214301 2021-02-10T04:31:38.540 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 202 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:38.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: cluster 2021-02-10T04:31:36.214319+0000 osd.1 (osd 2021-02-10T04:31:38.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: .1) 203 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:38.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:38.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: -10T04:31:36.214346+0000 osd.1 (osd.1 2021-02-10T04:31:38.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: ) 204 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:38.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: 2021-02-10T04:31:38.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: 2021 2021-02-10T04:31:38.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: -02-10T04:31:36. 2021-02-10T04:31:38.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: 214352+0000 osd.1 (osd.1) 205 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:38.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:38.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: -02-10T04:31:36.214358 2021-02-10T04:31:38.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 206 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:38.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:38.542 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: 02-10T04:31:36.214364+0000 2021-02-10T04:31:38.542 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: osd.1 (osd.1) 207 : cluster [WRN] 2021-02-10T04:31:38.542 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:38.542 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.542 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:31:38.542 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: T04:31:36.343193+0000 mon.a (mon.0) 1018 : cluster [DBG] 2021-02-10T04:31:38.542 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: osdmap e279: 8 total, 8 up, 8 in 2021-02-10T04:31:38.542 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.542 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: 2021-02-10T04:31:36.352614+0000 osd.4 (osd.4) 219 : cluster 2021-02-10T04:31:38.542 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:38.542 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.542 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: 2021-02-10T04:31:36.352623+0000 osd.4 (osd.4 2021-02-10T04:31:38.542 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: ) 220 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:38.543 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.543 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: 2021-02-10T04:31:36.352631+0000 osd.4 (osd.4) 2021-02-10T04:31:38.543 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: 221 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:38.543 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.543 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: 2021-02-10T04:31:36.352637+0000 osd.4 (osd. 2021-02-10T04:31:38.543 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: 4) 222 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:38.543 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.543 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: 2021-02-10T04:31:36.993293+0000 2021-02-10T04:31:38.543 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: osd.2 (osd.2) 214 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:38.543 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: 2021-02-10T04:31:38.543 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: cluster 2021-02-10T04:31: 2021-02-10T04:31:38.543 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: 36.993302+0000 osd.2 (osd.2) 215 : cluster 2021-02-10T04:31:38.543 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:38.544 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:38.544 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: 10T04:31:36.993308+0000 osd.2 (osd. 2021-02-10T04:31:38.544 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: 2) 216 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:38.544 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: cluster 2021-02-10T04:31:36.993314+0000 2021-02-10T04:31:38.544 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: osd.2 (osd.2) 217 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:38.544 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: cluster 2021-02-10T04:31:36. 2021-02-10T04:31:38.544 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: 993335+0000 osd.2 (osd.2) 218 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:38.544 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: cluster 2021-02-10T04:31 2021-02-10T04:31:38.544 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: :36.993340+0000 osd.2 (osd.2) 219 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:38.544 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: 2021-02-10T04:31:38.544 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:38.544 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: 31:36.993345+0000 osd.2 (osd.2) 220 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:38.544 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:38.544 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: :31:36.993350+0000 osd.2 (osd.2 2021-02-10T04:31:38.545 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:37 smithi085 conmon[35961]: ) 221 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:38.545 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: cluster 2021-02-10T04:31:37.167085+0000 2021-02-10T04:31:38.545 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: osd.1 (osd.1) 208 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:38.545 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:38.545 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 04:31:37.167092+0000 osd.1 (osd.1) 209 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:38.545 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:38.545 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 31:37.167102+0000 osd.1 (osd.1) 2021-02-10T04:31:38.545 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 210 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:38.545 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.643 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[37263]: debug 2021-02-10T04:31:38.363+0000 7f1c6c9ed700 -1 osd.5 pg_epoch: 281 pg[7.0( v 278'228 (0'0,278'228] local-lis/les=93/94 n=2 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [5,1,3] r=0 lpr=93 crt=278'228 lcod 278'226 mlcod 278'226 active+clean+snaptrim trimq=[b3~1] ps=42] removing snap head 2021-02-10T04:31:38.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[33677]: debug 2021-02-10T04:31:38.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[33677]: 2021-02-10T04:31:38.375+0000 7fbec19a1700 -1 osd.4 281 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:31:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: cluster 2021-02-10T04:31:37.167085+0000 osd.1 ( 2021-02-10T04:31:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: osd.1) 208 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: cluster 2021-02-10T04:31:37.167092+0000 osd.1 (osd.1) 209 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: cluster 2021-02-10T04:31: 2021-02-10T04:31:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: 37.167102+0000 osd.1 (osd.1) 210 : cluster [WRN] 2021-02-10T04:31:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: 2021-02-10T04:31:37.167107+0000 osd.1 2021-02-10T04:31:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: (osd.1) 211 : cluster [WRN] 2021-02-10T04:31:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: -10T04:31:37.167114 2021-02-10T04:31:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 212 : 2021-02-10T04:31:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:31:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: 31:37.167120+0000 osd.1 (osd.1) 213 : 2021-02-10T04:31:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:31:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: 04:31:37.353794+0000 mon.a (mon.0 2021-02-10T04:31:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: ) 1019 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2021-02-10T04:31:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: 02-10T04:31:37.367331+0000 osd.4 2021-02-10T04:31:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: (osd.4) 223 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: 2021-02-10T04:31:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: 2021-02-10T04:31:37.367338+0000 osd.4 (osd.4) 224 2021-02-10T04:31:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: -10T04:31:37.367351+0000 osd.4 (osd.4) 225 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: cluster 2021-02-10T04:31:37.367358 2021-02-10T04:31:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 226 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: cluster 2021-02-10T04:31:37.684195+0000 osd.7 2021-02-10T04:31:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: (osd.7) 44 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:31:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: cluster 2021-02-10T04:31:37.684403+0000 osd.7 (osd.7 2021-02-10T04:31:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: ) 45 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:31:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: audit 2021-02-10T04:31:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: 2021-02-10T04:31:37.855048+0000 mgr.y (mgr. 2021-02-10T04:31:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: 14140) 447 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:31:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38.037227+0000 2021-02-10T04:31:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: mgr.y (mgr.14140) 448 : cluster [DBG] pgmap v570: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 3 active+clean+snaptrim, 163 active+clean; 145 MiB data, 524 MiB used, 715 GiB / 715 GiB avail; 19 MiB/s rd, 4.6 MiB/s wr, 24 op/s 2021-02-10T04:31:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: 2021-02-10T04:31:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: 10T04:31:38.042397+0000 osd.2 (osd.2) 222 2021-02-10T04:31:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38 2021-02-10T04:31:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: .042402+0000 osd.2 (osd.2) 223 : cluster 2021-02-10T04:31:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: 2021-02-10T04:31:38.042407+0000 osd.2 ( 2021-02-10T04:31:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: osd.2) 224 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:31:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: 04:31:38.042411+0000 osd.2 (osd. 2021-02-10T04:31:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: 2) 225 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:31:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: :31:38.042414+0000 osd.2 (osd.2 2021-02-10T04:31:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: ) 226 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:31:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: 31:38.042418+0000 osd.2 (osd.2) 2021-02-10T04:31:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: 227 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: cluster 2021-02-10T04:31: 2021-02-10T04:31:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: 38.042421+0000 osd.2 (osd.2) 228 : 2021-02-10T04:31:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38.042426+0000 osd.2 (osd.2 2021-02-10T04:31:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: ) 229 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: audit 2021-02-10T04:31 2021-02-10T04:31:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: :38.063845+0000 mon.a (mon.0) 1020 2021-02-10T04:31:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:31:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: audit 2021-02-10T04:31: 2021-02-10T04:31:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: 38.064231+0000 mon.a (mon.0) 1021 : 2021-02-10T04:31:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:31:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38 2021-02-10T04:31:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:38 smithi164 conmon[29288]: .076514+0000 mon.a (mon.0) 1022 : cluster [WRN] Health check update: 18 slow ops, oldest one blocked for 142 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:31:38.691 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[52836]: debug 2021-02-10T04:31:38.691 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[52836]: 2021-02-10T04:31:38.375+0000 7fba70fb0700 -1 osd.3 pg_epoch: 281 pg[7.a( v 279'267 (0'0,279'267] local-lis/les=93/94 n=6 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [3,2,5] r=0 lpr=93 crt=279'266 lcod 279'266 mlcod 279'266 active+clean+snaptrim trimq=[b3~1] ps=42] removing snap head 2021-02-10T04:31:38.691 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[52836]: 2021-02-10T04:31:38.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 2021-02-10T04:31:38.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 2021 2021-02-10T04:31:38.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: - 2021-02-10T04:31:38.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 02 2021-02-10T04:31:38.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: - 2021-02-10T04:31:38.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 10 2021-02-10T04:31:38.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: T 2021-02-10T04:31:38.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 04 2021-02-10T04:31:38.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: : 2021-02-10T04:31:38.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 31 2021-02-10T04:31:38.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: : 2021-02-10T04:31:38.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 37 2021-02-10T04:31:38.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: . 2021-02-10T04:31:38.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 167107 2021-02-10T04:31:38.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: +0000 2021-02-10T04:31:38.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 2021-02-10T04:31:38.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: osd.1 2021-02-10T04:31:38.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: ( 2021-02-10T04:31:38.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: osd 2021-02-10T04:31:38.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: . 2021-02-10T04:31:38.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 1 2021-02-10T04:31:38.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: ) 2021-02-10T04:31:38.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 211 2021-02-10T04:31:38.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: : 2021-02-10T04:31:38.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 2021-02-10T04:31:38.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: [WRN] 2021-02-10T04:31:38.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 2021-02-10T04:31:38.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:38.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 2021-02-10T04:31:38.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 2021-02-10T04:31:38.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 2021 2021-02-10T04:31:38.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: - 2021-02-10T04:31:38.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 02 2021-02-10T04:31:38.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: - 2021-02-10T04:31:38.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 10 2021-02-10T04:31:38.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: T 2021-02-10T04:31:38.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 04 2021-02-10T04:31:38.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: : 2021-02-10T04:31:38.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 31 2021-02-10T04:31:38.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: : 2021-02-10T04:31:38.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 37 2021-02-10T04:31:38.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: . 2021-02-10T04:31:38.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 167114 2021-02-10T04:31:38.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: +0000 2021-02-10T04:31:38.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 2021-02-10T04:31:38.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: osd.1 2021-02-10T04:31:38.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: ( 2021-02-10T04:31:38.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: osd 2021-02-10T04:31:38.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: . 2021-02-10T04:31:38.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 1 2021-02-10T04:31:38.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: ) 2021-02-10T04:31:38.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 212 2021-02-10T04:31:38.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: : 2021-02-10T04:31:38.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 2021-02-10T04:31:38.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: [WRN] 2021-02-10T04:31:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 2021-02-10T04:31:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 2021-02-10T04:31:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 2021-02-10T04:31:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 2021 2021-02-10T04:31:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: - 2021-02-10T04:31:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 02 2021-02-10T04:31:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: - 2021-02-10T04:31:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 10 2021-02-10T04:31:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: T 2021-02-10T04:31:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 04 2021-02-10T04:31:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: : 2021-02-10T04:31:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 31 2021-02-10T04:31:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: : 2021-02-10T04:31:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 37 2021-02-10T04:31:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: . 2021-02-10T04:31:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 167120 2021-02-10T04:31:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: +0000 2021-02-10T04:31:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 2021-02-10T04:31:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: osd.1 (osd.1) 2021-02-10T04:31:38.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 213 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:38.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:38.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: -10T04:31:37.353794+0000 mon.a (mon.0) 2021-02-10T04:31:38.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 1019 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2021-02-10T04:31:38.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:38.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: -10T04:31:37.367331+0000 osd.4 (osd 2021-02-10T04:31:38.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: .4) 223 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:38.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: cluster 2021-02-10T04:31 2021-02-10T04:31:38.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: :37.367338+0000 osd.4 (osd.4) 224 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:38.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:38.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: :31:37.367351+0000 osd.4 (osd.4) 225 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:38.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:38.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: -02-10T04:31:37.367358+0000 osd.4 2021-02-10T04:31:38.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: (osd.4) 226 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:38.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:38.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: :31:37.684195+0000 osd.7 (osd.7) 44 : cluster 2021-02-10T04:31:38.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: [DBG] purged_snaps scrub starts 2021-02-10T04:31:38.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 2021-02-10T04:31:37.684403+0000 osd.7 (osd.7) 45 2021-02-10T04:31:38.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:31:38.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: audit 2021-02-10T04:31:38.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 2021-02-10T04:31:37.855048+0000 mgr.y (mgr.14140) 2021-02-10T04:31:38.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 447 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:31:38.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.037227+0000 2021-02-10T04:31:38.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: mgr.y (mgr.14140) 448 : cluster [DBG] pgmap v570: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 3 active+clean+snaptrim, 163 active+clean; 145 MiB data, 524 MiB used, 715 GiB / 715 GiB avail; 19 MiB/s rd, 4.6 MiB/s wr, 24 op/s 2021-02-10T04:31:38.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38 2021-02-10T04:31:38.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: .042397+0000 osd.2 (osd.2) 222 : cluster [WRN] 2021-02-10T04:31:38.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:38.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 2021-02-10T04:31:38.042402+0000 osd.2 (osd. 2021-02-10T04:31:38.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 2) 223 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:38.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: cluster 2021-02-10T04:31: 2021-02-10T04:31:38.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 38.042407+0000 osd.2 (osd.2) 224 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:38.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 2021-02-10T04:31:38.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:38.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: :31:38.042411+0000 osd.2 (osd.2) 225 : cluster 2021-02-10T04:31:38.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:38.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 2021-02-10T04:31:38.042414 2021-02-10T04:31:38.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 226 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:38.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:38.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: T04:31:38.042418+0000 osd.2 (osd.2) 227 2021-02-10T04:31:38.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:38.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 2021-02-10T04:31:38.042421 2021-02-10T04:31:38.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: +0000 osd.2 ( 2021-02-10T04:31:38.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: osd.2) 228 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:38.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:38.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 10T04:31:38.042426+0000 osd.2 (osd.2) 2021-02-10T04:31:38.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 229 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:38.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: audit 2021-02-10T04:31:38.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 2021-02-10T04:31:38.063845+0000 mon.a (mon.0) 2021-02-10T04:31:38.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 1020 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:31:38.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: audit 2021-02-10T04:31:38. 2021-02-10T04:31:38.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: 064231+0000 mon.a (mon.0) 1021 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:31:38.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:38.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: T04:31:38.076514+0000 mon.a (mon.0) 1022 2021-02-10T04:31:38.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:38 smithi085 conmon[35961]: : cluster [WRN] Health check update: 18 slow ops, oldest one blocked for 142 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:31:39.304 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:31:39.337 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:39.367 INFO:tasks.rados.rados.0.smithi085.stdout:964: finishing write tid 1 to smithi08575397-2 2021-02-10T04:31:39.367 INFO:tasks.rados.rados.0.smithi085.stdout:964: finishing write tid 2 to smithi08575397-2 2021-02-10T04:31:39.367 INFO:tasks.rados.rados.0.smithi085.stdout:964: finishing write tid 3 to smithi08575397-2 2021-02-10T04:31:39.367 INFO:tasks.rados.rados.0.smithi085.stdout:964: finishing write tid 4 to smithi08575397-2 2021-02-10T04:31:39.367 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 287 (ObjNum 361 snap 92 seq_num 361) dirty exists 2021-02-10T04:31:39.367 INFO:tasks.rados.rados.0.smithi085.stdout:964: left oid 2 (ObjNum 361 snap 92 seq_num 361) 2021-02-10T04:31:39.367 INFO:tasks.rados.rados.0.smithi085.stdout:965: finishing write tid 1 to smithi08575397-24 2021-02-10T04:31:39.367 INFO:tasks.rados.rados.0.smithi085.stdout:965: finishing write tid 2 to smithi08575397-24 2021-02-10T04:31:39.367 INFO:tasks.rados.rados.0.smithi085.stdout:965: finishing write tid 3 to smithi08575397-24 2021-02-10T04:31:39.367 INFO:tasks.rados.rados.0.smithi085.stdout:965: finishing write tid 4 to smithi08575397-24 2021-02-10T04:31:39.368 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 24 v 137 (ObjNum 362 snap 92 seq_num 362) dirty exists 2021-02-10T04:31:39.368 INFO:tasks.rados.rados.0.smithi085.stdout:965: left oid 24 (ObjNum 362 snap 92 seq_num 362) 2021-02-10T04:31:39.368 INFO:tasks.rados.rados.0.smithi085.stdout:963: done (15 left) 2021-02-10T04:31:39.368 INFO:tasks.rados.rados.0.smithi085.stdout:964: done (14 left) 2021-02-10T04:31:39.368 INFO:tasks.rados.rados.0.smithi085.stdout:965: done (13 left) 2021-02-10T04:31:39.368 INFO:tasks.rados.rados.0.smithi085.stdout:967: done (12 left) 2021-02-10T04:31:39.368 INFO:tasks.rados.rados.0.smithi085.stdout:968: snap_create 2021-02-10T04:31:39.369 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[49106]: debug 2021-02-10T04:31:39.007+0000 7f77a3e94700 -1 osd.2 281 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:31:39.374 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: cluster 2021-02-10T04:31:39.375 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[45443]: debug 2021-02-10T04:31:39.206+0000 7f7140d73700 -1 osd.1 281 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:31:39.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: cluster 2021-02-10T04:31:38.202289+0000 2021-02-10T04:31:39.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: osd.1 (osd.1) 214 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:39.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:39.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: -10T04:31:38.202298+0000 osd.1 (osd.1) 215 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:39.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: 2021-02-10T04:31:39.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:39.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: 10T04:31:38.202304+0000 osd.1 ( 2021-02-10T04:31:39.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: osd.1) 216 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:39.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: cluster 2021-02-10T04:31:39.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: 2021-02-10T04:31 2021-02-10T04:31:39.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: :38.202310+0000 osd.1 (osd.1) 2021-02-10T04:31:39.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: 217 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:39.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: cluster 2021-02-10T04:31:39.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:31:39.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: 10T04:31:38.202316+0000 osd.1 2021-02-10T04:31:39.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: (osd.1) 218 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:39.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: 2021-02-10T04:31:39.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:39.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: 10T04:31:38.202322+0000 osd.1 ( 2021-02-10T04:31:39.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: osd.1) 219 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:39.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: cluster 2021-02-10T04:31:39.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: 2021-02-10T04:31:38.363684+0000 2021-02-10T04:31:39.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: mon.a (mon.0) 1023 : cluster [DBG] 2021-02-10T04:31:39.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: osdmap e281: 8 total, 8 up, 8 in 2021-02-10T04:31:39.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:39.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: 02-10T04:31:38.376215+0000 2021-02-10T04:31:39.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: osd.4 (osd.4) 227 : cluster [WRN] 2021-02-10T04:31:39.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:39.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:39.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: -10T04:31:38.376224+0000 osd.4 2021-02-10T04:31:39.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: (osd.4) 228 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:39.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: 2021-02-10T04:31:39.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:39.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: 10T04:31:38.376232+0000 osd.4 ( 2021-02-10T04:31:39.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: osd.4) 229 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:39.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: cluster 2021-02-10T04:31:39.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: 2021-02-10T04:31 2021-02-10T04:31:39.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: :38.376238+0000 osd.4 (osd.4) 230 2021-02-10T04:31:39.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:39.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: cluster 2021-02-10T04:31:39.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: 2021-02-10T04:31:39.008029 2021-02-10T04:31:39.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 230 : cluster [WRN] 2021-02-10T04:31:39.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:39.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:39.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: 02-10T04:31:39.008038+0000 2021-02-10T04:31:39.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: osd.2 (osd.2) 231 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:39.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: 2021-02-10T04:31:39.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:39.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: 31:39.008045+0000 osd.2 (osd.2 2021-02-10T04:31:39.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: ) 232 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:39.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:39.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: 10T04:31:39.008051+0000 osd.2 ( 2021-02-10T04:31:39.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: osd.2) 233 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:39.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:39.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: -02-10T04:31:39.008056+0000 2021-02-10T04:31:39.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: osd.2 (osd.2) 234 : cluster [WRN] 2021-02-10T04:31:39.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:39.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: cluster 2021-02-10T04:31:39. 2021-02-10T04:31:39.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: 008062+0000 osd.2 (osd.2) 235 : cluster 2021-02-10T04:31:39.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:39.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: cluster 2021-02-10T04:31:39.008068+0000 osd.2 (osd.2) 236 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:39.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: cluster 2021-02-10T04:31:39.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:31:39.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: 04:31:39.008081+0000 2021-02-10T04:31:39.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: osd.2 (osd.2) 237 2021-02-10T04:31:39.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:39.473 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:31:39.476 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[40892]: debug 2021-02-10T04:31:39.218+0000 7f30364c7700 -1 osd.6 pg_epoch: 281 pg[7.d( v 281'196 (0'0,281'196] local-lis/les=93/94 n=6 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [6,1,0] r=0 lpr=93 luod=279'194 crt=281'196 lcod 278'192 mlcod 278'192 active+clean+snaptrim trimq=[b3~1] ps=42] removing snap head 2021-02-10T04:31:39.476 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[44544]: debug 2021-02-10T04:31:39.431+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:39.476 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[33677]: debug 2021-02-10T04:31:39.415+0000 7fbec19a1700 -1 osd.4 282 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:31:39.476 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38.202289+0000 osd.1 (osd.1) 2021-02-10T04:31:39.476 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: 214 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:39.476 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38.202298+0000 osd.1 (osd.1) 215 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:39.477 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38.202304+0000 osd.1 (osd.1) 216 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:39.477 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: 2021-02-10T04:31:39.477 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: cluster 2021-02-10T04:31:38 2021-02-10T04:31:39.477 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: .202310+0000 osd.1 (osd.1) 217 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:39.477 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:39.477 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: -02-10T04:31:38.202316+0000 osd.1 (osd.1) 218 : cluster [WRN] 2021-02-10T04:31:39.477 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:39.477 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: cluster 2021-02-10T04:31 2021-02-10T04:31:39.477 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: :38.202322+0000 osd.1 (osd.1) 219 2021-02-10T04:31:39.477 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:39.477 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:31:39.477 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: 04:31:38.363684+0000 mon.a (mon 2021-02-10T04:31:39.477 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: .0) 1023 : cluster [DBG] osdmap e281: 8 total, 8 up, 8 in 2021-02-10T04:31:39.477 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:39.478 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: -02-10T04:31:38.376215+0000 2021-02-10T04:31:39.478 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: osd.4 (osd.4) 227 : cluster [WRN] 2021-02-10T04:31:39.478 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:39.478 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: cluster 2021-02-10T04:31:39.478 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: 2021-02-10T04:31:38.376224+0000 2021-02-10T04:31:39.478 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: osd.4 (osd.4) 228 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:39.478 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: 2021-02-10T04:31:39.478 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:39.478 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: 02-10T04:31:38.376232+0000 osd.4 (osd. 2021-02-10T04:31:39.478 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: 4) 229 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:39.478 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:31:39.478 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: :31:38.376238+0000 osd.4 (osd.4 2021-02-10T04:31:39.479 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: ) 230 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:39.479 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: cluster 2021-02-10T04:31:39.479 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: 2021-02-10T04:31:39.008029+0000 osd.2 (osd.2) 2021-02-10T04:31:39.479 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: 230 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:39.480 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: cluster 2021-02-10T04:31:39.480 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: 2021-02-10T04:31:39.008038+0000 osd.2 (osd.2) 231 : cluster [WRN] 2021-02-10T04:31:39.480 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:39.480 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:39.480 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: 02-10T04:31:39.008045+0000 osd.2 (osd.2 2021-02-10T04:31:39.480 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: ) 232 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:39.480 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:31:39.480 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: 31:39.008051+0000 osd.2 (osd.2) 2021-02-10T04:31:39.480 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: 233 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:39.480 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: cluster 2021-02-10T04:31:39.008056 2021-02-10T04:31:39.480 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 234 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:39.480 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: cluster 2021-02-10T04:31:39 2021-02-10T04:31:39.480 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: .008062+0000 osd.2 (osd.2) 235 : cluster 2021-02-10T04:31:39.481 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:39.481 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:31:39.481 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: :31:39.008068+0000 osd.2 (osd 2021-02-10T04:31:39.481 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: .2) 236 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:39.481 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:39.481 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: -02-10T04:31:39.008081 2021-02-10T04:31:39.481 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 237 2021-02-10T04:31:39.481 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:39 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:39.583 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:31:39.611 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:39.626 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 2021-02-10T04:31:39.626 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 2021 2021-02-10T04:31:39.626 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: - 2021-02-10T04:31:39.626 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 02 2021-02-10T04:31:39.627 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: - 2021-02-10T04:31:39.627 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 10 2021-02-10T04:31:39.627 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: T 2021-02-10T04:31:39.627 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 04 2021-02-10T04:31:39.627 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: : 2021-02-10T04:31:39.627 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 31 2021-02-10T04:31:39.627 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: : 2021-02-10T04:31:39.627 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 38 2021-02-10T04:31:39.627 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: . 2021-02-10T04:31:39.627 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 202289 2021-02-10T04:31:39.627 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: +0000 2021-02-10T04:31:39.627 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 2021-02-10T04:31:39.627 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: osd.1 2021-02-10T04:31:39.627 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: ( 2021-02-10T04:31:39.628 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: osd 2021-02-10T04:31:39.628 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: . 2021-02-10T04:31:39.628 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 1 2021-02-10T04:31:39.628 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: ) 2021-02-10T04:31:39.628 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 214 2021-02-10T04:31:39.628 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: : 2021-02-10T04:31:39.628 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: cluster 2021-02-10T04:31:39.628 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 2021-02-10T04:31:39.628 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: [WRN] 2021-02-10T04:31:39.628 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 2021-02-10T04:31:39.628 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:39.628 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 2021-02-10T04:31:39.628 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: cluster 2021-02-10T04:31:39.629 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 2021-02-10T04:31:39.629 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 2021 2021-02-10T04:31:39.629 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: - 2021-02-10T04:31:39.629 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 02 2021-02-10T04:31:39.629 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: - 2021-02-10T04:31:39.629 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 10 2021-02-10T04:31:39.629 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: T 2021-02-10T04:31:39.629 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 04 2021-02-10T04:31:39.629 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: : 2021-02-10T04:31:39.629 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 31 2021-02-10T04:31:39.629 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: : 2021-02-10T04:31:39.629 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 38 2021-02-10T04:31:39.629 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: . 2021-02-10T04:31:39.630 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 202298 2021-02-10T04:31:39.630 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: +0000 2021-02-10T04:31:39.630 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 2021-02-10T04:31:39.630 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: osd.1 2021-02-10T04:31:39.630 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: ( 2021-02-10T04:31:39.630 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: osd 2021-02-10T04:31:39.630 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: . 2021-02-10T04:31:39.630 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 1 2021-02-10T04:31:39.630 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: ) 2021-02-10T04:31:39.630 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 215 2021-02-10T04:31:39.630 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: : 2021-02-10T04:31:39.630 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: cluster 2021-02-10T04:31:39.630 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 2021-02-10T04:31:39.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: [WRN] 2021-02-10T04:31:39.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 2021-02-10T04:31:39.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:39.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 2021-02-10T04:31:39.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: cluster 2021-02-10T04:31:39.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 2021-02-10T04:31:39.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 2021 2021-02-10T04:31:39.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: - 2021-02-10T04:31:39.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 02 2021-02-10T04:31:39.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: - 2021-02-10T04:31:39.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 10 2021-02-10T04:31:39.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: T 2021-02-10T04:31:39.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 04 2021-02-10T04:31:39.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: :31:38.202304+0000 osd.1 (osd 2021-02-10T04:31:39.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: .1) 216 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:39.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: cluster 2021-02-10T04:31:38.202310+0000 osd.1 ( 2021-02-10T04:31:39.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: osd.1) 217 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:39.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: cluster 2021-02-10T04:31:39.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 2021-02-10T04:31:39.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:31:39.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 31:38.202316+0000 osd.1 (osd.1) 218 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:39.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:39.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: -02-10T04:31:38.202322+0000 2021-02-10T04:31:39.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: osd.1 (osd.1) 219 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:39.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 2021-02-10T04:31:39.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:39.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: T04:31:38.363684+0000 2021-02-10T04:31:39.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: mon.a (mon.0) 1023 : cluster [DBG] 2021-02-10T04:31:39.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: osdmap e281: 8 total, 8 up, 8 in 2021-02-10T04:31:39.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:31:39.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 04:31:38.376215+0000 2021-02-10T04:31:39.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: osd.4 (osd.4) 227 : cluster 2021-02-10T04:31:39.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:39.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:31:39.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: :31:38.376224+0000 osd.4 (osd.4) 2021-02-10T04:31:39.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 228 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:39.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:31:39.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 04:31:38.376232+0000 osd.4 (osd.4 2021-02-10T04:31:39.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: ) 229 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:39.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:31:39.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: :31:38.376238+0000 osd.4 (osd.4) 2021-02-10T04:31:39.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 230 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:39.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:31:39.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 04:31:39.008029+0000 osd.2 (osd.2 2021-02-10T04:31:39.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: ) 230 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:39.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:39.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: T04:31:39.008038+0000 osd.2 (osd. 2021-02-10T04:31:39.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 2) 231 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:39.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:39.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: T04:31:39.008045+0000 osd.2 (osd.2 2021-02-10T04:31:39.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: ) 232 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:39.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:39.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: T04:31:39.008051+0000 osd.2 ( 2021-02-10T04:31:39.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: osd.2) 233 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:39.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:39.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: -02-10T04:31:39.008056+0000 2021-02-10T04:31:39.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: osd.2 (osd.2) 234 : cluster [WRN] 2021-02-10T04:31:39.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:39.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:39.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 10T04:31:39.008062 2021-02-10T04:31:39.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 235 : cluster 2021-02-10T04:31:39.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:39.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: cluster 2021-02-10T04:31 2021-02-10T04:31:39.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: :39.008068+0000 osd.2 (osd.2 2021-02-10T04:31:39.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: ) 236 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:39.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:39.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: 10T04:31:39.008081+0000 osd.2 ( 2021-02-10T04:31:39.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[29319]: osd.2) 237 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:39.656 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2021-02-10T04:31:39.656 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:31:39.849 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:31:39.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:39 smithi085 conmon[49106]: debug 2021-02-10T04:31:39.818+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:39.954 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:31:39.992 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:40.034 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:31:40.219 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[45443]: debug 2021-02-10T04:31:40.219 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[45443]: 2021-02-10T04:31:40.199+0000 7f7140d73700 -1 osd.1 282 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:31:40.219 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[49106]: debug 2021-02-10T04:31:40.219 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[49106]: 2021-02-10T04:31:40.000+0000 7f77a3e94700 -1 osd.2 282 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:31:40.219 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[49106]: 2021-02-10T04:31:40.296 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:31:40.296 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 20, 2021-02-10T04:31:40.296 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 600, 2021-02-10T04:31:40.296 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [ 2021-02-10T04:31:40.296 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.296 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:277 7.6 7:649ff6e5:::smithi08575397-4:head [copy-from ver 3] snapc 8=[8] ondisk+write+known_if_redirected e105)", 2021-02-10T04:31:40.297 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:28:41.366906+0000", 2021-02-10T04:31:40.297 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 178.926365137, 2021-02-10T04:31:40.297 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.14149879600000001, 2021-02-10T04:31:40.297 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:31:40.297 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:31:40.297 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:31:40.297 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:31:40.297 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:31:40.297 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 277 2021-02-10T04:31:40.297 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.297 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:31:40.297 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.297 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:31:40.298 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.366906+0000", 2021-02-10T04:31:40.298 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:31:40.298 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.298 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.298 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:31:40.298 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.366906+0000", 2021-02-10T04:31:40.298 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.1103e-05 2021-02-10T04:31:40.298 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.298 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.298 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:31:40.298 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.366977+0000", 2021-02-10T04:31:40.298 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.2088000000000002e-05 2021-02-10T04:31:40.299 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.299 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.299 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:31:40.299 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.367009+0000", 2021-02-10T04:31:40.299 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.1164000000000001e-05 2021-02-10T04:31:40.299 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.299 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.299 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:31:40.299 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.367050+0000", 2021-02-10T04:31:40.299 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.2283999999999998e-05 2021-02-10T04:31:40.299 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.299 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.299 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:31:40.300 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.367082+0000", 2021-02-10T04:31:40.300 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.2577000000000005e-05 2021-02-10T04:31:40.300 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.300 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.300 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:31:40.300 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.367175+0000", 2021-02-10T04:31:40.300 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00017677 2021-02-10T04:31:40.300 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.300 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.300 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:31:40.300 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.367352+0000", 2021-02-10T04:31:40.300 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.13094492499999999 2021-02-10T04:31:40.301 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.301 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.301 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for subops from 3,7", 2021-02-10T04:31:40.301 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.498297+0000", 2021-02-10T04:31:40.301 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0051660320000000001 2021-02-10T04:31:40.301 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.301 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.301 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op_commit", 2021-02-10T04:31:40.301 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.503463+0000", 2021-02-10T04:31:40.301 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0045000109999999999 2021-02-10T04:31:40.301 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.301 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.302 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:31:40.302 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.507963+0000", 2021-02-10T04:31:40.302 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000284574 2021-02-10T04:31:40.302 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.302 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.302 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:31:40.302 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.508247+0000", 2021-02-10T04:31:40.302 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.9922e-05 2021-02-10T04:31:40.302 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.302 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.302 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:31:40.302 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.508307+0000", 2021-02-10T04:31:40.303 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.7345999999999999e-05 2021-02-10T04:31:40.303 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.303 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.303 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:31:40.303 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.508404+0000", 2021-02-10T04:31:40.303 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.14149879600000001 2021-02-10T04:31:40.303 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.303 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:31:40.303 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.303 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.303 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.303 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:278 7.6 7:649ff6e5:::smithi08575397-4:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e105)", 2021-02-10T04:31:40.304 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:28:41.367153+0000", 2021-02-10T04:31:40.304 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 178.926118282, 2021-02-10T04:31:40.304 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.14146404900000001, 2021-02-10T04:31:40.304 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:31:40.304 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:31:40.304 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:31:40.304 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:31:40.304 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:31:40.304 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 278 2021-02-10T04:31:40.304 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.304 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:31:40.304 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.304 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:31:40.305 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.367153+0000", 2021-02-10T04:31:40.305 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:31:40.305 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.305 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.305 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:31:40.305 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.367153+0000", 2021-02-10T04:31:40.305 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.3748999999999998e-05 2021-02-10T04:31:40.305 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.305 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.305 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:31:40.305 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.367186+0000", 2021-02-10T04:31:40.306 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.2439999999999993e-06 2021-02-10T04:31:40.306 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.306 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.306 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:31:40.306 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.367195+0000", 2021-02-10T04:31:40.306 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.4347000000000002e-05 2021-02-10T04:31:40.306 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.306 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.306 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:31:40.306 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.367229+0000", 2021-02-10T04:31:40.306 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.707e-06 2021-02-10T04:31:40.306 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.307 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.307 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:31:40.307 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.367234+0000", 2021-02-10T04:31:40.307 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00027270899999999999 2021-02-10T04:31:40.307 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.307 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.307 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:31:40.307 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.367506+0000", 2021-02-10T04:31:40.307 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.4251000000000001e-05 2021-02-10T04:31:40.307 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.307 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.308 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for blocked object", 2021-02-10T04:31:40.308 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.367541+0000", 2021-02-10T04:31:40.308 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.13481500399999999 2021-02-10T04:31:40.308 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.308 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.308 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:31:40.308 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.502356+0000", 2021-02-10T04:31:40.308 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.4981000000000005e-05 2021-02-10T04:31:40.308 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.308 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.308 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for rw locks", 2021-02-10T04:31:40.308 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.502441+0000", 2021-02-10T04:31:40.309 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0059967199999999997 2021-02-10T04:31:40.309 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.309 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.309 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:31:40.309 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.508437+0000", 2021-02-10T04:31:40.309 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.8860000000000001e-05 2021-02-10T04:31:40.309 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.309 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.310 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:31:40.310 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.508516+0000", 2021-02-10T04:31:40.310 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00010047699999999999 2021-02-10T04:31:40.310 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.310 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.310 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:31:40.310 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.508617+0000", 2021-02-10T04:31:40.310 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.14146404900000001 2021-02-10T04:31:40.310 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.311 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:31:40.311 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.311 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.311 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.311 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(client.14817.0:292 7.d e105/93 7:b4cb5e6c:::smithi08575397-50:head v 105'15, mlcod=98'12)", 2021-02-10T04:31:40.311 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:28:41.477208+0000", 2021-02-10T04:31:40.311 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 178.81606293900001, 2021-02-10T04:31:40.311 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.026614787000000001, 2021-02-10T04:31:40.311 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:31:40.311 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:31:40.311 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:31:40.312 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.312 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:31:40.312 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.477208+0000", 2021-02-10T04:31:40.312 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:31:40.312 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.312 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.312 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:31:40.312 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.477208+0000", 2021-02-10T04:31:40.312 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.021062668 2021-02-10T04:31:40.312 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.312 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.312 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:31:40.313 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.498271+0000", 2021-02-10T04:31:40.313 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.8300000000000008e-06 2021-02-10T04:31:40.313 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.313 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.313 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:31:40.313 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.498280+0000", 2021-02-10T04:31:40.313 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.3501e-05 2021-02-10T04:31:40.313 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.313 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.313 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:31:40.313 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.498294+0000", 2021-02-10T04:31:40.313 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.4369999999999996e-06 2021-02-10T04:31:40.314 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.314 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.314 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:31:40.314 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.498303+0000", 2021-02-10T04:31:40.314 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.533e-05 2021-02-10T04:31:40.314 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.314 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.314 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:31:40.314 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.498369+0000", 2021-02-10T04:31:40.314 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.3921999999999997e-05 2021-02-10T04:31:40.315 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.315 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.315 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:31:40.315 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.498413+0000", 2021-02-10T04:31:40.315 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0053480180000000004 2021-02-10T04:31:40.315 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.315 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.315 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:31:40.316 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.503761+0000", 2021-02-10T04:31:40.316 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.2081000000000004e-05 2021-02-10T04:31:40.316 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.316 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.316 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:31:40.316 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:41.503823+0000", 2021-02-10T04:31:40.316 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.026614787000000001 2021-02-10T04:31:40.316 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.316 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:31:40.316 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.317 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.317 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.317 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:546 7.f 7:f473c801:::smithi08575397-49:head [write 2781526~586628] snapc 1e=[1e,1d,1c,1b,19,17,15] ondisk+write+known_if_redirected e127)", 2021-02-10T04:31:40.317 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:29:03.062337+0000", 2021-02-10T04:31:40.317 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 157.23093379400001, 2021-02-10T04:31:40.317 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.021398545000000001, 2021-02-10T04:31:40.317 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:31:40.317 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:31:40.317 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:31:40.317 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:31:40.317 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:31:40.317 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 546 2021-02-10T04:31:40.317 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.318 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:31:40.318 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.318 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:31:40.318 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:03.062337+0000", 2021-02-10T04:31:40.318 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:31:40.318 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.318 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.318 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:31:40.318 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:03.062337+0000", 2021-02-10T04:31:40.318 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00091129399999999999 2021-02-10T04:31:40.318 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.318 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.319 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:31:40.319 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:03.063248+0000", 2021-02-10T04:31:40.319 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1654999999999999e-05 2021-02-10T04:31:40.319 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.319 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.319 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:31:40.319 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:03.063260+0000", 2021-02-10T04:31:40.319 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.4589e-05 2021-02-10T04:31:40.319 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.319 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.319 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:31:40.319 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:03.063275+0000", 2021-02-10T04:31:40.320 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.8969999999999997e-06 2021-02-10T04:31:40.320 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.320 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.320 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:31:40.320 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:03.063283+0000", 2021-02-10T04:31:40.320 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0014358489999999999 2021-02-10T04:31:40.320 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.320 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.320 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:31:40.320 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:03.064719+0000", 2021-02-10T04:31:40.320 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000103786 2021-02-10T04:31:40.320 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.320 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.321 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:31:40.321 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:03.064823+0000", 2021-02-10T04:31:40.321 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00014043000000000001 2021-02-10T04:31:40.321 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.321 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.321 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for subops from 3,7", 2021-02-10T04:31:40.321 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:03.064964+0000", 2021-02-10T04:31:40.321 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0013915100000000001 2021-02-10T04:31:40.321 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.321 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.321 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op_commit", 2021-02-10T04:31:40.321 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:03.066355+0000", 2021-02-10T04:31:40.321 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0041402219999999998 2021-02-10T04:31:40.322 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.322 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.322 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:31:40.322 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:03.070495+0000", 2021-02-10T04:31:40.322 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.013096241 2021-02-10T04:31:40.322 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.322 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.322 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:31:40.322 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:03.083592+0000", 2021-02-10T04:31:40.322 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.1966999999999995e-05 2021-02-10T04:31:40.322 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.322 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.323 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:31:40.323 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:03.083683+0000", 2021-02-10T04:31:40.323 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.2104999999999997e-05 2021-02-10T04:31:40.323 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.323 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.323 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:31:40.323 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:03.083736+0000", 2021-02-10T04:31:40.323 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.021398545000000001 2021-02-10T04:31:40.323 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.323 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:31:40.323 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.323 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.323 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.324 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:616 7.f 7:fc15f6da:::smithi08575397-35:head [copy-from ver 29] snapc 20=[20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e131)", 2021-02-10T04:31:40.324 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:29:07.163805+0000", 2021-02-10T04:31:40.324 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 153.12946537799999, 2021-02-10T04:31:40.324 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.84101313899999997, 2021-02-10T04:31:40.324 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:31:40.324 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:31:40.324 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:31:40.324 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:31:40.324 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:31:40.324 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 616 2021-02-10T04:31:40.324 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.324 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:31:40.324 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.325 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:31:40.325 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:07.163805+0000", 2021-02-10T04:31:40.325 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:31:40.325 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.325 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.325 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:31:40.325 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:07.163805+0000", 2021-02-10T04:31:40.325 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.1975000000000003e-05 2021-02-10T04:31:40.325 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.325 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.325 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:31:40.325 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:07.163857+0000", 2021-02-10T04:31:40.325 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.1710000000000002e-06 2021-02-10T04:31:40.325 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.326 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.326 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:31:40.326 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:07.163866+0000", 2021-02-10T04:31:40.326 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.3302000000000001e-05 2021-02-10T04:31:40.326 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.326 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.326 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:31:40.326 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:07.163879+0000", 2021-02-10T04:31:40.326 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.3380000000000005e-06 2021-02-10T04:31:40.326 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.326 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.326 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:31:40.326 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:07.163887+0000", 2021-02-10T04:31:40.327 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0272000000000003e-05 2021-02-10T04:31:40.327 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.327 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.327 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:31:40.327 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:07.163938+0000", 2021-02-10T04:31:40.327 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.5087999999999995e-05 2021-02-10T04:31:40.327 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.327 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.327 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:31:40.327 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:07.164033+0000", 2021-02-10T04:31:40.327 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.75282117000000004 2021-02-10T04:31:40.327 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.327 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.327 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for subops from 3,7", 2021-02-10T04:31:40.328 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:07.916854+0000", 2021-02-10T04:31:40.328 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0047053709999999999 2021-02-10T04:31:40.328 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.328 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.328 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op_commit", 2021-02-10T04:31:40.328 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:07.921559+0000", 2021-02-10T04:31:40.328 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0026131879999999998 2021-02-10T04:31:40.328 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.329 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.329 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:31:40.329 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:07.924172+0000", 2021-02-10T04:31:40.329 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.080534976999999994 2021-02-10T04:31:40.329 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.329 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.329 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:31:40.329 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.004707+0000", 2021-02-10T04:31:40.329 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.6618000000000005e-05 2021-02-10T04:31:40.329 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.329 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.330 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:31:40.330 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.004774+0000", 2021-02-10T04:31:40.330 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4669000000000002e-05 2021-02-10T04:31:40.330 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.330 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.330 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:31:40.330 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.004819+0000", 2021-02-10T04:31:40.330 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.84101313899999997 2021-02-10T04:31:40.330 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.330 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:31:40.330 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.330 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.330 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.331 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:617 7.f 7:fc15f6da:::smithi08575397-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e131)", 2021-02-10T04:31:40.331 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:29:07.163933+0000", 2021-02-10T04:31:40.331 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 153.12933830099999, 2021-02-10T04:31:40.331 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.84108339399999998, 2021-02-10T04:31:40.331 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:31:40.331 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:31:40.331 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:31:40.331 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:31:40.331 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:31:40.332 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 617 2021-02-10T04:31:40.332 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.332 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:31:40.332 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.332 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:31:40.332 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:07.163933+0000", 2021-02-10T04:31:40.332 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:31:40.332 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.332 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.332 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:31:40.332 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:07.163933+0000", 2021-02-10T04:31:40.332 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.0059000000000001e-05 2021-02-10T04:31:40.332 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.332 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.333 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:31:40.333 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:07.163963+0000", 2021-02-10T04:31:40.333 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.4589999999999997e-06 2021-02-10T04:31:40.333 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.333 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.333 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:31:40.333 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:07.163968+0000", 2021-02-10T04:31:40.333 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0455000000000001e-05 2021-02-10T04:31:40.333 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.333 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.333 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:31:40.333 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:07.163978+0000", 2021-02-10T04:31:40.333 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.8459999999999999e-06 2021-02-10T04:31:40.333 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.334 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.334 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:31:40.334 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:07.163983+0000", 2021-02-10T04:31:40.334 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00019883600000000001 2021-02-10T04:31:40.334 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.334 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.334 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:31:40.334 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:07.164182+0000", 2021-02-10T04:31:40.334 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.3602000000000001e-05 2021-02-10T04:31:40.334 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.334 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.334 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for blocked object", 2021-02-10T04:31:40.334 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:07.164216+0000", 2021-02-10T04:31:40.335 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.75633279799999997 2021-02-10T04:31:40.335 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.335 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.335 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:31:40.335 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:07.920549+0000", 2021-02-10T04:31:40.335 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.9328999999999999e-05 2021-02-10T04:31:40.335 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.335 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.335 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for rw locks", 2021-02-10T04:31:40.335 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:07.920628+0000", 2021-02-10T04:31:40.335 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.084230861000000004 2021-02-10T04:31:40.335 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.335 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.335 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:31:40.336 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.004859+0000", 2021-02-10T04:31:40.336 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.2747e-05 2021-02-10T04:31:40.336 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.336 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.336 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:31:40.336 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.004932+0000", 2021-02-10T04:31:40.336 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.4401999999999996e-05 2021-02-10T04:31:40.336 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.336 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.336 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:31:40.336 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.005016+0000", 2021-02-10T04:31:40.336 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.84108339399999998 2021-02-10T04:31:40.336 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.336 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:31:40.337 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.337 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.337 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.337 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(client.14817.0:630 7.7 e132/93 7:ef952473:::smithi08575397-16:head v 132'82, mlcod=125'78)", 2021-02-10T04:31:40.337 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:29:08.059811+0000", 2021-02-10T04:31:40.337 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 152.23345961699999, 2021-02-10T04:31:40.337 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.032541694000000003, 2021-02-10T04:31:40.337 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:31:40.337 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:31:40.337 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:31:40.337 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.337 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:31:40.337 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.059811+0000", 2021-02-10T04:31:40.337 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:31:40.338 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.338 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.338 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:31:40.338 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.059811+0000", 2021-02-10T04:31:40.338 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.7342000000000002e-05 2021-02-10T04:31:40.338 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.338 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.338 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:31:40.338 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.059859+0000", 2021-02-10T04:31:40.338 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.2940000000000006e-06 2021-02-10T04:31:40.338 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.338 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.339 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:31:40.339 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.059867+0000", 2021-02-10T04:31:40.339 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.3436e-05 2021-02-10T04:31:40.339 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.339 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.339 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:31:40.339 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.059880+0000", 2021-02-10T04:31:40.339 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.0890000000000002e-06 2021-02-10T04:31:40.339 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.339 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.339 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:31:40.339 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.059887+0000", 2021-02-10T04:31:40.339 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.8213999999999993e-05 2021-02-10T04:31:40.339 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.340 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.340 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:31:40.340 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.059966+0000", 2021-02-10T04:31:40.340 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.1895e-05 2021-02-10T04:31:40.340 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.340 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.340 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:31:40.340 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.059987+0000", 2021-02-10T04:31:40.340 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.032305489999999999 2021-02-10T04:31:40.340 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.340 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.340 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:31:40.340 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.092293+0000", 2021-02-10T04:31:40.340 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.9933999999999999e-05 2021-02-10T04:31:40.341 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.341 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.341 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:31:40.341 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.092353+0000", 2021-02-10T04:31:40.341 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.032541694000000003 2021-02-10T04:31:40.341 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.341 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:31:40.341 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.341 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.341 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.341 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(client.14817.0:631 7.7 e132/93 7:ef952473:::smithi08575397-16:head v 132'83, mlcod=125'78)", 2021-02-10T04:31:40.341 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:29:08.059921+0000", 2021-02-10T04:31:40.341 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 152.233350073, 2021-02-10T04:31:40.341 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.032470143, 2021-02-10T04:31:40.342 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:31:40.342 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:31:40.342 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:31:40.342 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.342 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:31:40.342 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.059921+0000", 2021-02-10T04:31:40.342 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:31:40.342 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.342 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.342 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:31:40.342 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.059921+0000", 2021-02-10T04:31:40.342 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.2830000000000002e-05 2021-02-10T04:31:40.342 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.342 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.343 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:31:40.343 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.059964+0000", 2021-02-10T04:31:40.343 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.4139999999999999e-06 2021-02-10T04:31:40.343 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.343 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.343 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:31:40.343 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.059970+0000", 2021-02-10T04:31:40.343 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.4059999999999994e-06 2021-02-10T04:31:40.343 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.343 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.343 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:31:40.343 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.059979+0000", 2021-02-10T04:31:40.343 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4530000000000004e-06 2021-02-10T04:31:40.344 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.344 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.344 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:31:40.344 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.059984+0000", 2021-02-10T04:31:40.344 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000336263 2021-02-10T04:31:40.344 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.344 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.344 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:31:40.344 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.060320+0000", 2021-02-10T04:31:40.344 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.5143999999999998e-05 2021-02-10T04:31:40.344 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.344 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.344 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:31:40.344 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.060345+0000", 2021-02-10T04:31:40.345 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.032011351 2021-02-10T04:31:40.345 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.345 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.345 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:31:40.345 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.092357+0000", 2021-02-10T04:31:40.345 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.4282000000000001e-05 2021-02-10T04:31:40.345 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.345 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.345 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:31:40.345 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.092391+0000", 2021-02-10T04:31:40.345 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.032470143 2021-02-10T04:31:40.345 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.345 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:31:40.346 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.346 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.346 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.346 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(osd.6.0:206 7.7 e134/93 7:ef952473:::smithi08575397-16:23 v 134'85, mlcod=132'82)", 2021-02-10T04:31:40.346 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:29:10.044639+0000", 2021-02-10T04:31:40.346 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 150.24863178199999, 2021-02-10T04:31:40.346 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.068442896000000003, 2021-02-10T04:31:40.346 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:31:40.346 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:31:40.346 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:31:40.346 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.347 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:31:40.347 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.044639+0000", 2021-02-10T04:31:40.347 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:31:40.347 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.347 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.347 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:31:40.347 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.044639+0000", 2021-02-10T04:31:40.347 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.3741e-05 2021-02-10T04:31:40.347 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.347 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.347 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:31:40.347 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.044683+0000", 2021-02-10T04:31:40.348 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.1029999999999996e-06 2021-02-10T04:31:40.348 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.348 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.348 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:31:40.348 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.044690+0000", 2021-02-10T04:31:40.348 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0684e-05 2021-02-10T04:31:40.348 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.348 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.348 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:31:40.348 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.044701+0000", 2021-02-10T04:31:40.348 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.3239999999999999e-06 2021-02-10T04:31:40.348 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.348 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.348 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:31:40.349 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.044707+0000", 2021-02-10T04:31:40.349 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.064525964000000005 2021-02-10T04:31:40.349 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.349 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.349 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:31:40.349 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.109233+0000", 2021-02-10T04:31:40.349 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.4558000000000001e-05 2021-02-10T04:31:40.349 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.349 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.349 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:31:40.349 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.109257+0000", 2021-02-10T04:31:40.349 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00110636 2021-02-10T04:31:40.349 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.349 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.350 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:31:40.350 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.110364+0000", 2021-02-10T04:31:40.350 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0027181620000000001 2021-02-10T04:31:40.350 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.350 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.350 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:31:40.350 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.113082+0000", 2021-02-10T04:31:40.350 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.068442896000000003 2021-02-10T04:31:40.350 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.350 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:31:40.350 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.350 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.350 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.351 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "pg_update_log_missing(7.7 epoch 154/93 rep_tid 272 entries 154'138 (0'0) error 7:e7b4a0f7:::smithi08575397-11:head by client.14817.0:884 0.000000 -2 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 151'135)", 2021-02-10T04:31:40.351 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:29:30.146795+0000", 2021-02-10T04:31:40.351 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 130.14647608600001, 2021-02-10T04:31:40.351 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.026579041000000001, 2021-02-10T04:31:40.351 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:31:40.351 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:31:40.351 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:31:40.351 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.351 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:31:40.351 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:30.146795+0000", 2021-02-10T04:31:40.351 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:31:40.351 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.351 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.351 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:31:40.352 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:30.146795+0000", 2021-02-10T04:31:40.352 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4215999999999998e-05 2021-02-10T04:31:40.352 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.352 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.352 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:31:40.352 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:30.146839+0000", 2021-02-10T04:31:40.352 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1589e-05 2021-02-10T04:31:40.352 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.352 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.352 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:31:40.352 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:30.146851+0000", 2021-02-10T04:31:40.352 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.8570999999999999e-05 2021-02-10T04:31:40.352 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.352 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.353 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:31:40.353 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:30.146869+0000", 2021-02-10T04:31:40.353 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.0539999999999995e-06 2021-02-10T04:31:40.353 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.353 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.353 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:31:40.353 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:30.146877+0000", 2021-02-10T04:31:40.353 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00032153200000000002 2021-02-10T04:31:40.353 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.353 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.353 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:31:40.353 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:30.147199+0000", 2021-02-10T04:31:40.353 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.026175079 2021-02-10T04:31:40.353 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.354 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.354 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:31:40.354 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:30.173374+0000", 2021-02-10T04:31:40.354 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.026579041000000001 2021-02-10T04:31:40.354 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.354 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:31:40.354 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.354 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.354 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.354 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:1275 7.9 7:93db45b2:::smithi08575397-44:head [copy-from ver 0] snapc 56=[56] ondisk+write+known_if_redirected e183)", 2021-02-10T04:31:40.354 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:29:59.561992+0000", 2021-02-10T04:31:40.354 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 100.73127875900001, 2021-02-10T04:31:40.354 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.173932164, 2021-02-10T04:31:40.355 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:31:40.355 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:31:40.355 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:31:40.355 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:31:40.355 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:31:40.355 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 1275 2021-02-10T04:31:40.355 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.355 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:31:40.355 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.356 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:31:40.356 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:59.561992+0000", 2021-02-10T04:31:40.356 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:31:40.356 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.356 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.356 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:31:40.356 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:59.561992+0000", 2021-02-10T04:31:40.356 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.3980000000000001e-05 2021-02-10T04:31:40.357 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.357 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.357 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:31:40.357 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:59.562016+0000", 2021-02-10T04:31:40.357 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.8520000000000003e-06 2021-02-10T04:31:40.357 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.357 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.357 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:31:40.357 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:59.562021+0000", 2021-02-10T04:31:40.357 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1015e-05 2021-02-10T04:31:40.357 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.357 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.357 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:31:40.357 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:59.562032+0000", 2021-02-10T04:31:40.358 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.2599999999999996e-06 2021-02-10T04:31:40.358 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.358 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.358 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:31:40.358 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:59.562037+0000", 2021-02-10T04:31:40.358 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.4132000000000002e-05 2021-02-10T04:31:40.358 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.358 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.358 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:31:40.358 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:59.562101+0000", 2021-02-10T04:31:40.358 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00010671 2021-02-10T04:31:40.358 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.358 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.359 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:31:40.359 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:59.562208+0000", 2021-02-10T04:31:40.359 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.17371621500000001 2021-02-10T04:31:40.359 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.359 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.359 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:31:40.359 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:59.735924+0000", 2021-02-10T04:31:40.359 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.173932164 2021-02-10T04:31:40.359 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.359 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:31:40.359 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.359 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.359 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.359 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:1276 7.9 7:93db45b2:::smithi08575397-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e183)", 2021-02-10T04:31:40.360 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:29:59.562084+0000", 2021-02-10T04:31:40.360 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 100.731186712, 2021-02-10T04:31:40.360 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.17414471100000001, 2021-02-10T04:31:40.360 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:31:40.360 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:31:40.360 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:31:40.360 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:31:40.360 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:31:40.360 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 1276 2021-02-10T04:31:40.360 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.360 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:31:40.360 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.360 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:31:40.361 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:59.562084+0000", 2021-02-10T04:31:40.361 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:31:40.361 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.361 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.361 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:31:40.361 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:59.562084+0000", 2021-02-10T04:31:40.361 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.3349999999999998e-05 2021-02-10T04:31:40.361 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.361 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.361 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:31:40.361 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:59.562107+0000", 2021-02-10T04:31:40.361 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.659e-06 2021-02-10T04:31:40.361 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.362 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.362 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:31:40.362 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:59.562111+0000", 2021-02-10T04:31:40.362 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.8759999999999999e-06 2021-02-10T04:31:40.362 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.362 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.362 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:31:40.362 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:59.562118+0000", 2021-02-10T04:31:40.362 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.467e-06 2021-02-10T04:31:40.362 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.363 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.363 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:31:40.363 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:59.562121+0000", 2021-02-10T04:31:40.363 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00025292700000000001 2021-02-10T04:31:40.363 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.363 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.363 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:31:40.363 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:59.562374+0000", 2021-02-10T04:31:40.363 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.6449999999999997e-05 2021-02-10T04:31:40.363 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.363 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.363 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for blocked object", 2021-02-10T04:31:40.363 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:59.562421+0000", 2021-02-10T04:31:40.363 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.17357523899999999 2021-02-10T04:31:40.364 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.364 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.364 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:31:40.364 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:59.735996+0000", 2021-02-10T04:31:40.364 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000118202 2021-02-10T04:31:40.364 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.364 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.364 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:31:40.364 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:59.736114+0000", 2021-02-10T04:31:40.364 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000114541 2021-02-10T04:31:40.364 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.364 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.364 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:31:40.365 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:59.736229+0000", 2021-02-10T04:31:40.365 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.17414471100000001 2021-02-10T04:31:40.365 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.365 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:31:40.365 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.365 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.365 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.365 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:1672 7.9 7:93db45b2:::smithi08575397-44:head [copy-from ver 217] snapc 6f=[6f] ondisk+write+known_if_redirected e212)", 2021-02-10T04:31:40.365 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:30:28.445842+0000", 2021-02-10T04:31:40.365 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 71.847428659000002, 2021-02-10T04:31:40.366 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.4280552909999999, 2021-02-10T04:31:40.366 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:31:40.366 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:31:40.366 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:31:40.366 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:31:40.366 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:31:40.366 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 1672 2021-02-10T04:31:40.366 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.366 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:31:40.366 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.367 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:31:40.367 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.445842+0000", 2021-02-10T04:31:40.367 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:31:40.367 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.367 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.367 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:31:40.367 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.445842+0000", 2021-02-10T04:31:40.367 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.6919999999999998e-05 2021-02-10T04:31:40.367 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.367 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.367 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:31:40.367 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.445889+0000", 2021-02-10T04:31:40.367 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.6750000000000002e-06 2021-02-10T04:31:40.368 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.368 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.368 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:31:40.368 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.445897+0000", 2021-02-10T04:31:40.368 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.3869000000000001e-05 2021-02-10T04:31:40.368 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.368 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.368 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:31:40.368 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.445911+0000", 2021-02-10T04:31:40.368 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.3600000000000001e-06 2021-02-10T04:31:40.368 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.368 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.368 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:31:40.368 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.445917+0000", 2021-02-10T04:31:40.369 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0759e-05 2021-02-10T04:31:40.369 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.369 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.369 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:31:40.369 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.445968+0000", 2021-02-10T04:31:40.369 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.8629999999999996e-05 2021-02-10T04:31:40.369 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.369 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.369 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:31:40.369 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.446066+0000", 2021-02-10T04:31:40.369 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.41660147 2021-02-10T04:31:40.369 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.369 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.369 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for subops from 1,4", 2021-02-10T04:31:40.370 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:29.862668+0000", 2021-02-10T04:31:40.370 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.010650328000000001 2021-02-10T04:31:40.370 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.370 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.370 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:31:40.370 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:29.873318+0000", 2021-02-10T04:31:40.370 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.4790000000000002e-06 2021-02-10T04:31:40.370 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.370 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.370 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op_commit", 2021-02-10T04:31:40.370 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:29.873328+0000", 2021-02-10T04:31:40.370 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000416797 2021-02-10T04:31:40.370 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.371 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.371 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:31:40.371 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:29.873744+0000", 2021-02-10T04:31:40.371 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.5724999999999997e-05 2021-02-10T04:31:40.371 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.371 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.371 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:31:40.371 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:29.873820+0000", 2021-02-10T04:31:40.371 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.7279000000000003e-05 2021-02-10T04:31:40.371 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.371 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.371 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:31:40.371 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:29.873897+0000", 2021-02-10T04:31:40.371 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.4280552909999999 2021-02-10T04:31:40.372 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.372 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:31:40.372 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.372 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.372 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.372 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:1673 7.9 7:93db45b2:::smithi08575397-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e212)", 2021-02-10T04:31:40.372 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:30:28.445959+0000", 2021-02-10T04:31:40.372 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 71.847311711000003, 2021-02-10T04:31:40.372 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.4282873279999999, 2021-02-10T04:31:40.372 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:31:40.372 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:31:40.372 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:31:40.372 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:31:40.372 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:31:40.373 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 1673 2021-02-10T04:31:40.373 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.373 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:31:40.373 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.373 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:31:40.373 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.445959+0000", 2021-02-10T04:31:40.373 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:31:40.373 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.373 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.373 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:31:40.373 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.445959+0000", 2021-02-10T04:31:40.373 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.0720999999999999e-05 2021-02-10T04:31:40.373 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.373 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.374 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:31:40.374 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.445990+0000", 2021-02-10T04:31:40.374 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.9699999999999996e-06 2021-02-10T04:31:40.374 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.374 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.374 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:31:40.374 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.445996+0000", 2021-02-10T04:31:40.374 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1564e-05 2021-02-10T04:31:40.374 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.374 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.374 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:31:40.374 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.446007+0000", 2021-02-10T04:31:40.374 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0560000000000004e-06 2021-02-10T04:31:40.375 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.375 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.375 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:31:40.375 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.446012+0000", 2021-02-10T04:31:40.375 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00019481500000000001 2021-02-10T04:31:40.375 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.375 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.375 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:31:40.375 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.446207+0000", 2021-02-10T04:31:40.375 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.2522999999999999e-05 2021-02-10T04:31:40.375 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.375 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.375 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for blocked object", 2021-02-10T04:31:40.375 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.446250+0000", 2021-02-10T04:31:40.376 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.421146324 2021-02-10T04:31:40.376 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.376 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.376 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:31:40.376 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:29.867396+0000", 2021-02-10T04:31:40.376 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00010089 2021-02-10T04:31:40.376 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.376 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.376 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for rw locks", 2021-02-10T04:31:40.376 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:29.867497+0000", 2021-02-10T04:31:40.376 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0064370470000000004 2021-02-10T04:31:40.376 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.376 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.377 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:31:40.377 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:29.873934+0000", 2021-02-10T04:31:40.377 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00020304700000000001 2021-02-10T04:31:40.377 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.377 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.377 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:31:40.377 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:29.874137+0000", 2021-02-10T04:31:40.377 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00010937099999999999 2021-02-10T04:31:40.377 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.377 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.377 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:31:40.377 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:29.874246+0000", 2021-02-10T04:31:40.378 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.4282873279999999 2021-02-10T04:31:40.378 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.378 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:31:40.378 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.378 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.378 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.378 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(client.14817.0:1750 7.7 e218/93 7:ef952473:::smithi08575397-16:head v 218'296, mlcod=217'293)", 2021-02-10T04:31:40.378 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:30:34.479965+0000", 2021-02-10T04:31:40.378 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 65.813306193000003, 2021-02-10T04:31:40.378 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.020609902999999999, 2021-02-10T04:31:40.378 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:31:40.379 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:31:40.379 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:31:40.379 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.379 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:31:40.379 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:34.479965+0000", 2021-02-10T04:31:40.379 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:31:40.379 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.379 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.379 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:31:40.379 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:34.479965+0000", 2021-02-10T04:31:40.379 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0077503720000000002 2021-02-10T04:31:40.379 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.379 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.379 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:31:40.380 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:34.487715+0000", 2021-02-10T04:31:40.380 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.042e-05 2021-02-10T04:31:40.380 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.380 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.380 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:31:40.380 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:34.487725+0000", 2021-02-10T04:31:40.380 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.4575e-05 2021-02-10T04:31:40.380 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.380 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.380 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:31:40.380 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:34.487740+0000", 2021-02-10T04:31:40.380 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.0450000000000001e-06 2021-02-10T04:31:40.380 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.380 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.381 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:31:40.381 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:34.487748+0000", 2021-02-10T04:31:40.381 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00013837499999999999 2021-02-10T04:31:40.381 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.381 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.381 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:31:40.381 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:34.487886+0000", 2021-02-10T04:31:40.381 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.7177000000000001e-05 2021-02-10T04:31:40.381 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.381 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.381 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:31:40.381 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:34.487914+0000", 2021-02-10T04:31:40.381 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.012596796 2021-02-10T04:31:40.382 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.382 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.382 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:31:40.382 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:34.500510+0000", 2021-02-10T04:31:40.382 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.4142999999999999e-05 2021-02-10T04:31:40.382 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.382 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.382 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:31:40.382 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:34.500575+0000", 2021-02-10T04:31:40.382 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.020609902999999999 2021-02-10T04:31:40.382 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.382 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:31:40.382 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.382 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.383 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.383 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:2467 7.f 7:fe141921:::smithi08575397-2:head [copy-from ver 79] snapc b0=[] ondisk+write+known_if_redirected e275)", 2021-02-10T04:31:40.383 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:31:32.313117+0000", 2021-02-10T04:31:40.383 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 7.9801537629999997, 2021-02-10T04:31:40.383 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.020798829000000001, 2021-02-10T04:31:40.383 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:31:40.383 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:31:40.383 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:31:40.383 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:31:40.383 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:31:40.384 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 2467 2021-02-10T04:31:40.384 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.384 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:31:40.384 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.384 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:31:40.384 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.313117+0000", 2021-02-10T04:31:40.384 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:31:40.384 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.384 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.384 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:31:40.384 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.313117+0000", 2021-02-10T04:31:40.384 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.0037000000000003e-05 2021-02-10T04:31:40.385 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.385 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.385 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:31:40.385 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.313157+0000", 2021-02-10T04:31:40.385 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.8499999999999996e-06 2021-02-10T04:31:40.385 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.385 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.385 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:31:40.385 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.313164+0000", 2021-02-10T04:31:40.385 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.5140000000000001e-06 2021-02-10T04:31:40.385 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.385 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.385 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:31:40.386 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.313172+0000", 2021-02-10T04:31:40.386 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.6069999999999996e-06 2021-02-10T04:31:40.386 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.386 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.386 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:31:40.386 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.313179+0000", 2021-02-10T04:31:40.386 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.7942000000000003e-05 2021-02-10T04:31:40.386 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.386 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.386 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:31:40.386 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.313227+0000", 2021-02-10T04:31:40.386 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.7007000000000001e-05 2021-02-10T04:31:40.387 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.387 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.387 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:31:40.387 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.313324+0000", 2021-02-10T04:31:40.387 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0084651110000000009 2021-02-10T04:31:40.387 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.387 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.387 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for subops from 3,7", 2021-02-10T04:31:40.387 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.321789+0000", 2021-02-10T04:31:40.387 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0085505270000000005 2021-02-10T04:31:40.387 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.387 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.388 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:31:40.388 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.330340+0000", 2021-02-10T04:31:40.388 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0025239440000000002 2021-02-10T04:31:40.388 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.388 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.388 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op_commit", 2021-02-10T04:31:40.388 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.332864+0000", 2021-02-10T04:31:40.388 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00089329700000000004 2021-02-10T04:31:40.388 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.388 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.388 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:31:40.388 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.333757+0000", 2021-02-10T04:31:40.388 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.9853999999999999e-05 2021-02-10T04:31:40.389 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.389 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.389 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:31:40.389 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.333827+0000", 2021-02-10T04:31:40.389 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.9139000000000002e-05 2021-02-10T04:31:40.389 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.389 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.389 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:31:40.389 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.333916+0000", 2021-02-10T04:31:40.389 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.020798829000000001 2021-02-10T04:31:40.389 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.390 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:31:40.390 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.390 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.390 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.390 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:2468 7.f 7:fe141921:::smithi08575397-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e275)", 2021-02-10T04:31:40.390 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:31:32.313219+0000", 2021-02-10T04:31:40.390 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 7.9800521150000003, 2021-02-10T04:31:40.390 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.020929199999999998, 2021-02-10T04:31:40.390 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:31:40.390 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:31:40.390 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:31:40.390 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:31:40.391 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:31:40.391 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 2468 2021-02-10T04:31:40.391 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.391 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:31:40.391 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.391 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:31:40.391 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.313219+0000", 2021-02-10T04:31:40.391 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:31:40.391 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.391 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.391 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:31:40.391 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.313219+0000", 2021-02-10T04:31:40.392 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00017952500000000001 2021-02-10T04:31:40.392 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.392 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.392 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:31:40.392 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.313398+0000", 2021-02-10T04:31:40.392 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.7059999999999998e-06 2021-02-10T04:31:40.392 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.392 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.392 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:31:40.392 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.313405+0000", 2021-02-10T04:31:40.392 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.3679000000000001e-05 2021-02-10T04:31:40.392 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.392 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.393 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:31:40.393 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.313419+0000", 2021-02-10T04:31:40.393 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.3099999999999997e-06 2021-02-10T04:31:40.393 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.393 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.393 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:31:40.393 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.313425+0000", 2021-02-10T04:31:40.393 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.8164000000000003e-05 2021-02-10T04:31:40.393 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.393 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.393 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:31:40.393 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.313463+0000", 2021-02-10T04:31:40.393 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.7929999999999997e-05 2021-02-10T04:31:40.394 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.394 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.394 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for blocked object", 2021-02-10T04:31:40.394 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.313511+0000", 2021-02-10T04:31:40.394 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.013642389 2021-02-10T04:31:40.394 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.394 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.394 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:31:40.394 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.327153+0000", 2021-02-10T04:31:40.394 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00013125800000000001 2021-02-10T04:31:40.394 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.394 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.394 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for rw locks", 2021-02-10T04:31:40.395 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.327285+0000", 2021-02-10T04:31:40.395 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0066665520000000001 2021-02-10T04:31:40.395 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.395 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.395 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:31:40.395 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.333951+0000", 2021-02-10T04:31:40.395 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.3277999999999997e-05 2021-02-10T04:31:40.395 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.395 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.395 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:31:40.395 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.334045+0000", 2021-02-10T04:31:40.395 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000103409 2021-02-10T04:31:40.395 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.395 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.396 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:31:40.396 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.334148+0000", 2021-02-10T04:31:40.396 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.020929199999999998 2021-02-10T04:31:40.396 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.396 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:31:40.396 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.396 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.396 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.396 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:2540 7.4 7:2dcefd69:::smithi08575397-28:head [write 3244434~688437] snapc b3=[b3] ondisk+write+known_if_redirected e278)", 2021-02-10T04:31:40.396 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:31:35.657391+0000", 2021-02-10T04:31:40.396 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 4.6358798119999998, 2021-02-10T04:31:40.396 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.019434732, 2021-02-10T04:31:40.396 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:31:40.396 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:31:40.397 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:31:40.397 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:31:40.397 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:31:40.397 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 2540 2021-02-10T04:31:40.397 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.397 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:31:40.397 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.397 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:31:40.397 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.657391+0000", 2021-02-10T04:31:40.397 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:31:40.397 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.397 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.397 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:31:40.397 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.657391+0000", 2021-02-10T04:31:40.398 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00016835900000000001 2021-02-10T04:31:40.398 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.398 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.398 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:31:40.398 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.657559+0000", 2021-02-10T04:31:40.398 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.2050000000000001e-06 2021-02-10T04:31:40.398 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.398 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.398 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:31:40.398 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.657565+0000", 2021-02-10T04:31:40.398 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.1282000000000001e-05 2021-02-10T04:31:40.398 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.398 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.399 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:31:40.399 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.657586+0000", 2021-02-10T04:31:40.399 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.6610000000000003e-06 2021-02-10T04:31:40.399 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.399 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.399 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:31:40.399 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.657593+0000", 2021-02-10T04:31:40.399 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.7518e-05 2021-02-10T04:31:40.399 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.399 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.399 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:31:40.399 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.657660+0000", 2021-02-10T04:31:40.399 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00012691499999999999 2021-02-10T04:31:40.399 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.400 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.400 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:31:40.400 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.657787+0000", 2021-02-10T04:31:40.400 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00018677099999999999 2021-02-10T04:31:40.400 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.400 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.400 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for subops from 1,4", 2021-02-10T04:31:40.400 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.657974+0000", 2021-02-10T04:31:40.400 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001340576 2021-02-10T04:31:40.400 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.400 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.400 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op_commit", 2021-02-10T04:31:40.400 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.659314+0000", 2021-02-10T04:31:40.401 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.011333849999999999 2021-02-10T04:31:40.401 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.401 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.401 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:31:40.401 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.670648+0000", 2021-02-10T04:31:40.401 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0060998770000000001 2021-02-10T04:31:40.401 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.401 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.401 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:31:40.401 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.676748+0000", 2021-02-10T04:31:40.401 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.0719000000000001e-05 2021-02-10T04:31:40.401 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.402 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.402 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:31:40.402 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.676789+0000", 2021-02-10T04:31:40.402 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.6999000000000003e-05 2021-02-10T04:31:40.402 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.402 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.402 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:31:40.402 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.676826+0000", 2021-02-10T04:31:40.402 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.019434732 2021-02-10T04:31:40.402 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.402 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:31:40.402 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.403 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.403 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.403 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:2541 7.4 7:2dcefd69:::smithi08575397-28:head [setxattr _header (57) in=7b,truncate 3932871] snapc b3=[b3] ondisk+write+known_if_redirected e278)", 2021-02-10T04:31:40.403 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:31:35.657794+0000", 2021-02-10T04:31:40.403 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 4.6354766889999999, 2021-02-10T04:31:40.403 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.019299283, 2021-02-10T04:31:40.403 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:31:40.403 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:31:40.403 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:31:40.403 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:31:40.403 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:31:40.403 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 2541 2021-02-10T04:31:40.403 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.404 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:31:40.404 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.404 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:31:40.404 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.657794+0000", 2021-02-10T04:31:40.404 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:31:40.404 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.404 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.404 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:31:40.404 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.657794+0000", 2021-02-10T04:31:40.404 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.5881999999999999e-05 2021-02-10T04:31:40.404 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.404 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.404 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:31:40.404 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.657840+0000", 2021-02-10T04:31:40.405 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.2760000000000002e-06 2021-02-10T04:31:40.405 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.405 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.405 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:31:40.405 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.657844+0000", 2021-02-10T04:31:40.405 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.1740000000000003e-06 2021-02-10T04:31:40.405 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.405 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.405 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:31:40.405 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.657851+0000", 2021-02-10T04:31:40.405 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.0389999999999993e-06 2021-02-10T04:31:40.405 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.405 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.405 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:31:40.406 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.657861+0000", 2021-02-10T04:31:40.406 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000855298 2021-02-10T04:31:40.406 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.406 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.406 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:31:40.406 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.658716+0000", 2021-02-10T04:31:40.406 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00010185500000000001 2021-02-10T04:31:40.406 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.406 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.406 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:31:40.406 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.658818+0000", 2021-02-10T04:31:40.406 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000128794 2021-02-10T04:31:40.406 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.407 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.407 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for subops from 1,4", 2021-02-10T04:31:40.407 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.658946+0000", 2021-02-10T04:31:40.407 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.011709043000000001 2021-02-10T04:31:40.407 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.407 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.407 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op_commit", 2021-02-10T04:31:40.407 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.670655+0000", 2021-02-10T04:31:40.407 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000113261 2021-02-10T04:31:40.407 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.407 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.407 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:31:40.407 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.670769+0000", 2021-02-10T04:31:40.408 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0061891330000000003 2021-02-10T04:31:40.408 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.408 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.408 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:31:40.408 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.676958+0000", 2021-02-10T04:31:40.408 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.9312999999999999e-05 2021-02-10T04:31:40.408 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.408 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.408 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:31:40.408 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.677017+0000", 2021-02-10T04:31:40.408 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.6215000000000001e-05 2021-02-10T04:31:40.408 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.408 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.408 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:31:40.409 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.677093+0000", 2021-02-10T04:31:40.409 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.019299283 2021-02-10T04:31:40.409 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.409 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:31:40.409 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.409 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.409 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.409 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:2542 7.4 7:2dcefd69:::smithi08575397-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e278)", 2021-02-10T04:31:40.409 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:31:35.657895+0000", 2021-02-10T04:31:40.409 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 4.6353754680000003, 2021-02-10T04:31:40.409 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.019463920999999999, 2021-02-10T04:31:40.410 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:31:40.410 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:31:40.410 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:31:40.410 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:31:40.410 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:31:40.410 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 2542 2021-02-10T04:31:40.410 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.410 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:31:40.410 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.410 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:31:40.411 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.657895+0000", 2021-02-10T04:31:40.411 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:31:40.411 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.411 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.411 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:31:40.411 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.657895+0000", 2021-02-10T04:31:40.411 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.7344e-05 2021-02-10T04:31:40.411 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.411 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.411 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:31:40.411 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.657933+0000", 2021-02-10T04:31:40.411 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.7139999999999999e-06 2021-02-10T04:31:40.412 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.412 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.412 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:31:40.412 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.657936+0000", 2021-02-10T04:31:40.412 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.8089999999999993e-06 2021-02-10T04:31:40.412 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.412 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.412 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:31:40.412 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.657944+0000", 2021-02-10T04:31:40.412 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.2719999999999997e-06 2021-02-10T04:31:40.412 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.412 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.412 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:31:40.412 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.657949+0000", 2021-02-10T04:31:40.413 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001282116 2021-02-10T04:31:40.413 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.413 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.413 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:31:40.413 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.659232+0000", 2021-02-10T04:31:40.413 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.3158000000000007e-05 2021-02-10T04:31:40.413 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.413 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.413 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for rw locks", 2021-02-10T04:31:40.413 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.659305+0000", 2021-02-10T04:31:40.413 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.017835101999999999 2021-02-10T04:31:40.413 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.413 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.414 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:31:40.414 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.677140+0000", 2021-02-10T04:31:40.414 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.9555000000000007e-05 2021-02-10T04:31:40.414 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.414 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.414 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:31:40.414 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.677219+0000", 2021-02-10T04:31:40.414 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000139851 2021-02-10T04:31:40.414 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:31:40.414 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:31:40.414 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:31:40.414 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:35.677359+0000", 2021-02-10T04:31:40.415 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.019463920999999999 2021-02-10T04:31:40.415 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.415 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:31:40.415 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.415 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:31:40.415 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:31:40.415 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:31:40.416 INFO:tasks.rados.rados.0.smithi085.stdout:968: done (12 left) 2021-02-10T04:31:40.416 INFO:tasks.rados.rados.0.smithi085.stdout:969: write (excl) oid 13 current snap is 93 2021-02-10T04:31:40.417 INFO:tasks.rados.rados.0.smithi085.stdout:969: seq_num 363 ranges {500941=783978,1942140=158801} 2021-02-10T04:31:40.417 INFO:tasks.rados.rados.0.smithi085.stdout:969: writing smithi08575397-13 from 500941 to 1284919 tid 1 2021-02-10T04:31:40.417 INFO:tasks.rados.rados.0.smithi085.stdout:969: writing smithi08575397-13 from 1942140 to 2100941 tid 2 2021-02-10T04:31:40.417 INFO:tasks.rados.rados.0.smithi085.stdout:970: read oid 6 snap -1 2021-02-10T04:31:40.417 INFO:tasks.rados.rados.0.smithi085.stdout:970: expect (ObjNum 357 snap 90 seq_num 357) 2021-02-10T04:31:40.417 INFO:tasks.rados.rados.0.smithi085.stdout:971: write oid 28 current snap is 93 2021-02-10T04:31:40.417 INFO:tasks.rados.rados.0.smithi085.stdout:971: seq_num 364 ranges {517748=459227,1600622=406110} 2021-02-10T04:31:40.417 INFO:tasks.rados.rados.0.smithi085.stdout:971: writing smithi08575397-28 from 517748 to 976975 tid 1 2021-02-10T04:31:40.417 INFO:tasks.rados.rados.0.smithi085.stdout:971: writing smithi08575397-28 from 1600622 to 2006732 tid 2 2021-02-10T04:31:40.417 INFO:tasks.rados.rados.0.smithi085.stdout:970: expect (ObjNum 357 snap 90 seq_num 357) 2021-02-10T04:31:40.507 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:31:40.514 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[52836]: debug 2021-02-10T04:31:40.514 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[52836]: 2021-02-10T04:31:40.381+0000 7fba70fb0700 -1 osd.3 pg_epoch: 283 pg[7.a( v 281'271 (0'0,281'271] local-lis/les=93/94 n=3 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [3,2,5] r=0 lpr=93 crt=281'271 lcod 281'269 mlcod 281'269 active+clean+snaptrim trimq=[b3~1,b5~1] ps=42] removing snap head 2021-02-10T04:31:40.514 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[52836]: 2021-02-10T04:31:40.515 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: cluster 2021-02-10T04:31:39.207525+0000 osd.1 2021-02-10T04:31:40.515 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: (osd.1) 220 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:40.515 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: cluster 2021-02-10T04:31:39.207535+0000 osd.1 (osd.1) 221 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:40.515 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: cluster 2021-02-10T04:31:39.207541+0000 osd.1 (osd.1) 222 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:40.515 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: cluster 2021-02-10T04:31:39.207547+0000 osd.1 (osd.1) 223 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:40.515 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: cluster 2021-02-10T04:31:39.207552+0000 osd.1 (osd.1) 2021-02-10T04:31:40.515 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: 224 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:40.515 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: cluster 2021-02-10T04:31:39.207557+0000 osd.1 ( 2021-02-10T04:31:40.515 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: osd.1) 225 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:40.515 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:40.516 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: T04:31:39.363791+0000 mon.a (mon. 2021-02-10T04:31:40.516 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: 0) 1024 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2021-02-10T04:31:40.516 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: cluster 2021-02-10T04:31:40.516 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: 2021-02 2021-02-10T04:31:40.516 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: -10T04:31:39.415968+0000 osd.4 2021-02-10T04:31:40.516 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: (osd.4) 231 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:40.516 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: 2021-02-10T04:31:40.516 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: cluster 2021-02-10T04:31: 2021-02-10T04:31:40.516 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: 39.415974+0000 osd.4 (osd.4) 232 2021-02-10T04:31:40.516 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:40.516 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: cluster 2021-02-10T04:31:40.516 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: 2021-02-10T04:31:39.415979+0000 2021-02-10T04:31:40.517 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: osd.4 (osd.4) 233 : cluster [WRN] 2021-02-10T04:31:40.517 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:40.517 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:40.517 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: 02-10T04:31:39.415982+0000 osd.4 (osd 2021-02-10T04:31:40.518 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: .4) 234 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:40.518 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:40.518 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: -02-10T04:31:40.000708+0000 osd.2 2021-02-10T04:31:40.518 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: (osd.2) 238 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:40.518 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: 2021-02-10T04:31:40.518 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:31:40.518 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: 04:31:40.000718+0000 osd.2 (osd 2021-02-10T04:31:40.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: .2) 239 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:40.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:40.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: 02-10T04:31:40.000724+0000 osd.2 ( 2021-02-10T04:31:40.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: osd.2) 240 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:40.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:40.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: -10T04:31:40.000730+0000 osd.2 (osd 2021-02-10T04:31:40.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: .2) 241 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:40.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:40.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: -02-10T04:31:40.000736+0000 2021-02-10T04:31:40.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: osd.2 (osd.2) 242 : cluster [WRN] 2021-02-10T04:31:40.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:40.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:31:40.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: 04:31:40.000742+0000 osd.2 (osd 2021-02-10T04:31:40.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: .2) 243 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:40.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: cluster 2021-02-10T04:31:40.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: 2021-02-10T04:31:40.000749 2021-02-10T04:31:40.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 244 : cluster 2021-02-10T04:31:40.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:40.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:31:40.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: :31:40.000755+0000 osd.2 (osd. 2021-02-10T04:31:40.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: 2) 245 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:40.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: cluster 2021-02-10T04:31:40.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:31:40.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: T04:31:40.038932+0000 mgr.y ( 2021-02-10T04:31:40.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: mgr.14140) 449 : cluster [DBG] pgmap v573: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 3 active+clean+snaptrim, 163 active+clean; 140 MiB data, 521 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.3 MiB/s wr, 6 op/s 2021-02-10T04:31:40.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:40.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: -10T04:31:40.366300+0000 mon.a ( 2021-02-10T04:31:40.521 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[29319]: mon.0) 1025 : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2021-02-10T04:31:40.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: cluster 2021-02-10T04:31:40.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021 2021-02-10T04:31:40.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: - 2021-02-10T04:31:40.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 02 2021-02-10T04:31:40.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: - 2021-02-10T04:31:40.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 10 2021-02-10T04:31:40.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: T 2021-02-10T04:31:40.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 04 2021-02-10T04:31:40.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: : 2021-02-10T04:31:40.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 31 2021-02-10T04:31:40.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: : 2021-02-10T04:31:40.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 39 2021-02-10T04:31:40.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: . 2021-02-10T04:31:40.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 207525 2021-02-10T04:31:40.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: +0000 2021-02-10T04:31:40.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: osd.1 2021-02-10T04:31:40.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: ( 2021-02-10T04:31:40.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: osd 2021-02-10T04:31:40.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: . 2021-02-10T04:31:40.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 1 2021-02-10T04:31:40.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: ) 2021-02-10T04:31:40.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 220 2021-02-10T04:31:40.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: : 2021-02-10T04:31:40.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: cluster 2021-02-10T04:31:40.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: [WRN] 2021-02-10T04:31:40.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:40.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: cluster 2021-02-10T04:31:40.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021 2021-02-10T04:31:40.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: - 2021-02-10T04:31:40.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 02 2021-02-10T04:31:40.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: - 2021-02-10T04:31:40.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 10 2021-02-10T04:31:40.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: T 2021-02-10T04:31:40.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 04 2021-02-10T04:31:40.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: : 2021-02-10T04:31:40.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 31 2021-02-10T04:31:40.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: : 2021-02-10T04:31:40.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 39 2021-02-10T04:31:40.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: . 2021-02-10T04:31:40.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 207535 2021-02-10T04:31:40.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: +0000 2021-02-10T04:31:40.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: osd.1 2021-02-10T04:31:40.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: ( 2021-02-10T04:31:40.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: osd 2021-02-10T04:31:40.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: . 2021-02-10T04:31:40.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 1 2021-02-10T04:31:40.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: ) 2021-02-10T04:31:40.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 221 2021-02-10T04:31:40.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: : 2021-02-10T04:31:40.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: cluster 2021-02-10T04:31:40.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: [WRN] 2021-02-10T04:31:40.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:40.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.525 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: cluster 2021-02-10T04:31:40.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021 2021-02-10T04:31:40.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: - 2021-02-10T04:31:40.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 02 2021-02-10T04:31:40.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: - 2021-02-10T04:31:40.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 10 2021-02-10T04:31:40.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: T 2021-02-10T04:31:40.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 04 2021-02-10T04:31:40.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: : 2021-02-10T04:31:40.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 31 2021-02-10T04:31:40.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: : 2021-02-10T04:31:40.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 39 2021-02-10T04:31:40.526 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: . 2021-02-10T04:31:40.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 207541 2021-02-10T04:31:40.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: +0000 2021-02-10T04:31:40.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: osd.1 2021-02-10T04:31:40.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: ( 2021-02-10T04:31:40.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: osd 2021-02-10T04:31:40.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: . 2021-02-10T04:31:40.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 1 2021-02-10T04:31:40.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: ) 2021-02-10T04:31:40.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 222 2021-02-10T04:31:40.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: : 2021-02-10T04:31:40.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: cluster 2021-02-10T04:31:40.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.527 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: [WRN] 2021-02-10T04:31:40.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:40.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: cluster 2021-02-10T04:31:40.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021 2021-02-10T04:31:40.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: - 2021-02-10T04:31:40.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 02 2021-02-10T04:31:40.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: - 2021-02-10T04:31:40.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 10 2021-02-10T04:31:40.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: T 2021-02-10T04:31:40.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 04 2021-02-10T04:31:40.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: : 2021-02-10T04:31:40.528 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 31 2021-02-10T04:31:40.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: : 2021-02-10T04:31:40.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 39 2021-02-10T04:31:40.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: . 2021-02-10T04:31:40.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 207547 2021-02-10T04:31:40.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: +0000 2021-02-10T04:31:40.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: osd.1 2021-02-10T04:31:40.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: ( 2021-02-10T04:31:40.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: osd 2021-02-10T04:31:40.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: . 2021-02-10T04:31:40.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 1 2021-02-10T04:31:40.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: ) 2021-02-10T04:31:40.529 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 223 2021-02-10T04:31:40.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: : 2021-02-10T04:31:40.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: cluster 2021-02-10T04:31:40.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: [WRN] 2021-02-10T04:31:40.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:40.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: cluster 2021-02-10T04:31:40.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021 2021-02-10T04:31:40.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: - 2021-02-10T04:31:40.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 02 2021-02-10T04:31:40.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: - 2021-02-10T04:31:40.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 10 2021-02-10T04:31:40.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: T 2021-02-10T04:31:40.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 04 2021-02-10T04:31:40.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: : 2021-02-10T04:31:40.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 31 2021-02-10T04:31:40.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: : 2021-02-10T04:31:40.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 39 2021-02-10T04:31:40.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: . 2021-02-10T04:31:40.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 207552 2021-02-10T04:31:40.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: +0000 2021-02-10T04:31:40.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: osd.1 2021-02-10T04:31:40.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: ( 2021-02-10T04:31:40.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: osd 2021-02-10T04:31:40.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: . 2021-02-10T04:31:40.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 1 2021-02-10T04:31:40.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: ) 2021-02-10T04:31:40.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 224 2021-02-10T04:31:40.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: : 2021-02-10T04:31:40.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: cluster 2021-02-10T04:31:40.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: [WRN] 2021-02-10T04:31:40.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:40.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: cluster 2021-02-10T04:31:40.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021 2021-02-10T04:31:40.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: - 2021-02-10T04:31:40.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 02-10T04:31: 2021-02-10T04:31:40.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 39.207557+0000 osd.1 (osd.1) 225 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:40.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:40.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 02-10T04:31:39.363791+0000 2021-02-10T04:31:40.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: mon.a (mon.0) 1024 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2021-02-10T04:31:40.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: cluster 2021-02-10T04:31: 2021-02-10T04:31:40.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 39.415968+0000 osd.4 (osd.4) 231 2021-02-10T04:31:40.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:40.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:40.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: :31:39.415974+0000 osd.4 (osd.4 2021-02-10T04:31:40.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: ) 232 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:40.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:40.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 02-10T04:31:39.415979+0000 osd.4 2021-02-10T04:31:40.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: (osd.4) 233 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:40.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:40.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 31:39.415982+0000 osd.4 (osd.4 2021-02-10T04:31:40.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: ) 234 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:40.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: cluster 2021-02-10T04:31:40.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.000708 2021-02-10T04:31:40.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 238 : cluster 2021-02-10T04:31:40.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:40.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:40.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: :31:40.000718+0000 osd.2 (osd. 2021-02-10T04:31:40.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2) 239 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:40.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:40.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: -10T04:31:40.000724+0000 osd.2 2021-02-10T04:31:40.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: (osd.2) 240 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:40.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: cluster 2021-02-10T04:31:40.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.000730+0000 osd.2 (osd.2) 241 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:40.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.536 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: cluster 2021-02-10T04:31:40.536 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 2021-02-10T04:31:40.000736+0000 osd.2 (osd.2) 242 2021-02-10T04:31:40.536 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:40.536 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: cluster 2021-02-10T04:31 2021-02-10T04:31:40.536 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: :40.000742+0000 osd.2 (osd.2) 2021-02-10T04:31:40.536 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 243 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:40.536 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:40.536 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: T04:31:40.000749+0000 osd.2 (osd 2021-02-10T04:31:40.536 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: .2) 244 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:40.536 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:40.536 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 02-10T04:31:40.000755+0000 2021-02-10T04:31:40.536 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: osd.2 (osd.2) 245 : cluster [WRN] 2021-02-10T04:31:40.536 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:40.536 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: cluster 2021-02-10T04:31:40. 2021-02-10T04:31:40.537 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 038932+0000 mgr.y (mgr.14140) 449 : cluster 2021-02-10T04:31:40.537 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: [DBG] pgmap v573: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 3 active+clean+snaptrim, 163 active+clean; 140 MiB data, 521 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.3 MiB/s wr, 6 op/s 2021-02-10T04:31:40.537 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:40.537 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: 31:40.366300+0000 mon.a (mon.0 2021-02-10T04:31:40.537 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[35961]: ) 1025 : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2021-02-10T04:31:40.537 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:31:40 smithi085 conmon[41540]: debug 2021-02-10T04:31:40.278+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:40.539 INFO:tasks.rados.rados.0.smithi085.stdout:969: finishing write tid 1 to smithi08575397-13 2021-02-10T04:31:40.539 INFO:tasks.rados.rados.0.smithi085.stdout:970: done (14 left) 2021-02-10T04:31:40.539 INFO:tasks.rados.rados.0.smithi085.stdout:972: read oid 50 snap -1 2021-02-10T04:31:40.539 INFO:tasks.rados.rados.0.smithi085.stdout:972: expect deleted 2021-02-10T04:31:40.539 INFO:tasks.rados.rados.0.smithi085.stdout:973: write oid 40 current snap is 93 2021-02-10T04:31:40.539 INFO:tasks.rados.rados.0.smithi085.stdout:973: seq_num 365 ranges {534555=650829,1659104=475451} 2021-02-10T04:31:40.539 INFO:tasks.rados.rados.0.smithi085.stdout:973: writing smithi08575397-40 from 534555 to 1185384 tid 1 2021-02-10T04:31:40.540 INFO:tasks.rados.rados.0.smithi085.stdout:973: writing smithi08575397-40 from 1659104 to 2134555 tid 2 2021-02-10T04:31:40.540 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:40.540 INFO:tasks.rados.rados.0.smithi085.stdout:969: finishing write tid 2 to smithi08575397-13 2021-02-10T04:31:40.540 INFO:tasks.rados.rados.0.smithi085.stdout:969: finishing write tid 3 to smithi08575397-13 2021-02-10T04:31:40.540 INFO:tasks.rados.rados.0.smithi085.stdout:969: finishing write tid 4 to smithi08575397-13 2021-02-10T04:31:40.540 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 379 (ObjNum 363 snap 93 seq_num 363) dirty exists 2021-02-10T04:31:40.540 INFO:tasks.rados.rados.0.smithi085.stdout:969: left oid 13 (ObjNum 363 snap 93 seq_num 363) 2021-02-10T04:31:40.540 INFO:tasks.rados.rados.0.smithi085.stdout:971: finishing write tid 1 to smithi08575397-28 2021-02-10T04:31:40.540 INFO:tasks.rados.rados.0.smithi085.stdout:971: finishing write tid 2 to smithi08575397-28 2021-02-10T04:31:40.540 INFO:tasks.rados.rados.0.smithi085.stdout:969: done (15 left) 2021-02-10T04:31:40.540 INFO:tasks.rados.rados.0.smithi085.stdout:974: read oid 6 snap -1 2021-02-10T04:31:40.540 INFO:tasks.rados.rados.0.smithi085.stdout:974: expect (ObjNum 357 snap 90 seq_num 357) 2021-02-10T04:31:40.540 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:40.540 INFO:tasks.rados.rados.0.smithi085.stdout:971: finishing write tid 3 to smithi08575397-28 2021-02-10T04:31:40.541 INFO:tasks.rados.rados.0.smithi085.stdout:971: finishing write tid 4 to smithi08575397-28 2021-02-10T04:31:40.541 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 147 (ObjNum 364 snap 93 seq_num 364) dirty exists 2021-02-10T04:31:40.541 INFO:tasks.rados.rados.0.smithi085.stdout:971: left oid 28 (ObjNum 364 snap 93 seq_num 364) 2021-02-10T04:31:40.541 INFO:tasks.rados.rados.0.smithi085.stdout:973: finishing write tid 1 to smithi08575397-40 2021-02-10T04:31:40.541 INFO:tasks.rados.rados.0.smithi085.stdout:971: done (15 left) 2021-02-10T04:31:40.541 INFO:tasks.rados.rados.0.smithi085.stdout:972: done (14 left) 2021-02-10T04:31:40.541 INFO:tasks.rados.rados.0.smithi085.stdout:975: rollback oid 7 current snap is 93 2021-02-10T04:31:40.541 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 7 to 91 2021-02-10T04:31:40.541 INFO:tasks.rados.rados.0.smithi085.stdout:976: copy_from oid 28 from oid 23 current snap is 93 2021-02-10T04:31:40.541 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:40.543 INFO:tasks.rados.rados.0.smithi085.stdout:974: expect (ObjNum 357 snap 90 seq_num 357) 2021-02-10T04:31:40.613 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:31:40.639 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: cluster 2021-02-10T04:31:39.207525+0000 osd.1 (osd.1) 220 : cluster 2021-02-10T04:31:40.639 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:40.640 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: cluster 2021-02-10T04:31:39.207535+0000 osd.1 (osd.1) 221 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:40.640 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:40.640 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: -10T04:31:39.207541+0000 osd.1 (osd.1) 222 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:40.640 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:40.640 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: -10T04:31:39. 2021-02-10T04:31:40.640 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: 207547+0000 osd.1 (osd.1) 223 : cluster 2021-02-10T04:31:40.640 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:40.640 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: cluster 2021-02-10T04:31:40.640 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: 2021-02-10T04:31 2021-02-10T04:31:40.640 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: :39.207552+0000 osd.1 (osd 2021-02-10T04:31:40.640 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: .1) 224 : cluster [WRN] 2021-02-10T04:31:40.640 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:40.641 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: cluster 2021-02-10T04:31:40.641 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: 2021-02-10 2021-02-10T04:31:40.641 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: T04:31:39.207557 2021-02-10T04:31:40.641 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 225 : cluster 2021-02-10T04:31:40.641 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:40.641 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:40.641 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: 10T04:31:39.363791+0000 mon.a ( 2021-02-10T04:31:40.641 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: mon.0) 1024 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2021-02-10T04:31:40.641 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:40.641 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: 02-10T04:31:39. 2021-02-10T04:31:40.641 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: 415968+0000 osd.4 (osd.4) 2021-02-10T04:31:40.641 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: 231 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:40.642 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:40.642 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: -02-10T04:31:39.415974+0000 2021-02-10T04:31:40.642 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: osd.4 (osd.4) 232 : 2021-02-10T04:31:40.642 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:40.642 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:40.642 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: -02-10T04:31: 2021-02-10T04:31:40.642 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: 39.415979+0000 osd.4 (osd. 2021-02-10T04:31:40.642 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: 4) 233 : cluster [WRN] 2021-02-10T04:31:40.642 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:40.642 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:40.642 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: 02-10T04:31:39.415982+0000 osd.4 ( 2021-02-10T04:31:40.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: osd.4) 234 : cluster [WRN] 2021-02-10T04:31:40.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:40.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:40.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: T04:31:40.000708+0000 osd.2 (osd 2021-02-10T04:31:40.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: .2) 238 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:40.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:40.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: -02-10T04:31:40.000718 2021-02-10T04:31:40.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 239 : cluster [WRN] 2021-02-10T04:31:40.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: cluster 2021-02-10T04:31 2021-02-10T04:31:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: :40.000724+0000 osd.2 (osd.2) 2021-02-10T04:31:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: 240 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: -10T04:31:40.000730+0000 2021-02-10T04:31:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: osd.2 (osd.2) 241 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: 02-10T04:31:40.000736+0000 2021-02-10T04:31:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: osd.2 (osd.2) 242 : cluster [WRN] 2021-02-10T04:31:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: cluster 2021-02-10T04:31 2021-02-10T04:31:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: :40.000742+0000 osd.2 (osd.2) 2021-02-10T04:31:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: 243 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: cluster 2021-02-10T04:31:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: 2021-02-10T04:31:40 2021-02-10T04:31:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: .000749+0000 osd.2 (osd.2) 244 : cluster 2021-02-10T04:31:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: cluster 2021-02-10T04:31:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: 2021-02-10T04:31:40.000755+0000 osd.2 (osd 2021-02-10T04:31:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: .2) 245 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:40.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: 10T04:31:40.038932 2021-02-10T04:31:40.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 449 : 2021-02-10T04:31:40.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: cluster [DBG] pgmap v573: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 3 active+clean+snaptrim, 163 active+clean; 140 MiB data, 521 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.3 MiB/s wr, 6 op/s 2021-02-10T04:31:40.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:40.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: -02-10T04:31: 2021-02-10T04:31:40.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: 40.366300+0000 mon.a (mon.0) 1025 2021-02-10T04:31:40.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[29288]: : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2021-02-10T04:31:40.646 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[33677]: debug 2021-02-10T04:31:40.396+0000 7fbec19a1700 -1 osd.4 283 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:31:40.647 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:40.675 INFO:tasks.rados.rados.0.smithi085.stdout:973: finishing write tid 2 to smithi08575397-40 2021-02-10T04:31:40.675 INFO:tasks.rados.rados.0.smithi085.stdout:973: finishing write tid 3 to smithi08575397-40 2021-02-10T04:31:40.675 INFO:tasks.rados.rados.0.smithi085.stdout:973: finishing write tid 4 to smithi08575397-40 2021-02-10T04:31:40.675 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 508 (ObjNum 365 snap 93 seq_num 365) dirty exists 2021-02-10T04:31:40.675 INFO:tasks.rados.rados.0.smithi085.stdout:973: left oid 40 (ObjNum 365 snap 93 seq_num 365) 2021-02-10T04:31:40.675 INFO:tasks.rados.rados.0.smithi085.stdout:975: finishing rollback tid 0 to smithi08575397-7 2021-02-10T04:31:40.675 INFO:tasks.rados.rados.0.smithi085.stdout:975: finishing rollback tid 1 to smithi08575397-7 2021-02-10T04:31:40.675 INFO:tasks.rados.rados.0.smithi085.stdout:975: finishing rollback tid 2 to smithi08575397-7 2021-02-10T04:31:40.676 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 455 (ObjNum 352 snap 90 seq_num 352) dirty exists 2021-02-10T04:31:40.676 INFO:tasks.rados.rados.0.smithi085.stdout:976: finishing copy_from to smithi08575397-28 2021-02-10T04:31:40.676 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 150 (ObjNum 346 snap 88 seq_num 346) dirty exists 2021-02-10T04:31:40.676 INFO:tasks.rados.rados.0.smithi085.stdout:976: finishing copy_from racing read to smithi08575397-28 2021-02-10T04:31:40.676 INFO:tasks.rados.rados.0.smithi085.stdout:973: done (15 left) 2021-02-10T04:31:40.676 INFO:tasks.rados.rados.0.smithi085.stdout:974: done (14 left) 2021-02-10T04:31:40.676 INFO:tasks.rados.rados.0.smithi085.stdout:975: done (13 left) 2021-02-10T04:31:40.676 INFO:tasks.rados.rados.0.smithi085.stdout:976: done (12 left) 2021-02-10T04:31:40.676 INFO:tasks.rados.rados.0.smithi085.stdout:977: delete oid 24 current snap is 93 2021-02-10T04:31:40.677 INFO:tasks.rados.rados.0.smithi085.stdout:977: done (12 left) 2021-02-10T04:31:40.677 INFO:tasks.rados.rados.0.smithi085.stdout:978: snap_remove snap 91 2021-02-10T04:31:40.731 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:31:40.836 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:31:40.863 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:40.893 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[40892]: debug 2021-02-10T04:31:40.706+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:40.981 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:31:41.086 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:31:41.119 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:41.222 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:31:40 smithi164 conmon[44544]: debug 2021-02-10T04:31:40.938+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:41.222 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[40892]: debug 2021-02-10T04:31:41.221+0000 7f30364c7700 -1 osd.6 pg_epoch: 283 pg[7.d( v 281'198 (0'0,281'198] local-lis/les=93/94 n=4 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [6,1,0] r=0 lpr=93 crt=281'198 lcod 281'196 mlcod 281'196 active+clean+snaptrim trimq=[b5~1] ps=43] removing snap head 2021-02-10T04:31:41.387 INFO:tasks.rados.rados.0.smithi085.stdout:978: done (12 left) 2021-02-10T04:31:41.387 INFO:tasks.rados.rados.0.smithi085.stdout:979: read oid 36 snap -1 2021-02-10T04:31:41.387 INFO:tasks.rados.rados.0.smithi085.stdout:979: expect (ObjNum 306 snap 74 seq_num 306) 2021-02-10T04:31:41.388 INFO:tasks.rados.rados.0.smithi085.stdout:980: delete oid 37 current snap is 93 2021-02-10T04:31:41.390 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[49106]: debug 2021-02-10T04:31:41.047+0000 7f77a3e94700 -1 osd.2 283 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:31:41.391 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[49106]: debug 2021-02-10T04:31:41.192+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:41.391 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: cluster 2021-02-10T04:31:41.391 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[45443]: debug 2021-02-10T04:31:41.224+0000 7f7140d73700 -1 osd.1 283 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:31:41.399 INFO:tasks.rados.rados.0.smithi085.stdout:979: expect (ObjNum 306 snap 74 seq_num 306) 2021-02-10T04:31:41.440 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:31:41.553 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset noscrub 2021-02-10T04:31:42.424 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[33677]: debug 2021-02-10T04:31:41.412+0000 7fbec19a1700 -1 osd.4 284 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:31:42.424 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: cluster 2021-02-10T04:31:40.199867+0000 2021-02-10T04:31:42.424 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: osd.1 (osd.1) 226 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:42.424 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: cluster 2021-02-10T04:31:40.199877+0000 osd.1 (osd.1) 227 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:42.424 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: 2021-02-10T04:31:42.424 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:31:42.424 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: 31:40.199883+0000 osd.1 (osd.1) 228 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:42.424 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:42.424 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: 02-10T04:31:40.199889+0000 osd.1 (osd.1) 229 2021-02-10T04:31:42.425 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:42.425 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:42.425 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: -02-10T04:31:40.199928+0000 osd.1 (osd.1) 230 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:42.425 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:31:42.425 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: :31:40.199934+0000 osd.1 (osd.1) 231 : cluster 2021-02-10T04:31:42.425 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:42.425 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: cluster 2021-02-10T04:31:40.397011+0000 2021-02-10T04:31:42.425 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: osd.4 (osd.4) 235 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:42.425 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:31:42.425 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: 04:31:40.397019+0000 osd.4 (osd. 2021-02-10T04:31:42.425 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: 4) 236 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:42.425 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:42.426 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: T04:31:40.397023+0000 2021-02-10T04:31:42.426 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: osd.4 (osd.4) 237 : 2021-02-10T04:31:42.426 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:42.426 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: cluster 2021-02-10T04:31:42.426 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:31:42.426 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: 31:40.397026+0000 osd.4 ( 2021-02-10T04:31:42.426 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: osd.4) 238 : cluster [WRN] 2021-02-10T04:31:42.426 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:42.426 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: cluster 2021-02-10T04:31:42.427 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: 2021-02-10T04:31:41.048494+0000 osd.2 (osd.2) 246 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:42.427 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: cluster 2021-02-10T04:31:41.048503+0000 osd.2 2021-02-10T04:31:42.427 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: (osd.2) 247 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:42.427 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:42.427 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: 10T04:31:41.048523+0000 osd.2 (osd. 2021-02-10T04:31:42.427 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: 2) 248 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:42.427 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: cluster 2021-02-10T04:31:41.048530+0000 osd.2 (osd.2) 249 : cluster 2021-02-10T04:31:42.427 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:42.427 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: cluster 2021-02-10T04:31:42.427 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: 2021-02-10T04:31:41.048537+0000 osd.2 (osd. 2021-02-10T04:31:42.427 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: 2) 250 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:42.427 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:31:42.427 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: 04:31:41.048542+0000 osd.2 (osd.2) 251 : cluster 2021-02-10T04:31:42.428 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:42.428 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:42.428 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: 10T04:31:41.048546+0000 osd.2 (osd.2 2021-02-10T04:31:42.428 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: ) 252 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:42.428 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:42.428 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: 10T04:31:41.048551 2021-02-10T04:31:42.428 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:41 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 253 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:42.428 INFO:tasks.rados.rados.0.smithi085.stdout:979: done (13 left) 2021-02-10T04:31:42.428 INFO:tasks.rados.rados.0.smithi085.stdout:980: done (12 left) 2021-02-10T04:31:42.428 INFO:tasks.rados.rados.0.smithi085.stdout:981: snap_remove snap 92 2021-02-10T04:31:42.428 INFO:tasks.rados.rados.0.smithi085.stdout:981: done (12 left) 2021-02-10T04:31:42.428 INFO:tasks.rados.rados.0.smithi085.stdout:982: delete oid 28 current snap is 93 2021-02-10T04:31:42.429 INFO:tasks.rados.rados.0.smithi085.stdout:982: done (12 left) 2021-02-10T04:31:42.429 INFO:tasks.rados.rados.0.smithi085.stdout:983: snap_create 2021-02-10T04:31:42.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: cluster 2021-02-10T04:31:40.199867+0000 osd.1 (osd.1) 226 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:42.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: cluster 2021-02-10T04:31:40.199877+0000 osd.1 (osd.1) 227 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:42.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: 2021-02-10T04:31:42.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:42.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: 04:31:40.199883+0000 osd.1 (osd.1) 228 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:42.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:42.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: -10T04:31:40.199889+0000 osd.1 (osd. 2021-02-10T04:31:42.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: 1) 229 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:42.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:42.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: 10T04:31:40.199928+0000 osd.1 (osd.1 2021-02-10T04:31:42.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: ) 230 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:42.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:42.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: T04:31:40.199934+0000 osd.1 (osd. 2021-02-10T04:31:42.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: 1) 231 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:42.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:42.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: :31:40.397011+0000 osd.4 (osd.4) 2021-02-10T04:31:42.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: 235 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:42.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:42.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: 31:40.397019+0000 osd.4 (osd.4) 236 2021-02-10T04:31:42.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:42.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: cluster 2021-02-10T04:31: 2021-02-10T04:31:42.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: 40.397023+0000 osd.4 (osd.4) 237 : cluster 2021-02-10T04:31:42.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:42.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: cluster 2021-02-10T04:31:40. 2021-02-10T04:31:42.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: 397026+0000 osd.4 (osd.4) 238 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:42.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: 2021-02-10T04:31:42.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: cluster 2021-02-10T04:31:42.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: 2021-02-10T04:31:41.048494+0000 osd.2 2021-02-10T04:31:42.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: (osd.2) 246 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:42.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:42.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: -02-10T04:31:41.048503+0000 2021-02-10T04:31:42.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: osd.2 (osd.2) 247 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:42.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: 2021-02-10T04:31:42.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: cluster 2021-02-10T04:31:42.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: 2021-02-10T04:31:41.048523+0000 osd.2 (osd.2) 248 : cluster 2021-02-10T04:31:42.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:42.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: cluster 2021-02-10T04:31:41. 2021-02-10T04:31:42.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: 048530+0000 osd.2 (osd.2) 249 : cluster [WRN] 2021-02-10T04:31:42.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:42.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: cluster 2021-02-10T04:31:42.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: 2021-02-10T04:31:41.048537+0000 osd.2 (osd. 2021-02-10T04:31:42.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: 2) 250 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:42.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:42.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: 10T04:31:41.048542+0000 2021-02-10T04:31:42.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: osd.2 (osd.2) 251 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:42.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:42.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: -02-10T04:31:41.048546+0000 osd.2 ( 2021-02-10T04:31:42.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: osd.2) 252 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:42.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:42.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: T04:31:41.048551+0000 osd.2 (osd. 2021-02-10T04:31:42.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[35961]: 2) 253 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:42.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: 2021-02-10T04:31:42.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: 2021 2021-02-10T04:31:42.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: - 2021-02-10T04:31:42.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: 02 2021-02-10T04:31:42.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: - 2021-02-10T04:31:42.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: 10 2021-02-10T04:31:42.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: T04:31:40.199867 2021-02-10T04:31:42.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: +0000 osd.1 (osd.1) 226 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:42.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:42.434 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: 02-10T04:31:40.199877+0000 osd.1 2021-02-10T04:31:42.434 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: (osd.1) 227 : cluster [WRN] 2021-02-10T04:31:42.434 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:42.434 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:42.434 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: T04:31:40.199883+0000 osd.1 (osd 2021-02-10T04:31:42.434 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: .1) 228 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:42.434 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: cluster 2021-02-10T04:31:40 2021-02-10T04:31:42.434 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: .199889+0000 osd.1 (osd.1) 229 : cluster 2021-02-10T04:31:42.434 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:42.434 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: cluster 2021-02-10T04:31 2021-02-10T04:31:42.434 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: :40.199928+0000 osd.1 (osd.1) 230 : 2021-02-10T04:31:42.434 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:42.435 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: cluster 2021-02-10T04:31 2021-02-10T04:31:42.435 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: :40.199934+0000 osd.1 (osd.1) 231 2021-02-10T04:31:42.435 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:42.435 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:42.435 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: T04:31:40.397011+0000 osd.4 (osd 2021-02-10T04:31:42.435 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: .4) 235 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:42.435 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:42.435 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: 10T04:31:40.397019+0000 osd.4 ( 2021-02-10T04:31:42.435 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: osd.4) 236 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:42.435 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:42.435 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: 02-10T04:31:40.397023+0000 2021-02-10T04:31:42.435 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: osd.4 (osd.4) 237 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:42.435 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: 2021-02-10T04:31:42.436 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: cluster 2021-02-10T04:31:42.436 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: 2021-02-10T04:31:40.397026+0000 osd.4 (osd. 2021-02-10T04:31:42.436 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: 4) 238 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:42.436 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:42.436 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: -10T04:31:41.048494+0000 osd.2 2021-02-10T04:31:42.436 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: (osd.2) 246 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:42.436 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:42.436 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: 02-10T04:31:41.048503+0000 osd.2 ( 2021-02-10T04:31:42.436 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: osd.2) 247 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:42.436 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:31:42.436 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: 04:31:41.048523+0000 osd.2 (osd.2) 2021-02-10T04:31:42.436 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: 248 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:42.436 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:31:42.437 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: 04:31:41.048530+0000 osd.2 (osd 2021-02-10T04:31:42.437 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: .2) 249 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:42.437 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:42.437 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: -02-10T04:31:41.048537+0000 2021-02-10T04:31:42.437 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: osd.2 (osd.2) 250 : cluster [WRN] 2021-02-10T04:31:42.437 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:42.437 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: cluster 2021-02-10T04:31:41 2021-02-10T04:31:42.437 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: .048542+0000 osd.2 (osd.2) 251 : cluster 2021-02-10T04:31:42.437 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:42.437 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:31:42.437 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: 31:41.048546+0000 osd.2 (osd.2) 2021-02-10T04:31:42.437 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: 252 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:42.437 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:42.437 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: T04:31:41.048551+0000 osd.2 (osd. 2021-02-10T04:31:42.438 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:41 smithi085 conmon[29319]: 2) 253 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:42.438 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: cluster 2021-02-10T04:31:41.224669+0000 osd.1 (osd. 2021-02-10T04:31:42.438 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: 1) 232 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:42.438 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: cluster 2021-02-10T04:31:41.224677+0000 osd.1 (osd.1) 233 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:42.438 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: cluster 2021-02-10T04:31:41.224684+0000 osd.1 (osd.1) 234 : 2021-02-10T04:31:42.438 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:42.438 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: cluster 2021-02-10T04:31:41. 2021-02-10T04:31:42.438 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: 224691+0000 osd.1 (osd.1) 235 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:42.438 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:42.438 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: 02-10T04:31:41.224697+0000 osd.1 ( 2021-02-10T04:31:42.438 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: osd.1) 236 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:42.438 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:42.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: 02-10T04:31:41.224704+0000 osd.1 ( 2021-02-10T04:31:42.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: osd.1) 237 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:42.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:42.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: -10T04:31:41.386962+0000 mon.a ( 2021-02-10T04:31:42.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: mon.0) 1026 : cluster [DBG] 2021-02-10T04:31:42.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: osdmap e284: 8 total, 8 up, 8 in 2021-02-10T04:31:42.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: cluster 2021-02-10T04:31:42.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:31:42.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: 31:41.413485+0000 osd.4 ( 2021-02-10T04:31:42.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: osd.4) 239 : cluster 2021-02-10T04:31:42.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:42.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:42.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: -02-10T04:31:41 2021-02-10T04:31:42.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: .413493+0000 osd.4 (osd. 2021-02-10T04:31:42.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: 4) 240 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:42.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: 2021-02-10T04:31:42.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:31:42.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: 04:31:41.413500+0000 osd.4 (osd.4 2021-02-10T04:31:42.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: ) 241 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:42.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:42.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: -02-10T04:31:41.413503 2021-02-10T04:31:42.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 242 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:42.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: 2021-02-10T04:31:42.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: cluster 2021-02-10T04:31:42.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: 2021-02-10T04:31:42.006632+0000 osd.2 (osd.2) 254 : cluster [WRN] 2021-02-10T04:31:42.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:42.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:42.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: -02-10T04:31:42.006637+0000 osd.2 (osd.2) 255 : cluster 2021-02-10T04:31:42.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:42.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:42.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: -02-10T04:31:42.006640+0000 osd.2 (osd.2 2021-02-10T04:31:42.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: ) 256 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:42.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:42.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: 02-10T04:31:42.006643+0000 2021-02-10T04:31:42.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: osd.2 (osd.2) 257 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:42.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: 2021-02-10T04:31:42.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: cluster 2021-02-10T04:31:42.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: 2021-02-10T04:31:42.006648+0000 osd.2 (osd.2) 258 2021-02-10T04:31:42.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:42.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: cluster 2021-02-10T04:31:42.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: 2021-02-10T04:31:42.006651+0000 osd.2 (osd.2) 259 : 2021-02-10T04:31:42.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:42.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:42.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: -02-10T04:31:42.006666+0000 osd.2 (osd.2) 260 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:42.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: cluster 2021-02-10T04:31 2021-02-10T04:31:42.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: :42.006672+0000 osd.2 (osd.2) 261 : cluster [WRN] 2021-02-10T04:31:42.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:42.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:42.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[29319]: -10T04:31:42.039465+0000 mgr.y (mgr.14140) 450 : cluster [DBG] pgmap v576: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 137 MiB data, 501 MiB used, 715 GiB / 715 GiB avail; 4.3 MiB/s rd, 3.3 MiB/s wr, 17 op/s 2021-02-10T04:31:42.442 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[49106]: debug 2021-02-10T04:31:42.006+0000 7f77a3e94700 -1 osd.2 284 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:31:42.443 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[45443]: debug 2021-02-10T04:31:42.202+0000 7f7140d73700 -1 osd.1 284 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:31:42.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: cluster 2021-02-10T04:31:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 2021-02-10T04:31:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 2021 2021-02-10T04:31:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: - 2021-02-10T04:31:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 02 2021-02-10T04:31:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: - 2021-02-10T04:31:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 10 2021-02-10T04:31:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: T04:31:41 2021-02-10T04:31:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: .224669+0000 osd.1 (osd.1) 232 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 31:41.224677+0000 osd.1 (osd.1) 233 : 2021-02-10T04:31:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:42.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: :31:41.224684+0000 osd.1 (osd.1) 234 : 2021-02-10T04:31:42.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:42.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: cluster 2021-02-10T04:31: 2021-02-10T04:31:42.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 41.224691+0000 osd.1 (osd.1) 235 : cluster 2021-02-10T04:31:42.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:42.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: cluster 2021-02-10T04:31:41. 2021-02-10T04:31:42.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 224697+0000 osd.1 (osd.1) 236 : cluster [WRN] 2021-02-10T04:31:42.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:42.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: cluster 2021-02-10T04:31: 2021-02-10T04:31:42.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 41.224704+0000 osd.1 (osd.1) 237 : 2021-02-10T04:31:42.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:42.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:42.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: :31:41.386962+0000 mon.a (mon.0) 1026 : 2021-02-10T04:31:42.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: cluster [DBG] osdmap e284: 8 total, 8 up, 8 in 2021-02-10T04:31:42.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:42.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: :31:41.413485+0000 osd.4 ( 2021-02-10T04:31:42.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: osd.4) 239 : cluster [WRN] 2021-02-10T04:31:42.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:42.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: cluster 2021-02-10T04:31:42.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 2021-02-10T04:31:41. 2021-02-10T04:31:42.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 413493+0000 osd.4 (osd.4) 240 : cluster [WRN] 2021-02-10T04:31:42.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:42.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: cluster 2021-02-10T04:31:41 2021-02-10T04:31:42.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: .413500+0000 osd.4 (osd.4) 241 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:42.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:42.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: -10T04:31:41.413503+0000 osd.4 (osd 2021-02-10T04:31:42.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: .4) 242 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:42.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:42.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: -10T04:31:42.006632+0000 osd.2 2021-02-10T04:31:42.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: (osd.2) 254 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:42.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:42.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 04:31:42.006637+0000 osd.2 2021-02-10T04:31:42.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: (osd.2) 255 : cluster 2021-02-10T04:31:42.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:42.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: cluster 2021-02-10T04:31:42.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 2021-02-10T04:31:42.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 2021 2021-02-10T04:31:42.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: - 2021-02-10T04:31:42.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 02 2021-02-10T04:31:42.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: - 2021-02-10T04:31:42.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 10 2021-02-10T04:31:42.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: T 2021-02-10T04:31:42.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 04 2021-02-10T04:31:42.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: : 2021-02-10T04:31:42.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 31 2021-02-10T04:31:42.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: : 2021-02-10T04:31:42.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 42 2021-02-10T04:31:42.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: . 2021-02-10T04:31:42.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 006640 2021-02-10T04:31:42.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: +0000 2021-02-10T04:31:42.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 2021-02-10T04:31:42.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: osd.2 2021-02-10T04:31:42.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: ( 2021-02-10T04:31:42.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: osd 2021-02-10T04:31:42.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: . 2021-02-10T04:31:42.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 2 2021-02-10T04:31:42.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: ) 2021-02-10T04:31:42.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 256 2021-02-10T04:31:42.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: : 2021-02-10T04:31:42.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: cluster 2021-02-10T04:31:42.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 2021-02-10T04:31:42.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: [WRN] 2021-02-10T04:31:42.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 2021-02-10T04:31:42.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:42.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 2021-02-10T04:31:42.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: cluster 2021-02-10T04:31:42.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 2021-02-10T04:31:42.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 2021 2021-02-10T04:31:42.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: - 2021-02-10T04:31:42.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 02 2021-02-10T04:31:42.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: - 2021-02-10T04:31:42.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 10 2021-02-10T04:31:42.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: T 2021-02-10T04:31:42.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 04 2021-02-10T04:31:42.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: : 2021-02-10T04:31:42.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 31 2021-02-10T04:31:42.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: : 2021-02-10T04:31:42.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 42 2021-02-10T04:31:42.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: . 2021-02-10T04:31:42.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 006643 2021-02-10T04:31:42.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: +0000 2021-02-10T04:31:42.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 2021-02-10T04:31:42.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: osd.2 2021-02-10T04:31:42.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: ( 2021-02-10T04:31:42.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: osd 2021-02-10T04:31:42.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: . 2021-02-10T04:31:42.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 2 2021-02-10T04:31:42.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: ) 2021-02-10T04:31:42.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 257 2021-02-10T04:31:42.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: : 2021-02-10T04:31:42.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: cluster 2021-02-10T04:31:42.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 2021-02-10T04:31:42.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: [WRN] 2021-02-10T04:31:42.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 2021-02-10T04:31:42.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:42.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 2021-02-10T04:31:42.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: cluster 2021-02-10T04:31:42.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 2021-02-10T04:31:42.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 2021 2021-02-10T04:31:42.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: - 2021-02-10T04:31:42.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 02 2021-02-10T04:31:42.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: - 2021-02-10T04:31:42.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 10 2021-02-10T04:31:42.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: T 2021-02-10T04:31:42.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 04 2021-02-10T04:31:42.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: : 2021-02-10T04:31:42.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 31 2021-02-10T04:31:42.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: : 2021-02-10T04:31:42.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 42 2021-02-10T04:31:42.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: . 2021-02-10T04:31:42.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 006648 2021-02-10T04:31:42.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: +0000 2021-02-10T04:31:42.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 2021-02-10T04:31:42.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: osd.2 2021-02-10T04:31:42.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: ( 2021-02-10T04:31:42.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: osd 2021-02-10T04:31:42.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: . 2021-02-10T04:31:42.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 2 2021-02-10T04:31:42.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: ) 2021-02-10T04:31:42.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 258 2021-02-10T04:31:42.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: : 2021-02-10T04:31:42.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: cluster 2021-02-10T04:31:42.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 2021-02-10T04:31:42.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: [WRN] 2021-02-10T04:31:42.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 2021-02-10T04:31:42.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:42.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 2021-02-10T04:31:42.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: cluster 2021-02-10T04:31:42.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 2021-02-10T04:31 2021-02-10T04:31:42.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: :42.006651+0000 osd.2 (osd.2) 259 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:42.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: cluster 2021-02-10T04:31:42.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 2021-02-10T04:31:42.006666+0000 2021-02-10T04:31:42.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: osd.2 (osd.2) 260 : cluster [WRN] 2021-02-10T04:31:42.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:42.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:42.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 04:31:42.006672+0000 osd.2 (osd. 2021-02-10T04:31:42.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 2) 261 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:42.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: cluster 2021-02-10T04:31:42.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: 2021-02-10T04:31:42.039465 2021-02-10T04:31:42.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 450 : cluster 2021-02-10T04:31:42.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[35961]: [DBG] pgmap v576: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 137 MiB data, 501 MiB used, 715 GiB / 715 GiB avail; 4.3 MiB/s rd, 3.3 MiB/s wr, 17 op/s 2021-02-10T04:31:42.704 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[41540]: debug 2021-02-10T04:31:42.704 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[41540]: 2021-02-10T04:31:42.415+0000 7fb94835c700 -1 osd.0 pg_epoch: 285 pg[7.4( v 285'153 (0'0,285'153] local-lis/les=93/94 n=4 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [0,4,1] r=0 lpr=93 luod=285'151 crt=285'153 lcod 283'150 mlcod 283'150 active+clean+snaptrim trimq=[b6~1,b9~1] ps=44] removing snap head 2021-02-10T04:31:42.704 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[41540]: 2021-02-10T04:31:42.731 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:31:42.893 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[33677]: debug 2021-02-10T04:31:42.406+0000 7fbec19a1700 -1 osd.4 285 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:31:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: cluster 2021-02-10T04:31:41.224669+0000 osd.1 (osd.1) 232 : cluster 2021-02-10T04:31:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: cluster 2021-02-10T04:31:41.224677+0000 osd.1 (osd.1) 233 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: cluster 2021-02-10T04:31:41.224684+0000 osd.1 (osd.1) 234 : cluster 2021-02-10T04:31:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:31:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: 04:31:41.224691+0000 osd.1 (osd.1 2021-02-10T04:31:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: ) 235 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: 02-10T04:31:41.224697+0000 2021-02-10T04:31:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: osd.1 (osd.1) 236 : cluster [WRN] 2021-02-10T04:31:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: -10T04:31:41.224704+0000 osd.1 (osd 2021-02-10T04:31:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: .1) 237 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:31:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: 04:31:41.386962+0000 mon.a (mon.0) 1026 : 2021-02-10T04:31:42.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: cluster [DBG] osdmap e284: 8 total, 8 up, 8 in 2021-02-10T04:31:42.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:42.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: 02-10T04:31:41.413485+0000 osd.4 ( 2021-02-10T04:31:42.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: osd.4) 239 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:42.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:42.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: T04:31:41.413493+0000 osd.4 (osd.4) 240 : cluster 2021-02-10T04:31:42.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:42.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:42.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: -10T04:31:41.413500+0000 osd.4 (osd.4) 241 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:42.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: cluster 2021-02-10T04:31:41.413503+0000 2021-02-10T04:31:42.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: osd.4 (osd.4) 242 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:42.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: cluster 2021-02-10T04:31:42.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: 2021-02-10T04:31:42.006632+0000 osd.2 (osd 2021-02-10T04:31:42.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: .2) 254 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:42.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: cluster 2021-02-10T04:31 2021-02-10T04:31:42.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: :42.006637+0000 osd.2 (osd.2 2021-02-10T04:31:42.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: ) 255 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:42.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:42.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: -10T04:31:42.006640+0000 osd.2 (osd 2021-02-10T04:31:42.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: .2) 256 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:42.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:42.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: 10T04:31:42.006643+0000 2021-02-10T04:31:42.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: osd.2 (osd.2) 257 : cluster [WRN] 2021-02-10T04:31:42.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:42.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:42.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: 10T04:31:42. 2021-02-10T04:31:42.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: 006648+0000 osd.2 (osd.2) 2021-02-10T04:31:42.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: 258 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:42.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:42.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: -02-10T04:31:42 2021-02-10T04:31:42.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: .006651+0000 osd.2 (osd.2 2021-02-10T04:31:42.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: ) 259 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:42.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: 2021-02-10T04:31:42.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: cluster 2021-02-10T04:31:42.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: 2021-02-10T04:31 2021-02-10T04:31:42.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: :42.006666+0000 osd.2 (osd.2 2021-02-10T04:31:42.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: ) 260 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:42.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:42.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: -10T04:31:42.006672 2021-02-10T04:31:42.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 261 : cluster 2021-02-10T04:31:42.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:42.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: cluster 2021-02-10T04:31:42.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:31:42.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: 31:42.039465+0000 mgr.y 2021-02-10T04:31:42.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: (mgr.14140) 450 : cluster 2021-02-10T04:31:42.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:42 smithi164 conmon[29288]: [DBG] pgmap v576: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 137 MiB data, 501 MiB used, 715 GiB / 715 GiB avail; 4.3 MiB/s rd, 3.3 MiB/s wr, 17 op/s 2021-02-10T04:31:43.083 INFO:tasks.rados.rados.0.smithi085.stdout:983: done (12 left) 2021-02-10T04:31:43.083 INFO:tasks.rados.rados.0.smithi085.stdout:984: copy_from oid 31 from oid 7 current snap is 94 2021-02-10T04:31:43.083 INFO:tasks.rados.rados.0.smithi085.stdout:985: delete oid 5 current snap is 94 2021-02-10T04:31:43.090 INFO:tasks.rados.rados.0.smithi085.stdout:985: done (13 left) 2021-02-10T04:31:43.090 INFO:tasks.rados.rados.0.smithi085.stdout:986: write (excl) oid 13 current snap is 94 2021-02-10T04:31:43.090 INFO:tasks.rados.rados.0.smithi085.stdout:986: seq_num 370 ranges {618590=627074,1835161=383429} 2021-02-10T04:31:43.097 INFO:tasks.rados.rados.0.smithi085.stdout:986: writing smithi08575397-13 from 618590 to 1245664 tid 1 2021-02-10T04:31:43.104 INFO:tasks.rados.rados.0.smithi085.stdout:986: writing smithi08575397-13 from 1835161 to 2218590 tid 2 2021-02-10T04:31:43.104 INFO:tasks.rados.rados.0.smithi085.stdout:987: copy_from oid 22 from oid 24 current snap is 94 2021-02-10T04:31:43.104 INFO:tasks.rados.rados.0.smithi085.stdout:988: snap_create 2021-02-10T04:31:43.104 INFO:tasks.rados.rados.0.smithi085.stdout:986: finishing write tid 1 to smithi08575397-13 2021-02-10T04:31:43.106 INFO:tasks.rados.rados.0.smithi085.stdout:987: finishing copy_from to smithi08575397-22 2021-02-10T04:31:43.106 INFO:tasks.rados.rados.0.smithi085.stdout:987: got expected ENOENT (src dne) 2021-02-10T04:31:43.106 INFO:tasks.rados.rados.0.smithi085.stdout:987: finishing copy_from racing read to smithi08575397-22 2021-02-10T04:31:43.107 INFO:tasks.rados.rados.0.smithi085.stdout:986: finishing write tid 2 to smithi08575397-13 2021-02-10T04:31:43.107 INFO:tasks.rados.rados.0.smithi085.stdout:986: finishing write tid 3 to smithi08575397-13 2021-02-10T04:31:43.108 INFO:tasks.rados.rados.0.smithi085.stdout:986: finishing write tid 4 to smithi08575397-13 2021-02-10T04:31:43.108 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 382 (ObjNum 370 snap 94 seq_num 370) dirty exists 2021-02-10T04:31:43.108 INFO:tasks.rados.rados.0.smithi085.stdout:986: left oid 13 (ObjNum 370 snap 94 seq_num 370) 2021-02-10T04:31:43.404 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[45443]: debug 2021-02-10T04:31:43.156+0000 7f7140d73700 -1 osd.1 286 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:31:43.405 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: cluster 2021-02-10T04:31:43.405 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: 2021-02-10T04:31:43.405 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: 2021 2021-02-10T04:31:43.405 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: - 2021-02-10T04:31:43.405 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: 02 2021-02-10T04:31:43.405 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: - 2021-02-10T04:31:43.405 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: 10 2021-02-10T04:31:43.405 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: T04:31:42.202765 2021-02-10T04:31:43.405 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: +0000 osd.1 (osd.1) 238 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:43.406 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:43.406 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: 10T04:31:42.202774+0000 osd.1 (osd 2021-02-10T04:31:43.406 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: .1) 239 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:43.406 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:43.406 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: 10T04:31:42.202779+0000 2021-02-10T04:31:43.406 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: osd.1 (osd.1) 240 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:43.406 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:43.406 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: 10T04:31:42.202782+0000 osd.1 (osd.1) 241 : cluster [WRN] 2021-02-10T04:31:43.406 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:43.406 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:43.406 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: 10T04:31:42.202788+0000 osd.1 (osd.1) 242 : cluster 2021-02-10T04:31:43.406 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:43.407 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:31:43.407 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: 31:42.202796+0000 osd.1 (osd.1) 243 : cluster 2021-02-10T04:31:43.407 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:43.407 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: cluster 2021-02-10T04:31 2021-02-10T04:31:43.407 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: :42.360166+0000 osd.4 (osd.4) 243 : cluster [DBG] 2021-02-10T04:31:43.407 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: purged_snaps scrub starts 2021-02-10T04:31:43.407 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:43.407 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: T04:31:42.360327+0000 osd.4 (osd.4) 244 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:31:43.407 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: cluster 2021-02-10T04:31 2021-02-10T04:31:43.407 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: :42.403603+0000 mon.a (mon.0) 1027 : 2021-02-10T04:31:43.407 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2021-02-10T04:31:43.407 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:43.407 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: -10T04:31:42.406881+0000 osd.4 (osd.4 2021-02-10T04:31:43.408 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: ) 245 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:43.408 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:43.408 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: -10T04:31:42.406886+0000 osd.4 ( 2021-02-10T04:31:43.408 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: osd.4) 246 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:43.408 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:43.408 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: -10T04:31:42.406894+0000 osd.4 2021-02-10T04:31:43.408 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: (osd.4) 247 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:43.408 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:43.408 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: T04:31:42.406900+0000 osd.4 (osd.4) 248 2021-02-10T04:31:43.408 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:43.408 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:43.408 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: 10T 2021-02-10T04:31:43.408 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: 04:31:42.967505+0000 osd.2 ( 2021-02-10T04:31:43.409 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: osd.2) 262 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:43.409 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:43.409 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:42 smithi085 conmon[49106]: debug 2021-02-10T04:31:42.966+0000 7f77a3e94700 -1 osd.2 285 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:31:43.409 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: cluster 2021-02-10T04:31:43.409 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: 2021-02-10T04:31:43.410 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: 2021 2021-02-10T04:31:43.410 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: -02-10T04:31:42.202765 2021-02-10T04:31:43.410 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 238 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:43.410 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:43.410 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: T04:31:42.202774+0000 osd.1 (osd.1 2021-02-10T04:31:43.410 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: ) 239 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:43.410 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:43.410 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: 02-10T04:31:42.202779+0000 osd.1 ( 2021-02-10T04:31:43.410 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: osd.1) 240 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:43.410 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:43.410 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: 02-10T04:31:42.202782+0000 osd.1 ( 2021-02-10T04:31:43.411 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: osd.1) 241 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:43.411 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:43.411 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: 02-10T04:31:42. 2021-02-10T04:31:43.411 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: 202788+0000 osd.1 (osd.1 2021-02-10T04:31:43.411 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: ) 242 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:43.411 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: cluster 2021-02-10T04:31:43.411 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: 2021-02-10T04:31:43.411 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: 2021-02-10T04:31: 2021-02-10T04:31:43.411 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: 42.202796+0000 osd.1 (osd.1) 243 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:43.411 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:43.412 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: 02-10T04:31:42.360166+0000 2021-02-10T04:31:43.412 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: osd.4 (osd.4) 243 : cluster [DBG] 2021-02-10T04:31:43.412 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: purged_snaps scrub starts 2021-02-10T04:31:43.412 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:43.412 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: T04:31:42.360327+0000 osd.4 (osd 2021-02-10T04:31:43.412 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: .4) 244 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:31:43.412 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:43.412 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: 02-10T04:31:42.403603+0000 2021-02-10T04:31:43.412 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: mon.a (mon.0) 1027 : cluster [DBG] 2021-02-10T04:31:43.412 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: osdmap e285: 8 total, 8 up, 8 in 2021-02-10T04:31:43.412 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:43.412 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: T04:31:42.406881+0000 osd.4 ( 2021-02-10T04:31:43.412 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: osd.4) 245 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:43.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: cluster 2021-02-10T04:31:43.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: 2021-02-10T04:31:42. 2021-02-10T04:31:43.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: 406886+0000 osd.4 (osd.4) 246 : cluster 2021-02-10T04:31:43.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:43.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:43.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: 31:42.406894+0000 osd.4 (osd.4 2021-02-10T04:31:43.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: ) 247 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:43.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:43.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: 04:31:42.406900+0000 osd.4 (osd 2021-02-10T04:31:43.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: .4) 248 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:43.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: cluster 2021-02-10T04:31:43.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: 2021-02-10T04:31:42.967505+0000 2021-02-10T04:31:43.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: osd.2 (osd.2) 262 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:43.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: 2021-02-10T04:31:43.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:43.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: -02-10T04:31:42.967514+0000 osd.2 (osd.2 2021-02-10T04:31:43.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: ) 263 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:43.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: cluster 2021-02-10T04:31 2021-02-10T04:31:43.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: :42.967520+0000 osd.2 (osd. 2021-02-10T04:31:43.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: cluster 2021-02-10T04:31:42.202765+0000 osd.1 (osd.1) 238 2021-02-10T04:31:43.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:43.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: cluster 2021-02-10T04:31:42.202774+0000 osd.1 (osd.1) 239 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:43.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: cluster 2021-02-10T04:31:43.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: 2021-02-10T04:31:42.202779+0000 osd.1 (osd.1) 240 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:43.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: cluster 2021-02-10T04:31:43.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: 2021-02-10T04:31:42.202782+0000 osd.1 (osd.1) 241 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:43.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: cluster 2021-02-10T04:31:42.202788+0000 osd.1 ( 2021-02-10T04:31:43.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: osd.1) 242 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:43.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: cluster 2021-02-10T04:31:42.202796+0000 osd.1 (osd.1) 243 : cluster [WRN] 2021-02-10T04:31:43.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:43.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: cluster 2021-02-10T04:31:43.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: 2021-02-10T04:31:42.360166+0000 osd.4 (osd.4) 243 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:31:43.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: cluster 2021-02-10T04:31:43.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: 2021-02-10T04:31:42.360327+0000 osd.4 (osd.4) 244 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:31:43.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: cluster 2021-02-10T04:31 2021-02-10T04:31:43.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: :42.403603+0000 mon.a (mon.0) 1027 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2021-02-10T04:31:43.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: cluster 2021-02-10T04:31:43.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: 2021-02- 2021-02-10T04:31:43.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: 10T04:31:42.406881 2021-02-10T04:31:43.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 245 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:43.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:31:43.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: 04:31:42.406886+0000 osd.4 (osd.4) 246 : 2021-02-10T04:31:43.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:43.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: cluster 2021-02-10T04:31:42.406894+0000 2021-02-10T04:31:43.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: osd.4 (osd.4) 247 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:43.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:31:43.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: 04:31:42.406900+0000 osd.4 (osd.4) 248 2021-02-10T04:31:43.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:43.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:31:43.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: :31:42.967505+0000 osd.2 (osd.2) 262 2021-02-10T04:31:43.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:43.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:43.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: -10T04:31:42. 2021-02-10T04:31:43.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: 967514+0000 osd.2 (osd.2) 2021-02-10T04:31:43.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: 263 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:43.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:43.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: -02-10T04:31:42 2021-02-10T04:31:43.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: .967520+0000 osd.2 (osd.2 2021-02-10T04:31:43.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: ) 264 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:43.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:43.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: -02-10T04:31:42.967526+0000 2021-02-10T04:31:43.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: osd.2 (osd.2) 265 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:43.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:43.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: 02-10T04:31:42.967532+0000 osd.2 (osd.2 2021-02-10T04:31:43.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: ) 266 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:43.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:43.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: 10T04:31:42.967538+0000 osd.2 (osd. 2021-02-10T04:31:43.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: 2) 267 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:43.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:43.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: -02-10T04:31:42.967545+0000 2021-02-10T04:31:43.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: osd.2 (osd.2) 268 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:43.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:43.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: 02-10T04:31:42.967551+0000 osd.2 ( 2021-02-10T04:31:43.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: osd.2) 269 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:43.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:43.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: 02-10T04:31:43.077416+0000 mon.a 2021-02-10T04:31:43.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: (mon.0) 1028 : cluster [WRN] 2021-02-10T04:31:43.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: Health check update: 18 slow ops, oldest one blocked for 147 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:31:43.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: cluster 2021-02-10T04:31:43.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: 2021-02 2021-02-10T04:31:43.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: -10T04:31:43 2021-02-10T04:31:43.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: .082927+0000 mon.a (mon.0 2021-02-10T04:31:43.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[29288]: ) 1029 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2021-02-10T04:31:43.652 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:43 smithi164 conmon[33677]: debug 2021-02-10T04:31:43.373+0000 7fbec19a1700 -1 osd.4 286 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:31:43.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: -10T04:31:42.967514+0000 osd.2 2021-02-10T04:31:43.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: (osd.2 2021-02-10T04:31:43.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: ) 263 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:43.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:43.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: -02-10T04:31:42.967520 2021-02-10T04:31:43.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 264 : cluster 2021-02-10T04:31:43.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:43.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: cluster 2021-02-10T04:31:43.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: 2021-02-10T04:31:42.967526+0000 osd.2 (osd. 2021-02-10T04:31:43.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: 2) 265 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:43.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:43.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: -10T04:31 2021-02-10T04:31:43.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: :42.967532+0000 osd.2 (osd.2) 266 : cluster 2021-02-10T04:31:43.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:43.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:43.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: -02-10T04:31:42.967538+0000 osd.2 (osd.2) 267 : cluster [WRN] 2021-02-10T04:31:43.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:43.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: cluster 2021-02-10T04:31: 2021-02-10T04:31:43.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: 42.967545+0000 osd.2 (osd.2) 268 : cluster [WRN] 2021-02-10T04:31:43.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:43.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: cluster 2021-02-10T04:31:43.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: 2021-02-10T04:31:42.967551+0000 osd.2 (osd 2021-02-10T04:31:43.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: .2) 269 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:43.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:43.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: 10T04:31:43.077416+0000 mon.a (mon.0 2021-02-10T04:31:43.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: ) 1028 : cluster [WRN] Health check update: 18 slow ops, oldest one blocked for 147 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:31:43.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:43.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: 10T04:31:43.082927+0000 mon.a (mon 2021-02-10T04:31:43.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[29319]: .0) 1029 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2021-02-10T04:31:43.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: 2) 264 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:43.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:43.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: 10T04:31:42.967526+0000 osd.2 (osd 2021-02-10T04:31:43.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: .2) 265 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:43.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:43.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: 10T04:31:42.967532+0000 2021-02-10T04:31:43.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: osd.2 (osd.2) 266 : 2021-02-10T04:31:43.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:43.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: cluster 2021-02-10T04:31:43.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:31:43.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: 31:42.967538+0000 osd.2 ( 2021-02-10T04:31:43.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: osd.2) 267 : cluster 2021-02-10T04:31:43.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:43.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:43.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: 02-10T04:31:42 2021-02-10T04:31:43.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: .967545+0000 osd.2 (osd.2) 2021-02-10T04:31:43.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: 268 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:43.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:43.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: :31:42.967551+0000 osd.2 (osd.2) 269 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:43.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:43.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: -02-10T04:31:43.077416+0000 mon.a (mon.0) 1028 : cluster [WRN] 2021-02-10T04:31:43.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: Health check update: 18 slow ops, oldest one blocked for 147 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:31:43.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: cluster 2021-02-10T04:31:43.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: 2021-02-10T04:31:43.082927+0000 mon.a (mon.0 2021-02-10T04:31:43.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[35961]: ) 1029 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2021-02-10T04:31:43.697 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:31:43.727 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:44.092 INFO:tasks.rados.rados.0.smithi085.stdout:986: done (15 left) 2021-02-10T04:31:44.092 INFO:tasks.rados.rados.0.smithi085.stdout:987: done (14 left) 2021-02-10T04:31:44.092 INFO:tasks.rados.rados.0.smithi085.stdout:988: done (13 left) 2021-02-10T04:31:44.092 INFO:tasks.rados.rados.0.smithi085.stdout:989: copy_from oid 3 from oid 22 current snap is 95 2021-02-10T04:31:44.093 INFO:tasks.rados.rados.0.smithi085.stdout:990: read oid 2 snap -1 2021-02-10T04:31:44.093 INFO:tasks.rados.rados.0.smithi085.stdout:990: expect (ObjNum 361 snap 92 seq_num 361) 2021-02-10T04:31:44.093 INFO:tasks.rados.rados.0.smithi085.stdout:991: snap_create 2021-02-10T04:31:44.095 INFO:tasks.rados.rados.0.smithi085.stdout:990: expect (ObjNum 361 snap 92 seq_num 361) 2021-02-10T04:31:44.108 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:43 smithi085 conmon[49106]: debug 2021-02-10T04:31:43.945+0000 7f77a3e94700 -1 osd.2 286 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:31:44.248 INFO:tasks.rados.rados.0.smithi085.stdout:984: finishing copy_from to smithi08575397-31 2021-02-10T04:31:44.249 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 455 (ObjNum 352 snap 90 seq_num 352) dirty exists 2021-02-10T04:31:44.249 INFO:tasks.rados.rados.0.smithi085.stdout:984: finishing copy_from racing read to smithi08575397-31 2021-02-10T04:31:44.413 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[45443]: debug 2021-02-10T04:31:44.107+0000 7f7140d73700 -1 osd.1 286 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:31:44.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: cluster 2021-02-10T04:31:43. 2021-02-10T04:31:44.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: 156736+0000 osd.1 (osd.1) 244 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:44.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:44.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: -10T04:31:43.156748+0000 osd.1 (osd.1) 245 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:44.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:44.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: 04:31:43. 2021-02-10T04:31:44.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: 156756+0000 osd.1 (osd.1) 246 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:44.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:44.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: 10T04:31: 2021-02-10T04:31:44.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: 43.156763+0000 osd.1 (osd.1) 247 : cluster 2021-02-10T04:31:44.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:44.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:44.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: 02-10T04:31:43.156769+0000 osd.1 ( 2021-02-10T04:31:44.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: osd.1) 248 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:44.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:44.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: 04:31:43.156776+0000 osd.1 (osd.1) 249 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:44.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: cluster 2021-02-10T04:31:43. 2021-02-10T04:31:44.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: 373784+0000 osd.4 (osd.4) 249 : cluster [WRN] 2021-02-10T04:31:44.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:44.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: cluster 2021-02-10T04:31:43 2021-02-10T04:31:44.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: .373790+0000 osd.4 (osd.4) 250 : cluster 2021-02-10T04:31:44.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:44.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:44.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: 04:31:43.373795+0000 osd.4 (osd.4 2021-02-10T04:31:44.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: ) 251 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:44.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:44.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: -10T04:31:43.373798+0000 osd.4 2021-02-10T04:31:44.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: (osd.4) 252 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:44.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: 2021-02-10T04:31:44.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:44.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: 31:43.946290+0000 osd.2 (osd.2 2021-02-10T04:31:44.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: ) 270 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:44.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: cluster 2021-02-10T04:31:43 2021-02-10T04:31:44.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: .946301+0000 osd.2 (osd.2) 271 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:44.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:44.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: -10T04:31:43.946306+0000 osd.2 (osd.2) 272 : cluster 2021-02-10T04:31:44.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:44.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:44.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: 04:31:43.946314+0000 osd.2 (osd.2) 273 : 2021-02-10T04:31:44.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:44.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:44.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: -02-10T04:31:43.946321+0000 osd.2 (osd.2) 274 : cluster [WRN] 2021-02-10T04:31:44.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:44.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:44.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: 10T04:31:43.946328+0000 osd.2 (osd.2) 275 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:44.418 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: 2021-02-10T04:31:44.418 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:44.418 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: -10T04:31:43.946334+0000 osd.2 (osd.2) 276 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:44.418 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:44.418 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: -02-10T04:31: 2021-02-10T04:31:44.418 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: 43.946345+0000 osd.2 (osd. 2021-02-10T04:31:44.418 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: 2) 277 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:44.418 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: 2021-02-10T04:31:44.418 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: cluster 2021-02-10T04:31:44.418 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: 2021-02-10T04:31 2021-02-10T04:31:44.418 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: :44.039897+0000 2021-02-10T04:31:44.418 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: mgr.y (mgr.14140) 451 : 2021-02-10T04:31:44.419 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: cluster [DBG] pgmap v579: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 136 MiB data, 486 MiB used, 715 GiB / 715 GiB avail; 4.3 MiB/s rd, 2.2 MiB/s wr, 14 op/s 2021-02-10T04:31:44.419 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: cluster 2021-02-10T04:31:44.419 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: 2021-02-10T04:31:44.092087+0000 2021-02-10T04:31:44.419 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: mon.a (mon.0) 1030 : cluster [DBG] 2021-02-10T04:31:44.419 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[35961]: osdmap e287: 8 total, 8 up, 8 in 2021-02-10T04:31:44.572 INFO:tasks.rados.rados.0.smithi085.stdout:989: finishing copy_from to smithi08575397-3 2021-02-10T04:31:44.572 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 150 (ObjNum 346 snap 88 seq_num 346) dirty exists 2021-02-10T04:31:44.572 INFO:tasks.rados.rados.0.smithi085.stdout:989: finishing copy_from racing read to smithi08575397-3 2021-02-10T04:31:44.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[33677]: debug 2021-02-10T04:31:44.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[33677]: 2021-02-10T04:31:44.392+0000 7fbec19a1700 -1 osd.4 287 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:31:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: cluster 2021-02-10T04:31 2021-02-10T04:31:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: :43.156736+0000 osd.1 (osd.1) 244 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: cluster 2021-02-10T04:31:43.156748+0000 osd.1 (osd.1) 245 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: cluster 2021-02-10T04:31:43.156756+0000 osd.1 ( 2021-02-10T04:31:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: osd.1) 246 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: cluster 2021-02-10T04:31:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: 2021-02- 2021-02-10T04:31:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: 10T04:31:43.156763+0000 osd.1 (osd. 2021-02-10T04:31:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: 1) 247 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: cluster 2021-02-10T04:31: 2021-02-10T04:31:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: 43.156769+0000 osd.1 (osd.1) 248 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: 2021-02-10T04:31:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: 02-10T04:31:43.156776+0000 2021-02-10T04:31:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: osd.1 (osd.1) 249 : cluster [WRN] 2021-02-10T04:31:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: cluster 2021-02-10T04:31:43.373784 2021-02-10T04:31:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 249 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:31:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: :31:43.373790+0000 osd.4 (osd.4) 250 : cluster [WRN] 2021-02-10T04:31:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: cluster 2021-02-10T04:31: 2021-02-10T04:31:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: 43.373795+0000 osd.4 (osd.4) 251 : 2021-02-10T04:31:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: 10T04:31:43.373798+0000 osd.4 ( 2021-02-10T04:31:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: osd.4) 252 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:31:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: 31:43.946290+0000 osd.2 (osd.2) 270 2021-02-10T04:31:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: 02-10T04:31:43.946301+0000 osd.2 ( 2021-02-10T04:31:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: osd.2) 271 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: 02-10T04:31:43.946306+0000 osd.2 ( 2021-02-10T04:31:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: osd.2) 272 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: cluster 2021-02-10T04:31:43.946314+0000 2021-02-10T04:31:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: osd.2 (osd.2) 273 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: 10T04:31:43.946321+0000 osd.2 (osd.2) 274 : cluster 2021-02-10T04:31:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: -10T04:31:43.946328+0000 osd.2 (osd.2) 275 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: T04:31:43.946334+0000 2021-02-10T04:31:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: osd.2 (osd.2) 276 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:44.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: 02-10T04:31:43.946345+0000 osd.2 (osd 2021-02-10T04:31:44.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: .2) 277 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:44.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:44.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: 02-10T04:31:44.039897+0000 mgr.y 2021-02-10T04:31:44.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: (mgr.14140) 451 : cluster [DBG] pgmap v579: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 136 MiB data, 486 MiB used, 715 GiB / 715 GiB avail; 4.3 MiB/s rd, 2.2 MiB/s wr, 14 op/s 2021-02-10T04:31:44.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:31:44.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: 04:31:44.092087+0000 mon.a (mon. 2021-02-10T04:31:44.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:44 smithi164 conmon[29288]: 0) 1030 : cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2021-02-10T04:31:44.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: cluster 2021-02-10T04:31:44.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: 2021-02-10T04:31:44.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: 2021 2021-02-10T04:31:44.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: - 2021-02-10T04:31:44.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: 02-10T04:31:43. 2021-02-10T04:31:44.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: 156736+0000 osd.1 (osd.1) 244 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:44.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:44.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: -02-10T04:31:43.156748+0000 2021-02-10T04:31:44.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: osd.1 2021-02-10T04:31:44.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: (osd.1) 245 2021-02-10T04:31:44.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:44.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:31:44.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: :31:43.156756+0000 osd.1 (osd.1) 2021-02-10T04:31:44.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: 246 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:44.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: cluster 2021-02-10T04:31:44.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: 2021-02-10T04:31 2021-02-10T04:31:44.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: :43.156763+0000 osd.1 (osd 2021-02-10T04:31:44.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: .1) 2021-02-10T04:31:44.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: 247 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:44.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: cluster 2021-02-10T04:31:44.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: 2021-02-10T04:31 2021-02-10T04:31:44.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: :43.156769 2021-02-10T04:31:44.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: +0000 osd.1 (osd.1) 2021-02-10T04:31:44.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: 248 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:44.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:44.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: 02-10T04:31:43. 2021-02-10T04:31:44.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: 156776+0000 osd.1 (osd.1 2021-02-10T04:31:44.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: ) 249 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:44.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: 2021-02-10T04:31:44.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:44.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: 10T04:31:43.373784+0000 osd.4 2021-02-10T04:31:44.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: (osd.4) 249 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:44.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:44.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: -02-10T04:31:43.373790+0000 2021-02-10T04:31:44.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: osd.4 (osd.4) 250 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:44.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: 2021-02-10T04:31:44.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: cluster 2021-02-10T04:31:44.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:31:44.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: :31:43.373795+0000 osd.4 (osd.4) 251 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:44.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: cluster 2021-02-10T04:31:43.373798+0000 osd.4 (osd. 2021-02-10T04:31:44.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: 4) 252 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:44.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:44.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: 10T04:31:43.946290+0000 osd.2 (osd.2) 270 : cluster 2021-02-10T04:31:44.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:44.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:44.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: -02-10T04:31:43.946301+0000 osd.2 (osd. 2021-02-10T04:31:44.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: 2) 271 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:44.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:44.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: -10T04:31:43.946306+0000 osd.2 2021-02-10T04:31:44.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: (osd.2) 272 : cluster 2021-02-10T04:31:44.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:44.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: cluster 2021-02-10T04:31 2021-02-10T04:31:44.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: :43.946314+0000 osd.2 (osd 2021-02-10T04:31:44.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: .2) 273 : cluster [WRN] 2021-02-10T04:31:44.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:44.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:44.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: 02-10T04:31:43.946321+0000 2021-02-10T04:31:44.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: osd.2 (osd.2) 274 : cluster 2021-02-10T04:31:44.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:44.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:31:44.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: :31:43.946328+0000 osd.2 ( 2021-02-10T04:31:44.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: osd.2) 275 : cluster [WRN] 2021-02-10T04:31:44.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:44.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:44.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: 10T04:31:43.946334+0000 2021-02-10T04:31:44.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: osd.2 (osd.2) 276 : cluster 2021-02-10T04:31:44.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:44.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:44.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: -10T04:31:43.946345+0000 osd.2 (osd 2021-02-10T04:31:44.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: .2) 277 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:44.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:44.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: 02-10T04:31:44.039897+0000 mgr.y (mgr. 2021-02-10T04:31:44.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: 14140) 451 : cluster [DBG] pgmap v579: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 136 MiB data, 486 MiB used, 715 GiB / 715 GiB avail; 4.3 MiB/s rd, 2.2 MiB/s wr, 14 op/s 2021-02-10T04:31:44.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:44.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: -02-10T04:31:44.092087+0000 2021-02-10T04:31:44.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: mon.a (mon.0) 1030 : cluster [DBG] 2021-02-10T04:31:44.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[29319]: osdmap e287: 8 total, 8 up, 8 in 2021-02-10T04:31:44.700 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[41540]: debug 2021-02-10T04:31:44.419+0000 7fb94c364700 -1 osd.0 pg_epoch: 287 pg[7.4( v 285'155 (0'0,285'155] local-lis/les=93/94 n=2 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [0,4,1] r=0 lpr=93 crt=285'155 lcod 285'153 mlcod 285'153 active+clean+snaptrim trimq=[b9~1] ps=45] removing snap head 2021-02-10T04:31:45.094 INFO:tasks.rados.rados.0.smithi085.stdout:984: done (15 left) 2021-02-10T04:31:45.094 INFO:tasks.rados.rados.0.smithi085.stdout:989: done (14 left) 2021-02-10T04:31:45.094 INFO:tasks.rados.rados.0.smithi085.stdout:990: done (13 left) 2021-02-10T04:31:45.094 INFO:tasks.rados.rados.0.smithi085.stdout:991: done (12 left) 2021-02-10T04:31:45.095 INFO:tasks.rados.rados.0.smithi085.stdout:992: rollback oid 30 current snap is 96 2021-02-10T04:31:45.095 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 30 to 94 2021-02-10T04:31:45.095 INFO:tasks.rados.rados.0.smithi085.stdout:993: snap_remove snap 95 2021-02-10T04:31:45.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:44 smithi085 conmon[49106]: debug 2021-02-10T04:31:44.912+0000 7f77a3e94700 -1 osd.2 287 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:31:45.191 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[45443]: debug 2021-02-10T04:31:45.076+0000 7f7140d73700 -1 osd.1 287 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:31:45.668 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:31:45.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: cluster 2021-02-10T04:31:44.108285 2021-02-10T04:31:45.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 250 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:45.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:45.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: T04:31:44.108307+0000 osd.1 (osd. 2021-02-10T04:31:45.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: 1) 251 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:45.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: cluster 2021-02-10T04:31: 2021-02-10T04:31:45.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: 44.108314+0000 osd.1 (osd.1) 252 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:45.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: cluster 2021-02-10T04:31 2021-02-10T04:31:45.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: :44.108320+0000 osd.1 (osd.1) 253 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:45.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:45.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: 31:44.108326+0000 osd.1 (osd.1) 254 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:45.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: 2021-02-10T04:31:45.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:45.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: 31:44.108331+0000 osd.1 (osd.1) 255 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:45.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: cluster 2021-02-10T04:31:44.393245+0000 2021-02-10T04:31:45.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: osd.4 (osd.4) 253 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:45.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: cluster 2021-02-10T04:31:44.393253+0000 2021-02-10T04:31:45.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: osd.4 (osd.4) 254 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:45.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: cluster 2021-02-10T04:31:44.393257+0000 2021-02-10T04:31:45.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: osd.4 (osd.4) 255 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:45.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: cluster 2021-02-10T04:31:44.393261+0000 2021-02-10T04:31:45.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: osd.4 (osd.4) 256 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:45.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: cluster 2021-02-10T04:31:44.912712+0000 2021-02-10T04:31:45.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: osd.2 (osd.2) 278 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:45.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: cluster 2021-02-10T04:31:44.912720 2021-02-10T04:31:45.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 279 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:45.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: cluster 2021-02-10T04:31:44.912740 2021-02-10T04:31:45.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 280 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:45.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: cluster 2021-02-10T04:31:44 2021-02-10T04:31:45.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: .912745+0000 osd.2 (osd.2) 281 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:45.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: cluster 2021-02-10T04:31:44.912749 2021-02-10T04:31:45.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 282 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:45.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: cluster 2021-02-10T04:31:44.912752+0000 2021-02-10T04:31:45.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: osd.2 (osd.2) 283 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:45.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: cluster 2021-02-10T04:31:44.912757+0000 osd.2 (osd 2021-02-10T04:31:45.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: .2) 284 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:45.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:31:45.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: -02-10T04:31:44.912767+0000 osd.2 (osd.2) 285 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:45.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: cluster 2021-02-10T04:31:45.092803 2021-02-10T04:31:45.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1031 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2021-02-10T04:31:45.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: cluster 2021-02-10T04:31:45.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:31:45.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: 31:44.108285+0000 osd.1 (osd.1) 250 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:45.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:45.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: -02-10T04:31:44.108307+0000 2021-02-10T04:31:45.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: osd.1 (osd.1) 251 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:45.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: 2021-02-10T04:31:45.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: cluster 2021-02-10T04:31 2021-02-10T04:31:45.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: :44.108314+0000 osd.1 (osd.1) 2021-02-10T04:31:45.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: 252 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:45.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:45.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: 10T04:31:44.108320+0000 osd.1 ( 2021-02-10T04:31:45.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: osd.1) 253 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:45.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:45.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: 02-10T04:31:44.108326+0000 2021-02-10T04:31:45.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: osd.1 (osd.1) 254 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:45.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:31:45.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: :31:44.108331+0000 osd.1 (osd.1) 255 : cluster 2021-02-10T04:31:45.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:45.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:45.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: 10T04:31:44.393245+0000 osd.4 (osd.4) 253 : 2021-02-10T04:31:45.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:45.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:31:45.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: 04:31:44.393253+0000 osd.4 (osd. 2021-02-10T04:31:45.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: 4) 254 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:45.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:31:45.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: 02-10T04:31:44.393257+0000 2021-02-10T04:31:45.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: osd.4 (osd.4) 255 : cluster [WRN] 2021-02-10T04:31:45.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:45.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:45.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: T04:31:44.393261+0000 2021-02-10T04:31:45.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: osd.4 (osd.4) 256 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:45.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: 2021-02-10T04:31:45.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: cluster 2021-02-10T04:31:45.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: 2021-02-10T04:31:44.912712+0000 osd.2 (osd. 2021-02-10T04:31:45.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: 2) 278 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:45.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:45.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: T04:31:44.912720+0000 osd.2 (osd. 2021-02-10T04:31:45.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: 2) 279 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:45.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:45.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: 10T04:31:44.912740+0000 osd.2 (osd.2) 2021-02-10T04:31:45.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: 280 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:45.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: cluster 2021-02-10T04:31:44 2021-02-10T04:31:45.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: .912745+0000 osd.2 (osd.2) 281 : cluster [WRN] 2021-02-10T04:31:45.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:45.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:45.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: T04:31:44.912749+0000 osd.2 (osd.2) 282 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:45.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: 2021-02-10T04:31:45.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:31:45.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: 31:44.912752+0000 osd.2 (osd.2) 283 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:45.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:31:45.712 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: :31:44.912757+0000 osd.2 (osd.2) 284 : cluster 2021-02-10T04:31:45.712 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:45.712 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:31:45.712 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: -02-10T04:31:44.912767+0000 osd.2 (osd.2) 285 : cluster 2021-02-10T04:31:45.712 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:45.713 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: cluster 2021-02-10T04:31:45.092803+0000 mon.a 2021-02-10T04:31:45.713 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[29319]: (mon.0) 1031 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2021-02-10T04:31:45.714 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[41540]: debug 2021-02-10T04:31:45.714 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[41540]: 2021-02-10T04:31:45.633+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:45.852 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:31:49.166 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight 2021-02-10T04:31:49.166 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:45 smithi085 conmon[49106]: debug 2021-02-10T04:31:45.936+0000 7f77a3e94700 -1 osd.2 288 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:31:49.166 INFO:tasks.rados.rados.0.smithi085.stdout:993: done (13 left) 2021-02-10T04:31:49.167 INFO:tasks.rados.rados.0.smithi085.stdout:994: rollback oid 40 current snap is 96 2021-02-10T04:31:49.167 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 40 to 94 2021-02-10T04:31:49.167 INFO:tasks.rados.rados.0.smithi085.stdout:992: finishing rollback tid 0 to smithi08575397-30 2021-02-10T04:31:49.167 INFO:tasks.rados.rados.0.smithi085.stdout:992: finishing rollback tid 1 to smithi08575397-30 2021-02-10T04:31:49.167 INFO:tasks.rados.rados.0.smithi085.stdout:992: finishing rollback tid 2 to smithi08575397-30 2021-02-10T04:31:49.167 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 318 (ObjNum 342 snap 87 seq_num 342) dirty exists 2021-02-10T04:31:49.167 INFO:tasks.rados.rados.0.smithi085.stdout:992: done (13 left) 2021-02-10T04:31:49.167 INFO:tasks.rados.rados.0.smithi085.stdout:995: write (excl) oid 3 current snap is 96 2021-02-10T04:31:49.167 INFO:tasks.rados.rados.0.smithi085.stdout:995: seq_num 371 ranges {635397=702323,1893643=341754} 2021-02-10T04:31:49.167 INFO:tasks.rados.rados.0.smithi085.stdout:995: writing smithi08575397-3 from 635397 to 1337720 tid 1 2021-02-10T04:31:49.167 INFO:tasks.rados.rados.0.smithi085.stdout:995: writing smithi08575397-3 from 1893643 to 2235397 tid 2 2021-02-10T04:31:49.167 INFO:tasks.rados.rados.0.smithi085.stdout:994: finishing rollback tid 0 to smithi08575397-40 2021-02-10T04:31:49.168 INFO:tasks.rados.rados.0.smithi085.stdout:994: finishing rollback tid 1 to smithi08575397-40 2021-02-10T04:31:49.168 INFO:tasks.rados.rados.0.smithi085.stdout:994: finishing rollback tid 2 to smithi08575397-40 2021-02-10T04:31:49.168 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 520 (ObjNum 365 snap 93 seq_num 365) dirty exists 2021-02-10T04:31:49.168 INFO:tasks.rados.rados.0.smithi085.stdout:994: done (13 left) 2021-02-10T04:31:49.168 INFO:tasks.rados.rados.0.smithi085.stdout:996: write (excl) oid 30 current snap is 96 2021-02-10T04:31:49.168 INFO:tasks.rados.rados.0.smithi085.stdout:996: seq_num 372 ranges {652204=777572,1952125=300079} 2021-02-10T04:31:49.168 INFO:tasks.rados.rados.0.smithi085.stdout:996: writing smithi08575397-30 from 652204 to 1429776 tid 1 2021-02-10T04:31:49.168 INFO:tasks.rados.rados.0.smithi085.stdout:996: writing smithi08575397-30 from 1952125 to 2252204 tid 2 2021-02-10T04:31:49.168 INFO:tasks.rados.rados.0.smithi085.stdout:997: write oid 2 current snap is 96 2021-02-10T04:31:49.168 INFO:tasks.rados.rados.0.smithi085.stdout:995: finishing write tid 1 to smithi08575397-3 2021-02-10T04:31:49.168 INFO:tasks.rados.rados.0.smithi085.stdout:995: finishing write tid 2 to smithi08575397-3 2021-02-10T04:31:49.168 INFO:tasks.rados.rados.0.smithi085.stdout:995: finishing write tid 3 to smithi08575397-3 2021-02-10T04:31:49.168 INFO:tasks.rados.rados.0.smithi085.stdout:995: finishing write tid 4 to smithi08575397-3 2021-02-10T04:31:49.168 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 458 (ObjNum 371 snap 96 seq_num 371) dirty exists 2021-02-10T04:31:49.169 INFO:tasks.rados.rados.0.smithi085.stdout:995: left oid 3 (ObjNum 371 snap 96 seq_num 371) 2021-02-10T04:31:49.169 INFO:tasks.rados.rados.0.smithi085.stdout:997: seq_num 373 ranges {669011=569174,2010607=258404} 2021-02-10T04:31:49.169 INFO:tasks.rados.rados.0.smithi085.stdout:997: writing smithi08575397-2 from 669011 to 1238185 tid 1 2021-02-10T04:31:49.169 INFO:tasks.rados.rados.0.smithi085.stdout:997: writing smithi08575397-2 from 2010607 to 2269011 tid 2 2021-02-10T04:31:49.169 INFO:tasks.rados.rados.0.smithi085.stdout:995: done (14 left) 2021-02-10T04:31:49.169 INFO:tasks.rados.rados.0.smithi085.stdout:998: snap_remove snap 93 2021-02-10T04:31:49.169 INFO:tasks.rados.rados.0.smithi085.stdout:996: finishing write tid 1 to smithi08575397-30 2021-02-10T04:31:49.169 INFO:tasks.rados.rados.0.smithi085.stdout:996: finishing write tid 2 to smithi08575397-30 2021-02-10T04:31:49.169 INFO:tasks.rados.rados.0.smithi085.stdout:996: finishing write tid 3 to smithi08575397-30 2021-02-10T04:31:49.169 INFO:tasks.rados.rados.0.smithi085.stdout:996: finishing write tid 4 to smithi08575397-30 2021-02-10T04:31:49.169 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 323 (ObjNum 372 snap 96 seq_num 372) dirty exists 2021-02-10T04:31:49.169 INFO:tasks.rados.rados.0.smithi085.stdout:996: left oid 30 (ObjNum 372 snap 96 seq_num 372) 2021-02-10T04:31:49.169 INFO:tasks.rados.rados.0.smithi085.stdout:997: finishing write tid 1 to smithi08575397-2 2021-02-10T04:31:49.170 INFO:tasks.rados.rados.0.smithi085.stdout:997: finishing write tid 2 to smithi08575397-2 2021-02-10T04:31:49.170 INFO:tasks.rados.rados.0.smithi085.stdout:997: finishing write tid 3 to smithi08575397-2 2021-02-10T04:31:49.170 INFO:tasks.rados.rados.0.smithi085.stdout:997: finishing write tid 4 to smithi08575397-2 2021-02-10T04:31:49.170 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 290 (ObjNum 373 snap 96 seq_num 373) dirty exists 2021-02-10T04:31:49.170 INFO:tasks.rados.rados.0.smithi085.stdout:997: left oid 2 (ObjNum 373 snap 96 seq_num 373) 2021-02-10T04:31:49.170 INFO:tasks.rados.rados.0.smithi085.stdout:996: done (14 left) 2021-02-10T04:31:49.170 INFO:tasks.rados.rados.0.smithi085.stdout:997: done (13 left) 2021-02-10T04:31:49.170 INFO:tasks.rados.rados.0.smithi085.stdout:998: done (12 left) 2021-02-10T04:31:49.170 INFO:tasks.rados.rados.0.smithi085.stdout:999: delete oid 8 current snap is 96 2021-02-10T04:31:49.170 INFO:tasks.rados.rados.0.smithi085.stdout:999: done (12 left) 2021-02-10T04:31:49.170 INFO:tasks.rados.rados.0.smithi085.stdout:1000: snap_remove snap 94 2021-02-10T04:31:49.171 INFO:tasks.rados.rados.0.smithi085.stdout:1000: done (12 left) 2021-02-10T04:31:49.171 INFO:tasks.rados.rados.0.smithi085.stdout:1001: snap_create 2021-02-10T04:31:49.171 INFO:tasks.rados.rados.0.smithi085.stdout:1001: done (12 left) 2021-02-10T04:31:49.171 INFO:tasks.rados.rados.0.smithi085.stdout:1002: delete oid 23 current snap is 97 2021-02-10T04:31:49.171 INFO:tasks.rados.rados.0.smithi085.stdout:1002: done (12 left) 2021-02-10T04:31:49.171 INFO:tasks.rados.rados.0.smithi085.stdout:1003: delete oid 36 current snap is 97 2021-02-10T04:31:49.171 INFO:tasks.rados.rados.0.smithi085.stdout:1003: done (12 left) 2021-02-10T04:31:49.171 INFO:tasks.rados.rados.0.smithi085.stdout:1004: snap_remove snap 96 2021-02-10T04:31:49.171 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[45443]: debug 2021-02-10T04:31:46.049+0000 7f7140d73700 -1 osd.1 288 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:31:49.171 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:31:49.172 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:31:49.172 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:31:49.172 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:31:49.172 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:31:49.172 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:31:49.172 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:31:49.172 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:31:49.172 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[29319]: cluster 2021-02-10T04:31:45.076831+0000 osd.1 (osd.1) 256 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:49.173 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[29319]: cluster 2021-02-10T04:31:45.076843+0000 osd.1 (osd.1) 257 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:49.173 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[29319]: cluster 2021-02-10T04:31:45.076862+0000 osd.1 (osd.1) 258 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:49.173 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[29319]: cluster 2021-02-10T04:31:45.076868+0000 osd.1 (osd.1) 259 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:49.173 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[29319]: cluster 2021-02-10T04:31:45.076871+0000 osd.1 (osd.1) 260 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:49.173 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[29319]: cluster 2021-02-10T04:31:45.076880+0000 osd.1 (osd.1) 261 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:49.173 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[29319]: cluster 2021-02-10T04:31:45.428903+0000 osd.4 (osd.4) 257 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:49.173 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[29319]: cluster 2021-02-10T04:31:45.428912+0000 osd.4 (osd.4) 258 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:49.173 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[29319]: cluster 2021-02-10T04:31:45.428915+0000 osd.4 (osd.4) 259 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:49.173 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[29319]: cluster 2021-02-10T04:31:45.428918+0000 osd.4 (osd.4) 260 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:49.173 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[29319]: cluster 2021-02-10T04:31:45.937354+0000 osd.2 (osd.2) 286 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:49.173 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[29319]: cluster 2021-02-10T04:31:45.937360+0000 osd.2 (osd.2 2021-02-10T04:31:49.173 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[29319]: ) 287 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:49.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[29319]: cluster 2021-02-10T04:31:45.937367+0000 osd.2 (osd.2) 288 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:49.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[29319]: cluster 2021-02-10T04:31:45.937371+0000 osd.2 (osd.2) 289 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:49.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[29319]: cluster 2021-02-10T04:31:45.937375+0000 osd.2 (osd.2) 290 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:49.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[29319]: cluster 2021-02-10T04:31:45.937379+0000 osd.2 (osd.2) 291 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:49.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[29319]: cluster 2021-02-10T04:31:45.937383+0000 osd.2 (osd.2) 292 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:49.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[29319]: cluster 2021-02-10T04:31:45.937386+0000 osd.2 (osd.2) 293 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:49.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[29319]: cluster 2021-02-10T04:31:46.043727+0000 mgr.y (mgr.14140) 452 : cluster [DBG] pgmap v582: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 164 active+clean; 117 MiB data, 466 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 2.5 MiB/s wr, 17 op/s 2021-02-10T04:31:49.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[29319]: cluster 2021-02-10T04:31:46.096036+0000 mon.a (mon.0) 1032 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2021-02-10T04:31:49.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[29319]: cluster 2021-02-10T04:31:46.049783+0000 osd.1 (osd. 2021-02-10T04:31:49.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[29319]: 1) 262 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:49.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[29319]: cluster 2021-02-10T04:31:46.049792+0000 osd.1 (osd.1) 263 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:49.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[29319]: cluster 2021-02-10T04:31:46.049798+0000 osd.1 (osd.1) 264 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:49.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[29319]: cluster 2021-02-10T04:31:46.049803+0000 osd.1 (osd.1) 265 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:49.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:31:49.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[29319]: 04:31:46.049809+0000 osd.1 (osd.1) 266 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:49.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:31:49.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[29319]: :31:46.049814+0000 osd.1 (osd.1) 267 : cluster [WRN] 2021-02-10T04:31:49.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[29319]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:49.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[29319]: cluster 2021-02-10T04:31: 2021-02-10T04:31:49.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[29319]: 46.390542+0000 osd.4 (osd.4) 261 : cluster 2021-02-10T04:31:49.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:49.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:31:49.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[29319]: T04:31:46.390553+0000 osd.4 (osd. 2021-02-10T04:31:49.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[29319]: 4) 262 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:49.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[29319]: cluster 2021-02-10T04:31:49.176 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:31:49.176 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[29319]: 10T04:31:46.390561+0000 osd.4 ( 2021-02-10T04:31:49.176 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[29319]: osd.4) 263 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:49.176 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[29319]: cluster 2021-02-10T04:31:49.176 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[29319]: 2021-02-10T04:31:46.390569+0000 2021-02-10T04:31:49.176 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[29319]: osd.4 (osd.4) 264 : cluster [WRN] 2021-02-10T04:31:49.176 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:49.176 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[29319]: audit 2021-02-10T04:31: 2021-02-10T04:31:49.176 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[29319]: 46.489416+0000 mon.c (mon.1) 48 2021-02-10T04:31:49.176 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[29319]: : audit [DBG] from='client.? 172.21.15.85:0/3050165941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:31:49.176 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:31:49.176 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[29319]: -10T04:31:47.098903+0000 mon.a 2021-02-10T04:31:49.176 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[29319]: (mon.0) 1033 : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2021-02-10T04:31:49.176 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[29319]: cluster 2021-02-10T04:31:46.761924+0000 osd.1 (osd.1) 268 : cluster 2021-02-10T04:31:49.177 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[29319]: [DBG] purged_snaps scrub starts 2021-02-10T04:31:49.177 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[29319]: cluster 2021-02-10T04:31:46.762166+0000 osd.1 (osd.1) 269 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:31:49.177 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[29319]: audit 2021-02-10T04:31:47.865019+0000 mgr.y (mgr.14140) 453 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:31:49.177 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[29319]: cluster 2021-02-10T04:31 2021-02-10T04:31:49.177 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[29319]: :48.044325+0000 mgr.y (mgr.14140) 454 : cluster [DBG] pgmap v585: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 164 active+clean; 117 MiB data, 448 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 2.3 MiB/s wr, 15 op/s 2021-02-10T04:31:49.177 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:31:49.177 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[29319]: 10T04:31:48.079306+0000 mon.a (mon 2021-02-10T04:31:49.177 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[29319]: .0) 1034 : cluster [WRN] Health check update: 18 slow ops, oldest one blocked for 152 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:31:49.177 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[29319]: cluster 2021-02-10T04:31:49.177 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[29319]: 2021- 2021-02-10T04:31:49.177 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[29319]: 02-10T04:31:48.102104+0000 mon.a (mon.0) 1035 : cluster 2021-02-10T04:31:49.177 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[29319]: [DBG] osdmap e291: 8 total, 8 up, 8 in 2021-02-10T04:31:49.177 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: cluster 2021-02-10T04:31:49.178 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2021-02-10T04:31: 2021-02-10T04:31:49.178 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 45.076831+0000 osd.1 (osd.1) 256 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:49.178 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: cluster 2021-02-10T04:31 2021-02-10T04:31:49.178 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: :45.076843+0000 osd.1 (osd.1) 257 : cluster 2021-02-10T04:31:49.178 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:49.178 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: cluster 2021-02-10T04:31: 2021-02-10T04:31:49.178 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 45.076862+0000 osd.1 (osd.1) 258 : cluster 2021-02-10T04:31:49.178 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:49.178 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:49.178 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: :31:45.076868+0000 osd.1 (osd.1) 2021-02-10T04:31:49.178 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 259 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:49.178 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:49.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 04:31:45.076871+0000 osd.1 (osd.1) 2021-02-10T04:31:49.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 260 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:49.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: cluster 2021-02-10T04:31 2021-02-10T04:31:49.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: :45.076880+0000 osd.1 (osd.1) 261 : cluster 2021-02-10T04:31:49.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:49.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: cluster 2021-02-10T04:31:45. 2021-02-10T04:31:49.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 428903+0000 osd.4 (osd.4) 257 : cluster [WRN] 2021-02-10T04:31:49.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:49.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: cluster 2021-02-10T04:31:49.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2021-02-10T04:31:45.428912+0000 osd.4 ( 2021-02-10T04:31:49.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: osd.4) 258 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:49.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:49.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: T04:31:45.428915+0000 osd.4 (osd. 2021-02-10T04:31:49.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 4) 259 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:49.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: cluster 2021-02-10T04:31:49.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:31:49.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: T04:31:45.428918+0000 osd.4 (osd.4) 260 : cluster [WRN] 2021-02-10T04:31:49.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:49.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:31:49.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 31:45.937354+0000 osd.2 2021-02-10T04:31:49.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: ( 2021-02-10T04:31:49.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: osd 2021-02-10T04:31:49.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: . 2021-02-10T04:31:49.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2 2021-02-10T04:31:49.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: ) 2021-02-10T04:31:49.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 286 2021-02-10T04:31:49.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: : 2021-02-10T04:31:49.181 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: cluster 2021-02-10T04:31:49.181 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2021-02-10T04:31:49.181 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: [WRN] 2021-02-10T04:31:49.181 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2021-02-10T04:31:49.181 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:49.181 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2021-02-10T04:31:49.181 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: cluster 2021-02-10T04:31:49.181 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2021-02-10T04:31:49.181 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2021 2021-02-10T04:31:49.181 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: - 2021-02-10T04:31:49.181 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 02 2021-02-10T04:31:49.181 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: - 2021-02-10T04:31:49.181 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 10 2021-02-10T04:31:49.182 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: T 2021-02-10T04:31:49.182 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 04 2021-02-10T04:31:49.182 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: : 2021-02-10T04:31:49.182 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 31 2021-02-10T04:31:49.182 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: : 2021-02-10T04:31:49.182 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 45 2021-02-10T04:31:49.182 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: . 2021-02-10T04:31:49.182 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 937360 2021-02-10T04:31:49.182 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: +0000 2021-02-10T04:31:49.182 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2021-02-10T04:31:49.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: osd.2 2021-02-10T04:31:49.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: ( 2021-02-10T04:31:49.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: osd 2021-02-10T04:31:49.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: . 2021-02-10T04:31:49.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2 2021-02-10T04:31:49.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: ) 2021-02-10T04:31:49.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 287 2021-02-10T04:31:49.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: : 2021-02-10T04:31:49.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: cluster 2021-02-10T04:31:49.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2021-02-10T04:31:49.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: [WRN] 2021-02-10T04:31:49.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2021-02-10T04:31:49.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:49.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2021-02-10T04:31:49.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: cluster 2021-02-10T04:31:49.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2021-02-10T04:31:49.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2021 2021-02-10T04:31:49.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: - 2021-02-10T04:31:49.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 02 2021-02-10T04:31:49.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: - 2021-02-10T04:31:49.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 10 2021-02-10T04:31:49.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: T 2021-02-10T04:31:49.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 04 2021-02-10T04:31:49.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: : 2021-02-10T04:31:49.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 31 2021-02-10T04:31:49.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: : 2021-02-10T04:31:49.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 45 2021-02-10T04:31:49.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: . 2021-02-10T04:31:49.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 937367 2021-02-10T04:31:49.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: +0000 2021-02-10T04:31:49.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2021-02-10T04:31:49.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: osd.2 2021-02-10T04:31:49.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: ( 2021-02-10T04:31:49.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: osd 2021-02-10T04:31:49.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: . 2021-02-10T04:31:49.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2 2021-02-10T04:31:49.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: ) 2021-02-10T04:31:49.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 288 2021-02-10T04:31:49.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: : 2021-02-10T04:31:49.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: cluster 2021-02-10T04:31:49.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2021-02-10T04:31:49.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: [WRN] 2021-02-10T04:31:49.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2021-02-10T04:31:49.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:49.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2021-02-10T04:31:49.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: cluster 2021-02-10T04:31:49.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2021-02-10T04:31:49.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2021 2021-02-10T04:31:49.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: - 2021-02-10T04:31:49.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 02 2021-02-10T04:31:49.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: - 2021-02-10T04:31:49.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 10 2021-02-10T04:31:49.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: T 2021-02-10T04:31:49.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 04 2021-02-10T04:31:49.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: : 2021-02-10T04:31:49.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 31 2021-02-10T04:31:49.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: : 2021-02-10T04:31:49.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 45 2021-02-10T04:31:49.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: . 2021-02-10T04:31:49.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 937371 2021-02-10T04:31:49.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: +0000 2021-02-10T04:31:49.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2021-02-10T04:31:49.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: osd.2 2021-02-10T04:31:49.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: ( 2021-02-10T04:31:49.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: osd 2021-02-10T04:31:49.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: . 2021-02-10T04:31:49.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2 2021-02-10T04:31:49.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: ) 2021-02-10T04:31:49.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 289 2021-02-10T04:31:49.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: : 2021-02-10T04:31:49.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: cluster 2021-02-10T04:31:49.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2021-02-10T04:31:49.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: [WRN] 2021-02-10T04:31:49.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2021-02-10T04:31:49.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:49.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2021-02-10T04:31:49.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: cluster 2021-02-10T04:31:49.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2021-02-10T04:31:49.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2021 2021-02-10T04:31:49.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: - 2021-02-10T04:31:49.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 02 2021-02-10T04:31:49.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: -10T04:31:45 2021-02-10T04:31:49.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: .937375+0000 osd.2 (osd.2) 290 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:49.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:49.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 02-10T04:31:45. 2021-02-10T04:31:49.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 937379+0000 osd.2 (osd.2 2021-02-10T04:31:49.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: ) 291 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:49.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2021-02-10T04:31:49.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:49.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: :31:45.937383+0000 osd.2 (osd.2) 292 2021-02-10T04:31:49.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:49.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:49.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: T04:31:45.937386+0000 osd.2 (osd. 2021-02-10T04:31:49.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: 2) 293 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:49.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:31:49.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: T04:31:46.043727+0000 mgr.y (mgr.14140 2021-02-10T04:31:49.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: ) 452 : cluster [DBG] pgmap v582: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 164 active+clean; 117 MiB data, 466 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 2.5 MiB/s wr, 17 op/s 2021-02-10T04:31:49.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:49.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: -10T04:31:46.096036+0000 mon.a 2021-02-10T04:31:49.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:46 smithi085 conmon[35961]: (mon.0) 1032 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2021-02-10T04:31:49.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: cluster 2021-02-10T04:31:46.049783+0000 2021-02-10T04:31:49.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: osd.1 (osd.1) 262 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:49.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: cluster 2021-02-10T04:31:46.049792+0000 osd.1 (osd.1) 263 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:49.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: cluster 2021-02-10T04:31:49.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: 2021-02-10T04:31:46.049798+0000 osd.1 (osd.1) 264 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:49.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: 2021-02-10T04:31:49.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: cluster 2021-02-10T04:31:46.049803+0000 osd.1 (osd. 2021-02-10T04:31:49.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: 1) 265 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:49.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: cluster 2021-02-10T04:31:49.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: 2021-02-10T04:31:46.049809+0000 osd.1 ( 2021-02-10T04:31:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: osd.1) 266 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: 10T04:31:46.049814+0000 osd.1 (osd 2021-02-10T04:31:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: .1) 267 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: -10T04:31:46.390542+0000 osd.4 2021-02-10T04:31:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: (osd.4) 261 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: 02-10T04:31:46.390553+0000 2021-02-10T04:31:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: osd.4 (osd.4) 262 : cluster [WRN] 2021-02-10T04:31:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: cluster 2021-02-10T04:31:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: 2021-02-10T04:31:46.390561+0000 osd.4 ( 2021-02-10T04:31:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: osd.4) 263 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:49.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:49.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: 10T04:31:46.390569+0000 osd.4 ( 2021-02-10T04:31:49.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: osd.4) 264 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:49.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:31:49.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: -10T04:31:46.489416+0000 2021-02-10T04:31:49.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: mon.c (mon.1) 48 : audit [DBG] 2021-02-10T04:31:49.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/3050165941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:31:49.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: cluster 2021-02-10T04:31 2021-02-10T04:31:49.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: :47.098903+0000 mon.a (mon.0 2021-02-10T04:31:49.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:47 smithi085 conmon[35961]: ) 1033 : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2021-02-10T04:31:49.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[35961]: cluster 2021-02-10T04:31:46.761924+0000 osd.1 ( 2021-02-10T04:31:49.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[35961]: osd.1) 268 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:31:49.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:31:49.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[35961]: 04:31:46.762166+0000 osd.1 (osd.1) 269 : cluster [DBG] 2021-02-10T04:31:49.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:31:49.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[35961]: audit 2021 2021-02-10T04:31:49.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[35961]: -02-10T04:31:47.865019 2021-02-10T04:31:49.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 453 : audit 2021-02-10T04:31:49.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[35961]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:31:49.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[35961]: cluster 2021-02-10T04:31:49.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[35961]: 2021-02-10T04:31:48. 2021-02-10T04:31:49.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[35961]: 044325+0000 mgr.y (mgr.14140) 454 : cluster 2021-02-10T04:31:49.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[35961]: [DBG] pgmap v585: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 164 active+clean; 117 MiB data, 448 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 2.3 MiB/s wr, 15 op/s 2021-02-10T04:31:49.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[35961]: cluster 2021-02-10T04:31:49.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[35961]: 2021-02-10T04:31:48. 2021-02-10T04:31:49.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[35961]: 079306+0000 mon.a (mon.0) 1034 : cluster 2021-02-10T04:31:49.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[35961]: [WRN] Health check update: 18 slow ops, oldest one blocked for 152 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:31:49.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[35961]: cluster 2021-02-10T04:31:49.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:31:49.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[35961]: 31:48.102104+0000 mon.a (mon.0) 2021-02-10T04:31:49.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:48 smithi085 conmon[35961]: 1035 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2021-02-10T04:31:49.195 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:31:49.195 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:31:49.195 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:31:49.195 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:31:49.195 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:31:49.195 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:31:49.195 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:31:49.196 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:31:49.197 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[33677]: debug 2021-02-10T04:31:45.428+0000 7fbec19a1700 -1 osd.4 288 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:31:49.197 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[33677]: debug 2021-02-10T04:31:46.389+0000 7fbec19a1700 -1 osd.4 289 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:31:49.197 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: cluster 2021-02-10T04:31:44.108285+0000 osd.1 (osd. 2021-02-10T04:31:49.198 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: 1) 250 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:49.198 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: cluster 2021-02-10T04:31:44.108307+0000 osd.1 (osd.1) 251 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:49.198 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: 2021-02-10T04:31:49.198 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: cluster 2021-02-10T04:31:44.108314+0000 osd.1 (osd 2021-02-10T04:31:49.198 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: .1) 252 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:49.198 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: cluster 2021-02-10T04:31:49.198 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: 2021-02-10T04:31:44.108320+0000 2021-02-10T04:31:49.198 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: osd.1 (osd.1) 253 : cluster 2021-02-10T04:31:49.198 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:49.198 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: cluster 2021-02-10T04:31:49.198 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: 2021-02-10T04:31: 2021-02-10T04:31:49.198 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: 44.108326+0000 osd.1 (osd. 2021-02-10T04:31:49.198 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: 1) 254 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:49.199 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:49.199 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: 10T04:31:44.108331+0000 osd.1 (osd.1) 255 : 2021-02-10T04:31:49.199 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:49.199 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: cluster 2021-02-10T04:31:44.393245+0000 osd.4 ( 2021-02-10T04:31:49.199 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: osd.4) 253 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:49.199 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: cluster 2021-02-10T04:31: 2021-02-10T04:31:49.199 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: 44.393253+0000 osd.4 (osd.4) 254 : cluster [WRN] 2021-02-10T04:31:49.199 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:49.199 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:49.199 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: -02-10T04:31:44.393257+0000 osd.4 (osd.4) 255 : cluster [WRN] 2021-02-10T04:31:49.199 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:49.199 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: cluster 2021-02-10T04:31:49.200 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: 2021-02-10T04:31:44.393261+0000 osd.4 (osd.4) 256 : cluster [WRN] 2021-02-10T04:31:49.200 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:49.200 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: cluster 2021-02-10T04:31:49.200 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: 2021-02-10T04:31:44.912712+0000 osd.2 (osd.2) 278 : cluster 2021-02-10T04:31:49.200 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:49.200 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: cluster 2021-02-10T04:31:44.912720+0000 osd.2 (osd 2021-02-10T04:31:49.200 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: .2) 279 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:49.200 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: cluster 2021-02-10T04:31:44 2021-02-10T04:31:49.200 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: .912740+0000 osd.2 (osd.2) 280 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:49.200 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: cluster 2021-02-10T04:31:44.912745+0000 osd.2 (osd.2 2021-02-10T04:31:49.201 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: ) 281 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:49.201 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: cluster 2021-02-10T04:31:44.912749+0000 osd.2 (osd 2021-02-10T04:31:49.201 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: .2) 282 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:49.201 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: cluster 2021-02-10T04:31:44.912752 2021-02-10T04:31:49.201 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 283 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:49.201 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:49.201 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: T04:31:44.912757+0000 osd.2 (osd.2) 284 : cluster 2021-02-10T04:31:49.201 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:49.201 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: cluster 2021-02-10T04:31:44.912767+0000 osd.2 (osd 2021-02-10T04:31:49.201 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: .2) 285 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:49.201 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: cluster 2021-02-10T04:31 2021-02-10T04:31:49.201 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: :45.092803+0000 mon.a (mon.0) 1031 : cluster [DBG] 2021-02-10T04:31:49.201 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:45 smithi164 conmon[29288]: osdmap e288: 8 total, 8 up, 8 in 2021-02-10T04:31:49.202 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: cluster 2021-02-10T04:31:45.076831+0000 osd.1 ( 2021-02-10T04:31:49.202 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: osd.1) 256 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:49.202 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: cluster 2021-02-10T04:31:45.076843+0000 osd.1 (osd.1) 257 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:49.202 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: cluster 2021-02-10T04:31:45.076862+0000 2021-02-10T04:31:49.202 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: osd.1 (osd.1) 258 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:49.202 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:31:49.202 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: 04:31:45.076868+0000 osd.1 (osd.1) 259 : 2021-02-10T04:31:49.202 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:49.202 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:49.203 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: T04:31:45.076871+0000 osd.1 (osd. 2021-02-10T04:31:49.203 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: 1) 260 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:49.203 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: cluster 2021-02-10T04:31:49.203 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: 2021-02-10T04:31:45.076880+0000 2021-02-10T04:31:49.203 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: osd.1 (osd.1) 261 : cluster [WRN] 2021-02-10T04:31:49.203 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:49.203 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: cluster 2021-02-10T04:31: 2021-02-10T04:31:49.203 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: 45.428903+0000 osd.4 (osd.4) 257 2021-02-10T04:31:49.203 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:49.203 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:49.203 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: 10T04:31:45.428912+0000 osd.4 ( 2021-02-10T04:31:49.204 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: osd.4) 258 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:49.204 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: cluster 2021-02-10T04:31:49.204 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: 2021-02-10T04:31:45.428915+0000 2021-02-10T04:31:49.204 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: osd.4 (osd.4) 259 : cluster [WRN] 2021-02-10T04:31:49.204 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:49.204 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:49.204 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: -10T04:31:45.428918+0000 osd.4 ( 2021-02-10T04:31:49.204 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: osd.4) 260 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:49.204 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: cluster 2021-02-10T04:31:49.204 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: 2021-02-10T04:31:45.937354 2021-02-10T04:31:49.204 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 286 : cluster [WRN] 2021-02-10T04:31:49.204 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:31:49.204 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:49.205 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: 10T04:31:45.937360+0000 osd.2 (osd. 2021-02-10T04:31:49.205 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: 2) 287 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:31:49.205 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: cluster 2021-02-10T04:31:49.205 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: 2021-02-10T04:31:45.937367+0000 osd.2 (osd 2021-02-10T04:31:49.205 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: .2) 288 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:31:49.205 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: cluster 2021-02-10T04:31:49.205 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: 2021-02-10T04:31: 2021-02-10T04:31:49.205 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: 45.937371+0000 osd.2 (osd.2) 289 : 2021-02-10T04:31:49.205 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:31:49.205 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:49.205 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: 10T04:31:45.937375+0000 osd.2 ( 2021-02-10T04:31:49.205 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: osd.2) 290 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:31:49.205 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: cluster 2021-02-10T04:31:49.206 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: 2021-02-10T04:31: 2021-02-10T04:31:49.206 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: 45.937379+0000 osd.2 (osd.2) 291 2021-02-10T04:31:49.206 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:31:49.206 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: cluster 2021-02-10T04:31:49.206 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: 2021-02-10T04:31:45.937383 2021-02-10T04:31:49.206 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 292 : cluster [WRN] 2021-02-10T04:31:49.206 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:31:49.206 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:49.206 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: -10T04:31:45.937386+0000 osd.2 ( 2021-02-10T04:31:49.206 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: osd.2) 293 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:31:49.206 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: cluster 2021-02-10T04:31:49.206 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: 2021-02-10T04:31 2021-02-10T04:31:49.207 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: :46.043727+0000 mgr.y (mgr.14140) 2021-02-10T04:31:49.207 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: 452 : cluster [DBG] pgmap v582: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 164 active+clean; 117 MiB data, 466 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 2.5 MiB/s wr, 17 op/s 2021-02-10T04:31:49.207 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: cluster 2021-02-10T04:31:49.207 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:46 smithi164 conmon[29288]: 2021-02-10T04:31:46.096036+0000 mon.a (mon.0) 1032 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2021-02-10T04:31:49.207 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: cluster 2021-02-10T04:31:46.049783+0000 osd.1 (osd.1) 2021-02-10T04:31:49.207 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: 262 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:31:49.207 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: cluster 2021-02-10T04:31:46.049792+0000 osd.1 (osd.1) 263 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:31:49.207 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:31:49.207 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: 04:31:46.049798+0000 osd.1 (osd.1) 264 2021-02-10T04:31:49.208 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:31:49.208 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:49.208 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: -10T04:31:46.049803+0000 osd.1 (osd.1) 265 : cluster 2021-02-10T04:31:49.208 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:31:49.208 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:31:49.208 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: 04:31:46.049809+0000 osd.1 (osd. 2021-02-10T04:31:49.208 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: 1) 266 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:31:49.208 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:49.208 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: 10T04:31:46.049814+0000 osd.1 2021-02-10T04:31:49.208 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: (osd.1) 267 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:31:49.208 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:49.208 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: -10T04:31:46.390542+0000 osd.4 2021-02-10T04:31:49.208 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: (osd.4) 261 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:31:49.208 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:49.209 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: -10T04:31:46.390553+0000 2021-02-10T04:31:49.209 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: osd.4 (osd.4) 262 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:31:49.209 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: 2021-02-10T04:31:49.209 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:49.209 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: -02-10T04:31:46.390561+0000 osd.4 (osd 2021-02-10T04:31:49.209 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: .4) 263 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:31:49.209 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:31:49.209 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: 02-10T04:31:46.390569+0000 2021-02-10T04:31:49.209 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: osd.4 (osd.4) 264 : cluster [WRN] 2021-02-10T04:31:49.209 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:31:49.209 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: audit 2021-02-10T04:31:46 2021-02-10T04:31:49.209 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: .489416+0000 mon.c (mon.1) 48 : 2021-02-10T04:31:49.210 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: audit [DBG] from='client.? 172.21.15.85:0/3050165941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:31:49.210 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:31:49.210 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: T04:31:47.098903+0000 mon.a ( 2021-02-10T04:31:49.210 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[29288]: mon.0) 1033 : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2021-02-10T04:31:49.210 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:48 smithi164 conmon[29288]: cluster 2021-02-10T04:31:49.210 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:48 smithi164 conmon[29288]: 2021-02-10T04:31:49.210 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:48 smithi164 conmon[29288]: 2021-02-10T04:31:46 2021-02-10T04:31:49.210 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:48 smithi164 conmon[29288]: .761924+0000 osd.1 (osd.1) 268 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:31:49.210 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:48 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:31:49.210 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:48 smithi164 conmon[29288]: 31:46.762166+0000 osd.1 (osd.1) 269 : cluster 2021-02-10T04:31:49.210 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:48 smithi164 conmon[29288]: [DBG] purged_snaps scrub ok 2021-02-10T04:31:49.210 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:48 smithi164 conmon[29288]: audit 2021-02-10T04:31:47.865019+0000 mgr.y ( 2021-02-10T04:31:49.211 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:48 smithi164 conmon[29288]: mgr.14140) 453 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:31:49.211 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:48 smithi164 conmon[29288]: cluster 2021-02-10T04:31:49.211 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:48 smithi164 conmon[29288]: 2021-02-10T04:31 2021-02-10T04:31:49.211 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:48 smithi164 conmon[29288]: :48. 2021-02-10T04:31:49.211 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:48 smithi164 conmon[29288]: 044325 2021-02-10T04:31:49.211 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:48 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 454 : cluster [DBG] 2021-02-10T04:31:49.211 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:48 smithi164 conmon[29288]: pgmap v585: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 164 active+clean; 117 MiB data, 448 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 2.3 MiB/s wr, 15 op/s 2021-02-10T04:31:49.211 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:48 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:31:49.211 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:48 smithi164 conmon[29288]: 10T04:31:48.079306+0000 mon.a ( 2021-02-10T04:31:49.211 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:48 smithi164 conmon[29288]: mon.0) 1034 : cluster [WRN] Health check update: 18 slow ops, oldest one blocked for 152 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:31:49.211 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:48 smithi164 conmon[29288]: cluster 2021-02-10T04:31:49.211 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:48 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:31:49.211 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:48 smithi164 conmon[29288]: :31:48.102104+0000 mon.a (mon.0) 1035 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2021-02-10T04:31:49.212 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:48 smithi164 conmon[29288]: 2021-02-10T04:31:49.212 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:31:47 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:31:49.237 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:49.515 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:31:49.643 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:31:49 smithi164 conmon[40892]: debug 2021-02-10T04:31:49.318+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:49.763 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:31:49.868 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:31:49.900 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:50.105 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:50 smithi164 conmon[29288]: cluster 2021-02-10T04:31:49.102421+0000 mon.a (mon.0) 2021-02-10T04:31:50.105 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:50 smithi164 conmon[29288]: 1036 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2021-02-10T04:31:50.106 INFO:tasks.rados.rados.0.smithi085.stdout:1004: done (12 left) 2021-02-10T04:31:50.106 INFO:tasks.rados.rados.0.smithi085.stdout:1005: write (excl) oid 27 current snap is 97 2021-02-10T04:31:50.106 INFO:tasks.rados.rados.0.smithi085.stdout:1005: seq_num 377 ranges {736239=470170,1728182=571782} 2021-02-10T04:31:50.114 INFO:tasks.rados.rados.0.smithi085.stdout:1005: writing smithi08575397-27 from 736239 to 1206409 tid 1 2021-02-10T04:31:50.122 INFO:tasks.rados.rados.0.smithi085.stdout:1005: writing smithi08575397-27 from 1728182 to 2299964 tid 2 2021-02-10T04:31:50.122 INFO:tasks.rados.rados.0.smithi085.stdout:1005: finishing write tid 1 to smithi08575397-27 2021-02-10T04:31:50.122 INFO:tasks.rados.rados.0.smithi085.stdout:1006: snap_create 2021-02-10T04:31:50.125 INFO:tasks.rados.rados.0.smithi085.stdout:1005: finishing write tid 2 to smithi08575397-27 2021-02-10T04:31:50.125 INFO:tasks.rados.rados.0.smithi085.stdout:1005: finishing write tid 3 to smithi08575397-27 2021-02-10T04:31:50.125 INFO:tasks.rados.rados.0.smithi085.stdout:1005: finishing write tid 4 to smithi08575397-27 2021-02-10T04:31:50.126 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 27 v 77 (ObjNum 377 snap 97 seq_num 377) dirty exists 2021-02-10T04:31:50.126 INFO:tasks.rados.rados.0.smithi085.stdout:1005: left oid 27 (ObjNum 377 snap 97 seq_num 377) 2021-02-10T04:31:50.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:31:50 smithi085 conmon[52836]: debug 2021-02-10T04:31:50.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:31:50 smithi085 conmon[52836]: 2021-02-10T04:31:50.108+0000 7fba74fb8700 -1 osd.3 pg_epoch: 293 pg[7.2( v 292'154 (0'0,292'154] local-lis/les=93/94 n=4 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [3,1,2] r=0 lpr=93 crt=292'154 lcod 292'152 mlcod 292'152 active+clean+snaptrim trimq=[c2~1] ps=49] removing snap head 2021-02-10T04:31:50.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:31:50 smithi085 conmon[52836]: 2021-02-10T04:31:50.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:31:50 smithi085 conmon[52836]: debug 2021-02-10T04:31:50.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:31:50 smithi085 conmon[52836]: 2021-02-10T04:31:50.109+0000 7fba74fb8700 -1 osd.3 pg_epoch: 293 pg[7.2( v 293'156 (0'0,293'156] local-lis/les=93/94 n=2 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [3,1,2] r=0 lpr=93 luod=292'154 crt=293'156 lcod 292'152 mlcod 292'152 active+clean+snaptrim trimq=[c2~1] ps=49] removing snap head 2021-02-10T04:31:50.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:31:50 smithi085 conmon[52836]: 2021-02-10T04:31:50.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:50 smithi085 conmon[29319]: cluster 2021-02-10T04:31:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:50 smithi085 conmon[29319]: 2021-02-10T04:31:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:50 smithi085 conmon[29319]: 2021 2021-02-10T04:31:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:50 smithi085 conmon[29319]: - 2021-02-10T04:31:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:50 smithi085 conmon[29319]: 02-10T04:31: 2021-02-10T04:31:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:50 smithi085 conmon[29319]: 49.102421+0000 mon.a (mon.0) 1036 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2021-02-10T04:31:50.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:50 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:50.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:50 smithi085 conmon[35961]: 02-10T04:31:49.102421+0000 mon.a (mon.0) 1036 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2021-02-10T04:31:50.893 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:31:50 smithi164 conmon[40892]: debug 2021-02-10T04:31:50.893 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:31:50 smithi164 conmon[40892]: 2021-02-10T04:31:50.435+0000 7f30314bd700 -1 osd.6 pg_epoch: 293 pg[7.7( v 291'524 (0'0,291'524] local-lis/les=93/94 n=8 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [6,0,3] r=0 lpr=93 crt=291'524 lcod 289'522 mlcod 289'522 active+clean+snaptrim trimq=[b9~1,bc~3,c2~1] ps=45] removing snap head 2021-02-10T04:31:50.893 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:31:50 smithi164 conmon[40892]: 2021-02-10T04:31:51.109 INFO:tasks.rados.rados.0.smithi085.stdout:1005: done (13 left) 2021-02-10T04:31:51.110 INFO:tasks.rados.rados.0.smithi085.stdout:1006: done (12 left) 2021-02-10T04:31:51.110 INFO:tasks.rados.rados.0.smithi085.stdout:1007: snap_create 2021-02-10T04:31:51.246 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:31:51.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:51 smithi164 conmon[29288]: cluster 2021-02-10T04:31:50.044778+0000 mgr.y (mgr.14140) 455 2021-02-10T04:31:51.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:51 smithi164 conmon[29288]: : cluster [DBG] pgmap v588: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim, 162 active+clean; 113 MiB data, 452 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 MiB/s wr, 3 op/s 2021-02-10T04:31:51.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:51 smithi164 conmon[29288]: cluster 2021-02-10T04:31:50.106090+0000 mon.a (mon.0) 1037 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2021-02-10T04:31:51.441 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:31:50 smithi085 conmon[45443]: debug 2021-02-10T04:31:50.969+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:51.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:51 smithi085 conmon[29319]: cluster 2021-02-10T04:31:50.044778+0000 mgr.y 2021-02-10T04:31:51.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:51 smithi085 conmon[29319]: (mgr.14140) 455 : cluster [DBG] pgmap v588: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim, 162 active+clean; 113 MiB data, 452 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 MiB/s wr, 3 op/s 2021-02-10T04:31:51.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:51 smithi085 conmon[29319]: cluster 2021-02-10T04:31:50.106090+0000 mon.a (mon.0) 1037 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2021-02-10T04:31:51.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:51 smithi085 conmon[35961]: cluster 2021-02-10T04:31:50.044778+0000 2021-02-10T04:31:51.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:51 smithi085 conmon[35961]: mgr.y (mgr.14140) 455 : cluster [DBG] pgmap v588: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim, 162 active+clean; 113 MiB data, 452 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 MiB/s wr, 3 op/s 2021-02-10T04:31:51.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:51 smithi085 conmon[35961]: cluster 2021-02-10T04:31:50.106090+0000 mon.a (mon. 2021-02-10T04:31:51.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:51 smithi085 conmon[35961]: 0) 1037 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2021-02-10T04:31:52.116 INFO:tasks.rados.rados.0.smithi085.stdout:1007: done (12 left) 2021-02-10T04:31:54.273 INFO:tasks.rados.rados.0.smithi085.stdout:1008: write oid 36 current snap is 99 2021-02-10T04:31:54.273 INFO:tasks.rados.rados.0.smithi085.stdout:1008: seq_num 378 ranges {753046=545419,1786664=431793} 2021-02-10T04:31:54.278 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:52 smithi164 conmon[29288]: cluster 2021-02-10T04:31:54.278 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:52 smithi164 conmon[29288]: 2021-02-10T04:31:54.278 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:52 smithi164 conmon[29288]: 2021-02-10T04:31:51 2021-02-10T04:31:54.279 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:52 smithi164 conmon[29288]: .109003+0000 mon.a (mon.0) 1038 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2021-02-10T04:31:54.279 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:52 smithi164 conmon[29288]: cluster 2021-02-10T04:31:54.279 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:52 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:31:54.279 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:52 smithi164 conmon[29288]: 31:52.112782+0000 mon.a ( 2021-02-10T04:31:54.279 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:52 smithi164 conmon[29288]: mon.0) 1039 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2021-02-10T04:31:54.279 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:53 smithi164 conmon[29288]: cephadm 2021-02-10T04:31:51.995904 2021-02-10T04:31:54.279 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:53 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 456 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:31:54.279 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:53 smithi164 conmon[29288]: cluster 2021-02-10T04:31:52.045299+0000 mgr.y (mgr.14140) 457 : cluster [DBG] pgmap v591: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim, 162 active+clean; 113 MiB data, 435 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.1 MiB/s wr, 9 op/s 2021-02-10T04:31:54.279 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:53 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:31:54.279 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:53 smithi164 conmon[29288]: -02-10T04:31:52.114100+0000 mon.a (mon.0) 1040 : cluster [INF] Health check cleared: SLOW_OPS (was: 12 slow ops, oldest one blocked for 137 sec, daemons [osd.2,osd.4] have slow ops.) 2021-02-10T04:31:54.279 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:53 smithi164 conmon[29288]: audit 2021-02-10T04:31:52 2021-02-10T04:31:54.279 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:53 smithi164 conmon[29288]: .507982+0000 mon.a (mon.0) 1041 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:31:54.279 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:53 smithi164 conmon[29288]: 2021-02-10T04:31:54.280 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:53 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:31:54.280 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:53 smithi164 conmon[29288]: -10T04:31:53.055277+0000 mon.a (mon. 2021-02-10T04:31:54.280 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:53 smithi164 conmon[29288]: 0) 1042 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:31:54.280 INFO:tasks.rados.rados.0.smithi085.stdout:1008: writing smithi08575397-36 from 753046 to 1298465 tid 1 2021-02-10T04:31:54.280 INFO:tasks.rados.rados.0.smithi085.stdout:1008: writing smithi08575397-36 from 1786664 to 2218457 tid 2 2021-02-10T04:31:54.280 INFO:tasks.rados.rados.0.smithi085.stdout:1009: write (excl) oid 24 current snap is 99 2021-02-10T04:31:54.280 INFO:tasks.rados.rados.0.smithi085.stdout:1008: finishing write tid 1 to smithi08575397-36 2021-02-10T04:31:54.280 INFO:tasks.rados.rados.0.smithi085.stdout:1009: seq_num 379 ranges {769853=620668,1845146=524707} 2021-02-10T04:31:54.280 INFO:tasks.rados.rados.0.smithi085.stdout:1009: writing smithi08575397-24 from 769853 to 1390521 tid 1 2021-02-10T04:31:54.280 INFO:tasks.rados.rados.0.smithi085.stdout:1009: writing smithi08575397-24 from 1845146 to 2369853 tid 2 2021-02-10T04:31:54.280 INFO:tasks.rados.rados.0.smithi085.stdout:1010: read oid 6 snap -1 2021-02-10T04:31:54.280 INFO:tasks.rados.rados.0.smithi085.stdout:1010: expect (ObjNum 357 snap 90 seq_num 357) 2021-02-10T04:31:54.281 INFO:tasks.rados.rados.0.smithi085.stdout:1008: finishing write tid 2 to smithi08575397-36 2021-02-10T04:31:54.281 INFO:tasks.rados.rados.0.smithi085.stdout:1008: finishing write tid 3 to smithi08575397-36 2021-02-10T04:31:54.281 INFO:tasks.rados.rados.0.smithi085.stdout:1008: finishing write tid 4 to smithi08575397-36 2021-02-10T04:31:54.281 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 161 (ObjNum 378 snap 99 seq_num 378) dirty exists 2021-02-10T04:31:54.281 INFO:tasks.rados.rados.0.smithi085.stdout:1008: left oid 36 (ObjNum 378 snap 99 seq_num 378) 2021-02-10T04:31:54.281 INFO:tasks.rados.rados.0.smithi085.stdout:1008: done (14 left) 2021-02-10T04:31:54.281 INFO:tasks.rados.rados.0.smithi085.stdout:1011: snap_remove snap 98 2021-02-10T04:31:54.281 INFO:tasks.rados.rados.0.smithi085.stdout:1011: done (14 left) 2021-02-10T04:31:54.281 INFO:tasks.rados.rados.0.smithi085.stdout:1012: read oid 8 snap 97 2021-02-10T04:31:54.281 INFO:tasks.rados.rados.0.smithi085.stdout:1012: expect deleted 2021-02-10T04:31:54.281 INFO:tasks.rados.rados.0.smithi085.stdout:1013: write oid 7 current snap is 99 2021-02-10T04:31:54.281 INFO:tasks.rados.rados.0.smithi085.stdout:1013: seq_num 380 ranges {786660=695917,1903628=435462} 2021-02-10T04:31:54.281 INFO:tasks.rados.rados.0.smithi085.stdout:1013: writing smithi08575397-7 from 786660 to 1482577 tid 1 2021-02-10T04:31:54.281 INFO:tasks.rados.rados.0.smithi085.stdout:1013: writing smithi08575397-7 from 1903628 to 2339090 tid 2 2021-02-10T04:31:54.282 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:54.282 INFO:tasks.rados.rados.0.smithi085.stdout:1009: finishing write tid 1 to smithi08575397-24 2021-02-10T04:31:54.282 INFO:tasks.rados.rados.0.smithi085.stdout:1010: expect (ObjNum 357 snap 90 seq_num 357) 2021-02-10T04:31:54.282 INFO:tasks.rados.rados.0.smithi085.stdout:1009: finishing write tid 2 to smithi08575397-24 2021-02-10T04:31:54.282 INFO:tasks.rados.rados.0.smithi085.stdout:1009: finishing write tid 3 to smithi08575397-24 2021-02-10T04:31:54.282 INFO:tasks.rados.rados.0.smithi085.stdout:1009: finishing write tid 4 to smithi08575397-24 2021-02-10T04:31:54.282 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 24 v 160 (ObjNum 379 snap 99 seq_num 379) dirty exists 2021-02-10T04:31:54.282 INFO:tasks.rados.rados.0.smithi085.stdout:1009: left oid 24 (ObjNum 379 snap 99 seq_num 379) 2021-02-10T04:31:54.282 INFO:tasks.rados.rados.0.smithi085.stdout:1013: finishing write tid 1 to smithi08575397-7 2021-02-10T04:31:54.282 INFO:tasks.rados.rados.0.smithi085.stdout:1013: finishing write tid 2 to smithi08575397-7 2021-02-10T04:31:54.282 INFO:tasks.rados.rados.0.smithi085.stdout:1013: finishing write tid 3 to smithi08575397-7 2021-02-10T04:31:54.282 INFO:tasks.rados.rados.0.smithi085.stdout:1009: done (15 left) 2021-02-10T04:31:54.282 INFO:tasks.rados.rados.0.smithi085.stdout:1010: done (14 left) 2021-02-10T04:31:54.283 INFO:tasks.rados.rados.0.smithi085.stdout:1012: done (13 left) 2021-02-10T04:31:54.283 INFO:tasks.rados.rados.0.smithi085.stdout:1014: read oid 3 snap 97 2021-02-10T04:31:54.283 INFO:tasks.rados.rados.0.smithi085.stdout:1014: expect (ObjNum 371 snap 96 seq_num 371) 2021-02-10T04:31:54.283 INFO:tasks.rados.rados.0.smithi085.stdout:1015: delete oid 6 current snap is 99 2021-02-10T04:31:54.283 INFO:tasks.rados.rados.0.smithi085.stdout:1013: finishing write tid 4 to smithi08575397-7 2021-02-10T04:31:54.283 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 461 (ObjNum 380 snap 99 seq_num 380) dirty exists 2021-02-10T04:31:54.283 INFO:tasks.rados.rados.0.smithi085.stdout:1013: left oid 7 (ObjNum 380 snap 99 seq_num 380) 2021-02-10T04:31:54.283 INFO:tasks.rados.rados.0.smithi085.stdout:1014: expect (ObjNum 371 snap 96 seq_num 371) 2021-02-10T04:31:54.283 INFO:tasks.rados.rados.0.smithi085.stdout:1013: done (14 left) 2021-02-10T04:31:54.283 INFO:tasks.rados.rados.0.smithi085.stdout:1014: done (13 left) 2021-02-10T04:31:54.283 INFO:tasks.rados.rados.0.smithi085.stdout:1015: done (12 left) 2021-02-10T04:31:54.283 INFO:tasks.rados.rados.0.smithi085.stdout:1016: rollback oid 3 current snap is 99 2021-02-10T04:31:54.283 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 3 to 97 2021-02-10T04:31:54.283 INFO:tasks.rados.rados.0.smithi085.stdout:1017: copy_from oid 16 from oid 6 current snap is 99 2021-02-10T04:31:54.284 INFO:tasks.rados.rados.0.smithi085.stdout:1018: snap_create 2021-02-10T04:31:54.284 INFO:tasks.rados.rados.0.smithi085.stdout:1017: finishing copy_from to smithi08575397-16 2021-02-10T04:31:54.284 INFO:tasks.rados.rados.0.smithi085.stdout:1017: got expected ENOENT (src dne) 2021-02-10T04:31:54.284 INFO:tasks.rados.rados.0.smithi085.stdout:1017: finishing copy_from racing read to smithi08575397-16 2021-02-10T04:31:54.284 INFO:tasks.rados.rados.0.smithi085.stdout:1016: finishing rollback tid 0 to smithi08575397-3 2021-02-10T04:31:54.284 INFO:tasks.rados.rados.0.smithi085.stdout:1016: finishing rollback tid 1 to smithi08575397-3 2021-02-10T04:31:54.284 INFO:tasks.rados.rados.0.smithi085.stdout:1016: finishing rollback tid 2 to smithi08575397-3 2021-02-10T04:31:54.284 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 464 (ObjNum 371 snap 96 seq_num 371) dirty exists 2021-02-10T04:31:54.284 INFO:tasks.rados.rados.0.smithi085.stdout:1016: done (14 left) 2021-02-10T04:31:54.284 INFO:tasks.rados.rados.0.smithi085.stdout:1017: done (13 left) 2021-02-10T04:31:54.284 INFO:tasks.rados.rados.0.smithi085.stdout:1018: done (12 left) 2021-02-10T04:31:54.284 INFO:tasks.rados.rados.0.smithi085.stdout:1019: rollback oid 37 current snap is 100 2021-02-10T04:31:54.284 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 37 to 97 2021-02-10T04:31:54.285 INFO:tasks.rados.rados.0.smithi085.stdout:1020: read oid 2 snap -1 2021-02-10T04:31:54.285 INFO:tasks.rados.rados.0.smithi085.stdout:1020: expect (ObjNum 373 snap 96 seq_num 373) 2021-02-10T04:31:54.285 INFO:tasks.rados.rados.0.smithi085.stdout:1021: rollback oid 43 current snap is 100 2021-02-10T04:31:54.285 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 43 to 97 2021-02-10T04:31:54.285 INFO:tasks.rados.rados.0.smithi085.stdout:1022: write (excl) oid 46 current snap is 100 2021-02-10T04:31:54.285 INFO:tasks.rados.rados.0.smithi085.stdout:1022: seq_num 382 ranges {420274=562768,1620592=439131} 2021-02-10T04:31:54.285 INFO:tasks.rados.rados.0.smithi085.stdout:1022: writing smithi08575397-46 from 420274 to 983042 tid 1 2021-02-10T04:31:54.285 INFO:tasks.rados.rados.0.smithi085.stdout:1022: writing smithi08575397-46 from 1620592 to 2059723 tid 2 2021-02-10T04:31:54.285 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:54.285 INFO:tasks.rados.rados.0.smithi085.stdout:1019: finishing rollback tid 1 to smithi08575397-37 2021-02-10T04:31:54.285 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 0 (ObjNum 1 snap 32576 seq_num 82480432) dirty dne 2021-02-10T04:31:54.285 INFO:tasks.rados.rados.0.smithi085.stdout:1019: done (15 left) 2021-02-10T04:31:54.285 INFO:tasks.rados.rados.0.smithi085.stdout:1023: write oid 30 current snap is 100 2021-02-10T04:31:54.285 INFO:tasks.rados.rados.0.smithi085.stdout:1021: finishing rollback tid 1 to smithi08575397-43 2021-02-10T04:31:54.286 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 0 (ObjNum 3 snap 2 seq_num 4) dirty dne 2021-02-10T04:31:54.286 INFO:tasks.rados.rados.0.smithi085.stdout:1020: expect (ObjNum 373 snap 96 seq_num 373) 2021-02-10T04:31:54.286 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:52 smithi085 conmon[29319]: cluster 2021-02-10T04:31:51.109003+0000 mon.a ( 2021-02-10T04:31:54.287 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:52 smithi085 conmon[29319]: mon.0) 1038 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2021-02-10T04:31:54.287 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:52 smithi085 conmon[29319]: cluster 2021-02-10T04:31:52.112782+0000 mon.a (mon.0) 1039 : cluster 2021-02-10T04:31:54.287 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:52 smithi085 conmon[29319]: [DBG] osdmap e295: 8 total, 8 up, 8 in 2021-02-10T04:31:54.287 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:53 smithi085 conmon[29319]: cephadm 2021-02-10T04:31:51.995904+0000 mgr.y ( 2021-02-10T04:31:54.287 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:53 smithi085 conmon[29319]: mgr.14140) 456 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:31:54.287 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:53 smithi085 conmon[29319]: cluster 2021-02-10T04:31:52.045299+0000 mgr.y (mgr.14140) 457 : cluster [DBG] pgmap v591: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim, 162 active+clean; 113 MiB data, 435 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.1 MiB/s wr, 9 op/s 2021-02-10T04:31:54.287 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:53 smithi085 conmon[29319]: cluster 2021-02-10T04:31:52.114100+0000 mon.a (mon.0) 1040 : cluster [INF] Health check cleared: SLOW_OPS (was: 12 slow ops, oldest one blocked for 137 sec, daemons [osd.2,osd.4] have slow ops.) 2021-02-10T04:31:54.287 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:53 smithi085 conmon[29319]: audit 2021-02-10T04:31:52.507982+0000 mon.a (mon.0) 1041 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:31:54.287 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:53 smithi085 conmon[29319]: audit 2021-02-10T04:31:53.055277+0000 mon.a (mon.0 2021-02-10T04:31:54.287 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:53 smithi085 conmon[29319]: ) 1042 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:31:54.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:52 smithi085 conmon[35961]: cluster 2021-02-10T04:31:54.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:52 smithi085 conmon[35961]: 2021-02-10T04:31:54.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:52 smithi085 conmon[35961]: 2021 2021-02-10T04:31:54.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:52 smithi085 conmon[35961]: - 2021-02-10T04:31:54.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:52 smithi085 conmon[35961]: 02 2021-02-10T04:31:54.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:52 smithi085 conmon[35961]: - 2021-02-10T04:31:54.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:52 smithi085 conmon[35961]: 10 2021-02-10T04:31:54.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:52 smithi085 conmon[35961]: T 2021-02-10T04:31:54.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:52 smithi085 conmon[35961]: 04 2021-02-10T04:31:54.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:52 smithi085 conmon[35961]: : 2021-02-10T04:31:54.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:52 smithi085 conmon[35961]: 31 2021-02-10T04:31:54.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:52 smithi085 conmon[35961]: : 2021-02-10T04:31:54.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:52 smithi085 conmon[35961]: 51 2021-02-10T04:31:54.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:52 smithi085 conmon[35961]: . 2021-02-10T04:31:54.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:52 smithi085 conmon[35961]: 109003+0000 mon.a (mon.0) 1038 : cluster 2021-02-10T04:31:54.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:52 smithi085 conmon[35961]: [DBG] osdmap e294: 8 total, 8 up, 8 in 2021-02-10T04:31:54.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:52 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:31:54.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:52 smithi085 conmon[35961]: :31:52.112782+0000 mon.a (mon.0) 1039 : cluster [DBG] 2021-02-10T04:31:54.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:52 smithi085 conmon[35961]: osdmap e295: 8 total, 8 up, 8 in 2021-02-10T04:31:54.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:53 smithi085 conmon[35961]: cephadm 2021-02-10T04:31:51.995904+0000 2021-02-10T04:31:54.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:53 smithi085 conmon[35961]: mgr.y (mgr.14140) 456 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:31:54.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:53 smithi085 conmon[35961]: cluster 2021-02-10T04:31 2021-02-10T04:31:54.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:53 smithi085 conmon[35961]: :52.045299+0000 mgr.y (mgr.14140) 457 : cluster [DBG] pgmap v591: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim, 162 active+clean; 113 MiB data, 435 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.1 MiB/s wr, 9 op/s 2021-02-10T04:31:54.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:53 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:31:54.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:53 smithi085 conmon[35961]: -10T04:31:52.114100+0000 mon.a ( 2021-02-10T04:31:54.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:53 smithi085 conmon[35961]: mon.0) 1040 : cluster [INF] 2021-02-10T04:31:54.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:53 smithi085 conmon[35961]: Health check cleared: SLOW_OPS (was: 12 slow ops, oldest one blocked for 137 sec, daemons [osd.2,osd.4] have slow ops.) 2021-02-10T04:31:54.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:53 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:31:54.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:53 smithi085 conmon[35961]: 02-10T04:31: 2021-02-10T04:31:54.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:53 smithi085 conmon[35961]: 52.507982+0000 mon.a (mon. 2021-02-10T04:31:54.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:53 smithi085 conmon[35961]: 0) 1041 : audit [INF] 2021-02-10T04:31:54.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:53 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:31:54.292 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:53 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:31:54.292 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:53 smithi085 conmon[35961]: 02-10T04:31:53 2021-02-10T04:31:54.292 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:53 smithi085 conmon[35961]: .055277+0000 mon.a (mon.0 2021-02-10T04:31:54.292 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:53 smithi085 conmon[35961]: ) 1042 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:31:54.354 INFO:tasks.rados.rados.0.smithi085.stdout:1022: finishing write tid 1 to smithi08575397-46 2021-02-10T04:31:54.354 INFO:tasks.rados.rados.0.smithi085.stdout:1023: seq_num 383 ranges {437081=638017,1679074=582789} 2021-02-10T04:31:54.359 INFO:tasks.rados.rados.0.smithi085.stdout:1023: writing smithi08575397-30 from 437081 to 1075098 tid 1 2021-02-10T04:31:54.367 INFO:tasks.rados.rados.0.smithi085.stdout:1023: writing smithi08575397-30 from 1679074 to 2261863 tid 2 2021-02-10T04:31:54.367 INFO:tasks.rados.rados.0.smithi085.stdout:1022: finishing write tid 2 to smithi08575397-46 2021-02-10T04:31:54.367 INFO:tasks.rados.rados.0.smithi085.stdout:1022: finishing write tid 3 to smithi08575397-46 2021-02-10T04:31:54.367 INFO:tasks.rados.rados.0.smithi085.stdout:1022: finishing write tid 4 to smithi08575397-46 2021-02-10T04:31:54.367 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 330 (ObjNum 382 snap 100 seq_num 382) dirty exists 2021-02-10T04:31:54.368 INFO:tasks.rados.rados.0.smithi085.stdout:1022: left oid 46 (ObjNum 382 snap 100 seq_num 382) 2021-02-10T04:31:54.368 INFO:tasks.rados.rados.0.smithi085.stdout:1023: finishing write tid 1 to smithi08575397-30 2021-02-10T04:31:54.368 INFO:tasks.rados.rados.0.smithi085.stdout:1020: done (15 left) 2021-02-10T04:31:54.368 INFO:tasks.rados.rados.0.smithi085.stdout:1021: done (14 left) 2021-02-10T04:31:54.368 INFO:tasks.rados.rados.0.smithi085.stdout:1022: done (13 left) 2021-02-10T04:31:54.368 INFO:tasks.rados.rados.0.smithi085.stdout:1024: delete oid 11 current snap is 100 2021-02-10T04:31:54.371 INFO:tasks.rados.rados.0.smithi085.stdout:1023: finishing write tid 2 to smithi08575397-30 2021-02-10T04:31:54.371 INFO:tasks.rados.rados.0.smithi085.stdout:1024: done (13 left) 2021-02-10T04:31:54.371 INFO:tasks.rados.rados.0.smithi085.stdout:1025: write oid 7 current snap is 100 2021-02-10T04:31:54.371 INFO:tasks.rados.rados.0.smithi085.stdout:1025: seq_num 385 ranges {470695=788515,1679685=586458} 2021-02-10T04:31:54.380 INFO:tasks.rados.rados.0.smithi085.stdout:1025: writing smithi08575397-7 from 470695 to 1259210 tid 1 2021-02-10T04:31:54.385 INFO:tasks.rados.rados.0.smithi085.stdout:1025: writing smithi08575397-7 from 1679685 to 2266143 tid 2 2021-02-10T04:31:54.386 INFO:tasks.rados.rados.0.smithi085.stdout:1023: finishing write tid 3 to smithi08575397-30 2021-02-10T04:31:54.386 INFO:tasks.rados.rados.0.smithi085.stdout:1023: finishing write tid 4 to smithi08575397-30 2021-02-10T04:31:54.386 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 334 (ObjNum 383 snap 100 seq_num 383) dirty exists 2021-02-10T04:31:54.386 INFO:tasks.rados.rados.0.smithi085.stdout:1023: left oid 30 (ObjNum 383 snap 100 seq_num 383) 2021-02-10T04:31:54.386 INFO:tasks.rados.rados.0.smithi085.stdout:1025: finishing write tid 1 to smithi08575397-7 2021-02-10T04:31:54.386 INFO:tasks.rados.rados.0.smithi085.stdout:1023: done (13 left) 2021-02-10T04:31:54.386 INFO:tasks.rados.rados.0.smithi085.stdout:1026: copy_from oid 50 from oid 21 current snap is 100 2021-02-10T04:31:54.386 INFO:tasks.rados.rados.0.smithi085.stdout:1027: copy_from oid 11 from oid 40 current snap is 100 2021-02-10T04:31:54.386 INFO:tasks.rados.rados.0.smithi085.stdout:1028: rollback oid 3 current snap is 100 2021-02-10T04:31:54.386 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 3 to 99 2021-02-10T04:31:54.386 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:54.387 INFO:tasks.rados.rados.0.smithi085.stdout:1026: finishing copy_from to smithi08575397-50 2021-02-10T04:31:54.387 INFO:tasks.rados.rados.0.smithi085.stdout:1026: got expected ENOENT (src dne) 2021-02-10T04:31:54.387 INFO:tasks.rados.rados.0.smithi085.stdout:1026: finishing copy_from racing read to smithi08575397-50 2021-02-10T04:31:54.388 INFO:tasks.rados.rados.0.smithi085.stdout:1026: done (15 left) 2021-02-10T04:31:54.388 INFO:tasks.rados.rados.0.smithi085.stdout:1029: write (excl) oid 13 current snap is 100 2021-02-10T04:31:54.388 INFO:tasks.rados.rados.0.smithi085.stdout:1029: seq_num 386 ranges {487502=463764,1738167=446469} 2021-02-10T04:31:54.393 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:31:54.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:54 smithi164 conmon[29288]: cephadm 2021-02-10T04:31:52.508863+0000 mgr.y (mgr 2021-02-10T04:31:54.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:54 smithi164 conmon[29288]: .14140) 458 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:31:54.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:54 smithi164 conmon[29288]: cluster 2021-02-10T04:31:53.142626+0000 mon.a (mon.0) 1043 : 2021-02-10T04:31:54.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:54 smithi164 conmon[29288]: cluster [DBG] osdmap e296: 8 total, 8 up, 8 in 2021-02-10T04:31:54.394 INFO:tasks.rados.rados.0.smithi085.stdout:1029: writing smithi08575397-13 from 487502 to 951266 tid 1 2021-02-10T04:31:54.403 INFO:tasks.rados.rados.0.smithi085.stdout:1029: writing smithi08575397-13 from 1738167 to 2184636 tid 2 2021-02-10T04:31:54.403 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:54.403 INFO:tasks.rados.rados.0.smithi085.stdout:1027: finishing copy_from to smithi08575397-11 2021-02-10T04:31:54.403 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 520 (ObjNum 365 snap 93 seq_num 365) dirty exists 2021-02-10T04:31:54.403 INFO:tasks.rados.rados.0.smithi085.stdout:1025: finishing write tid 2 to smithi08575397-7 2021-02-10T04:31:54.404 INFO:tasks.rados.rados.0.smithi085.stdout:1027: finishing copy_from racing read to smithi08575397-11 2021-02-10T04:31:54.404 INFO:tasks.rados.rados.0.smithi085.stdout:1027: done (15 left) 2021-02-10T04:31:54.404 INFO:tasks.rados.rados.0.smithi085.stdout:1030: rollback oid 2 current snap is 100 2021-02-10T04:31:54.404 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 2 to 99 2021-02-10T04:31:54.404 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:54.414 INFO:tasks.rados.rados.0.smithi085.stdout:1025: finishing write tid 3 to smithi08575397-7 2021-02-10T04:31:54.415 INFO:tasks.rados.rados.0.smithi085.stdout:1025: finishing write tid 4 to smithi08575397-7 2021-02-10T04:31:54.415 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 467 (ObjNum 385 snap 100 seq_num 385) dirty exists 2021-02-10T04:31:54.415 INFO:tasks.rados.rados.0.smithi085.stdout:1025: left oid 7 (ObjNum 385 snap 100 seq_num 385) 2021-02-10T04:31:54.415 INFO:tasks.rados.rados.0.smithi085.stdout:1028: finishing rollback tid 0 to smithi08575397-3 2021-02-10T04:31:54.415 INFO:tasks.rados.rados.0.smithi085.stdout:1028: finishing rollback tid 1 to smithi08575397-3 2021-02-10T04:31:54.415 INFO:tasks.rados.rados.0.smithi085.stdout:1028: finishing rollback tid 2 to smithi08575397-3 2021-02-10T04:31:54.415 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 470 (ObjNum 371 snap 96 seq_num 371) dirty exists 2021-02-10T04:31:54.415 INFO:tasks.rados.rados.0.smithi085.stdout:1025: done (15 left) 2021-02-10T04:31:54.415 INFO:tasks.rados.rados.0.smithi085.stdout:1028: done (14 left) 2021-02-10T04:31:54.415 INFO:tasks.rados.rados.0.smithi085.stdout:1031: read oid 38 snap -1 2021-02-10T04:31:54.415 INFO:tasks.rados.rados.0.smithi085.stdout:1031: expect (ObjNum 287 snap 71 seq_num 287) 2021-02-10T04:31:54.416 INFO:tasks.rados.rados.0.smithi085.stdout:1032: rollback oid 40 current snap is 100 2021-02-10T04:31:54.416 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 40 to 99 2021-02-10T04:31:54.416 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:54.418 INFO:tasks.rados.rados.0.smithi085.stdout:1029: finishing write tid 1 to smithi08575397-13 2021-02-10T04:31:54.418 INFO:tasks.rados.rados.0.smithi085.stdout:1030: finishing rollback tid 0 to smithi08575397-2 2021-02-10T04:31:54.418 INFO:tasks.rados.rados.0.smithi085.stdout:1030: finishing rollback tid 1 to smithi08575397-2 2021-02-10T04:31:54.418 INFO:tasks.rados.rados.0.smithi085.stdout:1030: finishing rollback tid 2 to smithi08575397-2 2021-02-10T04:31:54.418 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 293 (ObjNum 373 snap 96 seq_num 373) dirty exists 2021-02-10T04:31:54.419 INFO:tasks.rados.rados.0.smithi085.stdout:1030: done (15 left) 2021-02-10T04:31:54.419 INFO:tasks.rados.rados.0.smithi085.stdout:1033: read oid 11 snap -1 2021-02-10T04:31:54.419 INFO:tasks.rados.rados.0.smithi085.stdout:1033: expect (ObjNum 365 snap 93 seq_num 365) 2021-02-10T04:31:54.419 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:54.419 INFO:tasks.rados.rados.0.smithi085.stdout:1031: expect (ObjNum 287 snap 71 seq_num 287) 2021-02-10T04:31:54.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:54 smithi085 conmon[29319]: cephadm 2021-02-10T04:31:52.508863+0000 mgr.y ( 2021-02-10T04:31:54.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:54 smithi085 conmon[29319]: mgr.14140) 458 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:31:54.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:54 smithi085 conmon[29319]: cluster 2021-02-10T04:31:53.142626+0000 mon.a (mon.0) 1043 : cluster [DBG] osdmap e296: 8 total, 8 up, 8 in 2021-02-10T04:31:54.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:54 smithi085 conmon[35961]: cephadm 2021-02-10T04:31:52. 2021-02-10T04:31:54.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:54 smithi085 conmon[35961]: 508863+0000 mgr.y (mgr.14140) 458 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:31:54.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:54 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:31:54.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:54 smithi085 conmon[35961]: 02-10T04:31:53.142626+0000 mon.a (mon.0) 1043 : cluster [DBG] 2021-02-10T04:31:54.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:54 smithi085 conmon[35961]: osdmap e296: 8 total, 8 up, 8 in 2021-02-10T04:31:54.431 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:54.626 INFO:tasks.rados.rados.0.smithi085.stdout:1031: done (15 left) 2021-02-10T04:31:54.626 INFO:tasks.rados.rados.0.smithi085.stdout:1034: read oid 50 snap 99 2021-02-10T04:31:54.626 INFO:tasks.rados.rados.0.smithi085.stdout:1034: expect deleted 2021-02-10T04:31:54.626 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:54.626 INFO:tasks.rados.rados.0.smithi085.stdout:1029: finishing write tid 2 to smithi08575397-13 2021-02-10T04:31:54.627 INFO:tasks.rados.rados.0.smithi085.stdout:1029: finishing write tid 3 to smithi08575397-13 2021-02-10T04:31:54.627 INFO:tasks.rados.rados.0.smithi085.stdout:1029: finishing write tid 4 to smithi08575397-13 2021-02-10T04:31:54.627 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 385 (ObjNum 386 snap 100 seq_num 386) dirty exists 2021-02-10T04:31:54.627 INFO:tasks.rados.rados.0.smithi085.stdout:1029: left oid 13 (ObjNum 386 snap 100 seq_num 386) 2021-02-10T04:31:54.627 INFO:tasks.rados.rados.0.smithi085.stdout:1032: finishing rollback tid 0 to smithi08575397-40 2021-02-10T04:31:54.627 INFO:tasks.rados.rados.0.smithi085.stdout:1032: finishing rollback tid 1 to smithi08575397-40 2021-02-10T04:31:54.627 INFO:tasks.rados.rados.0.smithi085.stdout:1032: finishing rollback tid 2 to smithi08575397-40 2021-02-10T04:31:54.627 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 535 (ObjNum 365 snap 93 seq_num 365) dirty exists 2021-02-10T04:31:54.627 INFO:tasks.rados.rados.0.smithi085.stdout:1029: done (15 left) 2021-02-10T04:31:54.627 INFO:tasks.rados.rados.0.smithi085.stdout:1032: done (14 left) 2021-02-10T04:31:54.627 INFO:tasks.rados.rados.0.smithi085.stdout:1035: copy_from oid 37 from oid 26 current snap is 100 2021-02-10T04:31:54.627 INFO:tasks.rados.rados.0.smithi085.stdout:1036: rollback oid 36 current snap is 100 2021-02-10T04:31:54.627 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 36 to 99 2021-02-10T04:31:54.628 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:54.628 INFO:tasks.rados.rados.0.smithi085.stdout:1033: expect (ObjNum 365 snap 93 seq_num 365) 2021-02-10T04:31:54.681 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:31:54 smithi085 conmon[41540]: debug 2021-02-10T04:31:54.644+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:54.688 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:31:54.776 INFO:tasks.rados.rados.0.smithi085.stdout:1033: done (15 left) 2021-02-10T04:31:54.776 INFO:tasks.rados.rados.0.smithi085.stdout:1037: snap_remove snap 97 2021-02-10T04:31:54.804 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:31:54.832 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:55.133 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:31:54 smithi164 conmon[44544]: debug 2021-02-10T04:31:54.911+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:55.145 INFO:tasks.rados.rados.0.smithi085.stdout:1037: done (15 left) 2021-02-10T04:31:55.146 INFO:tasks.rados.rados.0.smithi085.stdout:1038: rollback oid 30 current snap is 100 2021-02-10T04:31:55.146 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 30 to 99 2021-02-10T04:31:55.146 INFO:tasks.rados.rados.0.smithi085.stdout:1036: finishing rollback tid 0 to smithi08575397-36 2021-02-10T04:31:55.146 INFO:tasks.rados.rados.0.smithi085.stdout:1036: finishing rollback tid 1 to smithi08575397-36 2021-02-10T04:31:55.146 INFO:tasks.rados.rados.0.smithi085.stdout:1036: finishing rollback tid 2 to smithi08575397-36 2021-02-10T04:31:55.146 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 165 (ObjNum 378 snap 99 seq_num 378) dirty exists 2021-02-10T04:31:55.146 INFO:tasks.rados.rados.0.smithi085.stdout:1035: finishing copy_from to smithi08575397-37 2021-02-10T04:31:55.146 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 85 (ObjNum 343 snap 88 seq_num 343) dirty exists 2021-02-10T04:31:55.146 INFO:tasks.rados.rados.0.smithi085.stdout:1035: finishing copy_from racing read to smithi08575397-37 2021-02-10T04:31:55.146 INFO:tasks.rados.rados.0.smithi085.stdout:1034: done (15 left) 2021-02-10T04:31:55.146 INFO:tasks.rados.rados.0.smithi085.stdout:1035: done (14 left) 2021-02-10T04:31:55.147 INFO:tasks.rados.rados.0.smithi085.stdout:1036: done (13 left) 2021-02-10T04:31:55.147 INFO:tasks.rados.rados.0.smithi085.stdout:1039: write (excl) oid 6 current snap is 100 2021-02-10T04:31:55.147 INFO:tasks.rados.rados.0.smithi085.stdout:1039: seq_num 387 ranges {504309=539013,1796649=590127} 2021-02-10T04:31:55.152 INFO:tasks.rados.rados.0.smithi085.stdout:1039: writing smithi08575397-6 from 504309 to 1043322 tid 1 2021-02-10T04:31:55.161 INFO:tasks.rados.rados.0.smithi085.stdout:1039: writing smithi08575397-6 from 1796649 to 2386776 tid 2 2021-02-10T04:31:55.161 INFO:tasks.rados.rados.0.smithi085.stdout:1040: read oid 22 snap -1 2021-02-10T04:31:55.161 INFO:tasks.rados.rados.0.smithi085.stdout:1040: expect (ObjNum 346 snap 88 seq_num 346) 2021-02-10T04:31:55.161 INFO:tasks.rados.rados.0.smithi085.stdout:1041: write (excl) oid 24 current snap is 100 2021-02-10T04:31:55.161 INFO:tasks.rados.rados.0.smithi085.stdout:1041: seq_num 388 ranges {521116=730615,1971484=549632} 2021-02-10T04:31:55.169 INFO:tasks.rados.rados.0.smithi085.stdout:1041: writing smithi08575397-24 from 521116 to 1251731 tid 1 2021-02-10T04:31:55.177 INFO:tasks.rados.rados.0.smithi085.stdout:1041: writing smithi08575397-24 from 1971484 to 2521116 tid 2 2021-02-10T04:31:55.177 INFO:tasks.rados.rados.0.smithi085.stdout:1038: finishing rollback tid 0 to smithi08575397-30 2021-02-10T04:31:55.177 INFO:tasks.rados.rados.0.smithi085.stdout:1038: finishing rollback tid 1 to smithi08575397-30 2021-02-10T04:31:55.177 INFO:tasks.rados.rados.0.smithi085.stdout:1038: finishing rollback tid 2 to smithi08575397-30 2021-02-10T04:31:55.177 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 341 (ObjNum 372 snap 96 seq_num 372) dirty exists 2021-02-10T04:31:55.177 INFO:tasks.rados.rados.0.smithi085.stdout:1039: finishing write tid 1 to smithi08575397-6 2021-02-10T04:31:55.177 INFO:tasks.rados.rados.0.smithi085.stdout:1040: expect (ObjNum 346 snap 88 seq_num 346) 2021-02-10T04:31:55.339 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:31:55.369 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:31:55.373 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:31:55.392 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:55 smithi085 conmon[29319]: cluster 2021-02-10T04:31:54.045778+0000 mgr.y (mgr.14140) 459 : cluster [DBG] pgmap v594: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim, 163 active+clean; 113 MiB data, 427 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 975 KiB/s wr, 8 op/s 2021-02-10T04:31:55.392 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:55 smithi085 conmon[29319]: cluster 2021-02-10T04:31:54.143238+0000 mon.a (mon.0) 1044 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2021-02-10T04:31:55.393 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:55 smithi085 conmon[35961]: cluster 2021-02-10T04:31:54.045778+0000 mgr.y (mgr.14140) 459 : cluster [DBG] pgmap v594: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim, 163 active+clean; 113 MiB data, 427 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 975 KiB/s wr, 8 op/s 2021-02-10T04:31:55.393 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:55 smithi085 conmon[35961]: cluster 2021-02-10T04:31:54.143238+0000 mon.a (mon.0) 1044 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2021-02-10T04:31:55.393 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:31:55 smithi085 conmon[52836]: debug 2021-02-10T04:31:55.393 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:31:55 smithi085 conmon[52836]: 2021-02-10T04:31:55.147+0000 7fba747b7700 -1 osd.3 pg_epoch: 298 pg[7.3( v 298'336 (0'0,298'336] local-lis/les=93/94 n=6 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [3,2,4] r=0 lpr=93 luod=297'334 crt=298'336 lcod 297'333 mlcod 297'333 active+clean+snaptrim trimq=[c4~1] ps=51] removing snap head 2021-02-10T04:31:55.393 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:31:55 smithi085 conmon[52836]: 2021-02-10T04:31:55.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:55 smithi164 conmon[29288]: cluster 2021-02-10T04:31:54.045778+0000 mgr.y 2021-02-10T04:31:55.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:55 smithi164 conmon[29288]: (mgr.14140) 459 : cluster [DBG] pgmap v594: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim, 163 active+clean; 113 MiB data, 427 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 975 KiB/s wr, 8 op/s 2021-02-10T04:31:55.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:55 smithi164 conmon[29288]: cluster 2021-02-10T04:31:54.143238+0000 mon.a (mon.0 2021-02-10T04:31:55.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:55 smithi164 conmon[29288]: ) 1044 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2021-02-10T04:31:55.444 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:31:55.450 INFO:tasks.rados.rados.0.smithi085.stdout:1041: finishing write tid 1 to smithi08575397-24 2021-02-10T04:31:55.451 INFO:tasks.rados.rados.0.smithi085.stdout:1039: finishing write tid 2 to smithi08575397-6 2021-02-10T04:31:55.451 INFO:tasks.rados.rados.0.smithi085.stdout:1039: finishing write tid 3 to smithi08575397-6 2021-02-10T04:31:55.451 INFO:tasks.rados.rados.0.smithi085.stdout:1039: finishing write tid 4 to smithi08575397-6 2021-02-10T04:31:55.451 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 344 (ObjNum 387 snap 100 seq_num 387) dirty exists 2021-02-10T04:31:55.451 INFO:tasks.rados.rados.0.smithi085.stdout:1039: left oid 6 (ObjNum 387 snap 100 seq_num 387) 2021-02-10T04:31:55.451 INFO:tasks.rados.rados.0.smithi085.stdout:1038: done (15 left) 2021-02-10T04:31:55.452 INFO:tasks.rados.rados.0.smithi085.stdout:1039: done (14 left) 2021-02-10T04:31:55.452 INFO:tasks.rados.rados.0.smithi085.stdout:1040: done (13 left) 2021-02-10T04:31:55.452 INFO:tasks.rados.rados.0.smithi085.stdout:1042: rollback oid 1 current snap is 100 2021-02-10T04:31:55.452 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 1 to 99 2021-02-10T04:31:55.452 INFO:tasks.rados.rados.0.smithi085.stdout:1041: finishing write tid 2 to smithi08575397-24 2021-02-10T04:31:55.452 INFO:tasks.rados.rados.0.smithi085.stdout:1041: finishing write tid 3 to smithi08575397-24 2021-02-10T04:31:55.452 INFO:tasks.rados.rados.0.smithi085.stdout:1041: finishing write tid 4 to smithi08575397-24 2021-02-10T04:31:55.452 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 24 v 164 (ObjNum 388 snap 100 seq_num 388) dirty exists 2021-02-10T04:31:55.452 INFO:tasks.rados.rados.0.smithi085.stdout:1041: left oid 24 (ObjNum 388 snap 100 seq_num 388) 2021-02-10T04:31:55.453 INFO:tasks.rados.rados.0.smithi085.stdout:1041: done (13 left) 2021-02-10T04:31:55.454 INFO:tasks.rados.rados.0.smithi085.stdout:1043: write (excl) oid 40 current snap is 100 2021-02-10T04:31:55.454 INFO:tasks.rados.rados.0.smithi085.stdout:1043: seq_num 389 ranges {537923=405864,1513613=593796} 2021-02-10T04:31:55.455 INFO:tasks.rados.rados.0.smithi085.stdout:1043: writing smithi08575397-40 from 537923 to 943787 tid 1 2021-02-10T04:31:55.464 INFO:tasks.rados.rados.0.smithi085.stdout:1043: writing smithi08575397-40 from 1513613 to 2107409 tid 2 2021-02-10T04:31:55.464 INFO:tasks.rados.rados.0.smithi085.stdout:1044: write (excl) oid 43 current snap is 100 2021-02-10T04:31:55.464 INFO:tasks.rados.rados.0.smithi085.stdout:1042: finishing rollback tid 0 to smithi08575397-1 2021-02-10T04:31:55.464 INFO:tasks.rados.rados.0.smithi085.stdout:1042: finishing rollback tid 1 to smithi08575397-1 2021-02-10T04:31:55.464 INFO:tasks.rados.rados.0.smithi085.stdout:1042: finishing rollback tid 2 to smithi08575397-1 2021-02-10T04:31:55.464 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 268 (ObjNum 308 snap 74 seq_num 308) dirty exists 2021-02-10T04:31:55.464 INFO:tasks.rados.rados.0.smithi085.stdout:1043: finishing write tid 1 to smithi08575397-40 2021-02-10T04:31:55.464 INFO:tasks.rados.rados.0.smithi085.stdout:1044: seq_num 390 ranges {554730=481113,1572095=737454} 2021-02-10T04:31:55.469 INFO:tasks.rados.rados.0.smithi085.stdout:1044: writing smithi08575397-43 from 554730 to 1035843 tid 1 2021-02-10T04:31:55.478 INFO:tasks.rados.rados.0.smithi085.stdout:1044: writing smithi08575397-43 from 1572095 to 2309549 tid 2 2021-02-10T04:31:55.478 INFO:tasks.rados.rados.0.smithi085.stdout:1042: done (14 left) 2021-02-10T04:31:55.478 INFO:tasks.rados.rados.0.smithi085.stdout:1045: snap_create 2021-02-10T04:31:55.478 INFO:tasks.rados.rados.0.smithi085.stdout:1043: finishing write tid 2 to smithi08575397-40 2021-02-10T04:31:55.478 INFO:tasks.rados.rados.0.smithi085.stdout:1043: finishing write tid 3 to smithi08575397-40 2021-02-10T04:31:55.478 INFO:tasks.rados.rados.0.smithi085.stdout:1043: finishing write tid 4 to smithi08575397-40 2021-02-10T04:31:55.478 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 543 (ObjNum 389 snap 100 seq_num 389) dirty exists 2021-02-10T04:31:55.478 INFO:tasks.rados.rados.0.smithi085.stdout:1043: left oid 40 (ObjNum 389 snap 100 seq_num 389) 2021-02-10T04:31:55.478 INFO:tasks.rados.rados.0.smithi085.stdout:1044: finishing write tid 1 to smithi08575397-43 2021-02-10T04:31:55.481 INFO:tasks.rados.rados.0.smithi085.stdout:1044: finishing write tid 2 to smithi08575397-43 2021-02-10T04:31:55.481 INFO:tasks.rados.rados.0.smithi085.stdout:1044: finishing write tid 3 to smithi08575397-43 2021-02-10T04:31:55.481 INFO:tasks.rados.rados.0.smithi085.stdout:1044: finishing write tid 4 to smithi08575397-43 2021-02-10T04:31:55.482 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 473 (ObjNum 390 snap 100 seq_num 390) dirty exists 2021-02-10T04:31:55.482 INFO:tasks.rados.rados.0.smithi085.stdout:1044: left oid 43 (ObjNum 390 snap 100 seq_num 390) 2021-02-10T04:31:55.483 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:56.153 INFO:tasks.rados.rados.0.smithi085.stdout:1043: done (14 left) 2021-02-10T04:31:56.153 INFO:tasks.rados.rados.0.smithi085.stdout:1044: done (13 left) 2021-02-10T04:31:56.153 INFO:tasks.rados.rados.0.smithi085.stdout:1045: done (12 left) 2021-02-10T04:31:56.154 INFO:tasks.rados.rados.0.smithi085.stdout:1046: write (excl) oid 16 current snap is 101 2021-02-10T04:31:56.154 INFO:tasks.rados.rados.0.smithi085.stdout:1046: seq_num 391 ranges {571537=556362,1630577=597465} 2021-02-10T04:31:56.159 INFO:tasks.rados.rados.0.smithi085.stdout:1046: writing smithi08575397-16 from 571537 to 1127899 tid 1 2021-02-10T04:31:56.167 INFO:tasks.rados.rados.0.smithi085.stdout:1046: writing smithi08575397-16 from 1630577 to 2228042 tid 2 2021-02-10T04:31:56.167 INFO:tasks.rados.rados.0.smithi085.stdout:1047: write oid 23 current snap is 101 2021-02-10T04:31:56.167 INFO:tasks.rados.rados.0.smithi085.stdout:1046: finishing write tid 1 to smithi08575397-16 2021-02-10T04:31:56.167 INFO:tasks.rados.rados.0.smithi085.stdout:1047: seq_num 392 ranges {588344=631611,1689059=741123} 2021-02-10T04:31:56.174 INFO:tasks.rados.rados.0.smithi085.stdout:1047: writing smithi08575397-23 from 588344 to 1219955 tid 1 2021-02-10T04:31:56.182 INFO:tasks.rados.rados.0.smithi085.stdout:1047: writing smithi08575397-23 from 1689059 to 2430182 tid 2 2021-02-10T04:31:56.183 INFO:tasks.rados.rados.0.smithi085.stdout:1048: read oid 11 snap 100 2021-02-10T04:31:56.183 INFO:tasks.rados.rados.0.smithi085.stdout:1048: expect (ObjNum 365 snap 93 seq_num 365) 2021-02-10T04:31:56.183 INFO:tasks.rados.rados.0.smithi085.stdout:1049: rollback oid 24 current snap is 101 2021-02-10T04:31:56.183 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 24 to 100 2021-02-10T04:31:56.183 INFO:tasks.rados.rados.0.smithi085.stdout:1046: finishing write tid 2 to smithi08575397-16 2021-02-10T04:31:56.183 INFO:tasks.rados.rados.0.smithi085.stdout:1046: finishing write tid 3 to smithi08575397-16 2021-02-10T04:31:56.183 INFO:tasks.rados.rados.0.smithi085.stdout:1046: finishing write tid 4 to smithi08575397-16 2021-02-10T04:31:56.183 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 547 (ObjNum 391 snap 101 seq_num 391) dirty exists 2021-02-10T04:31:56.183 INFO:tasks.rados.rados.0.smithi085.stdout:1046: left oid 16 (ObjNum 391 snap 101 seq_num 391) 2021-02-10T04:31:56.183 INFO:tasks.rados.rados.0.smithi085.stdout:1047: finishing write tid 1 to smithi08575397-23 2021-02-10T04:31:56.183 INFO:tasks.rados.rados.0.smithi085.stdout:1046: done (15 left) 2021-02-10T04:31:56.184 INFO:tasks.rados.rados.0.smithi085.stdout:1050: delete oid 50 current snap is 101 2021-02-10T04:31:56.187 INFO:tasks.rados.rados.0.smithi085.stdout:1047: finishing write tid 2 to smithi08575397-23 2021-02-10T04:31:56.188 INFO:tasks.rados.rados.0.smithi085.stdout:1047: finishing write tid 3 to smithi08575397-23 2021-02-10T04:31:56.188 INFO:tasks.rados.rados.0.smithi085.stdout:1047: finishing write tid 4 to smithi08575397-23 2021-02-10T04:31:56.188 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 168 (ObjNum 392 snap 101 seq_num 392) dirty exists 2021-02-10T04:31:56.188 INFO:tasks.rados.rados.0.smithi085.stdout:1047: left oid 23 (ObjNum 392 snap 101 seq_num 392) 2021-02-10T04:31:56.188 INFO:tasks.rados.rados.0.smithi085.stdout:1049: finishing rollback tid 0 to smithi08575397-24 2021-02-10T04:31:56.189 INFO:tasks.rados.rados.0.smithi085.stdout:1049: finishing rollback tid 1 to smithi08575397-24 2021-02-10T04:31:56.189 INFO:tasks.rados.rados.0.smithi085.stdout:1049: finishing rollback tid 2 to smithi08575397-24 2021-02-10T04:31:56.190 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 24 v 168 (ObjNum 388 snap 100 seq_num 388) dirty exists 2021-02-10T04:31:56.194 INFO:tasks.rados.rados.0.smithi085.stdout:1048: expect (ObjNum 365 snap 93 seq_num 365) 2021-02-10T04:31:56.343 INFO:tasks.rados.rados.0.smithi085.stdout:1047: done (15 left) 2021-02-10T04:31:56.343 INFO:tasks.rados.rados.0.smithi085.stdout:1048: done (14 left) 2021-02-10T04:31:56.343 INFO:tasks.rados.rados.0.smithi085.stdout:1049: done (13 left) 2021-02-10T04:31:56.344 INFO:tasks.rados.rados.0.smithi085.stdout:1050: done (12 left) 2021-02-10T04:31:56.344 INFO:tasks.rados.rados.0.smithi085.stdout:1051: write oid 5 current snap is 101 2021-02-10T04:31:56.344 INFO:tasks.rados.rados.0.smithi085.stdout:1051: seq_num 394 ranges {621958=782109,2089670=532288} 2021-02-10T04:31:56.351 INFO:tasks.rados.rados.0.smithi085.stdout:1051: writing smithi08575397-5 from 621958 to 1404067 tid 1 2021-02-10T04:31:56.358 INFO:tasks.rados.rados.0.smithi085.stdout:1051: writing smithi08575397-5 from 2089670 to 2621958 tid 2 2021-02-10T04:31:56.358 INFO:tasks.rados.rados.0.smithi085.stdout:1052: snap_remove snap 99 2021-02-10T04:31:56.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:56 smithi164 conmon[29288]: cluster 2021-02-10T04:31:55.144818+0000 mon.a (mon.0) 1045 2021-02-10T04:31:56.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:56 smithi164 conmon[29288]: : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2021-02-10T04:31:56.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:56 smithi164 conmon[29288]: audit 2021-02-10T04:31:55.345601+0000 mgr.y (mgr.14140) 460 : audit [DBG] from='client.15024 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:31:56.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:56 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:31:56.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:56 smithi164 conmon[29288]: :31:56.046263+0000 mgr.y (mgr.14140) 461 : cluster [DBG] pgmap v597: 168 pgs: 1 active+clean+scrubbing, 1 active+clean+snaptrim, 166 active+clean; 113 MiB data, 436 MiB used, 715 GiB / 715 GiB avail; 8.6 MiB/s rd, 1.6 MiB/s wr, 20 op/s 2021-02-10T04:31:56.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:56 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:31:56.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:56 smithi085 conmon[29319]: 31:55.144818+0000 mon.a (mon.0) 1045 : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2021-02-10T04:31:56.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:56 smithi085 conmon[29319]: audit 2021-02-10T04:31:55.345601+0000 mgr.y (mgr.14140) 460 : audit [DBG] from='client.15024 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:31:56.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:56 smithi085 conmon[29319]: cluster 2021-02-10T04:31:56.046263+0000 mgr.y (mgr.14140 2021-02-10T04:31:56.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:56 smithi085 conmon[29319]: ) 461 : cluster [DBG] pgmap v597: 168 pgs: 1 active+clean+scrubbing, 1 active+clean+snaptrim, 166 active+clean; 113 MiB data, 436 MiB used, 715 GiB / 715 GiB avail; 8.6 MiB/s rd, 1.6 MiB/s wr, 20 op/s 2021-02-10T04:31:56.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:56 smithi085 conmon[35961]: cluster 2021-02-10T04:31:55.144818+0000 mon.a (mon. 2021-02-10T04:31:56.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:56 smithi085 conmon[35961]: 0) 1045 : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2021-02-10T04:31:56.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:56 smithi085 conmon[35961]: audit 2021-02-10T04:31:55.345601+0000 2021-02-10T04:31:56.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:56 smithi085 conmon[35961]: mgr.y (mgr.14140) 460 : audit [DBG] from='client.15024 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:31:56.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:56 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:31:56.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:56 smithi085 conmon[35961]: 10T04:31:56.046263+0000 mgr.y (mgr.14140) 461 : cluster [DBG] pgmap v597: 168 pgs: 1 active+clean+scrubbing, 1 active+clean+snaptrim, 166 active+clean; 113 MiB data, 436 MiB used, 715 GiB / 715 GiB avail; 8.6 MiB/s rd, 1.6 MiB/s wr, 20 op/s 2021-02-10T04:31:56.484 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:31:56.789 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:31:57.162 INFO:tasks.rados.rados.0.smithi085.stdout:1052: done (13 left) 2021-02-10T04:31:57.162 INFO:tasks.rados.rados.0.smithi085.stdout:1053: rollback oid 23 current snap is 101 2021-02-10T04:31:57.162 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 23 to 100 2021-02-10T04:31:57.162 INFO:tasks.rados.rados.0.smithi085.stdout:1054: write (excl) oid 46 current snap is 101 2021-02-10T04:31:57.162 INFO:tasks.rados.rados.0.smithi085.stdout:1054: seq_num 395 ranges {638765=457358,1748152=604803} 2021-02-10T04:31:57.164 INFO:tasks.rados.rados.0.smithi085.stdout:1054: writing smithi08575397-46 from 638765 to 1096123 tid 1 2021-02-10T04:31:57.173 INFO:tasks.rados.rados.0.smithi085.stdout:1054: writing smithi08575397-46 from 1748152 to 2352955 tid 2 2021-02-10T04:31:57.174 INFO:tasks.rados.rados.0.smithi085.stdout:1051: finishing write tid 1 to smithi08575397-5 2021-02-10T04:31:57.174 INFO:tasks.rados.rados.0.smithi085.stdout:1051: finishing write tid 2 to smithi08575397-5 2021-02-10T04:31:57.174 INFO:tasks.rados.rados.0.smithi085.stdout:1051: finishing write tid 3 to smithi08575397-5 2021-02-10T04:31:57.174 INFO:tasks.rados.rados.0.smithi085.stdout:1051: finishing write tid 4 to smithi08575397-5 2021-02-10T04:31:57.174 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 388 (ObjNum 394 snap 101 seq_num 394) dirty exists 2021-02-10T04:31:57.174 INFO:tasks.rados.rados.0.smithi085.stdout:1051: left oid 5 (ObjNum 394 snap 101 seq_num 394) 2021-02-10T04:31:57.174 INFO:tasks.rados.rados.0.smithi085.stdout:1054: finishing write tid 1 to smithi08575397-46 2021-02-10T04:31:57.174 INFO:tasks.rados.rados.0.smithi085.stdout:1053: finishing rollback tid 0 to smithi08575397-23 2021-02-10T04:31:57.174 INFO:tasks.rados.rados.0.smithi085.stdout:1053: finishing rollback tid 1 to smithi08575397-23 2021-02-10T04:31:57.174 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 170 (ObjNum 82252864 snap 22030 seq_num 0) dirty dne 2021-02-10T04:31:57.174 INFO:tasks.rados.rados.0.smithi085.stdout:1051: done (14 left) 2021-02-10T04:31:57.174 INFO:tasks.rados.rados.0.smithi085.stdout:1053: done (13 left) 2021-02-10T04:31:57.175 INFO:tasks.rados.rados.0.smithi085.stdout:1055: read oid 27 snap -1 2021-02-10T04:31:57.175 INFO:tasks.rados.rados.0.smithi085.stdout:1055: expect (ObjNum 377 snap 97 seq_num 377) 2021-02-10T04:31:57.175 INFO:tasks.rados.rados.0.smithi085.stdout:1056: write oid 11 current snap is 101 2021-02-10T04:31:57.175 INFO:tasks.rados.rados.0.smithi085.stdout:1056: seq_num 396 ranges {655572=648960,1922987=732585} 2021-02-10T04:31:57.181 INFO:tasks.rados.rados.0.smithi085.stdout:1056: writing smithi08575397-11 from 655572 to 1304532 tid 1 2021-02-10T04:31:57.190 INFO:tasks.rados.rados.0.smithi085.stdout:1056: writing smithi08575397-11 from 1922987 to 2655572 tid 2 2021-02-10T04:31:57.190 INFO:tasks.rados.rados.0.smithi085.stdout:1057: copy_from oid 43 from oid 8 current snap is 101 2021-02-10T04:31:57.190 INFO:tasks.rados.rados.0.smithi085.stdout:1054: finishing write tid 2 to smithi08575397-46 2021-02-10T04:31:57.190 INFO:tasks.rados.rados.0.smithi085.stdout:1054: finishing write tid 3 to smithi08575397-46 2021-02-10T04:31:57.190 INFO:tasks.rados.rados.0.smithi085.stdout:1054: finishing write tid 4 to smithi08575397-46 2021-02-10T04:31:57.190 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 350 (ObjNum 395 snap 101 seq_num 395) dirty exists 2021-02-10T04:31:57.190 INFO:tasks.rados.rados.0.smithi085.stdout:1054: left oid 46 (ObjNum 395 snap 101 seq_num 395) 2021-02-10T04:31:57.191 INFO:tasks.rados.rados.0.smithi085.stdout:1055: expect (ObjNum 377 snap 97 seq_num 377) 2021-02-10T04:31:57.300 INFO:tasks.rados.rados.0.smithi085.stdout:1054: done (15 left) 2021-02-10T04:31:57.300 INFO:tasks.rados.rados.0.smithi085.stdout:1055: done (14 left) 2021-02-10T04:31:57.300 INFO:tasks.rados.rados.0.smithi085.stdout:1058: read oid 21 snap -1 2021-02-10T04:31:57.301 INFO:tasks.rados.rados.0.smithi085.stdout:1058: expect deleted 2021-02-10T04:31:57.301 INFO:tasks.rados.rados.0.smithi085.stdout:1059: rollback oid 46 current snap is 101 2021-02-10T04:31:57.301 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 46 to 100 2021-02-10T04:31:57.301 INFO:tasks.rados.rados.0.smithi085.stdout:1056: finishing write tid 1 to smithi08575397-11 2021-02-10T04:31:57.301 INFO:tasks.rados.rados.0.smithi085.stdout:1057: finishing copy_from to smithi08575397-43 2021-02-10T04:31:57.302 INFO:tasks.rados.rados.0.smithi085.stdout:1057: got expected ENOENT (src dne) 2021-02-10T04:31:57.302 INFO:tasks.rados.rados.0.smithi085.stdout:1057: finishing copy_from racing read to smithi08575397-43 2021-02-10T04:31:57.302 INFO:tasks.rados.rados.0.smithi085.stdout:1056: finishing write tid 2 to smithi08575397-11 2021-02-10T04:31:57.303 INFO:tasks.rados.rados.0.smithi085.stdout:1056: finishing write tid 3 to smithi08575397-11 2021-02-10T04:31:57.304 INFO:tasks.rados.rados.0.smithi085.stdout:1056: finishing write tid 4 to smithi08575397-11 2021-02-10T04:31:57.304 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 555 (ObjNum 396 snap 101 seq_num 396) dirty exists 2021-02-10T04:31:57.304 INFO:tasks.rados.rados.0.smithi085.stdout:1056: left oid 11 (ObjNum 396 snap 101 seq_num 396) 2021-02-10T04:31:57.304 INFO:tasks.rados.rados.0.smithi085.stdout:1056: done (15 left) 2021-02-10T04:31:57.304 INFO:tasks.rados.rados.0.smithi085.stdout:1057: done (14 left) 2021-02-10T04:31:57.304 INFO:tasks.rados.rados.0.smithi085.stdout:1060: snap_create 2021-02-10T04:31:57.306 INFO:tasks.rados.rados.0.smithi085.stdout:1059: finishing rollback tid 0 to smithi08575397-46 2021-02-10T04:31:57.306 INFO:tasks.rados.rados.0.smithi085.stdout:1059: finishing rollback tid 1 to smithi08575397-46 2021-02-10T04:31:57.306 INFO:tasks.rados.rados.0.smithi085.stdout:1059: finishing rollback tid 2 to smithi08575397-46 2021-02-10T04:31:57.306 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 353 (ObjNum 382 snap 100 seq_num 382) dirty exists 2021-02-10T04:31:57.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:57 smithi085 conmon[29319]: cluster 2021-02-10T04:31:57.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:57 smithi085 conmon[29319]: 2021-02-10T04:31:57.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:57 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:31:57.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:57 smithi085 conmon[29319]: 04:31:56.149498+0000 mon.a (mon.0) 1046 : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2021-02-10T04:31:57.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:57 smithi085 conmon[35961]: cluster 2021-02-10T04:31:56 2021-02-10T04:31:57.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:57 smithi085 conmon[35961]: .149498+0000 mon.a (mon.0) 1046 : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2021-02-10T04:31:57.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:57 smithi164 conmon[29288]: cluster 2021-02-10T04:31:56.149498+0000 mon.a (mon.0) 1046 2021-02-10T04:31:57.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:57 smithi164 conmon[29288]: : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2021-02-10T04:31:58.145 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:31:57 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:31:58.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:58 smithi085 conmon[29319]: cluster 2021-02-10T04:31:57.157548+0000 mon.a 2021-02-10T04:31:58.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:58 smithi085 conmon[29319]: (mon.0) 1047 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2021-02-10T04:31:58.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:58 smithi085 conmon[29319]: audit 2021-02-10T04:31:57.875012+0000 mgr.y (mgr.14140) 462 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:31:58.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:58 smithi085 conmon[29319]: cluster 2021-02-10T04:31:58.046875+0000 2021-02-10T04:31:58.176 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:58 smithi085 conmon[29319]: mgr.y (mgr.14140) 463 : cluster 2021-02-10T04:31:58.176 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:58 smithi085 conmon[29319]: [DBG] pgmap v600: 168 pgs: 1 active+clean+scrubbing, 1 active+clean+snaptrim, 166 active+clean; 113 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 8.6 MiB/s rd, 1.6 MiB/s wr, 20 op/s 2021-02-10T04:31:58.180 INFO:tasks.rados.rados.0.smithi085.stdout:1058: done (14 left) 2021-02-10T04:31:58.180 INFO:tasks.rados.rados.0.smithi085.stdout:1059: done (13 left) 2021-02-10T04:31:58.180 INFO:tasks.rados.rados.0.smithi085.stdout:1060: done (12 left) 2021-02-10T04:31:58.180 INFO:tasks.rados.rados.0.smithi085.stdout:1061: snap_create 2021-02-10T04:31:58.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:31:58 smithi085 conmon[49106]: debug 2021-02-10T04:31:58.315+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:31:58.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:58 smithi085 conmon[35961]: cluster 2021-02-10T04:31:57.157548+0000 2021-02-10T04:31:58.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:58 smithi085 conmon[35961]: mon.a (mon.0) 1047 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2021-02-10T04:31:58.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:58 smithi085 conmon[35961]: audit 2021-02-10T04:31:57.875012+0000 mgr.y (mgr. 2021-02-10T04:31:58.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:58 smithi085 conmon[35961]: 14140) 462 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:31:58.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:58 smithi085 conmon[35961]: cluster 2021-02-10T04:31:58.046875+0000 2021-02-10T04:31:58.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:58 smithi085 conmon[35961]: mgr.y (mgr.14140) 463 : cluster [DBG] pgmap v600: 168 pgs: 1 active+clean+scrubbing, 1 active+clean+snaptrim, 166 active+clean; 113 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 8.6 MiB/s rd, 1.6 MiB/s wr, 20 op/s 2021-02-10T04:31:58.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:58 smithi085 conmon[35961]: 2021-02-10T04:31:58.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:58 smithi164 conmon[29288]: cluster 2021-02-10T04:31:57.157548+0000 mon.a (mon.0 2021-02-10T04:31:58.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:58 smithi164 conmon[29288]: ) 1047 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2021-02-10T04:31:58.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:58 smithi164 conmon[29288]: audit 2021-02-10T04:31:57.875012+0000 mgr.y (mgr.14140) 462 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:31:58.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:58 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:31:58.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:58 smithi164 conmon[29288]: -10T04:31:58.046875+0000 mgr.y (mgr.14140) 463 : cluster [DBG] pgmap v600: 168 pgs: 1 active+clean+scrubbing, 1 active+clean+snaptrim, 166 active+clean; 113 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 8.6 MiB/s rd, 1.6 MiB/s wr, 20 op/s 2021-02-10T04:31:58.671 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:31:59.189 INFO:tasks.rados.rados.0.smithi085.stdout:1061: done (12 left) 2021-02-10T04:31:59.189 INFO:tasks.rados.rados.0.smithi085.stdout:1062: write oid 8 current snap is 103 2021-02-10T04:31:59.189 INFO:tasks.rados.rados.0.smithi085.stdout:1062: seq_num 397 ranges {672379=724209,1865116=608472} 2021-02-10T04:31:59.189 INFO:tasks.rados.rados.0.smithi085.stdout:1062: writing smithi08575397-8 from 672379 to 1396588 tid 1 2021-02-10T04:31:59.195 INFO:tasks.rados.rados.0.smithi085.stdout:1062: writing smithi08575397-8 from 1865116 to 2473588 tid 2 2021-02-10T04:31:59.195 INFO:tasks.rados.rados.0.smithi085.stdout:1063: write (excl) oid 16 current snap is 103 2021-02-10T04:31:59.195 INFO:tasks.rados.rados.0.smithi085.stdout:1063: seq_num 398 ranges {689186=799458,1923598=752130} 2021-02-10T04:31:59.207 INFO:tasks.rados.rados.0.smithi085.stdout:1063: writing smithi08575397-16 from 689186 to 1488644 tid 1 2021-02-10T04:31:59.215 INFO:tasks.rados.rados.0.smithi085.stdout:1063: writing smithi08575397-16 from 1923598 to 2675728 tid 2 2021-02-10T04:31:59.215 INFO:tasks.rados.rados.0.smithi085.stdout:1064: copy_from oid 27 from oid 24 current snap is 103 2021-02-10T04:31:59.215 INFO:tasks.rados.rados.0.smithi085.stdout:1065: write oid 21 current snap is 103 2021-02-10T04:31:59.215 INFO:tasks.rados.rados.0.smithi085.stdout:1065: seq_num 399 ranges {705993=474707,1582080=612141} 2021-02-10T04:31:59.221 INFO:tasks.rados.rados.0.smithi085.stdout:1065: writing smithi08575397-21 from 705993 to 1180700 tid 1 2021-02-10T04:31:59.228 INFO:tasks.rados.rados.0.smithi085.stdout:1065: writing smithi08575397-21 from 1582080 to 2194221 tid 2 2021-02-10T04:31:59.228 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:31:59.228 INFO:tasks.rados.rados.0.smithi085.stdout:1062: finishing write tid 1 to smithi08575397-8 2021-02-10T04:31:59.228 INFO:tasks.rados.rados.0.smithi085.stdout:1062: finishing write tid 2 to smithi08575397-8 2021-02-10T04:31:59.228 INFO:tasks.rados.rados.0.smithi085.stdout:1062: finishing write tid 3 to smithi08575397-8 2021-02-10T04:31:59.229 INFO:tasks.rados.rados.0.smithi085.stdout:1062: finishing write tid 4 to smithi08575397-8 2021-02-10T04:31:59.229 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 271 (ObjNum 397 snap 103 seq_num 397) dirty exists 2021-02-10T04:31:59.229 INFO:tasks.rados.rados.0.smithi085.stdout:1062: left oid 8 (ObjNum 397 snap 103 seq_num 397) 2021-02-10T04:31:59.229 INFO:tasks.rados.rados.0.smithi085.stdout:1063: finishing write tid 1 to smithi08575397-16 2021-02-10T04:31:59.229 INFO:tasks.rados.rados.0.smithi085.stdout:1063: finishing write tid 2 to smithi08575397-16 2021-02-10T04:31:59.229 INFO:tasks.rados.rados.0.smithi085.stdout:1063: finishing write tid 3 to smithi08575397-16 2021-02-10T04:31:59.229 INFO:tasks.rados.rados.0.smithi085.stdout:1063: finishing write tid 4 to smithi08575397-16 2021-02-10T04:31:59.229 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 561 (ObjNum 398 snap 103 seq_num 398) dirty exists 2021-02-10T04:31:59.229 INFO:tasks.rados.rados.0.smithi085.stdout:1063: left oid 16 (ObjNum 398 snap 103 seq_num 398) 2021-02-10T04:31:59.229 INFO:tasks.rados.rados.0.smithi085.stdout:1065: finishing write tid 1 to smithi08575397-21 2021-02-10T04:31:59.229 INFO:tasks.rados.rados.0.smithi085.stdout:1062: done (15 left) 2021-02-10T04:31:59.229 INFO:tasks.rados.rados.0.smithi085.stdout:1063: done (14 left) 2021-02-10T04:31:59.229 INFO:tasks.rados.rados.0.smithi085.stdout:1066: read oid 23 snap -1 2021-02-10T04:31:59.230 INFO:tasks.rados.rados.0.smithi085.stdout:1066: expect deleted 2021-02-10T04:31:59.230 INFO:tasks.rados.rados.0.smithi085.stdout:1067: snap_remove snap 102 2021-02-10T04:31:59.619 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:59 smithi085 conmon[29319]: cluster 2021-02-10T04:31:59.619 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:59 smithi085 conmon[29319]: 2021-02-10T04:31:59.619 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:59 smithi085 conmon[29319]: 2021 2021-02-10T04:31:59.619 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:59 smithi085 conmon[29319]: - 2021-02-10T04:31:59.619 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:59 smithi085 conmon[29319]: 02 2021-02-10T04:31:59.619 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:59 smithi085 conmon[29319]: - 2021-02-10T04:31:59.619 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:59 smithi085 conmon[29319]: 10T04:31:58.178910+0000 2021-02-10T04:31:59.619 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:31:59 smithi085 conmon[29319]: mon.a (mon.0) 1048 : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2021-02-10T04:31:59.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:59 smithi085 conmon[35961]: cluster 2021-02-10T04:31:59.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:59 smithi085 conmon[35961]: 2021-02-10T04:31:59.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:59 smithi085 conmon[35961]: 2021 2021-02-10T04:31:59.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:59 smithi085 conmon[35961]: - 2021-02-10T04:31:59.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:59 smithi085 conmon[35961]: 02 2021-02-10T04:31:59.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:59 smithi085 conmon[35961]: - 2021-02-10T04:31:59.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:59 smithi085 conmon[35961]: 10 2021-02-10T04:31:59.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:59 smithi085 conmon[35961]: T 2021-02-10T04:31:59.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:59 smithi085 conmon[35961]: 04 2021-02-10T04:31:59.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:59 smithi085 conmon[35961]: : 2021-02-10T04:31:59.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:59 smithi085 conmon[35961]: 31 2021-02-10T04:31:59.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:59 smithi085 conmon[35961]: : 2021-02-10T04:31:59.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:59 smithi085 conmon[35961]: 58 2021-02-10T04:31:59.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:59 smithi085 conmon[35961]: . 2021-02-10T04:31:59.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:59 smithi085 conmon[35961]: 178910+0000 mon.a (mon.0) 1048 : 2021-02-10T04:31:59.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:31:59 smithi085 conmon[35961]: cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2021-02-10T04:31:59.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:59 smithi164 conmon[29288]: cluster 2021-02-10T04:31:59.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:59 smithi164 conmon[29288]: 2021-02-10T04:31:58. 2021-02-10T04:31:59.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:31:59 smithi164 conmon[29288]: 178910+0000 mon.a (mon.0) 1048 : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2021-02-10T04:31:59.725 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:31:59.759 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:31:59.956 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":596,"stamp":"2021-02-10T04:31:55.142094+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":118889474,"num_objects":404,"num_object_clones":9,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":404,"num_whiteouts":1,"num_read":3795,"num_read_kb":1831660,"num_write":2313,"num_write_kb":573663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":2080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3759,"ondisk_log_size":3759,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":440296,"kb_used_data":272168,"kb_used_omap":0,"kb_used_meta":168064,"kb_avail":749520920,"statfs":{"total":767960285184,"available":767509422080,"internally_reserved":0,"allocated":278700032,"data_stored":286205497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":172097536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,8,6],"upper_bound":524288},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3028577,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":-2,"num_read":50,"num_read_kb":35195,"num_write":37,"num_write_kb":7717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.035590"},"pg_stats":[{"pgid":"5.9","version":"131'3","reported_seq":"220","reported_epoch":"295","state":"active+clean","last_fresh":"2021-02-10T04:31:52.770385+0000","last_change":"2021-02-10T04:29:07.145921+0000","last_active":"2021-02-10T04:31:52.770385+0000","last_peered":"2021-02-10T04:31:52.770385+0000","last_clean":"2021-02-10T04:31:52.770385+0000","last_became_active":"2021-02-10T04:29:07.145525+0000","last_became_peered":"2021-02-10T04:29:07.145525+0000","last_unstale":"2021-02-10T04:31:52.770385+0000","last_undegraded":"2021-02-10T04:31:52.770385+0000","last_fullsized":"2021-02-10T04:31:52.770385+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:28:48.500425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:48.500425+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"63'30","reported_seq":"319","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.107999+0000","last_change":"2021-02-10T04:28:32.490336+0000","last_active":"2021-02-10T04:31:51.107999+0000","last_peered":"2021-02-10T04:31:51.107999+0000","last_clean":"2021-02-10T04:31:51.107999+0000","last_became_active":"2021-02-10T04:25:49.002058+0000","last_became_peered":"2021-02-10T04:25:49.002058+0000","last_unstale":"2021-02-10T04:31:51.107999+0000","last_undegraded":"2021-02-10T04:31:51.107999+0000","last_fullsized":"2021-02-10T04:31:51.107999+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2021-02-10T04:28:32.490246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:32.490246+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":59,"num_read_kb":42,"num_write":31,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"281","reported_epoch":"292","state":"active+clean","last_fresh":"2021-02-10T04:31:49.106588+0000","last_change":"2021-02-10T04:29:34.787349+0000","last_active":"2021-02-10T04:31:49.106588+0000","last_peered":"2021-02-10T04:31:49.106588+0000","last_clean":"2021-02-10T04:31:49.106588+0000","last_became_active":"2021-02-10T04:25:36.344641+0000","last_became_peered":"2021-02-10T04:25:36.344641+0000","last_unstale":"2021-02-10T04:31:49.106588+0000","last_undegraded":"2021-02-10T04:31:49.106588+0000","last_fullsized":"2021-02-10T04:31:49.106588+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:29:34.787265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:34.787265+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"254","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.108484+0000","last_change":"2021-02-10T04:29:19.405593+0000","last_active":"2021-02-10T04:31:51.108484+0000","last_peered":"2021-02-10T04:31:51.108484+0000","last_clean":"2021-02-10T04:31:51.108484+0000","last_became_active":"2021-02-10T04:25:53.148090+0000","last_became_peered":"2021-02-10T04:25:53.148090+0000","last_unstale":"2021-02-10T04:31:51.108484+0000","last_undegraded":"2021-02-10T04:31:51.108484+0000","last_fullsized":"2021-02-10T04:31:51.108484+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:19.405525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:29:19.405525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"213'180","reported_seq":"818","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.111805+0000","last_change":"2021-02-10T04:31:50.114529+0000","last_active":"2021-02-10T04:31:51.111805+0000","last_peered":"2021-02-10T04:31:51.111805+0000","last_clean":"2021-02-10T04:31:51.111805+0000","last_became_active":"2021-02-10T04:28:25.149288+0000","last_became_peered":"2021-02-10T04:28:25.149288+0000","last_unstale":"2021-02-10T04:31:51.111805+0000","last_undegraded":"2021-02-10T04:31:51.111805+0000","last_fullsized":"2021-02-10T04:31:51.111805+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":180,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13411920,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":114,"num_read_kb":107748,"num_write":88,"num_write_kb":23523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"}]},{"pgid":"4.b","version":"63'20","reported_seq":"301","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.144363+0000","last_change":"2021-02-10T04:28:50.696234+0000","last_active":"2021-02-10T04:31:53.144363+0000","last_peered":"2021-02-10T04:31:53.144363+0000","last_clean":"2021-02-10T04:31:53.144363+0000","last_became_active":"2021-02-10T04:25:49.004346+0000","last_became_peered":"2021-02-10T04:25:49.004346+0000","last_unstale":"2021-02-10T04:31:53.144363+0000","last_undegraded":"2021-02-10T04:31:53.144363+0000","last_fullsized":"2021-02-10T04:31:53.144363+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'20","last_scrub_stamp":"2021-02-10T04:28:50.696124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:50.696124+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":27,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"265","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.112754+0000","last_change":"2021-02-10T04:28:31.102622+0000","last_active":"2021-02-10T04:31:51.112754+0000","last_peered":"2021-02-10T04:31:51.112754+0000","last_clean":"2021-02-10T04:31:51.112754+0000","last_became_active":"2021-02-10T04:25:34.334427+0000","last_became_peered":"2021-02-10T04:25:34.334427+0000","last_unstale":"2021-02-10T04:31:51.112754+0000","last_undegraded":"2021-02-10T04:31:51.112754+0000","last_fullsized":"2021-02-10T04:31:51.112754+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:31.102558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:31.102558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"258","reported_epoch":"292","state":"active+clean","last_fresh":"2021-02-10T04:31:49.105761+0000","last_change":"2021-02-10T04:29:35.805411+0000","last_active":"2021-02-10T04:31:49.105761+0000","last_peered":"2021-02-10T04:31:49.105761+0000","last_clean":"2021-02-10T04:31:49.105761+0000","last_became_active":"2021-02-10T04:25:51.016118+0000","last_became_peered":"2021-02-10T04:25:51.016118+0000","last_unstale":"2021-02-10T04:31:49.105761+0000","last_undegraded":"2021-02-10T04:31:49.105761+0000","last_fullsized":"2021-02-10T04:31:49.105761+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:35.805326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.805326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"258","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.158716+0000","last_change":"2021-02-10T04:28:48.700480+0000","last_active":"2021-02-10T04:31:53.158716+0000","last_peered":"2021-02-10T04:31:53.158716+0000","last_clean":"2021-02-10T04:31:53.158716+0000","last_became_active":"2021-02-10T04:25:53.038249+0000","last_became_peered":"2021-02-10T04:25:53.038249+0000","last_unstale":"2021-02-10T04:31:53.158716+0000","last_undegraded":"2021-02-10T04:31:53.158716+0000","last_fullsized":"2021-02-10T04:31:53.158716+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:48.700418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:48.700418+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"259'74","reported_seq":"925","reported_epoch":"293","state":"active+clean","last_fresh":"2021-02-10T04:31:50.119131+0000","last_change":"2021-02-10T04:31:50.119400+0000","last_active":"2021-02-10T04:31:50.119131+0000","last_peered":"2021-02-10T04:31:50.119131+0000","last_clean":"2021-02-10T04:31:50.119131+0000","last_became_active":"2021-02-10T04:28:25.155557+0000","last_became_peered":"2021-02-10T04:28:25.155557+0000","last_unstale":"2021-02-10T04:31:50.119131+0000","last_undegraded":"2021-02-10T04:31:50.119131+0000","last_fullsized":"2021-02-10T04:31:50.119131+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'62","last_scrub_stamp":"2021-02-10T04:30:10.905905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:10.905905+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2437048,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":48,"num_read_kb":56455,"num_write":38,"num_write_kb":14043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"}]},{"pgid":"4.a","version":"63'29","reported_seq":"315","reported_epoch":"295","state":"active+clean","last_fresh":"2021-02-10T04:31:52.108108+0000","last_change":"2021-02-10T04:29:26.671051+0000","last_active":"2021-02-10T04:31:52.108108+0000","last_peered":"2021-02-10T04:31:52.108108+0000","last_clean":"2021-02-10T04:31:52.108108+0000","last_became_active":"2021-02-10T04:25:49.015913+0000","last_became_peered":"2021-02-10T04:25:49.015913+0000","last_unstale":"2021-02-10T04:31:52.108108+0000","last_undegraded":"2021-02-10T04:31:52.108108+0000","last_fullsized":"2021-02-10T04:31:52.108108+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'29","last_scrub_stamp":"2021-02-10T04:29:26.670950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.670950+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":56,"num_read_kb":36,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"62'2","reported_seq":"335","reported_epoch":"293","state":"active+clean","last_fresh":"2021-02-10T04:31:50.118356+0000","last_change":"2021-02-10T04:28:33.618291+0000","last_active":"2021-02-10T04:31:50.118356+0000","last_peered":"2021-02-10T04:31:50.118356+0000","last_clean":"2021-02-10T04:31:50.118356+0000","last_became_active":"2021-02-10T04:25:34.331199+0000","last_became_peered":"2021-02-10T04:25:34.331199+0000","last_unstale":"2021-02-10T04:31:50.118356+0000","last_undegraded":"2021-02-10T04:31:50.118356+0000","last_fullsized":"2021-02-10T04:31:50.118356+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2021-02-10T04:28:33.618170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:33.618170+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"253","reported_epoch":"292","state":"active+clean","last_fresh":"2021-02-10T04:31:49.106455+0000","last_change":"2021-02-10T04:28:37.645921+0000","last_active":"2021-02-10T04:31:49.106455+0000","last_peered":"2021-02-10T04:31:49.106455+0000","last_clean":"2021-02-10T04:31:49.106455+0000","last_became_active":"2021-02-10T04:25:51.025882+0000","last_became_peered":"2021-02-10T04:25:51.025882+0000","last_unstale":"2021-02-10T04:31:49.106455+0000","last_undegraded":"2021-02-10T04:31:49.106455+0000","last_fullsized":"2021-02-10T04:31:49.106455+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:37.645857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:37.645857+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"179","reported_epoch":"292","state":"active+clean","last_fresh":"2021-02-10T04:31:49.105872+0000","last_change":"2021-02-10T04:29:24.731524+0000","last_active":"2021-02-10T04:31:49.105872+0000","last_peered":"2021-02-10T04:31:49.105872+0000","last_clean":"2021-02-10T04:31:49.105872+0000","last_became_active":"2021-02-10T04:29:07.162878+0000","last_became_peered":"2021-02-10T04:29:07.162878+0000","last_unstale":"2021-02-10T04:31:49.105872+0000","last_undegraded":"2021-02-10T04:31:49.105872+0000","last_fullsized":"2021-02-10T04:31:49.105872+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.731458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.731458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"213'82","reported_seq":"713","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.159215+0000","last_change":"2021-02-10T04:31:53.159215+0000","last_active":"2021-02-10T04:31:53.159215+0000","last_peered":"2021-02-10T04:31:53.159215+0000","last_clean":"2021-02-10T04:31:53.159215+0000","last_became_active":"2021-02-10T04:28:25.150246+0000","last_became_peered":"2021-02-10T04:28:25.150246+0000","last_unstale":"2021-02-10T04:31:53.159215+0000","last_undegraded":"2021-02-10T04:31:53.159215+0000","last_fullsized":"2021-02-10T04:31:53.159215+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'63","last_scrub_stamp":"2021-02-10T04:29:39.700115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:29:39.700115+0000","log_size":82,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7025293,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":13996,"num_write":41,"num_write_kb":22530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c5","length":"1"}]},{"pgid":"4.d","version":"113'33","reported_seq":"305","reported_epoch":"293","state":"active+clean","last_fresh":"2021-02-10T04:31:50.108351+0000","last_change":"2021-02-10T04:28:49.515083+0000","last_active":"2021-02-10T04:31:50.108351+0000","last_peered":"2021-02-10T04:31:50.108351+0000","last_clean":"2021-02-10T04:31:50.108351+0000","last_became_active":"2021-02-10T04:25:49.011699+0000","last_became_peered":"2021-02-10T04:25:49.011699+0000","last_unstale":"2021-02-10T04:31:50.108351+0000","last_undegraded":"2021-02-10T04:31:50.108351+0000","last_fullsized":"2021-02-10T04:31:50.108351+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'33","last_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:28:49.514984+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"175","reported_epoch":"293","state":"active+clean","last_fresh":"2021-02-10T04:31:50.107252+0000","last_change":"2021-02-10T04:29:07.146266+0000","last_active":"2021-02-10T04:31:50.107252+0000","last_peered":"2021-02-10T04:31:50.107252+0000","last_clean":"2021-02-10T04:31:50.107252+0000","last_became_active":"2021-02-10T04:29:07.145616+0000","last_became_peered":"2021-02-10T04:29:07.145616+0000","last_unstale":"2021-02-10T04:31:50.107252+0000","last_undegraded":"2021-02-10T04:31:50.107252+0000","last_fullsized":"2021-02-10T04:31:50.107252+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:44.410817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:44.410817+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"256","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.112382+0000","last_change":"2021-02-10T04:29:35.958357+0000","last_active":"2021-02-10T04:31:51.112382+0000","last_peered":"2021-02-10T04:31:51.112382+0000","last_clean":"2021-02-10T04:31:51.112382+0000","last_became_active":"2021-02-10T04:25:51.028302+0000","last_became_peered":"2021-02-10T04:25:51.028302+0000","last_unstale":"2021-02-10T04:31:51.112382+0000","last_undegraded":"2021-02-10T04:31:51.112382+0000","last_fullsized":"2021-02-10T04:31:51.112382+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:35.958255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.958255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"251","reported_epoch":"292","state":"active+clean","last_fresh":"2021-02-10T04:31:49.106909+0000","last_change":"2021-02-10T04:28:32.623869+0000","last_active":"2021-02-10T04:31:49.106909+0000","last_peered":"2021-02-10T04:31:49.106909+0000","last_clean":"2021-02-10T04:31:49.106909+0000","last_became_active":"2021-02-10T04:25:53.038218+0000","last_became_peered":"2021-02-10T04:25:53.038218+0000","last_unstale":"2021-02-10T04:31:49.106909+0000","last_undegraded":"2021-02-10T04:31:49.106909+0000","last_fullsized":"2021-02-10T04:31:49.106909+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:32.623791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:32.623791+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"197'106","reported_seq":"1469","reported_epoch":"292","state":"active+clean+scrubbing","last_fresh":"2021-02-10T04:31:49.106100+0000","last_change":"2021-02-10T04:30:15.066819+0000","last_active":"2021-02-10T04:31:49.106100+0000","last_peered":"2021-02-10T04:31:49.106100+0000","last_clean":"2021-02-10T04:31:49.106100+0000","last_became_active":"2021-02-10T04:28:25.157931+0000","last_became_peered":"2021-02-10T04:28:25.157931+0000","last_unstale":"2021-02-10T04:31:49.106100+0000","last_undegraded":"2021-02-10T04:31:49.106100+0000","last_fullsized":"2021-02-10T04:31:49.106100+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":106,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":50,"stat_sum":{"num_bytes":12437781,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":69,"num_read_kb":61326,"num_write":71,"num_write_kb":20689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"197","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.107977+0000","last_change":"2021-02-10T04:29:37.336179+0000","last_active":"2021-02-10T04:31:51.107977+0000","last_peered":"2021-02-10T04:31:51.107977+0000","last_clean":"2021-02-10T04:31:51.107977+0000","last_became_active":"2021-02-10T04:28:49.046469+0000","last_became_peered":"2021-02-10T04:28:49.046469+0000","last_unstale":"2021-02-10T04:31:51.107977+0000","last_undegraded":"2021-02-10T04:31:51.107977+0000","last_fullsized":"2021-02-10T04:31:51.107977+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:37.336092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:37.336092+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"268","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.145749+0000","last_change":"2021-02-10T04:29:07.160624+0000","last_active":"2021-02-10T04:31:53.145749+0000","last_peered":"2021-02-10T04:31:53.145749+0000","last_clean":"2021-02-10T04:31:53.145749+0000","last_became_active":"2021-02-10T04:29:07.159274+0000","last_became_peered":"2021-02-10T04:29:07.159274+0000","last_unstale":"2021-02-10T04:31:53.145749+0000","last_undegraded":"2021-02-10T04:31:53.145749+0000","last_fullsized":"2021-02-10T04:31:53.145749+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:30.424326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:28:30.424326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"62'13","reported_seq":"278","reported_epoch":"293","state":"active+clean","last_fresh":"2021-02-10T04:31:50.108087+0000","last_change":"2021-02-10T04:28:41.427119+0000","last_active":"2021-02-10T04:31:50.108087+0000","last_peered":"2021-02-10T04:31:50.108087+0000","last_clean":"2021-02-10T04:31:50.108087+0000","last_became_active":"2021-02-10T04:25:49.001725+0000","last_became_peered":"2021-02-10T04:25:49.001725+0000","last_unstale":"2021-02-10T04:31:50.108087+0000","last_undegraded":"2021-02-10T04:31:50.108087+0000","last_fullsized":"2021-02-10T04:31:50.108087+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'13","last_scrub_stamp":"2021-02-10T04:28:41.427050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:41.427050+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"267","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.111743+0000","last_change":"2021-02-10T04:29:18.107974+0000","last_active":"2021-02-10T04:31:51.111743+0000","last_peered":"2021-02-10T04:31:51.111743+0000","last_clean":"2021-02-10T04:31:51.111743+0000","last_became_active":"2021-02-10T04:25:36.333883+0000","last_became_peered":"2021-02-10T04:25:36.333883+0000","last_unstale":"2021-02-10T04:31:51.111743+0000","last_undegraded":"2021-02-10T04:31:51.111743+0000","last_fullsized":"2021-02-10T04:31:51.111743+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:18.107867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:18.107867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"58'2","reported_seq":"327","reported_epoch":"292","state":"active+clean","last_fresh":"2021-02-10T04:31:49.106312+0000","last_change":"2021-02-10T04:28:35.648670+0000","last_active":"2021-02-10T04:31:49.106312+0000","last_peered":"2021-02-10T04:31:49.106312+0000","last_clean":"2021-02-10T04:31:49.106312+0000","last_became_active":"2021-02-10T04:25:51.025268+0000","last_became_peered":"2021-02-10T04:25:51.025268+0000","last_unstale":"2021-02-10T04:31:49.106312+0000","last_undegraded":"2021-02-10T04:31:49.106312+0000","last_fullsized":"2021-02-10T04:31:49.106312+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:28:35.648602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:35.648602+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"253","reported_epoch":"293","state":"active+clean","last_fresh":"2021-02-10T04:31:50.108129+0000","last_change":"2021-02-10T04:29:38.873167+0000","last_active":"2021-02-10T04:31:50.108129+0000","last_peered":"2021-02-10T04:31:50.108129+0000","last_clean":"2021-02-10T04:31:50.108129+0000","last_became_active":"2021-02-10T04:25:53.043027+0000","last_became_peered":"2021-02-10T04:25:53.043027+0000","last_unstale":"2021-02-10T04:31:50.108129+0000","last_undegraded":"2021-02-10T04:31:50.108129+0000","last_fullsized":"2021-02-10T04:31:50.108129+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:38.873102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.873102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"297'144","reported_seq":"2875","reported_epoch":"297","state":"active+clean","last_fresh":"2021-02-10T04:31:54.415755+0000","last_change":"2021-02-10T04:31:53.159079+0000","last_active":"2021-02-10T04:31:54.415755+0000","last_peered":"2021-02-10T04:31:54.415755+0000","last_clean":"2021-02-10T04:31:54.415755+0000","last_became_active":"2021-02-10T04:28:25.149226+0000","last_became_peered":"2021-02-10T04:28:25.149226+0000","last_unstale":"2021-02-10T04:31:54.415755+0000","last_undegraded":"2021-02-10T04:31:54.415755+0000","last_fullsized":"2021-02-10T04:31:54.415755+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":144,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8588410,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":159,"num_read_kb":176382,"num_write":80,"num_write_kb":30825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c5","length":"1"}]},{"pgid":"1.a","version":"0'0","reported_seq":"197","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.108363+0000","last_change":"2021-02-10T04:29:29.384240+0000","last_active":"2021-02-10T04:31:51.108363+0000","last_peered":"2021-02-10T04:31:51.108363+0000","last_clean":"2021-02-10T04:31:51.108363+0000","last_became_active":"2021-02-10T04:28:49.050935+0000","last_became_peered":"2021-02-10T04:28:49.050935+0000","last_unstale":"2021-02-10T04:31:51.108363+0000","last_undegraded":"2021-02-10T04:31:51.108363+0000","last_fullsized":"2021-02-10T04:31:51.108363+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.384170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.384170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"62'17","reported_seq":"287","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.113091+0000","last_change":"2021-02-10T04:29:37.933295+0000","last_active":"2021-02-10T04:31:51.113091+0000","last_peered":"2021-02-10T04:31:51.113091+0000","last_clean":"2021-02-10T04:31:51.113091+0000","last_became_active":"2021-02-10T04:25:49.004933+0000","last_became_peered":"2021-02-10T04:25:49.004933+0000","last_unstale":"2021-02-10T04:31:51.113091+0000","last_undegraded":"2021-02-10T04:31:51.113091+0000","last_fullsized":"2021-02-10T04:31:51.113091+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'17","last_scrub_stamp":"2021-02-10T04:29:37.933206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:37.933206+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"283","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.111607+0000","last_change":"2021-02-10T04:28:39.128857+0000","last_active":"2021-02-10T04:31:51.111607+0000","last_peered":"2021-02-10T04:31:51.111607+0000","last_clean":"2021-02-10T04:31:51.111607+0000","last_became_active":"2021-02-10T04:28:21.130021+0000","last_became_peered":"2021-02-10T04:28:21.130021+0000","last_unstale":"2021-02-10T04:31:51.111607+0000","last_undegraded":"2021-02-10T04:31:51.111607+0000","last_fullsized":"2021-02-10T04:31:51.111607+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_clean_scrub_stamp":"2021-02-10T04:28:39.128781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"78'3","reported_seq":"330","reported_epoch":"293","state":"active+clean","last_fresh":"2021-02-10T04:31:50.108097+0000","last_change":"2021-02-10T04:28:43.509446+0000","last_active":"2021-02-10T04:31:50.108097+0000","last_peered":"2021-02-10T04:31:50.108097+0000","last_clean":"2021-02-10T04:31:50.108097+0000","last_became_active":"2021-02-10T04:25:51.025218+0000","last_became_peered":"2021-02-10T04:25:51.025218+0000","last_unstale":"2021-02-10T04:31:50.108097+0000","last_undegraded":"2021-02-10T04:31:50.108097+0000","last_fullsized":"2021-02-10T04:31:50.108097+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:28:43.509353+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:28:43.509353+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"254","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.108947+0000","last_change":"2021-02-10T04:28:45.481538+0000","last_active":"2021-02-10T04:31:51.108947+0000","last_peered":"2021-02-10T04:31:51.108947+0000","last_clean":"2021-02-10T04:31:51.108947+0000","last_became_active":"2021-02-10T04:25:53.042697+0000","last_became_peered":"2021-02-10T04:25:53.042697+0000","last_unstale":"2021-02-10T04:31:51.108947+0000","last_undegraded":"2021-02-10T04:31:51.108947+0000","last_fullsized":"2021-02-10T04:31:51.108947+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:45.481472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:28:45.481472+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"290'166","reported_seq":"1812","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.148148+0000","last_change":"2021-02-10T04:31:53.148507+0000","last_active":"2021-02-10T04:31:53.148148+0000","last_peered":"2021-02-10T04:31:53.148148+0000","last_clean":"2021-02-10T04:31:53.148148+0000","last_became_active":"2021-02-10T04:28:25.148500+0000","last_became_peered":"2021-02-10T04:28:25.148500+0000","last_unstale":"2021-02-10T04:31:53.148148+0000","last_undegraded":"2021-02-10T04:31:53.148148+0000","last_fullsized":"2021-02-10T04:31:53.148148+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'110","last_scrub_stamp":"2021-02-10T04:30:18.463782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:18.463782+0000","log_size":166,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10201948,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":150,"num_read_kb":165813,"num_write":93,"num_write_kb":36206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c5","length":"1"}]},{"pgid":"4.e","version":"86'31","reported_seq":"309","reported_epoch":"293","state":"active+clean","last_fresh":"2021-02-10T04:31:50.107983+0000","last_change":"2021-02-10T04:28:11.693702+0000","last_active":"2021-02-10T04:31:50.107983+0000","last_peered":"2021-02-10T04:31:50.107983+0000","last_clean":"2021-02-10T04:31:50.107983+0000","last_became_active":"2021-02-10T04:25:49.011093+0000","last_became_peered":"2021-02-10T04:25:49.011093+0000","last_unstale":"2021-02-10T04:31:50.107983+0000","last_undegraded":"2021-02-10T04:31:50.107983+0000","last_fullsized":"2021-02-10T04:31:50.107983+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'31","last_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:28:11.693616+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":28,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"241","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.140762+0000","last_change":"2021-02-10T04:29:31.683314+0000","last_active":"2021-02-10T04:31:53.140762+0000","last_peered":"2021-02-10T04:31:53.140762+0000","last_clean":"2021-02-10T04:31:53.140762+0000","last_became_active":"2021-02-10T04:29:07.160593+0000","last_became_peered":"2021-02-10T04:29:07.160593+0000","last_unstale":"2021-02-10T04:31:53.140762+0000","last_undegraded":"2021-02-10T04:31:53.140762+0000","last_fullsized":"2021-02-10T04:31:53.140762+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:31.683213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.683213+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"261","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.109128+0000","last_change":"2021-02-10T04:30:10.487631+0000","last_active":"2021-02-10T04:31:51.109128+0000","last_peered":"2021-02-10T04:31:51.109128+0000","last_clean":"2021-02-10T04:31:51.109128+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:31:51.109128+0000","last_undegraded":"2021-02-10T04:31:51.109128+0000","last_fullsized":"2021-02-10T04:31:51.109128+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:10.487528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:30:10.487528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"250","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.149191+0000","last_change":"2021-02-10T04:27:52.383262+0000","last_active":"2021-02-10T04:31:53.149191+0000","last_peered":"2021-02-10T04:31:53.149191+0000","last_clean":"2021-02-10T04:31:53.149191+0000","last_became_active":"2021-02-10T04:25:53.148054+0000","last_became_peered":"2021-02-10T04:25:53.148054+0000","last_unstale":"2021-02-10T04:31:53.149191+0000","last_undegraded":"2021-02-10T04:31:53.149191+0000","last_fullsized":"2021-02-10T04:31:53.149191+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:52.383188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:52.383188+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"290'201","reported_seq":"975","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.147539+0000","last_change":"2021-02-10T04:31:53.147968+0000","last_active":"2021-02-10T04:31:53.147539+0000","last_peered":"2021-02-10T04:31:53.147539+0000","last_clean":"2021-02-10T04:31:53.147539+0000","last_became_active":"2021-02-10T04:28:25.152030+0000","last_became_peered":"2021-02-10T04:28:25.152030+0000","last_unstale":"2021-02-10T04:31:53.147539+0000","last_undegraded":"2021-02-10T04:31:53.147539+0000","last_fullsized":"2021-02-10T04:31:53.147539+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":201,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6605151,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":148,"num_read_kb":140533,"num_write":100,"num_write_kb":44746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c5","length":"1"}]},{"pgid":"5.0","version":"158'3","reported_seq":"333","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.145212+0000","last_change":"2021-02-10T04:29:34.388978+0000","last_active":"2021-02-10T04:31:53.145212+0000","last_peered":"2021-02-10T04:31:53.145212+0000","last_clean":"2021-02-10T04:31:53.145212+0000","last_became_active":"2021-02-10T04:25:51.031906+0000","last_became_peered":"2021-02-10T04:25:51.031906+0000","last_unstale":"2021-02-10T04:31:53.145212+0000","last_undegraded":"2021-02-10T04:31:53.145212+0000","last_fullsized":"2021-02-10T04:31:53.145212+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'3","last_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_deep_scrub":"158'3","last_deep_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_clean_scrub_stamp":"2021-02-10T04:29:34.388912+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"63'26","reported_seq":"305","reported_epoch":"293","state":"active+clean","last_fresh":"2021-02-10T04:31:50.106977+0000","last_change":"2021-02-10T04:29:36.813190+0000","last_active":"2021-02-10T04:31:50.106977+0000","last_peered":"2021-02-10T04:31:50.106977+0000","last_clean":"2021-02-10T04:31:50.106977+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:31:50.106977+0000","last_undegraded":"2021-02-10T04:31:50.106977+0000","last_fullsized":"2021-02-10T04:31:50.106977+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:29:36.813083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.813083+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":51,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"298","reported_epoch":"295","state":"active+clean","last_fresh":"2021-02-10T04:31:52.110391+0000","last_change":"2021-02-10T04:29:28.632178+0000","last_active":"2021-02-10T04:31:52.110391+0000","last_peered":"2021-02-10T04:31:52.110391+0000","last_clean":"2021-02-10T04:31:52.110391+0000","last_became_active":"2021-02-10T04:28:39.041833+0000","last_became_peered":"2021-02-10T04:28:39.041833+0000","last_unstale":"2021-02-10T04:31:52.110391+0000","last_undegraded":"2021-02-10T04:31:52.110391+0000","last_fullsized":"2021-02-10T04:31:52.110391+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.632085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.632085+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"182","reported_epoch":"295","state":"active+clean","last_fresh":"2021-02-10T04:31:52.108527+0000","last_change":"2021-02-10T04:29:24.658465+0000","last_active":"2021-02-10T04:31:52.108527+0000","last_peered":"2021-02-10T04:31:52.108527+0000","last_clean":"2021-02-10T04:31:52.108527+0000","last_became_active":"2021-02-10T04:29:07.148527+0000","last_became_peered":"2021-02-10T04:29:07.148527+0000","last_unstale":"2021-02-10T04:31:52.108527+0000","last_undegraded":"2021-02-10T04:31:52.108527+0000","last_fullsized":"2021-02-10T04:31:52.108527+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.658374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.658374+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"297'165","reported_seq":"1487","reported_epoch":"297","state":"active+clean","last_fresh":"2021-02-10T04:31:54.632627+0000","last_change":"2021-02-10T04:31:53.147688+0000","last_active":"2021-02-10T04:31:54.632627+0000","last_peered":"2021-02-10T04:31:54.632627+0000","last_clean":"2021-02-10T04:31:54.632627+0000","last_became_active":"2021-02-10T04:28:25.157306+0000","last_became_peered":"2021-02-10T04:28:25.157306+0000","last_unstale":"2021-02-10T04:31:54.632627+0000","last_undegraded":"2021-02-10T04:31:54.632627+0000","last_fullsized":"2021-02-10T04:31:54.632627+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":165,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2353046,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":163,"num_read_kb":152001,"num_write":84,"num_write_kb":20426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c5","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":"199","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.149260+0000","last_change":"2021-02-10T04:30:12.452072+0000","last_active":"2021-02-10T04:31:53.149260+0000","last_peered":"2021-02-10T04:31:53.149260+0000","last_clean":"2021-02-10T04:31:53.149260+0000","last_became_active":"2021-02-10T04:28:49.048525+0000","last_became_peered":"2021-02-10T04:28:49.048525+0000","last_unstale":"2021-02-10T04:31:53.149260+0000","last_undegraded":"2021-02-10T04:31:53.149260+0000","last_fullsized":"2021-02-10T04:31:53.149260+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:12.452000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:30:12.452000+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"255","reported_epoch":"293","state":"active+clean","last_fresh":"2021-02-10T04:31:50.108058+0000","last_change":"2021-02-10T04:29:40.859352+0000","last_active":"2021-02-10T04:31:50.108058+0000","last_peered":"2021-02-10T04:31:50.108058+0000","last_clean":"2021-02-10T04:31:50.108058+0000","last_became_active":"2021-02-10T04:25:51.018706+0000","last_became_peered":"2021-02-10T04:25:51.018706+0000","last_unstale":"2021-02-10T04:31:50.108058+0000","last_undegraded":"2021-02-10T04:31:50.108058+0000","last_fullsized":"2021-02-10T04:31:50.108058+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:40.859274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:40.859274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"152'48","reported_seq":"343","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.145074+0000","last_change":"2021-02-10T04:29:28.398522+0000","last_active":"2021-02-10T04:31:53.145074+0000","last_peered":"2021-02-10T04:31:53.145074+0000","last_clean":"2021-02-10T04:31:53.145074+0000","last_became_active":"2021-02-10T04:25:48.997481+0000","last_became_peered":"2021-02-10T04:25:48.997481+0000","last_unstale":"2021-02-10T04:31:53.145074+0000","last_undegraded":"2021-02-10T04:31:53.145074+0000","last_fullsized":"2021-02-10T04:31:53.145074+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'48","last_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_deep_scrub":"152'48","last_deep_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.398429+0000","log_size":48,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":70,"num_read_kb":48,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"279","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.112906+0000","last_change":"2021-02-10T04:28:35.010774+0000","last_active":"2021-02-10T04:31:51.112906+0000","last_peered":"2021-02-10T04:31:51.112906+0000","last_clean":"2021-02-10T04:31:51.112906+0000","last_became_active":"2021-02-10T04:28:21.133248+0000","last_became_peered":"2021-02-10T04:28:21.133248+0000","last_unstale":"2021-02-10T04:31:51.112906+0000","last_undegraded":"2021-02-10T04:31:51.112906+0000","last_fullsized":"2021-02-10T04:31:51.112906+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:28:35.010709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:35.010709+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"253","reported_epoch":"293","state":"active+clean","last_fresh":"2021-02-10T04:31:50.107776+0000","last_change":"2021-02-10T04:28:05.641668+0000","last_active":"2021-02-10T04:31:50.107776+0000","last_peered":"2021-02-10T04:31:50.107776+0000","last_clean":"2021-02-10T04:31:50.107776+0000","last_became_active":"2021-02-10T04:25:53.045085+0000","last_became_peered":"2021-02-10T04:25:53.045085+0000","last_unstale":"2021-02-10T04:31:50.107776+0000","last_undegraded":"2021-02-10T04:31:50.107776+0000","last_fullsized":"2021-02-10T04:31:50.107776+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:05.641577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:05.641577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"297'334","reported_seq":"997","reported_epoch":"297","state":"active+clean","last_fresh":"2021-02-10T04:31:54.416755+0000","last_change":"2021-02-10T04:31:53.150514+0000","last_active":"2021-02-10T04:31:54.416755+0000","last_peered":"2021-02-10T04:31:54.416755+0000","last_clean":"2021-02-10T04:31:54.416755+0000","last_became_active":"2021-02-10T04:28:25.161622+0000","last_became_peered":"2021-02-10T04:28:25.161622+0000","last_unstale":"2021-02-10T04:31:54.416755+0000","last_undegraded":"2021-02-10T04:31:54.416755+0000","last_fullsized":"2021-02-10T04:31:54.416755+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":334,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10892210,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":232,"num_read_kb":259937,"num_write":168,"num_write_kb":56566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c5","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":"196","reported_epoch":"293","state":"active+clean","last_fresh":"2021-02-10T04:31:50.107608+0000","last_change":"2021-02-10T04:29:33.839409+0000","last_active":"2021-02-10T04:31:50.107608+0000","last_peered":"2021-02-10T04:31:50.107608+0000","last_clean":"2021-02-10T04:31:50.107608+0000","last_became_active":"2021-02-10T04:28:49.045505+0000","last_became_peered":"2021-02-10T04:28:49.045505+0000","last_unstale":"2021-02-10T04:31:50.107608+0000","last_undegraded":"2021-02-10T04:31:50.107608+0000","last_fullsized":"2021-02-10T04:31:50.107608+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:33.839335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:33.839335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"281","reported_epoch":"295","state":"active+clean","last_fresh":"2021-02-10T04:31:52.108263+0000","last_change":"2021-02-10T04:30:57.722450+0000","last_active":"2021-02-10T04:31:52.108263+0000","last_peered":"2021-02-10T04:31:52.108263+0000","last_clean":"2021-02-10T04:31:52.108263+0000","last_became_active":"2021-02-10T04:25:51.025636+0000","last_became_peered":"2021-02-10T04:25:51.025636+0000","last_unstale":"2021-02-10T04:31:52.108263+0000","last_undegraded":"2021-02-10T04:31:52.108263+0000","last_fullsized":"2021-02-10T04:31:52.108263+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:37.215333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:37.215333+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"63'27","reported_seq":"315","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.144253+0000","last_change":"2021-02-10T04:28:36.754064+0000","last_active":"2021-02-10T04:31:53.144253+0000","last_peered":"2021-02-10T04:31:53.144253+0000","last_clean":"2021-02-10T04:31:53.144253+0000","last_became_active":"2021-02-10T04:25:49.002937+0000","last_became_peered":"2021-02-10T04:25:49.002937+0000","last_unstale":"2021-02-10T04:31:53.144253+0000","last_undegraded":"2021-02-10T04:31:53.144253+0000","last_fullsized":"2021-02-10T04:31:53.144253+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'27","last_scrub_stamp":"2021-02-10T04:28:36.753994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:36.753994+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":38,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"194","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.143781+0000","last_change":"2021-02-10T04:29:35.711635+0000","last_active":"2021-02-10T04:31:53.143781+0000","last_peered":"2021-02-10T04:31:53.143781+0000","last_clean":"2021-02-10T04:31:53.143781+0000","last_became_active":"2021-02-10T04:29:07.161341+0000","last_became_peered":"2021-02-10T04:29:07.161341+0000","last_unstale":"2021-02-10T04:31:53.143781+0000","last_undegraded":"2021-02-10T04:31:53.143781+0000","last_fullsized":"2021-02-10T04:31:53.143781+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_deep_scrub":"53'2","last_deep_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.711557+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"254","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.111706+0000","last_change":"2021-02-10T04:29:26.978134+0000","last_active":"2021-02-10T04:31:51.111706+0000","last_peered":"2021-02-10T04:31:51.111706+0000","last_clean":"2021-02-10T04:31:51.111706+0000","last_became_active":"2021-02-10T04:25:53.149552+0000","last_became_peered":"2021-02-10T04:25:53.149552+0000","last_unstale":"2021-02-10T04:31:51.111706+0000","last_undegraded":"2021-02-10T04:31:51.111706+0000","last_fullsized":"2021-02-10T04:31:51.111706+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:26.978062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.978062+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"290'240","reported_seq":"876","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.108153+0000","last_change":"2021-02-10T04:31:50.108459+0000","last_active":"2021-02-10T04:31:51.108153+0000","last_peered":"2021-02-10T04:31:51.108153+0000","last_clean":"2021-02-10T04:31:51.108153+0000","last_became_active":"2021-02-10T04:28:25.146766+0000","last_became_peered":"2021-02-10T04:28:25.146766+0000","last_unstale":"2021-02-10T04:31:51.108153+0000","last_undegraded":"2021-02-10T04:31:51.108153+0000","last_fullsized":"2021-02-10T04:31:51.108153+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":240,"ondisk_log_size":240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2218590,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":130,"num_read_kb":110106,"num_write":116,"num_write_kb":49232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":"279","reported_epoch":"295","state":"active+clean","last_fresh":"2021-02-10T04:31:52.109619+0000","last_change":"2021-02-10T04:29:38.793837+0000","last_active":"2021-02-10T04:31:52.109619+0000","last_peered":"2021-02-10T04:31:52.109619+0000","last_clean":"2021-02-10T04:31:52.109619+0000","last_became_active":"2021-02-10T04:28:49.054638+0000","last_became_peered":"2021-02-10T04:28:49.054638+0000","last_unstale":"2021-02-10T04:31:52.109619+0000","last_undegraded":"2021-02-10T04:31:52.109619+0000","last_fullsized":"2021-02-10T04:31:52.109619+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:38.793760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.793760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"256","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.109204+0000","last_change":"2021-02-10T04:29:40.451948+0000","last_active":"2021-02-10T04:31:51.109204+0000","last_peered":"2021-02-10T04:31:51.109204+0000","last_clean":"2021-02-10T04:31:51.109204+0000","last_became_active":"2021-02-10T04:25:51.032201+0000","last_became_peered":"2021-02-10T04:25:51.032201+0000","last_unstale":"2021-02-10T04:31:51.109204+0000","last_undegraded":"2021-02-10T04:31:51.109204+0000","last_fullsized":"2021-02-10T04:31:51.109204+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:40.451832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:29:40.451832+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"62'16","reported_seq":"286","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.158769+0000","last_change":"2021-02-10T04:28:34.694585+0000","last_active":"2021-02-10T04:31:53.158769+0000","last_peered":"2021-02-10T04:31:53.158769+0000","last_clean":"2021-02-10T04:31:53.158769+0000","last_became_active":"2021-02-10T04:25:49.003521+0000","last_became_peered":"2021-02-10T04:25:49.003521+0000","last_unstale":"2021-02-10T04:31:53.158769+0000","last_undegraded":"2021-02-10T04:31:53.158769+0000","last_fullsized":"2021-02-10T04:31:53.158769+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'16","last_scrub_stamp":"2021-02-10T04:28:34.694486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:34.694486+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":16,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"234","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.108111+0000","last_change":"2021-02-10T04:30:14.204629+0000","last_active":"2021-02-10T04:31:51.108111+0000","last_peered":"2021-02-10T04:31:51.108111+0000","last_clean":"2021-02-10T04:31:51.108111+0000","last_became_active":"2021-02-10T04:29:07.159327+0000","last_became_peered":"2021-02-10T04:29:07.159327+0000","last_unstale":"2021-02-10T04:31:51.108111+0000","last_undegraded":"2021-02-10T04:31:51.108111+0000","last_fullsized":"2021-02-10T04:31:51.108111+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:14.204533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:30:14.204533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"260","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.142056+0000","last_change":"2021-02-10T04:29:21.620573+0000","last_active":"2021-02-10T04:31:53.142056+0000","last_peered":"2021-02-10T04:31:53.142056+0000","last_clean":"2021-02-10T04:31:53.142056+0000","last_became_active":"2021-02-10T04:25:49.001782+0000","last_became_peered":"2021-02-10T04:25:49.001782+0000","last_unstale":"2021-02-10T04:31:53.142056+0000","last_undegraded":"2021-02-10T04:31:53.142056+0000","last_fullsized":"2021-02-10T04:31:53.142056+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:21.620482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:21.620482+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"254","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.112460+0000","last_change":"2021-02-10T04:29:23.098781+0000","last_active":"2021-02-10T04:31:51.112460+0000","last_peered":"2021-02-10T04:31:51.112460+0000","last_clean":"2021-02-10T04:31:51.112460+0000","last_became_active":"2021-02-10T04:25:53.040390+0000","last_became_peered":"2021-02-10T04:25:53.040390+0000","last_unstale":"2021-02-10T04:31:51.112460+0000","last_undegraded":"2021-02-10T04:31:51.112460+0000","last_fullsized":"2021-02-10T04:31:51.112460+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.098681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.098681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"215'85","reported_seq":"709","reported_epoch":"293","state":"active+clean","last_fresh":"2021-02-10T04:31:50.118521+0000","last_change":"2021-02-10T04:31:50.118859+0000","last_active":"2021-02-10T04:31:50.118521+0000","last_peered":"2021-02-10T04:31:50.118521+0000","last_clean":"2021-02-10T04:31:50.118521+0000","last_became_active":"2021-02-10T04:29:07.163659+0000","last_became_peered":"2021-02-10T04:29:07.163659+0000","last_unstale":"2021-02-10T04:31:50.118521+0000","last_undegraded":"2021-02-10T04:31:50.118521+0000","last_fullsized":"2021-02-10T04:31:50.118521+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2201750,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":44,"num_read_kb":45112,"num_write":42,"num_write_kb":10011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"}]},{"pgid":"1.3","version":"0'0","reported_seq":"197","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.111722+0000","last_change":"2021-02-10T04:29:28.928994+0000","last_active":"2021-02-10T04:31:51.111722+0000","last_peered":"2021-02-10T04:31:51.111722+0000","last_clean":"2021-02-10T04:31:51.111722+0000","last_became_active":"2021-02-10T04:28:49.043854+0000","last_became_peered":"2021-02-10T04:28:49.043854+0000","last_unstale":"2021-02-10T04:31:51.111722+0000","last_undegraded":"2021-02-10T04:31:51.111722+0000","last_fullsized":"2021-02-10T04:31:51.111722+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.928906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.928906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"179","reported_epoch":"292","state":"active+clean","last_fresh":"2021-02-10T04:31:49.106340+0000","last_change":"2021-02-10T04:29:29.805554+0000","last_active":"2021-02-10T04:31:49.106340+0000","last_peered":"2021-02-10T04:31:49.106340+0000","last_clean":"2021-02-10T04:31:49.106340+0000","last_became_active":"2021-02-10T04:29:07.148512+0000","last_became_peered":"2021-02-10T04:29:07.148512+0000","last_unstale":"2021-02-10T04:31:49.106340+0000","last_undegraded":"2021-02-10T04:31:49.106340+0000","last_fullsized":"2021-02-10T04:31:49.106340+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.805432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.805432+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"62'15","reported_seq":"283","reported_epoch":"295","state":"active+clean","last_fresh":"2021-02-10T04:31:52.109105+0000","last_change":"2021-02-10T04:28:40.322577+0000","last_active":"2021-02-10T04:31:52.109105+0000","last_peered":"2021-02-10T04:31:52.109105+0000","last_clean":"2021-02-10T04:31:52.109105+0000","last_became_active":"2021-02-10T04:25:49.002471+0000","last_became_peered":"2021-02-10T04:25:49.002471+0000","last_unstale":"2021-02-10T04:31:52.109105+0000","last_undegraded":"2021-02-10T04:31:52.109105+0000","last_fullsized":"2021-02-10T04:31:52.109105+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:28:40.322504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:40.322504+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":15,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"263","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.140811+0000","last_change":"2021-02-10T04:29:36.663903+0000","last_active":"2021-02-10T04:31:53.140811+0000","last_peered":"2021-02-10T04:31:53.140811+0000","last_clean":"2021-02-10T04:31:53.140811+0000","last_became_active":"2021-02-10T04:29:07.158075+0000","last_became_peered":"2021-02-10T04:29:07.158075+0000","last_unstale":"2021-02-10T04:31:53.140811+0000","last_undegraded":"2021-02-10T04:31:53.140811+0000","last_fullsized":"2021-02-10T04:31:53.140811+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.663840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.663840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"281","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.108438+0000","last_change":"2021-02-10T04:28:34.482961+0000","last_active":"2021-02-10T04:31:51.108438+0000","last_peered":"2021-02-10T04:31:51.108438+0000","last_clean":"2021-02-10T04:31:51.108438+0000","last_became_active":"2021-02-10T04:28:21.123027+0000","last_became_peered":"2021-02-10T04:28:21.123027+0000","last_unstale":"2021-02-10T04:31:51.108438+0000","last_undegraded":"2021-02-10T04:31:51.108438+0000","last_fullsized":"2021-02-10T04:31:51.108438+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:34.482906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:34.482906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"259","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.148374+0000","last_change":"2021-02-10T04:29:35.382459+0000","last_active":"2021-02-10T04:31:53.148374+0000","last_peered":"2021-02-10T04:31:53.148374+0000","last_clean":"2021-02-10T04:31:53.148374+0000","last_became_active":"2021-02-10T04:25:49.010614+0000","last_became_peered":"2021-02-10T04:25:49.010614+0000","last_unstale":"2021-02-10T04:31:53.148374+0000","last_undegraded":"2021-02-10T04:31:53.148374+0000","last_fullsized":"2021-02-10T04:31:53.148374+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:35.382341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.382341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"254","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.109228+0000","last_change":"2021-02-10T04:28:38.503398+0000","last_active":"2021-02-10T04:31:51.109228+0000","last_peered":"2021-02-10T04:31:51.109228+0000","last_clean":"2021-02-10T04:31:51.109228+0000","last_became_active":"2021-02-10T04:25:53.037506+0000","last_became_peered":"2021-02-10T04:25:53.037506+0000","last_unstale":"2021-02-10T04:31:51.109228+0000","last_undegraded":"2021-02-10T04:31:51.109228+0000","last_fullsized":"2021-02-10T04:31:51.109228+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:38.503309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:38.503309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"275'85","reported_seq":"858","reported_epoch":"297","state":"active+clean","last_fresh":"2021-02-10T04:31:54.987210+0000","last_change":"2021-02-10T04:31:53.145393+0000","last_active":"2021-02-10T04:31:54.987210+0000","last_peered":"2021-02-10T04:31:54.987210+0000","last_clean":"2021-02-10T04:31:54.987210+0000","last_became_active":"2021-02-10T04:28:25.150109+0000","last_became_peered":"2021-02-10T04:28:25.150109+0000","last_unstale":"2021-02-10T04:31:54.987210+0000","last_undegraded":"2021-02-10T04:31:54.987210+0000","last_fullsized":"2021-02-10T04:31:54.987210+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'36","last_scrub_stamp":"2021-02-10T04:30:13.863430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:13.863430+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5899257,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":56120,"num_write":58,"num_write_kb":17039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c5","length":"1"}]},{"pgid":"5.6","version":"0'0","reported_seq":"253","reported_epoch":"292","state":"active+clean","last_fresh":"2021-02-10T04:31:49.106780+0000","last_change":"2021-02-10T04:29:23.779839+0000","last_active":"2021-02-10T04:31:49.106780+0000","last_peered":"2021-02-10T04:31:49.106780+0000","last_clean":"2021-02-10T04:31:49.106780+0000","last_became_active":"2021-02-10T04:25:51.025306+0000","last_became_peered":"2021-02-10T04:25:51.025306+0000","last_unstale":"2021-02-10T04:31:49.106780+0000","last_undegraded":"2021-02-10T04:31:49.106780+0000","last_fullsized":"2021-02-10T04:31:49.106780+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.779767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.779767+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"82'29","reported_seq":"312","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.113130+0000","last_change":"2021-02-10T04:29:25.974354+0000","last_active":"2021-02-10T04:31:51.113130+0000","last_peered":"2021-02-10T04:31:51.113130+0000","last_clean":"2021-02-10T04:31:51.113130+0000","last_became_active":"2021-02-10T04:25:49.002843+0000","last_became_peered":"2021-02-10T04:25:49.002843+0000","last_unstale":"2021-02-10T04:31:51.113130+0000","last_undegraded":"2021-02-10T04:31:51.113130+0000","last_fullsized":"2021-02-10T04:31:51.113130+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'29","last_scrub_stamp":"2021-02-10T04:29:25.974280+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.974280+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":32,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"276","reported_epoch":"292","state":"active+clean","last_fresh":"2021-02-10T04:31:49.105744+0000","last_change":"2021-02-10T04:28:39.690493+0000","last_active":"2021-02-10T04:31:49.105744+0000","last_peered":"2021-02-10T04:31:49.105744+0000","last_clean":"2021-02-10T04:31:49.105744+0000","last_became_active":"2021-02-10T04:28:21.120083+0000","last_became_peered":"2021-02-10T04:28:21.120083+0000","last_unstale":"2021-02-10T04:31:49.105744+0000","last_undegraded":"2021-02-10T04:31:49.105744+0000","last_fullsized":"2021-02-10T04:31:49.105744+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:39.690436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:39.690436+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"622","reported_epoch":"295","state":"active+clean","last_fresh":"2021-02-10T04:31:52.127146+0000","last_change":"2021-02-10T04:28:50.992977+0000","last_active":"2021-02-10T04:31:52.127146+0000","last_peered":"2021-02-10T04:31:52.127146+0000","last_clean":"2021-02-10T04:31:52.127146+0000","last_became_active":"2021-02-10T04:25:49.013969+0000","last_became_peered":"2021-02-10T04:25:49.013969+0000","last_unstale":"2021-02-10T04:31:52.127146+0000","last_undegraded":"2021-02-10T04:31:52.127146+0000","last_fullsized":"2021-02-10T04:31:52.127146+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:28:50.992899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:50.992899+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":361,"num_read_kb":356,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"182","reported_epoch":"295","state":"active+clean","last_fresh":"2021-02-10T04:31:52.109268+0000","last_change":"2021-02-10T04:29:33.747527+0000","last_active":"2021-02-10T04:31:52.109268+0000","last_peered":"2021-02-10T04:31:52.109268+0000","last_clean":"2021-02-10T04:31:52.109268+0000","last_became_active":"2021-02-10T04:29:07.158869+0000","last_became_peered":"2021-02-10T04:29:07.158869+0000","last_unstale":"2021-02-10T04:31:52.109268+0000","last_undegraded":"2021-02-10T04:31:52.109268+0000","last_fullsized":"2021-02-10T04:31:52.109268+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:33.747448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:33.747448+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.4","version":"295'160","reported_seq":"807","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.163925+0000","last_change":"2021-02-10T04:31:53.163925+0000","last_active":"2021-02-10T04:31:53.163925+0000","last_peered":"2021-02-10T04:31:53.163925+0000","last_clean":"2021-02-10T04:31:53.163925+0000","last_became_active":"2021-02-10T04:28:25.151037+0000","last_became_peered":"2021-02-10T04:28:25.151037+0000","last_unstale":"2021-02-10T04:31:53.163925+0000","last_undegraded":"2021-02-10T04:31:53.163925+0000","last_fullsized":"2021-02-10T04:31:53.163925+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":160,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2369853,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":103,"num_read_kb":82049,"num_write":104,"num_write_kb":31134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c5","length":"1"}]},{"pgid":"1.2","version":"0'0","reported_seq":"273","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.141241+0000","last_change":"2021-02-10T04:29:29.683867+0000","last_active":"2021-02-10T04:31:53.141241+0000","last_peered":"2021-02-10T04:31:53.141241+0000","last_clean":"2021-02-10T04:31:53.141241+0000","last_became_active":"2021-02-10T04:29:07.159607+0000","last_became_peered":"2021-02-10T04:29:07.159607+0000","last_unstale":"2021-02-10T04:31:53.141241+0000","last_undegraded":"2021-02-10T04:31:53.141241+0000","last_fullsized":"2021-02-10T04:31:53.141241+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.683805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.683805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"258","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.143839+0000","last_change":"2021-02-10T04:28:43.740930+0000","last_active":"2021-02-10T04:31:53.143839+0000","last_peered":"2021-02-10T04:31:53.143839+0000","last_clean":"2021-02-10T04:31:53.143839+0000","last_became_active":"2021-02-10T04:25:51.027624+0000","last_became_peered":"2021-02-10T04:25:51.027624+0000","last_unstale":"2021-02-10T04:31:53.143839+0000","last_undegraded":"2021-02-10T04:31:53.143839+0000","last_fullsized":"2021-02-10T04:31:53.143839+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:43.740853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:28:43.740853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"63'31","reported_seq":"346","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.112118+0000","last_change":"2021-02-10T04:30:47.884226+0000","last_active":"2021-02-10T04:31:51.112118+0000","last_peered":"2021-02-10T04:31:51.112118+0000","last_clean":"2021-02-10T04:31:51.112118+0000","last_became_active":"2021-02-10T04:25:49.007872+0000","last_became_peered":"2021-02-10T04:25:49.007872+0000","last_unstale":"2021-02-10T04:31:51.112118+0000","last_undegraded":"2021-02-10T04:31:51.112118+0000","last_fullsized":"2021-02-10T04:31:51.112118+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'31","last_scrub_stamp":"2021-02-10T04:28:03.352778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:03.352778+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":66,"num_read_kb":42,"num_write":34,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"274","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.108539+0000","last_change":"2021-02-10T04:28:36.491041+0000","last_active":"2021-02-10T04:31:51.108539+0000","last_peered":"2021-02-10T04:31:51.108539+0000","last_clean":"2021-02-10T04:31:51.108539+0000","last_became_active":"2021-02-10T04:28:21.125541+0000","last_became_peered":"2021-02-10T04:28:21.125541+0000","last_unstale":"2021-02-10T04:31:51.108539+0000","last_undegraded":"2021-02-10T04:31:51.108539+0000","last_fullsized":"2021-02-10T04:31:51.108539+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:36.490961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:36.490961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"255","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.150097+0000","last_change":"2021-02-10T04:29:36.396044+0000","last_active":"2021-02-10T04:31:53.150097+0000","last_peered":"2021-02-10T04:31:53.150097+0000","last_clean":"2021-02-10T04:31:53.150097+0000","last_became_active":"2021-02-10T04:25:53.049487+0000","last_became_peered":"2021-02-10T04:25:53.049487+0000","last_unstale":"2021-02-10T04:31:53.150097+0000","last_undegraded":"2021-02-10T04:31:53.150097+0000","last_fullsized":"2021-02-10T04:31:53.150097+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.395943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.395943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"295'528","reported_seq":"3276","reported_epoch":"296","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:31:53.147283+0000","last_change":"2021-02-10T04:31:52.439001+0000","last_active":"2021-02-10T04:31:53.147283+0000","last_peered":"2021-02-10T04:31:53.147283+0000","last_clean":"2021-02-10T04:31:53.147283+0000","last_became_active":"2021-02-10T04:28:25.147959+0000","last_became_peered":"2021-02-10T04:28:25.147959+0000","last_unstale":"2021-02-10T04:31:53.147283+0000","last_undegraded":"2021-02-10T04:31:53.147283+0000","last_fullsized":"2021-02-10T04:31:53.147283+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'223","last_scrub_stamp":"2021-02-10T04:30:14.478639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:14.478639+0000","log_size":528,"ondisk_log_size":528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"stat_sum":{"num_bytes":15933168,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":296,"num_read_kb":237409,"num_write":286,"num_write_kb":120966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":"197","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.108196+0000","last_change":"2021-02-10T04:29:39.434301+0000","last_active":"2021-02-10T04:31:51.108196+0000","last_peered":"2021-02-10T04:31:51.108196+0000","last_clean":"2021-02-10T04:31:51.108196+0000","last_became_active":"2021-02-10T04:28:49.054267+0000","last_became_peered":"2021-02-10T04:28:49.054267+0000","last_unstale":"2021-02-10T04:31:51.108196+0000","last_undegraded":"2021-02-10T04:31:51.108196+0000","last_fullsized":"2021-02-10T04:31:51.108196+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:39.434210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:39.434210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"58'2","reported_seq":"333","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.144403+0000","last_change":"2021-02-10T04:28:33.656576+0000","last_active":"2021-02-10T04:31:53.144403+0000","last_peered":"2021-02-10T04:31:53.144403+0000","last_clean":"2021-02-10T04:31:53.144403+0000","last_became_active":"2021-02-10T04:25:51.026251+0000","last_became_peered":"2021-02-10T04:25:51.026251+0000","last_unstale":"2021-02-10T04:31:53.144403+0000","last_undegraded":"2021-02-10T04:31:53.144403+0000","last_fullsized":"2021-02-10T04:31:53.144403+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:28:33.656509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:33.656509+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"79'34","reported_seq":"312","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.111850+0000","last_change":"2021-02-10T04:28:49.959266+0000","last_active":"2021-02-10T04:31:51.111850+0000","last_peered":"2021-02-10T04:31:51.111850+0000","last_clean":"2021-02-10T04:31:51.111850+0000","last_became_active":"2021-02-10T04:25:49.006147+0000","last_became_peered":"2021-02-10T04:25:49.006147+0000","last_unstale":"2021-02-10T04:31:51.111850+0000","last_undegraded":"2021-02-10T04:31:51.111850+0000","last_fullsized":"2021-02-10T04:31:51.111850+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'34","last_scrub_stamp":"2021-02-10T04:28:49.959162+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:28:49.959162+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":22,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"292","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.112199+0000","last_change":"2021-02-10T04:28:34.053378+0000","last_active":"2021-02-10T04:31:51.112199+0000","last_peered":"2021-02-10T04:31:51.112199+0000","last_clean":"2021-02-10T04:31:51.112199+0000","last_became_active":"2021-02-10T04:28:21.133772+0000","last_became_peered":"2021-02-10T04:28:21.133772+0000","last_unstale":"2021-02-10T04:31:51.112199+0000","last_undegraded":"2021-02-10T04:31:51.112199+0000","last_fullsized":"2021-02-10T04:31:51.112199+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:34.053323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:34.053323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"256","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.140730+0000","last_change":"2021-02-10T04:28:42.701993+0000","last_active":"2021-02-10T04:31:53.140730+0000","last_peered":"2021-02-10T04:31:53.140730+0000","last_clean":"2021-02-10T04:31:53.140730+0000","last_became_active":"2021-02-10T04:25:53.045215+0000","last_became_peered":"2021-02-10T04:25:53.045215+0000","last_unstale":"2021-02-10T04:31:53.140730+0000","last_undegraded":"2021-02-10T04:31:53.140730+0000","last_fullsized":"2021-02-10T04:31:53.140730+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:42.701924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:42.701924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"293'77","reported_seq":"708","reported_epoch":"293","state":"active+clean","last_fresh":"2021-02-10T04:31:50.126422+0000","last_change":"2021-02-10T04:31:50.108382+0000","last_active":"2021-02-10T04:31:50.126422+0000","last_peered":"2021-02-10T04:31:50.126422+0000","last_clean":"2021-02-10T04:31:50.126422+0000","last_became_active":"2021-02-10T04:28:25.154434+0000","last_became_peered":"2021-02-10T04:28:25.154434+0000","last_unstale":"2021-02-10T04:31:50.126422+0000","last_undegraded":"2021-02-10T04:31:50.126422+0000","last_fullsized":"2021-02-10T04:31:50.126422+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7546343,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":63,"num_read_kb":60653,"num_write":37,"num_write_kb":13263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"}]},{"pgid":"1.7","version":"0'0","reported_seq":"279","reported_epoch":"295","state":"active+clean","last_fresh":"2021-02-10T04:31:52.110273+0000","last_change":"2021-02-10T04:29:39.829039+0000","last_active":"2021-02-10T04:31:52.110273+0000","last_peered":"2021-02-10T04:31:52.110273+0000","last_clean":"2021-02-10T04:31:52.110273+0000","last_became_active":"2021-02-10T04:28:49.053127+0000","last_became_peered":"2021-02-10T04:28:49.053127+0000","last_unstale":"2021-02-10T04:31:52.110273+0000","last_undegraded":"2021-02-10T04:31:52.110273+0000","last_fullsized":"2021-02-10T04:31:52.110273+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:39.828975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:39.828975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"255","reported_epoch":"292","state":"active+clean","last_fresh":"2021-02-10T04:31:49.105723+0000","last_change":"2021-02-10T04:29:20.757283+0000","last_active":"2021-02-10T04:31:49.105723+0000","last_peered":"2021-02-10T04:31:49.105723+0000","last_clean":"2021-02-10T04:31:49.105723+0000","last_became_active":"2021-02-10T04:25:51.023000+0000","last_became_peered":"2021-02-10T04:25:51.023000+0000","last_unstale":"2021-02-10T04:31:49.105723+0000","last_undegraded":"2021-02-10T04:31:49.105723+0000","last_fullsized":"2021-02-10T04:31:49.105723+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:20.757194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:20.757194+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"63'27","reported_seq":"316","reported_epoch":"293","state":"active+clean","last_fresh":"2021-02-10T04:31:50.107085+0000","last_change":"2021-02-10T04:28:47.476743+0000","last_active":"2021-02-10T04:31:50.107085+0000","last_peered":"2021-02-10T04:31:50.107085+0000","last_clean":"2021-02-10T04:31:50.107085+0000","last_became_active":"2021-02-10T04:25:49.013259+0000","last_became_peered":"2021-02-10T04:25:49.013259+0000","last_unstale":"2021-02-10T04:31:50.107085+0000","last_undegraded":"2021-02-10T04:31:50.107085+0000","last_fullsized":"2021-02-10T04:31:50.107085+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'27","last_scrub_stamp":"2021-02-10T04:28:47.476669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:47.476669+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":41,"num_write":29,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"334","reported_epoch":"293","state":"active+clean","last_fresh":"2021-02-10T04:31:50.108464+0000","last_change":"2021-02-10T04:30:57.399572+0000","last_active":"2021-02-10T04:31:50.108464+0000","last_peered":"2021-02-10T04:31:50.108464+0000","last_clean":"2021-02-10T04:31:50.108464+0000","last_became_active":"2021-02-10T04:28:39.041556+0000","last_became_peered":"2021-02-10T04:28:39.041556+0000","last_unstale":"2021-02-10T04:31:50.108464+0000","last_undegraded":"2021-02-10T04:31:50.108464+0000","last_fullsized":"2021-02-10T04:31:50.108464+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:27:53.635228+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:27:53.635228+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"255","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.149473+0000","last_change":"2021-02-10T04:29:37.364781+0000","last_active":"2021-02-10T04:31:53.149473+0000","last_peered":"2021-02-10T04:31:53.149473+0000","last_clean":"2021-02-10T04:31:53.149473+0000","last_became_active":"2021-02-10T04:25:53.047736+0000","last_became_peered":"2021-02-10T04:25:53.047736+0000","last_unstale":"2021-02-10T04:31:53.149473+0000","last_undegraded":"2021-02-10T04:31:53.149473+0000","last_fullsized":"2021-02-10T04:31:53.149473+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:37.364697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:37.364697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"297'301","reported_seq":"1684","reported_epoch":"297","state":"active+clean","last_fresh":"2021-02-10T04:31:54.415066+0000","last_change":"2021-02-10T04:31:53.148992+0000","last_active":"2021-02-10T04:31:54.415066+0000","last_peered":"2021-02-10T04:31:54.415066+0000","last_clean":"2021-02-10T04:31:54.415066+0000","last_became_active":"2021-02-10T04:28:25.157771+0000","last_became_peered":"2021-02-10T04:28:25.157771+0000","last_unstale":"2021-02-10T04:31:54.415066+0000","last_undegraded":"2021-02-10T04:31:54.415066+0000","last_fullsized":"2021-02-10T04:31:54.415066+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":301,"ondisk_log_size":301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8741848,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":131,"num_read_kb":104714,"num_write":152,"num_write_kb":62290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c5","length":"1"}]},{"pgid":"4.16","version":"63'28","reported_seq":"320","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.144687+0000","last_change":"2021-02-10T04:29:25.645462+0000","last_active":"2021-02-10T04:31:53.144687+0000","last_peered":"2021-02-10T04:31:53.144687+0000","last_clean":"2021-02-10T04:31:53.144687+0000","last_became_active":"2021-02-10T04:25:49.008942+0000","last_became_peered":"2021-02-10T04:25:49.008942+0000","last_unstale":"2021-02-10T04:31:53.144687+0000","last_undegraded":"2021-02-10T04:31:53.144687+0000","last_fullsized":"2021-02-10T04:31:53.144687+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'28","last_scrub_stamp":"2021-02-10T04:29:25.645396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.645396+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":62,"num_read_kb":40,"num_write":29,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"265","reported_epoch":"292","state":"active+clean","last_fresh":"2021-02-10T04:31:49.105638+0000","last_change":"2021-02-10T04:28:45.647998+0000","last_active":"2021-02-10T04:31:49.105638+0000","last_peered":"2021-02-10T04:31:49.105638+0000","last_clean":"2021-02-10T04:31:49.105638+0000","last_became_active":"2021-02-10T04:25:34.329694+0000","last_became_peered":"2021-02-10T04:25:34.329694+0000","last_unstale":"2021-02-10T04:31:49.105638+0000","last_undegraded":"2021-02-10T04:31:49.105638+0000","last_fullsized":"2021-02-10T04:31:49.105638+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:28:45.647916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:45.647916+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"257","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.149904+0000","last_change":"2021-02-10T04:28:33.321856+0000","last_active":"2021-02-10T04:31:53.149904+0000","last_peered":"2021-02-10T04:31:53.149904+0000","last_clean":"2021-02-10T04:31:53.149904+0000","last_became_active":"2021-02-10T04:25:51.030651+0000","last_became_peered":"2021-02-10T04:25:51.030651+0000","last_unstale":"2021-02-10T04:31:53.149904+0000","last_undegraded":"2021-02-10T04:31:53.149904+0000","last_fullsized":"2021-02-10T04:31:53.149904+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:33.321784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:33.321784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"251","reported_epoch":"292","state":"active+clean","last_fresh":"2021-02-10T04:31:49.106251+0000","last_change":"2021-02-10T04:29:28.776423+0000","last_active":"2021-02-10T04:31:49.106251+0000","last_peered":"2021-02-10T04:31:49.106251+0000","last_clean":"2021-02-10T04:31:49.106251+0000","last_became_active":"2021-02-10T04:25:53.035867+0000","last_became_peered":"2021-02-10T04:25:53.035867+0000","last_unstale":"2021-02-10T04:31:49.106251+0000","last_undegraded":"2021-02-10T04:31:49.106251+0000","last_fullsized":"2021-02-10T04:31:49.106251+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.776340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.776340+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"63'13","reported_seq":"306","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.149523+0000","last_change":"2021-02-10T04:28:11.311247+0000","last_active":"2021-02-10T04:31:53.149523+0000","last_peered":"2021-02-10T04:31:53.149523+0000","last_clean":"2021-02-10T04:31:53.149523+0000","last_became_active":"2021-02-10T04:26:39.003832+0000","last_became_peered":"2021-02-10T04:26:39.003832+0000","last_unstale":"2021-02-10T04:31:53.149523+0000","last_undegraded":"2021-02-10T04:31:53.149523+0000","last_fullsized":"2021-02-10T04:31:53.149523+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2021-02-10T04:28:11.311182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:11.311182+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"268","reported_epoch":"295","state":"active+clean","last_fresh":"2021-02-10T04:31:52.110543+0000","last_change":"2021-02-10T04:29:18.565481+0000","last_active":"2021-02-10T04:31:52.110543+0000","last_peered":"2021-02-10T04:31:52.110543+0000","last_clean":"2021-02-10T04:31:52.110543+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:31:52.110543+0000","last_undegraded":"2021-02-10T04:31:52.110543+0000","last_fullsized":"2021-02-10T04:31:52.110543+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:29:18.565413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:18.565413+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"256","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.108949+0000","last_change":"2021-02-10T04:29:18.429730+0000","last_active":"2021-02-10T04:31:51.108949+0000","last_peered":"2021-02-10T04:31:51.108949+0000","last_clean":"2021-02-10T04:31:51.108949+0000","last_became_active":"2021-02-10T04:25:51.031864+0000","last_became_peered":"2021-02-10T04:25:51.031864+0000","last_unstale":"2021-02-10T04:31:51.108949+0000","last_undegraded":"2021-02-10T04:31:51.108949+0000","last_fullsized":"2021-02-10T04:31:51.108949+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:18.429649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:18.429649+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"182","reported_epoch":"295","state":"active+clean","last_fresh":"2021-02-10T04:31:52.109467+0000","last_change":"2021-02-10T04:29:36.866949+0000","last_active":"2021-02-10T04:31:52.109467+0000","last_peered":"2021-02-10T04:31:52.109467+0000","last_clean":"2021-02-10T04:31:52.109467+0000","last_became_active":"2021-02-10T04:29:07.147249+0000","last_became_peered":"2021-02-10T04:29:07.147249+0000","last_unstale":"2021-02-10T04:31:52.109467+0000","last_undegraded":"2021-02-10T04:31:52.109467+0000","last_fullsized":"2021-02-10T04:31:52.109467+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.866833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.866833+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"63'26","reported_seq":"368","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.149395+0000","last_change":"2021-02-10T04:30:54.359947+0000","last_active":"2021-02-10T04:31:53.149395+0000","last_peered":"2021-02-10T04:31:53.149395+0000","last_clean":"2021-02-10T04:31:53.149395+0000","last_became_active":"2021-02-10T04:26:39.003971+0000","last_became_peered":"2021-02-10T04:26:39.003971+0000","last_unstale":"2021-02-10T04:31:53.149395+0000","last_undegraded":"2021-02-10T04:31:53.149395+0000","last_fullsized":"2021-02-10T04:31:53.149395+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:27:28.213454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:28.213454+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"84'7","reported_seq":"285","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.108452+0000","last_change":"2021-02-10T04:30:12.538670+0000","last_active":"2021-02-10T04:31:51.108452+0000","last_peered":"2021-02-10T04:31:51.108452+0000","last_clean":"2021-02-10T04:31:51.108452+0000","last_became_active":"2021-02-10T04:25:36.334366+0000","last_became_peered":"2021-02-10T04:25:36.334366+0000","last_unstale":"2021-02-10T04:31:51.108452+0000","last_undegraded":"2021-02-10T04:31:51.108452+0000","last_fullsized":"2021-02-10T04:31:51.108452+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'7","last_scrub_stamp":"2021-02-10T04:30:12.538549+0000","last_deep_scrub":"84'7","last_deep_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_clean_scrub_stamp":"2021-02-10T04:30:12.538549+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"58'2","reported_seq":"331","reported_epoch":"295","state":"active+clean","last_fresh":"2021-02-10T04:31:52.109003+0000","last_change":"2021-02-10T04:28:47.496534+0000","last_active":"2021-02-10T04:31:52.109003+0000","last_peered":"2021-02-10T04:31:52.109003+0000","last_clean":"2021-02-10T04:31:52.109003+0000","last_became_active":"2021-02-10T04:25:51.032328+0000","last_became_peered":"2021-02-10T04:25:51.032328+0000","last_unstale":"2021-02-10T04:31:52.109003+0000","last_undegraded":"2021-02-10T04:31:52.109003+0000","last_fullsized":"2021-02-10T04:31:52.109003+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:28:47.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:47.496468+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"256","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.143635+0000","last_change":"2021-02-10T04:28:45.697744+0000","last_active":"2021-02-10T04:31:53.143635+0000","last_peered":"2021-02-10T04:31:53.143635+0000","last_clean":"2021-02-10T04:31:53.143635+0000","last_became_active":"2021-02-10T04:25:53.045177+0000","last_became_peered":"2021-02-10T04:25:53.045177+0000","last_unstale":"2021-02-10T04:31:53.143635+0000","last_undegraded":"2021-02-10T04:31:53.143635+0000","last_fullsized":"2021-02-10T04:31:53.143635+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_clean_scrub_stamp":"2021-02-10T04:28:45.697677+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"63'19","reported_seq":"297","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.108820+0000","last_change":"2021-02-10T04:28:50.493712+0000","last_active":"2021-02-10T04:31:51.108820+0000","last_peered":"2021-02-10T04:31:51.108820+0000","last_clean":"2021-02-10T04:31:51.108820+0000","last_became_active":"2021-02-10T04:25:49.010890+0000","last_became_peered":"2021-02-10T04:25:49.010890+0000","last_unstale":"2021-02-10T04:31:51.108820+0000","last_undegraded":"2021-02-10T04:31:51.108820+0000","last_fullsized":"2021-02-10T04:31:51.108820+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2021-02-10T04:28:50.493641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:50.493641+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":26,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"272","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.144326+0000","last_change":"2021-02-10T04:29:19.564845+0000","last_active":"2021-02-10T04:31:53.144326+0000","last_peered":"2021-02-10T04:31:53.144326+0000","last_clean":"2021-02-10T04:31:53.144326+0000","last_became_active":"2021-02-10T04:25:34.321096+0000","last_became_peered":"2021-02-10T04:25:34.321096+0000","last_unstale":"2021-02-10T04:31:53.144326+0000","last_undegraded":"2021-02-10T04:31:53.144326+0000","last_fullsized":"2021-02-10T04:31:53.144326+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:19.564725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:29:19.564725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"58'2","reported_seq":"332","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.150128+0000","last_change":"2021-02-10T04:29:38.343946+0000","last_active":"2021-02-10T04:31:53.150128+0000","last_peered":"2021-02-10T04:31:53.150128+0000","last_clean":"2021-02-10T04:31:53.150128+0000","last_became_active":"2021-02-10T04:25:51.018300+0000","last_became_peered":"2021-02-10T04:25:51.018300+0000","last_unstale":"2021-02-10T04:31:53.150128+0000","last_undegraded":"2021-02-10T04:31:53.150128+0000","last_fullsized":"2021-02-10T04:31:53.150128+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:29:38.343834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.343834+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"253","reported_epoch":"293","state":"active+clean","last_fresh":"2021-02-10T04:31:50.108634+0000","last_change":"2021-02-10T04:29:23.934149+0000","last_active":"2021-02-10T04:31:50.108634+0000","last_peered":"2021-02-10T04:31:50.108634+0000","last_clean":"2021-02-10T04:31:50.108634+0000","last_became_active":"2021-02-10T04:25:53.031782+0000","last_became_peered":"2021-02-10T04:25:53.031782+0000","last_unstale":"2021-02-10T04:31:50.108634+0000","last_undegraded":"2021-02-10T04:31:50.108634+0000","last_fullsized":"2021-02-10T04:31:50.108634+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.934078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.934078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"62'15","reported_seq":"286","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.111849+0000","last_change":"2021-02-10T04:30:11.170100+0000","last_active":"2021-02-10T04:31:51.111849+0000","last_peered":"2021-02-10T04:31:51.111849+0000","last_clean":"2021-02-10T04:31:51.111849+0000","last_became_active":"2021-02-10T04:25:49.015543+0000","last_became_peered":"2021-02-10T04:25:49.015543+0000","last_unstale":"2021-02-10T04:31:51.111849+0000","last_undegraded":"2021-02-10T04:31:51.111849+0000","last_fullsized":"2021-02-10T04:31:51.111849+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:30:11.170034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.170034+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"266","reported_epoch":"295","state":"active+clean","last_fresh":"2021-02-10T04:31:52.108878+0000","last_change":"2021-02-10T04:28:36.204091+0000","last_active":"2021-02-10T04:31:52.108878+0000","last_peered":"2021-02-10T04:31:52.108878+0000","last_clean":"2021-02-10T04:31:52.108878+0000","last_became_active":"2021-02-10T04:25:34.330070+0000","last_became_peered":"2021-02-10T04:25:34.330070+0000","last_unstale":"2021-02-10T04:31:52.108878+0000","last_undegraded":"2021-02-10T04:31:52.108878+0000","last_fullsized":"2021-02-10T04:31:52.108878+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:36.204007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:36.204007+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"257","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.148952+0000","last_change":"2021-02-10T04:28:36.311808+0000","last_active":"2021-02-10T04:31:53.148952+0000","last_peered":"2021-02-10T04:31:53.148952+0000","last_clean":"2021-02-10T04:31:53.148952+0000","last_became_active":"2021-02-10T04:25:51.029155+0000","last_became_peered":"2021-02-10T04:25:51.029155+0000","last_unstale":"2021-02-10T04:31:53.148952+0000","last_undegraded":"2021-02-10T04:31:53.148952+0000","last_fullsized":"2021-02-10T04:31:53.148952+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:36.311725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:28:36.311725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"256","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.140635+0000","last_change":"2021-02-10T04:28:38.788947+0000","last_active":"2021-02-10T04:31:53.140635+0000","last_peered":"2021-02-10T04:31:53.140635+0000","last_clean":"2021-02-10T04:31:53.140635+0000","last_became_active":"2021-02-10T04:25:53.039322+0000","last_became_peered":"2021-02-10T04:25:53.039322+0000","last_unstale":"2021-02-10T04:31:53.140635+0000","last_undegraded":"2021-02-10T04:31:53.140635+0000","last_fullsized":"2021-02-10T04:31:53.140635+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:38.788851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:38.788851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"62'15","reported_seq":"279","reported_epoch":"293","state":"active+clean","last_fresh":"2021-02-10T04:31:50.107334+0000","last_change":"2021-02-10T04:27:42.487239+0000","last_active":"2021-02-10T04:31:50.107334+0000","last_peered":"2021-02-10T04:31:50.107334+0000","last_clean":"2021-02-10T04:31:50.107334+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:31:50.107334+0000","last_undegraded":"2021-02-10T04:31:50.107334+0000","last_fullsized":"2021-02-10T04:31:50.107334+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:27:42.487154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:42.487154+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"274","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.112672+0000","last_change":"2021-02-10T04:28:33.070791+0000","last_active":"2021-02-10T04:31:51.112672+0000","last_peered":"2021-02-10T04:31:51.112672+0000","last_clean":"2021-02-10T04:31:51.112672+0000","last_became_active":"2021-02-10T04:25:34.330242+0000","last_became_peered":"2021-02-10T04:25:34.330242+0000","last_unstale":"2021-02-10T04:31:51.112672+0000","last_undegraded":"2021-02-10T04:31:51.112672+0000","last_fullsized":"2021-02-10T04:31:51.112672+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:28:33.070726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:33.070726+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"256","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.111936+0000","last_change":"2021-02-10T04:29:33.027075+0000","last_active":"2021-02-10T04:31:51.111936+0000","last_peered":"2021-02-10T04:31:51.111936+0000","last_clean":"2021-02-10T04:31:51.111936+0000","last_became_active":"2021-02-10T04:25:51.030952+0000","last_became_peered":"2021-02-10T04:25:51.030952+0000","last_unstale":"2021-02-10T04:31:51.111936+0000","last_undegraded":"2021-02-10T04:31:51.111936+0000","last_fullsized":"2021-02-10T04:31:51.111936+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:33.026988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:29:33.026988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"255","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.146036+0000","last_change":"2021-02-10T04:29:24.421782+0000","last_active":"2021-02-10T04:31:53.146036+0000","last_peered":"2021-02-10T04:31:53.146036+0000","last_clean":"2021-02-10T04:31:53.146036+0000","last_became_active":"2021-02-10T04:25:53.043277+0000","last_became_peered":"2021-02-10T04:25:53.043277+0000","last_unstale":"2021-02-10T04:31:53.146036+0000","last_undegraded":"2021-02-10T04:31:53.146036+0000","last_fullsized":"2021-02-10T04:31:53.146036+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.421710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.421710+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"62'12","reported_seq":"307","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.145119+0000","last_change":"2021-02-10T04:27:48.284315+0000","last_active":"2021-02-10T04:31:53.145119+0000","last_peered":"2021-02-10T04:31:53.145119+0000","last_clean":"2021-02-10T04:31:53.145119+0000","last_became_active":"2021-02-10T04:26:39.005616+0000","last_became_peered":"2021-02-10T04:26:39.005616+0000","last_unstale":"2021-02-10T04:31:53.145119+0000","last_undegraded":"2021-02-10T04:31:53.145119+0000","last_fullsized":"2021-02-10T04:31:53.145119+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2021-02-10T04:27:48.284248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:48.284248+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"265","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.109007+0000","last_change":"2021-02-10T04:28:49.522178+0000","last_active":"2021-02-10T04:31:51.109007+0000","last_peered":"2021-02-10T04:31:51.109007+0000","last_clean":"2021-02-10T04:31:51.109007+0000","last_became_active":"2021-02-10T04:25:34.331787+0000","last_became_peered":"2021-02-10T04:25:34.331787+0000","last_unstale":"2021-02-10T04:31:51.109007+0000","last_undegraded":"2021-02-10T04:31:51.109007+0000","last_fullsized":"2021-02-10T04:31:51.109007+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:49.522124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:49.522124+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"257","reported_epoch":"295","state":"active+clean","last_fresh":"2021-02-10T04:31:52.110459+0000","last_change":"2021-02-10T04:29:23.645783+0000","last_active":"2021-02-10T04:31:52.110459+0000","last_peered":"2021-02-10T04:31:52.110459+0000","last_clean":"2021-02-10T04:31:52.110459+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:31:52.110459+0000","last_undegraded":"2021-02-10T04:31:52.110459+0000","last_fullsized":"2021-02-10T04:31:52.110459+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.645694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.645694+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"179","reported_epoch":"292","state":"active+clean","last_fresh":"2021-02-10T04:31:49.105922+0000","last_change":"2021-02-10T04:29:31.825006+0000","last_active":"2021-02-10T04:31:49.105922+0000","last_peered":"2021-02-10T04:31:49.105922+0000","last_clean":"2021-02-10T04:31:49.105922+0000","last_became_active":"2021-02-10T04:29:07.149764+0000","last_became_peered":"2021-02-10T04:29:07.149764+0000","last_unstale":"2021-02-10T04:31:49.105922+0000","last_undegraded":"2021-02-10T04:31:49.105922+0000","last_fullsized":"2021-02-10T04:31:49.105922+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:31.824925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.824925+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"63'21","reported_seq":"300","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.146530+0000","last_change":"2021-02-10T04:28:31.382776+0000","last_active":"2021-02-10T04:31:53.146530+0000","last_peered":"2021-02-10T04:31:53.146530+0000","last_clean":"2021-02-10T04:31:53.146530+0000","last_became_active":"2021-02-10T04:25:49.007049+0000","last_became_peered":"2021-02-10T04:25:49.007049+0000","last_unstale":"2021-02-10T04:31:53.146530+0000","last_undegraded":"2021-02-10T04:31:53.146530+0000","last_fullsized":"2021-02-10T04:31:53.146530+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2021-02-10T04:28:31.382671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:31.382671+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":27,"num_write":21,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"269","reported_epoch":"295","state":"active+clean","last_fresh":"2021-02-10T04:31:52.110496+0000","last_change":"2021-02-10T04:28:35.196412+0000","last_active":"2021-02-10T04:31:52.110496+0000","last_peered":"2021-02-10T04:31:52.110496+0000","last_clean":"2021-02-10T04:31:52.110496+0000","last_became_active":"2021-02-10T04:25:35.328828+0000","last_became_peered":"2021-02-10T04:25:35.328828+0000","last_unstale":"2021-02-10T04:31:52.110496+0000","last_undegraded":"2021-02-10T04:31:52.110496+0000","last_fullsized":"2021-02-10T04:31:52.110496+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:28:35.196342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:35.196342+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"175","reported_epoch":"293","state":"active+clean","last_fresh":"2021-02-10T04:31:50.106522+0000","last_change":"2021-02-10T04:29:07.150253+0000","last_active":"2021-02-10T04:31:50.106522+0000","last_peered":"2021-02-10T04:31:50.106522+0000","last_clean":"2021-02-10T04:31:50.106522+0000","last_became_active":"2021-02-10T04:29:07.149867+0000","last_became_peered":"2021-02-10T04:29:07.149867+0000","last_unstale":"2021-02-10T04:31:50.106522+0000","last_undegraded":"2021-02-10T04:31:50.106522+0000","last_fullsized":"2021-02-10T04:31:50.106522+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:47.473024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:47.473024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"250","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.149547+0000","last_change":"2021-02-10T04:27:50.300043+0000","last_active":"2021-02-10T04:31:53.149547+0000","last_peered":"2021-02-10T04:31:53.149547+0000","last_clean":"2021-02-10T04:31:53.149547+0000","last_became_active":"2021-02-10T04:25:53.150845+0000","last_became_peered":"2021-02-10T04:25:53.150845+0000","last_unstale":"2021-02-10T04:31:53.149547+0000","last_undegraded":"2021-02-10T04:31:53.149547+0000","last_fullsized":"2021-02-10T04:31:53.149547+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_clean_scrub_stamp":"2021-02-10T04:27:50.299942+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"63'22","reported_seq":"307","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.144978+0000","last_change":"2021-02-10T04:29:38.663351+0000","last_active":"2021-02-10T04:31:53.144978+0000","last_peered":"2021-02-10T04:31:53.144978+0000","last_clean":"2021-02-10T04:31:53.144978+0000","last_became_active":"2021-02-10T04:25:49.009177+0000","last_became_peered":"2021-02-10T04:25:49.009177+0000","last_unstale":"2021-02-10T04:31:53.144978+0000","last_undegraded":"2021-02-10T04:31:53.144978+0000","last_fullsized":"2021-02-10T04:31:53.144978+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2021-02-10T04:29:38.663272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.663272+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":22,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"274","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.108043+0000","last_change":"2021-02-10T04:28:37.515037+0000","last_active":"2021-02-10T04:31:51.108043+0000","last_peered":"2021-02-10T04:31:51.108043+0000","last_clean":"2021-02-10T04:31:51.108043+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:31:51.108043+0000","last_undegraded":"2021-02-10T04:31:51.108043+0000","last_fullsized":"2021-02-10T04:31:51.108043+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:37.514943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:37.514943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"58'2","reported_seq":"332","reported_epoch":"295","state":"active+clean","last_fresh":"2021-02-10T04:31:52.770440+0000","last_change":"2021-02-10T04:29:31.726022+0000","last_active":"2021-02-10T04:31:52.770440+0000","last_peered":"2021-02-10T04:31:52.770440+0000","last_clean":"2021-02-10T04:31:52.770440+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:31:52.770440+0000","last_undegraded":"2021-02-10T04:31:52.770440+0000","last_fullsized":"2021-02-10T04:31:52.770440+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:29:31.725932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.725932+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"181","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.108650+0000","last_change":"2021-02-10T04:29:26.437691+0000","last_active":"2021-02-10T04:31:51.108650+0000","last_peered":"2021-02-10T04:31:51.108650+0000","last_clean":"2021-02-10T04:31:51.108650+0000","last_became_active":"2021-02-10T04:29:07.150801+0000","last_became_peered":"2021-02-10T04:29:07.150801+0000","last_unstale":"2021-02-10T04:31:51.108650+0000","last_undegraded":"2021-02-10T04:31:51.108650+0000","last_fullsized":"2021-02-10T04:31:51.108650+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:26.437591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.437591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"76'35","reported_seq":"322","reported_epoch":"295","state":"active+clean","last_fresh":"2021-02-10T04:31:52.110888+0000","last_change":"2021-02-10T04:30:13.526264+0000","last_active":"2021-02-10T04:31:52.110888+0000","last_peered":"2021-02-10T04:31:52.110888+0000","last_clean":"2021-02-10T04:31:52.110888+0000","last_became_active":"2021-02-10T04:25:49.015760+0000","last_became_peered":"2021-02-10T04:25:49.015760+0000","last_unstale":"2021-02-10T04:31:52.110888+0000","last_undegraded":"2021-02-10T04:31:52.110888+0000","last_fullsized":"2021-02-10T04:31:52.110888+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'35","last_scrub_stamp":"2021-02-10T04:30:13.526170+0000","last_deep_scrub":"76'35","last_deep_scrub_stamp":"2021-02-10T04:26:58.198641+0000","last_clean_scrub_stamp":"2021-02-10T04:30:13.526170+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":50,"num_read_kb":30,"num_write":24,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"266","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.146317+0000","last_change":"2021-02-10T04:30:11.485059+0000","last_active":"2021-02-10T04:31:53.146317+0000","last_peered":"2021-02-10T04:31:53.146317+0000","last_clean":"2021-02-10T04:31:53.146317+0000","last_became_active":"2021-02-10T04:25:34.324981+0000","last_became_peered":"2021-02-10T04:25:34.324981+0000","last_unstale":"2021-02-10T04:31:53.146317+0000","last_undegraded":"2021-02-10T04:31:53.146317+0000","last_fullsized":"2021-02-10T04:31:53.146317+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:11.485004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.485004+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"258","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.145059+0000","last_change":"2021-02-10T04:29:23.589223+0000","last_active":"2021-02-10T04:31:53.145059+0000","last_peered":"2021-02-10T04:31:53.145059+0000","last_clean":"2021-02-10T04:31:53.145059+0000","last_became_active":"2021-02-10T04:25:51.022853+0000","last_became_peered":"2021-02-10T04:25:51.022853+0000","last_unstale":"2021-02-10T04:31:53.145059+0000","last_undegraded":"2021-02-10T04:31:53.145059+0000","last_fullsized":"2021-02-10T04:31:53.145059+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.589159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.589159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"254","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.112604+0000","last_change":"2021-02-10T04:29:25.076025+0000","last_active":"2021-02-10T04:31:51.112604+0000","last_peered":"2021-02-10T04:31:51.112604+0000","last_clean":"2021-02-10T04:31:51.112604+0000","last_became_active":"2021-02-10T04:25:53.041098+0000","last_became_peered":"2021-02-10T04:25:53.041098+0000","last_unstale":"2021-02-10T04:31:51.112604+0000","last_undegraded":"2021-02-10T04:31:51.112604+0000","last_fullsized":"2021-02-10T04:31:51.112604+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:25.075937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.075937+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"62'15","reported_seq":"280","reported_epoch":"292","state":"active+clean","last_fresh":"2021-02-10T04:31:49.106786+0000","last_change":"2021-02-10T04:28:46.680268+0000","last_active":"2021-02-10T04:31:49.106786+0000","last_peered":"2021-02-10T04:31:49.106786+0000","last_clean":"2021-02-10T04:31:49.106786+0000","last_became_active":"2021-02-10T04:25:49.010553+0000","last_became_peered":"2021-02-10T04:25:49.010553+0000","last_unstale":"2021-02-10T04:31:49.106786+0000","last_undegraded":"2021-02-10T04:31:49.106786+0000","last_fullsized":"2021-02-10T04:31:49.106786+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:28:46.680176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:46.680176+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"269","reported_epoch":"295","state":"active+clean","last_fresh":"2021-02-10T04:31:52.108189+0000","last_change":"2021-02-10T04:28:38.246850+0000","last_active":"2021-02-10T04:31:52.108189+0000","last_peered":"2021-02-10T04:31:52.108189+0000","last_clean":"2021-02-10T04:31:52.108189+0000","last_became_active":"2021-02-10T04:25:36.336442+0000","last_became_peered":"2021-02-10T04:25:36.336442+0000","last_unstale":"2021-02-10T04:31:52.108189+0000","last_undegraded":"2021-02-10T04:31:52.108189+0000","last_fullsized":"2021-02-10T04:31:52.108189+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:28:38.246768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:38.246768+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"256","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.112836+0000","last_change":"2021-02-10T04:29:22.078330+0000","last_active":"2021-02-10T04:31:51.112836+0000","last_peered":"2021-02-10T04:31:51.112836+0000","last_clean":"2021-02-10T04:31:51.112836+0000","last_became_active":"2021-02-10T04:25:51.029399+0000","last_became_peered":"2021-02-10T04:25:51.029399+0000","last_unstale":"2021-02-10T04:31:51.112836+0000","last_undegraded":"2021-02-10T04:31:51.112836+0000","last_fullsized":"2021-02-10T04:31:51.112836+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:22.078247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:22.078247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"253","reported_epoch":"293","state":"active+clean","last_fresh":"2021-02-10T04:31:50.108006+0000","last_change":"2021-02-10T04:29:31.903091+0000","last_active":"2021-02-10T04:31:50.108006+0000","last_peered":"2021-02-10T04:31:50.108006+0000","last_clean":"2021-02-10T04:31:50.108006+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:31:50.108006+0000","last_undegraded":"2021-02-10T04:31:50.108006+0000","last_fullsized":"2021-02-10T04:31:50.108006+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:31.902996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.902996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"62'22","reported_seq":"300","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.150038+0000","last_change":"2021-02-10T04:27:58.363322+0000","last_active":"2021-02-10T04:31:53.150038+0000","last_peered":"2021-02-10T04:31:53.150038+0000","last_clean":"2021-02-10T04:31:53.150038+0000","last_became_active":"2021-02-10T04:25:49.008997+0000","last_became_peered":"2021-02-10T04:25:49.008997+0000","last_unstale":"2021-02-10T04:31:53.150038+0000","last_undegraded":"2021-02-10T04:31:53.150038+0000","last_fullsized":"2021-02-10T04:31:53.150038+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'22","last_scrub_stamp":"2021-02-10T04:27:58.363248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:58.363248+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":49,"num_read_kb":31,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"278","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.148753+0000","last_change":"2021-02-10T04:28:30.415204+0000","last_active":"2021-02-10T04:31:53.148753+0000","last_peered":"2021-02-10T04:31:53.148753+0000","last_clean":"2021-02-10T04:31:53.148753+0000","last_became_active":"2021-02-10T04:25:36.336893+0000","last_became_peered":"2021-02-10T04:25:36.336893+0000","last_unstale":"2021-02-10T04:31:53.148753+0000","last_undegraded":"2021-02-10T04:31:53.148753+0000","last_fullsized":"2021-02-10T04:31:53.148753+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:30.415134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:26:46.425288+0000","last_clean_scrub_stamp":"2021-02-10T04:28:30.415134+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"250","reported_epoch":"293","state":"active+clean","last_fresh":"2021-02-10T04:31:50.108290+0000","last_change":"2021-02-10T04:27:48.598953+0000","last_active":"2021-02-10T04:31:50.108290+0000","last_peered":"2021-02-10T04:31:50.108290+0000","last_clean":"2021-02-10T04:31:50.108290+0000","last_became_active":"2021-02-10T04:25:51.018872+0000","last_became_peered":"2021-02-10T04:25:51.018872+0000","last_unstale":"2021-02-10T04:31:50.108290+0000","last_undegraded":"2021-02-10T04:31:50.108290+0000","last_fullsized":"2021-02-10T04:31:50.108290+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:48.598870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:48.598870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"255","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.149338+0000","last_change":"2021-02-10T04:29:26.355726+0000","last_active":"2021-02-10T04:31:53.149338+0000","last_peered":"2021-02-10T04:31:53.149338+0000","last_clean":"2021-02-10T04:31:53.149338+0000","last_became_active":"2021-02-10T04:25:53.147831+0000","last_became_peered":"2021-02-10T04:25:53.147831+0000","last_unstale":"2021-02-10T04:31:53.149338+0000","last_undegraded":"2021-02-10T04:31:53.149338+0000","last_fullsized":"2021-02-10T04:31:53.149338+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:26.355652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.355652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"63'25","reported_seq":"311","reported_epoch":"293","state":"active+clean","last_fresh":"2021-02-10T04:31:50.107206+0000","last_change":"2021-02-10T04:28:32.565457+0000","last_active":"2021-02-10T04:31:50.107206+0000","last_peered":"2021-02-10T04:31:50.107206+0000","last_clean":"2021-02-10T04:31:50.107206+0000","last_became_active":"2021-02-10T04:25:49.000911+0000","last_became_peered":"2021-02-10T04:25:49.000911+0000","last_unstale":"2021-02-10T04:31:50.107206+0000","last_undegraded":"2021-02-10T04:31:50.107206+0000","last_fullsized":"2021-02-10T04:31:50.107206+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2021-02-10T04:28:32.565374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:32.565374+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":53,"num_read_kb":37,"num_write":26,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"176","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.108747+0000","last_change":"2021-02-10T04:29:07.152012+0000","last_active":"2021-02-10T04:31:51.108747+0000","last_peered":"2021-02-10T04:31:51.108747+0000","last_clean":"2021-02-10T04:31:51.108747+0000","last_became_active":"2021-02-10T04:29:07.151577+0000","last_became_peered":"2021-02-10T04:29:07.151577+0000","last_unstale":"2021-02-10T04:31:51.108747+0000","last_undegraded":"2021-02-10T04:31:51.108747+0000","last_fullsized":"2021-02-10T04:31:51.108747+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:00.398480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:00.398480+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"256","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.108444+0000","last_change":"2021-02-10T04:28:46.464172+0000","last_active":"2021-02-10T04:31:51.108444+0000","last_peered":"2021-02-10T04:31:51.108444+0000","last_clean":"2021-02-10T04:31:51.108444+0000","last_became_active":"2021-02-10T04:25:51.025488+0000","last_became_peered":"2021-02-10T04:25:51.025488+0000","last_unstale":"2021-02-10T04:31:51.108444+0000","last_undegraded":"2021-02-10T04:31:51.108444+0000","last_fullsized":"2021-02-10T04:31:51.108444+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:46.464108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:46.464108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"256","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.141094+0000","last_change":"2021-02-10T04:28:44.719453+0000","last_active":"2021-02-10T04:31:53.141094+0000","last_peered":"2021-02-10T04:31:53.141094+0000","last_clean":"2021-02-10T04:31:53.141094+0000","last_became_active":"2021-02-10T04:25:53.039844+0000","last_became_peered":"2021-02-10T04:25:53.039844+0000","last_unstale":"2021-02-10T04:31:53.141094+0000","last_undegraded":"2021-02-10T04:31:53.141094+0000","last_fullsized":"2021-02-10T04:31:53.141094+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:44.719363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:44.719363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"63'12","reported_seq":"284","reported_epoch":"293","state":"active+clean","last_fresh":"2021-02-10T04:31:50.107175+0000","last_change":"2021-02-10T04:30:13.089576+0000","last_active":"2021-02-10T04:31:50.107175+0000","last_peered":"2021-02-10T04:31:50.107175+0000","last_clean":"2021-02-10T04:31:50.107175+0000","last_became_active":"2021-02-10T04:25:49.013196+0000","last_became_peered":"2021-02-10T04:25:49.013196+0000","last_unstale":"2021-02-10T04:31:50.107175+0000","last_undegraded":"2021-02-10T04:31:50.107175+0000","last_fullsized":"2021-02-10T04:31:50.107175+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2021-02-10T04:30:13.089494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:30:13.089494+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"182","reported_epoch":"295","state":"active+clean","last_fresh":"2021-02-10T04:31:52.109187+0000","last_change":"2021-02-10T04:30:11.537208+0000","last_active":"2021-02-10T04:31:52.109187+0000","last_peered":"2021-02-10T04:31:52.109187+0000","last_clean":"2021-02-10T04:31:52.109187+0000","last_became_active":"2021-02-10T04:29:07.157944+0000","last_became_peered":"2021-02-10T04:29:07.157944+0000","last_unstale":"2021-02-10T04:31:52.109187+0000","last_undegraded":"2021-02-10T04:31:52.109187+0000","last_fullsized":"2021-02-10T04:31:52.109187+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:11.537136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.537136+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"175","reported_epoch":"293","state":"active+clean","last_fresh":"2021-02-10T04:31:50.107337+0000","last_change":"2021-02-10T04:29:07.146984+0000","last_active":"2021-02-10T04:31:50.107337+0000","last_peered":"2021-02-10T04:31:50.107337+0000","last_clean":"2021-02-10T04:31:50.107337+0000","last_became_active":"2021-02-10T04:29:07.146525+0000","last_became_peered":"2021-02-10T04:29:07.146525+0000","last_unstale":"2021-02-10T04:31:50.107337+0000","last_undegraded":"2021-02-10T04:31:50.107337+0000","last_fullsized":"2021-02-10T04:31:50.107337+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:46.493066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:46.493066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"254","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.108073+0000","last_change":"2021-02-10T04:29:22.475613+0000","last_active":"2021-02-10T04:31:51.108073+0000","last_peered":"2021-02-10T04:31:51.108073+0000","last_clean":"2021-02-10T04:31:51.108073+0000","last_became_active":"2021-02-10T04:25:53.042601+0000","last_became_peered":"2021-02-10T04:25:53.042601+0000","last_unstale":"2021-02-10T04:31:51.108073+0000","last_undegraded":"2021-02-10T04:31:51.108073+0000","last_fullsized":"2021-02-10T04:31:51.108073+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:22.475504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:22.475504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"78'32","reported_seq":"311","reported_epoch":"293","state":"active+clean","last_fresh":"2021-02-10T04:31:50.106575+0000","last_change":"2021-02-10T04:28:35.520489+0000","last_active":"2021-02-10T04:31:50.106575+0000","last_peered":"2021-02-10T04:31:50.106575+0000","last_clean":"2021-02-10T04:31:50.106575+0000","last_became_active":"2021-02-10T04:25:49.010481+0000","last_became_peered":"2021-02-10T04:25:49.010481+0000","last_unstale":"2021-02-10T04:31:50.106575+0000","last_undegraded":"2021-02-10T04:31:50.106575+0000","last_fullsized":"2021-02-10T04:31:50.106575+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'32","last_scrub_stamp":"2021-02-10T04:28:35.520410+0000","last_deep_scrub":"78'32","last_deep_scrub_stamp":"2021-02-10T04:27:09.170428+0000","last_clean_scrub_stamp":"2021-02-10T04:28:35.520410+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":47,"num_read_kb":28,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"266","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.150014+0000","last_change":"2021-02-10T04:29:29.355356+0000","last_active":"2021-02-10T04:31:53.150014+0000","last_peered":"2021-02-10T04:31:53.150014+0000","last_clean":"2021-02-10T04:31:53.150014+0000","last_became_active":"2021-02-10T04:25:34.327162+0000","last_became_peered":"2021-02-10T04:25:34.327162+0000","last_unstale":"2021-02-10T04:31:53.150014+0000","last_undegraded":"2021-02-10T04:31:53.150014+0000","last_fullsized":"2021-02-10T04:31:53.150014+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.355257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.355257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"256","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.111639+0000","last_change":"2021-02-10T04:28:40.098300+0000","last_active":"2021-02-10T04:31:51.111639+0000","last_peered":"2021-02-10T04:31:51.111639+0000","last_clean":"2021-02-10T04:31:51.111639+0000","last_became_active":"2021-02-10T04:25:51.030405+0000","last_became_peered":"2021-02-10T04:25:51.030405+0000","last_unstale":"2021-02-10T04:31:51.111639+0000","last_undegraded":"2021-02-10T04:31:51.111639+0000","last_fullsized":"2021-02-10T04:31:51.111639+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:40.098239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:40.098239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"248","reported_epoch":"293","state":"active+clean","last_fresh":"2021-02-10T04:31:50.107120+0000","last_change":"2021-02-10T04:27:51.679613+0000","last_active":"2021-02-10T04:31:50.107120+0000","last_peered":"2021-02-10T04:31:50.107120+0000","last_clean":"2021-02-10T04:31:50.107120+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:31:50.107120+0000","last_undegraded":"2021-02-10T04:31:50.107120+0000","last_fullsized":"2021-02-10T04:31:50.107120+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:51.679515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:51.679515+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"63'44","reported_seq":"349","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.146657+0000","last_change":"2021-02-10T04:27:49.269619+0000","last_active":"2021-02-10T04:31:53.146657+0000","last_peered":"2021-02-10T04:31:53.146657+0000","last_clean":"2021-02-10T04:31:53.146657+0000","last_became_active":"2021-02-10T04:25:49.008451+0000","last_became_peered":"2021-02-10T04:25:49.008451+0000","last_unstale":"2021-02-10T04:31:53.146657+0000","last_undegraded":"2021-02-10T04:31:53.146657+0000","last_fullsized":"2021-02-10T04:31:53.146657+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'44","last_scrub_stamp":"2021-02-10T04:27:49.269539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:49.269539+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":89,"num_read_kb":66,"num_write":45,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"276","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.144084+0000","last_change":"2021-02-10T04:29:28.719237+0000","last_active":"2021-02-10T04:31:53.144084+0000","last_peered":"2021-02-10T04:31:53.144084+0000","last_clean":"2021-02-10T04:31:53.144084+0000","last_became_active":"2021-02-10T04:25:36.336112+0000","last_became_peered":"2021-02-10T04:25:36.336112+0000","last_unstale":"2021-02-10T04:31:53.144084+0000","last_undegraded":"2021-02-10T04:31:53.144084+0000","last_fullsized":"2021-02-10T04:31:53.144084+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.719147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.719147+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"250","reported_epoch":"293","state":"active+clean","last_fresh":"2021-02-10T04:31:50.106428+0000","last_change":"2021-02-10T04:27:49.628172+0000","last_active":"2021-02-10T04:31:50.106428+0000","last_peered":"2021-02-10T04:31:50.106428+0000","last_clean":"2021-02-10T04:31:50.106428+0000","last_became_active":"2021-02-10T04:25:51.029110+0000","last_became_peered":"2021-02-10T04:25:51.029110+0000","last_unstale":"2021-02-10T04:31:50.106428+0000","last_undegraded":"2021-02-10T04:31:50.106428+0000","last_fullsized":"2021-02-10T04:31:50.106428+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:49.628090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:49.628090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"255","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.148675+0000","last_change":"2021-02-10T04:29:25.382883+0000","last_active":"2021-02-10T04:31:53.148675+0000","last_peered":"2021-02-10T04:31:53.148675+0000","last_clean":"2021-02-10T04:31:53.148675+0000","last_became_active":"2021-02-10T04:25:53.150917+0000","last_became_peered":"2021-02-10T04:25:53.150917+0000","last_unstale":"2021-02-10T04:31:53.148675+0000","last_undegraded":"2021-02-10T04:31:53.148675+0000","last_fullsized":"2021-02-10T04:31:53.148675+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:25.382793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.382793+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.20","version":"0'0","reported_seq":"244","reported_epoch":"294","state":"active+clean","last_fresh":"2021-02-10T04:31:51.108155+0000","last_change":"2021-02-10T04:30:11.492843+0000","last_active":"2021-02-10T04:31:51.108155+0000","last_peered":"2021-02-10T04:31:51.108155+0000","last_clean":"2021-02-10T04:31:51.108155+0000","last_became_active":"2021-02-10T04:29:07.148937+0000","last_became_peered":"2021-02-10T04:29:07.148937+0000","last_unstale":"2021-02-10T04:31:51.108155+0000","last_undegraded":"2021-02-10T04:31:51.108155+0000","last_fullsized":"2021-02-10T04:31:51.108155+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:11.492788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:41.375276+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.492788+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":"293","reported_epoch":"292","state":"active+clean","last_fresh":"2021-02-10T04:31:49.105866+0000","last_change":"2021-02-10T04:29:21.708787+0000","last_active":"2021-02-10T04:31:49.105866+0000","last_peered":"2021-02-10T04:31:49.105866+0000","last_clean":"2021-02-10T04:31:49.105866+0000","last_became_active":"2021-02-10T04:28:39.045354+0000","last_became_peered":"2021-02-10T04:28:39.045354+0000","last_unstale":"2021-02-10T04:31:49.105866+0000","last_undegraded":"2021-02-10T04:31:49.105866+0000","last_fullsized":"2021-02-10T04:31:49.105866+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:21.708701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:21.708701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":"275","reported_epoch":"295","state":"active+clean","last_fresh":"2021-02-10T04:31:52.107926+0000","last_change":"2021-02-10T04:28:42.382533+0000","last_active":"2021-02-10T04:31:52.107926+0000","last_peered":"2021-02-10T04:31:52.107926+0000","last_clean":"2021-02-10T04:31:52.107926+0000","last_became_active":"2021-02-10T04:28:23.129832+0000","last_became_peered":"2021-02-10T04:28:23.129832+0000","last_unstale":"2021-02-10T04:31:52.107926+0000","last_undegraded":"2021-02-10T04:31:52.107926+0000","last_fullsized":"2021-02-10T04:31:52.107926+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:42.382464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:42.382464+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":"220","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.147720+0000","last_change":"2021-02-10T04:28:48.409915+0000","last_active":"2021-02-10T04:31:53.147720+0000","last_peered":"2021-02-10T04:31:53.147720+0000","last_clean":"2021-02-10T04:31:53.147720+0000","last_became_active":"2021-02-10T04:28:23.129523+0000","last_became_peered":"2021-02-10T04:28:23.129523+0000","last_unstale":"2021-02-10T04:31:53.147720+0000","last_undegraded":"2021-02-10T04:31:53.147720+0000","last_fullsized":"2021-02-10T04:31:53.147720+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:48.409826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:48.409826+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":"219","reported_epoch":"295","state":"active+clean","last_fresh":"2021-02-10T04:31:52.108743+0000","last_change":"2021-02-10T04:28:46.203018+0000","last_active":"2021-02-10T04:31:52.108743+0000","last_peered":"2021-02-10T04:31:52.108743+0000","last_clean":"2021-02-10T04:31:52.108743+0000","last_became_active":"2021-02-10T04:28:23.126368+0000","last_became_peered":"2021-02-10T04:28:23.126368+0000","last_unstale":"2021-02-10T04:31:52.108743+0000","last_undegraded":"2021-02-10T04:31:52.108743+0000","last_fullsized":"2021-02-10T04:31:52.108743+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:46.202944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:46.202944+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.25","version":"53'2","reported_seq":"235","reported_epoch":"293","state":"active+clean","last_fresh":"2021-02-10T04:31:50.108269+0000","last_change":"2021-02-10T04:28:40.422198+0000","last_active":"2021-02-10T04:31:50.108269+0000","last_peered":"2021-02-10T04:31:50.108269+0000","last_clean":"2021-02-10T04:31:50.108269+0000","last_became_active":"2021-02-10T04:28:23.467770+0000","last_became_peered":"2021-02-10T04:28:23.467770+0000","last_unstale":"2021-02-10T04:31:50.108269+0000","last_undegraded":"2021-02-10T04:31:50.108269+0000","last_fullsized":"2021-02-10T04:31:50.108269+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:28:40.422115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:40.422115+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.26","version":"53'1","reported_seq":"205","reported_epoch":"292","state":"active+clean","last_fresh":"2021-02-10T04:31:49.106632+0000","last_change":"2021-02-10T04:29:22.801988+0000","last_active":"2021-02-10T04:31:49.106632+0000","last_peered":"2021-02-10T04:31:49.106632+0000","last_clean":"2021-02-10T04:31:49.106632+0000","last_became_active":"2021-02-10T04:28:39.043300+0000","last_became_peered":"2021-02-10T04:28:39.043300+0000","last_unstale":"2021-02-10T04:31:49.106632+0000","last_undegraded":"2021-02-10T04:31:49.106632+0000","last_fullsized":"2021-02-10T04:31:49.106632+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:29:22.801929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:22.801929+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":"246","reported_epoch":"293","state":"active+clean","last_fresh":"2021-02-10T04:31:50.108036+0000","last_change":"2021-02-10T04:29:07.146256+0000","last_active":"2021-02-10T04:31:50.108036+0000","last_peered":"2021-02-10T04:31:50.108036+0000","last_clean":"2021-02-10T04:31:50.108036+0000","last_became_active":"2021-02-10T04:29:07.145907+0000","last_became_peered":"2021-02-10T04:29:07.145907+0000","last_unstale":"2021-02-10T04:31:50.108036+0000","last_undegraded":"2021-02-10T04:31:50.108036+0000","last_fullsized":"2021-02-10T04:31:50.108036+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:39.388884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:39.388884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":"220","reported_epoch":"296","state":"active+clean","last_fresh":"2021-02-10T04:31:53.145376+0000","last_change":"2021-02-10T04:29:23.440451+0000","last_active":"2021-02-10T04:31:53.145376+0000","last_peered":"2021-02-10T04:31:53.145376+0000","last_clean":"2021-02-10T04:31:53.145376+0000","last_became_active":"2021-02-10T04:28:23.129469+0000","last_became_peered":"2021-02-10T04:28:23.129469+0000","last_unstale":"2021-02-10T04:31:53.145376+0000","last_undegraded":"2021-02-10T04:31:53.145376+0000","last_fullsized":"2021-02-10T04:31:53.145376+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.440391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.440391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":"219","reported_epoch":"295","state":"active+clean","last_fresh":"2021-02-10T04:31:52.110276+0000","last_change":"2021-02-10T04:28:45.220352+0000","last_active":"2021-02-10T04:31:52.110276+0000","last_peered":"2021-02-10T04:31:52.110276+0000","last_clean":"2021-02-10T04:31:52.110276+0000","last_became_active":"2021-02-10T04:28:23.126795+0000","last_became_peered":"2021-02-10T04:28:23.126795+0000","last_unstale":"2021-02-10T04:31:52.110276+0000","last_undegraded":"2021-02-10T04:31:52.110276+0000","last_fullsized":"2021-02-10T04:31:52.110276+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_clean_scrub_stamp":"2021-02-10T04:28:45.220258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":118863616,"num_objects":47,"num_object_clones":9,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":1,"num_read":1957,"num_read_kb":1830354,"num_write":1558,"num_write_kb":573489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":280592384,"data_stored":296596384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2928,"ondisk_log_size":2928,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":1945,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":17,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":5835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25,"ondisk_log_size":25,"up":126,"acting":126,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":361,"num_read_kb":356,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":1453,"num_read_kb":926,"num_write":734,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":782,"ondisk_log_size":782,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748364877,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":42580,"kb_used_data":23820,"kb_used_omap":0,"kb_used_meta":18752,"kb_avail":93702572,"statfs":{"total":95995035648,"available":95951433728,"internally_reserved":0,"allocated":24391680,"data_stored":23796790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19202048},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:31:45 2021","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":1.0129999999999999,"15min":0.92800000000000005},"min":{"1min":0.53300000000000003,"5min":0.441,"15min":0.441},"max":{"1min":2.1120000000000001,"5min":18.681000000000001,"15min":18.681000000000001},"last":9.8379999999999992},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.121,"15min":0.94699999999999995},"min":{"1min":0.56299999999999994,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":4.1109999999999998,"5min":18.274999999999999,"15min":18.274999999999999},"last":9.7609999999999992}]},{"osd":1,"last update":"Wed Feb 10 04:31:45 2021","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":1.3300000000000001,"15min":1.038},"min":{"1min":0.55500000000000005,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.3720000000000001,"5min":41.966000000000001,"15min":41.966000000000001},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.88,"5min":1.4359999999999999,"15min":1.161},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.131,"5min":41.804000000000002,"15min":41.804000000000002},"last":0.97499999999999998}]},{"osd":2,"last update":"Wed Feb 10 04:31:45 2021","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":2.6549999999999998,"15min":1.5720000000000001},"min":{"1min":0.70099999999999996,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.3819999999999999,"5min":125.714,"15min":125.714},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.1699999999999999,"5min":2.7080000000000002,"15min":1.5429999999999999},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.75,"5min":125.53700000000001,"15min":125.53700000000001},"last":0.73199999999999998}]},{"osd":3,"last update":"Wed Feb 10 04:31:45 2021","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":4.6459999999999999,"15min":2.226},"min":{"1min":0.82399999999999995,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.591,"5min":192.95500000000001,"15min":192.95500000000001},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.151,"5min":4.7640000000000002,"15min":2.359},"min":{"1min":0.94099999999999995,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.339,"5min":192.601,"15min":192.601},"last":1.2969999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:31:45 2021","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.1910000000000001,"15min":1.0229999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.64900000000000002},"max":{"1min":3.8050000000000002,"5min":3.8050000000000002,"15min":3.8050000000000002},"last":1.004},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.1719999999999999,"15min":1.1339999999999999},"min":{"1min":0.748,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":2.1400000000000001,"5min":2.847,"15min":2.847},"last":1.3280000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:31:45 2021","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.294,"15min":1.194},"min":{"1min":0.85699999999999998,"5min":0.68899999999999995,"15min":0.624},"max":{"1min":1.5609999999999999,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":1.46},{"interface":"front","average":{"1min":1.488,"5min":1.3740000000000001,"15min":1.2569999999999999},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":4.6790000000000003,"5min":4.6790000000000003,"15min":4.6790000000000003},"last":1.2270000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:31:45 2021","interfaces":[{"interface":"back","average":{"1min":3.1349999999999998,"5min":1.8069999999999999,"15min":1.4299999999999999},"min":{"1min":0.98999999999999999,"5min":0.94199999999999995,"15min":0.86899999999999999},"max":{"1min":29.206,"5min":29.206,"15min":29.206},"last":1.3540000000000001},{"interface":"front","average":{"1min":2.8940000000000001,"5min":1.7470000000000001,"15min":1.399},"min":{"1min":1.0269999999999999,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":26.913,"5min":26.913,"15min":26.913},"last":1.427}]}]},{"osd":6,"up_from":43,"seq":184683593810,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64872,"kb_used_data":44448,"kb_used_omap":0,"kb_used_meta":20416,"kb_avail":93680280,"statfs":{"total":95995035648,"available":95928606720,"internally_reserved":0,"allocated":45514752,"data_stored":44758585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20905984},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:31:26 2021","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":1.921,"15min":1.2729999999999999},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":3.105,"5min":64.164000000000001,"15min":64.164000000000001},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.81299999999999994,"5min":1.998,"15min":1.347},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.3660000000000001,"5min":64.277000000000001,"15min":64.277000000000001},"last":0.77200000000000002}]},{"osd":1,"last update":"Wed Feb 10 04:31:26 2021","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.3029999999999999,"15min":1.1259999999999999},"min":{"1min":0.63500000000000001,"5min":0.61599999999999999,"15min":0.61099999999999999},"max":{"1min":10.331,"5min":18.177,"15min":18.177},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.2390000000000001,"15min":1.0649999999999999},"min":{"1min":0.624,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.7390000000000001,"5min":18.898,"15min":18.898},"last":0.57799999999999996}]},{"osd":2,"last update":"Wed Feb 10 04:31:26 2021","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":1.274,"15min":1.1319999999999999},"min":{"1min":0.63400000000000001,"5min":0.625,"15min":0.625},"max":{"1min":1.7,"5min":19.474,"15min":19.474},"last":0.67900000000000005},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.3340000000000001,"15min":1.1160000000000001},"min":{"1min":0.73999999999999999,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.663,"5min":19.227,"15min":19.227},"last":1.0269999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:31:26 2021","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.411,"15min":1.2270000000000001},"min":{"1min":0.80800000000000005,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":3.7480000000000002,"5min":15.316000000000001,"15min":15.316000000000001},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.4650000000000001,"15min":1.218},"min":{"1min":0.88900000000000001,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.7,"5min":15.35,"15min":15.35},"last":1.137}]},{"osd":4,"last update":"Wed Feb 10 04:31:26 2021","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.2749999999999999,"15min":1.0549999999999999},"min":{"1min":0.75700000000000001,"5min":0.61699999999999999,"15min":0.58999999999999997},"max":{"1min":3.4969999999999999,"5min":3.4969999999999999,"15min":3.4969999999999999},"last":1.002},{"interface":"front","average":{"1min":0.91300000000000003,"5min":1.012,"15min":0.92800000000000005},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.177,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.92500000000000004}]},{"osd":5,"last update":"Wed Feb 10 04:31:26 2021","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":1.1739999999999999,"15min":1.044},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.4339999999999999,"5min":5.3529999999999998,"15min":5.3529999999999998},"last":0.751},{"interface":"front","average":{"1min":1.26,"5min":1.4139999999999999,"15min":1.218},"min":{"1min":0.84099999999999997,"5min":0.81799999999999995,"15min":0.69499999999999995},"max":{"1min":1.651,"5min":2.4550000000000001,"15min":2.4550000000000001},"last":1.2470000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:31:50 2021","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.248,"15min":1.1419999999999999},"min":{"1min":0.76900000000000002,"5min":0.75800000000000001,"15min":0.73599999999999999},"max":{"1min":1.47,"5min":2.6680000000000001,"15min":2.6680000000000001},"last":1.0509999999999999},{"interface":"front","average":{"1min":1.425,"5min":1.54,"15min":1.4359999999999999},"min":{"1min":0.93500000000000005,"5min":0.90300000000000002,"15min":0.89000000000000001},"max":{"1min":1.7709999999999999,"5min":3.5049999999999999,"15min":3.5049999999999999},"last":1.333}]}]},{"osd":1,"up_from":13,"seq":55834574947,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":44724,"kb_used_data":24236,"kb_used_omap":0,"kb_used_meta":20480,"kb_avail":93700428,"statfs":{"total":95995035648,"available":95949238272,"internally_reserved":0,"allocated":24817664,"data_stored":24037137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20971520},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,2],"upper_bound":524288},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:31:05 2021","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.093,"15min":0.77600000000000002},"min":{"1min":0.59299999999999997,"5min":0.47899999999999998,"15min":0.36399999999999999},"max":{"1min":2.262,"5min":33.195,"15min":33.195},"last":0.879},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.1659999999999999,"15min":0.84799999999999998},"min":{"1min":0.53600000000000003,"5min":0.51800000000000002,"15min":0.41199999999999998},"max":{"1min":1.7490000000000001,"5min":32.965000000000003,"15min":32.965000000000003},"last":0.82299999999999995}]},{"osd":2,"last update":"Wed Feb 10 04:31:24 2021","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.1950000000000001,"15min":0.90600000000000003},"min":{"1min":0.57599999999999996,"5min":0.53800000000000003,"15min":0.48699999999999999},"max":{"1min":3.3519999999999999,"5min":33.231999999999999,"15min":33.231999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":1.294,"5min":1.2829999999999999,"15min":0.95899999999999996},"min":{"1min":0.73299999999999998,"5min":0.50800000000000001,"15min":0.46000000000000002},"max":{"1min":2.8519999999999999,"5min":33.064,"15min":33.064},"last":0.95999999999999996}]},{"osd":3,"last update":"Wed Feb 10 04:31:38 2021","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.085,"15min":0.97099999999999997},"min":{"1min":0.54700000000000004,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":3.883,"5min":3.883,"15min":3.883},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.1060000000000001,"15min":1.0449999999999999},"min":{"1min":0.60699999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":3.9129999999999998,"5min":3.9129999999999998,"15min":3.9129999999999998},"last":1.6479999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:30:56 2021","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.224,"15min":1.1679999999999999},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":3.0049999999999999,"5min":3.0049999999999999,"15min":3.0049999999999999},"last":1.175},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.246,"15min":1.22},"min":{"1min":0.81899999999999995,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":2.8130000000000002,"5min":2.8130000000000002,"15min":2.8130000000000002},"last":1.3100000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:31:15 2021","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.167,"15min":1.2490000000000001},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":4.7969999999999997,"5min":4.7969999999999997,"15min":5.5860000000000003},"last":0.998},{"interface":"front","average":{"1min":1.806,"5min":1.431,"15min":1.5960000000000001},"min":{"1min":0.88100000000000001,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":5.2069999999999999,"5min":5.2069999999999999,"15min":9.5310000000000006},"last":1.597}]},{"osd":6,"last update":"Wed Feb 10 04:31:27 2021","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.3320000000000001,"15min":1.304},"min":{"1min":0.90400000000000003,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.6469999999999998,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":1.421},{"interface":"front","average":{"1min":1.625,"5min":1.234,"15min":1.282},"min":{"1min":0.81599999999999995,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":4.5960000000000001,"5min":4.5960000000000001,"15min":4.5960000000000001},"last":1.1160000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:31:42 2021","interfaces":[{"interface":"back","average":{"1min":1.5660000000000001,"5min":1.47,"15min":1.633},"min":{"1min":1.0269999999999999,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":5.0099999999999998,"5min":5.0099999999999998,"15min":7.5540000000000003},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.4410000000000001,"15min":1.601},"min":{"1min":1.04,"5min":1.006,"15min":1.006},"max":{"1min":2.7570000000000001,"5min":2.7570000000000001,"15min":8.1020000000000003},"last":1.1990000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738473,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72892,"kb_used_data":50996,"kb_used_omap":0,"kb_used_meta":21888,"kb_avail":93672260,"statfs":{"total":95995035648,"available":95920394240,"internally_reserved":0,"allocated":52219904,"data_stored":55645193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":22413312},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 04:31:00 2021","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":0.91500000000000004,"15min":0.69099999999999995},"min":{"1min":0.52400000000000002,"5min":0.48099999999999998,"15min":0.40799999999999997},"max":{"1min":2.9399999999999999,"5min":5.125,"15min":5.125},"last":1.02},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.96599999999999997,"15min":0.80800000000000005},"min":{"1min":0.58899999999999997,"5min":0.51200000000000001,"15min":0.45000000000000001},"max":{"1min":1.302,"5min":7.5549999999999997,"15min":7.5549999999999997},"last":0.60199999999999998}]},{"osd":2,"last update":"Wed Feb 10 04:31:21 2021","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.044,"15min":0.86299999999999999},"min":{"1min":0.66400000000000003,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.46,"5min":3.5019999999999998,"15min":3.5019999999999998},"last":1.202},{"interface":"front","average":{"1min":1.151,"5min":0.91000000000000003,"15min":0.83399999999999996},"min":{"1min":0.59499999999999997,"5min":0.51300000000000001,"15min":0.495},"max":{"1min":2.7280000000000002,"5min":2.9279999999999999,"15min":2.9279999999999999},"last":0.77600000000000002}]},{"osd":3,"last update":"Wed Feb 10 04:31:35 2021","interfaces":[{"interface":"back","average":{"1min":1.784,"5min":2.3759999999999999,"15min":1.3380000000000001},"min":{"1min":0.72999999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":13.819000000000001,"5min":126.587,"15min":126.587},"last":0.95299999999999996},{"interface":"front","average":{"1min":1.665,"5min":2.5430000000000001,"15min":1.546},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":9.9149999999999991,"5min":126.88200000000001,"15min":126.88200000000001},"last":1.1279999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:30:53 2021","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.1240000000000001,"15min":1.1679999999999999},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.669,"5min":2.6440000000000001,"15min":2.6440000000000001},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.214,"5min":1.2849999999999999,"15min":1.202},"min":{"1min":0.77800000000000002,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":2.375,"5min":6.992,"15min":6.992},"last":1.0649999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:31:08 2021","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.3149999999999999,"15min":1.2569999999999999},"min":{"1min":0.70599999999999996,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":2.2890000000000001,"5min":5.3410000000000002,"15min":5.3410000000000002},"last":1.5880000000000001},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.3580000000000001,"15min":1.3240000000000001},"min":{"1min":0.88800000000000001,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":3.2989999999999999,"5min":3.2989999999999999,"15min":3.2989999999999999},"last":1.4079999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:31:31 2021","interfaces":[{"interface":"back","average":{"1min":2.1099999999999999,"5min":1.5529999999999999,"15min":1.391},"min":{"1min":1.075,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":14.063000000000001,"5min":14.063000000000001,"15min":14.063000000000001},"last":1.8360000000000001},{"interface":"front","average":{"1min":1.524,"5min":1.45,"15min":1.3959999999999999},"min":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":2.4790000000000001,"5min":3.6720000000000002,"15min":3.6720000000000002},"last":1.468}]},{"osd":7,"last update":"Wed Feb 10 04:31:41 2021","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.3069999999999999,"15min":1.2769999999999999},"min":{"1min":0.97099999999999997,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":2.9249999999999998,"5min":2.9249999999999998,"15min":2.9249999999999998},"last":1.107},{"interface":"front","average":{"1min":2.177,"5min":1.6379999999999999,"15min":1.46},"min":{"1min":0.98199999999999998,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":14.189,"5min":14.189,"15min":14.189},"last":1.7230000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411425,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":42472,"kb_used_data":22304,"kb_used_omap":0,"kb_used_meta":20160,"kb_avail":93702680,"statfs":{"total":95995035648,"available":95951544320,"internally_reserved":0,"allocated":22839296,"data_stored":22841919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20643840},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,4],"upper_bound":262144},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:31:20 2021","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.83499999999999996,"15min":0.73199999999999998},"min":{"1min":0.52700000000000002,"5min":0.496,"15min":0.48699999999999999},"max":{"1min":1.9079999999999999,"5min":2.222,"15min":2.222},"last":1.2390000000000001},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.85999999999999999,"15min":0.75900000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.46000000000000002},"max":{"1min":1.5680000000000001,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":0.86899999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:31:20 2021","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":0.996,"15min":0.84299999999999997},"min":{"1min":0.64200000000000002,"5min":0.51200000000000001,"15min":0.495},"max":{"1min":2.1110000000000002,"5min":5.0650000000000004,"15min":5.0650000000000004},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.80800000000000005,"15min":0.72699999999999998},"min":{"1min":0.55300000000000005,"5min":0.51200000000000001,"15min":0.45000000000000001},"max":{"1min":1.1299999999999999,"5min":2.6779999999999999,"15min":2.6779999999999999},"last":0.63}]},{"osd":3,"last update":"Wed Feb 10 04:31:39 2021","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":0.98899999999999999,"15min":0.87},"min":{"1min":0.72699999999999998,"5min":0.56100000000000005,"15min":0.53500000000000003},"max":{"1min":1.6319999999999999,"5min":3.1160000000000001,"15min":3.1160000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.232,"15min":1.0609999999999999},"min":{"1min":1.0509999999999999,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.9890000000000001,"5min":7.0309999999999997,"15min":7.0309999999999997},"last":0.78500000000000003}]},{"osd":4,"last update":"Wed Feb 10 04:30:52 2021","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.2709999999999999,"15min":1.2110000000000001},"min":{"1min":0.92200000000000004,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.891,"5min":2.089,"15min":2.089},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.1679999999999999,"15min":1.1379999999999999},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":2.5819999999999999,"5min":2.5819999999999999,"15min":2.5819999999999999},"last":0.91600000000000004}]},{"osd":5,"last update":"Wed Feb 10 04:31:12 2021","interfaces":[{"interface":"back","average":{"1min":1.5309999999999999,"5min":1.3380000000000001,"15min":1.3240000000000001},"min":{"1min":1.042,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":2.29,"5min":2.29,"15min":2.29},"last":1.3720000000000001},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.252,"15min":1.2030000000000001},"min":{"1min":0.85999999999999999,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":2.1389999999999998,"5min":2.4089999999999998,"15min":2.4089999999999998},"last":1.0589999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:31:30 2021","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.353,"15min":1.3220000000000001},"min":{"1min":1.0329999999999999,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.8029999999999999,"5min":5.2430000000000003,"15min":5.2430000000000003},"last":1.5800000000000001},{"interface":"front","average":{"1min":1.6719999999999999,"5min":1.468,"15min":1.454},"min":{"1min":1.196,"5min":1.006,"15min":1.006},"max":{"1min":2.181,"5min":2.6629999999999998,"15min":2.6629999999999998},"last":1.623}]},{"osd":7,"last update":"Wed Feb 10 04:31:46 2021","interfaces":[{"interface":"back","average":{"1min":1.7050000000000001,"5min":1.5149999999999999,"15min":1.5089999999999999},"min":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"max":{"1min":2.6560000000000001,"5min":2.6560000000000001,"15min":2.6560000000000001},"last":1.508},{"interface":"front","average":{"1min":1.6830000000000001,"5min":1.343,"15min":1.296},"min":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":9.9220000000000006,"5min":9.9220000000000006,"15min":9.9220000000000006},"last":1.095}]}]},{"osd":3,"up_from":25,"seq":107374182492,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":89616,"kb_used_data":61512,"kb_used_omap":0,"kb_used_meta":28096,"kb_avail":93655536,"statfs":{"total":95995035648,"available":95903268864,"internally_reserved":0,"allocated":62988288,"data_stored":69947627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":28770304},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:31:41 2021","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.046,"15min":0.84099999999999997},"min":{"1min":0.62,"5min":0.49199999999999999,"15min":0.46100000000000002},"max":{"1min":1.6970000000000001,"5min":17.233000000000001,"15min":17.233000000000001},"last":0.753},{"interface":"front","average":{"1min":0.88500000000000001,"5min":1.3340000000000001,"15min":1.0189999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.5529999999999999,"5min":20.216999999999999,"15min":20.216999999999999},"last":0.77900000000000003}]},{"osd":1,"last update":"Wed Feb 10 04:31:41 2021","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.0389999999999999,"15min":0.92900000000000005},"min":{"1min":0.71099999999999997,"5min":0.54400000000000004,"15min":0.52800000000000002},"max":{"1min":2.1379999999999999,"5min":5.9130000000000003,"15min":5.9130000000000003},"last":1.0509999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.97099999999999997,"15min":0.89000000000000001},"min":{"1min":0.55700000000000005,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.4610000000000001,"5min":5.4039999999999999,"15min":5.4039999999999999},"last":1.01}]},{"osd":2,"last update":"Wed Feb 10 04:31:41 2021","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1060000000000001,"15min":1.002},"min":{"1min":0.66200000000000003,"5min":0.59999999999999998,"15min":0.58699999999999997},"max":{"1min":1.609,"5min":4.649,"15min":4.649},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.155,"5min":0.999,"15min":0.98699999999999999},"min":{"1min":0.76000000000000001,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.9890000000000001,"5min":2.3570000000000002,"15min":2.3570000000000002},"last":1.131}]},{"osd":4,"last update":"Wed Feb 10 04:31:50 2021","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.6779999999999999,"15min":1.3620000000000001},"min":{"1min":0.90400000000000003,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.748,"5min":24.882999999999999,"15min":24.882999999999999},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.6579999999999999,"15min":1.3720000000000001},"min":{"1min":0.98699999999999999,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.871,"5min":24.911999999999999,"15min":24.911999999999999},"last":1.29}]},{"osd":5,"last update":"Wed Feb 10 04:31:12 2021","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.8129999999999999,"15min":1.385},"min":{"1min":0.88300000000000001,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":2.8220000000000001,"5min":25.004999999999999,"15min":25.004999999999999},"last":1.5820000000000001},{"interface":"front","average":{"1min":1.607,"5min":1.766,"15min":1.4279999999999999},"min":{"1min":0.93400000000000005,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":2.6739999999999999,"5min":25.055,"15min":25.055},"last":1.524}]},{"osd":6,"last update":"Wed Feb 10 04:31:31 2021","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.958,"15min":1.5820000000000001},"min":{"1min":1.101,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":2.2170000000000001,"5min":25.143000000000001,"15min":25.143000000000001},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.722,"5min":1.877,"15min":1.5980000000000001},"min":{"1min":1.206,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":2.1880000000000002,"5min":25.181000000000001,"15min":25.181000000000001},"last":1.6000000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:31:46 2021","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.9630000000000001,"15min":1.6559999999999999},"min":{"1min":1.169,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":1.996,"5min":25.242999999999999,"15min":25.242999999999999},"last":1.8009999999999999},{"interface":"front","average":{"1min":1.7030000000000001,"5min":1.968,"15min":1.694},"min":{"1min":1.284,"5min":1.0209999999999999,"15min":1.0209999999999999},"max":{"1min":2.3500000000000001,"5min":25.286999999999999,"15min":25.286999999999999},"last":1.6699999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986265,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":38944,"kb_used_data":19800,"kb_used_omap":0,"kb_used_meta":19136,"kb_avail":93706208,"statfs":{"total":95995035648,"available":95955156992,"internally_reserved":0,"allocated":20275200,"data_stored":19568723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19595264},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4],"upper_bound":524288},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:30:51 2021","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.5329999999999999,"15min":1.1859999999999999},"min":{"1min":0.48699999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":2.6110000000000002,"5min":62.273000000000003,"15min":62.273000000000003},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.88600000000000001,"5min":1.587,"15min":1.1850000000000001},"min":{"1min":0.54200000000000004,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.9390000000000001,"5min":62.375999999999998,"15min":62.375999999999998},"last":0.748}]},{"osd":1,"last update":"Wed Feb 10 04:30:51 2021","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.85599999999999998,"15min":0.90400000000000003},"min":{"1min":0.58399999999999996,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.706,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.091,"15min":1.0449999999999999},"min":{"1min":0.67000000000000004,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":2.7549999999999999,"5min":2.7549999999999999,"15min":2.7549999999999999},"last":0.93500000000000005}]},{"osd":2,"last update":"Wed Feb 10 04:30:51 2021","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.9590000000000001,"15min":1.3759999999999999},"min":{"1min":0.80700000000000005,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":2.8650000000000002,"5min":62.521000000000001,"15min":62.521000000000001},"last":1.052},{"interface":"front","average":{"1min":1.2090000000000001,"5min":2.0619999999999998,"15min":1.458},"min":{"1min":0.79200000000000004,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":2.2530000000000001,"5min":62.572000000000003,"15min":62.572000000000003},"last":0.96699999999999997}]},{"osd":3,"last update":"Wed Feb 10 04:30:51 2021","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.9119999999999999,"15min":1.3320000000000001},"min":{"1min":0.72299999999999998,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":3.3300000000000001,"5min":62.113,"15min":62.113},"last":1.1990000000000001},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.8580000000000001,"15min":1.351},"min":{"1min":0.73699999999999999,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.8420000000000001,"5min":62.146000000000001,"15min":62.146000000000001},"last":1.234}]},{"osd":5,"last update":"Wed Feb 10 04:31:14 2021","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.1850000000000001,"15min":0.93799999999999994},"min":{"1min":0.64100000000000001,"5min":0.63200000000000001,"15min":0.57499999999999996},"max":{"1min":2.544,"5min":2.544,"15min":2.544},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.1259999999999999,"15min":0.92200000000000004},"min":{"1min":0.73799999999999999,"5min":0.66800000000000004,"15min":0.624},"max":{"1min":2.0150000000000001,"5min":2.0150000000000001,"15min":2.0150000000000001},"last":1.6970000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:31:27 2021","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.6819999999999999,"15min":1.264},"min":{"1min":1.012,"5min":0.79200000000000004,"15min":0.63100000000000001},"max":{"1min":2.9700000000000002,"5min":29.686,"15min":29.686},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.448,"5min":1.802,"15min":1.429},"min":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.79000000000000004},"max":{"1min":2.6960000000000002,"5min":29.510000000000002,"15min":29.510000000000002},"last":1.3069999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:31:46 2021","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.3859999999999999,"15min":1.3100000000000001},"min":{"1min":1.0289999999999999,"5min":0.89600000000000002,"15min":0.88600000000000001},"max":{"1min":1.54,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":1.284},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.5149999999999999,"15min":1.4219999999999999},"min":{"1min":1.087,"5min":0.91200000000000003,"15min":0.83099999999999996},"max":{"1min":2.105,"5min":3.4830000000000001,"15min":3.4830000000000001},"last":1.387}]}]},{"osd":5,"up_from":37,"seq":158913790038,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":44196,"kb_used_data":25052,"kb_used_omap":0,"kb_used_meta":19136,"kb_avail":93700956,"statfs":{"total":95995035648,"available":95949778944,"internally_reserved":0,"allocated":25653248,"data_stored":25609523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19595264},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:31:08 2021","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":0.98899999999999999,"15min":0.998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":8.4830000000000005,"5min":8.4830000000000005,"15min":8.4830000000000005},"last":1.734},{"interface":"front","average":{"1min":1.198,"5min":0.97199999999999998,"15min":0.93200000000000005},"min":{"1min":0.53000000000000003,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":5.6340000000000003,"5min":5.6340000000000003,"15min":5.6340000000000003},"last":0.88200000000000001}]},{"osd":1,"last update":"Wed Feb 10 04:31:08 2021","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.89900000000000002,"15min":0.92200000000000004},"min":{"1min":0.54200000000000004,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2.7879999999999998,"5min":2.7879999999999998,"15min":2.7879999999999998},"last":0.85199999999999998},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.0229999999999999,"15min":1.01},"min":{"1min":0.56599999999999995,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":5.7679999999999998,"5min":5.7679999999999998,"15min":5.7679999999999998},"last":1.149}]},{"osd":2,"last update":"Wed Feb 10 04:31:08 2021","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.946,"15min":1.3120000000000001},"min":{"1min":0.60999999999999999,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":7.9980000000000002,"5min":67.852000000000004,"15min":67.852000000000004},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.92500000000000004,"5min":1.8049999999999999,"15min":1.2549999999999999},"min":{"1min":0.66700000000000004,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.462,"5min":67.548000000000002,"15min":67.548000000000002},"last":0.97499999999999998}]},{"osd":3,"last update":"Wed Feb 10 04:31:08 2021","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":2.169,"15min":1.4490000000000001},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":8.3699999999999992,"5min":67.376999999999995,"15min":67.376999999999995},"last":1.625},{"interface":"front","average":{"1min":1.129,"5min":2.0539999999999998,"15min":1.397},"min":{"1min":0.85699999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.52,"5min":66.888999999999996,"15min":66.888999999999996},"last":1.0840000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:31:08 2021","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.151,"15min":0.98199999999999998},"min":{"1min":0.746,"5min":0.65300000000000002,"15min":0.63900000000000001},"max":{"1min":5.2290000000000001,"5min":5.2290000000000001,"15min":5.2290000000000001},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.3660000000000001,"15min":1.0860000000000001},"min":{"1min":0.73299999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":8.1449999999999996,"5min":8.1449999999999996,"15min":8.1449999999999996},"last":1.4870000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:31:29 2021","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.5,"15min":1.236},"min":{"1min":0.84499999999999997,"5min":0.77400000000000002,"15min":0.70599999999999996},"max":{"1min":8.2569999999999997,"5min":8.2569999999999997,"15min":8.2569999999999997},"last":2.0529999999999999},{"interface":"front","average":{"1min":1.222,"5min":1.2949999999999999,"15min":1.125},"min":{"1min":0.90300000000000002,"5min":0.77800000000000002,"15min":0.74399999999999999},"max":{"1min":1.5680000000000001,"5min":2.931,"15min":2.931},"last":1.2030000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:31:45 2021","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.367,"15min":1.236},"min":{"1min":0.91700000000000004,"5min":0.879,"15min":0.78400000000000003},"max":{"1min":5.4770000000000003,"5min":5.4770000000000003,"15min":5.4770000000000003},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.778,"5min":1.613,"15min":1.4430000000000001},"min":{"1min":0.96999999999999997,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":8.7840000000000007,"5min":8.7840000000000007,"15min":8.7840000000000007},"last":1.944}]}]}]}} 2021-02-10T04:31:59.963 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:32:00.071 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:32:00.175 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:32:00.191 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:32:00 smithi085 conmon[41540]: debug 2021-02-10T04:32:00.191 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:32:00 smithi085 conmon[41540]: 2021-02-10T04:32:00.047+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:00.208 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:00.213 INFO:tasks.rados.rados.0.smithi085.stdout:1067: done (15 left) 2021-02-10T04:32:00.213 INFO:tasks.rados.rados.0.smithi085.stdout:1068: write oid 6 current snap is 103 2021-02-10T04:32:00.213 INFO:tasks.rados.rados.0.smithi085.stdout:1065: finishing write tid 2 to smithi08575397-21 2021-02-10T04:32:00.213 INFO:tasks.rados.rados.0.smithi085.stdout:1065: finishing write tid 3 to smithi08575397-21 2021-02-10T04:32:00.213 INFO:tasks.rados.rados.0.smithi085.stdout:1065: finishing write tid 4 to smithi08575397-21 2021-02-10T04:32:00.213 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 564 (ObjNum 399 snap 103 seq_num 399) dirty exists 2021-02-10T04:32:00.213 INFO:tasks.rados.rados.0.smithi085.stdout:1065: left oid 21 (ObjNum 399 snap 103 seq_num 399) 2021-02-10T04:32:00.216 INFO:tasks.rados.rados.0.smithi085.stdout:1068: seq_num 400 ranges {722800=549956,2040562=682238} 2021-02-10T04:32:00.221 INFO:tasks.rados.rados.0.smithi085.stdout:1068: writing smithi08575397-6 from 722800 to 1272756 tid 1 2021-02-10T04:32:00.231 INFO:tasks.rados.rados.0.smithi085.stdout:1068: writing smithi08575397-6 from 2040562 to 2722800 tid 2 2021-02-10T04:32:00.232 INFO:tasks.rados.rados.0.smithi085.stdout:1068: finishing write tid 1 to smithi08575397-6 2021-02-10T04:32:00.232 INFO:tasks.rados.rados.0.smithi085.stdout:1065: done (15 left) 2021-02-10T04:32:00.232 INFO:tasks.rados.rados.0.smithi085.stdout:1066: done (14 left) 2021-02-10T04:32:00.232 INFO:tasks.rados.rados.0.smithi085.stdout:1069: rollback oid 7 current snap is 103 2021-02-10T04:32:00.232 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 7 to 101 2021-02-10T04:32:00.232 INFO:tasks.rados.rados.0.smithi085.stdout:1070: delete oid 43 current snap is 103 2021-02-10T04:32:00.240 INFO:tasks.rados.rados.0.smithi085.stdout:1069: finishing rollback tid 0 to smithi08575397-7 2021-02-10T04:32:00.240 INFO:tasks.rados.rados.0.smithi085.stdout:1069: finishing rollback tid 1 to smithi08575397-7 2021-02-10T04:32:00.241 INFO:tasks.rados.rados.0.smithi085.stdout:1068: finishing write tid 2 to smithi08575397-6 2021-02-10T04:32:00.241 INFO:tasks.rados.rados.0.smithi085.stdout:1069: finishing rollback tid 2 to smithi08575397-7 2021-02-10T04:32:00.241 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 476 (ObjNum 385 snap 100 seq_num 385) dirty exists 2021-02-10T04:32:00.241 INFO:tasks.rados.rados.0.smithi085.stdout:1068: finishing write tid 3 to smithi08575397-6 2021-02-10T04:32:00.241 INFO:tasks.rados.rados.0.smithi085.stdout:1069: done (15 left) 2021-02-10T04:32:00.241 INFO:tasks.rados.rados.0.smithi085.stdout:1070: done (14 left) 2021-02-10T04:32:00.242 INFO:tasks.rados.rados.0.smithi085.stdout:1071: copy_from oid 8 from oid 22 current snap is 103 2021-02-10T04:32:00.242 INFO:tasks.rados.rados.0.smithi085.stdout:1072: read oid 26 snap -1 2021-02-10T04:32:00.242 INFO:tasks.rados.rados.0.smithi085.stdout:1072: expect (ObjNum 343 snap 88 seq_num 343) 2021-02-10T04:32:00.242 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:00.242 INFO:tasks.rados.rados.0.smithi085.stdout:1068: finishing write tid 4 to smithi08575397-6 2021-02-10T04:32:00.242 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 357 (ObjNum 400 snap 103 seq_num 400) dirty exists 2021-02-10T04:32:00.242 INFO:tasks.rados.rados.0.smithi085.stdout:1068: left oid 6 (ObjNum 400 snap 103 seq_num 400) 2021-02-10T04:32:00.242 INFO:tasks.rados.rados.0.smithi085.stdout:1068: done (15 left) 2021-02-10T04:32:00.242 INFO:tasks.rados.rados.0.smithi085.stdout:1073: snap_remove snap 100 2021-02-10T04:32:00.320 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:32:00.333 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:32:00.424 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:32:00.462 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:00 smithi085 conmon[29319]: cluster 2021-02-10T04:31:59.180459+0000 mon.a ( 2021-02-10T04:32:00.462 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:00 smithi085 conmon[29319]: mon.0) 1049 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2021-02-10T04:32:00.462 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:00 smithi085 conmon[29319]: cluster 2021-02-10T04:32:00.047909+0000 mgr.y (mgr.14140) 464 : cluster [DBG] pgmap v603: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 164 active+clean; 121 MiB data, 451 MiB used, 715 GiB / 715 GiB avail; 8.3 MiB/s rd, 2.8 MiB/s wr, 16 op/s 2021-02-10T04:32:00.462 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:00 smithi085 conmon[35961]: cluster 2021-02-10T04:32:00.462 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:00 smithi085 conmon[35961]: 2021-02-10T04:32:00.462 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:00 smithi085 conmon[35961]: 2021 2021-02-10T04:32:00.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:00 smithi085 conmon[35961]: - 2021-02-10T04:32:00.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:00 smithi085 conmon[35961]: 02 2021-02-10T04:32:00.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:00 smithi085 conmon[35961]: - 2021-02-10T04:32:00.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:00 smithi085 conmon[35961]: 10 2021-02-10T04:32:00.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:00 smithi085 conmon[35961]: T 2021-02-10T04:32:00.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:00 smithi085 conmon[35961]: 04:31:59.180459 2021-02-10T04:32:00.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:00 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1049 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2021-02-10T04:32:00.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:00 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:00.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:00 smithi085 conmon[35961]: 10T04:32:00.047909+0000 mgr.y (mgr.14140) 2021-02-10T04:32:00.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:00 smithi085 conmon[35961]: 464 : cluster [DBG] pgmap v603: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 164 active+clean; 121 MiB data, 451 MiB used, 715 GiB / 715 GiB avail; 8.3 MiB/s rd, 2.8 MiB/s wr, 16 op/s 2021-02-10T04:32:00.474 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:00.591 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:32:00.625 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:32:00.626 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:32:00.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:00 smithi164 conmon[29288]: cluster 2021-02-10T04:31:59.180459 2021-02-10T04:32:00.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:00 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1049 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2021-02-10T04:32:00.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:00 smithi164 conmon[29288]: cluster 2021-02-10T04:32 2021-02-10T04:32:00.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:00 smithi164 conmon[29288]: :00.047909+0000 mgr.y (mgr.14140) 464 : cluster [DBG] pgmap v603: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 164 active+clean; 121 MiB data, 451 MiB used, 715 GiB / 715 GiB avail; 8.3 MiB/s rd, 2.8 MiB/s wr, 16 op/s 2021-02-10T04:32:00.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:00 smithi164 conmon[33677]: debug 2021-02-10T04:32:00.287+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:00.745 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:32:00.758 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:00 smithi085 conmon[49106]: debug 2021-02-10T04:32:00.759 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:00 smithi085 conmon[49106]: 2021-02-10T04:32:00.555+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:00.759 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:00 smithi085 conmon[49106]: 2021-02-10T04:32:00.788 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:01.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:00 smithi085 conmon[52836]: debug 2021-02-10T04:32:01.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:00 smithi085 conmon[52836]: 2021-02-10T04:32:00.968+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:01.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:00 smithi085 conmon[52836]: 2021-02-10T04:32:01.222 INFO:tasks.rados.rados.0.smithi085.stdout:1073: done (15 left) 2021-02-10T04:32:01.222 INFO:tasks.rados.rados.0.smithi085.stdout:1074: read oid 2 snap -1 2021-02-10T04:32:01.222 INFO:tasks.rados.rados.0.smithi085.stdout:1074: expect (ObjNum 373 snap 96 seq_num 373) 2021-02-10T04:32:01.222 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:01.222 INFO:tasks.rados.rados.0.smithi085.stdout:1072: expect (ObjNum 343 snap 88 seq_num 343) 2021-02-10T04:32:01.236 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is unset 2021-02-10T04:32:01.254 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:32:01.455 INFO:tasks.rados.rados.0.smithi085.stdout:1071: finishing copy_from to smithi08575397-8 2021-02-10T04:32:01.455 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 150 (ObjNum 346 snap 88 seq_num 346) dirty exists 2021-02-10T04:32:01.455 INFO:tasks.rados.rados.0.smithi085.stdout:1071: finishing copy_from racing read to smithi08575397-8 2021-02-10T04:32:01.455 INFO:tasks.rados.rados.0.smithi085.stdout:1074: expect (ObjNum 373 snap 96 seq_num 373) 2021-02-10T04:32:01.468 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:01 smithi085 conmon[29319]: cluster 2021-02-10T04:32: 2021-02-10T04:32:01.468 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:01 smithi085 conmon[29319]: 00.209266+0000 mon.a (mon.0) 1050 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2021-02-10T04:32:01.469 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:01 smithi085 conmon[29319]: audit 2021-02-10T04:32: 2021-02-10T04:32:01.469 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:01 smithi085 conmon[29319]: 00.488979+0000 mon.c (mon.1) 49 : audit [INF] from='client.? 172.21.15.85:0/1352123141' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:32:01.469 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:01 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:32:01.469 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:01 smithi085 conmon[29319]: :32:00.489826+0000 mon.a (mon.0 2021-02-10T04:32:01.469 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:01 smithi085 conmon[29319]: ) 1051 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:32:01.469 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:01 smithi085 conmon[35961]: cluster 2021-02-10T04:32:00.209266+0000 mon.a ( 2021-02-10T04:32:01.469 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:01 smithi085 conmon[35961]: mon.0) 1050 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2021-02-10T04:32:01.469 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:01 smithi085 conmon[35961]: audit 2021-02-10T04:32:00.488979+0000 mon.c (mon.1) 49 : 2021-02-10T04:32:01.469 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:01 smithi085 conmon[35961]: audit [INF] from='client.? 172.21.15.85:0/1352123141' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:32:01.469 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:01 smithi085 conmon[35961]: audit 2021-02-10T04:32:01.469 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:01 smithi085 conmon[35961]: 2021-02-10T04:32:00.489826+0000 mon.a (mon 2021-02-10T04:32:01.470 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:01 smithi085 conmon[35961]: .0) 1051 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:32:01.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:01 smithi164 conmon[29288]: cluster 2021-02-10T04:32:00.209266+0000 mon.a (mon.0) 1050 : cluster 2021-02-10T04:32:01.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:01 smithi164 conmon[29288]: [DBG] osdmap e303: 8 total, 8 up, 8 in 2021-02-10T04:32:01.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:01 smithi164 conmon[29288]: audit 2021-02-10T04:32:00.488979+0000 mon.c (mon.1) 49 : audit [INF] from='client.? 172.21.15.85:0/1352123141' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:32:01.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:01 smithi164 conmon[29288]: audit 2021-02-10T04:32 2021-02-10T04:32:01.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:01 smithi164 conmon[29288]: :00.489826+0000 mon.a (mon.0) 1051 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:32:01.646 INFO:tasks.rados.rados.0.smithi085.stdout:1071: done (15 left) 2021-02-10T04:32:01.646 INFO:tasks.rados.rados.0.smithi085.stdout:1072: done (14 left) 2021-02-10T04:32:01.646 INFO:tasks.rados.rados.0.smithi085.stdout:1074: done (13 left) 2021-02-10T04:32:01.646 INFO:tasks.rados.rados.0.smithi085.stdout:1075: write (excl) oid 6 current snap is 103 2021-02-10T04:32:01.646 INFO:tasks.rados.rados.0.smithi085.stdout:1075: seq_num 402 ranges {756414=700454,2041173=715241} 2021-02-10T04:32:01.655 INFO:tasks.rados.rados.0.smithi085.stdout:1075: writing smithi08575397-6 from 756414 to 1456868 tid 1 2021-02-10T04:32:01.663 INFO:tasks.rados.rados.0.smithi085.stdout:1075: writing smithi08575397-6 from 2041173 to 2756414 tid 2 2021-02-10T04:32:01.664 INFO:tasks.rados.rados.0.smithi085.stdout:1076: write oid 22 current snap is 103 2021-02-10T04:32:01.664 INFO:tasks.rados.rados.0.smithi085.stdout:1075: finishing write tid 1 to smithi08575397-6 2021-02-10T04:32:01.664 INFO:tasks.rados.rados.0.smithi085.stdout:1076: seq_num 403 ranges {773221=492056,1816008=503126} 2021-02-10T04:32:01.672 INFO:tasks.rados.rados.0.smithi085.stdout:1076: writing smithi08575397-22 from 773221 to 1265277 tid 1 2021-02-10T04:32:01.679 INFO:tasks.rados.rados.0.smithi085.stdout:1076: writing smithi08575397-22 from 1816008 to 2319134 tid 2 2021-02-10T04:32:01.679 INFO:tasks.rados.rados.0.smithi085.stdout:1077: read oid 26 snap -1 2021-02-10T04:32:01.679 INFO:tasks.rados.rados.0.smithi085.stdout:1077: expect (ObjNum 343 snap 88 seq_num 343) 2021-02-10T04:32:01.679 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:01.679 INFO:tasks.rados.rados.0.smithi085.stdout:1075: finishing write tid 2 to smithi08575397-6 2021-02-10T04:32:01.680 INFO:tasks.rados.rados.0.smithi085.stdout:1075: finishing write tid 3 to smithi08575397-6 2021-02-10T04:32:01.680 INFO:tasks.rados.rados.0.smithi085.stdout:1075: finishing write tid 4 to smithi08575397-6 2021-02-10T04:32:01.680 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 364 (ObjNum 402 snap 103 seq_num 402) dirty exists 2021-02-10T04:32:01.680 INFO:tasks.rados.rados.0.smithi085.stdout:1075: left oid 6 (ObjNum 402 snap 103 seq_num 402) 2021-02-10T04:32:01.680 INFO:tasks.rados.rados.0.smithi085.stdout:1076: finishing write tid 1 to smithi08575397-22 2021-02-10T04:32:01.680 INFO:tasks.rados.rados.0.smithi085.stdout:1075: done (15 left) 2021-02-10T04:32:01.680 INFO:tasks.rados.rados.0.smithi085.stdout:1078: snap_create 2021-02-10T04:32:01.683 INFO:tasks.rados.rados.0.smithi085.stdout:1076: finishing write tid 2 to smithi08575397-22 2021-02-10T04:32:01.683 INFO:tasks.rados.rados.0.smithi085.stdout:1077: expect (ObjNum 343 snap 88 seq_num 343) 2021-02-10T04:32:01.938 INFO:tasks.rados.rados.0.smithi085.stdout:1076: finishing write tid 3 to smithi08575397-22 2021-02-10T04:32:01.939 INFO:tasks.rados.rados.0.smithi085.stdout:1076: finishing write tid 4 to smithi08575397-22 2021-02-10T04:32:01.939 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 458 (ObjNum 403 snap 103 seq_num 403) dirty exists 2021-02-10T04:32:01.939 INFO:tasks.rados.rados.0.smithi085.stdout:1076: left oid 22 (ObjNum 403 snap 103 seq_num 403) 2021-02-10T04:32:02.226 INFO:tasks.rados.rados.0.smithi085.stdout:1076: done (15 left) 2021-02-10T04:32:02.226 INFO:tasks.rados.rados.0.smithi085.stdout:1077: done (14 left) 2021-02-10T04:32:02.226 INFO:tasks.rados.rados.0.smithi085.stdout:1078: done (13 left) 2021-02-10T04:32:02.226 INFO:tasks.rados.rados.0.smithi085.stdout:1079: snap_remove snap 101 2021-02-10T04:32:02.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:02 smithi164 conmon[29288]: cluster 2021-02-10T04:32:00.883622+0000 osd.5 (osd 2021-02-10T04:32:02.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:02 smithi164 conmon[29288]: .5) 55 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:32:02.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:02 smithi164 conmon[29288]: cluster 2021-02-10T04:32 2021-02-10T04:32:02.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:02 smithi164 conmon[29288]: :00.883862+0000 osd.5 (osd.5) 56 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:32:02.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:02 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:32:02.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:02 smithi164 conmon[29288]: 10T04:32:01.204995+0000 mon.a (mon.0) 1052 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:32:02.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:02 smithi164 conmon[29288]: 2021-02-10T04:32:02.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:02 smithi164 conmon[29288]: audit 2021-02-10T04:32: 2021-02-10T04:32:02.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:02 smithi164 conmon[29288]: 01.213636+0000 mon.a (mon.0) 1053 2021-02-10T04:32:02.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:02 smithi164 conmon[29288]: : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:32:02.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:02 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:02.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:02 smithi164 conmon[29288]: 02-10T04:32:01.213686+0000 2021-02-10T04:32:02.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:02 smithi164 conmon[29288]: mon.a (mon.0) 1054 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2021-02-10T04:32:02.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:02 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:02.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:02 smithi164 conmon[29288]: T04:32:02. 2021-02-10T04:32:02.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:02 smithi164 conmon[29288]: 048381+0000 mgr.y (mgr.14140) 465 : cluster [DBG] 2021-02-10T04:32:02.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:02 smithi164 conmon[29288]: pgmap v606: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim, 163 active+clean; 120 MiB data, 463 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 4.2 MiB/s wr, 27 op/s 2021-02-10T04:32:02.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:02 smithi085 conmon[29319]: cluster 2021-02-10T04:32:00.883622+0000 osd.5 (osd.5) 55 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:32:02.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:02 smithi085 conmon[29319]: cluster 2021-02-10T04:32:00.883862+0000 osd.5 (osd.5) 56 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:32:02.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:02 smithi085 conmon[29319]: cluster 2021-02-10T04:32:01.204995+0000 mon.a (mon.0) 1052 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:32:02.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:02 smithi085 conmon[29319]: audit 2021-02-10T04:32:01.213636+0000 mon.a (mon.0) 1053 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:32:02.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:02 smithi085 conmon[29319]: cluster 2021-02-10T04:32:01.213686+0000 mon.a (mon.0) 1054 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2021-02-10T04:32:02.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:02 smithi085 conmon[29319]: cluster 2021-02-10T04:32:02.048381+0000 mgr.y (mgr.14140) 465 : cluster [DBG] pgmap v606: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim, 163 active+clean; 120 MiB data, 463 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 4.2 MiB/s wr, 27 op/s 2021-02-10T04:32:02.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:02 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:32:02.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:02 smithi085 conmon[35961]: 32:00.883622+0000 osd.5 (osd.5) 55 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:32:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:02 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:02 smithi085 conmon[35961]: 10T04:32:00.883862+0000 osd.5 (osd.5) 56 : 2021-02-10T04:32:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:02 smithi085 conmon[35961]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:32:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:02 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:32:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:02 smithi085 conmon[35961]: 32:01.204995+0000 mon.a (mon.0) 1052 : 2021-02-10T04:32:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:02 smithi085 conmon[35961]: cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:32:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:02 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:32:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:02 smithi085 conmon[35961]: -10T04:32:01.213636+0000 mon.a ( 2021-02-10T04:32:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:02 smithi085 conmon[35961]: mon.0) 1053 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:32:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:02 smithi085 conmon[35961]: 2021-02-10T04:32:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:02 smithi085 conmon[35961]: cluster 2021-02-10T04:32:01 2021-02-10T04:32:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:02 smithi085 conmon[35961]: .213686+0000 mon.a (mon.0) 1054 : cluster 2021-02-10T04:32:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:02 smithi085 conmon[35961]: [DBG] osdmap e304: 8 total, 8 up, 8 in 2021-02-10T04:32:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:02 smithi085 conmon[35961]: cluster 2021-02-10T04:32:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:02 smithi085 conmon[35961]: 2021-02-10T04:32:02.048381+0000 mgr.y 2021-02-10T04:32:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:02 smithi085 conmon[35961]: (mgr.14140) 465 : cluster [DBG] pgmap v606: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim, 163 active+clean; 120 MiB data, 463 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 4.2 MiB/s wr, 27 op/s 2021-02-10T04:32:03.095 INFO:tasks.rados.rados.0.smithi085.stdout:1079: done (13 left) 2021-02-10T04:32:03.095 INFO:tasks.rados.rados.0.smithi085.stdout:1080: snap_create 2021-02-10T04:32:03.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:03 smithi164 conmon[29288]: cluster 2021-02-10T04:32:02.223402+0000 2021-02-10T04:32:03.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:03 smithi164 conmon[29288]: mon.a (mon.0) 1055 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2021-02-10T04:32:03.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:03 smithi164 conmon[29288]: cluster 2021-02-10T04:32:03.091609+0000 mon.a (mon.0) 1056 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2021-02-10T04:32:03.691 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:03 smithi085 conmon[52836]: debug 2021-02-10T04:32:03.225+0000 7fba74fb8700 -1 osd.3 pg_epoch: 306 pg[7.a( v 304'322 (0'0,304'322] local-lis/les=93/94 n=5 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [3,2,5] r=0 lpr=93 crt=304'322 lcod 304'320 mlcod 304'320 active+clean+snaptrim trimq=[cb~1] ps=55] removing snap head 2021-02-10T04:32:03.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:03 smithi085 conmon[35961]: cluster 2021-02-10T04:32:02.223402 2021-02-10T04:32:03.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:03 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1055 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2021-02-10T04:32:03.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:03 smithi085 conmon[35961]: cluster 2021-02-10T04:32:03.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:03 smithi085 conmon[35961]: 2021-02-10T04:32:03.091609+0000 mon.a (mon.0 2021-02-10T04:32:03.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:03 smithi085 conmon[35961]: ) 1056 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2021-02-10T04:32:03.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:03 smithi085 conmon[29319]: cluster 2021-02-10T04:32:02.223402+0000 mon.a (mon. 2021-02-10T04:32:03.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:03 smithi085 conmon[29319]: 0) 1055 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2021-02-10T04:32:03.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:03 smithi085 conmon[29319]: cluster 2021-02-10T04:32:03.091609+0000 mon.a (mon.0) 1056 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2021-02-10T04:32:04.096 INFO:tasks.rados.rados.0.smithi085.stdout:1080: done (13 left) 2021-02-10T04:32:04.096 INFO:tasks.rados.rados.0.smithi085.stdout:1081: rollback oid 37 current snap is 105 2021-02-10T04:32:04.096 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 37 to 104 2021-02-10T04:32:04.096 INFO:tasks.rados.rados.0.smithi085.stdout:1082: read oid 46 snap -1 2021-02-10T04:32:04.096 INFO:tasks.rados.rados.0.smithi085.stdout:1082: expect (ObjNum 382 snap 100 seq_num 382) 2021-02-10T04:32:04.096 INFO:tasks.rados.rados.0.smithi085.stdout:1083: write oid 3 current snap is 105 2021-02-10T04:32:04.096 INFO:tasks.rados.rados.0.smithi085.stdout:1083: seq_num 404 ranges {790028=567305,1874490=763137} 2021-02-10T04:32:04.104 INFO:tasks.rados.rados.0.smithi085.stdout:1083: writing smithi08575397-3 from 790028 to 1357333 tid 1 2021-02-10T04:32:04.112 INFO:tasks.rados.rados.0.smithi085.stdout:1083: writing smithi08575397-3 from 1874490 to 2637627 tid 2 2021-02-10T04:32:04.112 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:04.113 INFO:tasks.rados.rados.0.smithi085.stdout:1082: expect (ObjNum 382 snap 100 seq_num 382) 2021-02-10T04:32:04.219 INFO:tasks.rados.rados.0.smithi085.stdout:1083: finishing write tid 1 to smithi08575397-3 2021-02-10T04:32:04.219 INFO:tasks.rados.rados.0.smithi085.stdout:1081: finishing rollback tid 0 to smithi08575397-37 2021-02-10T04:32:04.220 INFO:tasks.rados.rados.0.smithi085.stdout:1081: finishing rollback tid 1 to smithi08575397-37 2021-02-10T04:32:04.220 INFO:tasks.rados.rados.0.smithi085.stdout:1081: finishing rollback tid 2 to smithi08575397-37 2021-02-10T04:32:04.220 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 574 (ObjNum 343 snap 88 seq_num 343) dirty exists 2021-02-10T04:32:04.220 INFO:tasks.rados.rados.0.smithi085.stdout:1081: done (15 left) 2021-02-10T04:32:04.220 INFO:tasks.rados.rados.0.smithi085.stdout:1082: done (14 left) 2021-02-10T04:32:04.220 INFO:tasks.rados.rados.0.smithi085.stdout:1084: write (excl) oid 16 current snap is 105 2021-02-10T04:32:04.220 INFO:tasks.rados.rados.0.smithi085.stdout:1083: finishing write tid 2 to smithi08575397-3 2021-02-10T04:32:04.220 INFO:tasks.rados.rados.0.smithi085.stdout:1083: finishing write tid 3 to smithi08575397-3 2021-02-10T04:32:04.220 INFO:tasks.rados.rados.0.smithi085.stdout:1083: finishing write tid 4 to smithi08575397-3 2021-02-10T04:32:04.220 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 480 (ObjNum 404 snap 105 seq_num 404) dirty exists 2021-02-10T04:32:04.220 INFO:tasks.rados.rados.0.smithi085.stdout:1083: left oid 3 (ObjNum 404 snap 105 seq_num 404) 2021-02-10T04:32:04.220 INFO:tasks.rados.rados.0.smithi085.stdout:1084: seq_num 405 ranges {406835=642554,1532972=506795,2772312=34523} 2021-02-10T04:32:04.225 INFO:tasks.rados.rados.0.smithi085.stdout:1084: writing smithi08575397-16 from 406835 to 1049389 tid 1 2021-02-10T04:32:04.232 INFO:tasks.rados.rados.0.smithi085.stdout:1084: writing smithi08575397-16 from 1532972 to 2039767 tid 2 2021-02-10T04:32:04.237 INFO:tasks.rados.rados.0.smithi085.stdout:1084: writing smithi08575397-16 from 2772312 to 2806835 tid 3 2021-02-10T04:32:04.237 INFO:tasks.rados.rados.0.smithi085.stdout:1083: done (14 left) 2021-02-10T04:32:04.237 INFO:tasks.rados.rados.0.smithi085.stdout:1084: finishing write tid 1 to smithi08575397-16 2021-02-10T04:32:04.237 INFO:tasks.rados.rados.0.smithi085.stdout:1084: finishing write tid 2 to smithi08575397-16 2021-02-10T04:32:04.237 INFO:tasks.rados.rados.0.smithi085.stdout:1085: read oid 21 snap -1 2021-02-10T04:32:04.237 INFO:tasks.rados.rados.0.smithi085.stdout:1085: expect (ObjNum 399 snap 103 seq_num 399) 2021-02-10T04:32:04.237 INFO:tasks.rados.rados.0.smithi085.stdout:1086: write oid 1 current snap is 105 2021-02-10T04:32:04.237 INFO:tasks.rados.rados.0.smithi085.stdout:1086: seq_num 406 ranges {423642=717803,1875101=766806} 2021-02-10T04:32:04.244 INFO:tasks.rados.rados.0.smithi085.stdout:1086: writing smithi08575397-1 from 423642 to 1141445 tid 1 2021-02-10T04:32:04.254 INFO:tasks.rados.rados.0.smithi085.stdout:1086: writing smithi08575397-1 from 1875101 to 2641907 tid 2 2021-02-10T04:32:04.255 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:04.255 INFO:tasks.rados.rados.0.smithi085.stdout:1084: finishing write tid 3 to smithi08575397-16 2021-02-10T04:32:04.255 INFO:tasks.rados.rados.0.smithi085.stdout:1084: finishing write tid 4 to smithi08575397-16 2021-02-10T04:32:04.255 INFO:tasks.rados.rados.0.smithi085.stdout:1084: finishing write tid 5 to smithi08575397-16 2021-02-10T04:32:04.255 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 579 (ObjNum 405 snap 105 seq_num 405) dirty exists 2021-02-10T04:32:04.256 INFO:tasks.rados.rados.0.smithi085.stdout:1084: left oid 16 (ObjNum 405 snap 105 seq_num 405) 2021-02-10T04:32:04.256 INFO:tasks.rados.rados.0.smithi085.stdout:1085: expect (ObjNum 399 snap 103 seq_num 399) 2021-02-10T04:32:04.381 INFO:tasks.rados.rados.0.smithi085.stdout:1086: finishing write tid 1 to smithi08575397-1 2021-02-10T04:32:04.381 INFO:tasks.rados.rados.0.smithi085.stdout:1086: finishing write tid 2 to smithi08575397-1 2021-02-10T04:32:04.381 INFO:tasks.rados.rados.0.smithi085.stdout:1086: finishing write tid 3 to smithi08575397-1 2021-02-10T04:32:04.381 INFO:tasks.rados.rados.0.smithi085.stdout:1086: finishing write tid 4 to smithi08575397-1 2021-02-10T04:32:04.381 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 274 (ObjNum 406 snap 105 seq_num 406) dirty exists 2021-02-10T04:32:04.382 INFO:tasks.rados.rados.0.smithi085.stdout:1086: left oid 1 (ObjNum 406 snap 105 seq_num 406) 2021-02-10T04:32:04.382 INFO:tasks.rados.rados.0.smithi085.stdout:1084: done (15 left) 2021-02-10T04:32:04.382 INFO:tasks.rados.rados.0.smithi085.stdout:1085: done (14 left) 2021-02-10T04:32:04.382 INFO:tasks.rados.rados.0.smithi085.stdout:1086: done (13 left) 2021-02-10T04:32:04.382 INFO:tasks.rados.rados.0.smithi085.stdout:1087: snap_remove snap 103 2021-02-10T04:32:04.784 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:04 smithi085 conmon[52836]: debug 2021-02-10T04:32:04.515+0000 7fba6ffae700 -1 osd.3 pg_epoch: 307 pg[4.14( v 63'26 (0'0,63'26] local-lis/les=71/72 n=11 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,2,7] r=0 lpr=71 crt=63'26 lcod 63'25 mlcod 0'0 active+clean] scrubber pg(4.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55feb3a02160) 2021-02-10T04:32:04.885 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:32:04.916 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:05.092 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:05 smithi085 conmon[35961]: cluster 2021-02-10T04:32:05.092 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:05 smithi085 conmon[35961]: 2021-02-10T04:32:05.092 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:05 smithi085 conmon[35961]: 2021 2021-02-10T04:32:05.092 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:05 smithi085 conmon[35961]: - 2021-02-10T04:32:05.092 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:05 smithi085 conmon[35961]: 02 2021-02-10T04:32:05.092 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:05 smithi085 conmon[35961]: - 2021-02-10T04:32:05.092 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:05 smithi085 conmon[35961]: 10 2021-02-10T04:32:05.092 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:05 smithi085 conmon[35961]: T 2021-02-10T04:32:05.092 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:05 smithi085 conmon[35961]: 04 2021-02-10T04:32:05.092 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:05 smithi085 conmon[35961]: : 2021-02-10T04:32:05.093 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:05 smithi085 conmon[35961]: 32 2021-02-10T04:32:05.093 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:05 smithi085 conmon[35961]: : 2021-02-10T04:32:05.093 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:05 smithi085 conmon[35961]: 04 2021-02-10T04:32:05.093 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:05 smithi085 conmon[35961]: .048933+0000 mgr.y (mgr.14140) 466 2021-02-10T04:32:05.093 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:05 smithi085 conmon[35961]: : cluster [DBG] pgmap v609: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 164 active+clean; 120 MiB data, 466 MiB used, 715 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.4 MiB/s wr, 13 op/s 2021-02-10T04:32:05.093 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:05 smithi085 conmon[35961]: cluster 2021-02-10T04:32:04.096171+0000 mon.a (mon.0 2021-02-10T04:32:05.093 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:05 smithi085 conmon[35961]: ) 1057 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2021-02-10T04:32:05.231 INFO:tasks.rados.rados.0.smithi085.stdout:1087: done (13 left) 2021-02-10T04:32:05.231 INFO:tasks.rados.rados.0.smithi085.stdout:1088: snap_remove snap 104 2021-02-10T04:32:05.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:05 smithi164 conmon[29288]: cluster 2021-02-10T04:32:04.048933+0000 2021-02-10T04:32:05.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:05 smithi164 conmon[29288]: mgr.y (mgr.14140) 466 : cluster [DBG] pgmap v609: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 164 active+clean; 120 MiB data, 466 MiB used, 715 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.4 MiB/s wr, 13 op/s 2021-02-10T04:32:05.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:05 smithi164 conmon[29288]: cluster 2021-02-10T04:32:04.096171+0000 mon.a (mon.0) 1057 2021-02-10T04:32:05.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:05 smithi164 conmon[29288]: : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2021-02-10T04:32:05.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:05 smithi085 conmon[29319]: cluster 2021-02-10T04:32:04.048933+0000 mgr.y 2021-02-10T04:32:05.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:05 smithi085 conmon[29319]: (mgr.14140) 466 : cluster [DBG] pgmap v609: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 164 active+clean; 120 MiB data, 466 MiB used, 715 GiB / 715 GiB avail; 3.3 MiB/s rd, 1.4 MiB/s wr, 13 op/s 2021-02-10T04:32:05.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:05 smithi085 conmon[29319]: cluster 2021-02-10T04:32:04.096171+0000 mon.a (mon.0) 1057 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2021-02-10T04:32:06.240 INFO:tasks.rados.rados.0.smithi085.stdout:1088: done (13 left) 2021-02-10T04:32:06.240 INFO:tasks.rados.rados.0.smithi085.stdout:1089: copy_from oid 2 from oid 43 current snap is 105 2021-02-10T04:32:06.240 INFO:tasks.rados.rados.0.smithi085.stdout:1090: rollback oid 50 current snap is 105 2021-02-10T04:32:06.240 INFO:tasks.rados.rados.0.smithi085.stdout:1090: done (14 left) 2021-02-10T04:32:06.240 INFO:tasks.rados.rados.0.smithi085.stdout:1091: write (excl) oid 5 current snap is 105 2021-02-10T04:32:06.240 INFO:tasks.rados.rados.0.smithi085.stdout:1091: seq_num 407 ranges {440449=793052,1933583=510464} 2021-02-10T04:32:06.244 INFO:tasks.rados.rados.0.smithi085.stdout:1091: writing smithi08575397-5 from 440449 to 1233501 tid 1 2021-02-10T04:32:06.252 INFO:tasks.rados.rados.0.smithi085.stdout:1091: writing smithi08575397-5 from 1933583 to 2444047 tid 2 2021-02-10T04:32:06.252 INFO:tasks.rados.rados.0.smithi085.stdout:1092: copy_from oid 21 from oid 46 current snap is 105 2021-02-10T04:32:06.252 INFO:tasks.rados.rados.0.smithi085.stdout:1091: finishing write tid 1 to smithi08575397-5 2021-02-10T04:32:06.252 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:06.252 INFO:tasks.rados.rados.0.smithi085.stdout:1089: finishing copy_from to smithi08575397-2 2021-02-10T04:32:06.253 INFO:tasks.rados.rados.0.smithi085.stdout:1089: got expected ENOENT (src dne) 2021-02-10T04:32:06.253 INFO:tasks.rados.rados.0.smithi085.stdout:1089: finishing copy_from racing read to smithi08575397-2 2021-02-10T04:32:06.253 INFO:tasks.rados.rados.0.smithi085.stdout:1089: done (15 left) 2021-02-10T04:32:06.253 INFO:tasks.rados.rados.0.smithi085.stdout:1093: write (excl) oid 23 current snap is 105 2021-02-10T04:32:06.253 INFO:tasks.rados.rados.0.smithi085.stdout:1093: seq_num 408 ranges {457256=468301,1592065=770475,2854581=2675} 2021-02-10T04:32:06.258 INFO:tasks.rados.rados.0.smithi085.stdout:1093: writing smithi08575397-23 from 457256 to 925557 tid 1 2021-02-10T04:32:06.268 INFO:tasks.rados.rados.0.smithi085.stdout:1093: writing smithi08575397-23 from 1592065 to 2362540 tid 2 2021-02-10T04:32:06.271 INFO:tasks.rados.rados.0.smithi085.stdout:1093: writing smithi08575397-23 from 2854581 to 2857256 tid 3 2021-02-10T04:32:06.272 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:06.272 INFO:tasks.rados.rados.0.smithi085.stdout:1091: finishing write tid 2 to smithi08575397-5 2021-02-10T04:32:06.272 INFO:tasks.rados.rados.0.smithi085.stdout:1091: finishing write tid 3 to smithi08575397-5 2021-02-10T04:32:06.272 INFO:tasks.rados.rados.0.smithi085.stdout:1091: finishing write tid 4 to smithi08575397-5 2021-02-10T04:32:06.272 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 391 (ObjNum 407 snap 105 seq_num 407) dirty exists 2021-02-10T04:32:06.272 INFO:tasks.rados.rados.0.smithi085.stdout:1091: left oid 5 (ObjNum 407 snap 105 seq_num 407) 2021-02-10T04:32:06.272 INFO:tasks.rados.rados.0.smithi085.stdout:1093: finishing write tid 1 to smithi08575397-23 2021-02-10T04:32:06.272 INFO:tasks.rados.rados.0.smithi085.stdout:1091: done (15 left) 2021-02-10T04:32:06.273 INFO:tasks.rados.rados.0.smithi085.stdout:1094: write (excl) oid 3 current snap is 105 2021-02-10T04:32:06.273 INFO:tasks.rados.rados.0.smithi085.stdout:1094: seq_num 409 ranges {474063=543550,1650547=514133,2765651=108412} 2021-02-10T04:32:06.278 INFO:tasks.rados.rados.0.smithi085.stdout:1094: writing smithi08575397-3 from 474063 to 1017613 tid 1 2021-02-10T04:32:06.285 INFO:tasks.rados.rados.0.smithi085.stdout:1094: writing smithi08575397-3 from 1650547 to 2164680 tid 2 2021-02-10T04:32:06.289 INFO:tasks.rados.rados.0.smithi085.stdout:1094: writing smithi08575397-3 from 2765651 to 2874063 tid 3 2021-02-10T04:32:06.289 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:06.289 INFO:tasks.rados.rados.0.smithi085.stdout:1093: finishing write tid 2 to smithi08575397-23 2021-02-10T04:32:06.289 INFO:tasks.rados.rados.0.smithi085.stdout:1093: finishing write tid 3 to smithi08575397-23 2021-02-10T04:32:06.290 INFO:tasks.rados.rados.0.smithi085.stdout:1093: finishing write tid 4 to smithi08575397-23 2021-02-10T04:32:06.290 INFO:tasks.rados.rados.0.smithi085.stdout:1093: finishing write tid 5 to smithi08575397-23 2021-02-10T04:32:06.290 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 176 (ObjNum 408 snap 105 seq_num 408) dirty exists 2021-02-10T04:32:06.290 INFO:tasks.rados.rados.0.smithi085.stdout:1093: left oid 23 (ObjNum 408 snap 105 seq_num 408) 2021-02-10T04:32:06.290 INFO:tasks.rados.rados.0.smithi085.stdout:1094: finishing write tid 1 to smithi08575397-3 2021-02-10T04:32:06.290 INFO:tasks.rados.rados.0.smithi085.stdout:1094: finishing write tid 2 to smithi08575397-3 2021-02-10T04:32:06.290 INFO:tasks.rados.rados.0.smithi085.stdout:1093: done (15 left) 2021-02-10T04:32:06.290 INFO:tasks.rados.rados.0.smithi085.stdout:1095: snap_create 2021-02-10T04:32:06.292 INFO:tasks.rados.rados.0.smithi085.stdout:1094: finishing write tid 3 to smithi08575397-3 2021-02-10T04:32:06.292 INFO:tasks.rados.rados.0.smithi085.stdout:1094: finishing write tid 4 to smithi08575397-3 2021-02-10T04:32:06.293 INFO:tasks.rados.rados.0.smithi085.stdout:1094: finishing write tid 5 to smithi08575397-3 2021-02-10T04:32:06.293 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 484 (ObjNum 409 snap 105 seq_num 409) dirty exists 2021-02-10T04:32:06.293 INFO:tasks.rados.rados.0.smithi085.stdout:1094: left oid 3 (ObjNum 409 snap 105 seq_num 409) 2021-02-10T04:32:06.463 INFO:tasks.rados.rados.0.smithi085.stdout:1092: finishing copy_from to smithi08575397-21 2021-02-10T04:32:06.463 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 353 (ObjNum 382 snap 100 seq_num 382) dirty exists 2021-02-10T04:32:06.464 INFO:tasks.rados.rados.0.smithi085.stdout:1092: finishing copy_from racing read to smithi08575397-21 2021-02-10T04:32:06.587 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:06 smithi085 conmon[35961]: cluster 2021-02-10T04:32:05.230720 2021-02-10T04:32:06.587 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:06 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1058 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2021-02-10T04:32:06.587 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:06 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:06.587 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:06 smithi085 conmon[35961]: 10T04:32:06.049472+0000 mgr.y (mgr.14140) 467 : cluster 2021-02-10T04:32:06.587 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:06 smithi085 conmon[35961]: [DBG] pgmap v612: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim, 163 active+clean; 121 MiB data, 470 MiB used, 715 GiB / 715 GiB avail; 9.3 MiB/s rd, 3.0 MiB/s wr, 19 op/s 2021-02-10T04:32:06.587 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:06 smithi085 conmon[52836]: debug 2021-02-10T04:32:06.585+0000 7fba6ffae700 -1 osd.3 pg_epoch: 309 pg[4.14( v 63'26 (0'0,63'26] local-lis/les=71/72 n=11 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,2,7] r=0 lpr=71 crt=63'26 lcod 63'25 mlcod 0'0 active+clean] scrubber pg(4.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55fec0820580) 2021-02-10T04:32:06.588 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:06 smithi085 conmon[29319]: cluster 2021-02-10T04:32:06.588 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:06 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:32:06.588 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:06 smithi085 conmon[29319]: :32:05.230720+0000 mon.a (mon.0) 1058 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2021-02-10T04:32:06.588 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:06 smithi085 conmon[29319]: cluster 2021-02-10T04:32:06.588 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:06 smithi085 conmon[29319]: 2021-02-10T04:32:06.588 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:06 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:32:06.588 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:06 smithi085 conmon[29319]: :32:06.049472+0000 mgr.y (mgr.14140) 467 : cluster [DBG] pgmap v612: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim, 163 active+clean; 121 MiB data, 470 MiB used, 715 GiB / 715 GiB avail; 9.3 MiB/s rd, 3.0 MiB/s wr, 19 op/s 2021-02-10T04:32:06.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:06 smithi164 conmon[29288]: cluster 2021-02-10T04:32:05.230720+0000 mon.a (mon.0) 1058 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2021-02-10T04:32:06.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:06 smithi164 conmon[29288]: cluster 2021-02-10T04:32:06.049472+0000 mgr.y (mgr.14140) 467 : cluster [DBG] pgmap v612: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim, 163 active+clean; 121 MiB data, 470 MiB used, 715 GiB / 715 GiB avail; 9.3 MiB/s rd, 3.0 MiB/s wr, 19 op/s 2021-02-10T04:32:07.243 INFO:tasks.rados.rados.0.smithi085.stdout:1092: done (15 left) 2021-02-10T04:32:08.437 INFO:tasks.rados.rados.0.smithi085.stdout:1094: done (14 left) 2021-02-10T04:32:08.438 INFO:tasks.rados.rados.0.smithi085.stdout:1095: done (13 left) 2021-02-10T04:32:08.438 INFO:tasks.rados.rados.0.smithi085.stdout:1096: rollback oid 40 current snap is 106 2021-02-10T04:32:08.438 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 40 to 105 2021-02-10T04:32:08.438 INFO:tasks.rados.rados.0.smithi085.stdout:1097: read oid 28 snap 105 2021-02-10T04:32:08.438 INFO:tasks.rados.rados.0.smithi085.stdout:1097: expect deleted 2021-02-10T04:32:08.438 INFO:tasks.rados.rados.0.smithi085.stdout:1098: read oid 30 snap -1 2021-02-10T04:32:08.438 INFO:tasks.rados.rados.0.smithi085.stdout:1098: expect (ObjNum 372 snap 96 seq_num 372) 2021-02-10T04:32:08.438 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:08.442 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:07 smithi164 conmon[29288]: cluster 2021-02-10T04:32:05.644882+0000 osd.2 (osd.2) 2021-02-10T04:32:08.442 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:07 smithi164 conmon[29288]: 294 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:32:08.442 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:07 smithi164 conmon[29288]: cluster 2021-02-10T04:32:05.645057+0000 osd.2 (osd.2) 295 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:32:08.443 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:07 smithi164 conmon[29288]: cluster 2021-02-10T04:32:08.443 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:07 smithi164 conmon[29288]: 2021-02-10T04:32:06.236146+0000 mon.a (mon.0) 1059 : 2021-02-10T04:32:08.443 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:07 smithi164 conmon[29288]: cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2021-02-10T04:32:08.443 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:32:07 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:32:08.443 INFO:tasks.rados.rados.0.smithi085.stdout:1097: done (15 left) 2021-02-10T04:32:08.443 INFO:tasks.rados.rados.0.smithi085.stdout:1099: rollback oid 46 current snap is 106 2021-02-10T04:32:08.443 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 46 to 105 2021-02-10T04:32:08.443 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:08.443 INFO:tasks.rados.rados.0.smithi085.stdout:1096: finishing rollback tid 0 to smithi08575397-40 2021-02-10T04:32:08.443 INFO:tasks.rados.rados.0.smithi085.stdout:1096: finishing rollback tid 1 to smithi08575397-40 2021-02-10T04:32:08.443 INFO:tasks.rados.rados.0.smithi085.stdout:1096: finishing rollback tid 2 to smithi08575397-40 2021-02-10T04:32:08.444 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 590 (ObjNum 389 snap 100 seq_num 389) dirty exists 2021-02-10T04:32:08.444 INFO:tasks.rados.rados.0.smithi085.stdout:1096: done (15 left) 2021-02-10T04:32:08.444 INFO:tasks.rados.rados.0.smithi085.stdout:1100: snap_create 2021-02-10T04:32:08.444 INFO:tasks.rados.rados.0.smithi085.stdout:1098: expect (ObjNum 372 snap 96 seq_num 372) 2021-02-10T04:32:08.444 INFO:tasks.rados.rados.0.smithi085.stdout:1099: finishing rollback tid 0 to smithi08575397-46 2021-02-10T04:32:08.444 INFO:tasks.rados.rados.0.smithi085.stdout:1099: finishing rollback tid 1 to smithi08575397-46 2021-02-10T04:32:08.444 INFO:tasks.rados.rados.0.smithi085.stdout:1099: finishing rollback tid 2 to smithi08575397-46 2021-02-10T04:32:08.444 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 370 (ObjNum 382 snap 100 seq_num 382) dirty exists 2021-02-10T04:32:08.444 INFO:tasks.rados.rados.0.smithi085.stdout:1098: done (15 left) 2021-02-10T04:32:08.444 INFO:tasks.rados.rados.0.smithi085.stdout:1099: done (14 left) 2021-02-10T04:32:08.444 INFO:tasks.rados.rados.0.smithi085.stdout:1100: done (13 left) 2021-02-10T04:32:08.444 INFO:tasks.rados.rados.0.smithi085.stdout:1101: read oid 50 snap -1 2021-02-10T04:32:08.445 INFO:tasks.rados.rados.0.smithi085.stdout:1101: expect deleted 2021-02-10T04:32:08.445 INFO:tasks.rados.rados.0.smithi085.stdout:1102: delete oid 31 current snap is 107 2021-02-10T04:32:08.445 INFO:tasks.rados.rados.0.smithi085.stdout:1101: done (14 left) 2021-02-10T04:32:08.445 INFO:tasks.rados.rados.0.smithi085.stdout:1102: done (13 left) 2021-02-10T04:32:08.445 INFO:tasks.rados.rados.0.smithi085.stdout:1103: rollback oid 28 current snap is 107 2021-02-10T04:32:08.445 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 28 to 105 2021-02-10T04:32:08.446 INFO:tasks.rados.rados.0.smithi085.stdout:1104: delete oid 8 current snap is 107 2021-02-10T04:32:08.446 INFO:tasks.rados.rados.0.smithi085.stdout:1103: finishing rollback tid 1 to smithi08575397-28 2021-02-10T04:32:08.446 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 0 (ObjNum 1 snap 32576 seq_num 82372816) dirty dne 2021-02-10T04:32:08.446 INFO:tasks.rados.rados.0.smithi085.stdout:1103: done (14 left) 2021-02-10T04:32:08.446 INFO:tasks.rados.rados.0.smithi085.stdout:1104: done (13 left) 2021-02-10T04:32:08.446 INFO:tasks.rados.rados.0.smithi085.stdout:1105: write oid 22 current snap is 107 2021-02-10T04:32:08.446 INFO:tasks.rados.rados.0.smithi085.stdout:1105: seq_num 412 ranges {524484=485650,1425993=661460,2847920=76564} 2021-02-10T04:32:08.446 INFO:tasks.rados.rados.0.smithi085.stdout:1105: writing smithi08575397-22 from 524484 to 1010134 tid 1 2021-02-10T04:32:08.446 INFO:tasks.rados.rados.0.smithi085.stdout:1105: writing smithi08575397-22 from 1425993 to 2087453 tid 2 2021-02-10T04:32:08.446 INFO:tasks.rados.rados.0.smithi085.stdout:1105: writing smithi08575397-22 from 2847920 to 2924484 tid 3 2021-02-10T04:32:08.446 INFO:tasks.rados.rados.0.smithi085.stdout:1105: finishing write tid 1 to smithi08575397-22 2021-02-10T04:32:08.446 INFO:tasks.rados.rados.0.smithi085.stdout:1106: read oid 30 snap 105 2021-02-10T04:32:08.447 INFO:tasks.rados.rados.0.smithi085.stdout:1106: expect (ObjNum 372 snap 96 seq_num 372) 2021-02-10T04:32:08.447 INFO:tasks.rados.rados.0.smithi085.stdout:1107: copy_from oid 8 from oid 40 current snap is 107 2021-02-10T04:32:08.447 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:08.447 INFO:tasks.rados.rados.0.smithi085.stdout:1106: expect (ObjNum 372 snap 96 seq_num 372) 2021-02-10T04:32:08.447 INFO:tasks.rados.rados.0.smithi085.stdout:1105: finishing write tid 2 to smithi08575397-22 2021-02-10T04:32:08.447 INFO:tasks.rados.rados.0.smithi085.stdout:1105: finishing write tid 3 to smithi08575397-22 2021-02-10T04:32:08.447 INFO:tasks.rados.rados.0.smithi085.stdout:1105: finishing write tid 4 to smithi08575397-22 2021-02-10T04:32:08.447 INFO:tasks.rados.rados.0.smithi085.stdout:1106: done (15 left) 2021-02-10T04:32:08.447 INFO:tasks.rados.rados.0.smithi085.stdout:1108: rollback oid 46 current snap is 107 2021-02-10T04:32:08.447 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 46 to 105 2021-02-10T04:32:08.447 INFO:tasks.rados.rados.0.smithi085.stdout:1105: finishing write tid 5 to smithi08575397-22 2021-02-10T04:32:08.447 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 463 (ObjNum 412 snap 107 seq_num 412) dirty exists 2021-02-10T04:32:08.447 INFO:tasks.rados.rados.0.smithi085.stdout:1105: left oid 22 (ObjNum 412 snap 107 seq_num 412) 2021-02-10T04:32:08.448 INFO:tasks.rados.rados.0.smithi085.stdout:1107: finishing copy_from to smithi08575397-8 2021-02-10T04:32:08.448 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 590 (ObjNum 389 snap 100 seq_num 389) dirty exists 2021-02-10T04:32:08.448 INFO:tasks.rados.rados.0.smithi085.stdout:1107: finishing copy_from racing read to smithi08575397-8 2021-02-10T04:32:08.448 INFO:tasks.rados.rados.0.smithi085.stdout:1105: done (15 left) 2021-02-10T04:32:08.448 INFO:tasks.rados.rados.0.smithi085.stdout:1107: done (14 left) 2021-02-10T04:32:08.448 INFO:tasks.rados.rados.0.smithi085.stdout:1109: snap_create 2021-02-10T04:32:08.448 INFO:tasks.rados.rados.0.smithi085.stdout:1108: finishing rollback tid 0 to smithi08575397-46 2021-02-10T04:32:08.448 INFO:tasks.rados.rados.0.smithi085.stdout:1108: finishing rollback tid 1 to smithi08575397-46 2021-02-10T04:32:08.448 INFO:tasks.rados.rados.0.smithi085.stdout:1108: finishing rollback tid 2 to smithi08575397-46 2021-02-10T04:32:08.448 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 374 (ObjNum 382 snap 100 seq_num 382) dirty exists 2021-02-10T04:32:08.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:07 smithi085 conmon[35961]: cluster 2021-02-10T04:32:05.644882+0000 osd.2 (osd. 2021-02-10T04:32:08.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:07 smithi085 conmon[35961]: 2) 294 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:32:08.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:07 smithi085 conmon[35961]: cluster 2021-02-10T04:32:05.645057+0000 osd.2 (osd.2 2021-02-10T04:32:08.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:07 smithi085 conmon[35961]: ) 295 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:32:08.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:07 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:08.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:07 smithi085 conmon[35961]: 10T04:32:06.236146+0000 2021-02-10T04:32:08.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:07 smithi085 conmon[35961]: mon.a (mon.0) 1059 : cluster [DBG] 2021-02-10T04:32:08.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:07 smithi085 conmon[35961]: osdmap e309: 8 total, 8 up, 8 in 2021-02-10T04:32:08.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:07 smithi085 conmon[29319]: cluster 2021-02-10T04:32:05.644882+0000 osd.2 ( 2021-02-10T04:32:08.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:07 smithi085 conmon[29319]: osd.2) 294 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:32:08.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:07 smithi085 conmon[29319]: cluster 2021-02-10T04:32:05.645057+0000 osd.2 (osd.2) 295 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:32:08.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:07 smithi085 conmon[29319]: cluster 2021-02-10T04:32:06.236146+0000 mon.a (mon.0) 1059 : 2021-02-10T04:32:08.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:07 smithi085 conmon[29319]: cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2021-02-10T04:32:08.450 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:07 smithi085 conmon[45443]: debug 2021-02-10T04:32:07.310+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:08.450 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:32:08.450 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:07 smithi085 conmon[52836]: debug 2021-02-10T04:32:07.546+0000 7fba73fb6700 -1 osd.3 pg_epoch: 310 pg[4.14( v 63'26 (0'0,63'26] local-lis/les=71/72 n=11 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,2,7] r=0 lpr=71 crt=63'26 lcod 63'25 mlcod 0'0 active+clean] scrubber pg(4.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55fec073cc60) 2021-02-10T04:32:08.550 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:32:08.577 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:08 smithi164 conmon[29288]: cluster 2021-02-10T04:32:07.241980+0000 mon.a (mon.0) 1060 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2021-02-10T04:32:08.577 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:08 smithi164 conmon[29288]: 2021-02-10T04:32:08.577 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:08 smithi164 conmon[29288]: audit 2021-02-10T04:32:07.885045+0000 mgr.y (mgr.14140) 468 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:32:08.577 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:08 smithi164 conmon[29288]: cluster 2021-02-10T04:32:08.049981+0000 2021-02-10T04:32:08.577 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:08 smithi164 conmon[29288]: mgr.y (mgr.14140) 469 : cluster [DBG] pgmap v615: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim, 163 active+clean; 121 MiB data, 471 MiB used, 715 GiB / 715 GiB avail; 9.3 MiB/s rd, 3.0 MiB/s wr, 19 op/s 2021-02-10T04:32:08.577 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:08 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:08.578 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:08 smithi164 conmon[29288]: -02-10T04:32:08.091820+0000 mon.a (mon.0) 2021-02-10T04:32:08.578 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:08 smithi164 conmon[29288]: 1061 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2021-02-10T04:32:08.580 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:08.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[35961]: cluster 2021-02-10T04:32:08.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[35961]: 2021-02-10T04:32:08.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[35961]: 2021 2021-02-10T04:32:08.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[35961]: -02-10T04:32:07. 2021-02-10T04:32:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[35961]: 241980+0000 mon.a (mon.0) 1060 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2021-02-10T04:32:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[35961]: audit 2021 2021-02-10T04:32:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[35961]: -02-10T04:32:07.885045+0000 2021-02-10T04:32:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[35961]: mgr.y (mgr.14140) 468 : audit [DBG] 2021-02-10T04:32:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[35961]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:32:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:32:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[35961]: 32:08.049981+0000 mgr.y (mgr.14140 2021-02-10T04:32:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[35961]: ) 469 : cluster [DBG] pgmap v615: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim, 163 active+clean; 121 MiB data, 471 MiB used, 715 GiB / 715 GiB avail; 9.3 MiB/s rd, 3.0 MiB/s wr, 19 op/s 2021-02-10T04:32:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[35961]: 02-10T04:32:08.091820+0000 2021-02-10T04:32:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[35961]: mon.a (mon.0) 1061 : cluster [DBG] 2021-02-10T04:32:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[35961]: osdmap e311: 8 total, 8 up, 8 in 2021-02-10T04:32:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[29319]: cluster 2021-02-10T04:32:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[29319]: 2021-02-10T04:32:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[29319]: 2021 2021-02-10T04:32:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[29319]: - 2021-02-10T04:32:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[29319]: 02-10T04:32:07 2021-02-10T04:32:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[29319]: .241980+0000 mon.a (mon.0) 1060 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2021-02-10T04:32:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[29319]: audit 2021-02-10T04:32:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:32:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[29319]: 10T04:32:07.885045+0000 mgr.y (mgr 2021-02-10T04:32:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[29319]: .14140) 468 : audit [DBG] 2021-02-10T04:32:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[29319]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:32:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[29319]: cluster 2021-02-10T04:32:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[29319]: 2021 2021-02-10T04:32:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[29319]: -02-10T04:32:08.049981+0000 mgr.y (mgr.14140) 469 : cluster [DBG] pgmap v615: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim, 163 active+clean; 121 MiB data, 471 MiB used, 715 GiB / 715 GiB avail; 9.3 MiB/s rd, 3.0 MiB/s wr, 19 op/s 2021-02-10T04:32:08.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:08 smithi085 conmon[29319]: cluster 2021-02-10T04:32:08.091820+0000 mon.a (mon.0) 1061 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2021-02-10T04:32:08.894 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:32:08 smithi164 conmon[40892]: debug 2021-02-10T04:32:08.663+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:08.994 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:32:09.102 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:32:09.132 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:09.249 INFO:tasks.rados.rados.0.smithi085.stdout:1108: done (14 left) 2021-02-10T04:32:09.250 INFO:tasks.rados.rados.0.smithi085.stdout:1109: done (13 left) 2021-02-10T04:32:09.250 INFO:tasks.rados.rados.0.smithi085.stdout:1110: snap_create 2021-02-10T04:32:09.942 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:09 smithi085 conmon[52836]: debug 2021-02-10T04:32:09.550+0000 7fba6ffae700 -1 osd.3 pg_epoch: 312 pg[4.14( v 63'26 (0'0,63'26] local-lis/les=71/72 n=11 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,2,7] r=0 lpr=71 crt=63'26 lcod 63'25 mlcod 0'0 active+clean] scrubber pg(4.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55fec0658b00) 2021-02-10T04:32:10.250 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:10 smithi085 conmon[35961]: cluster 2021-02-10T04:32:09. 2021-02-10T04:32:10.250 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:10 smithi085 conmon[35961]: 250139+0000 mon.a (mon.0) 1062 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2021-02-10T04:32:10.250 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:10 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:32:10.250 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:10 smithi085 conmon[35961]: 32:10.050401+0000 mgr.y (mgr.14140) 470 2021-02-10T04:32:10.250 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:10 smithi085 conmon[35961]: : cluster [DBG] pgmap v618: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim, 163 active+clean; 123 MiB data, 477 MiB used, 715 GiB / 715 GiB avail; 800 KiB/s rd, 2.4 MiB/s wr, 11 op/s 2021-02-10T04:32:10.250 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:10 smithi085 conmon[29319]: cluster 2021-02-10T04:32:10.250 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:10 smithi085 conmon[29319]: 2021-02-10T04:32:09. 2021-02-10T04:32:10.250 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:10 smithi085 conmon[29319]: 250139+0000 mon.a (mon.0) 1062 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2021-02-10T04:32:10.251 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:10 smithi085 conmon[29319]: cluster 2021-02-10T04:32:10. 2021-02-10T04:32:10.251 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:10 smithi085 conmon[29319]: 050401+0000 mgr.y (mgr.14140) 470 : cluster [DBG] 2021-02-10T04:32:10.251 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:10 smithi085 conmon[29319]: pgmap v618: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim, 163 active+clean; 123 MiB data, 477 MiB used, 715 GiB / 715 GiB avail; 800 KiB/s rd, 2.4 MiB/s wr, 11 op/s 2021-02-10T04:32:10.257 INFO:tasks.rados.rados.0.smithi085.stdout:1110: done (13 left) 2021-02-10T04:32:10.257 INFO:tasks.rados.rados.0.smithi085.stdout:1111: rollback oid 36 current snap is 109 2021-02-10T04:32:10.257 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 36 to 107 2021-02-10T04:32:10.257 INFO:tasks.rados.rados.0.smithi085.stdout:1112: delete oid 5 current snap is 109 2021-02-10T04:32:10.264 INFO:tasks.rados.rados.0.smithi085.stdout:1111: finishing rollback tid 0 to smithi08575397-36 2021-02-10T04:32:10.264 INFO:tasks.rados.rados.0.smithi085.stdout:1111: finishing rollback tid 1 to smithi08575397-36 2021-02-10T04:32:10.264 INFO:tasks.rados.rados.0.smithi085.stdout:1112: done (14 left) 2021-02-10T04:32:10.265 INFO:tasks.rados.rados.0.smithi085.stdout:1113: write (excl) oid 22 current snap is 109 2021-02-10T04:32:10.265 INFO:tasks.rados.rados.0.smithi085.stdout:1113: seq_num 414 ranges {558098=636148,1826604=665129} 2021-02-10T04:32:10.272 INFO:tasks.rados.rados.0.smithi085.stdout:1113: writing smithi08575397-22 from 558098 to 1194246 tid 1 2021-02-10T04:32:10.280 INFO:tasks.rados.rados.0.smithi085.stdout:1113: writing smithi08575397-22 from 1826604 to 2491733 tid 2 2021-02-10T04:32:10.280 INFO:tasks.rados.rados.0.smithi085.stdout:1114: read oid 28 snap 108 2021-02-10T04:32:10.280 INFO:tasks.rados.rados.0.smithi085.stdout:1114: expect deleted 2021-02-10T04:32:10.280 INFO:tasks.rados.rados.0.smithi085.stdout:1111: finishing rollback tid 2 to smithi08575397-36 2021-02-10T04:32:10.281 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 180 (ObjNum 378 snap 99 seq_num 378) dirty exists 2021-02-10T04:32:10.281 INFO:tasks.rados.rados.0.smithi085.stdout:1113: finishing write tid 1 to smithi08575397-22 2021-02-10T04:32:10.281 INFO:tasks.rados.rados.0.smithi085.stdout:1111: done (15 left) 2021-02-10T04:32:10.281 INFO:tasks.rados.rados.0.smithi085.stdout:1115: read oid 5 snap 105 2021-02-10T04:32:10.281 INFO:tasks.rados.rados.0.smithi085.stdout:1115: expect (ObjNum 407 snap 105 seq_num 407) 2021-02-10T04:32:10.281 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:10.281 INFO:tasks.rados.rados.0.smithi085.stdout:1114: done (15 left) 2021-02-10T04:32:10.282 INFO:tasks.rados.rados.0.smithi085.stdout:1116: write oid 36 current snap is 109 2021-02-10T04:32:10.282 INFO:tasks.rados.rados.0.smithi085.stdout:1116: seq_num 415 ranges {574905=711397,1885086=525140,2930189=44716} 2021-02-10T04:32:10.290 INFO:tasks.rados.rados.0.smithi085.stdout:1116: writing smithi08575397-36 from 574905 to 1286302 tid 1 2021-02-10T04:32:10.300 INFO:tasks.rados.rados.0.smithi085.stdout:1116: writing smithi08575397-36 from 1885086 to 2410226 tid 2 2021-02-10T04:32:10.301 INFO:tasks.rados.rados.0.smithi085.stdout:1116: writing smithi08575397-36 from 2930189 to 2974905 tid 3 2021-02-10T04:32:10.301 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:10.301 INFO:tasks.rados.rados.0.smithi085.stdout:1113: finishing write tid 2 to smithi08575397-22 2021-02-10T04:32:10.301 INFO:tasks.rados.rados.0.smithi085.stdout:1113: finishing write tid 3 to smithi08575397-22 2021-02-10T04:32:10.301 INFO:tasks.rados.rados.0.smithi085.stdout:1113: finishing write tid 4 to smithi08575397-22 2021-02-10T04:32:10.302 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 466 (ObjNum 414 snap 109 seq_num 414) dirty exists 2021-02-10T04:32:10.302 INFO:tasks.rados.rados.0.smithi085.stdout:1113: left oid 22 (ObjNum 414 snap 109 seq_num 414) 2021-02-10T04:32:10.302 INFO:tasks.rados.rados.0.smithi085.stdout:1115: expect (ObjNum 407 snap 105 seq_num 407) 2021-02-10T04:32:10.471 INFO:tasks.rados.rados.0.smithi085.stdout:1116: finishing write tid 1 to smithi08575397-36 2021-02-10T04:32:10.471 INFO:tasks.rados.rados.0.smithi085.stdout:1116: finishing write tid 2 to smithi08575397-36 2021-02-10T04:32:10.471 INFO:tasks.rados.rados.0.smithi085.stdout:1116: finishing write tid 3 to smithi08575397-36 2021-02-10T04:32:10.471 INFO:tasks.rados.rados.0.smithi085.stdout:1113: done (15 left) 2021-02-10T04:32:10.471 INFO:tasks.rados.rados.0.smithi085.stdout:1115: done (14 left) 2021-02-10T04:32:10.471 INFO:tasks.rados.rados.0.smithi085.stdout:1117: delete oid 43 current snap is 109 2021-02-10T04:32:10.471 INFO:tasks.rados.rados.0.smithi085.stdout:1116: finishing write tid 4 to smithi08575397-36 2021-02-10T04:32:10.471 INFO:tasks.rados.rados.0.smithi085.stdout:1116: finishing write tid 5 to smithi08575397-36 2021-02-10T04:32:10.471 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 184 (ObjNum 415 snap 109 seq_num 415) dirty exists 2021-02-10T04:32:10.471 INFO:tasks.rados.rados.0.smithi085.stdout:1116: left oid 36 (ObjNum 415 snap 109 seq_num 415) 2021-02-10T04:32:10.472 INFO:tasks.rados.rados.0.smithi085.stdout:1116: done (14 left) 2021-02-10T04:32:10.473 INFO:tasks.rados.rados.0.smithi085.stdout:1117: done (13 left) 2021-02-10T04:32:10.473 INFO:tasks.rados.rados.0.smithi085.stdout:1118: read oid 46 snap -1 2021-02-10T04:32:10.473 INFO:tasks.rados.rados.0.smithi085.stdout:1118: expect (ObjNum 382 snap 100 seq_num 382) 2021-02-10T04:32:10.473 INFO:tasks.rados.rados.0.smithi085.stdout:1119: write (excl) oid 36 current snap is 109 2021-02-10T04:32:10.473 INFO:tasks.rados.rados.0.smithi085.stdout:1119: seq_num 417 ranges {608519=461895,1602050=528809,2585035=423484} 2021-02-10T04:32:10.479 INFO:tasks.rados.rados.0.smithi085.stdout:1119: writing smithi08575397-36 from 608519 to 1070414 tid 1 2021-02-10T04:32:10.486 INFO:tasks.rados.rados.0.smithi085.stdout:1119: writing smithi08575397-36 from 1602050 to 2130859 tid 2 2021-02-10T04:32:10.491 INFO:tasks.rados.rados.0.smithi085.stdout:1119: writing smithi08575397-36 from 2585035 to 3008519 tid 3 2021-02-10T04:32:10.491 INFO:tasks.rados.rados.0.smithi085.stdout:1120: read oid 31 snap -1 2021-02-10T04:32:10.491 INFO:tasks.rados.rados.0.smithi085.stdout:1120: expect deleted 2021-02-10T04:32:10.491 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:10.492 INFO:tasks.rados.rados.0.smithi085.stdout:1118: expect (ObjNum 382 snap 100 seq_num 382) 2021-02-10T04:32:10.607 INFO:tasks.rados.rados.0.smithi085.stdout:1119: finishing write tid 1 to smithi08575397-36 2021-02-10T04:32:10.607 INFO:tasks.rados.rados.0.smithi085.stdout:1119: finishing write tid 2 to smithi08575397-36 2021-02-10T04:32:10.607 INFO:tasks.rados.rados.0.smithi085.stdout:1119: finishing write tid 3 to smithi08575397-36 2021-02-10T04:32:10.607 INFO:tasks.rados.rados.0.smithi085.stdout:1119: finishing write tid 4 to smithi08575397-36 2021-02-10T04:32:10.607 INFO:tasks.rados.rados.0.smithi085.stdout:1119: finishing write tid 5 to smithi08575397-36 2021-02-10T04:32:10.607 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 188 (ObjNum 417 snap 109 seq_num 417) dirty exists 2021-02-10T04:32:10.607 INFO:tasks.rados.rados.0.smithi085.stdout:1119: left oid 36 (ObjNum 417 snap 109 seq_num 417) 2021-02-10T04:32:10.607 INFO:tasks.rados.rados.0.smithi085.stdout:1118: done (15 left) 2021-02-10T04:32:10.607 INFO:tasks.rados.rados.0.smithi085.stdout:1119: done (14 left) 2021-02-10T04:32:10.607 INFO:tasks.rados.rados.0.smithi085.stdout:1120: done (13 left) 2021-02-10T04:32:10.608 INFO:tasks.rados.rados.0.smithi085.stdout:1121: write (excl) oid 23 current snap is 109 2021-02-10T04:32:10.608 INFO:tasks.rados.rados.0.smithi085.stdout:1121: seq_num 418 ranges {625326=537144,1944179=672467} 2021-02-10T04:32:10.614 INFO:tasks.rados.rados.0.smithi085.stdout:1121: writing smithi08575397-23 from 625326 to 1162470 tid 1 2021-02-10T04:32:10.623 INFO:tasks.rados.rados.0.smithi085.stdout:1121: writing smithi08575397-23 from 1944179 to 2616646 tid 2 2021-02-10T04:32:10.624 INFO:tasks.rados.rados.0.smithi085.stdout:1122: write (excl) oid 6 current snap is 109 2021-02-10T04:32:10.624 INFO:tasks.rados.rados.0.smithi085.stdout:1122: seq_num 419 ranges {642133=728746,2119014=532478} 2021-02-10T04:32:10.633 INFO:tasks.rados.rados.0.smithi085.stdout:1122: writing smithi08575397-6 from 642133 to 1370879 tid 1 2021-02-10T04:32:10.645 INFO:tasks.rados.rados.0.smithi085.stdout:1122: writing smithi08575397-6 from 2119014 to 2651492 tid 2 2021-02-10T04:32:10.645 INFO:tasks.rados.rados.0.smithi085.stdout:1123: write (excl) oid 21 current snap is 109 2021-02-10T04:32:10.645 INFO:tasks.rados.rados.0.smithi085.stdout:1121: finishing write tid 1 to smithi08575397-23 2021-02-10T04:32:10.645 INFO:tasks.rados.rados.0.smithi085.stdout:1123: seq_num 420 ranges {658940=403995,1777496=676136,2950951=107989} 2021-02-10T04:32:10.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:10 smithi164 conmon[29288]: cluster 2021-02-10T04:32:09.250139+0000 mon.a (mon.0) 1062 : cluster [DBG] 2021-02-10T04:32:10.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:10 smithi164 conmon[29288]: osdmap e312: 8 total, 8 up, 8 in 2021-02-10T04:32:10.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:10 smithi164 conmon[29288]: cluster 2021-02-10T04:32:10.050401+0000 mgr.y (mgr.14140) 470 : cluster [DBG] pgmap v618: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim, 163 active+clean; 123 MiB data, 477 MiB used, 715 GiB / 715 GiB avail; 800 KiB/s rd, 2.4 MiB/s wr, 11 op/s 2021-02-10T04:32:10.647 INFO:tasks.rados.rados.0.smithi085.stdout:1123: writing smithi08575397-21 from 658940 to 1062935 tid 1 2021-02-10T04:32:10.655 INFO:tasks.rados.rados.0.smithi085.stdout:1123: writing smithi08575397-21 from 1777496 to 2453632 tid 2 2021-02-10T04:32:10.661 INFO:tasks.rados.rados.0.smithi085.stdout:1123: writing smithi08575397-21 from 2950951 to 3058940 tid 3 2021-02-10T04:32:10.661 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:10.661 INFO:tasks.rados.rados.0.smithi085.stdout:1121: finishing write tid 2 to smithi08575397-23 2021-02-10T04:32:10.661 INFO:tasks.rados.rados.0.smithi085.stdout:1121: finishing write tid 3 to smithi08575397-23 2021-02-10T04:32:10.662 INFO:tasks.rados.rados.0.smithi085.stdout:1121: finishing write tid 4 to smithi08575397-23 2021-02-10T04:32:10.662 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 192 (ObjNum 418 snap 109 seq_num 418) dirty exists 2021-02-10T04:32:10.662 INFO:tasks.rados.rados.0.smithi085.stdout:1121: left oid 23 (ObjNum 418 snap 109 seq_num 418) 2021-02-10T04:32:10.662 INFO:tasks.rados.rados.0.smithi085.stdout:1122: finishing write tid 1 to smithi08575397-6 2021-02-10T04:32:10.662 INFO:tasks.rados.rados.0.smithi085.stdout:1121: done (15 left) 2021-02-10T04:32:10.662 INFO:tasks.rados.rados.0.smithi085.stdout:1124: copy_from oid 16 from oid 28 current snap is 109 2021-02-10T04:32:10.662 INFO:tasks.rados.rados.0.smithi085.stdout:1122: finishing write tid 2 to smithi08575397-6 2021-02-10T04:32:10.663 INFO:tasks.rados.rados.0.smithi085.stdout:1122: finishing write tid 3 to smithi08575397-6 2021-02-10T04:32:10.663 INFO:tasks.rados.rados.0.smithi085.stdout:1122: finishing write tid 4 to smithi08575397-6 2021-02-10T04:32:10.663 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 378 (ObjNum 419 snap 109 seq_num 419) dirty exists 2021-02-10T04:32:10.664 INFO:tasks.rados.rados.0.smithi085.stdout:1122: left oid 6 (ObjNum 419 snap 109 seq_num 419) 2021-02-10T04:32:10.664 INFO:tasks.rados.rados.0.smithi085.stdout:1123: finishing write tid 1 to smithi08575397-21 2021-02-10T04:32:10.664 INFO:tasks.rados.rados.0.smithi085.stdout:1122: done (15 left) 2021-02-10T04:32:10.664 INFO:tasks.rados.rados.0.smithi085.stdout:1125: read oid 31 snap -1 2021-02-10T04:32:10.664 INFO:tasks.rados.rados.0.smithi085.stdout:1125: expect deleted 2021-02-10T04:32:10.664 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:10.664 INFO:tasks.rados.rados.0.smithi085.stdout:1125: done (15 left) 2021-02-10T04:32:10.664 INFO:tasks.rados.rados.0.smithi085.stdout:1126: copy_from oid 13 from oid 23 current snap is 109 2021-02-10T04:32:10.664 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:10.665 INFO:tasks.rados.rados.0.smithi085.stdout:1123: finishing write tid 2 to smithi08575397-21 2021-02-10T04:32:10.665 INFO:tasks.rados.rados.0.smithi085.stdout:1123: finishing write tid 3 to smithi08575397-21 2021-02-10T04:32:10.665 INFO:tasks.rados.rados.0.smithi085.stdout:1123: finishing write tid 4 to smithi08575397-21 2021-02-10T04:32:10.665 INFO:tasks.rados.rados.0.smithi085.stdout:1123: finishing write tid 5 to smithi08575397-21 2021-02-10T04:32:10.665 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 595 (ObjNum 420 snap 109 seq_num 420) dirty exists 2021-02-10T04:32:10.665 INFO:tasks.rados.rados.0.smithi085.stdout:1123: left oid 21 (ObjNum 420 snap 109 seq_num 420) 2021-02-10T04:32:10.665 INFO:tasks.rados.rados.0.smithi085.stdout:1123: done (15 left) 2021-02-10T04:32:10.666 INFO:tasks.rados.rados.0.smithi085.stdout:1127: write oid 36 current snap is 109 2021-02-10T04:32:10.666 INFO:tasks.rados.rados.0.smithi085.stdout:1127: seq_num 421 ranges {675747=479244,1835978=536147} 2021-02-10T04:32:10.670 INFO:tasks.rados.rados.0.smithi085.stdout:1127: writing smithi08575397-36 from 675747 to 1154991 tid 1 2021-02-10T04:32:10.677 INFO:tasks.rados.rados.0.smithi085.stdout:1127: writing smithi08575397-36 from 1835978 to 2372125 tid 2 2021-02-10T04:32:10.677 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:10.677 INFO:tasks.rados.rados.0.smithi085.stdout:1124: finishing copy_from to smithi08575397-16 2021-02-10T04:32:10.677 INFO:tasks.rados.rados.0.smithi085.stdout:1124: got expected ENOENT (src dne) 2021-02-10T04:32:10.677 INFO:tasks.rados.rados.0.smithi085.stdout:1124: finishing copy_from racing read to smithi08575397-16 2021-02-10T04:32:10.677 INFO:tasks.rados.rados.0.smithi085.stdout:1127: finishing write tid 1 to smithi08575397-36 2021-02-10T04:32:10.677 INFO:tasks.rados.rados.0.smithi085.stdout:1124: done (15 left) 2021-02-10T04:32:10.678 INFO:tasks.rados.rados.0.smithi085.stdout:1128: rollback oid 30 current snap is 109 2021-02-10T04:32:10.678 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 30 to 106 2021-02-10T04:32:10.678 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:10.684 INFO:tasks.rados.rados.0.smithi085.stdout:1127: finishing write tid 2 to smithi08575397-36 2021-02-10T04:32:10.686 INFO:tasks.rados.rados.0.smithi085.stdout:1127: finishing write tid 3 to smithi08575397-36 2021-02-10T04:32:10.686 INFO:tasks.rados.rados.0.smithi085.stdout:1127: finishing write tid 4 to smithi08575397-36 2021-02-10T04:32:10.686 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 195 (ObjNum 421 snap 109 seq_num 421) dirty exists 2021-02-10T04:32:10.686 INFO:tasks.rados.rados.0.smithi085.stdout:1127: left oid 36 (ObjNum 421 snap 109 seq_num 421) 2021-02-10T04:32:10.686 INFO:tasks.rados.rados.0.smithi085.stdout:1127: done (15 left) 2021-02-10T04:32:10.686 INFO:tasks.rados.rados.0.smithi085.stdout:1129: write oid 11 current snap is 109 2021-02-10T04:32:10.686 INFO:tasks.rados.rados.0.smithi085.stdout:1129: seq_num 422 ranges {692554=554493,1778107=679805,2889444=203110} 2021-02-10T04:32:10.690 INFO:tasks.rados.rados.0.smithi085.stdout:1129: writing smithi08575397-11 from 692554 to 1247047 tid 1 2021-02-10T04:32:10.697 INFO:tasks.rados.rados.0.smithi085.stdout:1129: writing smithi08575397-11 from 1778107 to 2457912 tid 2 2021-02-10T04:32:10.702 INFO:tasks.rados.rados.0.smithi085.stdout:1129: writing smithi08575397-11 from 2889444 to 3092554 tid 3 2021-02-10T04:32:10.702 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:10.702 INFO:tasks.rados.rados.0.smithi085.stdout:1128: finishing rollback tid 0 to smithi08575397-30 2021-02-10T04:32:10.702 INFO:tasks.rados.rados.0.smithi085.stdout:1128: finishing rollback tid 1 to smithi08575397-30 2021-02-10T04:32:10.702 INFO:tasks.rados.rados.0.smithi085.stdout:1128: finishing rollback tid 2 to smithi08575397-30 2021-02-10T04:32:10.702 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 382 (ObjNum 372 snap 96 seq_num 372) dirty exists 2021-02-10T04:32:10.702 INFO:tasks.rados.rados.0.smithi085.stdout:1129: finishing write tid 1 to smithi08575397-11 2021-02-10T04:32:10.702 INFO:tasks.rados.rados.0.smithi085.stdout:1128: done (15 left) 2021-02-10T04:32:10.702 INFO:tasks.rados.rados.0.smithi085.stdout:1130: copy_from oid 3 from oid 31 current snap is 109 2021-02-10T04:32:10.702 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:10.704 INFO:tasks.rados.rados.0.smithi085.stdout:1130: finishing copy_from to smithi08575397-3 2021-02-10T04:32:10.705 INFO:tasks.rados.rados.0.smithi085.stdout:1130: got expected ENOENT (src dne) 2021-02-10T04:32:10.705 INFO:tasks.rados.rados.0.smithi085.stdout:1130: finishing copy_from racing read to smithi08575397-3 2021-02-10T04:32:10.705 INFO:tasks.rados.rados.0.smithi085.stdout:1130: done (15 left) 2021-02-10T04:32:10.705 INFO:tasks.rados.rados.0.smithi085.stdout:1131: write oid 28 current snap is 109 2021-02-10T04:32:10.705 INFO:tasks.rados.rados.0.smithi085.stdout:1131: seq_num 423 ranges {709361=629742,1836589=539816,3033220=76141} 2021-02-10T04:32:10.714 INFO:tasks.rados.rados.0.smithi085.stdout:1131: writing smithi08575397-28 from 709361 to 1339103 tid 1 2021-02-10T04:32:10.719 INFO:tasks.rados.rados.0.smithi085.stdout:1131: writing smithi08575397-28 from 1836589 to 2376405 tid 2 2021-02-10T04:32:10.724 INFO:tasks.rados.rados.0.smithi085.stdout:1131: writing smithi08575397-28 from 3033220 to 3109361 tid 3 2021-02-10T04:32:10.724 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:10.724 INFO:tasks.rados.rados.0.smithi085.stdout:1129: finishing write tid 2 to smithi08575397-11 2021-02-10T04:32:10.724 INFO:tasks.rados.rados.0.smithi085.stdout:1129: finishing write tid 3 to smithi08575397-11 2021-02-10T04:32:10.724 INFO:tasks.rados.rados.0.smithi085.stdout:1129: finishing write tid 4 to smithi08575397-11 2021-02-10T04:32:10.725 INFO:tasks.rados.rados.0.smithi085.stdout:1129: finishing write tid 5 to smithi08575397-11 2021-02-10T04:32:10.725 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 600 (ObjNum 422 snap 109 seq_num 422) dirty exists 2021-02-10T04:32:10.725 INFO:tasks.rados.rados.0.smithi085.stdout:1129: left oid 11 (ObjNum 422 snap 109 seq_num 422) 2021-02-10T04:32:10.725 INFO:tasks.rados.rados.0.smithi085.stdout:1131: finishing write tid 1 to smithi08575397-28 2021-02-10T04:32:10.725 INFO:tasks.rados.rados.0.smithi085.stdout:1129: done (15 left) 2021-02-10T04:32:10.725 INFO:tasks.rados.rados.0.smithi085.stdout:1132: write (excl) oid 30 current snap is 109 2021-02-10T04:32:10.725 INFO:tasks.rados.rados.0.smithi085.stdout:1131: finishing write tid 2 to smithi08575397-28 2021-02-10T04:32:10.725 INFO:tasks.rados.rados.0.smithi085.stdout:1132: seq_num 424 ranges {726168=704991,1895071=683474} 2021-02-10T04:32:10.733 INFO:tasks.rados.rados.0.smithi085.stdout:1132: writing smithi08575397-30 from 726168 to 1431159 tid 1 2021-02-10T04:32:10.742 INFO:tasks.rados.rados.0.smithi085.stdout:1132: writing smithi08575397-30 from 1895071 to 2578545 tid 2 2021-02-10T04:32:10.742 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:10.742 INFO:tasks.rados.rados.0.smithi085.stdout:1131: finishing write tid 3 to smithi08575397-28 2021-02-10T04:32:10.742 INFO:tasks.rados.rados.0.smithi085.stdout:1131: finishing write tid 4 to smithi08575397-28 2021-02-10T04:32:10.742 INFO:tasks.rados.rados.0.smithi085.stdout:1131: finishing write tid 5 to smithi08575397-28 2021-02-10T04:32:10.743 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 176 (ObjNum 423 snap 109 seq_num 423) dirty exists 2021-02-10T04:32:10.743 INFO:tasks.rados.rados.0.smithi085.stdout:1131: left oid 28 (ObjNum 423 snap 109 seq_num 423) 2021-02-10T04:32:10.743 INFO:tasks.rados.rados.0.smithi085.stdout:1126: finishing copy_from to smithi08575397-13 2021-02-10T04:32:10.743 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 192 (ObjNum 418 snap 109 seq_num 418) dirty exists 2021-02-10T04:32:10.743 INFO:tasks.rados.rados.0.smithi085.stdout:1126: finishing copy_from racing read to smithi08575397-13 2021-02-10T04:32:10.743 INFO:tasks.rados.rados.0.smithi085.stdout:1132: finishing write tid 1 to smithi08575397-30 2021-02-10T04:32:10.743 INFO:tasks.rados.rados.0.smithi085.stdout:1126: done (15 left) 2021-02-10T04:32:10.743 INFO:tasks.rados.rados.0.smithi085.stdout:1131: done (14 left) 2021-02-10T04:32:10.743 INFO:tasks.rados.rados.0.smithi085.stdout:1133: read oid 2 snap 108 2021-02-10T04:32:10.743 INFO:tasks.rados.rados.0.smithi085.stdout:1133: expect (ObjNum 373 snap 96 seq_num 373) 2021-02-10T04:32:10.743 INFO:tasks.rados.rados.0.smithi085.stdout:1134: copy_from oid 13 from oid 37 current snap is 109 2021-02-10T04:32:10.743 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:10.744 INFO:tasks.rados.rados.0.smithi085.stdout:1132: finishing write tid 2 to smithi08575397-30 2021-02-10T04:32:10.744 INFO:tasks.rados.rados.0.smithi085.stdout:1133: expect (ObjNum 373 snap 96 seq_num 373) 2021-02-10T04:32:10.926 INFO:tasks.rados.rados.0.smithi085.stdout:1132: finishing write tid 3 to smithi08575397-30 2021-02-10T04:32:10.926 INFO:tasks.rados.rados.0.smithi085.stdout:1132: finishing write tid 4 to smithi08575397-30 2021-02-10T04:32:10.926 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 385 (ObjNum 424 snap 109 seq_num 424) dirty exists 2021-02-10T04:32:10.927 INFO:tasks.rados.rados.0.smithi085.stdout:1132: left oid 30 (ObjNum 424 snap 109 seq_num 424) 2021-02-10T04:32:10.927 INFO:tasks.rados.rados.0.smithi085.stdout:1134: finishing copy_from to smithi08575397-13 2021-02-10T04:32:10.927 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 574 (ObjNum 343 snap 88 seq_num 343) dirty exists 2021-02-10T04:32:10.927 INFO:tasks.rados.rados.0.smithi085.stdout:1134: finishing copy_from racing read to smithi08575397-13 2021-02-10T04:32:10.927 INFO:tasks.rados.rados.0.smithi085.stdout:1132: done (15 left) 2021-02-10T04:32:10.927 INFO:tasks.rados.rados.0.smithi085.stdout:1133: done (14 left) 2021-02-10T04:32:10.927 INFO:tasks.rados.rados.0.smithi085.stdout:1134: done (13 left) 2021-02-10T04:32:10.927 INFO:tasks.rados.rados.0.smithi085.stdout:1135: snap_create 2021-02-10T04:32:11.263 INFO:tasks.rados.rados.0.smithi085.stdout:1135: done (13 left) 2021-02-10T04:32:11.264 INFO:tasks.rados.rados.0.smithi085.stdout:1136: delete oid 8 current snap is 110 2021-02-10T04:32:11.269 INFO:tasks.rados.rados.0.smithi085.stdout:1136: done (13 left) 2021-02-10T04:32:11.269 INFO:tasks.rados.rados.0.smithi085.stdout:1137: snap_create 2021-02-10T04:32:11.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:11 smithi085 conmon[29319]: cluster 2021-02-10T04:32:11.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:11 smithi085 conmon[29319]: 2021-02 2021-02-10T04:32:11.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:11 smithi085 conmon[29319]: -10T04:32:10.255731+0000 mon.a (mon.0) 1063 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2021-02-10T04:32:11.566 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:11 smithi085 conmon[35961]: cluster 2021-02-10T04:32:10.255731 2021-02-10T04:32:11.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:11 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1063 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2021-02-10T04:32:11.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:11 smithi164 conmon[29288]: cluster 2021-02-10T04:32:10.255731+0000 mon.a (mon.0) 1063 2021-02-10T04:32:11.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:11 smithi164 conmon[29288]: : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2021-02-10T04:32:11.788 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.1 dump_blocked_ops 2021-02-10T04:32:11.852 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:11 smithi085 conmon[52836]: debug 2021-02-10T04:32:11.564+0000 7fba73fb6700 -1 osd.3 pg_epoch: 314 pg[4.14( v 63'26 (0'0,63'26] local-lis/les=71/72 n=11 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,2,7] r=0 lpr=71 crt=63'26 lcod 63'25 mlcod 0'0 active+clean] scrubber pg(4.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55fec0086f20) 2021-02-10T04:32:11.852 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:11 smithi085 conmon[52836]: debug 2021-02-10T04:32:11.788+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:11.900 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:32:12.009 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:32:12.042 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:12.136 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:32:12.274 INFO:tasks.rados.rados.0.smithi085.stdout:1137: done (13 left) 2021-02-10T04:32:12.275 INFO:tasks.rados.rados.0.smithi085.stdout:1138: snap_create 2021-02-10T04:32:12.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:32:12 smithi164 conmon[40892]: debug 2021-02-10T04:32:12.124+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:12.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:12 smithi164 conmon[29288]: cluster 2021-02-10T04:32:11.263334+0000 2021-02-10T04:32:12.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:12 smithi164 conmon[29288]: mon.a (mon.0) 1064 : cluster [DBG] osdmap e314: 8 total, 8 up, 8 in 2021-02-10T04:32:12.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:12 smithi164 conmon[29288]: cluster 2021-02-10T04:32:12.052993+0000 mgr.y (mgr. 2021-02-10T04:32:12.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:12 smithi164 conmon[29288]: 14140) 471 : cluster [DBG] pgmap v621: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim, 165 active+clean; 125 MiB data, 483 MiB used, 715 GiB / 715 GiB avail; 4.4 MiB/s rd, 3.0 MiB/s wr, 19 op/s 2021-02-10T04:32:12.426 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:32:12.531 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:32:12.558 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:12.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:12 smithi085 conmon[29319]: cluster 2021-02-10T04:32:12.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:12 smithi085 conmon[29319]: 2021-02-10T04:32: 2021-02-10T04:32:12.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:12 smithi085 conmon[29319]: 11.263334+0000 mon.a (mon.0) 1064 : cluster [DBG] osdmap e314: 8 total, 8 up, 8 in 2021-02-10T04:32:12.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:12 smithi085 conmon[29319]: cluster 2021-02-10T04:32:12.052993+0000 mgr.y (mgr.14140 2021-02-10T04:32:12.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:12 smithi085 conmon[29319]: ) 471 : cluster [DBG] pgmap v621: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim, 165 active+clean; 125 MiB data, 483 MiB used, 715 GiB / 715 GiB avail; 4.4 MiB/s rd, 3.0 MiB/s wr, 19 op/s 2021-02-10T04:32:12.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:12 smithi085 conmon[35961]: cluster 2021-02-10T04:32:11.263334+0000 2021-02-10T04:32:12.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:12 smithi085 conmon[35961]: mon.a (mon.0) 1064 : cluster [DBG] osdmap e314: 8 total, 8 up, 8 in 2021-02-10T04:32:12.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:12 smithi085 conmon[35961]: cluster 2021-02-10T04:32:12.052993 2021-02-10T04:32:12.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:12 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 471 : cluster [DBG] pgmap v621: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim, 165 active+clean; 125 MiB data, 483 MiB used, 715 GiB / 715 GiB avail; 4.4 MiB/s rd, 3.0 MiB/s wr, 19 op/s 2021-02-10T04:32:12.704 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:32:12.735 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:32:12.778 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:32:12.811 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:32:12.844 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:32:12.869 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:32:12.893 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:32:12 smithi164 conmon[40892]: debug 2021-02-10T04:32:12.626+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:12.900 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:32:12.908 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:32:12.911 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:32:12.932 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:32:12.933 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:32:12.934 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:32:12.934 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:32:12.934 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:32:12.934 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:32:12.934 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:32:12.934 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:32:12.934 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:32:12.993 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:32:13.097 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:32:13.139 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:13.279 INFO:tasks.rados.rados.0.smithi085.stdout:1138: done (13 left) 2021-02-10T04:32:13.279 INFO:tasks.rados.rados.0.smithi085.stdout:1139: read oid 22 snap -1 2021-02-10T04:32:13.279 INFO:tasks.rados.rados.0.smithi085.stdout:1139: expect (ObjNum 414 snap 109 seq_num 414) 2021-02-10T04:32:13.279 INFO:tasks.rados.rados.0.smithi085.stdout:1140: write (excl) oid 7 current snap is 112 2021-02-10T04:32:13.279 INFO:tasks.rados.rados.0.smithi085.stdout:1140: seq_num 426 ranges {759782=571842,2012035=687143} 2021-02-10T04:32:13.282 INFO:tasks.rados.rados.0.smithi085.stdout:1140: writing smithi08575397-7 from 759782 to 1331624 tid 1 2021-02-10T04:32:13.291 INFO:tasks.rados.rados.0.smithi085.stdout:1140: writing smithi08575397-7 from 2012035 to 2699178 tid 2 2021-02-10T04:32:13.292 INFO:tasks.rados.rados.0.smithi085.stdout:1141: snap_create 2021-02-10T04:32:13.292 INFO:tasks.rados.rados.0.smithi085.stdout:1140: finishing write tid 1 to smithi08575397-7 2021-02-10T04:32:13.292 INFO:tasks.rados.rados.0.smithi085.stdout:1139: expect (ObjNum 414 snap 109 seq_num 414) 2021-02-10T04:32:13.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[49106]: debug 2021-02-10T04:32:13.442 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[49106]: 2021-02-10T04:32:13.183+0000 7f77a3e94700 -1 osd.2 315 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:32:13.442 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[49106]: 2021-02-10T04:32:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[29319]: cluster 2021-02-10T04:32:12.274669+0000 mon.a 2021-02-10T04:32:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[29319]: (mon.0) 1065 : cluster [DBG] osdmap e315: 8 total, 8 up, 8 in 2021-02-10T04:32:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[29319]: audit 2021-02-10T04:32:12.669760+0000 mon.c (mon.1) 50 : audit [DBG] from='client.? 172.21.15.85:0/3355554583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:32:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[29319]: audit 2021-02-10T04:32:12.882491+0000 mgr.y (mgr.14140) 472 : audit 2021-02-10T04:32:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[29319]: [DBG] from='client.15033 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:32:13.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[35961]: cluster 2021-02-10T04:32:13.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[35961]: 2021-02-10T04:32:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[35961]: 2021 2021-02-10T04:32:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[35961]: - 2021-02-10T04:32:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[35961]: 02 2021-02-10T04:32:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[35961]: - 2021-02-10T04:32:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[35961]: 10 2021-02-10T04:32:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[35961]: T 2021-02-10T04:32:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[35961]: 04 2021-02-10T04:32:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[35961]: : 2021-02-10T04:32:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[35961]: 32 2021-02-10T04:32:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[35961]: : 2021-02-10T04:32:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[35961]: 12 2021-02-10T04:32:13.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[35961]: . 2021-02-10T04:32:13.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[35961]: 274669 2021-02-10T04:32:13.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[35961]: +0000 2021-02-10T04:32:13.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[35961]: 2021-02-10T04:32:13.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[35961]: mon.a (mon.0) 1065 : cluster [DBG] 2021-02-10T04:32:13.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[35961]: osdmap e315: 8 total, 8 up, 8 in 2021-02-10T04:32:13.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[35961]: audit 2021-02-10T04:32:12.669760+0000 mon.c 2021-02-10T04:32:13.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[35961]: (mon.1) 50 : audit [DBG] from='client.? 172.21.15.85:0/3355554583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:32:13.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[35961]: audit 2021-02-10T04:32:12.882491 2021-02-10T04:32:13.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 472 : audit [DBG] 2021-02-10T04:32:13.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[35961]: from='client.15033 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:32:13.445 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[45443]: debug 2021-02-10T04:32:13.327+0000 7f7140d73700 -1 osd.1 316 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:32:13.445 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[45443]: debug 2021-02-10T04:32:13.445 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[45443]: 2021-02-10T04:32:13.328+0000 7f7121cf0700 -1 osd.1 pg_epoch: 316 pg[4.4( v 63'31 (0'0,63'31] local-lis/les=55/56 n=14 ec=55/55 lis/c=55/55 les/c/f=56/56/0 sis=55) [1,2,3] r=0 lpr=55 crt=63'31 lcod 63'30 mlcod 63'30 active+clean] scrubber pg(4.4) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 3 (0x55dbcb41ec60) 2021-02-10T04:32:13.445 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[45443]: 2021-02-10T04:32:13.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:13 smithi164 conmon[29288]: cluster 2021-02-10T04:32:12.274669+0000 mon.a (mon. 2021-02-10T04:32:13.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:13 smithi164 conmon[29288]: 0) 1065 : cluster [DBG] osdmap e315: 8 total, 8 up, 8 in 2021-02-10T04:32:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:13 smithi164 conmon[29288]: audit 2021-02-10T04:32:12.669760+0000 mon.c (mon.1) 50 : audit [DBG] from='client.? 172.21.15.85:0/3355554583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:32:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:13 smithi164 conmon[29288]: audit 2021-02-10T04:32:12.882491+0000 mgr.y (mgr.14140) 472 : 2021-02-10T04:32:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:13 smithi164 conmon[29288]: audit [DBG] from='client.15033 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:32:13.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:13 smithi164 conmon[33677]: debug 2021-02-10T04:32:13.348+0000 7fbec19a1700 -1 osd.4 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:32:13.659 INFO:tasks.rados.rados.0.smithi085.stdout:1140: finishing write tid 2 to smithi08575397-7 2021-02-10T04:32:13.659 INFO:tasks.rados.rados.0.smithi085.stdout:1140: finishing write tid 3 to smithi08575397-7 2021-02-10T04:32:13.659 INFO:tasks.rados.rados.0.smithi085.stdout:1140: finishing write tid 4 to smithi08575397-7 2021-02-10T04:32:13.659 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 487 (ObjNum 426 snap 112 seq_num 426) dirty exists 2021-02-10T04:32:13.659 INFO:tasks.rados.rados.0.smithi085.stdout:1140: left oid 7 (ObjNum 426 snap 112 seq_num 426) 2021-02-10T04:32:13.935 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:32:13.940 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[49106]: debug 2021-02-10T04:32:13.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[49106]: 2021-02-10T04:32:13.631+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:13.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:13 smithi085 conmon[49106]: 2021-02-10T04:32:13.964 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset nodeep-scrub 2021-02-10T04:32:14.268 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:32:14.275 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[49106]: debug 2021-02-10T04:32:14.153+0000 7f77a3e94700 -1 osd.2 316 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:32:14.277 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: cluster 2021-02-10T04:32:13.185111+0000 osd.2 (osd 2021-02-10T04:32:14.277 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: .2) 296 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:14.277 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: cluster 2021-02-10T04:32:13.185123+0000 osd.2 (osd.2) 297 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:14.277 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: cluster 2021-02-10T04:32:13.185132+0000 osd.2 (osd.2) 298 : 2021-02-10T04:32:14.277 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:14.277 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: cluster 2021-02-10T04:32 2021-02-10T04:32:14.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: :13.185141+0000 osd.2 (osd.2) 2021-02-10T04:32:14.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: 299 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:14.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:32:14.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: 32:13.185148+0000 osd.2 (osd.2) 2021-02-10T04:32:14.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: 300 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:14.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:32:14.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: 04:32:13.185155+0000 osd.2 (osd. 2021-02-10T04:32:14.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: 2) 301 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:14.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:14.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: 02-10T04:32:13.185161+0000 osd.2 (osd.2) 302 : cluster 2021-02-10T04:32:14.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:14.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: cluster 2021-02-10T04:32:13 2021-02-10T04:32:14.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: .185167+0000 osd.2 (osd.2) 303 : cluster 2021-02-10T04:32:14.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:14.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:32:14.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: 32:13.270017+0000 mon.a (mon.0 2021-02-10T04:32:14.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: ) 1066 : cluster [DBG] osdmap e316: 8 total, 8 up, 8 in 2021-02-10T04:32:14.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:14.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: -10T04:32:13.328660+0000 osd.1 2021-02-10T04:32:14.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: (osd.1) 270 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:14.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: 2021-02-10T04:32:14.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: cluster 2021-02-10T04:32:14.295 INFO:tasks.rados.rados.0.smithi085.stdout:1139: done (15 left) 2021-02-10T04:32:14.295 INFO:tasks.rados.rados.0.smithi085.stdout:1140: done (14 left) 2021-02-10T04:32:14.295 INFO:tasks.rados.rados.0.smithi085.stdout:1141: done (13 left) 2021-02-10T04:32:14.295 INFO:tasks.rados.rados.0.smithi085.stdout:1142: write (excl) oid 2 current snap is 113 2021-02-10T04:32:14.295 INFO:tasks.rados.rados.0.smithi085.stdout:1142: seq_num 427 ranges {776589=647091,2070517=430801,3026559=150030} 2021-02-10T04:32:14.302 INFO:tasks.rados.rados.0.smithi085.stdout:1142: writing smithi08575397-2 from 776589 to 1423680 tid 1 2021-02-10T04:32:14.309 INFO:tasks.rados.rados.0.smithi085.stdout:1142: writing smithi08575397-2 from 2070517 to 2501318 tid 2 2021-02-10T04:32:14.313 INFO:tasks.rados.rados.0.smithi085.stdout:1142: writing smithi08575397-2 from 3026559 to 3176589 tid 3 2021-02-10T04:32:14.314 INFO:tasks.rados.rados.0.smithi085.stdout:1143: read oid 6 snap 106 2021-02-10T04:32:14.314 INFO:tasks.rados.rados.0.smithi085.stdout:1143: expect (ObjNum 402 snap 103 seq_num 402) 2021-02-10T04:32:14.314 INFO:tasks.rados.rados.0.smithi085.stdout:1142: finishing write tid 1 to smithi08575397-2 2021-02-10T04:32:14.314 INFO:tasks.rados.rados.0.smithi085.stdout:1142: finishing write tid 2 to smithi08575397-2 2021-02-10T04:32:14.314 INFO:tasks.rados.rados.0.smithi085.stdout:1144: read oid 28 snap -1 2021-02-10T04:32:14.314 INFO:tasks.rados.rados.0.smithi085.stdout:1144: expect (ObjNum 423 snap 109 seq_num 423) 2021-02-10T04:32:14.314 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:14.317 INFO:tasks.rados.rados.0.smithi085.stdout:1142: finishing write tid 3 to smithi08575397-2 2021-02-10T04:32:14.317 INFO:tasks.rados.rados.0.smithi085.stdout:1142: finishing write tid 4 to smithi08575397-2 2021-02-10T04:32:14.317 INFO:tasks.rados.rados.0.smithi085.stdout:1142: finishing write tid 5 to smithi08575397-2 2021-02-10T04:32:14.317 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 297 (ObjNum 427 snap 113 seq_num 427) dirty exists 2021-02-10T04:32:14.317 INFO:tasks.rados.rados.0.smithi085.stdout:1142: left oid 2 (ObjNum 427 snap 113 seq_num 427) 2021-02-10T04:32:14.318 INFO:tasks.rados.rados.0.smithi085.stdout:1142: done (15 left) 2021-02-10T04:32:14.318 INFO:tasks.rados.rados.0.smithi085.stdout:1145: write (excl) oid 5 current snap is 113 2021-02-10T04:32:14.318 INFO:tasks.rados.rados.0.smithi085.stdout:1145: seq_num 428 ranges {793396=722340,2128999=690812} 2021-02-10T04:32:14.327 INFO:tasks.rados.rados.0.smithi085.stdout:1145: writing smithi08575397-5 from 793396 to 1515736 tid 1 2021-02-10T04:32:14.335 INFO:tasks.rados.rados.0.smithi085.stdout:1145: writing smithi08575397-5 from 2128999 to 2819811 tid 2 2021-02-10T04:32:14.336 INFO:tasks.rados.rados.0.smithi085.stdout:1144: expect (ObjNum 423 snap 109 seq_num 423) 2021-02-10T04:32:14.474 INFO:tasks.rados.rados.0.smithi085.stdout:1143: expect (ObjNum 402 snap 103 seq_num 402) 2021-02-10T04:32:14.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[33677]: debug 2021-02-10T04:32:14.375+0000 7fbec19a1700 -1 osd.4 317 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:32:14.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: cluster 2021-02-10T04:32:13.185111+0000 osd.2 ( 2021-02-10T04:32:14.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: osd.2) 296 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:14.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: cluster 2021-02-10T04:32:13.185123+0000 osd.2 (osd.2) 297 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:14.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: cluster 2021-02-10T04:32:13.185132+0000 osd.2 (osd.2) 2021-02-10T04:32:14.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: 298 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:14.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:14.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: T04:32:13.185141+0000 osd.2 (osd.2) 299 : cluster 2021-02-10T04:32:14.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:14.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:14.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: 02-10T04:32:13.185148+0000 osd.2 (osd.2) 300 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:14.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:32:14.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: 10T04:32:13.185155+0000 osd.2 (osd.2) 301 : 2021-02-10T04:32:14.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:14.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: cluster 2021-02-10T04:32:13 2021-02-10T04:32:14.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: .185161+0000 osd.2 (osd.2) 302 : cluster [WRN] 2021-02-10T04:32:14.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:14.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:14.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: -02-10T04:32:13.185167+0000 osd.2 (osd.2) 303 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:14.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:14.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: 04:32:13.270017+0000 mon.a 2021-02-10T04:32:14.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: (mon.0) 1066 : cluster 2021-02-10T04:32:14.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: [DBG] osdmap e316: 8 total, 8 up, 8 in 2021-02-10T04:32:14.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:14.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: 02-10T04:32:13 2021-02-10T04:32:14.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: .328660+0000 osd.1 (osd.1) 270 : 2021-02-10T04:32:14.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:14.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: cluster 2021-02-10T04:32:13 2021-02-10T04:32:14.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: .328667+0000 osd.1 (osd.1) 271 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:14.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:32:14.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: 10T04:32:13.328675+0000 osd.1 (osd.1) 2021-02-10T04:32:14.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: 272 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:14.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: cluster 2021-02-10T04:32:14.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: 2021-02-10T04:32:13.328678+0000 osd.1 (osd.1) 2021-02-10T04:32:14.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: 273 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:14.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:14.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: 02-10T04:32:13.328682+0000 osd.1 (osd.1) 274 : cluster 2021-02-10T04:32:14.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:14.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: cluster 2021-02-10T04:32:13. 2021-02-10T04:32:14.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: 328685+0000 osd.1 (osd.1) 275 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:14.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: cluster 2021-02-10T04:32:13.349379+0000 osd.4 2021-02-10T04:32:14.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: (osd.4) 265 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:14.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:14.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: 04:32:13.349388+0000 osd.4 (osd.4) 266 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:14.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: cluster 2021-02-10T04:32:13.349392+0000 osd.4 (osd.4) 267 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:14.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: cluster 2021-02-10T04:32 2021-02-10T04:32:14.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: :13.349396+0000 osd.4 (osd.4) 268 : cluster [WRN] 2021-02-10T04:32:14.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:14.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:32:14.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:14 smithi164 conmon[29288]: 10T04:32:14.053414+0000 mgr.y (mgr.14140) 473 : cluster [DBG] pgmap v624: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim, 165 active+clean; 129 MiB data, 500 MiB used, 715 GiB / 715 GiB avail; 7.7 MiB/s rd, 2.2 MiB/s wr, 13 op/s 2021-02-10T04:32:14.658 INFO:tasks.rados.rados.0.smithi085.stdout:1145: finishing write tid 1 to smithi08575397-5 2021-02-10T04:32:14.658 INFO:tasks.rados.rados.0.smithi085.stdout:1143: done (15 left) 2021-02-10T04:32:14.658 INFO:tasks.rados.rados.0.smithi085.stdout:1144: done (14 left) 2021-02-10T04:32:14.658 INFO:tasks.rados.rados.0.smithi085.stdout:1146: write oid 26 current snap is 113 2021-02-10T04:32:14.659 INFO:tasks.rados.rados.0.smithi085.stdout:1145: finishing write tid 2 to smithi08575397-5 2021-02-10T04:32:14.659 INFO:tasks.rados.rados.0.smithi085.stdout:1146: seq_num 429 ranges {410203=797589,1787481=434470,2681405=528798} 2021-02-10T04:32:14.666 INFO:tasks.rados.rados.0.smithi085.stdout:1146: writing smithi08575397-26 from 410203 to 1207792 tid 1 2021-02-10T04:32:14.672 INFO:tasks.rados.rados.0.smithi085.stdout:1146: writing smithi08575397-26 from 1787481 to 2221951 tid 2 2021-02-10T04:32:14.678 INFO:tasks.rados.rados.0.smithi085.stdout:1146: writing smithi08575397-26 from 2681405 to 3210203 tid 3 2021-02-10T04:32:14.679 INFO:tasks.rados.rados.0.smithi085.stdout:1147: read oid 28 snap 110 2021-02-10T04:32:14.679 INFO:tasks.rados.rados.0.smithi085.stdout:1147: expect (ObjNum 423 snap 109 seq_num 423) 2021-02-10T04:32:14.679 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:14.679 INFO:tasks.rados.rados.0.smithi085.stdout:1145: finishing write tid 3 to smithi08575397-5 2021-02-10T04:32:14.679 INFO:tasks.rados.rados.0.smithi085.stdout:1145: finishing write tid 4 to smithi08575397-5 2021-02-10T04:32:14.679 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 577 (ObjNum 428 snap 113 seq_num 428) dirty exists 2021-02-10T04:32:14.679 INFO:tasks.rados.rados.0.smithi085.stdout:1145: left oid 5 (ObjNum 428 snap 113 seq_num 428) 2021-02-10T04:32:14.679 INFO:tasks.rados.rados.0.smithi085.stdout:1146: finishing write tid 1 to smithi08575397-26 2021-02-10T04:32:14.679 INFO:tasks.rados.rados.0.smithi085.stdout:1145: done (15 left) 2021-02-10T04:32:14.679 INFO:tasks.rados.rados.0.smithi085.stdout:1148: read oid 7 snap -1 2021-02-10T04:32:14.679 INFO:tasks.rados.rados.0.smithi085.stdout:1148: expect (ObjNum 426 snap 112 seq_num 426) 2021-02-10T04:32:14.679 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:14.679 INFO:tasks.rados.rados.0.smithi085.stdout:1146: finishing write tid 2 to smithi08575397-26 2021-02-10T04:32:14.682 INFO:tasks.rados.rados.0.smithi085.stdout:1147: expect (ObjNum 423 snap 109 seq_num 423) 2021-02-10T04:32:14.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: 2021-02-10T04:32:13.328667+0000 osd.1 ( 2021-02-10T04:32:14.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: osd.1) 271 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:14.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:14.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: 02-10T04:32:13.328675+0000 2021-02-10T04:32:14.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: osd.1 (osd.1) 272 : cluster [WRN] 2021-02-10T04:32:14.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:14.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: cluster 2021-02-10T04:32:14.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: 2021-02-10T04:32:13.328678+0000 osd.1 (osd. 2021-02-10T04:32:14.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: 1) 273 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:14.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:32:14.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: 04:32:13.328682+0000 osd.1 (osd. 2021-02-10T04:32:14.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: 1) 274 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:14.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:14.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: 10T04:32:13.328685+0000 osd.1 ( 2021-02-10T04:32:14.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: osd.1) 275 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:14.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:14.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: 02-10T04:32:13.349379+0000 osd.4 2021-02-10T04:32:14.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: (osd.4) 265 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:14.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: 2021-02-10T04:32:14.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: cluster 2021-02-10T04:32:14.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: 2021-02-10T04:32:13.349388+0000 osd.4 ( 2021-02-10T04:32:14.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: osd.4) 266 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:14.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:14.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: 02-10T04:32:13.349392+0000 2021-02-10T04:32:14.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: osd.4 (osd.4) 267 : cluster [WRN] 2021-02-10T04:32:14.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:14.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: cluster 2021-02-10T04:32:14.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: 2021-02-10T04:32:13.349396+0000 osd.4 2021-02-10T04:32:14.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: (osd.4) 268 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:14.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:14.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: -02-10T04:32:14.053414+0000 2021-02-10T04:32:14.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[29319]: mgr.y (mgr.14140) 473 : cluster [DBG] pgmap v624: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim, 165 active+clean; 129 MiB data, 500 MiB used, 715 GiB / 715 GiB avail; 7.7 MiB/s rd, 2.2 MiB/s wr, 13 op/s 2021-02-10T04:32:14.694 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[45443]: debug 2021-02-10T04:32:14.347+0000 7f7140d73700 -1 osd.1 317 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:32:14.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: cluster 2021-02-10T04:32:14.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: 2021-02-10T04:32:14.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: 2021 2021-02-10T04:32:14.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: - 2021-02-10T04:32:14.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: 02 2021-02-10T04:32:14.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: - 2021-02-10T04:32:14.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: 10T04:32:13.185111+0000 2021-02-10T04:32:14.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: osd.2 (osd.2) 296 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:14.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: cluster 2021-02-10T04:32:13.185123 2021-02-10T04:32:14.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 297 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:14.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: 2021-02-10T04:32:14.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: cluster 2021-02-10T04:32:14.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: 2021-02-10T04:32:13.185132+0000 osd.2 (osd. 2021-02-10T04:32:14.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: 2) 298 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:14.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:14.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: 04:32:13.185141+0000 osd.2 (osd.2) 2021-02-10T04:32:14.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: 299 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:14.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:14.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: :32:13.185148+0000 osd.2 (osd.2) 300 2021-02-10T04:32:14.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:14.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: cluster 2021-02-10T04:32: 2021-02-10T04:32:14.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: 13.185155+0000 osd.2 (osd.2) 301 : cluster 2021-02-10T04:32:14.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:14.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: cluster 2021-02-10T04:32: 2021-02-10T04:32:14.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: 13.185161+0000 osd.2 (osd.2) 302 : cluster 2021-02-10T04:32:14.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:14.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: cluster 2021-02-10T04:32 2021-02-10T04:32:14.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: :13.185167+0000 osd.2 (osd.2) 303 : 2021-02-10T04:32:14.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:14.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: cluster 2021-02-10T04:32: 2021-02-10T04:32:14.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: 13.270017+0000 mon.a (mon.0) 1066 : cluster 2021-02-10T04:32:14.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: [DBG] osdmap e316: 8 total, 8 up, 8 in 2021-02-10T04:32:14.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: cluster 2021-02-10T04:32: 2021-02-10T04:32:14.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: 13.328660+0000 osd.1 (osd.1) 270 : cluster 2021-02-10T04:32:14.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:14.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: cluster 2021-02-10T04:32:13 2021-02-10T04:32:14.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: .328667+0000 osd.1 (osd.1) 271 : cluster 2021-02-10T04:32:14.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:14.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: cluster 2021-02-10T04:32:13 2021-02-10T04:32:14.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: .328675+0000 osd.1 (osd.1) 272 : cluster [WRN] 2021-02-10T04:32:14.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:14.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: cluster 2021-02-10T04:32:14.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: 2021-02-10T04:32:13.328678+0000 osd.1 (osd.1) 2021-02-10T04:32:14.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: 273 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:14.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: cluster 2021-02-10T04:32:13.328682+0000 osd.1 (osd.1) 274 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:14.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: cluster 2021-02-10T04:32:13 2021-02-10T04:32:14.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: .328685+0000 osd.1 (osd.1) 275 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:14.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: cluster 2021-02-10T04:32:13. 2021-02-10T04:32:14.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: 349379+0000 osd.4 (osd.4) 265 : cluster [WRN] 2021-02-10T04:32:14.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:14.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: cluster 2021-02-10T04:32:14.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: 2021-02-10T04:32:13.349388+0000 osd.4 (osd.4 2021-02-10T04:32:14.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: ) 266 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:14.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: cluster 2021-02-10T04:32 2021-02-10T04:32:14.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: :13.349392+0000 osd.4 (osd.4) 267 : cluster 2021-02-10T04:32:14.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:14.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: cluster 2021-02-10T04:32:13 2021-02-10T04:32:14.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: .349396+0000 osd.4 (osd.4) 268 : cluster [WRN] 2021-02-10T04:32:14.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:14.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: cluster 2021-02-10T04:32:14.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: 2021-02-10T04:32:14.053414+0000 mgr.y (mgr.14140 2021-02-10T04:32:14.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:14 smithi085 conmon[35961]: ) 473 : cluster [DBG] pgmap v624: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim, 165 active+clean; 129 MiB data, 500 MiB used, 715 GiB / 715 GiB avail; 7.7 MiB/s rd, 2.2 MiB/s wr, 13 op/s 2021-02-10T04:32:14.829 INFO:tasks.rados.rados.0.smithi085.stdout:1147: done (15 left) 2021-02-10T04:32:14.829 INFO:tasks.rados.rados.0.smithi085.stdout:1149: snap_remove snap 105 2021-02-10T04:32:15.295 INFO:tasks.rados.rados.0.smithi085.stdout:1149: done (15 left) 2021-02-10T04:32:15.296 INFO:tasks.rados.rados.0.smithi085.stdout:1150: delete oid 43 current snap is 113 2021-02-10T04:32:15.296 INFO:tasks.rados.rados.0.smithi085.stdout:1146: finishing write tid 3 to smithi08575397-26 2021-02-10T04:32:15.296 INFO:tasks.rados.rados.0.smithi085.stdout:1146: finishing write tid 4 to smithi08575397-26 2021-02-10T04:32:15.296 INFO:tasks.rados.rados.0.smithi085.stdout:1146: finishing write tid 5 to smithi08575397-26 2021-02-10T04:32:15.296 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 90 (ObjNum 429 snap 113 seq_num 429) dirty exists 2021-02-10T04:32:15.296 INFO:tasks.rados.rados.0.smithi085.stdout:1146: left oid 26 (ObjNum 429 snap 113 seq_num 429) 2021-02-10T04:32:15.297 INFO:tasks.rados.rados.0.smithi085.stdout:1148: expect (ObjNum 426 snap 112 seq_num 426) 2021-02-10T04:32:15.555 INFO:tasks.rados.rados.0.smithi085.stdout:1146: done (15 left) 2021-02-10T04:32:15.555 INFO:tasks.rados.rados.0.smithi085.stdout:1148: done (14 left) 2021-02-10T04:32:15.555 INFO:tasks.rados.rados.0.smithi085.stdout:1150: done (13 left) 2021-02-10T04:32:15.556 INFO:tasks.rados.rados.0.smithi085.stdout:1151: copy_from oid 22 from oid 30 current snap is 113 2021-02-10T04:32:15.556 INFO:tasks.rados.rados.0.smithi085.stdout:1152: read oid 40 snap 108 2021-02-10T04:32:15.556 INFO:tasks.rados.rados.0.smithi085.stdout:1152: expect (ObjNum 389 snap 100 seq_num 389) 2021-02-10T04:32:15.556 INFO:tasks.rados.rados.0.smithi085.stdout:1153: snap_create 2021-02-10T04:32:15.561 INFO:tasks.rados.rados.0.smithi085.stdout:1152: expect (ObjNum 389 snap 100 seq_num 389) 2021-02-10T04:32:15.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[33677]: debug 2021-02-10T04:32:15.406+0000 7fbec19a1700 -1 osd.4 318 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:32:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: cluster 2021-02-10T04:32:14.154924+0000 osd.2 (osd.2) 304 : 2021-02-10T04:32:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: cluster 2021-02-10T04:32:14.154934+0000 osd.2 (osd.2) 305 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: -10T04:32:14. 2021-02-10T04:32:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: 154940+0000 osd.2 (osd.2 2021-02-10T04:32:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: ) 306 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:15.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:15.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: T04:32:14.154947+0000 osd.2 ( 2021-02-10T04:32:15.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: osd.2) 307 : cluster [WRN] 2021-02-10T04:32:15.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:15.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:32:15.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: 10T04:32:14.154953+0000 osd.2 (osd. 2021-02-10T04:32:15.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: 2) 308 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:15.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: 2021-02-10T04:32:15.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: cluster 2021-02-10T04:32:15.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: 2021-02-10T04:32:14.154959 2021-02-10T04:32:15.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 309 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:15.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:15.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: -02-10T04:32:14.154964+0000 2021-02-10T04:32:15.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: osd.2 (osd.2) 310 : cluster [WRN] 2021-02-10T04:32:15.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:15.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: cluster 2021-02-10T04:32: 2021-02-10T04:32:15.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: 14.154977+0000 osd.2 (osd.2) 311 2021-02-10T04:32:15.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:15.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:15.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: 04:32:14.288623+0000 mon.a (mon 2021-02-10T04:32:15.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: .0) 1067 : cluster [DBG] osdmap e317: 8 total, 8 up, 8 in 2021-02-10T04:32:15.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:15.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: 02-10T04:32:14.348561+0000 2021-02-10T04:32:15.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: osd.1 (osd.1) 276 : cluster [WRN] 2021-02-10T04:32:15.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:15.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: cluster 2021-02-10T04:32: 2021-02-10T04:32:15.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: 14.348579+0000 osd.1 (osd.1) 277 : 2021-02-10T04:32:15.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:15.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: cluster 2021-02-10T04:32:14.348586+0000 osd.1 (osd.1) 278 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:15.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: cluster 2021-02-10T04:32:14.348592+0000 2021-02-10T04:32:15.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: osd.1 (osd.1) 279 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:15.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:15.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: 04:32:14.348597+0000 osd.1 (osd 2021-02-10T04:32:15.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: .1) 280 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:15.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:15.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: -10T04:32:14.348604+0000 2021-02-10T04:32:15.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: osd.1 (osd.1) 281 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:15.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: 2021-02-10T04:32:15.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: cluster 2021-02-10T04:32:15.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: 2021-02-10T04:32:14.376326+0000 osd.4 ( 2021-02-10T04:32:15.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: osd.4) 269 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:15.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:15.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: 02-10T04:32:14.376336+0000 2021-02-10T04:32:15.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: osd.4 (osd.4) 270 : cluster [WRN] 2021-02-10T04:32:15.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:15.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: cluster 2021-02-10T04:32:15.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: 2021-02-10T04:32:14.376343+0000 osd.4 2021-02-10T04:32:15.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: (osd.4) 271 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:15.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: 2021-02-10T04:32:15.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: cluster 2021-02-10T04:32:15.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: 2021-02-10T04:32:14.376349+0000 osd.4 2021-02-10T04:32:15.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:15 smithi164 conmon[29288]: (osd.4) 272 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:15.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[49106]: debug 2021-02-10T04:32:15.195+0000 7f77a3e94700 -1 osd.2 317 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:32:15.691 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[45443]: debug 2021-02-10T04:32:15.302+0000 7f7140d73700 -1 osd.1 318 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:32:15.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: cluster 2021-02-10T04:32:15.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:32:15.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: 32:14.154924+0000 osd.2 (osd.2) 304 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:15.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: cluster 2021-02-10T04:32:14.154934+0000 osd.2 (osd.2) 305 2021-02-10T04:32:15.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:15.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:32:15.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: T04:32:14.154940+0000 osd.2 (osd.2) 306 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:15.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: cluster 2021-02-10T04:32:14.154947+0000 2021-02-10T04:32:15.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: osd.2 (osd.2) 307 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:15.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: cluster 2021-02-10T04:32:14.154953+0000 2021-02-10T04:32:15.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: osd.2 (osd.2) 308 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:15.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:32:15.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: 04:32:14.154959+0000 osd.2 (osd.2) 309 2021-02-10T04:32:15.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:15.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: cluster 2021-02-10T04:32:15.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: 2021-02-10T04:32:14 2021-02-10T04:32:15.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: .154964+0000 osd.2 (osd.2) 310 : cluster 2021-02-10T04:32:15.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:15.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:15.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: 10T04:32:14.154977+0000 osd.2 (osd. 2021-02-10T04:32:15.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: 2) 311 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:15.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:15.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: -10T04:32:14.288623+0000 mon.a (mon 2021-02-10T04:32:15.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: .0) 1067 : cluster [DBG] osdmap e317: 8 total, 8 up, 8 in 2021-02-10T04:32:15.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: cluster 2021-02-10T04:32:15.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: 2021-02-10T04:32:14. 2021-02-10T04:32:15.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: 348561+0000 osd.1 (osd.1) 276 : cluster [WRN] 2021-02-10T04:32:15.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:15.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:15.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: -10T04:32:14.348579+0000 osd.1 2021-02-10T04:32:15.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: (osd.1) 277 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:15.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:15.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: -02-10T04:32:14.348586+0000 2021-02-10T04:32:15.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: osd.1 (osd.1) 278 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:15.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: cluster 2021-02-10T04:32:15.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: 2021-02-10T04:32: 2021-02-10T04:32:15.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: 14.348592+0000 osd.1 (osd. 2021-02-10T04:32:15.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: 1) 279 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:15.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: 2021-02-10T04:32:15.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:15.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: -02-10T04:32:14.348597+0000 2021-02-10T04:32:15.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: osd.1 (osd.1) 280 : cluster [WRN] 2021-02-10T04:32:15.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:15.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: cluster 2021-02-10T04:32:15.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: 2021-02-10T04:32 2021-02-10T04:32:15.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: :14.348604+0000 osd.1 (osd.1) 281 2021-02-10T04:32:15.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: : cluster [WRN] 2021-02-10T04:32:15.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:15.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:32:15.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: 04:32:14.376326+0000 osd.4 (osd.4 2021-02-10T04:32:15.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: ) 269 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:15.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: cluster 2021-02-10T04:32:15.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: 2021-02-10T04:32:14.376336+0000 2021-02-10T04:32:15.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: osd.4 (osd.4) 270 : cluster 2021-02-10T04:32:15.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:15.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:15.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: -02-10T04:32:14.376343+0000 2021-02-10T04:32:15.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: osd.4 (osd.4) 271 : 2021-02-10T04:32:15.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:15.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: cluster 2021-02-10T04:32:15.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: 2021-02-10T04:32:14.376349 2021-02-10T04:32:15.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 272 : 2021-02-10T04:32:15.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:15.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: cluster 2021-02-10T04:32:14.154924 2021-02-10T04:32:15.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 304 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:15.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:15.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: -02-10T04:32:14.154934+0000 osd.2 (osd.2) 305 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:15.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:15.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: :32:14.154940+0000 osd.2 (osd.2) 306 2021-02-10T04:32:15.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:15.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:32:15.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: T04:32:14.154947+0000 osd.2 (osd.2) 2021-02-10T04:32:15.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: 307 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:15.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:15.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: 02-10T04:32:14.154953+0000 osd.2 2021-02-10T04:32:15.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: (osd.2) 308 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:15.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:15.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: -02-10T04:32:14.154959+0000 osd.2 (osd.2) 309 : 2021-02-10T04:32:15.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:15.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: cluster 2021-02-10T04:32:15.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: 2021-02-10T04:32:14.154964+0000 osd.2 (osd 2021-02-10T04:32:15.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: .2) 310 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:15.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: cluster 2021-02-10T04:32:14.154977+0000 2021-02-10T04:32:15.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: osd.2 (osd.2) 311 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:15.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: 2021-02-10T04:32:15.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:15.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: -02-10T04:32:14.288623+0000 mon.a (mon.0) 2021-02-10T04:32:15.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: 1067 : cluster [DBG] osdmap e317: 8 total, 8 up, 8 in 2021-02-10T04:32:15.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:15.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: 10T04:32:14.348561+0000 osd.1 (osd. 2021-02-10T04:32:15.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: 1) 276 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:15.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:15.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: -10T04:32:14.348579+0000 osd.1 2021-02-10T04:32:15.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: (osd.1) 277 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:15.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:15.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: -10T04:32:14.348586+0000 2021-02-10T04:32:15.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: osd.1 (osd.1) 278 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:15.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:15.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: :32:14.348592+0000 osd.1 (osd.1 2021-02-10T04:32:15.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: ) 279 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:15.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:15.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: 02-10T04:32:14.348597+0000 osd.1 ( 2021-02-10T04:32:15.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: osd.1) 280 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:15.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:15.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: 02-10T04:32:14.348604+0000 2021-02-10T04:32:15.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: osd.1 (osd.1) 281 : cluster [WRN] 2021-02-10T04:32:15.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:15.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: cluster 2021-02-10T04:32:15.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: 2021-02-10T04:32:14.376326+0000 osd.4 (osd 2021-02-10T04:32:15.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: .4) 269 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:15.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:15.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: :32:14.376336+0000 osd.4 (osd.4) 270 2021-02-10T04:32:15.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:15.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: cluster 2021-02-10T04:32:15.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: 2021-02-10T04:32:14.376343+0000 osd.4 (osd.4) 271 : cluster [WRN] 2021-02-10T04:32:15.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:15.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:15.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: -02-10T04:32:14.376349+0000 osd.4 (osd.4) 2021-02-10T04:32:15.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:15 smithi085 conmon[35961]: 272 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:15.730 INFO:tasks.rados.rados.0.smithi085.stdout:1151: finishing copy_from to smithi08575397-22 2021-02-10T04:32:15.730 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 385 (ObjNum 424 snap 109 seq_num 424) dirty exists 2021-02-10T04:32:15.730 INFO:tasks.rados.rados.0.smithi085.stdout:1151: finishing copy_from racing read to smithi08575397-22 2021-02-10T04:32:16.305 INFO:tasks.rados.rados.0.smithi085.stdout:1151: done (15 left) 2021-02-10T04:32:16.305 INFO:tasks.rados.rados.0.smithi085.stdout:1152: done (14 left) 2021-02-10T04:32:16.305 INFO:tasks.rados.rados.0.smithi085.stdout:1153: done (13 left) 2021-02-10T04:32:16.305 INFO:tasks.rados.rados.0.smithi085.stdout:1154: rollback oid 22 current snap is 114 2021-02-10T04:32:16.305 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 22 to 108 2021-02-10T04:32:16.305 INFO:tasks.rados.rados.0.smithi085.stdout:1155: snap_remove snap 113 2021-02-10T04:32:16.518 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:32:16.549 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: cluster 2021-02-10T04:32:15.197120+0000 osd.2 (osd. 2021-02-10T04:32:16.549 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: 2) 312 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:16.549 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: cluster 2021-02-10T04:32:15.197129+0000 osd.2 (osd.2) 313 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:16.549 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:16.549 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: -10T04:32:15.197135+0000 osd.2 (osd.2) 314 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:16.549 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:16.549 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: 04:32:15.197156+0000 osd.2 (osd.2 2021-02-10T04:32:16.549 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: ) 315 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:16.550 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:16.550 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: T04:32:15.197163+0000 osd.2 (osd. 2021-02-10T04:32:16.550 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: 2) 316 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:16.550 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:16.550 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: T04:32:15.197171+0000 osd.2 (osd. 2021-02-10T04:32:16.550 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: 2) 317 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:16.550 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:32:16.550 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: :32:15.197177+0000 osd.2 (osd.2) 318 2021-02-10T04:32:16.550 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:16.550 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: cluster 2021-02-10T04:32:15.197185 2021-02-10T04:32:16.550 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 319 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:16.550 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: cluster 2021-02-10T04:32:15.293940+0000 mon.a (mon.0 2021-02-10T04:32:16.550 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: ) 1068 : cluster [DBG] osdmap e318: 8 total, 8 up, 8 in 2021-02-10T04:32:16.551 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: cluster 2021-02-10T04:32: 2021-02-10T04:32:16.551 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: 15.304003+0000 osd.1 (osd.1) 282 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:16.551 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: 2021-02-10T04:32:16.551 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:16.551 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: -02-10T04:32:15.304011+0000 osd.1 (osd.1) 283 : cluster [WRN] 2021-02-10T04:32:16.551 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:16.551 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:32:16.551 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: :32:15.304017+0000 osd.1 (osd.1) 284 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:16.551 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: cluster 2021-02-10T04:32:15 2021-02-10T04:32:16.551 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: .304026+0000 osd.1 (osd.1) 285 : cluster 2021-02-10T04:32:16.551 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:16.551 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: cluster 2021-02-10T04:32:15. 2021-02-10T04:32:16.551 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: 304032+0000 osd.1 (osd.1) 286 : cluster [WRN] 2021-02-10T04:32:16.552 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:16.552 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: cluster 2021-02-10T04:32:16.552 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: 2021-02-10T04:32:15.407157+0000 osd.4 (osd.4) 2021-02-10T04:32:16.552 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: 273 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:16.552 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:32:16.552 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: :32:15.407165+0000 osd.4 (osd.4) 274 2021-02-10T04:32:16.552 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:16.552 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:32:16.552 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: 32:15.407172+0000 osd.4 (osd.4) 275 : cluster [WRN] 2021-02-10T04:32:16.552 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:16.552 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: cluster 2021-02-10T04:32:16.552 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: 2021-02-10T04:32:15.407175+0000 osd.4 (osd.4 2021-02-10T04:32:16.552 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: ) 276 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:16.552 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:32:16.553 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: :32:16.053957+0000 mgr.y (mgr.14140) 474 : 2021-02-10T04:32:16.553 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[29288]: cluster [DBG] pgmap v627: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 148 MiB data, 536 MiB used, 715 GiB / 715 GiB avail; 22 MiB/s rd, 5.7 MiB/s wr, 41 op/s 2021-02-10T04:32:16.553 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[33677]: debug 2021-02-10T04:32:16.410+0000 7fbec19a1700 -1 osd.4 318 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:32:16.553 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:16.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: cluster 2021-02-10T04:32:15.197120+0000 osd.2 (osd. 2021-02-10T04:32:16.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: 2) 312 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:16.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: cluster 2021-02-10T04:32:15.197129+0000 osd.2 (osd.2) 313 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:16.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:32:16.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: T04:32:15.197135+0000 2021-02-10T04:32:16.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: osd.2 (osd.2) 314 : cluster [WRN] 2021-02-10T04:32:16.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:16.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:16.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: 02-10T04:32:15.197156+0000 2021-02-10T04:32:16.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: osd.2 (osd.2) 315 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:16.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: 2021-02-10T04:32:16.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:32:16.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: T04:32:15.197163+0000 osd.2 ( 2021-02-10T04:32:16.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: osd.2) 316 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:16.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:16.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: 02-10T04:32:15.197171+0000 osd.2 (osd 2021-02-10T04:32:16.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: .2) 317 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:16.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: cluster 2021-02-10T04:32:15. 2021-02-10T04:32:16.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: 197177+0000 osd.2 (osd.2) 318 : cluster [WRN] 2021-02-10T04:32:16.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:16.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: cluster 2021-02-10T04:32:16.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: 2021-02-10T04:32:15.197185+0000 osd.2 ( 2021-02-10T04:32:16.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: osd.2) 319 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:16.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: cluster 2021-02-10T04:32:15.293940+0000 2021-02-10T04:32:16.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: mon.a (mon.0) 1068 : cluster [DBG] osdmap e318: 8 total, 8 up, 8 in 2021-02-10T04:32:16.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: cluster 2021-02-10T04:32:15.304003+0000 2021-02-10T04:32:16.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: osd.1 (osd.1) 282 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:16.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: cluster 2021-02-10T04:32:15. 2021-02-10T04:32:16.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: 304011+0000 osd.1 (osd.1) 283 : cluster 2021-02-10T04:32:16.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:16.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: cluster 2021-02-10T04:32:16.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: 2021-02-10T04:32:15.304017+0000 osd.1 (osd.1) 284 : cluster 2021-02-10T04:32:16.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:16.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:16.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: -10T04:32:15.304026+0000 osd.1 (osd.1) 285 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:16.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: cluster 2021-02-10T04:32 2021-02-10T04:32:16.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: :15.304032+0000 osd.1 (osd.1) 2021-02-10T04:32:16.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: 286 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:16.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: cluster 2021-02-10T04:32 2021-02-10T04:32:16.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: :15.407157+0000 osd.4 (osd.4) 273 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:16.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: cluster 2021-02-10T04:32:15. 2021-02-10T04:32:16.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: 407165+0000 osd.4 (osd.4) 274 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:16.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:16.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: -10T04:32:15.407172+0000 2021-02-10T04:32:16.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: osd.4 (osd.4) 275 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:16.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:32:16.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: 04:32:15.407175+0000 osd.4 (osd.4) 2021-02-10T04:32:16.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: 276 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:16.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:32:16.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: 32:16.053957 2021-02-10T04:32:16.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 474 : cluster 2021-02-10T04:32:16.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[29319]: [DBG] pgmap v627: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 148 MiB data, 536 MiB used, 715 GiB / 715 GiB avail; 22 MiB/s rd, 5.7 MiB/s wr, 41 op/s 2021-02-10T04:32:16.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: cluster 2021-02-10T04:32:15. 2021-02-10T04:32:16.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: 197120+0000 osd.2 (osd.2) 312 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:16.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: cluster 2021-02-10T04:32:16.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: 2021-02-10T04:32:15.197129+0000 osd.2 (osd.2) 313 : cluster 2021-02-10T04:32:16.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:16.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: cluster 2021-02-10T04:32:16.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: 2021-02-10T04:32:15.197135+0000 osd.2 (osd.2 2021-02-10T04:32:16.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: ) 314 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:16.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:16.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: :32:15.197156+0000 osd.2 (osd.2) 2021-02-10T04:32:16.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: 315 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:16.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:16.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: :32:15.197163+0000 osd.2 (osd. 2021-02-10T04:32:16.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: 2) 316 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:16.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:16.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: -02-10T04:32:15.197171+0000 2021-02-10T04:32:16.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: osd.2 (osd.2) 317 : cluster [WRN] 2021-02-10T04:32:16.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:16.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: cluster 2021-02-10T04:32:16.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: 2021-02-10T04:32:15.197177+0000 osd.2 (osd 2021-02-10T04:32:16.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: .2) 318 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:16.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:16.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: 04:32:15.197185+0000 osd.2 (osd 2021-02-10T04:32:16.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: .2) 319 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:16.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:16.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: 02-10T04:32:15.293940+0000 2021-02-10T04:32:16.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: mon.a (mon.0) 1068 : cluster [DBG] osdmap e318: 8 total, 8 up, 8 in 2021-02-10T04:32:16.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: 2021-02-10T04:32:16.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: cluster 2021-02-10T04:32: 2021-02-10T04:32:16.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: 15.304003+0000 osd.1 (osd.1) 282 : cluster 2021-02-10T04:32:16.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:16.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:32:16.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: 32:15.304011+0000 osd.1 (osd.1) 2021-02-10T04:32:16.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: 283 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:16.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: cluster 2021-02-10T04:32 2021-02-10T04:32:16.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: :15.304017+0000 osd.1 (osd.1) 284 : cluster 2021-02-10T04:32:16.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:16.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: cluster 2021-02-10T04:32:15.304026 2021-02-10T04:32:16.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 285 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:16.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: 2021-02-10T04:32:16.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: cluster 2021-02-10T04:32:16.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: 2021-02-10T04:32:15.304032+0000 osd.1 ( 2021-02-10T04:32:16.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: osd.1) 286 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:16.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:16.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: 02-10T04:32:15.407157+0000 osd.4 2021-02-10T04:32:16.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: (osd.4) 273 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:16.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:16.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: -10T04:32:15.407165+0000 osd.4 ( 2021-02-10T04:32:16.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: osd.4) 274 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:16.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:16.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: 02-10T04:32:15.407172+0000 osd.4 ( 2021-02-10T04:32:16.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: osd.4) 275 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:16.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: cluster 2021-02-10T04:32:15.407175+0000 osd.4 (osd.4 2021-02-10T04:32:16.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: ) 276 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:16.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:16.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: -02-10T04:32:16.053957+0000 mgr.y (mgr.14140) 474 : cluster [DBG] pgmap v627: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 148 MiB data, 536 MiB used, 715 GiB / 715 GiB avail; 22 MiB/s rd, 5.7 MiB/s wr, 41 op/s 2021-02-10T04:32:16.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[35961]: 2021-02-10T04:32:16.701 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[45443]: debug 2021-02-10T04:32:16.330+0000 7f7140d73700 -1 osd.1 319 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:32:16.702 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:16 smithi085 conmon[49106]: debug 2021-02-10T04:32:16.199+0000 7f77a3e94700 -1 osd.2 318 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:32:16.893 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:16 smithi164 conmon[33677]: debug 2021-02-10T04:32:16.643+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:16.909 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:32:17.014 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:32:17.045 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:17.299 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[44544]: debug 2021-02-10T04:32:17.299 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[44544]: 2021-02-10T04:32:17.130+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:17.315 INFO:tasks.rados.rados.0.smithi085.stdout:1155: done (14 left) 2021-02-10T04:32:17.315 INFO:tasks.rados.rados.0.smithi085.stdout:1156: write (excl) oid 21 current snap is 114 2021-02-10T04:32:17.315 INFO:tasks.rados.rados.0.smithi085.stdout:1156: seq_num 431 ranges {443817=548087,1788092=438139,3019898=223919} 2021-02-10T04:32:17.315 INFO:tasks.rados.rados.0.smithi085.stdout:1156: writing smithi08575397-21 from 443817 to 991904 tid 1 2021-02-10T04:32:17.323 INFO:tasks.rados.rados.0.smithi085.stdout:1156: writing smithi08575397-21 from 1788092 to 2226231 tid 2 2021-02-10T04:32:17.330 INFO:tasks.rados.rados.0.smithi085.stdout:1156: writing smithi08575397-21 from 3019898 to 3243817 tid 3 2021-02-10T04:32:17.330 INFO:tasks.rados.rados.0.smithi085.stdout:1154: finishing rollback tid 0 to smithi08575397-22 2021-02-10T04:32:17.330 INFO:tasks.rados.rados.0.smithi085.stdout:1154: finishing rollback tid 1 to smithi08575397-22 2021-02-10T04:32:17.330 INFO:tasks.rados.rados.0.smithi085.stdout:1154: finishing rollback tid 2 to smithi08575397-22 2021-02-10T04:32:17.330 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 469 (ObjNum 412 snap 107 seq_num 412) dirty exists 2021-02-10T04:32:17.330 INFO:tasks.rados.rados.0.smithi085.stdout:1156: finishing write tid 1 to smithi08575397-21 2021-02-10T04:32:17.331 INFO:tasks.rados.rados.0.smithi085.stdout:1156: finishing write tid 2 to smithi08575397-21 2021-02-10T04:32:17.331 INFO:tasks.rados.rados.0.smithi085.stdout:1154: done (14 left) 2021-02-10T04:32:17.331 INFO:tasks.rados.rados.0.smithi085.stdout:1157: snap_remove snap 107 2021-02-10T04:32:17.392 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:32:17.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: cluster 2021-02-10T04:32:16.201238+0000 osd.2 (osd.2) 320 : 2021-02-10T04:32:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: cluster 2021-02-10T04:32:16.201247+0000 osd.2 (osd.2) 321 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: cluster 2021-02-10T04:32 2021-02-10T04:32:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: :16.201260+0000 osd.2 (osd.2) 322 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: cluster 2021-02-10T04:32:16.201267+0000 2021-02-10T04:32:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: osd.2 (osd.2) 323 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:32:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: 04:32:16.201272+0000 osd.2 (osd. 2021-02-10T04:32:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: 2) 324 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: 10T04:32:16.201277+0000 osd.2 ( 2021-02-10T04:32:17.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: osd.2) 325 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:17.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:17.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: -02-10T04:32:16.201287+0000 osd.2 (osd.2) 326 : cluster 2021-02-10T04:32:17.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:17.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: cluster 2021-02-10T04:32:16.201292+0000 2021-02-10T04:32:17.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: osd.2 (osd.2) 327 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:17.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: 2021-02-10T04:32:17.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: cluster 2021-02-10T04:32:17.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: 2021-02-10T04:32:16.291345+0000 mon.a 2021-02-10T04:32:17.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: (mon.0) 1069 : cluster [WRN] Health check failed: 8 slow ops, oldest one blocked for 143 sec, osd.2 has slow ops (SLOW_OPS) 2021-02-10T04:32:17.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: 2021-02-10T04:32:17.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: cluster 2021-02-10T04:32:17.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: 2021-02-10T04:32:16.301897+0000 mon.a ( 2021-02-10T04:32:17.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: mon.0) 1070 : cluster [DBG] osdmap e319: 8 total, 8 up, 8 in 2021-02-10T04:32:17.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:17.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: -02-10T04:32:16.331987+0000 2021-02-10T04:32:17.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: osd.1 (osd.1) 287 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:17.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:17.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: 02-10T04:32:16.331994+0000 2021-02-10T04:32:17.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: osd.1 (osd.1) 288 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:17.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:17.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: 10T04:32:16.332000+0000 osd.1 ( 2021-02-10T04:32:17.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: osd.1) 289 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:17.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: cluster 2021-02-10T04:32:17.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: 2021-02-10T04:32:16.332004+0000 osd.1 (osd.1) 290 : cluster 2021-02-10T04:32:17.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:17.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: cluster 2021-02-10T04:32:16. 2021-02-10T04:32:17.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: 332008+0000 osd.1 (osd.1) 291 : cluster 2021-02-10T04:32:17.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:17.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:17.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: -02-10T04:32:16.332012+0000 osd.1 (osd.1) 2021-02-10T04:32:17.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: 292 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:17.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:32:17.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: 04:32:16.410698+0000 osd.4 (osd 2021-02-10T04:32:17.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: .4) 277 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:17.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:17.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: 10T04:32:16.410707+0000 osd.4 ( 2021-02-10T04:32:17.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: osd.4) 278 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:17.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:17.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: -02-10T04:32:16.410714+0000 2021-02-10T04:32:17.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: osd.4 (osd.4) 279 : cluster [WRN] 2021-02-10T04:32:17.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:17.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: cluster 2021-02-10T04:32:17.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: 2021-02-10T04:32:16.410727+0000 osd.4 ( 2021-02-10T04:32:17.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[29319]: osd.4) 280 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:17.446 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[45443]: debug 2021-02-10T04:32:17.283+0000 7f7140d73700 -1 osd.1 319 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:32:17.446 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[49106]: debug 2021-02-10T04:32:17.168+0000 7f77a3e94700 -1 osd.2 319 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:32:17.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: cluster 2021-02-10T04:32:17.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: 2021-02-10T04:32:16 2021-02-10T04:32:17.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: .201238+0000 osd.2 (osd.2) 320 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:17.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:17.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: 04:32:16.201247+0000 osd.2 (osd.2) 321 : cluster 2021-02-10T04:32:17.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:17.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: cluster 2021-02-10T04:32:17.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: 2021-02-10T04:32:17.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: 2021 2021-02-10T04:32:17.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: - 2021-02-10T04:32:17.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: 02 2021-02-10T04:32:17.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: - 2021-02-10T04:32:17.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: 10 2021-02-10T04:32:17.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: T 2021-02-10T04:32:17.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: 04 2021-02-10T04:32:17.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: : 2021-02-10T04:32:17.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: 32 2021-02-10T04:32:17.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: : 2021-02-10T04:32:17.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: 16.201260+0000 osd.2 (osd.2 2021-02-10T04:32:17.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: ) 322 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:17.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:17.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: 02-10T04:32:16.201267+0000 osd.2 (osd.2) 2021-02-10T04:32:17.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: 323 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:17.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:17.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: -10T04:32:16.201272+0000 osd.2 (osd.2) 324 : 2021-02-10T04:32:17.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:17.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: cluster 2021-02-10T04:32:17.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: 2021-02-10T04:32: 2021-02-10T04:32:17.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: 16.201277 2021-02-10T04:32:17.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 325 : cluster 2021-02-10T04:32:17.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:17.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: cluster 2021-02-10T04:32:17.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: 2021-02-10T04:32:17.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: 2021-02-10T04:32:16 2021-02-10T04:32:17.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: .201287+0000 osd.2 (osd.2) 326 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:17.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: cluster 2021-02-10T04:32:17.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:32:17.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: 04:32:16.201292+0000 osd.2 (osd. 2021-02-10T04:32:17.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: 2) 327 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:17.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:17.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: 02-10T04:32:16.291345+0000 mon.a ( 2021-02-10T04:32:17.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: mon.0) 1069 : cluster [WRN] Health check failed: 8 slow ops, oldest one blocked for 143 sec, osd.2 has slow ops (SLOW_OPS) 2021-02-10T04:32:17.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: cluster 2021-02-10T04:32:16. 2021-02-10T04:32:17.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: 301897+0000 mon.a (mon.0) 1070 : cluster [DBG] 2021-02-10T04:32:17.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: osdmap e319: 8 total, 8 up, 8 in 2021-02-10T04:32:17.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: cluster 2021-02-10T04:32: 2021-02-10T04:32:17.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: 16.331987+0000 osd.1 (osd.1) 287 : cluster 2021-02-10T04:32:17.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:17.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:17.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: -10T04:32:16.331994+0000 osd.1 (osd.1) 288 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:17.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: 2021-02-10T04:32:17.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: cluster 2021-02-10T04:32 2021-02-10T04:32:17.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: :16.332000+0000 osd.1 (osd.1) 2021-02-10T04:32:17.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: 289 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:17.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:17.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: 04:32:16.332004+0000 osd.1 (osd. 2021-02-10T04:32:17.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: 1) 290 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:17.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:17.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: 02-10T04:32:16.332008+0000 2021-02-10T04:32:17.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: osd.1 (osd.1) 291 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:17.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: cluster 2021-02-10T04:32:17.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: 2021-02-10T04:32:16.332012+0000 2021-02-10T04:32:17.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: osd.1 (osd.1) 292 : cluster [WRN] 2021-02-10T04:32:17.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:17.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:17.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: 02-10T04:32:16.410698+0000 osd.4 (osd.4) 277 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:17.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:17.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: -10T04:32:16.410707+0000 osd.4 (osd.4) 278 2021-02-10T04:32:17.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:17.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: cluster 2021-02-10T04:32:17.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: 2021-02-10T04:32:16.410714+0000 2021-02-10T04:32:17.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: osd.4 (osd.4) 279 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:17.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: 2021-02-10T04:32:17.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:17.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: -10T04:32:16.410727+0000 osd.4 (osd.4 2021-02-10T04:32:17.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[35961]: ) 280 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:17.497 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:32:17.529 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:17.614 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: cluster 2021-02-10T04:32:16.201238+0000 osd.2 (osd.2) 320 : cluster 2021-02-10T04:32:17.614 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:17.614 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: cluster 2021-02-10T04:32:16.201247+0000 osd.2 (osd.2) 321 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:17.614 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: cluster 2021-02-10T04:32:16.201260+0000 osd.2 (osd.2) 322 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:17.614 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:32:17.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: :32:16.201267+0000 osd.2 (osd.2) 323 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:17.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: cluster 2021-02-10T04:32:16.201272+0000 2021-02-10T04:32:17.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: osd.2 (osd.2) 324 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:17.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: cluster 2021-02-10T04:32:17.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: 2021-02-10T04:32:16.201277+0000 osd.2 (osd. 2021-02-10T04:32:17.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: 2) 325 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:17.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: cluster 2021-02-10T04:32:17.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: 2021-02-10T04:32:17.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: 2021 2021-02-10T04:32:17.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: - 2021-02-10T04:32:17.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: 02 2021-02-10T04:32:17.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: - 2021-02-10T04:32:17.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: 10 2021-02-10T04:32:17.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: T04:32:16. 2021-02-10T04:32:17.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: 201287+0000 osd.2 (osd.2) 326 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:17.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:17.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: T04:32:16.201292+0000 osd.2 (osd. 2021-02-10T04:32:17.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: 2) 327 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:17.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:17.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: T04:32:16.291345+0000 mon.a (mon. 2021-02-10T04:32:17.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: 0) 1069 : cluster [WRN] Health check failed: 8 slow ops, oldest one blocked for 143 sec, osd.2 has slow ops (SLOW_OPS) 2021-02-10T04:32:17.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:17.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: 04:32:16.301897+0000 mon.a (mon. 2021-02-10T04:32:17.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: 0) 1070 : cluster [DBG] osdmap e319: 8 total, 8 up, 8 in 2021-02-10T04:32:17.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:17.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: -10T04:32:16.331987+0000 osd.1 ( 2021-02-10T04:32:17.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: osd.1) 287 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:17.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:17.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: -02-10T04:32:16.331994+0000 2021-02-10T04:32:17.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: osd.1 (osd.1) 288 : cluster [WRN] 2021-02-10T04:32:17.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:17.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: cluster 2021-02-10T04:32:17.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: 2021-02-10T04:32:16.332000+0000 osd.1 ( 2021-02-10T04:32:17.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: osd.1) 289 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:17.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: cluster 2021-02-10T04:32:16. 2021-02-10T04:32:17.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: 332004+0000 osd.1 (osd.1) 290 : cluster 2021-02-10T04:32:17.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:17.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: cluster 2021-02-10T04:32: 2021-02-10T04:32:17.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: 16.332008+0000 osd.1 (osd.1) 291 : 2021-02-10T04:32:17.618 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:17.618 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:32:17.618 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: 32:16.332012+0000 osd.1 (osd.1) 2021-02-10T04:32:17.618 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: 292 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:17.618 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:17.618 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: 04:32:16.410698+0000 2021-02-10T04:32:17.618 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: osd.4 (osd.4) 277 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:17.618 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:17.618 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: -02-10T04:32:16.410707+0000 2021-02-10T04:32:17.619 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: osd.4 (osd.4) 278 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:17.619 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: 2021-02-10T04:32:17.619 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: cluster 2021-02-10T04:32:17.619 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: 2021-02-10T04:32:16.410714+0000 osd.4 (osd. 2021-02-10T04:32:17.619 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: 4) 279 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:17.619 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:17.619 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: T04:32:16.410727+0000 osd.4 (osd. 2021-02-10T04:32:17.619 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[29288]: 4) 280 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:17.620 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[33677]: debug 2021-02-10T04:32:17.443+0000 7fbec19a1700 -1 osd.4 320 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:32:17.620 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[33677]: debug 2021-02-10T04:32:17.612+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:17.633 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:32:17.739 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:32:17.772 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:17.941 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:17 smithi085 conmon[52836]: debug 2021-02-10T04:32:17.620+0000 7fba6ffae700 -1 osd.3 pg_epoch: 320 pg[4.14( v 63'26 (0'0,63'26] local-lis/les=71/72 n=11 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,2,7] r=0 lpr=71 crt=63'26 lcod 63'25 mlcod 0'0 active+clean] scrubber pg(4.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55fec4876580) 2021-02-10T04:32:18.306 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:32:17 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:32:18.315 INFO:tasks.rados.rados.0.smithi085.stdout:1157: done (14 left) 2021-02-10T04:32:18.315 INFO:tasks.rados.rados.0.smithi085.stdout:1158: write oid 16 current snap is 114 2021-02-10T04:32:18.315 INFO:tasks.rados.rados.0.smithi085.stdout:1156: finishing write tid 3 to smithi08575397-21 2021-02-10T04:32:18.315 INFO:tasks.rados.rados.0.smithi085.stdout:1158: seq_num 432 ranges {460624=623336,1846574=698150,3163674=96950} 2021-02-10T04:32:18.322 INFO:tasks.rados.rados.0.smithi085.stdout:1158: writing smithi08575397-16 from 460624 to 1083960 tid 1 2021-02-10T04:32:18.331 INFO:tasks.rados.rados.0.smithi085.stdout:1158: writing smithi08575397-16 from 1846574 to 2544724 tid 2 2021-02-10T04:32:18.335 INFO:tasks.rados.rados.0.smithi085.stdout:1158: writing smithi08575397-16 from 3163674 to 3260624 tid 3 2021-02-10T04:32:18.336 INFO:tasks.rados.rados.0.smithi085.stdout:1159: snap_remove snap 111 2021-02-10T04:32:18.441 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[45443]: debug 2021-02-10T04:32:18.256+0000 7f7140d73700 -1 osd.1 320 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:32:18.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[49106]: debug 2021-02-10T04:32:18.133+0000 7f77a3e94700 -1 osd.2 320 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:32:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: cluster 2021-02-10T04:32:17.170383+0000 osd.2 (osd 2021-02-10T04:32:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: .2) 328 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: cluster 2021-02-10T04:32:17.170395+0000 osd.2 (osd.2) 329 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: cluster 2021-02-10T04:32:17.170403+0000 2021-02-10T04:32:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: osd.2 (osd.2) 330 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: 2021-02-10T04:32:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: cluster 2021-02-10T04:32:17 2021-02-10T04:32:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: .170410+0000 osd.2 (osd. 2021-02-10T04:32:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: 2) 331 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:18.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: cluster 2021-02-10T04:32:18.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: 2021- 2021-02-10T04:32:18.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: 02-10T04:32:17.170416+0000 2021-02-10T04:32:18.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: osd.2 (osd.2) 332 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:18.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: 2021-02-10T04:32:18.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:18.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: 02-10T04:32:17.170424+0000 osd.2 (osd 2021-02-10T04:32:18.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: .2) 333 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:18.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:18.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: -02- 2021-02-10T04:32:18.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: 10T04:32:17.170430+0000 2021-02-10T04:32:18.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: osd.2 (osd.2) 334 : cluster 2021-02-10T04:32:18.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:18.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:18.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: -10T04:32 2021-02-10T04:32:18.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: :17.170437+0000 osd.2 (osd.2) 335 2021-02-10T04:32:18.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:18.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:18.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: 02-10T04:32:17.284672+0000 osd.1 (osd. 2021-02-10T04:32:18.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: 1) 293 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:18.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:18.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: 02-10T04:32:17.284678+0000 2021-02-10T04:32:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: cluster 2021-02-10T04:32:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: 2021- 2021-02-10T04:32:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: 02-10T04:32:17.170383+0000 osd.2 (osd.2) 328 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: 2021-02-10T04:32:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: 10T04:32:17.170395+0000 osd.2 ( 2021-02-10T04:32:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: osd.2) 329 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: cluster 2021-02-10T04:32:18.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: 2021-02-10T04:32:17.170403+0000 2021-02-10T04:32:18.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: osd.2 (osd.2) 330 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:18.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: 2021-02-10T04:32:18.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: cluster 2021-02-10T04:32: 2021-02-10T04:32:18.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: 17.170410+0000 osd.2 (osd.2) 331 : 2021-02-10T04:32:18.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:18.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: cluster 2021-02-10T04:32:18.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: 2021-02-10T04:32:17.170416 2021-02-10T04:32:18.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 332 : cluster [WRN] 2021-02-10T04:32:18.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:18.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:18.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: 04:32:17.170424+0000 osd.2 (osd. 2021-02-10T04:32:18.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: 2) 333 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:18.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: cluster 2021-02-10T04:32:18.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: 2021-02-10T04:32:17.170430 2021-02-10T04:32:18.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 334 : cluster [WRN] 2021-02-10T04:32:18.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:18.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:18.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: 02-10T04:32:17.170437+0000 2021-02-10T04:32:18.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: osd.2 (osd.2) 335 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:18.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: 2021-02-10T04:32:18.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:18.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: 10T04:32:17.284672+0000 osd.1 ( 2021-02-10T04:32:18.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: osd.1) 293 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:18.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: cluster 2021-02-10T04:32:18.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: 2021-02-10T04:32:18.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: osd.1 (osd.1) 294 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:18.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: 2021-02-10T04:32:18.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: cluster 2021-02-10T04:32:18.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: 2021-02-10T04:32:17.284681+0000 osd.1 2021-02-10T04:32:18.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: (osd.1) 295 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:18.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:18.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: 02-10T04:32:17.284686+0000 2021-02-10T04:32:18.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: osd.1 (osd.1) 296 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:18.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: cluster 2021-02-10T04:32 2021-02-10T04:32:18.454 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: :17.284691+0000 osd.1 (osd.1) 297 2021-02-10T04:32:18.454 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:18.454 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: cluster 2021-02-10T04:32:18.454 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: 2021-02-10T04:32: 2021-02-10T04:32:18.454 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: 17.284703+0000 osd.1 (osd.1) 298 : 2021-02-10T04:32:18.454 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:18.454 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:18.454 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: 10T04:32:17.308939+0000 mon.a ( 2021-02-10T04:32:18.454 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: mon.0) 1071 : cluster [DBG] osdmap e320: 8 total, 8 up, 8 in 2021-02-10T04:32:18.454 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:18.454 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: 02-10T04:32:17.444115+0000 osd.4 2021-02-10T04:32:18.454 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: (osd.4) 281 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:18.455 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: 2021-02-10T04:32:18.455 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: cluster 2021-02-10T04:32:18.455 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: 2021-02-10T04:32:17.444126+0000 osd.4 2021-02-10T04:32:18.455 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: (osd.4) 282 : cluster [WRN] 2021-02-10T04:32:18.455 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:18.455 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:32:18.455 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: 32:17.444146+0000 osd.4 (osd.4) 2021-02-10T04:32:18.455 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: 283 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:18.455 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:32:18.455 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: 04:32:17.444151+0000 osd.4 2021-02-10T04:32:18.455 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: (osd.4) 284 : cluster 2021-02-10T04:32:18.455 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:18.455 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: audit 2021 2021-02-10T04:32:18.456 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: -02-10T04:32:17.895032+0000 mgr.y ( 2021-02-10T04:32:18.456 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: mgr.14140) 475 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:32:18.456 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: cluster 2021-02-10T04:32:18.456 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: 2021-02-10T04:32:18. 2021-02-10T04:32:18.456 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: 054482+0000 mgr.y (mgr.14140) 476 : cluster [DBG] 2021-02-10T04:32:18.456 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[29319]: pgmap v630: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 148 MiB data, 536 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 4.1 MiB/s wr, 37 op/s 2021-02-10T04:32:18.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: 2021-02-10T04:32:17. 2021-02-10T04:32:18.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: 284678+0000 osd.1 (osd.1) 294 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:18.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:18.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: 02-10T04:32:17.284681+0000 osd.1 (osd.1) 295 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:18.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: cluster 2021-02-10T04:32 2021-02-10T04:32:18.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: :17.284686+0000 osd.1 (osd.1) 296 : cluster 2021-02-10T04:32:18.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:18.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:18.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: 04:32:17.284691+0000 osd.1 (osd.1) 297 : cluster 2021-02-10T04:32:18.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:18.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:18.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: 04:32:17.284703+0000 2021-02-10T04:32:18.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: osd.1 (osd.1) 298 : cluster 2021-02-10T04:32:18.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:18.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: cluster 2021-02-10T04:32:18.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: 2021-02-10T04:32:18.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: 2021 2021-02-10T04:32:18.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: - 2021-02-10T04:32:18.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: 02-10T04:32:17. 2021-02-10T04:32:18.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: 308939+0000 mon.a (mon.0) 1071 : cluster [DBG] osdmap e320: 8 total, 8 up, 8 in 2021-02-10T04:32:18.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:18.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: 02-10T04:32:17.444115+0000 2021-02-10T04:32:18.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: osd.4 (osd.4) 281 : cluster 2021-02-10T04:32:18.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:18.460 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:18.460 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: -10T04:32:17.444126+0000 osd.4 (osd.4 2021-02-10T04:32:18.460 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: ) 282 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:18.460 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:18.460 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: 10T04:32:17.444146+0000 osd.4 (osd 2021-02-10T04:32:18.461 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: .4) 283 : cluster [WRN] 2021-02-10T04:32:18.461 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:18.461 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:32:18.461 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: 32:17.444151+0000 osd.4 (osd.4) 2021-02-10T04:32:18.461 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: 284 : cluster 2021-02-10T04:32:18.462 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:18.462 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: audit 2021-02-10T04:32:18.462 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: 2021-02-10T04:32:17. 2021-02-10T04:32:18.462 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: 895032+0000 mgr.y (mgr.14140) 475 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:32:18.462 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: cluster 2021-02-10T04:32:18.054482+0000 mgr.y ( 2021-02-10T04:32:18.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[35961]: mgr.14140) 476 : cluster [DBG] pgmap v630: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 148 MiB data, 536 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 4.1 MiB/s wr, 37 op/s 2021-02-10T04:32:18.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[33677]: debug 2021-02-10T04:32:18.443+0000 7fbec19a1700 -1 osd.4 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:32:18.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: cluster 2021-02-10T04:32:17.170383+0000 osd.2 (osd 2021-02-10T04:32:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: .2) 328 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: cluster 2021-02-10T04:32:17 2021-02-10T04:32:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: .170395+0000 osd.2 (osd.2) 329 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: T04:32:17. 2021-02-10T04:32:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: 170403+0000 osd.2 (osd.2) 330 : cluster 2021-02-10T04:32:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: cluster 2021-02-10T04:32:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: 2021-02-10T04:32:17.170410+0000 2021-02-10T04:32:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: osd.2 (osd.2) 331 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: 2021-02-10T04:32:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: cluster 2021-02-10T04:32 2021-02-10T04:32:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: :17.170416+0000 osd.2 (osd. 2021-02-10T04:32:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: 2) 332 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: -02-10T04:32:17.170424+0000 2021-02-10T04:32:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: osd.2 (osd.2) 333 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: 2021-02-10T04:32:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: 04:32:17.170430+0000 osd.2 (osd. 2021-02-10T04:32:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: 2) 334 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: 2021-02-10T04:32:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:32:18.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: :32:17.170437+0000 osd.2 (osd.2 2021-02-10T04:32:18.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: ) 335 : cluster [WRN] 2021-02-10T04:32:18.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:18.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: cluster 2021-02-10T04:32 2021-02-10T04:32:18.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: :17.284672+0000 osd.1 (osd.1) 293 2021-02-10T04:32:18.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:18.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: cluster 2021-02-10T04:32:18.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: 2021-02-10T04:32:17.284678+0000 2021-02-10T04:32:18.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: osd.1 (osd.1) 294 : cluster [WRN] 2021-02-10T04:32:18.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:18.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: cluster 2021-02-10T04:32:17. 2021-02-10T04:32:18.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: 284681+0000 osd.1 (osd.1) 295 : cluster [WRN] 2021-02-10T04:32:18.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:18.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:18.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: -10T04:32:17.284686+0000 osd.1 ( 2021-02-10T04:32:18.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: osd.1) 296 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:18.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: cluster 2021-02-10T04:32:18.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: 2021-02-10T04:32:17.284691 2021-02-10T04:32:18.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 297 : cluster [WRN] 2021-02-10T04:32:18.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:18.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:32:18.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: 32:17.284703+0000 osd.1 (osd.1) 2021-02-10T04:32:18.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: 298 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:18.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: cluster 2021-02-10T04:32:18.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: 2021-02-10T04:32:17.308939 2021-02-10T04:32:18.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1071 : cluster 2021-02-10T04:32:18.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: [DBG] osdmap e320: 8 total, 8 up, 8 in 2021-02-10T04:32:18.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:18.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: -02-10T04:32:17.444115+0000 osd.4 2021-02-10T04:32:18.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: (osd.4) 281 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:18.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: 2021-02-10T04:32:18.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: cluster 2021-02-10T04:32:17. 2021-02-10T04:32:18.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: 444126+0000 osd.4 (osd.4) 282 : cluster 2021-02-10T04:32:18.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:18.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: cluster 2021-02-10T04:32:18.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: 2021-02-10T04:32:17.444146+0000 2021-02-10T04:32:18.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: osd.4 (osd.4) 283 : cluster [WRN] 2021-02-10T04:32:18.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:18.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:32:18.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: 32:17.444151+0000 osd.4 (osd. 2021-02-10T04:32:18.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: 4) 284 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:18.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: audit 2021-02-10T04:32:18.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: 2021-02-10T04:32:17.895032+0000 2021-02-10T04:32:18.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: mgr.y (mgr.14140) 475 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:32:18.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: 2021-02-10T04:32:18.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: cluster 2021-02-10T04:32:18.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: 2021-02-10T04:32:18.054482+0000 mgr.y 2021-02-10T04:32:18.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:18 smithi164 conmon[29288]: (mgr.14140) 476 : cluster [DBG] pgmap v630: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 148 MiB data, 536 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 4.1 MiB/s wr, 37 op/s 2021-02-10T04:32:19.112 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:32:19.178 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:18 smithi085 conmon[52836]: debug 2021-02-10T04:32:18.829+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:19.329 INFO:tasks.rados.rados.0.smithi085.stdout:1159: done (15 left) 2021-02-10T04:32:19.329 INFO:tasks.rados.rados.0.smithi085.stdout:1160: copy_from oid 13 from oid 22 current snap is 114 2021-02-10T04:32:19.329 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:19.330 INFO:tasks.rados.rados.0.smithi085.stdout:1156: finishing write tid 4 to smithi08575397-21 2021-02-10T04:32:19.330 INFO:tasks.rados.rados.0.smithi085.stdout:1156: finishing write tid 5 to smithi08575397-21 2021-02-10T04:32:19.330 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 611 (ObjNum 431 snap 114 seq_num 431) dirty exists 2021-02-10T04:32:19.330 INFO:tasks.rados.rados.0.smithi085.stdout:1156: left oid 21 (ObjNum 431 snap 114 seq_num 431) 2021-02-10T04:32:19.330 INFO:tasks.rados.rados.0.smithi085.stdout:1158: finishing write tid 1 to smithi08575397-16 2021-02-10T04:32:19.330 INFO:tasks.rados.rados.0.smithi085.stdout:1158: finishing write tid 2 to smithi08575397-16 2021-02-10T04:32:19.330 INFO:tasks.rados.rados.0.smithi085.stdout:1158: finishing write tid 3 to smithi08575397-16 2021-02-10T04:32:19.330 INFO:tasks.rados.rados.0.smithi085.stdout:1158: finishing write tid 4 to smithi08575397-16 2021-02-10T04:32:19.330 INFO:tasks.rados.rados.0.smithi085.stdout:1158: finishing write tid 5 to smithi08575397-16 2021-02-10T04:32:19.330 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 616 (ObjNum 432 snap 114 seq_num 432) dirty exists 2021-02-10T04:32:19.330 INFO:tasks.rados.rados.0.smithi085.stdout:1158: left oid 16 (ObjNum 432 snap 114 seq_num 432) 2021-02-10T04:32:19.330 INFO:tasks.rados.rados.0.smithi085.stdout:1156: done (15 left) 2021-02-10T04:32:19.330 INFO:tasks.rados.rados.0.smithi085.stdout:1158: done (14 left) 2021-02-10T04:32:19.331 INFO:tasks.rados.rados.0.smithi085.stdout:1161: delete oid 5 current snap is 114 2021-02-10T04:32:19.340 INFO:tasks.rados.rados.0.smithi085.stdout:1161: done (14 left) 2021-02-10T04:32:19.340 INFO:tasks.rados.rados.0.smithi085.stdout:1162: snap_create 2021-02-10T04:32:19.349 INFO:tasks.rados.rados.0.smithi085.stdout:1160: finishing copy_from to smithi08575397-13 2021-02-10T04:32:19.350 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 469 (ObjNum 412 snap 107 seq_num 412) dirty exists 2021-02-10T04:32:19.350 INFO:tasks.rados.rados.0.smithi085.stdout:1160: finishing copy_from racing read to smithi08575397-13 2021-02-10T04:32:19.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[49106]: debug 2021-02-10T04:32:19.177+0000 7f77a3e94700 -1 osd.2 321 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:32:19.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: cluster 2021-02-10T04:32:19.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: 2021-02-10T04:32:18.135162+0000 osd.2 (osd.2) 336 2021-02-10T04:32:19.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:19.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:19.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: 02-10T04: 2021-02-10T04:32:19.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: 32:18.135170 2021-02-10T04:32:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 337 : cluster [WRN] 2021-02-10T04:32:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: cluster 2021-02-10T04:32:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:32:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: 04:32:18.135175+0000 osd.2 (osd 2021-02-10T04:32:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: .2) 338 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: cluster 2021-02-10T04:32:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:32:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: 10T04:32:18.135180+0000 osd.2 (osd 2021-02-10T04:32:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: .2) 339 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: -02-10T04:32:18.135185+0000 osd.2 2021-02-10T04:32:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: (osd.2) 340 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: 2021-02-10T04:32:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: cluster 2021-02-10T04:32:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: 2021-02-10T04:32:18 2021-02-10T04:32:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: .135192+0000 osd.2 (osd.2) 341 : cluster 2021-02-10T04:32:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: cluster 2021-02-10T04:32:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:32:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: 32:18.135198+0000 osd.2 (osd. 2021-02-10T04:32:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: 2) 342 : cluster [WRN] 2021-02-10T04:32:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: cluster 2021-02-10T04:32 2021-02-10T04:32:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: :18.135204+0000 osd.2 (osd.2) 343 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: cluster 2021-02-10T04:32 2021-02-10T04:32:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: :18.258438+0000 osd.1 (osd.1) 299 : cluster 2021-02-10T04:32:19.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:19.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:19.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: 04:32:18 2021-02-10T04:32:19.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: .258447+0000 osd.1 (osd.1) 300 : cluster 2021-02-10T04:32:19.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:19.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:19.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: 04:32:18.258453+0000 osd.1 (osd. 2021-02-10T04:32:19.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: 1) 301 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:19.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:19.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: -02-10T04:32:18.258461+0000 2021-02-10T04:32:19.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: osd.1 (osd.1) 302 : cluster [WRN] 2021-02-10T04:32:19.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:19.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:32:19.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: 32:18.258471+0000 osd.1 (osd.1 2021-02-10T04:32:19.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: ) 303 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:19.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:19.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: 02-10T04:32:18.258479+0000 osd.1 2021-02-10T04:32:19.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: (osd.1) 304 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:19.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:19.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: -02-10T04:32:18 2021-02-10T04:32:19.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: .315791+0000 mon.a (mon.0) 1072 : 2021-02-10T04:32:19.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: cluster [DBG] osdmap e321: 8 total, 8 up, 8 in 2021-02-10T04:32:19.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:19.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: -02-10T04:32:18.443771+0000 osd.4 (osd.4) 285 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:19.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: cluster 2021-02-10T04:32 2021-02-10T04:32:19.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: :18.443781+0000 osd.4 (osd.4) 2021-02-10T04:32:19.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: 286 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:19.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:19.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: -10T04:32:18.443788+0000 2021-02-10T04:32:19.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: osd.4 (osd.4) 287 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:19.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:19.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: 04:32:18.443798+0000 osd.4 (osd 2021-02-10T04:32:19.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[35961]: .4) 288 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:19.448 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[45443]: debug 2021-02-10T04:32:19.269+0000 7f7140d73700 -1 osd.1 321 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:32:19.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: cluster 2021-02-10T04:32:19.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:32:19.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: 04:32:18.135162+0000 osd.2 (osd.2) 336 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:19.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: cluster 2021-02-10T04:32:19.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: 2021-02-10T04:32:18.135170+0000 2021-02-10T04:32:19.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: osd.2 (osd.2) 337 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:19.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:19.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: -10T04 2021-02-10T04:32:19.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: :32:18.135175+0000 osd.2 (osd 2021-02-10T04:32:19.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: .2) 338 : cluster [WRN] 2021-02-10T04:32:19.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:19.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:19.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: 02-10T04:32:18.135180+0000 2021-02-10T04:32:19.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: osd.2 (osd.2) 339 : cluster [WRN] 2021-02-10T04:32:19.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:19.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:19.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: -10T04:32:18. 2021-02-10T04:32:19.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: 135185+0000 osd.2 (osd.2) 340 2021-02-10T04:32:19.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:19.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: cluster 2021-02-10T04:32:19.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:32:19.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: :32:18.135192+0000 osd.2 (osd 2021-02-10T04:32:19.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: .2) 341 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:19.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: 2021-02-10T04:32:19.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: cluster 2021-02-10T04:32:19.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:32:19.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: T04:32:18.135198+0000 osd.2 (osd.2) 342 2021-02-10T04:32:19.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:19.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: cluster 2021-02-10T04:32:19.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: 2021-02-10T04:32:18 2021-02-10T04:32:19.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: .135204+0000 osd.2 (osd.2) 343 : cluster [WRN] 2021-02-10T04:32:19.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:19.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: cluster 2021-02-10T04:32:18 2021-02-10T04:32:19.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: .258438+0000 osd.1 (osd.1) 299 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:19.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: 2021-02-10T04:32:19.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: cluster 2021-02-10T04:32:19.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: 2021-02-10T04:32:18.258447+0000 osd.1 (osd.1) 300 : cluster 2021-02-10T04:32:19.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:19.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: cluster 2021-02-10T04:32:19.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: 2021-02-10T04:32: 2021-02-10T04:32:19.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: 18.258453+0000 osd.1 (osd.1) 2021-02-10T04:32:19.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: 301 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:19.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:19.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: -10T04:32:18.258461+0000 2021-02-10T04:32:19.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: osd.1 (osd.1) 302 : cluster [WRN] 2021-02-10T04:32:19.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:19.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: cluster 2021-02-10T04:32:18.258471+0000 osd.1 2021-02-10T04:32:19.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: (osd.1) 303 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:19.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:19.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: -02-10T04:32: 2021-02-10T04:32:19.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: 18.258479+0000 osd.1 (osd 2021-02-10T04:32:19.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: .1) 304 : cluster [WRN] 2021-02-10T04:32:19.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:19.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:19.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: 02-10T04:32:18.315791 2021-02-10T04:32:19.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1072 : cluster 2021-02-10T04:32:19.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: [DBG] osdmap e321: 8 total, 8 up, 8 in 2021-02-10T04:32:19.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: cluster 2021-02-10T04:32 2021-02-10T04:32:19.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: :18.443771+0000 osd.4 (osd.4) 285 : 2021-02-10T04:32:19.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:19.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: cluster 2021-02-10T04:32:19.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: 2021-02-10T04:32:18.443781+0000 osd.4 (osd.4) 286 : cluster 2021-02-10T04:32:19.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:19.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: cluster 2021-02-10T04:32:19.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: 2021-02-10T04:32:18.443788+0000 osd.4 ( 2021-02-10T04:32:19.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: osd.4) 287 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:19.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:19.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: -02-10T04:32:18.443798+0000 osd.4 ( 2021-02-10T04:32:19.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:19 smithi085 conmon[29319]: osd.4) 288 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:19.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[33677]: debug 2021-02-10T04:32:19.442+0000 7fbec19a1700 -1 osd.4 322 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:32:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: cluster 2021-02-10T04:32:18.135162 2021-02-10T04:32:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 336 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: cluster 2021-02-10T04:32:18.135170+0000 osd.2 (osd.2) 337 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: -10T04:32:18.135175+0000 osd.2 (osd 2021-02-10T04:32:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: .2) 338 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: cluster 2021-02-10T04:32:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: 2021-02-10T04:32 2021-02-10T04:32:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: :18.135180+0000 osd.2 (osd.2) 339 2021-02-10T04:32:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: -10T04:32:18.135185+0000 osd.2 ( 2021-02-10T04:32:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: osd.2) 340 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: -02-10T04:32:18.135192+0000 osd.2 (osd. 2021-02-10T04:32:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: 2) 341 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: -02-10T04:32:18.135198+0000 2021-02-10T04:32:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: osd.2 (osd.2) 342 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: 2021-02-10T04:32:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: cluster 2021-02-10T04:32 2021-02-10T04:32:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: :18.135204+0000 osd.2 (osd.2) 343 2021-02-10T04:32:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: -10T04:32:18.258438+0000 osd.1 2021-02-10T04:32:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: (osd.1) 299 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: cluster 2021-02-10T04:32:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: 2021-02-10T04:32:18.258447 2021-02-10T04:32:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 300 : cluster [WRN] 2021-02-10T04:32:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:32:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: 32:18.258453+0000 osd.1 (osd.1) 2021-02-10T04:32:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: 301 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:32:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: 10T04:32:18.258461+0000 osd.1 (osd. 2021-02-10T04:32:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: 1) 302 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: 02-10T04:32:18.258471+0000 osd.1 2021-02-10T04:32:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: (osd.1) 303 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: 2021-02-10T04:32:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: cluster 2021-02-10T04:32:18. 2021-02-10T04:32:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: 258479+0000 osd.1 (osd.1) 304 : cluster 2021-02-10T04:32:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:32:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: 32:18.315791+0000 mon.a (mon.0) 2021-02-10T04:32:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: 1072 : cluster [DBG] osdmap e321: 8 total, 8 up, 8 in 2021-02-10T04:32:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: cluster 2021-02-10T04:32:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: 2021-02-10T04:32: 2021-02-10T04:32:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: 18.443771+0000 osd.4 (osd.4) 285 2021-02-10T04:32:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:32:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: 10T04:32:18.443781+0000 osd.4 ( 2021-02-10T04:32:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: osd.4) 286 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: cluster 2021-02-10T04:32:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: 2021-02-10T04:32 2021-02-10T04:32:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: :18.443788+0000 osd.4 (osd.4) 287 2021-02-10T04:32:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:19.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: 02-10T04:32:18.443798+0000 2021-02-10T04:32:19.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: osd.4 (osd.4) 288 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:19.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:19 smithi164 conmon[29288]: 2021-02-10T04:32:20.343 INFO:tasks.rados.rados.0.smithi085.stdout:1160: done (14 left) 2021-02-10T04:32:20.343 INFO:tasks.rados.rados.0.smithi085.stdout:1162: done (13 left) 2021-02-10T04:32:20.343 INFO:tasks.rados.rados.0.smithi085.stdout:1163: delete oid 2 current snap is 115 2021-02-10T04:32:20.345 INFO:tasks.rados.rados.0.smithi085.stdout:1163: done (13 left) 2021-02-10T04:32:20.345 INFO:tasks.rados.rados.0.smithi085.stdout:1164: read oid 40 snap 108 2021-02-10T04:32:20.346 INFO:tasks.rados.rados.0.smithi085.stdout:1164: expect (ObjNum 389 snap 100 seq_num 389) 2021-02-10T04:32:20.346 INFO:tasks.rados.rados.0.smithi085.stdout:1165: snap_create 2021-02-10T04:32:20.352 INFO:tasks.rados.rados.0.smithi085.stdout:1164: expect (ObjNum 389 snap 100 seq_num 389) 2021-02-10T04:32:20.413 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:32:20.448 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:20.498 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[49106]: debug 2021-02-10T04:32:20.207+0000 7f77a3e94700 -1 osd.2 322 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:32:20.500 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":622,"stamp":"2021-02-10T04:32:12.268536+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":131445880,"num_objects":407,"num_object_clones":7,"num_object_copies":1221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":407,"num_whiteouts":1,"num_read":3945,"num_read_kb":1935958,"num_write":2440,"num_write_kb":615723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":2080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3974,"ondisk_log_size":3974,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":494904,"kb_used_data":315636,"kb_used_omap":0,"kb_used_meta":176320,"kb_avail":749466312,"statfs":{"total":767960285184,"available":767453503488,"internally_reserved":0,"allocated":323211264,"data_stored":334420214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":180551680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,10,8],"upper_bound":524288},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4099485,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":37,"num_read_kb":18228,"num_write":40,"num_write_kb":12274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.178899"},"pg_stats":[{"pgid":"5.9","version":"131'3","reported_seq":"238","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.773270+0000","last_change":"2021-02-10T04:29:07.145921+0000","last_active":"2021-02-10T04:32:07.773270+0000","last_peered":"2021-02-10T04:32:07.773270+0000","last_clean":"2021-02-10T04:32:07.773270+0000","last_became_active":"2021-02-10T04:29:07.145525+0000","last_became_peered":"2021-02-10T04:29:07.145525+0000","last_unstale":"2021-02-10T04:32:07.773270+0000","last_undegraded":"2021-02-10T04:32:07.773270+0000","last_fullsized":"2021-02-10T04:32:07.773270+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:28:48.500425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:48.500425+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"63'30","reported_seq":"335","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.239854+0000","last_change":"2021-02-10T04:28:32.490336+0000","last_active":"2021-02-10T04:32:07.239854+0000","last_peered":"2021-02-10T04:32:07.239854+0000","last_clean":"2021-02-10T04:32:07.239854+0000","last_became_active":"2021-02-10T04:25:49.002058+0000","last_became_peered":"2021-02-10T04:25:49.002058+0000","last_unstale":"2021-02-10T04:32:07.239854+0000","last_undegraded":"2021-02-10T04:32:07.239854+0000","last_fullsized":"2021-02-10T04:32:07.239854+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2021-02-10T04:28:32.490246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:32.490246+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":59,"num_read_kb":42,"num_write":31,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"301","reported_epoch":"312","state":"active+clean","last_fresh":"2021-02-10T04:32:09.251036+0000","last_change":"2021-02-10T04:29:34.787349+0000","last_active":"2021-02-10T04:32:09.251036+0000","last_peered":"2021-02-10T04:32:09.251036+0000","last_clean":"2021-02-10T04:32:09.251036+0000","last_became_active":"2021-02-10T04:25:36.344641+0000","last_became_peered":"2021-02-10T04:25:36.344641+0000","last_unstale":"2021-02-10T04:32:09.251036+0000","last_undegraded":"2021-02-10T04:32:09.251036+0000","last_fullsized":"2021-02-10T04:32:09.251036+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:29:34.787265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:34.787265+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"270","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.240208+0000","last_change":"2021-02-10T04:29:19.405593+0000","last_active":"2021-02-10T04:32:07.240208+0000","last_peered":"2021-02-10T04:32:07.240208+0000","last_clean":"2021-02-10T04:32:07.240208+0000","last_became_active":"2021-02-10T04:25:53.148090+0000","last_became_peered":"2021-02-10T04:25:53.148090+0000","last_unstale":"2021-02-10T04:32:07.240208+0000","last_undegraded":"2021-02-10T04:32:07.240208+0000","last_fullsized":"2021-02-10T04:32:07.240208+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:19.405525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:29:19.405525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"213'180","reported_seq":"865","reported_epoch":"309","state":"active+clean","last_fresh":"2021-02-10T04:32:06.242774+0000","last_change":"2021-02-10T04:32:06.242774+0000","last_active":"2021-02-10T04:32:06.242774+0000","last_peered":"2021-02-10T04:32:06.242774+0000","last_clean":"2021-02-10T04:32:06.242774+0000","last_became_active":"2021-02-10T04:28:25.149288+0000","last_became_peered":"2021-02-10T04:28:25.149288+0000","last_unstale":"2021-02-10T04:32:06.242774+0000","last_undegraded":"2021-02-10T04:32:06.242774+0000","last_fullsized":"2021-02-10T04:32:06.242774+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":180,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13411920,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":114,"num_read_kb":107748,"num_write":88,"num_write_kb":23523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"}]},{"pgid":"4.b","version":"63'20","reported_seq":"316","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.092850+0000","last_change":"2021-02-10T04:28:50.696234+0000","last_active":"2021-02-10T04:32:08.092850+0000","last_peered":"2021-02-10T04:32:08.092850+0000","last_clean":"2021-02-10T04:32:08.092850+0000","last_became_active":"2021-02-10T04:25:49.004346+0000","last_became_peered":"2021-02-10T04:25:49.004346+0000","last_unstale":"2021-02-10T04:32:08.092850+0000","last_undegraded":"2021-02-10T04:32:08.092850+0000","last_fullsized":"2021-02-10T04:32:08.092850+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'20","last_scrub_stamp":"2021-02-10T04:28:50.696124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:50.696124+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":27,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"280","reported_epoch":"309","state":"active+clean","last_fresh":"2021-02-10T04:32:06.242130+0000","last_change":"2021-02-10T04:28:31.102622+0000","last_active":"2021-02-10T04:32:06.242130+0000","last_peered":"2021-02-10T04:32:06.242130+0000","last_clean":"2021-02-10T04:32:06.242130+0000","last_became_active":"2021-02-10T04:25:34.334427+0000","last_became_peered":"2021-02-10T04:25:34.334427+0000","last_unstale":"2021-02-10T04:32:06.242130+0000","last_undegraded":"2021-02-10T04:32:06.242130+0000","last_fullsized":"2021-02-10T04:32:06.242130+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:31.102558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:31.102558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"278","reported_epoch":"312","state":"active+clean","last_fresh":"2021-02-10T04:32:09.254752+0000","last_change":"2021-02-10T04:29:35.805411+0000","last_active":"2021-02-10T04:32:09.254752+0000","last_peered":"2021-02-10T04:32:09.254752+0000","last_clean":"2021-02-10T04:32:09.254752+0000","last_became_active":"2021-02-10T04:25:51.016118+0000","last_became_peered":"2021-02-10T04:25:51.016118+0000","last_unstale":"2021-02-10T04:32:09.254752+0000","last_undegraded":"2021-02-10T04:32:09.254752+0000","last_fullsized":"2021-02-10T04:32:09.254752+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:35.805326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.805326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"273","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.092252+0000","last_change":"2021-02-10T04:28:48.700480+0000","last_active":"2021-02-10T04:32:08.092252+0000","last_peered":"2021-02-10T04:32:08.092252+0000","last_clean":"2021-02-10T04:32:08.092252+0000","last_became_active":"2021-02-10T04:25:53.038249+0000","last_became_peered":"2021-02-10T04:25:53.038249+0000","last_unstale":"2021-02-10T04:32:08.092252+0000","last_undegraded":"2021-02-10T04:32:08.092252+0000","last_fullsized":"2021-02-10T04:32:08.092252+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:48.700418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:48.700418+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"259'74","reported_seq":"977","reported_epoch":"312","state":"active+clean","last_fresh":"2021-02-10T04:32:09.249487+0000","last_change":"2021-02-10T04:32:06.236028+0000","last_active":"2021-02-10T04:32:09.249487+0000","last_peered":"2021-02-10T04:32:09.249487+0000","last_clean":"2021-02-10T04:32:09.249487+0000","last_became_active":"2021-02-10T04:28:25.155557+0000","last_became_peered":"2021-02-10T04:28:25.155557+0000","last_unstale":"2021-02-10T04:32:09.249487+0000","last_undegraded":"2021-02-10T04:32:09.249487+0000","last_fullsized":"2021-02-10T04:32:09.249487+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'62","last_scrub_stamp":"2021-02-10T04:30:10.905905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:10.905905+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2437048,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":48,"num_read_kb":56455,"num_write":38,"num_write_kb":14043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"}]},{"pgid":"4.a","version":"63'29","reported_seq":"330","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.241571+0000","last_change":"2021-02-10T04:29:26.671051+0000","last_active":"2021-02-10T04:32:07.241571+0000","last_peered":"2021-02-10T04:32:07.241571+0000","last_clean":"2021-02-10T04:32:07.241571+0000","last_became_active":"2021-02-10T04:25:49.015913+0000","last_became_peered":"2021-02-10T04:25:49.015913+0000","last_unstale":"2021-02-10T04:32:07.241571+0000","last_undegraded":"2021-02-10T04:32:07.241571+0000","last_fullsized":"2021-02-10T04:32:07.241571+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'29","last_scrub_stamp":"2021-02-10T04:29:26.670950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.670950+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":56,"num_read_kb":36,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"62'2","reported_seq":"359","reported_epoch":"312","state":"active+clean","last_fresh":"2021-02-10T04:32:10.214323+0000","last_change":"2021-02-10T04:28:33.618291+0000","last_active":"2021-02-10T04:32:10.214323+0000","last_peered":"2021-02-10T04:32:10.214323+0000","last_clean":"2021-02-10T04:32:10.214323+0000","last_became_active":"2021-02-10T04:25:34.331199+0000","last_became_peered":"2021-02-10T04:25:34.331199+0000","last_unstale":"2021-02-10T04:32:10.214323+0000","last_undegraded":"2021-02-10T04:32:10.214323+0000","last_fullsized":"2021-02-10T04:32:10.214323+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2021-02-10T04:28:33.618170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:33.618170+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"273","reported_epoch":"312","state":"active+clean","last_fresh":"2021-02-10T04:32:09.250301+0000","last_change":"2021-02-10T04:28:37.645921+0000","last_active":"2021-02-10T04:32:09.250301+0000","last_peered":"2021-02-10T04:32:09.250301+0000","last_clean":"2021-02-10T04:32:09.250301+0000","last_became_active":"2021-02-10T04:25:51.025882+0000","last_became_peered":"2021-02-10T04:25:51.025882+0000","last_unstale":"2021-02-10T04:32:09.250301+0000","last_undegraded":"2021-02-10T04:32:09.250301+0000","last_fullsized":"2021-02-10T04:32:09.250301+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:37.645857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:37.645857+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"199","reported_epoch":"312","state":"active+clean","last_fresh":"2021-02-10T04:32:09.249550+0000","last_change":"2021-02-10T04:29:24.731524+0000","last_active":"2021-02-10T04:32:09.249550+0000","last_peered":"2021-02-10T04:32:09.249550+0000","last_clean":"2021-02-10T04:32:09.249550+0000","last_became_active":"2021-02-10T04:29:07.162878+0000","last_became_peered":"2021-02-10T04:29:07.162878+0000","last_unstale":"2021-02-10T04:32:09.249550+0000","last_undegraded":"2021-02-10T04:32:09.249550+0000","last_fullsized":"2021-02-10T04:32:09.249550+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.731458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.731458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"213'82","reported_seq":"756","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.092361+0000","last_change":"2021-02-10T04:32:06.245812+0000","last_active":"2021-02-10T04:32:08.092361+0000","last_peered":"2021-02-10T04:32:08.092361+0000","last_clean":"2021-02-10T04:32:08.092361+0000","last_became_active":"2021-02-10T04:28:25.150246+0000","last_became_peered":"2021-02-10T04:28:25.150246+0000","last_unstale":"2021-02-10T04:32:08.092361+0000","last_undegraded":"2021-02-10T04:32:08.092361+0000","last_fullsized":"2021-02-10T04:32:08.092361+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'63","last_scrub_stamp":"2021-02-10T04:29:39.700115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:29:39.700115+0000","log_size":82,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7025293,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":13996,"num_write":41,"num_write_kb":22530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"}]},{"pgid":"4.d","version":"113'33","reported_seq":"325","reported_epoch":"313","state":"active+clean","last_fresh":"2021-02-10T04:32:10.259059+0000","last_change":"2021-02-10T04:28:49.515083+0000","last_active":"2021-02-10T04:32:10.259059+0000","last_peered":"2021-02-10T04:32:10.259059+0000","last_clean":"2021-02-10T04:32:10.259059+0000","last_became_active":"2021-02-10T04:25:49.011699+0000","last_became_peered":"2021-02-10T04:25:49.011699+0000","last_unstale":"2021-02-10T04:32:10.259059+0000","last_undegraded":"2021-02-10T04:32:10.259059+0000","last_fullsized":"2021-02-10T04:32:10.259059+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'33","last_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:28:49.514984+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"195","reported_epoch":"313","state":"active+clean","last_fresh":"2021-02-10T04:32:10.257777+0000","last_change":"2021-02-10T04:29:07.146266+0000","last_active":"2021-02-10T04:32:10.257777+0000","last_peered":"2021-02-10T04:32:10.257777+0000","last_clean":"2021-02-10T04:32:10.257777+0000","last_became_active":"2021-02-10T04:29:07.145616+0000","last_became_peered":"2021-02-10T04:29:07.145616+0000","last_unstale":"2021-02-10T04:32:10.257777+0000","last_undegraded":"2021-02-10T04:32:10.257777+0000","last_fullsized":"2021-02-10T04:32:10.257777+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:44.410817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:44.410817+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"271","reported_epoch":"309","state":"active+clean","last_fresh":"2021-02-10T04:32:06.242082+0000","last_change":"2021-02-10T04:29:35.958357+0000","last_active":"2021-02-10T04:32:06.242082+0000","last_peered":"2021-02-10T04:32:06.242082+0000","last_clean":"2021-02-10T04:32:06.242082+0000","last_became_active":"2021-02-10T04:25:51.028302+0000","last_became_peered":"2021-02-10T04:25:51.028302+0000","last_unstale":"2021-02-10T04:32:06.242082+0000","last_undegraded":"2021-02-10T04:32:06.242082+0000","last_fullsized":"2021-02-10T04:32:06.242082+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:35.958255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.958255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"271","reported_epoch":"312","state":"active+clean","last_fresh":"2021-02-10T04:32:09.251279+0000","last_change":"2021-02-10T04:28:32.623869+0000","last_active":"2021-02-10T04:32:09.251279+0000","last_peered":"2021-02-10T04:32:09.251279+0000","last_clean":"2021-02-10T04:32:09.251279+0000","last_became_active":"2021-02-10T04:25:53.038218+0000","last_became_peered":"2021-02-10T04:25:53.038218+0000","last_unstale":"2021-02-10T04:32:09.251279+0000","last_undegraded":"2021-02-10T04:32:09.251279+0000","last_fullsized":"2021-02-10T04:32:09.251279+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:32.623791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:32.623791+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"197'106","reported_seq":"1489","reported_epoch":"312","state":"active+clean+scrubbing","last_fresh":"2021-02-10T04:32:09.250943+0000","last_change":"2021-02-10T04:30:15.066819+0000","last_active":"2021-02-10T04:32:09.250943+0000","last_peered":"2021-02-10T04:32:09.250943+0000","last_clean":"2021-02-10T04:32:09.250943+0000","last_became_active":"2021-02-10T04:28:25.157931+0000","last_became_peered":"2021-02-10T04:28:25.157931+0000","last_unstale":"2021-02-10T04:32:09.250943+0000","last_undegraded":"2021-02-10T04:32:09.250943+0000","last_fullsized":"2021-02-10T04:32:09.250943+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":106,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":58,"stat_sum":{"num_bytes":12437781,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":69,"num_read_kb":61326,"num_write":71,"num_write_kb":20689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"213","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.239900+0000","last_change":"2021-02-10T04:29:37.336179+0000","last_active":"2021-02-10T04:32:07.239900+0000","last_peered":"2021-02-10T04:32:07.239900+0000","last_clean":"2021-02-10T04:32:07.239900+0000","last_became_active":"2021-02-10T04:28:49.046469+0000","last_became_peered":"2021-02-10T04:28:49.046469+0000","last_unstale":"2021-02-10T04:32:07.239900+0000","last_undegraded":"2021-02-10T04:32:07.239900+0000","last_fullsized":"2021-02-10T04:32:07.239900+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:37.336092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:37.336092+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"283","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.093592+0000","last_change":"2021-02-10T04:29:07.160624+0000","last_active":"2021-02-10T04:32:08.093592+0000","last_peered":"2021-02-10T04:32:08.093592+0000","last_clean":"2021-02-10T04:32:08.093592+0000","last_became_active":"2021-02-10T04:29:07.159274+0000","last_became_peered":"2021-02-10T04:29:07.159274+0000","last_unstale":"2021-02-10T04:32:08.093592+0000","last_undegraded":"2021-02-10T04:32:08.093592+0000","last_fullsized":"2021-02-10T04:32:08.093592+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:30.424326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:28:30.424326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"62'13","reported_seq":"298","reported_epoch":"313","state":"active+clean","last_fresh":"2021-02-10T04:32:10.258258+0000","last_change":"2021-02-10T04:28:41.427119+0000","last_active":"2021-02-10T04:32:10.258258+0000","last_peered":"2021-02-10T04:32:10.258258+0000","last_clean":"2021-02-10T04:32:10.258258+0000","last_became_active":"2021-02-10T04:25:49.001725+0000","last_became_peered":"2021-02-10T04:25:49.001725+0000","last_unstale":"2021-02-10T04:32:10.258258+0000","last_undegraded":"2021-02-10T04:32:10.258258+0000","last_fullsized":"2021-02-10T04:32:10.258258+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'13","last_scrub_stamp":"2021-02-10T04:28:41.427050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:41.427050+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"282","reported_epoch":"309","state":"active+clean","last_fresh":"2021-02-10T04:32:06.238691+0000","last_change":"2021-02-10T04:29:18.107974+0000","last_active":"2021-02-10T04:32:06.238691+0000","last_peered":"2021-02-10T04:32:06.238691+0000","last_clean":"2021-02-10T04:32:06.238691+0000","last_became_active":"2021-02-10T04:25:36.333883+0000","last_became_peered":"2021-02-10T04:25:36.333883+0000","last_unstale":"2021-02-10T04:32:06.238691+0000","last_undegraded":"2021-02-10T04:32:06.238691+0000","last_fullsized":"2021-02-10T04:32:06.238691+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:18.107867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:18.107867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"58'2","reported_seq":"351","reported_epoch":"312","state":"active+clean","last_fresh":"2021-02-10T04:32:09.251641+0000","last_change":"2021-02-10T04:28:35.648670+0000","last_active":"2021-02-10T04:32:09.251641+0000","last_peered":"2021-02-10T04:32:09.251641+0000","last_clean":"2021-02-10T04:32:09.251641+0000","last_became_active":"2021-02-10T04:25:51.025268+0000","last_became_peered":"2021-02-10T04:25:51.025268+0000","last_unstale":"2021-02-10T04:32:09.251641+0000","last_undegraded":"2021-02-10T04:32:09.251641+0000","last_fullsized":"2021-02-10T04:32:09.251641+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:28:35.648602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:35.648602+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"273","reported_epoch":"313","state":"active+clean","last_fresh":"2021-02-10T04:32:10.259760+0000","last_change":"2021-02-10T04:29:38.873167+0000","last_active":"2021-02-10T04:32:10.259760+0000","last_peered":"2021-02-10T04:32:10.259760+0000","last_clean":"2021-02-10T04:32:10.259760+0000","last_became_active":"2021-02-10T04:25:53.043027+0000","last_became_peered":"2021-02-10T04:25:53.043027+0000","last_unstale":"2021-02-10T04:32:10.259760+0000","last_undegraded":"2021-02-10T04:32:10.259760+0000","last_fullsized":"2021-02-10T04:32:10.259760+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:38.873102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.873102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"300'148","reported_seq":"2926","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.101500+0000","last_change":"2021-02-10T04:32:06.251515+0000","last_active":"2021-02-10T04:32:08.101500+0000","last_peered":"2021-02-10T04:32:08.101500+0000","last_clean":"2021-02-10T04:32:08.101500+0000","last_became_active":"2021-02-10T04:28:25.149226+0000","last_became_peered":"2021-02-10T04:28:25.149226+0000","last_unstale":"2021-02-10T04:32:08.101500+0000","last_undegraded":"2021-02-10T04:32:08.101500+0000","last_fullsized":"2021-02-10T04:32:08.101500+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":148,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8588410,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":168,"num_read_kb":185250,"num_write":80,"num_write_kb":30825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"}]},{"pgid":"1.a","version":"0'0","reported_seq":"213","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.240062+0000","last_change":"2021-02-10T04:29:29.384240+0000","last_active":"2021-02-10T04:32:07.240062+0000","last_peered":"2021-02-10T04:32:07.240062+0000","last_clean":"2021-02-10T04:32:07.240062+0000","last_became_active":"2021-02-10T04:28:49.050935+0000","last_became_peered":"2021-02-10T04:28:49.050935+0000","last_unstale":"2021-02-10T04:32:07.240062+0000","last_undegraded":"2021-02-10T04:32:07.240062+0000","last_fullsized":"2021-02-10T04:32:07.240062+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.384170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.384170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"62'17","reported_seq":"302","reported_epoch":"309","state":"active+clean","last_fresh":"2021-02-10T04:32:06.238761+0000","last_change":"2021-02-10T04:29:37.933295+0000","last_active":"2021-02-10T04:32:06.238761+0000","last_peered":"2021-02-10T04:32:06.238761+0000","last_clean":"2021-02-10T04:32:06.238761+0000","last_became_active":"2021-02-10T04:25:49.004933+0000","last_became_peered":"2021-02-10T04:25:49.004933+0000","last_unstale":"2021-02-10T04:32:06.238761+0000","last_undegraded":"2021-02-10T04:32:06.238761+0000","last_fullsized":"2021-02-10T04:32:06.238761+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'17","last_scrub_stamp":"2021-02-10T04:29:37.933206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:37.933206+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"298","reported_epoch":"309","state":"active+clean","last_fresh":"2021-02-10T04:32:06.237945+0000","last_change":"2021-02-10T04:28:39.128857+0000","last_active":"2021-02-10T04:32:06.237945+0000","last_peered":"2021-02-10T04:32:06.237945+0000","last_clean":"2021-02-10T04:32:06.237945+0000","last_became_active":"2021-02-10T04:28:21.130021+0000","last_became_peered":"2021-02-10T04:28:21.130021+0000","last_unstale":"2021-02-10T04:32:06.237945+0000","last_undegraded":"2021-02-10T04:32:06.237945+0000","last_fullsized":"2021-02-10T04:32:06.237945+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_clean_scrub_stamp":"2021-02-10T04:28:39.128781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"78'3","reported_seq":"354","reported_epoch":"313","state":"active+clean","last_fresh":"2021-02-10T04:32:10.259674+0000","last_change":"2021-02-10T04:28:43.509446+0000","last_active":"2021-02-10T04:32:10.259674+0000","last_peered":"2021-02-10T04:32:10.259674+0000","last_clean":"2021-02-10T04:32:10.259674+0000","last_became_active":"2021-02-10T04:25:51.025218+0000","last_became_peered":"2021-02-10T04:25:51.025218+0000","last_unstale":"2021-02-10T04:32:10.259674+0000","last_undegraded":"2021-02-10T04:32:10.259674+0000","last_fullsized":"2021-02-10T04:32:10.259674+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:28:43.509353+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:28:43.509353+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"270","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.240716+0000","last_change":"2021-02-10T04:28:45.481538+0000","last_active":"2021-02-10T04:32:07.240716+0000","last_peered":"2021-02-10T04:32:07.240716+0000","last_clean":"2021-02-10T04:32:07.240716+0000","last_became_active":"2021-02-10T04:25:53.042697+0000","last_became_peered":"2021-02-10T04:25:53.042697+0000","last_unstale":"2021-02-10T04:32:07.240716+0000","last_undegraded":"2021-02-10T04:32:07.240716+0000","last_fullsized":"2021-02-10T04:32:07.240716+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:45.481472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:28:45.481472+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"311'179","reported_seq":"1871","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.179370+0000","last_change":"2021-02-10T04:32:07.182650+0000","last_active":"2021-02-10T04:32:08.179370+0000","last_peered":"2021-02-10T04:32:08.179370+0000","last_clean":"2021-02-10T04:32:08.179370+0000","last_became_active":"2021-02-10T04:28:25.148500+0000","last_became_peered":"2021-02-10T04:28:25.148500+0000","last_unstale":"2021-02-10T04:32:08.179370+0000","last_undegraded":"2021-02-10T04:32:08.179370+0000","last_fullsized":"2021-02-10T04:32:08.179370+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'110","last_scrub_stamp":"2021-02-10T04:30:18.463782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:18.463782+0000","log_size":179,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10458320,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":159,"num_read_kb":176997,"num_write":103,"num_write_kb":38375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"}]},{"pgid":"4.e","version":"86'31","reported_seq":"329","reported_epoch":"313","state":"active+clean","last_fresh":"2021-02-10T04:32:10.259442+0000","last_change":"2021-02-10T04:28:11.693702+0000","last_active":"2021-02-10T04:32:10.259442+0000","last_peered":"2021-02-10T04:32:10.259442+0000","last_clean":"2021-02-10T04:32:10.259442+0000","last_became_active":"2021-02-10T04:25:49.011093+0000","last_became_peered":"2021-02-10T04:25:49.011093+0000","last_unstale":"2021-02-10T04:32:10.259442+0000","last_undegraded":"2021-02-10T04:32:10.259442+0000","last_fullsized":"2021-02-10T04:32:10.259442+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'31","last_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:28:11.693616+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":28,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"256","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.091271+0000","last_change":"2021-02-10T04:29:31.683314+0000","last_active":"2021-02-10T04:32:08.091271+0000","last_peered":"2021-02-10T04:32:08.091271+0000","last_clean":"2021-02-10T04:32:08.091271+0000","last_became_active":"2021-02-10T04:29:07.160593+0000","last_became_peered":"2021-02-10T04:29:07.160593+0000","last_unstale":"2021-02-10T04:32:08.091271+0000","last_undegraded":"2021-02-10T04:32:08.091271+0000","last_fullsized":"2021-02-10T04:32:08.091271+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:31.683213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.683213+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"277","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.241017+0000","last_change":"2021-02-10T04:30:10.487631+0000","last_active":"2021-02-10T04:32:07.241017+0000","last_peered":"2021-02-10T04:32:07.241017+0000","last_clean":"2021-02-10T04:32:07.241017+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:32:07.241017+0000","last_undegraded":"2021-02-10T04:32:07.241017+0000","last_fullsized":"2021-02-10T04:32:07.241017+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:10.487528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:30:10.487528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"265","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.098832+0000","last_change":"2021-02-10T04:27:52.383262+0000","last_active":"2021-02-10T04:32:08.098832+0000","last_peered":"2021-02-10T04:32:08.098832+0000","last_clean":"2021-02-10T04:32:08.098832+0000","last_became_active":"2021-02-10T04:25:53.148054+0000","last_became_peered":"2021-02-10T04:25:53.148054+0000","last_unstale":"2021-02-10T04:32:08.098832+0000","last_undegraded":"2021-02-10T04:32:08.098832+0000","last_fullsized":"2021-02-10T04:32:08.098832+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:52.383188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:52.383188+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"311'223","reported_seq":"1039","reported_epoch":"311","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:32:08.230540+0000","last_change":"2021-02-10T04:32:07.238094+0000","last_active":"2021-02-10T04:32:08.230540+0000","last_peered":"2021-02-10T04:32:08.230540+0000","last_clean":"2021-02-10T04:32:08.230540+0000","last_became_active":"2021-02-10T04:28:25.152030+0000","last_became_peered":"2021-02-10T04:28:25.152030+0000","last_unstale":"2021-02-10T04:32:08.230540+0000","last_undegraded":"2021-02-10T04:32:08.230540+0000","last_fullsized":"2021-02-10T04:32:08.230540+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":223,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":12605382,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":153,"num_read_kb":140535,"num_write":113,"num_write_kb":53704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"}]},{"pgid":"5.0","version":"158'3","reported_seq":"351","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.094024+0000","last_change":"2021-02-10T04:29:34.388978+0000","last_active":"2021-02-10T04:32:08.094024+0000","last_peered":"2021-02-10T04:32:08.094024+0000","last_clean":"2021-02-10T04:32:08.094024+0000","last_became_active":"2021-02-10T04:25:51.031906+0000","last_became_peered":"2021-02-10T04:25:51.031906+0000","last_unstale":"2021-02-10T04:32:08.094024+0000","last_undegraded":"2021-02-10T04:32:08.094024+0000","last_fullsized":"2021-02-10T04:32:08.094024+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'3","last_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_deep_scrub":"158'3","last_deep_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_clean_scrub_stamp":"2021-02-10T04:29:34.388912+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"63'26","reported_seq":"325","reported_epoch":"313","state":"active+clean","last_fresh":"2021-02-10T04:32:10.257505+0000","last_change":"2021-02-10T04:29:36.813190+0000","last_active":"2021-02-10T04:32:10.257505+0000","last_peered":"2021-02-10T04:32:10.257505+0000","last_clean":"2021-02-10T04:32:10.257505+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:32:10.257505+0000","last_undegraded":"2021-02-10T04:32:10.257505+0000","last_fullsized":"2021-02-10T04:32:10.257505+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:29:36.813083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.813083+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":51,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"313","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.242854+0000","last_change":"2021-02-10T04:29:28.632178+0000","last_active":"2021-02-10T04:32:07.242854+0000","last_peered":"2021-02-10T04:32:07.242854+0000","last_clean":"2021-02-10T04:32:07.242854+0000","last_became_active":"2021-02-10T04:28:39.041833+0000","last_became_peered":"2021-02-10T04:28:39.041833+0000","last_unstale":"2021-02-10T04:32:07.242854+0000","last_undegraded":"2021-02-10T04:32:07.242854+0000","last_fullsized":"2021-02-10T04:32:07.242854+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.632085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.632085+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"197","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.242084+0000","last_change":"2021-02-10T04:29:24.658465+0000","last_active":"2021-02-10T04:32:07.242084+0000","last_peered":"2021-02-10T04:32:07.242084+0000","last_clean":"2021-02-10T04:32:07.242084+0000","last_became_active":"2021-02-10T04:29:07.148527+0000","last_became_peered":"2021-02-10T04:29:07.148527+0000","last_unstale":"2021-02-10T04:32:07.242084+0000","last_undegraded":"2021-02-10T04:32:07.242084+0000","last_fullsized":"2021-02-10T04:32:07.242084+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.658374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.658374+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"309'176","reported_seq":"1543","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.095577+0000","last_change":"2021-02-10T04:32:06.239067+0000","last_active":"2021-02-10T04:32:08.095577+0000","last_peered":"2021-02-10T04:32:08.095577+0000","last_clean":"2021-02-10T04:32:08.095577+0000","last_became_active":"2021-02-10T04:28:25.157306+0000","last_became_peered":"2021-02-10T04:28:25.157306+0000","last_unstale":"2021-02-10T04:32:08.095577+0000","last_undegraded":"2021-02-10T04:32:08.095577+0000","last_fullsized":"2021-02-10T04:32:08.095577+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":176,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5210302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":166,"num_read_kb":152003,"num_write":95,"num_write_kb":22981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":"214","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.098916+0000","last_change":"2021-02-10T04:30:12.452072+0000","last_active":"2021-02-10T04:32:08.098916+0000","last_peered":"2021-02-10T04:32:08.098916+0000","last_clean":"2021-02-10T04:32:08.098916+0000","last_became_active":"2021-02-10T04:28:49.048525+0000","last_became_peered":"2021-02-10T04:28:49.048525+0000","last_unstale":"2021-02-10T04:32:08.098916+0000","last_undegraded":"2021-02-10T04:32:08.098916+0000","last_fullsized":"2021-02-10T04:32:08.098916+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:12.452000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:30:12.452000+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"275","reported_epoch":"313","state":"active+clean","last_fresh":"2021-02-10T04:32:10.257841+0000","last_change":"2021-02-10T04:29:40.859352+0000","last_active":"2021-02-10T04:32:10.257841+0000","last_peered":"2021-02-10T04:32:10.257841+0000","last_clean":"2021-02-10T04:32:10.257841+0000","last_became_active":"2021-02-10T04:25:51.018706+0000","last_became_peered":"2021-02-10T04:25:51.018706+0000","last_unstale":"2021-02-10T04:32:10.257841+0000","last_undegraded":"2021-02-10T04:32:10.257841+0000","last_fullsized":"2021-02-10T04:32:10.257841+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:40.859274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:40.859274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"152'48","reported_seq":"358","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.093938+0000","last_change":"2021-02-10T04:29:28.398522+0000","last_active":"2021-02-10T04:32:08.093938+0000","last_peered":"2021-02-10T04:32:08.093938+0000","last_clean":"2021-02-10T04:32:08.093938+0000","last_became_active":"2021-02-10T04:25:48.997481+0000","last_became_peered":"2021-02-10T04:25:48.997481+0000","last_unstale":"2021-02-10T04:32:08.093938+0000","last_undegraded":"2021-02-10T04:32:08.093938+0000","last_fullsized":"2021-02-10T04:32:08.093938+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'48","last_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_deep_scrub":"152'48","last_deep_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.398429+0000","log_size":48,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":70,"num_read_kb":48,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"294","reported_epoch":"309","state":"active+clean","last_fresh":"2021-02-10T04:32:06.238696+0000","last_change":"2021-02-10T04:28:35.010774+0000","last_active":"2021-02-10T04:32:06.238696+0000","last_peered":"2021-02-10T04:32:06.238696+0000","last_clean":"2021-02-10T04:32:06.238696+0000","last_became_active":"2021-02-10T04:28:21.133248+0000","last_became_peered":"2021-02-10T04:28:21.133248+0000","last_unstale":"2021-02-10T04:32:06.238696+0000","last_undegraded":"2021-02-10T04:32:06.238696+0000","last_fullsized":"2021-02-10T04:32:06.238696+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:28:35.010709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:35.010709+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"273","reported_epoch":"313","state":"active+clean","last_fresh":"2021-02-10T04:32:10.257243+0000","last_change":"2021-02-10T04:28:05.641668+0000","last_active":"2021-02-10T04:32:10.257243+0000","last_peered":"2021-02-10T04:32:10.257243+0000","last_clean":"2021-02-10T04:32:10.257243+0000","last_became_active":"2021-02-10T04:25:53.045085+0000","last_became_peered":"2021-02-10T04:25:53.045085+0000","last_unstale":"2021-02-10T04:32:10.257243+0000","last_undegraded":"2021-02-10T04:32:10.257243+0000","last_fullsized":"2021-02-10T04:32:10.257243+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:05.641577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:05.641577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"311'374","reported_seq":"1161","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.303669+0000","last_change":"2021-02-10T04:32:06.239542+0000","last_active":"2021-02-10T04:32:08.303669+0000","last_peered":"2021-02-10T04:32:08.303669+0000","last_clean":"2021-02-10T04:32:08.303669+0000","last_became_active":"2021-02-10T04:28:25.161622+0000","last_became_peered":"2021-02-10T04:28:25.161622+0000","last_unstale":"2021-02-10T04:32:08.303669+0000","last_undegraded":"2021-02-10T04:32:08.303669+0000","last_fullsized":"2021-02-10T04:32:08.303669+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":374,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10252501,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":262,"num_read_kb":277914,"num_write":192,"num_write_kb":61297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":"216","reported_epoch":"313","state":"active+clean","last_fresh":"2021-02-10T04:32:10.258488+0000","last_change":"2021-02-10T04:29:33.839409+0000","last_active":"2021-02-10T04:32:10.258488+0000","last_peered":"2021-02-10T04:32:10.258488+0000","last_clean":"2021-02-10T04:32:10.258488+0000","last_became_active":"2021-02-10T04:28:49.045505+0000","last_became_peered":"2021-02-10T04:28:49.045505+0000","last_unstale":"2021-02-10T04:32:10.258488+0000","last_undegraded":"2021-02-10T04:32:10.258488+0000","last_fullsized":"2021-02-10T04:32:10.258488+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:33.839335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:33.839335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"304","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.777834+0000","last_change":"2021-02-10T04:32:08.778589+0000","last_active":"2021-02-10T04:32:08.777834+0000","last_peered":"2021-02-10T04:32:08.777834+0000","last_clean":"2021-02-10T04:32:08.777834+0000","last_became_active":"2021-02-10T04:25:51.025636+0000","last_became_peered":"2021-02-10T04:25:51.025636+0000","last_unstale":"2021-02-10T04:32:08.777834+0000","last_undegraded":"2021-02-10T04:32:08.777834+0000","last_fullsized":"2021-02-10T04:32:08.777834+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:37.215333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:37.215333+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"63'27","reported_seq":"330","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.092650+0000","last_change":"2021-02-10T04:28:36.754064+0000","last_active":"2021-02-10T04:32:08.092650+0000","last_peered":"2021-02-10T04:32:08.092650+0000","last_clean":"2021-02-10T04:32:08.092650+0000","last_became_active":"2021-02-10T04:25:49.002937+0000","last_became_peered":"2021-02-10T04:25:49.002937+0000","last_unstale":"2021-02-10T04:32:08.092650+0000","last_undegraded":"2021-02-10T04:32:08.092650+0000","last_fullsized":"2021-02-10T04:32:08.092650+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'27","last_scrub_stamp":"2021-02-10T04:28:36.753994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:36.753994+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":38,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"209","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.094133+0000","last_change":"2021-02-10T04:29:35.711635+0000","last_active":"2021-02-10T04:32:08.094133+0000","last_peered":"2021-02-10T04:32:08.094133+0000","last_clean":"2021-02-10T04:32:08.094133+0000","last_became_active":"2021-02-10T04:29:07.161341+0000","last_became_peered":"2021-02-10T04:29:07.161341+0000","last_unstale":"2021-02-10T04:32:08.094133+0000","last_undegraded":"2021-02-10T04:32:08.094133+0000","last_fullsized":"2021-02-10T04:32:08.094133+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_deep_scrub":"53'2","last_deep_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.711557+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"269","reported_epoch":"309","state":"active+clean","last_fresh":"2021-02-10T04:32:06.238028+0000","last_change":"2021-02-10T04:29:26.978134+0000","last_active":"2021-02-10T04:32:06.238028+0000","last_peered":"2021-02-10T04:32:06.238028+0000","last_clean":"2021-02-10T04:32:06.238028+0000","last_became_active":"2021-02-10T04:25:53.149552+0000","last_became_peered":"2021-02-10T04:25:53.149552+0000","last_unstale":"2021-02-10T04:32:06.238028+0000","last_undegraded":"2021-02-10T04:32:06.238028+0000","last_fullsized":"2021-02-10T04:32:06.238028+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:26.978062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.978062+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"309'254","reported_seq":"940","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.240153+0000","last_change":"2021-02-10T04:32:06.240170+0000","last_active":"2021-02-10T04:32:07.240153+0000","last_peered":"2021-02-10T04:32:07.240153+0000","last_clean":"2021-02-10T04:32:07.240153+0000","last_became_active":"2021-02-10T04:28:25.146766+0000","last_became_peered":"2021-02-10T04:28:25.146766+0000","last_unstale":"2021-02-10T04:32:07.240153+0000","last_undegraded":"2021-02-10T04:32:07.240153+0000","last_fullsized":"2021-02-10T04:32:07.240153+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":254,"ondisk_log_size":254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5327951,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":135,"num_read_kb":110109,"num_write":128,"num_write_kb":52680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":"294","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.242621+0000","last_change":"2021-02-10T04:29:38.793837+0000","last_active":"2021-02-10T04:32:07.242621+0000","last_peered":"2021-02-10T04:32:07.242621+0000","last_clean":"2021-02-10T04:32:07.242621+0000","last_became_active":"2021-02-10T04:28:49.054638+0000","last_became_peered":"2021-02-10T04:28:49.054638+0000","last_unstale":"2021-02-10T04:32:07.242621+0000","last_undegraded":"2021-02-10T04:32:07.242621+0000","last_fullsized":"2021-02-10T04:32:07.242621+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:38.793760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.793760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"272","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.241090+0000","last_change":"2021-02-10T04:29:40.451948+0000","last_active":"2021-02-10T04:32:07.241090+0000","last_peered":"2021-02-10T04:32:07.241090+0000","last_clean":"2021-02-10T04:32:07.241090+0000","last_became_active":"2021-02-10T04:25:51.032201+0000","last_became_peered":"2021-02-10T04:25:51.032201+0000","last_unstale":"2021-02-10T04:32:07.241090+0000","last_undegraded":"2021-02-10T04:32:07.241090+0000","last_fullsized":"2021-02-10T04:32:07.241090+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:40.451832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:29:40.451832+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"62'16","reported_seq":"301","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.100679+0000","last_change":"2021-02-10T04:28:34.694585+0000","last_active":"2021-02-10T04:32:08.100679+0000","last_peered":"2021-02-10T04:32:08.100679+0000","last_clean":"2021-02-10T04:32:08.100679+0000","last_became_active":"2021-02-10T04:25:49.003521+0000","last_became_peered":"2021-02-10T04:25:49.003521+0000","last_unstale":"2021-02-10T04:32:08.100679+0000","last_undegraded":"2021-02-10T04:32:08.100679+0000","last_fullsized":"2021-02-10T04:32:08.100679+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'16","last_scrub_stamp":"2021-02-10T04:28:34.694486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:34.694486+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":16,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"249","reported_epoch":"309","state":"active+clean","last_fresh":"2021-02-10T04:32:06.238083+0000","last_change":"2021-02-10T04:30:14.204629+0000","last_active":"2021-02-10T04:32:06.238083+0000","last_peered":"2021-02-10T04:32:06.238083+0000","last_clean":"2021-02-10T04:32:06.238083+0000","last_became_active":"2021-02-10T04:29:07.159327+0000","last_became_peered":"2021-02-10T04:29:07.159327+0000","last_unstale":"2021-02-10T04:32:06.238083+0000","last_undegraded":"2021-02-10T04:32:06.238083+0000","last_fullsized":"2021-02-10T04:32:06.238083+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:14.204533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:30:14.204533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"275","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.092052+0000","last_change":"2021-02-10T04:29:21.620573+0000","last_active":"2021-02-10T04:32:08.092052+0000","last_peered":"2021-02-10T04:32:08.092052+0000","last_clean":"2021-02-10T04:32:08.092052+0000","last_became_active":"2021-02-10T04:25:49.001782+0000","last_became_peered":"2021-02-10T04:25:49.001782+0000","last_unstale":"2021-02-10T04:32:08.092052+0000","last_undegraded":"2021-02-10T04:32:08.092052+0000","last_fullsized":"2021-02-10T04:32:08.092052+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:21.620482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:21.620482+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"269","reported_epoch":"309","state":"active+clean","last_fresh":"2021-02-10T04:32:06.242136+0000","last_change":"2021-02-10T04:29:23.098781+0000","last_active":"2021-02-10T04:32:06.242136+0000","last_peered":"2021-02-10T04:32:06.242136+0000","last_clean":"2021-02-10T04:32:06.242136+0000","last_became_active":"2021-02-10T04:25:53.040390+0000","last_became_peered":"2021-02-10T04:25:53.040390+0000","last_unstale":"2021-02-10T04:32:06.242136+0000","last_undegraded":"2021-02-10T04:32:06.242136+0000","last_fullsized":"2021-02-10T04:32:06.242136+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.098681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.098681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"215'85","reported_seq":"761","reported_epoch":"312","state":"active+clean","last_fresh":"2021-02-10T04:32:09.251524+0000","last_change":"2021-02-10T04:32:06.240226+0000","last_active":"2021-02-10T04:32:09.251524+0000","last_peered":"2021-02-10T04:32:09.251524+0000","last_clean":"2021-02-10T04:32:09.251524+0000","last_became_active":"2021-02-10T04:29:07.163659+0000","last_became_peered":"2021-02-10T04:29:07.163659+0000","last_unstale":"2021-02-10T04:32:09.251524+0000","last_undegraded":"2021-02-10T04:32:09.251524+0000","last_fullsized":"2021-02-10T04:32:09.251524+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2201750,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":44,"num_read_kb":45112,"num_write":42,"num_write_kb":10011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"}]},{"pgid":"1.3","version":"0'0","reported_seq":"212","reported_epoch":"309","state":"active+clean","last_fresh":"2021-02-10T04:32:06.241707+0000","last_change":"2021-02-10T04:29:28.928994+0000","last_active":"2021-02-10T04:32:06.241707+0000","last_peered":"2021-02-10T04:32:06.241707+0000","last_clean":"2021-02-10T04:32:06.241707+0000","last_became_active":"2021-02-10T04:28:49.043854+0000","last_became_peered":"2021-02-10T04:28:49.043854+0000","last_unstale":"2021-02-10T04:32:06.241707+0000","last_undegraded":"2021-02-10T04:32:06.241707+0000","last_fullsized":"2021-02-10T04:32:06.241707+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.928906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.928906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"199","reported_epoch":"312","state":"active+clean","last_fresh":"2021-02-10T04:32:09.250559+0000","last_change":"2021-02-10T04:29:29.805554+0000","last_active":"2021-02-10T04:32:09.250559+0000","last_peered":"2021-02-10T04:32:09.250559+0000","last_clean":"2021-02-10T04:32:09.250559+0000","last_became_active":"2021-02-10T04:29:07.148512+0000","last_became_peered":"2021-02-10T04:29:07.148512+0000","last_unstale":"2021-02-10T04:32:09.250559+0000","last_undegraded":"2021-02-10T04:32:09.250559+0000","last_fullsized":"2021-02-10T04:32:09.250559+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.805432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.805432+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"62'15","reported_seq":"298","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.242614+0000","last_change":"2021-02-10T04:28:40.322577+0000","last_active":"2021-02-10T04:32:07.242614+0000","last_peered":"2021-02-10T04:32:07.242614+0000","last_clean":"2021-02-10T04:32:07.242614+0000","last_became_active":"2021-02-10T04:25:49.002471+0000","last_became_peered":"2021-02-10T04:25:49.002471+0000","last_unstale":"2021-02-10T04:32:07.242614+0000","last_undegraded":"2021-02-10T04:32:07.242614+0000","last_fullsized":"2021-02-10T04:32:07.242614+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:28:40.322504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:40.322504+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":15,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"278","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.091350+0000","last_change":"2021-02-10T04:29:36.663903+0000","last_active":"2021-02-10T04:32:08.091350+0000","last_peered":"2021-02-10T04:32:08.091350+0000","last_clean":"2021-02-10T04:32:08.091350+0000","last_became_active":"2021-02-10T04:29:07.158075+0000","last_became_peered":"2021-02-10T04:29:07.158075+0000","last_unstale":"2021-02-10T04:32:08.091350+0000","last_undegraded":"2021-02-10T04:32:08.091350+0000","last_fullsized":"2021-02-10T04:32:08.091350+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.663840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.663840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"297","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.240448+0000","last_change":"2021-02-10T04:28:34.482961+0000","last_active":"2021-02-10T04:32:07.240448+0000","last_peered":"2021-02-10T04:32:07.240448+0000","last_clean":"2021-02-10T04:32:07.240448+0000","last_became_active":"2021-02-10T04:28:21.123027+0000","last_became_peered":"2021-02-10T04:28:21.123027+0000","last_unstale":"2021-02-10T04:32:07.240448+0000","last_undegraded":"2021-02-10T04:32:07.240448+0000","last_fullsized":"2021-02-10T04:32:07.240448+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:34.482906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:34.482906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"274","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.096143+0000","last_change":"2021-02-10T04:29:35.382459+0000","last_active":"2021-02-10T04:32:08.096143+0000","last_peered":"2021-02-10T04:32:08.096143+0000","last_clean":"2021-02-10T04:32:08.096143+0000","last_became_active":"2021-02-10T04:25:49.010614+0000","last_became_peered":"2021-02-10T04:25:49.010614+0000","last_unstale":"2021-02-10T04:32:08.096143+0000","last_undegraded":"2021-02-10T04:32:08.096143+0000","last_fullsized":"2021-02-10T04:32:08.096143+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:35.382341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.382341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"270","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.241133+0000","last_change":"2021-02-10T04:28:38.503398+0000","last_active":"2021-02-10T04:32:07.241133+0000","last_peered":"2021-02-10T04:32:07.241133+0000","last_clean":"2021-02-10T04:32:07.241133+0000","last_became_active":"2021-02-10T04:25:53.037506+0000","last_became_peered":"2021-02-10T04:25:53.037506+0000","last_unstale":"2021-02-10T04:32:07.241133+0000","last_undegraded":"2021-02-10T04:32:07.241133+0000","last_fullsized":"2021-02-10T04:32:07.241133+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:38.503309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:38.503309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"275'85","reported_seq":"907","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.100783+0000","last_change":"2021-02-10T04:32:06.250888+0000","last_active":"2021-02-10T04:32:08.100783+0000","last_peered":"2021-02-10T04:32:08.100783+0000","last_clean":"2021-02-10T04:32:08.100783+0000","last_became_active":"2021-02-10T04:28:25.150109+0000","last_became_peered":"2021-02-10T04:28:25.150109+0000","last_unstale":"2021-02-10T04:32:08.100783+0000","last_undegraded":"2021-02-10T04:32:08.100783+0000","last_fullsized":"2021-02-10T04:32:08.100783+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'36","last_scrub_stamp":"2021-02-10T04:30:13.863430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:13.863430+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5899257,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":82528,"num_write":58,"num_write_kb":17039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"}]},{"pgid":"5.6","version":"0'0","reported_seq":"273","reported_epoch":"312","state":"active+clean","last_fresh":"2021-02-10T04:32:09.251192+0000","last_change":"2021-02-10T04:29:23.779839+0000","last_active":"2021-02-10T04:32:09.251192+0000","last_peered":"2021-02-10T04:32:09.251192+0000","last_clean":"2021-02-10T04:32:09.251192+0000","last_became_active":"2021-02-10T04:25:51.025306+0000","last_became_peered":"2021-02-10T04:25:51.025306+0000","last_unstale":"2021-02-10T04:32:09.251192+0000","last_undegraded":"2021-02-10T04:32:09.251192+0000","last_fullsized":"2021-02-10T04:32:09.251192+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.779767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.779767+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"82'29","reported_seq":"327","reported_epoch":"309","state":"active+clean","last_fresh":"2021-02-10T04:32:06.238849+0000","last_change":"2021-02-10T04:29:25.974354+0000","last_active":"2021-02-10T04:32:06.238849+0000","last_peered":"2021-02-10T04:32:06.238849+0000","last_clean":"2021-02-10T04:32:06.238849+0000","last_became_active":"2021-02-10T04:25:49.002843+0000","last_became_peered":"2021-02-10T04:25:49.002843+0000","last_unstale":"2021-02-10T04:32:06.238849+0000","last_undegraded":"2021-02-10T04:32:06.238849+0000","last_fullsized":"2021-02-10T04:32:06.238849+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'29","last_scrub_stamp":"2021-02-10T04:29:25.974280+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.974280+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":32,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"296","reported_epoch":"312","state":"active+clean","last_fresh":"2021-02-10T04:32:09.249891+0000","last_change":"2021-02-10T04:28:39.690493+0000","last_active":"2021-02-10T04:32:09.249891+0000","last_peered":"2021-02-10T04:32:09.249891+0000","last_clean":"2021-02-10T04:32:09.249891+0000","last_became_active":"2021-02-10T04:28:21.120083+0000","last_became_peered":"2021-02-10T04:28:21.120083+0000","last_unstale":"2021-02-10T04:32:09.249891+0000","last_undegraded":"2021-02-10T04:32:09.249891+0000","last_fullsized":"2021-02-10T04:32:09.249891+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:39.690436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:39.690436+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"652","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.572206+0000","last_change":"2021-02-10T04:28:50.992977+0000","last_active":"2021-02-10T04:32:07.572206+0000","last_peered":"2021-02-10T04:32:07.572206+0000","last_clean":"2021-02-10T04:32:07.572206+0000","last_became_active":"2021-02-10T04:25:49.013969+0000","last_became_peered":"2021-02-10T04:25:49.013969+0000","last_unstale":"2021-02-10T04:32:07.572206+0000","last_undegraded":"2021-02-10T04:32:07.572206+0000","last_fullsized":"2021-02-10T04:32:07.572206+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:28:50.992899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:50.992899+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":376,"num_read_kb":371,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"197","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.242741+0000","last_change":"2021-02-10T04:29:33.747527+0000","last_active":"2021-02-10T04:32:07.242741+0000","last_peered":"2021-02-10T04:32:07.242741+0000","last_clean":"2021-02-10T04:32:07.242741+0000","last_became_active":"2021-02-10T04:29:07.158869+0000","last_became_peered":"2021-02-10T04:29:07.158869+0000","last_unstale":"2021-02-10T04:32:07.242741+0000","last_undegraded":"2021-02-10T04:32:07.242741+0000","last_fullsized":"2021-02-10T04:32:07.242741+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:33.747448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:33.747448+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.4","version":"304'172","reported_seq":"862","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.101886+0000","last_change":"2021-02-10T04:32:06.249530+0000","last_active":"2021-02-10T04:32:08.101886+0000","last_peered":"2021-02-10T04:32:08.101886+0000","last_clean":"2021-02-10T04:32:08.101886+0000","last_became_active":"2021-02-10T04:28:25.151037+0000","last_became_peered":"2021-02-10T04:28:25.151037+0000","last_unstale":"2021-02-10T04:32:08.101886+0000","last_undegraded":"2021-02-10T04:32:08.101886+0000","last_fullsized":"2021-02-10T04:32:08.101886+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":172,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2521116,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":105,"num_read_kb":82050,"num_write":110,"num_write_kb":32385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"}]},{"pgid":"1.2","version":"0'0","reported_seq":"288","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.091050+0000","last_change":"2021-02-10T04:29:29.683867+0000","last_active":"2021-02-10T04:32:08.091050+0000","last_peered":"2021-02-10T04:32:08.091050+0000","last_clean":"2021-02-10T04:32:08.091050+0000","last_became_active":"2021-02-10T04:29:07.159607+0000","last_became_peered":"2021-02-10T04:29:07.159607+0000","last_unstale":"2021-02-10T04:32:08.091050+0000","last_undegraded":"2021-02-10T04:32:08.091050+0000","last_fullsized":"2021-02-10T04:32:08.091050+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.683805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.683805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"273","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.094210+0000","last_change":"2021-02-10T04:28:43.740930+0000","last_active":"2021-02-10T04:32:08.094210+0000","last_peered":"2021-02-10T04:32:08.094210+0000","last_clean":"2021-02-10T04:32:08.094210+0000","last_became_active":"2021-02-10T04:25:51.027624+0000","last_became_peered":"2021-02-10T04:25:51.027624+0000","last_unstale":"2021-02-10T04:32:08.094210+0000","last_undegraded":"2021-02-10T04:32:08.094210+0000","last_fullsized":"2021-02-10T04:32:08.094210+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:43.740853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:28:43.740853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"63'31","reported_seq":"363","reported_epoch":"309","state":"active+clean","last_fresh":"2021-02-10T04:32:06.241776+0000","last_change":"2021-02-10T04:32:03.235162+0000","last_active":"2021-02-10T04:32:06.241776+0000","last_peered":"2021-02-10T04:32:06.241776+0000","last_clean":"2021-02-10T04:32:06.241776+0000","last_became_active":"2021-02-10T04:25:49.007872+0000","last_became_peered":"2021-02-10T04:25:49.007872+0000","last_unstale":"2021-02-10T04:32:06.241776+0000","last_undegraded":"2021-02-10T04:32:06.241776+0000","last_fullsized":"2021-02-10T04:32:06.241776+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'31","last_scrub_stamp":"2021-02-10T04:28:03.352778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:03.352778+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":66,"num_read_kb":42,"num_write":34,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"290","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.240287+0000","last_change":"2021-02-10T04:28:36.491041+0000","last_active":"2021-02-10T04:32:07.240287+0000","last_peered":"2021-02-10T04:32:07.240287+0000","last_clean":"2021-02-10T04:32:07.240287+0000","last_became_active":"2021-02-10T04:28:21.125541+0000","last_became_peered":"2021-02-10T04:28:21.125541+0000","last_unstale":"2021-02-10T04:32:07.240287+0000","last_undegraded":"2021-02-10T04:32:07.240287+0000","last_fullsized":"2021-02-10T04:32:07.240287+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:36.490961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:36.490961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"270","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.096940+0000","last_change":"2021-02-10T04:29:36.396044+0000","last_active":"2021-02-10T04:32:08.096940+0000","last_peered":"2021-02-10T04:32:08.096940+0000","last_clean":"2021-02-10T04:32:08.096940+0000","last_became_active":"2021-02-10T04:25:53.049487+0000","last_became_peered":"2021-02-10T04:25:53.049487+0000","last_unstale":"2021-02-10T04:32:08.096940+0000","last_undegraded":"2021-02-10T04:32:08.096940+0000","last_fullsized":"2021-02-10T04:32:08.096940+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.395943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.395943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"310'590","reported_seq":"3403","reported_epoch":"311","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:32:08.212264+0000","last_change":"2021-02-10T04:32:07.182698+0000","last_active":"2021-02-10T04:32:08.212264+0000","last_peered":"2021-02-10T04:32:08.212264+0000","last_clean":"2021-02-10T04:32:08.212264+0000","last_became_active":"2021-02-10T04:28:25.147959+0000","last_became_peered":"2021-02-10T04:28:25.147959+0000","last_unstale":"2021-02-10T04:32:08.212264+0000","last_undegraded":"2021-02-10T04:32:08.212264+0000","last_fullsized":"2021-02-10T04:32:08.212264+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'223","last_scrub_stamp":"2021-02-10T04:30:14.478639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:14.478639+0000","log_size":590,"ondisk_log_size":590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":20151890,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":335,"num_read_kb":263548,"num_write":321,"num_write_kb":136282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"}]},{"pgid":"1.1","version":"0'0","reported_seq":"213","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.240054+0000","last_change":"2021-02-10T04:29:39.434301+0000","last_active":"2021-02-10T04:32:07.240054+0000","last_peered":"2021-02-10T04:32:07.240054+0000","last_clean":"2021-02-10T04:32:07.240054+0000","last_became_active":"2021-02-10T04:28:49.054267+0000","last_became_peered":"2021-02-10T04:28:49.054267+0000","last_unstale":"2021-02-10T04:32:07.240054+0000","last_undegraded":"2021-02-10T04:32:07.240054+0000","last_fullsized":"2021-02-10T04:32:07.240054+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:39.434210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:39.434210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"58'2","reported_seq":"351","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.101749+0000","last_change":"2021-02-10T04:28:33.656576+0000","last_active":"2021-02-10T04:32:08.101749+0000","last_peered":"2021-02-10T04:32:08.101749+0000","last_clean":"2021-02-10T04:32:08.101749+0000","last_became_active":"2021-02-10T04:25:51.026251+0000","last_became_peered":"2021-02-10T04:25:51.026251+0000","last_unstale":"2021-02-10T04:32:08.101749+0000","last_undegraded":"2021-02-10T04:32:08.101749+0000","last_fullsized":"2021-02-10T04:32:08.101749+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:28:33.656509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:33.656509+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"79'34","reported_seq":"327","reported_epoch":"309","state":"active+clean","last_fresh":"2021-02-10T04:32:06.238939+0000","last_change":"2021-02-10T04:28:49.959266+0000","last_active":"2021-02-10T04:32:06.238939+0000","last_peered":"2021-02-10T04:32:06.238939+0000","last_clean":"2021-02-10T04:32:06.238939+0000","last_became_active":"2021-02-10T04:25:49.006147+0000","last_became_peered":"2021-02-10T04:25:49.006147+0000","last_unstale":"2021-02-10T04:32:06.238939+0000","last_undegraded":"2021-02-10T04:32:06.238939+0000","last_fullsized":"2021-02-10T04:32:06.238939+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'34","last_scrub_stamp":"2021-02-10T04:28:49.959162+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:28:49.959162+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":22,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"307","reported_epoch":"309","state":"active+clean","last_fresh":"2021-02-10T04:32:06.241865+0000","last_change":"2021-02-10T04:28:34.053378+0000","last_active":"2021-02-10T04:32:06.241865+0000","last_peered":"2021-02-10T04:32:06.241865+0000","last_clean":"2021-02-10T04:32:06.241865+0000","last_became_active":"2021-02-10T04:28:21.133772+0000","last_became_peered":"2021-02-10T04:28:21.133772+0000","last_unstale":"2021-02-10T04:32:06.241865+0000","last_undegraded":"2021-02-10T04:32:06.241865+0000","last_fullsized":"2021-02-10T04:32:06.241865+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:34.053323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:34.053323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"271","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.091140+0000","last_change":"2021-02-10T04:28:42.701993+0000","last_active":"2021-02-10T04:32:08.091140+0000","last_peered":"2021-02-10T04:32:08.091140+0000","last_clean":"2021-02-10T04:32:08.091140+0000","last_became_active":"2021-02-10T04:25:53.045215+0000","last_became_peered":"2021-02-10T04:25:53.045215+0000","last_unstale":"2021-02-10T04:32:08.091140+0000","last_undegraded":"2021-02-10T04:32:08.091140+0000","last_fullsized":"2021-02-10T04:32:08.091140+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:42.701924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:42.701924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"293'77","reported_seq":"763","reported_epoch":"313","state":"active+clean","last_fresh":"2021-02-10T04:32:10.257331+0000","last_change":"2021-02-10T04:32:06.475334+0000","last_active":"2021-02-10T04:32:10.257331+0000","last_peered":"2021-02-10T04:32:10.257331+0000","last_clean":"2021-02-10T04:32:10.257331+0000","last_became_active":"2021-02-10T04:28:25.154434+0000","last_became_peered":"2021-02-10T04:28:25.154434+0000","last_unstale":"2021-02-10T04:32:10.257331+0000","last_undegraded":"2021-02-10T04:32:10.257331+0000","last_fullsized":"2021-02-10T04:32:10.257331+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7546343,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":74349,"num_write":37,"num_write_kb":13263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"}]},{"pgid":"1.7","version":"0'0","reported_seq":"294","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.242849+0000","last_change":"2021-02-10T04:29:39.829039+0000","last_active":"2021-02-10T04:32:07.242849+0000","last_peered":"2021-02-10T04:32:07.242849+0000","last_clean":"2021-02-10T04:32:07.242849+0000","last_became_active":"2021-02-10T04:28:49.053127+0000","last_became_peered":"2021-02-10T04:28:49.053127+0000","last_unstale":"2021-02-10T04:32:07.242849+0000","last_undegraded":"2021-02-10T04:32:07.242849+0000","last_fullsized":"2021-02-10T04:32:07.242849+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:39.828975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:39.828975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"275","reported_epoch":"312","state":"active+clean","last_fresh":"2021-02-10T04:32:09.249198+0000","last_change":"2021-02-10T04:29:20.757283+0000","last_active":"2021-02-10T04:32:09.249198+0000","last_peered":"2021-02-10T04:32:09.249198+0000","last_clean":"2021-02-10T04:32:09.249198+0000","last_became_active":"2021-02-10T04:25:51.023000+0000","last_became_peered":"2021-02-10T04:25:51.023000+0000","last_unstale":"2021-02-10T04:32:09.249198+0000","last_undegraded":"2021-02-10T04:32:09.249198+0000","last_fullsized":"2021-02-10T04:32:09.249198+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:20.757194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:20.757194+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"63'27","reported_seq":"336","reported_epoch":"313","state":"active+clean","last_fresh":"2021-02-10T04:32:10.257579+0000","last_change":"2021-02-10T04:28:47.476743+0000","last_active":"2021-02-10T04:32:10.257579+0000","last_peered":"2021-02-10T04:32:10.257579+0000","last_clean":"2021-02-10T04:32:10.257579+0000","last_became_active":"2021-02-10T04:25:49.013259+0000","last_became_peered":"2021-02-10T04:25:49.013259+0000","last_unstale":"2021-02-10T04:32:10.257579+0000","last_undegraded":"2021-02-10T04:32:10.257579+0000","last_fullsized":"2021-02-10T04:32:10.257579+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'27","last_scrub_stamp":"2021-02-10T04:28:47.476669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:47.476669+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":41,"num_write":29,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"360","reported_epoch":"313","state":"active+clean","last_fresh":"2021-02-10T04:32:10.260214+0000","last_change":"2021-02-10T04:32:07.519171+0000","last_active":"2021-02-10T04:32:10.260214+0000","last_peered":"2021-02-10T04:32:10.260214+0000","last_clean":"2021-02-10T04:32:10.260214+0000","last_became_active":"2021-02-10T04:28:39.041556+0000","last_became_peered":"2021-02-10T04:28:39.041556+0000","last_unstale":"2021-02-10T04:32:10.260214+0000","last_undegraded":"2021-02-10T04:32:10.260214+0000","last_fullsized":"2021-02-10T04:32:10.260214+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:27:53.635228+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:27:53.635228+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"270","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.103077+0000","last_change":"2021-02-10T04:29:37.364781+0000","last_active":"2021-02-10T04:32:08.103077+0000","last_peered":"2021-02-10T04:32:08.103077+0000","last_clean":"2021-02-10T04:32:08.103077+0000","last_became_active":"2021-02-10T04:25:53.047736+0000","last_became_peered":"2021-02-10T04:25:53.047736+0000","last_unstale":"2021-02-10T04:32:08.103077+0000","last_undegraded":"2021-02-10T04:32:08.103077+0000","last_fullsized":"2021-02-10T04:32:08.103077+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:37.364697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:37.364697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"309'338","reported_seq":"1761","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:09.248428+0000","last_change":"2021-02-10T04:32:09.248428+0000","last_active":"2021-02-10T04:32:09.248428+0000","last_peered":"2021-02-10T04:32:09.248428+0000","last_clean":"2021-02-10T04:32:09.248428+0000","last_became_active":"2021-02-10T04:28:25.157771+0000","last_became_peered":"2021-02-10T04:28:25.157771+0000","last_unstale":"2021-02-10T04:32:09.248428+0000","last_undegraded":"2021-02-10T04:32:09.248428+0000","last_fullsized":"2021-02-10T04:32:09.248428+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":338,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5344758,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":138,"num_read_kb":104717,"num_write":168,"num_write_kb":65922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"}]},{"pgid":"4.16","version":"63'28","reported_seq":"335","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.100899+0000","last_change":"2021-02-10T04:29:25.645462+0000","last_active":"2021-02-10T04:32:08.100899+0000","last_peered":"2021-02-10T04:32:08.100899+0000","last_clean":"2021-02-10T04:32:08.100899+0000","last_became_active":"2021-02-10T04:25:49.008942+0000","last_became_peered":"2021-02-10T04:25:49.008942+0000","last_unstale":"2021-02-10T04:32:08.100899+0000","last_undegraded":"2021-02-10T04:32:08.100899+0000","last_fullsized":"2021-02-10T04:32:08.100899+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'28","last_scrub_stamp":"2021-02-10T04:29:25.645396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.645396+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":62,"num_read_kb":40,"num_write":29,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"285","reported_epoch":"312","state":"active+clean","last_fresh":"2021-02-10T04:32:09.249137+0000","last_change":"2021-02-10T04:28:45.647998+0000","last_active":"2021-02-10T04:32:09.249137+0000","last_peered":"2021-02-10T04:32:09.249137+0000","last_clean":"2021-02-10T04:32:09.249137+0000","last_became_active":"2021-02-10T04:25:34.329694+0000","last_became_peered":"2021-02-10T04:25:34.329694+0000","last_unstale":"2021-02-10T04:32:09.249137+0000","last_undegraded":"2021-02-10T04:32:09.249137+0000","last_fullsized":"2021-02-10T04:32:09.249137+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:28:45.647916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:45.647916+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"272","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.097370+0000","last_change":"2021-02-10T04:28:33.321856+0000","last_active":"2021-02-10T04:32:08.097370+0000","last_peered":"2021-02-10T04:32:08.097370+0000","last_clean":"2021-02-10T04:32:08.097370+0000","last_became_active":"2021-02-10T04:25:51.030651+0000","last_became_peered":"2021-02-10T04:25:51.030651+0000","last_unstale":"2021-02-10T04:32:08.097370+0000","last_undegraded":"2021-02-10T04:32:08.097370+0000","last_fullsized":"2021-02-10T04:32:08.097370+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:33.321784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:33.321784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"271","reported_epoch":"312","state":"active+clean","last_fresh":"2021-02-10T04:32:09.250558+0000","last_change":"2021-02-10T04:29:28.776423+0000","last_active":"2021-02-10T04:32:09.250558+0000","last_peered":"2021-02-10T04:32:09.250558+0000","last_clean":"2021-02-10T04:32:09.250558+0000","last_became_active":"2021-02-10T04:25:53.035867+0000","last_became_peered":"2021-02-10T04:25:53.035867+0000","last_unstale":"2021-02-10T04:32:09.250558+0000","last_undegraded":"2021-02-10T04:32:09.250558+0000","last_fullsized":"2021-02-10T04:32:09.250558+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.776340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.776340+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"63'13","reported_seq":"321","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.097237+0000","last_change":"2021-02-10T04:28:11.311247+0000","last_active":"2021-02-10T04:32:08.097237+0000","last_peered":"2021-02-10T04:32:08.097237+0000","last_clean":"2021-02-10T04:32:08.097237+0000","last_became_active":"2021-02-10T04:26:39.003832+0000","last_became_peered":"2021-02-10T04:26:39.003832+0000","last_unstale":"2021-02-10T04:32:08.097237+0000","last_undegraded":"2021-02-10T04:32:08.097237+0000","last_fullsized":"2021-02-10T04:32:08.097237+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2021-02-10T04:28:11.311182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:11.311182+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"283","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.242207+0000","last_change":"2021-02-10T04:29:18.565481+0000","last_active":"2021-02-10T04:32:07.242207+0000","last_peered":"2021-02-10T04:32:07.242207+0000","last_clean":"2021-02-10T04:32:07.242207+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:32:07.242207+0000","last_undegraded":"2021-02-10T04:32:07.242207+0000","last_fullsized":"2021-02-10T04:32:07.242207+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:29:18.565413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:18.565413+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"272","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.240774+0000","last_change":"2021-02-10T04:29:18.429730+0000","last_active":"2021-02-10T04:32:07.240774+0000","last_peered":"2021-02-10T04:32:07.240774+0000","last_clean":"2021-02-10T04:32:07.240774+0000","last_became_active":"2021-02-10T04:25:51.031864+0000","last_became_peered":"2021-02-10T04:25:51.031864+0000","last_unstale":"2021-02-10T04:32:07.240774+0000","last_undegraded":"2021-02-10T04:32:07.240774+0000","last_fullsized":"2021-02-10T04:32:07.240774+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:18.429649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:18.429649+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"197","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.242838+0000","last_change":"2021-02-10T04:29:36.866949+0000","last_active":"2021-02-10T04:32:07.242838+0000","last_peered":"2021-02-10T04:32:07.242838+0000","last_clean":"2021-02-10T04:32:07.242838+0000","last_became_active":"2021-02-10T04:29:07.147249+0000","last_became_peered":"2021-02-10T04:29:07.147249+0000","last_unstale":"2021-02-10T04:32:07.242838+0000","last_undegraded":"2021-02-10T04:32:07.242838+0000","last_fullsized":"2021-02-10T04:32:07.242838+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.866833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.866833+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"63'26","reported_seq":"391","reported_epoch":"312","state":"active+clean","last_fresh":"2021-02-10T04:32:09.550652+0000","last_change":"2021-02-10T04:32:09.551533+0000","last_active":"2021-02-10T04:32:09.550652+0000","last_peered":"2021-02-10T04:32:09.550652+0000","last_clean":"2021-02-10T04:32:09.550652+0000","last_became_active":"2021-02-10T04:26:39.003971+0000","last_became_peered":"2021-02-10T04:26:39.003971+0000","last_unstale":"2021-02-10T04:32:09.550652+0000","last_undegraded":"2021-02-10T04:32:09.550652+0000","last_fullsized":"2021-02-10T04:32:09.550652+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:27:28.213454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:28.213454+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"84'7","reported_seq":"301","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.240266+0000","last_change":"2021-02-10T04:30:12.538670+0000","last_active":"2021-02-10T04:32:07.240266+0000","last_peered":"2021-02-10T04:32:07.240266+0000","last_clean":"2021-02-10T04:32:07.240266+0000","last_became_active":"2021-02-10T04:25:36.334366+0000","last_became_peered":"2021-02-10T04:25:36.334366+0000","last_unstale":"2021-02-10T04:32:07.240266+0000","last_undegraded":"2021-02-10T04:32:07.240266+0000","last_fullsized":"2021-02-10T04:32:07.240266+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'7","last_scrub_stamp":"2021-02-10T04:30:12.538549+0000","last_deep_scrub":"84'7","last_deep_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_clean_scrub_stamp":"2021-02-10T04:30:12.538549+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"58'2","reported_seq":"350","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.773521+0000","last_change":"2021-02-10T04:28:47.496534+0000","last_active":"2021-02-10T04:32:07.773521+0000","last_peered":"2021-02-10T04:32:07.773521+0000","last_clean":"2021-02-10T04:32:07.773521+0000","last_became_active":"2021-02-10T04:25:51.032328+0000","last_became_peered":"2021-02-10T04:25:51.032328+0000","last_unstale":"2021-02-10T04:32:07.773521+0000","last_undegraded":"2021-02-10T04:32:07.773521+0000","last_fullsized":"2021-02-10T04:32:07.773521+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:28:47.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:47.496468+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"271","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.100588+0000","last_change":"2021-02-10T04:28:45.697744+0000","last_active":"2021-02-10T04:32:08.100588+0000","last_peered":"2021-02-10T04:32:08.100588+0000","last_clean":"2021-02-10T04:32:08.100588+0000","last_became_active":"2021-02-10T04:25:53.045177+0000","last_became_peered":"2021-02-10T04:25:53.045177+0000","last_unstale":"2021-02-10T04:32:08.100588+0000","last_undegraded":"2021-02-10T04:32:08.100588+0000","last_fullsized":"2021-02-10T04:32:08.100588+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_clean_scrub_stamp":"2021-02-10T04:28:45.697677+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"63'19","reported_seq":"313","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.240428+0000","last_change":"2021-02-10T04:28:50.493712+0000","last_active":"2021-02-10T04:32:07.240428+0000","last_peered":"2021-02-10T04:32:07.240428+0000","last_clean":"2021-02-10T04:32:07.240428+0000","last_became_active":"2021-02-10T04:25:49.010890+0000","last_became_peered":"2021-02-10T04:25:49.010890+0000","last_unstale":"2021-02-10T04:32:07.240428+0000","last_undegraded":"2021-02-10T04:32:07.240428+0000","last_fullsized":"2021-02-10T04:32:07.240428+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2021-02-10T04:28:50.493641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:50.493641+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":26,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"287","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.092774+0000","last_change":"2021-02-10T04:29:19.564845+0000","last_active":"2021-02-10T04:32:08.092774+0000","last_peered":"2021-02-10T04:32:08.092774+0000","last_clean":"2021-02-10T04:32:08.092774+0000","last_became_active":"2021-02-10T04:25:34.321096+0000","last_became_peered":"2021-02-10T04:25:34.321096+0000","last_unstale":"2021-02-10T04:32:08.092774+0000","last_undegraded":"2021-02-10T04:32:08.092774+0000","last_fullsized":"2021-02-10T04:32:08.092774+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:19.564725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:29:19.564725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"58'2","reported_seq":"350","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.099195+0000","last_change":"2021-02-10T04:29:38.343946+0000","last_active":"2021-02-10T04:32:08.099195+0000","last_peered":"2021-02-10T04:32:08.099195+0000","last_clean":"2021-02-10T04:32:08.099195+0000","last_became_active":"2021-02-10T04:25:51.018300+0000","last_became_peered":"2021-02-10T04:25:51.018300+0000","last_unstale":"2021-02-10T04:32:08.099195+0000","last_undegraded":"2021-02-10T04:32:08.099195+0000","last_fullsized":"2021-02-10T04:32:08.099195+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:29:38.343834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.343834+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"273","reported_epoch":"313","state":"active+clean","last_fresh":"2021-02-10T04:32:10.260488+0000","last_change":"2021-02-10T04:29:23.934149+0000","last_active":"2021-02-10T04:32:10.260488+0000","last_peered":"2021-02-10T04:32:10.260488+0000","last_clean":"2021-02-10T04:32:10.260488+0000","last_became_active":"2021-02-10T04:25:53.031782+0000","last_became_peered":"2021-02-10T04:25:53.031782+0000","last_unstale":"2021-02-10T04:32:10.260488+0000","last_undegraded":"2021-02-10T04:32:10.260488+0000","last_fullsized":"2021-02-10T04:32:10.260488+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.934078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.934078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"62'15","reported_seq":"301","reported_epoch":"309","state":"active+clean","last_fresh":"2021-02-10T04:32:06.238855+0000","last_change":"2021-02-10T04:30:11.170100+0000","last_active":"2021-02-10T04:32:06.238855+0000","last_peered":"2021-02-10T04:32:06.238855+0000","last_clean":"2021-02-10T04:32:06.238855+0000","last_became_active":"2021-02-10T04:25:49.015543+0000","last_became_peered":"2021-02-10T04:25:49.015543+0000","last_unstale":"2021-02-10T04:32:06.238855+0000","last_undegraded":"2021-02-10T04:32:06.238855+0000","last_fullsized":"2021-02-10T04:32:06.238855+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:30:11.170034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.170034+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"281","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.241444+0000","last_change":"2021-02-10T04:28:36.204091+0000","last_active":"2021-02-10T04:32:07.241444+0000","last_peered":"2021-02-10T04:32:07.241444+0000","last_clean":"2021-02-10T04:32:07.241444+0000","last_became_active":"2021-02-10T04:25:34.330070+0000","last_became_peered":"2021-02-10T04:25:34.330070+0000","last_unstale":"2021-02-10T04:32:07.241444+0000","last_undegraded":"2021-02-10T04:32:07.241444+0000","last_fullsized":"2021-02-10T04:32:07.241444+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:36.204007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:36.204007+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"272","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.102989+0000","last_change":"2021-02-10T04:28:36.311808+0000","last_active":"2021-02-10T04:32:08.102989+0000","last_peered":"2021-02-10T04:32:08.102989+0000","last_clean":"2021-02-10T04:32:08.102989+0000","last_became_active":"2021-02-10T04:25:51.029155+0000","last_became_peered":"2021-02-10T04:25:51.029155+0000","last_unstale":"2021-02-10T04:32:08.102989+0000","last_undegraded":"2021-02-10T04:32:08.102989+0000","last_fullsized":"2021-02-10T04:32:08.102989+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:36.311725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:28:36.311725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"271","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.091053+0000","last_change":"2021-02-10T04:28:38.788947+0000","last_active":"2021-02-10T04:32:08.091053+0000","last_peered":"2021-02-10T04:32:08.091053+0000","last_clean":"2021-02-10T04:32:08.091053+0000","last_became_active":"2021-02-10T04:25:53.039322+0000","last_became_peered":"2021-02-10T04:25:53.039322+0000","last_unstale":"2021-02-10T04:32:08.091053+0000","last_undegraded":"2021-02-10T04:32:08.091053+0000","last_fullsized":"2021-02-10T04:32:08.091053+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:38.788851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:38.788851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"62'15","reported_seq":"299","reported_epoch":"313","state":"active+clean","last_fresh":"2021-02-10T04:32:10.257862+0000","last_change":"2021-02-10T04:27:42.487239+0000","last_active":"2021-02-10T04:32:10.257862+0000","last_peered":"2021-02-10T04:32:10.257862+0000","last_clean":"2021-02-10T04:32:10.257862+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:32:10.257862+0000","last_undegraded":"2021-02-10T04:32:10.257862+0000","last_fullsized":"2021-02-10T04:32:10.257862+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:27:42.487154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:42.487154+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"289","reported_epoch":"309","state":"active+clean","last_fresh":"2021-02-10T04:32:06.242020+0000","last_change":"2021-02-10T04:28:33.070791+0000","last_active":"2021-02-10T04:32:06.242020+0000","last_peered":"2021-02-10T04:32:06.242020+0000","last_clean":"2021-02-10T04:32:06.242020+0000","last_became_active":"2021-02-10T04:25:34.330242+0000","last_became_peered":"2021-02-10T04:25:34.330242+0000","last_unstale":"2021-02-10T04:32:06.242020+0000","last_undegraded":"2021-02-10T04:32:06.242020+0000","last_fullsized":"2021-02-10T04:32:06.242020+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:28:33.070726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:33.070726+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"271","reported_epoch":"309","state":"active+clean","last_fresh":"2021-02-10T04:32:06.239016+0000","last_change":"2021-02-10T04:29:33.027075+0000","last_active":"2021-02-10T04:32:06.239016+0000","last_peered":"2021-02-10T04:32:06.239016+0000","last_clean":"2021-02-10T04:32:06.239016+0000","last_became_active":"2021-02-10T04:25:51.030952+0000","last_became_peered":"2021-02-10T04:25:51.030952+0000","last_unstale":"2021-02-10T04:32:06.239016+0000","last_undegraded":"2021-02-10T04:32:06.239016+0000","last_fullsized":"2021-02-10T04:32:06.239016+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:33.026988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:29:33.026988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"270","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.095543+0000","last_change":"2021-02-10T04:29:24.421782+0000","last_active":"2021-02-10T04:32:08.095543+0000","last_peered":"2021-02-10T04:32:08.095543+0000","last_clean":"2021-02-10T04:32:08.095543+0000","last_became_active":"2021-02-10T04:25:53.043277+0000","last_became_peered":"2021-02-10T04:25:53.043277+0000","last_unstale":"2021-02-10T04:32:08.095543+0000","last_undegraded":"2021-02-10T04:32:08.095543+0000","last_fullsized":"2021-02-10T04:32:08.095543+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.421710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.421710+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"62'12","reported_seq":"322","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.093739+0000","last_change":"2021-02-10T04:27:48.284315+0000","last_active":"2021-02-10T04:32:08.093739+0000","last_peered":"2021-02-10T04:32:08.093739+0000","last_clean":"2021-02-10T04:32:08.093739+0000","last_became_active":"2021-02-10T04:26:39.005616+0000","last_became_peered":"2021-02-10T04:26:39.005616+0000","last_unstale":"2021-02-10T04:32:08.093739+0000","last_undegraded":"2021-02-10T04:32:08.093739+0000","last_fullsized":"2021-02-10T04:32:08.093739+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2021-02-10T04:27:48.284248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:48.284248+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"281","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.240956+0000","last_change":"2021-02-10T04:28:49.522178+0000","last_active":"2021-02-10T04:32:07.240956+0000","last_peered":"2021-02-10T04:32:07.240956+0000","last_clean":"2021-02-10T04:32:07.240956+0000","last_became_active":"2021-02-10T04:25:34.331787+0000","last_became_peered":"2021-02-10T04:25:34.331787+0000","last_unstale":"2021-02-10T04:32:07.240956+0000","last_undegraded":"2021-02-10T04:32:07.240956+0000","last_fullsized":"2021-02-10T04:32:07.240956+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:49.522124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:49.522124+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"272","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.242120+0000","last_change":"2021-02-10T04:29:23.645783+0000","last_active":"2021-02-10T04:32:07.242120+0000","last_peered":"2021-02-10T04:32:07.242120+0000","last_clean":"2021-02-10T04:32:07.242120+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:32:07.242120+0000","last_undegraded":"2021-02-10T04:32:07.242120+0000","last_fullsized":"2021-02-10T04:32:07.242120+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.645694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.645694+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"199","reported_epoch":"312","state":"active+clean","last_fresh":"2021-02-10T04:32:09.254909+0000","last_change":"2021-02-10T04:29:31.825006+0000","last_active":"2021-02-10T04:32:09.254909+0000","last_peered":"2021-02-10T04:32:09.254909+0000","last_clean":"2021-02-10T04:32:09.254909+0000","last_became_active":"2021-02-10T04:29:07.149764+0000","last_became_peered":"2021-02-10T04:29:07.149764+0000","last_unstale":"2021-02-10T04:32:09.254909+0000","last_undegraded":"2021-02-10T04:32:09.254909+0000","last_fullsized":"2021-02-10T04:32:09.254909+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:31.824925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.824925+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"63'21","reported_seq":"315","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.096018+0000","last_change":"2021-02-10T04:28:31.382776+0000","last_active":"2021-02-10T04:32:08.096018+0000","last_peered":"2021-02-10T04:32:08.096018+0000","last_clean":"2021-02-10T04:32:08.096018+0000","last_became_active":"2021-02-10T04:25:49.007049+0000","last_became_peered":"2021-02-10T04:25:49.007049+0000","last_unstale":"2021-02-10T04:32:08.096018+0000","last_undegraded":"2021-02-10T04:32:08.096018+0000","last_fullsized":"2021-02-10T04:32:08.096018+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2021-02-10T04:28:31.382671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:31.382671+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":27,"num_write":21,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"284","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.242920+0000","last_change":"2021-02-10T04:28:35.196412+0000","last_active":"2021-02-10T04:32:07.242920+0000","last_peered":"2021-02-10T04:32:07.242920+0000","last_clean":"2021-02-10T04:32:07.242920+0000","last_became_active":"2021-02-10T04:25:35.328828+0000","last_became_peered":"2021-02-10T04:25:35.328828+0000","last_unstale":"2021-02-10T04:32:07.242920+0000","last_undegraded":"2021-02-10T04:32:07.242920+0000","last_fullsized":"2021-02-10T04:32:07.242920+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:28:35.196342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:35.196342+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"195","reported_epoch":"313","state":"active+clean","last_fresh":"2021-02-10T04:32:10.256641+0000","last_change":"2021-02-10T04:29:07.150253+0000","last_active":"2021-02-10T04:32:10.256641+0000","last_peered":"2021-02-10T04:32:10.256641+0000","last_clean":"2021-02-10T04:32:10.256641+0000","last_became_active":"2021-02-10T04:29:07.149867+0000","last_became_peered":"2021-02-10T04:29:07.149867+0000","last_unstale":"2021-02-10T04:32:10.256641+0000","last_undegraded":"2021-02-10T04:32:10.256641+0000","last_fullsized":"2021-02-10T04:32:10.256641+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:47.473024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:47.473024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"265","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.103296+0000","last_change":"2021-02-10T04:27:50.300043+0000","last_active":"2021-02-10T04:32:08.103296+0000","last_peered":"2021-02-10T04:32:08.103296+0000","last_clean":"2021-02-10T04:32:08.103296+0000","last_became_active":"2021-02-10T04:25:53.150845+0000","last_became_peered":"2021-02-10T04:25:53.150845+0000","last_unstale":"2021-02-10T04:32:08.103296+0000","last_undegraded":"2021-02-10T04:32:08.103296+0000","last_fullsized":"2021-02-10T04:32:08.103296+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_clean_scrub_stamp":"2021-02-10T04:27:50.299942+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"63'22","reported_seq":"322","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.101162+0000","last_change":"2021-02-10T04:29:38.663351+0000","last_active":"2021-02-10T04:32:08.101162+0000","last_peered":"2021-02-10T04:32:08.101162+0000","last_clean":"2021-02-10T04:32:08.101162+0000","last_became_active":"2021-02-10T04:25:49.009177+0000","last_became_peered":"2021-02-10T04:25:49.009177+0000","last_unstale":"2021-02-10T04:32:08.101162+0000","last_undegraded":"2021-02-10T04:32:08.101162+0000","last_fullsized":"2021-02-10T04:32:08.101162+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2021-02-10T04:29:38.663272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.663272+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":22,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"290","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.239921+0000","last_change":"2021-02-10T04:28:37.515037+0000","last_active":"2021-02-10T04:32:07.239921+0000","last_peered":"2021-02-10T04:32:07.239921+0000","last_clean":"2021-02-10T04:32:07.239921+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:32:07.239921+0000","last_undegraded":"2021-02-10T04:32:07.239921+0000","last_fullsized":"2021-02-10T04:32:07.239921+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:37.514943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:37.514943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"58'2","reported_seq":"350","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.773185+0000","last_change":"2021-02-10T04:29:31.726022+0000","last_active":"2021-02-10T04:32:07.773185+0000","last_peered":"2021-02-10T04:32:07.773185+0000","last_clean":"2021-02-10T04:32:07.773185+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:32:07.773185+0000","last_undegraded":"2021-02-10T04:32:07.773185+0000","last_fullsized":"2021-02-10T04:32:07.773185+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:29:31.725932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.725932+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"197","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.240916+0000","last_change":"2021-02-10T04:29:26.437691+0000","last_active":"2021-02-10T04:32:07.240916+0000","last_peered":"2021-02-10T04:32:07.240916+0000","last_clean":"2021-02-10T04:32:07.240916+0000","last_became_active":"2021-02-10T04:29:07.150801+0000","last_became_peered":"2021-02-10T04:29:07.150801+0000","last_unstale":"2021-02-10T04:32:07.240916+0000","last_undegraded":"2021-02-10T04:32:07.240916+0000","last_fullsized":"2021-02-10T04:32:07.240916+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:26.437591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.437591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"76'35","reported_seq":"337","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.242983+0000","last_change":"2021-02-10T04:30:13.526264+0000","last_active":"2021-02-10T04:32:07.242983+0000","last_peered":"2021-02-10T04:32:07.242983+0000","last_clean":"2021-02-10T04:32:07.242983+0000","last_became_active":"2021-02-10T04:25:49.015760+0000","last_became_peered":"2021-02-10T04:25:49.015760+0000","last_unstale":"2021-02-10T04:32:07.242983+0000","last_undegraded":"2021-02-10T04:32:07.242983+0000","last_fullsized":"2021-02-10T04:32:07.242983+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'35","last_scrub_stamp":"2021-02-10T04:30:13.526170+0000","last_deep_scrub":"76'35","last_deep_scrub_stamp":"2021-02-10T04:26:58.198641+0000","last_clean_scrub_stamp":"2021-02-10T04:30:13.526170+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":50,"num_read_kb":30,"num_write":24,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"281","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.095777+0000","last_change":"2021-02-10T04:30:11.485059+0000","last_active":"2021-02-10T04:32:08.095777+0000","last_peered":"2021-02-10T04:32:08.095777+0000","last_clean":"2021-02-10T04:32:08.095777+0000","last_became_active":"2021-02-10T04:25:34.324981+0000","last_became_peered":"2021-02-10T04:25:34.324981+0000","last_unstale":"2021-02-10T04:32:08.095777+0000","last_undegraded":"2021-02-10T04:32:08.095777+0000","last_fullsized":"2021-02-10T04:32:08.095777+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:11.485004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.485004+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"273","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.101226+0000","last_change":"2021-02-10T04:29:23.589223+0000","last_active":"2021-02-10T04:32:08.101226+0000","last_peered":"2021-02-10T04:32:08.101226+0000","last_clean":"2021-02-10T04:32:08.101226+0000","last_became_active":"2021-02-10T04:25:51.022853+0000","last_became_peered":"2021-02-10T04:25:51.022853+0000","last_unstale":"2021-02-10T04:32:08.101226+0000","last_undegraded":"2021-02-10T04:32:08.101226+0000","last_fullsized":"2021-02-10T04:32:08.101226+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.589159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.589159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"269","reported_epoch":"309","state":"active+clean","last_fresh":"2021-02-10T04:32:06.241891+0000","last_change":"2021-02-10T04:29:25.076025+0000","last_active":"2021-02-10T04:32:06.241891+0000","last_peered":"2021-02-10T04:32:06.241891+0000","last_clean":"2021-02-10T04:32:06.241891+0000","last_became_active":"2021-02-10T04:25:53.041098+0000","last_became_peered":"2021-02-10T04:25:53.041098+0000","last_unstale":"2021-02-10T04:32:06.241891+0000","last_undegraded":"2021-02-10T04:32:06.241891+0000","last_fullsized":"2021-02-10T04:32:06.241891+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:25.075937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.075937+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"62'15","reported_seq":"300","reported_epoch":"312","state":"active+clean","last_fresh":"2021-02-10T04:32:09.250847+0000","last_change":"2021-02-10T04:28:46.680268+0000","last_active":"2021-02-10T04:32:09.250847+0000","last_peered":"2021-02-10T04:32:09.250847+0000","last_clean":"2021-02-10T04:32:09.250847+0000","last_became_active":"2021-02-10T04:25:49.010553+0000","last_became_peered":"2021-02-10T04:25:49.010553+0000","last_unstale":"2021-02-10T04:32:09.250847+0000","last_undegraded":"2021-02-10T04:32:09.250847+0000","last_fullsized":"2021-02-10T04:32:09.250847+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:28:46.680176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:46.680176+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"284","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.241654+0000","last_change":"2021-02-10T04:28:38.246850+0000","last_active":"2021-02-10T04:32:07.241654+0000","last_peered":"2021-02-10T04:32:07.241654+0000","last_clean":"2021-02-10T04:32:07.241654+0000","last_became_active":"2021-02-10T04:25:36.336442+0000","last_became_peered":"2021-02-10T04:25:36.336442+0000","last_unstale":"2021-02-10T04:32:07.241654+0000","last_undegraded":"2021-02-10T04:32:07.241654+0000","last_fullsized":"2021-02-10T04:32:07.241654+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:28:38.246768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:38.246768+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"271","reported_epoch":"309","state":"active+clean","last_fresh":"2021-02-10T04:32:06.242236+0000","last_change":"2021-02-10T04:29:22.078330+0000","last_active":"2021-02-10T04:32:06.242236+0000","last_peered":"2021-02-10T04:32:06.242236+0000","last_clean":"2021-02-10T04:32:06.242236+0000","last_became_active":"2021-02-10T04:25:51.029399+0000","last_became_peered":"2021-02-10T04:25:51.029399+0000","last_unstale":"2021-02-10T04:32:06.242236+0000","last_undegraded":"2021-02-10T04:32:06.242236+0000","last_fullsized":"2021-02-10T04:32:06.242236+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:22.078247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:22.078247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"273","reported_epoch":"313","state":"active+clean","last_fresh":"2021-02-10T04:32:10.259524+0000","last_change":"2021-02-10T04:29:31.903091+0000","last_active":"2021-02-10T04:32:10.259524+0000","last_peered":"2021-02-10T04:32:10.259524+0000","last_clean":"2021-02-10T04:32:10.259524+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:32:10.259524+0000","last_undegraded":"2021-02-10T04:32:10.259524+0000","last_fullsized":"2021-02-10T04:32:10.259524+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:31.902996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.902996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"62'22","reported_seq":"315","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.096513+0000","last_change":"2021-02-10T04:27:58.363322+0000","last_active":"2021-02-10T04:32:08.096513+0000","last_peered":"2021-02-10T04:32:08.096513+0000","last_clean":"2021-02-10T04:32:08.096513+0000","last_became_active":"2021-02-10T04:25:49.008997+0000","last_became_peered":"2021-02-10T04:25:49.008997+0000","last_unstale":"2021-02-10T04:32:08.096513+0000","last_undegraded":"2021-02-10T04:32:08.096513+0000","last_fullsized":"2021-02-10T04:32:08.096513+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'22","last_scrub_stamp":"2021-02-10T04:27:58.363248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:58.363248+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":49,"num_read_kb":31,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"293","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.102788+0000","last_change":"2021-02-10T04:28:30.415204+0000","last_active":"2021-02-10T04:32:08.102788+0000","last_peered":"2021-02-10T04:32:08.102788+0000","last_clean":"2021-02-10T04:32:08.102788+0000","last_became_active":"2021-02-10T04:25:36.336893+0000","last_became_peered":"2021-02-10T04:25:36.336893+0000","last_unstale":"2021-02-10T04:32:08.102788+0000","last_undegraded":"2021-02-10T04:32:08.102788+0000","last_fullsized":"2021-02-10T04:32:08.102788+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:30.415134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:26:46.425288+0000","last_clean_scrub_stamp":"2021-02-10T04:28:30.415134+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"270","reported_epoch":"313","state":"active+clean","last_fresh":"2021-02-10T04:32:10.258397+0000","last_change":"2021-02-10T04:27:48.598953+0000","last_active":"2021-02-10T04:32:10.258397+0000","last_peered":"2021-02-10T04:32:10.258397+0000","last_clean":"2021-02-10T04:32:10.258397+0000","last_became_active":"2021-02-10T04:25:51.018872+0000","last_became_peered":"2021-02-10T04:25:51.018872+0000","last_unstale":"2021-02-10T04:32:10.258397+0000","last_undegraded":"2021-02-10T04:32:10.258397+0000","last_fullsized":"2021-02-10T04:32:10.258397+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:48.598870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:48.598870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"270","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.097155+0000","last_change":"2021-02-10T04:29:26.355726+0000","last_active":"2021-02-10T04:32:08.097155+0000","last_peered":"2021-02-10T04:32:08.097155+0000","last_clean":"2021-02-10T04:32:08.097155+0000","last_became_active":"2021-02-10T04:25:53.147831+0000","last_became_peered":"2021-02-10T04:25:53.147831+0000","last_unstale":"2021-02-10T04:32:08.097155+0000","last_undegraded":"2021-02-10T04:32:08.097155+0000","last_fullsized":"2021-02-10T04:32:08.097155+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:26.355652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.355652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"63'25","reported_seq":"331","reported_epoch":"313","state":"active+clean","last_fresh":"2021-02-10T04:32:10.256910+0000","last_change":"2021-02-10T04:28:32.565457+0000","last_active":"2021-02-10T04:32:10.256910+0000","last_peered":"2021-02-10T04:32:10.256910+0000","last_clean":"2021-02-10T04:32:10.256910+0000","last_became_active":"2021-02-10T04:25:49.000911+0000","last_became_peered":"2021-02-10T04:25:49.000911+0000","last_unstale":"2021-02-10T04:32:10.256910+0000","last_undegraded":"2021-02-10T04:32:10.256910+0000","last_fullsized":"2021-02-10T04:32:10.256910+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2021-02-10T04:28:32.565374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:32.565374+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":53,"num_read_kb":37,"num_write":26,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"192","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.241045+0000","last_change":"2021-02-10T04:29:07.152012+0000","last_active":"2021-02-10T04:32:07.241045+0000","last_peered":"2021-02-10T04:32:07.241045+0000","last_clean":"2021-02-10T04:32:07.241045+0000","last_became_active":"2021-02-10T04:29:07.151577+0000","last_became_peered":"2021-02-10T04:29:07.151577+0000","last_unstale":"2021-02-10T04:32:07.241045+0000","last_undegraded":"2021-02-10T04:32:07.241045+0000","last_fullsized":"2021-02-10T04:32:07.241045+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:00.398480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:00.398480+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"272","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.241095+0000","last_change":"2021-02-10T04:28:46.464172+0000","last_active":"2021-02-10T04:32:07.241095+0000","last_peered":"2021-02-10T04:32:07.241095+0000","last_clean":"2021-02-10T04:32:07.241095+0000","last_became_active":"2021-02-10T04:25:51.025488+0000","last_became_peered":"2021-02-10T04:25:51.025488+0000","last_unstale":"2021-02-10T04:32:07.241095+0000","last_undegraded":"2021-02-10T04:32:07.241095+0000","last_fullsized":"2021-02-10T04:32:07.241095+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:46.464108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:46.464108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"271","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.091650+0000","last_change":"2021-02-10T04:28:44.719453+0000","last_active":"2021-02-10T04:32:08.091650+0000","last_peered":"2021-02-10T04:32:08.091650+0000","last_clean":"2021-02-10T04:32:08.091650+0000","last_became_active":"2021-02-10T04:25:53.039844+0000","last_became_peered":"2021-02-10T04:25:53.039844+0000","last_unstale":"2021-02-10T04:32:08.091650+0000","last_undegraded":"2021-02-10T04:32:08.091650+0000","last_fullsized":"2021-02-10T04:32:08.091650+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:44.719363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:44.719363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"63'12","reported_seq":"304","reported_epoch":"313","state":"active+clean","last_fresh":"2021-02-10T04:32:10.257695+0000","last_change":"2021-02-10T04:30:13.089576+0000","last_active":"2021-02-10T04:32:10.257695+0000","last_peered":"2021-02-10T04:32:10.257695+0000","last_clean":"2021-02-10T04:32:10.257695+0000","last_became_active":"2021-02-10T04:25:49.013196+0000","last_became_peered":"2021-02-10T04:25:49.013196+0000","last_unstale":"2021-02-10T04:32:10.257695+0000","last_undegraded":"2021-02-10T04:32:10.257695+0000","last_fullsized":"2021-02-10T04:32:10.257695+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2021-02-10T04:30:13.089494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:30:13.089494+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"197","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.242624+0000","last_change":"2021-02-10T04:30:11.537208+0000","last_active":"2021-02-10T04:32:07.242624+0000","last_peered":"2021-02-10T04:32:07.242624+0000","last_clean":"2021-02-10T04:32:07.242624+0000","last_became_active":"2021-02-10T04:29:07.157944+0000","last_became_peered":"2021-02-10T04:29:07.157944+0000","last_unstale":"2021-02-10T04:32:07.242624+0000","last_undegraded":"2021-02-10T04:32:07.242624+0000","last_fullsized":"2021-02-10T04:32:07.242624+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:11.537136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.537136+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"195","reported_epoch":"313","state":"active+clean","last_fresh":"2021-02-10T04:32:10.257035+0000","last_change":"2021-02-10T04:29:07.146984+0000","last_active":"2021-02-10T04:32:10.257035+0000","last_peered":"2021-02-10T04:32:10.257035+0000","last_clean":"2021-02-10T04:32:10.257035+0000","last_became_active":"2021-02-10T04:29:07.146525+0000","last_became_peered":"2021-02-10T04:29:07.146525+0000","last_unstale":"2021-02-10T04:32:10.257035+0000","last_undegraded":"2021-02-10T04:32:10.257035+0000","last_fullsized":"2021-02-10T04:32:10.257035+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:46.493066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:46.493066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"270","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.239981+0000","last_change":"2021-02-10T04:29:22.475613+0000","last_active":"2021-02-10T04:32:07.239981+0000","last_peered":"2021-02-10T04:32:07.239981+0000","last_clean":"2021-02-10T04:32:07.239981+0000","last_became_active":"2021-02-10T04:25:53.042601+0000","last_became_peered":"2021-02-10T04:25:53.042601+0000","last_unstale":"2021-02-10T04:32:07.239981+0000","last_undegraded":"2021-02-10T04:32:07.239981+0000","last_fullsized":"2021-02-10T04:32:07.239981+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:22.475504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:22.475504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"78'32","reported_seq":"331","reported_epoch":"313","state":"active+clean","last_fresh":"2021-02-10T04:32:10.256728+0000","last_change":"2021-02-10T04:28:35.520489+0000","last_active":"2021-02-10T04:32:10.256728+0000","last_peered":"2021-02-10T04:32:10.256728+0000","last_clean":"2021-02-10T04:32:10.256728+0000","last_became_active":"2021-02-10T04:25:49.010481+0000","last_became_peered":"2021-02-10T04:25:49.010481+0000","last_unstale":"2021-02-10T04:32:10.256728+0000","last_undegraded":"2021-02-10T04:32:10.256728+0000","last_fullsized":"2021-02-10T04:32:10.256728+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'32","last_scrub_stamp":"2021-02-10T04:28:35.520410+0000","last_deep_scrub":"78'32","last_deep_scrub_stamp":"2021-02-10T04:27:09.170428+0000","last_clean_scrub_stamp":"2021-02-10T04:28:35.520410+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":47,"num_read_kb":28,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"281","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.096860+0000","last_change":"2021-02-10T04:29:29.355356+0000","last_active":"2021-02-10T04:32:08.096860+0000","last_peered":"2021-02-10T04:32:08.096860+0000","last_clean":"2021-02-10T04:32:08.096860+0000","last_became_active":"2021-02-10T04:25:34.327162+0000","last_became_peered":"2021-02-10T04:25:34.327162+0000","last_unstale":"2021-02-10T04:32:08.096860+0000","last_undegraded":"2021-02-10T04:32:08.096860+0000","last_fullsized":"2021-02-10T04:32:08.096860+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.355257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.355257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"271","reported_epoch":"309","state":"active+clean","last_fresh":"2021-02-10T04:32:06.237990+0000","last_change":"2021-02-10T04:28:40.098300+0000","last_active":"2021-02-10T04:32:06.237990+0000","last_peered":"2021-02-10T04:32:06.237990+0000","last_clean":"2021-02-10T04:32:06.237990+0000","last_became_active":"2021-02-10T04:25:51.030405+0000","last_became_peered":"2021-02-10T04:25:51.030405+0000","last_unstale":"2021-02-10T04:32:06.237990+0000","last_undegraded":"2021-02-10T04:32:06.237990+0000","last_fullsized":"2021-02-10T04:32:06.237990+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:40.098239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:40.098239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"268","reported_epoch":"313","state":"active+clean","last_fresh":"2021-02-10T04:32:10.256827+0000","last_change":"2021-02-10T04:27:51.679613+0000","last_active":"2021-02-10T04:32:10.256827+0000","last_peered":"2021-02-10T04:32:10.256827+0000","last_clean":"2021-02-10T04:32:10.256827+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:32:10.256827+0000","last_undegraded":"2021-02-10T04:32:10.256827+0000","last_fullsized":"2021-02-10T04:32:10.256827+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:51.679515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:51.679515+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"63'44","reported_seq":"364","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.096106+0000","last_change":"2021-02-10T04:27:49.269619+0000","last_active":"2021-02-10T04:32:08.096106+0000","last_peered":"2021-02-10T04:32:08.096106+0000","last_clean":"2021-02-10T04:32:08.096106+0000","last_became_active":"2021-02-10T04:25:49.008451+0000","last_became_peered":"2021-02-10T04:25:49.008451+0000","last_unstale":"2021-02-10T04:32:08.096106+0000","last_undegraded":"2021-02-10T04:32:08.096106+0000","last_fullsized":"2021-02-10T04:32:08.096106+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'44","last_scrub_stamp":"2021-02-10T04:27:49.269539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:49.269539+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":89,"num_read_kb":66,"num_write":45,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"291","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.101719+0000","last_change":"2021-02-10T04:29:28.719237+0000","last_active":"2021-02-10T04:32:08.101719+0000","last_peered":"2021-02-10T04:32:08.101719+0000","last_clean":"2021-02-10T04:32:08.101719+0000","last_became_active":"2021-02-10T04:25:36.336112+0000","last_became_peered":"2021-02-10T04:25:36.336112+0000","last_unstale":"2021-02-10T04:32:08.101719+0000","last_undegraded":"2021-02-10T04:32:08.101719+0000","last_fullsized":"2021-02-10T04:32:08.101719+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.719147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.719147+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"270","reported_epoch":"313","state":"active+clean","last_fresh":"2021-02-10T04:32:10.256455+0000","last_change":"2021-02-10T04:27:49.628172+0000","last_active":"2021-02-10T04:32:10.256455+0000","last_peered":"2021-02-10T04:32:10.256455+0000","last_clean":"2021-02-10T04:32:10.256455+0000","last_became_active":"2021-02-10T04:25:51.029110+0000","last_became_peered":"2021-02-10T04:25:51.029110+0000","last_unstale":"2021-02-10T04:32:10.256455+0000","last_undegraded":"2021-02-10T04:32:10.256455+0000","last_fullsized":"2021-02-10T04:32:10.256455+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:49.628090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:49.628090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"270","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.102697+0000","last_change":"2021-02-10T04:29:25.382883+0000","last_active":"2021-02-10T04:32:08.102697+0000","last_peered":"2021-02-10T04:32:08.102697+0000","last_clean":"2021-02-10T04:32:08.102697+0000","last_became_active":"2021-02-10T04:25:53.150917+0000","last_became_peered":"2021-02-10T04:25:53.150917+0000","last_unstale":"2021-02-10T04:32:08.102697+0000","last_undegraded":"2021-02-10T04:32:08.102697+0000","last_fullsized":"2021-02-10T04:32:08.102697+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:25.382793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.382793+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.20","version":"0'0","reported_seq":"260","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.240048+0000","last_change":"2021-02-10T04:30:11.492843+0000","last_active":"2021-02-10T04:32:07.240048+0000","last_peered":"2021-02-10T04:32:07.240048+0000","last_clean":"2021-02-10T04:32:07.240048+0000","last_became_active":"2021-02-10T04:29:07.148937+0000","last_became_peered":"2021-02-10T04:29:07.148937+0000","last_unstale":"2021-02-10T04:32:07.240048+0000","last_undegraded":"2021-02-10T04:32:07.240048+0000","last_fullsized":"2021-02-10T04:32:07.240048+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:11.492788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:41.375276+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.492788+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":"313","reported_epoch":"312","state":"active+clean","last_fresh":"2021-02-10T04:32:09.250069+0000","last_change":"2021-02-10T04:29:21.708787+0000","last_active":"2021-02-10T04:32:09.250069+0000","last_peered":"2021-02-10T04:32:09.250069+0000","last_clean":"2021-02-10T04:32:09.250069+0000","last_became_active":"2021-02-10T04:28:39.045354+0000","last_became_peered":"2021-02-10T04:28:39.045354+0000","last_unstale":"2021-02-10T04:32:09.250069+0000","last_undegraded":"2021-02-10T04:32:09.250069+0000","last_fullsized":"2021-02-10T04:32:09.250069+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:21.708701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:21.708701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":"290","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.241365+0000","last_change":"2021-02-10T04:28:42.382533+0000","last_active":"2021-02-10T04:32:07.241365+0000","last_peered":"2021-02-10T04:32:07.241365+0000","last_clean":"2021-02-10T04:32:07.241365+0000","last_became_active":"2021-02-10T04:28:23.129832+0000","last_became_peered":"2021-02-10T04:28:23.129832+0000","last_unstale":"2021-02-10T04:32:07.241365+0000","last_undegraded":"2021-02-10T04:32:07.241365+0000","last_fullsized":"2021-02-10T04:32:07.241365+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:42.382464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:42.382464+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":"235","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.099661+0000","last_change":"2021-02-10T04:28:48.409915+0000","last_active":"2021-02-10T04:32:08.099661+0000","last_peered":"2021-02-10T04:32:08.099661+0000","last_clean":"2021-02-10T04:32:08.099661+0000","last_became_active":"2021-02-10T04:28:23.129523+0000","last_became_peered":"2021-02-10T04:28:23.129523+0000","last_unstale":"2021-02-10T04:32:08.099661+0000","last_undegraded":"2021-02-10T04:32:08.099661+0000","last_fullsized":"2021-02-10T04:32:08.099661+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:48.409826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:48.409826+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":"234","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.241311+0000","last_change":"2021-02-10T04:28:46.203018+0000","last_active":"2021-02-10T04:32:07.241311+0000","last_peered":"2021-02-10T04:32:07.241311+0000","last_clean":"2021-02-10T04:32:07.241311+0000","last_became_active":"2021-02-10T04:28:23.126368+0000","last_became_peered":"2021-02-10T04:28:23.126368+0000","last_unstale":"2021-02-10T04:32:07.241311+0000","last_undegraded":"2021-02-10T04:32:07.241311+0000","last_fullsized":"2021-02-10T04:32:07.241311+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:46.202944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:46.202944+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.25","version":"53'2","reported_seq":"255","reported_epoch":"313","state":"active+clean","last_fresh":"2021-02-10T04:32:10.258580+0000","last_change":"2021-02-10T04:28:40.422198+0000","last_active":"2021-02-10T04:32:10.258580+0000","last_peered":"2021-02-10T04:32:10.258580+0000","last_clean":"2021-02-10T04:32:10.258580+0000","last_became_active":"2021-02-10T04:28:23.467770+0000","last_became_peered":"2021-02-10T04:28:23.467770+0000","last_unstale":"2021-02-10T04:32:10.258580+0000","last_undegraded":"2021-02-10T04:32:10.258580+0000","last_fullsized":"2021-02-10T04:32:10.258580+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:28:40.422115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:40.422115+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.26","version":"53'1","reported_seq":"225","reported_epoch":"312","state":"active+clean","last_fresh":"2021-02-10T04:32:09.251042+0000","last_change":"2021-02-10T04:29:22.801988+0000","last_active":"2021-02-10T04:32:09.251042+0000","last_peered":"2021-02-10T04:32:09.251042+0000","last_clean":"2021-02-10T04:32:09.251042+0000","last_became_active":"2021-02-10T04:28:39.043300+0000","last_became_peered":"2021-02-10T04:28:39.043300+0000","last_unstale":"2021-02-10T04:32:09.251042+0000","last_undegraded":"2021-02-10T04:32:09.251042+0000","last_fullsized":"2021-02-10T04:32:09.251042+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:29:22.801929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:22.801929+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":"266","reported_epoch":"313","state":"active+clean","last_fresh":"2021-02-10T04:32:10.260071+0000","last_change":"2021-02-10T04:29:07.146256+0000","last_active":"2021-02-10T04:32:10.260071+0000","last_peered":"2021-02-10T04:32:10.260071+0000","last_clean":"2021-02-10T04:32:10.260071+0000","last_became_active":"2021-02-10T04:29:07.145907+0000","last_became_peered":"2021-02-10T04:29:07.145907+0000","last_unstale":"2021-02-10T04:32:10.260071+0000","last_undegraded":"2021-02-10T04:32:10.260071+0000","last_fullsized":"2021-02-10T04:32:10.260071+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:39.388884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:39.388884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":"235","reported_epoch":"311","state":"active+clean","last_fresh":"2021-02-10T04:32:08.094156+0000","last_change":"2021-02-10T04:29:23.440451+0000","last_active":"2021-02-10T04:32:08.094156+0000","last_peered":"2021-02-10T04:32:08.094156+0000","last_clean":"2021-02-10T04:32:08.094156+0000","last_became_active":"2021-02-10T04:28:23.129469+0000","last_became_peered":"2021-02-10T04:28:23.129469+0000","last_unstale":"2021-02-10T04:32:08.094156+0000","last_undegraded":"2021-02-10T04:32:08.094156+0000","last_fullsized":"2021-02-10T04:32:08.094156+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.440391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.440391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":"234","reported_epoch":"310","state":"active+clean","last_fresh":"2021-02-10T04:32:07.240953+0000","last_change":"2021-02-10T04:28:45.220352+0000","last_active":"2021-02-10T04:32:07.240953+0000","last_peered":"2021-02-10T04:32:07.240953+0000","last_clean":"2021-02-10T04:32:07.240953+0000","last_became_active":"2021-02-10T04:28:23.126795+0000","last_became_peered":"2021-02-10T04:28:23.126795+0000","last_unstale":"2021-02-10T04:32:07.240953+0000","last_undegraded":"2021-02-10T04:32:07.240953+0000","last_fullsized":"2021-02-10T04:32:07.240953+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_clean_scrub_stamp":"2021-02-10T04:28:45.220258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":131420022,"num_objects":50,"num_object_clones":7,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":1,"num_read":2092,"num_read_kb":1934637,"num_write":1685,"num_write_kb":615549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":321052672,"data_stored":340442356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3143,"ondisk_log_size":3143,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":1945,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":17,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":5835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25,"ondisk_log_size":25,"up":126,"acting":126,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":376,"num_read_kb":371,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":1453,"num_read_kb":926,"num_write":734,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":782,"ondisk_log_size":782,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748364881,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":43404,"kb_used_data":24260,"kb_used_omap":0,"kb_used_meta":19136,"kb_avail":93701748,"statfs":{"total":95995035648,"available":95950589952,"internally_reserved":0,"allocated":24842240,"data_stored":25754526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19595264},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:31:45 2021","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":1.0129999999999999,"15min":0.92800000000000005},"min":{"1min":0.53300000000000003,"5min":0.441,"15min":0.441},"max":{"1min":2.1120000000000001,"5min":18.681000000000001,"15min":18.681000000000001},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.121,"15min":0.94699999999999995},"min":{"1min":0.56299999999999994,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":4.1109999999999998,"5min":18.274999999999999,"15min":18.274999999999999},"last":0.74399999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:31:45 2021","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":1.3300000000000001,"15min":1.038},"min":{"1min":0.55500000000000005,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.3720000000000001,"5min":41.966000000000001,"15min":41.966000000000001},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.88,"5min":1.4359999999999999,"15min":1.161},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.131,"5min":41.804000000000002,"15min":41.804000000000002},"last":0.79400000000000004}]},{"osd":2,"last update":"Wed Feb 10 04:31:45 2021","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":2.6549999999999998,"15min":1.5720000000000001},"min":{"1min":0.70099999999999996,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.3819999999999999,"5min":125.714,"15min":125.714},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.1699999999999999,"5min":2.7080000000000002,"15min":1.5429999999999999},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.75,"5min":125.53700000000001,"15min":125.53700000000001},"last":0.97799999999999998}]},{"osd":3,"last update":"Wed Feb 10 04:31:45 2021","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":4.6459999999999999,"15min":2.226},"min":{"1min":0.82399999999999995,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.591,"5min":192.95500000000001,"15min":192.95500000000001},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.151,"5min":4.7640000000000002,"15min":2.359},"min":{"1min":0.94099999999999995,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.339,"5min":192.601,"15min":192.601},"last":1.0409999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:31:45 2021","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.1910000000000001,"15min":1.0229999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.64900000000000002},"max":{"1min":3.8050000000000002,"5min":3.8050000000000002,"15min":3.8050000000000002},"last":1.284},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.1719999999999999,"15min":1.1339999999999999},"min":{"1min":0.748,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":2.1400000000000001,"5min":2.847,"15min":2.847},"last":1.2509999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:31:45 2021","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.294,"15min":1.194},"min":{"1min":0.85699999999999998,"5min":0.68899999999999995,"15min":0.624},"max":{"1min":1.5609999999999999,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":1.982},{"interface":"front","average":{"1min":1.488,"5min":1.3740000000000001,"15min":1.2569999999999999},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":4.6790000000000003,"5min":4.6790000000000003,"15min":4.6790000000000003},"last":1.4530000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:31:45 2021","interfaces":[{"interface":"back","average":{"1min":3.1349999999999998,"5min":1.8069999999999999,"15min":1.4299999999999999},"min":{"1min":0.98999999999999999,"5min":0.94199999999999995,"15min":0.86899999999999999},"max":{"1min":29.206,"5min":29.206,"15min":29.206},"last":1.385},{"interface":"front","average":{"1min":2.8940000000000001,"5min":1.7470000000000001,"15min":1.399},"min":{"1min":1.0269999999999999,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":26.913,"5min":26.913,"15min":26.913},"last":1.73}]}]},{"osd":6,"up_from":43,"seq":184683593813,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77252,"kb_used_data":56252,"kb_used_omap":0,"kb_used_meta":20992,"kb_avail":93667900,"statfs":{"total":95995035648,"available":95915929600,"internally_reserved":0,"allocated":57602048,"data_stored":59695026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21495808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:31:26 2021","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":1.921,"15min":1.2729999999999999},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":3.105,"5min":64.164000000000001,"15min":64.164000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.81299999999999994,"5min":1.998,"15min":1.347},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.3660000000000001,"5min":64.277000000000001,"15min":64.277000000000001},"last":1.508}]},{"osd":1,"last update":"Wed Feb 10 04:31:26 2021","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.3029999999999999,"15min":1.1259999999999999},"min":{"1min":0.63500000000000001,"5min":0.61599999999999999,"15min":0.61099999999999999},"max":{"1min":10.331,"5min":18.177,"15min":18.177},"last":0.88},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.2390000000000001,"15min":1.0649999999999999},"min":{"1min":0.624,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.7390000000000001,"5min":18.898,"15min":18.898},"last":1.042}]},{"osd":2,"last update":"Wed Feb 10 04:31:26 2021","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":1.274,"15min":1.1319999999999999},"min":{"1min":0.63400000000000001,"5min":0.625,"15min":0.625},"max":{"1min":1.7,"5min":19.474,"15min":19.474},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.3340000000000001,"15min":1.1160000000000001},"min":{"1min":0.73999999999999999,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.663,"5min":19.227,"15min":19.227},"last":1.117}]},{"osd":3,"last update":"Wed Feb 10 04:31:26 2021","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.411,"15min":1.2270000000000001},"min":{"1min":0.80800000000000005,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":3.7480000000000002,"5min":15.316000000000001,"15min":15.316000000000001},"last":1.6499999999999999},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.4650000000000001,"15min":1.218},"min":{"1min":0.88900000000000001,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.7,"5min":15.35,"15min":15.35},"last":0.98599999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:31:26 2021","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.2749999999999999,"15min":1.0549999999999999},"min":{"1min":0.75700000000000001,"5min":0.61699999999999999,"15min":0.58999999999999997},"max":{"1min":3.4969999999999999,"5min":3.4969999999999999,"15min":3.4969999999999999},"last":1.3819999999999999},{"interface":"front","average":{"1min":0.91300000000000003,"5min":1.012,"15min":0.92800000000000005},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.177,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.94399999999999995}]},{"osd":5,"last update":"Wed Feb 10 04:31:26 2021","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":1.1739999999999999,"15min":1.044},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.4339999999999999,"5min":5.3529999999999998,"15min":5.3529999999999998},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.26,"5min":1.4139999999999999,"15min":1.218},"min":{"1min":0.84099999999999997,"5min":0.81799999999999995,"15min":0.69499999999999995},"max":{"1min":1.651,"5min":2.4550000000000001,"15min":2.4550000000000001},"last":1.5589999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:31:50 2021","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.248,"15min":1.1419999999999999},"min":{"1min":0.76900000000000002,"5min":0.75800000000000001,"15min":0.73599999999999999},"max":{"1min":1.47,"5min":2.6680000000000001,"15min":2.6680000000000001},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.425,"5min":1.54,"15min":1.4359999999999999},"min":{"1min":0.93500000000000005,"5min":0.90300000000000002,"15min":0.89000000000000001},"max":{"1min":1.7709999999999999,"5min":3.5049999999999999,"15min":3.5049999999999999},"last":1.3320000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574950,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":53580,"kb_used_data":32708,"kb_used_omap":0,"kb_used_meta":20864,"kb_avail":93691572,"statfs":{"total":95995035648,"available":95940169728,"internally_reserved":0,"allocated":33492992,"data_stored":32585572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21364736},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,2],"upper_bound":524288},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:32:05 2021","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.1279999999999999,"15min":0.80000000000000004},"min":{"1min":0.502,"5min":0.47899999999999998,"15min":0.36399999999999999},"max":{"1min":2.6579999999999999,"5min":33.195,"15min":33.195},"last":0.73799999999999999},{"interface":"front","average":{"1min":1.131,"5min":1.2250000000000001,"15min":0.879},"min":{"1min":0.53800000000000003,"5min":0.51800000000000002,"15min":0.41199999999999998},"max":{"1min":3.4969999999999999,"5min":32.965000000000003,"15min":32.965000000000003},"last":1.5780000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:31:24 2021","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.1950000000000001,"15min":0.90600000000000003},"min":{"1min":0.57599999999999996,"5min":0.53800000000000003,"15min":0.48699999999999999},"max":{"1min":3.3519999999999999,"5min":33.231999999999999,"15min":33.231999999999999},"last":1.242},{"interface":"front","average":{"1min":1.294,"5min":1.2829999999999999,"15min":0.95899999999999996},"min":{"1min":0.73299999999999998,"5min":0.50800000000000001,"15min":0.46000000000000002},"max":{"1min":2.8519999999999999,"5min":33.064,"15min":33.064},"last":1.0389999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:31:38 2021","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.085,"15min":0.97099999999999997},"min":{"1min":0.54700000000000004,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":3.883,"5min":3.883,"15min":3.883},"last":1.167},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.1060000000000001,"15min":1.0449999999999999},"min":{"1min":0.60699999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":3.9129999999999998,"5min":3.9129999999999998,"15min":3.9129999999999998},"last":1.341}]},{"osd":4,"last update":"Wed Feb 10 04:32:00 2021","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.214,"15min":1.171},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.859,"5min":3.0049999999999999,"15min":3.0049999999999999},"last":1.298},{"interface":"front","average":{"1min":1.274,"5min":1.2529999999999999,"15min":1.224},"min":{"1min":0.77200000000000002,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":2.5350000000000001,"5min":2.8130000000000002,"15min":2.8130000000000002},"last":1.6830000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:31:15 2021","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.167,"15min":1.2490000000000001},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":4.7969999999999997,"5min":4.7969999999999997,"15min":5.5860000000000003},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.806,"5min":1.431,"15min":1.5960000000000001},"min":{"1min":0.88100000000000001,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":5.2069999999999999,"5min":5.2069999999999999,"15min":9.5310000000000006},"last":1.4359999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:31:27 2021","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.3320000000000001,"15min":1.304},"min":{"1min":0.90400000000000003,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.6469999999999998,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":1.9139999999999999},{"interface":"front","average":{"1min":1.625,"5min":1.234,"15min":1.282},"min":{"1min":0.81599999999999995,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":4.5960000000000001,"5min":4.5960000000000001,"15min":4.5960000000000001},"last":1.21}]},{"osd":7,"last update":"Wed Feb 10 04:31:42 2021","interfaces":[{"interface":"back","average":{"1min":1.5660000000000001,"5min":1.47,"15min":1.633},"min":{"1min":1.0269999999999999,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":5.0099999999999998,"5min":5.0099999999999998,"15min":7.5540000000000003},"last":1.6140000000000001},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.4410000000000001,"15min":1.601},"min":{"1min":1.04,"5min":1.006,"15min":1.006},"max":{"1min":2.7570000000000001,"5min":2.7570000000000001,"15min":8.1020000000000003},"last":1.7909999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738476,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87412,"kb_used_data":60908,"kb_used_omap":0,"kb_used_meta":26496,"kb_avail":93657740,"statfs":{"total":95995035648,"available":95905525760,"internally_reserved":0,"allocated":62369792,"data_stored":63471791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27131904},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 04:32:01 2021","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.93600000000000005,"15min":0.71199999999999997},"min":{"1min":0.53200000000000003,"5min":0.48099999999999998,"15min":0.40799999999999997},"max":{"1min":1.722,"5min":5.125,"15min":5.125},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.97299999999999998,"15min":0.82199999999999995},"min":{"1min":0.53600000000000003,"5min":0.51200000000000001,"15min":0.45000000000000001},"max":{"1min":2.306,"5min":7.5549999999999997,"15min":7.5549999999999997},"last":0.81000000000000005}]},{"osd":2,"last update":"Wed Feb 10 04:31:21 2021","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.044,"15min":0.86299999999999999},"min":{"1min":0.66400000000000003,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.46,"5min":3.5019999999999998,"15min":3.5019999999999998},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.151,"5min":0.91000000000000003,"15min":0.83399999999999996},"min":{"1min":0.59499999999999997,"5min":0.51300000000000001,"15min":0.495},"max":{"1min":2.7280000000000002,"5min":2.9279999999999999,"15min":2.9279999999999999},"last":0.96199999999999997}]},{"osd":3,"last update":"Wed Feb 10 04:31:35 2021","interfaces":[{"interface":"back","average":{"1min":1.784,"5min":2.3759999999999999,"15min":1.3380000000000001},"min":{"1min":0.72999999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":13.819000000000001,"5min":126.587,"15min":126.587},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.665,"5min":2.5430000000000001,"15min":1.546},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":9.9149999999999991,"5min":126.88200000000001,"15min":126.88200000000001},"last":1.569}]},{"osd":4,"last update":"Wed Feb 10 04:31:55 2021","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.159,"15min":1.1719999999999999},"min":{"1min":0.71099999999999997,"5min":0.621,"15min":0.621},"max":{"1min":2.532,"5min":2.6440000000000001,"15min":2.6440000000000001},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.663,"5min":1.3899999999999999,"15min":1.2350000000000001},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.70099999999999996},"max":{"1min":13.891,"5min":13.891,"15min":13.891},"last":0.93500000000000005}]},{"osd":5,"last update":"Wed Feb 10 04:32:09 2021","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.456,"15min":1.2949999999999999},"min":{"1min":0.79900000000000004,"5min":0.70599999999999996,"15min":0.69599999999999995},"max":{"1min":13.949999999999999,"5min":13.949999999999999,"15min":13.949999999999999},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.948,"5min":1.504,"15min":1.367},"min":{"1min":0.90900000000000003,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":13.964,"5min":13.964,"15min":13.964},"last":1.8029999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:31:31 2021","interfaces":[{"interface":"back","average":{"1min":2.1099999999999999,"5min":1.5529999999999999,"15min":1.391},"min":{"1min":1.075,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":14.063000000000001,"5min":14.063000000000001,"15min":14.063000000000001},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.524,"5min":1.45,"15min":1.3959999999999999},"min":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":2.4790000000000001,"5min":3.6720000000000002,"15min":3.6720000000000002},"last":1.696}]},{"osd":7,"last update":"Wed Feb 10 04:31:41 2021","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.3069999999999999,"15min":1.2769999999999999},"min":{"1min":0.97099999999999997,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":2.9249999999999998,"5min":2.9249999999999998,"15min":2.9249999999999998},"last":1.6060000000000001},{"interface":"front","average":{"1min":2.177,"5min":1.6379999999999999,"15min":1.46},"min":{"1min":0.98199999999999998,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":14.189,"5min":14.189,"15min":14.189},"last":1.3700000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411429,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":45944,"kb_used_data":25136,"kb_used_omap":0,"kb_used_meta":20800,"kb_avail":93699208,"statfs":{"total":95995035648,"available":95947988992,"internally_reserved":0,"allocated":25739264,"data_stored":27623267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21299200},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,6,2],"upper_bound":524288},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:31:20 2021","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.83499999999999996,"15min":0.73199999999999998},"min":{"1min":0.52700000000000002,"5min":0.496,"15min":0.48699999999999999},"max":{"1min":1.9079999999999999,"5min":2.222,"15min":2.222},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.85999999999999999,"15min":0.75900000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.46000000000000002},"max":{"1min":1.5680000000000001,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":0.83499999999999996}]},{"osd":1,"last update":"Wed Feb 10 04:31:20 2021","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":0.996,"15min":0.84299999999999997},"min":{"1min":0.64200000000000002,"5min":0.51200000000000001,"15min":0.495},"max":{"1min":2.1110000000000002,"5min":5.0650000000000004,"15min":5.0650000000000004},"last":1.117},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.80800000000000005,"15min":0.72699999999999998},"min":{"1min":0.55300000000000005,"5min":0.51200000000000001,"15min":0.45000000000000001},"max":{"1min":1.1299999999999999,"5min":2.6779999999999999,"15min":2.6779999999999999},"last":0.77500000000000002}]},{"osd":3,"last update":"Wed Feb 10 04:31:39 2021","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":0.98899999999999999,"15min":0.87},"min":{"1min":0.72699999999999998,"5min":0.56100000000000005,"15min":0.53500000000000003},"max":{"1min":1.6319999999999999,"5min":3.1160000000000001,"15min":3.1160000000000001},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.232,"15min":1.0609999999999999},"min":{"1min":1.0509999999999999,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.9890000000000001,"5min":7.0309999999999997,"15min":7.0309999999999997},"last":0.98199999999999998}]},{"osd":4,"last update":"Wed Feb 10 04:31:54 2021","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.292,"15min":1.224},"min":{"1min":0.752,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.0350000000000001,"5min":2.089,"15min":2.089},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.538,"5min":1.258,"15min":1.1659999999999999},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":9.7609999999999992,"5min":9.7609999999999992,"15min":9.7609999999999992},"last":0.93600000000000005}]},{"osd":5,"last update":"Wed Feb 10 04:31:12 2021","interfaces":[{"interface":"back","average":{"1min":1.5309999999999999,"5min":1.3380000000000001,"15min":1.3240000000000001},"min":{"1min":1.042,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":2.29,"5min":2.29,"15min":2.29},"last":1.456},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.252,"15min":1.2030000000000001},"min":{"1min":0.85999999999999999,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":2.1389999999999998,"5min":2.4089999999999998,"15min":2.4089999999999998},"last":1.952}]},{"osd":6,"last update":"Wed Feb 10 04:31:30 2021","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.353,"15min":1.3220000000000001},"min":{"1min":1.0329999999999999,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.8029999999999999,"5min":5.2430000000000003,"15min":5.2430000000000003},"last":1.5},{"interface":"front","average":{"1min":1.6719999999999999,"5min":1.468,"15min":1.454},"min":{"1min":1.196,"5min":1.006,"15min":1.006},"max":{"1min":2.181,"5min":2.6629999999999998,"15min":2.6629999999999998},"last":1.611}]},{"osd":7,"last update":"Wed Feb 10 04:31:46 2021","interfaces":[{"interface":"back","average":{"1min":1.7050000000000001,"5min":1.5149999999999999,"15min":1.5089999999999999},"min":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"max":{"1min":2.6560000000000001,"5min":2.6560000000000001,"15min":2.6560000000000001},"last":1.75},{"interface":"front","average":{"1min":1.6830000000000001,"5min":1.343,"15min":1.296},"min":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":9.9220000000000006,"5min":9.9220000000000006,"15min":9.9220000000000006},"last":1.244}]}]},{"osd":3,"up_from":25,"seq":107374182495,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":92544,"kb_used_data":63608,"kb_used_omap":0,"kb_used_meta":28928,"kb_avail":93652608,"statfs":{"total":95995035648,"available":95900270592,"internally_reserved":0,"allocated":65134592,"data_stored":69049858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29622272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:31:41 2021","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.046,"15min":0.84099999999999997},"min":{"1min":0.62,"5min":0.49199999999999999,"15min":0.46100000000000002},"max":{"1min":1.6970000000000001,"5min":17.233000000000001,"15min":17.233000000000001},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.88500000000000001,"5min":1.3340000000000001,"15min":1.0189999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.5529999999999999,"5min":20.216999999999999,"15min":20.216999999999999},"last":0.70999999999999996}]},{"osd":1,"last update":"Wed Feb 10 04:31:41 2021","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.0389999999999999,"15min":0.92900000000000005},"min":{"1min":0.71099999999999997,"5min":0.54400000000000004,"15min":0.52800000000000002},"max":{"1min":2.1379999999999999,"5min":5.9130000000000003,"15min":5.9130000000000003},"last":1.034},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.97099999999999997,"15min":0.89000000000000001},"min":{"1min":0.55700000000000005,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.4610000000000001,"5min":5.4039999999999999,"15min":5.4039999999999999},"last":0.82099999999999995}]},{"osd":2,"last update":"Wed Feb 10 04:31:41 2021","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1060000000000001,"15min":1.002},"min":{"1min":0.66200000000000003,"5min":0.59999999999999998,"15min":0.58699999999999997},"max":{"1min":1.609,"5min":4.649,"15min":4.649},"last":0.94799999999999995},{"interface":"front","average":{"1min":1.155,"5min":0.999,"15min":0.98699999999999999},"min":{"1min":0.76000000000000001,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.9890000000000001,"5min":2.3570000000000002,"15min":2.3570000000000002},"last":0.85399999999999998}]},{"osd":4,"last update":"Wed Feb 10 04:31:50 2021","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.6779999999999999,"15min":1.3620000000000001},"min":{"1min":0.90400000000000003,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.748,"5min":24.882999999999999,"15min":24.882999999999999},"last":1.458},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.6579999999999999,"15min":1.3720000000000001},"min":{"1min":0.98699999999999999,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.871,"5min":24.911999999999999,"15min":24.911999999999999},"last":1.542}]},{"osd":5,"last update":"Wed Feb 10 04:31:12 2021","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.8129999999999999,"15min":1.385},"min":{"1min":0.88300000000000001,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":2.8220000000000001,"5min":25.004999999999999,"15min":25.004999999999999},"last":1.3400000000000001},{"interface":"front","average":{"1min":1.607,"5min":1.766,"15min":1.4279999999999999},"min":{"1min":0.93400000000000005,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":2.6739999999999999,"5min":25.055,"15min":25.055},"last":1.371}]},{"osd":6,"last update":"Wed Feb 10 04:31:31 2021","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.958,"15min":1.5820000000000001},"min":{"1min":1.101,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":2.2170000000000001,"5min":25.143000000000001,"15min":25.143000000000001},"last":1.5669999999999999},{"interface":"front","average":{"1min":1.722,"5min":1.877,"15min":1.5980000000000001},"min":{"1min":1.206,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":2.1880000000000002,"5min":25.181000000000001,"15min":25.181000000000001},"last":1.7509999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:31:46 2021","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.9630000000000001,"15min":1.6559999999999999},"min":{"1min":1.169,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":1.996,"5min":25.242999999999999,"15min":25.242999999999999},"last":1.6639999999999999},{"interface":"front","average":{"1min":1.7030000000000001,"5min":1.968,"15min":1.694},"min":{"1min":1.284,"5min":1.0209999999999999,"15min":1.0209999999999999},"max":{"1min":2.3500000000000001,"5min":25.286999999999999,"15min":25.286999999999999},"last":1.6479999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986269,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":45872,"kb_used_data":26280,"kb_used_omap":0,"kb_used_meta":19584,"kb_avail":93699280,"statfs":{"total":95995035648,"available":95948062720,"internally_reserved":0,"allocated":26910720,"data_stored":29253338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20054016},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4],"upper_bound":524288},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:31:56 2021","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":1.504,"15min":1.1699999999999999},"min":{"1min":0.51000000000000001,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.363,"5min":62.273000000000003,"15min":62.273000000000003},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.86399999999999999,"5min":1.575,"15min":1.177},"min":{"1min":0.57499999999999996,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.669,"5min":62.375999999999998,"15min":62.375999999999998},"last":0.80000000000000004}]},{"osd":1,"last update":"Wed Feb 10 04:31:56 2021","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.85699999999999998,"15min":0.89700000000000002},"min":{"1min":0.60399999999999998,"5min":0.58199999999999996,"15min":0.55400000000000005},"max":{"1min":1.3149999999999999,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.0920000000000001,"15min":1.042},"min":{"1min":0.76000000000000001,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.2909999999999999,"5min":2.7549999999999999,"15min":2.7549999999999999},"last":1.5449999999999999}]},{"osd":2,"last update":"Wed Feb 10 04:31:56 2021","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.9670000000000001,"15min":1.3759999999999999},"min":{"1min":0.71999999999999997,"5min":0.69399999999999995,"15min":0.57899999999999996},"max":{"1min":1.3600000000000001,"5min":62.521000000000001,"15min":62.521000000000001},"last":1.127},{"interface":"front","average":{"1min":1.127,"5min":2.04,"15min":1.456},"min":{"1min":0.80100000000000005,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.796,"5min":62.572000000000003,"15min":62.572000000000003},"last":1.917}]},{"osd":3,"last update":"Wed Feb 10 04:31:56 2021","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.9379999999999999,"15min":1.3480000000000001},"min":{"1min":0.83299999999999996,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":2.129,"5min":62.113,"15min":62.113},"last":2.4540000000000002},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.8899999999999999,"15min":1.359},"min":{"1min":0.94499999999999995,"5min":0.73699999999999999,"15min":0.69899999999999995},"max":{"1min":1.97,"5min":62.146000000000001,"15min":62.146000000000001},"last":1.444}]},{"osd":5,"last update":"Wed Feb 10 04:31:14 2021","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.1850000000000001,"15min":0.93799999999999994},"min":{"1min":0.64100000000000001,"5min":0.63200000000000001,"15min":0.57499999999999996},"max":{"1min":2.544,"5min":2.544,"15min":2.544},"last":1.276},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.1259999999999999,"15min":0.92200000000000004},"min":{"1min":0.73799999999999999,"5min":0.66800000000000004,"15min":0.624},"max":{"1min":2.0150000000000001,"5min":2.0150000000000001,"15min":2.0150000000000001},"last":1.6579999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:31:27 2021","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.6819999999999999,"15min":1.264},"min":{"1min":1.012,"5min":0.79200000000000004,"15min":0.63100000000000001},"max":{"1min":2.9700000000000002,"5min":29.686,"15min":29.686},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.448,"5min":1.802,"15min":1.429},"min":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.79000000000000004},"max":{"1min":2.6960000000000002,"5min":29.510000000000002,"15min":29.510000000000002},"last":1.776}]},{"osd":7,"last update":"Wed Feb 10 04:31:46 2021","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.3859999999999999,"15min":1.3100000000000001},"min":{"1min":1.0289999999999999,"5min":0.89600000000000002,"15min":0.88600000000000001},"max":{"1min":1.54,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.5149999999999999,"15min":1.4219999999999999},"min":{"1min":1.087,"5min":0.91200000000000003,"15min":0.83099999999999996},"max":{"1min":2.105,"5min":3.4830000000000001,"15min":3.4830000000000001},"last":1.6000000000000001}]}]},{"osd":5,"up_from":37,"seq":158913790041,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":48896,"kb_used_data":26484,"kb_used_omap":0,"kb_used_meta":19520,"kb_avail":93696256,"statfs":{"total":95995035648,"available":95944966144,"internally_reserved":0,"allocated":27119616,"data_stored":26986836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19988480},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:31:08 2021","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":0.98899999999999999,"15min":0.998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":8.4830000000000005,"5min":8.4830000000000005,"15min":8.4830000000000005},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.198,"5min":0.97199999999999998,"15min":0.93200000000000005},"min":{"1min":0.53000000000000003,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":5.6340000000000003,"5min":5.6340000000000003,"15min":5.6340000000000003},"last":1.248}]},{"osd":1,"last update":"Wed Feb 10 04:31:08 2021","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.89900000000000002,"15min":0.92200000000000004},"min":{"1min":0.54200000000000004,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2.7879999999999998,"5min":2.7879999999999998,"15min":2.7879999999999998},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.0229999999999999,"15min":1.01},"min":{"1min":0.56599999999999995,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":5.7679999999999998,"5min":5.7679999999999998,"15min":5.7679999999999998},"last":1.454}]},{"osd":2,"last update":"Wed Feb 10 04:31:08 2021","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.946,"15min":1.3120000000000001},"min":{"1min":0.60999999999999999,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":7.9980000000000002,"5min":67.852000000000004,"15min":67.852000000000004},"last":1.2709999999999999},{"interface":"front","average":{"1min":0.92500000000000004,"5min":1.8049999999999999,"15min":1.2549999999999999},"min":{"1min":0.66700000000000004,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.462,"5min":67.548000000000002,"15min":67.548000000000002},"last":1.4119999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:31:08 2021","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":2.169,"15min":1.4490000000000001},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":8.3699999999999992,"5min":67.376999999999995,"15min":67.376999999999995},"last":1.494},{"interface":"front","average":{"1min":1.129,"5min":2.0539999999999998,"15min":1.397},"min":{"1min":0.85699999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.52,"5min":66.888999999999996,"15min":66.888999999999996},"last":1.528}]},{"osd":4,"last update":"Wed Feb 10 04:31:08 2021","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.151,"15min":0.98199999999999998},"min":{"1min":0.746,"5min":0.65300000000000002,"15min":0.63900000000000001},"max":{"1min":5.2290000000000001,"5min":5.2290000000000001,"15min":5.2290000000000001},"last":1.881},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.3660000000000001,"15min":1.0860000000000001},"min":{"1min":0.73299999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":8.1449999999999996,"5min":8.1449999999999996,"15min":8.1449999999999996},"last":1.712}]},{"osd":6,"last update":"Wed Feb 10 04:31:29 2021","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.5,"15min":1.236},"min":{"1min":0.84499999999999997,"5min":0.77400000000000002,"15min":0.70599999999999996},"max":{"1min":8.2569999999999997,"5min":8.2569999999999997,"15min":8.2569999999999997},"last":1.9379999999999999},{"interface":"front","average":{"1min":1.222,"5min":1.2949999999999999,"15min":1.125},"min":{"1min":0.90300000000000002,"5min":0.77800000000000002,"15min":0.74399999999999999},"max":{"1min":1.5680000000000001,"5min":2.931,"15min":2.931},"last":1.841}]},{"osd":7,"last update":"Wed Feb 10 04:31:45 2021","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.367,"15min":1.236},"min":{"1min":0.91700000000000004,"5min":0.879,"15min":0.78400000000000003},"max":{"1min":5.4770000000000003,"5min":5.4770000000000003,"15min":5.4770000000000003},"last":1.7829999999999999},{"interface":"front","average":{"1min":1.778,"5min":1.613,"15min":1.4430000000000001},"min":{"1min":0.96999999999999997,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":8.7840000000000007,"5min":8.7840000000000007,"15min":8.7840000000000007},"last":2.161}]}]}]}} 2021-02-10T04:32:20.508 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2021-02-10T04:32:20.508 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: cluster 2021-02-10T04:32:18.738624+0000 osd.6 (osd.6 2021-02-10T04:32:20.508 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: ) 51 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:32:20.508 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: cluster 2021-02-10T04:32:18.738958+0000 osd.6 (osd.6) 52 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:32:20.508 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: cluster 2021-02-10T04:32:19.178967+0000 osd.2 (osd 2021-02-10T04:32:20.508 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: .2) 344 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:20.508 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:20.508 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: -02-10T04:32:19.178975+0000 osd.2 2021-02-10T04:32:20.508 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: (osd.2) 345 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:20.508 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:20.508 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: -02-10T04:32:19.178981+0000 osd.2 2021-02-10T04:32:20.509 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: (osd.2) 346 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:20.509 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:20.509 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: -10T04:32:19.178987+0000 osd.2 ( 2021-02-10T04:32:20.509 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: osd.2) 347 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:20.509 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:20.509 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: -10T04:32:19.178992+0000 osd.2 (osd 2021-02-10T04:32:20.509 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: .2) 348 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:20.509 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:20.509 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: 10T04:32:19.178998+0000 osd.2 (osd. 2021-02-10T04:32:20.509 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: 2) 349 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:20.509 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:20.509 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: -10T04:32:19.179003+0000 osd.2 ( 2021-02-10T04:32:20.509 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: osd.2) 350 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:20.510 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:20.510 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: -02-10T04:32:19.179035+0000 2021-02-10T04:32:20.510 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: osd.2 (osd.2) 351 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:20.514 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:20.514 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: -10T04:32:19.270645+0000 osd.1 ( 2021-02-10T04:32:20.514 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: osd.1) 305 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:20.514 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:20.515 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: cluster 2021-02-10T04:32:18.738624+0000 2021-02-10T04:32:20.515 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: osd.6 (osd.6) 51 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:32:20.515 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:20.515 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: :32:18.738958+0000 osd.6 (osd.6) 52 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:32:20.515 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:20.515 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: 02-10T04:32:19.178967+0000 osd.2 ( 2021-02-10T04:32:20.515 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: osd.2) 344 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:20.515 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:20.515 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: 04:32:19.178975+0000 osd.2 (osd.2) 2021-02-10T04:32:20.515 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: 345 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:20.515 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: cluster 2021-02-10T04:32 2021-02-10T04:32:20.515 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: :19.178981+0000 osd.2 (osd.2) 346 : cluster 2021-02-10T04:32:20.515 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:20.515 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: cluster 2021-02-10T04:32:19.178987 2021-02-10T04:32:20.516 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 347 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:20.516 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: 2021-02-10T04:32:20.516 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: cluster 2021-02-10T04:32:20.516 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: 2021-02-10T04:32:19.178992+0000 osd.2 (osd.2 2021-02-10T04:32:20.516 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: ) 348 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:20.516 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: cluster 2021-02-10T04:32 2021-02-10T04:32:20.516 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: :19.178998+0000 osd.2 (osd.2) 349 2021-02-10T04:32:20.516 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:20.516 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: cluster 2021-02-10T04:32: 2021-02-10T04:32:20.516 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: 19.179003+0000 osd.2 (osd.2) 350 : cluster 2021-02-10T04:32:20.516 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:20.516 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: cluster 2021-02-10T04:32:19. 2021-02-10T04:32:20.516 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: 179035+0000 osd.2 (osd.2) 2021-02-10T04:32:20.517 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: 351 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:20.517 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:20.517 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: 02-10T04:32:19.270645+0000 2021-02-10T04:32:20.517 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: osd.1 (osd.1) 305 : 2021-02-10T04:32:20.517 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:32:20.517 INFO:tasks.thrashosds.thrasher:inject_pause on 0 2021-02-10T04:32:20.517 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2021-02-10T04:32:20.517 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2021-02-10T04:32:20.517 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.0 config set filestore_inject_stall 3 2021-02-10T04:32:20.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: 02-10T04:32:19.270655+0000 2021-02-10T04:32:20.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: osd.1 (osd.1) 306 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:20.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: 2021-02-10T04:32:20.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: cluster 2021-02-10T04:32:20.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: 2021-02-10T04:32:19.270658+0000 osd.1 (osd.1) 307 : cluster 2021-02-10T04:32:20.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:20.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: cluster 2021-02-10T04:32:20.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: 2021-02-10T04:32:19.270662+0000 osd.1 (osd.1 2021-02-10T04:32:20.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: ) 308 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:20.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: cluster 2021-02-10T04:32:19.270667+0000 osd.1 (osd 2021-02-10T04:32:20.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: .1) 309 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:20.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:32:20.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: 32:19.270681+0000 osd.1 (osd.1) 2021-02-10T04:32:20.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: 310 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:20.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:20.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: 02-10T04:32:19.329199+0000 2021-02-10T04:32:20.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: mon.a (mon.0) 1073 : cluster [DBG] osdmap e322: 8 total, 8 up, 8 in 2021-02-10T04:32:20.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: 2021-02-10T04:32:20.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: cluster 2021-02-10T04:32:20.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:32:20.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: T04:32:19.443640+0000 osd.4 (osd. 2021-02-10T04:32:20.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: 4) 289 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:20.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: cluster 2021-02-10T04:32:20.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:32:20.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: 04:32:19.443652+0000 osd.4 (osd.4) 290 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:20.521 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:20.521 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: 02-10T04:32:19.443659+0000 osd.4 (osd.4) 2021-02-10T04:32:20.521 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: 291 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:20.521 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:20.521 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: -10T04:32:19.443669+0000 osd.4 2021-02-10T04:32:20.521 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: (osd.4) 292 : cluster 2021-02-10T04:32:20.521 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:20.521 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:20.521 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: -02-10T04:32:20.054868+0000 2021-02-10T04:32:20.521 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: mgr.y (mgr.14140) 477 : 2021-02-10T04:32:20.521 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[29319]: cluster [DBG] pgmap v633: 168 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 1 active+clean+scrubbing, 164 active+clean; 160 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 8 op/s 2021-02-10T04:32:20.521 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[45443]: debug 2021-02-10T04:32:20.241+0000 7f7140d73700 -1 osd.1 322 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:32:20.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:20.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:20.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: -02-10T04:32: 2021-02-10T04:32:20.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: 19.270655+0000 osd.1 (osd.1 2021-02-10T04:32:20.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: ) 306 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:20.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:20.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: 02-10T04:32:19. 2021-02-10T04:32:20.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: 270658+0000 osd.1 (osd.1) 2021-02-10T04:32:20.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: 307 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:20.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: cluster 2021-02-10T04:32:19.270662+0000 osd.1 (osd. 2021-02-10T04:32:20.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: 1) 308 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:20.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: cluster 2021-02-10T04:32:20.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: 2021-02-10T04:32:19.270667+0000 2021-02-10T04:32:20.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: osd.1 (osd.1) 309 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:20.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:32:20.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: T04:32:19.270681+0000 osd.1 (osd 2021-02-10T04:32:20.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: .1) 310 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:20.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:20.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: 02-10T04:32:19.329199+0000 2021-02-10T04:32:20.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: mon.a (mon.0) 1073 : cluster 2021-02-10T04:32:20.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: [DBG] osdmap e322: 8 total, 8 up, 8 in 2021-02-10T04:32:20.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: cluster 2021-02-10T04:32: 2021-02-10T04:32:20.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: 19.443640+0000 osd.4 (osd.4) 289 : cluster 2021-02-10T04:32:20.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:20.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: cluster 2021-02-10T04:32 2021-02-10T04:32:20.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: :19.443652+0000 osd.4 (osd.4) 290 : 2021-02-10T04:32:20.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:20.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: cluster 2021-02-10T04:32 2021-02-10T04:32:20.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: :19.443659+0000 osd.4 (osd.4) 291 2021-02-10T04:32:20.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:20.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:20.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: :32:19.443669+0000 osd.4 (osd.4 2021-02-10T04:32:20.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: ) 292 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:20.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:20.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: 10T04:32:20.054868+0000 mgr.y ( 2021-02-10T04:32:20.524 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[35961]: mgr.14140) 477 : cluster [DBG] pgmap v633: 168 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 1 active+clean+scrubbing, 164 active+clean; 160 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 8 op/s 2021-02-10T04:32:20.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[33677]: debug 2021-02-10T04:32:20.483+0000 7fbec19a1700 -1 osd.4 323 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:32:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: cluster 2021-02-10T04:32:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:32:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 32:18.738624+0000 osd.6 (osd.6) 51 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:32:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: cluster 2021-02-10T04:32: 2021-02-10T04:32:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 18.738958+0000 osd.6 (osd.6) 52 : cluster [DBG] 2021-02-10T04:32:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: purged_snaps scrub ok 2021-02-10T04:32:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: cluster 2021-02-10T04:32: 2021-02-10T04:32:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 19.178967+0000 osd.2 (osd.2) 344 : 2021-02-10T04:32:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: cluster 2021-02-10T04:32:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 2021-02-10T04:32:19.178975+0000 osd.2 (osd.2) 345 : cluster 2021-02-10T04:32:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: cluster 2021-02-10T04:32:19. 2021-02-10T04:32:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 178981+0000 osd.2 (osd.2) 346 2021-02-10T04:32:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: cluster 2021-02-10T04:32:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 2021-02-10T04:32:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 2021 2021-02-10T04:32:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: - 2021-02-10T04:32:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 02 2021-02-10T04:32:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: - 2021-02-10T04:32:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 10 2021-02-10T04:32:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: T 2021-02-10T04:32:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 04 2021-02-10T04:32:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: : 2021-02-10T04:32:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 32 2021-02-10T04:32:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: : 2021-02-10T04:32:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 19 2021-02-10T04:32:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: . 2021-02-10T04:32:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 178987 2021-02-10T04:32:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: +0000 2021-02-10T04:32:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 2021-02-10T04:32:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: osd.2 2021-02-10T04:32:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: ( 2021-02-10T04:32:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: osd 2021-02-10T04:32:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: . 2021-02-10T04:32:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 2 2021-02-10T04:32:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: ) 2021-02-10T04:32:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 347 2021-02-10T04:32:20.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: : 2021-02-10T04:32:20.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: cluster 2021-02-10T04:32:20.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 2021-02-10T04:32:20.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:20.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: cluster 2021-02-10T04:32:20.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 2021-02-10T04:32:19.178992+0000 2021-02-10T04:32:20.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: osd.2 (osd.2) 348 : cluster 2021-02-10T04:32:20.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:20.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: cluster 2021-02-10T04:32:20.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 2021-02-10T04:32:20.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 2021 2021-02-10T04:32:20.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: - 2021-02-10T04:32:20.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 02 2021-02-10T04:32:20.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: -10T04:32: 2021-02-10T04:32:20.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 19.178998+0000 osd.2 (osd.2) 349 : cluster 2021-02-10T04:32:20.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:20.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: cluster 2021-02-10T04:32:20.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 2021-02-10T04:32:20.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 2021 2021-02-10T04:32:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: - 2021-02-10T04:32:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 02 2021-02-10T04:32:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: - 2021-02-10T04:32:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 10 2021-02-10T04:32:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: T 2021-02-10T04:32:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 04 2021-02-10T04:32:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: : 2021-02-10T04:32:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 32 2021-02-10T04:32:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: : 2021-02-10T04:32:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 19 2021-02-10T04:32:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: . 2021-02-10T04:32:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 179003 2021-02-10T04:32:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: +0000 2021-02-10T04:32:20.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 2021-02-10T04:32:20.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: osd.2 2021-02-10T04:32:20.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: ( 2021-02-10T04:32:20.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: osd 2021-02-10T04:32:20.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: . 2021-02-10T04:32:20.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 2 2021-02-10T04:32:20.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: ) 2021-02-10T04:32:20.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 350 2021-02-10T04:32:20.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: : 2021-02-10T04:32:20.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: cluster 2021-02-10T04:32:20.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 2021-02-10T04:32:20.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: [WRN] 2021-02-10T04:32:20.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 2021-02-10T04:32:20.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:20.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 2021-02-10T04:32:20.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: cluster 2021-02-10T04:32:20.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 2021-02-10T04:32:20.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 2021 2021-02-10T04:32:20.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: - 2021-02-10T04:32:20.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 02 2021-02-10T04:32:20.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: - 2021-02-10T04:32:20.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 10 2021-02-10T04:32:20.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: T 2021-02-10T04:32:20.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 04 2021-02-10T04:32:20.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: : 2021-02-10T04:32:20.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 32 2021-02-10T04:32:20.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: : 2021-02-10T04:32:20.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 19 2021-02-10T04:32:20.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: . 2021-02-10T04:32:20.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 179035 2021-02-10T04:32:20.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: +0000 2021-02-10T04:32:20.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 2021-02-10T04:32:20.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: osd.2 2021-02-10T04:32:20.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: ( 2021-02-10T04:32:20.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: osd 2021-02-10T04:32:20.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: . 2021-02-10T04:32:20.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 2 2021-02-10T04:32:20.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: ) 2021-02-10T04:32:20.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 351 2021-02-10T04:32:20.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: : 2021-02-10T04:32:20.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: cluster 2021-02-10T04:32:20.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 2021-02-10T04:32:20.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: [WRN] 2021-02-10T04:32:20.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 2021-02-10T04:32:20.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:20.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 2021-02-10T04:32:20.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: cluster 2021-02-10T04:32:20.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 2021-02-10T04:32:20.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 2021 2021-02-10T04:32:20.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: - 2021-02-10T04:32:20.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 02 2021-02-10T04:32:20.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: - 2021-02-10T04:32:20.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 10 2021-02-10T04:32:20.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: T 2021-02-10T04:32:20.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 04 2021-02-10T04:32:20.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: : 2021-02-10T04:32:20.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 32 2021-02-10T04:32:20.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: : 2021-02-10T04:32:20.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 19 2021-02-10T04:32:20.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: . 2021-02-10T04:32:20.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 270645 2021-02-10T04:32:20.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: +0000 2021-02-10T04:32:20.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 2021-02-10T04:32:20.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: osd.1 2021-02-10T04:32:20.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: ( 2021-02-10T04:32:20.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: osd 2021-02-10T04:32:20.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: . 2021-02-10T04:32:20.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 1 2021-02-10T04:32:20.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: ) 2021-02-10T04:32:20.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 305 2021-02-10T04:32:20.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: : 2021-02-10T04:32:20.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:20.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: cluster 2021-02-10T04:32: 2021-02-10T04:32:20.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 19.270655+0000 osd.1 (osd.1) 306 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:20.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:20.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 02-10T04:32:19.270658+0000 osd.1 2021-02-10T04:32:20.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: (osd.1) 307 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:20.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:20.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: -10T04:32:19.270662+0000 osd.1 ( 2021-02-10T04:32:20.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: osd.1) 308 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:20.656 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:20.656 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: T04:32:19.270667+0000 osd.1 ( 2021-02-10T04:32:20.656 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: osd.1) 309 : cluster [WRN] 2021-02-10T04:32:20.656 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:20.656 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: cluster 2021-02-10T04:32 2021-02-10T04:32:20.656 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: :19.270681+0000 osd.1 (osd.1) 310 : cluster 2021-02-10T04:32:20.656 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:20.656 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: cluster 2021-02-10T04:32:20.656 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 2021-02-10T04:32:19.329199+0000 mon.a (mon.0) 1073 2021-02-10T04:32:20.656 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: : cluster [DBG] osdmap e322: 8 total, 8 up, 8 in 2021-02-10T04:32:20.656 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:32:20.656 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 32:19.443640+0000 osd.4 (osd.4) 289 : 2021-02-10T04:32:20.656 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:20.656 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: cluster 2021-02-10T04:32: 2021-02-10T04:32:20.657 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 19.443652+0000 osd.4 (osd.4) 290 : 2021-02-10T04:32:20.657 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:20.657 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: cluster 2021-02-10T04:32:19 2021-02-10T04:32:20.657 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: .443659+0000 osd.4 (osd.4) 291 : cluster [WRN] 2021-02-10T04:32:20.657 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:20.657 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: cluster 2021-02-10T04:32:19. 2021-02-10T04:32:20.657 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: 443669+0000 osd.4 (osd.4) 292 : cluster [WRN] 2021-02-10T04:32:20.657 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:20.657 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: cluster 2021-02-10T04:32:20.054868 2021-02-10T04:32:20.657 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 477 : cluster [DBG] 2021-02-10T04:32:20.657 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[29288]: pgmap v633: 168 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 1 active+clean+scrubbing, 164 active+clean; 160 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 8 op/s 2021-02-10T04:32:20.751 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:20 smithi085 conmon[49106]: debug 2021-02-10T04:32:20.749+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:20.784 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:32:20.821 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:32:20.889 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:32:20.916 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:21.025 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:32:21.130 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:32:21.155 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:21.219 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:32:20 smithi164 conmon[37263]: debug 2021-02-10T04:32:20.989+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:21.241 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:32:21.346 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:32:21.349 INFO:tasks.rados.rados.0.smithi085.stdout:1164: done (14 left) 2021-02-10T04:32:21.349 INFO:tasks.rados.rados.0.smithi085.stdout:1165: done (13 left) 2021-02-10T04:32:21.349 INFO:tasks.rados.rados.0.smithi085.stdout:1166: delete oid 7 current snap is 116 2021-02-10T04:32:21.358 INFO:tasks.rados.rados.0.smithi085.stdout:1166: done (13 left) 2021-02-10T04:32:21.358 INFO:tasks.rados.rados.0.smithi085.stdout:1167: rollback oid 6 current snap is 116 2021-02-10T04:32:21.358 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 6 to 115 2021-02-10T04:32:21.358 INFO:tasks.rados.rados.0.smithi085.stdout:1168: write (excl) oid 22 current snap is 116 2021-02-10T04:32:21.358 INFO:tasks.rados.rados.0.smithi085.stdout:1168: seq_num 436 ranges {527852=640685,1680502=589135,2757013=570839} 2021-02-10T04:32:21.364 INFO:tasks.rados.rados.0.smithi085.stdout:1168: writing smithi08575397-22 from 527852 to 1168537 tid 1 2021-02-10T04:32:21.372 INFO:tasks.rados.rados.0.smithi085.stdout:1168: writing smithi08575397-22 from 1680502 to 2269637 tid 2 2021-02-10T04:32:21.373 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:21.379 INFO:tasks.rados.rados.0.smithi085.stdout:1168: writing smithi08575397-22 from 2757013 to 3327852 tid 3 2021-02-10T04:32:21.379 INFO:tasks.rados.rados.0.smithi085.stdout:1169: write (excl) oid 3 current snap is 116 2021-02-10T04:32:21.379 INFO:tasks.rados.rados.0.smithi085.stdout:1167: finishing rollback tid 0 to smithi08575397-6 2021-02-10T04:32:21.379 INFO:tasks.rados.rados.0.smithi085.stdout:1167: finishing rollback tid 1 to smithi08575397-6 2021-02-10T04:32:21.379 INFO:tasks.rados.rados.0.smithi085.stdout:1167: finishing rollback tid 2 to smithi08575397-6 2021-02-10T04:32:21.379 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 397 (ObjNum 419 snap 109 seq_num 419) dirty exists 2021-02-10T04:32:21.379 INFO:tasks.rados.rados.0.smithi085.stdout:1168: finishing write tid 1 to smithi08575397-22 2021-02-10T04:32:21.379 INFO:tasks.rados.rados.0.smithi085.stdout:1168: finishing write tid 2 to smithi08575397-22 2021-02-10T04:32:21.379 INFO:tasks.rados.rados.0.smithi085.stdout:1169: seq_num 437 ranges {544659=715934,1738984=449146,2784436=560223} 2021-02-10T04:32:21.386 INFO:tasks.rados.rados.0.smithi085.stdout:1169: writing smithi08575397-3 from 544659 to 1260593 tid 1 2021-02-10T04:32:21.392 INFO:tasks.rados.rados.0.smithi085.stdout:1169: writing smithi08575397-3 from 1738984 to 2188130 tid 2 2021-02-10T04:32:21.399 INFO:tasks.rados.rados.0.smithi085.stdout:1169: writing smithi08575397-3 from 2784436 to 3344659 tid 3 2021-02-10T04:32:21.400 INFO:tasks.rados.rados.0.smithi085.stdout:1167: done (15 left) 2021-02-10T04:32:21.400 INFO:tasks.rados.rados.0.smithi085.stdout:1170: copy_from oid 1 from oid 30 current snap is 116 2021-02-10T04:32:21.400 INFO:tasks.rados.rados.0.smithi085.stdout:1168: finishing write tid 3 to smithi08575397-22 2021-02-10T04:32:21.401 INFO:tasks.rados.rados.0.smithi085.stdout:1168: finishing write tid 4 to smithi08575397-22 2021-02-10T04:32:21.401 INFO:tasks.rados.rados.0.smithi085.stdout:1168: finishing write tid 5 to smithi08575397-22 2021-02-10T04:32:21.401 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 473 (ObjNum 436 snap 116 seq_num 436) dirty exists 2021-02-10T04:32:21.401 INFO:tasks.rados.rados.0.smithi085.stdout:1168: left oid 22 (ObjNum 436 snap 116 seq_num 436) 2021-02-10T04:32:21.401 INFO:tasks.rados.rados.0.smithi085.stdout:1169: finishing write tid 1 to smithi08575397-3 2021-02-10T04:32:21.401 INFO:tasks.rados.rados.0.smithi085.stdout:1168: done (15 left) 2021-02-10T04:32:21.401 INFO:tasks.rados.rados.0.smithi085.stdout:1171: read oid 7 snap -1 2021-02-10T04:32:21.401 INFO:tasks.rados.rados.0.smithi085.stdout:1171: expect deleted 2021-02-10T04:32:21.401 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:21.402 INFO:tasks.rados.rados.0.smithi085.stdout:1169: finishing write tid 2 to smithi08575397-3 2021-02-10T04:32:21.405 INFO:tasks.rados.rados.0.smithi085.stdout:1169: finishing write tid 3 to smithi08575397-3 2021-02-10T04:32:21.406 INFO:tasks.rados.rados.0.smithi085.stdout:1169: finishing write tid 4 to smithi08575397-3 2021-02-10T04:32:21.406 INFO:tasks.rados.rados.0.smithi085.stdout:1169: finishing write tid 5 to smithi08575397-3 2021-02-10T04:32:21.406 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 492 (ObjNum 437 snap 116 seq_num 437) dirty exists 2021-02-10T04:32:21.406 INFO:tasks.rados.rados.0.smithi085.stdout:1169: left oid 3 (ObjNum 437 snap 116 seq_num 437) 2021-02-10T04:32:21.407 INFO:tasks.rados.rados.0.smithi085.stdout:1169: done (15 left) 2021-02-10T04:32:21.407 INFO:tasks.rados.rados.0.smithi085.stdout:1172: write oid 2 current snap is 116 2021-02-10T04:32:21.407 INFO:tasks.rados.rados.0.smithi085.stdout:1172: seq_num 438 ranges {561466=791183,2081113=592804,3095506=265960} 2021-02-10T04:32:21.415 INFO:tasks.rados.rados.0.smithi085.stdout:1172: writing smithi08575397-2 from 561466 to 1352649 tid 1 2021-02-10T04:32:21.421 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[49106]: debug 2021-02-10T04:32:21.225+0000 7f77a3e94700 -1 osd.2 323 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:32:21.422 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[45443]: debug 2021-02-10T04:32:21.220+0000 7f7140d73700 -1 osd.1 323 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:32:21.423 INFO:tasks.rados.rados.0.smithi085.stdout:1172: writing smithi08575397-2 from 2081113 to 2673917 tid 2 2021-02-10T04:32:21.427 INFO:tasks.rados.rados.0.smithi085.stdout:1172: writing smithi08575397-2 from 3095506 to 3361466 tid 3 2021-02-10T04:32:21.427 INFO:tasks.rados.rados.0.smithi085.stdout:1172: finishing write tid 1 to smithi08575397-2 2021-02-10T04:32:21.427 INFO:tasks.rados.rados.0.smithi085.stdout:1172: finishing write tid 2 to smithi08575397-2 2021-02-10T04:32:21.427 INFO:tasks.rados.rados.0.smithi085.stdout:1171: done (15 left) 2021-02-10T04:32:21.428 INFO:tasks.rados.rados.0.smithi085.stdout:1173: snap_create 2021-02-10T04:32:21.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: cluster 2021-02-10T04:32:21.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 2021-02-10T04:32:21.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 2021 2021-02-10T04:32:21.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: - 2021-02-10T04:32:21.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 02 2021-02-10T04:32:21.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: - 2021-02-10T04:32:21.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 10 2021-02-10T04:32:21.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: T 2021-02-10T04:32:21.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 04 2021-02-10T04:32:21.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: : 2021-02-10T04:32:21.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 32 2021-02-10T04:32:21.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: : 2021-02-10T04:32:21.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 20 2021-02-10T04:32:21.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: . 2021-02-10T04:32:21.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 208827 2021-02-10T04:32:21.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: +0000 2021-02-10T04:32:21.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 2021-02-10T04:32:21.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: osd.2 2021-02-10T04:32:21.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: ( 2021-02-10T04:32:21.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: osd 2021-02-10T04:32:21.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: . 2021-02-10T04:32:21.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 2 2021-02-10T04:32:21.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: ) 2021-02-10T04:32:21.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 352 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:21.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: cluster 2021-02-10T04:32:20.208836 2021-02-10T04:32:21.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 353 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:21.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:32:21.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: T04:32:20.208844+0000 osd.2 (osd.2) 2021-02-10T04:32:21.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 354 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:21.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:32:21.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 32:20.208866+0000 osd.2 (osd.2) 355 : 2021-02-10T04:32:21.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:21.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:21.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: -02-10T04:32:20.208871+0000 2021-02-10T04:32:21.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: osd.2 (osd.2) 356 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:21.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:21.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: -02-10T04:32:20.208876+0000 osd.2 2021-02-10T04:32:21.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: (osd.2) 357 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:21.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:21.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: -10T04:32:20.208882+0000 osd.2 ( 2021-02-10T04:32:21.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: osd.2) 358 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:21.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:32:21.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: T04:32:20.208888+0000 osd.2 (osd. 2021-02-10T04:32:21.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 2) 359 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:21.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:21.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 10T04:32:20.243332+0000 osd.1 ( 2021-02-10T04:32:21.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: osd.1) 311 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:21.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:32:21.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: T04:32:20.243337+0000 osd.1 (osd. 2021-02-10T04:32:21.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 1) 312 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:21.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: cluster 2021-02-10T04:32:21.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 2021-02-10T04:32:21.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 2021 2021-02-10T04:32:21.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: - 2021-02-10T04:32:21.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 02 2021-02-10T04:32:21.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: - 2021-02-10T04:32:21.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 10 2021-02-10T04:32:21.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: T04:32:20.243342+0000 2021-02-10T04:32:21.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: osd.1 (osd.1) 313 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:21.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:21.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: -10T04:32:20.243345+0000 osd.1 2021-02-10T04:32:21.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: (osd.1) 314 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:21.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: cluster 2021-02-10T04:32:21.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 2021-02-10T04:32:20.243349+0000 osd.1 (osd.1) 2021-02-10T04:32:21.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 315 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:21.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: cluster 2021-02-10T04:32:21.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 2021-02-10T04:32 2021-02-10T04:32:21.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: :20.243353+0000 osd.1 (osd 2021-02-10T04:32:21.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: .1) 316 : cluster [WRN] 2021-02-10T04:32:21.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:21.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: cluster 2021-02-10T04:32:21.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:32:21.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 32:20.338842+0000 mon.a 2021-02-10T04:32:21.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: (mon.0) 1074 : cluster 2021-02-10T04:32:21.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: [DBG] osdmap e323: 8 total, 8 up, 8 in 2021-02-10T04:32:21.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: cluster 2021-02-10T04:32:21.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 2021-02-10T04:32 2021-02-10T04:32:21.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: :20.484129+0000 osd.4 ( 2021-02-10T04:32:21.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: osd.4) 293 : cluster [WRN] 2021-02-10T04:32:21.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:21.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: cluster 2021-02-10T04:32:21.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:32:21.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 32:20.484140+0000 osd.4 (osd. 2021-02-10T04:32:21.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 4) 294 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:21.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:21.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 04:32:20.484144+0000 osd.4 (osd.4) 2021-02-10T04:32:21.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 295 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:21.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:21.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 04:32:20.484148+0000 osd.4 (osd.4 2021-02-10T04:32:21.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: ) 296 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:21.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:21.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 04:32:21.226690+0000 osd.2 (osd.2 2021-02-10T04:32:21.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: ) 360 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:21.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:21.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: :32:21.226714+0000 osd.2 (osd.2) 2021-02-10T04:32:21.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 361 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:21.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: cluster 2021-02-10T04:32:21.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 2021-02-10T04:32:21.226722+0000 2021-02-10T04:32:21.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: osd.2 (osd.2) 362 : 2021-02-10T04:32:21.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:21.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: cluster 2021-02-10T04:32:21.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:32:21.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 32:21.226729+0000 osd.2 ( 2021-02-10T04:32:21.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: osd.2) 363 : cluster [WRN] 2021-02-10T04:32:21.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:21.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: cluster 2021-02-10T04:32:21.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 2021-02-10T04:32:21.226738 2021-02-10T04:32:21.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 364 : cluster [WRN] 2021-02-10T04:32:21.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:21.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: cluster 2021-02-10T04:32:21.226746 2021-02-10T04:32:21.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 365 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:21.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: 2021-02-10T04:32:21.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:21.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: -02-10T04:32:21.226751+0000 osd.2 (osd.2 2021-02-10T04:32:21.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: ) 366 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:21.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:21.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: -02-10T04:32:21.226757+0000 osd.2 2021-02-10T04:32:21.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[35961]: (osd.2) 367 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:21.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: cluster 2021-02-10T04:32:21.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: 2021-02-10T04:32:20. 2021-02-10T04:32:21.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: 208827+0000 osd.2 (osd.2) 352 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:21.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: cluster 2021-02-10T04:32:20.208836 2021-02-10T04:32:21.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 353 : cluster [WRN] 2021-02-10T04:32:21.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:21.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: cluster 2021-02-10T04:32 2021-02-10T04:32:21.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: :20.208844+0000 osd.2 (osd.2) 354 : 2021-02-10T04:32:21.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:21.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:32:21.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: 04:32:20.208866+0000 osd.2 (osd.2) 2021-02-10T04:32:21.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: 355 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:21.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:21.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: 02-10T04:32:20.208871+0000 osd.2 2021-02-10T04:32:21.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: (osd.2) 356 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:21.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: cluster 2021-02-10T04:32:21.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: 2021-02-10T04:32:20.208876+0000 2021-02-10T04:32:21.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: osd.2 (osd.2) 357 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:21.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: 2021-02-10T04:32:21.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: cluster 2021-02-10T04:32:21.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: 2021-02-10T04:32:20.208882+0000 osd.2 (osd 2021-02-10T04:32:21.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: .2) 358 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:21.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:21.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: -10T04:32:20.208888+0000 osd.2 (osd.2) 359 : cluster [WRN] 2021-02-10T04:32:21.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:21.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: cluster 2021-02-10T04:32 2021-02-10T04:32:21.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: :20.243332+0000 osd.1 (osd.1) 311 2021-02-10T04:32:21.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:21.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:32:21.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: 04:32:20.243337+0000 osd.1 (osd. 2021-02-10T04:32:21.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: 1) 312 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:21.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:21.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: 02-10T04:32:20.243342+0000 osd.1 2021-02-10T04:32:21.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: (osd.1) 313 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:21.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:21.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: -10T04:32:20.243345+0000 osd.1 (osd 2021-02-10T04:32:21.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: .1) 314 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:21.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:32:21.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: :32:20.243349+0000 osd.1 (osd.1) 2021-02-10T04:32:21.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: 315 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:21.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: cluster 2021-02-10T04:32 2021-02-10T04:32:21.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: :20.243353+0000 osd.1 (osd.1) 316 : 2021-02-10T04:32:21.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:21.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: cluster 2021-02-10T04:32:20.338842 2021-02-10T04:32:21.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1074 : cluster [DBG] 2021-02-10T04:32:21.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: osdmap e323: 8 total, 8 up, 8 in 2021-02-10T04:32:21.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:21.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: 02-10T04:32:20.484129+0000 osd.4 (osd.4) 293 : 2021-02-10T04:32:21.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:21.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: cluster 2021-02-10T04:32:20.484140 2021-02-10T04:32:21.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 294 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:21.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: 2021-02-10T04:32:21.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: cluster 2021-02-10T04:32:21.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: 2021-02-10T04:32:20.484144+0000 osd.4 (osd. 2021-02-10T04:32:21.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: 4) 295 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:21.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: cluster 2021-02-10T04:32:21.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: 2021-02-10T04:32:20.484148+0000 osd.4 (osd.4) 296 : cluster [WRN] 2021-02-10T04:32:21.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:21.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: cluster 2021-02-10T04:32: 2021-02-10T04:32:21.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: 21.226690+0000 osd.2 (osd.2) 360 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:21.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: cluster 2021-02-10T04:32:21.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: 2021-02-10T04:32:21.226714+0000 osd.2 (osd.2) 361 : cluster [WRN] 2021-02-10T04:32:21.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:21.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: cluster 2021-02-10T04:32: 2021-02-10T04:32:21.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: 21.226722+0000 osd.2 (osd.2) 362 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:21.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: cluster 2021-02-10T04:32:21.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: 2021-02-10T04:32:21.226729+0000 osd.2 (osd.2) 363 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:21.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: cluster 2021-02-10T04:32:21. 2021-02-10T04:32:21.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: 226738+0000 osd.2 (osd.2) 364 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:21.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: cluster 2021-02-10T04:32:21.226746+0000 2021-02-10T04:32:21.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: osd.2 (osd.2) 365 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:21.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: cluster 2021-02-10T04:32:21.226751+0000 osd.2 (osd 2021-02-10T04:32:21.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: .2) 366 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:21.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:32:21.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: :32:21.226757+0000 osd.2 (osd.2 2021-02-10T04:32:21.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[29319]: ) 367 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:21.446 INFO:tasks.rados.rados.0.smithi085.stdout:1170: finishing copy_from to smithi08575397-1 2021-02-10T04:32:21.446 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 385 (ObjNum 424 snap 109 seq_num 424) dirty exists 2021-02-10T04:32:21.446 INFO:tasks.rados.rados.0.smithi085.stdout:1170: finishing copy_from racing read to smithi08575397-1 2021-02-10T04:32:21.446 INFO:tasks.rados.rados.0.smithi085.stdout:1172: finishing write tid 3 to smithi08575397-2 2021-02-10T04:32:21.446 INFO:tasks.rados.rados.0.smithi085.stdout:1172: finishing write tid 4 to smithi08575397-2 2021-02-10T04:32:21.446 INFO:tasks.rados.rados.0.smithi085.stdout:1172: finishing write tid 5 to smithi08575397-2 2021-02-10T04:32:21.446 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 302 (ObjNum 438 snap 116 seq_num 438) dirty exists 2021-02-10T04:32:21.446 INFO:tasks.rados.rados.0.smithi085.stdout:1172: left oid 2 (ObjNum 438 snap 116 seq_num 438) 2021-02-10T04:32:21.466 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:32:21.469 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[37263]: debug 2021-02-10T04:32:21.218+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:21.469 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[37263]: debug 2021-02-10T04:32:21.436+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:21.469 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[33677]: debug 2021-02-10T04:32:21.467+0000 7fbec19a1700 -1 osd.4 324 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:32:21.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: cluster 2021-02-10T04:32:20.208827+0000 2021-02-10T04:32:21.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: osd.2 (osd.2) 352 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:21.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: cluster 2021-02-10T04:32:20.208836+0000 osd.2 (osd. 2021-02-10T04:32:21.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: 2) 353 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:21.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:32:21.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: :32:20.208844+0000 osd.2 (osd.2) 354 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:21.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: 2021-02-10T04:32:21.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: cluster 2021-02-10T04:32:21.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:32:21.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: :32:20.208866+0000 osd.2 2021-02-10T04:32:21.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: (osd.2) 355 : cluster 2021-02-10T04:32:21.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:21.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:21.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: -10T04:32:20.208871+0000 osd.2 (osd. 2021-02-10T04:32:21.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: 2) 356 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:21.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:21.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: 04:32:20.208876+0000 osd.2 (osd.2) 357 : cluster [WRN] 2021-02-10T04:32:21.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:21.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: cluster 2021-02-10T04:32 2021-02-10T04:32:21.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: :20.208882+0000 osd.2 (osd.2) 358 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:21.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: cluster 2021-02-10T04:32: 2021-02-10T04:32:21.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: 20.208888+0000 osd.2 (osd 2021-02-10T04:32:21.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: .2) 359 : cluster [WRN] 2021-02-10T04:32:21.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:21.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:21.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: -02-10T04:32: 2021-02-10T04:32:21.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: 20.243332+0000 osd.1 (osd. 2021-02-10T04:32:21.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: 1) 311 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:21.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: 2021-02-10T04:32:21.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: cluster 2021-02-10T04:32:21.472 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: 2021-02-10T04:32 2021-02-10T04:32:21.472 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: :20.243337+0000 osd.1 (osd.1 2021-02-10T04:32:21.472 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: ) 312 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:21.473 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:32:21.473 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: :32:20.243342+0000 osd.1 (osd.1) 313 2021-02-10T04:32:21.473 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:21.473 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:32:21.473 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: 32:20.243345+0000 osd.1 (osd.1) 2021-02-10T04:32:21.473 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: 314 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:21.473 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:21.473 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: 04:32:20.243349+0000 osd.1 (osd.1 2021-02-10T04:32:21.474 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: ) 315 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:21.474 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:32:21.474 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: :32:20.243353+0000 osd.1 (osd.1) 2021-02-10T04:32:21.474 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: 316 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:21.474 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:21.475 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: 04:32:20.338842+0000 mon.a (mon.0 2021-02-10T04:32:21.475 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: ) 1074 : cluster [DBG] osdmap e323: 8 total, 8 up, 8 in 2021-02-10T04:32:21.475 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:21.475 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: -10T04:32:20.484129+0000 osd.4 (osd.4) 293 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:21.475 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:32:21.475 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: 10T04:32:20.484140+0000 osd.4 2021-02-10T04:32:21.475 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: (osd.4) 294 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:21.475 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: cluster 2021-02-10T04:32:21.476 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: 2021-02-10T04:32:20 2021-02-10T04:32:21.476 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: .484144+0000 osd.4 (osd.4) 295 : 2021-02-10T04:32:21.476 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:21.476 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:32:21.476 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: 10T04:32:20.484148+0000 osd.4 2021-02-10T04:32:21.476 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: (osd.4) 296 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:21.476 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:21.476 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: -02-10T04:32:21.226690 2021-02-10T04:32:21.476 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 360 : cluster 2021-02-10T04:32:21.476 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:21.476 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:21.476 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: T04:32:21.226714+0000 osd.2 ( 2021-02-10T04:32:21.476 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: osd.2) 361 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:21.477 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:21.477 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: 02-10T04:32:21.226722+0000 osd.2 2021-02-10T04:32:21.477 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: (osd.2) 362 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:21.477 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:21.477 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: -02-10T04:32:21.226729+0000 2021-02-10T04:32:21.477 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: osd.2 (osd.2) 363 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:21.477 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: cluster 2021-02-10T04:32:21.477 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: 2021-02-10T04:32:21 2021-02-10T04:32:21.477 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: .226738+0000 osd.2 (osd.2 2021-02-10T04:32:21.477 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: ) 364 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:21.477 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: cluster 2021-02-10T04:32:21.477 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: 2021-02-10T04:32:21.477 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: 2021-02-10T04:32:21.226746+0000 2021-02-10T04:32:21.477 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: osd.2 (osd.2) 365 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:21.478 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: cluster 2021-02-10T04:32:21 2021-02-10T04:32:21.478 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: .226751+0000 osd.2 (osd.2 2021-02-10T04:32:21.478 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: ) 366 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:21.478 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:21.478 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: 02-10T04:32:21.226757+0000 osd.2 (osd. 2021-02-10T04:32:21.478 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[29288]: 2) 367 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:21.521 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:32:21.521 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [ 2021-02-10T04:32:21.521 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.521 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215)", 2021-02-10T04:32:21.521 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:30:31.173983+0000", 2021-02-10T04:32:21.521 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 110.341257599, 2021-02-10T04:32:21.521 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 110.341284589, 2021-02-10T04:32:21.521 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:32:21.521 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:32:21.522 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:32:21.522 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:32:21.522 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:32:21.522 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 1683 2021-02-10T04:32:21.522 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.522 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:32:21.522 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.522 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:32:21.522 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.173983+0000", 2021-02-10T04:32:21.522 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:32:21.522 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.522 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.523 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:32:21.523 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.173983+0000", 2021-02-10T04:32:21.523 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.0055000000000002e-05 2021-02-10T04:32:21.523 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.523 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.523 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:32:21.523 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.174023+0000", 2021-02-10T04:32:21.523 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.3059999999999998e-06 2021-02-10T04:32:21.523 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.523 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.523 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:32:21.523 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.174032+0000", 2021-02-10T04:32:21.524 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.4224e-05 2021-02-10T04:32:21.524 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.524 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.524 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:32:21.524 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.174046+0000", 2021-02-10T04:32:21.524 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.0839999999999992e-06 2021-02-10T04:32:21.524 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.524 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.524 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:32:21.524 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.174054+0000", 2021-02-10T04:32:21.524 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.1035000000000002e-05 2021-02-10T04:32:21.525 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.525 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.525 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:21.525 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.174105+0000", 2021-02-10T04:32:21.525 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000103606 2021-02-10T04:32:21.525 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.525 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.525 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:32:21.525 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.174208+0000", 2021-02-10T04:32:21.525 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00022531 2021-02-10T04:32:21.525 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:21.526 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:32:21.526 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:21.526 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.526 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.527 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215)", 2021-02-10T04:32:21.527 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:30:31.174396+0000", 2021-02-10T04:32:21.527 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 110.340844335, 2021-02-10T04:32:21.527 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 110.34092550299999, 2021-02-10T04:32:21.527 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:32:21.527 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "delayed", 2021-02-10T04:32:21.527 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:32:21.527 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:32:21.527 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:32:21.527 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 1684 2021-02-10T04:32:21.527 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.527 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:32:21.528 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.528 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:32:21.528 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.174396+0000", 2021-02-10T04:32:21.528 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:32:21.528 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.528 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.528 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:32:21.528 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.174396+0000", 2021-02-10T04:32:21.528 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.6590000000000001e-05 2021-02-10T04:32:21.528 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.529 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.529 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:32:21.529 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.174423+0000", 2021-02-10T04:32:21.529 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.5430000000000001e-06 2021-02-10T04:32:21.529 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.529 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.529 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:32:21.529 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.174429+0000", 2021-02-10T04:32:21.529 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0336000000000001e-05 2021-02-10T04:32:21.529 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.529 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.530 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:32:21.530 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.174439+0000", 2021-02-10T04:32:21.530 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.3011999999999999e-05 2021-02-10T04:32:21.530 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.530 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.530 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:32:21.530 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.174462+0000", 2021-02-10T04:32:21.530 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.1209999999999999e-05 2021-02-10T04:32:21.530 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.530 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.530 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:21.530 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.174513+0000", 2021-02-10T04:32:21.531 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.1226999999999997e-05 2021-02-10T04:32:21.531 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.531 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.531 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for blocked object", 2021-02-10T04:32:21.531 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.174554+0000", 2021-02-10T04:32:21.531 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000157918 2021-02-10T04:32:21.531 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:21.531 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:32:21.531 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:21.531 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.531 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.531 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223)", 2021-02-10T04:32:21.531 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:30:39.230682+0000", 2021-02-10T04:32:21.532 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 102.284558488, 2021-02-10T04:32:21.532 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 102.284683747, 2021-02-10T04:32:21.532 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:32:21.532 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:32:21.532 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:32:21.532 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:32:21.532 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:32:21.532 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 1777 2021-02-10T04:32:21.532 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.532 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:32:21.532 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.532 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:32:21.532 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:39.230682+0000", 2021-02-10T04:32:21.532 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:32:21.533 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.533 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.533 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:32:21.533 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:39.230682+0000", 2021-02-10T04:32:21.533 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.5284999999999998e-05 2021-02-10T04:32:21.533 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.533 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.533 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:32:21.533 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:39.230728+0000", 2021-02-10T04:32:21.533 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.2739999999999996e-06 2021-02-10T04:32:21.533 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.533 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.534 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:32:21.534 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:39.230737+0000", 2021-02-10T04:32:21.534 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.5939000000000001e-05 2021-02-10T04:32:21.534 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.534 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.534 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:32:21.534 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:39.230753+0000", 2021-02-10T04:32:21.534 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.4569999999999994e-06 2021-02-10T04:32:21.534 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.534 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.534 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:32:21.534 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:39.230762+0000", 2021-02-10T04:32:21.535 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.3152999999999997e-05 2021-02-10T04:32:21.535 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.535 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.535 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:21.535 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:39.230825+0000", 2021-02-10T04:32:21.535 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00012603500000000001 2021-02-10T04:32:21.535 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.535 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.535 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:32:21.535 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:39.230951+0000", 2021-02-10T04:32:21.535 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00026914300000000001 2021-02-10T04:32:21.535 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:21.535 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:32:21.536 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:21.536 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.536 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.536 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223)", 2021-02-10T04:32:21.536 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:30:39.230812+0000", 2021-02-10T04:32:21.536 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 102.284429117, 2021-02-10T04:32:21.536 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 102.28459142600001, 2021-02-10T04:32:21.536 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:32:21.536 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "delayed", 2021-02-10T04:32:21.536 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:32:21.536 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:32:21.536 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:32:21.537 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 1778 2021-02-10T04:32:21.537 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.537 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:32:21.537 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.537 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:32:21.537 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:39.230812+0000", 2021-02-10T04:32:21.537 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:32:21.537 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.537 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.537 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:32:21.537 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:39.230812+0000", 2021-02-10T04:32:21.537 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.0914999999999998e-05 2021-02-10T04:32:21.537 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.538 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.538 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:32:21.538 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:39.230843+0000", 2021-02-10T04:32:21.538 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.2049999999999999e-06 2021-02-10T04:32:21.538 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.538 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.538 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:32:21.538 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:39.230849+0000", 2021-02-10T04:32:21.538 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1541e-05 2021-02-10T04:32:21.539 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.539 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.539 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:32:21.539 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:39.230860+0000", 2021-02-10T04:32:21.540 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.4489999999999998e-06 2021-02-10T04:32:21.540 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.540 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.540 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:32:21.541 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:39.230866+0000", 2021-02-10T04:32:21.541 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00023446 2021-02-10T04:32:21.541 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.541 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.541 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:21.541 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:39.231100+0000", 2021-02-10T04:32:21.541 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.6888999999999997e-05 2021-02-10T04:32:21.542 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.542 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.542 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for blocked object", 2021-02-10T04:32:21.542 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:39.231157+0000", 2021-02-10T04:32:21.542 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000345459 2021-02-10T04:32:21.542 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:21.542 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:32:21.542 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:21.542 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.543 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.543 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134)", 2021-02-10T04:32:21.543 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:29:10.111666+0000", 2021-02-10T04:32:21.543 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 191.403574771, 2021-02-10T04:32:21.543 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 191.40379767499999, 2021-02-10T04:32:21.543 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:32:21.543 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:32:21.543 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:32:21.544 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:32:21.544 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:32:21.544 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 650 2021-02-10T04:32:21.544 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.544 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:32:21.544 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.544 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:32:21.544 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.111666+0000", 2021-02-10T04:32:21.544 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:32:21.544 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.545 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.545 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:32:21.545 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.111666+0000", 2021-02-10T04:32:21.545 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4101999999999999e-05 2021-02-10T04:32:21.545 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.545 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.545 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:32:21.545 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.111710+0000", 2021-02-10T04:32:21.545 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.5870000000000004e-06 2021-02-10T04:32:21.545 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.545 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.545 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:32:21.546 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.111718+0000", 2021-02-10T04:32:21.546 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.3169e-05 2021-02-10T04:32:21.546 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.546 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.546 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:32:21.546 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.111731+0000", 2021-02-10T04:32:21.546 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.6460000000000004e-06 2021-02-10T04:32:21.547 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.547 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.547 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:32:21.547 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.111738+0000", 2021-02-10T04:32:21.547 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.5581999999999998e-05 2021-02-10T04:32:21.547 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.548 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.548 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:21.548 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.111783+0000", 2021-02-10T04:32:21.548 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000151488 2021-02-10T04:32:21.548 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.548 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.548 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:32:21.548 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.111935+0000", 2021-02-10T04:32:21.549 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00026857399999999999 2021-02-10T04:32:21.549 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:21.549 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:32:21.549 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:21.549 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.549 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.549 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134)", 2021-02-10T04:32:21.549 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:29:10.112136+0000", 2021-02-10T04:32:21.550 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 191.40310460200001, 2021-02-10T04:32:21.550 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 191.40339645500001, 2021-02-10T04:32:21.550 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:32:21.550 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "delayed", 2021-02-10T04:32:21.550 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:32:21.550 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:32:21.551 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:32:21.551 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 651 2021-02-10T04:32:21.551 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.551 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:32:21.551 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.551 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:32:21.551 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.112136+0000", 2021-02-10T04:32:21.551 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:32:21.551 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.551 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.551 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:32:21.552 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.112136+0000", 2021-02-10T04:32:21.552 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.5148999999999998e-05 2021-02-10T04:32:21.552 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.552 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.552 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:32:21.552 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.112171+0000", 2021-02-10T04:32:21.552 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.6880000000000002e-06 2021-02-10T04:32:21.552 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.552 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.552 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:32:21.552 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.112178+0000", 2021-02-10T04:32:21.552 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1945e-05 2021-02-10T04:32:21.552 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.553 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.553 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:32:21.553 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.112190+0000", 2021-02-10T04:32:21.553 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.7760000000000003e-06 2021-02-10T04:32:21.553 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.553 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.553 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:32:21.553 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.112196+0000", 2021-02-10T04:32:21.553 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4552999999999998e-05 2021-02-10T04:32:21.553 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.554 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.554 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:21.554 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.112240+0000", 2021-02-10T04:32:21.554 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0986000000000003e-05 2021-02-10T04:32:21.554 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:21.554 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:21.554 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for blocked object", 2021-02-10T04:32:21.554 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.112291+0000", 2021-02-10T04:32:21.554 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00015509700000000001 2021-02-10T04:32:21.554 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:21.554 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:32:21.554 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:21.555 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:21.555 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:32:21.555 INFO:teuthology.orchestra.run.smithi085.stdout: "complaint_time": 30, 2021-02-10T04:32:21.555 INFO:teuthology.orchestra.run.smithi085.stdout: "num_blocked_ops": 6 2021-02-10T04:32:21.555 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:32:21.579 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:32:21.605 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:21.691 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:21 smithi085 conmon[52836]: debug 2021-02-10T04:32:21.532+0000 7fba6ffae700 -1 osd.3 pg_epoch: 324 pg[4.14( v 63'26 (0'0,63'26] local-lis/les=71/72 n=11 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,2,7] r=0 lpr=71 crt=63'26 lcod 63'25 mlcod 0'0 active+clean] scrubber pg(4.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55fec03f5b80) 2021-02-10T04:32:21.699 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:32:21.805 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:32:21.835 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[33677]: debug 2021-02-10T04:32:21.668+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:21.838 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:21.958 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:32:22.304 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[40892]: debug 2021-02-10T04:32:22.304 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[40892]: 2021-02-10T04:32:21.926+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:22.304 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:32:21 smithi164 conmon[40892]: 2021-02-10T04:32:22.355 INFO:tasks.rados.rados.0.smithi085.stdout:1170: done (15 left) 2021-02-10T04:32:22.355 INFO:tasks.rados.rados.0.smithi085.stdout:1172: done (14 left) 2021-02-10T04:32:22.355 INFO:tasks.rados.rados.0.smithi085.stdout:1173: done (13 left) 2021-02-10T04:32:22.356 INFO:tasks.rados.rados.0.smithi085.stdout:1174: copy_from oid 2 from oid 3 current snap is 117 2021-02-10T04:32:22.356 INFO:tasks.rados.rados.0.smithi085.stdout:1175: snap_remove snap 108 2021-02-10T04:32:22.404 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:32:22.438 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:22.601 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:32:22.631 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[45443]: debug 2021-02-10T04:32:22.226+0000 7f7140d73700 -1 osd.1 324 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:32:22.632 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[49106]: debug 2021-02-10T04:32:22.253+0000 7f77a3e94700 -1 osd.2 324 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:32:22.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: cluster 2021-02-10T04:32:22.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: 2021-02-10T04:32:21 2021-02-10T04:32:22.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: .221843+0000 osd.1 (osd.1) 317 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:22.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: cluster 2021-02-10T04:32:22.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: 2021-02-10T04:32:21.221853+0000 osd.1 (osd.1) 318 : cluster [WRN] 2021-02-10T04:32:22.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:22.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:32:22.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: :32:21.221875+0000 osd.1 (osd. 2021-02-10T04:32:22.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: 1) 319 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:22.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:22.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: 02-10T04:32:21.221885+0000 2021-02-10T04:32:22.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: osd.1 (osd.1) 320 : cluster [WRN] 2021-02-10T04:32:22.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:22.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: cluster 2021-02-10T04:32:21 2021-02-10T04:32:22.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: .221889+0000 osd.1 (osd.1) 321 : cluster 2021-02-10T04:32:22.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:22.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:32:22.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: 04:32:21.221893+0000 osd.1 (osd 2021-02-10T04:32:22.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: .1) 322 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:22.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: cluster 2021-02-10T04:32:22.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: 2021-02-10T04:32:21.348034+0000 mon.a ( 2021-02-10T04:32:22.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: mon.0) 1075 : cluster [DBG] osdmap e324: 8 total, 8 up, 8 in 2021-02-10T04:32:22.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: cluster 2021-02-10T04:32:21.467843 2021-02-10T04:32:22.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 297 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:22.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: cluster 2021-02-10T04:32:21 2021-02-10T04:32:22.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: .467850+0000 osd.4 (osd.4) 298 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:22.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:32:22.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: 04:32:21.467854+0000 osd.4 (osd.4) 299 : 2021-02-10T04:32:22.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:22.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:22.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: -10T04:32:21.467858+0000 osd.4 (osd.4) 300 : cluster [WRN] 2021-02-10T04:32:22.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:22.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:22.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: -10T04:32:22.055359+0000 mgr.y (mgr.14140) 478 2021-02-10T04:32:22.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: : cluster [DBG] pgmap v636: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 160 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 7 op/s 2021-02-10T04:32:22.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: cluster 2021-02-10T04:32:22. 2021-02-10T04:32:22.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: 254256+0000 osd.2 (osd.2) 368 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:22.639 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: cluster 2021-02-10T04:32 2021-02-10T04:32:22.639 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: :22.254261+0000 osd.2 (osd.2) 369 : cluster 2021-02-10T04:32:22.639 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:22.639 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:22.640 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: -02-10T04:32:22.254265+0000 osd.2 (osd.2) 370 : 2021-02-10T04:32:22.640 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:22.640 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: cluster 2021-02-10T04:32:22.254268 2021-02-10T04:32:22.641 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 371 : cluster [WRN] 2021-02-10T04:32:22.641 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:22.641 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: cluster 2021-02-10T04:32:22.254271+0000 osd.2 2021-02-10T04:32:22.641 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: (osd.2) 372 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:22.641 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: cluster 2021-02-10T04:32:22.254275+0000 osd.2 ( 2021-02-10T04:32:22.642 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: osd.2) 373 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:22.642 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: cluster 2021-02-10T04:32:22.254279+0000 osd.2 2021-02-10T04:32:22.642 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: (osd.2) 374 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:22.642 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: cluster 2021-02-10T04:32:22.254283+0000 osd.2 2021-02-10T04:32:22.642 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[29319]: (osd.2) 375 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:22.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[33677]: debug 2021-02-10T04:32:22.496+0000 7fbec19a1700 -1 osd.4 325 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:32:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: cluster 2021-02-10T04:32:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:32:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: :32:21.221843+0000 osd.1 (osd.1) 317 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:32:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: 10T04:32:21.221853+0000 osd.1 2021-02-10T04:32:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: (osd.1) 318 : cluster 2021-02-10T04:32:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:32:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: :32:21.221875+0000 osd.1 (osd.1) 319 2021-02-10T04:32:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: cluster 2021-02-10T04:32:21.221885 2021-02-10T04:32:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 320 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: T04:32:21.221889+0000 osd.1 (osd.1) 2021-02-10T04:32:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: 321 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:22.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:32:22.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: :32:21.221893+0000 osd.1 2021-02-10T04:32:22.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: (osd.1) 322 : cluster 2021-02-10T04:32:22.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:22.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:22.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: T04:32:21.348034+0000 2021-02-10T04:32:22.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: mon.a (mon.0) 1075 : cluster 2021-02-10T04:32:22.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: [DBG] osdmap e324: 8 total, 8 up, 8 in 2021-02-10T04:32:22.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:22.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: -02-10T04:32:21 2021-02-10T04:32:22.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: .467843+0000 osd.4 (osd.4 2021-02-10T04:32:22.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: ) 297 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:22.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:22.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: -02-10T04:32:21.467850 2021-02-10T04:32:22.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 298 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:22.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:32:22.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: 10T04:32:21.467854 2021-02-10T04:32:22.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 299 : cluster 2021-02-10T04:32:22.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:22.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: cluster 2021-02-10T04:32:21.467858 2021-02-10T04:32:22.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 300 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:22.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: 2021-02-10T04:32:22.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:22.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: -02-10T04:32:22.055359+0000 mgr.y (mgr.14140) 478 2021-02-10T04:32:22.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: : cluster [DBG] pgmap v636: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 160 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 7 op/s 2021-02-10T04:32:22.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:32:22.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: :32:22.254256+0000 osd.2 (osd.2 2021-02-10T04:32:22.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: ) 368 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:22.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:32:22.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: :32:22.254261+0000 osd.2 (osd.2) 369 2021-02-10T04:32:22.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:22.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: cluster 2021-02-10T04:32 2021-02-10T04:32:22.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: :22.254265+0000 osd.2 (osd.2) 370 : cluster 2021-02-10T04:32:22.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:22.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: cluster 2021-02-10T04:32:22 2021-02-10T04:32:22.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: .254268+0000 osd.2 (osd.2) 371 : cluster 2021-02-10T04:32:22.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:22.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: cluster 2021-02-10T04:32:22 2021-02-10T04:32:22.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: .254271+0000 osd.2 (osd.2) 372 : cluster 2021-02-10T04:32:22.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:22.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: cluster 2021-02-10T04:32: 2021-02-10T04:32:22.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: 22.254275+0000 osd.2 (osd.2) 373 : cluster 2021-02-10T04:32:22.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:22.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:22.656 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: -10T04:32:22.254279+0000 osd.2 (osd.2) 374 2021-02-10T04:32:22.656 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:22.656 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:32:22.656 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: :32:22.254283+0000 osd.2 (osd.2) 2021-02-10T04:32:22.656 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:22 smithi164 conmon[29288]: 375 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:22.657 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: cluster 2021-02-10T04:32:22.657 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: 2021-02-10T04:32:22.657 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: 2021 2021-02-10T04:32:22.657 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: - 2021-02-10T04:32:22.657 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: 02 2021-02-10T04:32:22.658 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: - 2021-02-10T04:32:22.658 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: 10T04:32:21.221843+0000 2021-02-10T04:32:22.658 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: osd.1 (osd.1) 317 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:22.658 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: cluster 2021-02-10T04:32 2021-02-10T04:32:22.658 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: :21.221853+0000 osd.1 (osd. 2021-02-10T04:32:22.658 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: 1) 318 : cluster [WRN] 2021-02-10T04:32:22.659 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:22.659 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: cluster 2021-02-10T04:32:22.659 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: 2021-02-10T04:32:22.659 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: 2021 2021-02-10T04:32:22.659 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: - 2021-02-10T04:32:22.660 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: 02 2021-02-10T04:32:22.660 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: -10T04:32:21.221875 2021-02-10T04:32:22.660 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 319 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:22.660 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: cluster 2021-02-10T04:32:21.221885+0000 2021-02-10T04:32:22.660 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: osd.1 (osd.1) 320 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:22.660 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: 2021-02-10T04:32:22.661 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:22.661 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: 10T04:32:21.221889+0000 osd.1 (osd.1) 321 : 2021-02-10T04:32:22.661 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:22.661 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: cluster 2021-02-10T04:32:21 2021-02-10T04:32:22.661 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: .221893+0000 osd.1 (osd.1) 322 : cluster 2021-02-10T04:32:22.661 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:22.662 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: cluster 2021-02-10T04:32:22.662 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: 2021-02-10T04:32:21.348034+0000 mon.a ( 2021-02-10T04:32:22.662 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: mon.0) 1075 : cluster [DBG] osdmap e324: 8 total, 8 up, 8 in 2021-02-10T04:32:22.662 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:22.662 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: 10T04:32:21.467843+0000 osd.4 (osd 2021-02-10T04:32:22.663 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: .4) 297 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:22.663 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:22.663 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: 02-10T04:32:21.467850+0000 2021-02-10T04:32:22.663 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: osd.4 (osd.4) 298 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:22.663 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: 2021-02-10T04:32:22.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:22.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: 02-10T04:32:21.467854+0000 osd.4 (osd. 2021-02-10T04:32:22.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: 4) 299 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:22.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:32:22.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: T04:32:21.467858+0000 osd.4 (osd 2021-02-10T04:32:22.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: .4) 300 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:22.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: cluster 2021-02-10T04:32:22.055359 2021-02-10T04:32:22.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 478 : cluster 2021-02-10T04:32:22.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: [DBG] pgmap v636: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 160 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 7 op/s 2021-02-10T04:32:22.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: cluster 2021-02-10T04:32:22.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: 2021-02-10T04:32:22.254256+0000 osd.2 2021-02-10T04:32:22.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: (osd.2) 368 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:22.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: 2021-02-10T04:32:22.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: cluster 2021-02-10T04:32: 2021-02-10T04:32:22.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: 22.254261+0000 osd.2 (osd.2) 369 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:22.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: 2021-02-10T04:32:22.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:22.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: -02-10T04:32:22.254265+0000 osd.2 (osd 2021-02-10T04:32:22.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: .2) 370 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:22.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:22.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: 04:32:22.254268+0000 osd.2 (osd. 2021-02-10T04:32:22.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: 2) 371 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:22.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:22.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: 10T04:32:22.254271+0000 osd.2 ( 2021-02-10T04:32:22.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: osd.2) 372 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:22.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:22.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: 02-10T04:32:22.254275+0000 2021-02-10T04:32:22.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: osd.2 (osd.2) 373 : cluster [WRN] 2021-02-10T04:32:22.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:22.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: cluster 2021-02-10T04:32:22.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: 2021-02-10T04:32:22.254279+0000 osd.2 (osd 2021-02-10T04:32:22.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: .2) 374 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:22.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:32:22.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: T04:32:22.254283+0000 osd.2 (osd. 2021-02-10T04:32:22.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[35961]: 2) 375 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:22.893 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:32:22.941 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:22 smithi085 conmon[52836]: debug 2021-02-10T04:32:22.759+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:23.336 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:32:23.359 INFO:tasks.rados.rados.0.smithi085.stdout:1175: done (14 left) 2021-02-10T04:32:23.359 INFO:tasks.rados.rados.0.smithi085.stdout:1176: snap_remove snap 110 2021-02-10T04:32:23.363 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[45443]: debug 2021-02-10T04:32:23.268+0000 7f7140d73700 -1 osd.1 325 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:32:23.363 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[49106]: debug 2021-02-10T04:32:23.263+0000 7f77a3e94700 -1 osd.2 325 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:32:23.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[33677]: debug 2021-02-10T04:32:23.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[33677]: 2021-02-10T04:32:23.470+0000 7fbec19a1700 -1 osd.4 326 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:32:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: cluster 2021-02-10T04:32:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:32:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: 32:22.226966+0000 osd.1 (osd.1) 323 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: cluster 2021-02-10T04:32:22. 2021-02-10T04:32:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: 226976+0000 osd.1 (osd.1) 324 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: 2021-02-10T04:32:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: cluster 2021-02-10T04:32:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: 2021-02-10T04:32:22.226980+0000 osd.1 (osd. 2021-02-10T04:32:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: 1) 325 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: 04:32:22.226984+0000 osd.1 (osd.1 2021-02-10T04:32:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: ) 326 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: cluster 2021-02-10T04:32 2021-02-10T04:32:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: :22.226988+0000 osd.1 (osd.1) 327 : cluster 2021-02-10T04:32:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: cluster 2021-02-10T04:32:22 2021-02-10T04:32:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: .226991+0000 osd.1 (osd.1) 328 : cluster 2021-02-10T04:32:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: cluster 2021-02-10T04:32:22.355293+0000 mon.a 2021-02-10T04:32:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: (mon.0) 1076 : cluster [DBG] osdmap e325: 8 total, 8 up, 8 in 2021-02-10T04:32:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:32:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: 10T04:32:22.496840+0000 osd.4 (osd. 2021-02-10T04:32:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: 4) 301 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: cluster 2021-02-10T04:32:22 2021-02-10T04:32:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: .496850+0000 osd.4 (osd.4) 302 : cluster 2021-02-10T04:32:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:32:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: :32:22.496856+0000 osd.4 (osd.4) 2021-02-10T04:32:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: 303 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: 04:32:22.496864+0000 osd.4 (osd.4 2021-02-10T04:32:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: ) 304 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: 04:32:23.091055+0000 mon.a (mon. 2021-02-10T04:32:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: 0) 1077 : cluster [WRN] Health check update: 18 slow ops, oldest one blocked for 187 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:32:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: -10T04:32:23.264257+0000 osd.2 (osd 2021-02-10T04:32:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: .2) 376 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: cluster 2021-02-10T04:32:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: 2021-02-10T04:32:23.264266+0000 osd.2 (osd.2) 377 : cluster [WRN] 2021-02-10T04:32:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: cluster 2021-02-10T04:32:23.264271 2021-02-10T04:32:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 378 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: 2021-02-10T04:32:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: cluster 2021-02-10T04:32:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: 2021-02-10T04:32:23.264277+0000 osd.2 (osd 2021-02-10T04:32:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: .2) 379 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: T04:32:23.264283+0000 osd.2 (osd.2 2021-02-10T04:32:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: ) 380 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: T04:32:23.264288+0000 osd.2 (osd 2021-02-10T04:32:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: .2) 381 : cluster [WRN] 2021-02-10T04:32:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:23.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:23.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: -02-10T04:32: 2021-02-10T04:32:23.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: 23.264291+0000 osd.2 (osd 2021-02-10T04:32:23.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: .2) 382 : cluster [WRN] 2021-02-10T04:32:23.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:23.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:23.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: -10T04:32:23 2021-02-10T04:32:23.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: .264294+0000 osd.2 (osd. 2021-02-10T04:32:23.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: 2) 383 : cluster [WRN] 2021-02-10T04:32:23.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:23 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:23.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: cluster 2021-02-10T04:32:23.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: 2021- 2021-02-10T04:32:23.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: 02-10T04:32:22.226966+0000 osd.1 (osd.1) 323 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:23.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: 2021-02-10T04:32:23.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:23.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: 02-10T04:32:22.226976+0000 2021-02-10T04:32:23.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: osd.1 (osd.1) 324 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:23.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: 2021-02-10T04:32:23.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: cluster 2021-02-10T04:32: 2021-02-10T04:32:23.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: 22.226980+0000 osd.1 (osd.1) 325 2021-02-10T04:32:23.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:23.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: cluster 2021-02-10T04:32:22 2021-02-10T04:32:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: .226984+0000 osd.1 (osd.1) 326 : 2021-02-10T04:32:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: :32:22.226988+0000 osd.1 (osd. 2021-02-10T04:32:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: 1) 327 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: 02-10T04:32:22.226991+0000 osd.1 (osd.1) 328 : cluster [WRN] 2021-02-10T04:32:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: cluster 2021-02-10T04:32:22.355293 2021-02-10T04:32:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1076 : cluster 2021-02-10T04:32:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: [DBG] osdmap e325: 8 total, 8 up, 8 in 2021-02-10T04:32:23.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:32:23.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: 32:22.496840+0000 osd.4 (osd.4 2021-02-10T04:32:23.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: ) 301 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:23.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:32:23.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: 32:22.496850+0000 osd.4 (osd.4) 2021-02-10T04:32:23.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: 302 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:23.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:32:23.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: 32:22.496856+0000 osd.4 (osd.4 2021-02-10T04:32:23.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: ) 303 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:23.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:32:23.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: T04:32:22.496864+0000 osd.4 (osd.4) 304 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:23.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:23.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: 02-10T04:32:23.091055+0000 2021-02-10T04:32:23.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: mon.a (mon.0) 1077 : cluster [WRN] Health check update: 18 slow ops, oldest one blocked for 187 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:32:23.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: 2021-02-10T04:32:23.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: cluster 2021-02-10T04:32:23.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: 2021-02-10T04:32:23.264257+0000 osd.2 (osd 2021-02-10T04:32:23.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: .2) 376 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:23.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:23.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: 02-10T04:32:23.264266+0000 2021-02-10T04:32:23.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: osd.2 (osd.2) 377 : cluster [WRN] 2021-02-10T04:32:23.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:23.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: cluster 2021-02-10T04:32:23.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: 2021-02-10T04:32:23.264271+0000 osd.2 2021-02-10T04:32:23.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: (osd.2) 378 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:23.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: 2021-02-10T04:32:23.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: cluster 2021-02-10T04:32:23.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: 2021-02-10T04:32:23.264277+0000 osd.2 ( 2021-02-10T04:32:23.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: osd.2) 379 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:23.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:23.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: 02-10T04:32:23.264283+0000 2021-02-10T04:32:23.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: osd.2 (osd.2) 380 : cluster [WRN] 2021-02-10T04:32:23.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:23.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: cluster 2021-02-10T04:32:23 2021-02-10T04:32:23.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: .264288+0000 osd.2 (osd.2) 381 : 2021-02-10T04:32:23.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:23.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:23.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: :32:23.264291+0000 osd.2 (osd. 2021-02-10T04:32:23.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: 2) 382 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:23.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:23.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: -10T04:32:23.264294+0000 osd.2 2021-02-10T04:32:23.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: (osd.2) 383 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:23.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[35961]: 2021-02-10T04:32:23.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: cluster 2021-02-10T04:32:23.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: 2021-02-10T04:32:22.226966 2021-02-10T04:32:23.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: +0000 osd.1 (osd.1) 323 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:23.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:32:23.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: T04:32:22.226976+0000 osd.1 (osd.1) 324 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:23.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: cluster 2021-02-10T04:32:22. 2021-02-10T04:32:23.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: 226980+0000 osd.1 (osd.1) 325 : cluster [WRN] 2021-02-10T04:32:23.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:23.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: cluster 2021-02-10T04:32:22.226984 2021-02-10T04:32:23.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: +0000 osd.1 (osd.1) 326 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:23.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: 2021-02-10T04:32:23.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: cluster 2021-02-10T04:32:23.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: 2021-02-10T04:32:22.226988+0000 osd.1 (osd.1) 327 2021-02-10T04:32:23.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:23.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: cluster 2021-02-10T04:32:23.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: 2021-02-10T04:32:22.226991+0000 osd.1 (osd.1) 2021-02-10T04:32:23.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: 328 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:23.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: cluster 2021-02-10T04:32:22.355293+0000 mon.a 2021-02-10T04:32:23.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: (mon.0) 1076 : cluster [DBG] osdmap e325: 8 total, 8 up, 8 in 2021-02-10T04:32:23.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:32:23.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: 04:32:22.496840+0000 osd.4 (osd.4 2021-02-10T04:32:23.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: ) 301 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:23.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:23.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: 10T04:32:22.496850+0000 osd.4 (osd 2021-02-10T04:32:23.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: .4) 302 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:23.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:23.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: -02-10T04:32:22. 2021-02-10T04:32:23.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: 496856+0000 osd.4 (osd.4) 303 : cluster [WRN] 2021-02-10T04:32:23.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:23.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: cluster 2021-02-10T04:32:22.496864 2021-02-10T04:32:23.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 304 : cluster [WRN] 2021-02-10T04:32:23.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:23.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: cluster 2021-02-10T04:32: 2021-02-10T04:32:23.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: 23.091055+0000 mon.a (mon.0) 1077 : 2021-02-10T04:32:23.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: cluster [WRN] Health check update: 18 slow ops, oldest one blocked for 187 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:32:23.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:32:23.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: 04:32:23.264257+0000 osd.2 (osd 2021-02-10T04:32:23.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: .2) 376 : cluster [WRN] 2021-02-10T04:32:23.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:23.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: cluster 2021-02-10T04:32:23.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: 2021-02-10T04:32 2021-02-10T04:32:23.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: :23.264266+0000 osd.2 (osd 2021-02-10T04:32:23.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: .2) 377 : cluster [WRN] 2021-02-10T04:32:23.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:23.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: cluster 2021-02-10T04:32:23 2021-02-10T04:32:23.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: .264271+0000 osd.2 (osd. 2021-02-10T04:32:23.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: 2) 378 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:23.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: 2021-02-10T04:32:23.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: cluster 2021-02-10T04:32:23.264277+0000 osd.2 (osd.2) 379 : cluster 2021-02-10T04:32:23.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:23.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: cluster 2021-02-10T04:32: 2021-02-10T04:32:23.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: 23.264283+0000 osd.2 (osd.2) 380 : cluster 2021-02-10T04:32:23.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:23.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:32:23.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: T04:32:23.264288+0000 2021-02-10T04:32:23.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: osd.2 (osd.2) 381 : cluster 2021-02-10T04:32:23.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:23.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:32:23.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: :32:23.264291+0000 osd.2 (osd.2 2021-02-10T04:32:23.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: ) 382 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:23.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:32:23.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: 32:23.264294+0000 osd.2 (osd.2) 383 : 2021-02-10T04:32:23.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:23 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:24.361 INFO:tasks.rados.rados.0.smithi085.stdout:1176: done (14 left) 2021-02-10T04:32:24.361 INFO:tasks.rados.rados.0.smithi085.stdout:1177: read oid 31 snap -1 2021-02-10T04:32:24.361 INFO:tasks.rados.rados.0.smithi085.stdout:1177: expect deleted 2021-02-10T04:32:24.361 INFO:tasks.rados.rados.0.smithi085.stdout:1178: snap_remove snap 115 2021-02-10T04:32:24.630 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[49106]: debug 2021-02-10T04:32:24.235+0000 7f77a3e94700 -1 osd.2 326 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:32:24.631 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[52836]: debug 2021-02-10T04:32:24.630+0000 7fba6ffae700 -1 osd.3 pg_epoch: 327 pg[4.14( v 63'26 (0'0,63'26] local-lis/les=71/72 n=11 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,2,7] r=0 lpr=71 crt=63'26 lcod 63'25 mlcod 0'0 active+clean] scrubber pg(4.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55fec0601a20) 2021-02-10T04:32:24.631 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[45443]: debug 2021-02-10T04:32:24.253+0000 7f7140d73700 -1 osd.1 326 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:32:24.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: cluster 2021-02-10T04:32:24.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: 2021-02-10T04:32:24.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: 2021 2021-02-10T04:32:24.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: - 2021-02-10T04:32:24.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: 02-10T04:32:23.269072+0000 2021-02-10T04:32:24.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: osd.1 (osd.1) 329 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:24.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:24.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: -10T04:32:23.269080+0000 osd.1 (osd. 2021-02-10T04:32:24.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: 1) 330 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:24.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: cluster 2021-02-10T04:32:24.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: 2021-02-10T04:32:24.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: 2021 2021-02-10T04:32:24.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: -02-10T04:32:23 2021-02-10T04:32:24.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: .269086+0000 osd.1 (osd.1) 331 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:24.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:32:24.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: 32:23.269091+0000 osd.1 (osd.1) 332 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:24.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: 2021-02-10T04:32:24.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:24.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: -10T04:32:23.269097+0000 osd.1 2021-02-10T04:32:24.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: (osd.1) 333 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:24.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:24.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: -10T04:32:23.269104+0000 osd.1 (osd. 2021-02-10T04:32:24.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: 1) 334 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:24.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:24.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: -10T04:32:23.359497+0000 mon.a (mon. 2021-02-10T04:32:24.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: 0) 1078 : cluster [DBG] osdmap e326: 8 total, 8 up, 8 in 2021-02-10T04:32:24.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: 2021-02-10T04:32:24.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: cluster 2021-02-10T04:32:24.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: 2021-02-10T04:32: 2021-02-10T04:32:24.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: 23.471227+0000 osd.4 (osd.4) 305 : 2021-02-10T04:32:24.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:24.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:24.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: -10T04:32:23.471235+0000 osd.4 ( 2021-02-10T04:32:24.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: osd.4) 306 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:24.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:24.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: 10T04:32:23.471241+0000 osd.4 (osd.4) 307 2021-02-10T04:32:24.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:24.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:32:24.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: 32:23.471248+0000 osd.4 (osd.4) 308 2021-02-10T04:32:24.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:24.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:24.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: -10T04:32:24.055746+0000 mgr.y (mgr 2021-02-10T04:32:24.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: .14140) 479 : cluster [DBG] pgmap v639: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 160 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 714 KiB/s wr, 2 op/s 2021-02-10T04:32:24.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:24.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: -10T04:32:24.235911+0000 osd.2 2021-02-10T04:32:24.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: (osd.2) 384 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:24.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:24.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: -02-10T04:32:24.235917+0000 2021-02-10T04:32:24.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: osd.2 (osd.2) 385 : cluster [WRN] 2021-02-10T04:32:24.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:24.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: cluster 2021-02-10T04:32:24 2021-02-10T04:32:24.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: .235921+0000 osd.2 (osd.2) 386 : cluster 2021-02-10T04:32:24.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:24.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:32:24.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: 32:24.235924+0000 osd.2 (osd.2) 2021-02-10T04:32:24.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: 387 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:24.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: cluster 2021-02-10T04:32: 2021-02-10T04:32:24.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: 24.235928+0000 osd.2 (osd.2) 388 : cluster [WRN] 2021-02-10T04:32:24.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:24.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: cluster 2021-02-10T04:32 2021-02-10T04:32:24.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: :24.235931+0000 osd.2 (osd.2) 389 : 2021-02-10T04:32:24.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:24.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:24.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: 02-10T04:32:24.235947+0000 osd.2 ( 2021-02-10T04:32:24.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: osd.2) 390 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:24.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: 2021-02-10T04:32:24.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:32:24.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: :32:24.235950+0000 osd.2 (osd.2) 2021-02-10T04:32:24.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: 391 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:24.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:24.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: 02-10T04:32:24.359862+0000 2021-02-10T04:32:24.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: mon.a (mon.0) 1079 : cluster 2021-02-10T04:32:24.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[29319]: [DBG] osdmap e327: 8 total, 8 up, 8 in 2021-02-10T04:32:24.638 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: cluster 2021-02-10T04:32:24.638 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:32:24.638 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: :32:23.269072+0000 osd.1 (osd.1) 329 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:24.638 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:24.638 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: -02-10T04:32:23.269080+0000 osd.1 2021-02-10T04:32:24.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: (osd.1) 330 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:24.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:24.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: -02-10T04:32:23.269086+0000 osd.1 (osd.1 2021-02-10T04:32:24.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: ) 331 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:24.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: cluster 2021-02-10T04:32:24.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:32:24.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: 32:23.269091 2021-02-10T04:32:24.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 332 : cluster 2021-02-10T04:32:24.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:24.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:24.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: :32:23.269097+0000 osd.1 (osd.1) 333 2021-02-10T04:32:24.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:24.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: cluster 2021-02-10T04:32:24.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: 2021-02-10T04:32:23.269104+0000 osd.1 (osd.1) 334 : cluster [WRN] 2021-02-10T04:32:24.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:24.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: cluster 2021-02-10T04:32:23 2021-02-10T04:32:24.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: .359497+0000 mon.a (mon 2021-02-10T04:32:24.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: .0) 1078 : 2021-02-10T04:32:24.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: cluster [DBG] osdmap e326: 8 total, 8 up, 8 in 2021-02-10T04:32:24.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:24.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: -02-10T04:32:23.471227+0000 osd.4 (osd 2021-02-10T04:32:24.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: .4) 305 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:24.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: cluster 2021-02-10T04:32:24.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:32:24.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: T04:32:23.471235+0000 osd.4 (osd.4) 306 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:24.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: cluster 2021-02-10T04:32:24.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:32:24.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: 32:23.471241+0000 osd.4 (osd.4) 2021-02-10T04:32:24.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: 307 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:24.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:24.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: -02-10T04:32:23. 2021-02-10T04:32:24.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: 471248+0000 osd.4 (osd.4) 308 : cluster 2021-02-10T04:32:24.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:24.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:24.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: -10T04:32:24.055746+0000 mgr.y 2021-02-10T04:32:24.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: (mgr.14140) 479 : cluster [DBG] pgmap v639: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 160 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 714 KiB/s wr, 2 op/s 2021-02-10T04:32:24.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: 2021-02-10T04:32:24.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: cluster 2021-02-10T04:32: 2021-02-10T04:32:24.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: 24.235911+0000 osd.2 (osd.2) 384 2021-02-10T04:32:24.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:24.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: cluster 2021-02-10T04:32:24.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: 2021-02-10T04:32 2021-02-10T04:32:24.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: :24.235917+0000 2021-02-10T04:32:24.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: osd.2 ( 2021-02-10T04:32:24.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: osd.2) 385 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:24.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:24.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: 10T04:32:24.235921+0000 osd.2 (osd.2 2021-02-10T04:32:24.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: ) 386 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:24.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:24.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: -10T04:32:24.235924 2021-02-10T04:32:24.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 387 : cluster [WRN] 2021-02-10T04:32:24.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:24.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: cluster 2021-02-10T04:32 2021-02-10T04:32:24.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: :24.235928+0000 osd.2 (osd.2) 2021-02-10T04:32:24.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: 388 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:24.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:24.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: 04:32:24.235931+0000 osd.2 (osd.2) 389 : cluster 2021-02-10T04:32:24.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:24.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:32:24.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: 32 2021-02-10T04:32:24.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: :24.235947+0000 osd.2 ( 2021-02-10T04:32:24.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: osd.2) 390 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:24.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: cluster 2021-02-10T04:32:24.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: 2021-02-10T04:32:24.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: 2021 2021-02-10T04:32:24.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: - 2021-02-10T04:32:24.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: 02 2021-02-10T04:32:24.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: - 2021-02-10T04:32:24.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: 10 2021-02-10T04:32:24.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: T04:32:24.235950+0000 2021-02-10T04:32:24.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: osd.2 (osd.2) 391 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:24.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: cluster 2021-02-10T04:32:24.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: 2021-02-10T04:32:24.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:32:24.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:24 smithi085 conmon[35961]: 32:24.359862+0000 mon.a (mon.0) 1079 : cluster [DBG] osdmap e327: 8 total, 8 up, 8 in 2021-02-10T04:32:24.645 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:24 smithi164 conmon[33677]: debug 2021-02-10T04:32:24.486+0000 7fbec19a1700 -1 osd.4 327 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:32:24.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:24 smithi164 conmon[29288]: cluster 2021-02-10T04:32:23.269072 2021-02-10T04:32:24.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:24 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 329 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:24.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:24 smithi164 conmon[29288]: cluster 2021-02-10T04:32:23.269080+0000 osd.1 (osd.1) 330 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:24.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:24 smithi164 conmon[29288]: cluster 2021-02-10T04:32:23.269086+0000 osd.1 (osd.1) 331 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:24.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:24 smithi164 conmon[29288]: cluster 2021-02-10T04:32:23.269091+0000 osd.1 (osd.1) 332 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:24.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:24 smithi164 conmon[29288]: cluster 2021-02-10T04:32:23.269097+0000 osd.1 (osd.1) 333 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:24.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:24 smithi164 conmon[29288]: cluster 2021-02-10T04:32:23.269104+0000 osd.1 (osd.1) 334 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:24.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:24 smithi164 conmon[29288]: cluster 2021-02-10T04:32:23.359497+0000 mon.a (mon.0) 1078 : cluster [DBG] osdmap e326: 8 total, 8 up, 8 in 2021-02-10T04:32:24.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:24 smithi164 conmon[29288]: cluster 2021-02-10T04:32:23.471227+0000 osd.4 (osd.4) 305 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:24.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:24 smithi164 conmon[29288]: cluster 2021-02-10T04:32:23.471235+0000 osd.4 (osd.4) 306 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:24.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:24 smithi164 conmon[29288]: cluster 2021-02-10T04:32:23.471241+0000 osd.4 (osd.4) 307 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:24.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:24 smithi164 conmon[29288]: cluster 2021-02-10T04:32:23.471248+0000 osd.4 (osd.4) 308 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:24.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:24 smithi164 conmon[29288]: cluster 2021-02-10T04:32:24.055746+0000 mgr.y (mgr.14140) 479 : cluster [DBG] pgmap v639: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 160 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 714 KiB/s wr, 2 op/s 2021-02-10T04:32:24.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:24 smithi164 conmon[29288]: cluster 2021-02-10T04:32:24.235911+0000 osd.2 (osd.2) 384 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:24.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:24 smithi164 conmon[29288]: cluster 2021-02-10T04:32:24.235917+0000 osd.2 (osd.2) 385 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:24.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:24 smithi164 conmon[29288]: cluster 2021-02-10T04:32:24.235921+0000 osd.2 (osd.2) 386 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:24.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:24 smithi164 conmon[29288]: cluster 2021-02-10T04:32:24.235924+0000 osd.2 (osd.2) 387 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:24.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:24 smithi164 conmon[29288]: cluster 2021-02-10T04:32:24.235928+0000 osd.2 (osd.2) 388 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:24.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:24 smithi164 conmon[29288]: cluster 2021-02-10T04:32:24.235931+0000 osd.2 (osd.2) 389 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:24.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:24 smithi164 conmon[29288]: cluster 2021-02-10T04:32:24.235947+0000 osd.2 (osd.2) 390 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:24.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:24 smithi164 conmon[29288]: cluster 2021-02-10T04:32:24.235950+0000 osd.2 (osd.2) 391 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:24.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:24 smithi164 conmon[29288]: cluster 2021-02-10T04:32:24.359862+0000 mon.a (mon.0) 1079 : cluster [DBG] osdmap e327: 8 total, 8 up, 8 in 2021-02-10T04:32:25.364 INFO:tasks.rados.rados.0.smithi085.stdout:1178: done (15 left) 2021-02-10T04:32:25.364 INFO:tasks.rados.rados.0.smithi085.stdout:1179: read oid 28 snap -1 2021-02-10T04:32:25.364 INFO:tasks.rados.rados.0.smithi085.stdout:1179: expect (ObjNum 423 snap 109 seq_num 423) 2021-02-10T04:32:25.365 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:25.365 INFO:tasks.rados.rados.0.smithi085.stdout:1174: finishing copy_from to smithi08575397-2 2021-02-10T04:32:25.365 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 492 (ObjNum 437 snap 116 seq_num 437) dirty exists 2021-02-10T04:32:25.365 INFO:tasks.rados.rados.0.smithi085.stdout:1174: finishing copy_from racing read to smithi08575397-2 2021-02-10T04:32:25.365 INFO:tasks.rados.rados.0.smithi085.stdout:1174: done (15 left) 2021-02-10T04:32:25.365 INFO:tasks.rados.rados.0.smithi085.stdout:1177: done (14 left) 2021-02-10T04:32:25.365 INFO:tasks.rados.rados.0.smithi085.stdout:1180: delete oid 30 current snap is 117 2021-02-10T04:32:25.367 INFO:tasks.rados.rados.0.smithi085.stdout:1179: expect (ObjNum 423 snap 109 seq_num 423) 2021-02-10T04:32:25.369 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:32:25.398 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:25.499 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:32:25.513 INFO:tasks.rados.rados.0.smithi085.stdout:1179: done (14 left) 2021-02-10T04:32:25.513 INFO:tasks.rados.rados.0.smithi085.stdout:1180: done (13 left) 2021-02-10T04:32:25.513 INFO:tasks.rados.rados.0.smithi085.stdout:1181: snap_remove snap 112 2021-02-10T04:32:25.604 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:32:25.609 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[49106]: debug 2021-02-10T04:32:25.257+0000 7f77a3e94700 -1 osd.2 327 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:32:25.611 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[52836]: debug 2021-02-10T04:32:25.609+0000 7fba73fb6700 -1 osd.3 pg_epoch: 328 pg[4.14( v 63'26 (0'0,63'26] local-lis/les=71/72 n=11 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,2,7] r=0 lpr=71 crt=63'26 lcod 63'25 mlcod 0'0 active+clean] scrubber pg(4.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55fec05abce0) 2021-02-10T04:32:25.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: cluster 2021-02-10T04:32:24.254291+0000 osd.1 (osd.1) 335 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:25.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: cluster 2021-02-10T04:32:24.254300+0000 osd.1 (osd.1) 336 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:25.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: cluster 2021-02-10T04:32:24.254309+0000 osd.1 (osd.1) 337 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:25.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: cluster 2021-02-10T04:32:24.254315+0000 osd.1 (osd.1) 338 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:25.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: cluster 2021-02-10T04:32:24.254320+0000 osd.1 (osd.1) 339 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:25.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: cluster 2021-02-10T04:32:24.254338+0000 osd.1 (osd.1) 340 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:25.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: cluster 2021-02-10T04:32:24.487248+0000 osd.4 (osd.4) 309 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:25.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: cluster 2021-02-10T04:32:24.487257+0000 osd.4 (osd.4) 310 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:25.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: cluster 2021-02-10T04:32:24.487264+0000 osd.4 (osd.4) 311 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:25.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: cluster 2021-02-10T04:32:24.487270+0000 osd.4 (osd.4) 312 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:25.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: cluster 2021-02-10T04:32:25.257693+0000 osd.2 (osd.2) 392 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:25.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: cluster 2021-02-10T04:32:25.257700+0000 osd.2 (osd.2) 393 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:25.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: cluster 2021-02-10T04:32:25.257705+0000 osd.2 (osd.2) 394 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:25.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: cluster 2021-02-10T04:32:25.257708+0000 osd.2 (osd.2) 395 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:25.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: 2021-02-10T04:32:25.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: cluster 2021-02-10T04:32 2021-02-10T04:32:25.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: :25.257711+0000 osd.2 (osd.2) 396 : cluster [WRN] 2021-02-10T04:32:25.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:25.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: cluster 2021-02-10T04:32: 2021-02-10T04:32:25.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: 25.257715+0000 osd.2 (osd.2) 397 2021-02-10T04:32:25.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:25.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:32:25.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: 32:25.257718+0000 osd.2 (osd.2 2021-02-10T04:32:25.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: ) 398 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:25.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:32:25.616 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: T04:32:25.257721+0000 osd.2 ( 2021-02-10T04:32:25.616 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: osd.2) 399 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:25.616 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:25.616 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: 02-10T04:32:25.363858+0000 2021-02-10T04:32:25.617 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: mon.a (mon.0) 1080 : cluster [DBG] 2021-02-10T04:32:25.617 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[29319]: osdmap e328: 8 total, 8 up, 8 in 2021-02-10T04:32:25.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: cluster 2021-02-10T04:32:24.254291+0000 2021-02-10T04:32:25.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: osd.1 (osd.1) 335 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:25.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:25.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: 04:32:24.254300+0000 osd.1 (osd.1) 336 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:25.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: cluster 2021-02-10T04:32:25.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: 2021-02-10T04:32:24.254309 2021-02-10T04:32:25.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 337 : cluster [WRN] 2021-02-10T04:32:25.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:25.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: cluster 2021-02-10T04:32:24 2021-02-10T04:32:25.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: .254315+0000 osd.1 (osd.1) 338 : 2021-02-10T04:32:25.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:25.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: cluster 2021-02-10T04:32 2021-02-10T04:32:25.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: :24.254320+0000 osd.1 (osd.1) 339 2021-02-10T04:32:25.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:25.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:25.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: :32:24.254338+0000 osd.1 ( 2021-02-10T04:32:25.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: osd.1) 340 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:25.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: cluster 2021-02-10T04:32:25.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: 2021-02-10T04:32:24.487248 2021-02-10T04:32:25.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 309 : cluster 2021-02-10T04:32:25.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:25.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: cluster 2021-02-10T04:32: 2021-02-10T04:32:25.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: 24.487257+0000 osd.4 (osd.4) 310 : cluster 2021-02-10T04:32:25.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:25.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: cluster 2021-02-10T04:32:24.487264+0000 2021-02-10T04:32:25.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: osd.4 (osd.4) 311 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:25.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: 2021-02-10T04:32:25.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: cluster 2021-02-10T04:32:25.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: 2021-02-10T04:32:24.487270+0000 osd.4 ( 2021-02-10T04:32:25.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: osd.4) 312 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:25.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: cluster 2021-02-10T04:32:25.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: 2021-02-10T04:32:25 2021-02-10T04:32:25.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: .257693+0000 osd.2 (osd.2) 392 : 2021-02-10T04:32:25.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:25.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:32:25.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: T04:32:25.257700+0000 osd.2 (osd. 2021-02-10T04:32:25.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: 2) 393 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:25.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:25.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: -10T04:32:25.257705+0000 osd.2 2021-02-10T04:32:25.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: (osd.2) 394 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:25.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:25.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: 02-10T04:32:25.257708+0000 2021-02-10T04:32:25.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: osd.2 (osd.2) 395 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:25.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:25.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: -02-10T04:32:25.257711+0000 2021-02-10T04:32:25.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: osd.2 (osd.2) 396 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:25.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:25.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: -02-10T04:32:25.257715+0000 2021-02-10T04:32:25.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: osd.2 (osd.2) 397 : cluster [WRN] 2021-02-10T04:32:25.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:25.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: cluster 2021-02-10T04:32:25.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: 2021-02-10T04:32:25.257718+0000 osd.2 (osd 2021-02-10T04:32:25.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: .2) 398 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:25.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:25.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: 10T04:32:25.257721+0000 osd.2 ( 2021-02-10T04:32:25.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: osd.2) 399 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:25.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:25.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: -10T04:32:25.363858+0000 2021-02-10T04:32:25.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: mon.a (mon.0) 1080 : cluster [DBG] osdmap e328: 8 total, 8 up, 8 in 2021-02-10T04:32:25.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[35961]: 2021-02-10T04:32:25.624 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:25 smithi085 conmon[45443]: debug 2021-02-10T04:32:25.225+0000 7f7140d73700 -1 osd.1 327 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:32:25.635 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:32:25 smithi164 conmon[40892]: debug 2021-02-10T04:32:25.472+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:25.636 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:25 smithi164 conmon[29288]: cluster 2021-02-10T04:32:24.254291+0000 osd.1 (osd 2021-02-10T04:32:25.636 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:25 smithi164 conmon[29288]: .1) 335 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:25.636 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:25 smithi164 conmon[29288]: cluster 2021-02-10T04:32:24.254300+0000 osd.1 (osd.1) 336 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:25.636 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:25 smithi164 conmon[29288]: cluster 2021-02-10T04:32:24.254309+0000 osd.1 (osd.1) 337 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:25.636 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:25 smithi164 conmon[29288]: cluster 2021-02-10T04:32:24.254315+0000 osd.1 (osd.1) 338 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:25.637 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:25 smithi164 conmon[29288]: cluster 2021-02-10T04:32:24.254320+0000 osd.1 (osd.1) 339 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:25.637 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:25 smithi164 conmon[29288]: cluster 2021-02-10T04:32:24.254338+0000 osd.1 (osd.1) 340 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:25.637 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:25 smithi164 conmon[29288]: cluster 2021-02-10T04:32:24.487248+0000 osd.4 (osd.4) 309 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:25.637 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:25 smithi164 conmon[29288]: cluster 2021-02-10T04:32:24.487257+0000 osd.4 (osd.4) 310 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:25.637 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:25 smithi164 conmon[29288]: cluster 2021-02-10T04:32:24.487264+0000 osd.4 (osd.4) 311 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:25.637 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:25 smithi164 conmon[29288]: cluster 2021-02-10T04:32:24.487270+0000 osd.4 (osd.4) 312 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:25.637 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:25 smithi164 conmon[29288]: cluster 2021-02-10T04:32:25.257693+0000 2021-02-10T04:32:25.638 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:25 smithi164 conmon[29288]: osd.2 (osd.2) 392 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:25.638 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:25 smithi164 conmon[29288]: cluster 2021-02-10T04:32:25.257700+0000 osd.2 (osd.2) 393 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:25.638 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:25 smithi164 conmon[29288]: cluster 2021-02-10T04:32:25.257705+0000 osd.2 (osd.2) 394 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:25.638 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:25 smithi164 conmon[29288]: cluster 2021-02-10T04:32:25.257708+0000 osd.2 (osd.2) 395 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:25.639 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:25 smithi164 conmon[29288]: cluster 2021-02-10T04:32:25.257711+0000 osd.2 (osd.2) 396 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:25.639 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:25 smithi164 conmon[29288]: cluster 2021-02-10T04:32:25.257715+0000 osd.2 (osd.2) 397 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:25.639 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:25 smithi164 conmon[29288]: cluster 2021-02-10T04:32:25.257718+0000 osd.2 (osd.2) 398 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:25.639 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:25 smithi164 conmon[29288]: cluster 2021-02-10T04:32:25 2021-02-10T04:32:25.639 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:25 smithi164 conmon[29288]: .257721+0000 osd.2 (osd.2) 399 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:25.639 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:25 smithi164 conmon[29288]: cluster 2021-02-10T04:32:25.363858+0000 mon.a (mon.0) 1080 : cluster [DBG] osdmap e328: 8 total, 8 up, 8 in 2021-02-10T04:32:25.639 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:25 smithi164 conmon[33677]: debug 2021-02-10T04:32:25.529+0000 7fbec19a1700 -1 osd.4 328 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:32:25.640 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:25.796 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:32:25.893 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:25 smithi164 conmon[33677]: debug 2021-02-10T04:32:25.730+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:25.903 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:32:25.939 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:26.386 INFO:tasks.rados.rados.0.smithi085.stdout:1181: done (13 left) 2021-02-10T04:32:26.387 INFO:tasks.rados.rados.0.smithi085.stdout:1182: snap_remove snap 106 2021-02-10T04:32:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: cluster 2021-02-10T04:32:25.225897+0000 osd.1 (osd 2021-02-10T04:32:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: .1) 341 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: cluster 2021-02-10T04:32:25.225902 2021-02-10T04:32:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 342 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: T04:32:25.225906+0000 2021-02-10T04:32:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: osd.1 (osd.1) 343 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: 2021-02-10T04:32:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: cluster 2021-02-10T04:32:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: 2021-02-10T04:32: 2021-02-10T04:32:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: 25.225918+0000 osd.1 (osd.1) 2021-02-10T04:32:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: 344 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: -02-10T04:32:25 2021-02-10T04:32:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: .225922+0000 2021-02-10T04:32:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: osd.1 (osd 2021-02-10T04:32:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: .1) 345 : cluster [WRN] 2021-02-10T04:32:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: cluster 2021-02-10T04:32:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: 2021-02 2021-02-10T04:32:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: -10T04:32:25. 2021-02-10T04:32:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: 225927+0000 osd.1 (osd.1 2021-02-10T04:32:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: ) 346 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: -02-10T04: 2021-02-10T04:32:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: 32:25.530600+0000 osd.4 2021-02-10T04:32:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: (osd.4) 313 : cluster 2021-02-10T04:32:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: -02-10T04:32:25 2021-02-10T04:32:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: .530612+0000 osd.4 (osd.4 2021-02-10T04:32:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: ) 314 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: cluster 2021-02-10T04:32:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:32:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: 04:32:25.530624+0000 osd.4 (osd 2021-02-10T04:32:26.647 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[33677]: debug 2021-02-10T04:32:26.481+0000 7fbec19a1700 -1 osd.4 329 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:32:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: .4) 315 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: 02-10T04:32:25.530627+0000 osd.4 (osd 2021-02-10T04:32:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: .4) 316 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: -02-10T04:32:26.056274+0000 mgr.y 2021-02-10T04:32:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: (mgr.14140) 480 : cluster [DBG] pgmap v642: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 169 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.9 MiB/s rd, 3.5 MiB/s wr, 11 op/s 2021-02-10T04:32:26.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: cluster 2021-02-10T04:32:26.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: 2021-02-10T04:32 2021-02-10T04:32:26.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: :26.225090+0000 osd.2 (osd.2) 400 2021-02-10T04:32:26.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:26.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:26.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: 02-10T04:32:26.225098+0000 osd.2 (osd.2) 401 2021-02-10T04:32:26.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:26.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:26.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: 02-10T04:32:26.225123+0000 2021-02-10T04:32:26.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: osd.2 (osd.2) 402 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:26.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: 2021-02-10T04:32:26.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: cluster 2021-02-10T04:32 2021-02-10T04:32:26.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: :26.225129+0000 osd.2 (osd.2) 403 2021-02-10T04:32:26.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:26.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:26.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: -02-10T04:32:26.225137+0000 2021-02-10T04:32:26.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: osd.2 (osd.2) 404 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:26.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: 2021-02-10T04:32:26.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:32:26.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: :32:26.225145+0000 osd.2 (osd. 2021-02-10T04:32:26.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: 2) 405 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:26.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:26.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: 02-10T04:32:26.225152 2021-02-10T04:32:26.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 406 : cluster [WRN] 2021-02-10T04:32:26.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:26.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: cluster 2021-02-10T04:32:26.225157+0000 2021-02-10T04:32:26.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:26 smithi164 conmon[29288]: osd.2 (osd.2) 407 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:26.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[49106]: debug 2021-02-10T04:32:26.224+0000 7f77a3e94700 -1 osd.2 328 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:32:26.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: cluster 2021-02-10T04:32:25.225897+0000 osd.1 (osd.1) 341 : 2021-02-10T04:32:26.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:26.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: cluster 2021-02-10T04:32:25.225902+0000 osd.1 (osd.1) 342 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:26.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:26.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: :32:25.225906+0000 osd.1 (osd.1) 343 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:26.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: cluster 2021-02-10T04:32:26.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: 2021-02-10T04:32:25.225918+0000 osd.1 (osd.1) 344 : 2021-02-10T04:32:26.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:26.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:32:26.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: 32:25.225922+0000 osd.1 (osd.1) 345 2021-02-10T04:32:26.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:26.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:32:26.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: T04:32:25.225927+0000 osd.1 2021-02-10T04:32:26.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: (osd.1) 346 : cluster 2021-02-10T04:32:26.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:26.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:26.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: 10T04:32:25.530600+0000 osd.4 (osd.4) 313 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:26.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: cluster 2021-02-10T04:32:26.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: 2021-02-10T04:32:25.530612 2021-02-10T04:32:26.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 314 : cluster 2021-02-10T04:32:26.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:26.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:32:26.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: T04:32:25.530624+0000 osd.4 ( 2021-02-10T04:32:26.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: osd.4) 315 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:26.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: cluster 2021-02-10T04:32:26.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: 2021-02-10T04:32:25. 2021-02-10T04:32:26.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: 530627+0000 osd.4 (osd.4) 316 : cluster [WRN] 2021-02-10T04:32:26.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:26.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: cluster 2021-02-10T04:32: 2021-02-10T04:32:26.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: 26.056274+0000 mgr.y (mgr.14140) 480 : cluster 2021-02-10T04:32:26.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: [DBG] pgmap v642: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 169 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.9 MiB/s rd, 3.5 MiB/s wr, 11 op/s 2021-02-10T04:32:26.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:26.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: 10T04:32:26.225090+0000 osd.2 2021-02-10T04:32:26.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: (osd.2) 400 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:26.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: cluster 2021-02-10T04:32:26.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: 2021-02-10T04:32:26.225098 2021-02-10T04:32:26.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 401 : cluster [WRN] 2021-02-10T04:32:26.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:26.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: cluster 2021-02-10T04:32 2021-02-10T04:32:26.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: :26.225123+0000 osd.2 (osd.2) 402 : 2021-02-10T04:32:26.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:26.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:26.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: 02-10T04:32:26.225129+0000 2021-02-10T04:32:26.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: osd.2 (osd.2) 403 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:26.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: 2021-02-10T04:32:26.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: cluster 2021-02-10T04:32:26.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: 2021- 2021-02-10T04:32:26.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: 02-10T04:32:26.225137+0000 2021-02-10T04:32:26.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: osd.2 (osd.2) 404 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:26.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: 2021-02-10T04:32:26.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: cluster 2021-02-10T04:32:26.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: 2021-02-10T04:32:26.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: 2021 2021-02-10T04:32:26.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: - 2021-02-10T04:32:26.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: 02 2021-02-10T04:32:26.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: - 2021-02-10T04:32:26.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: 10T04:32:26.225145+0000 2021-02-10T04:32:26.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: osd.2 (osd.2) 405 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:26.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:32:26.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: T04:32:26.225152+0000 osd.2 ( 2021-02-10T04:32:26.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: osd.2) 406 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:26.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:26.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: -10T04:32:26.225157+0000 osd.2 2021-02-10T04:32:26.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[35961]: (osd.2) 407 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:26.700 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[45443]: debug 2021-02-10T04:32:26.191+0000 7f7140d73700 -1 osd.1 328 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:32:26.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: cluster 2021-02-10T04:32:26.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: 2021-02-10T04:32:26.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: 2021 2021-02-10T04:32:26.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: - 2021-02-10T04:32:26.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: 02 2021-02-10T04:32:26.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: - 2021-02-10T04:32:26.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: 10 2021-02-10T04:32:26.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: T 2021-02-10T04:32:26.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: 04 2021-02-10T04:32:26.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: : 2021-02-10T04:32:26.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: 32 2021-02-10T04:32:26.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: : 2021-02-10T04:32:26.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: 25 2021-02-10T04:32:26.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: . 2021-02-10T04:32:26.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: 225897 2021-02-10T04:32:26.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: +0000 osd.1 (osd.1) 341 : cluster [WRN] 2021-02-10T04:32:26.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:26.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:26.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: 02-10T04:32:25.225902+0000 osd.1 (osd.1) 342 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:26.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: cluster 2021-02-10T04:32:26.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: 2021-02-10T04:32: 2021-02-10T04:32:26.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: 25.225906+0000 osd.1 (osd.1) 343 : cluster 2021-02-10T04:32:26.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:26.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:26.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: -10T04:32:25.225918+0000 osd.1 2021-02-10T04:32:26.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: (osd.1) 344 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:26.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:26.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: -10T04:32:25.225922+0000 osd.1 (osd.1) 2021-02-10T04:32:26.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: 345 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:26.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: cluster 2021-02-10T04:32:26.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:32:26.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: T04:32:25.225927+0000 osd.1 (osd.1) 346 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:26.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: cluster 2021-02-10T04:32:26.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: 2021-02-10T04:32:25.530600+0000 2021-02-10T04:32:26.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: osd.4 (osd.4) 313 : cluster [WRN] 2021-02-10T04:32:26.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:26.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: cluster 2021-02-10T04:32:25. 2021-02-10T04:32:26.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: 530612+0000 osd.4 (osd.4) 314 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:26.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:26.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: -02-10T04:32:25.530624+0000 osd.4 (osd.4) 315 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:26.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:32:26.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: T04:32:25.530627+0000 osd.4 (osd 2021-02-10T04:32:26.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: .4) 316 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:26.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:32:26.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: T04:32:26.056274+0000 mgr.y (mgr.14140) 480 : cluster 2021-02-10T04:32:26.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: [DBG] pgmap v642: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 169 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.9 MiB/s rd, 3.5 MiB/s wr, 11 op/s 2021-02-10T04:32:26.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:32:26.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: 04:32:26.225090+0000 osd.2 (osd.2) 400 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:26.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: 2021-02-10T04:32:26.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: cluster 2021-02-10T04:32:26.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: 2021-02-10T04:32:26.225098+0000 osd.2 (osd.2) 401 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:26.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: 2021-02-10T04:32:26.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: cluster 2021-02-10T04:32:26.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: 2021-02-10T04:32:26.225123+0000 osd.2 ( 2021-02-10T04:32:26.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: osd.2) 402 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:26.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:32:26.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: 04:32:26.225129+0000 osd.2 (osd.2 2021-02-10T04:32:26.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: ) 403 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:26.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:26.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: -10T04:32:26.225137+0000 osd.2 (osd.2) 404 : cluster [WRN] 2021-02-10T04:32:26.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:26.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: cluster 2021-02-10T04:32:26.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: 2021-02-10T04:32:26.225145+0000 osd.2 ( 2021-02-10T04:32:26.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: osd.2) 405 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:26.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:26.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: 02-10T04:32:26.225152+0000 osd.2 (osd. 2021-02-10T04:32:26.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: 2) 406 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:26.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:26.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: -10T04:32:26.225157+0000 osd.2 (osd 2021-02-10T04:32:26.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:26 smithi085 conmon[29319]: .2) 407 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:27.387 INFO:tasks.rados.rados.0.smithi085.stdout:1182: done (13 left) 2021-02-10T04:32:27.387 INFO:tasks.rados.rados.0.smithi085.stdout:1183: read oid 5 snap -1 2021-02-10T04:32:27.387 INFO:tasks.rados.rados.0.smithi085.stdout:1183: expect deleted 2021-02-10T04:32:27.387 INFO:tasks.rados.rados.0.smithi085.stdout:1184: copy_from oid 31 from oid 16 current snap is 117 2021-02-10T04:32:27.388 INFO:tasks.rados.rados.0.smithi085.stdout:1185: read oid 8 snap -1 2021-02-10T04:32:27.388 INFO:tasks.rados.rados.0.smithi085.stdout:1185: expect deleted 2021-02-10T04:32:27.388 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:27.388 INFO:tasks.rados.rados.0.smithi085.stdout:1185: done (15 left) 2021-02-10T04:32:27.388 INFO:tasks.rados.rados.0.smithi085.stdout:1186: snap_create 2021-02-10T04:32:27.507 INFO:tasks.rados.rados.0.smithi085.stdout:1184: finishing copy_from to smithi08575397-31 2021-02-10T04:32:27.508 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 616 (ObjNum 432 snap 114 seq_num 432) dirty exists 2021-02-10T04:32:27.508 INFO:tasks.rados.rados.0.smithi085.stdout:1184: finishing copy_from racing read to smithi08575397-31 2021-02-10T04:32:27.578 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[49106]: debug 2021-02-10T04:32:27.578 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[49106]: 2021-02-10T04:32:27.227+0000 7f77a3e94700 -1 osd.2 329 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:32:27.578 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[49106]: 2021-02-10T04:32:27.579 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[45443]: debug 2021-02-10T04:32:27.579 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[45443]: 2021-02-10T04:32:27.199+0000 7f7140d73700 -1 osd.1 329 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:32:27.579 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[45443]: 2021-02-10T04:32:27.579 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[45443]: debug 2021-02-10T04:32:27.579 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[45443]: 2021-02-10T04:32:27.200+0000 7f7125cf8700 -1 osd.1 pg_epoch: 329 pg[4.4( v 63'31 (0'0,63'31] local-lis/les=55/56 n=14 ec=55/55 lis/c=55/55 les/c/f=56/56/0 sis=55) [1,2,3] r=0 lpr=55 crt=63'31 lcod 63'30 mlcod 63'30 active+clean] scrubber pg(4.4) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 3 (0x55dbcdb32dc0) 2021-02-10T04:32:27.579 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[45443]: 2021-02-10T04:32:27.580 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: cluster 2021-02-10T04:32:26.191865+0000 osd.1 (osd 2021-02-10T04:32:27.580 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: .1) 347 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:27.580 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: cluster 2021-02-10T04:32:26.191871+0000 osd.1 (osd.1) 348 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:27.580 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: cluster 2021-02-10T04:32:26.191875+0000 osd.1 (osd.1) 349 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:27.580 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: cluster 2021-02-10T04:32: 2021-02-10T04:32:27.580 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: 26.191878+0000 osd.1 (osd.1) 350 : cluster [WRN] 2021-02-10T04:32:27.580 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:27.580 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: cluster 2021-02-10T04:32 2021-02-10T04:32:27.580 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: :26.191881+0000 osd.1 (osd.1) 351 : cluster 2021-02-10T04:32:27.580 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:27.581 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: cluster 2021-02-10T04:32:26.191886+0000 2021-02-10T04:32:27.581 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: osd.1 (osd.1) 352 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:27.581 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:27.581 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: -10T04:32:26.385968+0000 mon.a ( 2021-02-10T04:32:27.581 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: mon.0) 1081 : cluster [DBG] osdmap e329: 8 total, 8 up, 8 in 2021-02-10T04:32:27.581 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:27.581 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: 10T04:32:26.481942+0000 osd.4 (osd. 2021-02-10T04:32:27.581 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: 4) 317 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:27.581 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:32:27.581 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: 04:32:26.481953+0000 osd.4 (osd. 2021-02-10T04:32:27.581 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: 4) 318 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:27.581 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:32:27.582 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: 04:32:26.481960+0000 osd.4 (osd.4) 2021-02-10T04:32:27.582 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: 319 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:27.582 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: cluster 2021-02-10T04:32 2021-02-10T04:32:27.582 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: :26.481980+0000 osd.4 (osd.4) 320 : cluster 2021-02-10T04:32:27.582 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:27.582 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: cluster 2021-02-10T04:32: 2021-02-10T04:32:27.582 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: 27.228313+0000 osd.2 (osd.2) 408 : cluster [WRN] 2021-02-10T04:32:27.582 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:27.582 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: cluster 2021-02-10T04:32:27.582 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: 2021-02-10T04:32:27.228322+0000 osd.2 (osd. 2021-02-10T04:32:27.582 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: 2) 409 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:27.583 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:32:27.583 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: 32:27.228328+0000 osd.2 (osd.2) 410 : cluster 2021-02-10T04:32:27.583 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:27.583 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: cluster 2021-02-10T04:32:27.228334 2021-02-10T04:32:27.583 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 411 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:27.583 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: 2021-02-10T04:32:27.584 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: cluster 2021-02-10T04:32:27.584 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: 2021-02-10T04:32:27.228340+0000 osd.2 (osd. 2021-02-10T04:32:27.584 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: 2) 412 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:27.584 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:32:27.584 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: T04:32:27.228345+0000 osd.2 (osd. 2021-02-10T04:32:27.584 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: 2) 413 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:27.584 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:27.584 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: 02-10T04:32:27.228351+0000 osd.2 ( 2021-02-10T04:32:27.584 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: osd.2) 414 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:27.585 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:27.585 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: 02-10T04:32:27.228357+0000 osd.2 2021-02-10T04:32:27.585 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[29319]: (osd.2) 415 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:27.585 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: cluster 2021-02-10T04:32:26.191865+0000 osd.1 2021-02-10T04:32:27.585 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: (osd.1) 347 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:27.585 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: cluster 2021-02-10T04:32:26.191871+0000 osd.1 (osd.1) 2021-02-10T04:32:27.585 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: 348 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:27.586 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:27.586 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: -10T04:32:26.191875+0000 osd.1 (osd. 2021-02-10T04:32:27.586 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: 1) 349 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:27.586 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:27.586 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: 04:32:26.191878+0000 osd.1 (osd.1) 350 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:27.586 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:32:27.586 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: 32:26.191881+0000 osd.1 (osd.1) 351 : 2021-02-10T04:32:27.586 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:27.587 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: cluster 2021-02-10T04:32:26 2021-02-10T04:32:27.587 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: .191886+0000 osd.1 (osd.1) 352 : cluster [WRN] 2021-02-10T04:32:27.587 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:27.587 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: cluster 2021-02-10T04:32:26.385968 2021-02-10T04:32:27.587 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1081 : cluster [DBG] 2021-02-10T04:32:27.587 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: osdmap e329: 8 total, 8 up, 8 in 2021-02-10T04:32:27.588 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: cluster 2021-02-10T04:32:27.588 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: 2021-02-10T04:32:26.481942+0000 osd.4 (osd 2021-02-10T04:32:27.588 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: .4) 317 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:27.588 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:27.588 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: 10T04:32:26.481953+0000 osd.4 (osd.4) 2021-02-10T04:32:27.588 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: 318 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:27.588 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:27.588 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: :32:26.481960+0000 osd.4 (osd.4) 319 2021-02-10T04:32:27.588 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:27.588 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: cluster 2021-02-10T04:32:26. 2021-02-10T04:32:27.588 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: 481980+0000 osd.4 (osd.4) 320 : cluster 2021-02-10T04:32:27.589 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:27.589 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:27.589 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: -02-10T04:32:27.228313 2021-02-10T04:32:27.589 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 408 2021-02-10T04:32:27.589 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:27.589 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: cluster 2021-02-10T04:32:27.589 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: 2021-02-10T04:32 2021-02-10T04:32:27.589 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: :27.228322+0000 osd.2 ( 2021-02-10T04:32:27.589 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: osd.2) 409 : cluster 2021-02-10T04:32:27.589 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:27.589 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:27.589 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: -02-10T04:32: 2021-02-10T04:32:27.590 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: 27.228328+0000 osd.2 (osd.2 2021-02-10T04:32:27.590 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: ) 410 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:27.590 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:27.590 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: 02-10T04:32:27. 2021-02-10T04:32:27.590 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: 228334+0000 osd.2 (osd.2 2021-02-10T04:32:27.590 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: ) 411 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:27.590 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: 2021-02-10T04:32:27.590 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:27.590 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: -10T04:32:27. 2021-02-10T04:32:27.591 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: 228340+0000 osd.2 (osd.2 2021-02-10T04:32:27.591 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: ) 412 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:27.591 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:27.591 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: 02-10T04:32:27.228345+0000 osd.2 2021-02-10T04:32:27.591 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: (osd.2) 413 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:27.591 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:27.591 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: -02-10T04:32: 2021-02-10T04:32:27.591 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: 27.228351+0000 osd.2 (osd 2021-02-10T04:32:27.591 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: .2) 414 : cluster [WRN] 2021-02-10T04:32:27.591 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:27.591 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: cluster 2021-02-10T04:32:27.591 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:32:27.591 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: 32:27.228357+0000 osd.2 ( 2021-02-10T04:32:27.592 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[35961]: osd.2) 415 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:27.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[33677]: debug 2021-02-10T04:32:27.504+0000 7fbec19a1700 -1 osd.4 330 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:32:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: cluster 2021-02-10T04:32:26.191865+0000 osd.1 2021-02-10T04:32:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: (osd.1) 347 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: cluster 2021-02-10T04:32:26 2021-02-10T04:32:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: .191871+0000 osd.1 (osd.1) 348 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: cluster 2021-02-10T04:32:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: 2021-02-10T04:32:26.191875+0000 2021-02-10T04:32:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: osd.1 (osd.1) 349 : cluster 2021-02-10T04:32:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: cluster 2021-02-10T04:32:26.191878 2021-02-10T04:32:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 350 : cluster [WRN] 2021-02-10T04:32:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: cluster 2021-02-10T04:32: 2021-02-10T04:32:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: 26.191881+0000 osd.1 (osd.1) 351 : cluster 2021-02-10T04:32:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: cluster 2021-02-10T04:32:26.191886 2021-02-10T04:32:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 352 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: 2021-02-10T04:32:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: -10T04:32:26.385968+0000 mon.a (mon.0) 1081 : cluster 2021-02-10T04:32:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: [DBG] osdmap e329: 8 total, 8 up, 8 in 2021-02-10T04:32:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:32:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: :32:26.481942+0000 osd.4 (osd.4) 2021-02-10T04:32:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: 317 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: T04:32:26.481953+0000 osd.4 (osd.4 2021-02-10T04:32:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: ) 318 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: cluster 2021-02-10T04:32:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: 2021-02- 2021-02-10T04:32:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: 10T04:32:26.481960+0000 osd.4 2021-02-10T04:32:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: (osd.4) 319 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: 2021-02-10T04:32:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: cluster 2021-02-10T04:32:26.481980+0000 osd.4 ( 2021-02-10T04:32:27.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: osd.4) 320 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:27.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:32:27.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: 32:27.228313+0000 osd.2 (osd.2) 408 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:27.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: cluster 2021-02-10T04:32: 2021-02-10T04:32:27.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: 27.228322+0000 osd.2 (osd.2) 409 : cluster [WRN] 2021-02-10T04:32:27.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:27.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:27.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: -10T04:32:27.228328+0000 osd.2 (osd 2021-02-10T04:32:27.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: .2) 410 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:27.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: cluster 2021-02-10T04:32:27.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: 2021-02-10T04:32 2021-02-10T04:32:27.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: :27.228334+0000 osd.2 ( 2021-02-10T04:32:27.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: osd.2) 411 : cluster [WRN] 2021-02-10T04:32:27.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:27.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: cluster 2021-02-10T04:32:27.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: 2021-02-10T04:32: 2021-02-10T04:32:27.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: 27.228340+0000 osd.2 (osd.2) 412 : cluster 2021-02-10T04:32:27.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:27.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:32:27.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: 10T04:32:27.228345+0000 osd.2 (osd 2021-02-10T04:32:27.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: .2) 413 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:27.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: cluster 2021-02-10T04:32:27.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:32:27.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: 04:32:27.228351+0000 osd.2 (osd 2021-02-10T04:32:27.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: .2) 414 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:27.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: cluster 2021-02-10T04:32:27.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: 2021-02-10T04:32:27.228357 2021-02-10T04:32:27.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 415 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:27.941 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:27 smithi085 conmon[52836]: debug 2021-02-10T04:32:27.577+0000 7fba6ffae700 -1 osd.3 pg_epoch: 330 pg[4.14( v 63'26 (0'0,63'26] local-lis/les=71/72 n=11 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,2,7] r=0 lpr=71 crt=63'26 lcod 63'25 mlcod 0'0 active+clean] scrubber pg(4.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55fec096fa20) 2021-02-10T04:32:28.387 INFO:tasks.rados.rados.0.smithi085.stdout:1183: done (15 left) 2021-02-10T04:32:28.387 INFO:tasks.rados.rados.0.smithi085.stdout:1184: done (14 left) 2021-02-10T04:32:28.387 INFO:tasks.rados.rados.0.smithi085.stdout:1186: done (13 left) 2021-02-10T04:32:28.387 INFO:tasks.rados.rados.0.smithi085.stdout:1187: snap_create 2021-02-10T04:32:28.393 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:32:27 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:32:28.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[49106]: debug 2021-02-10T04:32:28.195+0000 7f77a3e94700 -1 osd.2 330 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:32:28.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:28.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: -10T04:32:27.200116+0000 osd.1 (osd.1) 353 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:28.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: 2021-02-10T04:32:28.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:32:28.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: T04:32:27.200123+0000 osd.1 (osd.1) 2021-02-10T04:32:28.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: 354 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:28.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: cluster 2021-02-10T04:32:28.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: 2021-02-10T04:32:28.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: 2021-02-10T04:32:27 2021-02-10T04:32:28.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: .200129+0000 osd.1 (osd.1) 355 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:28.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: cluster 2021-02-10T04:32: 2021-02-10T04:32:28.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: 27.200134+0000 osd.1 (osd.1) 356 2021-02-10T04:32:28.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:28.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:28.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: 02-10T04:32:27.200139+0000 osd.1 (osd 2021-02-10T04:32:28.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: .1) 357 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:28.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:28.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: -10T04:32:27.200145+0000 osd.1 2021-02-10T04:32:28.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: (osd.1) 358 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:28.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:28.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: 02-10T04:32:27.386360+0000 mon.a (mon. 2021-02-10T04:32:28.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: 0) 1082 : cluster [DBG] osdmap e330: 8 total, 8 up, 8 in 2021-02-10T04:32:28.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: cluster 2021-02-10T04:32:28.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:32:28.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: 32:27.390599+0000 osd.3 (osd.3) 58 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:32:28.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: cluster 2021-02-10T04:32:27 2021-02-10T04:32:28.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: .391530+0000 osd.3 (osd.3) 59 : cluster 2021-02-10T04:32:28.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: [DBG] purged_snaps scrub ok 2021-02-10T04:32:28.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: cluster 2021-02-10T04:32: 2021-02-10T04:32:28.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: 27.504860+0000 osd.4 (osd.4) 321 : cluster [WRN] 2021-02-10T04:32:28.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:28.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: cluster 2021-02-10T04:32:27.504868 2021-02-10T04:32:28.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 322 : cluster [WRN] 2021-02-10T04:32:28.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:28.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:28.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: 02-10T04:32: 2021-02-10T04:32:28.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: 27.504887+0000 osd.4 (osd.4 2021-02-10T04:32:28.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: ) 323 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:28.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:28.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: 02-10T04:32:27 2021-02-10T04:32:28.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: .504893+0000 osd.4 (osd.4) 2021-02-10T04:32:28.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: 324 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:28.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: audit 2021-02-10T04:32: 2021-02-10T04:32:28.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: 27.905020+0000 mgr.y (mgr.14140 2021-02-10T04:32:28.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: ) 481 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:32:28.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:28.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: 10T04:32:28.056819+0000 mgr.y 2021-02-10T04:32:28.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: (mgr.14140) 482 : cluster [DBG] pgmap v645: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 169 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 10 op/s 2021-02-10T04:32:28.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: 2021-02-10T04:32:28.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:32:28.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: 04:32:28.092777+0000 mon.a (mon.0) 1083 : cluster 2021-02-10T04:32:28.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: [WRN] Health check update: 18 slow ops, oldest one blocked for 192 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:32:28.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: cluster 2021-02-10T04:32:28.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: 2021-02-10T04:32:28.196320+0000 osd.2 (osd.2) 2021-02-10T04:32:28.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: 416 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:28.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:28.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: 10T04:32:28.196347+0000 osd.2 2021-02-10T04:32:28.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: (osd.2) 417 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:28.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: cluster 2021-02-10T04:32:28.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: 2021-02-10T04:32:28 2021-02-10T04:32:28.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: .196354+0000 osd.2 (osd.2) 418 : 2021-02-10T04:32:28.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:28.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:32:28.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: 04:32:28.196358+0000 osd.2 ( 2021-02-10T04:32:28.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: osd.2) 419 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:28.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:32:28.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: T04:32:28.196361+0000 osd.2 ( 2021-02-10T04:32:28.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: osd.2) 420 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:28.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:28.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: -02-10T04:32:28.196366+0000 osd.2 ( 2021-02-10T04:32:28.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: osd.2) 421 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:28.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: cluster 2021-02-10T04:32:28.196370+0000 2021-02-10T04:32:28.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: osd.2 (osd.2) 422 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:28.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: cluster 2021-02-10T04:32:28. 2021-02-10T04:32:28.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: 196383+0000 osd.2 (osd.2) 423 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:28.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: cluster 2021-02-10T04:32: 2021-02-10T04:32:28.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: 28.387334+0000 mon.a (mon.0) 1084 : cluster [DBG] osdmap e331: 8 total, 8 up, 8 in 2021-02-10T04:32:28.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[29319]: 2021-02-10T04:32:28.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: cluster 2021-02-10T04:32:27.200116 2021-02-10T04:32:28.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 353 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:28.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: cluster 2021-02-10T04:32:27.200123 2021-02-10T04:32:28.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 354 : cluster [WRN] 2021-02-10T04:32:28.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:28.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:28.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: 04:32:27.200129+0000 osd.1 (osd 2021-02-10T04:32:28.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: .1) 355 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:28.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: cluster 2021-02-10T04:32:28.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: 2021-02-10T04:32: 2021-02-10T04:32:28.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: 27.200134+0000 osd.1 (osd.1) 356 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:28.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: cluster 2021-02-10T04:32:28.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:32:28.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: :32:27.200139+0000 osd.1 (osd.1) 357 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:28.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:28.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: -02-10T04:32:27.200145+0000 2021-02-10T04:32:28.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: osd.1 (osd.1) 358 : cluster [WRN] 2021-02-10T04:32:28.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:28.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: cluster 2021-02-10T04:32 2021-02-10T04:32:28.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: :27.386360+0000 mon.a (mon 2021-02-10T04:32:28.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: .0) 1082 : cluster [DBG] osdmap e330: 8 total, 8 up, 8 in 2021-02-10T04:32:28.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:28.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: 02-10T04: 2021-02-10T04:32:28.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: 32:27.390599+0000 osd.3 (osd.3) 2021-02-10T04:32:28.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: 58 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:32:28.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: cluster 2021-02-10T04:32:28.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:32:28.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: 10T04:32:27.391530+0000 osd.3 (osd.3) 59 : cluster 2021-02-10T04:32:28.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: [DBG] purged_snaps scrub ok 2021-02-10T04:32:28.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: cluster 2021-02-10T04:32:28.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: 2021-02-10T04:32:27.504860+0000 2021-02-10T04:32:28.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: osd.4 (osd.4) 321 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:28.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: cluster 2021-02-10T04:32:27 2021-02-10T04:32:28.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: .504868+0000 osd.4 (osd.4) 322 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:28.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:28.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: 10T04:32:27.504887+0000 osd.4 (osd.4) 323 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:28.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: cluster 2021-02-10T04:32:27.504893+0000 2021-02-10T04:32:28.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: osd.4 (osd.4) 324 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:28.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: audit 2021-02-10T04:32:27.905020+0000 2021-02-10T04:32:28.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: mgr.y (mgr.14140) 481 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:32:28.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: cluster 2021-02-10T04:32:28.056819+0000 2021-02-10T04:32:28.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: mgr.y (mgr.14140) 482 : cluster [DBG] pgmap v645: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 169 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 10 op/s 2021-02-10T04:32:28.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: cluster 2021-02-10T04:32:28. 2021-02-10T04:32:28.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: 092777+0000 mon.a (mon.0) 1083 : cluster [WRN] Health check update: 18 slow ops, oldest one blocked for 192 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:32:28.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: cluster 2021-02-10T04:32 2021-02-10T04:32:28.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: :28.196320+0000 osd.2 (osd.2) 416 2021-02-10T04:32:28.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:28.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: cluster 2021-02-10T04:32:28.196347+0000 2021-02-10T04:32:28.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: osd.2 (osd.2) 417 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:28.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: cluster 2021-02-10T04:32:28.196354+0000 2021-02-10T04:32:28.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: osd.2 (osd.2) 418 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:28.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: cluster 2021-02-10T04:32:28.196358 2021-02-10T04:32:28.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 419 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:28.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: cluster 2021-02-10T04:32: 2021-02-10T04:32:28.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: 28.196361+0000 osd.2 (osd.2) 420 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:28.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: 2021-02-10T04:32:28.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: cluster 2021-02-10T04:32 2021-02-10T04:32:28.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: :28.196366+0000 osd.2 (osd.2) 421 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:28.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: cluster 2021-02-10T04:32:28 2021-02-10T04:32:28.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: .196370+0000 osd.2 (osd.2) 422 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:28.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:28.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: :32:28.196383+0000 osd.2 (osd.2) 423 : cluster 2021-02-10T04:32:28.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:28.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:28.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: 02-10T04:32:28.387334+0000 mon.a (mon.0) 1084 : 2021-02-10T04:32:28.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[35961]: cluster [DBG] osdmap e331: 8 total, 8 up, 8 in 2021-02-10T04:32:28.453 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[45443]: debug 2021-02-10T04:32:28.188+0000 7f7140d73700 -1 osd.1 330 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:32:28.453 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[45443]: 2021-02-10T04:32:28.453 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[45443]: debug 2021-02-10T04:32:28.453 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[45443]: 2021-02-10T04:32:28.189+0000 7f7125cf8700 -1 osd.1 pg_epoch: 330 pg[4.4( v 63'31 (0'0,63'31] local-lis/les=55/56 n=14 ec=55/55 lis/c=55/55 les/c/f=56/56/0 sis=55) [1,2,3] r=0 lpr=55 crt=63'31 lcod 63'30 mlcod 63'30 active+clean] scrubber pg(4.4) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 3 (0x55dbcb0c7600) 2021-02-10T04:32:28.453 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[45443]: 2021-02-10T04:32:28.454 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:32:28 smithi085 conmon[41540]: debug 2021-02-10T04:32:28.281+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:28.557 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:32:28.893 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[33677]: debug 2021-02-10T04:32:28.463+0000 7fbec19a1700 -1 osd.4 331 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:32:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: cluster 2021-02-10T04:32:27.200116+0000 osd.1 (osd.1) 2021-02-10T04:32:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: 353 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: cluster 2021-02-10T04:32:27.200123+0000 osd.1 (osd.1) 354 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:32:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: :32:27.200129+0000 osd.1 (osd.1) 355 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: cluster 2021-02-10T04:32:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: 2021-02-10T04:32:27.200134+0000 osd.1 2021-02-10T04:32:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: (osd.1) 356 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: cluster 2021-02-10T04:32:27. 2021-02-10T04:32:28.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: 200139+0000 osd.1 (osd.1) 357 : cluster [WRN] 2021-02-10T04:32:28.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:28.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:28.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: 02-10T04:32:27.200145+0000 osd.1 ( 2021-02-10T04:32:28.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: osd.1) 358 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:28.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:32:28.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: 10T04:32:27.386360+0000 mon.a (mon.0) 1082 : cluster 2021-02-10T04:32:28.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: [DBG] osdmap e330: 8 total, 8 up, 8 in 2021-02-10T04:32:28.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:28.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: -10T04:32:27.390599+0000 osd.3 (osd.3) 58 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:32:28.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:32:28.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: 32:27.391530+0000 osd.3 (osd.3) 59 : cluster 2021-02-10T04:32:28.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: [DBG] purged_snaps scrub ok 2021-02-10T04:32:28.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:28.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: 02-10T04:32:27.504860+0000 osd.4 (osd.4) 321 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:28.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: cluster 2021-02-10T04:32:27.504868+0000 osd.4 (osd.4) 322 : cluster [WRN] 2021-02-10T04:32:28.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:28.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: cluster 2021-02-10T04:32:27 2021-02-10T04:32:28.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: .504887+0000 osd.4 (osd.4) 323 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:28.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:28.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: -10T04:32:27.504893+0000 osd.4 (osd.4) 324 : 2021-02-10T04:32:28.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:28.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: audit 2021-02-10T04:32: 2021-02-10T04:32:28.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: 27.905020+0000 mgr.y (mgr.14140) 481 : 2021-02-10T04:32:28.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:32:28.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: cluster 2021-02-10T04:32:28.056819+0000 2021-02-10T04:32:28.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: mgr.y (mgr.14140) 482 : cluster [DBG] pgmap v645: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 169 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 10 op/s 2021-02-10T04:32:28.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:28.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: 04:32:28.092777+0000 mon.a (mon.0) 1083 : cluster [WRN] 2021-02-10T04:32:28.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: Health check update: 18 slow ops, oldest one blocked for 192 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:32:28.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:32:28.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: 32:28.196320+0000 osd.2 (osd.2) 416 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:28.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: cluster 2021-02-10T04:32:28.196347+0000 osd.2 2021-02-10T04:32:28.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: (osd.2) 417 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:28.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: cluster 2021-02-10T04:32:28.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: 2021-02-10T04:32:28.196354+0000 osd.2 (osd.2) 418 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:28.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: cluster 2021-02-10T04:32:28.196358+0000 osd.2 2021-02-10T04:32:28.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: (osd.2) 419 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:28.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: cluster 2021-02-10T04:32:28.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: 2021-02-10T04:32:28.196361+0000 osd.2 (osd.2) 420 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:28.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: cluster 2021-02-10T04:32:28.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: 2021-02-10T04:32:28.196366+0000 osd.2 (osd.2) 421 : cluster [WRN] 2021-02-10T04:32:28.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:28.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:28.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: 02-10T04:32:28.196370+0000 osd.2 (osd.2) 422 2021-02-10T04:32:28.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:28.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:28.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: -02-10T04:32:28 2021-02-10T04:32:28.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: .196383+0000 osd.2 (osd.2) 423 : cluster [WRN] 2021-02-10T04:32:28.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:28.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: cluster 2021-02-10T04:32:28.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: 2021-02-10T04:32:28.387334+0000 mon.a (mon.0) 2021-02-10T04:32:28.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:28 smithi164 conmon[29288]: 1084 : cluster [DBG] osdmap e331: 8 total, 8 up, 8 in 2021-02-10T04:32:29.391 INFO:tasks.rados.rados.0.smithi085.stdout:1187: done (13 left) 2021-02-10T04:32:29.391 INFO:tasks.rados.rados.0.smithi085.stdout:1188: rollback oid 5 current snap is 119 2021-02-10T04:32:29.391 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 5 to 117 2021-02-10T04:32:29.392 INFO:tasks.rados.rados.0.smithi085.stdout:1189: read oid 30 snap -1 2021-02-10T04:32:29.392 INFO:tasks.rados.rados.0.smithi085.stdout:1189: expect deleted 2021-02-10T04:32:29.392 INFO:tasks.rados.rados.0.smithi085.stdout:1190: copy_from oid 6 from oid 31 current snap is 119 2021-02-10T04:32:29.392 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:29.392 INFO:tasks.rados.rados.0.smithi085.stdout:1188: finishing rollback tid 1 to smithi08575397-5 2021-02-10T04:32:29.392 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 0 (ObjNum 0 snap 32576 seq_num 82304672) dirty dne 2021-02-10T04:32:29.392 INFO:tasks.rados.rados.0.smithi085.stdout:1188: done (15 left) 2021-02-10T04:32:29.392 INFO:tasks.rados.rados.0.smithi085.stdout:1191: snap_remove snap 114 2021-02-10T04:32:29.478 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[49106]: debug 2021-02-10T04:32:29.216+0000 7f77a3e94700 -1 osd.2 331 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:32:29.480 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[45443]: debug 2021-02-10T04:32:29.480 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[45443]: 2021-02-10T04:32:29.232+0000 7f7140d73700 -1 osd.1 331 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:32:29.480 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[45443]: debug 2021-02-10T04:32:29.233+0000 7f7121cf0700 -1 osd.1 pg_epoch: 331 pg[4.4( v 63'31 (0'0,63'31] local-lis/les=55/56 n=14 ec=55/55 lis/c=55/55 les/c/f=56/56/0 sis=55) [1,2,3] r=0 lpr=55 crt=63'31 lcod 63'30 mlcod 63'30 active+clean] scrubber pg(4.4) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 3 (0x55dbcd85a840) 2021-02-10T04:32:29.481 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: cluster 2021-02-10T04:32:28.188909 2021-02-10T04:32:29.481 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 359 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:29.481 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: cluster 2021-02-10T04:32:29.482 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: 2021-02-10T04:32:28.188921+0000 osd.1 (osd.1) 2021-02-10T04:32:29.482 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: 360 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:29.482 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:32:29.482 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: 32:28.188927+0000 osd.1 (osd.1) 2021-02-10T04:32:29.482 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: 361 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:29.483 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:32:29.483 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: T04:32:28.188933+0000 osd.1 (osd 2021-02-10T04:32:29.483 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: .1) 362 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:29.483 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:29.483 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: -10T04:32:28.188939+0000 osd.1 2021-02-10T04:32:29.483 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: (osd.1) 363 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:29.483 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:29.483 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: -02-10T04:32:28.188945+0000 2021-02-10T04:32:29.483 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: osd.1 (osd.1) 364 : cluster [WRN] 2021-02-10T04:32:29.484 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:29.484 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: cluster 2021-02-10T04:32:29.484 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: 2021-02-10T04:32:28.463812+0000 osd.4 ( 2021-02-10T04:32:29.484 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: osd.4) 325 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:29.485 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:29.485 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: -02-10T04:32:28.463825+0000 2021-02-10T04:32:29.485 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: osd.4 (osd.4) 326 : cluster [WRN] 2021-02-10T04:32:29.485 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:29.485 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: cluster 2021-02-10T04:32:29.485 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: 2021-02-10T04:32:28.463833+0000 osd.4 ( 2021-02-10T04:32:29.485 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: osd.4) 327 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:29.485 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:29.485 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: 02-10T04:32:28.463840+0000 2021-02-10T04:32:29.485 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: osd.4 (osd.4) 328 : cluster [WRN] 2021-02-10T04:32:29.485 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:29.485 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: cluster 2021-02-10T04:32:29.485 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: 2021-02-10T04:32:29.216924+0000 osd.2 (osd. 2021-02-10T04:32:29.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: 2) 424 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:29.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:29.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: 10T04:32:29.216934+0000 osd.2 ( 2021-02-10T04:32:29.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: osd.2) 425 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:29.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:29.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: 02-10T04:32:29.216940+0000 2021-02-10T04:32:29.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: osd.2 (osd.2) 426 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:29.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: 2021-02-10T04:32:29.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: cluster 2021-02-10T04:32:29.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: 2021-02-10T04:32:29.216958+0000 osd.2 (osd 2021-02-10T04:32:29.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: .2) 427 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:29.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:29.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: -10T04:32:29.216963+0000 osd.2 2021-02-10T04:32:29.487 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: (osd.2) 428 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:29.487 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:29.487 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: -02-10T04:32:29.216969+0000 2021-02-10T04:32:29.487 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: osd.2 (osd.2) 429 : cluster [WRN] 2021-02-10T04:32:29.487 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:29.487 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: cluster 2021-02-10T04:32:29. 2021-02-10T04:32:29.487 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: 216974+0000 osd.2 (osd.2) 430 : cluster 2021-02-10T04:32:29.487 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:29.487 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: cluster 2021-02-10T04:32: 2021-02-10T04:32:29.487 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: 29.216979+0000 osd.2 (osd.2) 431 2021-02-10T04:32:29.487 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:29.487 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:29.487 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: :32:29.388854+0000 mon.a (mon. 2021-02-10T04:32:29.488 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[35961]: 0) 1085 : cluster [DBG] osdmap e332: 8 total, 8 up, 8 in 2021-02-10T04:32:29.488 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: cluster 2021-02-10T04:32:29.488 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: 2021-02-10T04:32:29.488 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: 2021 2021-02-10T04:32:29.488 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: - 2021-02-10T04:32:29.488 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: 02-10T04:32:28.188909 2021-02-10T04:32:29.488 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: +0000 osd.1 (osd.1) 359 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:29.488 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: cluster 2021-02-10T04:32 2021-02-10T04:32:29.488 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: :28.188921+0000 osd.1 (osd.1) 360 : cluster 2021-02-10T04:32:29.488 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:29.488 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: cluster 2021-02-10T04:32:28.188927+0000 osd.1 (osd.1) 361 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:29.489 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: cluster 2021-02-10T04:32 2021-02-10T04:32:29.489 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: :28.188933+0000 osd.1 (osd.1) 362 : cluster 2021-02-10T04:32:29.489 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:29.489 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:32:29.489 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: :32:28.188939+0000 osd.1 2021-02-10T04:32:29.489 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: (osd.1) 363 : cluster 2021-02-10T04:32:29.489 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:29.489 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: cluster 2021-02-10T04:32:28.188945 2021-02-10T04:32:29.489 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: +0000 osd.1 (osd.1) 364 : cluster 2021-02-10T04:32:29.489 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:29.489 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: cluster 2021-02-10T04:32:29.489 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: 2021-02-10T04:32:28.463812+0000 osd.4 (osd.4) 325 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:29.489 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:29.489 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: -02-10T04:32:28.463825+0000 osd.4 (osd.4) 326 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:29.490 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:32:29.490 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: 32:28.463833+0000 osd.4 (osd.4) 327 : cluster 2021-02-10T04:32:29.490 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:29.490 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: cluster 2021-02-10T04:32:28.463840+0000 2021-02-10T04:32:29.490 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: osd.4 (osd.4) 328 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:29.490 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: cluster 2021-02-10T04:32:29.216924+0000 osd.2 (osd.2) 424 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:29.490 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: cluster 2021-02-10T04:32:29.216934+0000 osd.2 (osd.2) 425 : cluster 2021-02-10T04:32:29.490 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:29.490 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: cluster 2021-02-10T04:32:29. 2021-02-10T04:32:29.490 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: 216940+0000 osd.2 (osd.2) 426 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:29.490 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:29.491 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: -10T04:32:29.216958 2021-02-10T04:32:29.491 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 427 2021-02-10T04:32:29.491 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:29.491 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:29.491 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: 10T04:32:29.216963+0000 osd.2 (osd 2021-02-10T04:32:29.491 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: .2) 428 2021-02-10T04:32:29.491 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:29.491 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: cluster 2021-02-10T04:32:29.491 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: 2021-02-10T04:32:29.491 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:32:29.491 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: :32:29.216969+0000 osd.2 (osd.2) 429 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:29.491 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: cluster 2021-02-10T04:32:29.492 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: 2021-02-10T04:32:29. 2021-02-10T04:32:29.492 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: 216974+0000 osd.2 (osd.2) 430 : cluster 2021-02-10T04:32:29.492 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:29.492 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:29.492 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: 10T04:32:29.216979+0000 osd.2 ( 2021-02-10T04:32:29.492 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: osd.2) 431 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:29.492 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: cluster 2021-02-10T04:32:29. 2021-02-10T04:32:29.492 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: 388854+0000 mon.a (mon.0) 1085 : cluster 2021-02-10T04:32:29.492 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:29 smithi085 conmon[29319]: [DBG] osdmap e332: 8 total, 8 up, 8 in 2021-02-10T04:32:29.582 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:32:29.614 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:29.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[33677]: debug 2021-02-10T04:32:29.459+0000 7fbec19a1700 -1 osd.4 332 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:32:29.645 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[37263]: debug 2021-02-10T04:32:29.384+0000 7f1c6c9ed700 -1 osd.5 pg_epoch: 331 pg[7.0( v 329'281 (0'0,329'281] local-lis/les=93/94 n=5 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [5,1,3] r=0 lpr=93 crt=329'281 lcod 328'279 mlcod 328'279 active+clean+snaptrim trimq=[d5~1,e2~1] ps=65] removing snap head 2021-02-10T04:32:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: cluster 2021-02-10T04:32:28.188909 2021-02-10T04:32:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 359 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: cluster 2021-02-10T04:32:28.188921+0000 osd.1 (osd 2021-02-10T04:32:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: .1) 360 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: -10T04:32:28.188927+0000 osd.1 (osd.1) 361 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: T04:32:28.188933+0000 osd.1 (osd.1) 362 : cluster 2021-02-10T04:32:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: cluster 2021-02-10T04:32:28.188939+0000 osd.1 (osd 2021-02-10T04:32:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: .1) 363 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: cluster 2021-02-10T04:32:28 2021-02-10T04:32:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: .188945+0000 osd.1 (osd.1) 364 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: 2021-02-10T04:32:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: -10T04:32:28.463812+0000 osd.4 (osd.4) 325 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: 2021-02-10T04:32:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:29.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: -10T04:32:28.463825+0000 osd.4 (osd.4) 326 : cluster [WRN] 2021-02-10T04:32:29.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:29.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: cluster 2021-02-10T04:32:29.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: 2021-02-10T04:32:28.463833+0000 osd.4 (osd.4) 327 : cluster 2021-02-10T04:32:29.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:29.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: cluster 2021-02-10T04:32:28.463840+0000 osd.4 ( 2021-02-10T04:32:29.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: osd.4) 328 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:29.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: cluster 2021-02-10T04:32:29.216924+0000 2021-02-10T04:32:29.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: osd.2 (osd.2) 424 : cluster [WRN] 2021-02-10T04:32:29.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:29.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:29.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: 02-10T04:32:29.216934+0000 2021-02-10T04:32:29.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: osd.2 (osd.2) 425 : 2021-02-10T04:32:29.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:29.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:29.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: T04:32:29.216940+0000 osd.2 2021-02-10T04:32:29.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: (osd.2) 426 : cluster 2021-02-10T04:32:29.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:29.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: cluster 2021-02-10T04:32:29.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: 2021-02-10T04:32:29. 2021-02-10T04:32:29.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: 216958+0000 osd.2 (osd.2) 427 2021-02-10T04:32:29.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:29.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: cluster 2021-02-10T04:32:29.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:32:29.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: :32:29.216963+0000 osd.2 (osd.2) 2021-02-10T04:32:29.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: 428 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:29.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:29.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: 04:32:29.216969+0000 osd.2 (osd.2) 429 : cluster [WRN] 2021-02-10T04:32:29.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:29.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:29.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: -10T04:32:29.216974+0000 osd.2 (osd.2) 430 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:29.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: cluster 2021-02-10T04:32:29 2021-02-10T04:32:29.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: .216979+0000 osd.2 (osd.2) 431 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:29.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: cluster 2021-02-10T04:32 2021-02-10T04:32:29.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[29288]: :29.388854+0000 mon.a (mon.0) 1085 : cluster [DBG] osdmap e332: 8 total, 8 up, 8 in 2021-02-10T04:32:29.875 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.1 dump_historic_ops 2021-02-10T04:32:30.002 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:32:30.006 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:32:29 smithi164 conmon[44544]: debug 2021-02-10T04:32:29.702+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:30.107 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:32:30.141 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:30.204 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:32:30.392 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:32:30.394 INFO:tasks.rados.rados.0.smithi085.stdout:1191: done (15 left) 2021-02-10T04:32:30.394 INFO:tasks.rados.rados.0.smithi085.stdout:1192: read oid 22 snap -1 2021-02-10T04:32:30.395 INFO:tasks.rados.rados.0.smithi085.stdout:1192: expect (ObjNum 436 snap 116 seq_num 436) 2021-02-10T04:32:30.395 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:30.395 INFO:tasks.rados.rados.0.smithi085.stdout:1190: finishing copy_from to smithi08575397-6 2021-02-10T04:32:30.395 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 616 (ObjNum 432 snap 114 seq_num 432) dirty exists 2021-02-10T04:32:30.395 INFO:tasks.rados.rados.0.smithi085.stdout:1190: finishing copy_from racing read to smithi08575397-6 2021-02-10T04:32:30.395 INFO:tasks.rados.rados.0.smithi085.stdout:1189: done (15 left) 2021-02-10T04:32:30.395 INFO:tasks.rados.rados.0.smithi085.stdout:1190: done (14 left) 2021-02-10T04:32:30.395 INFO:tasks.rados.rados.0.smithi085.stdout:1193: delete oid 8 current snap is 119 2021-02-10T04:32:30.398 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[45443]: debug 2021-02-10T04:32:30.262+0000 7f7140d73700 -1 osd.1 332 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:32:30.398 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[49106]: debug 2021-02-10T04:32:30.189+0000 7f77a3e94700 -1 osd.2 332 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:32:30.398 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[52836]: debug 2021-02-10T04:32:30.398 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[52836]: 2021-02-10T04:32:30.352+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:30.398 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[52836]: 2021-02-10T04:32:30.421 INFO:tasks.rados.rados.0.smithi085.stdout:1192: expect (ObjNum 436 snap 116 seq_num 436) 2021-02-10T04:32:30.500 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:32:30.531 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:32:30.531 INFO:teuthology.orchestra.run.smithi085.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2021-02-10T04:32:30.531 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:32:30.544 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:30.658 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:30.658 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: 10T04:32:29.233084+0000 osd.1 (osd.1) 365 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:30.658 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:30.658 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: 10T04:32:29.233093+0000 osd.1 (osd.1) 366 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:30.658 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: cluster 2021-02-10T04:32:30.658 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:32:30.659 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: 04:32:29.233096+0000 2021-02-10T04:32:30.659 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: osd.1 (osd.1) 367 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:30.659 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: 2021-02-10T04:32:30.659 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:32:30.659 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: 32:29.233101+0000 osd.1 (osd.1) 368 2021-02-10T04:32:30.659 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:30.659 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:30.659 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: -10T04:32:29.233104+0000 osd.1 ( 2021-02-10T04:32:30.659 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: osd.1) 369 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:30.659 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:30.659 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: 10T04:32:29.233108+0000 osd.1 (osd.1) 370 : cluster [WRN] 2021-02-10T04:32:30.659 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:30.659 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:32:30.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: T04:32:29.460193+0000 osd.4 ( 2021-02-10T04:32:30.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: osd.4) 329 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:30.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: 2021-02-10T04:32:30.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:32:30.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: :32:29.460199+0000 osd.4 (osd. 2021-02-10T04:32:30.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: 4) 330 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:30.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:30.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: -10T04:32:29.460213+0000 osd.4 (osd 2021-02-10T04:32:30.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: .4) 331 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:30.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:30.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: -10T04:32:29.460216+0000 osd.4 ( 2021-02-10T04:32:30.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: osd.4) 332 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:30.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:30.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: 02-10T04:32: 2021-02-10T04:32:30.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: 30.057225+0000 mgr.y (mgr.14140) 2021-02-10T04:32:30.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: 483 : cluster [DBG] pgmap v648: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 172 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 796 KiB/s wr, 1 op/s 2021-02-10T04:32:30.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: cluster 2021-02-10T04:32:30.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: 2021-02-10T04:32:30.189923 2021-02-10T04:32:30.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 432 : 2021-02-10T04:32:30.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: cluster [WRN] 2021-02-10T04:32:30.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:30.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:30.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: 02-10T04:32:30.189931 2021-02-10T04:32:30.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 433 : cluster 2021-02-10T04:32:30.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:30.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:30.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: 02-10T04:32:30.189940+0000 2021-02-10T04:32:30.662 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: osd.2 (osd.2) 434 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:30.662 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: 2021-02-10T04:32:30.662 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: cluster 2021-02-10T04:32:30 2021-02-10T04:32:30.662 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: .189945+0000 osd.2 (osd.2) 435 : cluster 2021-02-10T04:32:30.662 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:30.662 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:30.662 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: 10T04:32:30.189951+0000 2021-02-10T04:32:30.662 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: osd.2 (osd.2) 2021-02-10T04:32:30.662 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: 436 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:30.662 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:30.662 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: 10T04 2021-02-10T04:32:30.662 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: :32:30.189956+0000 osd.2 2021-02-10T04:32:30.663 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: (osd.2) 437 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:30.663 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:32:30.663 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: T04:32:30.189959+0000 osd.2 (osd.2) 438 : 2021-02-10T04:32:30.663 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:30.663 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:30.663 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: 02-10T04:32:30.189962+0000 2021-02-10T04:32:30.663 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: osd.2 (osd.2) 439 : cluster [WRN] 2021-02-10T04:32:30.663 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:30.663 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:30.663 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: -10T04:32:30.390593+0000 mon.a ( 2021-02-10T04:32:30.663 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[29319]: mon.0) 1086 : cluster [DBG] osdmap e333: 8 total, 8 up, 8 in 2021-02-10T04:32:30.663 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: cluster 2021-02-10T04:32:30.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: 2021-02-10T04:32 2021-02-10T04:32:30.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: :29.233084+0000 osd.1 (osd.1) 365 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:30.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:32:30.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: T04:32:29.233093+0000 osd.1 (osd. 2021-02-10T04:32:30.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: 1) 366 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:30.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:30.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: 02-10T04:32:29.233096+0000 osd.1 2021-02-10T04:32:30.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: (osd.1) 367 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:30.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: cluster 2021-02-10T04:32 2021-02-10T04:32:30.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: :29.233101+0000 osd.1 (osd.1) 368 : 2021-02-10T04:32:30.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:30.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: cluster 2021-02-10T04:32 2021-02-10T04:32:30.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: :29.233104+0000 osd.1 (osd.1 2021-02-10T04:32:30.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: ) 369 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:30.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:30.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: 10T04:32:29.233108+0000 osd.1 (osd 2021-02-10T04:32:30.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: .1) 370 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:30.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: 2021-02-10T04:32:30.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: cluster 2021-02-10T04:32:30.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: 2021-02-10T04:32:29.460193+0000 osd.4 (osd 2021-02-10T04:32:30.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: .4) 329 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:30.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:30.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: 10T04:32:29.460199+0000 osd.4 ( 2021-02-10T04:32:30.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: osd.4) 330 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:30.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:30.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: -02-10T04:32:29.460213+0000 2021-02-10T04:32:30.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: osd.4 (osd.4) 331 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:30.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: 2021-02-10T04:32:30.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: cluster 2021-02-10T04:32:29 2021-02-10T04:32:30.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: .460216+0000 osd.4 (osd.4) 332 : cluster 2021-02-10T04:32:30.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:30.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:30.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: 04:32:30.057225+0000 mgr.y (mgr 2021-02-10T04:32:30.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: .14140) 483 : cluster [DBG] 2021-02-10T04:32:30.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: pgmap v648: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 172 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 796 KiB/s wr, 1 op/s 2021-02-10T04:32:30.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: cluster 2021-02-10T04:32:30.189923+0000 osd.2 (osd. 2021-02-10T04:32:30.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: 2) 432 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:30.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:30.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: 02-10T04:32:30.189931+0000 2021-02-10T04:32:30.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: osd.2 (osd.2) 433 : cluster [WRN] 2021-02-10T04:32:30.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:30.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: cluster 2021-02-10T04:32: 2021-02-10T04:32:30.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: 30.189940+0000 osd.2 (osd.2) 434 : 2021-02-10T04:32:30.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:30.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:30.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: 04:32:30.189945+0000 osd.2 (osd 2021-02-10T04:32:30.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: .2) 435 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:30.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:30.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: 10T04:32:30.189951+0000 osd.2 ( 2021-02-10T04:32:30.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: osd.2) 436 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:30.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:30.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: -02-10T04:32:30.189956+0000 2021-02-10T04:32:30.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: osd.2 (osd.2) 437 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:30.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: 2021-02-10T04:32:30.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: cluster 2021-02-10T04:32: 2021-02-10T04:32:30.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: 30.189959+0000 osd.2 (osd. 2021-02-10T04:32:30.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: 2) 438 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:30.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: cluster 2021-02-10T04:32:30.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: 2021-02-10T04:32:30.189962+0000 2021-02-10T04:32:30.669 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: osd.2 (osd.2) 439 : cluster [WRN] 2021-02-10T04:32:30.669 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:30.669 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:30.669 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: :32:30.390593+0000 mon.a (mon.0) 2021-02-10T04:32:30.669 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[35961]: 1086 : cluster [DBG] osdmap e333: 8 total, 8 up, 8 in 2021-02-10T04:32:30.728 INFO:tasks.rados.rados.0.smithi085.stdout:1192: done (14 left) 2021-02-10T04:32:30.728 INFO:tasks.rados.rados.0.smithi085.stdout:1193: done (13 left) 2021-02-10T04:32:30.728 INFO:tasks.rados.rados.0.smithi085.stdout:1194: snap_remove snap 117 2021-02-10T04:32:30.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[33677]: debug 2021-02-10T04:32:30.483+0000 7fbec19a1700 -1 osd.4 333 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:32:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: cluster 2021-02-10T04:32:29.233084+0000 osd.1 (osd 2021-02-10T04:32:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: .1) 365 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: cluster 2021-02-10T04:32:29.233093+0000 osd.1 (osd.1) 366 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: cluster 2021-02-10T04:32:29.233096+0000 osd.1 (osd.1) 367 : cluster [WRN] 2021-02-10T04:32:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: T04:32:29.233101+0000 osd.1 (osd. 2021-02-10T04:32:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: 1) 368 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: cluster 2021-02-10T04:32:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: 2021-02-10T04:32:29 2021-02-10T04:32:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: .233104+0000 osd.1 (osd. 2021-02-10T04:32:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: 1) 369 : cluster [WRN] 2021-02-10T04:32:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: -02-10T04:32: 2021-02-10T04:32:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: 29.233108+0000 osd.1 2021-02-10T04:32:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: (osd.1) 370 : 2021-02-10T04:32:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:30.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: cluster 2021-02-10T04:32:30.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: 2021-02-10T04:32 2021-02-10T04:32:30.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: :29.460193+0000 2021-02-10T04:32:30.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: osd.4 (osd.4) 329 2021-02-10T04:32:30.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:30.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: cluster 2021-02-10T04:32:30.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: 2021-02-10T04:32 2021-02-10T04:32:30.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: :29.460199+0000 2021-02-10T04:32:30.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: osd.4 (osd.4) 330 2021-02-10T04:32:30.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:30.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: cluster 2021-02-10T04:32:30.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: 2021-02-10T04:32: 2021-02-10T04:32:30.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: 29.460213+0000 osd.4 ( 2021-02-10T04:32:30.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: osd.4) 331 : cluster 2021-02-10T04:32:30.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:30.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:30.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: -10T04:32:29.460216+0000 osd.4 2021-02-10T04:32:30.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: (osd.4) 332 : cluster 2021-02-10T04:32:30.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:30.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:30.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: -10T04:32:30.057225+0000 mgr.y (mgr 2021-02-10T04:32:30.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: .14140) 483 : cluster [DBG] pgmap v648: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 172 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 796 KiB/s wr, 1 op/s 2021-02-10T04:32:30.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: 2021-02-10T04:32:30.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: cluster 2021-02-10T04:32:30.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: 2021-02-10T04:32:30.189923+0000 2021-02-10T04:32:30.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: osd.2 (osd.2) 432 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:30.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: 2021-02-10T04:32:30.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:30.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: 04:32:30.189931+0000 osd.2 ( 2021-02-10T04:32:30.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: osd.2) 433 : cluster [WRN] 2021-02-10T04:32:30.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:30.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:30.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: -02-10T04:32:30.189940+0000 osd.2 (osd 2021-02-10T04:32:30.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: .2) 434 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:30.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:30.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: -10T04:32:30.189945+0000 osd.2 (osd.2) 435 2021-02-10T04:32:30.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:30.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: cluster 2021-02-10T04:32:30.189951+0000 2021-02-10T04:32:30.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: osd.2 (osd.2) 436 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:30.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:30.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: 04:32:30.189956+0000 osd.2 (osd.2) 437 : cluster 2021-02-10T04:32:30.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:30.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: cluster 2021-02-10T04:32:30.189959+0000 osd.2 (osd 2021-02-10T04:32:30.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: .2) 438 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:30.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: cluster 2021-02-10T04:32:30 2021-02-10T04:32:30.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: .189962+0000 osd.2 (osd.2) 439 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:30.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: 2021-02-10T04:32:30.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:30.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: -02-10T04:32:30.390593+0000 mon.a (mon.0) 1086 : cluster 2021-02-10T04:32:30.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:30 smithi164 conmon[29288]: [DBG] osdmap e333: 8 total, 8 up, 8 in 2021-02-10T04:32:30.941 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[45443]: debug 2021-02-10T04:32:30.941 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:30 smithi085 conmon[45443]: 2021-02-10T04:32:30.874+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:31.414 INFO:tasks.rados.rados.0.smithi085.stdout:1194: done (13 left) 2021-02-10T04:32:31.414 INFO:tasks.rados.rados.0.smithi085.stdout:1195: write oid 7 current snap is 119 2021-02-10T04:32:31.414 INFO:tasks.rados.rados.0.smithi085.stdout:1195: seq_num 441 ranges {611887=733283,1972912=456484,2894128=517759} 2021-02-10T04:32:31.418 INFO:tasks.rados.rados.0.smithi085.stdout:1195: writing smithi08575397-7 from 611887 to 1345170 tid 1 2021-02-10T04:32:31.426 INFO:tasks.rados.rados.0.smithi085.stdout:1195: writing smithi08575397-7 from 1972912 to 2429396 tid 2 2021-02-10T04:32:31.433 INFO:tasks.rados.rados.0.smithi085.stdout:1195: writing smithi08575397-7 from 2894128 to 3411887 tid 3 2021-02-10T04:32:31.433 INFO:tasks.rados.rados.0.smithi085.stdout:1196: copy_from oid 43 from oid 40 current snap is 119 2021-02-10T04:32:31.433 INFO:tasks.rados.rados.0.smithi085.stdout:1197: delete oid 50 current snap is 119 2021-02-10T04:32:31.435 INFO:tasks.rados.rados.0.smithi085.stdout:1197: done (15 left) 2021-02-10T04:32:31.435 INFO:tasks.rados.rados.0.smithi085.stdout:1198: write oid 38 current snap is 119 2021-02-10T04:32:31.435 INFO:tasks.rados.rados.0.smithi085.stdout:1198: seq_num 443 ranges {645501=483781,1573523=460153,2548974=763384} 2021-02-10T04:32:31.443 INFO:tasks.rados.rados.0.smithi085.stdout:1198: writing smithi08575397-38 from 645501 to 1129282 tid 1 2021-02-10T04:32:31.449 INFO:tasks.rados.rados.0.smithi085.stdout:1198: writing smithi08575397-38 from 1573523 to 2033676 tid 2 2021-02-10T04:32:31.458 INFO:tasks.rados.rados.0.smithi085.stdout:1198: writing smithi08575397-38 from 2548974 to 3312358 tid 3 2021-02-10T04:32:31.458 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:31.458 INFO:tasks.rados.rados.0.smithi085.stdout:1195: finishing write tid 1 to smithi08575397-7 2021-02-10T04:32:31.458 INFO:tasks.rados.rados.0.smithi085.stdout:1195: finishing write tid 2 to smithi08575397-7 2021-02-10T04:32:31.459 INFO:tasks.rados.rados.0.smithi085.stdout:1195: finishing write tid 3 to smithi08575397-7 2021-02-10T04:32:31.459 INFO:tasks.rados.rados.0.smithi085.stdout:1195: finishing write tid 4 to smithi08575397-7 2021-02-10T04:32:31.459 INFO:tasks.rados.rados.0.smithi085.stdout:1195: finishing write tid 5 to smithi08575397-7 2021-02-10T04:32:31.459 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 496 (ObjNum 441 snap 119 seq_num 441) dirty exists 2021-02-10T04:32:31.459 INFO:tasks.rados.rados.0.smithi085.stdout:1195: left oid 7 (ObjNum 441 snap 119 seq_num 441) 2021-02-10T04:32:31.459 INFO:tasks.rados.rados.0.smithi085.stdout:1198: finishing write tid 1 to smithi08575397-38 2021-02-10T04:32:31.459 INFO:tasks.rados.rados.0.smithi085.stdout:1198: finishing write tid 2 to smithi08575397-38 2021-02-10T04:32:31.459 INFO:tasks.rados.rados.0.smithi085.stdout:1195: done (15 left) 2021-02-10T04:32:31.459 INFO:tasks.rados.rados.0.smithi085.stdout:1199: snap_create 2021-02-10T04:32:31.462 INFO:tasks.rados.rados.0.smithi085.stdout:1198: finishing write tid 3 to smithi08575397-38 2021-02-10T04:32:31.462 INFO:tasks.rados.rados.0.smithi085.stdout:1198: finishing write tid 4 to smithi08575397-38 2021-02-10T04:32:31.462 INFO:tasks.rados.rados.0.smithi085.stdout:1198: finishing write tid 5 to smithi08575397-38 2021-02-10T04:32:31.463 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 620 (ObjNum 443 snap 119 seq_num 443) dirty exists 2021-02-10T04:32:31.463 INFO:tasks.rados.rados.0.smithi085.stdout:1198: left oid 38 (ObjNum 443 snap 119 seq_num 443) 2021-02-10T04:32:31.630 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[49106]: debug 2021-02-10T04:32:31.206+0000 7f77a3e94700 -1 osd.2 333 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:32:31.630 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[45443]: debug 2021-02-10T04:32:31.630 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[45443]: 2021-02-10T04:32:31.269+0000 7f7140d73700 -1 osd.1 333 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:32:31.630 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[45443]: 2021-02-10T04:32:31.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: cluster 2021-02-10T04:32:31.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: 2021-02-10T04:32:31.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:32:31.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: T04:32:30.262834+0000 osd.1 (osd.1) 371 : cluster [WRN] 2021-02-10T04:32:31.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:31.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: cluster 2021-02-10T04:32:31.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: 2021-02-10T04:32:30.262843 2021-02-10T04:32:31.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: +0000 osd.1 (osd.1) 372 : cluster 2021-02-10T04:32:31.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:31.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:32:31.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: T04:32:30.262861+0000 osd.1 (osd. 2021-02-10T04:32:31.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: 1) 373 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:31.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: cluster 2021-02-10T04:32:31.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: 2021-02-10T04:32:31.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: 2021 2021-02-10T04:32:31.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: - 2021-02-10T04:32:31.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: 02 2021-02-10T04:32:31.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: - 2021-02-10T04:32:31.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: 10 2021-02-10T04:32:31.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: T 2021-02-10T04:32:31.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: 04 2021-02-10T04:32:31.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: : 2021-02-10T04:32:31.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: 32 2021-02-10T04:32:31.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: : 2021-02-10T04:32:31.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: 30 2021-02-10T04:32:31.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: . 2021-02-10T04:32:31.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: 262867 2021-02-10T04:32:31.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: +0000 2021-02-10T04:32:31.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: 2021-02-10T04:32:31.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: osd.1 2021-02-10T04:32:31.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: ( 2021-02-10T04:32:31.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: osd 2021-02-10T04:32:31.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: . 2021-02-10T04:32:31.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: 1 2021-02-10T04:32:31.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: ) 2021-02-10T04:32:31.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: 374 2021-02-10T04:32:31.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: : 2021-02-10T04:32:31.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: cluster 2021-02-10T04:32:31.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: 2021-02-10T04:32:31.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: [WRN] 2021-02-10T04:32:31.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:31.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: cluster 2021-02-10T04:32:30.262874+0000 osd.1 (osd. 2021-02-10T04:32:31.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: 1) 375 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:31.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: cluster 2021-02-10T04:32 2021-02-10T04:32:31.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: :30.262880+0000 osd.1 (osd.1) 376 : 2021-02-10T04:32:31.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:31.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:31.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: 10T04:32:30.484664+0000 osd.4 2021-02-10T04:32:31.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: (osd.4) 333 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:31.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: cluster 2021-02-10T04:32:30.484672+0000 osd.4 (osd.4) 334 : cluster [WRN] 2021-02-10T04:32:31.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:31.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:31.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: -10T04:32:30.484676+0000 osd.4 (osd.4) 335 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:31.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:31.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: -02-10T04:32:30.484680+0000 osd.4 (osd.4) 336 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:31.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: cluster 2021-02-10T04:32: 2021-02-10T04:32:31.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: 31.207460+0000 osd.2 (osd.2) 440 : 2021-02-10T04:32:31.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:31.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:32:31.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: 32:31.207469+0000 osd.2 (osd.2) 2021-02-10T04:32:31.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: 441 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:31.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:32:31.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: :32:31.207477+0000 osd.2 (osd. 2021-02-10T04:32:31.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: 2) 442 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:31.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:31.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: 10T04:32:31.207483+0000 osd.2 2021-02-10T04:32:31.639 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: (osd.2) 443 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:31.639 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: 2021-02-10T04:32:31.639 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:31.639 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: 02-10T04:32:31.207489+0000 osd.2 (osd.2 2021-02-10T04:32:31.639 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: ) 444 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:31.639 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:32:31.639 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: :32:31.207495+0000 osd.2 (osd.2) 2021-02-10T04:32:31.639 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: 445 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:31.640 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:32:31.640 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: :32:31.207500+0000 osd.2 (osd.2) 446 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:31.640 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:32:31.640 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: 32:31.207505+0000 osd.2 (osd.2) 2021-02-10T04:32:31.640 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[29319]: 447 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:31.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: cluster 2021-02-10T04:32:31.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 2021-02-10T04:32:31.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 2021 2021-02-10T04:32:31.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: - 2021-02-10T04:32:31.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 02 2021-02-10T04:32:31.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: - 2021-02-10T04:32:31.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 10 2021-02-10T04:32:31.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: T 2021-02-10T04:32:31.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 04 2021-02-10T04:32:31.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: : 2021-02-10T04:32:31.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 32 2021-02-10T04:32:31.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: : 2021-02-10T04:32:31.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 30 2021-02-10T04:32:31.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: . 2021-02-10T04:32:31.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 262834 2021-02-10T04:32:31.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: +0000 2021-02-10T04:32:31.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 2021-02-10T04:32:31.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: osd.1 2021-02-10T04:32:31.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: ( 2021-02-10T04:32:31.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: osd 2021-02-10T04:32:31.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: . 2021-02-10T04:32:31.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 1 2021-02-10T04:32:31.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: ) 2021-02-10T04:32:31.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 371 2021-02-10T04:32:31.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: : 2021-02-10T04:32:31.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: cluster 2021-02-10T04:32:31.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 2021-02-10T04:32:31.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: [WRN] 2021-02-10T04:32:31.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 2021-02-10T04:32:31.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:31.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 2021-02-10T04:32:31.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: cluster 2021-02-10T04:32:31.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 2021-02-10T04:32:31.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 2021 2021-02-10T04:32:31.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: - 2021-02-10T04:32:31.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 02 2021-02-10T04:32:31.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: - 2021-02-10T04:32:31.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 10 2021-02-10T04:32:31.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: T 2021-02-10T04:32:31.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 04 2021-02-10T04:32:31.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: : 2021-02-10T04:32:31.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 32 2021-02-10T04:32:31.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: : 2021-02-10T04:32:31.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 30.262843+0000 osd.1 (osd.1) 372 2021-02-10T04:32:31.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:31.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:31.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 04:32:30.262861+0000 osd.1 (osd.1) 373 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:31.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 2021-02-10T04:32:31.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: cluster 2021-02-10T04:32:31.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 2021-02-10T04:32:30.262867+0000 osd.1 2021-02-10T04:32:31.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: (osd.1) 374 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:31.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 2021-02-10T04:32:31.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:31.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: -02-10T04:32:30.262874+0000 osd.1 ( 2021-02-10T04:32:31.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: osd.1) 375 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:31.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:31.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 02-10T04:32:30.262880+0000 2021-02-10T04:32:31.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: osd.1 (osd.1) 376 : cluster [WRN] 2021-02-10T04:32:31.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:31.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: cluster 2021-02-10T04:32:31.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 2021-02-10T04:32:30.484664+0000 osd.4 2021-02-10T04:32:31.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: (osd.4) 333 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:31.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 2021-02-10T04:32:31.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:31.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: -02-10T04:32:30.484672+0000 osd.4 (osd 2021-02-10T04:32:31.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: .4) 334 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:31.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:31.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 10T04:32:30.484676+0000 osd.4 2021-02-10T04:32:31.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: (osd.4) 335 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:31.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:31.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 02-10T04:32:30.484680+0000 osd.4 2021-02-10T04:32:31.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: (osd.4) 336 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:31.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:31.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: -10T04:32:31.207460+0000 2021-02-10T04:32:31.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: osd.2 (osd.2) 440 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:31.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 2021-02-10T04:32:31.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: cluster 2021-02-10T04:32:31.207469+0000 2021-02-10T04:32:31.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: osd.2 (osd.2) 441 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:31.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 2021-02-10T04:32:31.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:31.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 02-10T04:32:31.207477+0000 osd.2 (osd.2 2021-02-10T04:32:31.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: ) 442 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:31.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: cluster 2021-02-10T04:32:31. 2021-02-10T04:32:31.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 207483+0000 osd.2 (osd.2) 443 : cluster [WRN] 2021-02-10T04:32:31.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:31.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:31.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: -02-10T04:32:31.207489+0000 osd.2 (osd.2) 2021-02-10T04:32:31.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 444 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:31.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: cluster 2021-02-10T04:32: 2021-02-10T04:32:31.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 31.207495+0000 osd.2 (osd.2) 445 : 2021-02-10T04:32:31.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:31.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: cluster 2021-02-10T04:32:31. 2021-02-10T04:32:31.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 207500+0000 osd.2 (osd.2) 446 : cluster [WRN] 2021-02-10T04:32:31.650 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:31.650 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: cluster 2021-02-10T04:32:31.650 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 2021-02-10T04:32:31.207505+0000 osd.2 (osd. 2021-02-10T04:32:31.650 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[35961]: 2) 447 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:31.893 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[33677]: debug 2021-02-10T04:32:31.504+0000 7fbec19a1700 -1 osd.4 334 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:32:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: cluster 2021-02-10T04:32:30.262834+0000 osd.1 (osd.1) 371 : 2021-02-10T04:32:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: cluster 2021-02-10T04:32:30.262843+0000 osd.1 (osd.1) 372 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:32:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: 10T04:32:30.262861+0000 osd.1 (osd.1) 373 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: T04:32:30.262867+0000 osd.1 (osd.1) 374 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: 04:32:30.262874+0000 osd.1 (osd.1) 375 : cluster 2021-02-10T04:32:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: cluster 2021-02-10T04:32: 2021-02-10T04:32:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: 30.262880+0000 osd.1 (osd.1) 376 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:32:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: 10T04:32:30.484664+0000 osd.4 (osd.4) 333 2021-02-10T04:32:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: cluster 2021-02-10T04:32:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: 2021-02-10T04:32:30.484672+0000 osd.4 (osd.4) 334 : 2021-02-10T04:32:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: cluster 2021-02-10T04:32:31.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: 2021-02-10T04:32:30.484676+0000 osd.4 (osd.4) 335 2021-02-10T04:32:31.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:31.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:31.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: 04:32:30.484680+0000 osd.4 (osd.4) 2021-02-10T04:32:31.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: 336 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:31.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: cluster 2021-02-10T04:32:31.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: 2021-02-10T04:32:31.207460+0000 2021-02-10T04:32:31.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: osd.2 (osd.2) 440 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:31.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:31.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: 04:32:31.207469+0000 osd.2 (osd.2) 441 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:31.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: 2021-02-10T04:32:31.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:32:31.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: 10T04:32:31.207477+0000 osd.2 (osd.2) 442 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:31.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:32:31.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: 10T04:32:31.207483+0000 osd.2 (osd.2) 2021-02-10T04:32:31.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: 443 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:31.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: cluster 2021-02-10T04:32:31.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: 2021-02-10T04:32:31.207489+0000 osd.2 (osd.2) 444 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:31.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:31.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: 02-10T04:32:31. 2021-02-10T04:32:31.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: 207495+0000 osd.2 (osd.2) 445 : cluster 2021-02-10T04:32:31.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:31.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:31.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: T04:32:31.207500+0000 osd.2 2021-02-10T04:32:31.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: (osd.2) 446 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:31.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:31.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: -02-10T04:32: 2021-02-10T04:32:31.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: 31.207505+0000 osd.2 (osd.2) 447 2021-02-10T04:32:31.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:31 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:31.941 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:31 smithi085 conmon[52836]: debug 2021-02-10T04:32:31.629+0000 7fba6ffae700 -1 osd.3 pg_epoch: 334 pg[4.14( v 63'26 (0'0,63'26] local-lis/les=71/72 n=11 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,2,7] r=0 lpr=71 crt=63'26 lcod 63'25 mlcod 0'0 active+clean] scrubber pg(4.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55fec06f42c0) 2021-02-10T04:32:32.427 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[49106]: debug 2021-02-10T04:32:32.427 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[49106]: 2021-02-10T04:32:32.254+0000 7f77a3e94700 -1 osd.2 334 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:32:32.427 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[49106]: 2021-02-10T04:32:32.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: cluster 2021-02-10T04:32:31.270277+0000 2021-02-10T04:32:32.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: osd.1 (osd.1) 377 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:32.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: cluster 2021-02-10T04:32:31.270287+0000 osd.1 (osd.1) 378 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:32.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: cluster 2021-02-10T04:32: 2021-02-10T04:32:32.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: 31.270293+0000 osd.1 (osd.1) 379 : cluster 2021-02-10T04:32:32.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:32.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:32:32.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: T04:32:31.270323+0000 osd.1 (osd 2021-02-10T04:32:32.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: .1) 380 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:32.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: cluster 2021-02-10T04:32:32.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: 2021-02-10T04:32:31.270330+0000 2021-02-10T04:32:32.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: osd.1 (osd.1) 381 : cluster [WRN] 2021-02-10T04:32:32.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:32.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:32:32.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: 32:31.270336+0000 osd.1 (osd.1 2021-02-10T04:32:32.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: ) 382 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:32.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:32:32.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: T04:32:31.409216+0000 mon.a (mon 2021-02-10T04:32:32.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: .0) 1087 : cluster [DBG] osdmap e334: 8 total, 8 up, 8 in 2021-02-10T04:32:32.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: audit 2021 2021-02-10T04:32:32.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: -02-10T04:32:31.494480+0000 mon.c 2021-02-10T04:32:32.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: (mon.1) 51 : audit 2021-02-10T04:32:32.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: [INF] from='client.? 172.21.15.85:0/2161624276' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:32:32.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: audit 2021-02-10T04:32: 2021-02-10T04:32:32.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: 31.494814+0000 mon.a (mon.0) 2021-02-10T04:32:32.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: 1088 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:32:32.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:32.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: -10T04:32:31.504824+0000 osd.4 2021-02-10T04:32:32.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: (osd.4) 337 : cluster 2021-02-10T04:32:32.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:32.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:32:32.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: T04:32:31 2021-02-10T04:32:32.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: .504831+0000 osd.4 2021-02-10T04:32:32.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: (osd.4) 338 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:32.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:32.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: -02-10T04:32:31.504835+0000 osd.4 (osd.4) 339 : cluster 2021-02-10T04:32:32.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:32.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: cluster 2021-02-10T04:32: 2021-02-10T04:32:32.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: 31.504839+0000 osd.4 (osd.4) 340 : 2021-02-10T04:32:32.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:32.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:32:32.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: 32:32.057805+0000 mgr.y (mgr.14140 2021-02-10T04:32:32.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: ) 484 : cluster [DBG] pgmap v651: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 159 active+clean; 171 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 914 KiB/s rd, 1.6 MiB/s wr, 3 op/s 2021-02-10T04:32:32.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:32:32.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: T04:32:32.255260+0000 osd.2 (osd 2021-02-10T04:32:32.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: .2) 448 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:32.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:32.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: 02-10T04:32:32.255268+0000 2021-02-10T04:32:32.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: osd.2 (osd.2) 449 : cluster [WRN] 2021-02-10T04:32:32.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:32.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: cluster 2021-02-10T04:32:32.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: 2021-02-10T04:32:32.255273+0000 osd.2 ( 2021-02-10T04:32:32.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: osd.2) 450 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:32.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:32.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: 02-10T04:32:32.255278+0000 2021-02-10T04:32:32.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: osd.2 (osd.2) 451 : cluster [WRN] 2021-02-10T04:32:32.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:32.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: cluster 2021-02-10T04:32:32.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: 2021-02-10T04:32:32.255284+0000 osd.2 (osd 2021-02-10T04:32:32.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: .2) 452 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:32.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:32.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: -10T04:32:32.255289+0000 osd.2 2021-02-10T04:32:32.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: (osd.2) 453 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:32.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: cluster 2021-02-10T04:32:32.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: 2021-02-10T04:32:32.255295+0000 2021-02-10T04:32:32.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: osd.2 (osd.2) 2021-02-10T04:32:32.434 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: 454 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:32.434 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: cluster 2021-02-10T04:32:32.434 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: 2021-02-10T04:32:32.255300+0000 osd.2 (osd 2021-02-10T04:32:32.434 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[29319]: .2) 455 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:32.435 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[45443]: debug 2021-02-10T04:32:32.297+0000 7f7140d73700 -1 osd.1 334 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:32:32.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: cluster 2021-02-10T04:32:32.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 2021-02-10T04:32:32.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 2021 2021-02-10T04:32:32.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: - 2021-02-10T04:32:32.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 02 2021-02-10T04:32:32.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: - 2021-02-10T04:32:32.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 10 2021-02-10T04:32:32.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: T 2021-02-10T04:32:32.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 04:32:31.270277 2021-02-10T04:32:32.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 377 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:32.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: cluster 2021-02-10T04:32:32.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 2021-02-10T04:32: 2021-02-10T04:32:32.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 31.270287+0000 osd.1 (osd.1) 378 : 2021-02-10T04:32:32.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:32.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: cluster 2021-02-10T04:32:32.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 2021-02 2021-02-10T04:32:32.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: -10T04:32:31.270293+0000 osd.1 (osd.1 2021-02-10T04:32:32.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: ) 379 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:32.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:32.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: -10T04:32:31.270323+0000 osd.1 (osd.1) 2021-02-10T04:32:32.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 380 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:32.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:32.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 02-10T04:32:31.270330+0000 osd.1 ( 2021-02-10T04:32:32.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: osd.1) 381 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:32.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:32.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: -10T04:32:31.270336+0000 osd.1 (osd 2021-02-10T04:32:32.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: .1) 382 : cluster [WRN] 2021-02-10T04:32:32.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:32.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: cluster 2021-02-10T04:32:32.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 2021-02-10T04:32:31 2021-02-10T04:32:32.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: .409216+0000 mon.a (mon.0) 1087 : cluster 2021-02-10T04:32:32.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: [DBG] osdmap e334: 8 total, 8 up, 8 in 2021-02-10T04:32:32.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:32:32.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 02-10T04:32:31.494480+0000 mon.c (mon. 2021-02-10T04:32:32.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 1) 51 : audit [INF] from='client.? 172.21.15.85:0/2161624276' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:32:32.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: audit 2021-02-10T04:32:32.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:32:32.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 04:32:31.494814+0000 mon.a (mon.0) 1088 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:32:32.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 2021-02-10T04:32:32.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: cluster 2021-02-10T04:32:32.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:32:32.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: :32:31.504824+0000 osd.4 (osd 2021-02-10T04:32:32.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: .4) 337 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:32.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: cluster 2021-02-10T04:32:32.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:32:32.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 04:32:31.504831+0000 osd.4 (osd.4) 338 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:32.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:32.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 10T04:32:31.504835+0000 osd.4 (osd.4 2021-02-10T04:32:32.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: ) 339 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:32.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:32.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 10T04:32:31.504839+0000 osd.4 (osd 2021-02-10T04:32:32.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: .4) 340 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:32.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:32.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 10T04:32:32.057805+0000 mgr.y 2021-02-10T04:32:32.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: (mgr.14140) 484 : cluster [DBG] pgmap v651: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 159 active+clean; 171 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 914 KiB/s rd, 1.6 MiB/s wr, 3 op/s 2021-02-10T04:32:32.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:32.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 02-10T04:32:32.255260+0000 osd.2 2021-02-10T04:32:32.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: (osd.2) 448 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:32.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: cluster 2021-02-10T04:32:32.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 2021-02-10T04:32:32.255268+0000 2021-02-10T04:32:32.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: osd.2 (osd.2) 449 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:32.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 2021-02-10T04:32:32.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: cluster 2021-02-10T04:32:32.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 2021-02-10T04:32:32.255273+0000 osd.2 ( 2021-02-10T04:32:32.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: osd.2) 450 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:32.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:32.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 10T04:32:32.255278+0000 osd.2 ( 2021-02-10T04:32:32.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: osd.2) 451 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:32.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:32.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 02-10T04:32:32.255284+0000 osd.2 (osd. 2021-02-10T04:32:32.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 2) 452 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:32.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: cluster 2021-02-10T04:32:32.255289+0000 osd.2 (osd. 2021-02-10T04:32:32.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 2) 453 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:32.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: cluster 2021-02-10T04:32:32.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: 2021-02-10T04:32:32.255295+0000 osd.2 (osd 2021-02-10T04:32:32.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: .2) 454 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:32.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:32.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: :32:32.255300+0000 osd.2 (osd.2 2021-02-10T04:32:32.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:32 smithi085 conmon[35961]: ) 455 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:32.441 INFO:tasks.rados.rados.0.smithi085.stdout:1198: done (15 left) 2021-02-10T04:32:32.441 INFO:tasks.rados.rados.0.smithi085.stdout:1199: done (14 left) 2021-02-10T04:32:32.442 INFO:tasks.rados.rados.0.smithi085.stdout:1200: read oid 22 snap -1 2021-02-10T04:32:32.442 INFO:tasks.rados.rados.0.smithi085.stdout:1200: expect (ObjNum 436 snap 116 seq_num 436) 2021-02-10T04:32:32.442 INFO:tasks.rados.rados.0.smithi085.stdout:1201: read oid 37 snap 116 2021-02-10T04:32:32.442 INFO:tasks.rados.rados.0.smithi085.stdout:1201: expect (ObjNum 343 snap 88 seq_num 343) 2021-02-10T04:32:32.442 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:32.452 INFO:tasks.rados.rados.0.smithi085.stdout:1200: expect (ObjNum 436 snap 116 seq_num 436) 2021-02-10T04:32:32.733 INFO:tasks.rados.rados.0.smithi085.stdout:1196: finishing copy_from to smithi08575397-43 2021-02-10T04:32:32.733 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 590 (ObjNum 389 snap 100 seq_num 389) dirty exists 2021-02-10T04:32:32.733 INFO:tasks.rados.rados.0.smithi085.stdout:1196: finishing copy_from racing read to smithi08575397-43 2021-02-10T04:32:32.733 INFO:tasks.rados.rados.0.smithi085.stdout:1201: expect (ObjNum 343 snap 88 seq_num 343) 2021-02-10T04:32:32.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[33677]: debug 2021-02-10T04:32:32.508+0000 7fbec19a1700 -1 osd.4 335 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:32:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: cluster 2021-02-10T04:32:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: 2021-02-10T04:32:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: 2021 2021-02-10T04:32:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: -02-10T04:32:31.270277 2021-02-10T04:32:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 377 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: 02-10T04:32:31. 2021-02-10T04:32:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: 270287+0000 osd.1 (osd.1) 2021-02-10T04:32:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: 378 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: cluster 2021-02-10T04:32:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: 2021-02- 2021-02-10T04:32:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: 10T04:32:31.270293+0000 osd.1 (osd.1) 379 : cluster 2021-02-10T04:32:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: 04:32:31.270323+0000 osd.1 (osd. 2021-02-10T04:32:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: 1) 380 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: 02-10T04:32:31.270330+0000 osd.1 ( 2021-02-10T04:32:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: osd.1) 381 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:32.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: 02-10T04:32:31.270336+0000 osd.1 (osd 2021-02-10T04:32:32.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: .1) 382 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:32.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:32.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: -10T04:32:31.409216+0000 mon.a (mon 2021-02-10T04:32:32.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: .0) 1087 : cluster [DBG] osdmap e334: 8 total, 8 up, 8 in 2021-02-10T04:32:32.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:32:32.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: T04:32:31.494480+0000 mon.c (mon.1 2021-02-10T04:32:32.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: ) 51 : audit [INF] from='client.? 172.21.15.85:0/2161624276' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:32:32.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: audit 2021 2021-02-10T04:32:32.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: -02-10T04:32:31. 2021-02-10T04:32:32.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: 494814+0000 mon.a (mon.0) 2021-02-10T04:32:32.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: 1088 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:32:32.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:32.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: -02-10T04:32:31.504824+0000 2021-02-10T04:32:32.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: osd.4 (osd.4) 337 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:32.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:32.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: -02-10T04:32:31.504831+0000 osd.4 ( 2021-02-10T04:32:32.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: osd.4) 338 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:32.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:32.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: -02-10T04:32:31.504835+0000 osd.4 ( 2021-02-10T04:32:32.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: osd.4) 339 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:32.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:32.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: -02-10T04:32:31.504839+0000 2021-02-10T04:32:32.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: osd.4 (osd.4) 340 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:32.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:32.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: -10T04:32:32.057805+0000 2021-02-10T04:32:32.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: mgr.y (mgr.14140) 484 : cluster 2021-02-10T04:32:32.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: [DBG] pgmap v651: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 159 active+clean; 171 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 914 KiB/s rd, 1.6 MiB/s wr, 3 op/s 2021-02-10T04:32:32.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: cluster 2021-02-10T04:32:32.255260+0000 osd.2 (osd.2) 448 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:32.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: cluster 2021-02-10T04:32:32.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: 2021-02-10T04:32:32.255268 2021-02-10T04:32:32.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 449 : cluster [WRN] 2021-02-10T04:32:32.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:32.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: cluster 2021-02-10T04:32 2021-02-10T04:32:32.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: :32.255273+0000 osd.2 (osd.2 2021-02-10T04:32:32.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: ) 450 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:32.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:32.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: -02-10T04:32:32 2021-02-10T04:32:32.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: .255278+0000 osd.2 (osd.2 2021-02-10T04:32:32.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: ) 451 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:32.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:32.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: -02-10T04:32:32.255284+0000 2021-02-10T04:32:32.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: osd.2 (osd.2) 452 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:32.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: 2021-02-10T04:32:32.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: cluster 2021-02-10T04:32: 2021-02-10T04:32:32.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: 32.255289+0000 osd.2 (osd.2) 453 : cluster 2021-02-10T04:32:32.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:32.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: cluster 2021-02-10T04:32:32.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: 2021-02-10T04:32:32. 2021-02-10T04:32:32.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: 255295+0000 osd.2 (osd.2) 454 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:32.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: cluster 2021-02-10T04:32:32.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: 2021-02-10T04:32:32.255300 2021-02-10T04:32:32.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 455 : cluster [WRN] 2021-02-10T04:32:32.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:32 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:32.948 INFO:tasks.rados.rados.0.smithi085.stdout:1196: done (15 left) 2021-02-10T04:32:32.948 INFO:tasks.rados.rados.0.smithi085.stdout:1200: done (14 left) 2021-02-10T04:32:32.948 INFO:tasks.rados.rados.0.smithi085.stdout:1201: done (13 left) 2021-02-10T04:32:32.948 INFO:tasks.rados.rados.0.smithi085.stdout:1202: delete oid 36 current snap is 120 2021-02-10T04:32:32.949 INFO:tasks.rados.rados.0.smithi085.stdout:1202: done (13 left) 2021-02-10T04:32:32.950 INFO:tasks.rados.rados.0.smithi085.stdout:1203: read oid 30 snap -1 2021-02-10T04:32:32.950 INFO:tasks.rados.rados.0.smithi085.stdout:1203: expect deleted 2021-02-10T04:32:32.950 INFO:tasks.rados.rados.0.smithi085.stdout:1204: write oid 6 current snap is 120 2021-02-10T04:32:32.950 INFO:tasks.rados.rados.0.smithi085.stdout:1204: seq_num 445 ranges {679115=634279,2090487=463822,3003820=475295} 2021-02-10T04:32:32.957 INFO:tasks.rados.rados.0.smithi085.stdout:1204: writing smithi08575397-6 from 679115 to 1313394 tid 1 2021-02-10T04:32:32.964 INFO:tasks.rados.rados.0.smithi085.stdout:1204: writing smithi08575397-6 from 2090487 to 2554309 tid 2 2021-02-10T04:32:32.969 INFO:tasks.rados.rados.0.smithi085.stdout:1204: writing smithi08575397-6 from 3003820 to 3479115 tid 3 2021-02-10T04:32:32.970 INFO:tasks.rados.rados.0.smithi085.stdout:1205: copy_from oid 28 from oid 11 current snap is 120 2021-02-10T04:32:32.970 INFO:tasks.rados.rados.0.smithi085.stdout:1204: finishing write tid 1 to smithi08575397-6 2021-02-10T04:32:32.970 INFO:tasks.rados.rados.0.smithi085.stdout:1204: finishing write tid 2 to smithi08575397-6 2021-02-10T04:32:32.970 INFO:tasks.rados.rados.0.smithi085.stdout:1203: done (15 left) 2021-02-10T04:32:32.970 INFO:tasks.rados.rados.0.smithi085.stdout:1206: copy_from oid 46 from oid 36 current snap is 120 2021-02-10T04:32:32.970 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:32.973 INFO:tasks.rados.rados.0.smithi085.stdout:1204: finishing write tid 3 to smithi08575397-6 2021-02-10T04:32:32.974 INFO:tasks.rados.rados.0.smithi085.stdout:1204: finishing write tid 4 to smithi08575397-6 2021-02-10T04:32:32.975 INFO:tasks.rados.rados.0.smithi085.stdout:1204: finishing write tid 5 to smithi08575397-6 2021-02-10T04:32:32.975 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 624 (ObjNum 445 snap 120 seq_num 445) dirty exists 2021-02-10T04:32:32.975 INFO:tasks.rados.rados.0.smithi085.stdout:1204: left oid 6 (ObjNum 445 snap 120 seq_num 445) 2021-02-10T04:32:32.975 INFO:tasks.rados.rados.0.smithi085.stdout:1204: done (15 left) 2021-02-10T04:32:32.975 INFO:tasks.rados.rados.0.smithi085.stdout:1207: snap_create 2021-02-10T04:32:32.975 INFO:tasks.rados.rados.0.smithi085.stdout:1206: finishing copy_from to smithi08575397-46 2021-02-10T04:32:32.976 INFO:tasks.rados.rados.0.smithi085.stdout:1206: got expected ENOENT (src dne) 2021-02-10T04:32:32.976 INFO:tasks.rados.rados.0.smithi085.stdout:1206: finishing copy_from racing read to smithi08575397-46 2021-02-10T04:32:33.167 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:32:33.262 INFO:tasks.rados.rados.0.smithi085.stdout:1205: finishing copy_from to smithi08575397-28 2021-02-10T04:32:33.263 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 600 (ObjNum 422 snap 109 seq_num 422) dirty exists 2021-02-10T04:32:33.263 INFO:tasks.rados.rados.0.smithi085.stdout:1205: finishing copy_from racing read to smithi08575397-28 2021-02-10T04:32:33.440 INFO:tasks.rados.rados.0.smithi085.stdout:1205: done (15 left) 2021-02-10T04:32:33.440 INFO:tasks.rados.rados.0.smithi085.stdout:1206: done (14 left) 2021-02-10T04:32:33.440 INFO:tasks.rados.rados.0.smithi085.stdout:1207: done (13 left) 2021-02-10T04:32:33.441 INFO:tasks.rados.rados.0.smithi085.stdout:1208: write (excl) oid 21 current snap is 121 2021-02-10T04:32:33.441 INFO:tasks.rados.rados.0.smithi085.stdout:1208: seq_num 446 ranges {695922=709528,2032616=607480,3198537=297385} 2021-02-10T04:32:33.448 INFO:tasks.rados.rados.0.smithi085.stdout:1208: writing smithi08575397-21 from 695922 to 1405450 tid 1 2021-02-10T04:32:33.456 INFO:tasks.rados.rados.0.smithi085.stdout:1208: writing smithi08575397-21 from 2032616 to 2640096 tid 2 2021-02-10T04:32:33.462 INFO:tasks.rados.rados.0.smithi085.stdout:1208: writing smithi08575397-21 from 3198537 to 3495922 tid 3 2021-02-10T04:32:33.462 INFO:tasks.rados.rados.0.smithi085.stdout:1209: write (excl) oid 40 current snap is 121 2021-02-10T04:32:33.462 INFO:tasks.rados.rados.0.smithi085.stdout:1208: finishing write tid 1 to smithi08575397-21 2021-02-10T04:32:33.462 INFO:tasks.rados.rados.0.smithi085.stdout:1208: finishing write tid 2 to smithi08575397-21 2021-02-10T04:32:33.462 INFO:tasks.rados.rados.0.smithi085.stdout:1209: seq_num 447 ranges {712729=784777,2091098=467491,3342313=170416} 2021-02-10T04:32:33.471 INFO:tasks.rados.rados.0.smithi085.stdout:1209: writing smithi08575397-40 from 712729 to 1497506 tid 1 2021-02-10T04:32:33.478 INFO:tasks.rados.rados.0.smithi085.stdout:1209: writing smithi08575397-40 from 2091098 to 2558589 tid 2 2021-02-10T04:32:33.484 INFO:tasks.rados.rados.0.smithi085.stdout:1209: writing smithi08575397-40 from 3342313 to 3512729 tid 3 2021-02-10T04:32:33.484 INFO:tasks.rados.rados.0.smithi085.stdout:1210: write oid 26 current snap is 121 2021-02-10T04:32:33.484 INFO:tasks.rados.rados.0.smithi085.stdout:1208: finishing write tid 3 to smithi08575397-21 2021-02-10T04:32:33.484 INFO:tasks.rados.rados.0.smithi085.stdout:1210: seq_num 448 ranges {729536=460026,1749580=611149,2853383=452450} 2021-02-10T04:32:33.491 INFO:tasks.rados.rados.0.smithi085.stdout:1210: writing smithi08575397-26 from 729536 to 1189562 tid 1 2021-02-10T04:32:33.498 INFO:tasks.rados.rados.0.smithi085.stdout:1210: writing smithi08575397-26 from 1749580 to 2360729 tid 2 2021-02-10T04:32:33.506 INFO:tasks.rados.rados.0.smithi085.stdout:1210: writing smithi08575397-26 from 2853383 to 3305833 tid 3 2021-02-10T04:32:33.506 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:33.506 INFO:tasks.rados.rados.0.smithi085.stdout:1208: finishing write tid 4 to smithi08575397-21 2021-02-10T04:32:33.506 INFO:tasks.rados.rados.0.smithi085.stdout:1208: finishing write tid 5 to smithi08575397-21 2021-02-10T04:32:33.507 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 641 (ObjNum 446 snap 121 seq_num 446) dirty exists 2021-02-10T04:32:33.507 INFO:tasks.rados.rados.0.smithi085.stdout:1208: left oid 21 (ObjNum 446 snap 121 seq_num 446) 2021-02-10T04:32:33.507 INFO:tasks.rados.rados.0.smithi085.stdout:1209: finishing write tid 1 to smithi08575397-40 2021-02-10T04:32:33.507 INFO:tasks.rados.rados.0.smithi085.stdout:1209: finishing write tid 2 to smithi08575397-40 2021-02-10T04:32:33.507 INFO:tasks.rados.rados.0.smithi085.stdout:1209: finishing write tid 3 to smithi08575397-40 2021-02-10T04:32:33.507 INFO:tasks.rados.rados.0.smithi085.stdout:1209: finishing write tid 4 to smithi08575397-40 2021-02-10T04:32:33.507 INFO:tasks.rados.rados.0.smithi085.stdout:1208: done (15 left) 2021-02-10T04:32:33.507 INFO:tasks.rados.rados.0.smithi085.stdout:1211: read oid 1 snap -1 2021-02-10T04:32:33.507 INFO:tasks.rados.rados.0.smithi085.stdout:1211: expect (ObjNum 424 snap 109 seq_num 424) 2021-02-10T04:32:33.507 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:33.507 INFO:tasks.rados.rados.0.smithi085.stdout:1209: finishing write tid 5 to smithi08575397-40 2021-02-10T04:32:33.508 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 646 (ObjNum 447 snap 121 seq_num 447) dirty exists 2021-02-10T04:32:33.508 INFO:tasks.rados.rados.0.smithi085.stdout:1209: left oid 40 (ObjNum 447 snap 121 seq_num 447) 2021-02-10T04:32:33.508 INFO:tasks.rados.rados.0.smithi085.stdout:1210: finishing write tid 1 to smithi08575397-26 2021-02-10T04:32:33.508 INFO:tasks.rados.rados.0.smithi085.stdout:1210: finishing write tid 2 to smithi08575397-26 2021-02-10T04:32:33.508 INFO:tasks.rados.rados.0.smithi085.stdout:1209: done (15 left) 2021-02-10T04:32:33.508 INFO:tasks.rados.rados.0.smithi085.stdout:1212: snap_create 2021-02-10T04:32:33.509 INFO:tasks.rados.rados.0.smithi085.stdout:1210: finishing write tid 3 to smithi08575397-26 2021-02-10T04:32:33.509 INFO:tasks.rados.rados.0.smithi085.stdout:1210: finishing write tid 4 to smithi08575397-26 2021-02-10T04:32:33.509 INFO:tasks.rados.rados.0.smithi085.stdout:1210: finishing write tid 5 to smithi08575397-26 2021-02-10T04:32:33.509 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 109 (ObjNum 448 snap 121 seq_num 448) dirty exists 2021-02-10T04:32:33.509 INFO:tasks.rados.rados.0.smithi085.stdout:1210: left oid 26 (ObjNum 448 snap 121 seq_num 448) 2021-02-10T04:32:33.516 INFO:tasks.rados.rados.0.smithi085.stdout:1211: expect (ObjNum 424 snap 109 seq_num 424) 2021-02-10T04:32:33.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[49106]: debug 2021-02-10T04:32:33.287+0000 7f77a3e94700 -1 osd.2 335 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:32:33.692 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[45443]: debug 2021-02-10T04:32:33.337+0000 7f7140d73700 -1 osd.1 335 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:32:33.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: cluster 2021-02-10T04:32:32.249789+0000 osd.0 (osd. 2021-02-10T04:32:33.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: 0) 60 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:32:33.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: cluster 2021-02-10T04:32:32.250000 2021-02-10T04:32:33.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: +0000 osd.0 (osd.0) 61 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:32:33.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: cluster 2021-02-10T04:32 2021-02-10T04:32:33.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: :32.297922+0000 osd.1 (osd.1) 383 : 2021-02-10T04:32:33.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:33.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: cluster 2021-02-10T04:32 2021-02-10T04:32:33.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: :32.297932+0000 osd.1 (osd.1) 384 : cluster 2021-02-10T04:32:33.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:33.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:32:33.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: 32:32.297939+0000 osd.1 (osd.1) 385 2021-02-10T04:32:33.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:33.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:33.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: :32:32.297944+0000 osd.1 (osd.1 2021-02-10T04:32:33.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: ) 386 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:33.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:32:33.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: 32:32.297950+0000 osd.1 (osd.1) 387 : cluster [WRN] 2021-02-10T04:32:33.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:33.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: cluster 2021-02-10T04:32:33.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: 2021-02-10T04:32:32.297957+0000 osd.1 (osd. 2021-02-10T04:32:33.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: 1) 388 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:33.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:33.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: :32:32.428435+0000 mon.a (mon.0) 2021-02-10T04:32:33.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: 1089 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:32:33.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:32:33.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: T04:32:32.438656+0000 mon.a (mon.0) 1090 : 2021-02-10T04:32:33.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:32:33.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:32:33.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: T04:32:32.438728+0000 mon.a (mon.0) 1091 : cluster [DBG] osdmap e335: 8 total, 8 up, 8 in 2021-02-10T04:32:33.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: cluster 2021-02-10T04:32:32.509485+0000 2021-02-10T04:32:33.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: osd.4 (osd.4) 341 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:33.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: cluster 2021-02-10T04:32:32.509498+0000 osd.4 2021-02-10T04:32:33.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: (osd.4) 342 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:33.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: cluster 2021-02-10T04:32:32.509507+0000 2021-02-10T04:32:33.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: osd.4 (osd.4) 343 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:33.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: cluster 2021-02-10T04:32:32.509518+0000 osd.4 ( 2021-02-10T04:32:33.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: osd.4) 344 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:33.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: cluster 2021-02-10T04:32:33 2021-02-10T04:32:33.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: .094091+0000 mon.a (mon.0) 1092 : cluster [WRN] 2021-02-10T04:32:33.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: Health check update: 18 slow ops, oldest one blocked for 197 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:32:33.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:33.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: -02-10T04:32:33.287664+0000 osd.2 (osd.2) 2021-02-10T04:32:33.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: 456 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:33.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:32:33.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: T04:32:33.287673+0000 osd.2 (osd.2) 457 : cluster [WRN] 2021-02-10T04:32:33.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:33.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: cluster 2021-02-10T04:32:33.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: 2021-02-10T04:32:33.287680+0000 osd.2 (osd 2021-02-10T04:32:33.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: .2) 458 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:33.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:33.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: 02-10T04:32:33.287686+0000 2021-02-10T04:32:33.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: osd.2 (osd.2) 459 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:33.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:33.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: 10T04:32:33.287692+0000 osd.2 (osd 2021-02-10T04:32:33.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: .2) 460 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:33.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: 2021-02-10T04:32:33.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:33.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: 04:32:33.287699+0000 osd.2 (osd 2021-02-10T04:32:33.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: .2) 461 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:33.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:33.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: 10T04:32:33.287704+0000 osd.2 ( 2021-02-10T04:32:33.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: osd.2) 462 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:33.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:33.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: 10T04:32:33.287710+0000 osd.2 (osd 2021-02-10T04:32:33.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[35961]: .2) 463 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:33.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: cluster 2021-02-10T04:32:32.249789+0000 osd.0 (osd. 2021-02-10T04:32:33.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: 0) 60 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:32:33.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: cluster 2021-02-10T04:32:32.250000+0000 osd.0 (osd.0) 61 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:32:33.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: cluster 2021-02-10T04:32:32.297922+0000 2021-02-10T04:32:33.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: osd.1 (osd.1) 383 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:33.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: cluster 2021-02-10T04:32:33.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: 2021-02-10T04:32:32.297932+0000 2021-02-10T04:32:33.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: osd.1 (osd.1) 384 : cluster [WRN] 2021-02-10T04:32:33.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:33.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: cluster 2021-02-10T04:32:33.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: 2021-02-10T04:32:32.297939+0000 osd.1 (osd 2021-02-10T04:32:33.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: .1) 385 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:33.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:33.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: -10T04:32:32.297944+0000 osd.1 ( 2021-02-10T04:32:33.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: osd.1) 386 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:33.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: cluster 2021-02-10T04:32:33.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:32:33.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: 10T04:32:32.297950+0000 osd.1 ( 2021-02-10T04:32:33.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: osd.1) 387 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:33.712 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:33.712 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: -02-10T04:32:32.297957+0000 2021-02-10T04:32:33.712 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: osd.1 (osd.1) 388 : cluster [WRN] 2021-02-10T04:32:33.712 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:33.713 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: cluster 2021-02-10T04:32:32 2021-02-10T04:32:33.713 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: .428435+0000 mon.a (mon.0) 1089 : 2021-02-10T04:32:33.713 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:32:33.713 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: audit 2021-02-10T 2021-02-10T04:32:33.713 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: 04:32:32.438656+0000 mon.a (mon. 2021-02-10T04:32:33.713 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: 0) 1090 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:32:33.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:33.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: -10T04:32:32.438728+0000 2021-02-10T04:32:33.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: mon.a (mon.0) 1091 : cluster [DBG] osdmap e335: 8 total, 8 up, 8 in 2021-02-10T04:32:33.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: 2021-02-10T04:32:33.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:32:33.715 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: 32:32.509485+0000 osd.4 (osd.4 2021-02-10T04:32:33.715 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: ) 341 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:33.715 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:32:33.715 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: 04:32:32.509498+0000 osd.4 (osd. 2021-02-10T04:32:33.715 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: 4) 342 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:33.715 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:33.716 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: 10T04:32:32.509507+0000 osd.4 ( 2021-02-10T04:32:33.716 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: osd.4) 343 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:33.716 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:33.716 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: -10T04:32:32.509518+0000 osd.4 2021-02-10T04:32:33.716 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: (osd.4) 344 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:33.717 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:33.717 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: 02-10T04:32:33.094091+0000 2021-02-10T04:32:33.717 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: mon.a (mon.0) 1092 : cluster [WRN] 2021-02-10T04:32:33.717 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: Health check update: 18 slow ops, oldest one blocked for 197 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:32:33.717 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: cluster 2021-02-10T04:32:33.717 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: 2021-02-10T04:32:33.287664+0000 osd.2 ( 2021-02-10T04:32:33.718 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: osd.2) 456 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:33.718 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:33.718 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: -10T04:32:33.287673+0000 osd.2 2021-02-10T04:32:33.718 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: (osd.2) 457 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:33.718 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: 2021-02-10T04:32:33.719 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:33.719 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: -02-10T04:32:33.287680+0000 osd.2 (osd.2 2021-02-10T04:32:33.719 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: ) 458 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:33.719 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:32:33.719 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: 04:32:33.287686+0000 osd.2 (osd. 2021-02-10T04:32:33.719 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: 2) 459 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:33.720 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:32:33.720 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: T04:32:33.287692+0000 osd.2 (osd 2021-02-10T04:32:33.720 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: .2) 460 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:33.720 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:33.720 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: -10T04:32:33.287699+0000 osd.2 (osd 2021-02-10T04:32:33.721 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: .2) 461 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:33.721 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:33.721 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: 02-10T04:32:33.287704+0000 osd.2 2021-02-10T04:32:33.721 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: (osd.2) 462 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:33.721 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:33.722 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: 02-10T04:32:33.287710+0000 2021-02-10T04:32:33.722 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: osd.2 (osd.2) 463 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:33.722 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[29319]: 2021-02-10T04:32:33.723 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is unset 2021-02-10T04:32:33.893 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[33677]: debug 2021-02-10T04:32:33.522+0000 7fbec19a1700 -1 osd.4 336 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:32:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: cluster 2021-02-10T04:32:32.249789+0000 osd.0 ( 2021-02-10T04:32:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: osd.0) 60 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:32:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: cluster 2021-02-10T04:32:32.250000+0000 osd.0 (osd.0) 61 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:32:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: cluster 2021-02-10T04:32:32.297922+0000 osd.1 2021-02-10T04:32:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: (osd.1) 383 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: -10T04:32:32.297932+0000 osd.1 (osd.1 2021-02-10T04:32:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: ) 384 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:32:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: 32:32.297939+0000 osd.1 (osd.1) 385 : cluster [WRN] 2021-02-10T04:32:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: T04:32:32.297944+0000 osd.1 (osd.1) 386 : cluster [WRN] 2021-02-10T04:32:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: cluster 2021-02-10T04:32:32 2021-02-10T04:32:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: .297950+0000 osd.1 (osd.1) 387 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:33.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:32:33.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: :32:32.297957+0000 osd.1 (osd.1) 388 : cluster [WRN] 2021-02-10T04:32:33.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:33.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:33.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: T04:32:32.428435+0000 mon.a (mon.0) 1089 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:32:33.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: audit 2021-02-10T04: 2021-02-10T04:32:33.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: 32:32.438656+0000 mon.a (mon.0) 1090 : audit [INF] 2021-02-10T04:32:33.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:32:33.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: cluster 2021-02-10T04:32:33.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: 2021-02-10T04:32:32.438728+0000 mon.a (mon. 2021-02-10T04:32:33.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: 0) 1091 : cluster [DBG] osdmap e335: 8 total, 8 up, 8 in 2021-02-10T04:32:33.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: cluster 2021-02-10T04:32:32.509485+0000 osd.4 2021-02-10T04:32:33.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: (osd.4) 341 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:33.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: cluster 2021-02-10T04:32:32.509498+0000 2021-02-10T04:32:33.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: osd.4 (osd.4) 342 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:33.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: cluster 2021-02-10T04:32:32 2021-02-10T04:32:33.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: .509507+0000 osd.4 (osd.4) 343 : cluster [WRN] 2021-02-10T04:32:33.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:33.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:32:33.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: 10T04:32:32.509518+0000 osd.4 (osd.4) 344 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:33.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:32:33.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: 10T04:32:33.094091+0000 mon.a (mon.0) 1092 : cluster 2021-02-10T04:32:33.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: [WRN] Health check update: 18 slow ops, oldest one blocked for 197 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:32:33.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: cluster 2021-02-10T04:32:33. 2021-02-10T04:32:33.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: 287664+0000 osd.2 (osd.2) 456 : cluster [WRN] 2021-02-10T04:32:33.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:33.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: cluster 2021-02-10T04:32: 2021-02-10T04:32:33.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: 33.287673+0000 osd.2 (osd.2) 457 : cluster 2021-02-10T04:32:33.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:33.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: cluster 2021-02-10T04:32:33.287680 2021-02-10T04:32:33.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 458 : cluster [WRN] 2021-02-10T04:32:33.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:33.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: cluster 2021-02-10T04:32:33.287686+0000 osd.2 (osd.2) 459 : 2021-02-10T04:32:33.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:33.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:33.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: -10T04:32:33.287692+0000 osd.2 (osd.2) 460 : cluster 2021-02-10T04:32:33.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:33.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: cluster 2021-02-10T04:32:33.287699+0000 osd.2 (osd.2) 461 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:33.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: cluster 2021-02-10T04:32:33. 2021-02-10T04:32:33.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: 287704+0000 osd.2 (osd.2) 462 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:33.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:33 smithi164 conmon[29288]: cluster 2021-02-10T04:32:33.287710+0000 osd.2 (osd.2) 463 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:34.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[52836]: debug 2021-02-10T04:32:34.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:33 smithi085 conmon[52836]: 2021-02-10T04:32:33.709+0000 7fba6ffae700 -1 osd.3 pg_epoch: 336 pg[4.14( v 63'26 (0'0,63'26] local-lis/les=71/72 n=11 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,2,7] r=0 lpr=71 crt=63'26 lcod 63'25 mlcod 0'0 active+clean] scrubber pg(4.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55fec00458c0) 2021-02-10T04:32:34.455 INFO:tasks.rados.rados.0.smithi085.stdout:1210: done (15 left) 2021-02-10T04:32:34.455 INFO:tasks.rados.rados.0.smithi085.stdout:1211: done (14 left) 2021-02-10T04:32:34.455 INFO:tasks.rados.rados.0.smithi085.stdout:1212: done (13 left) 2021-02-10T04:32:34.455 INFO:tasks.rados.rados.0.smithi085.stdout:1213: write oid 28 current snap is 122 2021-02-10T04:32:34.455 INFO:tasks.rados.rados.0.smithi085.stdout:1213: seq_num 449 ranges {746343=651628,1924415=754807,3397159=149184} 2021-02-10T04:32:34.462 INFO:tasks.rados.rados.0.smithi085.stdout:1213: writing smithi08575397-28 from 746343 to 1397971 tid 1 2021-02-10T04:32:34.472 INFO:tasks.rados.rados.0.smithi085.stdout:1213: writing smithi08575397-28 from 1924415 to 2679222 tid 2 2021-02-10T04:32:34.477 INFO:tasks.rados.rados.0.smithi085.stdout:1213: writing smithi08575397-28 from 3397159 to 3546343 tid 3 2021-02-10T04:32:34.477 INFO:tasks.rados.rados.0.smithi085.stdout:1214: read oid 5 snap -1 2021-02-10T04:32:34.478 INFO:tasks.rados.rados.0.smithi085.stdout:1214: expect deleted 2021-02-10T04:32:34.478 INFO:tasks.rados.rados.0.smithi085.stdout:1215: copy_from oid 30 from oid 36 current snap is 122 2021-02-10T04:32:34.478 INFO:tasks.rados.rados.0.smithi085.stdout:1213: finishing write tid 1 to smithi08575397-28 2021-02-10T04:32:34.478 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:34.478 INFO:tasks.rados.rados.0.smithi085.stdout:1213: finishing write tid 2 to smithi08575397-28 2021-02-10T04:32:34.479 INFO:tasks.rados.rados.0.smithi085.stdout:1214: done (15 left) 2021-02-10T04:32:34.479 INFO:tasks.rados.rados.0.smithi085.stdout:1216: copy_from oid 2 from oid 11 current snap is 122 2021-02-10T04:32:34.479 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:34.480 INFO:tasks.rados.rados.0.smithi085.stdout:1215: finishing copy_from to smithi08575397-30 2021-02-10T04:32:34.480 INFO:tasks.rados.rados.0.smithi085.stdout:1215: got expected ENOENT (src dne) 2021-02-10T04:32:34.480 INFO:tasks.rados.rados.0.smithi085.stdout:1215: finishing copy_from racing read to smithi08575397-30 2021-02-10T04:32:34.480 INFO:tasks.rados.rados.0.smithi085.stdout:1215: done (15 left) 2021-02-10T04:32:34.480 INFO:tasks.rados.rados.0.smithi085.stdout:1217: copy_from oid 8 from oid 36 current snap is 122 2021-02-10T04:32:34.480 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:34.480 INFO:tasks.rados.rados.0.smithi085.stdout:1213: finishing write tid 3 to smithi08575397-28 2021-02-10T04:32:34.480 INFO:tasks.rados.rados.0.smithi085.stdout:1213: finishing write tid 4 to smithi08575397-28 2021-02-10T04:32:34.481 INFO:tasks.rados.rados.0.smithi085.stdout:1213: finishing write tid 5 to smithi08575397-28 2021-02-10T04:32:34.481 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 604 (ObjNum 449 snap 122 seq_num 449) dirty exists 2021-02-10T04:32:34.481 INFO:tasks.rados.rados.0.smithi085.stdout:1213: left oid 28 (ObjNum 449 snap 122 seq_num 449) 2021-02-10T04:32:34.481 INFO:tasks.rados.rados.0.smithi085.stdout:1213: done (15 left) 2021-02-10T04:32:34.481 INFO:tasks.rados.rados.0.smithi085.stdout:1218: snap_create 2021-02-10T04:32:34.481 INFO:tasks.rados.rados.0.smithi085.stdout:1217: finishing copy_from to smithi08575397-8 2021-02-10T04:32:34.482 INFO:tasks.rados.rados.0.smithi085.stdout:1217: got expected ENOENT (src dne) 2021-02-10T04:32:34.482 INFO:tasks.rados.rados.0.smithi085.stdout:1217: finishing copy_from racing read to smithi08575397-8 2021-02-10T04:32:34.495 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:32:34.525 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:34.692 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[45443]: debug 2021-02-10T04:32:34.361+0000 7f7140d73700 -1 osd.1 336 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:32:34.692 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[49106]: debug 2021-02-10T04:32:34.311+0000 7f77a3e94700 -1 osd.2 336 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:32:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: cluster 2021-02-10T04:32:33.337842+0000 osd.1 ( 2021-02-10T04:32:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: osd.1) 389 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: cluster 2021-02-10T04:32:33.337852+0000 osd.1 (osd.1) 390 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: 2021-02-10T04:32:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: cluster 2021-02-10T04:32:33.337858+0000 osd.1 (osd.1) 391 : cluster [WRN] 2021-02-10T04:32:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: cluster 2021-02-10T04:32:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: 2021-02-10T04:32:33.337865+0000 osd.1 (osd.1) 392 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: cluster 2021-02-10T04:32:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: 2021-02-10T04:32:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: 2021 2021-02-10T04:32:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: - 2021-02-10T04:32:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: 02 2021-02-10T04:32:34.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: - 2021-02-10T04:32:34.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: 10 2021-02-10T04:32:34.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: T 2021-02-10T04:32:34.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: 04 2021-02-10T04:32:34.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: : 2021-02-10T04:32:34.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: 32 2021-02-10T04:32:34.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: : 2021-02-10T04:32:34.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: 33 2021-02-10T04:32:34.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: . 2021-02-10T04:32:34.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: 337874 2021-02-10T04:32:34.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: +0000 osd.1 (osd.1) 393 : cluster 2021-02-10T04:32:34.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:34.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: cluster 2021-02-10T04:32:33 2021-02-10T04:32:34.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: .337882+0000 osd.1 (osd.1) 394 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:34.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:32:34.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: 32:33.440245+0000 mon.a (mon.0) 1093 2021-02-10T04:32:34.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: : cluster [DBG] osdmap e336: 8 total, 8 up, 8 in 2021-02-10T04:32:34.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:32:34.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: 32:33.523196+0000 osd.4 (osd.4) 345 : 2021-02-10T04:32:34.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:34.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:32:34.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: 04:32:33.523207+0000 osd.4 (osd.4 2021-02-10T04:32:34.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: ) 346 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:34.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:32:34.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: T04:32:33.523214+0000 osd.4 (osd 2021-02-10T04:32:34.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: .4) 347 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:34.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:34.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: -02-10T04:32:33.523221+0000 2021-02-10T04:32:34.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: osd.4 (osd.4) 348 : cluster [WRN] 2021-02-10T04:32:34.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:34.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: cluster 2021-02-10T04:32:34.058258 2021-02-10T04:32:34.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 485 : cluster [DBG] 2021-02-10T04:32:34.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: pgmap v654: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 160 active+clean; 166 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 915 KiB/s rd, 796 KiB/s wr, 3 op/s 2021-02-10T04:32:34.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: cluster 2021-02-10T04:32:34.311640 2021-02-10T04:32:34.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 464 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:34.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: 2021-02-10T04:32:34.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: cluster 2021-02-10T04:32:34.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: 2021-02-10T04:32:34.311649+0000 osd.2 (osd 2021-02-10T04:32:34.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: .2) 465 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:34.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:34.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: 10T04:32:34.311657+0000 osd.2 2021-02-10T04:32:34.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: (osd.2) 466 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:34.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: cluster 2021-02-10T04:32:34.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: 2021-02-10T04:32:34.311661+0000 2021-02-10T04:32:34.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: osd.2 (osd.2) 467 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:34.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:34.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: 02-10T04:32:34.311664+0000 osd.2 ( 2021-02-10T04:32:34.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: osd.2) 468 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:34.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:34.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: 10T04:32:34.311671+0000 2021-02-10T04:32:34.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: osd.2 (osd.2) 469 : 2021-02-10T04:32:34.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:34.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: cluster 2021-02-10T04:32:34.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:32:34.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: 04:32:34.311676+0000 osd.2 (osd.2) 470 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:34.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:34.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: 02-10T04:32:34. 2021-02-10T04:32:34.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: 311682+0000 osd.2 (osd.2) 2021-02-10T04:32:34.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[29319]: 471 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:34.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: cluster 2021-02-10T04:32:33.337842 2021-02-10T04:32:34.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 389 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:34.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: cluster 2021-02-10T04:32:34.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: 2021-02-10T04:32:34.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: 2021 2021-02-10T04:32:34.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: - 2021-02-10T04:32:34.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: 02 2021-02-10T04:32:34.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: -10T04:32: 2021-02-10T04:32:34.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: 33.337852+0000 osd.1 (osd.1) 390 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:34.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:34.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: 02-10T04:32:33.337858+0000 osd.1 (osd 2021-02-10T04:32:34.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: .1) 391 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:34.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:34.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: 02-10T04:32:33.337865+0000 osd.1 (osd 2021-02-10T04:32:34.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: .1) 392 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:34.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:34.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: 02-10T04:32:33.337874+0000 osd.1 ( 2021-02-10T04:32:34.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: osd.1) 393 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:34.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:34.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: 10T04:32:33.337882+0000 osd.1 (osd. 2021-02-10T04:32:34.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: 1) 394 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:34.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: cluster 2021-02-10T04:32:34.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: 2021-02-10T04:32:33.440245+0000 2021-02-10T04:32:34.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: mon.a (mon.0) 1093 : cluster [DBG] 2021-02-10T04:32:34.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: osdmap e336: 8 total, 8 up, 8 in 2021-02-10T04:32:34.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: cluster 2021-02-10T04:32: 2021-02-10T04:32:34.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: 33.523196+0000 osd.4 (osd.4) 345 : 2021-02-10T04:32:34.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:34.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:32:34.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: 32:33.523207+0000 osd.4 (osd.4) 2021-02-10T04:32:34.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: 346 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:34.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:34.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: 04:32:33.523214+0000 osd.4 (osd. 2021-02-10T04:32:34.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: 4) 347 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:34.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:34.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: -02-10T04:32:33.523221+0000 2021-02-10T04:32:34.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: osd.4 (osd.4) 348 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:34.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: 2021-02-10T04:32:34.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: cluster 2021-02-10T04:32:34 2021-02-10T04:32:34.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: .058258+0000 mgr.y (mgr.14140) 485 : cluster 2021-02-10T04:32:34.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: [DBG] pgmap v654: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 160 active+clean; 166 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 915 KiB/s rd, 796 KiB/s wr, 3 op/s 2021-02-10T04:32:34.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:34.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: :32:34.311640+0000 osd.2 (osd.2 2021-02-10T04:32:34.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: ) 464 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:34.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: cluster 2021-02-10T04:32:34.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:32:34.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: :32:34.311649+0000 osd.2 (osd.2 2021-02-10T04:32:34.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: ) 465 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:34.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:34.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: -02-10T04:32:34.311657+0000 2021-02-10T04:32:34.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: osd.2 (osd.2) 466 : cluster [WRN] 2021-02-10T04:32:34.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:34.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:32:34.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: 32:34.311661+0000 osd.2 (osd.2) 2021-02-10T04:32:34.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: 467 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:34.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:34.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: 02-10T04:32:34.311664+0000 2021-02-10T04:32:34.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: osd.2 (osd.2) 468 : cluster [WRN] 2021-02-10T04:32:34.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:34.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: cluster 2021-02-10T04:32:34.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: 2021-02-10T04:32:34.311671+0000 osd.2 ( 2021-02-10T04:32:34.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: osd.2) 469 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:34.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: cluster 2021-02-10T04:32:34.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: 2021-02-10T04:32:34.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: 2021 2021-02-10T04:32:34.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: - 2021-02-10T04:32:34.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: 02 2021-02-10T04:32:34.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: - 2021-02-10T04:32:34.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: 10 2021-02-10T04:32:34.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: T04:32:34 2021-02-10T04:32:34.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: .311676+0000 osd.2 (osd.2) 470 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:34.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:34.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: 02-10T04:32:34.311682+0000 osd.2 (osd 2021-02-10T04:32:34.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:34 smithi085 conmon[35961]: .2) 471 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: cluster 2021-02-10T04:32:33.337842+0000 osd.1 (osd.1) 2021-02-10T04:32:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: 389 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: cluster 2021-02-10T04:32:33.337852+0000 osd.1 (osd.1) 390 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: 04:32:33.337858+0000 osd.1 (osd.1) 391 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: cluster 2021-02-10T04:32:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: 2021-02-10T04:32:33.337865+0000 osd.1 (osd. 2021-02-10T04:32:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: 1) 392 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: T04:32:33.337874+0000 osd.1 (osd. 2021-02-10T04:32:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: 1) 393 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: cluster 2021-02-10T04:32 2021-02-10T04:32:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: :33.337882+0000 osd.1 (osd.1) 394 : 2021-02-10T04:32:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:32:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: 32:33.440245+0000 mon.a (mon.0) 2021-02-10T04:32:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: 1093 : cluster [DBG] osdmap e336: 8 total, 8 up, 8 in 2021-02-10T04:32:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:34.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: 04:32:33.523196+0000 osd.4 (osd.4 2021-02-10T04:32:34.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: ) 345 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:34.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:34.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: T04:32:33.523207+0000 osd.4 (osd 2021-02-10T04:32:34.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: .4) 346 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:34.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:34.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: -10T04:32:33.523214+0000 osd.4 ( 2021-02-10T04:32:34.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: osd.4) 347 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:34.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:34.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: -02-10T04:32:33.523221+0000 2021-02-10T04:32:34.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: osd.4 (osd.4) 348 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:34.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: 2021-02-10T04:32:34.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: cluster 2021-02-10T04:32:34.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: 2021-02-10T04:32:34.058258+0000 mgr.y (mgr 2021-02-10T04:32:34.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: .14140) 485 : cluster [DBG] pgmap v654: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 160 active+clean; 166 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 915 KiB/s rd, 796 KiB/s wr, 3 op/s 2021-02-10T04:32:34.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:34.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: -10T04:32:34.311640+0000 osd.2 ( 2021-02-10T04:32:34.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: osd.2) 464 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:34.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: cluster 2021-02-10T04:32:34. 2021-02-10T04:32:34.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: 311649+0000 osd.2 (osd.2) 465 : cluster 2021-02-10T04:32:34.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:34.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: cluster 2021-02-10T04:32:34 2021-02-10T04:32:34.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: .311657+0000 osd.2 (osd.2) 466 : 2021-02-10T04:32:34.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:34.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: cluster 2021-02-10T04:32 2021-02-10T04:32:34.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: :34.311661+0000 osd.2 (osd.2) 2021-02-10T04:32:34.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: 467 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:34.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:32:34.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: :32:34.311664+0000 osd.2 (osd.2 2021-02-10T04:32:34.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: ) 468 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:34.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:34.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: T04:32:34.311671+0000 osd.2 (osd 2021-02-10T04:32:34.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: .2) 469 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:34.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:34.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: -10T04:32:34.311676+0000 osd.2 2021-02-10T04:32:34.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: (osd.2) 470 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:34.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:34.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: -02-10T04:32:34.311682+0000 2021-02-10T04:32:34.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: osd.2 (osd.2) 471 : cluster [WRN] 2021-02-10T04:32:34.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:34.899 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[44544]: debug 2021-02-10T04:32:34.596+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:34.899 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:34 smithi164 conmon[33677]: debug 2021-02-10T04:32:34.564+0000 7fbec19a1700 -1 osd.4 337 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:32:34.906 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:32:35.010 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:32:35.038 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:35.383 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:32:35.386 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[44544]: debug 2021-02-10T04:32:35.105+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:35.457 INFO:tasks.rados.rados.0.smithi085.stdout:1217: done (15 left) 2021-02-10T04:32:35.457 INFO:tasks.rados.rados.0.smithi085.stdout:1218: done (14 left) 2021-02-10T04:32:35.457 INFO:tasks.rados.rados.0.smithi085.stdout:1219: snap_remove snap 116 2021-02-10T04:32:35.489 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:32:35.525 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:35.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[33677]: debug 2021-02-10T04:32:35.562+0000 7fbec19a1700 -1 osd.4 338 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:32:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: cluster 2021-02-10T04:32:34.361854+0000 osd.1 (osd.1) 2021-02-10T04:32:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: 395 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: cluster 2021-02-10T04:32:34.361864+0000 osd.1 (osd.1) 396 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: cluster 2021-02-10T04:32:34.361874+0000 osd.1 (osd.1) 2021-02-10T04:32:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: 397 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: 02-10T04:32:34.361881+0000 osd.1 2021-02-10T04:32:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: (osd.1) 398 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:32:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: 10T04:32:34.361886+0000 osd.1 (osd. 2021-02-10T04:32:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: 1) 399 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: -02-10T04:32:34.361892+0000 osd.1 (osd.1) 400 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: 02-10T04:32:34.450456+0000 mon.a (mon 2021-02-10T04:32:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: .0) 1094 : cluster [DBG] osdmap e337: 8 total, 8 up, 8 in 2021-02-10T04:32:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: -02-10T04:32:34.565516+0000 osd.4 (osd 2021-02-10T04:32:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: .4) 349 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:32:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: 10T04:32:34.565527+0000 osd.4 (osd. 2021-02-10T04:32:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: 4) 350 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: -02-10T04:32:34.565534+0000 osd.4 (osd.4) 351 2021-02-10T04:32:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:32:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: 32:34.565543+0000 osd.4 (osd.4) 352 : cluster 2021-02-10T04:32:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: cluster 2021-02-10T04:32:35.308282+0000 osd.2 (osd.2) 472 : cluster [WRN] 2021-02-10T04:32:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: cluster 2021-02-10T04:32:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: 2021-02-10T04:32:35.308293+0000 osd.2 (osd.2) 2021-02-10T04:32:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: 473 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: cluster 2021-02-10T04:32:35.308302+0000 2021-02-10T04:32:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: osd.2 (osd.2) 474 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: cluster 2021-02-10T04:32:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: 2021-02-10T04:32:35.308308+0000 2021-02-10T04:32:35.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: osd.2 (osd.2) 475 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:35.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: cluster 2021-02-10T04:32:35.308316+0000 osd.2 (osd.2) 476 2021-02-10T04:32:35.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:35.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: cluster 2021-02-10T04:32:35.308322 2021-02-10T04:32:35.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 477 : cluster [WRN] 2021-02-10T04:32:35.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:35.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:32:35.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: 32:35.308341+0000 osd.2 (osd.2) 478 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:35.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: 2021-02-10T04:32:35.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: cluster 2021-02-10T04:32:35.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: 2021-02-10T04:32:35.308366+0000 osd.2 (osd.2 2021-02-10T04:32:35.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:35 smithi164 conmon[29288]: ) 479 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:35.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[49106]: debug 2021-02-10T04:32:35.307+0000 7f77a3e94700 -1 osd.2 337 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:32:35.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: cluster 2021-02-10T04:32:35.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: 2021-02-10T04:32 2021-02-10T04:32:35.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: :34.361854+0000 osd.1 (osd.1) 395 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:35.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: cluster 2021-02-10T04:32:34.361864+0000 osd.1 (osd 2021-02-10T04:32:35.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: .1) 396 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: -02-10T04:32:34.361874+0000 osd.1 (osd.1) 397 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: cluster 2021-02-10T04:32:34.361881+0000 osd.1 2021-02-10T04:32:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: (osd.1) 398 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: cluster 2021-02-10T04:32:34.361886+0000 osd.1 (osd.1 2021-02-10T04:32:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: ) 399 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: cluster 2021-02-10T04:32:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: 2021-02-10T04:32:34.361892+0000 osd.1 (osd.1) 400 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: cluster 2021-02-10T04:32:34.450456 2021-02-10T04:32:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1094 : cluster [DBG] osdmap e337: 8 total, 8 up, 8 in 2021-02-10T04:32:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: cluster 2021-02-10T04:32:34.565516+0000 osd.4 (osd 2021-02-10T04:32:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: .4) 349 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:35.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: cluster 2021-02-10T04:32:34.565527+0000 osd.4 (osd.4) 2021-02-10T04:32:35.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: 350 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:35.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: cluster 2021-02-10T04:32:35.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: 2021-02-10T04:32:34.565534+0000 osd.4 (osd.4) 351 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:35.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: cluster 2021-02-10T04:32:34. 2021-02-10T04:32:35.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: 565543+0000 osd.4 (osd.4) 352 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:35.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: cluster 2021-02-10T04:32:35 2021-02-10T04:32:35.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: .308282+0000 osd.2 (osd.2) 472 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:35.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: cluster 2021-02-10T04:32:35 2021-02-10T04:32:35.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: .308293+0000 osd.2 (osd.2) 473 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:35.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: cluster 2021-02-10T04:32:35.308302 2021-02-10T04:32:35.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 474 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:35.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: cluster 2021-02-10T04:32:35.308308+0000 2021-02-10T04:32:35.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: osd.2 (osd.2) 475 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:35.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: cluster 2021-02-10T04:32:35.308316+0000 2021-02-10T04:32:35.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: osd.2 (osd.2) 476 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:35.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: cluster 2021-02-10T04:32:35.308322+0000 osd.2 (osd 2021-02-10T04:32:35.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: .2) 477 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:35.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:35.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: 02-10T04:32:35.308341+0000 osd.2 (osd.2) 478 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:35.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:32:35.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[29319]: :32:35.308366+0000 osd.2 (osd.2) 479 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:35.695 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[45443]: debug 2021-02-10T04:32:35.354+0000 7f7140d73700 -1 osd.1 337 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:32:35.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: cluster 2021-02-10T04:32:35.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: 2021-02-10T04:32:35.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: 2021 2021-02-10T04:32:35.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: - 2021-02-10T04:32:35.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: 02-10T04:32: 2021-02-10T04:32:35.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: 34.361854+0000 osd.1 (osd.1) 395 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:35.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: cluster 2021-02-10T04:32:35.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: 2021-02-10T04:32:35.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:32:35.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: 04:32:34.361864+0000 osd.1 (osd.1) 396 : cluster [WRN] 2021-02-10T04:32:35.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:35.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: cluster 2021-02-10T04:32:35.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: 2021-02-10T04:32:34. 2021-02-10T04:32:35.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: 361874+0000 osd.1 (osd.1) 397 : cluster 2021-02-10T04:32:35.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:35.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:35.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: -10T04:32:34.361881+0000 osd.1 2021-02-10T04:32:35.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: (osd.1) 398 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:35.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:35.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: -02-10T04:32:34.361886+0000 2021-02-10T04:32:35.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: osd.1 (osd.1) 399 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:35.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:35.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: -02-10T04:32:34.361892+0000 osd.1 (osd.1) 400 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:35.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: cluster 2021-02-10T04:32:34.450456+0000 mon.a (mon.0) 1094 : 2021-02-10T04:32:35.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: cluster [DBG] osdmap e337: 8 total, 8 up, 8 in 2021-02-10T04:32:35.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: cluster 2021-02-10T04:32:34 2021-02-10T04:32:35.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: .565516+0000 osd.4 (osd.4) 349 : 2021-02-10T04:32:35.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:35.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:35.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: 10T04:32:34.565527+0000 osd.4 (osd.4) 350 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:35.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: cluster 2021-02-10T04:32: 2021-02-10T04:32:35.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: 34.565534+0000 osd.4 (osd.4) 351 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:35.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: 2021-02-10T04:32:35.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: cluster 2021-02-10T04:32 2021-02-10T04:32:35.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: :34.565543+0000 osd.4 (osd.4) 2021-02-10T04:32:35.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: 352 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:35.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:35.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: 04:32:35.308282+0000 osd.2 (osd. 2021-02-10T04:32:35.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: 2) 472 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:35.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: cluster 2021-02-10T04:32:35. 2021-02-10T04:32:35.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: 308293+0000 osd.2 (osd.2) 473 : cluster 2021-02-10T04:32:35.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:35.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: cluster 2021-02-10T04:32:35.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: 2021-02-10T04:32:35.308302+0000 osd.2 (osd.2) 474 2021-02-10T04:32:35.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:35.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:35.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: 02-10T04:32:35.308308+0000 osd.2 (osd. 2021-02-10T04:32:35.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: 2) 475 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:35.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:35.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: -02-10T04:32:35.308316+0000 2021-02-10T04:32:35.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: osd.2 (osd.2) 476 : cluster [WRN] 2021-02-10T04:32:35.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:35.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:32:35.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: 32:35.308322+0000 osd.2 (osd.2) 477 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:35.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:35.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: 10T04:32:35.308341+0000 osd.2 (osd. 2021-02-10T04:32:35.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: 2) 478 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:35.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: cluster 2021-02-10T04:32:35.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: 2021-02-10T04:32 2021-02-10T04:32:35.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:35 smithi085 conmon[35961]: :35.308366+0000 osd.2 (osd.2) 479 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:36.463 INFO:tasks.rados.rados.0.smithi085.stdout:1219: done (14 left) 2021-02-10T04:32:36.463 INFO:tasks.rados.rados.0.smithi085.stdout:1220: read oid 13 snap -1 2021-02-10T04:32:36.463 INFO:tasks.rados.rados.0.smithi085.stdout:1220: expect (ObjNum 412 snap 107 seq_num 412) 2021-02-10T04:32:36.463 INFO:tasks.rados.rados.0.smithi085.stdout:1221: rollback oid 8 current snap is 123 2021-02-10T04:32:36.463 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 8 to 120 2021-02-10T04:32:36.463 INFO:tasks.rados.rados.0.smithi085.stdout:1216: finishing copy_from to smithi08575397-2 2021-02-10T04:32:36.463 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 600 (ObjNum 422 snap 109 seq_num 422) dirty exists 2021-02-10T04:32:36.463 INFO:tasks.rados.rados.0.smithi085.stdout:1216: finishing copy_from racing read to smithi08575397-2 2021-02-10T04:32:36.463 INFO:tasks.rados.rados.0.smithi085.stdout:1216: done (15 left) 2021-02-10T04:32:36.463 INFO:tasks.rados.rados.0.smithi085.stdout:1222: read oid 37 snap -1 2021-02-10T04:32:36.464 INFO:tasks.rados.rados.0.smithi085.stdout:1222: expect (ObjNum 343 snap 88 seq_num 343) 2021-02-10T04:32:36.464 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:36.473 INFO:tasks.rados.rados.0.smithi085.stdout:1221: finishing rollback tid 1 to smithi08575397-8 2021-02-10T04:32:36.473 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 591 (ObjNum 83264688 snap 22030 seq_num 82472784) dirty dne 2021-02-10T04:32:36.473 INFO:tasks.rados.rados.0.smithi085.stdout:1221: done (15 left) 2021-02-10T04:32:36.473 INFO:tasks.rados.rados.0.smithi085.stdout:1223: write oid 40 current snap is 123 2021-02-10T04:32:36.473 INFO:tasks.rados.rados.0.smithi085.stdout:1223: seq_num 450 ranges {763150=726877,1982897=614818,3024582=474994} 2021-02-10T04:32:36.478 INFO:tasks.rados.rados.0.smithi085.stdout:1223: writing smithi08575397-40 from 763150 to 1490027 tid 1 2021-02-10T04:32:36.486 INFO:tasks.rados.rados.0.smithi085.stdout:1223: writing smithi08575397-40 from 1982897 to 2597715 tid 2 2021-02-10T04:32:36.492 INFO:tasks.rados.rados.0.smithi085.stdout:1223: writing smithi08575397-40 from 3024582 to 3499576 tid 3 2021-02-10T04:32:36.492 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:36.493 INFO:tasks.rados.rados.0.smithi085.stdout:1222: expect (ObjNum 343 snap 88 seq_num 343) 2021-02-10T04:32:36.691 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[45443]: debug 2021-02-10T04:32:36.390+0000 7f7140d73700 -1 osd.1 338 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:32:36.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[49106]: debug 2021-02-10T04:32:36.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[49106]: 2021-02-10T04:32:36.281+0000 7f77a3e94700 -1 osd.2 338 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:32:36.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[49106]: 2021-02-10T04:32:36.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: cluster 2021-02-10T04:32:35.354868+0000 osd.1 (osd 2021-02-10T04:32:36.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: .1) 401 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:36.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: cluster 2021-02-10T04:32:35.354880+0000 osd.1 (osd.1) 402 : cluster [WRN] 2021-02-10T04:32:36.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:36.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: cluster 2021-02-10T04:32:36.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: 2021-02-10T04:32:35.354886+0000 osd.1 (osd 2021-02-10T04:32:36.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: .1) 403 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:36.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: cluster 2021-02-10T04:32:36.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: 2021-02-10T04:32:35.354892+0000 osd.1 2021-02-10T04:32:36.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: (osd.1) 404 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:36.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:36.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: 10T04:32:35.354898+0000 osd.1 (osd. 2021-02-10T04:32:36.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: 1) 405 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:36.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:32:36.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: T04:32:35.354904+0000 osd.1 (osd.1 2021-02-10T04:32:36.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: ) 406 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:36.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:32:36.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: 32:35.455512+0000 mon.a (mon.0) 2021-02-10T04:32:36.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: 1095 : cluster [DBG] osdmap e338: 8 total, 8 up, 8 in 2021-02-10T04:32:36.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: cluster 2021-02-10T04:32 2021-02-10T04:32:36.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: :35.563064+0000 osd.4 (osd.4) 353 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:36.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: cluster 2021-02-10T04:32 2021-02-10T04:32:36.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: :35.563075+0000 osd.4 (osd.4) 354 2021-02-10T04:32:36.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:36.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: cluster 2021-02-10T04:32:35 2021-02-10T04:32:36.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: .563081+0000 osd.4 (osd.4) 355 2021-02-10T04:32:36.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:36.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:32:36.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: :32:35.563087+0000 osd.4 (osd. 2021-02-10T04:32:36.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: 4) 356 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:36.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:36.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: -10T04:32:36.058836+0000 mgr.y (mgr 2021-02-10T04:32:36.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: .14140) 486 : cluster [DBG] pgmap v657: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 4 active+clean+snaptrim, 162 active+clean; 169 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 4.0 MiB/s wr, 22 op/s 2021-02-10T04:32:36.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:32:36.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: 04:32:36.282276+0000 osd.2 (osd.2) 2021-02-10T04:32:36.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: 480 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:36.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: cluster 2021-02-10T04:32: 2021-02-10T04:32:36.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: 36.282284+0000 osd.2 (osd.2) 481 : cluster 2021-02-10T04:32:36.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:36.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: cluster 2021-02-10T04:32:36.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: 2021-02-10T04:32:36.282302+0000 osd.2 (osd.2) 482 2021-02-10T04:32:36.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:36.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: cluster 2021-02-10T04:32:36.282316+0000 2021-02-10T04:32:36.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: osd.2 (osd.2) 483 : cluster [WRN] 2021-02-10T04:32:36.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:36.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: cluster 2021-02-10T04:32:36.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: 2021-02-10T04:32:36.282323+0000 2021-02-10T04:32:36.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: osd.2 (osd.2) 484 : cluster 2021-02-10T04:32:36.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:36.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:36.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: -02-10T04:32: 2021-02-10T04:32:36.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: 36.282329+0000 osd.2 (osd. 2021-02-10T04:32:36.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: 2) 485 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:36.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: cluster 2021-02-10T04:32:36.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: 2021-02-10T04:32: 2021-02-10T04:32:36.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: 36.282335+0000 osd.2 (osd.2) 486 : cluster 2021-02-10T04:32:36.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:36.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: cluster 2021-02-10T04:32:36.282340+0000 osd.2 (osd.2) 487 : cluster 2021-02-10T04:32:36.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:36.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: cluster 2021-02-10T04:32:36.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: 2021-02-10T04:32:36.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: 2021 2021-02-10T04:32:36.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: - 2021-02-10T04:32:36.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: 02 2021-02-10T04:32:36.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: - 2021-02-10T04:32:36.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: 10 2021-02-10T04:32:36.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: T 2021-02-10T04:32:36.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: 04 2021-02-10T04:32:36.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: : 2021-02-10T04:32:36.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: 32 2021-02-10T04:32:36.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: : 2021-02-10T04:32:36.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: 35 2021-02-10T04:32:36.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: . 2021-02-10T04:32:36.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: 354868 2021-02-10T04:32:36.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: +0000 2021-02-10T04:32:36.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: 2021-02-10T04:32:36.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: osd.1 2021-02-10T04:32:36.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: ( 2021-02-10T04:32:36.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: osd 2021-02-10T04:32:36.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: .1) 401 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:36.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: cluster 2021-02-10T04:32: 2021-02-10T04:32:36.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: 35.354880+0000 osd.1 (osd.1) 402 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:36.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: 2021-02-10T04:32:36.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:36.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: -02-10T04:32:35.354886+0000 osd.1 (osd.1 2021-02-10T04:32:36.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: ) 403 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:36.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: cluster 2021-02-10T04:32:35 2021-02-10T04:32:36.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: .354892+0000 osd.1 (osd.1) 404 : cluster 2021-02-10T04:32:36.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:36.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: cluster 2021-02-10T04:32:35 2021-02-10T04:32:36.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: .354898+0000 osd.1 (osd.1) 405 : cluster [WRN] 2021-02-10T04:32:36.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:36.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: cluster 2021-02-10T04:32:36.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: 2021-02-10T04:32:35.354904+0000 osd.1 (osd.1) 2021-02-10T04:32:36.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: 406 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:36.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: cluster 2021-02-10T04:32: 2021-02-10T04:32:36.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: 35.455512+0000 mon.a (mon.0) 1095 : cluster 2021-02-10T04:32:36.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: [DBG] osdmap e338: 8 total, 8 up, 8 in 2021-02-10T04:32:36.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:36.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: 04:32:35.563064+0000 osd.4 (osd.4) 353 : cluster [WRN] 2021-02-10T04:32:36.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:36.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:36.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: 02-10T04:32:35.563075+0000 osd.4 (osd.4) 354 : 2021-02-10T04:32:36.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:36.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: cluster 2021-02-10T04:32:36.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: 2021-02-10T04:32:35.563081+0000 osd.4 (osd 2021-02-10T04:32:36.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: .4) 355 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:36.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:36.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: :32:35.563087+0000 osd.4 (osd.4) 2021-02-10T04:32:36.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: 356 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:36.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:36.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: :32:36.058836+0000 mgr.y (mgr.14140 2021-02-10T04:32:36.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: ) 486 : cluster [DBG] pgmap v657: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 4 active+clean+snaptrim, 162 active+clean; 169 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 4.0 MiB/s wr, 22 op/s 2021-02-10T04:32:36.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:36.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: :32:36.282276+0000 osd.2 (osd.2 2021-02-10T04:32:36.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: ) 480 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:36.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:36.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: 04:32:36.282284+0000 osd.2 (osd. 2021-02-10T04:32:36.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: 2) 481 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:36.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:36.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: 04:32:36.282302+0000 osd.2 (osd. 2021-02-10T04:32:36.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: 2) 482 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:36.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:36.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: 04:32:36.282316+0000 osd.2 (osd. 2021-02-10T04:32:36.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: 2) 483 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:36.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:32:36.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: 32:36.282323+0000 osd.2 (osd.2) 484 2021-02-10T04:32:36.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:36.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: cluster 2021-02-10T04:32 2021-02-10T04:32:36.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: :36.282329+0000 osd.2 (osd.2) 485 : 2021-02-10T04:32:36.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:36.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: cluster 2021-02-10T04:32:36 2021-02-10T04:32:36.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: .282335+0000 osd.2 (osd.2) 486 : cluster 2021-02-10T04:32:36.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:36.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: cluster 2021-02-10T04:32:36.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: 2021-02-10T04:32:36.282340+0000 osd.2 2021-02-10T04:32:36.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[35961]: (osd.2) 487 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:36.733 INFO:tasks.rados.rados.0.smithi085.stdout:1223: finishing write tid 1 to smithi08575397-40 2021-02-10T04:32:36.928 INFO:tasks.rados.rados.0.smithi085.stdout:1223: finishing write tid 2 to smithi08575397-40 2021-02-10T04:32:36.929 INFO:tasks.rados.rados.0.smithi085.stdout:1222: done (15 left) 2021-02-10T04:32:36.929 INFO:tasks.rados.rados.0.smithi085.stdout:1224: snap_create 2021-02-10T04:32:36.929 INFO:tasks.rados.rados.0.smithi085.stdout:1223: finishing write tid 3 to smithi08575397-40 2021-02-10T04:32:36.929 INFO:tasks.rados.rados.0.smithi085.stdout:1223: finishing write tid 4 to smithi08575397-40 2021-02-10T04:32:36.929 INFO:tasks.rados.rados.0.smithi085.stdout:1223: finishing write tid 5 to smithi08575397-40 2021-02-10T04:32:36.930 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 653 (ObjNum 450 snap 123 seq_num 450) dirty exists 2021-02-10T04:32:36.930 INFO:tasks.rados.rados.0.smithi085.stdout:1223: left oid 40 (ObjNum 450 snap 123 seq_num 450) 2021-02-10T04:32:36.932 INFO:tasks.rados.rados.0.smithi085.stdout:1220: expect (ObjNum 412 snap 107 seq_num 412) 2021-02-10T04:32:36.932 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[33677]: debug 2021-02-10T04:32:36.523+0000 7fbec19a1700 -1 osd.4 339 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:32:36.933 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: cluster 2021-02-10T04:32:36.933 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:32:36.933 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: 32:35.354868+0000 osd.1 (osd.1) 401 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:36.933 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:36.933 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: 02-10T04:32:35.354880+0000 osd.1 ( 2021-02-10T04:32:36.933 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: osd.1) 402 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:36.933 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:36.933 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: -02-10T04:32:35.354886+0000 osd.1 2021-02-10T04:32:36.933 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: (osd.1) 403 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:36.933 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:36.933 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: -10T04:32:35.354892+0000 osd.1 (osd. 2021-02-10T04:32:36.934 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: 1) 404 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:36.934 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:32:36.934 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: 10T04:32:35.354898+0000 osd.1 ( 2021-02-10T04:32:36.934 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: osd.1) 405 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:36.934 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:36.934 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: -10T04:32:35.354904+0000 osd.1 ( 2021-02-10T04:32:36.934 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: osd.1) 406 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:36.934 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:36.934 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: -02-10T04:32:35.455512+0000 2021-02-10T04:32:36.934 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: mon.a (mon.0) 1095 : cluster [DBG] osdmap e338: 8 total, 8 up, 8 in 2021-02-10T04:32:36.934 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: cluster 2021-02-10T04:32:36.934 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:32:36.934 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: 04:32:35.563064+0000 osd.4 (osd.4) 353 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:36.935 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: 2021-02-10T04:32:36.935 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: cluster 2021-02-10T04:32:36.935 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: 2021-02-10T04:32:35.563075+0000 2021-02-10T04:32:36.935 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: osd.4 (osd.4) 354 : cluster [WRN] 2021-02-10T04:32:36.935 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:36.935 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: cluster 2021-02-10T04:32:36.935 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: 2021-02-10T04:32 2021-02-10T04:32:36.935 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: :35.563081+0000 osd.4 (osd.4) 355 2021-02-10T04:32:36.935 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:36.935 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:36.935 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: -10T04:32:35 2021-02-10T04:32:36.935 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: .563087+0000 osd.4 (osd.4) 356 2021-02-10T04:32:36.935 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:36.935 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: cluster 2021-02-10T04:32: 2021-02-10T04:32:36.936 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: 36.058836+0000 mgr.y (mgr.14140) 486 : cluster 2021-02-10T04:32:36.936 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: [DBG] pgmap v657: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 4 active+clean+snaptrim, 162 active+clean; 169 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 4.0 MiB/s wr, 22 op/s 2021-02-10T04:32:36.937 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: cluster 2021-02-10T04:32:36.937 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: 2021-02-10T04:32:36.282276+0000 osd.2 (osd. 2021-02-10T04:32:36.937 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: 2) 480 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:36.937 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:36.937 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: 02-10T04:32:36.282284+0000 osd.2 (osd.2) 481 : cluster 2021-02-10T04:32:36.937 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:36.937 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: cluster 2021-02-10T04:32:36.938 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: 2021-02-10T04:32:36.282302+0000 osd.2 (osd 2021-02-10T04:32:36.938 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: .2) 482 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:36.938 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:32:36.938 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: 10T04:32:36.282316+0000 osd.2 (osd 2021-02-10T04:32:36.938 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: .2) 483 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:36.938 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: cluster 2021-02-10T04:32:36.938 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: 2021-02-10T04:32:36.282323+0000 osd.2 (osd.2) 484 : cluster 2021-02-10T04:32:36.938 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:36.939 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:32:36.939 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: :32:36.282329+0000 osd.2 (osd.2) 485 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:36.939 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:32:36.939 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: 10T04:32:36.282335+0000 osd.2 (osd. 2021-02-10T04:32:36.939 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: 2) 486 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:36.939 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:36.939 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: T04:32:36.282340+0000 osd.2 (osd. 2021-02-10T04:32:36.939 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:36 smithi164 conmon[29288]: 2) 487 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:37.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:36 smithi085 conmon[52836]: debug 2021-02-10T04:32:36.764+0000 7fba73fb6700 -1 osd.3 pg_epoch: 339 pg[4.14( v 63'26 (0'0,63'26] local-lis/les=71/72 n=11 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,2,7] r=0 lpr=71 crt=63'26 lcod 63'25 mlcod 0'0 active+clean] scrubber pg(4.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55fec06b4160) 2021-02-10T04:32:37.475 INFO:tasks.rados.rados.0.smithi085.stdout:1220: done (15 left) 2021-02-10T04:32:37.475 INFO:tasks.rados.rados.0.smithi085.stdout:1223: done (14 left) 2021-02-10T04:32:37.475 INFO:tasks.rados.rados.0.smithi085.stdout:1224: done (13 left) 2021-02-10T04:32:37.475 INFO:tasks.rados.rados.0.smithi085.stdout:1225: write oid 11 current snap is 124 2021-02-10T04:32:37.475 INFO:tasks.rados.rados.0.smithi085.stdout:1225: seq_num 451 ranges {779957=402126,1925026=758476,3335652=244305} 2021-02-10T04:32:37.476 INFO:tasks.rados.rados.0.smithi085.stdout:1225: writing smithi08575397-11 from 779957 to 1182083 tid 1 2021-02-10T04:32:37.485 INFO:tasks.rados.rados.0.smithi085.stdout:1225: writing smithi08575397-11 from 1925026 to 2683502 tid 2 2021-02-10T04:32:37.492 INFO:tasks.rados.rados.0.smithi085.stdout:1225: writing smithi08575397-11 from 3335652 to 3579957 tid 3 2021-02-10T04:32:37.492 INFO:tasks.rados.rados.0.smithi085.stdout:1226: copy_from oid 6 from oid 1 current snap is 124 2021-02-10T04:32:37.492 INFO:tasks.rados.rados.0.smithi085.stdout:1225: finishing write tid 1 to smithi08575397-11 2021-02-10T04:32:37.492 INFO:tasks.rados.rados.0.smithi085.stdout:1225: finishing write tid 2 to smithi08575397-11 2021-02-10T04:32:37.492 INFO:tasks.rados.rados.0.smithi085.stdout:1227: read oid 46 snap -1 2021-02-10T04:32:37.492 INFO:tasks.rados.rados.0.smithi085.stdout:1227: expect (ObjNum 382 snap 100 seq_num 382) 2021-02-10T04:32:37.492 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:37.496 INFO:tasks.rados.rados.0.smithi085.stdout:1227: expect (ObjNum 382 snap 100 seq_num 382) 2021-02-10T04:32:37.585 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: cluster 2021-02-10T04:32:36.390537+0000 osd.1 2021-02-10T04:32:37.585 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: (osd.1) 407 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:37.585 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: cluster 2021-02-10T04:32:36.390548+0000 osd.1 (osd.1) 408 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:37.585 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: cluster 2021-02-10T04:32:36.390553+0000 2021-02-10T04:32:37.586 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: osd.1 (osd.1) 409 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:37.586 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:37.586 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: 02-10T04:32:36.390565+0000 osd.1 ( 2021-02-10T04:32:37.586 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: osd.1) 410 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:37.586 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: cluster 2021-02-10T04:32:37.586 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: 2021-02-10T04:32:36.390597+0000 2021-02-10T04:32:37.587 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: osd.1 (osd.1) 411 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:37.587 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:37.587 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: 02-10T04:32:36.390604+0000 osd.1 ( 2021-02-10T04:32:37.587 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: osd.1) 412 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:37.587 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:37.588 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: -10T04:32:36.461862+0000 mon.a 2021-02-10T04:32:37.588 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: (mon.0) 1096 : cluster [DBG] 2021-02-10T04:32:37.588 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: osdmap e339: 8 total, 8 up, 8 in 2021-02-10T04:32:37.588 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: cluster 2021-02-10T04:32:37.588 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: 2021-02-10T04:32:36 2021-02-10T04:32:37.588 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: .523724+0000 osd.4 (osd.4) 357 : cluster 2021-02-10T04:32:37.589 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:37.589 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:32:37.589 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: T04:32:36.523735+0000 osd.4 2021-02-10T04:32:37.589 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: (osd.4) 358 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:37.589 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:37.589 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: 02-10T04:32:36.523742+0000 osd.4 2021-02-10T04:32:37.590 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: (osd.4) 359 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:37.590 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:37.590 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: -02-10T04:32:36.523748+0000 2021-02-10T04:32:37.590 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: osd.4 (osd.4) 360 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:37.590 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:37.591 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: -02-10T04:32:37.269544+0000 osd.2 2021-02-10T04:32:37.591 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: (osd.2) 488 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:37.591 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:37.591 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: -02-10T04:32:37.269553+0000 2021-02-10T04:32:37.591 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: osd.2 (osd.2) 489 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:37.591 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: 2021-02-10T04:32:37.591 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: cluster 2021-02-10T04:32:37.592 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: 2021-02-10T04:32:37.269560+0000 osd.2 (osd.2 2021-02-10T04:32:37.592 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: ) 490 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:37.592 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:32:37.592 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: 32:37.269566+0000 osd.2 (osd.2) 491 : cluster 2021-02-10T04:32:37.592 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:37.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: cluster 2021-02-10T04:32:37.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: 2021-02-10T04:32:37.269577+0000 osd.2 (osd.2) 492 2021-02-10T04:32:37.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:37.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: cluster 2021-02-10T04:32:37.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: 2021-02-10T04:32:37.269583+0000 osd.2 (osd 2021-02-10T04:32:37.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: .2) 493 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:37.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: cluster 2021-02-10T04:32: 2021-02-10T04:32:37.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: 37.269588+0000 osd.2 (osd.2) 494 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:37.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: 2021-02-10T04:32:37.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:37.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: -10T04:32:37.269595+0000 osd.2 (osd.2) 495 : cluster [WRN] 2021-02-10T04:32:37.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:37.595 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: cluster 2021-02-10T04:32:37.595 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:32:37.595 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: 04:32:36.390537+0000 osd.1 (osd.1) 407 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:37.595 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:37.595 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: :32:36.390548+0000 osd.1 (osd.1 2021-02-10T04:32:37.596 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: ) 408 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:37.596 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: cluster 2021-02-10T04:32:37.596 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: 2021-02-10T04:32:36.390553+0000 osd.1 ( 2021-02-10T04:32:37.596 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: osd.1) 409 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:37.596 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:37.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: -02-10T04:32:36.390565+0000 2021-02-10T04:32:37.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: osd.1 (osd.1) 410 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:37.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: cluster 2021-02-10T04:32:37.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: 2021-02-10T04:32:36.390597+0000 osd.1 (osd.1) 411 : cluster [WRN] 2021-02-10T04:32:37.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:37.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:37.598 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: 04:32:36.390604+0000 osd.1 2021-02-10T04:32:37.598 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: (osd.1) 412 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:37.598 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: cluster 2021-02-10T04:32:37.598 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: 2021-02-10T04:32:36.461862+0000 mon.a (mon.0) 1096 : cluster 2021-02-10T04:32:37.598 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: [DBG] osdmap e339: 8 total, 8 up, 8 in 2021-02-10T04:32:37.598 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:37.599 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: -02-10T04:32: 2021-02-10T04:32:37.599 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: 36.523724+0000 osd.4 (osd.4) 357 : cluster 2021-02-10T04:32:37.599 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:37.599 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:37.599 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: :32:36.523735+0000 osd.4 (osd.4 2021-02-10T04:32:37.600 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: ) 358 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:37.600 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:37.600 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: 04:32:36.523742+0000 osd.4 (osd 2021-02-10T04:32:37.600 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: .4) 359 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:37.600 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:37.600 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: :32:36.523748+0000 osd.4 (osd 2021-02-10T04:32:37.601 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: .4) 360 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:37.601 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: 2021-02-10T04:32:37.601 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:37.601 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: -10T04:32:37.269544+0000 osd.2 2021-02-10T04:32:37.601 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: (osd.2) 488 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:37.601 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: cluster 2021-02-10T04:32:37.602 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: 2021-02-10T04:32: 2021-02-10T04:32:37.602 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: 37.269553+0000 osd.2 (osd.2) 489 : cluster 2021-02-10T04:32:37.602 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:37.603 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:37.603 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: :32:37.269560+0000 osd.2 (osd.2) 2021-02-10T04:32:37.603 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: 490 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:37.604 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:37.604 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: 04:32:37.269566+0000 osd.2 (osd.2 2021-02-10T04:32:37.604 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: ) 491 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:37.604 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:37.604 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: :32:37.269577+0000 osd.2 (osd.2) 492 2021-02-10T04:32:37.604 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:37.605 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: cluster 2021-02-10T04:32:37 2021-02-10T04:32:37.605 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: .269583+0000 osd.2 (osd.2) 493 : cluster 2021-02-10T04:32:37.605 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:37.605 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: cluster 2021-02-10T04:32:37.605 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: 2021-02-10T04:32:37.269588+0000 osd.2 (osd 2021-02-10T04:32:37.605 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: .2) 494 : cluster [WRN] 2021-02-10T04:32:37.606 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:37.606 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:37.606 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: 10T04:32:37.269595 2021-02-10T04:32:37.606 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 2021-02-10T04:32:37.606 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: 495 : cluster [WRN] 2021-02-10T04:32:37.607 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:37.607 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[45443]: debug 2021-02-10T04:32:37.607 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[45443]: 2021-02-10T04:32:37.409+0000 7f7140d73700 -1 osd.1 339 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:32:37.608 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[45443]: debug 2021-02-10T04:32:37.410+0000 7f7125cf8700 -1 osd.1 pg_epoch: 339 pg[4.4( v 63'31 (0'0,63'31] local-lis/les=55/56 n=14 ec=55/55 lis/c=55/55 les/c/f=56/56/0 sis=55) [1,2,3] r=0 lpr=55 crt=63'31 lcod 63'30 mlcod 63'30 active+clean] scrubber pg(4.4) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 3 (0x55dbc884de40) 2021-02-10T04:32:37.609 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[49106]: debug 2021-02-10T04:32:37.269+0000 7f77a3e94700 -1 osd.2 339 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:32:37.616 INFO:tasks.rados.rados.0.smithi085.stdout:1227: done (15 left) 2021-02-10T04:32:37.616 INFO:tasks.rados.rados.0.smithi085.stdout:1228: read oid 5 snap -1 2021-02-10T04:32:37.616 INFO:tasks.rados.rados.0.smithi085.stdout:1228: expect deleted 2021-02-10T04:32:37.616 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:37.617 INFO:tasks.rados.rados.0.smithi085.stdout:1225: finishing write tid 3 to smithi08575397-11 2021-02-10T04:32:37.617 INFO:tasks.rados.rados.0.smithi085.stdout:1225: finishing write tid 4 to smithi08575397-11 2021-02-10T04:32:37.617 INFO:tasks.rados.rados.0.smithi085.stdout:1225: finishing write tid 5 to smithi08575397-11 2021-02-10T04:32:37.617 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 662 (ObjNum 451 snap 124 seq_num 451) dirty exists 2021-02-10T04:32:37.617 INFO:tasks.rados.rados.0.smithi085.stdout:1225: left oid 11 (ObjNum 451 snap 124 seq_num 451) 2021-02-10T04:32:37.617 INFO:tasks.rados.rados.0.smithi085.stdout:1226: finishing copy_from to smithi08575397-6 2021-02-10T04:32:37.618 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 385 (ObjNum 424 snap 109 seq_num 424) dirty exists 2021-02-10T04:32:37.618 INFO:tasks.rados.rados.0.smithi085.stdout:1226: finishing copy_from racing read to smithi08575397-6 2021-02-10T04:32:37.618 INFO:tasks.rados.rados.0.smithi085.stdout:1225: done (15 left) 2021-02-10T04:32:37.619 INFO:tasks.rados.rados.0.smithi085.stdout:1226: done (14 left) 2021-02-10T04:32:37.619 INFO:tasks.rados.rados.0.smithi085.stdout:1229: write oid 43 current snap is 124 2021-02-10T04:32:37.619 INFO:tasks.rados.rados.0.smithi085.stdout:1229: seq_num 452 ranges {796764=477375,1983508=618487,3363075=233689} 2021-02-10T04:32:37.624 INFO:tasks.rados.rados.0.smithi085.stdout:1229: writing smithi08575397-43 from 796764 to 1274139 tid 1 2021-02-10T04:32:37.632 INFO:tasks.rados.rados.0.smithi085.stdout:1229: writing smithi08575397-43 from 1983508 to 2601995 tid 2 2021-02-10T04:32:37.639 INFO:tasks.rados.rados.0.smithi085.stdout:1229: writing smithi08575397-43 from 3363075 to 3596764 tid 3 2021-02-10T04:32:37.639 INFO:tasks.rados.rados.0.smithi085.stdout:1230: snap_remove snap 119 2021-02-10T04:32:37.887 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:32:37.893 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[33677]: debug 2021-02-10T04:32:37.484+0000 7fbec19a1700 -1 osd.4 340 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:32:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: cluster 2021-02-10T04:32:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:32:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: 32:36.390537+0000 osd.1 (osd.1) 407 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:32:37.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: :32:36.390548+0000 osd.1 (osd.1) 2021-02-10T04:32:37.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: 408 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:37.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:37.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: -10T04:32:36.390553+0000 osd.1 (osd.1) 2021-02-10T04:32:37.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: 409 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:37.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: cluster 2021-02-10T04:32:37.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: 2021-02-10T04:32:36.390565 2021-02-10T04:32:37.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 410 : cluster [WRN] 2021-02-10T04:32:37.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:37.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:37.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: 04:32:36.390597+0000 osd.1 (osd.1 2021-02-10T04:32:37.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: ) 411 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:37.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:37.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: -02-10T04:32:36.390604+0000 osd.1 2021-02-10T04:32:37.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: (osd.1) 412 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:37.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:37.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: 02-10T04:32:36.461862+0000 mon.a (mon 2021-02-10T04:32:37.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: .0) 1096 : cluster [DBG] osdmap e339: 8 total, 8 up, 8 in 2021-02-10T04:32:37.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:37.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: -02-10T04:32:36. 2021-02-10T04:32:37.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: 523724+0000 osd.4 (osd.4) 2021-02-10T04:32:37.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: 357 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:37.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:37.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: -02-10T04:32:36 2021-02-10T04:32:37.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: .523735+0000 osd.4 (osd. 2021-02-10T04:32:37.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: 4) 358 : cluster [WRN] 2021-02-10T04:32:37.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:37.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:37.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: 02-10T04:32: 2021-02-10T04:32:37.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: 36.523742+0000 osd.4 (osd. 2021-02-10T04:32:37.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: 4) 359 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:37.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: 2021-02-10T04:32:37.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: cluster 2021-02-10T04:32:37.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:32:37.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: 32:36.523748 2021-02-10T04:32:37.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2021-02-10T04:32:37.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: 360 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:37.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: cluster 2021-02-10T04:32:37.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: 2021-02- 2021-02-10T04:32:37.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: 10T04: 2021-02-10T04:32:37.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: 32:37.269544+0000 osd.2 ( 2021-02-10T04:32:37.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: osd.2) 488 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:37.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:37.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: 02-10T04:32:37.269553+0000 2021-02-10T04:32:37.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: osd.2 (osd.2) 489 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:37.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:37.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: -02-10T04:32: 2021-02-10T04:32:37.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: 37.269560+0000 osd.2 (osd. 2021-02-10T04:32:37.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: 2) 490 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:37.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: 2021-02-10T04:32:37.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: cluster 2021-02-10T04:32:37.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:32:37.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: :32:37.269566+0000 2021-02-10T04:32:37.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: osd.2 (osd.2) 491 : 2021-02-10T04:32:37.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:37.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:37.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: -02-10T04:32:37.269577+0000 osd.2 2021-02-10T04:32:37.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: (osd.2) 492 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:37.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:37.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: 02-10T04:32:37.269583+0000 osd.2 ( 2021-02-10T04:32:37.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: osd.2) 493 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:37.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:37.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: -02-10T04:32:37.269588+0000 osd.2 2021-02-10T04:32:37.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: (osd. 2021-02-10T04:32:37.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: 2) 494 : cluster [WRN] 2021-02-10T04:32:37.906 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:37.906 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: cluster 2021-02-10T04:32:37.906 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: 2021-02-10T04:32:37.269595 2021-02-10T04:32:37.906 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 495 2021-02-10T04:32:37.907 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:37.941 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[41540]: debug 2021-02-10T04:32:37.942 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[41540]: 2021-02-10T04:32:37.627+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:37.942 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:32:37 smithi085 conmon[41540]: 2021-02-10T04:32:38.394 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:32:37 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:32:38.481 INFO:tasks.rados.rados.0.smithi085.stdout:1229: finishing write tid 1 to smithi08575397-43 2021-02-10T04:32:39.172 INFO:tasks.rados.rados.0.smithi085.stdout:1229: finishing write tid 2 to smithi08575397-43 2021-02-10T04:32:39.172 INFO:tasks.rados.rados.0.smithi085.stdout:1229: finishing write tid 3 to smithi08575397-43 2021-02-10T04:32:39.172 INFO:tasks.rados.rados.0.smithi085.stdout:1229: finishing write tid 4 to smithi08575397-43 2021-02-10T04:32:39.173 INFO:tasks.rados.rados.0.smithi085.stdout:1229: finishing write tid 5 to smithi08575397-43 2021-02-10T04:32:39.173 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 594 (ObjNum 452 snap 124 seq_num 452) dirty exists 2021-02-10T04:32:39.173 INFO:tasks.rados.rados.0.smithi085.stdout:1229: left oid 43 (ObjNum 452 snap 124 seq_num 452) 2021-02-10T04:32:39.173 INFO:tasks.rados.rados.0.smithi085.stdout:1228: done (15 left) 2021-02-10T04:32:39.173 INFO:tasks.rados.rados.0.smithi085.stdout:1229: done (14 left) 2021-02-10T04:32:39.173 INFO:tasks.rados.rados.0.smithi085.stdout:1230: done (13 left) 2021-02-10T04:32:39.173 INFO:tasks.rados.rados.0.smithi085.stdout:1231: delete oid 30 current snap is 124 2021-02-10T04:32:39.175 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[33677]: debug 2021-02-10T04:32:38.523+0000 7fbec19a1700 -1 osd.4 341 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:32:39.175 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: cluster 2021-02-10T04:32:37.409807+0000 osd.1 ( 2021-02-10T04:32:39.176 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: osd.1) 413 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:39.176 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: cluster 2021-02-10T04:32:37.409815+0000 osd.1 (osd.1) 414 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:39.176 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: cluster 2021-02-10T04:32:37.409819+0000 osd.1 ( 2021-02-10T04:32:39.176 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: osd.1) 415 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:39.176 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:39.176 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: 02-10T04:32:37.409823+0000 osd.1 2021-02-10T04:32:39.176 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: (osd.1) 416 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:39.176 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:39.176 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: -10T04:32:37.409829+0000 osd.1 ( 2021-02-10T04:32:39.176 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: osd.1) 417 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:39.176 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:39.177 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: -10T04:32:37.409850+0000 osd.1 ( 2021-02-10T04:32:39.177 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: osd.1) 418 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:39.177 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:39.177 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: T04:32:37.467004+0000 mon.a (mon.0 2021-02-10T04:32:39.177 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: ) 1097 : cluster [DBG] osdmap e340: 8 total, 8 up, 8 in 2021-02-10T04:32:39.177 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:39.177 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: T04:32:37.485674+0000 osd.4 (osd. 2021-02-10T04:32:39.177 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: 4) 361 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:39.177 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:39.177 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: 04:32:37.485687+0000 osd.4 (osd.4) 2021-02-10T04:32:39.177 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: 362 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:39.177 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: cluster 2021-02-10T04:32 2021-02-10T04:32:39.178 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: :37.485694+0000 osd.4 (osd.4) 363 : 2021-02-10T04:32:39.178 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:39.178 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: cluster 2021-02-10T04:32:37. 2021-02-10T04:32:39.178 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: 485700+0000 osd.4 (osd.4) 364 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:39.178 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: 2021-02-10T04:32:39.178 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:39.178 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: -02-10T04:32:37.721737+0000 osd.7 (osd.7) 46 2021-02-10T04:32:39.178 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:32:39.178 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:32:39.178 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: 32:37.721932+0000 osd.7 (osd.7) 47 : cluster 2021-02-10T04:32:39.179 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: [DBG] purged_snaps scrub ok 2021-02-10T04:32:39.179 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: audit 2021-02-10T04:32:39.179 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: 2021-02-10T04:32:37.915097+0000 mgr.y (mgr.14140) 487 : audit 2021-02-10T04:32:39.179 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:32:39.179 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:39.179 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: T04:32:38.059467+0000 mgr.y (mgr 2021-02-10T04:32:39.179 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: .14140) 488 : cluster [DBG] pgmap v660: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 4 active+clean+snaptrim, 162 active+clean; 169 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 4.0 MiB/s wr, 22 op/s 2021-02-10T04:32:39.179 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: audit 2021-02-10T04:32:39.179 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:32:39.179 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: 04:32:38.070908+0000 mon.a (mon.0) 1098 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:32:39.179 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:32:39.179 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: 02-10T04:32:38.071267+0000 mon.a ( 2021-02-10T04:32:39.179 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: mon.0) 1099 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:32:39.179 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:39.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: -02-10T04:32:38.096135+0000 2021-02-10T04:32:39.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: mon.a (mon.0) 1100 : cluster [WRN] Health check update: 18 slow ops, oldest one blocked for 202 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:32:39.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: 2021-02-10T04:32:39.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: cluster 2021-02-10T04:32:39.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: 2021-02-10T04:32:38.257675+0000 osd.2 (osd.2) 496 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:39.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: cluster 2021-02-10T04:32 2021-02-10T04:32:39.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: :38.257681+0000 osd.2 (osd.2) 497 : 2021-02-10T04:32:39.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:39.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: cluster 2021-02-10T04:32 2021-02-10T04:32:39.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: :38.257685+0000 osd.2 (osd.2) 498 2021-02-10T04:32:39.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:39.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: cluster 2021-02-10T04:32 2021-02-10T04:32:39.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: :38.257689+0000 osd.2 (osd.2) 499 2021-02-10T04:32:39.181 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:39.181 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:32:39.181 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: 32:38.257693+0000 osd.2 (osd.2) 500 2021-02-10T04:32:39.181 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:39.181 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: cluster 2021-02-10T04:32 2021-02-10T04:32:39.181 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: :38.257696+0000 osd.2 (osd.2) 501 2021-02-10T04:32:39.181 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:39.181 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:32:39.181 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: 32:38.257699+0000 osd.2 (osd.2) 502 2021-02-10T04:32:39.181 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:39.181 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: cluster 2021-02-10T04:32 2021-02-10T04:32:39.181 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: :38.257702+0000 osd.2 (osd.2) 503 2021-02-10T04:32:39.181 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:38 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:39.182 INFO:tasks.rados.rados.0.smithi085.stdout:1231: done (13 left) 2021-02-10T04:32:39.182 INFO:tasks.rados.rados.0.smithi085.stdout:1232: rollback oid 40 current snap is 124 2021-02-10T04:32:39.182 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 40 to 121 2021-02-10T04:32:39.182 INFO:tasks.rados.rados.0.smithi085.stdout:1233: snap_create 2021-02-10T04:32:39.182 INFO:tasks.rados.rados.0.smithi085.stdout:1232: finishing rollback tid 0 to smithi08575397-40 2021-02-10T04:32:39.182 INFO:tasks.rados.rados.0.smithi085.stdout:1232: finishing rollback tid 1 to smithi08575397-40 2021-02-10T04:32:39.182 INFO:tasks.rados.rados.0.smithi085.stdout:1232: finishing rollback tid 2 to smithi08575397-40 2021-02-10T04:32:39.182 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 666 (ObjNum 447 snap 121 seq_num 447) dirty exists 2021-02-10T04:32:39.182 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[49106]: debug 2021-02-10T04:32:38.257+0000 7f77a3e94700 -1 osd.2 340 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:32:39.182 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[45443]: debug 2021-02-10T04:32:38.386+0000 7f7140d73700 -1 osd.1 340 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:32:39.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: cluster 2021-02-10T04:32:37.409807+0000 osd.1 (osd.1 2021-02-10T04:32:39.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: ) 413 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:39.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:39.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 10T04:32:37.409815+0000 osd.1 (osd.1) 414 2021-02-10T04:32:39.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:39.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:39.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 02-10T04:32:37.409819+0000 osd.1 ( 2021-02-10T04:32:39.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: osd.1) 415 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:39.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:39.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: -02-10T04:32:37.409823+0000 osd.1 ( 2021-02-10T04:32:39.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: osd.1) 416 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:39.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:39.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 02-10T04:32:37.409829+0000 osd.1 (osd.1) 2021-02-10T04:32:39.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 417 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:39.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:32:39.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 32:37.409850+0000 2021-02-10T04:32:39.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: osd.1 (osd.1) 418 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:39.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:39.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 02-10T04:32: 2021-02-10T04:32:39.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 37.467004+0000 mon.a (mon.0) 1097 : cluster 2021-02-10T04:32:39.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: [DBG] osdmap e340: 8 total, 8 up, 8 in 2021-02-10T04:32:39.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: cluster 2021-02-10T04:32:39.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 2021-02 2021-02-10T04:32:39.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: -10T04:32:37.485674+0000 osd.4 (osd.4) 361 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:39.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: cluster 2021-02-10T04:32:39.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 2021-02-10T04:32: 2021-02-10T04:32:39.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 37.485687+0000 osd.4 (osd.4) 362 2021-02-10T04:32:39.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:39.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: cluster 2021-02-10T04:32:39.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 2021-02-10T04:32 2021-02-10T04:32:39.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: :37.485694+0000 osd.4 (osd.4) 2021-02-10T04:32:39.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 363 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:39.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: cluster 2021-02-10T04:32:39.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 2021-02-10T04:32:37 2021-02-10T04:32:39.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: .485700+0000 osd.4 (osd.4) 364 : cluster 2021-02-10T04:32:39.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:39.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:32:39.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: T04:32:37.721737+0000 osd.7 (osd 2021-02-10T04:32:39.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: .7) 46 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:32:39.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: cluster 2021-02-10T04:32:39.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 2021-02-10T04:32: 2021-02-10T04:32:39.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 37.721932+0000 osd.7 (osd.7) 47 : 2021-02-10T04:32:39.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:32:39.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: audit 2021-02-10T04:32:39.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 2021-02-10T04:32: 2021-02-10T04:32:39.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 37.915097+0000 mgr.y (mgr.14140) 487 2021-02-10T04:32:39.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:32:39.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:39.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: -10T04:32:38.059467+0000 mgr.y 2021-02-10T04:32:39.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: (mgr.14140) 488 : cluster [DBG] pgmap v660: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 4 active+clean+snaptrim, 162 active+clean; 169 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 4.0 MiB/s wr, 22 op/s 2021-02-10T04:32:39.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 2021-02-10T04:32:39.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: audit 2021-02-10T04:32:39.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:32:39.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 10T04:32:38.070908+0000 mon.a ( 2021-02-10T04:32:39.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: mon.0) 1098 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:32:39.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: audit 2021-02-10T04:32:39.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 2021-02-10T04:32:38 2021-02-10T04:32:39.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: .071267+0000 mon.a (mon.0) 1099 : 2021-02-10T04:32:39.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:32:39.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: cluster 2021-02-10T04:32:39.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 2021-02-10T04:32:38.096135 2021-02-10T04:32:39.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1100 : cluster 2021-02-10T04:32:39.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: [WRN] Health check update: 18 slow ops, oldest one blocked for 202 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:32:39.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: cluster 2021-02-10T04:32:39.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 2021-02-10T04:32:38. 2021-02-10T04:32:39.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 257675+0000 osd.2 (osd.2) 496 2021-02-10T04:32:39.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:39.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: cluster 2021-02-10T04:32:39.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:32:39.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 32:38.257681+0000 osd.2 (osd.2 2021-02-10T04:32:39.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: ) 497 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:39.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:39.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: -02-10T04:32:38.257685+0000 2021-02-10T04:32:39.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: osd.2 (osd.2) 498 : cluster [WRN] 2021-02-10T04:32:39.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:39.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:39.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: :32:38.257689+0000 osd.2 (osd. 2021-02-10T04:32:39.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 2) 499 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:39.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:39.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 10T04:32:38.257693 2021-02-10T04:32:39.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 500 : cluster 2021-02-10T04:32:39.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:39.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: cluster 2021-02-10T04:32: 2021-02-10T04:32:39.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 38.257696+0000 osd.2 (osd.2) 501 2021-02-10T04:32:39.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:39.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:39.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: :32:38.257699+0000 osd.2 (osd. 2021-02-10T04:32:39.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 2) 502 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:39.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: cluster 2021-02-10T04:32: 2021-02-10T04:32:39.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: 38.257702+0000 osd.2 (osd.2) 503 2021-02-10T04:32:39.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:39.189 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: cluster 2021-02-10T04:32:39.189 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 2021-02-10T04:32:39.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 2021 2021-02-10T04:32:39.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: - 2021-02-10T04:32:39.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 02 2021-02-10T04:32:39.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: - 2021-02-10T04:32:39.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 10 2021-02-10T04:32:39.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: T 2021-02-10T04:32:39.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 04 2021-02-10T04:32:39.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: : 2021-02-10T04:32:39.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 32 2021-02-10T04:32:39.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: : 2021-02-10T04:32:39.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 37 2021-02-10T04:32:39.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: . 2021-02-10T04:32:39.190 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 409807 2021-02-10T04:32:39.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: +0000 2021-02-10T04:32:39.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 2021-02-10T04:32:39.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: osd.1 2021-02-10T04:32:39.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: ( 2021-02-10T04:32:39.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: osd 2021-02-10T04:32:39.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: . 2021-02-10T04:32:39.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 1 2021-02-10T04:32:39.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: ) 2021-02-10T04:32:39.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 413 2021-02-10T04:32:39.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: : 2021-02-10T04:32:39.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: cluster 2021-02-10T04:32:39.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:39.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:39.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 02-10T04:32:37.409815+0000 osd.1 (osd.1 2021-02-10T04:32:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: ) 414 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: cluster 2021-02-10T04:32:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 2021-02-10T04:32:37.409819+0000 2021-02-10T04:32:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: osd.1 (osd.1) 415 : cluster [WRN] 2021-02-10T04:32:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: cluster 2021-02-10T04:32:37 2021-02-10T04:32:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: .409823+0000 osd.1 (osd.1) 416 : 2021-02-10T04:32:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:32:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 32:37.409829+0000 osd.1 (osd.1) 2021-02-10T04:32:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 417 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:32:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: T04:32:37.409850+0000 osd.1 (osd 2021-02-10T04:32:39.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: .1) 418 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:39.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:39.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: -10T04:32:37.467004+0000 mon.a 2021-02-10T04:32:39.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: (mon.0) 1097 : cluster [DBG] osdmap e340: 8 total, 8 up, 8 in 2021-02-10T04:32:39.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:39.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: -10T04:32:37.485674+0000 osd.4 ( 2021-02-10T04:32:39.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: osd.4) 361 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:39.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:39.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: -02-10T04:32:37.485687+0000 2021-02-10T04:32:39.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: osd.4 (osd.4) 362 : cluster [WRN] 2021-02-10T04:32:39.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:39.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: cluster 2021-02-10T04:32:39.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 2021-02-10T04:32:37.485694+0000 osd.4 (osd 2021-02-10T04:32:39.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: .4) 363 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:39.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:39.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 10T04:32:37.485700+0000 osd.4 ( 2021-02-10T04:32:39.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: osd.4) 364 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:39.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:39.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: -10T04:32:37.721737+0000 osd.7 2021-02-10T04:32:39.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: (osd.7) 46 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:32:39.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:39.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 02-10T04:32:37.721932+0000 osd.7 2021-02-10T04:32:39.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: (osd.7) 47 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:32:39.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 2021-02-10T04:32:39.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: audit 2021-02-10T04:32:37 2021-02-10T04:32:39.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: .915097+0000 mgr.y (mgr.14140) 487 : 2021-02-10T04:32:39.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:32:39.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:32:39.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: :32:38.059467+0000 mgr.y (mgr. 2021-02-10T04:32:39.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 14140) 488 : cluster [DBG] pgmap v660: 168 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 4 active+clean+snaptrim, 162 active+clean; 169 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 4.0 MiB/s wr, 22 op/s 2021-02-10T04:32:39.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: audit 2021 2021-02-10T04:32:39.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: -02-10T04:32:38.070908+0000 2021-02-10T04:32:39.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: mon.a (mon.0) 1098 : audit [INF] 2021-02-10T04:32:39.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:32:39.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: audit 2021-02-10T04:32:38 2021-02-10T04:32:39.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: .071267+0000 mon.a (mon.0) 1099 : 2021-02-10T04:32:39.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:32:39.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:32:39.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 32:38.096135+0000 mon.a (mon.0) 2021-02-10T04:32:39.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 1100 : cluster [WRN] Health check update: 18 slow ops, oldest one blocked for 202 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:32:39.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:32:39.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 04:32:38.257675+0000 osd.2 (osd 2021-02-10T04:32:39.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: .2) 496 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:39.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:39.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: -10T04:32:38.257681+0000 osd.2 ( 2021-02-10T04:32:39.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: osd.2) 497 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:39.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:39.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 02-10T04:32:38.257685+0000 osd.2 2021-02-10T04:32:39.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: (osd.2) 498 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:39.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 2021-02-10T04:32:39.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:39.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: -02-10T04:32:38.257689+0000 osd.2 (osd.2 2021-02-10T04:32:39.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: ) 499 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:39.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:32:39.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 04:32:38.257693+0000 osd.2 (osd. 2021-02-10T04:32:39.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 2) 500 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:39.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:39.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 10T04:32:38.257696+0000 osd.2 ( 2021-02-10T04:32:39.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: osd.2) 501 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:39.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:39.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 02-10T04:32:38. 2021-02-10T04:32:39.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 257699+0000 osd.2 (osd.2) 502 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:39.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: 2021-02-10T04:32:39.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[29319]: cluster 2021-02-10T04:32:38.257702+0000 osd.2 (osd.2) 503 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:39.198 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:38 smithi085 conmon[52836]: debug 2021-02-10T04:32:38.807+0000 7fba73fb6700 -1 osd.3 pg_epoch: 341 pg[4.14( v 63'26 (0'0,63'26] local-lis/les=71/72 n=11 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,2,7] r=0 lpr=71 crt=63'26 lcod 63'25 mlcod 0'0 active+clean] scrubber pg(4.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55fec48762c0) 2021-02-10T04:32:39.486 INFO:tasks.rados.rados.0.smithi085.stdout:1232: done (14 left) 2021-02-10T04:32:39.486 INFO:tasks.rados.rados.0.smithi085.stdout:1233: done (13 left) 2021-02-10T04:32:39.487 INFO:tasks.rados.rados.0.smithi085.stdout:1234: read oid 36 snap 120 2021-02-10T04:32:39.487 INFO:tasks.rados.rados.0.smithi085.stdout:1234: expect deleted 2021-02-10T04:32:39.487 INFO:tasks.rados.rados.0.smithi085.stdout:1235: write (excl) oid 46 current snap is 125 2021-02-10T04:32:39.487 INFO:tasks.rados.rados.0.smithi085.stdout:1235: seq_num 454 ranges {430378=627873,1700472=622156,3017921=612457} 2021-02-10T04:32:39.494 INFO:tasks.rados.rados.0.smithi085.stdout:1235: writing smithi08575397-46 from 430378 to 1058251 tid 1 2021-02-10T04:32:39.502 INFO:tasks.rados.rados.0.smithi085.stdout:1235: writing smithi08575397-46 from 1700472 to 2322628 tid 2 2021-02-10T04:32:39.511 INFO:tasks.rados.rados.0.smithi085.stdout:1235: writing smithi08575397-46 from 3017921 to 3630378 tid 3 2021-02-10T04:32:39.511 INFO:tasks.rados.rados.0.smithi085.stdout:1236: read oid 3 snap 123 2021-02-10T04:32:39.511 INFO:tasks.rados.rados.0.smithi085.stdout:1236: expect (ObjNum 437 snap 116 seq_num 437) 2021-02-10T04:32:39.511 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:39.511 INFO:tasks.rados.rados.0.smithi085.stdout:1235: finishing write tid 1 to smithi08575397-46 2021-02-10T04:32:39.511 INFO:tasks.rados.rados.0.smithi085.stdout:1235: finishing write tid 2 to smithi08575397-46 2021-02-10T04:32:39.511 INFO:tasks.rados.rados.0.smithi085.stdout:1234: done (15 left) 2021-02-10T04:32:39.511 INFO:tasks.rados.rados.0.smithi085.stdout:1237: write (excl) oid 21 current snap is 125 2021-02-10T04:32:39.512 INFO:tasks.rados.rados.0.smithi085.stdout:1237: seq_num 455 ranges {447185=703122,1642601=765814,2928991=447707} 2021-02-10T04:32:39.518 INFO:tasks.rados.rados.0.smithi085.stdout:1237: writing smithi08575397-21 from 447185 to 1150307 tid 1 2021-02-10T04:32:39.526 INFO:tasks.rados.rados.0.smithi085.stdout:1237: writing smithi08575397-21 from 1642601 to 2408415 tid 2 2021-02-10T04:32:39.532 INFO:tasks.rados.rados.0.smithi085.stdout:1237: writing smithi08575397-21 from 2928991 to 3376698 tid 3 2021-02-10T04:32:39.532 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:39.532 INFO:tasks.rados.rados.0.smithi085.stdout:1236: expect (ObjNum 437 snap 116 seq_num 437) 2021-02-10T04:32:39.691 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[45443]: debug 2021-02-10T04:32:39.360+0000 7f7140d73700 -1 osd.1 341 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:32:39.692 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[49106]: debug 2021-02-10T04:32:39.284+0000 7f77a3e94700 -1 osd.2 341 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:32:39.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: cluster 2021-02-10T04:32:39.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: 2021-02-10T04:32:39.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: 2021 2021-02-10T04:32:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: - 2021-02-10T04:32:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: 02 2021-02-10T04:32:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: -10T04:32:38.386684+0000 2021-02-10T04:32:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: osd.1 (osd.1) 419 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: 02-10T04:32:38.386693+0000 osd.1 2021-02-10T04:32:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: (osd.1) 420 : cluster [WRN] 2021-02-10T04:32:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: -02-10T04:32:38. 2021-02-10T04:32:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: 386699+0000 osd.1 (osd.1) 2021-02-10T04:32:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: 421 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:39.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: cluster 2021-02-10T04:32:39.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:32:39.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: 04:32:38.386704+0000 osd.1 (osd.1) 422 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:39.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:39.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: 02-10T04:32: 2021-02-10T04:32:39.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: 38.386711+0000 osd.1 (osd.1) 423 : cluster 2021-02-10T04:32:39.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:39.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:39.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: 02-10T04:32:38.386720+0000 osd.1 (osd.1) 424 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:39.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:39.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: 04:32:38.481943+0000 mon.a (mon.0 2021-02-10T04:32:39.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: ) 1101 : cluster [DBG] osdmap e341: 8 total, 8 up, 8 in 2021-02-10T04:32:39.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: cluster 2021-02-10T04:32:39.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: 2021-02-10T04:32:38.524412+0000 osd.4 2021-02-10T04:32:39.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: (osd.4) 365 : cluster 2021-02-10T04:32:39.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:39.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: cluster 2021-02-10T04:32:39.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:32:39.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: 10T04:32:38. 2021-02-10T04:32:39.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: 524424+0000 osd.4 (osd.4) 2021-02-10T04:32:39.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: 366 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:39.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:39.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: -02-10T04:32:38.524431 2021-02-10T04:32:39.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 367 : cluster 2021-02-10T04:32:39.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:39.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:32:39.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: T04:32:38.524451+0000 osd.4 ( 2021-02-10T04:32:39.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: osd.4) 368 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:39.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:39.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: -02-10T04:32:39.285310+0000 osd.2 (osd 2021-02-10T04:32:39.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: .2) 504 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:39.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: 2021-02-10T04:32:39.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:39.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: 10T04:32:39.285322+0000 osd.2 (osd.2) 505 : cluster [WRN] 2021-02-10T04:32:39.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:39.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:39.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: 02-10T04:32:39.285328+0000 osd.2 (osd.2) 506 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:39.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: cluster 2021-02-10T04:32:39 2021-02-10T04:32:39.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: .285334+0000 osd.2 (osd.2) 507 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:39.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:32:39.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: T04:32:39.285341+0000 osd.2 (osd.2) 508 : 2021-02-10T04:32:39.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:39.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:32:39.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: T04:32:39.285347+0000 osd.2 (osd.2) 509 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:39.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: cluster 2021-02-10T04:32:39.285352 2021-02-10T04:32:39.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 510 : cluster [WRN] 2021-02-10T04:32:39.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:39.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:39.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: 04:32:39.285358+0000 osd.2 (osd 2021-02-10T04:32:39.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: .2) 511 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:39.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:39.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: -02-10T04:32:39 2021-02-10T04:32:39.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: .361195+0000 osd.1 (osd.1) 425 : 2021-02-10T04:32:39.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:39.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:39.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: -10T04:32:39.361203 2021-02-10T04:32:39.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 426 : cluster 2021-02-10T04:32:39.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:39.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:32:39.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: T04:32:39.361206+0000 2021-02-10T04:32:39.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: osd.1 (osd.1) 427 : 2021-02-10T04:32:39.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:39.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:39.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: 02-10T04:32:39. 2021-02-10T04:32:39.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: 361210+0000 osd.1 (osd.1) 428 : 2021-02-10T04:32:39.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:39.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:39.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: -10T04:32:39.361213+0000 osd.1 (osd.1) 429 : cluster 2021-02-10T04:32:39.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:39.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: cluster 2021-02-10T04:32:39 2021-02-10T04:32:39.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: .361217+0000 osd.1 (osd.1) 430 : 2021-02-10T04:32:39.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:39.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: cluster 2021-02-10T04:32:38.386684+0000 osd.1 (osd.1) 419 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:39.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: 2021-02-10T04:32:39.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: cluster 2021-02-10T04:32:38.386693+0000 osd.1 (osd.1) 420 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:39.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: cluster 2021-02-10T04:32:38. 2021-02-10T04:32:39.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: 386699+0000 osd.1 (osd.1) 421 : 2021-02-10T04:32:39.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:39.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:32:39.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: 04:32:38.386704+0000 osd.1 (osd. 2021-02-10T04:32:39.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: 1) 422 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:39.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:32:39.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: 32:38.386711+0000 osd.1 (osd.1) 423 : cluster 2021-02-10T04:32:39.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:39.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: cluster 2021-02-10T04:32:39.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: 2021-02-10T04:32:38. 2021-02-10T04:32:39.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: 386720+0000 osd.1 (osd.1) 424 : cluster [WRN] 2021-02-10T04:32:39.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:39.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:32:39.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: 32:38.481943+0000 mon.a (mon 2021-02-10T04:32:39.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: .0) 1101 : cluster [DBG] 2021-02-10T04:32:39.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: osdmap e341: 8 total, 8 up, 8 in 2021-02-10T04:32:39.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:39.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: 02-10T04:32: 2021-02-10T04:32:39.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: 38.524412+0000 osd.4 (osd.4) 365 : cluster [WRN] 2021-02-10T04:32:39.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:39.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:32:39.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: 04:32:38.524424+0000 osd.4 (osd.4) 2021-02-10T04:32:39.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: 366 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:39.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:39.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: 10T04:32:38.524431+0000 osd.4 (osd. 2021-02-10T04:32:39.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: 4) 367 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:39.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:39.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: -02-10T04:32:38.524451 2021-02-10T04:32:39.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 368 : cluster 2021-02-10T04:32:39.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:39.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:32:39.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: :32:39.285310+0000 osd.2 (osd.2) 504 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:39.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:39.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: 02-10T04:32:39.285322 2021-02-10T04:32:39.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 505 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:39.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:32:39.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: T04:32:39.285328+0000 osd.2 ( 2021-02-10T04:32:39.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: osd.2) 506 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:39.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:39.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: 10T04:32:39.285334+0000 osd.2 (osd 2021-02-10T04:32:39.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: .2) 507 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:39.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:39.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: -10T04:32:39.285341+0000 osd.2 ( 2021-02-10T04:32:39.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: osd.2) 508 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:39.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:39.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: -10T04:32:39.285347+0000 osd.2 ( 2021-02-10T04:32:39.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: osd.2) 509 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:39.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:39.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: -10T04:32:39.285352+0000 2021-02-10T04:32:39.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: osd.2 (osd.2) 510 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:39.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:39.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: -10T04:32:39.285358+0000 osd.2 (osd.2 2021-02-10T04:32:39.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: ) 511 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:39.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:32:39.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: :32:39.361195+0000 osd.1 (osd.1) 425 : 2021-02-10T04:32:39.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:39.712 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: cluster 2021-02-10T04:32: 2021-02-10T04:32:39.712 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: 39.361203+0000 osd.1 (osd.1) 426 : cluster [WRN] 2021-02-10T04:32:39.712 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:39.712 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: cluster 2021-02-10T04:32:39.712 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: 2021-02-10T04:32:39.361206+0000 osd.1 (osd. 2021-02-10T04:32:39.713 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: 1) 427 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:39.713 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:39.713 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: -02-10T04:32:39.361210+0000 osd.1 ( 2021-02-10T04:32:39.713 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: osd.1) 428 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:39.713 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:39.713 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: -10T04:32:39.361213+0000 osd.1 (osd.1) 429 : 2021-02-10T04:32:39.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:39.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: cluster 2021-02-10T04:32:39 2021-02-10T04:32:39.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: .361217+0000 osd.1 (osd.1) 430 : cluster 2021-02-10T04:32:39.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:39.840 INFO:tasks.rados.rados.0.smithi085.stdout:1235: finishing write tid 3 to smithi08575397-46 2021-02-10T04:32:39.840 INFO:tasks.rados.rados.0.smithi085.stdout:1235: finishing write tid 4 to smithi08575397-46 2021-02-10T04:32:39.840 INFO:tasks.rados.rados.0.smithi085.stdout:1236: done (15 left) 2021-02-10T04:32:39.840 INFO:tasks.rados.rados.0.smithi085.stdout:1238: read oid 26 snap -1 2021-02-10T04:32:39.840 INFO:tasks.rados.rados.0.smithi085.stdout:1238: expect (ObjNum 448 snap 121 seq_num 448) 2021-02-10T04:32:39.840 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:39.840 INFO:tasks.rados.rados.0.smithi085.stdout:1235: finishing write tid 5 to smithi08575397-46 2021-02-10T04:32:39.840 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 628 (ObjNum 454 snap 125 seq_num 454) dirty exists 2021-02-10T04:32:39.841 INFO:tasks.rados.rados.0.smithi085.stdout:1235: left oid 46 (ObjNum 454 snap 125 seq_num 454) 2021-02-10T04:32:39.841 INFO:tasks.rados.rados.0.smithi085.stdout:1237: finishing write tid 1 to smithi08575397-21 2021-02-10T04:32:39.841 INFO:tasks.rados.rados.0.smithi085.stdout:1237: finishing write tid 2 to smithi08575397-21 2021-02-10T04:32:39.841 INFO:tasks.rados.rados.0.smithi085.stdout:1235: done (15 left) 2021-02-10T04:32:39.841 INFO:tasks.rados.rados.0.smithi085.stdout:1239: read oid 22 snap -1 2021-02-10T04:32:39.841 INFO:tasks.rados.rados.0.smithi085.stdout:1239: expect (ObjNum 436 snap 116 seq_num 436) 2021-02-10T04:32:39.841 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:39.841 INFO:tasks.rados.rados.0.smithi085.stdout:1237: finishing write tid 3 to smithi08575397-21 2021-02-10T04:32:39.841 INFO:tasks.rados.rados.0.smithi085.stdout:1237: finishing write tid 4 to smithi08575397-21 2021-02-10T04:32:39.841 INFO:tasks.rados.rados.0.smithi085.stdout:1237: finishing write tid 5 to smithi08575397-21 2021-02-10T04:32:39.841 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 675 (ObjNum 455 snap 125 seq_num 455) dirty exists 2021-02-10T04:32:39.841 INFO:tasks.rados.rados.0.smithi085.stdout:1237: left oid 21 (ObjNum 455 snap 125 seq_num 455) 2021-02-10T04:32:39.841 INFO:tasks.rados.rados.0.smithi085.stdout:1237: done (15 left) 2021-02-10T04:32:39.842 INFO:tasks.rados.rados.0.smithi085.stdout:1240: snap_create 2021-02-10T04:32:39.847 INFO:tasks.rados.rados.0.smithi085.stdout:1238: expect (ObjNum 448 snap 121 seq_num 448) 2021-02-10T04:32:39.895 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[33677]: debug 2021-02-10T04:32:39.521+0000 7fbec19a1700 -1 osd.4 342 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:32:39.944 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: cluster 2021-02-10T04:32:38.386684+0000 osd.1 ( 2021-02-10T04:32:39.945 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: osd.1) 419 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:39.945 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: cluster 2021-02-10T04:32:38.386693+0000 osd.1 (osd. 2021-02-10T04:32:39.945 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: 1) 420 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:39.945 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: cluster 2021-02-10T04:32:39.945 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: 2021-02-10T04:32:38.386699+0000 2021-02-10T04:32:39.945 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: osd.1 (osd.1) 421 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:39.945 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: 2021-02-10T04:32:39.945 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:39.945 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: T04:32:38.386704+0000 osd.1 2021-02-10T04:32:39.945 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: (osd.1) 422 : cluster [WRN] 2021-02-10T04:32:39.945 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:39.945 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: cluster 2021-02-10T04:32:39.945 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: 2021-02-10T04:32:38.386711 2021-02-10T04:32:39.946 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 423 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:39.946 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: 2021-02-10T04:32:39.946 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: cluster 2021-02-10T04:32:39.946 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: 2021-02-10T04:32:38.386720+0000 osd.1 (osd.1 2021-02-10T04:32:39.946 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: ) 424 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:39.946 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: cluster 2021-02-10T04:32:39.946 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:32:39.946 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: :32:38.481943+0000 mon.a (mon.0) 1101 : cluster [DBG] osdmap e341: 8 total, 8 up, 8 in 2021-02-10T04:32:39.946 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:39.946 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: -02-10T04:32:38. 2021-02-10T04:32:39.946 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: 524412+0000 osd.4 (osd.4 2021-02-10T04:32:39.946 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: ) 365 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:39.947 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: 2021-02-10T04:32:39.947 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:32:39.947 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: :32:38.524424+0000 2021-02-10T04:32:39.947 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: osd.4 (osd.4) 366 : cluster 2021-02-10T04:32:39.947 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:39.947 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: cluster 2021-02-10T04:32:39.947 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:32:39.947 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: 32:38.524431+0000 osd.4 2021-02-10T04:32:39.947 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: (osd.4) 367 : cluster 2021-02-10T04:32:39.947 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:39.947 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: cluster 2021-02-10T04:32:39.947 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:32:39.947 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: 32:38.524451+0000 osd.4 2021-02-10T04:32:39.947 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: (osd.4) 368 : cluster 2021-02-10T04:32:39.948 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:39.948 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: cluster 2021-02-10T04:32:39.948 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: 2021-02-10T04:32: 2021-02-10T04:32:39.948 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: 39.285310+0000 osd.2 (osd.2) 504 : 2021-02-10T04:32:39.948 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:39.948 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:39.948 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: -02-10T04:32:39.285322+0000 2021-02-10T04:32:39.948 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: osd.2 (osd.2) 505 : cluster 2021-02-10T04:32:39.948 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:39.948 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: cluster 2021-02-10T04:32:39.948 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:32:39.948 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: 32:39.285328+0000 osd.2 ( 2021-02-10T04:32:39.948 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: osd.2) 506 : cluster 2021-02-10T04:32:39.949 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:39.949 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:39.949 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: 04:32:39.285334+0000 osd.2 ( 2021-02-10T04:32:39.949 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: osd.2) 507 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:39.949 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:39.949 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: 02-10T04:32:39.285341+0000 osd.2 (osd.2) 2021-02-10T04:32:39.949 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: 508 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:39.949 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:32:39.949 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: 32:39.285347+0000 osd.2 (osd.2) 509 2021-02-10T04:32:39.949 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:39.949 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: cluster 2021-02-10T04:32: 2021-02-10T04:32:39.949 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: 39.285352+0000 osd.2 (osd.2) 510 : cluster 2021-02-10T04:32:39.949 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:39.950 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: cluster 2021-02-10T04:32:39.285358+0000 2021-02-10T04:32:39.950 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: osd.2 (osd.2) 511 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:39.950 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:39.950 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: -10T04:32:39.361195+0000 osd.1 ( 2021-02-10T04:32:39.950 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: osd.1) 425 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:39.950 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:32:39.950 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: 10T04:32:39.361203+0000 osd.1 (osd. 2021-02-10T04:32:39.950 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: 1) 426 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:39.950 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:39.950 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: T04:32:39.361206+0000 osd.1 (osd.1 2021-02-10T04:32:39.950 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: ) 427 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:39.950 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:32:39.950 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: :32:39.361210+0000 osd.1 (osd 2021-02-10T04:32:39.951 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: .1) 428 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:39.951 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: cluster 2021-02-10T04:32: 2021-02-10T04:32:39.951 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: 39.361213+0000 osd.1 (osd.1) 429 : cluster 2021-02-10T04:32:39.951 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:39.951 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:32:39.951 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: 32:39.361217+0000 osd.1 (osd.1) 2021-02-10T04:32:39.951 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:39 smithi164 conmon[29288]: 430 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:40.171 INFO:tasks.rados.rados.0.smithi085.stdout:1239: expect (ObjNum 436 snap 116 seq_num 436) 2021-02-10T04:32:40.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:39 smithi085 conmon[52836]: debug 2021-02-10T04:32:39.790+0000 7fba6ffae700 -1 osd.3 pg_epoch: 342 pg[4.14( v 63'26 (0'0,63'26] local-lis/les=71/72 n=11 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,2,7] r=0 lpr=71 crt=63'26 lcod 63'25 mlcod 0'0 active+clean] scrubber pg(4.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55feb5c81a20) 2021-02-10T04:32:40.498 INFO:tasks.rados.rados.0.smithi085.stdout:1238: done (15 left) 2021-02-10T04:32:40.498 INFO:tasks.rados.rados.0.smithi085.stdout:1239: done (14 left) 2021-02-10T04:32:40.498 INFO:tasks.rados.rados.0.smithi085.stdout:1240: done (13 left) 2021-02-10T04:32:40.498 INFO:tasks.rados.rados.0.smithi085.stdout:1241: read oid 16 snap -1 2021-02-10T04:32:40.498 INFO:tasks.rados.rados.0.smithi085.stdout:1241: expect (ObjNum 432 snap 114 seq_num 432) 2021-02-10T04:32:40.498 INFO:tasks.rados.rados.0.smithi085.stdout:1242: rollback oid 28 current snap is 126 2021-02-10T04:32:40.499 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 28 to 109 2021-02-10T04:32:40.499 INFO:tasks.rados.rados.0.smithi085.stdout:1243: read oid 8 snap -1 2021-02-10T04:32:40.499 INFO:tasks.rados.rados.0.smithi085.stdout:1243: expect deleted 2021-02-10T04:32:40.499 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:40.499 INFO:tasks.rados.rados.0.smithi085.stdout:1243: done (15 left) 2021-02-10T04:32:40.500 INFO:tasks.rados.rados.0.smithi085.stdout:1244: snap_create 2021-02-10T04:32:40.507 INFO:tasks.rados.rados.0.smithi085.stdout:1241: expect (ObjNum 432 snap 114 seq_num 432) 2021-02-10T04:32:40.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[49106]: debug 2021-02-10T04:32:40.311+0000 7f77a3e94700 -1 osd.2 342 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:32:40.691 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[45443]: debug 2021-02-10T04:32:40.692 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[45443]: 2021-02-10T04:32:40.325+0000 7f7140d73700 -1 osd.1 342 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:32:40.692 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[45443]: debug 2021-02-10T04:32:40.692 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[45443]: 2021-02-10T04:32:40.326+0000 7f7125cf8700 -1 osd.1 pg_epoch: 342 pg[4.4( v 63'31 (0'0,63'31] local-lis/les=55/56 n=14 ec=55/55 lis/c=55/55 les/c/f=56/56/0 sis=55) [1,2,3] r=0 lpr=55 crt=63'31 lcod 63'30 mlcod 63'30 active+clean] scrubber pg(4.4) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 3 (0x55dbc85f29a0) 2021-02-10T04:32:40.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: cluster 2021-02-10T04:32:40.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: 2021-02-10T04:32:40.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:32:40.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: :32:39.485987+0000 mon.a (mon.0) 1102 : cluster [DBG] osdmap e342: 8 total, 8 up, 8 in 2021-02-10T04:32:40.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:40.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: -10T04:32:39.522107+0000 osd.4 (osd. 2021-02-10T04:32:40.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: 4) 369 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:40.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:40.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: -02-10T04:32:39.522113+0000 2021-02-10T04:32:40.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: osd.4 (osd.4) 370 : cluster [WRN] 2021-02-10T04:32:40.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:40.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: cluster 2021-02-10T04:32:39. 2021-02-10T04:32:40.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: 522117+0000 osd.4 (osd.4) 371 : cluster 2021-02-10T04:32:40.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:40.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:32:40.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: T04:32:39.522121+0000 osd.4 (osd 2021-02-10T04:32:40.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: .4) 372 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:40.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: cluster 2021-02-10T04:32:40.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: 2021-02-10T04:32:40.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: 2021 2021-02-10T04:32:40.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: - 2021-02-10T04:32:40.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: 02 2021-02-10T04:32:40.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: - 2021-02-10T04:32:40.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: 10 2021-02-10T04:32:40.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: T 2021-02-10T04:32:40.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: 04 2021-02-10T04:32:40.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: : 2021-02-10T04:32:40.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: 32 2021-02-10T04:32:40.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: :40.059897+0000 mgr.y (mgr. 2021-02-10T04:32:40.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: 14140) 489 : cluster [DBG] pgmap v663: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 164 active+clean; 173 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 6.6 MiB/s rd, 787 KiB/s wr, 8 op/s 2021-02-10T04:32:40.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:40.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: 02-10T04:32:40.311964+0000 osd.2 (osd.2) 512 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:40.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: cluster 2021-02-10T04:32:40.311970+0000 osd.2 (osd.2) 513 : cluster 2021-02-10T04:32:40.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:40.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: cluster 2021-02-10T04:32:40.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: 2021-02-10T04:32:40.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:32:40.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: :32:40.311977+0000 osd.2 (osd.2) 514 : cluster 2021-02-10T04:32:40.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:40.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: cluster 2021-02-10T04:32:40.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: 2021-02 2021-02-10T04:32:40.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: -10T04:32:40.311983+0000 osd.2 ( 2021-02-10T04:32:40.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: osd.2) 515 2021-02-10T04:32:40.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:40.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:40.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: -10T04:32:40.311990+0000 osd.2 2021-02-10T04:32:40.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: (osd.2) 516 : cluster [WRN] 2021-02-10T04:32:40.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:40.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:32:40.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: -10T04:32:40.311996+0000 osd.2 2021-02-10T04:32:40.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: (osd.2) 517 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:40.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: 2021-02-10T04:32:40.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: cluster 2021-02-10T04:32:40.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: 2021-02-10T04:32:40.312002+0000 osd.2 2021-02-10T04:32:40.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: (osd.2) 518 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:40.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: 2021-02-10T04:32:40.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: cluster 2021-02-10T04:32:40.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: 2021-02-10T04:32:40.312007+0000 osd.2 (osd.2) 519 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:40.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:32:40.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: 32:40.326254+0000 osd.1 (osd.1 2021-02-10T04:32:40.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: ) 431 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:40.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: cluster 2021-02-10T04:32:40.326264+0000 osd.1 (osd.1) 432 : cluster [WRN] 2021-02-10T04:32:40.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:40.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: cluster 2021-02-10T04:32:40.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:32:40.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: 32:40.326271+0000 osd.1 (osd.1) 433 : cluster 2021-02-10T04:32:40.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:40.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:40.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: 10T04:32:40.326277+0000 osd.1 2021-02-10T04:32:40.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: (osd.1) 434 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:40.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: 2021-02-10T04:32:40.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: cluster 2021-02-10T04:32:40.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: 2021-02-10T04:32:40.326284+0000 2021-02-10T04:32:40.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: osd.1 (osd.1) 435 : cluster [WRN] 2021-02-10T04:32:40.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:40.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: cluster 2021-02-10T04:32:40.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: 2021-02-10T04:32:40. 2021-02-10T04:32:40.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: 326291+0000 osd.1 (osd.1) 436 : 2021-02-10T04:32:40.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:40.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: cluster 2021-02-10T04:32:40.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: 2021-02-10T04:32:40.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: 2021 2021-02-10T04:32:40.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: - 2021-02-10T04:32:40.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: 02 2021-02-10T04:32:40.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: - 2021-02-10T04:32:40.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: 10 2021-02-10T04:32:40.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: T 2021-02-10T04:32:40.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: 04 2021-02-10T04:32:40.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: :32:39.485987+0000 mon.a (mon. 2021-02-10T04:32:40.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: 0) 1102 : cluster [DBG] osdmap e342: 8 total, 8 up, 8 in 2021-02-10T04:32:40.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: cluster 2021-02-10T04:32:40.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: 2021-02-10T04:32:39.522107+0000 osd.4 (osd.4) 369 : 2021-02-10T04:32:40.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:40.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: cluster 2021-02-10T04:32:39 2021-02-10T04:32:40.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: .522113+0000 osd.4 (osd.4) 370 : cluster [WRN] 2021-02-10T04:32:40.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:40.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: cluster 2021-02-10T04:32:40.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: 2021-02-10T04:32:39.522117+0000 osd.4 (osd.4) 2021-02-10T04:32:40.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: 371 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:40.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: cluster 2021-02-10T04:32: 2021-02-10T04:32:40.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: 39.522121+0000 osd.4 (osd.4) 372 : cluster [WRN] 2021-02-10T04:32:40.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:40.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:40.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: 02-10T04:32:40.059897+0000 mgr.y (mgr.14140) 489 : 2021-02-10T04:32:40.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: cluster [DBG] pgmap v663: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 164 active+clean; 173 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 6.6 MiB/s rd, 787 KiB/s wr, 8 op/s 2021-02-10T04:32:40.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: cluster 2021-02-10T04:32:40.311964 2021-02-10T04:32:40.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 512 : cluster [WRN] 2021-02-10T04:32:40.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:40.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: cluster 2021-02-10T04:32:40 2021-02-10T04:32:40.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: .311970+0000 osd.2 (osd.2 2021-02-10T04:32:40.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: ) 513 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:40.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: 2021-02-10T04:32:40.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:32:40.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: T04:32:40.311977+0000 2021-02-10T04:32:40.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: osd.2 (osd.2) 514 : cluster 2021-02-10T04:32:40.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:40.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:32:40.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: 32:40.311983+0000 osd.2 (osd. 2021-02-10T04:32:40.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: 2) 515 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:40.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: cluster 2021-02-10T04:32:40.311990+0000 2021-02-10T04:32:40.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: osd.2 (osd.2) 516 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:40.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:40.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: -10T04:32:40.311996+0000 osd.2 2021-02-10T04:32:40.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: (osd.2) 517 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:40.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:40.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: -02-10T04:32: 2021-02-10T04:32:40.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: 40.312002+0000 osd.2 (osd 2021-02-10T04:32:40.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: .2) 518 : cluster [WRN] 2021-02-10T04:32:40.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:40.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: cluster 2021-02-10T04:32:40.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: 2021-02-10T04:32:40. 2021-02-10T04:32:40.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: 312007+0000 osd.2 (osd.2) 519 : 2021-02-10T04:32:40.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:40.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: cluster 2021-02-10T04:32:40.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:32:40.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: 32:40.326254+0000 osd.1 ( 2021-02-10T04:32:40.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: osd.1) 431 : cluster 2021-02-10T04:32:40.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:40.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:32:40.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: T04:32:40.326264+0000 osd.1 (osd. 2021-02-10T04:32:40.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: 1) 432 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:40.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:40.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: 10T04:32:40.326271+0000 osd.1 2021-02-10T04:32:40.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: (osd.1) 433 : cluster 2021-02-10T04:32:40.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:40.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:40.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: 02-10T04:32:40 2021-02-10T04:32:40.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: .326277+0000 osd.1 (osd.1) 434 : 2021-02-10T04:32:40.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:40.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:32:40.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: 32:40.326284+0000 osd.1 (osd.1) 435 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:40.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: cluster 2021-02-10T04:32:40.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: 2021-02-10T04:32 2021-02-10T04:32:40.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: :40.326291+0000 osd.1 (osd. 2021-02-10T04:32:40.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: 1) 436 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:40.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[29319]: 2021-02-10T04:32:40.806 INFO:tasks.rados.rados.0.smithi085.stdout:1242: finishing rollback tid 0 to smithi08575397-28 2021-02-10T04:32:40.806 INFO:tasks.rados.rados.0.smithi085.stdout:1242: finishing rollback tid 1 to smithi08575397-28 2021-02-10T04:32:40.806 INFO:tasks.rados.rados.0.smithi085.stdout:1242: finishing rollback tid 2 to smithi08575397-28 2021-02-10T04:32:40.806 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 607 (ObjNum 423 snap 109 seq_num 423) dirty exists 2021-02-10T04:32:40.893 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[33677]: debug 2021-02-10T04:32:40.893 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[33677]: 2021-02-10T04:32:40.482+0000 7fbec19a1700 -1 osd.4 342 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:32:40.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: cluster 2021-02-10T04:32:39.485987+0000 mon.a (mon.0) 2021-02-10T04:32:40.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: 1102 : cluster [DBG] osdmap e342: 8 total, 8 up, 8 in 2021-02-10T04:32:40.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: cluster 2021-02-10T04:32:39.522107+0000 osd.4 (osd.4) 369 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:40.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: cluster 2021-02-10T04:32:39.522113+0000 osd.4 ( 2021-02-10T04:32:40.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: osd.4) 370 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:40.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: cluster 2021-02-10T04:32:39.522117 2021-02-10T04:32:40.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 371 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:40.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: 2021-02-10T04:32:40.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: cluster 2021-02-10T04:32:40.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: 2021-02-10T04:32:39.522121+0000 osd.4 ( 2021-02-10T04:32:40.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: osd.4) 372 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:40.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:40.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: 02-10T04:32:40.059897+0000 mgr.y ( 2021-02-10T04:32:40.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: mgr.14140) 489 : cluster [DBG] pgmap v663: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 164 active+clean; 173 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 6.6 MiB/s rd, 787 KiB/s wr, 8 op/s 2021-02-10T04:32:40.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: cluster 2021-02-10T04:32:40.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: 2021-02-10T04:32:40.311964+0000 2021-02-10T04:32:40.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: osd.2 (osd.2) 512 : cluster [WRN] 2021-02-10T04:32:40.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:40.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: cluster 2021-02-10T04:32:40.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:32:40.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: 32:40.311970+0000 osd.2 (osd.2) 2021-02-10T04:32:40.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: 513 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:40.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:40.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: T04:32:40.311977 2021-02-10T04:32:40.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 514 : cluster 2021-02-10T04:32:40.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:40.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: cluster 2021-02-10T04:32:40.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: 2021-02- 2021-02-10T04:32:40.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: 10T04:32:40.311983+0000 osd.2 (osd. 2021-02-10T04:32:40.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: 2) 515 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:40.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:40.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: 02-10T04:32:40.311990+0000 osd.2 (osd. 2021-02-10T04:32:40.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: 2) 516 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:40.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:40.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: -02-10T04:32:40.311996+0000 osd.2 (osd. 2021-02-10T04:32:40.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: 2) 517 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:40.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:40.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: -02-10T04:32:40.312002+0000 osd.2 ( 2021-02-10T04:32:40.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: osd.2) 518 : cluster [WRN] 2021-02-10T04:32:40.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:40.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:40.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: 02-10T04:32:40 2021-02-10T04:32:40.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: .312007+0000 osd.2 (osd. 2021-02-10T04:32:40.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: 2) 519 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:40.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: 2021-02-10T04:32:40.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:40.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: -10T04:32:40. 2021-02-10T04:32:40.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: 326254+0000 osd.1 (osd.1 2021-02-10T04:32:40.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: ) 431 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:40.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: 2021-02-10T04:32:40.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:40.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: 02-10T04:32:40 2021-02-10T04:32:40.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: .326264+0000 osd.1 (osd.1 2021-02-10T04:32:40.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: ) 432 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:40.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:40.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: T04:32:40.326271+0000 osd.1 (osd.1) 433 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:40.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:40.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: 04:32:40.326277+0000 osd.1 (osd.1 2021-02-10T04:32:40.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: ) 434 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:40.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:40.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: 04:32:40.326284+0000 osd.1 (osd.1) 2021-02-10T04:32:40.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: 435 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:40.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:32:40.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: 32:40.326291+0000 osd.1 (osd.1) 436 2021-02-10T04:32:40.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:40 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:41.081 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:32:41.091 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:40 smithi085 conmon[52836]: debug 2021-02-10T04:32:40.776+0000 7fba73fb6700 -1 osd.3 pg_epoch: 343 pg[4.14( v 63'26 (0'0,63'26] local-lis/les=71/72 n=11 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,2,7] r=0 lpr=71 crt=63'26 lcod 63'25 mlcod 0'0 active+clean] scrubber pg(4.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55feb3a038c0) 2021-02-10T04:32:41.118 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:41.231 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:32:41.338 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:32:41.354 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[49106]: debug 2021-02-10T04:32:41.307+0000 7f77a3e94700 -1 osd.2 343 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:32:41.354 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[45443]: debug 2021-02-10T04:32:41.202+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:41.354 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[45443]: debug 2021-02-10T04:32:41.353+0000 7f7140d73700 -1 osd.1 343 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) 2021-02-10T04:32:41.371 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:41.502 INFO:tasks.rados.rados.0.smithi085.stdout:1241: done (15 left) 2021-02-10T04:32:41.502 INFO:tasks.rados.rados.0.smithi085.stdout:1242: done (14 left) 2021-02-10T04:32:41.502 INFO:tasks.rados.rados.0.smithi085.stdout:1244: done (13 left) 2021-02-10T04:32:41.503 INFO:tasks.rados.rados.0.smithi085.stdout:1245: write (excl) oid 28 current snap is 127 2021-02-10T04:32:41.503 INFO:tasks.rados.rados.0.smithi085.stdout:1245: seq_num 456 ranges {463992=778371,1701083=625825,2956414=658979} 2021-02-10T04:32:41.506 INFO:tasks.rados.rados.0.smithi085.stdout:1245: writing smithi08575397-28 from 463992 to 1242363 tid 1 2021-02-10T04:32:41.513 INFO:tasks.rados.rados.0.smithi085.stdout:1245: writing smithi08575397-28 from 1701083 to 2326908 tid 2 2021-02-10T04:32:41.522 INFO:tasks.rados.rados.0.smithi085.stdout:1245: writing smithi08575397-28 from 2956414 to 3615393 tid 3 2021-02-10T04:32:41.522 INFO:tasks.rados.rados.0.smithi085.stdout:1246: rollback oid 21 current snap is 127 2021-02-10T04:32:41.522 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 21 to 121 2021-02-10T04:32:41.522 INFO:tasks.rados.rados.0.smithi085.stdout:1245: finishing write tid 1 to smithi08575397-28 2021-02-10T04:32:41.522 INFO:tasks.rados.rados.0.smithi085.stdout:1245: finishing write tid 2 to smithi08575397-28 2021-02-10T04:32:41.522 INFO:tasks.rados.rados.0.smithi085.stdout:1247: write (excl) oid 13 current snap is 127 2021-02-10T04:32:41.522 INFO:tasks.rados.rados.0.smithi085.stdout:1247: seq_num 457 ranges {480799=569973,1475918=769483,2700190=586604} 2021-02-10T04:32:41.527 INFO:tasks.rados.rados.0.smithi085.stdout:1247: writing smithi08575397-13 from 480799 to 1050772 tid 1 2021-02-10T04:32:41.534 INFO:tasks.rados.rados.0.smithi085.stdout:1247: writing smithi08575397-13 from 1475918 to 2245401 tid 2 2021-02-10T04:32:41.540 INFO:tasks.rados.rados.0.smithi085.stdout:1247: writing smithi08575397-13 from 2700190 to 3286794 tid 3 2021-02-10T04:32:41.541 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:41.541 INFO:tasks.rados.rados.0.smithi085.stdout:1245: finishing write tid 3 to smithi08575397-28 2021-02-10T04:32:41.541 INFO:tasks.rados.rados.0.smithi085.stdout:1245: finishing write tid 4 to smithi08575397-28 2021-02-10T04:32:41.541 INFO:tasks.rados.rados.0.smithi085.stdout:1245: finishing write tid 5 to smithi08575397-28 2021-02-10T04:32:41.541 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 611 (ObjNum 456 snap 127 seq_num 456) dirty exists 2021-02-10T04:32:41.541 INFO:tasks.rados.rados.0.smithi085.stdout:1245: left oid 28 (ObjNum 456 snap 127 seq_num 456) 2021-02-10T04:32:41.541 INFO:tasks.rados.rados.0.smithi085.stdout:1245: done (15 left) 2021-02-10T04:32:41.541 INFO:tasks.rados.rados.0.smithi085.stdout:1248: snap_create 2021-02-10T04:32:41.545 INFO:tasks.rados.rados.0.smithi085.stdout:1246: finishing rollback tid 0 to smithi08575397-21 2021-02-10T04:32:41.546 INFO:tasks.rados.rados.0.smithi085.stdout:1247: finishing write tid 1 to smithi08575397-13 2021-02-10T04:32:41.548 INFO:tasks.rados.rados.0.smithi085.stdout:1246: finishing rollback tid 1 to smithi08575397-21 2021-02-10T04:32:41.549 INFO:tasks.rados.rados.0.smithi085.stdout:1246: finishing rollback tid 2 to smithi08575397-21 2021-02-10T04:32:41.549 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 681 (ObjNum 446 snap 121 seq_num 446) dirty exists 2021-02-10T04:32:41.550 INFO:tasks.rados.rados.0.smithi085.stdout:1247: finishing write tid 2 to smithi08575397-13 2021-02-10T04:32:41.551 INFO:tasks.rados.rados.0.smithi085.stdout:1247: finishing write tid 3 to smithi08575397-13 2021-02-10T04:32:41.551 INFO:tasks.rados.rados.0.smithi085.stdout:1247: finishing write tid 4 to smithi08575397-13 2021-02-10T04:32:41.552 INFO:tasks.rados.rados.0.smithi085.stdout:1247: finishing write tid 5 to smithi08575397-13 2021-02-10T04:32:41.552 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 582 (ObjNum 457 snap 127 seq_num 457) dirty exists 2021-02-10T04:32:41.552 INFO:tasks.rados.rados.0.smithi085.stdout:1247: left oid 13 (ObjNum 457 snap 127 seq_num 457) 2021-02-10T04:32:41.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[33677]: debug 2021-02-10T04:32:41.458+0000 7fbec19a1700 -1 osd.4 343 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:32:41.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[33677]: debug 2021-02-10T04:32:41.471+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: cluster 2021-02-10T04:32:40.482802+0000 2021-02-10T04:32:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: osd.4 (osd.4) 373 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: cluster 2021-02-10T04:32:40.482812+0000 osd.4 (osd.4) 374 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: 2021-02-10T04:32:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: cluster 2021-02-10T04:32:40. 2021-02-10T04:32:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: 482819+0000 osd.4 (osd.4) 375 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: -02-10T04:32:40.482825+0000 2021-02-10T04:32:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: osd.4 (osd.4) 376 : cluster [WRN] 2021-02-10T04:32:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: -02-10T04:32:40.495272+0000 mon.a (mon.0 2021-02-10T04:32:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: ) 1103 : cluster [DBG] osdmap e343: 8 total, 8 up, 8 in 2021-02-10T04:32:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: -10T04:32:41.307823+0000 osd.2 (osd 2021-02-10T04:32:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: .2) 520 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: 04:32:41.307833+0000 osd.2 ( 2021-02-10T04:32:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: osd.2) 521 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:41.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: -10T04:32:41.307840+0000 2021-02-10T04:32:41.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: osd.2 (osd.2) 522 : cluster 2021-02-10T04:32:41.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:41.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: cluster 2021-02-10T04:32:41.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: 2021-02-10T04:32:41. 2021-02-10T04:32:41.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: 307846+0000 osd.2 (osd.2) 523 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:41.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: 2021-02-10T04:32:41.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: cluster 2021-02-10T04:32:41.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: 2021-02-10T04:32:41.307855+0000 osd.2 (osd.2) 524 : cluster 2021-02-10T04:32:41.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:41.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: cluster 2021-02-10T04:32:41. 2021-02-10T04:32:41.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: 307860+0000 osd.2 (osd.2) 525 : cluster [WRN] 2021-02-10T04:32:41.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:41.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: cluster 2021-02-10T04:32:41. 2021-02-10T04:32:41.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: 307866+0000 osd.2 (osd.2) 526 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:41.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: cluster 2021-02-10T04:32:41.307871 2021-02-10T04:32:41.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 527 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:41.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:41.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: -02-10T04:32:41.353873+0000 osd.1 2021-02-10T04:32:41.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: (osd.1) 437 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:41.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:32:41.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: 32:41.353882+0000 osd.1 (osd.1) 438 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:41.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: cluster 2021-02-10T04:32:41.353888 2021-02-10T04:32:41.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 439 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:41.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:32:41.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: 10T04:32:41.353894+0000 osd.1 (osd 2021-02-10T04:32:41.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: .1) 440 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:41.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:32:41.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: 10T04:32:41.353900+0000 osd.1 (osd 2021-02-10T04:32:41.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: .1) 441 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:41.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:32:41.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: T04:32:41.353906+0000 osd.1 (osd.1 2021-02-10T04:32:41.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:41 smithi164 conmon[29288]: ) 442 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:41.649 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: cluster 2021-02-10T04:32:41.649 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:32:41.649 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: :32:40.482802+0000 osd.4 (osd.4) 373 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:41.649 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:41.649 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: -02-10T04:32:40.482812+0000 osd.4 2021-02-10T04:32:41.649 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: (osd.4) 374 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:41.649 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:41.649 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: 10T04:32:40.482819+0000 osd.4 (osd.4 2021-02-10T04:32:41.649 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: ) 375 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:41.649 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:41.649 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: -10T04:32:40.482825+0000 osd.4 ( 2021-02-10T04:32:41.649 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: osd.4) 376 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:41.650 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:32:41.650 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: -02-10T04:32:40.495272+0000 2021-02-10T04:32:41.650 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: mon.a 2021-02-10T04:32:41.650 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: (mon.0) 1103 : cluster 2021-02-10T04:32:41.650 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: [DBG] osdmap e343: 8 total, 8 up, 8 in 2021-02-10T04:32:41.650 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:41.650 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: 02-10T04:32:41.307823+0000 osd.2 2021-02-10T04:32:41.650 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: (osd.2) 520 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:41.650 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:32:41.650 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: T04:32:41.307833+0000 osd.2 (osd. 2021-02-10T04:32:41.650 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: 2) 521 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:41.650 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: cluster 2021-02-10T04:32:41.651 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: 2021-02-10T04:32:41.307840+0000 2021-02-10T04:32:41.651 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: osd.2 (osd.2) 522 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:41.651 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: 2021-02-10T04:32:41.651 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: cluster 2021-02-10T04:32:41.651 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: 2021-02-10T04:32:41.307846+0000 osd.2 (osd.2) 523 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:41.651 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: cluster 2021-02-10T04:32:41.651 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: 2021-02-10T04:32:41.307855+0000 osd.2 (osd.2 2021-02-10T04:32:41.651 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: ) 524 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:41.651 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: cluster 2021-02-10T04:32:41.651 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: 2021-02-10T04:32:41.307860+0000 2021-02-10T04:32:41.651 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: osd.2 (osd.2) 525 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:41.651 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: 2021-02-10T04:32:41.652 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: cluster 2021-02-10T04:32:41.652 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: 2021-02-10T04:32:41.307866+0000 osd.2 ( 2021-02-10T04:32:41.652 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: osd.2) 526 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:41.652 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:41.652 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: 02-10T04:32:41.307871+0000 osd.2 ( 2021-02-10T04:32:41.652 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: osd.2) 527 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:41.652 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: cluster 2021-02-10T04:32:41 2021-02-10T04:32:41.652 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: .353873+0000 osd.1 (osd.1) 437 : cluster [WRN] 2021-02-10T04:32:41.652 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:41.652 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: cluster 2021-02-10T04:32:41.652 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: 2021-02-10T04:32:41.353882+0000 osd.1 ( 2021-02-10T04:32:41.652 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: osd.1) 438 : cluster [WRN] slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:41.652 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:32:41.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: 04:32:41.353888+0000 osd.1 (osd.1 2021-02-10T04:32:41.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: ) 439 : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:41.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:32:41.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: T04:32:41.353894+0000 osd.1 (osd. 2021-02-10T04:32:41.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: 1) 440 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:41.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:41.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: -10T04:32:41.353900+0000 osd.1 (osd 2021-02-10T04:32:41.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: .1) 441 : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:41.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:41.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: 02-10T04:32:41.353906+0000 2021-02-10T04:32:41.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: osd.1 (osd.1) 442 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:41.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[29319]: 2021-02-10T04:32:41.654 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: cluster 2021-02-10T04:32:40.482802+0000 2021-02-10T04:32:41.654 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: osd.4 (osd.4) 373 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:41.654 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:41.654 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: 10T04:32:40.482812+0000 osd.4 (osd.4) 374 : cluster 2021-02-10T04:32:41.654 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:41.654 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:32:41.655 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: :32:40.482819+0000 osd.4 (osd.4 2021-02-10T04:32:41.655 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: ) 375 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:41.655 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:41.655 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: 04:32:40.482825+0000 osd.4 (osd.4) 2021-02-10T04:32:41.655 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: 376 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:41.655 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: cluster 2021-02-10T04:32:41.655 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: 2021-02-10T04:32:40.495272+0000 mon.a (mon.0) 1103 : cluster [DBG] osdmap e343: 8 total, 8 up, 8 in 2021-02-10T04:32:41.655 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: cluster 2021-02-10T04:32:41.307823+0000 osd.2 2021-02-10T04:32:41.655 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: (osd.2) 520 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:32:41.655 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: cluster 2021-02-10T04:32:41.307833+0000 osd.2 2021-02-10T04:32:41.655 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: (osd.2) 521 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:32:41.655 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: cluster 2021-02-10T04:32:41.307840+0000 osd.2 (osd. 2021-02-10T04:32:41.656 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: 2) 522 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:32:41.656 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:41.656 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: -02-10T04:32:41.307846+0000 osd.2 (osd.2) 523 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:32:41.656 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: 2021-02-10T04:32:41.656 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: cluster 2021-02-10T04:32:41 2021-02-10T04:32:41.656 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: .307855+0000 osd.2 (osd.2) 524 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:32:41.656 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: cluster 2021-02-10T04:32:41.656 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: 2021-02-10T04:32:41.307860+0000 osd.2 (osd.2) 525 : cluster [WRN] 2021-02-10T04:32:41.657 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:32:41.657 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:41.657 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: 02-10T04:32:41.307866+0000 osd.2 (osd.2) 526 2021-02-10T04:32:41.657 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:32:41.657 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: cluster 2021-02-10T04:32:41.657 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: 2021-02-10T04:32:41.307871+0000 osd.2 (osd.2) 527 : 2021-02-10T04:32:41.657 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:32:41.657 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:41.657 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: -02-10T04:32:41.353873+0000 osd.1 (osd.1) 437 : cluster [WRN] slow request osd_op(client.14817.0:1683 7.b 7:d34c66fc:::smithi08575397-14:head [copy-from ver 217 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.173983+0000 currently started 2021-02-10T04:32:41.657 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: cluster 2021-02-10T04:32:41. 2021-02-10T04:32:41.657 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: 353882+0000 osd.1 (osd.1) 438 : cluster [WRN] 2021-02-10T04:32:41.657 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1684 7.b 7:d34c66fc:::smithi08575397-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.174396+0000 currently delayed 2021-02-10T04:32:41.658 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:32:41.658 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: 32:41.353888+0000 osd.1 (osd.1) 439 2021-02-10T04:32:41.658 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1777 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 0 in=55b] snapc 7b=[79,78] ondisk+write+known_if_redirected e223) initiated 2021-02-10T04:30:39.230682+0000 currently started 2021-02-10T04:32:41.658 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: cluster 2021-02-10T04:32:41 2021-02-10T04:32:41.658 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: .353894+0000 osd.1 (osd.1) 440 : cluster [WRN] slow request osd_op(client.14817.0:1778 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e223) initiated 2021-02-10T04:30:39.230812+0000 currently delayed 2021-02-10T04:32:41.658 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:41.659 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: 10T04:32:41.353900+0000 osd.1 (osd.1) 441 2021-02-10T04:32:41.659 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:650 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 29 in=55b] snapc 24=[24,20,1e,1d,1c,19,17,15] ondisk+write+known_if_redirected e134) initiated 2021-02-10T04:29:10.111666+0000 currently started 2021-02-10T04:32:41.659 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:41.659 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: 02-10T04:32:41.353906+0000 osd.1 ( 2021-02-10T04:32:41.659 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[35961]: osd.1) 442 : cluster [WRN] slow request osd_op(client.14817.0:651 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e134) initiated 2021-02-10T04:29:10.112136+0000 currently delayed 2021-02-10T04:32:41.694 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:32:41.694 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 20, 2021-02-10T04:32:41.694 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 600, 2021-02-10T04:32:41.694 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [ 2021-02-10T04:32:41.694 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.694 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:53 4.2 4:4347d321:::bucket.sync-source-hints.:head [call,read 0~0] snapc 0=[] ondisk+read+known_if_redirected e56)", 2021-02-10T04:32:41.695 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:25:48.995080+0000", 2021-02-10T04:32:41.695 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 412.696341555, 2021-02-10T04:32:41.695 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.011610191000000001, 2021-02-10T04:32:41.695 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:32:41.695 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:32:41.695 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:32:41.695 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:32:41.695 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:32:41.695 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 53 2021-02-10T04:32:41.695 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.695 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:32:41.695 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.696 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:32:41.696 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:48.995080+0000", 2021-02-10T04:32:41.696 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:32:41.696 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.696 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.696 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:32:41.696 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:48.995080+0000", 2021-02-10T04:32:41.696 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.1201000000000001e-05 2021-02-10T04:32:41.696 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.696 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.696 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:32:41.696 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:48.995121+0000", 2021-02-10T04:32:41.696 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.5220000000000001e-06 2021-02-10T04:32:41.696 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.697 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.697 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:32:41.697 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:48.995129+0000", 2021-02-10T04:32:41.697 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.3692e-05 2021-02-10T04:32:41.697 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.697 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.697 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:32:41.697 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:48.995142+0000", 2021-02-10T04:32:41.697 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.9789999999999996e-06 2021-02-10T04:32:41.697 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.697 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.697 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:32:41.697 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:48.995149+0000", 2021-02-10T04:32:41.698 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00279948 2021-02-10T04:32:41.698 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.698 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.698 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:41.698 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:48.997949+0000", 2021-02-10T04:32:41.698 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.1929999999999993e-06 2021-02-10T04:32:41.698 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.698 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.698 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for peered", 2021-02-10T04:32:41.698 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:48.997957+0000", 2021-02-10T04:32:41.698 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0086204409999999995 2021-02-10T04:32:41.698 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.698 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.699 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:41.699 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:49.006577+0000", 2021-02-10T04:32:41.699 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00011268300000000001 2021-02-10T04:32:41.699 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.699 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.699 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:32:41.699 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:25:49.006690+0000", 2021-02-10T04:32:41.699 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.011610191000000001 2021-02-10T04:32:41.699 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.699 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:32:41.699 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.699 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.699 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.700 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:346 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 6] snapc a=[] ondisk+write+known_if_redirected e108)", 2021-02-10T04:32:41.700 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:28:44.036688+0000", 2021-02-10T04:32:41.700 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 237.65473378499999, 2021-02-10T04:32:41.700 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.023712008, 2021-02-10T04:32:41.700 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:32:41.700 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:32:41.700 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:32:41.700 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:32:41.700 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:32:41.700 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 346 2021-02-10T04:32:41.700 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.700 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:32:41.700 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.701 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:32:41.701 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.036688+0000", 2021-02-10T04:32:41.701 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:32:41.701 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.701 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.701 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:32:41.701 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.036688+0000", 2021-02-10T04:32:41.701 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.5324999999999997e-05 2021-02-10T04:32:41.701 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.701 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.701 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:32:41.701 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.036733+0000", 2021-02-10T04:32:41.701 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.9160000000000001e-06 2021-02-10T04:32:41.701 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.702 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.702 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:32:41.702 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.036741+0000", 2021-02-10T04:32:41.702 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.3095000000000001e-05 2021-02-10T04:32:41.702 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.702 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.702 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:32:41.702 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.036754+0000", 2021-02-10T04:32:41.702 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.1219999999999997e-06 2021-02-10T04:32:41.702 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.702 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.702 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:32:41.702 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.036761+0000", 2021-02-10T04:32:41.703 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.2971000000000003e-05 2021-02-10T04:32:41.703 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.703 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.703 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:41.703 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.036834+0000", 2021-02-10T04:32:41.703 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.8089999999999999e-06 2021-02-10T04:32:41.703 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.703 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.703 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op must wait for map", 2021-02-10T04:32:41.703 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.036843+0000", 2021-02-10T04:32:41.703 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0033220150000000002 2021-02-10T04:32:41.704 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.704 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.704 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:41.704 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.040165+0000", 2021-02-10T04:32:41.704 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00010351100000000001 2021-02-10T04:32:41.704 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.704 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.704 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:32:41.704 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.040268+0000", 2021-02-10T04:32:41.704 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.004247664 2021-02-10T04:32:41.704 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.704 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.705 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for subops from 5,6", 2021-02-10T04:32:41.705 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.044516+0000", 2021-02-10T04:32:41.705 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0040944149999999997 2021-02-10T04:32:41.705 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.705 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.705 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op_commit", 2021-02-10T04:32:41.705 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.048611+0000", 2021-02-10T04:32:41.705 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0045914320000000003 2021-02-10T04:32:41.705 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.705 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.705 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:32:41.705 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.053202+0000", 2021-02-10T04:32:41.706 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0070608570000000002 2021-02-10T04:32:41.706 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.706 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.706 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:32:41.706 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.060263+0000", 2021-02-10T04:32:41.706 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.1599000000000003e-05 2021-02-10T04:32:41.706 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.706 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.706 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:32:41.706 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.060334+0000", 2021-02-10T04:32:41.706 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.5277000000000005e-05 2021-02-10T04:32:41.706 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.706 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.707 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:32:41.707 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.060400+0000", 2021-02-10T04:32:41.707 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.023712008 2021-02-10T04:32:41.707 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.707 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:32:41.707 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.707 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.707 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.707 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:347 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e108)", 2021-02-10T04:32:41.707 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:28:44.036854+0000", 2021-02-10T04:32:41.707 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 237.65456728000001, 2021-02-10T04:32:41.708 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.023765821999999999, 2021-02-10T04:32:41.708 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:32:41.708 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:32:41.708 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:32:41.708 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:32:41.708 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:32:41.708 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 347 2021-02-10T04:32:41.708 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.708 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:32:41.708 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.708 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:32:41.709 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.036854+0000", 2021-02-10T04:32:41.709 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:32:41.709 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.709 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.709 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:32:41.709 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.036854+0000", 2021-02-10T04:32:41.709 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000107419 2021-02-10T04:32:41.709 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.709 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.709 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:32:41.709 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.036962+0000", 2021-02-10T04:32:41.709 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.6640000000000001e-06 2021-02-10T04:32:41.710 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.710 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.710 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:32:41.710 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.036968+0000", 2021-02-10T04:32:41.710 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1936000000000001e-05 2021-02-10T04:32:41.710 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.710 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.710 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:32:41.710 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.036980+0000", 2021-02-10T04:32:41.710 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.3469999999999998e-06 2021-02-10T04:32:41.710 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.710 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.711 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:32:41.711 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.036986+0000", 2021-02-10T04:32:41.711 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.2969999999999997e-05 2021-02-10T04:32:41.711 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.711 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.711 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:41.711 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.037029+0000", 2021-02-10T04:32:41.711 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.1560000000000001e-06 2021-02-10T04:32:41.711 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.711 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.711 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting_for_map not empty", 2021-02-10T04:32:41.711 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.037038+0000", 2021-02-10T04:32:41.711 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0034051789999999999 2021-02-10T04:32:41.712 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.712 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.712 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:41.712 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.040443+0000", 2021-02-10T04:32:41.712 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9762000000000002e-05 2021-02-10T04:32:41.712 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.712 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.712 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for blocked object", 2021-02-10T04:32:41.712 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.040493+0000", 2021-02-10T04:32:41.712 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0076417509999999996 2021-02-10T04:32:41.712 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.712 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.712 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:41.713 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.048134+0000", 2021-02-10T04:32:41.713 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00018435899999999999 2021-02-10T04:32:41.713 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.713 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.713 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for rw locks", 2021-02-10T04:32:41.713 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.048319+0000", 2021-02-10T04:32:41.713 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.01211023 2021-02-10T04:32:41.713 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.713 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.713 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:41.713 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.060429+0000", 2021-02-10T04:32:41.713 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.7798000000000002e-05 2021-02-10T04:32:41.713 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.714 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.714 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:32:41.714 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.060517+0000", 2021-02-10T04:32:41.714 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000103251 2021-02-10T04:32:41.714 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.714 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.714 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:32:41.714 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:28:44.060620+0000", 2021-02-10T04:32:41.714 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.023765821999999999 2021-02-10T04:32:41.714 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.714 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:32:41.714 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.714 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.715 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.715 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(client.14817.0:539 7.d e127/93 7:b1749140:::smithi08575397-19:head v 127'31, mlcod=121'28)", 2021-02-10T04:32:41.715 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:29:03.051757+0000", 2021-02-10T04:32:41.715 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 218.63966405599999, 2021-02-10T04:32:41.715 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.062654109999999999, 2021-02-10T04:32:41.715 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:32:41.715 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:32:41.715 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:32:41.715 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.715 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:32:41.715 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:03.051757+0000", 2021-02-10T04:32:41.715 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:32:41.715 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.716 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.716 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:32:41.716 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:03.051757+0000", 2021-02-10T04:32:41.716 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.027609313999999999 2021-02-10T04:32:41.716 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.716 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.716 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:32:41.716 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:03.079367+0000", 2021-02-10T04:32:41.716 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0942e-05 2021-02-10T04:32:41.716 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.716 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.716 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:32:41.716 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:03.079378+0000", 2021-02-10T04:32:41.716 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1161999999999999e-05 2021-02-10T04:32:41.717 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.717 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.717 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:32:41.717 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:03.079389+0000", 2021-02-10T04:32:41.717 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.0949999999999996e-06 2021-02-10T04:32:41.717 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.717 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.717 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:32:41.717 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:03.079397+0000", 2021-02-10T04:32:41.717 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.8677999999999995e-05 2021-02-10T04:32:41.717 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.717 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.717 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:41.718 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:03.079466+0000", 2021-02-10T04:32:41.718 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.9949000000000001e-05 2021-02-10T04:32:41.718 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.718 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.718 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:32:41.718 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:03.079496+0000", 2021-02-10T04:32:41.718 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.034850301 2021-02-10T04:32:41.718 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.718 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.718 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:32:41.718 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:03.114346+0000", 2021-02-10T04:32:41.718 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.5668999999999998e-05 2021-02-10T04:32:41.718 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.719 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.719 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:32:41.719 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:03.114412+0000", 2021-02-10T04:32:41.719 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.062654109999999999 2021-02-10T04:32:41.719 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.719 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:32:41.719 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.719 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.719 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.719 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(client.14817.0:623 7.0 e132/93 7:03f22018:::smithi08575397-13:head v 132'32, mlcod=132'30)", 2021-02-10T04:32:41.719 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:29:08.057643+0000", 2021-02-10T04:32:41.719 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 213.63377881299999, 2021-02-10T04:32:41.719 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.034478448000000002, 2021-02-10T04:32:41.720 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:32:41.720 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:32:41.720 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:32:41.720 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.720 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:32:41.720 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.057643+0000", 2021-02-10T04:32:41.720 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:32:41.720 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.720 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.720 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:32:41.720 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.057643+0000", 2021-02-10T04:32:41.720 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00088847800000000005 2021-02-10T04:32:41.720 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.721 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.721 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:32:41.721 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.058531+0000", 2021-02-10T04:32:41.721 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0716e-05 2021-02-10T04:32:41.721 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.721 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.721 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:32:41.721 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.058542+0000", 2021-02-10T04:32:41.721 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1151999999999999e-05 2021-02-10T04:32:41.721 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.721 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.721 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:32:41.721 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.058553+0000", 2021-02-10T04:32:41.722 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.6490000000000008e-06 2021-02-10T04:32:41.722 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.722 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.722 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:32:41.722 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.058561+0000", 2021-02-10T04:32:41.722 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00040535400000000002 2021-02-10T04:32:41.722 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.722 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.722 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:41.722 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.058966+0000", 2021-02-10T04:32:41.722 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.0599999999999998e-05 2021-02-10T04:32:41.722 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.723 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.723 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:32:41.723 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.058997+0000", 2021-02-10T04:32:41.723 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.033070374999999999 2021-02-10T04:32:41.723 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.723 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.723 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:32:41.723 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.092067+0000", 2021-02-10T04:32:41.723 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.4123999999999999e-05 2021-02-10T04:32:41.723 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.724 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.724 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:32:41.724 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.092121+0000", 2021-02-10T04:32:41.724 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.034478448000000002 2021-02-10T04:32:41.724 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.724 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:32:41.724 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.724 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.724 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.724 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(client.14817.0:624 7.0 e132/93 7:03f22018:::smithi08575397-13:head v 132'33, mlcod=132'30)", 2021-02-10T04:32:41.724 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:29:08.058621+0000", 2021-02-10T04:32:41.724 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 213.63280091999999, 2021-02-10T04:32:41.724 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.033535599999999999, 2021-02-10T04:32:41.725 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:32:41.725 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:32:41.725 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:32:41.725 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.725 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:32:41.725 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.058621+0000", 2021-02-10T04:32:41.725 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:32:41.725 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.725 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.725 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:32:41.725 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.058621+0000", 2021-02-10T04:32:41.725 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.3763000000000001e-05 2021-02-10T04:32:41.725 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.726 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.726 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:32:41.726 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.058664+0000", 2021-02-10T04:32:41.726 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.7780000000000002e-06 2021-02-10T04:32:41.726 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.726 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.726 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:32:41.726 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.058671+0000", 2021-02-10T04:32:41.726 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0132999999999999e-05 2021-02-10T04:32:41.726 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.726 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.727 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:32:41.727 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.058681+0000", 2021-02-10T04:32:41.727 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0100000000000003e-06 2021-02-10T04:32:41.727 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.727 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.727 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:32:41.727 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.058686+0000", 2021-02-10T04:32:41.727 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00078517400000000003 2021-02-10T04:32:41.727 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.728 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.728 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:41.728 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.059471+0000", 2021-02-10T04:32:41.728 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.3003e-05 2021-02-10T04:32:41.728 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.728 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.728 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:32:41.728 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.059494+0000", 2021-02-10T04:32:41.728 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.032631782999999998 2021-02-10T04:32:41.728 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.729 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.729 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:32:41.729 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.092126+0000", 2021-02-10T04:32:41.729 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.9955999999999999e-05 2021-02-10T04:32:41.729 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.729 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.729 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:32:41.729 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.092156+0000", 2021-02-10T04:32:41.729 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.033535599999999999 2021-02-10T04:32:41.729 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.729 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:32:41.729 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.730 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.730 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.730 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(client.14817.0:618 7.0 e132/93 7:06aed406:::smithi08575397-5:head v 132'35, mlcod=132'32)", 2021-02-10T04:32:41.730 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:29:08.842985+0000", 2021-02-10T04:32:41.730 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 212.848436835, 2021-02-10T04:32:41.730 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.014410935, 2021-02-10T04:32:41.730 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:32:41.731 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:32:41.731 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:32:41.731 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.731 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:32:41.731 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.842985+0000", 2021-02-10T04:32:41.731 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:32:41.731 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.731 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.731 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:32:41.731 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.842985+0000", 2021-02-10T04:32:41.732 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0083104110000000002 2021-02-10T04:32:41.732 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.732 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.732 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:32:41.732 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.851295+0000", 2021-02-10T04:32:41.732 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.7360999999999999e-05 2021-02-10T04:32:41.732 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.732 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.732 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:32:41.732 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.851312+0000", 2021-02-10T04:32:41.732 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.9953e-05 2021-02-10T04:32:41.732 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.733 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.733 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:32:41.733 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.851332+0000", 2021-02-10T04:32:41.733 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0416e-05 2021-02-10T04:32:41.733 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.733 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.733 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:32:41.733 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.851343+0000", 2021-02-10T04:32:41.733 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000100407 2021-02-10T04:32:41.733 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.733 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.733 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:41.733 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.851443+0000", 2021-02-10T04:32:41.734 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.3171999999999997e-05 2021-02-10T04:32:41.734 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.734 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.734 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:32:41.734 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.851516+0000", 2021-02-10T04:32:41.734 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0058268210000000003 2021-02-10T04:32:41.734 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.734 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.734 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:32:41.734 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.857343+0000", 2021-02-10T04:32:41.734 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.2394e-05 2021-02-10T04:32:41.734 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.734 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.735 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:32:41.735 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:08.857396+0000", 2021-02-10T04:32:41.735 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.014410935 2021-02-10T04:32:41.735 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.735 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:32:41.735 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.735 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.735 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.735 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(osd.5.0:84 7.0 e134/93 7:03f22018:::smithi08575397-13:23 v 134'37, mlcod=132'33)", 2021-02-10T04:32:41.735 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:29:10.042859+0000", 2021-02-10T04:32:41.735 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 211.64856261400001, 2021-02-10T04:32:41.735 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.067695775999999999, 2021-02-10T04:32:41.735 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:32:41.736 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:32:41.736 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:32:41.736 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.736 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:32:41.736 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.042859+0000", 2021-02-10T04:32:41.736 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:32:41.736 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.736 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.736 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:32:41.736 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.042859+0000", 2021-02-10T04:32:41.736 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.0001999999999999e-05 2021-02-10T04:32:41.736 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.736 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.736 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:32:41.737 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.042919+0000", 2021-02-10T04:32:41.737 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.0090000000000001e-06 2021-02-10T04:32:41.737 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.737 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.737 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:32:41.737 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.042926+0000", 2021-02-10T04:32:41.737 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0489000000000001e-05 2021-02-10T04:32:41.737 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.737 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.737 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:32:41.737 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.042936+0000", 2021-02-10T04:32:41.737 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2853999999999999e-05 2021-02-10T04:32:41.737 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.738 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.738 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:32:41.738 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.042949+0000", 2021-02-10T04:32:41.738 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00025871500000000002 2021-02-10T04:32:41.738 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.738 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.738 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:41.738 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.043208+0000", 2021-02-10T04:32:41.738 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.3760999999999998e-05 2021-02-10T04:32:41.738 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.738 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.738 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:32:41.738 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.043232+0000", 2021-02-10T04:32:41.739 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.067238027000000006 2021-02-10T04:32:41.739 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.739 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.739 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:32:41.739 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.110470+0000", 2021-02-10T04:32:41.739 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.4919000000000005e-05 2021-02-10T04:32:41.739 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.739 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.739 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:32:41.739 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:10.110555+0000", 2021-02-10T04:32:41.739 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.067695775999999999 2021-02-10T04:32:41.739 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.739 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:32:41.740 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.740 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.740 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.740 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDScrubReserve(6.1d GRANT e148)", 2021-02-10T04:32:41.740 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:29:25.020701+0000", 2021-02-10T04:32:41.740 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 196.67072091099999, 2021-02-10T04:32:41.740 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.052446732000000003, 2021-02-10T04:32:41.740 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:32:41.740 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:32:41.740 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:32:41.740 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.740 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:32:41.740 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:25.020701+0000", 2021-02-10T04:32:41.741 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:32:41.741 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.741 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.741 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:32:41.741 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:25.020701+0000", 2021-02-10T04:32:41.741 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.052345045 2021-02-10T04:32:41.741 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.741 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.741 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:32:41.741 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:25.073046+0000", 2021-02-10T04:32:41.741 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.5760000000000002e-06 2021-02-10T04:32:41.741 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.742 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.742 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:32:41.742 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:25.073052+0000", 2021-02-10T04:32:41.742 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0208e-05 2021-02-10T04:32:41.742 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.742 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.742 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:32:41.742 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:25.073062+0000", 2021-02-10T04:32:41.742 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.1629999999999999e-06 2021-02-10T04:32:41.742 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.742 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.742 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:32:41.742 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:25.073070+0000", 2021-02-10T04:32:41.743 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.8606000000000003e-05 2021-02-10T04:32:41.743 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.743 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.743 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:41.743 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:25.073108+0000", 2021-02-10T04:32:41.743 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2614e-05 2021-02-10T04:32:41.743 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.743 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.743 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:32:41.743 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:25.073121+0000", 2021-02-10T04:32:41.743 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.4670000000000003e-06 2021-02-10T04:32:41.743 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.743 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.744 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:32:41.744 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:25.073126+0000", 2021-02-10T04:32:41.744 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.1053e-05 2021-02-10T04:32:41.744 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.744 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.744 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:32:41.744 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:25.073147+0000", 2021-02-10T04:32:41.744 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.052446732000000003 2021-02-10T04:32:41.744 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.744 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:32:41.744 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.744 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.744 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.745 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:876 7.b 7:d34c66fc:::smithi08575397-14:head [write 786594~779712] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153)", 2021-02-10T04:32:41.745 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:29:29.076946+0000", 2021-02-10T04:32:41.745 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 192.61447579200001, 2021-02-10T04:32:41.745 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.011178517000000001, 2021-02-10T04:32:41.745 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:32:41.745 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:32:41.745 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:32:41.745 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:32:41.745 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:32:41.745 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 876 2021-02-10T04:32:41.745 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.745 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:32:41.745 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.745 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:32:41.746 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:29.076946+0000", 2021-02-10T04:32:41.746 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:32:41.746 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.746 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.746 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:32:41.746 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:29.076946+0000", 2021-02-10T04:32:41.746 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00041959499999999998 2021-02-10T04:32:41.746 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.746 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.746 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:32:41.746 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:29.077365+0000", 2021-02-10T04:32:41.746 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.4769999999999992e-06 2021-02-10T04:32:41.747 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.747 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.747 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:32:41.747 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:29.077375+0000", 2021-02-10T04:32:41.747 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.3621999999999999e-05 2021-02-10T04:32:41.747 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.747 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.747 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:32:41.747 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:29.077388+0000", 2021-02-10T04:32:41.747 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.2770000000000005e-06 2021-02-10T04:32:41.747 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.747 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.747 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:32:41.748 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:29.077397+0000", 2021-02-10T04:32:41.748 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.0112000000000003e-05 2021-02-10T04:32:41.748 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.748 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.748 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:41.748 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:29.077457+0000", 2021-02-10T04:32:41.748 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000161805 2021-02-10T04:32:41.748 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.748 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.748 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:32:41.748 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:29.077619+0000", 2021-02-10T04:32:41.748 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000267614 2021-02-10T04:32:41.748 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.748 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.749 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for subops from 5,6", 2021-02-10T04:32:41.749 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:29.077886+0000", 2021-02-10T04:32:41.749 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003142288 2021-02-10T04:32:41.749 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.749 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.749 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op_commit", 2021-02-10T04:32:41.749 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:29.081028+0000", 2021-02-10T04:32:41.749 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00071196300000000003 2021-02-10T04:32:41.749 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.749 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.749 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:32:41.749 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:29.081740+0000", 2021-02-10T04:32:41.750 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0062412350000000004 2021-02-10T04:32:41.750 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.750 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.750 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:32:41.750 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:29.087982+0000", 2021-02-10T04:32:41.750 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.3508000000000002e-05 2021-02-10T04:32:41.750 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.750 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.750 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:32:41.750 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:29.088055+0000", 2021-02-10T04:32:41.750 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.9021000000000002e-05 2021-02-10T04:32:41.750 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.750 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.751 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:32:41.751 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:29.088124+0000", 2021-02-10T04:32:41.751 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.011178517000000001 2021-02-10T04:32:41.751 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.751 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:32:41.751 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.751 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.751 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.751 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:944 7.b 7:d39fd0bb:::smithi08575397-9:head [delete] snapc 3e=[3e,3d,2e] ondisk+write+peerstat_old+known_if_redirected e159)", 2021-02-10T04:32:41.751 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:29:35.158317+0000", 2021-02-10T04:32:41.751 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 186.53310495900001, 2021-02-10T04:32:41.751 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.014617342, 2021-02-10T04:32:41.751 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:32:41.752 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:32:41.752 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:32:41.752 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:32:41.752 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:32:41.752 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 944 2021-02-10T04:32:41.752 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.752 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:32:41.752 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.752 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:32:41.752 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:35.158317+0000", 2021-02-10T04:32:41.752 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:32:41.752 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.752 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.753 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:32:41.753 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:35.158317+0000", 2021-02-10T04:32:41.753 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.4279000000000002e-05 2021-02-10T04:32:41.753 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.753 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.753 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:32:41.753 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:35.158351+0000", 2021-02-10T04:32:41.753 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.1519999999999998e-06 2021-02-10T04:32:41.753 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.753 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.753 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:32:41.753 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:35.158360+0000", 2021-02-10T04:32:41.753 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.9511999999999999e-05 2021-02-10T04:32:41.753 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.754 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.754 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:32:41.754 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:35.158379+0000", 2021-02-10T04:32:41.754 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.3729999999999999e-06 2021-02-10T04:32:41.754 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.754 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.754 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:32:41.754 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:35.158389+0000", 2021-02-10T04:32:41.754 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.8179e-05 2021-02-10T04:32:41.754 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.754 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.754 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:41.754 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:35.158447+0000", 2021-02-10T04:32:41.755 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0023e-05 2021-02-10T04:32:41.755 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.755 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.755 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op must wait for map", 2021-02-10T04:32:41.755 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:35.158457+0000", 2021-02-10T04:32:41.755 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.011435097999999999 2021-02-10T04:32:41.755 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.755 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.755 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:41.755 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:35.169892+0000", 2021-02-10T04:32:41.755 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000117889 2021-02-10T04:32:41.755 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.755 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.756 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:32:41.756 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:35.170010+0000", 2021-02-10T04:32:41.756 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00024910999999999998 2021-02-10T04:32:41.756 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.756 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.756 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for subops from 5,6", 2021-02-10T04:32:41.756 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:35.170259+0000", 2021-02-10T04:32:41.756 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00088030499999999995 2021-02-10T04:32:41.756 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.756 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.756 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op_commit", 2021-02-10T04:32:41.756 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:35.171139+0000", 2021-02-10T04:32:41.757 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00061677599999999998 2021-02-10T04:32:41.757 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.757 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.757 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:32:41.757 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:35.171756+0000", 2021-02-10T04:32:41.757 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0010495560000000001 2021-02-10T04:32:41.757 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.758 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.758 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:32:41.758 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:35.172806+0000", 2021-02-10T04:32:41.758 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.4785999999999999e-05 2021-02-10T04:32:41.758 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.758 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.758 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:32:41.758 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:35.172871+0000", 2021-02-10T04:32:41.758 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.3304000000000002e-05 2021-02-10T04:32:41.758 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.758 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.758 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:32:41.759 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:35.172934+0000", 2021-02-10T04:32:41.759 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.014617342 2021-02-10T04:32:41.759 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.759 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:32:41.759 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.759 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.759 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.759 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(client.14817.0:1186 7.0 e176/93 7:06aed406:::smithi08575397-5:head v 176'96, mlcod=174'92)", 2021-02-10T04:32:41.759 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:29:52.225321+0000", 2021-02-10T04:32:41.759 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 169.46610021000001, 2021-02-10T04:32:41.759 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.013112564, 2021-02-10T04:32:41.759 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:32:41.760 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:32:41.760 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:32:41.760 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.760 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:32:41.760 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:52.225321+0000", 2021-02-10T04:32:41.760 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:32:41.760 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.760 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.761 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:32:41.761 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:52.225321+0000", 2021-02-10T04:32:41.761 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0047210339999999998 2021-02-10T04:32:41.761 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.761 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.761 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:32:41.762 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:52.230042+0000", 2021-02-10T04:32:41.762 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.2260000000000005e-06 2021-02-10T04:32:41.762 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.762 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.762 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:32:41.762 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:52.230051+0000", 2021-02-10T04:32:41.762 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.5892e-05 2021-02-10T04:32:41.762 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.762 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.762 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:32:41.762 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:52.230066+0000", 2021-02-10T04:32:41.763 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.798e-06 2021-02-10T04:32:41.763 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.763 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.763 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:32:41.763 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:52.230076+0000", 2021-02-10T04:32:41.763 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.8918999999999994e-05 2021-02-10T04:32:41.763 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.763 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.763 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:41.763 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:52.230155+0000", 2021-02-10T04:32:41.763 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.2478000000000003e-05 2021-02-10T04:32:41.763 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.763 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.764 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:32:41.764 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:52.230188+0000", 2021-02-10T04:32:41.764 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0081708400000000004 2021-02-10T04:32:41.764 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.764 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.764 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:32:41.764 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:52.238358+0000", 2021-02-10T04:32:41.764 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.5376999999999993e-05 2021-02-10T04:32:41.764 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.764 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.764 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:32:41.764 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:52.238434+0000", 2021-02-10T04:32:41.764 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.013112564 2021-02-10T04:32:41.765 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.765 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:32:41.765 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.765 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.765 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.765 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(osd.5.0:151 7.0 e211/93 7:06aed406:::smithi08575397-5:6f v 211'141, mlcod=210'138)", 2021-02-10T04:32:41.765 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:30:28.105185+0000", 2021-02-10T04:32:41.765 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 133.586236595, 2021-02-10T04:32:41.765 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.031950218000000002, 2021-02-10T04:32:41.765 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:32:41.765 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:32:41.765 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:32:41.765 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.766 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:32:41.766 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.105185+0000", 2021-02-10T04:32:41.766 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:32:41.766 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.766 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.766 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:32:41.766 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.105185+0000", 2021-02-10T04:32:41.766 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.2187000000000001e-05 2021-02-10T04:32:41.766 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.766 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.766 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:32:41.766 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.105227+0000", 2021-02-10T04:32:41.766 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.7720000000000002e-06 2021-02-10T04:32:41.767 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.767 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.767 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:32:41.767 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.105232+0000", 2021-02-10T04:32:41.767 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.4479999999999999e-06 2021-02-10T04:32:41.767 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.767 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.767 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:32:41.767 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.105238+0000", 2021-02-10T04:32:41.767 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0003000000000001e-05 2021-02-10T04:32:41.767 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.767 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.767 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:32:41.768 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.105248+0000", 2021-02-10T04:32:41.768 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4032999999999997e-05 2021-02-10T04:32:41.768 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.768 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.768 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:41.768 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.105292+0000", 2021-02-10T04:32:41.768 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.7489000000000001e-05 2021-02-10T04:32:41.768 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.768 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.768 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:32:41.768 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.105320+0000", 2021-02-10T04:32:41.768 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.031758833 2021-02-10T04:32:41.769 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.769 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.769 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:32:41.769 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.137079+0000", 2021-02-10T04:32:41.769 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.6453000000000003e-05 2021-02-10T04:32:41.769 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.769 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.769 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:32:41.770 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.137135+0000", 2021-02-10T04:32:41.770 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.031950218000000002 2021-02-10T04:32:41.771 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.771 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:32:41.771 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.771 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.771 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.771 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(osd.5.0:152 7.0 e211/93 7:03599452:::smithi08575397-34:6d v 211'143, mlcod=210'138)", 2021-02-10T04:32:41.771 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:30:28.105496+0000", 2021-02-10T04:32:41.771 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 133.58592508999999, 2021-02-10T04:32:41.771 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.031872533000000002, 2021-02-10T04:32:41.771 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:32:41.771 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:32:41.771 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:32:41.771 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.772 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:32:41.772 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.105496+0000", 2021-02-10T04:32:41.772 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:32:41.772 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.772 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.772 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:32:41.772 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.105496+0000", 2021-02-10T04:32:41.772 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.8788999999999997e-05 2021-02-10T04:32:41.772 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.772 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.772 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:32:41.772 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.105535+0000", 2021-02-10T04:32:41.772 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.241e-06 2021-02-10T04:32:41.773 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.773 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.773 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:32:41.773 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.105541+0000", 2021-02-10T04:32:41.773 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.4960000000000002e-06 2021-02-10T04:32:41.773 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.773 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.773 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:32:41.773 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.105551+0000", 2021-02-10T04:32:41.773 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.091e-06 2021-02-10T04:32:41.773 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.773 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.773 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:32:41.774 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.105556+0000", 2021-02-10T04:32:41.774 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000103941 2021-02-10T04:32:41.774 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.774 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.774 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:41.775 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.105660+0000", 2021-02-10T04:32:41.775 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.3876999999999999e-05 2021-02-10T04:32:41.775 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.775 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.775 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:32:41.776 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.105684+0000", 2021-02-10T04:32:41.776 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.031639569999999999 2021-02-10T04:32:41.776 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.776 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.776 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:32:41.776 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.137323+0000", 2021-02-10T04:32:41.776 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.5528000000000002e-05 2021-02-10T04:32:41.776 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.776 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.776 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:32:41.776 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:28.137369+0000", 2021-02-10T04:32:41.776 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.031872533000000002 2021-02-10T04:32:41.777 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.777 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:32:41.777 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.777 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.777 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.777 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(client.14817.0:1748 7.d e218/93 7:b09ccbba:::smithi08575397-1:head v 218'125, mlcod=218'121)", 2021-02-10T04:32:41.777 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:30:34.474789+0000", 2021-02-10T04:32:41.777 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 127.21663288800001, 2021-02-10T04:32:41.777 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.024088017, 2021-02-10T04:32:41.777 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:32:41.777 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:32:41.777 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:32:41.777 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.778 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:32:41.778 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:34.474789+0000", 2021-02-10T04:32:41.778 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:32:41.778 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.779 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.779 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:32:41.779 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:34.474789+0000", 2021-02-10T04:32:41.779 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.013440969000000001 2021-02-10T04:32:41.779 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.779 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.779 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:32:41.779 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:34.488230+0000", 2021-02-10T04:32:41.779 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.2990000000000002e-06 2021-02-10T04:32:41.779 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.779 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.779 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:32:41.780 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:34.488239+0000", 2021-02-10T04:32:41.780 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1731e-05 2021-02-10T04:32:41.780 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.780 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.780 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:32:41.780 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:34.488251+0000", 2021-02-10T04:32:41.780 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.9469999999999997e-06 2021-02-10T04:32:41.780 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.780 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.780 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:32:41.780 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:34.488258+0000", 2021-02-10T04:32:41.780 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.2067999999999997e-05 2021-02-10T04:32:41.780 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.781 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.781 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:41.781 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:34.488310+0000", 2021-02-10T04:32:41.781 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.9714e-05 2021-02-10T04:32:41.781 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.781 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.781 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:32:41.781 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:34.488339+0000", 2021-02-10T04:32:41.781 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.010445196 2021-02-10T04:32:41.781 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.781 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.781 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:32:41.781 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:34.498785+0000", 2021-02-10T04:32:41.781 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.2093000000000001e-05 2021-02-10T04:32:41.782 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.782 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.782 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:32:41.782 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:34.498877+0000", 2021-02-10T04:32:41.782 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.024088017 2021-02-10T04:32:41.782 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.782 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:32:41.782 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.782 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.782 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.782 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(client.14817.0:1710 7.0 e220/93 7:06aed406:::smithi08575397-5:head v 220'158, mlcod=215'155)", 2021-02-10T04:32:41.782 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:30:36.557104+0000", 2021-02-10T04:32:41.783 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 125.134317255, 2021-02-10T04:32:41.783 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.015306874999999999, 2021-02-10T04:32:41.783 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:32:41.783 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:32:41.783 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:32:41.783 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.783 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:32:41.783 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:36.557104+0000", 2021-02-10T04:32:41.783 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:32:41.783 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.783 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.783 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:32:41.783 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:36.557104+0000", 2021-02-10T04:32:41.784 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0083114060000000003 2021-02-10T04:32:41.784 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.784 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.784 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:32:41.784 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:36.565416+0000", 2021-02-10T04:32:41.784 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0890000000000001e-05 2021-02-10T04:32:41.784 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.784 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.784 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:32:41.784 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:36.565427+0000", 2021-02-10T04:32:41.784 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0063e-05 2021-02-10T04:32:41.785 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.785 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.785 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:32:41.785 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:36.565437+0000", 2021-02-10T04:32:41.785 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.9999999999999999e-06 2021-02-10T04:32:41.785 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.785 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.785 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:32:41.785 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:36.565444+0000", 2021-02-10T04:32:41.785 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.8070000000000004e-05 2021-02-10T04:32:41.785 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.785 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.785 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:41.786 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:36.565512+0000", 2021-02-10T04:32:41.786 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.2602000000000001e-05 2021-02-10T04:32:41.786 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.786 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.786 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:32:41.786 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:36.565534+0000", 2021-02-10T04:32:41.786 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0068175600000000003 2021-02-10T04:32:41.786 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.786 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.786 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:32:41.787 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:36.572352+0000", 2021-02-10T04:32:41.787 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.9283999999999997e-05 2021-02-10T04:32:41.787 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.787 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.787 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:32:41.787 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:36.572411+0000", 2021-02-10T04:32:41.787 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.015306874999999999 2021-02-10T04:32:41.787 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.787 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:32:41.787 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.788 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.788 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.788 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(client.14817.0:2345 7.0 e261/93 7:06aed406:::smithi08575397-5:head v 261'214, mlcod=259'210)", 2021-02-10T04:32:41.788 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:31:18.193032+0000", 2021-02-10T04:32:41.788 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 83.498389148000001, 2021-02-10T04:32:41.788 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.016639226, 2021-02-10T04:32:41.788 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:32:41.788 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:32:41.788 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:32:41.788 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.788 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:32:41.789 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:18.193032+0000", 2021-02-10T04:32:41.789 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:32:41.789 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.789 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.789 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:32:41.789 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:18.193032+0000", 2021-02-10T04:32:41.789 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0083172639999999996 2021-02-10T04:32:41.789 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.789 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.789 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:32:41.789 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:18.201350+0000", 2021-02-10T04:32:41.789 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.258e-05 2021-02-10T04:32:41.789 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.790 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.790 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:32:41.790 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:18.201362+0000", 2021-02-10T04:32:41.790 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.9400000000000001e-05 2021-02-10T04:32:41.790 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.790 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.790 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:32:41.790 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:18.201382+0000", 2021-02-10T04:32:41.790 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.6809999999999993e-06 2021-02-10T04:32:41.790 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.790 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.790 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:32:41.791 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:18.201391+0000", 2021-02-10T04:32:41.791 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.5593999999999995e-05 2021-02-10T04:32:41.791 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.791 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.791 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:41.791 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:18.201467+0000", 2021-02-10T04:32:41.791 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.7004000000000001e-05 2021-02-10T04:32:41.792 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.792 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.792 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:32:41.792 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:18.201494+0000", 2021-02-10T04:32:41.792 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0080988550000000003 2021-02-10T04:32:41.792 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.792 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.793 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:32:41.793 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:18.209593+0000", 2021-02-10T04:32:41.793 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.8848000000000001e-05 2021-02-10T04:32:41.793 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.793 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.793 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:32:41.793 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:18.209672+0000", 2021-02-10T04:32:41.793 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.016639226 2021-02-10T04:32:41.793 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.793 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:32:41.794 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.794 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.794 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.794 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(client.14817.0:2488 7.d e275/93 7:b09ccbba:::smithi08575397-1:head v 275'188, mlcod=272'185)", 2021-02-10T04:32:41.794 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:31:32.754044+0000", 2021-02-10T04:32:41.794 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 68.937377634000001, 2021-02-10T04:32:41.794 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.011158204, 2021-02-10T04:32:41.794 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:32:41.794 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:32:41.794 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:32:41.794 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.794 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:32:41.795 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.754044+0000", 2021-02-10T04:32:41.795 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:32:41.795 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.795 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.795 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:32:41.795 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.754044+0000", 2021-02-10T04:32:41.795 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0063601450000000002 2021-02-10T04:32:41.795 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.796 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.796 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:32:41.796 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.760404+0000", 2021-02-10T04:32:41.796 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2193e-05 2021-02-10T04:32:41.796 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.796 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.796 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:32:41.796 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.760416+0000", 2021-02-10T04:32:41.796 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.4423999999999999e-05 2021-02-10T04:32:41.797 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.797 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.797 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:32:41.797 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.760431+0000", 2021-02-10T04:32:41.797 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.5990000000000008e-06 2021-02-10T04:32:41.797 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.797 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.798 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:32:41.798 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.760440+0000", 2021-02-10T04:32:41.798 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.2494000000000002e-05 2021-02-10T04:32:41.798 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.798 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.798 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:41.798 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.760503+0000", 2021-02-10T04:32:41.798 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.9142999999999999e-05 2021-02-10T04:32:41.799 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.799 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.799 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:32:41.799 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.760532+0000", 2021-02-10T04:32:41.799 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0046011170000000001 2021-02-10T04:32:41.799 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.799 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.799 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:32:41.799 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.765133+0000", 2021-02-10T04:32:41.799 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.9089000000000002e-05 2021-02-10T04:32:41.799 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.799 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.800 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:32:41.800 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:31:32.765202+0000", 2021-02-10T04:32:41.800 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.011158204 2021-02-10T04:32:41.800 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.800 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:32:41.800 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.800 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.800 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.800 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(client.14817.0:3080 7.0 e322/93 7:03f22018:::smithi08575397-13:head v 322'273, mlcod=321'268)", 2021-02-10T04:32:41.800 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:32:19.336255+0000", 2021-02-10T04:32:41.800 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 22.355166038, 2021-02-10T04:32:41.801 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.013647876, 2021-02-10T04:32:41.801 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:32:41.801 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:32:41.801 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:32:41.801 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.801 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:32:41.801 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:32:19.336255+0000", 2021-02-10T04:32:41.801 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:32:41.801 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.801 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.801 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:32:41.801 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:32:19.336255+0000", 2021-02-10T04:32:41.802 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0055211000000000001 2021-02-10T04:32:41.802 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.802 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.802 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:32:41.802 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:32:19.341777+0000", 2021-02-10T04:32:41.802 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0223e-05 2021-02-10T04:32:41.802 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.802 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.802 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:32:41.802 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:32:19.341787+0000", 2021-02-10T04:32:41.802 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.6116e-05 2021-02-10T04:32:41.802 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.802 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.803 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:32:41.803 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:32:19.341803+0000", 2021-02-10T04:32:41.803 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0197000000000001e-05 2021-02-10T04:32:41.803 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.803 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.803 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:32:41.803 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:32:19.341813+0000", 2021-02-10T04:32:41.803 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9631e-05 2021-02-10T04:32:41.803 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.803 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.803 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:41.803 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:32:19.341863+0000", 2021-02-10T04:32:41.803 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.5457e-05 2021-02-10T04:32:41.804 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.804 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.804 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:32:41.804 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:32:19.341888+0000", 2021-02-10T04:32:41.804 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0079580120000000004 2021-02-10T04:32:41.804 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.804 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.804 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:32:41.804 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:32:19.349846+0000", 2021-02-10T04:32:41.804 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.7139999999999998e-05 2021-02-10T04:32:41.804 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.804 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.804 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:32:41.805 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:32:19.349903+0000", 2021-02-10T04:32:41.805 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.013647876 2021-02-10T04:32:41.805 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.805 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:32:41.805 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.805 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.805 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.805 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(client.14817.0:3103 7.d e324/93 7:b09ccbba:::smithi08575397-1:head v 324'231, mlcod=320'227)", 2021-02-10T04:32:41.805 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:32:21.409709+0000", 2021-02-10T04:32:41.805 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 20.281712742, 2021-02-10T04:32:41.805 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.017449042000000001, 2021-02-10T04:32:41.805 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:32:41.805 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:32:41.806 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:32:41.806 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.806 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:32:41.806 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:32:21.409709+0000", 2021-02-10T04:32:41.806 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:32:41.806 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.806 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.806 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:32:41.806 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:32:21.409709+0000", 2021-02-10T04:32:41.806 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0065394229999999999 2021-02-10T04:32:41.806 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.806 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.806 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:32:41.807 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:32:21.416248+0000", 2021-02-10T04:32:41.807 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.3409e-05 2021-02-10T04:32:41.807 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.807 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.807 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:32:41.807 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:32:21.416262+0000", 2021-02-10T04:32:41.807 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.3980000000000001e-05 2021-02-10T04:32:41.807 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.807 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.807 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:32:41.807 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:32:21.416286+0000", 2021-02-10T04:32:41.807 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0781000000000001e-05 2021-02-10T04:32:41.807 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.808 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.808 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:32:41.808 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:32:21.416296+0000", 2021-02-10T04:32:41.808 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.1499000000000002e-05 2021-02-10T04:32:41.808 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.808 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.808 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:32:41.808 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:32:21.416358+0000", 2021-02-10T04:32:41.808 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.5850000000000002e-05 2021-02-10T04:32:41.808 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.808 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.808 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:32:41.808 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:32:21.416384+0000", 2021-02-10T04:32:41.808 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.01070492 2021-02-10T04:32:41.809 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.809 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.809 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:32:41.809 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:32:21.427089+0000", 2021-02-10T04:32:41.809 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.9179999999999998e-05 2021-02-10T04:32:41.809 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:32:41.809 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:32:41.809 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:32:41.809 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:32:21.427158+0000", 2021-02-10T04:32:41.809 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.017449042000000001 2021-02-10T04:32:41.809 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.809 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:32:41.809 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.810 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:32:41.810 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:32:41.810 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:32:41.838 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:32:41.941 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:41 smithi085 conmon[52836]: debug 2021-02-10T04:32:41.809+0000 7fba73fb6700 -1 osd.3 pg_epoch: 344 pg[4.14( v 63'26 (0'0,63'26] local-lis/les=71/72 n=11 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,2,7] r=0 lpr=71 crt=63'26 lcod 63'25 mlcod 0'0 active+clean] scrubber pg(4.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55fec0659ce0) 2021-02-10T04:32:41.943 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:32:41.964 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:32:41.982 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:41.995 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:32:42.028 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:32:42.063 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:32:42.095 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:32:42.128 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:32:42.158 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:32:42.190 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:32:42.191 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:32:42.192 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:32:42.192 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:32:42.193 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:32:42.193 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:32:42.193 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:32:42.194 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:32:42.194 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:32:42.406 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:32:42.444 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[52836]: debug 2021-02-10T04:32:42.067+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:42.509 INFO:tasks.rados.rados.0.smithi085.stdout:1246: done (15 left) 2021-02-10T04:32:42.509 INFO:tasks.rados.rados.0.smithi085.stdout:1247: done (14 left) 2021-02-10T04:32:42.509 INFO:tasks.rados.rados.0.smithi085.stdout:1248: done (13 left) 2021-02-10T04:32:42.510 INFO:tasks.rados.rados.0.smithi085.stdout:1249: snap_create 2021-02-10T04:32:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: cluster 2021-02-10T04:32:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: 2021-02-10T04:32:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: 2021 2021-02-10T04:32:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: - 2021-02-10T04:32:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: 02 2021-02-10T04:32:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: - 2021-02-10T04:32:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: 10T04:32:41 2021-02-10T04:32:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: .459590+0000 osd.4 (osd.4) 377 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: cluster 2021-02-10T04:32:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: 2021-02-10T04:32:41. 2021-02-10T04:32:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: 459596+0000 osd.4 (osd.4) 378 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: cluster 2021-02-10T04:32:41.459601+0000 osd.4 ( 2021-02-10T04:32:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: osd.4) 379 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: cluster 2021-02-10T04:32:41.459607+0000 osd.4 ( 2021-02-10T04:32:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: osd.4) 380 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: cluster 2021-02-10T04:32:41.491713+0000 mon.a (mon 2021-02-10T04:32:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: .0) 1104 : cluster [DBG] osdmap e344: 8 total, 8 up, 8 in 2021-02-10T04:32:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: audit 2021-02-10T04:32:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: 2021-02-10T04:32:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: 2021 2021-02-10T04:32:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: - 2021-02-10T04:32:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: 02 2021-02-10T04:32:42.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: - 2021-02-10T04:32:42.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: 10 2021-02-10T04:32:42.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: T 2021-02-10T04:32:42.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: 04 2021-02-10T04:32:42.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: : 2021-02-10T04:32:42.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: 32 2021-02-10T04:32:42.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: : 2021-02-10T04:32:42.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: 41 2021-02-10T04:32:42.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: . 2021-02-10T04:32:42.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: 938148 2021-02-10T04:32:42.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: +0000 2021-02-10T04:32:42.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: 2021-02-10T04:32:42.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: mon.c 2021-02-10T04:32:42.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: ( 2021-02-10T04:32:42.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: mon 2021-02-10T04:32:42.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: . 2021-02-10T04:32:42.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: 1) 52 : audit [DBG] from='client.? 172.21.15.85:0/3227072542' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:32:42.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: 2021-02-10T04:32:42.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:42.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: -02-10T04:32:42.060595+0000 mgr.y (mgr.14140) 490 : 2021-02-10T04:32:42.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:42 smithi164 conmon[29288]: cluster [DBG] pgmap v666: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 5 active+clean+snaptrim, 160 active+clean; 180 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 16 MiB/s rd, 3.0 MiB/s wr, 18 op/s 2021-02-10T04:32:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[29319]: cluster 2021-02-10T04:32:41.459590+0000 osd.4 2021-02-10T04:32:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[29319]: (osd.4) 377 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[29319]: cluster 2021-02-10T04:32:41.459596+0000 osd.4 (osd.4) 378 : 2021-02-10T04:32:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:32:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[29319]: T04:32:41.459601+0000 osd.4 (osd.4) 379 : 2021-02-10T04:32:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:32:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[29319]: :32:41.459607+0000 osd.4 ( 2021-02-10T04:32:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[29319]: osd.4) 380 : cluster 2021-02-10T04:32:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[29319]: 10T04:32:41.491713+0000 mon.a (mon.0) 1104 : cluster [DBG] 2021-02-10T04:32:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[29319]: osdmap e344: 8 total, 8 up, 8 in 2021-02-10T04:32:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[29319]: audit 2021-02-10T04: 2021-02-10T04:32:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[29319]: 32:41.938148+0000 mon.c (mon.1) 2021-02-10T04:32:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[29319]: 52 : audit [DBG] from='client.? 172.21.15.85:0/3227072542' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:32:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[29319]: cluster 2021-02-10T04:32:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[29319]: 2021-02-10T04:32:42 2021-02-10T04:32:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[29319]: .060595+0000 mgr.y (mgr.14140) 2021-02-10T04:32:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[29319]: 490 : cluster [DBG] pgmap v666: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 5 active+clean+snaptrim, 160 active+clean; 180 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 16 MiB/s rd, 3.0 MiB/s wr, 18 op/s 2021-02-10T04:32:42.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[35961]: cluster 2021-02-10T04:32:41.459590+0000 2021-02-10T04:32:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[35961]: osd.4 (osd.4) 377 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:32:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[35961]: cluster 2021-02-10T04:32:41.459596+0000 osd.4 (osd.4 2021-02-10T04:32:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[35961]: ) 378 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:32:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:32:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[35961]: 04:32:41.459601+0000 osd.4 (osd.4) 379 : cluster 2021-02-10T04:32:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:32:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[35961]: cluster 2021-02-10T04:32:41 2021-02-10T04:32:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[35961]: .459607+0000 osd.4 (osd.4) 380 : cluster 2021-02-10T04:32:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:32:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[35961]: cluster 2021-02-10T04:32 2021-02-10T04:32:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[35961]: :41.491713+0000 mon.a (mon.0) 2021-02-10T04:32:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[35961]: 1104 : cluster [DBG] osdmap e344: 8 total, 8 up, 8 in 2021-02-10T04:32:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:32:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[35961]: 04:32:41.938148+0000 mon.c (mon 2021-02-10T04:32:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[35961]: .1) 52 : audit [DBG] from='client.? 172.21.15.85:0/3227072542' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:32:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[35961]: -02-10T04:32:42. 2021-02-10T04:32:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[35961]: 060595+0000 mgr.y (mgr.14140) 490 : cluster 2021-02-10T04:32:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:42 smithi085 conmon[35961]: [DBG] pgmap v666: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 5 active+clean+snaptrim, 160 active+clean; 180 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 16 MiB/s rd, 3.0 MiB/s wr, 18 op/s 2021-02-10T04:32:43.104 INFO:tasks.rados.rados.0.smithi085.stdout:1249: done (13 left) 2021-02-10T04:32:43.104 INFO:tasks.rados.rados.0.smithi085.stdout:1250: copy_from oid 37 from oid 31 current snap is 129 2021-02-10T04:32:43.104 INFO:tasks.rados.rados.0.smithi085.stdout:1251: rollback oid 36 current snap is 129 2021-02-10T04:32:43.104 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 36 to 128 2021-02-10T04:32:43.104 INFO:tasks.rados.rados.0.smithi085.stdout:1252: copy_from oid 21 from oid 1 current snap is 129 2021-02-10T04:32:43.104 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:43.111 INFO:tasks.rados.rados.0.smithi085.stdout:1251: finishing rollback tid 1 to smithi08575397-36 2021-02-10T04:32:43.111 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 213 (ObjNum 1215200496 snap 32576 seq_num 1215200496) dirty dne 2021-02-10T04:32:43.112 INFO:tasks.rados.rados.0.smithi085.stdout:1251: done (15 left) 2021-02-10T04:32:43.112 INFO:tasks.rados.rados.0.smithi085.stdout:1253: snap_remove snap 121 2021-02-10T04:32:43.237 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set noscrub 2021-02-10T04:32:43.500 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:32:43.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[35961]: cluster 2021-02-10T04:32:42. 2021-02-10T04:32:43.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[35961]: 508366+0000 mon.a (mon.0) 1105 : cluster [DBG] osdmap e345: 8 total, 8 up, 8 in 2021-02-10T04:32:43.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[35961]: cluster 2021-02-10T04:32:43.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[35961]: 2021-02-10T04:32:42.622365+0000 osd.4 (osd.4 2021-02-10T04:32:43.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[35961]: ) 381 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:32:43.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:32:43.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[35961]: T04:32:42.622573+0000 osd.4 (osd 2021-02-10T04:32:43.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[35961]: .4) 382 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:32:43.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:32:43.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[35961]: -02-10T04:32:43.097283+0000 2021-02-10T04:32:43.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[35961]: mon.a (mon.0) 1106 : cluster [WRN] Health check update: 18 slow ops, oldest one blocked for 207 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:32:43.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[35961]: 2021-02-10T04:32:43.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[35961]: cluster 2021-02-10T04:32:43.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[35961]: 2021-02-10T04:32:43.103685+0000 mon.a (mon.0 2021-02-10T04:32:43.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[35961]: ) 1107 : cluster [DBG] osdmap e346: 8 total, 8 up, 8 in 2021-02-10T04:32:43.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[29319]: cluster 2021-02-10T04:32:42.508366 2021-02-10T04:32:43.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1105 : cluster [DBG] osdmap e345: 8 total, 8 up, 8 in 2021-02-10T04:32:43.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:43.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[29319]: 02-10T04:32:42.622365+0000 osd.4 ( 2021-02-10T04:32:43.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[29319]: osd.4) 381 : cluster 2021-02-10T04:32:43.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[29319]: [DBG] purged_snaps scrub starts 2021-02-10T04:32:43.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:43.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[29319]: 10T04:32:42.622573+0000 2021-02-10T04:32:43.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[29319]: osd.4 (osd.4) 382 : cluster [DBG] 2021-02-10T04:32:43.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[29319]: purged_snaps scrub ok 2021-02-10T04:32:43.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:32:43.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[29319]: -10T04 2021-02-10T04:32:43.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[29319]: :32:43.097283+0000 mon.a 2021-02-10T04:32:43.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[29319]: (mon.0) 1106 : cluster [WRN] Health check update: 18 slow ops, oldest one blocked for 207 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:32:43.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:32:43.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[29319]: 02-10T04:32:43.103685+0000 mon.a 2021-02-10T04:32:43.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:43 smithi085 conmon[29319]: (mon.0) 1107 : cluster [DBG] osdmap e346: 8 total, 8 up, 8 in 2021-02-10T04:32:43.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:43 smithi164 conmon[29288]: cluster 2021-02-10T04:32:42. 2021-02-10T04:32:43.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:43 smithi164 conmon[29288]: 508366+0000 mon.a (mon.0) 1105 : cluster [DBG] osdmap e345: 8 total, 8 up, 8 in 2021-02-10T04:32:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:43 smithi164 conmon[29288]: cluster 2021-02-10T04:32 2021-02-10T04:32:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:43 smithi164 conmon[29288]: :42.622365+0000 osd.4 (osd.4) 381 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:32:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:43 smithi164 conmon[29288]: 2021-02-10T04:32:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:43 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:43 smithi164 conmon[29288]: 04:32:42.622573+0000 osd.4 (osd 2021-02-10T04:32:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:43 smithi164 conmon[29288]: .4) 382 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:32:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:43 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:32:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:43 smithi164 conmon[29288]: 04:32:43.097283+0000 mon.a (mon.0) 1106 2021-02-10T04:32:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:43 smithi164 conmon[29288]: : cluster [WRN] Health check update: 18 slow ops, oldest one blocked for 207 sec, daemons [osd.1,osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:32:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:43 smithi164 conmon[29288]: cluster 2021-02-10T04:32:43 2021-02-10T04:32:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:43 smithi164 conmon[29288]: .103685+0000 mon.a (mon.0) 1107 : 2021-02-10T04:32:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:43 smithi164 conmon[29288]: cluster [DBG] osdmap e346: 8 total, 8 up, 8 in 2021-02-10T04:32:44.114 INFO:tasks.rados.rados.0.smithi085.stdout:1253: done (15 left) 2021-02-10T04:32:44.114 INFO:tasks.rados.rados.0.smithi085.stdout:1254: copy_from oid 40 from oid 36 current snap is 129 2021-02-10T04:32:44.114 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:44.114 INFO:tasks.rados.rados.0.smithi085.stdout:1252: finishing copy_from to smithi08575397-21 2021-02-10T04:32:44.114 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 385 (ObjNum 424 snap 109 seq_num 424) dirty exists 2021-02-10T04:32:44.114 INFO:tasks.rados.rados.0.smithi085.stdout:1252: finishing copy_from racing read to smithi08575397-21 2021-02-10T04:32:44.115 INFO:tasks.rados.rados.0.smithi085.stdout:1250: finishing copy_from to smithi08575397-37 2021-02-10T04:32:44.115 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 616 (ObjNum 432 snap 114 seq_num 432) dirty exists 2021-02-10T04:32:44.115 INFO:tasks.rados.rados.0.smithi085.stdout:1250: finishing copy_from racing read to smithi08575397-37 2021-02-10T04:32:44.115 INFO:tasks.rados.rados.0.smithi085.stdout:1250: done (15 left) 2021-02-10T04:32:44.115 INFO:tasks.rados.rados.0.smithi085.stdout:1252: done (14 left) 2021-02-10T04:32:44.115 INFO:tasks.rados.rados.0.smithi085.stdout:1255: copy_from oid 16 from oid 37 current snap is 129 2021-02-10T04:32:44.115 INFO:tasks.rados.rados.0.smithi085.stdout:1256: read oid 26 snap -1 2021-02-10T04:32:44.115 INFO:tasks.rados.rados.0.smithi085.stdout:1256: expect (ObjNum 448 snap 121 seq_num 448) 2021-02-10T04:32:44.115 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:44.124 INFO:tasks.rados.rados.0.smithi085.stdout:1254: finishing copy_from to smithi08575397-40 2021-02-10T04:32:44.124 INFO:tasks.rados.rados.0.smithi085.stdout:1254: got expected ENOENT (src dne) 2021-02-10T04:32:44.124 INFO:tasks.rados.rados.0.smithi085.stdout:1256: expect (ObjNum 448 snap 121 seq_num 448) 2021-02-10T04:32:44.433 INFO:tasks.rados.rados.0.smithi085.stdout:1254: finishing copy_from racing read to smithi08575397-40 2021-02-10T04:32:44.433 INFO:tasks.rados.rados.0.smithi085.stdout:1255: finishing copy_from to smithi08575397-16 2021-02-10T04:32:44.433 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 616 (ObjNum 432 snap 114 seq_num 432) dirty exists 2021-02-10T04:32:44.434 INFO:tasks.rados.rados.0.smithi085.stdout:1255: finishing copy_from racing read to smithi08575397-16 2021-02-10T04:32:44.434 INFO:tasks.rados.rados.0.smithi085.stdout:1254: done (15 left) 2021-02-10T04:32:44.434 INFO:tasks.rados.rados.0.smithi085.stdout:1255: done (14 left) 2021-02-10T04:32:44.434 INFO:tasks.rados.rados.0.smithi085.stdout:1256: done (13 left) 2021-02-10T04:32:44.434 INFO:tasks.rados.rados.0.smithi085.stdout:1257: read oid 50 snap 122 2021-02-10T04:32:44.435 INFO:tasks.rados.rados.0.smithi085.stdout:1257: expect deleted 2021-02-10T04:32:44.435 INFO:tasks.rados.rados.0.smithi085.stdout:1258: read oid 31 snap 118 2021-02-10T04:32:44.435 INFO:tasks.rados.rados.0.smithi085.stdout:1258: expect (ObjNum 432 snap 114 seq_num 432) 2021-02-10T04:32:44.435 INFO:tasks.rados.rados.0.smithi085.stdout:1259: snap_remove snap 124 2021-02-10T04:32:45.107 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:44 smithi085 conmon[52836]: debug 2021-02-10T04:32:44.798+0000 7fba73fb6700 -1 osd.3 pg_epoch: 347 pg[4.14( v 63'26 (0'0,63'26] local-lis/les=71/72 n=11 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,2,7] r=0 lpr=71 crt=63'26 lcod 63'25 mlcod 0'0 active+clean] scrubber pg(4.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55fec0910000) 2021-02-10T04:32:45.109 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:45 smithi085 conmon[29319]: cluster 2021-02-10T04:32:45.109 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:45 smithi085 conmon[29319]: 2021-02-10T04:32 2021-02-10T04:32:45.109 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:45 smithi085 conmon[29319]: :44.061037+0000 mgr.y (mgr.14140) 491 : cluster [DBG] pgmap v669: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 5 active+clean+snaptrim, 160 active+clean; 182 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 9.6 MiB/s rd, 2.7 MiB/s wr, 13 op/s 2021-02-10T04:32:45.109 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:45 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:32:45.109 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:45 smithi085 conmon[29319]: 32:44.112002+0000 mon.a (mon.0) 1108 : 2021-02-10T04:32:45.109 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:45 smithi085 conmon[29319]: cluster [DBG] osdmap e347: 8 total, 8 up, 8 in 2021-02-10T04:32:45.110 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:45 smithi085 conmon[35961]: cluster 2021-02-10T04:32:44.061037+0000 2021-02-10T04:32:45.110 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:45 smithi085 conmon[35961]: mgr.y (mgr.14140) 491 : cluster [DBG] pgmap v669: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 5 active+clean+snaptrim, 160 active+clean; 182 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 9.6 MiB/s rd, 2.7 MiB/s wr, 13 op/s 2021-02-10T04:32:45.110 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:45 smithi085 conmon[35961]: cluster 2021-02-10T04:32:45.110 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:45 smithi085 conmon[35961]: 2021-02-10T04:32:44.112002+0000 mon.a (mon.0 2021-02-10T04:32:45.110 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:45 smithi085 conmon[35961]: ) 1108 : cluster [DBG] osdmap e347: 8 total, 8 up, 8 in 2021-02-10T04:32:45.115 INFO:tasks.rados.rados.0.smithi085.stdout:1259: done (15 left) 2021-02-10T04:32:45.116 INFO:tasks.rados.rados.0.smithi085.stdout:1260: write (excl) oid 38 current snap is 129 2021-02-10T04:32:45.116 INFO:tasks.rados.rados.0.smithi085.stdout:1258: expect (ObjNum 432 snap 114 seq_num 432) 2021-02-10T04:32:45.313 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:32:45.346 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:45.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:45 smithi164 conmon[29288]: cluster 2021-02-10T04:32:44.061037+0000 mgr.y 2021-02-10T04:32:45.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:45 smithi164 conmon[29288]: (mgr.14140) 491 : cluster [DBG] pgmap v669: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 5 active+clean+snaptrim, 160 active+clean; 182 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 9.6 MiB/s rd, 2.7 MiB/s wr, 13 op/s 2021-02-10T04:32:45.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:45 smithi164 conmon[29288]: cluster 2021-02-10T04:32:44.112002+0000 mon.a (mon.0) 1108 : 2021-02-10T04:32:45.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:45 smithi164 conmon[29288]: cluster [DBG] osdmap e347: 8 total, 8 up, 8 in 2021-02-10T04:32:45.400 INFO:tasks.rados.rados.0.smithi085.stdout:1260: seq_num 458 ranges {497606=645222,1934400=629494,3243966=453640} 2021-02-10T04:32:45.407 INFO:tasks.rados.rados.0.smithi085.stdout:1260: writing smithi08575397-38 from 497606 to 1142828 tid 1 2021-02-10T04:32:45.418 INFO:tasks.rados.rados.0.smithi085.stdout:1260: writing smithi08575397-38 from 1934400 to 2563894 tid 2 2021-02-10T04:32:45.425 INFO:tasks.rados.rados.0.smithi085.stdout:1260: writing smithi08575397-38 from 3243966 to 3697606 tid 3 2021-02-10T04:32:45.425 INFO:tasks.rados.rados.0.smithi085.stdout:1257: done (15 left) 2021-02-10T04:32:45.425 INFO:tasks.rados.rados.0.smithi085.stdout:1258: done (14 left) 2021-02-10T04:32:45.425 INFO:tasks.rados.rados.0.smithi085.stdout:1261: delete oid 7 current snap is 129 2021-02-10T04:32:45.425 INFO:tasks.rados.rados.0.smithi085.stdout:1260: finishing write tid 1 to smithi08575397-38 2021-02-10T04:32:45.425 INFO:tasks.rados.rados.0.smithi085.stdout:1260: finishing write tid 2 to smithi08575397-38 2021-02-10T04:32:45.428 INFO:tasks.rados.rados.0.smithi085.stdout:1260: finishing write tid 3 to smithi08575397-38 2021-02-10T04:32:45.429 INFO:tasks.rados.rados.0.smithi085.stdout:1260: finishing write tid 4 to smithi08575397-38 2021-02-10T04:32:45.429 INFO:tasks.rados.rados.0.smithi085.stdout:1261: done (14 left) 2021-02-10T04:32:45.429 INFO:tasks.rados.rados.0.smithi085.stdout:1262: copy_from oid 36 from oid 37 current snap is 129 2021-02-10T04:32:45.429 INFO:tasks.rados.rados.0.smithi085.stdout:1260: finishing write tid 5 to smithi08575397-38 2021-02-10T04:32:45.430 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 632 (ObjNum 458 snap 129 seq_num 458) dirty exists 2021-02-10T04:32:45.430 INFO:tasks.rados.rados.0.smithi085.stdout:1260: left oid 38 (ObjNum 458 snap 129 seq_num 458) 2021-02-10T04:32:45.430 INFO:tasks.rados.rados.0.smithi085.stdout:1260: done (14 left) 2021-02-10T04:32:45.430 INFO:tasks.rados.rados.0.smithi085.stdout:1263: snap_create 2021-02-10T04:32:46.007 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-02-10T04:32:46.007 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:32:46.007 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.000000 2021-02-10T04:32:46.007 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd primary-affinity 1 0 2021-02-10T04:32:46.119 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:45 smithi085 conmon[52836]: debug 2021-02-10T04:32:45.824+0000 7fba73fb6700 -1 osd.3 pg_epoch: 348 pg[4.14( v 63'26 (0'0,63'26] local-lis/les=71/72 n=11 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,2,7] r=0 lpr=71 crt=63'26 lcod 63'25 mlcod 0'0 active+clean] scrubber pg(4.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55fec06c0c60) 2021-02-10T04:32:46.120 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:46 smithi085 conmon[29319]: cluster 2021-02-10T04:32:45.114116+0000 mon.a 2021-02-10T04:32:46.120 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:46 smithi085 conmon[29319]: (mon.0) 1109 : cluster [DBG] osdmap e348: 8 total, 8 up, 8 in 2021-02-10T04:32:46.128 INFO:tasks.rados.rados.0.smithi085.stdout:1263: done (14 left) 2021-02-10T04:32:46.128 INFO:tasks.rados.rados.0.smithi085.stdout:1264: write (excl) oid 5 current snap is 130 2021-02-10T04:32:46.128 INFO:tasks.rados.rados.0.smithi085.stdout:1264: seq_num 460 ranges {531220=795720,1935011=516810,3066106=420420} 2021-02-10T04:32:46.132 INFO:tasks.rados.rados.0.smithi085.stdout:1264: writing smithi08575397-5 from 531220 to 1326940 tid 1 2021-02-10T04:32:46.140 INFO:tasks.rados.rados.0.smithi085.stdout:1264: writing smithi08575397-5 from 1935011 to 2451821 tid 2 2021-02-10T04:32:46.151 INFO:tasks.rados.rados.0.smithi085.stdout:1264: writing smithi08575397-5 from 3066106 to 3486526 tid 3 2021-02-10T04:32:46.151 INFO:tasks.rados.rados.0.smithi085.stdout:1265: read oid 31 snap -1 2021-02-10T04:32:46.151 INFO:tasks.rados.rados.0.smithi085.stdout:1265: expect (ObjNum 432 snap 114 seq_num 432) 2021-02-10T04:32:46.151 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:46.151 INFO:tasks.rados.rados.0.smithi085.stdout:1264: finishing write tid 1 to smithi08575397-5 2021-02-10T04:32:46.151 INFO:tasks.rados.rados.0.smithi085.stdout:1264: finishing write tid 2 to smithi08575397-5 2021-02-10T04:32:46.154 INFO:tasks.rados.rados.0.smithi085.stdout:1264: finishing write tid 3 to smithi08575397-5 2021-02-10T04:32:46.154 INFO:tasks.rados.rados.0.smithi085.stdout:1264: finishing write tid 4 to smithi08575397-5 2021-02-10T04:32:46.154 INFO:tasks.rados.rados.0.smithi085.stdout:1264: finishing write tid 5 to smithi08575397-5 2021-02-10T04:32:46.154 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 586 (ObjNum 460 snap 130 seq_num 460) dirty exists 2021-02-10T04:32:46.154 INFO:tasks.rados.rados.0.smithi085.stdout:1264: left oid 5 (ObjNum 460 snap 130 seq_num 460) 2021-02-10T04:32:46.155 INFO:tasks.rados.rados.0.smithi085.stdout:1264: done (15 left) 2021-02-10T04:32:46.155 INFO:tasks.rados.rados.0.smithi085.stdout:1266: write (excl) oid 30 current snap is 130 2021-02-10T04:32:46.155 INFO:tasks.rados.rados.0.smithi085.stdout:1266: seq_num 461 ranges {548027=470969,1593493=776821,3093529=631692} 2021-02-10T04:32:46.162 INFO:tasks.rados.rados.0.smithi085.stdout:1266: writing smithi08575397-30 from 548027 to 1018996 tid 1 2021-02-10T04:32:46.172 INFO:tasks.rados.rados.0.smithi085.stdout:1266: writing smithi08575397-30 from 1593493 to 2370314 tid 2 2021-02-10T04:32:46.190 INFO:tasks.rados.rados.0.smithi085.stdout:1266: writing smithi08575397-30 from 3093529 to 3725221 tid 3 2021-02-10T04:32:46.190 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:46.190 INFO:tasks.rados.rados.0.smithi085.stdout:1265: expect (ObjNum 432 snap 114 seq_num 432) 2021-02-10T04:32:46.302 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:32:46.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:46 smithi164 conmon[29288]: cluster 2021-02-10T04:32:45.114116+0000 mon.a (mon.0) 1109 : cluster 2021-02-10T04:32:46.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:46 smithi164 conmon[29288]: [DBG] osdmap e348: 8 total, 8 up, 8 in 2021-02-10T04:32:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:46 smithi085 conmon[35961]: cluster 2021-02-10T04:32:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:46 smithi085 conmon[35961]: 2021-02-10T04:32:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:46 smithi085 conmon[35961]: 2021 2021-02-10T04:32:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:46 smithi085 conmon[35961]: - 2021-02-10T04:32:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:46 smithi085 conmon[35961]: 02 2021-02-10T04:32:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:46 smithi085 conmon[35961]: -10T04:32: 2021-02-10T04:32:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:46 smithi085 conmon[35961]: 45.114116+0000 mon.a (mon.0) 1109 : cluster [DBG] osdmap e348: 8 total, 8 up, 8 in 2021-02-10T04:32:46.476 INFO:tasks.rados.rados.0.smithi085.stdout:1266: finishing write tid 1 to smithi08575397-30 2021-02-10T04:32:46.476 INFO:tasks.rados.rados.0.smithi085.stdout:1266: finishing write tid 2 to smithi08575397-30 2021-02-10T04:32:46.476 INFO:tasks.rados.rados.0.smithi085.stdout:1265: done (15 left) 2021-02-10T04:32:46.476 INFO:tasks.rados.rados.0.smithi085.stdout:1267: read oid 46 snap -1 2021-02-10T04:32:46.476 INFO:tasks.rados.rados.0.smithi085.stdout:1267: expect (ObjNum 454 snap 125 seq_num 454) 2021-02-10T04:32:46.476 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:46.476 INFO:tasks.rados.rados.0.smithi085.stdout:1266: finishing write tid 3 to smithi08575397-30 2021-02-10T04:32:46.477 INFO:tasks.rados.rados.0.smithi085.stdout:1266: finishing write tid 4 to smithi08575397-30 2021-02-10T04:32:46.477 INFO:tasks.rados.rados.0.smithi085.stdout:1266: finishing write tid 5 to smithi08575397-30 2021-02-10T04:32:46.477 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 636 (ObjNum 461 snap 130 seq_num 461) dirty exists 2021-02-10T04:32:46.477 INFO:tasks.rados.rados.0.smithi085.stdout:1266: left oid 30 (ObjNum 461 snap 130 seq_num 461) 2021-02-10T04:32:46.477 INFO:tasks.rados.rados.0.smithi085.stdout:1262: finishing copy_from to smithi08575397-36 2021-02-10T04:32:46.477 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 616 (ObjNum 432 snap 114 seq_num 432) dirty exists 2021-02-10T04:32:46.477 INFO:tasks.rados.rados.0.smithi085.stdout:1262: finishing copy_from racing read to smithi08575397-36 2021-02-10T04:32:46.477 INFO:tasks.rados.rados.0.smithi085.stdout:1262: done (15 left) 2021-02-10T04:32:46.477 INFO:tasks.rados.rados.0.smithi085.stdout:1266: done (14 left) 2021-02-10T04:32:46.477 INFO:tasks.rados.rados.0.smithi085.stdout:1268: read oid 40 snap -1 2021-02-10T04:32:46.477 INFO:tasks.rados.rados.0.smithi085.stdout:1268: expect (ObjNum 447 snap 121 seq_num 447) 2021-02-10T04:32:46.478 INFO:tasks.rados.rados.0.smithi085.stdout:1269: delete oid 21 current snap is 130 2021-02-10T04:32:46.485 INFO:tasks.rados.rados.0.smithi085.stdout:1267: expect (ObjNum 454 snap 125 seq_num 454) 2021-02-10T04:32:46.692 INFO:tasks.rados.rados.0.smithi085.stdout:1267: done (15 left) 2021-02-10T04:32:46.692 INFO:tasks.rados.rados.0.smithi085.stdout:1269: done (14 left) 2021-02-10T04:32:46.692 INFO:tasks.rados.rados.0.smithi085.stdout:1270: copy_from oid 11 from oid 16 current snap is 130 2021-02-10T04:32:46.693 INFO:tasks.rados.rados.0.smithi085.stdout:1268: expect (ObjNum 447 snap 121 seq_num 447) 2021-02-10T04:32:46.898 INFO:tasks.rados.rados.0.smithi085.stdout:1270: finishing copy_from to smithi08575397-11 2021-02-10T04:32:46.898 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 616 (ObjNum 432 snap 114 seq_num 432) dirty exists 2021-02-10T04:32:46.898 INFO:tasks.rados.rados.0.smithi085.stdout:1270: finishing copy_from racing read to smithi08575397-11 2021-02-10T04:32:46.898 INFO:tasks.rados.rados.0.smithi085.stdout:1268: done (14 left) 2021-02-10T04:32:46.898 INFO:tasks.rados.rados.0.smithi085.stdout:1270: done (13 left) 2021-02-10T04:32:46.898 INFO:tasks.rados.rados.0.smithi085.stdout:1271: write (excl) oid 28 current snap is 130 2021-02-10T04:32:46.899 INFO:tasks.rados.rados.0.smithi085.stdout:1271: seq_num 463 ranges {581641=621467,1994104=780490,3432022=349619} 2021-02-10T04:32:46.906 INFO:tasks.rados.rados.0.smithi085.stdout:1271: writing smithi08575397-28 from 581641 to 1203108 tid 1 2021-02-10T04:32:46.916 INFO:tasks.rados.rados.0.smithi085.stdout:1271: writing smithi08575397-28 from 1994104 to 2774594 tid 2 2021-02-10T04:32:46.923 INFO:tasks.rados.rados.0.smithi085.stdout:1271: writing smithi08575397-28 from 3432022 to 3781641 tid 3 2021-02-10T04:32:46.923 INFO:tasks.rados.rados.0.smithi085.stdout:1272: read oid 31 snap 118 2021-02-10T04:32:46.923 INFO:tasks.rados.rados.0.smithi085.stdout:1272: expect (ObjNum 432 snap 114 seq_num 432) 2021-02-10T04:32:46.923 INFO:tasks.rados.rados.0.smithi085.stdout:1273: copy_from oid 40 from oid 3 current snap is 130 2021-02-10T04:32:46.923 INFO:tasks.rados.rados.0.smithi085.stdout:1271: finishing write tid 1 to smithi08575397-28 2021-02-10T04:32:46.923 INFO:tasks.rados.rados.0.smithi085.stdout:1271: finishing write tid 2 to smithi08575397-28 2021-02-10T04:32:46.923 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:46.925 INFO:tasks.rados.rados.0.smithi085.stdout:1271: finishing write tid 3 to smithi08575397-28 2021-02-10T04:32:46.926 INFO:tasks.rados.rados.0.smithi085.stdout:1271: finishing write tid 4 to smithi08575397-28 2021-02-10T04:32:46.926 INFO:tasks.rados.rados.0.smithi085.stdout:1271: finishing write tid 5 to smithi08575397-28 2021-02-10T04:32:46.927 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 615 (ObjNum 463 snap 130 seq_num 463) dirty exists 2021-02-10T04:32:46.927 INFO:tasks.rados.rados.0.smithi085.stdout:1271: left oid 28 (ObjNum 463 snap 130 seq_num 463) 2021-02-10T04:32:46.927 INFO:tasks.rados.rados.0.smithi085.stdout:1271: done (15 left) 2021-02-10T04:32:46.927 INFO:tasks.rados.rados.0.smithi085.stdout:1274: copy_from oid 8 from oid 13 current snap is 130 2021-02-10T04:32:46.927 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:46.941 INFO:tasks.rados.rados.0.smithi085.stdout:1272: expect (ObjNum 432 snap 114 seq_num 432) 2021-02-10T04:32:47.223 INFO:tasks.rados.rados.0.smithi085.stdout:1272: done (15 left) 2021-02-10T04:32:47.224 INFO:tasks.rados.rados.0.smithi085.stdout:1275: snap_remove snap 127 2021-02-10T04:32:47.328 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.2 dump_ops_in_flight 2021-02-10T04:32:47.346 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:32:47.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: cluster 2021-02-10T04:32:47.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: 2021-02-10T04:32:47.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: 2021 2021-02-10T04:32:47.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: - 2021-02-10T04:32:47.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: 02 2021-02-10T04:32:47.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: - 2021-02-10T04:32:47.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: 10 2021-02-10T04:32:47.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: T 2021-02-10T04:32:47.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: 04 2021-02-10T04:32:47.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: : 2021-02-10T04:32:47.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: 32 2021-02-10T04:32:47.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: : 2021-02-10T04:32:47.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: 46 2021-02-10T04:32:47.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: . 2021-02-10T04:32:47.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: 061608 2021-02-10T04:32:47.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: +0000 2021-02-10T04:32:47.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: 2021-02-10T04:32:47.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: mgr.y 2021-02-10T04:32:47.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: ( 2021-02-10T04:32:47.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: mgr 2021-02-10T04:32:47.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: . 2021-02-10T04:32:47.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: 14140 2021-02-10T04:32:47.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: ) 2021-02-10T04:32:47.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: 492 2021-02-10T04:32:47.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: : 2021-02-10T04:32:47.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: cluster 2021-02-10T04:32:47.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: 2021-02-10T04:32:47.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: [DBG] 2021-02-10T04:32:47.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: 2021-02-10T04:32:47.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: pgmap v672: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim, 163 active+clean; 196 MiB data, 675 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 2.9 MiB/s wr, 15 op/s 2021-02-10T04:32:47.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: 2021-02-10T04:32:47.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: cluster 2021-02-10T04:32:47.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: 2021-02-10T04:32:46.123630+0000 mon.a (mon.0) 1110 : cluster [DBG] osdmap e349: 8 total, 8 up, 8 in 2021-02-10T04:32:47.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[29288]: 2021-02-10T04:32:47.423 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:47 smithi085 conmon[29319]: cluster 2021-02-10T04:32:46.061608+0000 mgr.y 2021-02-10T04:32:47.423 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:47 smithi085 conmon[29319]: (mgr.14140) 492 : cluster [DBG] pgmap v672: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim, 163 active+clean; 196 MiB data, 675 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 2.9 MiB/s wr, 15 op/s 2021-02-10T04:32:47.423 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:47 smithi085 conmon[29319]: cluster 2021-02-10T04:32:46.123630+0000 mon.a (mon.0) 1110 : cluster 2021-02-10T04:32:47.423 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:47 smithi085 conmon[29319]: [DBG] osdmap e349: 8 total, 8 up, 8 in 2021-02-10T04:32:47.424 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:47 smithi085 conmon[45443]: debug 2021-02-10T04:32:47.299+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:47.424 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:47 smithi085 conmon[45443]: debug 2021-02-10T04:32:47.331+0000 7f7125cf8700 -1 osd.1 pg_epoch: 350 pg[4.4( v 63'31 (0'0,63'31] local-lis/les=55/56 n=14 ec=55/55 lis/c=55/55 les/c/f=56/56/0 sis=55) [1,2,3] r=0 lpr=55 crt=63'31 lcod 63'30 mlcod 63'30 active+clean] scrubber pg(4.4) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 3 (0x55dbc881bce0) 2021-02-10T04:32:47.424 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:47 smithi085 conmon[35961]: cluster 2021-02-10T04:32:46.061608+0000 2021-02-10T04:32:47.424 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:47 smithi085 conmon[35961]: mgr.y (mgr.14140) 492 : cluster [DBG] pgmap v672: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim, 163 active+clean; 196 MiB data, 675 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 2.9 MiB/s wr, 15 op/s 2021-02-10T04:32:47.425 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:47 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:32:47.425 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:47 smithi085 conmon[35961]: 32:46.123630+0000 mon.a (mon.0) 1110 : cluster [DBG] osdmap e349: 8 total, 8 up, 8 in 2021-02-10T04:32:47.452 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:32:47.489 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:47.645 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:32:47.691 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:47 smithi085 conmon[52836]: debug 2021-02-10T04:32:47.691 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:47 smithi085 conmon[52836]: 2021-02-10T04:32:47.630+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:47.691 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:47 smithi085 conmon[52836]: 2021-02-10T04:32:47.780 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:32:47.889 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:32:47.921 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:48.115 INFO:tasks.rados.rados.0.smithi085.stdout:1275: done (15 left) 2021-02-10T04:32:48.115 INFO:tasks.rados.rados.0.smithi085.stdout:1276: write (excl) oid 22 current snap is 130 2021-02-10T04:32:48.115 INFO:tasks.rados.rados.0.smithi085.stdout:1276: seq_num 464 ranges {598448=413069,1768939=524148,2775798=581861,3781488=16960} 2021-02-10T04:32:48.132 INFO:tasks.rados.rados.0.smithi085.stdout:1276: writing smithi08575397-22 from 598448 to 1011517 tid 1 2021-02-10T04:32:48.132 INFO:tasks.rados.rados.0.smithi085.stdout:1276: writing smithi08575397-22 from 1768939 to 2293087 tid 2 2021-02-10T04:32:48.132 INFO:tasks.rados.rados.0.smithi085.stdout:1276: writing smithi08575397-22 from 2775798 to 3357659 tid 3 2021-02-10T04:32:48.134 INFO:tasks.rados.rados.0.smithi085.stdout:1276: writing smithi08575397-22 from 3781488 to 3798448 tid 4 2021-02-10T04:32:48.135 INFO:tasks.rados.rados.0.smithi085.stdout:1274: finishing copy_from to smithi08575397-8 2021-02-10T04:32:48.135 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 582 (ObjNum 457 snap 127 seq_num 457) dirty exists 2021-02-10T04:32:48.135 INFO:tasks.rados.rados.0.smithi085.stdout:1274: finishing copy_from racing read to smithi08575397-8 2021-02-10T04:32:48.135 INFO:tasks.rados.rados.0.smithi085.stdout:1273: finishing copy_from to smithi08575397-40 2021-02-10T04:32:48.135 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 492 (ObjNum 437 snap 116 seq_num 437) dirty exists 2021-02-10T04:32:48.135 INFO:tasks.rados.rados.0.smithi085.stdout:1273: finishing copy_from racing read to smithi08575397-40 2021-02-10T04:32:48.135 INFO:tasks.rados.rados.0.smithi085.stdout:1273: done (15 left) 2021-02-10T04:32:48.135 INFO:tasks.rados.rados.0.smithi085.stdout:1274: done (14 left) 2021-02-10T04:32:48.135 INFO:tasks.rados.rados.0.smithi085.stdout:1277: write oid 16 current snap is 130 2021-02-10T04:32:48.135 INFO:tasks.rados.rados.0.smithi085.stdout:1277: seq_num 465 ranges {615255=488318,1827421=784159,3203221=612034} 2021-02-10T04:32:48.142 INFO:tasks.rados.rados.0.smithi085.stdout:1277: writing smithi08575397-16 from 615255 to 1103573 tid 1 2021-02-10T04:32:48.152 INFO:tasks.rados.rados.0.smithi085.stdout:1277: writing smithi08575397-16 from 1827421 to 2611580 tid 2 2021-02-10T04:32:48.162 INFO:tasks.rados.rados.0.smithi085.stdout:1277: writing smithi08575397-16 from 3203221 to 3815255 tid 3 2021-02-10T04:32:48.162 INFO:tasks.rados.rados.0.smithi085.stdout:1276: finishing write tid 1 to smithi08575397-22 2021-02-10T04:32:48.162 INFO:tasks.rados.rados.0.smithi085.stdout:1276: finishing write tid 2 to smithi08575397-22 2021-02-10T04:32:48.162 INFO:tasks.rados.rados.0.smithi085.stdout:1276: finishing write tid 3 to smithi08575397-22 2021-02-10T04:32:48.162 INFO:tasks.rados.rados.0.smithi085.stdout:1276: finishing write tid 4 to smithi08575397-22 2021-02-10T04:32:48.163 INFO:tasks.rados.rados.0.smithi085.stdout:1276: finishing write tid 5 to smithi08575397-22 2021-02-10T04:32:48.163 INFO:tasks.rados.rados.0.smithi085.stdout:1276: finishing write tid 6 to smithi08575397-22 2021-02-10T04:32:48.163 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 621 (ObjNum 464 snap 130 seq_num 464) dirty exists 2021-02-10T04:32:48.163 INFO:tasks.rados.rados.0.smithi085.stdout:1276: left oid 22 (ObjNum 464 snap 130 seq_num 464) 2021-02-10T04:32:48.163 INFO:tasks.rados.rados.0.smithi085.stdout:1277: finishing write tid 1 to smithi08575397-16 2021-02-10T04:32:48.163 INFO:tasks.rados.rados.0.smithi085.stdout:1277: finishing write tid 2 to smithi08575397-16 2021-02-10T04:32:48.163 INFO:tasks.rados.rados.0.smithi085.stdout:1276: done (14 left) 2021-02-10T04:32:48.164 INFO:tasks.rados.rados.0.smithi085.stdout:1278: snap_create 2021-02-10T04:32:48.183 INFO:tasks.rados.rados.0.smithi085.stdout:1277: finishing write tid 3 to smithi08575397-16 2021-02-10T04:32:48.183 INFO:tasks.rados.rados.0.smithi085.stdout:1277: finishing write tid 4 to smithi08575397-16 2021-02-10T04:32:48.183 INFO:tasks.rados.rados.0.smithi085.stdout:1277: finishing write tid 5 to smithi08575397-16 2021-02-10T04:32:48.183 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 710 (ObjNum 465 snap 130 seq_num 465) dirty exists 2021-02-10T04:32:48.183 INFO:tasks.rados.rados.0.smithi085.stdout:1277: left oid 16 (ObjNum 465 snap 130 seq_num 465) 2021-02-10T04:32:48.242 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: cluster 2021-02-10T04:32:48.242 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 2021-02-10T04:32:48.242 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 2021 2021-02-10T04:32:48.242 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: - 2021-02-10T04:32:48.242 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 02 2021-02-10T04:32:48.242 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: - 2021-02-10T04:32:48.242 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 10 2021-02-10T04:32:48.242 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: T 2021-02-10T04:32:48.242 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 04 2021-02-10T04:32:48.242 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: : 2021-02-10T04:32:48.243 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 32 2021-02-10T04:32:48.243 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: : 2021-02-10T04:32:48.243 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 46 2021-02-10T04:32:48.243 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: . 2021-02-10T04:32:48.243 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 885307 2021-02-10T04:32:48.243 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: +0000 2021-02-10T04:32:48.243 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 2021-02-10T04:32:48.243 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: osd.1 2021-02-10T04:32:48.243 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: ( 2021-02-10T04:32:48.243 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: osd 2021-02-10T04:32:48.243 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: . 2021-02-10T04:32:48.243 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 1 2021-02-10T04:32:48.243 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: ) 2021-02-10T04:32:48.243 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 443 2021-02-10T04:32:48.244 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: : 2021-02-10T04:32:48.244 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: cluster 2021-02-10T04:32:48.244 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 2021-02-10T04:32:48.244 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: [DBG] 2021-02-10T04:32:48.244 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 2021-02-10T04:32:48.244 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: purged_snaps scrub starts 2021-02-10T04:32:48.244 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 2021-02-10T04:32:48.244 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: cluster 2021-02-10T04:32:48.244 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 2021-02-10T04:32:48.244 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 2021 2021-02-10T04:32:48.244 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: - 2021-02-10T04:32:48.244 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 02 2021-02-10T04:32:48.244 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: - 2021-02-10T04:32:48.245 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 10 2021-02-10T04:32:48.245 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: T 2021-02-10T04:32:48.245 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 04 2021-02-10T04:32:48.245 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: : 2021-02-10T04:32:48.245 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 32 2021-02-10T04:32:48.245 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: : 2021-02-10T04:32:48.245 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 46 2021-02-10T04:32:48.245 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: . 2021-02-10T04:32:48.245 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 885604 2021-02-10T04:32:48.245 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: +0000 2021-02-10T04:32:48.245 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 2021-02-10T04:32:48.245 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: osd.1 2021-02-10T04:32:48.245 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: ( 2021-02-10T04:32:48.245 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: osd 2021-02-10T04:32:48.246 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: . 2021-02-10T04:32:48.246 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 1 2021-02-10T04:32:48.246 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: ) 2021-02-10T04:32:48.246 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 444 2021-02-10T04:32:48.246 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: : 2021-02-10T04:32:48.246 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: cluster 2021-02-10T04:32:48.246 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 2021-02-10T04:32:48.246 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: [DBG] 2021-02-10T04:32:48.246 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 2021-02-10T04:32:48.246 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:32:48.246 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 2021-02-10T04:32:48.246 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: cluster 2021-02-10T04:32:48.246 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 2021-02-10T04:32:48.247 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 2021 2021-02-10T04:32:48.247 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: - 2021-02-10T04:32:48.247 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 02 2021-02-10T04:32:48.247 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: - 2021-02-10T04:32:48.247 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 10 2021-02-10T04:32:48.247 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: T 2021-02-10T04:32:48.247 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 04 2021-02-10T04:32:48.247 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: : 2021-02-10T04:32:48.247 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 32 2021-02-10T04:32:48.247 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: : 2021-02-10T04:32:48.247 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 47 2021-02-10T04:32:48.247 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: . 2021-02-10T04:32:48.247 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 131268 2021-02-10T04:32:48.248 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: +0000 2021-02-10T04:32:48.248 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 2021-02-10T04:32:48.248 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: mon.a 2021-02-10T04:32:48.248 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: ( 2021-02-10T04:32:48.248 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: mon 2021-02-10T04:32:48.248 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: . 2021-02-10T04:32:48.248 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 0 2021-02-10T04:32:48.248 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: ) 2021-02-10T04:32:48.248 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 1111 2021-02-10T04:32:48.248 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: : 2021-02-10T04:32:48.248 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: cluster 2021-02-10T04:32:48.248 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 2021-02-10T04:32:48.248 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: [DBG] 2021-02-10T04:32:48.248 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 2021-02-10T04:32:48.249 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: osdmap e350: 8 total, 8 up, 8 in 2021-02-10T04:32:48.249 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 2021-02-10T04:32:48.249 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: cluster 2021-02-10T04:32:48.249 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 2021-02-10T04:32:48.249 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 2021 2021-02-10T04:32:48.249 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: - 2021-02-10T04:32:48.249 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 02 2021-02-10T04:32:48.249 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: - 2021-02-10T04:32:48.249 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 10 2021-02-10T04:32:48.249 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: T 2021-02-10T04:32:48.249 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 04 2021-02-10T04:32:48.249 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: : 2021-02-10T04:32:48.250 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 32 2021-02-10T04:32:48.250 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: : 2021-02-10T04:32:48.250 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 48 2021-02-10T04:32:48.250 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: . 2021-02-10T04:32:48.250 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 099267 2021-02-10T04:32:48.250 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: +0000 2021-02-10T04:32:48.250 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 2021-02-10T04:32:48.250 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: mon.a 2021-02-10T04:32:48.250 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: ( 2021-02-10T04:32:48.250 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: mon 2021-02-10T04:32:48.250 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: . 2021-02-10T04:32:48.251 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 0 2021-02-10T04:32:48.251 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: ) 2021-02-10T04:32:48.251 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 1112 2021-02-10T04:32:48.251 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: : 2021-02-10T04:32:48.251 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: cluster 2021-02-10T04:32:48.251 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 2021-02-10T04:32:48.251 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: [WRN] 2021-02-10T04:32:48.251 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 2021-02-10T04:32:48.251 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: Health check update: 4 slow ops, oldest one blocked for 192 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:32:48.251 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 2021-02-10T04:32:48.251 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: cluster 2021-02-10T04:32:48.251 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 2021-02-10T04:32:48.251 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 2021 2021-02-10T04:32:48.252 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: - 2021-02-10T04:32:48.252 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 02 2021-02-10T04:32:48.252 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: - 2021-02-10T04:32:48.252 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 10 2021-02-10T04:32:48.252 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: T 2021-02-10T04:32:48.252 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 04 2021-02-10T04:32:48.252 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: : 2021-02-10T04:32:48.252 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 32 2021-02-10T04:32:48.252 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: : 2021-02-10T04:32:48.252 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 48 2021-02-10T04:32:48.252 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: . 2021-02-10T04:32:48.252 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 108088 2021-02-10T04:32:48.252 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: +0000 2021-02-10T04:32:48.253 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 2021-02-10T04:32:48.253 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: mon.a 2021-02-10T04:32:48.253 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: ( 2021-02-10T04:32:48.253 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: mon 2021-02-10T04:32:48.253 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: . 2021-02-10T04:32:48.253 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 0 2021-02-10T04:32:48.253 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: ) 2021-02-10T04:32:48.253 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 1113 2021-02-10T04:32:48.253 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: : 2021-02-10T04:32:48.253 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: cluster 2021-02-10T04:32:48.253 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 2021-02-10T04:32:48.253 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: [DBG] 2021-02-10T04:32:48.253 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 2021-02-10T04:32:48.253 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: osdmap e351: 8 total, 8 up, 8 in 2021-02-10T04:32:48.254 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[35961]: 2021-02-10T04:32:48.254 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: cluster 2021-02-10T04:32:48.254 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 2021-02-10T04:32:48.254 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 2021 2021-02-10T04:32:48.254 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: - 2021-02-10T04:32:48.254 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 02 2021-02-10T04:32:48.254 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: - 2021-02-10T04:32:48.255 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 10 2021-02-10T04:32:48.255 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: T 2021-02-10T04:32:48.255 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 04 2021-02-10T04:32:48.255 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: : 2021-02-10T04:32:48.255 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 32 2021-02-10T04:32:48.255 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: : 2021-02-10T04:32:48.255 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 46 2021-02-10T04:32:48.255 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: . 2021-02-10T04:32:48.255 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 885307 2021-02-10T04:32:48.255 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: +0000 2021-02-10T04:32:48.256 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 2021-02-10T04:32:48.256 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: osd.1 2021-02-10T04:32:48.256 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: ( 2021-02-10T04:32:48.256 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: osd 2021-02-10T04:32:48.256 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: . 2021-02-10T04:32:48.256 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 1 2021-02-10T04:32:48.256 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: ) 2021-02-10T04:32:48.256 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 443 2021-02-10T04:32:48.256 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: : 2021-02-10T04:32:48.256 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: cluster 2021-02-10T04:32:48.256 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 2021-02-10T04:32:48.256 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: [DBG] 2021-02-10T04:32:48.256 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 2021-02-10T04:32:48.256 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: purged_snaps scrub starts 2021-02-10T04:32:48.257 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 2021-02-10T04:32:48.257 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: cluster 2021-02-10T04:32:48.257 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 2021-02-10T04:32:48.257 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 2021 2021-02-10T04:32:48.257 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: - 2021-02-10T04:32:48.257 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 02 2021-02-10T04:32:48.257 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: - 2021-02-10T04:32:48.257 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 10 2021-02-10T04:32:48.257 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: T 2021-02-10T04:32:48.257 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 04 2021-02-10T04:32:48.257 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: : 2021-02-10T04:32:48.257 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 32 2021-02-10T04:32:48.257 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: : 2021-02-10T04:32:48.258 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 46 2021-02-10T04:32:48.258 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: . 2021-02-10T04:32:48.258 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 885604 2021-02-10T04:32:48.258 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: +0000 2021-02-10T04:32:48.258 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 2021-02-10T04:32:48.258 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: osd.1 2021-02-10T04:32:48.258 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: ( 2021-02-10T04:32:48.258 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: osd 2021-02-10T04:32:48.258 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: . 2021-02-10T04:32:48.258 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 1 2021-02-10T04:32:48.258 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: ) 2021-02-10T04:32:48.258 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 444 2021-02-10T04:32:48.258 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: : 2021-02-10T04:32:48.259 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: cluster 2021-02-10T04:32:48.259 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 2021-02-10T04:32:48.259 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: [DBG] 2021-02-10T04:32:48.259 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 2021-02-10T04:32:48.259 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: purged_snaps scrub ok 2021-02-10T04:32:48.259 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 2021-02-10T04:32:48.259 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: cluster 2021-02-10T04:32:48.259 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 2021-02-10T04:32:48.259 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 2021 2021-02-10T04:32:48.259 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: - 2021-02-10T04:32:48.259 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 02 2021-02-10T04:32:48.259 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: - 2021-02-10T04:32:48.259 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 10 2021-02-10T04:32:48.259 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: T 2021-02-10T04:32:48.260 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 04 2021-02-10T04:32:48.260 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: : 2021-02-10T04:32:48.260 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 32 2021-02-10T04:32:48.260 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: : 2021-02-10T04:32:48.260 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 47 2021-02-10T04:32:48.260 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: . 2021-02-10T04:32:48.260 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 131268 2021-02-10T04:32:48.260 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: +0000 2021-02-10T04:32:48.260 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 2021-02-10T04:32:48.260 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: mon.a 2021-02-10T04:32:48.260 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: ( 2021-02-10T04:32:48.260 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: mon 2021-02-10T04:32:48.260 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: . 2021-02-10T04:32:48.261 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 0 2021-02-10T04:32:48.261 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: ) 2021-02-10T04:32:48.261 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 1111 2021-02-10T04:32:48.261 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: : 2021-02-10T04:32:48.261 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: cluster 2021-02-10T04:32:48.261 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 2021-02-10T04:32:48.261 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: [DBG] 2021-02-10T04:32:48.261 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 2021-02-10T04:32:48.261 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: osdmap e350: 8 total, 8 up, 8 in 2021-02-10T04:32:48.261 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 2021-02-10T04:32:48.262 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: cluster 2021-02-10T04:32:48.262 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 2021-02-10T04:32:48.262 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 2021 2021-02-10T04:32:48.262 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: - 2021-02-10T04:32:48.262 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 02 2021-02-10T04:32:48.262 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: - 2021-02-10T04:32:48.263 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 10 2021-02-10T04:32:48.263 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: T 2021-02-10T04:32:48.263 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 04 2021-02-10T04:32:48.263 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: : 2021-02-10T04:32:48.263 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 32 2021-02-10T04:32:48.263 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: : 2021-02-10T04:32:48.264 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 48 2021-02-10T04:32:48.264 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: . 2021-02-10T04:32:48.264 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 099267 2021-02-10T04:32:48.264 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: +0000 2021-02-10T04:32:48.264 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 2021-02-10T04:32:48.264 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: mon.a 2021-02-10T04:32:48.264 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: ( 2021-02-10T04:32:48.264 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: mon 2021-02-10T04:32:48.264 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: . 2021-02-10T04:32:48.265 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 0 2021-02-10T04:32:48.265 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: ) 2021-02-10T04:32:48.265 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 1112 2021-02-10T04:32:48.265 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: : 2021-02-10T04:32:48.265 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: cluster 2021-02-10T04:32:48.265 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 2021-02-10T04:32:48.265 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: [WRN] 2021-02-10T04:32:48.265 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 2021-02-10T04:32:48.265 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: Health check update: 4 slow ops, oldest one blocked for 192 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:32:48.265 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 2021-02-10T04:32:48.265 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: cluster 2021-02-10T04:32:48.265 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 2021-02-10T04:32:48.265 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 2021 2021-02-10T04:32:48.266 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: - 2021-02-10T04:32:48.266 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 02 2021-02-10T04:32:48.266 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: - 2021-02-10T04:32:48.266 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 10 2021-02-10T04:32:48.266 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: T 2021-02-10T04:32:48.266 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 04 2021-02-10T04:32:48.266 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: : 2021-02-10T04:32:48.266 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 32 2021-02-10T04:32:48.266 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: : 2021-02-10T04:32:48.266 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 48 2021-02-10T04:32:48.266 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: . 2021-02-10T04:32:48.266 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 108088 2021-02-10T04:32:48.266 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: +0000 2021-02-10T04:32:48.267 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 2021-02-10T04:32:48.267 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: mon.a 2021-02-10T04:32:48.267 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: ( 2021-02-10T04:32:48.267 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: mon 2021-02-10T04:32:48.267 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: . 2021-02-10T04:32:48.267 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 0 2021-02-10T04:32:48.267 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: ) 2021-02-10T04:32:48.267 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 1113 2021-02-10T04:32:48.267 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: : 2021-02-10T04:32:48.267 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: cluster 2021-02-10T04:32:48.267 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 2021-02-10T04:32:48.267 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: [DBG] 2021-02-10T04:32:48.267 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 2021-02-10T04:32:48.267 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: osdmap e351: 8 total, 8 up, 8 in 2021-02-10T04:32:48.268 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:48 smithi085 conmon[29319]: 2021-02-10T04:32:48.393 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:32:47 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:32:48.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:48 smithi164 conmon[29288]: cluster 2021-02-10T04:32:46.885307 2021-02-10T04:32:48.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:48 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 443 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:32:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:48 smithi164 conmon[29288]: cluster 2021-02-10T04:32:46.885604+0000 2021-02-10T04:32:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:48 smithi164 conmon[29288]: osd.1 (osd.1) 444 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:32:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:48 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:48 smithi164 conmon[29288]: 02-10T04:32:47.131268+0000 2021-02-10T04:32:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:48 smithi164 conmon[29288]: mon.a (mon.0) 1111 : cluster [DBG] 2021-02-10T04:32:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:48 smithi164 conmon[29288]: osdmap e350: 8 total, 8 up, 8 in 2021-02-10T04:32:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:48 smithi164 conmon[29288]: cluster 2021-02-10T04:32:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:48 smithi164 conmon[29288]: 2021- 2021-02-10T04:32:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:48 smithi164 conmon[29288]: 02-10T04:32:48.099267+0000 2021-02-10T04:32:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:48 smithi164 conmon[29288]: mon.a (mon.0) 1112 : cluster [WRN] 2021-02-10T04:32:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:48 smithi164 conmon[29288]: Health check update: 4 slow ops, oldest one blocked for 192 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:32:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:48 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:32:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:48 smithi164 conmon[29288]: -10T04:32:48.108088+0000 2021-02-10T04:32:48.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:48 smithi164 conmon[29288]: mon.a (mon.0) 1113 : cluster 2021-02-10T04:32:48.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:48 smithi164 conmon[29288]: [DBG] osdmap e351: 8 total, 8 up, 8 in 2021-02-10T04:32:49.158 INFO:tasks.rados.rados.0.smithi085.stdout:1277: done (14 left) 2021-02-10T04:32:49.158 INFO:tasks.rados.rados.0.smithi085.stdout:1278: done (13 left) 2021-02-10T04:32:49.158 INFO:tasks.rados.rados.0.smithi085.stdout:1279: write (excl) oid 37 current snap is 131 2021-02-10T04:32:49.158 INFO:tasks.rados.rados.0.smithi085.stdout:1279: seq_num 466 ranges {632062=563567,1885903=527817,2830644=720758} 2021-02-10T04:32:49.159 INFO:tasks.rados.rados.0.smithi085.stdout:1279: writing smithi08575397-37 from 632062 to 1195629 tid 1 2021-02-10T04:32:49.167 INFO:tasks.rados.rados.0.smithi085.stdout:1279: writing smithi08575397-37 from 1885903 to 2413720 tid 2 2021-02-10T04:32:49.175 INFO:tasks.rados.rados.0.smithi085.stdout:1279: writing smithi08575397-37 from 2830644 to 3551402 tid 3 2021-02-10T04:32:49.175 INFO:tasks.rados.rados.0.smithi085.stdout:1280: rollback oid 2 current snap is 131 2021-02-10T04:32:49.175 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 2 to 125 2021-02-10T04:32:49.175 INFO:tasks.rados.rados.0.smithi085.stdout:1279: finishing write tid 1 to smithi08575397-37 2021-02-10T04:32:49.176 INFO:tasks.rados.rados.0.smithi085.stdout:1279: finishing write tid 2 to smithi08575397-37 2021-02-10T04:32:49.176 INFO:tasks.rados.rados.0.smithi085.stdout:1281: write (excl) oid 7 current snap is 131 2021-02-10T04:32:49.176 INFO:tasks.rados.rados.0.smithi085.stdout:1281: seq_num 467 ranges {648869=638816,1828032=787828,3141714=532030} 2021-02-10T04:32:49.182 INFO:tasks.rados.rados.0.smithi085.stdout:1281: writing smithi08575397-7 from 648869 to 1287685 tid 1 2021-02-10T04:32:49.191 INFO:tasks.rados.rados.0.smithi085.stdout:1281: writing smithi08575397-7 from 1828032 to 2615860 tid 2 2021-02-10T04:32:49.198 INFO:tasks.rados.rados.0.smithi085.stdout:1281: writing smithi08575397-7 from 3141714 to 3673744 tid 3 2021-02-10T04:32:49.198 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:49.199 INFO:tasks.rados.rados.0.smithi085.stdout:1280: finishing rollback tid 0 to smithi08575397-2 2021-02-10T04:32:49.199 INFO:tasks.rados.rados.0.smithi085.stdout:1280: finishing rollback tid 1 to smithi08575397-2 2021-02-10T04:32:49.199 INFO:tasks.rados.rados.0.smithi085.stdout:1280: finishing rollback tid 2 to smithi08575397-2 2021-02-10T04:32:49.199 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 603 (ObjNum 422 snap 109 seq_num 422) dirty exists 2021-02-10T04:32:49.199 INFO:tasks.rados.rados.0.smithi085.stdout:1279: finishing write tid 3 to smithi08575397-37 2021-02-10T04:32:49.199 INFO:tasks.rados.rados.0.smithi085.stdout:1280: done (15 left) 2021-02-10T04:32:49.199 INFO:tasks.rados.rados.0.smithi085.stdout:1282: rollback oid 2 current snap is 131 2021-02-10T04:32:49.199 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 2 to 122 2021-02-10T04:32:49.199 INFO:tasks.rados.rados.0.smithi085.stdout:1279: finishing write tid 4 to smithi08575397-37 2021-02-10T04:32:49.199 INFO:tasks.rados.rados.0.smithi085.stdout:1279: finishing write tid 5 to smithi08575397-37 2021-02-10T04:32:49.199 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 715 (ObjNum 466 snap 131 seq_num 466) dirty exists 2021-02-10T04:32:49.199 INFO:tasks.rados.rados.0.smithi085.stdout:1279: left oid 37 (ObjNum 466 snap 131 seq_num 466) 2021-02-10T04:32:49.199 INFO:tasks.rados.rados.0.smithi085.stdout:1281: finishing write tid 1 to smithi08575397-7 2021-02-10T04:32:49.200 INFO:tasks.rados.rados.0.smithi085.stdout:1279: done (15 left) 2021-02-10T04:32:49.200 INFO:tasks.rados.rados.0.smithi085.stdout:1283: delete oid 46 current snap is 131 2021-02-10T04:32:49.200 INFO:tasks.rados.rados.0.smithi085.stdout:1281: finishing write tid 2 to smithi08575397-7 2021-02-10T04:32:49.202 INFO:tasks.rados.rados.0.smithi085.stdout:1283: done (15 left) 2021-02-10T04:32:49.203 INFO:tasks.rados.rados.0.smithi085.stdout:1284: snap_create 2021-02-10T04:32:49.203 INFO:tasks.rados.rados.0.smithi085.stdout:1282: finishing rollback tid 0 to smithi08575397-2 2021-02-10T04:32:49.203 INFO:tasks.rados.rados.0.smithi085.stdout:1282: finishing rollback tid 1 to smithi08575397-2 2021-02-10T04:32:49.203 INFO:tasks.rados.rados.0.smithi085.stdout:1282: finishing rollback tid 2 to smithi08575397-2 2021-02-10T04:32:49.203 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 606 (ObjNum 422 snap 109 seq_num 422) dirty exists 2021-02-10T04:32:49.204 INFO:tasks.rados.rados.0.smithi085.stdout:1281: finishing write tid 3 to smithi08575397-7 2021-02-10T04:32:49.204 INFO:tasks.rados.rados.0.smithi085.stdout:1281: finishing write tid 4 to smithi08575397-7 2021-02-10T04:32:49.204 INFO:tasks.rados.rados.0.smithi085.stdout:1281: finishing write tid 5 to smithi08575397-7 2021-02-10T04:32:49.204 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 599 (ObjNum 467 snap 131 seq_num 467) dirty exists 2021-02-10T04:32:49.204 INFO:tasks.rados.rados.0.smithi085.stdout:1281: left oid 7 (ObjNum 467 snap 131 seq_num 467) 2021-02-10T04:32:49.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:49 smithi085 conmon[29319]: audit 2021-02-10T04:32:49.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:49 smithi085 conmon[29319]: 2021-02-10T04:32:47.924994+0000 mgr.y 2021-02-10T04:32:49.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:49 smithi085 conmon[29319]: (mgr.14140) 493 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:32:49.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:49 smithi085 conmon[29319]: cluster 2021-02-10T04:32:48 2021-02-10T04:32:49.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:49 smithi085 conmon[29319]: .062187+0000 mgr.y (mgr.14140) 494 : cluster [DBG] pgmap v675: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim, 163 active+clean; 196 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2021-02-10T04:32:49.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:49 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:49.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:49 smithi085 conmon[29319]: 10T04:32:48.128666 2021-02-10T04:32:49.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:49 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1114 2021-02-10T04:32:49.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:49 smithi085 conmon[29319]: : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 192 sec, osd.4 has slow ops) 2021-02-10T04:32:49.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:49 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:32:49.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:49 smithi085 conmon[29319]: T04:32:48.128730+0000 mon.a (mon.0) 1115 : cluster [INF] Cluster is now healthy 2021-02-10T04:32:49.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:49 smithi085 conmon[35961]: audit 2021-02-10T04:32:47. 2021-02-10T04:32:49.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:49 smithi085 conmon[35961]: 924994+0000 mgr.y (mgr.14140) 493 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:32:49.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:49 smithi085 conmon[35961]: cluster 2021-02-10T04:32:48.062187+0000 mgr.y (mgr.14140) 494 : cluster [DBG] pgmap v675: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim, 163 active+clean; 196 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2021-02-10T04:32:49.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:49 smithi085 conmon[35961]: cluster 2021-02-10T04:32:48.128666+0000 mon.a (mon.0) 1114 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 192 sec, osd.4 has slow ops) 2021-02-10T04:32:49.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:49 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:49.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:49 smithi085 conmon[35961]: 02-10T04:32:48.128730+0000 mon.a (mon.0) 1115 : cluster [INF] Cluster is now healthy 2021-02-10T04:32:49.449 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:32:49.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:49 smithi164 conmon[29288]: audit 2021-02-10T04:32:47.924994+0000 mgr.y ( 2021-02-10T04:32:49.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:49 smithi164 conmon[29288]: mgr.14140) 493 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:32:49.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:49 smithi164 conmon[29288]: cluster 2021-02-10T04:32:48 2021-02-10T04:32:49.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:49 smithi164 conmon[29288]: .062187+0000 mgr.y (mgr.14140) 494 : cluster [DBG] pgmap v675: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim, 163 active+clean; 196 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2021-02-10T04:32:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:49 smithi164 conmon[29288]: cluster 2021-02-10T04:32:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:49 smithi164 conmon[29288]: 2021-02-10T04:32:48.128666 2021-02-10T04:32:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:49 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1114 : cluster 2021-02-10T04:32:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:49 smithi164 conmon[29288]: [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 192 sec, osd.4 has slow ops) 2021-02-10T04:32:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:49 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:49 smithi164 conmon[29288]: 02-10T04:32:48.128730+0000 mon.a ( 2021-02-10T04:32:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:49 smithi164 conmon[29288]: mon.0) 1115 : cluster [INF] Cluster is now healthy 2021-02-10T04:32:49.745 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:32:50.163 INFO:tasks.rados.rados.0.smithi085.stdout:1281: done (15 left) 2021-02-10T04:32:50.163 INFO:tasks.rados.rados.0.smithi085.stdout:1282: done (14 left) 2021-02-10T04:32:50.163 INFO:tasks.rados.rados.0.smithi085.stdout:1284: done (13 left) 2021-02-10T04:32:50.163 INFO:tasks.rados.rados.0.smithi085.stdout:1285: rollback oid 46 current snap is 132 2021-02-10T04:32:50.163 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 46 to 120 2021-02-10T04:32:50.163 INFO:tasks.rados.rados.0.smithi085.stdout:1286: rollback oid 8 current snap is 132 2021-02-10T04:32:50.163 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 8 to 122 2021-02-10T04:32:50.163 INFO:tasks.rados.rados.0.smithi085.stdout:1287: write (excl) oid 3 current snap is 132 2021-02-10T04:32:50.163 INFO:tasks.rados.rados.0.smithi085.stdout:1287: seq_num 469 ranges {682483=789314,1944996=791497,3196560=554574} 2021-02-10T04:32:50.169 INFO:tasks.rados.rados.0.smithi085.stdout:1287: writing smithi08575397-3 from 682483 to 1471797 tid 1 2021-02-10T04:32:50.177 INFO:tasks.rados.rados.0.smithi085.stdout:1287: writing smithi08575397-3 from 1944996 to 2736493 tid 2 2021-02-10T04:32:50.185 INFO:tasks.rados.rados.0.smithi085.stdout:1287: writing smithi08575397-3 from 3196560 to 3751134 tid 3 2021-02-10T04:32:50.185 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:50.185 INFO:tasks.rados.rados.0.smithi085.stdout:1286: finishing rollback tid 0 to smithi08575397-8 2021-02-10T04:32:50.185 INFO:tasks.rados.rados.0.smithi085.stdout:1286: finishing rollback tid 1 to smithi08575397-8 2021-02-10T04:32:50.185 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 593 (ObjNum 1953066355 snap 942696808 seq_num 859125557) dirty dne 2021-02-10T04:32:50.186 INFO:tasks.rados.rados.0.smithi085.stdout:1285: finishing rollback tid 1 to smithi08575397-46 2021-02-10T04:32:50.186 INFO:tasks.rados.rados.0.smithi085.stdout:1286: done (15 left) 2021-02-10T04:32:50.186 INFO:tasks.rados.rados.0.smithi085.stdout:1288: write oid 7 current snap is 132 2021-02-10T04:32:50.186 INFO:tasks.rados.rados.0.smithi085.stdout:1285: finishing rollback tid 2 to smithi08575397-46 2021-02-10T04:32:50.186 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 639 (ObjNum 382 snap 100 seq_num 382) dirty exists 2021-02-10T04:32:50.187 INFO:tasks.rados.rados.0.smithi085.stdout:1288: seq_num 470 ranges {699290=464563,1603478=535155,2823983=765846} 2021-02-10T04:32:50.194 INFO:tasks.rados.rados.0.smithi085.stdout:1288: writing smithi08575397-7 from 699290 to 1163853 tid 1 2021-02-10T04:32:50.197 INFO:tasks.rados.rados.0.smithi085.stdout:1288: writing smithi08575397-7 from 1603478 to 2138633 tid 2 2021-02-10T04:32:50.208 INFO:tasks.rados.rados.0.smithi085.stdout:1288: writing smithi08575397-7 from 2823983 to 3589829 tid 3 2021-02-10T04:32:50.208 INFO:tasks.rados.rados.0.smithi085.stdout:1285: done (15 left) 2021-02-10T04:32:50.208 INFO:tasks.rados.rados.0.smithi085.stdout:1289: read oid 13 snap -1 2021-02-10T04:32:50.208 INFO:tasks.rados.rados.0.smithi085.stdout:1289: expect (ObjNum 457 snap 127 seq_num 457) 2021-02-10T04:32:50.208 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:50.208 INFO:tasks.rados.rados.0.smithi085.stdout:1287: finishing write tid 1 to smithi08575397-3 2021-02-10T04:32:50.208 INFO:tasks.rados.rados.0.smithi085.stdout:1287: finishing write tid 2 to smithi08575397-3 2021-02-10T04:32:50.208 INFO:tasks.rados.rados.0.smithi085.stdout:1287: finishing write tid 3 to smithi08575397-3 2021-02-10T04:32:50.209 INFO:tasks.rados.rados.0.smithi085.stdout:1287: finishing write tid 4 to smithi08575397-3 2021-02-10T04:32:50.209 INFO:tasks.rados.rados.0.smithi085.stdout:1287: finishing write tid 5 to smithi08575397-3 2021-02-10T04:32:50.209 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 603 (ObjNum 469 snap 132 seq_num 469) dirty exists 2021-02-10T04:32:50.209 INFO:tasks.rados.rados.0.smithi085.stdout:1287: left oid 3 (ObjNum 469 snap 132 seq_num 469) 2021-02-10T04:32:50.209 INFO:tasks.rados.rados.0.smithi085.stdout:1288: finishing write tid 1 to smithi08575397-7 2021-02-10T04:32:50.209 INFO:tasks.rados.rados.0.smithi085.stdout:1287: done (15 left) 2021-02-10T04:32:50.209 INFO:tasks.rados.rados.0.smithi085.stdout:1290: rollback oid 38 current snap is 132 2021-02-10T04:32:50.209 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 38 to 118 2021-02-10T04:32:50.209 INFO:tasks.rados.rados.0.smithi085.stdout:1288: finishing write tid 2 to smithi08575397-7 2021-02-10T04:32:50.209 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:50.211 INFO:tasks.rados.rados.0.smithi085.stdout:1290: finishing rollback tid 0 to smithi08575397-38 2021-02-10T04:32:50.211 INFO:tasks.rados.rados.0.smithi085.stdout:1290: finishing rollback tid 1 to smithi08575397-38 2021-02-10T04:32:50.211 INFO:tasks.rados.rados.0.smithi085.stdout:1290: finishing rollback tid 2 to smithi08575397-38 2021-02-10T04:32:50.211 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 642 (ObjNum 287 snap 71 seq_num 287) dirty exists 2021-02-10T04:32:50.211 INFO:tasks.rados.rados.0.smithi085.stdout:1290: done (15 left) 2021-02-10T04:32:50.211 INFO:tasks.rados.rados.0.smithi085.stdout:1291: snap_create 2021-02-10T04:32:50.212 INFO:tasks.rados.rados.0.smithi085.stdout:1288: finishing write tid 3 to smithi08575397-7 2021-02-10T04:32:50.212 INFO:tasks.rados.rados.0.smithi085.stdout:1288: finishing write tid 4 to smithi08575397-7 2021-02-10T04:32:50.213 INFO:tasks.rados.rados.0.smithi085.stdout:1288: finishing write tid 5 to smithi08575397-7 2021-02-10T04:32:50.213 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 607 (ObjNum 470 snap 132 seq_num 470) dirty exists 2021-02-10T04:32:50.213 INFO:tasks.rados.rados.0.smithi085.stdout:1288: left oid 7 (ObjNum 470 snap 132 seq_num 470) 2021-02-10T04:32:50.219 INFO:tasks.rados.rados.0.smithi085.stdout:1289: expect (ObjNum 457 snap 127 seq_num 457) 2021-02-10T04:32:50.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:50 smithi085 conmon[29319]: cluster 2021-02-10T04:32:49.152624+0000 mon.a 2021-02-10T04:32:50.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:50 smithi085 conmon[29319]: (mon.0) 1116 : cluster [DBG] osdmap e352: 8 total, 8 up, 8 in 2021-02-10T04:32:50.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:50 smithi085 conmon[29319]: cluster 2021-02-10T04:32:50.062654+0000 mgr.y (mgr.14140) 495 : cluster 2021-02-10T04:32:50.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:50 smithi085 conmon[29319]: [DBG] pgmap v678: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim, 161 active+clean; 216 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 4.5 MiB/s wr, 17 op/s 2021-02-10T04:32:50.441 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:50 smithi085 conmon[45443]: debug 2021-02-10T04:32:50.300+0000 7f7125cf8700 -1 osd.1 pg_epoch: 353 pg[4.4( v 63'31 (0'0,63'31] local-lis/les=55/56 n=14 ec=55/55 lis/c=55/55 les/c/f=56/56/0 sis=55) [1,2,3] r=0 lpr=55 crt=63'31 lcod 63'30 mlcod 63'30 active+clean] scrubber pg(4.4) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 3 (0x55dbc83d9ce0) 2021-02-10T04:32:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:50 smithi085 conmon[35961]: cluster 2021-02-10T04:32:49.152624 2021-02-10T04:32:50.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:50 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1116 : cluster [DBG] osdmap e352: 8 total, 8 up, 8 in 2021-02-10T04:32:50.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:50 smithi085 conmon[35961]: cluster 2021-02-10T04:32: 2021-02-10T04:32:50.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:50 smithi085 conmon[35961]: 50.062654+0000 mgr.y (mgr.14140) 495 : cluster [DBG] pgmap v678: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim, 161 active+clean; 216 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 4.5 MiB/s wr, 17 op/s 2021-02-10T04:32:50.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:50 smithi164 conmon[29288]: cluster 2021-02-10T04:32:49.152624+0000 mon.a 2021-02-10T04:32:50.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:50 smithi164 conmon[29288]: (mon.0) 1116 : cluster [DBG] osdmap e352: 8 total, 8 up, 8 in 2021-02-10T04:32:50.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:50 smithi164 conmon[29288]: cluster 2021-02-10T04:32:50.062654+0000 mgr.y (mgr.14140) 495 : cluster [DBG] pgmap v678: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim, 161 active+clean; 216 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 4.5 MiB/s wr, 17 op/s 2021-02-10T04:32:51.163 INFO:tasks.rados.rados.0.smithi085.stdout:1288: done (15 left) 2021-02-10T04:32:51.163 INFO:tasks.rados.rados.0.smithi085.stdout:1289: done (14 left) 2021-02-10T04:32:51.163 INFO:tasks.rados.rados.0.smithi085.stdout:1291: done (13 left) 2021-02-10T04:32:51.164 INFO:tasks.rados.rados.0.smithi085.stdout:1292: write oid 2 current snap is 133 2021-02-10T04:32:51.164 INFO:tasks.rados.rados.0.smithi085.stdout:1292: seq_num 471 ranges {716097=539812,1945607=795166,3535053=381044} 2021-02-10T04:32:51.170 INFO:tasks.rados.rados.0.smithi085.stdout:1292: writing smithi08575397-2 from 716097 to 1255909 tid 1 2021-02-10T04:32:51.179 INFO:tasks.rados.rados.0.smithi085.stdout:1292: writing smithi08575397-2 from 1945607 to 2740773 tid 2 2021-02-10T04:32:51.187 INFO:tasks.rados.rados.0.smithi085.stdout:1292: writing smithi08575397-2 from 3535053 to 3916097 tid 3 2021-02-10T04:32:51.187 INFO:tasks.rados.rados.0.smithi085.stdout:1292: finishing write tid 1 to smithi08575397-2 2021-02-10T04:32:51.187 INFO:tasks.rados.rados.0.smithi085.stdout:1293: read oid 5 snap -1 2021-02-10T04:32:51.187 INFO:tasks.rados.rados.0.smithi085.stdout:1293: expect (ObjNum 460 snap 130 seq_num 460) 2021-02-10T04:32:51.187 INFO:tasks.rados.rados.0.smithi085.stdout:1294: snap_create 2021-02-10T04:32:51.187 INFO:tasks.rados.rados.0.smithi085.stdout:1292: finishing write tid 2 to smithi08575397-2 2021-02-10T04:32:51.190 INFO:tasks.rados.rados.0.smithi085.stdout:1292: finishing write tid 3 to smithi08575397-2 2021-02-10T04:32:51.190 INFO:tasks.rados.rados.0.smithi085.stdout:1292: finishing write tid 4 to smithi08575397-2 2021-02-10T04:32:51.191 INFO:tasks.rados.rados.0.smithi085.stdout:1292: finishing write tid 5 to smithi08575397-2 2021-02-10T04:32:51.191 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 610 (ObjNum 471 snap 133 seq_num 471) dirty exists 2021-02-10T04:32:51.192 INFO:tasks.rados.rados.0.smithi085.stdout:1292: left oid 2 (ObjNum 471 snap 133 seq_num 471) 2021-02-10T04:32:51.200 INFO:tasks.rados.rados.0.smithi085.stdout:1293: expect (ObjNum 460 snap 130 seq_num 460) 2021-02-10T04:32:51.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:51 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:32:51.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:51 smithi085 conmon[29319]: 04:32:50.158601+0000 mon.a (mon.0) 1117 : cluster [DBG] osdmap e353: 8 total, 8 up, 8 in 2021-02-10T04:32:51.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:51 smithi085 conmon[35961]: cluster 2021-02-10T04:32:51.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:51 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:32:51.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:51 smithi085 conmon[35961]: 10T04:32:50.158601+0000 mon.a (mon.0) 1117 : cluster [DBG] osdmap e353: 8 total, 8 up, 8 in 2021-02-10T04:32:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:51 smithi164 conmon[29288]: cluster 2021-02-10T04:32:50.158601+0000 mon.a (mon.0) 1117 : cluster 2021-02-10T04:32:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:51 smithi164 conmon[29288]: [DBG] osdmap e353: 8 total, 8 up, 8 in 2021-02-10T04:32:52.170 INFO:tasks.rados.rados.0.smithi085.stdout:1292: done (15 left) 2021-02-10T04:32:52.170 INFO:tasks.rados.rados.0.smithi085.stdout:1293: done (14 left) 2021-02-10T04:32:52.170 INFO:tasks.rados.rados.0.smithi085.stdout:1294: done (13 left) 2021-02-10T04:32:52.170 INFO:tasks.rados.rados.0.smithi085.stdout:1295: delete oid 30 current snap is 134 2021-02-10T04:32:52.178 INFO:tasks.rados.rados.0.smithi085.stdout:1295: done (13 left) 2021-02-10T04:32:52.178 INFO:tasks.rados.rados.0.smithi085.stdout:1296: copy_from oid 11 from oid 13 current snap is 134 2021-02-10T04:32:52.178 INFO:tasks.rados.rados.0.smithi085.stdout:1297: write (excl) oid 3 current snap is 134 2021-02-10T04:32:52.178 INFO:tasks.rados.rados.0.smithi085.stdout:1297: seq_num 473 ranges {749711=406663,1778924=682482,3189899=716015} 2021-02-10T04:32:52.185 INFO:tasks.rados.rados.0.smithi085.stdout:1297: writing smithi08575397-3 from 749711 to 1156374 tid 1 2021-02-10T04:32:52.194 INFO:tasks.rados.rados.0.smithi085.stdout:1297: writing smithi08575397-3 from 1778924 to 2461406 tid 2 2021-02-10T04:32:52.204 INFO:tasks.rados.rados.0.smithi085.stdout:1297: writing smithi08575397-3 from 3189899 to 3905914 tid 3 2021-02-10T04:32:52.205 INFO:tasks.rados.rados.0.smithi085.stdout:1298: read oid 22 snap -1 2021-02-10T04:32:52.205 INFO:tasks.rados.rados.0.smithi085.stdout:1298: expect (ObjNum 464 snap 130 seq_num 464) 2021-02-10T04:32:52.205 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:52.205 INFO:tasks.rados.rados.0.smithi085.stdout:1297: finishing write tid 1 to smithi08575397-3 2021-02-10T04:32:52.205 INFO:tasks.rados.rados.0.smithi085.stdout:1296: finishing copy_from to smithi08575397-11 2021-02-10T04:32:52.205 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 582 (ObjNum 457 snap 127 seq_num 457) dirty exists 2021-02-10T04:32:52.205 INFO:tasks.rados.rados.0.smithi085.stdout:1296: finishing copy_from racing read to smithi08575397-11 2021-02-10T04:32:52.205 INFO:tasks.rados.rados.0.smithi085.stdout:1297: finishing write tid 2 to smithi08575397-3 2021-02-10T04:32:52.205 INFO:tasks.rados.rados.0.smithi085.stdout:1296: done (15 left) 2021-02-10T04:32:52.205 INFO:tasks.rados.rados.0.smithi085.stdout:1299: read oid 36 snap 118 2021-02-10T04:32:52.205 INFO:tasks.rados.rados.0.smithi085.stdout:1299: expect (ObjNum 421 snap 109 seq_num 421) 2021-02-10T04:32:52.205 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:52.211 INFO:tasks.rados.rados.0.smithi085.stdout:1297: finishing write tid 3 to smithi08575397-3 2021-02-10T04:32:52.211 INFO:tasks.rados.rados.0.smithi085.stdout:1297: finishing write tid 4 to smithi08575397-3 2021-02-10T04:32:52.212 INFO:tasks.rados.rados.0.smithi085.stdout:1299: expect (ObjNum 421 snap 109 seq_num 421) 2021-02-10T04:32:52.441 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:52 smithi085 conmon[45443]: debug 2021-02-10T04:32:52.282+0000 7f7125cf8700 -1 osd.1 pg_epoch: 355 pg[4.4( v 63'31 (0'0,63'31] local-lis/les=55/56 n=14 ec=55/55 lis/c=55/55 les/c/f=56/56/0 sis=55) [1,2,3] r=0 lpr=55 crt=63'31 lcod 63'30 mlcod 63'30 active+clean] scrubber pg(4.4) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 3 (0x55dbc86ade40) 2021-02-10T04:32:52.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:52 smithi085 conmon[29319]: cluster 2021-02-10T04:32:51.163265+0000 mon.a (mon.0) 1118 : cluster [DBG] 2021-02-10T04:32:52.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:52 smithi085 conmon[29319]: osdmap e354: 8 total, 8 up, 8 in 2021-02-10T04:32:52.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:52 smithi085 conmon[29319]: cluster 2021-02-10T04:32:52.063127+0000 mgr.y (mgr.14140) 496 : cluster [DBG] pgmap v681: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim, 162 active+clean; 230 MiB data, 746 MiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 7.0 MiB/s wr, 27 op/s 2021-02-10T04:32:52.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:52 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:52.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:52 smithi085 conmon[35961]: 10T04:32:51.163265+0000 mon.a (mon.0) 1118 : cluster [DBG] osdmap e354: 8 total, 8 up, 8 in 2021-02-10T04:32:52.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:52 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:52.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:52 smithi085 conmon[35961]: 02-10T04:32:52.063127+0000 mgr.y (mgr. 2021-02-10T04:32:52.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:52 smithi085 conmon[35961]: 14140) 496 : cluster [DBG] pgmap v681: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim, 162 active+clean; 230 MiB data, 746 MiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 7.0 MiB/s wr, 27 op/s 2021-02-10T04:32:52.494 INFO:tasks.rados.rados.0.smithi085.stdout:1297: finishing write tid 5 to smithi08575397-3 2021-02-10T04:32:52.494 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 611 (ObjNum 473 snap 134 seq_num 473) dirty exists 2021-02-10T04:32:52.494 INFO:tasks.rados.rados.0.smithi085.stdout:1297: left oid 3 (ObjNum 473 snap 134 seq_num 473) 2021-02-10T04:32:52.494 INFO:tasks.rados.rados.0.smithi085.stdout:1298: expect (ObjNum 464 snap 130 seq_num 464) 2021-02-10T04:32:52.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:52 smithi164 conmon[29288]: cluster 2021-02-10T04:32:51.163265+0000 2021-02-10T04:32:52.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:52 smithi164 conmon[29288]: mon.a (mon.0) 1118 : cluster [DBG] osdmap e354: 8 total, 8 up, 8 in 2021-02-10T04:32:52.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:52 smithi164 conmon[29288]: cluster 2021-02-10T04:32:52.063127+0000 mgr.y (mgr.14140) 496 : cluster [DBG] pgmap v681: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim, 162 active+clean; 230 MiB data, 746 MiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 7.0 MiB/s wr, 27 op/s 2021-02-10T04:32:52.894 INFO:tasks.rados.rados.0.smithi085.stdout:1297: done (15 left) 2021-02-10T04:32:52.895 INFO:tasks.rados.rados.0.smithi085.stdout:1298: done (14 left) 2021-02-10T04:32:52.895 INFO:tasks.rados.rados.0.smithi085.stdout:1299: done (13 left) 2021-02-10T04:32:52.895 INFO:tasks.rados.rados.0.smithi085.stdout:1300: copy_from oid 46 from oid 36 current snap is 134 2021-02-10T04:32:52.895 INFO:tasks.rados.rados.0.smithi085.stdout:1301: read oid 23 snap 128 2021-02-10T04:32:52.895 INFO:tasks.rados.rados.0.smithi085.stdout:1301: expect (ObjNum 418 snap 109 seq_num 418) 2021-02-10T04:32:52.895 INFO:tasks.rados.rados.0.smithi085.stdout:1302: snap_remove snap 118 2021-02-10T04:32:52.941 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:52 smithi085 conmon[45443]: debug 2021-02-10T04:32:52.521+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:53.071 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:32:53.187 INFO:tasks.rados.rados.0.smithi085.stdout:1302: done (15 left) 2021-02-10T04:32:53.187 INFO:tasks.rados.rados.0.smithi085.stdout:1303: rollback oid 50 current snap is 134 2021-02-10T04:32:53.187 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 50 to 126 2021-02-10T04:32:53.187 INFO:tasks.rados.rados.0.smithi085.stdout:1301: expect (ObjNum 418 snap 109 seq_num 418) 2021-02-10T04:32:53.379 INFO:tasks.rados.rados.0.smithi085.stdout:1300: finishing copy_from to smithi08575397-46 2021-02-10T04:32:53.380 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 616 (ObjNum 432 snap 114 seq_num 432) dirty exists 2021-02-10T04:32:53.380 INFO:tasks.rados.rados.0.smithi085.stdout:1300: finishing copy_from racing read to smithi08575397-46 2021-02-10T04:32:53.380 INFO:tasks.rados.rados.0.smithi085.stdout:1300: done (15 left) 2021-02-10T04:32:53.380 INFO:tasks.rados.rados.0.smithi085.stdout:1301: done (14 left) 2021-02-10T04:32:53.380 INFO:tasks.rados.rados.0.smithi085.stdout:1304: write (excl) oid 43 current snap is 134 2021-02-10T04:32:53.380 INFO:tasks.rados.rados.0.smithi085.stdout:1303: finishing rollback tid 1 to smithi08575397-50 2021-02-10T04:32:53.380 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-02-10T04:32:53.380 INFO:tasks.rados.rados.0.smithi085.stdout:1304: seq_num 474 ranges {766518=481912,1837406=542493,2933675=527287,3876584=89934} 2021-02-10T04:32:53.387 INFO:tasks.rados.rados.0.smithi085.stdout:1304: writing smithi08575397-43 from 766518 to 1248430 tid 1 2021-02-10T04:32:53.394 INFO:tasks.rados.rados.0.smithi085.stdout:1304: writing smithi08575397-43 from 1837406 to 2379899 tid 2 2021-02-10T04:32:53.401 INFO:tasks.rados.rados.0.smithi085.stdout:1304: writing smithi08575397-43 from 2933675 to 3460962 tid 3 2021-02-10T04:32:53.404 INFO:tasks.rados.rados.0.smithi085.stdout:1304: writing smithi08575397-43 from 3876584 to 3966518 tid 4 2021-02-10T04:32:53.404 INFO:tasks.rados.rados.0.smithi085.stdout:1303: done (14 left) 2021-02-10T04:32:53.404 INFO:tasks.rados.rados.0.smithi085.stdout:1305: snap_remove snap 126 2021-02-10T04:32:53.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:53 smithi085 conmon[29319]: cluster 2021-02-10T04:32:52.168766+0000 mon.a (mon.0 2021-02-10T04:32:53.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:53 smithi085 conmon[29319]: ) 1119 : cluster [DBG] osdmap e355: 8 total, 8 up, 8 in 2021-02-10T04:32:53.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:53 smithi085 conmon[29319]: cephadm 2021-02-10T04:32:53.061185+0000 mgr.y (mgr.14140) 497 : cephadm [INF] 2021-02-10T04:32:53.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:53 smithi085 conmon[29319]: Refreshing smithi085 facts 2021-02-10T04:32:53.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:53 smithi085 conmon[35961]: cluster 2021-02-10T04:32:52.168766+0000 2021-02-10T04:32:53.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:53 smithi085 conmon[35961]: mon.a (mon.0) 1119 : cluster [DBG] osdmap e355: 8 total, 8 up, 8 in 2021-02-10T04:32:53.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:53 smithi085 conmon[35961]: cephadm 2021-02 2021-02-10T04:32:53.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:53 smithi085 conmon[35961]: -10T04:32:53.061185 2021-02-10T04:32:53.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:53 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 497 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:32:53.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:53 smithi164 conmon[29288]: cluster 2021-02-10T04:32:53.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:53 smithi164 conmon[29288]: 2021-02-10T04:32: 2021-02-10T04:32:53.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:53 smithi164 conmon[29288]: 52.168766+0000 mon.a (mon.0) 1119 : cluster [DBG] osdmap e355: 8 total, 8 up, 8 in 2021-02-10T04:32:53.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:53 smithi164 conmon[29288]: cephadm 2021-02- 2021-02-10T04:32:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:53 smithi164 conmon[29288]: 10T04:32:53.061185+0000 mgr.y (mgr.14140) 497 : 2021-02-10T04:32:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:53 smithi164 conmon[29288]: cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:32:53.844 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:32:53.884 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:54.143 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:32:53 smithi164 conmon[40892]: debug 2021-02-10T04:32:54.143 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:32:53 smithi164 conmon[40892]: 2021-02-10T04:32:53.973+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:54.195 INFO:tasks.rados.rados.0.smithi085.stdout:1305: done (14 left) 2021-02-10T04:32:54.195 INFO:tasks.rados.rados.0.smithi085.stdout:1306: write oid 28 current snap is 134 2021-02-10T04:32:54.195 INFO:tasks.rados.rados.0.smithi085.stdout:1304: finishing write tid 1 to smithi08575397-43 2021-02-10T04:32:54.195 INFO:tasks.rados.rados.0.smithi085.stdout:1304: finishing write tid 2 to smithi08575397-43 2021-02-10T04:32:54.196 INFO:tasks.rados.rados.0.smithi085.stdout:1304: finishing write tid 3 to smithi08575397-43 2021-02-10T04:32:54.196 INFO:tasks.rados.rados.0.smithi085.stdout:1304: finishing write tid 4 to smithi08575397-43 2021-02-10T04:32:54.196 INFO:tasks.rados.rados.0.smithi085.stdout:1304: finishing write tid 5 to smithi08575397-43 2021-02-10T04:32:54.196 INFO:tasks.rados.rados.0.smithi085.stdout:1304: finishing write tid 6 to smithi08575397-43 2021-02-10T04:32:54.196 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 616 (ObjNum 474 snap 134 seq_num 474) dirty exists 2021-02-10T04:32:54.196 INFO:tasks.rados.rados.0.smithi085.stdout:1304: left oid 43 (ObjNum 474 snap 134 seq_num 474) 2021-02-10T04:32:54.202 INFO:tasks.rados.rados.0.smithi085.stdout:1306: seq_num 475 ranges {783325=557161,1779535=686151,3244745=454912} 2021-02-10T04:32:54.203 INFO:tasks.rados.rados.0.smithi085.stdout:1306: writing smithi08575397-28 from 783325 to 1340486 tid 1 2021-02-10T04:32:54.210 INFO:tasks.rados.rados.0.smithi085.stdout:1306: writing smithi08575397-28 from 1779535 to 2465686 tid 2 2021-02-10T04:32:54.218 INFO:tasks.rados.rados.0.smithi085.stdout:1306: writing smithi08575397-28 from 3244745 to 3699657 tid 3 2021-02-10T04:32:54.219 INFO:tasks.rados.rados.0.smithi085.stdout:1304: done (14 left) 2021-02-10T04:32:54.219 INFO:tasks.rados.rados.0.smithi085.stdout:1307: copy_from oid 5 from oid 1 current snap is 134 2021-02-10T04:32:54.219 INFO:tasks.rados.rados.0.smithi085.stdout:1306: finishing write tid 1 to smithi08575397-28 2021-02-10T04:32:54.219 INFO:tasks.rados.rados.0.smithi085.stdout:1306: finishing write tid 2 to smithi08575397-28 2021-02-10T04:32:54.219 INFO:tasks.rados.rados.0.smithi085.stdout:1308: read oid 2 snap -1 2021-02-10T04:32:54.219 INFO:tasks.rados.rados.0.smithi085.stdout:1308: expect (ObjNum 471 snap 133 seq_num 471) 2021-02-10T04:32:54.219 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:54.222 INFO:tasks.rados.rados.0.smithi085.stdout:1306: finishing write tid 3 to smithi08575397-28 2021-02-10T04:32:54.222 INFO:tasks.rados.rados.0.smithi085.stdout:1306: finishing write tid 4 to smithi08575397-28 2021-02-10T04:32:54.223 INFO:tasks.rados.rados.0.smithi085.stdout:1306: finishing write tid 5 to smithi08575397-28 2021-02-10T04:32:54.223 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 619 (ObjNum 475 snap 134 seq_num 475) dirty exists 2021-02-10T04:32:54.223 INFO:tasks.rados.rados.0.smithi085.stdout:1306: left oid 28 (ObjNum 475 snap 134 seq_num 475) 2021-02-10T04:32:54.223 INFO:tasks.rados.rados.0.smithi085.stdout:1306: done (15 left) 2021-02-10T04:32:54.223 INFO:tasks.rados.rados.0.smithi085.stdout:1309: copy_from oid 31 from oid 22 current snap is 134 2021-02-10T04:32:54.223 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:54.227 INFO:tasks.rados.rados.0.smithi085.stdout:1308: expect (ObjNum 471 snap 133 seq_num 471) 2021-02-10T04:32:54.316 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:32:54.418 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:32:57.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:54 smithi164 conmon[29288]: cluster 2021-02-10T04:32:53.187087+0000 mon.a ( 2021-02-10T04:32:57.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:54 smithi164 conmon[29288]: mon.0) 1120 : cluster [DBG] osdmap e356: 8 total, 8 up, 8 in 2021-02-10T04:32:57.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:54 smithi164 conmon[29288]: audit 2021-02-10T04:32:53.552174+0000 mon.a (mon.0) 1121 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:32:57.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:54 smithi164 conmon[29288]: cephadm 2021-02-10T04:32:53.552909+0000 mgr.y (mgr.14140 2021-02-10T04:32:57.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:54 smithi164 conmon[29288]: ) 498 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:32:57.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:54 smithi164 conmon[29288]: cluster 2021-02-10T04:32:54.063556 2021-02-10T04:32:57.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:54 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 499 : cluster [DBG] pgmap v684: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim, 163 active+clean; 230 MiB data, 754 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 2.5 MiB/s wr, 16 op/s 2021-02-10T04:32:57.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:54 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:32:57.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:54 smithi164 conmon[29288]: 04:32:54.124457+0000 mon.a (mon.0 2021-02-10T04:32:57.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:54 smithi164 conmon[29288]: ) 1122 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:32:57.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:55 smithi164 conmon[29288]: cluster 2021-02-10T04:32:57.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:55 smithi164 conmon[29288]: 2021-02-10T04:32:57.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:55 smithi164 conmon[29288]: 2021-02-10 2021-02-10T04:32:57.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:55 smithi164 conmon[29288]: T04:32:54.195391+0000 mon.a (mon.0) 1123 : 2021-02-10T04:32:57.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:55 smithi164 conmon[29288]: cluster [DBG] osdmap e357: 8 total, 8 up, 8 in 2021-02-10T04:32:57.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:56 smithi164 conmon[29288]: cluster 2021-02-10T04:32:57.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:56 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:32:57.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:56 smithi164 conmon[29288]: 04:32:55.555785+0000 mon.a (mon.0) 1124 : cluster [DBG] osdmap e358: 8 total, 8 up, 8 in 2021-02-10T04:32:57.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:56 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:32:57.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:56 smithi164 conmon[29288]: 02-10T04:32:56.064058 2021-02-10T04:32:57.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:56 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 2021-02-10T04:32:57.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:56 smithi164 conmon[29288]: 500 : cluster [DBG] pgmap v687: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 254 MiB data, 799 MiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 4.7 MiB/s wr, 27 op/s 2021-02-10T04:32:57.865 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:54 smithi085 conmon[29319]: cluster 2021-02-10T04:32:57.865 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:54 smithi085 conmon[29319]: 2021-02 2021-02-10T04:32:57.865 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:54 smithi085 conmon[29319]: -10T04:32:53.187087+0000 mon.a (mon.0) 1120 : cluster [DBG] osdmap e356: 8 total, 8 up, 8 in 2021-02-10T04:32:57.865 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:54 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:32:57.865 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:54 smithi085 conmon[29319]: 02-10T04:32:53.552174+0000 mon.a (mon.0) 1121 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:32:57.865 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:54 smithi085 conmon[29319]: cephadm 2021-02 2021-02-10T04:32:57.865 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:54 smithi085 conmon[29319]: -10T04:32:53.552909+0000 mgr.y (mgr. 2021-02-10T04:32:57.865 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:54 smithi085 conmon[29319]: 14140) 498 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:32:57.865 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:54 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:32:57.865 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:54 smithi085 conmon[29319]: 10T04:32:54.063556+0000 mgr.y (mgr.14140 2021-02-10T04:32:57.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:54 smithi085 conmon[29319]: ) 499 : cluster [DBG] pgmap v684: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim, 163 active+clean; 230 MiB data, 754 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 2.5 MiB/s wr, 16 op/s 2021-02-10T04:32:57.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:54 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:32:57.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:54 smithi085 conmon[29319]: 10T04:32:54.124457+0000 mon.a (mon.0 2021-02-10T04:32:57.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:54 smithi085 conmon[29319]: ) 1122 : audit [INF] 2021-02-10T04:32:57.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:54 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:32:57.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:55 smithi085 conmon[29319]: cluster 2021-02-10T04:32:54.195391+0000 mon.a (mon 2021-02-10T04:32:57.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:55 smithi085 conmon[29319]: .0) 1123 : cluster [DBG] osdmap e357: 8 total, 8 up, 8 in 2021-02-10T04:32:57.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: cluster 2021-02-10T04:32:57.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: 2021-02-10T04:32:57.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: 2021 2021-02-10T04:32:57.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: - 2021-02-10T04:32:57.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: 02 2021-02-10T04:32:57.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: - 2021-02-10T04:32:57.867 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: 10 2021-02-10T04:32:57.867 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: T 2021-02-10T04:32:57.867 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: 04 2021-02-10T04:32:57.867 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: : 2021-02-10T04:32:57.867 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: 32 2021-02-10T04:32:57.867 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: : 2021-02-10T04:32:57.867 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: 55 2021-02-10T04:32:57.867 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: . 2021-02-10T04:32:57.867 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: 555785 2021-02-10T04:32:57.867 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: +0000 2021-02-10T04:32:57.867 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: 2021-02-10T04:32:57.867 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: mon.a 2021-02-10T04:32:57.867 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: ( 2021-02-10T04:32:57.867 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: mon 2021-02-10T04:32:57.868 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: . 2021-02-10T04:32:57.868 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: 0 2021-02-10T04:32:57.868 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: ) 2021-02-10T04:32:57.868 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: 1124 2021-02-10T04:32:57.868 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: : 2021-02-10T04:32:57.868 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: cluster 2021-02-10T04:32:57.868 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: 2021-02-10T04:32:57.868 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: [DBG] 2021-02-10T04:32:57.868 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: 2021-02-10T04:32:57.868 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: osdmap e358: 8 total, 8 up, 8 in 2021-02-10T04:32:57.868 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: 2021-02-10T04:32:57.868 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:32:57.868 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[29319]: 32:56.064058+0000 mgr.y (mgr.14140) 500 : cluster [DBG] pgmap v687: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 254 MiB data, 799 MiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 4.7 MiB/s wr, 27 op/s 2021-02-10T04:32:57.869 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:54 smithi085 conmon[35961]: cluster 2021-02-10T04:32:53.187087+0000 2021-02-10T04:32:57.869 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:54 smithi085 conmon[35961]: mon.a (mon.0) 1120 : cluster [DBG] osdmap e356: 8 total, 8 up, 8 in 2021-02-10T04:32:57.869 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:54 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:32:57.869 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:54 smithi085 conmon[35961]: :32:53.552174+0000 mon.a (mon.0) 1121 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:32:57.869 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:54 smithi085 conmon[35961]: cephadm 2021- 2021-02-10T04:32:57.869 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:54 smithi085 conmon[35961]: 02-10T04:32:53.552909+0000 mgr.y 2021-02-10T04:32:57.869 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:54 smithi085 conmon[35961]: (mgr.14140) 498 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:32:57.869 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:54 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:57.869 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:54 smithi085 conmon[35961]: 02-10T04:32:54.063556+0000 mgr.y 2021-02-10T04:32:57.869 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:54 smithi085 conmon[35961]: (mgr.14140) 499 : cluster [DBG] pgmap v684: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim, 163 active+clean; 230 MiB data, 754 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 2.5 MiB/s wr, 16 op/s 2021-02-10T04:32:57.869 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:54 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:32:57.869 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:54 smithi085 conmon[35961]: 02-10T04:32:54.124457+0000 mon.a 2021-02-10T04:32:57.870 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:54 smithi085 conmon[35961]: (mon.0) 1122 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:32:57.870 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:54 smithi085 conmon[35961]: 2021-02-10T04:32:57.870 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:55 smithi085 conmon[35961]: cluster 2021-02-10T04:32:54.195391+0000 mon.a 2021-02-10T04:32:57.870 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:55 smithi085 conmon[35961]: (mon.0) 1123 : cluster [DBG] osdmap e357: 8 total, 8 up, 8 in 2021-02-10T04:32:57.870 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[35961]: cluster 2021-02-10T04:32:55.555785 2021-02-10T04:32:57.870 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1124 : cluster [DBG] osdmap e358: 8 total, 8 up, 8 in 2021-02-10T04:32:57.870 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:32:57.870 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[35961]: 10T04:32:56.064058+0000 mgr.y (mgr.14140) 500 : cluster [DBG] pgmap v687: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 254 MiB data, 799 MiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 4.7 MiB/s wr, 27 op/s 2021-02-10T04:32:57.870 INFO:tasks.rados.rados.0.smithi085.stdout:1307: finishing copy_from to smithi08575397-5 2021-02-10T04:32:57.870 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 385 (ObjNum 424 snap 109 seq_num 424) dirty exists 2021-02-10T04:32:57.870 INFO:tasks.rados.rados.0.smithi085.stdout:1307: finishing copy_from racing read to smithi08575397-5 2021-02-10T04:32:57.870 INFO:tasks.rados.rados.0.smithi085.stdout:1309: finishing copy_from to smithi08575397-31 2021-02-10T04:32:57.871 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 621 (ObjNum 464 snap 130 seq_num 464) dirty exists 2021-02-10T04:32:57.871 INFO:tasks.rados.rados.0.smithi085.stdout:1309: finishing copy_from racing read to smithi08575397-31 2021-02-10T04:32:57.871 INFO:tasks.rados.rados.0.smithi085.stdout:1307: done (15 left) 2021-02-10T04:32:57.871 INFO:tasks.rados.rados.0.smithi085.stdout:1308: done (14 left) 2021-02-10T04:32:57.871 INFO:tasks.rados.rados.0.smithi085.stdout:1309: done (13 left) 2021-02-10T04:32:57.871 INFO:tasks.rados.rados.0.smithi085.stdout:1310: snap_create 2021-02-10T04:32:57.871 INFO:tasks.rados.rados.0.smithi085.stdout:1310: done (13 left) 2021-02-10T04:32:57.871 INFO:tasks.rados.rados.0.smithi085.stdout:1311: snap_remove snap 132 2021-02-10T04:32:57.871 INFO:tasks.rados.rados.0.smithi085.stdout:1311: done (13 left) 2021-02-10T04:32:57.871 INFO:tasks.rados.rados.0.smithi085.stdout:1312: snap_remove snap 130 2021-02-10T04:32:57.871 INFO:tasks.rados.rados.0.smithi085.stdout:1312: done (13 left) 2021-02-10T04:32:57.871 INFO:tasks.rados.rados.0.smithi085.stdout:1313: write oid 31 current snap is 135 2021-02-10T04:32:57.871 INFO:tasks.rados.rados.0.smithi085.stdout:1313: seq_num 476 ranges {400132=632410,1438017=546162} 2021-02-10T04:32:57.872 INFO:tasks.rados.rados.0.smithi085.stdout:1313: writing smithi08575397-31 from 400132 to 1032542 tid 1 2021-02-10T04:32:57.872 INFO:tasks.rados.rados.0.smithi085.stdout:1313: writing smithi08575397-31 from 1438017 to 1984179 tid 2 2021-02-10T04:32:57.872 INFO:tasks.rados.rados.0.smithi085.stdout:1314: snap_create 2021-02-10T04:32:57.872 INFO:tasks.rados.rados.0.smithi085.stdout:1313: finishing write tid 1 to smithi08575397-31 2021-02-10T04:32:57.872 INFO:tasks.rados.rados.0.smithi085.stdout:1313: finishing write tid 2 to smithi08575397-31 2021-02-10T04:32:57.872 INFO:tasks.rados.rados.0.smithi085.stdout:1313: finishing write tid 3 to smithi08575397-31 2021-02-10T04:32:57.872 INFO:tasks.rados.rados.0.smithi085.stdout:1313: finishing write tid 4 to smithi08575397-31 2021-02-10T04:32:57.872 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 624 (ObjNum 476 snap 135 seq_num 476) dirty exists 2021-02-10T04:32:57.872 INFO:tasks.rados.rados.0.smithi085.stdout:1313: left oid 31 (ObjNum 476 snap 135 seq_num 476) 2021-02-10T04:32:57.872 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:32:57.872 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:32:57.872 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[45443]: debug 2021-02-10T04:32:56.269+0000 7f7121cf0700 -1 osd.1 pg_epoch: 358 pg[4.4( v 63'31 (0'0,63'31] local-lis/les=55/56 n=14 ec=55/55 lis/c=55/55 les/c/f=56/56/0 sis=55) [1,2,3] r=0 lpr=55 crt=63'31 lcod 63'30 mlcod 63'30 active+clean] scrubber pg(4.4) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 3 (0x55dbcd5fab00) 2021-02-10T04:32:57.873 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:32:56 smithi085 conmon[52836]: debug 2021-02-10T04:32:56.767+0000 7fba6ffae700 -1 osd.3 pg_epoch: 359 pg[4.14( v 63'26 (0'0,63'26] local-lis/les=71/72 n=11 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,2,7] r=0 lpr=71 crt=63'26 lcod 63'25 mlcod 0'0 active+clean] scrubber pg(4.14) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55fec06fe6e0) 2021-02-10T04:32:57.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:57 smithi164 conmon[29288]: cluster 2021-02-10T04:32:56.566388+0000 mon.a (mon. 2021-02-10T04:32:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:57 smithi164 conmon[29288]: 0) 1125 : cluster [DBG] osdmap e359: 8 total, 8 up, 8 in 2021-02-10T04:32:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:57 smithi164 conmon[29288]: audit 2021-02-10T04:32:56.897946+0000 mon.a (mon.0) 1126 : audit [INF] from='client.? 172.21.15.85:0/3722864708' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:32:57.904 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:57.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:57 smithi085 conmon[29319]: cluster 2021-02-10T04:32:56.566388+0000 mon.a ( 2021-02-10T04:32:57.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:57 smithi085 conmon[29319]: mon.0) 1125 : cluster [DBG] osdmap e359: 8 total, 8 up, 8 in 2021-02-10T04:32:57.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:57 smithi085 conmon[29319]: audit 2021-02-10T04:32:56.897946+0000 mon.a (mon.0) 1126 : audit [INF] from='client.? 172.21.15.85:0/3722864708' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:32:57.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:57 smithi085 conmon[35961]: cluster 2021-02-10T04:32:56.566388+0000 mon.a (mon.0) 1125 2021-02-10T04:32:57.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:57 smithi085 conmon[35961]: : cluster [DBG] osdmap e359: 8 total, 8 up, 8 in 2021-02-10T04:32:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:57 smithi085 conmon[35961]: audit 2021-02-10T04:32:56.897946+0000 mon.a (mon.0) 1126 : audit [INF] from='client.? 172.21.15.85:0/3722864708' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:32:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:57 smithi085 conmon[35961]: 2021-02-10T04:32:58.220 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is set 2021-02-10T04:32:58.376 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:32:58.380 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:32:57 smithi164 conmon[37263]: debug 2021-02-10T04:32:57.995+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:32:58.380 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:32:57 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:32:58.482 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:32:58.524 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:32:58.578 INFO:tasks.rados.rados.0.smithi085.stdout:1313: done (14 left) 2021-02-10T04:32:58.579 INFO:tasks.rados.rados.0.smithi085.stdout:1314: done (13 left) 2021-02-10T04:32:58.579 INFO:tasks.rados.rados.0.smithi085.stdout:1315: read oid 38 snap -1 2021-02-10T04:32:58.579 INFO:tasks.rados.rados.0.smithi085.stdout:1315: expect (ObjNum 287 snap 71 seq_num 287) 2021-02-10T04:32:58.579 INFO:tasks.rados.rados.0.smithi085.stdout:1316: read oid 22 snap -1 2021-02-10T04:32:58.579 INFO:tasks.rados.rados.0.smithi085.stdout:1316: expect (ObjNum 464 snap 130 seq_num 464) 2021-02-10T04:32:58.579 INFO:tasks.rados.rados.0.smithi085.stdout:1317: read oid 8 snap -1 2021-02-10T04:32:58.579 INFO:tasks.rados.rados.0.smithi085.stdout:1317: expect deleted 2021-02-10T04:32:58.579 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:58.586 INFO:tasks.rados.rados.0.smithi085.stdout:1315: expect (ObjNum 287 snap 71 seq_num 287) 2021-02-10T04:32:58.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:58 smithi164 conmon[29288]: cluster 2021-02-10T04:32:58.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:58 smithi164 conmon[29288]: 2021-02-10T04:32:58.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:58 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:32:58.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:58 smithi164 conmon[29288]: :32:57.558635+0000 mon.a (mon.0) 1127 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:32:58.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:58 smithi164 conmon[29288]: audit 2021 2021-02-10T04:32:58.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:58 smithi164 conmon[29288]: -02-10T04:32:57.566573+0000 mon.a (mon 2021-02-10T04:32:58.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:58 smithi164 conmon[29288]: .0) 1128 : audit [INF] from='client.? 172.21.15.85:0/3722864708' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:32:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:58 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:32:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:58 smithi164 conmon[29288]: 10T04:32:57.566634+0000 mon.a (mon. 2021-02-10T04:32:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:58 smithi164 conmon[29288]: 0) 1129 : cluster [DBG] osdmap e360: 8 total, 8 up, 8 in 2021-02-10T04:32:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:58 smithi164 conmon[29288]: audit 2021-02-10T04:32:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:58 smithi164 conmon[29288]: 2021-02-10T04:32 2021-02-10T04:32:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:58 smithi164 conmon[29288]: :57.935069+0000 mgr.y (mgr.14140) 501 2021-02-10T04:32:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:58 smithi164 conmon[29288]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:32:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:58 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:32:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:58 smithi164 conmon[29288]: -02-10T04:32:58.064528+0000 2021-02-10T04:32:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:58 smithi164 conmon[29288]: mgr.y (mgr.14140) 502 : cluster [DBG] pgmap v690: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 254 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 4.7 MiB/s wr, 23 op/s 2021-02-10T04:32:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:58 smithi164 conmon[29288]: 2021-02-10T04:32:58.784 INFO:tasks.rados.rados.0.smithi085.stdout:1315: done (15 left) 2021-02-10T04:32:58.784 INFO:tasks.rados.rados.0.smithi085.stdout:1318: rollback oid 36 current snap is 136 2021-02-10T04:32:58.784 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 36 to 122 2021-02-10T04:32:58.784 INFO:tasks.rados.rados.0.smithi085.stdout:1316: expect (ObjNum 464 snap 130 seq_num 464) 2021-02-10T04:32:58.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:58 smithi085 conmon[29319]: cluster 2021-02-10T04:32:57.558635+0000 mon.a (mon.0) 1127 : 2021-02-10T04:32:58.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:58 smithi085 conmon[29319]: cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:32:58.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:58 smithi085 conmon[29319]: audit 2021-02-10T04:32:57.566573+0000 mon.a (mon.0) 1128 : audit [INF] from='client.? 172.21.15.85:0/3722864708' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:32:58.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:58 smithi085 conmon[29319]: cluster 2021-02-10T04:32:57.566634+0000 mon.a (mon.0) 1129 : cluster [DBG] osdmap e360: 8 total, 8 up, 8 in 2021-02-10T04:32:58.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:58 smithi085 conmon[29319]: audit 2021-02-10T04:32:57.935069+0000 mgr.y (mgr.14140) 501 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:32:58.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:58 smithi085 conmon[29319]: cluster 2021-02-10T04:32:58.064528+0000 mgr.y (mgr.14140) 502 : cluster [DBG] pgmap v690: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 254 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 4.7 MiB/s wr, 23 op/s 2021-02-10T04:32:58.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:58 smithi085 conmon[35961]: cluster 2021-02-10T04:32: 2021-02-10T04:32:58.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:58 smithi085 conmon[35961]: 57.558635+0000 mon.a (mon.0) 1127 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:32:58.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:58 smithi085 conmon[35961]: audit 2021-02-10T04:32:57 2021-02-10T04:32:58.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:58 smithi085 conmon[35961]: .566573+0000 mon.a (mon.0) 1128 : audit 2021-02-10T04:32:58.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:58 smithi085 conmon[35961]: [INF] from='client.? 172.21.15.85:0/3722864708' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:32:58.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:58 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:32:58.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:58 smithi085 conmon[35961]: 32:57.566634+0000 mon.a (mon.0) 1129 2021-02-10T04:32:58.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:58 smithi085 conmon[35961]: : cluster [DBG] osdmap e360: 8 total, 8 up, 8 in 2021-02-10T04:32:58.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:58 smithi085 conmon[35961]: audit 2021-02-10T04:32:58.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:58 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:32:58.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:58 smithi085 conmon[35961]: 32:57.935069+0000 mgr.y 2021-02-10T04:32:58.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:58 smithi085 conmon[35961]: (mgr.14140) 501 : audit 2021-02-10T04:32:58.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:58 smithi085 conmon[35961]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:32:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:58 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:32:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:58 smithi085 conmon[35961]: 02-10T04:32:58.064528+0000 2021-02-10T04:32:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:58 smithi085 conmon[35961]: mgr.y (mgr.14140) 502 : cluster [DBG] 2021-02-10T04:32:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:58 smithi085 conmon[35961]: pgmap v690: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 157 active+clean; 254 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 4.7 MiB/s wr, 23 op/s 2021-02-10T04:32:59.169 INFO:tasks.rados.rados.0.smithi085.stdout:1316: done (15 left) 2021-02-10T04:32:59.169 INFO:tasks.rados.rados.0.smithi085.stdout:1317: done (14 left) 2021-02-10T04:32:59.170 INFO:tasks.rados.rados.0.smithi085.stdout:1319: snap_create 2021-02-10T04:32:59.170 INFO:tasks.rados.rados.0.smithi085.stdout:1318: finishing rollback tid 0 to smithi08575397-36 2021-02-10T04:32:59.170 INFO:tasks.rados.rados.0.smithi085.stdout:1318: finishing rollback tid 1 to smithi08575397-36 2021-02-10T04:32:59.170 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 618 (ObjNum 0 snap 0 seq_num 82403760) dirty dne 2021-02-10T04:32:59.596 INFO:tasks.rados.rados.0.smithi085.stdout:1318: done (14 left) 2021-02-10T04:32:59.596 INFO:tasks.rados.rados.0.smithi085.stdout:1319: done (13 left) 2021-02-10T04:32:59.596 INFO:tasks.rados.rados.0.smithi085.stdout:1320: write (excl) oid 11 current snap is 137 2021-02-10T04:32:59.596 INFO:tasks.rados.rados.0.smithi085.stdout:1320: seq_num 477 ranges {416939=707659,1896499=120440} 2021-02-10T04:32:59.597 INFO:tasks.rados.rados.0.smithi085.stdout:1320: writing smithi08575397-11 from 416939 to 1124598 tid 1 2021-02-10T04:32:59.602 INFO:tasks.rados.rados.0.smithi085.stdout:1320: writing smithi08575397-11 from 1896499 to 2016939 tid 2 2021-02-10T04:32:59.602 INFO:tasks.rados.rados.0.smithi085.stdout:1321: delete oid 50 current snap is 137 2021-02-10T04:32:59.603 INFO:tasks.rados.rados.0.smithi085.stdout:1320: finishing write tid 1 to smithi08575397-11 2021-02-10T04:32:59.605 INFO:tasks.rados.rados.0.smithi085.stdout:1321: done (14 left) 2021-02-10T04:32:59.605 INFO:tasks.rados.rados.0.smithi085.stdout:1322: rollback oid 21 current snap is 137 2021-02-10T04:32:59.605 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 21 to 120 2021-02-10T04:32:59.605 INFO:tasks.rados.rados.0.smithi085.stdout:1323: write (excl) oid 5 current snap is 137 2021-02-10T04:32:59.605 INFO:tasks.rados.rados.0.smithi085.stdout:1323: seq_num 479 ranges {450553=574510,1613463=437090} 2021-02-10T04:32:59.611 INFO:tasks.rados.rados.0.smithi085.stdout:1323: writing smithi08575397-5 from 450553 to 1025063 tid 1 2021-02-10T04:32:59.618 INFO:tasks.rados.rados.0.smithi085.stdout:1323: writing smithi08575397-5 from 1613463 to 2050553 tid 2 2021-02-10T04:32:59.618 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:59.618 INFO:tasks.rados.rados.0.smithi085.stdout:1320: finishing write tid 2 to smithi08575397-11 2021-02-10T04:32:59.619 INFO:tasks.rados.rados.0.smithi085.stdout:1320: finishing write tid 3 to smithi08575397-11 2021-02-10T04:32:59.619 INFO:tasks.rados.rados.0.smithi085.stdout:1320: finishing write tid 4 to smithi08575397-11 2021-02-10T04:32:59.619 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 737 (ObjNum 477 snap 137 seq_num 477) dirty exists 2021-02-10T04:32:59.619 INFO:tasks.rados.rados.0.smithi085.stdout:1320: left oid 11 (ObjNum 477 snap 137 seq_num 477) 2021-02-10T04:32:59.619 INFO:tasks.rados.rados.0.smithi085.stdout:1322: finishing rollback tid 1 to smithi08575397-21 2021-02-10T04:32:59.619 INFO:tasks.rados.rados.0.smithi085.stdout:1322: finishing rollback tid 2 to smithi08575397-21 2021-02-10T04:32:59.619 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 739 (ObjNum 431 snap 114 seq_num 431) dirty exists 2021-02-10T04:32:59.619 INFO:tasks.rados.rados.0.smithi085.stdout:1323: finishing write tid 1 to smithi08575397-5 2021-02-10T04:32:59.619 INFO:tasks.rados.rados.0.smithi085.stdout:1320: done (15 left) 2021-02-10T04:32:59.619 INFO:tasks.rados.rados.0.smithi085.stdout:1322: done (14 left) 2021-02-10T04:32:59.619 INFO:tasks.rados.rados.0.smithi085.stdout:1324: read oid 1 snap 120 2021-02-10T04:32:59.619 INFO:tasks.rados.rados.0.smithi085.stdout:1324: expect (ObjNum 424 snap 109 seq_num 424) 2021-02-10T04:32:59.620 INFO:tasks.rados.rados.0.smithi085.stdout:1325: write (excl) oid 3 current snap is 137 2021-02-10T04:32:59.620 INFO:tasks.rados.rados.0.smithi085.stdout:1325: seq_num 480 ranges {467360=649759,1671945=395415} 2021-02-10T04:32:59.625 INFO:tasks.rados.rados.0.smithi085.stdout:1325: writing smithi08575397-3 from 467360 to 1117119 tid 1 2021-02-10T04:32:59.632 INFO:tasks.rados.rados.0.smithi085.stdout:1325: writing smithi08575397-3 from 1671945 to 2067360 tid 2 2021-02-10T04:32:59.632 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:32:59.632 INFO:tasks.rados.rados.0.smithi085.stdout:1323: finishing write tid 2 to smithi08575397-5 2021-02-10T04:32:59.632 INFO:tasks.rados.rados.0.smithi085.stdout:1323: finishing write tid 3 to smithi08575397-5 2021-02-10T04:32:59.632 INFO:tasks.rados.rados.0.smithi085.stdout:1323: finishing write tid 4 to smithi08575397-5 2021-02-10T04:32:59.632 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 589 (ObjNum 479 snap 137 seq_num 479) dirty exists 2021-02-10T04:32:59.632 INFO:tasks.rados.rados.0.smithi085.stdout:1323: left oid 5 (ObjNum 479 snap 137 seq_num 479) 2021-02-10T04:32:59.632 INFO:tasks.rados.rados.0.smithi085.stdout:1324: expect (ObjNum 424 snap 109 seq_num 424) 2021-02-10T04:32:59.875 INFO:tasks.rados.rados.0.smithi085.stdout:1325: finishing write tid 1 to smithi08575397-3 2021-02-10T04:32:59.875 INFO:tasks.rados.rados.0.smithi085.stdout:1325: finishing write tid 2 to smithi08575397-3 2021-02-10T04:32:59.875 INFO:tasks.rados.rados.0.smithi085.stdout:1325: finishing write tid 3 to smithi08575397-3 2021-02-10T04:32:59.875 INFO:tasks.rados.rados.0.smithi085.stdout:1325: finishing write tid 4 to smithi08575397-3 2021-02-10T04:32:59.875 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 619 (ObjNum 480 snap 137 seq_num 480) dirty exists 2021-02-10T04:32:59.876 INFO:tasks.rados.rados.0.smithi085.stdout:1325: left oid 3 (ObjNum 480 snap 137 seq_num 480) 2021-02-10T04:32:59.876 INFO:tasks.rados.rados.0.smithi085.stdout:1323: done (15 left) 2021-02-10T04:32:59.876 INFO:tasks.rados.rados.0.smithi085.stdout:1324: done (14 left) 2021-02-10T04:32:59.876 INFO:tasks.rados.rados.0.smithi085.stdout:1325: done (13 left) 2021-02-10T04:32:59.876 INFO:tasks.rados.rados.0.smithi085.stdout:1326: snap_create 2021-02-10T04:32:59.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:59 smithi164 conmon[29288]: cluster 2021-02-10T04:32:58.576895+0000 mon.a ( 2021-02-10T04:32:59.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:32:59 smithi164 conmon[29288]: mon.0) 1130 : cluster [DBG] osdmap e361: 8 total, 8 up, 8 in 2021-02-10T04:32:59.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:59 smithi085 conmon[29319]: cluster 2021-02-10T04:32:59.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:59 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:32:59.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:32:59 smithi085 conmon[29319]: 10T04:32:58.576895+0000 mon.a (mon.0) 1130 : cluster [DBG] osdmap e361: 8 total, 8 up, 8 in 2021-02-10T04:32:59.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:59 smithi085 conmon[35961]: cluster 2021-02-10T04:32:58. 2021-02-10T04:32:59.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:32:59 smithi085 conmon[35961]: 576895+0000 mon.a (mon.0) 1130 : cluster [DBG] osdmap e361: 8 total, 8 up, 8 in 2021-02-10T04:33:00.593 INFO:tasks.rados.rados.0.smithi085.stdout:1326: done (13 left) 2021-02-10T04:33:00.593 INFO:tasks.rados.rados.0.smithi085.stdout:1327: rollback oid 16 current snap is 138 2021-02-10T04:33:00.594 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 16 to 135 2021-02-10T04:33:00.594 INFO:tasks.rados.rados.0.smithi085.stdout:1328: snap_create 2021-02-10T04:33:00.598 INFO:tasks.rados.rados.0.smithi085.stdout:1327: finishing rollback tid 0 to smithi08575397-16 2021-02-10T04:33:00.600 INFO:tasks.rados.rados.0.smithi085.stdout:1327: finishing rollback tid 1 to smithi08575397-16 2021-02-10T04:33:00.600 INFO:tasks.rados.rados.0.smithi085.stdout:1327: finishing rollback tid 2 to smithi08575397-16 2021-02-10T04:33:00.600 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 745 (ObjNum 465 snap 130 seq_num 465) dirty exists 2021-02-10T04:33:00.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:00 smithi164 conmon[29288]: cluster 2021-02-10T04:32:59.588234+0000 mon.a (mon.0) 1131 : cluster [DBG] osdmap e362: 8 total, 8 up, 8 in 2021-02-10T04:33:00.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:00 smithi164 conmon[29288]: cluster 2021-02-10T04:33:00.064940+0000 mgr.y (mgr.14140) 503 : cluster [DBG] pgmap v693: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 262 MiB data, 818 MiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 1.9 MiB/s wr, 5 op/s 2021-02-10T04:33:00.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: cluster 2021-02-10T04:33:00.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: 2021-02-10T04:33:00.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: 2021 2021-02-10T04:33:00.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: - 2021-02-10T04:33:00.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: 02 2021-02-10T04:33:00.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: - 2021-02-10T04:33:00.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: 10 2021-02-10T04:33:00.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: T 2021-02-10T04:33:00.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: 04 2021-02-10T04:33:00.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: : 2021-02-10T04:33:00.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: 32 2021-02-10T04:33:00.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: : 2021-02-10T04:33:00.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: 59 2021-02-10T04:33:00.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: . 2021-02-10T04:33:00.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: 588234 2021-02-10T04:33:00.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: +0000 2021-02-10T04:33:00.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: 2021-02-10T04:33:00.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: mon.a 2021-02-10T04:33:00.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: ( 2021-02-10T04:33:00.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: mon 2021-02-10T04:33:00.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: . 2021-02-10T04:33:00.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: 0 2021-02-10T04:33:00.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: ) 2021-02-10T04:33:00.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: 1131 2021-02-10T04:33:00.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: : 2021-02-10T04:33:00.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: cluster 2021-02-10T04:33:00.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: 2021-02-10T04:33:00.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: [DBG] 2021-02-10T04:33:00.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: 2021-02-10T04:33:00.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: osdmap e362: 8 total, 8 up, 8 in 2021-02-10T04:33:00.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: 2021-02-10T04:33:00.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: cluster 2021-02-10T04:33:00.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: 2021-02-10T04:33:00.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: 2021 2021-02-10T04:33:00.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: - 2021-02-10T04:33:00.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: 02 2021-02-10T04:33:00.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: - 2021-02-10T04:33:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: 10 2021-02-10T04:33:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: T 2021-02-10T04:33:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: 04 2021-02-10T04:33:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: : 2021-02-10T04:33:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: 33 2021-02-10T04:33:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: : 2021-02-10T04:33:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: 00 2021-02-10T04:33:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: . 2021-02-10T04:33:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: 064940 2021-02-10T04:33:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: +0000 2021-02-10T04:33:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: 2021-02-10T04:33:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: mgr.y 2021-02-10T04:33:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: (mgr.14140) 503 : cluster [DBG] pgmap v693: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 262 MiB data, 818 MiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 1.9 MiB/s wr, 5 op/s 2021-02-10T04:33:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[35961]: 2021-02-10T04:33:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[29319]: cluster 2021-02-10T04:32:59.588234+0000 mon.a (mon. 2021-02-10T04:33:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[29319]: 0) 1131 : cluster [DBG] osdmap e362: 8 total, 8 up, 8 in 2021-02-10T04:33:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:00 smithi085 conmon[29319]: cluster 2021-02-10T04:33:00.064940+0000 mgr.y (mgr.14140) 503 : cluster [DBG] pgmap v693: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 262 MiB data, 818 MiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 1.9 MiB/s wr, 5 op/s 2021-02-10T04:33:01.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:33:01 smithi085 conmon[52836]: debug 2021-02-10T04:33:01.027+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:01.302 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:33:01.604 INFO:tasks.rados.rados.0.smithi085.stdout:1327: done (14 left) 2021-02-10T04:33:01.604 INFO:tasks.rados.rados.0.smithi085.stdout:1328: done (13 left) 2021-02-10T04:33:01.604 INFO:tasks.rados.rados.0.smithi085.stdout:1329: rollback oid 46 current snap is 139 2021-02-10T04:33:01.604 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 46 to 109 2021-02-10T04:33:01.604 INFO:tasks.rados.rados.0.smithi085.stdout:1330: rollback oid 7 current snap is 139 2021-02-10T04:33:01.604 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 7 to 122 2021-02-10T04:33:01.604 INFO:tasks.rados.rados.0.smithi085.stdout:1331: write (excl) oid 26 current snap is 139 2021-02-10T04:33:01.604 INFO:tasks.rados.rados.0.smithi085.stdout:1331: seq_num 481 ranges {484167=725008,1730427=353740} 2021-02-10T04:33:01.605 INFO:tasks.rados.rados.0.smithi085.stdout:1331: writing smithi08575397-26 from 484167 to 1209175 tid 1 2021-02-10T04:33:01.609 INFO:tasks.rados.rados.0.smithi085.stdout:1331: writing smithi08575397-26 from 1730427 to 2084167 tid 2 2021-02-10T04:33:01.609 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:33:01.609 INFO:tasks.rados.rados.0.smithi085.stdout:1331: finishing write tid 1 to smithi08575397-26 2021-02-10T04:33:01.612 INFO:tasks.rados.rados.0.smithi085.stdout:1331: finishing write tid 2 to smithi08575397-26 2021-02-10T04:33:01.613 INFO:tasks.rados.rados.0.smithi085.stdout:1331: finishing write tid 3 to smithi08575397-26 2021-02-10T04:33:01.613 INFO:tasks.rados.rados.0.smithi085.stdout:1331: finishing write tid 4 to smithi08575397-26 2021-02-10T04:33:01.613 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 119 (ObjNum 481 snap 139 seq_num 481) dirty exists 2021-02-10T04:33:01.614 INFO:tasks.rados.rados.0.smithi085.stdout:1331: left oid 26 (ObjNum 481 snap 139 seq_num 481) 2021-02-10T04:33:01.614 INFO:tasks.rados.rados.0.smithi085.stdout:1331: done (15 left) 2021-02-10T04:33:01.614 INFO:tasks.rados.rados.0.smithi085.stdout:1332: write oid 38 current snap is 139 2021-02-10T04:33:01.614 INFO:tasks.rados.rados.0.smithi085.stdout:1332: seq_num 482 ranges {500974=400257,1388909=557169} 2021-02-10T04:33:01.619 INFO:tasks.rados.rados.0.smithi085.stdout:1332: writing smithi08575397-38 from 500974 to 901231 tid 1 2021-02-10T04:33:01.626 INFO:tasks.rados.rados.0.smithi085.stdout:1332: writing smithi08575397-38 from 1388909 to 1946078 tid 2 2021-02-10T04:33:01.626 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:33:01.626 INFO:tasks.rados.rados.0.smithi085.stdout:1329: finishing rollback tid 0 to smithi08575397-46 2021-02-10T04:33:01.627 INFO:tasks.rados.rados.0.smithi085.stdout:1330: finishing rollback tid 0 to smithi08575397-7 2021-02-10T04:33:01.627 INFO:tasks.rados.rados.0.smithi085.stdout:1329: finishing rollback tid 1 to smithi08575397-46 2021-02-10T04:33:01.627 INFO:tasks.rados.rados.0.smithi085.stdout:1330: finishing rollback tid 1 to smithi08575397-7 2021-02-10T04:33:01.627 INFO:tasks.rados.rados.0.smithi085.stdout:1329: finishing rollback tid 2 to smithi08575397-46 2021-02-10T04:33:01.627 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 646 (ObjNum 382 snap 100 seq_num 382) dirty exists 2021-02-10T04:33:01.627 INFO:tasks.rados.rados.0.smithi085.stdout:1330: finishing rollback tid 2 to smithi08575397-7 2021-02-10T04:33:01.627 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 622 (ObjNum 441 snap 119 seq_num 441) dirty exists 2021-02-10T04:33:01.627 INFO:tasks.rados.rados.0.smithi085.stdout:1332: finishing write tid 1 to smithi08575397-38 2021-02-10T04:33:01.627 INFO:tasks.rados.rados.0.smithi085.stdout:1329: done (15 left) 2021-02-10T04:33:01.627 INFO:tasks.rados.rados.0.smithi085.stdout:1330: done (14 left) 2021-02-10T04:33:01.627 INFO:tasks.rados.rados.0.smithi085.stdout:1333: read oid 50 snap -1 2021-02-10T04:33:01.627 INFO:tasks.rados.rados.0.smithi085.stdout:1333: expect deleted 2021-02-10T04:33:01.627 INFO:tasks.rados.rados.0.smithi085.stdout:1334: read oid 31 snap 122 2021-02-10T04:33:01.627 INFO:tasks.rados.rados.0.smithi085.stdout:1334: expect (ObjNum 432 snap 114 seq_num 432) 2021-02-10T04:33:01.628 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:33:01.628 INFO:tasks.rados.rados.0.smithi085.stdout:1333: done (15 left) 2021-02-10T04:33:01.628 INFO:tasks.rados.rados.0.smithi085.stdout:1335: delete oid 13 current snap is 139 2021-02-10T04:33:01.630 INFO:tasks.rados.rados.0.smithi085.stdout:1332: finishing write tid 2 to smithi08575397-38 2021-02-10T04:33:01.630 INFO:tasks.rados.rados.0.smithi085.stdout:1332: finishing write tid 3 to smithi08575397-38 2021-02-10T04:33:01.630 INFO:tasks.rados.rados.0.smithi085.stdout:1332: finishing write tid 4 to smithi08575397-38 2021-02-10T04:33:01.630 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 649 (ObjNum 482 snap 139 seq_num 482) dirty exists 2021-02-10T04:33:01.630 INFO:tasks.rados.rados.0.smithi085.stdout:1332: left oid 38 (ObjNum 482 snap 139 seq_num 482) 2021-02-10T04:33:01.630 INFO:tasks.rados.rados.0.smithi085.stdout:1332: done (15 left) 2021-02-10T04:33:01.631 INFO:tasks.rados.rados.0.smithi085.stdout:1335: done (14 left) 2021-02-10T04:33:01.631 INFO:tasks.rados.rados.0.smithi085.stdout:1336: read oid 30 snap -1 2021-02-10T04:33:01.631 INFO:tasks.rados.rados.0.smithi085.stdout:1336: expect deleted 2021-02-10T04:33:01.631 INFO:tasks.rados.rados.0.smithi085.stdout:1337: delete oid 38 current snap is 139 2021-02-10T04:33:01.635 INFO:tasks.rados.rados.0.smithi085.stdout:1336: done (15 left) 2021-02-10T04:33:01.635 INFO:tasks.rados.rados.0.smithi085.stdout:1337: done (14 left) 2021-02-10T04:33:01.635 INFO:tasks.rados.rados.0.smithi085.stdout:1338: delete oid 40 current snap is 139 2021-02-10T04:33:01.648 INFO:tasks.rados.rados.0.smithi085.stdout:1334: expect (ObjNum 432 snap 114 seq_num 432) 2021-02-10T04:33:01.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:01 smithi164 conmon[29288]: cluster 2021-02-10T04:33:00.589954+0000 mon.a (mon.0 2021-02-10T04:33:01.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:01 smithi164 conmon[29288]: ) 1132 : cluster [DBG] osdmap e363: 8 total, 8 up, 8 in 2021-02-10T04:33:01.925 INFO:tasks.rados.rados.0.smithi085.stdout:1334: done (14 left) 2021-02-10T04:33:01.925 INFO:tasks.rados.rados.0.smithi085.stdout:1338: done (13 left) 2021-02-10T04:33:01.925 INFO:tasks.rados.rados.0.smithi085.stdout:1339: write oid 6 current snap is 139 2021-02-10T04:33:01.925 INFO:tasks.rados.rados.0.smithi085.stdout:1339: seq_num 486 ranges {568202=701253,1906484=261718} 2021-02-10T04:33:01.932 INFO:tasks.rados.rados.0.smithi085.stdout:1339: writing smithi08575397-6 from 568202 to 1269455 tid 1 2021-02-10T04:33:01.937 INFO:tasks.rados.rados.0.smithi085.stdout:1339: writing smithi08575397-6 from 1906484 to 2168202 tid 2 2021-02-10T04:33:01.937 INFO:tasks.rados.rados.0.smithi085.stdout:1340: snap_remove snap 133 2021-02-10T04:33:01.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:01 smithi085 conmon[29319]: cluster 2021-02-10T04:33:00.589954 2021-02-10T04:33:01.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:01 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1132 : cluster [DBG] osdmap e363: 8 total, 8 up, 8 in 2021-02-10T04:33:01.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:01 smithi085 conmon[35961]: cluster 2021-02-10T04:33:01.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:01 smithi085 conmon[35961]: 2021-02-10T04:33:01.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:01 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:33:01.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:01 smithi085 conmon[35961]: T04:33:00.589954+0000 mon.a (mon.0) 1132 : cluster [DBG] osdmap e363: 8 total, 8 up, 8 in 2021-02-10T04:33:02.608 INFO:tasks.rados.rados.0.smithi085.stdout:1340: done (14 left) 2021-02-10T04:33:02.608 INFO:tasks.rados.rados.0.smithi085.stdout:1341: rollback oid 46 current snap is 139 2021-02-10T04:33:02.608 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 46 to 125 2021-02-10T04:33:02.608 INFO:tasks.rados.rados.0.smithi085.stdout:1339: finishing write tid 1 to smithi08575397-6 2021-02-10T04:33:02.608 INFO:tasks.rados.rados.0.smithi085.stdout:1339: finishing write tid 2 to smithi08575397-6 2021-02-10T04:33:02.608 INFO:tasks.rados.rados.0.smithi085.stdout:1339: finishing write tid 3 to smithi08575397-6 2021-02-10T04:33:02.608 INFO:tasks.rados.rados.0.smithi085.stdout:1339: finishing write tid 4 to smithi08575397-6 2021-02-10T04:33:02.608 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 653 (ObjNum 486 snap 139 seq_num 486) dirty exists 2021-02-10T04:33:02.608 INFO:tasks.rados.rados.0.smithi085.stdout:1339: left oid 6 (ObjNum 486 snap 139 seq_num 486) 2021-02-10T04:33:02.608 INFO:tasks.rados.rados.0.smithi085.stdout:1339: done (14 left) 2021-02-10T04:33:02.609 INFO:tasks.rados.rados.0.smithi085.stdout:1342: read oid 40 snap -1 2021-02-10T04:33:02.609 INFO:tasks.rados.rados.0.smithi085.stdout:1342: expect deleted 2021-02-10T04:33:02.609 INFO:tasks.rados.rados.0.smithi085.stdout:1343: read oid 30 snap -1 2021-02-10T04:33:02.609 INFO:tasks.rados.rados.0.smithi085.stdout:1343: expect deleted 2021-02-10T04:33:02.609 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:33:02.613 INFO:tasks.rados.rados.0.smithi085.stdout:1342: done (15 left) 2021-02-10T04:33:02.613 INFO:tasks.rados.rados.0.smithi085.stdout:1344: copy_from oid 28 from oid 37 current snap is 139 2021-02-10T04:33:02.613 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:33:02.726 INFO:tasks.rados.rados.0.smithi085.stdout:1341: finishing rollback tid 0 to smithi08575397-46 2021-02-10T04:33:02.726 INFO:tasks.rados.rados.0.smithi085.stdout:1343: done (15 left) 2021-02-10T04:33:02.726 INFO:tasks.rados.rados.0.smithi085.stdout:1345: copy_from oid 23 from oid 2 current snap is 139 2021-02-10T04:33:02.726 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:33:02.726 INFO:tasks.rados.rados.0.smithi085.stdout:1341: finishing rollback tid 1 to smithi08575397-46 2021-02-10T04:33:02.727 INFO:tasks.rados.rados.0.smithi085.stdout:1341: finishing rollback tid 2 to smithi08575397-46 2021-02-10T04:33:02.727 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 656 (ObjNum 454 snap 125 seq_num 454) dirty exists 2021-02-10T04:33:02.727 INFO:tasks.rados.rados.0.smithi085.stdout:1341: done (15 left) 2021-02-10T04:33:02.727 INFO:tasks.rados.rados.0.smithi085.stdout:1346: write (excl) oid 46 current snap is 139 2021-02-10T04:33:02.727 INFO:tasks.rados.rados.0.smithi085.stdout:1346: seq_num 487 ranges {585009=492855,1564966=620043} 2021-02-10T04:33:02.732 INFO:tasks.rados.rados.0.smithi085.stdout:1346: writing smithi08575397-46 from 585009 to 1077864 tid 1 2021-02-10T04:33:02.739 INFO:tasks.rados.rados.0.smithi085.stdout:1346: writing smithi08575397-46 from 1564966 to 2185009 tid 2 2021-02-10T04:33:02.739 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:33:02.750 INFO:tasks.rados.rados.0.smithi085.stdout:1344: finishing copy_from to smithi08575397-28 2021-02-10T04:33:02.750 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 715 (ObjNum 466 snap 131 seq_num 466) dirty exists 2021-02-10T04:33:02.751 INFO:tasks.rados.rados.0.smithi085.stdout:1344: finishing copy_from racing read to smithi08575397-28 2021-02-10T04:33:02.751 INFO:tasks.rados.rados.0.smithi085.stdout:1344: done (15 left) 2021-02-10T04:33:02.751 INFO:tasks.rados.rados.0.smithi085.stdout:1347: read oid 22 snap -1 2021-02-10T04:33:02.751 INFO:tasks.rados.rados.0.smithi085.stdout:1347: expect (ObjNum 464 snap 130 seq_num 464) 2021-02-10T04:33:02.751 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:33:02.752 INFO:tasks.rados.rados.0.smithi085.stdout:1345: finishing copy_from to smithi08575397-23 2021-02-10T04:33:02.752 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 610 (ObjNum 471 snap 133 seq_num 471) dirty exists 2021-02-10T04:33:02.752 INFO:tasks.rados.rados.0.smithi085.stdout:1346: finishing write tid 1 to smithi08575397-46 2021-02-10T04:33:02.752 INFO:tasks.rados.rados.0.smithi085.stdout:1345: finishing copy_from racing read to smithi08575397-23 2021-02-10T04:33:02.752 INFO:tasks.rados.rados.0.smithi085.stdout:1345: done (15 left) 2021-02-10T04:33:02.753 INFO:tasks.rados.rados.0.smithi085.stdout:1348: snap_remove snap 129 2021-02-10T04:33:02.876 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:33:02.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:02 smithi164 conmon[29288]: cluster 2021-02-10T04:33:01.106553+0000 osd.5 (osd 2021-02-10T04:33:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:02 smithi164 conmon[29288]: .5) 57 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:33:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:02 smithi164 conmon[29288]: cluster 2021-02-10T04:33:01.106689+0000 osd.5 (osd.5) 58 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:33:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:02 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:33:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:02 smithi164 conmon[29288]: 02-10T04:33:01.591199+0000 mon.a (mon.0) 1133 : cluster [DBG] osdmap e364: 8 total, 8 up, 8 in 2021-02-10T04:33:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:02 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:33:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:02 smithi164 conmon[29288]: 10T04:33:02.065397+0000 mgr.y (mgr 2021-02-10T04:33:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:02 smithi164 conmon[29288]: .14140) 504 : cluster [DBG] pgmap v696: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 158 active+clean; 261 MiB data, 825 MiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 2.2 MiB/s wr, 8 op/s 2021-02-10T04:33:02.904 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:02 smithi085 conmon[29319]: cluster 2021-02-10T04:33:01.106553+0000 osd.5 ( 2021-02-10T04:33:02.905 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:02 smithi085 conmon[29319]: osd.5) 57 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:33:02.905 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:02 smithi085 conmon[29319]: cluster 2021-02-10T04:33:01.106689+0000 osd.5 (osd.5) 58 : 2021-02-10T04:33:02.905 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:02 smithi085 conmon[29319]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:33:02.905 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:02 smithi085 conmon[29319]: cluster 2021-02-10T04:33 2021-02-10T04:33:02.905 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:02 smithi085 conmon[29319]: :01.591199+0000 mon.a (mon.0) 1133 : cluster [DBG] osdmap e364: 8 total, 8 up, 8 in 2021-02-10T04:33:02.905 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:02 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:02.905 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:02 smithi085 conmon[29319]: -02-10T04:33:02.065397+0000 2021-02-10T04:33:02.905 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:02 smithi085 conmon[29319]: mgr.y (mgr.14140) 504 : cluster 2021-02-10T04:33:02.905 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:02 smithi085 conmon[29319]: [DBG] pgmap v696: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 158 active+clean; 261 MiB data, 825 MiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 2.2 MiB/s wr, 8 op/s 2021-02-10T04:33:02.905 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:02 smithi085 conmon[35961]: cluster 2021-02-10T04:33:01.106553+0000 2021-02-10T04:33:02.906 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:02 smithi085 conmon[35961]: osd.5 (osd.5) 57 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:33:02.906 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:02 smithi085 conmon[35961]: cluster 2021-02-10T04:33:01. 2021-02-10T04:33:02.906 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:02 smithi085 conmon[35961]: 106689+0000 osd.5 (osd.5) 58 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:33:02.906 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:02 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:02.906 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:02 smithi085 conmon[35961]: T04:33:01.591199+0000 mon.a (mon. 2021-02-10T04:33:02.906 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:02 smithi085 conmon[35961]: 0) 1133 : cluster [DBG] osdmap e364: 8 total, 8 up, 8 in 2021-02-10T04:33:02.906 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:02 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:02.906 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:02 smithi085 conmon[35961]: -02-10T04:33:02.065397+0000 mgr.y 2021-02-10T04:33:02.906 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:02 smithi085 conmon[35961]: (mgr.14140) 504 : cluster [DBG] 2021-02-10T04:33:02.906 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:02 smithi085 conmon[35961]: pgmap v696: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 158 active+clean; 261 MiB data, 825 MiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 2.2 MiB/s wr, 8 op/s 2021-02-10T04:33:02.907 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:03.116 INFO:tasks.rados.rados.0.smithi085.stdout:1348: done (15 left) 2021-02-10T04:33:03.116 INFO:tasks.rados.rados.0.smithi085.stdout:1349: copy_from oid 7 from oid 3 current snap is 139 2021-02-10T04:33:03.116 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:33:03.116 INFO:tasks.rados.rados.0.smithi085.stdout:1346: finishing write tid 2 to smithi08575397-46 2021-02-10T04:33:03.117 INFO:tasks.rados.rados.0.smithi085.stdout:1346: finishing write tid 3 to smithi08575397-46 2021-02-10T04:33:03.117 INFO:tasks.rados.rados.0.smithi085.stdout:1346: finishing write tid 4 to smithi08575397-46 2021-02-10T04:33:03.117 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 659 (ObjNum 487 snap 139 seq_num 487) dirty exists 2021-02-10T04:33:03.117 INFO:tasks.rados.rados.0.smithi085.stdout:1346: left oid 46 (ObjNum 487 snap 139 seq_num 487) 2021-02-10T04:33:03.117 INFO:tasks.rados.rados.0.smithi085.stdout:1347: expect (ObjNum 464 snap 130 seq_num 464) 2021-02-10T04:33:03.507 INFO:tasks.rados.rados.0.smithi085.stdout:1346: done (15 left) 2021-02-10T04:33:03.507 INFO:tasks.rados.rados.0.smithi085.stdout:1347: done (14 left) 2021-02-10T04:33:03.507 INFO:tasks.rados.rados.0.smithi085.stdout:1350: read oid 26 snap -1 2021-02-10T04:33:03.508 INFO:tasks.rados.rados.0.smithi085.stdout:1350: expect (ObjNum 481 snap 139 seq_num 481) 2021-02-10T04:33:03.508 INFO:tasks.rados.rados.0.smithi085.stdout:1351: copy_from oid 5 from oid 30 current snap is 139 2021-02-10T04:33:03.508 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:33:03.508 INFO:tasks.rados.rados.0.smithi085.stdout:1349: finishing copy_from to smithi08575397-7 2021-02-10T04:33:03.508 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 619 (ObjNum 480 snap 137 seq_num 480) dirty exists 2021-02-10T04:33:03.508 INFO:tasks.rados.rados.0.smithi085.stdout:1349: finishing copy_from racing read to smithi08575397-7 2021-02-10T04:33:03.508 INFO:tasks.rados.rados.0.smithi085.stdout:1349: done (15 left) 2021-02-10T04:33:03.508 INFO:tasks.rados.rados.0.smithi085.stdout:1352: snap_create 2021-02-10T04:33:03.510 INFO:tasks.rados.rados.0.smithi085.stdout:1350: expect (ObjNum 481 snap 139 seq_num 481) 2021-02-10T04:33:03.687 INFO:tasks.rados.rados.0.smithi085.stdout:1351: finishing copy_from to smithi08575397-5 2021-02-10T04:33:03.687 INFO:tasks.rados.rados.0.smithi085.stdout:1351: got expected ENOENT (src dne) 2021-02-10T04:33:03.687 INFO:tasks.rados.rados.0.smithi085.stdout:1351: finishing copy_from racing read to smithi08575397-5 2021-02-10T04:33:03.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:03 smithi164 conmon[29288]: cluster 2021-02-10T04:33:02.607827+0000 mon.a ( 2021-02-10T04:33:03.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:03 smithi164 conmon[29288]: mon.0) 1134 : cluster [DBG] osdmap e365: 8 total, 8 up, 8 in 2021-02-10T04:33:03.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:03 smithi164 conmon[29288]: cluster 2021-02-10T04:33:03.114132+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e366: 8 total, 8 up, 8 in 2021-02-10T04:33:03.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:03 smithi164 conmon[29288]: audit 2021-02-10T04:33:03.572121+0000 mon.b (mon.2 2021-02-10T04:33:03.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:03 smithi164 conmon[29288]: ) 33 : audit [INF] from='client.? 172.21.15.85:0/400821729' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.0}]: dispatch 2021-02-10T04:33:03.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:03 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:33:03.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:03 smithi164 conmon[29288]: 02-10T04:33:03.572634+0000 mon.a ( 2021-02-10T04:33:03.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:03 smithi164 conmon[29288]: mon.0) 1136 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.0}]: dispatch 2021-02-10T04:33:03.941 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:33:03 smithi085 conmon[52836]: debug 2021-02-10T04:33:03.637+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:03.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:03 smithi085 conmon[29319]: cluster 2021-02-10T04:33:02.607827+0000 2021-02-10T04:33:03.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:03 smithi085 conmon[29319]: mon.a (mon.0) 1134 : cluster [DBG] osdmap e365: 8 total, 8 up, 8 in 2021-02-10T04:33:03.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:03 smithi085 conmon[29319]: cluster 2021-02-10T04:33:03.114132+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e366: 8 total, 8 up, 8 in 2021-02-10T04:33:03.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:03 smithi085 conmon[29319]: audit 2021-02-10T04:33:03.572121+0000 mon.b (mon.2) 33 : audit [INF] from='client.? 172.21.15.85:0/400821729' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.0}]: dispatch 2021-02-10T04:33:03.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:03 smithi085 conmon[29319]: audit 2021-02-10T04:33:03.572634+0000 mon.a (mon.0) 1136 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.0}]: dispatch 2021-02-10T04:33:03.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:03 smithi085 conmon[35961]: cluster 2021-02-10T04:33:02.607827+0000 2021-02-10T04:33:03.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:03 smithi085 conmon[35961]: mon.a (mon.0) 1134 : cluster [DBG] osdmap e365: 8 total, 8 up, 8 in 2021-02-10T04:33:03.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:03 smithi085 conmon[35961]: cluster 2021-02-10T04:33:03 2021-02-10T04:33:03.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:03 smithi085 conmon[35961]: .114132+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e366: 8 total, 8 up, 8 in 2021-02-10T04:33:03.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:03 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:33:03.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:03 smithi085 conmon[35961]: :33:03.572121+0000 mon.b (mon.2) 2021-02-10T04:33:03.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:03 smithi085 conmon[35961]: 33 : audit [INF] from='client.? 172.21.15.85:0/400821729' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.0}]: dispatch 2021-02-10T04:33:03.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:03 smithi085 conmon[35961]: audit 2021-02-10T04:33 2021-02-10T04:33:03.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:03 smithi085 conmon[35961]: :03.572634+0000 mon.a (mon.0) 1136 2021-02-10T04:33:03.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:03 smithi085 conmon[35961]: : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.0}]: dispatch 2021-02-10T04:33:04.125 INFO:tasks.rados.rados.0.smithi085.stdout:1350: done (15 left) 2021-02-10T04:33:04.126 INFO:tasks.rados.rados.0.smithi085.stdout:1351: done (14 left) 2021-02-10T04:33:04.126 INFO:tasks.rados.rados.0.smithi085.stdout:1352: done (13 left) 2021-02-10T04:33:04.126 INFO:tasks.rados.rados.0.smithi085.stdout:1353: delete oid 13 current snap is 140 2021-02-10T04:33:04.140 INFO:teuthology.orchestra.run.smithi085.stderr:set osd.1 primary-affinity to 0 (802) 2021-02-10T04:33:04.147 INFO:tasks.rados.rados.0.smithi085.stdout:1353: done (13 left) 2021-02-10T04:33:04.147 INFO:tasks.rados.rados.0.smithi085.stdout:1354: snap_remove snap 134 2021-02-10T04:33:04.219 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:33:04.525 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:33:04.535 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.2 dump_blocked_ops 2021-02-10T04:33:04.550 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:04.809 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:33:04.813 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:04 smithi164 conmon[33677]: debug 2021-02-10T04:33:04.615+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:04.843 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:33:04.914 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:33:04.944 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:05.068 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:33:05.073 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:33:05 smithi164 conmon[40892]: debug 2021-02-10T04:33:05.026+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:05.131 INFO:tasks.rados.rados.0.smithi085.stdout:1354: done (13 left) 2021-02-10T04:33:05.131 INFO:tasks.rados.rados.0.smithi085.stdout:1355: delete oid 6 current snap is 140 2021-02-10T04:33:05.155 INFO:tasks.rados.rados.0.smithi085.stdout:1355: done (13 left) 2021-02-10T04:33:05.155 INFO:tasks.rados.rados.0.smithi085.stdout:1356: read oid 2 snap -1 2021-02-10T04:33:05.155 INFO:tasks.rados.rados.0.smithi085.stdout:1356: expect (ObjNum 471 snap 133 seq_num 471) 2021-02-10T04:33:05.155 INFO:tasks.rados.rados.0.smithi085.stdout:1357: snap_remove snap 120 2021-02-10T04:33:05.174 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:33:05.213 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:05.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:05 smithi164 conmon[29288]: cluster 2021-02-10T04:33:04.065895+0000 mgr.y 2021-02-10T04:33:05.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:05 smithi164 conmon[29288]: (mgr.14140) 505 : cluster [DBG] pgmap v699: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 158 active+clean; 262 MiB data, 829 MiB used, 714 GiB / 715 GiB avail; 1.4 MiB/s rd, 503 KiB/s wr, 6 op/s 2021-02-10T04:33:05.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:05 smithi164 conmon[29288]: audit 2021-02-10T04:33:04.125353+0000 mon.a (mon.0) 1137 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.0}]': finished 2021-02-10T04:33:05.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:05 smithi164 conmon[29288]: cluster 2021-02-10T04:33:04.125451+0000 mon.a (mon.0) 1138 : cluster [DBG] osdmap e367: 8 total, 8 up, 8 in 2021-02-10T04:33:05.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:05 smithi164 conmon[29288]: 2021-02-10T04:33:05.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:05 smithi085 conmon[29319]: cluster 2021-02-10T04:33:04.065895+0000 mgr.y 2021-02-10T04:33:05.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:05 smithi085 conmon[29319]: (mgr.14140) 505 : cluster [DBG] pgmap v699: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 158 active+clean; 262 MiB data, 829 MiB used, 714 GiB / 715 GiB avail; 1.4 MiB/s rd, 503 KiB/s wr, 6 op/s 2021-02-10T04:33:05.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:05 smithi085 conmon[29319]: audit 2021-02-10T04:33:04.125353+0000 mon.a (mon.0) 1137 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.0}]': finished 2021-02-10T04:33:05.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:05 smithi085 conmon[29319]: cluster 2021-02-10T04:33:04.125451+0000 mon.a (mon.0) 1138 : cluster [DBG] osdmap e367: 8 total, 8 up, 8 in 2021-02-10T04:33:05.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:05 smithi085 conmon[35961]: cluster 2021-02-10T04:33:04. 2021-02-10T04:33:05.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:05 smithi085 conmon[35961]: 065895+0000 mgr.y (mgr.14140) 505 : cluster [DBG] pgmap v699: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 158 active+clean; 262 MiB data, 829 MiB used, 714 GiB / 715 GiB avail; 1.4 MiB/s rd, 503 KiB/s wr, 6 op/s 2021-02-10T04:33:05.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:05 smithi085 conmon[35961]: audit 2021-02-10T04:33:04.125353 2021-02-10T04:33:05.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:05 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1137 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.0}]': finished 2021-02-10T04:33:05.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:05 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:33:05.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:05 smithi085 conmon[35961]: :33:04.125451+0000 mon.a (mon.0) 1138 : cluster [DBG] osdmap e367: 8 total, 8 up, 8 in 2021-02-10T04:33:05.464 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:33:05.464 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:33:05.465 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:33:05.465 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:33:05.473 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:33:05.502 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:33:05.534 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:33:05.564 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:33:05.570 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:33:05.570 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:33:05.570 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:33:05.570 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:33:05.570 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:33:05.570 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:33:05.570 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:33:05.570 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:33:05.847 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:33:05.859 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:05 smithi085 conmon[49106]: debug 2021-02-10T04:33:05.859 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:05 smithi085 conmon[49106]: 2021-02-10T04:33:05.444+0000 7f77a3e94700 -1 osd.2 368 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:33:05.859 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:05 smithi085 conmon[49106]: debug 2021-02-10T04:33:05.496+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:05.893 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:05 smithi164 conmon[33677]: debug 2021-02-10T04:33:05.502+0000 7fbec19a1700 -1 osd.4 368 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:33:06.145 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[29319]: cluster 2021-02-10T04:33:06.145 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[29319]: 2021-02-10T04:33:06.145 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[29319]: 2021-02-10T04:33:05 2021-02-10T04:33:06.145 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[29319]: .129192+0000 mon.a (mon.0) 1139 : cluster [DBG] osdmap e368: 8 total, 8 up, 8 in 2021-02-10T04:33:06.145 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[29319]: audit 2021-02-10T04:33:05.321069+0000 mon.a 2021-02-10T04:33:06.146 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[29319]: (mon.0) 1140 : audit [DBG] from='client.? 172.21.15.85:0/2624737400' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:33:06.146 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[29319]: cluster 2021-02-10T04:33:06.146 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[29319]: 2021-02-10T04:33:05.504323 2021-02-10T04:33:06.146 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 383 : cluster [WRN] 2021-02-10T04:33:06.146 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:06.146 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:33:06.146 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[29319]: 33:05.504332+0000 osd.4 (osd.4 2021-02-10T04:33:06.146 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[29319]: ) 384 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:06.146 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:33:06.146 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[29319]: 04:33:05.504339+0000 osd.4 (osd 2021-02-10T04:33:06.146 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[29319]: .4) 385 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:06.146 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:33:06.147 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[29319]: 10T04:33:05.504344+0000 osd.4 ( 2021-02-10T04:33:06.147 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[29319]: osd.4) 386 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:06.149 INFO:tasks.rados.rados.0.smithi085.stdout:1357: done (14 left) 2021-02-10T04:33:06.150 INFO:tasks.rados.rados.0.smithi085.stdout:1358: read oid 43 snap -1 2021-02-10T04:33:06.150 INFO:tasks.rados.rados.0.smithi085.stdout:1358: expect (ObjNum 474 snap 134 seq_num 474) 2021-02-10T04:33:06.150 INFO:tasks.rados.rados.0.smithi085.stdout:1359: write (excl) oid 30 current snap is 140 2021-02-10T04:33:06.150 INFO:tasks.rados.rados.0.smithi085.stdout:1359: seq_num 490 ranges {635430=718602,2140412=95018} 2021-02-10T04:33:06.155 INFO:tasks.rados.rados.0.smithi085.stdout:1359: writing smithi08575397-30 from 635430 to 1354032 tid 1 2021-02-10T04:33:06.160 INFO:tasks.rados.rados.0.smithi085.stdout:1359: writing smithi08575397-30 from 2140412 to 2235430 tid 2 2021-02-10T04:33:06.160 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:33:06.160 INFO:tasks.rados.rados.0.smithi085.stdout:1356: expect (ObjNum 471 snap 133 seq_num 471) 2021-02-10T04:33:06.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:06 smithi164 conmon[29288]: cluster 2021-02-10T04:33:05.129192+0000 mon.a ( 2021-02-10T04:33:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:06 smithi164 conmon[29288]: mon.0) 1139 : cluster [DBG] osdmap e368: 8 total, 8 up, 8 in 2021-02-10T04:33:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:06 smithi164 conmon[29288]: audit 2021-02-10T04:33:05.321069+0000 mon.a (mon.0) 1140 : audit [DBG] from='client.? 172.21.15.85:0/2624737400' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:33:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:06 smithi164 conmon[29288]: cluster 2021-02-10T04:33:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:06 smithi164 conmon[29288]: 2021-02-10T04:33:05.504323+0000 osd.4 (osd.4) 383 : cluster [WRN] 2021-02-10T04:33:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:06 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:06 smithi164 conmon[29288]: cluster 2021-02-10T04:33:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:06 smithi164 conmon[29288]: 2021-02-10T04:33:05.504332+0000 osd.4 (osd.4) 384 2021-02-10T04:33:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:06 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:06 smithi164 conmon[29288]: cluster 2021-02-10T04:33:05 2021-02-10T04:33:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:06 smithi164 conmon[29288]: .504339+0000 osd.4 (osd.4) 385 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:06 smithi164 conmon[29288]: 2021-02-10T04:33:06.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:06 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:33:06.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:06 smithi164 conmon[29288]: 02-10T04:33:05.504344+0000 osd.4 (osd.4) 386 : 2021-02-10T04:33:06.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:06 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:06.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[35961]: cluster 2021-02-10T04:33:06.437 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[35961]: 2021-02-10T04:33:06.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[35961]: 2021 2021-02-10T04:33:06.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[35961]: - 2021-02-10T04:33:06.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[35961]: 02 2021-02-10T04:33:06.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[35961]: -10T04:33:05.129192+0000 2021-02-10T04:33:06.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[35961]: mon.a (mon.0) 1139 : cluster [DBG] osdmap e368: 8 total, 8 up, 8 in 2021-02-10T04:33:06.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:33:06.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[35961]: :33:05.321069+0000 mon.a (mon.0 2021-02-10T04:33:06.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[35961]: ) 1140 : audit [DBG] from='client.? 172.21.15.85:0/2624737400' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:33:06.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:33:06.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[35961]: 10T04:33:05.504323+0000 osd.4 ( 2021-02-10T04:33:06.438 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[35961]: osd.4) 383 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:06.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[35961]: cluster 2021-02-10T04:33:06.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[35961]: 2021-02-10T04:33:05.504332+0000 2021-02-10T04:33:06.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[35961]: osd.4 (osd.4) 384 : cluster [WRN] 2021-02-10T04:33:06.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:06.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[35961]: cluster 2021-02-10T04:33:06.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[35961]: 2021-02-10T04:33:05.504339+0000 osd.4 (osd. 2021-02-10T04:33:06.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[35961]: 4) 385 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:06.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[35961]: cluster 2021-02-10T04:33:05.504344+0000 osd.4 (osd.4 2021-02-10T04:33:06.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[35961]: ) 386 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:06.488 INFO:tasks.rados.rados.0.smithi085.stdout:1356: done (15 left) 2021-02-10T04:33:06.488 INFO:tasks.rados.rados.0.smithi085.stdout:1360: snap_create 2021-02-10T04:33:06.488 INFO:tasks.rados.rados.0.smithi085.stdout:665: finishing copy_from to smithi08575397-9 2021-02-10T04:33:06.488 INFO:tasks.rados.rados.0.smithi085.stdout:665: got expected ENOENT (src dne) 2021-02-10T04:33:06.489 INFO:tasks.rados.rados.0.smithi085.stdout:665: finishing copy_from racing read to smithi08575397-9 2021-02-10T04:33:06.489 INFO:tasks.rados.rados.0.smithi085.stdout:221: finishing copy_from to smithi08575397-12 2021-02-10T04:33:06.489 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 29 (ObjNum 85 snap 11 seq_num 85) dirty exists 2021-02-10T04:33:06.489 INFO:tasks.rados.rados.0.smithi085.stdout:221: finishing copy_from racing read to smithi08575397-12 2021-02-10T04:33:06.489 INFO:tasks.rados.rados.0.smithi085.stdout:250: finishing copy_from to smithi08575397-48 2021-02-10T04:33:06.489 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 39 (ObjNum 105 snap 22 seq_num 105) dirty exists 2021-02-10T04:33:06.489 INFO:tasks.rados.rados.0.smithi085.stdout:250: finishing copy_from racing read to smithi08575397-48 2021-02-10T04:33:06.489 INFO:tasks.rados.rados.0.smithi085.stdout:624: finishing copy_from to smithi08575397-14 2021-02-10T04:33:06.489 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 217 (ObjNum 194 snap 43 seq_num 194) dirty exists 2021-02-10T04:33:06.489 INFO:tasks.rados.rados.0.smithi085.stdout:624: finishing copy_from racing read to smithi08575397-14 2021-02-10T04:33:06.489 INFO:tasks.rados.rados.0.smithi085.stdout:538: finishing copy_from to smithi08575397-42 2021-02-10T04:33:06.489 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 97 (ObjNum 204 snap 44 seq_num 204) dirty exists 2021-02-10T04:33:06.490 INFO:tasks.rados.rados.0.smithi085.stdout:538: finishing copy_from racing read to smithi08575397-42 2021-02-10T04:33:06.490 INFO:tasks.rados.rados.0.smithi085.stdout:1358: expect (ObjNum 474 snap 134 seq_num 474) 2021-02-10T04:33:06.627 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set nodeep-scrub 2021-02-10T04:33:06.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:06 smithi085 conmon[49106]: debug 2021-02-10T04:33:06.435+0000 7f77a3e94700 -1 osd.2 369 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:33:06.792 INFO:tasks.rados.rados.0.smithi085.stdout:1359: finishing write tid 1 to smithi08575397-30 2021-02-10T04:33:06.792 INFO:tasks.rados.rados.0.smithi085.stdout:1359: finishing write tid 2 to smithi08575397-30 2021-02-10T04:33:06.792 INFO:tasks.rados.rados.0.smithi085.stdout:1359: finishing write tid 3 to smithi08575397-30 2021-02-10T04:33:06.792 INFO:tasks.rados.rados.0.smithi085.stdout:1359: finishing write tid 4 to smithi08575397-30 2021-02-10T04:33:06.792 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 663 (ObjNum 490 snap 140 seq_num 490) dirty exists 2021-02-10T04:33:06.792 INFO:tasks.rados.rados.0.smithi085.stdout:1359: left oid 30 (ObjNum 490 snap 140 seq_num 490) 2021-02-10T04:33:06.893 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:06 smithi164 conmon[33677]: debug 2021-02-10T04:33:06.549+0000 7fbec19a1700 -1 osd.4 369 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:33:06.936 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:33:07.155 INFO:tasks.rados.rados.0.smithi085.stdout:221: done (15 left) 2021-02-10T04:33:07.155 INFO:tasks.rados.rados.0.smithi085.stdout:250: done (14 left) 2021-02-10T04:33:07.155 INFO:tasks.rados.rados.0.smithi085.stdout:538: done (13 left) 2021-02-10T04:33:07.156 INFO:tasks.rados.rados.0.smithi085.stdout:624: done (12 left) 2021-02-10T04:33:07.156 INFO:tasks.rados.rados.0.smithi085.stdout:665: done (11 left) 2021-02-10T04:33:07.156 INFO:tasks.rados.rados.0.smithi085.stdout:1358: done (10 left) 2021-02-10T04:33:07.156 INFO:tasks.rados.rados.0.smithi085.stdout:1359: done (9 left) 2021-02-10T04:33:07.156 INFO:tasks.rados.rados.0.smithi085.stdout:1360: done (8 left) 2021-02-10T04:33:07.156 INFO:tasks.rados.rados.0.smithi085.stdout:1361: read oid 23 snap -1 2021-02-10T04:33:07.156 INFO:tasks.rados.rados.0.smithi085.stdout:1361: expect (ObjNum 471 snap 133 seq_num 471) 2021-02-10T04:33:07.156 INFO:tasks.rados.rados.0.smithi085.stdout:1362: write oid 43 current snap is 141 2021-02-10T04:33:07.156 INFO:tasks.rados.rados.0.smithi085.stdout:1362: seq_num 491 ranges {652237=793851,2198894=53343} 2021-02-10T04:33:07.165 INFO:tasks.rados.rados.0.smithi085.stdout:1362: writing smithi08575397-43 from 652237 to 1446088 tid 1 2021-02-10T04:33:07.170 INFO:tasks.rados.rados.0.smithi085.stdout:1362: writing smithi08575397-43 from 2198894 to 2252237 tid 2 2021-02-10T04:33:07.170 INFO:tasks.rados.rados.0.smithi085.stdout:1363: rollback oid 36 current snap is 141 2021-02-10T04:33:07.170 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 36 to 109 2021-02-10T04:33:07.170 INFO:tasks.rados.rados.0.smithi085.stdout:1361: expect (ObjNum 471 snap 133 seq_num 471) 2021-02-10T04:33:07.356 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:33:07.386 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:07.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: cluster 2021-02-10T04:33:05.445648+0000 osd.2 (osd.2 2021-02-10T04:33:07.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: ) 528 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:07.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: cluster 2021-02-10T04:33:05.445658+0000 osd.2 (osd.2) 529 : cluster 2021-02-10T04:33:07.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:07.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: cluster 2021-02-10T04:33:07.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: 2021-02-10T04:33:05.445664+0000 osd.2 (osd.2 2021-02-10T04:33:07.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: ) 530 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:07.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: cluster 2021-02-10T04:33:05.445669+0000 osd.2 ( 2021-02-10T04:33:07.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: osd.2) 531 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:07.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: cluster 2021-02-10T04:33:05.445675+0000 2021-02-10T04:33:07.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: osd.2 (osd.2) 532 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:07.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: cluster 2021-02-10T04:33:05.445784+0000 2021-02-10T04:33:07.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: osd.2 (osd.2) 533 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:07.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: cluster 2021-02-10T04:33:05.445791+0000 2021-02-10T04:33:07.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: osd.2 (osd.2) 534 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:07.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:33:07.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: 04:33:05.445795+0000 osd.2 (osd.2) 535 2021-02-10T04:33:07.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:07.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:33:07.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: :33:05.708789+0000 osd.2 (osd.2 2021-02-10T04:33:07.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: ) 536 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:33:07.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: cluster 2021-02-10T04:33:07.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: 2021-02-10T04:33:05 2021-02-10T04:33:07.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: .708988+0000 osd.2 (osd.2) 537 : cluster 2021-02-10T04:33:07.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: [DBG] purged_snaps scrub ok 2021-02-10T04:33:07.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:33:07.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: 10T04:33:06.066468+0000 mgr.y (mgr.14140) 506 2021-02-10T04:33:07.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: : cluster [DBG] pgmap v702: 168 pgs: 5 peering, 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 152 active+clean; 279 MiB data, 854 MiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s wr, 9 op/s 2021-02-10T04:33:07.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: cluster 2021-02-10T04:33:07.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: 2021-02-10T04:33:06.134233+0000 mon.a (mon.0) 1141 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2021-02-10T04:33:07.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: cluster 2021-02-10T04:33:06 2021-02-10T04:33:07.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: .134268+0000 mon.a (mon.0) 1142 : cluster [WRN] Health check failed: 8 slow ops, oldest one blocked for 193 sec, osd.2 has slow ops (SLOW_OPS) 2021-02-10T04:33:07.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:33:07.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: T04:33:06.145795+0000 mon.a (mon.0) 1143 : cluster 2021-02-10T04:33:07.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: [DBG] osdmap e369: 8 total, 8 up, 8 in 2021-02-10T04:33:07.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: cluster 2021-02-10T04:33:06 2021-02-10T04:33:07.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: .551176+0000 osd.4 (osd.4) 387 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:07.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:33:07.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: 33:06.551191+0000 osd.4 (osd.4) 388 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:07.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: cluster 2021-02-10T04:33 2021-02-10T04:33:07.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: :06.551197+0000 osd.4 (osd.4) 389 : cluster [WRN] 2021-02-10T04:33:07.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:07.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: cluster 2021-02-10T04:33:07.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: 2021-02-10T04:33:06.551223+0000 osd.4 (osd.4) 390 : 2021-02-10T04:33:07.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:07.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[49106]: debug 2021-02-10T04:33:07.390+0000 7f77a3e94700 -1 osd.2 370 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:33:07.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: cluster 2021-02-10T04:33:05.445648+0000 osd.2 (osd.2) 528 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:07.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: cluster 2021-02-10T04:33:05.445658+0000 osd.2 (osd.2) 529 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:07.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: cluster 2021-02-10T04:33:05.445664+0000 osd.2 (osd.2) 530 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:07.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: cluster 2021-02-10T04:33:05.445669+0000 osd.2 (osd.2) 531 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:07.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: cluster 2021-02-10T04:33:05.445675+0000 osd.2 (osd.2) 532 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:07.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: cluster 2021-02-10T04:33:05.445784+0000 osd.2 (osd.2) 533 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:07.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: cluster 2021-02-10T04:33:05.445791+0000 osd.2 (osd.2) 534 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:07.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: cluster 2021-02-10T04:33:07.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: 2021-02-10T04:33:05.445795+0000 osd.2 (osd.2) 535 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:07.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: cluster 2021-02-10T04:33:05.708789+0000 osd.2 (osd.2) 536 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:33:07.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: cluster 2021-02-10T04:33:05.708988+0000 osd.2 ( 2021-02-10T04:33:07.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: osd.2) 537 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:33:07.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:33:07.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: -10T04:33:06.066468+0000 mgr.y (mgr.14140) 506 : cluster [DBG] 2021-02-10T04:33:07.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: pgmap v702: 168 pgs: 5 peering, 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 152 active+clean; 279 MiB data, 854 MiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s wr, 9 op/s 2021-02-10T04:33:07.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: cluster 2021-02-10T04:33:07.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: 2021-02-10T04:33:06.134233+0000 mon.a (mon. 2021-02-10T04:33:07.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: 0) 1141 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2021-02-10T04:33:07.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: cluster 2021-02-10T04:33:06.134268+0000 2021-02-10T04:33:07.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: mon.a (mon.0) 1142 : cluster [WRN] Health check failed: 8 slow ops, oldest one blocked for 193 sec, osd.2 has slow ops (SLOW_OPS) 2021-02-10T04:33:07.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:33:07.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: T04:33:06.145795+0000 mon.a (mon.0 2021-02-10T04:33:07.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: ) 1143 : cluster [DBG] osdmap e369: 8 total, 8 up, 8 in 2021-02-10T04:33:07.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:33:07.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: 04:33:06.551176+0000 osd.4 (osd.4) 2021-02-10T04:33:07.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: 387 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:07.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: cluster 2021-02-10T04:33:06 2021-02-10T04:33:07.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: .551191+0000 osd.4 (osd.4) 388 : cluster [WRN] 2021-02-10T04:33:07.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:07.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:33:07.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: 10T04:33:06.551197+0000 osd.4 (osd.4) 389 : 2021-02-10T04:33:07.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:07.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: cluster 2021-02-10T04:33 2021-02-10T04:33:07.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: :06.551223+0000 osd.4 (osd.4) 390 : cluster 2021-02-10T04:33:07.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:07.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: cluster 2021-02-10T04:33:05.445648+0000 osd.2 ( 2021-02-10T04:33:07.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: osd.2) 528 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:07.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:33:07.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: 10T04:33:05.445658+0000 2021-02-10T04:33:07.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: osd.2 (osd.2) 529 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:07.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: cluster 2021-02-10T04:33:07.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: 2021-02-10T04:33:05. 2021-02-10T04:33:07.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: 445664+0000 osd.2 (osd.2) 2021-02-10T04:33:07.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: 530 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:07.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:33:07.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: 33:05.445669+0000 osd.2 (osd.2) 531 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:07.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:07.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: 02-10T04:33:05.445675+0000 osd.2 ( 2021-02-10T04:33:07.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: osd.2) 532 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:07.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:33:07.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: 33:05.445784+0000 osd.2 (osd.2) 533 : 2021-02-10T04:33:07.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:07.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:07.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: T04:33:05.445791+0000 osd.2 (osd. 2021-02-10T04:33:07.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: 2) 534 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:07.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: cluster 2021-02-10T04:33:07.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:33:07.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: :33:05.445795+0000 osd.2 (osd.2) 535 2021-02-10T04:33:07.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:07.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:07.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: 02-10T04:33:05.708789+0000 osd.2 2021-02-10T04:33:07.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: (osd.2) 536 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:33:07.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: cluster 2021-02-10T04:33:07.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: 2021-02-10T04:33 2021-02-10T04:33:07.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: :05.708988+0000 osd.2 (osd 2021-02-10T04:33:07.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: .2) 537 : cluster [DBG] 2021-02-10T04:33:07.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:33:07.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: cluster 2021-02-10T04:33:07.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:33:07.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: 10T04:33:06.066468+0000 2021-02-10T04:33:07.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: mgr.y (mgr.14140) 506 : cluster [DBG] pgmap v702: 168 pgs: 5 peering, 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 152 active+clean; 279 MiB data, 854 MiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s wr, 9 op/s 2021-02-10T04:33:07.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:07.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: T04:33:06.134233+0000 mon.a (mon.0) 1141 : cluster 2021-02-10T04:33:07.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2021-02-10T04:33:07.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: cluster 2021-02-10T04:33:07.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: 2021-02-10T04:33:06.134268+0000 2021-02-10T04:33:07.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: mon.a (mon.0) 1142 : cluster 2021-02-10T04:33:07.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: [WRN] Health check failed: 8 slow ops, oldest one blocked for 193 sec, osd.2 has slow ops (SLOW_OPS) 2021-02-10T04:33:07.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: cluster 2021-02-10T04:33:07.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: 2021-02-10T04:33:06.145795+0000 mon.a ( 2021-02-10T04:33:07.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: mon.0) 1143 : cluster [DBG] osdmap e369: 8 total, 8 up, 8 in 2021-02-10T04:33:07.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:07.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: 02-10T04:33:06.551176+0000 osd.4 (osd.4 2021-02-10T04:33:07.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: ) 387 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:07.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:33:07.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: 04:33:06.551191+0000 osd.4 (osd.4 2021-02-10T04:33:07.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: ) 388 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:07.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:07.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: -10T04:33:06.551197+0000 osd.4 (osd. 2021-02-10T04:33:07.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: 4) 389 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:07.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: cluster 2021-02-10T04:33 2021-02-10T04:33:07.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:07 smithi085 conmon[35961]: :06.551223+0000 osd.4 (osd.4) 390 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:07.510 INFO:tasks.rados.rados.0.smithi085.stdout:1361: done (10 left) 2021-02-10T04:33:07.510 INFO:tasks.rados.rados.0.smithi085.stdout:1364: write (excl) oid 30 current snap is 141 2021-02-10T04:33:07.510 INFO:tasks.rados.rados.0.smithi085.stdout:1362: finishing write tid 1 to smithi08575397-43 2021-02-10T04:33:07.510 INFO:tasks.rados.rados.0.smithi085.stdout:1363: finishing rollback tid 1 to smithi08575397-36 2021-02-10T04:33:07.510 INFO:tasks.rados.rados.0.smithi085.stdout:1362: finishing write tid 2 to smithi08575397-43 2021-02-10T04:33:07.510 INFO:tasks.rados.rados.0.smithi085.stdout:1363: finishing rollback tid 2 to smithi08575397-36 2021-02-10T04:33:07.510 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 620 (ObjNum 421 snap 109 seq_num 421) dirty exists 2021-02-10T04:33:07.510 INFO:tasks.rados.rados.0.smithi085.stdout:1362: finishing write tid 3 to smithi08575397-43 2021-02-10T04:33:07.511 INFO:tasks.rados.rados.0.smithi085.stdout:1364: seq_num 492 ranges {669044=469100,1741023=459161} 2021-02-10T04:33:07.516 INFO:tasks.rados.rados.0.smithi085.stdout:1364: writing smithi08575397-30 from 669044 to 1138144 tid 1 2021-02-10T04:33:07.523 INFO:tasks.rados.rados.0.smithi085.stdout:1364: writing smithi08575397-30 from 1741023 to 2200184 tid 2 2021-02-10T04:33:07.523 INFO:tasks.rados.rados.0.smithi085.stdout:1363: done (10 left) 2021-02-10T04:33:07.523 INFO:tasks.rados.rados.0.smithi085.stdout:1365: snap_create 2021-02-10T04:33:07.523 INFO:tasks.rados.rados.0.smithi085.stdout:1362: finishing write tid 4 to smithi08575397-43 2021-02-10T04:33:07.524 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 625 (ObjNum 491 snap 141 seq_num 491) dirty exists 2021-02-10T04:33:07.524 INFO:tasks.rados.rados.0.smithi085.stdout:1362: left oid 43 (ObjNum 491 snap 141 seq_num 491) 2021-02-10T04:33:07.524 INFO:tasks.rados.rados.0.smithi085.stdout:1364: finishing write tid 1 to smithi08575397-30 2021-02-10T04:33:07.526 INFO:tasks.rados.rados.0.smithi085.stdout:1364: finishing write tid 2 to smithi08575397-30 2021-02-10T04:33:07.526 INFO:tasks.rados.rados.0.smithi085.stdout:1364: finishing write tid 3 to smithi08575397-30 2021-02-10T04:33:07.527 INFO:tasks.rados.rados.0.smithi085.stdout:1364: finishing write tid 4 to smithi08575397-30 2021-02-10T04:33:07.527 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 666 (ObjNum 492 snap 141 seq_num 492) dirty exists 2021-02-10T04:33:07.527 INFO:tasks.rados.rados.0.smithi085.stdout:1364: left oid 30 (ObjNum 492 snap 141 seq_num 492) 2021-02-10T04:33:07.893 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[33677]: debug 2021-02-10T04:33:07.549+0000 7fbec19a1700 -1 osd.4 370 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:33:08.118 INFO:tasks.rados.rados.0.smithi085.stdout:1362: done (10 left) 2021-02-10T04:33:08.118 INFO:tasks.rados.rados.0.smithi085.stdout:1364: done (9 left) 2021-02-10T04:33:08.118 INFO:tasks.rados.rados.0.smithi085.stdout:1365: done (8 left) 2021-02-10T04:33:08.118 INFO:tasks.rados.rados.0.smithi085.stdout:1366: delete oid 7 current snap is 142 2021-02-10T04:33:08.123 INFO:tasks.rados.rados.0.smithi085.stdout:1366: done (8 left) 2021-02-10T04:33:08.123 INFO:tasks.rados.rados.0.smithi085.stdout:1367: rollback oid 2 current snap is 142 2021-02-10T04:33:08.123 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 2 to 123 2021-02-10T04:33:08.123 INFO:tasks.rados.rados.0.smithi085.stdout:1368: delete oid 6 current snap is 142 2021-02-10T04:33:08.130 INFO:tasks.rados.rados.0.smithi085.stdout:1368: done (9 left) 2021-02-10T04:33:08.130 INFO:tasks.rados.rados.0.smithi085.stdout:1369: write (excl) oid 5 current snap is 142 2021-02-10T04:33:08.130 INFO:tasks.rados.rados.0.smithi085.stdout:1369: seq_num 495 ranges {719465=411200,1632822=686643} 2021-02-10T04:33:08.137 INFO:tasks.rados.rados.0.smithi085.stdout:1369: writing smithi08575397-5 from 719465 to 1130665 tid 1 2021-02-10T04:33:08.144 INFO:tasks.rados.rados.0.smithi085.stdout:1369: writing smithi08575397-5 from 1632822 to 2319465 tid 2 2021-02-10T04:33:08.145 INFO:tasks.rados.rados.0.smithi085.stdout:1370: copy_from oid 7 from oid 31 current snap is 142 2021-02-10T04:33:08.145 INFO:tasks.rados.rados.0.smithi085.stdout:1367: finishing rollback tid 0 to smithi08575397-2 2021-02-10T04:33:08.145 INFO:tasks.rados.rados.0.smithi085.stdout:1367: finishing rollback tid 1 to smithi08575397-2 2021-02-10T04:33:08.145 INFO:tasks.rados.rados.0.smithi085.stdout:1371: delete oid 30 current snap is 142 2021-02-10T04:33:08.145 INFO:tasks.rados.rados.0.smithi085.stdout:1369: finishing write tid 1 to smithi08575397-5 2021-02-10T04:33:08.146 INFO:tasks.rados.rados.0.smithi085.stdout:1367: finishing rollback tid 2 to smithi08575397-2 2021-02-10T04:33:08.146 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 613 (ObjNum 422 snap 109 seq_num 422) dirty exists 2021-02-10T04:33:08.149 INFO:tasks.rados.rados.0.smithi085.stdout:1367: done (11 left) 2021-02-10T04:33:08.149 INFO:tasks.rados.rados.0.smithi085.stdout:1371: done (10 left) 2021-02-10T04:33:08.149 INFO:tasks.rados.rados.0.smithi085.stdout:1372: snap_create 2021-02-10T04:33:08.156 INFO:tasks.rados.rados.0.smithi085.stdout:1369: finishing write tid 2 to smithi08575397-5 2021-02-10T04:33:08.156 INFO:tasks.rados.rados.0.smithi085.stdout:1369: finishing write tid 3 to smithi08575397-5 2021-02-10T04:33:08.157 INFO:tasks.rados.rados.0.smithi085.stdout:1369: finishing write tid 4 to smithi08575397-5 2021-02-10T04:33:08.157 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 593 (ObjNum 495 snap 142 seq_num 495) dirty exists 2021-02-10T04:33:08.157 INFO:tasks.rados.rados.0.smithi085.stdout:1369: left oid 5 (ObjNum 495 snap 142 seq_num 495) 2021-02-10T04:33:08.208 INFO:tasks.rados.rados.0.smithi085.stdout:1370: finishing copy_from to smithi08575397-7 2021-02-10T04:33:08.208 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 624 (ObjNum 476 snap 135 seq_num 476) dirty exists 2021-02-10T04:33:08.208 INFO:tasks.rados.rados.0.smithi085.stdout:1370: finishing copy_from racing read to smithi08575397-7 2021-02-10T04:33:08.215 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:33:07 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:33:08.215 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:33:08.215 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: 04:33:06.437464+0000 osd.2 (osd.2) 538 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:08.215 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:33:08.517 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: 04 2021-02-10T04:33:08.517 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: : 2021-02-10T04:33:08.517 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: 33:06.437475+0000 osd.2 (osd 2021-02-10T04:33:08.517 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: .2) 539 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:08.517 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:33:08.517 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: 02-10T04:33:06.437485+0000 osd.2 (osd.2) 540 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:08.517 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:33:08.517 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: 04:33:06.437495+0000 osd.2 (osd.2) 2021-02-10T04:33:08.517 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: 541 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:08.517 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: cluster 2021-02-10T04:33:06.437502+0000 osd.2 (osd.2 2021-02-10T04:33:08.518 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: ) 542 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:08.518 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: cluster 2021-02-10T04:33:08.518 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: 2021-02-10T04:33:06.437507+0000 osd.2 (osd.2) 543 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:08.518 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:33:08.518 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: 33:06.437513+0000 osd.2 (osd.2) 544 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:08.518 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:33:08.518 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: 10T04:33:06.437520+0000 osd.2 (osd 2021-02-10T04:33:08.518 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: .2) 545 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:08.518 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: cluster 2021-02-10T04:33:07.154199+0000 mon.a (mon.0) 1144 : cluster [DBG] osdmap e370: 8 total, 8 up, 8 in 2021-02-10T04:33:08.518 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: cluster 2021-02-10T04:33:07.551055+0000 osd.4 (osd.4 2021-02-10T04:33:08.518 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: ) 391 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:08.518 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:33:08.518 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: 33:07.551067+0000 osd.4 (osd.4) 392 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:08.519 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: cluster 2021-02-10T04:33 2021-02-10T04:33:08.519 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: :07.551073+0000 osd.4 (osd 2021-02-10T04:33:08.519 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: .4) 393 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:08.519 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:33:08.519 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: 10T04:33:07.551079+0000 2021-02-10T04:33:08.519 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: osd.4 (osd.4) 394 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:08.519 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: 2021-02-10T04:33:08.519 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: audit 2021-02-10T04:33:07.945017+0000 mgr.y (mgr.14140) 507 : 2021-02-10T04:33:08.519 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:33:08.519 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: cluster 2021-02-10T04:33:08.066950+0000 mgr.y (mgr 2021-02-10T04:33:08.519 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: .14140) 508 : cluster [DBG] pgmap v705: 168 pgs: 5 peering, 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 152 active+clean; 279 MiB data, 880 MiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s wr, 8 op/s 2021-02-10T04:33:08.519 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[29288]: cluster 2021-02-10T04:33:08.117739+0000 mon.a (mon.0) 1145 : cluster [DBG] osdmap e371: 8 total, 8 up, 8 in 2021-02-10T04:33:08.520 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:08 smithi164 conmon[33677]: debug 2021-02-10T04:33:08.514+0000 7fbec19a1700 -1 osd.4 371 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:33:08.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[49106]: debug 2021-02-10T04:33:08.356+0000 7f77a3e94700 -1 osd.2 371 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:33:08.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: cluster 2021-02-10T04:33:06.437464+0000 2021-02-10T04:33:08.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: osd.2 (osd.2) 538 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:08.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: cluster 2021-02-10T04:33:06.437475+0000 osd.2 (osd.2) 539 : cluster [WRN] 2021-02-10T04:33:08.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:08.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:33:08.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: 33:06.437485+0000 osd.2 (osd.2) 540 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:08.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:08.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: -02-10T04:33:06.437495+0000 osd.2 (osd.2 2021-02-10T04:33:08.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: ) 541 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:08.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: cluster 2021-02-10T04:33:06.437502+0000 osd.2 2021-02-10T04:33:08.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: (osd.2) 542 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:08.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: 2021-02-10T04:33:08.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: cluster 2021-02-10T04:33:06 2021-02-10T04:33:08.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: .437507+0000 osd.2 (osd.2) 543 : 2021-02-10T04:33:08.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: cluster 2021-02-10T04:33: 2021-02-10T04:33:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: 06. 2021-02-10T04:33:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: 437513+0000 osd.2 (osd.2) 544 : cluster 2021-02-10T04:33:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:33:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: 10T04:33:06.437520+0000 2021-02-10T04:33:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: osd.2 (osd.2) 545 : cluster [WRN] 2021-02-10T04:33:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: cluster 2021-02-10T04:33:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: 2021 2021-02-10T04:33:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: -02-10T04:33:07. 2021-02-10T04:33:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: 154199+0000 mon.a (mon.0) 1144 : cluster [DBG] osdmap e370: 8 total, 8 up, 8 in 2021-02-10T04:33:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: cluster 2021-02-10T04:33:08.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:33:08.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: 33:07.551055+0000 osd.4 (osd.4) 2021-02-10T04:33:08.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: 391 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:08.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: cluster 2021-02-10T04:33:08.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: 2021-02-10T04:33:07. 2021-02-10T04:33:08.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: 551067+0000 osd.4 (osd.4) 392 : cluster 2021-02-10T04:33:08.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:08.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:33:08.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: -10T04:33:07.551073+0000 osd.4 2021-02-10T04:33:08.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: (osd.4) 393 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:08.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:33:08.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: 02-10T04:33:07.551079+0000 osd.4 2021-02-10T04:33:08.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: (osd.4) 394 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:08.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:33:08.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: 10T04:33:07.945017+0000 mgr.y (mgr.14140) 507 2021-02-10T04:33:08.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:33:08.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:33:08.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: -10T04:33:08.066950+0000 2021-02-10T04:33:08.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: mgr.y (mgr.14140) 508 : cluster [DBG] 2021-02-10T04:33:08.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: pgmap v705: 168 pgs: 5 peering, 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 152 active+clean; 279 MiB data, 880 MiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s wr, 8 op/s 2021-02-10T04:33:08.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:33:08.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: 02-10T04:33:08.117739+0000 mon.a (mon.0) 1145 2021-02-10T04:33:08.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[29319]: : cluster [DBG] osdmap e371: 8 total, 8 up, 8 in 2021-02-10T04:33:08.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: cluster 2021-02-10T04:33:08.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: 2021-02-10T04:33:08.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: 2021 2021-02-10T04:33:08.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: - 2021-02-10T04:33:08.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: 02 2021-02-10T04:33:08.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: - 2021-02-10T04:33:08.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: 10 2021-02-10T04:33:08.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: T 2021-02-10T04:33:08.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: 04 2021-02-10T04:33:08.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: : 2021-02-10T04:33:08.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: 33 2021-02-10T04:33:08.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: : 2021-02-10T04:33:08.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: 06 2021-02-10T04:33:08.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: . 2021-02-10T04:33:08.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: 437464+0000 osd.2 (osd.2) 2021-02-10T04:33:08.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: 538 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:08.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:08.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: -02-10T04:33:06.437475 2021-02-10T04:33:08.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 539 : cluster 2021-02-10T04:33:08.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:08.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:08.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: T04:33:06.437485+0000 osd.2 (osd.2 2021-02-10T04:33:08.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: ) 540 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:08.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:33:08.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: 33:06.437495 2021-02-10T04:33:08.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 541 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:08.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: 2021-02-10T04:33:08.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:08.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: 02-10T04:33:06.437502+0000 osd.2 ( 2021-02-10T04:33:08.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: osd.2) 542 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:08.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:08.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: 02-10T04:33:06.437507+0000 osd.2 2021-02-10T04:33:08.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: (osd.2) 543 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:08.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: cluster 2021-02-10T04:33:08.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: 2021-02-10T04:33:08.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: 2021 2021-02-10T04:33:08.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: -02-10T04:33: 2021-02-10T04:33:08.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: 06.437513+0000 osd.2 (osd.2) 544 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:08.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:08.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: -10T04:33:06.437520+0000 osd.2 (osd.2 2021-02-10T04:33:08.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: ) 545 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:08.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:08.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: -10T04:33:07.154199+0000 mon.a (mon 2021-02-10T04:33:08.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: .0) 1144 : cluster [DBG] osdmap e370: 8 total, 8 up, 8 in 2021-02-10T04:33:08.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: cluster 2021-02-10T04:33:08.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: 2021-02-10T04:33 2021-02-10T04:33:08.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: :07.551055+0000 osd.4 (osd.4) 391 2021-02-10T04:33:08.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:08.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: cluster 2021-02-10T04:33:08.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: 2021-02-10T04:33:07.551067 2021-02-10T04:33:08.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 392 : cluster [WRN] 2021-02-10T04:33:08.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:08.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:08.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: -10T04:33:07.551073+0000 osd.4 (osd.4) 393 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:08.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:33:08.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: 33:07.551079+0000 osd.4 (osd.4 2021-02-10T04:33:08.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: ) 394 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:08.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: audit 2021-02-10T04:33:08.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: 2021-02-10T04:33:08.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: 2021 2021-02-10T04:33:08.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: - 2021-02-10T04:33:08.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: 02 2021-02-10T04:33:08.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: - 2021-02-10T04:33:08.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: 10 2021-02-10T04:33:08.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: T 2021-02-10T04:33:08.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: 04 2021-02-10T04:33:08.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: : 2021-02-10T04:33:08.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: 33 2021-02-10T04:33:08.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: : 2021-02-10T04:33:08.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: 07 2021-02-10T04:33:08.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: . 2021-02-10T04:33:08.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: 945017 2021-02-10T04:33:08.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 507 2021-02-10T04:33:08.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:33:08.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:08.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: T04:33:08.066950+0000 mgr.y (mgr.14140) 508 : cluster [DBG] pgmap v705: 168 pgs: 5 peering, 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 152 active+clean; 279 MiB data, 880 MiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s wr, 8 op/s 2021-02-10T04:33:08.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:33:08.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: 10T04:33:08.117739+0000 mon.a (mon.0) 1145 : cluster [DBG] 2021-02-10T04:33:08.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:08 smithi085 conmon[35961]: osdmap e371: 8 total, 8 up, 8 in 2021-02-10T04:33:09.123 INFO:tasks.rados.rados.0.smithi085.stdout:1369: done (10 left) 2021-02-10T04:33:09.124 INFO:tasks.rados.rados.0.smithi085.stdout:1370: done (9 left) 2021-02-10T04:33:09.124 INFO:tasks.rados.rados.0.smithi085.stdout:1372: done (8 left) 2021-02-10T04:33:09.124 INFO:tasks.rados.rados.0.smithi085.stdout:1373: rollback oid 9 current snap is 143 2021-02-10T04:33:09.124 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 9 to 139 2021-02-10T04:33:09.124 INFO:tasks.rados.rados.0.smithi085.stdout:1374: read oid 2 snap -1 2021-02-10T04:33:09.124 INFO:tasks.rados.rados.0.smithi085.stdout:1374: expect (ObjNum 422 snap 109 seq_num 422) 2021-02-10T04:33:09.124 INFO:tasks.rados.rados.0.smithi085.stdout:1375: write (excl) oid 22 current snap is 143 2021-02-10T04:33:09.124 INFO:tasks.rados.rados.0.smithi085.stdout:1375: seq_num 497 ranges {753079=561698,2033433=319646} 2021-02-10T04:33:09.132 INFO:tasks.rados.rados.0.smithi085.stdout:1375: writing smithi08575397-22 from 753079 to 1314777 tid 1 2021-02-10T04:33:09.138 INFO:tasks.rados.rados.0.smithi085.stdout:1375: writing smithi08575397-22 from 2033433 to 2353079 tid 2 2021-02-10T04:33:09.138 INFO:tasks.rados.rados.0.smithi085.stdout:1376: read oid 36 snap 140 2021-02-10T04:33:09.138 INFO:tasks.rados.rados.0.smithi085.stdout:1376: expect deleted 2021-02-10T04:33:09.138 INFO:tasks.rados.rados.0.smithi085.stdout:1377: write oid 14 current snap is 143 2021-02-10T04:33:09.138 INFO:tasks.rados.rados.0.smithi085.stdout:1373: finishing rollback tid 0 to smithi08575397-9 2021-02-10T04:33:09.138 INFO:tasks.rados.rados.0.smithi085.stdout:1377: seq_num 498 ranges {769886=636947,2091915=277971} 2021-02-10T04:33:09.146 INFO:tasks.rados.rados.0.smithi085.stdout:1377: writing smithi08575397-14 from 769886 to 1406833 tid 1 2021-02-10T04:33:09.152 INFO:tasks.rados.rados.0.smithi085.stdout:1377: writing smithi08575397-14 from 2091915 to 2369886 tid 2 2021-02-10T04:33:09.153 INFO:tasks.rados.rados.0.smithi085.stdout:1378: copy_from oid 46 from oid 1 current snap is 143 2021-02-10T04:33:09.153 INFO:tasks.rados.rados.0.smithi085.stdout:1373: finishing rollback tid 1 to smithi08575397-9 2021-02-10T04:33:09.153 INFO:tasks.rados.rados.0.smithi085.stdout:1373: finishing rollback tid 2 to smithi08575397-9 2021-02-10T04:33:09.153 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 9 v 220 (ObjNum 229 snap 51 seq_num 229) dirty exists 2021-02-10T04:33:09.153 INFO:tasks.rados.rados.0.smithi085.stdout:1374: expect (ObjNum 422 snap 109 seq_num 422) 2021-02-10T04:33:09.395 INFO:tasks.rados.rados.0.smithi085.stdout:1375: finishing write tid 1 to smithi08575397-22 2021-02-10T04:33:09.396 INFO:tasks.rados.rados.0.smithi085.stdout:1373: done (13 left) 2021-02-10T04:33:09.396 INFO:tasks.rados.rados.0.smithi085.stdout:1374: done (12 left) 2021-02-10T04:33:09.396 INFO:tasks.rados.rados.0.smithi085.stdout:1379: copy_from oid 16 from oid 43 current snap is 143 2021-02-10T04:33:09.396 INFO:tasks.rados.rados.0.smithi085.stdout:1375: finishing write tid 2 to smithi08575397-22 2021-02-10T04:33:09.396 INFO:tasks.rados.rados.0.smithi085.stdout:1375: finishing write tid 3 to smithi08575397-22 2021-02-10T04:33:09.396 INFO:tasks.rados.rados.0.smithi085.stdout:1375: finishing write tid 4 to smithi08575397-22 2021-02-10T04:33:09.396 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 627 (ObjNum 497 snap 143 seq_num 497) dirty exists 2021-02-10T04:33:09.396 INFO:tasks.rados.rados.0.smithi085.stdout:1375: left oid 22 (ObjNum 497 snap 143 seq_num 497) 2021-02-10T04:33:09.396 INFO:tasks.rados.rados.0.smithi085.stdout:1377: finishing write tid 1 to smithi08575397-14 2021-02-10T04:33:09.396 INFO:tasks.rados.rados.0.smithi085.stdout:1377: finishing write tid 2 to smithi08575397-14 2021-02-10T04:33:09.396 INFO:tasks.rados.rados.0.smithi085.stdout:1377: finishing write tid 3 to smithi08575397-14 2021-02-10T04:33:09.397 INFO:tasks.rados.rados.0.smithi085.stdout:1377: finishing write tid 4 to smithi08575397-14 2021-02-10T04:33:09.397 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 223 (ObjNum 498 snap 143 seq_num 498) dirty exists 2021-02-10T04:33:09.397 INFO:tasks.rados.rados.0.smithi085.stdout:1377: left oid 14 (ObjNum 498 snap 143 seq_num 498) 2021-02-10T04:33:09.397 INFO:tasks.rados.rados.0.smithi085.stdout:1378: finishing copy_from to smithi08575397-46 2021-02-10T04:33:09.397 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 385 (ObjNum 424 snap 109 seq_num 424) dirty exists 2021-02-10T04:33:09.397 INFO:tasks.rados.rados.0.smithi085.stdout:1378: finishing copy_from racing read to smithi08575397-46 2021-02-10T04:33:09.397 INFO:tasks.rados.rados.0.smithi085.stdout:1375: done (12 left) 2021-02-10T04:33:09.397 INFO:tasks.rados.rados.0.smithi085.stdout:1376: done (11 left) 2021-02-10T04:33:09.397 INFO:tasks.rados.rados.0.smithi085.stdout:1377: done (10 left) 2021-02-10T04:33:09.397 INFO:tasks.rados.rados.0.smithi085.stdout:1378: done (9 left) 2021-02-10T04:33:09.397 INFO:tasks.rados.rados.0.smithi085.stdout:1380: read oid 36 snap 142 2021-02-10T04:33:09.397 INFO:tasks.rados.rados.0.smithi085.stdout:1380: expect (ObjNum 421 snap 109 seq_num 421) 2021-02-10T04:33:09.397 INFO:tasks.rados.rados.0.smithi085.stdout:1381: read oid 23 snap -1 2021-02-10T04:33:09.398 INFO:tasks.rados.rados.0.smithi085.stdout:1381: expect (ObjNum 471 snap 133 seq_num 471) 2021-02-10T04:33:09.398 INFO:tasks.rados.rados.0.smithi085.stdout:1382: rollback oid 22 current snap is 143 2021-02-10T04:33:09.398 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 22 to 140 2021-02-10T04:33:09.398 INFO:tasks.rados.rados.0.smithi085.stdout:1383: copy_from oid 48 from oid 7 current snap is 143 2021-02-10T04:33:09.398 INFO:tasks.rados.rados.0.smithi085.stdout:1384: read oid 6 snap -1 2021-02-10T04:33:09.398 INFO:tasks.rados.rados.0.smithi085.stdout:1384: expect deleted 2021-02-10T04:33:09.398 INFO:tasks.rados.rados.0.smithi085.stdout:1385: snap_create 2021-02-10T04:33:09.408 INFO:tasks.rados.rados.0.smithi085.stdout:1382: finishing rollback tid 0 to smithi08575397-22 2021-02-10T04:33:09.408 INFO:tasks.rados.rados.0.smithi085.stdout:1382: finishing rollback tid 1 to smithi08575397-22 2021-02-10T04:33:09.409 INFO:tasks.rados.rados.0.smithi085.stdout:1382: finishing rollback tid 2 to smithi08575397-22 2021-02-10T04:33:09.409 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 630 (ObjNum 464 snap 130 seq_num 464) dirty exists 2021-02-10T04:33:09.413 INFO:tasks.rados.rados.0.smithi085.stdout:1380: expect (ObjNum 421 snap 109 seq_num 421) 2021-02-10T04:33:09.467 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[49106]: debug 2021-02-10T04:33:09.468 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[49106]: 2021-02-10T04:33:09.404+0000 7f77a3e94700 -1 osd.2 372 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:33:09.468 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[49106]: 2021-02-10T04:33:09.468 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: cluster 2021-02-10T04:33:09.469 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 2021-02-10T04:33:09.469 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 2021 2021-02-10T04:33:09.469 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: - 2021-02-10T04:33:09.469 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 02-10T04:33:07. 2021-02-10T04:33:09.469 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 391748+0000 osd.2 (osd.2) 546 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:09.469 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: cluster 2021-02-10T04:33:09.469 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 2021-02-10T04:33:07.391756+0000 osd.2 (osd.2 2021-02-10T04:33:09.469 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: ) 547 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:09.469 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: cluster 2021-02-10T04:33:09.469 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 2021-02-10T04:33:09.469 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 2021 2021-02-10T04:33:09.469 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: -02-10T04:33: 2021-02-10T04:33:09.470 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 07.391762+0000 osd.2 (osd.2) 548 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:09.470 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: cluster 2021-02-10T04:33:09.470 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 2021-02-10T04:33:09.470 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:33:09.470 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: T04:33:07.391768+0000 osd.2 ( 2021-02-10T04:33:09.470 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: osd.2) 549 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:09.470 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: cluster 2021-02-10T04:33:09.470 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 2021-02-10T04:33:09.470 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:33:09.470 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: T04:33:07.391774+0000 osd.2 (osd. 2021-02-10T04:33:09.470 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 2) 550 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:09.470 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: cluster 2021-02-10T04:33:09.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 2021-02-10T04:33:09.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 2021-02-10T04:33 2021-02-10T04:33:09.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: :07.391780+0000 osd.2 (osd.2) 551 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:09.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: cluster 2021-02-10T04:33:09.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:33:09.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 10T04:33:07.391785+0000 osd.2 (osd 2021-02-10T04:33:09.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: .2) 552 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:09.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: cluster 2021-02-10T04:33:09.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 2021-02-10T04:33 2021-02-10T04:33:09.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: :07.391790+0000 osd.2 (osd 2021-02-10T04:33:09.471 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: .2) 553 : cluster [WRN] 2021-02-10T04:33:09.472 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:09.472 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: cluster 2021-02-10T04:33:09.472 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:33:09.472 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: T04:33:08.358285+0000 osd.2 (osd.2 2021-02-10T04:33:09.473 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: ) 554 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:09.474 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: cluster 2021-02-10T04:33:09.474 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:33:09.474 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: T04:33:08.358294+0000 2021-02-10T04:33:09.474 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: osd.2 (osd.2) 555 : 2021-02-10T04:33:09.474 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:09.474 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: cluster 2021-02-10T04:33:09.474 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 2021-02-10T04:33:08. 2021-02-10T04:33:09.474 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 358303+0000 osd.2 (osd.2) 556 2021-02-10T04:33:09.474 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:09.474 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: cluster 2021-02-10T04:33:09.475 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:33:09.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 10T04:33:08.358309 2021-02-10T04:33:09.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 2021-02-10T04:33:09.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 557 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:09.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: cluster 2021-02-10T04:33:09.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 2021-02-10T04:33: 2021-02-10T04:33:09.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 08.358318+0000 osd.2 (osd 2021-02-10T04:33:09.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: .2) 558 : cluster [WRN] 2021-02-10T04:33:09.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:09.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: cluster 2021-02-10T04:33:09.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 2021-02-10T04:33 2021-02-10T04:33:09.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: :08.358330+0000 osd.2 (osd.2) 559 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:09.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: cluster 2021-02-10T04:33:09.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 2021-02 2021-02-10T04:33:09.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: -10T04:33:08 2021-02-10T04:33:09.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: .358336+0000 osd.2 (osd. 2021-02-10T04:33:09.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 2) 560 : cluster [WRN] 2021-02-10T04:33:09.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:09.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: cluster 2021-02-10T04:33:09.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:33:09.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 10T04:33:08. 2021-02-10T04:33:09.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 358341+0000 osd.2 (osd.2 2021-02-10T04:33:09.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: ) 561 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:09.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: cluster 2021-02-10T04:33:09.483 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:33:09.483 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 04:33:08.516273+0000 osd.4 (osd.4 2021-02-10T04:33:09.483 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: ) 395 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:09.483 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: cluster 2021-02-10T04:33:09.483 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:33:09.483 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 10T04:33:08.516280+0000 2021-02-10T04:33:09.483 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: osd.4 (osd.4) 396 2021-02-10T04:33:09.483 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:09.483 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:09.483 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: -02-10T04:33: 2021-02-10T04:33:09.483 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 08.516283+0000 osd.4 (osd 2021-02-10T04:33:09.483 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: .4) 397 : cluster [WRN] 2021-02-10T04:33:09.483 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:09.484 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: cluster 2021-02-10T04:33:09.484 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 2021-02-10T04:33 2021-02-10T04:33:09.484 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: :08.516287+0000 osd.4 ( 2021-02-10T04:33:09.484 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: osd.4) 398 : cluster [WRN] 2021-02-10T04:33:09.484 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:09.484 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:33:09.484 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: 02-10T04:33:09 2021-02-10T04:33:09.484 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: .123660+0000 mon.a (mon.0 2021-02-10T04:33:09.484 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[29319]: ) 1146 : cluster [DBG] osdmap e372: 8 total, 8 up, 8 in 2021-02-10T04:33:09.484 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: cluster 2021-02-10T04:33:07.391748+0000 osd.2 ( 2021-02-10T04:33:09.484 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: osd.2) 546 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:09.485 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:09.485 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: T04:33:07.391756+0000 osd.2 (osd.2) 547 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:09.485 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: 2021-02-10T04:33:09.485 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: cluster 2021-02-10T04:33 2021-02-10T04:33:09.485 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: :07.391762+0000 osd.2 (osd.2) 548 : cluster 2021-02-10T04:33:09.485 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:09.485 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:33:09.485 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: 33:07.391768+0000 osd.2 (osd.2) 549 2021-02-10T04:33:09.485 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:09.485 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:09.485 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: 02-10T04:33:07.391774+0000 osd.2 2021-02-10T04:33:09.485 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: (osd.2) 550 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:09.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:09.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: -10T04:33:07.391780+0000 osd.2 (osd. 2021-02-10T04:33:09.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: 2) 551 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:09.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:09.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: -10T04:33:07.391785+0000 osd.2 ( 2021-02-10T04:33:09.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: osd.2) 552 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:09.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: cluster 2021-02-10T04:33:09.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: 2021-02-10T04:33:07.391790+0000 osd.2 2021-02-10T04:33:09.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: (osd.2) 553 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:09.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:09.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: -10T04:33:08.358285+0000 osd.2 (osd. 2021-02-10T04:33:09.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: 2) 554 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:09.487 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:09.487 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: -10T04:33:08.358294+0000 osd.2 2021-02-10T04:33:09.487 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: (osd.2) 555 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:09.487 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:09.487 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: -02-10T04:33:08.358303+0000 osd.2 2021-02-10T04:33:09.487 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: (osd.2) 556 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:09.487 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: cluster 2021-02-10T04:33:09.487 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:33:09.487 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: :33:08.358309+0000 osd.2 (osd.2) 557 : 2021-02-10T04:33:09.487 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:09.487 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: cluster 2021-02-10T04:33:09.487 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: 2021-02-10T04:33: 2021-02-10T04:33:09.487 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: 08.358318+0000 osd.2 (osd.2) 558 : cluster 2021-02-10T04:33:09.488 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:09.488 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:09.488 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: T04:33:08.358330+0000 osd.2 (osd. 2021-02-10T04:33:09.488 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: 2) 559 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:09.488 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:09.488 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: T04:33:08.358336+0000 osd.2 (osd.2 2021-02-10T04:33:09.488 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: ) 560 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:09.488 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:09.488 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: 02-10T04:33:08.358341+0000 osd.2 2021-02-10T04:33:09.488 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: (osd.2) 561 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:09.488 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:09.488 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: -10T04:33:08.516273+0000 osd.4 (osd. 2021-02-10T04:33:09.488 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: 4) 395 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:09.489 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: cluster 2021-02-10T04:33:09.489 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:33:09.489 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: 04:33:08.516280+0000 osd.4 (osd.4 2021-02-10T04:33:09.489 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: ) 396 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:09.489 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: cluster 2021-02-10T04:33:09.489 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:33:09.489 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: 04:33:08.516283+0000 osd.4 (osd. 2021-02-10T04:33:09.489 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: 4) 397 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:09.489 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: cluster 2021-02-10T04:33:09.489 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: 2021-02 2021-02-10T04:33:09.489 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: -10T04:33:08. 2021-02-10T04:33:09.489 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: 516287+0000 osd.4 (osd.4 2021-02-10T04:33:09.489 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: ) 398 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:09.490 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:33:09.490 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: 10T04:33:09.123660+0000 mon.a (mon.0) 1146 : cluster [DBG] 2021-02-10T04:33:09.490 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[35961]: osdmap e372: 8 total, 8 up, 8 in 2021-02-10T04:33:09.491 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[33677]: debug 2021-02-10T04:33:09.472+0000 7fbec19a1700 -1 osd.4 372 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:33:09.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: cluster 2021-02-10T04:33:09.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:33:09.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: 33:07.391748+0000 osd.2 (osd.2) 546 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:09.492 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: cluster 2021-02-10T04:33 2021-02-10T04:33:09.492 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: :07.391756+0000 osd.2 (osd.2) 547 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:09.492 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:33:09.492 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: 10T04:33:07.391762+0000 osd.2 (osd.2) 548 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:09.492 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: cluster 2021-02-10T04:33:09.492 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: 2021-02-10T04:33:07.391768 2021-02-10T04:33:09.492 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 549 : cluster [WRN] 2021-02-10T04:33:09.492 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:09.492 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: cluster 2021-02-10T04:33:09.492 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: 2021-02-10T04:33:07.391774+0000 osd.2 (osd.2 2021-02-10T04:33:09.492 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: ) 550 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:09.492 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:33:09.492 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: -02-10T04:33:07.391780+0000 2021-02-10T04:33:09.493 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: osd.2 (osd.2) 551 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:09.493 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:33:09.493 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: -02-10T04:33:07.391785+0000 2021-02-10T04:33:09.493 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: osd.2 (osd.2) 552 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:09.493 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: cluster 2021-02-10T04:33:07 2021-02-10T04:33:09.493 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: .391790+0000 osd.2 (osd.2) 553 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:09.493 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: cluster 2021-02-10T04:33:08.358285+0000 osd.2 ( 2021-02-10T04:33:09.493 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: osd.2) 554 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:09.493 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:33:09.493 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: 10T04:33:08.358294+0000 osd.2 (osd.2) 555 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:09.493 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:33:09.493 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: -10T04:33:08.358303+0000 osd.2 ( 2021-02-10T04:33:09.493 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: osd.2) 556 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:09.494 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:33:09.494 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: 04:33:08.358309+0000 osd.2 (osd. 2021-02-10T04:33:09.494 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: 2) 557 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:09.494 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: 2021-02-10T04:33:09.494 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:33:09.494 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: -10 2021-02-10T04:33:09.494 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: T04:33: 2021-02-10T04:33:09.494 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: 08.358318+0000 osd.2 ( 2021-02-10T04:33:09.494 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: osd.2) 558 : cluster [WRN] 2021-02-10T04:33:09.494 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:09.494 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: cluster 2021-02-10T04:33:09.494 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:33:09.494 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: 33:08.358330+0000 osd.2 2021-02-10T04:33:09.495 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: (osd.2) 559 : cluster [WRN] 2021-02-10T04:33:09.495 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:09.495 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: cluster 2021-02-10T04:33:08 2021-02-10T04:33:09.495 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: .358336+0000 osd.2 (osd.2) 560 : cluster [WRN] 2021-02-10T04:33:09.495 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:09.495 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: cluster 2021-02-10T04:33:08. 2021-02-10T04:33:09.495 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: 358341+0000 osd.2 (osd.2) 561 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:09.495 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: cluster 2021-02-10T04:33:08.516273 2021-02-10T04:33:09.495 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 395 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:09.495 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:33:09.495 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: T04:33:08.516280+0000 osd.4 (osd.4 2021-02-10T04:33:09.495 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: ) 396 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:09.495 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:33:09.495 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: 33:08.516283+0000 osd.4 (osd.4) 397 : 2021-02-10T04:33:09.496 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:09.496 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: cluster 2021-02-10T04:33:08.516287+0000 osd.4 ( 2021-02-10T04:33:09.496 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: osd.4) 398 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:09.496 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: cluster 2021-02-10T04:33 2021-02-10T04:33:09.496 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:09 smithi164 conmon[29288]: :09.123660+0000 mon.a (mon.0) 1146 : cluster [DBG] osdmap e372: 8 total, 8 up, 8 in 2021-02-10T04:33:09.518 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:33:09.680 INFO:tasks.rados.rados.0.smithi085.stdout:1381: expect (ObjNum 471 snap 133 seq_num 471) 2021-02-10T04:33:09.683 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:33:09.719 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:09.857 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[49106]: debug 2021-02-10T04:33:09.466+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:10.001 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:33:10.012 INFO:tasks.rados.rados.0.smithi085.stdout:1383: finishing copy_from to smithi08575397-48 2021-02-10T04:33:10.012 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 624 (ObjNum 476 snap 135 seq_num 476) dirty exists 2021-02-10T04:33:10.012 INFO:tasks.rados.rados.0.smithi085.stdout:1383: finishing copy_from racing read to smithi08575397-48 2021-02-10T04:33:10.012 INFO:tasks.rados.rados.0.smithi085.stdout:1379: finishing copy_from to smithi08575397-16 2021-02-10T04:33:10.013 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 625 (ObjNum 491 snap 141 seq_num 491) dirty exists 2021-02-10T04:33:10.013 INFO:tasks.rados.rados.0.smithi085.stdout:1379: finishing copy_from racing read to smithi08575397-16 2021-02-10T04:33:10.106 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:33:10.137 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:33:09 smithi085 conmon[41540]: debug 2021-02-10T04:33:09.958+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:10.140 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:10.235 INFO:tasks.rados.rados.0.smithi085.stdout:1379: done (14 left) 2021-02-10T04:33:10.235 INFO:tasks.rados.rados.0.smithi085.stdout:1380: done (13 left) 2021-02-10T04:33:10.235 INFO:tasks.rados.rados.0.smithi085.stdout:1381: done (12 left) 2021-02-10T04:33:10.235 INFO:tasks.rados.rados.0.smithi085.stdout:1382: done (11 left) 2021-02-10T04:33:10.235 INFO:tasks.rados.rados.0.smithi085.stdout:1383: done (10 left) 2021-02-10T04:33:10.235 INFO:tasks.rados.rados.0.smithi085.stdout:1384: done (9 left) 2021-02-10T04:33:10.235 INFO:tasks.rados.rados.0.smithi085.stdout:1385: done (8 left) 2021-02-10T04:33:10.235 INFO:tasks.rados.rados.0.smithi085.stdout:1386: read oid 11 snap -1 2021-02-10T04:33:10.235 INFO:tasks.rados.rados.0.smithi085.stdout:1386: expect (ObjNum 477 snap 137 seq_num 477) 2021-02-10T04:33:10.236 INFO:tasks.rados.rados.0.smithi085.stdout:1387: copy_from oid 3 from oid 32 current snap is 144 2021-02-10T04:33:10.236 INFO:tasks.rados.rados.0.smithi085.stdout:1388: read oid 7 snap -1 2021-02-10T04:33:10.236 INFO:tasks.rados.rados.0.smithi085.stdout:1388: expect (ObjNum 476 snap 135 seq_num 476) 2021-02-10T04:33:10.236 INFO:tasks.rados.rados.0.smithi085.stdout:1389: rollback oid 12 current snap is 144 2021-02-10T04:33:10.236 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 12 to 131 2021-02-10T04:33:10.236 INFO:tasks.rados.rados.0.smithi085.stdout:1390: snap_create 2021-02-10T04:33:10.240 INFO:tasks.rados.rados.0.smithi085.stdout:1388: expect (ObjNum 476 snap 135 seq_num 476) 2021-02-10T04:33:10.252 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:33:10.360 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:33:10.390 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:10.412 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:10 smithi085 conmon[49106]: debug 2021-02-10T04:33:10.224+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:10.498 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:10 smithi085 conmon[29319]: cluster 2021-02-10T04:33:10.498 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:10 smithi085 conmon[29319]: 2021-02-10T04:33:09. 2021-02-10T04:33:10.498 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:10 smithi085 conmon[29319]: 473665+0000 osd.4 (osd.4) 399 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:10.498 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:10 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:33:10.498 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:10 smithi085 conmon[29319]: -10T04:33:09.473671+0000 osd.4 (osd.4 2021-02-10T04:33:10.498 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:10 smithi085 conmon[29319]: ) 400 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:10.498 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:10 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:10.498 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:10 smithi085 conmon[29319]: -02-10T04:33:09.473675+0000 2021-02-10T04:33:10.499 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:10 smithi085 conmon[29319]: osd.4 (osd.4) 401 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:10.499 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:10 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:33:10.499 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:10 smithi085 conmon[29319]: 10T04:33:09.473679+0000 osd.4 (osd.4) 402 : 2021-02-10T04:33:10.499 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:10 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:10.499 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:10 smithi085 conmon[29319]: cluster 2021-02-10T04:33:10.067434+0000 2021-02-10T04:33:10.499 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:10 smithi085 conmon[29319]: mgr.y (mgr.14140) 509 : cluster [DBG] pgmap v708: 168 pgs: 5 peering, 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 153 active+clean; 286 MiB data, 905 MiB used, 714 GiB / 715 GiB avail; 3.3 MiB/s wr, 0 op/s 2021-02-10T04:33:10.499 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:10 smithi085 conmon[35961]: cluster 2021-02-10T04:33:09.473665+0000 osd.4 (osd 2021-02-10T04:33:10.499 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:10 smithi085 conmon[35961]: .4) 399 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:10.499 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:10 smithi085 conmon[35961]: cluster 2021-02-10T04:33:09.473671+0000 osd.4 (osd.4) 2021-02-10T04:33:10.499 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:10 smithi085 conmon[35961]: 400 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:10.499 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:10 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:33:10.500 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:10 smithi085 conmon[35961]: 10T04:33:09.473675+0000 osd.4 (osd.4) 401 : cluster [WRN] 2021-02-10T04:33:10.500 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:10 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:10.500 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:10 smithi085 conmon[35961]: cluster 2021-02-10T04:33:10.500 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:10 smithi085 conmon[35961]: 2021-02-10T04:33:09.473679+0000 osd.4 (osd. 2021-02-10T04:33:10.500 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:10 smithi085 conmon[35961]: 4) 402 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:10.500 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:10 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:10.500 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:10 smithi085 conmon[35961]: T04:33:10.067434+0000 mgr.y (mgr. 2021-02-10T04:33:10.500 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:10 smithi085 conmon[35961]: 14140) 509 : cluster [DBG] pgmap v708: 168 pgs: 5 peering, 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 153 active+clean; 286 MiB data, 905 MiB used, 714 GiB / 715 GiB avail; 3.3 MiB/s wr, 0 op/s 2021-02-10T04:33:10.500 INFO:tasks.rados.rados.0.smithi085.stdout:1389: finishing rollback tid 0 to smithi08575397-12 2021-02-10T04:33:10.500 INFO:tasks.rados.rados.0.smithi085.stdout:1386: expect (ObjNum 477 snap 137 seq_num 477) 2021-02-10T04:33:10.501 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:33:10.503 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:10 smithi164 conmon[29288]: cluster 2021-02-10T04:33:09.473665+0000 2021-02-10T04:33:10.503 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:10 smithi164 conmon[29288]: osd.4 (osd.4) 399 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:10.503 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:10 smithi164 conmon[29288]: cluster 2021-02-10T04:33:09.473671+0000 osd.4 (osd.4) 400 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:10.504 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:10 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:33:10.504 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:10 smithi164 conmon[29288]: 04:33:09.473675+0000 osd.4 (osd.4) 401 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:10.504 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:10 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:33:10.504 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:10 smithi164 conmon[29288]: 04:33:09.473679+0000 osd.4 (osd.4) 402 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:10.504 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:10 smithi164 conmon[29288]: cluster 2021-02-10T04:33:10.067434+0000 2021-02-10T04:33:10.504 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:10 smithi164 conmon[29288]: mgr.y (mgr.14140) 509 : cluster [DBG] pgmap v708: 168 pgs: 5 peering, 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 153 active+clean; 286 MiB data, 905 MiB used, 714 GiB / 715 GiB avail; 3.3 MiB/s wr, 0 op/s 2021-02-10T04:33:10.504 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:10 smithi164 conmon[33677]: debug 2021-02-10T04:33:10.433+0000 7fbec19a1700 -1 osd.4 373 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:33:10.504 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:10 smithi164 conmon[33677]: debug 2021-02-10T04:33:10.452+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:10.606 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:33:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:10.632 INFO:tasks.rados.rados.0.smithi085.stdout:1389: finishing rollback tid 1 to smithi08575397-12 2021-02-10T04:33:10.632 INFO:tasks.rados.rados.0.smithi085.stdout:1389: finishing rollback tid 2 to smithi08575397-12 2021-02-10T04:33:10.633 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 627 (ObjNum 85 snap 11 seq_num 85) dirty exists 2021-02-10T04:33:10.633 INFO:tasks.rados.rados.0.smithi085.stdout:1387: finishing copy_from to smithi08575397-3 2021-02-10T04:33:10.633 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 97 (ObjNum 204 snap 44 seq_num 204) dirty exists 2021-02-10T04:33:10.633 INFO:tasks.rados.rados.0.smithi085.stdout:1387: finishing copy_from racing read to smithi08575397-3 2021-02-10T04:33:10.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:10 smithi085 conmon[49106]: debug 2021-02-10T04:33:10.410+0000 7f77a3e94700 -1 osd.2 373 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:33:10.743 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:33:10.846 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:33:10.881 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:10.893 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:33:10 smithi164 conmon[44544]: debug 2021-02-10T04:33:10.695+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:11.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[49106]: debug 2021-02-10T04:33:11.055+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:11.233 INFO:tasks.rados.rados.0.smithi085.stdout:1386: done (12 left) 2021-02-10T04:33:11.233 INFO:tasks.rados.rados.0.smithi085.stdout:1387: done (11 left) 2021-02-10T04:33:11.234 INFO:tasks.rados.rados.0.smithi085.stdout:1388: done (10 left) 2021-02-10T04:33:11.234 INFO:tasks.rados.rados.0.smithi085.stdout:1389: done (9 left) 2021-02-10T04:33:11.234 INFO:tasks.rados.rados.0.smithi085.stdout:1390: done (8 left) 2021-02-10T04:33:11.234 INFO:tasks.rados.rados.0.smithi085.stdout:1391: read oid 3 snap 131 2021-02-10T04:33:11.234 INFO:tasks.rados.rados.0.smithi085.stdout:1391: expect (ObjNum 437 snap 116 seq_num 437) 2021-02-10T04:33:11.234 INFO:tasks.rados.rados.0.smithi085.stdout:1392: snap_create 2021-02-10T04:33:11.258 INFO:tasks.rados.rados.0.smithi085.stdout:1391: expect (ObjNum 437 snap 116 seq_num 437) 2021-02-10T04:33:11.376 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:33:11.631 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:33:11.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[33677]: debug 2021-02-10T04:33:11.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[33677]: 2021-02-10T04:33:11.396+0000 7fbec19a1700 -1 osd.4 374 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:33:11.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: cluster 2021-02-10T04:33:09.405556+0000 osd.2 2021-02-10T04:33:11.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: (osd.2) 562 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:11.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: cluster 2021-02-10T04:33:09.405563+0000 osd.2 (osd.2) 563 : cluster [WRN] 2021-02-10T04:33:11.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:11.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: cluster 2021-02-10T04:33:09.405567+0000 osd.2 (osd 2021-02-10T04:33:11.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: .2) 564 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:11.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:33:11.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: 33:09.405580+0000 osd.2 (osd.2) 565 : cluster 2021-02-10T04:33:11.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:11.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: cluster 2021-02-10T04:33:11.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: 2021-02-10T04:33:09.405586+0000 osd.2 (osd.2) 2021-02-10T04:33:11.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: 566 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:11.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: cluster 2021-02-10T04:33: 2021-02-10T04:33:11.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: 09.405604+0000 osd.2 (osd.2) 567 2021-02-10T04:33:11.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:11.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: cluster 2021-02-10T04:33:11.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: 2021-02-10T04:33:09. 2021-02-10T04:33:11.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: 405610+0000 osd.2 (osd.2) 568 : cluster [WRN] 2021-02-10T04:33:11.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:11.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: cluster 2021-02-10T04:33:11.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: 2021-02-10T04:33:09.405616+0000 osd.2 (osd.2 2021-02-10T04:33:11.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: ) 569 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:11.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: cluster 2021-02-10T04:33 2021-02-10T04:33:11.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: :10.231760+0000 mon.a (mon.0) 1147 : cluster 2021-02-10T04:33:11.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: [DBG] osdmap e373: 8 total, 8 up, 8 in 2021-02-10T04:33:11.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:33:11.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: T04:33:10.434504+0000 2021-02-10T04:33:11.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: osd.4 (osd.4) 403 : cluster 2021-02-10T04:33:11.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:11.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:33:11.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: -02-10T04:33:10.434514+0000 2021-02-10T04:33:11.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: osd.4 (osd.4) 404 : cluster 2021-02-10T04:33:11.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:11.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: cluster 2021-02-10T04:33:10 2021-02-10T04:33:11.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: .434520+0000 osd.4 (osd.4) 405 : cluster [WRN] 2021-02-10T04:33:11.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:11.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:33:11.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: 02-10T04:33:10.434537+0000 osd.4 (osd.4) 406 : cluster [WRN] 2021-02-10T04:33:11.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:11 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:11.662 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[49106]: debug 2021-02-10T04:33:11.432+0000 7f77a3e94700 -1 osd.2 374 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:33:11.662 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: cluster 2021-02-10T04:33:09.405556+0000 osd.2 (osd.2) 562 : cluster [WRN] 2021-02-10T04:33:11.662 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:11.663 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: cluster 2021-02-10T04:33:09.405563+0000 osd.2 (osd.2) 563 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:11.663 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: 2021-02-10T04:33:11.663 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:33:11.663 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: :33:09.405567+0000 osd.2 (osd.2) 564 : cluster [WRN] 2021-02-10T04:33:11.663 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:11.663 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: cluster 2021-02-10T04:33:09 2021-02-10T04:33:11.663 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: .405580+0000 osd.2 (osd.2) 565 : cluster [WRN] 2021-02-10T04:33:11.663 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:11.663 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: cluster 2021-02-10T04:33: 2021-02-10T04:33:11.663 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: 09.405586+0000 osd.2 (osd.2) 566 : cluster 2021-02-10T04:33:11.663 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:11.663 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: cluster 2021-02-10T04:33: 2021-02-10T04:33:11.664 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: 09.405604+0000 osd.2 (osd.2) 567 : cluster 2021-02-10T04:33:11.664 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:11.664 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:33:11.664 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: T04:33:09.405610+0000 osd.2 (osd.2 2021-02-10T04:33:11.664 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: ) 568 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:11.664 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:33:11.664 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: T04:33:09.405616+0000 osd.2 (osd. 2021-02-10T04:33:11.664 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: 2) 569 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:11.664 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:33:11.664 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: 10T04:33:10.231760+0000 mon.a (mon. 2021-02-10T04:33:11.664 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: 0) 1147 : cluster [DBG] osdmap e373: 8 total, 8 up, 8 in 2021-02-10T04:33:11.664 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:33:11.664 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: 02-10T04:33:10.434504+0000 osd.4 ( 2021-02-10T04:33:11.664 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: osd.4) 403 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:11.665 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:33:11.665 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: -10T04:33:10.434514+0000 osd.4 (osd. 2021-02-10T04:33:11.665 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: 4) 404 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:11.665 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:33:11.665 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: 04:33:10.434520+0000 osd.4 (osd.4 2021-02-10T04:33:11.665 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: ) 405 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:11.665 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:33:11.665 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: T04:33:10.434537+0000 osd.4 (osd.4) 2021-02-10T04:33:11.665 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[29319]: 406 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:11.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: cluster 2021-02-10T04:33:11.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: 2021-02-10T04:33:11.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: 2021 2021-02-10T04:33:11.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: - 2021-02-10T04:33:11.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: 02 2021-02-10T04:33:11.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: - 2021-02-10T04:33:11.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: 10 2021-02-10T04:33:11.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: T 2021-02-10T04:33:11.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: 04 2021-02-10T04:33:11.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: : 2021-02-10T04:33:11.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: 33 2021-02-10T04:33:11.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: :09.405556+0000 osd.2 ( 2021-02-10T04:33:11.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: osd.2) 562 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:11.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:11.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: T04:33:09.405563+0000 2021-02-10T04:33:11.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: osd.2 (osd.2) 563 : 2021-02-10T04:33:11.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:11.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:11.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: -10T04:33:09. 2021-02-10T04:33:11.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: 405567+0000 osd.2 (osd.2 2021-02-10T04:33:11.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: ) 564 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:11.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:11.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: T04:33:09.405580+0000 2021-02-10T04:33:11.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: osd.2 (osd.2) 565 : 2021-02-10T04:33:11.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:11.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: cluster 2021-02-10T04:33:09.405586+0000 2021-02-10T04:33:11.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: osd.2 (osd.2) 566 : cluster 2021-02-10T04:33:11.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:11.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:11.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: -02-10T04:33:09. 2021-02-10T04:33:11.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: 405604+0000 osd.2 (osd.2) 567 2021-02-10T04:33:11.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:11.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:33:11.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: 33:09.405610+0000 osd.2 ( 2021-02-10T04:33:11.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: osd.2) 568 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:11.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:11.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: -10T04:33:09.405616+0000 osd.2 ( 2021-02-10T04:33:11.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: osd.2) 569 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:11.669 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:11.669 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: -10T04:33:10.231760+0000 mon.a ( 2021-02-10T04:33:11.669 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: mon.0) 1147 : cluster [DBG] osdmap e373: 8 total, 8 up, 8 in 2021-02-10T04:33:11.669 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: cluster 2021-02-10T04:33:11.669 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:33:11.669 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: 33:10.434504+0000 osd.4 ( 2021-02-10T04:33:11.669 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: osd.4) 403 : cluster [WRN] 2021-02-10T04:33:11.669 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:11.669 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:33:11.669 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: 10T04:33:10.434514+0000 osd.4 ( 2021-02-10T04:33:11.669 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: osd.4) 404 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:11.669 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:33:11.669 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: :33:10.434520+0000 osd.4 (osd.4) 405 2021-02-10T04:33:11.669 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:11.670 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:33:11.670 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: :33:10.434537+0000 osd.4 (osd.4 2021-02-10T04:33:11.670 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:11 smithi085 conmon[35961]: ) 406 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:11.947 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:33:12.237 INFO:tasks.rados.rados.0.smithi085.stdout:1391: done (9 left) 2021-02-10T04:33:12.237 INFO:tasks.rados.rados.0.smithi085.stdout:1392: done (8 left) 2021-02-10T04:33:12.237 INFO:tasks.rados.rados.0.smithi085.stdout:1393: write (excl) oid 20 current snap is 146 2021-02-10T04:33:12.237 INFO:tasks.rados.rados.0.smithi085.stdout:1393: seq_num 499 ranges {786693=712196,2150397=236296} 2021-02-10T04:33:12.243 INFO:tasks.rados.rados.0.smithi085.stdout:1393: writing smithi08575397-20 from 786693 to 1498889 tid 1 2021-02-10T04:33:12.255 INFO:tasks.rados.rados.0.smithi085.stdout:1393: writing smithi08575397-20 from 2150397 to 2386693 tid 2 2021-02-10T04:33:12.255 INFO:tasks.rados.rados.0.smithi085.stdout:1394: snap_remove snap 139 2021-02-10T04:33:12.255 INFO:tasks.rados.rados.0.smithi085.stdout:1393: finishing write tid 1 to smithi08575397-20 2021-02-10T04:33:12.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[33677]: debug 2021-02-10T04:33:12.361+0000 7fbec19a1700 -1 osd.4 375 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:33:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: cluster 2021-02-10T04:33:10.412108+0000 osd.2 (osd.2) 570 : cluster [WRN] 2021-02-10T04:33:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: cluster 2021-02-10T04:33:10.412117+0000 osd.2 (osd.2) 571 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: cluster 2021-02-10T04:33:10.412123+0000 osd.2 (osd.2) 572 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:33:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: -10T04:33:10.412130+0000 osd.2 (osd.2) 573 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: cluster 2021-02-10T04:33:10.412137+0000 osd.2 (osd.2) 574 : cluster 2021-02-10T04:33:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: cluster 2021-02-10T04:33:10.412143+0000 osd.2 2021-02-10T04:33:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: (osd.2) 575 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: cluster 2021-02-10T04:33:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:33:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: 04:33:10.412149+0000 osd.2 (osd.2) 576 : cluster 2021-02-10T04:33:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: cluster 2021-02-10T04:33:10.412160+0000 2021-02-10T04:33:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: osd.2 (osd.2) 577 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:33:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: -02-10T04:33:11.232554+0000 mon.a (mon. 2021-02-10T04:33:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: 0) 1148 : cluster [DBG] osdmap e374: 8 total, 8 up, 8 in 2021-02-10T04:33:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: cluster 2021-02-10T04:33:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: 2021-02-10T04:33:11.397252+0000 osd.4 2021-02-10T04:33:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: (osd.4) 407 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:12.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: cluster 2021-02-10T04:33:11.397260+0000 2021-02-10T04:33:12.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: osd.4 (osd.4) 408 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:12.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: cluster 2021-02-10T04:33: 2021-02-10T04:33:12.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: 11.397281+0000 osd.4 (osd.4) 409 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:12.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: cluster 2021-02-10T04:33:11. 2021-02-10T04:33:12.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: 397287+0000 osd.4 (osd.4) 410 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:12.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: cluster 2021-02-10T04:33:12.067997+0000 mgr.y (mgr.14140) 510 2021-02-10T04:33:12.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: : cluster [DBG] pgmap v711: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 158 active+clean; 294 MiB data, 914 MiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 5.1 MiB/s wr, 21 op/s 2021-02-10T04:33:12.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:33:12.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: -10T04:33:12.233368+0000 mon.a (mon.0) 1149 : cluster [DBG] osdmap e375: 8 total, 8 up, 8 in 2021-02-10T04:33:12.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:12 smithi164 conmon[29288]: 2021-02-10T04:33:12.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[49106]: debug 2021-02-10T04:33:12.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[49106]: 2021-02-10T04:33:12.456+0000 7f77a3e94700 -1 osd.2 375 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:33:12.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: cluster 2021-02-10T04:33:10.412108+0000 osd.2 (osd. 2021-02-10T04:33:12.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: 2) 570 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:12.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: cluster 2021-02-10T04:33:10.412117+0000 osd.2 (osd.2) 571 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:12.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: cluster 2021-02-10T04:33:10.412123+0000 osd.2 (osd.2) 572 : cluster [WRN] 2021-02-10T04:33:12.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:12.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:12.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: -02-10T04:33:10.412130+0000 osd.2 (osd.2) 573 2021-02-10T04:33:12.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:12.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:33:12.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: 10T04:33:10.412137+0000 osd.2 (osd 2021-02-10T04:33:12.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: .2) 574 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:12.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:33:12.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: 04:33:10.412143+0000 osd.2 (osd.2) 575 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:12.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:33:12.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: 33:10.412149+0000 osd.2 (osd.2) 576 : cluster 2021-02-10T04:33:12.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:12.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: cluster 2021-02-10T04:33:10.412160 2021-02-10T04:33:12.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 577 : cluster [WRN] 2021-02-10T04:33:12.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:12.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: cluster 2021-02-10T04:33:11.232554+0000 2021-02-10T04:33:12.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: mon.a (mon.0) 1148 : cluster [DBG] osdmap e374: 8 total, 8 up, 8 in 2021-02-10T04:33:12.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: cluster 2021-02-10T04:33:11 2021-02-10T04:33:12.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: .397252+0000 osd.4 (osd.4) 407 : cluster [WRN] 2021-02-10T04:33:12.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:12.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: cluster 2021-02-10T04:33:12.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: 2021-02-10T04:33:11.397260+0000 osd.4 (osd.4) 408 : cluster 2021-02-10T04:33:12.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:12.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: cluster 2021-02-10T04:33:12.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: 2021-02-10T04:33:11.397281+0000 osd.4 (osd.4) 409 : cluster 2021-02-10T04:33:12.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:12.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:12.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: -02-10T04:33:11.397287+0000 osd.4 (osd.4) 410 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:12.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:33:12.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: 04:33:12.067997+0000 mgr.y (mgr.14140) 510 : cluster [DBG] pgmap v711: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 158 active+clean; 294 MiB data, 914 MiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 5.1 MiB/s wr, 21 op/s 2021-02-10T04:33:12.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:33:12.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: 04:33:12.233368+0000 mon.a (mon.0) 2021-02-10T04:33:12.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[29319]: 1149 : cluster [DBG] osdmap e375: 8 total, 8 up, 8 in 2021-02-10T04:33:12.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: cluster 2021-02-10T04:33:10.412108 2021-02-10T04:33:12.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 570 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:12.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:33:12.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: 10T04:33:10.412117+0000 osd.2 (osd.2) 571 : 2021-02-10T04:33:12.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:12.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:12.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: 02-10T04:33:10.412123+0000 osd.2 ( 2021-02-10T04:33:12.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: osd.2) 572 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:12.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:33:12.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: 10T04:33:10.412130+0000 osd.2 (osd 2021-02-10T04:33:12.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: .2) 573 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:12.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: cluster 2021-02-10T04:33: 2021-02-10T04:33:12.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: 10.412137+0000 osd.2 (osd.2) 574 : cluster 2021-02-10T04:33:12.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:12.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:12.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: T04:33:10.412143+0000 osd.2 ( 2021-02-10T04:33:12.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: osd.2) 575 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:12.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: cluster 2021-02-10T04:33:12.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: 2021-02-10T04:33:10. 2021-02-10T04:33:12.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: 412149+0000 osd.2 (osd.2) 576 : cluster 2021-02-10T04:33:12.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:12.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:33:12.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: 10T04:33:10.412160+0000 2021-02-10T04:33:12.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: osd.2 (osd.2) 577 : cluster [WRN] 2021-02-10T04:33:12.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:12.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:33:12.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: 33:11.232554+0000 mon.a (mon.0) 2021-02-10T04:33:12.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: 1148 : cluster [DBG] osdmap e374: 8 total, 8 up, 8 in 2021-02-10T04:33:12.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:33:12.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: 10T04:33:11.397252+0000 osd.4 (osd 2021-02-10T04:33:12.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: .4) 407 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:12.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:12.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: T04:33:11.397260+0000 osd.4 (osd. 2021-02-10T04:33:12.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: 4) 408 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:12.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:12.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: T04:33:11.397281+0000 osd.4 (osd 2021-02-10T04:33:12.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: .4) 409 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:12.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:12.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: -02-10T04:33:11.397287 2021-02-10T04:33:12.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 410 : cluster 2021-02-10T04:33:12.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:12.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: cluster 2021-02-10T04:33: 2021-02-10T04:33:12.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: 12.067997+0000 mgr.y (mgr.14140) 510 : 2021-02-10T04:33:12.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: cluster [DBG] pgmap v711: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 158 active+clean; 294 MiB data, 914 MiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 5.1 MiB/s wr, 21 op/s 2021-02-10T04:33:12.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:33:12.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: 33:12.233368+0000 mon.a (mon.0 2021-02-10T04:33:12.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:12 smithi085 conmon[35961]: ) 1149 : cluster [DBG] osdmap e375: 8 total, 8 up, 8 in 2021-02-10T04:33:13.274 INFO:tasks.rados.rados.0.smithi085.stdout:1394: done (9 left) 2021-02-10T04:33:13.357 INFO:tasks.rados.rados.0.smithi085.stdout:1395: write oid 46 current snap is 146 2021-02-10T04:33:13.358 INFO:tasks.rados.rados.0.smithi085.stdout:1395: seq_num 500 ranges {403500=787445,1692526=473837} 2021-02-10T04:33:13.359 INFO:tasks.rados.rados.0.smithi085.stdout:1395: writing smithi08575397-46 from 403500 to 1190945 tid 1 2021-02-10T04:33:13.359 INFO:tasks.rados.rados.0.smithi085.stdout:1395: writing smithi08575397-46 from 1692526 to 2166363 tid 2 2021-02-10T04:33:13.359 INFO:tasks.rados.rados.0.smithi085.stdout:1396: snap_create 2021-02-10T04:33:13.359 INFO:tasks.rados.rados.0.smithi085.stdout:1393: finishing write tid 2 to smithi08575397-20 2021-02-10T04:33:13.359 INFO:tasks.rados.rados.0.smithi085.stdout:1393: finishing write tid 3 to smithi08575397-20 2021-02-10T04:33:13.359 INFO:tasks.rados.rados.0.smithi085.stdout:1393: finishing write tid 4 to smithi08575397-20 2021-02-10T04:33:13.359 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 20 v 220 (ObjNum 499 snap 146 seq_num 499) dirty exists 2021-02-10T04:33:13.360 INFO:tasks.rados.rados.0.smithi085.stdout:1393: left oid 20 (ObjNum 499 snap 146 seq_num 499) 2021-02-10T04:33:13.360 INFO:tasks.rados.rados.0.smithi085.stdout:1395: finishing write tid 1 to smithi08575397-46 2021-02-10T04:33:13.360 INFO:tasks.rados.rados.0.smithi085.stdout:1395: finishing write tid 2 to smithi08575397-46 2021-02-10T04:33:13.360 INFO:tasks.rados.rados.0.smithi085.stdout:1395: finishing write tid 3 to smithi08575397-46 2021-02-10T04:33:13.360 INFO:tasks.rados.rados.0.smithi085.stdout:1395: finishing write tid 4 to smithi08575397-46 2021-02-10T04:33:13.360 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 670 (ObjNum 500 snap 146 seq_num 500) dirty exists 2021-02-10T04:33:13.360 INFO:tasks.rados.rados.0.smithi085.stdout:1395: left oid 46 (ObjNum 500 snap 146 seq_num 500) 2021-02-10T04:33:13.645 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[33677]: debug 2021-02-10T04:33:13.365+0000 7fbec19a1700 -1 osd.4 376 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:33:13.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: cluster 2021-02-10T04:33:11.434315+0000 osd.2 (osd.2) 578 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:13.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: cluster 2021-02-10T04:33:11.434322+0000 osd.2 (osd.2) 579 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:13.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: cluster 2021-02-10T04:33:11.434328+0000 osd.2 2021-02-10T04:33:13.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: (osd.2) 580 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:13.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: cluster 2021-02-10T04:33:13.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: 2021-02-10T04:33:11.434333+0000 osd.2 ( 2021-02-10T04:33:13.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: osd.2) 581 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:13.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:33:13.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: -02-10T04:33:11.434339+0000 osd.2 (osd.2) 2021-02-10T04:33:13.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: 582 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:13.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: cluster 2021-02-10T04:33:13.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: 2021-02-10T04:33:11.434344+0000 osd.2 (osd.2) 583 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:13.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: cluster 2021-02-10T04:33:11.434351+0000 osd.2 (osd.2) 584 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:13.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: cluster 2021-02-10T04:33:11.434357+0000 osd.2 2021-02-10T04:33:13.656 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: (osd.2) 585 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:13.656 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:33:13.657 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: -02-10T04:33:12.240483+0000 mon.a (mon 2021-02-10T04:33:13.657 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: .0) 1150 : cluster [WRN] Health check update: 12 slow ops, oldest one blocked for 222 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:33:13.657 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: cluster 2021-02-10T04:33:12.240515 2021-02-10T04:33:13.657 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1151 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2021-02-10T04:33:13.658 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:33:16.272 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: -10T04:33:12. 2021-02-10T04:33:16.272 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: 361858+0000 osd.4 (osd.4) 411 2021-02-10T04:33:16.273 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:16.273 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:33:16.273 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: -02-10T04:33:12 2021-02-10T04:33:16.273 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: .361869+0000 osd.4 (osd.4) 412 : 2021-02-10T04:33:16.273 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:16.273 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: cluster 2021-02-10T04:33:16.273 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: 2021-02-10T04:33:12.361875 2021-02-10T04:33:16.273 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 413 : cluster [WRN] 2021-02-10T04:33:16.273 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:16.273 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: cluster 2021-02-10T04:33:16.273 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:33:16.273 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: :33:12.361881+0000 osd.4 (osd. 2021-02-10T04:33:16.274 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:13 smithi164 conmon[29288]: 4) 414 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:16.274 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-02-10T04:33:16.274 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 0 1 2021-02-10T04:33:16.282 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[49106]: debug 2021-02-10T04:33:13.417+0000 7f77a3e94700 -1 osd.2 376 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:33:16.282 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[49106]: debug 2021-02-10T04:33:14.426+0000 7f77a3e94700 -1 osd.2 377 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:33:16.283 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[49106]: debug 2021-02-10T04:33:15.423+0000 7f77a3e94700 -1 osd.2 378 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:33:16.283 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: cluster 2021-02-10T04:33:11.434315+0000 osd.2 (osd. 2021-02-10T04:33:16.283 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: 2) 578 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:16.283 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:33:16.283 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: :33:11.434322+0000 osd.2 (osd.2) 579 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:16.283 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:33:16.283 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: 33:11.434328+0000 osd.2 (osd.2) 580 : cluster [WRN] 2021-02-10T04:33:16.284 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:16.284 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: cluster 2021-02-10T04:33:11 2021-02-10T04:33:16.284 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: .434333+0000 osd.2 (osd.2) 581 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:16.284 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:16.284 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: -10T04:33:11.434339+0000 osd.2 (osd. 2021-02-10T04:33:16.284 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: 2) 582 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:16.284 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: cluster 2021-02-10T04:33:16.284 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:33:16.284 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: :33:11.434344+0000 osd.2 (osd.2) 583 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:16.285 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:16.285 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: -02-10T04:33:11.434351+0000 2021-02-10T04:33:16.285 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: osd.2 (osd.2) 2021-02-10T04:33:16.285 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: 584 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:16.285 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:33:16.285 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: :33:11.434357+0000 osd.2 (osd 2021-02-10T04:33:16.285 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: .2) 585 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:16.285 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: cluster 2021-02-10T04:33:16.285 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: 2021-02-10T04:33: 2021-02-10T04:33:16.285 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: 12.240483+0000 mon.a (mon.0) 1150 : cluster [WRN] Health check update: 12 slow ops, oldest one blocked for 222 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:33:16.285 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: cluster 2021-02-10T04:33 2021-02-10T04:33:16.285 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: :12.240515+0000 mon.a (mon.0) 1151 : 2021-02-10T04:33:16.286 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2021-02-10T04:33:16.286 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: cluster 2021-02-10T04:33 2021-02-10T04:33:16.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: :12.361858+0000 osd.4 (osd. 2021-02-10T04:33:16.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: 4) 411 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:16.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: cluster 2021-02-10T04:33:16.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:33:16.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: 33:12.361869+0000 osd.4 (osd.4) 412 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:16.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: cluster 2021-02-10T04:33:12.361875 2021-02-10T04:33:16.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 413 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:16.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: cluster 2021-02-10T04:33:16.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:33:16.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[35961]: 04:33:12.361881+0000 osd.4 (osd.4) 414 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:16.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: cluster 2021-02-10T04:33:12. 2021-02-10T04:33:16.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: 457706+0000 osd.2 (osd.2) 586 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:16.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: cluster 2021-02-10T04:33:12.457714 2021-02-10T04:33:16.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 587 : cluster 2021-02-10T04:33:16.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:16.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: cluster 2021-02-10T04:33:12 2021-02-10T04:33:16.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: .457722+0000 osd.2 (osd.2) 588 : cluster 2021-02-10T04:33:16.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:16.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: cluster 2021-02-10T04:33: 2021-02-10T04:33:16.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: 12.457728+0000 osd.2 (osd.2) 589 2021-02-10T04:33:16.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:16.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:33:16.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: 33:12.457734+0000 osd.2 (osd.2) 2021-02-10T04:33:16.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: 590 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:16.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:33:16.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: 04:33:12.457739+0000 osd.2 (osd. 2021-02-10T04:33:16.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: 2) 591 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:16.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:16.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: -10T04:33:12.457744+0000 osd.2 2021-02-10T04:33:16.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: (osd.2) 592 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:16.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: 2021-02-10T04:33:16.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:16.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: -02-10T04:33:12.457751+0000 2021-02-10T04:33:16.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: osd.2 (osd.2) 593 : cluster [WRN] 2021-02-10T04:33:16.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:16.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: cluster 2021-02-10T04:33: 2021-02-10T04:33:16.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: 13.274274+0000 mon.a (mon.0) 1152 2021-02-10T04:33:16.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: : cluster [DBG] osdmap e376: 8 total, 8 up, 8 in 2021-02-10T04:33:16.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:33:16.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: 04:33:13.366376+0000 osd.4 (osd. 2021-02-10T04:33:16.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: 4) 415 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:16.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:33:16.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: 10T04:33:13.366382+0000 osd.4 ( 2021-02-10T04:33:16.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: osd.4) 416 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:16.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:16.292 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: -10T04:33:13.366386+0000 osd.4 2021-02-10T04:33:16.292 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: (osd.4) 417 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:16.292 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: 2021-02-10T04:33:16.292 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: cluster 2021-02-10T04:33:16.292 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: 2021-02-10T04:33:13.366389+0000 osd.4 ( 2021-02-10T04:33:16.292 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: osd.4) 418 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:16.292 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:16.292 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: 02-10T04:33:14.068450+0000 2021-02-10T04:33:16.292 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: mgr.y (mgr.14140) 511 : cluster [DBG] 2021-02-10T04:33:16.293 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[35961]: pgmap v714: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 159 active+clean; 297 MiB data, 927 MiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 2.8 MiB/s wr, 25 op/s 2021-02-10T04:33:16.293 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:33:16.293 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 04:33:13.419448+0000 osd.2 (osd.2) 594 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:16.293 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: cluster 2021-02-10T04:33:16.294 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 2021- 2021-02-10T04:33:16.294 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 02-10T04:33:13. 2021-02-10T04:33:16.294 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 419459+0000 osd.2 (osd.2 2021-02-10T04:33:16.294 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: ) 595 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:16.294 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 2021-02-10T04:33:16.294 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:33:16.294 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 33:13.419466+0000 osd.2 (osd.2) 596 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:16.294 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:16.295 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: -02-10T04:33:13.419471+0000 osd.2 (osd.2) 597 : cluster 2021-02-10T04:33:16.295 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:16.295 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: cluster 2021-02-10T04:33:16.295 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 2021-02-10T04:33:16.295 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 2021 2021-02-10T04:33:16.295 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: - 2021-02-10T04:33:16.296 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 02 2021-02-10T04:33:16.296 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: - 2021-02-10T04:33:16.296 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 10 2021-02-10T04:33:16.296 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: T 2021-02-10T04:33:16.296 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 04 2021-02-10T04:33:16.296 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: : 2021-02-10T04:33:16.296 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 33 2021-02-10T04:33:16.297 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: : 2021-02-10T04:33:16.297 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 13 2021-02-10T04:33:16.297 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: . 2021-02-10T04:33:16.297 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 419478 2021-02-10T04:33:16.297 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: +0000 2021-02-10T04:33:16.297 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: osd.2 (osd.2) 598 : cluster 2021-02-10T04:33:16.297 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:16.297 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: cluster 2021-02-10T04:33 2021-02-10T04:33:16.298 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: :13.419484+0000 osd.2 (osd.2) 599 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:16.298 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: cluster 2021-02-10T04:33:16.298 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 2021-02-10T04:33:16.298 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 2021 2021-02-10T04:33:16.298 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: - 2021-02-10T04:33:16.298 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 02 2021-02-10T04:33:16.298 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: - 2021-02-10T04:33:16.298 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 10 2021-02-10T04:33:16.298 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: T 2021-02-10T04:33:16.299 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 04 2021-02-10T04:33:16.299 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: : 2021-02-10T04:33:16.299 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 33 2021-02-10T04:33:16.299 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: : 2021-02-10T04:33:16.299 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 13 2021-02-10T04:33:16.299 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: . 2021-02-10T04:33:16.299 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 419490 2021-02-10T04:33:16.299 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: +0000 2021-02-10T04:33:16.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 2021-02-10T04:33:16.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: osd.2 2021-02-10T04:33:16.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: ( 2021-02-10T04:33:16.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: osd 2021-02-10T04:33:16.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: . 2021-02-10T04:33:16.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 2 2021-02-10T04:33:16.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: ) 2021-02-10T04:33:16.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 600 2021-02-10T04:33:16.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:16.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:16.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 02-10T04:33:13.419497+0000 osd.2 (osd.2 2021-02-10T04:33:16.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: ) 601 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:16.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: cluster 2021-02-10T04:33:16.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 2021-02-10T04:33:16.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 2021 2021-02-10T04:33:16.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: - 2021-02-10T04:33:16.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 02 2021-02-10T04:33:16.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: - 2021-02-10T04:33:16.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 10T04:33:14.281070+0000 2021-02-10T04:33:16.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: mon.a (mon.0) 1153 : cluster [DBG] osdmap e377: 8 total, 8 up, 8 in 2021-02-10T04:33:16.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:16.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 02-10T04:33:14.365691+0000 osd.4 (osd.4 2021-02-10T04:33:16.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: ) 419 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:16.303 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:16.303 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: -02-10T04:33:14 2021-02-10T04:33:16.303 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: .365697+0000 osd.4 (osd.4 2021-02-10T04:33:16.303 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: ) 420 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:16.303 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 2021-02-10T04:33:16.303 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: cluster 2021-02-10T04:33:16.303 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:33:16.304 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 33:14.365700+0000 osd.4 2021-02-10T04:33:16.304 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: (osd.4) 421 : cluster 2021-02-10T04:33:16.304 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:16.304 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:16.304 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: -02-10T04:33: 2021-02-10T04:33:16.304 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: 14.365704+0000 osd.4 (osd 2021-02-10T04:33:16.305 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: .4) 422 : cluster [WRN] 2021-02-10T04:33:16.305 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:16.305 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[29319]: cluster 2021-02-10T04:33:11.434315+0000 osd.2 ( 2021-02-10T04:33:16.305 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[29319]: osd.2) 578 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:16.305 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[29319]: cluster 2021-02-10T04:33:11.434322+0000 osd.2 (osd.2) 579 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:16.305 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[29319]: cluster 2021-02-10T04:33:11.434328+0000 osd.2 (osd.2) 580 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:16.306 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[29319]: cluster 2021-02-10T04:33:11.434333+0000 osd.2 (osd.2) 581 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:16.306 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[29319]: cluster 2021-02-10T04:33:11.434339+0000 osd.2 (osd.2) 582 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:16.306 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[29319]: cluster 2021-02-10T04:33:11.434344+0000 osd.2 (osd.2) 583 : cluster [WRN] 2021-02-10T04:33:16.306 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:16.306 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[29319]: cluster 2021-02-10T04:33:11.434351+0000 osd.2 (osd.2) 584 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:16.306 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[29319]: cluster 2021-02-10T04:33:11 2021-02-10T04:33:16.307 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[29319]: .434357+0000 osd.2 (osd.2) 585 : cluster 2021-02-10T04:33:16.307 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:16.307 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:33:16.307 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[29319]: 33:12.240483+0000 mon.a (mon.0 2021-02-10T04:33:16.307 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[29319]: ) 1150 : cluster [WRN] Health check update: 12 slow ops, oldest one blocked for 222 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:33:16.307 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:33:16.307 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[29319]: T04:33:12.240515+0000 mon.a (mon 2021-02-10T04:33:16.307 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[29319]: .0) 1151 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2021-02-10T04:33:16.307 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[29319]: cluster 2021-02-10T04:33:12.361858 2021-02-10T04:33:16.307 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 411 : cluster [WRN] 2021-02-10T04:33:16.307 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:16.308 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[29319]: cluster 2021-02-10T04:33:12 2021-02-10T04:33:16.308 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[29319]: .361869+0000 osd.4 (osd.4) 412 : 2021-02-10T04:33:16.308 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:16.308 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[29319]: cluster 2021-02-10T04:33:12.361875+0000 2021-02-10T04:33:16.308 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[29319]: osd.4 (osd.4) 413 : cluster [WRN] 2021-02-10T04:33:16.308 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:16.308 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[29319]: cluster 2021-02-10T04:33: 2021-02-10T04:33:16.308 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[29319]: 12.361881+0000 osd.4 (osd.4) 414 2021-02-10T04:33:16.308 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:13 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:16.308 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: cluster 2021-02-10T04:33:12.457706+0000 osd.2 (osd.2 2021-02-10T04:33:16.308 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: ) 586 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:16.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: cluster 2021-02-10T04:33:12.457714+0000 osd.2 (osd.2 2021-02-10T04:33:16.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: ) 587 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:16.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:33:16.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: 02-10T04:33:12.457722+0000 osd.2 (osd. 2021-02-10T04:33:16.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: 2) 588 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:16.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: cluster 2021-02-10T04:33:16.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: 2021-02-10T04:33: 2021-02-10T04:33:16.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: 12.457728+0000 osd.2 (osd.2) 589 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:16.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: cluster 2021-02-10T04:33:16.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:33:16.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: T04:33:12.457734+0000 osd.2 (osd 2021-02-10T04:33:16.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: .2) 590 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:16.310 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: cluster 2021-02-10T04:33:16.310 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: 2021-02-10T04:33:16.310 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: 2021-02-10T04:33 2021-02-10T04:33:16.310 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: :12.457739+0000 osd.2 (osd.2) 591 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:16.310 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: cluster 2021-02-10T04:33:16.310 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: 2021-02-10T04:33:12.457744+0000 osd.2 ( 2021-02-10T04:33:16.310 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: osd.2) 592 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:16.310 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: cluster 2021-02-10T04:33:16.311 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: 2021-02-10T04:33:12. 2021-02-10T04:33:16.311 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: 457751+0000 osd.2 (osd.2) 593 2021-02-10T04:33:16.311 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:16.311 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: cluster 2021-02-10T04:33:13 2021-02-10T04:33:16.311 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: .274274+0000 mon.a (mon.0) 1152 : cluster 2021-02-10T04:33:16.311 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: [DBG] osdmap e376: 8 total, 8 up, 8 in 2021-02-10T04:33:16.311 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:16.311 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: -02-10T04:33:13.366376+0000 osd.4 (osd.4) 415 : cluster [WRN] 2021-02-10T04:33:16.311 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:16.311 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:33:16.311 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: 04:33:13.366382+0000 osd.4 (osd.4) 416 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:16.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: cluster 2021-02-10T04:33 2021-02-10T04:33:16.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: :13.366386+0000 osd.4 (osd.4) 417 : cluster 2021-02-10T04:33:16.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:16.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: cluster 2021-02-10T04:33:13.366389+0000 2021-02-10T04:33:16.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: osd.4 (osd.4) 418 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:16.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:33:16.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: 10T04:33:14.068450+0000 mgr.y ( 2021-02-10T04:33:16.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:14 smithi085 conmon[29319]: mgr.14140) 511 : cluster [DBG] pgmap v714: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 159 active+clean; 297 MiB data, 927 MiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 2.8 MiB/s wr, 25 op/s 2021-02-10T04:33:16.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: cluster 2021-02-10T04:33:13.419448+0000 osd.2 (osd 2021-02-10T04:33:16.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: .2) 594 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:16.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: cluster 2021-02-10T04:33:13.419459+0000 osd.2 (osd.2) 595 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:16.312 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: cluster 2021-02-10T04:33:13.419466+0000 2021-02-10T04:33:16.313 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: osd.2 (osd.2) 596 : cluster [WRN] 2021-02-10T04:33:16.313 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:16.313 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:33:16.313 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: 02-10T04:33:13 2021-02-10T04:33:16.313 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: .419471+0000 osd.2 (osd.2) 597 : 2021-02-10T04:33:16.313 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:16.313 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:33:16.313 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: 04:33:13.419478+0000 osd.2 (osd 2021-02-10T04:33:16.313 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: .2) 598 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:16.313 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:33:16.313 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: -10T04:33:13.419484+0000 osd.2 ( 2021-02-10T04:33:16.313 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: osd.2) 599 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:16.314 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:33:16.314 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: T04:33:13.419490+0000 osd.2 (osd.2) 600 : cluster 2021-02-10T04:33:16.314 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:16.314 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: cluster 2021-02-10T04:33:16.314 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: 2021-02-10T04:33:13.419497+0000 osd.2 (osd.2) 601 : cluster 2021-02-10T04:33:16.314 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:16.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: cluster 2021-02-10T04:33:16.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: 2021-02-10T04:33:14.281070+0000 2021-02-10T04:33:16.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: mon.a (mon.0) 1153 : cluster [DBG] osdmap e377: 8 total, 8 up, 8 in 2021-02-10T04:33:16.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:33:16.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: :33:14.365691+0000 osd.4 (osd.4) 419 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:16.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: 2021-02-10T04:33:16.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:33:16.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: -10T04:33:14.365697+0000 osd.4 (osd.4) 420 : cluster 2021-02-10T04:33:16.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:16.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:33:16.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: T04:33:14.365700 2021-02-10T04:33:16.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 421 : cluster [WRN] 2021-02-10T04:33:16.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:16.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:33:16.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: 10T04:33:14. 2021-02-10T04:33:16.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: 365704+0000 osd.4 (osd.4) 2021-02-10T04:33:16.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:15 smithi085 conmon[29319]: 422 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:16.317 INFO:tasks.rados.rados.0.smithi085.stdout:1393: done (10 left) 2021-02-10T04:33:16.317 INFO:tasks.rados.rados.0.smithi085.stdout:1395: done (9 left) 2021-02-10T04:33:16.317 INFO:tasks.rados.rados.0.smithi085.stdout:1396: done (8 left) 2021-02-10T04:33:16.317 INFO:tasks.rados.rados.0.smithi085.stdout:1397: delete oid 13 current snap is 147 2021-02-10T04:33:16.317 INFO:tasks.rados.rados.0.smithi085.stdout:1397: done (8 left) 2021-02-10T04:33:16.317 INFO:tasks.rados.rados.0.smithi085.stdout:1398: delete oid 6 current snap is 147 2021-02-10T04:33:16.317 INFO:tasks.rados.rados.0.smithi085.stdout:1398: done (8 left) 2021-02-10T04:33:16.317 INFO:tasks.rados.rados.0.smithi085.stdout:1399: rollback oid 31 current snap is 147 2021-02-10T04:33:16.317 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 31 to 123 2021-02-10T04:33:16.317 INFO:tasks.rados.rados.0.smithi085.stdout:1400: write (excl) oid 7 current snap is 147 2021-02-10T04:33:16.317 INFO:tasks.rados.rados.0.smithi085.stdout:1400: seq_num 503 ranges {453921=729545,1584325=621164} 2021-02-10T04:33:16.318 INFO:tasks.rados.rados.0.smithi085.stdout:1400: writing smithi08575397-7 from 453921 to 1183466 tid 1 2021-02-10T04:33:16.318 INFO:tasks.rados.rados.0.smithi085.stdout:1400: writing smithi08575397-7 from 1584325 to 2205489 tid 2 2021-02-10T04:33:16.318 INFO:tasks.rados.rados.0.smithi085.stdout:1401: copy_from oid 12 from oid 30 current snap is 147 2021-02-10T04:33:16.318 INFO:tasks.rados.rados.0.smithi085.stdout:1399: finishing rollback tid 0 to smithi08575397-31 2021-02-10T04:33:16.318 INFO:tasks.rados.rados.0.smithi085.stdout:1399: finishing rollback tid 1 to smithi08575397-31 2021-02-10T04:33:16.318 INFO:tasks.rados.rados.0.smithi085.stdout:1399: finishing rollback tid 2 to smithi08575397-31 2021-02-10T04:33:16.318 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 633 (ObjNum 432 snap 114 seq_num 432) dirty exists 2021-02-10T04:33:16.318 INFO:tasks.rados.rados.0.smithi085.stdout:1400: finishing write tid 1 to smithi08575397-7 2021-02-10T04:33:16.318 INFO:tasks.rados.rados.0.smithi085.stdout:1399: done (10 left) 2021-02-10T04:33:16.319 INFO:tasks.rados.rados.0.smithi085.stdout:1402: write oid 37 current snap is 147 2021-02-10T04:33:16.319 INFO:tasks.rados.rados.0.smithi085.stdout:1402: seq_num 504 ranges {470728=404794,1526454=481175} 2021-02-10T04:33:16.319 INFO:tasks.rados.rados.0.smithi085.stdout:1402: writing smithi08575397-37 from 470728 to 875522 tid 1 2021-02-10T04:33:16.319 INFO:tasks.rados.rados.0.smithi085.stdout:1402: writing smithi08575397-37 from 1526454 to 2007629 tid 2 2021-02-10T04:33:16.319 INFO:tasks.rados.rados.0.smithi085.stdout:1403: delete oid 40 current snap is 147 2021-02-10T04:33:16.319 INFO:tasks.rados.rados.0.smithi085.stdout:1401: finishing copy_from to smithi08575397-12 2021-02-10T04:33:16.319 INFO:tasks.rados.rados.0.smithi085.stdout:1401: got expected ENOENT (src dne) 2021-02-10T04:33:16.319 INFO:tasks.rados.rados.0.smithi085.stdout:1401: finishing copy_from racing read to smithi08575397-12 2021-02-10T04:33:16.319 INFO:tasks.rados.rados.0.smithi085.stdout:1400: finishing write tid 2 to smithi08575397-7 2021-02-10T04:33:16.319 INFO:tasks.rados.rados.0.smithi085.stdout:1400: finishing write tid 3 to smithi08575397-7 2021-02-10T04:33:16.319 INFO:tasks.rados.rados.0.smithi085.stdout:1400: finishing write tid 4 to smithi08575397-7 2021-02-10T04:33:16.320 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 629 (ObjNum 503 snap 147 seq_num 503) dirty exists 2021-02-10T04:33:16.320 INFO:tasks.rados.rados.0.smithi085.stdout:1400: left oid 7 (ObjNum 503 snap 147 seq_num 503) 2021-02-10T04:33:16.320 INFO:tasks.rados.rados.0.smithi085.stdout:1402: finishing write tid 1 to smithi08575397-37 2021-02-10T04:33:16.320 INFO:tasks.rados.rados.0.smithi085.stdout:1402: finishing write tid 2 to smithi08575397-37 2021-02-10T04:33:16.320 INFO:tasks.rados.rados.0.smithi085.stdout:1402: finishing write tid 3 to smithi08575397-37 2021-02-10T04:33:16.320 INFO:tasks.rados.rados.0.smithi085.stdout:1402: finishing write tid 4 to smithi08575397-37 2021-02-10T04:33:16.320 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 777 (ObjNum 504 snap 147 seq_num 504) dirty exists 2021-02-10T04:33:16.320 INFO:tasks.rados.rados.0.smithi085.stdout:1402: left oid 37 (ObjNum 504 snap 147 seq_num 504) 2021-02-10T04:33:16.320 INFO:tasks.rados.rados.0.smithi085.stdout:1400: done (11 left) 2021-02-10T04:33:16.320 INFO:tasks.rados.rados.0.smithi085.stdout:1401: done (10 left) 2021-02-10T04:33:16.320 INFO:tasks.rados.rados.0.smithi085.stdout:1402: done (9 left) 2021-02-10T04:33:16.321 INFO:tasks.rados.rados.0.smithi085.stdout:1403: done (8 left) 2021-02-10T04:33:16.321 INFO:tasks.rados.rados.0.smithi085.stdout:1404: read oid 7 snap -1 2021-02-10T04:33:16.321 INFO:tasks.rados.rados.0.smithi085.stdout:1404: expect (ObjNum 503 snap 147 seq_num 503) 2021-02-10T04:33:16.321 INFO:tasks.rados.rados.0.smithi085.stdout:1405: write (excl) oid 37 current snap is 147 2021-02-10T04:33:16.321 INFO:tasks.rados.rados.0.smithi085.stdout:1405: seq_num 506 ranges {504342=555292,1643418=484844} 2021-02-10T04:33:16.321 INFO:tasks.rados.rados.0.smithi085.stdout:1405: writing smithi08575397-37 from 504342 to 1059634 tid 1 2021-02-10T04:33:16.321 INFO:tasks.rados.rados.0.smithi085.stdout:1405: writing smithi08575397-37 from 1643418 to 2128262 tid 2 2021-02-10T04:33:16.321 INFO:tasks.rados.rados.0.smithi085.stdout:1406: read oid 14 snap -1 2021-02-10T04:33:16.321 INFO:tasks.rados.rados.0.smithi085.stdout:1406: expect (ObjNum 498 snap 143 seq_num 498) 2021-02-10T04:33:16.321 INFO:tasks.rados.rados.0.smithi085.stdout:1407: write (excl) oid 3 current snap is 147 2021-02-10T04:33:16.321 INFO:tasks.rados.rados.0.smithi085.stdout:1407: seq_num 507 ranges {521149=630541,1701900=628502} 2021-02-10T04:33:16.321 INFO:tasks.rados.rados.0.smithi085.stdout:1407: writing smithi08575397-3 from 521149 to 1151690 tid 1 2021-02-10T04:33:16.322 INFO:tasks.rados.rados.0.smithi085.stdout:1407: writing smithi08575397-3 from 1701900 to 2330402 tid 2 2021-02-10T04:33:16.322 INFO:tasks.rados.rados.0.smithi085.stdout:1408: rollback oid 42 current snap is 147 2021-02-10T04:33:16.322 INFO:tasks.rados.rados.0.smithi085.stdout:1404: expect (ObjNum 503 snap 147 seq_num 503) 2021-02-10T04:33:16.322 INFO:tasks.rados.rados.0.smithi085.stdout:1405: finishing write tid 1 to smithi08575397-37 2021-02-10T04:33:16.322 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 42 to 128 2021-02-10T04:33:16.322 INFO:tasks.rados.rados.0.smithi085.stdout:1405: finishing write tid 2 to smithi08575397-37 2021-02-10T04:33:16.322 INFO:tasks.rados.rados.0.smithi085.stdout:1405: finishing write tid 3 to smithi08575397-37 2021-02-10T04:33:16.322 INFO:tasks.rados.rados.0.smithi085.stdout:1405: finishing write tid 4 to smithi08575397-37 2021-02-10T04:33:16.322 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 781 (ObjNum 506 snap 147 seq_num 506) dirty exists 2021-02-10T04:33:16.322 INFO:tasks.rados.rados.0.smithi085.stdout:1405: left oid 37 (ObjNum 506 snap 147 seq_num 506) 2021-02-10T04:33:16.322 INFO:tasks.rados.rados.0.smithi085.stdout:1406: expect (ObjNum 498 snap 143 seq_num 498) 2021-02-10T04:33:16.322 INFO:tasks.rados.rados.0.smithi085.stdout:1407: finishing write tid 1 to smithi08575397-3 2021-02-10T04:33:16.322 INFO:tasks.rados.rados.0.smithi085.stdout:1407: finishing write tid 2 to smithi08575397-3 2021-02-10T04:33:16.323 INFO:tasks.rados.rados.0.smithi085.stdout:1407: finishing write tid 3 to smithi08575397-3 2021-02-10T04:33:16.323 INFO:tasks.rados.rados.0.smithi085.stdout:1404: done (12 left) 2021-02-10T04:33:16.323 INFO:tasks.rados.rados.0.smithi085.stdout:1405: done (11 left) 2021-02-10T04:33:16.323 INFO:tasks.rados.rados.0.smithi085.stdout:1406: done (10 left) 2021-02-10T04:33:16.323 INFO:tasks.rados.rados.0.smithi085.stdout:1409: write (excl) oid 22 current snap is 147 2021-02-10T04:33:16.323 INFO:tasks.rados.rados.0.smithi085.stdout:1407: finishing write tid 4 to smithi08575397-3 2021-02-10T04:33:16.323 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 632 (ObjNum 507 snap 147 seq_num 507) dirty exists 2021-02-10T04:33:16.323 INFO:tasks.rados.rados.0.smithi085.stdout:1407: left oid 3 (ObjNum 507 snap 147 seq_num 507) 2021-02-10T04:33:16.323 INFO:tasks.rados.rados.0.smithi085.stdout:1409: seq_num 508 ranges {537956=705790,1644029=772160} 2021-02-10T04:33:16.323 INFO:tasks.rados.rados.0.smithi085.stdout:1409: writing smithi08575397-22 from 537956 to 1243746 tid 1 2021-02-10T04:33:16.323 INFO:tasks.rados.rados.0.smithi085.stdout:1409: writing smithi08575397-22 from 1644029 to 2416189 tid 2 2021-02-10T04:33:16.324 INFO:tasks.rados.rados.0.smithi085.stdout:1407: done (10 left) 2021-02-10T04:33:16.324 INFO:tasks.rados.rados.0.smithi085.stdout:1410: read oid 32 snap -1 2021-02-10T04:33:16.324 INFO:tasks.rados.rados.0.smithi085.stdout:1410: expect (ObjNum 204 snap 44 seq_num 204) 2021-02-10T04:33:16.324 INFO:tasks.rados.rados.0.smithi085.stdout:1411: snap_create 2021-02-10T04:33:16.324 INFO:tasks.rados.rados.0.smithi085.stdout:1408: finishing rollback tid 0 to smithi08575397-42 2021-02-10T04:33:16.324 INFO:tasks.rados.rados.0.smithi085.stdout:1408: finishing rollback tid 1 to smithi08575397-42 2021-02-10T04:33:16.325 INFO:tasks.rados.rados.0.smithi085.stdout:1408: finishing rollback tid 2 to smithi08575397-42 2021-02-10T04:33:16.325 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 630 (ObjNum 204 snap 44 seq_num 204) dirty exists 2021-02-10T04:33:16.325 INFO:tasks.rados.rados.0.smithi085.stdout:1409: finishing write tid 1 to smithi08575397-22 2021-02-10T04:33:16.325 INFO:tasks.rados.rados.0.smithi085.stdout:1409: finishing write tid 2 to smithi08575397-22 2021-02-10T04:33:16.325 INFO:tasks.rados.rados.0.smithi085.stdout:1409: finishing write tid 3 to smithi08575397-22 2021-02-10T04:33:16.325 INFO:tasks.rados.rados.0.smithi085.stdout:1409: finishing write tid 4 to smithi08575397-22 2021-02-10T04:33:16.325 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 636 (ObjNum 508 snap 147 seq_num 508) dirty exists 2021-02-10T04:33:16.325 INFO:tasks.rados.rados.0.smithi085.stdout:1409: left oid 22 (ObjNum 508 snap 147 seq_num 508) 2021-02-10T04:33:16.326 INFO:tasks.rados.rados.0.smithi085.stdout:1410: expect (ObjNum 204 snap 44 seq_num 204) 2021-02-10T04:33:16.326 INFO:tasks.rados.rados.0.smithi085.stdout:1408: done (11 left) 2021-02-10T04:33:16.326 INFO:tasks.rados.rados.0.smithi085.stdout:1409: done (10 left) 2021-02-10T04:33:16.326 INFO:tasks.rados.rados.0.smithi085.stdout:1410: done (9 left) 2021-02-10T04:33:16.326 INFO:tasks.rados.rados.0.smithi085.stdout:1411: done (8 left) 2021-02-10T04:33:16.326 INFO:tasks.rados.rados.0.smithi085.stdout:1412: read oid 6 snap 109 2021-02-10T04:33:16.326 INFO:tasks.rados.rados.0.smithi085.stdout:1412: expect (ObjNum 419 snap 109 seq_num 419) 2021-02-10T04:33:16.326 INFO:tasks.rados.rados.0.smithi085.stdout:1413: snap_remove snap 140 2021-02-10T04:33:16.327 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[33677]: debug 2021-02-10T04:33:14.364+0000 7fbec19a1700 -1 osd.4 377 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:33:16.327 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[33677]: debug 2021-02-10T04:33:15.394+0000 7fbec19a1700 -1 osd.4 378 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:33:16.327 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: cluster 2021-02-10T04:33:12.457706+0000 osd.2 (osd.2) 586 : 2021-02-10T04:33:16.327 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:16.328 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: cluster 2021-02-10T04:33:12.457714+0000 osd.2 (osd.2) 587 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:16.328 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: cluster 2021-02-10T04:33:12.457722+0000 osd.2 (osd.2) 588 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:16.328 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: cluster 2021-02-10T04:33:12.457728+0000 osd.2 (osd.2) 589 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:16.328 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:33:16.328 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: 33:12.457734+0000 osd.2 (osd.2) 590 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:16.328 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: cluster 2021-02-10T04:33: 2021-02-10T04:33:16.328 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: 12.457739+0000 osd.2 (osd.2) 591 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:16.329 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:33:16.329 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: T04:33:12.457744+0000 osd.2 (osd 2021-02-10T04:33:16.329 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: .2) 592 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:16.329 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:33:16.329 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: :33:12.457751+0000 osd.2 (osd.2) 593 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:16.329 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: cluster 2021-02-10T04:33: 2021-02-10T04:33:16.329 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: 13.274274+0000 mon.a (mon.0) 1152 : cluster [DBG] osdmap e376: 8 total, 8 up, 8 in 2021-02-10T04:33:16.330 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:33:16.330 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: -10T04:33:13.366376+0000 osd.4 (osd.4) 415 : cluster [WRN] 2021-02-10T04:33:16.330 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:16.330 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: cluster 2021-02-10T04:33:13.366382+0000 osd.4 (osd 2021-02-10T04:33:16.330 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: .4) 416 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:16.330 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: cluster 2021-02-10T04:33:16.331 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:33:16.331 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: 33:13.366386+0000 osd.4 ( 2021-02-10T04:33:16.331 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: osd.4) 417 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:16.331 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: cluster 2021-02-10T04:33:13.366389 2021-02-10T04:33:16.331 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 418 : cluster [WRN] 2021-02-10T04:33:16.331 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:16.331 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: cluster 2021-02-10T04:33:14.068450+0000 mgr.y (mgr 2021-02-10T04:33:16.332 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: .14140) 511 : 2021-02-10T04:33:16.332 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: cluster [DBG] 2021-02-10T04:33:16.332 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:14 smithi164 conmon[29288]: pgmap v714: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 159 active+clean; 297 MiB data, 927 MiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 2.8 MiB/s wr, 25 op/s 2021-02-10T04:33:16.332 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: cluster 2021-02-10T04:33:13.419448+0000 osd.2 (osd.2) 594 : 2021-02-10T04:33:16.332 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:16.332 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: cluster 2021-02-10T04:33:13.419459+0000 osd.2 (osd.2) 595 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:16.333 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: cluster 2021-02-10T04:33:13.419466 2021-02-10T04:33:16.333 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 596 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:16.333 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: cluster 2021-02-10T04:33:16.333 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: 2021-02-10T04:33:13.419471 2021-02-10T04:33:16.333 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 597 : cluster 2021-02-10T04:33:16.334 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:16.334 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:33:16.334 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: -02-10T04:33:13.419478+0000 2021-02-10T04:33:16.334 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: osd.2 (osd.2) 598 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:16.334 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:33:16.334 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: 10T04:33:13.419484+0000 osd.2 (osd.2) 599 : 2021-02-10T04:33:16.335 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:16.335 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: cluster 2021-02-10T04:33:16.335 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: 2021-02-10T04:33:13.419490+0000 2021-02-10T04:33:16.336 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: osd.2 (osd.2) 600 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:16.336 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: cluster 2021-02-10T04:33:16.336 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: 2021-02-10T04:33:13.419497 2021-02-10T04:33:16.336 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 601 : cluster [WRN] 2021-02-10T04:33:16.336 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:16.337 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:33:16.337 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: -10T04:33:14.281070+0000 mon.a (mon.0) 1153 2021-02-10T04:33:16.337 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: : cluster [DBG] osdmap e377: 8 total, 8 up, 8 in 2021-02-10T04:33:16.337 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:33:16.337 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: -02-10T04:33:14.365691+0000 2021-02-10T04:33:16.337 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: osd.4 (osd.4) 419 : cluster [WRN] 2021-02-10T04:33:16.338 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:16.338 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:33:16.338 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: 02-10T04:33:14.365697 2021-02-10T04:33:16.338 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 420 : cluster 2021-02-10T04:33:16.338 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:16.339 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: cluster 2021-02-10T04:33:16.339 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: 2021 2021-02-10T04:33:16.339 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: - 2021-02-10T04:33:16.339 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: 02-10T04:33:14.365700+0000 osd.4 (osd.4 2021-02-10T04:33:16.339 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: ) 421 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:16.339 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:33:16.339 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:15 smithi164 conmon[29288]: :33:14.365704+0000 osd.4 (osd.4) 422 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:16.340 INFO:tasks.rados.rados.0.smithi085.stdout:1413: done (9 left) 2021-02-10T04:33:16.340 INFO:tasks.rados.rados.0.smithi085.stdout:1414: write (excl) oid 20 current snap is 148 2021-02-10T04:33:16.340 INFO:tasks.rados.rados.0.smithi085.stdout:1412: expect (ObjNum 419 snap 109 seq_num 419) 2021-02-10T04:33:16.440 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:33:16.468 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:16.532 INFO:tasks.rados.rados.0.smithi085.stdout:1414: seq_num 509 ranges {554763=781039,2102511=452252} 2021-02-10T04:33:16.540 INFO:tasks.rados.rados.0.smithi085.stdout:1414: writing smithi08575397-20 from 554763 to 1335802 tid 1 2021-02-10T04:33:16.547 INFO:tasks.rados.rados.0.smithi085.stdout:1414: writing smithi08575397-20 from 2102511 to 2554763 tid 2 2021-02-10T04:33:16.547 INFO:tasks.rados.rados.0.smithi085.stdout:1412: done (9 left) 2021-02-10T04:33:16.547 INFO:tasks.rados.rados.0.smithi085.stdout:1415: read oid 10 snap -1 2021-02-10T04:33:16.547 INFO:tasks.rados.rados.0.smithi085.stdout:1415: expect deleted 2021-02-10T04:33:16.547 INFO:tasks.rados.rados.0.smithi085.stdout:1416: write (excl) oid 46 current snap is 148 2021-02-10T04:33:16.547 INFO:tasks.rados.rados.0.smithi085.stdout:1416: seq_num 510 ranges {571570=572641,1877346=694224} 2021-02-10T04:33:16.555 INFO:tasks.rados.rados.0.smithi085.stdout:1416: writing smithi08575397-46 from 571570 to 1144211 tid 1 2021-02-10T04:33:16.565 INFO:tasks.rados.rados.0.smithi085.stdout:1416: writing smithi08575397-46 from 1877346 to 2571570 tid 2 2021-02-10T04:33:16.565 INFO:tasks.rados.rados.0.smithi085.stdout:1414: finishing write tid 1 to smithi08575397-20 2021-02-10T04:33:16.565 INFO:tasks.rados.rados.0.smithi085.stdout:1414: finishing write tid 2 to smithi08575397-20 2021-02-10T04:33:16.566 INFO:tasks.rados.rados.0.smithi085.stdout:1414: finishing write tid 3 to smithi08575397-20 2021-02-10T04:33:16.566 INFO:tasks.rados.rados.0.smithi085.stdout:1414: finishing write tid 4 to smithi08575397-20 2021-02-10T04:33:16.566 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 20 v 223 (ObjNum 509 snap 148 seq_num 509) dirty exists 2021-02-10T04:33:16.566 INFO:tasks.rados.rados.0.smithi085.stdout:1414: left oid 20 (ObjNum 509 snap 148 seq_num 509) 2021-02-10T04:33:16.566 INFO:tasks.rados.rados.0.smithi085.stdout:1416: finishing write tid 1 to smithi08575397-46 2021-02-10T04:33:16.566 INFO:tasks.rados.rados.0.smithi085.stdout:1414: done (10 left) 2021-02-10T04:33:16.566 INFO:tasks.rados.rados.0.smithi085.stdout:1415: done (9 left) 2021-02-10T04:33:16.566 INFO:tasks.rados.rados.0.smithi085.stdout:1417: read oid 20 snap -1 2021-02-10T04:33:16.566 INFO:tasks.rados.rados.0.smithi085.stdout:1417: expect (ObjNum 509 snap 148 seq_num 509) 2021-02-10T04:33:16.566 INFO:tasks.rados.rados.0.smithi085.stdout:1418: snap_remove snap 131 2021-02-10T04:33:16.574 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:33:16.578 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[33677]: debug 2021-02-10T04:33:16.362+0000 7fbec19a1700 -1 osd.4 379 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:33:16.578 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[40892]: debug 2021-02-10T04:33:16.578 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[40892]: 2021-02-10T04:33:16.544+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:16.578 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[40892]: 2021-02-10T04:33:16.579 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: cluster 2021-02-10T04:33:16.579 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:33:16.579 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: :33:14.427921+0000 osd.2 (osd.2) 602 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:16.579 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:33:16.580 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: 02-10T04:33:14.427930+0000 osd.2 ( 2021-02-10T04:33:16.580 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: osd.2) 603 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:16.580 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: cluster 2021-02-10T04:33:16.580 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: 2021-02-10T04:33:14.427936+0000 2021-02-10T04:33:16.580 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: osd.2 (osd.2) 604 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:16.580 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:33:16.580 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: -10T04:33:14.427947+0000 osd.2 (osd.2) 605 : cluster [WRN] 2021-02-10T04:33:16.580 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:16.580 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: cluster 2021-02-10T04:33:16.580 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: 2021-02-10T04:33:14.427953+0000 osd.2 (osd.2) 606 : cluster 2021-02-10T04:33:16.580 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:16.580 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:33:16.580 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: 04:33:14.427959+0000 osd.2 (osd.2 2021-02-10T04:33:16.581 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: ) 607 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:16.581 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:33:16.581 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: :33:14.427965+0000 osd.2 (osd.2 2021-02-10T04:33:16.581 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: ) 608 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:16.581 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:33:16.581 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: :33:14.427970+0000 osd.2 (osd.2) 2021-02-10T04:33:16.581 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: 609 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:16.581 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:33:16.581 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: 33:15.293934+0000 mon.a (mon.0) 2021-02-10T04:33:16.581 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: 1154 : cluster [DBG] osdmap e378: 8 total, 8 up, 8 in 2021-02-10T04:33:16.581 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:33:16.581 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: 33:15.394778+0000 osd.4 (osd.4) 423 2021-02-10T04:33:16.581 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:16.582 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: cluster 2021-02-10T04:33: 2021-02-10T04:33:16.582 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: 15.394796+0000 osd.4 (osd.4) 424 : cluster 2021-02-10T04:33:16.582 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:16.582 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: cluster 2021-02-10T04:33:15. 2021-02-10T04:33:16.582 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: 394807+0000 osd.4 (osd.4) 425 : cluster 2021-02-10T04:33:16.582 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:16.582 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: cluster 2021-02-10T04:33:16.582 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: 2021-02-10T04:33:15.394812+0000 osd.4 ( 2021-02-10T04:33:16.582 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: osd.4) 426 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:16.582 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:33:16.582 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: 10T04:33:16.069055+0000 mgr.y ( 2021-02-10T04:33:16.582 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[29288]: mgr.14140) 512 : cluster [DBG] pgmap v717: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 159 active+clean; 303 MiB data, 938 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 3.7 MiB/s wr, 24 op/s 2021-02-10T04:33:16.680 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:33:16.712 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:16.712 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[49106]: debug 2021-02-10T04:33:16.455+0000 7f77a3e94700 -1 osd.2 379 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:33:16.713 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: cluster 2021-02-10T04:33:14.427921 2021-02-10T04:33:16.713 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 602 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:16.713 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: cluster 2021-02-10T04:33:14.427930+0000 osd.2 (osd.2) 603 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:16.713 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: cluster 2021-02-10T04:33:14.427936+0000 osd.2 2021-02-10T04:33:16.713 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: (osd.2) 604 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:16.713 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:16.713 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: -02-10T04:33:14.427947+0000 osd.2 2021-02-10T04:33:16.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: (osd.2) 605 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:16.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:33:16.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: 02-10T04:33:14.427953+0000 osd.2 2021-02-10T04:33:16.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: (osd.2) 606 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:16.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:33:16.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: 02-10T04:33:14.427959+0000 osd.2 2021-02-10T04:33:16.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: (osd.2) 607 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:16.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:33:16.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: 02-10T04:33:14.427965+0000 osd.2 2021-02-10T04:33:16.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: (osd.2) 608 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:16.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: 2021-02-10T04:33:16.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:16.715 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: -02-10T04:33:14.427970+0000 osd.2 (osd. 2021-02-10T04:33:16.715 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: 2) 609 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:16.715 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: cluster 2021-02-10T04:33:15.293934+0000 mon.a (mon.0) 1154 : cluster [DBG] osdmap e378: 8 total, 8 up, 8 in 2021-02-10T04:33:16.715 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:33:16.715 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: 33:15.394778+0000 osd.4 (osd.4) 423 2021-02-10T04:33:16.715 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:16.715 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:33:16.715 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: 33:15.394796+0000 osd.4 (osd.4) 2021-02-10T04:33:16.715 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: 424 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:16.715 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:33:16.715 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: 33:15.394807+0000 osd.4 (osd.4 2021-02-10T04:33:16.715 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: ) 425 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:16.715 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:33:16.716 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: 04:33:15.394812+0000 osd.4 (osd. 2021-02-10T04:33:16.716 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: 4) 426 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:16.716 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:33:16.716 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: 04:33:16.069055+0000 mgr.y (mgr.14140 2021-02-10T04:33:16.716 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[29319]: ) 512 : cluster [DBG] pgmap v717: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 159 active+clean; 303 MiB data, 938 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 3.7 MiB/s wr, 24 op/s 2021-02-10T04:33:16.716 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: cluster 2021-02-10T04:33:14.427921+0000 2021-02-10T04:33:16.716 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: osd.2 (osd.2) 602 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:16.716 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: cluster 2021-02-10T04:33 2021-02-10T04:33:16.716 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: :14.427930+0000 osd.2 (osd.2) 603 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:16.716 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:16.716 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: T04:33:14.427936+0000 osd.2 (osd. 2021-02-10T04:33:16.717 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: 2) 604 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:16.717 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:16.717 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: T04:33:14.427947+0000 osd.2 (osd 2021-02-10T04:33:16.717 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: .2) 605 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:16.717 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:33:16.717 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: 10T04:33:14.427953+0000 osd.2 ( 2021-02-10T04:33:16.717 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: osd.2) 606 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:16.717 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:16.717 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: -10T04:33:14.427959+0000 osd.2 ( 2021-02-10T04:33:16.717 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: osd.2) 607 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:16.717 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:16.717 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: -10T04:33:14.427965+0000 osd.2 ( 2021-02-10T04:33:16.717 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: osd.2) 608 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:16.717 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:16.718 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: -10T04:33:14.427970+0000 osd.2 ( 2021-02-10T04:33:16.718 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: osd.2) 609 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:16.718 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:16.718 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: 02-10T04:33:15.293934+0000 mon.a 2021-02-10T04:33:16.718 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: (mon.0) 1154 : cluster [DBG] osdmap e378: 8 total, 8 up, 8 in 2021-02-10T04:33:16.718 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: 2021-02-10T04:33:16.718 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:16.718 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: -02-10T04:33:15.394778+0000 osd.4 (osd.4 2021-02-10T04:33:16.718 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: ) 423 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:16.718 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: cluster 2021-02-10T04:33:15.394796+0000 osd.4 (osd.4) 424 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:16.718 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: cluster 2021-02-10T04:33:16.718 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: 2021-02-10T04:33:15.394807+0000 osd.4 (osd.4) 2021-02-10T04:33:16.718 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: 425 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:16.719 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:33:16.719 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: 33:15.394812+0000 osd.4 (osd.4) 426 2021-02-10T04:33:16.719 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:16.719 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: cluster 2021-02-10T04:33:16 2021-02-10T04:33:16.719 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: .069055+0000 mgr.y (mgr.14140) 512 : cluster [DBG] 2021-02-10T04:33:16.719 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:16 smithi085 conmon[35961]: pgmap v717: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 159 active+clean; 303 MiB data, 938 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 3.7 MiB/s wr, 24 op/s 2021-02-10T04:33:16.834 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:33:16.836 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:33:16 smithi164 conmon[44544]: debug 2021-02-10T04:33:16.782+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:16.938 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:33:17.309 INFO:tasks.rados.rados.0.smithi085.stdout:1418: done (10 left) 2021-02-10T04:33:17.309 INFO:tasks.rados.rados.0.smithi085.stdout:1419: copy_from oid 40 from oid 13 current snap is 148 2021-02-10T04:33:17.309 INFO:tasks.rados.rados.0.smithi085.stdout:1416: finishing write tid 2 to smithi08575397-46 2021-02-10T04:33:17.309 INFO:tasks.rados.rados.0.smithi085.stdout:1416: finishing write tid 3 to smithi08575397-46 2021-02-10T04:33:17.309 INFO:tasks.rados.rados.0.smithi085.stdout:1417: expect (ObjNum 509 snap 148 seq_num 509) 2021-02-10T04:33:17.519 INFO:tasks.rados.rados.0.smithi085.stdout:1416: finishing write tid 4 to smithi08575397-46 2021-02-10T04:33:17.519 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 673 (ObjNum 510 snap 148 seq_num 510) dirty exists 2021-02-10T04:33:17.519 INFO:tasks.rados.rados.0.smithi085.stdout:1416: left oid 46 (ObjNum 510 snap 148 seq_num 510) 2021-02-10T04:33:17.519 INFO:tasks.rados.rados.0.smithi085.stdout:1416: done (10 left) 2021-02-10T04:33:17.519 INFO:tasks.rados.rados.0.smithi085.stdout:1417: done (9 left) 2021-02-10T04:33:17.519 INFO:tasks.rados.rados.0.smithi085.stdout:1420: delete oid 5 current snap is 148 2021-02-10T04:33:17.519 INFO:tasks.rados.rados.0.smithi085.stdout:1419: finishing copy_from to smithi08575397-40 2021-02-10T04:33:17.519 INFO:tasks.rados.rados.0.smithi085.stdout:1419: got expected ENOENT (src dne) 2021-02-10T04:33:17.519 INFO:tasks.rados.rados.0.smithi085.stdout:1419: finishing copy_from racing read to smithi08575397-40 2021-02-10T04:33:17.549 INFO:tasks.rados.rados.0.smithi085.stdout:1419: done (9 left) 2021-02-10T04:33:17.551 INFO:tasks.rados.rados.0.smithi085.stdout:1420: done (8 left) 2021-02-10T04:33:17.551 INFO:tasks.rados.rados.0.smithi085.stdout:1421: rollback oid 3 current snap is 148 2021-02-10T04:33:17.551 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 3 to 144 2021-02-10T04:33:17.551 INFO:tasks.rados.rados.0.smithi085.stdout:1422: snap_remove snap 141 2021-02-10T04:33:17.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: cluster 2021-02-10T04:33:15.424933+0000 osd.2 (osd.2) 610 : cluster 2021-02-10T04:33:17.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:17.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: cluster 2021-02-10T04:33:15.424938+0000 osd.2 (osd.2) 611 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:17.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: cluster 2021-02-10T04:33:15.424943+0000 osd.2 (osd. 2021-02-10T04:33:17.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: 2) 612 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:17.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: cluster 2021-02-10T04:33:15.424949+0000 2021-02-10T04:33:17.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: osd.2 (osd.2) 613 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:17.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: cluster 2021-02-10T04:33:17.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: 2021-02-10T04:33:15.424952+0000 2021-02-10T04:33:17.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: osd.2 (osd.2) 614 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:17.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: cluster 2021-02-10T04:33:15.424958 2021-02-10T04:33:17.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 615 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:17.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: cluster 2021-02-10T04:33:15. 2021-02-10T04:33:17.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: 424961+0000 osd.2 (osd.2) 616 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:17.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:33:17.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: T04:33:15.424964+0000 2021-02-10T04:33:17.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: osd.2 (osd.2) 617 : 2021-02-10T04:33:17.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:17.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:33:17.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: 02-10T04:33:16. 2021-02-10T04:33:17.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: 302242+0000 mon.a (mon.0 2021-02-10T04:33:17.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: ) 1155 : cluster [DBG] osdmap e379: 8 total, 8 up, 8 in 2021-02-10T04:33:17.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: 2021-02-10T04:33:17.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:33:17.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: :33:16.362906+0000 osd.4 (osd.4) 427 : cluster 2021-02-10T04:33:17.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:17.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:33:17.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: :33:16.362912+0000 osd.4 (osd.4) 428 : cluster [WRN] 2021-02-10T04:33:17.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:17.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: cluster 2021-02-10T04:33:16.362915+0000 2021-02-10T04:33:17.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: osd.4 (osd.4) 429 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:17.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: cluster 2021-02-10T04:33:16.362919+0000 2021-02-10T04:33:17.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[29288]: osd.4 (osd.4) 430 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:17.647 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[33677]: debug 2021-02-10T04:33:17.397+0000 7fbec19a1700 -1 osd.4 380 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:33:17.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[49106]: debug 2021-02-10T04:33:17.459+0000 7f77a3e94700 -1 osd.2 380 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:33:17.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[29319]: cluster 2021-02-10T04:33:15.424933+0000 osd.2 (osd.2) 610 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:17.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[29319]: cluster 2021-02-10T04:33:15.424938+0000 osd.2 (osd.2) 611 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:17.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[29319]: cluster 2021-02-10T04:33:15.424943+0000 osd.2 (osd.2) 612 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:17.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[29319]: cluster 2021-02-10T04:33:15.424949+0000 osd.2 (osd.2) 613 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:17.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[29319]: cluster 2021-02-10T04:33:15.424952+0000 osd.2 (osd.2) 614 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:17.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[29319]: cluster 2021-02-10T04:33:15.424958+0000 osd.2 (osd.2) 615 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:17.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[29319]: cluster 2021-02-10T04:33:15.424961+0000 osd.2 (osd.2) 616 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:17.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[29319]: cluster 2021-02-10T04:33:15.424964+0000 osd.2 (osd.2) 617 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:17.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[29319]: cluster 2021-02-10T04:33:16.302242+0000 mon.a (mon.0) 1155 : cluster [DBG] osdmap e379: 8 total, 8 up, 8 in 2021-02-10T04:33:17.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[29319]: cluster 2021-02-10T04:33:16.362906+0000 osd.4 (osd.4) 427 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:17.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[29319]: cluster 2021-02-10T04:33:17.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[29319]: 2021-02-10T04:33:16.362912+0000 osd.4 (osd.4) 428 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:17.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[29319]: cluster 2021-02-10T04:33:17.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:33:17.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[29319]: 10T04:33:16.362915+0000 osd.4 (osd.4) 429 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:17.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:33:17.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[29319]: -10T04:33:16.362919+0000 2021-02-10T04:33:17.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[29319]: osd.4 (osd.4) 430 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:17.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: cluster 2021-02-10T04:33:17.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: 2021-02-10T04:33:17.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: 2021 2021-02-10T04:33:17.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: - 2021-02-10T04:33:17.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: 02 2021-02-10T04:33:17.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: - 2021-02-10T04:33:17.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: 10 2021-02-10T04:33:17.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: T 2021-02-10T04:33:17.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: 04:33:15.424933+0000 osd.2 (osd.2) 2021-02-10T04:33:17.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: 610 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:17.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:33:17.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: 10T04:33:15.424938+0000 osd.2 (osd.2) 611 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:17.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: cluster 2021-02-10T04:33:15.424943+0000 2021-02-10T04:33:17.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: osd.2 (osd.2) 612 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:17.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:33:17.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: 04:33:15.424949+0000 osd.2 (osd.2) 613 : cluster [WRN] 2021-02-10T04:33:17.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:17.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:17.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: -02-10T04:33:15.424952+0000 osd.2 (osd.2) 614 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:17.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: cluster 2021-02-10T04:33: 2021-02-10T04:33:17.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: 15.424958+0000 osd.2 (osd.2) 615 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:17.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: cluster 2021-02-10T04:33:15.424961+0000 osd.2 (osd. 2021-02-10T04:33:17.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: 2) 616 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:17.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: cluster 2021-02-10T04:33:15. 2021-02-10T04:33:17.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: 424964+0000 osd.2 (osd.2) 617 : cluster [WRN] 2021-02-10T04:33:17.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:17.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: cluster 2021-02-10T04:33: 2021-02-10T04:33:17.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: 16.302242+0000 mon.a (mon.0) 1155 : cluster 2021-02-10T04:33:17.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: [DBG] osdmap e379: 8 total, 8 up, 8 in 2021-02-10T04:33:17.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:17.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: -10T04:33:16 2021-02-10T04:33:17.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: .362906+0000 osd.4 (osd.4 2021-02-10T04:33:17.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: ) 427 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:17.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:17.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: T04:33:16.362912+0000 osd.4 (osd.4) 428 : 2021-02-10T04:33:17.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:17.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:17.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: 02-10T04:33:16.362915+0000 osd.4 (osd.4) 2021-02-10T04:33:17.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: 429 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:17.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: cluster 2021-02-10T04:33 2021-02-10T04:33:17.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: :16.362919+0000 osd.4 (osd.4) 430 2021-02-10T04:33:17.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:17 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:17.941 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:18.123 INFO:tasks.rados.rados.0.smithi085.stdout:1422: done (9 left) 2021-02-10T04:33:18.123 INFO:tasks.rados.rados.0.smithi085.stdout:1423: read oid 16 snap -1 2021-02-10T04:33:18.123 INFO:tasks.rados.rados.0.smithi085.stdout:1423: expect (ObjNum 491 snap 141 seq_num 491) 2021-02-10T04:33:18.123 INFO:tasks.rados.rados.0.smithi085.stdout:1424: delete oid 31 current snap is 148 2021-02-10T04:33:18.123 INFO:tasks.rados.rados.0.smithi085.stdout:1421: finishing rollback tid 0 to smithi08575397-3 2021-02-10T04:33:18.124 INFO:tasks.rados.rados.0.smithi085.stdout:1421: finishing rollback tid 1 to smithi08575397-3 2021-02-10T04:33:18.124 INFO:tasks.rados.rados.0.smithi085.stdout:1421: finishing rollback tid 2 to smithi08575397-3 2021-02-10T04:33:18.124 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 635 (ObjNum 204 snap 44 seq_num 204) dirty exists 2021-02-10T04:33:18.133 INFO:tasks.rados.rados.0.smithi085.stdout:1423: expect (ObjNum 491 snap 141 seq_num 491) 2021-02-10T04:33:18.293 INFO:tasks.rados.rados.0.smithi085.stdout:1421: done (10 left) 2021-02-10T04:33:18.293 INFO:tasks.rados.rados.0.smithi085.stdout:1423: done (9 left) 2021-02-10T04:33:18.293 INFO:tasks.rados.rados.0.smithi085.stdout:1424: done (8 left) 2021-02-10T04:33:18.294 INFO:tasks.rados.rados.0.smithi085.stdout:1425: rollback oid 23 current snap is 148 2021-02-10T04:33:18.294 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 23 to 138 2021-02-10T04:33:18.294 INFO:tasks.rados.rados.0.smithi085.stdout:1426: copy_from oid 3 from oid 10 current snap is 148 2021-02-10T04:33:18.294 INFO:tasks.rados.rados.0.smithi085.stdout:1427: snap_remove snap 144 2021-02-10T04:33:18.307 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:33:17 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:33:18.308 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: cluster 2021-02-10T04:33:16.456929+0000 osd.2 (osd.2) 618 : cluster [WRN] 2021-02-10T04:33:18.309 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:18.309 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: cluster 2021-02-10T04:33:16.456937+0000 osd.2 (osd.2) 619 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:18.309 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:33:18.309 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: 10T04:33:16.456943+0000 osd.2 (osd.2) 620 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:18.309 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: cluster 2021-02-10T04:33:18.309 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: 2021-02-10T04:33:16. 2021-02-10T04:33:18.309 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: 456949+0000 osd.2 (osd.2) 621 : cluster 2021-02-10T04:33:18.309 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:18.309 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: cluster 2021-02-10T04:33:18.309 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: 2021-02-10T04:33:16.456967+0000 2021-02-10T04:33:18.309 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: osd.2 (osd.2) 622 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:18.310 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: cluster 2021-02-10T04:33:18.310 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: 2021-02-10T04:33:16.456974+0000 osd.2 (osd.2 2021-02-10T04:33:18.310 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: ) 623 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:18.310 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:33:18.310 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: 02-10T04:33:16.456980+0000 osd.2 ( 2021-02-10T04:33:18.310 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: osd.2) 624 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:18.310 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: cluster 2021-02-10T04:33:16.456986+0000 osd.2 ( 2021-02-10T04:33:18.310 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: osd.2) 625 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:18.310 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:33:18.310 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: 10T04:33:17.308305 2021-02-10T04:33:18.310 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1156 2021-02-10T04:33:18.311 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: : cluster [DBG] osdmap e380: 8 total, 8 up, 8 in 2021-02-10T04:33:18.311 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:33:18.311 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: 10T04:33:17.398682+0000 osd.4 (osd. 2021-02-10T04:33:18.311 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: 4) 431 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:18.311 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: 2021-02-10T04:33:18.311 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: cluster 2021-02-10T04:33 2021-02-10T04:33:18.311 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: :17.398691+0000 osd.4 (osd.4) 432 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:18.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[35961]: cluster 2021-02-10T04:33:16.456929 2021-02-10T04:33:18.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 618 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[35961]: cluster 2021-02-10T04:33:16.456937+0000 osd.2 (osd.2) 619 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[35961]: cluster 2021-02-10T04:33:16.456943+0000 osd.2 (osd.2) 620 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[35961]: cluster 2021-02-10T04:33:16.456949+0000 osd.2 (osd.2) 621 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[35961]: cluster 2021-02-10T04:33:16.456967+0000 osd.2 (osd.2) 622 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[35961]: cluster 2021-02-10T04:33:16.456974+0000 osd.2 (osd.2) 623 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[35961]: cluster 2021-02-10T04:33:16.456980+0000 osd.2 (osd.2) 624 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[35961]: cluster 2021-02-10T04:33:16.456986+0000 osd.2 (osd.2) 625 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[35961]: cluster 2021-02-10T04:33:17.308305+0000 mon.a (mon. 2021-02-10T04:33:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[35961]: 0) 1156 : cluster [DBG] osdmap e380: 8 total, 8 up, 8 in 2021-02-10T04:33:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[35961]: cluster 2021-02-10T04:33:17.398682+0000 osd.4 (osd.4) 431 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[35961]: cluster 2021-02-10T04:33:17.398691+0000 osd.4 (osd.4) 432 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[35961]: cluster 2021-02-10T04:33:17.398697+0000 osd.4 ( 2021-02-10T04:33:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[35961]: osd.4) 433 : cluster [WRN] 2021-02-10T04:33:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[35961]: 02-10T04:33:17.398704+0000 2021-02-10T04:33:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[35961]: osd.4 (osd.4) 434 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:33:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[35961]: 10T04:33:17.955088+0000 mgr.y ( 2021-02-10T04:33:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[35961]: mgr.14140) 513 2021-02-10T04:33:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[35961]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:33:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[35961]: cluster 2021-02-10T04:33:18 2021-02-10T04:33:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[35961]: .069422+0000 mgr.y (mgr.14140 2021-02-10T04:33:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[35961]: ) 514 : cluster [DBG] pgmap v720: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 159 active+clean; 303 MiB data, 941 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.7 MiB/s wr, 20 op/s 2021-02-10T04:33:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[35961]: -02-10T04 2021-02-10T04:33:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[35961]: :33:18.117040+0000 mon.a 2021-02-10T04:33:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[35961]: (mon.0) 1157 : cluster [DBG] osdmap e381: 8 total, 8 up, 8 in 2021-02-10T04:33:18.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: cluster 2021-02-10T04:33:18.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: 2021-02-10T04:33 2021-02-10T04:33:18.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: :16.456929+0000 osd.2 (osd.2) 618 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:18.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:33:18.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: 10T04:33:16.456937+0000 osd.2 (osd.2) 619 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:18.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: cluster 2021-02-10T04:33:18.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: 2021-02-10T04:33:16.456943+0000 2021-02-10T04:33:18.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: osd.2 (osd.2) 620 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:18.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: 2021-02-10T04:33:18.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:33:18.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: :33:16.456949+0000 osd.2 (osd.2 2021-02-10T04:33:18.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: ) 621 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:18.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:33:18.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: 33:16.456967+0000 osd.2 (osd.2) 622 : 2021-02-10T04:33:18.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:18.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:18.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: -02-10T04:33:16.456974+0000 osd.2 ( 2021-02-10T04:33:18.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: osd.2) 623 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:18.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: cluster 2021-02-10T04:33:18.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: 2021 2021-02-10T04:33:18.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: -02-10T04:33:16.456980+0000 2021-02-10T04:33:18.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: osd.2 (osd.2) 624 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:18.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:33:18.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: -10T04:33:16.456986+0000 osd.2 (osd.2 2021-02-10T04:33:18.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: ) 625 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:18.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: cluster 2021-02-10T04:33:18.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: 2021-02-10T04:33:17 2021-02-10T04:33:18.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: .308305+0000 mon.a (mon.0) 1156 : cluster [DBG] 2021-02-10T04:33:18.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: osdmap e380: 8 total, 8 up, 8 in 2021-02-10T04:33:18.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: cluster 2021-02-10T04:33:17 2021-02-10T04:33:18.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: .398682+0000 osd.4 (osd.4) 431 : cluster 2021-02-10T04:33:18.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:18.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:33:18.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: 33:17.398691+0000 osd.4 (osd.4) 2021-02-10T04:33:18.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: 432 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:18.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: cluster 2021-02-10T04:33: 2021-02-10T04:33:18.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: 17.398697+0000 osd.4 (osd.4) 433 : cluster 2021-02-10T04:33:18.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:18.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:33:18.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: 04:33:17.398704+0000 osd.4 (osd.4) 2021-02-10T04:33:18.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: 434 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:18.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:33:18.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: T04:33:17.955088+0000 mgr.y (mgr. 2021-02-10T04:33:18.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: 14140) 513 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:33:18.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:33:18.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: -10T04:33:18.069422+0000 mgr.y ( 2021-02-10T04:33:18.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: mgr.14140) 514 : cluster [DBG] pgmap v720: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 159 active+clean; 303 MiB data, 941 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.7 MiB/s wr, 20 op/s 2021-02-10T04:33:18.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:18.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: -02-10T04:33:18.117040+0000 2021-02-10T04:33:18.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: mon.a (mon.0) 1157 : cluster [DBG] 2021-02-10T04:33:18.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[29319]: osdmap e381: 8 total, 8 up, 8 in 2021-02-10T04:33:18.471 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:33:18.589 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:33:18.613 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: cluster 2021-02-10T04:33: 2021-02-10T04:33:18.613 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: 17.398697+0000 osd.4 (osd. 2021-02-10T04:33:18.613 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: 4) 433 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:18.613 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: 2021-02-10T04:33:18.613 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:33:18.614 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: -10T04:33:17.398704+0000 osd.4 (osd.4) 2021-02-10T04:33:18.614 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: 434 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:18.614 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: audit 2021-02-10T04:33:17.955088+0000 2021-02-10T04:33:18.614 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: mgr.y (mgr.14140) 513 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:33:18.614 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:33:18.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: -10T04:33:18 2021-02-10T04:33:18.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: .069422+0000 mgr.y (mgr.14140) 514 : cluster 2021-02-10T04:33:18.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: [DBG] pgmap v720: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 159 active+clean; 303 MiB data, 941 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.7 MiB/s wr, 20 op/s 2021-02-10T04:33:18.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: cluster 2021-02-10T04:33:18.117040 2021-02-10T04:33:18.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1157 : cluster [DBG] 2021-02-10T04:33:18.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[29288]: osdmap e381: 8 total, 8 up, 8 in 2021-02-10T04:33:18.616 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[33677]: debug 2021-02-10T04:33:18.399+0000 7fbec19a1700 -1 osd.4 381 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:33:18.616 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:18.715 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:33:18.777 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[49106]: debug 2021-02-10T04:33:18.449+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:18.777 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:18 smithi085 conmon[49106]: debug 2021-02-10T04:33:18.467+0000 7f77a3e94700 -1 osd.2 381 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:33:18.820 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:33:18.857 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:18.893 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[37263]: debug 2021-02-10T04:33:18.893 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:33:18 smithi164 conmon[37263]: 2021-02-10T04:33:18.678+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:19.034 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:33:19.130 INFO:tasks.rados.rados.0.smithi085.stdout:1427: done (10 left) 2021-02-10T04:33:19.131 INFO:tasks.rados.rados.0.smithi085.stdout:1428: read oid 42 snap -1 2021-02-10T04:33:19.131 INFO:tasks.rados.rados.0.smithi085.stdout:1428: expect (ObjNum 204 snap 44 seq_num 204) 2021-02-10T04:33:19.131 INFO:tasks.rados.rados.0.smithi085.stdout:1429: snap_create 2021-02-10T04:33:19.131 INFO:tasks.rados.rados.0.smithi085.stdout:1425: finishing rollback tid 0 to smithi08575397-23 2021-02-10T04:33:19.131 INFO:tasks.rados.rados.0.smithi085.stdout:1425: finishing rollback tid 1 to smithi08575397-23 2021-02-10T04:33:19.131 INFO:tasks.rados.rados.0.smithi085.stdout:1426: finishing copy_from to smithi08575397-3 2021-02-10T04:33:19.131 INFO:tasks.rados.rados.0.smithi085.stdout:1426: got expected ENOENT (src dne) 2021-02-10T04:33:19.131 INFO:tasks.rados.rados.0.smithi085.stdout:1425: finishing rollback tid 2 to smithi08575397-23 2021-02-10T04:33:19.132 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 623 (ObjNum 418 snap 109 seq_num 418) dirty exists 2021-02-10T04:33:19.132 INFO:tasks.rados.rados.0.smithi085.stdout:1426: finishing copy_from racing read to smithi08575397-3 2021-02-10T04:33:19.148 INFO:tasks.rados.rados.0.smithi085.stdout:1428: expect (ObjNum 204 snap 44 seq_num 204) 2021-02-10T04:33:19.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: cluster 2021-02-10T04:33:19.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: 2021-02-10T04:33:19.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: 2021 2021-02-10T04:33:19.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: - 2021-02-10T04:33:19.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: 02-10T04:33:17.461512 2021-02-10T04:33:19.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 626 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:19.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: cluster 2021-02-10T04:33: 2021-02-10T04:33:19.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: 17.461524+0000 osd.2 (osd.2) 627 : cluster [WRN] 2021-02-10T04:33:19.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:19.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:19.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: 02-10T04:33:17.461528+0000 osd.2 (osd.2) 628 : cluster [WRN] 2021-02-10T04:33:19.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:19.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:19.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: 02-10T04:33:17.461531+0000 osd.2 (osd.2) 629 : cluster 2021-02-10T04:33:19.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:19.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: cluster 2021-02-10T04:33:19.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: 2021-02-10T04:33:17.461536+0000 osd.2 (osd.2) 2021-02-10T04:33:19.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: 630 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:19.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: cluster 2021-02-10T04:33:17 2021-02-10T04:33:19.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: .461539+0000 osd.2 (osd.2) 631 : cluster 2021-02-10T04:33:19.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:19.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: cluster 2021-02-10T04:33:19.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: 2021-02-10T04:33:17.461542+0000 osd.2 2021-02-10T04:33:19.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: (osd.2) 632 : cluster [WRN] 2021-02-10T04:33:19.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:19.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:33:19.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: 10T04:33:17.461545 2021-02-10T04:33:19.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 633 2021-02-10T04:33:19.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:19.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:19.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: -02-10T04:33 2021-02-10T04:33:19.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: :18.303909+0000 mon.a (mon 2021-02-10T04:33:19.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: .0) 1158 : cluster [WRN] 2021-02-10T04:33:19.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: Health check update: 12 slow ops, oldest one blocked for 227 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:33:19.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:33:19.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: 04:33:18.399819+0000 2021-02-10T04:33:19.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: osd.4 (osd.4) 435 : 2021-02-10T04:33:19.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:19.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: cluster 2021-02-10T04:33:19.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: 2021-02-10T04:33:18. 2021-02-10T04:33:19.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: 399827+0000 osd.4 (osd.4) 2021-02-10T04:33:19.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: 436 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:19.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: cluster 2021-02-10T04:33:19.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: 2021-02-10T04:33 2021-02-10T04:33:19.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: :18.399831+0000 osd.4 (osd 2021-02-10T04:33:19.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: .4) 437 : cluster [WRN] 2021-02-10T04:33:19.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:19.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:19.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: T04:33:18.399835+0000 2021-02-10T04:33:19.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: osd.4 (osd.4) 438 2021-02-10T04:33:19.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:19.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: cluster 2021-02-10T04:33:19.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: 2021-02-10T04:33 2021-02-10T04:33:19.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: :19.129883+0000 mon.a ( 2021-02-10T04:33:19.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: mon.0) 1159 : cluster [DBG] 2021-02-10T04:33:19.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: osdmap e382: 8 total, 8 up, 8 in 2021-02-10T04:33:19.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:33:19.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: 10T04:33:19. 2021-02-10T04:33:19.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: 139650+0000 mon.a (mon.0) 2021-02-10T04:33:19.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[35961]: 1160 : audit [INF] from='client.? 172.21.15.85:0/557611128' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:33:19.617 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[49106]: debug 2021-02-10T04:33:19.447+0000 7f77a3e94700 -1 osd.2 382 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:33:19.617 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: cluster 2021-02-10T04:33:17.461512+0000 2021-02-10T04:33:19.617 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: osd.2 (osd.2) 626 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:19.617 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: cluster 2021-02-10T04:33:17.461524+0000 osd.2 (osd.2) 627 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:19.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: cluster 2021-02-10T04:33:17.461528+0000 osd.2 (osd.2) 628 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:19.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: cluster 2021-02-10T04:33:17.461531+0000 osd.2 (osd.2) 629 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:19.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: cluster 2021-02-10T04:33:17.461536+0000 osd.2 (osd.2) 630 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:19.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: cluster 2021-02-10T04:33:17.461539+0000 osd.2 (osd.2) 631 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:19.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: cluster 2021-02-10T04:33:17.461542+0000 osd.2 (osd.2) 632 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:19.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: cluster 2021-02-10T04:33 2021-02-10T04:33:19.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: :17.461545+0000 2021-02-10T04:33:19.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: osd.2 (osd.2) 633 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:19.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: 2021-02-10T04:33:19.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: cluster 2021-02-10T04:33:18 2021-02-10T04:33:19.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: .303909+0000 mon.a (mon.0) 1158 : cluster 2021-02-10T04:33:19.619 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: [WRN] Health check update: 12 slow ops, oldest one blocked for 227 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:33:19.619 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: cluster 2021-02-10T04:33:18 2021-02-10T04:33:19.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: .399819+0000 osd.4 (osd.4) 435 : cluster 2021-02-10T04:33:19.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:19.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:33:19.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: T04:33:18.399827+0000 osd.4 ( 2021-02-10T04:33:19.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: osd.4) 436 : cluster [WRN] 2021-02-10T04:33:19.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:19.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: cluster 2021-02-10T04:33:19.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:33:19.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: :33:18.399831+0000 osd.4 (osd.4) 2021-02-10T04:33:19.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: 437 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:19.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:19.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: -02-10T04:33:18.399835+0000 2021-02-10T04:33:19.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: osd.4 (osd.4) 438 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:19.621 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: 2021-02-10T04:33:19.621 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: cluster 2021-02-10T04:33:19.621 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: 2021-02-10T04:33:19.129883+0000 mon.a ( 2021-02-10T04:33:19.621 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: mon.0) 1159 : cluster [DBG] osdmap e382: 8 total, 8 up, 8 in 2021-02-10T04:33:19.621 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: audit 2021 2021-02-10T04:33:19.621 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: -02-10T04:33:19.139650+0000 mon.a 2021-02-10T04:33:19.621 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[29319]: (mon.0) 1160 : audit [INF] from='client.? 172.21.15.85:0/557611128' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:33:19.621 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:33:19 smithi085 conmon[45443]: debug 2021-02-10T04:33:19.610+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: cluster 2021-02-10T04:33:17.461512+0000 osd.2 (osd.2) 2021-02-10T04:33:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: 626 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: cluster 2021-02-10T04:33:17.461524+0000 osd.2 (osd.2) 627 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: cluster 2021-02-10T04:33:17.461528+0000 osd.2 (osd.2) 628 : 2021-02-10T04:33:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:33:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: 10T04:33:17.461531+0000 osd.2 2021-02-10T04:33:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: (osd.2) 629 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: cluster 2021-02-10T04:33:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: 2021-02-10T04:33: 2021-02-10T04:33:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: 17.461536+0000 osd.2 (osd.2) 630 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: 2021-02-10T04:33:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: cluster 2021-02-10T04:33:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: 2021-02-10T04:33:17 2021-02-10T04:33:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: .461539+0000 osd.2 (osd. 2021-02-10T04:33:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: 2) 631 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: cluster 2021-02-10T04:33:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: 2021-02-10T04:33:17.461542+0000 2021-02-10T04:33:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: osd.2 (osd.2) 632 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: cluster 2021-02-10T04:33:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: 2021-02-10T04:33: 2021-02-10T04:33:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: 17.461545+0000 osd.2 (osd.2) 633 : 2021-02-10T04:33:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:33:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: -10T04:33:18.303909+0000 2021-02-10T04:33:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: mon.a (mon.0) 1158 : cluster [WRN] 2021-02-10T04:33:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: Health check update: 12 slow ops, oldest one blocked for 227 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:33:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:33:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: 04:33:18.399819+0000 osd.4 (osd 2021-02-10T04:33:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: .4) 435 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: cluster 2021-02-10T04:33:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: 2021-02-10T04:33 2021-02-10T04:33:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: :18.399827+0000 osd.4 (osd.4) 436 : cluster 2021-02-10T04:33:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: cluster 2021-02-10T04:33:18.399831+0000 osd.4 2021-02-10T04:33:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: (osd.4) 437 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: cluster 2021-02-10T04:33:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: 2021-02-10T04:33 2021-02-10T04:33:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: :18.399835+0000 osd.4 (osd.4) 438 2021-02-10T04:33:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:33:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: -10T04:33:19.129883 2021-02-10T04:33:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1159 : cluster [DBG] 2021-02-10T04:33:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: osdmap e382: 8 total, 8 up, 8 in 2021-02-10T04:33:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:33:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: T04:33:19.139650+0000 mon.a ( 2021-02-10T04:33:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[29288]: mon.0) 1160 : audit [INF] from='client.? 172.21.15.85:0/557611128' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:33:19.648 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:19 smithi164 conmon[33677]: debug 2021-02-10T04:33:19.422+0000 7fbec19a1700 -1 osd.4 382 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:33:19.868 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:33:20.138 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is set 2021-02-10T04:33:20.139 INFO:tasks.rados.rados.0.smithi085.stdout:1425: done (11 left) 2021-02-10T04:33:20.139 INFO:tasks.rados.rados.0.smithi085.stdout:1426: done (10 left) 2021-02-10T04:33:20.139 INFO:tasks.rados.rados.0.smithi085.stdout:1428: done (9 left) 2021-02-10T04:33:20.139 INFO:tasks.rados.rados.0.smithi085.stdout:1429: done (8 left) 2021-02-10T04:33:20.139 INFO:tasks.rados.rados.0.smithi085.stdout:1430: write oid 38 current snap is 149 2021-02-10T04:33:20.139 INFO:tasks.rados.rados.0.smithi085.stdout:1430: seq_num 513 ranges {621991=798388,1936439=639509} 2021-02-10T04:33:20.141 INFO:tasks.rados.rados.0.smithi085.stdout:1430: writing smithi08575397-38 from 621991 to 1420379 tid 1 2021-02-10T04:33:20.149 INFO:tasks.rados.rados.0.smithi085.stdout:1430: writing smithi08575397-38 from 1936439 to 2575948 tid 2 2021-02-10T04:33:20.149 INFO:tasks.rados.rados.0.smithi085.stdout:1431: snap_create 2021-02-10T04:33:20.149 INFO:tasks.rados.rados.0.smithi085.stdout:1430: finishing write tid 1 to smithi08575397-38 2021-02-10T04:33:20.154 INFO:tasks.rados.rados.0.smithi085.stdout:1430: finishing write tid 2 to smithi08575397-38 2021-02-10T04:33:20.154 INFO:tasks.rados.rados.0.smithi085.stdout:1430: finishing write tid 3 to smithi08575397-38 2021-02-10T04:33:20.154 INFO:tasks.rados.rados.0.smithi085.stdout:1430: finishing write tid 4 to smithi08575397-38 2021-02-10T04:33:20.154 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 676 (ObjNum 513 snap 149 seq_num 513) dirty exists 2021-02-10T04:33:20.155 INFO:tasks.rados.rados.0.smithi085.stdout:1430: left oid 38 (ObjNum 513 snap 149 seq_num 513) 2021-02-10T04:33:20.182 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:33:20.215 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:20.394 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: cluster 2021-02-10T04:33:18.469393+0000 osd.2 (osd.2) 634 : 2021-02-10T04:33:20.394 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:20.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:33:20.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: 33:18.469401+0000 osd.2 (osd.2) 635 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:20.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:20.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: T04:33:18.469407+0000 osd.2 (osd. 2021-02-10T04:33:20.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: 2) 636 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:20.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: cluster 2021-02-10T04:33:20.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:33:20.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: 33:18.469412+0000 osd.2 (osd.2) 2021-02-10T04:33:20.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: 637 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:20.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: cluster 2021-02-10T04:33:20.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:33:20.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: 33:18.469418+0000 osd.2 (osd.2) 638 : cluster [WRN] 2021-02-10T04:33:20.396 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:20.396 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: cluster 2021-02-10T04:33:20.396 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: 2021-02-10T04:33:18.469422 2021-02-10T04:33:20.396 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 639 : cluster [WRN] 2021-02-10T04:33:20.396 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:20.396 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: cluster 2021-02-10T04:33:18.469426 2021-02-10T04:33:20.396 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 640 : cluster [WRN] 2021-02-10T04:33:20.396 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:20.396 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: cluster 2021-02-10T04:33:18.469429+0000 osd.2 (osd.2) 2021-02-10T04:33:20.396 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: 641 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:20.396 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: cluster 2021-02-10T04:33:18.979149+0000 osd.6 2021-02-10T04:33:20.396 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: (osd.6) 53 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:33:20.396 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:20.396 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: -02-10T04:33:18.979318+0000 2021-02-10T04:33:20.397 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: osd.6 (osd.6) 54 : cluster [DBG] 2021-02-10T04:33:20.397 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: purged_snaps scrub found 2 strays 2021-02-10T04:33:20.397 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: cluster 2021-02-10T04:33:19.423095+0000 2021-02-10T04:33:20.397 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: osd.4 (osd.4) 439 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:20.397 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:20.397 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: T04:33:19.423106+0000 osd.4 (osd.4) 440 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:20.397 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: 2021-02-10T04:33:20.397 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:20.397 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: -10T04:33:19.423113+0000 osd.4 ( 2021-02-10T04:33:20.397 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: osd.4) 441 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:20.397 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:20.397 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: 02-10T04:33:19.423119+0000 osd.4 ( 2021-02-10T04:33:20.397 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: osd.4) 442 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:20.398 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: cluster 2021-02-10T04:33: 2021-02-10T04:33:20.398 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: 20.069816+0000 mgr.y (mgr.14140) 515 : cluster [DBG] pgmap v723: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 157 active+clean; 304 MiB data, 940 MiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 832 KiB/s wr, 9 op/s 2021-02-10T04:33:20.398 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: cluster 2021-02-10T04:33:20.126024+0000 mon.a ( 2021-02-10T04:33:20.398 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: mon.0) 1161 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:33:20.398 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:33:20.398 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: :33:20.131340+0000 mon.a (mon.0 2021-02-10T04:33:20.398 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: ) 1162 : audit [INF] from='client.? 172.21.15.85:0/557611128' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:33:20.398 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:20.398 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: -10T04:33:20.131432+0000 mon.a ( 2021-02-10T04:33:20.398 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[35961]: mon.0) 1163 : cluster [DBG] osdmap e383: 8 total, 8 up, 8 in 2021-02-10T04:33:20.399 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: cluster 2021-02-10T04:33:18.469393+0000 osd.2 (osd.2) 634 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:20.399 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: cluster 2021-02-10T04:33:18.469401+0000 osd.2 (osd.2) 635 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:20.399 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: cluster 2021-02-10T04:33:18.469407+0000 osd.2 (osd.2) 636 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:20.399 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: cluster 2021-02-10T04:33:18.469412+0000 osd.2 (osd.2) 637 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:20.399 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: cluster 2021-02-10T04:33:18.469418+0000 osd.2 (osd.2) 638 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:20.399 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: cluster 2021-02-10T04:33:18.469422+0000 osd.2 (osd.2) 639 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:20.399 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: cluster 2021-02-10T04:33:18.469426+0000 osd.2 (osd.2) 640 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:20.399 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: cluster 2021-02-10T04:33:18.469429+0000 osd.2 (osd.2) 641 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:20.399 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:33:20.399 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: 10T04:33:18.979149+0000 osd.6 (osd.6) 53 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:33:20.399 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: cluster 2021-02-10T04:33:20.400 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:33:20.400 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: :33:18.979318+0000 osd.6 (osd.6) 2021-02-10T04:33:20.400 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: 54 : cluster [DBG] purged_snaps scrub found 2 strays 2021-02-10T04:33:20.400 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: cluster 2021-02-10T04:33:20.400 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: 2021-02-10T04:33: 2021-02-10T04:33:20.401 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: 19.423095+0000 osd.4 (osd.4) 439 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:20.401 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:33:20.401 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: 33:19.423106+0000 osd.4 (osd.4) 440 : cluster [WRN] 2021-02-10T04:33:20.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:20.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: cluster 2021-02-10T04:33:20.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: 2021-02-10T04:33:19.423113+0000 osd.4 (osd.4) 441 : cluster [WRN] 2021-02-10T04:33:20.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:20.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:33:20.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: 02-10T04:33:19.423119+0000 2021-02-10T04:33:20.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: osd.4 (osd.4) 442 : cluster [WRN] 2021-02-10T04:33:20.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:20.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: cluster 2021-02-10T04:33:20.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: 2021-02-10T04:33: 2021-02-10T04:33:20.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: 20.069816+0000 mgr.y (mgr.14140) 515 2021-02-10T04:33:20.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: : cluster [DBG] pgmap v723: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 157 active+clean; 304 MiB data, 940 MiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 832 KiB/s wr, 9 op/s 2021-02-10T04:33:20.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:33:20.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: 10T04:33:20.126024+0000 mon.a 2021-02-10T04:33:20.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: (mon.0) 1161 : cluster [WRN] 2021-02-10T04:33:20.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:33:20.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: audit 2021-02-10T04:33:20. 2021-02-10T04:33:20.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: 131340+0000 mon.a (mon.0) 2021-02-10T04:33:20.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: 1162 : audit [INF] from='client.? 172.21.15.85:0/557611128' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:33:20.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:33:20.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: T04:33:20.131432+0000 mon.a ( 2021-02-10T04:33:20.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[29319]: mon.0) 1163 : cluster [DBG] osdmap e383: 8 total, 8 up, 8 in 2021-02-10T04:33:20.517 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:33:20.517 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [ 2021-02-10T04:33:20.517 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.517 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215)", 2021-02-10T04:33:20.517 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:30:31.176628+0000", 2021-02-10T04:33:20.517 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 168.619226419, 2021-02-10T04:33:20.517 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 168.61924682099999, 2021-02-10T04:33:20.517 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:33:20.518 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:33:20.518 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:33:20.518 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:33:20.518 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:33:20.518 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 1693 2021-02-10T04:33:20.518 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.518 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:33:20.518 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.518 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:33:20.518 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.176628+0000", 2021-02-10T04:33:20.518 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:33:20.519 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.519 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.519 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:33:20.519 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.176628+0000", 2021-02-10T04:33:20.519 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4158e-05 2021-02-10T04:33:20.519 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.519 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.519 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:33:20.519 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.176673+0000", 2021-02-10T04:33:20.519 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.6469999999999998e-06 2021-02-10T04:33:20.519 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.519 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.520 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:33:20.520 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.176680+0000", 2021-02-10T04:33:20.520 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.3864e-05 2021-02-10T04:33:20.520 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.520 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.520 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:33:20.520 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.176694+0000", 2021-02-10T04:33:20.520 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.9800000000000003e-06 2021-02-10T04:33:20.520 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.520 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.520 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:33:20.520 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.176700+0000", 2021-02-10T04:33:20.521 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.8934999999999998e-05 2021-02-10T04:33:20.521 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.521 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.521 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:33:20.521 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.176749+0000", 2021-02-10T04:33:20.521 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00016330300000000001 2021-02-10T04:33:20.521 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.521 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.521 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:33:20.521 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.176912+0000", 2021-02-10T04:33:20.522 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00028388699999999997 2021-02-10T04:33:20.522 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:33:20.522 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:33:20.522 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:33:20.522 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.522 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.522 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215)", 2021-02-10T04:33:20.522 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:30:31.176747+0000", 2021-02-10T04:33:20.522 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 168.619108224, 2021-02-10T04:33:20.522 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 168.619185515, 2021-02-10T04:33:20.522 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:33:20.523 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "delayed", 2021-02-10T04:33:20.523 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:33:20.523 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:33:20.523 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:33:20.523 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 1694 2021-02-10T04:33:20.523 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.523 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:33:20.523 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.523 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:33:20.523 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.176747+0000", 2021-02-10T04:33:20.523 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:33:20.523 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.524 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.524 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:33:20.524 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.176747+0000", 2021-02-10T04:33:20.524 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.6130000000000001e-05 2021-02-10T04:33:20.524 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.524 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.524 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:33:20.524 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.176783+0000", 2021-02-10T04:33:20.524 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4599999999999996e-06 2021-02-10T04:33:20.524 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.524 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.524 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:33:20.525 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.176787+0000", 2021-02-10T04:33:20.525 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.9730000000000003e-06 2021-02-10T04:33:20.525 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.525 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.525 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:33:20.525 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.176796+0000", 2021-02-10T04:33:20.525 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1965e-05 2021-02-10T04:33:20.525 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.525 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.525 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:33:20.525 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.176808+0000", 2021-02-10T04:33:20.525 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00067544199999999995 2021-02-10T04:33:20.526 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.526 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.526 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:33:20.526 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.177484+0000", 2021-02-10T04:33:20.526 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.2240000000000002e-05 2021-02-10T04:33:20.526 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.526 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.526 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for blocked object", 2021-02-10T04:33:20.526 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:31.177526+0000", 2021-02-10T04:33:20.526 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00077921000000000002 2021-02-10T04:33:20.526 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:33:20.527 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:33:20.527 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:33:20.527 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.527 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.527 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176)", 2021-02-10T04:33:20.527 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:29:52.087674+0000", 2021-02-10T04:33:20.527 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 207.708181317, 2021-02-10T04:33:20.527 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 207.70830991599999, 2021-02-10T04:33:20.527 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:33:20.527 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:33:20.527 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:33:20.527 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:33:20.527 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:33:20.528 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 1164 2021-02-10T04:33:20.528 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.528 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:33:20.528 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.528 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:33:20.528 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:52.087674+0000", 2021-02-10T04:33:20.528 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:33:20.528 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.528 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.528 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:33:20.528 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:52.087674+0000", 2021-02-10T04:33:20.528 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.5439999999999999e-05 2021-02-10T04:33:20.529 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.529 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.529 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:33:20.529 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:52.087719+0000", 2021-02-10T04:33:20.529 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.7420000000000003e-06 2021-02-10T04:33:20.529 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.529 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.529 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:33:20.529 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:52.087728+0000", 2021-02-10T04:33:20.529 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.3535e-05 2021-02-10T04:33:20.529 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.529 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.530 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:33:20.530 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:52.087741+0000", 2021-02-10T04:33:20.530 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.4950000000000002e-06 2021-02-10T04:33:20.530 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.530 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.530 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:33:20.530 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:52.087749+0000", 2021-02-10T04:33:20.530 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.4869999999999994e-05 2021-02-10T04:33:20.530 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.530 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.531 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:33:20.531 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:52.087814+0000", 2021-02-10T04:33:20.531 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000129152 2021-02-10T04:33:20.531 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.531 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.531 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:33:20.531 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:52.087943+0000", 2021-02-10T04:33:20.531 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00026923399999999998 2021-02-10T04:33:20.531 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:33:20.531 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:33:20.531 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:33:20.531 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.532 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.532 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176)", 2021-02-10T04:33:20.532 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:29:52.087796+0000", 2021-02-10T04:33:20.532 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 207.708059308, 2021-02-10T04:33:20.532 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 207.70822650400001, 2021-02-10T04:33:20.532 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:33:20.532 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "delayed", 2021-02-10T04:33:20.532 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:33:20.532 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:33:20.532 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:33:20.532 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 1165 2021-02-10T04:33:20.532 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.533 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:33:20.533 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.533 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:33:20.533 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:52.087796+0000", 2021-02-10T04:33:20.533 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:33:20.533 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.533 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.533 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:33:20.533 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:52.087796+0000", 2021-02-10T04:33:20.533 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.7659000000000002e-05 2021-02-10T04:33:20.533 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.533 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.534 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:33:20.534 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:52.087813+0000", 2021-02-10T04:33:20.534 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.9729999999999999e-06 2021-02-10T04:33:20.534 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.534 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.534 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:33:20.534 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:52.087817+0000", 2021-02-10T04:33:20.534 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.9250000000000001e-06 2021-02-10T04:33:20.534 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.534 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.534 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:33:20.534 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:52.087826+0000", 2021-02-10T04:33:20.535 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.3699999999999999e-06 2021-02-10T04:33:20.535 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.535 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.535 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:33:20.535 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:52.087830+0000", 2021-02-10T04:33:20.535 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00069767199999999998 2021-02-10T04:33:20.535 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.535 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.535 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:33:20.535 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:52.088527+0000", 2021-02-10T04:33:20.535 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4317000000000003e-05 2021-02-10T04:33:20.536 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.536 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.536 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for blocked object", 2021-02-10T04:33:20.536 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:29:52.088571+0000", 2021-02-10T04:33:20.536 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00077591599999999998 2021-02-10T04:33:20.536 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:33:20.536 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:33:20.536 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:33:20.536 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.536 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.536 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230)", 2021-02-10T04:33:20.536 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:30:46.071690+0000", 2021-02-10T04:33:20.537 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 153.72416440000001, 2021-02-10T04:33:20.537 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 153.72437470700001, 2021-02-10T04:33:20.537 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:33:20.537 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:33:20.537 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:33:20.537 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:33:20.537 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:33:20.537 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 1816 2021-02-10T04:33:20.537 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.537 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:33:20.537 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.537 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:33:20.537 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.071690+0000", 2021-02-10T04:33:20.538 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:33:20.538 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.538 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.538 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:33:20.538 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.071690+0000", 2021-02-10T04:33:20.538 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.6808999999999999e-05 2021-02-10T04:33:20.538 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.538 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.538 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:33:20.538 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.071737+0000", 2021-02-10T04:33:20.538 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.0420000000000008e-06 2021-02-10T04:33:20.538 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.539 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.539 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:33:20.539 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.071746+0000", 2021-02-10T04:33:20.539 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.8348000000000001e-05 2021-02-10T04:33:20.539 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.539 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.539 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:33:20.539 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.071765+0000", 2021-02-10T04:33:20.539 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0186999999999999e-05 2021-02-10T04:33:20.539 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.539 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.539 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:33:20.540 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.071775+0000", 2021-02-10T04:33:20.540 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000167374 2021-02-10T04:33:20.540 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.540 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.540 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:33:20.540 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.071942+0000", 2021-02-10T04:33:20.540 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00011369799999999999 2021-02-10T04:33:20.540 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.540 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.540 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:33:20.540 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.072056+0000", 2021-02-10T04:33:20.540 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00036545799999999998 2021-02-10T04:33:20.541 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:33:20.541 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:33:20.541 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:33:20.541 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.541 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.541 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230)", 2021-02-10T04:33:20.541 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:30:46.071824+0000", 2021-02-10T04:33:20.541 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 153.72403127300001, 2021-02-10T04:33:20.541 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 153.72431455099999, 2021-02-10T04:33:20.541 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:33:20.541 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "delayed", 2021-02-10T04:33:20.541 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:33:20.541 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:33:20.542 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:33:20.542 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 1817 2021-02-10T04:33:20.542 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.542 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:33:20.542 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.542 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:33:20.542 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.071824+0000", 2021-02-10T04:33:20.542 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:33:20.542 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.542 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.542 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:33:20.543 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.071824+0000", 2021-02-10T04:33:20.543 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.1028999999999997e-05 2021-02-10T04:33:20.543 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.543 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.543 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:33:20.543 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.071855+0000", 2021-02-10T04:33:20.543 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.8099999999999992e-06 2021-02-10T04:33:20.543 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.543 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.543 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:33:20.543 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.071864+0000", 2021-02-10T04:33:20.543 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.4374e-05 2021-02-10T04:33:20.544 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.544 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.544 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:33:20.544 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.071879+0000", 2021-02-10T04:33:20.544 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.1059999999999996e-06 2021-02-10T04:33:20.544 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.544 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.544 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:33:20.544 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.071885+0000", 2021-02-10T04:33:20.544 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00066733999999999999 2021-02-10T04:33:20.544 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.544 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.544 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:33:20.545 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.072552+0000", 2021-02-10T04:33:20.545 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.8199000000000003e-05 2021-02-10T04:33:20.545 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.545 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.545 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for blocked object", 2021-02-10T04:33:20.545 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.072610+0000", 2021-02-10T04:33:20.545 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00078685799999999998 2021-02-10T04:33:20.545 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:33:20.545 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:33:20.545 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:33:20.545 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.545 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.546 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230)", 2021-02-10T04:33:20.546 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:30:46.083500+0000", 2021-02-10T04:33:20.546 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 153.712354718, 2021-02-10T04:33:20.546 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 153.71271498999999, 2021-02-10T04:33:20.546 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:33:20.546 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:33:20.546 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:33:20.546 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:33:20.546 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:33:20.546 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 1822 2021-02-10T04:33:20.546 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.546 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:33:20.547 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.547 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:33:20.547 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.083500+0000", 2021-02-10T04:33:20.547 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:33:20.547 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.547 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.547 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:33:20.547 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.083500+0000", 2021-02-10T04:33:20.547 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.8988e-05 2021-02-10T04:33:20.547 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.547 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.548 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:33:20.548 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.083539+0000", 2021-02-10T04:33:20.548 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.3180000000000003e-06 2021-02-10T04:33:20.548 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.548 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.548 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:33:20.548 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.083545+0000", 2021-02-10T04:33:20.548 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.309e-05 2021-02-10T04:33:20.548 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.548 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.548 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:33:20.549 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.083559+0000", 2021-02-10T04:33:20.549 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.8680000000000001e-06 2021-02-10T04:33:20.549 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.549 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.549 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:33:20.549 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.083565+0000", 2021-02-10T04:33:20.549 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000339382 2021-02-10T04:33:20.549 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.549 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.549 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:33:20.549 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.083905+0000", 2021-02-10T04:33:20.549 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000105299 2021-02-10T04:33:20.550 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.550 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.550 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:33:20.550 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.084010+0000", 2021-02-10T04:33:20.550 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00050994500000000004 2021-02-10T04:33:20.550 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:33:20.550 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:33:20.550 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:33:20.550 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.550 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.550 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230)", 2021-02-10T04:33:20.551 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:30:46.083641+0000", 2021-02-10T04:33:20.551 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 153.71221399000001, 2021-02-10T04:33:20.551 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 153.712642913, 2021-02-10T04:33:20.551 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:33:20.551 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "delayed", 2021-02-10T04:33:20.551 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:33:20.551 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.14817", 2021-02-10T04:33:20.551 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:33:20.551 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 1823 2021-02-10T04:33:20.551 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.551 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:33:20.551 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.552 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:33:20.552 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.083641+0000", 2021-02-10T04:33:20.552 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:33:20.552 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.552 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.552 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:33:20.552 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.083641+0000", 2021-02-10T04:33:20.552 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.9726e-05 2021-02-10T04:33:20.552 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.552 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.552 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:33:20.552 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.083671+0000", 2021-02-10T04:33:20.552 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.7209999999999999e-06 2021-02-10T04:33:20.553 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.553 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.553 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:33:20.553 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.083676+0000", 2021-02-10T04:33:20.553 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0818e-05 2021-02-10T04:33:20.553 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.553 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.553 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:33:20.553 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.083687+0000", 2021-02-10T04:33:20.553 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.2680000000000002e-06 2021-02-10T04:33:20.553 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.553 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.554 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:33:20.554 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.083692+0000", 2021-02-10T04:33:20.554 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00068197400000000001 2021-02-10T04:33:20.554 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.554 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.554 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:33:20.554 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.084374+0000", 2021-02-10T04:33:20.554 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.2744999999999997e-05 2021-02-10T04:33:20.554 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:33:20.554 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:33:20.554 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for blocked object", 2021-02-10T04:33:20.555 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:30:46.084427+0000", 2021-02-10T04:33:20.555 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00078625199999999998 2021-02-10T04:33:20.555 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:33:20.555 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:33:20.555 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:33:20.555 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:33:20.555 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:33:20.555 INFO:teuthology.orchestra.run.smithi085.stdout: "complaint_time": 30, 2021-02-10T04:33:20.555 INFO:teuthology.orchestra.run.smithi085.stdout: "num_blocked_ops": 8 2021-02-10T04:33:20.555 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:33:20.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[33677]: debug 2021-02-10T04:33:20.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[33677]: 2021-02-10T04:33:20.449+0000 7fbec19a1700 -1 osd.4 383 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:33:20.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[33677]: 2021-02-10T04:33:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: cluster 2021-02-10T04:33:18.469393 2021-02-10T04:33:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 634 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: cluster 2021-02-10T04:33:18.469401+0000 osd.2 (osd.2) 635 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: 2021-02-10T04:33:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: cluster 2021-02-10T04:33:18.469407+0000 osd.2 (osd.2) 636 : cluster [WRN] 2021-02-10T04:33:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: cluster 2021-02-10T04:33:18.469412+0000 2021-02-10T04:33:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: osd.2 (osd.2) 637 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: cluster 2021-02-10T04:33:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: 2021-02-10T04:33:18.469418+0000 osd.2 (osd.2) 638 : cluster [WRN] 2021-02-10T04:33:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:33:20.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: 10T04:33:18.469422+0000 osd.2 (osd.2) 639 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:20.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:33:20.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: 33:18.469426+0000 osd.2 (osd.2) 2021-02-10T04:33:20.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: 640 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:20.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: cluster 2021-02-10T04:33:20.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: 2021-02-10T04:33: 2021-02-10T04:33:20.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: 18.469429+0000 osd.2 (osd.2) 641 2021-02-10T04:33:20.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:20.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:33:20.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: 02-10T04:33:18.979149 2021-02-10T04:33:20.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: +0000 osd.6 (osd.6) 53 : cluster 2021-02-10T04:33:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: [DBG] purged_snaps scrub starts 2021-02-10T04:33:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:33:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: 10T04:33:18.979318+0000 osd.6 (osd.6 2021-02-10T04:33:20.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: ) 54 : cluster [DBG] purged_snaps scrub found 2 strays 2021-02-10T04:33:20.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: cluster 2021-02-10T04:33:19. 2021-02-10T04:33:20.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: 423095+0000 osd.4 (osd.4) 439 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:20.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: cluster 2021-02-10T04:33:19 2021-02-10T04:33:20.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: .423106+0000 osd.4 (osd.4) 440 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:20.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:33:20.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: T04:33:19.423113+0000 osd.4 (osd.4) 441 : cluster 2021-02-10T04:33:20.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:20.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:33:20.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: -02-10T04:33:19.423119+0000 osd.4 (osd.4) 442 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:20.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:33:20.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: 04:33:20.069816+0000 mgr.y (mgr. 2021-02-10T04:33:20.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: 14140) 515 : cluster [DBG] pgmap v723: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 157 active+clean; 304 MiB data, 940 MiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 832 KiB/s wr, 9 op/s 2021-02-10T04:33:20.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: 2021-02-10T04:33:20.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:33:20.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: 04:33:20.126024+0000 mon.a ( 2021-02-10T04:33:20.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: mon.0) 1161 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:33:20.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: audit 2021-02-10T04:33:20.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: 2021-02-10T04:33: 2021-02-10T04:33:20.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: 20.131340+0000 mon.a (mon. 2021-02-10T04:33:20.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: 0) 1162 : audit [INF] from='client.? 172.21.15.85:0/557611128' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:33:20.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:33:20.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: 10T04:33:20.131432+0000 mon.a (mon.0) 1163 2021-02-10T04:33:20.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:20 smithi164 conmon[29288]: : cluster [DBG] osdmap e383: 8 total, 8 up, 8 in 2021-02-10T04:33:20.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:20 smithi085 conmon[49106]: debug 2021-02-10T04:33:20.404+0000 7f77a3e94700 -1 osd.2 383 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:33:21.136 INFO:tasks.rados.rados.0.smithi085.stdout:1430: done (9 left) 2021-02-10T04:33:21.137 INFO:tasks.rados.rados.0.smithi085.stdout:1431: done (8 left) 2021-02-10T04:33:21.137 INFO:tasks.rados.rados.0.smithi085.stdout:1432: read oid 7 snap -1 2021-02-10T04:33:21.137 INFO:tasks.rados.rados.0.smithi085.stdout:1432: expect (ObjNum 503 snap 147 seq_num 503) 2021-02-10T04:33:21.137 INFO:tasks.rados.rados.0.smithi085.stdout:1433: read oid 9 snap -1 2021-02-10T04:33:21.137 INFO:tasks.rados.rados.0.smithi085.stdout:1433: expect (ObjNum 229 snap 51 seq_num 229) 2021-02-10T04:33:21.137 INFO:tasks.rados.rados.0.smithi085.stdout:1434: snap_create 2021-02-10T04:33:21.145 INFO:tasks.rados.rados.0.smithi085.stdout:1432: expect (ObjNum 503 snap 147 seq_num 503) 2021-02-10T04:33:21.355 INFO:tasks.rados.rados.0.smithi085.stdout:1433: expect (ObjNum 229 snap 51 seq_num 229) 2021-02-10T04:33:21.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[33677]: debug 2021-02-10T04:33:21.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[33677]: 2021-02-10T04:33:21.487+0000 7fbec19a1700 -1 osd.4 384 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:33:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: cluster 2021-02-10T04:33:19.448089+0000 osd.2 ( 2021-02-10T04:33:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: osd.2) 642 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: cluster 2021-02-10T04:33:19.448099+0000 osd.2 (osd.2) 643 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: cluster 2021-02-10T04:33:19.448107+0000 osd.2 (osd.2) 644 : 2021-02-10T04:33:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: cluster 2021-02-10T04:33: 2021-02-10T04:33:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: 19.448114+0000 osd.2 (osd.2) 645 : cluster 2021-02-10T04:33:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: cluster 2021-02-10T04:33:19.448120 2021-02-10T04:33:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 646 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: 2021-02-10T04:33:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: cluster 2021-02-10T04:33:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: 2021-02-10T04:33:19.448126+0000 osd.2 (osd.2) 2021-02-10T04:33:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: 647 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:33:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: :33:19.448131+0000 osd.2 (osd.2) 648 2021-02-10T04:33:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: cluster 2021-02-10T04:33 2021-02-10T04:33:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: :19.448137+0000 osd.2 (osd.2) 649 : cluster 2021-02-10T04:33:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: cluster 2021-02-10T04:33:20.449838 2021-02-10T04:33:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 443 : cluster [WRN] 2021-02-10T04:33:21.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:21.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: cluster 2021-02-10T04:33:21.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: 2021-02-10T04:33:20.449847+0000 osd.4 (osd 2021-02-10T04:33:21.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: .4) 444 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:21.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: cluster 2021-02-10T04:33:20.449853 2021-02-10T04:33:21.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 445 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:21.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:33:21.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: 33:20.449863+0000 osd.4 (osd.4) 446 2021-02-10T04:33:21.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:21.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:33:21.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: 02-10T04:33:21.136199 2021-02-10T04:33:21.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1164 : 2021-02-10T04:33:21.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:21 smithi164 conmon[29288]: cluster [DBG] osdmap e384: 8 total, 8 up, 8 in 2021-02-10T04:33:21.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[49106]: debug 2021-02-10T04:33:21.360+0000 7f77a3e94700 -1 osd.2 384 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:33:21.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: cluster 2021-02-10T04:33:21.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:33:21.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: 33:19.448089+0000 osd.2 (osd.2) 642 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:21.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:33:21.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: 04:33:19.448099+0000 osd.2 (osd.2) 2021-02-10T04:33:21.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: 643 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:21.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:33:21.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: T04:33:19.448107+0000 osd.2 ( 2021-02-10T04:33:21.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: osd.2) 644 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:21.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:33:21.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: 02-10T04:33:19.448114+0000 osd.2 2021-02-10T04:33:21.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: (osd.2) 645 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:21.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:33:21.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: 10T04:33:19.448120+0000 osd.2 (osd 2021-02-10T04:33:21.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: .2) 646 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:21.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:21.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: -02-10T04:33: 2021-02-10T04:33:21.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: 19.448126+0000 osd.2 ( 2021-02-10T04:33:21.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: osd.2) 647 : cluster 2021-02-10T04:33:21.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:21.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:33:21.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: -10T04:33:19.448131+0000 osd.2 2021-02-10T04:33:21.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: (osd.2) 648 : cluster [WRN] 2021-02-10T04:33:21.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:21.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:21.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: -02-10T04:33:19.448137+0000 2021-02-10T04:33:21.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: osd.2 (osd.2) 649 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:21.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: 2021-02-10T04:33:21.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: cluster 2021-02-10T04:33:20 2021-02-10T04:33:21.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: .449838+0000 osd.4 (osd.4) 443 : cluster 2021-02-10T04:33:21.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:21.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: cluster 2021-02-10T04:33: 2021-02-10T04:33:21.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: 20.449847+0000 osd.4 (osd.4) 444 : cluster 2021-02-10T04:33:21.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:21.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:33:21.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: 33:20.449853+0000 osd.4 (osd.4) 2021-02-10T04:33:21.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: 445 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:21.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:33:21.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: 02-10T04:33:20.449863 2021-02-10T04:33:21.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 446 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:21.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: 2021-02-10T04:33:21.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:33:21.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: 02-10T04:33:21.136199+0000 mon.a (mon. 2021-02-10T04:33:21.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[29319]: 0) 1164 : cluster [DBG] osdmap e384: 8 total, 8 up, 8 in 2021-02-10T04:33:21.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: cluster 2021-02-10T04:33:21.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: 2021-02-10T04:33:21.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: 2021 2021-02-10T04:33:21.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: - 2021-02-10T04:33:21.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: 02 2021-02-10T04:33:21.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: - 2021-02-10T04:33:21.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: 10 2021-02-10T04:33:21.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: T 2021-02-10T04:33:21.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: 04 2021-02-10T04:33:21.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: : 2021-02-10T04:33:21.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: 33 2021-02-10T04:33:21.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: : 2021-02-10T04:33:21.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: 19 2021-02-10T04:33:21.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: . 2021-02-10T04:33:21.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: 448089 2021-02-10T04:33:21.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: +0000 2021-02-10T04:33:21.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: 2021-02-10T04:33:21.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: osd.2 2021-02-10T04:33:21.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: (osd.2) 642 : cluster [WRN] 2021-02-10T04:33:21.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:21.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: cluster 2021-02-10T04:33:21.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: 2021-02-10T04:33:19.448099+0000 osd.2 (osd.2) 2021-02-10T04:33:21.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: 643 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:21.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:21.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: -02-10T04:33:19.448107+0000 2021-02-10T04:33:21.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: osd.2 (osd.2) 644 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:21.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: 2021-02-10T04:33:21.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: cluster 2021-02-10T04:33:21.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: 2021-02-10T04:33:19.448114+0000 osd.2 ( 2021-02-10T04:33:21.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: osd.2) 645 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:21.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:33:21.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: 10T04:33:19.448120+0000 osd.2 (osd. 2021-02-10T04:33:21.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: 2) 646 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:21.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:33:21.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: 04:33:19.448126+0000 osd.2 ( 2021-02-10T04:33:21.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: osd.2) 647 : cluster [WRN] 2021-02-10T04:33:21.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:21.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: cluster 2021-02-10T04:33:21.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: 2021-02-10T04:33 2021-02-10T04:33:21.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: :19.448131+0000 osd.2 (osd.2) 2021-02-10T04:33:21.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: 648 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:21.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:21.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: T04:33:19.448137+0000 osd.2 (osd. 2021-02-10T04:33:21.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: 2) 649 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:21.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:21.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: -02-10T04:33:20.449838+0000 osd.4 2021-02-10T04:33:21.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: (osd.4) 443 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:21.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:21.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: -02-10T04:33:20.449847+0000 2021-02-10T04:33:21.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: osd.4 (osd.4) 444 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:21.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: 2021-02-10T04:33:21.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: cluster 2021-02-10T04:33:21.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: 2021-02-10T04:33:20.449853+0000 osd.4 (osd.4 2021-02-10T04:33:21.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: ) 445 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:21.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:33:21.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: 10T04:33:20.449863+0000 osd.4 (osd 2021-02-10T04:33:21.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: .4) 446 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:21.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:21.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: 02-10T04:33:21.136199+0000 mon.a 2021-02-10T04:33:21.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:21 smithi085 conmon[35961]: (mon.0) 1164 : cluster [DBG] osdmap e384: 8 total, 8 up, 8 in 2021-02-10T04:33:22.137 INFO:tasks.rados.rados.0.smithi085.stdout:1432: done (10 left) 2021-02-10T04:33:22.137 INFO:tasks.rados.rados.0.smithi085.stdout:1433: done (9 left) 2021-02-10T04:33:22.137 INFO:tasks.rados.rados.0.smithi085.stdout:1434: done (8 left) 2021-02-10T04:33:22.137 INFO:tasks.rados.rados.0.smithi085.stdout:1435: snap_remove snap 150 2021-02-10T04:33:22.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[33677]: debug 2021-02-10T04:33:22.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[33677]: 2021-02-10T04:33:22.459+0000 7fbec19a1700 -1 osd.4 385 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:33:22.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[33677]: 2021-02-10T04:33:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: cluster 2021-02-10T04:33:20.404665+0000 osd.2 (osd.2) 2021-02-10T04:33:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: 650 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: cluster 2021-02-10T04:33:20.404676+0000 osd.2 (osd.2) 651 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: cluster 2021-02-10T04:33:20.404682+0000 osd.2 (osd. 2021-02-10T04:33:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: 2) 652 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: cluster 2021-02-10T04:33:20.404697+0000 osd.2 (osd.2) 2021-02-10T04:33:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: 653 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: cluster 2021-02-10T04:33:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: 2021-02-10T04:33:20.404703+0000 osd.2 (osd.2 2021-02-10T04:33:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: ) 654 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: cluster 2021-02-10T04:33:20.404708+0000 osd.2 (osd 2021-02-10T04:33:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: .2) 655 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:33:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: 33:20.404713+0000 osd.2 (osd.2) 656 2021-02-10T04:33:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: cluster 2021-02-10T04:33:20. 2021-02-10T04:33:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: 404718+0000 osd.2 (osd.2) 657 2021-02-10T04:33:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:33:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: 04:33:21.360708+0000 osd.2 (osd.2) 2021-02-10T04:33:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: 658 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:33:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: -10T04:33:21.360722+0000 2021-02-10T04:33:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: osd.2 (osd.2) 659 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: 2021-02-10T04:33:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: cluster 2021-02-10T04:33:21.360729+0000 2021-02-10T04:33:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: osd.2 (osd.2) 660 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:33:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: 02-10T04:33:21.360736+0000 osd.2 (osd.2) 661 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:33:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: T04:33:21.360742+0000 osd.2 (osd. 2021-02-10T04:33:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: 2) 662 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: cluster 2021-02-10T04:33:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: 2021-02-10T04:33:21 2021-02-10T04:33:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: .360748+0000 osd.2 (osd.2) 663 : cluster 2021-02-10T04:33:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:33:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: :33:21.360755 2021-02-10T04:33:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 664 : cluster 2021-02-10T04:33:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:33:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: T04:33:21.360761+0000 osd.2 2021-02-10T04:33:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: (osd.2) 665 : cluster 2021-02-10T04:33:22.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:22.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: cluster 2021-02-10T04:33:21.488113 2021-02-10T04:33:22.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 447 2021-02-10T04:33:22.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:22.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:33:22.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: T04:33:21.488119+0000 osd.4 (osd 2021-02-10T04:33:22.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: .4) 448 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:22.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: cluster 2021-02-10T04:33:22.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:33:22.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: :33:21.488125+0000 osd.4 (osd. 2021-02-10T04:33:22.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: 4) 449 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:22.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:33:22.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: -02-10T04:33:21.488131 2021-02-10T04:33:22.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 450 : cluster [WRN] 2021-02-10T04:33:22.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:22.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: cluster 2021-02-10T04:33:22.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: 2021-02-10T04:33:22 2021-02-10T04:33:22.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: .070306+0000 mgr.y (mgr. 2021-02-10T04:33:22.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: 14140) 516 : cluster [DBG] 2021-02-10T04:33:22.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: pgmap v726: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 306 MiB data, 933 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 1.4 MiB/s wr, 19 op/s 2021-02-10T04:33:22.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:33:22.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: -02-10T04:33 2021-02-10T04:33:22.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: :22.135099+0000 mon.a (mon 2021-02-10T04:33:22.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: .0) 1165 : cluster [DBG] 2021-02-10T04:33:22.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:22 smithi164 conmon[29288]: osdmap e385: 8 total, 8 up, 8 in 2021-02-10T04:33:22.666 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:33:22.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[49106]: debug 2021-02-10T04:33:22.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[49106]: 2021-02-10T04:33:22.354+0000 7f77a3e94700 -1 osd.2 385 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:33:22.691 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[52836]: debug 2021-02-10T04:33:22.494+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:22.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: cluster 2021-02-10T04:33:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: 2021- 2021-02-10T04:33:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: 02-10T04:33:20.404665+0000 osd.2 (osd.2) 650 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: 2021-02-10T04:33:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:33:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: 02-10T04:33:20.404676+0000 osd.2 (osd. 2021-02-10T04:33:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: 2) 651 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:33:22.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: 04:33:20.404682+0000 osd.2 (osd 2021-02-10T04:33:22.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: .2) 652 : 2021-02-10T04:33:22.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:22.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: cluster 2021-02-10T04:33: 2021-02-10T04:33:22.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: 20.404697+0000 osd.2 (osd.2) 653 2021-02-10T04:33:22.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:22.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:33:22.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: :33:20.404703+0000 osd.2 (osd.2 2021-02-10T04:33:22.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: ) 654 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:22.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:33:22.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: 10T04:33:20.404708+0000 osd.2 ( 2021-02-10T04:33:22.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: osd.2) 655 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:22.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:33:22.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: 02-10T04:33:20.404713+0000 2021-02-10T04:33:22.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: osd.2 (osd.2) 656 : cluster [WRN] 2021-02-10T04:33:22.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:22.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: cluster 2021-02-10T04:33:22.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: 2021-02-10T04:33:20.404718+0000 osd.2 2021-02-10T04:33:22.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: (osd.2) 657 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:22.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: 2021-02-10T04:33:22.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:22.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: -02-10T04:33:21.360708+0000 osd.2 (osd 2021-02-10T04:33:22.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: .2) 658 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:22.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:33:22.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: 10T04:33:21.360722+0000 osd.2 2021-02-10T04:33:22.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: (osd.2) 659 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:22.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:22.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: -02-10T04:33:21.360729+0000 2021-02-10T04:33:22.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: osd.2 (osd.2) 660 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:22.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: 2021-02-10T04:33:22.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: cluster 2021-02-10T04:33:22.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: 2021-02-10T04:33:21.360736+0000 osd.2 (osd 2021-02-10T04:33:22.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: .2) 661 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:22.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:33:22.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: 10T04:33:21.360742+0000 osd.2 ( 2021-02-10T04:33:22.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: osd.2) 662 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:22.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:33:22.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: 02-10T04:33:21.360748+0000 2021-02-10T04:33:22.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: osd.2 (osd.2) 663 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:22.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: 2021-02-10T04:33:22.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: cluster 2021-02-10T04:33:22.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: 2021-02-10T04:33:21.360755+0000 osd.2 (osd 2021-02-10T04:33:22.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: .2) 664 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:22.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:33:22.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: 10T04:33:21.360761+0000 osd.2 ( 2021-02-10T04:33:22.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: osd.2) 665 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:22.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:33:22.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: 02-10T04:33:21.488113+0000 osd.4 (osd.4) 447 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:22.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:22.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: -02-10T04:33:21.488119+0000 2021-02-10T04:33:22.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: osd.4 (osd.4) 448 : cluster [WRN] 2021-02-10T04:33:22.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:22.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: cluster 2021-02-10T04:33:22.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: 2021-02-10T04:33:21.488125+0000 osd.4 ( 2021-02-10T04:33:22.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: osd.4) 449 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:22.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: 2021-02-10T04:33:22.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: cluster 2021-02-10T04:33:21.488131+0000 osd.4 ( 2021-02-10T04:33:22.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: osd.4) 450 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:22.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: cluster 2021-02-10T04:33: 2021-02-10T04:33:22.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: 22.070306+0000 mgr.y (mgr.14140) 516 : 2021-02-10T04:33:22.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: cluster [DBG] pgmap v726: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 306 MiB data, 933 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 1.4 MiB/s wr, 19 op/s 2021-02-10T04:33:22.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:33:22.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: :33:22.135099+0000 mon.a (mon.0 2021-02-10T04:33:22.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[29319]: ) 1165 : cluster [DBG] osdmap e385: 8 total, 8 up, 8 in 2021-02-10T04:33:22.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: cluster 2021-02-10T04:33:20.404665+0000 2021-02-10T04:33:22.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: osd.2 (osd.2) 650 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:22.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:22.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: -02-10T04:33:20.404676+0000 osd.2 (osd.2) 651 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:22.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: cluster 2021-02-10T04:33:20.404682+0000 2021-02-10T04:33:22.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: osd.2 (osd.2) 652 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:22.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: cluster 2021-02-10T04:33:20 2021-02-10T04:33:22.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: .404697+0000 osd.2 (osd.2) 653 : cluster 2021-02-10T04:33:22.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:22.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: cluster 2021-02-10T04:33:20.404703+0000 2021-02-10T04:33:22.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: osd.2 (osd.2) 654 : cluster [WRN] 2021-02-10T04:33:22.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:22.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:22.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: T04:33:20.404708+0000 osd.2 (osd.2) 655 : cluster [WRN] 2021-02-10T04:33:22.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:22.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: cluster 2021-02-10T04:33: 2021-02-10T04:33:22.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: 20.404713+0000 osd.2 (osd.2) 656 2021-02-10T04:33:22.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:22.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:33:22.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: 33:20.404718+0000 osd.2 (osd.2 2021-02-10T04:33:22.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: ) 657 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:22.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:22.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: -10T04:33:21.360708+0000 osd.2 ( 2021-02-10T04:33:22.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: osd.2) 658 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:22.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:22.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: 02-10T04:33:21.360722+0000 2021-02-10T04:33:22.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: osd.2 (osd.2) 659 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:22.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: 2021-02-10T04:33:22.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: cluster 2021-02-10T04:33: 2021-02-10T04:33:22.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: 21.360729+0000 osd.2 (osd.2) 660 2021-02-10T04:33:22.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:22.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:22.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: -02-10T04:33:21.360736+0000 2021-02-10T04:33:22.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: osd.2 (osd.2) 661 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:22.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:22.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: -02-10T04:33:21.360742+0000 2021-02-10T04:33:22.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: osd.2 (osd.2) 662 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:22.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: 2021-02-10T04:33:22.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: cluster 2021-02-10T04:33:22.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: 2021-02-10T04:33:21.360748+0000 osd.2 (osd.2 2021-02-10T04:33:22.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: ) 663 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:22.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:33:22.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: 10T04:33:21.360755+0000 osd.2 (osd.2) 664 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:22.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:33:22.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: :33:21.360761+0000 osd.2 (osd. 2021-02-10T04:33:22.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: 2) 665 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:22.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:22.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: -02-10T04:33:21. 2021-02-10T04:33:22.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: 488113+0000 osd.4 (osd.4) 447 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:22.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: cluster 2021-02-10T04:33:22.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: 2021-02-10T04:33:22.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:33:22.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: 10T04:33:21.488119+0000 osd.4 (osd.4) 448 2021-02-10T04:33:22.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:22.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:22.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: 02-10T04:33:21. 2021-02-10T04:33:22.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: 488125+0000 osd.4 (osd.4) 449 : cluster 2021-02-10T04:33:22.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:22.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: cluster 2021-02-10T04:33 2021-02-10T04:33:22.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: :21.488131+0000 osd.4 (osd.4) 2021-02-10T04:33:22.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: 450 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:22.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:22.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: 02-10T04:33:22.070306+0000 mgr.y (mgr.14140) 516 2021-02-10T04:33:22.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: : cluster [DBG] pgmap v726: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 306 MiB data, 933 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 1.4 MiB/s wr, 19 op/s 2021-02-10T04:33:22.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:33:22.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: :33:22.135099+0000 mon.a (mon. 2021-02-10T04:33:22.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:22 smithi085 conmon[35961]: 0) 1165 : cluster [DBG] osdmap e385: 8 total, 8 up, 8 in 2021-02-10T04:33:23.137 INFO:tasks.rados.rados.0.smithi085.stdout:1435: done (8 left) 2021-02-10T04:33:23.137 INFO:tasks.rados.rados.0.smithi085.stdout:1436: read oid 26 snap -1 2021-02-10T04:33:23.138 INFO:tasks.rados.rados.0.smithi085.stdout:1436: expect (ObjNum 481 snap 139 seq_num 481) 2021-02-10T04:33:23.138 INFO:tasks.rados.rados.0.smithi085.stdout:1437: read oid 42 snap -1 2021-02-10T04:33:23.138 INFO:tasks.rados.rados.0.smithi085.stdout:1437: expect (ObjNum 204 snap 44 seq_num 204) 2021-02-10T04:33:23.138 INFO:tasks.rados.rados.0.smithi085.stdout:1438: copy_from oid 44 from oid 16 current snap is 151 2021-02-10T04:33:23.138 INFO:tasks.rados.rados.0.smithi085.stdout:1439: delete oid 8 current snap is 151 2021-02-10T04:33:23.149 INFO:tasks.rados.rados.0.smithi085.stdout:1439: done (11 left) 2021-02-10T04:33:23.149 INFO:tasks.rados.rados.0.smithi085.stdout:1440: copy_from oid 9 from oid 12 current snap is 151 2021-02-10T04:33:23.149 INFO:tasks.rados.rados.0.smithi085.stdout:1436: expect (ObjNum 481 snap 139 seq_num 481) 2021-02-10T04:33:23.328 INFO:tasks.rados.rados.0.smithi085.stdout:1436: done (11 left) 2021-02-10T04:33:23.328 INFO:tasks.rados.rados.0.smithi085.stdout:1441: read oid 14 snap -1 2021-02-10T04:33:23.328 INFO:tasks.rados.rados.0.smithi085.stdout:1441: expect (ObjNum 498 snap 143 seq_num 498) 2021-02-10T04:33:23.328 INFO:tasks.rados.rados.0.smithi085.stdout:1442: delete oid 31 current snap is 151 2021-02-10T04:33:23.328 INFO:tasks.rados.rados.0.smithi085.stdout:1437: expect (ObjNum 204 snap 44 seq_num 204) 2021-02-10T04:33:23.573 INFO:tasks.rados.rados.0.smithi085.stdout:1437: done (12 left) 2021-02-10T04:33:23.574 INFO:tasks.rados.rados.0.smithi085.stdout:1442: done (11 left) 2021-02-10T04:33:23.574 INFO:tasks.rados.rados.0.smithi085.stdout:1443: rollback oid 6 current snap is 151 2021-02-10T04:33:23.574 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 6 to 135 2021-02-10T04:33:23.574 INFO:tasks.rados.rados.0.smithi085.stdout:1441: expect (ObjNum 498 snap 143 seq_num 498) 2021-02-10T04:33:23.630 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: cluster 2021-02-10T04:33:22. 2021-02-10T04:33:23.630 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: 355460+0000 osd.2 (osd.2) 666 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:23.630 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:23.630 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: 02-10T04:33:22.355468+0000 osd.2 (osd.2) 667 : 2021-02-10T04:33:23.630 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:23.630 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:33:23.631 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: 33:22.355474+0000 osd.2 (osd.2) 2021-02-10T04:33:23.631 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: 668 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:23.631 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:23.631 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: T04:33:22.355481+0000 osd.2 (osd 2021-02-10T04:33:23.631 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: .2) 669 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:23.631 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:23.631 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: -10T04:33:22.355485+0000 2021-02-10T04:33:23.631 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: osd.2 (osd.2) 670 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:23.631 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: 2021-02-10T04:33:23.631 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: cluster 2021-02-10T04:33:23.632 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: 2021-02-10T04:33:22.355489+0000 osd.2 ( 2021-02-10T04:33:23.632 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: osd.2) 671 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:23.632 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:23.632 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: -02-10T04:33:22.355495+0000 2021-02-10T04:33:23.632 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: osd.2 (osd.2) 672 : cluster [WRN] 2021-02-10T04:33:23.632 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:23.632 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:33:23.632 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: 33:22.355501+0000 osd.2 (osd.2) 2021-02-10T04:33:23.632 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: 673 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:23.632 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:23.632 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: T04:33:22.460072+0000 osd.4 (osd.4) 2021-02-10T04:33:23.632 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: 451 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:23.632 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: cluster 2021-02-10T04:33:22.460081+0000 osd.4 (osd.4) 452 2021-02-10T04:33:23.633 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:23.633 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: cluster 2021-02-10T04:33:23.633 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: 2021-02-10T04:33:22.460087+0000 osd.4 (osd.4) 453 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:23.633 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:23.633 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: -10T04:33:22.460094+0000 osd.4 2021-02-10T04:33:23.633 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: (osd.4) 454 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:23.633 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:33:23.633 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: :33:23.137154+0000 mon.a (mon.0) 1166 : cluster 2021-02-10T04:33:23.633 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[35961]: [DBG] osdmap e386: 8 total, 8 up, 8 in 2021-02-10T04:33:23.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: cluster 2021-02-10T04:33:23.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: 2021-02-10T04:33:23.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: 2021-02-10T04:33:22.355460+0000 osd.2 (osd.2 2021-02-10T04:33:23.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: ) 666 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:23.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: cluster 2021-02-10T04:33:23.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:33:23.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: T04:33:22.355468+0000 osd.2 (osd 2021-02-10T04:33:23.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: .2) 667 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:23.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: cluster 2021-02-10T04:33:23.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: 2021-02-10T04:33:23.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: 2021 2021-02-10T04:33:23.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: - 2021-02-10T04:33:23.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: 02 2021-02-10T04:33:23.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: - 2021-02-10T04:33:23.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: 10 2021-02-10T04:33:23.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: T 2021-02-10T04:33:23.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: 04 2021-02-10T04:33:23.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: :33:22.355474+0000 osd.2 2021-02-10T04:33:23.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: (osd.2) 668 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:23.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: cluster 2021-02-10T04:33:23.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: 2021-02-10T04:33:22.355481+0000 osd.2 (osd.2) 669 : 2021-02-10T04:33:23.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:23.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: cluster 2021-02-10T04:33:22 2021-02-10T04:33:23.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: .355485+0000 osd.2 (osd.2) 670 : cluster [WRN] 2021-02-10T04:33:23.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:23.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: cluster 2021-02-10T04:33:22 2021-02-10T04:33:23.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: .355489+0000 osd.2 (osd.2) 671 : cluster [WRN] 2021-02-10T04:33:23.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:23.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: cluster 2021-02-10T04:33:23.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: 2021-02-10T04:33:22.355495+0000 osd.2 (osd.2 2021-02-10T04:33:23.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: ) 672 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:23.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: cluster 2021-02-10T04:33 2021-02-10T04:33:23.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: :22.355501+0000 osd.2 (osd.2) 673 : 2021-02-10T04:33:23.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:23.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:23.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: -02-10T04:33: 2021-02-10T04:33:23.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: 22.460072+0000 osd.4 (osd. 2021-02-10T04:33:23.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: 4) 451 : cluster [WRN] 2021-02-10T04:33:23.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:23.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:33:23.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: 04:33:22.460081+0000 osd.4 (osd.4 2021-02-10T04:33:23.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: ) 452 : 2021-02-10T04:33:23.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:23.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: cluster 2021-02-10T04:33:22. 2021-02-10T04:33:23.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: 460087+0000 osd.4 (osd.4) 453 : cluster [WRN] 2021-02-10T04:33:23.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:23.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: cluster 2021-02-10T04:33:23.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: 2021-02-10T04:33:22.460094+0000 osd.4 (osd.4) 2021-02-10T04:33:23.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: 454 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:23.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:33:23.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: -10T04:33:23.137154 2021-02-10T04:33:23.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1166 : cluster 2021-02-10T04:33:23.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[29319]: [DBG] osdmap e386: 8 total, 8 up, 8 in 2021-02-10T04:33:23.645 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[33677]: debug 2021-02-10T04:33:23.452+0000 7fbec19a1700 -1 osd.4 386 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:33:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: cluster 2021-02-10T04:33:22.355460+0000 osd.2 (osd.2) 666 : cluster [WRN] 2021-02-10T04:33:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: cluster 2021-02-10T04:33:22.355468+0000 osd.2 (osd.2) 667 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: cluster 2021-02-10T04:33:22.355474+0000 osd.2 (osd.2) 668 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: cluster 2021-02-10T04:33 2021-02-10T04:33:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: :22.355481+0000 osd.2 (osd.2) 669 : 2021-02-10T04:33:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:33:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: -02-10T04:33:22.355485+0000 2021-02-10T04:33:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: osd.2 (osd.2) 670 : cluster [WRN] 2021-02-10T04:33:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:33:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: 10T04:33:22.355489+0000 2021-02-10T04:33:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: osd.2 (osd.2) 671 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: 2021-02-10T04:33:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: cluster 2021-02-10T04:33:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: 2021-02-10T04:33:22. 2021-02-10T04:33:23.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: 355495+0000 2021-02-10T04:33:23.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: osd.2 (osd.2) 672 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:23.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:33:23.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: T04:33:22.355501+0000 osd.2 (osd.2 2021-02-10T04:33:23.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: ) 673 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:23.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:33:23.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: 04:33:22.460072+0000 2021-02-10T04:33:23.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: osd.4 (osd.4) 451 : 2021-02-10T04:33:23.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:23.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: cluster 2021-02-10T04:33:23.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: 2021-02- 2021-02-10T04:33:23.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: 10T04:33:22. 2021-02-10T04:33:23.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: 460081+0000 osd.4 (osd.4 2021-02-10T04:33:23.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: ) 452 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:23.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:33:23.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: 02-10T04:33:22.460087+0000 osd.4 2021-02-10T04:33:23.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: (osd.4) 453 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:23.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: cluster 2021-02-10T04:33:23.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: 2021-02-10T04:33: 2021-02-10T04:33:23.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: 22.460094+0000 osd.4 (osd 2021-02-10T04:33:23.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: .4) 454 : cluster [WRN] 2021-02-10T04:33:23.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:23.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: cluster 2021-02-10T04:33:23.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: 2021-02-10T04:33:23.137154 2021-02-10T04:33:23.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1166 : cluster 2021-02-10T04:33:23.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:23 smithi164 conmon[29288]: [DBG] osdmap e386: 8 total, 8 up, 8 in 2021-02-10T04:33:23.718 INFO:tasks.rados.rados.0.smithi085.stdout:1440: finishing copy_from to smithi08575397-9 2021-02-10T04:33:23.718 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 9 v 627 (ObjNum 85 snap 11 seq_num 85) dirty exists 2021-02-10T04:33:23.718 INFO:tasks.rados.rados.0.smithi085.stdout:1440: finishing copy_from racing read to smithi08575397-9 2021-02-10T04:33:23.718 INFO:tasks.rados.rados.0.smithi085.stdout:1438: finishing copy_from to smithi08575397-44 2021-02-10T04:33:23.718 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 625 (ObjNum 491 snap 141 seq_num 491) dirty exists 2021-02-10T04:33:23.718 INFO:tasks.rados.rados.0.smithi085.stdout:1438: finishing copy_from racing read to smithi08575397-44 2021-02-10T04:33:23.718 INFO:tasks.rados.rados.0.smithi085.stdout:1438: done (11 left) 2021-02-10T04:33:23.718 INFO:tasks.rados.rados.0.smithi085.stdout:1440: done (10 left) 2021-02-10T04:33:23.718 INFO:tasks.rados.rados.0.smithi085.stdout:1441: done (9 left) 2021-02-10T04:33:23.719 INFO:tasks.rados.rados.0.smithi085.stdout:1444: rollback oid 32 current snap is 151 2021-02-10T04:33:23.719 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 32 to 122 2021-02-10T04:33:23.719 INFO:tasks.rados.rados.0.smithi085.stdout:1443: finishing rollback tid 1 to smithi08575397-6 2021-02-10T04:33:23.719 INFO:tasks.rados.rados.0.smithi085.stdout:1443: finishing rollback tid 2 to smithi08575397-6 2021-02-10T04:33:23.719 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 678 (ObjNum 424 snap 109 seq_num 424) dirty exists 2021-02-10T04:33:23.719 INFO:tasks.rados.rados.0.smithi085.stdout:1443: done (9 left) 2021-02-10T04:33:23.719 INFO:tasks.rados.rados.0.smithi085.stdout:1445: snap_remove snap 145 2021-02-10T04:33:23.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:23 smithi085 conmon[49106]: debug 2021-02-10T04:33:23.384+0000 7f77a3e94700 -1 osd.2 386 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:33:24.340 INFO:tasks.rados.rados.0.smithi085.stdout:1445: done (9 left) 2021-02-10T04:33:24.341 INFO:tasks.rados.rados.0.smithi085.stdout:1446: snap_create 2021-02-10T04:33:24.341 INFO:tasks.rados.rados.0.smithi085.stdout:1444: finishing rollback tid 0 to smithi08575397-32 2021-02-10T04:33:24.341 INFO:tasks.rados.rados.0.smithi085.stdout:1444: finishing rollback tid 1 to smithi08575397-32 2021-02-10T04:33:24.341 INFO:tasks.rados.rados.0.smithi085.stdout:1444: finishing rollback tid 2 to smithi08575397-32 2021-02-10T04:33:24.341 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 32 v 110 (ObjNum 204 snap 44 seq_num 204) dirty exists 2021-02-10T04:33:24.395 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:33:24.426 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:24.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:24 smithi164 conmon[33677]: debug 2021-02-10T04:33:24.486+0000 7fbec19a1700 -1 osd.4 387 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:33:24.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:24 smithi164 conmon[29288]: cluster 2021-02-10T04:33:24.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:24 smithi164 conmon[29288]: 2021-02-10T04:33:24.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:24 smithi164 conmon[29288]: 2021 2021-02-10T04:33:24.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:24 smithi164 conmon[29288]: - 2021-02-10T04:33:24.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:24 smithi164 conmon[29288]: 02 2021-02-10T04:33:24.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:24 smithi164 conmon[29288]: - 2021-02-10T04:33:24.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:24 smithi164 conmon[29288]: 10 2021-02-10T04:33:24.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:24 smithi164 conmon[29288]: T04:33:23.453456+0000 osd.4 2021-02-10T04:33:24.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:24 smithi164 conmon[29288]: (osd.4) 455 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:24.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:24 smithi164 conmon[29288]: cluster 2021-02-10T04:33 2021-02-10T04:33:24.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:24 smithi164 conmon[29288]: :23.453462+0000 osd.4 (osd.4) 456 : 2021-02-10T04:33:24.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:24 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:24.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:24 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:33:24.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:24 smithi164 conmon[29288]: -10T04:33:23.453466+0000 2021-02-10T04:33:24.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:24 smithi164 conmon[29288]: osd.4 (osd.4) 457 : 2021-02-10T04:33:24.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:24 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:24.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:24 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:33:24.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:24 smithi164 conmon[29288]: -02-10T04:33:23 2021-02-10T04:33:24.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:24 smithi164 conmon[29288]: .453470+0000 osd.4 (osd.4) 458 : cluster 2021-02-10T04:33:24.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:24 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:24.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:24 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:33:24.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:24 smithi164 conmon[29288]: 33:24.070706+0000 mgr.y (mgr.14140 2021-02-10T04:33:24.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:24 smithi164 conmon[29288]: ) 517 : cluster [DBG] pgmap v729: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 303 MiB data, 924 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 611 KiB/s wr, 13 op/s 2021-02-10T04:33:24.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:24 smithi085 conmon[49106]: debug 2021-02-10T04:33:24.410+0000 7f77a3e94700 -1 osd.2 387 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:33:24.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:24 smithi085 conmon[35961]: cluster 2021-02-10T04:33:23.453456+0000 osd.4 (osd.4) 455 2021-02-10T04:33:24.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:24 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:24.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:24 smithi085 conmon[35961]: cluster 2021-02-10T04:33: 2021-02-10T04:33:24.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:24 smithi085 conmon[35961]: 23.453462+0000 osd.4 (osd.4) 456 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:24.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:24 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:24.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:24 smithi085 conmon[35961]: 02-10T04:33:23.453466+0000 2021-02-10T04:33:24.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:24 smithi085 conmon[35961]: osd.4 (osd.4) 457 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:24.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:24 smithi085 conmon[35961]: 2021-02-10T04:33:24.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:24 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:24.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:24 smithi085 conmon[35961]: -02-10T04:33:23.453470+0000 osd.4 (osd.4) 458 : cluster 2021-02-10T04:33:24.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:24 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:24.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:24 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:33:24.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:24 smithi085 conmon[35961]: :33:24.070706+0000 mgr.y (mgr.14140 2021-02-10T04:33:24.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:24 smithi085 conmon[35961]: ) 517 : cluster [DBG] pgmap v729: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 303 MiB data, 924 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 611 KiB/s wr, 13 op/s 2021-02-10T04:33:24.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:24 smithi085 conmon[29319]: cluster 2021-02-10T04:33:23.453456+0000 osd.4 ( 2021-02-10T04:33:24.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:24 smithi085 conmon[29319]: osd.4) 455 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:24.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:24 smithi085 conmon[29319]: cluster 2021-02-10T04:33:23.453462+0000 osd.4 (osd.4) 456 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:24.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:24 smithi085 conmon[29319]: cluster 2021-02-10T04:33:23.453466+0000 osd.4 (osd.4) 457 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:24.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:24 smithi085 conmon[29319]: cluster 2021-02-10T04:33:23.453470+0000 osd.4 (osd.4) 458 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:24.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:24 smithi085 conmon[29319]: cluster 2021-02-10T04:33:24.070706+0000 mgr.y (mgr.14140) 517 : cluster [DBG] pgmap v729: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 303 MiB data, 924 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 611 KiB/s wr, 13 op/s 2021-02-10T04:33:25.345 INFO:tasks.rados.rados.0.smithi085.stdout:1444: done (9 left) 2021-02-10T04:33:25.345 INFO:tasks.rados.rados.0.smithi085.stdout:1446: done (8 left) 2021-02-10T04:33:25.345 INFO:tasks.rados.rados.0.smithi085.stdout:1447: snap_remove snap 143 2021-02-10T04:33:25.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[33677]: debug 2021-02-10T04:33:25.520+0000 7fbec19a1700 -1 osd.4 388 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:33:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: cluster 2021-02-10T04:33:23.384907+0000 osd.2 ( 2021-02-10T04:33:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: osd.2) 674 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: cluster 2021-02-10T04:33:23.384915+0000 osd.2 (osd.2) 675 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: 2021-02-10T04:33:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:33:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: :33:23.384920+0000 osd.2 (osd.2) 676 : cluster 2021-02-10T04:33:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:33:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: :33:23.384926+0000 osd.2 (osd.2) 2021-02-10T04:33:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: 677 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: cluster 2021-02-10T04:33: 2021-02-10T04:33:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: 23.384931+0000 osd.2 (osd.2) 678 : cluster [WRN] 2021-02-10T04:33:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: cluster 2021-02-10T04:33: 2021-02-10T04:33:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: 23.384937+0000 osd.2 (osd.2) 679 2021-02-10T04:33:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:33:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: 33:23.384942+0000 osd.2 (osd.2) 2021-02-10T04:33:25.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: 680 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:25.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:33:25.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: 04:33:23.384947+0000 osd.2 ( 2021-02-10T04:33:25.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: osd.2) 681 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:25.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:33:25.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: 02-10T04:33:24.341123+0000 2021-02-10T04:33:25.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: mon.a (mon.0) 1167 : cluster [DBG] 2021-02-10T04:33:25.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: osdmap e387: 8 total, 8 up, 8 in 2021-02-10T04:33:25.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: cluster 2021-02-10T04:33:25.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: 2021-02-10T04:33:24.487338+0000 osd.4 2021-02-10T04:33:25.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: (osd.4) 459 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: 2021-02-10T04:33:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: cluster 2021-02-10T04:33:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: 2021-02-10T04:33:24.487347+0000 osd.4 (osd. 2021-02-10T04:33:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: 4) 460 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:33:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: 04:33:24.487353+0000 osd.4 (osd. 2021-02-10T04:33:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: 4) 461 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:33:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: 10T04:33:24.487358+0000 osd.4 (osd.4) 462 2021-02-10T04:33:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:25 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:25.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[49106]: debug 2021-02-10T04:33:25.457+0000 7f77a3e94700 -1 osd.2 388 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:33:25.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: cluster 2021-02-10T04:33:23.384907+0000 2021-02-10T04:33:25.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: osd.2 (osd.2) 674 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:25.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: cluster 2021-02-10T04:33:23 2021-02-10T04:33:25.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: .384915+0000 osd.2 (osd.2) 675 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:25.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:33:25.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: 04:33:23.384920+0000 osd.2 (osd. 2021-02-10T04:33:25.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: 2) 676 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:25.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:25.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: -10T04:33:23.384926+0000 osd.2 ( 2021-02-10T04:33:25.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: osd.2) 677 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:25.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: cluster 2021-02-10T04:33:25.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: 2021-02-10T04:33:23.384931 2021-02-10T04:33:25.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 678 : cluster [WRN] 2021-02-10T04:33:25.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:25.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:33:25.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: :33:23.384937+0000 osd.2 (osd.2 2021-02-10T04:33:25.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: ) 679 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:25.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:25.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: -02-10T04:33:23.384942+0000 2021-02-10T04:33:25.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: osd.2 (osd.2) 680 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:25.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:25.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: -10T04:33:23.384947+0000 osd.2 (osd.2 2021-02-10T04:33:25.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: ) 681 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:25.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:25.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: -02-10T04:33:24.341123+0000 2021-02-10T04:33:25.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: mon.a (mon.0) 1167 : cluster [DBG] osdmap e387: 8 total, 8 up, 8 in 2021-02-10T04:33:25.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:25.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: 02-10T04:33:24.487338+0000 osd.4 (osd.4) 459 : cluster 2021-02-10T04:33:25.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:25.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: cluster 2021-02-10T04:33:24.487347+0000 osd.4 (osd.4) 2021-02-10T04:33:25.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: 460 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:25.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: cluster 2021-02-10T04:33:24 2021-02-10T04:33:25.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: .487353+0000 osd.4 (osd.4) 461 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:25.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: 2021-02-10T04:33:25.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: cluster 2021-02-10T04:33:25.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: 2021-02-10T04:33:25.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:33:25.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: :33:24.487358+0000 osd.4 (osd. 2021-02-10T04:33:25.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[35961]: 4) 462 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:25.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: cluster 2021-02-10T04:33:25.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: 2021-02-10T04:33 2021-02-10T04:33:25.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: :23.384907+0000 osd.2 (osd.2) 674 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:25.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: cluster 2021-02-10T04:33:25.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: 2021-02-10T04:33: 2021-02-10T04:33:25.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: 23.384915+0000 osd.2 (osd.2) 675 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:25.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: cluster 2021-02-10T04:33:25.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: 2021-02-10T04:33:23.384920 2021-02-10T04:33:25.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 676 : cluster [WRN] 2021-02-10T04:33:25.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:25.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: cluster 2021-02-10T04:33:25.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:33:25.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: T04:33:23. 2021-02-10T04:33:25.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: 384926+0000 osd.2 (osd.2) 2021-02-10T04:33:25.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: 677 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:25.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: cluster 2021-02-10T04:33:25.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: 2021-02-10T04:33:23 2021-02-10T04:33:25.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: .384931+0000 osd.2 (osd.2) 678 : 2021-02-10T04:33:25.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:25.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:33:25.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: 04:33:23.384937+0000 osd.2 (osd. 2021-02-10T04:33:25.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: 2) 679 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:25.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:25.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: -02-10T04:33:23.384942 2021-02-10T04:33:25.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 680 : cluster [WRN] 2021-02-10T04:33:25.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:25.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: cluster 2021-02-10T04:33 2021-02-10T04:33:25.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: :23.384947+0000 osd.2 (osd.2) 681 2021-02-10T04:33:25.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:25.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:33:25.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: -10T04:33:24.341123+0000 mon.a 2021-02-10T04:33:25.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: (mon.0) 1167 : cluster [DBG] osdmap e387: 8 total, 8 up, 8 in 2021-02-10T04:33:25.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: 2021-02-10T04:33:25.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:25.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: -02-10T04:33: 2021-02-10T04:33:25.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: 24.487338+0000 osd.4 (osd 2021-02-10T04:33:25.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: .4) 459 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:25.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:25.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: -02-10T04:33:24.487347+0000 osd.4 ( 2021-02-10T04:33:25.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: osd.4) 460 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:25.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:25.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: -02-10T04:33:24.487353+0000 osd.4 2021-02-10T04:33:25.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: (osd.4) 461 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:25.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: cluster 2021-02-10T04:33:25.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: 2021-02-10T04:33:24.487358 2021-02-10T04:33:25.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 462 : cluster 2021-02-10T04:33:25.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:25 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:26.346 INFO:tasks.rados.rados.0.smithi085.stdout:1447: done (8 left) 2021-02-10T04:33:26.346 INFO:tasks.rados.rados.0.smithi085.stdout:1448: delete oid 3 current snap is 152 2021-02-10T04:33:26.356 INFO:tasks.rados.rados.0.smithi085.stdout:1448: done (8 left) 2021-02-10T04:33:26.356 INFO:tasks.rados.rados.0.smithi085.stdout:1449: rollback oid 30 current snap is 152 2021-02-10T04:33:26.356 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 30 to 135 2021-02-10T04:33:26.356 INFO:tasks.rados.rados.0.smithi085.stdout:1450: write oid 31 current snap is 152 2021-02-10T04:33:26.356 INFO:tasks.rados.rados.0.smithi085.stdout:1450: seq_num 517 ranges {689219=415737,1770367=646847} 2021-02-10T04:33:26.361 INFO:tasks.rados.rados.0.smithi085.stdout:1450: writing smithi08575397-31 from 689219 to 1104956 tid 1 2021-02-10T04:33:26.370 INFO:tasks.rados.rados.0.smithi085.stdout:1450: writing smithi08575397-31 from 1770367 to 2417214 tid 2 2021-02-10T04:33:26.370 INFO:tasks.rados.rados.0.smithi085.stdout:1451: write oid 36 current snap is 152 2021-02-10T04:33:26.370 INFO:tasks.rados.rados.0.smithi085.stdout:1449: finishing rollback tid 1 to smithi08575397-30 2021-02-10T04:33:26.370 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 667 (ObjNum 0 snap 22030 seq_num 82513168) dirty dne 2021-02-10T04:33:26.371 INFO:tasks.rados.rados.0.smithi085.stdout:1450: finishing write tid 1 to smithi08575397-31 2021-02-10T04:33:26.371 INFO:tasks.rados.rados.0.smithi085.stdout:1451: seq_num 518 ranges {706026=490986,1828849=790505} 2021-02-10T04:33:26.377 INFO:tasks.rados.rados.0.smithi085.stdout:1451: writing smithi08575397-36 from 706026 to 1197012 tid 1 2021-02-10T04:33:26.386 INFO:tasks.rados.rados.0.smithi085.stdout:1451: writing smithi08575397-36 from 1828849 to 2619354 tid 2 2021-02-10T04:33:26.386 INFO:tasks.rados.rados.0.smithi085.stdout:1449: done (10 left) 2021-02-10T04:33:26.386 INFO:tasks.rados.rados.0.smithi085.stdout:1452: snap_create 2021-02-10T04:33:26.386 INFO:tasks.rados.rados.0.smithi085.stdout:1450: finishing write tid 2 to smithi08575397-31 2021-02-10T04:33:26.387 INFO:tasks.rados.rados.0.smithi085.stdout:1450: finishing write tid 3 to smithi08575397-31 2021-02-10T04:33:26.387 INFO:tasks.rados.rados.0.smithi085.stdout:1450: finishing write tid 4 to smithi08575397-31 2021-02-10T04:33:26.387 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 640 (ObjNum 517 snap 152 seq_num 517) dirty exists 2021-02-10T04:33:26.387 INFO:tasks.rados.rados.0.smithi085.stdout:1450: left oid 31 (ObjNum 517 snap 152 seq_num 517) 2021-02-10T04:33:26.387 INFO:tasks.rados.rados.0.smithi085.stdout:1451: finishing write tid 1 to smithi08575397-36 2021-02-10T04:33:26.389 INFO:tasks.rados.rados.0.smithi085.stdout:1451: finishing write tid 2 to smithi08575397-36 2021-02-10T04:33:26.390 INFO:tasks.rados.rados.0.smithi085.stdout:1451: finishing write tid 3 to smithi08575397-36 2021-02-10T04:33:26.390 INFO:tasks.rados.rados.0.smithi085.stdout:1451: finishing write tid 4 to smithi08575397-36 2021-02-10T04:33:26.390 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 626 (ObjNum 518 snap 152 seq_num 518) dirty exists 2021-02-10T04:33:26.390 INFO:tasks.rados.rados.0.smithi085.stdout:1451: left oid 36 (ObjNum 518 snap 152 seq_num 518) 2021-02-10T04:33:26.407 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:33:26.441 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[45443]: debug 2021-02-10T04:33:26.441 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[45443]: 2021-02-10T04:33:26.143+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:26.441 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[45443]: 2021-02-10T04:33:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: cluster 2021-02-10T04:33:24 2021-02-10T04:33:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: .410625+0000 osd.2 (osd.2) 682 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: cluster 2021-02-10T04:33: 2021-02-10T04:33:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: 24.410630+0000 osd.2 (osd.2) 683 : cluster 2021-02-10T04:33:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:33:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: :33:24.410634+0000 osd.2 (osd.2) 684 2021-02-10T04:33:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: -02-10T04:33:24.410637+0000 osd.2 (osd.2) 2021-02-10T04:33:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: 685 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: -10T04:33:24.410640+0000 osd.2 (osd 2021-02-10T04:33:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: .2) 686 : cluster [WRN] 2021-02-10T04:33:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:33:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: 04:33:24.410646+0000 osd.2 (osd.2 2021-02-10T04:33:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: ) 687 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: cluster 2021-02-10T04:33 2021-02-10T04:33:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: :24.410651+0000 osd.2 (osd.2) 688 2021-02-10T04:33:26.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:26.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: cluster 2021-02-10T04:33:24.410654+0000 2021-02-10T04:33:26.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: osd.2 (osd.2) 689 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:26.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:26.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: T04:33:25.344640+0000 mon.a (mon.0) 1168 : 2021-02-10T04:33:26.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: cluster [DBG] osdmap e388: 8 total, 8 up, 8 in 2021-02-10T04:33:26.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: cluster 2021-02-10T04:33:26.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: 2021-02-10T04:33:25.521072+0000 osd.4 (osd.4) 463 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:26.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:26.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: 02-10T04:33:25.521080+0000 osd.4 ( 2021-02-10T04:33:26.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: osd.4) 464 : cluster [WRN] 2021-02-10T04:33:26.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:26.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: cluster 2021-02-10T04:33:26.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:33:26.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: T04:33:25.521084+0000 2021-02-10T04:33:26.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: osd.4 (osd.4) 465 : cluster 2021-02-10T04:33:26.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:26.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:26.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: -10T04:33:25.521087+0000 osd.4 ( 2021-02-10T04:33:26.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: osd.4) 466 : cluster [WRN] 2021-02-10T04:33:26.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:26.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:26.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: -02-10T04:33:26.071320+0000 2021-02-10T04:33:26.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: mgr.y (mgr.14140) 518 : 2021-02-10T04:33:26.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: cluster [DBG] pgmap v732: 168 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 159 active+clean; 300 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 901 KiB/s wr, 12 op/s 2021-02-10T04:33:26.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:26.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: 02-10T04:33:26.345917 2021-02-10T04:33:26.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1169 2021-02-10T04:33:26.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[35961]: : cluster [DBG] osdmap e389: 8 total, 8 up, 8 in 2021-02-10T04:33:26.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: cluster 2021-02-10T04:33:24.410625+0000 osd.2 ( 2021-02-10T04:33:26.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: osd.2) 682 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:26.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: cluster 2021-02-10T04:33:24.410630+0000 osd.2 (osd.2) 683 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:26.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: cluster 2021-02-10T04:33:24.410634+0000 osd.2 (osd.2) 684 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:26.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: cluster 2021-02-10T04:33:24.410637+0000 osd.2 (osd.2) 685 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:26.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: cluster 2021-02-10T04:33:24.410640+0000 osd.2 (osd.2) 686 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:26.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:26.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: -02-10T04:33:24.410646+0000 osd.2 (osd.2) 687 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:26.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: cluster 2021-02-10T04:33:24.410651 2021-02-10T04:33:26.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 688 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:26.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:33:26.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: :33:24.410654+0000 osd.2 (osd.2) 689 : cluster 2021-02-10T04:33:26.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:26.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: cluster 2021-02-10T04:33:25 2021-02-10T04:33:26.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: .344640+0000 mon.a (mon.0) 1168 : cluster 2021-02-10T04:33:26.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: [DBG] osdmap e388: 8 total, 8 up, 8 in 2021-02-10T04:33:26.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:33:26.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: -10T04:33:25.521072 2021-02-10T04:33:26.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 463 2021-02-10T04:33:26.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:26.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:33:26.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: -10T04:33:25.521080 2021-02-10T04:33:26.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 464 2021-02-10T04:33:26.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:26.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:26.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: -02-10T04:33: 2021-02-10T04:33:26.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: 25.521084+0000 osd.4 (osd 2021-02-10T04:33:26.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: .4) 465 : cluster [WRN] 2021-02-10T04:33:26.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:26.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:26.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: -02-10T04:33:25.521087 2021-02-10T04:33:26.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 466 : cluster 2021-02-10T04:33:26.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:26.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: cluster 2021-02-10T04:33:26.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: 2021-02-10T04:33:26.071320+0000 mgr.y (mgr.14140) 518 : cluster 2021-02-10T04:33:26.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: [DBG] pgmap v732: 168 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 159 active+clean; 300 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 901 KiB/s wr, 12 op/s 2021-02-10T04:33:26.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: cluster 2021-02-10T04:33:26.345917+0000 mon.a 2021-02-10T04:33:26.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[29319]: (mon.0) 1169 : cluster [DBG] osdmap e389: 8 total, 8 up, 8 in 2021-02-10T04:33:26.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[33677]: debug 2021-02-10T04:33:26.505+0000 7fbec19a1700 -1 osd.4 389 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:33:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: cluster 2021-02-10T04:33:24.410625+0000 osd.2 (osd 2021-02-10T04:33:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: .2) 682 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: cluster 2021-02-10T04:33:24.410630+0000 osd.2 (osd.2) 683 2021-02-10T04:33:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:33:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: -10T04:33:24.410634+0000 osd.2 (osd.2) 684 : cluster 2021-02-10T04:33:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:33:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: -10T04:33:24.410637+0000 osd.2 (osd 2021-02-10T04:33:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: .2) 685 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: cluster 2021-02-10T04:33:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: 2021-02-10T04:33:24.410640+0000 osd.2 2021-02-10T04:33:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: (osd.2) 686 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: cluster 2021-02-10T04:33:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: 2021-02-10T04:33:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: 2021 2021-02-10T04:33:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: - 2021-02-10T04:33:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: 02 2021-02-10T04:33:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: -10T04:33:24.410646+0000 osd.2 (osd.2) 687 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: cluster 2021-02-10T04:33 2021-02-10T04:33:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: :24.410651+0000 osd.2 (osd.2) 688 : cluster 2021-02-10T04:33:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:33:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: 04:33:24.410654+0000 osd.2 (osd.2) 689 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: cluster 2021-02-10T04:33:25.344640+0000 mon.a (mon.0 2021-02-10T04:33:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: ) 1168 : cluster [DBG] osdmap e388: 8 total, 8 up, 8 in 2021-02-10T04:33:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: cluster 2021-02-10T04:33:25.521072 2021-02-10T04:33:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 463 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: 2021-02-10T04:33:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:33:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: 02-10T04:33:25.521080+0000 osd.4 (osd.4) 464 2021-02-10T04:33:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: cluster 2021-02-10T04:33:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: 2021-02-10T04:33:25.521084+0000 2021-02-10T04:33:26.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: osd.4 (osd.4) 465 : cluster 2021-02-10T04:33:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:33:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: -02-10T04:33:25 2021-02-10T04:33:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: .521087+0000 osd.4 (osd 2021-02-10T04:33:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: .4) 466 2021-02-10T04:33:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:33:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: 02-10T04:33:26.071320+0000 mgr.y ( 2021-02-10T04:33:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: mgr.14140) 518 : cluster 2021-02-10T04:33:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: [DBG] pgmap v732: 168 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 159 active+clean; 300 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 901 KiB/s wr, 12 op/s 2021-02-10T04:33:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:33:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: T04:33:26.345917+0000 mon.a (mon.0 2021-02-10T04:33:26.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:26 smithi164 conmon[29288]: ) 1169 : cluster [DBG] osdmap e389: 8 total, 8 up, 8 in 2021-02-10T04:33:26.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:26 smithi085 conmon[49106]: debug 2021-02-10T04:33:26.496+0000 7f77a3e94700 -1 osd.2 389 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:33:27.227 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:33:27.265 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:27.370 INFO:tasks.rados.rados.0.smithi085.stdout:1450: done (10 left) 2021-02-10T04:33:27.370 INFO:tasks.rados.rados.0.smithi085.stdout:1451: done (9 left) 2021-02-10T04:33:27.370 INFO:tasks.rados.rados.0.smithi085.stdout:1452: done (8 left) 2021-02-10T04:33:27.370 INFO:tasks.rados.rados.0.smithi085.stdout:1453: snap_create 2021-02-10T04:33:27.374 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[29319]: cluster 2021-02-10T04:33:25.458300+0000 2021-02-10T04:33:27.374 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[29319]: osd.2 (osd.2) 690 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:27.374 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[29319]: cluster 2021-02-10T04:33:25.458309+0000 osd.2 (osd.2) 691 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:27.374 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[29319]: cluster 2021-02-10T04:33:25.458316+0000 osd.2 (osd.2) 692 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:27.374 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[29319]: cluster 2021-02-10T04:33:25.458323+0000 osd.2 (osd.2) 693 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:27.374 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[29319]: cluster 2021-02-10T04:33:25.458330+0000 osd.2 (osd. 2021-02-10T04:33:27.374 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[29319]: 2) 694 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:27.374 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[29319]: cluster 2021-02-10T04:33:25.458337+0000 osd.2 (osd.2) 695 2021-02-10T04:33:27.374 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:27.374 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[29319]: cluster 2021-02-10T04:33:27.374 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[29319]: 2021-02-10T04:33:25.458343+0000 osd.2 (osd. 2021-02-10T04:33:27.375 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[29319]: 2) 696 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:27.375 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[29319]: cluster 2021-02-10T04:33 2021-02-10T04:33:27.375 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[29319]: :25.458351+0000 osd.2 (osd.2) 697 : 2021-02-10T04:33:27.375 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:27.375 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:33:27.375 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[29319]: :33:26.506496+0000 osd.4 (osd.4 2021-02-10T04:33:27.375 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[29319]: ) 467 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:27.375 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:33:27.375 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[29319]: 04:33:26.506507+0000 osd.4 (osd. 2021-02-10T04:33:27.375 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[29319]: 4) 468 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:27.375 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:33:27.375 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[29319]: T04:33:26.506513+0000 osd.4 (osd 2021-02-10T04:33:27.375 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[29319]: .4) 469 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:27.375 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:33:27.376 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[29319]: -10T04:33:26.506518+0000 osd.4 2021-02-10T04:33:27.376 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[29319]: (osd.4) 470 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:27.636 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[49106]: debug 2021-02-10T04:33:27.486+0000 7f77a3e94700 -1 osd.2 390 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:33:27.637 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[45443]: debug 2021-02-10T04:33:27.636+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:27.637 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: cluster 2021-02-10T04:33:25.458300+0000 2021-02-10T04:33:27.637 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: osd.2 (osd.2) 690 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:27.637 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: cluster 2021-02-10T04:33:25.458309+0000 2021-02-10T04:33:27.637 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: osd.2 (osd.2) 691 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:27.637 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:27.638 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: 02-10T04:33:25. 2021-02-10T04:33:27.638 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: 458316+0000 osd.2 (osd.2) 692 : cluster 2021-02-10T04:33:27.638 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:27.638 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: cluster 2021-02-10T04:33:27.638 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:33:27.638 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: 33:25.458323+0000 osd.2 2021-02-10T04:33:27.638 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: (osd.2) 693 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:27.638 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: 2021-02-10T04:33:27.638 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: cluster 2021-02-10T04:33:25. 2021-02-10T04:33:27.638 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: 458330+0000 osd.2 (osd.2) 694 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:27.638 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:33:27.638 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: 10T04:33:25.458337+0000 2021-02-10T04:33:27.638 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: osd.2 (osd.2) 695 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:27.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:27.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: -02-10T04:33:25.458343+0000 2021-02-10T04:33:27.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: osd.2 (osd.2) 696 : cluster [WRN] 2021-02-10T04:33:27.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:27.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: cluster 2021-02-10T04:33:25. 2021-02-10T04:33:27.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: 458351+0000 osd.2 (osd.2) 697 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:27.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:27.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: -10T04:33:26.506496+0000 osd.4 ( 2021-02-10T04:33:27.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: osd.4) 467 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:27.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: cluster 2021-02-10T04:33:26.506507+0000 osd.4 (osd 2021-02-10T04:33:27.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: .4) 468 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:27.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: cluster 2021-02-10T04:33:26.506513+0000 2021-02-10T04:33:27.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: osd.4 (osd.4) 469 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:27.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:27.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: T04:33:26.506518+0000 osd.4 (osd. 2021-02-10T04:33:27.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:27 smithi085 conmon[35961]: 4) 470 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:27.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[33677]: debug 2021-02-10T04:33:27.493+0000 7fbec19a1700 -1 osd.4 389 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:33:27.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: cluster 2021-02-10T04:33:25.458300+0000 osd.2 (osd 2021-02-10T04:33:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: .2) 690 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: cluster 2021-02-10T04:33:25.458309+0000 osd.2 (osd.2) 691 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: cluster 2021-02-10T04:33:25.458316+0000 osd.2 (osd.2) 2021-02-10T04:33:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: 692 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:33:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: T04:33:25.458323+0000 osd.2 (osd. 2021-02-10T04:33:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: 2) 693 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:33:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: -02-10T04:33:25.458330+0000 osd.2 2021-02-10T04:33:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: (osd.2) 694 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: 2021-02-10T04:33:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: cluster 2021-02-10T04:33:25. 2021-02-10T04:33:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: 458337+0000 osd.2 (osd.2) 695 : cluster 2021-02-10T04:33:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:33:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: 33:25.458343+0000 osd.2 (osd.2) 2021-02-10T04:33:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: 696 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:33:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: T04:33:25.458351+0000 osd.2 (osd 2021-02-10T04:33:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: .2) 697 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:33:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: 02-10T04:33:26.506496+0000 osd.4 2021-02-10T04:33:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: (osd.4) 467 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: cluster 2021-02-10T04:33:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: 2021-02-10T04:33:26.506507+0000 2021-02-10T04:33:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: osd.4 (osd.4) 468 : cluster [WRN] 2021-02-10T04:33:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: cluster 2021-02-10T04:33: 2021-02-10T04:33:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: 26.506513+0000 osd.4 (osd.4) 469 2021-02-10T04:33:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:33:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: T04:33:26.506518+0000 osd.4 (osd 2021-02-10T04:33:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[29288]: .4) 470 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:27.674 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:33:27.677 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.2 dump_historic_ops 2021-02-10T04:33:27.779 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:33:27.811 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:28.373 INFO:tasks.rados.rados.0.smithi085.stdout:1453: done (8 left) 2021-02-10T04:33:28.374 INFO:tasks.rados.rados.0.smithi085.stdout:1454: delete oid 11 current snap is 154 2021-02-10T04:33:28.384 INFO:tasks.rados.rados.0.smithi085.stdout:1454: done (8 left) 2021-02-10T04:33:28.384 INFO:tasks.rados.rados.0.smithi085.stdout:1455: snap_remove snap 151 2021-02-10T04:33:28.384 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:33:27 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:33:28.387 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: cluster 2021-02-10T04:33:26.497152+0000 osd.2 (osd.2 2021-02-10T04:33:28.387 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: ) 698 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:28.387 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: cluster 2021-02-10T04:33:26.497162+0000 osd.2 (osd.2) 699 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:28.388 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: cluster 2021-02-10T04:33: 2021-02-10T04:33:28.388 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: 26.497168+0000 osd.2 (osd.2) 700 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:28.388 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: 2021-02-10T04:33:28.388 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: cluster 2021-02-10T04:33:26 2021-02-10T04:33:28.388 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: .497186+0000 osd.2 (osd.2) 701 : 2021-02-10T04:33:28.388 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:28.388 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:33:28.388 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: :33:26.497192+0000 osd.2 (osd.2 2021-02-10T04:33:28.388 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: ) 702 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:28.388 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:33:28.388 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: -10T04:33:26.497207+0000 osd.2 (osd 2021-02-10T04:33:28.388 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: .2) 703 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:28.388 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:33:28.389 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: -02-10T04:33:26.497213+0000 osd.2 2021-02-10T04:33:28.389 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: (osd.2) 704 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:28.389 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: 2021-02-10T04:33:28.389 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: cluster 2021-02-10T04:33:26 2021-02-10T04:33:28.389 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: .497225+0000 osd.2 (osd.2) 2021-02-10T04:33:28.389 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: 705 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:28.389 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: cluster 2021-02-10T04:33:28.389 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: 2021-02-10T04:33:27.366424+0000 2021-02-10T04:33:28.389 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: mon.a (mon.0) 1170 : cluster [DBG] osdmap e390: 8 total, 8 up, 8 in 2021-02-10T04:33:28.389 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:33:28.389 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: -10T04:33:27.493784+0000 osd.4 (osd.4) 471 2021-02-10T04:33:28.389 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:28.389 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:33:28.390 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: :33:27.493791+0000 osd.4 (osd. 2021-02-10T04:33:28.390 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: 4) 472 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:28.390 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:33:28.390 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: 02-10T04:33:27.493795 2021-02-10T04:33:28.390 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 473 : cluster 2021-02-10T04:33:28.390 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:28.390 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: cluster 2021-02-10T04:33:27. 2021-02-10T04:33:28.390 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: 493798+0000 osd.4 (osd.4) 474 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:28.390 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:33:28.390 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: 33:27.746279+0000 osd.3 (osd.3) 60 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:33:28.390 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: 2021-02-10T04:33:28.390 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:33:28.390 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: 04:33:27.746509+0000 osd.3 (osd.3) 61 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:33:28.391 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: audit 2021-02-10T04:33 2021-02-10T04:33:28.391 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: :27.965068+0000 mgr.y (mgr.14140) 519 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:33:28.391 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: 2021-02-10T04:33:28.391 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:33:28.391 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: 04:33:28.071751+0000 mgr.y (mgr.14140) 520 : cluster [DBG] pgmap v735: 168 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 159 active+clean; 300 MiB data, 920 MiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 901 KiB/s wr, 8 op/s 2021-02-10T04:33:28.391 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: cluster 2021-02-10T04:33 2021-02-10T04:33:28.391 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: :28.115176+0000 mon.a ( 2021-02-10T04:33:28.391 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[29288]: mon.0) 1171 : cluster [WRN] Health check update: 12 slow ops, oldest one blocked for 232 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:33:28.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:28 smithi164 conmon[33677]: debug 2021-02-10T04:33:28.536+0000 7fbec19a1700 -1 osd.4 391 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:33:28.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[49106]: debug 2021-02-10T04:33:28.491+0000 7f77a3e94700 -1 osd.2 391 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:33:28.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[29319]: cluster 2021-02-10T04:33:26.497152+0000 osd.2 (osd.2) 698 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:28.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[29319]: cluster 2021-02-10T04:33:26.497162+0000 osd.2 (osd.2) 699 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:28.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[29319]: cluster 2021-02-10T04:33:26.497168+0000 osd.2 (osd.2) 700 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:28.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[29319]: cluster 2021-02-10T04:33:26.497186+0000 osd.2 (osd.2) 701 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:28.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[29319]: cluster 2021-02-10T04:33:26.497192+0000 osd.2 (osd.2) 702 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:28.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[29319]: cluster 2021-02-10T04:33:26.497207+0000 osd.2 (osd.2) 703 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:28.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[29319]: cluster 2021-02-10T04:33:26.497213+0000 osd.2 (osd.2) 704 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:28.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:33:28.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[29319]: -10T04:33:26.497225+0000 osd.2 (osd.2) 705 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:28.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[29319]: cluster 2021-02-10T04:33:27.366424+0000 mon.a (mon.0) 1170 : cluster [DBG] osdmap e390: 8 total, 8 up, 8 in 2021-02-10T04:33:28.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[29319]: cluster 2021-02-10T04:33:27.493784+0000 osd.4 (osd.4) 471 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:28.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[29319]: cluster 2021-02-10T04:33:27.493791+0000 osd.4 (osd.4) 472 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:28.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[29319]: cluster 2021-02-10T04:33:27.493795+0000 osd.4 (osd.4) 473 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:28.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[29319]: cluster 2021-02-10T04:33:27.493798+0000 osd.4 (osd.4) 474 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:28.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[29319]: cluster 2021-02-10T04:33:27.746279+0000 osd.3 (osd.3) 60 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:33:28.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[29319]: cluster 2021-02-10T04:33:27.746509+0000 osd.3 (osd.3) 61 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:33:28.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[29319]: audit 2021-02-10T04:33:27.965068+0000 mgr.y (mgr.14140) 519 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:33:28.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[29319]: cluster 2021-02-10T04:33:28.071751+0000 mgr.y (mgr.14140) 520 : cluster [DBG] pgmap v735: 168 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 159 active+clean; 300 MiB data, 920 MiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 901 KiB/s wr, 8 op/s 2021-02-10T04:33:28.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[29319]: cluster 2021-02-10T04:33:28.115176+0000 mon.a (mon.0) 1171 : cluster [WRN] Health check update: 12 slow ops, oldest one blocked for 232 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:33:28.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: cluster 2021-02-10T04:33:28.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: 2021-02-10T04:33:28.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: 2021 2021-02-10T04:33:28.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: - 2021-02-10T04:33:28.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: 02 2021-02-10T04:33:28.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: - 2021-02-10T04:33:28.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: 10 2021-02-10T04:33:28.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: T 2021-02-10T04:33:28.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: 04 2021-02-10T04:33:28.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: :33:26 2021-02-10T04:33:28.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: .497152+0000 osd.2 (osd.2) 698 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:28.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:33:28.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: :33:26.497162+0000 osd.2 (osd.2) 699 : cluster 2021-02-10T04:33:28.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:28.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:33:28.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: 33:26.497168 2021-02-10T04:33:28.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 700 2021-02-10T04:33:28.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: cluster 2021-02-10T04:33:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: 2021-02-10T04:33:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: 2021 2021-02-10T04:33:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: - 2021-02-10T04:33:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: 02-10 2021-02-10T04:33:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: T04:33:26.497186+0000 osd.2 (osd.2) 701 : 2021-02-10T04:33:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: 02-10T04:33:26.497192+0000 osd.2 (osd.2) 702 : cluster 2021-02-10T04:33:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: cluster 2021-02-10T04:33:26.497207+0000 2021-02-10T04:33:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: osd.2 (osd.2) 703 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: -10T04:33:26.497213+0000 osd.2 ( 2021-02-10T04:33:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: osd.2) 704 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:33:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: 10T04:33:26.497225+0000 osd.2 (osd.2) 705 2021-02-10T04:33:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: : cluster 2021-02-10T04:33:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: cluster 2021-02-10T04:33:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: 2021-02-10T04:33:27. 2021-02-10T04:33:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: 366424+0000 mon.a (mon.0) 1170 : 2021-02-10T04:33:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: cluster [DBG] osdmap e390: 8 total, 8 up, 8 in 2021-02-10T04:33:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: cluster 2021-02-10T04:33:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: 2021-02-10T04:33:27.493784+0000 2021-02-10T04:33:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: osd.4 (osd.4) 471 : cluster [WRN] 2021-02-10T04:33:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: -02-10T04:33:27 2021-02-10T04:33:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: .493791+0000 2021-02-10T04:33:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: osd.4 (osd.4) 472 : cluster [WRN] 2021-02-10T04:33:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: cluster 2021-02-10T04:33: 2021-02-10T04:33:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: 27.493795+0000 osd.4 (osd. 2021-02-10T04:33:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: 4) 473 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:33:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: 10T04:33:27.493798+0000 osd.4 (osd 2021-02-10T04:33:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: .4) 474 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: T04:33:27.746279+0000 osd.3 (osd.3) 60 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:33:28.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:28.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: -02-10T04:33:27.746509+0000 2021-02-10T04:33:28.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: osd.3 (osd.3) 2021-02-10T04:33:28.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: 61 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:33:28.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: audit 2021-02-10T04:33:28.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: 2021-02-10T04:33:27.965068+0000 2021-02-10T04:33:28.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: mgr.y (mgr.14140) 519 : audit 2021-02-10T04:33:28.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:33:28.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:28.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: -02-10T04:33:28.071751+0000 mgr.y (mgr.14140) 520 2021-02-10T04:33:28.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: : cluster [DBG] pgmap v735: 168 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 159 active+clean; 300 MiB data, 920 MiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 901 KiB/s wr, 8 op/s 2021-02-10T04:33:28.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:28.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: T04:33:28.115176+0000 mon.a (mon. 2021-02-10T04:33:28.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:28 smithi085 conmon[35961]: 0) 1171 : cluster [WRN] Health check update: 12 slow ops, oldest one blocked for 232 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:33:29.398 INFO:tasks.rados.rados.0.smithi085.stdout:1455: done (8 left) 2021-02-10T04:33:29.399 INFO:tasks.rados.rados.0.smithi085.stdout:1456: snap_create 2021-02-10T04:33:29.679 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset noscrub 2021-02-10T04:33:29.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[49106]: debug 2021-02-10T04:33:29.503+0000 7f77a3e94700 -1 osd.2 392 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:33:29.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: cluster 2021-02-10T04:33:29.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: 2021-02-10T04:33:27. 2021-02-10T04:33:29.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: 487488+0000 osd.2 (osd.2) 706 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:29.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:29.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: -02-10T04:33:27.487505+0000 2021-02-10T04:33:29.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: osd.2 (osd.2) 707 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:29.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:33:29.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: 02-10T04:33:27.487508+0000 osd.2 (osd 2021-02-10T04:33:29.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: .2) 708 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:29.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: cluster 2021-02-10T04:33:27.487512 2021-02-10T04:33:29.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 709 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:29.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:33:29.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: :33:27.487517+0000 osd.2 (osd.2) 710 : cluster 2021-02-10T04:33:29.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:29.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:33:29.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: 10T04:33:27.487521+0000 osd.2 (osd.2) 711 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:29.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:33:29.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: -10T04:33:27.487524+0000 osd.2 (osd. 2021-02-10T04:33:29.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: 2) 712 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:29.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: cluster 2021-02-10T04:33: 2021-02-10T04:33:29.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: 27.487528+0000 osd.2 (osd.2) 713 : cluster [WRN] 2021-02-10T04:33:29.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:29.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:33:29.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: -10T04:33:28.374099+0000 mon.a (mon.0) 1172 : cluster 2021-02-10T04:33:29.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: [DBG] osdmap e391: 8 total, 8 up, 8 in 2021-02-10T04:33:29.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: cluster 2021-02-10T04:33:28 2021-02-10T04:33:29.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: .537043+0000 osd.4 (osd.4) 475 : cluster 2021-02-10T04:33:29.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:29.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: cluster 2021-02-10T04:33:28.537048 2021-02-10T04:33:29.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 476 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:29.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: cluster 2021-02-10T04:33:28. 2021-02-10T04:33:29.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: 537052+0000 osd.4 (osd.4) 477 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:29.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:33:29.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: T04:33:28.537055+0000 osd.4 (osd.4) 2021-02-10T04:33:29.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[29319]: 478 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:29.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: cluster 2021-02-10T04:33:27.487488 2021-02-10T04:33:29.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 706 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:29.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: cluster 2021-02-10T04:33:29.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: 2021-02-10T04:33:27.487505+0000 osd.2 (osd.2 2021-02-10T04:33:29.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: ) 707 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:29.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:29.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: -10T04:33:27.487508+0000 osd.2 (osd 2021-02-10T04:33:29.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: .2) 708 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:29.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:29.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: -10T04:33:27.487512+0000 2021-02-10T04:33:29.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: osd.2 (osd.2) 709 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:29.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: 2021-02-10T04:33:29.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:29.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: T04:33:27.487517+0000 osd.2 ( 2021-02-10T04:33:29.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: osd.2) 710 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:29.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:29.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: -02-10T04:33:27.487521 2021-02-10T04:33:29.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 711 : cluster 2021-02-10T04:33:29.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:29.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:33:29.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: 33:27.487524+0000 osd.2 (osd.2 2021-02-10T04:33:29.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: ) 712 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:29.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:29.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: -02-10T04:33:27.487528 2021-02-10T04:33:29.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 713 : cluster [WRN] 2021-02-10T04:33:29.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:29.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:33:29.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: 04:33:28 2021-02-10T04:33:29.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: .374099+0000 2021-02-10T04:33:29.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: mon.a (mon.0) 1172 : cluster [DBG] 2021-02-10T04:33:29.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: osdmap e391: 8 total, 8 up, 8 in 2021-02-10T04:33:29.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:33:29.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: 10T04:33:28.537043+0000 osd.4 2021-02-10T04:33:29.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: (osd.4) 475 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:29.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:29.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: -10T04:33:28.537048+0000 osd.4 ( 2021-02-10T04:33:29.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: osd.4) 476 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:29.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: 2021-02-10T04:33:29.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:33:29.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: 04:33:28.537052+0000 osd.4 (osd 2021-02-10T04:33:29.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: .4) 477 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:29.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: cluster 2021-02-10T04:33:29.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: 2021-02-10T04:33:28.537055+0000 osd.4 (osd.4) 478 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:29.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:29 smithi085 conmon[35961]: 2021-02-10T04:33:29.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[33677]: debug 2021-02-10T04:33:29.569+0000 7fbec19a1700 -1 osd.4 391 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:33:29.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: cluster 2021-02-10T04:33:29.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: 2021-02-10T04:33 2021-02-10T04:33:29.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: :27.487488+0000 osd.2 (osd.2) 706 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:29.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:33:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: 02-10T04:33:27.487505+0000 osd.2 (osd 2021-02-10T04:33:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: .2) 707 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: cluster 2021-02-10T04:33:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: 2021-02-10T04:33:27.487508 2021-02-10T04:33:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 708 : cluster [WRN] 2021-02-10T04:33:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: cluster 2021-02-10T04:33 2021-02-10T04:33:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: :27.487512+0000 osd.2 (osd.2) 709 : cluster 2021-02-10T04:33:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: cluster 2021-02-10T04:33:27.487517+0000 osd.2 ( 2021-02-10T04:33:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: osd.2) 710 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:29.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:33:29.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: T04:33:27.487521+0000 osd.2 (osd.2 2021-02-10T04:33:29.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: ) 711 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:29.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:33:29.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: :33:27.487524+0000 osd.2 2021-02-10T04:33:29.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: (osd.2) 712 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:29.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: 2021-02-10T04:33:29.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: cluster 2021-02-10T04:33:29.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: 2021-02-10T04:33:27.487528+0000 osd.2 2021-02-10T04:33:29.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: (osd.2) 713 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:29.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:33:29.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: 02-10T04:33:28.374099+0000 2021-02-10T04:33:29.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: mon.a (mon.0) 1172 : cluster [DBG] 2021-02-10T04:33:29.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: osdmap e391: 8 total, 8 up, 8 in 2021-02-10T04:33:29.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: cluster 2021-02-10T04:33:29.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: 2021-02-10T04:33:28.537043+0000 osd.4 ( 2021-02-10T04:33:29.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: osd.4) 475 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:29.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:33:29.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: T04:33:28.537048+0000 osd.4 ( 2021-02-10T04:33:29.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: osd.4) 476 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:29.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:33:29.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: -02-10T04:33:28.537052+0000 osd.4 (osd.4 2021-02-10T04:33:29.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: ) 477 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:29.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: cluster 2021-02-10T04:33:29.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:33:29.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: 33:28.537055+0000 osd.4 2021-02-10T04:33:29.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: (osd.4) 478 : cluster 2021-02-10T04:33:29.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:29 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:30.397 INFO:tasks.rados.rados.0.smithi085.stdout:1456: done (8 left) 2021-02-10T04:33:30.397 INFO:tasks.rados.rados.0.smithi085.stdout:1457: rollback oid 1 current snap is 155 2021-02-10T04:33:30.397 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 1 to 148 2021-02-10T04:33:30.397 INFO:tasks.rados.rados.0.smithi085.stdout:1458: copy_from oid 38 from oid 6 current snap is 155 2021-02-10T04:33:30.397 INFO:tasks.rados.rados.0.smithi085.stdout:1459: write oid 30 current snap is 155 2021-02-10T04:33:30.397 INFO:tasks.rados.rados.0.smithi085.stdout:1459: seq_num 520 ranges {739640=641484,1829460=794174} 2021-02-10T04:33:30.399 INFO:tasks.rados.rados.0.smithi085.stdout:1459: writing smithi08575397-30 from 739640 to 1381124 tid 1 2021-02-10T04:33:30.408 INFO:tasks.rados.rados.0.smithi085.stdout:1459: writing smithi08575397-30 from 1829460 to 2623634 tid 2 2021-02-10T04:33:30.408 INFO:tasks.rados.rados.0.smithi085.stdout:1460: read oid 26 snap -1 2021-02-10T04:33:30.408 INFO:tasks.rados.rados.0.smithi085.stdout:1460: expect (ObjNum 481 snap 139 seq_num 481) 2021-02-10T04:33:30.408 INFO:tasks.rados.rados.0.smithi085.stdout:1461: write oid 8 current snap is 155 2021-02-10T04:33:30.408 INFO:tasks.rados.rados.0.smithi085.stdout:1461: seq_num 521 ranges {756447=716733,1887942=537832} 2021-02-10T04:33:30.417 INFO:tasks.rados.rados.0.smithi085.stdout:1461: writing smithi08575397-8 from 756447 to 1473180 tid 1 2021-02-10T04:33:30.423 INFO:tasks.rados.rados.0.smithi085.stdout:1461: writing smithi08575397-8 from 1887942 to 2425774 tid 2 2021-02-10T04:33:30.423 INFO:tasks.rados.rados.0.smithi085.stdout:1462: copy_from oid 23 from oid 9 current snap is 155 2021-02-10T04:33:30.424 INFO:tasks.rados.rados.0.smithi085.stdout:1457: finishing rollback tid 0 to smithi08575397-1 2021-02-10T04:33:30.424 INFO:tasks.rados.rados.0.smithi085.stdout:1457: finishing rollback tid 1 to smithi08575397-1 2021-02-10T04:33:30.424 INFO:tasks.rados.rados.0.smithi085.stdout:1457: finishing rollback tid 2 to smithi08575397-1 2021-02-10T04:33:30.424 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 596 (ObjNum 424 snap 109 seq_num 424) dirty exists 2021-02-10T04:33:30.424 INFO:tasks.rados.rados.0.smithi085.stdout:1459: finishing write tid 1 to smithi08575397-30 2021-02-10T04:33:30.424 INFO:tasks.rados.rados.0.smithi085.stdout:1460: expect (ObjNum 481 snap 139 seq_num 481) 2021-02-10T04:33:30.598 INFO:tasks.rados.rados.0.smithi085.stdout:1461: finishing write tid 1 to smithi08575397-8 2021-02-10T04:33:30.598 INFO:tasks.rados.rados.0.smithi085.stdout:1458: finishing copy_from to smithi08575397-38 2021-02-10T04:33:30.598 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 678 (ObjNum 424 snap 109 seq_num 424) dirty exists 2021-02-10T04:33:30.599 INFO:tasks.rados.rados.0.smithi085.stdout:1458: finishing copy_from racing read to smithi08575397-38 2021-02-10T04:33:30.599 INFO:tasks.rados.rados.0.smithi085.stdout:1461: finishing write tid 2 to smithi08575397-8 2021-02-10T04:33:30.599 INFO:tasks.rados.rados.0.smithi085.stdout:1457: done (13 left) 2021-02-10T04:33:30.599 INFO:tasks.rados.rados.0.smithi085.stdout:1458: done (12 left) 2021-02-10T04:33:30.599 INFO:tasks.rados.rados.0.smithi085.stdout:1460: done (11 left) 2021-02-10T04:33:30.599 INFO:tasks.rados.rados.0.smithi085.stdout:1463: write oid 26 current snap is 155 2021-02-10T04:33:30.599 INFO:tasks.rados.rados.0.smithi085.stdout:1461: finishing write tid 3 to smithi08575397-8 2021-02-10T04:33:30.599 INFO:tasks.rados.rados.0.smithi085.stdout:1461: finishing write tid 4 to smithi08575397-8 2021-02-10T04:33:30.599 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 599 (ObjNum 521 snap 155 seq_num 521) dirty exists 2021-02-10T04:33:30.599 INFO:tasks.rados.rados.0.smithi085.stdout:1461: left oid 8 (ObjNum 521 snap 155 seq_num 521) 2021-02-10T04:33:30.599 INFO:tasks.rados.rados.0.smithi085.stdout:1459: finishing write tid 2 to smithi08575397-30 2021-02-10T04:33:30.599 INFO:tasks.rados.rados.0.smithi085.stdout:1459: finishing write tid 3 to smithi08575397-30 2021-02-10T04:33:30.600 INFO:tasks.rados.rados.0.smithi085.stdout:1463: seq_num 522 ranges {773254=791982,2346424=426830} 2021-02-10T04:33:30.606 INFO:tasks.rados.rados.0.smithi085.stdout:1463: writing smithi08575397-26 from 773254 to 1565236 tid 1 2021-02-10T04:33:30.612 INFO:tasks.rados.rados.0.smithi085.stdout:1463: writing smithi08575397-26 from 2346424 to 2773254 tid 2 2021-02-10T04:33:30.613 INFO:tasks.rados.rados.0.smithi085.stdout:1461: done (11 left) 2021-02-10T04:33:30.613 INFO:tasks.rados.rados.0.smithi085.stdout:1464: snap_remove snap 125 2021-02-10T04:33:30.691 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[52836]: debug 2021-02-10T04:33:30.519+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:30.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[49106]: debug 2021-02-10T04:33:30.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[49106]: 2021-02-10T04:33:30.529+0000 7f77a3e94700 -1 osd.2 393 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:33:30.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: cluster 2021-02-10T04:33:28.492371+0000 osd.2 ( 2021-02-10T04:33:30.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: osd.2) 714 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:30.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: cluster 2021-02-10T04:33:28.492381+0000 osd.2 (osd.2) 715 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:30.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: cluster 2021-02-10T04:33:28.492386+0000 osd.2 (osd.2) 716 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:30.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: cluster 2021-02-10T04:33:28.492392+0000 osd.2 (osd.2) 717 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:30.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:33:30.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: 33:28.492399+0000 osd.2 (osd.2) 718 : cluster 2021-02-10T04:33:30.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:30.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:33:30.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: 33:28.492405+0000 osd.2 (osd.2) 719 : 2021-02-10T04:33:30.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:30.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:33:30.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: 33:28.492411+0000 osd.2 (osd.2) 720 2021-02-10T04:33:30.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:30.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:33:30.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: 02-10T 2021-02-10T04:33:30.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: 04:33:28.492417+0000 osd.2 (osd 2021-02-10T04:33:30.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: .2) 721 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:30.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:33:30.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: 04:33:29.391378+0000 mon.a (mon.0) 1173 : cluster [DBG] osdmap e392: 8 total, 8 up, 8 in 2021-02-10T04:33:30.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: cluster 2021-02-10T04:33:30.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: 2021-02-10T04:33:29.504154+0000 osd.2 (osd. 2021-02-10T04:33:30.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: 2) 722 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:30.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:30.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: -02-10T04:33:29.504160+0000 osd.2 2021-02-10T04:33:30.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: (osd.2) 723 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:30.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:33:30.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: 02-10T04:33:29.504163+0000 osd.2 2021-02-10T04:33:30.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: (osd.2) 724 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:30.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:33:30.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: 10T04:33:29.504166+0000 2021-02-10T04:33:30.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: osd.2 (osd.2) 725 : 2021-02-10T04:33:30.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:30.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:33:30.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: 10T04:33:29.504173 2021-02-10T04:33:30.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 726 2021-02-10T04:33:30.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:30.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:33:30.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: 02-10T04:33:29 2021-02-10T04:33:30.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: .504177+0000 osd.2 (osd.2 2021-02-10T04:33:30.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: ) 727 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:30.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: cluster 2021-02-10T04:33:30.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: 2021 2021-02-10T04:33:30.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: -02-10T 2021-02-10T04:33:30.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: 04:33:29.504186+0000 2021-02-10T04:33:30.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: osd.2 (osd.2) 728 : cluster 2021-02-10T04:33:30.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:30.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:30.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: -02-10T04:33: 2021-02-10T04:33:30.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: 29.504191+0000 osd.2 (osd. 2021-02-10T04:33:30.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: 2) 729 : cluster [WRN] 2021-02-10T04:33:30.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:30.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:30.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: -02-10T04:33:29.570549+0000 osd.4 (osd.4) 479 : 2021-02-10T04:33:30.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:30.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: cluster 2021-02-10T04:33:30.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: 2021-02-10T04:33:29.570557+0000 osd.4 (osd.4 2021-02-10T04:33:30.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: ) 480 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:30.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:33:30.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: 33:29.570564+0000 osd.4 (osd.4) 481 : 2021-02-10T04:33:30.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:30.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: cluster 2021-02-10T04:33:30.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: 2021-02-10T04:33:29.570567+0000 osd.4 ( 2021-02-10T04:33:30.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: osd.4) 482 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:30.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:33:30.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: 04:33:30.072178+0000 mgr.y (mgr.14140) 2021-02-10T04:33:30.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: 521 : cluster [DBG] pgmap v738: 168 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 161 active+clean; 306 MiB data, 923 MiB used, 714 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.1 MiB/s wr, 6 op/s 2021-02-10T04:33:30.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: cluster 2021-02-10T04:33:30 2021-02-10T04:33:30.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: .390684+0000 mon.a (mon.0) 1174 : cluster [DBG] 2021-02-10T04:33:30.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[29319]: osdmap e393: 8 total, 8 up, 8 in 2021-02-10T04:33:30.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: cluster 2021-02-10T04:33:28.492371+0000 osd.2 2021-02-10T04:33:30.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: (osd.2) 714 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:30.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:30.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: -02-10T04:33:28.492381+0000 osd.2 (osd.2) 715 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:30.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: 2021-02-10T04:33:30.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: cluster 2021-02-10T04:33:30.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: 2021 2021-02-10T04:33:30.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: - 2021-02-10T04:33:30.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: 02 2021-02-10T04:33:30.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: - 2021-02-10T04:33:30.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: 10 2021-02-10T04:33:30.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: T 2021-02-10T04:33:30.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: 04 2021-02-10T04:33:30.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: :33:28.492386+0000 osd.2 2021-02-10T04:33:30.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: (osd.2) 716 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:30.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:33:30.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: :33:28.492392+0000 osd.2 (osd.2) 717 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:30.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:30.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: -10T04:33:28.492399+0000 osd.2 (osd 2021-02-10T04:33:30.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: .2) 718 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:30.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:30.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: -02-10T04:33:28.492405 2021-02-10T04:33:30.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 719 : cluster 2021-02-10T04:33:30.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:30.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: cluster 2021-02-10T04:33:28.492411+0000 osd.2 2021-02-10T04:33:30.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: (osd.2) 720 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:30.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: cluster 2021-02-10T04:33:28 2021-02-10T04:33:30.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: .492417+0000 osd.2 (osd.2) 721 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:30.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: cluster 2021-02-10T04:33:29.391378+0000 2021-02-10T04:33:30.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: mon.a (mon.0) 1173 : cluster [DBG] osdmap e392: 8 total, 8 up, 8 in 2021-02-10T04:33:30.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: 2021-02-10T04:33:30.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: cluster 2021-02-10T04:33:30.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: 2021-02-10T04:33:29.504154+0000 osd.2 (osd 2021-02-10T04:33:30.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: .2) 722 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:30.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:30.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: T04:33:29.504160+0000 osd.2 (osd. 2021-02-10T04:33:30.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: 2) 723 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:30.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:33:30.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: :33:29.504163+0000 osd.2 (osd.2) 724 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:30.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: cluster 2021-02-10T04:33:29.504166+0000 2021-02-10T04:33:30.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: osd.2 (osd.2) 725 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:30.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:30.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: 02-10T04:33:29.504173+0000 osd.2 2021-02-10T04:33:30.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: (osd.2) 726 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:30.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:30.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: -02-10T04:33:29.504177+0000 2021-02-10T04:33:30.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: osd.2 (osd.2) 727 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:30.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: 2021-02-10T04:33:30.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:30.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: -10T04:33:29 2021-02-10T04:33:30.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: .504186+0000 osd.2 (osd. 2021-02-10T04:33:30.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: 2) 728 : cluster [WRN] 2021-02-10T04:33:30.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:30.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:30.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: T04:33:29.504191+0000 osd.2 (osd. 2021-02-10T04:33:30.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: 2) 729 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:30.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:30.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: -10T04:33:29.570549+0000 osd.4 2021-02-10T04:33:30.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: (osd.4) 479 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:30.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:30.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: 02-10T04:33:29.570557+0000 2021-02-10T04:33:30.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: osd.4 (osd.4) 480 : cluster 2021-02-10T04:33:30.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:30.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:33:30.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: :33:29.570564+0000 osd.4 2021-02-10T04:33:30.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: (osd.4) 481 : cluster 2021-02-10T04:33:30.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:30.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:33:30.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: 10T04:33:29.570567+0000 osd.4 ( 2021-02-10T04:33:30.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: osd.4) 482 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:30.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:30.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: T04:33:30.072178+0000 mgr.y (mgr. 2021-02-10T04:33:30.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: 14140) 521 : cluster [DBG] pgmap v738: 168 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 161 active+clean; 306 MiB data, 923 MiB used, 714 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.1 MiB/s wr, 6 op/s 2021-02-10T04:33:30.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:33:30.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: 33:30.390684+0000 mon.a (mon.0) 1174 : cluster 2021-02-10T04:33:30.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:30 smithi085 conmon[35961]: [DBG] osdmap e393: 8 total, 8 up, 8 in 2021-02-10T04:33:30.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[33677]: debug 2021-02-10T04:33:30.569+0000 7fbec19a1700 -1 osd.4 393 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:33:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: cluster 2021-02-10T04:33:28.492371+0000 2021-02-10T04:33:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: osd.2 (osd.2) 714 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: cluster 2021-02-10T04:33:28.492381+0000 osd.2 (osd.2) 715 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: cluster 2021-02-10T04:33:28.492386+0000 2021-02-10T04:33:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: osd.2 (osd.2) 716 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: cluster 2021-02-10T04:33 2021-02-10T04:33:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: :28.492392+0000 osd.2 (osd.2 2021-02-10T04:33:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: ) 717 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:33:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: 02-10T04:33:28.492399+0000 2021-02-10T04:33:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: osd.2 (osd.2) 718 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: cluster 2021-02-10T04:33:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: 2021-02-10T04:33:28.492405+0000 2021-02-10T04:33:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: osd.2 (osd.2) 719 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: 2021-02-10T04:33:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: cluster 2021-02-10T04:33:28 2021-02-10T04:33:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: .492411+0000 osd.2 (osd.2) 720 : cluster 2021-02-10T04:33:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:33:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: :33:28.492417+0000 osd.2 (osd.2 2021-02-10T04:33:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: ) 721 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:30.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:33:30.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: 02-10T04:33:29.391378+0000 2021-02-10T04:33:30.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: mon.a (mon.0) 1173 : cluster [DBG] osdmap e392: 8 total, 8 up, 8 in 2021-02-10T04:33:30.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: 2021-02-10T04:33:30.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:33:30.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: T04:33:29.504154+0000 osd.2 (osd.2) 722 : cluster [WRN] 2021-02-10T04:33:30.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:30.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: cluster 2021-02-10T04:33: 2021-02-10T04:33:30.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: 29.504160+0000 osd.2 (osd.2) 723 : 2021-02-10T04:33:30.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:30.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:33:30.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: 02-10T04:33:29.504163+0000 osd.2 2021-02-10T04:33:30.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: (osd.2) 724 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:30.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: cluster 2021-02-10T04:33:30.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: 2021-02-10T04:33:29.504166+0000 osd.2 (osd 2021-02-10T04:33:30.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: .2) 725 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:30.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:33:30.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: -10T04:33:29.504173+0000 osd.2 ( 2021-02-10T04:33:30.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: osd.2) 726 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:30.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: cluster 2021-02-10T04:33:30.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: 2021-02-10T04:33:29.504177+0000 2021-02-10T04:33:30.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: osd.2 (osd.2) 727 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:30.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: 2021-02-10T04:33:30.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:33:30.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: 10T04:33:29.504186+0000 osd.2 (osd 2021-02-10T04:33:30.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: .2) 728 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:30.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:33:30.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: -10T04:33:29.504191+0000 osd.2 (osd. 2021-02-10T04:33:30.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: 2) 729 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:30.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:33:30.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: 02-10T04:33:29.570549+0000 2021-02-10T04:33:30.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: osd.4 (osd.4) 479 : cluster [WRN] 2021-02-10T04:33:30.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:30.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: cluster 2021-02-10T04:33 2021-02-10T04:33:30.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: :29.570557 2021-02-10T04:33:30.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: +0000 osd.4 (osd. 2021-02-10T04:33:30.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: 4) 480 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:30.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: cluster 2021-02-10T04:33:30.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: 2021-02-10 2021-02-10T04:33:30.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: T04:33:29.570564+0000 osd.4 (osd 2021-02-10T04:33:30.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: .4) 481 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:30.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: cluster 2021-02-10T04:33:30.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: 2021-02-10T04:33:29.570567+0000 2021-02-10T04:33:30.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: osd.4 (osd.4) 482 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:30.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: 2021-02-10T04:33:30.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: cluster 2021-02-10T04:33:30.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: 2021-02-10T04:33:30.072178 2021-02-10T04:33:30.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 521 : cluster [DBG] pgmap v738: 168 pgs: 1 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 161 active+clean; 306 MiB data, 923 MiB used, 714 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.1 MiB/s wr, 6 op/s 2021-02-10T04:33:30.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:33:30.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: 10T04:33:30.390684+0000 mon.a (mon.0) 1174 : 2021-02-10T04:33:30.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:30 smithi164 conmon[29288]: cluster [DBG] osdmap e393: 8 total, 8 up, 8 in 2021-02-10T04:33:30.949 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:33:30.964 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:33:30.965 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:33:31.129 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:33:31.168 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:31.445 INFO:tasks.rados.rados.0.smithi085.stdout:1464: done (11 left) 2021-02-10T04:33:31.445 INFO:tasks.rados.rados.0.smithi085.stdout:1465: write oid 16 current snap is 155 2021-02-10T04:33:31.445 INFO:tasks.rados.rados.0.smithi085.stdout:1459: finishing write tid 4 to smithi08575397-30 2021-02-10T04:33:31.445 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 681 (ObjNum 520 snap 155 seq_num 520) dirty exists 2021-02-10T04:33:31.445 INFO:tasks.rados.rados.0.smithi085.stdout:1459: left oid 30 (ObjNum 520 snap 155 seq_num 520) 2021-02-10T04:33:31.445 INFO:tasks.rados.rados.0.smithi085.stdout:1462: finishing copy_from to smithi08575397-23 2021-02-10T04:33:31.445 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 627 (ObjNum 85 snap 11 seq_num 85) dirty exists 2021-02-10T04:33:31.446 INFO:tasks.rados.rados.0.smithi085.stdout:1462: finishing copy_from racing read to smithi08575397-23 2021-02-10T04:33:31.446 INFO:tasks.rados.rados.0.smithi085.stdout:1463: finishing write tid 1 to smithi08575397-26 2021-02-10T04:33:31.446 INFO:tasks.rados.rados.0.smithi085.stdout:1463: finishing write tid 2 to smithi08575397-26 2021-02-10T04:33:31.446 INFO:tasks.rados.rados.0.smithi085.stdout:1465: seq_num 523 ranges {790061=467231,2004906=541501} 2021-02-10T04:33:31.452 INFO:tasks.rados.rados.0.smithi085.stdout:1465: writing smithi08575397-16 from 790061 to 1257292 tid 1 2021-02-10T04:33:31.461 INFO:tasks.rados.rados.0.smithi085.stdout:1465: writing smithi08575397-16 from 2004906 to 2546407 tid 2 2021-02-10T04:33:31.461 INFO:tasks.rados.rados.0.smithi085.stdout:1459: done (11 left) 2021-02-10T04:33:31.461 INFO:tasks.rados.rados.0.smithi085.stdout:1462: done (10 left) 2021-02-10T04:33:31.461 INFO:tasks.rados.rados.0.smithi085.stdout:1466: delete oid 37 current snap is 155 2021-02-10T04:33:31.461 INFO:tasks.rados.rados.0.smithi085.stdout:1463: finishing write tid 3 to smithi08575397-26 2021-02-10T04:33:31.461 INFO:tasks.rados.rados.0.smithi085.stdout:1463: finishing write tid 4 to smithi08575397-26 2021-02-10T04:33:31.462 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 131 (ObjNum 522 snap 155 seq_num 522) dirty exists 2021-02-10T04:33:31.462 INFO:tasks.rados.rados.0.smithi085.stdout:1463: left oid 26 (ObjNum 522 snap 155 seq_num 522) 2021-02-10T04:33:31.463 INFO:tasks.rados.rados.0.smithi085.stdout:1465: finishing write tid 1 to smithi08575397-16 2021-02-10T04:33:31.466 INFO:tasks.rados.rados.0.smithi085.stdout:1465: finishing write tid 2 to smithi08575397-16 2021-02-10T04:33:31.466 INFO:tasks.rados.rados.0.smithi085.stdout:1465: finishing write tid 3 to smithi08575397-16 2021-02-10T04:33:31.466 INFO:tasks.rados.rados.0.smithi085.stdout:1465: finishing write tid 4 to smithi08575397-16 2021-02-10T04:33:31.466 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 805 (ObjNum 523 snap 155 seq_num 523) dirty exists 2021-02-10T04:33:31.466 INFO:tasks.rados.rados.0.smithi085.stdout:1465: left oid 16 (ObjNum 523 snap 155 seq_num 523) 2021-02-10T04:33:31.467 INFO:tasks.rados.rados.0.smithi085.stdout:1463: done (10 left) 2021-02-10T04:33:31.467 INFO:tasks.rados.rados.0.smithi085.stdout:1465: done (9 left) 2021-02-10T04:33:31.467 INFO:tasks.rados.rados.0.smithi085.stdout:1466: done (8 left) 2021-02-10T04:33:31.467 INFO:tasks.rados.rados.0.smithi085.stdout:1467: write oid 28 current snap is 155 2021-02-10T04:33:31.467 INFO:tasks.rados.rados.0.smithi085.stdout:1467: seq_num 525 ranges {423675=734082,1721870=545170,2717777=105898} 2021-02-10T04:33:31.474 INFO:tasks.rados.rados.0.smithi085.stdout:1467: writing smithi08575397-28 from 423675 to 1157757 tid 1 2021-02-10T04:33:31.481 INFO:tasks.rados.rados.0.smithi085.stdout:1467: writing smithi08575397-28 from 1721870 to 2267040 tid 2 2021-02-10T04:33:31.484 INFO:tasks.rados.rados.0.smithi085.stdout:1467: writing smithi08575397-28 from 2717777 to 2823675 tid 3 2021-02-10T04:33:31.484 INFO:tasks.rados.rados.0.smithi085.stdout:1468: read oid 36 snap 122 2021-02-10T04:33:31.484 INFO:tasks.rados.rados.0.smithi085.stdout:1468: expect deleted 2021-02-10T04:33:31.484 INFO:tasks.rados.rados.0.smithi085.stdout:1469: delete oid 7 current snap is 155 2021-02-10T04:33:31.485 INFO:tasks.rados.rados.0.smithi085.stdout:1467: finishing write tid 1 to smithi08575397-28 2021-02-10T04:33:31.485 INFO:tasks.rados.rados.0.smithi085.stdout:1467: finishing write tid 2 to smithi08575397-28 2021-02-10T04:33:31.486 INFO:tasks.rados.rados.0.smithi085.stdout:1467: finishing write tid 3 to smithi08575397-28 2021-02-10T04:33:31.487 INFO:tasks.rados.rados.0.smithi085.stdout:1467: finishing write tid 4 to smithi08575397-28 2021-02-10T04:33:31.487 INFO:tasks.rados.rados.0.smithi085.stdout:1467: finishing write tid 5 to smithi08575397-28 2021-02-10T04:33:31.487 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 719 (ObjNum 525 snap 155 seq_num 525) dirty exists 2021-02-10T04:33:31.487 INFO:tasks.rados.rados.0.smithi085.stdout:1467: left oid 28 (ObjNum 525 snap 155 seq_num 525) 2021-02-10T04:33:31.489 INFO:tasks.rados.rados.0.smithi085.stdout:1467: done (10 left) 2021-02-10T04:33:31.490 INFO:tasks.rados.rados.0.smithi085.stdout:1468: done (9 left) 2021-02-10T04:33:31.490 INFO:tasks.rados.rados.0.smithi085.stdout:1469: done (8 left) 2021-02-10T04:33:31.490 INFO:tasks.rados.rados.0.smithi085.stdout:1470: write oid 22 current snap is 155 2021-02-10T04:33:31.490 INFO:tasks.rados.rados.0.smithi085.stdout:1470: seq_num 527 ranges {457289=484580,1438834=548839,2772623=84666} 2021-02-10T04:33:31.495 INFO:tasks.rados.rados.0.smithi085.stdout:1470: writing smithi08575397-22 from 457289 to 941869 tid 1 2021-02-10T04:33:31.502 INFO:tasks.rados.rados.0.smithi085.stdout:1470: writing smithi08575397-22 from 1438834 to 1987673 tid 2 2021-02-10T04:33:31.508 INFO:tasks.rados.rados.0.smithi085.stdout:1470: writing smithi08575397-22 from 2772623 to 2857289 tid 3 2021-02-10T04:33:31.508 INFO:tasks.rados.rados.0.smithi085.stdout:1470: finishing write tid 1 to smithi08575397-22 2021-02-10T04:33:31.508 INFO:tasks.rados.rados.0.smithi085.stdout:1471: read oid 26 snap -1 2021-02-10T04:33:31.508 INFO:tasks.rados.rados.0.smithi085.stdout:1471: expect (ObjNum 522 snap 155 seq_num 522) 2021-02-10T04:33:31.508 INFO:tasks.rados.rados.0.smithi085.stdout:1472: write oid 14 current snap is 155 2021-02-10T04:33:31.508 INFO:tasks.rados.rados.0.smithi085.stdout:1470: finishing write tid 2 to smithi08575397-22 2021-02-10T04:33:31.509 INFO:tasks.rados.rados.0.smithi085.stdout:1472: seq_num 528 ranges {474096=559829,1780963=408850,2800046=74050} 2021-02-10T04:33:31.514 INFO:tasks.rados.rados.0.smithi085.stdout:1472: writing smithi08575397-14 from 474096 to 1033925 tid 1 2021-02-10T04:33:31.521 INFO:tasks.rados.rados.0.smithi085.stdout:1472: writing smithi08575397-14 from 1780963 to 2189813 tid 2 2021-02-10T04:33:31.525 INFO:tasks.rados.rados.0.smithi085.stdout:1472: writing smithi08575397-14 from 2800046 to 2874096 tid 3 2021-02-10T04:33:31.526 INFO:tasks.rados.rados.0.smithi085.stdout:1473: delete oid 35 current snap is 155 2021-02-10T04:33:31.526 INFO:tasks.rados.rados.0.smithi085.stdout:1470: finishing write tid 3 to smithi08575397-22 2021-02-10T04:33:31.526 INFO:tasks.rados.rados.0.smithi085.stdout:1470: finishing write tid 4 to smithi08575397-22 2021-02-10T04:33:31.526 INFO:tasks.rados.rados.0.smithi085.stdout:1470: finishing write tid 5 to smithi08575397-22 2021-02-10T04:33:31.526 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 644 (ObjNum 527 snap 155 seq_num 527) dirty exists 2021-02-10T04:33:31.526 INFO:tasks.rados.rados.0.smithi085.stdout:1470: left oid 22 (ObjNum 527 snap 155 seq_num 527) 2021-02-10T04:33:31.527 INFO:tasks.rados.rados.0.smithi085.stdout:1471: expect (ObjNum 522 snap 155 seq_num 522) 2021-02-10T04:33:31.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[29319]: cluster 2021-02-10T04:33:30.530339+0000 2021-02-10T04:33:31.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[29319]: osd.2 (osd.2) 730 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:31.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[29319]: cluster 2021-02-10T04:33:30.530347+0000 osd.2 (osd.2) 731 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:31.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[29319]: cluster 2021-02-10T04:33:30.530352+0000 osd.2 (osd.2) 732 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:31.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[29319]: cluster 2021-02-10T04:33:30.530368+0000 osd.2 2021-02-10T04:33:31.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[29319]: (osd.2) 733 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:31.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[29319]: cluster 2021-02-10T04:33:30.530374+0000 osd.2 2021-02-10T04:33:31.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[29319]: (osd.2) 734 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:31.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:33:31.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[29319]: -10T04:33:30.530379+0000 osd.2 (osd.2) 735 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:31.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[29319]: cluster 2021-02-10T04:33 2021-02-10T04:33:31.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[29319]: :30.530385+0000 osd.2 (osd.2) 736 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:31.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[29319]: 2021-02-10T04:33:31.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[29319]: cluster 2021-02-10T04:33:30 2021-02-10T04:33:31.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[29319]: .530390+0000 osd.2 (osd.2) 737 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:31.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[29319]: cluster 2021-02-10T04:33:31.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[29319]: 2021-02-10T04:33:30.570018+0000 osd.4 (osd.4) 483 : cluster 2021-02-10T04:33:31.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:31.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:33:31.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[29319]: 04:33:30.570024+0000 osd.4 (osd. 2021-02-10T04:33:31.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[29319]: 4) 484 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:31.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:33:31.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[29319]: 02-10T04:33:30.570028+0000 2021-02-10T04:33:31.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[29319]: osd.4 (osd.4) 485 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:31.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[29319]: 2021-02-10T04:33:31.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:33:31.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[29319]: :33:30.570031+0000 osd.4 (osd.4) 2021-02-10T04:33:31.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[29319]: 486 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:31.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: cluster 2021-02-10T04:33:30.530339 2021-02-10T04:33:31.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 730 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:31.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:31.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: -02-10T04:33:30.530347+0000 osd.2 2021-02-10T04:33:31.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: (osd.2) 731 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:31.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:33:31.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: :33:30.530352+0000 osd.2 (osd.2) 732 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:31.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:33:31.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: 10T04:33:30.530368+0000 osd.2 (osd.2) 733 : 2021-02-10T04:33:31.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:31.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:31.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: 02-10T04:33:30.530374+0000 osd.2 (osd 2021-02-10T04:33:31.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: .2) 734 : cluster [WRN] 2021-02-10T04:33:31.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:31.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:31.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: 02-10T04:33:30.530379+0000 osd.2 (osd 2021-02-10T04:33:31.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: .2) 735 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:31.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: cluster 2021-02-10T04:33:31.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: 2021- 2021-02-10T04:33:31.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: 02-10T04:33:30.530385+0000 osd.2 (osd.2) 736 : cluster 2021-02-10T04:33:31.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:31.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:31.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: 02-10T04:33:30.530390+0000 osd.2 (osd.2 2021-02-10T04:33:31.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: ) 737 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:31.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:31.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: -10T04:33:30.570018+0000 osd.4 (osd.4 2021-02-10T04:33:31.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: ) 483 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:31.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:31.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: T04:33:30.570024+0000 2021-02-10T04:33:31.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: osd.4 (osd.4) 484 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:31.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: cluster 2021-02-10T04:33:31.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: 2021-02-10T04:33:30.570028+0000 2021-02-10T04:33:31.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: osd.4 (osd.4) 485 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:31.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: 2021-02-10T04:33:31.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: cluster 2021-02-10T04:33:31.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: 2021-02-10T04:33:30.570031+0000 osd.4 2021-02-10T04:33:31.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[35961]: (osd.4) 486 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:31.699 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:31 smithi085 conmon[49106]: debug 2021-02-10T04:33:31.560+0000 7f77a3e94700 -1 osd.2 394 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:33:31.719 INFO:tasks.rados.rados.0.smithi085.stdout:1472: finishing write tid 1 to smithi08575397-14 2021-02-10T04:33:31.719 INFO:tasks.rados.rados.0.smithi085.stdout:1472: finishing write tid 2 to smithi08575397-14 2021-02-10T04:33:31.719 INFO:tasks.rados.rados.0.smithi085.stdout:1472: finishing write tid 3 to smithi08575397-14 2021-02-10T04:33:31.719 INFO:tasks.rados.rados.0.smithi085.stdout:1472: finishing write tid 4 to smithi08575397-14 2021-02-10T04:33:31.719 INFO:tasks.rados.rados.0.smithi085.stdout:1470: done (11 left) 2021-02-10T04:33:31.719 INFO:tasks.rados.rados.0.smithi085.stdout:1471: done (10 left) 2021-02-10T04:33:31.719 INFO:tasks.rados.rados.0.smithi085.stdout:1473: done (9 left) 2021-02-10T04:33:31.719 INFO:tasks.rados.rados.0.smithi085.stdout:1474: snap_create 2021-02-10T04:33:31.719 INFO:tasks.rados.rados.0.smithi085.stdout:1472: finishing write tid 5 to smithi08575397-14 2021-02-10T04:33:31.719 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 634 (ObjNum 528 snap 155 seq_num 528) dirty exists 2021-02-10T04:33:31.719 INFO:tasks.rados.rados.0.smithi085.stdout:1472: left oid 14 (ObjNum 528 snap 155 seq_num 528) 2021-02-10T04:33:31.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:31 smithi164 conmon[33677]: debug 2021-02-10T04:33:31.585+0000 7fbec19a1700 -1 osd.4 394 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:33:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:31 smithi164 conmon[29288]: cluster 2021-02-10T04:33:30.530339+0000 osd.2 (osd 2021-02-10T04:33:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:31 smithi164 conmon[29288]: .2) 730 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:31 smithi164 conmon[29288]: cluster 2021-02-10T04:33:30.530347+0000 osd.2 (osd.2) 731 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:31 smithi164 conmon[29288]: cluster 2021-02-10T04:33:30.530352+0000 osd.2 (osd.2) 732 : cluster 2021-02-10T04:33:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:31 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:31 smithi164 conmon[29288]: cluster 2021-02-10T04:33:30. 2021-02-10T04:33:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:31 smithi164 conmon[29288]: 530368+0000 osd.2 (osd.2) 733 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:31 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:33:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:31 smithi164 conmon[29288]: -10T04:33:30.530374+0000 osd.2 (osd.2 2021-02-10T04:33:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:31 smithi164 conmon[29288]: ) 734 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:31 smithi164 conmon[29288]: cluster 2021-02-10T04:33:30 2021-02-10T04:33:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:31 smithi164 conmon[29288]: .530379+0000 osd.2 (osd.2) 735 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:31 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:33:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:31 smithi164 conmon[29288]: 10T04:33:30.530385+0000 osd.2 (osd.2) 2021-02-10T04:33:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:31 smithi164 conmon[29288]: 736 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:31 smithi164 conmon[29288]: cluster 2021-02-10T04:33:30.530390 2021-02-10T04:33:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:31 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 737 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:31 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:33:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:31 smithi164 conmon[29288]: T04:33:30.570018+0000 osd.4 (osd.4) 483 : 2021-02-10T04:33:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:31 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:31 smithi164 conmon[29288]: cluster 2021-02-10T04:33:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:31 smithi164 conmon[29288]: 2021-02-10T04:33:30.570024+0000 osd.4 (osd.4) 484 2021-02-10T04:33:31.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:31 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:31.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:31 smithi164 conmon[29288]: cluster 2021-02-10T04:33:31.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:31 smithi164 conmon[29288]: 2021-02-10T04:33:30.570028+0000 osd.4 (osd 2021-02-10T04:33:31.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:31 smithi164 conmon[29288]: .4) 485 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:31.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:31 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:33:31.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:31 smithi164 conmon[29288]: 33:30.570031+0000 osd.4 (osd.4) 486 : cluster [WRN] 2021-02-10T04:33:31.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:31 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:32.407 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:33:32.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: cluster 2021-02-10T04:33:32.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:33:32.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: :33:31.442192+0000 mon.a (mon.0) 1175 : cluster [DBG] osdmap e394: 8 total, 8 up, 8 in 2021-02-10T04:33:32.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: cluster 2021-02-10T04:33:32.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:33:32.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: :33:31.560620+0000 osd.2 (osd.2) 738 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:32.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:33:32.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: :33:31.560628+0000 osd.2 (osd.2) 739 : cluster [WRN] 2021-02-10T04:33:32.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:32.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: cluster 2021-02-10T04:33:32.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: 2021-02-10T04:33:31. 2021-02-10T04:33:32.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: 560643+0000 osd.2 (osd.2) 740 : cluster 2021-02-10T04:33:32.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:32.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: cluster 2021-02-10T04:33: 2021-02-10T04:33:32.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: 31.560649+0000 osd.2 (osd.2) 741 2021-02-10T04:33:32.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:32.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:33:32.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: 02-10T04:33:31.560657+0000 2021-02-10T04:33:32.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: osd.2 (osd.2) 742 : cluster 2021-02-10T04:33:32.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:32.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:33:32.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: -10T04:33:31.560663+0000 osd.2 2021-02-10T04:33:32.446 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[52836]: debug 2021-02-10T04:33:32.049+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:32.454 INFO:tasks.rados.rados.0.smithi085.stdout:1472: done (9 left) 2021-02-10T04:33:32.454 INFO:tasks.rados.rados.0.smithi085.stdout:1474: done (8 left) 2021-02-10T04:33:32.454 INFO:tasks.rados.rados.0.smithi085.stdout:1475: read oid 48 snap 147 2021-02-10T04:33:32.454 INFO:tasks.rados.rados.0.smithi085.stdout:1475: expect (ObjNum 476 snap 135 seq_num 476) 2021-02-10T04:33:32.454 INFO:tasks.rados.rados.0.smithi085.stdout:1476: write oid 38 current snap is 156 2021-02-10T04:33:32.454 INFO:tasks.rados.rados.0.smithi085.stdout:1476: seq_num 530 ranges {507710=710327,1897927=412519,2854892=52818} 2021-02-10T04:33:32.462 INFO:tasks.rados.rados.0.smithi085.stdout:1476: writing smithi08575397-38 from 507710 to 1218037 tid 1 2021-02-10T04:33:32.468 INFO:tasks.rados.rados.0.smithi085.stdout:1476: writing smithi08575397-38 from 1897927 to 2310446 tid 2 2021-02-10T04:33:32.472 INFO:tasks.rados.rados.0.smithi085.stdout:1476: writing smithi08575397-38 from 2854892 to 2907710 tid 3 2021-02-10T04:33:32.472 INFO:tasks.rados.rados.0.smithi085.stdout:1477: copy_from oid 7 from oid 32 current snap is 156 2021-02-10T04:33:32.472 INFO:tasks.rados.rados.0.smithi085.stdout:1475: expect (ObjNum 476 snap 135 seq_num 476) 2021-02-10T04:33:32.682 INFO:tasks.rados.rados.0.smithi085.stdout:1476: finishing write tid 1 to smithi08575397-38 2021-02-10T04:33:32.682 INFO:tasks.rados.rados.0.smithi085.stdout:1475: done (10 left) 2021-02-10T04:33:32.682 INFO:tasks.rados.rados.0.smithi085.stdout:1478: read oid 48 snap -1 2021-02-10T04:33:32.682 INFO:tasks.rados.rados.0.smithi085.stdout:1478: expect (ObjNum 476 snap 135 seq_num 476) 2021-02-10T04:33:32.682 INFO:tasks.rados.rados.0.smithi085.stdout:1479: read oid 21 snap -1 2021-02-10T04:33:32.682 INFO:tasks.rados.rados.0.smithi085.stdout:1479: expect (ObjNum 431 snap 114 seq_num 431) 2021-02-10T04:33:32.682 INFO:tasks.rados.rados.0.smithi085.stdout:1480: rollback oid 1 current snap is 156 2021-02-10T04:33:32.682 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 1 to 136 2021-02-10T04:33:32.682 INFO:tasks.rados.rados.0.smithi085.stdout:1476: finishing write tid 2 to smithi08575397-38 2021-02-10T04:33:32.682 INFO:tasks.rados.rados.0.smithi085.stdout:1476: finishing write tid 3 to smithi08575397-38 2021-02-10T04:33:32.683 INFO:tasks.rados.rados.0.smithi085.stdout:1476: finishing write tid 4 to smithi08575397-38 2021-02-10T04:33:32.683 INFO:tasks.rados.rados.0.smithi085.stdout:1476: finishing write tid 5 to smithi08575397-38 2021-02-10T04:33:32.683 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 685 (ObjNum 530 snap 156 seq_num 530) dirty exists 2021-02-10T04:33:32.683 INFO:tasks.rados.rados.0.smithi085.stdout:1476: left oid 38 (ObjNum 530 snap 156 seq_num 530) 2021-02-10T04:33:32.683 INFO:tasks.rados.rados.0.smithi085.stdout:1477: finishing copy_from to smithi08575397-7 2021-02-10T04:33:32.683 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 110 (ObjNum 204 snap 44 seq_num 204) dirty exists 2021-02-10T04:33:32.683 INFO:tasks.rados.rados.0.smithi085.stdout:1477: finishing copy_from racing read to smithi08575397-7 2021-02-10T04:33:32.683 INFO:tasks.rados.rados.0.smithi085.stdout:1476: done (12 left) 2021-02-10T04:33:32.683 INFO:tasks.rados.rados.0.smithi085.stdout:1477: done (11 left) 2021-02-10T04:33:32.683 INFO:tasks.rados.rados.0.smithi085.stdout:1481: write oid 11 current snap is 156 2021-02-10T04:33:32.683 INFO:tasks.rados.rados.0.smithi085.stdout:1481: seq_num 531 ranges {524517=785576,1956409=556177} 2021-02-10T04:33:32.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[49106]: debug 2021-02-10T04:33:32.596+0000 7f77a3e94700 -1 osd.2 395 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:33:32.691 INFO:tasks.rados.rados.0.smithi085.stdout:1481: writing smithi08575397-11 from 524517 to 1310093 tid 1 2021-02-10T04:33:32.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: (osd.2) 743 : cluster [WRN] 2021-02-10T04:33:32.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:32.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: cluster 2021-02-10T04:33:32.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: 2021-02-10T04:33:31.560668+0000 2021-02-10T04:33:32.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: osd.2 (osd.2) 744 : cluster 2021-02-10T04:33:32.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:32.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:32.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: -02-10T04:33:31.560678+0000 2021-02-10T04:33:32.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: osd.2 (osd.2) 745 : cluster 2021-02-10T04:33:32.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:32.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: cluster 2021-02-10T04:33: 2021-02-10T04:33:32.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: 31.586232+0000 osd.4 (osd.4) 487 : 2021-02-10T04:33:32.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:32.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:32.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: -02-10T04:33:31.586251+0000 2021-02-10T04:33:32.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: osd.4 (osd.4) 488 : cluster 2021-02-10T04:33:32.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:32.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: cluster 2021-02-10T04:33 2021-02-10T04:33:32.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: :31.586255+0000 osd.4 (osd.4) 489 : 2021-02-10T04:33:32.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:32.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:33:32.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: 10T04:33:31.586259+0000 osd.4 2021-02-10T04:33:32.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: (osd.4) 490 : cluster 2021-02-10T04:33:32.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:32.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:33:32.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: -10T04:33:32.072729+0000 mgr.y (mgr.14140) 2021-02-10T04:33:32.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[29319]: 522 : cluster [DBG] pgmap v741: 168 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 162 active+clean; 304 MiB data, 934 MiB used, 714 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.4 MiB/s wr, 9 op/s 2021-02-10T04:33:32.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: cluster 2021-02-10T04:33:32.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:33:32.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 04:33:31.442192+0000 mon.a (mon.0) 1175 : cluster [DBG] osdmap e394: 8 total, 8 up, 8 in 2021-02-10T04:33:32.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 2021-02-10T04:33:32.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: cluster 2021-02-10T04:33:32.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 2021-02-10T04:33:32.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 2021 2021-02-10T04:33:32.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: - 2021-02-10T04:33:32.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 02 2021-02-10T04:33:32.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: - 2021-02-10T04:33:32.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 10 2021-02-10T04:33:32.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: T 2021-02-10T04:33:32.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 04 2021-02-10T04:33:32.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: : 2021-02-10T04:33:32.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 33 2021-02-10T04:33:32.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: : 2021-02-10T04:33:32.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 31 2021-02-10T04:33:32.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: . 2021-02-10T04:33:32.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 560620 2021-02-10T04:33:32.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: +0000 2021-02-10T04:33:32.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 2021-02-10T04:33:32.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: osd.2 2021-02-10T04:33:32.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: ( 2021-02-10T04:33:32.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: osd 2021-02-10T04:33:32.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: . 2021-02-10T04:33:32.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 2 2021-02-10T04:33:32.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: ) 2021-02-10T04:33:32.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 738 2021-02-10T04:33:32.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: : 2021-02-10T04:33:32.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: cluster 2021-02-10T04:33:32.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 2021-02-10T04:33:32.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: [WRN] 2021-02-10T04:33:32.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 2021-02-10T04:33:32.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:32.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 2021-02-10T04:33:32.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: cluster 2021-02-10T04:33:32.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 2021-02-10T04:33:32.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 2021 2021-02-10T04:33:32.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: - 2021-02-10T04:33:32.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 02 2021-02-10T04:33:32.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: - 2021-02-10T04:33:32.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 10 2021-02-10T04:33:32.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: T 2021-02-10T04:33:32.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 04:33:31.560628+0000 osd.2 (osd.2) 739 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:32.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: cluster 2021-02-10T04:33:32.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 2021-02-10T04:33:31.560643 2021-02-10T04:33:32.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 740 : cluster [WRN] 2021-02-10T04:33:32.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:32.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: cluster 2021-02-10T04:33:32.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 2021-02-10T04:33:32.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 2021 2021-02-10T04:33:32.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: - 2021-02-10T04:33:32.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 02 2021-02-10T04:33:32.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: -10T04:33:31. 2021-02-10T04:33:32.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 560649+0000 osd.2 (osd.2) 741 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:32.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:32.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: -02-10T04:33: 2021-02-10T04:33:32.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 31.560657+0000 osd.2 (osd 2021-02-10T04:33:32.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: .2) 742 : cluster [WRN] 2021-02-10T04:33:32.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:32.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: cluster 2021-02-10T04:33:32.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:33:32.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 10T04:33:31.560663+0000 osd.2 (osd 2021-02-10T04:33:32.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: .2) 743 : cluster [WRN] 2021-02-10T04:33:32.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:32.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:33:32.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 33:31.560668+0000 osd.2 (osd.2) 744 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:32.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:32.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 02-10T04:33:31.560678+0000 osd.2 (osd.2 2021-02-10T04:33:32.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: ) 745 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:32.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:33:32.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 33:31.586232+0000 osd.4 (osd.4) 487 : cluster 2021-02-10T04:33:32.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:32.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: cluster 2021-02-10T04:33:32.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 2021-02-10T04:33:31.586251 2021-02-10T04:33:32.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 488 : cluster [WRN] 2021-02-10T04:33:32.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:32.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: cluster 2021-02-10T04:33:31 2021-02-10T04:33:32.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: .586255+0000 osd.4 (osd.4) 489 : cluster 2021-02-10T04:33:32.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:32.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: cluster 2021-02-10T04:33:32.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:33:32.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 33:31.586259+0000 osd.4 (osd.4) 490 : cluster 2021-02-10T04:33:32.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:32.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: cluster 2021-02-10T04:33:32.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 2021-02-10T04:33:32.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 2021 2021-02-10T04:33:32.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: - 2021-02-10T04:33:32.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 02 2021-02-10T04:33:32.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: -10T04:33:32. 2021-02-10T04:33:32.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:32 smithi085 conmon[35961]: 072729+0000 mgr.y (mgr.14140) 522 : cluster [DBG] pgmap v741: 168 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 162 active+clean; 304 MiB data, 934 MiB used, 714 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.4 MiB/s wr, 9 op/s 2021-02-10T04:33:32.701 INFO:tasks.rados.rados.0.smithi085.stdout:1481: writing smithi08575397-11 from 1956409 to 2512586 tid 2 2021-02-10T04:33:32.701 INFO:tasks.rados.rados.0.smithi085.stdout:1482: copy_from oid 32 from oid 9 current snap is 156 2021-02-10T04:33:32.701 INFO:tasks.rados.rados.0.smithi085.stdout:1478: expect (ObjNum 476 snap 135 seq_num 476) 2021-02-10T04:33:32.881 INFO:tasks.rados.rados.0.smithi085.stdout:1479: expect (ObjNum 431 snap 114 seq_num 431) 2021-02-10T04:33:32.893 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[33677]: debug 2021-02-10T04:33:32.632+0000 7fbec19a1700 -1 osd.4 395 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:33:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: cluster 2021-02-10T04:33: 2021-02-10T04:33:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: 31.442192+0000 mon.a (mon.0) 1175 : cluster [DBG] osdmap e394: 8 total, 8 up, 8 in 2021-02-10T04:33:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: cluster 2021-02-10T04:33:31.560620+0000 2021-02-10T04:33:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: osd.2 (osd.2) 738 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: cluster 2021-02-10T04:33:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: 2021-02-10T04:33:31. 2021-02-10T04:33:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: 560628+0000 osd.2 (osd.2) 739 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:33:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: -10T04:33:31.560643+0000 osd.2 (osd.2) 2021-02-10T04:33:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: 740 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: cluster 2021-02-10T04:33:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: 2021-02-10T04:33: 2021-02-10T04:33:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: 31.560649+0000 osd.2 (osd. 2021-02-10T04:33:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: 2) 741 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: 2021-02-10T04:33:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:33:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: -10T04:33:31.560657 2021-02-10T04:33:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 742 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:33:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: T04:33:31.560663+0000 osd.2 (osd.2) 743 2021-02-10T04:33:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:33:32.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: T04:33:31.560668+0000 osd.2 (osd 2021-02-10T04:33:32.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: .2) 744 : cluster 2021-02-10T04:33:32.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:32.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:33:32.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: -10T04:33:31.560678+0000 osd.2 ( 2021-02-10T04:33:32.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: osd.2) 745 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:32.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:33:32.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: -10T04:33:31.586232+0000 osd.4 (osd.4) 2021-02-10T04:33:32.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: 487 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:32.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: cluster 2021-02-10T04:33:31 2021-02-10T04:33:32.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: .586251+0000 osd.4 (osd.4) 488 : 2021-02-10T04:33:32.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:32.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:33:32.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: 04:33:31.586255+0000 2021-02-10T04:33:32.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: osd.4 (osd.4) 489 : 2021-02-10T04:33:32.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:32.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:33:32.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: :33:31.586259+0000 osd.4 (osd.4) 490 : 2021-02-10T04:33:32.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:32.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:33:32.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: 02-10T04:33:32.072729+0000 2021-02-10T04:33:32.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: mgr.y (mgr.14140) 522 : cluster 2021-02-10T04:33:32.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:32 smithi164 conmon[29288]: [DBG] pgmap v741: 168 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 162 active+clean; 304 MiB data, 934 MiB used, 714 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.4 MiB/s wr, 9 op/s 2021-02-10T04:33:33.085 INFO:tasks.rados.rados.0.smithi085.stdout:1480: finishing rollback tid 0 to smithi08575397-1 2021-02-10T04:33:33.086 INFO:tasks.rados.rados.0.smithi085.stdout:1480: finishing rollback tid 1 to smithi08575397-1 2021-02-10T04:33:33.086 INFO:tasks.rados.rados.0.smithi085.stdout:1478: done (12 left) 2021-02-10T04:33:33.086 INFO:tasks.rados.rados.0.smithi085.stdout:1479: done (11 left) 2021-02-10T04:33:33.086 INFO:tasks.rados.rados.0.smithi085.stdout:1483: delete oid 26 current snap is 156 2021-02-10T04:33:33.086 INFO:tasks.rados.rados.0.smithi085.stdout:1480: finishing rollback tid 2 to smithi08575397-1 2021-02-10T04:33:33.086 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 602 (ObjNum 424 snap 109 seq_num 424) dirty exists 2021-02-10T04:33:33.086 INFO:tasks.rados.rados.0.smithi085.stdout:1481: finishing write tid 1 to smithi08575397-11 2021-02-10T04:33:33.086 INFO:tasks.rados.rados.0.smithi085.stdout:1481: finishing write tid 2 to smithi08575397-11 2021-02-10T04:33:33.086 INFO:tasks.rados.rados.0.smithi085.stdout:1481: finishing write tid 3 to smithi08575397-11 2021-02-10T04:33:33.086 INFO:tasks.rados.rados.0.smithi085.stdout:1481: finishing write tid 4 to smithi08575397-11 2021-02-10T04:33:33.086 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 810 (ObjNum 531 snap 156 seq_num 531) dirty exists 2021-02-10T04:33:33.087 INFO:tasks.rados.rados.0.smithi085.stdout:1481: left oid 11 (ObjNum 531 snap 156 seq_num 531) 2021-02-10T04:33:33.088 INFO:tasks.rados.rados.0.smithi085.stdout:1480: done (11 left) 2021-02-10T04:33:33.088 INFO:tasks.rados.rados.0.smithi085.stdout:1481: done (10 left) 2021-02-10T04:33:33.088 INFO:tasks.rados.rados.0.smithi085.stdout:1483: done (9 left) 2021-02-10T04:33:33.088 INFO:tasks.rados.rados.0.smithi085.stdout:1484: snap_create 2021-02-10T04:33:33.236 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:33:33.268 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:33.461 INFO:tasks.rados.rados.0.smithi085.stdout:1484: done (9 left) 2021-02-10T04:33:33.461 INFO:tasks.rados.rados.0.smithi085.stdout:1485: rollback oid 35 current snap is 157 2021-02-10T04:33:33.461 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 35 to 149 2021-02-10T04:33:33.461 INFO:tasks.rados.rados.0.smithi085.stdout:1486: snap_create 2021-02-10T04:33:33.468 INFO:tasks.rados.rados.0.smithi085.stdout:1485: finishing rollback tid 1 to smithi08575397-35 2021-02-10T04:33:33.469 INFO:tasks.rados.rados.0.smithi085.stdout:1485: finishing rollback tid 2 to smithi08575397-35 2021-02-10T04:33:33.469 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 35 v 616 (ObjNum 85 snap 11 seq_num 85) dirty exists 2021-02-10T04:33:33.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[33677]: debug 2021-02-10T04:33:33.359+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: cluster 2021-02-10T04:33:32.453205+0000 mon.a 2021-02-10T04:33:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: (mon.0) 1176 : cluster [DBG] osdmap e395: 8 total, 8 up, 8 in 2021-02-10T04:33:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: cluster 2021-02-10T04:33:32.482833+0000 osd.0 (osd.0) 62 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:33:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: 2021-02-10T04:33:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: cluster 2021-02-10T04:33:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: 2021 2021-02-10T04:33:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: - 2021-02-10T04:33:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: 02 2021-02-10T04:33:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: - 2021-02-10T04:33:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: 10 2021-02-10T04:33:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: T 2021-02-10T04:33:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: 04 2021-02-10T04:33:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: : 2021-02-10T04:33:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: 33 2021-02-10T04:33:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: : 2021-02-10T04:33:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: 32.483163+0000 osd.0 2021-02-10T04:33:33.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: (osd.0) 63 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:33:33.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: cluster 2021-02-10T04:33:33.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: 2021-02-10T04:33: 2021-02-10T04:33:33.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: 32.596853+0000 osd.2 (osd.2) 746 : 2021-02-10T04:33:33.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:33.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:33:33.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: -10T04:33:32.596863+0000 osd.2 ( 2021-02-10T04:33:33.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: osd.2) 747 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:33.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:33:33.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: 02-10T04:33:32.596875+0000 osd.2 ( 2021-02-10T04:33:33.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: osd.2) 748 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:33.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:33:33.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: 02-10T04:33:32.596882+0000 osd.2 2021-02-10T04:33:33.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: (osd.2) 749 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:33.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: 2021-02-10T04:33:33.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: cluster 2021-02-10T04:33:33.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: 2021-02-10T04:33:32.596889+0000 osd.2 (osd. 2021-02-10T04:33:33.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: 2) 750 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:33.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: cluster 2021-02-10T04:33:33.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: 2021-02-10T04:33:32.596896+0000 osd.2 (osd.2) 2021-02-10T04:33:33.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: 751 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:33.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: cluster 2021-02-10T04:33:32.596901+0000 osd.2 (osd.2 2021-02-10T04:33:33.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: ) 752 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:33.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:33:33.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: 04:33:32.596907+0000 osd.2 (osd.2 2021-02-10T04:33:33.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: ) 753 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:33.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:33:33.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: 10T04:33:32.633483+0000 2021-02-10T04:33:33.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: osd.4 (osd.4 2021-02-10T04:33:33.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: ) 491 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:33.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:33:33.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: 10T04:33:32.633492+0000 osd.4 (osd 2021-02-10T04:33:33.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: .4) 492 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:33.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:33:33.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: 10T04:33:32.633498+0000 osd.4 (osd 2021-02-10T04:33:33.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: .4) 493 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:33.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:33:33.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: -02-10T04: 2021-02-10T04:33:33.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: 33:32.633504+0000 osd.4 (osd.4) 2021-02-10T04:33:33.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: 494 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:33.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:33:33.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: 02-10T04:33:33.116844+0000 mon.a 2021-02-10T04:33:33.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: (mon.0) 1177 : cluster [WRN] Health check update: 12 slow ops, oldest one blocked for 242 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:33:33.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: 2021-02-10T04:33:33.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: audit 2021-02-10T04:33:33.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: 2021-02-10T04:33:33.258497+0000 mon.a (mon 2021-02-10T04:33:33.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:33 smithi164 conmon[29288]: .0) 1178 : audit [DBG] from='client.? 172.21.15.85:0/2146983925' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:33:33.697 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:33:33.722 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: cluster 2021-02-10T04:33:32 2021-02-10T04:33:33.722 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: .453205+0000 mon.a (mon.0) 1176 : cluster [DBG] osdmap e395: 8 total, 8 up, 8 in 2021-02-10T04:33:33.722 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: cluster 2021-02-10T04:33:32.482833+0000 osd.0 (osd.0) 62 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:33:33.722 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: cluster 2021-02-10T04:33:32.483163+0000 osd.0 (osd.0) 63 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:33:33.722 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:33:33.722 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: :33:32.596853+0000 osd.2 (osd.2) 746 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:33.722 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: cluster 2021-02-10T04:33:33.723 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: 2021-02-10T04:33:32.596863+0000 2021-02-10T04:33:33.723 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: osd.2 (osd.2) 747 : cluster 2021-02-10T04:33:33.723 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:33.723 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: cluster 2021-02-10T04:33:33.723 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:33:33.723 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: 33:32.596875+0000 osd.2 2021-02-10T04:33:33.723 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: (osd.2) 748 : cluster 2021-02-10T04:33:33.723 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:33.723 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: cluster 2021-02-10T04:33:33.723 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: 2021-02-10T04:33:32.596882 2021-02-10T04:33:33.723 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 749 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:33.723 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:33:33.724 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: T04:33:32.596889+0000 osd.2 (osd.2) 750 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:33.724 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: cluster 2021-02-10T04:33:32. 2021-02-10T04:33:33.724 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: 596896+0000 osd.2 (osd.2) 751 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:33.724 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:33:33.724 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: 04:33:32.596901+0000 osd.2 (osd.2) 752 : cluster 2021-02-10T04:33:33.724 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:33.724 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: cluster 2021-02-10T04:33:33.724 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: 2021-02-10T04:33:32.596907+0000 osd.2 (osd.2) 753 : cluster [WRN] 2021-02-10T04:33:33.724 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:33.724 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:33:33.724 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: T04:33:32.633483+0000 osd.4 (osd.4) 491 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:33.724 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: 2021-02-10T04:33:33.724 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:33:33.725 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: :33:32.633492+0000 osd.4 (osd.4) 492 : cluster 2021-02-10T04:33:33.725 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:33.725 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:33:33.725 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: :33:32.633498+0000 osd.4 (osd.4) 493 2021-02-10T04:33:33.725 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:33.725 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:33:33.725 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: -10T04:33:32.633504+0000 2021-02-10T04:33:33.725 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: osd.4 (osd.4) 494 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:33.725 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:33:33.725 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: 02-10T04:33:33.116844 2021-02-10T04:33:33.725 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1177 : cluster 2021-02-10T04:33:33.725 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: [WRN] Health check update: 12 slow ops, oldest one blocked for 242 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:33:33.725 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:33:33.726 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: 10T04:33:33.258497+0000 2021-02-10T04:33:33.726 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: mon.a (mon.0) 1178 : audit [DBG] 2021-02-10T04:33:33.726 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[29319]: from='client.? 172.21.15.85:0/2146983925' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:33:33.727 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: cluster 2021-02-10T04:33:32.453205 2021-02-10T04:33:33.727 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1176 : cluster [DBG] osdmap e395: 8 total, 8 up, 8 in 2021-02-10T04:33:33.727 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:33.727 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: -10T04:33:32.482833+0000 osd.0 (osd.0) 62 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:33:33.727 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: cluster 2021-02-10T04:33:33.727 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: 2021-02-10T04:33:32.483163 2021-02-10T04:33:33.727 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: +0000 osd.0 (osd.0) 63 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:33:33.728 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: 2021-02-10T04:33:33.728 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:33.728 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: T04:33:32.596853+0000 osd.2 (osd.2 2021-02-10T04:33:33.728 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: ) 746 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:33:33.728 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:33:33.728 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: 04:33:32.596863+0000 osd.2 (osd 2021-02-10T04:33:33.728 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: .2) 747 : cluster [WRN] 2021-02-10T04:33:33.728 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:33:33.728 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: cluster 2021-02-10T04:33:33.728 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:33:33.728 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: 04:33:32.596875+0000 osd.2 (osd 2021-02-10T04:33:33.728 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: .2) 748 : cluster [WRN] 2021-02-10T04:33:33.728 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:33:33.729 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: cluster 2021-02-10T04:33:33.729 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: 2021-02-10T04:33:33.729 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: 2021 2021-02-10T04:33:33.729 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: - 2021-02-10T04:33:33.729 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: 02 2021-02-10T04:33:33.729 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: - 2021-02-10T04:33:33.729 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: 10 2021-02-10T04:33:33.729 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: T 2021-02-10T04:33:33.729 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: 04 2021-02-10T04:33:33.729 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: : 2021-02-10T04:33:33.729 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: 33 2021-02-10T04:33:33.729 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: : 2021-02-10T04:33:33.730 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: 32 2021-02-10T04:33:33.730 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: . 2021-02-10T04:33:33.730 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: 596882 2021-02-10T04:33:33.730 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: +0000 2021-02-10T04:33:33.730 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: 2021-02-10T04:33:33.730 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: osd.2 2021-02-10T04:33:33.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: ( 2021-02-10T04:33:33.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: osd 2021-02-10T04:33:33.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: . 2021-02-10T04:33:33.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: 2) 749 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:33:33.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: 2021-02-10T04:33:33.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:33.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: -10T04:33:32.596889+0000 osd.2 2021-02-10T04:33:33.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: (osd.2) 750 : cluster 2021-02-10T04:33:33.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:33:33.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:33.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: 02-10T04:33:32.596896+0000 2021-02-10T04:33:33.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: osd.2 (osd.2) 751 : cluster 2021-02-10T04:33:33.732 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:33:33.732 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: cluster 2021-02-10T04:33:33.732 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: 2021-02-10T04:33:33.732 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: 2021 2021-02-10T04:33:33.732 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: - 2021-02-10T04:33:33.732 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: 02 2021-02-10T04:33:33.732 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: - 2021-02-10T04:33:33.732 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: 10T04:33:32. 2021-02-10T04:33:33.732 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: 596901+0000 osd.2 (osd.2) 752 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:33:33.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: cluster 2021-02-10T04:33:33.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:33:33.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: T04:33:32.596907+0000 osd.2 (osd.2) 753 : cluster 2021-02-10T04:33:33.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:33:33.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:33.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: 02-10T04:33:32.633483+0000 2021-02-10T04:33:33.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: osd.4 (osd.4) 491 : cluster 2021-02-10T04:33:33.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:33:33.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: cluster 2021-02-10T04:33:32 2021-02-10T04:33:33.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: .633492+0000 osd.4 (osd.4) 492 : cluster 2021-02-10T04:33:33.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:33:33.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:33.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: 02-10T04:33:32.633498+0000 osd.4 (osd 2021-02-10T04:33:33.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: .4) 493 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:33:33.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:33:33.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: 10T04:33:32.633504+0000 osd.4 (osd 2021-02-10T04:33:33.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: .4) 494 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:33:33.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: cluster 2021-02-10T04:33:33.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: 2021-02-10T04:33:33 2021-02-10T04:33:33.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: .116844+0000 mon.a (mon.0) 1177 : cluster [WRN] 2021-02-10T04:33:33.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: Health check update: 12 slow ops, oldest one blocked for 242 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:33:33.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:33:33.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: 33:33.258497+0000 mon.a (mon.0) 2021-02-10T04:33:33.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:33 smithi085 conmon[35961]: 1178 : audit [DBG] from='client.? 172.21.15.85:0/2146983925' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:33:33.802 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:33:33.843 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:33.865 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:33:33.865 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:33:33.865 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:33:33.865 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:33:33.865 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:33:33.865 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:33:33.865 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:33:33.865 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:33:33.865 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:33:33.866 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:33:33.866 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:33:33.866 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:33:33.866 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:33:33.866 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:33:33.866 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:33:33.866 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:33:34.337 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:34 smithi085 conmon[49106]: debug 2021-02-10T04:33:34.192+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:34.474 INFO:tasks.rados.rados.0.smithi085.stdout:1485: done (10 left) 2021-02-10T04:33:34.474 INFO:tasks.rados.rados.0.smithi085.stdout:1486: done (9 left) 2021-02-10T04:33:34.474 INFO:tasks.rados.rados.0.smithi085.stdout:1487: read oid 8 snap -1 2021-02-10T04:33:34.474 INFO:tasks.rados.rados.0.smithi085.stdout:1487: expect (ObjNum 521 snap 155 seq_num 521) 2021-02-10T04:33:34.474 INFO:tasks.rados.rados.0.smithi085.stdout:1488: write (excl) oid 3 current snap is 158 2021-02-10T04:33:34.474 INFO:tasks.rados.rados.0.smithi085.stdout:1488: seq_num 533 ranges {558131=652427,1673373=559846,2820808=137323} 2021-02-10T04:33:34.475 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.0 to 1 (10000) 2021-02-10T04:33:34.480 INFO:tasks.rados.rados.0.smithi085.stdout:1488: writing smithi08575397-3 from 558131 to 1210558 tid 1 2021-02-10T04:33:34.486 INFO:tasks.rados.rados.0.smithi085.stdout:1488: writing smithi08575397-3 from 1673373 to 2233219 tid 2 2021-02-10T04:33:34.491 INFO:tasks.rados.rados.0.smithi085.stdout:1488: writing smithi08575397-3 from 2820808 to 2958131 tid 3 2021-02-10T04:33:34.491 INFO:tasks.rados.rados.0.smithi085.stdout:1489: delete oid 38 current snap is 158 2021-02-10T04:33:34.491 INFO:tasks.rados.rados.0.smithi085.stdout:1487: expect (ObjNum 521 snap 155 seq_num 521) 2021-02-10T04:33:34.603 INFO:tasks.rados.rados.0.smithi085.stdout:1487: done (11 left) 2021-02-10T04:33:34.603 INFO:tasks.rados.rados.0.smithi085.stdout:1489: done (10 left) 2021-02-10T04:33:34.604 INFO:tasks.rados.rados.0.smithi085.stdout:1490: read oid 48 snap 157 2021-02-10T04:33:34.604 INFO:tasks.rados.rados.0.smithi085.stdout:1490: expect (ObjNum 476 snap 135 seq_num 476) 2021-02-10T04:33:34.604 INFO:tasks.rados.rados.0.smithi085.stdout:1491: delete oid 36 current snap is 158 2021-02-10T04:33:34.604 INFO:tasks.rados.rados.0.smithi085.stdout:1488: finishing write tid 1 to smithi08575397-3 2021-02-10T04:33:34.604 INFO:tasks.rados.rados.0.smithi085.stdout:1488: finishing write tid 2 to smithi08575397-3 2021-02-10T04:33:34.604 INFO:tasks.rados.rados.0.smithi085.stdout:1488: finishing write tid 3 to smithi08575397-3 2021-02-10T04:33:34.604 INFO:tasks.rados.rados.0.smithi085.stdout:1488: finishing write tid 4 to smithi08575397-3 2021-02-10T04:33:34.604 INFO:tasks.rados.rados.0.smithi085.stdout:1488: finishing write tid 5 to smithi08575397-3 2021-02-10T04:33:34.604 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 641 (ObjNum 533 snap 158 seq_num 533) dirty exists 2021-02-10T04:33:34.604 INFO:tasks.rados.rados.0.smithi085.stdout:1488: left oid 3 (ObjNum 533 snap 158 seq_num 533) 2021-02-10T04:33:34.606 INFO:tasks.rados.rados.0.smithi085.stdout:1488: done (11 left) 2021-02-10T04:33:34.606 INFO:tasks.rados.rados.0.smithi085.stdout:1491: done (10 left) 2021-02-10T04:33:34.606 INFO:tasks.rados.rados.0.smithi085.stdout:1492: snap_create 2021-02-10T04:33:34.610 INFO:tasks.rados.rados.0.smithi085.stdout:1490: expect (ObjNum 476 snap 135 seq_num 476) 2021-02-10T04:33:34.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:34 smithi085 conmon[29319]: cluster 2021-02-10T04:33:33.461207+0000 mon.a ( 2021-02-10T04:33:34.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:34 smithi085 conmon[29319]: mon.0) 1179 : cluster [DBG] osdmap e396: 8 total, 8 up, 8 in 2021-02-10T04:33:34.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:34 smithi085 conmon[29319]: cluster 2021-02-10T04:33:34.073240+0000 mgr.y (mgr.14140) 523 : cluster [DBG] pgmap v744: 168 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 162 active+clean; 305 MiB data, 946 MiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 568 KiB/s wr, 8 op/s 2021-02-10T04:33:34.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:34 smithi085 conmon[29319]: audit 2021-02-10T04:33:34.201985+0000 mon.a (mon.0) 1180 : audit [INF] from='client.? 172.21.15.85:0/2903889160' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T04:33:34.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:34 smithi085 conmon[35961]: cluster 2021-02-10T04:33:34.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:34 smithi085 conmon[35961]: 2021-02-10T04:33:34.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:34 smithi085 conmon[35961]: 2021 2021-02-10T04:33:34.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:34 smithi085 conmon[35961]: - 2021-02-10T04:33:34.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:34 smithi085 conmon[35961]: 02-10T04:33:33.461207+0000 mon.a (mon.0) 1179 : cluster [DBG] 2021-02-10T04:33:34.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:34 smithi085 conmon[35961]: osdmap e396: 8 total, 8 up, 8 in 2021-02-10T04:33:34.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:34 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:33:34.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:34 smithi085 conmon[35961]: 33:34.073240+0000 mgr.y (mgr.14140) 2021-02-10T04:33:34.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:34 smithi085 conmon[35961]: 523 : cluster [DBG] pgmap v744: 168 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 162 active+clean; 305 MiB data, 946 MiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 568 KiB/s wr, 8 op/s 2021-02-10T04:33:34.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:34 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:33:34.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:34 smithi085 conmon[35961]: -10T04:33:34.201985+0000 mon.a (mon.0) 1180 : audit [INF] from='client.? 172.21.15.85:0/2903889160' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T04:33:34.862 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:33:34.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:34 smithi164 conmon[29288]: cluster 2021-02-10T04:33:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:34 smithi164 conmon[29288]: 2021-02-10T04:33:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:34 smithi164 conmon[29288]: 2021 2021-02-10T04:33:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:34 smithi164 conmon[29288]: - 2021-02-10T04:33:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:34 smithi164 conmon[29288]: 02-10T04:33:33.461207+0000 mon.a (mon 2021-02-10T04:33:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:34 smithi164 conmon[29288]: .0) 1179 : cluster [DBG] osdmap e396: 8 total, 8 up, 8 in 2021-02-10T04:33:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:34 smithi164 conmon[29288]: cluster 2021-02-10T04:33:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:34 smithi164 conmon[29288]: 2021-02-10T04:33:34.073240 2021-02-10T04:33:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:34 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 523 : cluster 2021-02-10T04:33:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:34 smithi164 conmon[29288]: [DBG] pgmap v744: 168 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 162 active+clean; 305 MiB data, 946 MiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 568 KiB/s wr, 8 op/s 2021-02-10T04:33:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:34 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:33:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:34 smithi164 conmon[29288]: 02-10T04:33:34.201985+0000 2021-02-10T04:33:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:34 smithi164 conmon[29288]: mon.a (mon.0) 1180 : audit 2021-02-10T04:33:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:34 smithi164 conmon[29288]: [INF] from='client.? 172.21.15.85:0/2903889160' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T04:33:35.472 INFO:tasks.rados.rados.0.smithi085.stdout:1490: done (10 left) 2021-02-10T04:33:35.472 INFO:tasks.rados.rados.0.smithi085.stdout:1492: done (9 left) 2021-02-10T04:33:35.472 INFO:tasks.rados.rados.0.smithi085.stdout:1493: snap_create 2021-02-10T04:33:35.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:35 smithi164 conmon[29288]: audit 2021-02-10T04:33:34.470564+0000 mon.a (mon 2021-02-10T04:33:35.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:35 smithi164 conmon[29288]: .0) 1181 : audit [INF] from='client.? 172.21.15.85:0/2903889160' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2021-02-10T04:33:35.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:35 smithi164 conmon[29288]: cluster 2021-02-10T04:33:34.470642+0000 mon.a (mon.0) 1182 : cluster [DBG] osdmap e397: 8 total, 8 up, 8 in 2021-02-10T04:33:35.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:35 smithi085 conmon[29319]: audit 2021-02-10T04:33:34.470564+0000 mon.a (mon.0) 1181 : audit 2021-02-10T04:33:35.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:35 smithi085 conmon[29319]: [INF] from='client.? 172.21.15.85:0/2903889160' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2021-02-10T04:33:35.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:35 smithi085 conmon[29319]: cluster 2021-02-10T04:33:34.470642+0000 mon.a (mon.0) 1182 : cluster [DBG] osdmap e397: 8 total, 8 up, 8 in 2021-02-10T04:33:35.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:35 smithi085 conmon[35961]: audit 2021-02-10T04:33:34.470564+0000 2021-02-10T04:33:35.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:35 smithi085 conmon[35961]: mon.a (mon.0) 1181 : audit [INF] from='client.? 172.21.15.85:0/2903889160' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2021-02-10T04:33:35.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:35 smithi085 conmon[35961]: cluster 2021-02-10T04:33:35.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:35 smithi085 conmon[35961]: 2021-02-10T04:33:34.470642+0000 mon.a (mon.0) 1182 : cluster 2021-02-10T04:33:35.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:35 smithi085 conmon[35961]: [DBG] osdmap e397: 8 total, 8 up, 8 in 2021-02-10T04:33:36.482 INFO:tasks.rados.rados.0.smithi085.stdout:1493: done (9 left) 2021-02-10T04:33:36.483 INFO:tasks.rados.rados.0.smithi085.stdout:1494: read oid 48 snap -1 2021-02-10T04:33:36.483 INFO:tasks.rados.rados.0.smithi085.stdout:1494: expect (ObjNum 476 snap 135 seq_num 476) 2021-02-10T04:33:36.483 INFO:tasks.rados.rados.0.smithi085.stdout:1495: copy_from oid 14 from oid 28 current snap is 160 2021-02-10T04:33:36.483 INFO:tasks.rados.rados.0.smithi085.stdout:1496: read oid 20 snap -1 2021-02-10T04:33:36.483 INFO:tasks.rados.rados.0.smithi085.stdout:1496: expect (ObjNum 509 snap 148 seq_num 509) 2021-02-10T04:33:36.483 INFO:tasks.rados.rados.0.smithi085.stdout:1497: read oid 1 snap -1 2021-02-10T04:33:36.483 INFO:tasks.rados.rados.0.smithi085.stdout:1497: expect (ObjNum 424 snap 109 seq_num 424) 2021-02-10T04:33:36.483 INFO:tasks.rados.rados.0.smithi085.stdout:1498: rollback oid 3 current snap is 160 2021-02-10T04:33:36.483 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 3 to 146 2021-02-10T04:33:36.483 INFO:tasks.rados.rados.0.smithi085.stdout:1499: snap_remove snap 122 2021-02-10T04:33:36.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:36 smithi164 conmon[29288]: cluster 2021-02-10T04:33:35.472031+0000 mon.a (mon. 2021-02-10T04:33:36.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:36 smithi164 conmon[29288]: 0) 1183 : cluster [DBG] osdmap e398: 8 total, 8 up, 8 in 2021-02-10T04:33:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:36 smithi164 conmon[29288]: cluster 2021-02-10T04:33:36.073697+0000 mgr.y (mgr.14140) 524 : cluster [DBG] pgmap v747: 168 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 160 active+clean; 331 MiB data, 975 MiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 5.3 MiB/s wr, 31 op/s 2021-02-10T04:33:36.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:36 smithi085 conmon[29319]: cluster 2021-02-10T04:33:35.472031+0000 mon.a (mon 2021-02-10T04:33:36.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:36 smithi085 conmon[29319]: .0) 1183 : cluster [DBG] osdmap e398: 8 total, 8 up, 8 in 2021-02-10T04:33:36.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:36 smithi085 conmon[29319]: cluster 2021-02-10T04:33:36.073697+0000 mgr.y (mgr. 2021-02-10T04:33:36.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:36 smithi085 conmon[29319]: 14140) 524 : cluster [DBG] pgmap v747: 168 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 160 active+clean; 331 MiB data, 975 MiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 5.3 MiB/s wr, 31 op/s 2021-02-10T04:33:36.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:36 smithi085 conmon[35961]: cluster 2021-02-10T04:33:35.472031+0000 mon.a (mon.0) 1183 2021-02-10T04:33:36.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:36 smithi085 conmon[35961]: : cluster [DBG] osdmap e398: 8 total, 8 up, 8 in 2021-02-10T04:33:36.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:36 smithi085 conmon[35961]: cluster 2021-02-10T04:33:36.073697+0000 mgr.y 2021-02-10T04:33:36.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:36 smithi085 conmon[35961]: (mgr.14140) 524 : cluster [DBG] pgmap v747: 168 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 160 active+clean; 331 MiB data, 975 MiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 5.3 MiB/s wr, 31 op/s 2021-02-10T04:33:37.483 INFO:tasks.rados.rados.0.smithi085.stdout:1499: done (14 left) 2021-02-10T04:33:37.483 INFO:tasks.rados.rados.0.smithi085.stdout:1500: snap_create 2021-02-10T04:33:37.493 INFO:tasks.rados.rados.0.smithi085.stdout:1496: expect (ObjNum 509 snap 148 seq_num 509) 2021-02-10T04:33:37.677 INFO:tasks.rados.rados.0.smithi085.stdout:1498: finishing rollback tid 0 to smithi08575397-3 2021-02-10T04:33:37.677 INFO:tasks.rados.rados.0.smithi085.stdout:1498: finishing rollback tid 1 to smithi08575397-3 2021-02-10T04:33:37.678 INFO:tasks.rados.rados.0.smithi085.stdout:1498: finishing rollback tid 2 to smithi08575397-3 2021-02-10T04:33:37.678 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 644 (ObjNum 204 snap 44 seq_num 204) dirty exists 2021-02-10T04:33:37.678 INFO:tasks.rados.rados.0.smithi085.stdout:1494: expect (ObjNum 476 snap 135 seq_num 476) 2021-02-10T04:33:37.877 INFO:tasks.rados.rados.0.smithi085.stdout:1497: expect (ObjNum 424 snap 109 seq_num 424) 2021-02-10T04:33:37.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:37 smithi164 conmon[29288]: cluster 2021-02-10T04:33:36.481745+0000 mon.a (mon.0) 1184 : cluster [DBG] osdmap e399: 8 total, 8 up, 8 in 2021-02-10T04:33:37.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:37 smithi085 conmon[29319]: cluster 2021-02-10T04:33:37.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:37 smithi085 conmon[29319]: 2021-02-10T04:33:37.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:37 smithi085 conmon[29319]: 2021-02-10T04:33:36.481745 2021-02-10T04:33:37.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:37 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1184 : cluster [DBG] osdmap e399: 8 total, 8 up, 8 in 2021-02-10T04:33:37.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:37 smithi085 conmon[35961]: cluster 2021-02-10T04:33:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:37 smithi085 conmon[35961]: 2021-02-10T04:33:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:37 smithi085 conmon[35961]: 2021 2021-02-10T04:33:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:37 smithi085 conmon[35961]: - 2021-02-10T04:33:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:37 smithi085 conmon[35961]: 02 2021-02-10T04:33:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:37 smithi085 conmon[35961]: - 2021-02-10T04:33:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:37 smithi085 conmon[35961]: 10 2021-02-10T04:33:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:37 smithi085 conmon[35961]: T 2021-02-10T04:33:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:37 smithi085 conmon[35961]: 04 2021-02-10T04:33:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:37 smithi085 conmon[35961]: : 2021-02-10T04:33:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:37 smithi085 conmon[35961]: 33 2021-02-10T04:33:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:37 smithi085 conmon[35961]: : 2021-02-10T04:33:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:37 smithi085 conmon[35961]: 36 2021-02-10T04:33:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:37 smithi085 conmon[35961]: . 2021-02-10T04:33:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:37 smithi085 conmon[35961]: 481745 2021-02-10T04:33:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:37 smithi085 conmon[35961]: +0000 2021-02-10T04:33:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:37 smithi085 conmon[35961]: 2021-02-10T04:33:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:37 smithi085 conmon[35961]: mon.a 2021-02-10T04:33:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:37 smithi085 conmon[35961]: ( 2021-02-10T04:33:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:37 smithi085 conmon[35961]: mon 2021-02-10T04:33:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:37 smithi085 conmon[35961]: . 2021-02-10T04:33:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:37 smithi085 conmon[35961]: 0 2021-02-10T04:33:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:37 smithi085 conmon[35961]: ) 2021-02-10T04:33:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:37 smithi085 conmon[35961]: 1184 2021-02-10T04:33:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:37 smithi085 conmon[35961]: : cluster [DBG] osdmap e399: 8 total, 8 up, 8 in 2021-02-10T04:33:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:37 smithi085 conmon[35961]: 2021-02-10T04:33:38.098 INFO:tasks.rados.rados.0.smithi085.stdout:1495: finishing copy_from to smithi08575397-14 2021-02-10T04:33:38.098 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 719 (ObjNum 525 snap 155 seq_num 525) dirty exists 2021-02-10T04:33:38.099 INFO:tasks.rados.rados.0.smithi085.stdout:1495: finishing copy_from racing read to smithi08575397-14 2021-02-10T04:33:38.393 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:33:37 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:33:38.484 INFO:tasks.rados.rados.0.smithi085.stdout:1494: done (14 left) 2021-02-10T04:33:38.485 INFO:tasks.rados.rados.0.smithi085.stdout:1495: done (13 left) 2021-02-10T04:33:38.485 INFO:tasks.rados.rados.0.smithi085.stdout:1496: done (12 left) 2021-02-10T04:33:38.485 INFO:tasks.rados.rados.0.smithi085.stdout:1497: done (11 left) 2021-02-10T04:33:38.485 INFO:tasks.rados.rados.0.smithi085.stdout:1498: done (10 left) 2021-02-10T04:33:38.485 INFO:tasks.rados.rados.0.smithi085.stdout:1500: done (9 left) 2021-02-10T04:33:38.485 INFO:tasks.rados.rados.0.smithi085.stdout:1501: copy_from oid 30 from oid 38 current snap is 161 2021-02-10T04:33:38.485 INFO:tasks.rados.rados.0.smithi085.stdout:1502: snap_create 2021-02-10T04:33:38.496 INFO:tasks.rados.rados.0.smithi085.stdout:1501: finishing copy_from to smithi08575397-30 2021-02-10T04:33:38.496 INFO:tasks.rados.rados.0.smithi085.stdout:1501: got expected ENOENT (src dne) 2021-02-10T04:33:38.496 INFO:tasks.rados.rados.0.smithi085.stdout:1501: finishing copy_from racing read to smithi08575397-30 2021-02-10T04:33:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:38 smithi164 conmon[29288]: cluster 2021-02-10T04:33:37.482711+0000 mon.a ( 2021-02-10T04:33:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:38 smithi164 conmon[29288]: mon.0) 1185 : cluster [DBG] osdmap e400: 8 total, 8 up, 8 in 2021-02-10T04:33:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:38 smithi164 conmon[29288]: cluster 2021-02-10T04:33:37.757752+0000 osd.7 (osd.7) 48 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:33:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:38 smithi164 conmon[29288]: cluster 2021-02-10T04:33 2021-02-10T04:33:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:38 smithi164 conmon[29288]: :37.757929+0000 osd.7 (osd.7) 49 : cluster 2021-02-10T04:33:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:38 smithi164 conmon[29288]: [DBG] purged_snaps scrub ok 2021-02-10T04:33:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:38 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:33:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:38 smithi164 conmon[29288]: T04:33:37.974981+0000 mgr.y (mgr. 2021-02-10T04:33:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:38 smithi164 conmon[29288]: 14140) 525 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:33:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:38 smithi164 conmon[29288]: cluster 2021-02-10T04:33:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:38 smithi164 conmon[29288]: 2021-02-10T04:33 2021-02-10T04:33:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:38 smithi164 conmon[29288]: :38.074045+0000 mgr.y (mgr.14140) 526 2021-02-10T04:33:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:38 smithi164 conmon[29288]: : cluster [DBG] pgmap v750: 168 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 160 active+clean; 331 MiB data, 979 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.1 MiB/s wr, 26 op/s 2021-02-10T04:33:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:38 smithi164 conmon[29288]: audit 2021-02-10T04:33:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:38 smithi164 conmon[29288]: 2021-02-10T04:33:38 2021-02-10T04:33:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:38 smithi164 conmon[29288]: .077566+0000 mon.a (mon.0) 1186 : audit 2021-02-10T04:33:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:38 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:33:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:38 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:33:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:38 smithi164 conmon[29288]: 02-10T04:33:38.077832+0000 2021-02-10T04:33:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:38 smithi164 conmon[29288]: mon.a (mon.0) 1187 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:33:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:38 smithi164 conmon[29288]: 2021-02-10T04:33:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:38 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:33:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:38 smithi164 conmon[29288]: -10T04:33:38.118622+0000 mon.a 2021-02-10T04:33:38.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:38 smithi164 conmon[29288]: (mon.0) 1188 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 242 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:33:38.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[29319]: cluster 2021-02-10T04:33:37.482711+0000 mon.a (mon.0 2021-02-10T04:33:38.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[29319]: ) 1185 : cluster [DBG] osdmap e400: 8 total, 8 up, 8 in 2021-02-10T04:33:38.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[29319]: cluster 2021-02-10T04:33:37.757752+0000 osd.7 (osd.7) 48 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:33:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[29319]: cluster 2021-02-10T04:33:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[29319]: 2021-02-10T04:33:37.757929+0000 osd.7 (osd.7) 49 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:33:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[29319]: audit 2021 2021-02-10T04:33:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[29319]: -02-10T04:33:37.974981+0000 2021-02-10T04:33:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[29319]: mgr.y (mgr.14140) 525 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:33:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:33:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[29319]: 02-10T04:33:38.074045+0000 mgr.y 2021-02-10T04:33:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[29319]: (mgr.14140) 526 : cluster [DBG] pgmap v750: 168 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 160 active+clean; 331 MiB data, 979 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.1 MiB/s wr, 26 op/s 2021-02-10T04:33:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[29319]: audit 2021-02-10T04:33:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:33:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[29319]: 33:38.077566+0000 mon.a (mon.0) 1186 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:33:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[29319]: audit 2021 2021-02-10T04:33:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[29319]: -02-10T04:33:38.077832+0000 mon.a ( 2021-02-10T04:33:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[29319]: mon.0) 1187 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:33:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:33:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[29319]: -02-10T04:33:38.118622+0000 mon.a ( 2021-02-10T04:33:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[29319]: mon.0) 1188 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 242 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:33:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[35961]: cluster 2021-02-10T04:33:37.482711 2021-02-10T04:33:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1185 : cluster [DBG] osdmap e400: 8 total, 8 up, 8 in 2021-02-10T04:33:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[35961]: -10T04:33:37.757752+0000 osd.7 ( 2021-02-10T04:33:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[35961]: osd.7) 48 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:33:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[35961]: cluster 2021-02-10T04:33:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:33:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[35961]: :33:37.757929+0000 osd.7 (osd.7) 49 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:33:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:33:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[35961]: T04:33:37.974981+0000 mgr.y (mgr.14140) 2021-02-10T04:33:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[35961]: 525 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:33:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[35961]: -02-10T04:33:38.074045 2021-02-10T04:33:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 526 : cluster 2021-02-10T04:33:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[35961]: [DBG] pgmap v750: 168 pgs: 1 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 160 active+clean; 331 MiB data, 979 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.1 MiB/s wr, 26 op/s 2021-02-10T04:33:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[35961]: audit 2021 2021-02-10T04:33:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[35961]: -02-10T04:33:38.077566+0000 mon.a (mon.0 2021-02-10T04:33:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[35961]: ) 1186 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:33:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[35961]: audit 2021-02-10T04:33:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[35961]: 2021-02-10T04:33 2021-02-10T04:33:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[35961]: :38.077832+0000 mon.a (mon.0) 1187 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:33:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:33:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[35961]: -10T04:33:38.118622+0000 mon.a (mon. 2021-02-10T04:33:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:38 smithi085 conmon[35961]: 0) 1188 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 242 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:33:39.265 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:33:41.662 INFO:tasks.rados.rados.0.smithi085.stdout:1501: done (10 left) 2021-02-10T04:33:41.662 INFO:tasks.rados.rados.0.smithi085.stdout:1502: done (9 left) 2021-02-10T04:33:41.662 INFO:tasks.rados.rados.0.smithi085.stdout:1503: read oid 3 snap -1 2021-02-10T04:33:41.662 INFO:tasks.rados.rados.0.smithi085.stdout:1503: expect (ObjNum 204 snap 44 seq_num 204) 2021-02-10T04:33:41.662 INFO:tasks.rados.rados.0.smithi085.stdout:1504: read oid 12 snap -1 2021-02-10T04:33:41.663 INFO:tasks.rados.rados.0.smithi085.stdout:1504: expect (ObjNum 85 snap 11 seq_num 85) 2021-02-10T04:33:41.663 INFO:tasks.rados.rados.0.smithi085.stdout:1505: delete oid 36 current snap is 162 2021-02-10T04:33:41.663 INFO:tasks.rados.rados.0.smithi085.stdout:1504: expect (ObjNum 85 snap 11 seq_num 85) 2021-02-10T04:33:41.663 INFO:tasks.rados.rados.0.smithi085.stdout:1503: expect (ObjNum 204 snap 44 seq_num 204) 2021-02-10T04:33:41.663 INFO:tasks.rados.rados.0.smithi085.stdout:1503: done (11 left) 2021-02-10T04:33:41.663 INFO:tasks.rados.rados.0.smithi085.stdout:1504: done (10 left) 2021-02-10T04:33:41.663 INFO:tasks.rados.rados.0.smithi085.stdout:1505: done (9 left) 2021-02-10T04:33:41.663 INFO:tasks.rados.rados.0.smithi085.stdout:1506: snap_create 2021-02-10T04:33:41.663 INFO:tasks.rados.rados.0.smithi085.stdout:1506: done (9 left) 2021-02-10T04:33:41.663 INFO:tasks.rados.rados.0.smithi085.stdout:1507: read oid 7 snap -1 2021-02-10T04:33:41.663 INFO:tasks.rados.rados.0.smithi085.stdout:1507: expect (ObjNum 204 snap 44 seq_num 204) 2021-02-10T04:33:41.663 INFO:tasks.rados.rados.0.smithi085.stdout:1508: read oid 16 snap -1 2021-02-10T04:33:41.664 INFO:tasks.rados.rados.0.smithi085.stdout:1508: expect (ObjNum 523 snap 155 seq_num 523) 2021-02-10T04:33:41.664 INFO:tasks.rados.rados.0.smithi085.stdout:1509: read oid 50 snap 147 2021-02-10T04:33:41.664 INFO:tasks.rados.rados.0.smithi085.stdout:1509: expect deleted 2021-02-10T04:33:41.664 INFO:tasks.rados.rados.0.smithi085.stdout:1510: write (excl) oid 42 current snap is 163 2021-02-10T04:33:41.664 INFO:tasks.rados.rados.0.smithi085.stdout:1510: seq_num 537 ranges {625359=553423,1790948=567184,2930500=94859} 2021-02-10T04:33:41.664 INFO:tasks.rados.rados.0.smithi085.stdout:1510: writing smithi08575397-42 from 625359 to 1178782 tid 1 2021-02-10T04:33:41.664 INFO:tasks.rados.rados.0.smithi085.stdout:1510: writing smithi08575397-42 from 1790948 to 2358132 tid 2 2021-02-10T04:33:41.664 INFO:tasks.rados.rados.0.smithi085.stdout:1510: writing smithi08575397-42 from 2930500 to 3025359 tid 3 2021-02-10T04:33:41.664 INFO:tasks.rados.rados.0.smithi085.stdout:1511: delete oid 6 current snap is 163 2021-02-10T04:33:41.664 INFO:tasks.rados.rados.0.smithi085.stdout:1508: expect (ObjNum 523 snap 155 seq_num 523) 2021-02-10T04:33:41.664 INFO:tasks.rados.rados.0.smithi085.stdout:1508: done (13 left) 2021-02-10T04:33:41.664 INFO:tasks.rados.rados.0.smithi085.stdout:1509: done (12 left) 2021-02-10T04:33:41.665 INFO:tasks.rados.rados.0.smithi085.stdout:1511: done (11 left) 2021-02-10T04:33:41.665 INFO:tasks.rados.rados.0.smithi085.stdout:1512: snap_remove snap 158 2021-02-10T04:33:41.665 INFO:tasks.rados.rados.0.smithi085.stdout:1512: done (11 left) 2021-02-10T04:33:41.665 INFO:tasks.rados.rados.0.smithi085.stdout:1513: rollback oid 10 current snap is 163 2021-02-10T04:33:41.665 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 10 to 162 2021-02-10T04:33:41.665 INFO:tasks.rados.rados.0.smithi085.stdout:1507: expect (ObjNum 204 snap 44 seq_num 204) 2021-02-10T04:33:41.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:39 smithi085 conmon[35961]: cluster 2021-02-10T04:33:38.484954+0000 2021-02-10T04:33:41.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:39 smithi085 conmon[35961]: mon.a (mon.0) 1189 : cluster [DBG] osdmap e401: 8 total, 8 up, 8 in 2021-02-10T04:33:41.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:39 smithi085 conmon[35961]: cluster 2021-02-10T04:33:38.485611 2021-02-10T04:33:41.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:39 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1190 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 242 sec, osd.4 has slow ops) 2021-02-10T04:33:41.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:39 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:33:41.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:39 smithi085 conmon[35961]: T04:33:39.487891+0000 mon.a (mon. 2021-02-10T04:33:41.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:39 smithi085 conmon[35961]: 0) 1191 : cluster [DBG] osdmap e402: 8 total, 8 up, 8 in 2021-02-10T04:33:41.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:40 smithi085 conmon[35961]: cluster 2021-02-10T04:33:41.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:40 smithi085 conmon[35961]: 2021-02-10T04:33:41.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:40 smithi085 conmon[35961]: 2021 2021-02-10T04:33:41.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:40 smithi085 conmon[35961]: - 2021-02-10T04:33:41.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:40 smithi085 conmon[35961]: 02 2021-02-10T04:33:41.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:40 smithi085 conmon[35961]: - 2021-02-10T04:33:41.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:40 smithi085 conmon[35961]: 10 2021-02-10T04:33:41.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:40 smithi085 conmon[35961]: T 2021-02-10T04:33:41.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:40 smithi085 conmon[35961]: 04 2021-02-10T04:33:41.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:40 smithi085 conmon[35961]: :33:40.074474+0000 mgr.y ( 2021-02-10T04:33:41.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:40 smithi085 conmon[35961]: mgr.14140) 527 : cluster [DBG] pgmap v753: 168 pgs: 2 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 158 active+clean; 334 MiB data, 986 MiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 689 KiB/s wr, 8 op/s 2021-02-10T04:33:41.667 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:39 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:33:41.667 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:39 smithi085 conmon[29319]: :33:38.484954+0000 mon.a (mon.0) 1189 : cluster [DBG] osdmap e401: 8 total, 8 up, 8 in 2021-02-10T04:33:41.667 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:39 smithi085 conmon[29319]: cluster 2021-02-10T04:33:38.485611+0000 mon.a (mon.0) 1190 : cluster 2021-02-10T04:33:41.667 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:39 smithi085 conmon[29319]: [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 242 sec, osd.4 has slow ops) 2021-02-10T04:33:41.667 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:39 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:33:41.667 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:39 smithi085 conmon[29319]: 10T04:33:39.487891+0000 mon.a (mon.0) 1191 : cluster [DBG] osdmap e402: 8 total, 8 up, 8 in 2021-02-10T04:33:41.668 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:40 smithi085 conmon[29319]: cluster 2021-02-10T04:33:41.668 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:40 smithi085 conmon[29319]: 2021-02-10T04:33:41.668 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:40 smithi085 conmon[29319]: 2021 2021-02-10T04:33:41.668 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:40 smithi085 conmon[29319]: - 2021-02-10T04:33:41.668 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:40 smithi085 conmon[29319]: 02 2021-02-10T04:33:41.668 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:40 smithi085 conmon[29319]: - 2021-02-10T04:33:41.668 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:40 smithi085 conmon[29319]: 10 2021-02-10T04:33:41.668 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:40 smithi085 conmon[29319]: T 2021-02-10T04:33:41.668 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:40 smithi085 conmon[29319]: 04:33:40.074474+0000 mgr.y (mgr 2021-02-10T04:33:41.668 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:40 smithi085 conmon[29319]: .14140) 527 : cluster [DBG] pgmap v753: 168 pgs: 2 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 158 active+clean; 334 MiB data, 986 MiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 689 KiB/s wr, 8 op/s 2021-02-10T04:33:41.669 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:39 smithi164 conmon[29288]: cluster 2021-02-10T04:33:38.484954+0000 mon.a ( 2021-02-10T04:33:41.669 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:39 smithi164 conmon[29288]: mon.0) 1189 : cluster [DBG] osdmap e401: 8 total, 8 up, 8 in 2021-02-10T04:33:41.669 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:39 smithi164 conmon[29288]: cluster 2021-02-10T04:33:38.485611+0000 mon.a (mon.0) 1190 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 242 sec, osd.4 has slow ops) 2021-02-10T04:33:41.669 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:39 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:33:41.670 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:39 smithi164 conmon[29288]: 02-10T04:33:39.487891+0000 mon.a (mon.0) 1191 : cluster [DBG] osdmap e402: 8 total, 8 up, 8 in 2021-02-10T04:33:41.670 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:40 smithi164 conmon[29288]: cluster 2021-02-10T04:33:40.074474+0000 mgr.y ( 2021-02-10T04:33:41.670 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:40 smithi164 conmon[29288]: mgr.14140) 527 : cluster [DBG] pgmap v753: 168 pgs: 2 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 158 active+clean; 334 MiB data, 986 MiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 689 KiB/s wr, 8 op/s 2021-02-10T04:33:41.697 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:41.777 INFO:tasks.rados.rados.0.smithi085.stdout:1510: finishing write tid 1 to smithi08575397-42 2021-02-10T04:33:41.777 INFO:tasks.rados.rados.0.smithi085.stdout:1510: finishing write tid 2 to smithi08575397-42 2021-02-10T04:33:41.777 INFO:tasks.rados.rados.0.smithi085.stdout:1510: finishing write tid 3 to smithi08575397-42 2021-02-10T04:33:41.777 INFO:tasks.rados.rados.0.smithi085.stdout:1510: finishing write tid 4 to smithi08575397-42 2021-02-10T04:33:41.777 INFO:tasks.rados.rados.0.smithi085.stdout:1510: finishing write tid 5 to smithi08575397-42 2021-02-10T04:33:41.777 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 723 (ObjNum 537 snap 163 seq_num 537) dirty exists 2021-02-10T04:33:41.777 INFO:tasks.rados.rados.0.smithi085.stdout:1510: left oid 42 (ObjNum 537 snap 163 seq_num 537) 2021-02-10T04:33:41.778 INFO:tasks.rados.rados.0.smithi085.stdout:1513: finishing rollback tid 1 to smithi08575397-10 2021-02-10T04:33:41.778 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 10 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-02-10T04:33:41.778 INFO:tasks.rados.rados.0.smithi085.stdout:1507: done (11 left) 2021-02-10T04:33:41.778 INFO:tasks.rados.rados.0.smithi085.stdout:1510: done (10 left) 2021-02-10T04:33:41.778 INFO:tasks.rados.rados.0.smithi085.stdout:1513: done (9 left) 2021-02-10T04:33:41.778 INFO:tasks.rados.rados.0.smithi085.stdout:1514: snap_remove snap 157 2021-02-10T04:33:41.780 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:41 smithi164 conmon[29288]: cluster 2021-02-10T04:33:40.506841+0000 2021-02-10T04:33:41.780 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:41 smithi164 conmon[29288]: mon.a (mon.0) 1192 : cluster [DBG] osdmap e403: 8 total, 8 up, 8 in 2021-02-10T04:33:41.854 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:41 smithi085 conmon[35961]: cluster 2021-02-10T04:33:40.506841+0000 mon.a (mon.0 2021-02-10T04:33:41.854 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:41 smithi085 conmon[35961]: ) 1192 : cluster [DBG] osdmap e403: 8 total, 8 up, 8 in 2021-02-10T04:33:41.854 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:41 smithi085 conmon[29319]: cluster 2021-02-10T04:33:41.854 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:41 smithi085 conmon[29319]: 2021-02-10T04:33:41.854 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:41 smithi085 conmon[29319]: 2021 2021-02-10T04:33:41.854 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:41 smithi085 conmon[29319]: -02-10T04: 2021-02-10T04:33:41.854 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:41 smithi085 conmon[29319]: 33:40.506841+0000 mon.a (mon.0) 1192 : cluster [DBG] osdmap e403: 8 total, 8 up, 8 in 2021-02-10T04:33:42.112 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:33:42.116 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:33:41 smithi164 conmon[40892]: debug 2021-02-10T04:33:41.778+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:42.217 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:33:42.247 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:42.393 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:33:42 smithi164 conmon[37263]: debug 2021-02-10T04:33:42.333+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:42.525 INFO:tasks.rados.rados.0.smithi085.stdout:1514: done (9 left) 2021-02-10T04:33:42.526 INFO:tasks.rados.rados.0.smithi085.stdout:1515: rollback oid 6 current snap is 163 2021-02-10T04:33:42.526 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 6 to 147 2021-02-10T04:33:42.526 INFO:tasks.rados.rados.0.smithi085.stdout:1516: snap_create 2021-02-10T04:33:42.529 INFO:tasks.rados.rados.0.smithi085.stdout:1515: finishing rollback tid 1 to smithi08575397-6 2021-02-10T04:33:42.529 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 687 (ObjNum 1215200336 snap 32576 seq_num 82182464) dirty dne 2021-02-10T04:33:42.559 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:33:42.664 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:33:42.701 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:42.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:42 smithi164 conmon[29288]: cluster 2021-02-10T04:33:41.519922+0000 mon.a (mon.0) 1193 2021-02-10T04:33:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:42 smithi164 conmon[29288]: : cluster [DBG] osdmap e404: 8 total, 8 up, 8 in 2021-02-10T04:33:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:42 smithi164 conmon[29288]: cluster 2021-02-10T04:33:42.075022+0000 mgr.y (mgr.14140) 2021-02-10T04:33:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:42 smithi164 conmon[29288]: 528 : cluster [DBG] pgmap v756: 168 pgs: 2 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 158 active+clean; 337 MiB data, 986 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 689 KiB/s wr, 14 op/s 2021-02-10T04:33:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[29319]: cluster 2021-02-10T04:33:41.519922+0000 mon.a ( 2021-02-10T04:33:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[29319]: mon.0) 1193 : cluster [DBG] osdmap e404: 8 total, 8 up, 8 in 2021-02-10T04:33:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[29319]: cluster 2021-02-10T04:33:42.075022 2021-02-10T04:33:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 528 : 2021-02-10T04:33:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[29319]: cluster [DBG] pgmap v756: 168 pgs: 2 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 158 active+clean; 337 MiB data, 986 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 689 KiB/s wr, 14 op/s 2021-02-10T04:33:42.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[35961]: cluster 2021-02-10T04:33:42.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[35961]: 2021-02-10T04:33:42.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[35961]: 2021 2021-02-10T04:33:42.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[35961]: - 2021-02-10T04:33:42.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[35961]: 02 2021-02-10T04:33:42.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[35961]: - 2021-02-10T04:33:42.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[35961]: 10 2021-02-10T04:33:42.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[35961]: T 2021-02-10T04:33:42.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[35961]: 04 2021-02-10T04:33:42.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[35961]: : 2021-02-10T04:33:42.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[35961]: 33 2021-02-10T04:33:42.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[35961]: : 2021-02-10T04:33:42.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[35961]: 41 2021-02-10T04:33:42.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[35961]: . 2021-02-10T04:33:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[35961]: 519922 2021-02-10T04:33:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[35961]: +0000 2021-02-10T04:33:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[35961]: 2021-02-10T04:33:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[35961]: mon.a 2021-02-10T04:33:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[35961]: ( 2021-02-10T04:33:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[35961]: mon 2021-02-10T04:33:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[35961]: . 2021-02-10T04:33:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[35961]: 0 2021-02-10T04:33:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[35961]: ) 2021-02-10T04:33:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[35961]: 1193 2021-02-10T04:33:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[35961]: : 2021-02-10T04:33:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[35961]: cluster [DBG] osdmap e404: 8 total, 8 up, 8 in 2021-02-10T04:33:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:33:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[35961]: 02-10T04:33:42.075022+0000 mgr.y (mgr.14140) 2021-02-10T04:33:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:42 smithi085 conmon[35961]: 528 : cluster [DBG] pgmap v756: 168 pgs: 2 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 158 active+clean; 337 MiB data, 986 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 689 KiB/s wr, 14 op/s 2021-02-10T04:33:43.125 INFO:tasks.rados.rados.0.smithi085.stdout:1515: done (10 left) 2021-02-10T04:33:43.125 INFO:tasks.rados.rados.0.smithi085.stdout:1516: done (9 left) 2021-02-10T04:33:43.125 INFO:tasks.rados.rados.0.smithi085.stdout:1517: snap_create 2021-02-10T04:33:43.788 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:33:43.881 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:43 smithi085 conmon[29319]: cluster 2021-02-10T04:33:42.523463+0000 mon.a (mon 2021-02-10T04:33:43.881 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:43 smithi085 conmon[29319]: .0) 1194 : cluster [DBG] osdmap e405: 8 total, 8 up, 8 in 2021-02-10T04:33:43.882 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:43 smithi085 conmon[29319]: cluster 2021-02-10T04:33:42.648608+0000 osd.4 (osd.4) 495 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:33:43.882 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:43 smithi085 conmon[29319]: cluster 2021-02-10T04:33:42 2021-02-10T04:33:43.882 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:43 smithi085 conmon[29319]: .648806+0000 osd.4 (osd.4) 496 : cluster 2021-02-10T04:33:43.882 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:43 smithi085 conmon[29319]: [DBG] purged_snaps scrub ok 2021-02-10T04:33:43.882 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:43 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:33:43.882 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:43 smithi085 conmon[29319]: -10T04:33:43.124597+0000 2021-02-10T04:33:43.882 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:43 smithi085 conmon[29319]: mon.a (mon.0) 1195 : cluster [DBG] 2021-02-10T04:33:43.882 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:43 smithi085 conmon[29319]: osdmap e406: 8 total, 8 up, 8 in 2021-02-10T04:33:43.882 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:43 smithi085 conmon[35961]: cluster 2021-02-10T04:33:42 2021-02-10T04:33:43.882 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:43 smithi085 conmon[35961]: .523463+0000 mon.a (mon.0) 1194 : cluster [DBG] osdmap e405: 8 total, 8 up, 8 in 2021-02-10T04:33:43.882 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:43 smithi085 conmon[35961]: cluster 2021-02-10T04:33 2021-02-10T04:33:43.882 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:43 smithi085 conmon[35961]: :42.648608+0000 osd.4 (osd.4) 495 : cluster 2021-02-10T04:33:43.883 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:43 smithi085 conmon[35961]: [DBG] purged_snaps scrub starts 2021-02-10T04:33:43.883 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:43 smithi085 conmon[35961]: cluster 2021-02-10T04:33:43.883 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:43 smithi085 conmon[35961]: 2021-02-10T04:33 2021-02-10T04:33:43.883 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:43 smithi085 conmon[35961]: :42.648806+0000 osd.4 ( 2021-02-10T04:33:43.883 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:43 smithi085 conmon[35961]: osd.4) 496 : cluster 2021-02-10T04:33:43.883 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:43 smithi085 conmon[35961]: [DBG] purged_snaps scrub ok 2021-02-10T04:33:43.883 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:43 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:43.883 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:43 smithi085 conmon[35961]: -02-10T04:33:43 2021-02-10T04:33:43.883 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:43 smithi085 conmon[35961]: .124597+0000 mon.a (mon.0) 1195 2021-02-10T04:33:43.883 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:43 smithi085 conmon[35961]: : cluster [DBG] osdmap e406: 8 total, 8 up, 8 in 2021-02-10T04:33:43.883 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:33:43 smithi085 conmon[41540]: debug 2021-02-10T04:33:43.883 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:33:43 smithi085 conmon[41540]: 2021-02-10T04:33:43.491+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:43.884 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:33:43 smithi085 conmon[41540]: 2021-02-10T04:33:43.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:43 smithi164 conmon[29288]: cluster 2021-02-10T04:33:43.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:43 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:33:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:43 smithi164 conmon[29288]: 33:42.523463+0000 mon.a (mon.0) 1194 : cluster [DBG] osdmap e405: 8 total, 8 up, 8 in 2021-02-10T04:33:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:43 smithi164 conmon[29288]: cluster 2021-02-10T04:33:42 2021-02-10T04:33:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:43 smithi164 conmon[29288]: .648608+0000 osd.4 (osd.4) 495 : cluster [DBG] 2021-02-10T04:33:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:43 smithi164 conmon[29288]: purged_snaps scrub starts 2021-02-10T04:33:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:43 smithi164 conmon[29288]: cluster 2021-02-10T04:33:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:43 smithi164 conmon[29288]: 2021-02-10T04:33:42. 2021-02-10T04:33:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:43 smithi164 conmon[29288]: 648806+0000 osd.4 (osd.4) 496 : cluster [DBG] 2021-02-10T04:33:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:43 smithi164 conmon[29288]: purged_snaps scrub ok 2021-02-10T04:33:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:43 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:33:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:43 smithi164 conmon[29288]: -10T04:33:43.124597 2021-02-10T04:33:43.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:43 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2021-02-10T04:33:43.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:43 smithi164 conmon[29288]: 1195 : cluster [DBG] osdmap e406: 8 total, 8 up, 8 in 2021-02-10T04:33:44.061 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:33:44.107 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:44.130 INFO:tasks.rados.rados.0.smithi085.stdout:1517: done (9 left) 2021-02-10T04:33:44.130 INFO:tasks.rados.rados.0.smithi085.stdout:1518: delete oid 36 current snap is 165 2021-02-10T04:33:44.140 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 1 1 2021-02-10T04:33:44.141 INFO:tasks.rados.rados.0.smithi085.stdout:1518: done (9 left) 2021-02-10T04:33:44.141 INFO:tasks.rados.rados.0.smithi085.stdout:1519: snap_create 2021-02-10T04:33:44.379 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:33:44.441 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:33:44 smithi085 conmon[41540]: debug 2021-02-10T04:33:44.223+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:44.485 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:33:44.519 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:44.551 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:33:44.809 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:33:44 smithi085 conmon[52836]: debug 2021-02-10T04:33:44.809 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:33:44 smithi085 conmon[52836]: 2021-02-10T04:33:44.807+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:44.854 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:33:44.960 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:33:44.996 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:45.099 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:45 smithi085 conmon[49106]: debug 2021-02-10T04:33:45.099 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:45 smithi085 conmon[49106]: 2021-02-10T04:33:45.097+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:45.099 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:33:45 smithi085 conmon[49106]: 2021-02-10T04:33:45.121 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:33:45.121 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:33:45.139 INFO:tasks.rados.rados.0.smithi085.stdout:1519: done (9 left) 2021-02-10T04:33:45.139 INFO:tasks.rados.rados.0.smithi085.stdout:1520: read oid 14 snap -1 2021-02-10T04:33:45.139 INFO:tasks.rados.rados.0.smithi085.stdout:1520: expect (ObjNum 525 snap 155 seq_num 525) 2021-02-10T04:33:45.139 INFO:tasks.rados.rados.0.smithi085.stdout:1521: snap_create 2021-02-10T04:33:45.152 INFO:tasks.rados.rados.0.smithi085.stdout:1520: expect (ObjNum 525 snap 155 seq_num 525) 2021-02-10T04:33:45.154 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:33:45.260 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:33:45.291 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:45.381 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:33:45 smithi164 conmon[37263]: debug 2021-02-10T04:33:45.381 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:33:45 smithi164 conmon[37263]: 2021-02-10T04:33:45.379+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:45.381 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:33:45 smithi164 conmon[37263]: 2021-02-10T04:33:45.381 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:45 smithi164 conmon[29288]: cluster 2021-02-10T04:33:44.075500+0000 mgr.y (mgr.14140) 529 : cluster 2021-02-10T04:33:45.381 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:45 smithi164 conmon[29288]: [DBG] pgmap v759: 168 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 159 active+clean; 339 MiB data, 989 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 297 KiB/s wr, 10 op/s 2021-02-10T04:33:45.381 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:45 smithi164 conmon[29288]: cluster 2021-02-10T04:33:44.130124+0000 mon.a (mon.0) 1196 : cluster [DBG] osdmap e407: 8 total, 8 up, 8 in 2021-02-10T04:33:45.412 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:33:45.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:45 smithi085 conmon[29319]: cluster 2021-02-10T04:33:44.075500+0000 mgr.y (mgr.14140) 529 : cluster 2021-02-10T04:33:45.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:45 smithi085 conmon[29319]: [DBG] pgmap v759: 168 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 159 active+clean; 339 MiB data, 989 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 297 KiB/s wr, 10 op/s 2021-02-10T04:33:45.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:45 smithi085 conmon[29319]: cluster 2021-02-10T04:33:44.130124+0000 mon.a ( 2021-02-10T04:33:45.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:45 smithi085 conmon[29319]: mon.0) 1196 : cluster [DBG] osdmap e407: 8 total, 8 up, 8 in 2021-02-10T04:33:45.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:45 smithi085 conmon[35961]: cluster 2021-02-10T04:33:45.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:45 smithi085 conmon[35961]: 2021-02-10T04:33:45.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:45 smithi085 conmon[35961]: 2021 2021-02-10T04:33:45.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:45 smithi085 conmon[35961]: - 2021-02-10T04:33:45.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:45 smithi085 conmon[35961]: 02 2021-02-10T04:33:45.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:45 smithi085 conmon[35961]: - 2021-02-10T04:33:45.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:45 smithi085 conmon[35961]: 10T04:33:44.075500 2021-02-10T04:33:45.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:45 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 529 : cluster [DBG] pgmap v759: 168 pgs: 2 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 159 active+clean; 339 MiB data, 989 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 297 KiB/s wr, 10 op/s 2021-02-10T04:33:45.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:45 smithi085 conmon[35961]: cluster 2021-02-10T04:33:45.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:45 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:33:45.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:45 smithi085 conmon[35961]: 10T04:33:44.130124+0000 mon.a (mon.0) 1196 : 2021-02-10T04:33:45.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:45 smithi085 conmon[35961]: cluster [DBG] osdmap e407: 8 total, 8 up, 8 in 2021-02-10T04:33:45.518 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:33:45.546 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:46.141 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:33:46.143 INFO:tasks.rados.rados.0.smithi085.stdout:1520: done (10 left) 2021-02-10T04:33:46.143 INFO:tasks.rados.rados.0.smithi085.stdout:1521: done (9 left) 2021-02-10T04:33:46.143 INFO:tasks.rados.rados.0.smithi085.stdout:1522: snap_remove snap 155 2021-02-10T04:33:46.151 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is unset 2021-02-10T04:33:46.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:46 smithi164 conmon[29288]: cluster 2021-02-10T04:33:45.137936+0000 2021-02-10T04:33:46.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:46 smithi164 conmon[29288]: mon.a (mon.0) 1197 : cluster [DBG] osdmap e408: 8 total, 8 up, 8 in 2021-02-10T04:33:46.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:46 smithi164 conmon[29288]: audit 2021-02-10T04:33:45.188791+0000 mon.a 2021-02-10T04:33:46.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:46 smithi164 conmon[29288]: (mon.0) 1198 : audit [INF] from='client.? 172.21.15.85:0/1937300544' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:33:46.399 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:33:46.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:46 smithi085 conmon[29319]: cluster 2021-02-10T04:33:45.137936+0000 mon.a (mon 2021-02-10T04:33:46.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:46 smithi085 conmon[29319]: .0) 1197 : cluster [DBG] osdmap e408: 8 total, 8 up, 8 in 2021-02-10T04:33:46.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:46 smithi085 conmon[29319]: audit 2021-02-10T04:33:45.188791+0000 mon.a (mon.0) 1198 : audit [INF] from='client.? 172.21.15.85:0/1937300544' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:33:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:46 smithi085 conmon[35961]: cluster 2021-02-10T04:33:45.137936 2021-02-10T04:33:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:46 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1197 : cluster [DBG] osdmap e408: 8 total, 8 up, 8 in 2021-02-10T04:33:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:46 smithi085 conmon[35961]: audit 2021-02-10T04:33:45.188791+0000 mon.a (mon.0) 1198 : audit [INF] 2021-02-10T04:33:46.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:46 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/1937300544' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:33:47.167 INFO:tasks.rados.rados.0.smithi085.stdout:1522: done (9 left) 2021-02-10T04:33:47.167 INFO:tasks.rados.rados.0.smithi085.stdout:1523: read oid 3 snap -1 2021-02-10T04:33:47.167 INFO:tasks.rados.rados.0.smithi085.stdout:1523: expect (ObjNum 204 snap 44 seq_num 204) 2021-02-10T04:33:47.167 INFO:tasks.rados.rados.0.smithi085.stdout:1524: write (excl) oid 42 current snap is 167 2021-02-10T04:33:47.167 INFO:tasks.rados.rados.0.smithi085.stdout:1524: seq_num 540 ranges {675780=495523,1682747=714511,3012769=63011} 2021-02-10T04:33:47.167 INFO:tasks.rados.rados.0.smithi085.stdout:1524: writing smithi08575397-42 from 675780 to 1171303 tid 1 2021-02-10T04:33:47.172 INFO:tasks.rados.rados.0.smithi085.stdout:1524: writing smithi08575397-42 from 1682747 to 2397258 tid 2 2021-02-10T04:33:47.176 INFO:tasks.rados.rados.0.smithi085.stdout:1524: writing smithi08575397-42 from 3012769 to 3075780 tid 3 2021-02-10T04:33:47.176 INFO:tasks.rados.rados.0.smithi085.stdout:1525: rollback oid 1 current snap is 167 2021-02-10T04:33:47.176 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 1 to 165 2021-02-10T04:33:47.177 INFO:tasks.rados.rados.0.smithi085.stdout:1524: finishing write tid 1 to smithi08575397-42 2021-02-10T04:33:47.177 INFO:tasks.rados.rados.0.smithi085.stdout:1523: expect (ObjNum 204 snap 44 seq_num 204) 2021-02-10T04:33:47.431 INFO:tasks.rados.rados.0.smithi085.stdout:1523: done (11 left) 2021-02-10T04:33:47.431 INFO:tasks.rados.rados.0.smithi085.stdout:1526: delete oid 16 current snap is 167 2021-02-10T04:33:47.431 INFO:tasks.rados.rados.0.smithi085.stdout:1524: finishing write tid 2 to smithi08575397-42 2021-02-10T04:33:47.431 INFO:tasks.rados.rados.0.smithi085.stdout:1524: finishing write tid 3 to smithi08575397-42 2021-02-10T04:33:47.432 INFO:tasks.rados.rados.0.smithi085.stdout:1524: finishing write tid 4 to smithi08575397-42 2021-02-10T04:33:47.432 INFO:tasks.rados.rados.0.smithi085.stdout:1524: finishing write tid 5 to smithi08575397-42 2021-02-10T04:33:47.432 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 727 (ObjNum 540 snap 167 seq_num 540) dirty exists 2021-02-10T04:33:47.432 INFO:tasks.rados.rados.0.smithi085.stdout:1524: left oid 42 (ObjNum 540 snap 167 seq_num 540) 2021-02-10T04:33:47.432 INFO:tasks.rados.rados.0.smithi085.stdout:1525: finishing rollback tid 0 to smithi08575397-1 2021-02-10T04:33:47.432 INFO:tasks.rados.rados.0.smithi085.stdout:1525: finishing rollback tid 1 to smithi08575397-1 2021-02-10T04:33:47.432 INFO:tasks.rados.rados.0.smithi085.stdout:1525: finishing rollback tid 2 to smithi08575397-1 2021-02-10T04:33:47.433 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 605 (ObjNum 424 snap 109 seq_num 424) dirty exists 2021-02-10T04:33:47.433 INFO:tasks.rados.rados.0.smithi085.stdout:1524: done (11 left) 2021-02-10T04:33:47.433 INFO:tasks.rados.rados.0.smithi085.stdout:1525: done (10 left) 2021-02-10T04:33:47.434 INFO:tasks.rados.rados.0.smithi085.stdout:1526: done (9 left) 2021-02-10T04:33:47.434 INFO:tasks.rados.rados.0.smithi085.stdout:1527: snap_remove snap 109 2021-02-10T04:33:47.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[29319]: cluster 2021-02-10T04:33:46.076033+0000 2021-02-10T04:33:47.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[29319]: mgr.y (mgr.14140) 530 : cluster [DBG] pgmap v762: 168 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 163 active+clean; 337 MiB data, 991 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 297 KiB/s wr, 10 op/s 2021-02-10T04:33:47.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:33:47.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[29319]: -10T04:33:46.134187+0000 mon.a (mon.0) 1199 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:33:47.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[29319]: audit 2021-02-10T04:33:47.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:33:47.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[29319]: T04:33:46.139136+0000 mon.a (mon.0) 1200 : audit 2021-02-10T04:33:47.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[29319]: [INF] from='client.? 172.21.15.85:0/1937300544' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:33:47.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:33:47.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[29319]: 10T04:33:46.139408+0000 mon.a (mon. 2021-02-10T04:33:47.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[29319]: 0) 1201 : cluster [DBG] osdmap e409: 8 total, 8 up, 8 in 2021-02-10T04:33:47.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:33:47.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[29319]: 02-10T04 2021-02-10T04:33:47.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[29319]: :33:46.270471+0000 osd.0 (osd. 2021-02-10T04:33:47.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[29319]: 0) 64 : cluster [DBG] 7.4 scrub ok 2021-02-10T04:33:47.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: cluster 2021-02-10T04:33:46 2021-02-10T04:33:47.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: .076033+0000 mgr.y (mgr.14140) 530 : cluster [DBG] pgmap v762: 168 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 163 active+clean; 337 MiB data, 991 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 297 KiB/s wr, 10 op/s 2021-02-10T04:33:47.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:47.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: -02-10T04:33:46.134187+0000 mon.a 2021-02-10T04:33:47.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: (mon. 2021-02-10T04:33:47.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: 0) 1199 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:33:47.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: audit 2021-02-10T04:33:47.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: 2021-02-10T04:33:47.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: 2021 2021-02-10T04:33:47.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: - 2021-02-10T04:33:47.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: 02 2021-02-10T04:33:47.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: - 2021-02-10T04:33:47.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: 10 2021-02-10T04:33:47.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: T 2021-02-10T04:33:47.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: 04 2021-02-10T04:33:47.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: : 2021-02-10T04:33:47.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: 33 2021-02-10T04:33:47.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: : 2021-02-10T04:33:47.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: 46 2021-02-10T04:33:47.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: . 2021-02-10T04:33:47.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: 139136 2021-02-10T04:33:47.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: +0000 2021-02-10T04:33:47.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: 2021-02-10T04:33:47.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: mon.a 2021-02-10T04:33:47.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: ( 2021-02-10T04:33:47.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: mon 2021-02-10T04:33:47.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: . 2021-02-10T04:33:47.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: 0 2021-02-10T04:33:47.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: ) 1200 : audit [INF] from='client.? 172.21.15.85:0/1937300544' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:33:47.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: cluster 2021-02-10T04:33:46.139408 2021-02-10T04:33:47.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1201 : cluster [DBG] osdmap e409: 8 total, 8 up, 8 in 2021-02-10T04:33:47.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:47.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: -02-10T04:33:46.270471+0000 osd.0 2021-02-10T04:33:47.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:47 smithi085 conmon[35961]: (osd.0) 64 : cluster [DBG] 7.4 scrub ok 2021-02-10T04:33:47.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:47 smithi164 conmon[29288]: cluster 2021-02-10T04:33:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:47 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:33:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:47 smithi164 conmon[29288]: :33:46.076033+0000 mgr.y (mgr.14140) 530 : cluster [DBG] pgmap v762: 168 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 163 active+clean; 337 MiB data, 991 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 297 KiB/s wr, 10 op/s 2021-02-10T04:33:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:47 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:33:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:47 smithi164 conmon[29288]: -02-10T04:33:46.134187+0000 mon.a (mon.0) 1199 : 2021-02-10T04:33:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:47 smithi164 conmon[29288]: cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:33:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:47 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:33:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:47 smithi164 conmon[29288]: -10T04:33:46.139136+0000 mon.a (mon 2021-02-10T04:33:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:47 smithi164 conmon[29288]: .0) 1200 : audit [INF] from='client.? 172.21.15.85:0/1937300544' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:33:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:47 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:33:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:47 smithi164 conmon[29288]: 02-10T04:33:46.139408+0000 2021-02-10T04:33:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:47 smithi164 conmon[29288]: mon.a (mon.0) 1201 : cluster [DBG] osdmap e409: 8 total, 8 up, 8 in 2021-02-10T04:33:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:47 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:33:47.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:47 smithi164 conmon[29288]: -02-10T04:33:46.270471+0000 2021-02-10T04:33:47.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:47 smithi164 conmon[29288]: osd.0 (osd.0) 2021-02-10T04:33:47.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:47 smithi164 conmon[29288]: 64 : cluster [DBG] 7.4 scrub ok 2021-02-10T04:33:48.128 INFO:tasks.rados.rados.0.smithi085.stdout:1527: done (9 left) 2021-02-10T04:33:48.128 INFO:tasks.rados.rados.0.smithi085.stdout:1528: snap_create 2021-02-10T04:33:48.393 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:33:47 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:33:48.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:48 smithi164 conmon[29288]: cluster 2021-02-10T04:33:46.785901+0000 osd.3 (osd 2021-02-10T04:33:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:48 smithi164 conmon[29288]: .3) 62 : cluster [DBG] 4.14 scrub ok 2021-02-10T04:33:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:48 smithi164 conmon[29288]: cluster 2021-02-10T04:33:47.156657+0000 mon.a (mon.0) 1202 : cluster [DBG] osdmap e410: 8 total, 8 up, 8 in 2021-02-10T04:33:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:48 smithi164 conmon[29288]: cluster 2021-02-10T04:33:48.127640+0000 2021-02-10T04:33:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:48 smithi164 conmon[29288]: mon.a (mon.0) 1203 : cluster [DBG] osdmap e411: 8 total, 8 up, 8 in 2021-02-10T04:33:48.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:48 smithi085 conmon[29319]: cluster 2021-02-10T04:33:46.785901 2021-02-10T04:33:48.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:48 smithi085 conmon[29319]: +0000 osd.3 (osd.3) 62 : cluster [DBG] 4.14 scrub ok 2021-02-10T04:33:48.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:48 smithi085 conmon[29319]: cluster 2021-02-10T04:33: 2021-02-10T04:33:48.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:48 smithi085 conmon[29319]: 47.156657+0000 mon.a (mon.0) 1202 : cluster [DBG] osdmap e410: 8 total, 8 up, 8 in 2021-02-10T04:33:48.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:48 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:33:48.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:48 smithi085 conmon[29319]: -10T04:33:48.127640+0000 mon.a (mon.0) 1203 : cluster [DBG] 2021-02-10T04:33:48.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:48 smithi085 conmon[29319]: osdmap e411: 8 total, 8 up, 8 in 2021-02-10T04:33:48.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:48 smithi085 conmon[35961]: cluster 2021-02-10T04:33:46.785901+0000 osd.3 (osd. 2021-02-10T04:33:48.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:48 smithi085 conmon[35961]: 3) 62 : cluster [DBG] 4.14 scrub ok 2021-02-10T04:33:48.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:48 smithi085 conmon[35961]: cluster 2021-02-10T04:33:47.156657+0000 mon.a (mon.0) 1202 : cluster [DBG] osdmap e410: 8 total, 8 up, 8 in 2021-02-10T04:33:48.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:48 smithi085 conmon[35961]: cluster 2021-02-10T04:33:48.127640+0000 mon.a ( 2021-02-10T04:33:48.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:48 smithi085 conmon[35961]: mon.0) 1203 : cluster [DBG] osdmap e411: 8 total, 8 up, 8 in 2021-02-10T04:33:49.128 INFO:tasks.rados.rados.0.smithi085.stdout:1528: done (9 left) 2021-02-10T04:33:49.907 INFO:tasks.rados.rados.0.smithi085.stdout:1529: write (excl) oid 7 current snap is 168 2021-02-10T04:33:49.907 INFO:tasks.rados.rados.0.smithi085.stdout:1529: seq_num 542 ranges {709394=646021,2083358=718180} 2021-02-10T04:33:49.913 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:49 smithi164 conmon[29288]: cluster 2021-02-10T04:33:47.618265+0000 osd.1 (osd 2021-02-10T04:33:49.913 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:49 smithi164 conmon[29288]: .1) 445 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:33:49.913 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:49 smithi164 conmon[29288]: cluster 2021-02-10T04:33:47.618459+0000 osd.1 (osd.1) 446 : cluster [DBG] purged_snaps scrub found 2 strays 2021-02-10T04:33:49.914 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:49 smithi164 conmon[29288]: audit 2021-02-10T04:33:47. 2021-02-10T04:33:49.914 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:49 smithi164 conmon[29288]: 985034+0000 mgr.y (mgr.14140) 2021-02-10T04:33:49.914 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:49 smithi164 conmon[29288]: 531 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:33:49.914 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:49 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:33:49.914 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:49 smithi164 conmon[29288]: -02-10T04:33:48.076422+0000 2021-02-10T04:33:49.914 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:49 smithi164 conmon[29288]: mgr.y (mgr.14140) 532 : cluster [DBG] 2021-02-10T04:33:49.915 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:49 smithi164 conmon[29288]: pgmap v765: 168 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 163 active+clean; 337 MiB data, 991 MiB used, 714 GiB / 715 GiB avail; 7.5 MiB/s rd, 0 B/s wr, 6 op/s 2021-02-10T04:33:49.915 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:49 smithi164 conmon[29288]: cluster 2021-02-10T04:33 2021-02-10T04:33:49.915 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:49 smithi164 conmon[29288]: :48.646799+0000 osd.5 (osd.5) 59 2021-02-10T04:33:49.915 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:49 smithi164 conmon[29288]: : cluster [DBG] 2.1c scrub ok 2021-02-10T04:33:49.915 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:49 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:33:49.915 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:49 smithi164 conmon[29288]: T04:33:49.127302+0000 mon.a (mon 2021-02-10T04:33:49.915 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:49 smithi164 conmon[29288]: .0) 1204 : cluster [DBG] osdmap e412: 8 total, 8 up, 8 in 2021-02-10T04:33:49.916 INFO:tasks.rados.rados.0.smithi085.stdout:1529: writing smithi08575397-7 from 709394 to 1355415 tid 1 2021-02-10T04:33:49.916 INFO:tasks.rados.rados.0.smithi085.stdout:1529: writing smithi08575397-7 from 2083358 to 2801538 tid 2 2021-02-10T04:33:49.916 INFO:tasks.rados.rados.0.smithi085.stdout:1530: write (excl) oid 37 current snap is 168 2021-02-10T04:33:49.916 INFO:tasks.rados.rados.0.smithi085.stdout:1529: finishing write tid 1 to smithi08575397-7 2021-02-10T04:33:49.916 INFO:tasks.rados.rados.0.smithi085.stdout:1530: seq_num 543 ranges {726201=721270,2141840=578191} 2021-02-10T04:33:49.917 INFO:tasks.rados.rados.0.smithi085.stdout:1530: writing smithi08575397-37 from 726201 to 1447471 tid 1 2021-02-10T04:33:49.917 INFO:tasks.rados.rados.0.smithi085.stdout:1530: writing smithi08575397-37 from 2141840 to 2720031 tid 2 2021-02-10T04:33:49.917 INFO:tasks.rados.rados.0.smithi085.stdout:1531: delete oid 10 current snap is 168 2021-02-10T04:33:49.917 INFO:tasks.rados.rados.0.smithi085.stdout:1529: finishing write tid 2 to smithi08575397-7 2021-02-10T04:33:49.917 INFO:tasks.rados.rados.0.smithi085.stdout:1529: finishing write tid 3 to smithi08575397-7 2021-02-10T04:33:49.917 INFO:tasks.rados.rados.0.smithi085.stdout:1529: finishing write tid 4 to smithi08575397-7 2021-02-10T04:33:49.918 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 647 (ObjNum 542 snap 168 seq_num 542) dirty exists 2021-02-10T04:33:49.918 INFO:tasks.rados.rados.0.smithi085.stdout:1529: left oid 7 (ObjNum 542 snap 168 seq_num 542) 2021-02-10T04:33:49.918 INFO:tasks.rados.rados.0.smithi085.stdout:1530: finishing write tid 1 to smithi08575397-37 2021-02-10T04:33:49.918 INFO:tasks.rados.rados.0.smithi085.stdout:1529: done (11 left) 2021-02-10T04:33:49.918 INFO:tasks.rados.rados.0.smithi085.stdout:1531: done (10 left) 2021-02-10T04:33:49.919 INFO:tasks.rados.rados.0.smithi085.stdout:1532: read oid 7 snap -1 2021-02-10T04:33:49.919 INFO:tasks.rados.rados.0.smithi085.stdout:1532: expect (ObjNum 542 snap 168 seq_num 542) 2021-02-10T04:33:49.919 INFO:tasks.rados.rados.0.smithi085.stdout:1533: write oid 28 current snap is 168 2021-02-10T04:33:49.919 INFO:tasks.rados.rados.0.smithi085.stdout:1533: seq_num 545 ranges {759815=471768,1742451=465507,2917178=242637} 2021-02-10T04:33:49.919 INFO:tasks.rados.rados.0.smithi085.stdout:1533: writing smithi08575397-28 from 759815 to 1231583 tid 1 2021-02-10T04:33:49.920 INFO:tasks.rados.rados.0.smithi085.stdout:1533: writing smithi08575397-28 from 1742451 to 2207958 tid 2 2021-02-10T04:33:49.920 INFO:tasks.rados.rados.0.smithi085.stdout:1533: writing smithi08575397-28 from 2917178 to 3159815 tid 3 2021-02-10T04:33:49.920 INFO:tasks.rados.rados.0.smithi085.stdout:1534: rollback oid 13 current snap is 168 2021-02-10T04:33:49.920 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 13 to 166 2021-02-10T04:33:49.920 INFO:tasks.rados.rados.0.smithi085.stdout:1535: read oid 6 snap -1 2021-02-10T04:33:49.921 INFO:tasks.rados.rados.0.smithi085.stdout:1535: expect deleted 2021-02-10T04:33:49.921 INFO:tasks.rados.rados.0.smithi085.stdout:1536: write oid 46 current snap is 168 2021-02-10T04:33:49.921 INFO:tasks.rados.rados.0.smithi085.stdout:1536: seq_num 546 ranges {776622=547017,1800933=725518,2944601=232021} 2021-02-10T04:33:49.921 INFO:tasks.rados.rados.0.smithi085.stdout:1536: writing smithi08575397-46 from 776622 to 1323639 tid 1 2021-02-10T04:33:49.921 INFO:tasks.rados.rados.0.smithi085.stdout:1536: writing smithi08575397-46 from 1800933 to 2526451 tid 2 2021-02-10T04:33:49.921 INFO:tasks.rados.rados.0.smithi085.stdout:1536: writing smithi08575397-46 from 2944601 to 3176622 tid 3 2021-02-10T04:33:49.921 INFO:tasks.rados.rados.0.smithi085.stdout:1537: read oid 21 snap -1 2021-02-10T04:33:49.921 INFO:tasks.rados.rados.0.smithi085.stdout:1537: expect (ObjNum 431 snap 114 seq_num 431) 2021-02-10T04:33:49.921 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:33:49.921 INFO:tasks.rados.rados.0.smithi085.stdout:1530: finishing write tid 2 to smithi08575397-37 2021-02-10T04:33:49.921 INFO:tasks.rados.rados.0.smithi085.stdout:1530: finishing write tid 3 to smithi08575397-37 2021-02-10T04:33:49.921 INFO:tasks.rados.rados.0.smithi085.stdout:1530: finishing write tid 4 to smithi08575397-37 2021-02-10T04:33:49.921 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 835 (ObjNum 543 snap 168 seq_num 543) dirty exists 2021-02-10T04:33:49.922 INFO:tasks.rados.rados.0.smithi085.stdout:1530: left oid 37 (ObjNum 543 snap 168 seq_num 543) 2021-02-10T04:33:49.922 INFO:tasks.rados.rados.0.smithi085.stdout:1532: expect (ObjNum 542 snap 168 seq_num 542) 2021-02-10T04:33:49.922 INFO:tasks.rados.rados.0.smithi085.stdout:1533: finishing write tid 1 to smithi08575397-28 2021-02-10T04:33:49.922 INFO:tasks.rados.rados.0.smithi085.stdout:1533: finishing write tid 2 to smithi08575397-28 2021-02-10T04:33:49.922 INFO:tasks.rados.rados.0.smithi085.stdout:1530: done (15 left) 2021-02-10T04:33:49.922 INFO:tasks.rados.rados.0.smithi085.stdout:1532: done (14 left) 2021-02-10T04:33:49.922 INFO:tasks.rados.rados.0.smithi085.stdout:1538: read oid 2 snap -1 2021-02-10T04:33:49.922 INFO:tasks.rados.rados.0.smithi085.stdout:1538: expect (ObjNum 422 snap 109 seq_num 422) 2021-02-10T04:33:49.922 INFO:tasks.rados.rados.0.smithi085.stdout:1539: rollback oid 1 current snap is 168 2021-02-10T04:33:49.923 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 1 to 138 2021-02-10T04:33:49.923 INFO:tasks.rados.rados.0.smithi085.stdout:1534: finishing rollback tid 1 to smithi08575397-13 2021-02-10T04:33:49.923 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 590 (ObjNum 82279728 snap 22030 seq_num 80674832) dirty dne 2021-02-10T04:33:49.923 INFO:tasks.rados.rados.0.smithi085.stdout:1533: finishing write tid 3 to smithi08575397-28 2021-02-10T04:33:49.923 INFO:tasks.rados.rados.0.smithi085.stdout:1533: finishing write tid 4 to smithi08575397-28 2021-02-10T04:33:49.923 INFO:tasks.rados.rados.0.smithi085.stdout:1533: finishing write tid 5 to smithi08575397-28 2021-02-10T04:33:49.924 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 723 (ObjNum 545 snap 168 seq_num 545) dirty exists 2021-02-10T04:33:49.924 INFO:tasks.rados.rados.0.smithi085.stdout:1533: left oid 28 (ObjNum 545 snap 168 seq_num 545) 2021-02-10T04:33:49.924 INFO:tasks.rados.rados.0.smithi085.stdout:1536: finishing write tid 1 to smithi08575397-46 2021-02-10T04:33:49.924 INFO:tasks.rados.rados.0.smithi085.stdout:1536: finishing write tid 2 to smithi08575397-46 2021-02-10T04:33:49.924 INFO:tasks.rados.rados.0.smithi085.stdout:1536: finishing write tid 3 to smithi08575397-46 2021-02-10T04:33:49.924 INFO:tasks.rados.rados.0.smithi085.stdout:1536: finishing write tid 4 to smithi08575397-46 2021-02-10T04:33:49.925 INFO:tasks.rados.rados.0.smithi085.stdout:1536: finishing write tid 5 to smithi08575397-46 2021-02-10T04:33:49.925 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 691 (ObjNum 546 snap 168 seq_num 546) dirty exists 2021-02-10T04:33:49.925 INFO:tasks.rados.rados.0.smithi085.stdout:1536: left oid 46 (ObjNum 546 snap 168 seq_num 546) 2021-02-10T04:33:49.925 INFO:tasks.rados.rados.0.smithi085.stdout:1533: done (15 left) 2021-02-10T04:33:49.925 INFO:tasks.rados.rados.0.smithi085.stdout:1534: done (14 left) 2021-02-10T04:33:49.925 INFO:tasks.rados.rados.0.smithi085.stdout:1535: done (13 left) 2021-02-10T04:33:49.925 INFO:tasks.rados.rados.0.smithi085.stdout:1536: done (12 left) 2021-02-10T04:33:49.925 INFO:tasks.rados.rados.0.smithi085.stdout:1540: snap_create 2021-02-10T04:33:49.926 INFO:tasks.rados.rados.0.smithi085.stdout:1537: expect (ObjNum 431 snap 114 seq_num 431) 2021-02-10T04:33:49.926 INFO:tasks.rados.rados.0.smithi085.stdout:1538: expect (ObjNum 422 snap 109 seq_num 422) 2021-02-10T04:33:49.926 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[35961]: cluster 2021-02-10T04:33:47.618265+0000 osd.1 (osd.1) 445 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:33:49.926 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[35961]: cluster 2021-02-10T04:33:47.618459+0000 osd.1 (osd.1) 446 : cluster [DBG] purged_snaps scrub found 2 strays 2021-02-10T04:33:49.926 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[35961]: audit 2021-02-10T04:33:47. 2021-02-10T04:33:49.926 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[35961]: 985034+0000 mgr.y (mgr.14140) 531 : audit [DBG] 2021-02-10T04:33:49.926 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[35961]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:33:49.926 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[35961]: cluster 2021-02-10T04:33: 2021-02-10T04:33:49.927 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[35961]: 48.076422+0000 mgr.y (mgr.14140) 532 : cluster 2021-02-10T04:33:49.927 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[35961]: [DBG] pgmap v765: 168 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 163 active+clean; 337 MiB data, 991 MiB used, 714 GiB / 715 GiB avail; 7.5 MiB/s rd, 0 B/s wr, 6 op/s 2021-02-10T04:33:49.928 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:33:49.928 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[35961]: 33:48.646799+0000 osd.5 (osd.5) 59 2021-02-10T04:33:49.928 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[35961]: : cluster [DBG] 2.1c scrub ok 2021-02-10T04:33:49.928 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:33:49.928 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[35961]: :33:49.127302+0000 mon.a (mon.0) 1204 : cluster [DBG] osdmap e412: 8 total, 8 up, 8 in 2021-02-10T04:33:49.929 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[29319]: cluster 2021-02-10T04:33:49.929 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[29319]: 2021-02-10T04:33:47.618265 2021-02-10T04:33:49.929 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[29319]: +0000 osd.1 (osd.1) 445 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:33:49.929 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[29319]: cluster 2021-02-10T04:33:49.929 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[29319]: 2021-02-10T04:33:47 2021-02-10T04:33:49.929 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[29319]: .618459+0000 osd.1 (osd.1) 446 2021-02-10T04:33:49.929 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[29319]: : cluster [DBG] purged_snaps scrub found 2 strays 2021-02-10T04:33:49.929 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:33:49.930 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[29319]: 02-10T04:33:47.985034+0000 mgr.y 2021-02-10T04:33:49.930 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[29319]: (mgr.14140) 531 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:33:49.930 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:33:49.930 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[29319]: -10T04:33:48.076422+0000 mgr.y 2021-02-10T04:33:49.930 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[29319]: (mgr.14140) 532 : cluster [DBG] pgmap v765: 168 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 163 active+clean; 337 MiB data, 991 MiB used, 714 GiB / 715 GiB avail; 7.5 MiB/s rd, 0 B/s wr, 6 op/s 2021-02-10T04:33:49.930 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:33:49.930 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[29319]: 02-10T04:33:48.646799+0000 2021-02-10T04:33:49.930 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[29319]: osd.5 (osd.5) 59 : cluster [DBG] 2.1c scrub ok 2021-02-10T04:33:49.930 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[29319]: 2021-02-10T04:33:49.930 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[29319]: cluster 2021-02-10T04:33: 2021-02-10T04:33:49.930 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[29319]: 49.127302+0000 mon.a (mon.0) 1204 : 2021-02-10T04:33:49.931 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:49 smithi085 conmon[29319]: cluster [DBG] osdmap e412: 8 total, 8 up, 8 in 2021-02-10T04:33:49.977 INFO:tasks.rados.rados.0.smithi085.stdout:1539: finishing rollback tid 0 to smithi08575397-1 2021-02-10T04:33:49.977 INFO:tasks.rados.rados.0.smithi085.stdout:1539: finishing rollback tid 1 to smithi08575397-1 2021-02-10T04:33:49.977 INFO:tasks.rados.rados.0.smithi085.stdout:1539: finishing rollback tid 2 to smithi08575397-1 2021-02-10T04:33:49.977 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 608 (ObjNum 424 snap 109 seq_num 424) dirty exists 2021-02-10T04:33:50.218 INFO:tasks.rados.rados.0.smithi085.stdout:1537: done (12 left) 2021-02-10T04:33:50.219 INFO:tasks.rados.rados.0.smithi085.stdout:1538: done (11 left) 2021-02-10T04:33:50.219 INFO:tasks.rados.rados.0.smithi085.stdout:1539: done (10 left) 2021-02-10T04:33:50.219 INFO:tasks.rados.rados.0.smithi085.stdout:1540: done (9 left) 2021-02-10T04:33:50.219 INFO:tasks.rados.rados.0.smithi085.stdout:1541: copy_from oid 38 from oid 30 current snap is 169 2021-02-10T04:33:50.219 INFO:tasks.rados.rados.0.smithi085.stdout:1542: write (excl) oid 3 current snap is 169 2021-02-10T04:33:50.219 INFO:tasks.rados.rados.0.smithi085.stdout:1542: seq_num 547 ranges {793429=622266,1859415=469176,2972024=221405} 2021-02-10T04:33:50.226 INFO:tasks.rados.rados.0.smithi085.stdout:1542: writing smithi08575397-3 from 793429 to 1415695 tid 1 2021-02-10T04:33:50.232 INFO:tasks.rados.rados.0.smithi085.stdout:1542: writing smithi08575397-3 from 1859415 to 2328591 tid 2 2021-02-10T04:33:50.238 INFO:tasks.rados.rados.0.smithi085.stdout:1542: writing smithi08575397-3 from 2972024 to 3193429 tid 3 2021-02-10T04:33:50.238 INFO:tasks.rados.rados.0.smithi085.stdout:1543: read oid 50 snap -1 2021-02-10T04:33:50.238 INFO:tasks.rados.rados.0.smithi085.stdout:1543: expect deleted 2021-02-10T04:33:50.238 INFO:tasks.rados.rados.0.smithi085.stdout:1544: snap_remove snap 138 2021-02-10T04:33:50.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:50 smithi164 conmon[29288]: cluster 2021-02-10T04:33:49.338200+0000 osd.0 (osd. 2021-02-10T04:33:50.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:50 smithi164 conmon[29288]: 0) 65 : cluster [DBG] 6.10 scrub ok 2021-02-10T04:33:50.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:50 smithi164 conmon[29288]: cluster 2021-02-10T04:33:49.760985+0000 osd.4 (osd.4) 497 : cluster [DBG] 2.f scrub ok 2021-02-10T04:33:50.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:50 smithi164 conmon[29288]: cluster 2021-02-10T04:33:50.076846+0000 mgr.y (mgr.14140) 533 2021-02-10T04:33:50.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:50 smithi164 conmon[29288]: : cluster [DBG] pgmap v768: 168 pgs: 2 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 161 active+clean; 335 MiB data, 992 MiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 311 KiB/s wr, 4 op/s 2021-02-10T04:33:50.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:50 smithi085 conmon[35961]: cluster 2021-02-10T04:33:50.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:50 smithi085 conmon[35961]: 2021-02-10T04:33:50.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:50 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:33:50.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:50 smithi085 conmon[35961]: :33:49.338200+0000 osd.0 (osd.0) 65 : cluster [DBG] 6.10 scrub ok 2021-02-10T04:33:50.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:50 smithi085 conmon[35961]: cluster 2021-02-10T04:33:50.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:50 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:33:50.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:50 smithi085 conmon[35961]: 33:49.760985+0000 osd.4 (osd.4) 497 : cluster [DBG] 2.f scrub ok 2021-02-10T04:33:50.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:50 smithi085 conmon[35961]: cluster 2021-02-10T04:33:50.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:50 smithi085 conmon[35961]: 2021-02-10T04:33:50.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:50 smithi085 conmon[35961]: 2021-02-10T04:33:50. 2021-02-10T04:33:50.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:50 smithi085 conmon[35961]: 076846+0000 mgr.y (mgr.14140) 533 : cluster [DBG] pgmap v768: 168 pgs: 2 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 161 active+clean; 335 MiB data, 992 MiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 311 KiB/s wr, 4 op/s 2021-02-10T04:33:50.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:50 smithi085 conmon[29319]: cluster 2021-02-10T04:33:49.338200+0000 osd.0 2021-02-10T04:33:50.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:50 smithi085 conmon[29319]: (osd.0) 65 : cluster [DBG] 6.10 scrub ok 2021-02-10T04:33:50.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:50 smithi085 conmon[29319]: cluster 2021-02-10T04:33:49.760985 2021-02-10T04:33:50.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:50 smithi085 conmon[29319]: +0000 osd.4 (osd.4 2021-02-10T04:33:50.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:50 smithi085 conmon[29319]: ) 497 : cluster [DBG] 2.f scrub ok 2021-02-10T04:33:50.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:50 smithi085 conmon[29319]: cluster 2021-02-10T04:33:50.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:50 smithi085 conmon[29319]: 2021-02-10T04:33:50.076846+0000 mgr.y (mgr 2021-02-10T04:33:50.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:50 smithi085 conmon[29319]: .14140) 533 : cluster [DBG] pgmap v768: 168 pgs: 2 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 161 active+clean; 335 MiB data, 992 MiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 311 KiB/s wr, 4 op/s 2021-02-10T04:33:51.226 INFO:tasks.rados.rados.0.smithi085.stdout:1544: done (12 left) 2021-02-10T04:33:51.226 INFO:tasks.rados.rados.0.smithi085.stdout:1545: write (excl) oid 5 current snap is 169 2021-02-10T04:33:51.226 INFO:tasks.rados.rados.0.smithi085.stdout:1542: finishing write tid 1 to smithi08575397-3 2021-02-10T04:33:51.226 INFO:tasks.rados.rados.0.smithi085.stdout:1542: finishing write tid 2 to smithi08575397-3 2021-02-10T04:33:51.226 INFO:tasks.rados.rados.0.smithi085.stdout:1542: finishing write tid 3 to smithi08575397-3 2021-02-10T04:33:51.227 INFO:tasks.rados.rados.0.smithi085.stdout:1542: finishing write tid 4 to smithi08575397-3 2021-02-10T04:33:51.227 INFO:tasks.rados.rados.0.smithi085.stdout:1542: finishing write tid 5 to smithi08575397-3 2021-02-10T04:33:51.227 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 651 (ObjNum 547 snap 169 seq_num 547) dirty exists 2021-02-10T04:33:51.227 INFO:tasks.rados.rados.0.smithi085.stdout:1542: left oid 3 (ObjNum 547 snap 169 seq_num 547) 2021-02-10T04:33:51.227 INFO:tasks.rados.rados.0.smithi085.stdout:1541: finishing copy_from to smithi08575397-38 2021-02-10T04:33:51.227 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 681 (ObjNum 520 snap 155 seq_num 520) dirty exists 2021-02-10T04:33:51.227 INFO:tasks.rados.rados.0.smithi085.stdout:1541: finishing copy_from racing read to smithi08575397-38 2021-02-10T04:33:51.227 INFO:tasks.rados.rados.0.smithi085.stdout:1545: seq_num 548 ranges {410236=413868,1234250=729187,2715800=423063} 2021-02-10T04:33:51.228 INFO:tasks.rados.rados.0.smithi085.stdout:1545: writing smithi08575397-5 from 410236 to 824104 tid 1 2021-02-10T04:33:51.235 INFO:tasks.rados.rados.0.smithi085.stdout:1545: writing smithi08575397-5 from 1234250 to 1963437 tid 2 2021-02-10T04:33:51.243 INFO:tasks.rados.rados.0.smithi085.stdout:1545: writing smithi08575397-5 from 2715800 to 3138863 tid 3 2021-02-10T04:33:51.243 INFO:tasks.rados.rados.0.smithi085.stdout:1541: done (12 left) 2021-02-10T04:33:51.243 INFO:tasks.rados.rados.0.smithi085.stdout:1542: done (11 left) 2021-02-10T04:33:51.244 INFO:tasks.rados.rados.0.smithi085.stdout:1543: done (10 left) 2021-02-10T04:33:51.244 INFO:tasks.rados.rados.0.smithi085.stdout:1546: snap_create 2021-02-10T04:33:51.244 INFO:tasks.rados.rados.0.smithi085.stdout:1545: finishing write tid 1 to smithi08575397-5 2021-02-10T04:33:51.244 INFO:tasks.rados.rados.0.smithi085.stdout:1545: finishing write tid 2 to smithi08575397-5 2021-02-10T04:33:51.246 INFO:tasks.rados.rados.0.smithi085.stdout:1545: finishing write tid 3 to smithi08575397-5 2021-02-10T04:33:51.247 INFO:tasks.rados.rados.0.smithi085.stdout:1545: finishing write tid 4 to smithi08575397-5 2021-02-10T04:33:51.247 INFO:tasks.rados.rados.0.smithi085.stdout:1545: finishing write tid 5 to smithi08575397-5 2021-02-10T04:33:51.247 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 598 (ObjNum 548 snap 169 seq_num 548) dirty exists 2021-02-10T04:33:51.247 INFO:tasks.rados.rados.0.smithi085.stdout:1545: left oid 5 (ObjNum 548 snap 169 seq_num 548) 2021-02-10T04:33:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:51 smithi164 conmon[29288]: cluster 2021-02-10T04:33:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:51 smithi164 conmon[29288]: 2021-02-10T04:33: 2021-02-10T04:33:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:51 smithi164 conmon[29288]: 49.423076+0000 osd.2 (osd.2) 754 : cluster [DBG] 4.4 scrub ok 2021-02-10T04:33:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:51 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:33:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:51 smithi164 conmon[29288]: 33:50.217335+0000 mon.a (mon.0) 1205 : cluster [DBG] 2021-02-10T04:33:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:51 smithi164 conmon[29288]: osdmap e413: 8 total, 8 up, 8 in 2021-02-10T04:33:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:51 smithi164 conmon[29288]: cluster 2021-02-10T04:33 2021-02-10T04:33:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:51 smithi164 conmon[29288]: :50.351530+0000 osd.0 (osd.0) 66 : cluster 2021-02-10T04:33:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:51 smithi164 conmon[29288]: [DBG] 7.f scrub ok 2021-02-10T04:33:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:51 smithi164 conmon[29288]: cluster 2021-02-10T04:33 2021-02-10T04:33:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:51 smithi164 conmon[29288]: :50.581996+0000 osd.5 (osd.5) 2021-02-10T04:33:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:51 smithi164 conmon[29288]: 60 : cluster [DBG] 7.0 scrub ok 2021-02-10T04:33:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:51 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:33:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:51 smithi164 conmon[29288]: -10T04:33:50.809465+0000 osd.3 ( 2021-02-10T04:33:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:51 smithi164 conmon[29288]: osd.3) 63 : cluster [DBG] 6.13 scrub ok 2021-02-10T04:33:51.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:51 smithi085 conmon[29319]: cluster 2021-02-10T04:33:49.423076+0000 osd.2 2021-02-10T04:33:51.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:51 smithi085 conmon[29319]: (osd.2) 754 : cluster [DBG] 4.4 scrub ok 2021-02-10T04:33:51.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:51 smithi085 conmon[29319]: cluster 2021-02-10T04:33:50.217335+0000 mon.a (mon 2021-02-10T04:33:51.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:51 smithi085 conmon[29319]: .0) 1205 : cluster [DBG] osdmap e413: 8 total, 8 up, 8 in 2021-02-10T04:33:51.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:51 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:33:51.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:51 smithi085 conmon[29319]: 33:50.351530+0000 osd.0 (osd.0) 66 : cluster [DBG] 2021-02-10T04:33:51.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:51 smithi085 conmon[29319]: 7.f scrub ok 2021-02-10T04:33:51.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:51 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:33:51.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:51 smithi085 conmon[29319]: 04:33:50.581996+0000 osd.5 (osd.5) 60 : cluster [DBG] 7.0 scrub ok 2021-02-10T04:33:51.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:51 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:33:51.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:51 smithi085 conmon[29319]: T04:33:50.809465+0000 osd.3 (osd 2021-02-10T04:33:51.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:51 smithi085 conmon[29319]: .3) 63 : cluster [DBG] 6.13 scrub ok 2021-02-10T04:33:51.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:51 smithi085 conmon[35961]: cluster 2021-02-10T04:33:49.423076 2021-02-10T04:33:51.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:51 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 754 : cluster [DBG] 4.4 scrub ok 2021-02-10T04:33:51.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:51 smithi085 conmon[35961]: cluster 2021-02-10T04:33:50.217335+0000 2021-02-10T04:33:51.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:51 smithi085 conmon[35961]: mon.a (mon.0) 1205 : cluster [DBG] osdmap e413: 8 total, 8 up, 8 in 2021-02-10T04:33:51.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:51 smithi085 conmon[35961]: 2021-02-10T04:33:51.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:51 smithi085 conmon[35961]: cluster 2021-02-10T04:33:51.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:51 smithi085 conmon[35961]: 2021-02-10T04:33:50.351530+0000 osd.0 (osd. 2021-02-10T04:33:51.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:51 smithi085 conmon[35961]: 0) 66 : cluster [DBG] 7.f scrub ok 2021-02-10T04:33:51.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:51 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:33:51.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:51 smithi085 conmon[35961]: :33:50.581996+0000 osd.5 (osd.5 2021-02-10T04:33:51.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:51 smithi085 conmon[35961]: ) 60 : cluster [DBG] 7.0 scrub ok 2021-02-10T04:33:51.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:51 smithi085 conmon[35961]: cluster 2021-02-10T04:33 2021-02-10T04:33:51.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:51 smithi085 conmon[35961]: :50.809465+0000 osd.3 (osd.3) 63 : 2021-02-10T04:33:51.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:51 smithi085 conmon[35961]: cluster [DBG] 6.13 scrub ok 2021-02-10T04:33:51.696 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:33:51 smithi085 conmon[45443]: debug 2021-02-10T04:33:51.309+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:51.762 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:33:52.230 INFO:tasks.rados.rados.0.smithi085.stdout:1545: done (10 left) 2021-02-10T04:33:52.231 INFO:tasks.rados.rados.0.smithi085.stdout:1546: done (9 left) 2021-02-10T04:33:52.231 INFO:tasks.rados.rados.0.smithi085.stdout:1547: snap_create 2021-02-10T04:33:52.484 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:52 smithi085 conmon[29319]: cluster 2021-02-10T04:33:51.222988+0000 2021-02-10T04:33:52.485 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:52 smithi085 conmon[29319]: mon.a (mon.0) 1206 : cluster [DBG] osdmap e414: 8 total, 8 up, 8 in 2021-02-10T04:33:52.485 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:52 smithi085 conmon[29319]: cluster 2021-02-10T04:33:51.858380+0000 osd.3 (osd.3) 64 : cluster [DBG] 1.9 scrub ok 2021-02-10T04:33:52.485 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:52 smithi085 conmon[29319]: cluster 2021-02-10T04:33:52.077343+0000 mgr.y (mgr.14140) 534 : cluster [DBG] pgmap v771: 168 pgs: 4 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 157 active+clean; 325 MiB data, 980 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 1.3 MiB/s wr, 16 op/s 2021-02-10T04:33:52.485 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:52 smithi085 conmon[35961]: cluster 2021-02-10T04:33:51.222988+0000 mon.a 2021-02-10T04:33:52.485 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:52 smithi085 conmon[35961]: (mon.0) 1206 : cluster [DBG] osdmap e414: 8 total, 8 up, 8 in 2021-02-10T04:33:52.485 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:52 smithi085 conmon[35961]: cluster 2021-02-10T04:33:51.858380+0000 osd.3 (osd.3) 2021-02-10T04:33:52.485 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:52 smithi085 conmon[35961]: 64 : cluster [DBG] 1.9 scrub ok 2021-02-10T04:33:52.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:52 smithi085 conmon[35961]: cluster 2021-02-10T04:33:52.077343 2021-02-10T04:33:52.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:52 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 534 : cluster [DBG] 2021-02-10T04:33:52.486 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:52 smithi085 conmon[35961]: pgmap v771: 168 pgs: 4 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 157 active+clean; 325 MiB data, 980 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 1.3 MiB/s wr, 16 op/s 2021-02-10T04:33:52.542 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:33:52.577 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:52 smithi164 conmon[29288]: cluster 2021-02-10T04:33:52.577 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:52 smithi164 conmon[29288]: 2021-02-10T04:33:52.577 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:52 smithi164 conmon[29288]: 2021-02-10T04:33:51 2021-02-10T04:33:52.577 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:52 smithi164 conmon[29288]: .222988+0000 mon.a (mon.0) 1206 : cluster [DBG] osdmap e414: 8 total, 8 up, 8 in 2021-02-10T04:33:52.577 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:52 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:33:52.577 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:52 smithi164 conmon[29288]: 02-10T04:33:51.858380+0000 osd.3 (osd.3) 2021-02-10T04:33:52.577 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:52 smithi164 conmon[29288]: 64 : cluster [DBG] 1.9 scrub ok 2021-02-10T04:33:52.577 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:52 smithi164 conmon[29288]: cluster 2021-02-10T04:33:52.578 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:52 smithi164 conmon[29288]: 2021-02-10T04:33: 2021-02-10T04:33:52.578 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:52 smithi164 conmon[29288]: 52.077343+0000 mgr.y (mgr.14140) 534 : 2021-02-10T04:33:52.578 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:52 smithi164 conmon[29288]: cluster [DBG] pgmap v771: 168 pgs: 4 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 157 active+clean; 325 MiB data, 980 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 1.3 MiB/s wr, 16 op/s 2021-02-10T04:33:52.579 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:52.760 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.3 dump_ops_in_flight 2021-02-10T04:33:52.893 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:33:52 smithi164 conmon[40892]: debug 2021-02-10T04:33:52.647+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:53.025 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:33:53.130 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:33:53.159 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:53.211 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:33:53.235 INFO:tasks.rados.rados.0.smithi085.stdout:1547: done (9 left) 2021-02-10T04:33:53.235 INFO:tasks.rados.rados.0.smithi085.stdout:1548: copy_from oid 46 from oid 21 current snap is 171 2021-02-10T04:33:53.235 INFO:tasks.rados.rados.0.smithi085.stdout:1549: copy_from oid 40 from oid 44 current snap is 171 2021-02-10T04:33:53.235 INFO:tasks.rados.rados.0.smithi085.stdout:1550: delete oid 16 current snap is 171 2021-02-10T04:33:53.247 INFO:tasks.rados.rados.0.smithi085.stdout:1550: done (11 left) 2021-02-10T04:33:53.247 INFO:tasks.rados.rados.0.smithi085.stdout:1551: delete oid 26 current snap is 171 2021-02-10T04:33:53.251 INFO:tasks.rados.rados.0.smithi085.stdout:1551: done (11 left) 2021-02-10T04:33:53.251 INFO:tasks.rados.rados.0.smithi085.stdout:1552: write (excl) oid 36 current snap is 171 2021-02-10T04:33:53.252 INFO:tasks.rados.rados.0.smithi085.stdout:1552: seq_num 551 ranges {460657=639615,1693343=476514,2681716=578941} 2021-02-10T04:33:53.258 INFO:tasks.rados.rados.0.smithi085.stdout:1552: writing smithi08575397-36 from 460657 to 1100272 tid 1 2021-02-10T04:33:53.267 INFO:tasks.rados.rados.0.smithi085.stdout:1552: writing smithi08575397-36 from 1693343 to 2169857 tid 2 2021-02-10T04:33:53.275 INFO:tasks.rados.rados.0.smithi085.stdout:1552: writing smithi08575397-36 from 2681716 to 3260657 tid 3 2021-02-10T04:33:53.275 INFO:tasks.rados.rados.0.smithi085.stdout:1553: snap_create 2021-02-10T04:33:53.275 INFO:tasks.rados.rados.0.smithi085.stdout:1548: finishing copy_from to smithi08575397-46 2021-02-10T04:33:53.275 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 739 (ObjNum 431 snap 114 seq_num 431) dirty exists 2021-02-10T04:33:53.275 INFO:tasks.rados.rados.0.smithi085.stdout:1548: finishing copy_from racing read to smithi08575397-46 2021-02-10T04:33:53.276 INFO:tasks.rados.rados.0.smithi085.stdout:1549: finishing copy_from to smithi08575397-40 2021-02-10T04:33:53.276 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 625 (ObjNum 491 snap 141 seq_num 491) dirty exists 2021-02-10T04:33:53.276 INFO:tasks.rados.rados.0.smithi085.stdout:1549: finishing copy_from racing read to smithi08575397-40 2021-02-10T04:33:53.276 INFO:tasks.rados.rados.0.smithi085.stdout:1552: finishing write tid 1 to smithi08575397-36 2021-02-10T04:33:53.277 INFO:tasks.rados.rados.0.smithi085.stdout:1552: finishing write tid 2 to smithi08575397-36 2021-02-10T04:33:53.280 INFO:tasks.rados.rados.0.smithi085.stdout:1552: finishing write tid 3 to smithi08575397-36 2021-02-10T04:33:53.281 INFO:tasks.rados.rados.0.smithi085.stdout:1552: finishing write tid 4 to smithi08575397-36 2021-02-10T04:33:53.281 INFO:tasks.rados.rados.0.smithi085.stdout:1552: finishing write tid 5 to smithi08575397-36 2021-02-10T04:33:53.281 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 632 (ObjNum 551 snap 171 seq_num 551) dirty exists 2021-02-10T04:33:53.281 INFO:tasks.rados.rados.0.smithi085.stdout:1552: left oid 36 (ObjNum 551 snap 171 seq_num 551) 2021-02-10T04:33:53.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:53 smithi164 conmon[29288]: cluster 2021-02-10T04:33:52.230716+0000 mon.a (mon.0) 1207 : cluster [DBG] osdmap e415: 8 total, 8 up, 8 in 2021-02-10T04:33:53.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:53 smithi164 conmon[29288]: cluster 2021-02-10T04:33:52.643095+0000 osd.4 (osd.4) 498 : cluster [DBG] 5.1c scrub ok 2021-02-10T04:33:53.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:53 smithi164 conmon[33677]: debug 2021-02-10T04:33:53.242+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:53.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:53 smithi085 conmon[29319]: cluster 2021-02-10T04:33:53.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:53 smithi085 conmon[29319]: 2021-02-10T04:33:53.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:53 smithi085 conmon[29319]: 2021 2021-02-10T04:33:53.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:53 smithi085 conmon[29319]: -02-10T04:33:52. 2021-02-10T04:33:53.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:53 smithi085 conmon[29319]: 230716+0000 mon.a (mon.0) 1207 : cluster [DBG] osdmap e415: 8 total, 8 up, 8 in 2021-02-10T04:33:53.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:53 smithi085 conmon[29319]: cluster 2021-02-10T04:33:53.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:53 smithi085 conmon[29319]: 2021-02-10T04:33:53.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:53 smithi085 conmon[29319]: 2021 2021-02-10T04:33:53.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:53 smithi085 conmon[29319]: - 2021-02-10T04:33:53.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:53 smithi085 conmon[29319]: 02 2021-02-10T04:33:53.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:53 smithi085 conmon[29319]: - 2021-02-10T04:33:53.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:53 smithi085 conmon[29319]: 10 2021-02-10T04:33:53.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:53 smithi085 conmon[29319]: T 2021-02-10T04:33:53.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:53 smithi085 conmon[29319]: 04 2021-02-10T04:33:53.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:53 smithi085 conmon[29319]: :33:52.643095 2021-02-10T04:33:53.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:53 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 498 : cluster [DBG] 5.1c scrub ok 2021-02-10T04:33:53.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:53 smithi085 conmon[35961]: cluster 2021-02-10T04:33:53.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:53 smithi085 conmon[35961]: 2021 2021-02-10T04:33:53.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:53 smithi085 conmon[35961]: -02-10T04:33:52.230716+0000 mon.a (mon.0) 1207 : cluster 2021-02-10T04:33:53.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:53 smithi085 conmon[35961]: [DBG] osdmap e415: 8 total, 8 up, 8 in 2021-02-10T04:33:53.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:53 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:53.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:53 smithi085 conmon[35961]: -02-10T04:33:52.643095+0000 2021-02-10T04:33:53.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:53 smithi085 conmon[35961]: osd.4 (osd.4) 498 : cluster [DBG] 2021-02-10T04:33:53.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:53 smithi085 conmon[35961]: 5.1c scrub ok 2021-02-10T04:33:53.488 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:33:53.594 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:33:53.624 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:54.266 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[29319]: cluster 2021-02-10T04:33:54.266 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[29319]: 2021-02-10T04:33:54.266 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[29319]: 2021 2021-02-10T04:33:54.266 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[29319]: - 2021-02-10T04:33:54.267 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[29319]: 02 2021-02-10T04:33:54.267 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[29319]: - 2021-02-10T04:33:54.267 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[29319]: 10 2021-02-10T04:33:54.267 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[29319]: T 2021-02-10T04:33:54.267 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[29319]: 04:33:52.827250+0000 osd.6 2021-02-10T04:33:54.267 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[29319]: (osd.6) 55 : cluster [DBG] 5.2 scrub ok 2021-02-10T04:33:54.267 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:33:54.267 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[29319]: -10T04:33:53.234650+0000 mon.a ( 2021-02-10T04:33:54.267 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[29319]: mon.0) 1208 : cluster [DBG] osdmap e416: 8 total, 8 up, 8 in 2021-02-10T04:33:54.267 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[29319]: 2021-02-10T04:33:54.267 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:33:54.267 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[29319]: 33:54.077763+0000 mgr.y (mgr.14140) 2021-02-10T04:33:54.267 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[29319]: 535 : cluster [DBG] pgmap v774: 168 pgs: 4 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 156 active+clean; 325 MiB data, 972 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 1.3 MiB/s wr, 13 op/s 2021-02-10T04:33:54.268 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[29319]: cephadm 2021-02-10T04:33:54.268 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[29319]: 2021-02-10T04:33 2021-02-10T04:33:54.268 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[29319]: :54.133219+0000 mgr.y (mgr.14140) 536 2021-02-10T04:33:54.268 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[29319]: : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:33:54.268 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[29319]: cephadm 2021-02-10T 2021-02-10T04:33:54.268 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[29319]: 04:33:54.133749+0000 mgr.y (mgr. 2021-02-10T04:33:54.268 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[29319]: 14140) 537 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:33:54.268 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: cluster 2021-02-10T04:33:54.268 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 2021-02-10T04:33:54.268 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 2021 2021-02-10T04:33:54.268 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: - 2021-02-10T04:33:54.268 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 02 2021-02-10T04:33:54.269 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: - 2021-02-10T04:33:54.269 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 10 2021-02-10T04:33:54.269 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: T 2021-02-10T04:33:54.269 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 04 2021-02-10T04:33:54.269 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: : 2021-02-10T04:33:54.269 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 33 2021-02-10T04:33:54.269 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: : 2021-02-10T04:33:54.269 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 52 2021-02-10T04:33:54.269 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: . 2021-02-10T04:33:54.269 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 827250 2021-02-10T04:33:54.269 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: +0000 2021-02-10T04:33:54.269 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 2021-02-10T04:33:54.269 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: osd.6 2021-02-10T04:33:54.270 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: ( 2021-02-10T04:33:54.270 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: osd 2021-02-10T04:33:54.270 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: . 2021-02-10T04:33:54.270 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 6 2021-02-10T04:33:54.270 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: ) 2021-02-10T04:33:54.270 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 55 2021-02-10T04:33:54.270 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: : 2021-02-10T04:33:54.270 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: cluster 2021-02-10T04:33:54.270 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 2021-02-10T04:33:54.270 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: [DBG] 2021-02-10T04:33:54.270 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 2021-02-10T04:33:54.270 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 5.2 scrub ok 2021-02-10T04:33:54.270 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 2021-02-10T04:33:54.271 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: cluster 2021-02-10T04:33:54.271 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 2021-02-10T04:33:54.271 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 2021 2021-02-10T04:33:54.271 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: - 2021-02-10T04:33:54.271 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 02 2021-02-10T04:33:54.271 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: - 2021-02-10T04:33:54.271 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 10 2021-02-10T04:33:54.271 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: T 2021-02-10T04:33:54.271 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 04 2021-02-10T04:33:54.271 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: : 2021-02-10T04:33:54.271 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 33 2021-02-10T04:33:54.271 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: : 2021-02-10T04:33:54.271 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 53 2021-02-10T04:33:54.272 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: . 2021-02-10T04:33:54.272 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 234650 2021-02-10T04:33:54.272 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: +0000 2021-02-10T04:33:54.272 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 2021-02-10T04:33:54.272 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: mon.a 2021-02-10T04:33:54.272 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: ( 2021-02-10T04:33:54.272 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: mon 2021-02-10T04:33:54.272 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: . 2021-02-10T04:33:54.272 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 0 2021-02-10T04:33:54.272 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: ) 2021-02-10T04:33:54.272 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 1208 2021-02-10T04:33:54.272 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: : 2021-02-10T04:33:54.272 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: cluster 2021-02-10T04:33:54.272 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 2021-02-10T04:33:54.273 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: [DBG] 2021-02-10T04:33:54.273 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 2021-02-10T04:33:54.273 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: osdmap e416: 8 total, 8 up, 8 in 2021-02-10T04:33:54.273 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 2021-02-10T04:33:54.273 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: cluster 2021-02-10T04:33:54.273 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 2021-02-10T04:33:54.273 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 2021 2021-02-10T04:33:54.273 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: - 2021-02-10T04:33:54.273 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 02 2021-02-10T04:33:54.273 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: - 2021-02-10T04:33:54.273 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 10 2021-02-10T04:33:54.273 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: T 2021-02-10T04:33:54.273 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 04 2021-02-10T04:33:54.273 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: : 2021-02-10T04:33:54.274 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 33 2021-02-10T04:33:54.274 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: : 2021-02-10T04:33:54.274 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 54.077763+0000 mgr.y (mgr. 2021-02-10T04:33:54.274 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 14140) 535 : cluster [DBG] pgmap v774: 168 pgs: 4 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 156 active+clean; 325 MiB data, 972 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 1.3 MiB/s wr, 13 op/s 2021-02-10T04:33:54.274 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: cephadm 2021-02 2021-02-10T04:33:54.274 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: -10T04:33:54. 2021-02-10T04:33:54.274 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 133219+0000 mgr.y (mgr. 2021-02-10T04:33:54.274 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: 14140) 536 : cephadm [INF] 2021-02-10T04:33:54.274 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: Refreshing smithi085 facts 2021-02-10T04:33:54.274 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: cephadm 2021 2021-02-10T04:33:54.274 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: -02-10T04:33 2021-02-10T04:33:54.274 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: :54.133749+0000 mgr.y ( 2021-02-10T04:33:54.274 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: mgr.14140) 537 : cephadm [INF] 2021-02-10T04:33:54.275 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[35961]: Refreshing smithi164 facts 2021-02-10T04:33:54.275 INFO:tasks.rados.rados.0.smithi085.stdout:1548: done (12 left) 2021-02-10T04:33:54.275 INFO:tasks.rados.rados.0.smithi085.stdout:1549: done (11 left) 2021-02-10T04:33:54.275 INFO:tasks.rados.rados.0.smithi085.stdout:1552: done (10 left) 2021-02-10T04:33:54.275 INFO:tasks.rados.rados.0.smithi085.stdout:1553: done (9 left) 2021-02-10T04:33:54.275 INFO:tasks.rados.rados.0.smithi085.stdout:1554: copy_from oid 26 from oid 31 current snap is 172 2021-02-10T04:33:54.275 INFO:tasks.rados.rados.0.smithi085.stdout:1555: delete oid 36 current snap is 172 2021-02-10T04:33:54.275 INFO:tasks.rados.rados.0.smithi085.stdout:1555: done (10 left) 2021-02-10T04:33:54.275 INFO:tasks.rados.rados.0.smithi085.stdout:1556: read oid 30 snap -1 2021-02-10T04:33:54.275 INFO:tasks.rados.rados.0.smithi085.stdout:1556: expect (ObjNum 520 snap 155 seq_num 520) 2021-02-10T04:33:54.275 INFO:tasks.rados.rados.0.smithi085.stdout:1557: write (excl) oid 7 current snap is 172 2021-02-10T04:33:54.275 INFO:tasks.rados.rados.0.smithi085.stdout:1557: seq_num 553 ranges {494271=790113,1693954=480183,2620209=512129} 2021-02-10T04:33:54.276 INFO:tasks.rados.rados.0.smithi085.stdout:1557: writing smithi08575397-7 from 494271 to 1284384 tid 1 2021-02-10T04:33:54.278 INFO:tasks.rados.rados.0.smithi085.stdout:1557: writing smithi08575397-7 from 1693954 to 2174137 tid 2 2021-02-10T04:33:54.284 INFO:tasks.rados.rados.0.smithi085.stdout:1557: writing smithi08575397-7 from 2620209 to 3132338 tid 3 2021-02-10T04:33:54.290 INFO:tasks.rados.rados.0.smithi085.stdout:1556: expect (ObjNum 520 snap 155 seq_num 520) 2021-02-10T04:33:54.399 INFO:tasks.rados.rados.0.smithi085.stdout:1557: finishing write tid 1 to smithi08575397-7 2021-02-10T04:33:54.400 INFO:tasks.rados.rados.0.smithi085.stdout:1556: done (11 left) 2021-02-10T04:33:54.400 INFO:tasks.rados.rados.0.smithi085.stdout:1558: copy_from oid 44 from oid 14 current snap is 172 2021-02-10T04:33:54.400 INFO:tasks.rados.rados.0.smithi085.stdout:1557: finishing write tid 2 to smithi08575397-7 2021-02-10T04:33:54.400 INFO:tasks.rados.rados.0.smithi085.stdout:1557: finishing write tid 3 to smithi08575397-7 2021-02-10T04:33:54.400 INFO:tasks.rados.rados.0.smithi085.stdout:1557: finishing write tid 4 to smithi08575397-7 2021-02-10T04:33:54.400 INFO:tasks.rados.rados.0.smithi085.stdout:1557: finishing write tid 5 to smithi08575397-7 2021-02-10T04:33:54.400 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 655 (ObjNum 553 snap 172 seq_num 553) dirty exists 2021-02-10T04:33:54.400 INFO:tasks.rados.rados.0.smithi085.stdout:1557: left oid 7 (ObjNum 553 snap 172 seq_num 553) 2021-02-10T04:33:54.400 INFO:tasks.rados.rados.0.smithi085.stdout:1557: done (11 left) 2021-02-10T04:33:54.400 INFO:tasks.rados.rados.0.smithi085.stdout:1559: copy_from oid 46 from oid 8 current snap is 172 2021-02-10T04:33:54.400 INFO:tasks.rados.rados.0.smithi085.stdout:1560: read oid 21 snap -1 2021-02-10T04:33:54.401 INFO:tasks.rados.rados.0.smithi085.stdout:1560: expect (ObjNum 431 snap 114 seq_num 431) 2021-02-10T04:33:54.401 INFO:tasks.rados.rados.0.smithi085.stdout:1561: snap_remove snap 137 2021-02-10T04:33:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:54 smithi164 conmon[29288]: cluster 2021-02-10T04:33:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:54 smithi164 conmon[29288]: 2021-02-10T04:33:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:54 smithi164 conmon[29288]: 2021 2021-02-10T04:33:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:54 smithi164 conmon[29288]: - 2021-02-10T04:33:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:54 smithi164 conmon[29288]: 02 2021-02-10T04:33:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:54 smithi164 conmon[29288]: -10T04:33:52 2021-02-10T04:33:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:54 smithi164 conmon[29288]: .827250+0000 osd.6 (osd.6) 55 : cluster [DBG] 5.2 scrub ok 2021-02-10T04:33:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:54 smithi164 conmon[29288]: cluster 2021-02-10T04:33: 2021-02-10T04:33:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:54 smithi164 conmon[29288]: 53.234650+0000 mon.a (mon.0) 1208 : cluster [DBG] osdmap e416: 8 total, 8 up, 8 in 2021-02-10T04:33:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:54 smithi164 conmon[29288]: 2021-02-10T04:33:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:54 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:33:54.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:54 smithi164 conmon[29288]: :33:54.077763+0000 mgr.y (mgr. 2021-02-10T04:33:54.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:54 smithi164 conmon[29288]: 14140) 535 : cluster [DBG] pgmap v774: 168 pgs: 4 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 156 active+clean; 325 MiB data, 972 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 1.3 MiB/s wr, 13 op/s 2021-02-10T04:33:54.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:54 smithi164 conmon[29288]: cephadm 2021-02 2021-02-10T04:33:54.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:54 smithi164 conmon[29288]: -10T04:33:54.133219+0000 mgr.y 2021-02-10T04:33:54.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:54 smithi164 conmon[29288]: (mgr.14140) 536 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:33:54.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:54 smithi164 conmon[29288]: cephadm 2021-02 2021-02-10T04:33:54.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:54 smithi164 conmon[29288]: -10T04:33:54.133749+0000 mgr.y (mgr.14140) 2021-02-10T04:33:54.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:54 smithi164 conmon[29288]: 537 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:33:54.791 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset nodeep-scrub 2021-02-10T04:33:55.142 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:33:54 smithi085 conmon[41540]: debug 2021-02-10T04:33:54.938+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:55.151 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:33:55.178 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:33:55.275 INFO:tasks.rados.rados.0.smithi085.stdout:1561: done (13 left) 2021-02-10T04:33:55.275 INFO:tasks.rados.rados.0.smithi085.stdout:1562: copy_from oid 50 from oid 40 current snap is 172 2021-02-10T04:33:55.275 INFO:tasks.rados.rados.0.smithi085.stdout:1563: read oid 6 snap -1 2021-02-10T04:33:55.275 INFO:tasks.rados.rados.0.smithi085.stdout:1563: expect deleted 2021-02-10T04:33:55.276 INFO:tasks.rados.rados.0.smithi085.stdout:1564: delete oid 30 current snap is 172 2021-02-10T04:33:55.276 INFO:tasks.rados.rados.0.smithi085.stdout:1560: expect (ObjNum 431 snap 114 seq_num 431) 2021-02-10T04:33:55.287 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:33:55.315 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:55.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[29319]: cluster 2021-02-10T04:33:53.834623+0000 2021-02-10T04:33:55.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[29319]: osd.6 (osd.6) 56 : cluster [DBG] 7.d scrub ok 2021-02-10T04:33:55.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[29319]: cluster 2021-02-10T04:33:54.251741+0000 mon.a (mon.0) 1209 : cluster [DBG] osdmap e417: 8 total, 8 up, 8 in 2021-02-10T04:33:55.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[29319]: audit 2021-02-10T04:33:55.117385+0000 mon.a (mon.0) 1210 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:33:55.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: cluster 2021-02-10T04:33:55.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: 2021-02-10T04:33:55.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: 2021 2021-02-10T04:33:55.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: - 2021-02-10T04:33:55.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: 02 2021-02-10T04:33:55.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: - 2021-02-10T04:33:55.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: 10 2021-02-10T04:33:55.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: T 2021-02-10T04:33:55.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: 04 2021-02-10T04:33:55.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: : 2021-02-10T04:33:55.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: 33 2021-02-10T04:33:55.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: : 2021-02-10T04:33:55.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: 53 2021-02-10T04:33:55.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: . 2021-02-10T04:33:55.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: 834623 2021-02-10T04:33:55.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: +0000 2021-02-10T04:33:55.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: 2021-02-10T04:33:55.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: osd.6 2021-02-10T04:33:55.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: ( 2021-02-10T04:33:55.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: osd 2021-02-10T04:33:55.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: . 2021-02-10T04:33:55.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: 6 2021-02-10T04:33:55.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: ) 2021-02-10T04:33:55.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: 56 2021-02-10T04:33:55.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: : 2021-02-10T04:33:55.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: cluster 2021-02-10T04:33:55.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: 2021-02-10T04:33:55.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: [DBG] 2021-02-10T04:33:55.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: 2021-02-10T04:33:55.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: 7.d scrub ok 2021-02-10T04:33:55.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: 2021-02-10T04:33:55.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: cluster 2021-02-10T04:33:55.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: 2021-02-10T04:33: 2021-02-10T04:33:55.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: 54.251741+0000 mon.a (mon.0) 1209 : cluster [DBG] osdmap e417: 8 total, 8 up, 8 in 2021-02-10T04:33:55.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:33:55.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: -10T04:33:55.117385+0000 mon.a (mon.0) 2021-02-10T04:33:55.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:55 smithi085 conmon[35961]: 1210 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:33:55.496 INFO:tasks.rados.rados.0.smithi085.stdout:1560: done (15 left) 2021-02-10T04:33:55.496 INFO:tasks.rados.rados.0.smithi085.stdout:1564: done (14 left) 2021-02-10T04:33:55.496 INFO:tasks.rados.rados.0.smithi085.stdout:1565: snap_create 2021-02-10T04:33:55.496 INFO:tasks.rados.rados.0.smithi085.stdout:1559: finishing copy_from to smithi08575397-46 2021-02-10T04:33:55.496 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 599 (ObjNum 521 snap 155 seq_num 521) dirty exists 2021-02-10T04:33:55.496 INFO:tasks.rados.rados.0.smithi085.stdout:1559: finishing copy_from racing read to smithi08575397-46 2021-02-10T04:33:55.496 INFO:tasks.rados.rados.0.smithi085.stdout:1554: finishing copy_from to smithi08575397-26 2021-02-10T04:33:55.496 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 640 (ObjNum 517 snap 152 seq_num 517) dirty exists 2021-02-10T04:33:55.496 INFO:tasks.rados.rados.0.smithi085.stdout:1554: finishing copy_from racing read to smithi08575397-26 2021-02-10T04:33:55.496 INFO:tasks.rados.rados.0.smithi085.stdout:1558: finishing copy_from to smithi08575397-44 2021-02-10T04:33:55.497 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 719 (ObjNum 525 snap 155 seq_num 525) dirty exists 2021-02-10T04:33:55.497 INFO:tasks.rados.rados.0.smithi085.stdout:1558: finishing copy_from racing read to smithi08575397-44 2021-02-10T04:33:55.497 INFO:tasks.rados.rados.0.smithi085.stdout:1562: finishing copy_from to smithi08575397-50 2021-02-10T04:33:55.497 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 625 (ObjNum 491 snap 141 seq_num 491) dirty exists 2021-02-10T04:33:55.497 INFO:tasks.rados.rados.0.smithi085.stdout:1562: finishing copy_from racing read to smithi08575397-50 2021-02-10T04:33:55.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:55 smithi164 conmon[29288]: cluster 2021-02-10T04:33:53.834623+0000 2021-02-10T04:33:55.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:55 smithi164 conmon[29288]: osd.6 (osd.6) 56 : cluster [DBG] 7.d scrub ok 2021-02-10T04:33:55.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:55 smithi164 conmon[29288]: cluster 2021-02-10T04:33: 2021-02-10T04:33:55.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:55 smithi164 conmon[29288]: 54.251741+0000 mon.a (mon.0) 1209 : cluster [DBG] osdmap e417: 8 total, 8 up, 8 in 2021-02-10T04:33:55.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:55 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:33:55.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:55 smithi164 conmon[29288]: 02-10T04:33:55.117385+0000 mon.a (mon.0) 2021-02-10T04:33:55.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:55 smithi164 conmon[29288]: 1210 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:33:55.645 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:33:55 smithi164 conmon[44544]: debug 2021-02-10T04:33:55.377+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:55.801 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:33:55.906 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:33:55.944 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:33:56.268 INFO:tasks.rados.rados.0.smithi085.stdout:1554: done (14 left) 2021-02-10T04:33:56.268 INFO:tasks.rados.rados.0.smithi085.stdout:1558: done (13 left) 2021-02-10T04:33:56.268 INFO:tasks.rados.rados.0.smithi085.stdout:1559: done (12 left) 2021-02-10T04:33:56.268 INFO:tasks.rados.rados.0.smithi085.stdout:1562: done (11 left) 2021-02-10T04:33:56.268 INFO:tasks.rados.rados.0.smithi085.stdout:1563: done (10 left) 2021-02-10T04:33:56.269 INFO:tasks.rados.rados.0.smithi085.stdout:1565: done (9 left) 2021-02-10T04:33:56.269 INFO:tasks.rados.rados.0.smithi085.stdout:1566: read oid 37 snap 167 2021-02-10T04:33:56.269 INFO:tasks.rados.rados.0.smithi085.stdout:1566: expect deleted 2021-02-10T04:33:56.269 INFO:tasks.rados.rados.0.smithi085.stdout:1567: rollback oid 12 current snap is 173 2021-02-10T04:33:56.269 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 12 to 147 2021-02-10T04:33:56.269 INFO:tasks.rados.rados.0.smithi085.stdout:1568: read oid 11 snap -1 2021-02-10T04:33:56.269 INFO:tasks.rados.rados.0.smithi085.stdout:1568: expect (ObjNum 531 snap 156 seq_num 531) 2021-02-10T04:33:56.269 INFO:tasks.rados.rados.0.smithi085.stdout:1569: copy_from oid 6 from oid 5 current snap is 173 2021-02-10T04:33:56.269 INFO:tasks.rados.rados.0.smithi085.stdout:1570: read oid 36 snap -1 2021-02-10T04:33:56.269 INFO:tasks.rados.rados.0.smithi085.stdout:1570: expect deleted 2021-02-10T04:33:56.269 INFO:tasks.rados.rados.0.smithi085.stdout:1571: delete oid 16 current snap is 173 2021-02-10T04:33:56.273 INFO:tasks.rados.rados.0.smithi085.stdout:1567: finishing rollback tid 0 to smithi08575397-12 2021-02-10T04:33:56.275 INFO:tasks.rados.rados.0.smithi085.stdout:1567: finishing rollback tid 1 to smithi08575397-12 2021-02-10T04:33:56.275 INFO:tasks.rados.rados.0.smithi085.stdout:1567: finishing rollback tid 2 to smithi08575397-12 2021-02-10T04:33:56.275 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 730 (ObjNum 85 snap 11 seq_num 85) dirty exists 2021-02-10T04:33:56.286 INFO:tasks.rados.rados.0.smithi085.stdout:1568: expect (ObjNum 531 snap 156 seq_num 531) 2021-02-10T04:33:56.372 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:56 smithi085 conmon[29319]: cluster 2021-02-10T04:33:55.261390+0000 mon.a (mon.0) 1211 : cluster [DBG] osdmap e418: 8 total, 8 up, 8 in 2021-02-10T04:33:56.373 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:56 smithi085 conmon[29319]: cluster 2021-02-10T04:33:55.288029+0000 osd.0 (osd.0) 67 : cluster [DBG] 4.6 scrub ok 2021-02-10T04:33:56.373 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:56 smithi085 conmon[29319]: cluster 2021-02-10T04:33:55.585803+0000 osd.5 (osd.5) 61 : cluster [DBG] 4.8 scrub ok 2021-02-10T04:33:56.373 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:56 smithi085 conmon[29319]: cluster 2021-02-10T04:33:55.849566+0000 osd.3 (osd.3) 65 : cluster [DBG] 6.c scrub ok 2021-02-10T04:33:56.373 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:56 smithi085 conmon[29319]: cluster 2021-02-10T04:33:56.078275+0000 mgr.y (mgr.14140) 538 : cluster [DBG] pgmap v777: 168 pgs: 3 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 157 active+clean; 331 MiB data, 982 MiB used, 714 GiB / 715 GiB avail; 6.7 MiB/s rd, 5.8 MiB/s wr, 17 op/s 2021-02-10T04:33:56.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:56 smithi085 conmon[35961]: cluster 2021-02-10T04:33:56.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:56 smithi085 conmon[35961]: 2021-02-10T04:33:56.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:56 smithi085 conmon[35961]: 2021 2021-02-10T04:33:56.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:56 smithi085 conmon[35961]: - 2021-02-10T04:33:56.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:56 smithi085 conmon[35961]: 02 2021-02-10T04:33:56.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:56 smithi085 conmon[35961]: - 2021-02-10T04:33:56.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:56 smithi085 conmon[35961]: 10 2021-02-10T04:33:56.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:56 smithi085 conmon[35961]: T04:33:55.261390 2021-02-10T04:33:56.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:56 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1211 : cluster [DBG] osdmap e418: 8 total, 8 up, 8 in 2021-02-10T04:33:56.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:56 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:33:56.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:56 smithi085 conmon[35961]: -02-10T04:33:55.288029+0000 2021-02-10T04:33:56.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:56 smithi085 conmon[35961]: osd.0 (osd.0) 67 : cluster [DBG] 4.6 scrub ok 2021-02-10T04:33:56.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:56 smithi085 conmon[35961]: 2021-02-10T04:33:56.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:56 smithi085 conmon[35961]: cluster 2021-02-10T04:33: 2021-02-10T04:33:56.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:56 smithi085 conmon[35961]: 55.585803+0000 osd.5 (osd.5) 61 : 2021-02-10T04:33:56.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:56 smithi085 conmon[35961]: cluster [DBG] 4.8 scrub ok 2021-02-10T04:33:56.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:56 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:33:56.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:56 smithi085 conmon[35961]: 04:33:55.849566+0000 osd.3 (osd.3 2021-02-10T04:33:56.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:56 smithi085 conmon[35961]: ) 65 : cluster [DBG] 6.c scrub ok 2021-02-10T04:33:56.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:56 smithi085 conmon[35961]: cluster 2021-02-10T04:33:56.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:56 smithi085 conmon[35961]: 2021-02-10T04:33:56.078275+0000 2021-02-10T04:33:56.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:56 smithi085 conmon[35961]: mgr.y (mgr.14140) 538 : cluster [DBG] pgmap v777: 168 pgs: 3 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 157 active+clean; 331 MiB data, 982 MiB used, 714 GiB / 715 GiB avail; 6.7 MiB/s rd, 5.8 MiB/s wr, 17 op/s 2021-02-10T04:33:56.406 INFO:tasks.rados.rados.0.smithi085.stdout:1566: done (14 left) 2021-02-10T04:33:56.406 INFO:tasks.rados.rados.0.smithi085.stdout:1567: done (13 left) 2021-02-10T04:33:56.407 INFO:tasks.rados.rados.0.smithi085.stdout:1568: done (12 left) 2021-02-10T04:33:56.407 INFO:tasks.rados.rados.0.smithi085.stdout:1570: done (11 left) 2021-02-10T04:33:56.407 INFO:tasks.rados.rados.0.smithi085.stdout:1571: done (10 left) 2021-02-10T04:33:56.407 INFO:tasks.rados.rados.0.smithi085.stdout:1572: snap_remove snap 164 2021-02-10T04:33:56.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:56 smithi164 conmon[29288]: cluster 2021-02-10T04:33:56.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:56 smithi164 conmon[29288]: 2021-02-10T04:33:56.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:56 smithi164 conmon[29288]: 2021 2021-02-10T04:33:56.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:56 smithi164 conmon[29288]: - 2021-02-10T04:33:56.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:56 smithi164 conmon[29288]: 02 2021-02-10T04:33:56.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:56 smithi164 conmon[29288]: - 2021-02-10T04:33:56.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:56 smithi164 conmon[29288]: 10T04:33:55.261390+0000 mon.a ( 2021-02-10T04:33:56.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:56 smithi164 conmon[29288]: mon.0) 1211 : cluster [DBG] osdmap e418: 8 total, 8 up, 8 in 2021-02-10T04:33:56.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:56 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:33:56.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:56 smithi164 conmon[29288]: -02-10T04:33:55.288029+0000 osd.0 (osd.0) 67 : cluster [DBG] 4.6 scrub ok 2021-02-10T04:33:56.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:56 smithi164 conmon[29288]: cluster 2021-02-10T04:33:55.585803+0000 osd.5 ( 2021-02-10T04:33:56.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:56 smithi164 conmon[29288]: osd.5) 61 : cluster [DBG] 4.8 scrub ok 2021-02-10T04:33:56.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:56 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:33:56.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:56 smithi164 conmon[29288]: 10T04:33:55.849566+0000 osd.3 ( 2021-02-10T04:33:56.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:56 smithi164 conmon[29288]: osd.3) 65 : cluster [DBG] 6.c scrub ok 2021-02-10T04:33:56.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:56 smithi164 conmon[29288]: 2021-02-10T04:33:56.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:56 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:33:56.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:56 smithi164 conmon[29288]: T04:33:56.078275+0000 mgr.y (mgr. 2021-02-10T04:33:56.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:56 smithi164 conmon[29288]: 14140) 538 : cluster [DBG] pgmap v777: 168 pgs: 3 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 157 active+clean; 331 MiB data, 982 MiB used, 714 GiB / 715 GiB avail; 6.7 MiB/s rd, 5.8 MiB/s wr, 17 op/s 2021-02-10T04:33:56.691 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:33:56 smithi085 conmon[52836]: debug 2021-02-10T04:33:56.691 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:33:56 smithi085 conmon[52836]: 2021-02-10T04:33:56.478+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:33:56.691 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:33:56 smithi085 conmon[52836]: 2021-02-10T04:33:56.736 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:33:57.269 INFO:tasks.rados.rados.0.smithi085.stdout:1572: done (10 left) 2021-02-10T04:33:57.270 INFO:tasks.rados.rados.0.smithi085.stdout:1573: write oid 48 current snap is 173 2021-02-10T04:33:57.270 INFO:tasks.rados.rados.0.smithi085.stdout:1573: seq_num 556 ranges {544692=732213,1985753=743863,3335184=9508} 2021-02-10T04:33:57.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[29319]: cluster 2021-02-10T04:33:56.267029+0000 2021-02-10T04:33:57.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[29319]: mon.a (mon.0) 1212 : cluster [DBG] osdmap e419: 8 total, 8 up, 8 in 2021-02-10T04:33:57.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[29319]: cluster 2021-02-10T04:33:56.571916+0000 osd.5 (osd.5) 62 : cluster [DBG] 7.b scrub ok 2021-02-10T04:33:57.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[29319]: audit 2021-02-10T04:33:56.816849+0000 mon.c (mon.1) 53 : audit [INF] from='client.? 172.21.15.85:0/2147750950' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-02-10T04:33:57.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:33:57.279 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[35961]: cluster 2021-02-10T04:33:56.267029 2021-02-10T04:33:57.279 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1212 : cluster [DBG] osdmap e419: 8 total, 8 up, 8 in 2021-02-10T04:33:57.280 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[35961]: cluster 2021-02-10T04:33 2021-02-10T04:33:57.280 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[35961]: :56.571916+0000 osd.5 (osd.5) 62 2021-02-10T04:33:57.280 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[35961]: : cluster [DBG] 7.b scrub ok 2021-02-10T04:33:57.280 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:33:57.280 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[35961]: 02-10T04:33:56.816849+0000 2021-02-10T04:33:57.280 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[35961]: mon.c (mon.1) 53 : audit [INF] 2021-02-10T04:33:57.280 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/2147750950' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-02-10T04:33:57.280 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:33:57.280 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[35961]: 33:56.817524+0000 mon.a (mon.0) 2021-02-10T04:33:57.280 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[35961]: 1213 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-02-10T04:33:57.281 INFO:tasks.rados.rados.0.smithi085.stdout:1573: writing smithi08575397-48 from 544692 to 1276905 tid 1 2021-02-10T04:33:57.284 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.1 to 1 (10000) 2021-02-10T04:33:57.287 INFO:tasks.rados.rados.0.smithi085.stdout:1573: writing smithi08575397-48 from 1985753 to 2729616 tid 2 2021-02-10T04:33:57.291 INFO:tasks.rados.rados.0.smithi085.stdout:1573: writing smithi08575397-48 from 3335184 to 3344692 tid 3 2021-02-10T04:33:57.292 INFO:tasks.rados.rados.0.smithi085.stdout:1569: finishing copy_from to smithi08575397-6 2021-02-10T04:33:57.292 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 598 (ObjNum 548 snap 169 seq_num 548) dirty exists 2021-02-10T04:33:57.292 INFO:tasks.rados.rados.0.smithi085.stdout:1569: finishing copy_from racing read to smithi08575397-6 2021-02-10T04:33:57.292 INFO:tasks.rados.rados.0.smithi085.stdout:1573: finishing write tid 1 to smithi08575397-48 2021-02-10T04:33:57.292 INFO:tasks.rados.rados.0.smithi085.stdout:1569: done (10 left) 2021-02-10T04:33:57.292 INFO:tasks.rados.rados.0.smithi085.stdout:1574: read oid 31 snap -1 2021-02-10T04:33:57.292 INFO:tasks.rados.rados.0.smithi085.stdout:1574: expect (ObjNum 517 snap 152 seq_num 517) 2021-02-10T04:33:57.292 INFO:tasks.rados.rados.0.smithi085.stdout:1575: copy_from oid 43 from oid 38 current snap is 173 2021-02-10T04:33:57.292 INFO:tasks.rados.rados.0.smithi085.stdout:1576: read oid 28 snap -1 2021-02-10T04:33:57.292 INFO:tasks.rados.rados.0.smithi085.stdout:1576: expect (ObjNum 545 snap 168 seq_num 545) 2021-02-10T04:33:57.292 INFO:tasks.rados.rados.0.smithi085.stdout:1577: read oid 42 snap -1 2021-02-10T04:33:57.293 INFO:tasks.rados.rados.0.smithi085.stdout:1577: expect (ObjNum 540 snap 167 seq_num 540) 2021-02-10T04:33:57.293 INFO:tasks.rados.rados.0.smithi085.stdout:1578: copy_from oid 13 from oid 22 current snap is 173 2021-02-10T04:33:57.293 INFO:tasks.rados.rados.0.smithi085.stdout:1579: delete oid 12 current snap is 173 2021-02-10T04:33:57.293 INFO:tasks.rados.rados.0.smithi085.stdout:1573: finishing write tid 2 to smithi08575397-48 2021-02-10T04:33:57.301 INFO:tasks.rados.rados.0.smithi085.stdout:1576: expect (ObjNum 545 snap 168 seq_num 545) 2021-02-10T04:33:57.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:57 smithi164 conmon[29288]: cluster 2021-02-10T04:33:56.267029+0000 mon.a (mon.0) 1212 : cluster [DBG] osdmap e419: 8 total, 8 up, 8 in 2021-02-10T04:33:57.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:57 smithi164 conmon[29288]: cluster 2021-02-10T04:33:56.571916+0000 osd.5 (osd.5) 62 : cluster [DBG] 7.b scrub ok 2021-02-10T04:33:57.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:57 smithi164 conmon[29288]: audit 2021-02-10T04:33:56.816849+0000 mon.c (mon.1) 53 : audit [INF] from='client.? 172.21.15.85:0/2147750950' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-02-10T04:33:57.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:57 smithi164 conmon[29288]: audit 2021-02-10T04:33:56.817524+0000 mon.a (mon.0) 1213 : audit 2021-02-10T04:33:57.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:57 smithi164 conmon[29288]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-02-10T04:33:57.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[29319]: T 2021-02-10T04:33:57.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[29319]: 04 2021-02-10T04:33:57.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[29319]: : 2021-02-10T04:33:57.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[29319]: 33 2021-02-10T04:33:57.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[29319]: : 2021-02-10T04:33:57.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[29319]: 56 2021-02-10T04:33:57.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[29319]: . 2021-02-10T04:33:57.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[29319]: 817524 2021-02-10T04:33:57.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[29319]: +0000 2021-02-10T04:33:57.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[29319]: 2021-02-10T04:33:57.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[29319]: mon.a 2021-02-10T04:33:57.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[29319]: ( 2021-02-10T04:33:57.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[29319]: mon 2021-02-10T04:33:57.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[29319]: . 2021-02-10T04:33:57.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:57 smithi085 conmon[29319]: 0) 1213 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-02-10T04:33:57.693 INFO:tasks.rados.rados.0.smithi085.stdout:1574: expect (ObjNum 517 snap 152 seq_num 517) 2021-02-10T04:33:57.819 INFO:tasks.rados.rados.0.smithi085.stdout:1573: finishing write tid 3 to smithi08575397-48 2021-02-10T04:33:57.819 INFO:tasks.rados.rados.0.smithi085.stdout:1573: finishing write tid 4 to smithi08575397-48 2021-02-10T04:33:57.819 INFO:tasks.rados.rados.0.smithi085.stdout:1573: finishing write tid 5 to smithi08575397-48 2021-02-10T04:33:57.819 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 734 (ObjNum 556 snap 173 seq_num 556) dirty exists 2021-02-10T04:33:57.819 INFO:tasks.rados.rados.0.smithi085.stdout:1573: left oid 48 (ObjNum 556 snap 173 seq_num 556) 2021-02-10T04:33:57.819 INFO:tasks.rados.rados.0.smithi085.stdout:1577: expect (ObjNum 540 snap 167 seq_num 540) 2021-02-10T04:33:58.115 INFO:tasks.rados.rados.0.smithi085.stdout:1575: finishing copy_from to smithi08575397-43 2021-02-10T04:33:58.115 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 681 (ObjNum 520 snap 155 seq_num 520) dirty exists 2021-02-10T04:33:58.116 INFO:tasks.rados.rados.0.smithi085.stdout:1575: finishing copy_from racing read to smithi08575397-43 2021-02-10T04:33:58.116 INFO:tasks.rados.rados.0.smithi085.stdout:1578: finishing copy_from to smithi08575397-13 2021-02-10T04:33:58.116 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 644 (ObjNum 527 snap 155 seq_num 527) dirty exists 2021-02-10T04:33:58.116 INFO:tasks.rados.rados.0.smithi085.stdout:1578: finishing copy_from racing read to smithi08575397-13 2021-02-10T04:33:58.116 INFO:tasks.rados.rados.0.smithi085.stdout:1573: done (15 left) 2021-02-10T04:33:58.116 INFO:tasks.rados.rados.0.smithi085.stdout:1574: done (14 left) 2021-02-10T04:33:58.116 INFO:tasks.rados.rados.0.smithi085.stdout:1575: done (13 left) 2021-02-10T04:33:58.116 INFO:tasks.rados.rados.0.smithi085.stdout:1576: done (12 left) 2021-02-10T04:33:58.116 INFO:tasks.rados.rados.0.smithi085.stdout:1577: done (11 left) 2021-02-10T04:33:58.116 INFO:tasks.rados.rados.0.smithi085.stdout:1578: done (10 left) 2021-02-10T04:33:58.116 INFO:tasks.rados.rados.0.smithi085.stdout:1579: done (9 left) 2021-02-10T04:33:58.116 INFO:tasks.rados.rados.0.smithi085.stdout:1580: write oid 1 current snap is 173 2021-02-10T04:33:58.117 INFO:tasks.rados.rados.0.smithi085.stdout:1580: seq_num 558 ranges {578306=482711,1586364=631179,2757324=484842} 2021-02-10T04:33:58.121 INFO:tasks.rados.rados.0.smithi085.stdout:1580: writing smithi08575397-1 from 578306 to 1061017 tid 1 2021-02-10T04:33:58.127 INFO:tasks.rados.rados.0.smithi085.stdout:1580: writing smithi08575397-1 from 1586364 to 2217543 tid 2 2021-02-10T04:33:58.134 INFO:tasks.rados.rados.0.smithi085.stdout:1580: writing smithi08575397-1 from 2757324 to 3242166 tid 3 2021-02-10T04:33:58.134 INFO:tasks.rados.rados.0.smithi085.stdout:1581: write (excl) oid 37 current snap is 173 2021-02-10T04:33:58.134 INFO:tasks.rados.rados.0.smithi085.stdout:1581: seq_num 559 ranges {595113=557960,1644846=491190,2784747=610366} 2021-02-10T04:33:58.141 INFO:tasks.rados.rados.0.smithi085.stdout:1581: writing smithi08575397-37 from 595113 to 1153073 tid 1 2021-02-10T04:33:58.147 INFO:tasks.rados.rados.0.smithi085.stdout:1581: writing smithi08575397-37 from 1644846 to 2136036 tid 2 2021-02-10T04:33:58.155 INFO:tasks.rados.rados.0.smithi085.stdout:1581: writing smithi08575397-37 from 2784747 to 3395113 tid 3 2021-02-10T04:33:58.155 INFO:tasks.rados.rados.0.smithi085.stdout:1582: read oid 20 snap 171 2021-02-10T04:33:58.156 INFO:tasks.rados.rados.0.smithi085.stdout:1582: expect (ObjNum 509 snap 148 seq_num 509) 2021-02-10T04:33:58.156 INFO:tasks.rados.rados.0.smithi085.stdout:1583: read oid 16 snap 161 2021-02-10T04:33:58.156 INFO:tasks.rados.rados.0.smithi085.stdout:1583: expect (ObjNum 523 snap 155 seq_num 523) 2021-02-10T04:33:58.156 INFO:tasks.rados.rados.0.smithi085.stdout:1584: delete oid 23 current snap is 173 2021-02-10T04:33:58.173 INFO:tasks.rados.rados.0.smithi085.stdout:1583: expect (ObjNum 523 snap 155 seq_num 523) 2021-02-10T04:33:58.280 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:33:57 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:33:58.398 INFO:tasks.rados.rados.0.smithi085.stdout:1583: done (13 left) 2021-02-10T04:33:58.399 INFO:tasks.rados.rados.0.smithi085.stdout:1584: done (12 left) 2021-02-10T04:33:58.399 INFO:tasks.rados.rados.0.smithi085.stdout:1585: snap_remove snap 152 2021-02-10T04:33:58.600 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:58 smithi164 conmon[29288]: cluster 2021-02-10T04:33:56.837807+0000 osd.6 (osd.6) 57 2021-02-10T04:33:58.601 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:58 smithi164 conmon[29288]: : cluster [DBG] 4.5 scrub ok 2021-02-10T04:33:58.601 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:58 smithi164 conmon[29288]: audit 2021-02-10T04:33:57.268119+0000 mon.a (mon.0) 1214 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-02-10T04:33:58.601 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:58 smithi164 conmon[29288]: cluster 2021-02-10T04:33:57.268188 2021-02-10T04:33:58.601 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:58 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1215 : cluster [DBG] osdmap e420: 8 total, 8 up, 8 in 2021-02-10T04:33:58.601 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:58 smithi164 conmon[29288]: audit 2021-02-10T04:33:57.994985+0000 mgr.y 2021-02-10T04:33:58.601 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:58 smithi164 conmon[29288]: (mgr.14140) 539 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:33:58.601 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:58 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:33:58.601 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:58 smithi164 conmon[29288]: 04:33:58.078740+0000 mgr.y (mgr. 2021-02-10T04:33:58.601 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:58 smithi164 conmon[29288]: 14140) 540 : cluster [DBG] pgmap v780: 168 pgs: 3 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 157 active+clean; 331 MiB data, 996 MiB used, 714 GiB / 715 GiB avail; 6.7 MiB/s rd, 5.4 MiB/s wr, 16 op/s 2021-02-10T04:33:58.601 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:58 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:33:58.601 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:58 smithi164 conmon[29288]: 33:58.128588+0000 mon.a (mon.0) 1216 : cluster [DBG] 2021-02-10T04:33:58.601 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:58 smithi164 conmon[29288]: osdmap e421: 8 total, 8 up, 8 in 2021-02-10T04:33:58.602 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:33:58 smithi164 conmon[33677]: debug 2021-02-10T04:33:58.598+0000 7fbea5924700 -1 osd.4 pg_epoch: 421 pg[6.1a( empty local-lis/les=59/60 n=0 ec=59/59 lis/c=59/59 les/c/f=60/60/0 sis=59) [4,5,1] r=0 lpr=59 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.1a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x562760a5a2c0) 2021-02-10T04:33:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:58 smithi085 conmon[29319]: cluster 2021-02-10T04:33:56.837807+0000 osd.6 ( 2021-02-10T04:33:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:58 smithi085 conmon[29319]: osd.6) 57 : cluster [DBG] 4.5 scrub ok 2021-02-10T04:33:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:58 smithi085 conmon[29319]: audit 2021-02-10T04:33:57.268119+0000 mon.a (mon.0) 1214 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-02-10T04:33:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:58 smithi085 conmon[29319]: cluster 2021-02-10T04:33:57.268188+0000 mon.a (mon.0) 1215 : cluster [DBG] osdmap e420: 8 total, 8 up, 8 in 2021-02-10T04:33:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:58 smithi085 conmon[29319]: audit 2021-02-10T04:33:57.994985+0000 mgr.y (mgr.14140) 539 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:33:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:58 smithi085 conmon[29319]: cluster 2021-02-10T04:33:58.078740+0000 mgr.y (mgr.14140) 540 : cluster [DBG] pgmap v780: 168 pgs: 3 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 157 active+clean; 331 MiB data, 996 MiB used, 714 GiB / 715 GiB avail; 6.7 MiB/s rd, 5.4 MiB/s wr, 16 op/s 2021-02-10T04:33:58.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:58 smithi085 conmon[29319]: cluster 2021-02-10T04:33:58.128588+0000 mon.a (mon.0) 1216 : cluster [DBG] osdmap e421: 8 total, 8 up, 8 in 2021-02-10T04:33:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:58 smithi085 conmon[35961]: cluster 2021-02-10T04:33:56.837807 2021-02-10T04:33:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:58 smithi085 conmon[35961]: +0000 osd.6 (osd.6) 57 : cluster [DBG] 4.5 scrub ok 2021-02-10T04:33:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:58 smithi085 conmon[35961]: audit 2021-02-10T04:33:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:58 smithi085 conmon[35961]: 2021-02-10T04:33:57.268119+0000 mon.a (mon.0) 1214 : 2021-02-10T04:33:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:58 smithi085 conmon[35961]: audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-02-10T04:33:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:58 smithi085 conmon[35961]: cluster 2021-02-10T04:33:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:58 smithi085 conmon[35961]: 2021-02-10T04:33:57 2021-02-10T04:33:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:58 smithi085 conmon[35961]: .268188+0000 mon.a (mon.0) 1215 : 2021-02-10T04:33:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:58 smithi085 conmon[35961]: cluster [DBG] osdmap e420: 8 total, 8 up, 8 in 2021-02-10T04:33:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:58 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:33:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:58 smithi085 conmon[35961]: T04:33:57.994985+0000 mgr.y (mgr. 2021-02-10T04:33:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:58 smithi085 conmon[35961]: 14140) 539 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:33:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:58 smithi085 conmon[35961]: cluster 2021-02-10T04:33:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:58 smithi085 conmon[35961]: 2021-02-10T04:33:58. 2021-02-10T04:33:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:58 smithi085 conmon[35961]: 078740+0000 mgr.y (mgr.14140) 540 : cluster 2021-02-10T04:33:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:58 smithi085 conmon[35961]: [DBG] pgmap v780: 168 pgs: 3 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 157 active+clean; 331 MiB data, 996 MiB used, 714 GiB / 715 GiB avail; 6.7 MiB/s rd, 5.4 MiB/s wr, 16 op/s 2021-02-10T04:33:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:58 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:33:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:58 smithi085 conmon[35961]: 10T04:33:58.128588+0000 mon.a ( 2021-02-10T04:33:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:58 smithi085 conmon[35961]: mon.0) 1216 : cluster 2021-02-10T04:33:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:58 smithi085 conmon[35961]: [DBG] osdmap e421: 8 total, 8 up, 8 in 2021-02-10T04:33:59.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:33:58 smithi085 conmon[52836]: debug 2021-02-10T04:33:58.926+0000 7fba75fba700 -1 osd.3 pg_epoch: 421 pg[4.10( v 62'12 (0'0,62'12] local-lis/les=71/72 n=10 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,5,6] r=0 lpr=71 crt=62'12 lcod 62'11 mlcod 0'0 active+clean] scrubber pg(4.10) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6 (0x55feb38a9600) 2021-02-10T04:33:59.285 INFO:tasks.rados.rados.0.smithi085.stdout:1585: done (12 left) 2021-02-10T04:33:59.286 INFO:tasks.rados.rados.0.smithi085.stdout:1586: write (excl) oid 48 current snap is 173 2021-02-10T04:33:59.286 INFO:tasks.rados.rados.0.smithi085.stdout:1586: seq_num 561 ranges {628727=708458,2045457=494859,3123240=305487} 2021-02-10T04:33:59.300 INFO:tasks.rados.rados.0.smithi085.stdout:1586: writing smithi08575397-48 from 628727 to 1337185 tid 1 2021-02-10T04:33:59.307 INFO:tasks.rados.rados.0.smithi085.stdout:1586: writing smithi08575397-48 from 2045457 to 2540316 tid 2 2021-02-10T04:33:59.313 INFO:tasks.rados.rados.0.smithi085.stdout:1586: writing smithi08575397-48 from 3123240 to 3428727 tid 3 2021-02-10T04:33:59.313 INFO:tasks.rados.rados.0.smithi085.stdout:1580: finishing write tid 1 to smithi08575397-1 2021-02-10T04:33:59.313 INFO:tasks.rados.rados.0.smithi085.stdout:1580: finishing write tid 2 to smithi08575397-1 2021-02-10T04:33:59.313 INFO:tasks.rados.rados.0.smithi085.stdout:1580: finishing write tid 3 to smithi08575397-1 2021-02-10T04:33:59.313 INFO:tasks.rados.rados.0.smithi085.stdout:1580: finishing write tid 4 to smithi08575397-1 2021-02-10T04:33:59.313 INFO:tasks.rados.rados.0.smithi085.stdout:1580: finishing write tid 5 to smithi08575397-1 2021-02-10T04:33:59.313 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 629 (ObjNum 558 snap 173 seq_num 558) dirty exists 2021-02-10T04:33:59.313 INFO:tasks.rados.rados.0.smithi085.stdout:1580: left oid 1 (ObjNum 558 snap 173 seq_num 558) 2021-02-10T04:33:59.313 INFO:tasks.rados.rados.0.smithi085.stdout:1581: finishing write tid 1 to smithi08575397-37 2021-02-10T04:33:59.313 INFO:tasks.rados.rados.0.smithi085.stdout:1582: expect (ObjNum 509 snap 148 seq_num 509) 2021-02-10T04:33:59.520 INFO:tasks.rados.rados.0.smithi085.stdout:1581: finishing write tid 2 to smithi08575397-37 2021-02-10T04:33:59.520 INFO:tasks.rados.rados.0.smithi085.stdout:1581: finishing write tid 3 to smithi08575397-37 2021-02-10T04:33:59.520 INFO:tasks.rados.rados.0.smithi085.stdout:1581: finishing write tid 4 to smithi08575397-37 2021-02-10T04:33:59.520 INFO:tasks.rados.rados.0.smithi085.stdout:1581: finishing write tid 5 to smithi08575397-37 2021-02-10T04:33:59.520 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 861 (ObjNum 559 snap 173 seq_num 559) dirty exists 2021-02-10T04:33:59.520 INFO:tasks.rados.rados.0.smithi085.stdout:1581: left oid 37 (ObjNum 559 snap 173 seq_num 559) 2021-02-10T04:33:59.520 INFO:tasks.rados.rados.0.smithi085.stdout:1580: done (12 left) 2021-02-10T04:33:59.521 INFO:tasks.rados.rados.0.smithi085.stdout:1581: done (11 left) 2021-02-10T04:33:59.521 INFO:tasks.rados.rados.0.smithi085.stdout:1582: done (10 left) 2021-02-10T04:33:59.521 INFO:tasks.rados.rados.0.smithi085.stdout:1587: write (excl) oid 35 current snap is 173 2021-02-10T04:33:59.521 INFO:tasks.rados.rados.0.smithi085.stdout:1586: finishing write tid 1 to smithi08575397-48 2021-02-10T04:33:59.521 INFO:tasks.rados.rados.0.smithi085.stdout:1587: seq_num 562 ranges {645534=783707,2103939=638517,3150663=294871} 2021-02-10T04:33:59.529 INFO:tasks.rados.rados.0.smithi085.stdout:1587: writing smithi08575397-35 from 645534 to 1429241 tid 1 2021-02-10T04:33:59.537 INFO:tasks.rados.rados.0.smithi085.stdout:1587: writing smithi08575397-35 from 2103939 to 2742456 tid 2 2021-02-10T04:33:59.542 INFO:tasks.rados.rados.0.smithi085.stdout:1587: writing smithi08575397-35 from 3150663 to 3445534 tid 3 2021-02-10T04:33:59.542 INFO:tasks.rados.rados.0.smithi085.stdout:1588: rollback oid 6 current snap is 173 2021-02-10T04:33:59.542 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 6 to 169 2021-02-10T04:33:59.542 INFO:tasks.rados.rados.0.smithi085.stdout:1586: finishing write tid 2 to smithi08575397-48 2021-02-10T04:33:59.542 INFO:tasks.rados.rados.0.smithi085.stdout:1586: finishing write tid 3 to smithi08575397-48 2021-02-10T04:33:59.543 INFO:tasks.rados.rados.0.smithi085.stdout:1586: finishing write tid 4 to smithi08575397-48 2021-02-10T04:33:59.543 INFO:tasks.rados.rados.0.smithi085.stdout:1586: finishing write tid 5 to smithi08575397-48 2021-02-10T04:33:59.543 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 739 (ObjNum 561 snap 173 seq_num 561) dirty exists 2021-02-10T04:33:59.543 INFO:tasks.rados.rados.0.smithi085.stdout:1586: left oid 48 (ObjNum 561 snap 173 seq_num 561) 2021-02-10T04:33:59.543 INFO:tasks.rados.rados.0.smithi085.stdout:1587: finishing write tid 1 to smithi08575397-35 2021-02-10T04:33:59.543 INFO:tasks.rados.rados.0.smithi085.stdout:1586: done (11 left) 2021-02-10T04:33:59.543 INFO:tasks.rados.rados.0.smithi085.stdout:1589: copy_from oid 26 from oid 23 current snap is 173 2021-02-10T04:33:59.543 INFO:tasks.rados.rados.0.smithi085.stdout:1590: delete oid 3 current snap is 173 2021-02-10T04:33:59.545 INFO:tasks.rados.rados.0.smithi085.stdout:1588: finishing rollback tid 0 to smithi08575397-6 2021-02-10T04:33:59.545 INFO:tasks.rados.rados.0.smithi085.stdout:1587: finishing write tid 2 to smithi08575397-35 2021-02-10T04:33:59.547 INFO:tasks.rados.rados.0.smithi085.stdout:1588: finishing rollback tid 1 to smithi08575397-6 2021-02-10T04:33:59.547 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 742 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-02-10T04:33:59.547 INFO:tasks.rados.rados.0.smithi085.stdout:1588: done (12 left) 2021-02-10T04:33:59.547 INFO:tasks.rados.rados.0.smithi085.stdout:1590: done (11 left) 2021-02-10T04:33:59.547 INFO:tasks.rados.rados.0.smithi085.stdout:1591: write oid 22 current snap is 173 2021-02-10T04:33:59.547 INFO:tasks.rados.rados.0.smithi085.stdout:1591: seq_num 564 ranges {679148=650558,1937256=642186,3321862=157286} 2021-02-10T04:33:59.554 INFO:tasks.rados.rados.0.smithi085.stdout:1591: writing smithi08575397-22 from 679148 to 1329706 tid 1 2021-02-10T04:33:59.562 INFO:tasks.rados.rados.0.smithi085.stdout:1591: writing smithi08575397-22 from 1937256 to 2579442 tid 2 2021-02-10T04:33:59.568 INFO:tasks.rados.rados.0.smithi085.stdout:1591: writing smithi08575397-22 from 3321862 to 3479148 tid 3 2021-02-10T04:33:59.568 INFO:tasks.rados.rados.0.smithi085.stdout:1592: snap_create 2021-02-10T04:33:59.568 INFO:tasks.rados.rados.0.smithi085.stdout:1589: finishing copy_from to smithi08575397-26 2021-02-10T04:33:59.568 INFO:tasks.rados.rados.0.smithi085.stdout:1589: got expected ENOENT (src dne) 2021-02-10T04:33:59.568 INFO:tasks.rados.rados.0.smithi085.stdout:1589: finishing copy_from racing read to smithi08575397-26 2021-02-10T04:33:59.568 INFO:tasks.rados.rados.0.smithi085.stdout:1587: finishing write tid 3 to smithi08575397-35 2021-02-10T04:33:59.568 INFO:tasks.rados.rados.0.smithi085.stdout:1587: finishing write tid 4 to smithi08575397-35 2021-02-10T04:33:59.569 INFO:tasks.rados.rados.0.smithi085.stdout:1587: finishing write tid 5 to smithi08575397-35 2021-02-10T04:33:59.569 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 35 v 620 (ObjNum 562 snap 173 seq_num 562) dirty exists 2021-02-10T04:33:59.569 INFO:tasks.rados.rados.0.smithi085.stdout:1587: left oid 35 (ObjNum 562 snap 173 seq_num 562) 2021-02-10T04:33:59.569 INFO:tasks.rados.rados.0.smithi085.stdout:1591: finishing write tid 1 to smithi08575397-22 2021-02-10T04:33:59.569 INFO:tasks.rados.rados.0.smithi085.stdout:1591: finishing write tid 2 to smithi08575397-22 2021-02-10T04:33:59.570 INFO:tasks.rados.rados.0.smithi085.stdout:1591: finishing write tid 3 to smithi08575397-22 2021-02-10T04:33:59.571 INFO:tasks.rados.rados.0.smithi085.stdout:1591: finishing write tid 4 to smithi08575397-22 2021-02-10T04:33:59.571 INFO:tasks.rados.rados.0.smithi085.stdout:1591: finishing write tid 5 to smithi08575397-22 2021-02-10T04:33:59.571 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 648 (ObjNum 564 snap 173 seq_num 564) dirty exists 2021-02-10T04:33:59.571 INFO:tasks.rados.rados.0.smithi085.stdout:1591: left oid 22 (ObjNum 564 snap 173 seq_num 564) 2021-02-10T04:33:59.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:59 smithi164 conmon[29288]: cluster 2021-02-10T04:33:57.871394+0000 osd.6 2021-02-10T04:33:59.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:33:59 smithi164 conmon[29288]: (osd.6) 58 : cluster [DBG] 5.9 scrub ok 2021-02-10T04:33:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:59 smithi085 conmon[29319]: cluster 2021-02-10T04:33:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:59 smithi085 conmon[29319]: 2021-02-10T04:33:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:59 smithi085 conmon[29319]: 2021 2021-02-10T04:33:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:59 smithi085 conmon[29319]: - 2021-02-10T04:33:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:59 smithi085 conmon[29319]: 02 2021-02-10T04:33:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:59 smithi085 conmon[29319]: - 2021-02-10T04:33:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:59 smithi085 conmon[29319]: 10 2021-02-10T04:33:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:59 smithi085 conmon[29319]: T 2021-02-10T04:33:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:59 smithi085 conmon[29319]: 04 2021-02-10T04:33:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:59 smithi085 conmon[29319]: : 2021-02-10T04:33:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:59 smithi085 conmon[29319]: 33 2021-02-10T04:33:59.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:59 smithi085 conmon[29319]: : 2021-02-10T04:33:59.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:59 smithi085 conmon[29319]: 57 2021-02-10T04:33:59.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:59 smithi085 conmon[29319]: . 2021-02-10T04:33:59.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:59 smithi085 conmon[29319]: 871394 2021-02-10T04:33:59.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:59 smithi085 conmon[29319]: +0000 2021-02-10T04:33:59.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:59 smithi085 conmon[29319]: 2021-02-10T04:33:59.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:59 smithi085 conmon[29319]: osd.6 2021-02-10T04:33:59.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:59 smithi085 conmon[29319]: ( 2021-02-10T04:33:59.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:59 smithi085 conmon[29319]: osd 2021-02-10T04:33:59.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:59 smithi085 conmon[29319]: .6) 58 : cluster 2021-02-10T04:33:59.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:33:59 smithi085 conmon[29319]: [DBG] 5.9 scrub ok 2021-02-10T04:33:59.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:59 smithi085 conmon[35961]: cluster 2021-02-10T04:33:59.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:59 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:33:59.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:33:59 smithi085 conmon[35961]: :33:57.871394+0000 osd.6 (osd.6) 58 : cluster [DBG] 5.9 scrub ok 2021-02-10T04:34:00.124 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:34:00.157 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:34:00.300 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:00 smithi085 conmon[29319]: cluster 2021-02-10T04:33:58. 2021-02-10T04:34:00.300 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:00 smithi085 conmon[29319]: 892497+0000 osd.6 (osd.6) 59 : cluster [DBG] 2.22 scrub ok 2021-02-10T04:34:00.301 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:00 smithi085 conmon[29319]: cluster 2021-02-10T04:33:59.285420+0000 mon.a (mon.0) 1217 : cluster [DBG] osdmap e422: 8 total, 8 up, 8 in 2021-02-10T04:34:00.301 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:00 smithi085 conmon[29319]: cluster 2021-02-10T04:34:00.079142+0000 mgr.y (mgr.14140) 541 : cluster [DBG] pgmap v783: 168 pgs: 3 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 158 active+clean; 340 MiB data, 1011 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 2.4 MiB/s wr, 17 op/s 2021-02-10T04:34:00.304 INFO:tasks.rados.rados.0.smithi085.stdout:1587: done (12 left) 2021-02-10T04:34:00.304 INFO:tasks.rados.rados.0.smithi085.stdout:1589: done (11 left) 2021-02-10T04:34:00.304 INFO:tasks.rados.rados.0.smithi085.stdout:1591: done (10 left) 2021-02-10T04:34:00.304 INFO:tasks.rados.rados.0.smithi085.stdout:1592: done (9 left) 2021-02-10T04:34:00.304 INFO:tasks.rados.rados.0.smithi085.stdout:1593: snap_remove snap 163 2021-02-10T04:34:00.566 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:34:00.570 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:00 smithi164 conmon[29288]: cluster 2021-02-10T04:34:00.571 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:00 smithi164 conmon[29288]: 2021-02-10T04:34:00.571 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:00 smithi164 conmon[29288]: 2021-02-10T04:33:58.892497+0000 osd.6 (osd.6) 2021-02-10T04:34:00.571 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:00 smithi164 conmon[29288]: 59 : cluster [DBG] 2.22 scrub ok 2021-02-10T04:34:00.571 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:00 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:00.571 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:00 smithi164 conmon[29288]: 02-10T04:33:59.285420+0000 mon.a (mon.0) 1217 : cluster [DBG] osdmap e422: 8 total, 8 up, 8 in 2021-02-10T04:34:00.571 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:00 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:00.572 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:00 smithi164 conmon[29288]: -02-10T04:34:00.079142+0000 2021-02-10T04:34:00.572 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:00 smithi164 conmon[29288]: mgr.y (mgr.14140) 541 : cluster [DBG] pgmap v783: 168 pgs: 3 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 158 active+clean; 340 MiB data, 1011 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 2.4 MiB/s wr, 17 op/s 2021-02-10T04:34:00.572 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:00 smithi164 conmon[33677]: debug 2021-02-10T04:34:00.572 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:00 smithi164 conmon[33677]: 2021-02-10T04:34:00.242+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:34:00.673 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:34:00.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:00 smithi085 conmon[35961]: cluster 2021-02-10T04:33:58.892497 2021-02-10T04:34:00.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:00 smithi085 conmon[35961]: +0000 osd.6 (osd.6) 59 : cluster [DBG] 2.22 scrub ok 2021-02-10T04:34:00.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:00 smithi085 conmon[35961]: cluster 2021-02-10T04:33:59 2021-02-10T04:34:00.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:00 smithi085 conmon[35961]: .285420+0000 mon.a (mon.0) 1217 : cluster [DBG] osdmap e422: 8 total, 8 up, 8 in 2021-02-10T04:34:00.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:00 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:00.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:00 smithi085 conmon[35961]: T04:34:00.079142+0000 mgr.y (mgr 2021-02-10T04:34:00.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:00 smithi085 conmon[35961]: .14140) 541 : cluster [DBG] pgmap v783: 168 pgs: 3 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 158 active+clean; 340 MiB data, 1011 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 2.4 MiB/s wr, 17 op/s 2021-02-10T04:34:00.710 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:34:00.998 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:00 smithi085 conmon[49106]: debug 2021-02-10T04:34:00.996+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:34:01.264 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:34:01.310 INFO:tasks.rados.rados.0.smithi085.stdout:1593: done (9 left) 2021-02-10T04:34:01.310 INFO:tasks.rados.rados.0.smithi085.stdout:1594: snap_remove snap 136 2021-02-10T04:34:01.311 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:01 smithi085 conmon[29319]: cluster 2021-02-10T04:34:00.303871+0000 mon.a ( 2021-02-10T04:34:01.311 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:01 smithi085 conmon[29319]: mon.0) 1218 : cluster [DBG] osdmap e423: 8 total, 8 up, 8 in 2021-02-10T04:34:01.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:01 smithi164 conmon[29288]: cluster 2021-02-10T04:34:00.303871+0000 mon.a (mon. 2021-02-10T04:34:01.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:01 smithi164 conmon[29288]: 0) 1218 : cluster [DBG] osdmap e423: 8 total, 8 up, 8 in 2021-02-10T04:34:01.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:01 smithi164 conmon[33677]: debug 2021-02-10T04:34:01.545+0000 7fbea992c700 -1 osd.4 pg_epoch: 424 pg[6.1a( empty local-lis/les=59/60 n=0 ec=59/59 lis/c=59/59 les/c/f=60/60/0 sis=59) [4,5,1] r=0 lpr=59 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.1a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x562760aac2c0) 2021-02-10T04:34:01.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:01 smithi085 conmon[35961]: cluster 2021-02-10T04:34:01.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:01 smithi085 conmon[35961]: 2021-02-10T04:34:01.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:01 smithi085 conmon[35961]: 2021-02-10T04:34:00 2021-02-10T04:34:01.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:01 smithi085 conmon[35961]: .303871+0000 mon.a (mon.0) 1218 : cluster [DBG] osdmap e423: 8 total, 8 up, 8 in 2021-02-10T04:34:02.312 INFO:tasks.rados.rados.0.smithi085.stdout:1594: done (9 left) 2021-02-10T04:34:02.312 INFO:tasks.rados.rados.0.smithi085.stdout:1595: write oid 36 current snap is 174 2021-02-10T04:34:02.312 INFO:tasks.rados.rados.0.smithi085.stdout:1595: seq_num 565 ranges {695955=725807,1879385=502197,2949285=480099} 2021-02-10T04:34:02.317 INFO:tasks.rados.rados.0.smithi085.stdout:1595: writing smithi08575397-36 from 695955 to 1421762 tid 1 2021-02-10T04:34:02.323 INFO:tasks.rados.rados.0.smithi085.stdout:1595: writing smithi08575397-36 from 1879385 to 2381582 tid 2 2021-02-10T04:34:02.330 INFO:tasks.rados.rados.0.smithi085.stdout:1595: writing smithi08575397-36 from 2949285 to 3429384 tid 3 2021-02-10T04:34:02.331 INFO:tasks.rados.rados.0.smithi085.stdout:1596: rollback oid 37 current snap is 174 2021-02-10T04:34:02.331 INFO:tasks.rados.rados.0.smithi085.stdout:1595: finishing write tid 1 to smithi08575397-36 2021-02-10T04:34:02.331 INFO:tasks.rados.rados.0.smithi085.stdout:1595: finishing write tid 2 to smithi08575397-36 2021-02-10T04:34:02.331 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 37 to 171 2021-02-10T04:34:02.331 INFO:tasks.rados.rados.0.smithi085.stdout:1597: read oid 20 snap -1 2021-02-10T04:34:02.331 INFO:tasks.rados.rados.0.smithi085.stdout:1597: expect (ObjNum 509 snap 148 seq_num 509) 2021-02-10T04:34:02.331 INFO:tasks.rados.rados.0.smithi085.stdout:1598: write oid 5 current snap is 174 2021-02-10T04:34:02.331 INFO:tasks.rados.rados.0.smithi085.stdout:1598: seq_num 566 ranges {712762=401056,1537867=645855,2860355=652407} 2021-02-10T04:34:02.337 INFO:tasks.rados.rados.0.smithi085.stdout:1598: writing smithi08575397-5 from 712762 to 1113818 tid 1 2021-02-10T04:34:02.344 INFO:tasks.rados.rados.0.smithi085.stdout:1598: writing smithi08575397-5 from 1537867 to 2183722 tid 2 2021-02-10T04:34:02.353 INFO:tasks.rados.rados.0.smithi085.stdout:1598: writing smithi08575397-5 from 2860355 to 3512762 tid 3 2021-02-10T04:34:02.353 INFO:tasks.rados.rados.0.smithi085.stdout:1599: read oid 7 snap -1 2021-02-10T04:34:02.353 INFO:tasks.rados.rados.0.smithi085.stdout:1599: expect (ObjNum 553 snap 172 seq_num 553) 2021-02-10T04:34:02.353 INFO:tasks.rados.rados.0.smithi085.stdout:1600: write oid 44 current snap is 174 2021-02-10T04:34:02.353 INFO:tasks.rados.rados.0.smithi085.stdout:1600: seq_num 567 ranges {729569=476305,1996349=505866,3004131=525438} 2021-02-10T04:34:02.360 INFO:tasks.rados.rados.0.smithi085.stdout:1600: writing smithi08575397-44 from 729569 to 1205874 tid 1 2021-02-10T04:34:02.367 INFO:tasks.rados.rados.0.smithi085.stdout:1600: writing smithi08575397-44 from 1996349 to 2502215 tid 2 2021-02-10T04:34:02.374 INFO:tasks.rados.rados.0.smithi085.stdout:1600: writing smithi08575397-44 from 3004131 to 3529569 tid 3 2021-02-10T04:34:02.374 INFO:tasks.rados.rados.0.smithi085.stdout:1601: read oid 23 snap -1 2021-02-10T04:34:02.374 INFO:tasks.rados.rados.0.smithi085.stdout:1601: expect deleted 2021-02-10T04:34:02.374 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:34:02.374 INFO:tasks.rados.rados.0.smithi085.stdout:1597: expect (ObjNum 509 snap 148 seq_num 509) 2021-02-10T04:34:02.594 INFO:tasks.rados.rados.0.smithi085.stdout:1596: finishing rollback tid 0 to smithi08575397-37 2021-02-10T04:34:03.403 INFO:tasks.rados.rados.0.smithi085.stdout:1596: finishing rollback tid 1 to smithi08575397-37 2021-02-10T04:34:03.403 INFO:tasks.rados.rados.0.smithi085.stdout:1596: finishing rollback tid 2 to smithi08575397-37 2021-02-10T04:34:03.403 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 871 (ObjNum 543 snap 168 seq_num 543) dirty exists 2021-02-10T04:34:03.403 INFO:tasks.rados.rados.0.smithi085.stdout:1595: finishing write tid 3 to smithi08575397-36 2021-02-10T04:34:03.403 INFO:tasks.rados.rados.0.smithi085.stdout:1598: finishing write tid 1 to smithi08575397-5 2021-02-10T04:34:03.403 INFO:tasks.rados.rados.0.smithi085.stdout:1595: finishing write tid 4 to smithi08575397-36 2021-02-10T04:34:03.403 INFO:tasks.rados.rados.0.smithi085.stdout:1595: finishing write tid 5 to smithi08575397-36 2021-02-10T04:34:03.403 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 638 (ObjNum 565 snap 174 seq_num 565) dirty exists 2021-02-10T04:34:03.404 INFO:tasks.rados.rados.0.smithi085.stdout:1595: left oid 36 (ObjNum 565 snap 174 seq_num 565) 2021-02-10T04:34:03.404 INFO:tasks.rados.rados.0.smithi085.stdout:1598: finishing write tid 2 to smithi08575397-5 2021-02-10T04:34:03.404 INFO:tasks.rados.rados.0.smithi085.stdout:1595: done (15 left) 2021-02-10T04:34:03.404 INFO:tasks.rados.rados.0.smithi085.stdout:1596: done (14 left) 2021-02-10T04:34:03.404 INFO:tasks.rados.rados.0.smithi085.stdout:1597: done (13 left) 2021-02-10T04:34:03.404 INFO:tasks.rados.rados.0.smithi085.stdout:1602: write oid 31 current snap is 174 2021-02-10T04:34:03.404 INFO:tasks.rados.rados.0.smithi085.stdout:1598: finishing write tid 3 to smithi08575397-5 2021-02-10T04:34:03.404 INFO:tasks.rados.rados.0.smithi085.stdout:1598: finishing write tid 4 to smithi08575397-5 2021-02-10T04:34:03.404 INFO:tasks.rados.rados.0.smithi085.stdout:1602: seq_num 568 ranges {746376=551554,2054831=649524,3315201=231175} 2021-02-10T04:34:03.406 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:02 smithi164 conmon[29288]: cluster 2021-02-10T04:34:01.308611+0000 2021-02-10T04:34:03.406 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:02 smithi164 conmon[29288]: mon.a (mon.0) 1219 : cluster [DBG] osdmap e424: 8 total, 8 up, 8 in 2021-02-10T04:34:03.406 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:02 smithi164 conmon[29288]: cluster 2021-02-10T04:34:02.079668+0000 mgr.y (mgr.14140) 542 : cluster 2021-02-10T04:34:03.407 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:02 smithi164 conmon[29288]: [DBG] pgmap v786: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 344 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 4.2 MiB/s wr, 27 op/s 2021-02-10T04:34:03.407 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:02 smithi164 conmon[29288]: cluster 2021-02-10T04:34:02 2021-02-10T04:34:03.407 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:02 smithi164 conmon[29288]: .307109+0000 mon.a (mon.0) 1220 : 2021-02-10T04:34:03.407 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:02 smithi164 conmon[29288]: cluster [DBG] osdmap e425: 8 total, 8 up, 8 in 2021-02-10T04:34:03.407 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:02 smithi164 conmon[33677]: debug 2021-02-10T04:34:02.529+0000 7fbea5924700 -1 osd.4 pg_epoch: 425 pg[6.1a( empty local-lis/les=59/60 n=0 ec=59/59 lis/c=59/59 les/c/f=60/60/0 sis=59) [4,5,1] r=0 lpr=59 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.1a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x562761027080) 2021-02-10T04:34:03.407 INFO:tasks.rados.rados.0.smithi085.stdout:1602: writing smithi08575397-31 from 746376 to 1297930 tid 1 2021-02-10T04:34:03.407 INFO:tasks.rados.rados.0.smithi085.stdout:1602: writing smithi08575397-31 from 2054831 to 2704355 tid 2 2021-02-10T04:34:03.407 INFO:tasks.rados.rados.0.smithi085.stdout:1602: writing smithi08575397-31 from 3315201 to 3546376 tid 3 2021-02-10T04:34:03.407 INFO:tasks.rados.rados.0.smithi085.stdout:1603: snap_create 2021-02-10T04:34:03.407 INFO:tasks.rados.rados.0.smithi085.stdout:1598: finishing write tid 5 to smithi08575397-5 2021-02-10T04:34:03.408 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 648 (ObjNum 566 snap 174 seq_num 566) dirty exists 2021-02-10T04:34:03.408 INFO:tasks.rados.rados.0.smithi085.stdout:1598: left oid 5 (ObjNum 566 snap 174 seq_num 566) 2021-02-10T04:34:03.408 INFO:tasks.rados.rados.0.smithi085.stdout:1599: expect (ObjNum 553 snap 172 seq_num 553) 2021-02-10T04:34:03.409 INFO:tasks.rados.rados.0.smithi085.stdout:1600: finishing write tid 1 to smithi08575397-44 2021-02-10T04:34:03.409 INFO:tasks.rados.rados.0.smithi085.stdout:1600: finishing write tid 2 to smithi08575397-44 2021-02-10T04:34:03.409 INFO:tasks.rados.rados.0.smithi085.stdout:1600: finishing write tid 3 to smithi08575397-44 2021-02-10T04:34:03.409 INFO:tasks.rados.rados.0.smithi085.stdout:1600: finishing write tid 4 to smithi08575397-44 2021-02-10T04:34:03.409 INFO:tasks.rados.rados.0.smithi085.stdout:1600: finishing write tid 5 to smithi08575397-44 2021-02-10T04:34:03.409 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 723 (ObjNum 567 snap 174 seq_num 567) dirty exists 2021-02-10T04:34:03.409 INFO:tasks.rados.rados.0.smithi085.stdout:1600: left oid 44 (ObjNum 567 snap 174 seq_num 567) 2021-02-10T04:34:03.409 INFO:tasks.rados.rados.0.smithi085.stdout:1602: finishing write tid 1 to smithi08575397-31 2021-02-10T04:34:03.409 INFO:tasks.rados.rados.0.smithi085.stdout:1602: finishing write tid 2 to smithi08575397-31 2021-02-10T04:34:03.409 INFO:tasks.rados.rados.0.smithi085.stdout:1602: finishing write tid 3 to smithi08575397-31 2021-02-10T04:34:03.409 INFO:tasks.rados.rados.0.smithi085.stdout:1602: finishing write tid 4 to smithi08575397-31 2021-02-10T04:34:03.409 INFO:tasks.rados.rados.0.smithi085.stdout:1602: finishing write tid 5 to smithi08575397-31 2021-02-10T04:34:03.409 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 652 (ObjNum 568 snap 174 seq_num 568) dirty exists 2021-02-10T04:34:03.410 INFO:tasks.rados.rados.0.smithi085.stdout:1602: left oid 31 (ObjNum 568 snap 174 seq_num 568) 2021-02-10T04:34:03.410 INFO:tasks.rados.rados.0.smithi085.stdout:1598: done (14 left) 2021-02-10T04:34:03.410 INFO:tasks.rados.rados.0.smithi085.stdout:1599: done (13 left) 2021-02-10T04:34:03.410 INFO:tasks.rados.rados.0.smithi085.stdout:1600: done (12 left) 2021-02-10T04:34:03.410 INFO:tasks.rados.rados.0.smithi085.stdout:1601: done (11 left) 2021-02-10T04:34:03.410 INFO:tasks.rados.rados.0.smithi085.stdout:1602: done (10 left) 2021-02-10T04:34:03.410 INFO:tasks.rados.rados.0.smithi085.stdout:1603: done (9 left) 2021-02-10T04:34:03.410 INFO:tasks.rados.rados.0.smithi085.stdout:1604: write (excl) oid 31 current snap is 175 2021-02-10T04:34:03.410 INFO:tasks.rados.rados.0.smithi085.stdout:1604: seq_num 569 ranges {763183=626803,1996960=793182,3226271=336912} 2021-02-10T04:34:03.410 INFO:tasks.rados.rados.0.smithi085.stdout:1604: writing smithi08575397-31 from 763183 to 1389986 tid 1 2021-02-10T04:34:03.410 INFO:tasks.rados.rados.0.smithi085.stdout:1604: writing smithi08575397-31 from 1996960 to 2790142 tid 2 2021-02-10T04:34:03.410 INFO:tasks.rados.rados.0.smithi085.stdout:1604: writing smithi08575397-31 from 3226271 to 3563183 tid 3 2021-02-10T04:34:03.410 INFO:tasks.rados.rados.0.smithi085.stdout:1605: snap_remove snap 171 2021-02-10T04:34:03.411 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:02 smithi085 conmon[35961]: cluster 2021-02-10T04:34:01.308611+0000 mon.a (mon. 2021-02-10T04:34:03.411 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:02 smithi085 conmon[35961]: 0) 1219 : cluster [DBG] osdmap e424: 8 total, 8 up, 8 in 2021-02-10T04:34:03.411 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:02 smithi085 conmon[35961]: cluster 2021-02-10T04:34:02.079668 2021-02-10T04:34:03.411 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:02 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 542 : cluster [DBG] pgmap v786: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 344 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 4.2 MiB/s wr, 27 op/s 2021-02-10T04:34:03.411 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:02 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:34:03.411 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:02 smithi085 conmon[35961]: :34:02.307109+0000 mon.a (mon.0) 1220 : cluster [DBG] osdmap e425: 8 total, 8 up, 8 in 2021-02-10T04:34:03.411 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:02 smithi085 conmon[29319]: cluster 2021-02-10T04:34:03.411 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:02 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:34:03.411 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:02 smithi085 conmon[29319]: T04:34:01.308611+0000 mon.a (mon.0) 1219 : cluster [DBG] osdmap e424: 8 total, 8 up, 8 in 2021-02-10T04:34:03.411 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:02 smithi085 conmon[29319]: cluster 2021-02-10T04:34:03.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:02 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:34:03.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:02 smithi085 conmon[29319]: 04:34:02.079668+0000 mgr.y (mgr. 2021-02-10T04:34:03.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:02 smithi085 conmon[29319]: 14140) 542 : cluster [DBG] pgmap v786: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 344 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 4.2 MiB/s wr, 27 op/s 2021-02-10T04:34:03.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:02 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:03.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:02 smithi085 conmon[29319]: -10T04:34:02.307109+0000 mon.a 2021-02-10T04:34:03.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:02 smithi085 conmon[29319]: (mon.0) 1220 : cluster [DBG] osdmap e425: 8 total, 8 up, 8 in 2021-02-10T04:34:03.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:02 smithi085 conmon[29319]: 2021-02-10T04:34:03.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:03 smithi164 conmon[29288]: cluster 2021-02-10T04:34:01.895508+0000 osd.6 (osd.6) 60 : cluster [DBG] 2.24 scrub ok 2021-02-10T04:34:03.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:03 smithi164 conmon[29288]: cluster 2021-02-10T04:34:02.064395+0000 osd.5 (osd.5) 63 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:34:03.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:03 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:03.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:03 smithi164 conmon[29288]: :34:02.064586+0000 osd.5 (osd.5) 64 : cluster [DBG] purged_snaps scrub found 2 strays 2021-02-10T04:34:03.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:03 smithi085 conmon[35961]: cluster 2021-02-10T04:34:01.895508+0000 2021-02-10T04:34:03.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:03 smithi085 conmon[35961]: osd.6 (osd.6) 60 : cluster [DBG] 2.24 scrub ok 2021-02-10T04:34:03.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:03 smithi085 conmon[35961]: cluster 2021-02-10T04:34:02 2021-02-10T04:34:03.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:03 smithi085 conmon[35961]: .064395+0000 osd.5 (osd.5) 63 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:34:03.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:03 smithi085 conmon[35961]: cluster 2021-02-10T04:34 2021-02-10T04:34:03.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:03 smithi085 conmon[35961]: :02.064586+0000 osd.5 (osd.5) 64 : cluster [DBG] purged_snaps scrub found 2 strays 2021-02-10T04:34:03.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:03 smithi085 conmon[29319]: cluster 2021-02-10T04:34:01.895508+0000 osd.6 2021-02-10T04:34:03.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:03 smithi085 conmon[29319]: (osd.6) 60 : cluster [DBG] 2.24 scrub ok 2021-02-10T04:34:03.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:03 smithi085 conmon[29319]: cluster 2021-02-10T04:34:02.064395+0000 osd.5 (osd.5) 63 2021-02-10T04:34:03.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:03 smithi085 conmon[29319]: : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:34:03.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:03 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:03.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:03 smithi085 conmon[29319]: -10T04:34:02.064586+0000 2021-02-10T04:34:03.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:03 smithi085 conmon[29319]: osd.5 (osd.5) 64 : cluster 2021-02-10T04:34:03.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:03 smithi085 conmon[29319]: [DBG] purged_snaps scrub found 2 strays 2021-02-10T04:34:04.342 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:04 smithi085 conmon[35961]: cluster 2021-02-10T04:34:04.343 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:04 smithi085 conmon[35961]: 2021-02-10T04:34:04.343 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:04 smithi085 conmon[35961]: 2021-02-10T04:34:03 2021-02-10T04:34:04.343 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:04 smithi085 conmon[35961]: .333683+0000 mon.a (mon.0) 1221 : cluster [DBG] osdmap e426: 8 total, 8 up, 8 in 2021-02-10T04:34:04.343 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:04 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:04.343 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:04 smithi085 conmon[35961]: -10T04:34:04.080120+0000 mgr.y (mgr. 2021-02-10T04:34:04.343 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:04 smithi085 conmon[35961]: 14140) 543 : cluster [DBG] pgmap v789: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 346 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 2.6 MiB/s wr, 13 op/s 2021-02-10T04:34:04.343 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:04 smithi085 conmon[29319]: cluster 2021-02-10T04:34:04.343 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:04 smithi085 conmon[29319]: 2021-02-10T04:34:04.343 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:04 smithi085 conmon[29319]: 2021 2021-02-10T04:34:04.343 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:04 smithi085 conmon[29319]: - 2021-02-10T04:34:04.343 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:04 smithi085 conmon[29319]: 02 2021-02-10T04:34:04.344 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:04 smithi085 conmon[29319]: - 2021-02-10T04:34:04.344 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:04 smithi085 conmon[29319]: 10 2021-02-10T04:34:04.344 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:04 smithi085 conmon[29319]: T04:34:03.333683+0000 2021-02-10T04:34:04.344 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:04 smithi085 conmon[29319]: mon.a (mon.0) 1221 : cluster [DBG] osdmap e426: 8 total, 8 up, 8 in 2021-02-10T04:34:04.344 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:04 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:04.344 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:04 smithi085 conmon[29319]: -02-10T04:34:04 2021-02-10T04:34:04.344 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:04 smithi085 conmon[29319]: .080120+0000 mgr.y (mgr. 2021-02-10T04:34:04.344 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:04 smithi085 conmon[29319]: 14140) 543 : cluster [DBG] pgmap v789: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 346 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 2.6 MiB/s wr, 13 op/s 2021-02-10T04:34:04.344 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:04 smithi085 conmon[29319]: 2021-02-10T04:34:04.346 INFO:tasks.rados.rados.0.smithi085.stdout:1605: done (10 left) 2021-02-10T04:34:04.346 INFO:tasks.rados.rados.0.smithi085.stdout:1606: rollback oid 23 current snap is 175 2021-02-10T04:34:04.346 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 23 to 172 2021-02-10T04:34:04.346 INFO:tasks.rados.rados.0.smithi085.stdout:1604: finishing write tid 1 to smithi08575397-31 2021-02-10T04:34:04.346 INFO:tasks.rados.rados.0.smithi085.stdout:1604: finishing write tid 2 to smithi08575397-31 2021-02-10T04:34:04.346 INFO:tasks.rados.rados.0.smithi085.stdout:1604: finishing write tid 3 to smithi08575397-31 2021-02-10T04:34:04.346 INFO:tasks.rados.rados.0.smithi085.stdout:1604: finishing write tid 4 to smithi08575397-31 2021-02-10T04:34:04.346 INFO:tasks.rados.rados.0.smithi085.stdout:1604: finishing write tid 5 to smithi08575397-31 2021-02-10T04:34:04.346 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 656 (ObjNum 569 snap 175 seq_num 569) dirty exists 2021-02-10T04:34:04.346 INFO:tasks.rados.rados.0.smithi085.stdout:1604: left oid 31 (ObjNum 569 snap 175 seq_num 569) 2021-02-10T04:34:04.346 INFO:tasks.rados.rados.0.smithi085.stdout:1604: done (10 left) 2021-02-10T04:34:04.347 INFO:tasks.rados.rados.0.smithi085.stdout:1607: write (excl) oid 22 current snap is 175 2021-02-10T04:34:04.347 INFO:tasks.rados.rados.0.smithi085.stdout:1607: seq_num 570 ranges {779990=702052,2055442=653193,3253694=326296} 2021-02-10T04:34:04.355 INFO:tasks.rados.rados.0.smithi085.stdout:1607: writing smithi08575397-22 from 779990 to 1482042 tid 1 2021-02-10T04:34:04.363 INFO:tasks.rados.rados.0.smithi085.stdout:1607: writing smithi08575397-22 from 2055442 to 2708635 tid 2 2021-02-10T04:34:04.369 INFO:tasks.rados.rados.0.smithi085.stdout:1607: writing smithi08575397-22 from 3253694 to 3579990 tid 3 2021-02-10T04:34:04.369 INFO:tasks.rados.rados.0.smithi085.stdout:1608: delete oid 43 current snap is 175 2021-02-10T04:34:04.369 INFO:tasks.rados.rados.0.smithi085.stdout:1606: finishing rollback tid 1 to smithi08575397-23 2021-02-10T04:34:04.369 INFO:tasks.rados.rados.0.smithi085.stdout:1606: finishing rollback tid 2 to smithi08575397-23 2021-02-10T04:34:04.370 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 640 (ObjNum 85 snap 11 seq_num 85) dirty exists 2021-02-10T04:34:04.370 INFO:tasks.rados.rados.0.smithi085.stdout:1607: finishing write tid 1 to smithi08575397-22 2021-02-10T04:34:04.372 INFO:tasks.rados.rados.0.smithi085.stdout:1606: done (11 left) 2021-02-10T04:34:04.372 INFO:tasks.rados.rados.0.smithi085.stdout:1608: done (10 left) 2021-02-10T04:34:04.372 INFO:tasks.rados.rados.0.smithi085.stdout:1609: copy_from oid 11 from oid 12 current snap is 175 2021-02-10T04:34:04.372 INFO:tasks.rados.rados.0.smithi085.stdout:1610: delete oid 5 current snap is 175 2021-02-10T04:34:04.373 INFO:tasks.rados.rados.0.smithi085.stdout:1607: finishing write tid 2 to smithi08575397-22 2021-02-10T04:34:04.373 INFO:tasks.rados.rados.0.smithi085.stdout:1607: finishing write tid 3 to smithi08575397-22 2021-02-10T04:34:04.373 INFO:tasks.rados.rados.0.smithi085.stdout:1607: finishing write tid 4 to smithi08575397-22 2021-02-10T04:34:04.373 INFO:tasks.rados.rados.0.smithi085.stdout:1607: finishing write tid 5 to smithi08575397-22 2021-02-10T04:34:04.373 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 660 (ObjNum 570 snap 175 seq_num 570) dirty exists 2021-02-10T04:34:04.373 INFO:tasks.rados.rados.0.smithi085.stdout:1607: left oid 22 (ObjNum 570 snap 175 seq_num 570) 2021-02-10T04:34:04.374 INFO:tasks.rados.rados.0.smithi085.stdout:1609: finishing copy_from to smithi08575397-11 2021-02-10T04:34:04.374 INFO:tasks.rados.rados.0.smithi085.stdout:1609: got expected ENOENT (src dne) 2021-02-10T04:34:04.374 INFO:tasks.rados.rados.0.smithi085.stdout:1609: finishing copy_from racing read to smithi08575397-11 2021-02-10T04:34:04.374 INFO:tasks.rados.rados.0.smithi085.stdout:1607: done (11 left) 2021-02-10T04:34:04.375 INFO:tasks.rados.rados.0.smithi085.stdout:1609: done (10 left) 2021-02-10T04:34:04.375 INFO:tasks.rados.rados.0.smithi085.stdout:1610: done (9 left) 2021-02-10T04:34:04.375 INFO:tasks.rados.rados.0.smithi085.stdout:1611: delete oid 43 current snap is 175 2021-02-10T04:34:04.377 INFO:tasks.rados.rados.0.smithi085.stdout:1611: done (9 left) 2021-02-10T04:34:04.377 INFO:tasks.rados.rados.0.smithi085.stdout:1612: read oid 30 snap -1 2021-02-10T04:34:04.377 INFO:tasks.rados.rados.0.smithi085.stdout:1612: expect deleted 2021-02-10T04:34:04.377 INFO:tasks.rados.rados.0.smithi085.stdout:1613: read oid 8 snap -1 2021-02-10T04:34:04.377 INFO:tasks.rados.rados.0.smithi085.stdout:1613: expect (ObjNum 521 snap 155 seq_num 521) 2021-02-10T04:34:04.378 INFO:tasks.rados.rados.0.smithi085.stdout:1614: write (excl) oid 38 current snap is 175 2021-02-10T04:34:04.378 INFO:tasks.rados.rados.0.smithi085.stdout:1614: seq_num 574 ranges {447218=719401,1889370=660531,2963386=614253} 2021-02-10T04:34:04.384 INFO:tasks.rados.rados.0.smithi085.stdout:1614: writing smithi08575397-38 from 447218 to 1166619 tid 1 2021-02-10T04:34:04.393 INFO:tasks.rados.rados.0.smithi085.stdout:1614: writing smithi08575397-38 from 1889370 to 2549901 tid 2 2021-02-10T04:34:04.400 INFO:tasks.rados.rados.0.smithi085.stdout:1614: writing smithi08575397-38 from 2963386 to 3577639 tid 3 2021-02-10T04:34:04.400 INFO:tasks.rados.rados.0.smithi085.stdout:1615: snap_create 2021-02-10T04:34:04.400 INFO:tasks.rados.rados.0.smithi085.stdout:1613: expect (ObjNum 521 snap 155 seq_num 521) 2021-02-10T04:34:04.503 INFO:tasks.rados.rados.0.smithi085.stdout:1614: finishing write tid 1 to smithi08575397-38 2021-02-10T04:34:04.504 INFO:tasks.rados.rados.0.smithi085.stdout:1614: finishing write tid 2 to smithi08575397-38 2021-02-10T04:34:04.504 INFO:tasks.rados.rados.0.smithi085.stdout:1614: finishing write tid 3 to smithi08575397-38 2021-02-10T04:34:04.504 INFO:tasks.rados.rados.0.smithi085.stdout:1614: finishing write tid 4 to smithi08575397-38 2021-02-10T04:34:04.504 INFO:tasks.rados.rados.0.smithi085.stdout:1614: finishing write tid 5 to smithi08575397-38 2021-02-10T04:34:04.504 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 746 (ObjNum 574 snap 175 seq_num 574) dirty exists 2021-02-10T04:34:04.504 INFO:tasks.rados.rados.0.smithi085.stdout:1614: left oid 38 (ObjNum 574 snap 175 seq_num 574) 2021-02-10T04:34:04.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:04 smithi164 conmon[29288]: cluster 2021-02-10T04:34:03.333683+0000 mon.a (mon. 2021-02-10T04:34:04.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:04 smithi164 conmon[29288]: 0) 1221 : cluster [DBG] osdmap e426: 8 total, 8 up, 8 in 2021-02-10T04:34:04.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:04 smithi164 conmon[29288]: cluster 2021-02-10T04:34:04.080120+0000 mgr.y (mgr.14140) 543 : cluster [DBG] pgmap v789: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 346 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 2.6 MiB/s wr, 13 op/s 2021-02-10T04:34:05.343 INFO:tasks.rados.rados.0.smithi085.stdout:1612: done (12 left) 2021-02-10T04:34:05.343 INFO:tasks.rados.rados.0.smithi085.stdout:1613: done (11 left) 2021-02-10T04:34:05.343 INFO:tasks.rados.rados.0.smithi085.stdout:1614: done (10 left) 2021-02-10T04:34:05.343 INFO:tasks.rados.rados.0.smithi085.stdout:1615: done (9 left) 2021-02-10T04:34:05.343 INFO:tasks.rados.rados.0.smithi085.stdout:1616: snap_remove snap 168 2021-02-10T04:34:05.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:05 smithi164 conmon[29288]: cluster 2021-02-10T04:34:04.344826+0000 mon.a ( 2021-02-10T04:34:05.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:05 smithi164 conmon[29288]: mon.0) 1222 : cluster [DBG] osdmap e427: 8 total, 8 up, 8 in 2021-02-10T04:34:05.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:05 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:34:05.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:05 smithi085 conmon[35961]: :34:04.344826+0000 mon.a (mon.0) 1222 : cluster [DBG] osdmap e427: 8 total, 8 up, 8 in 2021-02-10T04:34:05.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:05 smithi085 conmon[29319]: cluster 2021-02-10T04:34:04.344826+0000 2021-02-10T04:34:05.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:05 smithi085 conmon[29319]: mon.a (mon.0) 1222 : cluster [DBG] osdmap e427: 8 total, 8 up, 8 in 2021-02-10T04:34:06.358 INFO:tasks.rados.rados.0.smithi085.stdout:1616: done (9 left) 2021-02-10T04:34:06.358 INFO:tasks.rados.rados.0.smithi085.stdout:1617: read oid 11 snap -1 2021-02-10T04:34:06.358 INFO:tasks.rados.rados.0.smithi085.stdout:1617: expect (ObjNum 531 snap 156 seq_num 531) 2021-02-10T04:34:06.358 INFO:tasks.rados.rados.0.smithi085.stdout:1618: write (excl) oid 50 current snap is 176 2021-02-10T04:34:06.359 INFO:tasks.rados.rados.0.smithi085.stdout:1618: seq_num 575 ranges {464025=794650,1947852=404189,2990809=425525} 2021-02-10T04:34:06.365 INFO:tasks.rados.rados.0.smithi085.stdout:1618: writing smithi08575397-50 from 464025 to 1258675 tid 1 2021-02-10T04:34:06.374 INFO:tasks.rados.rados.0.smithi085.stdout:1618: writing smithi08575397-50 from 1947852 to 2352041 tid 2 2021-02-10T04:34:06.379 INFO:tasks.rados.rados.0.smithi085.stdout:1618: writing smithi08575397-50 from 2990809 to 3416334 tid 3 2021-02-10T04:34:06.379 INFO:tasks.rados.rados.0.smithi085.stdout:1619: read oid 13 snap -1 2021-02-10T04:34:06.379 INFO:tasks.rados.rados.0.smithi085.stdout:1619: expect (ObjNum 527 snap 155 seq_num 527) 2021-02-10T04:34:06.379 INFO:tasks.rados.rados.0.smithi085.stdout:1620: read oid 46 snap -1 2021-02-10T04:34:06.379 INFO:tasks.rados.rados.0.smithi085.stdout:1620: expect (ObjNum 521 snap 155 seq_num 521) 2021-02-10T04:34:06.379 INFO:tasks.rados.rados.0.smithi085.stdout:1621: write (excl) oid 12 current snap is 176 2021-02-10T04:34:06.379 INFO:tasks.rados.rados.0.smithi085.stdout:1617: expect (ObjNum 531 snap 156 seq_num 531) 2021-02-10T04:34:06.514 INFO:tasks.rados.rados.0.smithi085.stdout:1618: finishing write tid 1 to smithi08575397-50 2021-02-10T04:34:06.514 INFO:tasks.rados.rados.0.smithi085.stdout:1618: finishing write tid 2 to smithi08575397-50 2021-02-10T04:34:06.514 INFO:tasks.rados.rados.0.smithi085.stdout:1621: seq_num 576 ranges {480832=469899,1606334=664200,3018232=662600} 2021-02-10T04:34:06.519 INFO:tasks.rados.rados.0.smithi085.stdout:1621: writing smithi08575397-12 from 480832 to 950731 tid 1 2021-02-10T04:34:06.527 INFO:tasks.rados.rados.0.smithi085.stdout:1621: writing smithi08575397-12 from 1606334 to 2270534 tid 2 2021-02-10T04:34:06.538 INFO:tasks.rados.rados.0.smithi085.stdout:1621: writing smithi08575397-12 from 3018232 to 3680832 tid 3 2021-02-10T04:34:06.538 INFO:tasks.rados.rados.0.smithi085.stdout:1617: done (13 left) 2021-02-10T04:34:06.538 INFO:tasks.rados.rados.0.smithi085.stdout:1622: snap_remove snap 154 2021-02-10T04:34:06.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: cluster 2021-02-10T04:34:06.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: 2021-02-10T04:34:06.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: 2021 2021-02-10T04:34:06.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: - 2021-02-10T04:34:06.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: 02 2021-02-10T04:34:06.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: - 2021-02-10T04:34:06.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: 10T04:34:05.342796+0000 2021-02-10T04:34:06.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: mon.a (mon.0) 1223 : cluster [DBG] osdmap e428: 8 total, 8 up, 8 in 2021-02-10T04:34:06.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: cluster 2021-02-10T04:34:06.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: 2021-02-10T04:34:06.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: 2021 2021-02-10T04:34:06.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: - 2021-02-10T04:34:06.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: 02 2021-02-10T04:34:06.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: - 2021-02-10T04:34:06.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: 10 2021-02-10T04:34:06.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: T 2021-02-10T04:34:06.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: 04 2021-02-10T04:34:06.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: : 2021-02-10T04:34:06.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: 34 2021-02-10T04:34:06.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: : 2021-02-10T04:34:06.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: 06 2021-02-10T04:34:06.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: . 2021-02-10T04:34:06.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: 080565 2021-02-10T04:34:06.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: +0000 2021-02-10T04:34:06.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: 2021-02-10T04:34:06.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: mgr.y 2021-02-10T04:34:06.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: ( 2021-02-10T04:34:06.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: mgr 2021-02-10T04:34:06.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: . 2021-02-10T04:34:06.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: 14140 2021-02-10T04:34:06.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: ) 2021-02-10T04:34:06.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: 544 2021-02-10T04:34:06.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: : 2021-02-10T04:34:06.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: cluster 2021-02-10T04:34:06.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: 2021-02-10T04:34:06.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: [DBG] 2021-02-10T04:34:06.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: 2021-02-10T04:34:06.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: pgmap v792: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 363 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.1 MiB/s rd, 3.1 MiB/s wr, 19 op/s 2021-02-10T04:34:06.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:06 smithi164 conmon[29288]: 2021-02-10T04:34:06.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:06 smithi085 conmon[29319]: cluster 2021-02-10T04:34:05.342796+0000 2021-02-10T04:34:06.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:06 smithi085 conmon[29319]: mon.a (mon.0) 1223 : cluster [DBG] osdmap e428: 8 total, 8 up, 8 in 2021-02-10T04:34:06.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:06 smithi085 conmon[29319]: cluster 2021-02-10T04:34:06.080565+0000 mgr.y (mgr.14140) 544 : cluster [DBG] 2021-02-10T04:34:06.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:06 smithi085 conmon[29319]: pgmap v792: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 363 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.1 MiB/s rd, 3.1 MiB/s wr, 19 op/s 2021-02-10T04:34:06.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:06 smithi085 conmon[35961]: cluster 2021-02-10T04:34:05. 2021-02-10T04:34:06.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:06 smithi085 conmon[35961]: 342796+0000 mon.a (mon.0) 1223 : cluster [DBG] osdmap e428: 8 total, 8 up, 8 in 2021-02-10T04:34:06.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:06 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:06.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:06 smithi085 conmon[35961]: 10T04:34:06.080565+0000 mgr.y (mgr.14140) 544 : cluster [DBG] pgmap v792: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 363 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.1 MiB/s rd, 3.1 MiB/s wr, 19 op/s 2021-02-10T04:34:07.178 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:34:07.214 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:34:07.350 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:07 smithi085 conmon[29319]: cluster 2021-02-10T04:34:07.350 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:07 smithi085 conmon[29319]: 2021-02-10T04:34:07.350 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:07 smithi085 conmon[29319]: 2021 2021-02-10T04:34:07.350 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:07 smithi085 conmon[29319]: - 2021-02-10T04:34:07.350 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:07 smithi085 conmon[29319]: 02 2021-02-10T04:34:07.351 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:07 smithi085 conmon[29319]: - 2021-02-10T04:34:07.351 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:07 smithi085 conmon[29319]: 10T04:34:06.353400+0000 2021-02-10T04:34:07.351 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:07 smithi085 conmon[29319]: mon.a (mon.0) 1224 : cluster [DBG] osdmap e429: 8 total, 8 up, 8 in 2021-02-10T04:34:07.358 INFO:tasks.rados.rados.0.smithi085.stdout:1622: done (13 left) 2021-02-10T04:34:07.359 INFO:tasks.rados.rados.0.smithi085.stdout:1623: copy_from oid 14 from oid 20 current snap is 176 2021-02-10T04:34:07.359 INFO:tasks.rados.rados.0.smithi085.stdout:1624: snap_remove snap 135 2021-02-10T04:34:07.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:07 smithi164 conmon[29288]: cluster 2021-02-10T04:34:06.353400+0000 mon.a 2021-02-10T04:34:07.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:07 smithi164 conmon[29288]: (mon.0) 1224 : cluster [DBG] osdmap e429: 8 total, 8 up, 8 in 2021-02-10T04:34:07.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:07 smithi085 conmon[35961]: cluster 2021-02-10T04:34:07.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:07 smithi085 conmon[35961]: 2021-02-10T04:34:07.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:07 smithi085 conmon[35961]: 2021-02-10T04:34:06 2021-02-10T04:34:07.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:07 smithi085 conmon[35961]: .353400+0000 mon.a (mon.0) 1224 : cluster [DBG] osdmap e429: 8 total, 8 up, 8 in 2021-02-10T04:34:08.133 INFO:tasks.rados.rados.0.smithi085.stdout:1624: done (14 left) 2021-02-10T04:34:08.133 INFO:tasks.rados.rados.0.smithi085.stdout:1625: write oid 23 current snap is 176 2021-02-10T04:34:08.133 INFO:tasks.rados.rados.0.smithi085.stdout:1625: seq_num 577 ranges {497639=545148,1548463=407858,2529302=564422,3528143=169496} 2021-02-10T04:34:08.139 INFO:tasks.rados.rados.0.smithi085.stdout:1625: writing smithi08575397-23 from 497639 to 1042787 tid 1 2021-02-10T04:34:08.147 INFO:tasks.rados.rados.0.smithi085.stdout:1625: writing smithi08575397-23 from 1548463 to 1956321 tid 2 2021-02-10T04:34:08.152 INFO:tasks.rados.rados.0.smithi085.stdout:1625: writing smithi08575397-23 from 2529302 to 3093724 tid 3 2021-02-10T04:34:08.156 INFO:tasks.rados.rados.0.smithi085.stdout:1625: writing smithi08575397-23 from 3528143 to 3697639 tid 4 2021-02-10T04:34:08.157 INFO:tasks.rados.rados.0.smithi085.stdout:1626: write (excl) oid 35 current snap is 176 2021-02-10T04:34:08.157 INFO:tasks.rados.rados.0.smithi085.stdout:1626: seq_num 578 ranges {514446=736750,1723298=667869,3073078=641368} 2021-02-10T04:34:08.164 INFO:tasks.rados.rados.0.smithi085.stdout:1626: writing smithi08575397-35 from 514446 to 1251196 tid 1 2021-02-10T04:34:08.175 INFO:tasks.rados.rados.0.smithi085.stdout:1626: writing smithi08575397-35 from 1723298 to 2391167 tid 2 2021-02-10T04:34:08.180 INFO:tasks.rados.rados.0.smithi085.stdout:1626: writing smithi08575397-35 from 3073078 to 3714446 tid 3 2021-02-10T04:34:08.181 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:34:08.181 INFO:tasks.rados.rados.0.smithi085.stdout:1620: expect (ObjNum 521 snap 155 seq_num 521) 2021-02-10T04:34:08.306 INFO:tasks.rados.rados.0.smithi085.stdout:1618: finishing write tid 3 to smithi08575397-50 2021-02-10T04:34:08.306 INFO:tasks.rados.rados.0.smithi085.stdout:1618: finishing write tid 4 to smithi08575397-50 2021-02-10T04:34:08.306 INFO:tasks.rados.rados.0.smithi085.stdout:1620: done (15 left) 2021-02-10T04:34:08.306 INFO:tasks.rados.rados.0.smithi085.stdout:1627: snap_create 2021-02-10T04:34:08.306 INFO:tasks.rados.rados.0.smithi085.stdout:1618: finishing write tid 5 to smithi08575397-50 2021-02-10T04:34:08.306 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 633 (ObjNum 575 snap 176 seq_num 575) dirty exists 2021-02-10T04:34:08.306 INFO:tasks.rados.rados.0.smithi085.stdout:1618: left oid 50 (ObjNum 575 snap 176 seq_num 575) 2021-02-10T04:34:08.306 INFO:tasks.rados.rados.0.smithi085.stdout:1619: expect (ObjNum 527 snap 155 seq_num 527) 2021-02-10T04:34:08.359 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:34:08 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:34:08.571 INFO:tasks.rados.rados.0.smithi085.stdout:1621: finishing write tid 1 to smithi08575397-12 2021-02-10T04:34:08.571 INFO:tasks.rados.rados.0.smithi085.stdout:1621: finishing write tid 2 to smithi08575397-12 2021-02-10T04:34:08.571 INFO:tasks.rados.rados.0.smithi085.stdout:1621: finishing write tid 3 to smithi08575397-12 2021-02-10T04:34:08.572 INFO:tasks.rados.rados.0.smithi085.stdout:1621: finishing write tid 4 to smithi08575397-12 2021-02-10T04:34:08.572 INFO:tasks.rados.rados.0.smithi085.stdout:1621: finishing write tid 5 to smithi08575397-12 2021-02-10T04:34:08.572 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 743 (ObjNum 576 snap 176 seq_num 576) dirty exists 2021-02-10T04:34:08.572 INFO:tasks.rados.rados.0.smithi085.stdout:1621: left oid 12 (ObjNum 576 snap 176 seq_num 576) 2021-02-10T04:34:08.572 INFO:tasks.rados.rados.0.smithi085.stdout:1623: finishing copy_from to smithi08575397-14 2021-02-10T04:34:08.572 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 223 (ObjNum 509 snap 148 seq_num 509) dirty exists 2021-02-10T04:34:08.572 INFO:tasks.rados.rados.0.smithi085.stdout:1623: finishing copy_from racing read to smithi08575397-14 2021-02-10T04:34:08.572 INFO:tasks.rados.rados.0.smithi085.stdout:1625: finishing write tid 1 to smithi08575397-23 2021-02-10T04:34:08.572 INFO:tasks.rados.rados.0.smithi085.stdout:1625: finishing write tid 2 to smithi08575397-23 2021-02-10T04:34:08.572 INFO:tasks.rados.rados.0.smithi085.stdout:1625: finishing write tid 3 to smithi08575397-23 2021-02-10T04:34:08.572 INFO:tasks.rados.rados.0.smithi085.stdout:1625: finishing write tid 4 to smithi08575397-23 2021-02-10T04:34:08.572 INFO:tasks.rados.rados.0.smithi085.stdout:1625: finishing write tid 5 to smithi08575397-23 2021-02-10T04:34:08.572 INFO:tasks.rados.rados.0.smithi085.stdout:1625: finishing write tid 6 to smithi08575397-23 2021-02-10T04:34:08.573 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 645 (ObjNum 577 snap 176 seq_num 577) dirty exists 2021-02-10T04:34:08.573 INFO:tasks.rados.rados.0.smithi085.stdout:1625: left oid 23 (ObjNum 577 snap 176 seq_num 577) 2021-02-10T04:34:08.573 INFO:tasks.rados.rados.0.smithi085.stdout:1626: finishing write tid 1 to smithi08575397-35 2021-02-10T04:34:08.573 INFO:tasks.rados.rados.0.smithi085.stdout:1626: finishing write tid 2 to smithi08575397-35 2021-02-10T04:34:08.573 INFO:tasks.rados.rados.0.smithi085.stdout:1626: finishing write tid 3 to smithi08575397-35 2021-02-10T04:34:08.573 INFO:tasks.rados.rados.0.smithi085.stdout:1626: finishing write tid 4 to smithi08575397-35 2021-02-10T04:34:08.573 INFO:tasks.rados.rados.0.smithi085.stdout:1626: finishing write tid 5 to smithi08575397-35 2021-02-10T04:34:08.573 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 35 v 624 (ObjNum 578 snap 176 seq_num 578) dirty exists 2021-02-10T04:34:08.573 INFO:tasks.rados.rados.0.smithi085.stdout:1626: left oid 35 (ObjNum 578 snap 176 seq_num 578) 2021-02-10T04:34:08.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:08 smithi164 conmon[29288]: cluster 2021-02-10T04:34:06.635841+0000 osd.2 (osd.2) 2021-02-10T04:34:08.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:08 smithi164 conmon[29288]: 755 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:34:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:08 smithi164 conmon[29288]: cluster 2021-02-10T04:34:06.636113+0000 osd.2 (osd.2) 756 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:34:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:08 smithi164 conmon[29288]: cluster 2021-02-10T04:34:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:08 smithi164 conmon[29288]: 2021-02-10T04:34:07.357765+0000 mon.a (mon.0) 1225 : cluster 2021-02-10T04:34:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:08 smithi164 conmon[29288]: [DBG] osdmap e430: 8 total, 8 up, 8 in 2021-02-10T04:34:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:08 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:34:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:08 smithi164 conmon[29288]: 02-10T04:34:08.005029+0000 2021-02-10T04:34:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:08 smithi164 conmon[29288]: mgr.y (mgr.14140) 545 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:34:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:08 smithi164 conmon[29288]: 2021-02-10T04:34:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:08 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:08 smithi164 conmon[29288]: 34:08.081068+0000 mgr.y (mgr.14140) 546 2021-02-10T04:34:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:08 smithi164 conmon[29288]: : cluster [DBG] pgmap v795: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 363 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.1 MiB/s rd, 2.4 MiB/s wr, 15 op/s 2021-02-10T04:34:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:08 smithi164 conmon[29288]: cluster 2021-02-10T04:34:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:08 smithi164 conmon[29288]: 2021-02-10T04:34 2021-02-10T04:34:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:08 smithi164 conmon[29288]: :08.132080+0000 mon.a (mon. 2021-02-10T04:34:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:08 smithi164 conmon[29288]: 0) 1226 : cluster [DBG] osdmap e431: 8 total, 8 up, 8 in 2021-02-10T04:34:08.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[35961]: cluster 2021-02-10T04:34:06.635841 2021-02-10T04:34:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 755 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:34:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[35961]: cluster 2021-02-10T04:34:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[35961]: 2021- 2021-02-10T04:34:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[35961]: 02-10T04:34:06.636113+0000 2021-02-10T04:34:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[35961]: osd.2 (osd.2) 756 : cluster [DBG] 2021-02-10T04:34:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:34:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[35961]: cluster 2021-02-10T04:34:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[35961]: 2021-02-10T04:34:07.357765+0000 2021-02-10T04:34:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[35961]: mon.a (mon.0) 1225 : cluster [DBG] 2021-02-10T04:34:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[35961]: osdmap e430: 8 total, 8 up, 8 in 2021-02-10T04:34:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[35961]: audit 2021-02-10T04:34:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[35961]: 2021-02-10T04:34:08.005029+0000 mgr.y (mgr.14140) 2021-02-10T04:34:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[35961]: 545 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:34:08.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[35961]: cluster 2021-02-10T04:34 2021-02-10T04:34:08.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[35961]: :08.081068+0000 mgr.y (mgr.14140) 546 2021-02-10T04:34:08.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[35961]: : cluster [DBG] pgmap v795: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 363 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.1 MiB/s rd, 2.4 MiB/s wr, 15 op/s 2021-02-10T04:34:08.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:08.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[35961]: 34:08.132080+0000 mon.a (mon.0) 2021-02-10T04:34:08.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[35961]: 1226 : cluster [DBG] osdmap e431: 8 total, 8 up, 8 in 2021-02-10T04:34:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[29319]: cluster 2021-02-10T04:34:06.635841+0000 osd.2 2021-02-10T04:34:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[29319]: (osd.2) 755 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:34:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[29319]: cluster 2021-02-10T04:34:06.636113+0000 osd.2 (osd.2) 756 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:34:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[29319]: cluster 2021-02-10T04:34:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[29319]: 2021-02-10T04:34:07.357765+0000 mon.a (mon.0) 1225 : cluster [DBG] osdmap e430: 8 total, 8 up, 8 in 2021-02-10T04:34:08.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[29319]: 2021-02-10T04:34:08.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[29319]: audit 2021-02-10T04:34 2021-02-10T04:34:08.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[29319]: :08.005029+0000 mgr.y (mgr.14140) 545 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:34:08.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[29319]: cluster 2021-02-10T04:34:08.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[29319]: 2021-02-10T04:34:08.081068+0000 mgr.y (mgr.14140) 546 2021-02-10T04:34:08.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[29319]: : cluster [DBG] pgmap v795: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 363 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.1 MiB/s rd, 2.4 MiB/s wr, 15 op/s 2021-02-10T04:34:08.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[29319]: cluster 2021-02-10T04:34 2021-02-10T04:34:08.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[29319]: :08.132080+0000 mon.a (mon.0) 1226 2021-02-10T04:34:08.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:08 smithi085 conmon[29319]: : cluster [DBG] osdmap e431: 8 total, 8 up, 8 in 2021-02-10T04:34:09.142 INFO:tasks.rados.rados.0.smithi085.stdout:1618: done (15 left) 2021-02-10T04:34:09.142 INFO:tasks.rados.rados.0.smithi085.stdout:1619: done (14 left) 2021-02-10T04:34:09.142 INFO:tasks.rados.rados.0.smithi085.stdout:1621: done (13 left) 2021-02-10T04:34:09.142 INFO:tasks.rados.rados.0.smithi085.stdout:1623: done (12 left) 2021-02-10T04:34:09.142 INFO:tasks.rados.rados.0.smithi085.stdout:1625: done (11 left) 2021-02-10T04:34:09.142 INFO:tasks.rados.rados.0.smithi085.stdout:1626: done (10 left) 2021-02-10T04:34:09.142 INFO:tasks.rados.rados.0.smithi085.stdout:1627: done (9 left) 2021-02-10T04:34:09.143 INFO:tasks.rados.rados.0.smithi085.stdout:1628: write (excl) oid 36 current snap is 177 2021-02-10T04:34:09.143 INFO:tasks.rados.rados.0.smithi085.stdout:1628: seq_num 579 ranges {531253=411999,1381780=411527,2300501=586966,3376974=354279} 2021-02-10T04:34:09.148 INFO:tasks.rados.rados.0.smithi085.stdout:1628: writing smithi08575397-36 from 531253 to 943252 tid 1 2021-02-10T04:34:09.153 INFO:tasks.rados.rados.0.smithi085.stdout:1628: writing smithi08575397-36 from 1381780 to 1793307 tid 2 2021-02-10T04:34:09.160 INFO:tasks.rados.rados.0.smithi085.stdout:1628: writing smithi08575397-36 from 2300501 to 2887467 tid 3 2021-02-10T04:34:09.166 INFO:tasks.rados.rados.0.smithi085.stdout:1628: writing smithi08575397-36 from 3376974 to 3731253 tid 4 2021-02-10T04:34:09.166 INFO:tasks.rados.rados.0.smithi085.stdout:1629: write oid 6 current snap is 177 2021-02-10T04:34:09.166 INFO:tasks.rados.rados.0.smithi085.stdout:1628: finishing write tid 1 to smithi08575397-36 2021-02-10T04:34:09.166 INFO:tasks.rados.rados.0.smithi085.stdout:1629: seq_num 580 ranges {548060=487248,1440262=671538,2844277=798238} 2021-02-10T04:34:09.172 INFO:tasks.rados.rados.0.smithi085.stdout:1629: writing smithi08575397-6 from 548060 to 1035308 tid 1 2021-02-10T04:34:09.178 INFO:tasks.rados.rados.0.smithi085.stdout:1629: writing smithi08575397-6 from 1440262 to 2111800 tid 2 2021-02-10T04:34:09.188 INFO:tasks.rados.rados.0.smithi085.stdout:1629: writing smithi08575397-6 from 2844277 to 3642515 tid 3 2021-02-10T04:34:09.188 INFO:tasks.rados.rados.0.smithi085.stdout:1630: delete oid 23 current snap is 177 2021-02-10T04:34:09.188 INFO:tasks.rados.rados.0.smithi085.stdout:1628: finishing write tid 2 to smithi08575397-36 2021-02-10T04:34:09.188 INFO:tasks.rados.rados.0.smithi085.stdout:1628: finishing write tid 3 to smithi08575397-36 2021-02-10T04:34:09.188 INFO:tasks.rados.rados.0.smithi085.stdout:1628: finishing write tid 4 to smithi08575397-36 2021-02-10T04:34:09.188 INFO:tasks.rados.rados.0.smithi085.stdout:1628: finishing write tid 5 to smithi08575397-36 2021-02-10T04:34:09.188 INFO:tasks.rados.rados.0.smithi085.stdout:1628: finishing write tid 6 to smithi08575397-36 2021-02-10T04:34:09.188 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 650 (ObjNum 579 snap 177 seq_num 579) dirty exists 2021-02-10T04:34:09.188 INFO:tasks.rados.rados.0.smithi085.stdout:1628: left oid 36 (ObjNum 579 snap 177 seq_num 579) 2021-02-10T04:34:09.188 INFO:tasks.rados.rados.0.smithi085.stdout:1629: finishing write tid 1 to smithi08575397-6 2021-02-10T04:34:09.189 INFO:tasks.rados.rados.0.smithi085.stdout:1629: finishing write tid 2 to smithi08575397-6 2021-02-10T04:34:09.191 INFO:tasks.rados.rados.0.smithi085.stdout:1628: done (11 left) 2021-02-10T04:34:09.191 INFO:tasks.rados.rados.0.smithi085.stdout:1630: done (10 left) 2021-02-10T04:34:09.191 INFO:tasks.rados.rados.0.smithi085.stdout:1631: rollback oid 7 current snap is 177 2021-02-10T04:34:09.191 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 7 to 176 2021-02-10T04:34:09.191 INFO:tasks.rados.rados.0.smithi085.stdout:1632: write (excl) oid 5 current snap is 177 2021-02-10T04:34:09.191 INFO:tasks.rados.rados.0.smithi085.stdout:1632: seq_num 582 ranges {581674=637746,1840873=558854,3066417=537135} 2021-02-10T04:34:09.198 INFO:tasks.rados.rados.0.smithi085.stdout:1632: writing smithi08575397-5 from 581674 to 1219420 tid 1 2021-02-10T04:34:09.205 INFO:tasks.rados.rados.0.smithi085.stdout:1632: writing smithi08575397-5 from 1840873 to 2399727 tid 2 2021-02-10T04:34:09.212 INFO:tasks.rados.rados.0.smithi085.stdout:1632: writing smithi08575397-5 from 3066417 to 3603552 tid 3 2021-02-10T04:34:09.212 INFO:tasks.rados.rados.0.smithi085.stdout:1633: write oid 12 current snap is 177 2021-02-10T04:34:09.213 INFO:tasks.rados.rados.0.smithi085.stdout:1629: finishing write tid 3 to smithi08575397-6 2021-02-10T04:34:09.213 INFO:tasks.rados.rados.0.smithi085.stdout:1633: seq_num 583 ranges {598481=712995,1899355=418865,3093840=704641} 2021-02-10T04:34:09.219 INFO:tasks.rados.rados.0.smithi085.stdout:1633: writing smithi08575397-12 from 598481 to 1311476 tid 1 2021-02-10T04:34:09.225 INFO:tasks.rados.rados.0.smithi085.stdout:1633: writing smithi08575397-12 from 1899355 to 2318220 tid 2 2021-02-10T04:34:09.234 INFO:tasks.rados.rados.0.smithi085.stdout:1633: writing smithi08575397-12 from 3093840 to 3798481 tid 3 2021-02-10T04:34:09.234 INFO:tasks.rados.rados.0.smithi085.stdout:1634: snap_remove snap 174 2021-02-10T04:34:10.109 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:34:10.112 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:34:09 smithi085 conmon[52836]: debug 2021-02-10T04:34:09.856+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:34:10.147 INFO:tasks.rados.rados.0.smithi085.stdout:1634: done (13 left) 2021-02-10T04:34:10.147 INFO:tasks.rados.rados.0.smithi085.stdout:1635: read oid 40 snap -1 2021-02-10T04:34:10.147 INFO:tasks.rados.rados.0.smithi085.stdout:1635: expect (ObjNum 491 snap 141 seq_num 491) 2021-02-10T04:34:10.147 INFO:tasks.rados.rados.0.smithi085.stdout:1636: delete oid 14 current snap is 177 2021-02-10T04:34:10.147 INFO:tasks.rados.rados.0.smithi085.stdout:1629: finishing write tid 4 to smithi08575397-6 2021-02-10T04:34:10.147 INFO:tasks.rados.rados.0.smithi085.stdout:1629: finishing write tid 5 to smithi08575397-6 2021-02-10T04:34:10.147 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 750 (ObjNum 580 snap 177 seq_num 580) dirty exists 2021-02-10T04:34:10.148 INFO:tasks.rados.rados.0.smithi085.stdout:1629: left oid 6 (ObjNum 580 snap 177 seq_num 580) 2021-02-10T04:34:10.148 INFO:tasks.rados.rados.0.smithi085.stdout:1631: finishing rollback tid 0 to smithi08575397-7 2021-02-10T04:34:10.148 INFO:tasks.rados.rados.0.smithi085.stdout:1631: finishing rollback tid 1 to smithi08575397-7 2021-02-10T04:34:10.148 INFO:tasks.rados.rados.0.smithi085.stdout:1631: finishing rollback tid 2 to smithi08575397-7 2021-02-10T04:34:10.148 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 686 (ObjNum 553 snap 172 seq_num 553) dirty exists 2021-02-10T04:34:10.148 INFO:tasks.rados.rados.0.smithi085.stdout:1632: finishing write tid 1 to smithi08575397-5 2021-02-10T04:34:10.148 INFO:tasks.rados.rados.0.smithi085.stdout:1632: finishing write tid 2 to smithi08575397-5 2021-02-10T04:34:10.148 INFO:tasks.rados.rados.0.smithi085.stdout:1632: finishing write tid 3 to smithi08575397-5 2021-02-10T04:34:10.148 INFO:tasks.rados.rados.0.smithi085.stdout:1632: finishing write tid 4 to smithi08575397-5 2021-02-10T04:34:10.148 INFO:tasks.rados.rados.0.smithi085.stdout:1632: finishing write tid 5 to smithi08575397-5 2021-02-10T04:34:10.148 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 653 (ObjNum 582 snap 177 seq_num 582) dirty exists 2021-02-10T04:34:10.148 INFO:tasks.rados.rados.0.smithi085.stdout:1632: left oid 5 (ObjNum 582 snap 177 seq_num 582) 2021-02-10T04:34:10.148 INFO:tasks.rados.rados.0.smithi085.stdout:1633: finishing write tid 1 to smithi08575397-12 2021-02-10T04:34:10.148 INFO:tasks.rados.rados.0.smithi085.stdout:1633: finishing write tid 2 to smithi08575397-12 2021-02-10T04:34:10.149 INFO:tasks.rados.rados.0.smithi085.stdout:1633: finishing write tid 3 to smithi08575397-12 2021-02-10T04:34:10.149 INFO:tasks.rados.rados.0.smithi085.stdout:1633: finishing write tid 4 to smithi08575397-12 2021-02-10T04:34:10.149 INFO:tasks.rados.rados.0.smithi085.stdout:1633: finishing write tid 5 to smithi08575397-12 2021-02-10T04:34:10.149 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 747 (ObjNum 583 snap 177 seq_num 583) dirty exists 2021-02-10T04:34:10.149 INFO:tasks.rados.rados.0.smithi085.stdout:1633: left oid 12 (ObjNum 583 snap 177 seq_num 583) 2021-02-10T04:34:10.149 INFO:tasks.rados.rados.0.smithi085.stdout:1629: done (14 left) 2021-02-10T04:34:10.150 INFO:tasks.rados.rados.0.smithi085.stdout:1631: done (13 left) 2021-02-10T04:34:10.150 INFO:tasks.rados.rados.0.smithi085.stdout:1632: done (12 left) 2021-02-10T04:34:10.150 INFO:tasks.rados.rados.0.smithi085.stdout:1633: done (11 left) 2021-02-10T04:34:10.150 INFO:tasks.rados.rados.0.smithi085.stdout:1636: done (10 left) 2021-02-10T04:34:10.150 INFO:tasks.rados.rados.0.smithi085.stdout:1637: rollback oid 10 current snap is 177 2021-02-10T04:34:10.150 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 10 to 167 2021-02-10T04:34:10.150 INFO:tasks.rados.rados.0.smithi085.stdout:1638: write oid 28 current snap is 177 2021-02-10T04:34:10.150 INFO:tasks.rados.rados.0.smithi085.stdout:1638: seq_num 585 ranges {632095=463493,1616319=422534,2748686=487304} 2021-02-10T04:34:10.156 INFO:tasks.rados.rados.0.smithi085.stdout:1638: writing smithi08575397-28 from 632095 to 1095588 tid 1 2021-02-10T04:34:10.162 INFO:tasks.rados.rados.0.smithi085.stdout:1638: writing smithi08575397-28 from 1616319 to 2038853 tid 2 2021-02-10T04:34:10.170 INFO:tasks.rados.rados.0.smithi085.stdout:1638: writing smithi08575397-28 from 2748686 to 3235990 tid 3 2021-02-10T04:34:10.170 INFO:tasks.rados.rados.0.smithi085.stdout:1637: finishing rollback tid 1 to smithi08575397-10 2021-02-10T04:34:10.170 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 10 v 0 (ObjNum 1 snap 22030 seq_num 83266288) dirty dne 2021-02-10T04:34:10.170 INFO:tasks.rados.rados.0.smithi085.stdout:1635: expect (ObjNum 491 snap 141 seq_num 491) 2021-02-10T04:34:10.214 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:34:10.242 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:34:10.335 INFO:tasks.rados.rados.0.smithi085.stdout:1638: finishing write tid 1 to smithi08575397-28 2021-02-10T04:34:10.335 INFO:tasks.rados.rados.0.smithi085.stdout:1638: finishing write tid 2 to smithi08575397-28 2021-02-10T04:34:10.336 INFO:tasks.rados.rados.0.smithi085.stdout:1635: done (11 left) 2021-02-10T04:34:10.336 INFO:tasks.rados.rados.0.smithi085.stdout:1637: done (10 left) 2021-02-10T04:34:10.336 INFO:tasks.rados.rados.0.smithi085.stdout:1639: read oid 22 snap -1 2021-02-10T04:34:10.336 INFO:tasks.rados.rados.0.smithi085.stdout:1639: expect (ObjNum 570 snap 175 seq_num 570) 2021-02-10T04:34:10.336 INFO:tasks.rados.rados.0.smithi085.stdout:1638: finishing write tid 3 to smithi08575397-28 2021-02-10T04:34:10.336 INFO:tasks.rados.rados.0.smithi085.stdout:1638: finishing write tid 4 to smithi08575397-28 2021-02-10T04:34:10.336 INFO:tasks.rados.rados.0.smithi085.stdout:1638: finishing write tid 5 to smithi08575397-28 2021-02-10T04:34:10.336 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 727 (ObjNum 585 snap 177 seq_num 585) dirty exists 2021-02-10T04:34:10.336 INFO:tasks.rados.rados.0.smithi085.stdout:1638: left oid 28 (ObjNum 585 snap 177 seq_num 585) 2021-02-10T04:34:10.336 INFO:tasks.rados.rados.0.smithi085.stdout:1640: read oid 6 snap -1 2021-02-10T04:34:10.336 INFO:tasks.rados.rados.0.smithi085.stdout:1640: expect (ObjNum 580 snap 177 seq_num 580) 2021-02-10T04:34:10.336 INFO:tasks.rados.rados.0.smithi085.stdout:1638: done (11 left) 2021-02-10T04:34:10.336 INFO:tasks.rados.rados.0.smithi085.stdout:1641: read oid 31 snap 149 2021-02-10T04:34:10.337 INFO:tasks.rados.rados.0.smithi085.stdout:1641: expect deleted 2021-02-10T04:34:10.337 INFO:tasks.rados.rados.0.smithi085.stdout:1642: delete oid 48 current snap is 177 2021-02-10T04:34:10.338 INFO:tasks.rados.rados.0.smithi085.stdout:1642: done (12 left) 2021-02-10T04:34:10.338 INFO:tasks.rados.rados.0.smithi085.stdout:1643: delete oid 30 current snap is 177 2021-02-10T04:34:10.342 INFO:tasks.rados.rados.0.smithi085.stdout:1640: expect (ObjNum 580 snap 177 seq_num 580) 2021-02-10T04:34:10.357 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:34:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:10 smithi085 conmon[29319]: cluster 2021-02-10T04:34:09.142442+0000 mon.a (mon.0) 2021-02-10T04:34:10.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:10 smithi085 conmon[29319]: 1227 : cluster [DBG] osdmap e432: 8 total, 8 up, 8 in 2021-02-10T04:34:10.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:10 smithi085 conmon[35961]: cluster 2021-02-10T04:34:09. 2021-02-10T04:34:10.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:10 smithi085 conmon[35961]: 142442+0000 mon.a (mon.0) 1227 : cluster [DBG] osdmap e432: 8 total, 8 up, 8 in 2021-02-10T04:34:10.462 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:34:10.486 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:10 smithi164 conmon[29288]: cluster 2021-02-10T04:34:09.142442+0000 mon.a (mon.0) 2021-02-10T04:34:10.486 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:10 smithi164 conmon[29288]: 1227 : cluster [DBG] osdmap e432: 8 total, 8 up, 8 in 2021-02-10T04:34:10.486 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:34:10 smithi164 conmon[44544]: debug 2021-02-10T04:34:10.298+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:34:10.488 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:34:10.522 INFO:tasks.rados.rados.0.smithi085.stdout:1640: done (12 left) 2021-02-10T04:34:10.522 INFO:tasks.rados.rados.0.smithi085.stdout:1643: done (11 left) 2021-02-10T04:34:10.522 INFO:tasks.rados.rados.0.smithi085.stdout:1644: read oid 35 snap 123 2021-02-10T04:34:10.522 INFO:tasks.rados.rados.0.smithi085.stdout:1644: expect (ObjNum 85 snap 11 seq_num 85) 2021-02-10T04:34:10.522 INFO:tasks.rados.rados.0.smithi085.stdout:1645: copy_from oid 11 from oid 43 current snap is 177 2021-02-10T04:34:10.522 INFO:tasks.rados.rados.0.smithi085.stdout:1639: expect (ObjNum 570 snap 175 seq_num 570) 2021-02-10T04:34:10.582 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:34:10.687 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:34:10.714 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:34:10.781 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:34:10 smithi164 conmon[44544]: debug 2021-02-10T04:34:10.553+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:34:10.799 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 2 1 2021-02-10T04:34:10.816 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:34:10.823 INFO:tasks.rados.rados.0.smithi085.stdout:1645: finishing copy_from to smithi08575397-11 2021-02-10T04:34:10.823 INFO:tasks.rados.rados.0.smithi085.stdout:1645: got expected ENOENT (src dne) 2021-02-10T04:34:10.823 INFO:tasks.rados.rados.0.smithi085.stdout:1639: done (12 left) 2021-02-10T04:34:10.824 INFO:tasks.rados.rados.0.smithi085.stdout:1641: done (11 left) 2021-02-10T04:34:10.824 INFO:tasks.rados.rados.0.smithi085.stdout:1646: snap_create 2021-02-10T04:34:10.824 INFO:tasks.rados.rados.0.smithi085.stdout:1645: finishing copy_from racing read to smithi08575397-11 2021-02-10T04:34:10.824 INFO:tasks.rados.rados.0.smithi085.stdout:1644: expect (ObjNum 85 snap 11 seq_num 85) 2021-02-10T04:34:10.921 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:34:10.953 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:34:11.143 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:34:10 smithi164 conmon[44544]: debug 2021-02-10T04:34:10.781+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:34:11.164 INFO:tasks.rados.rados.0.smithi085.stdout:1644: done (11 left) 2021-02-10T04:34:11.164 INFO:tasks.rados.rados.0.smithi085.stdout:1645: done (10 left) 2021-02-10T04:34:11.164 INFO:tasks.rados.rados.0.smithi085.stdout:1646: done (9 left) 2021-02-10T04:34:11.164 INFO:tasks.rados.rados.0.smithi085.stdout:1647: delete oid 26 current snap is 178 2021-02-10T04:34:11.168 INFO:tasks.rados.rados.0.smithi085.stdout:1647: done (9 left) 2021-02-10T04:34:11.168 INFO:tasks.rados.rados.0.smithi085.stdout:1648: snap_remove snap 169 2021-02-10T04:34:11.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:11 smithi085 conmon[29319]: cluster 2021-02-10T04:34:10.081580+0000 mgr.y (mgr 2021-02-10T04:34:11.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:11 smithi085 conmon[29319]: .14140) 547 : cluster [DBG] pgmap v798: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 367 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.6 MiB/s rd, 1.8 MiB/s wr, 11 op/s 2021-02-10T04:34:11.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:11 smithi085 conmon[29319]: cluster 2021-02-10T04:34:10.141824+0000 mon.a (mon.0) 1228 : cluster 2021-02-10T04:34:11.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:11 smithi085 conmon[29319]: [DBG] osdmap e433: 8 total, 8 up, 8 in 2021-02-10T04:34:11.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:11 smithi085 conmon[35961]: cluster 2021-02-10T04:34:10.081580+0000 2021-02-10T04:34:11.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:11 smithi085 conmon[35961]: mgr.y (mgr.14140) 547 : cluster [DBG] pgmap v798: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 367 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.6 MiB/s rd, 1.8 MiB/s wr, 11 op/s 2021-02-10T04:34:11.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:11 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:11.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:11 smithi085 conmon[35961]: T04:34:10.141824+0000 mon.a (mon.0) 1228 : cluster [DBG] osdmap e433: 8 total, 8 up, 8 in 2021-02-10T04:34:11.443 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:34:11 smithi085 conmon[45443]: debug 2021-02-10T04:34:11.351+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:34:11.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:11 smithi164 conmon[29288]: cluster 2021-02-10T04:34:11.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:11 smithi164 conmon[29288]: 2021-02-10T04:34: 2021-02-10T04:34:11.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:11 smithi164 conmon[29288]: 10.081580+0000 mgr.y (mgr.14140) 547 : cluster [DBG] pgmap v798: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 367 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.6 MiB/s rd, 1.8 MiB/s wr, 11 op/s 2021-02-10T04:34:11.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:11 smithi164 conmon[29288]: cluster 2021-02-10T04:34:10.141824+0000 mon.a (mon.0) 1228 : cluster [DBG] osdmap e433: 8 total, 8 up, 8 in 2021-02-10T04:34:11.588 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:11 smithi164 conmon[33677]: debug 2021-02-10T04:34:11.586+0000 7fbea5924700 -1 osd.4 pg_epoch: 434 pg[6.1a( empty local-lis/les=59/60 n=0 ec=59/59 lis/c=59/59 les/c/f=60/60/0 sis=59) [4,5,1] r=0 lpr=59 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.1a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x56275da72b00) 2021-02-10T04:34:11.608 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:34:11.614 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:34:11.714 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:34:11.743 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:34:12.166 INFO:tasks.rados.rados.0.smithi085.stdout:1648: done (9 left) 2021-02-10T04:34:12.166 INFO:tasks.rados.rados.0.smithi085.stdout:1649: read oid 38 snap -1 2021-02-10T04:34:12.166 INFO:tasks.rados.rados.0.smithi085.stdout:1649: expect (ObjNum 574 snap 175 seq_num 574) 2021-02-10T04:34:12.166 INFO:tasks.rados.rados.0.smithi085.stdout:1650: read oid 16 snap 161 2021-02-10T04:34:12.166 INFO:tasks.rados.rados.0.smithi085.stdout:1650: expect (ObjNum 523 snap 155 seq_num 523) 2021-02-10T04:34:12.167 INFO:tasks.rados.rados.0.smithi085.stdout:1651: rollback oid 26 current snap is 178 2021-02-10T04:34:12.167 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 26 to 147 2021-02-10T04:34:12.167 INFO:tasks.rados.rados.0.smithi085.stdout:1652: snap_remove snap 128 2021-02-10T04:34:12.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:12 smithi085 conmon[29319]: cluster 2021-02-10T04:34:12.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:12 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:34:12.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:12 smithi085 conmon[29319]: :34:11.159531+0000 mon.a (mon.0) 1229 : cluster [DBG] osdmap e434: 8 total, 8 up, 8 in 2021-02-10T04:34:12.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:12 smithi085 conmon[35961]: cluster 2021-02-10T04:34:12.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:12 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:34:12.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:12 smithi085 conmon[35961]: 34:11.159531+0000 mon.a (mon.0) 1229 : cluster [DBG] osdmap e434: 8 total, 8 up, 8 in 2021-02-10T04:34:12.442 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:34:12 smithi085 conmon[41540]: debug 2021-02-10T04:34:12.219+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:34:12.455 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:34:12.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:12 smithi164 conmon[29288]: cluster 2021-02-10T04:34:11.159531+0000 mon.a (mon.0) 1229 2021-02-10T04:34:12.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:12 smithi164 conmon[29288]: : cluster [DBG] osdmap e434: 8 total, 8 up, 8 in 2021-02-10T04:34:12.920 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:34:12.948 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:34:13.040 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:34:13.145 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:34:13.182 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:34:13.216 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:34:13 smithi164 conmon[37263]: debug 2021-02-10T04:34:13.015+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:34:13.232 INFO:tasks.rados.rados.0.smithi085.stdout:1652: done (12 left) 2021-02-10T04:34:13.232 INFO:tasks.rados.rados.0.smithi085.stdout:1653: write (excl) oid 28 current snap is 178 2021-02-10T04:34:13.232 INFO:tasks.rados.rados.0.smithi085.stdout:1653: seq_num 589 ranges {699323=480842,1850247=429872,2858378=532392} 2021-02-10T04:34:13.235 INFO:tasks.rados.rados.0.smithi085.stdout:1653: writing smithi08575397-28 from 699323 to 1180165 tid 1 2021-02-10T04:34:13.244 INFO:tasks.rados.rados.0.smithi085.stdout:1653: writing smithi08575397-28 from 1850247 to 2280119 tid 2 2021-02-10T04:34:13.251 INFO:tasks.rados.rados.0.smithi085.stdout:1653: writing smithi08575397-28 from 2858378 to 3390770 tid 3 2021-02-10T04:34:13.264 INFO:tasks.rados.rados.0.smithi085.stdout:1651: finishing rollback tid 1 to smithi08575397-26 2021-02-10T04:34:13.264 INFO:tasks.rados.rados.0.smithi085.stdout:1651: finishing rollback tid 2 to smithi08575397-26 2021-02-10T04:34:13.264 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 643 (ObjNum 481 snap 139 seq_num 481) dirty exists 2021-02-10T04:34:13.264 INFO:tasks.rados.rados.0.smithi085.stdout:1649: expect (ObjNum 574 snap 175 seq_num 574) 2021-02-10T04:34:13.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:13 smithi085 conmon[29319]: cluster 2021-02-10T04:34:12.082151+0000 2021-02-10T04:34:13.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:13 smithi085 conmon[29319]: mgr.y (mgr.14140) 548 : cluster [DBG] pgmap v801: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 363 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 3.6 MiB/s wr, 23 op/s 2021-02-10T04:34:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:13 smithi085 conmon[29319]: cluster 2021-02-10T04:34:12.165551+0000 mon.a (mon.0) 1230 : cluster [DBG] osdmap e435: 8 total, 8 up, 8 in 2021-02-10T04:34:13.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:13 smithi085 conmon[35961]: cluster 2021-02-10T04:34:12.082151+0000 mgr.y ( 2021-02-10T04:34:13.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:13 smithi085 conmon[35961]: mgr.14140) 548 : cluster [DBG] pgmap v801: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 363 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 3.6 MiB/s wr, 23 op/s 2021-02-10T04:34:13.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:13 smithi085 conmon[35961]: cluster 2021-02-10T04:34:12.165551+0000 mon.a (mon.0) 1230 : cluster 2021-02-10T04:34:13.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:13 smithi085 conmon[35961]: [DBG] osdmap e435: 8 total, 8 up, 8 in 2021-02-10T04:34:13.480 INFO:tasks.rados.rados.0.smithi085.stdout:1650: expect (ObjNum 523 snap 155 seq_num 523) 2021-02-10T04:34:13.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:13 smithi164 conmon[29288]: cluster 2021-02-10T04:34:12.082151+0000 mgr.y (mgr.14140) 2021-02-10T04:34:13.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:13 smithi164 conmon[29288]: 548 : cluster [DBG] pgmap v801: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 363 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 3.6 MiB/s wr, 23 op/s 2021-02-10T04:34:13.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:13 smithi164 conmon[29288]: cluster 2021-02-10T04:34:12.165551+0000 mon.a (mon.0) 1230 : cluster [DBG] osdmap e435: 8 total, 8 up, 8 in 2021-02-10T04:34:13.721 INFO:tasks.rados.rados.0.smithi085.stdout:1653: finishing write tid 1 to smithi08575397-28 2021-02-10T04:34:13.721 INFO:tasks.rados.rados.0.smithi085.stdout:1653: finishing write tid 2 to smithi08575397-28 2021-02-10T04:34:13.721 INFO:tasks.rados.rados.0.smithi085.stdout:1649: done (12 left) 2021-02-10T04:34:13.721 INFO:tasks.rados.rados.0.smithi085.stdout:1650: done (11 left) 2021-02-10T04:34:13.721 INFO:tasks.rados.rados.0.smithi085.stdout:1651: done (10 left) 2021-02-10T04:34:13.721 INFO:tasks.rados.rados.0.smithi085.stdout:1654: delete oid 6 current snap is 178 2021-02-10T04:34:13.721 INFO:tasks.rados.rados.0.smithi085.stdout:1653: finishing write tid 3 to smithi08575397-28 2021-02-10T04:34:13.721 INFO:tasks.rados.rados.0.smithi085.stdout:1653: finishing write tid 4 to smithi08575397-28 2021-02-10T04:34:13.721 INFO:tasks.rados.rados.0.smithi085.stdout:1653: finishing write tid 5 to smithi08575397-28 2021-02-10T04:34:13.722 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 731 (ObjNum 589 snap 178 seq_num 589) dirty exists 2021-02-10T04:34:13.722 INFO:tasks.rados.rados.0.smithi085.stdout:1653: left oid 28 (ObjNum 589 snap 178 seq_num 589) 2021-02-10T04:34:13.723 INFO:tasks.rados.rados.0.smithi085.stdout:1653: done (10 left) 2021-02-10T04:34:13.723 INFO:tasks.rados.rados.0.smithi085.stdout:1654: done (9 left) 2021-02-10T04:34:13.723 INFO:tasks.rados.rados.0.smithi085.stdout:1655: rollback oid 26 current snap is 178 2021-02-10T04:34:13.723 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 26 to 142 2021-02-10T04:34:13.723 INFO:tasks.rados.rados.0.smithi085.stdout:1656: write oid 3 current snap is 178 2021-02-10T04:34:13.723 INFO:tasks.rados.rados.0.smithi085.stdout:1656: seq_num 591 ranges {732937=631340,1850858=433541,2796871=671289,3888195=44742} 2021-02-10T04:34:13.732 INFO:tasks.rados.rados.0.smithi085.stdout:1656: writing smithi08575397-3 from 732937 to 1364277 tid 1 2021-02-10T04:34:13.740 INFO:tasks.rados.rados.0.smithi085.stdout:1656: writing smithi08575397-3 from 1850858 to 2284399 tid 2 2021-02-10T04:34:13.747 INFO:tasks.rados.rados.0.smithi085.stdout:1656: writing smithi08575397-3 from 2796871 to 3468160 tid 3 2021-02-10T04:34:13.750 INFO:tasks.rados.rados.0.smithi085.stdout:1656: writing smithi08575397-3 from 3888195 to 3932937 tid 4 2021-02-10T04:34:13.751 INFO:tasks.rados.rados.0.smithi085.stdout:1657: rollback oid 48 current snap is 178 2021-02-10T04:34:13.751 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 48 to 170 2021-02-10T04:34:13.751 INFO:tasks.rados.rados.0.smithi085.stdout:1655: finishing rollback tid 0 to smithi08575397-26 2021-02-10T04:34:13.751 INFO:tasks.rados.rados.0.smithi085.stdout:1655: finishing rollback tid 1 to smithi08575397-26 2021-02-10T04:34:13.751 INFO:tasks.rados.rados.0.smithi085.stdout:1655: finishing rollback tid 2 to smithi08575397-26 2021-02-10T04:34:13.751 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 646 (ObjNum 481 snap 139 seq_num 481) dirty exists 2021-02-10T04:34:13.751 INFO:tasks.rados.rados.0.smithi085.stdout:1656: finishing write tid 1 to smithi08575397-3 2021-02-10T04:34:13.751 INFO:tasks.rados.rados.0.smithi085.stdout:1656: finishing write tid 2 to smithi08575397-3 2021-02-10T04:34:13.751 INFO:tasks.rados.rados.0.smithi085.stdout:1655: done (11 left) 2021-02-10T04:34:13.751 INFO:tasks.rados.rados.0.smithi085.stdout:1658: rollback oid 43 current snap is 178 2021-02-10T04:34:13.751 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 43 to 160 2021-02-10T04:34:13.751 INFO:tasks.rados.rados.0.smithi085.stdout:1659: delete oid 50 current snap is 178 2021-02-10T04:34:13.752 INFO:tasks.rados.rados.0.smithi085.stdout:1656: finishing write tid 3 to smithi08575397-3 2021-02-10T04:34:13.753 INFO:tasks.rados.rados.0.smithi085.stdout:1657: finishing rollback tid 1 to smithi08575397-48 2021-02-10T04:34:13.754 INFO:tasks.rados.rados.0.smithi085.stdout:1659: done (12 left) 2021-02-10T04:34:13.754 INFO:tasks.rados.rados.0.smithi085.stdout:1660: snap_remove snap 173 2021-02-10T04:34:13.858 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:34:13.864 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:34:13 smithi085 conmon[52836]: debug 2021-02-10T04:34:13.670+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:34:13.917 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:34:13.917 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:34:13.965 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:34:13.989 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:34:14.246 INFO:tasks.rados.rados.0.smithi085.stdout:1660: done (12 left) 2021-02-10T04:34:14.247 INFO:tasks.rados.rados.0.smithi085.stdout:1661: rollback oid 16 current snap is 178 2021-02-10T04:34:14.247 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 16 to 162 2021-02-10T04:34:14.247 INFO:tasks.rados.rados.0.smithi085.stdout:1656: finishing write tid 4 to smithi08575397-3 2021-02-10T04:34:14.247 INFO:tasks.rados.rados.0.smithi085.stdout:1657: finishing rollback tid 2 to smithi08575397-48 2021-02-10T04:34:14.247 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 751 (ObjNum 476 snap 135 seq_num 476) dirty exists 2021-02-10T04:34:14.247 INFO:tasks.rados.rados.0.smithi085.stdout:1656: finishing write tid 5 to smithi08575397-3 2021-02-10T04:34:14.247 INFO:tasks.rados.rados.0.smithi085.stdout:1656: finishing write tid 6 to smithi08575397-3 2021-02-10T04:34:14.247 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 691 (ObjNum 591 snap 178 seq_num 591) dirty exists 2021-02-10T04:34:14.247 INFO:tasks.rados.rados.0.smithi085.stdout:1656: left oid 3 (ObjNum 591 snap 178 seq_num 591) 2021-02-10T04:34:14.247 INFO:tasks.rados.rados.0.smithi085.stdout:1658: finishing rollback tid 1 to smithi08575397-43 2021-02-10T04:34:14.248 INFO:tasks.rados.rados.0.smithi085.stdout:1658: finishing rollback tid 2 to smithi08575397-43 2021-02-10T04:34:14.248 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 693 (ObjNum 491 snap 141 seq_num 491) dirty exists 2021-02-10T04:34:14.248 INFO:tasks.rados.rados.0.smithi085.stdout:1656: done (12 left) 2021-02-10T04:34:14.248 INFO:tasks.rados.rados.0.smithi085.stdout:1657: done (11 left) 2021-02-10T04:34:14.248 INFO:tasks.rados.rados.0.smithi085.stdout:1658: done (10 left) 2021-02-10T04:34:14.248 INFO:tasks.rados.rados.0.smithi085.stdout:1662: read oid 20 snap -1 2021-02-10T04:34:14.248 INFO:tasks.rados.rados.0.smithi085.stdout:1662: expect (ObjNum 509 snap 148 seq_num 509) 2021-02-10T04:34:14.248 INFO:tasks.rados.rados.0.smithi085.stdout:1663: write (excl) oid 7 current snap is 178 2021-02-10T04:34:14.248 INFO:tasks.rados.rados.0.smithi085.stdout:1663: seq_num 593 ranges {766551=498191,1684175=720857,2851717=693833} 2021-02-10T04:34:14.248 INFO:tasks.rados.rados.0.smithi085.stdout:1663: writing smithi08575397-7 from 766551 to 1264742 tid 1 2021-02-10T04:34:14.256 INFO:tasks.rados.rados.0.smithi085.stdout:1663: writing smithi08575397-7 from 1684175 to 2405032 tid 2 2021-02-10T04:34:14.263 INFO:tasks.rados.rados.0.smithi085.stdout:1663: writing smithi08575397-7 from 2851717 to 3545550 tid 3 2021-02-10T04:34:14.263 INFO:tasks.rados.rados.0.smithi085.stdout:1664: write (excl) oid 10 current snap is 178 2021-02-10T04:34:14.263 INFO:tasks.rados.rados.0.smithi085.stdout:1662: expect (ObjNum 509 snap 148 seq_num 509) 2021-02-10T04:34:14.365 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:34:14.369 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:14 smithi164 conmon[29288]: cluster 2021-02-10T04:34:14.369 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:14 smithi164 conmon[29288]: 2021-02-10 2021-02-10T04:34:14.369 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:14 smithi164 conmon[29288]: T04:34:13.228658+0000 mon.a (mon.0) 1231 : cluster [DBG] osdmap e436: 8 total, 8 up, 8 in 2021-02-10T04:34:14.369 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:14 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:34:14.369 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:14 smithi164 conmon[29288]: 02-10T04:34:13.883834+0000 mon.a (mon.0 2021-02-10T04:34:14.369 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:14 smithi164 conmon[29288]: ) 1232 : audit [DBG] from='client.? 172.21.15.85:0/295028155' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:34:14.369 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:14 smithi164 conmon[29288]: cluster 2021-02-10T04:34:14.369 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:14 smithi164 conmon[29288]: 2021-02-10T04:34:14.082588+0000 2021-02-10T04:34:14.370 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:14 smithi164 conmon[29288]: mgr.y (mgr.14140) 549 : cluster [DBG] pgmap v804: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 365 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 14 op/s 2021-02-10T04:34:14.370 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:34:14 smithi164 conmon[44544]: debug 2021-02-10T04:34:14.054+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:34:14.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:14 smithi085 conmon[29319]: cluster 2021-02-10T04:34:13.228658+0000 mon.a 2021-02-10T04:34:14.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:14 smithi085 conmon[29319]: (mon.0) 1231 : cluster [DBG] osdmap e436: 8 total, 8 up, 8 in 2021-02-10T04:34:14.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:14 smithi085 conmon[29319]: audit 2021-02-10T04:34:13.883834+0000 mon.a (mon. 2021-02-10T04:34:14.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:14 smithi085 conmon[29319]: 0) 1232 : audit [DBG] from='client.? 172.21.15.85:0/295028155' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:34:14.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:14 smithi085 conmon[29319]: cluster 2021-02-10T04:34:14.082588+0000 2021-02-10T04:34:14.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:14 smithi085 conmon[29319]: mgr.y (mgr.14140) 549 : cluster [DBG] pgmap v804: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 365 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 14 op/s 2021-02-10T04:34:14.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:14 smithi085 conmon[35961]: cluster 2021-02-10T04:34:13.228658+0000 mon.a (mon.0) 1231 : cluster [DBG] osdmap e436: 8 total, 8 up, 8 in 2021-02-10T04:34:14.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:14 smithi085 conmon[35961]: audit 2021-02-10T04:34:13.883834+0000 mon.a (mon.0) 1232 : audit [DBG] from='client.? 172.21.15.85:0/295028155' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:34:14.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:14 smithi085 conmon[35961]: cluster 2021-02-10T04:34:14.082588+0000 mgr.y (mgr.14140) 549 : cluster [DBG] pgmap v804: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 365 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 14 op/s 2021-02-10T04:34:14.453 INFO:tasks.rados.rados.0.smithi085.stdout:1663: finishing write tid 1 to smithi08575397-7 2021-02-10T04:34:14.453 INFO:tasks.rados.rados.0.smithi085.stdout:1663: finishing write tid 2 to smithi08575397-7 2021-02-10T04:34:14.453 INFO:tasks.rados.rados.0.smithi085.stdout:1664: seq_num 594 ranges {783358=573440,2026304=580868,3279140=621458} 2021-02-10T04:34:14.461 INFO:tasks.rados.rados.0.smithi085.stdout:1664: writing smithi08575397-10 from 783358 to 1356798 tid 1 2021-02-10T04:34:14.468 INFO:tasks.rados.rados.0.smithi085.stdout:1664: writing smithi08575397-10 from 2026304 to 2607172 tid 2 2021-02-10T04:34:14.471 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:34:14.478 INFO:tasks.rados.rados.0.smithi085.stdout:1664: writing smithi08575397-10 from 3279140 to 3900598 tid 3 2021-02-10T04:34:14.478 INFO:tasks.rados.rados.0.smithi085.stdout:1662: done (12 left) 2021-02-10T04:34:14.478 INFO:tasks.rados.rados.0.smithi085.stdout:1665: write (excl) oid 13 current snap is 178 2021-02-10T04:34:14.478 INFO:tasks.rados.rados.0.smithi085.stdout:1661: finishing rollback tid 1 to smithi08575397-16 2021-02-10T04:34:14.479 INFO:tasks.rados.rados.0.smithi085.stdout:1661: finishing rollback tid 2 to smithi08575397-16 2021-02-10T04:34:14.479 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 893 (ObjNum 523 snap 155 seq_num 523) dirty exists 2021-02-10T04:34:14.479 INFO:tasks.rados.rados.0.smithi085.stdout:1663: finishing write tid 3 to smithi08575397-7 2021-02-10T04:34:14.479 INFO:tasks.rados.rados.0.smithi085.stdout:1663: finishing write tid 4 to smithi08575397-7 2021-02-10T04:34:14.479 INFO:tasks.rados.rados.0.smithi085.stdout:1663: finishing write tid 5 to smithi08575397-7 2021-02-10T04:34:14.479 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 697 (ObjNum 593 snap 178 seq_num 593) dirty exists 2021-02-10T04:34:14.479 INFO:tasks.rados.rados.0.smithi085.stdout:1663: left oid 7 (ObjNum 593 snap 178 seq_num 593) 2021-02-10T04:34:14.479 INFO:tasks.rados.rados.0.smithi085.stdout:1664: finishing write tid 1 to smithi08575397-10 2021-02-10T04:34:14.479 INFO:tasks.rados.rados.0.smithi085.stdout:1664: finishing write tid 2 to smithi08575397-10 2021-02-10T04:34:14.479 INFO:tasks.rados.rados.0.smithi085.stdout:1665: seq_num 595 ranges {400165=648689,1684786=315379} 2021-02-10T04:34:14.485 INFO:tasks.rados.rados.0.smithi085.stdout:1665: writing smithi08575397-13 from 400165 to 1048854 tid 1 2021-02-10T04:34:14.492 INFO:tasks.rados.rados.0.smithi085.stdout:1665: writing smithi08575397-13 from 1684786 to 2000165 tid 2 2021-02-10T04:34:14.492 INFO:tasks.rados.rados.0.smithi085.stdout:1661: done (12 left) 2021-02-10T04:34:14.492 INFO:tasks.rados.rados.0.smithi085.stdout:1663: done (11 left) 2021-02-10T04:34:14.492 INFO:tasks.rados.rados.0.smithi085.stdout:1666: rollback oid 12 current snap is 178 2021-02-10T04:34:14.492 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 12 to 148 2021-02-10T04:34:14.493 INFO:tasks.rados.rados.0.smithi085.stdout:1664: finishing write tid 3 to smithi08575397-10 2021-02-10T04:34:14.493 INFO:tasks.rados.rados.0.smithi085.stdout:1664: finishing write tid 4 to smithi08575397-10 2021-02-10T04:34:14.493 INFO:tasks.rados.rados.0.smithi085.stdout:1664: finishing write tid 5 to smithi08575397-10 2021-02-10T04:34:14.493 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 10 v 628 (ObjNum 594 snap 178 seq_num 594) dirty exists 2021-02-10T04:34:14.493 INFO:tasks.rados.rados.0.smithi085.stdout:1664: left oid 10 (ObjNum 594 snap 178 seq_num 594) 2021-02-10T04:34:14.493 INFO:tasks.rados.rados.0.smithi085.stdout:1664: done (11 left) 2021-02-10T04:34:14.493 INFO:tasks.rados.rados.0.smithi085.stdout:1667: read oid 31 snap -1 2021-02-10T04:34:14.493 INFO:tasks.rados.rados.0.smithi085.stdout:1667: expect (ObjNum 569 snap 175 seq_num 569) 2021-02-10T04:34:14.493 INFO:tasks.rados.rados.0.smithi085.stdout:1668: read oid 36 snap -1 2021-02-10T04:34:14.494 INFO:tasks.rados.rados.0.smithi085.stdout:1668: expect (ObjNum 579 snap 177 seq_num 579) 2021-02-10T04:34:14.494 INFO:tasks.rados.rados.0.smithi085.stdout:1669: snap_remove snap 146 2021-02-10T04:34:14.499 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:34:14.643 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:34:14 smithi164 conmon[44544]: debug 2021-02-10T04:34:14.563+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:34:14.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:14 smithi164 conmon[33677]: debug 2021-02-10T04:34:14.552+0000 7fbec19a1700 -1 osd.4 436 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:14.788 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:34:14.788 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:34:14.789 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:34:14.789 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:34:14.789 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:34:14.789 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:34:14.789 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:34:14.789 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:34:14.789 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:34:14.789 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:34:14.789 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:34:14.789 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:34:14.789 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:34:14.790 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:34:14.790 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:34:14.790 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:34:14.883 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:34:14.942 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:14 smithi085 conmon[49106]: debug 2021-02-10T04:34:14.942 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:14 smithi085 conmon[49106]: 2021-02-10T04:34:14.492+0000 7f77a3e94700 -1 osd.2 437 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:14.942 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:14 smithi085 conmon[49106]: 2021-02-10T04:34:14.988 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:34:15.029 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:34:15.246 INFO:tasks.rados.rados.0.smithi085.stdout:1669: done (13 left) 2021-02-10T04:34:15.247 INFO:tasks.rados.rados.0.smithi085.stdout:1670: read oid 38 snap -1 2021-02-10T04:34:15.247 INFO:tasks.rados.rados.0.smithi085.stdout:1670: expect (ObjNum 574 snap 175 seq_num 574) 2021-02-10T04:34:15.247 INFO:tasks.rados.rados.0.smithi085.stdout:1671: snap_remove snap 167 2021-02-10T04:34:15.251 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is unset 2021-02-10T04:34:15.378 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: cluster 2021-02-10T04:34:15.378 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: 2021-02-10T04:34:15.378 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: 2021 2021-02-10T04:34:15.378 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: - 2021-02-10T04:34:15.378 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: 02 2021-02-10T04:34:15.378 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: - 2021-02-10T04:34:15.378 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: 10 2021-02-10T04:34:15.378 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: T 2021-02-10T04:34:15.379 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: 04 2021-02-10T04:34:15.379 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: : 2021-02-10T04:34:15.379 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: 34 2021-02-10T04:34:15.379 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: : 2021-02-10T04:34:15.379 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: 14 2021-02-10T04:34:15.379 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: . 2021-02-10T04:34:15.379 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: 239602 2021-02-10T04:34:15.379 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: +0000 2021-02-10T04:34:15.379 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: 2021-02-10T04:34:15.379 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: mon.a 2021-02-10T04:34:15.379 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: ( 2021-02-10T04:34:15.379 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: mon 2021-02-10T04:34:15.380 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: .0) 1233 : cluster [DBG] osdmap e437: 8 total, 8 up, 8 in 2021-02-10T04:34:15.380 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:34:15.380 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: 02-10T04:34:14.262138+0000 mon.c (mon.1) 54 : 2021-02-10T04:34:15.380 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: audit [INF] from='client.? 172.21.15.85:0/2799570523' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:34:15.380 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: audit 2021-02-10T04:34 2021-02-10T04:34:15.380 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: :14.262949+0000 mon.a (mon.0) 1234 : 2021-02-10T04:34:15.380 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:34:15.380 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:15.380 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: 04:34:14.552749+0000 osd.4 (osd. 2021-02-10T04:34:15.380 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: 4) 499 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:15.380 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:15.380 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: 02-10T04:34:14.552759+0000 2021-02-10T04:34:15.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: osd.4 (osd.4) 500 : 2021-02-10T04:34:15.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:15.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:34:15.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: :34:14.552766+0000 osd.4 (osd. 2021-02-10T04:34:15.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: 4) 501 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:15.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:34:15.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: :34:14.552772+0000 osd.4 ( 2021-02-10T04:34:15.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: osd.4) 502 : cluster [WRN] 2021-02-10T04:34:15.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:15.381 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[35961]: cluster 2021-02-10T04:34:14.239602+0000 2021-02-10T04:34:15.381 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[35961]: mon.a (mon.0) 1233 : cluster [DBG] osdmap e437: 8 total, 8 up, 8 in 2021-02-10T04:34:15.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[35961]: audit 2021 2021-02-10T04:34:15.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[35961]: -02-10T04:34:14.262138+0000 mon.c (mon.1) 54 : audit [INF] from='client.? 172.21.15.85:0/2799570523' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:34:15.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[35961]: 2021-02-10T04:34:15.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[35961]: audit 2021-02-10T04:34 2021-02-10T04:34:15.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[35961]: :14.262949+0000 mon.a (mon.0) 1234 : audit 2021-02-10T04:34:15.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[35961]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:34:15.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[35961]: cluster 2021-02-10T04:34 2021-02-10T04:34:15.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[35961]: :14.552749+0000 osd.4 (osd.4) 499 : cluster 2021-02-10T04:34:15.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:15.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:34:15.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[35961]: :34:14.552759+0000 osd.4 (osd.4 2021-02-10T04:34:15.383 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[35961]: ) 500 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:15.383 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[35961]: cluster 2021-02-10T04:34:14.552766 2021-02-10T04:34:15.383 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 501 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:15.383 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[35961]: cluster 2021-02-10T04:34: 2021-02-10T04:34:15.383 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[35961]: 14.552772+0000 osd.4 (osd.4) 502 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:15.524 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:15 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:15.524 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:15 smithi164 conmon[29288]: :34:14.239602+0000 mon.a (mon.0) 1233 : cluster [DBG] osdmap e437: 8 total, 8 up, 8 in 2021-02-10T04:34:15.524 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:15 smithi164 conmon[29288]: audit 2021-02-10T04:34:15.524 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:15 smithi164 conmon[29288]: 2021-02-10T04:34: 2021-02-10T04:34:15.524 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:15 smithi164 conmon[29288]: 14.262138+0000 mon.c (mon.1 2021-02-10T04:34:15.524 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:15 smithi164 conmon[29288]: ) 54 : audit [INF] from='client.? 172.21.15.85:0/2799570523' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:34:15.524 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:15 smithi164 conmon[29288]: audit 2021-02-10T04:34:15.525 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:15 smithi164 conmon[29288]: 2021-02-10T04:34:15.525 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:15 smithi164 conmon[29288]: 2021 2021-02-10T04:34:15.525 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:15 smithi164 conmon[29288]: -02-10T04:34: 2021-02-10T04:34:15.525 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:15 smithi164 conmon[29288]: 14.262949+0000 mon.a (mon.0) 1234 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:34:15.525 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:15 smithi164 conmon[29288]: cluster 2021-02-10T04:34:14.552749+0000 osd.4 (osd.4) 499 : 2021-02-10T04:34:15.525 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:15 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:15.525 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:15 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:15.525 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:15 smithi164 conmon[29288]: T04:34:14.552759+0000 osd.4 (osd. 2021-02-10T04:34:15.525 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:15 smithi164 conmon[29288]: 4) 500 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:15.525 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:15 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:15.525 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:15 smithi164 conmon[29288]: -10T04:34:14.552766+0000 osd.4 ( 2021-02-10T04:34:15.525 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:15 smithi164 conmon[29288]: osd.4) 501 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:15.525 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:15 smithi164 conmon[29288]: 2021-02-10T04:34:15.526 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:15 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:15.526 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:15 smithi164 conmon[29288]: 02-10T04:34:14.552772+0000 osd.4 ( 2021-02-10T04:34:15.526 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:15 smithi164 conmon[29288]: osd.4) 502 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:15.526 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:15 smithi164 conmon[29288]: 2021-02-10T04:34:15.526 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:15 smithi164 conmon[33677]: debug 2021-02-10T04:34:15.522+0000 7fbec19a1700 -1 osd.4 438 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:15.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[49106]: debug 2021-02-10T04:34:15.484+0000 7f77a3e94700 -1 osd.2 438 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:16.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:34:15 smithi085 conmon[52836]: debug 2021-02-10T04:34:15.906+0000 7fba75fba700 -1 osd.3 pg_epoch: 438 pg[4.10( v 62'12 (0'0,62'12] local-lis/les=71/72 n=10 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,5,6] r=0 lpr=71 crt=62'12 lcod 62'11 mlcod 0'0 active+clean] scrubber pg(4.10) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6 (0x55fec097cc60) 2021-02-10T04:34:16.255 INFO:tasks.rados.rados.0.smithi085.stdout:1671: done (14 left) 2021-02-10T04:34:16.255 INFO:tasks.rados.rados.0.smithi085.stdout:1672: read oid 44 snap 165 2021-02-10T04:34:16.255 INFO:tasks.rados.rados.0.smithi085.stdout:1672: expect (ObjNum 491 snap 141 seq_num 491) 2021-02-10T04:34:16.255 INFO:tasks.rados.rados.0.smithi085.stdout:1673: rollback oid 6 current snap is 178 2021-02-10T04:34:16.255 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 6 to 147 2021-02-10T04:34:16.255 INFO:tasks.rados.rados.0.smithi085.stdout:1665: finishing write tid 1 to smithi08575397-13 2021-02-10T04:34:16.255 INFO:tasks.rados.rados.0.smithi085.stdout:1666: finishing rollback tid 0 to smithi08575397-12 2021-02-10T04:34:16.255 INFO:tasks.rados.rados.0.smithi085.stdout:1665: finishing write tid 2 to smithi08575397-13 2021-02-10T04:34:16.256 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:34:16.256 INFO:tasks.rados.rados.0.smithi085.stdout:1665: finishing write tid 3 to smithi08575397-13 2021-02-10T04:34:16.256 INFO:tasks.rados.rados.0.smithi085.stdout:1666: finishing rollback tid 1 to smithi08575397-12 2021-02-10T04:34:16.256 INFO:tasks.rados.rados.0.smithi085.stdout:1666: finishing rollback tid 2 to smithi08575397-12 2021-02-10T04:34:16.256 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 754 (ObjNum 85 snap 11 seq_num 85) dirty exists 2021-02-10T04:34:16.256 INFO:tasks.rados.rados.0.smithi085.stdout:1665: finishing write tid 4 to smithi08575397-13 2021-02-10T04:34:16.256 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 656 (ObjNum 595 snap 178 seq_num 595) dirty exists 2021-02-10T04:34:16.256 INFO:tasks.rados.rados.0.smithi085.stdout:1665: left oid 13 (ObjNum 595 snap 178 seq_num 595) 2021-02-10T04:34:16.256 INFO:tasks.rados.rados.0.smithi085.stdout:1668: expect (ObjNum 579 snap 177 seq_num 579) 2021-02-10T04:34:16.466 INFO:tasks.rados.rados.0.smithi085.stdout:1667: expect (ObjNum 569 snap 175 seq_num 569) 2021-02-10T04:34:16.488 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: cluster 2021-02-10T04:34:14.494265+0000 2021-02-10T04:34:16.488 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: osd.2 (osd.2) 757 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:16.488 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: cluster 2021-02-10T04:34:14.494279+0000 osd.2 (osd 2021-02-10T04:34:16.488 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: .2) 758 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:16.488 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: cluster 2021-02-10T04:34:14.494286+0000 osd.2 (osd.2) 759 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:16.488 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: 2021-02-10T04:34:16.489 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: cluster 2021-02-10T04:34:16.489 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: 2021-02-10T04:34:14.494292+0000 osd.2 (osd.2) 760 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:16.489 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:16.489 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: -02-10T04:34:14.494298+0000 osd.2 (osd.2) 761 2021-02-10T04:34:16.489 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:16.489 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: cluster 2021-02-10T04:34 2021-02-10T04:34:16.489 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: :14.494303+0000 osd.2 (osd.2) 2021-02-10T04:34:16.489 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: 762 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:16.489 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:16.489 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: 10T04:34:14.494309+0000 osd.2 (osd.2 2021-02-10T04:34:16.489 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: ) 763 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:16.489 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: cluster 2021-02-10T04:34:16.489 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: 2021-02-10T04:34:14.494314+0000 osd.2 (osd.2 2021-02-10T04:34:16.490 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: ) 764 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:16.490 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:16.490 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: 04:34:14.494320+0000 osd.2 (osd.2) 765 : cluster [WRN] 2021-02-10T04:34:16.490 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:16.490 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: cluster 2021-02-10T04:34: 2021-02-10T04:34:16.490 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: 14.494328+0000 osd.2 (osd.2) 766 2021-02-10T04:34:16.490 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:16.490 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:16.490 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: 02-10T04:34:15.234874+0000 mon.a ( 2021-02-10T04:34:16.490 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: mon.0) 1235 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:34:16.490 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:16.490 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: :34:15.234906+0000 mon.a (mon.0) 1236 : cluster [INF] Cluster is now healthy 2021-02-10T04:34:16.490 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: 2021-02-10T04:34:16.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:34:16.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: T04:34:15.245294+0000 mon.a (mon.0) 1237 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:34:16.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: cluster 2021-02-10T04:34:15.245385+0000 mon.a (mon.0) 1238 : cluster [DBG] osdmap e438: 8 total, 8 up, 8 in 2021-02-10T04:34:16.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: cluster 2021-02-10T04:34:15.523575+0000 osd.4 2021-02-10T04:34:16.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: (osd.4) 503 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:16.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: cluster 2021-02-10T04:34:15.523586 2021-02-10T04:34:16.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 504 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:16.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:16.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: T04:34:15.523593+0000 osd.4 (osd.4) 505 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:16.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: cluster 2021-02-10T04:34:15. 2021-02-10T04:34:16.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: 523601+0000 osd.4 (osd.4) 506 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:16.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:16.491 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[29288]: -02-10T04:34:16.083177+0000 mgr.y (mgr.14140) 550 : cluster [DBG] pgmap v807: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 367 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 29 MiB/s rd, 2.8 MiB/s wr, 33 op/s 2021-02-10T04:34:16.492 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:16 smithi164 conmon[33677]: debug 2021-02-10T04:34:16.487+0000 7fbec19a1700 -1 osd.4 439 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:16.530 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[49106]: debug 2021-02-10T04:34:16.527+0000 7f77a3e94700 -1 osd.2 439 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:16.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: cluster 2021-02-10T04:34:14.494265+0000 2021-02-10T04:34:16.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: osd.2 (osd.2) 757 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:16.530 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: cluster 2021-02-10T04:34:14.494279+0000 osd.2 2021-02-10T04:34:16.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: (osd.2) 758 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:16.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: cluster 2021-02-10T04:34:16.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: 2021-02-10T04:34:14.494286+0000 osd.2 (osd.2) 759 : 2021-02-10T04:34:16.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:16.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: cluster 2021-02-10T04:34:14 2021-02-10T04:34:16.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: .494292 2021-02-10T04:34:16.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 760 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:16.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:16.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: 34:14.494298+0000 osd.2 (osd.2) 761 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:16.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: cluster 2021-02-10T04:34:14.494303+0000 osd.2 (osd.2) 762 : 2021-02-10T04:34:16.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:16.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:34:16.531 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: 04:34:14.494309+0000 osd.2 (osd.2) 2021-02-10T04:34:16.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: 763 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:16.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: cluster 2021-02-10T04:34:14 2021-02-10T04:34:16.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: .494314+0000 osd.2 (osd.2) 764 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:16.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: cluster 2021-02-10T04:34: 2021-02-10T04:34:16.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: 14.494320+0000 osd.2 (osd.2) 765 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:16.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:16.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: 34:14.494328+0000 osd.2 (osd.2) 766 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:16.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: 2021-02-10T04:34:16.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:16.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: -10T04:34:15.234874+0000 mon.a (mon.0) 1235 : cluster [INF] 2021-02-10T04:34:16.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:34:16.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: cluster 2021-02-10T04:34:16.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: 2021-02-10T04:34:15 2021-02-10T04:34:16.532 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: .234906+0000 mon.a (mon.0) 1236 : cluster 2021-02-10T04:34:16.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: [INF] Cluster is now healthy 2021-02-10T04:34:16.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: audit 2021 2021-02-10T04:34:16.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: -02-10T04:34 2021-02-10T04:34:16.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: :15.245294+0000 mon.a ( 2021-02-10T04:34:16.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: mon.0) 1237 : audit 2021-02-10T04:34:16.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:34:16.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: cluster 2021-02-10T04:34:16.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: 2021-02 2021-02-10T04:34:16.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: -10T04:34:15 2021-02-10T04:34:16.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: .245385+0000 mon.a (mon. 2021-02-10T04:34:16.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: 0) 1238 : cluster [DBG] osdmap e438: 8 total, 8 up, 8 in 2021-02-10T04:34:16.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: 2021-02-10T04:34:16.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: cluster 2021-02-10T04:34:16.533 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: 2021-02 2021-02-10T04:34:16.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: -10T04:34:15. 2021-02-10T04:34:16.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: 523575+0000 osd.4 (osd.4) 503 : 2021-02-10T04:34:16.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:16.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: cluster 2021-02-10T04:34:16.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:34:16.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: T04:34:15.523586+0000 osd.4 (osd.4) 504 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:16.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: cluster 2021-02-10T04:34:16.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: 2021-02-10T04:34:15 2021-02-10T04:34:16.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: .523593+0000 osd.4 (osd.4) 2021-02-10T04:34:16.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: 505 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:16.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: cluster 2021-02-10T04:34:16.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: 2021-02-10T04:34:15.523601+0000 2021-02-10T04:34:16.534 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: osd.4 (osd.4) 506 : cluster [WRN] 2021-02-10T04:34:16.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:16.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: cluster 2021-02-10T04:34:16.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: 2021- 2021-02-10T04:34:16.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: 02-10T04:34:16.083177+0000 2021-02-10T04:34:16.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: mgr.y (mgr.14140) 550 : cluster [DBG] 2021-02-10T04:34:16.535 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[35961]: pgmap v807: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 367 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 29 MiB/s rd, 2.8 MiB/s wr, 33 op/s 2021-02-10T04:34:16.535 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: cluster 2021-02-10T04:34:14.494265+0000 osd.2 (osd.2) 2021-02-10T04:34:16.535 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: 757 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:16.535 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: cluster 2021-02-10T04:34:14.494279+0000 osd.2 (osd.2) 758 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:16.536 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: cluster 2021-02-10T04:34:14.494286+0000 osd.2 (osd.2) 759 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:16.536 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:16.536 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: 10T04:34:14.494292+0000 osd.2 (osd. 2021-02-10T04:34:16.536 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: 2) 760 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:16.536 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: cluster 2021-02-10T04:34:14. 2021-02-10T04:34:16.536 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: 494298+0000 osd.2 (osd.2) 761 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:16.536 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: 2021-02-10T04:34:16.536 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:34:16.536 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: T04:34:14.494303+0000 2021-02-10T04:34:16.536 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: osd.2 (osd.2) 762 : 2021-02-10T04:34:16.536 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:16.536 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:16.536 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: -10T04:34:14. 2021-02-10T04:34:16.537 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: 494309+0000 osd.2 (osd.2) 2021-02-10T04:34:16.537 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: 763 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:16.537 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: cluster 2021-02-10T04:34: 2021-02-10T04:34:16.537 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: 14.494314+0000 osd.2 (osd.2) 764 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:16.537 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:34:16.537 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: :34:14.494320+0000 osd.2 (osd.2) 765 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:16.537 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:34:16.537 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: 34:14.494328+0000 osd.2 (osd.2) 766 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:16.537 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:16.537 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: 10T04:34:15.234874+0000 mon.a (mon.0) 1235 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:34:16.537 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: cluster 2021-02-10T04:34:16.537 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: 2021-02-10T04:34:15.234906+0000 mon.a (mon. 2021-02-10T04:34:16.537 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: 0) 1236 : cluster [INF] Cluster is now healthy 2021-02-10T04:34:16.538 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: audit 2021-02-10T 2021-02-10T04:34:16.538 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: 04:34:15.245294+0000 mon.a (mon.0) 1237 : 2021-02-10T04:34:16.538 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:34:16.538 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: cluster 2021-02-10T04:34:15 2021-02-10T04:34:16.538 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: .245385+0000 mon.a (mon.0) 1238 : cluster 2021-02-10T04:34:16.538 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: [DBG] osdmap e438: 8 total, 8 up, 8 in 2021-02-10T04:34:16.538 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:16.538 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: -02-10T04:34:15.523575+0000 osd.4 (osd.4) 2021-02-10T04:34:16.538 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: 503 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:16.538 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: cluster 2021-02-10T04:34:16.538 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: 2021-02-10T04:34:15.523586+0000 2021-02-10T04:34:16.538 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: osd.4 (osd.4) 504 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:16.538 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: 2021-02-10T04:34:16.538 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:16.539 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: 02-10T04:34:15.523593+0000 osd.4 (osd.4) 505 : cluster 2021-02-10T04:34:16.539 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:16.539 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: cluster 2021-02-10T04:34:16.539 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: 2021-02-10T04:34:15.523601+0000 osd.4 (osd.4) 506 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:16.539 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: cluster 2021-02-10T04:34:16 2021-02-10T04:34:16.539 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[29319]: .083177+0000 mgr.y (mgr.14140) 550 : cluster [DBG] pgmap v807: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 367 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 29 MiB/s rd, 2.8 MiB/s wr, 33 op/s 2021-02-10T04:34:16.683 INFO:tasks.rados.rados.0.smithi085.stdout:1665: done (15 left) 2021-02-10T04:34:16.683 INFO:tasks.rados.rados.0.smithi085.stdout:1666: done (14 left) 2021-02-10T04:34:16.683 INFO:tasks.rados.rados.0.smithi085.stdout:1667: done (13 left) 2021-02-10T04:34:16.683 INFO:tasks.rados.rados.0.smithi085.stdout:1668: done (12 left) 2021-02-10T04:34:16.684 INFO:tasks.rados.rados.0.smithi085.stdout:1674: write oid 20 current snap is 178 2021-02-10T04:34:16.684 INFO:tasks.rados.rados.0.smithi085.stdout:1674: seq_num 596 ranges {416972=723938,1743268=273704} 2021-02-10T04:34:16.690 INFO:tasks.rados.rados.0.smithi085.stdout:1674: writing smithi08575397-20 from 416972 to 1140910 tid 1 2021-02-10T04:34:16.696 INFO:tasks.rados.rados.0.smithi085.stdout:1674: writing smithi08575397-20 from 1743268 to 2016972 tid 2 2021-02-10T04:34:16.696 INFO:tasks.rados.rados.0.smithi085.stdout:1675: snap_remove snap 149 2021-02-10T04:34:17.239 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: cluster 2021-02-10T04:34:15.485697+0000 osd.2 (osd.2 2021-02-10T04:34:17.486 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: ) 767 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:17.486 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: cluster 2021-02-10T04:34:15.485719+0000 osd.2 (osd.2) 2021-02-10T04:34:17.486 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: 768 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:17.488 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[52836]: debug 2021-02-10T04:34:17.488 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:34:16 smithi085 conmon[52836]: 2021-02-10T04:34:16.955+0000 7fba75fba700 -1 osd.3 pg_epoch: 439 pg[4.10( v 62'12 (0'0,62'12] local-lis/les=71/72 n=10 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,5,6] r=0 lpr=71 crt=62'12 lcod 62'11 mlcod 0'0 active+clean] scrubber pg(4.10) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6 (0x55fec0838420) 2021-02-10T04:34:17.488 INFO:tasks.rados.rados.0.smithi085.stdout:1675: done (13 left) 2021-02-10T04:34:17.489 INFO:tasks.rados.rados.0.smithi085.stdout:1676: snap_create 2021-02-10T04:34:17.489 INFO:tasks.rados.rados.0.smithi085.stdout:1670: expect (ObjNum 574 snap 175 seq_num 574) 2021-02-10T04:34:17.489 INFO:tasks.rados.rados.0.smithi085.stdout:1673: finishing rollback tid 1 to smithi08575397-6 2021-02-10T04:34:17.489 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 751 (ObjNum 83228544 snap 22030 seq_num 82489152) dirty dne 2021-02-10T04:34:17.489 INFO:tasks.rados.rados.0.smithi085.stdout:1672: expect (ObjNum 491 snap 141 seq_num 491) 2021-02-10T04:34:17.491 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:34:17.574 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: cluster 2021-02-10T04:34:15 2021-02-10T04:34:17.574 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: .485725+0000 osd.2 (osd.2) 769 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:17.574 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: 2021-02-10T04:34:17.574 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:17.575 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: -02-10T04:34:15.485733+0000 osd.2 (osd.2) 770 2021-02-10T04:34:17.575 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:17.575 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:17.575 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: 10T04:34:15.485739+0000 osd.2 (osd. 2021-02-10T04:34:17.575 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: 2) 771 : cluster [WRN] 2021-02-10T04:34:17.575 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:17.575 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:17.576 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: -02-10T04:34: 2021-02-10T04:34:17.576 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: 15.485745+0000 osd.2 (osd 2021-02-10T04:34:17.576 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: .2) 772 : cluster [WRN] 2021-02-10T04:34:17.576 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:17.576 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:17.576 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: 10T04:34:15.485753+0000 2021-02-10T04:34:17.576 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: osd.2 (osd.2) 773 : cluster [WRN] 2021-02-10T04:34:17.576 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:17.577 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: cluster 2021-02-10T04:34:17.577 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: 2021-02-10T04:34:15.485759+0000 2021-02-10T04:34:17.577 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: osd.2 (osd.2) 774 : 2021-02-10T04:34:17.577 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:17.577 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: cluster 2021-02-10T04:34: 2021-02-10T04:34:17.577 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: 15.485765+0000 osd.2 (osd.2) 775 : cluster 2021-02-10T04:34:17.577 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:17.578 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:17.578 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: -10T04:34:15.485772+0000 osd.2 (osd 2021-02-10T04:34:17.578 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: .2) 776 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:17.578 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:17.578 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: 04:34:16.241600+0000 mon.a (mon.0) 1239 : cluster [WRN] 2021-02-10T04:34:17.578 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: Health check failed: 10 slow ops, oldest one blocked for 263 sec, osd.2 has slow ops (SLOW_OPS) 2021-02-10T04:34:17.579 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:34:17.579 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: T04:34:16.254657+0000 mon.a (mon.0) 1240 : 2021-02-10T04:34:17.579 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: cluster [DBG] osdmap e439: 8 total, 8 up, 8 in 2021-02-10T04:34:17.579 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: cluster 2021-02-10T04:34:17.579 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:34:17.579 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: 34:16.487984+0000 osd.4 ( 2021-02-10T04:34:17.579 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: osd.4) 507 : cluster [WRN] 2021-02-10T04:34:17.579 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:17.580 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:34:17.580 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: :34:16.487992+0000 osd.4 (osd.4) 508 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:17.580 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: cluster 2021-02-10T04:34:16.487996+0000 osd.4 (osd.4) 509 : cluster [WRN] 2021-02-10T04:34:17.580 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:17.580 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: cluster 2021-02-10T04:34:16.487999+0000 2021-02-10T04:34:17.580 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[29319]: osd.4 (osd.4) 510 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:17.581 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[45443]: debug 2021-02-10T04:34:17.322+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:34:17.581 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: cluster 2021-02-10T04:34:15.485697+0000 2021-02-10T04:34:17.581 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: osd.2 (osd.2) 767 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:17.581 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: cluster 2021-02-10T04:34:17.581 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: 2021-02-10T04:34:15.485719+0000 osd.2 ( 2021-02-10T04:34:17.581 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: osd.2) 768 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:17.581 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:17.582 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: -10T04:34:15.485725+0000 2021-02-10T04:34:17.582 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: osd.2 (osd.2) 769 : cluster [WRN] 2021-02-10T04:34:17.582 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:17.582 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:17.582 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: -02-10T04:34:15.485733+0000 osd.2 2021-02-10T04:34:17.582 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: (osd.2) 770 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:17.582 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:17.583 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: -02-10T04:34:15.485739+0000 osd.2 (osd.2) 771 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:17.583 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: 2021-02-10T04:34:17.583 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: cluster 2021-02-10T04:34:15.485745 2021-02-10T04:34:17.583 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 772 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:17.583 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:34:17.583 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: :34:15.485753+0000 osd.2 (osd.2) 773 : cluster [WRN] 2021-02-10T04:34:17.583 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:17.584 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:34:17.584 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: 04:34:15.485759+0000 osd.2 (osd.2) 774 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:17.584 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: cluster 2021-02-10T04:34 2021-02-10T04:34:17.584 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: :15.485765+0000 osd.2 (osd.2) 775 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:17.584 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: cluster 2021-02-10T04:34:15.485772 2021-02-10T04:34:17.584 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 776 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:17.584 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:17.585 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: T04:34:16.241600+0000 mon.a (mon.0) 1239 : cluster [WRN] 2021-02-10T04:34:17.585 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: Health check failed: 10 slow ops, oldest one blocked for 263 sec, osd.2 has slow ops (SLOW_OPS) 2021-02-10T04:34:17.585 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:34:17.585 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: 04:34:16.254657+0000 mon.a (mon.0) 1240 : cluster [DBG] osdmap e439: 8 total, 8 up, 8 in 2021-02-10T04:34:17.585 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:17.585 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: 02-10T04:34:16.487984+0000 osd.4 (osd.4) 507 : 2021-02-10T04:34:17.585 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:17.585 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: cluster 2021-02-10T04:34:16.487992+0000 osd.4 (osd.4 2021-02-10T04:34:17.586 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: ) 508 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:17.586 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:17.586 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: T04:34:16.487996+0000 osd.4 (osd.4) 509 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:17.586 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:17.586 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: T04:34:16.487999+0000 osd.4 (osd.4) 510 : cluster 2021-02-10T04:34:17.586 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:17.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: cluster 2021-02-10T04:34:17.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: 2021-02-10T04:34:17.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:34:17.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: :34:15.485697+0000 osd.2 (osd.2) 767 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:17.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:17.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: 02-10T04:34:15.485719+0000 osd.2 (osd.2 2021-02-10T04:34:17.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: ) 768 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:17.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:17.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: 10T04:34:15.485725+0000 osd.2 (osd 2021-02-10T04:34:17.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: .2) 769 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:17.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:17.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: -10T04:34:15.485733+0000 osd.2 ( 2021-02-10T04:34:17.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: osd.2) 770 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:17.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:17.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: -10T04:34:15.485739+0000 osd.2 ( 2021-02-10T04:34:17.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: osd.2) 771 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:17.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:17.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: 02-10T04:34:15.485745+0000 osd.2 2021-02-10T04:34:17.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: (osd.2) 772 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:17.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: 2021-02-10T04:34:17.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:17.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: -02-10T04:34:15.485753+0000 osd.2 (osd.2 2021-02-10T04:34:17.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: ) 773 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:17.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:17.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: 04:34:15.485759+0000 osd.2 (osd. 2021-02-10T04:34:17.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: 2) 774 : cluster 2021-02-10T04:34:17.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:17.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: cluster 2021-02-10T04:34: 2021-02-10T04:34:17.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: 15.485765+0000 osd.2 (osd.2) 775 : cluster 2021-02-10T04:34:17.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:17.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: cluster 2021-02-10T04:34: 2021-02-10T04:34:17.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: 15.485772+0000 osd.2 (osd.2) 776 : 2021-02-10T04:34:17.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:17.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:17.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: -10T04:34:16.241600+0000 mon.a ( 2021-02-10T04:34:17.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: mon.0) 1239 : cluster [WRN] Health check failed: 10 slow ops, oldest one blocked for 263 sec, osd.2 has slow ops (SLOW_OPS) 2021-02-10T04:34:17.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:17.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: 02-10T04:34:16.254657+0000 mon.a 2021-02-10T04:34:17.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: (mon.0) 1240 : cluster [DBG] osdmap e439: 8 total, 8 up, 8 in 2021-02-10T04:34:17.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: 2021-02-10T04:34:17.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: cluster 2021-02-10T04:34:17.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: 2021-02-10T04:34:16.487984+0000 osd.4 (osd.4) 507 2021-02-10T04:34:17.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:17.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: cluster 2021-02-10T04:34:16.487992+0000 osd.4 (osd.4) 508 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:17.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:17.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: -10T04:34:16.487996+0000 osd.4 (osd.4 2021-02-10T04:34:17.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: ) 509 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:17.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: cluster 2021-02-10T04:34:16.487999+0000 2021-02-10T04:34:17.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: osd.4 (osd.4) 510 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:17.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[29288]: 2021-02-10T04:34:17.652 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:17 smithi164 conmon[33677]: debug 2021-02-10T04:34:17.439+0000 7fbec19a1700 -1 osd.4 440 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:17.653 INFO:tasks.rados.rados.0.smithi085.stdout:1674: finishing write tid 1 to smithi08575397-20 2021-02-10T04:34:17.653 INFO:tasks.rados.rados.0.smithi085.stdout:1674: finishing write tid 2 to smithi08575397-20 2021-02-10T04:34:17.653 INFO:tasks.rados.rados.0.smithi085.stdout:1674: finishing write tid 3 to smithi08575397-20 2021-02-10T04:34:17.654 INFO:tasks.rados.rados.0.smithi085.stdout:1674: finishing write tid 4 to smithi08575397-20 2021-02-10T04:34:17.654 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 20 v 726 (ObjNum 596 snap 178 seq_num 596) dirty exists 2021-02-10T04:34:17.654 INFO:tasks.rados.rados.0.smithi085.stdout:1674: left oid 20 (ObjNum 596 snap 178 seq_num 596) 2021-02-10T04:34:17.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[49106]: debug 2021-02-10T04:34:17.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[49106]: 2021-02-10T04:34:17.572+0000 7f77a3e94700 -1 osd.2 440 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:17.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:17 smithi085 conmon[49106]: 2021-02-10T04:34:18.139 INFO:tasks.rados.rados.0.smithi085.stdout:1670: done (13 left) 2021-02-10T04:34:18.139 INFO:tasks.rados.rados.0.smithi085.stdout:1672: done (12 left) 2021-02-10T04:34:18.139 INFO:tasks.rados.rados.0.smithi085.stdout:1673: done (11 left) 2021-02-10T04:34:18.139 INFO:tasks.rados.rados.0.smithi085.stdout:1674: done (10 left) 2021-02-10T04:34:18.139 INFO:tasks.rados.rados.0.smithi085.stdout:1676: done (9 left) 2021-02-10T04:34:18.139 INFO:tasks.rados.rados.0.smithi085.stdout:1677: rollback oid 23 current snap is 179 2021-02-10T04:34:18.139 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 23 to 159 2021-02-10T04:34:18.139 INFO:tasks.rados.rados.0.smithi085.stdout:1678: snap_create 2021-02-10T04:34:18.147 INFO:tasks.rados.rados.0.smithi085.stdout:1677: finishing rollback tid 1 to smithi08575397-23 2021-02-10T04:34:18.148 INFO:tasks.rados.rados.0.smithi085.stdout:1677: finishing rollback tid 2 to smithi08575397-23 2021-02-10T04:34:18.148 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 653 (ObjNum 85 snap 11 seq_num 85) dirty exists 2021-02-10T04:34:18.393 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:34:18.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: cluster 2021-02-10T04:34:16.529426 2021-02-10T04:34:18.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 777 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:18.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: cluster 2021-02-10T04:34:16.529435+0000 osd.2 (osd.2) 778 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:18.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: 2021-02-10T04:34:18.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: cluster 2021-02-10T04:34:16.529449+0000 osd.2 (osd.2) 779 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:18.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: cluster 2021-02-10T04:34:16.529455 2021-02-10T04:34:18.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 780 : cluster [WRN] 2021-02-10T04:34:18.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:18.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: cluster 2021-02-10T04:34:16. 2021-02-10T04:34:18.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: 529458+0000 osd.2 (osd.2) 781 : cluster 2021-02-10T04:34:18.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:18.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: cluster 2021-02-10T04:34:16.529461+0000 osd.2 (osd 2021-02-10T04:34:18.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: .2) 782 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:18.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:18.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: -10T04:34:16.529466+0000 osd.2 ( 2021-02-10T04:34:18.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: osd.2) 783 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:18.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:18.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: 02-10T04:34:16.529469+0000 2021-02-10T04:34:18.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: osd.2 (osd.2) 784 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:18.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:18.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: -02-10T04:34:16.529472 2021-02-10T04:34:18.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 785 : cluster [WRN] 2021-02-10T04:34:18.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:18.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: cluster 2021-02-10T04:34: 2021-02-10T04:34:18.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: 16.529475+0000 osd.2 (osd.2) 786 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:18.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: 2021-02-10T04:34:18.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: cluster 2021-02-10T04:34:18.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: 2021-02-10T04:34:17.257486+0000 2021-02-10T04:34:18.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: mon.a (mon.0) 1241 : cluster [DBG] 2021-02-10T04:34:18.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: osdmap e440: 8 total, 8 up, 8 in 2021-02-10T04:34:18.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:18.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: 04:34:17.440648+0000 osd.4 (osd 2021-02-10T04:34:18.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: .4) 511 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:18.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:18.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: -02-10T04:34:17.440662+0000 osd.4 (osd.4 2021-02-10T04:34:18.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: ) 512 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:18.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:18.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: 02-10T04:34:17.440667+0000 osd.4 (osd.4) 513 : cluster [WRN] 2021-02-10T04:34:18.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:18.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: cluster 2021-02-10T04:34:18.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: 2021-02-10T04:34:17.440671+0000 osd.4 ( 2021-02-10T04:34:18.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: osd.4) 514 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:18.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: audit 2021 2021-02-10T04:34:18.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: -02-10T04:34:18.015055+0000 mgr.y 2021-02-10T04:34:18.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: (mgr.14140) 551 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:34:18.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:18.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: -10T04:34:18.083693+0000 mgr.y ( 2021-02-10T04:34:18.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: mgr.14140) 552 : cluster [DBG] pgmap v810: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 367 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 29 MiB/s rd, 2.0 MiB/s wr, 29 op/s 2021-02-10T04:34:18.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:18.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: 10T04:34:18.136924+0000 mon.a ( 2021-02-10T04:34:18.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[29288]: mon.0) 1242 : cluster [DBG] osdmap e441: 8 total, 8 up, 8 in 2021-02-10T04:34:18.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: cluster 2021-02-10T04:34:18.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: 2021-02-10T04:34:18.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: 2021 2021-02-10T04:34:18.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: - 2021-02-10T04:34:18.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: 02 2021-02-10T04:34:18.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: - 2021-02-10T04:34:18.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: 10 2021-02-10T04:34:18.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: T 2021-02-10T04:34:18.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: 04 2021-02-10T04:34:18.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: : 2021-02-10T04:34:18.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: 34:16.529426+0000 osd.2 (osd 2021-02-10T04:34:18.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: .2) 777 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:18.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:18.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: -02-10T04:34: 2021-02-10T04:34:18.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: 16.529435+0000 osd.2 (osd 2021-02-10T04:34:18.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: .2) 778 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:18.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: cluster 2021-02-10T04:34:18.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: 2021-02-10T04:34:16. 2021-02-10T04:34:18.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: 529449+0000 osd.2 (osd.2) 779 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:18.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:18.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: 10T04:34:16.529455+0000 2021-02-10T04:34:18.616 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: osd.2 (osd.2) 780 : cluster [WRN] 2021-02-10T04:34:18.616 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:18.616 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:34:18.616 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: T04:34:16.529458+0000 2021-02-10T04:34:18.616 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: osd.2 (osd.2) 781 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:18.616 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:34:18.616 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: T04:34:16.529461+0000 osd.2 (osd.2) 782 : cluster 2021-02-10T04:34:18.616 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:18.616 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: cluster 2021-02-10T04:34:18.616 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: 2021-02-10T04:34:16.529466+0000 osd.2 (osd.2) 783 : cluster [WRN] 2021-02-10T04:34:18.616 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:18.616 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:18.616 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: -10T04:34:16.529469+0000 osd.2 (osd.2) 784 : cluster 2021-02-10T04:34:18.617 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:18.617 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: cluster 2021-02-10T04:34:18.617 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: 2021-02-10T04:34:16.529472+0000 osd.2 (osd. 2021-02-10T04:34:18.617 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: 2) 785 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:18.617 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:34:18.617 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: 34:16.529475+0000 osd.2 (osd.2) 786 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:18.617 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: 2021-02-10T04:34:18.617 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: cluster 2021-02-10T04:34:17.257486 2021-02-10T04:34:18.617 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1241 : cluster [DBG] osdmap e440: 8 total, 8 up, 8 in 2021-02-10T04:34:18.617 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: cluster 2021-02-10T04:34:17.440648+0000 osd.4 (osd. 2021-02-10T04:34:18.617 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: 4) 511 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:18.617 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:18.617 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: 10T04:34:17.440662+0000 osd.4 (osd.4) 512 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:18.617 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: cluster 2021-02-10T04:34:17.440667+0000 2021-02-10T04:34:18.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: osd.4 (osd.4) 513 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:18.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: cluster 2021-02-10T04:34:17.440671+0000 osd.4 (osd. 2021-02-10T04:34:18.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: 4) 514 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:18.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: audit 2021-02-10T04:34:18.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: 2021-02-10T04:34:18.015055+0000 mgr.y (mgr.14140) 551 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:34:18.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: cluster 2021-02-10T04:34:18.083693+0000 2021-02-10T04:34:18.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: mgr.y (mgr.14140) 552 : cluster [DBG] pgmap v810: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 367 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 29 MiB/s rd, 2.0 MiB/s wr, 29 op/s 2021-02-10T04:34:18.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: cluster 2021-02-10T04:34:18.136924+0000 mon.a (mon. 2021-02-10T04:34:18.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[29319]: 0) 1242 : cluster [DBG] osdmap e441: 8 total, 8 up, 8 in 2021-02-10T04:34:18.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: cluster 2021-02-10T04:34:18.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: 2021-02-10T04:34:18.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: 2021 2021-02-10T04:34:18.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: - 2021-02-10T04:34:18.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: 02 2021-02-10T04:34:18.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: - 2021-02-10T04:34:18.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: 10 2021-02-10T04:34:18.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: T 2021-02-10T04:34:18.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: 04 2021-02-10T04:34:18.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: : 2021-02-10T04:34:18.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: 34 2021-02-10T04:34:18.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: : 2021-02-10T04:34:18.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: 16 2021-02-10T04:34:18.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: . 2021-02-10T04:34:18.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: 529426+0000 osd.2 (osd.2) 777 : cluster [WRN] 2021-02-10T04:34:18.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:18.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: cluster 2021-02-10T04:34:18.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: 2021-02 2021-02-10T04:34:18.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: -10T04:34:16 2021-02-10T04:34:18.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: . 2021-02-10T04:34:18.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: 529435+0000 osd.2 (osd.2 2021-02-10T04:34:18.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: ) 778 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:18.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:18.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: -10T04:34:16.529449+0000 osd.2 ( 2021-02-10T04:34:18.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: osd.2) 779 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:18.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:18.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: 02-10T04:34:16.529455+0000 osd.2 (osd.2) 780 : cluster [WRN] 2021-02-10T04:34:18.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:18.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:34:18.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: :34:16.529458+0000 osd.2 (osd.2) 2021-02-10T04:34:18.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: 781 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:18.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: cluster 2021-02-10T04:34:18.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:34:18.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: :34:16.529461+0000 osd.2 (osd.2 2021-02-10T04:34:18.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: ) 782 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:18.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:18.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: -02-10T04:34:16.529466+0000 osd.2 (osd 2021-02-10T04:34:18.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: .2) 783 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:18.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: cluster 2021-02-10T04:34:18.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:34:18.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: 34:16.529469+0000 osd.2 (osd.2) 2021-02-10T04:34:18.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: 784 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:18.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:18.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: -02-10T04:34:16.529472+0000 2021-02-10T04:34:18.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: osd.2 (osd.2) 785 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:18.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: 2021-02-10T04:34:18.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: cluster 2021-02-10T04:34:18.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: 2021-02-10T04:34:16.529475+0000 osd.2 2021-02-10T04:34:18.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: (osd.2) 786 : cluster 2021-02-10T04:34:18.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:18.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:18.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: -10T04:34:17.257486+0000 mon.a (mon 2021-02-10T04:34:18.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: .0) 1241 : cluster [DBG] 2021-02-10T04:34:18.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: osdmap e440: 8 total, 8 up, 8 in 2021-02-10T04:34:18.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:18.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: -02-10T04:34 2021-02-10T04:34:18.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: :17.440648+0000 osd.4 (osd 2021-02-10T04:34:18.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: .4) 511 : cluster [WRN] 2021-02-10T04:34:18.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:18.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:18.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: 10T04:34:17.440662+0000 osd.4 ( 2021-02-10T04:34:18.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: osd.4) 512 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:18.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:18.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: T04:34:17.440667+0000 osd.4 (osd. 2021-02-10T04:34:18.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: 4) 513 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:18.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:18.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: T04:34:17.440671+0000 osd.4 (osd 2021-02-10T04:34:18.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: .4) 514 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:18.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: audit 2021-02-10T04:34:18.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: 2021-02-10T04:34:18.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: 2021 2021-02-10T04:34:18.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: -02-10T04:34 2021-02-10T04:34:18.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: :18.015055+0000 mgr.y (mgr.14140) 551 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:34:18.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:18.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: -10T04:34:18.083693+0000 mgr.y (mgr.14140 2021-02-10T04:34:18.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: ) 552 : cluster [DBG] pgmap v810: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 367 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 29 MiB/s rd, 2.0 MiB/s wr, 29 op/s 2021-02-10T04:34:18.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:18.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: -10T04:34:18.136924+0000 mon.a (mon 2021-02-10T04:34:18.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: .0 2021-02-10T04:34:18.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: ) 1242 : cluster [DBG] osdmap e441: 8 total, 8 up, 8 in 2021-02-10T04:34:18.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[35961]: 2021-02-10T04:34:18.895 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:18 smithi164 conmon[33677]: debug 2021-02-10T04:34:18.476+0000 7fbec19a1700 -1 osd.4 441 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:22.356 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:18 smithi085 conmon[49106]: debug 2021-02-10T04:34:18.607+0000 7f77a3e94700 -1 osd.2 441 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:22.357 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[49106]: debug 2021-02-10T04:34:19.622+0000 7f77a3e94700 -1 osd.2 442 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:22.357 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[49106]: debug 2021-02-10T04:34:20.613+0000 7f77a3e94700 -1 osd.2 443 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:22.357 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[49106]: debug 2021-02-10T04:34:21.621+0000 7f77a3e94700 -1 osd.2 444 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:22.357 INFO:tasks.rados.rados.0.smithi085.stdout:1677: done (10 left) 2021-02-10T04:34:22.357 INFO:tasks.rados.rados.0.smithi085.stdout:1678: done (9 left) 2021-02-10T04:34:22.357 INFO:tasks.rados.rados.0.smithi085.stdout:1679: rollback oid 2 current snap is 180 2021-02-10T04:34:22.357 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 2 to 177 2021-02-10T04:34:22.357 INFO:tasks.rados.rados.0.smithi085.stdout:1680: read oid 13 snap -1 2021-02-10T04:34:22.357 INFO:tasks.rados.rados.0.smithi085.stdout:1680: expect (ObjNum 595 snap 178 seq_num 595) 2021-02-10T04:34:22.357 INFO:tasks.rados.rados.0.smithi085.stdout:1681: rollback oid 50 current snap is 180 2021-02-10T04:34:22.358 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 50 to 178 2021-02-10T04:34:22.358 INFO:tasks.rados.rados.0.smithi085.stdout:1682: copy_from oid 1 from oid 42 current snap is 180 2021-02-10T04:34:22.358 INFO:tasks.rados.rados.0.smithi085.stdout:1683: write oid 16 current snap is 180 2021-02-10T04:34:22.358 INFO:tasks.rados.rados.0.smithi085.stdout:1683: seq_num 597 ranges {433779=799187,1801750=232029} 2021-02-10T04:34:22.358 INFO:tasks.rados.rados.0.smithi085.stdout:1683: writing smithi08575397-16 from 433779 to 1232966 tid 1 2021-02-10T04:34:22.358 INFO:tasks.rados.rados.0.smithi085.stdout:1683: writing smithi08575397-16 from 1801750 to 2033779 tid 2 2021-02-10T04:34:22.358 INFO:tasks.rados.rados.0.smithi085.stdout:1684: write oid 7 current snap is 180 2021-02-10T04:34:22.358 INFO:tasks.rados.rados.0.smithi085.stdout:1684: seq_num 598 ranges {450586=474436,1343879=588206} 2021-02-10T04:34:22.358 INFO:tasks.rados.rados.0.smithi085.stdout:1684: writing smithi08575397-7 from 450586 to 925022 tid 1 2021-02-10T04:34:22.358 INFO:tasks.rados.rados.0.smithi085.stdout:1684: writing smithi08575397-7 from 1343879 to 1932085 tid 2 2021-02-10T04:34:22.358 INFO:tasks.rados.rados.0.smithi085.stdout:1685: snap_remove snap 142 2021-02-10T04:34:22.358 INFO:tasks.rados.rados.0.smithi085.stdout:1680: expect (ObjNum 595 snap 178 seq_num 595) 2021-02-10T04:34:22.358 INFO:tasks.rados.rados.0.smithi085.stdout:1680: done (15 left) 2021-02-10T04:34:22.359 INFO:tasks.rados.rados.0.smithi085.stdout:1685: done (14 left) 2021-02-10T04:34:22.359 INFO:tasks.rados.rados.0.smithi085.stdout:1686: delete oid 26 current snap is 180 2021-02-10T04:34:22.359 INFO:tasks.rados.rados.0.smithi085.stdout:1681: finishing rollback tid 1 to smithi08575397-50 2021-02-10T04:34:22.359 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 634 (ObjNum 0 snap 0 seq_num 82824528) dirty dne 2021-02-10T04:34:22.359 INFO:tasks.rados.rados.0.smithi085.stdout:1679: finishing rollback tid 0 to smithi08575397-2 2021-02-10T04:34:22.359 INFO:tasks.rados.rados.0.smithi085.stdout:1682: finishing copy_from to smithi08575397-1 2021-02-10T04:34:22.359 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 727 (ObjNum 540 snap 167 seq_num 540) dirty exists 2021-02-10T04:34:22.359 INFO:tasks.rados.rados.0.smithi085.stdout:1684: finishing write tid 1 to smithi08575397-7 2021-02-10T04:34:22.359 INFO:tasks.rados.rados.0.smithi085.stdout:1679: finishing rollback tid 1 to smithi08575397-2 2021-02-10T04:34:22.359 INFO:tasks.rados.rados.0.smithi085.stdout:1679: finishing rollback tid 2 to smithi08575397-2 2021-02-10T04:34:22.359 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 631 (ObjNum 422 snap 109 seq_num 422) dirty exists 2021-02-10T04:34:22.359 INFO:tasks.rados.rados.0.smithi085.stdout:1683: finishing write tid 1 to smithi08575397-16 2021-02-10T04:34:22.359 INFO:tasks.rados.rados.0.smithi085.stdout:1682: finishing copy_from racing read to smithi08575397-1 2021-02-10T04:34:22.359 INFO:tasks.rados.rados.0.smithi085.stdout:1683: finishing write tid 2 to smithi08575397-16 2021-02-10T04:34:22.360 INFO:tasks.rados.rados.0.smithi085.stdout:1683: finishing write tid 3 to smithi08575397-16 2021-02-10T04:34:22.360 INFO:tasks.rados.rados.0.smithi085.stdout:1683: finishing write tid 4 to smithi08575397-16 2021-02-10T04:34:22.360 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 903 (ObjNum 597 snap 180 seq_num 597) dirty exists 2021-02-10T04:34:22.360 INFO:tasks.rados.rados.0.smithi085.stdout:1683: left oid 16 (ObjNum 597 snap 180 seq_num 597) 2021-02-10T04:34:22.360 INFO:tasks.rados.rados.0.smithi085.stdout:1684: finishing write tid 2 to smithi08575397-7 2021-02-10T04:34:22.360 INFO:tasks.rados.rados.0.smithi085.stdout:1684: finishing write tid 3 to smithi08575397-7 2021-02-10T04:34:22.360 INFO:tasks.rados.rados.0.smithi085.stdout:1684: finishing write tid 4 to smithi08575397-7 2021-02-10T04:34:22.360 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 700 (ObjNum 598 snap 180 seq_num 598) dirty exists 2021-02-10T04:34:22.360 INFO:tasks.rados.rados.0.smithi085.stdout:1684: left oid 7 (ObjNum 598 snap 180 seq_num 598) 2021-02-10T04:34:22.360 INFO:tasks.rados.rados.0.smithi085.stdout:1679: done (14 left) 2021-02-10T04:34:22.360 INFO:tasks.rados.rados.0.smithi085.stdout:1681: done (13 left) 2021-02-10T04:34:22.360 INFO:tasks.rados.rados.0.smithi085.stdout:1682: done (12 left) 2021-02-10T04:34:22.360 INFO:tasks.rados.rados.0.smithi085.stdout:1683: done (11 left) 2021-02-10T04:34:22.360 INFO:tasks.rados.rados.0.smithi085.stdout:1684: done (10 left) 2021-02-10T04:34:22.361 INFO:tasks.rados.rados.0.smithi085.stdout:1686: done (9 left) 2021-02-10T04:34:22.361 INFO:tasks.rados.rados.0.smithi085.stdout:1687: write oid 21 current snap is 180 2021-02-10T04:34:22.361 INFO:tasks.rados.rados.0.smithi085.stdout:1687: seq_num 600 ranges {484200=624934,1860843=223357} 2021-02-10T04:34:22.361 INFO:tasks.rados.rados.0.smithi085.stdout:1687: writing smithi08575397-21 from 484200 to 1109134 tid 1 2021-02-10T04:34:22.361 INFO:tasks.rados.rados.0.smithi085.stdout:1687: writing smithi08575397-21 from 1860843 to 2084200 tid 2 2021-02-10T04:34:22.361 INFO:tasks.rados.rados.0.smithi085.stdout:1688: delete oid 1 current snap is 180 2021-02-10T04:34:22.361 INFO:tasks.rados.rados.0.smithi085.stdout:1687: finishing write tid 1 to smithi08575397-21 2021-02-10T04:34:22.361 INFO:tasks.rados.rados.0.smithi085.stdout:1688: done (10 left) 2021-02-10T04:34:22.361 INFO:tasks.rados.rados.0.smithi085.stdout:1689: snap_remove snap 178 2021-02-10T04:34:22.361 INFO:tasks.rados.rados.0.smithi085.stdout:1689: done (10 left) 2021-02-10T04:34:22.361 INFO:tasks.rados.rados.0.smithi085.stdout:1690: read oid 22 snap -1 2021-02-10T04:34:22.361 INFO:tasks.rados.rados.0.smithi085.stdout:1690: expect (ObjNum 570 snap 175 seq_num 570) 2021-02-10T04:34:22.361 INFO:tasks.rados.rados.0.smithi085.stdout:1691: write oid 2 current snap is 180 2021-02-10T04:34:22.362 INFO:tasks.rados.rados.0.smithi085.stdout:1691: seq_num 602 ranges {517814=491785,1577807=540007} 2021-02-10T04:34:22.362 INFO:tasks.rados.rados.0.smithi085.stdout:1691: writing smithi08575397-2 from 517814 to 1009599 tid 1 2021-02-10T04:34:22.362 INFO:tasks.rados.rados.0.smithi085.stdout:1691: writing smithi08575397-2 from 1577807 to 2117814 tid 2 2021-02-10T04:34:22.362 INFO:tasks.rados.rados.0.smithi085.stdout:1692: snap_remove snap 172 2021-02-10T04:34:22.362 INFO:tasks.rados.rados.0.smithi085.stdout:1692: done (12 left) 2021-02-10T04:34:22.362 INFO:tasks.rados.rados.0.smithi085.stdout:1693: snap_create 2021-02-10T04:34:22.362 INFO:tasks.rados.rados.0.smithi085.stdout:1687: finishing write tid 2 to smithi08575397-21 2021-02-10T04:34:22.362 INFO:tasks.rados.rados.0.smithi085.stdout:1687: finishing write tid 3 to smithi08575397-21 2021-02-10T04:34:22.362 INFO:tasks.rados.rados.0.smithi085.stdout:1687: finishing write tid 4 to smithi08575397-21 2021-02-10T04:34:22.362 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 907 (ObjNum 600 snap 180 seq_num 600) dirty exists 2021-02-10T04:34:22.362 INFO:tasks.rados.rados.0.smithi085.stdout:1687: left oid 21 (ObjNum 600 snap 180 seq_num 600) 2021-02-10T04:34:22.362 INFO:tasks.rados.rados.0.smithi085.stdout:1690: expect (ObjNum 570 snap 175 seq_num 570) 2021-02-10T04:34:22.363 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: cluster 2021-02-10T04:34:22.363 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 2021- 2021-02-10T04:34:22.363 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 02-10T04:34:17 2021-02-10T04:34:22.363 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: .572603+0000 osd.2 2021-02-10T04:34:22.363 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: (osd.2) 787 : cluster [WRN] 2021-02-10T04:34:22.363 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:22.363 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: cluster 2021-02-10T04:34:22.363 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 2021-02-10T04:34: 2021-02-10T04:34:22.363 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 17.572612+0000 osd.2 (osd.2) 2021-02-10T04:34:22.363 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 788 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:22.363 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: cluster 2021-02-10T04:34:22.363 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 2021-02-10T04:34:22.363 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 2021- 2021-02-10T04:34:22.364 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 02-10T04:34 2021-02-10T04:34:22.364 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: :17.572620+0000 osd.2 ( 2021-02-10T04:34:22.364 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: osd.2) 789 : cluster 2021-02-10T04:34:22.364 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:22.364 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:22.364 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: -02-10 2021-02-10T04:34:22.364 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: T04: 2021-02-10T04:34:22.364 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 34:17.572627+0000 osd.2 2021-02-10T04:34:22.364 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: (osd.2) 790 : cluster 2021-02-10T04:34:22.364 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:22.364 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: cluster 2021-02-10T04:34:22.364 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 2021-02-10T04:34:22.364 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 2021 2021-02-10T04:34:22.364 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: - 2021-02-10T04:34:22.365 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 02 2021-02-10T04:34:22.365 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: - 2021-02-10T04:34:22.365 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 10 2021-02-10T04:34:22.365 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: T 2021-02-10T04:34:22.365 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 04 2021-02-10T04:34:22.365 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: : 2021-02-10T04:34:22.365 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 34 2021-02-10T04:34:22.365 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: : 2021-02-10T04:34:22.365 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 17 2021-02-10T04:34:22.365 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: . 2021-02-10T04:34:22.365 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 572632 2021-02-10T04:34:22.365 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: +0000 2021-02-10T04:34:22.365 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 2021-02-10T04:34:22.366 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: osd.2 2021-02-10T04:34:22.366 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: (osd.2) 791 : cluster [WRN] 2021-02-10T04:34:22.366 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:22.366 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:22.366 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: -02-10T04:34:17.572638+0000 osd.2 (osd.2) 792 : 2021-02-10T04:34:22.366 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:22.366 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: cluster 2021-02-10T04:34:22.366 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 2021-02-10T04:34:17.572645 2021-02-10T04:34:22.366 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 793 : cluster [WRN] 2021-02-10T04:34:22.366 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:22.366 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:34:22.366 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: :34:17.572650+0000 osd.2 (osd.2 2021-02-10T04:34:22.366 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: ) 794 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:22.367 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:22.367 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 02-10T04:34:17.572669+0000 2021-02-10T04:34:22.367 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: osd.2 (osd.2) 795 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:22.367 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 2021-02-10T04:34:22.367 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:22.367 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 02-10T04:34:17.572675+0000 2021-02-10T04:34:22.367 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: osd.2 (osd.2) 796 : cluster 2021-02-10T04:34:22.367 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:22.367 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:22.367 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: -02-10T04:34:18.476989+0000 osd.4 ( 2021-02-10T04:34:22.368 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: osd.4) 515 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:22.368 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: cluster 2021-02-10T04:34:22.368 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 2021-02-10T04:34:22.368 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 2021 2021-02-10T04:34:22.368 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: -02-10T04:34: 2021-02-10T04:34:22.368 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 18.476998+0000 osd.4 (osd.4) 516 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:22.368 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:22.368 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 02-10T 2021-02-10T04:34:22.368 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 04:34:18.477004+0000 osd.4 2021-02-10T04:34:22.368 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: (osd.4) 517 : cluster 2021-02-10T04:34:22.368 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:22.369 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: cluster 2021-02-10T04:34:22.369 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:34:22.369 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 10T04:34:18.477010+0000 osd.4 (osd. 2021-02-10T04:34:22.369 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 4) 518 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:22.369 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: cluster 2021-02-10T04:34:22.369 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: 2021-02-10T04:34:19.136957 2021-02-10T04:34:22.369 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1243 : cluster [DBG] osdmap e442: 8 total, 8 up, 8 in 2021-02-10T04:34:22.369 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: cluster 2021-02-10T04:34:22.369 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: 2021-02-10T04:34:22.369 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: 2021 2021-02-10T04:34:22.369 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: - 2021-02-10T04:34:22.369 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: 02 2021-02-10T04:34:22.369 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: - 2021-02-10T04:34:22.370 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: 10 2021-02-10T04:34:22.370 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: T 2021-02-10T04:34:22.370 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: 04 2021-02-10T04:34:22.370 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: : 2021-02-10T04:34:22.370 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: 34 2021-02-10T04:34:22.370 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: : 2021-02-10T04:34:22.370 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: 18 2021-02-10T04:34:22.370 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: . 2021-02-10T04:34:22.370 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: 608081 2021-02-10T04:34:22.370 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: +0000 2021-02-10T04:34:22.370 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: 2021-02-10T04:34:22.370 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: osd.2 2021-02-10T04:34:22.370 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: (osd.2) 797 2021-02-10T04:34:22.371 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:22.371 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: cluster 2021-02-10T04:34 2021-02-10T04:34:22.371 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: :18.608102+0000 osd.2 (osd.2) 798 2021-02-10T04:34:22.371 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:22.371 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:22.371 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: 34:18.608109+0000 osd.2 (osd.2 2021-02-10T04:34:22.371 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: ) 799 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:22.371 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:22.371 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: T04:34:18.608115+0000 osd.2 (osd 2021-02-10T04:34:22.371 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: .2) 800 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:22.371 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:22.371 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: -02-10T04:34:18.608120+0000 2021-02-10T04:34:22.371 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: osd.2 (osd.2) 801 : cluster [WRN] 2021-02-10T04:34:22.371 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:22.372 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: cluster 2021-02-10T04:34:18 2021-02-10T04:34:22.372 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: .608127+0000 osd.2 (osd.2) 802 : 2021-02-10T04:34:22.372 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:22.372 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:34:22.372 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: :34:18.608139+0000 osd.2 (osd.2 2021-02-10T04:34:22.372 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: ) 803 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:22.372 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:22.372 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: T04:34:18.608147+0000 osd.2 ( 2021-02-10T04:34:22.372 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: osd.2) 804 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:22.372 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:22.372 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: 02-10T04:34:18.608153+0000 2021-02-10T04:34:22.372 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: osd.2 (osd.2) 805 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:22.372 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:22.372 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: -10T04:34:18.608160+0000 osd.2 2021-02-10T04:34:22.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: (osd.2) 806 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:22.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:22.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: -10T04:34:19.484163+0000 osd.4 2021-02-10T04:34:22.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: (osd.4) 519 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:22.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: 2021-02-10T04:34:22.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:22.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: 02-10T04:34:19.484172+0000 osd.4 (osd.4) 520 : 2021-02-10T04:34:22.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:22.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: cluster 2021-02-10T04:34: 2021-02-10T04:34:22.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: 19.484178+0000 osd.4 (osd.4) 521 : cluster 2021-02-10T04:34:22.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:22.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: cluster 2021-02-10T04:34 2021-02-10T04:34:22.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: :19.484185+0000 osd.4 (osd.4) 2021-02-10T04:34:22.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: 522 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:22.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: cluster 2021-02-10T04:34:20.084166+0000 mgr.y (mgr.14140) 553 : cluster [DBG] pgmap v813: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 368 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 235 KiB/s wr, 4 op/s 2021-02-10T04:34:22.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:22.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[35961]: 10T04:34:20.141161+0000 mon.a (mon.0) 1244 : cluster [DBG] osdmap e443: 8 total, 8 up, 8 in 2021-02-10T04:34:22.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: cluster 2021-02-10T04:34:19.623003 2021-02-10T04:34:22.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 807 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:22.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:22.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: -10T04:34:19.623010+0000 osd.2 (osd.2) 808 : cluster [WRN] 2021-02-10T04:34:22.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:22.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: cluster 2021-02-10T04:34:22.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: 2021-02-10T04:34:19.623015+0000 osd.2 2021-02-10T04:34:22.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: (osd.2) 809 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:22.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:22.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: -02-10T04:34:19.623018+0000 2021-02-10T04:34:22.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: osd.2 (osd.2) 810 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:22.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: 2021-02-10T04:34:22.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: cluster 2021-02-10T04:34:22.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: 2021-02-10T04:34:19.623022+0000 osd.2 (osd. 2021-02-10T04:34:22.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: 2) 811 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:22.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:22.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: T04:34:19.623025+0000 osd.2 (osd 2021-02-10T04:34:22.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: .2) 812 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:22.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: cluster 2021-02-10T04:34 2021-02-10T04:34:22.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: :19.623029+0000 osd.2 (osd.2) 813 2021-02-10T04:34:22.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:22.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:22.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: 02-10T04:34:19.623033+0000 2021-02-10T04:34:22.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: osd.2 (osd.2) 814 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:22.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: 2021-02-10T04:34:22.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: cluster 2021-02-10T04:34:22.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: 2021 2021-02-10T04:34:22.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: - 2021-02-10T04:34:22.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: 02 2021-02-10T04:34:22.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: - 2021-02-10T04:34:22.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: 10T04:34:19.623036+0000 2021-02-10T04:34:22.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: osd.2 (osd.2) 815 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:22.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:22.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: -10T04:34:19.623042+0000 osd.2 ( 2021-02-10T04:34:22.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: osd.2) 816 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:22.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: cluster 2021-02-10T04:34:22.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: 2021-02-10T04:34:19.925664 2021-02-10T04:34:22.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: +0000 osd.6 (osd.6) 61 : cluster [DBG] 2021-02-10T04:34:22.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: purged_snaps scrub starts 2021-02-10T04:34:22.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: cluster 2021-02-10T04:34: 2021-02-10T04:34:22.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: 19.925842+0000 osd.6 (osd.6) 62 : cluster 2021-02-10T04:34:22.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: [DBG] purged_snaps scrub ok 2021-02-10T04:34:22.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:22.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: T04:34:20.464511+0000 osd.4 (osd. 2021-02-10T04:34:22.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: 4) 523 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:22.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: cluster 2021-02-10T04:34:20. 2021-02-10T04:34:22.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: 464520+0000 osd.4 (osd.4) 524 : cluster 2021-02-10T04:34:22.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:22.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: cluster 2021-02-10T04:34:22.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: 2021-02-10T04:34 2021-02-10T04:34:22.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: :20.464526+0000 osd.4 ( 2021-02-10T04:34:22.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: osd.4) 525 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:22.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: 2021-02-10T04:34:22.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: cluster 2021-02-10T04:34:22.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: 2021-02-10T04:34:20.464532+0000 osd.4 (osd. 2021-02-10T04:34:22.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[35961]: 4) 526 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:22.379 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: cluster 2021-02-10T04:34:22.379 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: 2021-02-10T04:34:22.379 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: 2021-02-10T04:34:17. 2021-02-10T04:34:22.379 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: 572603+0000 osd.2 (osd.2) 787 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:22.379 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: cluster 2021-02-10T04:34:17. 2021-02-10T04:34:22.379 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: 572612+0000 osd.2 (osd.2) 788 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:22.379 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:22.379 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: -10T04:34:17.572620+0000 osd.2 (osd. 2021-02-10T04:34:22.379 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: 2) 789 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:22.379 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:22.379 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: 02-10T04:34:17.572627+0000 osd.2 ( 2021-02-10T04:34:22.379 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: osd.2) 790 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:22.379 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: cluster 2021-02-10T04:34:22.380 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: 2021-02-10T04:34:22.380 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: 2021 2021-02-10T04:34:22.380 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: -02-10T04:34:17 2021-02-10T04:34:22.380 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: .572632+0000 osd.2 (osd.2) 791 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:22.380 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:22.380 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: -02-10T04:34:17.572638+0000 2021-02-10T04:34:22.380 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: osd.2 (osd.2) 792 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:22.380 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: cluster 2021-02-10T04:34:22.380 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:34:22.380 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: 04:34:17.572645+0000 osd.2 (osd.2) 793 2021-02-10T04:34:22.380 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:22.380 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: cluster 2021-02-10T04:34:22.380 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: 2021-02-10T04:34: 2021-02-10T04:34:22.380 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: 17.572650+0000 osd.2 (osd. 2021-02-10T04:34:22.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: 2) 794 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:22.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: 2021-02-10T04:34:22.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:22.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: 02-10T04:34:17. 2021-02-10T04:34:22.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: 572669+0000 osd.2 (osd.2 2021-02-10T04:34:22.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: ) 795 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:22.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: 2021-02-10T04:34:22.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: cluster 2021-02-10T04:34:22.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: 2021-02-10T04:34:17 2021-02-10T04:34:22.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: .572675+0000 osd.2 (osd.2) 796 2021-02-10T04:34:22.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:22.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: cluster 2021-02-10T04:34:22.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: 2021-02-10T04:34:18. 2021-02-10T04:34:22.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: 476989+0000 osd.4 (osd.4) 515 : cluster 2021-02-10T04:34:22.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:22.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: cluster 2021-02-10T04:34:22.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:34:22.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: T04:34:18. 2021-02-10T04:34:22.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: 476998+0000 osd.4 (osd.4) 516 : cluster 2021-02-10T04:34:22.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:22.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: cluster 2021-02-10T04:34:22.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: 2021-02-10T04:34 2021-02-10T04:34:22.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: :18.477004+0000 osd.4 (osd 2021-02-10T04:34:22.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: .4) 517 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:22.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:22.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: 02-10T04:34:18.477010+0000 osd.4 2021-02-10T04:34:22.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: (osd.4) 518 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:22.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: cluster 2021-02-10T04:34:22.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: 2021-02-10T04:34:19. 2021-02-10T04:34:22.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: 136957+0000 mon.a (mon.0) 1243 : cluster 2021-02-10T04:34:22.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:19 smithi085 conmon[29319]: [DBG] osdmap e442: 8 total, 8 up, 8 in 2021-02-10T04:34:22.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: cluster 2021-02-10T04:34:18.608081+0000 osd.2 ( 2021-02-10T04:34:22.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: osd.2) 797 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:22.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: cluster 2021-02-10T04:34:18.608102+0000 osd.2 (osd.2) 798 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:22.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: cluster 2021-02-10T04:34:18.608109+0000 osd.2 (osd.2) 799 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:22.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: cluster 2021-02-10T04:34:18.608115+0000 2021-02-10T04:34:22.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: osd.2 (osd.2) 800 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:22.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: cluster 2021-02-10T04:34:22.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: 2021-02-10T04:34:18.608120+0000 osd.2 (osd.2) 801 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:22.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: cluster 2021-02-10T04:34:18.608127+0000 2021-02-10T04:34:22.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: osd.2 (osd.2) 802 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:22.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:22.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: 10T04:34:18.608139+0000 osd.2 (osd 2021-02-10T04:34:22.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: .2) 803 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:22.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:22.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: -02-10T04:34:18.608147+0000 2021-02-10T04:34:22.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: osd.2 (osd.2) 804 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:22.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:22.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: 04:34:18.608153+0000 osd.2 (osd.2) 805 2021-02-10T04:34:22.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:22.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: cluster 2021-02-10T04:34:22.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: 2021-02-10T04:34:18.608160+0000 osd.2 (osd.2) 806 : 2021-02-10T04:34:22.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:22.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: cluster 2021-02-10T04:34:19 2021-02-10T04:34:22.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: .484163+0000 osd.4 (osd.4) 519 : cluster 2021-02-10T04:34:22.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:22.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:34:22.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: 34:19.484172+0000 osd.4 (osd.4) 520 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:22.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: cluster 2021-02-10T04:34: 2021-02-10T04:34:22.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: 19.484178+0000 osd.4 (osd.4) 521 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:22.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:22.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: 02-10T04:34:19.484185+0000 2021-02-10T04:34:22.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: osd.4 (osd.4) 522 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:22.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: 2021-02-10T04:34:22.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: cluster 2021-02-10T04:34: 2021-02-10T04:34:22.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: 20.084166+0000 mgr.y (mgr.14140) 553 : 2021-02-10T04:34:22.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: cluster [DBG] pgmap v813: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 368 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 235 KiB/s wr, 4 op/s 2021-02-10T04:34:22.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:22.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: 02-10T04:34:20.141161+0000 mon.a (mon.0) 2021-02-10T04:34:22.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:20 smithi085 conmon[29319]: 1244 : cluster [DBG] osdmap e443: 8 total, 8 up, 8 in 2021-02-10T04:34:22.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: cluster 2021-02-10T04:34:19.623003+0000 osd.2 (osd.2) 807 : cluster 2021-02-10T04:34:22.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:22.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: cluster 2021-02-10T04:34:19.623010+0000 osd.2 (osd.2) 808 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:22.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: cluster 2021-02-10T04:34 2021-02-10T04:34:22.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: :19.623015+0000 osd.2 (osd.2) 809 : cluster 2021-02-10T04:34:22.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:22.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: cluster 2021-02-10T04:34:22.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: 2021- 2021-02-10T04:34:22.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: 02-10T04:34:19.623018+0000 2021-02-10T04:34:22.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: osd.2 (osd.2 2021-02-10T04:34:22.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: ) 810 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:22.387 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: cluster 2021-02-10T04:34:22.387 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: 2021-02-10T04:34:22.387 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: 2021 2021-02-10T04:34:22.387 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: -02-10T04:34:19 2021-02-10T04:34:22.387 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: .623022+0000 osd.2 (osd.2) 811 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:22.387 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:34:22.387 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: 34:19.623025+0000 osd.2 (osd.2) 812 : 2021-02-10T04:34:22.387 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:22.387 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: cluster 2021-02-10T04:34:22.387 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:34:22.387 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: 04:34:19.623029+0000 osd.2 (osd 2021-02-10T04:34:22.387 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: .2) 813 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:22.387 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:22.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: 10T04:34:19.623033+0000 osd.2 (osd. 2021-02-10T04:34:22.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: 2) 814 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:22.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:22.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: 04:34:19.623036+0000 osd.2 2021-02-10T04:34:22.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: (osd.2) 815 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:22.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: 2021-02-10T04:34:22.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: cluster 2021-02-10T04:34: 2021-02-10T04:34:22.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: 19.623042+0000 osd.2 (osd.2) 2021-02-10T04:34:22.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: 816 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:22.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:22.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: -10T04:34:19. 2021-02-10T04:34:22.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: 925664 2021-02-10T04:34:22.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: +0000 osd.6 (osd.6) 2021-02-10T04:34:22.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: 61 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:34:22.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: cluster 2021-02-10T04:34:22.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: 2021-02-10T04:34:22.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: 2021 2021-02-10T04:34:22.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: - 2021-02-10T04:34:22.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: 02 2021-02-10T04:34:22.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: - 2021-02-10T04:34:22.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: 10 2021-02-10T04:34:22.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: T 2021-02-10T04:34:22.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: 04 2021-02-10T04:34:22.390 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: : 2021-02-10T04:34:22.390 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: 34 2021-02-10T04:34:22.390 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: : 2021-02-10T04:34:22.390 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: 19 2021-02-10T04:34:22.390 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: . 2021-02-10T04:34:22.390 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: 925842+0000 osd.6 (osd.6 2021-02-10T04:34:22.390 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: ) 62 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:34:22.390 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:22.390 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: 02-10T04:34:20.464511+0000 osd.4 (osd.4 2021-02-10T04:34:22.390 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: ) 523 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:22.390 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:22.390 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: 02-10T04:34:20.464520+0000 2021-02-10T04:34:22.390 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: osd.4 (osd.4) 524 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:22.391 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: 2021-02-10T04:34:22.391 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: cluster 2021-02-10T04:34:22.391 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: 2021-02-10T04:34:20.464526+0000 osd.4 (osd.4 2021-02-10T04:34:22.391 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: ) 525 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:22.391 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:34:22.391 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: 34:20.464532+0000 osd.4 (osd.4 2021-02-10T04:34:22.391 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:21 smithi085 conmon[29319]: ) 526 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:22.391 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[33677]: debug 2021-02-10T04:34:19.483+0000 7fbec19a1700 -1 osd.4 442 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:22.391 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[33677]: debug 2021-02-10T04:34:22.391 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[33677]: 2021-02-10T04:34:19.484+0000 7fbea5924700 -1 osd.4 pg_epoch: 442 pg[6.1a( empty local-lis/les=59/60 n=0 ec=59/59 lis/c=59/59 les/c/f=60/60/0 sis=59) [4,5,1] r=0 lpr=59 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.1a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x562760a58f20) 2021-02-10T04:34:22.391 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[33677]: debug 2021-02-10T04:34:20.463+0000 7fbec19a1700 -1 osd.4 443 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:22.391 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[33677]: debug 2021-02-10T04:34:21.423+0000 7fbec19a1700 -1 osd.4 444 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:22.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: cluster 2021-02-10T04:34:22.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: 2021-02-10T04:34:22.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: 2021-02-10T04:34:17 2021-02-10T04:34:22.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: .572603+0000 osd.2 (osd.2) 787 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:22.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: cluster 2021-02-10T04:34:22.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:34:22.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: :34:17.572612+0000 osd.2 (osd.2) 2021-02-10T04:34:22.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: 788 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:22.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: cluster 2021-02-10T04:34:22.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: 2021-02-10T04:34:17.572620+0000 2021-02-10T04:34:22.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: osd.2 (osd.2) 789 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:22.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:22.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: -10T04:34:17.572627+0000 osd.2 (osd.2) 790 : cluster 2021-02-10T04:34:22.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:22.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:22.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: T04:34:17.572632+0000 osd.2 (osd.2) 791 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:22.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: cluster 2021-02-10T04:34:17.572638+0000 osd.2 2021-02-10T04:34:22.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: (osd.2) 792 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:22.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: cluster 2021-02-10T04:34:17.572645+0000 osd.2 2021-02-10T04:34:22.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: (osd.2) 793 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:22.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: cluster 2021-02-10T04:34:17.572650+0000 osd.2 (osd. 2021-02-10T04:34:22.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: 2) 794 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:22.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: cluster 2021-02-10T04:34:17.572669 2021-02-10T04:34:22.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 795 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:22.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:22.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: 34:17.572675+0000 osd.2 (osd.2) 796 : cluster [WRN] 2021-02-10T04:34:22.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:22.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: cluster 2021-02-10T04:34:22.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: 2021-02-10T04:34:18.476989+0000 osd.4 (osd.4) 515 : 2021-02-10T04:34:22.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:22.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: cluster 2021-02-10T04:34:18. 2021-02-10T04:34:22.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: 476998+0000 osd.4 (osd.4) 516 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:22.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: 2021-02-10T04:34:22.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: cluster 2021-02-10T04:34:22.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: 2021-02-10T04:34:18. 2021-02-10T04:34:22.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: 477004+0000 osd.4 (osd.4) 2021-02-10T04:34:22.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: 517 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:22.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:22.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: -02-10T04:34:18.477010 2021-02-10T04:34:22.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 518 : cluster [WRN] 2021-02-10T04:34:22.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:22.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:22.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: 34:19.136957+0000 mon.a (mon.0) 1243 : 2021-02-10T04:34:22.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:19 smithi164 conmon[29288]: cluster [DBG] osdmap e442: 8 total, 8 up, 8 in 2021-02-10T04:34:22.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: cluster 2021-02-10T04:34:18.608081+0000 2021-02-10T04:34:22.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: osd.2 (osd.2) 797 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:22.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: cluster 2021-02-10T04:34:18.608102+0000 osd.2 (osd.2) 798 : cluster [WRN] 2021-02-10T04:34:22.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:22.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: cluster 2021-02-10T04:34 2021-02-10T04:34:22.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: :18.608109+0000 osd.2 (osd.2) 799 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:22.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:22.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: T04:34:18.608115+0000 osd.2 (osd.2 2021-02-10T04:34:22.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: ) 800 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:22.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: cluster 2021-02-10T04:34:18.608120+0000 osd.2 (osd 2021-02-10T04:34:22.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: .2) 801 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:22.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:22.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: 34:18.608127+0000 osd.2 (osd.2) 802 2021-02-10T04:34:22.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:22.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:22.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: :34:18.608139+0000 osd.2 (osd.2) 803 : cluster 2021-02-10T04:34:22.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:22.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: cluster 2021-02-10T04:34:22.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: 2021-02-10T04:34:18.608147+0000 osd.2 (osd 2021-02-10T04:34:22.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: .2) 804 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:22.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:22.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: 04:34:18.608153+0000 osd.2 (osd.2 2021-02-10T04:34:22.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: ) 805 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:22.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:22.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: T04:34:18.608160+0000 osd.2 (osd 2021-02-10T04:34:22.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: .2) 806 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:22.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: cluster 2021-02-10T04:34:19.484163+0000 osd.4 (osd.4) 519 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:22.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: 2021-02-10T04:34:22.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: cluster 2021-02-10T04:34 2021-02-10T04:34:22.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: :19.484172+0000 osd.4 (osd.4) 520 : cluster [WRN] 2021-02-10T04:34:22.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:22.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:22.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: -02-10T04:34:19.484178+0000 osd.4 (osd.4) 521 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:22.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: 2021-02-10T04:34:22.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:22.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: 34:19.484185+0000 osd.4 (osd.4) 522 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:22.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:22.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: 04:34:20.084166+0000 mgr.y (mgr.14140) 553 : cluster [DBG] pgmap v813: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 368 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 235 KiB/s wr, 4 op/s 2021-02-10T04:34:22.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:20 smithi164 conmon[29288]: cluster 2021-02-10T04:34:20.141161+0000 mon.a (mon.0) 1244 : cluster [DBG] osdmap e443: 8 total, 8 up, 8 in 2021-02-10T04:34:22.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:22.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: 04:34:19.623003+0000 osd.2 (osd.2) 807 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:22.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: cluster 2021-02-10T04:34:22.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: 2021-02-10T04:34:19.623010+0000 osd.2 (osd.2) 808 : cluster 2021-02-10T04:34:22.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:22.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: cluster 2021-02-10T04:34: 2021-02-10T04:34:22.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: 19.623015+0000 osd.2 (osd.2) 809 : cluster [WRN] 2021-02-10T04:34:22.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:22.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:22.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: 34:19.623018+0000 osd.2 (osd.2) 2021-02-10T04:34:22.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: 810 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:22.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:22.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: :34:19.623022+0000 osd.2 (osd. 2021-02-10T04:34:22.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: 2) 811 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:22.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:22.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: :34:19.623025+0000 osd.2 (osd. 2021-02-10T04:34:22.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: 2) 812 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:22.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:22.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: :34:19.623029+0000 osd.2 (osd.2) 813 : cluster 2021-02-10T04:34:22.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:22.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:22.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: -10T04:34:19.623033+0000 osd.2 (osd.2) 814 2021-02-10T04:34:22.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:22.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: cluster 2021-02-10T04:34: 2021-02-10T04:34:22.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: 19.623036+0000 osd.2 (osd.2) 815 2021-02-10T04:34:22.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:22.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:22.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: -10T04:34:19.623042+0000 osd.2 ( 2021-02-10T04:34:22.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: osd.2) 816 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:22.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:22.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: -10T04:34:19.925664+0000 2021-02-10T04:34:22.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: osd.6 (osd.6) 61 : cluster 2021-02-10T04:34:22.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: [DBG] purged_snaps scrub starts 2021-02-10T04:34:22.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:22.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: -02-10T04:34:19 2021-02-10T04:34:22.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: .925842+0000 osd.6 (osd.6 2021-02-10T04:34:22.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: ) 62 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:34:22.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:22.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: -10T04:34:20.464511+0000 osd.4 (osd.4) 523 : cluster 2021-02-10T04:34:22.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:22.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:22.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: 04:34:20.464520+0000 osd.4 (osd.4) 524 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:22.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: cluster 2021-02-10T04:34:20 2021-02-10T04:34:22.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: .464526+0000 osd.4 (osd.4) 525 : cluster 2021-02-10T04:34:22.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:22.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: cluster 2021-02-10T04:34:22.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: 2021-02-10T04:34:20.464532 2021-02-10T04:34:22.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 526 2021-02-10T04:34:22.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:21 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:22.588 INFO:tasks.rados.rados.0.smithi085.stdout:1691: finishing write tid 1 to smithi08575397-2 2021-02-10T04:34:22.589 INFO:tasks.rados.rados.0.smithi085.stdout:1691: finishing write tid 2 to smithi08575397-2 2021-02-10T04:34:22.589 INFO:tasks.rados.rados.0.smithi085.stdout:1691: finishing write tid 3 to smithi08575397-2 2021-02-10T04:34:22.589 INFO:tasks.rados.rados.0.smithi085.stdout:1691: finishing write tid 4 to smithi08575397-2 2021-02-10T04:34:22.589 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 634 (ObjNum 602 snap 180 seq_num 602) dirty exists 2021-02-10T04:34:22.589 INFO:tasks.rados.rados.0.smithi085.stdout:1691: left oid 2 (ObjNum 602 snap 180 seq_num 602) 2021-02-10T04:34:22.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[29319]: cluster 2021-02-10T04:34:20.613544+0000 2021-02-10T04:34:22.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[29319]: osd.2 (osd.2) 817 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:22.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[29319]: cluster 2021-02-10T04:34:20.613551+0000 osd.2 (osd.2 2021-02-10T04:34:22.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[29319]: ) 818 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:22.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[29319]: cluster 2021-02-10T04:34:22.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[29319]: 2021-02-10T04:34:20.613556+0000 osd.2 (osd.2) 819 2021-02-10T04:34:22.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:22.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:34:22.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[29319]: :34:20.613559+0000 osd.2 (osd.2) 820 2021-02-10T04:34:22.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:22.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:22.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[29319]: 10T04:34:20.613562+0000 osd.2 2021-02-10T04:34:22.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[29319]: (osd.2) 821 : cluster 2021-02-10T04:34:22.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:22.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[29319]: cluster 2021-02-10T04:34:20.613566+0000 osd.2 (osd.2) 822 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:22.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[29319]: cluster 2021-02-10T04:34:20.613576+0000 osd.2 (osd.2) 823 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:22.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[29319]: cluster 2021-02-10T04:34:20.613582+0000 osd.2 (osd.2) 824 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:22.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[29319]: cluster 2021-02-10T04:34:20.613586+0000 osd.2 (osd.2) 825 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:22.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[29319]: cluster 2021-02-10T04:34:20.613589+0000 osd.2 (osd.2) 826 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:22.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[29319]: cluster 2021-02-10T04:34:21.284594+0000 mon.a (mon.0) 1245 : cluster [DBG] osdmap e444: 8 total, 8 up, 8 in 2021-02-10T04:34:22.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[29319]: cluster 2021-02-10T04:34:21.424175+0000 osd.4 (osd.4) 527 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:22.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[29319]: cluster 2021-02-10T04:34:21.424184+0000 osd.4 (osd.4) 528 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:22.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[29319]: cluster 2021-02-10T04:34:21.424207+0000 osd.4 (osd.4) 529 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:22.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[29319]: cluster 2021-02-10T04:34:21.424214+0000 osd.4 (osd.4) 530 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:22.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[29319]: cluster 2021-02-10T04:34:22.084813+0000 mgr.y (mgr.14140) 554 : cluster [DBG] pgmap v816: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 366 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 739 KiB/s wr, 13 op/s 2021-02-10T04:34:22.637 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: cluster 2021-02-10T04:34:20.613544+0000 osd.2 2021-02-10T04:34:22.637 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: (osd.2) 817 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:22.637 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: cluster 2021-02-10T04:34:22.637 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: 2021-02-10T04:34:20.613551+0000 osd.2 (osd.2) 2021-02-10T04:34:22.637 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: 818 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:22.638 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: cluster 2021-02-10T04:34: 2021-02-10T04:34:22.638 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: 20.613556+0000 osd.2 (osd.2) 819 : 2021-02-10T04:34:22.638 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:22.638 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:22.638 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: 34:20.613559+0000 osd.2 (osd.2) 2021-02-10T04:34:22.638 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: 820 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:22.638 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:22.638 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: 02-10T04:34:20.613562+0000 osd.2 2021-02-10T04:34:22.638 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: (osd.2) 821 : cluster [WRN] 2021-02-10T04:34:22.638 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:22.638 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:22.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: 10T04:34:20.613566+0000 osd.2 2021-02-10T04:34:22.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: (osd.2) 822 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:22.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:22.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: 02-10T04:34:20.613576+0000 osd.2 2021-02-10T04:34:22.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: (osd.2) 823 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:22.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: 2021-02-10T04:34:22.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:22.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: -10T04:34:20.613582+0000 osd.2 (osd.2) 2021-02-10T04:34:22.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: 824 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:22.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:22.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: T04:34:20.613586+0000 osd.2 (osd 2021-02-10T04:34:22.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: .2) 825 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:22.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:22.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: 10T04:34:20.613589+0000 osd.2 2021-02-10T04:34:22.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: (osd.2) 826 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:22.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:22.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: 10T04:34:21.284594+0000 mon.a (mon.0) 1245 : cluster 2021-02-10T04:34:22.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: [DBG] osdmap e444: 8 total, 8 up, 8 in 2021-02-10T04:34:22.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: cluster 2021-02-10T04:34:22.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: 2021-02-10T04:34:21.424175+0000 osd.4 (osd. 2021-02-10T04:34:22.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: 4) 527 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:22.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: cluster 2021-02-10T04:34 2021-02-10T04:34:22.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: :21.424184+0000 osd.4 (osd.4) 528 : 2021-02-10T04:34:22.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:22.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: cluster 2021-02-10T04:34:22.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: 2021-02-10T04:34:21.424207+0000 osd.4 (osd.4) 529 : cluster 2021-02-10T04:34:22.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:22.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:22.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: -02-10T04:34:21.424214+0000 osd.4 (osd.4) 530 : cluster [WRN] 2021-02-10T04:34:22.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:22.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:22.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: 02-10T04:34:22.084813+0000 mgr.y (mgr.14140) 554 : cluster 2021-02-10T04:34:22.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[35961]: [DBG] pgmap v816: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 366 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 739 KiB/s wr, 13 op/s 2021-02-10T04:34:22.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[33677]: debug 2021-02-10T04:34:22.430+0000 7fbec19a1700 -1 osd.4 445 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: cluster 2021-02-10T04:34:20.613544+0000 osd.2 2021-02-10T04:34:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: (osd.2) 817 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: cluster 2021-02-10T04:34:20.613551+0000 osd.2 (osd.2) 818 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: cluster 2021-02-10T04:34:20.613556+0000 osd.2 ( 2021-02-10T04:34:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: osd.2) 819 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: cluster 2021-02-10T04:34:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: 2021-02-10T04:34:20.613559+0000 osd.2 (osd.2) 820 : cluster 2021-02-10T04:34:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: cluster 2021-02-10T04:34:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: 2021-02-10T04:34:20.613562+0000 osd.2 (osd.2 2021-02-10T04:34:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: ) 821 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: cluster 2021-02-10T04:34: 2021-02-10T04:34:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: 20.613566+0000 osd.2 (osd.2) 822 : cluster 2021-02-10T04:34:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: cluster 2021-02-10T04:34 2021-02-10T04:34:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: :20.613576+0000 osd.2 (osd.2) 823 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: -10T04:34:20.613582+0000 osd.2 (osd. 2021-02-10T04:34:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: 2) 824 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: 34:20.613586+0000 osd.2 (osd.2) 825 2021-02-10T04:34:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: 02-10T04:34:20.613589+0000 osd.2 ( 2021-02-10T04:34:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: osd.2) 826 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: -02-10T04:34:21.284594+0000 mon.a 2021-02-10T04:34:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: (mon.0) 1245 : cluster [DBG] osdmap e444: 8 total, 8 up, 8 in 2021-02-10T04:34:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: 04:34:21.424175+0000 osd.4 (osd.4) 527 2021-02-10T04:34:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: cluster 2021-02-10T04:34:21.424184+0000 2021-02-10T04:34:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: osd.4 (osd.4) 528 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: 02-10T04:34:21.424207+0000 osd.4 (osd.4 2021-02-10T04:34:22.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: ) 529 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: cluster 2021-02-10T04:34:21.424214+0000 2021-02-10T04:34:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: osd.4 (osd.4) 530 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: 10T04:34:22.084813+0000 mgr.y ( 2021-02-10T04:34:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: mgr.14140) 554 : cluster [DBG] 2021-02-10T04:34:22.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:22 smithi164 conmon[29288]: pgmap v816: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 366 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 739 KiB/s wr, 13 op/s 2021-02-10T04:34:22.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:22 smithi085 conmon[49106]: debug 2021-02-10T04:34:22.632+0000 7f77a3e94700 -1 osd.2 445 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:23.307 INFO:tasks.rados.rados.0.smithi085.stdout:1687: done (12 left) 2021-02-10T04:34:23.307 INFO:tasks.rados.rados.0.smithi085.stdout:1690: done (11 left) 2021-02-10T04:34:23.307 INFO:tasks.rados.rados.0.smithi085.stdout:1691: done (10 left) 2021-02-10T04:34:23.307 INFO:tasks.rados.rados.0.smithi085.stdout:1693: done (9 left) 2021-02-10T04:34:23.308 INFO:tasks.rados.rados.0.smithi085.stdout:1694: delete oid 30 current snap is 181 2021-02-10T04:34:23.316 INFO:tasks.rados.rados.0.smithi085.stdout:1694: done (9 left) 2021-02-10T04:34:23.316 INFO:tasks.rados.rados.0.smithi085.stdout:1695: write (excl) oid 28 current snap is 181 2021-02-10T04:34:23.316 INFO:tasks.rados.rados.0.smithi085.stdout:1695: seq_num 604 ranges {551428=642283,1694771=456657} 2021-02-10T04:34:23.323 INFO:tasks.rados.rados.0.smithi085.stdout:1695: writing smithi08575397-28 from 551428 to 1193711 tid 1 2021-02-10T04:34:23.330 INFO:tasks.rados.rados.0.smithi085.stdout:1695: writing smithi08575397-28 from 1694771 to 2151428 tid 2 2021-02-10T04:34:23.330 INFO:tasks.rados.rados.0.smithi085.stdout:1696: rollback oid 7 current snap is 181 2021-02-10T04:34:23.330 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 7 to 170 2021-02-10T04:34:23.331 INFO:tasks.rados.rados.0.smithi085.stdout:1695: finishing write tid 1 to smithi08575397-28 2021-02-10T04:34:23.331 INFO:tasks.rados.rados.0.smithi085.stdout:1697: snap_create 2021-02-10T04:34:23.334 INFO:tasks.rados.rados.0.smithi085.stdout:1695: finishing write tid 2 to smithi08575397-28 2021-02-10T04:34:23.335 INFO:tasks.rados.rados.0.smithi085.stdout:1695: finishing write tid 3 to smithi08575397-28 2021-02-10T04:34:23.335 INFO:tasks.rados.rados.0.smithi085.stdout:1695: finishing write tid 4 to smithi08575397-28 2021-02-10T04:34:23.335 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 734 (ObjNum 604 snap 181 seq_num 604) dirty exists 2021-02-10T04:34:23.335 INFO:tasks.rados.rados.0.smithi085.stdout:1695: left oid 28 (ObjNum 604 snap 181 seq_num 604) 2021-02-10T04:34:23.335 INFO:tasks.rados.rados.0.smithi085.stdout:1696: finishing rollback tid 0 to smithi08575397-7 2021-02-10T04:34:23.335 INFO:tasks.rados.rados.0.smithi085.stdout:1696: finishing rollback tid 1 to smithi08575397-7 2021-02-10T04:34:23.336 INFO:tasks.rados.rados.0.smithi085.stdout:1696: finishing rollback tid 2 to smithi08575397-7 2021-02-10T04:34:23.336 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 703 (ObjNum 542 snap 168 seq_num 542) dirty exists 2021-02-10T04:34:23.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: cluster 2021-02-10T04:34:23.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: 2021-02-10T04:34:23.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: 2021 2021-02-10T04:34:23.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: - 2021-02-10T04:34:23.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: 02 2021-02-10T04:34:23.619 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: - 2021-02-10T04:34:23.619 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: 10 2021-02-10T04:34:23.619 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: T 2021-02-10T04:34:23.619 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: 04 2021-02-10T04:34:23.619 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: : 2021-02-10T04:34:23.619 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: 34 2021-02-10T04:34:23.619 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: : 2021-02-10T04:34:23.619 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: 21 2021-02-10T04:34:23.619 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: . 2021-02-10T04:34:23.619 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: 621604 2021-02-10T04:34:23.619 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: +0000 2021-02-10T04:34:23.619 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: osd.2 (osd.2) 827 : cluster 2021-02-10T04:34:23.619 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:23.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:23.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: -02-10T04:34:21.621612+0000 osd.2 (osd.2) 828 2021-02-10T04:34:23.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:23.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:23.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: -10T04:34:21.621639+0000 osd.2 2021-02-10T04:34:23.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: (osd.2) 829 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:23.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: cluster 2021-02-10T04:34:23.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: 2021-02-10T04:34:23.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: 2021 2021-02-10T04:34:23.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: - 2021-02-10T04:34:23.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: 02 2021-02-10T04:34:23.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: - 2021-02-10T04:34:23.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: 10 2021-02-10T04:34:23.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: T 2021-02-10T04:34:23.621 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: 04:34:21.621650+0000 2021-02-10T04:34:23.621 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: osd.2 (osd.2) 830 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:23.621 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:23.621 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: 10T04:34:21.621656+0000 osd.2 ( 2021-02-10T04:34:23.621 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: osd.2) 831 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:23.621 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: cluster 2021-02-10T04:34:23.621 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:34:23.621 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: 34:21.621674+0000 osd.2 (osd. 2021-02-10T04:34:23.621 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: 2) 832 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:23.621 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: cluster 2021-02-10T04:34:23.621 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: 2021-02-10T04:34:21 2021-02-10T04:34:23.621 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: .621681+0000 osd.2 (osd.2) 833 : 2021-02-10T04:34:23.621 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:23.622 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:23.622 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: -10T04:34:21.621687+0000 2021-02-10T04:34:23.622 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: osd.2 (osd.2) 834 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:23.622 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: 2021-02-10T04:34:23.622 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:23.622 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: 02-10T04:34:21.621693+0000 2021-02-10T04:34:23.623 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: osd.2 (osd.2) 835 : cluster [WRN] 2021-02-10T04:34:23.623 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:23.623 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:23.623 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: 04:34:21.621699+0000 osd.2 ( 2021-02-10T04:34:23.623 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: osd.2) 836 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:23.623 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: cluster 2021-02-10T04:34:23.623 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: 2021-02-10T04:34:22. 2021-02-10T04:34:23.623 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: 281080+0000 mon.a (mon.0) 1246 : cluster 2021-02-10T04:34:23.623 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: [WRN] Health check update: 14 slow ops, oldest one blocked for 292 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:34:23.623 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:34:23.623 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: T04:34:22.293321+0000 mon.a (mon. 2021-02-10T04:34:23.623 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: 0) 1247 : cluster [DBG] osdmap e445: 8 total, 8 up, 8 in 2021-02-10T04:34:23.623 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: cluster 2021-02-10T04:34:23.623 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: 2021-02-10T04:34:23.624 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: 2021 2021-02-10T04:34:23.624 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: -02-10T04:34: 2021-02-10T04:34:23.624 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: 22.431275+0000 osd.4 (osd.4) 531 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:23.624 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:23.624 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: -10T04:34:22.431282+0000 osd.4 (osd. 2021-02-10T04:34:23.624 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: 4) 532 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:23.624 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:23.624 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: 04:34:22.431286+0000 osd.4 (osd.4 2021-02-10T04:34:23.624 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: ) 533 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:23.624 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:34:23.624 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: 34:22.431289+0000 osd.4 (osd.4 2021-02-10T04:34:23.624 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[29319]: ) 534 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:23.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: cluster 2021-02-10T04:34:21.621604+0000 osd.2 (osd. 2021-02-10T04:34:23.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: 2) 827 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:23.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:23.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: 02-10T04:34:21.621612+0000 osd.2 (osd.2) 828 : 2021-02-10T04:34:23.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:23.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:34:23.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: :34:21.621639+0000 osd.2 (osd. 2021-02-10T04:34:23.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: 2) 829 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:23.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:23.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: 10T04:34:21.621650+0000 osd.2 (osd 2021-02-10T04:34:23.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: .2) 830 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:23.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:23.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: 02-10T04:34:21.621656+0000 2021-02-10T04:34:23.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: osd.2 (osd.2) 831 : cluster [WRN] 2021-02-10T04:34:23.626 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:23.626 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: cluster 2021-02-10T04:34:21 2021-02-10T04:34:23.626 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: .621674+0000 osd.2 (osd.2) 832 : cluster [WRN] 2021-02-10T04:34:23.626 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:23.626 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: cluster 2021-02-10T04:34:21. 2021-02-10T04:34:23.626 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: 621681+0000 osd.2 (osd.2) 833 : cluster [WRN] 2021-02-10T04:34:23.626 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:23.626 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: cluster 2021-02-10T04:34:21. 2021-02-10T04:34:23.626 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: 621687+0000 osd.2 (osd.2) 834 : cluster 2021-02-10T04:34:23.626 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:23.626 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: cluster 2021-02-10T04:34:23.626 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: 2021-02-10T04:34:21 2021-02-10T04:34:23.626 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: .621693+0000 osd.2 (osd.2) 835 : cluster 2021-02-10T04:34:23.627 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:23.627 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:34:23.627 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: :34:21.621699+0000 osd.2 (osd. 2021-02-10T04:34:23.627 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: 2) 836 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:23.627 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:23.627 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: 02-10T04:34:22.281080 2021-02-10T04:34:23.627 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1246 : cluster [WRN] 2021-02-10T04:34:23.627 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: Health check update: 14 slow ops, oldest one blocked for 292 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:34:23.627 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: cluster 2021-02-10T04:34 2021-02-10T04:34:23.627 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: :22.293321+0000 mon.a (mon.0) 1247 2021-02-10T04:34:23.627 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: : cluster [DBG] osdmap e445: 8 total, 8 up, 8 in 2021-02-10T04:34:23.627 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:34:23.627 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: 04:34:22.431275+0000 osd.4 (osd 2021-02-10T04:34:23.627 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: .4) 531 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:23.628 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:23.628 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: -02-10T04:34:22.431282 2021-02-10T04:34:23.628 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 532 : cluster 2021-02-10T04:34:23.628 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:23.628 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:23.628 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: -02-10T04:34:22.431286 2021-02-10T04:34:23.628 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 533 : cluster 2021-02-10T04:34:23.628 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:23.628 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: cluster 2021-02-10T04:34: 2021-02-10T04:34:23.628 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: 22.431289+0000 osd.4 (osd.4) 534 2021-02-10T04:34:23.628 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:23.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[33677]: debug 2021-02-10T04:34:23.468+0000 7fbec19a1700 -1 osd.4 446 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: cluster 2021-02-10T04:34:21.621604 2021-02-10T04:34:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 827 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: cluster 2021-02-10T04:34:21. 2021-02-10T04:34:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: 621612+0000 osd.2 (osd.2) 828 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: -10T04:34:21.621639+0000 osd.2 (osd.2) 829 : cluster 2021-02-10T04:34:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: cluster 2021-02-10T04:34:21.621650+0000 osd.2 (osd.2 2021-02-10T04:34:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: ) 830 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: cluster 2021-02-10T04:34:21.621656+0000 2021-02-10T04:34:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: osd.2 (osd.2) 831 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: :34:21.621674+0000 osd.2 (osd.2) 832 : cluster 2021-02-10T04:34:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: cluster 2021-02-10T04:34:21.621681+0000 osd.2 (osd. 2021-02-10T04:34:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: 2) 833 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: cluster 2021-02-10T04:34:21.621687+0000 osd.2 (osd. 2021-02-10T04:34:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: 2) 834 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: cluster 2021-02-10T04:34:21. 2021-02-10T04:34:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: 621693+0000 osd.2 (osd.2) 835 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: -10T04:34:21.621699+0000 osd.2 (osd.2 2021-02-10T04:34:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: ) 836 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: 34:22.281080+0000 mon.a (mon.0) 1246 : cluster [WRN] 2021-02-10T04:34:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: Health check update: 14 slow ops, oldest one blocked for 292 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:34:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: cluster 2021-02-10T04:34:22.293321+0000 2021-02-10T04:34:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: mon.a (mon.0) 1247 : cluster [DBG] osdmap e445: 8 total, 8 up, 8 in 2021-02-10T04:34:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: T04:34:22.431275+0000 osd.4 (osd.4) 531 : cluster 2021-02-10T04:34:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: cluster 2021-02-10T04:34:22.431282+0000 2021-02-10T04:34:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: osd.4 (osd.4) 532 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: T04:34:22.431286+0000 osd.4 (osd.4) 533 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: cluster 2021-02-10T04:34:22.431289+0000 2021-02-10T04:34:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:23 smithi164 conmon[29288]: osd.4 (osd.4) 534 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:23.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:23 smithi085 conmon[49106]: debug 2021-02-10T04:34:23.610+0000 7f77a3e94700 -1 osd.2 446 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:24.315 INFO:tasks.rados.rados.0.smithi085.stdout:1695: done (11 left) 2021-02-10T04:34:24.315 INFO:tasks.rados.rados.0.smithi085.stdout:1696: done (10 left) 2021-02-10T04:34:24.316 INFO:tasks.rados.rados.0.smithi085.stdout:1697: done (9 left) 2021-02-10T04:34:24.316 INFO:tasks.rados.rados.0.smithi085.stdout:1698: rollback oid 36 current snap is 182 2021-02-10T04:34:24.316 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 36 to 165 2021-02-10T04:34:24.316 INFO:tasks.rados.rados.0.smithi085.stdout:1699: snap_create 2021-02-10T04:34:24.326 INFO:tasks.rados.rados.0.smithi085.stdout:1698: finishing rollback tid 0 to smithi08575397-36 2021-02-10T04:34:24.326 INFO:tasks.rados.rados.0.smithi085.stdout:1698: finishing rollback tid 1 to smithi08575397-36 2021-02-10T04:34:24.326 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 655 (ObjNum 82531232 snap 22030 seq_num 82758704) dirty dne 2021-02-10T04:34:24.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: cluster 2021-02-10T04:34:24.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 2021-02-10T04:34:24.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 2021 2021-02-10T04:34:24.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: - 2021-02-10T04:34:24.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 02 2021-02-10T04:34:24.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: - 2021-02-10T04:34:24.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 10 2021-02-10T04:34:24.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: T 2021-02-10T04:34:24.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 04 2021-02-10T04:34:24.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: : 2021-02-10T04:34:24.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 34 2021-02-10T04:34:24.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: : 2021-02-10T04:34:24.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 22 2021-02-10T04:34:24.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: . 2021-02-10T04:34:24.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 632791 2021-02-10T04:34:24.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: +0000 2021-02-10T04:34:24.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 2021-02-10T04:34:24.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: osd.2 2021-02-10T04:34:24.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: ( 2021-02-10T04:34:24.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: osd 2021-02-10T04:34:24.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: . 2021-02-10T04:34:24.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 2 2021-02-10T04:34:24.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: ) 2021-02-10T04:34:24.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 837 2021-02-10T04:34:24.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: : 2021-02-10T04:34:24.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: cluster 2021-02-10T04:34:24.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 2021-02-10T04:34:24.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: [WRN] 2021-02-10T04:34:24.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 2021-02-10T04:34:24.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:24.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 2021-02-10T04:34:24.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: cluster 2021-02-10T04:34:24.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 2021-02-10T04:34:24.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 2021 2021-02-10T04:34:24.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: - 2021-02-10T04:34:24.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 02 2021-02-10T04:34:24.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: - 2021-02-10T04:34:24.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 10 2021-02-10T04:34:24.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: T 2021-02-10T04:34:24.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 04 2021-02-10T04:34:24.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: : 2021-02-10T04:34:24.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 34 2021-02-10T04:34:24.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: :22.632801+0000 osd.2 (osd. 2021-02-10T04:34:24.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 2) 838 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:24.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:24.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 02-10T04:34:22.632808+0000 osd.2 2021-02-10T04:34:24.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: (osd.2) 839 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:24.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 2021-02-10T04:34:24.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:24.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 02-10T04:34:22.632814+0000 osd.2 (osd. 2021-02-10T04:34:24.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 2) 840 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:24.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:24.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 02-10T04:34:22.632820+0000 osd.2 2021-02-10T04:34:24.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: (osd.2) 841 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:24.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 2021-02-10T04:34:24.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:24.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: T04:34:22.632824+0000 osd.2 (osd.2) 842 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:24.650 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:24.650 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: -10T04:34:22.632830+0000 osd.2 (osd. 2021-02-10T04:34:24.650 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 2) 843 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:24.650 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:24.650 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 02-10T04:34:22.632836+0000 osd.2 (osd.2 2021-02-10T04:34:24.650 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: ) 844 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:24.650 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: cluster 2021-02-10T04:34:24.650 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 2021-02-10T04:34:24.650 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 2021 2021-02-10T04:34:24.650 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: -02-10T 2021-02-10T04:34:24.650 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 04:34:22.632841+0000 osd.2 (osd.2) 845 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:24.650 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: cluster 2021-02-10T04:34:24.651 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 2021-02-10T04:34:24.651 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 2021 2021-02-10T04:34:24.651 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: - 2021-02-10T04:34:24.651 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 02 2021-02-10T04:34:24.651 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: - 2021-02-10T04:34:24.651 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 10 2021-02-10T04:34:24.651 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: T 2021-02-10T04:34:24.651 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 04 2021-02-10T04:34:24.651 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: : 2021-02-10T04:34:24.651 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 34:22.632844+0000 2021-02-10T04:34:24.651 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: osd.2 (osd.2) 846 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:24.651 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: cluster 2021-02-10T04:34:24.652 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 2021-02-10T04:34:24.652 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 2021 2021-02-10T04:34:24.652 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: - 2021-02-10T04:34:24.652 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 02 2021-02-10T04:34:24.652 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: -10T04:34 2021-02-10T04:34:24.652 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: :23.307500+0000 mon.a (mon.0) 1248 : cluster [DBG] osdmap e446: 8 total, 8 up, 8 in 2021-02-10T04:34:24.652 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: cluster 2021-02-10T04:34:24.652 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 2021-02 2021-02-10T04:34:24.652 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: -10T04:34:23.469068+0000 osd.4 (osd.4 2021-02-10T04:34:24.652 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: ) 535 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:24.652 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:24.652 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: -10T04:34:23.469076+0000 osd.4 (osd.4) 536 : cluster 2021-02-10T04:34:24.653 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:24.653 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:24.653 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: -10T04:34:23.469080 2021-02-10T04:34:24.653 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: +0000 osd.4 ( 2021-02-10T04:34:24.653 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: osd.4) 537 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:24.653 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: cluster 2021-02-10T04:34:24.653 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 2021-02-10T04:34:24.653 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 2021 2021-02-10T04:34:24.653 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: - 2021-02-10T04:34:24.653 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 02 2021-02-10T04:34:24.653 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: - 2021-02-10T04:34:24.653 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 10 2021-02-10T04:34:24.653 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: T 2021-02-10T04:34:24.654 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 04 2021-02-10T04:34:24.654 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: : 2021-02-10T04:34:24.654 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 34 2021-02-10T04:34:24.654 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: :23.469085+0000 2021-02-10T04:34:24.654 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: osd.4 (osd.4) 538 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:24.654 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: cluster 2021-02-10T04:34:24.654 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:34:24.654 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: 10T04:34:24.085286+0000 mgr.y (mgr.14140) 555 2021-02-10T04:34:24.654 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[35961]: : cluster [DBG] pgmap v819: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 366 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 8.9 MiB/s rd, 503 KiB/s wr, 10 op/s 2021-02-10T04:34:24.654 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: cluster 2021-02-10T04:34:22.632791+0000 osd.2 (osd. 2021-02-10T04:34:24.654 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: 2) 837 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:24.655 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: cluster 2021-02-10T04:34:22.632801+0000 osd.2 (osd.2) 838 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:24.655 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: cluster 2021-02-10T04:34:22.632808+0000 osd.2 (osd.2) 839 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:24.655 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: cluster 2021-02-10T04:34 2021-02-10T04:34:24.655 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: :22.632814+0000 osd.2 (osd.2) 840 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:24.655 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: cluster 2021-02-10T04:34:24.655 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: 2021-02-10T04:34:22.632820 2021-02-10T04:34:24.655 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 841 2021-02-10T04:34:24.655 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:24.655 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:24.655 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: 02-10T04:34:22.632824+0000 osd.2 2021-02-10T04:34:24.655 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: (osd.2) 842 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:24.655 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:24.656 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: -10T04:34:22.632830+0000 osd.2 (osd. 2021-02-10T04:34:24.656 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: 2) 843 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:24.656 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:24.656 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: -10T04:34:22.632836+0000 osd.2 ( 2021-02-10T04:34:24.656 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: osd.2) 844 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:24.656 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:24.656 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: -10T04:34:22.632841+0000 osd.2 ( 2021-02-10T04:34:24.656 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: osd.2) 845 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:24.656 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:24.656 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: -02-10T04:34:22.632844 2021-02-10T04:34:24.656 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 846 : cluster 2021-02-10T04:34:24.656 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:24.657 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:34:24.657 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: T04:34 2021-02-10T04:34:24.657 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: : 2021-02-10T04:34:24.657 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: 23.307500+0000 mon.a 2021-02-10T04:34:24.657 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: (mon.0) 1248 : cluster [DBG] osdmap e446: 8 total, 8 up, 8 in 2021-02-10T04:34:24.657 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:24.657 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: 10T04:34:23.469068+0000 osd.4 (osd. 2021-02-10T04:34:24.657 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: 4) 535 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:24.657 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:34:24.657 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: :34:23.469076+0000 osd.4 (osd. 2021-02-10T04:34:24.657 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: 4) 536 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:24.658 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:24.658 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: 02-10T04:34:23.469080+0000 osd.4 ( 2021-02-10T04:34:24.658 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: osd.4) 537 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:24.658 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:24.658 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: 04:34:23.469085+0000 osd.4 (osd.4) 538 : cluster [WRN] 2021-02-10T04:34:24.658 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:24.658 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:34:24.658 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[29319]: T04:34:24.085286+0000 mgr.y (mgr.14140) 555 : cluster [DBG] pgmap v819: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 366 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 8.9 MiB/s rd, 503 KiB/s wr, 10 op/s 2021-02-10T04:34:24.713 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[33677]: debug 2021-02-10T04:34:24.493+0000 7fbec19a1700 -1 osd.4 447 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:24.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: cluster 2021-02-10T04:34:24.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: 2021-02-10T04:34:24.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: 2021 2021-02-10T04:34:24.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: - 2021-02-10T04:34:24.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: 02 2021-02-10T04:34:24.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: - 2021-02-10T04:34:24.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: 10 2021-02-10T04:34:24.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: T 2021-02-10T04:34:24.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: 04 2021-02-10T04:34:24.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: : 2021-02-10T04:34:24.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: 34 2021-02-10T04:34:24.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: : 2021-02-10T04:34:24.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: 22 2021-02-10T04:34:24.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: . 2021-02-10T04:34:24.714 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: 632791 2021-02-10T04:34:24.715 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: +0000 2021-02-10T04:34:24.715 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: 2021-02-10T04:34:24.715 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: osd.2 2021-02-10T04:34:24.715 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: ( 2021-02-10T04:34:24.715 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: osd.2) 837 : cluster [WRN] 2021-02-10T04:34:24.715 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:24.715 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: cluster 2021-02-10T04:34:24.715 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: 2021-02-10T04:34:22.632801+0000 osd.2 (osd.2) 838 : cluster 2021-02-10T04:34:24.715 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:24.715 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:24.715 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: :34:22.632808+0000 osd.2 (osd.2) 839 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:24.715 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:24.715 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: 10T04:34:22.632814+0000 osd.2 (osd. 2021-02-10T04:34:24.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: 2) 840 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:24.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:24.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: 10T04:34:22.632820+0000 osd.2 ( 2021-02-10T04:34:24.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: osd.2) 841 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:24.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:24.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: -10T04:34:22.632824+0000 osd.2 ( 2021-02-10T04:34:24.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: osd.2) 842 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:24.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:24.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: 02-10T04:34:22.632830+0000 osd.2 ( 2021-02-10T04:34:24.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: osd.2) 843 : cluster [WRN] 2021-02-10T04:34:24.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:24.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: cluster 2021-02-10T04:34:24.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: 2021-02-10T04:34:22.632836 2021-02-10T04:34:24.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 844 : cluster 2021-02-10T04:34:24.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:24.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: cluster 2021-02-10T04:34:22 2021-02-10T04:34:24.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: .632841+0000 osd.2 (osd.2) 845 : cluster 2021-02-10T04:34:24.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:24.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:24.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: 34:22.632844+0000 osd.2 (osd.2) 2021-02-10T04:34:24.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: 846 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:24.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:24.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: 04:34:23.307500+0000 mon.a (mon. 2021-02-10T04:34:24.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: 0) 1248 : cluster [DBG] osdmap e446: 8 total, 8 up, 8 in 2021-02-10T04:34:24.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:24.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: 04:34:23.469068+0000 osd.4 (osd.4) 2021-02-10T04:34:24.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: 535 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:24.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:24.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: :34:23.469076+0000 osd.4 (osd.4 2021-02-10T04:34:24.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: ) 536 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:24.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:24.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: 04:34:23.469080+0000 osd.4 (osd.4 2021-02-10T04:34:24.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: ) 537 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:24.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:24.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: 10T04:34:23.469085+0000 osd.4 ( 2021-02-10T04:34:24.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: osd.4) 538 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:24.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:24.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: -02-10T04:34:24. 2021-02-10T04:34:24.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: 085286+0000 mgr.y (mgr.14140) 555 : cluster 2021-02-10T04:34:24.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:24 smithi164 conmon[29288]: [DBG] pgmap v819: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 366 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 8.9 MiB/s rd, 503 KiB/s wr, 10 op/s 2021-02-10T04:34:24.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:24 smithi085 conmon[49106]: debug 2021-02-10T04:34:24.643+0000 7f77a3e94700 -1 osd.2 447 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:25.316 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[52836]: debug 2021-02-10T04:34:25.026+0000 7fba71fb2700 -1 osd.3 pg_epoch: 447 pg[4.10( v 62'12 (0'0,62'12] local-lis/les=71/72 n=10 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,5,6] r=0 lpr=71 crt=62'12 lcod 62'11 mlcod 0'0 active+clean] scrubber pg(4.10) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6 (0x55fec06c91e0) 2021-02-10T04:34:25.317 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: cluster 2021-02-10T04:34:23.610949+0000 osd.2 (osd.2) 847 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:25.317 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: 2021-02-10T04:34:25.317 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: cluster 2021-02-10T04:34:23.610957+0000 osd.2 (osd.2) 848 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:25.317 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: cluster 2021-02-10T04:34:23.610964+0000 osd.2 (osd.2) 849 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:25.317 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: cluster 2021-02-10T04:34:23 2021-02-10T04:34:25.317 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: .610970+0000 osd.2 (osd.2) 850 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:25.317 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: 2021-02-10T04:34:25.317 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: cluster 2021-02-10T04:34:25.317 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: 2021-02-10T04:34:23.610976+0000 osd.2 (osd 2021-02-10T04:34:25.317 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: .2) 851 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:25.317 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: cluster 2021-02-10T04:34:23. 2021-02-10T04:34:25.317 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: 610982+0000 osd.2 (osd.2) 852 : cluster [WRN] 2021-02-10T04:34:25.318 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:25.318 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: cluster 2021-02-10T04:34 2021-02-10T04:34:25.318 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: :23.610993+0000 osd.2 (osd.2) 853 2021-02-10T04:34:25.318 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:25.318 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: cluster 2021-02-10T04:34:25.318 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: 2021-02-10T04:34:23.610998 2021-02-10T04:34:25.318 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 854 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:25.318 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:34:25.318 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: T04:34:23.611001+0000 osd.2 (osd. 2021-02-10T04:34:25.318 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: 2) 855 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:25.318 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:25.318 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: 02-10T04:34:23.611004+0000 osd.2 (osd 2021-02-10T04:34:25.319 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: .2) 856 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:25.319 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: cluster 2021-02-10T04:34:24.314456+0000 2021-02-10T04:34:25.319 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: mon.a (mon.0) 1249 : cluster 2021-02-10T04:34:25.319 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: [DBG] osdmap e447: 8 total, 8 up, 8 in 2021-02-10T04:34:25.319 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: cluster 2021-02-10T04:34:25.319 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: 2021-02-10T04:34:24.494432+0000 osd.4 (osd.4) 539 : 2021-02-10T04:34:25.319 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:25.319 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: cluster 2021-02-10T04:34: 2021-02-10T04:34:25.319 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: 24.494443+0000 osd.4 (osd.4) 540 : cluster 2021-02-10T04:34:25.319 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:25.319 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: cluster 2021-02-10T04:34:25.319 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: 2021-02-10T04:34:24. 2021-02-10T04:34:25.319 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: 494455+0000 osd.4 (osd.4) 541 : cluster 2021-02-10T04:34:25.319 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:25.320 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:34:25.320 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: T04:34:24.494462+0000 osd.4 ( 2021-02-10T04:34:25.320 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[29319]: osd.4) 542 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:25.321 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: cluster 2021-02-10T04:34:23.610949 2021-02-10T04:34:25.321 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 847 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:25.321 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: cluster 2021-02-10T04:34:25.321 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: 2021-02-10T04:34:23.610957+0000 osd.2 (osd.2) 848 : cluster 2021-02-10T04:34:25.321 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:25.321 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: cluster 2021-02-10T04:34:23 2021-02-10T04:34:25.321 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: .610964+0000 osd.2 (osd.2) 849 : cluster 2021-02-10T04:34:25.321 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:25.321 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: cluster 2021-02-10T04:34:23. 2021-02-10T04:34:25.321 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: 610970+0000 osd.2 (osd.2) 850 : cluster [WRN] 2021-02-10T04:34:25.322 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:25.322 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: cluster 2021-02-10T04:34:25.322 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: 2021-02-10T04:34:23.610976+0000 osd.2 (osd. 2021-02-10T04:34:25.322 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: 2) 851 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:25.322 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:25.322 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: 10T04:34:23.610982+0000 osd.2 (osd. 2021-02-10T04:34:25.322 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: 2) 852 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:25.322 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:25.322 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: T04:34:23.610993+0000 osd.2 (osd. 2021-02-10T04:34:25.323 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: 2) 853 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:25.323 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:25.323 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: 10T04:34:23.610998+0000 osd.2 (osd. 2021-02-10T04:34:25.323 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: 2) 854 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:25.323 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:25.323 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: -10T04:34:23.611001+0000 osd.2 2021-02-10T04:34:25.323 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: (osd.2) 855 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:25.323 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: 2021-02-10T04:34:25.323 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: cluster 2021-02-10T04:34:25.323 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: 2021-02-10T04:34:23.611004+0000 osd.2 ( 2021-02-10T04:34:25.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: osd.2) 856 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:25.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:25.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: -02-10T04:34:24.314456 2021-02-10T04:34:25.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1249 : cluster 2021-02-10T04:34:25.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: [DBG] osdmap e447: 8 total, 8 up, 8 in 2021-02-10T04:34:25.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:25.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: 34:24.494432+0000 osd.4 (osd.4) 2021-02-10T04:34:25.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: 539 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:25.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:25.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: 10T04:34:24.494443+0000 osd.4 2021-02-10T04:34:25.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: (osd.4) 540 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:25.324 INFO:tasks.rados.rados.0.smithi085.stdout:1698: done (10 left) 2021-02-10T04:34:25.325 INFO:tasks.rados.rados.0.smithi085.stdout:1699: done (9 left) 2021-02-10T04:34:25.325 INFO:tasks.rados.rados.0.smithi085.stdout:1700: rollback oid 12 current snap is 183 2021-02-10T04:34:25.325 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 12 to 123 2021-02-10T04:34:25.325 INFO:tasks.rados.rados.0.smithi085.stdout:1701: read oid 40 snap 182 2021-02-10T04:34:25.325 INFO:tasks.rados.rados.0.smithi085.stdout:1701: expect (ObjNum 491 snap 141 seq_num 491) 2021-02-10T04:34:25.325 INFO:tasks.rados.rados.0.smithi085.stdout:1702: delete oid 30 current snap is 183 2021-02-10T04:34:25.326 INFO:tasks.rados.rados.0.smithi085.stdout:1700: finishing rollback tid 0 to smithi08575397-12 2021-02-10T04:34:25.326 INFO:tasks.rados.rados.0.smithi085.stdout:1700: finishing rollback tid 1 to smithi08575397-12 2021-02-10T04:34:25.326 INFO:tasks.rados.rados.0.smithi085.stdout:1700: finishing rollback tid 2 to smithi08575397-12 2021-02-10T04:34:25.326 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 757 (ObjNum 85 snap 11 seq_num 85) dirty exists 2021-02-10T04:34:25.334 INFO:tasks.rados.rados.0.smithi085.stdout:1701: expect (ObjNum 491 snap 141 seq_num 491) 2021-02-10T04:34:25.491 INFO:tasks.rados.rados.0.smithi085.stdout:1700: done (11 left) 2021-02-10T04:34:25.491 INFO:tasks.rados.rados.0.smithi085.stdout:1701: done (10 left) 2021-02-10T04:34:25.492 INFO:tasks.rados.rados.0.smithi085.stdout:1702: done (9 left) 2021-02-10T04:34:25.492 INFO:tasks.rados.rados.0.smithi085.stdout:1703: copy_from oid 43 from oid 40 current snap is 183 2021-02-10T04:34:25.492 INFO:tasks.rados.rados.0.smithi085.stdout:1704: read oid 1 snap -1 2021-02-10T04:34:25.492 INFO:tasks.rados.rados.0.smithi085.stdout:1704: expect deleted 2021-02-10T04:34:25.492 INFO:tasks.rados.rados.0.smithi085.stdout:1705: copy_from oid 30 from oid 14 current snap is 183 2021-02-10T04:34:25.493 INFO:tasks.rados.rados.0.smithi085.stdout:1706: copy_from oid 28 from oid 21 current snap is 183 2021-02-10T04:34:25.493 INFO:tasks.rados.rados.0.smithi085.stdout:1707: snap_create 2021-02-10T04:34:25.494 INFO:tasks.rados.rados.0.smithi085.stdout:1705: finishing copy_from to smithi08575397-30 2021-02-10T04:34:25.494 INFO:tasks.rados.rados.0.smithi085.stdout:1705: got expected ENOENT (src dne) 2021-02-10T04:34:25.494 INFO:tasks.rados.rados.0.smithi085.stdout:1705: finishing copy_from racing read to smithi08575397-30 2021-02-10T04:34:25.510 INFO:tasks.rados.rados.0.smithi085.stdout:1706: finishing copy_from to smithi08575397-28 2021-02-10T04:34:25.510 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 907 (ObjNum 600 snap 180 seq_num 600) dirty exists 2021-02-10T04:34:25.511 INFO:tasks.rados.rados.0.smithi085.stdout:1706: finishing copy_from racing read to smithi08575397-28 2021-02-10T04:34:25.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[33677]: debug 2021-02-10T04:34:25.444+0000 7fbec19a1700 -1 osd.4 448 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: cluster 2021-02-10T04:34:23.610949+0000 osd.2 (osd.2) 2021-02-10T04:34:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: 847 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: cluster 2021-02-10T04:34:23.610957+0000 osd.2 (osd.2) 848 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: cluster 2021-02-10T04:34:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: 2021-02-10T04:34:23.610964+0000 osd.2 (osd.2) 849 : cluster 2021-02-10T04:34:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:25.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: cluster 2021-02-10T04:34:25.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: 2021-02-10T04:34:23 2021-02-10T04:34:25.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: .610970+0000 osd.2 (osd.2 2021-02-10T04:34:25.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: ) 850 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:25.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:25.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: -02-10T04:34:23.610976+0000 2021-02-10T04:34:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: osd.2 (osd.2) 851 : cluster 2021-02-10T04:34:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: -02-10T04:34:23 2021-02-10T04:34:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: .610982+0000 osd.2 (osd.2 2021-02-10T04:34:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: ) 852 : cluster 2021-02-10T04:34:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: -02-10T04:34:23 2021-02-10T04:34:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: .610993+0000 osd.2 (osd.2 2021-02-10T04:34:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: ) 853 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: cluster 2021-02-10T04:34:25.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: 2021-02-10T04:34:23 2021-02-10T04:34:25.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: .610998+0000 osd.2 (osd.2 2021-02-10T04:34:25.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: ) 854 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:25.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:25.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: -02-10T04:34:23 2021-02-10T04:34:25.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: .611001+0000 osd.2 (osd.2 2021-02-10T04:34:25.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: ) 855 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:25.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:25.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: 02-10T04:34:23.611004+0000 osd.2 (osd.2 2021-02-10T04:34:25.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: ) 856 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:25.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: cluster 2021-02-10T04:34:24.314456+0000 mon.a 2021-02-10T04:34:25.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: (mon.0) 1249 : cluster [DBG] osdmap e447: 8 total, 8 up, 8 in 2021-02-10T04:34:25.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:25.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: 02-10T04:34:24.494432+0000 osd.4 2021-02-10T04:34:25.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: (osd.4) 539 : cluster [WRN] 2021-02-10T04:34:25.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:25.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:25.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: -02-10T04:34:24 2021-02-10T04:34:25.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: .494443+0000 osd.4 (osd.4) 540 : cluster 2021-02-10T04:34:25.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:25.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:25.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: -02-10T04:34: 2021-02-10T04:34:25.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: 24.494455+0000 osd.4 (osd 2021-02-10T04:34:25.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: .4) 541 : cluster [WRN] 2021-02-10T04:34:25.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:25.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: cluster 2021-02-10T04:34:25.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:34:25.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: :34:24.494462+0000 2021-02-10T04:34:25.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: osd.4 (osd.4) 542 : 2021-02-10T04:34:25.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:25 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:25.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: cluster 2021-02-10T04:34 2021-02-10T04:34:25.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: :24.494455+0000 osd.4 (osd.4) 541 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:25.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: cluster 2021-02-10T04:34:24.494462+0000 2021-02-10T04:34:25.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[35961]: osd.4 (osd.4) 542 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:25.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:25 smithi085 conmon[49106]: debug 2021-02-10T04:34:25.689+0000 7f77a3e94700 -1 osd.2 448 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:25.955 INFO:tasks.rados.rados.0.smithi085.stdout:1703: finishing copy_from to smithi08575397-43 2021-02-10T04:34:25.955 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 625 (ObjNum 491 snap 141 seq_num 491) dirty exists 2021-02-10T04:34:25.956 INFO:tasks.rados.rados.0.smithi085.stdout:1703: finishing copy_from racing read to smithi08575397-43 2021-02-10T04:34:26.120 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:34:26.326 INFO:tasks.rados.rados.0.smithi085.stdout:1703: done (13 left) 2021-02-10T04:34:26.326 INFO:tasks.rados.rados.0.smithi085.stdout:1704: done (12 left) 2021-02-10T04:34:26.326 INFO:tasks.rados.rados.0.smithi085.stdout:1705: done (11 left) 2021-02-10T04:34:26.326 INFO:tasks.rados.rados.0.smithi085.stdout:1706: done (10 left) 2021-02-10T04:34:26.326 INFO:tasks.rados.rados.0.smithi085.stdout:1707: done (9 left) 2021-02-10T04:34:26.326 INFO:tasks.rados.rados.0.smithi085.stdout:1708: rollback oid 38 current snap is 184 2021-02-10T04:34:26.327 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 38 to 156 2021-02-10T04:34:26.327 INFO:tasks.rados.rados.0.smithi085.stdout:1709: delete oid 37 current snap is 184 2021-02-10T04:34:26.327 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[52836]: debug 2021-02-10T04:34:26.058+0000 7fba71fb2700 -1 osd.3 pg_epoch: 448 pg[4.10( v 62'12 (0'0,62'12] local-lis/les=71/72 n=10 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,5,6] r=0 lpr=71 crt=62'12 lcod 62'11 mlcod 0'0 active+clean] scrubber pg(4.10) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6 (0x55feb5c809a0) 2021-02-10T04:34:26.335 INFO:tasks.rados.rados.0.smithi085.stdout:1709: done (10 left) 2021-02-10T04:34:26.336 INFO:tasks.rados.rados.0.smithi085.stdout:1710: snap_create 2021-02-10T04:34:26.337 INFO:tasks.rados.rados.0.smithi085.stdout:1708: finishing rollback tid 0 to smithi08575397-38 2021-02-10T04:34:26.337 INFO:tasks.rados.rados.0.smithi085.stdout:1708: finishing rollback tid 1 to smithi08575397-38 2021-02-10T04:34:26.337 INFO:tasks.rados.rados.0.smithi085.stdout:1708: finishing rollback tid 2 to smithi08575397-38 2021-02-10T04:34:26.337 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 754 (ObjNum 530 snap 156 seq_num 530) dirty exists 2021-02-10T04:34:26.366 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:34:26.571 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:34:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:26 smithi164 conmon[29288]: cluster 2021-02-10T04:34:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:26 smithi164 conmon[29288]: 2021-02-10T04:34:24.644214+0000 osd.2 (osd.2) 857 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:26 smithi164 conmon[29288]: cluster 2021-02-10T04:34:24.644222+0000 osd.2 (osd.2) 858 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:26 smithi164 conmon[29288]: cluster 2021-02-10T04:34:24.644229+0000 osd.2 (osd.2) 859 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:26 smithi164 conmon[29288]: cluster 2021-02-10T04:34:24.644235+0000 osd.2 (osd.2) 860 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:26 smithi164 conmon[29288]: cluster 2021-02-10T04:34:24.644241+0000 osd.2 (osd.2) 861 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:26 smithi164 conmon[29288]: cluster 2021-02-10T04:34:24.644246+0000 osd.2 (osd.2) 862 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:26 smithi164 conmon[29288]: cluster 2021-02-10T04:34:24.644254+0000 osd.2 (osd.2) 863 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:26 smithi164 conmon[29288]: cluster 2021-02-10T04:34:24.644262+0000 osd.2 (osd.2) 864 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:26 smithi164 conmon[29288]: cluster 2021-02-10T04:34:24.644268+0000 osd.2 (osd.2) 865 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:26 smithi164 conmon[29288]: cluster 2021-02-10T04:34:24.644273+0000 osd.2 (osd.2) 866 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:26 smithi164 conmon[29288]: cluster 2021-02-10T04:34:25.319734+0000 mon.a (mon.0) 1250 : cluster [DBG] osdmap e448: 8 total, 8 up, 8 in 2021-02-10T04:34:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:26 smithi164 conmon[29288]: cluster 2021-02-10T04:34:25.445326+0000 osd.4 (osd.4) 543 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:26 smithi164 conmon[29288]: cluster 2021-02-10T04:34:25.445334+0000 osd.4 (osd.4) 544 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:26 smithi164 conmon[29288]: cluster 2021-02-10T04:34:25.445341+0000 osd.4 (osd.4) 545 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:26 smithi164 conmon[29288]: cluster 2021-02-10T04:34:25.445347+0000 osd.4 (osd.4) 546 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:26 smithi164 conmon[29288]: cluster 2021-02-10T04:34:26.085810+0000 mgr.y (mgr.14140) 556 : cluster [DBG] pgmap v822: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 356 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.1 MiB/s rd, 1.7 MiB/s wr, 12 op/s 2021-02-10T04:34:26.645 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:26 smithi164 conmon[33677]: debug 2021-02-10T04:34:26.646 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:26 smithi164 conmon[33677]: 2021-02-10T04:34:26.436+0000 7fbec19a1700 -1 osd.4 449 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:26.646 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:26 smithi164 conmon[33677]: 2021-02-10T04:34:26.646 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:26 smithi164 conmon[33677]: debug 2021-02-10T04:34:26.503+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:34:26.678 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:34:26.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: cluster 2021-02-10T04:34:26.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:34:26.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: 34:24.644214+0000 osd.2 (osd.2) 857 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:26.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: cluster 2021-02-10T04:34:24.644222+0000 osd.2 (osd.2) 858 : cluster [WRN] 2021-02-10T04:34:26.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:26.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:34:26.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: :34:24.644229+0000 osd.2 (osd.2 2021-02-10T04:34:26.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: ) 859 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:26.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:26.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: -10T04:34:24.644235+0000 2021-02-10T04:34:26.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: osd.2 (osd.2) 860 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:26.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: 2021-02-10T04:34:26.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:34:26.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: :34:24.644241+0000 osd.2 (osd.2 2021-02-10T04:34:26.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: ) 861 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:26.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:26.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: -02-10T04:34:24.644246+0000 2021-02-10T04:34:26.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: osd.2 (osd.2) 862 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:26.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: 2021-02-10T04:34:26.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: cluster 2021-02-10T04:34:26.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: 2021-02-10T04:34:24.644254+0000 osd.2 ( 2021-02-10T04:34:26.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: osd.2) 863 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:26.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:26.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: 10T04:34:24.644262+0000 osd.2 (osd.2) 864 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:26.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: cluster 2021-02-10T04:34:24. 2021-02-10T04:34:26.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: 644268+0000 osd.2 (osd.2) 865 : cluster [WRN] 2021-02-10T04:34:26.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:26.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: cluster 2021-02-10T04:34: 2021-02-10T04:34:26.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: 24.644273+0000 osd.2 (osd.2) 866 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:26.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:34:26.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: T04:34:25.319734+0000 mon.a (mon.0) 1250 : cluster 2021-02-10T04:34:26.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: [DBG] osdmap e448: 8 total, 8 up, 8 in 2021-02-10T04:34:26.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: cluster 2021-02-10T04:34:26.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: 2021-02-10T04:34:25.445326+0000 osd.4 (osd.4 2021-02-10T04:34:26.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: ) 543 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:26.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:26.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: -02-10T04:34:25.445334+0000 osd.4 (osd. 2021-02-10T04:34:26.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: 4) 544 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:26.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: cluster 2021-02-10T04:34:26.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: 2021-02-10T04:34:25. 2021-02-10T04:34:26.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: 445341+0000 osd.4 (osd.4) 545 : cluster 2021-02-10T04:34:26.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:26.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: cluster 2021-02-10T04:34:25 2021-02-10T04:34:26.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: .445347+0000 osd.4 (osd.4) 546 : cluster [WRN] 2021-02-10T04:34:26.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:26.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: cluster 2021-02-10T04:34:26.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: 2021-02-10T04:34:26.085810+0000 mgr.y (mgr.14140 2021-02-10T04:34:26.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[29319]: ) 556 : cluster [DBG] pgmap v822: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 356 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.1 MiB/s rd, 1.7 MiB/s wr, 12 op/s 2021-02-10T04:34:26.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: cluster 2021-02-10T04:34:26.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: 2021-02 2021-02-10T04:34:26.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: -10T04:34:24.644214+0000 osd.2 (osd.2) 857 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:26.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:26.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: -10T04:34:24.644222+0000 osd.2 (osd.2 2021-02-10T04:34:26.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: ) 858 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:26.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: cluster 2021-02-10T04:34:26.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: 2021-02 2021-02-10T04:34:26.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: -10T04:34:24.644229+0000 osd.2 2021-02-10T04:34:26.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: (osd.2) 859 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:26.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: 2021-02-10T04:34:26.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:26.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: 02-10T04:34:24.644235+0000 2021-02-10T04:34:26.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: osd.2 (osd.2) 860 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:26.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: 2021-02-10T04:34:26.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:26.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: T04:34:24.644241+0000 osd.2 (osd 2021-02-10T04:34:26.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: .2) 861 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:26.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: cluster 2021-02-10T04:34:26.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: 2021-02-10T04:34: 2021-02-10T04:34:26.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: 24.644246+0000 osd.2 (osd.2) 862 2021-02-10T04:34:26.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:26.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: cluster 2021-02-10T04:34:26.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: 2021-02-10T04:34: 2021-02-10T04:34:26.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: 24.644254+0000 osd.2 (osd.2) 863 2021-02-10T04:34:26.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:26.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:26.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: -10T04:34:24.644262+0000 osd.2 2021-02-10T04:34:26.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: (osd.2) 864 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:26.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: 2021-02-10T04:34:26.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:26.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: T04:34:24.644268+0000 osd.2 (osd.2) 865 : cluster 2021-02-10T04:34:26.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:26.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: cluster 2021-02-10T04:34:26.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: 2021-02-10T04:34:24.644273+0000 2021-02-10T04:34:26.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: osd.2 (osd.2) 866 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:26.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: 2021-02-10T04:34:26.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:34:26.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: 04:34:25.319734+0000 mon.a (mon. 2021-02-10T04:34:26.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: 0) 1250 : cluster [DBG] osdmap e448: 8 total, 8 up, 8 in 2021-02-10T04:34:26.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:26.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: -02-10T04:34:25.445326+0000 2021-02-10T04:34:26.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: osd.4 (osd.4) 543 : cluster [WRN] 2021-02-10T04:34:26.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:26.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: cluster 2021-02-10T04:34 2021-02-10T04:34:26.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: :25.445334+0000 osd.4 (osd.4) 2021-02-10T04:34:26.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: 544 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:26.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:26.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: -10T04:34:25.445341+0000 osd.4 2021-02-10T04:34:26.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: (osd.4) 545 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:26.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: 2021-02-10T04:34:26.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: cluster 2021-02-10T04:34:26.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: 2021-02-10T04:34:25.445347+0000 osd.4 ( 2021-02-10T04:34:26.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: osd.4) 546 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:26.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:26.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: 02-10T04:34:26.085810+0000 mgr.y ( 2021-02-10T04:34:26.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[35961]: mgr.14140) 556 : cluster [DBG] pgmap v822: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 356 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.1 MiB/s rd, 1.7 MiB/s wr, 12 op/s 2021-02-10T04:34:26.712 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:34:27.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[49106]: debug 2021-02-10T04:34:26.702+0000 7f77a3e94700 -1 osd.2 449 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:27.191 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[45443]: debug 2021-02-10T04:34:27.191 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:34:26 smithi085 conmon[45443]: 2021-02-10T04:34:26.795+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:34:27.336 INFO:tasks.rados.rados.0.smithi085.stdout:1708: done (10 left) 2021-02-10T04:34:27.336 INFO:tasks.rados.rados.0.smithi085.stdout:1710: done (9 left) 2021-02-10T04:34:27.336 INFO:tasks.rados.rados.0.smithi085.stdout:1711: write (excl) oid 43 current snap is 185 2021-02-10T04:34:27.336 INFO:tasks.rados.rados.0.smithi085.stdout:1711: seq_num 607 ranges {601849=468030,1753864=447985} 2021-02-10T04:34:27.343 INFO:tasks.rados.rados.0.smithi085.stdout:1711: writing smithi08575397-43 from 601849 to 1069879 tid 1 2021-02-10T04:34:27.350 INFO:tasks.rados.rados.0.smithi085.stdout:1711: writing smithi08575397-43 from 1753864 to 2201849 tid 2 2021-02-10T04:34:27.350 INFO:tasks.rados.rados.0.smithi085.stdout:1712: rollback oid 21 current snap is 185 2021-02-10T04:34:27.350 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 21 to 175 2021-02-10T04:34:27.350 INFO:tasks.rados.rados.0.smithi085.stdout:1711: finishing write tid 1 to smithi08575397-43 2021-02-10T04:34:27.350 INFO:tasks.rados.rados.0.smithi085.stdout:1713: write oid 10 current snap is 185 2021-02-10T04:34:27.350 INFO:tasks.rados.rados.0.smithi085.stdout:1713: seq_num 608 ranges {618656=543279,1812346=406310} 2021-02-10T04:34:27.357 INFO:tasks.rados.rados.0.smithi085.stdout:1713: writing smithi08575397-10 from 618656 to 1161935 tid 1 2021-02-10T04:34:27.364 INFO:tasks.rados.rados.0.smithi085.stdout:1713: writing smithi08575397-10 from 1812346 to 2218656 tid 2 2021-02-10T04:34:27.364 INFO:tasks.rados.rados.0.smithi085.stdout:1714: snap_remove snap 176 2021-02-10T04:34:27.393 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:34:27.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[33677]: debug 2021-02-10T04:34:27.462+0000 7fbec19a1700 -1 osd.4 450 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:27.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[33677]: debug 2021-02-10T04:34:27.463+0000 7fbea5924700 -1 osd.4 pg_epoch: 450 pg[6.1a( empty local-lis/les=59/60 n=0 ec=59/59 lis/c=59/59 les/c/f=60/60/0 sis=59) [4,5,1] r=0 lpr=59 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.1a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x562760ecba20) 2021-02-10T04:34:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: cluster 2021-02-10T04:34:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: 2021-02-10T04:34:25. 2021-02-10T04:34:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: 690482+0000 osd.2 (osd.2) 867 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: cluster 2021-02-10T04:34:25.690490+0000 osd.2 2021-02-10T04:34:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: (osd.2) 868 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: cluster 2021-02-10T04:34:25.690497+0000 osd.2 (osd.2) 2021-02-10T04:34:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: 869 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: cluster 2021-02-10T04:34:25.690503+0000 2021-02-10T04:34:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: osd.2 (osd.2) 870 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: 34:25.690511+0000 osd.2 (osd.2) 871 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: cluster 2021-02-10T04:34 2021-02-10T04:34:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: :25.690517+0000 osd.2 (osd.2) 872 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: 02-10T04:34:25.690523+0000 osd.2 (osd.2) 873 : cluster 2021-02-10T04:34:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:27.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: :34:25.690529+0000 osd.2 (osd.2) 874 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: cluster 2021-02-10T04:34:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: 2021-02-10T04:34:25.690535+0000 osd.2 (osd.2) 875 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: 2021-02-10T04:34:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: cluster 2021-02-10T04:34:25 2021-02-10T04:34:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: .690544+0000 osd.2 (osd.2) 876 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: -02-10T04:34:26.325258+0000 mon.a (mon.0) 1251 : cluster [DBG] osdmap e449: 8 total, 8 up, 8 in 2021-02-10T04:34:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: cluster 2021-02-10T04:34:26.437383+0000 osd.4 2021-02-10T04:34:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: (osd.4) 547 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: cluster 2021-02-10T04:34:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: 2021-02-10T04:34:26.437393+0000 osd.4 (osd.4) 548 : cluster [WRN] 2021-02-10T04:34:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:27.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:27.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: -10T04:34:26.437399+0000 osd.4 (osd.4) 549 2021-02-10T04:34:27.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:27.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:27.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:27 smithi164 conmon[29288]: -10T04:34:26.437405+0000 osd.4 (osd.4) 550 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:27.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: cluster 2021-02-10T04:34:25.690482+0000 osd.2 (osd. 2021-02-10T04:34:27.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: 2) 867 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:27.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: cluster 2021-02-10T04:34:25.690490+0000 osd.2 (osd.2) 868 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:27.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: 2021-02-10T04:34:27.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:27.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: 02-10T04:34:25.690497+0000 osd.2 2021-02-10T04:34:27.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: (osd.2) 869 : cluster 2021-02-10T04:34:27.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:27.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:27.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: -10T04:34:25. 2021-02-10T04:34:27.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: 690503+0000 osd.2 (osd.2 2021-02-10T04:34:27.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: ) 870 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:27.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:27.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: 10T04:34:25.690511+0000 osd.2 (osd.2) 871 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:27.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: cluster 2021-02-10T04:34:27.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:34:27.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: T04:34:25.690517+0000 osd.2 ( 2021-02-10T04:34:27.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: osd.2) 872 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:27.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: 2021-02-10T04:34:27.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:27.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: -02-10T04:34:25.690523 2021-02-10T04:34:27.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 873 : cluster 2021-02-10T04:34:27.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:27.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:27.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: 02-10T04:34:25.690529+0000 2021-02-10T04:34:27.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: osd.2 (osd.2) 874 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:27.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: 2021-02-10T04:34:27.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: cluster 2021-02-10T04:34:27.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: 2021-02-10T04:34:25.690535+0000 osd.2 ( 2021-02-10T04:34:27.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: osd.2) 875 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:27.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:27.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: 10T04:34:25.690544+0000 osd.2 (osd. 2021-02-10T04:34:27.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: 2) 876 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:27.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:27.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: 04:34:26.325258+0000 mon.a (mon.0) 1251 : cluster 2021-02-10T04:34:27.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: [DBG] osdmap e449: 8 total, 8 up, 8 in 2021-02-10T04:34:27.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: cluster 2021-02-10T04:34:26.437383 2021-02-10T04:34:27.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 547 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:27.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:34:27.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: 34:26.437393+0000 osd.4 (osd.4 2021-02-10T04:34:27.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: ) 548 : cluster [WRN] 2021-02-10T04:34:27.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:27.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:27.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: -10T04:34:26.437399+0000 osd.4 2021-02-10T04:34:27.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: (osd.4) 549 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:27.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:27.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: -10T04:34:26.437405+0000 osd.4 (osd.4) 550 2021-02-10T04:34:27.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:27.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: cluster 2021-02-10T04:34:27.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: 2021-02-10T04:34: 2021-02-10T04:34:27.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: 25.690482+0000 osd.2 (osd.2) 867 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:27.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: cluster 2021-02-10T04:34:27.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: 2021-02-10T04:34:25.690490+0000 osd.2 (osd.2) 868 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:27.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: cluster 2021-02-10T04:34:27.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:34:27.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: 34:25.690497+0000 osd.2 (osd 2021-02-10T04:34:27.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: .2) 869 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:27.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: cluster 2021-02-10T04:34:27.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:34:27.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: 04:34:25.690503+0000 osd.2 (osd. 2021-02-10T04:34:27.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: 2) 870 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:27.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:27.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: -02-10T04:34:25.690511+0000 osd.2 2021-02-10T04:34:27.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: (osd.2) 871 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:27.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:27.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: 10T04:34:25.690517+0000 osd.2 (osd.2 2021-02-10T04:34:27.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: ) 872 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:27.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: cluster 2021-02-10T04:34:27.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: 2021-02-10T04:34 2021-02-10T04:34:27.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: :25.690523+0000 osd.2 (osd.2) 873 2021-02-10T04:34:27.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:27.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:27.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: T04:34:25.690529+0000 osd.2 ( 2021-02-10T04:34:27.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: osd.2) 874 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:27.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:34:27.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: 04:34:25.690535+0000 osd.2 (osd.2) 875 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:27.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:27.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: 02-10T04:34:25.690544+0000 2021-02-10T04:34:27.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: osd.2 (osd.2) 876 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:27.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:27.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: -10T04:34:26.325258+0000 mon.a (mon. 2021-02-10T04:34:27.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: 0) 1251 : cluster [DBG] osdmap e449: 8 total, 8 up, 8 in 2021-02-10T04:34:27.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: cluster 2021-02-10T04:34: 2021-02-10T04:34:27.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: 26.437383+0000 osd.4 (osd.4) 547 : 2021-02-10T04:34:27.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:27.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:27.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: -02-10T04:34:26.437393+0000 2021-02-10T04:34:27.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: osd.4 (osd.4) 548 : cluster [WRN] 2021-02-10T04:34:27.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:27.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:27.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: T04:34:26.437399+0000 osd.4 (osd.4) 549 2021-02-10T04:34:27.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:27.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: cluster 2021-02-10T04:34: 2021-02-10T04:34:27.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: 26.437405+0000 osd.4 (osd.4) 550 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:27.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[35961]: 2021-02-10T04:34:28.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:27 smithi085 conmon[49106]: debug 2021-02-10T04:34:27.697+0000 7f77a3e94700 -1 osd.2 450 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:28.340 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:34:28.341 INFO:tasks.rados.rados.0.smithi085.stdout:1712: finishing rollback tid 0 to smithi08575397-21 2021-02-10T04:34:28.341 INFO:tasks.rados.rados.0.smithi085.stdout:1712: finishing rollback tid 1 to smithi08575397-21 2021-02-10T04:34:28.341 INFO:tasks.rados.rados.0.smithi085.stdout:1711: finishing write tid 2 to smithi08575397-43 2021-02-10T04:34:28.341 INFO:tasks.rados.rados.0.smithi085.stdout:1712: finishing rollback tid 2 to smithi08575397-21 2021-02-10T04:34:28.341 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 921 (ObjNum 431 snap 114 seq_num 431) dirty exists 2021-02-10T04:34:28.342 INFO:tasks.rados.rados.0.smithi085.stdout:1711: finishing write tid 3 to smithi08575397-43 2021-02-10T04:34:28.342 INFO:tasks.rados.rados.0.smithi085.stdout:1711: finishing write tid 4 to smithi08575397-43 2021-02-10T04:34:28.342 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 706 (ObjNum 607 snap 185 seq_num 607) dirty exists 2021-02-10T04:34:28.342 INFO:tasks.rados.rados.0.smithi085.stdout:1711: left oid 43 (ObjNum 607 snap 185 seq_num 607) 2021-02-10T04:34:28.342 INFO:tasks.rados.rados.0.smithi085.stdout:1713: finishing write tid 1 to smithi08575397-10 2021-02-10T04:34:28.342 INFO:tasks.rados.rados.0.smithi085.stdout:1713: finishing write tid 2 to smithi08575397-10 2021-02-10T04:34:28.342 INFO:tasks.rados.rados.0.smithi085.stdout:1713: finishing write tid 3 to smithi08575397-10 2021-02-10T04:34:28.342 INFO:tasks.rados.rados.0.smithi085.stdout:1713: finishing write tid 4 to smithi08575397-10 2021-02-10T04:34:28.342 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 10 v 637 (ObjNum 608 snap 185 seq_num 608) dirty exists 2021-02-10T04:34:28.342 INFO:tasks.rados.rados.0.smithi085.stdout:1713: left oid 10 (ObjNum 608 snap 185 seq_num 608) 2021-02-10T04:34:28.342 INFO:tasks.rados.rados.0.smithi085.stdout:1711: done (12 left) 2021-02-10T04:34:28.342 INFO:tasks.rados.rados.0.smithi085.stdout:1712: done (11 left) 2021-02-10T04:34:28.342 INFO:tasks.rados.rados.0.smithi085.stdout:1713: done (10 left) 2021-02-10T04:34:28.343 INFO:tasks.rados.rados.0.smithi085.stdout:1714: done (9 left) 2021-02-10T04:34:28.343 INFO:tasks.rados.rados.0.smithi085.stdout:1715: snap_remove snap 153 2021-02-10T04:34:28.343 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: cluster 2021-02-10T04:34:26.703199+0000 osd.2 (osd. 2021-02-10T04:34:28.343 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: 2) 877 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:28.343 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: cluster 2021-02-10T04:34:26.703209+0000 osd.2 (osd.2) 878 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:28.343 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:28.343 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: -10T04:34:26.703216+0000 osd.2 (osd.2) 879 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:28.343 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: cluster 2021-02-10T04:34:26 2021-02-10T04:34:28.343 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: .703223+0000 osd.2 (osd.2) 880 : cluster 2021-02-10T04:34:28.343 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:28.343 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: cluster 2021-02-10T04:34:26. 2021-02-10T04:34:28.343 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: 703229+0000 osd.2 (osd.2) 881 : cluster 2021-02-10T04:34:28.344 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:28.344 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: cluster 2021-02-10T04:34:26 2021-02-10T04:34:28.344 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: .703234+0000 osd.2 (osd.2) 882 : cluster 2021-02-10T04:34:28.344 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:28.345 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: cluster 2021-02-10T04:34:28.345 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: 2021-02-10T04:34:26.703242+0000 osd.2 (osd. 2021-02-10T04:34:28.345 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: 2) 883 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:28.345 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: cluster 2021-02-10T04:34 2021-02-10T04:34:28.345 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: :26.703247+0000 osd.2 (osd.2) 884 2021-02-10T04:34:28.345 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:28.345 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:28.345 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: :34:26.703253+0000 osd.2 (osd.2 2021-02-10T04:34:28.345 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: ) 885 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:28.346 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:28.346 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: 04:34:26.703259+0000 osd.2 (osd. 2021-02-10T04:34:28.346 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: 2) 886 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:28.346 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:28.346 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: 10T04:34:27.335731+0000 mon.a (mon 2021-02-10T04:34:28.346 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: .0) 1252 : cluster [DBG] osdmap e450: 8 total, 8 up, 8 in 2021-02-10T04:34:28.346 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:28.346 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: 02-10T04:34:27.463445+0000 osd.4 2021-02-10T04:34:28.346 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: (osd.4) 551 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:28.346 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:28.346 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: 02-10T04:34:27.463453+0000 osd.4 2021-02-10T04:34:28.346 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: (osd.4) 552 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:28.347 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:28.347 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: 10T04:34:27.463460+0000 osd.4 (osd 2021-02-10T04:34:28.347 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: .4) 553 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:28.347 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: cluster 2021-02-10T04:34:28.347 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: 2021-02-10T04:34:27.463466+0000 2021-02-10T04:34:28.347 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: osd.4 (osd.4) 554 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:28.347 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:28.347 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: -02-10T04:34:27.940616+0000 osd.3 2021-02-10T04:34:28.347 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: (osd.3) 66 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:34:28.347 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:28.347 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: 02-10T04:34:27.940970+0000 osd.3 ( 2021-02-10T04:34:28.347 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: osd.3) 67 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:34:28.347 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: audit 2021-02-10T04:34:28.347 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:34:28.348 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: 04:34:28.026034+0000 mgr.y (mgr.14140) 557 2021-02-10T04:34:28.348 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:34:28.348 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:28.348 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: 02-10T04:34:28.086252+0000 mgr.y (mgr 2021-02-10T04:34:28.348 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[29288]: .14140) 558 : cluster [DBG] pgmap v825: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 356 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 10 op/s 2021-02-10T04:34:28.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:28 smithi164 conmon[33677]: debug 2021-02-10T04:34:28.458+0000 7fbec19a1700 -1 osd.4 451 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:28.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: cluster 2021-02-10T04:34:28.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: 2021-02-10T04:34:28.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: 2021 2021-02-10T04:34:28.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: - 2021-02-10T04:34:28.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: 02 2021-02-10T04:34:28.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: - 2021-02-10T04:34:28.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: 10 2021-02-10T04:34:28.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: T 2021-02-10T04:34:28.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: 04 2021-02-10T04:34:28.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: : 2021-02-10T04:34:28.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: 34 2021-02-10T04:34:28.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: : 2021-02-10T04:34:28.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: 26 2021-02-10T04:34:28.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: . 2021-02-10T04:34:28.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: 703199 2021-02-10T04:34:28.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: +0000 2021-02-10T04:34:28.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: 2021-02-10T04:34:28.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: osd.2 2021-02-10T04:34:28.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: (osd.2) 877 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:28.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:28.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: 04:34:26.703209+0000 osd.2 (osd.2) 878 : cluster 2021-02-10T04:34:28.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:28.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: cluster 2021-02-10T04:34:26.703216 2021-02-10T04:34:28.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 879 : cluster [WRN] 2021-02-10T04:34:28.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:28.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: cluster 2021-02-10T04:34:28.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: 2021-02-10T04:34:26.703223+0000 osd.2 (osd.2 2021-02-10T04:34:28.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: ) 880 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:28.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: cluster 2021-02-10T04:34:28.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: 2021-02-10T04:34:26.703229 2021-02-10T04:34:28.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 881 2021-02-10T04:34:28.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:28.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:34:28.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: 34:26.703234+0000 osd.2 (osd.2) 882 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:28.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: 2021-02-10T04:34:28.687 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:28.687 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: -02-10T04:34:26.703242+0000 osd.2 (osd. 2021-02-10T04:34:28.687 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: 2) 883 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:28.687 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: cluster 2021-02-10T04:34:28.687 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: 2021-02-10T04:34:26. 2021-02-10T04:34:28.687 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: 703247+0000 osd.2 (osd.2) 884 : cluster 2021-02-10T04:34:28.687 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:28.687 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:34:28.687 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: 34:26.703253+0000 2021-02-10T04:34:28.687 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: osd.2 (osd.2) 885 : cluster [WRN] 2021-02-10T04:34:28.687 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:28.687 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: cluster 2021-02-10T04:34: 2021-02-10T04:34:28.688 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: 26.703259+0000 osd.2 (osd.2) 886 2021-02-10T04:34:28.688 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:28.688 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: cluster 2021-02-10T04:34:28.688 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: 2021-02-10T04:34:27.335731 2021-02-10T04:34:28.688 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1252 : cluster [DBG] 2021-02-10T04:34:28.688 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: osdmap e450: 8 total, 8 up, 8 in 2021-02-10T04:34:28.688 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: cluster 2021-02-10T04:34:28.688 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: 2021-02-10T04:34:27.463445+0000 osd.4 ( 2021-02-10T04:34:28.688 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: osd.4) 551 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:28.688 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:28.688 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: 04:34:27.463453+0000 osd.4 (osd.4) 552 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:28.688 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:28.689 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: 10T04:34:27.463460+0000 osd.4 ( 2021-02-10T04:34:28.689 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: osd.4) 553 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:28.689 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:28.689 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: 10T04:34:27.463466+0000 osd.4 ( 2021-02-10T04:34:28.689 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: osd.4) 554 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:28.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:28.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: 04:34:27.940616+0000 osd.3 (osd.3 2021-02-10T04:34:28.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: ) 66 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:34:28.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:34:28.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: 34:27.940970+0000 osd.3 (osd.3) 67 : 2021-02-10T04:34:28.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:34:28.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: audit 2021-02-10T04:34: 2021-02-10T04:34:28.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: 28.026034+0000 mgr.y (mgr.14140) 557 : audit 2021-02-10T04:34:28.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:34:28.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: cluster 2021-02-10T04:34:28.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: 2021-02-10T04:34:28.086252+0000 mgr.y (mgr. 2021-02-10T04:34:28.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[29319]: 14140) 558 : cluster [DBG] pgmap v825: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 356 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 10 op/s 2021-02-10T04:34:28.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: cluster 2021-02-10T04:34:28.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: 2021-02-10T04:34:28.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: 2021 2021-02-10T04:34:28.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: -02-10T04:34:26.703199+0000 osd.2 (osd 2021-02-10T04:34:28.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: .2) 877 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:28.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:28.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: -02-10T04:34:26.703209+0000 osd.2 (osd.2) 878 : cluster [WRN] 2021-02-10T04:34:28.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:28.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: cluster 2021-02-10T04:34:26.703216+0000 osd.2 (osd.2) 2021-02-10T04:34:28.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: 879 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:28.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: cluster 2021-02-10T04:34: 2021-02-10T04:34:28.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: 26.703223+0000 osd.2 (osd.2) 880 : cluster 2021-02-10T04:34:28.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:28.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:28.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: -02-10T04:34:26.703229+0000 osd.2 (osd.2) 881 : 2021-02-10T04:34:28.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:28.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: cluster 2021-02-10T04:34:26.703234 2021-02-10T04:34:28.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 882 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:28.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:28.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: -02-10T04:34:26.703242+0000 osd.2 (osd.2) 883 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:28.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: 2021-02-10T04:34:28.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: cluster 2021-02-10T04:34:26.703247 2021-02-10T04:34:28.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 884 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:28.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: cluster 2021-02-10T04:34: 2021-02-10T04:34:28.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: 26.703253+0000 osd.2 (osd.2) 885 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:28.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: cluster 2021-02-10T04:34:28.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: 2021-02-10T04:34:26.703259+0000 2021-02-10T04:34:28.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: osd.2 (osd.2) 886 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:28.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:28.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: -10T04:34:27.335731+0000 mon.a (mon. 2021-02-10T04:34:28.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: 0) 1252 : cluster [DBG] osdmap e450: 8 total, 8 up, 8 in 2021-02-10T04:34:28.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: cluster 2021-02-10T04:34:27.463445+0000 osd.4 (osd 2021-02-10T04:34:28.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: .4) 551 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: cluster 2021-02-10T04:34:27.463453 2021-02-10T04:34:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 552 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: 02-10T04:34:27.463460+0000 osd.4 (osd.4) 553 : cluster 2021-02-10T04:34:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: 34:27.463466+0000 osd.4 (osd.4) 554 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: -10T04:34:27.940616+0000 osd.3 (osd 2021-02-10T04:34:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: .3) 66 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:34:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: cluster 2021-02-10T04:34:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: 2021-02-10T04:34:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: 2021-02-10T04:34:27.940970+0000 2021-02-10T04:34:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: osd.3 (osd.3) 67 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:34:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:34:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: 02-10T04:34:28.026034+0000 mgr.y (mgr.14140) 557 : audit 2021-02-10T04:34:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:34:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[35961]: 34:28.086252+0000 mgr.y (mgr.14140) 558 : cluster [DBG] pgmap v825: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 356 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 10 op/s 2021-02-10T04:34:28.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:28 smithi085 conmon[49106]: debug 2021-02-10T04:34:28.681+0000 7f77a3e94700 -1 osd.2 450 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:29.341 INFO:tasks.rados.rados.0.smithi085.stdout:1715: done (9 left) 2021-02-10T04:34:29.341 INFO:tasks.rados.rados.0.smithi085.stdout:1716: write (excl) oid 2 current snap is 185 2021-02-10T04:34:29.341 INFO:tasks.rados.rados.0.smithi085.stdout:1716: seq_num 609 ranges {635463=734881,1987181=248282} 2021-02-10T04:34:29.350 INFO:tasks.rados.rados.0.smithi085.stdout:1716: writing smithi08575397-2 from 635463 to 1370344 tid 1 2021-02-10T04:34:29.355 INFO:tasks.rados.rados.0.smithi085.stdout:1716: writing smithi08575397-2 from 1987181 to 2235463 tid 2 2021-02-10T04:34:29.356 INFO:tasks.rados.rados.0.smithi085.stdout:1717: read oid 43 snap -1 2021-02-10T04:34:29.356 INFO:tasks.rados.rados.0.smithi085.stdout:1717: expect (ObjNum 607 snap 185 seq_num 607) 2021-02-10T04:34:29.356 INFO:tasks.rados.rados.0.smithi085.stdout:1718: snap_create 2021-02-10T04:34:29.357 INFO:tasks.rados.rados.0.smithi085.stdout:1716: finishing write tid 1 to smithi08575397-2 2021-02-10T04:34:29.360 INFO:tasks.rados.rados.0.smithi085.stdout:1716: finishing write tid 2 to smithi08575397-2 2021-02-10T04:34:29.361 INFO:tasks.rados.rados.0.smithi085.stdout:1716: finishing write tid 3 to smithi08575397-2 2021-02-10T04:34:29.362 INFO:tasks.rados.rados.0.smithi085.stdout:1716: finishing write tid 4 to smithi08575397-2 2021-02-10T04:34:29.362 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 640 (ObjNum 609 snap 185 seq_num 609) dirty exists 2021-02-10T04:34:29.362 INFO:tasks.rados.rados.0.smithi085.stdout:1716: left oid 2 (ObjNum 609 snap 185 seq_num 609) 2021-02-10T04:34:29.363 INFO:tasks.rados.rados.0.smithi085.stdout:1717: expect (ObjNum 607 snap 185 seq_num 607) 2021-02-10T04:34:29.379 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:34:29.414 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:34:29.549 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: cluster 2021-02-10T04:34:29.549 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 2021-02-10T04:34:29.549 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 2021-02-10T04:34: 2021-02-10T04:34:29.549 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 27.697708+0000 osd.2 (osd.2) 887 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:29.549 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: cluster 2021-02-10T04:34:29.549 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 2021 2021-02-10T04:34:29.549 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: - 2021-02-10T04:34:29.549 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 02 2021-02-10T04:34:29.549 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: - 2021-02-10T04:34:29.549 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 10 2021-02-10T04:34:29.549 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: T 2021-02-10T04:34:29.550 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 04 2021-02-10T04:34:29.550 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: : 2021-02-10T04:34:29.550 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 34 2021-02-10T04:34:29.550 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: : 2021-02-10T04:34:29.550 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 27 2021-02-10T04:34:29.550 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: . 2021-02-10T04:34:29.550 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 697718 2021-02-10T04:34:29.550 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: +0000 2021-02-10T04:34:29.550 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 2021-02-10T04:34:29.550 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: osd.2 2021-02-10T04:34:29.550 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: ( 2021-02-10T04:34:29.550 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: osd 2021-02-10T04:34:29.551 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: . 2021-02-10T04:34:29.551 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 2 2021-02-10T04:34:29.551 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: ) 2021-02-10T04:34:29.551 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 888 2021-02-10T04:34:29.551 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: : 2021-02-10T04:34:29.551 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: cluster 2021-02-10T04:34:29.551 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 2021-02-10T04:34:29.551 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: [WRN] 2021-02-10T04:34:29.551 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 2021-02-10T04:34:29.551 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:29.551 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: cluster 2021-02-10T04:34:27.697724+0000 osd.2 (osd.2) 889 : cluster 2021-02-10T04:34:29.551 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:29.551 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: cluster 2021-02-10T04:34:29.551 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 2021-02-10T04:34:27.697730+0000 osd.2 (osd. 2021-02-10T04:34:29.552 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 2) 890 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:29.552 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:29.552 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: -02-10T04:34: 2021-02-10T04:34:29.552 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 27.697737+0000 osd.2 (osd.2) 891 : cluster 2021-02-10T04:34:29.552 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:29.552 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:29.552 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 02-10T04:34:27.697743+0000 osd.2 (osd.2) 2021-02-10T04:34:29.552 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 892 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:29.552 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:29.552 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 02-10T04:34:27.697749+0000 osd.2 (osd.2 2021-02-10T04:34:29.553 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: cluster 2021-02-10T04:34:29.553 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: 2021-02-10T04:34:29.553 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: 2021 2021-02-10T04:34:29.553 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: - 2021-02-10T04:34:29.553 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: 02 2021-02-10T04:34:29.553 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: -10T04:34:27 2021-02-10T04:34:29.554 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: .697708+0000 osd.2 (osd.2) 887 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:29.554 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:29.554 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: -10T04:34:27.697718+0000 osd.2 (osd.2) 888 2021-02-10T04:34:29.554 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:29.554 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:29.554 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: 02-10T04:34:27.697724+0000 osd.2 ( 2021-02-10T04:34:29.554 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: osd.2) 889 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:29.554 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:29.554 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: -02-10T04:34:27. 2021-02-10T04:34:29.554 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: 697730+0000 osd.2 (osd.2) 890 : cluster 2021-02-10T04:34:29.554 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:29.554 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:29.555 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: -10T04:34:27.697737+0000 osd.2 (osd 2021-02-10T04:34:29.555 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: .2) 891 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:29.555 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:29.555 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: -10T04:34:27.697743+0000 osd.2 (osd.2) 892 : 2021-02-10T04:34:29.555 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:29.555 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: cluster 2021-02-10T04:34:27.697749 2021-02-10T04:34:29.555 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 893 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:29.555 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:29.555 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: -02-10T04:34:27.697753+0000 osd.2 2021-02-10T04:34:29.555 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: (osd.2) 894 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:29.555 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: 2021-02-10T04:34:29.555 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: cluster 2021-02-10T04:34:29.555 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:34:29.555 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: 34:27.697756+0000 osd.2 (osd 2021-02-10T04:34:29.556 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: .2) 895 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:29.556 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:29.556 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: -10T04:34:27.697760+0000 osd.2 ( 2021-02-10T04:34:29.556 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: osd.2) 896 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:29.556 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:29.556 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: -02-10T04:34:28.327880 2021-02-10T04:34:29.556 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1253 2021-02-10T04:34:29.556 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: : cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 297 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:34:29.556 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:29.556 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: -02-10T04:34:28.340938+0000 mon.a ( 2021-02-10T04:34:29.556 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: mon.0) 1254 : cluster [DBG] osdmap e451: 8 total, 8 up, 8 in 2021-02-10T04:34:29.556 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:29.556 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: 02-10T04:34:28.459595+0000 osd.4 (osd. 2021-02-10T04:34:29.557 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: 4) 555 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:29.557 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:29.557 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: 02-10T04:34:28.459601+0000 osd.4 (osd. 2021-02-10T04:34:29.557 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: 4) 556 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:29.557 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: cluster 2021-02-10T04:34:29.557 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: 2021-02-10T04:34:28. 2021-02-10T04:34:29.557 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: 459605+0000 osd.4 (osd.4) 557 : cluster 2021-02-10T04:34:29.557 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:29.557 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:29.557 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: 02-10T04:34:28.459612+0000 osd.4 (osd 2021-02-10T04:34:29.557 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[29319]: .4) 558 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:29.557 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: ) 893 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:29.557 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:29.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 02-10T04:34: 2021-02-10T04:34:29.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 27.697753+0000 osd.2 (osd.2) 894 : cluster 2021-02-10T04:34:29.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:29.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: cluster 2021-02-10T04:34:29.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:34:29.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: T04:34:27.697756+0000 osd.2 (osd.2) 895 : 2021-02-10T04:34:29.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:29.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:29.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 02-10T04:34:27.697760+0000 osd.2 (osd. 2021-02-10T04:34:29.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 2) 896 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:29.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:29.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 02-10T04:34 2021-02-10T04:34:29.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: :28.327880+0000 mon.a ( 2021-02-10T04:34:29.559 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: mon.0) 1253 : cluster [WRN] 2021-02-10T04:34:29.559 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: Health check update: 14 slow ops, oldest one blocked for 297 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:34:29.559 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:29.559 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: T04:34:28.340938+0000 mon.a ( 2021-02-10T04:34:29.559 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: mon.0) 1254 : cluster [DBG] 2021-02-10T04:34:29.559 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: osdmap e451: 8 total, 8 up, 8 in 2021-02-10T04:34:29.559 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:29.559 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 02-10T04:34:28.459595+0000 2021-02-10T04:34:29.559 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: osd.4 (osd.4) 555 2021-02-10T04:34:29.559 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:29.559 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:29.559 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 02-10T04:34:28.459601+0000 2021-02-10T04:34:29.559 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: osd.4 (osd.4) 556 : 2021-02-10T04:34:29.559 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:29.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: cluster 2021-02-10T04:34:29.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 2021-02-10T04:34:29.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 2021 2021-02-10T04:34:29.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: - 2021-02-10T04:34:29.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 02 2021-02-10T04:34:29.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: - 2021-02-10T04:34:29.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 10 2021-02-10T04:34:29.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: T 2021-02-10T04:34:29.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 04 2021-02-10T04:34:29.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: : 2021-02-10T04:34:29.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 34 2021-02-10T04:34:29.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: : 2021-02-10T04:34:29.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 28 2021-02-10T04:34:29.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: . 2021-02-10T04:34:29.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 459605 2021-02-10T04:34:29.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: +0000 2021-02-10T04:34:29.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 2021-02-10T04:34:29.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: osd.4 2021-02-10T04:34:29.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: ( 2021-02-10T04:34:29.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: osd 2021-02-10T04:34:29.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: . 2021-02-10T04:34:29.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 4 2021-02-10T04:34:29.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: ) 2021-02-10T04:34:29.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 557 2021-02-10T04:34:29.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: : 2021-02-10T04:34:29.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: cluster 2021-02-10T04:34:29.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 2021-02-10T04:34:29.562 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: [WRN] 2021-02-10T04:34:29.562 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 2021-02-10T04:34:29.562 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:29.562 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 2021-02-10T04:34:29.562 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: cluster 2021-02-10T04:34:29.562 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 2021-02-10T04:34:29.562 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 2021 2021-02-10T04:34:29.562 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: - 2021-02-10T04:34:29.562 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 02-10T04:34:28. 2021-02-10T04:34:29.562 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[35961]: 459612+0000 osd.4 (osd.4) 558 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:29.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[33677]: debug 2021-02-10T04:34:29.485+0000 7fbec19a1700 -1 osd.4 451 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: cluster 2021-02-10T04:34:27.697708+0000 osd.2 (osd.2) 887 2021-02-10T04:34:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: cluster 2021-02-10T04:34:27.697718+0000 osd.2 (osd.2) 888 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: 34:27.697724+0000 osd.2 (osd.2) 889 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: cluster 2021-02-10T04:34:27. 2021-02-10T04:34:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: 697730+0000 osd.2 (osd.2) 890 : cluster 2021-02-10T04:34:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: cluster 2021-02-10T04:34: 2021-02-10T04:34:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: 27.697737+0000 osd.2 (osd.2) 891 : 2021-02-10T04:34:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: 34:27.697743+0000 osd.2 (osd.2) 2021-02-10T04:34:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: 892 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: 04:34:27.697749+0000 osd.2 (osd 2021-02-10T04:34:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: .2) 893 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: 2021-02-10T04:34:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: 04:34:27.697753+0000 2021-02-10T04:34:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: osd.2 (osd.2) 894 : 2021-02-10T04:34:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: -10T04:34:27.697756 2021-02-10T04:34:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 895 : cluster [WRN] 2021-02-10T04:34:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: 02-10T04:34:27.697760+0000 2021-02-10T04:34:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: osd.2 (osd.2) 896 : cluster [WRN] 2021-02-10T04:34:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: 04:34:28.327880+0000 2021-02-10T04:34:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: mon.a (mon.0) 1253 : cluster 2021-02-10T04:34:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: [WRN] Health check update: 14 slow ops, oldest one blocked for 297 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:34:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: 34:28.340938+0000 mon.a (mon.0) 1254 2021-02-10T04:34:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: : cluster [DBG] osdmap e451: 8 total, 8 up, 8 in 2021-02-10T04:34:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: 04:34:28.459595+0000 osd.4 (osd. 2021-02-10T04:34:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: 4) 555 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: 10T04:34:28.459601+0000 osd.4 ( 2021-02-10T04:34:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: osd.4) 556 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:29.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: 02-10T04:34:28.459605+0000 osd.4 (osd.4) 557 : cluster [WRN] 2021-02-10T04:34:29.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:29.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: cluster 2021-02-10T04:34:29.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: 2021-02-10T04:34:28.459612+0000 osd.4 ( 2021-02-10T04:34:29.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:29 smithi164 conmon[29288]: osd.4) 558 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:29.715 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.3 dump_blocked_ops 2021-02-10T04:34:29.872 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:34:29.876 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[49106]: debug 2021-02-10T04:34:29.547+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:34:29.877 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:29 smithi085 conmon[49106]: debug 2021-02-10T04:34:29.676+0000 7f77a3e94700 -1 osd.2 452 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:29.978 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:34:30.026 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:34:30.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[52836]: debug 2021-02-10T04:34:30.069+0000 7fba75fba700 -1 osd.3 pg_epoch: 452 pg[4.10( v 62'12 (0'0,62'12] local-lis/les=71/72 n=10 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,5,6] r=0 lpr=71 crt=62'12 lcod 62'11 mlcod 0'0 active+clean] scrubber pg(4.10) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6 (0x55fec06f54a0) 2021-02-10T04:34:30.342 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:34:30.371 INFO:tasks.rados.rados.0.smithi085.stdout:1716: done (11 left) 2021-02-10T04:34:30.371 INFO:tasks.rados.rados.0.smithi085.stdout:1717: done (10 left) 2021-02-10T04:34:30.371 INFO:tasks.rados.rados.0.smithi085.stdout:1718: done (9 left) 2021-02-10T04:34:30.371 INFO:tasks.rados.rados.0.smithi085.stdout:1719: rollback oid 40 current snap is 186 2021-02-10T04:34:30.371 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 40 to 148 2021-02-10T04:34:30.371 INFO:tasks.rados.rados.0.smithi085.stdout:1720: delete oid 3 current snap is 186 2021-02-10T04:34:30.374 INFO:tasks.rados.rados.0.smithi085.stdout:1719: finishing rollback tid 0 to smithi08575397-40 2021-02-10T04:34:30.374 INFO:tasks.rados.rados.0.smithi085.stdout:1719: finishing rollback tid 1 to smithi08575397-40 2021-02-10T04:34:30.374 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 926 (ObjNum 6 snap 3 seq_num 7) dirty dne 2021-02-10T04:34:30.378 INFO:tasks.rados.rados.0.smithi085.stdout:1719: done (10 left) 2021-02-10T04:34:30.378 INFO:tasks.rados.rados.0.smithi085.stdout:1720: done (9 left) 2021-02-10T04:34:30.378 INFO:tasks.rados.rados.0.smithi085.stdout:1721: read oid 38 snap 184 2021-02-10T04:34:30.378 INFO:tasks.rados.rados.0.smithi085.stdout:1721: expect (ObjNum 530 snap 156 seq_num 530) 2021-02-10T04:34:30.382 INFO:tasks.rados.rados.0.smithi085.stdout:1722: write (excl) oid 30 current snap is 186 2021-02-10T04:34:30.382 INFO:tasks.rados.rados.0.smithi085.stdout:1722: seq_num 611 ranges {669077=485379,1587792=681285} 2021-02-10T04:34:30.385 INFO:tasks.rados.rados.0.smithi085.stdout:1722: writing smithi08575397-30 from 669077 to 1154456 tid 1 2021-02-10T04:34:30.393 INFO:tasks.rados.rados.0.smithi085.stdout:1722: writing smithi08575397-30 from 1587792 to 2269077 tid 2 2021-02-10T04:34:30.393 INFO:tasks.rados.rados.0.smithi085.stdout:1723: delete oid 28 current snap is 186 2021-02-10T04:34:30.393 INFO:tasks.rados.rados.0.smithi085.stdout:1721: expect (ObjNum 530 snap 156 seq_num 530) 2021-02-10T04:34:30.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[29319]: cluster 2021-02-10T04:34:28.681579 2021-02-10T04:34:30.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 897 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:30.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[29319]: cluster 2021-02-10T04:34:28.681593+0000 osd.2 (osd.2) 898 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:30.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[29319]: cluster 2021-02-10T04:34:28.681600+0000 osd.2 (osd.2) 899 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:30.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[29319]: cluster 2021-02-10T04:34:28.681608+0000 osd.2 (osd.2) 900 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:30.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[29319]: cluster 2021-02-10T04:34:28.681613+0000 osd.2 (osd.2) 901 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:30.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[29319]: cluster 2021-02-10T04:34:28.681619+0000 osd.2 (osd.2) 902 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:30.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[29319]: cluster 2021-02-10T04:34:28.681625+0000 osd.2 (osd.2) 903 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:30.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[29319]: cluster 2021-02-10T04:34:28.681630+0000 osd.2 (osd.2) 904 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:30.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[29319]: cluster 2021-02-10T04:34:28.681639+0000 osd.2 (osd.2) 905 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:30.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[29319]: cluster 2021-02-10T04:34:28.681644+0000 osd.2 (osd.2) 906 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:30.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[29319]: cluster 2021-02-10T04:34:29.339419+0000 mon.a (mon.0) 1255 : cluster [DBG] osdmap e452: 8 total, 8 up, 8 in 2021-02-10T04:34:30.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[29319]: cluster 2021-02-10T04:34:29.486305+0000 osd.4 (osd.4) 559 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:30.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[29319]: cluster 2021-02-10T04:34:29.486314+0000 osd.4 (osd.4) 560 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:30.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[29319]: cluster 2021-02-10T04:34:29. 2021-02-10T04:34:30.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[29319]: 486320+0000 osd.4 (osd.4) 561 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:30.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[29319]: cluster 2021-02-10T04:34:29.486326+0000 osd.4 (osd.4) 562 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:30.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[29319]: cluster 2021-02-10T04:34:30.086645+0000 mgr.y (mgr.14140 2021-02-10T04:34:30.636 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[29319]: ) 559 : cluster [DBG] pgmap v828: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 361 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 0 B/s wr, 3 op/s 2021-02-10T04:34:30.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: cluster 2021-02-10T04:34:30.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 2021-02-10T04:34 2021-02-10T04:34:30.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: :28.681579+0000 osd.2 (osd.2) 897 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:30.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: cluster 2021-02-10T04:34:30.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 2021-02-10T04:34:30.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 2021-02-10T04:34 2021-02-10T04:34:30.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: :28.681593+0000 osd.2 (osd.2) 898 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:30.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: cluster 2021-02-10T04:34:30.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 2021-02-10T04:34:30.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 2021 2021-02-10T04:34:30.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: - 2021-02-10T04:34:30.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 02 2021-02-10T04:34:30.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: - 2021-02-10T04:34:30.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 10 2021-02-10T04:34:30.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: T 2021-02-10T04:34:30.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 04 2021-02-10T04:34:30.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: : 2021-02-10T04:34:30.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 34 2021-02-10T04:34:30.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: : 2021-02-10T04:34:30.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 28 2021-02-10T04:34:30.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: . 2021-02-10T04:34:30.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 681600 2021-02-10T04:34:30.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: +0000 2021-02-10T04:34:30.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 2021-02-10T04:34:30.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: osd.2 2021-02-10T04:34:30.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: ( 2021-02-10T04:34:30.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: osd 2021-02-10T04:34:30.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: . 2021-02-10T04:34:30.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 2 2021-02-10T04:34:30.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: ) 2021-02-10T04:34:30.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 899 2021-02-10T04:34:30.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: : 2021-02-10T04:34:30.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: cluster 2021-02-10T04:34:30.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 2021-02-10T04:34:30.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: [WRN] 2021-02-10T04:34:30.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 2021-02-10T04:34:30.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:30.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 2021-02-10T04:34:30.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: cluster 2021-02-10T04:34:30.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 2021-02-10T04:34:30.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 2021 2021-02-10T04:34:30.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: - 2021-02-10T04:34:30.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 02 2021-02-10T04:34:30.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: - 2021-02-10T04:34:30.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 10 2021-02-10T04:34:30.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: T 2021-02-10T04:34:30.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 04 2021-02-10T04:34:30.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: : 2021-02-10T04:34:30.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 34 2021-02-10T04:34:30.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: : 2021-02-10T04:34:30.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 28 2021-02-10T04:34:30.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: . 2021-02-10T04:34:30.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 681608 2021-02-10T04:34:30.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: +0000 2021-02-10T04:34:30.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 2021-02-10T04:34:30.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: osd.2 2021-02-10T04:34:30.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: ( 2021-02-10T04:34:30.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: osd 2021-02-10T04:34:30.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: . 2021-02-10T04:34:30.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 2 2021-02-10T04:34:30.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: ) 2021-02-10T04:34:30.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 900 2021-02-10T04:34:30.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: : 2021-02-10T04:34:30.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: cluster 2021-02-10T04:34:30.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 2021-02-10T04:34:30.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: [WRN] 2021-02-10T04:34:30.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 2021-02-10T04:34:30.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:30.652 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[33677]: debug 2021-02-10T04:34:30.461+0000 7fbec19a1700 -1 osd.4 453 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:30.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: cluster 2021-02-10T04:34:28.681579+0000 osd.2 (osd.2) 897 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:30.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: cluster 2021-02-10T04:34:28.681593+0000 osd.2 (osd.2) 898 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:30.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: cluster 2021-02-10T04:34:28.681600 2021-02-10T04:34:30.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 899 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:30.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:30.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: 02-10T04:34:28.681608+0000 osd.2 (osd. 2021-02-10T04:34:30.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: 2) 900 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:30.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:30.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: 10T04:34:28.681613+0000 osd.2 ( 2021-02-10T04:34:30.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: osd.2) 901 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:30.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:30.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: :34:28.681619+0000 osd.2 (osd.2) 902 : cluster 2021-02-10T04:34:30.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:30.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: cluster 2021-02-10T04:34:28. 2021-02-10T04:34:30.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: 681625+0000 osd.2 (osd.2) 903 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:30.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: 2021-02-10T04:34:30.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: cluster 2021-02-10T04:34:30.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: 2021-02-10T04:34:28.681630+0000 osd.2 2021-02-10T04:34:30.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: (osd.2) 904 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:30.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:30.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: -02-10T04:34:28.681639+0000 osd.2 2021-02-10T04:34:30.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: (osd.2) 905 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:30.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:30.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: 02-10T04:34:28.681644+0000 osd.2 2021-02-10T04:34:30.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: (osd.2) 906 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:30.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:30.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: -02-10T04:34:29.339419+0000 mon.a 2021-02-10T04:34:30.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: (mon.0) 1255 : cluster [DBG] osdmap e452: 8 total, 8 up, 8 in 2021-02-10T04:34:30.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: cluster 2021-02-10T04:34:30.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: 2021 2021-02-10T04:34:30.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: -02-10T04 2021-02-10T04:34:30.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: :34:29.486305+0000 osd.4 (osd.4) 559 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:30.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:30.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: T04:34:29.486314+0000 osd.4 (osd.4) 560 : 2021-02-10T04:34:30.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:30.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:30.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: :34:29.486320+0000 osd.4 (osd.4) 2021-02-10T04:34:30.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: 561 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:30.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:30.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: :34:29.486326+0000 2021-02-10T04:34:30.656 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: osd.4 (osd.4) 562 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:30.656 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: 2021-02-10T04:34:30.656 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: cluster 2021-02-10T04:34:30.656 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: 2021-02-10T04:34:30.086645+0000 mgr.y ( 2021-02-10T04:34:30.656 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:30 smithi164 conmon[29288]: mgr.14140) 559 : cluster [DBG] pgmap v828: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 361 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 0 B/s wr, 3 op/s 2021-02-10T04:34:30.656 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 2021-02-10T04:34:30.657 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: cluster 2021-02-10T04:34:30.657 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 2021-02-10T04:34:30.657 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 2021 2021-02-10T04:34:30.657 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: - 2021-02-10T04:34:30.657 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 02 2021-02-10T04:34:30.657 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: - 2021-02-10T04:34:30.657 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 10 2021-02-10T04:34:30.657 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: T04:34:28.681613+0000 osd.2 2021-02-10T04:34:30.657 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: (osd.2) 901 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:30.657 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:30.657 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 34:28.681619+0000 osd.2 (osd.2) 902 2021-02-10T04:34:30.657 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:30.658 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: cluster 2021-02-10T04:34:30.658 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:34:30.658 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 34:28.681625+0000 osd.2 (osd.2) 2021-02-10T04:34:30.658 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 903 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:30.658 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:30.658 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 02-10T04:34:28.681630+0000 osd.2 ( 2021-02-10T04:34:30.658 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: osd.2) 904 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:30.658 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: cluster 2021-02-10T04:34:30.658 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 2021-02-10T04:34:28.681639 2021-02-10T04:34:30.659 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 905 : cluster [WRN] 2021-02-10T04:34:30.659 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:30.659 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:30.659 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 34:28.681644+0000 osd.2 (osd.2) 2021-02-10T04:34:30.659 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 906 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:30.659 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: cluster 2021-02-10T04:34:30.659 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:34:30.659 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 34:29.339419+0000 mon.a (mon.0) 2021-02-10T04:34:30.659 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 1255 : cluster [DBG] osdmap e452: 8 total, 8 up, 8 in 2021-02-10T04:34:30.660 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: cluster 2021-02-10T04:34:30.660 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:34:30.660 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 34:29.486305+0000 osd.4 ( 2021-02-10T04:34:30.660 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: osd.4) 559 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:30.660 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:30.660 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 10T04:34:29.486314+0000 osd.4 (osd. 2021-02-10T04:34:30.660 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 4) 560 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:30.660 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:30.660 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: -02-10T04:34:29.486320+0000 2021-02-10T04:34:30.660 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: osd.4 (osd.4) 561 : cluster [WRN] 2021-02-10T04:34:30.660 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:30.660 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: cluster 2021-02-10T04:34:30.660 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 2021-02-10T04:34 2021-02-10T04:34:30.660 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: :29.486326+0000 osd.4 (osd.4) 2021-02-10T04:34:30.661 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: 562 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:30.661 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:30.661 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: -10T04:34:30.086645+0000 mgr.y (mgr 2021-02-10T04:34:30.661 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[35961]: .14140) 559 : cluster [DBG] pgmap v828: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 361 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 0 B/s wr, 3 op/s 2021-02-10T04:34:30.672 INFO:tasks.rados.rados.0.smithi085.stdout:1722: finishing write tid 1 to smithi08575397-30 2021-02-10T04:34:30.672 INFO:tasks.rados.rados.0.smithi085.stdout:1722: finishing write tid 2 to smithi08575397-30 2021-02-10T04:34:30.673 INFO:tasks.rados.rados.0.smithi085.stdout:1722: finishing write tid 3 to smithi08575397-30 2021-02-10T04:34:30.673 INFO:tasks.rados.rados.0.smithi085.stdout:1722: finishing write tid 4 to smithi08575397-30 2021-02-10T04:34:30.673 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 757 (ObjNum 611 snap 186 seq_num 611) dirty exists 2021-02-10T04:34:30.673 INFO:tasks.rados.rados.0.smithi085.stdout:1722: left oid 30 (ObjNum 611 snap 186 seq_num 611) 2021-02-10T04:34:30.673 INFO:tasks.rados.rados.0.smithi085.stdout:1721: done (11 left) 2021-02-10T04:34:30.673 INFO:tasks.rados.rados.0.smithi085.stdout:1722: done (10 left) 2021-02-10T04:34:30.673 INFO:tasks.rados.rados.0.smithi085.stdout:1723: done (9 left) 2021-02-10T04:34:30.673 INFO:tasks.rados.rados.0.smithi085.stdout:1724: write (excl) oid 14 current snap is 186 2021-02-10T04:34:30.673 INFO:tasks.rados.rados.0.smithi085.stdout:1724: seq_num 613 ranges {702691=635877,2104756=197935} 2021-02-10T04:34:30.686 INFO:tasks.rados.rados.0.smithi085.stdout:1724: writing smithi08575397-14 from 702691 to 1338568 tid 1 2021-02-10T04:34:30.693 INFO:tasks.rados.rados.0.smithi085.stdout:1724: writing smithi08575397-14 from 2104756 to 2302691 tid 2 2021-02-10T04:34:30.693 INFO:tasks.rados.rados.0.smithi085.stdout:1725: copy_from oid 38 from oid 43 current snap is 186 2021-02-10T04:34:30.693 INFO:tasks.rados.rados.0.smithi085.stdout:1724: finishing write tid 1 to smithi08575397-14 2021-02-10T04:34:30.693 INFO:tasks.rados.rados.0.smithi085.stdout:1726: write oid 5 current snap is 186 2021-02-10T04:34:30.693 INFO:tasks.rados.rados.0.smithi085.stdout:1726: seq_num 614 ranges {719498=711126,2046885=272613} 2021-02-10T04:34:30.703 INFO:tasks.rados.rados.0.smithi085.stdout:1726: writing smithi08575397-5 from 719498 to 1430624 tid 1 2021-02-10T04:34:30.709 INFO:tasks.rados.rados.0.smithi085.stdout:1726: writing smithi08575397-5 from 2046885 to 2319498 tid 2 2021-02-10T04:34:30.709 INFO:tasks.rados.rados.0.smithi085.stdout:1727: delete oid 37 current snap is 186 2021-02-10T04:34:30.710 INFO:tasks.rados.rados.0.smithi085.stdout:1724: finishing write tid 2 to smithi08575397-14 2021-02-10T04:34:30.710 INFO:tasks.rados.rados.0.smithi085.stdout:1724: finishing write tid 3 to smithi08575397-14 2021-02-10T04:34:30.710 INFO:tasks.rados.rados.0.smithi085.stdout:1724: finishing write tid 4 to smithi08575397-14 2021-02-10T04:34:30.710 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 760 (ObjNum 613 snap 186 seq_num 613) dirty exists 2021-02-10T04:34:30.710 INFO:tasks.rados.rados.0.smithi085.stdout:1724: left oid 14 (ObjNum 613 snap 186 seq_num 613) 2021-02-10T04:34:30.720 INFO:tasks.rados.rados.0.smithi085.stdout:1725: finishing copy_from to smithi08575397-38 2021-02-10T04:34:30.720 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 706 (ObjNum 607 snap 185 seq_num 607) dirty exists 2021-02-10T04:34:30.720 INFO:tasks.rados.rados.0.smithi085.stdout:1725: finishing copy_from racing read to smithi08575397-38 2021-02-10T04:34:30.729 INFO:tasks.rados.rados.0.smithi085.stdout:1724: done (12 left) 2021-02-10T04:34:30.729 INFO:tasks.rados.rados.0.smithi085.stdout:1725: done (11 left) 2021-02-10T04:34:30.729 INFO:tasks.rados.rados.0.smithi085.stdout:1727: done (10 left) 2021-02-10T04:34:30.730 INFO:tasks.rados.rados.0.smithi085.stdout:1728: write oid 36 current snap is 186 2021-02-10T04:34:30.730 INFO:tasks.rados.rados.0.smithi085.stdout:1728: seq_num 616 ranges {753112=577977,1880202=472910} 2021-02-10T04:34:30.737 INFO:tasks.rados.rados.0.smithi085.stdout:1728: writing smithi08575397-36 from 753112 to 1331089 tid 1 2021-02-10T04:34:30.744 INFO:tasks.rados.rados.0.smithi085.stdout:1728: writing smithi08575397-36 from 1880202 to 2353112 tid 2 2021-02-10T04:34:30.744 INFO:tasks.rados.rados.0.smithi085.stdout:1729: read oid 8 snap 180 2021-02-10T04:34:30.744 INFO:tasks.rados.rados.0.smithi085.stdout:1729: expect (ObjNum 521 snap 155 seq_num 521) 2021-02-10T04:34:30.744 INFO:tasks.rados.rados.0.smithi085.stdout:1730: snap_remove snap 162 2021-02-10T04:34:30.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:30 smithi085 conmon[49106]: debug 2021-02-10T04:34:30.633+0000 7f77a3e94700 -1 osd.2 453 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:31.204 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:34:31.309 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:34:31.337 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[40892]: debug 2021-02-10T04:34:31.059+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:34:31.340 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:34:31.362 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: cluster 2021-02-10T04:34:31.362 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: 2021-02-10T04:34:31.362 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: 2021 2021-02-10T04:34:31.363 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: - 2021-02-10T04:34:31.363 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: 02-10T04:34:29 2021-02-10T04:34:31.363 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: .677148+0000 osd.2 (osd.2) 907 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:31.363 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:31.363 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: 02-10T04:34:29.677158+0000 osd.2 (osd.2 2021-02-10T04:34:31.363 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: ) 908 : cluster 2021-02-10T04:34:31.363 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:31.363 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:31.363 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: 02-10T04:34:29.677165+0000 osd.2 2021-02-10T04:34:31.363 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: (osd.2) 909 2021-02-10T04:34:31.363 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:31.367 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[52836]: debug 2021-02-10T04:34:31.034+0000 7fba75fba700 -1 osd.3 pg_epoch: 453 pg[4.10( v 62'12 (0'0,62'12] local-lis/les=71/72 n=10 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,5,6] r=0 lpr=71 crt=62'12 lcod 62'11 mlcod 0'0 active+clean] scrubber pg(4.10) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6 (0x55fec06e1080) 2021-02-10T04:34:31.368 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: cluster 2021-02-10T04:34:29.677148+0000 osd.2 (osd 2021-02-10T04:34:31.368 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: .2) 907 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:31.368 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: cluster 2021-02-10T04:34:29.677158+0000 osd.2 (osd.2) 908 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:31.368 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: cluster 2021-02-10T04:34:29.677165+0000 osd.2 ( 2021-02-10T04:34:31.368 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: osd.2) 909 : cluster [WRN] 2021-02-10T04:34:31.368 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:31.368 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: cluster 2021-02-10T04:34:31.368 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: 2021 2021-02-10T04:34:31.368 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: - 2021-02-10T04:34:31.368 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: 02 2021-02-10T04:34:31.368 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: -10T04 2021-02-10T04:34:31.368 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: :34:29.677171+0000 osd.2 (osd.2 2021-02-10T04:34:31.369 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: ) 910 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:31.369 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: cluster 2021-02-10T04:34:31.369 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: 2021-02-10T04:34:31.369 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: 2021 2021-02-10T04:34:31.369 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: - 2021-02-10T04:34:31.369 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: 02 2021-02-10T04:34:31.369 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: - 2021-02-10T04:34:31.369 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: 10 2021-02-10T04:34:31.369 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: T 2021-02-10T04:34:31.369 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: 04:34:29.677177+0000 osd.2 (osd 2021-02-10T04:34:31.369 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: .2) 911 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:31.369 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: cluster 2021-02-10T04:34 2021-02-10T04:34:31.369 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: :29.677182+0000 osd.2 (osd.2) 912 2021-02-10T04:34:31.370 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:31.370 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:31.370 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: 10T04:34:29.677188+0000 osd.2 2021-02-10T04:34:31.370 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: (osd.2) 913 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:31.370 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:31.370 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: 02-10T04:34:29.677194+0000 2021-02-10T04:34:31.370 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: osd.2 (osd.2) 914 : cluster 2021-02-10T04:34:31.370 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:31.370 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:34:31.370 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: 34:29.677199+0000 osd.2 (osd.2 2021-02-10T04:34:31.371 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: ) 915 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:31.371 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:31.371 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: -10T04:34:29.677207+0000 2021-02-10T04:34:31.372 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: osd.2 (osd.2) 916 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:31.372 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: 2021-02-10T04:34:31.372 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: cluster 2021-02-10T04:34:31.372 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: 2021-02-10T04:34:30.364476+0000 2021-02-10T04:34:31.372 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: mon.a (mon.0) 1256 : cluster [DBG] 2021-02-10T04:34:31.372 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: osdmap e453: 8 total, 8 up, 8 in 2021-02-10T04:34:31.372 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: cluster 2021-02-10T04:34 2021-02-10T04:34:31.373 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: :30.461775+0000 osd.4 2021-02-10T04:34:31.373 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: (osd.4) 563 : cluster [WRN] 2021-02-10T04:34:31.373 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:31.373 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: cluster 2021-02-10T04:34:30.461791+0000 osd.4 2021-02-10T04:34:31.373 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: (osd.4) 564 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:31.373 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:34:31.373 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: T04:34:30.461795+0000 osd.4 ( 2021-02-10T04:34:31.373 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: osd.4) 565 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:31.373 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:31.373 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: -02-10T04:34:30.461800+0000 2021-02-10T04:34:31.373 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: osd.4 (osd.4) 566 : cluster [WRN] 2021-02-10T04:34:31.374 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:31.374 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: audit 2021-02-10T04:34 2021-02-10T04:34:31.374 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: :30.922975+0000 mon.c (mon.1) 2021-02-10T04:34:31.374 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: 55 : audit [INF] from='client.? 172.21.15.85:0/2381520962' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-02-10T04:34:31.374 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: audit 2021-02-10T04:34:30 2021-02-10T04:34:31.374 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: .923403+0000 mon.a (mon.0) 1257 2021-02-10T04:34:31.374 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[29319]: : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-02-10T04:34:31.374 INFO:tasks.rados.rados.0.smithi085.stdout:1730: done (12 left) 2021-02-10T04:34:31.375 INFO:tasks.rados.rados.0.smithi085.stdout:1731: snap_create 2021-02-10T04:34:31.375 INFO:tasks.rados.rados.0.smithi085.stdout:1728: finishing write tid 1 to smithi08575397-36 2021-02-10T04:34:31.375 INFO:tasks.rados.rados.0.smithi085.stdout:1726: finishing write tid 1 to smithi08575397-5 2021-02-10T04:34:31.375 INFO:tasks.rados.rados.0.smithi085.stdout:1726: finishing write tid 2 to smithi08575397-5 2021-02-10T04:34:31.375 INFO:tasks.rados.rados.0.smithi085.stdout:1726: finishing write tid 3 to smithi08575397-5 2021-02-10T04:34:31.375 INFO:tasks.rados.rados.0.smithi085.stdout:1726: finishing write tid 4 to smithi08575397-5 2021-02-10T04:34:31.375 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 659 (ObjNum 614 snap 186 seq_num 614) dirty exists 2021-02-10T04:34:31.375 INFO:tasks.rados.rados.0.smithi085.stdout:1726: left oid 5 (ObjNum 614 snap 186 seq_num 614) 2021-02-10T04:34:31.375 INFO:tasks.rados.rados.0.smithi085.stdout:1728: finishing write tid 2 to smithi08575397-36 2021-02-10T04:34:31.375 INFO:tasks.rados.rados.0.smithi085.stdout:1728: finishing write tid 3 to smithi08575397-36 2021-02-10T04:34:31.376 INFO:tasks.rados.rados.0.smithi085.stdout:1728: finishing write tid 4 to smithi08575397-36 2021-02-10T04:34:31.376 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 658 (ObjNum 616 snap 186 seq_num 616) dirty exists 2021-02-10T04:34:31.376 INFO:tasks.rados.rados.0.smithi085.stdout:1728: left oid 36 (ObjNum 616 snap 186 seq_num 616) 2021-02-10T04:34:31.376 INFO:tasks.rados.rados.0.smithi085.stdout:1729: expect (ObjNum 521 snap 155 seq_num 521) 2021-02-10T04:34:31.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: cluster 2021-02-10T04:34:31.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: 2021-02-10T04:34:31.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: 2021-02-10T04:34:29 2021-02-10T04:34:31.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: .677148+0000 osd.2 (osd.2) 907 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:31.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:31.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: 02-10T04:34:29.677158+0000 2021-02-10T04:34:31.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: osd.2 (osd.2) 908 : cluster [WRN] 2021-02-10T04:34:31.376 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:31.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: cluster 2021-02-10T04:34:29 2021-02-10T04:34:31.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: .677165+0000 osd.2 (osd.2) 909 : 2021-02-10T04:34:31.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:31.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:34:31.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: 04:34:29.677171+0000 osd.2 (osd 2021-02-10T04:34:31.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: .2) 910 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:31.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:31.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: -02-10T04:34:29.677177 2021-02-10T04:34:31.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 911 : cluster 2021-02-10T04:34:31.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:31.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: cluster 2021-02-10T04:34 2021-02-10T04:34:31.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: :29.677182+0000 osd.2 (osd.2) 2021-02-10T04:34:31.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: 912 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:31.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:31.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: T04:34:29.677188+0000 osd.2 ( 2021-02-10T04:34:31.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: osd.2) 913 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:31.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:31.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: 02-10T04:34:29.677194 2021-02-10T04:34:31.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 914 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:31.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:31.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: -02-10T04:34:29.677199+0000 2021-02-10T04:34:31.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: osd.2 (osd.2) 915 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:31.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: 2021-02-10T04:34:31.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:31.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: 34:29.677207+0000 osd.2 (osd. 2021-02-10T04:34:31.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: 2) 916 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:31.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:31.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: 02-10T04:34:30.364476 2021-02-10T04:34:31.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1256 : cluster 2021-02-10T04:34:31.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: [DBG] osdmap e453: 8 total, 8 up, 8 in 2021-02-10T04:34:31.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:34:31.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: :34:30.461775+0000 osd.4 (osd. 2021-02-10T04:34:31.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: 4) 563 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:31.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:31.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: 10T04:34:30.461791+0000 osd.4 2021-02-10T04:34:31.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: (osd.4) 564 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:31.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:31.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: 02-10T04:34:30.461795+0000 2021-02-10T04:34:31.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: osd.4 (osd.4) 565 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:31.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: 2021-02-10T04:34:31.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:31.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: -02-10T04:34:30.461800+0000 osd.4 (osd. 2021-02-10T04:34:31.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: 4) 566 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:31.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:34:31.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: 10T04:34 2021-02-10T04:34:31.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: :30.922975+0000 mon.c (mon. 2021-02-10T04:34:31.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: 1) 55 : audit [INF] from='client.? 172.21.15.85:0/2381520962' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-02-10T04:34:31.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: audit 2021 2021-02-10T04:34:31.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: -02-10T04:34:30.923403 2021-02-10T04:34:31.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1257 : audit [INF] 2021-02-10T04:34:31.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[35961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-02-10T04:34:31.389 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.2 to 1 (10000) 2021-02-10T04:34:31.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[33677]: debug 2021-02-10T04:34:31.435+0000 7fbec19a1700 -1 osd.4 454 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:31.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[33677]: debug 2021-02-10T04:34:31.474+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:34:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: 10T04:34:29.677171+0000 2021-02-10T04:34:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: osd.2 (osd.2) 910 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: 2021-02-10T04:34:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: cluster 2021-02-10T04:34:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: 2021-02-10T04:34:29.677177+0000 osd.2 ( 2021-02-10T04:34:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: osd.2) 911 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: cluster 2021-02-10T04:34:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:34:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: 04:34:29.677182+0000 2021-02-10T04:34:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: osd.2 (osd.2) 912 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: cluster 2021-02-10T04:34:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: 2021-02-10T04:34:29.677188+0000 2021-02-10T04:34:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: osd.2 (osd.2) 913 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: 2021-02-10T04:34:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: cluster 2021-02-10T04:34:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: 2021-02-10T04:34:29.677194+0000 osd.2 (osd 2021-02-10T04:34:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: .2) 914 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: 10T04:34:29.677199+0000 osd.2 (osd 2021-02-10T04:34:31.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: .2) 915 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:31.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:31.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: -10T04:34:29.677207+0000 osd.2 (osd 2021-02-10T04:34:31.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: .2) 916 : cluster 2021-02-10T04:34:31.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:31.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: cluster 2021-02-10T04:34 2021-02-10T04:34:31.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: :30.364476+0000 mon.a (mon.0) 2021-02-10T04:34:31.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: 1256 : cluster [DBG] osdmap e453: 8 total, 8 up, 8 in 2021-02-10T04:34:31.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:31.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: :34:30.461775+0000 2021-02-10T04:34:31.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: osd.4 (osd.4) 563 : cluster [WRN] 2021-02-10T04:34:31.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:31.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: cluster 2021-02-10T04:34:30. 2021-02-10T04:34:31.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: 461791+0000 osd.4 (osd.4) 564 : cluster 2021-02-10T04:34:31.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:31.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: cluster 2021-02-10T04:34: 2021-02-10T04:34:31.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: 30.461795+0000 osd.4 (osd.4) 565 : 2021-02-10T04:34:31.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:31.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: cluster 2021-02-10T04:34 2021-02-10T04:34:31.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: :30.461800+0000 osd.4 (osd.4) 566 2021-02-10T04:34:31.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:31.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:34:31.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: :34:30.922975+0000 mon.c (mon.1 2021-02-10T04:34:31.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: ) 55 : audit [INF] from='client.? 172.21.15.85:0/2381520962' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-02-10T04:34:31.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:34:31.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: T04:34:30.923403+0000 mon.a (mon. 2021-02-10T04:34:31.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:31 smithi164 conmon[29288]: 0) 1257 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-02-10T04:34:31.671 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:34:31.776 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:34:31.814 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:34:31.897 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set noscrub 2021-02-10T04:34:31.927 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:31 smithi085 conmon[49106]: debug 2021-02-10T04:34:31.648+0000 7f77a3e94700 -1 osd.2 454 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:31.973 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:34:32.193 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:34:32.267 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:34:32.385 INFO:tasks.rados.rados.0.smithi085.stdout:1726: done (12 left) 2021-02-10T04:34:32.386 INFO:tasks.rados.rados.0.smithi085.stdout:1728: done (11 left) 2021-02-10T04:34:32.386 INFO:tasks.rados.rados.0.smithi085.stdout:1729: done (10 left) 2021-02-10T04:34:32.386 INFO:tasks.rados.rados.0.smithi085.stdout:1731: done (9 left) 2021-02-10T04:34:32.386 INFO:tasks.rados.rados.0.smithi085.stdout:1732: snap_remove snap 186 2021-02-10T04:34:32.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[33677]: debug 2021-02-10T04:34:32.395+0000 7fbec19a1700 -1 osd.4 455 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:32.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[33677]: debug 2021-02-10T04:34:32.395+0000 7fbea5924700 -1 osd.4 pg_epoch: 455 pg[6.1a( empty local-lis/les=59/60 n=0 ec=59/59 lis/c=59/59 les/c/f=60/60/0 sis=59) [4,5,1] r=0 lpr=59 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.1a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x562760a5a420) 2021-02-10T04:34:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: cluster 2021-02-10T04:34:30.633758+0000 osd.2 (osd. 2021-02-10T04:34:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: 2) 917 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: cluster 2021-02-10T04:34:30.633768+0000 osd.2 (osd.2) 918 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: cluster 2021-02-10T04:34:30.633774+0000 osd.2 (osd.2 2021-02-10T04:34:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: ) 919 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: T04:34:30.633780+0000 osd.2 (osd. 2021-02-10T04:34:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: 2) 920 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: -02-10T04:34:30.633786+0000 2021-02-10T04:34:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: osd.2 (osd.2) 921 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: 2021-02-10T04:34:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: cluster 2021-02-10T04:34:30 2021-02-10T04:34:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: .633792+0000 osd.2 (osd.2) 922 : cluster 2021-02-10T04:34:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: 04:34:30.633797+0000 osd.2 (osd.2 2021-02-10T04:34:32.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: ) 923 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:32.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:32.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: -02-10T04:34:30.633805+0000 2021-02-10T04:34:32.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: osd.2 (osd.2) 924 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:32.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: 2021-02-10T04:34:32.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:32.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: 02-10T04:34:30.633811+0000 osd.2 2021-02-10T04:34:32.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: (osd.2) 925 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:32.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: 2021-02-10T04:34:32.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: cluster 2021-02-10T04:34:30 2021-02-10T04:34:32.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: .633816+0000 osd.2 (osd.2) 926 : cluster 2021-02-10T04:34:32.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:32.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:34:32.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: 04:34:31.367893+0000 mon.a (mon. 2021-02-10T04:34:32.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: 0) 1258 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2021-02-10T04:34:32.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: cluster 2021-02-10T04:34:32.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: 2021-02-10T04:34:31.367944+0000 2021-02-10T04:34:32.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: mon.a (mon.0) 1259 : cluster [DBG] osdmap e454: 8 total, 8 up, 8 in 2021-02-10T04:34:32.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: 2021-02-10T04:34:32.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:32.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: T04:34:31.436494+0000 osd.4 (osd.4) 567 : 2021-02-10T04:34:32.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:32.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:32.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: 04:34:31.436505+0000 osd.4 (osd.4 2021-02-10T04:34:32.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: ) 568 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:32.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: cluster 2021-02-10T04:34:32.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: 2021-02-10T04:34: 2021-02-10T04:34:32.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: 31.436512+0000 osd.4 (osd.4) 569 2021-02-10T04:34:32.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:32.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: cluster 2021-02-10T04:34:32.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: 2021-02-10T04:34:31.436518 2021-02-10T04:34:32.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 570 : cluster 2021-02-10T04:34:32.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:32.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:32.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: :34:32.087176+0000 mgr.y (mgr. 2021-02-10T04:34:32.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:32 smithi164 conmon[29288]: 14140) 560 : cluster [DBG] pgmap v831: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 368 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.8 MiB/s rd, 1.7 MiB/s wr, 12 op/s 2021-02-10T04:34:32.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: cluster 2021-02-10T04:34:32.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: 2021-02-10T04:34:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: 2021 2021-02-10T04:34:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: - 2021-02-10T04:34:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: 02 2021-02-10T04:34:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: - 2021-02-10T04:34:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: 10 2021-02-10T04:34:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: T04:34:30 2021-02-10T04:34:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: .633758+0000 osd.2 (osd.2 2021-02-10T04:34:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: ) 917 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: -10T04:34:30.633768+0000 osd.2 (osd.2) 918 2021-02-10T04:34:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: cluster 2021-02-10T04:34:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: 2021-02-10T04:34:30.633774+0000 2021-02-10T04:34:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: osd.2 (osd.2) 919 : cluster [WRN] 2021-02-10T04:34:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: T04:34:30.633780+0000 osd.2 ( 2021-02-10T04:34:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: osd.2) 920 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: -02-10T04:34:30.633786+0000 osd.2 2021-02-10T04:34:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: (osd.2) 921 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: cluster 2021-02-10T04:34:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: 2021-02-10T04:34:30 2021-02-10T04:34:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: .633792+0000 osd.2 (osd.2) 922 : 2021-02-10T04:34:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: cluster 2021-02-10T04:34:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: 2021-02-10T04:34 2021-02-10T04:34:32.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: :30.633797+0000 osd.2 (osd.2) 923 2021-02-10T04:34:32.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:32.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: cluster 2021-02-10T04:34:32.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: 2021-02-10T04:34: 2021-02-10T04:34:32.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: 30.633805+0000 osd.2 (osd.2) 924 : 2021-02-10T04:34:32.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:32.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: cluster 2021-02-10T04:34:32.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: 2021-02-10T04:34:30.633811 2021-02-10T04:34:32.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 925 : cluster 2021-02-10T04:34:32.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: cluster 2021-02-10T04:34:32.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: 2021-02-10T04:34 2021-02-10T04:34:32.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: :30.633758+0000 osd.2 (osd.2) 917 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:32.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: cluster 2021-02-10T04:34:32.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: 2021-02-10T04:34:32.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: 2021 2021-02-10T04:34:32.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: - 2021-02-10T04:34:32.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: 02 2021-02-10T04:34:32.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: - 2021-02-10T04:34:32.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: 10 2021-02-10T04:34:32.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: T 2021-02-10T04:34:32.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: 04 2021-02-10T04:34:32.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: : 2021-02-10T04:34:32.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: 34 2021-02-10T04:34:32.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: : 2021-02-10T04:34:32.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: 30 2021-02-10T04:34:32.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: . 2021-02-10T04:34:32.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: 633768 2021-02-10T04:34:32.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: +0000 2021-02-10T04:34:32.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: 2021-02-10T04:34:32.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: osd.2 2021-02-10T04:34:32.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: ( 2021-02-10T04:34:32.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: osd 2021-02-10T04:34:32.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: . 2021-02-10T04:34:32.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: 2) 918 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:32.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:32.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: -10T04:34:30.633774+0000 osd.2 (osd.2) 919 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:32.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: cluster 2021-02-10T04:34:32.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:34:32.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: 34:30.633780+0000 osd.2 (osd.2 2021-02-10T04:34:32.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: ) 920 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:32.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:32.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: 02-10T04:34:30.633786+0000 osd.2 (osd 2021-02-10T04:34:32.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: .2) 921 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:32.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:32.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: -10T04:34:30.633792+0000 osd.2 (osd.2) 922 : cluster 2021-02-10T04:34:32.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:32.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:32.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: 10T04:34:30.633797+0000 osd.2 (osd.2) 923 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:32.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: cluster 2021-02-10T04:34:30.633805 2021-02-10T04:34:32.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 924 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:32.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: 2021-02-10T04:34:32.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: cluster 2021-02-10T04:34:32.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: 2021-02-10T04:34:30.633811+0000 osd.2 (osd.2 2021-02-10T04:34:32.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: ) 925 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:32.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: cluster 2021-02-10T04:34:32.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: 2021-02-10T04:34 2021-02-10T04:34:32.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: :30.633816+0000 osd.2 (osd.2) 926 : 2021-02-10T04:34:32.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:32.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: audit 2021-02-10T04:34:31 2021-02-10T04:34:32.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: .367893+0000 mon.a (mon.0) 1258 : audit [INF] 2021-02-10T04:34:32.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2021-02-10T04:34:32.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: cluster 2021-02-10T04:34 2021-02-10T04:34:32.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: :31.367944+0000 mon.a (mon.0) 1259 : cluster 2021-02-10T04:34:32.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: [DBG] osdmap e454: 8 total, 8 up, 8 in 2021-02-10T04:34:32.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:32.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: 02-10T04:34:31.436494+0000 osd.4 2021-02-10T04:34:32.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: (osd.4) 567 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:32.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:32.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: -10T04:34:31.436505+0000 osd.4 (osd 2021-02-10T04:34:32.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: .4) 568 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:32.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:32.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: 02-10T04:34:31.436512+0000 osd.4 (osd.4) 2021-02-10T04:34:32.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: 569 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:32.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: cluster 2021-02-10T04:34 2021-02-10T04:34:32.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: :31.436518+0000 osd.4 (osd.4) 570 : cluster 2021-02-10T04:34:32.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:32.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: cluster 2021-02-10T04:34:32 2021-02-10T04:34:32.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: .087176+0000 mgr.y (mgr.14140) 560 : cluster 2021-02-10T04:34:32.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[29319]: [DBG] pgmap v831: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 368 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.8 MiB/s rd, 1.7 MiB/s wr, 12 op/s 2021-02-10T04:34:32.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:32.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:32.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: -10T04:34:30.633816+0000 osd.2 (osd.2) 926 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:32.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: audit 2021-02-10T04:34:32.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: 2021-02-10T04:34:31. 2021-02-10T04:34:32.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: 367893+0000 mon.a (mon.0) 1258 : audit 2021-02-10T04:34:32.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2021-02-10T04:34:32.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:32.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: 02-10T04:34:31.367944+0000 mon.a (mon.0 2021-02-10T04:34:32.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: ) 1259 : cluster [DBG] osdmap e454: 8 total, 8 up, 8 in 2021-02-10T04:34:32.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: cluster 2021-02-10T04:34 2021-02-10T04:34:32.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: :31.436494+0000 osd.4 (osd.4) 567 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:32.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:32.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: -10T04:34:31.436505+0000 osd.4 (osd.4) 2021-02-10T04:34:32.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: 568 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:32.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:32.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: 10T04:34:31.436512+0000 osd.4 (osd.4) 569 2021-02-10T04:34:32.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:32.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: cluster 2021-02-10T04:34 2021-02-10T04:34:32.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: :31.436518+0000 osd.4 (osd.4) 570 : cluster [WRN] 2021-02-10T04:34:32.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:32.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:32.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: 10T04:34:32.087176+0000 mgr.y (mgr.14140) 560 : cluster 2021-02-10T04:34:32.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[35961]: [DBG] pgmap v831: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 368 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.8 MiB/s rd, 1.7 MiB/s wr, 12 op/s 2021-02-10T04:34:33.067 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:34:33.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[49106]: debug 2021-02-10T04:34:32.695+0000 7f77a3e94700 -1 osd.2 455 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:33.191 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:34:32 smithi085 conmon[41540]: debug 2021-02-10T04:34:32.804+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:34:33.390 INFO:tasks.rados.rados.0.smithi085.stdout:1732: done (9 left) 2021-02-10T04:34:33.390 INFO:tasks.rados.rados.0.smithi085.stdout:1733: read oid 16 snap -1 2021-02-10T04:34:33.390 INFO:tasks.rados.rados.0.smithi085.stdout:1733: expect (ObjNum 597 snap 180 seq_num 597) 2021-02-10T04:34:33.390 INFO:tasks.rados.rados.0.smithi085.stdout:1734: rollback oid 13 current snap is 187 2021-02-10T04:34:33.390 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 13 to 180 2021-02-10T04:34:33.391 INFO:tasks.rados.rados.0.smithi085.stdout:1735: rollback oid 11 current snap is 187 2021-02-10T04:34:33.391 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 11 to 185 2021-02-10T04:34:33.391 INFO:tasks.rados.rados.0.smithi085.stdout:1736: read oid 44 snap -1 2021-02-10T04:34:33.391 INFO:tasks.rados.rados.0.smithi085.stdout:1736: expect (ObjNum 567 snap 174 seq_num 567) 2021-02-10T04:34:33.391 INFO:tasks.rados.rados.0.smithi085.stdout:1737: rollback oid 43 current snap is 187 2021-02-10T04:34:33.391 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 43 to 165 2021-02-10T04:34:33.391 INFO:tasks.rados.rados.0.smithi085.stdout:1738: copy_from oid 38 from oid 36 current snap is 187 2021-02-10T04:34:33.391 INFO:tasks.rados.rados.0.smithi085.stdout:1739: write (excl) oid 23 current snap is 187 2021-02-10T04:34:33.391 INFO:tasks.rados.rados.0.smithi085.stdout:1739: seq_num 617 ranges {769919=653226,1938684=431235} 2021-02-10T04:34:33.395 INFO:tasks.rados.rados.0.smithi085.stdout:1739: writing smithi08575397-23 from 769919 to 1423145 tid 1 2021-02-10T04:34:33.402 INFO:tasks.rados.rados.0.smithi085.stdout:1739: writing smithi08575397-23 from 1938684 to 2369919 tid 2 2021-02-10T04:34:33.404 INFO:tasks.rados.rados.0.smithi085.stdout:1733: expect (ObjNum 597 snap 180 seq_num 597) 2021-02-10T04:34:33.575 INFO:tasks.rados.rados.0.smithi085.stdout:1734: finishing rollback tid 0 to smithi08575397-13 2021-02-10T04:34:33.576 INFO:tasks.rados.rados.0.smithi085.stdout:1734: finishing rollback tid 1 to smithi08575397-13 2021-02-10T04:34:33.576 INFO:tasks.rados.rados.0.smithi085.stdout:1735: finishing rollback tid 0 to smithi08575397-11 2021-02-10T04:34:33.576 INFO:tasks.rados.rados.0.smithi085.stdout:1735: finishing rollback tid 1 to smithi08575397-11 2021-02-10T04:34:33.576 INFO:tasks.rados.rados.0.smithi085.stdout:1735: finishing rollback tid 2 to smithi08575397-11 2021-02-10T04:34:33.576 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 935 (ObjNum 531 snap 156 seq_num 531) dirty exists 2021-02-10T04:34:33.576 INFO:tasks.rados.rados.0.smithi085.stdout:1734: finishing rollback tid 2 to smithi08575397-13 2021-02-10T04:34:33.576 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 662 (ObjNum 595 snap 178 seq_num 595) dirty exists 2021-02-10T04:34:33.576 INFO:tasks.rados.rados.0.smithi085.stdout:1733: done (15 left) 2021-02-10T04:34:33.576 INFO:tasks.rados.rados.0.smithi085.stdout:1734: done (14 left) 2021-02-10T04:34:33.576 INFO:tasks.rados.rados.0.smithi085.stdout:1735: done (13 left) 2021-02-10T04:34:33.576 INFO:tasks.rados.rados.0.smithi085.stdout:1740: read oid 3 snap 165 2021-02-10T04:34:33.576 INFO:tasks.rados.rados.0.smithi085.stdout:1740: expect (ObjNum 204 snap 44 seq_num 204) 2021-02-10T04:34:33.577 INFO:tasks.rados.rados.0.smithi085.stdout:1741: snap_remove snap 147 2021-02-10T04:34:33.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[33677]: debug 2021-02-10T04:34:33.395+0000 7fbec19a1700 -1 osd.4 456 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:33.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: cluster 2021-02-10T04:34:31.649335+0000 osd.2 2021-02-10T04:34:33.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: (osd.2) 927 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:33.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: cluster 2021-02-10T04:34:31.649344+0000 osd.2 (osd.2) 928 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:33.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: cluster 2021-02-10T04:34:31.649352+0000 osd.2 ( 2021-02-10T04:34:33.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: osd.2) 929 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: :34:31.649359+0000 osd.2 ( 2021-02-10T04:34:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: osd.2) 930 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: T04:34:31.649366+0000 osd.2 (osd 2021-02-10T04:34:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: .2) 931 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: 02-10T04:34:31.649371+0000 2021-02-10T04:34:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: osd.2 (osd.2) 932 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: 2021-02-10T04:34:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: cluster 2021-02-10T04:34:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: 2021-02-10T04:34:31.649378+0000 osd.2 ( 2021-02-10T04:34:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: osd.2) 933 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: -02-10T04:34:31.649383+0000 2021-02-10T04:34:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: osd.2 (osd.2) 934 : cluster [WRN] 2021-02-10T04:34:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: cluster 2021-02-10T04:34:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: 2021-02-10T04:34:31.649391+0000 2021-02-10T04:34:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: osd.2 (osd.2) 935 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: cluster 2021-02-10T04:34:31.649397+0000 osd.2 2021-02-10T04:34:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: (osd.2) 936 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: cluster 2021-02-10T04:34:32.381035+0000 2021-02-10T04:34:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: mon.a (mon.0) 1260 : cluster [DBG] osdmap e455: 8 total, 8 up, 8 in 2021-02-10T04:34:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: 34:32.395765+0000 osd.4 (osd.4) 571 : cluster 2021-02-10T04:34:33.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:33.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:33.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: -10T04:34:32.395771+0000 osd.4 (osd 2021-02-10T04:34:33.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: .4) 572 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:33.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: cluster 2021-02-10T04:34:33.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: 2021-02 2021-02-10T04:34:33.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: -10T04:34:32.395774+0000 2021-02-10T04:34:33.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: osd.4 (osd.4) 573 : cluster 2021-02-10T04:34:33.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:33.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: cluster 2021-02-10T04:34:33.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: 2021-02-10T04:34 2021-02-10T04:34:33.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: :32.395782+0000 osd.4 (osd.4) 574 2021-02-10T04:34:33.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:33.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:33.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: 10T04:34:32.660626+0000 osd.0 2021-02-10T04:34:33.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: (osd.0) 68 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:34:33.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: cluster 2021-02-10T04:34 2021-02-10T04:34:33.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: :32.660932+0000 osd.0 (osd.0) 2021-02-10T04:34:33.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:33 smithi164 conmon[29288]: 69 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:34:33.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: cluster 2021-02-10T04:34:31.649335+0000 osd.2 (osd.2) 927 : cluster 2021-02-10T04:34:33.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:33.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: cluster 2021-02-10T04:34:31.649344+0000 osd.2 (osd.2) 928 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:33.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: cluster 2021-02-10T04:34:31.649352+0000 2021-02-10T04:34:33.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: osd.2 (osd.2) 929 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:33.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: 2021-02-10T04:34:33.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: cluster 2021-02-10T04:34:33.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: 2021-02-10T04:34:31.649359+0000 osd.2 (osd. 2021-02-10T04:34:33.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: 2) 930 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:33.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:34:33.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: T04:34:31.649366+0000 2021-02-10T04:34:33.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: osd.2 (osd.2) 931 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:33.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: 2021-02-10T04:34:33.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: cluster 2021-02-10T04:34:33.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: 2021-02-10T04:34:31.649371+0000 2021-02-10T04:34:33.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: osd.2 (osd.2) 932 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:33.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: 2021-02-10T04:34:33.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: cluster 2021-02-10T04:34:33.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: 2021-02-10T04:34:31.649378+0000 osd.2 (osd 2021-02-10T04:34:33.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: .2) 933 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:33.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:33.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: -10T04:34:31.649383+0000 osd.2 2021-02-10T04:34:33.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: (osd.2) 934 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:33.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: 2021-02-10T04:34:33.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: cluster 2021-02-10T04:34:33.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: 2021-02-10T04:34:31.649391+0000 osd.2 ( 2021-02-10T04:34:33.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: osd.2) 935 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:33.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:33.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: -10T04:34:31.649397 2021-02-10T04:34:33.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 936 : cluster [WRN] 2021-02-10T04:34:33.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:33.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: cluster 2021-02-10T04:34: 2021-02-10T04:34:33.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: 32.381035+0000 mon.a (mon.0) 1260 2021-02-10T04:34:33.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: : cluster [DBG] osdmap e455: 8 total, 8 up, 8 in 2021-02-10T04:34:33.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:33.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: 04:34:32.395765+0000 osd.4 (osd.4) 571 : 2021-02-10T04:34:33.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:33.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: cluster 2021-02-10T04:34:32 2021-02-10T04:34:33.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: .395771+0000 osd.4 (osd.4) 572 2021-02-10T04:34:33.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:33.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: cluster 2021-02-10T04:34 2021-02-10T04:34:33.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: :32.395774+0000 osd.4 (osd.4) 573 2021-02-10T04:34:33.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:33.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:33.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: 10T04:34:32.395782+0000 osd.4 ( 2021-02-10T04:34:33.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: osd.4) 574 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:33.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:33.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: -02-10T04:34:32.660626+0000 2021-02-10T04:34:33.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: osd.0 (osd.0) 68 : cluster [DBG] 2021-02-10T04:34:33.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: purged_snaps scrub starts 2021-02-10T04:34:33.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: cluster 2021-02-10T04:34:33.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: 2021-02-10T04:34:32.660932+0000 osd.0 2021-02-10T04:34:33.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[29319]: (osd.0) 69 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:34:33.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: cluster 2021-02-10T04:34:31. 2021-02-10T04:34:33.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: 649335+0000 osd.2 (osd.2) 927 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:33.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: cluster 2021-02-10T04:34:33.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: 2021-02-10T04:34:31.649344+0000 osd.2 (osd.2) 928 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:33.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: 2021-02-10T04:34:33.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:33.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: -02-10T04:34:31.649352+0000 osd.2 (osd.2) 2021-02-10T04:34:33.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: 929 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:33.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:34:33.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: :34:31.649359+0000 osd.2 (osd.2) 2021-02-10T04:34:33.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: 930 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:33.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: cluster 2021-02-10T04:34:31.649366+0000 2021-02-10T04:34:33.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: osd.2 (osd.2) 931 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:33.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: cluster 2021-02-10T04:34:31.649371+0000 osd.2 (osd 2021-02-10T04:34:33.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: .2) 932 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:33.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: cluster 2021-02-10T04:34:33.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: 2021-02-10T04:34:31.649378+0000 osd.2 (osd.2) 933 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:33.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: cluster 2021-02-10T04:34:31.649383+0000 osd.2 2021-02-10T04:34:33.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: (osd.2) 934 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:33.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: cluster 2021-02-10T04:34:33.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: 2021-02-10T04:34:31.649391+0000 osd.2 (osd.2) 935 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:33.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: 2021-02-10T04:34:33.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: cluster 2021-02-10T04:34 2021-02-10T04:34:33.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: :31.649397+0000 osd.2 (osd.2) 936 : cluster [WRN] 2021-02-10T04:34:33.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:33.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: cluster 2021-02-10T04:34:33.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: 2021-02-10T04:34:32.381035+0000 mon.a (mon.0 2021-02-10T04:34:33.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: ) 1260 : cluster [DBG] osdmap e455: 8 total, 8 up, 8 in 2021-02-10T04:34:33.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: cluster 2021-02-10T04:34:33.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: 2021-02-10T04:34:32.395765 2021-02-10T04:34:33.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 571 : cluster [WRN] 2021-02-10T04:34:33.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:33.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: cluster 2021-02-10T04:34:33.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: 2021-02-10T04:34:32.395771+0000 osd.4 ( 2021-02-10T04:34:33.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: osd.4) 572 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:33.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:34:33.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: 04:34:32.395774+0000 osd.4 (osd.4) 573 : cluster 2021-02-10T04:34:33.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:33.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:34:33.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: :34:32.395782+0000 osd.4 (osd.4) 574 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:33.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: cluster 2021-02-10T04:34:32.660626+0000 osd.0 (osd 2021-02-10T04:34:33.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: .0) 68 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:34:33.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: cluster 2021-02-10T04:34:33.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: 2021-02-10T04:34:32.660932+0000 osd.0 ( 2021-02-10T04:34:33.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: osd.0) 69 : cluster [DBG] 2021-02-10T04:34:33.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:34:34.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:33 smithi085 conmon[49106]: debug 2021-02-10T04:34:33.707+0000 7f77a3e94700 -1 osd.2 456 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:34.396 INFO:tasks.rados.rados.0.smithi085.stdout:1741: done (14 left) 2021-02-10T04:34:34.397 INFO:tasks.rados.rados.0.smithi085.stdout:1742: snap_remove snap 177 2021-02-10T04:34:34.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[33677]: debug 2021-02-10T04:34:34.358+0000 7fbec19a1700 -1 osd.4 456 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:34.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: cluster 2021-02-10T04:34:34.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: 2021-02-10T04:34:32. 2021-02-10T04:34:34.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: 695923+0000 osd.2 (osd.2) 937 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:34.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:34.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: -02-10T04:34:32 2021-02-10T04:34:34.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: .695932+0000 osd.2 (osd.2) 938 : cluster 2021-02-10T04:34:34.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:34.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:34.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: 02-10T04:34:32.695940+0000 osd.2 (osd.2) 939 : cluster 2021-02-10T04:34:34.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:34.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: cluster 2021-02-10T04:34:34.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: 2021-02-10T04:34:32.695947+0000 osd.2 (osd.2 2021-02-10T04:34:34.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: ) 940 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:34.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:34.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: 02-10T04:34:32.695953+0000 osd.2 (osd.2) 2021-02-10T04:34:34.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: 941 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:34.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: cluster 2021-02-10T04:34:34.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: 2021-02-10T04:34:32.695961+0000 osd.2 (osd.2 2021-02-10T04:34:34.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: ) 942 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:34.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:34.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: T04:34:32.695975+0000 osd.2 (osd 2021-02-10T04:34:34.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: .2) 943 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:34.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:34.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: -10T04:34:32.695981+0000 osd.2 2021-02-10T04:34:34.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: (osd.2) 944 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:34.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:34.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: 02-10T04:34:32.695986+0000 2021-02-10T04:34:34.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: osd.2 (osd.2) 945 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:34.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: 2021-02-10T04:34:34.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: cluster 2021-02-10T04:34:34.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: 2021-02-10T04:34:32.695992+0000 osd.2 (osd 2021-02-10T04:34:34.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: .2) 946 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:34.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:34.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: 02-10T04:34:33.386553+0000 2021-02-10T04:34:34.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: mon.a (mon.0) 1261 : cluster [DBG] osdmap e456: 8 total, 8 up, 8 in 2021-02-10T04:34:34.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:34.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: -10T04:34:33.395994+0000 osd.4 2021-02-10T04:34:34.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: (osd.4) 575 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:34.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:34.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: -10T04:34:33.396007+0000 2021-02-10T04:34:34.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: osd.4 (osd.4) 576 : cluster [WRN] 2021-02-10T04:34:34.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:34.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: cluster 2021-02-10T04:34:34.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: 2021-02-10T04:34:33.396014+0000 osd.4 (osd.4) 2021-02-10T04:34:34.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: 577 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:34.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: cluster 2021-02-10T04:34:34.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: 2021-02-10T04:34:33.396040+0000 osd.4 (osd.4 2021-02-10T04:34:34.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: ) 578 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:34.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:34.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: -02-10T04:34:34.087651+0000 2021-02-10T04:34:34.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: mgr.y (mgr.14140) 561 : cluster [DBG] pgmap v834: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 371 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 11 op/s 2021-02-10T04:34:34.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:34 smithi164 conmon[29288]: 2021-02-10T04:34:34.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: cluster 2021-02-10T04:34:32.695923+0000 osd.2 (osd.2) 2021-02-10T04:34:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: 937 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: cluster 2021-02-10T04:34:32.695932+0000 osd.2 (osd.2) 938 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:34:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: T04:34:32.695940+0000 osd.2 (osd.2) 939 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: cluster 2021-02-10T04:34:32.695947 2021-02-10T04:34:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 940 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: 04:34:32.695953+0000 osd.2 (osd.2) 941 2021-02-10T04:34:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: cluster 2021-02-10T04:34:32 2021-02-10T04:34:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: .695961+0000 osd.2 (osd.2) 942 : cluster 2021-02-10T04:34:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: cluster 2021-02-10T04:34:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: 2021-02-10T04:34:32.695975+0000 osd.2 (osd.2) 2021-02-10T04:34:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: 943 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: cluster 2021-02-10T04:34:32. 2021-02-10T04:34:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: 695981+0000 osd.2 (osd.2) 944 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: 2021-02-10T04:34:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: -02-10T04:34:32.695986+0000 osd.2 (osd.2 2021-02-10T04:34:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: ) 945 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: cluster 2021-02-10T04:34: 2021-02-10T04:34:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: 32.695992+0000 osd.2 (osd.2) 946 : cluster 2021-02-10T04:34:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: cluster 2021-02-10T04:34:33 2021-02-10T04:34:34.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: .386553+0000 mon.a (mon.0) 1261 : cluster 2021-02-10T04:34:34.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: [DBG] osdmap e456: 8 total, 8 up, 8 in 2021-02-10T04:34:34.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: cluster 2021-02-10T04:34:33. 2021-02-10T04:34:34.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: 395994+0000 osd.4 (osd.4) 575 : cluster [WRN] 2021-02-10T04:34:34.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:34.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: cluster 2021-02-10T04:34: 2021-02-10T04:34:34.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: 33.396007+0000 osd.4 (osd.4) 576 : 2021-02-10T04:34:34.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:34.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:34.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: 02-10T04:34:33.396014+0000 osd.4 (osd.4) 577 : cluster 2021-02-10T04:34:34.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:34.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: cluster 2021-02-10T04:34:33.396040 2021-02-10T04:34:34.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 578 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:34.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:34:34.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: T04:34:34.087651+0000 mgr.y (mgr.14140) 561 : 2021-02-10T04:34:34.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[29319]: cluster [DBG] pgmap v834: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 371 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 11 op/s 2021-02-10T04:34:34.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: cluster 2021-02-10T04:34:32.695923+0000 2021-02-10T04:34:34.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: osd.2 (osd.2) 937 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:34.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: cluster 2021-02-10T04:34 2021-02-10T04:34:34.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: :32.695932+0000 osd.2 (osd.2) 938 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:34.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:34.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: -02-10T04:34:32.695940+0000 osd.2 (osd 2021-02-10T04:34:34.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: .2) 939 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:34.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:34.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: -02-10T04:34:32.695947+0000 2021-02-10T04:34:34.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: osd.2 (osd.2) 940 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:34.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:34.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: -10T04:34:32.695953+0000 osd.2 ( 2021-02-10T04:34:34.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: osd.2) 941 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:34.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:34.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: T04:34:32.695961+0000 osd.2 (osd.2) 942 : cluster 2021-02-10T04:34:34.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:34.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:34.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: 10T04:34:32.695975+0000 osd.2 (osd.2) 943 : cluster 2021-02-10T04:34:34.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:34.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:34.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: 10T04:34:32.695981+0000 osd.2 (osd.2) 944 : 2021-02-10T04:34:34.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:34.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:34.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: -10T04:34:32.695986+0000 osd.2 (osd.2 2021-02-10T04:34:34.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: ) 945 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:34.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:34.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: 34:32.695992+0000 osd.2 (osd.2) 946 : cluster 2021-02-10T04:34:34.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:34.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:34.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: -10T04:34:33.386553+0000 mon.a (mon.0) 1261 2021-02-10T04:34:34.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: : cluster [DBG] osdmap e456: 8 total, 8 up, 8 in 2021-02-10T04:34:34.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:34.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: -10T04:34:33.395994+0000 osd.4 (osd.4) 2021-02-10T04:34:34.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: 575 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:34.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:34.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: 02-10T04:34:33.396007+0000 osd.4 (osd.4 2021-02-10T04:34:34.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: ) 576 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:34.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: cluster 2021-02-10T04:34:34.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: 2021-02-10T04:34:33. 2021-02-10T04:34:34.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: 396014+0000 osd.4 (osd.4) 577 : cluster [WRN] 2021-02-10T04:34:34.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:34.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:34.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: T04:34:33.396040+0000 osd.4 (osd.4 2021-02-10T04:34:34.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: ) 578 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:34.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:34.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: -10T04:34:34.087651 2021-02-10T04:34:34.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 561 : cluster [DBG] 2021-02-10T04:34:34.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[35961]: pgmap v834: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 371 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 11 op/s 2021-02-10T04:34:35.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:34 smithi085 conmon[49106]: debug 2021-02-10T04:34:34.744+0000 7f77a3e94700 -1 osd.2 457 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:35.394 INFO:tasks.rados.rados.0.smithi085.stdout:1742: done (14 left) 2021-02-10T04:34:35.395 INFO:tasks.rados.rados.0.smithi085.stdout:1743: rollback oid 12 current snap is 187 2021-02-10T04:34:35.395 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 12 to 184 2021-02-10T04:34:35.395 INFO:tasks.rados.rados.0.smithi085.stdout:1736: expect (ObjNum 567 snap 174 seq_num 567) 2021-02-10T04:34:35.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[33677]: debug 2021-02-10T04:34:35.388+0000 7fbec19a1700 -1 osd.4 457 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: cluster 2021-02-10T04:34:33.708051+0000 osd.2 ( 2021-02-10T04:34:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: osd.2) 947 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: cluster 2021-02-10T04:34:33.708071+0000 osd.2 (osd.2) 948 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: cluster 2021-02-10T04:34:33.708077 2021-02-10T04:34:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 949 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: 2021-02-10T04:34:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: cluster 2021-02-10T04:34:33 2021-02-10T04:34:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: .708085+0000 osd.2 (osd.2) 950 : cluster 2021-02-10T04:34:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: 34:33.708091+0000 osd.2 (osd.2) 951 2021-02-10T04:34:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: cluster 2021-02-10T04:34:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: 2021-02-10T04:34:33. 2021-02-10T04:34:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: 708099+0000 osd.2 (osd.2) 952 : cluster [WRN] 2021-02-10T04:34:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: 04:34:33.708106+0000 osd.2 (osd.2 2021-02-10T04:34:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: ) 953 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: T04:34:33.708111+0000 osd.2 (osd.2 2021-02-10T04:34:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: ) 954 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: 02-10T04:34:33.708114+0000 2021-02-10T04:34:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: osd.2 (osd.2) 955 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: 2021-02-10T04:34:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: cluster 2021-02-10T04:34 2021-02-10T04:34:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: :33.708118+0000 osd.2 (osd.2) 2021-02-10T04:34:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: 956 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: 02-10T04:34:34.358876+0000 2021-02-10T04:34:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: osd.4 (osd.4) 579 : cluster [WRN] 2021-02-10T04:34:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: cluster 2021-02-10T04:34:34.358886 2021-02-10T04:34:35.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 580 : cluster [WRN] 2021-02-10T04:34:35.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:35.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: cluster 2021-02-10T04:34: 2021-02-10T04:34:35.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: 34.358895+0000 osd.4 (osd.4) 581 : 2021-02-10T04:34:35.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:35.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:35.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: :34:34.358903+0000 osd.4 (osd.4 2021-02-10T04:34:35.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: ) 582 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:35.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:35.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: -02-10T04:34:34.392008+0000 2021-02-10T04:34:35.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: mon.a (mon.0) 1262 : cluster [DBG] 2021-02-10T04:34:35.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:35 smithi164 conmon[29288]: osdmap e457: 8 total, 8 up, 8 in 2021-02-10T04:34:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: cluster 2021-02-10T04:34:33.708051+0000 osd.2 (osd 2021-02-10T04:34:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: .2) 947 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: cluster 2021-02-10T04:34:33.708071+0000 osd.2 (osd.2) 948 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: cluster 2021-02-10T04:34:33.708077+0000 osd.2 (osd.2) 949 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: cluster 2021-02-10T04:34:33 2021-02-10T04:34:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: .708085+0000 osd.2 (osd.2) 950 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: cluster 2021-02-10T04:34:33.708091 2021-02-10T04:34:35.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 951 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:35.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: cluster 2021-02-10T04:34:33.708099+0000 osd.2 2021-02-10T04:34:35.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: (osd.2) 952 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:35.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:35.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: 02-10T04:34:33. 2021-02-10T04:34:35.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: 708106+0000 osd.2 (osd.2) 953 : cluster 2021-02-10T04:34:35.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:35.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:34:35.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: T04:34:33.708111+0000 osd.2 (osd 2021-02-10T04:34:35.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: .2) 954 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:35.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:35.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: -02-10T04:34:33.708114+0000 2021-02-10T04:34:35.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: osd.2 (osd.2) 955 : cluster [WRN] 2021-02-10T04:34:35.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:35.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:35.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: 04:34:33.708118+0000 osd.2 (osd 2021-02-10T04:34:35.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: .2) 956 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:35.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:35.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: 02-10T04:34:34. 2021-02-10T04:34:35.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: 358876+0000 osd.4 (osd.4) 579 : cluster 2021-02-10T04:34:35.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:35.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:35.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: 04:34:34.358886+0000 osd.4 (osd. 2021-02-10T04:34:35.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: 4) 580 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:35.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:35.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: 10T04:34:34.358895+0000 osd.4 2021-02-10T04:34:35.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: (osd.4) 581 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:35.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: 2021-02-10T04:34:35.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: cluster 2021-02-10T04:34:35.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: 2021-02-10T04:34:34.358903+0000 osd.4 (osd.4) 582 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:35.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: 2021-02-10T04:34:35.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:35.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: 02-10T04:34:34.392008+0000 mon.a (mon. 2021-02-10T04:34:35.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[29319]: 0) 1262 : cluster [DBG] osdmap e457: 8 total, 8 up, 8 in 2021-02-10T04:34:35.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: cluster 2021-02-10T04:34:35.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: 2021-02-10T04:34:35.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: 2021 2021-02-10T04:34:35.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: - 2021-02-10T04:34:35.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: 02 2021-02-10T04:34:35.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: - 2021-02-10T04:34:35.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: 10 2021-02-10T04:34:35.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: T 2021-02-10T04:34:35.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: 04 2021-02-10T04:34:35.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: : 2021-02-10T04:34:35.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: 34 2021-02-10T04:34:35.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: : 2021-02-10T04:34:35.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: 33 2021-02-10T04:34:35.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: . 2021-02-10T04:34:35.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: 708051 2021-02-10T04:34:35.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: +0000 2021-02-10T04:34:35.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: osd.2 (osd.2) 947 : cluster 2021-02-10T04:34:35.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:35.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: cluster 2021-02-10T04:34:35.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: 2021-02-10T04:34:33.708071 2021-02-10T04:34:35.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 948 : cluster 2021-02-10T04:34:35.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:35.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: cluster 2021-02-10T04:34:35.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: 2021-02-10T04:34:33. 2021-02-10T04:34:35.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: 708077+0000 osd.2 (osd.2) 949 : cluster [WRN] 2021-02-10T04:34:35.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:35.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: cluster 2021-02-10T04:34:35.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:34:35.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: 34:33.708085+0000 osd.2 (osd.2 2021-02-10T04:34:35.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: ) 950 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:35.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: cluster 2021-02-10T04:34:35.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:34:35.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: :34:33.708091+0000 osd.2 (osd.2 2021-02-10T04:34:35.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: ) 951 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:35.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:35.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: -10T04:34:33.708099+0000 osd.2 ( 2021-02-10T04:34:35.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: osd.2) 952 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:35.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:34:35.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: 04:34:33.708106+0000 osd.2 (osd.2) 2021-02-10T04:34:35.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: 953 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:35.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: cluster 2021-02-10T04:34:35.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: 2021-02 2021-02-10T04:34:35.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: -10T04:34:33.708111+0000 2021-02-10T04:34:35.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: osd.2 (osd.2) 954 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:35.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: 2021-02-10T04:34:35.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: cluster 2021-02-10T04:34:35.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: 2021-02-10T04:34 2021-02-10T04:34:35.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: :33.708114+0000 osd.2 (osd.2) 2021-02-10T04:34:35.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: 955 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:35.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:35.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: 02-10T04:34:33.708118+0000 2021-02-10T04:34:35.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: osd.2 (osd.2) 956 : cluster 2021-02-10T04:34:35.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:35.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:34:35.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: 04:34:34.358876+0000 osd.4 (osd 2021-02-10T04:34:35.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: .4) 579 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:35.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:35.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: 02-10T04:34:34.358886+0000 osd.4 2021-02-10T04:34:35.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: (osd.4) 580 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:35.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: 2021-02-10T04:34:35.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: cluster 2021-02-10T04:34:35.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: 2021-02-10T04:34 2021-02-10T04:34:35.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: :34.358895+0000 osd.4 (osd.4) 2021-02-10T04:34:35.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: 581 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:35.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:35.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: 02-10T04:34:34.358903+0000 2021-02-10T04:34:35.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: osd.4 (osd.4) 582 : cluster [WRN] 2021-02-10T04:34:35.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:35.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:34:35.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: 04:34:34.392008+0000 mon.a ( 2021-02-10T04:34:35.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[35961]: mon.0) 1262 : cluster [DBG] osdmap e457: 8 total, 8 up, 8 in 2021-02-10T04:34:35.767 INFO:tasks.rados.rados.0.smithi085.stdout:1736: done (14 left) 2021-02-10T04:34:35.768 INFO:tasks.rados.rados.0.smithi085.stdout:1744: snap_remove snap 159 2021-02-10T04:34:36.148 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[52836]: debug 2021-02-10T04:34:36.147+0000 7fba71fb2700 -1 osd.3 pg_epoch: 458 pg[4.10( v 62'12 (0'0,62'12] local-lis/les=71/72 n=10 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,5,6] r=0 lpr=71 crt=62'12 lcod 62'11 mlcod 0'0 active+clean] scrubber pg(4.10) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6 (0x55fec06fe160) 2021-02-10T04:34:36.148 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[49106]: debug 2021-02-10T04:34:36.148 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:35 smithi085 conmon[49106]: 2021-02-10T04:34:35.747+0000 7f77a3e94700 -1 osd.2 458 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:36.411 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: cluster 2021-02-10T04:34:34.744926+0000 osd.2 (osd.2 2021-02-10T04:34:36.411 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: ) 957 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:36.411 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: cluster 2021-02-10T04:34:34.744931+0000 osd.2 (osd.2) 958 : cluster 2021-02-10T04:34:36.411 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:36.411 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:36.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: 04:34:34.744935+0000 osd.2 (osd.2) 959 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:36.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:36.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: -02-10T04:34:34.744938+0000 2021-02-10T04:34:36.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: osd.2 (osd.2) 960 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:36.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:36.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: 02-10T04:34:34.744942+0000 osd.2 (osd 2021-02-10T04:34:36.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: .2) 961 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:36.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:36.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: -10T04:34:34.744945+0000 osd.2 ( 2021-02-10T04:34:36.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: osd.2) 962 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:36.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:36.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: -02-10T04:34:34.744948+0000 2021-02-10T04:34:36.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: osd.2 (osd.2) 963 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:36.413 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: cluster 2021-02-10T04:34:36.413 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: 2021-02-10T04:34:34.744951+0000 2021-02-10T04:34:36.413 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: osd.2 (osd.2) 964 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:36.413 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: 2021-02-10T04:34:36.413 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:36.413 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: 10T04:34:34.744955+0000 osd.2 (osd.2) 965 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:36.413 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: cluster 2021-02-10T04:34:34.744957+0000 2021-02-10T04:34:36.413 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: osd.2 (osd.2) 966 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:36.413 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: cluster 2021-02-10T04:34:35.388795+0000 osd.4 ( 2021-02-10T04:34:36.413 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: osd.4) 583 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:36.413 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: cluster 2021-02-10T04:34:36.413 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: 2021-02-10T04:34:35.388804+0000 osd.4 (osd.4) 584 : cluster 2021-02-10T04:34:36.413 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:36.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:36.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: -02-10T04:34:35.388811+0000 osd.4 (osd.4) 2021-02-10T04:34:36.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: 585 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:36.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:34:36.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: 34:35.388816+0000 2021-02-10T04:34:36.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: osd.4 (osd.4) 586 : cluster 2021-02-10T04:34:36.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:36.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:34:36.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: :34:35.393161+0000 mon.a (mon.0) 2021-02-10T04:34:36.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: 1263 : cluster [DBG] osdmap e458: 8 total, 8 up, 8 in 2021-02-10T04:34:36.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:36.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: 02-10T04:34:36.088150+0000 mgr.y 2021-02-10T04:34:36.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: (mgr.14140) 562 : cluster 2021-02-10T04:34:36.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[29319]: [DBG] pgmap v837: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 379 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.3 MiB/s wr, 21 op/s 2021-02-10T04:34:36.417 INFO:tasks.rados.rados.0.smithi085.stdout:1744: done (14 left) 2021-02-10T04:34:36.417 INFO:tasks.rados.rados.0.smithi085.stdout:1745: write (excl) oid 30 current snap is 187 2021-02-10T04:34:36.417 INFO:tasks.rados.rados.0.smithi085.stdout:1737: finishing rollback tid 0 to smithi08575397-43 2021-02-10T04:34:36.417 INFO:tasks.rados.rados.0.smithi085.stdout:1745: seq_num 618 ranges {786726=728475,2280813=105913} 2021-02-10T04:34:36.426 INFO:tasks.rados.rados.0.smithi085.stdout:1745: writing smithi08575397-30 from 786726 to 1515201 tid 1 2021-02-10T04:34:36.433 INFO:tasks.rados.rados.0.smithi085.stdout:1745: writing smithi08575397-30 from 2280813 to 2386726 tid 2 2021-02-10T04:34:36.433 INFO:tasks.rados.rados.0.smithi085.stdout:1746: snap_create 2021-02-10T04:34:36.433 INFO:tasks.rados.rados.0.smithi085.stdout:1737: finishing rollback tid 1 to smithi08575397-43 2021-02-10T04:34:36.433 INFO:tasks.rados.rados.0.smithi085.stdout:1737: finishing rollback tid 2 to smithi08575397-43 2021-02-10T04:34:36.433 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 710 (ObjNum 491 snap 141 seq_num 491) dirty exists 2021-02-10T04:34:36.433 INFO:tasks.rados.rados.0.smithi085.stdout:1739: finishing write tid 1 to smithi08575397-23 2021-02-10T04:34:36.433 INFO:tasks.rados.rados.0.smithi085.stdout:1739: finishing write tid 2 to smithi08575397-23 2021-02-10T04:34:36.433 INFO:tasks.rados.rados.0.smithi085.stdout:1739: finishing write tid 3 to smithi08575397-23 2021-02-10T04:34:36.433 INFO:tasks.rados.rados.0.smithi085.stdout:1739: finishing write tid 4 to smithi08575397-23 2021-02-10T04:34:36.433 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 661 (ObjNum 617 snap 187 seq_num 617) dirty exists 2021-02-10T04:34:36.433 INFO:tasks.rados.rados.0.smithi085.stdout:1739: left oid 23 (ObjNum 617 snap 187 seq_num 617) 2021-02-10T04:34:36.433 INFO:tasks.rados.rados.0.smithi085.stdout:1738: finishing copy_from to smithi08575397-38 2021-02-10T04:34:36.434 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 658 (ObjNum 616 snap 186 seq_num 616) dirty exists 2021-02-10T04:34:36.434 INFO:tasks.rados.rados.0.smithi085.stdout:1738: finishing copy_from racing read to smithi08575397-38 2021-02-10T04:34:36.434 INFO:tasks.rados.rados.0.smithi085.stdout:1740: expect (ObjNum 204 snap 44 seq_num 204) 2021-02-10T04:34:36.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[33677]: debug 2021-02-10T04:34:36.375+0000 7fbec19a1700 -1 osd.4 458 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:36.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: cluster 2021-02-10T04:34:36.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: 2021-02-10T04:34:36.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: 2021-02-10T04:34:34.744926 2021-02-10T04:34:36.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 957 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:36.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: cluster 2021-02-10T04:34 2021-02-10T04:34:36.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: :34.744931+0000 osd.2 (osd.2) 958 : 2021-02-10T04:34:36.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:36.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: cluster 2021-02-10T04:34:36.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: 2021-02-10T04:34 2021-02-10T04:34:36.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: :34.744935+0000 osd.2 (osd 2021-02-10T04:34:36.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: .2) 959 : cluster [WRN] 2021-02-10T04:34:36.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:36.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:36.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: -10T04:34:34.744938+0000 osd.2 (osd 2021-02-10T04:34:36.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: .2) 960 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:36.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:36.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: 02-10T04:34:34.744942+0000 osd.2 2021-02-10T04:34:36.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: (osd.2) 961 : cluster 2021-02-10T04:34:36.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:36.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:36.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: 02-10T04:34:34.744945 2021-02-10T04:34:36.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 962 2021-02-10T04:34:36.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:36.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: cluster 2021-02-10T04:34:36.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: 2021-02-10T04:34: 2021-02-10T04:34:36.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: 34.744948+0000 osd.2 (osd. 2021-02-10T04:34:36.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: 2) 963 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:36.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: 2021-02-10T04:34:36.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: cluster 2021-02-10T04:34:34.744951 2021-02-10T04:34:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 964 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: 04:34:34.744955+0000 osd.2 (osd.2) 2021-02-10T04:34:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: 965 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: :34:34.744957+0000 osd.2 (osd.2) 2021-02-10T04:34:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: 966 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: cluster 2021-02-10T04:34:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: 2021-02-10T04:34:35.388795+0000 osd.4 (osd 2021-02-10T04:34:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: .4) 583 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: cluster 2021-02-10T04:34: 2021-02-10T04:34:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: 35.388804+0000 osd.4 (osd. 2021-02-10T04:34:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: 4) 584 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:36.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: cluster 2021-02-10T04:34:36.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: 2021-02-10T04:34 2021-02-10T04:34:36.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: :35.388811+0000 osd.4 (osd 2021-02-10T04:34:36.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: .4) 585 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:36.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: 2021-02-10T04:34:36.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:36.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: 02-10T04:34:35.388816+0000 osd.4 (osd.4) 586 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:36.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:36.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: T04:34:35.393161+0000 mon.a (mon.0) 1263 : cluster 2021-02-10T04:34:36.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: [DBG] osdmap e458: 8 total, 8 up, 8 in 2021-02-10T04:34:36.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:36.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: -10T04:34:36 2021-02-10T04:34:36.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: .088150+0000 mgr.y (mgr. 2021-02-10T04:34:36.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:36 smithi164 conmon[29288]: 14140) 562 : cluster [DBG] pgmap v837: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 379 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.3 MiB/s wr, 21 op/s 2021-02-10T04:34:36.691 INFO:tasks.rados.rados.0.smithi085.stdout:1743: finishing rollback tid 0 to smithi08575397-12 2021-02-10T04:34:36.691 INFO:tasks.rados.rados.0.smithi085.stdout:1743: finishing rollback tid 1 to smithi08575397-12 2021-02-10T04:34:36.691 INFO:tasks.rados.rados.0.smithi085.stdout:1743: finishing rollback tid 2 to smithi08575397-12 2021-02-10T04:34:36.691 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 763 (ObjNum 85 snap 11 seq_num 85) dirty exists 2021-02-10T04:34:36.691 INFO:tasks.rados.rados.0.smithi085.stdout:1745: finishing write tid 1 to smithi08575397-30 2021-02-10T04:34:36.691 INFO:tasks.rados.rados.0.smithi085.stdout:1745: finishing write tid 2 to smithi08575397-30 2021-02-10T04:34:36.691 INFO:tasks.rados.rados.0.smithi085.stdout:1745: finishing write tid 3 to smithi08575397-30 2021-02-10T04:34:36.691 INFO:tasks.rados.rados.0.smithi085.stdout:1745: finishing write tid 4 to smithi08575397-30 2021-02-10T04:34:36.692 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 760 (ObjNum 618 snap 187 seq_num 618) dirty exists 2021-02-10T04:34:36.692 INFO:tasks.rados.rados.0.smithi085.stdout:1745: left oid 30 (ObjNum 618 snap 187 seq_num 618) 2021-02-10T04:34:36.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: cluster 2021-02-10T04:34:34.744926+0000 2021-02-10T04:34:36.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: osd.2 (osd.2) 957 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:36.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: cluster 2021-02-10T04:34:34.744931+0000 osd.2 (osd.2) 2021-02-10T04:34:36.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: 958 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:36.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:34:36.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: 04:34:34.744935+0000 osd.2 (osd.2) 959 : cluster [WRN] 2021-02-10T04:34:36.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:36.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: cluster 2021-02-10T04:34:36.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: 2021-02-10T04:34:34.744938+0000 osd.2 (osd 2021-02-10T04:34:36.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: .2) 960 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:36.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:36.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: T04:34:34.744942+0000 osd.2 (osd. 2021-02-10T04:34:36.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: 2) 961 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:36.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:36.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: 02-10T04:34:34.744945+0000 2021-02-10T04:34:36.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: osd.2 (osd.2) 962 : cluster [WRN] 2021-02-10T04:34:36.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:36.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: cluster 2021-02-10T04:34:36.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: 2021-02-10T04:34:34.744948+0000 osd.2 2021-02-10T04:34:36.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: (osd.2) 963 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:36.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:36.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: 10T04:34:34.744951+0000 osd.2 ( 2021-02-10T04:34:36.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: osd.2) 964 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:36.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:36.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: -10T04:34:34.744955+0000 osd.2 2021-02-10T04:34:36.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: (osd.2) 965 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:36.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:36.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: 02-10T04:34:34.744957+0000 2021-02-10T04:34:36.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: osd.2 (osd.2) 966 : cluster [WRN] 2021-02-10T04:34:36.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:36.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: cluster 2021-02-10T04:34:36.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: 2021-02-10T04:34:35.388795+0000 osd.4 (osd.4 2021-02-10T04:34:36.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: ) 583 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:36.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:36.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: 34:35.388804+0000 osd.4 (osd.4) 584 : 2021-02-10T04:34:36.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:36.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:36.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: -10T04:34:35.388811+0000 osd.4 (osd 2021-02-10T04:34:36.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: .4) 585 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:36.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:36.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: T04:34:35.388816+0000 osd.4 (osd. 2021-02-10T04:34:36.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: 4) 586 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:36.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:36.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: 02-10T04:34:35.393161+0000 mon.a 2021-02-10T04:34:36.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: (mon.0) 1263 : cluster [DBG] osdmap e458: 8 total, 8 up, 8 in 2021-02-10T04:34:36.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: cluster 2021-02-10T04:34:36.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: 2021-02-10T04:34:36.088150+0000 2021-02-10T04:34:36.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: mgr.y (mgr.14140) 562 : cluster [DBG] pgmap v837: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 379 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.3 MiB/s wr, 21 op/s 2021-02-10T04:34:36.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[35961]: 2021-02-10T04:34:37.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:36 smithi085 conmon[49106]: debug 2021-02-10T04:34:36.778+0000 7f77a3e94700 -1 osd.2 459 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:37.421 INFO:tasks.rados.rados.0.smithi085.stdout:1737: done (15 left) 2021-02-10T04:34:37.422 INFO:tasks.rados.rados.0.smithi085.stdout:1738: done (14 left) 2021-02-10T04:34:37.422 INFO:tasks.rados.rados.0.smithi085.stdout:1739: done (13 left) 2021-02-10T04:34:37.422 INFO:tasks.rados.rados.0.smithi085.stdout:1740: done (12 left) 2021-02-10T04:34:37.422 INFO:tasks.rados.rados.0.smithi085.stdout:1743: done (11 left) 2021-02-10T04:34:37.422 INFO:tasks.rados.rados.0.smithi085.stdout:1745: done (10 left) 2021-02-10T04:34:37.422 INFO:tasks.rados.rados.0.smithi085.stdout:1746: done (9 left) 2021-02-10T04:34:37.422 INFO:tasks.rados.rados.0.smithi085.stdout:1747: write oid 13 current snap is 188 2021-02-10T04:34:37.422 INFO:tasks.rados.rados.0.smithi085.stdout:1747: seq_num 619 ranges {403533=403724,1539295=652201} 2021-02-10T04:34:37.426 INFO:tasks.rados.rados.0.smithi085.stdout:1747: writing smithi08575397-13 from 403533 to 807257 tid 1 2021-02-10T04:34:37.435 INFO:tasks.rados.rados.0.smithi085.stdout:1747: writing smithi08575397-13 from 1539295 to 2191496 tid 2 2021-02-10T04:34:37.436 INFO:tasks.rados.rados.0.smithi085.stdout:1748: write oid 43 current snap is 188 2021-02-10T04:34:37.436 INFO:tasks.rados.rados.0.smithi085.stdout:1748: seq_num 620 ranges {420340=478973,1597777=512212} 2021-02-10T04:34:37.441 INFO:tasks.rados.rados.0.smithi085.stdout:1748: writing smithi08575397-43 from 420340 to 899313 tid 1 2021-02-10T04:34:37.449 INFO:tasks.rados.rados.0.smithi085.stdout:1748: writing smithi08575397-43 from 1597777 to 2109989 tid 2 2021-02-10T04:34:37.449 INFO:tasks.rados.rados.0.smithi085.stdout:1749: write oid 7 current snap is 188 2021-02-10T04:34:37.449 INFO:tasks.rados.rados.0.smithi085.stdout:1747: finishing write tid 1 to smithi08575397-13 2021-02-10T04:34:37.450 INFO:tasks.rados.rados.0.smithi085.stdout:1749: seq_num 621 ranges {437147=554222,1656259=655870} 2021-02-10T04:34:37.455 INFO:tasks.rados.rados.0.smithi085.stdout:1749: writing smithi08575397-7 from 437147 to 991369 tid 1 2021-02-10T04:34:37.465 INFO:tasks.rados.rados.0.smithi085.stdout:1749: writing smithi08575397-7 from 1656259 to 2312129 tid 2 2021-02-10T04:34:37.465 INFO:tasks.rados.rados.0.smithi085.stdout:1750: copy_from oid 50 from oid 26 current snap is 188 2021-02-10T04:34:37.465 INFO:tasks.rados.rados.0.smithi085.stdout:1747: finishing write tid 2 to smithi08575397-13 2021-02-10T04:34:37.465 INFO:tasks.rados.rados.0.smithi085.stdout:1747: finishing write tid 3 to smithi08575397-13 2021-02-10T04:34:37.465 INFO:tasks.rados.rados.0.smithi085.stdout:1747: finishing write tid 4 to smithi08575397-13 2021-02-10T04:34:37.465 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 665 (ObjNum 619 snap 188 seq_num 619) dirty exists 2021-02-10T04:34:37.465 INFO:tasks.rados.rados.0.smithi085.stdout:1747: left oid 13 (ObjNum 619 snap 188 seq_num 619) 2021-02-10T04:34:37.465 INFO:tasks.rados.rados.0.smithi085.stdout:1748: finishing write tid 1 to smithi08575397-43 2021-02-10T04:34:37.465 INFO:tasks.rados.rados.0.smithi085.stdout:1747: done (12 left) 2021-02-10T04:34:37.466 INFO:tasks.rados.rados.0.smithi085.stdout:1751: read oid 48 snap -1 2021-02-10T04:34:37.466 INFO:tasks.rados.rados.0.smithi085.stdout:1751: expect (ObjNum 476 snap 135 seq_num 476) 2021-02-10T04:34:37.466 INFO:tasks.rados.rados.0.smithi085.stdout:1752: read oid 36 snap -1 2021-02-10T04:34:37.466 INFO:tasks.rados.rados.0.smithi085.stdout:1752: expect (ObjNum 616 snap 186 seq_num 616) 2021-02-10T04:34:37.466 INFO:tasks.rados.rados.0.smithi085.stdout:1753: snap_create 2021-02-10T04:34:37.466 INFO:tasks.rados.rados.0.smithi085.stdout:1748: finishing write tid 2 to smithi08575397-43 2021-02-10T04:34:37.466 INFO:tasks.rados.rados.0.smithi085.stdout:1748: finishing write tid 3 to smithi08575397-43 2021-02-10T04:34:37.466 INFO:tasks.rados.rados.0.smithi085.stdout:1748: finishing write tid 4 to smithi08575397-43 2021-02-10T04:34:37.466 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 713 (ObjNum 620 snap 188 seq_num 620) dirty exists 2021-02-10T04:34:37.466 INFO:tasks.rados.rados.0.smithi085.stdout:1748: left oid 43 (ObjNum 620 snap 188 seq_num 620) 2021-02-10T04:34:37.466 INFO:tasks.rados.rados.0.smithi085.stdout:1749: finishing write tid 1 to smithi08575397-7 2021-02-10T04:34:37.468 INFO:tasks.rados.rados.0.smithi085.stdout:1750: finishing copy_from to smithi08575397-50 2021-02-10T04:34:37.468 INFO:tasks.rados.rados.0.smithi085.stdout:1750: got expected ENOENT (src dne) 2021-02-10T04:34:37.468 INFO:tasks.rados.rados.0.smithi085.stdout:1750: finishing copy_from racing read to smithi08575397-50 2021-02-10T04:34:37.469 INFO:tasks.rados.rados.0.smithi085.stdout:1752: expect (ObjNum 616 snap 186 seq_num 616) 2021-02-10T04:34:37.574 INFO:tasks.rados.rados.0.smithi085.stdout:1749: finishing write tid 2 to smithi08575397-7 2021-02-10T04:34:37.574 INFO:tasks.rados.rados.0.smithi085.stdout:1749: finishing write tid 3 to smithi08575397-7 2021-02-10T04:34:37.575 INFO:tasks.rados.rados.0.smithi085.stdout:1749: finishing write tid 4 to smithi08575397-7 2021-02-10T04:34:37.575 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 716 (ObjNum 621 snap 188 seq_num 621) dirty exists 2021-02-10T04:34:37.575 INFO:tasks.rados.rados.0.smithi085.stdout:1749: left oid 7 (ObjNum 621 snap 188 seq_num 621) 2021-02-10T04:34:37.575 INFO:tasks.rados.rados.0.smithi085.stdout:1751: expect (ObjNum 476 snap 135 seq_num 476) 2021-02-10T04:34:37.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[33677]: debug 2021-02-10T04:34:37.340+0000 7fbec19a1700 -1 osd.4 459 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:37.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[33677]: debug 2021-02-10T04:34:37.341+0000 7fbea992c700 -1 osd.4 pg_epoch: 459 pg[6.1a( empty local-lis/les=59/60 n=0 ec=59/59 lis/c=59/59 les/c/f=60/60/0 sis=59) [4,5,1] r=0 lpr=59 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.1a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x562760a598c0) 2021-02-10T04:34:37.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: cluster 2021-02-10T04:34:37.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: 2021-02-10T04:34:35.747724 2021-02-10T04:34:37.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 967 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:37.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:37.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: 10T04:34:35.747732+0000 osd.2 (osd.2) 968 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:37.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: cluster 2021-02-10T04:34:35.747739+0000 osd.2 ( 2021-02-10T04:34:37.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: osd.2) 969 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:37.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: cluster 2021-02-10T04:34:35.747746+0000 osd.2 ( 2021-02-10T04:34:37.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: osd.2) 970 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:37.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: cluster 2021-02-10T04:34:37.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: 2021-02-10T04:34:35.747752+0000 osd.2 (osd.2) 971 : cluster 2021-02-10T04:34:37.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:37.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: cluster 2021-02-10T04:34:35.747758+0000 osd.2 (osd.2) 972 : cluster [WRN] 2021-02-10T04:34:37.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:37.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: cluster 2021-02-10T04:34:37.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: 2021-02-10T04:34:35.747766+0000 osd.2 (osd.2) 2021-02-10T04:34:37.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: 973 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:37.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: cluster 2021-02-10T04:34:37.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: 2021-02-10T04:34: 2021-02-10T04:34:37.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: 35.747771+0000 osd.2 (osd.2) 974 : cluster 2021-02-10T04:34:37.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:37.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:37.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: -10T04:34:35.747777+0000 osd.2 2021-02-10T04:34:37.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: (osd.2) 975 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:37.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: cluster 2021-02-10T04:34:37.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: 2021-02-10T04:34:35.747784+0000 osd.2 (osd 2021-02-10T04:34:37.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: .2) 976 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:37.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: cluster 2021-02-10T04:34:37.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: 2021-02-10T04:34:36.376265+0000 osd.4 (osd.4) 587 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:37.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: 2021-02-10T04:34:37.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:37.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: 34:36.376276+0000 osd.4 (osd.4) 588 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:37.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:37.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: T04:34:36.376281+0000 osd.4 (osd.4) 589 : cluster 2021-02-10T04:34:37.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:37.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:37.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: 10T04:34:36.376285+0000 osd.4 (osd.4) 590 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:37.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: cluster 2021-02-10T04:34:36.410300+0000 2021-02-10T04:34:37.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:37 smithi164 conmon[29288]: mon.a (mon.0) 1264 : cluster [DBG] osdmap e459: 8 total, 8 up, 8 in 2021-02-10T04:34:37.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: cluster 2021-02-10T04:34:35.747724+0000 osd.2 (osd. 2021-02-10T04:34:37.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: 2) 967 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:37.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: cluster 2021-02-10T04:34:35.747732+0000 osd.2 (osd.2) 968 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:37.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: cluster 2021-02-10T04:34:35.747739+0000 osd.2 (osd.2) 969 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:37.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:37.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: 04:34:35.747746+0000 2021-02-10T04:34:37.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: osd.2 (osd.2) 970 : 2021-02-10T04:34:37.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:37.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:37.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: -10T04:34:35.747752+0000 osd.2 (osd.2 2021-02-10T04:34:37.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: ) 971 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:37.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:34:37.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: :34:35.747758+0000 osd.2 (osd.2) 972 : cluster 2021-02-10T04:34:37.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:37.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: cluster 2021-02-10T04:34:35.747766+0000 osd.2 (osd.2) 973 2021-02-10T04:34:37.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:37.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:37.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: -02-10T04:34:35.747771+0000 osd.2 (osd.2) 974 : cluster 2021-02-10T04:34:37.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:37.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: cluster 2021-02-10T04:34:35.747777+0000 osd.2 (osd.2) 975 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:37.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:34:37.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: T04:34:35.747784+0000 osd.2 (osd.2) 2021-02-10T04:34:37.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: 976 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:37.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:34:37.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: T04:34:36.376265+0000 osd.4 (osd.4) 587 : 2021-02-10T04:34:37.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:37.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: cluster 2021-02-10T04:34:37.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: 2021-02-10T04:34:36.376276+0000 osd.4 (osd.4) 588 2021-02-10T04:34:37.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:37.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: cluster 2021-02-10T04:34 2021-02-10T04:34:37.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: :36.376281+0000 osd.4 (osd.4) 589 : cluster 2021-02-10T04:34:37.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:37.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: cluster 2021-02-10T04:34 2021-02-10T04:34:37.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: :36.376285+0000 osd.4 (osd. 2021-02-10T04:34:37.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: 4) 590 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:37.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: 2021-02-10T04:34:37.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: cluster 2021-02-10T04:34:36. 2021-02-10T04:34:37.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: 410300+0000 mon.a (mon.0) 1264 : cluster 2021-02-10T04:34:37.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[29319]: [DBG] osdmap e459: 8 total, 8 up, 8 in 2021-02-10T04:34:37.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: cluster 2021-02-10T04:34:35.747724+0000 2021-02-10T04:34:37.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: osd.2 (osd.2) 967 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:37.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:34:37.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: 04:34:35.747732+0000 osd.2 (osd.2) 968 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:37.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:37.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: -02-10T04:34:35.747739+0000 osd.2 (osd.2) 969 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:37.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: cluster 2021-02-10T04:34:37.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: 2021-02-10T04:34:35.747746+0000 2021-02-10T04:34:37.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: osd.2 (osd.2) 970 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:37.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: 2021-02-10T04:34:37.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:37.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: -02-10T04:34:35.747752+0000 osd.2 (osd.2) 971 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:37.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:37.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: 02-10T04:34:35.747758+0000 osd.2 2021-02-10T04:34:37.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: (osd.2) 972 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:37.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: cluster 2021-02-10T04:34:37.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: 2021-02-10T04:34 2021-02-10T04:34:37.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: :35.747766+0000 osd.2 (osd.2) 973 : cluster 2021-02-10T04:34:37.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:37.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: cluster 2021-02-10T04:34 2021-02-10T04:34:37.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: :35.747771+0000 osd.2 (osd.2) 974 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:37.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:37.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: 10T04:34:35.747777+0000 osd.2 (osd.2) 975 : cluster 2021-02-10T04:34:37.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:37.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:37.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: -10T04:34:35.747784+0000 osd.2 (osd.2 2021-02-10T04:34:37.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: ) 976 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:37.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: cluster 2021-02-10T04:34:37.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: 2021-02-10T04:34:37.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: 2021 2021-02-10T04:34:37.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: -02-10T04:34 2021-02-10T04:34:37.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: :36.376265+0000 osd.4 (osd.4) 587 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:37.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: cluster 2021-02-10T04:34:37.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: 2021-02-10T04:34:37.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:34:37.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: :34:36.376276+0000 osd.4 (osd.4) 588 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:37.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:37.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: -02-10T04:34:36.376281+0000 osd.4 (osd. 2021-02-10T04:34:37.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: 4) 589 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:37.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: cluster 2021-02-10T04:34:37.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: 2021-02-10T04:34:37.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: 2021 2021-02-10T04:34:37.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: - 2021-02-10T04:34:37.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: 02 2021-02-10T04:34:37.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: - 2021-02-10T04:34:37.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: 10 2021-02-10T04:34:37.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: T 2021-02-10T04:34:37.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: 04 2021-02-10T04:34:37.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: : 2021-02-10T04:34:37.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: 34 2021-02-10T04:34:37.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: : 2021-02-10T04:34:37.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: 36 2021-02-10T04:34:37.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: . 2021-02-10T04:34:37.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: 376285 2021-02-10T04:34:37.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: +0000 2021-02-10T04:34:37.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: 2021-02-10T04:34:37.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: osd.4 2021-02-10T04:34:37.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: ( 2021-02-10T04:34:37.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: osd 2021-02-10T04:34:37.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: . 2021-02-10T04:34:37.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: 4 2021-02-10T04:34:37.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: ) 2021-02-10T04:34:37.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: 590 2021-02-10T04:34:37.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: : 2021-02-10T04:34:37.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: cluster 2021-02-10T04:34:37.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: 2021-02-10T04:34:37.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: [WRN] 2021-02-10T04:34:37.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: 2021-02-10T04:34:37.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:37.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: 2021-02-10T04:34:37.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: cluster 2021-02-10T04:34:37.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: 2021-02-10T04:34:37.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: 2021-02 2021-02-10T04:34:37.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[35961]: -10T04:34:36.410300+0000 mon.a (mon.0) 1264 : cluster [DBG] osdmap e459: 8 total, 8 up, 8 in 2021-02-10T04:34:38.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:37 smithi085 conmon[49106]: debug 2021-02-10T04:34:37.768+0000 7f77a3e94700 -1 osd.2 460 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:38.304 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:34:38.305 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[33677]: debug 2021-02-10T04:34:38.304+0000 7fbec19a1700 -1 osd.4 460 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:38.426 INFO:tasks.rados.rados.0.smithi085.stdout:1748: done (14 left) 2021-02-10T04:34:38.427 INFO:tasks.rados.rados.0.smithi085.stdout:1749: done (13 left) 2021-02-10T04:34:38.427 INFO:tasks.rados.rados.0.smithi085.stdout:1750: done (12 left) 2021-02-10T04:34:38.427 INFO:tasks.rados.rados.0.smithi085.stdout:1751: done (11 left) 2021-02-10T04:34:38.427 INFO:tasks.rados.rados.0.smithi085.stdout:1752: done (10 left) 2021-02-10T04:34:38.427 INFO:tasks.rados.rados.0.smithi085.stdout:1753: done (9 left) 2021-02-10T04:34:38.427 INFO:tasks.rados.rados.0.smithi085.stdout:1754: snap_remove snap 187 2021-02-10T04:34:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: cluster 2021-02-10T04:34:36.778534+0000 osd.2 (osd.2) 977 : cluster 2021-02-10T04:34:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: cluster 2021-02-10T04:34:36.778543+0000 osd.2 (osd.2) 978 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: 02-10T04:34:36.778549+0000 osd.2 (osd.2) 979 : cluster [WRN] 2021-02-10T04:34:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: -02-10T04:34:36.778553+0000 osd.2 (osd.2 2021-02-10T04:34:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: ) 980 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: cluster 2021-02-10T04:34:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:34:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: 04:34:36.778556+0000 2021-02-10T04:34:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: osd.2 (osd.2) 981 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: cluster 2021-02-10T04:34:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: 2021-02-10T04:34:36.778559+0000 2021-02-10T04:34:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: osd.2 (osd.2) 982 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: -10T04:34:36.778563+0000 osd.2 (osd.2) 983 : cluster 2021-02-10T04:34:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: cluster 2021-02-10T04:34:36.778566+0000 osd.2 2021-02-10T04:34:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: (osd.2) 984 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: cluster 2021-02-10T04:34:36.778580+0000 osd.2 (osd.2 2021-02-10T04:34:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: ) 985 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: cluster 2021-02-10T04:34:36.778583+0000 osd.2 2021-02-10T04:34:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: (osd.2) 986 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: -02-10T04:34:37.341325+0000 2021-02-10T04:34:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: osd.4 (osd.4) 591 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: 10T04:34:37.341334+0000 osd.4 (osd 2021-02-10T04:34:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: .4) 592 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: 02-10T04:34:37 2021-02-10T04:34:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: .341341+0000 osd.4 (osd.4) 593 : cluster [WRN] 2021-02-10T04:34:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: cluster 2021-02-10T04:34:37.341347 2021-02-10T04:34:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 594 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: cluster 2021-02-10T04:34: 2021-02-10T04:34:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: 37.420256+0000 mon.a (mon.0) 1265 : cluster 2021-02-10T04:34:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: [DBG] osdmap e460: 8 total, 8 up, 8 in 2021-02-10T04:34:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: T04:34:37.768955+0000 osd.2 (osd 2021-02-10T04:34:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: .2) 987 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: -10T04:34:37.768964+0000 osd.2 2021-02-10T04:34:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: (osd.2) 988 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: 2021-02-10T04:34:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: cluster 2021-02-10T04:34:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: 2021-02-10T04:34:37.768970+0000 osd.2 2021-02-10T04:34:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: (osd.2) 989 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: 10T04:34:37.768976+0000 2021-02-10T04:34:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: osd.2 (osd.2) 990 : 2021-02-10T04:34:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: 02-10T04:34:37.768979 2021-02-10T04:34:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 991 : cluster 2021-02-10T04:34:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: cluster 2021-02-10T04:34:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: 2021-02-10T04:34 2021-02-10T04:34:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: :37.768982+0000 osd.2 (osd. 2021-02-10T04:34:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: 2) 992 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: cluster 2021-02-10T04:34:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:34:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: 34:37.768986+0000 osd.2 (osd.2 2021-02-10T04:34:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: ) 993 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: 02-10T04:34:37.768989+0000 osd.2 ( 2021-02-10T04:34:38.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: osd.2) 994 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:38.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: cluster 2021-02-10T04:34: 2021-02-10T04:34:38.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: 37.768992+0000 osd.2 (osd.2 2021-02-10T04:34:38.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: ) 995 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:38.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:38.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: -02-10T04:34:37.768996 2021-02-10T04:34:38.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 996 : 2021-02-10T04:34:38.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:38.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:34:38.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: 04:34:38.036049+0000 2021-02-10T04:34:38.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: mgr.y (mgr.14140) 563 : 2021-02-10T04:34:38.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:34:38.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: audit 2021 2021-02-10T04:34:38.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: -02-10T04:34:38 2021-02-10T04:34:38.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: .081495+0000 mon.a (mon.0) 1266 : 2021-02-10T04:34:38.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:34:38.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:38.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: T04:34:38.088526+0000 mgr.y ( 2021-02-10T04:34:38.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: mgr.14140) 564 : cluster [DBG] 2021-02-10T04:34:38.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: pgmap v840: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 379 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 1.9 MiB/s wr, 18 op/s 2021-02-10T04:34:38.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: audit 2021-02-10T04:34:38.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: 2021-02-10T04:34 2021-02-10T04:34:38.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: :38.089836+0000 mon.a (mon.0 2021-02-10T04:34:38.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: ) 1267 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:34:38.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:38.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: -10T04:34:38.137333+0000 mon.a 2021-02-10T04:34:38.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:38 smithi164 conmon[29288]: (mon.0) 1268 : cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 302 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:34:38.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: cluster 2021-02-10T04:34:36.778534+0000 osd.2 (osd 2021-02-10T04:34:38.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: .2) 977 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:38.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: cluster 2021-02-10T04:34:36.778543+0000 osd.2 (osd.2) 978 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:38.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:38.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 02-10T04:34:36.778549+0000 osd.2 (osd.2) 979 : cluster [WRN] 2021-02-10T04:34:38.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:38.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:38.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: -02-10T04:34:36.778553 2021-02-10T04:34:38.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 980 : cluster [WRN] 2021-02-10T04:34:38.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:38.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: cluster 2021-02-10T04:34:38.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 2021-02-10T04:34:36.778556+0000 osd.2 2021-02-10T04:34:38.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: (osd.2) 981 : cluster [WRN] 2021-02-10T04:34:38.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:38.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:38.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: -02-10T 2021-02-10T04:34:38.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 04:34:36.778559+0000 osd.2 (osd. 2021-02-10T04:34:38.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 2) 982 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:38.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 2021-02-10T04:34:38.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:38.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: -02-10T04:34: 2021-02-10T04:34:38.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 36.778563+0000 osd.2 (osd 2021-02-10T04:34:38.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: .2) 983 : cluster 2021-02-10T04:34:38.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:38.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: cluster 2021-02-10T04:34: 2021-02-10T04:34:38.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 36.778566+0000 osd.2 (osd.2) 984 : cluster 2021-02-10T04:34:38.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:38.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: cluster 2021-02-10T04:34:36.778580+0000 2021-02-10T04:34:38.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: osd.2 (osd.2) 985 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:38.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:38.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 02-10T04:34:36.778583+0000 osd.2 ( 2021-02-10T04:34:38.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: osd.2) 986 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:38.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:38.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 10T04:34:37.341325+0000 osd.4 (osd. 2021-02-10T04:34:38.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 4) 591 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:38.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:38.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 04:34:37.341334+0000 osd.4 (osd.4) 592 : 2021-02-10T04:34:38.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: cluster 2021-02-10T04:34:38.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:38.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:38.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 04:34:37.341341+0000 osd.4 (osd. 2021-02-10T04:34:38.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 4) 593 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:38.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:34:38.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: T04:34:37.341347+0000 osd.4 (osd.4 2021-02-10T04:34:38.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: ) 594 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:38.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: cluster 2021-02-10T04:34:38.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 2021 2021-02-10T04:34:38.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: -02- 2021-02-10T04:34:38.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 10T04:34:37.420256+0000 mon.a 2021-02-10T04:34:38.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: (mon.0) 1265 : cluster [DBG] osdmap e460: 8 total, 8 up, 8 in 2021-02-10T04:34:38.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 2021-02-10T04:34:38.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:38.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: -10T04:34:37.768955+0000 osd.2 2021-02-10T04:34:38.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: (osd.2) 987 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:38.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:38.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: -02-10T04:34: 2021-02-10T04:34:38.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 37.768964+0000 osd.2 (osd.2) 988 : 2021-02-10T04:34:38.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:38.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: cluster 2021-02-10T04:34 2021-02-10T04:34:38.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: :37.768970+0000 osd.2 (osd. 2021-02-10T04:34:38.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 2) 989 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:38.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:38.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 04:34:37.768976+0000 osd.2 (osd.2 2021-02-10T04:34:38.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: ) 990 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:38.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: cluster 2021-02-10T04:34:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: 2021-02-10T04:34:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: 2021 2021-02-10T04:34:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: - 2021-02-10T04:34:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: 02 2021-02-10T04:34:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: - 2021-02-10T04:34:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: 10 2021-02-10T04:34:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: T 2021-02-10T04:34:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: 04 2021-02-10T04:34:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: : 2021-02-10T04:34:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: 34 2021-02-10T04:34:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: : 2021-02-10T04:34:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: 36.778534+0000 2021-02-10T04:34:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: osd.2 (osd.2) 977 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: 02-10T04:34:36.778543+0000 osd.2 (osd.2 2021-02-10T04:34:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: ) 978 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: cluster 2021-02-10T04:34:36.778549+0000 osd.2 (osd.2) 2021-02-10T04:34:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: 979 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: cluster 2021-02-10T04:34:36 2021-02-10T04:34:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: .778553+0000 osd.2 (osd.2) 980 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:38.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: T04:34:36.778556+0000 osd.2 (osd 2021-02-10T04:34:38.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: .2) 981 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:38.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:38.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: -10T04:34:36.778559+0000 osd.2 2021-02-10T04:34:38.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: (osd.2) 982 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:38.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:38.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: 02-10T04:34:36.778563+0000 osd.2 2021-02-10T04:34:38.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: (osd.2) 983 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:38.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: 2021-02-10T04:34:38.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: cluster 2021-02-10T04:34:38.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: 2021-02-10T04:34:36.778566+0000 2021-02-10T04:34:38.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: osd.2 (osd.2) 984 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:38.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: 2021-02-10T04:34:38.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: cluster 2021-02-10T04:34:36 2021-02-10T04:34:38.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: .778580+0000 osd.2 (osd.2) 985 : 2021-02-10T04:34:38.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:38.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:38.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: T04:34:36.778583+0000 osd.2 (osd 2021-02-10T04:34:38.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: .2) 986 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:38.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:38.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: -10T04:34:37.341325+0000 osd.4 2021-02-10T04:34:38.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: (osd.4) 591 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:38.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:38.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: -02-10T04:34:37.341334+0000 osd.4 2021-02-10T04:34:38.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: (osd.4) 592 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:38.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:38.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: 02-10T04:34:37.341341+0000 2021-02-10T04:34:38.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: osd.4 (osd.4) 593 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:38.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: 2021-02-10T04:34:38.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: cluster 2021-02-10T04:34:38.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: 2021-02-10T04:34 2021-02-10T04:34:38.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: :37.341347+0000 osd.4 (osd.4) 2021-02-10T04:34:38.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: 594 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:38.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:38.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: 02-10T04:34:37.420256+0000 2021-02-10T04:34:38.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: mon.a (mon.0) 1265 : cluster [DBG] 2021-02-10T04:34:38.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: osdmap e460: 8 total, 8 up, 8 in 2021-02-10T04:34:38.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:38.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: T04:34:37.768955+0000 osd.2 (osd 2021-02-10T04:34:38.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: .2) 987 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:38.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:38.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: -02-10T04:34:37.768964+0000 2021-02-10T04:34:38.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: osd.2 (osd.2) 988 : cluster [WRN] 2021-02-10T04:34:38.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:38.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: cluster 2021-02-10T04:34:37. 2021-02-10T04:34:38.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: 768970+0000 osd.2 (osd.2) 989 : cluster 2021-02-10T04:34:38.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:38.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: cluster 2021-02-10T04:34 2021-02-10T04:34:38.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: :37.768976+0000 osd.2 (osd.2) 990 : cluster 2021-02-10T04:34:38.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:38.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: cluster 2021-02-10T04:34: 2021-02-10T04:34:38.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: 37.768979+0000 osd.2 (osd.2) 991 2021-02-10T04:34:38.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:38.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:34:38.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: :34:37.768982+0000 osd.2 (osd.2 2021-02-10T04:34:38.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: ) 992 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:38.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: cluster 2021-02-10T04:34:38.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: 2021-02-10T04:34:37.768986 2021-02-10T04:34:38.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 993 : cluster 2021-02-10T04:34:38.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:38.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:38.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: 10T04:34:37.768989+0000 osd.2 ( 2021-02-10T04:34:38.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: osd.2) 994 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:38.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:38.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: -02-10T04:34:37.768992+0000 2021-02-10T04:34:38.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: osd.2 (osd.2) 995 : cluster [WRN] 2021-02-10T04:34:38.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:38.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: cluster 2021-02-10T04:34:37. 2021-02-10T04:34:38.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: 768996+0000 osd.2 (osd.2) 996 : cluster 2021-02-10T04:34:38.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:38.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: audit 2021-02-10T04:34 2021-02-10T04:34:38.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: :38.036049+0000 mgr.y (mgr.14140) 2021-02-10T04:34:38.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: 563 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:34:38.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:34:38.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: 04:34:38.081495+0000 mon.a (mon 2021-02-10T04:34:38.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: .0) 1266 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:34:38.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:38.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: 02-10T04:34:38.088526+0000 2021-02-10T04:34:38.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: mgr.y (mgr.14140) 564 : cluster [DBG] pgmap v840: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 379 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 1.9 MiB/s wr, 18 op/s 2021-02-10T04:34:38.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: 2021-02-10T04:34:38.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: audit 2021-02-10T04:34:38.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: 2021-02-10T04:34:38.089836+0000 mon.a ( 2021-02-10T04:34:38.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: mon.0) 1267 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:34:38.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:38.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: 02-10T04:34:38.137333+0000 2021-02-10T04:34:38.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: mon.a (mon.0) 1268 : cluster [WRN] 2021-02-10T04:34:38.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[35961]: Health check update: 14 slow ops, oldest one blocked for 302 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:34:38.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 02-10T04:34:37.768979+0000 osd.2 ( 2021-02-10T04:34:38.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: osd.2) 991 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:38.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:38.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 04:34:37.768982+0000 osd.2 (osd.2) 2021-02-10T04:34:38.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 992 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:38.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:38.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 02-10T04:34: 2021-02-10T04:34:38.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 37.768986+0000 osd.2 (osd.2) 993 : cluster 2021-02-10T04:34:38.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:38.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: cluster 2021-02-10T04:34 2021-02-10T04:34:38.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: :37.768989+0000 osd.2 (osd.2) 994 : cluster 2021-02-10T04:34:38.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:38.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: cluster 2021-02-10T04:34: 2021-02-10T04:34:38.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 37.768992+0000 osd.2 (osd.2) 995 : cluster 2021-02-10T04:34:38.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:38.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:34:38.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: :34:37.768996+0000 osd.2 ( 2021-02-10T04:34:38.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: osd.2) 996 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:38.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: audit 2021-02-10T04:34:38.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:34:38.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 34:38.036049+0000 mgr.y 2021-02-10T04:34:38.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: (mgr.14140) 563 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:34:38.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:34:38.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: -10T04:34:38.081495+0000 2021-02-10T04:34:38.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: mon.a (mon.0) 1266 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:34:38.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 2021-02-10T04:34:38.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: cluster 2021-02-10T04:34:38.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 2021-02-10T04:34:38.088526+0000 2021-02-10T04:34:38.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: mgr.y (mgr.14140) 564 : cluster [DBG] pgmap v840: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 379 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 1.9 MiB/s wr, 18 op/s 2021-02-10T04:34:38.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: audit 2021 2021-02-10T04:34:38.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: -02-10T04:34:38.089836+0000 2021-02-10T04:34:38.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: mon.a (mon.0) 1267 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:34:38.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 2021-02-10T04:34:38.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: cluster 2021-02-10T04:34:38.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 2021-02-10T04:34:38.137333+0000 mon.a (mon. 2021-02-10T04:34:38.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: 0) 1268 : cluster 2021-02-10T04:34:38.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[29319]: [WRN] Health check update: 14 slow ops, oldest one blocked for 302 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:34:39.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:38 smithi085 conmon[49106]: debug 2021-02-10T04:34:38.755+0000 7f77a3e94700 -1 osd.2 461 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:39.433 INFO:tasks.rados.rados.0.smithi085.stdout:1754: done (9 left) 2021-02-10T04:34:39.433 INFO:tasks.rados.rados.0.smithi085.stdout:1755: read oid 26 snap -1 2021-02-10T04:34:39.433 INFO:tasks.rados.rados.0.smithi085.stdout:1755: expect deleted 2021-02-10T04:34:39.433 INFO:tasks.rados.rados.0.smithi085.stdout:1756: write oid 3 current snap is 189 2021-02-10T04:34:39.433 INFO:tasks.rados.rados.0.smithi085.stdout:1756: seq_num 622 ranges {453954=629471,1598388=515881} 2021-02-10T04:34:39.435 INFO:tasks.rados.rados.0.smithi085.stdout:1756: writing smithi08575397-3 from 453954 to 1083425 tid 1 2021-02-10T04:34:39.441 INFO:tasks.rados.rados.0.smithi085.stdout:1756: writing smithi08575397-3 from 1598388 to 2114269 tid 2 2021-02-10T04:34:39.441 INFO:tasks.rados.rados.0.smithi085.stdout:1756: finishing write tid 1 to smithi08575397-3 2021-02-10T04:34:39.442 INFO:tasks.rados.rados.0.smithi085.stdout:1755: done (10 left) 2021-02-10T04:34:39.442 INFO:tasks.rados.rados.0.smithi085.stdout:1757: read oid 40 snap -1 2021-02-10T04:34:39.442 INFO:tasks.rados.rados.0.smithi085.stdout:1757: expect deleted 2021-02-10T04:34:39.442 INFO:tasks.rados.rados.0.smithi085.stdout:1758: snap_create 2021-02-10T04:34:39.445 INFO:tasks.rados.rados.0.smithi085.stdout:1756: finishing write tid 2 to smithi08575397-3 2021-02-10T04:34:39.446 INFO:tasks.rados.rados.0.smithi085.stdout:1756: finishing write tid 3 to smithi08575397-3 2021-02-10T04:34:39.446 INFO:tasks.rados.rados.0.smithi085.stdout:1756: finishing write tid 4 to smithi08575397-3 2021-02-10T04:34:39.446 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 719 (ObjNum 622 snap 189 seq_num 622) dirty exists 2021-02-10T04:34:39.446 INFO:tasks.rados.rados.0.smithi085.stdout:1756: left oid 3 (ObjNum 622 snap 189 seq_num 622) 2021-02-10T04:34:39.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[33677]: debug 2021-02-10T04:34:39.335+0000 7fbec19a1700 -1 osd.4 461 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: cluster 2021-02-10T04:34:38.304837+0000 osd.4 2021-02-10T04:34:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: (osd.4) 595 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: cluster 2021-02-10T04:34:38.304843+0000 osd.4 (osd.4) 596 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: cluster 2021-02-10T04:34:38.304847+0000 2021-02-10T04:34:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: osd.4 (osd.4) 597 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: -02-10T04:34:38.304850+0000 osd.4 ( 2021-02-10T04:34:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: osd.4) 598 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: -10T04:34:38.426870+0000 2021-02-10T04:34:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: mon.a (mon.0) 1269 : cluster [DBG] 2021-02-10T04:34:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: osdmap e461: 8 total, 8 up, 8 in 2021-02-10T04:34:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: :34:38.682587+0000 osd.7 (osd. 2021-02-10T04:34:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: 7) 50 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:34:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: 10T04:34:38.682727+0000 osd.7 ( 2021-02-10T04:34:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: osd.7) 51 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:34:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: 02-10T04:34:38.755985+0000 2021-02-10T04:34:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: osd.2 (osd.2) 997 : cluster [WRN] 2021-02-10T04:34:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: cluster 2021-02-10T04:34:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: 2021-02-10T04:34:38.755994+0000 osd.2 2021-02-10T04:34:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: (osd.2) 998 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: 2021-02-10T04:34:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: cluster 2021-02-10T04:34:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: 2021-02-10T04:34:38.756000+0000 osd.2 ( 2021-02-10T04:34:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: osd.2) 999 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: 02-10T04:34:38.756009+0000 2021-02-10T04:34:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: osd.2 (osd.2) 1000 : cluster [WRN] 2021-02-10T04:34:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: cluster 2021-02-10T04:34:39.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: 2021-02-10T04:34:38.756015+0000 osd.2 (osd. 2021-02-10T04:34:39.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: 2) 1001 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:39.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:39.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: T04:34:38.756044+0000 osd.2 (osd 2021-02-10T04:34:39.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: .2) 1002 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:39.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:39.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: 02-10T04:34:38.756051+0000 2021-02-10T04:34:39.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: osd.2 (osd.2) 1003 : cluster [WRN] 2021-02-10T04:34:39.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:39.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: cluster 2021-02-10T04:34:39.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: 2021-02-10T04:34:38.756057+0000 2021-02-10T04:34:39.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: osd.2 (osd.2) 1004 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:39.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: 2021-02-10T04:34:39.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: cluster 2021-02-10T04:34:39.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: 2021-02-10T04:34:38.756062+0000 osd.2 ( 2021-02-10T04:34:39.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: osd.2) 1005 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:39.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:39.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: -02-10T04:34:38.756074+0000 2021-02-10T04:34:39.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: osd.2 (osd.2) 1006 : cluster [WRN] 2021-02-10T04:34:39.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:39 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:39.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: cluster 2021-02-10T04:34:39.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:34:39.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: 10T04:34:38.304837+0000 osd.4 (osd.4) 595 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:39.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:39.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: 04:34:38.304843+0000 osd.4 (osd.4) 596 : cluster 2021-02-10T04:34:39.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:39.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: -10T04:34:38.304847+0000 osd.4 2021-02-10T04:34:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: (osd.4) 597 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: 2021-02-10T04:34:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: cluster 2021-02-10T04:34: 2021-02-10T04:34:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: 38.304850+0000 osd.4 (osd.4) 598 2021-02-10T04:34:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: 10T04:34:38.426870+0000 mon.a 2021-02-10T04:34:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: (mon.0) 1269 : cluster [DBG] osdmap e461: 8 total, 8 up, 8 in 2021-02-10T04:34:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: 2021-02-10T04:34:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: cluster 2021-02-10T04:34 2021-02-10T04:34:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: :38.682587+0000 osd.7 (osd.7) 2021-02-10T04:34:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: 50 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:34:39.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:39.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: -02-10T04:34:38.682727+0000 2021-02-10T04:34:39.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: osd.7 (osd.7) 51 : cluster [DBG] 2021-02-10T04:34:39.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: purged_snaps scrub ok 2021-02-10T04:34:39.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:39.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: 04:34:38.755985+0000 osd.2 (osd.2) 997 : cluster 2021-02-10T04:34:39.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:39.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:34:39.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: 34:38.755994+0000 osd.2 (osd.2 2021-02-10T04:34:39.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: ) 998 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:39.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:39.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: -02-10T04:34:38.756000+0000 2021-02-10T04:34:39.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: osd.2 (osd.2) 999 : cluster [WRN] 2021-02-10T04:34:39.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:39.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:34:39.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: 34:38.756009+0000 osd.2 (osd.2) 2021-02-10T04:34:39.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: 1000 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:39.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:39.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: 02-10T04:34:38.756015+0000 2021-02-10T04:34:39.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: osd.2 (osd.2) 1001 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:39.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: 2021-02-10T04:34:39.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: cluster 2021-02-10T04:34:39.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: 2021-02-10T04:34:38.756044+0000 osd.2 ( 2021-02-10T04:34:39.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: osd.2) 1002 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:39.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:39.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: -02-10T04:34:38.756051+0000 2021-02-10T04:34:39.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: osd.2 (osd.2) 1003 : cluster [WRN] 2021-02-10T04:34:39.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:39.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: cluster 2021-02-10T04:34:38 2021-02-10T04:34:39.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: .756057+0000 osd.2 (osd.2) 1004 : 2021-02-10T04:34:39.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:39.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:34:39.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: 34:38.756062+0000 osd.2 (osd.2 2021-02-10T04:34:39.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: ) 1005 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:39.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:39.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: 10T04:34:38.756074+0000 osd.2 2021-02-10T04:34:39.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: (osd.2) 1006 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:39.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[29319]: 2021-02-10T04:34:39.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: cluster 2021-02-10T04:34:38.304837+0000 osd.4 ( 2021-02-10T04:34:39.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: osd.4) 595 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:39.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: cluster 2021-02-10T04:34:38.304843+0000 osd.4 (osd. 2021-02-10T04:34:39.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: 4) 596 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:39.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:39.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: -10T04:34:38.304847+0000 osd.4 (osd.4) 597 2021-02-10T04:34:39.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:39.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: cluster 2021-02-10T04:34: 2021-02-10T04:34:39.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: 38.304850+0000 osd.4 (osd.4) 598 2021-02-10T04:34:39.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:39.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: cluster 2021-02-10T04:34 2021-02-10T04:34:39.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: :38.426870+0000 mon.a (mon.0) 2021-02-10T04:34:39.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: 1269 : cluster [DBG] osdmap e461: 8 total, 8 up, 8 in 2021-02-10T04:34:39.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:34:39.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: :34:38.682587+0000 osd.7 (osd.7 2021-02-10T04:34:39.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: ) 50 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:34:39.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:34:39.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: :34:38.682727+0000 osd.7 (osd.7 2021-02-10T04:34:39.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: ) 51 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:34:39.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:34:39.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: :34:38.755985+0000 osd.2 (osd.2 2021-02-10T04:34:39.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: ) 997 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:39.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:39.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: 34:38.755994+0000 osd.2 (osd.2 2021-02-10T04:34:39.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: ) 998 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:39.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:34:39.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: :34:38.756000+0000 osd.2 (osd. 2021-02-10T04:34:39.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: 2) 999 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:39.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:34:39.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: :34:38.756009+0000 osd.2 (osd.2 2021-02-10T04:34:39.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: ) 1000 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:39.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:39.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: 34:38.756015+0000 osd.2 (osd.2 2021-02-10T04:34:39.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: ) 1001 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:39.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:39.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: 34:38.756044+0000 osd.2 (osd.2) 2021-02-10T04:34:39.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: 1002 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:39.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: cluster 2021-02-10T04:34:38.756051+0000 osd.2 (osd.2) 2021-02-10T04:34:39.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: 1003 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:39.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:39.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: 02-10T04:34:38.756057+0000 osd.2 (osd.2) 2021-02-10T04:34:39.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: 1004 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:39.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:39.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: 34:38.756062+0000 osd.2 (osd.2) 2021-02-10T04:34:39.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: 1005 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:39.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:39.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: 34:38.756074+0000 osd.2 (osd.2 2021-02-10T04:34:39.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[35961]: ) 1006 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:40.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:39 smithi085 conmon[49106]: debug 2021-02-10T04:34:39.865+0000 7f77a3e94700 -1 osd.2 462 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:40.444 INFO:tasks.rados.rados.0.smithi085.stdout:1756: done (11 left) 2021-02-10T04:34:40.445 INFO:tasks.rados.rados.0.smithi085.stdout:1757: done (10 left) 2021-02-10T04:34:40.445 INFO:tasks.rados.rados.0.smithi085.stdout:1758: done (9 left) 2021-02-10T04:34:40.445 INFO:tasks.rados.rados.0.smithi085.stdout:1759: read oid 2 snap -1 2021-02-10T04:34:40.445 INFO:tasks.rados.rados.0.smithi085.stdout:1759: expect (ObjNum 609 snap 185 seq_num 609) 2021-02-10T04:34:40.445 INFO:tasks.rados.rados.0.smithi085.stdout:1760: rollback oid 14 current snap is 190 2021-02-10T04:34:40.445 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 14 to 188 2021-02-10T04:34:40.446 INFO:tasks.rados.rados.0.smithi085.stdout:1761: read oid 10 snap -1 2021-02-10T04:34:40.446 INFO:tasks.rados.rados.0.smithi085.stdout:1761: expect (ObjNum 608 snap 185 seq_num 608) 2021-02-10T04:34:40.446 INFO:tasks.rados.rados.0.smithi085.stdout:1762: delete oid 3 current snap is 190 2021-02-10T04:34:40.449 INFO:tasks.rados.rados.0.smithi085.stdout:1760: finishing rollback tid 0 to smithi08575397-14 2021-02-10T04:34:40.449 INFO:tasks.rados.rados.0.smithi085.stdout:1760: finishing rollback tid 1 to smithi08575397-14 2021-02-10T04:34:40.449 INFO:tasks.rados.rados.0.smithi085.stdout:1760: finishing rollback tid 2 to smithi08575397-14 2021-02-10T04:34:40.450 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 766 (ObjNum 613 snap 186 seq_num 613) dirty exists 2021-02-10T04:34:40.455 INFO:tasks.rados.rados.0.smithi085.stdout:1760: done (12 left) 2021-02-10T04:34:40.455 INFO:tasks.rados.rados.0.smithi085.stdout:1762: done (11 left) 2021-02-10T04:34:40.455 INFO:tasks.rados.rados.0.smithi085.stdout:1763: write (excl) oid 21 current snap is 190 2021-02-10T04:34:40.455 INFO:tasks.rados.rados.0.smithi085.stdout:1763: seq_num 624 ranges {487568=496322,1431705=519550} 2021-02-10T04:34:40.461 INFO:tasks.rados.rados.0.smithi085.stdout:1763: writing smithi08575397-21 from 487568 to 983890 tid 1 2021-02-10T04:34:40.467 INFO:tasks.rados.rados.0.smithi085.stdout:1763: writing smithi08575397-21 from 1431705 to 1951255 tid 2 2021-02-10T04:34:40.469 INFO:tasks.rados.rados.0.smithi085.stdout:1759: expect (ObjNum 609 snap 185 seq_num 609) 2021-02-10T04:34:40.618 INFO:tasks.rados.rados.0.smithi085.stdout:1761: expect (ObjNum 608 snap 185 seq_num 608) 2021-02-10T04:34:40.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[33677]: debug 2021-02-10T04:34:40.318+0000 7fbec19a1700 -1 osd.4 462 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: cluster 2021-02-10T04:34:39.336186+0000 osd.4 (osd 2021-02-10T04:34:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: .4) 599 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: cluster 2021-02-10T04:34:39.336195+0000 osd.4 (osd.4) 600 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: cluster 2021-02-10T04:34:39 2021-02-10T04:34:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: .336202+0000 osd.4 (osd.4) 601 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: -02-10T04:34:39.336208+0000 osd.4 (osd 2021-02-10T04:34:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: .4) 602 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:40.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: 10T04:34:39.426557+0000 mon.a (mon.0) 2021-02-10T04:34:40.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: 1270 : cluster [DBG] osdmap e462: 8 total, 8 up, 8 in 2021-02-10T04:34:40.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: cluster 2021-02-10T04:34:39.865622+0000 2021-02-10T04:34:40.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: osd.2 (osd.2) 1007 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:40.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: cluster 2021-02-10T04:34:39.865631+0000 osd.2 (osd.2) 1008 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:40.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: cluster 2021-02-10T04:34:39.865639+0000 2021-02-10T04:34:40.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: osd.2 (osd.2) 1009 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:40.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:40.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: -02-10T04:34:39.865646+0000 osd.2 (osd 2021-02-10T04:34:40.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: .2) 1010 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:40.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: cluster 2021-02-10T04:34:39.865655+0000 2021-02-10T04:34:40.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: osd.2 (osd.2) 1011 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:40.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: cluster 2021-02-10T04:34:39 2021-02-10T04:34:40.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: .865661+0000 osd.2 (osd.2) 1012 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:40.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:40.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: 04:34:39.865668+0000 osd.2 (osd.2) 2021-02-10T04:34:40.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: 1013 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:40.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:40.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: 02-10T04:34:39.865676 2021-02-10T04:34:40.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1014 : cluster [WRN] 2021-02-10T04:34:40.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:40.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:40.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: 04:34:39.865682+0000 osd.2 (osd 2021-02-10T04:34:40.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: .2) 1015 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:40.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: cluster 2021-02-10T04:34:40.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: 2021-02-10T04:34: 2021-02-10T04:34:40.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: 39.865688+0000 osd.2 (osd.2 2021-02-10T04:34:40.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: ) 1016 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:40.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:40.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: -02-10T04:34:40.089014+0000 mgr.y 2021-02-10T04:34:40.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:40 smithi164 conmon[29288]: (mgr.14140) 565 : cluster [DBG] pgmap v843: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 375 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 258 KiB/s wr, 4 op/s 2021-02-10T04:34:40.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: cluster 2021-02-10T04:34:40.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: 2021-02-10T04:34:40.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: 2021-02-10T04:34:39. 2021-02-10T04:34:40.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: 336186+0000 osd.4 (osd.4) 599 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:40.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:40.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: 04:34:39.336195+0000 osd.4 (osd. 2021-02-10T04:34:40.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: 4) 600 : cluster [WRN] 2021-02-10T04:34:40.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:40.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:40.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: 04:34:39.336202+0000 osd.4 (osd 2021-02-10T04:34:40.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: .4) 601 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:40.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:40.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: -02-10T04:34:39.336208 2021-02-10T04:34:40.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 602 : cluster [WRN] 2021-02-10T04:34:40.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:40.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:34:40.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: 34:39.426557+0000 mon.a (mon.0) 2021-02-10T04:34:40.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: 1270 : cluster [DBG] osdmap e462: 8 total, 8 up, 8 in 2021-02-10T04:34:40.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:34:40.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: T04:34:39.865622+0000 osd.2 (osd 2021-02-10T04:34:40.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: .2) 1007 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:40.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:34:40.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: :34:39.865631+0000 osd.2 (osd.2) 2021-02-10T04:34:40.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: 1008 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:40.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:34:40.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: 34:39.865639+0000 osd.2 (osd.2) 1009 2021-02-10T04:34:40.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:40.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:34:40.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: :34:39.865646+0000 osd.2 (osd.2) 1010 2021-02-10T04:34:40.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:40.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: cluster 2021-02-10T04:34 2021-02-10T04:34:40.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: :39.865655+0000 osd.2 (osd.2) 1011 2021-02-10T04:34:40.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:40.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: cluster 2021-02-10T04:34: 2021-02-10T04:34:40.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: 39.865661+0000 osd.2 (osd.2) 1012 : cluster 2021-02-10T04:34:40.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:40.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:40.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: 04:34:39.865668+0000 osd.2 (osd.2) 1013 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:40.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: 2021-02-10T04:34:40.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:40.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: 02-10T04:34:39.865676+0000 osd.2 (osd.2) 1014 2021-02-10T04:34:40.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:40.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: cluster 2021-02-10T04:34:39.865682 2021-02-10T04:34:40.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1015 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:40.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: 2021-02-10T04:34:40.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: cluster 2021-02-10T04:34:40.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: 2021-02-10T04:34:39.865688+0000 osd.2 (osd.2) 1016 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:40.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:40.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: 02-10T04:34:40.089014+0000 2021-02-10T04:34:40.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: mgr.y (mgr.14140) 565 : cluster [DBG] 2021-02-10T04:34:40.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[29319]: pgmap v843: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 375 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 258 KiB/s wr, 4 op/s 2021-02-10T04:34:40.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: cluster 2021-02-10T04:34:39.336186+0000 2021-02-10T04:34:40.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: osd.4 (osd.4) 599 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:40.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:34:40.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: 04:34:39.336195+0000 osd.4 (osd.4) 600 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:40.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:40.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: -10T04:34:39.336202+0000 osd.4 ( 2021-02-10T04:34:40.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: osd.4) 601 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:40.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:40.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: 02-10T04:34:39.336208+0000 2021-02-10T04:34:40.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: osd.4 (osd.4) 602 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:40.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: 2021-02-10T04:34:40.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:40.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: -02-10T04:34:39.426557+0000 mon.a (mon. 2021-02-10T04:34:40.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: 0) 1270 : cluster [DBG] osdmap e462: 8 total, 8 up, 8 in 2021-02-10T04:34:40.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:40.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: T04:34:39.865622+0000 osd.2 (osd. 2021-02-10T04:34:40.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: 2) 1007 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:40.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:34:40.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: 04:34:39.865631+0000 osd.2 (osd. 2021-02-10T04:34:40.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: 2) 1008 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:40.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:40.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: T04:34:39.865639+0000 osd.2 (osd 2021-02-10T04:34:40.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: .2) 1009 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:40.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:40.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: -10T04:34:39.865646+0000 osd.2 2021-02-10T04:34:40.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: (osd.2) 1010 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:40.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: cluster 2021-02-10T04:34:39.865655+0000 osd.2 (osd.2) 1011 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:40.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:40.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: 34:39.865661+0000 osd.2 (osd.2) 2021-02-10T04:34:40.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: 1012 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:40.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:40.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: 02-10T04:34:39.865668+0000 2021-02-10T04:34:40.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: osd.2 (osd.2) 1013 : cluster [WRN] 2021-02-10T04:34:40.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:40.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: cluster 2021-02-10T04:34:40.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: 2021-02-10T04:34:39.865676+0000 osd.2 (osd. 2021-02-10T04:34:40.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: 2) 1014 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:40.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:40.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: T04:34:39.865682+0000 osd.2 (osd 2021-02-10T04:34:40.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: .2) 1015 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:40.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:40.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: 02-10T04:34:39.865688+0000 osd.2 2021-02-10T04:34:40.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: (osd.2) 1016 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:40.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: 2021-02-10T04:34:40.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: cluster 2021-02-10T04:34:40.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: 2021-02-10T04:34:40.089014+0000 mgr.y ( 2021-02-10T04:34:40.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[35961]: mgr.14140) 565 : cluster [DBG] pgmap v843: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 375 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 258 KiB/s wr, 4 op/s 2021-02-10T04:34:40.743 INFO:tasks.rados.rados.0.smithi085.stdout:1763: finishing write tid 1 to smithi08575397-21 2021-02-10T04:34:40.743 INFO:tasks.rados.rados.0.smithi085.stdout:1763: finishing write tid 2 to smithi08575397-21 2021-02-10T04:34:40.744 INFO:tasks.rados.rados.0.smithi085.stdout:1763: finishing write tid 3 to smithi08575397-21 2021-02-10T04:34:40.744 INFO:tasks.rados.rados.0.smithi085.stdout:1763: finishing write tid 4 to smithi08575397-21 2021-02-10T04:34:40.744 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 951 (ObjNum 624 snap 190 seq_num 624) dirty exists 2021-02-10T04:34:40.744 INFO:tasks.rados.rados.0.smithi085.stdout:1763: left oid 21 (ObjNum 624 snap 190 seq_num 624) 2021-02-10T04:34:40.744 INFO:tasks.rados.rados.0.smithi085.stdout:1759: done (11 left) 2021-02-10T04:34:40.744 INFO:tasks.rados.rados.0.smithi085.stdout:1761: done (10 left) 2021-02-10T04:34:40.744 INFO:tasks.rados.rados.0.smithi085.stdout:1763: done (9 left) 2021-02-10T04:34:40.744 INFO:tasks.rados.rados.0.smithi085.stdout:1764: snap_remove snap 161 2021-02-10T04:34:41.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:40 smithi085 conmon[49106]: debug 2021-02-10T04:34:40.857+0000 7f77a3e94700 -1 osd.2 463 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:41.449 INFO:tasks.rados.rados.0.smithi085.stdout:1764: done (9 left) 2021-02-10T04:34:41.450 INFO:tasks.rados.rados.0.smithi085.stdout:1765: read oid 12 snap 165 2021-02-10T04:34:41.450 INFO:tasks.rados.rados.0.smithi085.stdout:1765: expect (ObjNum 85 snap 11 seq_num 85) 2021-02-10T04:34:41.450 INFO:tasks.rados.rados.0.smithi085.stdout:1766: read oid 30 snap -1 2021-02-10T04:34:41.450 INFO:tasks.rados.rados.0.smithi085.stdout:1766: expect (ObjNum 618 snap 187 seq_num 618) 2021-02-10T04:34:41.450 INFO:tasks.rados.rados.0.smithi085.stdout:1767: snap_remove snap 156 2021-02-10T04:34:41.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[33677]: debug 2021-02-10T04:34:41.272+0000 7fbec19a1700 -1 osd.4 463 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: cluster 2021-02-10T04:34:40.318896+0000 osd.4 2021-02-10T04:34:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: (osd.4) 603 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: cluster 2021-02-10T04:34:40.318907+0000 2021-02-10T04:34:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: osd.4 (osd.4) 604 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: :34:40.318914+0000 osd.4 (osd.4) 2021-02-10T04:34:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: 605 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: cluster 2021-02-10T04:34:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: 2021-02-10T04:34:40.318920+0000 osd.4 (osd 2021-02-10T04:34:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: .4) 606 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: -10T04:34:40.442285 2021-02-10T04:34:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2021-02-10T04:34:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: 1271 : cluster [DBG] osdmap e463: 8 total, 8 up, 8 in 2021-02-10T04:34:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:41.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: 34:40.857540+0000 osd.2 (osd.2) 1017 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:41.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: cluster 2021-02-10T04:34: 2021-02-10T04:34:41.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: 40.857562+0000 osd.2 (osd.2) 1018 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:41.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: cluster 2021-02-10T04:34:40. 2021-02-10T04:34:41.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: 857577+0000 osd.2 (osd.2) 1019 : cluster [WRN] 2021-02-10T04:34:41.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:41.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: cluster 2021-02-10T04:34: 2021-02-10T04:34:41.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: 40.857584+0000 osd.2 (osd.2 2021-02-10T04:34:41.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: ) 1020 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:41.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:41.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: -10T04:34:40.857591+0000 osd.2 (osd 2021-02-10T04:34:41.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: .2) 1021 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:41.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:41.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: 10T04:34:40.857598+0000 2021-02-10T04:34:41.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: osd.2 (osd.2) 1022 : cluster [WRN] 2021-02-10T04:34:41.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:41.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: cluster 2021-02-10T04:34:40. 2021-02-10T04:34:41.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: 857605+0000 osd.2 (osd.2) 1023 : cluster 2021-02-10T04:34:41.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:41.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: cluster 2021-02-10T04:34:40 2021-02-10T04:34:41.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: .857611+0000 2021-02-10T04:34:41.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: osd.2 (osd.2) 1024 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:41.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:41.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: -02-10T04:34:40.857617+0000 osd.2 (osd 2021-02-10T04:34:41.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: .2) 1025 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:41.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:41.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: 10T04:34:40.857623+0000 osd.2 (osd.2) 1026 : 2021-02-10T04:34:41.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:41 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:41.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: cluster 2021-02-10T04:34:40. 2021-02-10T04:34:41.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: 318896+0000 osd.4 (osd.4) 603 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:41.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:41.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: 10T04:34:40.318907+0000 osd.4 (osd. 2021-02-10T04:34:41.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: 4) 604 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:41.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:41.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: 02-10T04:34:40.318914+0000 2021-02-10T04:34:41.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: osd.4 (osd.4) 605 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:41.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: 2021-02-10T04:34:41.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:41.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: 02-10T04:34:40.318920+0000 osd.4 (osd.4 2021-02-10T04:34:41.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: ) 606 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:41.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: cluster 2021-02-10T04:34: 2021-02-10T04:34:41.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: 40.442285+0000 mon.a (mon.0) 1271 : cluster [DBG] osdmap e463: 8 total, 8 up, 8 in 2021-02-10T04:34:41.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: -02-10T04:34:40.857540+0000 osd.2 2021-02-10T04:34:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: (osd.2) 1017 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:34:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: 04:34:40.857562+0000 osd.2 (osd.2 2021-02-10T04:34:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: ) 1018 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: T04:34:40.857577+0000 osd.2 (osd 2021-02-10T04:34:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: .2) 1019 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: 10T04:34:40.857584+0000 osd.2 2021-02-10T04:34:41.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: (osd.2) 1020 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:41.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:41.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: -10T04:34:40.857591+0000 osd.2 ( 2021-02-10T04:34:41.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: osd.2) 1021 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:41.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:41.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: -02-10T04:34:40.857598+0000 2021-02-10T04:34:41.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: osd.2 (osd.2) 1022 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:41.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: 2021-02-10T04:34:41.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: cluster 2021-02-10T04:34:41.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: 2021 2021-02-10T04:34:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: - 2021-02-10T04:34:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: 02 2021-02-10T04:34:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: - 2021-02-10T04:34:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: 10T04:34: 2021-02-10T04:34:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: 40.857605+0000 osd.2 (osd.2) 1023 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: 2021-02-10T04:34:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: -10T04:34:40.857611+0000 osd.2 (osd.2) 2021-02-10T04:34:41.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: 1024 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:41.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: cluster 2021-02-10T04:34:41.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: 2021-02-10T04:34:40. 2021-02-10T04:34:41.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: 857617+0000 osd.2 (osd.2) 1025 : cluster 2021-02-10T04:34:41.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:41.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:41.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: 34:40.857623+0000 osd.2 (osd.2) 1026 : 2021-02-10T04:34:41.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:41.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: cluster 2021-02-10T04:34:40.318896+0000 osd.4 (osd 2021-02-10T04:34:41.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: .4) 603 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:41.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: cluster 2021-02-10T04:34:40.318907+0000 osd.4 (osd.4) 604 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:41.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: cluster 2021-02-10T04:34:40.318914+0000 osd.4 (osd.4) 2021-02-10T04:34:41.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: 605 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:41.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:41.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: 02-10T04:34:40. 2021-02-10T04:34:41.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: 318920+0000 osd.4 (osd.4) 606 : cluster 2021-02-10T04:34:41.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:41.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:41.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: -10T04:34:40.442285+0000 2021-02-10T04:34:41.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: mon.a (mon.0) 1271 : cluster [DBG] osdmap e463: 8 total, 8 up, 8 in 2021-02-10T04:34:41.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:34:41.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: T04:34:40.857540+0000 osd.2 (osd. 2021-02-10T04:34:41.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: 2) 1017 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:41.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: cluster 2021-02-10T04:34:41.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: 2021-02-10T04:34 2021-02-10T04:34:41.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: :40.857562+0000 osd.2 (osd.2 2021-02-10T04:34:41.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: ) 1018 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:41.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:41.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: -02-10T04:34: 2021-02-10T04:34:41.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: 40.857577+0000 osd.2 (osd.2) 1019 2021-02-10T04:34:41.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:41.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:41.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: -10T04:34:40.857584 2021-02-10T04:34:41.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1020 : cluster 2021-02-10T04:34:41.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:41.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:34:41.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: T04:34:40.857591+0000 osd.2 2021-02-10T04:34:41.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: (osd.2) 1021 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:41.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: 2021-02-10T04:34:41.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: cluster 2021-02-10T04:34:41.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: 2021-02-10T04:34:40 2021-02-10T04:34:41.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: .857598+0000 osd.2 (osd.2) 1022 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:41.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:41.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: 02-10T04:34:40. 2021-02-10T04:34:41.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: 857605+0000 osd.2 (osd.2) 1023 : cluster 2021-02-10T04:34:41.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:41.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:41.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: 10T04:34:40. 2021-02-10T04:34:41.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: 857611+0000 osd.2 (osd.2) 1024 : cluster 2021-02-10T04:34:41.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:41.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:41.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: -10T04:34:40.857617 2021-02-10T04:34:41.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1025 : cluster 2021-02-10T04:34:41.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:41.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:41.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: 10T04:34:40.857623+0000 2021-02-10T04:34:41.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: osd.2 (osd.2) 1026 : cluster [WRN] 2021-02-10T04:34:41.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:42.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:41 smithi085 conmon[49106]: debug 2021-02-10T04:34:41.809+0000 7f77a3e94700 -1 osd.2 464 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:42.458 INFO:tasks.rados.rados.0.smithi085.stdout:1767: done (11 left) 2021-02-10T04:34:42.459 INFO:tasks.rados.rados.0.smithi085.stdout:1768: rollback oid 2 current snap is 190 2021-02-10T04:34:42.459 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 2 to 123 2021-02-10T04:34:42.459 INFO:tasks.rados.rados.0.smithi085.stdout:1766: expect (ObjNum 618 snap 187 seq_num 618) 2021-02-10T04:34:42.607 INFO:tasks.rados.rados.0.smithi085.stdout:1765: expect (ObjNum 85 snap 11 seq_num 85) 2021-02-10T04:34:42.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[33677]: debug 2021-02-10T04:34:42.307+0000 7fbec19a1700 -1 osd.4 464 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:42.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: cluster 2021-02-10T04:34:41.273624+0000 osd.4 (osd. 2021-02-10T04:34:42.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: 4) 607 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:42.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: cluster 2021-02-10T04:34:41.273630+0000 osd.4 (osd.4) 608 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:42.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: cluster 2021-02-10T04:34:41.273636+0000 osd.4 (osd.4) 2021-02-10T04:34:42.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: 609 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:42.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: cluster 2021-02-10T04:34:41. 2021-02-10T04:34:42.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: 273642+0000 osd.4 (osd.4) 610 : cluster 2021-02-10T04:34:42.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:42.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: cluster 2021-02-10T04:34:41. 2021-02-10T04:34:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: 448902+0000 mon.a (mon.0) 1272 : cluster 2021-02-10T04:34:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: [DBG] osdmap e464: 8 total, 8 up, 8 in 2021-02-10T04:34:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: cluster 2021-02-10T04:34:41. 2021-02-10T04:34:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: 810083+0000 osd.2 (osd.2) 1027 : cluster 2021-02-10T04:34:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: cluster 2021-02-10T04:34:41. 2021-02-10T04:34:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: 810091+0000 osd.2 (osd.2) 1028 : cluster 2021-02-10T04:34:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: cluster 2021-02-10T04:34:41 2021-02-10T04:34:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: .810097+0000 osd.2 (osd.2) 1029 : cluster 2021-02-10T04:34:42.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:42.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: cluster 2021-02-10T04:34:41 2021-02-10T04:34:42.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: .810104+0000 osd.2 (osd.2) 1030 : 2021-02-10T04:34:42.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:42.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: cluster 2021-02-10T04:34: 2021-02-10T04:34:42.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: 41.810110+0000 osd.2 (osd.2) 1031 : 2021-02-10T04:34:42.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:42.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: cluster 2021-02-10T04:34 2021-02-10T04:34:42.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: :41.810116+0000 osd.2 (osd.2) 1032 2021-02-10T04:34:42.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:42.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:42.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: 34:41.810123+0000 osd.2 (osd.2) 2021-02-10T04:34:42.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: 1033 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:42.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:42.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: :34:41.810132+0000 osd.2 (osd.2) 2021-02-10T04:34:42.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: 1034 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:42.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:42.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: 34:41.810138+0000 osd.2 (osd.2) 2021-02-10T04:34:42.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: 1035 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:42.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:42.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: :34:41.810145+0000 osd.2 (osd. 2021-02-10T04:34:42.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: 2) 1036 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:42.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:42.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: T04:34:42.089625+0000 mgr.y (mgr 2021-02-10T04:34:42.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:42 smithi164 conmon[29288]: .14140) 566 : cluster [DBG] pgmap v846: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 373 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.6 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2021-02-10T04:34:42.784 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: cluster 2021-02-10T04:34:42.784 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: 2021-02-10T04:34:42.784 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: 2021-02-10T04:34:41 2021-02-10T04:34:42.784 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: .273624+0000 osd.4 (osd.4) 607 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:42.784 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: cluster 2021-02-10T04:34:42.784 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: 2021-02-10T04:34:41.273630+0000 osd.4 2021-02-10T04:34:42.784 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: (osd.4) 608 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:42.784 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:42.784 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: 10T04:34:41.273636+0000 osd.4 (osd 2021-02-10T04:34:42.785 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: .4) 609 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:42.785 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:34:42.785 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: :34:41.273642+0000 osd.4 (osd.4) 610 2021-02-10T04:34:42.785 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:42.785 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: cluster 2021-02-10T04:34:41. 2021-02-10T04:34:42.785 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: 448902+0000 mon.a (mon.0) 1272 : cluster 2021-02-10T04:34:42.785 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: [DBG] osdmap e464: 8 total, 8 up, 8 in 2021-02-10T04:34:42.785 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: cluster 2021-02-10T04:34 2021-02-10T04:34:42.785 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: :41.810083+0000 osd.2 (osd.2) 1027 : 2021-02-10T04:34:42.785 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:42.785 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: cluster 2021-02-10T04:34:41 2021-02-10T04:34:42.785 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: .810091+0000 osd.2 (osd.2) 1028 : cluster [WRN] 2021-02-10T04:34:42.785 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:42.786 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: cluster 2021-02-10T04:34:42.786 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: 2021-02-10T04:34:41.810097+0000 osd.2 (osd.2) 2021-02-10T04:34:42.786 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: 1029 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:42.786 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: cluster 2021-02-10T04:34:41 2021-02-10T04:34:42.786 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: .810104+0000 osd.2 (osd.2) 1030 : cluster [WRN] 2021-02-10T04:34:42.786 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:42.786 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: cluster 2021-02-10T04:34:42.786 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: 2021-02-10T04:34:41.810110+0000 osd.2 (osd.2) 1031 : 2021-02-10T04:34:42.786 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:42.786 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:34:42.786 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: 34:41.810116+0000 osd.2 (osd.2) 1032 : 2021-02-10T04:34:42.786 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:42.787 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: cluster 2021-02-10T04:34: 2021-02-10T04:34:42.787 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: 41.810123+0000 osd.2 (osd.2) 1033 : cluster 2021-02-10T04:34:42.787 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:42.787 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: cluster 2021-02-10T04:34:41.810132+0000 2021-02-10T04:34:42.787 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: osd.2 (osd.2) 1034 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:42.787 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:42.787 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: 02-10T04:34:41.810138+0000 osd.2 ( 2021-02-10T04:34:42.787 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: osd.2) 1035 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:42.787 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:34:42.787 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: :34:41.810145+0000 osd.2 (osd.2) 1036 : cluster 2021-02-10T04:34:42.787 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:42.787 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: cluster 2021-02-10T04:34:42.089625 2021-02-10T04:34:42.787 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 566 : cluster [DBG] 2021-02-10T04:34:42.787 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[29319]: pgmap v846: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 373 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.6 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2021-02-10T04:34:42.788 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: cluster 2021-02-10T04:34:42.788 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: 2021-02-10T04:34:42.788 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: 2021 2021-02-10T04:34:42.788 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: - 2021-02-10T04:34:42.788 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: 02 2021-02-10T04:34:42.788 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: - 2021-02-10T04:34:42.788 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: 10 2021-02-10T04:34:42.788 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: T 2021-02-10T04:34:42.788 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: 04:34:41.273624+0000 osd.4 2021-02-10T04:34:42.788 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: (osd.4) 607 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:42.788 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: cluster 2021-02-10T04:34:42.788 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: 2021-02-10T04:34:42.789 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: 2021-02-10T04:34:41.273630+0000 2021-02-10T04:34:42.789 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: osd.4 (osd.4) 608 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:42.789 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: cluster 2021-02-10T04:34:42.789 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: 2021 2021-02-10T04:34:42.789 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: - 2021-02-10T04:34:42.789 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: 02 2021-02-10T04:34:42.789 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: - 2021-02-10T04:34:42.789 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: 10 2021-02-10T04:34:42.789 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: T 2021-02-10T04:34:42.789 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: 04 2021-02-10T04:34:42.789 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: : 2021-02-10T04:34:42.789 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: 34 2021-02-10T04:34:42.789 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: : 2021-02-10T04:34:42.789 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: 41 2021-02-10T04:34:42.790 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: . 2021-02-10T04:34:42.790 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: 273636 2021-02-10T04:34:42.790 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: +0000 2021-02-10T04:34:42.790 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: 2021-02-10T04:34:42.790 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: osd.4 2021-02-10T04:34:42.790 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: ( 2021-02-10T04:34:42.790 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: osd.4) 609 : cluster [WRN] 2021-02-10T04:34:42.790 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:42.790 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: cluster 2021-02-10T04:34 2021-02-10T04:34:42.790 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: :41.273642+0000 osd.4 (osd.4) 610 : cluster 2021-02-10T04:34:42.790 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:42.790 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:42.791 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: 02-10T04:34:41.448902+0000 2021-02-10T04:34:42.791 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: mon.a (mon.0) 1272 : cluster [DBG] osdmap e464: 8 total, 8 up, 8 in 2021-02-10T04:34:42.791 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: 2021-02-10T04:34:42.791 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:34:42.791 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: :34:41.810083+0000 osd.2 (osd.2 2021-02-10T04:34:42.791 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: ) 1027 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:42.791 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:42.791 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: 02-10T04:34:41.810091+0000 osd.2 2021-02-10T04:34:42.791 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: (osd.2) 1028 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:42.791 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: cluster 2021-02-10T04:34:42.791 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: 2021-02-10T04:34:41.810097 2021-02-10T04:34:42.791 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1029 : cluster 2021-02-10T04:34:42.791 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:42.792 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: cluster 2021-02-10T04:34:42.792 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: 2021 2021-02-10T04:34:42.792 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: - 2021-02-10T04:34:42.792 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: 02 2021-02-10T04:34:42.792 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: - 2021-02-10T04:34:42.792 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: 10 2021-02-10T04:34:42.792 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: T 2021-02-10T04:34:42.792 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: 04 2021-02-10T04:34:42.792 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: :34:41.810104+0000 2021-02-10T04:34:42.792 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: osd.2 (osd.2) 1030 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:42.792 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: cluster 2021-02-10T04:34:42.792 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: 2021-02-10T04:34 2021-02-10T04:34:42.792 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: :41.810110+0000 osd.2 (osd 2021-02-10T04:34:42.793 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: .2) 1031 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:42.793 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: cluster 2021-02-10T04:34:42.793 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:34:42.793 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: T04:34:41.810116+0000 osd.2 (osd.2) 1032 : 2021-02-10T04:34:42.793 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:42.793 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:42.793 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: T04:34:41.810123+0000 osd.2 (osd. 2021-02-10T04:34:42.793 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: 2) 1033 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:42.793 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:42.793 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: 10T04:34:41.810132+0000 osd.2 (osd. 2021-02-10T04:34:42.793 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: 2) 1034 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:42.793 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:42.793 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: -10T04:34:41.810138+0000 osd.2 ( 2021-02-10T04:34:42.793 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: osd.2) 1035 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:42.794 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:42.794 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: -02-10T04:34:41.810145+0000 2021-02-10T04:34:42.794 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: osd.2 (osd.2) 1036 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:42.794 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:42.794 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: 02-10T04:34:42.089625+0000 2021-02-10T04:34:42.794 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[35961]: mgr.y (mgr.14140) 566 : cluster [DBG] pgmap v846: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 373 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.6 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2021-02-10T04:34:42.877 INFO:tasks.rados.rados.0.smithi085.stdout:1768: finishing rollback tid 0 to smithi08575397-2 2021-02-10T04:34:42.877 INFO:tasks.rados.rados.0.smithi085.stdout:1768: finishing rollback tid 1 to smithi08575397-2 2021-02-10T04:34:42.877 INFO:tasks.rados.rados.0.smithi085.stdout:1768: finishing rollback tid 2 to smithi08575397-2 2021-02-10T04:34:42.878 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 643 (ObjNum 422 snap 109 seq_num 422) dirty exists 2021-02-10T04:34:42.878 INFO:tasks.rados.rados.0.smithi085.stdout:1765: done (11 left) 2021-02-10T04:34:42.878 INFO:tasks.rados.rados.0.smithi085.stdout:1766: done (10 left) 2021-02-10T04:34:42.878 INFO:tasks.rados.rados.0.smithi085.stdout:1768: done (9 left) 2021-02-10T04:34:42.878 INFO:tasks.rados.rados.0.smithi085.stdout:1769: read oid 6 snap -1 2021-02-10T04:34:42.878 INFO:tasks.rados.rados.0.smithi085.stdout:1769: expect deleted 2021-02-10T04:34:42.878 INFO:tasks.rados.rados.0.smithi085.stdout:1770: rollback oid 12 current snap is 190 2021-02-10T04:34:42.878 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 12 to 181 2021-02-10T04:34:42.878 INFO:tasks.rados.rados.0.smithi085.stdout:1771: read oid 23 snap -1 2021-02-10T04:34:42.878 INFO:tasks.rados.rados.0.smithi085.stdout:1771: expect (ObjNum 617 snap 187 seq_num 617) 2021-02-10T04:34:42.878 INFO:tasks.rados.rados.0.smithi085.stdout:1772: read oid 37 snap 184 2021-02-10T04:34:42.878 INFO:tasks.rados.rados.0.smithi085.stdout:1772: expect deleted 2021-02-10T04:34:42.878 INFO:tasks.rados.rados.0.smithi085.stdout:1773: read oid 46 snap 165 2021-02-10T04:34:42.879 INFO:tasks.rados.rados.0.smithi085.stdout:1773: expect (ObjNum 510 snap 148 seq_num 510) 2021-02-10T04:34:42.879 INFO:tasks.rados.rados.0.smithi085.stdout:1774: copy_from oid 42 from oid 8 current snap is 190 2021-02-10T04:34:42.879 INFO:tasks.rados.rados.0.smithi085.stdout:1775: write oid 1 current snap is 190 2021-02-10T04:34:42.879 INFO:tasks.rados.rados.0.smithi085.stdout:1775: seq_num 625 ranges {504375=571571,1490187=663208} 2021-02-10T04:34:42.883 INFO:tasks.rados.rados.0.smithi085.stdout:1775: writing smithi08575397-1 from 504375 to 1075946 tid 1 2021-02-10T04:34:42.890 INFO:tasks.rados.rados.0.smithi085.stdout:1775: writing smithi08575397-1 from 1490187 to 2153395 tid 2 2021-02-10T04:34:42.890 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:34:42.891 INFO:tasks.rados.rados.0.smithi085.stdout:1770: finishing rollback tid 0 to smithi08575397-12 2021-02-10T04:34:42.891 INFO:tasks.rados.rados.0.smithi085.stdout:1773: expect (ObjNum 510 snap 148 seq_num 510) 2021-02-10T04:34:43.091 INFO:tasks.rados.rados.0.smithi085.stdout:1770: finishing rollback tid 1 to smithi08575397-12 2021-02-10T04:34:43.092 INFO:tasks.rados.rados.0.smithi085.stdout:1770: finishing rollback tid 2 to smithi08575397-12 2021-02-10T04:34:43.092 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 769 (ObjNum 85 snap 11 seq_num 85) dirty exists 2021-02-10T04:34:43.092 INFO:tasks.rados.rados.0.smithi085.stdout:1769: done (15 left) 2021-02-10T04:34:43.092 INFO:tasks.rados.rados.0.smithi085.stdout:1770: done (14 left) 2021-02-10T04:34:43.092 INFO:tasks.rados.rados.0.smithi085.stdout:1772: done (13 left) 2021-02-10T04:34:43.092 INFO:tasks.rados.rados.0.smithi085.stdout:1773: done (12 left) 2021-02-10T04:34:43.092 INFO:tasks.rados.rados.0.smithi085.stdout:1776: delete oid 10 current snap is 190 2021-02-10T04:34:43.092 INFO:tasks.rados.rados.0.smithi085.stdout:1775: finishing write tid 1 to smithi08575397-1 2021-02-10T04:34:43.092 INFO:tasks.rados.rados.0.smithi085.stdout:1771: expect (ObjNum 617 snap 187 seq_num 617) 2021-02-10T04:34:43.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[49106]: debug 2021-02-10T04:34:43.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[49106]: 2021-02-10T04:34:42.781+0000 7f77a3e94700 -1 osd.2 465 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:43.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:42 smithi085 conmon[49106]: 2021-02-10T04:34:43.286 INFO:tasks.rados.rados.0.smithi085.stdout:1775: finishing write tid 2 to smithi08575397-1 2021-02-10T04:34:43.286 INFO:tasks.rados.rados.0.smithi085.stdout:1775: finishing write tid 3 to smithi08575397-1 2021-02-10T04:34:43.286 INFO:tasks.rados.rados.0.smithi085.stdout:1775: finishing write tid 4 to smithi08575397-1 2021-02-10T04:34:43.287 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 731 (ObjNum 625 snap 190 seq_num 625) dirty exists 2021-02-10T04:34:43.287 INFO:tasks.rados.rados.0.smithi085.stdout:1775: left oid 1 (ObjNum 625 snap 190 seq_num 625) 2021-02-10T04:34:43.287 INFO:tasks.rados.rados.0.smithi085.stdout:1774: finishing copy_from to smithi08575397-42 2021-02-10T04:34:43.287 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 599 (ObjNum 521 snap 155 seq_num 521) dirty exists 2021-02-10T04:34:43.287 INFO:tasks.rados.rados.0.smithi085.stdout:1774: finishing copy_from racing read to smithi08575397-42 2021-02-10T04:34:43.287 INFO:tasks.rados.rados.0.smithi085.stdout:1771: done (12 left) 2021-02-10T04:34:43.287 INFO:tasks.rados.rados.0.smithi085.stdout:1774: done (11 left) 2021-02-10T04:34:43.287 INFO:tasks.rados.rados.0.smithi085.stdout:1775: done (10 left) 2021-02-10T04:34:43.287 INFO:tasks.rados.rados.0.smithi085.stdout:1776: done (9 left) 2021-02-10T04:34:43.287 INFO:tasks.rados.rados.0.smithi085.stdout:1777: write oid 28 current snap is 190 2021-02-10T04:34:43.287 INFO:tasks.rados.rados.0.smithi085.stdout:1777: seq_num 627 ranges {537989=722069,1890798=647191} 2021-02-10T04:34:43.293 INFO:tasks.rados.rados.0.smithi085.stdout:1777: writing smithi08575397-28 from 537989 to 1260058 tid 1 2021-02-10T04:34:43.302 INFO:tasks.rados.rados.0.smithi085.stdout:1777: writing smithi08575397-28 from 1890798 to 2537989 tid 2 2021-02-10T04:34:43.303 INFO:tasks.rados.rados.0.smithi085.stdout:1778: snap_create 2021-02-10T04:34:43.303 INFO:tasks.rados.rados.0.smithi085.stdout:1777: finishing write tid 1 to smithi08575397-28 2021-02-10T04:34:43.305 INFO:tasks.rados.rados.0.smithi085.stdout:1777: finishing write tid 2 to smithi08575397-28 2021-02-10T04:34:43.305 INFO:tasks.rados.rados.0.smithi085.stdout:1777: finishing write tid 3 to smithi08575397-28 2021-02-10T04:34:43.306 INFO:tasks.rados.rados.0.smithi085.stdout:1777: finishing write tid 4 to smithi08575397-28 2021-02-10T04:34:43.306 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 911 (ObjNum 627 snap 190 seq_num 627) dirty exists 2021-02-10T04:34:43.306 INFO:tasks.rados.rados.0.smithi085.stdout:1777: left oid 28 (ObjNum 627 snap 190 seq_num 627) 2021-02-10T04:34:43.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[33677]: debug 2021-02-10T04:34:43.270+0000 7fbec19a1700 -1 osd.4 466 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:43.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: cluster 2021-02-10T04:34:42.307909+0000 osd.4 ( 2021-02-10T04:34:43.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: osd.4) 611 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:43.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: cluster 2021-02-10T04:34:42.307915+0000 osd.4 (osd. 2021-02-10T04:34:43.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: 4) 612 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:43.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: cluster 2021-02-10T04:34:43.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: 2021-02-10T04:34:42.307918+0000 osd.4 (osd 2021-02-10T04:34:43.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: .4) 613 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:43.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:43.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: -10T04:34:42.307922+0000 osd.4 (osd 2021-02-10T04:34:43.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: .4) 614 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:43.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:43.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: 02-10T04:34:42.457460+0000 mon.a (mon.0 2021-02-10T04:34:43.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: ) 1273 : cluster [DBG] osdmap e465: 8 total, 8 up, 8 in 2021-02-10T04:34:43.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: cluster 2021-02-10T04:34:42. 2021-02-10T04:34:43.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: 781930+0000 osd.2 (osd.2) 1037 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:43.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:43.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: 04:34:42.781937+0000 osd.2 (osd.2) 1038 : cluster 2021-02-10T04:34:43.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:43.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:43.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: -02-10T04:34:42.781943+0000 osd.2 (osd.2) 1039 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:43.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:43.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: :34:42.781949+0000 osd.2 (osd.2) 1040 2021-02-10T04:34:43.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:43.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: cluster 2021-02-10T04:34 2021-02-10T04:34:43.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: :42.781955+0000 osd.2 (osd.2) 1041 : cluster 2021-02-10T04:34:43.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:43.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:43.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: 04:34:42.781960+0000 osd.2 (osd.2 2021-02-10T04:34:43.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: ) 1042 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:43.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: cluster 2021-02-10T04:34:42 2021-02-10T04:34:43.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: .781968+0000 osd.2 (osd.2) 1043 : cluster 2021-02-10T04:34:43.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:43.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:43.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: -02-10T04:34:42.781974+0000 osd.2 (osd.2) 1044 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:43.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: 2021-02-10T04:34:43.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: cluster 2021-02-10T04:34:42. 2021-02-10T04:34:43.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: 781979+0000 osd.2 (osd.2) 1045 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:43.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: cluster 2021-02-10T04:34:43.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: 2021-02-10T04:34:42.781986+0000 osd.2 (osd.2) 1046 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:43.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:43.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: 04:34:43.138538+0000 mon.a (mon.0 2021-02-10T04:34:43.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: ) 1274 : cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 312 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:34:43.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:43.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: 34:43.143965+0000 mon.a (mon.0) 1275 : cluster [DBG] osdmap e466: 8 total, 8 up, 8 in 2021-02-10T04:34:43.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:43 smithi164 conmon[29288]: 2021-02-10T04:34:43.817 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: cluster 2021-02-10T04:34:43.817 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: 2021-02-10T04:34:43.817 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:34:43.817 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: :34:42.307909+0000 osd.4 (osd.4) 611 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:43.817 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: cluster 2021-02-10T04:34:43.817 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: 2021-02-10T04:34 2021-02-10T04:34:43.817 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: :42.307915+0000 osd.4 (osd.4) 612 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:43.817 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: cluster 2021-02-10T04:34:42.307918 2021-02-10T04:34:43.817 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 613 : cluster 2021-02-10T04:34:43.817 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: [WRN] 2021-02-10T04:34:43.818 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:43.818 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: cluster 2021-02-10T04:34 2021-02-10T04:34:43.818 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: :42.307922+0000 osd.4 (osd.4) 2021-02-10T04:34:43.818 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: 614 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:43.818 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:43.818 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: 10T04:34:42.457460+0000 mon.a ( 2021-02-10T04:34:43.818 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: mon.0) 1273 : cluster [DBG] osdmap e465: 8 total, 8 up, 8 in 2021-02-10T04:34:43.818 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: cluster 2021-02-10T04:34:43.818 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: 2021-02-10T04:34:42.781930+0000 osd.2 2021-02-10T04:34:43.818 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: (osd.2) 1037 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:43.818 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:43.818 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: 02-10T04:34:42.781937+0000 osd.2 2021-02-10T04:34:43.819 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: (osd.2) 1038 : cluster 2021-02-10T04:34:43.819 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:43.819 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:43.819 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: -02-10T04:34:42.781943 2021-02-10T04:34:43.819 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1039 : cluster [WRN] 2021-02-10T04:34:43.819 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:43.819 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: cluster 2021-02-10T04:34:42 2021-02-10T04:34:43.819 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: .781949+0000 osd.2 ( 2021-02-10T04:34:43.819 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: osd.2) 1040 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:43.819 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: cluster 2021-02-10T04:34:43.819 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: 2021-02-10T04:34:42.781955 2021-02-10T04:34:43.819 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1041 : cluster [WRN] 2021-02-10T04:34:43.819 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:43.820 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: cluster 2021-02-10T04:34:42 2021-02-10T04:34:43.820 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: .781960+0000 osd.2 (osd.2) 1042 : cluster 2021-02-10T04:34:43.820 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:43.820 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: cluster 2021-02-10T04:34:42.781968+0000 2021-02-10T04:34:43.820 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: osd.2 (osd.2) 1043 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:43.820 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: 2021-02-10T04:34:43.820 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:43.820 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: -02-10T04:34:42.781974+0000 osd.2 (osd 2021-02-10T04:34:43.820 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: .2) 1044 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:43.820 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:34:43.820 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: T04:34:42.781979+0000 osd.2 (osd 2021-02-10T04:34:43.820 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: .2) 1045 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:43.820 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: 2021-02-10T04:34:43.820 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: cluster 2021-02-10T04:34:43.821 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: 2021-02-10T04:34 2021-02-10T04:34:43.821 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: :42.781986+0000 osd.2 (osd.2 2021-02-10T04:34:43.821 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: ) 1046 : cluster [WRN] 2021-02-10T04:34:43.821 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:43.821 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:43.821 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: 10T04:34:43.138538+0000 mon.a ( 2021-02-10T04:34:43.821 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: mon.0) 1274 : cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 312 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:34:43.821 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:43.821 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: -02-10T04:34:43.143965 2021-02-10T04:34:43.821 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1275 : cluster [DBG] osdmap e466: 8 total, 8 up, 8 in 2021-02-10T04:34:43.822 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: cluster 2021-02-10T04:34:43.822 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: 2021-02-10T04:34:43.822 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: 2021-02-10T04:34:42.307909 2021-02-10T04:34:43.822 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 611 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:43.822 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:43.822 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: 34:42.307915+0000 osd.4 (osd.4 2021-02-10T04:34:43.822 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: ) 612 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:43.822 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:43.822 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: -02-10T04:34:42.307918+0000 2021-02-10T04:34:43.822 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: osd.4 (osd.4) 613 : cluster [WRN] 2021-02-10T04:34:43.822 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:43.822 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:43.823 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: 10T04:34:42.307922+0000 2021-02-10T04:34:43.823 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: osd.4 (osd.4) 614 : cluster [WRN] 2021-02-10T04:34:43.823 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:43.823 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: cluster 2021-02-10T04:34 2021-02-10T04:34:43.823 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: :42.457460+0000 mon.a (mon.0) 2021-02-10T04:34:43.823 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: 1273 : cluster [DBG] osdmap e465: 8 total, 8 up, 8 in 2021-02-10T04:34:43.823 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: cluster 2021-02-10T04:34:43.823 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: 2021-02-10T04:34:42.781930 2021-02-10T04:34:43.823 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1037 : cluster 2021-02-10T04:34:43.823 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:43.823 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:43.823 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: T04:34:42.781937+0000 osd.2 ( 2021-02-10T04:34:43.823 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: osd.2) 1038 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:43.823 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: cluster 2021-02-10T04:34:43.824 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: 2021-02-10T04:34:42. 2021-02-10T04:34:43.824 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: 781943+0000 osd.2 (osd.2) 1039 : 2021-02-10T04:34:43.824 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:43.824 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:43.824 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: T04:34:42.781949+0000 osd.2 ( 2021-02-10T04:34:43.824 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: osd.2) 1040 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:43.824 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:43.824 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: 02-10T04:34:42.781955+0000 2021-02-10T04:34:43.824 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: osd.2 (osd.2) 1041 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:43.824 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: 2021-02-10T04:34:43.824 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: cluster 2021-02-10T04:34:43.824 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: 2021-02-10T04:34:42.781960+0000 osd.2 (osd.2) 1042 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:43.824 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:43.825 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: -10T04:34:42.781968+0000 2021-02-10T04:34:43.825 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: osd.2 (osd.2) 1043 : cluster [WRN] 2021-02-10T04:34:43.825 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:43.825 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:43.825 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: 34:42.781974+0000 osd.2 (osd.2) 1044 : cluster [WRN] 2021-02-10T04:34:43.825 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:43.825 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: cluster 2021-02-10T04:34:42. 2021-02-10T04:34:43.825 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: 781979+0000 osd.2 (osd.2) 1045 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:43.825 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:43.825 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: T04:34:42.781986+0000 osd.2 ( 2021-02-10T04:34:43.825 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: osd.2) 1046 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:43.825 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: 2021-02-10T04:34:43.825 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: cluster 2021-02-10T04:34:43.826 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: 2021-02 2021-02-10T04:34:43.826 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: -10T04:34:43.138538+0000 mon.a (mon.0) 1274 : cluster [WRN] 2021-02-10T04:34:43.826 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: Health check update: 14 slow ops, oldest one blocked for 312 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:34:43.826 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: cluster 2021-02-10T04:34:43.143965+0000 2021-02-10T04:34:43.826 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[35961]: mon.a (mon.0) 1275 : cluster [DBG] osdmap e466: 8 total, 8 up, 8 in 2021-02-10T04:34:44.152 INFO:tasks.rados.rados.0.smithi085.stdout:1777: done (10 left) 2021-02-10T04:34:44.152 INFO:tasks.rados.rados.0.smithi085.stdout:1778: done (9 left) 2021-02-10T04:34:44.153 INFO:tasks.rados.rados.0.smithi085.stdout:1779: snap_create 2021-02-10T04:34:44.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:43 smithi085 conmon[49106]: debug 2021-02-10T04:34:43.815+0000 7f77a3e94700 -1 osd.2 466 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:44.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[33677]: debug 2021-02-10T04:34:44.316+0000 7fbec19a1700 -1 osd.4 467 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: cluster 2021-02-10T04:34:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: 2021-02 2021-02-10T04:34:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: -10T04:34:43.271213+0000 osd.4 (osd.4) 615 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: 2021-02-10T04:34:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: cluster 2021-02-10T04:34:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:34:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: 34:43.271219+0000 osd.4 (osd.4 2021-02-10T04:34:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: ) 616 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: -02-10T04:34:43.271226+0000 2021-02-10T04:34:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: osd.4 (osd.4) 617 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: 2021-02-10T04:34:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: cluster 2021-02-10T04:34:43 2021-02-10T04:34:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: .271230+0000 osd.4 (osd.4) 618 : 2021-02-10T04:34:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: T04:34:43.659252+0000 osd.4 (osd 2021-02-10T04:34:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: .4) 619 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:34:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: cluster 2021-02-10T04:34:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: 2021- 2021-02-10T04:34:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: 02-10T04:34:43.659450+0000 osd.4 (osd.4 2021-02-10T04:34:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: ) 620 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:34:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: cluster 2021-02-10T04:34:43 2021-02-10T04:34:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: .816042+0000 osd.2 (osd.2) 1047 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: cluster 2021-02-10T04:34:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: 2021-02-10T04:34:43. 2021-02-10T04:34:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: 816054+0000 osd.2 (osd.2) 1048 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: 2021-02-10T04:34:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: 04:34:43.816063+0000 osd.2 2021-02-10T04:34:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: (osd.2) 1049 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: 02-10T04:34:43.816070+0000 osd.2 (osd.2) 1050 : 2021-02-10T04:34:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:44.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: 02-10T04:34:43.816077+0000 osd.2 (osd 2021-02-10T04:34:44.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: .2) 1051 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:44.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: cluster 2021-02-10T04:34:43.816083+0000 osd.2 (osd. 2021-02-10T04:34:44.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: 2) 1052 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:44.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: cluster 2021-02-10T04:34:44.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: 2021-02 2021-02-10T04:34:44.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: -10T04:34:43.816089+0000 osd.2 2021-02-10T04:34:44.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: (osd.2) 1053 : cluster [WRN] 2021-02-10T04:34:44.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:44.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: cluster 2021-02-10T04:34:44.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: 2021- 2021-02-10T04:34:44.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: 02-10T04:34:43.816095+0000 osd.2 2021-02-10T04:34:44.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: (osd.2) 1054 : cluster [WRN] 2021-02-10T04:34:44.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:44.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:44.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: -02-10T04:34:43.816100+0000 osd.2 ( 2021-02-10T04:34:44.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: osd.2) 1055 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:44.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: cluster 2021-02-10T04:34:44.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: 2021-02-10T04:34:43.816106+0000 2021-02-10T04:34:44.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: osd.2 (osd.2) 1056 : 2021-02-10T04:34:44.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:44.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:44.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: :34:44.090108+0000 mgr.y (mgr.14140) 567 : cluster [DBG] pgmap v849: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 378 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 1.7 MiB/s wr, 11 op/s 2021-02-10T04:34:44.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: cluster 2021-02-10T04:34:44.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:34:44.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: 34:44.151621+0000 mon.a (mon 2021-02-10T04:34:44.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:44 smithi164 conmon[29288]: .0) 1276 : cluster [DBG] osdmap e467: 8 total, 8 up, 8 in 2021-02-10T04:34:44.771 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: cluster 2021-02-10T04:34:43.271213+0000 2021-02-10T04:34:44.772 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: osd.4 (osd.4) 615 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:44.772 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: cluster 2021-02-10T04:34:43.271219+0000 osd.4 (osd.4) 616 : cluster [WRN] 2021-02-10T04:34:44.772 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:44.772 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: cluster 2021-02-10T04:34:43 2021-02-10T04:34:44.772 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: .271226+0000 osd.4 (osd.4) 617 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:44.772 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:44.772 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: -02-10T04:34:43.271230+0000 osd.4 (osd. 2021-02-10T04:34:44.772 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: 4) 618 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:44.772 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:34:44.772 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: :34:43.659252+0000 osd.4 (osd.4) 619 : 2021-02-10T04:34:44.773 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: cluster [DBG] purged_snaps scrub starts 2021-02-10T04:34:44.773 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:44.773 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: -10T04:34:43.659450+0000 osd.4 (osd.4) 2021-02-10T04:34:44.773 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: 620 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:34:44.773 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:44.773 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: -02-10T04:34:43.816042+0000 osd.2 2021-02-10T04:34:44.773 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: (osd.2) 1047 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:44.773 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: cluster 2021-02-10T04:34:44.773 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: 2021-02-10T04:34:43. 2021-02-10T04:34:44.773 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: 816054+0000 osd.2 (osd.2) 1048 : cluster 2021-02-10T04:34:44.773 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:44.773 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: cluster 2021-02-10T04:34:43.816063+0000 2021-02-10T04:34:44.774 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: osd.2 (osd.2) 1049 : 2021-02-10T04:34:44.774 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:44.774 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: cluster 2021-02-10T04:34:44.774 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: 2021-02 2021-02-10T04:34:44.774 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: -10T04:34:43. 2021-02-10T04:34:44.774 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: 816070+0000 osd.2 (osd. 2021-02-10T04:34:44.774 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: 2) 1050 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:44.774 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:44.774 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: 02-10T04:34:43.816077+0000 osd.2 ( 2021-02-10T04:34:44.774 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: osd.2) 1051 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:44.774 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:44.774 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: -02-10T04:34:43.816083+0000 2021-02-10T04:34:44.774 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: osd.2 (osd.2) 1052 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:44.775 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: cluster 2021-02-10T04:34:44.775 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:34:44.775 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: 10T04:34:43.816089+0000 osd.2 ( 2021-02-10T04:34:44.775 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: osd.2) 1053 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:44.775 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:44.775 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: 02-10T04:34:43.816095+0000 osd.2 ( 2021-02-10T04:34:44.775 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: osd.2) 1054 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:44.775 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: cluster 2021-02-10T04:34:44.775 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: 2021-02-10T04:34:43.816100+0000 2021-02-10T04:34:44.775 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: osd.2 (osd.2) 1055 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:44.775 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: 2021-02-10T04:34:44.775 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:44.775 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: -02-10T04:34:43.816106+0000 osd.2 2021-02-10T04:34:44.776 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: (osd.2) 1056 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:44.776 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:44.776 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: 10T04:34:44.090108+0000 mgr.y (mgr 2021-02-10T04:34:44.776 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: .14140) 567 : cluster [DBG] pgmap v849: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 378 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 1.7 MiB/s wr, 11 op/s 2021-02-10T04:34:44.776 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:44.776 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: -02-10T04:34:44.151621+0000 2021-02-10T04:34:44.776 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: mon.a (mon.0) 1276 : cluster [DBG] osdmap e467: 8 total, 8 up, 8 in 2021-02-10T04:34:44.776 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[29319]: 2021-02-10T04:34:44.776 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: cluster 2021-02-10T04:34:44.776 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: 2021-02-10T04:34: 2021-02-10T04:34:44.776 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: 43.271213+0000 osd.4 (osd.4) 615 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:44.776 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: cluster 2021-02-10T04:34:44.777 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: 2021-02-10T04:34:43.271219 2021-02-10T04:34:44.777 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 616 : cluster 2021-02-10T04:34:44.777 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:44.777 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: cluster 2021-02-10T04:34:44.777 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:34:44.777 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: :34:43.271226+0000 osd.4 (osd.4 2021-02-10T04:34:44.777 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: ) 617 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:44.777 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: cluster 2021-02-10T04:34: 2021-02-10T04:34:44.777 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: 43.271230+0000 osd.4 (osd.4) 618 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:44.777 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: cluster 2021-02-10T04:34:43 2021-02-10T04:34:44.777 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: .659252+0000 osd.4 (osd.4) 619 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:34:44.777 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:44.777 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: -02-10T04:34:43.659450+0000 osd.4 (osd.4) 2021-02-10T04:34:44.778 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: 620 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:34:44.778 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:44.778 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: 10T04:34:43.816042+0000 osd.2 (osd. 2021-02-10T04:34:44.778 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: 2) 1047 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:44.778 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:44.778 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: -02-10T04:34:43.816054 2021-02-10T04:34:44.778 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1048 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:44.778 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: cluster 2021-02-10T04:34:44.778 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: 2021-02-10T04:34:43.816063+0000 osd.2 ( 2021-02-10T04:34:44.778 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: osd.2) 1049 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:44.778 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:44.778 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: 10T04:34:43.816070+0000 osd.2 (osd.2) 2021-02-10T04:34:44.779 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: 1050 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:44.779 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: cluster 2021-02-10T04:34:44.779 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: 2021-02 2021-02-10T04:34:44.779 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: -10T04:34:43.816077 2021-02-10T04:34:44.779 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1051 : cluster [WRN] 2021-02-10T04:34:44.779 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:44.779 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:44.779 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: -02-10T04:34:43.816083 2021-02-10T04:34:44.779 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1052 2021-02-10T04:34:44.779 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:44.779 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: cluster 2021-02-10T04:34:44.779 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: 2021-02-10T04:34:43 2021-02-10T04:34:44.779 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: .816089+0000 osd.2 (osd.2 2021-02-10T04:34:44.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: ) 1053 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:44.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: cluster 2021-02-10T04:34:44.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: 2021-02 2021-02-10T04:34:44.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: -10T04:34:43 2021-02-10T04:34:44.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: .816095+0000 osd.2 (osd.2) 1054 2021-02-10T04:34:44.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:44.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: cluster 2021-02-10T04:34:44.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: 2021-02-10T04:34 2021-02-10T04:34:44.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: :43.816100+0000 osd.2 2021-02-10T04:34:44.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: (osd.2) 1055 : cluster 2021-02-10T04:34:44.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:44.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: cluster 2021-02-10T04:34:44.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: 2021-02 2021-02-10T04:34:44.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: -10T04:34:43 2021-02-10T04:34:44.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: .816106+0000 osd.2 (osd. 2021-02-10T04:34:44.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: 2) 1056 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:44.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: cluster 2021-02-10T04:34:44.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:34:44.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: T04:34:44.090108+0000 mgr.y (mgr.14140) 567 : cluster 2021-02-10T04:34:44.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: [DBG] pgmap v849: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 378 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 1.7 MiB/s wr, 11 op/s 2021-02-10T04:34:44.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:44.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: T04:34:44.151621+0000 mon.a (mon. 2021-02-10T04:34:44.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[35961]: 0) 1276 : cluster [DBG] osdmap e467: 8 total, 8 up, 8 in 2021-02-10T04:34:44.905 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:34:44.935 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:34:45.153 INFO:tasks.rados.rados.0.smithi085.stdout:1779: done (9 left) 2021-02-10T04:34:45.153 INFO:tasks.rados.rados.0.smithi085.stdout:1780: snap_create 2021-02-10T04:34:45.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:44 smithi085 conmon[49106]: debug 2021-02-10T04:34:44.769+0000 7f77a3e94700 -1 osd.2 467 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:45.327 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[33677]: debug 2021-02-10T04:34:45.325+0000 7fbec19a1700 -1 osd.4 468 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: cluster 2021-02-10T04:34:44.317139+0000 osd.4 ( 2021-02-10T04:34:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: osd.4) 621 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: cluster 2021-02-10T04:34:44.317145+0000 osd.4 (osd.4) 622 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: cluster 2021-02-10T04:34:44.317151+0000 osd.4 (osd.4) 623 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: 2021-02-10T04:34:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: cluster 2021-02-10T04:34:44.317154+0000 osd.4 (osd.4) 624 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: cluster 2021-02-10T04:34:44.770220+0000 osd.2 (osd.2) 1057 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:45.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: 2021-02-10T04:34:45.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:45.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: T04:34:44.770230+0000 osd.2 (osd.2) 1058 : 2021-02-10T04:34:45.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:45.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:45.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: 34:44.770237+0000 osd.2 (osd.2 2021-02-10T04:34:45.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: ) 1059 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:45.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: cluster 2021-02-10T04:34:45.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: 2021-02-10T04:34:44.770243 2021-02-10T04:34:45.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1060 : cluster 2021-02-10T04:34:45.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:45.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:45.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: 02-10T04:34:44.770252+0000 osd.2 (osd.2) 2021-02-10T04:34:45.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: 1061 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:45.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: cluster 2021-02-10T04:34:45.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: 2021-02-10T04:34:44.770258+0000 osd.2 (osd.2) 1062 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:45.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: 2021-02-10T04:34:45.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:45.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: 34:44.770265+0000 osd.2 (osd.2) 1063 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:45.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:45.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: 02-10T04:34:44.770270+0000 2021-02-10T04:34:45.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: osd.2 (osd.2) 1064 : 2021-02-10T04:34:45.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:45.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:45.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: 02-10T04:34:44.770277+0000 osd.2 (osd. 2021-02-10T04:34:45.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: 2) 1065 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:45.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:45.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: -10T04:34:44.770282+0000 osd.2 (osd.2 2021-02-10T04:34:45.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: ) 1066 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:45.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:45.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: T04:34:45.152064+0000 mon.a (mon 2021-02-10T04:34:45.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:45 smithi164 conmon[29288]: .0) 1277 : cluster [DBG] osdmap e468: 8 total, 8 up, 8 in 2021-02-10T04:34:45.676 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: cluster 2021-02-10T04:34:44.317139+0000 2021-02-10T04:34:45.676 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: osd.4 (osd.4) 621 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:45.676 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:45.676 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: 34:44.317145+0000 osd.4 (osd.4) 622 : cluster 2021-02-10T04:34:45.676 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:45.676 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: 2021-02-10T04:34:45.676 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: cluster 2021-02-10T04:34:45.676 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: 2021-02-10T04:34:44.317151+0000 osd.4 ( 2021-02-10T04:34:45.677 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: osd.4) 623 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:45.677 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:45.677 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: 02-10T04:34:44.317154+0000 2021-02-10T04:34:45.677 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: osd.4 (osd.4) 624 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:45.677 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: 2021-02-10T04:34:45.677 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: cluster 2021-02-10T04:34:45.677 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: 2021-02-10T04:34:44.770220+0000 2021-02-10T04:34:45.677 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: osd.2 (osd.2) 1057 : cluster [WRN] 2021-02-10T04:34:45.677 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:45.677 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: cluster 2021-02-10T04:34:44.770230+0000 2021-02-10T04:34:45.677 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: osd.2 (osd.2) 1058 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:45.677 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:45.678 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: -02-10T04:34:44.770237+0000 2021-02-10T04:34:45.678 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: osd.2 (osd.2) 1059 : cluster [WRN] 2021-02-10T04:34:45.678 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:45.678 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:45.678 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: 10T04:34:44.770243+0000 osd.2 ( 2021-02-10T04:34:45.678 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: osd.2) 1060 : cluster 2021-02-10T04:34:45.678 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:45.678 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:45.678 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: 02-10T04:34:44.770252+0000 osd.2 ( 2021-02-10T04:34:45.678 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: osd.2) 1061 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:45.678 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:45.678 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: -10T04:34:44.770258+0000 osd.2 (osd.2) 2021-02-10T04:34:45.679 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: 1062 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:45.679 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:45.679 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: T04:34:44.770265+0000 osd.2 (osd 2021-02-10T04:34:45.679 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: .2) 1063 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:45.679 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:45.679 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: -10T04:34:44.770270+0000 osd.2 2021-02-10T04:34:45.679 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: (osd.2) 1064 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:45.679 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: 2021-02-10T04:34:45.680 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: cluster 2021-02-10T04:34:45.680 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: 2021-02-10T04:34:44.770277+0000 osd.2 (osd 2021-02-10T04:34:45.680 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: .2) 1065 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:45.680 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:45.680 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: -10T04:34:44.770282+0000 osd.2 2021-02-10T04:34:45.680 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: (osd.2) 1066 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:45.680 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: 2021-02-10T04:34:45.680 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: cluster 2021-02-10T04:34:45.680 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: 2021-02-10T04:34:45.152064+0000 mon.a ( 2021-02-10T04:34:45.680 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: mon.0) 1277 : cluster [DBG] osdmap e468: 8 total, 8 up, 8 in 2021-02-10T04:34:45.680 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[35961]: 2021-02-10T04:34:45.680 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: cluster 2021-02-10T04:34:44.317139+0000 osd.4 (osd 2021-02-10T04:34:45.681 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: .4) 621 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:45.681 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: cluster 2021-02-10T04:34:44.317145+0000 osd.4 (osd.4) 622 : cluster 2021-02-10T04:34:45.681 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:45.681 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: cluster 2021-02-10T04:34:45.681 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: 2021-02-10T04:34:44.317151+0000 osd.4 2021-02-10T04:34:45.681 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: (osd.4) 623 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:45.681 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: 2021-02-10T04:34:45.681 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: cluster 2021-02-10T04:34:45.681 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: 2021-02-10T04:34:44 2021-02-10T04:34:45.681 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: .317154+0000 osd.4 (osd 2021-02-10T04:34:45.681 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: .4) 624 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:45.681 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: cluster 2021-02-10T04:34:45.681 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:34:45.682 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: 10T04:34:44.770220+0000 osd.2 2021-02-10T04:34:45.682 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: (osd.2) 1057 2021-02-10T04:34:45.682 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:45.682 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: 2021-02-10T04:34:45.682 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: cluster 2021-02-10T04:34:45.682 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: 2021-02-10T04:34:44.770230+0000 osd.2 (osd.2) 1058 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:45.682 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:45.682 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: 10T04:34:44.770237+0000 osd.2 (osd.2 2021-02-10T04:34:45.682 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: ) 1059 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:45.682 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:45.682 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: 02-10T04:34:44.770243+0000 2021-02-10T04:34:45.682 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: osd.2 (osd.2) 1060 : cluster 2021-02-10T04:34:45.682 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:45.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: cluster 2021-02-10T04:34:44. 2021-02-10T04:34:45.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: 770252+0000 osd.2 (osd.2) 1061 : cluster [WRN] 2021-02-10T04:34:45.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:45.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: cluster 2021-02-10T04:34: 2021-02-10T04:34:45.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: 44.770258+0000 osd.2 (osd.2) 1062 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:45.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:45.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: -10T04:34:44.770265+0000 osd.2 (osd.2) 1063 : cluster [WRN] 2021-02-10T04:34:45.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:45.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:34:45.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: 34:44.770270+0000 osd.2 (osd.2) 1064 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:45.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: cluster 2021-02-10T04:34:45.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: 2021-02-10T04:34:44.770277 2021-02-10T04:34:45.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1065 : cluster [WRN] 2021-02-10T04:34:45.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:45.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:45.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: 10T04:34:44.770282+0000 osd.2 (osd 2021-02-10T04:34:45.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: .2) 1066 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:45.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:45.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: 02-10T04:34:45.152064+0000 mon.a (mon 2021-02-10T04:34:45.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[29319]: .0) 1277 : cluster [DBG] osdmap e468: 8 total, 8 up, 8 in 2021-02-10T04:34:45.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:45 smithi085 conmon[49106]: debug 2021-02-10T04:34:45.758+0000 7f77a3e94700 -1 osd.2 468 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:46.162 INFO:tasks.rados.rados.0.smithi085.stdout:1780: done (9 left) 2021-02-10T04:34:46.162 INFO:tasks.rados.rados.0.smithi085.stdout:1781: write (excl) oid 23 current snap is 193 2021-02-10T04:34:46.162 INFO:tasks.rados.rados.0.smithi085.stdout:1781: seq_num 628 ranges {554796=797318,1949280=526888} 2021-02-10T04:34:46.163 INFO:tasks.rados.rados.0.smithi085.stdout:1781: writing smithi08575397-23 from 554796 to 1352114 tid 1 2021-02-10T04:34:46.170 INFO:tasks.rados.rados.0.smithi085.stdout:1781: writing smithi08575397-23 from 1949280 to 2476168 tid 2 2021-02-10T04:34:46.170 INFO:tasks.rados.rados.0.smithi085.stdout:1782: delete oid 43 current snap is 193 2021-02-10T04:34:46.173 INFO:tasks.rados.rados.0.smithi085.stdout:1781: finishing write tid 1 to smithi08575397-23 2021-02-10T04:34:46.177 INFO:tasks.rados.rados.0.smithi085.stdout:1781: finishing write tid 2 to smithi08575397-23 2021-02-10T04:34:46.178 INFO:tasks.rados.rados.0.smithi085.stdout:1781: finishing write tid 3 to smithi08575397-23 2021-02-10T04:34:46.178 INFO:tasks.rados.rados.0.smithi085.stdout:1781: finishing write tid 4 to smithi08575397-23 2021-02-10T04:34:46.178 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 664 (ObjNum 628 snap 193 seq_num 628) dirty exists 2021-02-10T04:34:46.179 INFO:tasks.rados.rados.0.smithi085.stdout:1781: left oid 23 (ObjNum 628 snap 193 seq_num 628) 2021-02-10T04:34:46.179 INFO:tasks.rados.rados.0.smithi085.stdout:1781: done (10 left) 2021-02-10T04:34:46.179 INFO:tasks.rados.rados.0.smithi085.stdout:1782: done (9 left) 2021-02-10T04:34:46.179 INFO:tasks.rados.rados.0.smithi085.stdout:1783: read oid 30 snap -1 2021-02-10T04:34:46.179 INFO:tasks.rados.rados.0.smithi085.stdout:1783: expect (ObjNum 618 snap 187 seq_num 618) 2021-02-10T04:34:46.179 INFO:tasks.rados.rados.0.smithi085.stdout:1784: snap_create 2021-02-10T04:34:46.182 INFO:tasks.rados.rados.0.smithi085.stdout:1783: expect (ObjNum 618 snap 187 seq_num 618) 2021-02-10T04:34:46.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[33677]: debug 2021-02-10T04:34:46.275+0000 7fbec19a1700 -1 osd.4 468 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:46.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[33677]: debug 2021-02-10T04:34:46.276+0000 7fbea992c700 -1 osd.4 pg_epoch: 468 pg[6.1a( empty local-lis/les=59/60 n=0 ec=59/59 lis/c=59/59 les/c/f=60/60/0 sis=59) [4,5,1] r=0 lpr=59 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.1a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x562760a5b8c0) 2021-02-10T04:34:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: cluster 2021-02-10T04:34:45.326502+0000 osd.4 (osd.4) 625 : cluster 2021-02-10T04:34:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: cluster 2021-02-10T04:34:45.326507+0000 osd.4 (osd.4) 626 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: cluster 2021-02-10T04:34:45.326511+0000 2021-02-10T04:34:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: osd.4 (osd.4) 627 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: cluster 2021-02-10T04:34:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: 2021-02-10T04:34:45.326514+0000 osd.4 (osd.4) 2021-02-10T04:34:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: 628 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: 34:45.758550+0000 osd.2 (osd.2) 1067 2021-02-10T04:34:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: :34:45.758560+0000 osd.2 (osd.2) 2021-02-10T04:34:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: 1068 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: 04:34:45.758566+0000 osd.2 (osd. 2021-02-10T04:34:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: 2) 1069 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: cluster 2021-02-10T04:34: 2021-02-10T04:34:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: 45.758580+0000 osd.2 (osd.2) 1070 : cluster [WRN] 2021-02-10T04:34:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: cluster 2021-02-10T04:34:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: 2021-02-10T04:34:45.758586+0000 osd.2 (osd.2) 1071 : 2021-02-10T04:34:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: 34:45.758591+0000 osd.2 ( 2021-02-10T04:34:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: osd.2) 1072 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: -02-10T04:34:45.758594+0000 2021-02-10T04:34:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: osd.2 (osd.2) 1073 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: 2021-02-10T04:34:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: 34:45.758598+0000 osd.2 (osd.2) 1074 2021-02-10T04:34:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: 04:34:45.758602+0000 osd.2 (osd.2 2021-02-10T04:34:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: ) 1075 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: T04:34:45.758605+0000 osd.2 (osd.2 2021-02-10T04:34:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: ) 1076 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: -10T04:34:46.090798+0000 2021-02-10T04:34:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: mgr.y (mgr.14140) 568 : cluster [DBG] pgmap v852: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 385 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 1.5 MiB/s wr, 20 op/s 2021-02-10T04:34:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: cluster 2021-02-10T04:34:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: 2021-02-10T04:34:46. 2021-02-10T04:34:46.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: 153962+0000 mon.a (mon.0) 1278 : cluster 2021-02-10T04:34:46.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:46 smithi164 conmon[29288]: [DBG] osdmap e469: 8 total, 8 up, 8 in 2021-02-10T04:34:46.799 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[49106]: debug 2021-02-10T04:34:46.792+0000 7f77a3e94700 -1 osd.2 469 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:46.799 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: cluster 2021-02-10T04:34:45.326502+0000 osd.4 (osd.4) 625 2021-02-10T04:34:46.800 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:46.800 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: cluster 2021-02-10T04:34:45.326507+0000 osd.4 (osd.4) 626 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:46.800 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: cluster 2021-02-10T04:34:45.326511+0000 osd.4 (osd.4 2021-02-10T04:34:46.800 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: ) 627 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:46.800 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:46.800 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: 02-10T04:34:45.326514+0000 2021-02-10T04:34:46.800 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: osd.4 (osd.4) 628 : cluster [WRN] 2021-02-10T04:34:46.800 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:46.801 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:34:46.801 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: 34:45.758550+0000 osd.2 (osd.2 2021-02-10T04:34:46.801 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: ) 1067 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:46.801 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:34:46.801 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: T04:34:45.758560+0000 osd.2 (osd 2021-02-10T04:34:46.801 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: .2) 1068 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:46.801 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:46.801 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: -02-10T04:34:45.758566+0000 2021-02-10T04:34:46.801 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: osd.2 (osd.2) 1069 : cluster [WRN] 2021-02-10T04:34:46.801 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:46.801 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:34:46.801 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: 34:45.758580+0000 osd.2 (osd.2 2021-02-10T04:34:46.801 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: ) 1070 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:46.802 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:34:46.802 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: T04:34:45.758586+0000 osd.2 (osd. 2021-02-10T04:34:46.802 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: 2) 1071 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:46.802 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:46.802 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: -02-10T04:34:45.758591 2021-02-10T04:34:46.802 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1072 : cluster [WRN] 2021-02-10T04:34:46.802 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:46.802 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: cluster 2021-02-10T04:34: 2021-02-10T04:34:46.802 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: 45.758594+0000 osd.2 (osd.2) 2021-02-10T04:34:46.802 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: 1073 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:46.802 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:46.802 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: -02-10T04:34:45.758598 2021-02-10T04:34:46.803 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1074 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:46.803 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: 2021-02-10T04:34:46.803 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: cluster 2021-02-10T04:34:46.803 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: 2021-02-10T04:34:45.758602+0000 osd.2 (osd 2021-02-10T04:34:46.803 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: .2) 1075 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:46.803 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:46.803 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: 02-10T04:34:45.758605+0000 osd.2 ( 2021-02-10T04:34:46.803 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: osd.2) 1076 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:46.803 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:46.803 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: 02-10T04:34:46.090798+0000 2021-02-10T04:34:46.804 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: mgr.y (mgr.14140) 568 : cluster [DBG] pgmap v852: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 385 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 1.5 MiB/s wr, 20 op/s 2021-02-10T04:34:46.804 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: cluster 2021-02-10T04:34:46.804 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: 2021-02-10T04:34:46.153962 2021-02-10T04:34:46.804 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1278 : cluster [DBG] 2021-02-10T04:34:46.804 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[29319]: osdmap e469: 8 total, 8 up, 8 in 2021-02-10T04:34:46.804 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: cluster 2021-02-10T04:34:45 2021-02-10T04:34:46.805 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: .326502+0000 osd.4 (osd.4) 625 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:46.805 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: cluster 2021-02-10T04:34:46.805 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: 2021-02-10T04:34:45.326507+0000 osd.4 (osd.4) 626 : cluster 2021-02-10T04:34:46.805 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:46.805 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: cluster 2021-02-10T04:34:45. 2021-02-10T04:34:46.805 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: 326511+0000 osd.4 (osd.4) 627 : cluster [WRN] 2021-02-10T04:34:46.805 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:46.805 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:34:46.805 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: 04:34:45.326514+0000 osd.4 (osd. 2021-02-10T04:34:46.805 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: 4) 628 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:46.805 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:46.806 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: T04:34:45.758550+0000 osd.2 (osd 2021-02-10T04:34:46.806 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: .2) 1067 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:46.806 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:46.806 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: 10T04:34:45.758560+0000 osd.2 (osd. 2021-02-10T04:34:46.806 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: 2) 1068 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:46.806 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:46.806 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: -10T04:34:45.758566+0000 osd.2 2021-02-10T04:34:46.807 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: (osd.2) 1069 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:46.807 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: 2021-02-10T04:34:46.807 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: cluster 2021-02-10T04:34: 2021-02-10T04:34:46.807 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: 45.758580+0000 osd.2 (osd.2) 2021-02-10T04:34:46.807 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: 1070 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:46.807 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:34:46.807 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: :34:45.758586+0000 osd.2 (osd.2) 1071 2021-02-10T04:34:46.807 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:46.807 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:34:46.807 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: :34:45.758591+0000 osd.2 (osd.2) 1072 2021-02-10T04:34:46.808 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:46.808 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:46.808 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: 34:45.758594+0000 osd.2 (osd.2) 2021-02-10T04:34:46.808 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: 1073 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:46.808 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: cluster 2021-02-10T04:34:46.808 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: 2021 2021-02-10T04:34:46.808 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: - 2021-02-10T04:34:46.808 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: 02 2021-02-10T04:34:46.808 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: - 2021-02-10T04:34:46.808 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: 10 2021-02-10T04:34:46.808 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: T 2021-02-10T04:34:46.808 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: 04 2021-02-10T04:34:46.808 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: :34:45.758598+0000 2021-02-10T04:34:46.808 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: osd.2 (osd.2) 1074 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:46.809 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:46.809 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: 34:45.758602+0000 osd.2 (osd.2) 1075 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:46.809 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: cluster 2021-02-10T04:34:46.809 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: 2021-02-10T04:34:45 2021-02-10T04:34:46.809 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: .758605+0000 osd.2 (osd.2) 1076 : 2021-02-10T04:34:46.809 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:46.809 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:46.809 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: -10T04:34:46.090798+0000 mgr.y 2021-02-10T04:34:46.809 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: (mgr.14140) 568 : cluster [DBG] pgmap v852: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 385 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 1.5 MiB/s wr, 20 op/s 2021-02-10T04:34:46.809 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: 2021-02-10T04:34:46.809 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:46.809 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: 34:46.153962+0000 mon.a (mon.0 2021-02-10T04:34:46.810 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:46 smithi085 conmon[35961]: ) 1278 : cluster [DBG] osdmap e469: 8 total, 8 up, 8 in 2021-02-10T04:34:47.090 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 3 1 2021-02-10T04:34:47.157 INFO:tasks.rados.rados.0.smithi085.stdout:1783: done (10 left) 2021-02-10T04:34:47.157 INFO:tasks.rados.rados.0.smithi085.stdout:1784: done (9 left) 2021-02-10T04:34:47.157 INFO:tasks.rados.rados.0.smithi085.stdout:1785: rollback oid 12 current snap is 194 2021-02-10T04:34:47.157 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 12 to 179 2021-02-10T04:34:47.157 INFO:tasks.rados.rados.0.smithi085.stdout:1786: delete oid 3 current snap is 194 2021-02-10T04:34:47.161 INFO:tasks.rados.rados.0.smithi085.stdout:1785: finishing rollback tid 0 to smithi08575397-12 2021-02-10T04:34:47.161 INFO:tasks.rados.rados.0.smithi085.stdout:1785: finishing rollback tid 1 to smithi08575397-12 2021-02-10T04:34:47.161 INFO:tasks.rados.rados.0.smithi085.stdout:1785: finishing rollback tid 2 to smithi08575397-12 2021-02-10T04:34:47.162 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 772 (ObjNum 85 snap 11 seq_num 85) dirty exists 2021-02-10T04:34:47.169 INFO:tasks.rados.rados.0.smithi085.stdout:1785: done (10 left) 2021-02-10T04:34:47.169 INFO:tasks.rados.rados.0.smithi085.stdout:1786: done (9 left) 2021-02-10T04:34:47.169 INFO:tasks.rados.rados.0.smithi085.stdout:1787: delete oid 7 current snap is 194 2021-02-10T04:34:47.173 INFO:tasks.rados.rados.0.smithi085.stdout:1787: done (9 left) 2021-02-10T04:34:47.173 INFO:tasks.rados.rados.0.smithi085.stdout:1788: rollback oid 42 current snap is 194 2021-02-10T04:34:47.173 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 42 to 182 2021-02-10T04:34:47.173 INFO:tasks.rados.rados.0.smithi085.stdout:1789: read oid 40 snap -1 2021-02-10T04:34:47.173 INFO:tasks.rados.rados.0.smithi085.stdout:1789: expect deleted 2021-02-10T04:34:47.174 INFO:tasks.rados.rados.0.smithi085.stdout:1790: snap_create 2021-02-10T04:34:47.176 INFO:tasks.rados.rados.0.smithi085.stdout:1788: finishing rollback tid 0 to smithi08575397-42 2021-02-10T04:34:47.176 INFO:tasks.rados.rados.0.smithi085.stdout:1788: finishing rollback tid 1 to smithi08575397-42 2021-02-10T04:34:47.177 INFO:tasks.rados.rados.0.smithi085.stdout:1788: finishing rollback tid 2 to smithi08575397-42 2021-02-10T04:34:47.177 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 775 (ObjNum 540 snap 167 seq_num 540) dirty exists 2021-02-10T04:34:47.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[33677]: debug 2021-02-10T04:34:47.244+0000 7fbec19a1700 -1 osd.4 470 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: cluster 2021-02-10T04:34:46.277444+0000 osd.4 (osd. 2021-02-10T04:34:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: 4) 629 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: cluster 2021-02-10T04:34:46.277454+0000 osd.4 (osd.4) 630 2021-02-10T04:34:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: cluster 2021-02-10T04:34:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: 2021-02-10T04:34: 2021-02-10T04:34:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: 46.277462+0000 osd.4 (osd. 2021-02-10T04:34:47.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: 4) 631 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:47.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: 2021-02-10T04:34:47.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:47.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: 10T04:34:46.277471 2021-02-10T04:34:47.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 632 2021-02-10T04:34:47.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:47.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: cluster 2021-02-10T04:34:47.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: 2021-02-10T04:34: 2021-02-10T04:34:47.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: 46.793295+0000 osd.2 (osd.2) 1077 : cluster 2021-02-10T04:34:47.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:47.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: cluster 2021-02-10T04:34:46.793305+0000 2021-02-10T04:34:47.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: osd.2 (osd.2) 1078 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:47.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:47.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: 02-10T04:34:46.793311+0000 osd.2 (osd.2) 1079 2021-02-10T04:34:47.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:47.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: cluster 2021-02-10T04:34:47.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:34:47.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: 04:34:46.793317+0000 osd.2 2021-02-10T04:34:47.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: (osd.2) 1080 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:47.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: 2021-02-10T04:34:47.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:47.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: :34:46.793323+0000 osd.2 2021-02-10T04:34:47.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: (osd.2) 1081 : cluster 2021-02-10T04:34:47.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:47.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:47.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: 02-10T04:34:46 2021-02-10T04:34:47.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: .793329+0000 osd.2 (osd.2 2021-02-10T04:34:47.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: ) 1082 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:47.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:47.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: -02-10T04:34:46.793335+0000 osd.2 (osd.2 2021-02-10T04:34:47.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: ) 1083 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:47.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:47.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: T04:34:46.793340+0000 osd.2 (osd.2 2021-02-10T04:34:47.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: ) 1084 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:47.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: cluster 2021-02-10T04:34 2021-02-10T04:34:47.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: :46.793346+0000 osd.2 (osd.2) 1085 : cluster 2021-02-10T04:34:47.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: [WRN] 2021-02-10T04:34:47.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:47.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:47.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: -02-10T04:34:46 2021-02-10T04:34:47.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: .793351+0000 osd.2 (osd 2021-02-10T04:34:47.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: .2) 1086 : cluster 2021-02-10T04:34:47.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:47.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:47.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: 02-10T04:34:47.156717+0000 2021-02-10T04:34:47.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:47 smithi164 conmon[29288]: mon.a (mon.0) 1279 : cluster [DBG] osdmap e470: 8 total, 8 up, 8 in 2021-02-10T04:34:47.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: cluster 2021-02-10T04:34:46.277444+0000 osd.4 (osd.4) 629 : cluster 2021-02-10T04:34:47.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:47.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: cluster 2021-02-10T04:34:46.277454+0000 osd.4 (osd.4) 630 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:47.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: cluster 2021-02-10T04:34:46.277462+0000 osd.4 (osd.4) 631 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:47.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: 2021-02-10T04:34:47.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:47.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: -10T04:34:46.277471+0000 osd.4 (osd.4) 632 : cluster 2021-02-10T04:34:47.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:47.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: cluster 2021-02-10T04:34:47.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: 2021-02-10T04:34:46 2021-02-10T04:34:47.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: .793295+0000 osd.2 (osd.2) 1077 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:47.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:47.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: 34:46.793305+0000 osd.2 (osd.2) 2021-02-10T04:34:47.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: 1078 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:47.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: cluster 2021-02-10T04:34:47.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: 2021-02-10T04:34:46.793311 2021-02-10T04:34:47.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1079 : cluster 2021-02-10T04:34:47.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:47.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: cluster 2021-02-10T04:34:47.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: 2021-02-10T04:34:46. 2021-02-10T04:34:47.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: 793317+0000 osd.2 (osd.2) 1080 : cluster 2021-02-10T04:34:47.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:47.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: cluster 2021-02-10T04:34:47.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: 2021-02-10T04:34:46. 2021-02-10T04:34:47.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: 793323+0000 osd.2 (osd.2) 1081 : cluster 2021-02-10T04:34:47.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:47.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:47.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: 02-10T04:34:46.793329+0000 2021-02-10T04:34:47.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: osd.2 (osd.2) 1082 : cluster [WRN] 2021-02-10T04:34:47.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:47.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:47.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: -02-10T04:34:46.793335+0000 2021-02-10T04:34:47.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: osd.2 (osd.2) 1083 : cluster [WRN] 2021-02-10T04:34:47.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:47.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:47.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: -02-10T04:34:46.793340 2021-02-10T04:34:47.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1084 : cluster 2021-02-10T04:34:47.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:47.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: cluster 2021-02-10T04:34:47.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: 2021-02-10T04:34:47.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: 2021 2021-02-10T04:34:47.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: -02-10T04:34:46 2021-02-10T04:34:47.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: .793346+0000 osd.2 (osd.2) 1085 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:47.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: cluster 2021-02-10T04:34:47.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: 2021-02 2021-02-10T04:34:47.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: -10T04:34:46.793351 2021-02-10T04:34:47.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1086 : cluster 2021-02-10T04:34:47.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:47.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:47.785 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: -10T04:34:47. 2021-02-10T04:34:47.785 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: 156717+0000 mon.a (mon.0) 1279 : cluster 2021-02-10T04:34:47.785 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[35961]: [DBG] osdmap e470: 8 total, 8 up, 8 in 2021-02-10T04:34:47.785 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: cluster 2021-02-10T04:34:46.277444+0000 osd.4 (osd.4) 629 : cluster [WRN] 2021-02-10T04:34:47.785 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:47.785 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: cluster 2021-02-10T04:34:46.277454+0000 osd.4 (osd.4) 630 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:47.786 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: cluster 2021-02-10T04:34:46.277462+0000 osd.4 (osd.4) 631 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:47.786 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: cluster 2021-02-10T04:34:46.277471+0000 osd.4 (osd.4) 632 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:47.786 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: cluster 2021-02-10T04:34:46.793295+0000 osd.2 (osd.2) 1077 : cluster [WRN] 2021-02-10T04:34:47.786 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:47.786 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:34:47.786 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: 34:46.793305+0000 osd.2 (osd.2) 1078 : cluster 2021-02-10T04:34:47.786 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:47.786 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:47.786 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: -02-10T04:34:46.793311+0000 osd.2 (osd.2) 2021-02-10T04:34:47.786 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: 1079 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:47.786 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:47.786 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: 10T04:34:46.793317+0000 osd.2 ( 2021-02-10T04:34:47.786 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: osd.2) 1080 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:47.787 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: cluster 2021-02-10T04:34 2021-02-10T04:34:47.787 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: :46.793323+0000 osd.2 (osd.2) 1081 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:47.787 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: cluster 2021-02-10T04:34:46.793329+0000 osd.2 2021-02-10T04:34:47.787 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: (osd.2) 1082 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:47.787 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:47.787 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: -10T04:34:46.793335+0000 osd.2 ( 2021-02-10T04:34:47.787 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: osd.2) 1083 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:47.787 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:47.787 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: 02-10T04:34:46.793340+0000 2021-02-10T04:34:47.787 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: osd.2 (osd.2) 1084 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:47.787 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:47.787 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: 02-10T04:34:46.793346+0000 2021-02-10T04:34:47.788 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: osd.2 (osd.2) 1085 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:47.788 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:47.788 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: -02-10T04:34:46.793351+0000 2021-02-10T04:34:47.788 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: osd.2 (osd.2) 1086 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:47.788 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: 2021-02-10T04:34:47.788 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: cluster 2021-02-10T04:34:47.788 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: 2021-02-10T04:34:47.156717+0000 mon.a ( 2021-02-10T04:34:47.788 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[29319]: mon.0) 1279 : cluster [DBG] osdmap e470: 8 total, 8 up, 8 in 2021-02-10T04:34:47.878 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:34:48.154 INFO:tasks.rados.rados.0.smithi085.stdout:1788: done (11 left) 2021-02-10T04:34:48.154 INFO:tasks.rados.rados.0.smithi085.stdout:1789: done (10 left) 2021-02-10T04:34:48.154 INFO:tasks.rados.rados.0.smithi085.stdout:1790: done (9 left) 2021-02-10T04:34:48.154 INFO:tasks.rados.rados.0.smithi085.stdout:1791: delete oid 28 current snap is 195 2021-02-10T04:34:48.163 INFO:tasks.rados.rados.0.smithi085.stdout:1791: done (9 left) 2021-02-10T04:34:48.163 INFO:tasks.rados.rados.0.smithi085.stdout:1792: read oid 7 snap 192 2021-02-10T04:34:48.163 INFO:tasks.rados.rados.0.smithi085.stdout:1792: expect (ObjNum 621 snap 188 seq_num 621) 2021-02-10T04:34:48.163 INFO:tasks.rados.rados.0.smithi085.stdout:1793: copy_from oid 31 from oid 43 current snap is 195 2021-02-10T04:34:48.163 INFO:tasks.rados.rados.0.smithi085.stdout:1794: rollback oid 44 current snap is 195 2021-02-10T04:34:48.163 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 44 to 166 2021-02-10T04:34:48.163 INFO:tasks.rados.rados.0.smithi085.stdout:1795: write oid 40 current snap is 195 2021-02-10T04:34:48.163 INFO:tasks.rados.rados.0.smithi085.stdout:1795: seq_num 633 ranges {638831=489916,1841690=677884} 2021-02-10T04:34:48.166 INFO:tasks.rados.rados.0.smithi085.stdout:1795: writing smithi08575397-40 from 638831 to 1128747 tid 1 2021-02-10T04:34:48.176 INFO:tasks.rados.rados.0.smithi085.stdout:1795: writing smithi08575397-40 from 1841690 to 2519574 tid 2 2021-02-10T04:34:48.176 INFO:tasks.rados.rados.0.smithi085.stdout:1794: finishing rollback tid 0 to smithi08575397-44 2021-02-10T04:34:48.176 INFO:tasks.rados.rados.0.smithi085.stdout:1794: finishing rollback tid 1 to smithi08575397-44 2021-02-10T04:34:48.176 INFO:tasks.rados.rados.0.smithi085.stdout:1794: finishing rollback tid 2 to smithi08575397-44 2021-02-10T04:34:48.176 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 729 (ObjNum 491 snap 141 seq_num 491) dirty exists 2021-02-10T04:34:48.176 INFO:tasks.rados.rados.0.smithi085.stdout:1794: done (12 left) 2021-02-10T04:34:48.176 INFO:tasks.rados.rados.0.smithi085.stdout:1796: snap_create 2021-02-10T04:34:48.177 INFO:tasks.rados.rados.0.smithi085.stdout:1792: expect (ObjNum 621 snap 188 seq_num 621) 2021-02-10T04:34:48.192 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:47 smithi085 conmon[49106]: debug 2021-02-10T04:34:47.778+0000 7f77a3e94700 -1 osd.2 470 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:48.372 INFO:tasks.rados.rados.0.smithi085.stdout:1793: finishing copy_from to smithi08575397-31 2021-02-10T04:34:48.372 INFO:tasks.rados.rados.0.smithi085.stdout:1793: got expected ENOENT (src dne) 2021-02-10T04:34:48.372 INFO:tasks.rados.rados.0.smithi085.stdout:1793: finishing copy_from racing read to smithi08575397-31 2021-02-10T04:34:48.372 INFO:tasks.rados.rados.0.smithi085.stdout:1795: finishing write tid 1 to smithi08575397-40 2021-02-10T04:34:48.373 INFO:tasks.rados.rados.0.smithi085.stdout:1795: finishing write tid 2 to smithi08575397-40 2021-02-10T04:34:48.373 INFO:tasks.rados.rados.0.smithi085.stdout:1795: finishing write tid 3 to smithi08575397-40 2021-02-10T04:34:48.373 INFO:tasks.rados.rados.0.smithi085.stdout:1795: finishing write tid 4 to smithi08575397-40 2021-02-10T04:34:48.373 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 964 (ObjNum 633 snap 195 seq_num 633) dirty exists 2021-02-10T04:34:48.373 INFO:tasks.rados.rados.0.smithi085.stdout:1795: left oid 40 (ObjNum 633 snap 195 seq_num 633) 2021-02-10T04:34:48.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[40892]: debug 2021-02-10T04:34:48.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[40892]: 2021-02-10T04:34:48.167+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:34:48.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[40892]: 2021-02-10T04:34:48.395 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:34:48.395 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[33677]: debug 2021-02-10T04:34:48.395 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[33677]: 2021-02-10T04:34:48.238+0000 7fbec19a1700 -1 osd.4 471 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:48.395 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[33677]: debug 2021-02-10T04:34:48.239+0000 7fbea5924700 -1 osd.4 pg_epoch: 471 pg[6.1a( empty local-lis/les=59/60 n=0 ec=59/59 lis/c=59/59 les/c/f=60/60/0 sis=59) [4,5,1] r=0 lpr=59 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.1a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x56275de4b600) 2021-02-10T04:34:48.535 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:34:48.640 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:34:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: cluster 2021-02-10T04:34:47. 2021-02-10T04:34:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: 245977+0000 osd.4 (osd.4) 633 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: cluster 2021-02-10T04:34:47.245983+0000 osd.4 (osd.4) 634 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: cluster 2021-02-10T04:34:47.245987+0000 osd.4 (osd 2021-02-10T04:34:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: .4) 635 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: cluster 2021-02-10T04:34:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: 2021-02-10T04:34: 2021-02-10T04:34:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: 47.245990+0000 osd.4 (osd.4) 636 : 2021-02-10T04:34:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: 10T04:34:47.778953+0000 osd.2 2021-02-10T04:34:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: (osd.2) 1087 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: 2021-02-10T04:34:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:34:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: :34:47.778958+0000 osd.2 (osd.2 2021-02-10T04:34:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: ) 1088 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: cluster 2021-02-10T04:34:48.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: 2021-02-10T04:34:47. 2021-02-10T04:34:48.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: 778962+0000 osd.2 (osd.2) 1089 : cluster 2021-02-10T04:34:48.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:48.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: cluster 2021-02-10T04:34:48.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:34:48.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: :34:47.778965+0000 osd.2 (osd.2) 1090 2021-02-10T04:34:48.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: : cluster [WRN] 2021-02-10T04:34:48.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:48.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: cluster 2021-02-10T04:34:48.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: 2021-02-10T04:34:47. 2021-02-10T04:34:48.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: 778968+0000 osd.2 ( 2021-02-10T04:34:48.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: osd.2) 1091 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:48.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:48.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: 02-10T04:34:47.778971+0000 osd.2 2021-02-10T04:34:48.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: (osd.2) 1092 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:48.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: cluster 2021-02-10T04:34: 2021-02-10T04:34:48.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: 47.778975+0000 osd.2 (osd.2) 1093 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:48.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:48.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: 04:34:47.778979+0000 osd.2 (osd 2021-02-10T04:34:48.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: .2) 1094 : cluster [WRN] 2021-02-10T04:34:48.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:48.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: cluster 2021-02-10T04:34:48.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: 2021-02-10T04:34:47.778982 2021-02-10T04:34:48.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1095 2021-02-10T04:34:48.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:48.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:48.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: 02-10T04:34:47 2021-02-10T04:34:48.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: .778985+0000 osd.2 (osd 2021-02-10T04:34:48.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: .2) 1096 : cluster [WRN] 2021-02-10T04:34:48.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:48.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:34:48.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: -10T04:34:48.046029+0000 mgr.y 2021-02-10T04:34:48.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: (mgr.14140) 569 : audit [DBG] 2021-02-10T04:34:48.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:34:48.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: cluster 2021-02-10T04:34:48.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: 2021-02-10T04:34:48.091421 2021-02-10T04:34:48.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: +0000 mgr.y (mgr 2021-02-10T04:34:48.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: .14140) 570 : cluster [DBG] 2021-02-10T04:34:48.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: pgmap v855: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 385 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 884 KiB/s wr, 16 op/s 2021-02-10T04:34:48.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:48.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: 10T04:34:48.151035+0000 mon.a ( 2021-02-10T04:34:48.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[29319]: mon.0) 1280 : cluster [DBG] osdmap e471: 8 total, 8 up, 8 in 2021-02-10T04:34:48.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: cluster 2021-02-10T04:34:48.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: 2021-02-10T04:34:48.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: 2021 2021-02-10T04:34:48.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: - 2021-02-10T04:34:48.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: 02 2021-02-10T04:34:48.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: -10T04:34:47.245977+0000 2021-02-10T04:34:48.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: osd.4 (osd.4) 633 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:48.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:48.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: -10T04:34:47.245983+0000 osd.4 (osd.4) 634 : cluster [WRN] 2021-02-10T04:34:48.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:48.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: cluster 2021-02-10T04:34:48.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: 2021-02-10T04:34:47.245987+0000 osd.4 (osd.4) 2021-02-10T04:34:48.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: 635 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:48.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: cluster 2021-02-10T04:34:48.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: 2021-02-10T04:34: 2021-02-10T04:34:48.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: 47.245990+0000 osd.4 (osd.4) 636 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:48.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: cluster 2021-02-10T04:34:48.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: 2021-02-10T04:34: 2021-02-10T04:34:48.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: 47.778953+0000 osd.2 (osd.2) 1087 : 2021-02-10T04:34:48.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:48.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:34:48.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: 04:34:47.778958+0000 osd.2 (osd.2 2021-02-10T04:34:48.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: ) 1088 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:48.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:48.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: 34:47.778962+0000 osd.2 (osd.2) 1089 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:48.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: 2021-02-10T04:34:48.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:48.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: T04:34:47.778965+0000 osd.2 (osd.2) 1090 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:48.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:48.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: -10T04:34:47.778968+0000 osd.2 (osd.2) 1091 2021-02-10T04:34:48.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:48.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:48.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: -10T04:34:47.778971+0000 osd.2 (osd.2) 1092 : cluster [WRN] 2021-02-10T04:34:48.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:48.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: cluster 2021-02-10T04:34:48.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: 2021-02-10T04:34:47.778975+0000 2021-02-10T04:34:48.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: osd.2 (osd.2) 1093 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:48.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:34:48.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: 04:34:47.778979+0000 osd.2 (osd.2) 1094 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:48.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: 2021-02-10T04:34:48.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: cluster 2021-02-10T04:34:48.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: 2021-02-10T04:34:47.778982+0000 osd.2 2021-02-10T04:34:48.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: (osd.2) 1095 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:48.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: cluster 2021-02-10T04:34:48.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: 2021-02-10T04:34:47 2021-02-10T04:34:48.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: .778985+0000 osd.2 (osd.2) 1096 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:48.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:34:48.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: 10T04:34:48.046029+0000 mgr.y (mgr.14140 2021-02-10T04:34:48.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: ) 569 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:34:48.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: cluster 2021-02-10T04:34:48 2021-02-10T04:34:48.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: .091421+0000 mgr.y (mgr.14140) 570 : cluster [DBG] pgmap v855: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 385 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 884 KiB/s wr, 16 op/s 2021-02-10T04:34:48.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:48.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: -02-10T04:34:48.151035+0000 mon.a ( 2021-02-10T04:34:48.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[35961]: mon.0) 1280 : cluster [DBG] osdmap e471: 8 total, 8 up, 8 in 2021-02-10T04:34:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: cluster 2021-02-10T04:34:47.245977+0000 osd.4 (osd.4) 633 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: cluster 2021-02-10T04:34:47.245983+0000 osd.4 (osd.4) 634 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: T04:34:47.245987+0000 osd.4 (osd.4) 635 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: -02-10T04:34:47.245990+0000 2021-02-10T04:34:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: osd.4 (osd.4) 636 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: 2021-02-10T04:34:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: cluster 2021-02-10T04:34:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: 2021-02-10T04:34:47.778953+0000 osd.2 (osd 2021-02-10T04:34:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: .2) 1087 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: 10T04:34:47.778958+0000 osd.2 ( 2021-02-10T04:34:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: osd.2) 1088 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: 10T04:34:47.778962+0000 osd.2 (osd.2 2021-02-10T04:34:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: ) 1089 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: 02-10T04:34:47.778965+0000 2021-02-10T04:34:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: osd.2 (osd.2) 1090 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: 2021-02-10T04:34:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: cluster 2021-02-10T04:34:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: 2021-02-10T04:34:47 2021-02-10T04:34:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: .778968+0000 osd.2 (osd.2) 1091 : cluster 2021-02-10T04:34:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: :34:47.778971+0000 osd.2 (osd.2 2021-02-10T04:34:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: ) 1092 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: 10T04:34:47.778975+0000 osd.2 ( 2021-02-10T04:34:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: osd.2) 1093 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: -02-10T04:34:47.778979+0000 2021-02-10T04:34:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: osd.2 (osd.2) 1094 : cluster [WRN] 2021-02-10T04:34:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: -02-10T04:34:47.778982+0000 osd.2 (osd.2 2021-02-10T04:34:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: ) 1095 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: T04:34:47.778985+0000 osd.2 (osd 2021-02-10T04:34:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: .2) 1096 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: audit 2021-02-10T04:34:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: 2021-02-10T04:34: 2021-02-10T04:34:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: 48.046029+0000 mgr.y (mgr.14140) 569 : 2021-02-10T04:34:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:34:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: :34:48.091421+0000 mgr.y (mgr.14140 2021-02-10T04:34:48.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: ) 570 : cluster [DBG] pgmap v855: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 385 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 884 KiB/s wr, 16 op/s 2021-02-10T04:34:48.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:48.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: -10T04:34:48.151035+0000 mon.a 2021-02-10T04:34:48.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:48 smithi164 conmon[29288]: (mon.0) 1280 : cluster [DBG] osdmap e471: 8 total, 8 up, 8 in 2021-02-10T04:34:48.920 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:34:49.164 INFO:tasks.rados.rados.0.smithi085.stdout:1792: done (12 left) 2021-02-10T04:34:49.165 INFO:tasks.rados.rados.0.smithi085.stdout:1793: done (11 left) 2021-02-10T04:34:49.165 INFO:tasks.rados.rados.0.smithi085.stdout:1795: done (10 left) 2021-02-10T04:34:49.165 INFO:tasks.rados.rados.0.smithi085.stdout:1796: done (9 left) 2021-02-10T04:34:49.165 INFO:tasks.rados.rados.0.smithi085.stdout:1797: read oid 42 snap 148 2021-02-10T04:34:49.165 INFO:tasks.rados.rados.0.smithi085.stdout:1797: expect (ObjNum 204 snap 44 seq_num 204) 2021-02-10T04:34:49.165 INFO:tasks.rados.rados.0.smithi085.stdout:1798: rollback oid 2 current snap is 196 2021-02-10T04:34:49.165 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 2 to 181 2021-02-10T04:34:49.165 INFO:tasks.rados.rados.0.smithi085.stdout:1799: delete oid 8 current snap is 196 2021-02-10T04:34:49.173 INFO:tasks.rados.rados.0.smithi085.stdout:1798: finishing rollback tid 0 to smithi08575397-2 2021-02-10T04:34:49.175 INFO:tasks.rados.rados.0.smithi085.stdout:1798: finishing rollback tid 1 to smithi08575397-2 2021-02-10T04:34:49.175 INFO:tasks.rados.rados.0.smithi085.stdout:1798: finishing rollback tid 2 to smithi08575397-2 2021-02-10T04:34:49.175 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 647 (ObjNum 602 snap 180 seq_num 602) dirty exists 2021-02-10T04:34:49.178 INFO:tasks.rados.rados.0.smithi085.stdout:1798: done (11 left) 2021-02-10T04:34:49.178 INFO:tasks.rados.rados.0.smithi085.stdout:1799: done (10 left) 2021-02-10T04:34:49.178 INFO:tasks.rados.rados.0.smithi085.stdout:1800: delete oid 10 current snap is 196 2021-02-10T04:34:49.181 INFO:tasks.rados.rados.0.smithi085.stdout:1800: done (10 left) 2021-02-10T04:34:49.181 INFO:tasks.rados.rados.0.smithi085.stdout:1801: rollback oid 30 current snap is 196 2021-02-10T04:34:49.181 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 30 to 185 2021-02-10T04:34:49.181 INFO:tasks.rados.rados.0.smithi085.stdout:1802: snap_remove snap 183 2021-02-10T04:34:49.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[49106]: debug 2021-02-10T04:34:49.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[49106]: 2021-02-10T04:34:48.775+0000 7f77a3e94700 -1 osd.2 471 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:49.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:48 smithi085 conmon[49106]: 2021-02-10T04:34:49.482 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[33677]: debug 2021-02-10T04:34:49.482 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[33677]: 2021-02-10T04:34:49.207+0000 7fbec19a1700 -1 osd.4 472 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:49.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: cluster 2021-02-10T04:34:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: 2021-02-10T04:34:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: 2021-02-10T04:34:48.240521+0000 osd.4 (osd.4) 637 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: cluster 2021-02-10T04:34:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: 2021-02-10T04:34:48.240530+0000 osd.4 (osd. 2021-02-10T04:34:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: 4) 638 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:34:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: T04:34:48.240550+0000 osd.4 (osd.4) 639 : 2021-02-10T04:34:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: cluster 2021-02-10T04:34:48. 2021-02-10T04:34:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: 240556+0000 osd.4 (osd.4) 640 : cluster [WRN] 2021-02-10T04:34:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:49.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: cluster 2021-02-10T04:34:48. 2021-02-10T04:34:49.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: 467211+0000 mon.a (mon.0) 1281 : cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 317 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:34:49.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:34:49.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: 34:48.775891+0000 osd.2 (osd.2) 1097 : cluster 2021-02-10T04:34:49.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:49.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: cluster 2021-02-10T04:34:48 2021-02-10T04:34:49.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: .775900+0000 osd.2 (osd.2) 1098 : cluster 2021-02-10T04:34:49.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:49.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: cluster 2021-02-10T04:34: 2021-02-10T04:34:49.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: 48.775904+0000 osd.2 (osd.2) 1099 : cluster 2021-02-10T04:34:49.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:49.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: cluster 2021-02-10T04:34:49.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: 2021-02-10T04:34:48.775908+0000 osd.2 ( 2021-02-10T04:34:49.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: osd.2) 1100 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:49.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:34:49.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: T04:34:48.775912+0000 osd.2 (osd 2021-02-10T04:34:49.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: .2) 1101 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:49.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:49.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: 10T04:34:48.775916+0000 osd.2 (osd 2021-02-10T04:34:49.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: .2) 1102 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:49.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:34:49.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: 34:48.775920+0000 osd.2 (osd.2) 1103 2021-02-10T04:34:49.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:49.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: 2021-02-10T04:34:49.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: cluster 2021-02-10T04:34:48.775924 2021-02-10T04:34:49.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1104 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:49.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: cluster 2021-02-10T04:34:48.775928+0000 2021-02-10T04:34:49.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: osd.2 (osd.2) 1105 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:49.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:34:49.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: 34:48.775932+0000 osd.2 (osd.2) 1106 2021-02-10T04:34:49.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:49.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: cluster 2021-02-10T04:34 2021-02-10T04:34:49.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: :49.164876+0000 mon.a (mon.0) 1282 : cluster 2021-02-10T04:34:49.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[29319]: [DBG] osdmap e472: 8 total, 8 up, 8 in 2021-02-10T04:34:49.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: cluster 2021-02-10T04:34:48. 2021-02-10T04:34:49.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: 240521+0000 osd.4 (osd.4) 637 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:49.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: cluster 2021-02-10T04:34:48.240530+0000 osd.4 (osd.4) 638 : cluster 2021-02-10T04:34:49.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:49.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:34:49.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: 04:34:48.240550+0000 osd.4 (osd.4) 639 : cluster [WRN] 2021-02-10T04:34:49.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:49.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: cluster 2021-02-10T04:34:48 2021-02-10T04:34:49.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: .240556+0000 osd.4 (osd.4) 640 : 2021-02-10T04:34:49.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:49.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:49.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: 34:48.467211+0000 mon.a (mon.0) 2021-02-10T04:34:49.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: 1281 : cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 317 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:34:49.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:34:49.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: 04:34:48.775891+0000 osd.2 (osd. 2021-02-10T04:34:49.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: 2) 1097 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:49.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:49.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: -10T04:34:48.775900+0000 osd.2 2021-02-10T04:34:49.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: (osd.2) 1098 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:49.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: 2021-02-10T04:34:49.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: cluster 2021-02-10T04:34:49.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: 2021-02-10T04:34:48.775904+0000 osd.2 (osd 2021-02-10T04:34:49.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: .2) 1099 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:49.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:49.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: 10T04:34:48.775908+0000 osd.2 ( 2021-02-10T04:34:49.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: osd.2) 1100 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:49.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:49.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: -02-10T04:34:48.775912+0000 2021-02-10T04:34:49.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: osd.2 (osd.2) 1101 : cluster [WRN] 2021-02-10T04:34:49.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:49.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: cluster 2021-02-10T04:34:48. 2021-02-10T04:34:49.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: 775916+0000 osd.2 (osd.2) 1102 : cluster 2021-02-10T04:34:49.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:49.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: cluster 2021-02-10T04:34:48 2021-02-10T04:34:49.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: .775920+0000 osd.2 (osd.2) 1103 : cluster 2021-02-10T04:34:49.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:49.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: cluster 2021-02-10T04:34: 2021-02-10T04:34:49.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: 48.775924+0000 osd.2 (osd.2) 1104 : 2021-02-10T04:34:49.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:49.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:49.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: 34:48.775928+0000 osd.2 (osd.2) 2021-02-10T04:34:49.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: 1105 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:49.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:34:49.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: 04:34:48.775932+0000 osd.2 (osd. 2021-02-10T04:34:49.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: 2) 1106 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:49.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:49.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: -10T04:34:49.164876+0000 mon.a 2021-02-10T04:34:49.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: (mon.0) 1282 : cluster [DBG] osdmap e472: 8 total, 8 up, 8 in 2021-02-10T04:34:49.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[35961]: 2021-02-10T04:34:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: cluster 2021-02-10T04:34:48.240521+0000 osd.4 (osd.4) 637 : cluster [WRN] 2021-02-10T04:34:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: cluster 2021-02-10T04:34:48.240530+0000 osd.4 (osd.4) 638 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: cluster 2021-02-10T04:34:48.240550+0000 osd.4 (osd.4) 639 : cluster 2021-02-10T04:34:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: cluster 2021-02-10T04:34:48.240556+0000 osd.4 2021-02-10T04:34:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: (osd.4) 640 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: :34:48.467211+0000 mon.a (mon.0) 1281 : cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 317 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:34:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: 2021-02-10T04:34:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: cluster 2021-02-10T04:34:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: 2021-02-10T04:34: 2021-02-10T04:34:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: 48.775891+0000 osd.2 (osd.2) 1097 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: cluster 2021-02-10T04:34:48.775900+0000 osd.2 2021-02-10T04:34:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: (osd.2) 1098 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: 2021-02-10T04:34:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: 04:34:48.775904+0000 osd.2 (osd.2 2021-02-10T04:34:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: ) 1099 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: 10T04:34:48.775908+0000 osd.2 (osd.2) 1100 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: 02-10T04:34:48.775912+0000 osd.2 (osd.2) 1101 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: 10T04:34:48.775916+0000 osd.2 (osd.2) 1102 : 2021-02-10T04:34:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: cluster 2021-02-10T04:34 2021-02-10T04:34:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: :48.775920+0000 osd.2 (osd.2) 1103 2021-02-10T04:34:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: cluster 2021-02-10T04:34:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: 2021-02-10T04:34:48.775924+0000 osd.2 (osd. 2021-02-10T04:34:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: 2) 1104 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: -10T04:34:48.775928+0000 osd.2 (osd.2) 1105 : 2021-02-10T04:34:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:49.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: cluster 2021-02-10T04:34:49.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: 2021-02-10T04:34:48.775932+0000 osd.2 (osd.2 2021-02-10T04:34:49.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: ) 1106 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:49.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:49.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: 10T04:34:49.164876+0000 mon.a (mon.0) 1282 : cluster 2021-02-10T04:34:49.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:49 smithi164 conmon[29288]: [DBG] osdmap e472: 8 total, 8 up, 8 in 2021-02-10T04:34:50.012 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:49 smithi085 conmon[49106]: debug 2021-02-10T04:34:49.735+0000 7f77a3e94700 -1 osd.2 472 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:50.170 INFO:tasks.rados.rados.0.smithi085.stdout:1802: done (11 left) 2021-02-10T04:34:50.170 INFO:tasks.rados.rados.0.smithi085.stdout:1803: write oid 16 current snap is 196 2021-02-10T04:34:50.170 INFO:tasks.rados.rados.0.smithi085.stdout:1803: seq_num 636 ranges {689252=715663,1900783=425211} 2021-02-10T04:34:50.172 INFO:tasks.rados.rados.0.smithi085.stdout:1803: writing smithi08575397-16 from 689252 to 1404915 tid 1 2021-02-10T04:34:50.178 INFO:tasks.rados.rados.0.smithi085.stdout:1803: writing smithi08575397-16 from 1900783 to 2325994 tid 2 2021-02-10T04:34:50.178 INFO:tasks.rados.rados.0.smithi085.stdout:1804: rollback oid 50 current snap is 196 2021-02-10T04:34:50.178 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 50 to 190 2021-02-10T04:34:50.186 INFO:tasks.rados.rados.0.smithi085.stdout:1797: expect (ObjNum 204 snap 44 seq_num 204) 2021-02-10T04:34:50.417 INFO:tasks.rados.rados.0.smithi085.stdout:1801: finishing rollback tid 0 to smithi08575397-30 2021-02-10T04:34:50.417 INFO:tasks.rados.rados.0.smithi085.stdout:1801: finishing rollback tid 1 to smithi08575397-30 2021-02-10T04:34:50.417 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 762 (ObjNum 82554944 snap 22030 seq_num 82786656) dirty dne 2021-02-10T04:34:50.417 INFO:tasks.rados.rados.0.smithi085.stdout:1803: finishing write tid 1 to smithi08575397-16 2021-02-10T04:34:50.417 INFO:tasks.rados.rados.0.smithi085.stdout:1803: finishing write tid 2 to smithi08575397-16 2021-02-10T04:34:50.417 INFO:tasks.rados.rados.0.smithi085.stdout:1803: finishing write tid 3 to smithi08575397-16 2021-02-10T04:34:50.418 INFO:tasks.rados.rados.0.smithi085.stdout:1797: done (12 left) 2021-02-10T04:34:50.418 INFO:tasks.rados.rados.0.smithi085.stdout:1801: done (11 left) 2021-02-10T04:34:50.418 INFO:tasks.rados.rados.0.smithi085.stdout:1805: write (excl) oid 44 current snap is 196 2021-02-10T04:34:50.418 INFO:tasks.rados.rados.0.smithi085.stdout:1805: seq_num 637 ranges {706059=790912,1959265=685222} 2021-02-10T04:34:50.426 INFO:tasks.rados.rados.0.smithi085.stdout:1805: writing smithi08575397-44 from 706059 to 1496971 tid 1 2021-02-10T04:34:50.434 INFO:tasks.rados.rados.0.smithi085.stdout:1805: writing smithi08575397-44 from 1959265 to 2644487 tid 2 2021-02-10T04:34:50.434 INFO:tasks.rados.rados.0.smithi085.stdout:1806: rollback oid 28 current snap is 196 2021-02-10T04:34:50.434 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 28 to 193 2021-02-10T04:34:50.434 INFO:tasks.rados.rados.0.smithi085.stdout:1803: finishing write tid 4 to smithi08575397-16 2021-02-10T04:34:50.434 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 968 (ObjNum 636 snap 196 seq_num 636) dirty exists 2021-02-10T04:34:50.434 INFO:tasks.rados.rados.0.smithi085.stdout:1803: left oid 16 (ObjNum 636 snap 196 seq_num 636) 2021-02-10T04:34:50.434 INFO:tasks.rados.rados.0.smithi085.stdout:1804: finishing rollback tid 1 to smithi08575397-50 2021-02-10T04:34:50.435 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 634 (ObjNum 82483952 snap 22030 seq_num 82275568) dirty dne 2021-02-10T04:34:50.435 INFO:tasks.rados.rados.0.smithi085.stdout:1803: done (12 left) 2021-02-10T04:34:50.435 INFO:tasks.rados.rados.0.smithi085.stdout:1804: done (11 left) 2021-02-10T04:34:50.435 INFO:tasks.rados.rados.0.smithi085.stdout:1807: read oid 21 snap -1 2021-02-10T04:34:50.435 INFO:tasks.rados.rados.0.smithi085.stdout:1807: expect (ObjNum 624 snap 190 seq_num 624) 2021-02-10T04:34:50.435 INFO:tasks.rados.rados.0.smithi085.stdout:1808: rollback oid 8 current snap is 196 2021-02-10T04:34:50.435 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 8 to 184 2021-02-10T04:34:50.435 INFO:tasks.rados.rados.0.smithi085.stdout:1809: copy_from oid 13 from oid 26 current snap is 196 2021-02-10T04:34:50.435 INFO:tasks.rados.rados.0.smithi085.stdout:1810: delete oid 12 current snap is 196 2021-02-10T04:34:50.435 INFO:tasks.rados.rados.0.smithi085.stdout:1805: finishing write tid 1 to smithi08575397-44 2021-02-10T04:34:50.437 INFO:tasks.rados.rados.0.smithi085.stdout:1810: done (14 left) 2021-02-10T04:34:50.437 INFO:tasks.rados.rados.0.smithi085.stdout:1811: write (excl) oid 14 current snap is 196 2021-02-10T04:34:50.437 INFO:tasks.rados.rados.0.smithi085.stdout:1811: seq_num 639 ranges {739673=657763,2076229=663444} 2021-02-10T04:34:50.441 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[52836]: debug 2021-02-10T04:34:50.441 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[52836]: 2021-02-10T04:34:50.010+0000 7fba75fba700 -1 osd.3 pg_epoch: 472 pg[4.10( v 62'12 (0'0,62'12] local-lis/les=71/72 n=10 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,5,6] r=0 lpr=71 crt=62'12 lcod 62'11 mlcod 0'0 active+clean] scrubber pg(4.10) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6 (0x55fec07a9340) 2021-02-10T04:34:50.441 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[52836]: 2021-02-10T04:34:50.447 INFO:tasks.rados.rados.0.smithi085.stdout:1811: writing smithi08575397-14 from 739673 to 1397436 tid 1 2021-02-10T04:34:50.456 INFO:tasks.rados.rados.0.smithi085.stdout:1811: writing smithi08575397-14 from 2076229 to 2739673 tid 2 2021-02-10T04:34:50.456 INFO:tasks.rados.rados.0.smithi085.stdout:1812: read oid 12 snap -1 2021-02-10T04:34:50.457 INFO:tasks.rados.rados.0.smithi085.stdout:1812: expect deleted 2021-02-10T04:34:50.457 INFO:tasks.rados.rados.0.smithi085.stdout:1806: finishing rollback tid 1 to smithi08575397-28 2021-02-10T04:34:50.457 INFO:tasks.rados.rados.0.smithi085.stdout:1806: finishing rollback tid 2 to smithi08575397-28 2021-02-10T04:34:50.457 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 914 (ObjNum 627 snap 190 seq_num 627) dirty exists 2021-02-10T04:34:50.457 INFO:tasks.rados.rados.0.smithi085.stdout:1805: finishing write tid 2 to smithi08575397-44 2021-02-10T04:34:50.457 INFO:tasks.rados.rados.0.smithi085.stdout:1805: finishing write tid 3 to smithi08575397-44 2021-02-10T04:34:50.457 INFO:tasks.rados.rados.0.smithi085.stdout:1805: finishing write tid 4 to smithi08575397-44 2021-02-10T04:34:50.457 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 732 (ObjNum 637 snap 196 seq_num 637) dirty exists 2021-02-10T04:34:50.457 INFO:tasks.rados.rados.0.smithi085.stdout:1805: left oid 44 (ObjNum 637 snap 196 seq_num 637) 2021-02-10T04:34:50.457 INFO:tasks.rados.rados.0.smithi085.stdout:1808: finishing rollback tid 1 to smithi08575397-8 2021-02-10T04:34:50.457 INFO:tasks.rados.rados.0.smithi085.stdout:1807: expect (ObjNum 624 snap 190 seq_num 624) 2021-02-10T04:34:50.492 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[33677]: debug 2021-02-10T04:34:50.177+0000 7fbec19a1700 -1 osd.4 473 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:50.495 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: cluster 2021-02-10T04:34:48.554010+0000 osd.1 (osd.1) 2021-02-10T04:34:50.495 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: 447 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:34:50.495 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: cluster 2021-02-10T04:34:48.554252+0000 osd.1 (osd.1) 448 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:34:50.495 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: cluster 2021-02-10T04:34:49.208996+0000 osd.4 (osd 2021-02-10T04:34:50.495 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: .4) 641 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:50.496 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:50.496 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: -02-10T04:34:49.209005+0000 osd.4 2021-02-10T04:34:50.496 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: (osd.4) 642 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:50.496 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:50.496 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: 02-10T04:34:49.209028+0000 osd.4 (osd 2021-02-10T04:34:50.496 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: .4) 643 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:50.496 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:50.496 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: -02-10T04:34:49.209034+0000 osd.4 2021-02-10T04:34:50.496 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: (osd.4) 644 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:50.496 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: cluster 2021-02-10T04:34:50.496 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: 2021-02-10T04:34 2021-02-10T04:34:50.496 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: :49.736312+0000 osd.2 (osd.2) 1107 : 2021-02-10T04:34:50.497 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:50.497 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:50.497 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: 02-10T04:34:49.736323+0000 osd.2 2021-02-10T04:34:50.497 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: (osd.2) 1108 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:50.497 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:50.497 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: 02-10T04:34:49.736329+0000 osd.2 (osd.2) 1109 2021-02-10T04:34:50.497 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:50.497 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: cluster 2021-02-10T04:34:50.497 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: 2021-02-10T04:34:49.736335+0000 osd.2 (osd.2) 1110 : cluster 2021-02-10T04:34:50.497 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:50.497 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:50.498 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: -02-10T04:34:49.736342+0000 osd.2 (osd.2) 1111 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:50.498 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: cluster 2021-02-10T04:34 2021-02-10T04:34:50.498 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: :49.736348+0000 osd.2 (osd.2) 1112 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:50.498 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: 2021-02-10T04:34:50.498 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:50.498 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: 04:34:49.736354+0000 osd.2 (osd.2) 1113 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:50.498 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: cluster 2021-02-10T04:34:49.736357+0000 2021-02-10T04:34:50.498 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: osd.2 (osd.2) 1114 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:50.498 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: 2021-02-10T04:34:50.498 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:50.498 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: 02-10T04:34:49.736360+0000 2021-02-10T04:34:50.498 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: osd.2 (osd.2) 1115 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:50.499 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:50.499 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: T04:34:49.736364+0000 osd.2 (osd.2) 1116 : cluster 2021-02-10T04:34:50.499 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:50.499 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: cluster 2021-02-10T04:34: 2021-02-10T04:34:50.499 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: 50.091979+0000 mgr.y (mgr.14140) 571 : cluster 2021-02-10T04:34:50.499 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: [DBG] pgmap v858: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 391 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 285 KiB/s wr, 2 op/s 2021-02-10T04:34:50.499 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: cluster 2021-02-10T04:34:50. 2021-02-10T04:34:50.499 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: 163356+0000 mon.a (mon.0) 1283 : cluster 2021-02-10T04:34:50.499 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:50 smithi164 conmon[29288]: [DBG] osdmap e473: 8 total, 8 up, 8 in 2021-02-10T04:34:50.697 INFO:tasks.rados.rados.0.smithi085.stdout:1808: finishing rollback tid 2 to smithi08575397-8 2021-02-10T04:34:50.697 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 734 (ObjNum 521 snap 155 seq_num 521) dirty exists 2021-02-10T04:34:50.697 INFO:tasks.rados.rados.0.smithi085.stdout:1811: finishing write tid 1 to smithi08575397-14 2021-02-10T04:34:50.697 INFO:tasks.rados.rados.0.smithi085.stdout:1811: finishing write tid 2 to smithi08575397-14 2021-02-10T04:34:50.697 INFO:tasks.rados.rados.0.smithi085.stdout:1811: finishing write tid 3 to smithi08575397-14 2021-02-10T04:34:50.698 INFO:tasks.rados.rados.0.smithi085.stdout:1811: finishing write tid 4 to smithi08575397-14 2021-02-10T04:34:50.698 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 779 (ObjNum 639 snap 196 seq_num 639) dirty exists 2021-02-10T04:34:50.698 INFO:tasks.rados.rados.0.smithi085.stdout:1811: left oid 14 (ObjNum 639 snap 196 seq_num 639) 2021-02-10T04:34:50.698 INFO:tasks.rados.rados.0.smithi085.stdout:1809: finishing copy_from to smithi08575397-13 2021-02-10T04:34:50.698 INFO:tasks.rados.rados.0.smithi085.stdout:1809: got expected ENOENT (src dne) 2021-02-10T04:34:50.698 INFO:tasks.rados.rados.0.smithi085.stdout:1805: done (15 left) 2021-02-10T04:34:50.698 INFO:tasks.rados.rados.0.smithi085.stdout:1806: done (14 left) 2021-02-10T04:34:50.698 INFO:tasks.rados.rados.0.smithi085.stdout:1807: done (13 left) 2021-02-10T04:34:50.698 INFO:tasks.rados.rados.0.smithi085.stdout:1808: done (12 left) 2021-02-10T04:34:50.698 INFO:tasks.rados.rados.0.smithi085.stdout:1811: done (11 left) 2021-02-10T04:34:50.698 INFO:tasks.rados.rados.0.smithi085.stdout:1813: read oid 1 snap -1 2021-02-10T04:34:50.698 INFO:tasks.rados.rados.0.smithi085.stdout:1813: expect (ObjNum 625 snap 190 seq_num 625) 2021-02-10T04:34:50.698 INFO:tasks.rados.rados.0.smithi085.stdout:1814: delete oid 40 current snap is 196 2021-02-10T04:34:50.699 INFO:tasks.rados.rados.0.smithi085.stdout:1809: finishing copy_from racing read to smithi08575397-13 2021-02-10T04:34:50.708 INFO:tasks.rados.rados.0.smithi085.stdout:1813: expect (ObjNum 625 snap 190 seq_num 625) 2021-02-10T04:34:50.822 INFO:tasks.rados.rados.0.smithi085.stdout:1809: done (12 left) 2021-02-10T04:34:50.823 INFO:tasks.rados.rados.0.smithi085.stdout:1812: done (11 left) 2021-02-10T04:34:50.823 INFO:tasks.rados.rados.0.smithi085.stdout:1813: done (10 left) 2021-02-10T04:34:50.823 INFO:tasks.rados.rados.0.smithi085.stdout:1814: done (9 left) 2021-02-10T04:34:50.823 INFO:tasks.rados.rados.0.smithi085.stdout:1815: snap_remove snap 160 2021-02-10T04:34:50.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[49106]: debug 2021-02-10T04:34:50.706+0000 7f77a3e94700 -1 osd.2 473 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:50.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: cluster 2021-02-10T04:34:48.554010+0000 osd.1 (osd.1) 447 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:34:50.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:50.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: 04:34:48.554252+0000 osd.1 (osd.1) 448 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:34:50.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: cluster 2021-02-10T04:34:49.208996+0000 osd.4 (osd.4) 641 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:50.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: cluster 2021-02-10T04:34: 2021-02-10T04:34:50.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: 49.209005+0000 osd.4 (osd.4) 642 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:50.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: cluster 2021-02-10T04:34:49.209028 2021-02-10T04:34:50.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 643 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:50.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:50.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: -10T04:34:49.209034+0000 osd.4 2021-02-10T04:34:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: (osd.4) 644 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: 04:34:49.736312+0000 osd.2 (osd.2) 1107 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: cluster 2021-02-10T04:34:49.736323+0000 2021-02-10T04:34:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: osd.2 (osd.2) 1108 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: cluster 2021-02-10T04:34:49.736329+0000 osd.2 (osd.2) 1109 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: 2021-02-10T04:34:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: cluster 2021-02-10T04:34:49.736335+0000 osd.2 ( 2021-02-10T04:34:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: osd.2) 1110 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: cluster 2021-02-10T04:34:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: 2021-02-10T04:34:49.736342+0000 osd.2 ( 2021-02-10T04:34:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: osd.2) 1111 : cluster 2021-02-10T04:34:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: cluster 2021-02-10T04:34:49.736348+0000 osd.2 (osd.2) 1112 : cluster 2021-02-10T04:34:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: cluster 2021-02-10T04:34:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: 2021-02-10T04:34:49.736354+0000 osd.2 (osd.2) 1113 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: cluster 2021-02-10T04:34:49.736357+0000 osd.2 (osd.2) 1114 : cluster [WRN] 2021-02-10T04:34:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: cluster 2021-02-10T04:34:49.736360+0000 osd.2 (osd 2021-02-10T04:34:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: .2) 1115 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: cluster 2021-02-10T04:34:49.736364+0000 osd.2 (osd.2) 1116 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: cluster 2021-02-10T04:34:50.091979+0000 mgr.y (mgr.14140) 571 : 2021-02-10T04:34:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: cluster [DBG] pgmap v858: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 391 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 285 KiB/s wr, 2 op/s 2021-02-10T04:34:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:34:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: 34:50.163356+0000 mon.a (mon.0) 1283 : cluster 2021-02-10T04:34:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[29319]: [DBG] osdmap e473: 8 total, 8 up, 8 in 2021-02-10T04:34:50.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: cluster 2021-02-10T04:34:48.554010+0000 osd.1 ( 2021-02-10T04:34:50.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: osd.1) 447 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:34:50.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:50.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: 34:48.554252+0000 osd.1 (osd.1) 448 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:34:50.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:34:50.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: 04:34:49.208996+0000 osd.4 (osd.4) 641 : cluster [WRN] 2021-02-10T04:34:50.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:50.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: cluster 2021-02-10T04:34:49.209005+0000 osd.4 (osd.4 2021-02-10T04:34:50.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: ) 642 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:50.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: cluster 2021-02-10T04:34:49.209028+0000 osd.4 (osd.4) 643 2021-02-10T04:34:50.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:50.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: cluster 2021-02-10T04:34:49.209034+0000 osd.4 (osd.4) 2021-02-10T04:34:50.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: 644 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:50.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:50.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: 10T04:34:49.736312+0000 osd.2 (osd.2) 1107 : 2021-02-10T04:34:50.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:50.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: cluster 2021-02-10T04:34:49.736323+0000 osd.2 (osd.2) 1108 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:50.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:34:50.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: :34:49.736329+0000 osd.2 (osd.2) 1109 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:50.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:50.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: 10T04:34:49.736335+0000 osd.2 (osd.2) 1110 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:50.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: cluster 2021-02-10T04:34:49. 2021-02-10T04:34:50.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: 736342+0000 osd.2 (osd.2) 1111 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:50.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: cluster 2021-02-10T04:34:49.736348 2021-02-10T04:34:50.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1112 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:50.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: cluster 2021-02-10T04:34:49.736354+0000 osd.2 (osd.2) 1113 : cluster 2021-02-10T04:34:50.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:50.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: cluster 2021-02-10T04:34:49.736357+0000 osd.2 (osd.2) 1114 2021-02-10T04:34:50.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:50.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: cluster 2021-02-10T04:34:49.736360+0000 osd.2 ( 2021-02-10T04:34:50.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: osd.2) 1115 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:50.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: cluster 2021-02-10T04:34:49 2021-02-10T04:34:50.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: .736364+0000 osd.2 (osd.2) 1116 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:50.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: cluster 2021-02-10T04:34:50.091979+0000 mgr.y (mgr.14140) 571 : cluster [DBG] pgmap v858: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 391 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 285 KiB/s wr, 2 op/s 2021-02-10T04:34:50.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: cluster 2021-02-10T04:34:50.163356 2021-02-10T04:34:50.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1283 : cluster [DBG] osdmap e473: 8 total, 8 up, 8 in 2021-02-10T04:34:50.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:50 smithi085 conmon[35961]: 2021-02-10T04:34:51.302 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:34:51.306 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[49106]: debug 2021-02-10T04:34:51.012+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:34:51.408 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:34:51.444 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:34:51.500 INFO:tasks.rados.rados.0.smithi085.stdout:1815: done (9 left) 2021-02-10T04:34:51.500 INFO:tasks.rados.rados.0.smithi085.stdout:1816: read oid 36 snap 194 2021-02-10T04:34:51.500 INFO:tasks.rados.rados.0.smithi085.stdout:1816: expect (ObjNum 616 snap 186 seq_num 616) 2021-02-10T04:34:51.500 INFO:tasks.rados.rados.0.smithi085.stdout:1817: snap_remove snap 170 2021-02-10T04:34:51.502 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[33677]: debug 2021-02-10T04:34:51.166+0000 7fbec19a1700 -1 osd.4 473 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:51.651 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[52836]: debug 2021-02-10T04:34:51.541+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:34:51.655 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: cluster 2021-02-10T04:34:51.655 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: 2021-02-10T04:34:51.655 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:34:51.656 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: 34:50.178813+0000 osd.4 (osd.4) 645 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:51.656 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: 2021-02-10T04:34:51.656 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: cluster 2021-02-10T04:34:51.656 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: 2021-02-10T04:34:51.656 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: 2021 2021-02-10T04:34:51.656 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: - 2021-02-10T04:34:51.656 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: 02 2021-02-10T04:34:51.656 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: - 2021-02-10T04:34:51.656 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: 10 2021-02-10T04:34:51.656 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: T 2021-02-10T04:34:51.656 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: 04 2021-02-10T04:34:51.656 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: : 2021-02-10T04:34:51.657 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: 34 2021-02-10T04:34:51.657 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: : 2021-02-10T04:34:51.657 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: 50 2021-02-10T04:34:51.657 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: . 2021-02-10T04:34:51.657 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: 178821 2021-02-10T04:34:51.657 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: +0000 2021-02-10T04:34:51.657 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: 2021-02-10T04:34:51.657 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: osd.4 2021-02-10T04:34:51.657 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: ( 2021-02-10T04:34:51.657 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: osd.4) 646 : 2021-02-10T04:34:51.657 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:51.657 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:51.657 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: -02-10T04:34:50.178827+0000 osd.4 ( 2021-02-10T04:34:51.658 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: osd.4) 647 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:51.658 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:34:51.658 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: 34:50.178831+0000 osd.4 (osd.4) 648 : cluster [WRN] 2021-02-10T04:34:51.658 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:51.658 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:51.658 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: 02-10T04:34:50.707598+0000 osd.2 (osd.2 2021-02-10T04:34:51.658 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: ) 1117 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:51.658 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:34:51.658 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: :34:50.707608+0000 osd.2 (osd.2) 1118 : cluster [WRN] 2021-02-10T04:34:51.658 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:51.658 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:51.658 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: 02-10T04:34:50.707614+0000 osd.2 (osd.2) 1119 2021-02-10T04:34:51.658 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:51.659 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: cluster 2021-02-10T04:34:51.659 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: 2021-02-10T04:34:51.659 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: 2021 2021-02-10T04:34:51.659 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: - 2021-02-10T04:34:51.659 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: 02 2021-02-10T04:34:51.659 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: - 2021-02-10T04:34:51.659 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: 10 2021-02-10T04:34:51.659 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: T 2021-02-10T04:34:51.659 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: 04 2021-02-10T04:34:51.659 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: :34:50.707620 2021-02-10T04:34:51.659 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1120 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:51.659 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: cluster 2021-02-10T04:34:51.659 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:34:51.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: 04:34:50.707626+0000 osd.2 (osd.2) 1121 : 2021-02-10T04:34:51.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:51.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:51.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: -02-10T04:34:50.707634+0000 2021-02-10T04:34:51.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: osd.2 (osd.2) 1122 : cluster [WRN] 2021-02-10T04:34:51.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:51.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: cluster 2021-02-10T04:34:50. 2021-02-10T04:34:51.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: 707647+0000 osd.2 (osd.2) 1123 : cluster 2021-02-10T04:34:51.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:51.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:34:51.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: :34:50.707653+0000 osd.2 (osd.2 2021-02-10T04:34:51.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: ) 1124 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:51.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:51.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: 10T04:34:50.707659+0000 osd.2 (osd 2021-02-10T04:34:51.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: .2) 1125 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:51.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:51.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: 02-10T04:34:50.707664+0000 2021-02-10T04:34:51.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: osd.2 (osd.2) 1126 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:51.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[29319]: 2021-02-10T04:34:51.662 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: cluster 2021-02-10T04:34:50.178813+0000 osd.4 ( 2021-02-10T04:34:51.662 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: osd.4) 645 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:51.662 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: cluster 2021-02-10T04:34:50.178821+0000 osd.4 (osd.4) 646 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:51.662 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: cluster 2021-02-10T04:34:50.178827+0000 osd.4 (osd.4) 647 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:51.662 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: cluster 2021-02-10T04:34:50.178831+0000 osd.4 (osd.4) 648 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:51.662 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: cluster 2021-02-10T04:34:50.707598+0000 osd.2 (osd.2) 1117 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:51.662 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: cluster 2021-02-10T04:34:50.707608+0000 osd.2 (osd.2) 1118 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:51.662 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: cluster 2021-02-10T04:34:50. 2021-02-10T04:34:51.663 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: 707614+0000 osd.2 (osd.2) 1119 : cluster 2021-02-10T04:34:51.663 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:51.663 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: cluster 2021-02-10T04:34:50 2021-02-10T04:34:51.663 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: .707620+0000 osd.2 (osd.2) 1120 : cluster 2021-02-10T04:34:51.663 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:51.663 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: cluster 2021-02-10T04:34: 2021-02-10T04:34:51.663 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: 50.707626+0000 osd.2 (osd.2) 1121 2021-02-10T04:34:51.663 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:51.663 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:34:51.663 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: 34:50.707634+0000 osd.2 (osd.2) 2021-02-10T04:34:51.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: 1122 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:51.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:34:51.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: 04:34:50.707647+0000 osd.2 (osd. 2021-02-10T04:34:51.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: 2) 1123 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:51.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:51.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: -10T04:34:50.707653+0000 osd.2 2021-02-10T04:34:51.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: (osd.2) 1124 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:51.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: 2021-02-10T04:34:51.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: cluster 2021-02-10T04:34:51.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: 2021-02-10T04:34:50.707659+0000 osd.2 (osd 2021-02-10T04:34:51.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: .2) 1125 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:51.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:51.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: T04:34:50.707664+0000 osd.2 (osd 2021-02-10T04:34:51.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[35961]: .2) 1126 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:51.669 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:34:51.669 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [], 2021-02-10T04:34:51.669 INFO:teuthology.orchestra.run.smithi085.stdout: "complaint_time": 30, 2021-02-10T04:34:51.669 INFO:teuthology.orchestra.run.smithi085.stdout: "num_blocked_ops": 0 2021-02-10T04:34:51.669 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:34:51.841 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:34:51.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: cluster 2021-02-10T04:34:50.178813+0000 osd.4 (osd.4) 2021-02-10T04:34:51.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: 645 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:51.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: cluster 2021-02-10T04:34:50.178821+0000 osd.4 (osd.4) 646 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:51.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: cluster 2021-02-10T04:34:50.178827+0000 osd.4 (osd.4 2021-02-10T04:34:51.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: ) 647 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:51.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:51.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: T04:34:50.178831+0000 osd.4 (osd 2021-02-10T04:34:51.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: .4) 648 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:51.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:51.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: 02-10T04:34:50.707598+0000 osd.2 2021-02-10T04:34:51.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: (osd.2) 1117 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:51.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: cluster 2021-02-10T04:34:51.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: 2021-02-10T04:34:50.707608 2021-02-10T04:34:51.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1118 : cluster [WRN] 2021-02-10T04:34:51.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:51.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: cluster 2021-02-10T04:34 2021-02-10T04:34:51.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: :50.707614+0000 osd.2 (osd.2) 1119 2021-02-10T04:34:51.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:51.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:51.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: 04:34:50.707620+0000 osd.2 (osd. 2021-02-10T04:34:51.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: 2) 1120 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:51.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:51.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: 02-10T04:34:50.707626+0000 2021-02-10T04:34:51.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: osd.2 (osd.2) 1121 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:51.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: 2021-02-10T04:34:51.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: cluster 2021-02-10T04:34:51.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: 2021-02-10T04:34:50.707634+0000 osd.2 ( 2021-02-10T04:34:51.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: osd.2) 1122 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:51.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:51.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: -02-10T04:34:50.707647 2021-02-10T04:34:51.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1123 : cluster [WRN] 2021-02-10T04:34:51.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:51.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:51.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: 34:50.707653+0000 osd.2 (osd.2) 2021-02-10T04:34:51.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: 1124 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:51.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:51.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: 04:34:50.707659+0000 osd.2 (osd. 2021-02-10T04:34:51.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: 2) 1125 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:51.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:51.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: -02-10T04:34:50.707664+0000 2021-02-10T04:34:51.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: osd.2 (osd.2) 1126 : cluster [WRN] 2021-02-10T04:34:51.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:51 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:51.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:51 smithi085 conmon[49106]: debug 2021-02-10T04:34:51.742+0000 7f77a3e94700 -1 osd.2 474 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:52.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[33677]: debug 2021-02-10T04:34:52.117+0000 7fbec19a1700 -1 osd.4 474 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:52.441 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[52836]: debug 2021-02-10T04:34:52.012+0000 7fba71fb2700 -1 osd.3 pg_epoch: 474 pg[4.10( v 62'12 (0'0,62'12] local-lis/les=71/72 n=10 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,5,6] r=0 lpr=71 crt=62'12 lcod 62'11 mlcod 0'0 active+clean] scrubber pg(4.10) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6 (0x55fec06c82c0) 2021-02-10T04:34:52.509 INFO:tasks.rados.rados.0.smithi085.stdout:1817: done (10 left) 2021-02-10T04:34:52.509 INFO:tasks.rados.rados.0.smithi085.stdout:1818: rollback oid 35 current snap is 196 2021-02-10T04:34:52.509 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 35 to 148 2021-02-10T04:34:52.509 INFO:tasks.rados.rados.0.smithi085.stdout:1816: expect (ObjNum 616 snap 186 seq_num 616) 2021-02-10T04:34:52.615 INFO:tasks.rados.rados.0.smithi085.stdout:1816: done (10 left) 2021-02-10T04:34:52.616 INFO:tasks.rados.rados.0.smithi085.stdout:1819: write oid 8 current snap is 196 2021-02-10T04:34:52.616 INFO:tasks.rados.rados.0.smithi085.stdout:1818: finishing rollback tid 0 to smithi08575397-35 2021-02-10T04:34:52.616 INFO:tasks.rados.rados.0.smithi085.stdout:1819: seq_num 641 ranges {773287=408261,1793193=576207} 2021-02-10T04:34:52.622 INFO:tasks.rados.rados.0.smithi085.stdout:1819: writing smithi08575397-8 from 773287 to 1181548 tid 1 2021-02-10T04:34:52.630 INFO:tasks.rados.rados.0.smithi085.stdout:1819: writing smithi08575397-8 from 1793193 to 2369400 tid 2 2021-02-10T04:34:52.630 INFO:tasks.rados.rados.0.smithi085.stdout:1820: delete oid 28 current snap is 196 2021-02-10T04:34:52.630 INFO:tasks.rados.rados.0.smithi085.stdout:1818: finishing rollback tid 1 to smithi08575397-35 2021-02-10T04:34:52.630 INFO:tasks.rados.rados.0.smithi085.stdout:1818: finishing rollback tid 2 to smithi08575397-35 2021-02-10T04:34:52.630 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 35 v 650 (ObjNum 85 snap 11 seq_num 85) dirty exists 2021-02-10T04:34:52.630 INFO:tasks.rados.rados.0.smithi085.stdout:1819: finishing write tid 1 to smithi08575397-8 2021-02-10T04:34:52.632 INFO:tasks.rados.rados.0.smithi085.stdout:1818: done (11 left) 2021-02-10T04:34:52.632 INFO:tasks.rados.rados.0.smithi085.stdout:1820: done (10 left) 2021-02-10T04:34:52.632 INFO:tasks.rados.rados.0.smithi085.stdout:1821: snap_remove snap 188 2021-02-10T04:34:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: cluster 2021-02-10T04:34:51.168317+0000 osd.4 (osd.4) 649 2021-02-10T04:34:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: cluster 2021-02-10T04:34:51.168327+0000 osd.4 (osd.4) 650 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: cluster 2021-02-10T04:34:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: 2021-02-10T04:34:51.168334+0000 osd.4 (osd.4) 651 2021-02-10T04:34:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: cluster 2021-02-10T04:34:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: 2021-02-10T04:34:51.168339+0000 osd.4 ( 2021-02-10T04:34:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: osd.4) 652 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: cluster 2021-02-10T04:34:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: 2021-02-10T04:34:51.495069+0000 mon.a ( 2021-02-10T04:34:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: mon.0) 1284 : cluster [DBG] osdmap e474: 8 total, 8 up, 8 in 2021-02-10T04:34:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: 02-10T04:34:51.744358+0000 osd.2 (osd 2021-02-10T04:34:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: .2) 1127 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: -02-10T04:34:51.744363+0000 2021-02-10T04:34:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: osd.2 (osd.2) 1128 : cluster [WRN] 2021-02-10T04:34:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: 34:51.744366+0000 osd.2 (osd.2) 1129 : 2021-02-10T04:34:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: -10T04:34:51.744369+0000 osd.2 ( 2021-02-10T04:34:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: osd.2) 1130 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: cluster 2021-02-10T04:34:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: 2021-02 2021-02-10T04:34:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: -10T04:34:51.744372+0000 osd.2 ( 2021-02-10T04:34:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: osd.2) 1131 : cluster [WRN] 2021-02-10T04:34:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: -10T04:34:51.744375+0000 osd.2 ( 2021-02-10T04:34:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: osd.2) 1132 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: :34:51.744379+0000 osd.2 (osd.2 2021-02-10T04:34:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: ) 1133 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: cluster 2021-02-10T04:34:51.744382+0000 osd.2 2021-02-10T04:34:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: (osd.2) 1134 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: cluster 2021-02-10T04:34:51.744384+0000 2021-02-10T04:34:52.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: osd.2 (osd.2) 1135 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:52.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: 2021-02-10T04:34:52.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:52.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: -10T04:34:51.744387+0000 osd.2 (osd.2) 1136 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:52.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: cluster 2021-02-10T04:34:52. 2021-02-10T04:34:52.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:52 smithi164 conmon[29288]: 092655+0000 mgr.y (mgr.14140) 572 : cluster [DBG] pgmap v861: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 387 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.7 MiB/s rd, 609 KiB/s wr, 12 op/s 2021-02-10T04:34:52.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[49106]: debug 2021-02-10T04:34:52.707+0000 7f77a3e94700 -1 osd.2 475 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:52.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: cluster 2021-02-10T04:34:52.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 2021-02-10T04:34:51.168317+0000 osd.4 (osd.4) 649 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:52.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: cluster 2021-02-10T04:34:51 2021-02-10T04:34:52.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: .168327+0000 osd.4 (osd.4) 650 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: cluster 2021-02-10T04:34:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 2021-02-10T04:34:51. 2021-02-10T04:34:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 168334+0000 osd.4 (osd.4) 651 : cluster [WRN] 2021-02-10T04:34:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: cluster 2021-02-10T04:34:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 2021-02-10T04:34:51.168339+0000 osd.4 (osd. 2021-02-10T04:34:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 4) 652 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: cluster 2021-02-10T04:34:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:34:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: :34:51.495069+0000 mon.a (mon. 2021-02-10T04:34:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 0) 1284 : cluster [DBG] osdmap e474: 8 total, 8 up, 8 in 2021-02-10T04:34:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: cluster 2021-02-10T04:34 2021-02-10T04:34:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: :51.744358+0000 osd.2 (osd.2) 1127 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: cluster 2021-02-10T04:34:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:34:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: :34:51.744363+0000 osd.2 (osd.2 2021-02-10T04:34:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: ) 1128 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: cluster 2021-02-10T04:34:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 2021-02-10T04:34:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 2021 2021-02-10T04:34:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: - 2021-02-10T04:34:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 02 2021-02-10T04:34:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: - 2021-02-10T04:34:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 10T04:34:51.744366 2021-02-10T04:34:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1129 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:52.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: cluster 2021-02-10T04:34:52.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 2021 2021-02-10T04:34:52.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: -02-10T04:34:51.744369+0000 2021-02-10T04:34:52.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: osd.2 (osd.2) 1130 : cluster [WRN] 2021-02-10T04:34:52.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:52.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: cluster 2021-02-10T04:34:52.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:34:52.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 04:34:51.744372+0000 osd.2 (osd.2 2021-02-10T04:34:52.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: ) 1131 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:52.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: cluster 2021-02-10T04:34:52.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 2021-02 2021-02-10T04:34:52.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: -10T04:34:51 2021-02-10T04:34:52.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: .744375+0000 osd.2 (osd.2) 1132 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:52.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: cluster 2021-02-10T04:34:52.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 2021-02-10T04:34:52.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:34:52.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: T04:34:51.744379+0000 2021-02-10T04:34:52.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: osd.2 (osd.2) 1133 2021-02-10T04:34:52.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:52.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: cluster 2021-02-10T04:34:52.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 2021-02-10T04:34:52.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 2021 2021-02-10T04:34:52.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: -02-10T04:34: 2021-02-10T04:34:52.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 51.744382+0000 osd.2 (osd.2) 1134 : cluster [WRN] 2021-02-10T04:34:52.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:52.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: cluster 2021-02-10T04:34:52.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:34:52.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 04:34:51.744384+0000 osd.2 (osd.2) 1135 : cluster 2021-02-10T04:34:52.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:52.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: cluster 2021-02-10T04:34:52.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 2021 2021-02-10T04:34:52.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: - 2021-02-10T04:34:52.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 02 2021-02-10T04:34:52.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: - 2021-02-10T04:34:52.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 10 2021-02-10T04:34:52.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: T 2021-02-10T04:34:52.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 04 2021-02-10T04:34:52.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: : 2021-02-10T04:34:52.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 34 2021-02-10T04:34:52.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: :51.744387+0000 osd.2 (osd 2021-02-10T04:34:52.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: .2) 1136 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:52.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: cluster 2021-02-10T04:34:52.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 2021-02-10T04:34:52.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:34:52.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: :34:52.092655+0000 mgr.y (mgr.14140) 2021-02-10T04:34:52.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[29319]: 572 : cluster [DBG] pgmap v861: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 387 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.7 MiB/s rd, 609 KiB/s wr, 12 op/s 2021-02-10T04:34:52.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: cluster 2021-02-10T04:34:52.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: 2021-02-10T04:34:52.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: 2021 2021-02-10T04:34:52.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: - 2021-02-10T04:34:52.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: 02 2021-02-10T04:34:52.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: - 2021-02-10T04:34:52.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: 10 2021-02-10T04:34:52.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: T 2021-02-10T04:34:52.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: 04 2021-02-10T04:34:52.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: : 2021-02-10T04:34:52.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: 34 2021-02-10T04:34:52.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: : 2021-02-10T04:34:52.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: 51 2021-02-10T04:34:52.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: . 2021-02-10T04:34:52.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: 168317 2021-02-10T04:34:52.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: +0000 2021-02-10T04:34:52.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: 2021-02-10T04:34:52.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: osd.4 2021-02-10T04:34:52.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: ( 2021-02-10T04:34:52.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: osd 2021-02-10T04:34:52.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: .4) 649 : cluster [WRN] 2021-02-10T04:34:52.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:52.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: cluster 2021-02-10T04:34:52.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:34:52.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: :34:51.168327+0000 osd.4 ( 2021-02-10T04:34:52.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: osd.4) 650 : cluster 2021-02-10T04:34:52.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:52.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:52.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: 02-10T04:34:51.168334+0000 osd.4 ( 2021-02-10T04:34:52.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: osd.4) 651 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:52.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: cluster 2021-02-10T04:34:52.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: 2021-02-10T04:34:51.168339+0000 osd.4 2021-02-10T04:34:52.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: (osd.4) 652 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:52.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: 2021-02-10T04:34:52.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: cluster 2021-02-10T04:34:52.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: 2021-02-10T04:34:51.495069+0000 mon.a (mon.0) 1284 : 2021-02-10T04:34:52.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: cluster [DBG] osdmap e474: 8 total, 8 up, 8 in 2021-02-10T04:34:52.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: cluster 2021-02-10T04:34:51.744358 2021-02-10T04:34:52.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1127 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:52.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: 2021-02-10T04:34:52.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:34:52.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: 04:34:51.744363+0000 osd.2 (osd.2) 1128 : cluster 2021-02-10T04:34:52.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:52.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: cluster 2021-02-10T04:34:51.744366+0000 2021-02-10T04:34:52.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: osd.2 (osd.2) 1129 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:52.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:52.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: 10T04:34:51.744369+0000 osd.2 (osd.2) 1130 2021-02-10T04:34:52.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:52.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: cluster 2021-02-10T04:34:51.744372 2021-02-10T04:34:52.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1131 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:52.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: 2021-02-10T04:34:52.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:52.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: 02-10T04:34:51.744375+0000 osd.2 (osd.2) 1132 : cluster 2021-02-10T04:34:52.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:52.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: cluster 2021-02-10T04:34:51.744379+0000 osd.2 (osd 2021-02-10T04:34:52.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: .2) 1133 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:52.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: cluster 2021-02-10T04:34 2021-02-10T04:34:52.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: :51.744382+0000 osd.2 (osd.2) 1134 : cluster [WRN] 2021-02-10T04:34:52.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:52.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: cluster 2021-02-10T04:34:52.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: 2021-02-10T04:34:51.744384+0000 osd.2 (osd.2) 1135 : cluster 2021-02-10T04:34:52.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:52.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: cluster 2021-02-10T04:34:51.744387+0000 osd.2 2021-02-10T04:34:52.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: (osd.2) 1136 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:52.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:52.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: -10T04:34:52.092655+0000 mgr.y (mgr.14140) 2021-02-10T04:34:52.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:52 smithi085 conmon[35961]: 572 : cluster [DBG] pgmap v861: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 387 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.7 MiB/s rd, 609 KiB/s wr, 12 op/s 2021-02-10T04:34:53.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[33677]: debug 2021-02-10T04:34:53.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[33677]: 2021-02-10T04:34:53.083+0000 7fbec19a1700 -1 osd.4 475 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:53.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[33677]: 2021-02-10T04:34:53.530 INFO:tasks.rados.rados.0.smithi085.stdout:1821: done (10 left) 2021-02-10T04:34:53.530 INFO:tasks.rados.rados.0.smithi085.stdout:1822: write (excl) oid 2 current snap is 196 2021-02-10T04:34:53.530 INFO:tasks.rados.rados.0.smithi085.stdout:1822: seq_num 643 ranges {406901=558759,1393804=579876,2608396=198505} 2021-02-10T04:34:53.532 INFO:tasks.rados.rados.0.smithi085.stdout:1822: writing smithi08575397-2 from 406901 to 965660 tid 1 2021-02-10T04:34:53.539 INFO:tasks.rados.rados.0.smithi085.stdout:1822: writing smithi08575397-2 from 1393804 to 1973680 tid 2 2021-02-10T04:34:53.544 INFO:tasks.rados.rados.0.smithi085.stdout:1822: writing smithi08575397-2 from 2608396 to 2806901 tid 3 2021-02-10T04:34:53.544 INFO:tasks.rados.rados.0.smithi085.stdout:1819: finishing write tid 2 to smithi08575397-8 2021-02-10T04:34:53.544 INFO:tasks.rados.rados.0.smithi085.stdout:1823: copy_from oid 43 from oid 38 current snap is 196 2021-02-10T04:34:53.544 INFO:tasks.rados.rados.0.smithi085.stdout:1819: finishing write tid 3 to smithi08575397-8 2021-02-10T04:34:53.545 INFO:tasks.rados.rados.0.smithi085.stdout:1819: finishing write tid 4 to smithi08575397-8 2021-02-10T04:34:53.545 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 737 (ObjNum 641 snap 196 seq_num 641) dirty exists 2021-02-10T04:34:53.545 INFO:tasks.rados.rados.0.smithi085.stdout:1819: left oid 8 (ObjNum 641 snap 196 seq_num 641) 2021-02-10T04:34:53.545 INFO:tasks.rados.rados.0.smithi085.stdout:1822: finishing write tid 1 to smithi08575397-2 2021-02-10T04:34:53.545 INFO:tasks.rados.rados.0.smithi085.stdout:1819: done (11 left) 2021-02-10T04:34:53.545 INFO:tasks.rados.rados.0.smithi085.stdout:1824: write oid 14 current snap is 196 2021-02-10T04:34:53.545 INFO:tasks.rados.rados.0.smithi085.stdout:1824: seq_num 644 ranges {423708=634008,1852286=439887,2752172=71536} 2021-02-10T04:34:53.551 INFO:tasks.rados.rados.0.smithi085.stdout:1824: writing smithi08575397-14 from 423708 to 1057716 tid 1 2021-02-10T04:34:53.558 INFO:tasks.rados.rados.0.smithi085.stdout:1824: writing smithi08575397-14 from 1852286 to 2292173 tid 2 2021-02-10T04:34:53.562 INFO:tasks.rados.rados.0.smithi085.stdout:1824: writing smithi08575397-14 from 2752172 to 2823708 tid 3 2021-02-10T04:34:53.562 INFO:tasks.rados.rados.0.smithi085.stdout:1825: write oid 21 current snap is 196 2021-02-10T04:34:53.562 INFO:tasks.rados.rados.0.smithi085.stdout:1822: finishing write tid 2 to smithi08575397-2 2021-02-10T04:34:53.562 INFO:tasks.rados.rados.0.smithi085.stdout:1825: seq_num 645 ranges {440515=709257,1910768=583545} 2021-02-10T04:34:53.569 INFO:tasks.rados.rados.0.smithi085.stdout:1825: writing smithi08575397-21 from 440515 to 1149772 tid 1 2021-02-10T04:34:53.578 INFO:tasks.rados.rados.0.smithi085.stdout:1825: writing smithi08575397-21 from 1910768 to 2494313 tid 2 2021-02-10T04:34:53.578 INFO:tasks.rados.rados.0.smithi085.stdout:1822: finishing write tid 3 to smithi08575397-2 2021-02-10T04:34:53.578 INFO:tasks.rados.rados.0.smithi085.stdout:1822: finishing write tid 4 to smithi08575397-2 2021-02-10T04:34:53.578 INFO:tasks.rados.rados.0.smithi085.stdout:1822: finishing write tid 5 to smithi08575397-2 2021-02-10T04:34:53.578 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 654 (ObjNum 643 snap 196 seq_num 643) dirty exists 2021-02-10T04:34:53.578 INFO:tasks.rados.rados.0.smithi085.stdout:1822: left oid 2 (ObjNum 643 snap 196 seq_num 643) 2021-02-10T04:34:53.578 INFO:tasks.rados.rados.0.smithi085.stdout:1823: finishing copy_from to smithi08575397-43 2021-02-10T04:34:53.579 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 658 (ObjNum 616 snap 186 seq_num 616) dirty exists 2021-02-10T04:34:53.579 INFO:tasks.rados.rados.0.smithi085.stdout:1824: finishing write tid 1 to smithi08575397-14 2021-02-10T04:34:53.579 INFO:tasks.rados.rados.0.smithi085.stdout:1823: finishing copy_from racing read to smithi08575397-43 2021-02-10T04:34:53.579 INFO:tasks.rados.rados.0.smithi085.stdout:1824: finishing write tid 2 to smithi08575397-14 2021-02-10T04:34:53.579 INFO:tasks.rados.rados.0.smithi085.stdout:1824: finishing write tid 3 to smithi08575397-14 2021-02-10T04:34:53.579 INFO:tasks.rados.rados.0.smithi085.stdout:1824: finishing write tid 4 to smithi08575397-14 2021-02-10T04:34:53.579 INFO:tasks.rados.rados.0.smithi085.stdout:1824: finishing write tid 5 to smithi08575397-14 2021-02-10T04:34:53.579 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 783 (ObjNum 644 snap 196 seq_num 644) dirty exists 2021-02-10T04:34:53.579 INFO:tasks.rados.rados.0.smithi085.stdout:1824: left oid 14 (ObjNum 644 snap 196 seq_num 644) 2021-02-10T04:34:53.579 INFO:tasks.rados.rados.0.smithi085.stdout:1825: finishing write tid 1 to smithi08575397-21 2021-02-10T04:34:53.579 INFO:tasks.rados.rados.0.smithi085.stdout:1822: done (12 left) 2021-02-10T04:34:53.579 INFO:tasks.rados.rados.0.smithi085.stdout:1823: done (11 left) 2021-02-10T04:34:53.579 INFO:tasks.rados.rados.0.smithi085.stdout:1824: done (10 left) 2021-02-10T04:34:53.580 INFO:tasks.rados.rados.0.smithi085.stdout:1826: copy_from oid 16 from oid 6 current snap is 196 2021-02-10T04:34:53.580 INFO:tasks.rados.rados.0.smithi085.stdout:1827: write oid 46 current snap is 196 2021-02-10T04:34:53.580 INFO:tasks.rados.rados.0.smithi085.stdout:1827: seq_num 646 ranges {457322=784506,1969250=443556} 2021-02-10T04:34:53.585 INFO:tasks.rados.rados.0.smithi085.stdout:1827: writing smithi08575397-46 from 457322 to 1241828 tid 1 2021-02-10T04:34:53.593 INFO:tasks.rados.rados.0.smithi085.stdout:1827: writing smithi08575397-46 from 1969250 to 2412806 tid 2 2021-02-10T04:34:53.593 INFO:tasks.rados.rados.0.smithi085.stdout:1825: finishing write tid 2 to smithi08575397-21 2021-02-10T04:34:53.593 INFO:tasks.rados.rados.0.smithi085.stdout:1828: delete oid 20 current snap is 196 2021-02-10T04:34:53.593 INFO:tasks.rados.rados.0.smithi085.stdout:1825: finishing write tid 3 to smithi08575397-21 2021-02-10T04:34:53.593 INFO:tasks.rados.rados.0.smithi085.stdout:1825: finishing write tid 4 to smithi08575397-21 2021-02-10T04:34:53.594 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 980 (ObjNum 645 snap 196 seq_num 645) dirty exists 2021-02-10T04:34:53.594 INFO:tasks.rados.rados.0.smithi085.stdout:1825: left oid 21 (ObjNum 645 snap 196 seq_num 645) 2021-02-10T04:34:53.594 INFO:tasks.rados.rados.0.smithi085.stdout:1826: finishing copy_from to smithi08575397-16 2021-02-10T04:34:53.594 INFO:tasks.rados.rados.0.smithi085.stdout:1826: got expected ENOENT (src dne) 2021-02-10T04:34:53.594 INFO:tasks.rados.rados.0.smithi085.stdout:1826: finishing copy_from racing read to smithi08575397-16 2021-02-10T04:34:53.594 INFO:tasks.rados.rados.0.smithi085.stdout:1827: finishing write tid 1 to smithi08575397-46 2021-02-10T04:34:53.596 INFO:tasks.rados.rados.0.smithi085.stdout:1825: done (12 left) 2021-02-10T04:34:53.596 INFO:tasks.rados.rados.0.smithi085.stdout:1826: done (11 left) 2021-02-10T04:34:53.596 INFO:tasks.rados.rados.0.smithi085.stdout:1828: done (10 left) 2021-02-10T04:34:53.596 INFO:tasks.rados.rados.0.smithi085.stdout:1829: delete oid 13 current snap is 196 2021-02-10T04:34:53.597 INFO:tasks.rados.rados.0.smithi085.stdout:1827: finishing write tid 2 to smithi08575397-46 2021-02-10T04:34:53.597 INFO:tasks.rados.rados.0.smithi085.stdout:1827: finishing write tid 3 to smithi08575397-46 2021-02-10T04:34:53.598 INFO:tasks.rados.rados.0.smithi085.stdout:1827: finishing write tid 4 to smithi08575397-46 2021-02-10T04:34:53.598 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 765 (ObjNum 646 snap 196 seq_num 646) dirty exists 2021-02-10T04:34:53.598 INFO:tasks.rados.rados.0.smithi085.stdout:1827: left oid 46 (ObjNum 646 snap 196 seq_num 646) 2021-02-10T04:34:53.599 INFO:tasks.rados.rados.0.smithi085.stdout:1827: done (10 left) 2021-02-10T04:34:53.600 INFO:tasks.rados.rados.0.smithi085.stdout:1829: done (9 left) 2021-02-10T04:34:53.600 INFO:tasks.rados.rados.0.smithi085.stdout:1830: rollback oid 5 current snap is 196 2021-02-10T04:34:53.600 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 5 to 189 2021-02-10T04:34:53.600 INFO:tasks.rados.rados.0.smithi085.stdout:1831: read oid 8 snap -1 2021-02-10T04:34:53.600 INFO:tasks.rados.rados.0.smithi085.stdout:1831: expect (ObjNum 641 snap 196 seq_num 641) 2021-02-10T04:34:53.600 INFO:tasks.rados.rados.0.smithi085.stdout:1832: snap_create 2021-02-10T04:34:53.603 INFO:tasks.rados.rados.0.smithi085.stdout:1830: finishing rollback tid 0 to smithi08575397-5 2021-02-10T04:34:53.603 INFO:tasks.rados.rados.0.smithi085.stdout:1830: finishing rollback tid 1 to smithi08575397-5 2021-02-10T04:34:53.604 INFO:tasks.rados.rados.0.smithi085.stdout:1830: finishing rollback tid 2 to smithi08575397-5 2021-02-10T04:34:53.604 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 669 (ObjNum 614 snap 186 seq_num 614) dirty exists 2021-02-10T04:34:53.606 INFO:tasks.rados.rados.0.smithi085.stdout:1831: expect (ObjNum 641 snap 196 seq_num 641) 2021-02-10T04:34:53.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: cluster 2021-02-10T04:34:53.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:34:53.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: :34:52.118770+0000 osd.4 (osd.4) 653 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:53.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:53.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: 04:34:52.118780+0000 osd.4 (osd.4) 654 : cluster 2021-02-10T04:34:53.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:53.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: cluster 2021-02-10T04:34:53.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: 2021-02-10T04:34:52.118787+0000 osd.4 (osd.4) 655 : cluster 2021-02-10T04:34:53.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:53.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: cluster 2021-02-10T04:34:52.118794+0000 2021-02-10T04:34:53.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: osd.4 (osd.4) 656 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:53.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:53.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: -02-10T04:34:52.505400+0000 mon.a 2021-02-10T04:34:53.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: (mon.0) 1285 : cluster [DBG] osdmap e475: 8 total, 8 up, 8 in 2021-02-10T04:34:53.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:53.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: -02-10T04:34:52.708963+0000 2021-02-10T04:34:53.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: osd.2 (osd.2) 1137 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:53.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:53.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: 02-10T04:34:52.708969+0000 osd.2 (osd.2) 1138 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:53.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: 2021-02-10T04:34:53.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: cluster 2021-02-10T04:34:53.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: 2021-02-10T04:34:52.708974+0000 osd.2 (osd 2021-02-10T04:34:53.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: .2) 1139 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:53.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:34:53.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: :34:52.708979+0000 osd.2 (osd.2) 1140 : cluster [WRN] 2021-02-10T04:34:53.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:53.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:53.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: 10T04:34:52.708987+0000 osd.2 (osd.2) 1141 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:53.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:53.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: 02-10T04:34:52.708991+0000 2021-02-10T04:34:53.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: osd.2 (osd.2) 1142 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:53.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:53.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: -02-10T04:34:52.708999+0000 2021-02-10T04:34:53.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: osd.2 (osd.2) 1143 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:53.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: cluster 2021-02-10T04:34:52. 2021-02-10T04:34:53.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: 709006+0000 osd.2 (osd.2) 1144 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:53.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: cluster 2021-02-10T04:34:52.709011+0000 2021-02-10T04:34:53.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: osd.2 (osd.2) 1145 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:53.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: cluster 2021-02-10T04:34:53.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: 2021-02-10T04:34:52.709018+0000 osd.2 (osd.2) 1146 : cluster 2021-02-10T04:34:53.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:53.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: cluster 2021-02-10T04:34:52.118770 2021-02-10T04:34:53.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 653 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:53.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: 02-10T04:34:52.118780+0000 osd.4 (osd.4) 654 : cluster [WRN] 2021-02-10T04:34:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: cluster 2021-02-10T04:34:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: 2021-02-10T04:34:52.118787+0000 osd.4 (osd. 2021-02-10T04:34:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: 4) 655 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: cluster 2021-02-10T04:34:52 2021-02-10T04:34:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: .118794+0000 osd.4 (osd.4) 656 : cluster [WRN] 2021-02-10T04:34:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: cluster 2021-02-10T04:34:52 2021-02-10T04:34:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: .505400+0000 mon.a (mon.0) 1285 : cluster [DBG] osdmap e475: 8 total, 8 up, 8 in 2021-02-10T04:34:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: -10T04:34:52.708963+0000 osd.2 (osd.2) 1137 2021-02-10T04:34:53.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:53.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: cluster 2021-02-10T04:34:52. 2021-02-10T04:34:53.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: 708969 2021-02-10T04:34:53.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1138 2021-02-10T04:34:53.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:53.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:53.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: 02-10T04:34: 2021-02-10T04:34:53.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: 52.708974+0000 osd.2 (osd.2) 2021-02-10T04:34:53.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: 1139 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:53.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:53.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: -02-10T04:34:52.708979+0000 osd.2 (osd.2) 1140 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:53.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: cluster 2021-02-10T04:34 2021-02-10T04:34:53.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: :52.708987+0000 osd.2 (osd.2) 1141 : cluster [WRN] 2021-02-10T04:34:53.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:53.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: cluster 2021-02-10T04:34:53.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: 2021-02-10T04:34:52.708991+0000 osd.2 (osd.2 2021-02-10T04:34:53.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: ) 1142 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:53.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:34:53.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: :34:52.708999+0000 osd.2 (osd.2) 1143 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:53.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: cluster 2021-02-10T04:34:52.709006+0000 osd.2 2021-02-10T04:34:53.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: (osd.2) 1144 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:53.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: cluster 2021-02-10T04:34:52.709011+0000 2021-02-10T04:34:53.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: osd.2 (osd.2) 1145 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:53.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:53.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: -02-10T04:34:52.709018+0000 2021-02-10T04:34:53.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: osd.2 (osd.2) 1146 : cluster [WRN] 2021-02-10T04:34:53.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: cluster 2021-02-10T04:34:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: 2021-02-10T04:34:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: 2021 2021-02-10T04:34:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: - 2021-02-10T04:34:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: 02 2021-02-10T04:34:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: - 2021-02-10T04:34:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: 10 2021-02-10T04:34:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: T 2021-02-10T04:34:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: 04 2021-02-10T04:34:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: : 2021-02-10T04:34:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: 34 2021-02-10T04:34:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: : 2021-02-10T04:34:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: 52 2021-02-10T04:34:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: . 2021-02-10T04:34:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: 118770 2021-02-10T04:34:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: +0000 2021-02-10T04:34:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: 2021-02-10T04:34:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: osd.4 2021-02-10T04:34:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: ( 2021-02-10T04:34:53.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: osd 2021-02-10T04:34:53.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: .4) 653 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:53.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: 2021-02-10T04:34:53.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:53.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: -10T04:34:52.118780+0000 osd.4 (osd.4) 654 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:53.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: 2021-02-10T04:34:53.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:53.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: -10T04:34:52.118787+0000 osd.4 2021-02-10T04:34:53.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: (osd.4) 655 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:53.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:53.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: -02-10T04:34:52.118794+0000 2021-02-10T04:34:53.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: osd.4 (osd.4) 656 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:53.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:53.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: 02-10T04:34:52.505400+0000 mon.a ( 2021-02-10T04:34:53.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: mon.0) 1285 : cluster [DBG] osdmap e475: 8 total, 8 up, 8 in 2021-02-10T04:34:53.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:53.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: -02-10T04:34:52.708963+0000 osd.2 2021-02-10T04:34:53.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: (osd.2) 1137 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:53.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:53.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: 02-10T04:34:52.708969+0000 2021-02-10T04:34:53.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: osd.2 (osd.2) 1138 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:53.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: 2021-02-10T04:34:53.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: cluster 2021-02-10T04:34:53.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: 2021-02-10T04:34:52.708974+0000 osd.2 (osd 2021-02-10T04:34:53.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: .2) 1139 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:53.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:53.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: T04:34:52.708979+0000 osd.2 (osd.2 2021-02-10T04:34:53.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: ) 1140 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:53.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:53.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: T04:34:52.708987+0000 osd.2 (osd.2 2021-02-10T04:34:53.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: ) 1141 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:53.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:53.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: 10T04:34:52.708991+0000 osd.2 (osd 2021-02-10T04:34:53.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: .2) 1142 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:53.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:53.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: 04:34:52.708999+0000 osd.2 (osd.2) 2021-02-10T04:34:53.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: 1143 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:53.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:53.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: :34:52.709006+0000 osd.2 (osd.2) 2021-02-10T04:34:53.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: 1144 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:53.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:53.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: T04:34:52.709011+0000 osd.2 (osd 2021-02-10T04:34:53.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: .2) 1145 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:53.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:53.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: -02-10T04:34:52.709018+0000 2021-02-10T04:34:53.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: osd.2 (osd.2) 1146 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:53.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:53 smithi164 conmon[29288]: 2021-02-10T04:34:53.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:53 smithi085 conmon[49106]: debug 2021-02-10T04:34:53.686+0000 7f77a3e94700 -1 osd.2 476 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:54.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[33677]: debug 2021-02-10T04:34:54.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[33677]: 2021-02-10T04:34:54.133+0000 7fbec19a1700 -1 osd.4 476 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:54.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[33677]: debug 2021-02-10T04:34:54.134+0000 7fbea992c700 -1 osd.4 pg_epoch: 476 pg[6.1a( empty local-lis/les=59/60 n=0 ec=59/59 lis/c=59/59 les/c/f=60/60/0 sis=59) [4,5,1] r=0 lpr=59 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.1a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x562760ec9600) 2021-02-10T04:34:54.531 INFO:tasks.rados.rados.0.smithi085.stdout:1830: done (11 left) 2021-02-10T04:34:54.531 INFO:tasks.rados.rados.0.smithi085.stdout:1831: done (10 left) 2021-02-10T04:34:54.532 INFO:tasks.rados.rados.0.smithi085.stdout:1832: done (9 left) 2021-02-10T04:34:54.532 INFO:tasks.rados.rados.0.smithi085.stdout:1833: read oid 35 snap 184 2021-02-10T04:34:54.532 INFO:tasks.rados.rados.0.smithi085.stdout:1833: expect (ObjNum 578 snap 176 seq_num 578) 2021-02-10T04:34:54.532 INFO:tasks.rados.rados.0.smithi085.stdout:1834: delete oid 43 current snap is 197 2021-02-10T04:34:54.538 INFO:tasks.rados.rados.0.smithi085.stdout:1834: done (10 left) 2021-02-10T04:34:54.538 INFO:tasks.rados.rados.0.smithi085.stdout:1835: delete oid 50 current snap is 197 2021-02-10T04:34:54.543 INFO:tasks.rados.rados.0.smithi085.stdout:1835: done (10 left) 2021-02-10T04:34:54.543 INFO:tasks.rados.rados.0.smithi085.stdout:1836: rollback oid 48 current snap is 197 2021-02-10T04:34:54.543 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 48 to 184 2021-02-10T04:34:54.543 INFO:tasks.rados.rados.0.smithi085.stdout:1837: write oid 14 current snap is 197 2021-02-10T04:34:54.543 INFO:tasks.rados.rados.0.smithi085.stdout:1837: seq_num 651 ranges {541357=477104,1745307=594552} 2021-02-10T04:34:54.546 INFO:tasks.rados.rados.0.smithi085.stdout:1837: writing smithi08575397-14 from 541357 to 1018461 tid 1 2021-02-10T04:34:54.553 INFO:tasks.rados.rados.0.smithi085.stdout:1837: writing smithi08575397-14 from 1745307 to 2339859 tid 2 2021-02-10T04:34:54.553 INFO:tasks.rados.rados.0.smithi085.stdout:1838: read oid 8 snap -1 2021-02-10T04:34:54.554 INFO:tasks.rados.rados.0.smithi085.stdout:1838: expect (ObjNum 641 snap 196 seq_num 641) 2021-02-10T04:34:54.554 INFO:tasks.rados.rados.0.smithi085.stdout:1839: write oid 2 current snap is 197 2021-02-10T04:34:54.554 INFO:tasks.rados.rados.0.smithi085.stdout:1839: seq_num 652 ranges {558164=552353,1803789=454563,2855203=102961} 2021-02-10T04:34:54.560 INFO:tasks.rados.rados.0.smithi085.stdout:1839: writing smithi08575397-2 from 558164 to 1110517 tid 1 2021-02-10T04:34:54.566 INFO:tasks.rados.rados.0.smithi085.stdout:1839: writing smithi08575397-2 from 1803789 to 2258352 tid 2 2021-02-10T04:34:54.570 INFO:tasks.rados.rados.0.smithi085.stdout:1839: writing smithi08575397-2 from 2855203 to 2958164 tid 3 2021-02-10T04:34:54.570 INFO:tasks.rados.rados.0.smithi085.stdout:1840: delete oid 1 current snap is 197 2021-02-10T04:34:54.570 INFO:tasks.rados.rados.0.smithi085.stdout:1836: finishing rollback tid 0 to smithi08575397-48 2021-02-10T04:34:54.571 INFO:tasks.rados.rados.0.smithi085.stdout:1833: expect (ObjNum 578 snap 176 seq_num 578) 2021-02-10T04:34:54.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: cluster 2021-02-10T04:34:54.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:34:54.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: :34:53.085312+0000 osd.4 (osd.4) 657 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:54.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: cluster 2021-02-10T04:34: 2021-02-10T04:34:54.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: 53.085320+0000 osd.4 (osd.4) 658 : cluster 2021-02-10T04:34:54.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: cluster 2021-02-10T04:34:53.085326+0000 2021-02-10T04:34:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: osd.4 (osd.4) 659 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: 02-10T04:34:53.085332+0000 osd.4 2021-02-10T04:34:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: (osd.4) 660 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: -10T04:34:53.526874+0000 mon.a (mon.0 2021-02-10T04:34:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: ) 1286 : cluster [DBG] osdmap e476: 8 total, 8 up, 8 in 2021-02-10T04:34:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: 10T04:34:53.687798+0000 osd.2 ( 2021-02-10T04:34:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: osd.2) 1147 2021-02-10T04:34:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:54.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:54.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: 04:34:53.687805+0000 osd.2 (osd. 2021-02-10T04:34:54.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: 2) 1148 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:54.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:54.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: :34:53.687810+0000 osd.2 (osd 2021-02-10T04:34:54.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: .2) 1149 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: cluster 2021-02-10T04:34:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: 2021-02 2021-02-10T04:34:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: -10T04:34:53.687815+0000 osd.2 2021-02-10T04:34:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: (osd.2) 1150 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: cluster 2021-02-10T04:34:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:34:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: 34:53.687824+0000 osd.2 (osd.2) 2021-02-10T04:34:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: 1151 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: -10T04:34:53.687831+0000 2021-02-10T04:34:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: osd.2 (osd.2) 1152 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: cluster 2021-02-10T04:34:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:34:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: 34:53.687838+0000 osd.2 (osd.2) 1153 2021-02-10T04:34:54.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:54.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:54.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: 04:34:53.687845+0000 osd.2 (osd.2) 2021-02-10T04:34:54.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: 1154 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:54.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:54.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: 10T04:34:53.687851+0000 osd.2 (osd. 2021-02-10T04:34:54.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: 2) 1155 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:54.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: cluster 2021-02-10T04:34:54.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: 2021-02-10T04:34:53. 2021-02-10T04:34:54.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: 687856+0000 osd.2 (osd.2) 1156 : cluster 2021-02-10T04:34:54.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:54.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:54.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: :34:54.093106+0000 mgr.y (mgr.14140) 2021-02-10T04:34:54.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:54 smithi164 conmon[29288]: 573 : cluster [DBG] pgmap v864: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 386 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 646 KiB/s wr, 14 op/s 2021-02-10T04:34:54.923 INFO:tasks.rados.rados.0.smithi085.stdout:1836: finishing rollback tid 1 to smithi08575397-48 2021-02-10T04:34:54.923 INFO:tasks.rados.rados.0.smithi085.stdout:1836: finishing rollback tid 2 to smithi08575397-48 2021-02-10T04:34:54.923 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 786 (ObjNum 476 snap 135 seq_num 476) dirty exists 2021-02-10T04:34:54.924 INFO:tasks.rados.rados.0.smithi085.stdout:1837: finishing write tid 1 to smithi08575397-14 2021-02-10T04:34:54.924 INFO:tasks.rados.rados.0.smithi085.stdout:1837: finishing write tid 2 to smithi08575397-14 2021-02-10T04:34:54.924 INFO:tasks.rados.rados.0.smithi085.stdout:1837: finishing write tid 3 to smithi08575397-14 2021-02-10T04:34:54.924 INFO:tasks.rados.rados.0.smithi085.stdout:1837: finishing write tid 4 to smithi08575397-14 2021-02-10T04:34:54.924 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 789 (ObjNum 651 snap 197 seq_num 651) dirty exists 2021-02-10T04:34:54.924 INFO:tasks.rados.rados.0.smithi085.stdout:1837: left oid 14 (ObjNum 651 snap 197 seq_num 651) 2021-02-10T04:34:54.925 INFO:tasks.rados.rados.0.smithi085.stdout:1838: expect (ObjNum 641 snap 196 seq_num 641) 2021-02-10T04:34:54.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[49106]: debug 2021-02-10T04:34:54.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[49106]: 2021-02-10T04:34:54.675+0000 7f77a3e94700 -1 osd.2 477 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:54.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[49106]: 2021-02-10T04:34:54.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: cluster 2021-02-10T04:34:53.085312+0000 osd.4 (osd.4 2021-02-10T04:34:54.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: ) 657 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:54.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: cluster 2021-02-10T04:34:53.085320+0000 osd.4 (osd.4) 658 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:54.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: cluster 2021-02-10T04:34:53.085326+0000 osd.4 (osd. 2021-02-10T04:34:54.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: 4) 659 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:54.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:54.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: -02-10T04:34:53.085332+0000 2021-02-10T04:34:54.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: osd.4 (osd.4) 660 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:54.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: 2021-02-10T04:34:54.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: cluster 2021-02-10T04:34:54.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: 2021-02-10T04:34:53.526874+0000 mon.a (mon. 2021-02-10T04:34:54.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: 0) 1286 : cluster [DBG] osdmap e476: 8 total, 8 up, 8 in 2021-02-10T04:34:54.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:54.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: 02-10T04:34:53.687798+0000 osd.2 ( 2021-02-10T04:34:54.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: osd.2) 1147 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:54.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: cluster 2021-02-10T04:34:53.687805 2021-02-10T04:34:54.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1148 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:54.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: cluster 2021-02-10T04:34:54.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: 2021-02-10T04:34:54.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: 2021 2021-02-10T04:34:54.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: - 2021-02-10T04:34:54.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: 02 2021-02-10T04:34:54.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: - 2021-02-10T04:34:54.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: 10T04:34:53.687810+0000 osd.2 2021-02-10T04:34:54.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: (osd.2) 1149 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:54.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: cluster 2021-02-10T04:34:54.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: 2021-02-10T04:34:53.687815+0000 osd.2 (osd.2) 1150 : cluster 2021-02-10T04:34:54.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:54.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:54.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: -10T04:34:53.687824 2021-02-10T04:34:54.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1151 2021-02-10T04:34:54.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:54.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: cluster 2021-02-10T04:34:54.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: 2021-02-10T04:34 2021-02-10T04:34:54.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: :53.687831+0000 osd.2 ( 2021-02-10T04:34:54.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: osd.2) 1152 : cluster [WRN] 2021-02-10T04:34:54.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:54.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:54.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: 02-10T04:34:53 2021-02-10T04:34:54.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: .687838+0000 osd.2 (osd. 2021-02-10T04:34:54.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: 2) 1153 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:54.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:54.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: -02-10T04:34:53.687845+0000 2021-02-10T04:34:54.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: osd.2 (osd.2) 1154 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:54.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: 2021-02-10T04:34:54.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:54.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: 02-10T04:34:53.687851+0000 osd.2 2021-02-10T04:34:54.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: (osd.2) 1155 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:54.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:54.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: -10T04:34:53.687856+0000 osd.2 2021-02-10T04:34:54.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: (osd.2) 1156 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:54.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:54.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: 02-10T04:34:54.093106+0000 2021-02-10T04:34:54.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: mgr.y (mgr.14140) 573 : cluster [DBG] 2021-02-10T04:34:54.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[29319]: pgmap v864: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 386 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 646 KiB/s wr, 14 op/s 2021-02-10T04:34:54.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: cluster 2021-02-10T04:34:54.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: 2021-02-10T04:34 2021-02-10T04:34:54.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: :53.085312+0000 osd.4 (osd.4) 657 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:54.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: cluster 2021-02-10T04:34 2021-02-10T04:34:54.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: :53.085320+0000 osd.4 (osd.4) 658 : 2021-02-10T04:34:54.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:54.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:54.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: 02-10T04:34: 2021-02-10T04:34:54.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: 53.085326+0000 osd.4 (osd.4) 2021-02-10T04:34:54.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: 659 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:54.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: cluster 2021-02-10T04:34:54.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:34:54.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: :34:53.085332+0000 osd.4 ( 2021-02-10T04:34:54.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: osd.4) 660 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:54.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: cluster 2021-02-10T04:34:54.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:34:54.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: 04:34:53.526874+0000 mon.a (mon 2021-02-10T04:34:54.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: .0) 1286 : cluster [DBG] osdmap e476: 8 total, 8 up, 8 in 2021-02-10T04:34:54.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: cluster 2021-02-10T04:34:54.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: 2021-02-10T04:34: 2021-02-10T04:34:54.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: 53.687798+0000 osd.2 (osd.2) 1147 : 2021-02-10T04:34:54.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:54.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:54.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: 10T04:34:53.687805 2021-02-10T04:34:54.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 2021-02-10T04:34:54.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: 1148 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:54.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:54.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: -10T04:34:53.687810 2021-02-10T04:34:54.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 2021-02-10T04:34:54.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: 1149 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:54.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: cluster 2021-02-10T04:34:54.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: 2021-02-10T04:34:53.687815+0000 2021-02-10T04:34:54.958 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: osd.2 (osd.2) 1150 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:54.958 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: 2021-02-10T04:34:54.958 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: cluster 2021-02-10T04:34:54.958 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: 2021- 2021-02-10T04:34:54.958 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: 02-10T04:34:53.687824+0000 2021-02-10T04:34:54.958 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: osd.2 (osd.2) 1151 : cluster [WRN] 2021-02-10T04:34:54.959 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:54.959 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:54.959 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: -10T04:34:53.687831+0000 osd.2 (osd.2) 2021-02-10T04:34:54.959 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: 1152 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:54.959 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:54.959 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: 10T04:34:53.687838+0000 osd.2 (osd.2) 2021-02-10T04:34:54.960 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: 1153 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:54.960 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:54.960 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: 02-10T04:34:53.687845+0000 osd.2 (osd 2021-02-10T04:34:54.960 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: .2) 1154 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:54.960 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:54.961 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: -10T04:34:53.687851+0000 osd.2 (osd.2) 1155 2021-02-10T04:34:54.961 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:54.961 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:54.961 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: -10T04:34:53.687856+0000 osd.2 (osd.2 2021-02-10T04:34:54.961 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: ) 1156 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:54.961 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:54.961 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: -10T04:34:54.093106+0000 mgr.y (mgr.14140 2021-02-10T04:34:54.961 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:54 smithi085 conmon[35961]: ) 573 : cluster [DBG] pgmap v864: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 386 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 646 KiB/s wr, 14 op/s 2021-02-10T04:34:55.095 INFO:tasks.rados.rados.0.smithi085.stdout:1839: finishing write tid 1 to smithi08575397-2 2021-02-10T04:34:55.096 INFO:tasks.rados.rados.0.smithi085.stdout:1839: finishing write tid 2 to smithi08575397-2 2021-02-10T04:34:55.096 INFO:tasks.rados.rados.0.smithi085.stdout:1839: finishing write tid 3 to smithi08575397-2 2021-02-10T04:34:55.096 INFO:tasks.rados.rados.0.smithi085.stdout:1839: finishing write tid 4 to smithi08575397-2 2021-02-10T04:34:55.096 INFO:tasks.rados.rados.0.smithi085.stdout:1839: finishing write tid 5 to smithi08575397-2 2021-02-10T04:34:55.096 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 658 (ObjNum 652 snap 197 seq_num 652) dirty exists 2021-02-10T04:34:55.096 INFO:tasks.rados.rados.0.smithi085.stdout:1839: left oid 2 (ObjNum 652 snap 197 seq_num 652) 2021-02-10T04:34:55.096 INFO:tasks.rados.rados.0.smithi085.stdout:1833: done (14 left) 2021-02-10T04:34:55.096 INFO:tasks.rados.rados.0.smithi085.stdout:1836: done (13 left) 2021-02-10T04:34:55.097 INFO:tasks.rados.rados.0.smithi085.stdout:1837: done (12 left) 2021-02-10T04:34:55.097 INFO:tasks.rados.rados.0.smithi085.stdout:1838: done (11 left) 2021-02-10T04:34:55.097 INFO:tasks.rados.rados.0.smithi085.stdout:1839: done (10 left) 2021-02-10T04:34:55.097 INFO:tasks.rados.rados.0.smithi085.stdout:1840: done (9 left) 2021-02-10T04:34:55.097 INFO:tasks.rados.rados.0.smithi085.stdout:1841: write oid 30 current snap is 197 2021-02-10T04:34:55.097 INFO:tasks.rados.rados.0.smithi085.stdout:1841: seq_num 654 ranges {591778=419204,1637106=741879,2910049=81729} 2021-02-10T04:34:55.101 INFO:tasks.rados.rados.0.smithi085.stdout:1841: writing smithi08575397-30 from 591778 to 1010982 tid 1 2021-02-10T04:34:55.110 INFO:tasks.rados.rados.0.smithi085.stdout:1841: writing smithi08575397-30 from 1637106 to 2378985 tid 2 2021-02-10T04:34:55.114 INFO:tasks.rados.rados.0.smithi085.stdout:1841: writing smithi08575397-30 from 2910049 to 2991778 tid 3 2021-02-10T04:34:55.114 INFO:tasks.rados.rados.0.smithi085.stdout:1842: delete oid 31 current snap is 197 2021-02-10T04:34:55.114 INFO:tasks.rados.rados.0.smithi085.stdout:1841: finishing write tid 1 to smithi08575397-30 2021-02-10T04:34:55.114 INFO:tasks.rados.rados.0.smithi085.stdout:1841: finishing write tid 2 to smithi08575397-30 2021-02-10T04:34:55.117 INFO:tasks.rados.rados.0.smithi085.stdout:1841: finishing write tid 3 to smithi08575397-30 2021-02-10T04:34:55.117 INFO:tasks.rados.rados.0.smithi085.stdout:1842: done (10 left) 2021-02-10T04:34:55.117 INFO:tasks.rados.rados.0.smithi085.stdout:1843: snap_create 2021-02-10T04:34:55.117 INFO:tasks.rados.rados.0.smithi085.stdout:1841: finishing write tid 4 to smithi08575397-30 2021-02-10T04:34:55.118 INFO:tasks.rados.rados.0.smithi085.stdout:1841: finishing write tid 5 to smithi08575397-30 2021-02-10T04:34:55.118 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 769 (ObjNum 654 snap 197 seq_num 654) dirty exists 2021-02-10T04:34:55.118 INFO:tasks.rados.rados.0.smithi085.stdout:1841: left oid 30 (ObjNum 654 snap 197 seq_num 654) 2021-02-10T04:34:55.542 INFO:tasks.rados.rados.0.smithi085.stdout:1841: done (10 left) 2021-02-10T04:34:55.542 INFO:tasks.rados.rados.0.smithi085.stdout:1843: done (9 left) 2021-02-10T04:34:55.542 INFO:tasks.rados.rados.0.smithi085.stdout:1844: write (excl) oid 48 current snap is 198 2021-02-10T04:34:55.542 INFO:tasks.rados.rados.0.smithi085.stdout:1844: seq_num 656 ranges {625392=569702,1637717=745548,2848542=176850} 2021-02-10T04:34:55.546 INFO:tasks.rados.rados.0.smithi085.stdout:1844: writing smithi08575397-48 from 625392 to 1195094 tid 1 2021-02-10T04:34:55.555 INFO:tasks.rados.rados.0.smithi085.stdout:1844: writing smithi08575397-48 from 1637717 to 2383265 tid 2 2021-02-10T04:34:55.559 INFO:tasks.rados.rados.0.smithi085.stdout:1844: writing smithi08575397-48 from 2848542 to 3025392 tid 3 2021-02-10T04:34:55.560 INFO:tasks.rados.rados.0.smithi085.stdout:1844: finishing write tid 1 to smithi08575397-48 2021-02-10T04:34:55.560 INFO:tasks.rados.rados.0.smithi085.stdout:1845: read oid 20 snap -1 2021-02-10T04:34:55.560 INFO:tasks.rados.rados.0.smithi085.stdout:1845: expect deleted 2021-02-10T04:34:55.560 INFO:tasks.rados.rados.0.smithi085.stdout:1846: copy_from oid 5 from oid 35 current snap is 198 2021-02-10T04:34:55.560 INFO:tasks.rados.rados.0.smithi085.stdout:1847: snap_create 2021-02-10T04:34:55.561 INFO:tasks.rados.rados.0.smithi085.stdout:1844: finishing write tid 2 to smithi08575397-48 2021-02-10T04:34:55.562 INFO:tasks.rados.rados.0.smithi085.stdout:1844: finishing write tid 3 to smithi08575397-48 2021-02-10T04:34:55.562 INFO:tasks.rados.rados.0.smithi085.stdout:1844: finishing write tid 4 to smithi08575397-48 2021-02-10T04:34:55.566 INFO:tasks.rados.rados.0.smithi085.stdout:1844: finishing write tid 5 to smithi08575397-48 2021-02-10T04:34:55.566 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 793 (ObjNum 656 snap 198 seq_num 656) dirty exists 2021-02-10T04:34:55.566 INFO:tasks.rados.rados.0.smithi085.stdout:1844: left oid 48 (ObjNum 656 snap 198 seq_num 656) 2021-02-10T04:34:55.743 INFO:tasks.rados.rados.0.smithi085.stdout:1846: finishing copy_from to smithi08575397-5 2021-02-10T04:34:55.743 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 650 (ObjNum 85 snap 11 seq_num 85) dirty exists 2021-02-10T04:34:55.743 INFO:tasks.rados.rados.0.smithi085.stdout:1846: finishing copy_from racing read to smithi08575397-5 2021-02-10T04:34:55.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: cluster 2021-02-10T04:34:54.135465 2021-02-10T04:34:55.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 661 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:55.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: cluster 2021-02-10T04:34:54.135474+0000 osd.4 ( 2021-02-10T04:34:55.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: osd.4) 662 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:55.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: cluster 2021-02-10T04:34:55.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: 2021-02-10T04:34:54.135482 2021-02-10T04:34:55.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 663 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:55.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: cluster 2021-02-10T04:34:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: 2021-02-10T04:34 2021-02-10T04:34:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: :54.135487+0000 osd.4 (osd.4 2021-02-10T04:34:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: ) 664 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: -10T04:34:54.530891+0000 mon.a (mon.0) 2021-02-10T04:34:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: 1287 : cluster [DBG] osdmap e477: 8 total, 8 up, 8 in 2021-02-10T04:34:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: -02-10T04:34:54. 2021-02-10T04:34:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: 677063+0000 osd.2 (osd.2 2021-02-10T04:34:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: ) 1157 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: 02-10T04:34:54.677071+0000 osd.2 (osd. 2021-02-10T04:34:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: 2) 1158 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: 02-10T04:34:54.677077+0000 osd.2 ( 2021-02-10T04:34:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: osd.2) 1159 : cluster [WRN] 2021-02-10T04:34:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: cluster 2021-02-10T04:34:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: 2021-02-10T04:34: 2021-02-10T04:34:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: 54.677083+0000 osd.2 (osd.2) 1160 : cluster 2021-02-10T04:34:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: cluster 2021-02-10T04:34:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:34:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: 04:34:54.677089+0000 osd.2 (osd.2) 2021-02-10T04:34:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: 1161 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: cluster 2021-02-10T04:34:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: 2021-02- 2021-02-10T04:34:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: 10T04:34:54.677094+0000 osd.2 (osd 2021-02-10T04:34:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: .2) 1162 : cluster [WRN] 2021-02-10T04:34:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: 02-10T04:34:54.677100 2021-02-10T04:34:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1163 : cluster 2021-02-10T04:34:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: cluster 2021-02-10T04:34:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: 2021-02-10T04:34:54 2021-02-10T04:34:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: .677106+0000 osd.2 (osd.2) 2021-02-10T04:34:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: 1164 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:55.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: cluster 2021-02-10T04:34:55.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: 2021-02-10T04:34: 2021-02-10T04:34:55.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: 54.677111+0000 osd.2 (osd.2 2021-02-10T04:34:55.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: ) 1165 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:55.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: cluster 2021-02-10T04:34:55.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: 2021-02-10T04:34 2021-02-10T04:34:55.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: :54.677116+0000 osd.2 (osd. 2021-02-10T04:34:55.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: 2) 1166 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:55.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: cephadm 2021 2021-02-10T04:34:55.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: -02-10T04:34: 2021-02-10T04:34:55.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: 55.124467+0000 mgr.y (mgr. 2021-02-10T04:34:55.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: 14140) 574 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:34:55.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: 2021-02-10T04:34:55.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: cephadm 2021 2021-02-10T04:34:55.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: -02-10T04:34: 2021-02-10T04:34:55.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: 55.124991+0000 mgr.y (mgr. 2021-02-10T04:34:55.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: 14140) 575 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:34:55.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:55 smithi164 conmon[29288]: 2021-02-10T04:34:55.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[49106]: debug 2021-02-10T04:34:55.711+0000 7f77a3e94700 -1 osd.2 477 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:55.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[29319]: cluster 2021-02-10T04:34:54.135465+0000 osd.4 (osd.4) 661 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:55.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[29319]: cluster 2021-02-10T04:34:54.135474+0000 osd.4 (osd.4) 662 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:55.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[29319]: cluster 2021-02-10T04:34:54.135482+0000 osd.4 (osd.4) 663 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:55.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[29319]: cluster 2021-02-10T04:34:54.135487+0000 osd.4 (osd.4) 664 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:55.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[29319]: cluster 2021-02-10T04:34:54.530891+0000 mon.a (mon.0) 1287 : cluster [DBG] osdmap e477: 8 total, 8 up, 8 in 2021-02-10T04:34:55.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[29319]: cluster 2021-02-10T04:34:54.677063+0000 osd.2 (osd.2) 1157 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:55.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[29319]: cluster 2021-02-10T04:34:54.677071+0000 osd.2 (osd.2) 1158 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:55.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[29319]: cluster 2021-02-10T04:34:54.677077+0000 osd.2 (osd.2) 1159 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:55.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[29319]: cluster 2021-02-10T04:34:54.677083+0000 osd.2 (osd.2) 1160 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:55.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[29319]: cluster 2021-02-10T04:34:54.677089+0000 osd.2 (osd.2) 1161 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:55.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[29319]: cluster 2021-02-10T04:34:54.677094+0000 osd.2 (osd.2) 1162 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:55.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[29319]: cluster 2021-02-10T04:34:54.677100+0000 osd.2 (osd.2) 1163 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:55.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[29319]: cluster 2021-02-10T04:34:54.677106+0000 osd.2 (osd.2) 1164 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:55.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:55.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[29319]: -02-10T04:34:54.677111+0000 osd.2 (osd.2) 1165 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:55.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[29319]: cluster 2021-02-10T04:34:54.677116+0000 osd.2 (osd.2) 1166 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:55.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[29319]: cephadm 2021-02-10T04:34:55.124467+0000 mgr.y (mgr.14140) 574 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:34:55.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[29319]: cephadm 2021-02-10T04:34:55.124991+0000 mgr.y ( 2021-02-10T04:34:55.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[29319]: mgr.14140) 575 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:34:55.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: cluster 2021-02-10T04:34:54.135465+0000 2021-02-10T04:34:55.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: osd.4 (osd.4) 661 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:55.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: cluster 2021-02-10T04:34:55.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: 2021-02-10T04:34:54.135474+0000 osd.4 (osd.4) 662 : cluster 2021-02-10T04:34:55.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:55.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:34:55.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: 04:34:54.135482+0000 osd.4 (osd. 2021-02-10T04:34:55.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: 4) 663 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:55.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:55.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: 02-10T04:34:54.135487+0000 osd.4 2021-02-10T04:34:55.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: (osd.4) 664 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:55.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: 2021-02-10T04:34:55.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: cluster 2021-02-10T04:34:55.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: 2021-02-10T04:34:55.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:34:55.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: 04:34:54.530891+0000 mon.a (mon.0) 1287 : 2021-02-10T04:34:55.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: cluster [DBG] osdmap e477: 8 total, 8 up, 8 in 2021-02-10T04:34:55.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:55.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: -10T04:34:54.677063+0000 osd.2 (osd.2) 2021-02-10T04:34:55.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: 1157 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:55.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:55.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: -10T04:34:54.677071+0000 osd.2 (osd.2 2021-02-10T04:34:55.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: ) 1158 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:55.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:55.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: -10T04:34:54.677077+0000 osd.2 (osd.2 2021-02-10T04:34:55.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: ) 1159 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:55.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:55.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: -10T04:34:54.677083+0000 osd.2 (osd. 2021-02-10T04:34:55.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: 2) 1160 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:55.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:55.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: -10T04:34:54.677089+0000 osd.2 (osd.2 2021-02-10T04:34:55.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: ) 1161 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:55.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:55.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: -10T04:34:54.677094+0000 osd.2 (osd.2 2021-02-10T04:34:55.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: ) 1162 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:55.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:55.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: -10T04:34:54.677100+0000 osd.2 (osd.2) 2021-02-10T04:34:55.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: 1163 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:55.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:55.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: -10T04:34:54.677106+0000 osd.2 (osd.2 2021-02-10T04:34:55.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: ) 1164 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:55.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:55.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: -10T04:34:54.677111+0000 osd.2 (osd.2) 2021-02-10T04:34:55.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: 1165 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:55.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:55.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: -10T04:34:54.677116+0000 osd.2 (osd.2 2021-02-10T04:34:55.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: ) 1166 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:55.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: cephadm 2021-02 2021-02-10T04:34:55.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: -10T04:34:55.124467+0000 mgr.y (mgr.14140 2021-02-10T04:34:55.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: ) 574 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:34:55.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: cephadm 2021-02-10T04:34:55.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: 2021-02-10T04:34:55.124991 2021-02-10T04:34:55.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 575 : cephadm 2021-02-10T04:34:55.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:55 smithi085 conmon[35961]: [INF] Refreshing smithi164 facts 2021-02-10T04:34:56.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[33677]: debug 2021-02-10T04:34:56.089+0000 7fbec19a1700 -1 osd.4 478 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:56.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[33677]: debug 2021-02-10T04:34:56.090+0000 7fbea992c700 -1 osd.4 pg_epoch: 478 pg[6.1a( empty local-lis/les=59/60 n=0 ec=59/59 lis/c=59/59 les/c/f=60/60/0 sis=59) [4,5,1] r=0 lpr=59 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.1a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x56275d82f4a0) 2021-02-10T04:34:56.549 INFO:tasks.rados.rados.0.smithi085.stdout:1844: done (12 left) 2021-02-10T04:34:56.549 INFO:tasks.rados.rados.0.smithi085.stdout:1845: done (11 left) 2021-02-10T04:34:56.549 INFO:tasks.rados.rados.0.smithi085.stdout:1846: done (10 left) 2021-02-10T04:34:56.549 INFO:tasks.rados.rados.0.smithi085.stdout:1847: done (9 left) 2021-02-10T04:34:56.549 INFO:tasks.rados.rados.0.smithi085.stdout:1848: snap_create 2021-02-10T04:34:56.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: cluster 2021-02-10T04:34:56.988 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: 2021-02-10T04:34:56.989 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:34:56.989 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: 34:55.537941+0000 mon.a (mon.0) 1288 : cluster [DBG] osdmap e478: 8 total, 8 up, 8 in 2021-02-10T04:34:56.989 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:56.990 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: 02-10T04:34:55.712948+0000 osd.2 (osd.2 2021-02-10T04:34:56.990 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: ) 1167 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:56.991 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:56.991 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: T04:34:55.712954+0000 osd.2 (osd. 2021-02-10T04:34:56.991 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: 2) 1168 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:56.991 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:56.992 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: T04:34:55.712958+0000 osd.2 (osd 2021-02-10T04:34:56.992 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: .2) 1169 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:56.992 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:56.992 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: T04:34:55.712962+0000 osd.2 (osd 2021-02-10T04:34:56.993 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: .2) 1170 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:56.993 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:56.993 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: -10T04:34:55.712966+0000 osd.2 ( 2021-02-10T04:34:56.994 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: osd.2) 1171 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:56.994 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:56.994 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: 34:55.712973+0000 osd.2 (osd.2) 1172 2021-02-10T04:34:56.995 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:56.995 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:56.995 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: 34:55.712976+0000 osd.2 (osd.2) 1173 : cluster 2021-02-10T04:34:56.995 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:56.995 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:56.996 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: 04:34:55.712979+0000 osd.2 (osd.2) 2021-02-10T04:34:56.996 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: 1174 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:56.997 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:56.997 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: T04:34:55.712983+0000 osd.2 (osd 2021-02-10T04:34:56.998 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: .2) 1175 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:56.998 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:57.000 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: -10T04:34:55.712986+0000 osd.2 2021-02-10T04:34:57.000 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: (osd.2) 1176 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:57.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:34:57.002 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: 02-10T04:34:56.024731+0000 mon.a 2021-02-10T04:34:57.003 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: (mon.0) 1289 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:34:57.004 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:57.004 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: 02-10T04:34:56.093641+0000 mgr.y ( 2021-02-10T04:34:57.005 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:56 smithi164 conmon[29288]: mgr.14140) 576 : cluster [DBG] pgmap v867: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 397 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 3.2 MiB/s wr, 30 op/s 2021-02-10T04:34:57.009 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[49106]: debug 2021-02-10T04:34:57.009 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[49106]: 2021-02-10T04:34:56.685+0000 7f77a3e94700 -1 osd.2 478 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:57.009 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: cluster 2021-02-10T04:34:55.537941 2021-02-10T04:34:57.009 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1288 : cluster [DBG] osdmap e478: 8 total, 8 up, 8 in 2021-02-10T04:34:57.010 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:57.010 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: T04:34:55.712948+0000 osd.2 (osd.2) 1167 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:57.011 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: 2021-02-10T04:34:57.011 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:57.011 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: -02-10T04:34:55.712954+0000 osd.2 (osd.2) 2021-02-10T04:34:57.011 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: 1168 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:57.012 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: cluster 2021-02-10T04:34:55. 2021-02-10T04:34:57.012 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: 712958+0000 osd.2 (osd.2) 1169 : cluster [WRN] 2021-02-10T04:34:57.013 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:57.013 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: cluster 2021-02-10T04:34:57.013 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: 2021-02-10T04:34:55.712962+0000 osd.2 (osd. 2021-02-10T04:34:57.014 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: 2) 1170 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:57.014 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:57.015 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: -10T04:34:55.712966+0000 osd.2 ( 2021-02-10T04:34:57.015 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: osd.2) 1171 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:57.016 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:57.016 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: 02-10T04:34:55.712973+0000 2021-02-10T04:34:57.017 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: osd.2 (osd.2) 1172 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:57.017 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:57.017 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: -02-10T04:34:55.712976+0000 osd.2 2021-02-10T04:34:57.018 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: (osd.2) 1173 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:57.018 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:57.019 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: -02-10T04:34:55.712979 2021-02-10T04:34:57.019 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1174 : cluster [WRN] 2021-02-10T04:34:57.020 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:57.021 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: cluster 2021-02-10T04:34:57.021 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: 2021-02-10T04:34:55.712983+0000 osd.2 (osd. 2021-02-10T04:34:57.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: 2) 1175 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:57.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:57.023 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: 10T04:34:55.712986+0000 osd.2 (osd 2021-02-10T04:34:57.024 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: .2) 1176 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:57.024 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:34:57.024 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: 10T04:34:56.024731+0000 mon.a (mon. 2021-02-10T04:34:57.024 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: 0) 1289 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:34:57.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:34:57.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: 10T04:34:56.093641+0000 mgr.y ( 2021-02-10T04:34:57.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[35961]: mgr.14140) 576 : cluster [DBG] pgmap v867: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 397 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 3.2 MiB/s wr, 30 op/s 2021-02-10T04:34:57.025 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: cluster 2021-02-10T04:34:57.025 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:34:57.025 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: T04:34:55.537941+0000 mon.a (mon.0) 1288 : cluster [DBG] osdmap e478: 8 total, 8 up, 8 in 2021-02-10T04:34:57.026 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: cluster 2021-02-10T04:34:57.026 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:34:57.026 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: 10T04:34:55.712948+0000 osd.2 (osd.2) 1167 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:57.026 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: cluster 2021-02-10T04:34:57.026 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: 2021- 2021-02-10T04:34:57.026 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: 02-10T04:34:55.712954 2021-02-10T04:34:57.026 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1168 : cluster 2021-02-10T04:34:57.027 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:57.027 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:57.027 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: 02-10T04:34:55.712958+0000 osd.2 (osd.2) 1169 : cluster 2021-02-10T04:34:57.027 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:57.027 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: cluster 2021-02-10T04:34:57.027 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: 2021-02-10T04:34:55.712962+0000 osd.2 (osd.2) 1170 : cluster [WRN] 2021-02-10T04:34:57.027 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:57.028 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:57.028 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: 02-10T04:34:55.712966+0000 osd.2 2021-02-10T04:34:57.028 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: (osd.2) 1171 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:57.028 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: cluster 2021-02-10T04:34:57.028 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: 2021-02-10T04:34:57.028 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: 2021 2021-02-10T04:34:57.028 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: -02-10T04: 2021-02-10T04:34:57.028 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: 34:55.712973+0000 osd.2 (osd.2) 1172 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:57.029 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: cluster 2021-02-10T04:34:57.029 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: 2021-02-10T04:34: 2021-02-10T04:34:57.029 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: 55.712976+0000 osd.2 (osd.2) 1173 2021-02-10T04:34:57.029 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:57.029 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:57.029 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: -02-10T04:34:55.712979+0000 2021-02-10T04:34:57.030 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: osd.2 (osd.2) 1174 : cluster [WRN] 2021-02-10T04:34:57.030 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:57.030 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:34:57.032 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: -02-10T04:34:55.712983+0000 2021-02-10T04:34:57.032 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: osd.2 (osd.2) 1175 : cluster [WRN] 2021-02-10T04:34:57.032 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:57.033 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: cluster 2021-02-10T04:34:55.712986+0000 osd.2 (osd.2) 2021-02-10T04:34:57.033 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: 1176 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:57.033 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: audit 2021-02-10T04:34:57.033 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: 2021-02-10T04:34 2021-02-10T04:34:57.033 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: :56.024731+0000 mon.a (mon.0) 2021-02-10T04:34:57.034 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: 1289 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:34:57.034 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: cluster 2021-02-10T04:34:57.034 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: 2021-02-10T04:34 2021-02-10T04:34:57.034 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: :56.093641+0000 mgr.y (mgr.14140) 2021-02-10T04:34:57.035 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:56 smithi085 conmon[29319]: 576 : cluster [DBG] pgmap v867: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 397 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 3.2 MiB/s wr, 30 op/s 2021-02-10T04:34:57.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[33677]: debug 2021-02-10T04:34:57.130+0000 7fbec19a1700 -1 osd.4 479 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:57.441 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[52836]: debug 2021-02-10T04:34:57.441 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[52836]: 2021-02-10T04:34:57.048+0000 7fba71fb2700 -1 osd.3 pg_epoch: 479 pg[4.10( v 62'12 (0'0,62'12] local-lis/les=71/72 n=10 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,5,6] r=0 lpr=71 crt=62'12 lcod 62'11 mlcod 0'0 active+clean] scrubber pg(4.10) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6 (0x55fec06fec60) 2021-02-10T04:34:57.441 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[52836]: 2021-02-10T04:34:57.548 INFO:tasks.rados.rados.0.smithi085.stdout:1848: done (9 left) 2021-02-10T04:34:57.548 INFO:tasks.rados.rados.0.smithi085.stdout:1849: snap_remove snap 175 2021-02-10T04:34:57.558 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: cluster 2021-02-10T04:34:56.091270+0000 osd.4 (osd 2021-02-10T04:34:57.558 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: .4) 665 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:57.558 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: cluster 2021-02-10T04:34:56.091279+0000 osd.4 (osd.4) 666 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:57.558 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: cluster 2021-02-10T04:34:56.091285 2021-02-10T04:34:57.558 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 667 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:57.558 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: 2021-02-10T04:34:57.558 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: cluster 2021-02-10T04:34:57.558 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: 2021-02-10T04:34:56.091291+0000 osd.4 (osd 2021-02-10T04:34:57.558 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: .4) 668 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:57.558 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:57.559 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: 02-10T04:34:56.546521+0000 mon.a 2021-02-10T04:34:57.559 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: (mon.0) 1290 : cluster [DBG] osdmap e479: 8 total, 8 up, 8 in 2021-02-10T04:34:57.559 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: cluster 2021-02-10T04:34:57.559 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: 2021-02-10T04:34:56.687178+0000 osd.2 2021-02-10T04:34:57.559 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: (osd.2) 1177 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:57.559 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:57.559 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: 02-10T04:34:56.687205+0000 osd.2 2021-02-10T04:34:57.559 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: (osd.2) 1178 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:57.559 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:57.559 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: 10T04:34:56.687213+0000 osd.2 (osd.2) 2021-02-10T04:34:57.559 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: 1179 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:57.559 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: cluster 2021-02-10T04:34:57.559 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: 2021-02-10T04:34:56.687222 2021-02-10T04:34:57.559 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1180 : cluster [WRN] 2021-02-10T04:34:57.560 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:57.560 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: cluster 2021-02-10T04:34:57.560 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: 2021-02-10T04:34 2021-02-10T04:34:57.560 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: :56.687229+0000 osd.2 (osd.2) 1181 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:57.560 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:57.560 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: -10T04:34:56.687235+0000 osd.2 (osd. 2021-02-10T04:34:57.560 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: 2) 1182 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:57.560 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:57.560 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: 02-10T04:34:56.687242+0000 osd.2 (osd. 2021-02-10T04:34:57.560 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: 2) 1183 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:57.560 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:57.560 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: 10T04:34:56.687248+0000 osd.2 (osd 2021-02-10T04:34:57.560 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: .2) 1184 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:57.561 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: cluster 2021-02-10T04:34: 2021-02-10T04:34:57.561 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: 56.687254+0000 osd.2 (osd.2) 1185 : cluster [WRN] 2021-02-10T04:34:57.561 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:57.561 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: cluster 2021-02-10T04:34:57.561 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: 2021-02-10T04:34:56.687276+0000 osd.2 (osd 2021-02-10T04:34:57.561 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[29319]: .2) 1186 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: cluster 2021-02-10T04:34:56.091270+0000 2021-02-10T04:34:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: osd.4 (osd.4) 665 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: cluster 2021-02-10T04:34:56.091279+0000 osd.4 (osd.4) 666 : 2021-02-10T04:34:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: 34:56.091285+0000 osd.4 (osd.4) 667 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: cluster 2021-02-10T04:34:56.091291+0000 osd.4 (osd 2021-02-10T04:34:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: .4) 668 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:34:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: :34:56.546521+0000 mon.a (mon.0) 2021-02-10T04:34:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: 1290 : cluster [DBG] osdmap e479: 8 total, 8 up, 8 in 2021-02-10T04:34:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: T04:34:56.687178+0000 osd.2 (osd.2) 1177 : cluster 2021-02-10T04:34:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: T04:34:56.687205+0000 2021-02-10T04:34:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: osd.2 (osd.2) 1178 : cluster 2021-02-10T04:34:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:34:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: 02-10T04:34:56 2021-02-10T04:34:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: .687213+0000 osd.2 (osd.2) 1179 : cluster 2021-02-10T04:34:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:34:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: 10T04:34:56.687222 2021-02-10T04:34:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1180 : cluster 2021-02-10T04:34:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: -10T04:34:56.687229 2021-02-10T04:34:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1181 : cluster 2021-02-10T04:34:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: T04:34:56.687235+0000 osd.2 ( 2021-02-10T04:34:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: osd.2) 1182 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:34:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: -02-10T04:34:56. 2021-02-10T04:34:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: 687242+0000 osd.2 (osd.2) 1183 : cluster 2021-02-10T04:34:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:34:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: T04:34:56.687248+0000 osd.2 ( 2021-02-10T04:34:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: osd.2) 1184 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: cluster 2021-02-10T04:34:56.687254+0000 2021-02-10T04:34:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: osd.2 (osd.2) 1185 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:57.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: cluster 2021-02-10T04:34:56.687276+0000 osd.2 2021-02-10T04:34:57.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:57 smithi164 conmon[29288]: (osd.2) 1186 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:57.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[49106]: debug 2021-02-10T04:34:57.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[49106]: 2021-02-10T04:34:57.681+0000 7f77a3e94700 -1 osd.2 479 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:57.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[49106]: 2021-02-10T04:34:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: cluster 2021-02-10T04:34:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: 2021-02-10T04:34:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: 2021-02-10T04:34:56.091270+0000 2021-02-10T04:34:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: osd.4 (osd.4) 665 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: cluster 2021-02-10T04:34:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:34:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: T04:34:56.091279 2021-02-10T04:34:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 666 2021-02-10T04:34:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: -10T04:34:56.091285+0000 osd.4 2021-02-10T04:34:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: (osd.4) 667 : cluster 2021-02-10T04:34:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: -10T04: 2021-02-10T04:34:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: 34:56.091291+0000 osd.4 (osd.4) 668 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: -10T04:34:56.546521+0000 mon.a (mon.0) 1290 : 2021-02-10T04:34:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: cluster [DBG] osdmap e479: 8 total, 8 up, 8 in 2021-02-10T04:34:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: cluster 2021-02-10T04:34:56 2021-02-10T04:34:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: .687178+0000 osd.2 (osd.2) 1177 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:57.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:57.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: 02-10T04:34:56.687205+0000 osd.2 (osd 2021-02-10T04:34:57.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: .2) 1178 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:57.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: cluster 2021-02-10T04:34: 2021-02-10T04:34:57.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: 56.687213+0000 osd.2 (osd.2) 1179 : cluster [WRN] 2021-02-10T04:34:57.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:57.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:57.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: -10T04:34:56. 2021-02-10T04:34:57.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: 687222+0000 osd.2 (osd.2) 2021-02-10T04:34:57.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: 1180 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:57.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:34:57.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: 04:34:56.687229+0000 osd.2 (osd.2) 1181 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:57.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:57.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: T04:34:56.687235+0000 osd.2 (osd 2021-02-10T04:34:57.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: .2) 1182 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:57.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:34:57.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: 04:34:56.687242+0000 osd.2 (osd 2021-02-10T04:34:57.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: .2) 1183 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:57.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:57.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: T04:34:56.687248+0000 osd.2 (osd 2021-02-10T04:34:57.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: .2) 1184 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:57.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:57.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: -10T04:34:56.687254+0000 osd.2 ( 2021-02-10T04:34:57.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: osd.2) 1185 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:57.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:57.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: 02-10T04:34:56.687276 2021-02-10T04:34:57.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1186 : cluster [WRN] 2021-02-10T04:34:57.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:57 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:58.159 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:34:58.189 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:34:58.393 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:34:58.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[44544]: debug 2021-02-10T04:34:58.275+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:34:58.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[33677]: debug 2021-02-10T04:34:58.081+0000 7fbec19a1700 -1 osd.4 480 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:58.441 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[52836]: debug 2021-02-10T04:34:58.098+0000 7fba71fb2700 -1 osd.3 pg_epoch: 480 pg[4.10( v 62'12 (0'0,62'12] local-lis/les=71/72 n=10 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,5,6] r=0 lpr=71 crt=62'12 lcod 62'11 mlcod 0'0 active+clean] scrubber pg(4.10) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6 (0x55fec00458c0) 2021-02-10T04:34:58.550 INFO:tasks.rados.rados.0.smithi085.stdout:1849: done (9 left) 2021-02-10T04:34:58.550 INFO:tasks.rados.rados.0.smithi085.stdout:1850: snap_remove snap 181 2021-02-10T04:34:58.731 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:34:58.836 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:34:58.862 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: cluster 2021-02-10T04:34:57.132041+0000 osd.4 (osd 2021-02-10T04:34:58.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: .4) 669 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:58.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: cluster 2021-02-10T04:34:57.132053+0000 osd.4 (osd.4) 670 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:58.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: cluster 2021-02-10T04:34:57.132060+0000 osd.4 (osd.4) 2021-02-10T04:34:58.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: 671 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:58.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:58.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: -10T04:34:57.132066+0000 osd.4 (osd 2021-02-10T04:34:58.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: .4) 672 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:58.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: cluster 2021-02-10T04:34: 2021-02-10T04:34:58.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: 57.547123+0000 mon.a (mon.0) 1291 : cluster [DBG] osdmap e480: 8 total, 8 up, 8 in 2021-02-10T04:34:58.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: cluster 2021-02-10T04:34:57.682606+0000 2021-02-10T04:34:58.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: osd.2 (osd.2) 1187 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:58.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: cluster 2021-02-10T04:34: 2021-02-10T04:34:58.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: 57.682632+0000 osd.2 (osd.2) 1188 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:58.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:58.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: -10T04:34:57.682637+0000 osd.2 (osd.2) 1189 : cluster 2021-02-10T04:34:58.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:58.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:58.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: -10T04:34:57.682641+0000 osd.2 (osd.2) 1190 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:58.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: cluster 2021-02-10T04:34:57.682644+0000 2021-02-10T04:34:58.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: osd.2 (osd.2) 1191 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:58.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: cluster 2021-02-10T04:34:57.682648 2021-02-10T04:34:58.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1192 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:58.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: cluster 2021-02-10T04:34:57. 2021-02-10T04:34:58.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: 682651+0000 osd.2 (osd.2) 1193 : cluster [WRN] 2021-02-10T04:34:58.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:58.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:34:58.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: -10T04:34:57.682655+0000 osd.2 (osd.2) 1194 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:58.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:34:58.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: 04:34:57.682658+0000 osd.2 (osd.2) 2021-02-10T04:34:58.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: 1195 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:58.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:34:58.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: 34:57.682661+0000 osd.2 (osd.2) 1196 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:58.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: 2021-02-10T04:34:58.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: audit 2021 2021-02-10T04:34:58.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: -02-10T04:34:58.056010+0000 mgr.y (mgr.14140) 577 : 2021-02-10T04:34:58.866 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:34:58.866 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: cluster 2021-02-10T04:34:58.866 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: 2021-02-10T04:34:58.094125+0000 2021-02-10T04:34:58.866 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: mgr.y (mgr.14140) 578 : cluster [DBG] 2021-02-10T04:34:58.866 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: pgmap v870: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 397 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 2.9 MiB/s wr, 26 op/s 2021-02-10T04:34:58.866 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: cluster 2021-02-10T04:34:58 2021-02-10T04:34:58.866 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: .143770+0000 mon.a (mon.0) 1292 : 2021-02-10T04:34:58.866 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[29288]: cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 322 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:34:58.867 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:34:58.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[49106]: debug 2021-02-10T04:34:58.662+0000 7f77a3e94700 -1 osd.2 481 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:34:58.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: cluster 2021-02-10T04:34:57.132041 2021-02-10T04:34:58.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 669 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:58.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: cluster 2021-02-10T04:34:58.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: 2021-02-10T04:34:57.132053+0000 2021-02-10T04:34:58.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: osd.4 (osd.4) 670 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:58.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: 2021-02-10T04:34:58.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: 02-10T04:34:57.132060+0000 osd.4 (osd.4) 2021-02-10T04:34:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: 671 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: 02-10T04:34:57.132066+0000 osd.4 (osd. 2021-02-10T04:34:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: 4) 672 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: 02-10T04:34:57.547123+0000 mon.a (mon. 2021-02-10T04:34:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: 0) 1291 : cluster [DBG] osdmap e480: 8 total, 8 up, 8 in 2021-02-10T04:34:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: 02-10T04:34:57.682606+0000 osd.2 (osd 2021-02-10T04:34:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: .2) 1187 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: 02-10T04:34:57.682632+0000 osd.2 (osd 2021-02-10T04:34:58.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: .2) 1188 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:58.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:58.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: -02-10T04:34:57.682637+0000 2021-02-10T04:34:58.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: osd.2 (osd.2) 1189 : 2021-02-10T04:34:58.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:58.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: cluster 2021-02-10T04:34:58.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: 2021-02-10T04:34:58.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: 2021 2021-02-10T04:34:58.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: - 2021-02-10T04:34:58.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: 02-10T04:34:57 2021-02-10T04:34:58.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: .682641+0000 osd.2 (osd.2) 1190 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:58.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: cluster 2021-02-10T04:34:58.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: 2021-02-10T04:34:57.682644 2021-02-10T04:34:58.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1191 : cluster [WRN] 2021-02-10T04:34:58.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: cluster 2021-02-10T04:34:58.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: 2021-02-10T04:34:58.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: 2021 2021-02-10T04:34:58.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: - 2021-02-10T04:34:58.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: 02 2021-02-10T04:34:58.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: -10T04:34:57.132041 2021-02-10T04:34:58.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 669 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:34:58.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: cluster 2021-02-10T04:34:58.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:34:58.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: :34:57.132053+0000 osd.4 (osd.4) 670 2021-02-10T04:34:58.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:34:58.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:34:58.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: -10T04:34:57.132060+0000 osd.4 (osd. 2021-02-10T04:34:58.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: 4) 671 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:34:58.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: cluster 2021-02-10T04:34:57.132066+0000 osd.4 (osd.4) 672 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:34:58.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: cluster 2021-02-10T04:34:57 2021-02-10T04:34:58.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: .547123+0000 mon.a (mon.0) 1291 : cluster 2021-02-10T04:34:58.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: [DBG] osdmap e480: 8 total, 8 up, 8 in 2021-02-10T04:34:58.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: cluster 2021-02-10T04:34: 2021-02-10T04:34:58.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: 57.682606+0000 osd.2 (osd.2) 1187 : cluster 2021-02-10T04:34:58.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:34:58.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: cluster 2021-02-10T04:34:57.682632 2021-02-10T04:34:58.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1188 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:34:58.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: 2021-02-10T04:34:58.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:34:58.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: 10T04:34:57.682637+0000 osd.2 (osd.2) 1189 : cluster 2021-02-10T04:34:58.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:34:58.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: cluster 2021-02-10T04:34:57 2021-02-10T04:34:58.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: .682641+0000 osd.2 (osd.2) 1190 : cluster 2021-02-10T04:34:58.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:34:58.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: cluster 2021-02-10T04:34:57 2021-02-10T04:34:58.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: .682644+0000 osd.2 (osd.2) 1191 : cluster [WRN] 2021-02-10T04:34:58.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:58.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:34:58.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: 04:34:57.682648+0000 osd.2 (osd. 2021-02-10T04:34:58.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: 2) 1192 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:58.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:34:58.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: -10T04:34:57.682651+0000 osd.2 2021-02-10T04:34:58.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: (osd.2) 1193 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:58.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: cluster 2021-02-10T04:34:58.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: 2021-02-10T04:34:57.682655 2021-02-10T04:34:58.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1194 : cluster 2021-02-10T04:34:58.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:58.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:34:58.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: :34:57.682658+0000 osd.2 (osd.2 2021-02-10T04:34:58.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: ) 1195 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:58.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:34:58.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: 02-10T04:34:57.682661+0000 2021-02-10T04:34:58.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: osd.2 (osd.2) 1196 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:58.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: 2021-02-10T04:34:58.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: audit 2021-02-10T04:34: 2021-02-10T04:34:58.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: 58.056010+0000 mgr.y (mgr.14140) 577 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:34:58.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: 2021-02-10T04:34:58.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:34:58.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: -02-10T04:34:58.094125+0000 mgr.y (mgr.14140 2021-02-10T04:34:58.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: ) 578 : cluster [DBG] pgmap v870: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 397 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 2.9 MiB/s wr, 26 op/s 2021-02-10T04:34:58.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:34:58.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: T04:34:58.143770+0000 mon.a (mon 2021-02-10T04:34:58.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[35961]: .0) 1292 : cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 322 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:34:58.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:34:58.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: cluster 2021-02-10T04:34:58.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: 2021-02-10T04:34:57.682648+0000 osd.2 (osd.2 2021-02-10T04:34:58.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: ) 1192 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:34:58.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:34:58.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: 04:34:57.682651+0000 osd.2 (osd.2 2021-02-10T04:34:58.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: ) 1193 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:34:58.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: cluster 2021-02-10T04:34 2021-02-10T04:34:58.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: :57.682655+0000 osd.2 (osd.2) 1194 : cluster 2021-02-10T04:34:58.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:34:58.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: cluster 2021-02-10T04:34:57 2021-02-10T04:34:58.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: .682658+0000 osd.2 (osd.2) 1195 : cluster [WRN] 2021-02-10T04:34:58.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:34:58.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: cluster 2021-02-10T04:34:57. 2021-02-10T04:34:58.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: 682661+0000 osd.2 (osd.2) 1196 : cluster [WRN] 2021-02-10T04:34:58.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:34:58.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: audit 2021-02-10T04:34:58.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: 2021-02-10T04:34:58.056010+0000 mgr.y (mgr 2021-02-10T04:34:58.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: .14140) 577 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:34:58.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:34:58.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: 02-10T04:34:58 2021-02-10T04:34:58.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: .094125+0000 mgr.y (mgr.14140) 578 2021-02-10T04:34:58.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: : cluster [DBG] pgmap v870: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 397 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 2.9 MiB/s wr, 26 op/s 2021-02-10T04:34:58.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: cluster 2021-02-10T04:34 2021-02-10T04:34:58.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: :58.143770+0000 mon.a (mon.0) 1292 : 2021-02-10T04:34:58.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:58 smithi085 conmon[29319]: cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 322 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:34:59.143 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:34:58 smithi164 conmon[44544]: debug 2021-02-10T04:34:58.930+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:34:59.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[33677]: debug 2021-02-10T04:34:59.103+0000 7fbec19a1700 -1 osd.4 481 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:34:59.159 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:34:59.265 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:34:59.556 INFO:tasks.rados.rados.0.smithi085.stdout:1850: done (9 left) 2021-02-10T04:35:02.002 INFO:tasks.rados.rados.0.smithi085.stdout:1851: read oid 40 snap -1 2021-02-10T04:35:02.003 INFO:tasks.rados.rados.0.smithi085.stdout:1851: expect deleted 2021-02-10T04:35:02.003 INFO:tasks.rados.rados.0.smithi085.stdout:1852: write oid 22 current snap is 200 2021-02-10T04:35:02.003 INFO:tasks.rados.rados.0.smithi085.stdout:1852: seq_num 657 ranges {642199=644951,1696199=605559,2992318=49881} 2021-02-10T04:35:02.015 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: cluster 2021-02-10T04:34:58.083406+0000 2021-02-10T04:35:02.016 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: osd.4 (osd.4) 673 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:02.016 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: cluster 2021-02-10T04:34:58.083424+0000 osd.4 (osd. 2021-02-10T04:35:02.016 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: 4) 674 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:02.016 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: cluster 2021-02-10T04:35:02.016 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: 2021 2021-02-10T04:35:02.016 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: -02-10T04:34:58.083446 2021-02-10T04:35:02.016 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 675 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:02.016 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: 2021-02-10T04:35:02.016 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:35:02.016 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: T04:34:58.083454+0000 osd.4 (osd.4) 2021-02-10T04:35:02.016 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: 676 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:02.017 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:35:02.017 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: :34:58.550856+0000 mon.a (mon.0) 1293 2021-02-10T04:35:02.017 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: : cluster [DBG] osdmap e481: 8 total, 8 up, 8 in 2021-02-10T04:35:02.017 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: cluster 2021-02-10T04:35:02.017 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: 2021-02-10T04:34:58.663988+0000 osd.2 ( 2021-02-10T04:35:02.017 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: osd.2) 1197 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:02.017 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:35:02.017 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: 02-10T04:34:58.663997+0000 osd.2 (osd 2021-02-10T04:35:02.017 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: .2) 1198 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:02.017 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: cluster 2021-02-10T04:35:02.017 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: 2021-02-10T04:34:58.664004+0000 2021-02-10T04:35:02.017 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: osd.2 (osd.2) 1199 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:02.017 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: cluster 2021-02-10T04:35:02.018 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: 2021-02 2021-02-10T04:35:02.018 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: -10T04:34:58.664011+0000 osd.2 ( 2021-02-10T04:35:02.018 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: osd.2) 1200 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:02.018 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: cluster 2021-02-10T04:35:02.018 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: 2021-02-10T04:35:02.018 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: 2021-02-10T04:34:58.664016 2021-02-10T04:35:02.018 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1201 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:02.018 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: cluster 2021-02-10T04:34 2021-02-10T04:35:02.018 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: :58.664021+0000 osd.2 (osd.2) 1202 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:02.018 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: cluster 2021-02-10T04:34:58. 2021-02-10T04:35:02.018 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: 664026+0000 osd.2 (osd.2) 1203 : cluster [WRN] 2021-02-10T04:35:02.018 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:02.018 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: cluster 2021-02-10T04:34 2021-02-10T04:35:02.019 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: :58.664029+0000 osd.2 (osd.2) 1204 2021-02-10T04:35:02.019 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:02.019 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:35:02.019 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: 04:34:58.664034+0000 osd.2 (osd.2) 2021-02-10T04:35:02.019 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: 1205 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:02.019 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: cluster 2021-02-10T04:35:02.019 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: 2021-02-10T04:34:58.664037+0000 osd.2 (osd.2) 1206 2021-02-10T04:35:02.019 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:02.019 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: cluster 2021-02-10T04:34:59.549588+0000 mon.a 2021-02-10T04:35:02.019 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:34:59 smithi164 conmon[29288]: (mon.0) 1294 : cluster [DBG] osdmap e482: 8 total, 8 up, 8 in 2021-02-10T04:35:02.019 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: cluster 2021-02-10T04:35:02.019 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: 2021-02-10T04:35:02.019 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: 2021 2021-02-10T04:35:02.020 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: -02-10T04:34:59 2021-02-10T04:35:02.020 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: .105242+0000 osd.4 (osd.4) 677 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:02.020 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: cluster 2021-02-10T04:34:59 2021-02-10T04:35:02.020 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: .105251+0000 osd.4 (osd.4) 678 : cluster [WRN] 2021-02-10T04:35:02.020 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:02.020 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:02.020 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: -10T04:34:59.105257+0000 osd.4 (osd 2021-02-10T04:35:02.020 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: .4) 679 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:02.020 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:02.020 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: -10T04:34:59.105276+0000 osd.4 (osd 2021-02-10T04:35:02.020 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: .4) 680 : cluster [WRN] 2021-02-10T04:35:02.020 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:02.021 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:35:02.021 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: 02-10T04:34:59.630820+0000 osd.2 ( 2021-02-10T04:35:02.021 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: osd.2) 1207 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:02.021 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: 2021-02-10T04:35:02.021 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:35:02.021 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: -02-10T04:34:59.630831+0000 2021-02-10T04:35:02.021 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: osd.2 (osd.2) 1208 : 2021-02-10T04:35:02.021 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:02.021 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:35:02.021 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: :34:59.630840+0000 osd.2 (osd.2) 1209 : cluster [WRN] 2021-02-10T04:35:02.021 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:02.021 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: cluster 2021-02-10T04:35:02.021 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: 2021-02-10T04:34:59.630868+0000 osd.2 (osd.2 2021-02-10T04:35:02.022 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: ) 1210 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:02.022 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: cluster 2021-02-10T04:34:59. 2021-02-10T04:35:02.022 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: 630877+0000 osd.2 (osd.2) 1211 : cluster [WRN] 2021-02-10T04:35:02.022 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:02.022 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: cluster 2021-02-10T04:34:59.630881 2021-02-10T04:35:02.022 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1212 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:02.022 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: 2021-02-10T04:35:02.023 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:02.023 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: -10T04:34:59.630886+0000 osd.2 (osd.2) 1213 : cluster 2021-02-10T04:35:02.023 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:02.023 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: cluster 2021-02-10T04:34:59.630890+0000 2021-02-10T04:35:02.023 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: osd.2 (osd.2) 1214 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:02.023 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: cluster 2021-02-10T04:35:02.023 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: 2021-02-10T04:34:59 2021-02-10T04:35:02.023 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: .630894+0000 osd.2 (osd.2) 1215 : cluster [WRN] 2021-02-10T04:35:02.023 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:02.023 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:35:02.023 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: 02-10T04:34:59.630900+0000 osd.2 (osd.2) 1216 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:02.023 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[29288]: cluster 2021-02-10T04:35:00.094615+0000 mgr.y (mgr.14140) 579 : cluster [DBG] pgmap v873: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 402 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.7 MiB/s wr, 9 op/s 2021-02-10T04:35:02.023 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: cluster 2021-02-10T04:35:00.087459+0000 osd.4 (osd.4) 681 : cluster 2021-02-10T04:35:02.024 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:02.024 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: cluster 2021-02-10T04:35:00.087464+0000 osd.4 (osd.4) 682 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:02.024 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: cluster 2021-02-10T04:35:00.087467+0000 osd.4 (osd.4) 683 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:02.024 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:35:02.024 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: 02-10T04:35:00. 2021-02-10T04:35:02.024 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: 087471+0000 osd.4 (osd.4) 2021-02-10T04:35:02.024 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: 684 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:02.024 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: cluster 2021-02-10T04:35:02.024 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: 2021-02- 2021-02-10T04:35:02.024 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: 10T04:35:00.579231+0000 2021-02-10T04:35:02.024 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: mon.a (mon.0) 1295 : cluster [DBG] 2021-02-10T04:35:02.024 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: osdmap e483: 8 total, 8 up, 8 in 2021-02-10T04:35:02.025 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: cluster 2021-02-10T04:35: 2021-02-10T04:35:02.025 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: 00.734412+0000 osd.2 (osd.2) 1217 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:02.025 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: 2021-02-10T04:35:02.025 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: cluster 2021-02-10T04:35: 2021-02-10T04:35:02.025 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: 00.734423+0000 osd.2 (osd.2) 1218 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:02.025 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:35:02.025 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: T04:35:00.734429+0000 osd.2 (osd. 2021-02-10T04:35:02.025 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: 2) 1219 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:02.025 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:35:02.026 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: 02-10T04:35:00. 2021-02-10T04:35:02.026 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: 734435+0000 osd.2 (osd.2) 2021-02-10T04:35:02.026 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: 1220 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:02.026 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: cluster 2021-02-10T04:35:02.026 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: 2021-02-10T04:35:00.734441+0000 osd.2 2021-02-10T04:35:02.026 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: (osd.2) 1221 : cluster 2021-02-10T04:35:02.026 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:02.026 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:35:02.026 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: T04:35:00.734447+0000 osd.2 (osd.2) 1222 : cluster 2021-02-10T04:35:02.026 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:02.026 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: cluster 2021-02-10T04:35:02.026 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: 2021-02-10T04:35:00.734454+0000 osd.2 (osd.2) 1223 : 2021-02-10T04:35:02.026 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:02.027 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:35:02.027 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: -02-10T04:35: 2021-02-10T04:35:02.027 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: 00.734461+0000 osd.2 (osd.2) 1224 : 2021-02-10T04:35:02.027 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:02.027 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:35:02.027 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: -02-10T04:35:00.734466+0000 2021-02-10T04:35:02.027 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: osd.2 (osd.2) 2021-02-10T04:35:02.027 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: 1225 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:02.027 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:02.027 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: -10T04:35:00.734471+0000 osd.2 (osd.2 2021-02-10T04:35:02.027 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: ) 1226 : cluster 2021-02-10T04:35:02.027 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:02.028 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:00 smithi164 conmon[33677]: debug 2021-02-10T04:35:00.085+0000 7fbec19a1700 -1 osd.4 482 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:35:02.028 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[33677]: debug 2021-02-10T04:35:01.134+0000 7fbec19a1700 -1 osd.4 483 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:35:02.028 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[33677]: debug 2021-02-10T04:35:01.134+0000 7fbea992c700 -1 osd.4 pg_epoch: 483 pg[6.1a( empty local-lis/les=59/60 n=0 ec=59/59 lis/c=59/59 les/c/f=60/60/0 sis=59) [4,5,1] r=0 lpr=59 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.1a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x562760a81340) 2021-02-10T04:35:02.028 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[33677]: 2021-02-10T04:35:02.028 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:35:01 smithi164 conmon[40892]: debug 2021-02-10T04:35:01.366+0000 7f30364c7700 -1 osd.6 pg_epoch: 483 pg[7.d( v 481'362 (0'0,481'362] local-lis/les=93/94 n=10 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [6,1,0] r=0 lpr=93 crt=481'362 lcod 481'360 mlcod 481'360 active+clean+snaptrim trimq=[148~1,15b~1] ps=131] removing snap head 2021-02-10T04:35:02.028 INFO:tasks.rados.rados.0.smithi085.stdout:1852: writing smithi08575397-22 from 642199 to 1287150 tid 1 2021-02-10T04:35:02.028 INFO:tasks.rados.rados.0.smithi085.stdout:1852: writing smithi08575397-22 from 1696199 to 2301758 tid 2 2021-02-10T04:35:02.029 INFO:tasks.rados.rados.0.smithi085.stdout:1852: writing smithi08575397-22 from 2992318 to 3042199 tid 3 2021-02-10T04:35:02.029 INFO:tasks.rados.rados.0.smithi085.stdout:1853: write oid 48 current snap is 200 2021-02-10T04:35:02.029 INFO:tasks.rados.rados.0.smithi085.stdout:1853: seq_num 658 ranges {659006=720200,2154681=749217} 2021-02-10T04:35:02.029 INFO:tasks.rados.rados.0.smithi085.stdout:1853: writing smithi08575397-48 from 659006 to 1379206 tid 1 2021-02-10T04:35:02.029 INFO:tasks.rados.rados.0.smithi085.stdout:1853: writing smithi08575397-48 from 2154681 to 2903898 tid 2 2021-02-10T04:35:02.029 INFO:tasks.rados.rados.0.smithi085.stdout:1851: done (11 left) 2021-02-10T04:35:02.029 INFO:tasks.rados.rados.0.smithi085.stdout:1854: snap_create 2021-02-10T04:35:02.029 INFO:tasks.rados.rados.0.smithi085.stdout:1852: finishing write tid 1 to smithi08575397-22 2021-02-10T04:35:02.029 INFO:tasks.rados.rados.0.smithi085.stdout:1852: finishing write tid 2 to smithi08575397-22 2021-02-10T04:35:02.029 INFO:tasks.rados.rados.0.smithi085.stdout:1852: finishing write tid 3 to smithi08575397-22 2021-02-10T04:35:02.029 INFO:tasks.rados.rados.0.smithi085.stdout:1852: finishing write tid 4 to smithi08575397-22 2021-02-10T04:35:02.030 INFO:tasks.rados.rados.0.smithi085.stdout:1852: finishing write tid 5 to smithi08575397-22 2021-02-10T04:35:02.030 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 665 (ObjNum 657 snap 200 seq_num 657) dirty exists 2021-02-10T04:35:02.030 INFO:tasks.rados.rados.0.smithi085.stdout:1852: left oid 22 (ObjNum 657 snap 200 seq_num 657) 2021-02-10T04:35:02.030 INFO:tasks.rados.rados.0.smithi085.stdout:1853: finishing write tid 1 to smithi08575397-48 2021-02-10T04:35:02.030 INFO:tasks.rados.rados.0.smithi085.stdout:1853: finishing write tid 2 to smithi08575397-48 2021-02-10T04:35:02.030 INFO:tasks.rados.rados.0.smithi085.stdout:1853: finishing write tid 3 to smithi08575397-48 2021-02-10T04:35:02.030 INFO:tasks.rados.rados.0.smithi085.stdout:1853: finishing write tid 4 to smithi08575397-48 2021-02-10T04:35:02.030 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 796 (ObjNum 658 snap 200 seq_num 658) dirty exists 2021-02-10T04:35:02.030 INFO:tasks.rados.rados.0.smithi085.stdout:1853: left oid 48 (ObjNum 658 snap 200 seq_num 658) 2021-02-10T04:35:02.030 INFO:tasks.rados.rados.0.smithi085.stdout:1852: done (11 left) 2021-02-10T04:35:02.030 INFO:tasks.rados.rados.0.smithi085.stdout:1853: done (10 left) 2021-02-10T04:35:02.030 INFO:tasks.rados.rados.0.smithi085.stdout:1854: done (9 left) 2021-02-10T04:35:02.030 INFO:tasks.rados.rados.0.smithi085.stdout:1855: copy_from oid 22 from oid 30 current snap is 201 2021-02-10T04:35:02.031 INFO:tasks.rados.rados.0.smithi085.stdout:1856: write (excl) oid 31 current snap is 201 2021-02-10T04:35:02.031 INFO:tasks.rados.rados.0.smithi085.stdout:1856: seq_num 659 ranges {675813=795449,2096810=609228} 2021-02-10T04:35:02.031 INFO:tasks.rados.rados.0.smithi085.stdout:1856: writing smithi08575397-31 from 675813 to 1471262 tid 1 2021-02-10T04:35:02.031 INFO:tasks.rados.rados.0.smithi085.stdout:1856: writing smithi08575397-31 from 2096810 to 2706038 tid 2 2021-02-10T04:35:02.031 INFO:tasks.rados.rados.0.smithi085.stdout:1857: snap_create 2021-02-10T04:35:02.031 INFO:tasks.rados.rados.0.smithi085.stdout:1856: finishing write tid 1 to smithi08575397-31 2021-02-10T04:35:02.031 INFO:tasks.rados.rados.0.smithi085.stdout:1856: finishing write tid 2 to smithi08575397-31 2021-02-10T04:35:02.031 INFO:tasks.rados.rados.0.smithi085.stdout:1856: finishing write tid 3 to smithi08575397-31 2021-02-10T04:35:02.031 INFO:tasks.rados.rados.0.smithi085.stdout:1856: finishing write tid 4 to smithi08575397-31 2021-02-10T04:35:02.031 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 668 (ObjNum 659 snap 201 seq_num 659) dirty exists 2021-02-10T04:35:02.031 INFO:tasks.rados.rados.0.smithi085.stdout:1856: left oid 31 (ObjNum 659 snap 201 seq_num 659) 2021-02-10T04:35:02.032 INFO:tasks.rados.rados.0.smithi085.stdout:1855: finishing copy_from to smithi08575397-22 2021-02-10T04:35:02.032 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 769 (ObjNum 654 snap 197 seq_num 654) dirty exists 2021-02-10T04:35:02.032 INFO:tasks.rados.rados.0.smithi085.stdout:1855: finishing copy_from racing read to smithi08575397-22 2021-02-10T04:35:02.032 INFO:tasks.rados.rados.0.smithi085.stdout:1855: done (11 left) 2021-02-10T04:35:02.032 INFO:tasks.rados.rados.0.smithi085.stdout:1856: done (10 left) 2021-02-10T04:35:02.032 INFO:tasks.rados.rados.0.smithi085.stdout:1857: done (9 left) 2021-02-10T04:35:02.032 INFO:tasks.rados.rados.0.smithi085.stdout:1858: write oid 28 current snap is 202 2021-02-10T04:35:02.032 INFO:tasks.rados.rados.0.smithi085.stdout:1858: seq_num 660 ranges {692620=470698,1755292=752886} 2021-02-10T04:35:02.032 INFO:tasks.rados.rados.0.smithi085.stdout:1858: writing smithi08575397-28 from 692620 to 1163318 tid 1 2021-02-10T04:35:02.033 INFO:tasks.rados.rados.0.smithi085.stdout:1858: writing smithi08575397-28 from 1755292 to 2508178 tid 2 2021-02-10T04:35:02.033 INFO:tasks.rados.rados.0.smithi085.stdout:1859: delete oid 3 current snap is 202 2021-02-10T04:35:02.033 INFO:tasks.rados.rados.0.smithi085.stdout:1858: finishing write tid 1 to smithi08575397-28 2021-02-10T04:35:02.033 INFO:tasks.rados.rados.0.smithi085.stdout:1858: finishing write tid 2 to smithi08575397-28 2021-02-10T04:35:02.033 INFO:tasks.rados.rados.0.smithi085.stdout:1859: done (10 left) 2021-02-10T04:35:02.033 INFO:tasks.rados.rados.0.smithi085.stdout:1860: delete oid 10 current snap is 202 2021-02-10T04:35:02.033 INFO:tasks.rados.rados.0.smithi085.stdout:1858: finishing write tid 3 to smithi08575397-28 2021-02-10T04:35:02.033 INFO:tasks.rados.rados.0.smithi085.stdout:1858: finishing write tid 4 to smithi08575397-28 2021-02-10T04:35:02.033 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 918 (ObjNum 660 snap 202 seq_num 660) dirty exists 2021-02-10T04:35:02.033 INFO:tasks.rados.rados.0.smithi085.stdout:1858: left oid 28 (ObjNum 660 snap 202 seq_num 660) 2021-02-10T04:35:02.033 INFO:tasks.rados.rados.0.smithi085.stdout:1858: done (10 left) 2021-02-10T04:35:02.033 INFO:tasks.rados.rados.0.smithi085.stdout:1860: done (9 left) 2021-02-10T04:35:02.033 INFO:tasks.rados.rados.0.smithi085.stdout:1861: read oid 1 snap 196 2021-02-10T04:35:02.034 INFO:tasks.rados.rados.0.smithi085.stdout:1861: expect (ObjNum 625 snap 190 seq_num 625) 2021-02-10T04:35:02.034 INFO:tasks.rados.rados.0.smithi085.stdout:1862: write (excl) oid 5 current snap is 202 2021-02-10T04:35:02.034 INFO:tasks.rados.rados.0.smithi085.stdout:1862: seq_num 663 ranges {743041=412798,1930738=616566,3040503=102538} 2021-02-10T04:35:02.034 INFO:tasks.rados.rados.0.smithi085.stdout:1862: writing smithi08575397-5 from 743041 to 1155839 tid 1 2021-02-10T04:35:02.034 INFO:tasks.rados.rados.0.smithi085.stdout:1862: writing smithi08575397-5 from 1930738 to 2547304 tid 2 2021-02-10T04:35:02.034 INFO:tasks.rados.rados.0.smithi085.stdout:1862: writing smithi08575397-5 from 3040503 to 3143041 tid 3 2021-02-10T04:35:02.034 INFO:tasks.rados.rados.0.smithi085.stdout:1863: delete oid 37 current snap is 202 2021-02-10T04:35:02.034 INFO:tasks.rados.rados.0.smithi085.stdout:1861: expect (ObjNum 625 snap 190 seq_num 625) 2021-02-10T04:35:02.034 INFO:tasks.rados.rados.0.smithi085.stdout:1862: finishing write tid 1 to smithi08575397-5 2021-02-10T04:35:02.034 INFO:tasks.rados.rados.0.smithi085.stdout:1861: done (11 left) 2021-02-10T04:35:02.034 INFO:tasks.rados.rados.0.smithi085.stdout:1863: done (10 left) 2021-02-10T04:35:02.034 INFO:tasks.rados.rados.0.smithi085.stdout:1864: copy_from oid 44 from oid 43 current snap is 202 2021-02-10T04:35:02.034 INFO:tasks.rados.rados.0.smithi085.stdout:1862: finishing write tid 2 to smithi08575397-5 2021-02-10T04:35:02.035 INFO:tasks.rados.rados.0.smithi085.stdout:1862: finishing write tid 3 to smithi08575397-5 2021-02-10T04:35:02.035 INFO:tasks.rados.rados.0.smithi085.stdout:1862: finishing write tid 4 to smithi08575397-5 2021-02-10T04:35:02.035 INFO:tasks.rados.rados.0.smithi085.stdout:1862: finishing write tid 5 to smithi08575397-5 2021-02-10T04:35:02.035 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 673 (ObjNum 663 snap 202 seq_num 663) dirty exists 2021-02-10T04:35:02.035 INFO:tasks.rados.rados.0.smithi085.stdout:1862: left oid 5 (ObjNum 663 snap 202 seq_num 663) 2021-02-10T04:35:02.035 INFO:tasks.rados.rados.0.smithi085.stdout:1862: done (10 left) 2021-02-10T04:35:02.035 INFO:tasks.rados.rados.0.smithi085.stdout:1865: read oid 8 snap -1 2021-02-10T04:35:02.035 INFO:tasks.rados.rados.0.smithi085.stdout:1865: expect (ObjNum 641 snap 196 seq_num 641) 2021-02-10T04:35:02.035 INFO:tasks.rados.rados.0.smithi085.stdout:1866: copy_from oid 40 from oid 28 current snap is 202 2021-02-10T04:35:02.035 INFO:tasks.rados.rados.0.smithi085.stdout:1867: copy_from oid 11 from oid 6 current snap is 202 2021-02-10T04:35:02.035 INFO:tasks.rados.rados.0.smithi085.stdout:1868: snap_remove snap 197 2021-02-10T04:35:02.035 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:35:02.036 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[49106]: debug 2021-02-10T04:34:59.629+0000 7f77a3e94700 -1 osd.2 482 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:35:02.036 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[49106]: debug 2021-02-10T04:34:59.745+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:35:02.036 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[49106]: debug 2021-02-10T04:35:00.732+0000 7f77a3e94700 -1 osd.2 482 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:35:02.036 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[49106]: debug 2021-02-10T04:35:01.717+0000 7f77a3e94700 -1 osd.2 484 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:35:02.036 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: cluster 2021-02-10T04:34:58.083406+0000 osd.4 (osd.4) 673 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:02.036 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: cluster 2021-02-10T04:34:58.083424+0000 osd.4 (osd.4) 674 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:02.036 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: cluster 2021-02-10T04:34:58 2021-02-10T04:35:02.036 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: .083446+0000 osd.4 (osd.4) 675 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:02.036 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:35:02.036 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: 34:58.083454+0000 osd.4 (osd.4) 676 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:02.036 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:35:02.037 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: -02-10T04:34:58.550856+0000 2021-02-10T04:35:02.037 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: mon.a (mon.0) 1293 : cluster [DBG] osdmap e481: 8 total, 8 up, 8 in 2021-02-10T04:35:02.037 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: 2021-02-10T04:35:02.037 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: cluster 2021-02-10T04:35:02.037 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: 2021-02-10T04:34:58.663988+0000 osd.2 ( 2021-02-10T04:35:02.037 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: osd.2) 1197 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:02.037 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:35:02.037 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: -10T04:34:58.663997+0000 osd.2 ( 2021-02-10T04:35:02.037 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: osd.2) 1198 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:02.037 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:02.037 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: 02-10T04:34:58.664004+0000 2021-02-10T04:35:02.037 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: osd.2 (osd.2) 1199 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:02.037 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: 2021-02-10T04:35:02.038 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: cluster 2021-02-10T04:34:58.664011 2021-02-10T04:35:02.038 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1200 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:02.038 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:35:02.038 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: 10T04:34:58.664016+0000 osd.2 (osd.2) 1201 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:02.038 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:02.038 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: 02-10T04:34:58.664021+0000 2021-02-10T04:35:02.038 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: osd.2 (osd.2) 1202 : cluster [WRN] 2021-02-10T04:35:02.038 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:02.038 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: cluster 2021-02-10T04:35:02.038 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: 2021-02-10T04:34:58.664026+0000 osd.2 (osd.2 2021-02-10T04:35:02.038 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: ) 1203 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:02.038 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:35:02.039 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: 04:34:58.664029+0000 osd.2 (osd.2) 1204 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:02.039 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:35:02.039 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: -10T04:34:58.664034+0000 osd.2 (osd.2) 1205 2021-02-10T04:35:02.039 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:02.039 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:35:02.039 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: 34:58.664037+0000 osd.2 (osd.2) 2021-02-10T04:35:02.039 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: 1206 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:02.039 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: cluster 2021-02-10T04:34:59 2021-02-10T04:35:02.039 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: .549588+0000 mon.a (mon.0) 1294 : cluster [DBG] 2021-02-10T04:35:02.039 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[29319]: osdmap e482: 8 total, 8 up, 8 in 2021-02-10T04:35:02.039 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: cluster 2021-02-10T04:34:59.105242+0000 osd.4 (osd 2021-02-10T04:35:02.039 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: .4) 677 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:02.039 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: cluster 2021-02-10T04:34:59.105251+0000 osd.4 (osd.4) 678 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:02.040 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: cluster 2021-02-10T04:34:59.105257 2021-02-10T04:35:02.040 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 679 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:02.040 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: cluster 2021-02-10T04:35:02.040 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: 2021-02-10T04:34:59.105276+0000 2021-02-10T04:35:02.040 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: osd.4 (osd.4) 680 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:02.040 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: cluster 2021-02-10T04:34:59. 2021-02-10T04:35:02.040 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: 630820+0000 osd.2 (osd.2) 1207 : cluster [WRN] 2021-02-10T04:35:02.041 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:02.041 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: cluster 2021-02-10T04:34:59.630831+0000 2021-02-10T04:35:02.041 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: osd.2 (osd.2) 1208 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:02.041 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: cluster 2021-02-10T04:34: 2021-02-10T04:35:02.041 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: 59.630840+0000 osd.2 (osd.2) 1209 : cluster 2021-02-10T04:35:02.041 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:02.042 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: cluster 2021-02-10T04:34: 2021-02-10T04:35:02.042 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: 59.630868+0000 osd.2 (osd.2) 1210 : cluster 2021-02-10T04:35:02.042 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:02.042 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: cluster 2021-02-10T04:34: 2021-02-10T04:35:02.043 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: 59.630877+0000 osd.2 (osd.2) 1211 : cluster 2021-02-10T04:35:02.043 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:02.043 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: cluster 2021-02-10T04:34:59 2021-02-10T04:35:02.043 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: .630881+0000 osd.2 (osd.2) 1212 : cluster [WRN] 2021-02-10T04:35:02.044 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:02.044 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:35:02.044 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: 34:59.630886+0000 osd.2 (osd.2) 2021-02-10T04:35:02.044 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: 1213 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:02.044 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: cluster 2021-02-10T04:34: 2021-02-10T04:35:02.044 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: 59.630890+0000 osd.2 (osd.2) 1214 : cluster [WRN] 2021-02-10T04:35:02.045 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:02.045 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: cluster 2021-02-10T04:34 2021-02-10T04:35:02.045 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: :59.630894+0000 osd.2 (osd.2 2021-02-10T04:35:02.045 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: ) 1215 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:02.045 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:35:02.045 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: -10T04:34:59.630900+0000 osd.2 (osd 2021-02-10T04:35:02.046 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: .2) 1216 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:02.046 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:35:02.046 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: -10T04:35:00.094615+0000 mgr.y ( 2021-02-10T04:35:02.046 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[29319]: mgr.14140) 579 : cluster [DBG] pgmap v873: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 402 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.7 MiB/s wr, 9 op/s 2021-02-10T04:35:02.046 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: cluster 2021-02-10T04:35:02.046 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: 2021-02-10T04:35:02.047 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: 2021 2021-02-10T04:35:02.048 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: - 2021-02-10T04:35:02.048 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: 02 2021-02-10T04:35:02.048 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: - 2021-02-10T04:35:02.048 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: 10 2021-02-10T04:35:02.048 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: T04:35:00. 2021-02-10T04:35:02.048 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: 087459+0000 osd.4 (osd.4) 681 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:02.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: cluster 2021-02-10T04:35:02.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: 2021- 2021-02-10T04:35:02.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: 02-10T04:35:00.087464+0000 osd.4 ( 2021-02-10T04:35:02.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: osd.4) 682 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:02.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:02.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: 02-10T04:35:00.087467+0000 osd.4 (osd 2021-02-10T04:35:02.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: .4) 683 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:02.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:35:02.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: T04:35:00.087471+0000 osd.4 (osd. 2021-02-10T04:35:02.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: 4) 684 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:02.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: cluster 2021-02-10T04:35:02.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: 2021-02-10T04:35:00.579231+0000 2021-02-10T04:35:02.050 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: mon.a (mon.0) 1295 : cluster [DBG] 2021-02-10T04:35:02.050 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: osdmap e483: 8 total, 8 up, 8 in 2021-02-10T04:35:02.050 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:02.050 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: 02-10T04:35:00.734412+0000 2021-02-10T04:35:02.050 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: osd.2 (osd.2) 1217 : cluster [WRN] 2021-02-10T04:35:02.050 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:02.050 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:35:02.050 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: 35:00.734423+0000 osd.2 (osd.2 2021-02-10T04:35:02.050 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: ) 1218 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:02.050 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:35:02.050 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: 04:35:00.734429+0000 osd.2 (osd 2021-02-10T04:35:02.050 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: .2) 1219 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:02.050 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: cluster 2021-02-10T04:35:02.051 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: 2021-02-10T04:35: 2021-02-10T04:35:02.051 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: 00.734435+0000 osd.2 (osd.2) 1220 2021-02-10T04:35:02.051 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:02.051 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: cluster 2021-02-10T04:35:02.051 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:35:02.051 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: 35:00.734441+0000 osd.2 (osd.2 2021-02-10T04:35:02.051 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: ) 1221 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:02.051 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:35:02.051 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: -02-10T04:35:00.734447+0000 2021-02-10T04:35:02.051 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: osd.2 (osd.2) 1222 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:02.051 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: 2021-02-10T04:35:02.051 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: cluster 2021-02-10T04:35:02.052 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: 2021-02-10T04:35:00.734454 2021-02-10T04:35:02.052 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: +0000 osd.2 (osd.2 2021-02-10T04:35:02.052 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: ) 1223 : cluster 2021-02-10T04:35:02.052 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:02.052 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:35:02.052 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: 10T04:35:00.734461+0000 osd.2 ( 2021-02-10T04:35:02.052 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: osd.2) 1224 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:02.052 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: cluster 2021-02-10T04:35:02.052 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: 2021-02-10T04:35:02.052 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: 2021 2021-02-10T04:35:02.052 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: -02-10T04:35:00 2021-02-10T04:35:02.052 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: .734466+0000 osd.2 (osd.2) 1225 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:02.053 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:02.053 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: 02-10T04:35:00.734471+0000 osd.2 2021-02-10T04:35:02.053 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: (osd.2) 1226 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:02.053 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[29319]: 2021-02-10T04:35:02.053 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: cluster 2021-02-10T04:35:02.053 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: 2021-02-10T04:35:02.053 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: 2021 2021-02-10T04:35:02.053 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: - 2021-02-10T04:35:02.053 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: 02 2021-02-10T04:35:02.053 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: - 2021-02-10T04:35:02.053 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: 10T04:34:58 2021-02-10T04:35:02.053 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: .083406+0000 osd.4 (osd.4) 673 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:02.054 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: cluster 2021-02-10T04:35:02.054 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: 2021-02-10T04:34 2021-02-10T04:35:02.054 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: :58.083424+0000 osd.4 (osd.4) 674 2021-02-10T04:35:02.054 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:02.054 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: cluster 2021-02-10T04:35:02.054 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: 2021-02-10T04:34:58. 2021-02-10T04:35:02.054 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: 083446+0000 osd.4 (osd.4) 675 : cluster 2021-02-10T04:35:02.054 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:02.054 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: cluster 2021-02-10T04:35:02.054 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: 2021-02-10T04:34 2021-02-10T04:35:02.054 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: :58.083454+0000 osd.4 (osd.4) 2021-02-10T04:35:02.054 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: 676 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:02.055 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: cluster 2021-02-10T04:35:02.055 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:35:02.055 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: 34:58.550856+0000 mon.a (mon.0 2021-02-10T04:35:02.055 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: ) 1293 : cluster [DBG] osdmap e481: 8 total, 8 up, 8 in 2021-02-10T04:35:02.055 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: cluster 2021-02-10T04:35:02.055 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: 2021-02-10T04:34:58.663988+0000 osd.2 (osd.2) 1197 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:02.055 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: cluster 2021-02-10T04:35:02.055 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: 2021-02-10T04:34:58. 2021-02-10T04:35:02.055 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: 663997+0000 osd.2 (osd.2) 1198 : cluster 2021-02-10T04:35:02.055 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:02.055 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:35:02.055 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: 10T04:34:58.664004+0000 osd.2 ( 2021-02-10T04:35:02.055 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: osd.2) 1199 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:02.056 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: 2021-02-10T04:35:02.056 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:35:02.056 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: :34:58.664011+0000 osd.2 (osd. 2021-02-10T04:35:02.056 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: 2) 1200 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:02.056 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: cluster 2021-02-10T04:35:02.056 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:35:02.056 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: 04:34:58.664016+0000 osd.2 (osd.2 2021-02-10T04:35:02.056 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: ) 1201 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:02.056 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:35:02.056 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: -02-10T04:34:58.664021+0000 2021-02-10T04:35:02.056 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: osd.2 (osd.2) 1202 : cluster [WRN] 2021-02-10T04:35:02.056 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:02.056 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: cluster 2021-02-10T04:35:02.056 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: 2021-02-10T04:34:58. 2021-02-10T04:35:02.057 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: 664026+0000 osd.2 (osd.2) 1203 : cluster 2021-02-10T04:35:02.057 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:02.057 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:35:02.057 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: T04:34:58.664029+0000 osd.2 (osd 2021-02-10T04:35:02.057 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: .2) 1204 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:02.057 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: cluster 2021-02-10T04:35:02.057 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: 2021-02-10T04:34:58. 2021-02-10T04:35:02.057 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: 664034+0000 osd.2 (osd.2) 1205 : cluster 2021-02-10T04:35:02.057 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:02.057 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:35:02.057 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: 04:34:58.664037+0000 osd.2 (osd 2021-02-10T04:35:02.057 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: .2) 1206 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:02.057 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: cluster 2021-02-10T04:35:02.058 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: 2021-02-10T04:34:59.549588 2021-02-10T04:35:02.058 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1294 : cluster 2021-02-10T04:35:02.058 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:34:59 smithi085 conmon[35961]: [DBG] osdmap e482: 8 total, 8 up, 8 in 2021-02-10T04:35:02.058 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: cluster 2021-02-10T04:34:59. 2021-02-10T04:35:02.058 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: 105242+0000 osd.4 (osd.4) 677 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:02.058 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: cluster 2021-02-10T04:34: 2021-02-10T04:35:02.058 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: 59.105251+0000 osd.4 (osd. 2021-02-10T04:35:02.058 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: 4) 678 : cluster [WRN] 2021-02-10T04:35:02.058 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:02.058 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:35:02.058 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: -02-10T04:34 2021-02-10T04:35:02.058 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: :59.105257+0000 osd.4 ( 2021-02-10T04:35:02.058 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: osd.4) 679 : cluster [WRN] 2021-02-10T04:35:02.059 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:02.059 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:35:02.059 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: :34:59.105276+0000 osd.4 (osd. 2021-02-10T04:35:02.059 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: 4) 680 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:02.059 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:35:02.059 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: 10T04:34:59.630820+0000 osd.2 2021-02-10T04:35:02.059 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: (osd.2) 1207 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:02.059 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: 2021-02-10T04:35:02.059 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:35:02.059 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: 10T04:34:59.630831+0000 osd.2 (osd.2) 2021-02-10T04:35:02.059 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: 1208 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:02.059 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:35:02.060 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: 34:59.630840+0000 osd.2 (osd.2 2021-02-10T04:35:02.060 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: ) 1209 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:02.060 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:35:02.060 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: :34:59.630868+0000 osd.2 (osd. 2021-02-10T04:35:02.060 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: 2) 1210 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:02.060 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:35:02.060 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: 04:34:59.630877+0000 osd.2 (osd 2021-02-10T04:35:02.060 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: .2) 1211 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:02.060 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:35:02.060 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: T04:34:59.630881+0000 osd.2 ( 2021-02-10T04:35:02.060 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: osd.2) 1212 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:02.060 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:35:02.060 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: 04:34:59.630886+0000 osd.2 (osd. 2021-02-10T04:35:02.061 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: 2) 1213 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:02.061 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:35:02.061 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: :34:59.630890+0000 osd.2 (osd.2 2021-02-10T04:35:02.061 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: ) 1214 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:02.061 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:35:02.061 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: -10T04:34:59.630894 2021-02-10T04:35:02.061 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1215 : cluster 2021-02-10T04:35:02.061 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:02.062 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: cluster 2021-02-10T04:34: 2021-02-10T04:35:02.062 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: 59.630900+0000 osd.2 (osd.2) 1216 2021-02-10T04:35:02.062 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:02.062 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:35:02.062 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: 35:00.094615+0000 mgr.y (mgr.14140 2021-02-10T04:35:02.062 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:00 smithi085 conmon[35961]: ) 579 : cluster [DBG] pgmap v873: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 402 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.7 MiB/s wr, 9 op/s 2021-02-10T04:35:02.062 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: cluster 2021-02-10T04:35:00.087459+0000 osd.4 2021-02-10T04:35:02.062 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: (osd.4) 681 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:02.062 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: cluster 2021-02-10T04:35:00.087464+0000 osd.4 (osd 2021-02-10T04:35:02.062 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: .4) 682 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:02.062 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: cluster 2021-02-10T04:35:02.063 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: 2021-02-10T04:35:00.087467+0000 osd.4 (osd.4) 683 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:02.063 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:35:02.063 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: T04:35:00.087471+0000 osd.4 (osd.4) 684 : cluster [WRN] 2021-02-10T04:35:02.063 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:02.063 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:35:02.063 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: T04:35:00.579231+0000 mon.a (mon.0) 1295 : cluster [DBG] osdmap e483: 8 total, 8 up, 8 in 2021-02-10T04:35:02.063 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: 2021-02-10T04:35:02.064 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: cluster 2021-02-10T04:35:00.734412 2021-02-10T04:35:02.064 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1217 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:02.064 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: cluster 2021-02-10T04:35 2021-02-10T04:35:02.064 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: :00.734423+0000 osd.2 (osd.2) 1218 : 2021-02-10T04:35:02.064 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:02.064 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: cluster 2021-02-10T04:35:00.734429+0000 osd.2 (osd.2) 2021-02-10T04:35:02.064 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: 1219 2021-02-10T04:35:02.064 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: : 2021-02-10T04:35:02.065 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: cluster 2021-02-10T04:35:02.065 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: 2021-02-10T04:35:02.065 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: [WRN] 2021-02-10T04:35:02.065 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: 2021-02-10T04:35:02.065 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:02.065 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: 2021-02-10T04:35:02.065 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: cluster 2021-02-10T04:35: 2021-02-10T04:35:02.065 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: 00.734435+0000 osd.2 (osd.2) 1220 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:02.065 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:02.065 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: 02-10T04:35:00.734441 2021-02-10T04:35:02.065 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1221 : 2021-02-10T04:35:02.065 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:02.066 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:02.066 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: 02-10T04:35:00.734447+0000 osd.2 (osd.2) 1222 : 2021-02-10T04:35:02.066 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:02.066 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:35:02.066 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: T04:35:00.734454+0000 osd.2 (osd.2 2021-02-10T04:35:02.066 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: ) 1223 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:02.066 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: cluster 2021-02-10T04:35:02.066 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:35:02.066 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: 35:00.734461+0000 osd.2 (osd.2) 1224 2021-02-10T04:35:02.066 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:02.066 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:35:02.067 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: 10T04:35:00.734466+0000 osd.2 (osd 2021-02-10T04:35:02.067 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: .2) 1225 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:02.067 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: cluster 2021-02-10T04:35:02.067 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: 2021-02-10T04:35 2021-02-10T04:35:02.067 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: :00.734471+0000 osd.2 (osd.2) 1226 : 2021-02-10T04:35:02.067 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:01 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:02.068 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:35:02.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[33677]: debug 2021-02-10T04:35:02.091+0000 7fbec19a1700 -1 osd.4 484 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:35:02.601 INFO:tasks.rados.rados.0.smithi085.stdout:1868: done (13 left) 2021-02-10T04:35:02.601 INFO:tasks.rados.rados.0.smithi085.stdout:1869: snap_create 2021-02-10T04:35:02.612 INFO:tasks.rados.rados.0.smithi085.stdout:1865: expect (ObjNum 641 snap 196 seq_num 641) 2021-02-10T04:35:02.777 INFO:tasks.rados.rados.0.smithi085.stdout:1867: finishing copy_from to smithi08575397-11 2021-02-10T04:35:02.777 INFO:tasks.rados.rados.0.smithi085.stdout:1867: got expected ENOENT (src dne) 2021-02-10T04:35:02.777 INFO:tasks.rados.rados.0.smithi085.stdout:1867: finishing copy_from racing read to smithi08575397-11 2021-02-10T04:35:02.777 INFO:tasks.rados.rados.0.smithi085.stdout:1864: finishing copy_from to smithi08575397-44 2021-02-10T04:35:02.778 INFO:tasks.rados.rados.0.smithi085.stdout:1864: got expected ENOENT (src dne) 2021-02-10T04:35:02.778 INFO:tasks.rados.rados.0.smithi085.stdout:1864: finishing copy_from racing read to smithi08575397-44 2021-02-10T04:35:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: cluster 2021-02-10T04:35:01.135834+0000 osd.4 (osd. 2021-02-10T04:35:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: 4) 685 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: cluster 2021-02-10T04:35:01.135840+0000 osd.4 (osd.4) 686 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: cluster 2021-02-10T04:35:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: 2021-02-10T04:35:01.135844+0000 osd.4 (osd.4) 687 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:35:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: 02-10T04:35:01.135847+0000 2021-02-10T04:35:02.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: osd.4 (osd.4) 688 : cluster [WRN] 2021-02-10T04:35:02.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:02.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: cluster 2021-02-10T04:35:02.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: 2021-02-10T04:35:01.590970+0000 mon.a (mon.0) 1296 2021-02-10T04:35:02.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: : cluster [DBG] osdmap e484: 8 total, 8 up, 8 in 2021-02-10T04:35:02.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:35:02.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: 35:01.718712+0000 osd.2 (osd.2) 1227 : cluster 2021-02-10T04:35:02.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:02.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:02.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: -10T04:35:01.718722+0000 osd.2 (osd 2021-02-10T04:35:02.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: .2) 1228 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:02.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:35:02.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: 02-10T04:35:01.718728+0000 2021-02-10T04:35:02.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: osd.2 (osd.2) 1229 : cluster [WRN] 2021-02-10T04:35:02.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:02.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: cluster 2021-02-10T04:35:01.718734+0000 2021-02-10T04:35:02.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: osd.2 (osd.2) 1230 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:02.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:35:02.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: 04:35:01.718739+0000 osd.2 (osd.2) 1231 : cluster 2021-02-10T04:35:02.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:02.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: cluster 2021-02-10T04:35:02.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: 2021-02-10T04:35:01.718745+0000 osd.2 (osd.2) 1232 : cluster 2021-02-10T04:35:02.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:02.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: cluster 2021-02-10T04:35:01.718751+0000 osd.2 (osd.2) 1233 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:02.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: cluster 2021-02-10T04:35:01.718757+0000 2021-02-10T04:35:02.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: osd.2 (osd.2) 1234 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:02.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:35:02.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: :35:01.718763+0000 osd.2 (osd.2) 1235 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:02.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:35:02.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: 02-10T04:35:01.718769+0000 osd.2 ( 2021-02-10T04:35:02.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: osd.2) 1236 : cluster [WRN] 2021-02-10T04:35:02.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:02.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:35:02.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: -02-10T04:35:02.095158+0000 2021-02-10T04:35:02.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: mgr.y (mgr.14140 2021-02-10T04:35:02.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: ) 580 : cluster [DBG] 2021-02-10T04:35:02.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:02 smithi164 conmon[29288]: pgmap v876: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 395 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.7 MiB/s wr, 10 op/s 2021-02-10T04:35:02.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[49106]: debug 2021-02-10T04:35:02.705+0000 7f77a3e94700 -1 osd.2 485 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:35:02.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: cluster 2021-02-10T04:35:02.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:35:02.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: 04:35:01.135834+0000 osd.4 (osd.4) 685 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:02.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: cluster 2021-02-10T04:35:02.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: 2021-02 2021-02-10T04:35:02.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: -10T04:35:01 2021-02-10T04:35:02.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: .135840+0000 osd.4 (osd. 2021-02-10T04:35:02.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: 4) 686 : cluster [WRN] 2021-02-10T04:35:02.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:02.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:35:02.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: -02-10T04:35:01 2021-02-10T04:35:02.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: .135844+0000 osd.4 (osd. 2021-02-10T04:35:02.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: 4) 687 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:02.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: 2021-02-10T04:35:02.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: cluster 2021-02-10T04:35:02.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: 2021-02-10T04:35:02.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: 2021 2021-02-10T04:35:02.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: - 2021-02-10T04:35:02.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: 02 2021-02-10T04:35:02.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: - 2021-02-10T04:35:02.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: 10 2021-02-10T04:35:02.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: T04:35:01.135847 2021-02-10T04:35:02.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 688 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:02.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: cluster 2021-02-10T04:35:02.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: 2021-02 2021-02-10T04:35:02.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: -10T04:35:01.590970+0000 mon.a 2021-02-10T04:35:02.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: (mon.0) 1296 : cluster [DBG] osdmap e484: 8 total, 8 up, 8 in 2021-02-10T04:35:02.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: 2021-02-10T04:35:02.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:35:02.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: 10T04:35:01.718712+0000 osd.2 (osd 2021-02-10T04:35:02.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: .2) 1227 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:02.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:02.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: 02-10T04:35:01.718722+0000 osd.2 ( 2021-02-10T04:35:02.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: osd.2) 1228 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:02.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:35:02.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: 10T04:35:01.718728+0000 osd.2 ( 2021-02-10T04:35:02.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: osd.2) 1229 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:02.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:35:02.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: -02-10T04:35: 2021-02-10T04:35:02.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: 01.718734+0000 osd.2 (osd 2021-02-10T04:35:02.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: .2) 1230 : cluster [WRN] 2021-02-10T04:35:02.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:02.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:02.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: 02-10T04:35: 2021-02-10T04:35:02.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: 01.718739+0000 osd.2 (osd 2021-02-10T04:35:02.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: .2) 1231 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:02.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: 2021-02-10T04:35:02.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: cluster 2021-02-10T04:35:02.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: 2021-02-10T04:35:01.718745+0000 osd.2 (osd. 2021-02-10T04:35:02.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: 2) 1232 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:02.953 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:35:02.953 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: 10T04:35:01.718751+0000 osd.2 (osd.2) 1233 : cluster 2021-02-10T04:35:02.953 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:02.953 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: cluster 2021-02-10T04:35:01.718757 2021-02-10T04:35:02.953 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1234 : cluster [WRN] 2021-02-10T04:35:02.953 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:02.953 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: cluster 2021-02-10T04:35:02.954 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: 2021-02-10T04:35:01.718763+0000 osd.2 (osd.2 2021-02-10T04:35:02.954 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: ) 1235 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:02.954 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:35:02.954 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: T04:35:01.718769+0000 osd.2 (osd 2021-02-10T04:35:02.954 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: .2) 1236 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:02.954 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:35:02.954 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[29319]: :35:02.095158+0000 mgr.y (mgr.14140) 580 : cluster [DBG] pgmap v876: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 395 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.7 MiB/s wr, 10 op/s 2021-02-10T04:35:02.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: cluster 2021-02-10T04:35:02.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: 2021-02-10T04:35:02.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: 2021 2021-02-10T04:35:02.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: - 2021-02-10T04:35:02.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: 02 2021-02-10T04:35:02.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: -10T04:35:01.135834+0000 2021-02-10T04:35:02.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: osd.4 (osd.4) 685 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:02.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: cluster 2021-02-10T04:35 2021-02-10T04:35:02.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: :01.135840+0000 osd.4 (osd.4) 686 2021-02-10T04:35:02.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:02.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:35:02.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: :35:01.135844+0000 osd.4 (osd. 2021-02-10T04:35:02.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: 4) 687 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:02.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:02.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: 02-10T04:35:01.135847+0000 2021-02-10T04:35:02.958 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: osd.4 (osd.4) 688 : cluster [WRN] 2021-02-10T04:35:02.958 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:02.958 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: cluster 2021-02-10T04:35: 2021-02-10T04:35:02.958 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: 01.590970+0000 mon.a (mon.0) 1296 : cluster [DBG] 2021-02-10T04:35:02.958 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: osdmap e484: 8 total, 8 up, 8 in 2021-02-10T04:35:02.958 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: cluster 2021-02-10T04:35:02.959 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: 2021-02-10T04:35:01.718712+0000 osd.2 (osd.2) 2021-02-10T04:35:02.959 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: 1227 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:02.959 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: cluster 2021-02-10T04:35:01.718722 2021-02-10T04:35:02.959 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1228 : cluster [WRN] 2021-02-10T04:35:02.959 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:02.959 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:35:02.959 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: 10T04:35:01.718728+0000 osd.2 (osd.2) 1229 : cluster [WRN] 2021-02-10T04:35:02.960 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:02.960 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:35:02.960 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: -02-10T04:35:01.718734+0000 osd.2 (osd. 2021-02-10T04:35:02.960 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: 2) 1230 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:02.960 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:35:02.960 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: 04:35:01.718739+0000 osd.2 (osd 2021-02-10T04:35:02.961 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: .2) 1231 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:02.961 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:35:02.961 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: 10T04:35:01.718745+0000 2021-02-10T04:35:02.961 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: osd.2 (osd.2) 1232 : cluster [WRN] 2021-02-10T04:35:02.961 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:02.961 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: cluster 2021-02-10T04:35:02.961 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: 2021-02-10T04:35:01.718751+0000 osd.2 ( 2021-02-10T04:35:02.961 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: osd.2) 1233 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:02.962 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:35:02.962 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: -10T04:35:01.718757+0000 2021-02-10T04:35:02.962 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: osd.2 (osd.2) 1234 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:02.962 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: 2021-02-10T04:35:02.962 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:35:02.962 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: 10T04:35:01.718763+0000 osd.2 (osd.2) 1235 2021-02-10T04:35:02.962 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:02.962 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:35:02.963 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: 35:01.718769+0000 osd.2 (osd.2 2021-02-10T04:35:02.963 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: ) 1236 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:02.963 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:35:02.963 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: 10T04:35:02.095158+0000 mgr.y 2021-02-10T04:35:02.963 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: (mgr.14140) 580 : cluster [DBG] pgmap v876: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 395 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.7 MiB/s wr, 10 op/s 2021-02-10T04:35:02.963 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:02 smithi085 conmon[35961]: 2021-02-10T04:35:03.153 INFO:tasks.rados.rados.0.smithi085.stdout:1864: done (13 left) 2021-02-10T04:35:03.154 INFO:tasks.rados.rados.0.smithi085.stdout:1865: done (12 left) 2021-02-10T04:35:03.154 INFO:tasks.rados.rados.0.smithi085.stdout:1867: done (11 left) 2021-02-10T04:35:03.154 INFO:tasks.rados.rados.0.smithi085.stdout:1869: done (10 left) 2021-02-10T04:35:03.154 INFO:tasks.rados.rados.0.smithi085.stdout:1870: read oid 43 snap 166 2021-02-10T04:35:03.154 INFO:tasks.rados.rados.0.smithi085.stdout:1870: expect (ObjNum 491 snap 141 seq_num 491) 2021-02-10T04:35:03.154 INFO:tasks.rados.rados.0.smithi085.stdout:1871: snap_remove snap 195 2021-02-10T04:35:03.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[33677]: debug 2021-02-10T04:35:03.067+0000 7fbec19a1700 -1 osd.4 485 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:35:03.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: cluster 2021-02-10T04:35:02.092573+0000 2021-02-10T04:35:03.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: osd.4 (osd.4) 689 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:03.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: cluster 2021-02-10T04:35:02.092583+0000 osd.4 (osd.4) 690 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:03.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: 2021-02-10T04:35:03.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: cluster 2021-02-10T04:35:02 2021-02-10T04:35:03.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: .092590+0000 osd.4 (osd.4) 691 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:03.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: cluster 2021-02-10T04:35 2021-02-10T04:35:03.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: :02.092596+0000 osd.4 (osd.4) 692 : cluster [WRN] 2021-02-10T04:35:03.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:03.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: cluster 2021-02-10T04:35:03.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: 2021-02-10T04:35:02.600766+0000 mon.a (mon.0) 2021-02-10T04:35:03.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: 1297 : cluster [DBG] osdmap e485: 8 total, 8 up, 8 in 2021-02-10T04:35:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: cluster 2021-02-10T04:35 2021-02-10T04:35:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: :02.706831+0000 osd.2 (osd.2) 1237 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: -10T04:35:02.706838+0000 osd.2 (osd 2021-02-10T04:35:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: .2) 1238 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:35:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: 04:35:02.706845+0000 osd.2 (osd 2021-02-10T04:35:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: .2) 1239 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: 2021-02-10T04:35:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: -10T04:35 2021-02-10T04:35:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: :02.706851+0000 osd.2 ( 2021-02-10T04:35:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: osd.2) 1240 : cluster [WRN] 2021-02-10T04:35:03.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:03.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: cluster 2021-02-10T04:35: 2021-02-10T04:35:03.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: 02.706858+0000 osd.2 (osd.2) 1241 2021-02-10T04:35:03.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:03.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: cluster 2021-02-10T04:35: 2021-02-10T04:35:03.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: 02.706865+0000 osd.2 (osd.2) 1242 : cluster 2021-02-10T04:35:03.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:03.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: cluster 2021-02-10T04:35:02.706870 2021-02-10T04:35:03.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1243 : cluster 2021-02-10T04:35:03.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:03.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:35:03.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: 02-10T04:35:02. 2021-02-10T04:35:03.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: 706876+0000 osd.2 (osd.2) 2021-02-10T04:35:03.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: 1244 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:03.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:35:03.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: 35:02.706882+0000 osd.2 (osd.2) 1245 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:03.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: cluster 2021-02-10T04:35 2021-02-10T04:35:03.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: :02.706888+0000 osd.2 (osd 2021-02-10T04:35:03.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: .2) 1246 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:03.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:35:03.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: -02-10T04:35: 2021-02-10T04:35:03.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: 02.798271+0000 osd.5 (osd 2021-02-10T04:35:03.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: .5) 65 : cluster [DBG] 2021-02-10T04:35:03.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: purged_snaps scrub starts 2021-02-10T04:35:03.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:35:03.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: 10T04:35:02.798438+0000 2021-02-10T04:35:03.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: osd.5 (osd.5) 66 2021-02-10T04:35:03.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:35:03.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: cluster 2021-02-10T04:35:03.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:35:03.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: 35:03.144979+0000 mon.a 2021-02-10T04:35:03.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: (mon.0) 1298 : cluster [WRN] 2021-02-10T04:35:03.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: Health check update: 14 slow ops, oldest one blocked for 332 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:35:03.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:35:03.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: 02-10T04:35: 2021-02-10T04:35:03.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: 03.149850+0000 mon.a (mon 2021-02-10T04:35:03.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: .0) 1299 : cluster [DBG] 2021-02-10T04:35:03.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:03 smithi164 conmon[29288]: osdmap e486: 8 total, 8 up, 8 in 2021-02-10T04:35:03.942 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[49106]: debug 2021-02-10T04:35:03.942 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[49106]: 2021-02-10T04:35:03.728+0000 7f77a3e94700 -1 osd.2 486 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:35:03.942 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[49106]: 2021-02-10T04:35:03.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: cluster 2021-02-10T04:35:02.092573 2021-02-10T04:35:03.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 689 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:03.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:35:03.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: -02-10T04:35:02.092583+0000 osd.4 (osd.4) 690 : cluster 2021-02-10T04:35:03.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:03.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:35:03.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: 10T04:35:02.092590+0000 osd.4 (osd 2021-02-10T04:35:03.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: .4) 691 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:03.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: 2021-02-10T04:35:03.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:35:03.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: -02-10T04:35:02.092596+0000 2021-02-10T04:35:03.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: osd.4 (osd.4) 692 : cluster [WRN] 2021-02-10T04:35:03.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:03.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: cluster 2021-02-10T04:35:03.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:35:03.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: 10T04:35:02.600766+0000 mon.a (mon.0) 1297 : cluster 2021-02-10T04:35:03.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: [DBG] osdmap e485: 8 total, 8 up, 8 in 2021-02-10T04:35:03.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: cluster 2021-02-10T04:35:02.706831+0000 osd.2 ( 2021-02-10T04:35:03.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: osd.2) 1237 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:03.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: cluster 2021-02-10T04:35:02.706838+0000 2021-02-10T04:35:03.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: osd.2 (osd.2) 1238 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:03.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:35:03.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: 04:35:02.706845+0000 osd.2 (osd.2) 2021-02-10T04:35:03.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: 1239 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:03.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:03.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: 02-10T04:35:02.706851+0000 2021-02-10T04:35:03.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: osd.2 (osd.2) 1240 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:03.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: cluster 2021-02-10T04:35:02 2021-02-10T04:35:03.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: .706858+0000 osd.2 (osd.2) 1241 : cluster [WRN] 2021-02-10T04:35:03.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:03.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: cluster 2021-02-10T04:35:02 2021-02-10T04:35:03.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: .706865+0000 osd.2 (osd.2) 1242 : cluster 2021-02-10T04:35:03.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:03.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: cluster 2021-02-10T04:35:03.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: 2021-02-10T04:35:02.706870+0000 osd.2 (osd.2) 1243 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:03.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:35:03.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: -02-10T04:35:02.706876+0000 osd.2 (osd.2) 1244 : cluster 2021-02-10T04:35:03.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:03.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: cluster 2021-02-10T04:35:03.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:35:03.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: 04:35:02.706882+0000 2021-02-10T04:35:03.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: osd.2 (osd.2) 1245 : 2021-02-10T04:35:03.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:03.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: cluster 2021-02-10T04:35:03.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:35:03.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: 04:35:02.706888+0000 osd.2 (osd.2) 2021-02-10T04:35:03.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: 1246 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:03.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:03.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: 02-10T04:35:02.798271+0000 osd.5 ( 2021-02-10T04:35:03.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: osd.5) 65 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:35:03.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: 2021-02-10T04:35:03.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:35:03.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: 04:35:02.798438+0000 osd.5 (osd.5 2021-02-10T04:35:03.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: ) 66 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:35:03.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:35:03.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: -10T04:35:03.144979+0000 mon.a ( 2021-02-10T04:35:03.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: mon.0) 1298 : cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 332 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:35:03.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: 2021-02-10T04:35:03.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: cluster 2021-02-10T04:35:03.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: 2021-02-10T04:35: 2021-02-10T04:35:03.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: 03.149850+0000 mon.a (mon.0 2021-02-10T04:35:03.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[35961]: ) 1299 : cluster [DBG] osdmap e486: 8 total, 8 up, 8 in 2021-02-10T04:35:03.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: cluster 2021-02-10T04:35:02.092573+0000 osd.4 (osd.4) 689 : cluster 2021-02-10T04:35:03.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:03.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: cluster 2021-02-10T04:35:02.092583+0000 osd.4 (osd.4) 690 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: cluster 2021-02-10T04:35:02.092590+0000 osd.4 (osd.4) 691 : cluster [WRN] 2021-02-10T04:35:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: cluster 2021-02-10T04:35:02.092596+0000 osd.4 2021-02-10T04:35:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: (osd.4) 692 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: cluster 2021-02-10T04:35 2021-02-10T04:35:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: :02.600766+0000 mon.a (mon.0) 1297 : 2021-02-10T04:35:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: cluster [DBG] osdmap e485: 8 total, 8 up, 8 in 2021-02-10T04:35:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:35:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: :35:02.706831+0000 osd.2 (osd.2) 2021-02-10T04:35:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: 1237 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: cluster 2021-02-10T04:35:02. 2021-02-10T04:35:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: 706838+0000 osd.2 (osd.2) 1238 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: 2021-02-10T04:35:03.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: cluster 2021-02-10T04:35:02. 2021-02-10T04:35:03.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: 706845+0000 osd.2 (osd.2) 1239 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:03.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:35:03.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: -02-10T04:35: 2021-02-10T04:35:03.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: 02.706851+0000 osd.2 (osd 2021-02-10T04:35:03.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: .2) 1240 : cluster [WRN] 2021-02-10T04:35:03.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:03.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: cluster 2021-02-10T04:35 2021-02-10T04:35:03.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: :02.706858+0000 osd.2 (osd 2021-02-10T04:35:03.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: .2) 1241 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:03.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:35:03.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: -02-10T04:35:02. 2021-02-10T04:35:03.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: 706865+0000 osd.2 (osd.2) 2021-02-10T04:35:03.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: 1242 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:03.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:03.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: 02-10T04:35:02 2021-02-10T04:35:03.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: .706870+0000 osd.2 (osd. 2021-02-10T04:35:03.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: 2) 1243 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:03.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:35:03.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: T04:35:02.706876+0000 osd.2 ( 2021-02-10T04:35:03.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: osd.2) 1244 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:03.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: cluster 2021-02-10T04:35:02.706882 2021-02-10T04:35:03.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1245 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:03.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: cluster 2021-02-10T04:35:02.706888+0000 osd.2 (osd 2021-02-10T04:35:03.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: .2) 1246 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:03.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:03.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: 02-10T04:35:02.798271+0000 osd.5 (osd.5) 65 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:35:03.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: cluster 2021-02-10T04:35:02 2021-02-10T04:35:03.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: .798438+0000 osd.5 (osd.5) 66 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:35:03.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: cluster 2021-02-10T04:35:03.144979+0000 2021-02-10T04:35:03.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: mon.a (mon.0) 1298 : cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 332 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:35:03.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: cluster 2021-02-10T04:35: 2021-02-10T04:35:03.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:03 smithi085 conmon[29319]: 03.149850+0000 mon.a (mon.0) 1299 : cluster [DBG] osdmap e486: 8 total, 8 up, 8 in 2021-02-10T04:35:04.076 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:35:04.104 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:35:04.160 INFO:tasks.rados.rados.0.smithi085.stdout:1871: done (11 left) 2021-02-10T04:35:04.161 INFO:tasks.rados.rados.0.smithi085.stdout:1872: delete oid 50 current snap is 203 2021-02-10T04:35:04.161 INFO:tasks.rados.rados.0.smithi085.stdout:1870: expect (ObjNum 491 snap 141 seq_num 491) 2021-02-10T04:35:04.321 INFO:tasks.rados.rados.0.smithi085.stdout:1866: finishing copy_from to smithi08575397-40 2021-02-10T04:35:04.321 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 918 (ObjNum 660 snap 202 seq_num 660) dirty exists 2021-02-10T04:35:04.321 INFO:tasks.rados.rados.0.smithi085.stdout:1866: finishing copy_from racing read to smithi08575397-40 2021-02-10T04:35:04.322 INFO:tasks.rados.rados.0.smithi085.stdout:1866: done (11 left) 2021-02-10T04:35:04.322 INFO:tasks.rados.rados.0.smithi085.stdout:1870: done (10 left) 2021-02-10T04:35:04.322 INFO:tasks.rados.rados.0.smithi085.stdout:1872: done (9 left) 2021-02-10T04:35:04.322 INFO:tasks.rados.rados.0.smithi085.stdout:1873: rollback oid 42 current snap is 203 2021-02-10T04:35:04.322 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 42 to 190 2021-02-10T04:35:04.322 INFO:tasks.rados.rados.0.smithi085.stdout:1874: snap_remove snap 166 2021-02-10T04:35:04.393 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[40892]: debug 2021-02-10T04:35:04.183+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:35:04.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[33677]: debug 2021-02-10T04:35:04.019+0000 7fbec19a1700 -1 osd.4 486 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:35:04.432 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:35:04.538 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:35:04.574 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:35:04.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: cluster 2021-02-10T04:35:03.068602+0000 osd.4 (osd 2021-02-10T04:35:04.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: .4) 693 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:04.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: cluster 2021-02-10T04:35:03.068607+0000 osd.4 (osd.4) 694 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:04.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: cluster 2021-02-10T04:35:03.068611+0000 osd.4 (osd.4) 2021-02-10T04:35:04.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: 695 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:04.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:35:04.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: -02-10T04:35:03.068615+0000 osd.4 (osd.4) 696 : 2021-02-10T04:35:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: cluster 2021-02-10T04:35:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: 2021-02-10T04:35:03.730062+0000 2021-02-10T04:35:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: osd.2 (osd.2) 1247 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:35:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: 02-10T04:35:03.730071+0000 osd.2 2021-02-10T04:35:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: (osd.2) 1248 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:35:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: 04:35:03.730078+0000 osd.2 (osd.2) 1249 : cluster 2021-02-10T04:35:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: cluster 2021-02-10T04:35:03.730084+0000 2021-02-10T04:35:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: osd.2 (osd.2) 1250 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:04.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: cluster 2021-02-10T04:35 2021-02-10T04:35:04.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: :03.730089+0000 osd.2 (osd.2) 1251 : cluster 2021-02-10T04:35:04.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:04.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:35:04.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: :35:03 2021-02-10T04:35:04.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: .730094+0000 2021-02-10T04:35:04.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: osd.2 (osd.2) 1252 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:04.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:35:04.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: 10T04:35:03.730102+0000 osd.2 (osd. 2021-02-10T04:35:04.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: 2) 1253 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:04.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:35:04.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: T04:35:03.730105+0000 osd.2 ( 2021-02-10T04:35:04.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: osd.2) 1254 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:04.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: cluster 2021-02-10T04:35: 2021-02-10T04:35:04.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: 03.730110+0000 osd.2 (osd.2) 1255 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:04.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: cluster 2021-02-10T04:35:04.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: 2021-02-10T04:35:03. 2021-02-10T04:35:04.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: 730116+0000 osd.2 (osd.2) 1256 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:04.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: cluster 2021-02-10T04:35:04.020211 2021-02-10T04:35:04.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 697 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:04.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: cluster 2021-02-10T04:35:04.020222 2021-02-10T04:35:04.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 698 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:04.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: cluster 2021-02-10T04:35:04.020228+0000 2021-02-10T04:35:04.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: osd.4 (osd.4) 699 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:04.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: cluster 2021-02-10T04:35:04.020249+0000 osd.4 2021-02-10T04:35:04.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: (osd.4) 700 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:04.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: cluster 2021-02-10T04:35:04.095597+0000 mgr.y 2021-02-10T04:35:04.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: (mgr.14140) 581 : cluster [DBG] pgmap v879: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 393 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 636 KiB/s wr, 4 op/s 2021-02-10T04:35:04.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: cluster 2021-02-10T04:35: 2021-02-10T04:35:04.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: 04.160776+0000 mon.a (mon.0) 1300 : cluster 2021-02-10T04:35:04.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:04 smithi164 conmon[29288]: [DBG] osdmap e487: 8 total, 8 up, 8 in 2021-02-10T04:35:04.945 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[49106]: debug 2021-02-10T04:35:04.763+0000 7f77a3e94700 -1 osd.2 487 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:35:04.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: cluster 2021-02-10T04:35:03. 2021-02-10T04:35:04.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: 068602+0000 osd.4 (osd.4) 693 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:04.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: cluster 2021-02-10T04:35:04.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: 2021-02-10T04:35 2021-02-10T04:35:04.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: :03.068607+0000 osd.4 (osd.4) 694 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:04.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: cluster 2021-02-10T04:35: 2021-02-10T04:35:04.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: 03.068611+0000 osd.4 (osd.4) 695 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:04.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:35:04.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: -10T04:35:03.068615+0000 osd.4 (osd.4) 696 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:04.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: cluster 2021-02-10T04:35:04.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: 2021-02-10T04:35:03.730062+0000 osd.2 (osd.2) 1247 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:04.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: 2021-02-10T04:35:04.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:35:04.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: :35:03.730071+0000 osd.2 (osd.2) 1248 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:04.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: cluster 2021-02-10T04:35:04.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:35:04.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: T04:35:03.730078+0000 osd.2 (osd 2021-02-10T04:35:04.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: .2) 1249 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:04.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: cluster 2021-02-10T04:35:04.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: 2021-02-10T04:35:03.730084+0000 osd.2 (osd. 2021-02-10T04:35:04.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: 2) 1250 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:04.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: cluster 2021-02-10T04:35:03.730089+0000 osd.2 (osd. 2021-02-10T04:35:04.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: 2) 1251 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:04.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:35:04.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: 10T04:35:03.730094+0000 osd.2 (osd.2) 1252 : cluster 2021-02-10T04:35:04.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:04.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:35:04.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: -02-10T04:35:03.730102+0000 osd.2 (osd.2) 1253 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:04.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: cluster 2021-02-10T04:35:03.730105 2021-02-10T04:35:04.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1254 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:04.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: cluster 2021-02-10T04:35: 2021-02-10T04:35:04.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: 03.730110+0000 osd.2 (osd.2) 1255 : cluster [WRN] 2021-02-10T04:35:04.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:04.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: cluster 2021-02-10T04:35:04.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: 2021-02-10T04:35:03.730116+0000 osd.2 (osd.2) 1256 : 2021-02-10T04:35:04.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:04.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: cluster 2021-02-10T04:35:04.020211+0000 osd.4 (osd. 2021-02-10T04:35:04.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: 4) 697 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:04.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:35:04.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: T04:35:04.020222+0000 osd.4 (osd.4) 698 : cluster 2021-02-10T04:35:04.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:04.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:35:04.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: :35:04.020228+0000 osd.4 (osd.4) 699 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:04.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: cluster 2021-02-10T04:35:04.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: 2021-02-10T04:35:04.020249+0000 2021-02-10T04:35:04.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: osd.4 (osd.4) 700 : cluster [WRN] 2021-02-10T04:35:04.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:04.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: cluster 2021-02-10T04:35:04.095597+0000 mgr.y (mgr 2021-02-10T04:35:04.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: .14140) 581 : cluster [DBG] pgmap v879: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 393 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 636 KiB/s wr, 4 op/s 2021-02-10T04:35:04.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:35:04.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: -02-10T04:35:04.160776+0000 mon.a (mon.0 2021-02-10T04:35:04.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[35961]: ) 1300 : cluster [DBG] osdmap e487: 8 total, 8 up, 8 in 2021-02-10T04:35:04.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: cluster 2021-02-10T04:35:04.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: 2021-02-10T04:35:04.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: 2021-02-10T04:35:03. 2021-02-10T04:35:04.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: 068602+0000 osd.4 (osd.4) 693 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:04.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: cluster 2021-02-10T04:35 2021-02-10T04:35:04.953 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: :03.068607+0000 osd.4 (osd.4) 694 : cluster 2021-02-10T04:35:04.953 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:04.953 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: cluster 2021-02-10T04:35: 2021-02-10T04:35:04.953 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: 03.068611+0000 osd.4 (osd.4) 695 : cluster 2021-02-10T04:35:04.953 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:04.953 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: cluster 2021-02-10T04:35:04.953 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: 2021-02-10T04:35:03.068615+0000 osd.4 (osd.4) 2021-02-10T04:35:04.953 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: 696 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:04.953 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: cluster 2021-02-10T04:35:04.953 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: 2021-02-10T04:35:03.730062 2021-02-10T04:35:04.953 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1247 : cluster [WRN] 2021-02-10T04:35:04.954 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:04.954 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:35:04.954 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: -10T04:35:03.730071+0000 osd.2 (osd 2021-02-10T04:35:04.954 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: .2) 1248 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:04.954 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: cluster 2021-02-10T04:35:04.954 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: 2021-02-10T04:35:03.730078 2021-02-10T04:35:04.954 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 2021-02-10T04:35:04.954 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: 1249 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:04.954 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: cluster 2021-02-10T04:35:04.954 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: 2021-02-10T04:35:03 2021-02-10T04:35:04.954 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: .730084+0000 osd.2 (osd.2) 1250 : cluster 2021-02-10T04:35:04.955 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:04.955 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:35:04.955 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: 10T04:35:03.730089 2021-02-10T04:35:04.955 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1251 : cluster 2021-02-10T04:35:04.955 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:04.955 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: cluster 2021-02-10T04:35:04.955 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: 2021-02-10T04:35: 2021-02-10T04:35:04.955 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: 03.730094+0000 osd.2 (osd.2) 1252 2021-02-10T04:35:04.955 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:04.955 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: cluster 2021-02-10T04:35:04.955 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: 2021-02 2021-02-10T04:35:04.955 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: -10T04:35:03.730102+0000 osd.2 2021-02-10T04:35:04.955 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: (osd.2) 1253 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:04.956 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:35:04.956 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: 10T04:35:03.730105+0000 osd.2 (osd.2) 2021-02-10T04:35:04.956 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: 1254 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:04.956 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: cluster 2021-02-10T04:35:04.956 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:35:04.956 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: :35:03.730110+0000 osd.2 (osd.2) 2021-02-10T04:35:04.956 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: 1255 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:04.956 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: cluster 2021-02-10T04:35:04.956 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:35:04.956 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: 04:35:03.730116+0000 2021-02-10T04:35:04.956 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: osd.2 (osd.2) 2021-02-10T04:35:04.956 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: 1256 : cluster [WRN] 2021-02-10T04:35:04.956 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:04.957 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: cluster 2021-02-10T04:35:04.957 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: 2021-02-10T04:35:04.957 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: 2021 2021-02-10T04:35:04.957 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: - 2021-02-10T04:35:04.957 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: 02-10T04 2021-02-10T04:35:04.957 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: :35:04.020211+0000 osd.4 (osd.4) 697 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:04.957 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: 2021-02-10T04:35:04.957 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: cluster 2021-02-10T04:35:04.957 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: 2021-02-10T04:35:04.020222 2021-02-10T04:35:04.958 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 698 : cluster [WRN] 2021-02-10T04:35:04.958 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:04.958 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:35:04.958 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: T04:35:04.020228+0000 osd.4 (osd 2021-02-10T04:35:04.958 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: .4) 699 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:04.958 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:35:04.958 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: -02-10T04:35:04.020249+0000 osd.4 2021-02-10T04:35:04.958 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: (osd.4) 700 : cluster 2021-02-10T04:35:04.958 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:04.958 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: cluster 2021-02-10T04:35:04.958 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: 2021-02 2021-02-10T04:35:04.958 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: -10T04:35:04 2021-02-10T04:35:04.959 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: .095597+0000 mgr.y (mgr. 2021-02-10T04:35:04.959 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: 14140) 581 : cluster [DBG] pgmap v879: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 393 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 636 KiB/s wr, 4 op/s 2021-02-10T04:35:04.959 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: 2021-02-10T04:35:04.959 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: cluster 2021-02-10T04:35:04.959 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:35:04.959 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: :35:04.160776+0000 mon.a ( 2021-02-10T04:35:04.959 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:04 smithi085 conmon[29319]: mon.0) 1300 : cluster [DBG] osdmap e487: 8 total, 8 up, 8 in 2021-02-10T04:35:05.161 INFO:tasks.rados.rados.0.smithi085.stdout:1874: done (10 left) 2021-02-10T04:35:05.161 INFO:tasks.rados.rados.0.smithi085.stdout:1875: delete oid 28 current snap is 203 2021-02-10T04:35:05.161 INFO:tasks.rados.rados.0.smithi085.stdout:1873: finishing rollback tid 0 to smithi08575397-42 2021-02-10T04:35:05.161 INFO:tasks.rados.rados.0.smithi085.stdout:1873: finishing rollback tid 1 to smithi08575397-42 2021-02-10T04:35:05.161 INFO:tasks.rados.rados.0.smithi085.stdout:1873: finishing rollback tid 2 to smithi08575397-42 2021-02-10T04:35:05.161 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 799 (ObjNum 521 snap 155 seq_num 521) dirty exists 2021-02-10T04:35:05.172 INFO:tasks.rados.rados.0.smithi085.stdout:1873: done (10 left) 2021-02-10T04:35:05.172 INFO:tasks.rados.rados.0.smithi085.stdout:1875: done (9 left) 2021-02-10T04:35:05.172 INFO:tasks.rados.rados.0.smithi085.stdout:1876: snap_remove snap 182 2021-02-10T04:35:05.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[33677]: debug 2021-02-10T04:35:05.008+0000 7fbec19a1700 -1 osd.4 487 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:35:05.872 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[49106]: debug 2021-02-10T04:35:05.873 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[49106]: 2021-02-10T04:35:05.804+0000 7f77a3e94700 -1 osd.2 488 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:35:05.873 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[49106]: 2021-02-10T04:35:05.873 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: cluster 2021-02-10T04:35:04.764693+0000 osd.2 ( 2021-02-10T04:35:05.873 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: osd.2) 1257 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:05.873 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: cluster 2021-02-10T04:35:04.764703+0000 osd.2 (osd.2 2021-02-10T04:35:05.873 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: ) 1258 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:05.873 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:35:05.873 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: -10T04:35:04.764709+0000 osd.2 (osd.2) 1259 : 2021-02-10T04:35:05.873 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:05.873 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: cluster 2021-02-10T04:35:05.873 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: 2021-02-10T04:35:04.764714+0000 osd.2 (osd 2021-02-10T04:35:05.874 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: .2) 1260 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:05.874 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:35:05.874 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: :35:04.764720+0000 osd.2 (osd.2) 1261 : 2021-02-10T04:35:05.874 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:05.874 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: cluster 2021-02-10T04:35 2021-02-10T04:35:05.874 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: :04.764724+0000 osd.2 (osd.2) 1262 2021-02-10T04:35:05.874 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:05.874 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:35:05.874 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: T04:35:04.764732+0000 osd.2 (osd 2021-02-10T04:35:05.874 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: .2) 1263 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:05.874 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: cluster 2021-02-10T04:35:04.764738+0000 2021-02-10T04:35:05.874 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: osd.2 (osd.2) 1264 : cluster [WRN] 2021-02-10T04:35:05.874 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:05.875 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:35:05.875 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: T04:35: 2021-02-10T04:35:05.875 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: 04.764742+0000 osd.2 (osd.2) 1265 : cluster 2021-02-10T04:35:05.875 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:05.875 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:05.875 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: 02-10T04:35:04.764751+0000 osd.2 2021-02-10T04:35:05.875 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: (osd.2) 1266 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:05.875 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:35:05.875 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: -02-10T04:35:05.008797 2021-02-10T04:35:05.875 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 701 : cluster [WRN] 2021-02-10T04:35:05.875 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:05.875 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: cluster 2021-02-10T04:35 2021-02-10T04:35:05.875 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: :05.008803+0000 osd.4 (osd.4 2021-02-10T04:35:05.876 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: ) 702 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:05.876 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:35:05.876 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: -02-10T04:35:05.008806 2021-02-10T04:35:05.876 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 703 : 2021-02-10T04:35:05.876 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:05.876 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: cluster 2021-02-10T04:35: 2021-02-10T04:35:05.876 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: 05.008811+0000 osd.4 (osd.4) 704 : cluster 2021-02-10T04:35:05.876 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:05.876 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: cluster 2021-02-10T04:35:05 2021-02-10T04:35:05.876 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: .160214+0000 mon.a (mon.0) 1301 2021-02-10T04:35:05.876 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[29319]: : cluster [DBG] osdmap e488: 8 total, 8 up, 8 in 2021-02-10T04:35:05.877 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: cluster 2021-02-10T04:35:04 2021-02-10T04:35:05.877 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: .764693+0000 osd.2 (osd.2) 1257 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:05.877 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: cluster 2021-02-10T04:35:05.877 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: 2021-02-10T04:35:04.764703+0000 osd.2 2021-02-10T04:35:05.877 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: (osd.2) 1258 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:05.877 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: 2021-02-10T04:35:05.877 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:05.877 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: 02-10T04:35:04.764709+0000 osd.2 (osd.2 2021-02-10T04:35:05.877 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: ) 1259 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:05.877 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: cluster 2021-02-10T04:35 2021-02-10T04:35:05.877 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: :04.764714+0000 osd.2 (osd.2) 1260 2021-02-10T04:35:05.878 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:05.878 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: cluster 2021-02-10T04:35:04 2021-02-10T04:35:05.878 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: .764720+0000 osd.2 (osd.2) 1261 : cluster 2021-02-10T04:35:05.878 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:05.878 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: cluster 2021-02-10T04:35:05.878 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: 2021-02-10T04:35:04.764724+0000 osd.2 (osd 2021-02-10T04:35:05.878 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: .2) 1262 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:05.878 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:35:05.878 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: 04:35:04.764732+0000 osd.2 (osd.2 2021-02-10T04:35:05.878 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: ) 1263 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:05.878 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:35:05.878 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: :35:04.764738+0000 osd.2 (osd.2) 2021-02-10T04:35:05.879 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: 1264 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:05.879 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:35:05.879 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: 10T04:35:04.764742 2021-02-10T04:35:05.879 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1265 : cluster [WRN] 2021-02-10T04:35:05.879 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:05.879 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: cluster 2021-02-10T04:35 2021-02-10T04:35:05.879 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: :04.764751+0000 osd.2 (osd.2) 1266 : 2021-02-10T04:35:05.879 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:05.879 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:35:05.879 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: 10T04:35:05. 2021-02-10T04:35:05.879 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: 008797+0000 osd.4 (osd.4) 701 2021-02-10T04:35:05.879 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:05.879 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:05.879 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: 02-10T04:35:05.008803 2021-02-10T04:35:05.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 702 2021-02-10T04:35:05.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:05.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:35:05.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: T04:35:05.008806+0000 osd.4 (osd. 2021-02-10T04:35:05.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: 4) 703 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:05.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: 2021-02-10T04:35:05.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: cluster 2021-02-10T04:35:05 2021-02-10T04:35:05.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: .008811+0000 osd.4 (osd.4) 704 : cluster 2021-02-10T04:35:05.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:05.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: cluster 2021-02-10T04:35 2021-02-10T04:35:05.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: :05.160214+0000 mon.a (mon.0) 1301 : 2021-02-10T04:35:05.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[35961]: cluster [DBG] osdmap e488: 8 total, 8 up, 8 in 2021-02-10T04:35:05.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: cluster 2021-02-10T04:35:04.764693 2021-02-10T04:35:05.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1257 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:05.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: cluster 2021-02-10T04:35:04.764703+0000 osd.2 (osd 2021-02-10T04:35:05.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: .2) 1258 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:05.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: cluster 2021-02-10T04:35:05.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: 2021-02-10T04:35:04.764709+0000 2021-02-10T04:35:05.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: osd.2 (osd.2) 1259 : cluster [WRN] 2021-02-10T04:35:05.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:05.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:35:05.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: T04:35:04.764714+0000 osd.2 2021-02-10T04:35:05.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: (osd.2) 1260 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:05.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: 2021-02-10T04:35:05.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:35:05.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: 04:35:04.764720+0000 osd.2 (osd 2021-02-10T04:35:05.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: .2) 1261 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:05.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: cluster 2021-02-10T04:35:05.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:35:05.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: 35:04.764724+0000 osd.2 (osd.2 2021-02-10T04:35:05.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: ) 1262 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:05.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: cluster 2021-02-10T04:35:05.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: 2021-02-10T04:35 2021-02-10T04:35:05.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: :04.764732+0000 osd.2 (osd 2021-02-10T04:35:05.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: .2) 1263 : cluster [WRN] 2021-02-10T04:35:05.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:05.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:35:05.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: 02-10T04:35:04 2021-02-10T04:35:05.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: .764738+0000 osd.2 (osd.2) 1264 : 2021-02-10T04:35:05.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:05.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:05.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: -10T04:35:04.764742 2021-02-10T04:35:05.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1265 : cluster 2021-02-10T04:35:05.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:05.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:35:05.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: 10T04:35:04.764751+0000 2021-02-10T04:35:05.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: osd.2 (osd.2) 1266 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:05.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: 2021-02-10T04:35:05.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: cluster 2021-02-10T04:35:05.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: 2021-02-10T04:35: 2021-02-10T04:35:05.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: 05.008797+0000 osd.4 (osd.4 2021-02-10T04:35:05.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: ) 701 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:05.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:35:05.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: -02-10T04:35: 2021-02-10T04:35:05.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: 05.008803+0000 osd.4 (osd.4) 702 2021-02-10T04:35:05.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:05.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:35:05.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: -02-10T04:35: 2021-02-10T04:35:05.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: 05.008806+0000 osd.4 (osd. 2021-02-10T04:35:05.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: 4) 703 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:05.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: 2021-02-10T04:35:05.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:05.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: -10T04:35:05. 2021-02-10T04:35:05.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: 008811+0000 osd.4 (osd.4) 704 : cluster 2021-02-10T04:35:05.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:05.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:05.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: -10T04:35:05.160214 2021-02-10T04:35:05.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1301 : cluster 2021-02-10T04:35:05.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[29288]: [DBG] osdmap e488: 8 total, 8 up, 8 in 2021-02-10T04:35:05.900 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.3 dump_historic_ops 2021-02-10T04:35:05.995 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:35:06.165 INFO:tasks.rados.rados.0.smithi085.stdout:1876: done (9 left) 2021-02-10T04:35:06.165 INFO:tasks.rados.rados.0.smithi085.stdout:1877: read oid 23 snap -1 2021-02-10T04:35:06.165 INFO:tasks.rados.rados.0.smithi085.stdout:1877: expect (ObjNum 628 snap 193 seq_num 628) 2021-02-10T04:35:06.165 INFO:tasks.rados.rados.0.smithi085.stdout:1878: snap_create 2021-02-10T04:35:06.169 INFO:tasks.rados.rados.0.smithi085.stdout:1877: expect (ObjNum 628 snap 193 seq_num 628) 2021-02-10T04:35:06.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[52836]: debug 2021-02-10T04:35:06.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[52836]: 2021-02-10T04:35:05.945+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:35:06.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:35:05 smithi085 conmon[52836]: 2021-02-10T04:35:06.200 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:35:06.232 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:35:06.242 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:35:06.320 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:05 smithi164 conmon[33677]: debug 2021-02-10T04:35:05.983+0000 7fbec19a1700 -1 osd.4 488 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:35:06.390 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:35:06.496 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:35:06.535 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:35:06.611 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[37263]: debug 2021-02-10T04:35:06.319+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:35:06.613 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: cluster 2021-02-10T04:35:05.806310+0000 osd.2 (osd.2 2021-02-10T04:35:06.613 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: ) 1267 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:06.614 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: cluster 2021-02-10T04:35:05.806320+0000 osd.2 (osd.2) 1268 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:06.614 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:35:06.614 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: -02-10T04:35:05.806326+0000 osd.2 (osd.2) 1269 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:06.614 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:06.614 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: -10T04:35:05.806332+0000 osd.2 (osd.2) 2021-02-10T04:35:06.614 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: 1270 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:06.614 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:35:06.614 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: 02-10T04:35:05.806339+0000 2021-02-10T04:35:06.614 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: osd.2 (osd.2) 1271 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:06.614 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:06.614 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: -10T04:35:05.806344+0000 osd.2 (osd.2) 2021-02-10T04:35:06.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: 1272 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:06.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: cluster 2021-02-10T04:35:06.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: 2021-02-10T04:35:05. 2021-02-10T04:35:06.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: 806351+0000 osd.2 (osd.2) 1273 : cluster 2021-02-10T04:35:06.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:06.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:35:06.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: 10T04:35:05.806357+0000 2021-02-10T04:35:06.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: osd.2 (osd.2) 1274 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:06.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: 2021-02-10T04:35:06.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: cluster 2021-02-10T04:35:06.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: 2021-02-10T04:35:05.806363+0000 osd.2 (osd.2) 1275 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:06.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: cluster 2021-02-10T04:35:06.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: 2021-02-10T04:35:05.806369+0000 osd.2 (osd.2) 1276 : cluster [WRN] 2021-02-10T04:35:06.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:06.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: cluster 2021-02-10T04:35:06.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: 2021-02-10T04:35:05.983813+0000 osd.4 (osd.4) 705 : cluster [WRN] 2021-02-10T04:35:06.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:06.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: cluster 2021-02-10T04:35:06.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: 2021-02-10T04:35:05.983821+0000 osd.4 (osd.4) 706 : cluster [WRN] 2021-02-10T04:35:06.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:06.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:35:06.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: 02-10T04:35:05.983825+0000 osd.4 (osd.4) 707 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:06.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:35:06.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: :35:05.983829+0000 osd.4 (osd.4) 708 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:06.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: 2021-02-10T04:35:06.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:06.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: -10T04:35:06.096175+0000 mgr.y (mgr.14140) 582 : cluster 2021-02-10T04:35:06.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: [DBG] pgmap v882: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 396 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.8 MiB/s rd, 3.0 MiB/s wr, 16 op/s 2021-02-10T04:35:06.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:35:06.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: 04:35:06.162882+0000 mon.a (mon 2021-02-10T04:35:06.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[29288]: .0) 1302 : cluster [DBG] osdmap e489: 8 total, 8 up, 8 in 2021-02-10T04:35:06.617 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[29319]: cluster 2021-02-10T04:35:05.806310+0000 osd.2 (osd.2) 1267 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:06.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[29319]: cluster 2021-02-10T04:35:05.806320+0000 osd.2 (osd.2) 1268 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:06.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[29319]: cluster 2021-02-10T04:35:05.806326+0000 osd.2 (osd.2) 1269 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:06.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[29319]: cluster 2021-02-10T04:35:05.806332+0000 osd.2 (osd.2) 1270 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:06.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[29319]: cluster 2021-02-10T04:35:05.806339+0000 osd.2 (osd.2) 1271 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:06.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[29319]: cluster 2021-02-10T04:35:05.806344+0000 osd.2 (osd.2) 1272 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:06.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[29319]: cluster 2021-02-10T04:35:05.806351+0000 osd.2 (osd.2) 1273 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:06.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[29319]: cluster 2021-02-10T04:35:05.806357+0000 osd.2 (osd.2) 1274 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:06.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[29319]: cluster 2021-02-10T04:35:05.806363+0000 osd.2 (osd.2) 1275 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:06.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[29319]: cluster 2021-02-10T04:35:05.806369+0000 osd.2 (osd.2) 1276 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:06.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[29319]: cluster 2021-02-10T04:35:05.983813+0000 osd.4 (osd.4) 705 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:06.619 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[29319]: cluster 2021-02-10T04:35:05.983821+0000 osd.4 (osd.4) 706 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:06.619 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[29319]: cluster 2021-02-10T04:35:05.983825+0000 osd.4 (osd.4) 707 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:06.619 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[29319]: cluster 2021-02-10T04:35:05.983829+0000 osd.4 (osd.4) 708 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:06.619 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[29319]: cluster 2021-02-10T04:35:06.096175+0000 mgr.y (mgr.14140) 582 : cluster [DBG] pgmap v882: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 396 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.8 MiB/s rd, 3.0 MiB/s wr, 16 op/s 2021-02-10T04:35:06.619 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[29319]: cluster 2021-02-10T04:35:06.162882+0000 mon.a (mon.0) 1302 : cluster [DBG] osdmap e489: 8 total, 8 up, 8 in 2021-02-10T04:35:06.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[35961]: cluster 2021-02-10T04:35:05.806310+0000 osd.2 (osd.2) 1267 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:06.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[35961]: cluster 2021-02-10T04:35:05.806320+0000 osd.2 (osd.2) 1268 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:06.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[35961]: cluster 2021-02-10T04:35:05.806326+0000 osd.2 (osd.2) 1269 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:06.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[35961]: cluster 2021-02-10T04:35:05.806332+0000 osd.2 (osd.2) 1270 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:06.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[35961]: cluster 2021-02-10T04:35:05.806339 2021-02-10T04:35:06.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1271 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:06.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[35961]: cluster 2021-02-10T04:35:05.806344+0000 osd.2 (osd.2) 1272 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:06.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[35961]: cluster 2021-02-10T04:35:05.806351+0000 osd.2 (osd.2) 1273 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:06.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[35961]: cluster 2021-02-10T04:35:05.806357+0000 osd.2 (osd.2) 1274 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:06.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[35961]: cluster 2021-02-10T04:35:05.806363+0000 osd.2 (osd.2) 1275 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:06.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[35961]: cluster 2021-02-10T04:35:05.806369+0000 osd.2 (osd.2) 1276 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:06.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[35961]: cluster 2021-02-10T04:35:05.983813+0000 osd.4 (osd.4) 705 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:06.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[35961]: cluster 2021-02-10T04:35:05.983821+0000 osd.4 (osd.4) 706 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:06.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[35961]: cluster 2021-02-10T04:35:05.983825+0000 osd.4 (osd.4) 707 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:06.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[35961]: cluster 2021-02-10T04:35:05.983829+0000 osd.4 (osd.4) 708 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:06.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[35961]: cluster 2021-02-10T04:35:06.096175+0000 mgr.y (mgr.14140) 582 : cluster [DBG] pgmap v882: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 396 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.8 MiB/s rd, 3.0 MiB/s wr, 16 op/s 2021-02-10T04:35:06.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[35961]: cluster 2021-02-10T04:35:06.162882+0000 mon.a 2021-02-10T04:35:06.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[35961]: (mon.0) 1302 : cluster [DBG] osdmap e489: 8 total, 8 up, 8 in 2021-02-10T04:35:06.905 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[49106]: debug 2021-02-10T04:35:06.905 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[49106]: 2021-02-10T04:35:06.836+0000 7f77a3e94700 -1 osd.2 489 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:35:06.996 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:35:07.102 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:35:07.141 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:35:07.169 INFO:tasks.rados.rados.0.smithi085.stdout:1877: done (10 left) 2021-02-10T04:35:07.169 INFO:tasks.rados.rados.0.smithi085.stdout:1878: done (9 left) 2021-02-10T04:35:07.169 INFO:tasks.rados.rados.0.smithi085.stdout:1879: delete oid 8 current snap is 204 2021-02-10T04:35:07.175 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.3 to 1 (10000) 2021-02-10T04:35:07.175 INFO:tasks.rados.rados.0.smithi085.stdout:1879: done (9 left) 2021-02-10T04:35:07.175 INFO:tasks.rados.rados.0.smithi085.stdout:1880: write oid 42 current snap is 204 2021-02-10T04:35:07.175 INFO:tasks.rados.rados.0.smithi085.stdout:1880: seq_num 668 ranges {427076=789043,1706795=767562,2944912=282164} 2021-02-10T04:35:07.179 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is set 2021-02-10T04:35:07.180 INFO:tasks.rados.rados.0.smithi085.stdout:1880: writing smithi08575397-42 from 427076 to 1216119 tid 1 2021-02-10T04:35:07.188 INFO:tasks.rados.rados.0.smithi085.stdout:1880: writing smithi08575397-42 from 1706795 to 2474357 tid 2 2021-02-10T04:35:07.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[52836]: debug 2021-02-10T04:35:07.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:35:06 smithi085 conmon[52836]: 2021-02-10T04:35:06.948+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:35:07.194 INFO:tasks.rados.rados.0.smithi085.stdout:1880: writing smithi08575397-42 from 2944912 to 3227076 tid 3 2021-02-10T04:35:07.194 INFO:tasks.rados.rados.0.smithi085.stdout:1881: write oid 21 current snap is 204 2021-02-10T04:35:07.194 INFO:tasks.rados.rados.0.smithi085.stdout:1881: seq_num 669 ranges {443883=464292,1365277=511220,2572335=671548} 2021-02-10T04:35:07.203 INFO:tasks.rados.rados.0.smithi085.stdout:1881: writing smithi08575397-21 from 443883 to 908175 tid 1 2021-02-10T04:35:07.211 INFO:tasks.rados.rados.0.smithi085.stdout:1881: writing smithi08575397-21 from 1365277 to 1876497 tid 2 2021-02-10T04:35:07.221 INFO:tasks.rados.rados.0.smithi085.stdout:1881: writing smithi08575397-21 from 2572335 to 3243883 tid 3 2021-02-10T04:35:07.222 INFO:tasks.rados.rados.0.smithi085.stdout:1880: finishing write tid 1 to smithi08575397-42 2021-02-10T04:35:07.222 INFO:tasks.rados.rados.0.smithi085.stdout:1880: finishing write tid 2 to smithi08575397-42 2021-02-10T04:35:07.222 INFO:tasks.rados.rados.0.smithi085.stdout:1880: finishing write tid 3 to smithi08575397-42 2021-02-10T04:35:07.222 INFO:tasks.rados.rados.0.smithi085.stdout:1880: finishing write tid 4 to smithi08575397-42 2021-02-10T04:35:07.222 INFO:tasks.rados.rados.0.smithi085.stdout:1880: finishing write tid 5 to smithi08575397-42 2021-02-10T04:35:07.222 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 803 (ObjNum 668 snap 204 seq_num 668) dirty exists 2021-02-10T04:35:07.222 INFO:tasks.rados.rados.0.smithi085.stdout:1880: left oid 42 (ObjNum 668 snap 204 seq_num 668) 2021-02-10T04:35:07.223 INFO:tasks.rados.rados.0.smithi085.stdout:1881: finishing write tid 1 to smithi08575397-21 2021-02-10T04:35:07.223 INFO:tasks.rados.rados.0.smithi085.stdout:1881: finishing write tid 2 to smithi08575397-21 2021-02-10T04:35:07.223 INFO:tasks.rados.rados.0.smithi085.stdout:1880: done (10 left) 2021-02-10T04:35:07.223 INFO:tasks.rados.rados.0.smithi085.stdout:1882: snap_create 2021-02-10T04:35:07.229 INFO:tasks.rados.rados.0.smithi085.stdout:1881: finishing write tid 3 to smithi08575397-21 2021-02-10T04:35:07.229 INFO:tasks.rados.rados.0.smithi085.stdout:1881: finishing write tid 4 to smithi08575397-21 2021-02-10T04:35:07.229 INFO:tasks.rados.rados.0.smithi085.stdout:1881: finishing write tid 5 to smithi08575397-21 2021-02-10T04:35:07.229 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 997 (ObjNum 669 snap 204 seq_num 669) dirty exists 2021-02-10T04:35:07.229 INFO:tasks.rados.rados.0.smithi085.stdout:1881: left oid 21 (ObjNum 669 snap 204 seq_num 669) 2021-02-10T04:35:07.292 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:35:07.325 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:35:07.357 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:35:07.382 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:35:07.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:06 smithi164 conmon[33677]: debug 2021-02-10T04:35:06.934+0000 7fbec19a1700 -1 osd.4 489 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:35:07.411 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:35:07.443 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:35:07.470 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:35:07.498 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:35:07.500 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:35:07.500 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:35:07.500 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:35:07.500 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:35:07.500 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:35:07.500 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:35:07.500 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:35:07.500 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:35:07.526 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[52836]: debug 2021-02-10T04:35:07.254+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:35:07.717 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:35:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:35:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: 10T04:35:06.837820+0000 osd.2 (osd.2) 1277 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: cluster 2021-02-10T04:35:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: 2021-02-10T04:35:06.837832+0000 osd.2 (osd.2) 1278 2021-02-10T04:35:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:35:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: :35:06.837839+0000 osd.2 (osd.2) 2021-02-10T04:35:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: 1279 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:07.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:35:07.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: 04:35:06.837846+0000 osd.2 (osd.2 2021-02-10T04:35:07.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: ) 1280 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:07.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:07.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: -10T04:35:06.837851+0000 osd.2 2021-02-10T04:35:07.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: (osd.2) 1281 : cluster 2021-02-10T04:35:07.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:07.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: cluster 2021-02-10T04:35 2021-02-10T04:35:07.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: :06.837858+0000 osd.2 (osd.2) 1282 : cluster 2021-02-10T04:35:07.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:07.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:35:07.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: :35:06.837864+0000 osd.2 (osd.2) 1283 2021-02-10T04:35:07.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:07.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: cluster 2021-02-10T04:35:07.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: 2021-02-10T04:35: 2021-02-10T04:35:07.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: 06.837870+0000 osd.2 (osd.2) 1284 : 2021-02-10T04:35:07.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:07.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:35:07.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: :35:06.837875+0000 osd.2 (osd. 2021-02-10T04:35:07.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: 2) 1285 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:07.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:35:07.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: -02-10T04:35:06.837882 2021-02-10T04:35:07.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1286 : cluster [WRN] 2021-02-10T04:35:07.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:07.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:35:07.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: -02-10T04:35:06. 2021-02-10T04:35:07.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: 931241+0000 osd.2 (osd.2) 2021-02-10T04:35:07.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: 1287 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:35:07.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: cluster 2021-02-10T04:35:06.931494 2021-02-10T04:35:07.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1288 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:35:07.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:35:07.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: 02-10T04:35:06.935482+0000 osd.4 (osd.4) 709 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:07.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: 2021-02-10T04:35:07.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: cluster 2021-02-10T04:35:07.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: 2021-02-10T04:35:06.935492+0000 osd.4 (osd.4) 710 2021-02-10T04:35:07.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:07.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: cluster 2021-02-10T04:35:06.935498+0000 osd.4 ( 2021-02-10T04:35:07.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: osd.4) 711 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:07.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: cluster 2021-02-10T04:35:07.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: 2021-02-10T04:35:06.935506+0000 osd.4 ( 2021-02-10T04:35:07.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: osd.4) 712 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:07.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:35:07.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: 04:35:07.012714+0000 mon.c (mon.1) 56 : audit [INF] 2021-02-10T04:35:07.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/3019133229' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-02-10T04:35:07.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: audit 2021-02-10T04:35:07.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: 2021-02-10T04:35:07.013274+0000 mon.a (mon.0) 1303 : audit [INF] 2021-02-10T04:35:07.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-02-10T04:35:07.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: audit 2021-02-10T04:35:07.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: 2021-02-10T04:35:07.093368+0000 mon.c (mon.1) 57 : audit [INF] 2021-02-10T04:35:07.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/734501128' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:35:07.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: audit 2021 2021-02-10T04:35:07.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: -02-10T04:35:07.093975+0000 mon.a (mon 2021-02-10T04:35:07.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: .0) 1304 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:35:07.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:35:07.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: 02-10T04:35:07.158843+0000 2021-02-10T04:35:07.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: mon.a (mon.0) 1305 : cluster [WRN] 2021-02-10T04:35:07.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:35:07.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: audit 2021-02-10T04: 2021-02-10T04:35:07.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: 35:07.163841+0000 mon.a (mon.0) 1306 : audit 2021-02-10T04:35:07.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2021-02-10T04:35:07.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: audit 2021-02-10T04:35:07.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: 2021-02-10T04:35:07.163969+0000 mon.a (mon.0) 1307 : audit 2021-02-10T04:35:07.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:35:07.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: cluster 2021-02-10T04:35:07.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: 2021-02-10T04:35:07.164058+0000 mon.a (mon. 2021-02-10T04:35:07.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: 0) 1308 : cluster [DBG] osdmap e490: 8 total, 8 up, 8 in 2021-02-10T04:35:07.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:35:07.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: 10T04:35:07.255820+0000 2021-02-10T04:35:07.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: mon.a (mon.0) 1309 : audit [DBG] from='client.? 172.21.15.85:0/1283558562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:35:07.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:07 smithi164 conmon[29288]: 2021-02-10T04:35:07.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: cluster 2021-02-10T04:35:06.837820+0000 2021-02-10T04:35:07.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: osd.2 (osd.2) 1277 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:07.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: cluster 2021-02-10T04:35:06.837832+0000 osd.2 (osd.2) 1278 : cluster 2021-02-10T04:35:07.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:07.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:35:07.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: 35:06.837839+0000 osd.2 (osd.2) 1279 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:07.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:35:07.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: 10T04:35:06.837846+0000 osd.2 ( 2021-02-10T04:35:07.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: osd.2) 1280 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:07.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: cluster 2021-02-10T04:35:06. 2021-02-10T04:35:07.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: 837851+0000 osd.2 (osd.2) 1281 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:07.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: cluster 2021-02-10T04:35:06.837858+0000 osd.2 (osd.2) 1282 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:07.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:07.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: 02-10T04:35:06.837864+0000 osd.2 (osd.2) 1283 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:07.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: cluster 2021-02-10T04:35 2021-02-10T04:35:07.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: :06.837870+0000 osd.2 (osd.2) 1284 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:07.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: cluster 2021-02-10T04:35:06 2021-02-10T04:35:07.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: .837875+0000 osd.2 (osd.2) 1285 : cluster 2021-02-10T04:35:07.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:07.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: cluster 2021-02-10T04:35:06 2021-02-10T04:35:07.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: .837882+0000 osd.2 (osd.2) 1286 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:07.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:35:07.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: 04:35:06.931241+0000 osd.2 (osd.2) 1287 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:35:07.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: cluster 2021-02-10T04:35:06 2021-02-10T04:35:07.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: .931494+0000 osd.2 (osd.2) 1288 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:35:07.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: cluster 2021-02-10T04:35:06.935482+0000 2021-02-10T04:35:07.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: osd.4 (osd.4) 709 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:07.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:35:07.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: :35:06.935492+0000 osd.4 (osd. 2021-02-10T04:35:07.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: 4) 710 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:07.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: cluster 2021-02-10T04:35:07.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: 2021-02-10T04:35:07.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: 2021 2021-02-10T04:35:07.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: - 2021-02-10T04:35:07.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: 02 2021-02-10T04:35:07.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: - 2021-02-10T04:35:07.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: 10 2021-02-10T04:35:07.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: T 2021-02-10T04:35:07.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: 04 2021-02-10T04:35:07.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: : 2021-02-10T04:35:07.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: 35:06.935498+0000 2021-02-10T04:35:07.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: osd.4 (osd.4) 711 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:07.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:35:07.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: 04:35:06.935506+0000 osd.4 (osd.4 2021-02-10T04:35:07.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: ) 712 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:07.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: audit 2021-02-10T04:35:07.012714 2021-02-10T04:35:07.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: +0000 mon.c (mon.1) 56 : audit [INF] from='client.? 172.21.15.85:0/3019133229' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-02-10T04:35:07.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:35:07.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: - 2021-02-10T04:35:07.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: 10T04:35:07.013274 2021-02-10T04:35:07.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1303 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-02-10T04:35:07.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: audit 2021-02-10T04:35:07.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:35:07.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: 10T04:35:07.093368+0000 mon.c 2021-02-10T04:35:07.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: (mon.1) 57 : audit [INF] from='client.? 172.21.15.85:0/734501128' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:35:07.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: audit 2021 2021-02-10T04:35:07.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: -02-10T04 2021-02-10T04:35:07.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: :35:07.093975+0000 mon.a (mon.0 2021-02-10T04:35:07.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: ) 1304 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:35:07.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: cluster 2021-02-10T04:35:07.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:35:07.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: T04:35:07.158843+0000 mon.a (mon 2021-02-10T04:35:07.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: .0) 1305 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:35:07.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:35:07.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: -10T04:35:07.163841+0000 mon.a ( 2021-02-10T04:35:07.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: mon.0) 1306 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2021-02-10T04:35:07.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:35:07.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: 10T04:35:07.163969+0000 mon.a (mon. 2021-02-10T04:35:07.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: 0) 1307 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:35:07.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:07.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: 02-10T04:35:07.164058+0000 2021-02-10T04:35:07.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: mon.a (mon.0) 1308 : cluster [DBG] osdmap e490: 8 total, 8 up, 8 in 2021-02-10T04:35:07.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: 2021-02-10T04:35:07.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: audit 2021-02-10T04:35:07.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: 2021-02-10T04:35:07.255820+0000 2021-02-10T04:35:07.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[29319]: mon.a (mon.0) 1309 : audit [DBG] from='client.? 172.21.15.85:0/1283558562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:35:07.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: cluster 2021-02-10T04:35:06.837820 2021-02-10T04:35:07.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1277 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:07.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: cluster 2021-02-10T04:35:06 2021-02-10T04:35:07.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: .837832+0000 osd.2 (osd.2) 1278 : cluster 2021-02-10T04:35:07.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:07.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:35:07.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: -02-10T04:35:06. 2021-02-10T04:35:07.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: 837839+0000 osd.2 (osd.2) 1279 : cluster 2021-02-10T04:35:07.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:07.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:35:07.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: :35:06.837846+0000 osd.2 (osd.2) 2021-02-10T04:35:07.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: 1280 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:07.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:07.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: 02-10T04:35:06.837851+0000 osd.2 2021-02-10T04:35:07.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: (osd.2) 1281 : cluster 2021-02-10T04:35:07.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:07.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: cluster 2021-02-10T04:35:07.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: 2021-02-10T04:35:06 2021-02-10T04:35:07.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: .837858+0000 osd.2 (osd.2) 1282 : 2021-02-10T04:35:07.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:07.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:07.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: 02-10T04:35:06.837864+0000 osd.2 2021-02-10T04:35:07.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: (osd.2) 1283 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:07.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: 2021-02-10T04:35:07.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: cluster 2021-02-10T04:35:07.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: 2021-02-10T04:35:06. 2021-02-10T04:35:07.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: 837870+0000 osd.2 (osd.2) 1284 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:07.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: cluster 2021-02-10T04:35:07.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:35:07.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: 35:06.837875+0000 osd.2 (osd.2) 1285 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:07.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:35:07.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: -10T04:35:06.837882+0000 osd.2 (osd.2 2021-02-10T04:35:07.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: ) 1286 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:07.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:35:07.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: -10T04:35:06.931241+0000 osd.2 (osd 2021-02-10T04:35:07.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: .2) 1287 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:35:07.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:35:07.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: T04:35:06.931494+0000 osd.2 (osd.2) 1288 : cluster [DBG] 2021-02-10T04:35:07.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:35:07.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: cluster 2021-02-10T04:35:06.935482+0000 osd.4 (osd.4) 2021-02-10T04:35:07.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: 709 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:07.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:35:07.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: 35:06.935492+0000 osd.4 (osd.4 2021-02-10T04:35:07.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: ) 710 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:07.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:35:07.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: T04:35:06.935498+0000 osd.4 (osd.4) 2021-02-10T04:35:07.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: 711 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:07.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:35:07.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: -02-10T04:35:06.935506 2021-02-10T04:35:07.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 712 : cluster [WRN] 2021-02-10T04:35:07.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:07.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: audit 2021 2021-02-10T04:35:07.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: -02-10T04:35:07.012714+0000 mon.c (mon. 2021-02-10T04:35:07.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: 1) 56 : audit [INF] from='client.? 172.21.15.85:0/3019133229' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-02-10T04:35:07.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: audit 2021 2021-02-10T04:35:07.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: -02-10T04:35:07.013274+0000 2021-02-10T04:35:07.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: mon.a (mon.0) 1303 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-02-10T04:35:07.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: audit 2021-02-10T04:35:07. 2021-02-10T04:35:07.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: 093368+0000 mon.c (mon.1) 57 : audit [INF] from='client.? 172.21.15.85:0/734501128' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:35:07.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:35:07.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: -10T04:35:07.093975+0000 mon.a 2021-02-10T04:35:07.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: (mon.0) 1304 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:35:07.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:35:07.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: :35:07.158843+0000 mon.a (mon.0) 1305 2021-02-10T04:35:07.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:35:07.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:35:07.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: :35:07.163841+0000 mon.a ( 2021-02-10T04:35:07.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: mon.0) 1306 : audit [INF] 2021-02-10T04:35:07.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2021-02-10T04:35:07.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: audit 2021 2021-02-10T04:35:07.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: -02-10T04:35:07 2021-02-10T04:35:07.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: .163969+0000 mon.a (mon. 2021-02-10T04:35:07.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: 0) 1307 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:35:07.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: 2021-02-10T04:35:07.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:07.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: 02-10T04:35:07 2021-02-10T04:35:07.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: .164058+0000 mon.a (mon.0) 1308 : cluster [DBG] osdmap e490: 8 total, 8 up, 8 in 2021-02-10T04:35:07.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:35:07.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: -10T04:35:07.255820+0000 mon.a ( 2021-02-10T04:35:07.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:07 smithi085 conmon[35961]: mon.0) 1309 : audit [DBG] from='client.? 172.21.15.85:0/1283558562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:35:08.159 INFO:tasks.rados.rados.0.smithi085.stdout:1881: done (10 left) 2021-02-10T04:35:08.159 INFO:tasks.rados.rados.0.smithi085.stdout:1882: done (9 left) 2021-02-10T04:35:08.159 INFO:tasks.rados.rados.0.smithi085.stdout:1883: read oid 35 snap 202 2021-02-10T04:35:08.159 INFO:tasks.rados.rados.0.smithi085.stdout:1883: expect (ObjNum 85 snap 11 seq_num 85) 2021-02-10T04:35:08.159 INFO:tasks.rados.rados.0.smithi085.stdout:1884: copy_from oid 12 from oid 5 current snap is 205 2021-02-10T04:35:08.159 INFO:tasks.rados.rados.0.smithi085.stdout:1885: read oid 22 snap -1 2021-02-10T04:35:08.159 INFO:tasks.rados.rados.0.smithi085.stdout:1885: expect (ObjNum 654 snap 197 seq_num 654) 2021-02-10T04:35:08.160 INFO:tasks.rados.rados.0.smithi085.stdout:1886: snap_remove snap 179 2021-02-10T04:35:08.394 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:35:08 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:35:09.159 INFO:tasks.rados.rados.0.smithi085.stdout:1886: done (12 left) 2021-02-10T04:35:09.159 INFO:tasks.rados.rados.0.smithi085.stdout:1887: read oid 6 snap -1 2021-02-10T04:35:09.159 INFO:tasks.rados.rados.0.smithi085.stdout:1887: expect deleted 2021-02-10T04:35:09.159 INFO:tasks.rados.rados.0.smithi085.stdout:1888: read oid 38 snap -1 2021-02-10T04:35:09.159 INFO:tasks.rados.rados.0.smithi085.stdout:1888: expect (ObjNum 616 snap 186 seq_num 616) 2021-02-10T04:35:09.159 INFO:tasks.rados.rados.0.smithi085.stdout:1889: write oid 43 current snap is 205 2021-02-10T04:35:09.160 INFO:tasks.rados.rados.0.smithi085.stdout:1889: seq_num 670 ranges {460690=655894,1540112=771231,2716111=539778} 2021-02-10T04:35:09.162 INFO:tasks.rados.rados.0.smithi085.stdout:1889: writing smithi08575397-43 from 460690 to 1116584 tid 1 2021-02-10T04:35:09.170 INFO:tasks.rados.rados.0.smithi085.stdout:1889: writing smithi08575397-43 from 1540112 to 2311343 tid 2 2021-02-10T04:35:09.176 INFO:tasks.rados.rados.0.smithi085.stdout:1889: writing smithi08575397-43 from 2716111 to 3255889 tid 3 2021-02-10T04:35:09.176 INFO:tasks.rados.rados.0.smithi085.stdout:1890: snap_remove snap 196 2021-02-10T04:35:09.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:09 smithi085 conmon[29319]: audit 2021-02-10T04:35:09.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:09 smithi085 conmon[29319]: 2021-02-10T04:35:09.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:09 smithi085 conmon[29319]: 2021-02-10T04:35 2021-02-10T04:35:09.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:09 smithi085 conmon[29319]: :08.066013+0000 mgr.y (mgr.14140) 583 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:35:09.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:09 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:35:09.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:09 smithi085 conmon[29319]: 04:35:08.096625+0000 mgr.y (mgr. 2021-02-10T04:35:09.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:09 smithi085 conmon[29319]: 14140) 584 : cluster [DBG] pgmap v885: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 396 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.8 MiB/s rd, 2.4 MiB/s wr, 13 op/s 2021-02-10T04:35:09.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:09 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:09.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:09 smithi085 conmon[29319]: 02-10T04:35:08.154148+0000 mon.a (mon 2021-02-10T04:35:09.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:09 smithi085 conmon[29319]: .0) 1310 : cluster [DBG] osdmap e491: 8 total, 8 up, 8 in 2021-02-10T04:35:09.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:09 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:35:09.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:09 smithi085 conmon[29319]: :35:08.611104+0000 mon.a (mon.0) 2021-02-10T04:35:09.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:09 smithi085 conmon[29319]: 1311 : cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 337 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:35:09.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:09 smithi085 conmon[35961]: audit 2021-02-10T04:35:08.066013+0000 2021-02-10T04:35:09.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:09 smithi085 conmon[35961]: mgr.y (mgr.14140) 583 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:35:09.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:09 smithi085 conmon[35961]: cluster 2021-02-10T04:35:08. 2021-02-10T04:35:09.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:09 smithi085 conmon[35961]: 096625+0000 mgr.y (mgr.14140) 584 : cluster 2021-02-10T04:35:09.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:09 smithi085 conmon[35961]: [DBG] pgmap v885: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 396 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.8 MiB/s rd, 2.4 MiB/s wr, 13 op/s 2021-02-10T04:35:09.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:09 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:35:09.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:09 smithi085 conmon[35961]: 35:08.154148+0000 mon.a (mon.0 2021-02-10T04:35:09.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:09 smithi085 conmon[35961]: ) 1310 : cluster [DBG] osdmap e491: 8 total, 8 up, 8 in 2021-02-10T04:35:09.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:09 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:09.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:09 smithi085 conmon[35961]: 02-10T04:35:08.611104+0000 mon.a 2021-02-10T04:35:09.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:09 smithi085 conmon[35961]: (mon.0) 1311 : cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 337 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:35:09.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:09 smithi164 conmon[29288]: audit 2021-02-10T04:35:08.066013+0000 mgr.y (mgr.14140) 583 2021-02-10T04:35:09.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:09 smithi164 conmon[29288]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:35:09.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:09 smithi164 conmon[29288]: cluster 2021-02-10T04:35:08.096625+0000 mgr.y (mgr.14140) 584 : cluster [DBG] pgmap v885: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 396 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.8 MiB/s rd, 2.4 MiB/s wr, 13 op/s 2021-02-10T04:35:09.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:09 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:09.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:09 smithi164 conmon[29288]: -10T04:35:08.154148+0000 mon.a (mon.0) 1310 : cluster [DBG] osdmap e491: 8 total, 8 up, 8 in 2021-02-10T04:35:09.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:09 smithi164 conmon[29288]: 2021-02-10T04:35:09.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:09 smithi164 conmon[29288]: cluster 2021-02-10T04:35:09.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:09 smithi164 conmon[29288]: 2021-02-10 2021-02-10T04:35:09.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:09 smithi164 conmon[29288]: T04:35:08.611104+0000 2021-02-10T04:35:09.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:09 smithi164 conmon[29288]: mon.a (mon.0) 1311 : cluster 2021-02-10T04:35:09.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:09 smithi164 conmon[29288]: [WRN] Health check update: 14 slow ops, oldest one blocked for 337 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:35:10.173 INFO:tasks.rados.rados.0.smithi085.stdout:1890: done (15 left) 2021-02-10T04:35:10.173 INFO:tasks.rados.rados.0.smithi085.stdout:1891: snap_remove snap 199 2021-02-10T04:35:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:10 smithi085 conmon[29319]: cluster 2021-02-10T04:35:09.155986+0000 2021-02-10T04:35:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:10 smithi085 conmon[29319]: mon.a (mon.0) 1312 : cluster [DBG] osdmap e492: 8 total, 8 up, 8 in 2021-02-10T04:35:10.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:10 smithi085 conmon[35961]: cluster 2021-02-10T04:35:09.155986+0000 mon.a (mon.0) 1312 2021-02-10T04:35:10.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:10 smithi085 conmon[35961]: : cluster [DBG] osdmap e492: 8 total, 8 up, 8 in 2021-02-10T04:35:10.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:10 smithi164 conmon[29288]: cluster 2021-02-10T04:35:09.155986+0000 mon.a (mon.0) 1312 : cluster [DBG] osdmap e492: 8 total, 8 up, 8 in 2021-02-10T04:35:11.189 INFO:tasks.rados.rados.0.smithi085.stdout:1883: expect (ObjNum 85 snap 11 seq_num 85) 2021-02-10T04:35:11.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:11 smithi085 conmon[29319]: cluster 2021-02-10T04:35:10.097102+0000 mgr.y (mgr.14140) 585 : cluster 2021-02-10T04:35:11.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:11 smithi085 conmon[29319]: [DBG] pgmap v888: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 398 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.6 MiB/s rd, 852 KiB/s wr, 5 op/s 2021-02-10T04:35:11.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:11 smithi085 conmon[29319]: cluster 2021-02-10T04:35:10.172987+0000 mon.a (mon.0) 1313 : cluster [DBG] osdmap e493: 8 total, 8 up, 8 in 2021-02-10T04:35:11.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:11 smithi085 conmon[35961]: cluster 2021-02-10T04:35:11.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:11 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:35:11.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:11 smithi085 conmon[35961]: 10T04:35:10.097102+0000 mgr.y (mgr.14140) 585 : cluster [DBG] pgmap v888: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 398 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.6 MiB/s rd, 852 KiB/s wr, 5 op/s 2021-02-10T04:35:11.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:11 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:35:11.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:11 smithi085 conmon[35961]: 10T04:35:10.172987+0000 mon.a (mon. 2021-02-10T04:35:11.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:11 smithi085 conmon[35961]: 0) 1313 : cluster [DBG] osdmap e493: 8 total, 8 up, 8 in 2021-02-10T04:35:11.454 INFO:tasks.rados.rados.0.smithi085.stdout:1885: expect (ObjNum 654 snap 197 seq_num 654) 2021-02-10T04:35:11.596 INFO:tasks.rados.rados.0.smithi085.stdout:1884: finishing copy_from to smithi08575397-12 2021-02-10T04:35:11.596 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 673 (ObjNum 663 snap 202 seq_num 663) dirty exists 2021-02-10T04:35:11.597 INFO:tasks.rados.rados.0.smithi085.stdout:1884: finishing copy_from racing read to smithi08575397-12 2021-02-10T04:35:11.597 INFO:tasks.rados.rados.0.smithi085.stdout:1883: done (15 left) 2021-02-10T04:35:11.597 INFO:tasks.rados.rados.0.smithi085.stdout:1884: done (14 left) 2021-02-10T04:35:11.597 INFO:tasks.rados.rados.0.smithi085.stdout:1885: done (13 left) 2021-02-10T04:35:11.597 INFO:tasks.rados.rados.0.smithi085.stdout:1891: done (12 left) 2021-02-10T04:35:11.597 INFO:tasks.rados.rados.0.smithi085.stdout:1892: snap_create 2021-02-10T04:35:11.597 INFO:tasks.rados.rados.0.smithi085.stdout:1888: expect (ObjNum 616 snap 186 seq_num 616) 2021-02-10T04:35:11.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:11 smithi164 conmon[29288]: cluster 2021-02-10T04:35:10.097102+0000 mgr.y (mgr. 2021-02-10T04:35:11.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:11 smithi164 conmon[29288]: 14140) 585 : cluster [DBG] pgmap v888: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 398 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.6 MiB/s rd, 852 KiB/s wr, 5 op/s 2021-02-10T04:35:11.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:11 smithi164 conmon[29288]: cluster 2021-02-10T04:35:10.172987+0000 mon.a (mon.0) 1313 : cluster [DBG] osdmap e493: 8 total, 8 up, 8 in 2021-02-10T04:35:11.702 INFO:tasks.rados.rados.0.smithi085.stdout:1889: finishing write tid 1 to smithi08575397-43 2021-02-10T04:35:11.703 INFO:tasks.rados.rados.0.smithi085.stdout:1889: finishing write tid 2 to smithi08575397-43 2021-02-10T04:35:11.703 INFO:tasks.rados.rados.0.smithi085.stdout:1889: finishing write tid 3 to smithi08575397-43 2021-02-10T04:35:11.703 INFO:tasks.rados.rados.0.smithi085.stdout:1889: finishing write tid 4 to smithi08575397-43 2021-02-10T04:35:11.703 INFO:tasks.rados.rados.0.smithi085.stdout:1889: finishing write tid 5 to smithi08575397-43 2021-02-10T04:35:11.703 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 727 (ObjNum 670 snap 205 seq_num 670) dirty exists 2021-02-10T04:35:11.703 INFO:tasks.rados.rados.0.smithi085.stdout:1889: left oid 43 (ObjNum 670 snap 205 seq_num 670) 2021-02-10T04:35:12.193 INFO:tasks.rados.rados.0.smithi085.stdout:1887: done (12 left) 2021-02-10T04:35:12.193 INFO:tasks.rados.rados.0.smithi085.stdout:1888: done (11 left) 2021-02-10T04:35:12.193 INFO:tasks.rados.rados.0.smithi085.stdout:1889: done (10 left) 2021-02-10T04:35:12.193 INFO:tasks.rados.rados.0.smithi085.stdout:1892: done (9 left) 2021-02-10T04:35:12.193 INFO:tasks.rados.rados.0.smithi085.stdout:1893: snap_create 2021-02-10T04:35:12.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:12 smithi085 conmon[35961]: cluster 2021-02-10T04:35:11.184138+0000 2021-02-10T04:35:12.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:12 smithi085 conmon[35961]: mon.a (mon.0) 1314 : cluster [DBG] osdmap e494: 8 total, 8 up, 8 in 2021-02-10T04:35:12.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:12 smithi085 conmon[35961]: cluster 2021-02-10T04:35:12.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:12 smithi085 conmon[35961]: 2021-02-10T04:35: 2021-02-10T04:35:12.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:12 smithi085 conmon[35961]: 12.097735+0000 mgr.y (mgr.14140) 586 : cluster 2021-02-10T04:35:12.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:12 smithi085 conmon[35961]: [DBG] pgmap v891: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 394 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.3 MiB/s wr, 15 op/s 2021-02-10T04:35:12.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:12 smithi085 conmon[29319]: cluster 2021-02-10T04:35:11.184138+0000 mon.a (mon.0) 2021-02-10T04:35:12.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:12 smithi085 conmon[29319]: 1314 : cluster [DBG] osdmap e494: 8 total, 8 up, 8 in 2021-02-10T04:35:12.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:12 smithi085 conmon[29319]: cluster 2021-02-10T04:35:12.097735+0000 mgr.y (mgr.14140) 586 : cluster [DBG] pgmap v891: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 394 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.3 MiB/s wr, 15 op/s 2021-02-10T04:35:12.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:12 smithi164 conmon[29288]: cluster 2021-02-10T04:35:12.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:12 smithi164 conmon[29288]: 2021-02-10T04:35:11.184138+0000 2021-02-10T04:35:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:12 smithi164 conmon[29288]: mon.a (mon.0) 1314 : cluster [DBG] osdmap e494: 8 total, 8 up, 8 in 2021-02-10T04:35:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:12 smithi164 conmon[29288]: cluster 2021-02-10T04:35:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:12 smithi164 conmon[29288]: 2021-02-10T04:35:12.097735+0000 mgr.y (mgr.14140) 586 : cluster 2021-02-10T04:35:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:12 smithi164 conmon[29288]: [DBG] pgmap v891: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 394 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.3 MiB/s wr, 15 op/s 2021-02-10T04:35:13.240 INFO:tasks.rados.rados.0.smithi085.stdout:1893: done (9 left) 2021-02-10T04:35:13.240 INFO:tasks.rados.rados.0.smithi085.stdout:1894: write (excl) oid 44 current snap is 207 2021-02-10T04:35:13.240 INFO:tasks.rados.rados.0.smithi085.stdout:1894: seq_num 671 ranges {477497=731143,1998594=514889,3143534=133963} 2021-02-10T04:35:13.248 INFO:tasks.rados.rados.0.smithi085.stdout:1894: writing smithi08575397-44 from 477497 to 1208640 tid 1 2021-02-10T04:35:13.256 INFO:tasks.rados.rados.0.smithi085.stdout:1894: writing smithi08575397-44 from 1998594 to 2513483 tid 2 2021-02-10T04:35:13.261 INFO:tasks.rados.rados.0.smithi085.stdout:1894: writing smithi08575397-44 from 3143534 to 3277497 tid 3 2021-02-10T04:35:13.261 INFO:tasks.rados.rados.0.smithi085.stdout:1895: read oid 13 snap 185 2021-02-10T04:35:13.261 INFO:tasks.rados.rados.0.smithi085.stdout:1895: expect (ObjNum 595 snap 178 seq_num 595) 2021-02-10T04:35:13.261 INFO:tasks.rados.rados.0.smithi085.stdout:1896: snap_remove snap 206 2021-02-10T04:35:13.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:13 smithi164 conmon[29288]: cluster 2021-02-10T04:35:12.178206+0000 mon.a (mon 2021-02-10T04:35:13.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:13 smithi164 conmon[29288]: .0) 1315 : cluster [INF] Health check cleared: SLOW_OPS (was: 14 slow ops, oldest one blocked for 337 sec, daemons [osd.2,osd.4] have slow ops.) 2021-02-10T04:35:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:13 smithi164 conmon[29288]: cluster 2021-02-10T04:35:12.192213+0000 mon.a (mon.0) 1316 : cluster [DBG] osdmap e495: 8 total, 8 up, 8 in 2021-02-10T04:35:13.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:13 smithi085 conmon[35961]: cluster 2021-02-10T04:35:12.178206+0000 2021-02-10T04:35:13.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:13 smithi085 conmon[35961]: mon.a (mon.0) 1315 : cluster [INF] Health check cleared: SLOW_OPS (was: 14 slow ops, oldest one blocked for 337 sec, daemons [osd.2,osd.4] have slow ops.) 2021-02-10T04:35:13.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:13 smithi085 conmon[35961]: cluster 2021-02-10T04:35: 2021-02-10T04:35:13.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:13 smithi085 conmon[35961]: 12.192213+0000 mon.a (mon.0) 1316 : cluster [DBG] osdmap e495: 8 total, 8 up, 8 in 2021-02-10T04:35:13.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:13 smithi085 conmon[29319]: cluster 2021-02-10T04:35:12.178206+0000 2021-02-10T04:35:13.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:13 smithi085 conmon[29319]: mon.a (mon.0) 1315 : cluster [INF] Health check cleared: SLOW_OPS (was: 14 slow ops, oldest one blocked for 337 sec, daemons [osd.2,osd.4] have slow ops.) 2021-02-10T04:35:13.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:13 smithi085 conmon[29319]: cluster 2021-02-10T04:35:12.192213+0000 mon.a (mon.0) 2021-02-10T04:35:13.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:13 smithi085 conmon[29319]: 1316 : cluster [DBG] osdmap e495: 8 total, 8 up, 8 in 2021-02-10T04:35:14.256 INFO:tasks.rados.rados.0.smithi085.stdout:1896: done (11 left) 2021-02-10T04:35:14.256 INFO:tasks.rados.rados.0.smithi085.stdout:1897: snap_remove snap 194 2021-02-10T04:35:14.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:14 smithi164 conmon[29288]: cluster 2021-02-10T04:35:13.240736+0000 mon.a (mon.0 2021-02-10T04:35:14.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:14 smithi164 conmon[29288]: ) 1317 : cluster [DBG] osdmap e496: 8 total, 8 up, 8 in 2021-02-10T04:35:14.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:14 smithi164 conmon[29288]: cluster 2021-02-10T04:35:14.098116+0000 mgr.y (mgr.14140) 587 : cluster [DBG] pgmap v894: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 394 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 8.5 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2021-02-10T04:35:14.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:14 smithi085 conmon[29319]: cluster 2021-02-10T04:35:13.240736+0000 mon.a (mon.0) 1317 : cluster [DBG] osdmap e496: 8 total, 8 up, 8 in 2021-02-10T04:35:15.779 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:14 smithi085 conmon[29319]: 2021-02-10T04:35:15.779 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:14 smithi085 conmon[29319]: cluster 2021-02-10T04:35:14.098116+0000 mgr.y (mgr.14140) 587 : cluster [DBG] pgmap v894: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 394 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 8.5 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2021-02-10T04:35:15.779 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:14 smithi085 conmon[35961]: cluster 2021-02-10T04:35:15.779 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:14 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:35:15.779 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:14 smithi085 conmon[35961]: :35:13.240736+0000 mon.a (mon.0) 1317 : cluster [DBG] osdmap e496: 8 total, 8 up, 8 in 2021-02-10T04:35:15.779 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:14 smithi085 conmon[35961]: cluster 2021-02-10T04:35:14.098116+0000 2021-02-10T04:35:15.779 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:14 smithi085 conmon[35961]: mgr.y (mgr.14140) 587 : cluster [DBG] pgmap v894: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 394 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 8.5 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2021-02-10T04:35:15.782 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:15 smithi164 conmon[29288]: cluster 2021-02-10T04:35:14.255559+0000 mon.a (mon.0) 1318 : 2021-02-10T04:35:15.782 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:15 smithi164 conmon[29288]: cluster [DBG] osdmap e497: 8 total, 8 up, 8 in 2021-02-10T04:35:15.782 INFO:tasks.rados.rados.0.smithi085.stdout:1897: done (11 left) 2021-02-10T04:35:15.783 INFO:tasks.rados.rados.0.smithi085.stdout:1898: write oid 23 current snap is 207 2021-02-10T04:35:15.783 INFO:tasks.rados.rados.0.smithi085.stdout:1898: seq_num 672 ranges {494304=406392,1540723=774900,2770957=523347} 2021-02-10T04:35:15.783 INFO:tasks.rados.rados.0.smithi085.stdout:1898: writing smithi08575397-23 from 494304 to 900696 tid 1 2021-02-10T04:35:15.783 INFO:tasks.rados.rados.0.smithi085.stdout:1898: writing smithi08575397-23 from 1540723 to 2315623 tid 2 2021-02-10T04:35:15.783 INFO:tasks.rados.rados.0.smithi085.stdout:1898: writing smithi08575397-23 from 2770957 to 3294304 tid 3 2021-02-10T04:35:15.783 INFO:tasks.rados.rados.0.smithi085.stdout:1899: write oid 5 current snap is 207 2021-02-10T04:35:15.783 INFO:tasks.rados.rados.0.smithi085.stdout:1894: finishing write tid 1 to smithi08575397-44 2021-02-10T04:35:15.783 INFO:tasks.rados.rados.0.smithi085.stdout:1899: seq_num 673 ranges {511111=481641,1599205=518558,2682027=489947} 2021-02-10T04:35:15.783 INFO:tasks.rados.rados.0.smithi085.stdout:1899: writing smithi08575397-5 from 511111 to 992752 tid 1 2021-02-10T04:35:15.783 INFO:tasks.rados.rados.0.smithi085.stdout:1899: writing smithi08575397-5 from 1599205 to 2117763 tid 2 2021-02-10T04:35:15.784 INFO:tasks.rados.rados.0.smithi085.stdout:1899: writing smithi08575397-5 from 2682027 to 3171974 tid 3 2021-02-10T04:35:15.784 INFO:tasks.rados.rados.0.smithi085.stdout:1900: copy_from oid 42 from oid 46 current snap is 207 2021-02-10T04:35:15.784 INFO:tasks.rados.rados.0.smithi085.stdout:1894: finishing write tid 2 to smithi08575397-44 2021-02-10T04:35:15.784 INFO:tasks.rados.rados.0.smithi085.stdout:1894: finishing write tid 3 to smithi08575397-44 2021-02-10T04:35:15.784 INFO:tasks.rados.rados.0.smithi085.stdout:1894: finishing write tid 4 to smithi08575397-44 2021-02-10T04:35:15.784 INFO:tasks.rados.rados.0.smithi085.stdout:1894: finishing write tid 5 to smithi08575397-44 2021-02-10T04:35:15.784 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 737 (ObjNum 671 snap 207 seq_num 671) dirty exists 2021-02-10T04:35:15.784 INFO:tasks.rados.rados.0.smithi085.stdout:1894: left oid 44 (ObjNum 671 snap 207 seq_num 671) 2021-02-10T04:35:15.784 INFO:tasks.rados.rados.0.smithi085.stdout:1895: expect (ObjNum 595 snap 178 seq_num 595) 2021-02-10T04:35:15.784 INFO:tasks.rados.rados.0.smithi085.stdout:1898: finishing write tid 1 to smithi08575397-23 2021-02-10T04:35:15.784 INFO:tasks.rados.rados.0.smithi085.stdout:1898: finishing write tid 2 to smithi08575397-23 2021-02-10T04:35:15.784 INFO:tasks.rados.rados.0.smithi085.stdout:1898: finishing write tid 3 to smithi08575397-23 2021-02-10T04:35:15.784 INFO:tasks.rados.rados.0.smithi085.stdout:1898: finishing write tid 4 to smithi08575397-23 2021-02-10T04:35:15.785 INFO:tasks.rados.rados.0.smithi085.stdout:1898: finishing write tid 5 to smithi08575397-23 2021-02-10T04:35:15.785 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 668 (ObjNum 672 snap 207 seq_num 672) dirty exists 2021-02-10T04:35:15.785 INFO:tasks.rados.rados.0.smithi085.stdout:1898: left oid 23 (ObjNum 672 snap 207 seq_num 672) 2021-02-10T04:35:15.785 INFO:tasks.rados.rados.0.smithi085.stdout:1899: finishing write tid 1 to smithi08575397-5 2021-02-10T04:35:15.785 INFO:tasks.rados.rados.0.smithi085.stdout:1899: finishing write tid 2 to smithi08575397-5 2021-02-10T04:35:15.785 INFO:tasks.rados.rados.0.smithi085.stdout:1899: finishing write tid 3 to smithi08575397-5 2021-02-10T04:35:15.785 INFO:tasks.rados.rados.0.smithi085.stdout:1899: finishing write tid 4 to smithi08575397-5 2021-02-10T04:35:15.785 INFO:tasks.rados.rados.0.smithi085.stdout:1899: finishing write tid 5 to smithi08575397-5 2021-02-10T04:35:15.785 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 677 (ObjNum 673 snap 207 seq_num 673) dirty exists 2021-02-10T04:35:15.785 INFO:tasks.rados.rados.0.smithi085.stdout:1899: left oid 5 (ObjNum 673 snap 207 seq_num 673) 2021-02-10T04:35:15.785 INFO:tasks.rados.rados.0.smithi085.stdout:1900: finishing copy_from to smithi08575397-42 2021-02-10T04:35:15.785 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 765 (ObjNum 646 snap 196 seq_num 646) dirty exists 2021-02-10T04:35:15.785 INFO:tasks.rados.rados.0.smithi085.stdout:1900: finishing copy_from racing read to smithi08575397-42 2021-02-10T04:35:15.786 INFO:tasks.rados.rados.0.smithi085.stdout:1894: done (13 left) 2021-02-10T04:35:15.786 INFO:tasks.rados.rados.0.smithi085.stdout:1895: done (12 left) 2021-02-10T04:35:15.786 INFO:tasks.rados.rados.0.smithi085.stdout:1898: done (11 left) 2021-02-10T04:35:15.786 INFO:tasks.rados.rados.0.smithi085.stdout:1899: done (10 left) 2021-02-10T04:35:15.786 INFO:tasks.rados.rados.0.smithi085.stdout:1900: done (9 left) 2021-02-10T04:35:15.786 INFO:tasks.rados.rados.0.smithi085.stdout:1901: snap_create 2021-02-10T04:35:15.786 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:15 smithi085 conmon[35961]: cluster 2021-02-10T04:35:14.255559+0000 2021-02-10T04:35:15.786 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:15 smithi085 conmon[35961]: mon.a (mon.0) 1318 : cluster [DBG] osdmap e497: 8 total, 8 up, 8 in 2021-02-10T04:35:15.786 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:15 smithi085 conmon[29319]: cluster 2021-02-10T04:35:14.255559+0000 mon.a (mon. 2021-02-10T04:35:15.786 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:15 smithi085 conmon[29319]: 0) 1318 : cluster [DBG] osdmap e497: 8 total, 8 up, 8 in 2021-02-10T04:35:16.270 INFO:tasks.rados.rados.0.smithi085.stdout:1901: done (9 left) 2021-02-10T04:35:16.270 INFO:tasks.rados.rados.0.smithi085.stdout:1902: write (excl) oid 3 current snap is 208 2021-02-10T04:35:16.270 INFO:tasks.rados.rados.0.smithi085.stdout:1902: seq_num 674 ranges {527918=556890,1657687=778569,3225803=102115} 2021-02-10T04:35:16.276 INFO:tasks.rados.rados.0.smithi085.stdout:1902: writing smithi08575397-3 from 527918 to 1084808 tid 1 2021-02-10T04:35:16.285 INFO:tasks.rados.rados.0.smithi085.stdout:1902: writing smithi08575397-3 from 1657687 to 2436256 tid 2 2021-02-10T04:35:16.290 INFO:tasks.rados.rados.0.smithi085.stdout:1902: writing smithi08575397-3 from 3225803 to 3327918 tid 3 2021-02-10T04:35:16.291 INFO:tasks.rados.rados.0.smithi085.stdout:1903: rollback oid 7 current snap is 208 2021-02-10T04:35:16.291 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 7 to 165 2021-02-10T04:35:16.291 INFO:tasks.rados.rados.0.smithi085.stdout:1904: write (excl) oid 13 current snap is 208 2021-02-10T04:35:16.291 INFO:tasks.rados.rados.0.smithi085.stdout:1904: seq_num 675 ranges {544725=632139,1716169=522227,2736873=512491} 2021-02-10T04:35:16.298 INFO:tasks.rados.rados.0.smithi085.stdout:1904: writing smithi08575397-13 from 544725 to 1176864 tid 1 2021-02-10T04:35:16.304 INFO:tasks.rados.rados.0.smithi085.stdout:1904: writing smithi08575397-13 from 1716169 to 2238396 tid 2 2021-02-10T04:35:16.310 INFO:tasks.rados.rados.0.smithi085.stdout:1904: writing smithi08575397-13 from 2736873 to 3249364 tid 3 2021-02-10T04:35:16.311 INFO:tasks.rados.rados.0.smithi085.stdout:1905: write oid 30 current snap is 208 2021-02-10T04:35:16.311 INFO:tasks.rados.rados.0.smithi085.stdout:1902: finishing write tid 1 to smithi08575397-3 2021-02-10T04:35:16.311 INFO:tasks.rados.rados.0.smithi085.stdout:1905: seq_num 676 ranges {561532=707388,2058298=782238} 2021-02-10T04:35:16.318 INFO:tasks.rados.rados.0.smithi085.stdout:1905: writing smithi08575397-30 from 561532 to 1268920 tid 1 2021-02-10T04:35:16.328 INFO:tasks.rados.rados.0.smithi085.stdout:1905: writing smithi08575397-30 from 2058298 to 2840536 tid 2 2021-02-10T04:35:16.328 INFO:tasks.rados.rados.0.smithi085.stdout:1906: read oid 26 snap -1 2021-02-10T04:35:16.329 INFO:tasks.rados.rados.0.smithi085.stdout:1906: expect deleted 2021-02-10T04:35:16.329 INFO:tasks.rados.rados.0.smithi085.stdout:1902: finishing write tid 2 to smithi08575397-3 2021-02-10T04:35:16.329 INFO:tasks.rados.rados.0.smithi085.stdout:1902: finishing write tid 3 to smithi08575397-3 2021-02-10T04:35:16.329 INFO:tasks.rados.rados.0.smithi085.stdout:1902: finishing write tid 4 to smithi08575397-3 2021-02-10T04:35:16.329 INFO:tasks.rados.rados.0.smithi085.stdout:1902: finishing write tid 5 to smithi08575397-3 2021-02-10T04:35:16.329 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 731 (ObjNum 674 snap 208 seq_num 674) dirty exists 2021-02-10T04:35:16.329 INFO:tasks.rados.rados.0.smithi085.stdout:1902: left oid 3 (ObjNum 674 snap 208 seq_num 674) 2021-02-10T04:35:16.329 INFO:tasks.rados.rados.0.smithi085.stdout:1903: finishing rollback tid 1 to smithi08575397-7 2021-02-10T04:35:16.329 INFO:tasks.rados.rados.0.smithi085.stdout:1903: finishing rollback tid 2 to smithi08575397-7 2021-02-10T04:35:16.330 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 733 (ObjNum 204 snap 44 seq_num 204) dirty exists 2021-02-10T04:35:16.330 INFO:tasks.rados.rados.0.smithi085.stdout:1904: finishing write tid 1 to smithi08575397-13 2021-02-10T04:35:16.330 INFO:tasks.rados.rados.0.smithi085.stdout:1904: finishing write tid 2 to smithi08575397-13 2021-02-10T04:35:16.330 INFO:tasks.rados.rados.0.smithi085.stdout:1904: finishing write tid 3 to smithi08575397-13 2021-02-10T04:35:16.330 INFO:tasks.rados.rados.0.smithi085.stdout:1904: finishing write tid 4 to smithi08575397-13 2021-02-10T04:35:16.330 INFO:tasks.rados.rados.0.smithi085.stdout:1904: finishing write tid 5 to smithi08575397-13 2021-02-10T04:35:16.330 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 681 (ObjNum 675 snap 208 seq_num 675) dirty exists 2021-02-10T04:35:16.330 INFO:tasks.rados.rados.0.smithi085.stdout:1904: left oid 13 (ObjNum 675 snap 208 seq_num 675) 2021-02-10T04:35:16.330 INFO:tasks.rados.rados.0.smithi085.stdout:1905: finishing write tid 1 to smithi08575397-30 2021-02-10T04:35:16.330 INFO:tasks.rados.rados.0.smithi085.stdout:1907: read oid 20 snap -1 2021-02-10T04:35:16.330 INFO:tasks.rados.rados.0.smithi085.stdout:1907: expect deleted 2021-02-10T04:35:16.330 INFO:tasks.rados.rados.0.smithi085.stdout:1902: done (14 left) 2021-02-10T04:35:16.330 INFO:tasks.rados.rados.0.smithi085.stdout:1903: done (13 left) 2021-02-10T04:35:16.331 INFO:tasks.rados.rados.0.smithi085.stdout:1904: done (12 left) 2021-02-10T04:35:16.331 INFO:tasks.rados.rados.0.smithi085.stdout:1908: snap_create 2021-02-10T04:35:16.332 INFO:tasks.rados.rados.0.smithi085.stdout:1905: finishing write tid 2 to smithi08575397-30 2021-02-10T04:35:16.332 INFO:tasks.rados.rados.0.smithi085.stdout:1905: finishing write tid 3 to smithi08575397-30 2021-02-10T04:35:16.332 INFO:tasks.rados.rados.0.smithi085.stdout:1905: finishing write tid 4 to smithi08575397-30 2021-02-10T04:35:16.332 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 772 (ObjNum 676 snap 208 seq_num 676) dirty exists 2021-02-10T04:35:16.332 INFO:tasks.rados.rados.0.smithi085.stdout:1905: left oid 30 (ObjNum 676 snap 208 seq_num 676) 2021-02-10T04:35:16.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:16 smithi164 conmon[29288]: cluster 2021-02-10T04:35:16.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:16 smithi164 conmon[29288]: 2021-02-10T04:35:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:16 smithi164 conmon[29288]: 2021 2021-02-10T04:35:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:16 smithi164 conmon[29288]: - 2021-02-10T04:35:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:16 smithi164 conmon[29288]: 02 2021-02-10T04:35:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:16 smithi164 conmon[29288]: - 2021-02-10T04:35:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:16 smithi164 conmon[29288]: 10 2021-02-10T04:35:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:16 smithi164 conmon[29288]: T04:35:15.252123+0000 mon.a ( 2021-02-10T04:35:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:16 smithi164 conmon[29288]: mon.0) 1319 : cluster [DBG] osdmap e498: 8 total, 8 up, 8 in 2021-02-10T04:35:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:16 smithi164 conmon[29288]: cluster 2021-02-10T04:35:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:16 smithi164 conmon[29288]: 2021-02-10T04:35:16.098575+0000 mgr.y ( 2021-02-10T04:35:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:16 smithi164 conmon[29288]: mgr.14140) 588 : cluster [DBG] pgmap v897: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 388 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.1 MiB/s wr, 3 op/s 2021-02-10T04:35:16.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:16 smithi085 conmon[35961]: cluster 2021-02-10T04:35:15.252123+0000 mon.a (mon.0) 1319 : cluster 2021-02-10T04:35:16.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:16 smithi085 conmon[35961]: [DBG] osdmap e498: 8 total, 8 up, 8 in 2021-02-10T04:35:16.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:16 smithi085 conmon[35961]: cluster 2021-02-10T04:35:16.098575+0000 2021-02-10T04:35:16.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:16 smithi085 conmon[35961]: mgr.y (mgr.14140) 588 : cluster [DBG] pgmap v897: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 388 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.1 MiB/s wr, 3 op/s 2021-02-10T04:35:16.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:16 smithi085 conmon[29319]: cluster 2021-02-10T04:35:15.252123+0000 mon.a (mon.0) 1319 : cluster 2021-02-10T04:35:16.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:16 smithi085 conmon[29319]: [DBG] osdmap e498: 8 total, 8 up, 8 in 2021-02-10T04:35:16.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:16 smithi085 conmon[29319]: cluster 2021-02-10T04:35:16.098575+0000 mgr.y (mgr.14140) 588 : cluster [DBG] pgmap v897: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 388 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.1 MiB/s wr, 3 op/s 2021-02-10T04:35:17.060 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:35:17.087 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:35:17.267 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:17 smithi085 conmon[35961]: cluster 2021-02-10T04:35:16.268147+0000 2021-02-10T04:35:17.267 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:17 smithi085 conmon[35961]: mon.a (mon.0) 1320 : cluster [DBG] osdmap e499: 8 total, 8 up, 8 in 2021-02-10T04:35:17.267 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:17 smithi085 conmon[29319]: cluster 2021-02-10T04:35:16.268147+0000 mon.a (mon 2021-02-10T04:35:17.268 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:17 smithi085 conmon[29319]: .0) 1320 : cluster [DBG] osdmap e499: 8 total, 8 up, 8 in 2021-02-10T04:35:17.277 INFO:tasks.rados.rados.0.smithi085.stdout:1905: done (12 left) 2021-02-10T04:35:17.277 INFO:tasks.rados.rados.0.smithi085.stdout:1906: done (11 left) 2021-02-10T04:35:17.277 INFO:tasks.rados.rados.0.smithi085.stdout:1907: done (10 left) 2021-02-10T04:35:17.278 INFO:tasks.rados.rados.0.smithi085.stdout:1908: done (9 left) 2021-02-10T04:35:17.278 INFO:tasks.rados.rados.0.smithi085.stdout:1909: write (excl) oid 6 current snap is 209 2021-02-10T04:35:17.278 INFO:tasks.rados.rados.0.smithi085.stdout:1909: seq_num 677 ranges {578339=498990,1833133=525896,2791719=586620} 2021-02-10T04:35:17.283 INFO:tasks.rados.rados.0.smithi085.stdout:1909: writing smithi08575397-6 from 578339 to 1077329 tid 1 2021-02-10T04:35:17.291 INFO:tasks.rados.rados.0.smithi085.stdout:1909: writing smithi08575397-6 from 1833133 to 2359029 tid 2 2021-02-10T04:35:17.298 INFO:tasks.rados.rados.0.smithi085.stdout:1909: writing smithi08575397-6 from 2791719 to 3378339 tid 3 2021-02-10T04:35:17.298 INFO:tasks.rados.rados.0.smithi085.stdout:1910: delete oid 38 current snap is 209 2021-02-10T04:35:17.298 INFO:tasks.rados.rados.0.smithi085.stdout:1909: finishing write tid 1 to smithi08575397-6 2021-02-10T04:35:17.298 INFO:tasks.rados.rados.0.smithi085.stdout:1909: finishing write tid 2 to smithi08575397-6 2021-02-10T04:35:17.303 INFO:tasks.rados.rados.0.smithi085.stdout:1909: finishing write tid 3 to smithi08575397-6 2021-02-10T04:35:17.303 INFO:tasks.rados.rados.0.smithi085.stdout:1909: finishing write tid 4 to smithi08575397-6 2021-02-10T04:35:17.304 INFO:tasks.rados.rados.0.smithi085.stdout:1909: finishing write tid 5 to smithi08575397-6 2021-02-10T04:35:17.304 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 776 (ObjNum 677 snap 209 seq_num 677) dirty exists 2021-02-10T04:35:17.304 INFO:tasks.rados.rados.0.smithi085.stdout:1909: left oid 6 (ObjNum 677 snap 209 seq_num 677) 2021-02-10T04:35:17.304 INFO:tasks.rados.rados.0.smithi085.stdout:1909: done (10 left) 2021-02-10T04:35:17.304 INFO:tasks.rados.rados.0.smithi085.stdout:1910: done (9 left) 2021-02-10T04:35:17.305 INFO:tasks.rados.rados.0.smithi085.stdout:1911: delete oid 2 current snap is 209 2021-02-10T04:35:17.307 INFO:tasks.rados.rados.0.smithi085.stdout:1911: done (9 left) 2021-02-10T04:35:17.307 INFO:tasks.rados.rados.0.smithi085.stdout:1912: read oid 11 snap -1 2021-02-10T04:35:17.307 INFO:tasks.rados.rados.0.smithi085.stdout:1912: expect (ObjNum 531 snap 156 seq_num 531) 2021-02-10T04:35:17.307 INFO:tasks.rados.rados.0.smithi085.stdout:1913: snap_create 2021-02-10T04:35:17.317 INFO:tasks.rados.rados.0.smithi085.stdout:1912: expect (ObjNum 531 snap 156 seq_num 531) 2021-02-10T04:35:17.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:17 smithi164 conmon[33677]: debug 2021-02-10T04:35:17.298+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:35:17.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:17 smithi164 conmon[29288]: cluster 2021-02-10T04:35:16.268147+0000 mon.a (mon.0) 1320 : cluster [DBG] osdmap e499: 8 total, 8 up, 8 in 2021-02-10T04:35:17.653 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:35:17.758 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:35:17.793 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:35:17.905 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:35:17.908 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:17 smithi164 conmon[33677]: debug 2021-02-10T04:35:17.875+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:35:18.010 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:35:18.056 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:35:18.274 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:35:18 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:35:18.274 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:18 smithi164 conmon[29288]: cluster 2021-02-10T04:35:18.285 INFO:tasks.rados.rados.0.smithi085.stdout:1912: done (10 left) 2021-02-10T04:35:18.285 INFO:tasks.rados.rados.0.smithi085.stdout:1913: done (9 left) 2021-02-10T04:35:18.285 INFO:tasks.rados.rados.0.smithi085.stdout:1914: rollback oid 2 current snap is 210 2021-02-10T04:35:18.285 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 2 to 123 2021-02-10T04:35:18.286 INFO:tasks.rados.rados.0.smithi085.stdout:1915: read oid 26 snap -1 2021-02-10T04:35:18.286 INFO:tasks.rados.rados.0.smithi085.stdout:1915: expect deleted 2021-02-10T04:35:18.286 INFO:tasks.rados.rados.0.smithi085.stdout:1916: snap_remove snap 165 2021-02-10T04:35:18.390 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:35:18.441 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:35:18 smithi085 conmon[45443]: debug 2021-02-10T04:35:18.134+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:35:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:18 smithi085 conmon[29319]: cluster 2021-02-10T04:35:17.276940+0000 mon.a (mon.0) 1321 : 2021-02-10T04:35:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:18 smithi085 conmon[29319]: cluster [DBG] osdmap e500: 8 total, 8 up, 8 in 2021-02-10T04:35:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:18 smithi085 conmon[29319]: audit 2021-02-10T04:35:18.076059+0000 mgr.y (mgr.14140) 589 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:35:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:18 smithi085 conmon[29319]: 2021-02-10T04:35:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:18 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:35:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:18 smithi085 conmon[29319]: 10T04:35:18.099076+0000 mgr.y (mgr.14140) 2021-02-10T04:35:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:18 smithi085 conmon[29319]: 590 : cluster [DBG] pgmap v900: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 388 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 337 KiB/s wr, 2 op/s 2021-02-10T04:35:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:18 smithi085 conmon[35961]: cluster 2021-02-10T04:35:17.276940+0000 mon.a ( 2021-02-10T04:35:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:18 smithi085 conmon[35961]: mon.0) 1321 : cluster [DBG] osdmap e500: 8 total, 8 up, 8 in 2021-02-10T04:35:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:18 smithi085 conmon[35961]: audit 2021-02-10T04:35:18.076059+0000 mgr.y (mgr 2021-02-10T04:35:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:18 smithi085 conmon[35961]: .14140) 589 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:35:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:18 smithi085 conmon[35961]: 2021-02-10T04:35:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:18 smithi085 conmon[35961]: cluster 2021-02-10T04:35:18 2021-02-10T04:35:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:18 smithi085 conmon[35961]: .099076+0000 mgr.y (mgr.14140) 590 : cluster 2021-02-10T04:35:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:18 smithi085 conmon[35961]: [DBG] pgmap v900: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 388 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 337 KiB/s wr, 2 op/s 2021-02-10T04:35:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:18 smithi164 conmon[29288]: 2021-02-10T04:35: 2021-02-10T04:35:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:18 smithi164 conmon[29288]: 17.276940+0000 mon.a (mon.0) 1321 : cluster [DBG] osdmap e500: 8 total, 8 up, 8 in 2021-02-10T04:35:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:18 smithi164 conmon[29288]: audit 2021-02-10T04:35:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:18 smithi164 conmon[29288]: 2021-02-10T04:35:18.076059+0000 mgr.y 2021-02-10T04:35:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:18 smithi164 conmon[29288]: (mgr.14140) 589 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:35:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:18 smithi164 conmon[29288]: cluster 2021-02-10T04:35:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:18 smithi164 conmon[29288]: 2021-02-10T04:35:18 2021-02-10T04:35:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:18 smithi164 conmon[29288]: .099076+0000 mgr.y 2021-02-10T04:35:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:18 smithi164 conmon[29288]: (mgr.14140) 590 : cluster [DBG] pgmap v900: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 388 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 337 KiB/s wr, 2 op/s 2021-02-10T04:35:19.290 INFO:tasks.rados.rados.0.smithi085.stdout:1916: done (11 left) 2021-02-10T04:35:19.291 INFO:tasks.rados.rados.0.smithi085.stdout:1917: rollback oid 14 current snap is 210 2021-02-10T04:35:19.292 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 14 to 202 2021-02-10T04:35:19.292 INFO:tasks.rados.rados.0.smithi085.stdout:1918: snap_remove snap 208 2021-02-10T04:35:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:19 smithi164 conmon[29288]: cluster 2021-02-10T04:35:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:19 smithi164 conmon[29288]: 2021-02-10T04:35:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:19 smithi164 conmon[29288]: 2021 2021-02-10T04:35:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:19 smithi164 conmon[29288]: - 2021-02-10T04:35:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:19 smithi164 conmon[29288]: 02 2021-02-10T04:35:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:19 smithi164 conmon[29288]: - 2021-02-10T04:35:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:19 smithi164 conmon[29288]: 10 2021-02-10T04:35:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:19 smithi164 conmon[29288]: T 2021-02-10T04:35:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:19 smithi164 conmon[29288]: 04 2021-02-10T04:35:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:19 smithi164 conmon[29288]: : 2021-02-10T04:35:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:19 smithi164 conmon[29288]: 35 2021-02-10T04:35:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:19 smithi164 conmon[29288]: :18.284914+0000 mon.a (mon.0) 1322 : cluster [DBG] 2021-02-10T04:35:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:19 smithi164 conmon[29288]: osdmap e501: 8 total, 8 up, 8 in 2021-02-10T04:35:19.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:19 smithi085 conmon[29319]: cluster 2021-02-10T04:35:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:19 smithi085 conmon[29319]: 2021-02-10T04:35:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:19 smithi085 conmon[29319]: 2021 2021-02-10T04:35:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:19 smithi085 conmon[29319]: - 2021-02-10T04:35:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:19 smithi085 conmon[29319]: 02 2021-02-10T04:35:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:19 smithi085 conmon[29319]: - 2021-02-10T04:35:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:19 smithi085 conmon[29319]: 10 2021-02-10T04:35:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:19 smithi085 conmon[29319]: T 2021-02-10T04:35:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:19 smithi085 conmon[29319]: 04 2021-02-10T04:35:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:19 smithi085 conmon[29319]: : 2021-02-10T04:35:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:19 smithi085 conmon[29319]: 35 2021-02-10T04:35:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:19 smithi085 conmon[29319]: : 2021-02-10T04:35:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:19 smithi085 conmon[29319]: 18.284914+0000 mon.a (mon.0) 2021-02-10T04:35:19.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:19 smithi085 conmon[29319]: 1322 : cluster [DBG] osdmap e501: 8 total, 8 up, 8 in 2021-02-10T04:35:19.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:19 smithi085 conmon[35961]: cluster 2021-02-10T04:35:18.284914+0000 2021-02-10T04:35:19.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:19 smithi085 conmon[35961]: mon.a (mon.0) 1322 : cluster [DBG] osdmap e501: 8 total, 8 up, 8 in 2021-02-10T04:35:20.290 INFO:tasks.rados.rados.0.smithi085.stdout:1918: done (12 left) 2021-02-10T04:35:20.290 INFO:tasks.rados.rados.0.smithi085.stdout:1919: read oid 38 snap 192 2021-02-10T04:35:20.290 INFO:tasks.rados.rados.0.smithi085.stdout:1919: expect (ObjNum 616 snap 186 seq_num 616) 2021-02-10T04:35:20.290 INFO:tasks.rados.rados.0.smithi085.stdout:1920: write (excl) oid 21 current snap is 210 2021-02-10T04:35:20.290 INFO:tasks.rados.rados.0.smithi085.stdout:1920: seq_num 680 ranges {628760=724737,1892226=673223,3157635=271125} 2021-02-10T04:35:20.294 INFO:tasks.rados.rados.0.smithi085.stdout:1920: writing smithi08575397-21 from 628760 to 1353497 tid 1 2021-02-10T04:35:20.302 INFO:tasks.rados.rados.0.smithi085.stdout:1920: writing smithi08575397-21 from 1892226 to 2565449 tid 2 2021-02-10T04:35:20.307 INFO:tasks.rados.rados.0.smithi085.stdout:1920: writing smithi08575397-21 from 3157635 to 3428760 tid 3 2021-02-10T04:35:20.308 INFO:tasks.rados.rados.0.smithi085.stdout:1921: read oid 46 snap -1 2021-02-10T04:35:20.308 INFO:tasks.rados.rados.0.smithi085.stdout:1921: expect (ObjNum 646 snap 196 seq_num 646) 2021-02-10T04:35:20.308 INFO:tasks.rados.rados.0.smithi085.stdout:1922: write oid 11 current snap is 210 2021-02-10T04:35:20.308 INFO:tasks.rados.rados.0.smithi085.stdout:1922: seq_num 681 ranges {645567=799986,1950708=533234,3185058=260509} 2021-02-10T04:35:20.316 INFO:tasks.rados.rados.0.smithi085.stdout:1922: writing smithi08575397-11 from 645567 to 1445553 tid 1 2021-02-10T04:35:20.323 INFO:tasks.rados.rados.0.smithi085.stdout:1922: writing smithi08575397-11 from 1950708 to 2483942 tid 2 2021-02-10T04:35:20.329 INFO:tasks.rados.rados.0.smithi085.stdout:1922: writing smithi08575397-11 from 3185058 to 3445567 tid 3 2021-02-10T04:35:20.330 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:35:20.330 INFO:tasks.rados.rados.0.smithi085.stdout:1914: finishing rollback tid 1 to smithi08575397-2 2021-02-10T04:35:20.330 INFO:tasks.rados.rados.0.smithi085.stdout:1914: finishing rollback tid 2 to smithi08575397-2 2021-02-10T04:35:20.330 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 661 (ObjNum 422 snap 109 seq_num 422) dirty exists 2021-02-10T04:35:20.330 INFO:tasks.rados.rados.0.smithi085.stdout:1917: finishing rollback tid 0 to smithi08575397-14 2021-02-10T04:35:20.330 INFO:tasks.rados.rados.0.smithi085.stdout:1917: finishing rollback tid 1 to smithi08575397-14 2021-02-10T04:35:20.330 INFO:tasks.rados.rados.0.smithi085.stdout:1914: done (15 left) 2021-02-10T04:35:20.330 INFO:tasks.rados.rados.0.smithi085.stdout:1915: done (14 left) 2021-02-10T04:35:20.330 INFO:tasks.rados.rados.0.smithi085.stdout:1923: read oid 16 snap 202 2021-02-10T04:35:20.330 INFO:tasks.rados.rados.0.smithi085.stdout:1923: expect (ObjNum 636 snap 196 seq_num 636) 2021-02-10T04:35:20.331 INFO:tasks.rados.rados.0.smithi085.stdout:1924: snap_remove snap 123 2021-02-10T04:35:20.606 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:20 smithi085 conmon[29319]: cluster 2021-02-10T04:35:19.283876+0000 mon.a (mon 2021-02-10T04:35:20.607 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:20 smithi085 conmon[29319]: .0) 1323 : cluster [DBG] osdmap e502: 8 total, 8 up, 8 in 2021-02-10T04:35:20.607 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:20 smithi085 conmon[29319]: cluster 2021-02-10T04:35:20.607 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:20 smithi085 conmon[29319]: 2021-02-10T04:35:20.099565+0000 2021-02-10T04:35:20.607 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:20 smithi085 conmon[29319]: mgr.y (mgr.14140) 591 : cluster [DBG] 2021-02-10T04:35:20.607 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:20 smithi085 conmon[29319]: pgmap v903: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 394 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.5 MiB/s rd, 1.4 MiB/s wr, 7 op/s 2021-02-10T04:35:20.607 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:20 smithi085 conmon[29319]: cluster 2021-02-10T04:35:20.607 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:20 smithi085 conmon[29319]: 2021-02-10T04:35:20.285913 2021-02-10T04:35:20.607 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:20 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1324 : cluster [DBG] 2021-02-10T04:35:20.607 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:20 smithi085 conmon[29319]: osdmap e503: 8 total, 8 up, 8 in 2021-02-10T04:35:20.607 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:20 smithi085 conmon[35961]: cluster 2021-02-10T04:35:19.283876 2021-02-10T04:35:20.607 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:20 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1323 : cluster [DBG] osdmap e502: 8 total, 8 up, 8 in 2021-02-10T04:35:20.607 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:20 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:20.608 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:20 smithi085 conmon[35961]: 02-10T04:35:20.099565+0000 mgr.y ( 2021-02-10T04:35:20.608 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:20 smithi085 conmon[35961]: mgr.14140) 591 : cluster [DBG] 2021-02-10T04:35:20.608 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:20 smithi085 conmon[35961]: pgmap v903: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 394 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.5 MiB/s rd, 1.4 MiB/s wr, 7 op/s 2021-02-10T04:35:20.608 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:20 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:35:20.608 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:20 smithi085 conmon[35961]: 10T04:35:20.285913 2021-02-10T04:35:20.608 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:20 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1324 2021-02-10T04:35:20.608 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:20 smithi085 conmon[35961]: : cluster [DBG] osdmap e503: 8 total, 8 up, 8 in 2021-02-10T04:35:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:20 smithi164 conmon[29288]: cluster 2021-02-10T04:35:19.283876+0000 2021-02-10T04:35:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:20 smithi164 conmon[29288]: mon.a (mon.0) 1323 : cluster [DBG] osdmap e502: 8 total, 8 up, 8 in 2021-02-10T04:35:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:20 smithi164 conmon[29288]: cluster 2021-02-10T04:35:20.099565+0000 mgr.y (mgr.14140) 591 : cluster [DBG] pgmap v903: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 394 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.5 MiB/s rd, 1.4 MiB/s wr, 7 op/s 2021-02-10T04:35:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:20 smithi164 conmon[29288]: cluster 2021-02-10T04:35:20 2021-02-10T04:35:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:20 smithi164 conmon[29288]: .285913+0000 mon.a (mon.0) 1324 2021-02-10T04:35:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:20 smithi164 conmon[29288]: : cluster [DBG] osdmap e503: 8 total, 8 up, 8 in 2021-02-10T04:35:21.313 INFO:tasks.rados.rados.0.smithi085.stdout:1917: finishing rollback tid 2 to smithi08575397-14 2021-02-10T04:35:21.313 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 806 (ObjNum 651 snap 197 seq_num 651) dirty exists 2021-02-10T04:35:21.313 INFO:tasks.rados.rados.0.smithi085.stdout:1919: expect (ObjNum 616 snap 186 seq_num 616) 2021-02-10T04:35:21.411 INFO:tasks.rados.rados.0.smithi085.stdout:1917: done (15 left) 2021-02-10T04:35:21.412 INFO:tasks.rados.rados.0.smithi085.stdout:1919: done (14 left) 2021-02-10T04:35:21.412 INFO:tasks.rados.rados.0.smithi085.stdout:1924: done (13 left) 2021-02-10T04:35:21.412 INFO:tasks.rados.rados.0.smithi085.stdout:1925: write oid 37 current snap is 210 2021-02-10T04:35:21.412 INFO:tasks.rados.rados.0.smithi085.stdout:1925: seq_num 682 ranges {662374=475235,1609190=676892,2812481=624101} 2021-02-10T04:35:21.418 INFO:tasks.rados.rados.0.smithi085.stdout:1925: writing smithi08575397-37 from 662374 to 1137609 tid 1 2021-02-10T04:35:21.425 INFO:tasks.rados.rados.0.smithi085.stdout:1925: writing smithi08575397-37 from 1609190 to 2286082 tid 2 2021-02-10T04:35:21.433 INFO:tasks.rados.rados.0.smithi085.stdout:1925: writing smithi08575397-37 from 2812481 to 3436582 tid 3 2021-02-10T04:35:21.433 INFO:tasks.rados.rados.0.smithi085.stdout:1926: read oid 5 snap -1 2021-02-10T04:35:21.433 INFO:tasks.rados.rados.0.smithi085.stdout:1926: expect (ObjNum 673 snap 207 seq_num 673) 2021-02-10T04:35:21.433 INFO:tasks.rados.rados.0.smithi085.stdout:1927: read oid 30 snap -1 2021-02-10T04:35:21.433 INFO:tasks.rados.rados.0.smithi085.stdout:1927: expect (ObjNum 676 snap 208 seq_num 676) 2021-02-10T04:35:21.433 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:35:21.433 INFO:tasks.rados.rados.0.smithi085.stdout:1920: finishing write tid 1 to smithi08575397-21 2021-02-10T04:35:21.433 INFO:tasks.rados.rados.0.smithi085.stdout:1921: expect (ObjNum 646 snap 196 seq_num 646) 2021-02-10T04:35:21.612 INFO:tasks.rados.rados.0.smithi085.stdout:1920: finishing write tid 2 to smithi08575397-21 2021-02-10T04:35:21.612 INFO:tasks.rados.rados.0.smithi085.stdout:1920: finishing write tid 3 to smithi08575397-21 2021-02-10T04:35:21.612 INFO:tasks.rados.rados.0.smithi085.stdout:1920: finishing write tid 4 to smithi08575397-21 2021-02-10T04:35:21.612 INFO:tasks.rados.rados.0.smithi085.stdout:1920: finishing write tid 5 to smithi08575397-21 2021-02-10T04:35:21.612 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 1014 (ObjNum 680 snap 210 seq_num 680) dirty exists 2021-02-10T04:35:21.612 INFO:tasks.rados.rados.0.smithi085.stdout:1920: left oid 21 (ObjNum 680 snap 210 seq_num 680) 2021-02-10T04:35:21.612 INFO:tasks.rados.rados.0.smithi085.stdout:1922: finishing write tid 1 to smithi08575397-11 2021-02-10T04:35:21.612 INFO:tasks.rados.rados.0.smithi085.stdout:1922: finishing write tid 2 to smithi08575397-11 2021-02-10T04:35:21.612 INFO:tasks.rados.rados.0.smithi085.stdout:1922: finishing write tid 3 to smithi08575397-11 2021-02-10T04:35:21.612 INFO:tasks.rados.rados.0.smithi085.stdout:1922: finishing write tid 4 to smithi08575397-11 2021-02-10T04:35:21.613 INFO:tasks.rados.rados.0.smithi085.stdout:1920: done (15 left) 2021-02-10T04:35:21.613 INFO:tasks.rados.rados.0.smithi085.stdout:1921: done (14 left) 2021-02-10T04:35:21.613 INFO:tasks.rados.rados.0.smithi085.stdout:1928: read oid 50 snap 201 2021-02-10T04:35:21.613 INFO:tasks.rados.rados.0.smithi085.stdout:1928: expect deleted 2021-02-10T04:35:21.613 INFO:tasks.rados.rados.0.smithi085.stdout:1929: delete oid 10 current snap is 210 2021-02-10T04:35:21.613 INFO:tasks.rados.rados.0.smithi085.stdout:1922: finishing write tid 5 to smithi08575397-11 2021-02-10T04:35:21.613 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 1019 (ObjNum 681 snap 210 seq_num 681) dirty exists 2021-02-10T04:35:21.613 INFO:tasks.rados.rados.0.smithi085.stdout:1922: left oid 11 (ObjNum 681 snap 210 seq_num 681) 2021-02-10T04:35:21.613 INFO:tasks.rados.rados.0.smithi085.stdout:1923: expect (ObjNum 636 snap 196 seq_num 636) 2021-02-10T04:35:21.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:21 smithi164 conmon[29288]: cluster 2021-02-10T04:35:20.111904+0000 osd.6 2021-02-10T04:35:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:21 smithi164 conmon[29288]: (osd.6) 63 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:35:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:21 smithi164 conmon[29288]: cluster 2021-02-10T04:35:20.112140+0000 osd.6 (osd.6) 64 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:35:21.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:21 smithi085 conmon[35961]: cluster 2021-02-10T04:35:20.111904+0000 osd.6 2021-02-10T04:35:21.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:21 smithi085 conmon[35961]: (osd.6) 63 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:35:21.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:21 smithi085 conmon[35961]: cluster 2021-02-10T04:35:20.112140+0000 osd.6 ( 2021-02-10T04:35:21.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:21 smithi085 conmon[35961]: osd.6) 64 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:35:21.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:21 smithi085 conmon[29319]: cluster 2021-02-10T04:35:20.111904 2021-02-10T04:35:21.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:21 smithi085 conmon[29319]: +0000 osd.6 (osd.6) 63 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:35:21.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:21 smithi085 conmon[29319]: cluster 2021-02-10T04:35:20 2021-02-10T04:35:21.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:21 smithi085 conmon[29319]: .112140+0000 osd.6 (osd.6) 64 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:35:21.821 INFO:tasks.rados.rados.0.smithi085.stdout:1922: done (15 left) 2021-02-10T04:35:21.821 INFO:tasks.rados.rados.0.smithi085.stdout:1923: done (14 left) 2021-02-10T04:35:21.821 INFO:tasks.rados.rados.0.smithi085.stdout:1929: done (13 left) 2021-02-10T04:35:21.821 INFO:tasks.rados.rados.0.smithi085.stdout:1930: rollback oid 8 current snap is 210 2021-02-10T04:35:21.821 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 8 to 191 2021-02-10T04:35:21.821 INFO:tasks.rados.rados.0.smithi085.stdout:1925: finishing write tid 1 to smithi08575397-37 2021-02-10T04:35:21.821 INFO:tasks.rados.rados.0.smithi085.stdout:1925: finishing write tid 2 to smithi08575397-37 2021-02-10T04:35:21.822 INFO:tasks.rados.rados.0.smithi085.stdout:1927: expect (ObjNum 676 snap 208 seq_num 676) 2021-02-10T04:35:22.028 INFO:tasks.rados.rados.0.smithi085.stdout:1926: expect (ObjNum 673 snap 207 seq_num 673) 2021-02-10T04:35:22.231 INFO:tasks.rados.rados.0.smithi085.stdout:1925: finishing write tid 3 to smithi08575397-37 2021-02-10T04:35:22.231 INFO:tasks.rados.rados.0.smithi085.stdout:1925: finishing write tid 4 to smithi08575397-37 2021-02-10T04:35:22.231 INFO:tasks.rados.rados.0.smithi085.stdout:1926: done (13 left) 2021-02-10T04:35:22.231 INFO:tasks.rados.rados.0.smithi085.stdout:1927: done (12 left) 2021-02-10T04:35:22.231 INFO:tasks.rados.rados.0.smithi085.stdout:1931: copy_from oid 48 from oid 2 current snap is 210 2021-02-10T04:35:22.231 INFO:tasks.rados.rados.0.smithi085.stdout:1925: finishing write tid 5 to smithi08575397-37 2021-02-10T04:35:22.232 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 1027 (ObjNum 682 snap 210 seq_num 682) dirty exists 2021-02-10T04:35:22.232 INFO:tasks.rados.rados.0.smithi085.stdout:1925: left oid 37 (ObjNum 682 snap 210 seq_num 682) 2021-02-10T04:35:22.232 INFO:tasks.rados.rados.0.smithi085.stdout:1930: finishing rollback tid 1 to smithi08575397-8 2021-02-10T04:35:22.232 INFO:tasks.rados.rados.0.smithi085.stdout:1930: finishing rollback tid 2 to smithi08575397-8 2021-02-10T04:35:22.232 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 741 (ObjNum 521 snap 155 seq_num 521) dirty exists 2021-02-10T04:35:22.232 INFO:tasks.rados.rados.0.smithi085.stdout:1925: done (12 left) 2021-02-10T04:35:22.232 INFO:tasks.rados.rados.0.smithi085.stdout:1928: done (11 left) 2021-02-10T04:35:22.232 INFO:tasks.rados.rados.0.smithi085.stdout:1930: done (10 left) 2021-02-10T04:35:22.232 INFO:tasks.rados.rados.0.smithi085.stdout:1932: snap_create 2021-02-10T04:35:22.317 INFO:tasks.rados.rados.0.smithi085.stdout:1932: done (10 left) 2021-02-10T04:35:22.317 INFO:tasks.rados.rados.0.smithi085.stdout:1933: snap_create 2021-02-10T04:35:22.519 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:35:22.556 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:35:22.594 INFO:tasks.rados.rados.0.smithi085.stdout:1931: finishing copy_from to smithi08575397-48 2021-02-10T04:35:22.594 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 661 (ObjNum 422 snap 109 seq_num 422) dirty exists 2021-02-10T04:35:22.594 INFO:tasks.rados.rados.0.smithi085.stdout:1931: finishing copy_from racing read to smithi08575397-48 2021-02-10T04:35:22.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:22 smithi085 conmon[29319]: cluster 2021-02-10T04:35:22.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:22 smithi085 conmon[29319]: 2021-02-10T04:35:21. 2021-02-10T04:35:22.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:22 smithi085 conmon[29319]: 309766+0000 mon.a (mon.0) 1325 : cluster [DBG] osdmap e504: 8 total, 8 up, 8 in 2021-02-10T04:35:22.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:22 smithi085 conmon[29319]: cluster 2021-02-10T04:35:22.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:22 smithi085 conmon[29319]: 2021-02-10T04:35:22.100137+0000 2021-02-10T04:35:22.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:22 smithi085 conmon[29319]: mgr.y (mgr.14140) 592 : cluster [DBG] pgmap v906: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 395 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.5 MiB/s rd, 2.9 MiB/s wr, 16 op/s 2021-02-10T04:35:22.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:22 smithi085 conmon[35961]: cluster 2021-02-10T04:35:22.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:22 smithi085 conmon[35961]: 2021-02-10T04:35:22.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:22 smithi085 conmon[35961]: 2021-02-10T04:35:21. 2021-02-10T04:35:22.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:22 smithi085 conmon[35961]: 309766+0000 mon.a (mon.0) 1325 : cluster [DBG] osdmap e504: 8 total, 8 up, 8 in 2021-02-10T04:35:22.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:22 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:35:22.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:22 smithi085 conmon[35961]: 35:22.100137 2021-02-10T04:35:22.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:22 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 592 : cluster [DBG] pgmap v906: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 395 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.5 MiB/s rd, 2.9 MiB/s wr, 16 op/s 2021-02-10T04:35:22.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:22 smithi164 conmon[29288]: cluster 2021-02-10T04:35:21.309766+0000 mon.a (mon 2021-02-10T04:35:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:22 smithi164 conmon[29288]: .0) 1325 : cluster [DBG] osdmap e504: 8 total, 8 up, 8 in 2021-02-10T04:35:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:22 smithi164 conmon[29288]: cluster 2021-02-10T04:35:22.100137+0000 mgr.y (mgr.14140) 592 : cluster [DBG] pgmap v906: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 395 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.5 MiB/s rd, 2.9 MiB/s wr, 16 op/s 2021-02-10T04:35:23.159 INFO:tasks.rados.rados.0.smithi085.stdout:1931: done (10 left) 2021-02-10T04:35:23.159 INFO:tasks.rados.rados.0.smithi085.stdout:1933: done (9 left) 2021-02-10T04:35:23.159 INFO:tasks.rados.rados.0.smithi085.stdout:1934: delete oid 13 current snap is 212 2021-02-10T04:35:23.163 INFO:tasks.rados.rados.0.smithi085.stdout:1934: done (9 left) 2021-02-10T04:35:23.163 INFO:tasks.rados.rados.0.smithi085.stdout:1935: write (excl) oid 50 current snap is 212 2021-02-10T04:35:23.164 INFO:tasks.rados.rados.0.smithi085.stdout:1935: seq_num 685 ranges {712795=417335,1784636=540572,2894750=457917} 2021-02-10T04:35:23.171 INFO:tasks.rados.rados.0.smithi085.stdout:1935: writing smithi08575397-50 from 712795 to 1130130 tid 1 2021-02-10T04:35:23.178 INFO:tasks.rados.rados.0.smithi085.stdout:1935: writing smithi08575397-50 from 1784636 to 2325208 tid 2 2021-02-10T04:35:23.185 INFO:tasks.rados.rados.0.smithi085.stdout:1935: writing smithi08575397-50 from 2894750 to 3352667 tid 3 2021-02-10T04:35:23.185 INFO:tasks.rados.rados.0.smithi085.stdout:1936: read oid 42 snap 148 2021-02-10T04:35:23.186 INFO:tasks.rados.rados.0.smithi085.stdout:1936: expect (ObjNum 204 snap 44 seq_num 204) 2021-02-10T04:35:23.186 INFO:tasks.rados.rados.0.smithi085.stdout:1937: rollback oid 22 current snap is 212 2021-02-10T04:35:23.186 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 22 to 210 2021-02-10T04:35:23.186 INFO:tasks.rados.rados.0.smithi085.stdout:1935: finishing write tid 1 to smithi08575397-50 2021-02-10T04:35:23.186 INFO:tasks.rados.rados.0.smithi085.stdout:1938: read oid 3 snap -1 2021-02-10T04:35:23.186 INFO:tasks.rados.rados.0.smithi085.stdout:1938: expect (ObjNum 674 snap 208 seq_num 674) 2021-02-10T04:35:23.186 INFO:tasks.rados.rados.0.smithi085.stdout:1939: snap_create 2021-02-10T04:35:23.189 INFO:tasks.rados.rados.0.smithi085.stdout:1935: finishing write tid 2 to smithi08575397-50 2021-02-10T04:35:23.192 INFO:tasks.rados.rados.0.smithi085.stdout:1935: finishing write tid 3 to smithi08575397-50 2021-02-10T04:35:23.192 INFO:tasks.rados.rados.0.smithi085.stdout:1938: expect (ObjNum 674 snap 208 seq_num 674) 2021-02-10T04:35:23.342 INFO:tasks.rados.rados.0.smithi085.stdout:1937: finishing rollback tid 0 to smithi08575397-22 2021-02-10T04:35:23.342 INFO:tasks.rados.rados.0.smithi085.stdout:1937: finishing rollback tid 1 to smithi08575397-22 2021-02-10T04:35:23.342 INFO:tasks.rados.rados.0.smithi085.stdout:1935: finishing write tid 4 to smithi08575397-50 2021-02-10T04:35:23.342 INFO:tasks.rados.rados.0.smithi085.stdout:1937: finishing rollback tid 2 to smithi08575397-22 2021-02-10T04:35:23.342 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 772 (ObjNum 654 snap 197 seq_num 654) dirty exists 2021-02-10T04:35:23.342 INFO:tasks.rados.rados.0.smithi085.stdout:1935: finishing write tid 5 to smithi08575397-50 2021-02-10T04:35:23.342 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 745 (ObjNum 685 snap 212 seq_num 685) dirty exists 2021-02-10T04:35:23.342 INFO:tasks.rados.rados.0.smithi085.stdout:1935: left oid 50 (ObjNum 685 snap 212 seq_num 685) 2021-02-10T04:35:23.342 INFO:tasks.rados.rados.0.smithi085.stdout:1936: expect (ObjNum 204 snap 44 seq_num 204) 2021-02-10T04:35:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:23 smithi164 conmon[29288]: cluster 2021-02-10T04:35:22.316182+0000 mon.a (mon.0) 1326 : cluster [DBG] 2021-02-10T04:35:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:23 smithi164 conmon[29288]: osdmap e505: 8 total, 8 up, 8 in 2021-02-10T04:35:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:23 smithi164 conmon[29288]: cluster 2021-02-10T04:35:23.156888+0000 mon.a (mon.0) 1327 : cluster [DBG] osdmap e506: 8 total, 8 up, 8 in 2021-02-10T04:35:23.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:23 smithi085 conmon[29319]: cluster 2021-02-10T04:35:22.316182+0000 mon.a 2021-02-10T04:35:23.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:23 smithi085 conmon[29319]: (mon.0) 1326 : cluster [DBG] osdmap e505: 8 total, 8 up, 8 in 2021-02-10T04:35:23.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:23 smithi085 conmon[29319]: cluster 2021-02-10T04:35:23.156888+0000 mon.a (mon.0) 1327 : cluster [DBG] osdmap e506: 8 total, 8 up, 8 in 2021-02-10T04:35:23.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:23 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:35:23.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:23 smithi085 conmon[35961]: 04:35:22.316182+0000 mon.a (mon.0) 1326 : cluster [DBG] osdmap e505: 8 total, 8 up, 8 in 2021-02-10T04:35:23.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:23 smithi085 conmon[35961]: cluster 2021-02-10T04:35:23 2021-02-10T04:35:23.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:23 smithi085 conmon[35961]: .156888+0000 mon.a (mon.0) 1327 : cluster [DBG] osdmap e506: 8 total, 8 up, 8 in 2021-02-10T04:35:24.128 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 4 1 2021-02-10T04:35:24.165 INFO:tasks.rados.rados.0.smithi085.stdout:1935: done (13 left) 2021-02-10T04:35:24.165 INFO:tasks.rados.rados.0.smithi085.stdout:1936: done (12 left) 2021-02-10T04:35:24.165 INFO:tasks.rados.rados.0.smithi085.stdout:1937: done (11 left) 2021-02-10T04:35:24.165 INFO:tasks.rados.rados.0.smithi085.stdout:1938: done (10 left) 2021-02-10T04:35:24.165 INFO:tasks.rados.rados.0.smithi085.stdout:1939: done (9 left) 2021-02-10T04:35:24.165 INFO:tasks.rados.rados.0.smithi085.stdout:1940: snap_create 2021-02-10T04:35:24.490 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:35:24.505 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:35:24.619 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:35:24.646 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:35:24.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:24 smithi085 conmon[49106]: debug 2021-02-10T04:35:24.429+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:35:24.893 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:35:24 smithi164 conmon[44544]: debug 2021-02-10T04:35:24.718+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:35:24.995 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:35:25.100 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:35:25.128 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:35:25.175 INFO:tasks.rados.rados.0.smithi085.stdout:1940: done (9 left) 2021-02-10T04:35:25.176 INFO:tasks.rados.rados.0.smithi085.stdout:1941: snap_remove snap 204 2021-02-10T04:35:25.290 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:35:25.290 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:35:25.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:25 smithi164 conmon[29288]: cluster 2021-02-10T04:35:24.100639+0000 mgr.y ( 2021-02-10T04:35:25.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:25 smithi164 conmon[29288]: mgr.14140) 593 : cluster [DBG] pgmap v909: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 391 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.9 MiB/s rd, 2.2 MiB/s wr, 11 op/s 2021-02-10T04:35:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:25 smithi164 conmon[29288]: cluster 2021-02-10T04:35:24.165266+0000 mon.a (mon.0) 1328 : cluster [DBG] osdmap e507: 8 total, 8 up, 8 in 2021-02-10T04:35:25.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:25 smithi164 conmon[33677]: debug 2021-02-10T04:35:25.204+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:35:25.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:25 smithi085 conmon[35961]: cluster 2021-02-10T04:35:24.100639 2021-02-10T04:35:25.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:25 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 593 : cluster [DBG] pgmap v909: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 391 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.9 MiB/s rd, 2.2 MiB/s wr, 11 op/s 2021-02-10T04:35:25.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:25 smithi085 conmon[35961]: cluster 2021-02-10T04:35:24. 2021-02-10T04:35:25.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:25 smithi085 conmon[35961]: 165266+0000 mon.a (mon.0) 1328 : cluster [DBG] osdmap e507: 8 total, 8 up, 8 in 2021-02-10T04:35:25.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:25 smithi085 conmon[29319]: cluster 2021-02-10T04:35:24.100639+0000 mgr.y (mgr.14140) 593 2021-02-10T04:35:25.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:25 smithi085 conmon[29319]: : cluster [DBG] pgmap v909: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 391 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.9 MiB/s rd, 2.2 MiB/s wr, 11 op/s 2021-02-10T04:35:25.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:25 smithi085 conmon[29319]: cluster 2021-02-10T04:35:24.165266+0000 mon.a (mon.0) 1328 : cluster [DBG] osdmap e507: 8 total, 8 up, 8 in 2021-02-10T04:35:25.450 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:35:25.556 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:35:25.593 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:35:26.183 INFO:tasks.rados.rados.0.smithi085.stdout:1941: done (9 left) 2021-02-10T04:35:26.184 INFO:tasks.rados.rados.0.smithi085.stdout:1942: read oid 11 snap -1 2021-02-10T04:35:26.184 INFO:tasks.rados.rados.0.smithi085.stdout:1942: expect (ObjNum 681 snap 210 seq_num 681) 2021-02-10T04:35:26.184 INFO:tasks.rados.rados.0.smithi085.stdout:1943: copy_from oid 28 from oid 36 current snap is 214 2021-02-10T04:35:26.184 INFO:tasks.rados.rados.0.smithi085.stdout:1944: write oid 5 current snap is 214 2021-02-10T04:35:26.184 INFO:tasks.rados.rados.0.smithi085.stdout:1944: seq_num 686 ranges {729602=492584,1843118=684230,3322173=207429} 2021-02-10T04:35:26.191 INFO:tasks.rados.rados.0.smithi085.stdout:1944: writing smithi08575397-5 from 729602 to 1222186 tid 1 2021-02-10T04:35:26.199 INFO:tasks.rados.rados.0.smithi085.stdout:1944: writing smithi08575397-5 from 1843118 to 2527348 tid 2 2021-02-10T04:35:26.205 INFO:tasks.rados.rados.0.smithi085.stdout:1944: writing smithi08575397-5 from 3322173 to 3529602 tid 3 2021-02-10T04:35:26.205 INFO:tasks.rados.rados.0.smithi085.stdout:1945: write (excl) oid 26 current snap is 214 2021-02-10T04:35:26.206 INFO:tasks.rados.rados.0.smithi085.stdout:1945: seq_num 687 ranges {746409=567833,1901600=544241,3065949=480460} 2021-02-10T04:35:26.213 INFO:tasks.rados.rados.0.smithi085.stdout:1945: writing smithi08575397-26 from 746409 to 1314242 tid 1 2021-02-10T04:35:26.220 INFO:tasks.rados.rados.0.smithi085.stdout:1945: writing smithi08575397-26 from 1901600 to 2445841 tid 2 2021-02-10T04:35:26.227 INFO:tasks.rados.rados.0.smithi085.stdout:1945: writing smithi08575397-26 from 3065949 to 3546409 tid 3 2021-02-10T04:35:26.227 INFO:tasks.rados.rados.0.smithi085.stdout:1946: copy_from oid 37 from oid 1 current snap is 214 2021-02-10T04:35:26.227 INFO:tasks.rados.rados.0.smithi085.stdout:1943: finishing copy_from to smithi08575397-28 2021-02-10T04:35:26.227 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 658 (ObjNum 616 snap 186 seq_num 616) dirty exists 2021-02-10T04:35:26.227 INFO:tasks.rados.rados.0.smithi085.stdout:1943: finishing copy_from racing read to smithi08575397-28 2021-02-10T04:35:26.227 INFO:tasks.rados.rados.0.smithi085.stdout:1942: expect (ObjNum 681 snap 210 seq_num 681) 2021-02-10T04:35:26.327 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:35:26.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:26 smithi085 conmon[29319]: cluster 2021-02-10T04:35:26.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:26 smithi085 conmon[29319]: 2021-02-10T04:35:26.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:26 smithi085 conmon[29319]: 2021-02-10T04:35:25.173512+0000 mon.a 2021-02-10T04:35:26.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:26 smithi085 conmon[29319]: (mon.0) 1329 : cluster [DBG] osdmap e508: 8 total, 8 up, 8 in 2021-02-10T04:35:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:26 smithi085 conmon[35961]: cluster 2021-02-10T04:35:25.173512+0000 2021-02-10T04:35:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:26 smithi085 conmon[35961]: mon.a (mon.0) 1329 : cluster [DBG] osdmap e508: 8 total, 8 up, 8 in 2021-02-10T04:35:26.445 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set nodeep-scrub 2021-02-10T04:35:26.446 INFO:tasks.rados.rados.0.smithi085.stdout:1944: finishing write tid 1 to smithi08575397-5 2021-02-10T04:35:26.446 INFO:tasks.rados.rados.0.smithi085.stdout:1944: finishing write tid 2 to smithi08575397-5 2021-02-10T04:35:26.447 INFO:tasks.rados.rados.0.smithi085.stdout:1944: finishing write tid 3 to smithi08575397-5 2021-02-10T04:35:26.447 INFO:tasks.rados.rados.0.smithi085.stdout:1944: finishing write tid 4 to smithi08575397-5 2021-02-10T04:35:26.447 INFO:tasks.rados.rados.0.smithi085.stdout:1944: finishing write tid 5 to smithi08575397-5 2021-02-10T04:35:26.447 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 686 (ObjNum 686 snap 214 seq_num 686) dirty exists 2021-02-10T04:35:26.447 INFO:tasks.rados.rados.0.smithi085.stdout:1944: left oid 5 (ObjNum 686 snap 214 seq_num 686) 2021-02-10T04:35:26.447 INFO:tasks.rados.rados.0.smithi085.stdout:1942: done (13 left) 2021-02-10T04:35:26.447 INFO:tasks.rados.rados.0.smithi085.stdout:1943: done (12 left) 2021-02-10T04:35:26.447 INFO:tasks.rados.rados.0.smithi085.stdout:1944: done (11 left) 2021-02-10T04:35:26.447 INFO:tasks.rados.rados.0.smithi085.stdout:1947: write (excl) oid 20 current snap is 214 2021-02-10T04:35:26.447 INFO:tasks.rados.rados.0.smithi085.stdout:1947: seq_num 688 ranges {763216=643082,1843729=687899,3260666=302550} 2021-02-10T04:35:26.455 INFO:tasks.rados.rados.0.smithi085.stdout:1947: writing smithi08575397-20 from 763216 to 1406298 tid 1 2021-02-10T04:35:26.464 INFO:tasks.rados.rados.0.smithi085.stdout:1947: writing smithi08575397-20 from 1843729 to 2531628 tid 2 2021-02-10T04:35:26.473 INFO:tasks.rados.rados.0.smithi085.stdout:1947: writing smithi08575397-20 from 3260666 to 3563216 tid 3 2021-02-10T04:35:26.473 INFO:tasks.rados.rados.0.smithi085.stdout:1948: delete oid 13 current snap is 214 2021-02-10T04:35:26.473 INFO:tasks.rados.rados.0.smithi085.stdout:1945: finishing write tid 1 to smithi08575397-26 2021-02-10T04:35:26.473 INFO:tasks.rados.rados.0.smithi085.stdout:1945: finishing write tid 2 to smithi08575397-26 2021-02-10T04:35:26.473 INFO:tasks.rados.rados.0.smithi085.stdout:1945: finishing write tid 3 to smithi08575397-26 2021-02-10T04:35:26.473 INFO:tasks.rados.rados.0.smithi085.stdout:1945: finishing write tid 4 to smithi08575397-26 2021-02-10T04:35:26.473 INFO:tasks.rados.rados.0.smithi085.stdout:1945: finishing write tid 5 to smithi08575397-26 2021-02-10T04:35:26.473 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 651 (ObjNum 687 snap 214 seq_num 687) dirty exists 2021-02-10T04:35:26.473 INFO:tasks.rados.rados.0.smithi085.stdout:1945: left oid 26 (ObjNum 687 snap 214 seq_num 687) 2021-02-10T04:35:26.473 INFO:tasks.rados.rados.0.smithi085.stdout:1946: finishing copy_from to smithi08575397-37 2021-02-10T04:35:26.473 INFO:tasks.rados.rados.0.smithi085.stdout:1946: got expected ENOENT (src dne) 2021-02-10T04:35:26.474 INFO:tasks.rados.rados.0.smithi085.stdout:1946: finishing copy_from racing read to smithi08575397-37 2021-02-10T04:35:26.474 INFO:tasks.rados.rados.0.smithi085.stdout:1947: finishing write tid 1 to smithi08575397-20 2021-02-10T04:35:26.474 INFO:tasks.rados.rados.0.smithi085.stdout:1947: finishing write tid 2 to smithi08575397-20 2021-02-10T04:35:26.475 INFO:tasks.rados.rados.0.smithi085.stdout:1947: finishing write tid 3 to smithi08575397-20 2021-02-10T04:35:26.475 INFO:tasks.rados.rados.0.smithi085.stdout:1947: finishing write tid 4 to smithi08575397-20 2021-02-10T04:35:26.476 INFO:tasks.rados.rados.0.smithi085.stdout:1945: done (12 left) 2021-02-10T04:35:26.476 INFO:tasks.rados.rados.0.smithi085.stdout:1946: done (11 left) 2021-02-10T04:35:26.476 INFO:tasks.rados.rados.0.smithi085.stdout:1948: done (10 left) 2021-02-10T04:35:26.476 INFO:tasks.rados.rados.0.smithi085.stdout:1949: snap_remove snap 201 2021-02-10T04:35:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:26 smithi164 conmon[29288]: cluster 2021-02-10T04:35:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:26 smithi164 conmon[29288]: 2021-02-10T04:35:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:26 smithi164 conmon[29288]: 2021 2021-02-10T04:35:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:26 smithi164 conmon[29288]: - 2021-02-10T04:35:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:26 smithi164 conmon[29288]: 02 2021-02-10T04:35:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:26 smithi164 conmon[29288]: - 2021-02-10T04:35:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:26 smithi164 conmon[29288]: 10 2021-02-10T04:35:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:26 smithi164 conmon[29288]: T 2021-02-10T04:35:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:26 smithi164 conmon[29288]: 04:35:25.173512 2021-02-10T04:35:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:26 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1329 : cluster [DBG] osdmap e508: 8 total, 8 up, 8 in 2021-02-10T04:35:26.659 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:35:26.790 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:35:27.183 INFO:tasks.rados.rados.0.smithi085.stdout:1949: done (10 left) 2021-02-10T04:35:27.183 INFO:tasks.rados.rados.0.smithi085.stdout:1950: snap_create 2021-02-10T04:35:27.184 INFO:tasks.rados.rados.0.smithi085.stdout:1947: finishing write tid 5 to smithi08575397-20 2021-02-10T04:35:27.184 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 20 v 741 (ObjNum 688 snap 214 seq_num 688) dirty exists 2021-02-10T04:35:27.184 INFO:tasks.rados.rados.0.smithi085.stdout:1947: left oid 20 (ObjNum 688 snap 214 seq_num 688) 2021-02-10T04:35:27.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:27 smithi085 conmon[29319]: cluster 2021-02-10T04:35:26.101198+0000 mgr.y 2021-02-10T04:35:27.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:27 smithi085 conmon[29319]: (mgr.14140) 594 : cluster [DBG] pgmap v912: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 357 MiB data, 978 MiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 2.3 MiB/s wr, 22 op/s 2021-02-10T04:35:27.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:27 smithi085 conmon[29319]: cluster 2021-02-10T04:35:26.180106+0000 mon.a (mon.0) 1330 : cluster [DBG] osdmap e509: 8 total, 8 up, 8 in 2021-02-10T04:35:27.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:27 smithi085 conmon[35961]: cluster 2021-02-10T04:35:26.101198 2021-02-10T04:35:27.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:27 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 594 : cluster [DBG] pgmap v912: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 357 MiB data, 978 MiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 2.3 MiB/s wr, 22 op/s 2021-02-10T04:35:27.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:27 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:35:27.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:27 smithi085 conmon[35961]: 35:26.180106+0000 mon.a (mon.0) 1330 : cluster [DBG] osdmap e509: 8 total, 8 up, 8 in 2021-02-10T04:35:27.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:27 smithi164 conmon[29288]: cluster 2021-02-10T04:35:26.101198+0000 mgr.y (mgr. 2021-02-10T04:35:27.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:27 smithi164 conmon[29288]: 14140) 594 : cluster [DBG] pgmap v912: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 357 MiB data, 978 MiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 2.3 MiB/s wr, 22 op/s 2021-02-10T04:35:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:27 smithi164 conmon[29288]: cluster 2021-02-10T04:35:26.180106+0000 mon.a (mon.0) 1330 : cluster [DBG] osdmap e509: 8 total, 8 up, 8 in 2021-02-10T04:35:28.163 INFO:tasks.rados.rados.0.smithi085.stdout:1947: done (10 left) 2021-02-10T04:35:28.164 INFO:tasks.rados.rados.0.smithi085.stdout:1950: done (9 left) 2021-02-10T04:35:28.164 INFO:tasks.rados.rados.0.smithi085.stdout:1951: delete oid 46 current snap is 215 2021-02-10T04:35:28.171 INFO:tasks.rados.rados.0.smithi085.stdout:1951: done (9 left) 2021-02-10T04:35:28.171 INFO:tasks.rados.rados.0.smithi085.stdout:1952: read oid 50 snap -1 2021-02-10T04:35:28.171 INFO:tasks.rados.rados.0.smithi085.stdout:1952: expect (ObjNum 685 snap 212 seq_num 685) 2021-02-10T04:35:28.171 INFO:tasks.rados.rados.0.smithi085.stdout:1953: delete oid 7 current snap is 215 2021-02-10T04:35:28.175 INFO:tasks.rados.rados.0.smithi085.stdout:1953: done (10 left) 2021-02-10T04:35:28.175 INFO:tasks.rados.rados.0.smithi085.stdout:1954: copy_from oid 31 from oid 10 current snap is 215 2021-02-10T04:35:28.175 INFO:tasks.rados.rados.0.smithi085.stdout:1955: write oid 8 current snap is 215 2021-02-10T04:35:28.175 INFO:tasks.rados.rados.0.smithi085.stdout:1955: seq_num 692 ranges {430444=660431,1677657=695237,2970358=569527} 2021-02-10T04:35:28.182 INFO:tasks.rados.rados.0.smithi085.stdout:1955: writing smithi08575397-8 from 430444 to 1090875 tid 1 2021-02-10T04:35:28.191 INFO:tasks.rados.rados.0.smithi085.stdout:1955: writing smithi08575397-8 from 1677657 to 2372894 tid 2 2021-02-10T04:35:28.199 INFO:tasks.rados.rados.0.smithi085.stdout:1955: writing smithi08575397-8 from 2970358 to 3539885 tid 3 2021-02-10T04:35:28.200 INFO:tasks.rados.rados.0.smithi085.stdout:1956: write (excl) oid 21 current snap is 215 2021-02-10T04:35:28.200 INFO:tasks.rados.rados.0.smithi085.stdout:1956: seq_num 693 ranges {447251=735680,1736139=438895,2597781=780799} 2021-02-10T04:35:28.207 INFO:tasks.rados.rados.0.smithi085.stdout:1956: writing smithi08575397-21 from 447251 to 1182931 tid 1 2021-02-10T04:35:28.213 INFO:tasks.rados.rados.0.smithi085.stdout:1956: writing smithi08575397-21 from 1736139 to 2175034 tid 2 2021-02-10T04:35:28.221 INFO:tasks.rados.rados.0.smithi085.stdout:1956: writing smithi08575397-21 from 2597781 to 3378580 tid 3 2021-02-10T04:35:28.221 INFO:tasks.rados.rados.0.smithi085.stdout:1957: write (excl) oid 36 current snap is 215 2021-02-10T04:35:28.221 INFO:tasks.rados.rados.0.smithi085.stdout:1952: expect (ObjNum 685 snap 212 seq_num 685) 2021-02-10T04:35:28.392 INFO:tasks.rados.rados.0.smithi085.stdout:1957: seq_num 694 ranges {464058=410929,1394621=698906,2625204=592071,3656107=7951} 2021-02-10T04:35:28.393 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:35:28 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:35:28.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:28 smithi164 conmon[29288]: cluster 2021-02-10T04:35:27.182830+0000 2021-02-10T04:35:28.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:28 smithi164 conmon[29288]: mon.a (mon.0) 1331 : cluster [DBG] osdmap e510: 8 total, 8 up, 8 in 2021-02-10T04:35:28.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:28 smithi164 conmon[29288]: audit 2021-02-10T04:35:28.087063+0000 mgr.y (mgr.14140) 595 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:35:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:28 smithi164 conmon[29288]: 2021-02-10T04:35:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:28 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:28 smithi164 conmon[29288]: -10T04:35:28.101708+0000 mgr.y (mgr.14140) 2021-02-10T04:35:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:28 smithi164 conmon[29288]: 596 : cluster [DBG] pgmap v915: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 357 MiB data, 969 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 1.5 MiB/s wr, 19 op/s 2021-02-10T04:35:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:28 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:35:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:28 smithi164 conmon[29288]: 04:35:28.161354+0000 mon.a (mon.0) 1332 : cluster 2021-02-10T04:35:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:28 smithi164 conmon[29288]: [DBG] osdmap e511: 8 total, 8 up, 8 in 2021-02-10T04:35:28.397 INFO:tasks.rados.rados.0.smithi085.stdout:1957: writing smithi08575397-36 from 464058 to 874987 tid 1 2021-02-10T04:35:28.405 INFO:tasks.rados.rados.0.smithi085.stdout:1957: writing smithi08575397-36 from 1394621 to 2093527 tid 2 2021-02-10T04:35:28.412 INFO:tasks.rados.rados.0.smithi085.stdout:1957: writing smithi08575397-36 from 2625204 to 3217275 tid 3 2021-02-10T04:35:28.415 INFO:tasks.rados.rados.0.smithi085.stdout:1957: writing smithi08575397-36 from 3656107 to 3664058 tid 4 2021-02-10T04:35:28.415 INFO:tasks.rados.rados.0.smithi085.stdout:1952: done (13 left) 2021-02-10T04:35:28.415 INFO:tasks.rados.rados.0.smithi085.stdout:1958: snap_create 2021-02-10T04:35:28.415 INFO:tasks.rados.rados.0.smithi085.stdout:1955: finishing write tid 1 to smithi08575397-8 2021-02-10T04:35:28.416 INFO:tasks.rados.rados.0.smithi085.stdout:1955: finishing write tid 2 to smithi08575397-8 2021-02-10T04:35:28.416 INFO:tasks.rados.rados.0.smithi085.stdout:1955: finishing write tid 3 to smithi08575397-8 2021-02-10T04:35:28.416 INFO:tasks.rados.rados.0.smithi085.stdout:1955: finishing write tid 4 to smithi08575397-8 2021-02-10T04:35:28.416 INFO:tasks.rados.rados.0.smithi085.stdout:1955: finishing write tid 5 to smithi08575397-8 2021-02-10T04:35:28.416 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 749 (ObjNum 692 snap 215 seq_num 692) dirty exists 2021-02-10T04:35:28.416 INFO:tasks.rados.rados.0.smithi085.stdout:1955: left oid 8 (ObjNum 692 snap 215 seq_num 692) 2021-02-10T04:35:28.416 INFO:tasks.rados.rados.0.smithi085.stdout:1956: finishing write tid 1 to smithi08575397-21 2021-02-10T04:35:28.416 INFO:tasks.rados.rados.0.smithi085.stdout:1956: finishing write tid 2 to smithi08575397-21 2021-02-10T04:35:28.416 INFO:tasks.rados.rados.0.smithi085.stdout:1956: finishing write tid 3 to smithi08575397-21 2021-02-10T04:35:28.416 INFO:tasks.rados.rados.0.smithi085.stdout:1956: finishing write tid 4 to smithi08575397-21 2021-02-10T04:35:28.416 INFO:tasks.rados.rados.0.smithi085.stdout:1956: finishing write tid 5 to smithi08575397-21 2021-02-10T04:35:28.416 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 1042 (ObjNum 693 snap 215 seq_num 693) dirty exists 2021-02-10T04:35:28.417 INFO:tasks.rados.rados.0.smithi085.stdout:1956: left oid 21 (ObjNum 693 snap 215 seq_num 693) 2021-02-10T04:35:28.417 INFO:tasks.rados.rados.0.smithi085.stdout:1954: finishing copy_from to smithi08575397-31 2021-02-10T04:35:28.417 INFO:tasks.rados.rados.0.smithi085.stdout:1954: got expected ENOENT (src dne) 2021-02-10T04:35:28.417 INFO:tasks.rados.rados.0.smithi085.stdout:1954: finishing copy_from racing read to smithi08575397-31 2021-02-10T04:35:28.417 INFO:tasks.rados.rados.0.smithi085.stdout:1957: finishing write tid 1 to smithi08575397-36 2021-02-10T04:35:28.417 INFO:tasks.rados.rados.0.smithi085.stdout:1957: finishing write tid 2 to smithi08575397-36 2021-02-10T04:35:28.417 INFO:tasks.rados.rados.0.smithi085.stdout:1957: finishing write tid 3 to smithi08575397-36 2021-02-10T04:35:28.419 INFO:tasks.rados.rados.0.smithi085.stdout:1957: finishing write tid 4 to smithi08575397-36 2021-02-10T04:35:28.419 INFO:tasks.rados.rados.0.smithi085.stdout:1957: finishing write tid 5 to smithi08575397-36 2021-02-10T04:35:28.419 INFO:tasks.rados.rados.0.smithi085.stdout:1957: finishing write tid 6 to smithi08575397-36 2021-02-10T04:35:28.419 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 673 (ObjNum 694 snap 215 seq_num 694) dirty exists 2021-02-10T04:35:28.419 INFO:tasks.rados.rados.0.smithi085.stdout:1957: left oid 36 (ObjNum 694 snap 215 seq_num 694) 2021-02-10T04:35:28.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[35961]: cluster 2021-02-10T04:35:27.182830+0000 mon.a ( 2021-02-10T04:35:28.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[35961]: mon.0) 1331 : cluster [DBG] osdmap e510: 8 total, 8 up, 8 in 2021-02-10T04:35:28.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[35961]: audit 2021-02-10T04:35:28. 2021-02-10T04:35:28.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[35961]: 087063+0000 mgr.y (mgr.14140) 595 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:35:28.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:35:28.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[35961]: 35:28.101708+0000 mgr.y (mgr.14140) 2021-02-10T04:35:28.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[35961]: 596 : cluster [DBG] pgmap v915: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 357 MiB data, 969 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 1.5 MiB/s wr, 19 op/s 2021-02-10T04:35:28.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:35:28.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[35961]: 04:35:28.161354+0000 mon.a (mon. 2021-02-10T04:35:28.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[35961]: 0) 1332 : cluster [DBG] osdmap e511: 8 total, 8 up, 8 in 2021-02-10T04:35:28.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[29319]: cluster 2021-02-10T04:35:28.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[29319]: 2021-02-10T04:35:28.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[29319]: 2021 2021-02-10T04:35:28.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[29319]: - 2021-02-10T04:35:28.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[29319]: 02 2021-02-10T04:35:28.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[29319]: - 2021-02-10T04:35:28.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[29319]: 10 2021-02-10T04:35:28.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[29319]: T 2021-02-10T04:35:28.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[29319]: 04 2021-02-10T04:35:28.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[29319]: : 2021-02-10T04:35:28.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[29319]: 35 2021-02-10T04:35:28.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[29319]: : 2021-02-10T04:35:28.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[29319]: 27 2021-02-10T04:35:28.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[29319]: . 2021-02-10T04:35:28.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[29319]: 182830 2021-02-10T04:35:28.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2021-02-10T04:35:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[29319]: 1331 : cluster [DBG] osdmap e510: 8 total, 8 up, 8 in 2021-02-10T04:35:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:35:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[29319]: 02-10T04:35:28.087063+0000 mgr.y (mgr.14140) 595 2021-02-10T04:35:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[29319]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:35:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:35:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[29319]: -02-10T04:35:28. 2021-02-10T04:35:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[29319]: 101708+0000 mgr.y (mgr.14140) 596 : cluster 2021-02-10T04:35:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[29319]: [DBG] pgmap v915: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 357 MiB data, 969 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 1.5 MiB/s wr, 19 op/s 2021-02-10T04:35:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:35:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[29319]: -10T04:35:28.161354+0000 mon.a 2021-02-10T04:35:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[29319]: (mon.0) 1332 : cluster [DBG] osdmap e511: 8 total, 8 up, 8 in 2021-02-10T04:35:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:28 smithi085 conmon[29319]: 2021-02-10T04:35:29.197 INFO:tasks.rados.rados.0.smithi085.stdout:1954: done (13 left) 2021-02-10T04:35:29.197 INFO:tasks.rados.rados.0.smithi085.stdout:1955: done (12 left) 2021-02-10T04:35:29.197 INFO:tasks.rados.rados.0.smithi085.stdout:1956: done (11 left) 2021-02-10T04:35:29.197 INFO:tasks.rados.rados.0.smithi085.stdout:1957: done (10 left) 2021-02-10T04:35:29.197 INFO:tasks.rados.rados.0.smithi085.stdout:1958: done (9 left) 2021-02-10T04:35:29.197 INFO:tasks.rados.rados.0.smithi085.stdout:1959: read oid 14 snap 203 2021-02-10T04:35:29.197 INFO:tasks.rados.rados.0.smithi085.stdout:1959: expect (ObjNum 651 snap 197 seq_num 651) 2021-02-10T04:35:29.197 INFO:tasks.rados.rados.0.smithi085.stdout:1960: read oid 40 snap -1 2021-02-10T04:35:29.197 INFO:tasks.rados.rados.0.smithi085.stdout:1960: expect (ObjNum 660 snap 202 seq_num 660) 2021-02-10T04:35:29.197 INFO:tasks.rados.rados.0.smithi085.stdout:1961: rollback oid 20 current snap is 216 2021-02-10T04:35:29.198 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 20 to 189 2021-02-10T04:35:29.198 INFO:tasks.rados.rados.0.smithi085.stdout:1962: read oid 2 snap -1 2021-02-10T04:35:29.198 INFO:tasks.rados.rados.0.smithi085.stdout:1962: expect (ObjNum 422 snap 109 seq_num 422) 2021-02-10T04:35:29.198 INFO:tasks.rados.rados.0.smithi085.stdout:1963: write oid 31 current snap is 216 2021-02-10T04:35:29.198 INFO:tasks.rados.rados.0.smithi085.stdout:1963: seq_num 695 ranges {480865=486178,1453103=442564,2652627=519696} 2021-02-10T04:35:29.203 INFO:tasks.rados.rados.0.smithi085.stdout:1963: writing smithi08575397-31 from 480865 to 967043 tid 1 2021-02-10T04:35:29.208 INFO:tasks.rados.rados.0.smithi085.stdout:1963: writing smithi08575397-31 from 1453103 to 1895667 tid 2 2021-02-10T04:35:29.219 INFO:tasks.rados.rados.0.smithi085.stdout:1963: writing smithi08575397-31 from 2652627 to 3172323 tid 3 2021-02-10T04:35:29.219 INFO:tasks.rados.rados.0.smithi085.stdout:1964: read oid 43 snap -1 2021-02-10T04:35:29.220 INFO:tasks.rados.rados.0.smithi085.stdout:1964: expect (ObjNum 670 snap 205 seq_num 670) 2021-02-10T04:35:29.220 INFO:tasks.rados.rados.0.smithi085.stdout:1965: snap_create 2021-02-10T04:35:29.220 INFO:tasks.rados.rados.0.smithi085.stdout:1962: expect (ObjNum 422 snap 109 seq_num 422) 2021-02-10T04:35:29.464 INFO:tasks.rados.rados.0.smithi085.stdout:1961: finishing rollback tid 0 to smithi08575397-20 2021-02-10T04:35:29.464 INFO:tasks.rados.rados.0.smithi085.stdout:1961: finishing rollback tid 1 to smithi08575397-20 2021-02-10T04:35:29.464 INFO:tasks.rados.rados.0.smithi085.stdout:1959: expect (ObjNum 651 snap 197 seq_num 651) 2021-02-10T04:35:29.685 INFO:tasks.rados.rados.0.smithi085.stdout:1961: finishing rollback tid 2 to smithi08575397-20 2021-02-10T04:35:29.685 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 20 v 744 (ObjNum 596 snap 178 seq_num 596) dirty exists 2021-02-10T04:35:29.685 INFO:tasks.rados.rados.0.smithi085.stdout:1960: expect (ObjNum 660 snap 202 seq_num 660) 2021-02-10T04:35:29.822 INFO:tasks.rados.rados.0.smithi085.stdout:1963: finishing write tid 1 to smithi08575397-31 2021-02-10T04:35:29.823 INFO:tasks.rados.rados.0.smithi085.stdout:1963: finishing write tid 2 to smithi08575397-31 2021-02-10T04:35:29.823 INFO:tasks.rados.rados.0.smithi085.stdout:1964: expect (ObjNum 670 snap 205 seq_num 670) 2021-02-10T04:35:29.967 INFO:tasks.rados.rados.0.smithi085.stdout:1963: finishing write tid 3 to smithi08575397-31 2021-02-10T04:35:29.967 INFO:tasks.rados.rados.0.smithi085.stdout:1963: finishing write tid 4 to smithi08575397-31 2021-02-10T04:35:29.967 INFO:tasks.rados.rados.0.smithi085.stdout:1963: finishing write tid 5 to smithi08575397-31 2021-02-10T04:35:29.967 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 776 (ObjNum 695 snap 216 seq_num 695) dirty exists 2021-02-10T04:35:29.967 INFO:tasks.rados.rados.0.smithi085.stdout:1963: left oid 31 (ObjNum 695 snap 216 seq_num 695) 2021-02-10T04:35:30.204 INFO:tasks.rados.rados.0.smithi085.stdout:1959: done (15 left) 2021-02-10T04:35:30.204 INFO:tasks.rados.rados.0.smithi085.stdout:1960: done (14 left) 2021-02-10T04:35:30.204 INFO:tasks.rados.rados.0.smithi085.stdout:1961: done (13 left) 2021-02-10T04:35:30.204 INFO:tasks.rados.rados.0.smithi085.stdout:1962: done (12 left) 2021-02-10T04:35:30.204 INFO:tasks.rados.rados.0.smithi085.stdout:1963: done (11 left) 2021-02-10T04:35:30.204 INFO:tasks.rados.rados.0.smithi085.stdout:1964: done (10 left) 2021-02-10T04:35:30.204 INFO:tasks.rados.rados.0.smithi085.stdout:1965: done (9 left) 2021-02-10T04:35:30.204 INFO:tasks.rados.rados.0.smithi085.stdout:1966: rollback oid 22 current snap is 217 2021-02-10T04:35:30.204 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 22 to 184 2021-02-10T04:35:30.204 INFO:tasks.rados.rados.0.smithi085.stdout:1967: snap_create 2021-02-10T04:35:30.211 INFO:tasks.rados.rados.0.smithi085.stdout:1966: finishing rollback tid 0 to smithi08575397-22 2021-02-10T04:35:30.211 INFO:tasks.rados.rados.0.smithi085.stdout:1966: finishing rollback tid 1 to smithi08575397-22 2021-02-10T04:35:30.212 INFO:tasks.rados.rados.0.smithi085.stdout:1966: finishing rollback tid 2 to smithi08575397-22 2021-02-10T04:35:30.212 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 779 (ObjNum 570 snap 175 seq_num 570) dirty exists 2021-02-10T04:35:30.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:30 smithi164 conmon[29288]: cluster 2021-02-10T04:35:28.829611+0000 osd.3 (osd.3) 68 2021-02-10T04:35:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:30 smithi164 conmon[29288]: : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:35:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:30 smithi164 conmon[29288]: cluster 2021-02-10T04:35:28.830112+0000 osd.3 (osd.3) 69 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:35:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:30 smithi164 conmon[29288]: cluster 2021-02-10T04:35:29.196306+0000 mon.a ( 2021-02-10T04:35:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:30 smithi164 conmon[29288]: mon.0) 1333 : cluster [DBG] osdmap e512: 8 total, 8 up, 8 in 2021-02-10T04:35:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:30 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:35:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:30 smithi164 conmon[29288]: 02-10T04:35:30.102203+0000 mgr.y ( 2021-02-10T04:35:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:30 smithi164 conmon[29288]: mgr.14140) 597 : cluster [DBG] pgmap v918: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 354 MiB data, 959 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2021-02-10T04:35:30.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:30 smithi085 conmon[29319]: cluster 2021-02-10T04:35:28.829611+0000 osd.3 ( 2021-02-10T04:35:30.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:30 smithi085 conmon[29319]: osd.3) 68 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:35:30.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:30 smithi085 conmon[29319]: cluster 2021-02-10T04:35:28.830112+0000 osd.3 (osd.3) 69 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:35:30.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:30 smithi085 conmon[29319]: cluster 2021-02-10T04:35:29.196306+0000 mon.a (mon.0) 1333 : cluster 2021-02-10T04:35:30.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:30 smithi085 conmon[29319]: [DBG] osdmap e512: 8 total, 8 up, 8 in 2021-02-10T04:35:30.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:30 smithi085 conmon[29319]: cluster 2021-02-10T04:35:30.102203+0000 mgr.y (mgr.14140 2021-02-10T04:35:30.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:30 smithi085 conmon[29319]: ) 597 : cluster [DBG] pgmap v918: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 354 MiB data, 959 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2021-02-10T04:35:30.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:30 smithi085 conmon[35961]: cluster 2021-02-10T04:35:28.829611 2021-02-10T04:35:30.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:30 smithi085 conmon[35961]: +0000 osd.3 (osd.3) 68 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:35:30.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:30 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:35:30.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:30 smithi085 conmon[35961]: -10T04:35:28.830112+0000 osd.3 2021-02-10T04:35:30.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:30 smithi085 conmon[35961]: (osd.3) 69 : cluster [DBG] 2021-02-10T04:35:30.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:30 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:35:30.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:30 smithi085 conmon[35961]: cluster 2021-02-10T04:35:30.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:30 smithi085 conmon[35961]: 2021-02-10T04:35:30.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:30 smithi085 conmon[35961]: 2021 2021-02-10T04:35:30.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:30 smithi085 conmon[35961]: - 2021-02-10T04:35:30.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:30 smithi085 conmon[35961]: 02 2021-02-10T04:35:30.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:30 smithi085 conmon[35961]: - 2021-02-10T04:35:30.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:30 smithi085 conmon[35961]: 10 2021-02-10T04:35:30.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:30 smithi085 conmon[35961]: T 2021-02-10T04:35:30.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:30 smithi085 conmon[35961]: 04:35:29.196306+0000 mon.a 2021-02-10T04:35:30.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:30 smithi085 conmon[35961]: (mon.0) 1333 : cluster [DBG] osdmap e512: 8 total, 8 up, 8 in 2021-02-10T04:35:30.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:30 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:30.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:30 smithi085 conmon[35961]: 02-10T04:35:30.102203+0000 mgr.y (mgr.14140) 2021-02-10T04:35:30.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:30 smithi085 conmon[35961]: 597 : cluster [DBG] pgmap v918: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 354 MiB data, 959 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2021-02-10T04:35:31.212 INFO:tasks.rados.rados.0.smithi085.stdout:1966: done (10 left) 2021-02-10T04:35:31.212 INFO:tasks.rados.rados.0.smithi085.stdout:1967: done (9 left) 2021-02-10T04:35:31.212 INFO:tasks.rados.rados.0.smithi085.stdout:1968: write oid 37 current snap is 218 2021-02-10T04:35:31.212 INFO:tasks.rados.rados.0.smithi085.stdout:1968: seq_num 696 ranges {497672=561427,1795232=702575,2963697=730968} 2021-02-10T04:35:31.219 INFO:tasks.rados.rados.0.smithi085.stdout:1968: writing smithi08575397-37 from 497672 to 1059099 tid 1 2021-02-10T04:35:31.229 INFO:tasks.rados.rados.0.smithi085.stdout:1968: writing smithi08575397-37 from 1795232 to 2497807 tid 2 2021-02-10T04:35:31.237 INFO:tasks.rados.rados.0.smithi085.stdout:1968: writing smithi08575397-37 from 2963697 to 3694665 tid 3 2021-02-10T04:35:31.237 INFO:tasks.rados.rados.0.smithi085.stdout:1968: finishing write tid 1 to smithi08575397-37 2021-02-10T04:35:31.237 INFO:tasks.rados.rados.0.smithi085.stdout:1968: finishing write tid 2 to smithi08575397-37 2021-02-10T04:35:31.237 INFO:tasks.rados.rados.0.smithi085.stdout:1969: write oid 8 current snap is 218 2021-02-10T04:35:31.237 INFO:tasks.rados.rados.0.smithi085.stdout:1969: seq_num 697 ranges {514479=636676,1853714=446233,2991120=542240} 2021-02-10T04:35:31.244 INFO:tasks.rados.rados.0.smithi085.stdout:1969: writing smithi08575397-8 from 514479 to 1151155 tid 1 2021-02-10T04:35:31.251 INFO:tasks.rados.rados.0.smithi085.stdout:1969: writing smithi08575397-8 from 1853714 to 2299947 tid 2 2021-02-10T04:35:31.259 INFO:tasks.rados.rados.0.smithi085.stdout:1969: writing smithi08575397-8 from 2991120 to 3533360 tid 3 2021-02-10T04:35:31.259 INFO:tasks.rados.rados.0.smithi085.stdout:1970: write (excl) oid 6 current snap is 218 2021-02-10T04:35:31.259 INFO:tasks.rados.rados.0.smithi085.stdout:1970: seq_num 698 ranges {531286=711925,1912196=706244,3018543=712743} 2021-02-10T04:35:31.267 INFO:tasks.rados.rados.0.smithi085.stdout:1970: writing smithi08575397-6 from 531286 to 1243211 tid 1 2021-02-10T04:35:31.275 INFO:tasks.rados.rados.0.smithi085.stdout:1970: writing smithi08575397-6 from 1912196 to 2618440 tid 2 2021-02-10T04:35:31.283 INFO:tasks.rados.rados.0.smithi085.stdout:1970: writing smithi08575397-6 from 3018543 to 3731286 tid 3 2021-02-10T04:35:31.284 INFO:tasks.rados.rados.0.smithi085.stdout:1971: copy_from oid 35 from oid 2 current snap is 218 2021-02-10T04:35:31.284 INFO:tasks.rados.rados.0.smithi085.stdout:1968: finishing write tid 3 to smithi08575397-37 2021-02-10T04:35:31.284 INFO:tasks.rados.rados.0.smithi085.stdout:1968: finishing write tid 4 to smithi08575397-37 2021-02-10T04:35:31.284 INFO:tasks.rados.rados.0.smithi085.stdout:1968: finishing write tid 5 to smithi08575397-37 2021-02-10T04:35:31.284 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 1049 (ObjNum 696 snap 218 seq_num 696) dirty exists 2021-02-10T04:35:31.284 INFO:tasks.rados.rados.0.smithi085.stdout:1968: left oid 37 (ObjNum 696 snap 218 seq_num 696) 2021-02-10T04:35:31.284 INFO:tasks.rados.rados.0.smithi085.stdout:1969: finishing write tid 1 to smithi08575397-8 2021-02-10T04:35:31.284 INFO:tasks.rados.rados.0.smithi085.stdout:1969: finishing write tid 2 to smithi08575397-8 2021-02-10T04:35:31.284 INFO:tasks.rados.rados.0.smithi085.stdout:1969: finishing write tid 3 to smithi08575397-8 2021-02-10T04:35:31.284 INFO:tasks.rados.rados.0.smithi085.stdout:1968: done (12 left) 2021-02-10T04:35:31.284 INFO:tasks.rados.rados.0.smithi085.stdout:1972: rollback oid 7 current snap is 218 2021-02-10T04:35:31.284 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 7 to 210 2021-02-10T04:35:31.284 INFO:tasks.rados.rados.0.smithi085.stdout:1969: finishing write tid 4 to smithi08575397-8 2021-02-10T04:35:31.285 INFO:tasks.rados.rados.0.smithi085.stdout:1969: finishing write tid 5 to smithi08575397-8 2021-02-10T04:35:31.285 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 753 (ObjNum 697 snap 218 seq_num 697) dirty exists 2021-02-10T04:35:31.285 INFO:tasks.rados.rados.0.smithi085.stdout:1969: left oid 8 (ObjNum 697 snap 218 seq_num 697) 2021-02-10T04:35:31.285 INFO:tasks.rados.rados.0.smithi085.stdout:1970: finishing write tid 1 to smithi08575397-6 2021-02-10T04:35:31.285 INFO:tasks.rados.rados.0.smithi085.stdout:1970: finishing write tid 2 to smithi08575397-6 2021-02-10T04:35:31.285 INFO:tasks.rados.rados.0.smithi085.stdout:1969: done (12 left) 2021-02-10T04:35:31.285 INFO:tasks.rados.rados.0.smithi085.stdout:1973: copy_from oid 38 from oid 11 current snap is 218 2021-02-10T04:35:31.285 INFO:tasks.rados.rados.0.smithi085.stdout:1974: rollback oid 31 current snap is 218 2021-02-10T04:35:31.285 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 31 to 184 2021-02-10T04:35:31.285 INFO:tasks.rados.rados.0.smithi085.stdout:1975: read oid 44 snap -1 2021-02-10T04:35:31.285 INFO:tasks.rados.rados.0.smithi085.stdout:1975: expect (ObjNum 671 snap 207 seq_num 671) 2021-02-10T04:35:31.285 INFO:tasks.rados.rados.0.smithi085.stdout:1976: write oid 8 current snap is 218 2021-02-10T04:35:31.285 INFO:tasks.rados.rados.0.smithi085.stdout:1976: seq_num 699 ranges {548093=787174,1970678=449902,3045966=681137} 2021-02-10T04:35:31.292 INFO:tasks.rados.rados.0.smithi085.stdout:1976: writing smithi08575397-8 from 548093 to 1335267 tid 1 2021-02-10T04:35:31.298 INFO:tasks.rados.rados.0.smithi085.stdout:1976: writing smithi08575397-8 from 1970678 to 2420580 tid 2 2021-02-10T04:35:31.307 INFO:tasks.rados.rados.0.smithi085.stdout:1976: writing smithi08575397-8 from 3045966 to 3727103 tid 3 2021-02-10T04:35:31.307 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:35:31.307 INFO:tasks.rados.rados.0.smithi085.stdout:1975: expect (ObjNum 671 snap 207 seq_num 671) 2021-02-10T04:35:31.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:31 smithi085 conmon[29319]: cluster 2021-02-10T04:35:30.202504+0000 mon.a (mon.0) 1334 : cluster [DBG] osdmap e513: 8 total, 8 up, 8 in 2021-02-10T04:35:31.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:31 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:35:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:31 smithi085 conmon[35961]: 04:35:30.202504+0000 mon.a (mon.0) 1334 : cluster [DBG] osdmap e513: 8 total, 8 up, 8 in 2021-02-10T04:35:31.606 INFO:tasks.rados.rados.0.smithi085.stdout:1970: finishing write tid 3 to smithi08575397-6 2021-02-10T04:35:31.606 INFO:tasks.rados.rados.0.smithi085.stdout:1970: finishing write tid 4 to smithi08575397-6 2021-02-10T04:35:31.606 INFO:tasks.rados.rados.0.smithi085.stdout:1970: finishing write tid 5 to smithi08575397-6 2021-02-10T04:35:31.607 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 782 (ObjNum 698 snap 218 seq_num 698) dirty exists 2021-02-10T04:35:31.607 INFO:tasks.rados.rados.0.smithi085.stdout:1970: left oid 6 (ObjNum 698 snap 218 seq_num 698) 2021-02-10T04:35:31.607 INFO:tasks.rados.rados.0.smithi085.stdout:1972: finishing rollback tid 1 to smithi08575397-7 2021-02-10T04:35:31.607 INFO:tasks.rados.rados.0.smithi085.stdout:1972: finishing rollback tid 2 to smithi08575397-7 2021-02-10T04:35:31.607 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 736 (ObjNum 204 snap 44 seq_num 204) dirty exists 2021-02-10T04:35:31.607 INFO:tasks.rados.rados.0.smithi085.stdout:1974: finishing rollback tid 0 to smithi08575397-31 2021-02-10T04:35:31.607 INFO:tasks.rados.rados.0.smithi085.stdout:1974: finishing rollback tid 1 to smithi08575397-31 2021-02-10T04:35:31.607 INFO:tasks.rados.rados.0.smithi085.stdout:1974: finishing rollback tid 2 to smithi08575397-31 2021-02-10T04:35:31.607 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 782 (ObjNum 569 snap 175 seq_num 569) dirty exists 2021-02-10T04:35:31.607 INFO:tasks.rados.rados.0.smithi085.stdout:1976: finishing write tid 1 to smithi08575397-8 2021-02-10T04:35:31.607 INFO:tasks.rados.rados.0.smithi085.stdout:1971: finishing copy_from to smithi08575397-35 2021-02-10T04:35:31.607 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 35 v 661 (ObjNum 422 snap 109 seq_num 422) dirty exists 2021-02-10T04:35:31.607 INFO:tasks.rados.rados.0.smithi085.stdout:1971: finishing copy_from racing read to smithi08575397-35 2021-02-10T04:35:31.608 INFO:tasks.rados.rados.0.smithi085.stdout:1976: finishing write tid 2 to smithi08575397-8 2021-02-10T04:35:31.608 INFO:tasks.rados.rados.0.smithi085.stdout:1976: finishing write tid 3 to smithi08575397-8 2021-02-10T04:35:31.608 INFO:tasks.rados.rados.0.smithi085.stdout:1976: finishing write tid 4 to smithi08575397-8 2021-02-10T04:35:31.608 INFO:tasks.rados.rados.0.smithi085.stdout:1976: finishing write tid 5 to smithi08575397-8 2021-02-10T04:35:31.608 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 757 (ObjNum 699 snap 218 seq_num 699) dirty exists 2021-02-10T04:35:31.608 INFO:tasks.rados.rados.0.smithi085.stdout:1976: left oid 8 (ObjNum 699 snap 218 seq_num 699) 2021-02-10T04:35:31.608 INFO:tasks.rados.rados.0.smithi085.stdout:1973: finishing copy_from to smithi08575397-38 2021-02-10T04:35:31.608 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 1019 (ObjNum 681 snap 210 seq_num 681) dirty exists 2021-02-10T04:35:31.608 INFO:tasks.rados.rados.0.smithi085.stdout:1973: finishing copy_from racing read to smithi08575397-38 2021-02-10T04:35:31.608 INFO:tasks.rados.rados.0.smithi085.stdout:1970: done (15 left) 2021-02-10T04:35:31.608 INFO:tasks.rados.rados.0.smithi085.stdout:1971: done (14 left) 2021-02-10T04:35:31.608 INFO:tasks.rados.rados.0.smithi085.stdout:1972: done (13 left) 2021-02-10T04:35:31.608 INFO:tasks.rados.rados.0.smithi085.stdout:1973: done (12 left) 2021-02-10T04:35:31.609 INFO:tasks.rados.rados.0.smithi085.stdout:1974: done (11 left) 2021-02-10T04:35:31.609 INFO:tasks.rados.rados.0.smithi085.stdout:1975: done (10 left) 2021-02-10T04:35:31.609 INFO:tasks.rados.rados.0.smithi085.stdout:1976: done (9 left) 2021-02-10T04:35:31.609 INFO:tasks.rados.rados.0.smithi085.stdout:1977: write (excl) oid 8 current snap is 218 2021-02-10T04:35:31.609 INFO:tasks.rados.rados.0.smithi085.stdout:1977: seq_num 700 ranges {564900=578776,1629160=709913,3073389=492409} 2021-02-10T04:35:31.612 INFO:tasks.rados.rados.0.smithi085.stdout:1977: writing smithi08575397-8 from 564900 to 1143676 tid 1 2021-02-10T04:35:31.620 INFO:tasks.rados.rados.0.smithi085.stdout:1977: writing smithi08575397-8 from 1629160 to 2339073 tid 2 2021-02-10T04:35:31.628 INFO:tasks.rados.rados.0.smithi085.stdout:1977: writing smithi08575397-8 from 3073389 to 3565798 tid 3 2021-02-10T04:35:31.629 INFO:tasks.rados.rados.0.smithi085.stdout:1977: finishing write tid 1 to smithi08575397-8 2021-02-10T04:35:31.629 INFO:tasks.rados.rados.0.smithi085.stdout:1978: delete oid 37 current snap is 218 2021-02-10T04:35:31.631 INFO:tasks.rados.rados.0.smithi085.stdout:1977: finishing write tid 2 to smithi08575397-8 2021-02-10T04:35:31.633 INFO:tasks.rados.rados.0.smithi085.stdout:1977: finishing write tid 3 to smithi08575397-8 2021-02-10T04:35:31.634 INFO:tasks.rados.rados.0.smithi085.stdout:1977: finishing write tid 4 to smithi08575397-8 2021-02-10T04:35:31.634 INFO:tasks.rados.rados.0.smithi085.stdout:1977: finishing write tid 5 to smithi08575397-8 2021-02-10T04:35:31.634 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 761 (ObjNum 700 snap 218 seq_num 700) dirty exists 2021-02-10T04:35:31.634 INFO:tasks.rados.rados.0.smithi085.stdout:1977: left oid 8 (ObjNum 700 snap 218 seq_num 700) 2021-02-10T04:35:31.634 INFO:tasks.rados.rados.0.smithi085.stdout:1977: done (10 left) 2021-02-10T04:35:31.634 INFO:tasks.rados.rados.0.smithi085.stdout:1978: done (9 left) 2021-02-10T04:35:31.634 INFO:tasks.rados.rados.0.smithi085.stdout:1979: rollback oid 2 current snap is 218 2021-02-10T04:35:31.634 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 2 to 200 2021-02-10T04:35:31.634 INFO:tasks.rados.rados.0.smithi085.stdout:1980: write oid 40 current snap is 218 2021-02-10T04:35:31.634 INFO:tasks.rados.rados.0.smithi085.stdout:1980: seq_num 702 ranges {598514=729274,1746124=597229,3128235=514953} 2021-02-10T04:35:31.642 INFO:tasks.rados.rados.0.smithi085.stdout:1980: writing smithi08575397-40 from 598514 to 1327788 tid 1 2021-02-10T04:35:31.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:31 smithi164 conmon[29288]: cluster 2021-02-10T04:35:30.202504+0000 mon.a (mon.0) 2021-02-10T04:35:31.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:31 smithi164 conmon[29288]: 1334 : cluster [DBG] osdmap e513: 8 total, 8 up, 8 in 2021-02-10T04:35:31.649 INFO:tasks.rados.rados.0.smithi085.stdout:1980: writing smithi08575397-40 from 1746124 to 2343353 tid 2 2021-02-10T04:35:31.657 INFO:tasks.rados.rados.0.smithi085.stdout:1980: writing smithi08575397-40 from 3128235 to 3643188 tid 3 2021-02-10T04:35:31.658 INFO:tasks.rados.rados.0.smithi085.stdout:1981: rollback oid 38 current snap is 218 2021-02-10T04:35:31.658 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 38 to 200 2021-02-10T04:35:31.658 INFO:tasks.rados.rados.0.smithi085.stdout:1979: finishing rollback tid 0 to smithi08575397-2 2021-02-10T04:35:31.658 INFO:tasks.rados.rados.0.smithi085.stdout:1979: finishing rollback tid 1 to smithi08575397-2 2021-02-10T04:35:31.658 INFO:tasks.rados.rados.0.smithi085.stdout:1979: finishing rollback tid 2 to smithi08575397-2 2021-02-10T04:35:31.658 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 664 (ObjNum 652 snap 197 seq_num 652) dirty exists 2021-02-10T04:35:31.658 INFO:tasks.rados.rados.0.smithi085.stdout:1980: finishing write tid 1 to smithi08575397-40 2021-02-10T04:35:31.658 INFO:tasks.rados.rados.0.smithi085.stdout:1980: finishing write tid 2 to smithi08575397-40 2021-02-10T04:35:31.658 INFO:tasks.rados.rados.0.smithi085.stdout:1979: done (11 left) 2021-02-10T04:35:31.658 INFO:tasks.rados.rados.0.smithi085.stdout:1982: snap_create 2021-02-10T04:35:31.660 INFO:tasks.rados.rados.0.smithi085.stdout:1981: finishing rollback tid 0 to smithi08575397-38 2021-02-10T04:35:31.661 INFO:tasks.rados.rados.0.smithi085.stdout:1981: finishing rollback tid 1 to smithi08575397-38 2021-02-10T04:35:31.662 INFO:tasks.rados.rados.0.smithi085.stdout:1980: finishing write tid 3 to smithi08575397-40 2021-02-10T04:35:31.662 INFO:tasks.rados.rados.0.smithi085.stdout:1981: finishing rollback tid 2 to smithi08575397-38 2021-02-10T04:35:31.663 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 1022 (ObjNum 616 snap 186 seq_num 616) dirty exists 2021-02-10T04:35:31.663 INFO:tasks.rados.rados.0.smithi085.stdout:1980: finishing write tid 4 to smithi08575397-40 2021-02-10T04:35:31.663 INFO:tasks.rados.rados.0.smithi085.stdout:1980: finishing write tid 5 to smithi08575397-40 2021-02-10T04:35:31.663 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 1059 (ObjNum 702 snap 218 seq_num 702) dirty exists 2021-02-10T04:35:31.663 INFO:tasks.rados.rados.0.smithi085.stdout:1980: left oid 40 (ObjNum 702 snap 218 seq_num 702) 2021-02-10T04:35:32.088 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:35:32.191 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:35:31 smithi085 conmon[41540]: debug 2021-02-10T04:35:32.191 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:35:31 smithi085 conmon[41540]: 2021-02-10T04:35:31.825+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:35:32.220 INFO:tasks.rados.rados.0.smithi085.stdout:1980: done (11 left) 2021-02-10T04:35:32.221 INFO:tasks.rados.rados.0.smithi085.stdout:1981: done (10 left) 2021-02-10T04:35:32.221 INFO:tasks.rados.rados.0.smithi085.stdout:1982: done (9 left) 2021-02-10T04:35:32.221 INFO:tasks.rados.rados.0.smithi085.stdout:1983: read oid 48 snap -1 2021-02-10T04:35:32.221 INFO:tasks.rados.rados.0.smithi085.stdout:1983: expect (ObjNum 422 snap 109 seq_num 422) 2021-02-10T04:35:32.221 INFO:tasks.rados.rados.0.smithi085.stdout:1984: read oid 10 snap 192 2021-02-10T04:35:32.221 INFO:tasks.rados.rados.0.smithi085.stdout:1984: expect deleted 2021-02-10T04:35:32.221 INFO:tasks.rados.rados.0.smithi085.stdout:1985: snap_create 2021-02-10T04:35:32.242 INFO:tasks.rados.rados.0.smithi085.stdout:1983: expect (ObjNum 422 snap 109 seq_num 422) 2021-02-10T04:35:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:32 smithi164 conmon[29288]: cluster 2021-02-10T04:35:31.211773+0000 mon.a (mon. 2021-02-10T04:35:39.841 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:32 smithi164 conmon[29288]: 0) 1335 : cluster [DBG] osdmap e514: 8 total, 8 up, 8 in 2021-02-10T04:35:39.842 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:32 smithi164 conmon[29288]: cluster 2021-02-10T04:35:32.102803+0000 mgr.y (mgr.14140) 598 : cluster [DBG] pgmap v921: 168 pgs: 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 156 active+clean; 354 MiB data, 959 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 3.0 MiB/s wr, 24 op/s 2021-02-10T04:35:39.852 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:32 smithi085 conmon[29319]: cluster 2021-02-10T04:35:31.211773+0000 mon.a (mon.0) 1335 : cluster [DBG] osdmap e514: 8 total, 8 up, 8 in 2021-02-10T04:35:39.853 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:32 smithi085 conmon[29319]: cluster 2021-02-10T04:35:32.102803+0000 mgr.y (mgr.14140) 598 : cluster [DBG] pgmap v921: 168 pgs: 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 156 active+clean; 354 MiB data, 959 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 3.0 MiB/s wr, 24 op/s 2021-02-10T04:35:39.853 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:33 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:35:39.853 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:33 smithi085 conmon[29319]: T04:35:32.220085+0000 mon.a (mon.0) 1336 : cluster [DBG] osdmap e515: 8 total, 8 up, 8 in 2021-02-10T04:35:39.853 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:33 smithi085 conmon[29319]: cluster 2021-02-10T04:35:39.853 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:33 smithi085 conmon[29319]: 2021-02-10T04:35:39.853 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:33 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:35:39.853 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:33 smithi085 conmon[29319]: 35:32.675921+0000 osd.0 (osd.0) 70 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:35:39.853 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:33 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:39.853 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:33 smithi085 conmon[29319]: 02-10T04:35:32.676206+0000 osd.0 (osd.0) 71 : cluster 2021-02-10T04:35:39.853 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:33 smithi085 conmon[29319]: [DBG] purged_snaps scrub ok 2021-02-10T04:35:39.853 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:34 smithi085 conmon[29319]: cluster 2021-02-10T04:35:33.296905+0000 mon.a (mon.0) 1337 : cluster [DBG] osdmap e516: 8 total, 8 up, 8 in 2021-02-10T04:35:39.853 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:34 smithi085 conmon[29319]: cluster 2021-02-10T04:35:34.103554+0000 mgr.y (mgr.14140) 599 : cluster [DBG] pgmap v924: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 159 active+clean; 362 MiB data, 971 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 4.3 MiB/s wr, 29 op/s 2021-02-10T04:35:39.853 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:34 smithi085 conmon[29319]: audit 2021-02-10T04:35:34.199351+0000 mon.a (mon.0) 1338 : audit [INF] from='client.? 172.21.15.85:0/3555700038' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-02-10T04:35:39.854 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:34 smithi085 conmon[29319]: audit 2021-02-10T04:35:34.280690+0000 mon.a (mon.0) 1339 : audit [INF] from='client.? 172.21.15.85:0/3555700038' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-02-10T04:35:39.854 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:34 smithi085 conmon[29319]: cluster 2021-02-10T04:35:34.280758+0000 mon.a (mon.0) 1340 : cluster [DBG] osdmap e517: 8 total, 8 up, 8 in 2021-02-10T04:35:39.854 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:36 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:35:39.854 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:36 smithi085 conmon[29319]: :35:35.312032+0000 mon.a (mon.0) 1341 : cluster [DBG] osdmap e518: 8 total, 8 up, 8 in 2021-02-10T04:35:39.854 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:36 smithi085 conmon[29319]: cluster 2021-02-10T04:35 2021-02-10T04:35:39.854 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:36 smithi085 conmon[29319]: :36.104059+0000 mgr.y (mgr.14140) 600 : cluster [DBG] pgmap v927: 168 pgs: 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 158 active+clean; 374 MiB data, 986 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 4.6 MiB/s wr, 25 op/s 2021-02-10T04:35:39.854 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:37 smithi085 conmon[29319]: cluster 2021-02-10T04:35:36.317684+0000 mon.a (mon 2021-02-10T04:35:39.854 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:37 smithi085 conmon[29319]: .0) 1342 : cluster [DBG] osdmap e519: 8 total, 8 up, 8 in 2021-02-10T04:35:39.854 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[29319]: cluster 2021-02-10T04:35:39.854 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[29319]: 2021-02-10T04:35:39.854 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[29319]: 2021 2021-02-10T04:35:39.854 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[29319]: - 2021-02-10T04:35:39.855 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[29319]: 02 2021-02-10T04:35:39.855 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[29319]: - 2021-02-10T04:35:39.855 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[29319]: 10T04:35:37. 2021-02-10T04:35:39.855 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[29319]: 322581+0000 mon.a (mon.0) 1343 : cluster [DBG] osdmap e520: 8 total, 8 up, 8 in 2021-02-10T04:35:39.855 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[29319]: audit 2021-02-10T04:35:39.855 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[29319]: 2021- 2021-02-10T04:35:39.855 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[29319]: 02-10T04:35:38.096983+0000 mgr.y (mgr. 2021-02-10T04:35:39.855 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[29319]: 14140) 601 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:35:39.855 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[29319]: audit 2021-02-10T04:35:39.855 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[29319]: 2021-02-10T04:35:38.097100 2021-02-10T04:35:39.855 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1344 : audit 2021-02-10T04:35:39.855 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[29319]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:35:39.856 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:35:39.856 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[29319]: -10T04:35:38.097581+0000 mon.a 2021-02-10T04:35:39.856 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[29319]: (mon.0 2021-02-10T04:35:39.856 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[29319]: ) 2021-02-10T04:35:39.856 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[29319]: 1345 2021-02-10T04:35:39.856 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[29319]: : 2021-02-10T04:35:39.856 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[29319]: audit 2021-02-10T04:35:39.856 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[29319]: 2021-02-10T04:35:39.856 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[29319]: [INF] 2021-02-10T04:35:39.856 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:35:39.856 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[29319]: cluster 2021-02-10T04:35:38 2021-02-10T04:35:39.856 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[29319]: .104434+0000 mgr.y (mgr.14140) 602 : cluster [DBG] 2021-02-10T04:35:39.857 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[29319]: pgmap v930: 168 pgs: 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 158 active+clean; 374 MiB data, 981 MiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 7 op/s 2021-02-10T04:35:39.857 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:39 smithi085 conmon[29319]: cluster 2021-02-10T04:35:38.328396+0000 mon.a (mon.0 2021-02-10T04:35:39.857 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:39 smithi085 conmon[29319]: ) 1346 : cluster [DBG] osdmap e521: 8 total, 8 up, 8 in 2021-02-10T04:35:39.857 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:32 smithi085 conmon[35961]: cluster 2021-02-10T04:35:31.211773+0000 2021-02-10T04:35:39.857 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:32 smithi085 conmon[35961]: mon.a (mon.0) 1335 : cluster [DBG] osdmap e514: 8 total, 8 up, 8 in 2021-02-10T04:35:39.857 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:32 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:35:39.857 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:32 smithi085 conmon[35961]: 04:35:32.102803+0000 mgr.y (mgr.14140) 598 : cluster [DBG] pgmap v921: 168 pgs: 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 156 active+clean; 354 MiB data, 959 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 3.0 MiB/s wr, 24 op/s 2021-02-10T04:35:39.857 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:32 smithi085 conmon[35961]: 2021-02-10T04:35:39.857 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:33 smithi085 conmon[35961]: cluster 2021-02-10T04:35: 2021-02-10T04:35:39.857 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:33 smithi085 conmon[35961]: 32.220085+0000 mon.a (mon.0) 1336 : cluster [DBG] osdmap e515: 8 total, 8 up, 8 in 2021-02-10T04:35:39.857 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:33 smithi085 conmon[35961]: cluster 2021-02-10T04:35:32.675921+0000 osd.0 (osd.0) 70 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:35:39.858 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:33 smithi085 conmon[35961]: cluster 2021-02-10T04:35:32.676206+0000 osd.0 (osd.0) 71 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:35:39.858 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:34 smithi085 conmon[35961]: cluster 2021-02-10T04:35:39.858 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:34 smithi085 conmon[35961]: 2021-02-10T04:35:39.858 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:34 smithi085 conmon[35961]: 2021 2021-02-10T04:35:39.858 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:34 smithi085 conmon[35961]: - 2021-02-10T04:35:39.858 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:34 smithi085 conmon[35961]: 02 2021-02-10T04:35:39.858 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:34 smithi085 conmon[35961]: -10T04:35:33.296905+0000 2021-02-10T04:35:39.858 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:34 smithi085 conmon[35961]: mon.a (mon.0) 1337 : cluster [DBG] osdmap e516: 8 total, 8 up, 8 in 2021-02-10T04:35:39.858 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:34 smithi085 conmon[35961]: cluster 2021-02-10T04:35:34 2021-02-10T04:35:39.858 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:34 smithi085 conmon[35961]: .103554+0000 mgr.y (mgr.14140) 599 : cluster [DBG] 2021-02-10T04:35:39.858 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:34 smithi085 conmon[35961]: pgmap v924: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 159 active+clean; 362 MiB data, 971 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 4.3 MiB/s wr, 29 op/s 2021-02-10T04:35:39.859 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:34 smithi085 conmon[35961]: audit 2021-02-10T04:35:34.199351 2021-02-10T04:35:39.859 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:34 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1338 : audit [INF] 2021-02-10T04:35:39.859 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:34 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/3555700038' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-02-10T04:35:39.859 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:34 smithi085 conmon[35961]: audit 2021-02-10T04:35:39.859 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:34 smithi085 conmon[35961]: 2021-02-10T04:35:34.280690+0000 mon.a (mon 2021-02-10T04:35:39.859 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:34 smithi085 conmon[35961]: .0) 1339 : audit [INF] from='client.? 172.21.15.85:0/3555700038' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-02-10T04:35:39.859 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:34 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:35:39.859 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:34 smithi085 conmon[35961]: 04:35:34.280758+0000 mon.a (mon 2021-02-10T04:35:39.859 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:34 smithi085 conmon[35961]: .0) 1340 : cluster [DBG] osdmap e517: 8 total, 8 up, 8 in 2021-02-10T04:35:39.859 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:36 smithi085 conmon[35961]: cluster 2021-02-10T04:35:35 2021-02-10T04:35:39.859 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:36 smithi085 conmon[35961]: .312032+0000 mon.a (mon.0) 1341 : cluster [DBG] osdmap e518: 8 total, 8 up, 8 in 2021-02-10T04:35:39.859 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:36 smithi085 conmon[35961]: cluster 2021-02-10T04:35:36.104059+0000 mgr.y (mgr.14140) 600 : 2021-02-10T04:35:39.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:36 smithi085 conmon[35961]: cluster [DBG] pgmap v927: 168 pgs: 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 158 active+clean; 374 MiB data, 986 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 4.6 MiB/s wr, 25 op/s 2021-02-10T04:35:39.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:37 smithi085 conmon[35961]: cluster 2021-02-10T04:35:36.317684 2021-02-10T04:35:39.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:37 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1342 : cluster [DBG] osdmap e519: 8 total, 8 up, 8 in 2021-02-10T04:35:39.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[35961]: cluster 2021-02-10T04:35:37.322581+0000 2021-02-10T04:35:39.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[35961]: mon.a (mon.0) 1343 : cluster [DBG] osdmap e520: 8 total, 8 up, 8 in 2021-02-10T04:35:39.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[35961]: audit 2021-02-10T04:35:38.096983+0000 mgr.y (mgr.14140) 601 : audit 2021-02-10T04:35:39.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[35961]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:35:39.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:35:39.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[35961]: :35:38.097100+0000 mon.a (mon.0) 1344 2021-02-10T04:35:39.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[35961]: : 2021-02-10T04:35:39.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[35961]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:35:39.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[35961]: audit 2021-02-10T04:35:39.861 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:35:39.861 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[35961]: T04:35:38.097581+0000 mon.a (mon. 2021-02-10T04:35:39.861 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[35961]: 0) 1345 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:35:39.861 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:35:39.861 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[35961]: 04:35:38.104434+0000 mgr.y (mgr.14140 2021-02-10T04:35:39.861 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:38 smithi085 conmon[35961]: ) 602 : cluster [DBG] pgmap v930: 168 pgs: 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 158 active+clean; 374 MiB data, 981 MiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 7 op/s 2021-02-10T04:35:39.861 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:39 smithi085 conmon[35961]: cluster 2021-02-10T04:35:38.328396+0000 mon.a (mon.0 2021-02-10T04:35:39.861 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:39 smithi085 conmon[35961]: ) 1346 : cluster [DBG] osdmap e521: 8 total, 8 up, 8 in 2021-02-10T04:35:39.862 INFO:tasks.rados.rados.0.smithi085.stdout:1983: done (11 left) 2021-02-10T04:35:39.862 INFO:tasks.rados.rados.0.smithi085.stdout:1984: done (10 left) 2021-02-10T04:35:39.862 INFO:tasks.rados.rados.0.smithi085.stdout:1985: done (9 left) 2021-02-10T04:35:39.862 INFO:tasks.rados.rados.0.smithi085.stdout:1986: snap_remove snap 180 2021-02-10T04:35:39.862 INFO:tasks.rados.rados.0.smithi085.stdout:1986: done (9 left) 2021-02-10T04:35:39.862 INFO:tasks.rados.rados.0.smithi085.stdout:1987: rollback oid 14 current snap is 220 2021-02-10T04:35:39.862 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 14 to 202 2021-02-10T04:35:39.862 INFO:tasks.rados.rados.0.smithi085.stdout:1988: write (excl) oid 16 current snap is 220 2021-02-10T04:35:39.862 INFO:tasks.rados.rados.0.smithi085.stdout:1988: seq_num 703 ranges {615321=404523,1804606=457240,2755658=726225} 2021-02-10T04:35:39.862 INFO:tasks.rados.rados.0.smithi085.stdout:1988: writing smithi08575397-16 from 615321 to 1019844 tid 1 2021-02-10T04:35:39.862 INFO:tasks.rados.rados.0.smithi085.stdout:1988: writing smithi08575397-16 from 1804606 to 2261846 tid 2 2021-02-10T04:35:39.863 INFO:tasks.rados.rados.0.smithi085.stdout:1988: writing smithi08575397-16 from 2755658 to 3481883 tid 3 2021-02-10T04:35:39.863 INFO:tasks.rados.rados.0.smithi085.stdout:1989: delete oid 48 current snap is 220 2021-02-10T04:35:39.863 INFO:tasks.rados.rados.0.smithi085.stdout:1987: finishing rollback tid 0 to smithi08575397-14 2021-02-10T04:35:39.863 INFO:tasks.rados.rados.0.smithi085.stdout:1987: finishing rollback tid 1 to smithi08575397-14 2021-02-10T04:35:39.863 INFO:tasks.rados.rados.0.smithi085.stdout:1988: finishing write tid 1 to smithi08575397-16 2021-02-10T04:35:39.863 INFO:tasks.rados.rados.0.smithi085.stdout:1987: finishing rollback tid 2 to smithi08575397-14 2021-02-10T04:35:39.863 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 809 (ObjNum 651 snap 197 seq_num 651) dirty exists 2021-02-10T04:35:39.863 INFO:tasks.rados.rados.0.smithi085.stdout:1988: finishing write tid 2 to smithi08575397-16 2021-02-10T04:35:39.863 INFO:tasks.rados.rados.0.smithi085.stdout:1987: done (11 left) 2021-02-10T04:35:39.863 INFO:tasks.rados.rados.0.smithi085.stdout:1989: done (10 left) 2021-02-10T04:35:39.863 INFO:tasks.rados.rados.0.smithi085.stdout:1990: rollback oid 38 current snap is 220 2021-02-10T04:35:39.863 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 38 to 202 2021-02-10T04:35:39.863 INFO:tasks.rados.rados.0.smithi085.stdout:1991: snap_create 2021-02-10T04:35:39.864 INFO:tasks.rados.rados.0.smithi085.stdout:1988: finishing write tid 3 to smithi08575397-16 2021-02-10T04:35:39.864 INFO:tasks.rados.rados.0.smithi085.stdout:1988: finishing write tid 4 to smithi08575397-16 2021-02-10T04:35:39.864 INFO:tasks.rados.rados.0.smithi085.stdout:1988: finishing write tid 5 to smithi08575397-16 2021-02-10T04:35:39.864 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 1066 (ObjNum 703 snap 220 seq_num 703) dirty exists 2021-02-10T04:35:39.864 INFO:tasks.rados.rados.0.smithi085.stdout:1988: left oid 16 (ObjNum 703 snap 220 seq_num 703) 2021-02-10T04:35:39.864 INFO:tasks.rados.rados.0.smithi085.stdout:1990: finishing rollback tid 0 to smithi08575397-38 2021-02-10T04:35:39.864 INFO:tasks.rados.rados.0.smithi085.stdout:1990: finishing rollback tid 1 to smithi08575397-38 2021-02-10T04:35:39.864 INFO:tasks.rados.rados.0.smithi085.stdout:1990: finishing rollback tid 2 to smithi08575397-38 2021-02-10T04:35:39.864 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 1025 (ObjNum 616 snap 186 seq_num 616) dirty exists 2021-02-10T04:35:39.864 INFO:tasks.rados.rados.0.smithi085.stdout:1988: done (11 left) 2021-02-10T04:35:39.864 INFO:tasks.rados.rados.0.smithi085.stdout:1990: done (10 left) 2021-02-10T04:35:39.864 INFO:tasks.rados.rados.0.smithi085.stdout:1991: done (9 left) 2021-02-10T04:35:39.865 INFO:tasks.rados.rados.0.smithi085.stdout:1992: copy_from oid 28 from oid 14 current snap is 221 2021-02-10T04:35:39.865 INFO:tasks.rados.rados.0.smithi085.stdout:1993: write oid 20 current snap is 221 2021-02-10T04:35:39.865 INFO:tasks.rados.rados.0.smithi085.stdout:1993: seq_num 705 ranges {648935=555021,1805217=460909,2694151=465122,3817442=31493} 2021-02-10T04:35:39.865 INFO:tasks.rados.rados.0.smithi085.stdout:1993: writing smithi08575397-20 from 648935 to 1203956 tid 1 2021-02-10T04:35:39.865 INFO:tasks.rados.rados.0.smithi085.stdout:1993: writing smithi08575397-20 from 1805217 to 2266126 tid 2 2021-02-10T04:35:39.865 INFO:tasks.rados.rados.0.smithi085.stdout:1993: writing smithi08575397-20 from 2694151 to 3159273 tid 3 2021-02-10T04:35:39.865 INFO:tasks.rados.rados.0.smithi085.stdout:1993: writing smithi08575397-20 from 3817442 to 3848935 tid 4 2021-02-10T04:35:39.865 INFO:tasks.rados.rados.0.smithi085.stdout:1993: finishing write tid 1 to smithi08575397-20 2021-02-10T04:35:39.865 INFO:tasks.rados.rados.0.smithi085.stdout:1993: finishing write tid 2 to smithi08575397-20 2021-02-10T04:35:39.865 INFO:tasks.rados.rados.0.smithi085.stdout:1993: finishing write tid 3 to smithi08575397-20 2021-02-10T04:35:39.865 INFO:tasks.rados.rados.0.smithi085.stdout:1994: read oid 13 snap -1 2021-02-10T04:35:39.865 INFO:tasks.rados.rados.0.smithi085.stdout:1994: expect deleted 2021-02-10T04:35:39.866 INFO:tasks.rados.rados.0.smithi085.stdout:1995: read oid 2 snap -1 2021-02-10T04:35:39.866 INFO:tasks.rados.rados.0.smithi085.stdout:1995: expect (ObjNum 652 snap 197 seq_num 652) 2021-02-10T04:35:39.866 INFO:tasks.rados.rados.0.smithi085.stdout:1996: read oid 3 snap 218 2021-02-10T04:35:39.866 INFO:tasks.rados.rados.0.smithi085.stdout:1996: expect (ObjNum 674 snap 208 seq_num 674) 2021-02-10T04:35:39.866 INFO:tasks.rados.rados.0.smithi085.stdout:1997: read oid 11 snap -1 2021-02-10T04:35:39.866 INFO:tasks.rados.rados.0.smithi085.stdout:1997: expect (ObjNum 681 snap 210 seq_num 681) 2021-02-10T04:35:39.866 INFO:tasks.rados.rados.0.smithi085.stdout:1998: snap_create 2021-02-10T04:35:39.866 INFO:tasks.rados.rados.0.smithi085.stdout:1993: finishing write tid 4 to smithi08575397-20 2021-02-10T04:35:39.866 INFO:tasks.rados.rados.0.smithi085.stdout:1993: finishing write tid 5 to smithi08575397-20 2021-02-10T04:35:39.866 INFO:tasks.rados.rados.0.smithi085.stdout:1993: finishing write tid 6 to smithi08575397-20 2021-02-10T04:35:39.866 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 20 v 749 (ObjNum 705 snap 221 seq_num 705) dirty exists 2021-02-10T04:35:39.866 INFO:tasks.rados.rados.0.smithi085.stdout:1993: left oid 20 (ObjNum 705 snap 221 seq_num 705) 2021-02-10T04:35:39.867 INFO:tasks.rados.rados.0.smithi085.stdout:1995: expect (ObjNum 652 snap 197 seq_num 652) 2021-02-10T04:35:39.867 INFO:tasks.rados.rados.0.smithi085.stdout:1996: expect (ObjNum 674 snap 208 seq_num 674) 2021-02-10T04:35:39.867 INFO:tasks.rados.rados.0.smithi085.stdout:1997: expect (ObjNum 681 snap 210 seq_num 681) 2021-02-10T04:35:39.867 INFO:tasks.rados.rados.0.smithi085.stdout:1992: finishing copy_from to smithi08575397-28 2021-02-10T04:35:39.867 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 809 (ObjNum 651 snap 197 seq_num 651) dirty exists 2021-02-10T04:35:39.867 INFO:tasks.rados.rados.0.smithi085.stdout:1992: finishing copy_from racing read to smithi08575397-28 2021-02-10T04:35:39.867 INFO:tasks.rados.rados.0.smithi085.stdout:1992: done (15 left) 2021-02-10T04:35:39.867 INFO:tasks.rados.rados.0.smithi085.stdout:1993: done (14 left) 2021-02-10T04:35:39.867 INFO:tasks.rados.rados.0.smithi085.stdout:1994: done (13 left) 2021-02-10T04:35:39.867 INFO:tasks.rados.rados.0.smithi085.stdout:1995: done (12 left) 2021-02-10T04:35:39.867 INFO:tasks.rados.rados.0.smithi085.stdout:1996: done (11 left) 2021-02-10T04:35:39.867 INFO:tasks.rados.rados.0.smithi085.stdout:1997: done (10 left) 2021-02-10T04:35:39.868 INFO:tasks.rados.rados.0.smithi085.stdout:1998: done (9 left) 2021-02-10T04:35:39.868 INFO:tasks.rados.rados.0.smithi085.stdout:1999: delete oid 11 current snap is 222 2021-02-10T04:35:39.868 INFO:tasks.rados.rados.0.smithi085.stdout:1999: done (9 left) 2021-02-10T04:35:39.868 INFO:tasks.rados.rados.0.smithi085.stdout:2000: rollback oid 21 current snap is 222 2021-02-10T04:35:39.868 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 21 to 189 2021-02-10T04:35:39.868 INFO:tasks.rados.rados.0.smithi085.stdout:2001: delete oid 23 current snap is 222 2021-02-10T04:35:39.868 INFO:tasks.rados.rados.0.smithi085.stdout:2000: finishing rollback tid 0 to smithi08575397-21 2021-02-10T04:35:39.868 INFO:tasks.rados.rados.0.smithi085.stdout:2000: finishing rollback tid 1 to smithi08575397-21 2021-02-10T04:35:39.868 INFO:tasks.rados.rados.0.smithi085.stdout:2000: finishing rollback tid 2 to smithi08575397-21 2021-02-10T04:35:39.868 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 1076 (ObjNum 431 snap 114 seq_num 431) dirty exists 2021-02-10T04:35:39.868 INFO:tasks.rados.rados.0.smithi085.stdout:2000: done (10 left) 2021-02-10T04:35:39.869 INFO:tasks.rados.rados.0.smithi085.stdout:2001: done (9 left) 2021-02-10T04:35:39.869 INFO:tasks.rados.rados.0.smithi085.stdout:2002: read oid 44 snap -1 2021-02-10T04:35:39.869 INFO:tasks.rados.rados.0.smithi085.stdout:2002: expect (ObjNum 671 snap 207 seq_num 671) 2021-02-10T04:35:39.869 INFO:tasks.rados.rados.0.smithi085.stdout:2003: write oid 7 current snap is 222 2021-02-10T04:35:39.869 INFO:tasks.rados.rados.0.smithi085.stdout:2003: seq_num 708 ranges {699356=497121,1980663=608236,3176420=415291} 2021-02-10T04:35:39.870 INFO:tasks.rados.rados.0.smithi085.stdout:2003: writing smithi08575397-7 from 699356 to 1196477 tid 1 2021-02-10T04:35:39.870 INFO:tasks.rados.rados.0.smithi085.stdout:2003: writing smithi08575397-7 from 1980663 to 2588899 tid 2 2021-02-10T04:35:39.870 INFO:tasks.rados.rados.0.smithi085.stdout:2003: writing smithi08575397-7 from 3176420 to 3591711 tid 3 2021-02-10T04:35:39.870 INFO:tasks.rados.rados.0.smithi085.stdout:2004: snap_remove snap 198 2021-02-10T04:35:39.870 INFO:tasks.rados.rados.0.smithi085.stdout:2002: expect (ObjNum 671 snap 207 seq_num 671) 2021-02-10T04:35:39.870 INFO:tasks.rados.rados.0.smithi085.stdout:2003: finishing write tid 1 to smithi08575397-7 2021-02-10T04:35:39.870 INFO:tasks.rados.rados.0.smithi085.stdout:2003: finishing write tid 2 to smithi08575397-7 2021-02-10T04:35:39.870 INFO:tasks.rados.rados.0.smithi085.stdout:2002: done (11 left) 2021-02-10T04:35:39.870 INFO:tasks.rados.rados.0.smithi085.stdout:2004: done (10 left) 2021-02-10T04:35:39.870 INFO:tasks.rados.rados.0.smithi085.stdout:2005: snap_create 2021-02-10T04:35:39.870 INFO:tasks.rados.rados.0.smithi085.stdout:2003: finishing write tid 3 to smithi08575397-7 2021-02-10T04:35:39.870 INFO:tasks.rados.rados.0.smithi085.stdout:2003: finishing write tid 4 to smithi08575397-7 2021-02-10T04:35:39.871 INFO:tasks.rados.rados.0.smithi085.stdout:2003: finishing write tid 5 to smithi08575397-7 2021-02-10T04:35:39.871 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 740 (ObjNum 708 snap 222 seq_num 708) dirty exists 2021-02-10T04:35:39.871 INFO:tasks.rados.rados.0.smithi085.stdout:2003: left oid 7 (ObjNum 708 snap 222 seq_num 708) 2021-02-10T04:35:39.871 INFO:tasks.rados.rados.0.smithi085.stdout:2003: done (10 left) 2021-02-10T04:35:39.871 INFO:tasks.rados.rados.0.smithi085.stdout:2005: done (9 left) 2021-02-10T04:35:39.871 INFO:tasks.rados.rados.0.smithi085.stdout:2006: read oid 48 snap 218 2021-02-10T04:35:39.871 INFO:tasks.rados.rados.0.smithi085.stdout:2006: expect (ObjNum 422 snap 109 seq_num 422) 2021-02-10T04:35:39.871 INFO:tasks.rados.rados.0.smithi085.stdout:2007: read oid 38 snap 184 2021-02-10T04:35:39.871 INFO:tasks.rados.rados.0.smithi085.stdout:2007: expect (ObjNum 530 snap 156 seq_num 530) 2021-02-10T04:35:39.872 INFO:tasks.rados.rados.0.smithi085.stdout:2008: rollback oid 20 current snap is 223 2021-02-10T04:35:39.872 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 20 to 189 2021-02-10T04:35:39.872 INFO:tasks.rados.rados.0.smithi085.stdout:2009: delete oid 42 current snap is 223 2021-02-10T04:35:39.872 INFO:tasks.rados.rados.0.smithi085.stdout:2008: finishing rollback tid 0 to smithi08575397-20 2021-02-10T04:35:39.872 INFO:tasks.rados.rados.0.smithi085.stdout:2008: finishing rollback tid 1 to smithi08575397-20 2021-02-10T04:35:39.872 INFO:tasks.rados.rados.0.smithi085.stdout:2008: finishing rollback tid 2 to smithi08575397-20 2021-02-10T04:35:39.872 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 20 v 752 (ObjNum 596 snap 178 seq_num 596) dirty exists 2021-02-10T04:35:39.872 INFO:tasks.rados.rados.0.smithi085.stdout:2006: expect (ObjNum 422 snap 109 seq_num 422) 2021-02-10T04:35:39.872 INFO:tasks.rados.rados.0.smithi085.stdout:2007: expect (ObjNum 530 snap 156 seq_num 530) 2021-02-10T04:35:39.872 INFO:tasks.rados.rados.0.smithi085.stdout:2006: done (12 left) 2021-02-10T04:35:39.872 INFO:tasks.rados.rados.0.smithi085.stdout:2007: done (11 left) 2021-02-10T04:35:39.873 INFO:tasks.rados.rados.0.smithi085.stdout:2008: done (10 left) 2021-02-10T04:35:39.873 INFO:tasks.rados.rados.0.smithi085.stdout:2009: done (9 left) 2021-02-10T04:35:39.873 INFO:tasks.rados.rados.0.smithi085.stdout:2010: rollback oid 35 current snap is 223 2021-02-10T04:35:39.873 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 35 to 185 2021-02-10T04:35:39.873 INFO:tasks.rados.rados.0.smithi085.stdout:2011: read oid 36 snap -1 2021-02-10T04:35:39.873 INFO:tasks.rados.rados.0.smithi085.stdout:2011: expect (ObjNum 694 snap 215 seq_num 694) 2021-02-10T04:35:39.873 INFO:tasks.rados.rados.0.smithi085.stdout:2012: copy_from oid 16 from oid 44 current snap is 223 2021-02-10T04:35:39.873 INFO:tasks.rados.rados.0.smithi085.stdout:2013: snap_remove snap 216 2021-02-10T04:35:39.873 INFO:tasks.rados.rados.0.smithi085.stdout:2013: done (12 left) 2021-02-10T04:35:39.873 INFO:tasks.rados.rados.0.smithi085.stdout:2014: copy_from oid 20 from oid 48 current snap is 223 2021-02-10T04:35:39.873 INFO:tasks.rados.rados.0.smithi085.stdout:2010: finishing rollback tid 0 to smithi08575397-35 2021-02-10T04:35:39.873 INFO:tasks.rados.rados.0.smithi085.stdout:2010: finishing rollback tid 1 to smithi08575397-35 2021-02-10T04:35:39.874 INFO:tasks.rados.rados.0.smithi085.stdout:2010: finishing rollback tid 2 to smithi08575397-35 2021-02-10T04:35:39.874 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 35 v 667 (ObjNum 578 snap 176 seq_num 578) dirty exists 2021-02-10T04:35:39.874 INFO:tasks.rados.rados.0.smithi085.stdout:2011: expect (ObjNum 694 snap 215 seq_num 694) 2021-02-10T04:35:39.874 INFO:tasks.rados.rados.0.smithi085.stdout:2012: finishing copy_from to smithi08575397-16 2021-02-10T04:35:39.874 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 737 (ObjNum 671 snap 207 seq_num 671) dirty exists 2021-02-10T04:35:39.874 INFO:tasks.rados.rados.0.smithi085.stdout:2012: finishing copy_from racing read to smithi08575397-16 2021-02-10T04:35:39.874 INFO:tasks.rados.rados.0.smithi085.stdout:2014: finishing copy_from to smithi08575397-20 2021-02-10T04:35:39.874 INFO:tasks.rados.rados.0.smithi085.stdout:2014: got expected ENOENT (src dne) 2021-02-10T04:35:39.874 INFO:tasks.rados.rados.0.smithi085.stdout:2010: done (12 left) 2021-02-10T04:35:39.874 INFO:tasks.rados.rados.0.smithi085.stdout:2011: done (11 left) 2021-02-10T04:35:39.874 INFO:tasks.rados.rados.0.smithi085.stdout:2012: done (10 left) 2021-02-10T04:35:39.874 INFO:tasks.rados.rados.0.smithi085.stdout:2015: copy_from oid 37 from oid 42 current snap is 223 2021-02-10T04:35:39.875 INFO:tasks.rados.rados.0.smithi085.stdout:2016: snap_remove snap 209 2021-02-10T04:35:39.875 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.4 to 1 (10000) 2021-02-10T04:35:39.875 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:33 smithi164 conmon[29288]: cluster 2021-02-10T04:35:32. 2021-02-10T04:35:39.875 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:33 smithi164 conmon[29288]: 220085+0000 mon.a (mon.0) 1336 : cluster [DBG] osdmap e515: 8 total, 8 up, 8 in 2021-02-10T04:35:39.875 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:33 smithi164 conmon[29288]: cluster 2021-02-10T04:35:32.675921+0000 osd.0 (osd.0) 70 : cluster 2021-02-10T04:35:39.875 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:33 smithi164 conmon[29288]: [DBG] purged_snaps scrub starts 2021-02-10T04:35:39.875 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:33 smithi164 conmon[29288]: cluster 2021-02-10T04:35:32. 2021-02-10T04:35:39.875 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:33 smithi164 conmon[29288]: 676206+0000 osd.0 (osd.0) 71 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:35:39.875 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:34 smithi164 conmon[29288]: cluster 2021-02-10T04:35:33.296905+0000 mon.a ( 2021-02-10T04:35:39.875 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:34 smithi164 conmon[29288]: mon.0) 1337 : cluster [DBG] osdmap e516: 8 total, 8 up, 8 in 2021-02-10T04:35:39.876 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:34 smithi164 conmon[29288]: cluster 2021-02-10T04:35:34.103554+0000 mgr.y (mgr.14140) 599 : cluster [DBG] pgmap v924: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 159 active+clean; 362 MiB data, 971 MiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 4.3 MiB/s wr, 29 op/s 2021-02-10T04:35:39.876 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:34 smithi164 conmon[29288]: audit 2021-02-10T04:35:34.199351+0000 mon.a (mon 2021-02-10T04:35:39.876 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:34 smithi164 conmon[29288]: .0) 1338 : audit [INF] from='client.? 172.21.15.85:0/3555700038' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-02-10T04:35:39.876 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:34 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:35:39.876 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:34 smithi164 conmon[29288]: 02-10T04:35:34.280690+0000 mon.a (mon 2021-02-10T04:35:39.876 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:34 smithi164 conmon[29288]: .0) 1339 : audit [INF] from='client.? 172.21.15.85:0/3555700038' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-02-10T04:35:39.876 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:34 smithi164 conmon[29288]: cluster 2021-02-10T04:35:34. 2021-02-10T04:35:39.876 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:34 smithi164 conmon[29288]: 280758+0000 mon.a (mon.0) 1340 : cluster [DBG] osdmap e517: 8 total, 8 up, 8 in 2021-02-10T04:35:39.876 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:36 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:35:39.876 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:36 smithi164 conmon[29288]: 35:35.312032+0000 mon.a (mon.0) 1341 : cluster [DBG] osdmap e518: 8 total, 8 up, 8 in 2021-02-10T04:35:39.876 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:36 smithi164 conmon[29288]: cluster 2021-02-10T04:35: 2021-02-10T04:35:39.876 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:36 smithi164 conmon[29288]: 36.104059+0000 mgr.y (mgr.14140) 600 : cluster [DBG] pgmap v927: 168 pgs: 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 158 active+clean; 374 MiB data, 986 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 4.6 MiB/s wr, 25 op/s 2021-02-10T04:35:39.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:36 smithi164 conmon[29288]: 2021-02-10T04:35:39.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:37 smithi164 conmon[29288]: cluster 2021-02-10T04:35:36.317684+0000 mon.a (mon.0) 1342 2021-02-10T04:35:39.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:37 smithi164 conmon[29288]: : cluster [DBG] osdmap e519: 8 total, 8 up, 8 in 2021-02-10T04:35:39.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:38 smithi164 conmon[29288]: cluster 2021-02-10T04:35:37.322581+0000 2021-02-10T04:35:39.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:38 smithi164 conmon[29288]: mon.a (mon.0) 1343 : cluster [DBG] osdmap e520: 8 total, 8 up, 8 in 2021-02-10T04:35:39.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:38 smithi164 conmon[29288]: audit 2021-02-10T04:35:38.096983+0000 mgr.y (mgr.14140) 601 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:35:39.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:38 smithi164 conmon[29288]: audit 2021-02-10T04:35 2021-02-10T04:35:39.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:38 smithi164 conmon[29288]: :38.097100+0000 mon.a (mon.0) 1344 : audit 2021-02-10T04:35:39.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:38 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:35:39.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:38 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:35:39.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:38 smithi164 conmon[29288]: 10T04:35:38.097581+0000 mon.a ( 2021-02-10T04:35:39.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:38 smithi164 conmon[29288]: mon.0) 1345 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:35:39.878 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:38 smithi164 conmon[29288]: cluster 2021-02-10T04:35:39.878 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:38 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:35:39.878 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:38 smithi164 conmon[29288]: 35:38.104434+0000 mgr.y (mgr.14140) 602 : cluster [DBG] pgmap v930: 168 pgs: 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 6 active+clean+snaptrim, 158 active+clean; 374 MiB data, 981 MiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 2.1 MiB/s wr, 7 op/s 2021-02-10T04:35:39.878 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:39 smithi164 conmon[29288]: cluster 2021-02-10T04:35:38.328396+0000 mon.a (mon.0) 1346 : cluster [DBG] osdmap e521: 8 total, 8 up, 8 in 2021-02-10T04:35:39.878 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:35:38 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:35:39.878 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.4 dump_ops_in_flight 2021-02-10T04:35:39.963 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:35:39.992 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:35:40.110 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:35:40.216 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:35:40.240 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:40 smithi164 conmon[33677]: debug 2021-02-10T04:35:40.072+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:35:40.243 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:35:40.343 INFO:tasks.rados.rados.0.smithi085.stdout:2016: done (11 left) 2021-02-10T04:35:40.343 INFO:tasks.rados.rados.0.smithi085.stdout:2017: delete oid 10 current snap is 223 2021-02-10T04:35:40.343 INFO:tasks.rados.rados.0.smithi085.stdout:2014: finishing copy_from racing read to smithi08575397-20 2021-02-10T04:35:40.343 INFO:tasks.rados.rados.0.smithi085.stdout:2015: finishing copy_from to smithi08575397-37 2021-02-10T04:35:40.343 INFO:tasks.rados.rados.0.smithi085.stdout:2015: got expected ENOENT (src dne) 2021-02-10T04:35:40.343 INFO:tasks.rados.rados.0.smithi085.stdout:2015: finishing copy_from racing read to smithi08575397-37 2021-02-10T04:35:40.351 INFO:tasks.rados.rados.0.smithi085.stdout:2014: done (11 left) 2021-02-10T04:35:40.351 INFO:tasks.rados.rados.0.smithi085.stdout:2015: done (10 left) 2021-02-10T04:35:40.351 INFO:tasks.rados.rados.0.smithi085.stdout:2017: done (9 left) 2021-02-10T04:35:40.351 INFO:tasks.rados.rados.0.smithi085.stdout:2018: snap_remove snap 189 2021-02-10T04:35:40.369 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:35:40.376 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:35:40.482 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:35:40.569 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:35:40.641 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[29319]: cluster 2021-02-10T04:35:39.331779+0000 mon.a (mon.0 2021-02-10T04:35:40.641 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[29319]: ) 1347 : cluster [DBG] osdmap e522: 8 total, 8 up, 8 in 2021-02-10T04:35:40.642 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[29319]: cluster 2021-02-10T04:35:39.586905+0000 osd.7 (osd.7) 52 2021-02-10T04:35:40.642 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[29319]: : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:35:40.642 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[29319]: cluster 2021-02-10T04:35:40.642 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[29319]: 2021-02-10T04:35:39.587160+0000 osd.7 (osd.7) 53 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:35:40.642 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:35:40.642 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[29319]: T04:35:40.104854+0000 mgr.y (mgr.14140) 603 : cluster [DBG] pgmap v933: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 372 MiB data, 980 MiB used, 714 GiB / 715 GiB avail; 2.5 MiB/s rd, 388 KiB/s wr, 5 op/s 2021-02-10T04:35:40.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: cluster 2021-02-10T04:35:39.331779+0000 2021-02-10T04:35:40.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: mon.a (mon.0) 1347 : cluster [DBG] osdmap e522: 8 total, 8 up, 8 in 2021-02-10T04:35:40.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: cluster 2021-02-10T04:35:39.586905+0000 osd.7 (osd.7 2021-02-10T04:35:40.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: ) 52 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:35:40.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:35:40.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: - 2021-02-10T04:35:40.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: 10 2021-02-10T04:35:40.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: T 2021-02-10T04:35:40.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: 04 2021-02-10T04:35:40.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: : 2021-02-10T04:35:40.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: 35 2021-02-10T04:35:40.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: : 2021-02-10T04:35:40.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: 39 2021-02-10T04:35:40.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: . 2021-02-10T04:35:40.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: 587160 2021-02-10T04:35:40.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: +0000 2021-02-10T04:35:40.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: 2021-02-10T04:35:40.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: osd.7 2021-02-10T04:35:40.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: ( 2021-02-10T04:35:40.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: osd 2021-02-10T04:35:40.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: . 2021-02-10T04:35:40.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: 7 2021-02-10T04:35:40.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: ) 2021-02-10T04:35:40.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: 53 2021-02-10T04:35:40.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: : 2021-02-10T04:35:40.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: cluster 2021-02-10T04:35:40.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: 2021-02-10T04:35:40.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: [DBG] 2021-02-10T04:35:40.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: 2021-02-10T04:35:40.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:35:40.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: 2021-02-10T04:35:40.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: cluster 2021-02-10T04:35:40.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: 2021-02-10T04:35:40.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: 2021 2021-02-10T04:35:40.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: - 2021-02-10T04:35:40.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: 02 2021-02-10T04:35:40.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: - 2021-02-10T04:35:40.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: 10 2021-02-10T04:35:40.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: T 2021-02-10T04:35:40.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: 04 2021-02-10T04:35:40.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: : 2021-02-10T04:35:40.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: 35:40.104854+0000 mgr.y 2021-02-10T04:35:40.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[35961]: (mgr.14140) 603 : cluster [DBG] pgmap v933: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 372 MiB data, 980 MiB used, 714 GiB / 715 GiB avail; 2.5 MiB/s rd, 388 KiB/s wr, 5 op/s 2021-02-10T04:35:40.648 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:35:40 smithi085 conmon[41540]: debug 2021-02-10T04:35:40.639+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:35:40.648 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:35:40 smithi164 conmon[37263]: debug 2021-02-10T04:35:40.321+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:35:40.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:40 smithi164 conmon[29288]: cluster 2021-02-10T04:35:40.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:40 smithi164 conmon[29288]: 2021-02-10T04:35:40.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:40 smithi164 conmon[29288]: 2021-02-10T04:35:39. 2021-02-10T04:35:40.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:40 smithi164 conmon[29288]: 331779+0000 mon.a (mon.0) 1347 : cluster [DBG] osdmap e522: 8 total, 8 up, 8 in 2021-02-10T04:35:40.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:40 smithi164 conmon[29288]: cluster 2021-02-10T04:35:39.586905+0000 osd.7 (osd 2021-02-10T04:35:40.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:40 smithi164 conmon[29288]: .7) 52 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:35:40.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:40 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:35:40.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:40 smithi164 conmon[29288]: 02-10T04:35:39.587160+0000 osd.7 (osd.7 2021-02-10T04:35:40.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:40 smithi164 conmon[29288]: ) 53 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:35:40.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:40 smithi164 conmon[29288]: cluster 2021-02-10T04:35: 2021-02-10T04:35:40.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:40 smithi164 conmon[29288]: 40.104854+0000 mgr.y (mgr.14140) 603 : cluster [DBG] pgmap v933: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 372 MiB data, 980 MiB used, 714 GiB / 715 GiB avail; 2.5 MiB/s rd, 388 KiB/s wr, 5 op/s 2021-02-10T04:35:40.953 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:35:41.352 INFO:tasks.rados.rados.0.smithi085.stdout:2018: done (9 left) 2021-02-10T04:35:41.352 INFO:tasks.rados.rados.0.smithi085.stdout:2019: delete oid 22 current snap is 223 2021-02-10T04:35:41.355 INFO:tasks.rados.rados.0.smithi085.stdout:2019: done (9 left) 2021-02-10T04:35:41.355 INFO:tasks.rados.rados.0.smithi085.stdout:2020: write oid 50 current snap is 223 2021-02-10T04:35:41.355 INFO:tasks.rados.rados.0.smithi085.stdout:2020: seq_num 712 ranges {766584=798117,2098238=615574,3169759=460379} 2021-02-10T04:35:41.364 INFO:tasks.rados.rados.0.smithi085.stdout:2020: writing smithi08575397-50 from 766584 to 1564701 tid 1 2021-02-10T04:35:41.370 INFO:tasks.rados.rados.0.smithi085.stdout:2020: writing smithi08575397-50 from 2098238 to 2713812 tid 2 2021-02-10T04:35:41.377 INFO:tasks.rados.rados.0.smithi085.stdout:2020: writing smithi08575397-50 from 3169759 to 3630138 tid 3 2021-02-10T04:35:41.377 INFO:tasks.rados.rados.0.smithi085.stdout:2021: read oid 26 snap -1 2021-02-10T04:35:41.377 INFO:tasks.rados.rados.0.smithi085.stdout:2021: expect (ObjNum 687 snap 214 seq_num 687) 2021-02-10T04:35:41.377 INFO:tasks.rados.rados.0.smithi085.stdout:2022: rollback oid 40 current snap is 223 2021-02-10T04:35:41.377 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 40 to 211 2021-02-10T04:35:41.377 INFO:tasks.rados.rados.0.smithi085.stdout:2020: finishing write tid 1 to smithi08575397-50 2021-02-10T04:35:41.377 INFO:tasks.rados.rados.0.smithi085.stdout:2023: rollback oid 6 current snap is 223 2021-02-10T04:35:41.377 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 6 to 185 2021-02-10T04:35:41.377 INFO:tasks.rados.rados.0.smithi085.stdout:2024: read oid 8 snap -1 2021-02-10T04:35:41.377 INFO:tasks.rados.rados.0.smithi085.stdout:2024: expect (ObjNum 700 snap 218 seq_num 700) 2021-02-10T04:35:41.377 INFO:tasks.rados.rados.0.smithi085.stdout:2025: copy_from oid 28 from oid 36 current snap is 223 2021-02-10T04:35:41.378 INFO:tasks.rados.rados.0.smithi085.stdout:2026: delete oid 22 current snap is 223 2021-02-10T04:35:41.378 INFO:tasks.rados.rados.0.smithi085.stdout:2020: finishing write tid 2 to smithi08575397-50 2021-02-10T04:35:41.383 INFO:tasks.rados.rados.0.smithi085.stdout:2023: finishing rollback tid 0 to smithi08575397-6 2021-02-10T04:35:41.384 INFO:tasks.rados.rados.0.smithi085.stdout:2023: finishing rollback tid 1 to smithi08575397-6 2021-02-10T04:35:41.384 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 1027 (ObjNum 82529824 snap 22030 seq_num 0) dirty dne 2021-02-10T04:35:41.384 INFO:tasks.rados.rados.0.smithi085.stdout:2021: expect (ObjNum 687 snap 214 seq_num 687) 2021-02-10T04:35:41.546 INFO:tasks.rados.rados.0.smithi085.stdout:2021: done (15 left) 2021-02-10T04:35:41.546 INFO:tasks.rados.rados.0.smithi085.stdout:2023: done (14 left) 2021-02-10T04:35:41.546 INFO:tasks.rados.rados.0.smithi085.stdout:2026: done (13 left) 2021-02-10T04:35:41.546 INFO:tasks.rados.rados.0.smithi085.stdout:2027: write oid 46 current snap is 223 2021-02-10T04:35:41.546 INFO:tasks.rados.rados.0.smithi085.stdout:2020: finishing write tid 3 to smithi08575397-50 2021-02-10T04:35:41.546 INFO:tasks.rados.rados.0.smithi085.stdout:2027: seq_num 714 ranges {400198=548615,1415202=584996} 2021-02-10T04:35:41.551 INFO:tasks.rados.rados.0.smithi085.stdout:2027: writing smithi08575397-46 from 400198 to 948813 tid 1 2021-02-10T04:35:41.557 INFO:tasks.rados.rados.0.smithi085.stdout:2027: writing smithi08575397-46 from 1415202 to 2000198 tid 2 2021-02-10T04:35:41.557 INFO:tasks.rados.rados.0.smithi085.stdout:2028: copy_from oid 5 from oid 7 current snap is 223 2021-02-10T04:35:41.558 INFO:tasks.rados.rados.0.smithi085.stdout:2024: expect (ObjNum 700 snap 218 seq_num 700) 2021-02-10T04:35:41.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:41 smithi164 conmon[29288]: cluster 2021-02-10T04:35:40.342324+0000 mon.a (mon.0) 1348 : cluster [DBG] osdmap e523: 8 total, 8 up, 8 in 2021-02-10T04:35:41.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:41 smithi085 conmon[29319]: cluster 2021-02-10T04:35:40.342324+0000 2021-02-10T04:35:41.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:41 smithi085 conmon[29319]: mon.a (mon.0) 1348 : cluster [DBG] osdmap e523: 8 total, 8 up, 8 in 2021-02-10T04:35:41.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:41 smithi085 conmon[35961]: cluster 2021-02-10T04:35:41.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:41 smithi085 conmon[35961]: 2021-02-10T04:35:41.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:41 smithi085 conmon[35961]: 2021 2021-02-10T04:35:41.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:41 smithi085 conmon[35961]: - 2021-02-10T04:35:41.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:41 smithi085 conmon[35961]: 02 2021-02-10T04:35:41.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:41 smithi085 conmon[35961]: - 2021-02-10T04:35:41.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:41 smithi085 conmon[35961]: 10 2021-02-10T04:35:41.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:41 smithi085 conmon[35961]: T 2021-02-10T04:35:41.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:41 smithi085 conmon[35961]: 04 2021-02-10T04:35:41.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:41 smithi085 conmon[35961]: : 2021-02-10T04:35:41.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:41 smithi085 conmon[35961]: 35 2021-02-10T04:35:41.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:41 smithi085 conmon[35961]: : 2021-02-10T04:35:41.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:41 smithi085 conmon[35961]: 40 2021-02-10T04:35:41.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:41 smithi085 conmon[35961]: . 2021-02-10T04:35:41.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:41 smithi085 conmon[35961]: 342324 2021-02-10T04:35:41.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:41 smithi085 conmon[35961]: +0000 2021-02-10T04:35:41.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:41 smithi085 conmon[35961]: 2021-02-10T04:35:41.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:41 smithi085 conmon[35961]: mon.a 2021-02-10T04:35:41.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:41 smithi085 conmon[35961]: ( 2021-02-10T04:35:41.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:41 smithi085 conmon[35961]: mon 2021-02-10T04:35:41.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:41 smithi085 conmon[35961]: . 2021-02-10T04:35:41.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:41 smithi085 conmon[35961]: 0) 1348 : cluster [DBG] osdmap e523: 8 total, 8 up, 8 in 2021-02-10T04:35:41.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:41 smithi085 conmon[35961]: 2021-02-10T04:35:41.964 INFO:tasks.rados.rados.0.smithi085.stdout:2024: done (14 left) 2021-02-10T04:35:41.964 INFO:tasks.rados.rados.0.smithi085.stdout:2029: write oid 13 current snap is 223 2021-02-10T04:35:41.964 INFO:tasks.rados.rados.0.smithi085.stdout:2020: finishing write tid 4 to smithi08575397-50 2021-02-10T04:35:41.964 INFO:tasks.rados.rados.0.smithi085.stdout:2029: seq_num 715 ranges {417005=740217,1590037=426968} 2021-02-10T04:35:41.970 INFO:tasks.rados.rados.0.smithi085.stdout:2029: writing smithi08575397-13 from 417005 to 1157222 tid 1 2021-02-10T04:35:41.976 INFO:tasks.rados.rados.0.smithi085.stdout:2029: writing smithi08575397-13 from 1590037 to 2017005 tid 2 2021-02-10T04:35:41.976 INFO:tasks.rados.rados.0.smithi085.stdout:2030: snap_create 2021-02-10T04:35:41.976 INFO:tasks.rados.rados.0.smithi085.stdout:2020: finishing write tid 5 to smithi08575397-50 2021-02-10T04:35:41.976 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 765 (ObjNum 712 snap 223 seq_num 712) dirty exists 2021-02-10T04:35:41.976 INFO:tasks.rados.rados.0.smithi085.stdout:2020: left oid 50 (ObjNum 712 snap 223 seq_num 712) 2021-02-10T04:35:41.977 INFO:tasks.rados.rados.0.smithi085.stdout:2025: finishing copy_from to smithi08575397-28 2021-02-10T04:35:41.977 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 673 (ObjNum 694 snap 215 seq_num 694) dirty exists 2021-02-10T04:35:41.977 INFO:tasks.rados.rados.0.smithi085.stdout:2025: finishing copy_from racing read to smithi08575397-28 2021-02-10T04:35:41.977 INFO:tasks.rados.rados.0.smithi085.stdout:2022: finishing rollback tid 0 to smithi08575397-40 2021-02-10T04:35:41.977 INFO:tasks.rados.rados.0.smithi085.stdout:2022: finishing rollback tid 1 to smithi08575397-40 2021-02-10T04:35:41.977 INFO:tasks.rados.rados.0.smithi085.stdout:2022: finishing rollback tid 2 to smithi08575397-40 2021-02-10T04:35:41.977 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 1088 (ObjNum 660 snap 202 seq_num 660) dirty exists 2021-02-10T04:35:41.977 INFO:tasks.rados.rados.0.smithi085.stdout:2027: finishing write tid 1 to smithi08575397-46 2021-02-10T04:35:41.977 INFO:tasks.rados.rados.0.smithi085.stdout:2027: finishing write tid 2 to smithi08575397-46 2021-02-10T04:35:41.977 INFO:tasks.rados.rados.0.smithi085.stdout:2027: finishing write tid 3 to smithi08575397-46 2021-02-10T04:35:41.977 INFO:tasks.rados.rados.0.smithi085.stdout:2027: finishing write tid 4 to smithi08575397-46 2021-02-10T04:35:41.977 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 1030 (ObjNum 714 snap 223 seq_num 714) dirty exists 2021-02-10T04:35:41.978 INFO:tasks.rados.rados.0.smithi085.stdout:2027: left oid 46 (ObjNum 714 snap 223 seq_num 714) 2021-02-10T04:35:41.978 INFO:tasks.rados.rados.0.smithi085.stdout:2028: finishing copy_from to smithi08575397-5 2021-02-10T04:35:41.978 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 740 (ObjNum 708 snap 222 seq_num 708) dirty exists 2021-02-10T04:35:41.978 INFO:tasks.rados.rados.0.smithi085.stdout:2028: finishing copy_from racing read to smithi08575397-5 2021-02-10T04:35:41.978 INFO:tasks.rados.rados.0.smithi085.stdout:2029: finishing write tid 1 to smithi08575397-13 2021-02-10T04:35:41.979 INFO:tasks.rados.rados.0.smithi085.stdout:2029: finishing write tid 2 to smithi08575397-13 2021-02-10T04:35:41.979 INFO:tasks.rados.rados.0.smithi085.stdout:2029: finishing write tid 3 to smithi08575397-13 2021-02-10T04:35:41.980 INFO:tasks.rados.rados.0.smithi085.stdout:2029: finishing write tid 4 to smithi08575397-13 2021-02-10T04:35:41.980 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 743 (ObjNum 715 snap 223 seq_num 715) dirty exists 2021-02-10T04:35:41.980 INFO:tasks.rados.rados.0.smithi085.stdout:2029: left oid 13 (ObjNum 715 snap 223 seq_num 715) 2021-02-10T04:35:42.051 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:35:42.082 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:35:42.329 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 5 1 2021-02-10T04:35:42.352 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:35:42 smithi164 conmon[37263]: debug 2021-02-10T04:35:42.352 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:35:42 smithi164 conmon[37263]: 2021-02-10T04:35:42.154+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:35:42.352 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:35:42 smithi164 conmon[37263]: 2021-02-10T04:35:42.353 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:42 smithi164 conmon[29288]: cluster 2021-02-10T04:35:41.349223+0000 mon.a (mon.0) 1349 : cluster [DBG] osdmap e524: 8 total, 8 up, 8 in 2021-02-10T04:35:42.353 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:42 smithi164 conmon[29288]: cluster 2021-02-10T04:35:42.105426+0000 mgr.y (mgr.14140) 604 : cluster [DBG] pgmap v936: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 374 MiB data, 982 MiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 1.8 MiB/s wr, 23 op/s 2021-02-10T04:35:42.357 INFO:tasks.rados.rados.0.smithi085.stdout:2020: done (15 left) 2021-02-10T04:35:42.358 INFO:tasks.rados.rados.0.smithi085.stdout:2022: done (14 left) 2021-02-10T04:35:42.358 INFO:tasks.rados.rados.0.smithi085.stdout:2025: done (13 left) 2021-02-10T04:35:42.358 INFO:tasks.rados.rados.0.smithi085.stdout:2027: done (12 left) 2021-02-10T04:35:42.358 INFO:tasks.rados.rados.0.smithi085.stdout:2028: done (11 left) 2021-02-10T04:35:42.358 INFO:tasks.rados.rados.0.smithi085.stdout:2029: done (10 left) 2021-02-10T04:35:42.358 INFO:tasks.rados.rados.0.smithi085.stdout:2030: done (9 left) 2021-02-10T04:35:42.358 INFO:tasks.rados.rados.0.smithi085.stdout:2031: rollback oid 11 current snap is 224 2021-02-10T04:35:42.358 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 11 to 190 2021-02-10T04:35:42.358 INFO:tasks.rados.rados.0.smithi085.stdout:2032: rollback oid 28 current snap is 224 2021-02-10T04:35:42.358 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 28 to 221 2021-02-10T04:35:42.358 INFO:tasks.rados.rados.0.smithi085.stdout:2033: snap_remove snap 210 2021-02-10T04:35:42.509 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:35:42.558 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:42 smithi085 conmon[29319]: cluster 2021-02-10T04:35:41.349223+0000 mon.a (mon.0) 1349 : cluster [DBG] osdmap e524: 8 total, 8 up, 8 in 2021-02-10T04:35:42.558 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:42 smithi085 conmon[29319]: cluster 2021-02-10T04:35:42.105426+0000 mgr.y (mgr.14140) 604 : cluster [DBG] pgmap v936: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 374 MiB data, 982 MiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 1.8 MiB/s wr, 23 op/s 2021-02-10T04:35:42.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:42 smithi085 conmon[35961]: cluster 2021-02-10T04:35:41.349223+0000 mon.a 2021-02-10T04:35:42.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:42 smithi085 conmon[35961]: (mon.0) 1349 : cluster [DBG] osdmap e524: 8 total, 8 up, 8 in 2021-02-10T04:35:42.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:42 smithi085 conmon[35961]: cluster 2021-02-10T04:35:42.105426+0000 mgr.y (mgr.14140) 604 : cluster [DBG] pgmap v936: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 374 MiB data, 982 MiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 1.8 MiB/s wr, 23 op/s 2021-02-10T04:35:42.657 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:35:42.951 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:35:42.982 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:35:43.012 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:35:43.033 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:35:43.068 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:35:43.093 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:35:43.129 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:35:43.157 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:35:43.162 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:35:43.162 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:35:43.162 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:35:43.163 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:35:43.163 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:35:43.163 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:35:43.163 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:35:43.163 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:35:43.361 INFO:tasks.rados.rados.0.smithi085.stdout:2033: done (11 left) 2021-02-10T04:35:43.361 INFO:tasks.rados.rados.0.smithi085.stdout:2034: delete oid 38 current snap is 224 2021-02-10T04:35:43.362 INFO:tasks.rados.rados.0.smithi085.stdout:2031: finishing rollback tid 1 to smithi08575397-11 2021-02-10T04:35:43.362 INFO:tasks.rados.rados.0.smithi085.stdout:2031: finishing rollback tid 2 to smithi08575397-11 2021-02-10T04:35:43.362 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 1090 (ObjNum 531 snap 156 seq_num 531) dirty exists 2021-02-10T04:35:43.362 INFO:tasks.rados.rados.0.smithi085.stdout:2032: finishing rollback tid 0 to smithi08575397-28 2021-02-10T04:35:43.362 INFO:tasks.rados.rados.0.smithi085.stdout:2032: finishing rollback tid 1 to smithi08575397-28 2021-02-10T04:35:43.362 INFO:tasks.rados.rados.0.smithi085.stdout:2032: finishing rollback tid 2 to smithi08575397-28 2021-02-10T04:35:43.362 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 922 (ObjNum 651 snap 197 seq_num 651) dirty exists 2021-02-10T04:35:43.371 INFO:tasks.rados.rados.0.smithi085.stdout:2031: done (11 left) 2021-02-10T04:35:43.371 INFO:tasks.rados.rados.0.smithi085.stdout:2032: done (10 left) 2021-02-10T04:35:43.372 INFO:tasks.rados.rados.0.smithi085.stdout:2034: done (9 left) 2021-02-10T04:35:43.372 INFO:tasks.rados.rados.0.smithi085.stdout:2035: read oid 30 snap 222 2021-02-10T04:35:43.372 INFO:tasks.rados.rados.0.smithi085.stdout:2035: expect (ObjNum 676 snap 208 seq_num 676) 2021-02-10T04:35:43.372 INFO:tasks.rados.rados.0.smithi085.stdout:2036: snap_remove snap 192 2021-02-10T04:35:43.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:43 smithi085 conmon[29319]: cluster 2021-02-10T04:35:42.356282+0000 mon.a 2021-02-10T04:35:43.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:43 smithi085 conmon[29319]: (mon.0) 1350 : cluster [DBG] osdmap e525: 8 total, 8 up, 8 in 2021-02-10T04:35:43.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:43 smithi085 conmon[29319]: audit 2021-02-10T04:35:42.921857+0000 mon.c (mon.1) 58 : audit [DBG] from='client.? 172.21.15.85:0/3914310094' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:35:43.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:43 smithi085 conmon[35961]: cluster 2021-02-10T04:35:42.356282+0000 2021-02-10T04:35:43.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:43 smithi085 conmon[35961]: mon.a (mon.0) 1350 : cluster [DBG] osdmap e525: 8 total, 8 up, 8 in 2021-02-10T04:35:43.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:43 smithi085 conmon[35961]: audit 2021-02-10T04:35:42.921857+0000 mon.c (mon.1) 58 : audit [DBG] from='client.? 172.21.15.85:0/3914310094' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:35:43.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:43 smithi164 conmon[29288]: cluster 2021-02-10T04:35:42.356282 2021-02-10T04:35:43.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:43 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e525: 8 total, 8 up, 8 in 2021-02-10T04:35:43.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:43 smithi164 conmon[29288]: audit 2021-02-10T04:35:42.921857+0000 mon.c (mon.1) 58 : audit 2021-02-10T04:35:43.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:43 smithi164 conmon[29288]: [DBG] from='client.? 172.21.15.85:0/3914310094' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:35:44.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:43 smithi085 conmon[49106]: debug 2021-02-10T04:35:43.790+0000 7f77a3e94700 -1 osd.2 526 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:35:44.267 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:35:44.306 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:35:44.369 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:43 smithi164 conmon[33677]: debug 2021-02-10T04:35:43.979+0000 7fbec19a1700 -1 osd.4 526 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:35:44.369 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:44 smithi164 conmon[29288]: cluster 2021-02-10T04:35:43.361792+0000 mon.a 2021-02-10T04:35:44.369 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:44 smithi164 conmon[29288]: (mon.0) 1351 : cluster [DBG] osdmap e526: 8 total, 8 up, 8 in 2021-02-10T04:35:44.369 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:44 smithi164 conmon[29288]: audit 2021-02-10T04:35:43.409893+0000 mon.b (mon.2) 34 : audit 2021-02-10T04:35:44.369 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:44 smithi164 conmon[29288]: [INF] from='client.? 172.21.15.85:0/931513426' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:35:44.370 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:44 smithi164 conmon[29288]: audit 2021-02-10T04:35:44.370 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:44 smithi164 conmon[29288]: 2021-02-10T04:35:43.410335+0000 mon.a ( 2021-02-10T04:35:44.370 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:44 smithi164 conmon[29288]: mon.0) 1352 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:35:44.370 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:44 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:35:44.370 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:44 smithi164 conmon[29288]: -02-10T04:35:43.980919+0000 2021-02-10T04:35:44.370 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:44 smithi164 conmon[29288]: osd.4 (osd.4) 713 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:44.370 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:44 smithi164 conmon[29288]: 2021-02-10T04:35:44.370 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:44 smithi164 conmon[29288]: cluster 2021-02-10T04:35:44.370 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:44 smithi164 conmon[29288]: 2021-02-10T04:35:43.980928+0000 osd.4 2021-02-10T04:35:44.371 INFO:tasks.rados.rados.0.smithi085.stdout:2036: done (10 left) 2021-02-10T04:35:44.371 INFO:tasks.rados.rados.0.smithi085.stdout:2037: read oid 50 snap -1 2021-02-10T04:35:44.371 INFO:tasks.rados.rados.0.smithi085.stdout:2037: expect (ObjNum 712 snap 223 seq_num 712) 2021-02-10T04:35:44.371 INFO:tasks.rados.rados.0.smithi085.stdout:2038: write oid 48 current snap is 224 2021-02-10T04:35:44.371 INFO:tasks.rados.rados.0.smithi085.stdout:2035: expect (ObjNum 676 snap 208 seq_num 676) 2021-02-10T04:35:44.380 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is set 2021-02-10T04:35:44.562 INFO:tasks.rados.rados.0.smithi085.stdout:2038: seq_num 717 ranges {450619=490715,1590648=459971} 2021-02-10T04:35:44.568 INFO:tasks.rados.rados.0.smithi085.stdout:2038: writing smithi08575397-48 from 450619 to 941334 tid 1 2021-02-10T04:35:44.576 INFO:tasks.rados.rados.0.smithi085.stdout:2038: writing smithi08575397-48 from 1590648 to 2050619 tid 2 2021-02-10T04:35:44.576 INFO:tasks.rados.rados.0.smithi085.stdout:2035: done (11 left) 2021-02-10T04:35:44.576 INFO:tasks.rados.rados.0.smithi085.stdout:2039: snap_create 2021-02-10T04:35:44.576 INFO:tasks.rados.rados.0.smithi085.stdout:2037: expect (ObjNum 712 snap 223 seq_num 712) 2021-02-10T04:35:44.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:44 smithi164 conmon[29288]: (osd.4) 714 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:44 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:35:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:44 smithi164 conmon[29288]: 02-10T04:35:43.980932+0000 osd.4 (osd. 2021-02-10T04:35:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:44 smithi164 conmon[29288]: 4) 715 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:44 smithi164 conmon[29288]: cluster 2021-02-10T04:35:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:44 smithi164 conmon[29288]: 2021-02-10T04:35:43.980936+0000 osd.4 (osd.4) 716 : 2021-02-10T04:35:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:44 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:44 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:35:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:44 smithi164 conmon[29288]: 02-10T04:35:44.105992+0000 mgr.y (mgr.14140) 605 : cluster [DBG] pgmap v939: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 389 MiB data, 1003 MiB used, 714 GiB / 715 GiB avail; 23 MiB/s rd, 3.9 MiB/s wr, 26 op/s 2021-02-10T04:35:44.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:44 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:35:44.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:44 smithi085 conmon[29319]: 04:35:43.361792+0000 mon.a (mon.0) 1351 : cluster [DBG] osdmap e526: 8 total, 8 up, 8 in 2021-02-10T04:35:44.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:44 smithi085 conmon[29319]: audit 2021-02-10T04:35:43.409893+0000 mon.b (mon.2) 34 : audit [INF] from='client.? 172.21.15.85:0/931513426' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:35:44.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:44 smithi085 conmon[29319]: audit 2021-02-10T04:35:43.410335+0000 mon.a (mon.0) 1352 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:35:44.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:44 smithi085 conmon[29319]: cluster 2021-02-10T04:35:43.980919+0000 osd.4 (osd.4) 713 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:44.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:44 smithi085 conmon[29319]: cluster 2021-02-10T04:35:43.980928+0000 osd.4 (osd.4) 714 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:44.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:44 smithi085 conmon[29319]: cluster 2021-02-10T04:35:43.980932+0000 osd.4 (osd.4) 715 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:44.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:44 smithi085 conmon[29319]: cluster 2021-02-10T04:35:43.980936+0000 osd.4 (osd.4) 716 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:44.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:44 smithi085 conmon[29319]: cluster 2021-02-10T04:35:44.105992+0000 mgr.y (mgr.14140) 605 : cluster [DBG] pgmap v939: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 389 MiB data, 1003 MiB used, 714 GiB / 715 GiB avail; 23 MiB/s rd, 3.9 MiB/s wr, 26 op/s 2021-02-10T04:35:44.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:44 smithi085 conmon[35961]: cluster 2021-02-10T04:35:43.361792+0000 2021-02-10T04:35:44.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:44 smithi085 conmon[35961]: mon.a (mon.0) 1351 : cluster [DBG] osdmap e526: 8 total, 8 up, 8 in 2021-02-10T04:35:44.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:44 smithi085 conmon[35961]: audit 2021-02-10T04:35:43.409893+0000 mon.b (mon 2021-02-10T04:35:44.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:44 smithi085 conmon[35961]: .2) 34 : audit [INF] from='client.? 172.21.15.85:0/931513426' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:35:44.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:44 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:35:44.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:44 smithi085 conmon[35961]: 02-10T04:35:43.410335+0000 mon.a (mon.0) 1352 : audit 2021-02-10T04:35:44.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:44 smithi085 conmon[35961]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:35:44.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:44 smithi085 conmon[35961]: cluster 2021-02-10T04:35:44.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:44 smithi085 conmon[35961]: 2021-02-10T04:35:43.980919+0000 osd.4 (osd.4) 713 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:44.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:44 smithi085 conmon[35961]: cluster 2021-02-10T04:35: 2021-02-10T04:35:44.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:44 smithi085 conmon[35961]: 43.980928+0000 osd.4 (osd.4) 714 : cluster 2021-02-10T04:35:44.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:44 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:44.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:44 smithi085 conmon[35961]: cluster 2021-02-10T04:35:44.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:44 smithi085 conmon[35961]: 2021-02-10T04:35:43.980932+0000 osd.4 (osd.4 2021-02-10T04:35:44.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:44 smithi085 conmon[35961]: ) 715 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:44.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:44 smithi085 conmon[35961]: cluster 2021-02-10T04:35:43. 2021-02-10T04:35:44.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:44 smithi085 conmon[35961]: 980936+0000 osd.4 (osd.4) 716 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:44.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:44 smithi085 conmon[35961]: cluster 2021-02-10T04:35:44 2021-02-10T04:35:44.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:44 smithi085 conmon[35961]: .105992+0000 mgr.y (mgr.14140) 605 : cluster [DBG] pgmap v939: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 389 MiB data, 1003 MiB used, 714 GiB / 715 GiB avail; 23 MiB/s rd, 3.9 MiB/s wr, 26 op/s 2021-02-10T04:35:44.817 INFO:tasks.rados.rados.0.smithi085.stdout:2038: finishing write tid 1 to smithi08575397-48 2021-02-10T04:35:44.818 INFO:tasks.rados.rados.0.smithi085.stdout:2038: finishing write tid 2 to smithi08575397-48 2021-02-10T04:35:44.818 INFO:tasks.rados.rados.0.smithi085.stdout:2038: finishing write tid 3 to smithi08575397-48 2021-02-10T04:35:44.818 INFO:tasks.rados.rados.0.smithi085.stdout:2038: finishing write tid 4 to smithi08575397-48 2021-02-10T04:35:44.818 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 814 (ObjNum 717 snap 224 seq_num 717) dirty exists 2021-02-10T04:35:44.818 INFO:tasks.rados.rados.0.smithi085.stdout:2038: left oid 48 (ObjNum 717 snap 224 seq_num 717) 2021-02-10T04:35:45.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:44 smithi085 conmon[49106]: debug 2021-02-10T04:35:44.792+0000 7f77a3e94700 -1 osd.2 527 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:35:45.368 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:44 smithi164 conmon[33677]: debug 2021-02-10T04:35:44.980+0000 7fbec19a1700 -1 osd.4 527 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:35:45.376 INFO:tasks.rados.rados.0.smithi085.stdout:2037: done (11 left) 2021-02-10T04:35:45.377 INFO:tasks.rados.rados.0.smithi085.stdout:2038: done (10 left) 2021-02-10T04:35:45.377 INFO:tasks.rados.rados.0.smithi085.stdout:2039: done (9 left) 2021-02-10T04:35:45.377 INFO:tasks.rados.rados.0.smithi085.stdout:2040: rollback oid 42 current snap is 225 2021-02-10T04:35:45.377 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 42 to 217 2021-02-10T04:35:45.377 INFO:tasks.rados.rados.0.smithi085.stdout:2041: snap_create 2021-02-10T04:35:45.385 INFO:tasks.rados.rados.0.smithi085.stdout:2040: finishing rollback tid 1 to smithi08575397-42 2021-02-10T04:35:45.385 INFO:tasks.rados.rados.0.smithi085.stdout:2040: finishing rollback tid 2 to smithi08575397-42 2021-02-10T04:35:45.385 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 816 (ObjNum 646 snap 196 seq_num 646) dirty exists 2021-02-10T04:35:45.586 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:35:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: cluster 2021-02-10T04:35:43.790793+0000 2021-02-10T04:35:47.697 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: osd.2 (osd.2) 1289 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:47.697 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: cluster 2021-02-10T04:35:43.790806+0000 osd.2 (osd.2) 1290 : cluster [WRN] 2021-02-10T04:35:47.697 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:47.697 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: cluster 2021-02-10T04:35:43 2021-02-10T04:35:47.697 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: .790813+0000 osd.2 (osd.2) 1291 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:47.697 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:35:47.697 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: 35:43.790819+0000 osd.2 (osd.2) 1292 : 2021-02-10T04:35:47.697 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:47.697 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: cluster 2021-02-10T04:35:43. 2021-02-10T04:35:47.697 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: 790837+0000 osd.2 (osd.2) 1293 : cluster 2021-02-10T04:35:47.698 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:47.698 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: cluster 2021-02-10T04:35:47.698 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: 2021-02-10T04:35:43.790851+0000 osd.2 (osd.2 2021-02-10T04:35:47.698 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: ) 1294 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:47.698 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: cluster 2021-02-10T04:35 2021-02-10T04:35:47.698 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: :43.790858+0000 osd.2 (osd.2) 1295 : 2021-02-10T04:35:47.698 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:47.698 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:35:47.698 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: 04:35:43.790866+0000 2021-02-10T04:35:47.698 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: osd.2 (osd.2) 1296 : cluster 2021-02-10T04:35:47.699 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:47.699 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: cluster 2021-02-10T04:35:43.790873 2021-02-10T04:35:47.699 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1297 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:47.699 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: 2021-02-10T04:35:47.699 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: cluster 2021-02-10T04:35:47.699 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: 2021-02-10T04:35:43.790879+0000 2021-02-10T04:35:47.699 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: osd.2 (osd.2) 1298 : 2021-02-10T04:35:47.699 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:47.699 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: cluster 2021-02-10T04:35:47.699 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: 2021-02-10T04:35 2021-02-10T04:35:47.699 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: :44.359332+0000 mon.a ( 2021-02-10T04:35:47.700 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: mon.0) 1353 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:35:47.700 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: audit 2021-02-10T04:35 2021-02-10T04:35:47.700 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: :44.370544+0000 mon.a (mon.0) 1354 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:35:47.700 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:35:47.700 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: 04:35:44.370628+0000 mon.a (mon.0) 1355 : cluster [DBG] 2021-02-10T04:35:47.700 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: osdmap e527: 8 total, 8 up, 8 in 2021-02-10T04:35:47.700 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: cluster 2021-02-10T04:35 2021-02-10T04:35:47.700 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: :44.376280+0000 osd.4 (osd.4) 717 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:35:47.700 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: cluster 2021-02-10T04:35:47.700 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: 2021-02-10T04:35:44.376549+0000 osd.4 (osd.4) 718 : cluster [DBG] 2021-02-10T04:35:47.700 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: purged_snaps scrub ok 2021-02-10T04:35:47.701 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: cluster 2021-02-10T04:35:44 2021-02-10T04:35:47.701 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: .982460+0000 osd.4 (osd.4) 719 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:47.701 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: cluster 2021-02-10T04:35:44. 2021-02-10T04:35:47.701 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: 982487+0000 osd.4 (osd.4) 720 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:47.701 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:35:47.701 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: 10T04:35:44.982506+0000 osd.4 ( 2021-02-10T04:35:47.701 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: osd.4) 721 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:47.701 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:35:47.701 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: :35:44.982510+0000 osd.4 (osd.4) 722 2021-02-10T04:35:47.701 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:47.709 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[49106]: debug 2021-02-10T04:35:45.287+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:35:47.709 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[49106]: debug 2021-02-10T04:35:47.709 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[49106]: 2021-02-10T04:35:45.754+0000 7f77a3e94700 -1 osd.2 528 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:35:47.709 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[49106]: 2021-02-10T04:35:47.709 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[49106]: debug 2021-02-10T04:35:46.766+0000 7f77a3e94700 -1 osd.2 529 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:35:47.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: cluster 2021-02-10T04:35:43 2021-02-10T04:35:47.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: .790793+0000 osd.2 (osd.2) 1289 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:47.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: cluster 2021-02-10T04:35:43.790806+0000 2021-02-10T04:35:47.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: osd.2 (osd.2) 1290 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:47.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:35:47.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: 10T04:35:43.790813+0000 osd.2 (osd.2) 1291 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:47.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: cluster 2021-02-10T04:35:43.790819+0000 2021-02-10T04:35:47.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: osd.2 (osd.2) 1292 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:47.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: cluster 2021-02-10T04:35:43.790837 2021-02-10T04:35:47.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1293 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:47.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: cluster 2021-02-10T04:35:43.790851+0000 2021-02-10T04:35:47.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: osd.2 (osd.2) 1294 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:47.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: cluster 2021-02-10T04:35:43.790858+0000 2021-02-10T04:35:47.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: osd.2 (osd.2) 1295 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:47.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: cluster 2021-02-10T04:35:43.790866+0000 2021-02-10T04:35:47.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: osd.2 (osd.2) 1296 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:47.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: cluster 2021-02-10T04:35:43.790873+0000 2021-02-10T04:35:47.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: osd.2 (osd.2) 1297 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:47.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: cluster 2021-02-10T04:35:43.790879+0000 2021-02-10T04:35:47.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: osd.2 (osd.2) 1298 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:47.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: 2021-02-10T04:35:47.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:35:47.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: 35:44.359332+0000 mon.a (mon.0) 1353 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:35:47.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: audit 2021-02-10T04:35:44.370544+0000 2021-02-10T04:35:47.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: mon.a (mon.0) 1354 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:35:47.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: cluster 2021-02-10T04:35:44.370628+0000 mon.a (mon.0) 1355 : cluster [DBG] osdmap e527: 8 total, 8 up, 8 in 2021-02-10T04:35:47.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: cluster 2021-02-10T04:35:44.376280+0000 2021-02-10T04:35:47.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: osd.4 (osd.4) 717 : cluster [DBG] 2021-02-10T04:35:47.712 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: purged_snaps scrub starts 2021-02-10T04:35:47.712 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: cluster 2021-02-10T04:35:47.712 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: 2021-02-10T04:35:44.376549+0000 osd.4 (osd 2021-02-10T04:35:47.712 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: .4) 718 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:35:47.712 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: cluster 2021-02-10T04:35:44 2021-02-10T04:35:47.712 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: .982460+0000 osd.4 (osd.4) 719 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:47.712 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: cluster 2021-02-10T04:35:47.712 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: 2021-02-10T04:35:44.982487+0000 osd.4 (osd.4) 720 : cluster 2021-02-10T04:35:47.712 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:47.712 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:35:47.712 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: T04:35:44.982506+0000 osd.4 (osd.4) 721 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:47.712 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: cluster 2021-02-10T04:35:44.982510+0000 osd.4 (osd. 2021-02-10T04:35:47.712 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[29319]: 4) 722 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:47.713 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: cluster 2021-02-10T04:35:44.793369+0000 osd.2 (osd 2021-02-10T04:35:47.713 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: .2) 1299 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:47.713 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: cluster 2021-02-10T04:35:44.793381+0000 osd.2 (osd.2) 1300 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:47.713 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: cluster 2021-02-10T04:35:44.793389+0000 2021-02-10T04:35:47.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: osd.2 (osd.2) 1301 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:47.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:47.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: 02-10T04:35:44.793397+0000 osd.2 2021-02-10T04:35:47.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: (osd.2) 1302 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:47.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: 2021-02-10T04:35:47.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: cluster 2021-02-10T04:35:47.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: 2021-02-10T04:35:44.793403+0000 osd.2 (osd 2021-02-10T04:35:47.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: .2) 1303 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:47.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:35:47.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: 10T04:35:44.793429+0000 osd.2 ( 2021-02-10T04:35:47.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: osd.2) 1304 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:47.714 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:35:47.715 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: -10T04:35:44.793444+0000 osd.2 2021-02-10T04:35:47.715 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: (osd.2) 1305 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:47.715 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: 2021-02-10T04:35:47.715 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: cluster 2021-02-10T04:35:47.715 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: 2021-02-10T04:35:44.793448+0000 osd.2 (osd. 2021-02-10T04:35:47.715 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: 2) 1306 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:47.715 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:35:47.715 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: 10T04:35:44.793454+0000 osd.2 ( 2021-02-10T04:35:47.715 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: osd.2) 1307 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:47.716 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: cluster 2021-02-10T04:35:44 2021-02-10T04:35:47.716 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: .793459+0000 osd.2 (osd.2) 1308 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:47.716 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:47.716 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: 02-10T04:35:45.376197+0000 mon.a (mon.0) 1356 : cluster 2021-02-10T04:35:47.716 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: [DBG] osdmap e528: 8 total, 8 up, 8 in 2021-02-10T04:35:47.716 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: cluster 2021-02-10T04:35:45 2021-02-10T04:35:47.716 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: .951436+0000 osd.4 (osd.4) 723 : cluster 2021-02-10T04:35:47.716 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:47.717 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:35:47.717 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: 35:45.951456+0000 osd.4 (osd.4) 724 2021-02-10T04:35:47.717 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:47.717 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: cluster 2021-02-10T04:35:47.717 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: 2021-02-10T04:35:45 2021-02-10T04:35:47.717 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: .951460+0000 osd.4 (osd.4) 725 : cluster 2021-02-10T04:35:47.717 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:47.717 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:35:47.717 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: T04:35:45.951463+0000 osd.4 (osd 2021-02-10T04:35:47.718 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: .4) 726 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:47.718 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:35:47.718 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: -02-10T04:35:46.106508 2021-02-10T04:35:47.718 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 606 : cluster [DBG] 2021-02-10T04:35:47.718 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[29319]: pgmap v942: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 396 MiB data, 1023 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.6 MiB/s wr, 17 op/s 2021-02-10T04:35:47.718 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: cluster 2021-02-10T04:35:45.755470+0000 2021-02-10T04:35:47.718 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: osd.2 (osd.2) 1309 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:47.718 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: cluster 2021-02-10T04:35:45.755478+0000 osd.2 (osd.2) 1310 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:47.719 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: cluster 2021-02-10T04:35:45.755485+0000 osd.2 2021-02-10T04:35:47.719 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: (osd.2) 1311 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:47.719 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:35:47.719 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: T04:35:45.755492+0000 osd.2 (osd.2) 1312 : 2021-02-10T04:35:47.719 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:47.719 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: cluster 2021-02-10T04:35:45.755499+0000 osd.2 (osd 2021-02-10T04:35:47.719 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: .2) 1313 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:47.719 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:35:47.720 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: 04:35:45.755506+0000 osd.2 (osd.2) 1314 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:47.720 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: cluster 2021-02-10T04:35:45.755513+0000 2021-02-10T04:35:47.720 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: osd.2 (osd.2) 1315 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:47.720 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:35:47.720 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: -10T04:35:45.755520+0000 osd.2 (osd 2021-02-10T04:35:47.720 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: .2) 1316 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:47.720 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:47.720 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: 02-10T04:35:45.755525+0000 2021-02-10T04:35:47.720 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: osd.2 (osd.2) 1317 : cluster [WRN] 2021-02-10T04:35:47.721 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:47.721 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:47.721 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: 02-10T04:35:45.755531+0000 2021-02-10T04:35:47.721 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: osd.2 (osd.2) 1318 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:47.721 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: 2021-02-10T04:35:47.721 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: cluster 2021-02-10T04:35:47.721 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: 2021-02-10T04:35:46.372270+0000 mon.a ( 2021-02-10T04:35:47.721 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: mon.0) 1357 : cluster [WRN] Health check failed: 10 slow ops, oldest one blocked for 353 sec, osd.2 has slow ops (SLOW_OPS) 2021-02-10T04:35:47.721 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:35:47.722 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: -02-10T04:35:46.381326 2021-02-10T04:35:47.722 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1358 : cluster [DBG] 2021-02-10T04:35:47.722 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: osdmap e529: 8 total, 8 up, 8 in 2021-02-10T04:35:47.722 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:35:47.722 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: -10T04:35:46.967149+0000 2021-02-10T04:35:47.722 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: osd.4 (osd.4) 727 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:47.722 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:35:47.722 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: 35:46.967157+0000 osd.4 (osd.4) 728 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:47.723 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: cluster 2021-02-10T04:35: 2021-02-10T04:35:47.724 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: 46.967165+0000 osd.4 (osd.4) 729 : cluster [WRN] 2021-02-10T04:35:47.724 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:47.724 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: cluster 2021-02-10T04:35:46 2021-02-10T04:35:47.724 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: .967170+0000 osd.4 (osd.4) 730 : cluster [WRN] 2021-02-10T04:35:47.724 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:47.724 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: cluster 2021-02-10T04:35:43.790793+0000 2021-02-10T04:35:47.724 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: osd.2 (osd.2) 1289 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:47.724 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:35:47.725 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: -10T04:35:43.790806+0000 osd.2 (osd.2) 1290 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:47.725 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: 2021-02-10T04:35:47.725 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:47.725 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: 02-10T04:35:43.790813+0000 2021-02-10T04:35:47.725 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: osd.2 (osd.2) 1291 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:47.725 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: 2021-02-10T04:35:47.725 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:35:47.725 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: -10T04:35:43.790819+0000 osd.2 (osd.2) 2021-02-10T04:35:47.725 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: 1292 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:47.725 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: cluster 2021-02-10T04:35 2021-02-10T04:35:47.726 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: :43.790837+0000 osd.2 (osd.2) 1293 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:47.726 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:47.726 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: 02-10T04:35:43.790851+0000 osd.2 (osd.2) 1294 : cluster 2021-02-10T04:35:47.726 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:47.726 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: cluster 2021-02-10T04:35:43.790858+0000 osd.2 (osd.2 2021-02-10T04:35:47.726 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: ) 1295 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:47.726 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: cluster 2021-02-10T04:35:47.726 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:35:47.726 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: 35:43.790866+0000 osd.2 (osd.2) 1296 : cluster 2021-02-10T04:35:47.727 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:47.727 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:47.727 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: 02-10T04:35:43.790873+0000 osd.2 (osd.2) 1297 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:47.727 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: cluster 2021-02-10T04:35:43.790879 2021-02-10T04:35:47.727 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1298 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:47.727 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: cluster 2021-02-10T04:35:44.359332+0000 2021-02-10T04:35:47.727 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: mon.a (mon.0) 1353 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:35:47.728 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: audit 2021-02-10T04:35:44.370544+0000 2021-02-10T04:35:47.728 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: mon.a (mon.0) 1354 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:35:47.728 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: cluster 2021-02-10T04:35:44. 2021-02-10T04:35:47.728 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: 370628+0000 mon.a (mon.0) 1355 : cluster [DBG] osdmap e527: 8 total, 8 up, 8 in 2021-02-10T04:35:47.728 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: cluster 2021-02-10T04:35 2021-02-10T04:35:47.728 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: :44.376280+0000 osd.4 (osd.4) 717 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:35:47.728 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: 2021-02-10T04:35:47.728 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:47.729 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: 02-10T04:35:44.376549+0000 osd.4 (osd.4 2021-02-10T04:35:47.729 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: ) 718 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:35:47.729 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:35:47.729 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: 35:44.982460+0000 osd.4 (osd.4) 719 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:47.729 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: 2021-02-10T04:35:47.729 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:35:47.729 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: 10T04:35:44.982487+0000 osd.4 (osd.4) 720 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:47.729 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: 2021-02-10T04:35:47.729 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: cluster 2021-02-10T04:35:44 2021-02-10T04:35:47.730 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: .982506+0000 osd.4 (osd.4) 721 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:47.730 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:35:47.730 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: T04:35:44.982510+0000 osd.4 (osd. 2021-02-10T04:35:47.730 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:45 smithi085 conmon[35961]: 4) 722 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:47.730 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: cluster 2021-02-10T04:35:47.730 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: 2021-02-10T04:35:47.730 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: 2021 2021-02-10T04:35:47.730 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: - 2021-02-10T04:35:47.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: 02 2021-02-10T04:35:47.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: - 2021-02-10T04:35:47.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: 10 2021-02-10T04:35:47.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: T 2021-02-10T04:35:47.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: 04:35:44.793369 2021-02-10T04:35:47.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1299 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:47.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:47.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: 02-10T04:35:44.793381+0000 osd.2 (osd 2021-02-10T04:35:47.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: .2) 1300 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:47.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:35:47.732 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: -02-10T04:35:44.793389+0000 2021-02-10T04:35:47.732 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: osd.2 (osd.2) 1301 : cluster [WRN] 2021-02-10T04:35:47.732 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:47.732 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: cluster 2021-02-10T04:35:44.793397 2021-02-10T04:35:47.732 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1302 : cluster 2021-02-10T04:35:47.732 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:47.732 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: cluster 2021-02-10T04:35: 2021-02-10T04:35:47.732 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: 44.793403+0000 osd.2 (osd.2) 1303 2021-02-10T04:35:47.732 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:47.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:35:47.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: :35:44.793429+0000 osd.2 (osd. 2021-02-10T04:35:47.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: 2) 1304 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:47.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:35:47.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: 10T04:35:44.793444+0000 osd.2 ( 2021-02-10T04:35:47.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: osd.2) 1305 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:47.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:47.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: 02-10T04:35:44.793448+0000 2021-02-10T04:35:47.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: osd.2 (osd.2) 1306 : cluster [WRN] 2021-02-10T04:35:47.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:47.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: cluster 2021-02-10T04:35:47.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: 2021-02-10T04:35:44.793454+0000 osd.2 ( 2021-02-10T04:35:47.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: osd.2) 1307 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:47.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:47.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: 02-10T04:35:44.793459+0000 2021-02-10T04:35:47.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: osd.2 (osd.2) 1308 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:47.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: 2021-02-10T04:35:47.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: cluster 2021-02-10T04:35:47.735 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: 2021-02-10T04:35:45.376197+0000 mon.a (mon 2021-02-10T04:35:47.735 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: .0) 1356 : cluster [DBG] osdmap e528: 8 total, 8 up, 8 in 2021-02-10T04:35:47.735 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:35:47.735 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: 10T04:35:45.951436+0000 osd.4 ( 2021-02-10T04:35:47.735 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: osd.4) 723 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:47.735 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:47.735 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: 02-10T04:35:45.951456+0000 2021-02-10T04:35:47.735 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: osd.4 (osd.4) 724 : cluster [WRN] 2021-02-10T04:35:47.735 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:47.736 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: cluster 2021-02-10T04:35:47.736 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: 2021-02-10T04:35:45.951460+0000 osd.4 ( 2021-02-10T04:35:47.736 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: osd.4) 725 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:47.736 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:35:47.736 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: -02-10T04:35:45.951463+0000 2021-02-10T04:35:47.736 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: osd.4 (osd.4) 726 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:47.737 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:35:47.737 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: -02-10T04:35:46.106508+0000 2021-02-10T04:35:47.737 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: mgr.y (mgr.14140) 606 : cluster [DBG] 2021-02-10T04:35:47.737 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:46 smithi085 conmon[35961]: pgmap v942: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 396 MiB data, 1023 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.6 MiB/s wr, 17 op/s 2021-02-10T04:35:47.737 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: cluster 2021-02-10T04:35:47.737 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:35:47.737 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: 10T04:35:45.755470+0000 osd.2 (osd.2) 1309 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:47.738 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: cluster 2021-02-10T04:35:47.738 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: 2021-02 2021-02-10T04:35:47.738 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: -10T04:35:45.755478+0000 osd.2 (osd.2) 2021-02-10T04:35:47.738 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: 1310 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:47.739 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:47.739 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: 02-10T04:35:45.755485+0000 osd.2 (osd 2021-02-10T04:35:47.739 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: .2) 1311 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:47.739 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: cluster 2021-02-10T04:35:47.739 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:35:47.739 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: 04:35:45.755492+0000 osd.2 (osd 2021-02-10T04:35:47.739 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: .2) 1312 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:47.739 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: cluster 2021-02-10T04:35:47.740 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: 2021-02-10T04:35:45.755499+0000 osd.2 (osd 2021-02-10T04:35:47.740 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: .2) 1313 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:47.740 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:35:47.740 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: -10T04:35:45 2021-02-10T04:35:47.740 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: .755506+0000 osd.2 (osd.2) 1314 : cluster 2021-02-10T04:35:47.740 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:47.741 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:47.741 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: 02-10T04:35:45.755513+0000 osd.2 (osd. 2021-02-10T04:35:47.741 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: 2) 1315 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:47.741 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:35:47.742 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: -10T04:35:45.755520+0000 osd.2 (osd.2) 1316 2021-02-10T04:35:47.742 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:47.742 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:47.742 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: 02-10T04:35:45.755525+0000 osd.2 (osd 2021-02-10T04:35:47.742 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: .2) 1317 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:47.743 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:35:47.743 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: -10T04:35:45.755531+0000 osd.2 (osd.2) 2021-02-10T04:35:47.743 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: 1318 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:47.743 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:47.743 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: 02-10T04:35:46.372270+0000 mon.a (mon 2021-02-10T04:35:47.743 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: .0) 1357 : cluster [WRN] Health check failed: 10 slow ops, oldest one blocked for 353 sec, osd.2 has slow ops (SLOW_OPS) 2021-02-10T04:35:47.743 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:35:47.744 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: -10T04:35:46.381326+0000 mon.a (mon.0 2021-02-10T04:35:47.744 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: ) 1358 : cluster [DBG] osdmap e529: 8 total, 8 up, 8 in 2021-02-10T04:35:47.744 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:35:47.744 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: -10T04:35:46.967149+0000 osd.4 (osd.4 2021-02-10T04:35:47.744 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: ) 727 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:47.744 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:35:47.744 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: -02-10T04:35:46.967157+0000 2021-02-10T04:35:47.744 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: osd.4 (osd.4) 728 : cluster [WRN] 2021-02-10T04:35:47.744 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:47.745 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:35:47.745 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: 35:46.967165+0000 osd.4 (osd.4 2021-02-10T04:35:47.745 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: ) 729 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:47.745 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:35:47.745 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: -02-10T04:35:46.967170+0000 2021-02-10T04:35:47.746 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: osd.4 (osd.4) 730 : cluster [WRN] 2021-02-10T04:35:47.746 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:47.746 INFO:tasks.rados.rados.0.smithi085.stdout:2040: done (10 left) 2021-02-10T04:35:47.746 INFO:tasks.rados.rados.0.smithi085.stdout:2041: done (9 left) 2021-02-10T04:35:47.746 INFO:tasks.rados.rados.0.smithi085.stdout:2042: write oid 30 current snap is 226 2021-02-10T04:35:47.746 INFO:tasks.rados.rados.0.smithi085.stdout:2042: seq_num 718 ranges {467426=565964,1649130=418296} 2021-02-10T04:35:47.747 INFO:tasks.rados.rados.0.smithi085.stdout:2042: writing smithi08575397-30 from 467426 to 1033390 tid 1 2021-02-10T04:35:47.747 INFO:tasks.rados.rados.0.smithi085.stdout:2042: writing smithi08575397-30 from 1649130 to 2067426 tid 2 2021-02-10T04:35:47.747 INFO:tasks.rados.rados.0.smithi085.stdout:2042: finishing write tid 1 to smithi08575397-30 2021-02-10T04:35:47.749 INFO:tasks.rados.rados.0.smithi085.stdout:2043: read oid 5 snap -1 2021-02-10T04:35:47.749 INFO:tasks.rados.rados.0.smithi085.stdout:2043: expect (ObjNum 708 snap 222 seq_num 708) 2021-02-10T04:35:47.749 INFO:tasks.rados.rados.0.smithi085.stdout:2044: write oid 1 current snap is 226 2021-02-10T04:35:47.750 INFO:tasks.rados.rados.0.smithi085.stdout:2044: seq_num 719 ranges {484233=641213,1707612=376621} 2021-02-10T04:35:47.750 INFO:tasks.rados.rados.0.smithi085.stdout:2044: writing smithi08575397-1 from 484233 to 1125446 tid 1 2021-02-10T04:35:47.750 INFO:tasks.rados.rados.0.smithi085.stdout:2044: writing smithi08575397-1 from 1707612 to 2084233 tid 2 2021-02-10T04:35:47.750 INFO:tasks.rados.rados.0.smithi085.stdout:2045: rollback oid 16 current snap is 226 2021-02-10T04:35:47.750 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 16 to 224 2021-02-10T04:35:47.750 INFO:tasks.rados.rados.0.smithi085.stdout:2042: finishing write tid 2 to smithi08575397-30 2021-02-10T04:35:47.751 INFO:tasks.rados.rados.0.smithi085.stdout:2042: finishing write tid 3 to smithi08575397-30 2021-02-10T04:35:47.751 INFO:tasks.rados.rados.0.smithi085.stdout:2042: finishing write tid 4 to smithi08575397-30 2021-02-10T04:35:47.751 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 1034 (ObjNum 718 snap 226 seq_num 718) dirty exists 2021-02-10T04:35:47.751 INFO:tasks.rados.rados.0.smithi085.stdout:2042: left oid 30 (ObjNum 718 snap 226 seq_num 718) 2021-02-10T04:35:47.751 INFO:tasks.rados.rados.0.smithi085.stdout:2042: done (12 left) 2021-02-10T04:35:47.751 INFO:tasks.rados.rados.0.smithi085.stdout:2046: read oid 2 snap -1 2021-02-10T04:35:47.751 INFO:tasks.rados.rados.0.smithi085.stdout:2046: expect (ObjNum 652 snap 197 seq_num 652) 2021-02-10T04:35:47.751 INFO:tasks.rados.rados.0.smithi085.stdout:2047: read oid 10 snap 222 2021-02-10T04:35:47.752 INFO:tasks.rados.rados.0.smithi085.stdout:2047: expect deleted 2021-02-10T04:35:47.752 INFO:tasks.rados.rados.0.smithi085.stdout:2048: snap_remove snap 205 2021-02-10T04:35:47.752 INFO:tasks.rados.rados.0.smithi085.stdout:2043: expect (ObjNum 708 snap 222 seq_num 708) 2021-02-10T04:35:47.752 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:45 smithi164 conmon[33677]: debug 2021-02-10T04:35:45.949+0000 7fbec19a1700 -1 osd.4 528 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:35:47.752 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:46 smithi164 conmon[33677]: debug 2021-02-10T04:35:46.965+0000 7fbec19a1700 -1 osd.4 529 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:35:47.752 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:35:47.753 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [ 2021-02-10T04:35:47.753 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:47.753 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156)", 2021-02-10T04:35:47.753 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:32.164063+0000", 2021-02-10T04:35:47.753 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 373.82670554100002, 2021-02-10T04:35:47.753 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 373.82674483099998, 2021-02-10T04:35:47.753 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:35:47.754 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:35:47.754 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:35:47.754 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:35:47.754 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:35:47.754 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 910 2021-02-10T04:35:47.754 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:47.755 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:35:47.755 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:47.755 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:35:47.755 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164063+0000", 2021-02-10T04:35:47.755 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:35:47.755 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:47.755 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:47.756 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:35:47.756 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164063+0000", 2021-02-10T04:35:47.756 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.8188e-05 2021-02-10T04:35:47.756 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:47.756 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:47.756 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:35:47.756 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164111+0000", 2021-02-10T04:35:47.756 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0791000000000001e-05 2021-02-10T04:35:47.757 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:47.757 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:47.757 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:35:47.757 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164122+0000", 2021-02-10T04:35:47.757 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5682999999999999e-05 2021-02-10T04:35:47.757 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:47.757 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:47.757 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:35:47.758 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164138+0000", 2021-02-10T04:35:47.758 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0522e-05 2021-02-10T04:35:47.758 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:47.758 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:47.758 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:35:47.758 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164148+0000", 2021-02-10T04:35:47.758 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.9583999999999997e-05 2021-02-10T04:35:47.758 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:47.759 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:47.759 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:35:47.759 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164208+0000", 2021-02-10T04:35:47.759 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00013592500000000001 2021-02-10T04:35:47.759 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:47.759 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:47.759 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:35:47.759 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164344+0000", 2021-02-10T04:35:47.760 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00028069299999999999 2021-02-10T04:35:47.760 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:35:47.760 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:35:47.760 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:35:47.760 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:47.760 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:47.760 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156)", 2021-02-10T04:35:47.761 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:32.164227+0000", 2021-02-10T04:35:47.761 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 373.82654110999999, 2021-02-10T04:35:47.761 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 373.82667669800003, 2021-02-10T04:35:47.761 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:35:47.761 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "delayed", 2021-02-10T04:35:47.761 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:35:47.761 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:35:47.762 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:35:47.762 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 911 2021-02-10T04:35:47.762 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:47.762 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:35:47.762 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:47.762 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:35:47.763 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164227+0000", 2021-02-10T04:35:47.763 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:35:47.763 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:47.763 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:47.763 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:35:47.763 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164227+0000", 2021-02-10T04:35:47.763 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.3152999999999997e-05 2021-02-10T04:35:47.764 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:47.764 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:47.764 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:35:47.764 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164291+0000", 2021-02-10T04:35:47.764 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.8859999999999993e-06 2021-02-10T04:35:47.764 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:47.765 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:47.765 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:35:47.765 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164299+0000", 2021-02-10T04:35:47.765 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5695000000000001e-05 2021-02-10T04:35:47.765 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:47.765 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:47.765 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:35:47.765 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164315+0000", 2021-02-10T04:35:47.766 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.3790000000000003e-06 2021-02-10T04:35:47.766 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:47.766 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:47.766 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:35:47.766 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164322+0000", 2021-02-10T04:35:47.766 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00017179399999999999 2021-02-10T04:35:47.766 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:47.766 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:47.767 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:35:47.767 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164493+0000", 2021-02-10T04:35:47.767 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.9685000000000002e-05 2021-02-10T04:35:47.767 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:47.767 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:47.767 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:35:47.767 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164553+0000", 2021-02-10T04:35:47.767 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00032559199999999998 2021-02-10T04:35:47.768 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:35:47.775 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:35:47.775 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:35:47.775 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:47.775 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:47.776 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153)", 2021-02-10T04:35:47.776 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:29.061753+0000", 2021-02-10T04:35:47.776 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 376.92901547100001, 2021-02-10T04:35:47.776 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 376.92923363099999, 2021-02-10T04:35:47.776 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:35:47.776 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:35:47.776 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:35:47.776 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:35:47.777 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:35:47.777 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 865 2021-02-10T04:35:47.777 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:47.777 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:35:47.777 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:47.777 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:35:47.777 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061753+0000", 2021-02-10T04:35:47.777 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:35:47.778 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:47.778 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:47.778 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:35:47.778 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061753+0000", 2021-02-10T04:35:47.778 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.3646999999999997e-05 2021-02-10T04:35:47.778 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:47.778 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:47.779 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:35:49.856 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061797+0000", 2021-02-10T04:35:49.856 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.4220000000000003e-06 2021-02-10T04:35:49.856 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:49.856 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:49.856 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:35:49.856 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061804+0000", 2021-02-10T04:35:49.857 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.9582000000000002e-05 2021-02-10T04:35:49.857 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:49.857 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:49.857 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:35:49.857 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061844+0000", 2021-02-10T04:35:49.857 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.9789999999999996e-06 2021-02-10T04:35:49.857 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:49.857 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:49.857 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:35:49.857 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061851+0000", 2021-02-10T04:35:49.857 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00012051 2021-02-10T04:35:49.857 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:49.857 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:49.858 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:35:49.858 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061971+0000", 2021-02-10T04:35:49.858 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.0452999999999998e-05 2021-02-10T04:35:49.858 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:49.858 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:49.858 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:35:49.858 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.062042+0000", 2021-02-10T04:35:49.858 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00028859300000000002 2021-02-10T04:35:49.858 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:35:49.858 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:35:49.858 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:35:49.858 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:49.858 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:49.859 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153)", 2021-02-10T04:35:49.859 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:29.061891+0000", 2021-02-10T04:35:49.859 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 376.92887772900002, 2021-02-10T04:35:49.859 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 376.92916971900001, 2021-02-10T04:35:49.859 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:35:49.859 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "delayed", 2021-02-10T04:35:49.859 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:35:49.859 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:35:49.859 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:35:49.859 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 866 2021-02-10T04:35:49.859 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:49.859 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:35:49.859 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:49.860 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:35:49.860 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061891+0000", 2021-02-10T04:35:49.860 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:35:49.860 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:49.860 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:49.860 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:35:49.860 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061891+0000", 2021-02-10T04:35:49.860 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1696000000000003e-05 2021-02-10T04:35:49.860 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:49.860 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:49.860 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:35:49.860 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061923+0000", 2021-02-10T04:35:49.860 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.7699999999999998e-06 2021-02-10T04:35:49.861 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:49.861 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:49.861 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:35:49.861 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061928+0000", 2021-02-10T04:35:49.861 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1918e-05 2021-02-10T04:35:49.861 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:49.861 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:49.861 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:35:49.861 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061940+0000", 2021-02-10T04:35:49.861 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.8640000000000004e-06 2021-02-10T04:35:49.861 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:49.861 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:49.861 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:35:49.861 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061945+0000", 2021-02-10T04:35:49.862 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00022406100000000001 2021-02-10T04:35:49.862 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:49.862 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:49.862 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:35:49.862 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.062169+0000", 2021-02-10T04:35:49.862 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.0158999999999999e-05 2021-02-10T04:35:49.862 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:49.862 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:49.862 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:35:49.862 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.062219+0000", 2021-02-10T04:35:49.862 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000328468 2021-02-10T04:35:49.862 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:35:49.862 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:35:49.863 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:35:49.863 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:35:49.863 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2021-02-10T04:35:49.863 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 4 2021-02-10T04:35:49.863 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:35:49.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:46 smithi164 conmon[29288]: cluster 2021-02-10T04:35:44.793369+0000 osd.2 (osd.2) 1299 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:49.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:46 smithi164 conmon[29288]: cluster 2021-02-10T04:35:44.793381+0000 osd.2 (osd.2) 1300 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:49.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:46 smithi164 conmon[29288]: cluster 2021-02-10T04:35:44.793389+0000 osd.2 (osd.2) 1301 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:49.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:46 smithi164 conmon[29288]: cluster 2021-02-10T04:35:44.793397+0000 osd.2 (osd.2) 1302 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:49.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:46 smithi164 conmon[29288]: cluster 2021-02-10T04:35:44.793403+0000 osd.2 (osd.2) 1303 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:49.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:46 smithi164 conmon[29288]: cluster 2021-02-10T04:35:44.793429+0000 osd.2 (osd.2) 1304 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:49.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:46 smithi164 conmon[29288]: cluster 2021-02-10T04:35:44.793444+0000 osd.2 (osd.2) 1305 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:49.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:46 smithi164 conmon[29288]: cluster 2021-02-10T04:35:44.793448+0000 osd.2 (osd.2) 1306 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:49.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:46 smithi164 conmon[29288]: cluster 2021-02-10T04:35:44.793454+0000 osd.2 (osd.2) 1307 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:49.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:46 smithi164 conmon[29288]: cluster 2021-02-10T04:35:44.793459+0000 osd.2 (osd. 2021-02-10T04:35:49.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:46 smithi164 conmon[29288]: 2) 1308 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:49.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:46 smithi164 conmon[29288]: cluster 2021-02-10T04:35:45.376197+0000 mon.a (mon.0) 1356 : cluster [DBG] osdmap e528: 8 total, 8 up, 8 in 2021-02-10T04:35:49.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:46 smithi164 conmon[29288]: cluster 2021-02-10T04:35:45.951436+0000 osd.4 (osd.4) 723 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:49.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:46 smithi164 conmon[29288]: cluster 2021-02-10T04:35:45.951456+0000 osd.4 (osd.4) 724 : 2021-02-10T04:35:49.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:46 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:49.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:46 smithi164 conmon[29288]: cluster 2021-02-10T04:35:45.951460 2021-02-10T04:35:49.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:46 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 725 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:49.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:46 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:35:49.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:46 smithi164 conmon[29288]: T04:35:45.951463+0000 osd.4 (osd. 2021-02-10T04:35:49.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:46 smithi164 conmon[29288]: 4) 726 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:49.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:46 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:35:49.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:46 smithi164 conmon[29288]: T04:35:46.106508+0000 mgr.y (mgr. 2021-02-10T04:35:49.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:46 smithi164 conmon[29288]: 14140) 606 : cluster [DBG] pgmap v942: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 396 MiB data, 1023 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.6 MiB/s wr, 17 op/s 2021-02-10T04:35:49.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: cluster 2021-02-10T04:35:45.755470 2021-02-10T04:35:49.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1309 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:49.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:35:49.866 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: 10T04:35:45.755478+0000 osd.2 (osd.2) 1310 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:49.866 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:35:49.866 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: 04:35:45.755485+0000 osd.2 (osd.2) 1311 : 2021-02-10T04:35:49.866 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:49.866 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: cluster 2021-02-10T04:35:45.755492+0000 osd.2 (osd.2) 1312 : cluster 2021-02-10T04:35:49.866 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:49.866 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:35:49.866 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: -02-10T04:35:45.755499+0000 2021-02-10T04:35:49.866 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: osd.2 (osd.2) 1313 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:49.866 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: cluster 2021-02-10T04:35:45.755506+0000 osd.2 (osd.2) 1314 : 2021-02-10T04:35:49.866 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:49.866 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: cluster 2021-02-10T04:35:45.755513 2021-02-10T04:35:49.866 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1315 : cluster [WRN] 2021-02-10T04:35:49.867 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:49.867 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:49.867 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: -10T04:35:45.755520+0000 osd.2 (osd.2) 1316 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:49.867 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:35:49.867 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: 04:35:45.755525+0000 osd.2 (osd.2) 2021-02-10T04:35:49.867 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: 1317 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:49.867 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:35:49.867 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: -02-10T04:35:45.755531+0000 osd.2 (osd.2) 1318 : cluster 2021-02-10T04:35:49.867 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:49.867 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: cluster 2021-02-10T04:35:49.867 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: 2021-02-10T04:35:46.372270+0000 2021-02-10T04:35:49.867 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: mon.a (mon.0) 1357 : cluster 2021-02-10T04:35:49.868 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: [WRN] Health check failed: 10 slow ops, oldest one blocked for 353 sec, osd.2 has slow ops (SLOW_OPS) 2021-02-10T04:35:49.868 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:35:49.868 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: :35:46.381326+0000 mon.a (mon.0) 1358 : cluster [DBG] osdmap e529: 8 total, 8 up, 8 in 2021-02-10T04:35:49.868 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: cluster 2021-02-10T04:35:46. 2021-02-10T04:35:49.868 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: 967149+0000 osd.4 (osd.4) 727 : cluster [WRN] 2021-02-10T04:35:49.868 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:49.868 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: cluster 2021-02-10T04:35:46.967157 2021-02-10T04:35:49.868 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 728 : cluster [WRN] 2021-02-10T04:35:49.868 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:49.868 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: cluster 2021-02-10T04:35:49.868 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: 2021-02-10T04:35:46.967165+0000 osd.4 (osd. 2021-02-10T04:35:49.868 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: 4) 729 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:49.868 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:35:49.869 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: 04:35:46.967170+0000 osd.4 (osd.4) 2021-02-10T04:35:49.869 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[29288]: 730 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:49.875 INFO:tasks.rados.rados.0.smithi085.stdout:2046: expect (ObjNum 652 snap 197 seq_num 652) 2021-02-10T04:35:49.875 INFO:tasks.rados.rados.0.smithi085.stdout:2044: finishing write tid 1 to smithi08575397-1 2021-02-10T04:35:49.875 INFO:tasks.rados.rados.0.smithi085.stdout:2044: finishing write tid 2 to smithi08575397-1 2021-02-10T04:35:49.875 INFO:tasks.rados.rados.0.smithi085.stdout:2043: done (14 left) 2021-02-10T04:35:49.875 INFO:tasks.rados.rados.0.smithi085.stdout:2046: done (13 left) 2021-02-10T04:35:49.875 INFO:tasks.rados.rados.0.smithi085.stdout:2048: done (12 left) 2021-02-10T04:35:49.875 INFO:tasks.rados.rados.0.smithi085.stdout:2049: write (excl) oid 36 current snap is 226 2021-02-10T04:35:49.875 INFO:tasks.rados.rados.0.smithi085.stdout:2045: finishing rollback tid 0 to smithi08575397-16 2021-02-10T04:35:49.875 INFO:tasks.rados.rados.0.smithi085.stdout:2044: finishing write tid 3 to smithi08575397-1 2021-02-10T04:35:49.875 INFO:tasks.rados.rados.0.smithi085.stdout:2045: finishing rollback tid 1 to smithi08575397-16 2021-02-10T04:35:49.875 INFO:tasks.rados.rados.0.smithi085.stdout:2044: finishing write tid 4 to smithi08575397-1 2021-02-10T04:35:49.875 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 768 (ObjNum 719 snap 226 seq_num 719) dirty exists 2021-02-10T04:35:49.876 INFO:tasks.rados.rados.0.smithi085.stdout:2044: left oid 1 (ObjNum 719 snap 226 seq_num 719) 2021-02-10T04:35:49.876 INFO:tasks.rados.rados.0.smithi085.stdout:2045: finishing rollback tid 2 to smithi08575397-16 2021-02-10T04:35:49.876 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 1100 (ObjNum 671 snap 207 seq_num 671) dirty exists 2021-02-10T04:35:49.876 INFO:tasks.rados.rados.0.smithi085.stdout:2049: seq_num 720 ranges {501040=716462,1766094=334946} 2021-02-10T04:35:49.876 INFO:tasks.rados.rados.0.smithi085.stdout:2049: writing smithi08575397-36 from 501040 to 1217502 tid 1 2021-02-10T04:35:49.876 INFO:tasks.rados.rados.0.smithi085.stdout:2049: writing smithi08575397-36 from 1766094 to 2101040 tid 2 2021-02-10T04:35:49.876 INFO:tasks.rados.rados.0.smithi085.stdout:2044: done (12 left) 2021-02-10T04:35:49.876 INFO:tasks.rados.rados.0.smithi085.stdout:2045: done (11 left) 2021-02-10T04:35:49.876 INFO:tasks.rados.rados.0.smithi085.stdout:2047: done (10 left) 2021-02-10T04:35:49.876 INFO:tasks.rados.rados.0.smithi085.stdout:2050: write (excl) oid 26 current snap is 226 2021-02-10T04:35:49.876 INFO:tasks.rados.rados.0.smithi085.stdout:2050: seq_num 721 ranges {517847=791711,2108223=9624} 2021-02-10T04:35:49.876 INFO:tasks.rados.rados.0.smithi085.stdout:2050: writing smithi08575397-26 from 517847 to 1309558 tid 1 2021-02-10T04:35:49.876 INFO:tasks.rados.rados.0.smithi085.stdout:2050: writing smithi08575397-26 from 2108223 to 2117847 tid 2 2021-02-10T04:35:49.876 INFO:tasks.rados.rados.0.smithi085.stdout:2051: read oid 2 snap -1 2021-02-10T04:35:49.877 INFO:tasks.rados.rados.0.smithi085.stdout:2051: expect (ObjNum 652 snap 197 seq_num 652) 2021-02-10T04:35:49.877 INFO:tasks.rados.rados.0.smithi085.stdout:2052: copy_from oid 44 from oid 21 current snap is 226 2021-02-10T04:35:49.877 INFO:tasks.rados.rados.0.smithi085.stdout:2053: copy_from oid 50 from oid 43 current snap is 226 2021-02-10T04:35:49.877 INFO:tasks.rados.rados.0.smithi085.stdout:2049: finishing write tid 1 to smithi08575397-36 2021-02-10T04:35:49.877 INFO:tasks.rados.rados.0.smithi085.stdout:2049: finishing write tid 2 to smithi08575397-36 2021-02-10T04:35:49.877 INFO:tasks.rados.rados.0.smithi085.stdout:2049: finishing write tid 3 to smithi08575397-36 2021-02-10T04:35:49.877 INFO:tasks.rados.rados.0.smithi085.stdout:2049: finishing write tid 4 to smithi08575397-36 2021-02-10T04:35:49.877 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 677 (ObjNum 720 snap 226 seq_num 720) dirty exists 2021-02-10T04:35:49.877 INFO:tasks.rados.rados.0.smithi085.stdout:2049: left oid 36 (ObjNum 720 snap 226 seq_num 720) 2021-02-10T04:35:49.877 INFO:tasks.rados.rados.0.smithi085.stdout:2050: finishing write tid 1 to smithi08575397-26 2021-02-10T04:35:49.877 INFO:tasks.rados.rados.0.smithi085.stdout:2049: done (13 left) 2021-02-10T04:35:49.877 INFO:tasks.rados.rados.0.smithi085.stdout:2054: read oid 28 snap -1 2021-02-10T04:35:49.877 INFO:tasks.rados.rados.0.smithi085.stdout:2054: expect (ObjNum 651 snap 197 seq_num 651) 2021-02-10T04:35:49.877 INFO:tasks.rados.rados.0.smithi085.stdout:2055: read oid 42 snap -1 2021-02-10T04:35:49.878 INFO:tasks.rados.rados.0.smithi085.stdout:2055: expect (ObjNum 646 snap 196 seq_num 646) 2021-02-10T04:35:49.878 INFO:tasks.rados.rados.0.smithi085.stdout:2056: write (excl) oid 23 current snap is 226 2021-02-10T04:35:49.878 INFO:tasks.rados.rados.0.smithi085.stdout:2056: seq_num 722 ranges {534654=466960,1766705=367949} 2021-02-10T04:35:49.878 INFO:tasks.rados.rados.0.smithi085.stdout:2056: writing smithi08575397-23 from 534654 to 1001614 tid 1 2021-02-10T04:35:49.878 INFO:tasks.rados.rados.0.smithi085.stdout:2056: writing smithi08575397-23 from 1766705 to 2134654 tid 2 2021-02-10T04:35:49.878 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:35:49.878 INFO:tasks.rados.rados.0.smithi085.stdout:2051: expect (ObjNum 652 snap 197 seq_num 652) 2021-02-10T04:35:49.878 INFO:tasks.rados.rados.0.smithi085.stdout:2050: finishing write tid 2 to smithi08575397-26 2021-02-10T04:35:49.878 INFO:tasks.rados.rados.0.smithi085.stdout:2050: finishing write tid 3 to smithi08575397-26 2021-02-10T04:35:49.878 INFO:tasks.rados.rados.0.smithi085.stdout:2050: finishing write tid 4 to smithi08575397-26 2021-02-10T04:35:49.878 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 654 (ObjNum 721 snap 226 seq_num 721) dirty exists 2021-02-10T04:35:49.878 INFO:tasks.rados.rados.0.smithi085.stdout:2050: left oid 26 (ObjNum 721 snap 226 seq_num 721) 2021-02-10T04:35:49.878 INFO:tasks.rados.rados.0.smithi085.stdout:2054: expect (ObjNum 651 snap 197 seq_num 651) 2021-02-10T04:35:49.879 INFO:tasks.rados.rados.0.smithi085.stdout:2055: expect (ObjNum 646 snap 196 seq_num 646) 2021-02-10T04:35:49.879 INFO:tasks.rados.rados.0.smithi085.stdout:2056: finishing write tid 1 to smithi08575397-23 2021-02-10T04:35:49.879 INFO:tasks.rados.rados.0.smithi085.stdout:2050: done (15 left) 2021-02-10T04:35:49.879 INFO:tasks.rados.rados.0.smithi085.stdout:2051: done (14 left) 2021-02-10T04:35:49.879 INFO:tasks.rados.rados.0.smithi085.stdout:2054: done (13 left) 2021-02-10T04:35:49.879 INFO:tasks.rados.rados.0.smithi085.stdout:2055: done (12 left) 2021-02-10T04:35:49.879 INFO:tasks.rados.rados.0.smithi085.stdout:2057: snap_create 2021-02-10T04:35:49.879 INFO:tasks.rados.rados.0.smithi085.stdout:2056: finishing write tid 2 to smithi08575397-23 2021-02-10T04:35:49.879 INFO:tasks.rados.rados.0.smithi085.stdout:2056: finishing write tid 3 to smithi08575397-23 2021-02-10T04:35:49.879 INFO:tasks.rados.rados.0.smithi085.stdout:2056: finishing write tid 4 to smithi08575397-23 2021-02-10T04:35:49.879 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 680 (ObjNum 722 snap 226 seq_num 722) dirty exists 2021-02-10T04:35:49.879 INFO:tasks.rados.rados.0.smithi085.stdout:2056: left oid 23 (ObjNum 722 snap 226 seq_num 722) 2021-02-10T04:35:49.879 INFO:tasks.rados.rados.0.smithi085.stdout:2052: finishing copy_from to smithi08575397-44 2021-02-10T04:35:49.879 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 1076 (ObjNum 431 snap 114 seq_num 431) dirty exists 2021-02-10T04:35:49.880 INFO:tasks.rados.rados.0.smithi085.stdout:2052: finishing copy_from racing read to smithi08575397-44 2021-02-10T04:35:49.880 INFO:tasks.rados.rados.0.smithi085.stdout:2053: finishing copy_from to smithi08575397-50 2021-02-10T04:35:49.880 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 727 (ObjNum 670 snap 205 seq_num 670) dirty exists 2021-02-10T04:35:49.880 INFO:tasks.rados.rados.0.smithi085.stdout:2053: finishing copy_from racing read to smithi08575397-50 2021-02-10T04:35:49.880 INFO:tasks.rados.rados.0.smithi085.stdout:2052: done (12 left) 2021-02-10T04:35:49.880 INFO:tasks.rados.rados.0.smithi085.stdout:2053: done (11 left) 2021-02-10T04:35:49.880 INFO:tasks.rados.rados.0.smithi085.stdout:2056: done (10 left) 2021-02-10T04:35:49.880 INFO:tasks.rados.rados.0.smithi085.stdout:2057: done (9 left) 2021-02-10T04:35:49.880 INFO:tasks.rados.rados.0.smithi085.stdout:2058: copy_from oid 1 from oid 16 current snap is 227 2021-02-10T04:35:49.880 INFO:tasks.rados.rados.0.smithi085.stdout:2059: snap_create 2021-02-10T04:35:49.880 INFO:tasks.rados.rados.0.smithi085.stdout:2058: finishing copy_from to smithi08575397-1 2021-02-10T04:35:49.880 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 1100 (ObjNum 671 snap 207 seq_num 671) dirty exists 2021-02-10T04:35:49.880 INFO:tasks.rados.rados.0.smithi085.stdout:2058: finishing copy_from racing read to smithi08575397-1 2021-02-10T04:35:49.881 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[49106]: debug 2021-02-10T04:35:49.881 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[49106]: 2021-02-10T04:35:47.770+0000 7f77a3e94700 -1 osd.2 530 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:35:49.881 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:47 smithi085 conmon[49106]: 2021-02-10T04:35:49.881 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[49106]: debug 2021-02-10T04:35:48.777+0000 7f77a3e94700 -1 osd.2 531 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:35:49.881 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: cluster 2021-02-10T04:35:46.766544+0000 osd.2 (osd.2) 1319 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:49.881 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: cluster 2021-02-10T04:35:46.766551+0000 osd.2 (osd.2) 1320 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:49.881 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: cluster 2021-02-10T04:35:46.766555+0000 osd.2 (osd.2) 1321 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:49.881 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: cluster 2021-02-10T04:35:46.766558+0000 osd.2 (osd.2) 1322 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:49.881 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: cluster 2021-02-10T04:35:46.766563+0000 osd.2 (osd.2) 1323 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:49.881 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: cluster 2021-02-10T04:35:46.766576+0000 osd.2 (osd.2) 1324 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:49.881 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: cluster 2021-02-10T04:35:46.766582+0000 osd.2 (osd.2) 1325 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:49.882 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: cluster 2021-02-10T04:35:46.766589+0000 osd.2 (osd.2) 1326 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:49.882 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: cluster 2021-02-10T04:35:46.766596+0000 osd.2 (osd.2) 1327 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:49.882 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:35:49.882 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: 10T04:35:46.766602+0000 osd.2 (osd.2) 1328 : cluster [WRN] 2021-02-10T04:35:49.882 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:49.882 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: cluster 2021-02-10T04:35:47 2021-02-10T04:35:49.882 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: .389174+0000 mon.a (mon.0) 1359 : 2021-02-10T04:35:49.882 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: cluster [DBG] osdmap e530: 8 total, 8 up, 8 in 2021-02-10T04:35:49.882 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:35:49.882 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: T04:35:47.990866+0000 osd.4 (osd.4) 731 : 2021-02-10T04:35:49.882 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:49.882 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:35:49.883 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: -02-10T04:35:47.990875+0000 osd.4 (osd.4) 732 : cluster 2021-02-10T04:35:49.883 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:49.883 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:35:49.883 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: 04:35:47.990898+0000 osd.4 (osd.4) 733 : cluster 2021-02-10T04:35:49.883 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:49.883 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: cluster 2021-02-10T04:35: 2021-02-10T04:35:49.883 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: 47.990902+0000 osd.4 (osd.4) 734 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:49.883 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: cluster 2021-02-10T04:35:48. 2021-02-10T04:35:49.883 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: 106948+0000 mgr.y (mgr.14140) 607 : cluster [DBG] pgmap v945: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 396 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.8 MiB/s rd, 1.1 MiB/s wr, 8 op/s 2021-02-10T04:35:49.883 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: audit 2021-02-10T04:35: 2021-02-10T04:35:49.883 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: 48.107078+0000 mgr.y (mgr.14140) 608 : audit 2021-02-10T04:35:49.883 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:35:49.883 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: cluster 2021-02-10T04:35:48.168863 2021-02-10T04:35:49.883 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1360 : cluster [DBG] osdmap e531: 8 total, 8 up, 8 in 2021-02-10T04:35:49.884 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: cluster 2021-02-10T04:35:47.770996+0000 osd.2 (osd 2021-02-10T04:35:49.884 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: .2) 1329 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:49.884 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: cluster 2021-02-10T04:35:47.771004+0000 osd.2 (osd 2021-02-10T04:35:49.884 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: .2) 1330 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:49.884 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:35:49.884 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: -02-10T04:35:47.771010+0000 osd.2 (osd.2) 1331 2021-02-10T04:35:49.884 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:49.884 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:35:49.884 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: 35:47.771015+0000 osd.2 (osd. 2021-02-10T04:35:49.884 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: 2) 1332 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:49.884 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:35:49.884 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: -10T04:35:47.771022+0000 osd.2 ( 2021-02-10T04:35:49.885 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: osd.2) 1333 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:49.885 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:49.885 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: 02-10T04:35:47.771027+0000 osd.2 2021-02-10T04:35:49.885 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: (osd.2) 1334 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:49.885 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: 2021-02-10T04:35:49.885 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: cluster 2021-02-10T04:35:49.885 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: 2021-02-10T04:35:47.771033+0000 osd.2 (osd 2021-02-10T04:35:49.885 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: .2) 1335 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:49.885 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:49.885 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: 02-10T04:35:47.771038+0000 osd.2 2021-02-10T04:35:49.885 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: (osd.2) 1336 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:49.885 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: 2021-02-10T04:35:49.885 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: cluster 2021-02-10T04:35:49.885 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: 2021-02-10T04:35:47.771044+0000 osd.2 (osd 2021-02-10T04:35:49.886 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: .2) 1337 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:49.886 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:35:49.886 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: -10T04:35:47.771049+0000 osd.2 ( 2021-02-10T04:35:49.886 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: osd.2) 1338 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:49.886 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:35:49.886 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: -02-10T04:35:49.030169+0000 2021-02-10T04:35:49.886 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: osd.4 (osd.4) 735 : cluster [WRN] 2021-02-10T04:35:49.886 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:49.886 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: cluster 2021-02-10T04:35:49.886 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: 2021-02-10T04:35:49.030178+0000 osd.4 (osd 2021-02-10T04:35:49.886 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: .4) 736 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:49.886 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:35:49.886 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: -10T04:35:49.030188+0000 osd.4 ( 2021-02-10T04:35:49.887 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: osd.4) 737 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:49.887 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:49.887 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: 02-10T04:35:49.030213+0000 2021-02-10T04:35:49.887 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: osd.4 (osd.4) 738 : cluster [WRN] 2021-02-10T04:35:49.887 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:49.887 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: cluster 2021-02-10T04:35:46.766544+0000 osd.2 (osd. 2021-02-10T04:35:49.887 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: 2) 1319 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:49.887 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: cluster 2021-02-10T04:35:49.888 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: 2021-02-10T04:35:46.766551+0000 osd.2 (osd.2 2021-02-10T04:35:49.888 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: ) 1320 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:49.888 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: cluster 2021-02-10T04:35:49.888 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:35:49.888 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: 04:35:46.766555+0000 osd.2 2021-02-10T04:35:49.888 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: (osd.2) 1321 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:49.888 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: 2021-02-10T04:35:49.888 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:35:49.888 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: -10T04:35:46.766558+0000 2021-02-10T04:35:49.888 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: osd.2 (osd.2) 1322 : cluster [WRN] 2021-02-10T04:35:49.888 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:49.888 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:35:49.888 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: :35:46.766563+0000 osd.2 (osd.2 2021-02-10T04:35:49.888 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: ) 1323 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:49.889 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: cluster 2021-02-10T04:35:49.889 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:35:49.889 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: 35:46.766576+0000 osd.2 (osd 2021-02-10T04:35:49.889 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: .2) 1324 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:49.889 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: 2021-02-10T04:35:49.889 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:35:49.889 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: 10T04:35:46.766582+0000 osd.2 ( 2021-02-10T04:35:49.889 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: osd.2) 1325 : cluster [WRN] 2021-02-10T04:35:49.889 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:49.889 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: cluster 2021-02-10T04:35:49.889 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: 2021-02-10T04:35:46 2021-02-10T04:35:49.889 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: .766589+0000 osd.2 (osd.2) 2021-02-10T04:35:49.889 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: 1326 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:49.889 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:35:49.890 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: -02-10T04:35:46.766596+0000 osd.2 (osd.2 2021-02-10T04:35:49.890 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: ) 1327 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:49.890 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:49.890 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: 02-10T04:35:46.766602+0000 osd.2 ( 2021-02-10T04:35:49.890 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: osd.2) 1328 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:49.890 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:35:49.890 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: -02-10T04:35:47.389174+0000 2021-02-10T04:35:49.890 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: mon.a (mon.0) 1359 : cluster [DBG] 2021-02-10T04:35:49.890 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: osdmap e530: 8 total, 8 up, 8 in 2021-02-10T04:35:49.890 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:35:49.891 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: 35:47.990866+0000 osd.4 (osd.4 2021-02-10T04:35:49.891 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: ) 731 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:49.891 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:35:49.891 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: -02-10T04:35:47.990875+0000 2021-02-10T04:35:49.891 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: osd.4 (osd.4) 732 : cluster [WRN] 2021-02-10T04:35:49.891 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:49.891 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:49.891 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: 02-10T04:35:47.990898+0000 2021-02-10T04:35:49.891 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: osd.4 (osd.4) 733 : cluster [WRN] 2021-02-10T04:35:49.891 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:49.891 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: cluster 2021-02-10T04:35: 2021-02-10T04:35:49.891 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: 47.990902+0000 osd.4 (osd. 2021-02-10T04:35:49.891 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: 4) 734 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:49.892 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:35:49.892 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: -02-10T04:35:48.106948+0000 2021-02-10T04:35:49.892 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: mgr.y (mgr.14140) 607 : cluster [DBG] 2021-02-10T04:35:49.892 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: pgmap v945: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 396 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.8 MiB/s rd, 1.1 MiB/s wr, 8 op/s 2021-02-10T04:35:49.892 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:35:49.892 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: T04:35:48.107078+0000 mgr.y (mgr 2021-02-10T04:35:49.892 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: .14140) 608 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:35:49.892 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: cluster 2021-02-10T04:35:49.892 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: 2021-02 2021-02-10T04:35:49.892 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: -10T04:35:48.168863 2021-02-10T04:35:49.892 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1360 : cluster [DBG] 2021-02-10T04:35:49.892 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:48 smithi085 conmon[35961]: osdmap e531: 8 total, 8 up, 8 in 2021-02-10T04:35:49.892 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: cluster 2021-02-10T04:35:47.770996+0000 2021-02-10T04:35:49.892 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: osd.2 (osd.2) 1329 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:49.893 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: cluster 2021-02-10T04:35:47. 2021-02-10T04:35:49.893 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: 771004+0000 osd.2 (osd.2) 1330 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:49.893 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: cluster 2021-02-10T04:35: 2021-02-10T04:35:49.893 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: 47.771010+0000 osd.2 (osd.2) 1331 : 2021-02-10T04:35:49.893 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:49.893 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: cluster 2021-02-10T04:35 2021-02-10T04:35:49.893 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: :47.771015+0000 osd.2 (osd.2) 1332 2021-02-10T04:35:49.893 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:49.893 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:35:49.893 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: 35:47.771022+0000 osd.2 (osd.2) 2021-02-10T04:35:49.893 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: 1333 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:49.893 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:35:49.893 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: 04:35:47.771027+0000 osd.2 (osd. 2021-02-10T04:35:49.894 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: 2) 1334 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:49.894 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: cluster 2021-02-10T04:35:47 2021-02-10T04:35:49.894 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: .771033+0000 osd.2 (osd.2) 1335 : cluster 2021-02-10T04:35:49.894 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:49.894 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: cluster 2021-02-10T04:35:47 2021-02-10T04:35:49.894 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: .771038+0000 osd.2 (osd.2) 1336 : 2021-02-10T04:35:49.894 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:49.894 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: cluster 2021-02-10T04:35 2021-02-10T04:35:49.894 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: :47.771044+0000 osd.2 (osd.2) 1337 : cluster 2021-02-10T04:35:49.894 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:49.894 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: cluster 2021-02-10T04:35:47 2021-02-10T04:35:49.894 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: .771049+0000 osd.2 (osd.2) 1338 : cluster 2021-02-10T04:35:49.894 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:49.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: cluster 2021-02-10T04:35 2021-02-10T04:35:49.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: :49.030169+0000 osd.4 (osd.4) 735 2021-02-10T04:35:49.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:49.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:35:49.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: 35:49.030178+0000 osd.4 (osd.4 2021-02-10T04:35:49.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: ) 736 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:49.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:35:49.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: 35:49.030188+0000 osd.4 (osd.4) 2021-02-10T04:35:49.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: 737 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:49.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:35:49.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: 04:35:49.030213+0000 osd.4 (osd. 2021-02-10T04:35:49.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[35961]: 4) 738 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:49.895 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:47 smithi164 conmon[33677]: debug 2021-02-10T04:35:47.989+0000 7fbec19a1700 -1 osd.4 530 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:35:49.896 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[33677]: debug 2021-02-10T04:35:49.896 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[33677]: 2021-02-10T04:35:49.028+0000 7fbec19a1700 -1 osd.4 531 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:35:49.896 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[33677]: 2021-02-10T04:35:49.896 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:35:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: cluster 2021-02-10T04:35:46.766544+0000 osd.2 ( 2021-02-10T04:35:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: osd.2) 1319 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: cluster 2021-02-10T04:35:46.766551+0000 osd.2 2021-02-10T04:35:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: (osd.2) 1320 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:49.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:35:49.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: 02-10T04:35:46.766555+0000 osd.2 (osd.2) 1321 : cluster 2021-02-10T04:35:49.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:49.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: cluster 2021-02-10T04:35 2021-02-10T04:35:49.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: :46.766558+0000 osd.2 (osd.2) 1322 : 2021-02-10T04:35:49.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:49.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:35:49.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: -02-10T04:35:46.766563+0000 osd.2 (osd.2) 1323 2021-02-10T04:35:49.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:49.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:35:49.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: 10T04:35:46 2021-02-10T04:35:49.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: .766576+0000 osd.2 (osd.2) 1324 : cluster 2021-02-10T04:35:49.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:49.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: cluster 2021-02-10T04:35 2021-02-10T04:35:49.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: :46.766582+0000 osd.2 (osd.2) 1325 2021-02-10T04:35:49.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:49.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:35:49.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: T04:35:46.766589+0000 osd.2 2021-02-10T04:35:49.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: (osd.2) 2021-02-10T04:35:49.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: 1326 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:49.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: cluster 2021-02-10T04:35:49.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: 2021-02-10T04:35:46.766596+0000 2021-02-10T04:35:49.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: osd.2 (osd.2) 1327 : cluster [WRN] 2021-02-10T04:35:49.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:49.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: cluster 2021-02-10T04:35:46.766602+0000 2021-02-10T04:35:49.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: osd.2 (osd.2) 1328 : cluster [WRN] 2021-02-10T04:35:49.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:49.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: cluster 2021-02-10T04:35 2021-02-10T04:35:49.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: :47.389174+0000 mon.a (mon.0) 1359 2021-02-10T04:35:49.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: : cluster [DBG] osdmap e530: 8 total, 8 up, 8 in 2021-02-10T04:35:49.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:35:49.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: 35:47.990866+0000 osd.4 (osd.4) 2021-02-10T04:35:49.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: 731 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:49.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:35:49.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: 04:35:47.990875+0000 osd.4 (osd. 2021-02-10T04:35:49.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: 4) 732 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:49.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: cluster 2021-02-10T04:35:47.990898+0000 2021-02-10T04:35:49.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: osd.4 (osd.4) 733 : cluster [WRN] 2021-02-10T04:35:49.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:49.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: cluster 2021-02-10T04:35 2021-02-10T04:35:49.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: :47.990902+0000 osd.4 (osd.4) 734 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:49.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: cluster 2021-02-10T04:35:49.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: 2021-02-10T04:35:48.106948+0000 mgr.y (mgr.14140) 607 : 2021-02-10T04:35:49.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: cluster [DBG] pgmap v945: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 396 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.8 MiB/s rd, 1.1 MiB/s wr, 8 op/s 2021-02-10T04:35:49.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:35:49.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: T04:35:48.107078+0000 mgr.y (mgr 2021-02-10T04:35:49.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: .14140) 608 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:35:49.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: cluster 2021-02-10T04:35:49.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: 2021-02-10T04:35:48.168863+0000 2021-02-10T04:35:49.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: mon.a (mon.0) 1360 : cluster [DBG] 2021-02-10T04:35:49.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:48 smithi164 conmon[29288]: osdmap e531: 8 total, 8 up, 8 in 2021-02-10T04:35:49.901 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.4 dump_blocked_ops 2021-02-10T04:35:49.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: cluster 2021-02-10T04:35:47.770996+0000 osd.2 (osd.2) 2021-02-10T04:35:49.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: 1329 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:49.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: cluster 2021-02-10T04:35:47.771004+0000 osd.2 (osd.2) 1330 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:49.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: cluster 2021-02-10T04:35:47.771010+0000 osd.2 (osd. 2021-02-10T04:35:49.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: 2) 1331 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:49.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:35:49.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: -02-10T04:35:47.771015+0000 osd.2 (osd.2) 1332 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:49.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: cluster 2021-02-10T04:35: 2021-02-10T04:35:49.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: 47.771022+0000 osd.2 (osd.2) 1333 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:49.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:49.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: -10T04:35:47.771027+0000 osd.2 (osd.2) 1334 : 2021-02-10T04:35:49.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:49.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: cluster 2021-02-10T04:35:47.771033+0000 osd.2 (osd. 2021-02-10T04:35:49.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: 2) 1335 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:49.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: cluster 2021-02-10T04:35:47.771038 2021-02-10T04:35:49.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1336 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:49.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: cluster 2021-02-10T04:35:47.771044+0000 osd.2 (osd 2021-02-10T04:35:49.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: .2) 1337 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:49.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: cluster 2021-02-10T04:35:47. 2021-02-10T04:35:49.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: 771049+0000 osd.2 (osd.2) 1338 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:49.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:35:49.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: 10T04:35:49.030169+0000 osd.4 (osd.4) 735 : 2021-02-10T04:35:49.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:49.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: cluster 2021-02-10T04:35:49.030178+0000 osd.4 ( 2021-02-10T04:35:49.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: osd.4) 736 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:49.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: cluster 2021-02-10T04:35: 2021-02-10T04:35:49.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: 49.030188+0000 osd.4 (osd.4) 737 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:49.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: 2021-02-10T04:35:49.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:35:49.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: 02-10T04:35:49.030213+0000 osd.4 (osd.4) 738 : cluster [WRN] 2021-02-10T04:35:49.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:49 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:50.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:49 smithi085 conmon[49106]: debug 2021-02-10T04:35:49.795+0000 7f77a3e94700 -1 osd.2 532 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:35:50.231 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:35:50.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[33677]: debug 2021-02-10T04:35:50.001+0000 7fbec19a1700 -1 osd.4 532 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:35:50.419 INFO:tasks.rados.rados.0.smithi085.stdout:2058: done (10 left) 2021-02-10T04:35:50.420 INFO:tasks.rados.rados.0.smithi085.stdout:2059: done (9 left) 2021-02-10T04:35:50.420 INFO:tasks.rados.rados.0.smithi085.stdout:2060: write oid 2 current snap is 228 2021-02-10T04:35:50.420 INFO:tasks.rados.rados.0.smithi085.stdout:2060: seq_num 723 ranges {551461=658562,1941540=209921} 2021-02-10T04:35:50.427 INFO:tasks.rados.rados.0.smithi085.stdout:2060: writing smithi08575397-2 from 551461 to 1210023 tid 1 2021-02-10T04:35:50.433 INFO:tasks.rados.rados.0.smithi085.stdout:2060: writing smithi08575397-2 from 1941540 to 2151461 tid 2 2021-02-10T04:35:50.433 INFO:tasks.rados.rados.0.smithi085.stdout:2061: snap_remove snap 222 2021-02-10T04:35:50.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: cluster 2021-02-10T04:35:50.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: 2021-02-10T04:35:50.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: 2021 2021-02-10T04:35:50.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: -02-10T04:35:48. 2021-02-10T04:35:50.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: 627367+0000 osd.1 (osd.1) 449 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:35:50.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:35:50.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: -02-10T04:35:48.627657+0000 2021-02-10T04:35:50.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: osd.1 (osd.1) 450 : cluster [DBG] 2021-02-10T04:35:50.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: purged_snaps scrub ok 2021-02-10T04:35:50.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: cluster 2021-02-10T04:35:50.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: 2021-02-10T04:35:48.778399 2021-02-10T04:35:50.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1339 : cluster 2021-02-10T04:35:50.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:50.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: cluster 2021-02-10T04:35:50.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: 2021-02-10T04:35: 2021-02-10T04:35:50.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: 48.778408+0000 osd.2 (osd.2) 1340 : 2021-02-10T04:35:50.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:50.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:35:50.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: -10T04:35:48.778415+0000 osd.2 (osd.2) 2021-02-10T04:35:50.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: 1341 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:50.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: cluster 2021-02-10T04:35:50.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: 2021-02-10T04:35:48.778421+0000 osd.2 (osd 2021-02-10T04:35:50.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: .2) 1342 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:50.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: cluster 2021-02-10T04:35:50.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: 2021-02-10T04:35:48.778429+0000 2021-02-10T04:35:50.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: osd.2 (osd.2) 1343 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:50.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: cluster 2021-02-10T04:35:50.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:35:50.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: 10T04:35:48.778435+0000 osd.2 (osd.2) 1344 : cluster 2021-02-10T04:35:50.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:50.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:50.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: 02-10T04:35:48.778441+0000 osd.2 2021-02-10T04:35:50.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: (osd 2021-02-10T04:35:50.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: .2) 1345 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:50.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: cluster 2021-02-10T04:35:50.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: 2021-02-10T04:35:50.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:35:50.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: :35:48.778446+0000 osd.2 (osd.2) 1346 : cluster [WRN] 2021-02-10T04:35:50.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:50.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: cluster 2021-02-10T04:35 2021-02-10T04:35:50.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: :48.778452+0000 osd.2 (osd 2021-02-10T04:35:50.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: .2) 1347 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:50.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:50.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: 02-10T04:35:48.778458+0000 osd.2 ( 2021-02-10T04:35:50.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: osd.2) 1348 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:50.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: cluster 2021-02-10T04:35:50.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:35:50.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: 35:49.408788+0000 mon.a (mon.0) 1361 : cluster [DBG] osdmap e532: 8 total, 8 up, 8 in 2021-02-10T04:35:50.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: cluster 2021-02-10T04:35 2021-02-10T04:35:50.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: :50.002769+0000 osd.4 (osd.4) 739 : cluster 2021-02-10T04:35:50.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:50.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:35:50.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: 35:50.002780+0000 osd.4 (osd.4) 2021-02-10T04:35:50.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: 740 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:50.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:35:50.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: T04:35:50.002787+0000 osd.4 (osd.4) 741 2021-02-10T04:35:50.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:50.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: cluster 2021-02-10T04:35: 2021-02-10T04:35:50.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: 50.002794+0000 osd.4 (osd.4) 742 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:50.709 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:35:50.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: 04:35:50.107538+0000 mgr.y (mgr.14140 2021-02-10T04:35:50.710 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[29319]: ) 609 : cluster [DBG] pgmap v948: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 157 active+clean; 409 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 2.2 MiB/s wr, 18 op/s 2021-02-10T04:35:50.711 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: cluster 2021-02-10T04:35:50.712 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: 2021-02-10T04:35:50.712 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: 2021 2021-02-10T04:35:50.712 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: - 2021-02-10T04:35:50.712 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: 02 2021-02-10T04:35:50.712 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: - 2021-02-10T04:35:50.712 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: 10T04:35:48.627367+0000 2021-02-10T04:35:50.713 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: osd.1 (osd.1) 449 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:35:50.713 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:35:50.713 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: 10T04:35:48.627657+0000 osd.1 (osd.1) 2021-02-10T04:35:50.713 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: 450 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:35:50.713 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:50.713 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: 02-10T04:35:48.778399+0000 2021-02-10T04:35:50.713 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: osd.2 (osd.2) 1339 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:50.714 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: 2021-02-10T04:35:50.714 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: cluster 2021-02-10T04:35:50.714 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: 2021-02-10T04:35:48.778408+0000 2021-02-10T04:35:50.714 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: osd.2 (osd.2) 1340 : cluster [WRN] 2021-02-10T04:35:50.715 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:50.715 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:35:50.715 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: T04:35:48.778415+0000 osd.2 (osd.2) 2021-02-10T04:35:50.715 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: 1341 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:50.715 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:35:50.715 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: T04:35:48.778421+0000 osd.2 (osd 2021-02-10T04:35:50.715 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: .2) 1342 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:50.716 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:50.716 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: 02-10T04:35:48.778429+0000 osd.2 2021-02-10T04:35:50.716 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: (osd.2) 1343 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:50.716 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:35:50.716 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: -02-10T04:35:48.778435+0000 2021-02-10T04:35:50.716 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: osd.2 (osd.2) 1344 : cluster [WRN] 2021-02-10T04:35:50.716 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:50.717 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: cluster 2021-02-10T04:35: 2021-02-10T04:35:50.717 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: 48.778441+0000 osd.2 (osd.2) 1345 : cluster 2021-02-10T04:35:50.717 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:50.717 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: cluster 2021-02-10T04:35:50.717 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: 2021-02-10T04:35:48.778446+0000 osd.2 2021-02-10T04:35:50.717 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: (osd.2) 1346 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:50.717 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:35:50.718 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: 10T04:35:48.778452+0000 osd.2 (osd.2) 1347 : cluster 2021-02-10T04:35:50.718 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:50.718 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: cluster 2021-02-10T04:35:50.718 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: 2021-02-10T04:35:48.778458+0000 osd.2 ( 2021-02-10T04:35:50.718 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: osd.2) 1348 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:50.719 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:35:50.719 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: -10T04:35:49.408788+0000 mon.a ( 2021-02-10T04:35:50.719 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: mon.0) 1361 : cluster [DBG] osdmap e532: 8 total, 8 up, 8 in 2021-02-10T04:35:50.719 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:35:50.719 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: -02-10T04:35:50.002769+0000 2021-02-10T04:35:50.719 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: osd.4 (osd.4) 739 : cluster [WRN] 2021-02-10T04:35:50.719 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:50.719 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: cluster 2021-02-10T04:35:50. 2021-02-10T04:35:50.719 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: 002780+0000 osd.4 (osd.4) 740 : cluster 2021-02-10T04:35:50.720 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:50.720 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: cluster 2021-02-10T04:35:50 2021-02-10T04:35:50.720 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: .002787+0000 osd.4 (osd.4) 741 : cluster 2021-02-10T04:35:50.720 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:50.720 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: cluster 2021-02-10T04:35: 2021-02-10T04:35:50.720 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: 50.002794+0000 osd.4 (osd.4) 742 : cluster 2021-02-10T04:35:50.720 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:50.721 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: cluster 2021-02-10T04:35 2021-02-10T04:35:50.721 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: :50.107538+0000 mgr.y (mgr.14140) 609 : 2021-02-10T04:35:50.721 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[35961]: cluster [DBG] pgmap v948: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 157 active+clean; 409 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 2.2 MiB/s wr, 18 op/s 2021-02-10T04:35:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: cluster 2021-02-10T04:35:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:35:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: :35:48.627367+0000 osd.1 (osd.1) 449 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:35:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:35:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: -02-10T04:35:48.627657+0000 osd.1 (osd. 2021-02-10T04:35:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: 1) 450 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:35:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:35:50.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: -02-10T04:35:48.778399+0000 osd.2 ( 2021-02-10T04:35:50.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: osd.2) 1339 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:50.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: cluster 2021-02-10T04:35:50.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: 2021-02-10T04:35:48.778408+0000 osd.2 (osd.2) 1340 : 2021-02-10T04:35:50.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:50.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: cluster 2021-02-10T04:35 2021-02-10T04:35:50.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: :48.778415+0000 osd.2 (osd.2) 1341 : cluster 2021-02-10T04:35:50.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:50.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: cluster 2021-02-10T04:35:48.778421+0000 2021-02-10T04:35:50.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: osd.2 (osd.2) 1342 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:50.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: 2021-02-10T04:35:50.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: cluster 2021-02-10T04:35:48. 2021-02-10T04:35:50.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: 778429+0000 osd.2 (osd.2) 1343 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:50.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: 2021-02-10T04:35:50.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: cluster 2021-02-10T04:35:50.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: 2021-02-10T04:35:48.778435+0000 osd.2 (osd. 2021-02-10T04:35:50.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: 2) 1344 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:50.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:35:50.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: 04:35:48.778441+0000 osd.2 (osd.2) 1345 2021-02-10T04:35:50.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:50.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: cluster 2021-02-10T04:35:48.778446 2021-02-10T04:35:50.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1346 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:50.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: 2021-02-10T04:35:50.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:35:50.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: -02-10T04:35:48.778452+0000 osd.2 (osd. 2021-02-10T04:35:50.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: 2) 1347 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:50.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:35:50.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: 35:48.778458+0000 osd.2 (osd.2) 1348 : cluster 2021-02-10T04:35:50.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:50.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: cluster 2021-02-10T04:35:50.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: 2021-02-10T04:35:49.408788+0000 mon.a (mon.0) 1361 2021-02-10T04:35:50.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: : cluster [DBG] osdmap e532: 8 total, 8 up, 8 in 2021-02-10T04:35:50.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: cluster 2021-02-10T04:35:50.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: 2021-02-10 2021-02-10T04:35:50.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: T04:35:50.002769+0000 osd.4 (osd 2021-02-10T04:35:50.906 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: .4) 739 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:50.906 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: cluster 2021-02-10T04:35:50.906 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:35:50.906 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: 04:35:50.002780+0000 osd.4 (osd. 2021-02-10T04:35:50.906 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: 4) 740 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:50.907 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:35:50.907 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: 35:50.002787+0000 osd.4 (osd.4) 741 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:50.907 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: cluster 2021-02-10T04:35 2021-02-10T04:35:50.907 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: :50.002794+0000 osd.4 (osd.4) 742 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:50.907 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: cluster 2021-02-10T04:35 2021-02-10T04:35:50.908 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: :50.107538+0000 mgr.y (mgr.14140) 609 : cluster [DBG] pgmap v948: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 157 active+clean; 409 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 2.2 MiB/s wr, 18 op/s 2021-02-10T04:35:50.908 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:50 smithi164 conmon[29288]: 2021-02-10T04:35:51.058 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:35:51.088 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:50 smithi085 conmon[49106]: debug 2021-02-10T04:35:50.819+0000 7f77a3e94700 -1 osd.2 533 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:35:51.091 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:35:51.268 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[33677]: debug 2021-02-10T04:35:51.017+0000 7fbec19a1700 -1 osd.4 533 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:35:51.423 INFO:tasks.rados.rados.0.smithi085.stdout:2061: done (10 left) 2021-02-10T04:35:51.423 INFO:tasks.rados.rados.0.smithi085.stdout:2062: delete oid 10 current snap is 228 2021-02-10T04:35:51.423 INFO:tasks.rados.rados.0.smithi085.stdout:2060: finishing write tid 1 to smithi08575397-2 2021-02-10T04:35:51.423 INFO:tasks.rados.rados.0.smithi085.stdout:2060: finishing write tid 2 to smithi08575397-2 2021-02-10T04:35:51.423 INFO:tasks.rados.rados.0.smithi085.stdout:2060: finishing write tid 3 to smithi08575397-2 2021-02-10T04:35:51.423 INFO:tasks.rados.rados.0.smithi085.stdout:2060: finishing write tid 4 to smithi08575397-2 2021-02-10T04:35:51.423 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 670 (ObjNum 723 snap 228 seq_num 723) dirty exists 2021-02-10T04:35:51.423 INFO:tasks.rados.rados.0.smithi085.stdout:2060: left oid 2 (ObjNum 723 snap 228 seq_num 723) 2021-02-10T04:35:51.432 INFO:tasks.rados.rados.0.smithi085.stdout:2060: done (10 left) 2021-02-10T04:35:51.432 INFO:tasks.rados.rados.0.smithi085.stdout:2062: done (9 left) 2021-02-10T04:35:51.432 INFO:tasks.rados.rados.0.smithi085.stdout:2063: delete oid 5 current snap is 228 2021-02-10T04:35:51.435 INFO:tasks.rados.rados.0.smithi085.stdout:2063: done (9 left) 2021-02-10T04:35:51.436 INFO:tasks.rados.rados.0.smithi085.stdout:2064: delete oid 14 current snap is 228 2021-02-10T04:35:51.438 INFO:tasks.rados.rados.0.smithi085.stdout:2064: done (9 left) 2021-02-10T04:35:51.438 INFO:tasks.rados.rados.0.smithi085.stdout:2065: snap_create 2021-02-10T04:35:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: cluster 2021-02-10T04:35 2021-02-10T04:35:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: :49.796904+0000 osd.2 (osd.2) 1349 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: cluster 2021-02-10T04:35:49.796913+0000 osd.2 (osd. 2021-02-10T04:35:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: 2) 1350 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: cluster 2021-02-10T04:35:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: 2021-02-10T04:35:49. 2021-02-10T04:35:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: 796921+0000 osd.2 (osd.2) 1351 : cluster 2021-02-10T04:35:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:35:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: 04:35:49.796927+0000 osd.2 (osd. 2021-02-10T04:35:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: 2) 1352 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:35:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: -02-10T04:35:49. 2021-02-10T04:35:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: 796931+0000 osd.2 (osd.2) 1353 : cluster [WRN] 2021-02-10T04:35:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:35:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: 04:35:49.796934+0000 osd.2 (osd. 2021-02-10T04:35:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: 2) 1354 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:35:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: -02-10T04:35:49.796938+0000 2021-02-10T04:35:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: osd.2 (osd.2) 1355 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: 2021-02-10T04:35:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: cluster 2021-02-10T04:35 2021-02-10T04:35:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: :49.796941+0000 osd.2 (osd.2) 1356 2021-02-10T04:35:51.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:51.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:51.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: -10T04:35:49.796944+0000 osd.2 2021-02-10T04:35:51.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: (osd.2) 1357 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:51.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:35:51.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: -02-10T04:35:49.796947+0000 osd.2 ( 2021-02-10T04:35:51.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: osd.2) 1358 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:51.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: cluster 2021-02-10T04:35:51.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: 2021-02-10T04:35:50.418436 2021-02-10T04:35:51.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1362 : cluster 2021-02-10T04:35:51.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: [DBG] osdmap e533: 8 total, 8 up, 8 in 2021-02-10T04:35:51.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:35:51.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: 10T04:35:51.019046+0000 osd.4 ( 2021-02-10T04:35:51.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: osd.4) 743 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:51.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:35:51.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: -02-10T04:35:51.019055+0000 2021-02-10T04:35:51.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: osd.4 (osd.4) 744 : cluster [WRN] 2021-02-10T04:35:51.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:51.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: cluster 2021-02-10T04:35:51.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: 2021-02-10T04:35:51.019062+0000 osd.4 (osd 2021-02-10T04:35:51.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: .4) 745 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:51.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:35:51.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: 02-10T04:35:51.019093+0000 osd.4 2021-02-10T04:35:51.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: (osd.4) 746 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:51.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:51 smithi164 conmon[29288]: 2021-02-10T04:35:51.694 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[41540]: debug 2021-02-10T04:35:51.518+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:35:51.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: cluster 2021-02-10T04:35:49.796904 2021-02-10T04:35:51.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1349 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:51.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: cluster 2021-02-10T04:35:49.796913+0000 2021-02-10T04:35:51.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: osd.2 (osd.2) 1350 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:51.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:35:51.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: -10T04:35:49.796921+0000 2021-02-10T04:35:51.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: osd.2 (osd.2) 1351 : cluster [WRN] 2021-02-10T04:35:51.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:51.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: cluster 2021-02-10T04:35: 2021-02-10T04:35:51.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: 49.796927+0000 osd.2 (osd.2) 1352 : 2021-02-10T04:35:51.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:51.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:35:51.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: T04:35:49.796931+0000 osd.2 (osd 2021-02-10T04:35:51.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: .2) 1353 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:51.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:35:51.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: -02-10T04:35:49.796934+0000 2021-02-10T04:35:51.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: osd.2 (osd.2) 1354 : cluster [WRN] 2021-02-10T04:35:51.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:51.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:35:51.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: :35:49.796938+0000 osd.2 (osd.2) 1355 : cluster [WRN] 2021-02-10T04:35:51.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:51.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: cluster 2021-02-10T04:35 2021-02-10T04:35:51.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: :49.796941+0000 osd.2 (osd.2) 1356 : 2021-02-10T04:35:51.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:51.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:35:51.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: 04:35:49.796944+0000 osd.2 (osd 2021-02-10T04:35:51.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: .2) 1357 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:51.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:35:51.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: -10T04:35:49.796947+0000 2021-02-10T04:35:51.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: osd.2 (osd.2) 1358 : cluster [WRN] 2021-02-10T04:35:51.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:51.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: cluster 2021-02-10T04:35:50 2021-02-10T04:35:51.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: .418436+0000 mon.a (mon.0) 1362 : cluster 2021-02-10T04:35:51.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: [DBG] osdmap e533: 8 total, 8 up, 8 in 2021-02-10T04:35:51.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:35:51.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: 10T04:35:51.019046+0000 osd.4 (osd.4) 743 : 2021-02-10T04:35:51.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:51.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:35:51.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: 35:51.019055+0000 osd.4 (osd.4) 2021-02-10T04:35:51.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: 744 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:51.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:35:51.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: 35:51.019062+0000 osd.4 (osd.4) 2021-02-10T04:35:51.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: 745 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:51.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:35:51.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: 35:51.019093+0000 osd.4 (osd.4) 2021-02-10T04:35:51.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[35961]: 746 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:51.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: cluster 2021-02-10T04:35:51.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:35:51.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: 10T04:35:49.796904+0000 osd.2 (osd.2) 1349 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:51.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:35:51.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: T04:35:49.796913+0000 osd.2 (osd.2) 1350 : cluster [WRN] 2021-02-10T04:35:51.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:51.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:35:51.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: T04:35:49.796921+0000 osd.2 (osd. 2021-02-10T04:35:51.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: 2) 1351 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:51.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:35:51.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: -02-10T04:35:49.796927+0000 2021-02-10T04:35:51.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: osd.2 (osd.2) 1352 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:51.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: 2021-02-10T04:35:51.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:51.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: 02-10T04:35:49.796931+0000 osd.2 (osd. 2021-02-10T04:35:51.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: 2) 1353 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:51.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:35:51.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: :35:49.796934+0000 osd.2 (osd.2) 1354 : 2021-02-10T04:35:51.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:51.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: cluster 2021-02-10T04:35:49.796938 2021-02-10T04:35:51.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1355 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:51.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: cluster 2021-02-10T04:35 2021-02-10T04:35:51.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: :49.796941+0000 osd.2 (osd.2) 1356 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:51.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:35:51.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: T04:35:49.796944+0000 osd.2 (osd.2) 1357 2021-02-10T04:35:51.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:51.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: cluster 2021-02-10T04:35:51.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: 2021-02-10T04:35:49.796947+0000 osd.2 (osd.2) 1358 : cluster 2021-02-10T04:35:51.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:51.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:35:51.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: -02-10T04:35:50.418436+0000 mon.a (mon.0) 1362 : cluster [DBG] 2021-02-10T04:35:51.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: osdmap e533: 8 total, 8 up, 8 in 2021-02-10T04:35:51.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:51.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: 02-10T04:35:51.019046+0000 osd.4 (osd.4) 743 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:51.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: 2021-02-10T04:35:51.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:35:51.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: -10T04:35:51.019055+0000 osd.4 (osd.4) 2021-02-10T04:35:51.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: 744 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:51.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:35:51.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: -02-10T04:35:51.019062 2021-02-10T04:35:51.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 745 : cluster 2021-02-10T04:35:51.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:51.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:35:51.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: -02-10T04:35:51.019093+0000 2021-02-10T04:35:51.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: osd.4 (osd.4) 746 : cluster [WRN] 2021-02-10T04:35:51.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:51.751 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:35:52.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:51 smithi085 conmon[49106]: debug 2021-02-10T04:35:51.836+0000 7f77a3e94700 -1 osd.2 534 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:35:52.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[33677]: debug 2021-02-10T04:35:52.050+0000 7fbec19a1700 -1 osd.4 534 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:35:52.433 INFO:tasks.rados.rados.0.smithi085.stdout:2065: done (9 left) 2021-02-10T04:35:52.433 INFO:tasks.rados.rados.0.smithi085.stdout:2066: rollback oid 38 current snap is 229 2021-02-10T04:35:52.433 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 38 to 225 2021-02-10T04:35:52.433 INFO:tasks.rados.rados.0.smithi085.stdout:2067: read oid 1 snap 227 2021-02-10T04:35:52.433 INFO:tasks.rados.rados.0.smithi085.stdout:2067: expect (ObjNum 671 snap 207 seq_num 671) 2021-02-10T04:35:52.433 INFO:tasks.rados.rados.0.smithi085.stdout:2068: rollback oid 48 current snap is 229 2021-02-10T04:35:52.433 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 48 to 184 2021-02-10T04:35:52.433 INFO:tasks.rados.rados.0.smithi085.stdout:2069: rollback oid 14 current snap is 229 2021-02-10T04:35:52.433 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 14 to 228 2021-02-10T04:35:52.433 INFO:tasks.rados.rados.0.smithi085.stdout:2070: snap_create 2021-02-10T04:35:52.439 INFO:tasks.rados.rados.0.smithi085.stdout:2068: finishing rollback tid 0 to smithi08575397-48 2021-02-10T04:35:52.439 INFO:tasks.rados.rados.0.smithi085.stdout:2068: finishing rollback tid 1 to smithi08575397-48 2021-02-10T04:35:52.440 INFO:tasks.rados.rados.0.smithi085.stdout:2068: finishing rollback tid 2 to smithi08575397-48 2021-02-10T04:35:52.440 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 820 (ObjNum 476 snap 135 seq_num 476) dirty exists 2021-02-10T04:35:52.440 INFO:tasks.rados.rados.0.smithi085.stdout:2069: finishing rollback tid 1 to smithi08575397-14 2021-02-10T04:35:52.440 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 817 (ObjNum 82332432 snap 22030 seq_num 1215200352) dirty dne 2021-02-10T04:35:52.445 INFO:tasks.rados.rados.0.smithi085.stdout:2066: finishing rollback tid 1 to smithi08575397-38 2021-02-10T04:35:52.445 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 1031 (ObjNum 6 snap 1 seq_num 0) dirty dne 2021-02-10T04:35:52.445 INFO:tasks.rados.rados.0.smithi085.stdout:2067: expect (ObjNum 671 snap 207 seq_num 671) 2021-02-10T04:35:52.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: cluster 2021-02-10T04:35:52.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: 2021-02-10T04:35:52.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: 2021 2021-02-10T04:35:52.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: - 2021-02-10T04:35:52.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: 02 2021-02-10T04:35:52.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: - 2021-02-10T04:35:52.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: 10 2021-02-10T04:35:52.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: T 2021-02-10T04:35:52.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: 04 2021-02-10T04:35:52.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: : 2021-02-10T04:35:52.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: 35 2021-02-10T04:35:52.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: : 2021-02-10T04:35:52.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: 50 2021-02-10T04:35:52.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: . 2021-02-10T04:35:52.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: 821483+0000 osd.2 (osd 2021-02-10T04:35:52.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: .2) 1359 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:52.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:52.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: 02-10T04:35:50.821490+0000 2021-02-10T04:35:52.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: osd.2 (osd.2) 1360 : cluster [WRN] 2021-02-10T04:35:52.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:52.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:35:52.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: 35:50.821494+0000 osd.2 (osd.2) 1361 : cluster 2021-02-10T04:35:52.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:52.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:35:52.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: :35:50.821498+0000 osd.2 (osd.2 2021-02-10T04:35:52.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: ) 1362 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:52.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:52.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: 02-10T04:35:50.821502+0000 2021-02-10T04:35:52.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: osd.2 (osd.2) 1363 : cluster [WRN] 2021-02-10T04:35:52.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:52.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: cluster 2021-02-10T04:35:50. 2021-02-10T04:35:52.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: 821505+0000 osd.2 (osd.2) 1364 : cluster 2021-02-10T04:35:52.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:52.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: cluster 2021-02-10T04:35:52.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: 2021-02-10T04:35 2021-02-10T04:35:52.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: :50.821509+0000 osd.2 (osd.2 2021-02-10T04:35:52.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: ) 1365 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:52.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: cluster 2021-02-10T04:35:52.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: 2021-02-10T04:35: 2021-02-10T04:35:52.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: 50.821512+0000 osd.2 (osd.2) 1366 2021-02-10T04:35:52.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:52.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:35:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: -10T04:35:50.821516+0000 osd.2 2021-02-10T04:35:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: (osd.2) 1367 : cluster 2021-02-10T04:35:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:35:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: -10T04:35:50.821519+0000 osd.2 2021-02-10T04:35:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: (osd.2) 1368 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: cluster 2021-02-10T04:35:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: 2021-02-10T04:35:51.421760+0000 2021-02-10T04:35:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: mon.a (mon.0) 1363 : cluster [DBG] 2021-02-10T04:35:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: osdmap e534: 8 total, 8 up, 8 in 2021-02-10T04:35:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: cluster 2021-02-10T04:35: 2021-02-10T04:35:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: 52.051780+0000 osd.4 (osd.4) 747 2021-02-10T04:35:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:35:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: 10T04:35:52.051791+0000 osd.4 ( 2021-02-10T04:35:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: osd.4) 748 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: cluster 2021-02-10T04:35:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: 2021-02-10T04:35:52.051798+0000 2021-02-10T04:35:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: osd.4 (osd.4) 749 : cluster [WRN] 2021-02-10T04:35:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:35:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: :35:52.051803+0000 osd.4 (osd.4) 750 : 2021-02-10T04:35:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: cluster 2021-02-10T04:35 2021-02-10T04:35:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: :52.108054+0000 mgr.y (mgr.14140) 610 2021-02-10T04:35:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[35961]: : cluster [DBG] pgmap v951: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 157 active+clean; 411 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 2.9 MiB/s wr, 24 op/s 2021-02-10T04:35:52.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: cluster 2021-02-10T04:35:50.821483+0000 osd.2 (osd.2 2021-02-10T04:35:52.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: ) 1359 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:52.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: cluster 2021-02-10T04:35:50.821490+0000 osd.2 (osd.2) 1360 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:52.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: cluster 2021-02-10T04:35 2021-02-10T04:35:52.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: :50.821494+0000 osd.2 (osd.2) 1361 : 2021-02-10T04:35:52.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:52.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:35:52.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: 35:50.821498+0000 osd.2 (osd.2) 2021-02-10T04:35:52.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: 1362 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:52.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:35:52.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: :35:50.821502+0000 osd.2 (osd. 2021-02-10T04:35:52.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: 2) 1363 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:52.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:35:52.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: 10T04:35:50.821505+0000 osd.2 ( 2021-02-10T04:35:52.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: osd.2) 1364 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:52.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:52.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: 02-10T04:35:50.821509+0000 2021-02-10T04:35:52.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: osd.2 (osd.2) 1365 : cluster [WRN] 2021-02-10T04:35:52.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:52.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: cluster 2021-02-10T04:35:50. 2021-02-10T04:35:52.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: 821512+0000 osd.2 (osd.2) 1366 : cluster 2021-02-10T04:35:52.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:52.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:35:52.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: 35:50.821516+0000 osd.2 (osd.2) 2021-02-10T04:35:52.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: 1367 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:52.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:35:52.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: T04:35:50.821519+0000 osd.2 (osd 2021-02-10T04:35:52.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: .2) 1368 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:52.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:52.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: 02-10T04:35:51.421760+0000 2021-02-10T04:35:52.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: mon.a (mon.0) 1363 : cluster [DBG] 2021-02-10T04:35:52.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: osdmap e534: 8 total, 8 up, 8 in 2021-02-10T04:35:52.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: cluster 2021-02-10T04:35:52.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: 2021-02-10T04:35:52.051780+0000 osd.4 2021-02-10T04:35:52.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: (osd.4) 747 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:52.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:52.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: 02-10T04:35:52.051791+0000 2021-02-10T04:35:52.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: osd.4 (osd.4) 748 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:52.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: 2021-02-10T04:35:52.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: cluster 2021-02-10T04:35:52.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: 2021-02-10T04:35:52.051798+0000 osd.4 ( 2021-02-10T04:35:52.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: osd.4) 749 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:52.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:35:52.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: -02-10T04:35:52.051803+0000 2021-02-10T04:35:52.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: osd.4 (osd.4) 750 : cluster [WRN] 2021-02-10T04:35:52.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:52.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: cluster 2021-02-10T04:35:52 2021-02-10T04:35:52.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: .108054+0000 mgr.y (mgr.14140) 610 : 2021-02-10T04:35:52.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[29319]: cluster [DBG] pgmap v951: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 157 active+clean; 411 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 2.9 MiB/s wr, 24 op/s 2021-02-10T04:35:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: cluster 2021-02-10T04:35:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:35:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: :35:50.821483+0000 osd.2 (osd.2) 1359 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:35:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: T04:35:50.821490+0000 osd.2 (osd.2) 1360 : cluster 2021-02-10T04:35:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:35:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: 02-10T04:35:50.821494+0000 osd.2 (osd.2) 1361 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: 2021-02-10T04:35:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: cluster 2021-02-10T04:35 2021-02-10T04:35:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: :50.821498+0000 osd.2 (osd.2) 1362 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: cluster 2021-02-10T04:35: 2021-02-10T04:35:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: 50.821502+0000 osd.2 (osd.2) 1363 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: 2021-02-10T04:35:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:35:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: 35:50.821505+0000 osd.2 2021-02-10T04:35:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: (osd.2) 1364 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: cluster 2021-02-10T04:35: 2021-02-10T04:35:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: 50.821509+0000 osd.2 (osd.2) 1365 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: 2021-02-10T04:35:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: cluster 2021-02-10T04:35 2021-02-10T04:35:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: :50.821512+0000 osd.2 (osd.2) 1366 : 2021-02-10T04:35:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: cluster 2021-02-10T04:35:50. 2021-02-10T04:35:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: 821516+0000 osd.2 (osd.2) 1367 : cluster 2021-02-10T04:35:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: cluster 2021-02-10T04:35:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: 2021-02-10T04:35:50.821519+0000 osd.2 (osd. 2021-02-10T04:35:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: 2) 1368 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: cluster 2021-02-10T04:35: 2021-02-10T04:35:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: 51.421760+0000 mon.a (mon.0) 1363 : cluster [DBG] osdmap e534: 8 total, 8 up, 8 in 2021-02-10T04:35:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: 2021-02-10T04:35:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:35:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: 04:35:52.051780+0000 osd.4 (osd.4) 747 : cluster [WRN] 2021-02-10T04:35:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:35:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: 35:52.051791+0000 osd.4 (osd.4) 748 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: cluster 2021-02-10T04:35:52. 2021-02-10T04:35:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: 051798+0000 osd.4 (osd.4) 749 : cluster [WRN] 2021-02-10T04:35:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: cluster 2021-02-10T04:35:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: 2021-02-10T04:35:52.051803+0000 osd.4 (osd.4) 750 : cluster 2021-02-10T04:35:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: cluster 2021-02-10T04:35:52. 2021-02-10T04:35:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: 108054+0000 mgr.y (mgr.14140) 610 : cluster [DBG] 2021-02-10T04:35:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:52 smithi164 conmon[29288]: pgmap v951: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 157 active+clean; 411 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 2.9 MiB/s wr, 24 op/s 2021-02-10T04:35:53.114 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:52 smithi085 conmon[49106]: debug 2021-02-10T04:35:52.850+0000 7f77a3e94700 -1 osd.2 535 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:35:53.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[33677]: debug 2021-02-10T04:35:53.018+0000 7fbec19a1700 -1 osd.4 535 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:35:53.432 INFO:tasks.rados.rados.0.smithi085.stdout:2066: done (13 left) 2021-02-10T04:35:53.432 INFO:tasks.rados.rados.0.smithi085.stdout:2067: done (12 left) 2021-02-10T04:35:53.432 INFO:tasks.rados.rados.0.smithi085.stdout:2068: done (11 left) 2021-02-10T04:35:53.432 INFO:tasks.rados.rados.0.smithi085.stdout:2069: done (10 left) 2021-02-10T04:35:53.432 INFO:tasks.rados.rados.0.smithi085.stdout:2070: done (9 left) 2021-02-10T04:35:53.432 INFO:tasks.rados.rados.0.smithi085.stdout:2071: read oid 22 snap -1 2021-02-10T04:35:53.432 INFO:tasks.rados.rados.0.smithi085.stdout:2071: expect deleted 2021-02-10T04:35:53.433 INFO:tasks.rados.rados.0.smithi085.stdout:2072: write (excl) oid 7 current snap is 230 2021-02-10T04:35:53.433 INFO:tasks.rados.rados.0.smithi085.stdout:2072: seq_num 727 ranges {618689=559558,1659115=559574} 2021-02-10T04:35:53.438 INFO:tasks.rados.rados.0.smithi085.stdout:2072: writing smithi08575397-7 from 618689 to 1178247 tid 1 2021-02-10T04:35:53.445 INFO:tasks.rados.rados.0.smithi085.stdout:2072: writing smithi08575397-7 from 1659115 to 2218689 tid 2 2021-02-10T04:35:53.445 INFO:tasks.rados.rados.0.smithi085.stdout:2073: delete oid 44 current snap is 230 2021-02-10T04:35:53.445 INFO:tasks.rados.rados.0.smithi085.stdout:2072: finishing write tid 1 to smithi08575397-7 2021-02-10T04:35:53.449 INFO:tasks.rados.rados.0.smithi085.stdout:2071: done (11 left) 2021-02-10T04:35:53.449 INFO:tasks.rados.rados.0.smithi085.stdout:2073: done (10 left) 2021-02-10T04:35:53.449 INFO:tasks.rados.rados.0.smithi085.stdout:2074: read oid 22 snap 218 2021-02-10T04:35:53.449 INFO:tasks.rados.rados.0.smithi085.stdout:2074: expect (ObjNum 570 snap 175 seq_num 570) 2021-02-10T04:35:53.449 INFO:tasks.rados.rados.0.smithi085.stdout:2075: copy_from oid 20 from oid 14 current snap is 230 2021-02-10T04:35:53.449 INFO:tasks.rados.rados.0.smithi085.stdout:2076: write (excl) oid 6 current snap is 230 2021-02-10T04:35:53.449 INFO:tasks.rados.rados.0.smithi085.stdout:2076: seq_num 729 ranges {652303=710056,2059726=192577} 2021-02-10T04:35:53.458 INFO:tasks.rados.rados.0.smithi085.stdout:2076: writing smithi08575397-6 from 652303 to 1362359 tid 1 2021-02-10T04:35:53.463 INFO:tasks.rados.rados.0.smithi085.stdout:2076: writing smithi08575397-6 from 2059726 to 2252303 tid 2 2021-02-10T04:35:53.463 INFO:tasks.rados.rados.0.smithi085.stdout:2077: snap_remove snap 226 2021-02-10T04:35:53.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: cluster 2021-02-10T04:35:51.837690+0000 osd.2 (osd.2) 2021-02-10T04:35:53.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: 1369 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:53.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: cluster 2021-02-10T04:35:53.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: 2021 2021-02-10T04:35:53.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: - 2021-02-10T04:35:53.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: 02 2021-02-10T04:35:53.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: - 2021-02-10T04:35:53.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: 10 2021-02-10T04:35:53.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: T04:35:51.837700+0000 osd.2 (osd 2021-02-10T04:35:53.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: .2) 1370 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:53.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:35:53.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: 10T04:35:51.837707+0000 osd.2 ( 2021-02-10T04:35:53.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: osd.2) 1371 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:53.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: cluster 2021-02-10T04:35:51.837713+0000 osd.2 (osd 2021-02-10T04:35:53.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: .2) 1372 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:53.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:35:53.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: -10T04:35:51.837718+0000 osd.2 2021-02-10T04:35:53.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: (osd.2) 1373 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:53.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:35:53.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: 10T04:35:51.837724+0000 osd.2 (osd. 2021-02-10T04:35:53.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: 2) 1374 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:53.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:35:53.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: -02-10T04:35:51.837730+0000 2021-02-10T04:35:53.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: osd.2 (osd.2) 1375 : cluster [WRN] 2021-02-10T04:35:53.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:53.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: cluster 2021-02-10T04:35 2021-02-10T04:35:53.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: :51.837736+0000 osd.2 (osd.2) 2021-02-10T04:35:53.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: 1376 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:53.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:53.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: 02-10T04:35:51.837742+0000 2021-02-10T04:35:53.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: osd.2 (osd.2) 1377 : cluster [WRN] 2021-02-10T04:35:53.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:53.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:35:53.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: :35:51.837747+0000 osd.2 (osd. 2021-02-10T04:35:53.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: 2) 1378 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:53.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: cluster 2021-02-10T04:35:53.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: 2021-02-10T04:35:52.419210 2021-02-10T04:35:53.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1364 : cluster [WRN] 2021-02-10T04:35:53.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: Health check update: 14 slow ops, oldest one blocked for 381 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:35:53.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:35:53.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: 04:35:52.432027+0000 mon.a (mon 2021-02-10T04:35:53.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: .0) 1365 : cluster [DBG] osdmap e535: 8 total, 8 up, 8 in 2021-02-10T04:35:53.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:35:53.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: -02-10T04:35:53. 2021-02-10T04:35:53.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: 020530+0000 osd.4 (osd.4) 751 2021-02-10T04:35:53.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:53.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:35:53.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: 04:35:53.020538+0000 osd.4 (osd 2021-02-10T04:35:53.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: .4) 752 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:53.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: cluster 2021-02-10T04:35:53.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: 2021-02-10T04:35:53. 2021-02-10T04:35:53.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: 020543+0000 osd.4 (osd.4) 753 : cluster 2021-02-10T04:35:53.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:53.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:35:53.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: 04:35:53.020546+0000 osd.4 (osd 2021-02-10T04:35:53.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[29319]: .4) 754 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:53.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: cluster 2021-02-10T04:35:51. 2021-02-10T04:35:53.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: 837690+0000 osd.2 (osd.2) 1369 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:53.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: cluster 2021-02-10T04:35:51. 2021-02-10T04:35:53.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: 837700+0000 osd.2 (osd.2) 1370 : cluster [WRN] 2021-02-10T04:35:53.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:53.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: cluster 2021-02-10T04:35:53.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: 2021-02-10T04:35 2021-02-10T04:35:53.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: :51.837707+0000 osd.2 (osd.2 2021-02-10T04:35:53.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: ) 1371 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:53.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: cluster 2021-02-10T04:35:53.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: 2021-02-10T04:35:51 2021-02-10T04:35:53.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: .837713+0000 osd.2 (osd.2) 1372 2021-02-10T04:35:53.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:53.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:35:53.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: -02-10T04:35:51. 2021-02-10T04:35:53.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: 837718+0000 osd.2 (osd.2) 1373 : 2021-02-10T04:35:53.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:53.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:35:53.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: -02-10T04:35:51. 2021-02-10T04:35:53.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: 837724+0000 osd.2 (osd.2) 1374 : 2021-02-10T04:35:53.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:53.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: cluster 2021-02-10T04:35:53.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: 2021-02-10T04:35:51.837730+0000 osd.2 (osd.2) 1375 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:53.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: 2021-02-10T04:35:53.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:35:53.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: -10T04:35:51.837736+0000 2021-02-10T04:35:53.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: osd.2 (osd.2) 1376 : cluster [WRN] 2021-02-10T04:35:53.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:53.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:35:53.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: -10T04:35:51.837742+0000 2021-02-10T04:35:53.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: osd.2 (osd.2) 1377 : cluster [WRN] 2021-02-10T04:35:53.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:53.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:35:53.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: -10T04:35:51.837747+0000 2021-02-10T04:35:53.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: osd.2 (osd.2) 1378 : cluster [WRN] 2021-02-10T04:35:53.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:53.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:35:53.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: 04:35:52.419210+0000 mon.a (mon 2021-02-10T04:35:53.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: .0) 1364 : cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 381 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:35:53.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:53.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: 02-10T04:35:52.432027+0000 mon.a 2021-02-10T04:35:53.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: (mon.0) 1365 : cluster [DBG] osdmap e535: 8 total, 8 up, 8 in 2021-02-10T04:35:53.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:35:53.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: -02-10T04:35:53.020530+0000 2021-02-10T04:35:53.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: osd.4 (osd.4) 751 : cluster [WRN] 2021-02-10T04:35:53.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:53.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: cluster 2021-02-10T04:35:53.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: 2021-02-10T04:35:53. 2021-02-10T04:35:53.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: 020538+0000 osd.4 (osd.4) 752 2021-02-10T04:35:53.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:53.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: cluster 2021-02-10T04:35:53.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: 2021-02-10T04:35:53 2021-02-10T04:35:53.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: .020543+0000 osd.4 (osd.4) 753 2021-02-10T04:35:53.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:53.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: cluster 2021-02-10T04:35:53.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: 2021-02-10T04:35: 2021-02-10T04:35:53.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: 53.020546+0000 osd.4 (osd.4) 2021-02-10T04:35:53.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[35961]: 754 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: cluster 2021-02-10T04:35:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: 2021-02-10 2021-02-10T04:35:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: T04:35:51.837690+0000 osd.2 (osd.2) 1369 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:35:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: 02-10T04:35:51.837700+0000 osd.2 (osd.2 2021-02-10T04:35:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: ) 1370 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: cluster 2021-02-10T04:35 2021-02-10T04:35:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: :51.837707+0000 osd.2 (osd.2) 1371 : cluster [WRN] 2021-02-10T04:35:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: cluster 2021-02-10T04:35:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: 2021-02-10T04:35:51.837713+0000 2021-02-10T04:35:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: osd.2 (osd.2) 1372 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: 2021-02-10T04:35:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: cluster 2021-02-10T04:35:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: 2021-02-10T04:35:51 2021-02-10T04:35:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: .837718+0000 osd.2 (osd.2) 1373 : 2021-02-10T04:35:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: -10T04:35:51.837724 2021-02-10T04:35:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1374 : cluster 2021-02-10T04:35:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:35:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: 02-10T04:35: 2021-02-10T04:35:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: 51.837730+0000 osd.2 (osd. 2021-02-10T04:35:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: 2) 1375 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: cluster 2021-02-10T04:35:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:35:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: 35:51.837736+0000 osd.2 (osd.2) 1376 : cluster 2021-02-10T04:35:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: cluster 2021-02-10T04:35:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: 2021-02-10T04:35:51.837742+0000 osd.2 (osd.2) 1377 : cluster [WRN] 2021-02-10T04:35:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: cluster 2021-02-10T04:35:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: 2021-02-10T04:35:51. 2021-02-10T04:35:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: 837747+0000 osd.2 (osd.2) 1378 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: -10T04:35:52.419210+0000 mon.a (mon.0) 2021-02-10T04:35:53.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: 1364 : cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 381 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:35:53.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: cluster 2021-02-10T04:35:52. 2021-02-10T04:35:53.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: 432027+0000 mon.a (mon.0) 1365 : cluster [DBG] osdmap e535: 8 total, 8 up, 8 in 2021-02-10T04:35:53.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:35:53.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: 10T04:35:53.020530+0000 osd.4 ( 2021-02-10T04:35:53.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: osd.4) 751 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:53.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:35:53.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: -02-10T04:35:53. 2021-02-10T04:35:53.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: 020538+0000 osd.4 (osd.4) 752 : cluster 2021-02-10T04:35:53.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:53.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:53.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: -10T04:35:53.020543 2021-02-10T04:35:53.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 753 : cluster 2021-02-10T04:35:53.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:53.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:35:53.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: 10T04:35:53.020546+0000 2021-02-10T04:35:53.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: osd.4 (osd.4) 754 : cluster [WRN] 2021-02-10T04:35:53.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:53 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:54.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:53 smithi085 conmon[49106]: debug 2021-02-10T04:35:53.865+0000 7f77a3e94700 -1 osd.2 536 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:35:54.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[33677]: debug 2021-02-10T04:35:54.015+0000 7fbec19a1700 -1 osd.4 536 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:35:54.463 INFO:tasks.rados.rados.0.smithi085.stdout:2077: done (13 left) 2021-02-10T04:35:54.463 INFO:tasks.rados.rados.0.smithi085.stdout:2078: rollback oid 46 current snap is 230 2021-02-10T04:35:54.463 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 46 to 148 2021-02-10T04:35:54.463 INFO:tasks.rados.rados.0.smithi085.stdout:2072: finishing write tid 2 to smithi08575397-7 2021-02-10T04:35:54.463 INFO:tasks.rados.rados.0.smithi085.stdout:2072: finishing write tid 3 to smithi08575397-7 2021-02-10T04:35:54.463 INFO:tasks.rados.rados.0.smithi085.stdout:2072: finishing write tid 4 to smithi08575397-7 2021-02-10T04:35:54.463 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 751 (ObjNum 727 snap 230 seq_num 727) dirty exists 2021-02-10T04:35:54.463 INFO:tasks.rados.rados.0.smithi085.stdout:2072: left oid 7 (ObjNum 727 snap 230 seq_num 727) 2021-02-10T04:35:54.463 INFO:tasks.rados.rados.0.smithi085.stdout:2075: finishing copy_from to smithi08575397-20 2021-02-10T04:35:54.464 INFO:tasks.rados.rados.0.smithi085.stdout:2075: got expected ENOENT (src dne) 2021-02-10T04:35:54.464 INFO:tasks.rados.rados.0.smithi085.stdout:2075: finishing copy_from racing read to smithi08575397-20 2021-02-10T04:35:54.464 INFO:tasks.rados.rados.0.smithi085.stdout:2074: expect (ObjNum 570 snap 175 seq_num 570) 2021-02-10T04:35:54.739 INFO:tasks.rados.rados.0.smithi085.stdout:2072: done (13 left) 2021-02-10T04:35:54.740 INFO:tasks.rados.rados.0.smithi085.stdout:2074: done (12 left) 2021-02-10T04:35:54.740 INFO:tasks.rados.rados.0.smithi085.stdout:2075: done (11 left) 2021-02-10T04:35:54.740 INFO:tasks.rados.rados.0.smithi085.stdout:2079: snap_remove snap 212 2021-02-10T04:35:54.771 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: cluster 2021-02-10T04:35:52.852325+0000 osd.2 2021-02-10T04:35:54.771 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: (osd.2) 1379 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:54.772 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: cluster 2021-02-10T04:35:52.852331+0000 osd.2 ( 2021-02-10T04:35:54.772 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: osd.2) 1380 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:54.772 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: cluster 2021-02-10T04:35:54.772 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: 2021-02-10T04:35: 2021-02-10T04:35:54.772 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: 52.852337+0000 osd.2 (osd.2) 1381 : cluster 2021-02-10T04:35:54.772 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:54.772 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:35:54.772 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: T04:35:52.852345+0000 osd.2 (osd.2 2021-02-10T04:35:54.772 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: ) 1382 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:54.772 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: cluster 2021-02-10T04:35:54.772 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: 2021-02-10T04:35:52 2021-02-10T04:35:54.773 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: .852351+0000 osd.2 (osd.2) 1383 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:54.773 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:35:54.773 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: -02-10T04:35:52.852356+0000 osd.2 (osd.2) 1384 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:54.773 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: cluster 2021-02-10T04:35:52.852363+0000 osd.2 (osd.2) 1385 : cluster 2021-02-10T04:35:54.773 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:54.773 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:35:54.773 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: 10T04:35:52.852370+0000 osd.2 (osd.2) 1386 : cluster 2021-02-10T04:35:54.773 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:54.773 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: cluster 2021-02-10T04:35:52. 2021-02-10T04:35:54.773 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: 852376+0000 osd.2 (osd.2) 1387 : cluster [WRN] 2021-02-10T04:35:54.773 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:54.773 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: cluster 2021-02-10T04:35:52 2021-02-10T04:35:54.773 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: .852382+0000 osd.2 (osd.2) 1388 : cluster 2021-02-10T04:35:54.774 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:54.775 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: cluster 2021-02-10T04:35: 2021-02-10T04:35:54.775 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: 53.431221+0000 mon.a (mon.0) 1366 : 2021-02-10T04:35:54.775 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: cluster [DBG] osdmap e536: 8 total, 8 up, 8 in 2021-02-10T04:35:54.775 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:35:54.775 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: 35:53.866662+0000 osd.2 (osd.2) 1389 : cluster [WRN] 2021-02-10T04:35:54.775 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:54.775 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:35:54.775 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: -10T04:35:53.866671+0000 osd.2 (osd.2) 1390 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:54.776 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:35:54.776 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: 35:53.866678+0000 osd.2 (osd.2) 2021-02-10T04:35:54.776 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: 1391 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:54.776 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:35:54.776 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: 35:53.866684+0000 osd.2 (osd.2 2021-02-10T04:35:54.776 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: ) 1392 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:54.776 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:35:54.776 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: 04:35:53.866690+0000 osd.2 (osd 2021-02-10T04:35:54.776 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: .2) 1393 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:54.776 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:35:54.776 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: 10T04:35:53.866696+0000 osd.2 (osd.2) 1394 2021-02-10T04:35:54.776 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:54.776 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:35:54.777 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: -10T04:35:53.866704+0000 osd.2 (osd.2) 1395 2021-02-10T04:35:54.777 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:54.777 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: cluster 2021-02-10T04:35:53. 2021-02-10T04:35:54.777 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: 866712+0000 osd.2 (osd.2) 1396 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:54.777 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:35:54.777 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: :35:53.866716+0000 osd.2 (osd.2) 1397 : cluster 2021-02-10T04:35:54.777 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:54.777 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: cluster 2021-02-10T04:35 2021-02-10T04:35:54.777 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: :53.866720+0000 osd.2 (osd 2021-02-10T04:35:54.777 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: .2) 1398 : cluster [WRN] 2021-02-10T04:35:54.777 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:54.777 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:35:54.777 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: 10T04:35:54.016940 2021-02-10T04:35:54.778 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 755 2021-02-10T04:35:54.778 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:54.778 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:35:54.778 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: 10T04:35:54.016946 2021-02-10T04:35:54.778 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2021-02-10T04:35:54.778 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: 756 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:54.778 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: cluster 2021-02-10T04:35:54.778 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: 2021-02-10T04:35 2021-02-10T04:35:54.778 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: :54.016950+0000 osd.4 ( 2021-02-10T04:35:54.778 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: osd. 2021-02-10T04:35:54.778 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: 4) 757 : cluster [WRN] 2021-02-10T04:35:54.778 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:54.779 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:35:54.779 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: :35:54.016954+0000 osd.4 (osd.4) 758 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:54.779 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:35:54.779 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: -10T04:35:54.108630+0000 mgr.y ( 2021-02-10T04:35:54.779 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[29319]: mgr.14140) 611 : cluster [DBG] pgmap v954: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 157 active+clean; 412 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 1.5 MiB/s wr, 11 op/s 2021-02-10T04:35:54.779 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: cluster 2021-02-10T04:35:52.852325+0000 osd.2 ( 2021-02-10T04:35:54.779 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: osd.2) 1379 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:54.779 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: cluster 2021-02-10T04:35:52.852331+0000 osd.2 (osd.2) 1380 : 2021-02-10T04:35:54.779 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:54.779 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: cluster 2021-02-10T04:35:54.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: 2021-02-10T04:35:52.852337 2021-02-10T04:35:54.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1381 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:54.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: cluster 2021-02-10T04:35:54.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:35:54.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: 04:35:52.852345+0000 osd.2 (osd.2) 2021-02-10T04:35:54.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: 1382 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:54.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:35:54.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: -10T04:35:52.852351+0000 osd.2 (osd 2021-02-10T04:35:54.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: .2) 1383 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:54.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:35:54.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: -10T04:35:52.852356+0000 osd.2 (osd. 2021-02-10T04:35:54.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: 2) 1384 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:54.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: cluster 2021-02-10T04:35:54.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: 2021-02-10T04:35: 2021-02-10T04:35:54.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: 52.852363+0000 osd.2 (osd.2) 1385 : cluster 2021-02-10T04:35:54.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:54.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: cluster 2021-02-10T04:35:54.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: 2021-02-10T04:35: 2021-02-10T04:35:54.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: 52.852370+0000 osd.2 (osd.2) 1386 : cluster 2021-02-10T04:35:54.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:54.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: cluster 2021-02-10T04:35:54.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:35:54.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: 35:52.852376+0000 osd.2 (osd.2) 1387 2021-02-10T04:35:54.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:54.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:35:54.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: :35:52.852382+0000 osd.2 (osd.2) 1388 2021-02-10T04:35:54.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:54.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:35:54.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: 35:53.431221+0000 mon.a (mon.0) 1366 : cluster 2021-02-10T04:35:54.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: [DBG] osdmap e536: 8 total, 8 up, 8 in 2021-02-10T04:35:54.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: cluster 2021-02-10T04:35:53.866662+0000 2021-02-10T04:35:54.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: osd.2 (osd.2) 1389 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:54.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: cluster 2021-02-10T04:35:54.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: 2021-02-10T04:35:53.866671+0000 2021-02-10T04:35:54.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: osd.2 (osd.2) 1390 : cluster 2021-02-10T04:35:54.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:54.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: cluster 2021-02-10T04:35 2021-02-10T04:35:54.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: :53.866678+0000 osd.2 (osd.2) 1391 2021-02-10T04:35:54.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:54.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: cluster 2021-02-10T04:35:54.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:35:54.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: 35:53.866684+0000 osd.2 (osd.2) 1392 2021-02-10T04:35:54.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:54.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:35:54.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: 04:35:53.866690+0000 osd.2 (osd.2) 1393 2021-02-10T04:35:54.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:54.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:35:54.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: :35:53.866696+0000 osd.2 (osd.2) 1394 : 2021-02-10T04:35:54.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:54.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: cluster 2021-02-10T04:35: 2021-02-10T04:35:54.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: 53.866704+0000 osd.2 (osd.2) 1395 2021-02-10T04:35:54.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:54.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:35:54.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: T04:35:53.866712+0000 osd.2 (osd.2 2021-02-10T04:35:54.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: ) 1396 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:54.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:35:54.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: :35:53.866716+0000 osd.2 (osd.2) 1397 2021-02-10T04:35:54.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:54.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: cluster 2021-02-10T04:35:53. 2021-02-10T04:35:54.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: 866720+0000 osd.2 (osd.2) 1398 : cluster 2021-02-10T04:35:54.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:54.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: cluster 2021-02-10T04:35: 2021-02-10T04:35:54.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: 54.016940+0000 osd.4 (osd.4) 755 : 2021-02-10T04:35:54.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:54.785 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: cluster 2021-02-10T04:35:54. 2021-02-10T04:35:54.785 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: 016946+0000 osd.4 (osd.4) 756 : cluster 2021-02-10T04:35:54.785 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:54.785 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:35:54.785 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: T04:35:54.016950+0000 osd.4 (osd. 2021-02-10T04:35:54.785 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: 4) 757 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:54.785 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:35:54.785 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: -02-10T04:35:54.016954 2021-02-10T04:35:54.785 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 758 : cluster [WRN] 2021-02-10T04:35:54.785 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:54.785 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: cluster 2021-02-10T04:35:54 2021-02-10T04:35:54.785 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: .108630+0000 mgr.y (mgr.14140) 611 : cluster 2021-02-10T04:35:54.785 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[35961]: [DBG] pgmap v954: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 157 active+clean; 412 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 1.5 MiB/s wr, 11 op/s 2021-02-10T04:35:54.887 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:35:54.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: cluster 2021-02-10T04:35:54.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: 2021-02-10T04:35:52. 2021-02-10T04:35:54.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: 852325+0000 osd.2 (osd.2) 1379 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:54.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:54.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: -10T04:35:52.852331+0000 osd.2 (osd.2) 1380 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:54.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:35:54.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: -02-10T04:35:52.852337+0000 osd.2 ( 2021-02-10T04:35:54.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: osd.2) 1381 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:54.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:35:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: T04:35:52.852345+0000 osd.2 (osd.2) 1382 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:35:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: 35:52.852351+0000 osd.2 (osd.2) 1383 : cluster [WRN] 2021-02-10T04:35:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: -10T04:35:52.852356+0000 osd.2 (osd.2) 1384 : cluster 2021-02-10T04:35:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: cluster 2021-02-10T04:35:52 2021-02-10T04:35:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: .852363+0000 osd.2 (osd.2) 1385 : cluster 2021-02-10T04:35:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: cluster 2021-02-10T04:35:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: 2021-02-10T04:35:52.852370+0000 osd.2 (osd. 2021-02-10T04:35:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: 2) 1386 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:54.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:35:54.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: T04:35:52.852376+0000 osd.2 (osd 2021-02-10T04:35:54.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: .2) 1387 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:54.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:54.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: -10T04:35:52.852382+0000 osd.2 ( 2021-02-10T04:35:54.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: osd.2) 1388 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:54.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:35:54.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: 10T04:35:53.431221+0000 mon.a (mon. 2021-02-10T04:35:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: 0) 1366 : cluster [DBG] osdmap e536: 8 total, 8 up, 8 in 2021-02-10T04:35:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: -10T04:35:53.866662+0000 osd.2 (osd.2) 1389 : 2021-02-10T04:35:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: cluster 2021-02-10T04:35:53.866671+0000 2021-02-10T04:35:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: osd.2 (osd.2) 1390 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:35:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: 35:53.866678+0000 osd.2 (osd.2) 1391 2021-02-10T04:35:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:35:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: 10T04:35:53.866684+0000 osd.2 (osd 2021-02-10T04:35:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: .2) 1392 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:35:54.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: 35:53.866690+0000 osd.2 (osd.2) 1393 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:54.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: cluster 2021-02-10T04:35:53.866696+0000 osd.2 2021-02-10T04:35:54.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: (osd.2) 1394 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:54.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:35:54.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: 04:35:53.866704+0000 osd.2 (osd.2) 1395 2021-02-10T04:35:54.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:54.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:35:54.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: 02-10T04:35:53.866712+0000 osd.2 (osd.2) 1396 2021-02-10T04:35:54.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:54.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: cluster 2021-02-10T04:35: 2021-02-10T04:35:54.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: 53.866716+0000 osd.2 (osd.2) 1397 : 2021-02-10T04:35:54.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:54.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: cluster 2021-02-10T04:35:53.866720+0000 2021-02-10T04:35:54.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: osd.2 (osd.2) 1398 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:54.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:35:54.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: -02-10T04:35:54.016940+0000 2021-02-10T04:35:54.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: osd.4 (osd.4) 755 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:54.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: 2021-02-10T04:35:54.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:35:54.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: 02-10T04:35:54.016946+0000 osd.4 (osd.4) 2021-02-10T04:35:54.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: 756 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:54.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:35:54.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: T04:35:54.016950+0000 osd.4 (osd. 2021-02-10T04:35:54.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: 4) 757 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:54.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:35:54.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: 10T04:35:54.016954+0000 osd.4 (osd.4) 758 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:54.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: cluster 2021-02-10T04:35:54.108630+0000 mgr.y 2021-02-10T04:35:54.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:54 smithi164 conmon[29288]: (mgr.14140) 611 : cluster [DBG] pgmap v954: 168 pgs: 2 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 157 active+clean; 412 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 1.5 MiB/s wr, 11 op/s 2021-02-10T04:35:54.924 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:35:55.049 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:35:55.059 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:54 smithi085 conmon[49106]: debug 2021-02-10T04:35:54.906+0000 7f77a3e94700 -1 osd.2 537 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:35:55.156 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:35:55.181 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[37263]: debug 2021-02-10T04:35:55.012+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:35:55.181 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[33677]: debug 2021-02-10T04:35:55.058+0000 7fbec19a1700 -1 osd.4 537 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:35:55.183 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:35:55.291 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:35:55.398 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:35:55.433 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:35:55.455 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[37263]: debug 2021-02-10T04:35:55.455 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[37263]: 2021-02-10T04:35:55.269+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:35:55.455 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: cluster 2021-02-10T04:35:55.456 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:35:55.456 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: 04:35:54.454551+0000 mon.a (mon.0) 1367 : cluster [DBG] osdmap e537: 8 total, 8 up, 8 in 2021-02-10T04:35:55.456 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: cluster 2021-02-10T04:35:54.907974 2021-02-10T04:35:55.456 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1399 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:55.456 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: cluster 2021-02-10T04:35: 2021-02-10T04:35:55.456 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: 54.907981+0000 osd.2 (osd.2) 1400 : cluster 2021-02-10T04:35:55.456 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:55.456 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: cluster 2021-02-10T04:35:54 2021-02-10T04:35:55.456 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: .907984+0000 osd.2 (osd.2) 1401 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:55.456 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: cluster 2021-02-10T04:35:54.907988+0000 osd.2 (osd 2021-02-10T04:35:55.456 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: .2) 1402 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:55.456 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:35:55.457 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: 35:54.907991+0000 osd.2 (osd.2) 2021-02-10T04:35:55.457 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: 1403 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:55.457 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:35:55.457 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: T04:35:54.907994+0000 osd.2 (osd 2021-02-10T04:35:55.457 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: .2) 1404 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:55.457 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:55.457 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: -10T04:35:54.907997+0000 osd.2 2021-02-10T04:35:55.457 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: (osd.2) 1405 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:55.457 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: 2021-02-10T04:35:55.457 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:35:55.457 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: -02-10T04:35:54.908002+0000 osd.2 (osd 2021-02-10T04:35:55.457 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: .2) 1406 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:55.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: cluster 2021-02-10T04:35:54.908005+0000 osd.2 (osd.2) 1407 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:55.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: cluster 2021-02-10T04:35:54 2021-02-10T04:35:55.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: .908008+0000 osd.2 (osd.2) 1408 : cluster 2021-02-10T04:35:55.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:55.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: cluster 2021-02-10T04:35: 2021-02-10T04:35:55.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: 55.059895+0000 osd.4 (osd.4) 759 : cluster 2021-02-10T04:35:55.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:55.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: cluster 2021-02-10T04:35 2021-02-10T04:35:55.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: :55.059905+0000 osd.4 (osd.4) 760 2021-02-10T04:35:55.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:55.458 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: cluster 2021-02-10T04:35: 2021-02-10T04:35:55.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: 55.059912+0000 osd.4 (osd.4) 761 : cluster 2021-02-10T04:35:55.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:55.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: cluster 2021-02-10T04:35:55 2021-02-10T04:35:55.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: .059919+0000 osd.4 (osd.4) 762 : cluster 2021-02-10T04:35:55.459 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:55 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:55.467 INFO:tasks.rados.rados.0.smithi085.stdout:2079: done (11 left) 2021-02-10T04:35:55.467 INFO:tasks.rados.rados.0.smithi085.stdout:2080: snap_remove snap 220 2021-02-10T04:35:55.573 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:35:55.574 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [ 2021-02-10T04:35:55.574 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.574 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156)", 2021-02-10T04:35:55.574 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:32.164063+0000", 2021-02-10T04:35:55.574 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 383.40891845599998, 2021-02-10T04:35:55.574 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 383.40894247799997, 2021-02-10T04:35:55.575 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:35:55.575 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:35:55.575 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:35:55.575 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:35:55.575 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:35:55.575 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 910 2021-02-10T04:35:55.575 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.575 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:35:55.575 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.575 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:35:55.575 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164063+0000", 2021-02-10T04:35:55.575 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:35:55.575 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.576 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.576 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:35:55.576 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164063+0000", 2021-02-10T04:35:55.576 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.8188e-05 2021-02-10T04:35:55.576 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.576 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.576 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:35:55.576 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164111+0000", 2021-02-10T04:35:55.576 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0791000000000001e-05 2021-02-10T04:35:55.576 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.576 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.576 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:35:55.576 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164122+0000", 2021-02-10T04:35:55.577 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5682999999999999e-05 2021-02-10T04:35:55.577 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.577 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.577 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:35:55.577 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164138+0000", 2021-02-10T04:35:55.577 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0522e-05 2021-02-10T04:35:55.577 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.577 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.577 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:35:55.577 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164148+0000", 2021-02-10T04:35:55.577 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.9583999999999997e-05 2021-02-10T04:35:55.577 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.577 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.577 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:35:55.578 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164208+0000", 2021-02-10T04:35:55.578 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00013592500000000001 2021-02-10T04:35:55.578 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.578 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.578 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:35:55.578 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164344+0000", 2021-02-10T04:35:55.578 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00028069299999999999 2021-02-10T04:35:55.578 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:35:55.578 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:35:55.578 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:35:55.578 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.578 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.579 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156)", 2021-02-10T04:35:55.579 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:32.164227+0000", 2021-02-10T04:35:55.579 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 383.40875402500001, 2021-02-10T04:35:55.579 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 383.40885477699999, 2021-02-10T04:35:55.579 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:35:55.579 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "delayed", 2021-02-10T04:35:55.579 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:35:55.579 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:35:55.579 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:35:55.579 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 911 2021-02-10T04:35:55.579 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.579 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:35:55.580 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.580 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:35:55.580 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164227+0000", 2021-02-10T04:35:55.580 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:35:55.580 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.580 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.580 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:35:55.580 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164227+0000", 2021-02-10T04:35:55.580 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.3152999999999997e-05 2021-02-10T04:35:55.580 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.580 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.580 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:35:55.580 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164291+0000", 2021-02-10T04:35:55.580 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.8859999999999993e-06 2021-02-10T04:35:55.581 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.581 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.581 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:35:55.581 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164299+0000", 2021-02-10T04:35:55.581 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5695000000000001e-05 2021-02-10T04:35:55.581 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.581 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.581 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:35:55.581 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164315+0000", 2021-02-10T04:35:55.581 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.3790000000000003e-06 2021-02-10T04:35:55.581 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.581 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.581 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:35:55.581 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164322+0000", 2021-02-10T04:35:55.582 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00017179399999999999 2021-02-10T04:35:55.582 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.582 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.582 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:35:55.582 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164493+0000", 2021-02-10T04:35:55.582 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.9685000000000002e-05 2021-02-10T04:35:55.582 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.582 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.582 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:35:55.582 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:32.164553+0000", 2021-02-10T04:35:55.582 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00032559199999999998 2021-02-10T04:35:55.582 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:35:55.583 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:35:55.583 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:35:55.583 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.583 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.583 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153)", 2021-02-10T04:35:55.583 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:29.061753+0000", 2021-02-10T04:35:55.583 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 386.51122838600003, 2021-02-10T04:35:55.583 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 386.51137280699999, 2021-02-10T04:35:55.583 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:35:55.583 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:35:55.583 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:35:55.583 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:35:55.583 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:35:55.583 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 865 2021-02-10T04:35:55.583 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.584 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:35:55.584 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.584 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:35:55.584 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061753+0000", 2021-02-10T04:35:55.584 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:35:55.584 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.584 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.584 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:35:55.584 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061753+0000", 2021-02-10T04:35:55.584 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.3646999999999997e-05 2021-02-10T04:35:55.584 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.584 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.584 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:35:55.585 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061797+0000", 2021-02-10T04:35:55.585 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.4220000000000003e-06 2021-02-10T04:35:55.585 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.585 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.585 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:35:55.585 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061804+0000", 2021-02-10T04:35:55.585 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.9582000000000002e-05 2021-02-10T04:35:55.585 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.585 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.585 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:35:55.585 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061844+0000", 2021-02-10T04:35:55.585 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.9789999999999996e-06 2021-02-10T04:35:55.585 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.586 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.586 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:35:55.586 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061851+0000", 2021-02-10T04:35:55.586 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00012051 2021-02-10T04:35:55.586 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.586 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.586 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:35:55.586 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061971+0000", 2021-02-10T04:35:55.586 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.0452999999999998e-05 2021-02-10T04:35:55.586 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.586 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.586 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:35:55.586 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.062042+0000", 2021-02-10T04:35:55.587 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00028859300000000002 2021-02-10T04:35:55.587 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:35:55.587 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:35:55.587 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:35:55.587 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.587 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.587 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153)", 2021-02-10T04:35:55.587 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:29.061891+0000", 2021-02-10T04:35:55.587 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 386.51109064399998, 2021-02-10T04:35:55.587 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 386.51127577900002, 2021-02-10T04:35:55.587 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:35:55.587 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "delayed", 2021-02-10T04:35:55.587 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:35:55.587 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:35:55.588 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:35:55.588 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 866 2021-02-10T04:35:55.588 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.588 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:35:55.588 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.588 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:35:55.588 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061891+0000", 2021-02-10T04:35:55.588 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:35:55.588 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.588 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.588 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:35:55.588 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061891+0000", 2021-02-10T04:35:55.588 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1696000000000003e-05 2021-02-10T04:35:55.588 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.589 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.589 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:35:55.589 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061923+0000", 2021-02-10T04:35:55.589 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.7699999999999998e-06 2021-02-10T04:35:55.589 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.589 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.589 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:35:55.589 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061928+0000", 2021-02-10T04:35:55.589 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1918e-05 2021-02-10T04:35:55.589 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.589 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.589 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:35:55.589 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061940+0000", 2021-02-10T04:35:55.590 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.8640000000000004e-06 2021-02-10T04:35:55.590 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.590 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.590 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:35:55.590 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.061945+0000", 2021-02-10T04:35:55.590 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00022406100000000001 2021-02-10T04:35:55.590 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.590 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.590 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:35:55.590 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.062169+0000", 2021-02-10T04:35:55.590 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.0158999999999999e-05 2021-02-10T04:35:55.590 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:35:55.590 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:35:55.590 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:35:55.591 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:29.062219+0000", 2021-02-10T04:35:55.591 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000328468 2021-02-10T04:35:55.591 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:35:55.591 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:35:55.591 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:35:55.591 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:35:55.591 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2021-02-10T04:35:55.591 INFO:teuthology.orchestra.run.smithi164.stdout: "complaint_time": 30, 2021-02-10T04:35:55.591 INFO:teuthology.orchestra.run.smithi164.stdout: "num_blocked_ops": 4 2021-02-10T04:35:55.591 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:35:55.691 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[45443]: debug 2021-02-10T04:35:55.519+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:35:55.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[29319]: cluster 2021-02-10T04:35:54.454551+0000 2021-02-10T04:35:55.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[29319]: mon.a (mon.0) 1367 : cluster [DBG] osdmap e537: 8 total, 8 up, 8 in 2021-02-10T04:35:55.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[29319]: cluster 2021-02-10T04:35:54.907974+0000 osd.2 (osd.2) 1399 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:55.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[29319]: cluster 2021-02-10T04:35:54.907981+0000 osd.2 (osd.2) 1400 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:55.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[29319]: cluster 2021-02-10T04:35:54.907984+0000 osd.2 (osd.2) 1401 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:55.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[29319]: cluster 2021-02-10T04:35:54.907988+0000 osd.2 (osd.2) 1402 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:55.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[29319]: cluster 2021-02-10T04:35:54.907991+0000 osd.2 (osd.2) 1403 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:55.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[29319]: cluster 2021-02-10T04:35:54.907994+0000 osd.2 (osd.2) 1404 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:55.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[29319]: cluster 2021-02-10T04:35:54.907997+0000 osd.2 (osd.2) 1405 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:55.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[29319]: cluster 2021-02-10T04:35:54.908002+0000 osd.2 (osd.2) 1406 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:55.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[29319]: cluster 2021-02-10T04:35:54.908005+0000 osd.2 (osd.2) 1407 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:55.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[29319]: cluster 2021-02-10T04:35:54.908008+0000 osd.2 (osd.2) 1408 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:55.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[29319]: cluster 2021-02-10T04:35:55.059895+0000 osd.4 (osd.4) 759 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:55.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[29319]: cluster 2021-02-10T04:35:55.059905+0000 osd.4 (osd.4) 760 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:55.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[29319]: cluster 2021-02-10T04:35:55.059912+0000 osd.4 (osd.4) 761 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:55.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[29319]: cluster 2021-02-10T04:35:55.059919+0000 osd.4 (osd.4) 762 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:55.693 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:35:55.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: cluster 2021-02-10T04:35:54.454551+0000 2021-02-10T04:35:55.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: mon.a (mon.0) 1367 : cluster [DBG] osdmap e537: 8 total, 8 up, 8 in 2021-02-10T04:35:55.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: cluster 2021-02-10T04:35:54.907974+0000 osd.2 (osd.2) 1399 : cluster 2021-02-10T04:35:55.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:55.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:35:55.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: 35:54.907981+0000 osd.2 (osd.2) 1400 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:55.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: cluster 2021-02-10T04:35:54.907984 2021-02-10T04:35:55.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1401 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:55.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: 2021-02-10T04:35:55.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:35:55.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: 35:54.907988+0000 osd.2 (osd.2) 1402 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:55.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: cluster 2021-02-10T04:35:54.907991 2021-02-10T04:35:55.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1403 : cluster [WRN] 2021-02-10T04:35:55.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:55.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: cluster 2021-02-10T04:35:55.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: 2021-02-10T04:35:54.907994+0000 osd.2 (osd.2) 1404 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:55.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: cluster 2021-02-10T04:35:54.907997+0000 osd.2 (osd.2) 1405 : cluster 2021-02-10T04:35:55.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:55.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:35:55.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: 04:35:54.908002+0000 osd.2 (osd.2) 1406 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:55.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: 2021-02-10T04:35:55.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:35:55.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: 10T04:35:54.908005+0000 osd.2 (osd.2) 1407 : cluster 2021-02-10T04:35:55.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:55.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: cluster 2021-02-10T04:35:55.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: 2021-02-10T04:35:54.908008+0000 osd.2 (osd.2) 1408 2021-02-10T04:35:55.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:55.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:35:55.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: -10T04:35:55.059895+0000 osd.4 (osd.4) 759 : cluster 2021-02-10T04:35:55.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:55.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: cluster 2021-02-10T04:35:55. 2021-02-10T04:35:55.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: 059905+0000 osd.4 (osd.4) 760 : cluster [WRN] 2021-02-10T04:35:55.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:55.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: cluster 2021-02-10T04:35:55.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: 2021-02-10T04:35:55.059912+0000 osd.4 ( 2021-02-10T04:35:55.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: osd.4) 761 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:55.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: cluster 2021-02-10T04:35:55.059919 2021-02-10T04:35:55.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 762 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:55.800 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:35:55.833 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:35:55.959 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:35:55.962 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[49106]: debug 2021-02-10T04:35:55.962 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[49106]: 2021-02-10T04:35:55.880+0000 7f77a3e94700 -1 osd.2 538 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:35:55.963 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[45443]: debug 2021-02-10T04:35:55.963 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[45443]: 2021-02-10T04:35:55.921+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:35:55.963 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:35:55 smithi085 conmon[45443]: 2021-02-10T04:35:56.064 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:35:56.097 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:35:56.219 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:35:56.223 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[45443]: debug 2021-02-10T04:35:56.183+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:35:56.323 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:35:56.361 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:35:56.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[33677]: debug 2021-02-10T04:35:56.104+0000 7fbec19a1700 -1 osd.4 538 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:35:56.474 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: cluster 2021-02-10T04:35:55.465286 2021-02-10T04:35:56.474 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1368 : cluster [DBG] osdmap e538: 8 total, 8 up, 8 in 2021-02-10T04:35:56.474 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:35:56.475 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: 10T04:35:55.882148+0000 osd.2 ( 2021-02-10T04:35:56.475 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: osd.2) 1409 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:56.475 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: cluster 2021-02-10T04:35:55.882153 2021-02-10T04:35:56.475 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1410 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:56.475 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: cluster 2021-02-10T04:35:55. 2021-02-10T04:35:56.475 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: 882157+0000 osd.2 (osd.2) 1411 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:56.475 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: cluster 2021-02-10T04:35:55 2021-02-10T04:35:56.475 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: .882160+0000 osd.2 (osd.2) 1412 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:56.475 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: cluster 2021-02-10T04:35 2021-02-10T04:35:56.476 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[49106]: debug 2021-02-10T04:35:56.476 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[49106]: 2021-02-10T04:35:56.445+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:35:56.476 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[49106]: 2021-02-10T04:35:56.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: cluster 2021-02-10T04:35:56.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: 2021-02-10T04:35:56.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: 2021 2021-02-10T04:35:56.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: - 2021-02-10T04:35:56.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: 02 2021-02-10T04:35:56.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: - 2021-02-10T04:35:56.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: 10 2021-02-10T04:35:56.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: T 2021-02-10T04:35:56.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: 04 2021-02-10T04:35:56.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: :35:55.465286+0000 mon.a ( 2021-02-10T04:35:56.477 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: mon.0) 1368 : cluster [DBG] osdmap e538: 8 total, 8 up, 8 in 2021-02-10T04:35:56.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: cluster 2021-02-10T04:35:56.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: 2021-02-10T04:35:56.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: 2021 2021-02-10T04:35:56.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: - 2021-02-10T04:35:56.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: 02 2021-02-10T04:35:56.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: - 2021-02-10T04:35:56.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: 10 2021-02-10T04:35:56.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: T 2021-02-10T04:35:56.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: 04 2021-02-10T04:35:56.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: : 2021-02-10T04:35:56.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: 35 2021-02-10T04:35:56.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: : 2021-02-10T04:35:56.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: 55 2021-02-10T04:35:56.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: .882148+0000 osd.2 (osd. 2021-02-10T04:35:56.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: 2) 1409 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:56.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:35:56.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: -02-10T04: 2021-02-10T04:35:56.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: 35:55.882153+0000 osd.2 ( 2021-02-10T04:35:56.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: osd.2) 1410 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:56.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: cluster 2021-02-10T04:35:56.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:35:56.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: 35:55.882157+0000 osd.2 2021-02-10T04:35:56.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: (osd. 2021-02-10T04:35:56.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: 2) 1411 : cluster [WRN] 2021-02-10T04:35:56.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:56.479 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: cluster 2021-02-10T04:35 2021-02-10T04:35:56.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: :55.882160+0000 osd.2 (osd 2021-02-10T04:35:56.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: .2) 1412 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:56.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: 2021-02-10T04:35:56.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:35:56.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: -02-10T04:35:55.882164 2021-02-10T04:35:56.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1413 2021-02-10T04:35:56.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:56.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: cluster 2021-02-10T04:35:56.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: 2021-02-10T04:35:55.882167+0000 2021-02-10T04:35:56.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: osd.2 (osd.2) 1414 : cluster [WRN] 2021-02-10T04:35:56.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:56.480 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:56.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: 02-10T04:35:55.882172+0000 2021-02-10T04:35:56.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: osd.2 (osd.2) 1415 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:56.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:56.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: 02-10T04:35:55.882177+0000 2021-02-10T04:35:56.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: osd.2 (osd.2) 1416 : cluster [WRN] 2021-02-10T04:35:56.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:56.481 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:35:56.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: T04:35:55.882182+0000 2021-02-10T04:35:56.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: osd.2 (osd.2) 1417 : cluster [WRN] 2021-02-10T04:35:56.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:56.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:35:56.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: 35:55.882187+0000 osd.2 (osd.2) 1418 2021-02-10T04:35:56.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:56.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: cephadm 2021-02- 2021-02-10T04:35:56.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: 10T04:35:56.032601+0000 mgr.y ( 2021-02-10T04:35:56.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: mgr.14140) 612 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:35:56.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: cephadm 2021 2021-02-10T04:35:56.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: -02-10T04:35:56.033277+0000 2021-02-10T04:35:56.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: mgr.y (mgr.14140) 613 : cephadm [INF] 2021-02-10T04:35:56.482 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: Refreshing smithi164 facts 2021-02-10T04:35:56.483 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: cluster 2021-02-10T04:35:56.483 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:35:56.483 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: T04:35:56.109221+0000 mgr.y (mgr 2021-02-10T04:35:56.483 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.5 to 1 (10000) 2021-02-10T04:35:56.484 INFO:tasks.rados.rados.0.smithi085.stdout:2080: done (11 left) 2021-02-10T04:35:56.484 INFO:tasks.rados.rados.0.smithi085.stdout:2081: read oid 7 snap -1 2021-02-10T04:35:56.484 INFO:tasks.rados.rados.0.smithi085.stdout:2081: expect (ObjNum 727 snap 230 seq_num 727) 2021-02-10T04:35:56.484 INFO:tasks.rados.rados.0.smithi085.stdout:2082: delete oid 44 current snap is 230 2021-02-10T04:35:56.487 INFO:tasks.rados.rados.0.smithi085.stdout:2082: done (12 left) 2021-02-10T04:35:56.487 INFO:tasks.rados.rados.0.smithi085.stdout:2083: snap_create 2021-02-10T04:35:56.488 INFO:tasks.rados.rados.0.smithi085.stdout:2076: finishing write tid 1 to smithi08575397-6 2021-02-10T04:35:56.488 INFO:tasks.rados.rados.0.smithi085.stdout:2076: finishing write tid 2 to smithi08575397-6 2021-02-10T04:35:56.488 INFO:tasks.rados.rados.0.smithi085.stdout:2076: finishing write tid 3 to smithi08575397-6 2021-02-10T04:35:56.488 INFO:tasks.rados.rados.0.smithi085.stdout:2076: finishing write tid 4 to smithi08575397-6 2021-02-10T04:35:56.488 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 1037 (ObjNum 729 snap 230 seq_num 729) dirty exists 2021-02-10T04:35:56.488 INFO:tasks.rados.rados.0.smithi085.stdout:2076: left oid 6 (ObjNum 729 snap 230 seq_num 729) 2021-02-10T04:35:56.488 INFO:tasks.rados.rados.0.smithi085.stdout:2078: finishing rollback tid 0 to smithi08575397-46 2021-02-10T04:35:56.488 INFO:tasks.rados.rados.0.smithi085.stdout:2078: finishing rollback tid 1 to smithi08575397-46 2021-02-10T04:35:56.488 INFO:tasks.rados.rados.0.smithi085.stdout:2078: finishing rollback tid 2 to smithi08575397-46 2021-02-10T04:35:56.489 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 1040 (ObjNum 510 snap 148 seq_num 510) dirty exists 2021-02-10T04:35:56.494 INFO:tasks.rados.rados.0.smithi085.stdout:2081: expect (ObjNum 727 snap 230 seq_num 727) 2021-02-10T04:35:56.518 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:35:56.623 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:35:56.669 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:35:56.758 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: .14140) 614 : cluster [DBG] pgmap v957: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 417 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 1.5 MiB/s wr, 26 op/s 2021-02-10T04:35:56.758 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:35:56.758 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: 02-10T04:35:56.119046+0000 2021-02-10T04:35:56.758 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: mon.a (mon.0) 1369 : audit [INF] 2021-02-10T04:35:56.758 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[29319]: from='client.? 172.21.15.85:0/1047474168' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-02-10T04:35:56.759 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: :55.882164+0000 osd.2 (osd.2) 1413 : cluster [WRN] 2021-02-10T04:35:56.759 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:56.759 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:35:56.759 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: 35:55.882167+0000 osd.2 (osd.2) 1414 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:56.759 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: cluster 2021-02-10T04:35:55.882172+0000 2021-02-10T04:35:56.759 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: osd.2 (osd.2) 1415 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:56.759 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: cluster 2021-02-10T04:35:56.759 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: 2021-02-10T04:35:55.882177 2021-02-10T04:35:56.759 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1416 : cluster [WRN] 2021-02-10T04:35:56.759 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:56.759 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:35:56.759 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: 35:55.882182+0000 osd.2 (osd.2 2021-02-10T04:35:56.760 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: ) 1417 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:56.760 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: 2021-02-10T04:35:56.760 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: cluster 2021-02-10T04:35:56.760 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: 2021-02-10T04:35:55. 2021-02-10T04:35:56.760 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: 882187+0000 osd.2 (osd.2 2021-02-10T04:35:56.760 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: ) 1418 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:56.760 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: cephadm 2021-02-10 2021-02-10T04:35:56.760 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: T04:35:56.032601+0000 mgr.y (mgr. 2021-02-10T04:35:56.760 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: 14140) 612 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:35:56.760 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: cephadm 2021-02-10T04 2021-02-10T04:35:56.760 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: :35:56.033277+0000 mgr.y (mgr.14140) 613 : cephadm [INF] 2021-02-10T04:35:56.760 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: Refreshing smithi164 facts 2021-02-10T04:35:56.760 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: cluster 2021-02-10T04:35:56.760 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: 2021-02-10T04:35:56.109221+0000 mgr.y (mgr.14140 2021-02-10T04:35:56.761 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: ) 614 : cluster [DBG] pgmap v957: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 417 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 1.5 MiB/s wr, 26 op/s 2021-02-10T04:35:56.761 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:35:56.761 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: 10T04:35:56.119046+0000 mon.a (mon.0) 1369 : audit 2021-02-10T04:35:56.761 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[35961]: [INF] from='client.? 172.21.15.85:0/1047474168' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-02-10T04:35:56.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: cluster 2021-02-10T04:35:56.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: 2021-02-10T04:35:56.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: 2021-02-10T04:35: 2021-02-10T04:35:56.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: 55.465286+0000 mon.a (mon.0) 1368 : cluster [DBG] osdmap e538: 8 total, 8 up, 8 in 2021-02-10T04:35:56.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:56.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: -10T04:35:55.882148+0000 osd.2 (osd.2) 1409 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:56.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:35:56.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: -02-10T04:35:55.882153+0000 2021-02-10T04:35:56.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: osd.2 (osd.2) 1410 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:56.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: cluster 2021-02-10T04:35:56.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: 2021-02-10T04:35:56.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: 2021-02-10T04:35:55. 2021-02-10T04:35:56.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: 882157+0000 osd.2 (osd.2) 1411 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:56.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: cluster 2021-02-10T04:35:55.882160 2021-02-10T04:35:56.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1412 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:56.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:35:56.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: 10T04:35:55.882164+0000 2021-02-10T04:35:56.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: osd.2 (osd.2) 1413 : cluster 2021-02-10T04:35:56.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:56.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:35:56.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: 10T04:35:55.882167 2021-02-10T04:35:56.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1414 : 2021-02-10T04:35:56.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:56.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:35:56.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: 35:55.882172+0000 osd.2 (osd.2) 1415 : 2021-02-10T04:35:56.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:56.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: cluster 2021-02-10T04:35 2021-02-10T04:35:56.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: :55.882177+0000 osd.2 (osd.2) 1416 : cluster 2021-02-10T04:35:56.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:56.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: cluster 2021-02-10T04:35:55. 2021-02-10T04:35:56.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: 882182+0000 osd.2 (osd.2) 1417 : cluster [WRN] 2021-02-10T04:35:56.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:56.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: cluster 2021-02-10T04:35:56.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: 2021-02-10T04:35:55.882187+0000 osd.2 (osd. 2021-02-10T04:35:56.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: 2) 1418 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:56.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: cephadm 2021-02-10T04 2021-02-10T04:35:56.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: :35:56.032601+0000 mgr.y (mgr.14140) 2021-02-10T04:35:56.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: 612 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:35:56.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: cephadm 2021-02-10T04:35: 2021-02-10T04:35:56.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: 56.033277+0000 mgr.y (mgr.14140) 613 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:35:56.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: cluster 2021-02-10T04:35:56.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: 2021-02-10T04:35:56.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: 2021 2021-02-10T04:35:56.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: - 2021-02-10T04:35:56.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: 02-10T04:35:56. 2021-02-10T04:35:56.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: 109221+0000 mgr.y (mgr.14140) 614 : cluster [DBG] pgmap v957: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 417 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 1.5 MiB/s wr, 26 op/s 2021-02-10T04:35:56.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: audit 2021-02-10T04:35:56.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: 2021-02-10 2021-02-10T04:35:56.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: T04:35:56.119046+0000 mon.a (mon.0) 1369 : audit [INF] 2021-02-10T04:35:56.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:56 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/1047474168' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-02-10T04:35:56.982 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:35:57.009 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:35:57.011 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[49106]: debug 2021-02-10T04:35:57.011 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[49106]: 2021-02-10T04:35:56.925+0000 7f77a3e94700 -1 osd.2 539 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:35:57.011 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:35:56 smithi085 conmon[52836]: debug 2021-02-10T04:35:56.755+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:35:57.158 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset noscrub 2021-02-10T04:35:57.273 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:35:57.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[33677]: debug 2021-02-10T04:35:57.083+0000 7fbec19a1700 -1 osd.4 539 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:35:57.465 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:35:57.476 INFO:tasks.rados.rados.0.smithi085.stdout:2076: done (12 left) 2021-02-10T04:35:57.476 INFO:tasks.rados.rados.0.smithi085.stdout:2078: done (11 left) 2021-02-10T04:35:57.476 INFO:tasks.rados.rados.0.smithi085.stdout:2081: done (10 left) 2021-02-10T04:35:57.476 INFO:tasks.rados.rados.0.smithi085.stdout:2083: done (9 left) 2021-02-10T04:35:57.476 INFO:tasks.rados.rados.0.smithi085.stdout:2084: delete oid 46 current snap is 231 2021-02-10T04:35:57.486 INFO:tasks.rados.rados.0.smithi085.stdout:2084: done (9 left) 2021-02-10T04:35:57.486 INFO:tasks.rados.rados.0.smithi085.stdout:2085: snap_create 2021-02-10T04:35:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: cluster 2021-02-10T04:35:56.105825 2021-02-10T04:35:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 763 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: cluster 2021-02-10T04:35:56.105833+0000 2021-02-10T04:35:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: osd.4 (osd.4) 764 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:35:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: 04:35:56.105839+0000 osd.4 (osd.4 2021-02-10T04:35:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: ) 765 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: cluster 2021-02-10T04:35:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: 2021-02-10T04:35:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:35:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: 35:56.105845+0000 osd.4 (osd.4) 766 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: audit 2021-02-10T04:35:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: 2021-02-10T04:35:56.474222+0000 mon.a 2021-02-10T04:35:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: (mon.0) 1370 : audit [INF] from='client.? 172.21.15.85:0/1047474168' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-02-10T04:35:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: 2021-02-10T04:35:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: cluster 2021-02-10T04:35:56 2021-02-10T04:35:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: .474275+0000 mon.a (mon.0) 1371 : cluster 2021-02-10T04:35:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: [DBG] osdmap e539: 8 total, 8 up, 8 in 2021-02-10T04:35:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: audit 2021-02-10T04: 2021-02-10T04:35:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: 35:56.623540+0000 mon.a (mon.0) 2021-02-10T04:35:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: 1372 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:35:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:35:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: T04:35:56.926821+0000 osd.2 (osd 2021-02-10T04:35:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: .2) 1419 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:35:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: :35:56.926835+0000 osd.2 (osd.2) 2021-02-10T04:35:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: 1420 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:35:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: 04:35:56.926840+0000 osd.2 (osd. 2021-02-10T04:35:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: 2) 1421 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:35:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: 04:35:56.926845+0000 osd.2 (osd. 2021-02-10T04:35:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: 2) 1422 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: -10T04:35:56.926849+0000 osd.2 (osd.2) 1423 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: cluster 2021-02-10T04:35:56 2021-02-10T04:35:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: .926854+0000 osd.2 (osd.2) 1424 : cluster 2021-02-10T04:35:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: cluster 2021-02-10T04:35:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: 2021-02-10T04:35:56.926858+0000 osd.2 (osd.2) 1425 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:35:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: -02-10T04:35:56.926863+0000 2021-02-10T04:35:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: osd.2 (osd.2) 1426 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: 2021-02-10T04:35:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:35:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: 02-10T04:35:56.926868+0000 osd.2 (osd.2) 1427 : cluster 2021-02-10T04:35:57.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:57.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: cluster 2021-02-10T04:35:56 2021-02-10T04:35:57.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: .926872+0000 osd.2 ( 2021-02-10T04:35:57.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:57 smithi164 conmon[29288]: osd.2) 1428 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:57.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: cluster 2021-02-10T04:35:57.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: 2021-02-10T04:35:57.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: 2021 2021-02-10T04:35:57.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: - 2021-02-10T04:35:57.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: 02 2021-02-10T04:35:57.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: - 2021-02-10T04:35:57.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: 10 2021-02-10T04:35:57.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: T 2021-02-10T04:35:57.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: 04 2021-02-10T04:35:57.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: : 2021-02-10T04:35:57.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: 35 2021-02-10T04:35:57.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: : 2021-02-10T04:35:57.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: 56 2021-02-10T04:35:57.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: .105825+0000 osd.4 ( 2021-02-10T04:35:57.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: osd.4) 763 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:57.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: cluster 2021-02-10T04:35:57.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: 2021-02-10T04:35:56.105833 2021-02-10T04:35:57.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 764 2021-02-10T04:35:57.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:57.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:57.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: 02-10T04:35:56.105839+0000 osd.4 ( 2021-02-10T04:35:57.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: osd.4) 765 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:57.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: cluster 2021-02-10T04:35:56.105845+0000 2021-02-10T04:35:57.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: osd.4 (osd.4) 766 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:57.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: audit 2021 2021-02-10T04:35:57.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: -02-10T04:35:56.474222+0000 2021-02-10T04:35:57.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: mon.a (mon.0) 1370 : audit [INF] 2021-02-10T04:35:57.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: from='client.? 172.21.15.85:0/1047474168' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-02-10T04:35:57.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: cluster 2021-02-10T04:35:56. 2021-02-10T04:35:57.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: 474275+0000 mon.a (mon.0) 1371 : cluster [DBG] 2021-02-10T04:35:57.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: osdmap e539: 8 total, 8 up, 8 in 2021-02-10T04:35:57.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: audit 2021-02-10T04:35:57.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: 2021-02-10T04:35:56.623540+0000 mon.a (mon 2021-02-10T04:35:57.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: .0) 1372 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:35:57.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:35:57.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: T04:35:56.926821+0000 osd.2 (osd. 2021-02-10T04:35:57.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: 2) 1419 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:57.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: cluster 2021-02-10T04:35: 2021-02-10T04:35:57.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: 56.926835+0000 osd.2 (osd.2) 1420 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:57.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:35:57.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: -10T04:35:56.926840+0000 osd.2 (osd.2) 2021-02-10T04:35:57.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: 1421 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:57.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: cluster 2021-02-10T04:35 2021-02-10T04:35:57.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: :56.926845+0000 osd.2 (osd.2) 1422 : cluster 2021-02-10T04:35:57.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:57.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: cluster 2021-02-10T04:35:57.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: 2021-02-10T04:35:56.926849+0000 osd.2 (osd.2) 1423 2021-02-10T04:35:57.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:57.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: cluster 2021-02-10T04:35:57.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: 2021-02-10T04:35:56.926854+0000 osd.2 (osd.2) 1424 : cluster 2021-02-10T04:35:57.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:57.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: cluster 2021-02-10T04:35:57.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: 2021-02-10T04:35:56.926858+0000 osd.2 (osd.2 2021-02-10T04:35:57.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: ) 1425 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:57.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: cluster 2021-02-10T04:35: 2021-02-10T04:35:57.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: 56.926863+0000 osd.2 (osd.2) 1426 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:57.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: 2021-02-10T04:35:57.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:35:57.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: 04:35:56.926868+0000 osd.2 (osd. 2021-02-10T04:35:57.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: 2) 1427 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:57.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: 2021-02-10T04:35:57.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:35:57.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: T04:35:56.926872 2021-02-10T04:35:57.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1428 2021-02-10T04:35:57.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:57.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: cluster 2021-02-10T04:35:56.105825+0000 2021-02-10T04:35:57.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: osd.4 (osd.4) 763 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:57.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:35:57.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: 10T04:35:56.105833+0000 osd.4 (osd.4) 764 : cluster [WRN] 2021-02-10T04:35:57.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:57.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: cluster 2021-02-10T04:35:57.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: 2021-02-10T04:35:56.105839+0000 osd.4 (osd.4) 765 : 2021-02-10T04:35:57.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:57.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: cluster 2021-02-10T04:35: 2021-02-10T04:35:57.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: 56.105845+0000 osd.4 (osd.4) 766 : cluster 2021-02-10T04:35:57.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:57.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: audit 2021-02-10T04:35: 2021-02-10T04:35:57.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: 56.474222+0000 mon.a (mon.0) 1370 : audit 2021-02-10T04:35:57.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: [INF] from='client.? 172.21.15.85:0/1047474168' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-02-10T04:35:57.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:35:57.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: 02-10T04:35:56 2021-02-10T04:35:57.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: .474275+0000 mon.a (mon.0 2021-02-10T04:35:57.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: ) 1371 : cluster [DBG] osdmap e539: 8 total, 8 up, 8 in 2021-02-10T04:35:57.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:35:57.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: T04:35:56.623540+0000 mon.a (mon. 2021-02-10T04:35:57.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: 0) 1372 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:35:57.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: cluster 2021-02-10T04:35:57.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: 2021-02-10T04:35:56.926821 2021-02-10T04:35:57.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1419 : cluster [WRN] 2021-02-10T04:35:57.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:57.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:35:57.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: -02-10T04:35:56.926835+0000 2021-02-10T04:35:57.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: osd.2 (osd.2) 1420 : 2021-02-10T04:35:57.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:57.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: cluster 2021-02-10T04:35:57.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: 2021-02-10T04:35 2021-02-10T04:35:57.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: :56.926840+0000 osd.2 ( 2021-02-10T04:35:57.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: osd.2) 1421 : cluster [WRN] 2021-02-10T04:35:57.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:57.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: cluster 2021-02-10T04:35 2021-02-10T04:35:57.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: :56.926845+0000 osd.2 (osd.2) 1422 : 2021-02-10T04:35:57.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:57.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: cluster 2021-02-10T04:35:56 2021-02-10T04:35:57.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: .926849+0000 osd.2 (osd.2) 1423 : cluster [WRN] 2021-02-10T04:35:57.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:57.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: cluster 2021-02-10T04:35:57.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: 2021-02-10T04:35:56.926854+0000 osd.2 (osd.2) 2021-02-10T04:35:57.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: 1424 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:57.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:35:57.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: 35:56.926858+0000 osd.2 (osd.2) 1425 : 2021-02-10T04:35:57.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:57.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:35:57.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: -10T04:35:56. 2021-02-10T04:35:57.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: 926863+0000 osd.2 (osd.2) 2021-02-10T04:35:57.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: 1426 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:57.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: cluster 2021-02-10T04:35:57.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: 2021-02-10T04:35 2021-02-10T04:35:57.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: :56.926868+0000 osd.2 ( 2021-02-10T04:35:57.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: osd.2) 1427 : cluster 2021-02-10T04:35:57.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:57.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:35:57.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: -02-10T04:35: 2021-02-10T04:35:57.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: 56.926872+0000 osd.2 (osd. 2021-02-10T04:35:57.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: 2) 1428 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:57.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[35961]: 2021-02-10T04:35:58.393 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:35:58.554 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[33677]: debug 2021-02-10T04:35:58.075+0000 7fbec19a1700 -1 osd.4 540 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:35:58.555 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[49106]: debug 2021-02-10T04:35:58.555 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[49106]: 2021-02-10T04:35:57.948+0000 7f77a3e94700 -1 osd.2 540 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:35:58.555 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:57 smithi085 conmon[49106]: 2021-02-10T04:35:58.555 INFO:tasks.rados.rados.0.smithi085.stdout:2085: done (9 left) 2021-02-10T04:35:58.555 INFO:tasks.rados.rados.0.smithi085.stdout:2086: snap_remove snap 228 2021-02-10T04:35:58.576 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.4 dump_historic_ops 2021-02-10T04:35:58.876 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:35:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: cluster 2021-02-10T04:35:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 2021-02-10T04:35:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 2021 2021-02-10T04:35:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: - 2021-02-10T04:35:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 02 2021-02-10T04:35:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: - 2021-02-10T04:35:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 10 2021-02-10T04:35:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: T 2021-02-10T04:35:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 04:35:57.084780 2021-02-10T04:35:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 767 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: cluster 2021-02-10T04:35:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 2021-02-10T04:35:57 2021-02-10T04:35:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: .084789+0000 osd.4 (osd.4) 768 : 2021-02-10T04:35:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:35:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: :35:57.084795+0000 osd.4 (osd.4) 2021-02-10T04:35:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 769 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: cluster 2021-02-10T04:35: 2021-02-10T04:35:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 57.084800+0000 osd.4 (osd.4) 770 : cluster [WRN] 2021-02-10T04:35:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: cluster 2021-02-10T04:35:57. 2021-02-10T04:35:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 473075+0000 mon.a (mon.0) 1373 : cluster 2021-02-10T04:35:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: [DBG] osdmap e540: 8 total, 8 up, 8 in 2021-02-10T04:35:58.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:58.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: -10T04:35:57.949886+0000 osd.2 2021-02-10T04:35:58.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: (osd.2) 1429 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:58.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: cluster 2021-02-10T04:35:58.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 2021-02-10T04:35:57.949894+0000 2021-02-10T04:35:58.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: osd.2 (osd.2) 1430 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:58.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 2021-02-10T04:35:58.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: cluster 2021-02-10T04:35:57 2021-02-10T04:35:58.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: .949900+0000 osd.2 (osd.2) 1431 : 2021-02-10T04:35:58.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:58.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:35:58.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 02-10T04:35:57.949906+0000 osd.2 2021-02-10T04:35:58.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: (osd.2) 1432 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:58.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: cluster 2021-02-10T04:35:58.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 2021-02-10T04:35:57. 2021-02-10T04:35:58.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 949912+0000 osd.2 (osd.2) 1433 : cluster 2021-02-10T04:35:58.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:58.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: cluster 2021-02-10T04:35 2021-02-10T04:35:58.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: :57.949918+0000 osd.2 (osd.2) 1434 : cluster [WRN] 2021-02-10T04:35:58.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:58.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: cluster 2021-02-10T04:35: 2021-02-10T04:35:58.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 57.949925+0000 osd.2 (osd.2) 1435 2021-02-10T04:35:58.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:58.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:35:58.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: -10T04:35:57.949932+0000 osd.2 2021-02-10T04:35:58.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: (osd.2) 1436 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:58.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 2021-02-10T04:35:58.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: cluster 2021-02-10T04:35:57 2021-02-10T04:35:58.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: .949937+0000 osd.2 (osd.2) 1437 : 2021-02-10T04:35:58.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:58.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:35:58.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 10T04:35:57.949943+0000 osd.2 ( 2021-02-10T04:35:58.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: osd.2) 1438 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:58.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: cluster 2021-02-10T04:35:58.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 2021-02-10T04:35:58.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 2021 2021-02-10T04:35:58.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: - 2021-02-10T04:35:58.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 02 2021-02-10T04:35:58.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: - 2021-02-10T04:35:58.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 10 2021-02-10T04:35:58.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: T 2021-02-10T04:35:58.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 04 2021-02-10T04:35:58.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: : 2021-02-10T04:35:58.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 35 2021-02-10T04:35:58.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: : 2021-02-10T04:35:58.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 58 2021-02-10T04:35:58.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: . 2021-02-10T04:35:58.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 109694 2021-02-10T04:35:58.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: +0000 2021-02-10T04:35:58.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 2021-02-10T04:35:58.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: mgr.y 2021-02-10T04:35:58.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: ( 2021-02-10T04:35:58.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: mgr 2021-02-10T04:35:58.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: . 2021-02-10T04:35:58.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 14140 2021-02-10T04:35:58.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: ) 2021-02-10T04:35:58.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 615 2021-02-10T04:35:58.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: : 2021-02-10T04:35:58.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: cluster 2021-02-10T04:35:58.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 2021-02-10T04:35:58.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: [DBG] 2021-02-10T04:35:58.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 2021-02-10T04:35:58.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: pgmap v960: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 417 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 38 KiB/s rd, 707 KiB/s wr, 66 op/s 2021-02-10T04:35:58.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 2021-02-10T04:35:58.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: audit 2021-02-10T04:35:58.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 2021-02-10T04:35:58.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 2021 2021-02-10T04:35:58.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: - 2021-02-10T04:35:58.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 02 2021-02-10T04:35:58.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: - 2021-02-10T04:35:58.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 10 2021-02-10T04:35:58.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: T 2021-02-10T04:35:58.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 04 2021-02-10T04:35:58.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: : 2021-02-10T04:35:58.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 35 2021-02-10T04:35:58.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: : 2021-02-10T04:35:58.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 58 2021-02-10T04:35:58.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: . 2021-02-10T04:35:58.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 116989 2021-02-10T04:35:58.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: +0000 2021-02-10T04:35:58.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 2021-02-10T04:35:58.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: mgr.y 2021-02-10T04:35:58.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: ( 2021-02-10T04:35:58.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: mgr 2021-02-10T04:35:58.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: . 2021-02-10T04:35:58.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 14140 2021-02-10T04:35:58.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: ) 2021-02-10T04:35:58.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 616 2021-02-10T04:35:58.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: : 2021-02-10T04:35:58.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: audit 2021-02-10T04:35:58.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 2021-02-10T04:35:58.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: [DBG] 2021-02-10T04:35:58.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 2021-02-10T04:35:58.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:35:58.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:58 smithi164 conmon[29288]: 2021-02-10T04:35:58.919 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: cluster 2021-02-10T04:35:57 2021-02-10T04:35:58.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: .084780+0000 osd.4 (osd.4) 767 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:58.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: cluster 2021-02-10T04:35:57.084789+0000 osd.4 (osd.4) 768 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:58.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: cluster 2021-02-10T04:35:57.084795+0000 osd.4 (osd.4) 769 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:58.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: cluster 2021-02-10T04:35:57.084800+0000 osd.4 (osd.4) 770 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:58.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: cluster 2021-02-10T04:35:57.473075+0000 mon.a (mon.0) 1373 : cluster 2021-02-10T04:35:58.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: [DBG] osdmap e540: 8 total, 8 up, 8 in 2021-02-10T04:35:58.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:35:58.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: 10T04:35:57.949886+0000 osd.2 2021-02-10T04:35:58.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: (osd.2) 1429 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:58.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: cluster 2021-02-10T04:35:58.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: 2021-02-10T04:35:57. 2021-02-10T04:35:58.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: 949894+0000 osd.2 (osd.2) 1430 : cluster 2021-02-10T04:35:58.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:58.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: cluster 2021-02-10T04:35 2021-02-10T04:35:58.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: :57.949900+0000 osd.2 (osd.2) 1431 : cluster 2021-02-10T04:35:58.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:58.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:35:58.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: 02-10T04:35:57.949906+0000 osd.2 (osd.2) 1432 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:58.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: cluster 2021-02-10T04:35:57 2021-02-10T04:35:58.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: .949912+0000 osd.2 (osd.2) 1433 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:58.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: cluster 2021-02-10T04:35 2021-02-10T04:35:58.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: :57.949918+0000 osd.2 (osd.2) 1434 : cluster [WRN] 2021-02-10T04:35:58.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:58.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:35:58.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: :35:57.949925+0000 osd.2 (osd.2) 1435 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:58.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: cluster 2021-02-10T04:35:57.949932+0000 2021-02-10T04:35:58.922 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: osd.2 (osd.2) 1436 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:58.922 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: cluster 2021-02-10T04:35:57.949937+0000 2021-02-10T04:35:58.922 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: osd.2 (osd.2) 1437 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:58.922 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: cluster 2021-02-10T04:35:57.949943+0000 2021-02-10T04:35:58.922 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: osd.2 (osd.2) 1438 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:58.922 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: cluster 2021-02-10T04:35:58. 2021-02-10T04:35:58.922 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: 109694+0000 mgr.y (mgr.14140) 615 : cluster [DBG] pgmap v960: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 417 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 38 KiB/s rd, 707 KiB/s wr, 66 op/s 2021-02-10T04:35:58.922 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: audit 2021-02-10T04:35: 2021-02-10T04:35:58.922 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: 58.116989+0000 mgr.y (mgr.14140) 616 : audit [DBG] 2021-02-10T04:35:58.922 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[29319]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:35:58.923 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: cluster 2021-02-10T04:35:57.084780+0000 2021-02-10T04:35:58.923 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: osd.4 (osd.4) 767 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:35:58.923 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: cluster 2021-02-10T04:35:57.084789+0000 osd.4 (osd.4 2021-02-10T04:35:58.923 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: ) 768 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:35:58.923 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:35:58.923 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: 10T04:35:57.084795+0000 osd.4 (osd.4) 769 : cluster 2021-02-10T04:35:58.923 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:35:58.923 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: cluster 2021-02-10T04:35: 2021-02-10T04:35:58.923 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: 57.084800+0000 osd.4 (osd.4) 770 : cluster 2021-02-10T04:35:58.923 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:35:58.923 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:35:58.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: -10T04:35:57.473075+0000 mon.a 2021-02-10T04:35:58.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: (mon.0) 1373 : cluster [DBG] osdmap e540: 8 total, 8 up, 8 in 2021-02-10T04:35:58.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:35:58.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: -02-10T04:35:57.949886+0000 2021-02-10T04:35:58.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: osd.2 (osd.2) 1429 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:35:58.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: 2021-02-10T04:35:58.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: cluster 2021-02-10T04:35:58.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: 2021-02-10T04:35:57.949894+0000 osd.2 2021-02-10T04:35:58.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: (osd.2) 1430 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:35:58.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: 2021-02-10T04:35:58.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: cluster 2021-02-10T04:35:58.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: 2021-02-10T04:35:57.949900+0000 2021-02-10T04:35:58.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: osd.2 (osd.2) 1431 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:35:58.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: 2021-02-10T04:35:58.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: cluster 2021-02-10T04:35:58.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: 2021-02-10T04:35:57.949906+0000 2021-02-10T04:35:58.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: osd.2 (osd.2) 1432 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:35:58.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: 2021-02-10T04:35:58.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: cluster 2021-02-10T04:35:58.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: 2021-02-10T04:35:57.949912+0000 osd.2 2021-02-10T04:35:58.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: (osd.2) 1433 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:35:58.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:35:58.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: :35:57.949918+0000 osd.2 (osd.2 2021-02-10T04:35:58.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: ) 1434 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:35:58.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:35:58.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: 04:35:57.949925+0000 osd.2 (osd.2 2021-02-10T04:35:58.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: ) 1435 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:35:58.926 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:35:58.926 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: 04:35:57.949932+0000 osd.2 (osd 2021-02-10T04:35:58.926 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: .2) 1436 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:35:58.926 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:35:58.926 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: 10T04:35:57.949937+0000 osd.2 ( 2021-02-10T04:35:58.926 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: osd.2) 1437 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:35:58.926 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:35:58.926 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: -02-10T04:35:57.949943 2021-02-10T04:35:58.926 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1438 : cluster [WRN] 2021-02-10T04:35:58.926 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:35:58.926 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: cluster 2021-02-10T04:35:58. 2021-02-10T04:35:58.926 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: 109694+0000 mgr.y (mgr.14140) 615 : cluster 2021-02-10T04:35:58.927 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: [DBG] pgmap v960: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 417 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 38 KiB/s rd, 707 KiB/s wr, 66 op/s 2021-02-10T04:35:58.927 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:35:58.927 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: 35:58.116989+0000 mgr.y (mgr.14140) 616 2021-02-10T04:35:58.927 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[35961]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:35:59.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[49106]: debug 2021-02-10T04:35:59.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[49106]: 2021-02-10T04:35:58.917+0000 7f77a3e94700 -1 osd.2 541 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:35:59.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:58 smithi085 conmon[49106]: 2021-02-10T04:35:59.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[33677]: debug 2021-02-10T04:35:59.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[33677]: 2021-02-10T04:35:59.099+0000 7fbec19a1700 -1 osd.4 541 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:35:59.494 INFO:tasks.rados.rados.0.smithi085.stdout:2086: done (9 left) 2021-02-10T04:36:02.083 INFO:tasks.rados.rados.0.smithi085.stdout:2087: snap_create 2021-02-10T04:36:02.087 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: cluster 2021-02-10T04:35:58.077437+0000 2021-02-10T04:36:02.087 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: osd.4 (osd.4) 771 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:02.087 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: cluster 2021-02-10T04:35:58.077444+0000 osd.4 (osd.4) 772 : cluster 2021-02-10T04:36:02.087 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:02.087 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: cluster 2021-02-10T04:36:02.087 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: 2021-02-10T04:35:58.077448+0000 osd.4 (osd.4) 2021-02-10T04:36:02.088 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: 773 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:02.088 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:02.088 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: 02-10T04 2021-02-10T04:36:02.088 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: :35:58.077454+0000 osd.4 (osd.4) 774 : cluster 2021-02-10T04:36:02.088 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:02.088 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: cluster 2021-02-10T04:35: 2021-02-10T04:36:02.088 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: 58.476999+0000 mon.a (mon.0) 1374 : cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 387 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:36:02.088 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:36:02.088 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: 10T04:35:58.488207+0000 mon.a (mon.0) 1375 2021-02-10T04:36:02.088 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: : cluster [DBG] osdmap e541: 8 total, 8 up, 8 in 2021-02-10T04:36:02.088 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: cluster 2021-02-10T04:36:02.088 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: 2021-02-10T04:35:58.919121+0000 osd.2 (osd.2) 1439 2021-02-10T04:36:02.088 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:02.089 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:36:02.089 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: 04:35:58.919131+0000 osd.2 ( 2021-02-10T04:36:02.089 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: osd.2) 1440 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:02.089 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:02.089 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: -10T04:35:58.919138+0000 osd.2 (osd.2) 1441 2021-02-10T04:36:02.089 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:02.089 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: cluster 2021-02-10T04:36:02.089 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: 2021-02-10T04:35:58.919148+0000 osd.2 (osd.2) 1442 : cluster 2021-02-10T04:36:02.089 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:02.089 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:02.089 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: -02-10T04:35:58.919154+0000 osd.2 (osd.2) 1443 : cluster [WRN] 2021-02-10T04:36:02.089 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:02.089 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:02.090 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: -10T04:35:58.919161+0000 osd.2 (osd.2) 2021-02-10T04:36:02.090 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: 1444 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:02.090 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:02.090 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: 02-10T04:35:58 2021-02-10T04:36:02.090 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: .919167+0000 osd.2 (osd.2 2021-02-10T04:36:02.090 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: ) 1445 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:02.090 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:02.090 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: -02-10T04:35:58. 2021-02-10T04:36:02.090 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: 919173+0000 osd.2 (osd.2) 1446 : cluster 2021-02-10T04:36:02.090 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:02.090 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:36:02.090 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: 10T04:35:58.919180+0000 2021-02-10T04:36:02.090 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: osd.2 (osd.2) 1447 : cluster [WRN] 2021-02-10T04:36:02.091 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:02.091 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:36:02.091 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: 04:35:58.919186+0000 osd.2 ( 2021-02-10T04:36:02.091 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:35:59 smithi164 conmon[29288]: osd.2) 1448 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:02.091 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: cluster 2021-02-10T04:36:02.091 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 2021-02-10T04:36:02.091 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 2021 2021-02-10T04:36:02.091 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: - 2021-02-10T04:36:02.091 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 02 2021-02-10T04:36:02.091 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: - 2021-02-10T04:36:02.091 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 10 2021-02-10T04:36:02.091 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: T 2021-02-10T04:36:02.091 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 04 2021-02-10T04:36:02.092 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: : 2021-02-10T04:36:02.092 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 35: 2021-02-10T04:36:02.092 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 59.099914+0000 osd.4 (osd.4) 775 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:02.092 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: cluster 2021-02-10T04:36:02.092 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 2021 2021-02-10T04:36:02.092 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: - 2021-02-10T04:36:02.092 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 02 2021-02-10T04:36:02.092 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: - 2021-02-10T04:36:02.092 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 10 2021-02-10T04:36:02.092 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: T 2021-02-10T04:36:02.092 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 04 2021-02-10T04:36:02.092 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: : 2021-02-10T04:36:02.092 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 35 2021-02-10T04:36:02.093 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: : 2021-02-10T04:36:02.093 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 59 2021-02-10T04:36:02.093 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: . 2021-02-10T04:36:02.093 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 099925 2021-02-10T04:36:02.093 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: +0000 2021-02-10T04:36:02.093 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 2021-02-10T04:36:02.093 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: osd.4 2021-02-10T04:36:02.093 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: ( 2021-02-10T04:36:02.093 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: osd 2021-02-10T04:36:02.093 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: . 2021-02-10T04:36:02.093 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 4 2021-02-10T04:36:02.093 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: ) 2021-02-10T04:36:02.093 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 776 2021-02-10T04:36:02.094 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: : 2021-02-10T04:36:02.094 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: cluster 2021-02-10T04:36:02.094 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 2021-02-10T04:36:02.094 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: [WRN] 2021-02-10T04:36:02.094 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 2021-02-10T04:36:02.094 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:02.094 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 2021-02-10T04:36:02.094 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: cluster 2021-02-10T04:36:02.094 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 2021-02-10T04:36:02.094 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 2021 2021-02-10T04:36:02.094 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: - 2021-02-10T04:36:02.094 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 02 2021-02-10T04:36:02.095 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: - 2021-02-10T04:36:02.095 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 10 2021-02-10T04:36:02.095 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: T 2021-02-10T04:36:02.095 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 04 2021-02-10T04:36:02.095 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: : 2021-02-10T04:36:02.095 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 35 2021-02-10T04:36:02.095 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: : 2021-02-10T04:36:02.095 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 59 2021-02-10T04:36:02.095 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: . 2021-02-10T04:36:02.095 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 099931 2021-02-10T04:36:02.095 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: +0000 2021-02-10T04:36:02.096 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 2021-02-10T04:36:02.096 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: osd.4 2021-02-10T04:36:02.096 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: ( 2021-02-10T04:36:02.096 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: osd 2021-02-10T04:36:02.096 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: . 2021-02-10T04:36:02.096 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 4 2021-02-10T04:36:02.096 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: ) 2021-02-10T04:36:02.096 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 777 2021-02-10T04:36:02.096 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: : 2021-02-10T04:36:02.096 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: cluster 2021-02-10T04:36:02.096 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 2021-02-10T04:36:02.096 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: [WRN] 2021-02-10T04:36:02.096 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 2021-02-10T04:36:02.097 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:02.097 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 2021-02-10T04:36:02.097 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: cluster 2021-02-10T04:36:02.097 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 2021-02-10T04:36:02.097 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 2021 2021-02-10T04:36:02.097 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: - 2021-02-10T04:36:02.097 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 02 2021-02-10T04:36:02.097 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: - 2021-02-10T04:36:02.097 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 10 2021-02-10T04:36:02.097 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: T 2021-02-10T04:36:02.097 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 04 2021-02-10T04:36:02.097 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: : 2021-02-10T04:36:02.098 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 35 2021-02-10T04:36:02.098 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: : 2021-02-10T04:36:02.098 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 59 2021-02-10T04:36:02.098 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: . 2021-02-10T04:36:02.098 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 099936 2021-02-10T04:36:02.098 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: +0000 2021-02-10T04:36:02.098 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 2021-02-10T04:36:02.098 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: osd.4 2021-02-10T04:36:02.098 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: ( 2021-02-10T04:36:02.099 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: osd 2021-02-10T04:36:02.099 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: . 2021-02-10T04:36:02.099 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 4 2021-02-10T04:36:02.099 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: ) 2021-02-10T04:36:02.099 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 778 2021-02-10T04:36:02.099 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: : 2021-02-10T04:36:02.099 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: cluster 2021-02-10T04:36:02.099 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 2021-02-10T04:36:02.099 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: [WRN] 2021-02-10T04:36:02.099 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 2021-02-10T04:36:02.099 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:02.099 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 2021-02-10T04:36:02.099 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: cluster 2021-02-10T04:36:02.099 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 2021-02-10T04:36:02.100 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 2021 2021-02-10T04:36:02.100 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: - 2021-02-10T04:36:02.100 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 02 2021-02-10T04:36:02.100 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: - 2021-02-10T04:36:02.100 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 10 2021-02-10T04:36:02.100 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: T 2021-02-10T04:36:02.100 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 04 2021-02-10T04:36:02.100 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: : 2021-02-10T04:36:02.100 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 35 2021-02-10T04:36:02.100 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: : 2021-02-10T04:36:02.100 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 59 2021-02-10T04:36:02.100 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: . 2021-02-10T04:36:02.100 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 493863 2021-02-10T04:36:02.101 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: +0000 2021-02-10T04:36:02.101 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 2021-02-10T04:36:02.101 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: mon.a 2021-02-10T04:36:02.101 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: ( 2021-02-10T04:36:02.101 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: mon 2021-02-10T04:36:02.101 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: . 2021-02-10T04:36:02.101 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 0 2021-02-10T04:36:02.101 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: ) 2021-02-10T04:36:02.101 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 1376 2021-02-10T04:36:02.101 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: : 2021-02-10T04:36:02.101 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: cluster 2021-02-10T04:36:02.101 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 2021-02-10T04:36:02.101 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: [DBG] 2021-02-10T04:36:02.102 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 2021-02-10T04:36:02.102 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: osdmap e542: 8 total, 8 up, 8 in 2021-02-10T04:36:02.102 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 2021-02-10T04:36:02.102 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: cluster 2021-02-10T04:36:02.102 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 2021-02-10T04:36:02.102 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 2021 2021-02-10T04:36:02.102 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: - 2021-02-10T04:36:02.102 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 02 2021-02-10T04:36:02.102 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: - 2021-02-10T04:36:02.102 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 10 2021-02-10T04:36:02.102 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: T 2021-02-10T04:36:02.102 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 04 2021-02-10T04:36:02.102 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: : 2021-02-10T04:36:02.103 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 35 2021-02-10T04:36:02.103 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: : 2021-02-10T04:36:02.103 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 59 2021-02-10T04:36:02.103 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: .900146+0000 osd.2 (osd.2) 1449 2021-02-10T04:36:02.103 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:02.103 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: cluster 2021-02-10T04:36:02.103 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 2021-02-10T04:35:59.900154+0000 osd.2 (osd.2) 1450 : cluster 2021-02-10T04:36:02.103 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:02.103 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: cluster 2021-02-10T04:35: 2021-02-10T04:36:02.103 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 59.900160+0000 osd.2 (osd.2) 1451 : 2021-02-10T04:36:02.103 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:02.103 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: cluster 2021-02-10T04:35 2021-02-10T04:36:02.103 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: :59.900166+0000 osd.2 (osd.2) 1452 : 2021-02-10T04:36:02.104 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:02.104 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: cluster 2021-02-10T04:35 2021-02-10T04:36:02.104 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: :59.900172+0000 osd.2 (osd.2) 1453 : 2021-02-10T04:36:02.104 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:02.104 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: cluster 2021-02-10T04:35 2021-02-10T04:36:02.104 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: :59.900177+0000 osd.2 (osd.2) 1454 2021-02-10T04:36:02.104 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:02.104 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:36:02.104 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 35:59.900182+0000 osd.2 (osd.2) 1455 : 2021-02-10T04:36:02.104 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:02.104 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:36:02.104 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 35:59.900188+0000 osd.2 (osd.2) 1456 2021-02-10T04:36:02.104 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:02.105 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:36:02.105 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: :35:59.900193+0000 osd.2 (osd.2 2021-02-10T04:36:02.105 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: ) 1457 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:02.105 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:36:02.105 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 04:35:59.900198+0000 osd.2 (osd.2) 2021-02-10T04:36:02.105 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 1458 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:02.105 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:36:02.105 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: T04:36:00.110242+0000 mgr.y (mgr. 2021-02-10T04:36:02.105 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[29288]: 14140) 617 : cluster [DBG] pgmap v963: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 415 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2021-02-10T04:36:02.105 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: cluster 2021-02-10T04:36:00.105783+0000 osd.4 2021-02-10T04:36:02.105 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: (osd.4) 779 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:02.105 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: cluster 2021-02-10T04:36:00.105797+0000 osd.4 (osd.4) 780 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:02.105 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:02.106 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: -02-10T04:36:00.105802+0000 osd.4 (osd.4) 781 : cluster [WRN] 2021-02-10T04:36:02.106 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:02.106 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: cluster 2021-02-10T04:36:02.106 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: 2021-02-10T04:36:00.105808+0000 osd.4 (osd.4) 782 : 2021-02-10T04:36:02.106 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:02.106 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: cluster 2021-02-10T04:36:02.106 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: 2021-02-10T04:36:00.499277+0000 mon.a (mon.0) 1377 : 2021-02-10T04:36:02.106 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: cluster [DBG] osdmap e543: 8 total, 8 up, 8 in 2021-02-10T04:36:02.106 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: cluster 2021-02-10T04:36:02.106 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: 2021-02-10T04:36:00.943663+0000 osd.2 (osd.2) 1459 2021-02-10T04:36:02.106 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:02.106 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: cluster 2021-02-10T04:36:00.943669 2021-02-10T04:36:02.107 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1460 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:02.107 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: 2021-02-10T04:36:02.107 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: cluster 2021-02-10T04:36:00.943673+0000 osd.2 (osd.2 2021-02-10T04:36:02.107 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: ) 1461 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:02.107 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:36:02.107 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: 10T04:36:00.943677+0000 osd.2 (osd.2) 1462 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:02.107 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:36:02.107 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: 04:36:00.943681+0000 osd.2 (osd.2) 2021-02-10T04:36:02.107 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: 1463 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:02.107 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: cluster 2021-02-10T04:36:00.943684+0000 2021-02-10T04:36:02.107 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: osd.2 (osd.2) 1464 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:02.107 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:36:02.107 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: 04:36:00.943687+0000 osd.2 2021-02-10T04:36:02.107 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: (osd.2) 1465 : cluster 2021-02-10T04:36:02.108 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:02.108 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:02.108 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: -02-10T04:36: 2021-02-10T04:36:02.108 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: 00.943690+0000 osd.2 (osd.2) 1466 2021-02-10T04:36:02.108 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:02.108 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: cluster 2021-02-10T04:36:00. 2021-02-10T04:36:02.108 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: 943694+0000 osd.2 (osd.2) 1467 : cluster 2021-02-10T04:36:02.108 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:02.108 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:36:02.108 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: T04:36:00.943697+0000 osd.2 ( 2021-02-10T04:36:02.108 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[29288]: osd.2) 1468 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:02.108 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:00 smithi164 conmon[33677]: debug 2021-02-10T04:36:00.105+0000 7fbec19a1700 -1 osd.4 542 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:02.109 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:01 smithi164 conmon[33677]: debug 2021-02-10T04:36:01.140+0000 7fbec19a1700 -1 osd.4 543 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:02.109 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: cluster 2021-02-10T04:35:58.077437+0000 2021-02-10T04:36:02.109 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: osd.4 (osd.4) 771 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:02.109 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: cluster 2021-02-10T04:35:58.077444 2021-02-10T04:36:02.109 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 772 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:02.109 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:36:02.109 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: T04:35:58.077448+0000 osd.4 (osd.4 2021-02-10T04:36:02.109 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: ) 773 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:02.109 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:36:02.109 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: 10T04:35:58.077454+0000 osd.4 2021-02-10T04:36:02.109 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: (osd.4) 774 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:02.110 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:02.110 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: -10T04:35:58.476999+0000 mon.a (mon 2021-02-10T04:36:02.110 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: .0) 1374 : cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 387 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:36:02.110 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:02.110 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: 02-10T04:35:58.488207+0000 2021-02-10T04:36:02.110 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: mon.a (mon.0) 1375 : cluster [DBG] osdmap e541: 8 total, 8 up, 8 in 2021-02-10T04:36:02.110 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:36:02.110 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: 10T04:35:58.919121+0000 osd.2 (osd 2021-02-10T04:36:02.110 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: .2) 1439 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:02.110 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:36:02.110 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: 10T04:35:58.919131+0000 osd.2 ( 2021-02-10T04:36:02.110 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: osd.2) 1440 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:02.110 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:02.111 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: -02-10T04:35:58.919138+0000 2021-02-10T04:36:02.111 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: osd.2 (osd.2) 1441 : cluster [WRN] 2021-02-10T04:36:02.111 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:02.111 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:02.111 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: -10T04:35:58.919148+0000 osd.2 (osd.2) 1442 : 2021-02-10T04:36:02.111 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:02.111 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: cluster 2021-02-10T04:35 2021-02-10T04:36:02.111 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: :58.919154+0000 osd.2 (osd.2) 1443 : cluster [WRN] 2021-02-10T04:36:02.111 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:02.111 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: cluster 2021-02-10T04:35:58. 2021-02-10T04:36:02.111 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: 919161+0000 osd.2 (osd.2) 1444 : cluster [WRN] 2021-02-10T04:36:02.111 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:02.111 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: cluster 2021-02-10T04:35: 2021-02-10T04:36:02.112 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: 58.919167+0000 osd.2 (osd.2) 1445 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:02.112 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: 2021-02-10T04:36:02.112 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: cluster 2021-02-10T04:36:02.112 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: 2021-02-10T04:35:58.919173 2021-02-10T04:36:02.112 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1446 : cluster 2021-02-10T04:36:02.112 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:02.112 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: cluster 2021-02-10T04:35 2021-02-10T04:36:02.112 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: :58.919180+0000 osd.2 (osd.2) 1447 2021-02-10T04:36:02.112 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:02.112 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: cluster 2021-02-10T04:36:02.112 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: 2021-02-10T04:35:58 2021-02-10T04:36:02.112 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[29319]: .919186+0000 osd.2 (osd.2) 1448 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:02.112 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: cluster 2021-02-10T04:36:02.113 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: 2021-02-10T04:36:02.113 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: 2021 2021-02-10T04:36:02.113 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: - 2021-02-10T04:36:02.113 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: 02-10T04:35:59.099914+0000 osd.4 (osd.4) 775 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:02.113 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:36:02.113 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: 04:35:59.099925+0000 osd.4 (osd.4) 776 : cluster 2021-02-10T04:36:02.113 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:02.113 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:36:02.113 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: 35:59.099931+0000 osd.4 (osd.4) 777 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:02.113 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: cluster 2021-02-10T04:35:59.099936+0000 2021-02-10T04:36:02.113 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: osd.4 (osd.4) 778 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:02.113 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:02.113 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: 02-10T04:35:59.493863+0000 mon.a 2021-02-10T04:36:02.114 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: (mon.0) 1376 : cluster [DBG] 2021-02-10T04:36:02.114 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: osdmap e542: 8 total, 8 up, 8 in 2021-02-10T04:36:02.114 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:36:02.114 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: :35:59.900146+0000 osd.2 (osd.2) 1449 : 2021-02-10T04:36:02.114 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:02.114 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: cluster 2021-02-10T04:36:02.114 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: 2021-02-10T04:35 2021-02-10T04:36:02.114 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: :59.900154+0000 osd.2 (osd.2 2021-02-10T04:36:02.114 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: ) 1450 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:02.114 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:02.114 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: -02-10T04:35:59.900160 2021-02-10T04:36:02.114 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1451 : cluster 2021-02-10T04:36:02.114 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:02.115 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:02.115 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: 02-10T04:35:59.900166 2021-02-10T04:36:02.115 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1452 : cluster 2021-02-10T04:36:02.115 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:02.115 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: cluster 2021-02-10T04:35:59.900172+0000 osd.2 ( 2021-02-10T04:36:02.115 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: osd.2) 1453 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:02.115 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: cluster 2021-02-10T04:35:59.900177 2021-02-10T04:36:02.115 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1454 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:02.115 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: cluster 2021-02-10T04:36:02.115 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: 2021-02-10T04:35:59.900182+0000 osd.2 (osd.2) 1455 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:02.115 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: cluster 2021-02-10T04:35: 2021-02-10T04:36:02.115 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: 59.900188+0000 osd.2 (osd.2) 1456 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:02.115 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: cluster 2021-02-10T04:35:59 2021-02-10T04:36:02.116 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: .900193+0000 osd.2 (osd.2) 1457 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:02.116 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: cluster 2021-02-10T04:35:59.900198+0000 2021-02-10T04:36:02.116 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: osd.2 (osd.2) 1458 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:02.116 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: cluster 2021-02-10T04:36:00.110242 2021-02-10T04:36:02.116 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 617 : cluster [DBG] pgmap v963: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 415 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2021-02-10T04:36:02.116 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: cluster 2021-02-10T04:36:02.116 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: 2021-02-10T04:36:02.116 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: 2021 2021-02-10T04:36:02.116 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: - 2021-02-10T04:36:02.116 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: 02 2021-02-10T04:36:02.116 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: -10T04:36:00.105783+0000 osd.4 2021-02-10T04:36:02.116 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: (osd.4) 779 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:02.116 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:02.117 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: 02-10T04:36:00.105797+0000 osd.4 ( 2021-02-10T04:36:02.117 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: osd.4) 780 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:02.117 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: cluster 2021-02-10T04:36:02.117 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: 2021-02-10T04:36:00.105802 2021-02-10T04:36:02.117 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 781 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:02.117 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: 2021-02-10T04:36:02.117 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:02.117 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: -02-10T04:36:00 2021-02-10T04:36:02.117 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: .105808+0000 osd.4 (osd. 2021-02-10T04:36:02.117 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: 4) 782 : cluster [WRN] 2021-02-10T04:36:02.117 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:02.117 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:02.117 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: 02-10T04:36:00.499277+0000 mon.a 2021-02-10T04:36:02.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: (mon.0) 1377 : cluster [DBG] osdmap e543: 8 total, 8 up, 8 in 2021-02-10T04:36:02.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: 2021-02-10T04:36:02.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:02.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: -02-10T04:36:00.943663+0000 2021-02-10T04:36:02.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: osd.2 (osd.2) 1459 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:02.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:02.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: -10T04:36:00.943669+0000 osd.2 (osd.2 2021-02-10T04:36:02.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: ) 1460 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:02.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: cluster 2021-02-10T04:36:00.943673 2021-02-10T04:36:02.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1461 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:02.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:36:02.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: 04:36:00.943677+0000 osd.2 (osd.2) 1462 : cluster 2021-02-10T04:36:02.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:02.119 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:02.119 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: -02-10T04:36:00.943681+0000 osd.2 (osd.2) 1463 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:02.119 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:36:02.119 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: T04:36:00.943684+0000 2021-02-10T04:36:02.119 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: osd.2 (osd.2) 1464 : cluster [WRN] 2021-02-10T04:36:02.119 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:02.119 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: cluster 2021-02-10T04:36:02.119 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: 2021-02-10T04:36:00. 2021-02-10T04:36:02.119 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: 943687+0000 osd.2 (osd.2) 2021-02-10T04:36:02.119 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: 1465 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:02.119 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: cluster 2021-02-10T04:36:02.119 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: 2021-02-10T04:36:00 2021-02-10T04:36:02.119 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: .943690+0000 osd.2 (osd.2) 1466 : 2021-02-10T04:36:02.120 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:02.120 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: cluster 2021-02-10T04:36:02.120 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:36:02.120 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: 36:00.943694+0000 osd.2 (osd 2021-02-10T04:36:02.120 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: .2) 1467 : cluster [WRN] 2021-02-10T04:36:02.120 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:02.120 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: cluster 2021-02-10T04:36:00.943697+0000 2021-02-10T04:36:02.120 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[29319]: osd.2 (osd.2) 1468 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:02.120 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: cluster 2021-02-10T04:36:02.120 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 2021-02-10T04:36:02.120 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 2021 2021-02-10T04:36:02.120 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: - 2021-02-10T04:36:02.121 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 02 2021-02-10T04:36:02.121 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: - 2021-02-10T04:36:02.121 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 10 2021-02-10T04:36:02.121 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: T 2021-02-10T04:36:02.121 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 04:35:58.077437+0000 osd.4 ( 2021-02-10T04:36:02.121 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: osd.4) 771 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:02.121 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: cluster 2021-02-10T04:36:02.121 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:36:02.121 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 04:35:58.077444+0000 osd.4 (osd.4) 772 : cluster [WRN] 2021-02-10T04:36:02.121 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:02.121 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:02.121 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 02-10T04:35:58.077448+0000 osd.4 (osd.4) 773 : cluster [WRN] 2021-02-10T04:36:02.121 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:02.122 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: cluster 2021-02-10T04:36:02.122 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:36:02.122 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 10T04:35:58.077454+0000 osd.4 (osd.4) 774 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:02.122 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:02.122 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 02-10T04:35:58.476999+0000 mon.a (mon.0 2021-02-10T04:36:02.122 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: ) 1374 : cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 387 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:36:02.122 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: cluster 2021-02-10T04:36:02.122 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:36:02.122 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 04:35:58.488207+0000 mon.a (mon.0) 1375 : cluster [DBG] 2021-02-10T04:36:02.122 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: osdmap e541: 8 total, 8 up, 8 in 2021-02-10T04:36:02.122 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:02.122 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: -10T04:35:58.919121+0000 osd.2 (osd.2) 2021-02-10T04:36:02.122 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 1439 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:02.123 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:02.123 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 02-10T04:35:58.919131+0000 2021-02-10T04:36:02.123 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: osd.2 (osd.2) 1440 : cluster 2021-02-10T04:36:02.123 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:02.123 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:02.123 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 02-10T04:35:58.919138+0000 osd.2 (osd.2) 2021-02-10T04:36:02.123 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 1441 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:02.123 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:02.123 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 02-10T04:35:58.919148+0000 osd.2 ( 2021-02-10T04:36:02.123 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: osd.2) 1442 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:02.123 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:02.123 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: -02-10T04:35:58 2021-02-10T04:36:02.123 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: .919154+0000 osd.2 (osd. 2021-02-10T04:36:02.124 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 2) 1443 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:02.124 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:02.124 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 02-10T04:35:58.919161+0000 osd.2 (osd.2 2021-02-10T04:36:02.124 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: ) 1444 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:02.124 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: cluster 2021-02-10T04:36:02.124 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 2021-02-10T04:36:02.124 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 2021 2021-02-10T04:36:02.124 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: - 2021-02-10T04:36:02.124 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 02 2021-02-10T04:36:02.124 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: - 2021-02-10T04:36:02.124 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 10 2021-02-10T04:36:02.124 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: T 2021-02-10T04:36:02.124 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 04 2021-02-10T04:36:02.124 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: : 2021-02-10T04:36:02.125 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 35 2021-02-10T04:36:02.125 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: : 2021-02-10T04:36:02.125 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 58 2021-02-10T04:36:02.125 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: . 2021-02-10T04:36:02.125 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 919167 2021-02-10T04:36:02.125 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: +0000 2021-02-10T04:36:02.125 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 2021-02-10T04:36:02.125 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: osd.2 2021-02-10T04:36:02.125 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: ( 2021-02-10T04:36:02.125 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: osd.2) 1445 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:02.125 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 2021-02-10T04:36:02.125 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:02.125 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: -02-10T04:35:58.919173+0000 2021-02-10T04:36:02.126 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: osd.2 (osd.2) 1446 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:02.126 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: cluster 2021-02-10T04:36:02.126 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:36:02.126 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 04:35:58.919180+0000 osd.2 (osd.2) 1447 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:02.126 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: cluster 2021-02-10T04:36:02.126 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 2021-02-10T04:36:02.126 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 2021 2021-02-10T04:36:02.126 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: - 2021-02-10T04:36:02.126 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 02 2021-02-10T04:36:02.126 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: - 2021-02-10T04:36:02.126 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 10 2021-02-10T04:36:02.126 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: T 2021-02-10T04:36:02.126 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 04 2021-02-10T04:36:02.127 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: : 2021-02-10T04:36:02.127 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 35 2021-02-10T04:36:02.127 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: : 2021-02-10T04:36:02.127 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 58 2021-02-10T04:36:02.127 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: . 2021-02-10T04:36:02.127 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 919186 2021-02-10T04:36:02.127 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: +0000 2021-02-10T04:36:02.127 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 2021-02-10T04:36:02.127 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: osd.2 2021-02-10T04:36:02.127 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: ( 2021-02-10T04:36:02.127 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: osd 2021-02-10T04:36:02.127 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: . 2021-02-10T04:36:02.127 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 2 2021-02-10T04:36:02.128 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: ) 2021-02-10T04:36:02.128 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 1448 2021-02-10T04:36:02.128 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: : 2021-02-10T04:36:02.128 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: cluster 2021-02-10T04:36:02.128 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 2021-02-10T04:36:02.128 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: [WRN] 2021-02-10T04:36:02.128 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 2021-02-10T04:36:02.128 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:02.128 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[35961]: 2021-02-10T04:36:02.128 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: cluster 2021-02-10T04:36:02.128 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:36:02.128 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: 04:35:59.099914+0000 osd.4 (osd.4) 775 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:02.128 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:02.129 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: 02-10T04:35:59.099925+0000 2021-02-10T04:36:02.129 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: osd.4 (osd.4) 776 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:02.129 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:02.129 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: -10T04:35:59.099931+0000 osd.4 (osd 2021-02-10T04:36:02.129 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: .4) 777 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:02.129 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:02.129 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: 02-10T04:35:59.099936+0000 2021-02-10T04:36:02.129 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: osd.4 (osd.4) 778 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:02.129 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: 2021-02-10T04:36:02.129 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: cluster 2021-02-10T04:36:02.129 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: 2021-02-10T04:35:59.493863+0000 mon.a ( 2021-02-10T04:36:02.129 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: mon.0) 1376 : cluster [DBG] osdmap e542: 8 total, 8 up, 8 in 2021-02-10T04:36:02.129 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:02.129 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: -10T04:35:59.900146+0000 osd.2 ( 2021-02-10T04:36:02.130 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: osd.2) 1449 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:02.130 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:02.130 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: -02-10T04:35:59.900154+0000 2021-02-10T04:36:02.130 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: osd.2 (osd.2) 1450 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:02.130 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: cluster 2021-02-10T04:36:02.130 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: 2021-02-10T04:36:02.130 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: 2021 2021-02-10T04:36:02.130 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: - 2021-02-10T04:36:02.130 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: 02 2021-02-10T04:36:02.130 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: - 2021-02-10T04:36:02.130 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: 10 2021-02-10T04:36:02.130 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: T 2021-02-10T04:36:02.130 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: 04 2021-02-10T04:36:02.131 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: : 2021-02-10T04:36:02.131 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: 35 2021-02-10T04:36:02.131 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: : 2021-02-10T04:36:02.131 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: 59 2021-02-10T04:36:02.131 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: . 2021-02-10T04:36:02.131 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: 900160 2021-02-10T04:36:02.131 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: +0000 2021-02-10T04:36:02.131 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: 2021-02-10T04:36:02.131 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: osd.2 2021-02-10T04:36:02.131 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: ( 2021-02-10T04:36:02.131 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: osd.2) 1451 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:02.131 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: 2021-02-10T04:36:02.131 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:36:02.132 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: 10T04:35:59.900166+0000 osd.2 (osd.2) 1452 : cluster 2021-02-10T04:36:02.132 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:02.132 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:02.132 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: 02-10T04:35:59.900172+0000 osd.2 (osd 2021-02-10T04:36:02.132 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: .2) 1453 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:02.132 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:36:02.132 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: 10T04:35:59.900177+0000 osd.2 (osd.2) 1454 2021-02-10T04:36:02.132 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:02.132 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:36:02.132 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: :35:59.900182+0000 osd.2 (osd.2) 1455 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:02.132 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: cluster 2021-02-10T04:36:02.132 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: 2021-02-10T04:35:59 2021-02-10T04:36:02.132 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: .900188+0000 osd.2 (osd.2) 1456 : 2021-02-10T04:36:02.133 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:02.133 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: cluster 2021-02-10T04:35 2021-02-10T04:36:02.133 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: :59.900193+0000 osd.2 ( 2021-02-10T04:36:02.133 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: osd.2) 1457 : cluster 2021-02-10T04:36:02.133 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:02.133 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:02.133 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: -10T04:35:59.900198+0000 osd.2 2021-02-10T04:36:02.133 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: (osd.2) 1458 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:02.133 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:02.133 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: -02-10T04:36:00.110242+0000 mgr.y (mgr 2021-02-10T04:36:02.133 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[35961]: .14140) 617 : cluster [DBG] pgmap v963: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 415 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2021-02-10T04:36:02.133 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: cluster 2021-02-10T04:36:00.105783 2021-02-10T04:36:02.133 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 779 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:02.134 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:36:02.134 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: 36:00.105797+0000 osd.4 (osd.4) 780 2021-02-10T04:36:02.134 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:02.134 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:02.134 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: 02-10T04:36:00.105802+0000 osd.4 2021-02-10T04:36:02.134 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: (osd.4) 781 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:02.134 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: 2021-02-10T04:36:02.134 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: cluster 2021-02-10T04:36:02.134 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: 2021-02-10T04:36:00 2021-02-10T04:36:02.134 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: .105808+0000 osd.4 (osd.4) 782 : 2021-02-10T04:36:02.134 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:02.134 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:02.135 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: 02-10T04:36:00.499277+0000 2021-02-10T04:36:02.135 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: mon.a (mon.0) 1377 : cluster [DBG] osdmap e543: 8 total, 8 up, 8 in 2021-02-10T04:36:02.135 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: cluster 2021-02-10T04:36:02.135 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: 2021-02-10T04:36:00.943663+0000 osd.2 2021-02-10T04:36:02.135 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: (osd.2) 1459 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:02.135 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: cluster 2021-02-10T04:36:00.943669+0000 osd.2 (osd.2) 1460 : cluster 2021-02-10T04:36:02.135 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:02.135 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: cluster 2021-02-10T04:36:02.135 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: 2021-02-10T04:36:00.943673+0000 osd.2 (osd 2021-02-10T04:36:02.135 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: .2) 1461 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:02.135 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:02.135 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: 02-10T04:36:00.943677+0000 osd.2 2021-02-10T04:36:02.135 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: (osd.2) 1462 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:02.135 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: 2021-02-10T04:36:02.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: cluster 2021-02-10T04:36:00.943681+0000 osd.2 (osd 2021-02-10T04:36:02.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: .2) 1463 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:02.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:36:02.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: 10T04:36:00.943684+0000 osd.2 ( 2021-02-10T04:36:02.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: osd.2) 1464 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:02.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:36:02.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: 04:36:00.943687+0000 osd.2 (osd.2 2021-02-10T04:36:02.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: ) 1465 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:02.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:02.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: T04:36:00.943690+0000 osd.2 (osd.2) 1466 : cluster 2021-02-10T04:36:02.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:02.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: cluster 2021-02-10T04:36:02.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: 2021-02-10T04:36 2021-02-10T04:36:02.137 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: :00.943694+0000 osd.2 (osd.2) 2021-02-10T04:36:02.137 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: 1467 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:02.137 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:02.137 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: -02-10T04:36:00.943697+0000 2021-02-10T04:36:02.137 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: osd.2 (osd.2) 1468 : cluster [WRN] 2021-02-10T04:36:02.137 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:02.137 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[49106]: debug 2021-02-10T04:36:02.137 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[49106]: 2021-02-10T04:35:59.898+0000 7f77a3e94700 -1 osd.2 542 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:02.137 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:35:59 smithi085 conmon[49106]: 2021-02-10T04:36:02.137 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:00 smithi085 conmon[49106]: debug 2021-02-10T04:36:00.942+0000 7f77a3e94700 -1 osd.2 543 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:02.137 INFO:tasks.rados.rados.0.smithi085.stdout:2087: done (9 left) 2021-02-10T04:36:02.138 INFO:tasks.rados.rados.0.smithi085.stdout:2088: write oid 40 current snap is 233 2021-02-10T04:36:02.138 INFO:tasks.rados.rados.0.smithi085.stdout:2088: seq_num 732 ranges {702724=652156,1951525=351199} 2021-02-10T04:36:02.138 INFO:tasks.rados.rados.0.smithi085.stdout:2088: writing smithi08575397-40 from 702724 to 1354880 tid 1 2021-02-10T04:36:02.138 INFO:tasks.rados.rados.0.smithi085.stdout:2088: writing smithi08575397-40 from 1951525 to 2302724 tid 2 2021-02-10T04:36:02.138 INFO:tasks.rados.rados.0.smithi085.stdout:2089: read oid 42 snap -1 2021-02-10T04:36:02.138 INFO:tasks.rados.rados.0.smithi085.stdout:2089: expect (ObjNum 646 snap 196 seq_num 646) 2021-02-10T04:36:02.138 INFO:tasks.rados.rados.0.smithi085.stdout:2090: snap_create 2021-02-10T04:36:02.138 INFO:tasks.rados.rados.0.smithi085.stdout:2088: finishing write tid 1 to smithi08575397-40 2021-02-10T04:36:02.138 INFO:tasks.rados.rados.0.smithi085.stdout:2088: finishing write tid 2 to smithi08575397-40 2021-02-10T04:36:02.138 INFO:tasks.rados.rados.0.smithi085.stdout:2088: finishing write tid 3 to smithi08575397-40 2021-02-10T04:36:02.138 INFO:tasks.rados.rados.0.smithi085.stdout:2088: finishing write tid 4 to smithi08575397-40 2021-02-10T04:36:02.138 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 1114 (ObjNum 732 snap 233 seq_num 732) dirty exists 2021-02-10T04:36:02.138 INFO:tasks.rados.rados.0.smithi085.stdout:2088: left oid 40 (ObjNum 732 snap 233 seq_num 732) 2021-02-10T04:36:02.139 INFO:tasks.rados.rados.0.smithi085.stdout:2089: expect (ObjNum 646 snap 196 seq_num 646) 2021-02-10T04:36:02.139 INFO:tasks.rados.rados.0.smithi085.stdout:2088: done (11 left) 2021-02-10T04:36:02.139 INFO:tasks.rados.rados.0.smithi085.stdout:2089: done (10 left) 2021-02-10T04:36:02.139 INFO:tasks.rados.rados.0.smithi085.stdout:2090: done (9 left) 2021-02-10T04:36:02.139 INFO:tasks.rados.rados.0.smithi085.stdout:2091: write oid 46 current snap is 234 2021-02-10T04:36:02.139 INFO:tasks.rados.rados.0.smithi085.stdout:2091: seq_num 733 ranges {719531=727405,1893654=425877} 2021-02-10T04:36:02.139 INFO:tasks.rados.rados.0.smithi085.stdout:2091: writing smithi08575397-46 from 719531 to 1446936 tid 1 2021-02-10T04:36:02.139 INFO:tasks.rados.rados.0.smithi085.stdout:2091: writing smithi08575397-46 from 1893654 to 2319531 tid 2 2021-02-10T04:36:02.139 INFO:tasks.rados.rados.0.smithi085.stdout:2092: delete oid 16 current snap is 234 2021-02-10T04:36:02.139 INFO:tasks.rados.rados.0.smithi085.stdout:2091: finishing write tid 1 to smithi08575397-46 2021-02-10T04:36:02.139 INFO:tasks.rados.rados.0.smithi085.stdout:2092: done (10 left) 2021-02-10T04:36:02.139 INFO:tasks.rados.rados.0.smithi085.stdout:2093: snap_remove snap 185 2021-02-10T04:36:02.140 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:02 smithi164 conmon[33677]: debug 2021-02-10T04:36:02.110+0000 7fbec19a1700 -1 osd.4 544 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:02.381 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:36:02.382 INFO:teuthology.orchestra.run.smithi164.stdout: "size": 20, 2021-02-10T04:36:02.382 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 600, 2021-02-10T04:36:02.382 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [ 2021-02-10T04:36:02.382 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.382 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:232 7.1 7:8132b738:::smithi08575397-33:head [read 0~2890771,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e101)", 2021-02-10T04:36:02.382 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:28:37.203216+0000", 2021-02-10T04:36:02.382 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 445.17566832900002, 2021-02-10T04:36:02.382 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010334987, 2021-02-10T04:36:02.382 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:36:02.382 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:36:02.382 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:36:02.382 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:36:02.383 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:36:02.383 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 232 2021-02-10T04:36:02.383 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.383 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:36:02.383 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.383 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:36:02.383 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203216+0000", 2021-02-10T04:36:02.383 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:36:02.383 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.383 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.383 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:36:02.383 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203216+0000", 2021-02-10T04:36:02.383 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.4169999999999999e-05 2021-02-10T04:36:02.384 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.384 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.384 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:36:02.384 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203260+0000", 2021-02-10T04:36:02.384 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.064e-06 2021-02-10T04:36:02.384 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.384 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.384 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:36:02.384 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203269+0000", 2021-02-10T04:36:02.384 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5914999999999999e-05 2021-02-10T04:36:02.384 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.384 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.384 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:36:02.385 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203285+0000", 2021-02-10T04:36:02.385 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.4010000000000003e-06 2021-02-10T04:36:02.385 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.385 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.385 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:36:02.385 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203295+0000", 2021-02-10T04:36:02.385 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1817999999999995e-05 2021-02-10T04:36:02.385 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.385 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.385 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:36:02.385 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203377+0000", 2021-02-10T04:36:02.385 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.4599999999999992e-06 2021-02-10T04:36:02.385 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.386 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.386 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2021-02-10T04:36:02.386 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203386+0000", 2021-02-10T04:36:02.386 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0059418359999999998 2021-02-10T04:36:02.386 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.386 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.386 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:36:02.386 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.209328+0000", 2021-02-10T04:36:02.386 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000121338 2021-02-10T04:36:02.386 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.386 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.386 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:36:02.386 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.209449+0000", 2021-02-10T04:36:02.387 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0041019849999999998 2021-02-10T04:36:02.387 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.387 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.387 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:36:02.387 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.213551+0000", 2021-02-10T04:36:02.387 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010334987 2021-02-10T04:36:02.387 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.387 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:36:02.387 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.387 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.387 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.387 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:233 7.1 7:8132b738:::smithi08575397-33:head [read 0~2890771,checksum] snapc 0=[] ondisk+read+known_if_redirected e101)", 2021-02-10T04:36:02.387 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:28:37.203339+0000", 2021-02-10T04:36:02.388 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 445.17554533999999, 2021-02-10T04:36:02.388 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012015658, 2021-02-10T04:36:02.388 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:36:02.388 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:36:02.388 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:36:02.388 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:36:02.388 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:36:02.388 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 233 2021-02-10T04:36:02.388 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.388 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:36:02.388 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.388 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:36:02.388 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203339+0000", 2021-02-10T04:36:02.388 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:36:02.389 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.389 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.389 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:36:02.389 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203339+0000", 2021-02-10T04:36:02.389 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.3578999999999998e-05 2021-02-10T04:36:02.389 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.389 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.389 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:36:02.389 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203373+0000", 2021-02-10T04:36:02.389 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.9829999999999998e-06 2021-02-10T04:36:02.389 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.389 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.389 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:36:02.390 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203377+0000", 2021-02-10T04:36:02.390 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.499e-06 2021-02-10T04:36:02.390 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.390 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.390 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:36:02.390 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203386+0000", 2021-02-10T04:36:02.390 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.0490000000000004e-06 2021-02-10T04:36:02.390 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.390 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.390 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:36:02.390 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203394+0000", 2021-02-10T04:36:02.390 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000101165 2021-02-10T04:36:02.390 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.391 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.391 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:36:02.391 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203495+0000", 2021-02-10T04:36:02.391 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6439999999999997e-06 2021-02-10T04:36:02.391 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.391 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.391 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting_for_map not empty", 2021-02-10T04:36:02.391 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203501+0000", 2021-02-10T04:36:02.391 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010104657 2021-02-10T04:36:02.391 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.391 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.391 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:36:02.391 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.213606+0000", 2021-02-10T04:36:02.391 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00015320000000000001 2021-02-10T04:36:02.392 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.392 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.392 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:36:02.392 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.213759+0000", 2021-02-10T04:36:02.392 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0015958820000000001 2021-02-10T04:36:02.392 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.392 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.392 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:36:02.392 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.215355+0000", 2021-02-10T04:36:02.392 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012015658 2021-02-10T04:36:02.392 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.392 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:36:02.392 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.393 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.393 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.393 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:234 7.1 7:8132b738:::smithi08575397-33:head [sparse-read 0~1312313] snapc 0=[] ondisk+read+known_if_redirected e101)", 2021-02-10T04:36:02.393 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:28:37.203458+0000", 2021-02-10T04:36:02.393 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 445.17542639800001, 2021-02-10T04:36:02.393 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012303939, 2021-02-10T04:36:02.393 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:36:02.393 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:36:02.393 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:36:02.393 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:36:02.393 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:36:02.393 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 234 2021-02-10T04:36:02.393 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.394 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:36:02.394 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.394 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:36:02.394 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203458+0000", 2021-02-10T04:36:02.394 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:36:02.394 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.394 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.394 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:36:02.394 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203458+0000", 2021-02-10T04:36:02.394 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2258000000000001e-05 2021-02-10T04:36:02.394 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.394 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.394 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:36:02.395 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203500+0000", 2021-02-10T04:36:02.395 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.4709999999999998e-06 2021-02-10T04:36:02.395 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.395 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.395 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:36:02.395 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203506+0000", 2021-02-10T04:36:02.395 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0747e-05 2021-02-10T04:36:02.395 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.395 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.395 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:36:02.395 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203527+0000", 2021-02-10T04:36:02.395 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.7590000000000003e-06 2021-02-10T04:36:02.395 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.395 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.396 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:36:02.396 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203535+0000", 2021-02-10T04:36:02.396 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.5235999999999997e-05 2021-02-10T04:36:02.396 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.396 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.396 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:36:02.396 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203601+0000", 2021-02-10T04:36:02.396 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1539999999999996e-06 2021-02-10T04:36:02.396 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.396 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.396 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting_for_map not empty", 2021-02-10T04:36:02.396 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.203608+0000", 2021-02-10T04:36:02.396 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011817237 2021-02-10T04:36:02.397 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.397 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.397 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:36:02.397 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.215425+0000", 2021-02-10T04:36:02.397 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00013556399999999999 2021-02-10T04:36:02.397 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.397 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.397 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:36:02.397 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.215561+0000", 2021-02-10T04:36:02.397 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00020151300000000001 2021-02-10T04:36:02.397 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.397 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.397 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:36:02.398 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:37.215762+0000", 2021-02-10T04:36:02.398 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012303939 2021-02-10T04:36:02.398 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.398 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:36:02.398 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.398 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.398 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.398 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:257 7.8 e104/93 7:10dac65a:::smithi08575397-39:head v 104'4, mlcod=98'2)", 2021-02-10T04:36:02.398 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:28:40.225588+0000", 2021-02-10T04:36:02.398 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 442.15329675200002, 2021-02-10T04:36:02.398 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0099257589999999993, 2021-02-10T04:36:02.398 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:36:02.398 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:36:02.399 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:36:02.399 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.399 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:36:02.399 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:40.225588+0000", 2021-02-10T04:36:02.399 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:36:02.399 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.399 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.399 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:36:02.399 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:40.225588+0000", 2021-02-10T04:36:02.399 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0075234569999999999 2021-02-10T04:36:02.399 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.399 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.399 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:36:02.400 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:40.233111+0000", 2021-02-10T04:36:02.400 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1969e-05 2021-02-10T04:36:02.400 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.400 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.400 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:36:02.400 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:40.233123+0000", 2021-02-10T04:36:02.400 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6528000000000002e-05 2021-02-10T04:36:02.400 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.400 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.400 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:36:02.400 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:40.233140+0000", 2021-02-10T04:36:02.400 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2013e-05 2021-02-10T04:36:02.400 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.400 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.401 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:36:02.401 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:40.233152+0000", 2021-02-10T04:36:02.401 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.4579e-05 2021-02-10T04:36:02.401 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.401 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.401 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:36:02.401 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:40.233206+0000", 2021-02-10T04:36:02.401 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7978999999999998e-05 2021-02-10T04:36:02.401 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.401 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.401 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:36:02.401 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:40.233224+0000", 2021-02-10T04:36:02.401 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0022181760000000001 2021-02-10T04:36:02.402 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.402 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.402 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:36:02.402 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:40.235443+0000", 2021-02-10T04:36:02.402 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1057999999999996e-05 2021-02-10T04:36:02.402 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.402 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.402 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:36:02.402 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:40.235514+0000", 2021-02-10T04:36:02.402 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0099257589999999993 2021-02-10T04:36:02.402 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.402 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:36:02.402 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.403 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.403 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.403 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:265 7.3 e105/93 7:c765b4cc:::smithi08575397-30:head v 105'14, mlcod=98'11)", 2021-02-10T04:36:02.403 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:28:41.238549+0000", 2021-02-10T04:36:02.403 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 441.140335736, 2021-02-10T04:36:02.403 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.009041561, 2021-02-10T04:36:02.403 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:36:02.403 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:36:02.403 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:36:02.403 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.403 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:36:02.403 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.238549+0000", 2021-02-10T04:36:02.403 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:36:02.404 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.404 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.404 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:36:02.404 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.238549+0000", 2021-02-10T04:36:02.404 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.003393877 2021-02-10T04:36:02.404 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.404 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.404 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:36:02.404 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.241943+0000", 2021-02-10T04:36:02.404 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2551999999999999e-05 2021-02-10T04:36:02.404 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.404 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.404 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:36:02.404 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.241955+0000", 2021-02-10T04:36:02.405 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7842999999999998e-05 2021-02-10T04:36:02.405 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.405 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.405 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:36:02.405 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.241973+0000", 2021-02-10T04:36:02.405 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0155e-05 2021-02-10T04:36:02.405 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.405 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.405 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:36:02.405 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.241983+0000", 2021-02-10T04:36:02.405 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.0531999999999994e-05 2021-02-10T04:36:02.405 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.405 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.405 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:36:02.406 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.242064+0000", 2021-02-10T04:36:02.406 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8025e-05 2021-02-10T04:36:02.406 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.406 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.406 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:36:02.406 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.242092+0000", 2021-02-10T04:36:02.406 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0054341579999999997 2021-02-10T04:36:02.406 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.406 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.406 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:36:02.406 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.247526+0000", 2021-02-10T04:36:02.406 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.4418999999999994e-05 2021-02-10T04:36:02.406 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.407 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.407 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:36:02.407 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:28:41.247590+0000", 2021-02-10T04:36:02.407 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.009041561 2021-02-10T04:36:02.407 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.407 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:36:02.407 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.407 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.407 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.407 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:553 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 17] snapc 1e=[1e,1d,1c,1b,19,17,15] ondisk+write+known_if_redirected e127)", 2021-02-10T04:36:02.407 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:03.299307+0000", 2021-02-10T04:36:02.407 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 419.07957787999999, 2021-02-10T04:36:02.407 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012622955, 2021-02-10T04:36:02.408 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:36:02.408 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:36:02.408 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:36:02.408 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:36:02.408 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:36:02.408 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 553 2021-02-10T04:36:02.408 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.408 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:36:02.408 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.408 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:36:02.408 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299307+0000", 2021-02-10T04:36:02.408 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:36:02.408 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.408 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.409 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:36:02.409 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299307+0000", 2021-02-10T04:36:02.409 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.8406000000000003e-05 2021-02-10T04:36:02.409 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.409 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.409 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:36:02.409 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299365+0000", 2021-02-10T04:36:02.409 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7580000000000004e-06 2021-02-10T04:36:02.409 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.409 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.409 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:36:02.409 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299375+0000", 2021-02-10T04:36:02.409 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6067e-05 2021-02-10T04:36:02.410 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.410 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.410 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:36:02.410 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299391+0000", 2021-02-10T04:36:02.410 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.3809999999999999e-06 2021-02-10T04:36:02.410 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.410 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.410 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:36:02.410 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299399+0000", 2021-02-10T04:36:02.410 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6042999999999999e-05 2021-02-10T04:36:02.410 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.410 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.410 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:36:02.410 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299455+0000", 2021-02-10T04:36:02.411 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.2917999999999994e-05 2021-02-10T04:36:02.411 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.411 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.411 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:36:02.411 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299548+0000", 2021-02-10T04:36:02.411 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0037641939999999998 2021-02-10T04:36:02.411 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.411 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.411 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 3,6", 2021-02-10T04:36:02.411 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.303312+0000", 2021-02-10T04:36:02.411 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0044879940000000004 2021-02-10T04:36:02.411 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.412 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.412 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:36:02.412 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.307800+0000", 2021-02-10T04:36:02.412 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0023093699999999998 2021-02-10T04:36:02.412 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.412 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.412 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:36:02.412 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.310110+0000", 2021-02-10T04:36:02.412 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0017106650000000001 2021-02-10T04:36:02.412 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.412 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.412 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:36:02.412 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.311820+0000", 2021-02-10T04:36:02.412 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.2655000000000003e-05 2021-02-10T04:36:02.413 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.413 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.413 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:36:02.413 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.311873+0000", 2021-02-10T04:36:02.413 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6504e-05 2021-02-10T04:36:02.413 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.413 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.413 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:36:02.413 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.311930+0000", 2021-02-10T04:36:02.413 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012622955 2021-02-10T04:36:02.413 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.413 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:36:02.413 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.414 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.414 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.414 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:554 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e127)", 2021-02-10T04:36:02.414 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:03.299463+0000", 2021-02-10T04:36:02.414 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 419.07942183699998, 2021-02-10T04:36:02.414 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012735955, 2021-02-10T04:36:02.414 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:36:02.414 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:36:02.414 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:36:02.414 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:36:02.414 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:36:02.414 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 554 2021-02-10T04:36:02.414 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.415 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:36:02.415 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.415 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:36:02.415 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299463+0000", 2021-02-10T04:36:02.415 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:36:02.415 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.415 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.415 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:36:02.415 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299463+0000", 2021-02-10T04:36:02.415 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.3636e-05 2021-02-10T04:36:02.415 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.415 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.415 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:36:02.416 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299506+0000", 2021-02-10T04:36:02.416 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.7109999999999998e-06 2021-02-10T04:36:02.416 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.416 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.416 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:36:02.416 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299513+0000", 2021-02-10T04:36:02.416 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7858000000000001e-05 2021-02-10T04:36:02.416 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.416 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.416 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:36:02.416 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299531+0000", 2021-02-10T04:36:02.416 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9790000000000001e-06 2021-02-10T04:36:02.416 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.416 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.417 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:36:02.417 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299536+0000", 2021-02-10T04:36:02.417 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00019002799999999999 2021-02-10T04:36:02.417 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.417 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.417 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:36:02.417 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299726+0000", 2021-02-10T04:36:02.417 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5302000000000001e-05 2021-02-10T04:36:02.417 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.417 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.417 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:36:02.417 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.299771+0000", 2021-02-10T04:36:02.417 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0078913030000000006 2021-02-10T04:36:02.418 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.418 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.418 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:36:02.418 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.307663+0000", 2021-02-10T04:36:02.418 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.8472000000000002e-05 2021-02-10T04:36:02.418 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.418 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.418 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2021-02-10T04:36:02.418 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.307761+0000", 2021-02-10T04:36:02.418 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0042153260000000001 2021-02-10T04:36:02.418 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.418 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.418 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:36:02.418 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.311976+0000", 2021-02-10T04:36:02.419 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.6063999999999994e-05 2021-02-10T04:36:02.419 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.419 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.419 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:36:02.419 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.312072+0000", 2021-02-10T04:36:02.419 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00012627600000000001 2021-02-10T04:36:02.419 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.419 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.419 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:36:02.419 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:03.312199+0000", 2021-02-10T04:36:02.419 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012735955 2021-02-10T04:36:02.419 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.419 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:36:02.420 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.420 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.420 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.420 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:558 7.3 e129/93 7:cf4c8c3b:::smithi08575397-6:head v 129'48, mlcod=123'44)", 2021-02-10T04:36:02.420 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:05.162978+0000", 2021-02-10T04:36:02.420 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 417.21590692299998, 2021-02-10T04:36:02.420 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0073914510000000003, 2021-02-10T04:36:02.420 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:36:02.420 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:36:02.420 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:36:02.420 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.420 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:36:02.420 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.162978+0000", 2021-02-10T04:36:02.420 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:36:02.421 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.421 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.421 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:36:02.421 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.162978+0000", 2021-02-10T04:36:02.421 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0044459269999999997 2021-02-10T04:36:02.421 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.421 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.421 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:36:02.421 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.167424+0000", 2021-02-10T04:36:02.421 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.3649999999999993e-06 2021-02-10T04:36:02.421 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.421 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.421 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:36:02.422 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.167433+0000", 2021-02-10T04:36:02.422 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1695999999999999e-05 2021-02-10T04:36:02.422 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.422 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.422 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:36:02.422 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.167445+0000", 2021-02-10T04:36:02.422 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.8929999999999999e-06 2021-02-10T04:36:02.422 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.422 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.422 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:36:02.422 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.167452+0000", 2021-02-10T04:36:02.422 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.0009000000000002e-05 2021-02-10T04:36:02.422 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.422 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.423 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:36:02.423 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.167502+0000", 2021-02-10T04:36:02.423 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0502000000000001e-05 2021-02-10T04:36:02.423 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.423 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.423 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:36:02.423 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.167522+0000", 2021-02-10T04:36:02.423 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0028030339999999998 2021-02-10T04:36:02.423 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.423 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.423 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:36:02.423 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.170325+0000", 2021-02-10T04:36:02.423 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.4024999999999999e-05 2021-02-10T04:36:02.424 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.424 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.424 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:36:02.424 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:05.170369+0000", 2021-02-10T04:36:02.424 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0073914510000000003 2021-02-10T04:36:02.424 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.424 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:36:02.424 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.424 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.424 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.424 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:1175 7.3 e176/93 7:cf4c8c3b:::smithi08575397-6:head v 176'131, mlcod=172'127)", 2021-02-10T04:36:02.424 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:29:52.233097+0000", 2021-02-10T04:36:02.424 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 370.14578733399998, 2021-02-10T04:36:02.425 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011081865, 2021-02-10T04:36:02.425 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:36:02.425 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:36:02.425 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:36:02.425 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.425 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:36:02.425 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.233097+0000", 2021-02-10T04:36:02.425 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:36:02.425 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.425 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.426 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:36:02.426 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.233097+0000", 2021-02-10T04:36:02.426 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.005275587 2021-02-10T04:36:02.427 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.427 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.427 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:36:02.427 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.238373+0000", 2021-02-10T04:36:02.427 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.367e-05 2021-02-10T04:36:02.427 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.427 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.427 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:36:02.427 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.238386+0000", 2021-02-10T04:36:02.427 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5914e-05 2021-02-10T04:36:02.427 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.427 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.427 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:36:02.428 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.238402+0000", 2021-02-10T04:36:02.428 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.6599999999999995e-06 2021-02-10T04:36:02.428 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.428 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.428 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:36:02.428 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.238410+0000", 2021-02-10T04:36:02.428 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.1702000000000001e-05 2021-02-10T04:36:02.428 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.428 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.428 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:36:02.428 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.238462+0000", 2021-02-10T04:36:02.428 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.6571e-05 2021-02-10T04:36:02.428 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.429 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.429 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:36:02.429 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.238488+0000", 2021-02-10T04:36:02.429 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0056189509999999996 2021-02-10T04:36:02.429 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.429 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.429 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:36:02.429 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.244107+0000", 2021-02-10T04:36:02.429 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1810000000000005e-05 2021-02-10T04:36:02.429 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.429 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.429 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:36:02.429 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:29:52.244179+0000", 2021-02-10T04:36:02.429 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011081865 2021-02-10T04:36:02.430 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.430 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:36:02.430 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.430 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.430 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.430 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:1672 7.9 e213/93 7:93db45b2:::smithi08575397-44:head v 213'82, mlcod=182'80)", 2021-02-10T04:36:02.430 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:30:29.863683+0000", 2021-02-10T04:36:02.430 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 332.515201413, 2021-02-10T04:36:02.430 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0069915899999999998, 2021-02-10T04:36:02.430 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:36:02.430 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:36:02.430 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:36:02.431 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.431 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:36:02.431 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:29.863683+0000", 2021-02-10T04:36:02.431 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:36:02.431 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.431 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.431 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:36:02.431 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:29.863683+0000", 2021-02-10T04:36:02.431 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0034923749999999998 2021-02-10T04:36:02.431 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.431 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.431 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:36:02.431 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:29.867176+0000", 2021-02-10T04:36:02.432 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2147000000000001e-05 2021-02-10T04:36:02.432 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.432 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.432 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:36:02.432 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:29.867188+0000", 2021-02-10T04:36:02.432 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8363999999999999e-05 2021-02-10T04:36:02.432 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.432 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.432 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:36:02.432 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:29.867206+0000", 2021-02-10T04:36:02.432 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.3549999999999994e-06 2021-02-10T04:36:02.432 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.432 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.432 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:36:02.433 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:29.867215+0000", 2021-02-10T04:36:02.433 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1843999999999998e-05 2021-02-10T04:36:02.433 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.433 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.433 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:36:02.433 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:29.867297+0000", 2021-02-10T04:36:02.433 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.0994000000000001e-05 2021-02-10T04:36:02.433 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.433 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.433 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:36:02.433 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:29.867338+0000", 2021-02-10T04:36:02.433 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.003267547 2021-02-10T04:36:02.433 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.434 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.434 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:36:02.434 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:29.870606+0000", 2021-02-10T04:36:02.434 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.8963999999999999e-05 2021-02-10T04:36:02.434 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.434 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.434 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:36:02.434 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:29.870675+0000", 2021-02-10T04:36:02.434 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0069915899999999998 2021-02-10T04:36:02.434 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.434 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:36:02.434 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.434 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.435 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.435 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:1772 7.1 7:8eb2e6b5:::smithi08575397-27:head [read 0~3899224,checksum] snapc 0=[] ondisk+read+known_if_redirected e222)", 2021-02-10T04:36:02.435 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:30:38.224445+0000", 2021-02-10T04:36:02.435 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 324.15443987899999, 2021-02-10T04:36:02.435 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0062361650000000001, 2021-02-10T04:36:02.435 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:36:02.435 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:36:02.435 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:36:02.435 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:36:02.435 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:36:02.435 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 1772 2021-02-10T04:36:02.435 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.435 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:36:02.436 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.436 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:36:02.436 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:38.224445+0000", 2021-02-10T04:36:02.436 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:36:02.436 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.436 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.436 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:36:02.436 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:38.224445+0000", 2021-02-10T04:36:02.436 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.0787999999999997e-05 2021-02-10T04:36:02.436 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.436 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.436 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:36:02.436 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:38.224475+0000", 2021-02-10T04:36:02.437 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.371e-06 2021-02-10T04:36:02.437 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.437 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.437 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:36:02.437 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:38.224479+0000", 2021-02-10T04:36:02.437 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.0129999999999996e-06 2021-02-10T04:36:02.437 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.437 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.437 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:36:02.437 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:38.224486+0000", 2021-02-10T04:36:02.437 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.9890000000000003e-06 2021-02-10T04:36:02.437 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.437 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.437 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:36:02.438 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:38.224490+0000", 2021-02-10T04:36:02.438 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0046025010000000002 2021-02-10T04:36:02.438 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.438 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.438 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:36:02.438 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:38.229092+0000", 2021-02-10T04:36:02.438 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.8963999999999996e-05 2021-02-10T04:36:02.438 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.438 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.438 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:36:02.438 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:38.229191+0000", 2021-02-10T04:36:02.438 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001489539 2021-02-10T04:36:02.438 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.439 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.439 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:36:02.439 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:30:38.230681+0000", 2021-02-10T04:36:02.439 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0062361650000000001 2021-02-10T04:36:02.439 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.439 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:36:02.439 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.439 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.439 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.439 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:2606 7.4 e283/93 7:2dcefd69:::smithi08575397-28:head v 283'148, mlcod=283'146)", 2021-02-10T04:36:02.439 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:31:40.554364+0000", 2021-02-10T04:36:02.439 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 261.824521006, 2021-02-10T04:36:02.439 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0089953659999999994, 2021-02-10T04:36:02.440 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:36:02.440 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:36:02.440 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:36:02.440 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.440 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:36:02.440 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:31:40.554364+0000", 2021-02-10T04:36:02.440 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:36:02.440 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.440 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.440 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:36:02.440 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:31:40.554364+0000", 2021-02-10T04:36:02.440 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.003679468 2021-02-10T04:36:02.440 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.440 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.441 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:36:02.441 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:31:40.558043+0000", 2021-02-10T04:36:02.441 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3509000000000001e-05 2021-02-10T04:36:02.441 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.441 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.441 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:36:02.441 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:31:40.558057+0000", 2021-02-10T04:36:02.441 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6458000000000001e-05 2021-02-10T04:36:02.441 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.441 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.441 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:36:02.441 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:31:40.558073+0000", 2021-02-10T04:36:02.441 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.0969999999999995e-06 2021-02-10T04:36:02.442 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.442 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.442 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:36:02.442 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:31:40.558082+0000", 2021-02-10T04:36:02.442 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.3733999999999997e-05 2021-02-10T04:36:02.442 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.442 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.442 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:36:02.442 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:31:40.558136+0000", 2021-02-10T04:36:02.442 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4589000000000001e-05 2021-02-10T04:36:02.442 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.442 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.442 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:36:02.443 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:31:40.558160+0000", 2021-02-10T04:36:02.443 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0051118680000000003 2021-02-10T04:36:02.443 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.443 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.443 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:36:02.443 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:31:40.563272+0000", 2021-02-10T04:36:02.443 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.6643000000000004e-05 2021-02-10T04:36:02.443 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.443 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.443 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:36:02.443 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:31:40.563359+0000", 2021-02-10T04:36:02.443 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0089953659999999994 2021-02-10T04:36:02.443 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.444 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:36:02.444 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.444 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.444 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.444 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:3142 7.3 e332/93 7:cf4c8c3b:::smithi08575397-6:head v 332'411, mlcod=332'407)", 2021-02-10T04:36:02.444 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:32:29.485052+0000", 2021-02-10T04:36:02.444 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 212.89383265399999, 2021-02-10T04:36:02.444 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.007547916, 2021-02-10T04:36:02.444 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:36:02.444 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:36:02.444 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:36:02.444 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.444 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:36:02.445 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:32:29.485052+0000", 2021-02-10T04:36:02.445 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:36:02.445 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.445 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.445 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:36:02.445 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:32:29.485052+0000", 2021-02-10T04:36:02.445 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0039315610000000001 2021-02-10T04:36:02.445 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.445 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.445 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:36:02.445 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:32:29.488983+0000", 2021-02-10T04:36:02.445 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.272e-06 2021-02-10T04:36:02.445 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.446 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.446 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:36:02.446 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:32:29.488991+0000", 2021-02-10T04:36:02.446 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.0480000000000004e-06 2021-02-10T04:36:02.446 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.446 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.446 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:36:02.446 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:32:29.489000+0000", 2021-02-10T04:36:02.446 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.2319999999999998e-06 2021-02-10T04:36:02.446 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.446 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.446 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:36:02.446 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:32:29.489006+0000", 2021-02-10T04:36:02.447 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.7348999999999997e-05 2021-02-10T04:36:02.447 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.447 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.447 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:36:02.447 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:32:29.489053+0000", 2021-02-10T04:36:02.447 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.9835000000000001e-05 2021-02-10T04:36:02.447 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.447 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.447 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:36:02.447 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:32:29.489083+0000", 2021-02-10T04:36:02.447 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0034677789999999998 2021-02-10T04:36:02.447 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.447 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.448 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:36:02.448 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:32:29.492551+0000", 2021-02-10T04:36:02.448 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.884e-05 2021-02-10T04:36:02.448 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.448 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.448 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:36:02.448 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:32:29.492600+0000", 2021-02-10T04:36:02.448 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.007547916 2021-02-10T04:36:02.448 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.448 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:36:02.448 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.448 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.448 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.448 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:3179 7.4 e335/93 7:2dcefd69:::smithi08575397-28:head v 335'178, mlcod=313'175)", 2021-02-10T04:36:02.449 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:32:33.254466+0000", 2021-02-10T04:36:02.449 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 209.124418835, 2021-02-10T04:36:02.449 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0060880190000000001, 2021-02-10T04:36:02.449 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:36:02.449 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:36:02.449 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:36:02.449 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.449 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:36:02.449 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:32:33.254466+0000", 2021-02-10T04:36:02.449 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:36:02.449 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.449 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.449 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:36:02.450 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:32:33.254466+0000", 2021-02-10T04:36:02.450 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.003037709 2021-02-10T04:36:02.450 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.450 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.450 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:36:02.450 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:32:33.257503+0000", 2021-02-10T04:36:02.450 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.5329999999999997e-06 2021-02-10T04:36:02.450 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.450 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.450 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:36:02.450 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:32:33.257511+0000", 2021-02-10T04:36:02.450 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0852e-05 2021-02-10T04:36:02.450 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.450 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.451 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:36:02.451 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:32:33.257522+0000", 2021-02-10T04:36:02.451 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.6869999999999994e-06 2021-02-10T04:36:02.451 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.451 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.451 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:36:02.451 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:32:33.257530+0000", 2021-02-10T04:36:02.451 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.3340000000000002e-05 2021-02-10T04:36:02.451 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.451 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.451 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:36:02.451 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:32:33.257573+0000", 2021-02-10T04:36:02.451 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0445000000000002e-05 2021-02-10T04:36:02.452 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.452 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.452 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:36:02.452 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:32:33.257593+0000", 2021-02-10T04:36:02.452 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0028955059999999999 2021-02-10T04:36:02.452 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.452 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.452 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:36:02.452 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:32:33.260489+0000", 2021-02-10T04:36:02.452 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.4947e-05 2021-02-10T04:36:02.452 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.452 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.452 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:36:02.452 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:32:33.260554+0000", 2021-02-10T04:36:02.453 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0060880190000000001 2021-02-10T04:36:02.453 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.453 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:36:02.453 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.453 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.453 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.453 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:3237 7.3 e340/93 7:cf4c8c3b:::smithi08575397-6:head v 340'433, mlcod=339'429)", 2021-02-10T04:36:02.453 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:32:37.585846+0000", 2021-02-10T04:36:02.453 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 204.79303832599999, 2021-02-10T04:36:02.453 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0069367589999999998, 2021-02-10T04:36:02.453 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:36:02.453 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:36:02.453 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:36:02.454 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.454 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:36:02.454 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:32:37.585846+0000", 2021-02-10T04:36:02.454 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:36:02.454 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.454 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.454 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:36:02.454 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:32:37.585846+0000", 2021-02-10T04:36:02.454 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.003272329 2021-02-10T04:36:02.454 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.454 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.454 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:36:02.454 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:32:37.589119+0000", 2021-02-10T04:36:02.455 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.5760000000000002e-06 2021-02-10T04:36:02.455 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.455 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.455 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:36:02.455 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:32:37.589125+0000", 2021-02-10T04:36:02.455 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.824e-06 2021-02-10T04:36:02.455 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.455 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.455 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:36:02.455 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:32:37.589133+0000", 2021-02-10T04:36:02.455 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1300000000000003e-06 2021-02-10T04:36:02.455 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.455 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.455 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:36:02.456 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:32:37.589140+0000", 2021-02-10T04:36:02.456 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.2003000000000003e-05 2021-02-10T04:36:02.456 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.456 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.456 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:36:02.456 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:32:37.589192+0000", 2021-02-10T04:36:02.456 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.9893999999999999e-05 2021-02-10T04:36:02.456 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.456 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.456 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:36:02.456 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:32:37.589222+0000", 2021-02-10T04:36:02.456 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0035110929999999999 2021-02-10T04:36:02.457 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.457 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.457 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:36:02.457 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:32:37.592733+0000", 2021-02-10T04:36:02.457 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9910000000000002e-05 2021-02-10T04:36:02.457 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.457 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.457 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:36:02.457 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:32:37.592783+0000", 2021-02-10T04:36:02.457 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0069367589999999998 2021-02-10T04:36:02.457 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.457 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:36:02.457 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.457 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.458 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.458 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:3630 7.3 e372/93 7:cd29c741:::smithi08575397-46:head v 372'529, mlcod=371'525)", 2021-02-10T04:36:02.458 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:33:09.170397+0000", 2021-02-10T04:36:02.458 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 173.20848796000001, 2021-02-10T04:36:02.458 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.007091918, 2021-02-10T04:36:02.458 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:36:02.458 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:36:02.458 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:36:02.458 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.458 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:36:02.458 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:33:09.170397+0000", 2021-02-10T04:36:02.458 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:36:02.458 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.459 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.459 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:36:02.459 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:33:09.170397+0000", 2021-02-10T04:36:02.459 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0031900819999999999 2021-02-10T04:36:02.459 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.459 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.459 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:36:02.459 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:33:09.173587+0000", 2021-02-10T04:36:02.459 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.9520000000000006e-06 2021-02-10T04:36:02.459 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.459 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.459 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:36:02.459 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:33:09.173597+0000", 2021-02-10T04:36:02.460 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1296e-05 2021-02-10T04:36:02.460 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.460 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.460 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:36:02.460 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:33:09.173608+0000", 2021-02-10T04:36:02.460 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.0110000000000006e-06 2021-02-10T04:36:02.460 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.460 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.460 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:36:02.460 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:33:09.173617+0000", 2021-02-10T04:36:02.460 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.5831999999999997e-05 2021-02-10T04:36:02.460 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.460 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.461 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:36:02.461 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:33:09.173673+0000", 2021-02-10T04:36:02.461 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.1160000000000001e-05 2021-02-10T04:36:02.461 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.461 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.461 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:36:02.461 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:33:09.173694+0000", 2021-02-10T04:36:02.461 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0037277709999999999 2021-02-10T04:36:02.461 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.461 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.461 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:36:02.461 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:33:09.177422+0000", 2021-02-10T04:36:02.461 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6814000000000001e-05 2021-02-10T04:36:02.461 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.462 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.462 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:36:02.462 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:33:09.177489+0000", 2021-02-10T04:36:02.462 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.007091918 2021-02-10T04:36:02.462 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.462 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:36:02.462 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.462 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.462 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.462 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:3810 7.3 e393/93 7:c1415b8e:::smithi08575397-38:head v 393'573, mlcod=391'570)", 2021-02-10T04:36:02.462 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:33:30.413978+0000", 2021-02-10T04:36:02.462 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 151.964906762, 2021-02-10T04:36:02.462 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0062524310000000001, 2021-02-10T04:36:02.463 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:36:02.463 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:36:02.463 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:36:02.463 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.463 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:36:02.463 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:33:30.413978+0000", 2021-02-10T04:36:02.463 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:36:02.463 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.463 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.463 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:36:02.463 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:33:30.413978+0000", 2021-02-10T04:36:02.463 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0029339470000000001 2021-02-10T04:36:02.463 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.464 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.464 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:36:02.464 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:33:30.416912+0000", 2021-02-10T04:36:02.464 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0801e-05 2021-02-10T04:36:02.464 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.464 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.464 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:36:02.464 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:33:30.416923+0000", 2021-02-10T04:36:02.464 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5432000000000001e-05 2021-02-10T04:36:02.464 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.464 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.464 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:36:02.464 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:33:30.416938+0000", 2021-02-10T04:36:02.464 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0567000000000001e-05 2021-02-10T04:36:02.465 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.465 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.465 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:36:02.465 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:33:30.416949+0000", 2021-02-10T04:36:02.465 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.1475000000000003e-05 2021-02-10T04:36:02.465 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.465 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.465 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:36:02.465 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:33:30.417010+0000", 2021-02-10T04:36:02.465 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.1987000000000001e-05 2021-02-10T04:36:02.465 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.465 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.465 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:36:02.466 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:33:30.417032+0000", 2021-02-10T04:36:02.466 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.003154043 2021-02-10T04:36:02.466 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.466 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.466 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:36:02.466 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:33:30.420186+0000", 2021-02-10T04:36:02.466 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.4178999999999998e-05 2021-02-10T04:36:02.466 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.466 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.466 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:36:02.466 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:33:30.420230+0000", 2021-02-10T04:36:02.466 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0062524310000000001 2021-02-10T04:36:02.466 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.467 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:36:02.467 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.467 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.467 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.467 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:5292 7.9 e530/93 7:93db45b2:::smithi08575397-44:head v 530'270, mlcod=527'266)", 2021-02-10T04:36:02.467 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:35:48.071222+0000", 2021-02-10T04:36:02.467 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 14.307662435999999, 2021-02-10T04:36:02.467 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.007640333, 2021-02-10T04:36:02.467 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:36:02.467 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:36:02.467 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:36:02.467 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.467 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:36:02.467 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:48.071222+0000", 2021-02-10T04:36:02.468 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:36:02.468 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.468 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.468 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:36:02.468 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:48.071222+0000", 2021-02-10T04:36:02.468 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0039200750000000003 2021-02-10T04:36:02.468 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.468 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.468 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:36:02.468 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:48.075142+0000", 2021-02-10T04:36:02.468 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0345e-05 2021-02-10T04:36:02.468 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.468 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.468 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:36:02.469 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:48.075153+0000", 2021-02-10T04:36:02.469 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0157999999999999e-05 2021-02-10T04:36:02.469 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.469 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.469 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:36:02.469 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:48.075173+0000", 2021-02-10T04:36:02.469 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0556e-05 2021-02-10T04:36:02.469 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.469 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.469 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:36:02.469 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:48.075183+0000", 2021-02-10T04:36:02.469 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.1347999999999997e-05 2021-02-10T04:36:02.469 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.470 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.470 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:36:02.470 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:48.075235+0000", 2021-02-10T04:36:02.470 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2847e-05 2021-02-10T04:36:02.470 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.470 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.470 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:36:02.470 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:48.075257+0000", 2021-02-10T04:36:02.470 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0035359549999999999 2021-02-10T04:36:02.470 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.470 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.470 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:36:02.470 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:48.078793+0000", 2021-02-10T04:36:02.471 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.9048999999999996e-05 2021-02-10T04:36:02.471 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.471 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.471 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:36:02.471 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:48.078862+0000", 2021-02-10T04:36:02.471 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.007640333 2021-02-10T04:36:02.471 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.471 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:36:02.471 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.471 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.471 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.471 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.24382.0:1166 4.d 4:b3052b9f:reshard::reshard.0000000013:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e537)", 2021-02-10T04:36:02.471 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:35:55.266242+0000", 2021-02-10T04:36:02.472 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 7.1126429050000004, 2021-02-10T04:36:02.472 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0056989789999999999, 2021-02-10T04:36:02.472 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:36:02.472 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:36:02.472 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:36:02.472 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.24382", 2021-02-10T04:36:02.472 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/3915233377", 2021-02-10T04:36:02.472 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 1166 2021-02-10T04:36:02.472 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.472 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:36:02.472 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.472 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:36:02.472 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:55.266242+0000", 2021-02-10T04:36:02.473 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:36:02.473 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.473 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.473 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:36:02.473 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:55.266242+0000", 2021-02-10T04:36:02.473 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.9409999999999997e-05 2021-02-10T04:36:02.473 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.473 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.473 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:36:02.473 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:55.266281+0000", 2021-02-10T04:36:02.473 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5619999999999997e-06 2021-02-10T04:36:02.473 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.473 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.473 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:36:02.474 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:55.266286+0000", 2021-02-10T04:36:02.474 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7989999999999994e-06 2021-02-10T04:36:02.474 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.474 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.474 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:36:02.474 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:55.266293+0000", 2021-02-10T04:36:02.474 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.3760000000000001e-06 2021-02-10T04:36:02.474 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.474 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.474 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:36:02.474 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:55.266298+0000", 2021-02-10T04:36:02.474 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00027532700000000002 2021-02-10T04:36:02.474 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.475 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.475 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:36:02.475 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:55.266573+0000", 2021-02-10T04:36:02.475 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1346000000000004e-05 2021-02-10T04:36:02.475 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.475 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.475 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:36:02.475 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:55.266644+0000", 2021-02-10T04:36:02.475 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0041080810000000004 2021-02-10T04:36:02.475 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.475 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.475 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 1,2", 2021-02-10T04:36:02.475 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:55.270753+0000", 2021-02-10T04:36:02.475 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0010341389999999999 2021-02-10T04:36:02.476 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.476 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.476 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:36:02.476 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:55.271787+0000", 2021-02-10T04:36:02.476 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.943e-05 2021-02-10T04:36:02.476 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.476 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.476 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:36:02.476 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:55.271826+0000", 2021-02-10T04:36:02.476 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0621e-05 2021-02-10T04:36:02.476 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.476 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.476 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:36:02.477 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:55.271837+0000", 2021-02-10T04:36:02.477 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.8593e-05 2021-02-10T04:36:02.477 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.477 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.477 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:36:02.477 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:55.271895+0000", 2021-02-10T04:36:02.477 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5294999999999999e-05 2021-02-10T04:36:02.477 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.477 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.477 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:36:02.477 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:55.271941+0000", 2021-02-10T04:36:02.477 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0056989789999999999 2021-02-10T04:36:02.477 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.477 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:36:02.478 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.478 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.478 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.478 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.24382.0:1385 4.13 4:cc909b6b:::obj_delete_at_hint.0000000106:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e538)", 2021-02-10T04:36:02.478 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:35:56.137730+0000", 2021-02-10T04:36:02.478 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 6.2411547949999999, 2021-02-10T04:36:02.478 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0045290840000000001, 2021-02-10T04:36:02.478 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:36:02.478 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:36:02.478 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:36:02.478 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.24382", 2021-02-10T04:36:02.478 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/3915233377", 2021-02-10T04:36:02.478 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 1385 2021-02-10T04:36:02.479 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.479 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:36:02.479 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.479 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:36:02.479 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:56.137730+0000", 2021-02-10T04:36:02.479 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:36:02.479 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.479 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.479 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:36:02.479 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:56.137730+0000", 2021-02-10T04:36:02.479 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.9999000000000001e-05 2021-02-10T04:36:02.479 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.479 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.479 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:36:02.480 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:56.137770+0000", 2021-02-10T04:36:02.480 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.0679999999999999e-06 2021-02-10T04:36:02.480 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.480 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.480 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:36:02.480 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:56.137777+0000", 2021-02-10T04:36:02.480 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6467e-05 2021-02-10T04:36:02.480 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.480 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.480 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:36:02.480 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:56.137793+0000", 2021-02-10T04:36:02.480 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.2709999999999998e-06 2021-02-10T04:36:02.480 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.481 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.481 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:36:02.481 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:56.137803+0000", 2021-02-10T04:36:02.481 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.0948000000000001e-05 2021-02-10T04:36:02.481 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.481 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.481 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:36:02.481 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:56.137854+0000", 2021-02-10T04:36:02.481 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.8041999999999994e-05 2021-02-10T04:36:02.481 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.481 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.481 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:36:02.481 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:56.137952+0000", 2021-02-10T04:36:02.481 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00017618199999999999 2021-02-10T04:36:02.482 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.482 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.482 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 1,6", 2021-02-10T04:36:02.482 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:56.138128+0000", 2021-02-10T04:36:02.482 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00046283700000000001 2021-02-10T04:36:02.482 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.482 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.482 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:36:02.482 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:56.138591+0000", 2021-02-10T04:36:02.482 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00034823200000000002 2021-02-10T04:36:02.482 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.482 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.482 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:36:02.483 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:56.138939+0000", 2021-02-10T04:36:02.483 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0032396629999999998 2021-02-10T04:36:02.483 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.483 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.483 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:36:02.483 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:56.142178+0000", 2021-02-10T04:36:02.483 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.7808999999999997e-05 2021-02-10T04:36:02.483 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.483 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.483 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:36:02.483 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:56.142216+0000", 2021-02-10T04:36:02.483 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2565999999999999e-05 2021-02-10T04:36:02.483 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:36:02.483 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:36:02.484 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:36:02.484 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:35:56.142259+0000", 2021-02-10T04:36:02.484 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0045290840000000001 2021-02-10T04:36:02.484 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.484 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:36:02.484 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.484 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:36:02.484 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:36:02.484 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:36:02.485 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:01 smithi085 conmon[49106]: debug 2021-02-10T04:36:01.980+0000 7f77a3e94700 -1 osd.2 544 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:02.522 INFO:tasks.rados.rados.0.smithi085.stdout:2093: done (10 left) 2021-02-10T04:36:02.523 INFO:tasks.rados.rados.0.smithi085.stdout:2094: write (excl) oid 2 current snap is 234 2021-02-10T04:36:02.523 INFO:tasks.rados.rados.0.smithi085.stdout:2091: finishing write tid 2 to smithi08575397-46 2021-02-10T04:36:02.523 INFO:tasks.rados.rados.0.smithi085.stdout:2094: seq_num 735 ranges {753145=477903,2010618=342527} 2021-02-10T04:36:02.530 INFO:tasks.rados.rados.0.smithi085.stdout:2094: writing smithi08575397-2 from 753145 to 1231048 tid 1 2021-02-10T04:36:02.539 INFO:tasks.rados.rados.0.smithi085.stdout:2094: writing smithi08575397-2 from 2010618 to 2353145 tid 2 2021-02-10T04:36:02.539 INFO:tasks.rados.rados.0.smithi085.stdout:2095: delete oid 1 current snap is 234 2021-02-10T04:36:02.539 INFO:tasks.rados.rados.0.smithi085.stdout:2091: finishing write tid 3 to smithi08575397-46 2021-02-10T04:36:02.539 INFO:tasks.rados.rados.0.smithi085.stdout:2091: finishing write tid 4 to smithi08575397-46 2021-02-10T04:36:02.540 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 1044 (ObjNum 733 snap 234 seq_num 733) dirty exists 2021-02-10T04:36:02.540 INFO:tasks.rados.rados.0.smithi085.stdout:2091: left oid 46 (ObjNum 733 snap 234 seq_num 733) 2021-02-10T04:36:02.540 INFO:tasks.rados.rados.0.smithi085.stdout:2094: finishing write tid 1 to smithi08575397-2 2021-02-10T04:36:02.541 INFO:tasks.rados.rados.0.smithi085.stdout:2091: done (11 left) 2021-02-10T04:36:02.541 INFO:tasks.rados.rados.0.smithi085.stdout:2095: done (10 left) 2021-02-10T04:36:02.542 INFO:tasks.rados.rados.0.smithi085.stdout:2096: snap_create 2021-02-10T04:36:02.542 INFO:tasks.rados.rados.0.smithi085.stdout:2094: finishing write tid 2 to smithi08575397-2 2021-02-10T04:36:02.543 INFO:tasks.rados.rados.0.smithi085.stdout:2094: finishing write tid 3 to smithi08575397-2 2021-02-10T04:36:02.543 INFO:tasks.rados.rados.0.smithi085.stdout:2094: finishing write tid 4 to smithi08575397-2 2021-02-10T04:36:02.543 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 673 (ObjNum 735 snap 234 seq_num 735) dirty exists 2021-02-10T04:36:02.543 INFO:tasks.rados.rados.0.smithi085.stdout:2094: left oid 2 (ObjNum 735 snap 234 seq_num 735) 2021-02-10T04:36:02.843 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:02 smithi164 conmon[29288]: cluster 2021-02-10T04:36:01.141400+0000 osd.4 (osd.4) 783 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:02.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:02 smithi164 conmon[29288]: cluster 2021-02-10T04:36:01.141410+0000 osd.4 (osd.4) 784 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:02.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:02 smithi164 conmon[29288]: cluster 2021-02-10T04:36:01.141416+0000 osd.4 (osd.4) 785 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:02.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:02 smithi164 conmon[29288]: cluster 2021-02-10T04:36:01.141422+0000 osd.4 (osd.4) 786 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:02.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:02 smithi164 conmon[29288]: cluster 2021-02-10T04:36:01.510080+0000 mon.a (mon.0) 1378 : cluster [DBG] osdmap e544: 8 total, 8 up, 8 in 2021-02-10T04:36:02.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:02 smithi164 conmon[29288]: cluster 2021-02-10T04:36:01.982189+0000 osd.2 (osd. 2021-02-10T04:36:02.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:02 smithi164 conmon[29288]: 2) 1469 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:02.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:02 smithi164 conmon[29288]: cluster 2021-02-10T04:36:01.982199+0000 osd.2 (osd.2) 1470 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:02.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:02 smithi164 conmon[29288]: cluster 2021-02-10T04:36:01.982206+0000 osd.2 (osd.2) 1471 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:02.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:02 smithi164 conmon[29288]: cluster 2021-02-10T04:36:01.982213+0000 osd.2 (osd.2) 1472 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:02.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:02 smithi164 conmon[29288]: cluster 2021-02-10T04:36:01.982219+0000 osd.2 (osd.2) 1473 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:02.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:02 smithi164 conmon[29288]: cluster 2021-02-10T04:36:01.982225+0000 osd.2 (osd.2) 1474 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:02.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:02 smithi164 conmon[29288]: cluster 2021-02-10T04:36:01.982231+0000 osd.2 (osd.2) 1475 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:02.845 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:02 smithi164 conmon[29288]: cluster 2021-02-10T04:36:01.982236+0000 osd.2 (osd.2) 1476 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:02.845 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:02 smithi164 conmon[29288]: cluster 2021-02-10T04:36:01.982242+0000 osd.2 (osd.2) 1477 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:02.845 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:02 smithi164 conmon[29288]: cluster 2021-02-10T04:36:01.982248+0000 osd.2 (osd.2) 1478 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:02.845 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:02 smithi164 conmon[29288]: cluster 2021-02-10T04:36:02.110931+0000 mgr.y (mgr.14140) 618 : cluster [DBG] pgmap v966: 168 pgs: 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 156 active+clean; 415 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 0 B/s wr, 122 op/s 2021-02-10T04:36:02.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: cluster 2021-02-10T04:36:01.141400+0000 2021-02-10T04:36:02.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: osd.4 (osd.4) 783 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:02.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: cluster 2021-02-10T04:36:01 2021-02-10T04:36:02.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: .141410+0000 osd.4 (osd.4) 784 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:02.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:36:02.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: 36:01.141416+0000 osd.4 (osd.4 2021-02-10T04:36:02.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: ) 785 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:02.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:02.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: -02-10T04:36:01.141422+0000 2021-02-10T04:36:02.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: osd.4 (osd.4) 786 : cluster [WRN] 2021-02-10T04:36:02.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:02.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: cluster 2021-02-10T04:36:02.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:36:02.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: 10T04:36:01.510080+0000 2021-02-10T04:36:02.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: mon.a (mon.0) 1378 : cluster [DBG] 2021-02-10T04:36:02.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: osdmap e544: 8 total, 8 up, 8 in 2021-02-10T04:36:02.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:02.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: T04:36:01.982189+0000 osd.2 ( 2021-02-10T04:36:02.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: osd.2) 1469 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:02.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: 2021-02-10T04:36:02.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: cluster 2021-02-10T04:36 2021-02-10T04:36:02.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: :01.982199+0000 osd.2 (osd.2 2021-02-10T04:36:02.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: ) 1470 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:02.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: cluster 2021-02-10T04:36:02.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:36:02.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: T04:36:01.982206+0000 osd.2 ( 2021-02-10T04:36:02.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: osd.2) 1471 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:02.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: 2021-02-10T04:36:02.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: cluster 2021-02-10T04:36:02.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: 2021-02-10T04:36:01. 2021-02-10T04:36:02.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: 982213+0000 osd.2 (osd.2) 1472 : 2021-02-10T04:36:02.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:02.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: cluster 2021-02-10T04:36:02.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: 2021-02-10T04:36:01. 2021-02-10T04:36:02.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: 982219+0000 osd.2 (osd.2) 1473 : cluster 2021-02-10T04:36:02.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:02.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: cluster 2021-02-10T04:36:02.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:36:02.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: :36:01.982225+0000 osd.2 (osd 2021-02-10T04:36:02.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: .2) 1474 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:02.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:02.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: 02-10T04:36:01.982231+0000 2021-02-10T04:36:02.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: osd.2 (osd.2) 1475 : cluster 2021-02-10T04:36:02.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:02.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:36:02.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: 10T04:36:01.982236+0000 2021-02-10T04:36:02.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: osd.2 (osd.2) 1476 : cluster 2021-02-10T04:36:02.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:02.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:02.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: T04:36:01.982242+0000 osd.2 2021-02-10T04:36:02.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: (osd.2) 1477 : cluster [WRN] 2021-02-10T04:36:02.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:02.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:36:02.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: :36:01.982248+0000 osd.2 (osd 2021-02-10T04:36:02.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: .2) 1478 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:02.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:36:02.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: 10T04:36:02.110931+0000 mgr.y ( 2021-02-10T04:36:02.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[35961]: mgr.14140) 618 : cluster [DBG] pgmap v966: 168 pgs: 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 156 active+clean; 415 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 0 B/s wr, 122 op/s 2021-02-10T04:36:02.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: cluster 2021-02-10T04:36:01.141400+0000 osd.4 (osd. 2021-02-10T04:36:02.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: 4) 783 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:02.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: cluster 2021-02-10T04:36:01.141410+0000 osd.4 (osd.4) 784 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:02.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: cluster 2021-02-10T04:36:01.141416+0000 osd.4 2021-02-10T04:36:02.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: (osd.4) 785 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:02.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:02.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: 02-10T04:36:01.141422+0000 2021-02-10T04:36:02.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: osd.4 (osd.4) 786 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:02.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: cluster 2021-02-10T04:36:02.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: 2021-02-10T04:36:01.510080+0000 mon.a (mon.0) 1378 : 2021-02-10T04:36:02.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: cluster [DBG] osdmap e544: 8 total, 8 up, 8 in 2021-02-10T04:36:02.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:36:02.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: 36:01.982189+0000 osd.2 (osd.2) 2021-02-10T04:36:02.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: 1469 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:02.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:36:02.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: T04:36:01.982199+0000 osd.2 (osd.2) 1470 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:02.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:02.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: -02-10T04:36:01.982206+0000 osd.2 (osd.2) 1471 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:02.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:02.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: -02-10T04:36:01. 2021-02-10T04:36:02.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: 982213+0000 osd.2 (osd.2) 2021-02-10T04:36:02.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: 1472 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:02.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:02.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: -02-10T04:36: 2021-02-10T04:36:02.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: 01.982219+0000 osd.2 (osd 2021-02-10T04:36:02.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: .2) 1473 : cluster [WRN] 2021-02-10T04:36:02.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:02.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:02.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: -02-10T04:36:01.982225+0000 2021-02-10T04:36:02.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: osd.2 (osd.2) 1474 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:02.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:02.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: -10T04:36:01.982231+0000 osd.2 (osd 2021-02-10T04:36:02.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: .2) 1475 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:02.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:02.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: -02-10T04:36:01.982236+0000 osd.2 2021-02-10T04:36:02.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: (osd.2) 1476 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:02.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: cluster 2021-02-10T04:36 2021-02-10T04:36:02.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: :01.982242+0000 osd.2 (osd.2) 1477 2021-02-10T04:36:02.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:02.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:02.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: 02-10T04:36:01.982248+0000 osd.2 2021-02-10T04:36:02.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: (osd.2) 1478 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:02.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: cluster 2021-02-10T04:36:02.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: 2021-02-10T04:36:02 2021-02-10T04:36:02.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: .110931+0000 mgr.y (mgr.14140) 618 : cluster [DBG] 2021-02-10T04:36:02.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:02 smithi085 conmon[29319]: pgmap v966: 168 pgs: 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 156 active+clean; 415 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 0 B/s wr, 122 op/s 2021-02-10T04:36:03.143 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[33677]: debug 2021-02-10T04:36:03.143 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[33677]: 2021-02-10T04:36:03.090+0000 7fbec19a1700 -1 osd.4 545 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:03.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[33677]: 2021-02-10T04:36:03.170 INFO:tasks.rados.rados.0.smithi085.stdout:2094: done (10 left) 2021-02-10T04:36:03.171 INFO:tasks.rados.rados.0.smithi085.stdout:2096: done (9 left) 2021-02-10T04:36:03.171 INFO:tasks.rados.rados.0.smithi085.stdout:2097: delete oid 13 current snap is 235 2021-02-10T04:36:03.182 INFO:tasks.rados.rados.0.smithi085.stdout:2097: done (9 left) 2021-02-10T04:36:03.182 INFO:tasks.rados.rados.0.smithi085.stdout:2098: write (excl) oid 8 current snap is 235 2021-02-10T04:36:03.182 INFO:tasks.rados.rados.0.smithi085.stdout:2098: seq_num 738 ranges {403566=420003,1386064=546918} 2021-02-10T04:36:03.187 INFO:tasks.rados.rados.0.smithi085.stdout:2098: writing smithi08575397-8 from 403566 to 823569 tid 1 2021-02-10T04:36:03.193 INFO:tasks.rados.rados.0.smithi085.stdout:2098: writing smithi08575397-8 from 1386064 to 1932982 tid 2 2021-02-10T04:36:03.194 INFO:tasks.rados.rados.0.smithi085.stdout:2099: snap_remove snap 223 2021-02-10T04:36:03.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[49106]: debug 2021-02-10T04:36:03.014+0000 7f77a3e94700 -1 osd.2 545 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:03.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: cluster 2021-02-10T04:36:02.111314+0000 osd.4 (osd.4) 787 : 2021-02-10T04:36:03.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:03.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: cluster 2021-02-10T04:36:02.111323+0000 osd.4 (osd.4) 788 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:03.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: cluster 2021-02-10T04:36:02.111329+0000 osd.4 (osd.4) 789 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:03.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: cluster 2021-02-10T04:36:02.111335+0000 osd.4 (osd.4) 790 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:03.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: cluster 2021-02-10T04:36:02.515065+0000 mon.a (mon.0) 1379 : cluster [DBG] osdmap e545: 8 total, 8 up, 8 in 2021-02-10T04:36:03.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: cluster 2021-02-10T04:36: 2021-02-10T04:36:03.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: 03.015760+0000 osd.2 (osd.2) 1479 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:03.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: cluster 2021-02-10T04:36:03.015769+0000 osd.2 (osd.2) 1480 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:03.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: cluster 2021-02-10T04:36:03.015776+0000 osd.2 (osd.2) 1481 : cluster 2021-02-10T04:36:03.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:03.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:03.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: -10T04:36:03.015783+0000 osd.2 (osd.2) 1482 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:03.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:36:03.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: :36:03.015788+0000 osd.2 (osd.2) 1483 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:03.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:03.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: 02-10T04:36:03.015794+0000 osd.2 (osd.2 2021-02-10T04:36:03.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: ) 1484 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:03.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:03.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: -02-10T04:36:03.015800+0000 2021-02-10T04:36:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: osd.2 (osd.2) 1485 : cluster [WRN] 2021-02-10T04:36:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:36:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: 04:36:03.015806+0000 osd.2 (osd.2) 1486 : cluster 2021-02-10T04:36:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: cluster 2021-02-10T04:36:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: 2021-02-10T04:36:03.015811+0000 osd.2 (osd.2) 1487 : cluster [WRN] 2021-02-10T04:36:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: cluster 2021-02-10T04:36:03. 2021-02-10T04:36:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: 015817+0000 osd.2 (osd.2) 1488 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: cluster 2021-02-10T04:36:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: 2021-02-10T04:36:03.170407+0000 mon.a (mon.0) 1380 : cluster [DBG] osdmap e546: 8 total, 8 up, 8 in 2021-02-10T04:36:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:03 smithi164 conmon[29288]: 2021-02-10T04:36:03.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: cluster 2021-02-10T04:36:02.111314+0000 osd.4 (osd.4) 787 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:03.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: cluster 2021-02-10T04:36:02.111323+0000 osd.4 (osd.4) 788 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:03.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: cluster 2021-02-10T04:36:02.111329+0000 osd.4 ( 2021-02-10T04:36:03.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: osd.4) 789 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:03.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: cluster 2021-02-10T04:36:02 2021-02-10T04:36:03.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: .111335+0000 osd.4 (osd.4) 790 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:03.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:36:03.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: 04:36:02.515065+0000 mon.a (mon.0) 2021-02-10T04:36:03.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: 1379 : cluster [DBG] osdmap e545: 8 total, 8 up, 8 in 2021-02-10T04:36:03.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: cluster 2021-02-10T04:36 2021-02-10T04:36:03.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: :03.015760+0000 osd.2 (osd.2) 1479 : cluster 2021-02-10T04:36:03.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:03.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:03.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: -10T04:36:03.015769+0000 2021-02-10T04:36:03.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: osd.2 (osd.2) 1480 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:03.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:03.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: -02-10T04:36:03.015776 2021-02-10T04:36:03.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1481 : cluster 2021-02-10T04:36:03.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:03.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: cluster 2021-02-10T04:36 2021-02-10T04:36:03.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: :03.015783+0000 osd.2 (osd.2) 1482 2021-02-10T04:36:03.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:03.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:36:03.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: 04:36:03.015788+0000 osd.2 (osd.2 2021-02-10T04:36:03.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: ) 1483 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:03.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:03.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: 02-10T04:36:03.015794+0000 2021-02-10T04:36:03.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: osd.2 (osd.2) 1484 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:03.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: 2021-02-10T04:36:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: cluster 2021-02-10T04:36:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: 2021-02-10T04:36:03.015800+0000 2021-02-10T04:36:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: osd.2 (osd.2) 1485 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: 2021-02-10T04:36:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: cluster 2021-02-10T04:36:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: 2021-02-10T04:36: 2021-02-10T04:36:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: 03.015806+0000 osd.2 (osd.2) 1486 : cluster 2021-02-10T04:36:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: cluster 2021-02-10T04:36:03.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: 2021-02-10T04:36:03.015811+0000 2021-02-10T04:36:03.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: osd.2 (osd.2) 1487 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:03.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:36:03.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: 36:03.015817+0000 osd.2 (osd.2) 1488 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:03.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: 2021-02-10T04:36:03.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:03.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: -10T04:36:03.170407+0000 mon.a (mon.0) 1380 : cluster [DBG] osdmap e546: 8 total, 8 up, 8 in 2021-02-10T04:36:03.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[29319]: 2021-02-10T04:36:03.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: cluster 2021-02-10T04:36:03.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: 2021-02-10T04:36:03.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: 2021 2021-02-10T04:36:03.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: - 2021-02-10T04:36:03.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: 02-10T04:36:02.111314 2021-02-10T04:36:03.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 787 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:03.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: cluster 2021-02-10T04:36:03.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: 2021-02-10T04:36:02.111323+0000 osd.4 ( 2021-02-10T04:36:03.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: osd.4) 788 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:03.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:03.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: -02-10T04:36:02.111329+0000 2021-02-10T04:36:03.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: osd.4 (osd.4) 789 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:03.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:03.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: 02-10T04:36:02.111335+0000 osd.4 (osd. 2021-02-10T04:36:03.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: 4) 790 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:03.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: cluster 2021-02-10T04:36:03.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: 2021-02-10T04:36:02. 2021-02-10T04:36:03.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: 515065+0000 mon.a (mon.0) 1379 2021-02-10T04:36:03.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: : cluster 2021-02-10T04:36:03.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: [DBG] osdmap e545: 8 total, 8 up, 8 in 2021-02-10T04:36:03.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: 2021-02-10T04:36:03.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:03.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: -10 2021-02-10T04:36:03.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: T04:36:03.015760+0000 2021-02-10T04:36:03.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: osd.2 (osd.2) 1479 : 2021-02-10T04:36:03.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:03.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:03.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: -02-10T04:36:03.015769+0000 2021-02-10T04:36:03.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: osd.2 (osd.2) 1480 : cluster [WRN] 2021-02-10T04:36:03.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:03.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:36:03.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: 10T04:36:03.015776+0000 osd.2 (osd.2) 1481 2021-02-10T04:36:03.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:03.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:03.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: T04:36:03.015783+0000 osd.2 (osd 2021-02-10T04:36:03.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: .2) 1482 : cluster [WRN] 2021-02-10T04:36:03.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:03.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: cluster 2021-02-10T04:36 2021-02-10T04:36:03.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: :03.015788+0000 osd.2 (osd.2) 1483 : 2021-02-10T04:36:03.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:03.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:03.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: -10T04:36:03.015794+0000 2021-02-10T04:36:03.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: osd.2 (osd.2) 1484 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:03.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:03.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: -02-10T04:36:03.015800 2021-02-10T04:36:03.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1485 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:03.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:03.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: 02-10T04:36:03.015806+0000 osd.2 (osd.2 2021-02-10T04:36:03.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: ) 1486 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:03.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: cluster 2021-02-10T04:36 2021-02-10T04:36:03.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: :03.015811+0000 osd.2 (osd.2) 1487 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:03.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:03.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: -02-10T04:36:03 2021-02-10T04:36:03.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: .015817+0000 osd.2 (osd.2) 1488 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:03.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:03.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: T04:36:03.170407+0000 mon.a (mon.0) 1380 : cluster 2021-02-10T04:36:03.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:03 smithi085 conmon[35961]: [DBG] osdmap e546: 8 total, 8 up, 8 in 2021-02-10T04:36:04.181 INFO:tasks.rados.rados.0.smithi085.stdout:2099: done (10 left) 2021-02-10T04:36:04.181 INFO:tasks.rados.rados.0.smithi085.stdout:2100: write oid 48 current snap is 235 2021-02-10T04:36:04.182 INFO:tasks.rados.rados.0.smithi085.stdout:2100: seq_num 739 ranges {420373=495252,1444546=690576} 2021-02-10T04:36:04.187 INFO:tasks.rados.rados.0.smithi085.stdout:2100: writing smithi08575397-48 from 420373 to 915625 tid 1 2021-02-10T04:36:04.195 INFO:tasks.rados.rados.0.smithi085.stdout:2100: writing smithi08575397-48 from 1444546 to 2135122 tid 2 2021-02-10T04:36:04.196 INFO:tasks.rados.rados.0.smithi085.stdout:2098: finishing write tid 1 to smithi08575397-8 2021-02-10T04:36:04.196 INFO:tasks.rados.rados.0.smithi085.stdout:2098: finishing write tid 2 to smithi08575397-8 2021-02-10T04:36:04.196 INFO:tasks.rados.rados.0.smithi085.stdout:2098: finishing write tid 3 to smithi08575397-8 2021-02-10T04:36:04.196 INFO:tasks.rados.rados.0.smithi085.stdout:2098: finishing write tid 4 to smithi08575397-8 2021-02-10T04:36:04.196 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 1104 (ObjNum 738 snap 235 seq_num 738) dirty exists 2021-02-10T04:36:04.196 INFO:tasks.rados.rados.0.smithi085.stdout:2098: left oid 8 (ObjNum 738 snap 235 seq_num 738) 2021-02-10T04:36:04.196 INFO:tasks.rados.rados.0.smithi085.stdout:2100: finishing write tid 1 to smithi08575397-48 2021-02-10T04:36:04.196 INFO:tasks.rados.rados.0.smithi085.stdout:2098: done (10 left) 2021-02-10T04:36:04.196 INFO:tasks.rados.rados.0.smithi085.stdout:2101: delete oid 30 current snap is 235 2021-02-10T04:36:04.200 INFO:tasks.rados.rados.0.smithi085.stdout:2101: done (10 left) 2021-02-10T04:36:04.200 INFO:tasks.rados.rados.0.smithi085.stdout:2102: read oid 30 snap -1 2021-02-10T04:36:04.200 INFO:tasks.rados.rados.0.smithi085.stdout:2102: expect deleted 2021-02-10T04:36:04.200 INFO:tasks.rados.rados.0.smithi085.stdout:2103: read oid 5 snap -1 2021-02-10T04:36:04.200 INFO:tasks.rados.rados.0.smithi085.stdout:2103: expect deleted 2021-02-10T04:36:04.200 INFO:tasks.rados.rados.0.smithi085.stdout:2104: read oid 36 snap -1 2021-02-10T04:36:04.200 INFO:tasks.rados.rados.0.smithi085.stdout:2104: expect (ObjNum 720 snap 226 seq_num 720) 2021-02-10T04:36:04.200 INFO:tasks.rados.rados.0.smithi085.stdout:2105: write (excl) oid 11 current snap is 235 2021-02-10T04:36:04.200 INFO:tasks.rados.rados.0.smithi085.stdout:2105: seq_num 741 ranges {453987=645750,1845157=608830} 2021-02-10T04:36:04.207 INFO:tasks.rados.rados.0.smithi085.stdout:2105: writing smithi08575397-11 from 453987 to 1099737 tid 1 2021-02-10T04:36:04.215 INFO:tasks.rados.rados.0.smithi085.stdout:2105: writing smithi08575397-11 from 1845157 to 2453987 tid 2 2021-02-10T04:36:04.215 INFO:tasks.rados.rados.0.smithi085.stdout:2106: snap_remove snap 219 2021-02-10T04:36:04.311 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[49106]: debug 2021-02-10T04:36:04.006+0000 7f77a3e94700 -1 osd.2 546 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:04.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[33677]: debug 2021-02-10T04:36:04.088+0000 7fbec19a1700 -1 osd.4 546 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:04.413 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:36:04.444 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:36:04.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: cluster 2021-02-10T04:36:04.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:36:04.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: 04:36:03.091316+0000 osd.4 (osd.4) 791 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:04.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:04.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: 02-10T04:36:03.091323+0000 osd.4 2021-02-10T04:36:04.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: (osd.4) 792 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:04.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:04.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: -02-10T04:36: 2021-02-10T04:36:04.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: 03.091328+0000 osd.4 (osd. 2021-02-10T04:36:04.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: 4) 793 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:04.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:36:04.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: 04:36:03.091336+0000 osd.4 (osd. 2021-02-10T04:36:04.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: 4) 794 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:04.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:36:04.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: 10T04:36:03. 2021-02-10T04:36:04.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: 808055+0000 osd.5 (osd.5 2021-02-10T04:36:04.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: ) 67 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:36:04.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: 2021-02-10T04:36:04.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:36:04.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: 10T04:36:03. 2021-02-10T04:36:04.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: 808241+0000 osd.5 (osd.5 2021-02-10T04:36:04.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: ) 68 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:36:04.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: 2021-02-10T04:36:04.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:04.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: 02-10T04:36: 2021-02-10T04:36:04.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: 04.007591+0000 osd.2 (osd 2021-02-10T04:36:04.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: .2) 1489 : cluster [WRN] 2021-02-10T04:36:04.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:04.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: cluster 2021-02-10T04:36:04.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:36:04.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: 36:04.007599+0000 osd.2 2021-02-10T04:36:04.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: (osd.2) 1490 : cluster 2021-02-10T04:36:04.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:04.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:04.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: 02-10T04:36:04. 2021-02-10T04:36:04.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: 007603+0000 osd.2 (osd.2) 2021-02-10T04:36:04.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: 1491 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:04.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:04.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: -10T04:36:04.007607+0000 osd.2 2021-02-10T04:36:04.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: (osd.2) 1492 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:04.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:04.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: 02-10T04:36:04.007610+0000 osd.2 2021-02-10T04:36:04.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: (osd.2) 1493 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:04.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:04.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: -02-10T04:36:04.007614+0000 2021-02-10T04:36:04.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: osd.2 (osd.2) 1494 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:04.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: 2021-02-10T04:36:04.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:36:04.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: 10T04:36:04.007617+0000 osd.2 2021-02-10T04:36:04.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: (osd.2) 1495 : cluster [WRN] 2021-02-10T04:36:04.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:04.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:04.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: -10T04:36:04 2021-02-10T04:36:04.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: .007622+0000 osd.2 (osd. 2021-02-10T04:36:04.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: 2) 1496 : cluster [WRN] 2021-02-10T04:36:04.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:04.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:36:04.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: T04:36:04.007625+0000 osd.2 2021-02-10T04:36:04.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: (osd.2) 1497 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:04.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: 2021-02-10T04:36:04.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:04.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: -02-10T04:36:04.007629+0000 osd.2 (osd 2021-02-10T04:36:04.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: .2) 1498 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:04.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:36:04.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: T04:36:04.111476+0000 mgr.y (mgr 2021-02-10T04:36:04.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: .14140) 619 : cluster [DBG] pgmap v969: 168 pgs: 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 156 active+clean; 415 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.4 MiB/s rd, 481 KiB/s wr, 54 op/s 2021-02-10T04:36:04.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:04.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: -10T04:36:04.181237+0000 mon.a (mon.0) 1381 : cluster 2021-02-10T04:36:04.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[29319]: [DBG] osdmap e547: 8 total, 8 up, 8 in 2021-02-10T04:36:04.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: cluster 2021-02-10T04:36:03.091316 2021-02-10T04:36:04.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 791 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:04.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:36:04.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: 10T04:36:03.091323+0000 osd.4 (osd.4) 792 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:04.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:04.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: 02-10T04:36:03.091328+0000 osd.4 2021-02-10T04:36:04.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: (osd.4) 793 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:04.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:04.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: -10T04:36:03.091336+0000 osd.4 (osd.4) 2021-02-10T04:36:04.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: 794 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:04.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: cluster 2021-02-10T04:36:04.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: 2021 2021-02-10T04:36:04.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: -02-10T04:36:03 2021-02-10T04:36:04.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: .808055+0000 osd.5 (osd.5) 67 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:36:04.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:04.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: 02-10T04:36:03.808241+0000 osd.5 (osd 2021-02-10T04:36:04.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: .5) 68 : cluster [DBG] 2021-02-10T04:36:04.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:36:04.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: cluster 2021-02-10T04:36:04.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: 2021-02-10T04:36:04 2021-02-10T04:36:04.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: .007591+0000 osd.2 (osd.2) 1489 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:04.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: cluster 2021-02-10T04:36:04.007599+0000 2021-02-10T04:36:04.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: osd.2 (osd.2) 1490 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:04.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: 2021-02-10T04:36:04.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: cluster 2021-02-10T04:36:04.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: 2021-02-10T04:36:04.007603+0000 osd.2 ( 2021-02-10T04:36:04.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: osd.2) 1491 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:04.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: cluster 2021-02-10T04:36 2021-02-10T04:36:04.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: :04.007607+0000 osd.2 (osd.2) 1492 : 2021-02-10T04:36:04.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:04.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:36:04.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: 04:36:04.007610+0000 osd.2 (osd. 2021-02-10T04:36:04.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: 2) 1493 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:04.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:36:04.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: 04:36:04.007614+0000 osd.2 (osd.2 2021-02-10T04:36:04.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: ) 1494 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:04.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:04.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: -10T04:36:04.007617+0000 osd.2 ( 2021-02-10T04:36:04.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: osd.2) 1495 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:04.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:04.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: -10T04:36:04.007622+0000 osd.2 (osd.2) 1496 2021-02-10T04:36:04.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:04.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:04.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: T04:36:04.007625+0000 osd.2 (osd.2) 1497 : cluster [WRN] 2021-02-10T04:36:04.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:04.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:04.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: -02-10T04:36:04.007629+0000 osd.2 (osd.2) 1498 2021-02-10T04:36:04.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:04.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: cluster 2021-02-10T04:36 2021-02-10T04:36:04.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: :04.111476+0000 mgr.y (mgr.14140) 619 : 2021-02-10T04:36:04.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: cluster [DBG] pgmap v969: 168 pgs: 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 156 active+clean; 415 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.4 MiB/s rd, 481 KiB/s wr, 54 op/s 2021-02-10T04:36:04.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: cluster 2021-02-10T04:36: 2021-02-10T04:36:04.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: 04.181237+0000 mon.a (mon.0) 1381 : cluster 2021-02-10T04:36:04.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:04 smithi085 conmon[35961]: [DBG] osdmap e547: 8 total, 8 up, 8 in 2021-02-10T04:36:04.845 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 6 1 2021-02-10T04:36:04.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: cluster 2021-02-10T04:36:03.091316+0000 osd.4 (osd.4) 791 2021-02-10T04:36:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: cluster 2021-02-10T04:36:03.091323+0000 osd.4 (osd.4) 792 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: cluster 2021-02-10T04:36:03.091328+0000 osd.4 (osd. 2021-02-10T04:36:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: 4) 793 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: cluster 2021-02-10T04:36: 2021-02-10T04:36:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: 03.091336+0000 osd.4 (osd.4) 794 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: 2021-02-10T04:36:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: 02-10T04:36:03.808055+0000 osd.5 (osd.5) 67 : cluster [DBG] 2021-02-10T04:36:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: purged_snaps scrub starts 2021-02-10T04:36:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: cluster 2021-02-10T04:36:04.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: 2021-02-10T04:36:03.808241+0000 osd.5 (osd.5) 68 : cluster 2021-02-10T04:36:04.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: [DBG] purged_snaps scrub ok 2021-02-10T04:36:04.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: cluster 2021-02-10T04:36:04.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: 2021-02-10T04:36:04.007591+0000 osd.2 (osd.2) 1489 2021-02-10T04:36:04.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:04.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: cluster 2021-02-10T04:36:04.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: 2021-02-10T04:36:04.007599+0000 osd.2 (osd.2) 1490 2021-02-10T04:36:04.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:04.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: cluster 2021-02-10T04:36:04.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: 2021-02-10T04:36:04.007603+0000 osd.2 (osd.2) 2021-02-10T04:36:04.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: 1491 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:04.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: cluster 2021-02-10T04:36:04.007607 2021-02-10T04:36:04.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1492 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:04.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:36:04.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: 04:36:04.007610+0000 osd.2 (osd.2) 1493 : cluster 2021-02-10T04:36:04.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:04.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: cluster 2021-02-10T04:36:04.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: 2021-02-10T04:36:04.007614+0000 osd.2 (osd.2) 1494 2021-02-10T04:36:04.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:04.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: cluster 2021-02-10T04:36:04.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: 2021-02-10T04:36:04.007617+0000 osd.2 (osd. 2021-02-10T04:36:04.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: 2) 1495 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:04.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: cluster 2021-02-10T04:36: 2021-02-10T04:36:04.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: 04.007622+0000 osd.2 (osd.2) 1496 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:04.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: 2021-02-10T04:36:04.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:04.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: -02-10T04:36:04.007625+0000 osd.2 (osd.2) 1497 2021-02-10T04:36:04.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:04.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: cluster 2021-02-10T04:36:04.007629+0000 2021-02-10T04:36:04.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: osd.2 (osd.2) 1498 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:04.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: cluster 2021-02-10T04:36: 2021-02-10T04:36:04.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: 04.111476+0000 mgr.y (mgr.14140) 619 : cluster [DBG] pgmap v969: 168 pgs: 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 156 active+clean; 415 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.4 MiB/s rd, 481 KiB/s wr, 54 op/s 2021-02-10T04:36:04.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:36:04.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: 10T04:36:04.181237+0000 mon.a (mon.0) 1381 : 2021-02-10T04:36:04.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:04 smithi164 conmon[29288]: cluster [DBG] osdmap e547: 8 total, 8 up, 8 in 2021-02-10T04:36:05.016 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[49106]: debug 2021-02-10T04:36:05.016 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[49106]: 2021-02-10T04:36:05.013+0000 7f77a3e94700 -1 osd.2 547 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:05.016 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[49106]: 2021-02-10T04:36:05.192 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:36:05.192 INFO:tasks.rados.rados.0.smithi085.stdout:2106: done (14 left) 2021-02-10T04:36:05.192 INFO:tasks.rados.rados.0.smithi085.stdout:2107: read oid 43 snap 202 2021-02-10T04:36:05.192 INFO:tasks.rados.rados.0.smithi085.stdout:2107: expect deleted 2021-02-10T04:36:05.192 INFO:tasks.rados.rados.0.smithi085.stdout:2108: snap_create 2021-02-10T04:36:05.192 INFO:tasks.rados.rados.0.smithi085.stdout:2100: finishing write tid 2 to smithi08575397-48 2021-02-10T04:36:05.193 INFO:tasks.rados.rados.0.smithi085.stdout:2100: finishing write tid 3 to smithi08575397-48 2021-02-10T04:36:05.193 INFO:tasks.rados.rados.0.smithi085.stdout:2100: finishing write tid 4 to smithi08575397-48 2021-02-10T04:36:05.193 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 823 (ObjNum 739 snap 235 seq_num 739) dirty exists 2021-02-10T04:36:05.193 INFO:tasks.rados.rados.0.smithi085.stdout:2100: left oid 48 (ObjNum 739 snap 235 seq_num 739) 2021-02-10T04:36:05.193 INFO:tasks.rados.rados.0.smithi085.stdout:2104: expect (ObjNum 720 snap 226 seq_num 720) 2021-02-10T04:36:05.273 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:36:05.311 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:36:05.334 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[33677]: debug 2021-02-10T04:36:05.061+0000 7fbec19a1700 -1 osd.4 547 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:05.341 INFO:tasks.rados.rados.0.smithi085.stdout:2105: finishing write tid 1 to smithi08575397-11 2021-02-10T04:36:05.341 INFO:tasks.rados.rados.0.smithi085.stdout:2105: finishing write tid 2 to smithi08575397-11 2021-02-10T04:36:05.341 INFO:tasks.rados.rados.0.smithi085.stdout:2105: finishing write tid 3 to smithi08575397-11 2021-02-10T04:36:05.341 INFO:tasks.rados.rados.0.smithi085.stdout:2105: finishing write tid 4 to smithi08575397-11 2021-02-10T04:36:05.342 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 1122 (ObjNum 741 snap 235 seq_num 741) dirty exists 2021-02-10T04:36:05.342 INFO:tasks.rados.rados.0.smithi085.stdout:2105: left oid 11 (ObjNum 741 snap 235 seq_num 741) 2021-02-10T04:36:05.346 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:36:05.351 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.5 dump_ops_in_flight 2021-02-10T04:36:05.442 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[52836]: debug 2021-02-10T04:36:05.442 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[52836]: 2021-02-10T04:36:05.136+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:36:05.447 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:36:05.552 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:36:05.585 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:36:05.645 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[40892]: debug 2021-02-10T04:36:05.410+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:36:05.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:36:05.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: :36:04.088733+0000 osd.4 (osd.4) 795 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:05.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: cluster 2021-02-10T04:36:04.088747+0000 osd.4 (osd.4) 796 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:05.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: cluster 2021-02-10T04:36:04.088754+0000 osd.4 (osd.4) 797 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:05.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: cluster 2021-02-10T04:36:04.088759+0000 osd.4 (osd.4) 798 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:05.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: cluster 2021-02-10T04:36:05.015205+0000 osd.2 (osd.2) 1499 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:05.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: cluster 2021-02-10T04:36:05.015214+0000 osd.2 (osd.2) 1500 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:05.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: cluster 2021-02-10T04:36:05.015219+0000 osd.2 (osd.2) 1501 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:05.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: cluster 2021-02-10T04:36:05.015226+0000 osd.2 (osd.2) 1502 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:05.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: cluster 2021-02-10T04:36:05.015231+0000 osd.2 (osd.2) 1503 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:05.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: cluster 2021-02-10T04:36:05.015238+0000 osd.2 (osd.2) 1504 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:05.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: cluster 2021-02-10T04:36:05.015244+0000 osd.2 (osd.2) 1505 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:05.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:05.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: -02-10T04:36:05.015250+0000 osd.2 (osd.2) 1506 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:05.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: cluster 2021-02-10T04:36:05.015255+0000 osd.2 (osd.2) 2021-02-10T04:36:05.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: 1507 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:05.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:05.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: 02-10T04:36:05.015261+0000 osd.2 (osd.2) 1508 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:05.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: 2021-02-10T04:36:05.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:36:05.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: 04:36:05.062252+0000 osd.4 (osd.4) 799 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:05.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:36:05.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: 36:05.062262+0000 osd.4 (osd.4) 800 : cluster 2021-02-10T04:36:05.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:05.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:36:05.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: 10T04:36:05.062275+0000 osd.4 ( 2021-02-10T04:36:05.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: osd.4) 801 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:05.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:36:05.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: 10T04:36:05.062286+0000 osd.4 (osd.4) 802 : cluster 2021-02-10T04:36:05.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:05.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:36:05.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: 10T04:36:05.182225+0000 mon.a (mon.0) 1382 : cluster [DBG] 2021-02-10T04:36:05.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[29288]: osdmap e548: 8 total, 8 up, 8 in 2021-02-10T04:36:05.671 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:36:05.697 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:36:05.789 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: cluster 2021-02-10T04:36:04.088733+0000 osd.4 (osd 2021-02-10T04:36:05.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: .4) 795 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:05.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: cluster 2021-02-10T04:36:04.088747+0000 osd.4 ( 2021-02-10T04:36:05.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: osd.4) 796 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:05.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: cluster 2021-02-10T04:36:05.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: 2021-02-10T04:36:04 2021-02-10T04:36:05.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: .088754+0000 osd.4 (osd.4) 2021-02-10T04:36:05.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: 797 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:05.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:36:05.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: 10T04:36:04.088759+0000 osd.4 (osd. 2021-02-10T04:36:05.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: 4) 798 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:05.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:36:05.791 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: :36:05.015205+0000 osd.2 (osd.2) 1499 : 2021-02-10T04:36:05.791 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:05.791 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:36:05.791 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: 36:05.015214+0000 osd.2 (osd.2) 1500 : cluster 2021-02-10T04:36:05.791 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:05.791 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: cluster 2021-02-10T04:36:05.791 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: 2021-02-10T04:36:05.015219+0000 osd.2 (osd.2) 1501 2021-02-10T04:36:05.791 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:05.791 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: cluster 2021-02-10T04:36:05. 2021-02-10T04:36:05.791 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: 015226+0000 osd.2 (osd.2) 1502 : cluster 2021-02-10T04:36:05.791 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:05.791 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:05.792 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: -02-10T04:36:05.015231+0000 osd.2 (osd.2) 2021-02-10T04:36:05.792 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: 1503 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:05.792 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:05.792 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: -10T04:36:05.015238+0000 osd.2 2021-02-10T04:36:05.792 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: (osd.2) 1504 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:05.792 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:36:05.792 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: 10T04:36:05.015244+0000 osd.2 (osd.2) 1505 : cluster [WRN] 2021-02-10T04:36:05.792 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:05.792 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:05.792 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: 02-10T04:36:05.015250 2021-02-10T04:36:05.792 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1506 2021-02-10T04:36:05.792 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:05.792 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: cluster 2021-02-10T04:36:05.793 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: 2021-02-10T04:36 2021-02-10T04:36:05.793 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: :05.015255+0000 osd.2 (osd.2) 1507 : 2021-02-10T04:36:05.793 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:05.793 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: cluster 2021-02-10T04:36:05.793 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: 2021-02-10T04:36:05.015261+0000 osd.2 ( 2021-02-10T04:36:05.793 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: osd.2) 1508 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:05.793 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: cluster 2021-02-10T04:36:05.793 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: 2021-02-10T04:36:05.062252+0000 2021-02-10T04:36:05.793 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: osd.4 (osd.4) 799 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:05.793 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: 2021-02-10T04:36:05.793 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: cluster 2021-02-10T04:36:05.793 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: 2021-02-10T04:36:05.062262+0000 osd.4 2021-02-10T04:36:05.793 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: (osd.4) 800 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:05.794 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: cluster 2021-02-10T04:36:05 2021-02-10T04:36:05.794 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: .062275+0000 osd.4 (osd.4) 801 : cluster [WRN] 2021-02-10T04:36:05.794 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:05.794 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:36:05.794 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: 10T04:36:05.062286+0000 osd.4 (osd.4) 802 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:05.794 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: cluster 2021-02-10T04:36:05.182225+0000 2021-02-10T04:36:05.794 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[29319]: mon.a (mon.0) 1382 : cluster [DBG] osdmap e548: 8 total, 8 up, 8 in 2021-02-10T04:36:05.794 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: cluster 2021-02-10T04:36 2021-02-10T04:36:05.794 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: :04.088733+0000 osd.4 (osd.4) 795 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:05.794 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:36:05.794 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: :36:04.088747+0000 osd.4 (osd.4) 796 : cluster 2021-02-10T04:36:05.795 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:05.795 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: cluster 2021-02-10T04:36:04. 2021-02-10T04:36:05.795 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: 088754+0000 osd.4 (osd.4) 797 : cluster [WRN] 2021-02-10T04:36:05.795 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:05.795 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:05.795 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: -02-10T04:36:04.088759+0000 osd.4 (osd.4) 798 : cluster 2021-02-10T04:36:05.795 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:05.795 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: cluster 2021-02-10T04:36:05.795 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: 2021-02-10T04:36:05.015205+0000 osd.2 (osd.2) 2021-02-10T04:36:05.795 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: 1499 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:05.795 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: cluster 2021-02-10T04:36:05 2021-02-10T04:36:05.795 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: .015214+0000 osd.2 (osd.2) 1500 : cluster [WRN] 2021-02-10T04:36:05.796 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:05.796 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: cluster 2021-02-10T04:36:05.015219 2021-02-10T04:36:05.796 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1501 : cluster [WRN] 2021-02-10T04:36:05.796 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:05.796 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: cluster 2021-02-10T04:36:05.796 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: 2021-02-10T04:36:05.015226+0000 osd.2 2021-02-10T04:36:05.796 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: (osd.2) 1502 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:05.796 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: 2021-02-10T04:36:05.796 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: cluster 2021-02-10T04:36:05.796 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: 2021-02-10T04:36:05.015231+0000 osd.2 (osd 2021-02-10T04:36:05.796 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: .2) 1503 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:05.796 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:36:05.796 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: 36:05.015238+0000 osd.2 (osd 2021-02-10T04:36:05.797 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: .2) 1504 : cluster [WRN] 2021-02-10T04:36:05.797 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:05.797 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:05.797 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: -02-10T04:36 2021-02-10T04:36:05.797 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: :05.015244+0000 osd.2 (osd 2021-02-10T04:36:05.797 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: .2) 1505 : cluster [WRN] 2021-02-10T04:36:05.797 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:05.797 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:05.798 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: -10T04:36:05.015250 2021-02-10T04:36:05.798 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 2021-02-10T04:36:05.798 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: 1506 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:05.798 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:05.798 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: -10T04:36:05.015255+0000 osd.2 ( 2021-02-10T04:36:05.798 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: osd.2) 1507 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:05.798 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:05.798 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: 02-10T04:36:05.015261 2021-02-10T04:36:05.798 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1508 : cluster [WRN] 2021-02-10T04:36:05.798 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:05.798 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: cluster 2021-02-10T04:36:05.798 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: 2021-02-10T04:36:05.062252+0000 osd.4 ( 2021-02-10T04:36:05.799 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: osd.4) 799 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:05.799 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: cluster 2021-02-10T04:36:05.799 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:36:05.799 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: 36:05.062262+0000 osd.4 (osd 2021-02-10T04:36:05.799 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: .4) 800 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:05.799 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:36:05.799 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: :36:05.062275+0000 osd.4 (osd.4) 801 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:05.799 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:05.799 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: 02-10T04:36:05. 2021-02-10T04:36:05.799 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: 062286+0000 osd.4 (osd.4) 2021-02-10T04:36:05.799 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: 802 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:05.799 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:05.799 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: 02-10T04:36:05.182225+0000 mon.a (mon.0) 1382 : 2021-02-10T04:36:05.800 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[35961]: cluster [DBG] osdmap e548: 8 total, 8 up, 8 in 2021-02-10T04:36:05.802 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:36:05.841 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:36:05.923 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[44544]: debug 2021-02-10T04:36:05.666+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:36:05.923 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:36:05 smithi164 conmon[44544]: debug 2021-02-10T04:36:05.920+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:36:05.947 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:36:06.053 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:36:06.081 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:36:06.182 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:36:06.183 INFO:tasks.rados.rados.0.smithi085.stdout:2100: done (15 left) 2021-02-10T04:36:06.183 INFO:tasks.rados.rados.0.smithi085.stdout:2102: done (14 left) 2021-02-10T04:36:06.183 INFO:tasks.rados.rados.0.smithi085.stdout:2103: done (13 left) 2021-02-10T04:36:06.183 INFO:tasks.rados.rados.0.smithi085.stdout:2104: done (12 left) 2021-02-10T04:36:06.183 INFO:tasks.rados.rados.0.smithi085.stdout:2105: done (11 left) 2021-02-10T04:36:06.183 INFO:tasks.rados.rados.0.smithi085.stdout:2107: done (10 left) 2021-02-10T04:36:06.183 INFO:tasks.rados.rados.0.smithi085.stdout:2108: done (9 left) 2021-02-10T04:36:06.183 INFO:tasks.rados.rados.0.smithi085.stdout:2109: snap_create 2021-02-10T04:36:06.184 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[40892]: debug 2021-02-10T04:36:06.149+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:36:06.185 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[33677]: debug 2021-02-10T04:36:06.062+0000 7fbec19a1700 -1 osd.4 548 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:06.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:05 smithi085 conmon[49106]: debug 2021-02-10T04:36:05.995+0000 7f77a3e94700 -1 osd.2 548 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:06.288 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:36:06.317 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:36:06.431 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:36:06.435 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[33677]: debug 2021-02-10T04:36:06.391+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:36:06.537 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:36:06.567 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:36:06.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: cluster 2021-02-10T04:36:05 2021-02-10T04:36:06.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: .996697+0000 osd.2 (osd.2) 1509 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:06.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: cluster 2021-02-10T04:36:05.996706+0000 osd.2 (osd.2) 1510 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:06.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: cluster 2021-02-10T04:36:05.996713+0000 osd.2 (osd.2) 1511 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:06.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: cluster 2021-02-10T04:36:05.996727+0000 osd.2 (osd.2) 1512 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:06.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: cluster 2021-02-10T04:36:05.996733+0000 osd.2 2021-02-10T04:36:06.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: (osd.2) 1513 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:06.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: cluster 2021-02-10T04:36:05.996739 2021-02-10T04:36:06.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1514 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:06.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:06.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: -10T04:36:05.996745+0000 osd.2 (osd.2) 1515 2021-02-10T04:36:06.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:06.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:36:06.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: 10T04:36:05.996758+0000 osd.2 (osd 2021-02-10T04:36:06.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: .2) 1516 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:06.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:06.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: -02-10T04:36:05.996764+0000 2021-02-10T04:36:06.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: osd.2 (osd.2) 1517 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:06.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: 2021-02-10T04:36:06.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: cluster 2021-02-10T04:36:05 2021-02-10T04:36:06.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: .996770+0000 osd.2 (osd.2) 1518 : 2021-02-10T04:36:06.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:06.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:36:06.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: :36:06.062920+0000 osd.4 (osd.4 2021-02-10T04:36:06.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: ) 803 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:06.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:06.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: -02-10T04:36:06.062930+0000 2021-02-10T04:36:06.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: osd.4 (osd.4) 804 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:06.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: 2021-02-10T04:36:06.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: cluster 2021-02-10T04:36:06 2021-02-10T04:36:06.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: .062936+0000 osd.4 (osd.4) 805 : 2021-02-10T04:36:06.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:06.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: cluster 2021-02-10T04:36 2021-02-10T04:36:06.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: :06.062942+0000 osd.4 (osd.4) 806 : 2021-02-10T04:36:06.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:06.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:36:06.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: 10T04:36:06.111995+0000 mgr.y ( 2021-02-10T04:36:06.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: mgr.14140) 620 : cluster [DBG] pgmap v972: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 417 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 963 KiB/s wr, 11 op/s 2021-02-10T04:36:06.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:06.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: -02-10T04:36:06.182886+0000 2021-02-10T04:36:06.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: mon.a (mon.0) 1383 : cluster [DBG] 2021-02-10T04:36:06.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:06 smithi164 conmon[29288]: osdmap e549: 8 total, 8 up, 8 in 2021-02-10T04:36:06.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: cluster 2021-02-10T04:36:05.996697+0000 osd.2 2021-02-10T04:36:06.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: (osd.2) 1509 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:06.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: cluster 2021-02-10T04:36:05.996706+0000 osd.2 (osd.2) 1510 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:06.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: cluster 2021-02-10T04:36:05.996713+0000 osd.2 (osd.2) 1511 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:06.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: cluster 2021-02-10T04:36:05.996727+0000 2021-02-10T04:36:06.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: osd.2 (osd.2) 1512 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:06.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:36:06.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: 10T04:36:05.996733+0000 osd.2 (osd 2021-02-10T04:36:06.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: .2) 1513 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:06.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:06.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: -02-10T04:36:05.996739+0000 osd.2 2021-02-10T04:36:06.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: (osd.2) 1514 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:06.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: cluster 2021-02-10T04:36:06.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: 2021-02-10T04:36:05.996745+0000 2021-02-10T04:36:06.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: osd.2 (osd.2) 1515 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:06.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: cluster 2021-02-10T04:36:06.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:36:06.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: 04:36:05.996758+0000 osd.2 (osd.2 2021-02-10T04:36:06.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: ) 1516 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:06.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: cluster 2021-02-10T04:36:06.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: 2021-02-10T04:36:05. 2021-02-10T04:36:06.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: 996764+0000 osd.2 (osd.2) 1517 : cluster 2021-02-10T04:36:06.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:06.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: cluster 2021-02-10T04:36:06.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: 2021-02-10T04:36:05 2021-02-10T04:36:06.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: .996770+0000 osd.2 (osd.2) 2021-02-10T04:36:06.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: 1518 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:06.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:06.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: 02-10T04:36:06.062920+0000 2021-02-10T04:36:06.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: osd.4 (osd.4) 803 : cluster 2021-02-10T04:36:06.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:06.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:06.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: 02-10T04:36:06 2021-02-10T04:36:06.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: .062930+0000 osd.4 (osd.4 2021-02-10T04:36:06.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: ) 804 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:06.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:06.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: 02-10T04:36:06.062936+0000 osd.4 ( 2021-02-10T04:36:06.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: osd.4) 805 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:06.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: cluster 2021-02-10T04:36:06.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: 2021-02-10T04:36:06.062942 2021-02-10T04:36:06.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 806 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:06.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: 2021-02-10T04:36:06.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:36:06.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: :36:06.111995+0000 2021-02-10T04:36:06.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: mgr.y (mgr.14140) 620 : cluster 2021-02-10T04:36:06.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: [DBG] pgmap v972: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 417 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 963 KiB/s wr, 11 op/s 2021-02-10T04:36:06.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:36:06.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: 10T04:36:06.182886+0000 mon.a (mon.0) 1383 2021-02-10T04:36:06.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[29319]: : cluster [DBG] osdmap e549: 8 total, 8 up, 8 in 2021-02-10T04:36:06.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: cluster 2021-02-10T04:36:05.996697+0000 osd.2 (osd 2021-02-10T04:36:06.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: .2) 1509 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:06.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: cluster 2021-02-10T04:36:05. 2021-02-10T04:36:06.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: 996706+0000 osd.2 (osd.2) 1510 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:06.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: cluster 2021-02-10T04:36:06.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: 2021-02-10T04:36:05.996713+0000 osd.2 (osd.2) 1511 : 2021-02-10T04:36:06.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:06.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:06.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: T04:36:05.996727+0000 osd.2 (osd.2) 1512 : cluster 2021-02-10T04:36:06.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:06.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: cluster 2021-02-10T04:36:05. 2021-02-10T04:36:06.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: 996733+0000 osd.2 (osd.2) 1513 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:06.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:06.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: -02-10T04:36:05.996739+0000 osd.2 2021-02-10T04:36:06.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: (osd.2) 1514 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:06.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:36:06.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: 10T04:36:05.996745+0000 osd.2 (osd.2 2021-02-10T04:36:06.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: ) 1515 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:06.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:36:06.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: :36:05.996758+0000 osd.2 (osd.2) 2021-02-10T04:36:06.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: 1516 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:06.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:06.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: T04:36:05.996764+0000 osd.2 (osd.2) 1517 : 2021-02-10T04:36:06.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:06.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: cluster 2021-02-10T04:36:05.996770 2021-02-10T04:36:06.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1518 : cluster [WRN] 2021-02-10T04:36:06.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:06.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: cluster 2021-02-10T04:36 2021-02-10T04:36:06.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: :06.062920+0000 osd.4 (osd.4) 803 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:06.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:06.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: -10T04:36:06.062930+0000 osd.4 (osd.4 2021-02-10T04:36:06.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: ) 804 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:06.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: cluster 2021-02-10T04:36: 2021-02-10T04:36:06.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: 06.062936+0000 osd.4 (osd.4) 805 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:06.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:06.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: T04:36:06.062942+0000 osd.4 (osd 2021-02-10T04:36:06.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: .4) 806 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:06.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:06.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: 02-10T04:36:06.111995+0000 mgr.y 2021-02-10T04:36:06.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: (mgr.14140) 620 : cluster [DBG] pgmap v972: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 417 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 963 KiB/s wr, 11 op/s 2021-02-10T04:36:06.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: 2021-02-10T04:36:06.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: cluster 2021-02-10T04:36:06.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: 2021-02-10T04:36:06.182886+0000 2021-02-10T04:36:06.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: mon.a (mon.0) 1383 : cluster [DBG] osdmap e549: 8 total, 8 up, 8 in 2021-02-10T04:36:06.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:06 smithi085 conmon[35961]: 2021-02-10T04:36:07.185 INFO:tasks.rados.rados.0.smithi085.stdout:2109: done (9 left) 2021-02-10T04:36:07.185 INFO:tasks.rados.rados.0.smithi085.stdout:2110: read oid 43 snap -1 2021-02-10T04:36:07.185 INFO:tasks.rados.rados.0.smithi085.stdout:2110: expect (ObjNum 670 snap 205 seq_num 670) 2021-02-10T04:36:07.185 INFO:tasks.rados.rados.0.smithi085.stdout:2111: read oid 1 snap -1 2021-02-10T04:36:07.186 INFO:tasks.rados.rados.0.smithi085.stdout:2111: expect deleted 2021-02-10T04:36:07.186 INFO:tasks.rados.rados.0.smithi085.stdout:2112: read oid 22 snap 225 2021-02-10T04:36:07.186 INFO:tasks.rados.rados.0.smithi085.stdout:2112: expect deleted 2021-02-10T04:36:07.186 INFO:tasks.rados.rados.0.smithi085.stdout:2113: copy_from oid 37 from oid 42 current snap is 237 2021-02-10T04:36:07.186 INFO:tasks.rados.rados.0.smithi085.stdout:2114: read oid 16 snap -1 2021-02-10T04:36:07.186 INFO:tasks.rados.rados.0.smithi085.stdout:2114: expect deleted 2021-02-10T04:36:07.186 INFO:tasks.rados.rados.0.smithi085.stdout:2115: snap_create 2021-02-10T04:36:07.200 INFO:tasks.rados.rados.0.smithi085.stdout:2110: expect (ObjNum 670 snap 205 seq_num 670) 2021-02-10T04:36:07.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:07 smithi164 conmon[33677]: debug 2021-02-10T04:36:07.087+0000 7fbec19a1700 -1 osd.4 549 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:07.396 INFO:tasks.rados.rados.0.smithi085.stdout:2113: finishing copy_from to smithi08575397-37 2021-02-10T04:36:07.396 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 816 (ObjNum 646 snap 196 seq_num 646) dirty exists 2021-02-10T04:36:07.396 INFO:tasks.rados.rados.0.smithi085.stdout:2113: finishing copy_from racing read to smithi08575397-37 2021-02-10T04:36:07.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:07 smithi085 conmon[49106]: debug 2021-02-10T04:36:07.041+0000 7f77a3e94700 -1 osd.2 549 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:08.173 INFO:tasks.rados.rados.0.smithi085.stdout:2110: done (14 left) 2021-02-10T04:36:08.173 INFO:tasks.rados.rados.0.smithi085.stdout:2111: done (13 left) 2021-02-10T04:36:08.173 INFO:tasks.rados.rados.0.smithi085.stdout:2112: done (12 left) 2021-02-10T04:36:08.173 INFO:tasks.rados.rados.0.smithi085.stdout:2113: done (11 left) 2021-02-10T04:36:08.174 INFO:tasks.rados.rados.0.smithi085.stdout:2114: done (10 left) 2021-02-10T04:36:08.174 INFO:tasks.rados.rados.0.smithi085.stdout:2115: done (9 left) 2021-02-10T04:36:08.174 INFO:tasks.rados.rados.0.smithi085.stdout:2116: write oid 13 current snap is 238 2021-02-10T04:36:08.174 INFO:tasks.rados.rados.0.smithi085.stdout:2116: seq_num 742 ranges {470794=720999,1903639=554256} 2021-02-10T04:36:08.180 INFO:tasks.rados.rados.0.smithi085.stdout:2116: writing smithi08575397-13 from 470794 to 1191793 tid 1 2021-02-10T04:36:08.189 INFO:tasks.rados.rados.0.smithi085.stdout:2116: writing smithi08575397-13 from 1903639 to 2457895 tid 2 2021-02-10T04:36:08.189 INFO:tasks.rados.rados.0.smithi085.stdout:2117: rollback oid 37 current snap is 238 2021-02-10T04:36:08.189 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 37 to 203 2021-02-10T04:36:08.189 INFO:tasks.rados.rados.0.smithi085.stdout:2118: snap_remove snap 231 2021-02-10T04:36:08.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[33677]: debug 2021-02-10T04:36:08.104+0000 7fbec19a1700 -1 osd.4 550 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:08.394 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:36:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: cluster 2021-02-10T04:36:07.043079+0000 osd.2 (osd.2) 1519 : cluster 2021-02-10T04:36:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: cluster 2021-02-10T04:36:07.043091+0000 osd.2 (osd.2) 1520 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:36:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: 04:36:07.043095+0000 osd.2 2021-02-10T04:36:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: (osd.2) 1521 : cluster [WRN] 2021-02-10T04:36:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: cluster 2021-02-10T04:36:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: 2021-02-10T04:36 2021-02-10T04:36:08.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: :07.043099+0000 osd.2 (osd.2) 1522 : cluster 2021-02-10T04:36:08.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:08.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: cluster 2021-02-10T04:36 2021-02-10T04:36:08.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: :07.043103+0000 osd.2 (osd.2) 1523 : cluster 2021-02-10T04:36:08.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:08.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:08.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: 02-10T04:36:07.043106+0000 osd.2 (osd 2021-02-10T04:36:08.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: .2) 1524 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:08.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:08.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: 02-10T04:36:07.043113 2021-02-10T04:36:08.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1525 2021-02-10T04:36:08.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:08.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:08.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: -02-10T04:36: 2021-02-10T04:36:08.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: 07.043118+0000 osd.2 (osd. 2021-02-10T04:36:08.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: 2) 1526 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:08.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: 2021-02-10T04:36:08.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:36:08.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: 36:07.043122+0000 osd.2 (osd.2) 1527 : 2021-02-10T04:36:08.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:08.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:36:08.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: :36:07.043128+0000 osd.2 (osd.2) 1528 2021-02-10T04:36:08.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:08.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:36:08.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: 36:07.088194+0000 osd.4 (osd.4) 807 2021-02-10T04:36:08.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:08.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: cluster 2021-02-10T04:36:07 2021-02-10T04:36:08.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: .088200+0000 osd.4 (osd.4) 808 : cluster [WRN] 2021-02-10T04:36:08.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:08.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: cluster 2021-02-10T04:36:08.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: 2021-02-10T04:36:07.088204+0000 osd.4 (osd. 2021-02-10T04:36:08.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: 4) 809 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:08.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: 2021-02-10T04:36:08.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: cluster 2021-02-10T04:36:08.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: 2021-02-10T04:36 2021-02-10T04:36:08.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: :07.088208+0000 osd.4 ( 2021-02-10T04:36:08.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: osd.4) 810 : cluster [WRN] 2021-02-10T04:36:08.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:08.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:08.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: 02-10T04:36:07. 2021-02-10T04:36:08.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: 184682+0000 mon.a (mon.0) 2021-02-10T04:36:08.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: 1384 : cluster [DBG] osdmap e550: 8 total, 8 up, 8 in 2021-02-10T04:36:08.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:08.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: 02-10T04:36:08.112427+0000 2021-02-10T04:36:08.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: mgr.y (mgr.14140) 621 : cluster [DBG] pgmap v975: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 417 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 482 KiB/s wr, 6 op/s 2021-02-10T04:36:08.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:36:08.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: -10T04:36:08.126958+0000 mgr.y (mgr 2021-02-10T04:36:08.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: .14140) 622 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:36:08.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: cluster 2021-02-10T04:36:08.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: 2021-02-10T04:36 2021-02-10T04:36:08.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: :08.167036+0000 mon.a ( 2021-02-10T04:36:08.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: mon.0) 1385 : cluster [WRN] 2021-02-10T04:36:08.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: Health check update: 14 slow ops, oldest one blocked for 392 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:36:08.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: cluster 2021-02-10T04:36:08.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:36:08.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: 36:08.172233+0000 mon.a 2021-02-10T04:36:08.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: (mon.0) 1386 : cluster 2021-02-10T04:36:08.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:08 smithi164 conmon[29288]: [DBG] osdmap e551: 8 total, 8 up, 8 in 2021-02-10T04:36:08.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[49106]: debug 2021-02-10T04:36:08.442 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[49106]: 2021-02-10T04:36:08.024+0000 7f77a3e94700 -1 osd.2 550 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: cluster 2021-02-10T04:36:07.043079+0000 osd.2 (osd.2 2021-02-10T04:36:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: ) 1519 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: cluster 2021-02-10T04:36:07.043091+0000 osd.2 (osd.2) 1520 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: cluster 2021-02-10T04:36:07.043095+0000 osd.2 (osd.2) 1521 : cluster [WRN] 2021-02-10T04:36:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: cluster 2021-02-10T04:36:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: 2021-02-10T04:36:07.043099+0000 osd.2 (osd.2) 1522 : cluster 2021-02-10T04:36:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: cluster 2021-02-10T04:36:07. 2021-02-10T04:36:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: 043103+0000 osd.2 (osd.2) 1523 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:08.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: cluster 2021-02-10T04:36:07.043106+0000 osd.2 (osd 2021-02-10T04:36:08.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: .2) 1524 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:08.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:08.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: -02-10T04:36:07.043113+0000 osd.2 (osd.2) 1525 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:08.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:08.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: -02-10T04:36:07.043118+0000 osd.2 (osd.2) 1526 : cluster [WRN] 2021-02-10T04:36:08.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:08.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:08.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: T04:36:07.043122+0000 osd.2 (osd.2) 1527 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:08.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: cluster 2021-02-10T04:36:08.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: 2021-02 2021-02-10T04:36:08.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: -10T04:36:07.043128+0000 osd.2 (osd.2 2021-02-10T04:36:08.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: ) 1528 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:08.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:36:08.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: 10T04:36:07.088194+0000 osd.4 (osd.4) 807 : cluster 2021-02-10T04:36:08.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:08.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: cluster 2021-02-10T04:36:07.088200+0000 osd.4 ( 2021-02-10T04:36:08.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: osd.4) 808 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:08.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: cluster 2021-02-10T04:36:07. 2021-02-10T04:36:08.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: 088204+0000 osd.4 (osd.4) 809 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:08.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:36:08.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: 36:07.088208+0000 osd.4 (osd.4) 810 : cluster [WRN] 2021-02-10T04:36:08.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:08.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:08.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: -02-10T04:36:07.184682+0000 mon.a (mon.0) 1384 : cluster [DBG] osdmap e550: 8 total, 8 up, 8 in 2021-02-10T04:36:08.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: 2021-02-10T04:36:08.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:08.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: 02-10T04:36:08.112427+0000 mgr.y (mgr.14140) 621 : 2021-02-10T04:36:08.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: cluster [DBG] pgmap v975: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 417 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 482 KiB/s wr, 6 op/s 2021-02-10T04:36:08.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: audit 2021-02-10T04:36:08.126958+0000 2021-02-10T04:36:08.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: mgr.y (mgr.14140) 622 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:36:08.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:08.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: -10T04:36:08.167036+0000 mon.a (mon 2021-02-10T04:36:08.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: .0) 1385 : cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 392 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:36:08.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:36:08.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: :36:08.172233+0000 mon.a (mon.0) 1386 : 2021-02-10T04:36:08.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[35961]: cluster [DBG] osdmap e551: 8 total, 8 up, 8 in 2021-02-10T04:36:08.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: cluster 2021-02-10T04:36:07.043079+0000 osd.2 (osd.2) 1519 : cluster [WRN] 2021-02-10T04:36:08.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:08.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: cluster 2021-02-10T04:36:08.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: 2021-02-10T04:36 2021-02-10T04:36:08.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: :07.043091+0000 osd.2 ( 2021-02-10T04:36:08.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: osd.2) 2021-02-10T04:36:08.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: 1520 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:08.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: cluster 2021-02-10T04:36:08.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: 2021-02-10T04:36:08.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: 2021 2021-02-10T04:36:08.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: - 2021-02-10T04:36:08.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: 02 2021-02-10T04:36:08.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: - 2021-02-10T04:36:08.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: 10T04:36:07 2021-02-10T04:36:08.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: .043095+0000 osd.2 (osd.2) 1521 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:08.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: cluster 2021-02-10T04:36:08.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: 2021-02 2021-02-10T04:36:08.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: -10T04:36:07.043099+0000 osd.2 (osd.2) 2021-02-10T04:36:08.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: 1522 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:08.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: cluster 2021-02-10T04:36:08.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: 2021-02 2021-02-10T04:36:08.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: -10T04:36:07.043103+0000 osd.2 (osd. 2021-02-10T04:36:08.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: 2) 1523 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:08.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: cluster 2021-02-10T04:36:08.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: 2021-02-10T04:36:07.043106+0000 2021-02-10T04:36:08.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: osd.2 (osd.2) 1524 : cluster [WRN] 2021-02-10T04:36:08.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:08.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: cluster 2021-02-10T04:36:07 2021-02-10T04:36:08.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: .043113+0000 osd.2 (osd.2) 1525 : cluster 2021-02-10T04:36:08.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:08.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:36:08.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: 36:07.043118+0000 osd.2 (osd.2) 2021-02-10T04:36:08.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: 1526 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:08.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:36:08.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: 04:36:07.043122+0000 osd.2 (osd. 2021-02-10T04:36:08.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: 2) 1527 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:08.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:08.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: 02-10T04:36:07.043128+0000 2021-02-10T04:36:08.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: osd.2 (osd.2) 1528 : cluster [WRN] 2021-02-10T04:36:08.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:08.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: cluster 2021-02-10T04:36:07 2021-02-10T04:36:08.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: .088194+0000 osd.4 (osd.4) 807 : 2021-02-10T04:36:08.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:08.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:36:08.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: 36:07.088200+0000 osd.4 (osd.4 2021-02-10T04:36:08.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: ) 808 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:08.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:36:08.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: 10T04:36:07.088204+0000 osd.4 2021-02-10T04:36:08.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: (osd.4) 809 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:08.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:08.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: 02-10T04:36:07.088208+0000 2021-02-10T04:36:08.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: osd.4 (osd.4) 810 : cluster [WRN] 2021-02-10T04:36:08.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:08.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: cluster 2021-02-10T04:36:07 2021-02-10T04:36:08.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: .184682+0000 mon.a (mon.0) 1384 : 2021-02-10T04:36:08.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: cluster [DBG] osdmap e550: 8 total, 8 up, 8 in 2021-02-10T04:36:08.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:36:08.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: 04:36:08.112427+0000 mgr.y (mgr 2021-02-10T04:36:08.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: .14140) 621 : cluster [DBG] pgmap v975: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 417 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 482 KiB/s wr, 6 op/s 2021-02-10T04:36:08.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:36:08.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: 02-10T04:36:08.126958+0000 2021-02-10T04:36:08.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: mgr.y (mgr.14140) 622 : audit [DBG] 2021-02-10T04:36:08.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:36:08.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: cluster 2021-02-10T04:36:08 2021-02-10T04:36:08.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: .167036+0000 mon.a (mon.0) 1385 : 2021-02-10T04:36:08.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 392 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:36:08.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:36:08.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: 04:36:08.172233+0000 mon.a (mon 2021-02-10T04:36:08.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[29319]: .0) 1386 : cluster [DBG] osdmap e551: 8 total, 8 up, 8 in 2021-02-10T04:36:09.194 INFO:tasks.rados.rados.0.smithi085.stdout:2116: finishing write tid 1 to smithi08575397-13 2021-02-10T04:36:09.194 INFO:tasks.rados.rados.0.smithi085.stdout:2116: finishing write tid 2 to smithi08575397-13 2021-02-10T04:36:09.194 INFO:tasks.rados.rados.0.smithi085.stdout:2116: finishing write tid 3 to smithi08575397-13 2021-02-10T04:36:09.194 INFO:tasks.rados.rados.0.smithi085.stdout:2116: finishing write tid 4 to smithi08575397-13 2021-02-10T04:36:09.194 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 748 (ObjNum 742 snap 238 seq_num 742) dirty exists 2021-02-10T04:36:09.194 INFO:tasks.rados.rados.0.smithi085.stdout:2116: left oid 13 (ObjNum 742 snap 238 seq_num 742) 2021-02-10T04:36:09.194 INFO:tasks.rados.rados.0.smithi085.stdout:2117: finishing rollback tid 0 to smithi08575397-37 2021-02-10T04:36:09.194 INFO:tasks.rados.rados.0.smithi085.stdout:2117: finishing rollback tid 1 to smithi08575397-37 2021-02-10T04:36:09.194 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 1134 (ObjNum 82867360 snap 22030 seq_num 82533296) dirty dne 2021-02-10T04:36:09.194 INFO:tasks.rados.rados.0.smithi085.stdout:2116: done (11 left) 2021-02-10T04:36:09.195 INFO:tasks.rados.rados.0.smithi085.stdout:2117: done (10 left) 2021-02-10T04:36:09.195 INFO:tasks.rados.rados.0.smithi085.stdout:2118: done (9 left) 2021-02-10T04:36:09.195 INFO:tasks.rados.rados.0.smithi085.stdout:2119: read oid 22 snap -1 2021-02-10T04:36:09.195 INFO:tasks.rados.rados.0.smithi085.stdout:2119: expect deleted 2021-02-10T04:36:09.195 INFO:tasks.rados.rados.0.smithi085.stdout:2120: snap_create 2021-02-10T04:36:09.386 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:36:09.421 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2021-02-10T04:36:09.421 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 0 2021-02-10T04:36:09.421 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:36:09.422 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: cluster 2021-02-10T04:36:07.843315+0000 osd.2 (osd.2) 1529 : 2021-02-10T04:36:09.422 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: cluster [DBG] purged_snaps scrub starts 2021-02-10T04:36:09.422 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: cluster 2021-02-10T04:36:07.843563+0000 osd.2 (osd.2) 1530 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:36:09.423 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: cluster 2021-02-10T04:36:08.026024+0000 osd.2 (osd.2) 1531 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:09.423 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:36:09.423 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: 04:36:08.026043+0000 osd.2 (osd.2) 1532 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:09.423 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: cluster 2021-02-10T04:36:09.423 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: 2021-02-10T04:36: 2021-02-10T04:36:09.423 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: 08.026049+0000 osd.2 (osd. 2021-02-10T04:36:09.423 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: 2) 1533 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:09.424 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: 2021-02-10T04:36:09.424 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:36:09.424 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: :36:08.026056+0000 osd.2 2021-02-10T04:36:09.424 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: (osd.2) 1534 : cluster 2021-02-10T04:36:09.424 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:09.424 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:09.424 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: -10T04:36:08. 2021-02-10T04:36:09.424 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: 026062+0000 osd.2 (osd.2) 1535 2021-02-10T04:36:09.425 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:09.425 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:09.425 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: -02-10T04:36:08. 2021-02-10T04:36:09.425 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: 026067+0000 osd.2 (osd.2) 1536 2021-02-10T04:36:09.425 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:09.425 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: cluster 2021-02-10T04:36:09.425 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: 2021-02 2021-02-10T04:36:09.425 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: -10T04:36:08.026074 2021-02-10T04:36:09.426 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1537 : cluster 2021-02-10T04:36:09.426 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:09.426 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:09.426 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: -10T04:36:08.026080+0000 2021-02-10T04:36:09.426 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: osd.2 (osd.2) 1538 : cluster 2021-02-10T04:36:09.427 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:09.427 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:09.427 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: 02-10T04:36:08. 2021-02-10T04:36:09.427 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: 026085+0000 osd.2 (osd.2) 1539 : cluster 2021-02-10T04:36:09.427 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:09.427 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:09.427 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: -10T04:36:08.026093+0000 2021-02-10T04:36:09.427 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: osd.2 (osd.2) 1540 : cluster 2021-02-10T04:36:09.427 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:09.428 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:09.428 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: 02-10T04:36:08.104809 2021-02-10T04:36:09.428 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 811 : cluster 2021-02-10T04:36:09.428 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:09.428 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:09.428 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: 02-10T04:36:08.104818 2021-02-10T04:36:09.428 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 812 : 2021-02-10T04:36:09.428 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:09.428 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:09.429 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: -02-10T04:36:08 2021-02-10T04:36:09.429 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: .104826+0000 osd.4 (osd.4) 813 2021-02-10T04:36:09.429 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:09.429 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: cluster 2021-02-10T04:36:09.429 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: 2021-02-10T04:36: 2021-02-10T04:36:09.429 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: 08.104832+0000 osd.4 (osd.4 2021-02-10T04:36:09.429 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: ) 814 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:09.429 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: cluster 2021-02-10T04:36:09.429 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: 2021-02- 2021-02-10T04:36:09.430 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: 10T04:36:09.193564+0000 2021-02-10T04:36:09.430 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: mon.a (mon.0) 1387 : cluster 2021-02-10T04:36:09.430 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[29288]: [DBG] osdmap e552: 8 total, 8 up, 8 in 2021-02-10T04:36:09.430 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:09 smithi164 conmon[33677]: debug 2021-02-10T04:36:09.130+0000 7fbec19a1700 -1 osd.4 551 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:09.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:08 smithi085 conmon[49106]: debug 2021-02-10T04:36:08.981+0000 7f77a3e94700 -1 osd.2 551 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:09.442 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[52836]: debug 2021-02-10T04:36:09.165+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:36:09.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: cluster 2021-02-10T04:36: 2021-02-10T04:36:09.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: 07.843315+0000 osd.2 (osd.2) 1529 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:36:09.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:09.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: T04:36:07.843563+0000 osd.2 2021-02-10T04:36:09.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: (osd.2) 1530 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:36:09.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: 2021-02-10T04:36:09.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: cluster 2021-02-10T04:36:09.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: 2021-02-10T04:36:08.026024+0000 2021-02-10T04:36:09.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: osd.2 (osd.2) 1531 : cluster [WRN] 2021-02-10T04:36:09.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:09.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: cluster 2021-02-10T04:36:09.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: 2021-02-10T04:36:08.026043+0000 2021-02-10T04:36:09.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: osd.2 (osd.2) 1532 : cluster [WRN] 2021-02-10T04:36:09.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:09.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: cluster 2021-02-10T04:36:09.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: 2021-02-10T04:36:08.026049+0000 2021-02-10T04:36:09.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: osd.2 (osd.2) 1533 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:09.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: 2021-02-10T04:36:09.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:09.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: -02-10T04:36:08. 2021-02-10T04:36:09.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: 026056+0000 osd.2 (osd.2) 1534 : 2021-02-10T04:36:09.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:09.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:36:09.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: :36:08.026062+0000 osd.2 (osd. 2021-02-10T04:36:09.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: 2) 1535 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:09.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:09.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: -02-10T04:36:08 2021-02-10T04:36:09.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: .026067+0000 osd.2 (osd.2) 1536 : 2021-02-10T04:36:09.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:09.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: cluster 2021-02-10T04:36:09.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: 2021-02-10T04:36:08 2021-02-10T04:36:09.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: .026074+0000 osd.2 (osd.2) 1537 2021-02-10T04:36:09.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:09.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: cluster 2021-02-10T04:36:09.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:36:09.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: 36:08.026080+0000 osd.2 (osd.2 2021-02-10T04:36:09.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: cluster 2021-02-10T04:36:07.843315+0000 2021-02-10T04:36:09.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: osd.2 (osd.2) 1529 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:36:09.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: cluster 2021-02-10T04:36:07.843563+0000 osd.2 (osd.2) 1530 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:36:09.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: cluster 2021-02-10T04:36:08.026024+0000 osd.2 (osd.2) 1531 : cluster [WRN] 2021-02-10T04:36:09.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:09.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:09.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: -02-10T04:36:08.026043+0000 osd.2 2021-02-10T04:36:09.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: (osd.2) 1532 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:09.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:36:09.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: 10T04:36:08.026049+0000 osd.2 (osd.2) 1533 2021-02-10T04:36:09.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:09.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:36:09.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: 04:36:08.026056+0000 osd.2 (osd. 2021-02-10T04:36:09.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: 2) 1534 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:09.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:36:09.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: 10T04:36:08.026062+0000 osd.2 2021-02-10T04:36:09.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: (osd.2) 1535 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:09.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: cluster 2021-02-10T04:36:09.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: 2021-02-10T04:36:08. 2021-02-10T04:36:09.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: 026067+0000 osd.2 (osd.2) 2021-02-10T04:36:09.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: 1536 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:09.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: 2021-02-10T04:36:09.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:36:09.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: :36:08. 2021-02-10T04:36:09.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: 026074+0000 osd.2 (osd 2021-02-10T04:36:09.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: .2) 1537 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:09.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:09.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: -02-10T04:36:08. 2021-02-10T04:36:09.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: 026080+0000 osd.2 (osd.2) 1538 : 2021-02-10T04:36:09.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:09.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:09.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: -10T04:36:08.026085+0000 2021-02-10T04:36:09.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: osd.2 (osd.2) 1539 : cluster [WRN] 2021-02-10T04:36:09.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: ) 1538 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:09.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:09.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: 02-10T04:36:08.026085+0000 2021-02-10T04:36:09.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: osd.2 (osd.2) 1539 : cluster 2021-02-10T04:36:09.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:09.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: cluster 2021-02-10T04:36 2021-02-10T04:36:09.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: :08.026093+0000 osd.2 (osd.2) 2021-02-10T04:36:09.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: 1540 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:09.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:36:09.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: 04:36:08.104809+0000 osd.4 (osd 2021-02-10T04:36:09.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: .4) 811 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:09.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:09.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: 02-10T04:36:08.104818 2021-02-10T04:36:09.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 812 : cluster 2021-02-10T04:36:09.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:09.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: cluster 2021-02-10T04:36 2021-02-10T04:36:09.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: :08.104826+0000 osd.4 (osd.4) 2021-02-10T04:36:09.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: 813 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:09.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:09.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: T04:36:08.104832+0000 osd.4 2021-02-10T04:36:09.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: (osd.4) 814 : cluster [WRN] 2021-02-10T04:36:09.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:09.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: cluster 2021-02-10T04:36:09.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: 2021-02-10T04:36:09.193564+0000 2021-02-10T04:36:09.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: mon.a (mon.0) 1387 : cluster 2021-02-10T04:36:09.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[35961]: [DBG] osdmap e552: 8 total, 8 up, 8 in 2021-02-10T04:36:09.454 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:09.454 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: cluster 2021-02-10T04:36:08 2021-02-10T04:36:09.455 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: .026093+0000 osd.2 (osd.2) 1540 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:09.455 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:36:09.455 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: 04:36:08.104809+0000 osd.4 (osd.4) 811 : 2021-02-10T04:36:09.455 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:09.455 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:09.455 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: -10T04:36:08.104818+0000 osd.4 (osd.4) 812 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:09.455 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: cluster 2021-02-10T04:36 2021-02-10T04:36:09.455 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: :08.104826+0000 osd.4 (osd.4) 813 : cluster 2021-02-10T04:36:09.456 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:09.456 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:36:09.456 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: 04:36:08.104832+0000 osd.4 ( 2021-02-10T04:36:09.456 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: osd.4) 814 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:09.456 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:09.456 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: -02-10T04:36:09.193564 2021-02-10T04:36:09.456 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1387 : cluster [DBG] osdmap e552: 8 total, 8 up, 8 in 2021-02-10T04:36:09.456 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:09 smithi085 conmon[29319]: 2021-02-10T04:36:09.639 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:36:10.191 INFO:tasks.rados.rados.0.smithi085.stdout:2119: done (10 left) 2021-02-10T04:36:10.192 INFO:tasks.rados.rados.0.smithi085.stdout:2120: done (9 left) 2021-02-10T04:36:10.192 INFO:tasks.rados.rados.0.smithi085.stdout:2121: read oid 35 snap -1 2021-02-10T04:36:10.192 INFO:tasks.rados.rados.0.smithi085.stdout:2121: expect (ObjNum 578 snap 176 seq_num 578) 2021-02-10T04:36:10.192 INFO:tasks.rados.rados.0.smithi085.stdout:2122: write oid 8 current snap is 239 2021-02-10T04:36:10.192 INFO:tasks.rados.rados.0.smithi085.stdout:2122: seq_num 743 ranges {487601=796248,1962121=525480} 2021-02-10T04:36:10.199 INFO:tasks.rados.rados.0.smithi085.stdout:2122: writing smithi08575397-8 from 487601 to 1283849 tid 1 2021-02-10T04:36:10.207 INFO:tasks.rados.rados.0.smithi085.stdout:2122: writing smithi08575397-8 from 1962121 to 2487601 tid 2 2021-02-10T04:36:10.208 INFO:tasks.rados.rados.0.smithi085.stdout:2123: write (excl) oid 11 current snap is 239 2021-02-10T04:36:10.208 INFO:tasks.rados.rados.0.smithi085.stdout:2123: seq_num 744 ranges {504408=471497,1620603=557925} 2021-02-10T04:36:10.214 INFO:tasks.rados.rados.0.smithi085.stdout:2123: writing smithi08575397-11 from 504408 to 975905 tid 1 2021-02-10T04:36:10.222 INFO:tasks.rados.rados.0.smithi085.stdout:2123: writing smithi08575397-11 from 1620603 to 2178528 tid 2 2021-02-10T04:36:10.222 INFO:tasks.rados.rados.0.smithi085.stdout:2124: snap_create 2021-02-10T04:36:10.222 INFO:tasks.rados.rados.0.smithi085.stdout:2121: expect (ObjNum 578 snap 176 seq_num 578) 2021-02-10T04:36:10.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[49106]: debug 2021-02-10T04:36:10.014+0000 7f77a3e94700 -1 osd.2 552 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:10.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: cluster 2021-02-10T04:36:08.983109+0000 osd.2 ( 2021-02-10T04:36:10.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: osd.2) 1541 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:10.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:36:10.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: T04:36:08.983118+0000 osd.2 (osd.2) 1542 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:10.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:10.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: -10T04:36:08.983124+0000 osd.2 (osd. 2021-02-10T04:36:10.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: 2) 1543 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:10.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:10.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: 02-10T04:36:08.983129+0000 osd.2 2021-02-10T04:36:10.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: (osd.2) 1544 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:10.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:10.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: 02-10T04:36:08.983132+0000 osd.2 2021-02-10T04:36:10.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: (osd.2) 1545 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:10.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:10.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: -10T04:36:08.983136+0000 osd.2 (osd 2021-02-10T04:36:10.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: .2) 1546 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:10.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:36:10.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: 10T04:36:08.983143+0000 osd.2 (osd 2021-02-10T04:36:10.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: .2) 1547 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:10.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:36:10.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: 04:36:08.983149+0000 osd.2 (osd.2) 2021-02-10T04:36:10.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: 1548 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:10.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: cluster 2021-02-10T04:36 2021-02-10T04:36:10.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: :08.983155+0000 osd.2 (osd.2) 1549 : cluster 2021-02-10T04:36:10.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:10.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: cluster 2021-02-10T04:36:08. 2021-02-10T04:36:10.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: 983159+0000 osd.2 (osd.2) 1550 : cluster [WRN] 2021-02-10T04:36:10.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: cluster 2021-02-10T04:36:10.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: 2021-02-10T04:36:10.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: 2021 2021-02-10T04:36:10.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: - 2021-02-10T04:36:10.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: 02-10T04:36 2021-02-10T04:36:10.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: :08.983109+0000 osd.2 (osd.2) 1541 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:10.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:10.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: -10T04:36:08.983118+0000 osd.2 (osd.2 2021-02-10T04:36:10.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: ) 1542 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:10.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:10.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: -02-10T04:36:08.983124+0000 2021-02-10T04:36:10.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: osd.2 (osd.2) 1543 : cluster [WRN] 2021-02-10T04:36:10.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:10.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: cluster 2021-02-10T04:36 2021-02-10T04:36:10.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: :08.983129+0000 osd.2 (osd.2) 2021-02-10T04:36:10.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: 1544 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:10.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:10.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: -10T04:36:08.983132+0000 osd.2 2021-02-10T04:36:10.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: (osd.2) 1545 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:10.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: 2021-02-10T04:36:10.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: cluster 2021-02-10T04:36:08 2021-02-10T04:36:10.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: .983136+0000 osd.2 (osd.2) 1546 : 2021-02-10T04:36:10.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:10.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:36:10.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: 10T04:36:08.983143+0000 osd.2 2021-02-10T04:36:10.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: (osd.2) 1547 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:10.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: 2021-02-10T04:36:10.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: cluster 2021-02-10T04:36: 2021-02-10T04:36:10.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: 08.983149+0000 osd.2 (osd.2) 1548 2021-02-10T04:36:10.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:10.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:10.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: T04:36:08.983155+0000 osd.2 ( 2021-02-10T04:36:10.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: osd.2) 1549 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:10.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:10.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: -02-10T04:36:08.983159+0000 2021-02-10T04:36:10.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: osd.2 (osd.2) 1550 : cluster [WRN] 2021-02-10T04:36:10.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:10.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: cluster 2021-02-10T04:36: 2021-02-10T04:36:10.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: 09.131110+0000 osd.4 (osd.4) 815 2021-02-10T04:36:10.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:10.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:10.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: T04:36:09.131122+0000 osd.4 (osd 2021-02-10T04:36:10.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: .4) 816 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:10.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:10.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: -02-10T04:36:09.131129+0000 2021-02-10T04:36:10.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: osd.4 (osd.4) 817 : cluster [WRN] 2021-02-10T04:36:10.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:10.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: cluster 2021-02-10T04:36 2021-02-10T04:36:10.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: :09.131135+0000 osd.4 (osd.4) 818 2021-02-10T04:36:10.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:10.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:10.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: -10T04:36:10.114041+0000 mgr.y 2021-02-10T04:36:10.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: (mgr.14140) 623 : cluster [DBG] pgmap v978: 168 pgs: 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 156 active+clean; 419 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 1.3 MiB/s wr, 4 op/s 2021-02-10T04:36:10.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: 2021-02-10T04:36:10.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: cluster 2021-02-10T04:36 2021-02-10T04:36:10.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: :10.189886+0000 mon.a (mon.0) 2021-02-10T04:36:10.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[35961]: 1388 : cluster [DBG] osdmap e553: 8 total, 8 up, 8 in 2021-02-10T04:36:10.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:10.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: cluster 2021-02-10T04:36:10.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:36:10.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: 36:09.131110+0000 osd.4 (osd.4) 815 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:10.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:10.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: -10T04:36:09.131122+0000 osd.4 (osd. 2021-02-10T04:36:10.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: 4) 816 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:10.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: 2021-02-10T04:36:10.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:10.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: -02-10T04: 2021-02-10T04:36:10.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: 36:09.131129+0000 osd.4 2021-02-10T04:36:10.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: (osd.4) 817 : cluster [WRN] 2021-02-10T04:36:10.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:10.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:10.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: 02-10T04:36:09.131135+0000 osd.4 2021-02-10T04:36:10.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: (osd.4) 818 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:10.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:10.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: 02-10T04:36:10.114041+0000 mgr.y (mgr 2021-02-10T04:36:10.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: .14140) 623 : cluster [DBG] pgmap v978: 168 pgs: 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 156 active+clean; 419 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 1.3 MiB/s wr, 4 op/s 2021-02-10T04:36:10.454 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: cluster 2021-02-10T04:36 2021-02-10T04:36:10.454 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[29319]: :10.189886+0000 mon.a (mon.0) 1388 : cluster [DBG] osdmap e553: 8 total, 8 up, 8 in 2021-02-10T04:36:10.576 INFO:tasks.rados.rados.0.smithi085.stdout:2122: finishing write tid 1 to smithi08575397-8 2021-02-10T04:36:10.576 INFO:tasks.rados.rados.0.smithi085.stdout:2122: finishing write tid 2 to smithi08575397-8 2021-02-10T04:36:10.576 INFO:tasks.rados.rados.0.smithi085.stdout:2122: finishing write tid 3 to smithi08575397-8 2021-02-10T04:36:10.576 INFO:tasks.rados.rados.0.smithi085.stdout:2122: finishing write tid 4 to smithi08575397-8 2021-02-10T04:36:10.576 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 1107 (ObjNum 743 snap 239 seq_num 743) dirty exists 2021-02-10T04:36:10.577 INFO:tasks.rados.rados.0.smithi085.stdout:2122: left oid 8 (ObjNum 743 snap 239 seq_num 743) 2021-02-10T04:36:10.577 INFO:tasks.rados.rados.0.smithi085.stdout:2123: finishing write tid 1 to smithi08575397-11 2021-02-10T04:36:10.577 INFO:tasks.rados.rados.0.smithi085.stdout:2123: finishing write tid 2 to smithi08575397-11 2021-02-10T04:36:10.577 INFO:tasks.rados.rados.0.smithi085.stdout:2123: finishing write tid 3 to smithi08575397-11 2021-02-10T04:36:10.577 INFO:tasks.rados.rados.0.smithi085.stdout:2123: finishing write tid 4 to smithi08575397-11 2021-02-10T04:36:10.577 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 1140 (ObjNum 744 snap 239 seq_num 744) dirty exists 2021-02-10T04:36:10.577 INFO:tasks.rados.rados.0.smithi085.stdout:2123: left oid 11 (ObjNum 744 snap 239 seq_num 744) 2021-02-10T04:36:10.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[33677]: debug 2021-02-10T04:36:10.166+0000 7fbec19a1700 -1 osd.4 552 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:10.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: cluster 2021-02-10T04:36:08.983109+0000 osd.2 2021-02-10T04:36:10.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: (osd.2) 1541 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:10.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: cluster 2021-02-10T04:36:08.983118+0000 osd.2 (osd.2) 1542 : cluster [WRN] 2021-02-10T04:36:10.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:10.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: cluster 2021-02-10T04:36:10.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: 2021-02-10T04:36:08.983124+0000 osd.2 (osd. 2021-02-10T04:36:10.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: 2) 1543 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:10.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:36:10.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: 10T04:36:08 2021-02-10T04:36:10.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: .983129+0000 osd.2 (osd.2) 1544 : 2021-02-10T04:36:10.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:10.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: cluster 2021-02-10T04:36 2021-02-10T04:36:10.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: :08.983132+0000 osd.2 (osd.2 2021-02-10T04:36:10.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: ) 1545 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:10.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:36:10.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: 10T04:36:08.983136+0000 osd.2 ( 2021-02-10T04:36:10.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: osd.2) 1546 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:10.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:36:10.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: 04:36:08.983143+0000 osd.2 (osd 2021-02-10T04:36:10.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: .2) 1547 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:10.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: cluster 2021-02-10T04:36:10.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: 2021-02-10T04:36 2021-02-10T04:36:10.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: :08.983149+0000 osd.2 ( 2021-02-10T04:36:10.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: osd.2) 1548 : cluster 2021-02-10T04:36:10.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:10.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:10.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: -02- 2021-02-10T04:36:10.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: 10T04:36:08. 2021-02-10T04:36:10.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: 983155+0000 osd.2 (osd.2 2021-02-10T04:36:10.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: ) 1549 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:10.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:10.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: -10T04:36:08.983159 2021-02-10T04:36:10.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1550 : 2021-02-10T04:36:10.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:10.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:36:10.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: 36:09.131110+0000 osd.4 (osd.4) 2021-02-10T04:36:10.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: 815 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:10.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:36:10.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: T04:36:09.131122+0000 osd.4 2021-02-10T04:36:10.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: (osd.4) 816 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:10.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: cluster 2021-02-10T04:36:10.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:36:10.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: :36:09.131129+0000 2021-02-10T04:36:10.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: osd.4 (osd.4) 817 : cluster 2021-02-10T04:36:10.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:10.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: cluster 2021-02-10T04:36 2021-02-10T04:36:10.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: :09.131135+0000 osd.4 (osd.4) 2021-02-10T04:36:10.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: 818 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:10.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:36:10.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: :36:10.114041+0000 mgr.y (mgr. 2021-02-10T04:36:10.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: 14140) 623 : cluster [DBG] 2021-02-10T04:36:10.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: pgmap v978: 168 pgs: 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 156 active+clean; 419 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 1.3 MiB/s wr, 4 op/s 2021-02-10T04:36:10.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:36:10.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: T04:36:10.189886 2021-02-10T04:36:10.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1388 : 2021-02-10T04:36:10.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:10 smithi164 conmon[29288]: cluster [DBG] osdmap e553: 8 total, 8 up, 8 in 2021-02-10T04:36:11.216 INFO:tasks.rados.rados.0.smithi085.stdout:2121: done (12 left) 2021-02-10T04:36:11.216 INFO:tasks.rados.rados.0.smithi085.stdout:2122: done (11 left) 2021-02-10T04:36:11.216 INFO:tasks.rados.rados.0.smithi085.stdout:2123: done (10 left) 2021-02-10T04:36:11.217 INFO:tasks.rados.rados.0.smithi085.stdout:2124: done (9 left) 2021-02-10T04:36:11.217 INFO:tasks.rados.rados.0.smithi085.stdout:2125: snap_remove snap 218 2021-02-10T04:36:11.227 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:10 smithi085 conmon[49106]: debug 2021-02-10T04:36:10.976+0000 7f77a3e94700 -1 osd.2 553 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:11.228 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: cluster 2021-02-10T04:36:10.016076+0000 osd.2 ( 2021-02-10T04:36:11.228 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: osd.2) 1551 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:11.228 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: cluster 2021-02-10T04:36:10.016086+0000 osd.2 (osd.2) 1552 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:11.228 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: cluster 2021-02-10T04:36:10.016091+0000 osd.2 (osd 2021-02-10T04:36:11.228 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: .2) 1553 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:11.228 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:36:11.229 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: :36:10.016095+0000 osd.2 (osd.2) 1554 2021-02-10T04:36:11.229 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:11.229 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:36:11.229 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: 36:10.016099+0000 osd.2 (osd.2) 1555 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:11.229 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: cluster 2021-02-10T04:36:10.016103 2021-02-10T04:36:11.229 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1556 : cluster [WRN] 2021-02-10T04:36:11.229 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:11.229 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: cluster 2021-02-10T04:36:11.229 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: 2021-02-10T04:36:10.016110+0000 osd.2 ( 2021-02-10T04:36:11.229 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: osd.2) 1557 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:11.229 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:11.229 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: 02-10T04:36:10.016113+0000 2021-02-10T04:36:11.229 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: osd.2 (osd.2) 1558 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:11.230 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: 2021-02-10T04:36:11.230 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: cluster 2021-02-10T04:36:11.230 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: 2021-02-10T04:36:10.016117+0000 osd.2 2021-02-10T04:36:11.230 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: (osd.2) 1559 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:11.230 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:11.230 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: 02-10T04:36:10.016125+0000 2021-02-10T04:36:11.230 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: osd.2 (osd.2) 1560 : cluster [WRN] 2021-02-10T04:36:11.230 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:11.230 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: cluster 2021-02-10T04:36:11.230 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: 2021-02-10T04:36:10.167063+0000 osd.4 (osd.4) 819 2021-02-10T04:36:11.230 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:11.230 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:36:11.230 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: 10T04:36:10.167076+0000 2021-02-10T04:36:11.231 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: osd.4 (osd.4) 820 : cluster [WRN] 2021-02-10T04:36:11.231 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:11.231 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: cluster 2021-02-10T04:36:10.167081 2021-02-10T04:36:11.231 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 821 : cluster [WRN] 2021-02-10T04:36:11.231 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:11.231 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: cluster 2021-02-10T04:36:10. 2021-02-10T04:36:11.231 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: 167085+0000 osd.4 (osd.4) 822 : cluster 2021-02-10T04:36:11.231 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:11.232 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: cluster 2021-02-10T04:36:10.016076+0000 osd.2 (osd. 2021-02-10T04:36:11.232 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: 2) 1551 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:11.232 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:36:11.232 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: 36:10.016086+0000 osd.2 (osd.2) 1552 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:11.232 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: cluster 2021-02-10T04:36:11.232 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:36:11.232 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: 36:10.016091+0000 osd.2 (osd.2 2021-02-10T04:36:11.232 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: ) 1553 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:11.232 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:11.232 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: -10T04:36:10.016095+0000 osd.2 (osd.2) 2021-02-10T04:36:11.232 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: 1554 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:11.232 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: cluster 2021-02-10T04:36:11.233 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:36:11.233 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: 36:10.016099+0000 osd.2 (osd.2) 2021-02-10T04:36:11.233 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: 1555 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:11.233 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:11.233 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: -02-10T04:36:10 2021-02-10T04:36:11.233 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: .016103+0000 osd.2 (osd.2) 1556 : 2021-02-10T04:36:11.233 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:11.233 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: cluster 2021-02-10T04:36:11.233 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: 2021-02-10T04:36 2021-02-10T04:36:11.233 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: :10.016110+0000 osd.2 (osd.2) 1557 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:11.233 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:11.233 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: 02-10T04:36:10.016113+0000 osd.2 (osd 2021-02-10T04:36:11.233 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: .2) 1558 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:11.234 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:11.234 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: -02-10T04:36:10.016117+0000 osd.2 (osd.2) 1559 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:11.234 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:11.234 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: 02-10T04:36:10.016125+0000 osd.2 (osd 2021-02-10T04:36:11.234 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: .2) 1560 : cluster [WRN] 2021-02-10T04:36:11.234 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:11.234 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:36:11.234 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: :36:10.167063+0000 osd.4 (osd.4) 819 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:11.234 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: cluster 2021-02-10T04:36:10 2021-02-10T04:36:11.234 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: .167076+0000 osd.4 (osd.4 2021-02-10T04:36:11.234 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: ) 820 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:11.234 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: 2021-02-10T04:36:11.234 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:11.235 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: -02-10T04:36: 2021-02-10T04:36:11.235 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: 10.167081+0000 osd.4 (osd.4 2021-02-10T04:36:11.235 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: ) 821 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:11.235 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:36:11.235 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: :36:10.167085+0000 osd.4 (osd.4) 822 2021-02-10T04:36:11.235 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:11.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[33677]: debug 2021-02-10T04:36:11.144+0000 7fbec19a1700 -1 osd.4 553 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:11.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: cluster 2021-02-10T04:36:10.016076 2021-02-10T04:36:11.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1551 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:11.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: cluster 2021-02-10T04:36:10.016086+0000 osd.2 (osd.2) 1552 : cluster 2021-02-10T04:36:11.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:11.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: cluster 2021-02-10T04:36:10 2021-02-10T04:36:11.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: .016091+0000 osd.2 (osd.2) 1553 : cluster [WRN] 2021-02-10T04:36:11.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:11.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:36:11.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: 04:36:10.016095+0000 osd.2 (osd.2) 2021-02-10T04:36:11.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: 1554 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:11.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: cluster 2021-02-10T04:36:11.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: 2021-02-10T04:36 2021-02-10T04:36:11.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: :10.016099+0000 osd.2 ( 2021-02-10T04:36:11.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: osd.2) 1555 : cluster [WRN] 2021-02-10T04:36:11.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:11.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:36:11.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: T04:36:10.016103+0000 osd.2 (osd 2021-02-10T04:36:11.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: .2) 1556 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:11.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: cluster 2021-02-10T04:36:11.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: 2021-02-10T04:36:10. 2021-02-10T04:36:11.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: 016110+0000 osd.2 (osd.2) 1557 : cluster 2021-02-10T04:36:11.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:11.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:36:11.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: 36:10.016113+0000 osd.2 (osd.2) 2021-02-10T04:36:11.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: 1558 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:11.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: cluster 2021-02-10T04:36:11.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: 2021-02-10T04:36:10. 2021-02-10T04:36:11.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: 016117+0000 osd.2 (osd.2) 1559 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:11.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:36:11.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: 04:36:10.016125+0000 osd.2 (osd 2021-02-10T04:36:11.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: .2) 1560 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:11.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: cluster 2021-02-10T04:36:11.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: 2021-02-10T04:36:10. 2021-02-10T04:36:11.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: 167063+0000 osd.4 (osd.4) 819 : cluster 2021-02-10T04:36:11.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:11.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:36:11.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: 10T04:36:10.167076+0000 osd.4 2021-02-10T04:36:11.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: (osd.4) 820 : cluster 2021-02-10T04:36:11.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:11.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:11.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: -10T04:36:10.167081 2021-02-10T04:36:11.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 821 2021-02-10T04:36:11.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:11.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:11.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: -02-10T04:36:10. 2021-02-10T04:36:11.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: 167085+0000 osd.4 (osd.4 2021-02-10T04:36:11.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[29288]: ) 822 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:11.687 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:36:11.719 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:36:11.872 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.5 dump_blocked_ops 2021-02-10T04:36:11.905 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:36:12.011 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:36:12.039 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:36:11 smithi164 conmon[37263]: debug 2021-02-10T04:36:11.869+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:36:12.042 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:36:12.171 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:36:12.214 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:36:12.215 INFO:tasks.rados.rados.0.smithi085.stdout:2125: done (9 left) 2021-02-10T04:36:12.216 INFO:tasks.rados.rados.0.smithi085.stdout:2126: write oid 5 current snap is 240 2021-02-10T04:36:12.216 INFO:tasks.rados.rados.0.smithi085.stdout:2126: seq_num 745 ranges {521215=546746,1562732=701583} 2021-02-10T04:36:12.222 INFO:tasks.rados.rados.0.smithi085.stdout:2126: writing smithi08575397-5 from 521215 to 1067961 tid 1 2021-02-10T04:36:12.230 INFO:tasks.rados.rados.0.smithi085.stdout:2126: writing smithi08575397-5 from 1562732 to 2264315 tid 2 2021-02-10T04:36:12.230 INFO:tasks.rados.rados.0.smithi085.stdout:2127: write oid 23 current snap is 240 2021-02-10T04:36:12.231 INFO:tasks.rados.rados.0.smithi085.stdout:2127: seq_num 746 ranges {538022=738348,1737567=561594} 2021-02-10T04:36:12.238 INFO:tasks.rados.rados.0.smithi085.stdout:2127: writing smithi08575397-23 from 538022 to 1276370 tid 1 2021-02-10T04:36:12.245 INFO:tasks.rados.rados.0.smithi085.stdout:2127: writing smithi08575397-23 from 1737567 to 2299161 tid 2 2021-02-10T04:36:12.245 INFO:tasks.rados.rados.0.smithi085.stdout:2128: snap_remove snap 224 2021-02-10T04:36:12.277 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:36:12.315 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:11 smithi085 conmon[49106]: debug 2021-02-10T04:36:11.982+0000 7f77a3e94700 -1 osd.2 554 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:12.321 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: cluster 2021-02-10T04:36:10.977778+0000 osd.2 (osd.2 2021-02-10T04:36:12.321 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: ) 1561 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:12.321 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: cluster 2021-02-10T04:36:10.977785+0000 osd.2 (osd.2) 1562 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:12.321 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: cluster 2021-02-10T04:36:12.322 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: 2021-02-10T04:36:10.977789+0000 osd.2 (osd.2) 1563 : cluster 2021-02-10T04:36:12.322 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:12.322 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:36:12.322 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: 10T04:36:10.977793 2021-02-10T04:36:12.322 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1564 : cluster 2021-02-10T04:36:12.322 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:12.322 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:36:12.322 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: T04:36:10.977796+0000 osd.2 2021-02-10T04:36:12.322 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: (osd.2) 1565 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:12.322 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: 2021-02-10T04:36:12.322 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: cluster 2021-02-10T04:36:12.322 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: 2021-02-10T04:36:10 2021-02-10T04:36:12.322 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: .977799+0000 osd.2 (osd.2) 1566 : 2021-02-10T04:36:12.323 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:12.323 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:12.323 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: -10T04:36:10.977803 2021-02-10T04:36:12.323 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1567 : cluster [WRN] 2021-02-10T04:36:12.323 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:12.323 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:36:12.323 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: T04:36:10.977807+0000 2021-02-10T04:36:12.323 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: osd.2 (osd.2) 1568 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:12.323 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: 2021-02-10T04:36:12.323 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: cluster 2021-02-10T04:36:12.323 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: 2021-02-10T04:36:10.977811+0000 2021-02-10T04:36:12.323 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: osd.2 (osd.2) 1569 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:12.324 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:36:12.324 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: 36:10.977815+0000 osd.2 (osd.2) 1570 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:12.324 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: 2021-02-10T04:36:12.324 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:36:12.324 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: 10T04:36:11.145536+0000 osd.4 (osd.4) 823 : cluster [WRN] 2021-02-10T04:36:12.324 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:12.324 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: cluster 2021-02-10T04:36:12.324 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: 2021-02-10T04:36:11.145546+0000 2021-02-10T04:36:12.324 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: osd.4 (osd.4) 824 : cluster [WRN] 2021-02-10T04:36:12.324 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:12.324 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:12.324 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: 02-10T04:36:11.145552+0000 osd.4 (osd. 2021-02-10T04:36:12.324 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: 4) 825 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:12.324 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:12.325 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: -10T04:36:11. 2021-02-10T04:36:12.325 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: 145558+0000 osd.4 (osd.4) 826 : cluster 2021-02-10T04:36:12.325 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:12.325 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: cluster 2021-02-10T04:36:12.325 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: 2021-02-10T04:36:11.215435+0000 2021-02-10T04:36:12.325 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: mon.a (mon.0) 1389 : cluster [DBG] 2021-02-10T04:36:12.325 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: osdmap e554: 8 total, 8 up, 8 in 2021-02-10T04:36:12.325 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: cluster 2021-02-10T04:36 2021-02-10T04:36:12.325 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: :12.114610+0000 mgr.y (mgr.14140) 624 : cluster 2021-02-10T04:36:12.325 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: [DBG] pgmap v981: 168 pgs: 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 156 active+clean; 419 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 1.3 MiB/s wr, 7 op/s 2021-02-10T04:36:12.325 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:36:12.325 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: 10T04:36:12.215381+0000 mon.a 2021-02-10T04:36:12.326 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[29319]: (mon.0) 1390 : cluster [DBG] osdmap e555: 8 total, 8 up, 8 in 2021-02-10T04:36:12.326 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: cluster 2021-02-10T04:36:10.977778+0000 2021-02-10T04:36:12.326 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: osd.2 (osd.2) 1561 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:12.326 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:12.326 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: -10T04:36:10.977785+0000 osd.2 (osd.2) 1562 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:12.326 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: 2021-02-10T04:36:12.326 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: cluster 2021-02-10T04:36:10 2021-02-10T04:36:12.327 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: .977789+0000 osd.2 (osd.2) 1563 : cluster 2021-02-10T04:36:12.327 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:12.327 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:36:12.327 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: 36:10.977793+0000 osd.2 (osd.2) 2021-02-10T04:36:12.327 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: 1564 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:12.328 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:12.328 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: -10T04:36:10.977796+0000 osd.2 2021-02-10T04:36:12.328 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: (osd.2) 1565 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:12.328 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: cluster 2021-02-10T04:36:12.328 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: 2021-02-10T04:36:10.977799 2021-02-10T04:36:12.328 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1566 : cluster 2021-02-10T04:36:12.328 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:12.328 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:36:12.328 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: 36:10.977803+0000 osd.2 (osd.2 2021-02-10T04:36:12.328 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: ) 1567 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:12.328 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: cluster 2021-02-10T04:36:12.328 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:36:12.329 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: :36:10.977807+0000 osd.2 (osd. 2021-02-10T04:36:12.329 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: 2) 1568 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:12.329 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: cluster 2021-02-10T04:36:12.329 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: 2021-02-10T04:36:10.977811 2021-02-10T04:36:12.329 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1569 : cluster 2021-02-10T04:36:12.329 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:12.329 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:12.329 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: T04:36:10.977815+0000 osd.2 ( 2021-02-10T04:36:12.329 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: osd.2) 1570 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:12.329 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: cluster 2021-02-10T04:36:11.145536+0000 osd.4 (osd.4) 2021-02-10T04:36:12.330 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: 823 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:12.330 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:36:12.330 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: 10T04:36:11.145546+0000 osd.4 2021-02-10T04:36:12.330 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: (osd.4) 824 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:12.330 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: 2021-02-10T04:36:12.330 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:36:12.330 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: 36:11.145552+0000 2021-02-10T04:36:12.330 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: osd.4 (osd.4) 825 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:12.330 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: 2021-02-10T04:36:12.330 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:12.330 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: T04:36:11.145558+0000 osd.4 (osd.4) 826 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:12.330 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: cluster 2021-02-10T04:36:11.215435+0000 2021-02-10T04:36:12.330 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: mon.a (mon.0) 1389 : cluster [DBG] 2021-02-10T04:36:12.331 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: osdmap e554: 8 total, 8 up, 8 in 2021-02-10T04:36:12.331 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:36:12.331 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: 10T04:36:12.114610 2021-02-10T04:36:12.331 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 624 : cluster 2021-02-10T04:36:12.331 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: [DBG] pgmap v981: 168 pgs: 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 156 active+clean; 419 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 1.3 MiB/s wr, 7 op/s 2021-02-10T04:36:12.331 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: cluster 2021-02-10T04:36:12.331 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: 2021-02-10T04:36:12.215381+0000 mon.a (mon.0 2021-02-10T04:36:12.331 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[35961]: ) 1390 : cluster [DBG] osdmap e555: 8 total, 8 up, 8 in 2021-02-10T04:36:12.331 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:36:12.393 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[37263]: debug 2021-02-10T04:36:12.126+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:36:12.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[33677]: debug 2021-02-10T04:36:12.123+0000 7fbec19a1700 -1 osd.4 554 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:12.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: cluster 2021-02-10T04:36:10.977778+0000 osd.2 ( 2021-02-10T04:36:12.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: osd.2) 1561 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:12.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: cluster 2021-02-10T04:36:12.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: 2021 2021-02-10T04:36:12.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: - 2021-02-10T04:36:12.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: 02 2021-02-10T04:36:12.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: -10T04:36:10. 2021-02-10T04:36:12.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: 977785+0000 osd.2 (osd.2) 1562 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:12.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: cluster 2021-02-10T04:36:12.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: 2021-02-10T04:36:12.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:36:12.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: :36:10.977789+0000 osd.2 (osd.2) 1563 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:12.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:12.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: -10T04:36:10.977793+0000 osd.2 (osd. 2021-02-10T04:36:12.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: 2) 1564 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:12.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: cluster 2021-02-10T04:36:12.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:36:12.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: 36:10.977796+0000 osd.2 (osd.2) 2021-02-10T04:36:12.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: 1565 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:12.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: cluster 2021-02-10T04:36:12.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: 2021-02-10T04:36:10. 2021-02-10T04:36:12.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: 977799+0000 osd.2 (osd.2) 1566 : cluster 2021-02-10T04:36:12.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:12.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:12.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: -02-10T04:36:10.977803+0000 2021-02-10T04:36:12.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: osd.2 (osd.2) 1567 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:12.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: cluster 2021-02-10T04:36:12.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: 2021-02-10T04:36:10.977807+0000 2021-02-10T04:36:12.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: osd.2 (osd.2) 1568 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:12.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: 2021-02-10T04:36:12.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:12.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: -10T04:36:10.977811+0000 osd.2 ( 2021-02-10T04:36:12.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: osd.2) 1569 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:12.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: cluster 2021-02-10T04:36:12.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: 2021-02-10T04:36: 2021-02-10T04:36:12.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: 10.977815+0000 osd.2 (osd.2) 1570 2021-02-10T04:36:12.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:12.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: cluster 2021-02-10T04:36:12.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: 2021-02-10T04:36:11. 2021-02-10T04:36:12.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: 145536+0000 osd.4 (osd.4) 823 : cluster 2021-02-10T04:36:12.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:12.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:12.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: 02-10T04:36:11.145546+0000 osd.4 2021-02-10T04:36:12.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: (osd.4) 824 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:12.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: cluster 2021-02-10T04:36:12.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: 2021-02-10T04:36:11 2021-02-10T04:36:12.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: .145552+0000 osd.4 (osd.4) 825 : cluster 2021-02-10T04:36:12.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:12.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: cluster 2021-02-10T04:36:12.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: 2021-02-10T04:36:11.145558+0000 2021-02-10T04:36:12.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: osd.4 (osd.4) 826 : cluster [WRN] 2021-02-10T04:36:12.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:12.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:12.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: -02-10T04:36:11.215435+0000 2021-02-10T04:36:12.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: mon.a (mon.0) 1389 : cluster [DBG] osdmap e554: 8 total, 8 up, 8 in 2021-02-10T04:36:12.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: 2021-02-10T04:36:12.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: cluster 2021-02-10T04:36 2021-02-10T04:36:12.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: :12.114610+0000 mgr.y (mgr.14140) 2021-02-10T04:36:12.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: 624 : cluster [DBG] pgmap v981: 168 pgs: 3 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 156 active+clean; 419 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 1.3 MiB/s wr, 7 op/s 2021-02-10T04:36:12.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: cluster 2021-02-10T04:36:12.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: 2021-02-10T04:36:12 2021-02-10T04:36:12.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: .215381+0000 mon.a (mon.0) 1390 : cluster 2021-02-10T04:36:12.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:12 smithi164 conmon[29288]: [DBG] osdmap e555: 8 total, 8 up, 8 in 2021-02-10T04:36:12.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[49106]: debug 2021-02-10T04:36:12.400+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:36:12.699 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:36:13.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:12 smithi085 conmon[49106]: debug 2021-02-10T04:36:12.934+0000 7f77a3e94700 -1 osd.2 555 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:13.281 INFO:tasks.rados.rados.0.smithi085.stdout:2128: done (11 left) 2021-02-10T04:36:13.281 INFO:tasks.rados.rados.0.smithi085.stdout:2129: snap_remove snap 148 2021-02-10T04:36:13.301 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[33677]: debug 2021-02-10T04:36:13.134+0000 7fbec19a1700 -1 osd.4 555 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: cluster 2021-02-10T04:36:11.983748+0000 osd.2 2021-02-10T04:36:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: (osd.2) 1571 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: cluster 2021-02-10T04:36:11.983758+0000 osd.2 (osd.2) 1572 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: cluster 2021-02-10T04:36:11.983764+0000 osd.2 (osd.2) 1573 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: cluster 2021-02-10T04:36:11.983771+0000 osd.2 ( 2021-02-10T04:36:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: osd.2) 1574 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:13.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: 02-10T04:36:11.983776+0000 osd.2 2021-02-10T04:36:13.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: (osd.2) 1575 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:13.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: cluster 2021-02-10T04:36: 2021-02-10T04:36:13.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: 11.983782+0000 osd.2 (osd.2) 1576 : 2021-02-10T04:36:13.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:13.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: cluster 2021-02-10T04:36:11.983786+0000 2021-02-10T04:36:13.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: osd.2 (osd.2) 1577 : cluster [WRN] 2021-02-10T04:36:13.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:13.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:13.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: -02-10T04:36:11.983789+0000 osd.2 (osd.2) 2021-02-10T04:36:13.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: 1578 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:13.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: cluster 2021-02-10T04:36:11.983794+0000 2021-02-10T04:36:13.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: osd.2 (osd.2) 1579 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:13.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:36:13.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: 10T04:36:11.983799+0000 osd.2 (osd 2021-02-10T04:36:13.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: .2) 1580 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:13.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:13.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: -10T04:36:12.123976+0000 osd.4 ( 2021-02-10T04:36:13.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: osd.4) 827 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:13.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:13.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: 02-10T04:36:12.123984+0000 osd.4 2021-02-10T04:36:13.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: (osd.4) 828 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:13.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:13.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: -02-10T04:36:12.123989+0000 2021-02-10T04:36:13.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: osd.4 (osd.4) 829 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:13.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: 2021-02-10T04:36:13.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: cluster 2021-02-10T04:36:13.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: 2021-02-10T04:36:12.123992+0000 osd.4 (osd.4) 830 : cluster 2021-02-10T04:36:13.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:13.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: cluster 2021-02-10T04:36:13. 2021-02-10T04:36:13.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: 168473+0000 mon.a (mon.0) 1391 : cluster 2021-02-10T04:36:13.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:13 smithi164 conmon[29288]: [WRN] Health check update: 14 slow ops, oldest one blocked for 402 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:36:13.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: cluster 2021-02-10T04:36:11.983748+0000 2021-02-10T04:36:13.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: osd.2 (osd.2) 1571 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:13.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: cluster 2021-02-10T04:36:11.983758 2021-02-10T04:36:13.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1572 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:13.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:13.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: -02-10T04:36:11.983764 2021-02-10T04:36:13.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1573 : 2021-02-10T04:36:13.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:13.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:13.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: -02-10T04:36:11.983771+0000 osd.2 (osd 2021-02-10T04:36:13.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: .2) 1574 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:13.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:13.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: -02-10T04:36: 2021-02-10T04:36:13.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: 11.983776+0000 osd.2 (osd 2021-02-10T04:36:13.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: .2) 1575 : cluster [WRN] 2021-02-10T04:36:13.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:13.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:13.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: -02-10T04:36:11.983782+0000 2021-02-10T04:36:13.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: osd.2 (osd.2) 1576 : 2021-02-10T04:36:13.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:13.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: cluster 2021-02-10T04:36:13.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: 2021-02-10T04:36 2021-02-10T04:36:13.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: :11.983786+0000 osd.2 (osd 2021-02-10T04:36:13.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: .2) 1577 : cluster [WRN] 2021-02-10T04:36:13.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:13.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:13.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: -02-10T04:36: 2021-02-10T04:36:13.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: 11.983789+0000 osd.2 (osd 2021-02-10T04:36:13.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: .2) 1578 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:13.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: 2021-02-10T04:36:13.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: cluster 2021-02-10T04:36:13.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: 2021-02-10T04:36:11. 2021-02-10T04:36:13.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: 983794+0000 osd.2 (osd.2) 2021-02-10T04:36:13.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: 1579 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:13.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:13.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: -02-10T04:36:11 2021-02-10T04:36:13.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: .983799+0000 osd.2 (osd.2 2021-02-10T04:36:13.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: ) 1580 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:13.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: 2021-02-10T04:36:13.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:36:13.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: :36:12.123976+0000 osd.4 (osd 2021-02-10T04:36:13.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: .4) 827 : cluster [WRN] 2021-02-10T04:36:13.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:13.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:13.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: -10T04:36:12. 2021-02-10T04:36:13.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: 123984+0000 osd.4 (osd.4 2021-02-10T04:36:13.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: ) 828 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:13.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:13.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: -02-10T04:36: 2021-02-10T04:36:13.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: 12.123989+0000 osd.4 (osd. 2021-02-10T04:36:13.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: 4) 829 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:13.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: 2021-02-10T04:36:13.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:13.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: 02-10T04:36:12 2021-02-10T04:36:13.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: .123992+0000 osd.4 (osd.4 2021-02-10T04:36:13.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: ) 830 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:13.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:13.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: -02-10T04:36: 2021-02-10T04:36:13.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: 13.168473+0000 mon.a (mon. 2021-02-10T04:36:13.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: 0) 1391 : cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 402 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:36:13.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[35961]: 2021-02-10T04:36:13.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: cluster 2021-02-10T04:36:13.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: 2021-02-10T04:36:11.983748+0000 2021-02-10T04:36:13.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: osd.2 (osd.2) 1571 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:13.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: cluster 2021-02-10T04:36:11.983758+0000 osd.2 (osd.2) 1572 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:13.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:13.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: -02 2021-02-10T04:36:13.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: -10T 2021-02-10T04:36:13.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: 04:36:11.983764+0000 osd.2 2021-02-10T04:36:13.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: (osd 2021-02-10T04:36:13.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: .2) 1573 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:13.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:13.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: -10T04:36:11.983771+0000 osd.2 (osd.2 2021-02-10T04:36:13.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: ) 1574 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:13.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:13.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: 02-10T04:36:11.983776+0000 2021-02-10T04:36:13.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: osd.2 (osd.2) 1575 : cluster 2021-02-10T04:36:13.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: [WRN] 2021-02-10T04:36:13.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:13.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:36:13.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: 10T04:36:11.983782+0000 osd.2 (osd.2 2021-02-10T04:36:13.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: ) 1576 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:13.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:13.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: -02-10T04:36:11.983786+0000 2021-02-10T04:36:13.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: osd.2 (osd.2) 1577 : 2021-02-10T04:36:13.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:13.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:36:13.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: 36:11.983789+0000 osd.2 (osd.2) 1578 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:13.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: cluster 2021-02-10T04:36 2021-02-10T04:36:13.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: :11.983794+0000 osd.2 (osd.2) 1579 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:13.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: cluster 2021-02-10T04:36:13.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: 2021-02-10T04:36:11.983799 2021-02-10T04:36:13.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1580 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:13.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:13.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: 02-10T04:36:12.123976+0000 2021-02-10T04:36:13.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: osd.4 (osd.4) 827 : cluster 2021-02-10T04:36:13.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:13.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:36:13.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: 04:36:12.123984+0000 osd.4 (osd.4) 2021-02-10T04:36:13.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: 828 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:13.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: cluster 2021-02-10T04:36:13.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: 2021-02-10T04:36:12.123989 2021-02-10T04:36:13.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 829 : cluster [WRN] 2021-02-10T04:36:13.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:13.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: cluster 2021-02-10T04:36:12 2021-02-10T04:36:13.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: .123992+0000 osd.4 (osd.4) 830 : cluster 2021-02-10T04:36:13.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:13.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:36:13.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: 04:36:13.168473+0000 mon.a (mon.0 2021-02-10T04:36:13.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[29319]: ) 1391 : cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 402 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:36:14.290 INFO:tasks.rados.rados.0.smithi085.stdout:2129: done (11 left) 2021-02-10T04:36:14.290 INFO:tasks.rados.rados.0.smithi085.stdout:2130: read oid 36 snap -1 2021-02-10T04:36:14.290 INFO:tasks.rados.rados.0.smithi085.stdout:2130: expect (ObjNum 720 snap 226 seq_num 720) 2021-02-10T04:36:14.290 INFO:tasks.rados.rados.0.smithi085.stdout:2131: write oid 3 current snap is 240 2021-02-10T04:36:14.290 INFO:tasks.rados.rados.0.smithi085.stdout:2131: seq_num 747 ranges {554829=413597,1396049=705252} 2021-02-10T04:36:14.291 INFO:tasks.rados.rados.0.smithi085.stdout:2131: writing smithi08575397-3 from 554829 to 968426 tid 1 2021-02-10T04:36:14.298 INFO:tasks.rados.rados.0.smithi085.stdout:2131: writing smithi08575397-3 from 1396049 to 2101301 tid 2 2021-02-10T04:36:14.305 INFO:tasks.rados.rados.0.smithi085.stdout:2126: finishing write tid 1 to smithi08575397-5 2021-02-10T04:36:14.306 INFO:tasks.rados.rados.0.smithi085.stdout:2126: finishing write tid 2 to smithi08575397-5 2021-02-10T04:36:14.306 INFO:tasks.rados.rados.0.smithi085.stdout:2126: finishing write tid 3 to smithi08575397-5 2021-02-10T04:36:14.306 INFO:tasks.rados.rados.0.smithi085.stdout:2126: finishing write tid 4 to smithi08575397-5 2021-02-10T04:36:14.306 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 751 (ObjNum 745 snap 240 seq_num 745) dirty exists 2021-02-10T04:36:14.306 INFO:tasks.rados.rados.0.smithi085.stdout:2126: left oid 5 (ObjNum 745 snap 240 seq_num 745) 2021-02-10T04:36:14.307 INFO:tasks.rados.rados.0.smithi085.stdout:2127: finishing write tid 1 to smithi08575397-23 2021-02-10T04:36:14.307 INFO:tasks.rados.rados.0.smithi085.stdout:2127: finishing write tid 2 to smithi08575397-23 2021-02-10T04:36:14.307 INFO:tasks.rados.rados.0.smithi085.stdout:2127: finishing write tid 3 to smithi08575397-23 2021-02-10T04:36:14.307 INFO:tasks.rados.rados.0.smithi085.stdout:2127: finishing write tid 4 to smithi08575397-23 2021-02-10T04:36:14.307 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 683 (ObjNum 746 snap 240 seq_num 746) dirty exists 2021-02-10T04:36:14.307 INFO:tasks.rados.rados.0.smithi085.stdout:2127: left oid 23 (ObjNum 746 snap 240 seq_num 746) 2021-02-10T04:36:14.308 INFO:tasks.rados.rados.0.smithi085.stdout:2130: expect (ObjNum 720 snap 226 seq_num 720) 2021-02-10T04:36:14.312 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:13 smithi085 conmon[49106]: debug 2021-02-10T04:36:13.940+0000 7f77a3e94700 -1 osd.2 556 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:14.313 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: cluster 2021-02-10T04:36:12.936276+0000 osd.2 ( 2021-02-10T04:36:14.313 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: osd.2) 1581 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:14.313 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:36:14.313 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: T04:36:12.936285+0000 osd.2 (osd.2) 1582 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:14.313 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: cluster 2021-02-10T04:36:12.936301+0000 2021-02-10T04:36:14.314 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: osd.2 (osd.2) 1583 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:14.314 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:36:14.314 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: :36:12.936336+0000 osd.2 (osd.2) 1584 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:14.314 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: cluster 2021-02-10T04:36: 2021-02-10T04:36:14.314 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: 12.936343+0000 osd.2 (osd.2) 1585 : cluster 2021-02-10T04:36:14.314 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:14.314 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: cluster 2021-02-10T04:36 2021-02-10T04:36:14.314 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: :12.936349+0000 osd.2 (osd.2) 1586 : 2021-02-10T04:36:14.314 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:14.314 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: cluster 2021-02-10T04:36:12 2021-02-10T04:36:14.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: .936355+0000 osd.2 (osd.2) 1587 2021-02-10T04:36:14.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:14.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: cluster 2021-02-10T04:36: 2021-02-10T04:36:14.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: 12.936361+0000 osd.2 (osd.2) 1588 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:14.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:14.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: -02-10T04:36:12.936367+0000 osd.2 (osd.2) 1589 2021-02-10T04:36:14.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:14.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:36:14.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: T04:36:12.936373+0000 osd.2 (osd 2021-02-10T04:36:14.315 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: .2) 1590 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:14.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:14.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: 02-10T04:36:13.135665+0000 2021-02-10T04:36:14.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: osd.4 (osd.4) 831 : cluster [WRN] 2021-02-10T04:36:14.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:14.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: cluster 2021-02-10T04:36:13 2021-02-10T04:36:14.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: .135677+0000 osd.4 (osd.4) 832 : 2021-02-10T04:36:14.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:14.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:36:14.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: :36:13.135686+0000 osd.4 (osd. 2021-02-10T04:36:14.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: 4) 833 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:14.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:14.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: -10T04:36:13.135693+0000 2021-02-10T04:36:14.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: osd.4 (osd.4) 834 : cluster [WRN] 2021-02-10T04:36:14.317 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:14.317 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: cluster 2021-02-10T04:36:13.281163+0000 mon.a (mon.0) 1392 : cluster [DBG] osdmap e556: 8 total, 8 up, 8 in 2021-02-10T04:36:14.317 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:36:14.317 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: :36:14.115213+0000 mgr.y (mgr. 2021-02-10T04:36:14.317 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: 14140) 625 : cluster [DBG] pgmap v984: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 426 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 1.2 MiB/s wr, 8 op/s 2021-02-10T04:36:14.317 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:14.317 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: 02-10T04:36:14.283472+0000 2021-02-10T04:36:14.317 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: mon.a (mon.0) 1393 : cluster [DBG] 2021-02-10T04:36:14.317 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[29319]: osdmap e557: 8 total, 8 up, 8 in 2021-02-10T04:36:14.318 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: cluster 2021-02-10T04:36:12.936276 2021-02-10T04:36:14.318 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1581 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:14.318 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: cluster 2021-02-10T04:36:14.318 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:36:14.318 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: :36:12.936285+0000 osd.2 (osd 2021-02-10T04:36:14.318 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: .2) 1582 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:14.318 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: cluster 2021-02-10T04:36:14.318 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: 2021-02-10T04:36:12. 2021-02-10T04:36:14.319 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: 936301+0000 osd.2 (osd.2) 1583 : cluster 2021-02-10T04:36:14.319 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:14.319 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:36:14.319 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: 04:36:12.936336+0000 osd.2 (osd 2021-02-10T04:36:14.319 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: .2) 1584 : cluster [WRN] 2021-02-10T04:36:14.319 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:14.319 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:36:14.319 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: 10T04:36:12.936343+0000 osd.2 ( 2021-02-10T04:36:14.319 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: osd.2) 1585 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:14.320 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:14.320 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: -10T04:36:12.936349+0000 osd.2 ( 2021-02-10T04:36:14.320 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: osd.2) 1586 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:14.320 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: cluster 2021-02-10T04:36:14.320 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:36:14.320 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: 10T04:36:12.936355+0000 osd.2 2021-02-10T04:36:14.320 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: (osd.2) 1587 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:14.320 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: 2021-02-10T04:36:14.320 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:14.321 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: T04:36:12.936361+0000 osd.2 ( 2021-02-10T04:36:14.321 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: osd.2) 1588 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:14.321 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:36:14.321 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: 10T04:36:12.936367+0000 osd.2 ( 2021-02-10T04:36:14.321 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: osd.2) 1589 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:14.321 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: cluster 2021-02-10T04:36:14.321 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: 2021-02-10T04:36:12. 2021-02-10T04:36:14.321 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: 936373+0000 osd.2 (osd.2) 1590 : cluster [WRN] 2021-02-10T04:36:14.321 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:14.321 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: cluster 2021-02-10T04:36:13 2021-02-10T04:36:14.321 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: .135665+0000 osd.4 (osd.4) 831 : cluster 2021-02-10T04:36:14.321 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:14.322 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: cluster 2021-02-10T04:36:13 2021-02-10T04:36:14.322 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: .135677+0000 osd.4 (osd.4) 832 : cluster 2021-02-10T04:36:14.322 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:14.322 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: cluster 2021-02-10T04:36:13 2021-02-10T04:36:14.322 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: .135686+0000 osd.4 (osd.4) 833 : cluster 2021-02-10T04:36:14.322 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:14.322 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: cluster 2021-02-10T04:36:13 2021-02-10T04:36:14.322 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: .135693+0000 osd.4 (osd.4) 834 : cluster 2021-02-10T04:36:14.322 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:14.323 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:36:14.323 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: 36:13.281163+0000 mon.a (mon.0) 2021-02-10T04:36:14.323 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: 1392 : cluster [DBG] osdmap e556: 8 total, 8 up, 8 in 2021-02-10T04:36:14.323 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:36:14.323 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: 04:36:14.115213+0000 mgr.y (mgr. 2021-02-10T04:36:14.323 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: 14140) 625 : cluster [DBG] pgmap v984: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 426 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 1.2 MiB/s wr, 8 op/s 2021-02-10T04:36:14.323 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:14.323 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: T04:36:14.283472+0000 mon.a (mon 2021-02-10T04:36:14.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[35961]: .0) 1393 : cluster [DBG] osdmap e557: 8 total, 8 up, 8 in 2021-02-10T04:36:14.438 INFO:tasks.rados.rados.0.smithi085.stdout:2131: finishing write tid 1 to smithi08575397-3 2021-02-10T04:36:14.438 INFO:tasks.rados.rados.0.smithi085.stdout:2131: finishing write tid 2 to smithi08575397-3 2021-02-10T04:36:14.438 INFO:tasks.rados.rados.0.smithi085.stdout:2131: finishing write tid 3 to smithi08575397-3 2021-02-10T04:36:14.438 INFO:tasks.rados.rados.0.smithi085.stdout:2126: done (12 left) 2021-02-10T04:36:14.438 INFO:tasks.rados.rados.0.smithi085.stdout:2127: done (11 left) 2021-02-10T04:36:14.438 INFO:tasks.rados.rados.0.smithi085.stdout:2130: done (10 left) 2021-02-10T04:36:14.439 INFO:tasks.rados.rados.0.smithi085.stdout:2132: copy_from oid 36 from oid 10 current snap is 240 2021-02-10T04:36:14.439 INFO:tasks.rados.rados.0.smithi085.stdout:2133: write oid 43 current snap is 240 2021-02-10T04:36:14.439 INFO:tasks.rados.rados.0.smithi085.stdout:2133: seq_num 748 ranges {571636=488846,1854531=565263} 2021-02-10T04:36:14.443 INFO:tasks.rados.rados.0.smithi085.stdout:2133: writing smithi08575397-43 from 571636 to 1060482 tid 1 2021-02-10T04:36:14.452 INFO:tasks.rados.rados.0.smithi085.stdout:2133: writing smithi08575397-43 from 1854531 to 2419794 tid 2 2021-02-10T04:36:14.452 INFO:tasks.rados.rados.0.smithi085.stdout:2134: rollback oid 20 current snap is 240 2021-02-10T04:36:14.452 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 20 to 221 2021-02-10T04:36:14.452 INFO:tasks.rados.rados.0.smithi085.stdout:2135: write oid 1 current snap is 240 2021-02-10T04:36:14.452 INFO:tasks.rados.rados.0.smithi085.stdout:2135: seq_num 749 ranges {588443=564095,1796660=708921} 2021-02-10T04:36:14.459 INFO:tasks.rados.rados.0.smithi085.stdout:2135: writing smithi08575397-1 from 588443 to 1152538 tid 1 2021-02-10T04:36:14.468 INFO:tasks.rados.rados.0.smithi085.stdout:2135: writing smithi08575397-1 from 1796660 to 2505581 tid 2 2021-02-10T04:36:14.469 INFO:tasks.rados.rados.0.smithi085.stdout:2136: copy_from oid 26 from oid 13 current snap is 240 2021-02-10T04:36:14.469 INFO:tasks.rados.rados.0.smithi085.stdout:2131: finishing write tid 4 to smithi08575397-3 2021-02-10T04:36:14.469 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 773 (ObjNum 747 snap 240 seq_num 747) dirty exists 2021-02-10T04:36:14.469 INFO:tasks.rados.rados.0.smithi085.stdout:2131: left oid 3 (ObjNum 747 snap 240 seq_num 747) 2021-02-10T04:36:14.469 INFO:tasks.rados.rados.0.smithi085.stdout:2132: finishing copy_from to smithi08575397-36 2021-02-10T04:36:14.469 INFO:tasks.rados.rados.0.smithi085.stdout:2132: got expected ENOENT (src dne) 2021-02-10T04:36:14.469 INFO:tasks.rados.rados.0.smithi085.stdout:2132: finishing copy_from racing read to smithi08575397-36 2021-02-10T04:36:14.469 INFO:tasks.rados.rados.0.smithi085.stdout:2133: finishing write tid 1 to smithi08575397-43 2021-02-10T04:36:14.469 INFO:tasks.rados.rados.0.smithi085.stdout:2134: finishing rollback tid 0 to smithi08575397-20 2021-02-10T04:36:14.469 INFO:tasks.rados.rados.0.smithi085.stdout:2134: finishing rollback tid 1 to smithi08575397-20 2021-02-10T04:36:14.469 INFO:tasks.rados.rados.0.smithi085.stdout:2133: finishing write tid 2 to smithi08575397-43 2021-02-10T04:36:14.470 INFO:tasks.rados.rados.0.smithi085.stdout:2134: finishing rollback tid 2 to smithi08575397-20 2021-02-10T04:36:14.470 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 20 v 1080 (ObjNum 705 snap 221 seq_num 705) dirty exists 2021-02-10T04:36:14.470 INFO:tasks.rados.rados.0.smithi085.stdout:2133: finishing write tid 3 to smithi08575397-43 2021-02-10T04:36:14.470 INFO:tasks.rados.rados.0.smithi085.stdout:2131: done (14 left) 2021-02-10T04:36:14.470 INFO:tasks.rados.rados.0.smithi085.stdout:2132: done (13 left) 2021-02-10T04:36:14.470 INFO:tasks.rados.rados.0.smithi085.stdout:2134: done (12 left) 2021-02-10T04:36:14.470 INFO:tasks.rados.rados.0.smithi085.stdout:2137: snap_create 2021-02-10T04:36:14.470 INFO:tasks.rados.rados.0.smithi085.stdout:2133: finishing write tid 4 to smithi08575397-43 2021-02-10T04:36:14.470 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 777 (ObjNum 748 snap 240 seq_num 748) dirty exists 2021-02-10T04:36:14.470 INFO:tasks.rados.rados.0.smithi085.stdout:2133: left oid 43 (ObjNum 748 snap 240 seq_num 748) 2021-02-10T04:36:14.470 INFO:tasks.rados.rados.0.smithi085.stdout:2135: finishing write tid 1 to smithi08575397-1 2021-02-10T04:36:14.473 INFO:tasks.rados.rados.0.smithi085.stdout:2135: finishing write tid 2 to smithi08575397-1 2021-02-10T04:36:14.473 INFO:tasks.rados.rados.0.smithi085.stdout:2135: finishing write tid 3 to smithi08575397-1 2021-02-10T04:36:14.473 INFO:tasks.rados.rados.0.smithi085.stdout:2135: finishing write tid 4 to smithi08575397-1 2021-02-10T04:36:14.473 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 1110 (ObjNum 749 snap 240 seq_num 749) dirty exists 2021-02-10T04:36:14.474 INFO:tasks.rados.rados.0.smithi085.stdout:2135: left oid 1 (ObjNum 749 snap 240 seq_num 749) 2021-02-10T04:36:14.481 INFO:tasks.rados.rados.0.smithi085.stdout:2136: finishing copy_from to smithi08575397-26 2021-02-10T04:36:14.481 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 748 (ObjNum 742 snap 238 seq_num 742) dirty exists 2021-02-10T04:36:14.482 INFO:tasks.rados.rados.0.smithi085.stdout:2136: finishing copy_from racing read to smithi08575397-26 2021-02-10T04:36:14.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: cluster 2021-02-10T04:36:12.936276+0000 osd.2 (osd.2) 1581 2021-02-10T04:36:14.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:14.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: cluster 2021-02-10T04:36:12.936285+0000 osd.2 (osd.2) 1582 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:14.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:14.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: 02-10T04:36:12.936301+0000 osd.2 (osd.2) 1583 : cluster 2021-02-10T04:36:14.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:14.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:14.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: -02-10T04:36:12. 2021-02-10T04:36:14.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: 936336+0000 osd.2 (osd.2) 2021-02-10T04:36:14.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: 1584 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:14.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:14.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: -02-10T04:36:12. 2021-02-10T04:36:14.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: 936343+0000 osd.2 (osd.2) 2021-02-10T04:36:14.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: 1585 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:14.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:14.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: -02-10T04:36:12 2021-02-10T04:36:14.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: .936349+0000 osd.2 (osd. 2021-02-10T04:36:14.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: 2) 1586 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:14.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: 2021-02-10T04:36:14.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:14.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: 02-10T04:36:12. 2021-02-10T04:36:14.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: 936355+0000 osd.2 (osd.2) 2021-02-10T04:36:14.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: 1587 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:14.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:14.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: -02-10T04:36:12 2021-02-10T04:36:14.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: .936361+0000 osd.2 (osd.2 2021-02-10T04:36:14.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: ) 1588 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:14.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:14.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: -02-10T04:36:12. 2021-02-10T04:36:14.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: 936367+0000 osd.2 (osd.2) 2021-02-10T04:36:14.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: 1589 : cluster [WRN] 2021-02-10T04:36:14.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:14.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: cluster 2021-02-10T04:36:14.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:36:14.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: 36:12.936373+0000 osd.2 (osd.2) 1590 : cluster 2021-02-10T04:36:14.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:14.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: cluster 2021-02-10T04:36:14.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: 2021-02-10T04:36 2021-02-10T04:36:14.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: :13.135665+0000 osd.4 (osd. 2021-02-10T04:36:14.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: 4) 831 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:14.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: 2021-02-10T04:36:14.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:36:14.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: 04:36:13.135677+0000 osd.4 (osd.4 2021-02-10T04:36:14.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: ) 832 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:14.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:14.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: -10T04:36:13.135686+0000 osd.4 (osd.4) 2021-02-10T04:36:14.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: 833 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:14.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:14.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: 02-10T04:36:13.135693+0000 osd.4 2021-02-10T04:36:14.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: (osd.4) 834 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:14.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: 2021-02-10T04:36:14.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:14.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: -10T04:36:13.281163+0000 mon.a (mon 2021-02-10T04:36:14.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: .0) 1392 : cluster [DBG] osdmap e556: 8 total, 8 up, 8 in 2021-02-10T04:36:14.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: cluster 2021-02-10T04:36:14.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:36:14.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: 04:36:14.115213+0000 mgr.y (mgr.14140 2021-02-10T04:36:14.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: ) 625 : cluster [DBG] pgmap v984: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 426 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 1.2 MiB/s wr, 8 op/s 2021-02-10T04:36:14.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:14.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: -02-10T04:36:14 2021-02-10T04:36:14.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: .283472+0000 mon.a (mon.0) 1393 : cluster 2021-02-10T04:36:14.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[29288]: [DBG] osdmap e557: 8 total, 8 up, 8 in 2021-02-10T04:36:14.649 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:14 smithi164 conmon[33677]: debug 2021-02-10T04:36:14.154+0000 7fbec19a1700 -1 osd.4 556 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:15.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:14 smithi085 conmon[49106]: debug 2021-02-10T04:36:14.947+0000 7f77a3e94700 -1 osd.2 557 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:15.323 INFO:tasks.rados.rados.0.smithi085.stdout:2133: done (12 left) 2021-02-10T04:36:15.323 INFO:tasks.rados.rados.0.smithi085.stdout:2135: done (11 left) 2021-02-10T04:36:15.323 INFO:tasks.rados.rados.0.smithi085.stdout:2136: done (10 left) 2021-02-10T04:36:15.323 INFO:tasks.rados.rados.0.smithi085.stdout:2137: done (9 left) 2021-02-10T04:36:15.324 INFO:tasks.rados.rados.0.smithi085.stdout:2138: delete oid 36 current snap is 241 2021-02-10T04:36:15.335 INFO:tasks.rados.rados.0.smithi085.stdout:2138: done (9 left) 2021-02-10T04:36:15.335 INFO:tasks.rados.rados.0.smithi085.stdout:2139: snap_remove snap 240 2021-02-10T04:36:15.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[33677]: debug 2021-02-10T04:36:15.185+0000 7fbec19a1700 -1 osd.4 557 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: cluster 2021-02-10T04:36:13.942243+0000 2021-02-10T04:36:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: osd.2 (osd.2) 1591 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: cluster 2021-02-10T04:36: 2021-02-10T04:36:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: 13.942252+0000 osd.2 (osd.2) 1592 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: 02-10T04:36:13.942260+0000 2021-02-10T04:36:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: osd.2 (osd.2) 1593 : cluster [WRN] 2021-02-10T04:36:15.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:15.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:36:15.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: 36:13.942266+0000 osd.2 (osd.2 2021-02-10T04:36:15.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: ) 1594 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:15.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: cluster 2021-02-10T04:36:15.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: 2021-02-10T04:36: 2021-02-10T04:36:15.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: 13.942273+0000 osd.2 (osd.2) 1595 : 2021-02-10T04:36:15.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:15.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:15.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: 02-10T04:36:13.942280+0000 2021-02-10T04:36:15.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: osd.2 (osd.2) 1596 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:15.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: 2021-02-10T04:36:15.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: cluster 2021-02-10T04:36 2021-02-10T04:36:15.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: :13.942285+0000 osd.2 (osd.2) 2021-02-10T04:36:15.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: 1597 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:15.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:15.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: -10T04:36:13.942291+0000 osd.2 ( 2021-02-10T04:36:15.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: osd.2) 1598 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:15.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: cluster 2021-02-10T04:36:15.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: 2021-02-10T04:36:13.942297+0000 2021-02-10T04:36:15.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: osd.2 (osd.2) 1599 : cluster [WRN] 2021-02-10T04:36:15.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:15.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:36:15.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: 36:13.942302+0000 osd.2 (osd.2 2021-02-10T04:36:15.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: ) 1600 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:15.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:15.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: 02-10T04:36:14.155267 2021-02-10T04:36:15.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 835 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:15.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:15.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: 02-10T04:36:14.155274+0000 osd.4 2021-02-10T04:36:15.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: (osd.4) 836 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:15.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:15.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: -10T04:36:14.155278+0000 osd.4 2021-02-10T04:36:15.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: (osd.4) 837 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:15.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: 2021-02-10T04:36:15.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:15.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: 02-10T04:36:14.155286+0000 osd.4 (osd.4 2021-02-10T04:36:15.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:15 smithi164 conmon[29288]: ) 838 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:15.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: cluster 2021-02-10T04:36:13.942243+0000 2021-02-10T04:36:15.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: osd.2 (osd.2) 1591 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:15.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: cluster 2021-02-10T04:36 2021-02-10T04:36:15.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: :13.942252+0000 osd.2 (osd.2) 1592 : cluster 2021-02-10T04:36:15.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:15.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: cluster 2021-02-10T04:36:15.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: 2021-02-10T04:36 2021-02-10T04:36:15.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: :13.942260+0000 osd.2 2021-02-10T04:36:15.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: (osd.2) 1593 : cluster 2021-02-10T04:36:15.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:15.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: cluster 2021-02-10T04:36:15.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: 2021-02-10T04:36 2021-02-10T04:36:15.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: :13.942266+0000 osd.2 (osd. 2021-02-10T04:36:15.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: 2) 1594 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:15.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: 2021-02-10T04:36:15.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:36:15.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: T04:36:13.942273+0000 2021-02-10T04:36:15.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: osd.2 (osd.2) 1595 : cluster 2021-02-10T04:36:15.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:15.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:15.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: -10T04:36:13.942280+0000 osd.2 (osd 2021-02-10T04:36:15.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: .2) 1596 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:15.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: cluster 2021-02-10T04:36:13. 2021-02-10T04:36:15.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: 942285+0000 osd.2 (osd.2) 1597 : cluster [WRN] 2021-02-10T04:36:15.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:15.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:36:15.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: 36:13.942291+0000 osd.2 (osd.2) 2021-02-10T04:36:15.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: 1598 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:15.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: cluster 2021-02-10T04:36 2021-02-10T04:36:15.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: :13.942297+0000 osd.2 (osd.2) 1599 : 2021-02-10T04:36:15.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:15.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: cluster 2021-02-10T04:36: 2021-02-10T04:36:15.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: 13.942302+0000 osd.2 (osd.2) 1600 : cluster 2021-02-10T04:36:15.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:15.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: cluster 2021-02-10T04:36:14.155267 2021-02-10T04:36:15.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 835 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:15.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: 2021-02-10T04:36:15.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: cluster 2021-02-10T04:36:15.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: 2021-02-10T04:36:14.155274+0000 2021-02-10T04:36:15.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: osd.4 (osd.4) 836 : cluster [WRN] 2021-02-10T04:36:15.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:15.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: cluster 2021-02-10T04:36: 2021-02-10T04:36:15.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: 14.155278+0000 osd.4 (osd.4) 837 : 2021-02-10T04:36:15.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:15.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:36:15.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: 36:14.155286+0000 osd.4 (osd.4 2021-02-10T04:36:15.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[29319]: ) 838 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:15.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: cluster 2021-02-10T04:36:15.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2021-02-10T04:36:15.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2021 2021-02-10T04:36:15.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: - 2021-02-10T04:36:15.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 02 2021-02-10T04:36:15.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: - 2021-02-10T04:36:15.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 10 2021-02-10T04:36:15.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: T 2021-02-10T04:36:15.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 04 2021-02-10T04:36:15.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: : 2021-02-10T04:36:15.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 36 2021-02-10T04:36:15.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: : 2021-02-10T04:36:15.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 13 2021-02-10T04:36:15.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: . 2021-02-10T04:36:15.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 942243 2021-02-10T04:36:15.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: +0000 2021-02-10T04:36:15.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2021-02-10T04:36:15.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: osd.2 2021-02-10T04:36:15.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: ( 2021-02-10T04:36:15.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: osd 2021-02-10T04:36:15.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: . 2021-02-10T04:36:15.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2 2021-02-10T04:36:15.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: ) 2021-02-10T04:36:15.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 1591 2021-02-10T04:36:15.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: : 2021-02-10T04:36:15.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: cluster 2021-02-10T04:36:15.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2021-02-10T04:36:15.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: [WRN] 2021-02-10T04:36:15.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2021-02-10T04:36:15.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:15.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2021-02-10T04:36:15.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: cluster 2021-02-10T04:36:15.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2021-02-10T04:36:15.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2021 2021-02-10T04:36:15.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: - 2021-02-10T04:36:15.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 02 2021-02-10T04:36:15.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: - 2021-02-10T04:36:15.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 10 2021-02-10T04:36:15.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: T 2021-02-10T04:36:15.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 04 2021-02-10T04:36:15.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: : 2021-02-10T04:36:15.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 36 2021-02-10T04:36:15.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: : 2021-02-10T04:36:15.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 13 2021-02-10T04:36:15.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: . 2021-02-10T04:36:15.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 942252 2021-02-10T04:36:15.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: +0000 2021-02-10T04:36:15.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2021-02-10T04:36:15.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: osd.2 2021-02-10T04:36:15.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: ( 2021-02-10T04:36:15.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: osd 2021-02-10T04:36:15.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: . 2021-02-10T04:36:15.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2 2021-02-10T04:36:15.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: ) 2021-02-10T04:36:15.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 1592 2021-02-10T04:36:15.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: : 2021-02-10T04:36:15.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: cluster 2021-02-10T04:36:15.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2021-02-10T04:36:15.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: [WRN] 2021-02-10T04:36:15.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2021-02-10T04:36:15.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:15.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2021-02-10T04:36:15.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: cluster 2021-02-10T04:36:15.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2021-02-10T04:36:15.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2021 2021-02-10T04:36:15.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: - 2021-02-10T04:36:15.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 02 2021-02-10T04:36:15.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: - 2021-02-10T04:36:15.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 10 2021-02-10T04:36:15.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: T 2021-02-10T04:36:15.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 04 2021-02-10T04:36:15.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: : 2021-02-10T04:36:15.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 36 2021-02-10T04:36:15.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: : 2021-02-10T04:36:15.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 13 2021-02-10T04:36:15.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: . 2021-02-10T04:36:15.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 942260 2021-02-10T04:36:15.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: +0000 2021-02-10T04:36:15.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2021-02-10T04:36:15.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: osd.2 2021-02-10T04:36:15.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: ( 2021-02-10T04:36:15.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: osd 2021-02-10T04:36:15.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: . 2021-02-10T04:36:15.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2 2021-02-10T04:36:15.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: ) 2021-02-10T04:36:15.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 1593 2021-02-10T04:36:15.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: : 2021-02-10T04:36:15.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: cluster 2021-02-10T04:36:15.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2021-02-10T04:36:15.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: [WRN] 2021-02-10T04:36:15.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2021-02-10T04:36:15.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:15.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2021-02-10T04:36:15.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: cluster 2021-02-10T04:36:15.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2021-02-10T04:36:15.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2021 2021-02-10T04:36:15.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: - 2021-02-10T04:36:15.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 02 2021-02-10T04:36:15.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: - 2021-02-10T04:36:15.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 10 2021-02-10T04:36:15.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: T 2021-02-10T04:36:15.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 04 2021-02-10T04:36:15.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: : 2021-02-10T04:36:15.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 36 2021-02-10T04:36:15.710 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: : 2021-02-10T04:36:15.710 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 13 2021-02-10T04:36:15.710 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: . 2021-02-10T04:36:15.710 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 942266 2021-02-10T04:36:15.710 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: +0000 2021-02-10T04:36:15.710 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2021-02-10T04:36:15.710 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: osd.2 2021-02-10T04:36:15.710 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: ( 2021-02-10T04:36:15.710 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: osd 2021-02-10T04:36:15.710 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: . 2021-02-10T04:36:15.710 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2 2021-02-10T04:36:15.710 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: ) 2021-02-10T04:36:15.710 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 1594 2021-02-10T04:36:15.711 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: : 2021-02-10T04:36:15.711 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: cluster 2021-02-10T04:36:15.711 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2021-02-10T04:36:15.711 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: [WRN] 2021-02-10T04:36:15.711 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2021-02-10T04:36:15.711 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:15.711 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2021-02-10T04:36:15.711 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: cluster 2021-02-10T04:36:15.711 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2021-02-10T04:36:15.711 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2021 2021-02-10T04:36:15.711 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: -02-10T04:36:13 2021-02-10T04:36:15.711 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: .942273+0000 osd.2 (osd.2) 1595 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:15.711 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: cluster 2021-02-10T04:36:15.712 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2021-02-10T04:36:15.712 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2021 2021-02-10T04:36:15.712 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: - 2021-02-10T04:36:15.712 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 02 2021-02-10T04:36:15.712 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: - 2021-02-10T04:36:15.712 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 10 2021-02-10T04:36:15.712 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: T 2021-02-10T04:36:15.712 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 04 2021-02-10T04:36:15.712 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: : 2021-02-10T04:36:15.712 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 36 2021-02-10T04:36:15.712 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: : 2021-02-10T04:36:15.712 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 13 2021-02-10T04:36:15.712 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: . 2021-02-10T04:36:15.713 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 942280+0000 osd.2 (osd.2) 1596 : 2021-02-10T04:36:15.713 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:15.713 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: cluster 2021-02-10T04:36:13.942285+0000 osd.2 ( 2021-02-10T04:36:15.713 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: osd.2) 1597 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:15.713 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: cluster 2021-02-10T04:36 2021-02-10T04:36:15.713 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: :13.942291+0000 osd.2 (osd.2) 2021-02-10T04:36:15.713 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 1598 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:15.713 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:15.713 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 02-10T04:36:13 2021-02-10T04:36:15.713 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: .942297+0000 osd.2 (osd.2 2021-02-10T04:36:15.713 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: ) 1599 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:15.713 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: cluster 2021-02-10T04:36:15.713 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:36:15.714 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 36:13.942302+0000 osd.2 (osd. 2021-02-10T04:36:15.714 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2) 1600 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:15.714 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 2021-02-10T04:36:15.714 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:15.714 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: T04:36:14.155267+0000 osd.4 (osd. 2021-02-10T04:36:15.714 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 4) 835 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:15.714 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:36:15.714 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 36:14.155274+0000 osd.4 (osd.4) 836 : cluster 2021-02-10T04:36:15.714 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:15.714 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:36:15.714 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: 10T04:36:14.155278+0000 osd.4 (osd.4) 837 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:15.714 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[35961]: cluster 2021-02-10T04:36:14.155286+0000 osd.4 (osd.4) 838 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:16.333 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[49106]: debug 2021-02-10T04:36:16.333 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:15 smithi085 conmon[49106]: 2021-02-10T04:36:15.981+0000 7f77a3e94700 -1 osd.2 558 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:16.335 INFO:tasks.rados.rados.0.smithi085.stdout:2139: done (9 left) 2021-02-10T04:36:16.335 INFO:tasks.rados.rados.0.smithi085.stdout:2140: snap_remove snap 213 2021-02-10T04:36:16.339 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: cluster 2021-02-10T04:36:14.947808+0000 2021-02-10T04:36:16.340 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: osd.2 (osd.2) 1601 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:16.340 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: cluster 2021-02-10T04:36:14.947814+0000 osd.2 (osd.2) 1602 : cluster [WRN] 2021-02-10T04:36:16.340 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:16.340 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: cluster 2021-02-10T04:36:14 2021-02-10T04:36:16.340 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: .947820+0000 osd.2 (osd.2) 1603 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:16.340 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:36:16.340 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: T04:36:14.947824+0000 2021-02-10T04:36:16.340 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: osd.2 (osd.2) 1604 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:16.340 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: cluster 2021-02-10T04:36:14.947828+0000 osd.2 (osd. 2021-02-10T04:36:16.340 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: 2) 1605 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:16.341 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: cluster 2021-02-10T04:36:16.341 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: 2021-02-10T04:36:14. 2021-02-10T04:36:16.341 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: 947832+0000 osd.2 (osd.2 2021-02-10T04:36:16.341 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: ) 1606 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:16.341 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:16.341 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: 02-10T04:36:14.947836 2021-02-10T04:36:16.341 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 2021-02-10T04:36:16.341 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: 1607 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:16.341 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:16.341 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: -10T04:36:14.947840+0000 osd.2 (osd.2 2021-02-10T04:36:16.341 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: ) 1608 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:16.341 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: cluster 2021-02-10T04:36:16.341 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: 2021-02 2021-02-10T04:36:16.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: -10T04:36:14.947844 2021-02-10T04:36:16.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 2021-02-10T04:36:16.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: 1609 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:16.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: cluster 2021-02-10T04:36:16.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:36:16.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: 10T04:36:14.947864+0000 osd.2 (osd 2021-02-10T04:36:16.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: .2) 1610 : cluster [WRN] 2021-02-10T04:36:16.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:16.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:16.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: -02-10T04:36:15.185989+0000 2021-02-10T04:36:16.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: osd.4 (osd.4) 839 : cluster [WRN] 2021-02-10T04:36:16.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:16.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:36:16.343 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: 10T04:36:15.185997+0000 osd.4 2021-02-10T04:36:16.343 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: (osd.4) 840 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:16.343 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: 2021-02-10T04:36:16.343 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: cluster 2021-02-10T04:36: 2021-02-10T04:36:16.343 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: 15.186001+0000 osd.4 (osd.4) 841 : cluster 2021-02-10T04:36:16.343 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:16.343 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: cluster 2021-02-10T04:36:15 2021-02-10T04:36:16.343 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: .186006+0000 osd.4 (osd.4) 842 : cluster [WRN] 2021-02-10T04:36:16.343 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:16.343 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: cluster 2021-02-10T04:36:16.343 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: 2021-02-10T04:36:15.322441+0000 mon.a (mon.0) 2021-02-10T04:36:16.343 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: 1394 : cluster [DBG] osdmap e558: 8 total, 8 up, 8 in 2021-02-10T04:36:16.343 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:36:16.344 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: 36:16.115804+0000 mgr.y (mgr.14140 2021-02-10T04:36:16.344 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: ) 626 : cluster [DBG] pgmap v987: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 432 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 7.4 MiB/s rd, 2.6 MiB/s wr, 16 op/s 2021-02-10T04:36:16.344 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[29319]: 2021-02-10T04:36:16.344 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: cluster 2021-02-10T04:36:16.344 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2021-02-10T04:36:16.344 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2021 2021-02-10T04:36:16.344 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: - 2021-02-10T04:36:16.345 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 02 2021-02-10T04:36:16.345 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: - 2021-02-10T04:36:16.345 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 10 2021-02-10T04:36:16.345 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: T 2021-02-10T04:36:16.345 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 04 2021-02-10T04:36:16.345 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: : 2021-02-10T04:36:16.345 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 36 2021-02-10T04:36:16.345 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: : 2021-02-10T04:36:16.345 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 14 2021-02-10T04:36:16.345 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: . 2021-02-10T04:36:16.346 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 947808 2021-02-10T04:36:16.346 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: +0000 2021-02-10T04:36:16.346 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2021-02-10T04:36:16.346 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: osd.2 2021-02-10T04:36:16.346 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: ( 2021-02-10T04:36:16.346 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: osd 2021-02-10T04:36:16.346 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: . 2021-02-10T04:36:16.346 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2 2021-02-10T04:36:16.346 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: ) 2021-02-10T04:36:16.346 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 1601 2021-02-10T04:36:16.346 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: : 2021-02-10T04:36:16.346 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: cluster 2021-02-10T04:36:16.346 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2021-02-10T04:36:16.346 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: [WRN] 2021-02-10T04:36:16.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2021-02-10T04:36:16.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:16.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2021-02-10T04:36:16.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: cluster 2021-02-10T04:36:16.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2021-02-10T04:36:16.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2021 2021-02-10T04:36:16.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: - 2021-02-10T04:36:16.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 02 2021-02-10T04:36:16.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: - 2021-02-10T04:36:16.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 10 2021-02-10T04:36:16.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: T 2021-02-10T04:36:16.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 04 2021-02-10T04:36:16.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: : 2021-02-10T04:36:16.348 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 36 2021-02-10T04:36:16.348 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: : 2021-02-10T04:36:16.348 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 14 2021-02-10T04:36:16.348 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: . 2021-02-10T04:36:16.348 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 947814 2021-02-10T04:36:16.348 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: +0000 2021-02-10T04:36:16.348 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2021-02-10T04:36:16.348 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: osd.2 2021-02-10T04:36:16.348 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: ( 2021-02-10T04:36:16.348 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: osd 2021-02-10T04:36:16.348 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: . 2021-02-10T04:36:16.348 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2 2021-02-10T04:36:16.348 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: ) 2021-02-10T04:36:16.348 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 1602 2021-02-10T04:36:16.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: : 2021-02-10T04:36:16.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: cluster 2021-02-10T04:36:16.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2021-02-10T04:36:16.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: [WRN] 2021-02-10T04:36:16.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2021-02-10T04:36:16.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:16.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2021-02-10T04:36:16.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: cluster 2021-02-10T04:36:16.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2021-02-10T04:36:16.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2021 2021-02-10T04:36:16.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: - 2021-02-10T04:36:16.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 02 2021-02-10T04:36:16.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: - 2021-02-10T04:36:16.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 10 2021-02-10T04:36:16.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: T 2021-02-10T04:36:16.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 04 2021-02-10T04:36:16.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: : 2021-02-10T04:36:16.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 36 2021-02-10T04:36:16.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: : 2021-02-10T04:36:16.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 14 2021-02-10T04:36:16.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: . 2021-02-10T04:36:16.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 947820 2021-02-10T04:36:16.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: +0000 2021-02-10T04:36:16.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2021-02-10T04:36:16.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: osd.2 2021-02-10T04:36:16.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: ( 2021-02-10T04:36:16.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: osd 2021-02-10T04:36:16.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: . 2021-02-10T04:36:16.351 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2 2021-02-10T04:36:16.351 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: ) 2021-02-10T04:36:16.351 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 1603 2021-02-10T04:36:16.607 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: cluster 2021-02-10T04:36:14.947808+0000 osd.2 (osd.2 2021-02-10T04:36:16.607 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: ) 1601 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:16.607 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: cluster 2021-02-10T04:36:14.947814+0000 osd.2 (osd.2) 1602 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:16.607 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:36:16.608 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: 04:36:14.947820+0000 osd.2 (osd.2) 1603 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:16.608 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:36:16.608 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: 36:14 2021-02-10T04:36:16.608 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: .947824+0000 osd.2 (osd. 2021-02-10T04:36:16.608 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: 2) 1604 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:16.608 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:16.608 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: -02-10T04:36:14. 2021-02-10T04:36:16.608 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: 947828+0000 osd.2 (osd.2) 1605 2021-02-10T04:36:16.608 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:16.608 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: cluster 2021-02-10T04:36:16.608 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:36:16.608 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: :36:14.947832+0000 osd.2 2021-02-10T04:36:16.609 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: (osd.2) 1606 : cluster 2021-02-10T04:36:16.609 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:16.609 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:16.609 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: -02-10T04:36:14.947836 2021-02-10T04:36:16.609 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1607 : 2021-02-10T04:36:16.609 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:16.609 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:16.609 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: 02-10T04:36:14.947840+0000 osd.2 (osd 2021-02-10T04:36:16.609 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: .2) 1608 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:16.609 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:16.609 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: -02-10T04:36:14.947844+0000 2021-02-10T04:36:16.609 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: osd.2 (osd.2) 1609 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:16.609 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: cluster 2021-02-10T04:36:16.610 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: 2021-02-10T04:36 2021-02-10T04:36:16.610 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: :14.947864+0000 osd.2 (osd 2021-02-10T04:36:16.610 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: .2) 1610 : cluster [WRN] 2021-02-10T04:36:16.610 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:16.610 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:36:16.610 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: 04:36:15.185989+0000 osd.4 (osd.4) 839 : cluster 2021-02-10T04:36:16.610 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:16.610 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: cluster 2021-02-10T04:36:16.610 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: 2021-02-10T04:36:15.185997+0000 osd.4 (osd. 2021-02-10T04:36:16.610 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: 4) 840 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:16.610 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: cluster 2021-02-10T04:36: 2021-02-10T04:36:16.610 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: 15.186001+0000 osd.4 (osd.4) 841 : cluster 2021-02-10T04:36:16.611 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:16.611 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: cluster 2021-02-10T04:36:15.186006+0000 osd.4 ( 2021-02-10T04:36:16.611 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: osd.4) 842 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:16.611 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: cluster 2021-02-10T04:36:15. 2021-02-10T04:36:16.611 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: 322441+0000 mon.a (mon.0) 1394 : cluster 2021-02-10T04:36:16.611 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: [DBG] osdmap e558: 8 total, 8 up, 8 in 2021-02-10T04:36:16.611 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:36:16.611 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: 36:16.115804+0000 mgr.y (mgr.14140 2021-02-10T04:36:16.611 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[29288]: ) 626 : cluster [DBG] pgmap v987: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 432 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 7.4 MiB/s rd, 2.6 MiB/s wr, 16 op/s 2021-02-10T04:36:16.611 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:16 smithi164 conmon[33677]: debug 2021-02-10T04:36:16.212+0000 7fbec19a1700 -1 osd.4 558 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:16.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: : 2021-02-10T04:36:16.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: cluster 2021-02-10T04:36:16.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2021-02-10T04:36:16.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: [WRN] 2021-02-10T04:36:16.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2021-02-10T04:36:16.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:16.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2021-02-10T04:36:16.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: cluster 2021-02-10T04:36:16.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2021-02-10T04:36:16.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2021 2021-02-10T04:36:16.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: - 2021-02-10T04:36:16.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 02 2021-02-10T04:36:16.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: - 2021-02-10T04:36:16.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 10 2021-02-10T04:36:16.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: T 2021-02-10T04:36:16.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 04 2021-02-10T04:36:16.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: : 2021-02-10T04:36:16.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 36 2021-02-10T04:36:16.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: : 2021-02-10T04:36:16.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 14 2021-02-10T04:36:16.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: . 2021-02-10T04:36:16.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 947824 2021-02-10T04:36:16.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: +0000 2021-02-10T04:36:16.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2021-02-10T04:36:16.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: osd.2 2021-02-10T04:36:16.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: ( 2021-02-10T04:36:16.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: osd 2021-02-10T04:36:16.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: . 2021-02-10T04:36:16.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2 2021-02-10T04:36:16.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: ) 2021-02-10T04:36:16.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 1604 2021-02-10T04:36:16.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: : 2021-02-10T04:36:16.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: cluster 2021-02-10T04:36:16.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2021-02-10T04:36:16.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: [WRN] 2021-02-10T04:36:16.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2021-02-10T04:36:16.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:16.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2021-02-10T04:36:16.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: cluster 2021-02-10T04:36:16.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2021-02-10T04:36:16.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2021 2021-02-10T04:36:16.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: - 2021-02-10T04:36:16.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 02 2021-02-10T04:36:16.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: - 2021-02-10T04:36:16.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 10 2021-02-10T04:36:16.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: T 2021-02-10T04:36:16.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 04 2021-02-10T04:36:16.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: : 2021-02-10T04:36:16.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 36 2021-02-10T04:36:16.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: : 2021-02-10T04:36:16.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 14 2021-02-10T04:36:16.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: . 2021-02-10T04:36:16.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 947828 2021-02-10T04:36:16.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: +0000 2021-02-10T04:36:16.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2021-02-10T04:36:16.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: osd.2 2021-02-10T04:36:16.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: ( 2021-02-10T04:36:16.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: osd 2021-02-10T04:36:16.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: . 2021-02-10T04:36:16.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2 2021-02-10T04:36:16.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: ) 2021-02-10T04:36:16.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 1605 2021-02-10T04:36:16.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: : 2021-02-10T04:36:16.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: cluster 2021-02-10T04:36:16.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2021-02-10T04:36:16.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: [WRN] 2021-02-10T04:36:16.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2021-02-10T04:36:16.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:16.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: cluster 2021-02-10T04:36:14.947832+0000 osd.2 (osd.2) 1606 : cluster 2021-02-10T04:36:16.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:16.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: cluster 2021-02-10T04:36: 2021-02-10T04:36:16.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 14.947836+0000 osd.2 (osd.2) 2021-02-10T04:36:16.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 1607 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:16.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:36:16.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 04:36:14.947840+0000 osd.2 (osd 2021-02-10T04:36:16.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: .2) 1608 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:16.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:16.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: -10T04:36:14.947844+0000 2021-02-10T04:36:16.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: osd.2 2021-02-10T04:36:16.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: (osd.2) 1609 : cluster [WRN] 2021-02-10T04:36:16.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:16.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: cluster 2021-02-10T04:36 2021-02-10T04:36:16.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: :14.947864+0000 osd.2 (osd.2) 1610 2021-02-10T04:36:16.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:16.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: cluster 2021-02-10T04:36: 2021-02-10T04:36:16.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 15.185989+0000 osd.4 (osd.4) 839 2021-02-10T04:36:16.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:16.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:36:16.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: :36:15.185997+0000 osd.4 (osd. 2021-02-10T04:36:16.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 4) 840 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:16.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: cluster 2021-02-10T04:36:16.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 2021-02-10T04:36 2021-02-10T04:36:16.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: :15.186001+0000 osd.4 (osd.4) 841 2021-02-10T04:36:16.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:16.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:36:16.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 10T04:36:15.186006+0000 osd.4 ( 2021-02-10T04:36:16.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: osd.4) 842 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:16.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:16.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: -02-10T04:36:15.322441 2021-02-10T04:36:16.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1394 : cluster [DBG] 2021-02-10T04:36:16.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: osdmap e558: 8 total, 8 up, 8 in 2021-02-10T04:36:16.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: cluster 2021-02-10T04:36: 2021-02-10T04:36:16.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: 16.115804+0000 mgr.y (mgr.14140) 626 : cluster 2021-02-10T04:36:16.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[35961]: [DBG] pgmap v987: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 432 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 7.4 MiB/s rd, 2.6 MiB/s wr, 16 op/s 2021-02-10T04:36:17.181 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:36:17.181 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2021-02-10T04:36:17.181 INFO:teuthology.orchestra.run.smithi164.stdout: "complaint_time": 30, 2021-02-10T04:36:17.181 INFO:teuthology.orchestra.run.smithi164.stdout: "num_blocked_ops": 0 2021-02-10T04:36:17.181 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:36:17.338 INFO:tasks.rados.rados.0.smithi085.stdout:2140: done (9 left) 2021-02-10T04:36:17.338 INFO:tasks.rados.rados.0.smithi085.stdout:2141: copy_from oid 22 from oid 7 current snap is 241 2021-02-10T04:36:17.338 INFO:tasks.rados.rados.0.smithi085.stdout:2142: write (excl) oid 31 current snap is 241 2021-02-10T04:36:17.338 INFO:tasks.rados.rados.0.smithi085.stdout:2142: seq_num 751 ranges {622057=714593,1913624=708433} 2021-02-10T04:36:17.346 INFO:tasks.rados.rados.0.smithi085.stdout:2142: writing smithi08575397-31 from 622057 to 1336650 tid 1 2021-02-10T04:36:17.349 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:16 smithi085 conmon[49106]: debug 2021-02-10T04:36:16.956+0000 7f77a3e94700 -1 osd.2 559 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:17.350 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: cluster 2021-02-10T04:36:17.350 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 2021-02-10T04:36:17.350 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 2021 2021-02-10T04:36:17.350 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: - 2021-02-10T04:36:17.350 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 02 2021-02-10T04:36:17.350 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: - 2021-02-10T04:36:17.350 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 10 2021-02-10T04:36:17.350 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: T 2021-02-10T04:36:17.351 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 04 2021-02-10T04:36:17.351 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: : 2021-02-10T04:36:17.351 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 36 2021-02-10T04:36:17.351 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: : 2021-02-10T04:36:17.351 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 15 2021-02-10T04:36:17.351 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: . 2021-02-10T04:36:17.351 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 981928 2021-02-10T04:36:17.351 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: +0000 2021-02-10T04:36:17.351 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 2021-02-10T04:36:17.351 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: osd.2 2021-02-10T04:36:17.351 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: ( 2021-02-10T04:36:17.351 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: osd 2021-02-10T04:36:17.351 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: . 2021-02-10T04:36:17.352 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 2 2021-02-10T04:36:17.352 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: ) 2021-02-10T04:36:17.352 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 1611 2021-02-10T04:36:17.352 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: : 2021-02-10T04:36:17.352 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: cluster 2021-02-10T04:36:17.352 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 2021-02-10T04:36:17.352 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: [WRN] 2021-02-10T04:36:17.352 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 2021-02-10T04:36:17.352 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:17.352 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 2021-02-10T04:36:17.352 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: cluster 2021-02-10T04:36:17.352 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 2021-02-10T04:36:17.353 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 2021 2021-02-10T04:36:17.353 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: - 2021-02-10T04:36:17.353 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 02 2021-02-10T04:36:17.353 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: - 2021-02-10T04:36:17.353 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 10 2021-02-10T04:36:17.353 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: T 2021-02-10T04:36:17.353 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 04 2021-02-10T04:36:17.353 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: : 2021-02-10T04:36:17.353 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 36 2021-02-10T04:36:17.353 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: : 2021-02-10T04:36:17.353 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 15 2021-02-10T04:36:17.353 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: . 2021-02-10T04:36:17.353 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 981937 2021-02-10T04:36:17.353 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: +0000 2021-02-10T04:36:17.354 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: osd.2 (osd.2) 1612 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:17.354 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: cluster 2021-02-10T04:36:15.981943+0000 osd.2 (osd.2) 1613 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:17.354 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:17.354 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 02-10T04:36:15.981950+0000 osd.2 (osd.2) 1614 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:17.354 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 2021-02-10T04:36:17.354 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:36:17.354 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: T04:36:15.981956+0000 osd.2 (osd.2) 1615 : cluster [WRN] 2021-02-10T04:36:17.354 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:17.354 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:36:17.355 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 36:15.981962+0000 osd.2 (osd.2) 1616 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:17.355 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: cluster 2021-02-10T04:36:17.355 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 2021-02-10T04:36:15.981968+0000 osd.2 (osd.2) 1617 : 2021-02-10T04:36:17.355 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:17.355 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: cluster 2021-02-10T04:36:17.356 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 2021-02-10T04:36:15.981973+0000 osd.2 (osd.2) 2021-02-10T04:36:17.356 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 1618 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:17.356 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:17.356 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: -10T04:36:15.981980+0000 osd.2 (osd.2) 1619 : cluster [WRN] 2021-02-10T04:36:17.356 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:17.356 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:17.356 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 02-10T04:36:15.981986+0000 osd.2 (osd.2) 1620 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:17.356 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: cluster 2021-02-10T04:36:16.212768+0000 osd.4 (osd.4) 843 2021-02-10T04:36:17.356 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:17.356 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: cluster 2021-02-10T04:36:17.356 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 2021-02-10T04:36:16.212775+0000 osd.4 (osd. 2021-02-10T04:36:17.356 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 4) 844 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:17.356 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: cluster 2021-02-10T04:36: 2021-02-10T04:36:17.357 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 16.212778+0000 osd.4 (osd.4) 845 : cluster 2021-02-10T04:36:17.357 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:17.357 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:17.357 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: -02-10T04:36:16.212782+0000 osd.4 (osd.4) 846 2021-02-10T04:36:17.357 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:17.357 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: cluster 2021-02-10T04:36:16.335594+0000 mon.a (mon. 2021-02-10T04:36:17.357 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[29319]: 0) 1395 : cluster [DBG] osdmap e559: 8 total, 8 up, 8 in 2021-02-10T04:36:17.357 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: cluster 2021-02-10T04:36:17.357 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:36:17.357 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: 36:15.981928+0000 osd.2 (osd.2) 1611 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:17.357 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:17.357 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: 02-10T04:36:15.981937+0000 osd.2 (osd 2021-02-10T04:36:17.358 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: .2) 1612 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:17.358 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:17.358 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: -10T04:36:15.981943+0000 osd.2 2021-02-10T04:36:17.358 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: (osd.2) 1613 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:17.358 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:17.358 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: 02-10T04:36:15.981950+0000 osd.2 ( 2021-02-10T04:36:17.358 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: osd.2) 1614 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:17.358 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:36:17.358 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: 10T04:36:15.981956+0000 osd.2 2021-02-10T04:36:17.358 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: (osd.2) 1615 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:17.358 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:17.358 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: 02-10T04:36:15.981962+0000 2021-02-10T04:36:17.358 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: osd.2 (osd.2) 1616 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:17.359 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: 2021-02-10T04:36:17.359 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: cluster 2021-02-10T04:36:17.359 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: 2021-02-10T04:36:15.981968+0000 osd.2 ( 2021-02-10T04:36:17.359 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: osd.2) 1617 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:17.359 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:36:17.359 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: 10T04:36:15.981973+0000 osd.2 (osd 2021-02-10T04:36:17.359 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: .2) 1618 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:17.359 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:17.359 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: 02-10T04:36:15.981980+0000 osd.2 (osd 2021-02-10T04:36:17.359 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: .2) 1619 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:17.359 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:36:17.359 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: 10T04:36:15.981986+0000 osd.2 (osd 2021-02-10T04:36:17.360 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: .2) 1620 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:17.360 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:17.360 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: -02-10T04:36:16.212768+0000 2021-02-10T04:36:17.360 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: osd.4 (osd.4) 843 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:17.360 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:17.360 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: -10T04:36:16.212775+0000 osd.4 ( 2021-02-10T04:36:17.360 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: osd.4) 844 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:17.360 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:36:17.360 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: 10T04:36:16.212778+0000 osd.4 (osd 2021-02-10T04:36:17.360 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: .4) 845 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:17.360 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:17.360 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: 02-10T04:36:16.212782+0000 2021-02-10T04:36:17.360 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: osd.4 (osd.4) 846 : cluster 2021-02-10T04:36:17.361 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:17.361 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:17.361 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: T04:36:16.335594+0000 mon.a (mon 2021-02-10T04:36:17.361 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:17 smithi085 conmon[35961]: .0) 1395 : cluster [DBG] osdmap e559: 8 total, 8 up, 8 in 2021-02-10T04:36:17.361 INFO:tasks.rados.rados.0.smithi085.stdout:2142: writing smithi08575397-31 from 1913624 to 2622057 tid 2 2021-02-10T04:36:17.361 INFO:tasks.rados.rados.0.smithi085.stdout:2143: copy_from oid 12 from oid 5 current snap is 241 2021-02-10T04:36:17.361 INFO:tasks.rados.rados.0.smithi085.stdout:2144: write oid 23 current snap is 241 2021-02-10T04:36:17.361 INFO:tasks.rados.rados.0.smithi085.stdout:2144: seq_num 752 ranges {638864=789842,1972106=456248} 2021-02-10T04:36:17.364 INFO:tasks.rados.rados.0.smithi085.stdout:2144: writing smithi08575397-23 from 638864 to 1428706 tid 1 2021-02-10T04:36:17.370 INFO:tasks.rados.rados.0.smithi085.stdout:2144: writing smithi08575397-23 from 1972106 to 2428354 tid 2 2021-02-10T04:36:17.371 INFO:tasks.rados.rados.0.smithi085.stdout:2141: finishing copy_from to smithi08575397-22 2021-02-10T04:36:17.371 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 751 (ObjNum 727 snap 230 seq_num 727) dirty exists 2021-02-10T04:36:17.372 INFO:tasks.rados.rados.0.smithi085.stdout:2141: finishing copy_from racing read to smithi08575397-22 2021-02-10T04:36:17.372 INFO:tasks.rados.rados.0.smithi085.stdout:2142: finishing write tid 1 to smithi08575397-31 2021-02-10T04:36:17.372 INFO:tasks.rados.rados.0.smithi085.stdout:2142: finishing write tid 2 to smithi08575397-31 2021-02-10T04:36:17.372 INFO:tasks.rados.rados.0.smithi085.stdout:2142: finishing write tid 3 to smithi08575397-31 2021-02-10T04:36:17.373 INFO:tasks.rados.rados.0.smithi085.stdout:2142: finishing write tid 4 to smithi08575397-31 2021-02-10T04:36:17.373 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 786 (ObjNum 751 snap 241 seq_num 751) dirty exists 2021-02-10T04:36:17.373 INFO:tasks.rados.rados.0.smithi085.stdout:2142: left oid 31 (ObjNum 751 snap 241 seq_num 751) 2021-02-10T04:36:17.373 INFO:tasks.rados.rados.0.smithi085.stdout:2141: done (12 left) 2021-02-10T04:36:17.374 INFO:tasks.rados.rados.0.smithi085.stdout:2142: done (11 left) 2021-02-10T04:36:17.374 INFO:tasks.rados.rados.0.smithi085.stdout:2145: write oid 14 current snap is 241 2021-02-10T04:36:17.374 INFO:tasks.rados.rados.0.smithi085.stdout:2145: seq_num 753 ranges {655671=581444,2030588=625083} 2021-02-10T04:36:17.379 INFO:tasks.rados.rados.0.smithi085.stdout:2145: writing smithi08575397-14 from 655671 to 1237115 tid 1 2021-02-10T04:36:17.387 INFO:tasks.rados.rados.0.smithi085.stdout:2145: writing smithi08575397-14 from 2030588 to 2655671 tid 2 2021-02-10T04:36:17.387 INFO:tasks.rados.rados.0.smithi085.stdout:2146: snap_remove snap 232 2021-02-10T04:36:17.527 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:36:17.565 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: cluster 2021-02-10T04:36:15.981928 2021-02-10T04:36:17.565 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1611 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:17.566 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:36:17.566 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: 10T04:36:15.981937+0000 osd.2 (osd.2) 1612 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:17.566 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:17.566 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: -02-10T04:36:15.981943+0000 osd.2 ( 2021-02-10T04:36:17.566 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: osd.2) 1613 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:17.566 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:36:17.566 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: 36:15.981950+0000 osd.2 (osd.2) 1614 : cluster 2021-02-10T04:36:17.566 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:17.566 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: cluster 2021-02-10T04:36:15.981956+0000 2021-02-10T04:36:17.566 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: osd.2 (osd.2) 1615 : cluster [WRN] 2021-02-10T04:36:17.566 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:17.566 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:17.567 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: -10T04:36:15.981962+0000 osd.2 (osd 2021-02-10T04:36:17.567 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: .2) 1616 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:17.567 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:17.567 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: -02-10T04:36:15.981968+0000 osd.2 ( 2021-02-10T04:36:17.567 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: osd.2) 1617 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:17.567 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: cluster 2021-02-10T04:36:17.567 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:36:17.567 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: 36:15.981973+0000 osd.2 (osd.2) 1618 : 2021-02-10T04:36:17.567 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:17.567 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: cluster 2021-02-10T04:36:17.567 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: 2021-02-10T04:36:15.981980+0000 2021-02-10T04:36:17.567 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: osd.2 (osd.2) 1619 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:17.567 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: cluster 2021-02-10T04:36:15.981986+0000 osd.2 (osd.2) 1620 : cluster 2021-02-10T04:36:17.568 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:17.568 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:17.568 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: 02-10T04:36:16.212768+0000 osd.4 ( 2021-02-10T04:36:17.568 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: osd.4) 843 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:17.568 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: cluster 2021-02-10T04:36:17.568 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: 2021-02-10T04:36: 2021-02-10T04:36:17.568 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: 16.212775+0000 osd.4 (osd.4) 844 : cluster 2021-02-10T04:36:17.568 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:17.568 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: cluster 2021-02-10T04:36: 2021-02-10T04:36:17.568 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: 16.212778+0000 osd.4 (osd.4) 845 : cluster 2021-02-10T04:36:17.568 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:17.568 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: cluster 2021-02-10T04:36:16.212782 2021-02-10T04:36:17.568 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 846 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:17.569 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: 2021-02-10T04:36:17.569 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:36:17.569 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: 10T04:36:16.335594+0000 mon.a ( 2021-02-10T04:36:17.569 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[29288]: mon.0) 1395 : cluster [DBG] osdmap e559: 8 total, 8 up, 8 in 2021-02-10T04:36:17.569 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[33677]: debug 2021-02-10T04:36:17.569 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[33677]: 2021-02-10T04:36:17.181+0000 7fbec19a1700 -1 osd.4 559 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:17.569 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:36:17.851 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:17 smithi164 conmon[33677]: debug 2021-02-10T04:36:17.849+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:36:17.904 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:36:18.137 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:36:18.192 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[49106]: debug 2021-02-10T04:36:18.004+0000 7f77a3e94700 -1 osd.2 560 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:18.358 INFO:tasks.rados.rados.0.smithi085.stdout:2146: done (12 left) 2021-02-10T04:36:18.358 INFO:tasks.rados.rados.0.smithi085.stdout:2147: write oid 37 current snap is 241 2021-02-10T04:36:18.358 INFO:tasks.rados.rados.0.smithi085.stdout:2147: seq_num 754 ranges {672478=656693,2089070=459917} 2021-02-10T04:36:18.366 INFO:tasks.rados.rados.0.smithi085.stdout:2147: writing smithi08575397-37 from 672478 to 1329171 tid 1 2021-02-10T04:36:18.374 INFO:tasks.rados.rados.0.smithi085.stdout:2147: writing smithi08575397-37 from 2089070 to 2548987 tid 2 2021-02-10T04:36:18.375 INFO:tasks.rados.rados.0.smithi085.stdout:2144: finishing write tid 1 to smithi08575397-23 2021-02-10T04:36:18.375 INFO:tasks.rados.rados.0.smithi085.stdout:2143: finishing copy_from to smithi08575397-12 2021-02-10T04:36:18.375 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 751 (ObjNum 745 snap 240 seq_num 745) dirty exists 2021-02-10T04:36:18.375 INFO:tasks.rados.rados.0.smithi085.stdout:2143: finishing copy_from racing read to smithi08575397-12 2021-02-10T04:36:18.375 INFO:tasks.rados.rados.0.smithi085.stdout:2144: finishing write tid 2 to smithi08575397-23 2021-02-10T04:36:18.375 INFO:tasks.rados.rados.0.smithi085.stdout:2144: finishing write tid 3 to smithi08575397-23 2021-02-10T04:36:18.375 INFO:tasks.rados.rados.0.smithi085.stdout:2144: finishing write tid 4 to smithi08575397-23 2021-02-10T04:36:18.376 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 687 (ObjNum 752 snap 241 seq_num 752) dirty exists 2021-02-10T04:36:18.376 INFO:tasks.rados.rados.0.smithi085.stdout:2144: left oid 23 (ObjNum 752 snap 241 seq_num 752) 2021-02-10T04:36:18.376 INFO:tasks.rados.rados.0.smithi085.stdout:2145: finishing write tid 1 to smithi08575397-14 2021-02-10T04:36:18.376 INFO:tasks.rados.rados.0.smithi085.stdout:2145: finishing write tid 2 to smithi08575397-14 2021-02-10T04:36:18.376 INFO:tasks.rados.rados.0.smithi085.stdout:2145: finishing write tid 3 to smithi08575397-14 2021-02-10T04:36:18.376 INFO:tasks.rados.rados.0.smithi085.stdout:2145: finishing write tid 4 to smithi08575397-14 2021-02-10T04:36:18.376 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 826 (ObjNum 753 snap 241 seq_num 753) dirty exists 2021-02-10T04:36:18.376 INFO:tasks.rados.rados.0.smithi085.stdout:2145: left oid 14 (ObjNum 753 snap 241 seq_num 753) 2021-02-10T04:36:18.376 INFO:tasks.rados.rados.0.smithi085.stdout:2143: done (12 left) 2021-02-10T04:36:18.376 INFO:tasks.rados.rados.0.smithi085.stdout:2144: done (11 left) 2021-02-10T04:36:18.376 INFO:tasks.rados.rados.0.smithi085.stdout:2145: done (10 left) 2021-02-10T04:36:18.376 INFO:tasks.rados.rados.0.smithi085.stdout:2148: write oid 12 current snap is 241 2021-02-10T04:36:18.376 INFO:tasks.rados.rados.0.smithi085.stdout:2148: seq_num 755 ranges {689285=731942,2147552=541733} 2021-02-10T04:36:18.384 INFO:tasks.rados.rados.0.smithi085.stdout:2148: writing smithi08575397-12 from 689285 to 1421227 tid 1 2021-02-10T04:36:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: cluster 2021-02-10T04:36:16.957053+0000 osd.2 (osd 2021-02-10T04:36:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: .2) 1621 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: cluster 2021-02-10T04:36:16.957063+0000 osd.2 (osd.2) 1622 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: cluster 2021-02-10T04:36:16.957081+0000 osd.2 ( 2021-02-10T04:36:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: osd.2) 1623 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: cluster 2021-02-10T04:36:16.957086+0000 osd.2 (osd.2) 1624 : 2021-02-10T04:36:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: cluster 2021-02-10T04:36:16.957088+0000 2021-02-10T04:36:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: osd.2 (osd.2) 1625 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:18.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: cluster 2021-02-10T04:36 2021-02-10T04:36:18.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: :16.957093+0000 osd.2 (osd.2) 1626 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:18.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: 2021-02-10T04:36:18.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:36:18.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: 04:36:16.957098+0000 osd.2 (osd.2) 1627 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:18.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: cluster 2021-02-10T04:36:16 2021-02-10T04:36:18.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: .957105+0000 osd.2 (osd.2) 1628 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:18.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: cluster 2021-02-10T04:36:16 2021-02-10T04:36:18.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: .957110+0000 osd.2 (osd.2) 1629 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:18.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: cluster 2021-02-10T04:36:16.957127+0000 osd.2 (osd 2021-02-10T04:36:18.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: .2) 1630 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:18.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: cluster 2021-02-10T04:36:17 2021-02-10T04:36:18.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: .182215+0000 osd.4 (osd.4) 847 : cluster [WRN] 2021-02-10T04:36:18.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:18.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: cluster 2021-02-10T04:36:17.182224 2021-02-10T04:36:18.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 848 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:18.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: cluster 2021-02-10T04:36:17.182230+0000 2021-02-10T04:36:18.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: osd.4 (osd.4) 849 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:18.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: cluster 2021-02-10T04:36:17.182236+0000 2021-02-10T04:36:18.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: osd.4 (osd.4) 850 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:18.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:18.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: -02-10T04:36:17.334525+0000 2021-02-10T04:36:18.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: mon.a (mon.0) 1396 : cluster [DBG] osdmap e560: 8 total, 8 up, 8 in 2021-02-10T04:36:18.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:36:18.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: T04:36:18.116392+0000 mgr.y (mgr 2021-02-10T04:36:18.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: .14140) 627 : cluster [DBG] pgmap v990: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 433 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.4 MiB/s rd, 2.3 MiB/s wr, 12 op/s 2021-02-10T04:36:18.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:36:18.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: -10T04:36:18.137046+0000 mgr.y ( 2021-02-10T04:36:18.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[29288]: mgr.14140) 628 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:36:18.397 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:18 smithi164 conmon[33677]: debug 2021-02-10T04:36:18.208+0000 7fbec19a1700 -1 osd.4 560 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:18.398 INFO:tasks.rados.rados.0.smithi085.stdout:2148: writing smithi08575397-12 from 2147552 to 2689285 tid 2 2021-02-10T04:36:18.398 INFO:tasks.rados.rados.0.smithi085.stdout:2148: finishing write tid 1 to smithi08575397-12 2021-02-10T04:36:18.398 INFO:tasks.rados.rados.0.smithi085.stdout:2149: read oid 31 snap -1 2021-02-10T04:36:18.398 INFO:tasks.rados.rados.0.smithi085.stdout:2149: expect (ObjNum 751 snap 241 seq_num 751) 2021-02-10T04:36:18.398 INFO:tasks.rados.rados.0.smithi085.stdout:2150: read oid 35 snap -1 2021-02-10T04:36:18.398 INFO:tasks.rados.rados.0.smithi085.stdout:2150: expect (ObjNum 578 snap 176 seq_num 578) 2021-02-10T04:36:18.398 INFO:tasks.rados.rados.0.smithi085.stdout:2151: write (excl) oid 3 current snap is 241 2021-02-10T04:36:18.398 INFO:tasks.rados.rados.0.smithi085.stdout:2151: seq_num 756 ranges {706092=407191,1806034=463586} 2021-02-10T04:36:18.404 INFO:tasks.rados.rados.0.smithi085.stdout:2151: writing smithi08575397-3 from 706092 to 1113283 tid 1 2021-02-10T04:36:18.411 INFO:tasks.rados.rados.0.smithi085.stdout:2151: writing smithi08575397-3 from 1806034 to 2269620 tid 2 2021-02-10T04:36:18.411 INFO:tasks.rados.rados.0.smithi085.stdout:2152: snap_remove snap 207 2021-02-10T04:36:18.659 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:36:18.690 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:36:18.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: cluster 2021-02-10T04:36:16.957053+0000 2021-02-10T04:36:18.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: osd.2 (osd.2) 1621 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:18.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: cluster 2021-02-10T04:36:16.957063+0000 osd.2 (osd.2) 2021-02-10T04:36:18.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: 1622 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:18.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:18.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: -02-10T04:36:16.957081+0000 osd.2 (osd.2) 1623 2021-02-10T04:36:18.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:18.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:18.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: 02-10T04:36:16.957086+0000 2021-02-10T04:36:18.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: osd.2 (osd.2) 1624 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:18.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: 2021-02-10T04:36:18.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:36:18.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: 36:16.957088+0000 osd.2 (osd.2) 2021-02-10T04:36:18.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: 1625 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:18.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:18.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: -02-10T04:36:16.957093+0000 2021-02-10T04:36:18.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: osd.2 (osd.2) 1626 : cluster [WRN] 2021-02-10T04:36:18.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:18.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:18.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: -10T04: 2021-02-10T04:36:18.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: 36:16.957098+0000 osd.2 (osd.2) 1627 : 2021-02-10T04:36:18.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:18.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:18.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: 02-10T04:36:16.957105+0000 2021-02-10T04:36:18.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: osd.2 (osd.2) 1628 : cluster [WRN] 2021-02-10T04:36:18.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:18.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:36:18.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: 04:36:16.957110+0000 osd.2 (osd. 2021-02-10T04:36:18.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: 2) 1629 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:18.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:36:18.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: 36:16.957127+0000 osd.2 (osd.2) 1630 : cluster 2021-02-10T04:36:18.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:18.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:18.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: -10T04:36:17.182215+0000 osd.4 ( 2021-02-10T04:36:18.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: osd.4) 847 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:18.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:18.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: 02-10T04:36:17.182224+0000 2021-02-10T04:36:18.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: osd.4 (osd.4) 848 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:18.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: 2021-02-10T04:36:18.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:36:18.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: 36:17.182230+0000 osd.4 (osd.4) 2021-02-10T04:36:18.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: 849 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:18.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:18.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: 02-10T04:36:17.182236+0000 2021-02-10T04:36:18.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: osd.4 (osd.4) 850 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:18.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: 2021-02-10T04:36:18.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:36:18.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: 36:17.334525+0000 mon.a (mon.0 2021-02-10T04:36:18.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: ) 1396 : cluster [DBG] osdmap e560: 8 total, 8 up, 8 in 2021-02-10T04:36:18.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: cluster 2021-02-10T04:36:18.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: 2021-02-10T04:36:18.116392 2021-02-10T04:36:18.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 627 : cluster [DBG] 2021-02-10T04:36:18.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: pgmap v990: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 433 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.4 MiB/s rd, 2.3 MiB/s wr, 12 op/s 2021-02-10T04:36:18.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:36:18.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: T04:36:18.137046+0000 mgr.y ( 2021-02-10T04:36:18.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[35961]: mgr.14140) 628 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:36:18.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: cluster 2021-02-10T04:36:16.957053+0000 osd.2 ( 2021-02-10T04:36:18.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: osd.2) 1621 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:18.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: cluster 2021-02-10T04:36:16.957063+0000 osd.2 (osd.2) 1622 : cluster [WRN] 2021-02-10T04:36:18.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:18.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:18.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: -02-10T04:36:16.957081+0000 osd.2 (osd.2) 2021-02-10T04:36:18.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 1623 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:18.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: cluster 2021-02-10T04:36:18.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 2021-02-10T04:36:16.957086+0000 osd.2 2021-02-10T04:36:18.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: (osd.2) 1624 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:18.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: cluster 2021-02-10T04:36:18.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 2021-02-10T04:36:18.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 2021 2021-02-10T04:36:18.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: - 2021-02-10T04:36:18.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 02 2021-02-10T04:36:18.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: - 2021-02-10T04:36:18.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 10 2021-02-10T04:36:18.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: T 2021-02-10T04:36:18.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 04 2021-02-10T04:36:18.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: : 2021-02-10T04:36:18.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 36 2021-02-10T04:36:18.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: : 2021-02-10T04:36:18.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 16 2021-02-10T04:36:18.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: . 2021-02-10T04:36:18.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 957088 2021-02-10T04:36:18.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: +0000 2021-02-10T04:36:18.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: osd.2 (osd.2) 2021-02-10T04:36:18.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 1625 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:18.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: cluster 2021-02-10T04:36:18.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 2021 2021-02-10T04:36:18.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: - 2021-02-10T04:36:18.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 02 2021-02-10T04:36:18.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: - 2021-02-10T04:36:18.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 10 2021-02-10T04:36:18.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: T 2021-02-10T04:36:18.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 04 2021-02-10T04:36:18.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: : 2021-02-10T04:36:18.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 36 2021-02-10T04:36:18.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: : 2021-02-10T04:36:18.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 16 2021-02-10T04:36:18.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: . 2021-02-10T04:36:18.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 957093 2021-02-10T04:36:18.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: +0000 2021-02-10T04:36:18.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 2021-02-10T04:36:18.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: osd.2 2021-02-10T04:36:18.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: ( 2021-02-10T04:36:18.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: osd 2021-02-10T04:36:18.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: . 2021-02-10T04:36:18.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 2 2021-02-10T04:36:18.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: ) 2021-02-10T04:36:18.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 1626 2021-02-10T04:36:18.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: : 2021-02-10T04:36:18.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: cluster 2021-02-10T04:36:18.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:18.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:36:18.705 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 36:16.957098+0000 osd.2 (osd.2) 1627 : cluster [WRN] 2021-02-10T04:36:18.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:18.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:36:18.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 04:36:16.957105+0000 osd.2 (osd. 2021-02-10T04:36:18.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 2) 1628 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:18.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: cluster 2021-02-10T04:36:18.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 2021- 2021-02-10T04:36:18.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 02-10T04:36:16.957110+0000 osd.2 (osd.2) 1629 : cluster 2021-02-10T04:36:18.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:18.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: cluster 2021-02-10T04:36:18.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 2021-02-10T04:36:16.957127 2021-02-10T04:36:18.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1630 : cluster 2021-02-10T04:36:18.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:18.706 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:36:18.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: T04:36:17.182215+0000 osd.4 (osd 2021-02-10T04:36:18.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: .4) 847 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:18.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: cluster 2021-02-10T04:36:18.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:36:18.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: T04:36:17.182224+0000 osd.4 ( 2021-02-10T04:36:18.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: osd.4) 848 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:18.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: cluster 2021-02-10T04:36:18.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 2021-02-10T04:36:17.182230+0000 2021-02-10T04:36:18.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: osd.4 (osd.4) 849 : cluster [WRN] 2021-02-10T04:36:18.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:18.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:36:18.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 36:17.182236+0000 osd.4 (osd.4 2021-02-10T04:36:18.707 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: ) 850 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:18.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: cluster 2021-02-10T04:36:18.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: 2021-02-10T04:36:17 2021-02-10T04:36:18.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: .334525+0000 mon.a (mon.0) 1396 : 2021-02-10T04:36:18.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: cluster [DBG] osdmap e560: 8 total, 8 up, 8 in 2021-02-10T04:36:18.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:18.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: -02-10T04:36:18.116392+0000 2021-02-10T04:36:18.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: mgr.y (mgr.14140) 627 : cluster [DBG] 2021-02-10T04:36:18.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: pgmap v990: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 433 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.4 MiB/s rd, 2.3 MiB/s wr, 12 op/s 2021-02-10T04:36:18.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:36:18.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: :36:18.137046+0000 mgr.y (mgr.14140 2021-02-10T04:36:18.708 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:18 smithi085 conmon[29319]: ) 628 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:36:18.720 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:36:18.755 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:36:18.793 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:36:18.829 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:36:18.863 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:36:18.894 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:36:18.895 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:36:18.896 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:36:18.896 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:36:18.896 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:36:18.896 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:36:18.896 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:36:18.896 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:36:18.896 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:36:19.363 INFO:tasks.rados.rados.0.smithi085.stdout:2152: done (14 left) 2021-02-10T04:36:19.363 INFO:tasks.rados.rados.0.smithi085.stdout:2153: write oid 40 current snap is 241 2021-02-10T04:36:19.363 INFO:tasks.rados.rados.0.smithi085.stdout:2153: seq_num 757 ranges {722899=482440,1748163=723597} 2021-02-10T04:36:19.369 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is unset 2021-02-10T04:36:19.370 INFO:tasks.rados.rados.0.smithi085.stdout:2153: writing smithi08575397-40 from 722899 to 1205339 tid 1 2021-02-10T04:36:19.374 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.6 to 1 (10000) 2021-02-10T04:36:19.378 INFO:tasks.rados.rados.0.smithi085.stdout:2153: writing smithi08575397-40 from 1748163 to 2471760 tid 2 2021-02-10T04:36:19.378 INFO:tasks.rados.rados.0.smithi085.stdout:2154: rollback oid 26 current snap is 241 2021-02-10T04:36:19.378 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 26 to 190 2021-02-10T04:36:19.378 INFO:tasks.rados.rados.0.smithi085.stdout:2148: finishing write tid 2 to smithi08575397-12 2021-02-10T04:36:19.378 INFO:tasks.rados.rados.0.smithi085.stdout:2148: finishing write tid 3 to smithi08575397-12 2021-02-10T04:36:19.378 INFO:tasks.rados.rados.0.smithi085.stdout:2148: finishing write tid 4 to smithi08575397-12 2021-02-10T04:36:19.378 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 829 (ObjNum 755 snap 241 seq_num 755) dirty exists 2021-02-10T04:36:19.378 INFO:tasks.rados.rados.0.smithi085.stdout:2148: left oid 12 (ObjNum 755 snap 241 seq_num 755) 2021-02-10T04:36:19.379 INFO:tasks.rados.rados.0.smithi085.stdout:2149: expect (ObjNum 751 snap 241 seq_num 751) 2021-02-10T04:36:19.531 INFO:tasks.rados.rados.0.smithi085.stdout:2150: expect (ObjNum 578 snap 176 seq_num 578) 2021-02-10T04:36:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: cluster 2021-02-10T04:36:18.004724+0000 osd.2 (osd.2) 1631 : cluster 2021-02-10T04:36:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: cluster 2021-02-10T04:36:18.004734+0000 osd.2 (osd.2) 1632 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: cluster 2021-02-10T04:36:18 2021-02-10T04:36:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: .004741+0000 osd.2 (osd.2) 1633 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: cluster 2021-02-10T04:36:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: 2021-02-10T04:36:18.004748 2021-02-10T04:36:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1634 : cluster [WRN] 2021-02-10T04:36:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: cluster 2021-02-10T04:36: 2021-02-10T04:36:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: 18.004753+0000 osd.2 (osd.2) 1635 : 2021-02-10T04:36:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:36:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: :36:18.004759+0000 osd.2 (osd.2 2021-02-10T04:36:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: ) 1636 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:36:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: 10T04:36:18.004766+0000 osd.2 ( 2021-02-10T04:36:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: osd.2) 1637 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: cluster 2021-02-10T04:36:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: 2021-02-10T04:36:18. 2021-02-10T04:36:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: 004772+0000 osd.2 (osd.2) 1638 : cluster 2021-02-10T04:36:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: cluster 2021-02-10T04:36: 2021-02-10T04:36:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: 18.004779+0000 osd.2 (osd.2) 1639 2021-02-10T04:36:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:36:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: 04:36:18.004785+0000 osd.2 (osd. 2021-02-10T04:36:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: 2) 1640 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: -10T04:36:18.209625+0000 osd.4 2021-02-10T04:36:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: (osd.4) 851 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: -02-10T04:36:18.209631+0000 2021-02-10T04:36:19.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: osd.4 (osd.4) 852 : cluster [WRN] 2021-02-10T04:36:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: cluster 2021-02-10T04:36:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: 2021-02-10T04:36:18.209637+0000 2021-02-10T04:36:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: osd.4 (osd.4) 853 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: -02-10T04:36:18.209643+0000 2021-02-10T04:36:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: osd.4 (osd.4) 854 : cluster [WRN] 2021-02-10T04:36:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: cluster 2021-02-10T04:36:18.340715+0000 2021-02-10T04:36:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: mon.a (mon.0) 1397 : cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 407 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:36:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: 2021-02-10T04:36:19.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: cluster 2021-02-10T04:36:19.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: 2021-02-10T04:36:18.358521+0000 2021-02-10T04:36:19.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: mon.a (mon.0) 1398 : cluster [DBG] osdmap e561: 8 total, 8 up, 8 in 2021-02-10T04:36:19.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: 2021-02-10T04:36:19.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: audit 2021-02-10T04:36:19.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: 2021-02-10T04:36:18.583380+0000 2021-02-10T04:36:19.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: mon.b (mon 2021-02-10T04:36:19.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: .2) 35 2021-02-10T04:36:19.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: : audit [INF] from='client.? 172.21.15.85:0/658408887' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T04:36:19.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: audit 2021 2021-02-10T04:36:19.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: -02-10T04:36:18.583815 2021-02-10T04:36:19.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: +0000 mon.a (mon. 2021-02-10T04:36:19.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: 0) 1399 2021-02-10T04:36:19.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T04:36:19.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: audit 2021 2021-02-10T04:36:19.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: -02-10T04:36:18.628977+0000 2021-02-10T04:36:19.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: mon.b (mon.2) 36 : audit [DBG] from='client.? 172.21.15.85:0/1260347953' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:36:19.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: audit 2021-02-10T04:36:18.818597 2021-02-10T04:36:19.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: +0000 mon.b (mon.2) 37 : audit [INF] from='client.? 172.21.15.85:0/3166515971' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:36:19.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: audit 2021-02-10T04:36:18 2021-02-10T04:36:19.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:19 smithi164 conmon[29288]: .819055+0000 mon.a (mon.0) 1400 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:36:19.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: cluster 2021-02-10T04:36:18.004724+0000 osd.2 (osd.2) 1631 : cluster 2021-02-10T04:36:19.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: cluster 2021-02-10T04:36:18.004734+0000 osd.2 (osd.2) 1632 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: cluster 2021-02-10T04:36:18. 2021-02-10T04:36:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: 004741+0000 osd.2 (osd.2) 1633 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: cluster 2021-02-10T04:36:18 2021-02-10T04:36:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: .004748+0000 osd.2 (osd.2) 1634 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: cluster 2021-02-10T04:36:18.004753+0000 2021-02-10T04:36:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: osd.2 (osd.2) 1635 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: cluster 2021-02-10T04:36:18.004759 2021-02-10T04:36:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1636 : cluster [WRN] 2021-02-10T04:36:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: cluster 2021-02-10T04:36:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: 2021-02-10T04:36:18.004766+0000 osd.2 (osd.2) 1637 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: 2021-02-10T04:36:19.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:36:19.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: 04:36:18.004772+0000 osd.2 (osd.2) 1638 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:19.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: 2021-02-10T04:36:19.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:19.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: 02-10T04:36:18.004779+0000 osd.2 (osd.2) 1639 2021-02-10T04:36:19.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:19.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: cluster 2021-02-10T04:36:18.004785+0000 osd.2 ( 2021-02-10T04:36:19.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: osd.2) 1640 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:19.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: cluster 2021-02-10T04:36:18.209625+0000 osd.4 (osd 2021-02-10T04:36:19.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: .4) 851 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:19.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: cluster 2021-02-10T04:36:18. 2021-02-10T04:36:19.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: 209631+0000 osd.4 (osd.4) 852 : cluster 2021-02-10T04:36:19.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:19.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:36:19.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: :36:18.209637+0000 osd.4 (osd.4) 2021-02-10T04:36:19.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: 853 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:19.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: cluster 2021-02-10T04:36:18. 2021-02-10T04:36:19.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: 209643+0000 osd.4 (osd.4) 854 : cluster [WRN] 2021-02-10T04:36:19.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:19.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:36:19.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: :36:18.340715+0000 mon.a (mon. 2021-02-10T04:36:19.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: 0) 1397 : cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 407 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:36:19.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: 2021-02-10T04:36:19.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:19.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: 02-10T04:36: 2021-02-10T04:36:19.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: 18.358521+0000 mon.a (mon. 2021-02-10T04:36:19.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: 0) 1398 : cluster [DBG] 2021-02-10T04:36:19.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: osdmap e561: 8 total, 8 up, 8 in 2021-02-10T04:36:19.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: audit 2021-02-10T 2021-02-10T04:36:19.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: 04:36:18.583380+0000 mon.b (mon.2 2021-02-10T04:36:19.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: ) 35 : audit [INF] from='client.? 172.21.15.85:0/658408887' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T04:36:19.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: audit 2021-02-10T04:36:19.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: 2021-02-10T04:36:18.583815+0000 2021-02-10T04:36:19.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: mon.a (mon.0) 1399 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T04:36:19.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: 2021-02-10T04:36:19.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: audit 2021-02-10T04:36:19.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: 2021-02-10T04:36:18.628977+0000 2021-02-10T04:36:19.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: mon.b (mon.2) 36 : audit [DBG] from='client.? 172.21.15.85:0/1260347953' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:36:19.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: 2021-02-10T04:36:19.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: audit 2021-02-10T04:36:19.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: 2021-02-10T04:36:18.818597+0000 mon.b (mon 2021-02-10T04:36:19.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: .2) 37 : audit [INF] from='client.? 172.21.15.85:0/3166515971' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:36:19.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:36:19.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: 10T04:36:18.819055+0000 2021-02-10T04:36:19.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: mon.a (mon.0) 1400 : audit 2021-02-10T04:36:19.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[29319]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:36:19.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: cluster 2021-02-10T04:36:19.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: 2021-02-10T04:36: 2021-02-10T04:36:19.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: 18.004724+0000 osd.2 (osd.2) 1631 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:19.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:19.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: -02-10T04:36:18.004734+0000 osd.2 2021-02-10T04:36:19.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: (osd.2) 1632 : cluster [WRN] 2021-02-10T04:36:19.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:19.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:19.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: 02-10T04:36:18.004741+0000 2021-02-10T04:36:19.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: osd.2 (osd.2) 1633 : cluster 2021-02-10T04:36:19.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:19.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:19.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: 02-10T04:36:18.004748+0000 2021-02-10T04:36:19.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: osd.2 (osd.2) 1634 : cluster 2021-02-10T04:36:19.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:19.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: cluster 2021-02-10T04:36:19.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: 2021-02-10T04:36:18 2021-02-10T04:36:19.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: .004753+0000 osd.2 (osd.2) 1635 : cluster 2021-02-10T04:36:19.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:19.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: cluster 2021-02-10T04:36:18.004759+0000 2021-02-10T04:36:19.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: osd.2 (osd.2) 1636 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:19.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: 2021-02-10T04:36:19.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: cluster 2021-02-10T04:36:19.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: 2021-02-10T04:36:18.004766+0000 2021-02-10T04:36:19.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: osd.2 (osd.2) 1637 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:19.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:19.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: -02-10T04:36:18.004772+0000 osd.2 ( 2021-02-10T04:36:19.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: osd.2) 1638 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:19.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:36:19.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: 36:18.004779+0000 osd.2 (osd.2) 1639 : cluster [WRN] 2021-02-10T04:36:19.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:19.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:19.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: -02-10T04:36:18.004785+0000 osd.2 (osd.2) 1640 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:19.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:19.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: T04:36:18.209625+0000 osd.4 (osd. 2021-02-10T04:36:19.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: 4) 851 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:19.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:19.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: 02-10T04:36:18.209631+0000 osd.4 2021-02-10T04:36:19.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: (osd.4) 852 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:19.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: cluster 2021-02-10T04:36:19.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: 2021-02-10T04:36: 2021-02-10T04:36:19.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: 18.209637+0000 osd.4 (osd.4) 853 : cluster 2021-02-10T04:36:19.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:19.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:19.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: -02-10T04:36:18.209643+0000 2021-02-10T04:36:19.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: osd.4 (osd.4) 854 : cluster 2021-02-10T04:36:19.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:19.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: cluster 2021-02-10T04:36:19.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: 2021-02-10T04:36:18.340715+0000 mon.a 2021-02-10T04:36:19.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: (mon.0) 1397 : cluster 2021-02-10T04:36:19.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: [WRN] Health check update: 14 slow ops, oldest one blocked for 407 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:36:19.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: cluster 2021-02-10T04:36:19.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: 2021-02-10T04:36:18. 2021-02-10T04:36:19.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: 358521+0000 mon.a (mon.0) 2021-02-10T04:36:19.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: 1398 : cluster [DBG] osdmap e561: 8 total, 8 up, 8 in 2021-02-10T04:36:19.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:36:19.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: 10T04:36:18.583380+0000 mon.b (mon.2) 35 : 2021-02-10T04:36:19.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: audit [INF] from='client.? 172.21.15.85:0/658408887' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T04:36:19.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:36:19.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: 04:36:18.583815+0000 mon.a (mon.0) 1399 : audit 2021-02-10T04:36:19.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T04:36:19.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: audit 2021-02-10T04:36:18.628977 2021-02-10T04:36:19.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: +0000 mon.b (mon.2) 36 : audit [DBG] from='client.? 172.21.15.85:0/1260347953' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:36:19.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: 2021-02-10T04:36:19.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: audit 2021-02-10T04:36:18. 2021-02-10T04:36:19.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: 818597+0000 mon.b (mon.2) 37 : audit [INF] 2021-02-10T04:36:19.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/3166515971' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:36:19.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: audit 2021-02-10T04:36:18. 2021-02-10T04:36:19.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: 819055+0000 mon.a (mon.0) 1400 : audit [INF] 2021-02-10T04:36:19.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:19 smithi085 conmon[35961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:36:19.883 INFO:tasks.rados.rados.0.smithi085.stdout:2147: finishing write tid 1 to smithi08575397-37 2021-02-10T04:36:19.883 INFO:tasks.rados.rados.0.smithi085.stdout:2147: finishing write tid 2 to smithi08575397-37 2021-02-10T04:36:19.884 INFO:tasks.rados.rados.0.smithi085.stdout:2148: done (15 left) 2021-02-10T04:36:19.884 INFO:tasks.rados.rados.0.smithi085.stdout:2149: done (14 left) 2021-02-10T04:36:19.884 INFO:tasks.rados.rados.0.smithi085.stdout:2150: done (13 left) 2021-02-10T04:36:19.884 INFO:tasks.rados.rados.0.smithi085.stdout:2155: write oid 38 current snap is 241 2021-02-10T04:36:19.884 INFO:tasks.rados.rados.0.smithi085.stdout:2147: finishing write tid 3 to smithi08575397-37 2021-02-10T04:36:19.884 INFO:tasks.rados.rados.0.smithi085.stdout:2155: seq_num 758 ranges {739706=557689,1806645=467255} 2021-02-10T04:36:19.891 INFO:tasks.rados.rados.0.smithi085.stdout:2155: writing smithi08575397-38 from 739706 to 1297395 tid 1 2021-02-10T04:36:19.899 INFO:tasks.rados.rados.0.smithi085.stdout:2155: writing smithi08575397-38 from 1806645 to 2273900 tid 2 2021-02-10T04:36:19.899 INFO:tasks.rados.rados.0.smithi085.stdout:2156: copy_from oid 8 from oid 10 current snap is 241 2021-02-10T04:36:19.899 INFO:tasks.rados.rados.0.smithi085.stdout:2147: finishing write tid 4 to smithi08575397-37 2021-02-10T04:36:19.899 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 1157 (ObjNum 754 snap 241 seq_num 754) dirty exists 2021-02-10T04:36:19.899 INFO:tasks.rados.rados.0.smithi085.stdout:2147: left oid 37 (ObjNum 754 snap 241 seq_num 754) 2021-02-10T04:36:19.899 INFO:tasks.rados.rados.0.smithi085.stdout:2151: finishing write tid 1 to smithi08575397-3 2021-02-10T04:36:19.899 INFO:tasks.rados.rados.0.smithi085.stdout:2151: finishing write tid 2 to smithi08575397-3 2021-02-10T04:36:19.899 INFO:tasks.rados.rados.0.smithi085.stdout:2151: finishing write tid 3 to smithi08575397-3 2021-02-10T04:36:19.899 INFO:tasks.rados.rados.0.smithi085.stdout:2151: finishing write tid 4 to smithi08575397-3 2021-02-10T04:36:19.899 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 784 (ObjNum 756 snap 241 seq_num 756) dirty exists 2021-02-10T04:36:19.899 INFO:tasks.rados.rados.0.smithi085.stdout:2151: left oid 3 (ObjNum 756 snap 241 seq_num 756) 2021-02-10T04:36:19.899 INFO:tasks.rados.rados.0.smithi085.stdout:2154: finishing rollback tid 0 to smithi08575397-26 2021-02-10T04:36:19.900 INFO:tasks.rados.rados.0.smithi085.stdout:2154: finishing rollback tid 1 to smithi08575397-26 2021-02-10T04:36:19.900 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 750 (ObjNum 82684320 snap 22030 seq_num 83146048) dirty dne 2021-02-10T04:36:19.900 INFO:tasks.rados.rados.0.smithi085.stdout:2153: finishing write tid 1 to smithi08575397-40 2021-02-10T04:36:19.900 INFO:tasks.rados.rados.0.smithi085.stdout:2147: done (14 left) 2021-02-10T04:36:19.900 INFO:tasks.rados.rados.0.smithi085.stdout:2151: done (13 left) 2021-02-10T04:36:19.900 INFO:tasks.rados.rados.0.smithi085.stdout:2154: done (12 left) 2021-02-10T04:36:19.900 INFO:tasks.rados.rados.0.smithi085.stdout:2157: snap_create 2021-02-10T04:36:19.900 INFO:tasks.rados.rados.0.smithi085.stdout:2153: finishing write tid 2 to smithi08575397-40 2021-02-10T04:36:19.900 INFO:tasks.rados.rados.0.smithi085.stdout:2153: finishing write tid 3 to smithi08575397-40 2021-02-10T04:36:19.900 INFO:tasks.rados.rados.0.smithi085.stdout:2153: finishing write tid 4 to smithi08575397-40 2021-02-10T04:36:19.900 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 1161 (ObjNum 757 snap 241 seq_num 757) dirty exists 2021-02-10T04:36:19.900 INFO:tasks.rados.rados.0.smithi085.stdout:2153: left oid 40 (ObjNum 757 snap 241 seq_num 757) 2021-02-10T04:36:19.900 INFO:tasks.rados.rados.0.smithi085.stdout:2155: finishing write tid 1 to smithi08575397-38 2021-02-10T04:36:19.901 INFO:tasks.rados.rados.0.smithi085.stdout:2155: finishing write tid 2 to smithi08575397-38 2021-02-10T04:36:19.902 INFO:tasks.rados.rados.0.smithi085.stdout:2155: finishing write tid 3 to smithi08575397-38 2021-02-10T04:36:19.902 INFO:tasks.rados.rados.0.smithi085.stdout:2155: finishing write tid 4 to smithi08575397-38 2021-02-10T04:36:19.902 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 1048 (ObjNum 758 snap 241 seq_num 758) dirty exists 2021-02-10T04:36:19.902 INFO:tasks.rados.rados.0.smithi085.stdout:2155: left oid 38 (ObjNum 758 snap 241 seq_num 758) 2021-02-10T04:36:19.912 INFO:tasks.rados.rados.0.smithi085.stdout:2156: finishing copy_from to smithi08575397-8 2021-02-10T04:36:19.912 INFO:tasks.rados.rados.0.smithi085.stdout:2156: got expected ENOENT (src dne) 2021-02-10T04:36:19.912 INFO:tasks.rados.rados.0.smithi085.stdout:2156: finishing copy_from racing read to smithi08575397-8 2021-02-10T04:36:20.065 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:36:20.106 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:36:20.252 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:20 smithi164 conmon[33677]: debug 2021-02-10T04:36:20.131+0000 7fbea992c700 -1 osd.4 pg_epoch: 562 pg[6.1a( empty local-lis/les=59/60 n=0 ec=59/59 lis/c=59/59 les/c/f=60/60/0 sis=59) [4,5,1] r=0 lpr=59 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.1a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x562760af4580) 2021-02-10T04:36:20.270 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.5 dump_historic_ops 2021-02-10T04:36:20.358 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:20 smithi085 conmon[29319]: cluster 2021-02-10T04:36:19. 2021-02-10T04:36:20.399 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:20 smithi085 conmon[29319]: 353803+0000 mon.a (mon.0) 1401 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:36:20.399 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:20 smithi085 conmon[29319]: audit 2021-02-10T04:36:19.362087+0000 mon.a (mon.0) 1402 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2021-02-10T04:36:20.399 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:20 smithi085 conmon[29319]: audit 2021-02-10T04:36:19.362207+0000 mon.a (mon.0) 1403 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:36:20.399 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:20 smithi085 conmon[29319]: cluster 2021-02-10T04:36:19.362274+0000 mon.a (mon.0) 1404 : cluster [DBG] osdmap e562: 8 total, 8 up, 8 in 2021-02-10T04:36:20.399 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:20 smithi085 conmon[29319]: cluster 2021-02-10T04:36:20.117091+0000 mgr.y (mgr.14140) 629 : cluster [DBG] pgmap v993: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 411 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.3 MiB/s wr, 10 op/s 2021-02-10T04:36:20.399 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:20 smithi085 conmon[35961]: cluster 2021-02-10T04:36:19. 2021-02-10T04:36:20.399 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:20 smithi085 conmon[35961]: 353803+0000 mon.a (mon.0) 1401 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:36:20.399 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:20 smithi085 conmon[35961]: audit 2021-02-10T04:36:19.362087+0000 mon.a (mon.0) 1402 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2021-02-10T04:36:20.400 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:20 smithi085 conmon[35961]: audit 2021-02-10T04:36:19.362207+0000 mon.a (mon.0) 1403 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:36:20.400 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:20 smithi085 conmon[35961]: cluster 2021-02-10T04:36:19.362274+0000 mon.a (mon.0) 1404 : cluster [DBG] osdmap e562: 8 total, 8 up, 8 in 2021-02-10T04:36:20.400 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:20 smithi085 conmon[35961]: cluster 2021-02-10T04:36:20.117091+0000 mgr.y (mgr.14140) 629 : cluster [DBG] pgmap v993: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 411 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.3 MiB/s wr, 10 op/s 2021-02-10T04:36:20.400 INFO:tasks.rados.rados.0.smithi085.stdout:2153: done (12 left) 2021-02-10T04:36:20.400 INFO:tasks.rados.rados.0.smithi085.stdout:2155: done (11 left) 2021-02-10T04:36:20.401 INFO:tasks.rados.rados.0.smithi085.stdout:2156: done (10 left) 2021-02-10T04:36:20.401 INFO:tasks.rados.rados.0.smithi085.stdout:2157: done (9 left) 2021-02-10T04:36:20.401 INFO:tasks.rados.rados.0.smithi085.stdout:2158: rollback oid 38 current snap is 242 2021-02-10T04:36:20.401 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 38 to 190 2021-02-10T04:36:20.401 INFO:tasks.rados.rados.0.smithi085.stdout:2159: snap_remove snap 211 2021-02-10T04:36:20.556 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:36:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:20 smithi164 conmon[29288]: cluster 2021-02-10T04:36:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:20 smithi164 conmon[29288]: 2021-02-10T04:36:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:20 smithi164 conmon[29288]: 2021 2021-02-10T04:36:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:20 smithi164 conmon[29288]: -02-10T04:36:19. 2021-02-10T04:36:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:20 smithi164 conmon[29288]: 353803+0000 mon.a (mon.0) 1401 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:36:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:20 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:36:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:20 smithi164 conmon[29288]: 02-10T04:36:19.362087+0000 mon.a (mon.0 2021-02-10T04:36:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:20 smithi164 conmon[29288]: ) 1402 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2021-02-10T04:36:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:20 smithi164 conmon[29288]: audit 2021-02-10T04:36:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:20 smithi164 conmon[29288]: 2021-02-10T04:36:19.362207 2021-02-10T04:36:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:20 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1403 : audit [INF] 2021-02-10T04:36:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:20 smithi164 conmon[29288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:36:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:20 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:36:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:20 smithi164 conmon[29288]: :36:19.362274+0000 mon.a (mon. 2021-02-10T04:36:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:20 smithi164 conmon[29288]: 0) 1404 : cluster [DBG] osdmap e562: 8 total, 8 up, 8 in 2021-02-10T04:36:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:20 smithi164 conmon[29288]: cluster 2021-02-10T04:36:20.117091 2021-02-10T04:36:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:20 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 629 : cluster [DBG] pgmap v993: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 411 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.3 MiB/s wr, 10 op/s 2021-02-10T04:36:21.370 INFO:tasks.rados.rados.0.smithi085.stdout:2159: done (10 left) 2021-02-10T04:36:21.370 INFO:tasks.rados.rados.0.smithi085.stdout:2160: snap_remove snap 236 2021-02-10T04:36:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:21 smithi164 conmon[29288]: cluster 2021-02-10T04:36:20.368944 2021-02-10T04:36:22.684 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:21 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e563: 8 total, 8 up, 8 in 2021-02-10T04:36:22.685 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:36:21 smithi085 conmon[52836]: debug 2021-02-10T04:36:21.646+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:36:22.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:21 smithi085 conmon[29319]: cluster 2021-02-10T04:36:22.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:21 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:36:22.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:21 smithi085 conmon[29319]: 04:36:20.368944+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e563: 8 total, 8 up, 8 in 2021-02-10T04:36:22.685 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:21 smithi085 conmon[35961]: cluster 2021-02-10T04:36:20. 2021-02-10T04:36:22.686 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:21 smithi085 conmon[35961]: 368944+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e563: 8 total, 8 up, 8 in 2021-02-10T04:36:22.686 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:36:22.686 INFO:tasks.rados.rados.0.smithi085.stdout:2160: done (10 left) 2021-02-10T04:36:22.686 INFO:tasks.rados.rados.0.smithi085.stdout:2161: snap_create 2021-02-10T04:36:22.686 INFO:tasks.rados.rados.0.smithi085.stdout:2158: finishing rollback tid 0 to smithi08575397-38 2021-02-10T04:36:22.686 INFO:tasks.rados.rados.0.smithi085.stdout:2158: finishing rollback tid 1 to smithi08575397-38 2021-02-10T04:36:22.686 INFO:tasks.rados.rados.0.smithi085.stdout:2158: finishing rollback tid 2 to smithi08575397-38 2021-02-10T04:36:22.686 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 1051 (ObjNum 616 snap 186 seq_num 616) dirty exists 2021-02-10T04:36:22.686 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:22 smithi164 conmon[29288]: cluster 2021-02-10T04:36:20.883463+0000 osd.6 (osd 2021-02-10T04:36:22.686 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:22 smithi164 conmon[29288]: .6) 65 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:36:22.687 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:22 smithi164 conmon[29288]: cluster 2021-02-10T04:36:20.883713+0000 osd.6 (osd.6) 66 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:36:22.687 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:22 smithi164 conmon[29288]: cluster 2021-02-10T04:36:21.369374+0000 2021-02-10T04:36:22.687 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:22 smithi164 conmon[29288]: mon.a (mon.0) 1406 : cluster [DBG] osdmap e564: 8 total, 8 up, 8 in 2021-02-10T04:36:22.687 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:22 smithi164 conmon[29288]: 2021-02-10T04:36:22.687 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:22 smithi164 conmon[29288]: cluster 2021-02-10T04:36: 2021-02-10T04:36:22.687 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:22 smithi164 conmon[29288]: 22.117642+0000 mgr.y (mgr.14140) 630 2021-02-10T04:36:22.687 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:22 smithi164 conmon[29288]: : cluster [DBG] pgmap v996: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 398 MiB data, 1022 MiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.2 MiB/s wr, 14 op/s 2021-02-10T04:36:22.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:22 smithi085 conmon[35961]: cluster 2021-02-10T04:36:22.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:22 smithi085 conmon[35961]: 2021-02-10T04:36 2021-02-10T04:36:22.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:22 smithi085 conmon[35961]: :20.883463+0000 osd.6 (osd.6) 65 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:36:22.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:22 smithi085 conmon[35961]: cluster 2021-02-10T04:36:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:22 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:36:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:22 smithi085 conmon[35961]: 10T04:36:20.883713+0000 osd.6 (osd. 2021-02-10T04:36:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:22 smithi085 conmon[35961]: 6) 66 : cluster [DBG] 2021-02-10T04:36:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:22 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:36:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:22 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:22 smithi085 conmon[35961]: 02-10T04:36:21.369374+0000 2021-02-10T04:36:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:22 smithi085 conmon[35961]: mon.a (mon.0) 1406 : 2021-02-10T04:36:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:22 smithi085 conmon[35961]: cluster [DBG] osdmap e564: 8 total, 8 up, 8 in 2021-02-10T04:36:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:22 smithi085 conmon[35961]: cluster 2021-02-10T04:36:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:22 smithi085 conmon[35961]: 2021-02-10T04:36 2021-02-10T04:36:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:22 smithi085 conmon[35961]: :22.117642+0000 mgr.y (mgr.14140) 630 : 2021-02-10T04:36:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:22 smithi085 conmon[35961]: cluster [DBG] pgmap v996: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 398 MiB data, 1022 MiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.2 MiB/s wr, 14 op/s 2021-02-10T04:36:22.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:22 smithi085 conmon[29319]: cluster 2021-02-10T04:36:20.883463 2021-02-10T04:36:22.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:22 smithi085 conmon[29319]: +0000 osd.6 (osd.6) 65 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:36:22.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:22 smithi085 conmon[29319]: cluster 2021-02-10T04:36: 2021-02-10T04:36:22.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:22 smithi085 conmon[29319]: 20.883713+0000 osd.6 (osd.6) 66 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:36:22.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:22 smithi085 conmon[29319]: 2021-02-10T04:36:22.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:22 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:22.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:22 smithi085 conmon[29319]: -02-10T04:36:21.369374+0000 2021-02-10T04:36:22.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:22 smithi085 conmon[29319]: mon.a (mon.0) 1406 : cluster 2021-02-10T04:36:22.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:22 smithi085 conmon[29319]: [DBG] osdmap e564: 8 total, 8 up, 8 in 2021-02-10T04:36:22.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:22 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:22.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:22 smithi085 conmon[29319]: -02-10T04:36:22.117642+0000 2021-02-10T04:36:22.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:22 smithi085 conmon[29319]: mgr.y (mgr.14140) 630 : cluster [DBG] pgmap v996: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 398 MiB data, 1022 MiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.2 MiB/s wr, 14 op/s 2021-02-10T04:36:23.182 INFO:tasks.rados.rados.0.smithi085.stdout:2158: done (10 left) 2021-02-10T04:36:23.182 INFO:tasks.rados.rados.0.smithi085.stdout:2161: done (9 left) 2021-02-10T04:36:23.182 INFO:tasks.rados.rados.0.smithi085.stdout:2162: write oid 7 current snap is 243 2021-02-10T04:36:23.182 INFO:tasks.rados.rados.0.smithi085.stdout:2162: seq_num 759 ranges {756513=632938,1865127=727266} 2021-02-10T04:36:23.185 INFO:tasks.rados.rados.0.smithi085.stdout:2162: writing smithi08575397-7 from 756513 to 1389451 tid 1 2021-02-10T04:36:23.194 INFO:tasks.rados.rados.0.smithi085.stdout:2162: writing smithi08575397-7 from 1865127 to 2592393 tid 2 2021-02-10T04:36:23.194 INFO:tasks.rados.rados.0.smithi085.stdout:2163: copy_from oid 31 from oid 13 current snap is 243 2021-02-10T04:36:23.194 INFO:tasks.rados.rados.0.smithi085.stdout:2162: finishing write tid 1 to smithi08575397-7 2021-02-10T04:36:23.194 INFO:tasks.rados.rados.0.smithi085.stdout:2164: read oid 40 snap 242 2021-02-10T04:36:23.194 INFO:tasks.rados.rados.0.smithi085.stdout:2164: expect (ObjNum 757 snap 241 seq_num 757) 2021-02-10T04:36:23.194 INFO:tasks.rados.rados.0.smithi085.stdout:2165: write oid 44 current snap is 243 2021-02-10T04:36:23.194 INFO:tasks.rados.rados.0.smithi085.stdout:2165: seq_num 760 ranges {773320=708187,1923609=470924} 2021-02-10T04:36:23.203 INFO:tasks.rados.rados.0.smithi085.stdout:2165: writing smithi08575397-44 from 773320 to 1481507 tid 1 2021-02-10T04:36:23.209 INFO:tasks.rados.rados.0.smithi085.stdout:2165: writing smithi08575397-44 from 1923609 to 2394533 tid 2 2021-02-10T04:36:23.209 INFO:tasks.rados.rados.0.smithi085.stdout:2166: snap_create 2021-02-10T04:36:23.210 INFO:tasks.rados.rados.0.smithi085.stdout:2162: finishing write tid 2 to smithi08575397-7 2021-02-10T04:36:23.210 INFO:tasks.rados.rados.0.smithi085.stdout:2162: finishing write tid 3 to smithi08575397-7 2021-02-10T04:36:23.210 INFO:tasks.rados.rados.0.smithi085.stdout:2162: finishing write tid 4 to smithi08575397-7 2021-02-10T04:36:23.210 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 792 (ObjNum 759 snap 243 seq_num 759) dirty exists 2021-02-10T04:36:23.210 INFO:tasks.rados.rados.0.smithi085.stdout:2162: left oid 7 (ObjNum 759 snap 243 seq_num 759) 2021-02-10T04:36:23.210 INFO:tasks.rados.rados.0.smithi085.stdout:2164: expect (ObjNum 757 snap 241 seq_num 757) 2021-02-10T04:36:23.384 INFO:tasks.rados.rados.0.smithi085.stdout:2165: finishing write tid 1 to smithi08575397-44 2021-02-10T04:36:23.385 INFO:tasks.rados.rados.0.smithi085.stdout:2163: finishing copy_from to smithi08575397-31 2021-02-10T04:36:23.385 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 748 (ObjNum 742 snap 238 seq_num 742) dirty exists 2021-02-10T04:36:23.385 INFO:tasks.rados.rados.0.smithi085.stdout:2163: finishing copy_from racing read to smithi08575397-31 2021-02-10T04:36:23.385 INFO:tasks.rados.rados.0.smithi085.stdout:2165: finishing write tid 2 to smithi08575397-44 2021-02-10T04:36:23.385 INFO:tasks.rados.rados.0.smithi085.stdout:2165: finishing write tid 3 to smithi08575397-44 2021-02-10T04:36:23.385 INFO:tasks.rados.rados.0.smithi085.stdout:2165: finishing write tid 4 to smithi08575397-44 2021-02-10T04:36:23.385 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 1083 (ObjNum 760 snap 243 seq_num 760) dirty exists 2021-02-10T04:36:23.385 INFO:tasks.rados.rados.0.smithi085.stdout:2165: left oid 44 (ObjNum 760 snap 243 seq_num 760) 2021-02-10T04:36:23.386 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:23 smithi164 conmon[33677]: debug 2021-02-10T04:36:23.237+0000 7fbea5924700 -1 osd.4 pg_epoch: 566 pg[6.1a( empty local-lis/les=59/60 n=0 ec=59/59 lis/c=59/59 les/c/f=60/60/0 sis=59) [4,5,1] r=0 lpr=59 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.1a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x5627631b9e40) 2021-02-10T04:36:23.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:23 smithi164 conmon[29288]: cluster 2021-02-10T04:36:22.378001+0000 mon.a (mon.0) 1407 : 2021-02-10T04:36:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:23 smithi164 conmon[29288]: cluster [DBG] osdmap e565: 8 total, 8 up, 8 in 2021-02-10T04:36:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:23 smithi164 conmon[29288]: cluster 2021-02-10T04:36:22.379060+0000 mon.a (mon.0) 1408 : cluster [INF] Health check cleared: SLOW_OPS (was: 14 slow ops, oldest one blocked for 407 sec, daemons [osd.2,osd.4] have slow ops.) 2021-02-10T04:36:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:23 smithi164 conmon[29288]: cluster 2021-02-10T04:36:23.176709+0000 mon.a (mon.0) 1409 : cluster [DBG] osdmap e566: 8 total, 8 up, 8 in 2021-02-10T04:36:23.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:23 smithi085 conmon[29319]: cluster 2021-02-10T04:36:23.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:23 smithi085 conmon[29319]: 2021-02-10T04:36:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:23 smithi085 conmon[29319]: 2021 2021-02-10T04:36:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:23 smithi085 conmon[29319]: - 2021-02-10T04:36:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:23 smithi085 conmon[29319]: 02 2021-02-10T04:36:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:23 smithi085 conmon[29319]: -10T04:36:22.378001 2021-02-10T04:36:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:23 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1407 : cluster [DBG] osdmap e565: 8 total, 8 up, 8 in 2021-02-10T04:36:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:23 smithi085 conmon[29319]: cluster 2021-02-10T04:36:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:23 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:36:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:23 smithi085 conmon[29319]: :36:22.379060+0000 mon.a (mon.0) 2021-02-10T04:36:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:23 smithi085 conmon[29319]: 1408 : cluster [INF] Health check cleared: SLOW_OPS (was: 14 slow ops, oldest one blocked for 407 sec, daemons [osd.2,osd.4] have slow ops.) 2021-02-10T04:36:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:23 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:36:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:23 smithi085 conmon[29319]: :36:23.176709+0000 mon.a (mon.0) 2021-02-10T04:36:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:23 smithi085 conmon[29319]: 1409 : cluster [DBG] osdmap e566: 8 total, 8 up, 8 in 2021-02-10T04:36:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:23 smithi085 conmon[35961]: cluster 2021-02-10T04:36:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:23 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:36:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:23 smithi085 conmon[35961]: 04:36:22.378001+0000 mon.a (mon.0) 1407 : cluster [DBG] osdmap e565: 8 total, 8 up, 8 in 2021-02-10T04:36:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:23 smithi085 conmon[35961]: cluster 2021-02-10T04:36:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:23 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:36:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:23 smithi085 conmon[35961]: 36:22.379060+0000 mon.a (mon.0) 1408 : cluster [INF] Health check cleared: SLOW_OPS (was: 14 slow ops, oldest one blocked for 407 sec, daemons [osd.2,osd.4] have slow ops.) 2021-02-10T04:36:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:23 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:36:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:23 smithi085 conmon[35961]: :36:23.176709+0000 mon.a (mon.0) 2021-02-10T04:36:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:23 smithi085 conmon[35961]: 1409 : cluster [DBG] osdmap e566: 8 total, 8 up, 8 in 2021-02-10T04:36:24.187 INFO:tasks.rados.rados.0.smithi085.stdout:2162: done (13 left) 2021-02-10T04:36:24.187 INFO:tasks.rados.rados.0.smithi085.stdout:2163: done (12 left) 2021-02-10T04:36:24.187 INFO:tasks.rados.rados.0.smithi085.stdout:2164: done (11 left) 2021-02-10T04:36:24.187 INFO:tasks.rados.rados.0.smithi085.stdout:2165: done (10 left) 2021-02-10T04:36:24.187 INFO:tasks.rados.rados.0.smithi085.stdout:2166: done (9 left) 2021-02-10T04:36:24.187 INFO:tasks.rados.rados.0.smithi085.stdout:2167: rollback oid 23 current snap is 244 2021-02-10T04:36:24.187 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 23 to 234 2021-02-10T04:36:24.187 INFO:tasks.rados.rados.0.smithi085.stdout:2168: read oid 28 snap -1 2021-02-10T04:36:24.187 INFO:tasks.rados.rados.0.smithi085.stdout:2168: expect (ObjNum 651 snap 197 seq_num 651) 2021-02-10T04:36:24.187 INFO:tasks.rados.rados.0.smithi085.stdout:2169: write oid 37 current snap is 244 2021-02-10T04:36:24.188 INFO:tasks.rados.rados.0.smithi085.stdout:2169: seq_num 761 ranges {790127=499789,1698444=730935} 2021-02-10T04:36:24.194 INFO:tasks.rados.rados.0.smithi085.stdout:2169: writing smithi08575397-37 from 790127 to 1289916 tid 1 2021-02-10T04:36:24.201 INFO:tasks.rados.rados.0.smithi085.stdout:2169: writing smithi08575397-37 from 1698444 to 2429379 tid 2 2021-02-10T04:36:24.201 INFO:tasks.rados.rados.0.smithi085.stdout:2170: read oid 43 snap -1 2021-02-10T04:36:24.202 INFO:tasks.rados.rados.0.smithi085.stdout:2170: expect (ObjNum 748 snap 240 seq_num 748) 2021-02-10T04:36:24.202 INFO:tasks.rados.rados.0.smithi085.stdout:2168: expect (ObjNum 651 snap 197 seq_num 651) 2021-02-10T04:36:24.202 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:36:24.202 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:36:24.418 INFO:tasks.rados.rados.0.smithi085.stdout:2168: done (12 left) 2021-02-10T04:36:24.418 INFO:tasks.rados.rados.0.smithi085.stdout:2171: snap_remove snap 243 2021-02-10T04:36:24.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:24 smithi164 conmon[29288]: cluster 2021-02-10T04:36:24.118187+0000 mgr.y (mgr.14140) 631 : cluster [DBG] pgmap v999: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 392 MiB data, 1017 MiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 1.9 MiB/s wr, 11 op/s 2021-02-10T04:36:24.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:24 smithi164 conmon[29288]: cluster 2021-02-10T04:36:24.186891+0000 mon.a (mon.0) 1410 : cluster [DBG] osdmap e567: 8 total, 8 up, 8 in 2021-02-10T04:36:24.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:24 smithi085 conmon[35961]: cluster 2021-02-10T04:36:24.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:24 smithi085 conmon[35961]: 2021-02-10T04:36:24.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:24 smithi085 conmon[35961]: 2021 2021-02-10T04:36:24.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:24 smithi085 conmon[35961]: -02-10T04: 2021-02-10T04:36:24.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:24 smithi085 conmon[35961]: 36:24.118187+0000 mgr.y (mgr.14140) 631 : cluster [DBG] pgmap v999: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 392 MiB data, 1017 MiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 1.9 MiB/s wr, 11 op/s 2021-02-10T04:36:24.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:24 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:24.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:24 smithi085 conmon[35961]: 02-10T04:36:24.186891+0000 mon.a (mon. 2021-02-10T04:36:24.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:24 smithi085 conmon[35961]: 0) 1410 : cluster [DBG] osdmap e567: 8 total, 8 up, 8 in 2021-02-10T04:36:24.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:24 smithi085 conmon[29319]: cluster 2021-02-10T04:36 2021-02-10T04:36:24.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:24 smithi085 conmon[29319]: :24.118187+0000 mgr.y (mgr.14140) 631 : cluster [DBG] pgmap v999: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 392 MiB data, 1017 MiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 1.9 MiB/s wr, 11 op/s 2021-02-10T04:36:24.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:24 smithi085 conmon[29319]: cluster 2021-02-10T04:36:24.186891+0000 mon.a (mon.0) 1410 : cluster [DBG] osdmap e567: 8 total, 8 up, 8 in 2021-02-10T04:36:25.403 INFO:tasks.rados.rados.0.smithi085.stdout:2171: done (12 left) 2021-02-10T04:36:25.403 INFO:tasks.rados.rados.0.smithi085.stdout:2172: read oid 12 snap -1 2021-02-10T04:36:25.403 INFO:tasks.rados.rados.0.smithi085.stdout:2172: expect (ObjNum 755 snap 241 seq_num 755) 2021-02-10T04:36:25.403 INFO:tasks.rados.rados.0.smithi085.stdout:2173: write (excl) oid 5 current snap is 244 2021-02-10T04:36:25.403 INFO:tasks.rados.rados.0.smithi085.stdout:2173: seq_num 762 ranges {406934=575038,1640573=474593,2642791=164143} 2021-02-10T04:36:25.408 INFO:tasks.rados.rados.0.smithi085.stdout:2173: writing smithi08575397-5 from 406934 to 981972 tid 1 2021-02-10T04:36:25.415 INFO:tasks.rados.rados.0.smithi085.stdout:2173: writing smithi08575397-5 from 1640573 to 2115166 tid 2 2021-02-10T04:36:25.419 INFO:tasks.rados.rados.0.smithi085.stdout:2173: writing smithi08575397-5 from 2642791 to 2806934 tid 3 2021-02-10T04:36:25.419 INFO:tasks.rados.rados.0.smithi085.stdout:2174: snap_create 2021-02-10T04:36:25.420 INFO:tasks.rados.rados.0.smithi085.stdout:2169: finishing write tid 1 to smithi08575397-37 2021-02-10T04:36:25.420 INFO:tasks.rados.rados.0.smithi085.stdout:2167: finishing rollback tid 0 to smithi08575397-23 2021-02-10T04:36:25.420 INFO:tasks.rados.rados.0.smithi085.stdout:2167: finishing rollback tid 1 to smithi08575397-23 2021-02-10T04:36:25.420 INFO:tasks.rados.rados.0.smithi085.stdout:2167: finishing rollback tid 2 to smithi08575397-23 2021-02-10T04:36:25.420 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 690 (ObjNum 722 snap 226 seq_num 722) dirty exists 2021-02-10T04:36:25.420 INFO:tasks.rados.rados.0.smithi085.stdout:2170: expect (ObjNum 748 snap 240 seq_num 748) 2021-02-10T04:36:25.547 INFO:tasks.rados.rados.0.smithi085.stdout:2169: finishing write tid 2 to smithi08575397-37 2021-02-10T04:36:25.548 INFO:tasks.rados.rados.0.smithi085.stdout:2169: finishing write tid 3 to smithi08575397-37 2021-02-10T04:36:25.548 INFO:tasks.rados.rados.0.smithi085.stdout:2169: finishing write tid 4 to smithi08575397-37 2021-02-10T04:36:25.548 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 1173 (ObjNum 761 snap 244 seq_num 761) dirty exists 2021-02-10T04:36:25.548 INFO:tasks.rados.rados.0.smithi085.stdout:2169: left oid 37 (ObjNum 761 snap 244 seq_num 761) 2021-02-10T04:36:25.548 INFO:tasks.rados.rados.0.smithi085.stdout:2172: expect (ObjNum 755 snap 241 seq_num 755) 2021-02-10T04:36:25.706 INFO:tasks.rados.rados.0.smithi085.stdout:2173: finishing write tid 1 to smithi08575397-5 2021-02-10T04:36:25.706 INFO:tasks.rados.rados.0.smithi085.stdout:2173: finishing write tid 2 to smithi08575397-5 2021-02-10T04:36:25.706 INFO:tasks.rados.rados.0.smithi085.stdout:2173: finishing write tid 3 to smithi08575397-5 2021-02-10T04:36:25.706 INFO:tasks.rados.rados.0.smithi085.stdout:2173: finishing write tid 4 to smithi08575397-5 2021-02-10T04:36:25.706 INFO:tasks.rados.rados.0.smithi085.stdout:2173: finishing write tid 5 to smithi08575397-5 2021-02-10T04:36:25.706 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 755 (ObjNum 762 snap 244 seq_num 762) dirty exists 2021-02-10T04:36:25.706 INFO:tasks.rados.rados.0.smithi085.stdout:2173: left oid 5 (ObjNum 762 snap 244 seq_num 762) 2021-02-10T04:36:26.307 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.6 dump_ops_in_flight 2021-02-10T04:36:26.411 INFO:tasks.rados.rados.0.smithi085.stdout:2167: done (14 left) 2021-02-10T04:36:26.411 INFO:tasks.rados.rados.0.smithi085.stdout:2169: done (13 left) 2021-02-10T04:36:26.411 INFO:tasks.rados.rados.0.smithi085.stdout:2170: done (12 left) 2021-02-10T04:36:26.411 INFO:tasks.rados.rados.0.smithi085.stdout:2172: done (11 left) 2021-02-10T04:36:26.411 INFO:tasks.rados.rados.0.smithi085.stdout:2173: done (10 left) 2021-02-10T04:36:26.411 INFO:tasks.rados.rados.0.smithi085.stdout:2174: done (9 left) 2021-02-10T04:36:26.411 INFO:tasks.rados.rados.0.smithi085.stdout:2175: snap_remove snap 215 2021-02-10T04:36:26.572 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:36:26.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:26 smithi164 conmon[29288]: cluster 2021-02-10T04:36:25.402210+0000 mon.a (mon.0) 1411 2021-02-10T04:36:26.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:26 smithi164 conmon[29288]: : cluster [DBG] osdmap e568: 8 total, 8 up, 8 in 2021-02-10T04:36:26.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:26 smithi164 conmon[29288]: cluster 2021-02-10T04:36:26.118848+0000 mgr.y (mgr.14140) 632 : cluster [DBG] pgmap v1002: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 400 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2021-02-10T04:36:26.682 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:36:26.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:26 smithi085 conmon[29319]: cluster 2021-02-10T04:36:25.402210+0000 mon.a ( 2021-02-10T04:36:26.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:26 smithi085 conmon[29319]: mon.0) 1411 : cluster [DBG] osdmap e568: 8 total, 8 up, 8 in 2021-02-10T04:36:26.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:26 smithi085 conmon[29319]: cluster 2021-02-10T04:36:26.118848+0000 mgr.y (mgr.14140) 632 : cluster [DBG] pgmap v1002: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 400 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2021-02-10T04:36:26.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:26 smithi085 conmon[35961]: cluster 2021-02-10T04:36:25. 2021-02-10T04:36:26.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:26 smithi085 conmon[35961]: 402210+0000 mon.a (mon.0) 1411 : cluster [DBG] osdmap e568: 8 total, 8 up, 8 in 2021-02-10T04:36:26.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:26 smithi085 conmon[35961]: cluster 2021-02-10T04:36:26.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:26 smithi085 conmon[35961]: 2021-02-10T04:36:26.118848+0000 mgr.y (mgr.14140) 632 2021-02-10T04:36:26.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:26 smithi085 conmon[35961]: : cluster [DBG] pgmap v1002: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 400 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2021-02-10T04:36:26.707 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:36:27.410 INFO:tasks.rados.rados.0.smithi085.stdout:2175: done (9 left) 2021-02-10T04:36:27.410 INFO:tasks.rados.rados.0.smithi085.stdout:2176: read oid 43 snap -1 2021-02-10T04:36:27.410 INFO:tasks.rados.rados.0.smithi085.stdout:2176: expect (ObjNum 748 snap 240 seq_num 748) 2021-02-10T04:36:27.410 INFO:tasks.rados.rados.0.smithi085.stdout:2177: snap_create 2021-02-10T04:36:27.417 INFO:tasks.rados.rados.0.smithi085.stdout:2176: expect (ObjNum 748 snap 240 seq_num 748) 2021-02-10T04:36:27.579 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:36:27.617 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:36:27 smithi085 conmon[52836]: debug 2021-02-10T04:36:27.615+0000 7fba75fba700 -1 osd.3 pg_epoch: 570 pg[4.10( v 538'16 (0'0,538'16] local-lis/les=71/72 n=10 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,5,6] r=0 lpr=71 crt=538'16 lcod 538'15 mlcod 538'15 active+clean] scrubber pg(4.10) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6 (0x55fec096f600) 2021-02-10T04:36:27.617 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:27 smithi085 conmon[29319]: cluster 2021-02-10T04:36:26.410274+0000 mon.a (mon.0) 1412 : cluster [DBG] 2021-02-10T04:36:27.617 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:27 smithi085 conmon[29319]: osdmap e569: 8 total, 8 up, 8 in 2021-02-10T04:36:27.617 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:36:27 smithi085 conmon[45443]: debug 2021-02-10T04:36:27.242+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:36:27.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:27 smithi085 conmon[35961]: cluster 2021-02-10T04:36:26.410274+0000 2021-02-10T04:36:27.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:27 smithi085 conmon[35961]: mon.a (mon.0) 1412 : cluster [DBG] osdmap e569: 8 total, 8 up, 8 in 2021-02-10T04:36:27.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:27 smithi164 conmon[29288]: cluster 2021-02-10T04:36:26.410274+0000 mon.a (mon. 2021-02-10T04:36:27.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:27 smithi164 conmon[29288]: 0) 1412 : cluster [DBG] osdmap e569: 8 total, 8 up, 8 in 2021-02-10T04:36:28.185 INFO:tasks.rados.rados.0.smithi085.stdout:2176: done (10 left) 2021-02-10T04:36:28.185 INFO:tasks.rados.rados.0.smithi085.stdout:2177: done (9 left) 2021-02-10T04:36:28.185 INFO:tasks.rados.rados.0.smithi085.stdout:2178: write (excl) oid 1 current snap is 246 2021-02-10T04:36:28.185 INFO:tasks.rados.rados.0.smithi085.stdout:2178: seq_num 763 ranges {423741=650287,1699055=618251} 2021-02-10T04:36:28.186 INFO:tasks.rados.rados.0.smithi085.stdout:2178: writing smithi08575397-1 from 423741 to 1074028 tid 1 2021-02-10T04:36:28.194 INFO:tasks.rados.rados.0.smithi085.stdout:2178: writing smithi08575397-1 from 1699055 to 2317306 tid 2 2021-02-10T04:36:28.195 INFO:tasks.rados.rados.0.smithi085.stdout:2179: read oid 35 snap -1 2021-02-10T04:36:28.195 INFO:tasks.rados.rados.0.smithi085.stdout:2179: expect (ObjNum 578 snap 176 seq_num 578) 2021-02-10T04:36:28.195 INFO:tasks.rados.rados.0.smithi085.stdout:2180: copy_from oid 40 from oid 23 current snap is 246 2021-02-10T04:36:28.195 INFO:tasks.rados.rados.0.smithi085.stdout:2181: snap_remove snap 203 2021-02-10T04:36:28.408 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:36:28 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:36:28.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:28 smithi085 conmon[29319]: cluster 2021-02-10T04:36:28.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:28 smithi085 conmon[29319]: 2021-02-10T04:36:28.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:28 smithi085 conmon[29319]: 2021-02-10T04:36: 2021-02-10T04:36:28.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:28 smithi085 conmon[29319]: 27.409459+0000 mon.a (mon.0) 1413 : cluster [DBG] osdmap e570: 8 total, 8 up, 8 in 2021-02-10T04:36:28.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:28 smithi085 conmon[29319]: cluster 2021-02-10T04:36:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:28 smithi085 conmon[29319]: 2021-02-10T04:36:28 2021-02-10T04:36:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:28 smithi085 conmon[29319]: .119467+0000 mgr.y (mgr.14140) 633 : 2021-02-10T04:36:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:28 smithi085 conmon[29319]: cluster [DBG] pgmap v1005: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 398 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 917 KiB/s wr, 11 op/s 2021-02-10T04:36:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:28 smithi085 conmon[29319]: audit 2021 2021-02-10T04:36:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:28 smithi085 conmon[29319]: -02-10T 2021-02-10T04:36:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:28 smithi085 conmon[29319]: 04:36:28.147035+0000 mgr.y (mgr.14140) 2021-02-10T04:36:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:28 smithi085 conmon[29319]: 634 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:36:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:28 smithi085 conmon[29319]: cluster 2021-02-10T04:36:28.178266 2021-02-10T04:36:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:28 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1414 : cluster [DBG] osdmap e571: 8 total, 8 up, 8 in 2021-02-10T04:36:28.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:28 smithi085 conmon[35961]: cluster 2021-02-10T04:36:27.409459 2021-02-10T04:36:28.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:28 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1413 : cluster [DBG] osdmap e570: 8 total, 8 up, 8 in 2021-02-10T04:36:28.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:28 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:28 smithi085 conmon[35961]: 02-10T04:36:28.119467+0000 mgr.y (mgr.14140) 633 : cluster [DBG] 2021-02-10T04:36:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:28 smithi085 conmon[35961]: pgmap v1005: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 398 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 917 KiB/s wr, 11 op/s 2021-02-10T04:36:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:28 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:36:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:28 smithi085 conmon[35961]: -10T04:36:28.147035+0000 mgr.y 2021-02-10T04:36:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:28 smithi085 conmon[35961]: (mgr.14140) 634 : audit 2021-02-10T04:36:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:28 smithi085 conmon[35961]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:36:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:28 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:36:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:28 smithi085 conmon[35961]: 10T04:36:28.178266+0000 mon.a ( 2021-02-10T04:36:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:28 smithi085 conmon[35961]: mon.0) 1414 : cluster [DBG] osdmap e571: 8 total, 8 up, 8 in 2021-02-10T04:36:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:28 smithi164 conmon[29288]: cluster 2021-02-10T04:36:27.409459 2021-02-10T04:36:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:28 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1413 : cluster [DBG] osdmap e570: 8 total, 8 up, 8 in 2021-02-10T04:36:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:28 smithi164 conmon[29288]: cluster 2021-02-10T04:36:28.119467+0000 mgr.y (mgr.14140) 633 : cluster [DBG] 2021-02-10T04:36:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:28 smithi164 conmon[29288]: pgmap v1005: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 398 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.1 MiB/s rd, 917 KiB/s wr, 11 op/s 2021-02-10T04:36:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:28 smithi164 conmon[29288]: audit 2021-02-10T04:36:28.147035+0000 2021-02-10T04:36:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:28 smithi164 conmon[29288]: mgr.y (mgr.14140) 634 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:36:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:28 smithi164 conmon[29288]: cluster 2021-02-10T04:36:28.178266+0000 mon.a (mon. 2021-02-10T04:36:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:28 smithi164 conmon[29288]: 0) 1414 : cluster [DBG] osdmap e571: 8 total, 8 up, 8 in 2021-02-10T04:36:29.186 INFO:tasks.rados.rados.0.smithi085.stdout:2181: done (12 left) 2021-02-10T04:36:29.186 INFO:tasks.rados.rados.0.smithi085.stdout:2182: rollback oid 31 current snap is 246 2021-02-10T04:36:29.186 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 31 to 230 2021-02-10T04:36:29.186 INFO:tasks.rados.rados.0.smithi085.stdout:2178: finishing write tid 1 to smithi08575397-1 2021-02-10T04:36:29.186 INFO:tasks.rados.rados.0.smithi085.stdout:2179: expect (ObjNum 578 snap 176 seq_num 578) 2021-02-10T04:36:29.195 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:36:29.238 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:36:29.532 INFO:tasks.rados.rados.0.smithi085.stdout:2178: finishing write tid 2 to smithi08575397-1 2021-02-10T04:36:29.533 INFO:tasks.rados.rados.0.smithi085.stdout:2178: finishing write tid 3 to smithi08575397-1 2021-02-10T04:36:29.533 INFO:tasks.rados.rados.0.smithi085.stdout:2178: finishing write tid 4 to smithi08575397-1 2021-02-10T04:36:29.533 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 1113 (ObjNum 763 snap 246 seq_num 763) dirty exists 2021-02-10T04:36:29.533 INFO:tasks.rados.rados.0.smithi085.stdout:2178: left oid 1 (ObjNum 763 snap 246 seq_num 763) 2021-02-10T04:36:29.533 INFO:tasks.rados.rados.0.smithi085.stdout:2180: finishing copy_from to smithi08575397-40 2021-02-10T04:36:29.533 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 690 (ObjNum 722 snap 226 seq_num 722) dirty exists 2021-02-10T04:36:29.533 INFO:tasks.rados.rados.0.smithi085.stdout:2180: finishing copy_from racing read to smithi08575397-40 2021-02-10T04:36:29.533 INFO:tasks.rados.rados.0.smithi085.stdout:2182: finishing rollback tid 0 to smithi08575397-31 2021-02-10T04:36:29.533 INFO:tasks.rados.rados.0.smithi085.stdout:2182: finishing rollback tid 1 to smithi08575397-31 2021-02-10T04:36:29.533 INFO:tasks.rados.rados.0.smithi085.stdout:2182: finishing rollback tid 2 to smithi08575397-31 2021-02-10T04:36:29.533 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 789 (ObjNum 569 snap 175 seq_num 569) dirty exists 2021-02-10T04:36:29.534 INFO:tasks.rados.rados.0.smithi085.stdout:2178: done (12 left) 2021-02-10T04:36:29.534 INFO:tasks.rados.rados.0.smithi085.stdout:2179: done (11 left) 2021-02-10T04:36:29.534 INFO:tasks.rados.rados.0.smithi085.stdout:2180: done (10 left) 2021-02-10T04:36:29.534 INFO:tasks.rados.rados.0.smithi085.stdout:2182: done (9 left) 2021-02-10T04:36:29.534 INFO:tasks.rados.rados.0.smithi085.stdout:2183: write oid 42 current snap is 246 2021-02-10T04:36:29.534 INFO:tasks.rados.rados.0.smithi085.stdout:2183: seq_num 764 ranges {440548=725536,1757537=478262,2813990=26558} 2021-02-10T04:36:29.539 INFO:tasks.rados.rados.0.smithi085.stdout:2183: writing smithi08575397-42 from 440548 to 1166084 tid 1 2021-02-10T04:36:29.546 INFO:tasks.rados.rados.0.smithi085.stdout:2183: writing smithi08575397-42 from 1757537 to 2235799 tid 2 2021-02-10T04:36:29.550 INFO:tasks.rados.rados.0.smithi085.stdout:2183: writing smithi08575397-42 from 2813990 to 2840548 tid 3 2021-02-10T04:36:29.551 INFO:tasks.rados.rados.0.smithi085.stdout:2184: write oid 2 current snap is 246 2021-02-10T04:36:29.551 INFO:tasks.rados.rados.0.smithi085.stdout:2183: finishing write tid 1 to smithi08575397-42 2021-02-10T04:36:29.551 INFO:tasks.rados.rados.0.smithi085.stdout:2184: seq_num 765 ranges {457355=400785,1416019=621920,2725060=132295} 2021-02-10T04:36:29.556 INFO:tasks.rados.rados.0.smithi085.stdout:2184: writing smithi08575397-2 from 457355 to 858140 tid 1 2021-02-10T04:36:29.564 INFO:tasks.rados.rados.0.smithi085.stdout:2184: writing smithi08575397-2 from 1416019 to 2037939 tid 2 2021-02-10T04:36:29.570 INFO:tasks.rados.rados.0.smithi085.stdout:2184: writing smithi08575397-2 from 2725060 to 2857355 tid 3 2021-02-10T04:36:29.570 INFO:tasks.rados.rados.0.smithi085.stdout:2185: read oid 6 snap -1 2021-02-10T04:36:29.570 INFO:tasks.rados.rados.0.smithi085.stdout:2185: expect (ObjNum 729 snap 230 seq_num 729) 2021-02-10T04:36:29.570 INFO:tasks.rados.rados.0.smithi085.stdout:2186: rollback oid 30 current snap is 246 2021-02-10T04:36:29.570 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 30 to 227 2021-02-10T04:36:29.570 INFO:tasks.rados.rados.0.smithi085.stdout:2183: finishing write tid 2 to smithi08575397-42 2021-02-10T04:36:29.570 INFO:tasks.rados.rados.0.smithi085.stdout:2183: finishing write tid 3 to smithi08575397-42 2021-02-10T04:36:29.570 INFO:tasks.rados.rados.0.smithi085.stdout:2183: finishing write tid 4 to smithi08575397-42 2021-02-10T04:36:29.570 INFO:tasks.rados.rados.0.smithi085.stdout:2183: finishing write tid 5 to smithi08575397-42 2021-02-10T04:36:29.570 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 833 (ObjNum 764 snap 246 seq_num 764) dirty exists 2021-02-10T04:36:29.571 INFO:tasks.rados.rados.0.smithi085.stdout:2183: left oid 42 (ObjNum 764 snap 246 seq_num 764) 2021-02-10T04:36:29.571 INFO:tasks.rados.rados.0.smithi085.stdout:2184: finishing write tid 1 to smithi08575397-2 2021-02-10T04:36:29.571 INFO:tasks.rados.rados.0.smithi085.stdout:2184: finishing write tid 2 to smithi08575397-2 2021-02-10T04:36:29.571 INFO:tasks.rados.rados.0.smithi085.stdout:2183: done (12 left) 2021-02-10T04:36:29.571 INFO:tasks.rados.rados.0.smithi085.stdout:2187: copy_from oid 40 from oid 3 current snap is 246 2021-02-10T04:36:29.571 INFO:tasks.rados.rados.0.smithi085.stdout:2188: snap_remove snap 245 2021-02-10T04:36:30.198 INFO:tasks.rados.rados.0.smithi085.stdout:2184: finishing write tid 3 to smithi08575397-2 2021-02-10T04:36:30.198 INFO:tasks.rados.rados.0.smithi085.stdout:2184: finishing write tid 4 to smithi08575397-2 2021-02-10T04:36:30.198 INFO:tasks.rados.rados.0.smithi085.stdout:2184: finishing write tid 5 to smithi08575397-2 2021-02-10T04:36:30.198 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 677 (ObjNum 765 snap 246 seq_num 765) dirty exists 2021-02-10T04:36:30.198 INFO:tasks.rados.rados.0.smithi085.stdout:2184: left oid 2 (ObjNum 765 snap 246 seq_num 765) 2021-02-10T04:36:30.198 INFO:tasks.rados.rados.0.smithi085.stdout:2185: expect (ObjNum 729 snap 230 seq_num 729) 2021-02-10T04:36:30.288 INFO:tasks.rados.rados.0.smithi085.stdout:2186: finishing rollback tid 1 to smithi08575397-30 2021-02-10T04:36:30.288 INFO:tasks.rados.rados.0.smithi085.stdout:2186: finishing rollback tid 2 to smithi08575397-30 2021-02-10T04:36:30.288 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 1053 (ObjNum 718 snap 226 seq_num 718) dirty exists 2021-02-10T04:36:30.289 INFO:tasks.rados.rados.0.smithi085.stdout:2187: finishing copy_from to smithi08575397-40 2021-02-10T04:36:30.289 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 784 (ObjNum 756 snap 241 seq_num 756) dirty exists 2021-02-10T04:36:30.289 INFO:tasks.rados.rados.0.smithi085.stdout:2187: finishing copy_from racing read to smithi08575397-40 2021-02-10T04:36:30.289 INFO:tasks.rados.rados.0.smithi085.stdout:2184: done (13 left) 2021-02-10T04:36:30.289 INFO:tasks.rados.rados.0.smithi085.stdout:2185: done (12 left) 2021-02-10T04:36:30.289 INFO:tasks.rados.rados.0.smithi085.stdout:2186: done (11 left) 2021-02-10T04:36:30.289 INFO:tasks.rados.rados.0.smithi085.stdout:2187: done (10 left) 2021-02-10T04:36:30.289 INFO:tasks.rados.rados.0.smithi085.stdout:2188: done (9 left) 2021-02-10T04:36:30.289 INFO:tasks.rados.rados.0.smithi085.stdout:2189: write (excl) oid 3 current snap is 246 2021-02-10T04:36:30.289 INFO:tasks.rados.rados.0.smithi085.stdout:2189: seq_num 766 ranges {474162=476034,1358148=481931,2352483=465303} 2021-02-10T04:36:30.294 INFO:tasks.rados.rados.0.smithi085.stdout:2189: writing smithi08575397-3 from 474162 to 950196 tid 1 2021-02-10T04:36:30.300 INFO:tasks.rados.rados.0.smithi085.stdout:2189: writing smithi08575397-3 from 1358148 to 1840079 tid 2 2021-02-10T04:36:30.306 INFO:tasks.rados.rados.0.smithi085.stdout:2189: writing smithi08575397-3 from 2352483 to 2817786 tid 3 2021-02-10T04:36:30.306 INFO:tasks.rados.rados.0.smithi085.stdout:2189: finishing write tid 1 to smithi08575397-3 2021-02-10T04:36:30.306 INFO:tasks.rados.rados.0.smithi085.stdout:2189: finishing write tid 2 to smithi08575397-3 2021-02-10T04:36:30.307 INFO:tasks.rados.rados.0.smithi085.stdout:2190: snap_create 2021-02-10T04:36:30.310 INFO:tasks.rados.rados.0.smithi085.stdout:2189: finishing write tid 3 to smithi08575397-3 2021-02-10T04:36:30.310 INFO:tasks.rados.rados.0.smithi085.stdout:2189: finishing write tid 4 to smithi08575397-3 2021-02-10T04:36:30.311 INFO:tasks.rados.rados.0.smithi085.stdout:2189: finishing write tid 5 to smithi08575397-3 2021-02-10T04:36:30.311 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 807 (ObjNum 766 snap 246 seq_num 766) dirty exists 2021-02-10T04:36:30.311 INFO:tasks.rados.rados.0.smithi085.stdout:2189: left oid 3 (ObjNum 766 snap 246 seq_num 766) 2021-02-10T04:36:30.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[35961]: cluster 2021-02-10T04:36:30.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[35961]: 2021-02-10T04:36:30.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:36:30.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[35961]: :36:28.969004+0000 osd.3 (osd.3) 70 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:36:30.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[35961]: cluster 2021-02-10T04:36:28.969745 2021-02-10T04:36:30.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[35961]: +0000 osd.3 (osd.3) 71 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:36:30.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:30.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[35961]: 02-10T04:36:29.186484+0000 mon.a (mon. 2021-02-10T04:36:30.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[35961]: 0) 1415 : cluster [DBG] osdmap e572: 8 total, 8 up, 8 in 2021-02-10T04:36:30.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:36:30.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[35961]: :36:30.119936+0000 mgr.y (mgr.14140) 635 2021-02-10T04:36:30.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[35961]: : cluster [DBG] pgmap v1008: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 404 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 1.7 MiB/s wr, 10 op/s 2021-02-10T04:36:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[29319]: cluster 2021-02-10T04:36:30.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[29319]: 2021-02-10T04:36:30.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[29319]: 2021 2021-02-10T04:36:30.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[29319]: -02-10T04 2021-02-10T04:36:30.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[29319]: :36:28.969004+0000 osd.3 (osd.3) 70 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:36:30.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[29319]: cluster 2021-02-10T04:36:30.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[29319]: 2021-02 2021-02-10T04:36:30.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[29319]: -10T04:36:28.969745+0000 osd.3 (osd.3 2021-02-10T04:36:30.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[29319]: ) 71 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:36:30.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:30.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[29319]: -02-10T04:36:29.186484+0000 mon.a (mon 2021-02-10T04:36:30.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[29319]: .0) 1415 : cluster [DBG] osdmap e572: 8 total, 8 up, 8 in 2021-02-10T04:36:30.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:30.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[29319]: -02-10T04:36: 2021-02-10T04:36:30.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[29319]: 30.119936+0000 mgr.y (mgr.14140) 635 : 2021-02-10T04:36:30.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[29319]: cluster [DBG] pgmap v1008: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 404 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 1.7 MiB/s wr, 10 op/s 2021-02-10T04:36:30.511 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:30 smithi164 conmon[29288]: cluster 2021-02-10T04:36:28.969004+0000 osd.3 (osd. 2021-02-10T04:36:30.511 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:30 smithi164 conmon[29288]: 3) 70 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:36:30.511 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:30 smithi164 conmon[29288]: cluster 2021-02-10T04:36:28.969745+0000 osd.3 (osd.3) 71 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:36:30.511 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:30 smithi164 conmon[29288]: cluster 2021-02-10T04:36:29.186484+0000 mon.a ( 2021-02-10T04:36:30.511 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:30 smithi164 conmon[29288]: mon.0) 1415 : cluster [DBG] osdmap e572: 8 total, 8 up, 8 in 2021-02-10T04:36:30.511 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:30 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:30.512 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:30 smithi164 conmon[29288]: 02-10T04:36:30.119936+0000 mgr.y ( 2021-02-10T04:36:30.512 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:30 smithi164 conmon[29288]: mgr.14140) 635 : cluster [DBG] pgmap v1008: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 404 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.4 MiB/s rd, 1.7 MiB/s wr, 10 op/s 2021-02-10T04:36:30.778 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 7 1 2021-02-10T04:36:30.796 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:36:30.907 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:36:30.941 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[45443]: debug 2021-02-10T04:36:30.941 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[45443]: 2021-02-10T04:36:30.764+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:36:30.941 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:36:30 smithi085 conmon[45443]: 2021-02-10T04:36:30.944 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:36:31.064 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:36:31.064 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:36:31.104 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:36:31.118 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:36:31.196 INFO:tasks.rados.rados.0.smithi085.stdout:2189: done (10 left) 2021-02-10T04:36:31.196 INFO:tasks.rados.rados.0.smithi085.stdout:2190: done (9 left) 2021-02-10T04:36:31.196 INFO:tasks.rados.rados.0.smithi085.stdout:2191: delete oid 3 current snap is 247 2021-02-10T04:36:31.203 INFO:tasks.rados.rados.0.smithi085.stdout:2191: done (9 left) 2021-02-10T04:36:31.203 INFO:tasks.rados.rados.0.smithi085.stdout:2192: snap_create 2021-02-10T04:36:31.223 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:36:31.253 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:36:31.322 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:31 smithi164 conmon[29288]: cluster 2021-02-10T04:36:30.191300+0000 mon.a ( 2021-02-10T04:36:31.322 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:31 smithi164 conmon[29288]: mon.0) 1416 : cluster [DBG] osdmap e573: 8 total, 8 up, 8 in 2021-02-10T04:36:31.323 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:31 smithi164 conmon[33677]: debug 2021-02-10T04:36:31.036+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:36:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:31 smithi085 conmon[29319]: cluster 2021-02-10T04:36:30.191300+0000 2021-02-10T04:36:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:31 smithi085 conmon[29319]: mon.a (mon.0) 1416 : cluster [DBG] osdmap e573: 8 total, 8 up, 8 in 2021-02-10T04:36:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:31 smithi085 conmon[35961]: cluster 2021-02-10T04:36:30. 2021-02-10T04:36:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:31 smithi085 conmon[35961]: 191300+0000 mon.a (mon.0) 1416 : cluster [DBG] osdmap e573: 8 total, 8 up, 8 in 2021-02-10T04:36:31.560 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:36:31.644 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:36:31 smithi164 conmon[40892]: debug 2021-02-10T04:36:31.321+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:36:32.204 INFO:tasks.rados.rados.0.smithi085.stdout:2192: done (9 left) 2021-02-10T04:36:32.204 INFO:tasks.rados.rados.0.smithi085.stdout:2193: snap_remove snap 193 2021-02-10T04:36:32.635 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:36:32.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:32 smithi164 conmon[29288]: cluster 2021-02-10T04:36:32.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:32 smithi164 conmon[29288]: 2021-02-10T04:36:32.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:32 smithi164 conmon[29288]: 2021-02-10T04:36:31.195336+0000 2021-02-10T04:36:32.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:32 smithi164 conmon[29288]: mon.a (mon.0) 1417 : cluster [DBG] osdmap e574: 8 total, 8 up, 8 in 2021-02-10T04:36:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:32 smithi164 conmon[29288]: cluster 2021-02-10T04:36:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:32 smithi164 conmon[29288]: 2021-02-10T04:36:32.120448+0000 mgr.y (mgr.14140) 636 : 2021-02-10T04:36:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:32 smithi164 conmon[29288]: cluster [DBG] pgmap v1011: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 406 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 1.4 MiB/s wr, 12 op/s 2021-02-10T04:36:32.669 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:32 smithi085 conmon[35961]: cluster 2021-02-10T04:36:31.195336 2021-02-10T04:36:32.669 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:32 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1417 : cluster [DBG] osdmap e574: 8 total, 8 up, 8 in 2021-02-10T04:36:32.669 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:32 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:36:32.669 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:32 smithi085 conmon[35961]: :36:32.120448+0000 mgr.y (mgr.14140) 2021-02-10T04:36:32.669 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:32 smithi085 conmon[35961]: 636 : cluster [DBG] pgmap v1011: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 406 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 1.4 MiB/s wr, 12 op/s 2021-02-10T04:36:32.670 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:32 smithi085 conmon[29319]: cluster 2021-02-10T04:36:31.195336+0000 2021-02-10T04:36:32.670 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:32 smithi085 conmon[29319]: mon.a (mon.0) 1417 : cluster [DBG] osdmap e574: 8 total, 8 up, 8 in 2021-02-10T04:36:32.670 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:32 smithi085 conmon[29319]: cluster 2021-02-10T04:36:32.670 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:32 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:36:32.670 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:32 smithi085 conmon[29319]: 36:32.120448+0000 mgr.y (mgr.14140) 636 : cluster [DBG] pgmap v1011: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 406 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 1.4 MiB/s wr, 12 op/s 2021-02-10T04:36:32.699 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset nodeep-scrub 2021-02-10T04:36:32.952 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:36:33.005 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:36:33.144 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:36:33.175 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:36:33.215 INFO:tasks.rados.rados.0.smithi085.stdout:2193: done (9 left) 2021-02-10T04:36:33.215 INFO:tasks.rados.rados.0.smithi085.stdout:2194: rollback oid 38 current snap is 248 2021-02-10T04:36:33.215 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 38 to 217 2021-02-10T04:36:33.215 INFO:tasks.rados.rados.0.smithi085.stdout:2195: read oid 40 snap -1 2021-02-10T04:36:33.215 INFO:tasks.rados.rados.0.smithi085.stdout:2195: expect (ObjNum 756 snap 241 seq_num 756) 2021-02-10T04:36:33.215 INFO:tasks.rados.rados.0.smithi085.stdout:2196: snap_create 2021-02-10T04:36:33.231 INFO:tasks.rados.rados.0.smithi085.stdout:2195: expect (ObjNum 756 snap 241 seq_num 756) 2021-02-10T04:36:33.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:33 smithi164 conmon[29288]: cluster 2021-02-10T04:36:32.203267+0000 mon.a 2021-02-10T04:36:33.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:33 smithi164 conmon[29288]: (mon.0) 1418 : cluster [DBG] osdmap e575: 8 total, 8 up, 8 in 2021-02-10T04:36:33.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:33 smithi085 conmon[29319]: cluster 2021-02-10T04:36:32.203267+0000 mon.a (mon.0 2021-02-10T04:36:33.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:33 smithi085 conmon[29319]: ) 1418 : cluster [DBG] osdmap e575: 8 total, 8 up, 8 in 2021-02-10T04:36:33.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:33 smithi085 conmon[35961]: cluster 2021-02-10T04:36:32.203267+0000 2021-02-10T04:36:33.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:33 smithi085 conmon[35961]: mon.a (mon.0) 1418 : cluster [DBG] osdmap e575: 8 total, 8 up, 8 in 2021-02-10T04:36:33.448 INFO:tasks.rados.rados.0.smithi085.stdout:2194: finishing rollback tid 0 to smithi08575397-38 2021-02-10T04:36:33.448 INFO:tasks.rados.rados.0.smithi085.stdout:2194: finishing rollback tid 1 to smithi08575397-38 2021-02-10T04:36:33.448 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 1055 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-02-10T04:36:34.210 INFO:tasks.rados.rados.0.smithi085.stdout:2194: done (11 left) 2021-02-10T04:36:34.210 INFO:tasks.rados.rados.0.smithi085.stdout:2195: done (10 left) 2021-02-10T04:36:34.210 INFO:tasks.rados.rados.0.smithi085.stdout:2196: done (9 left) 2021-02-10T04:36:34.210 INFO:tasks.rados.rados.0.smithi085.stdout:2197: read oid 7 snap -1 2021-02-10T04:36:34.211 INFO:tasks.rados.rados.0.smithi085.stdout:2197: expect (ObjNum 759 snap 243 seq_num 759) 2021-02-10T04:36:34.211 INFO:tasks.rados.rados.0.smithi085.stdout:2198: write (excl) oid 31 current snap is 249 2021-02-10T04:36:34.211 INFO:tasks.rados.rados.0.smithi085.stdout:2198: seq_num 768 ranges {507776=742885,1991465=485600} 2021-02-10T04:36:34.215 INFO:tasks.rados.rados.0.smithi085.stdout:2198: writing smithi08575397-31 from 507776 to 1250661 tid 1 2021-02-10T04:36:34.223 INFO:tasks.rados.rados.0.smithi085.stdout:2198: writing smithi08575397-31 from 1991465 to 2477065 tid 2 2021-02-10T04:36:34.224 INFO:tasks.rados.rados.0.smithi085.stdout:2199: rollback oid 38 current snap is 249 2021-02-10T04:36:34.224 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 38 to 214 2021-02-10T04:36:34.224 INFO:tasks.rados.rados.0.smithi085.stdout:2197: expect (ObjNum 759 snap 243 seq_num 759) 2021-02-10T04:36:34.273 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.6 dump_blocked_ops 2021-02-10T04:36:34.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:34 smithi164 conmon[33677]: debug 2021-02-10T04:36:34.104+0000 7fbea992c700 -1 osd.4 pg_epoch: 576 pg[6.1a( empty local-lis/les=59/60 n=0 ec=59/59 lis/c=59/59 les/c/f=60/60/0 sis=59) [4,5,1] r=0 lpr=59 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.1a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x5627605fedc0) 2021-02-10T04:36:34.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:34 smithi164 conmon[29288]: cluster 2021-02-10T04:36:33.209456+0000 mon.a (mon.0) 1419 : cluster [DBG] osdmap e576: 8 total, 8 up, 8 in 2021-02-10T04:36:34.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:34 smithi164 conmon[29288]: cluster 2021-02-10T04:36:33.620969+0000 osd.0 (osd.0) 72 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:36:34.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:34 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:34.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:34 smithi164 conmon[29288]: -02-10T04:36:33.621211+0000 osd.0 (osd.0) 73 : cluster [DBG] 2021-02-10T04:36:34.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:34 smithi164 conmon[29288]: purged_snaps scrub ok 2021-02-10T04:36:34.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:34 smithi164 conmon[29288]: cluster 2021-02-10T04:36 2021-02-10T04:36:34.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:34 smithi164 conmon[29288]: :34.121033+0000 mgr.y ( 2021-02-10T04:36:34.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:34 smithi164 conmon[29288]: mgr.14140) 637 : cluster [DBG] pgmap v1014: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 408 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.9 MiB/s rd, 961 KiB/s wr, 10 op/s 2021-02-10T04:36:34.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:34 smithi164 conmon[29288]: cluster 2021-02-10T04:36:34.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:34 smithi164 conmon[29288]: 2021-02-10T04:36:34.207544+0000 2021-02-10T04:36:34.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:34 smithi164 conmon[29288]: mon.a (mon.0) 1420 : cluster [DBG] osdmap e577: 8 total, 8 up, 8 in 2021-02-10T04:36:34.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:34 smithi164 conmon[29288]: 2021-02-10T04:36:34.430 INFO:tasks.rados.rados.0.smithi085.stdout:2198: finishing write tid 1 to smithi08575397-31 2021-02-10T04:36:34.430 INFO:tasks.rados.rados.0.smithi085.stdout:2198: finishing write tid 2 to smithi08575397-31 2021-02-10T04:36:34.430 INFO:tasks.rados.rados.0.smithi085.stdout:2198: finishing write tid 3 to smithi08575397-31 2021-02-10T04:36:34.430 INFO:tasks.rados.rados.0.smithi085.stdout:2198: finishing write tid 4 to smithi08575397-31 2021-02-10T04:36:34.430 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 792 (ObjNum 768 snap 249 seq_num 768) dirty exists 2021-02-10T04:36:34.430 INFO:tasks.rados.rados.0.smithi085.stdout:2198: left oid 31 (ObjNum 768 snap 249 seq_num 768) 2021-02-10T04:36:34.430 INFO:tasks.rados.rados.0.smithi085.stdout:2199: finishing rollback tid 1 to smithi08575397-38 2021-02-10T04:36:34.430 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 1055 (ObjNum 82401776 snap 22030 seq_num 82549808) dirty dne 2021-02-10T04:36:34.430 INFO:tasks.rados.rados.0.smithi085.stdout:2197: done (11 left) 2021-02-10T04:36:34.430 INFO:tasks.rados.rados.0.smithi085.stdout:2198: done (10 left) 2021-02-10T04:36:34.430 INFO:tasks.rados.rados.0.smithi085.stdout:2199: done (9 left) 2021-02-10T04:36:34.431 INFO:tasks.rados.rados.0.smithi085.stdout:2200: write oid 50 current snap is 249 2021-02-10T04:36:34.431 INFO:tasks.rados.rados.0.smithi085.stdout:2200: seq_num 769 ranges {524583=418134,1649947=629258,2834752=89831} 2021-02-10T04:36:34.435 INFO:tasks.rados.rados.0.smithi085.stdout:2200: writing smithi08575397-50 from 524583 to 942717 tid 1 2021-02-10T04:36:34.444 INFO:tasks.rados.rados.0.smithi085.stdout:2200: writing smithi08575397-50 from 1649947 to 2279205 tid 2 2021-02-10T04:36:34.448 INFO:tasks.rados.rados.0.smithi085.stdout:2200: writing smithi08575397-50 from 2834752 to 2924583 tid 3 2021-02-10T04:36:34.449 INFO:tasks.rados.rados.0.smithi085.stdout:2201: read oid 11 snap -1 2021-02-10T04:36:34.449 INFO:tasks.rados.rados.0.smithi085.stdout:2201: expect (ObjNum 744 snap 239 seq_num 744) 2021-02-10T04:36:34.449 INFO:tasks.rados.rados.0.smithi085.stdout:2202: copy_from oid 30 from oid 22 current snap is 249 2021-02-10T04:36:34.449 INFO:tasks.rados.rados.0.smithi085.stdout:2200: finishing write tid 1 to smithi08575397-50 2021-02-10T04:36:34.449 INFO:tasks.rados.rados.0.smithi085.stdout:2203: snap_create 2021-02-10T04:36:34.451 INFO:tasks.rados.rados.0.smithi085.stdout:2200: finishing write tid 2 to smithi08575397-50 2021-02-10T04:36:34.453 INFO:tasks.rados.rados.0.smithi085.stdout:2200: finishing write tid 3 to smithi08575397-50 2021-02-10T04:36:34.453 INFO:tasks.rados.rados.0.smithi085.stdout:2200: finishing write tid 4 to smithi08575397-50 2021-02-10T04:36:34.456 INFO:tasks.rados.rados.0.smithi085.stdout:2200: finishing write tid 5 to smithi08575397-50 2021-02-10T04:36:34.456 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 1117 (ObjNum 769 snap 249 seq_num 769) dirty exists 2021-02-10T04:36:34.456 INFO:tasks.rados.rados.0.smithi085.stdout:2200: left oid 50 (ObjNum 769 snap 249 seq_num 769) 2021-02-10T04:36:34.460 INFO:tasks.rados.rados.0.smithi085.stdout:2201: expect (ObjNum 744 snap 239 seq_num 744) 2021-02-10T04:36:34.586 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:36:34.643 INFO:tasks.rados.rados.0.smithi085.stdout:2202: finishing copy_from to smithi08575397-30 2021-02-10T04:36:34.643 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 751 (ObjNum 727 snap 230 seq_num 727) dirty exists 2021-02-10T04:36:34.644 INFO:tasks.rados.rados.0.smithi085.stdout:2202: finishing copy_from racing read to smithi08575397-30 2021-02-10T04:36:34.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[29319]: cluster 2021-02-10T04:36:34.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[29319]: 2021-02-10T04:36:34.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[29319]: 2021-02-10T04:36: 2021-02-10T04:36:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[29319]: 33.209456+0000 mon.a (mon.0) 1419 : cluster [DBG] osdmap e576: 8 total, 8 up, 8 in 2021-02-10T04:36:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[29319]: cluster 2021-02-10T04:36:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[29319]: 2021-02-10T04:36:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[29319]: 2021 2021-02-10T04:36:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[29319]: - 2021-02-10T04:36:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[29319]: 02-10T04:36: 2021-02-10T04:36:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[29319]: 33.620969+0000 osd.0 (osd.0) 72 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:36:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[29319]: 2021-02-10T04:36:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[29319]: cluster 2021-02-10T04:36:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:36:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[29319]: 36:33.621211+0000 osd.0 (osd 2021-02-10T04:36:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[29319]: .0) 73 : cluster [DBG] 2021-02-10T04:36:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[29319]: purged_snaps scrub ok 2021-02-10T04:36:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[29319]: cluster 2021-02-10T04:36:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[29319]: 2021-02-10T04:36:34. 2021-02-10T04:36:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[29319]: 121033+0000 mgr.y (mgr.14140) 637 : cluster [DBG] 2021-02-10T04:36:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[29319]: pgmap v1014: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 408 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.9 MiB/s rd, 961 KiB/s wr, 10 op/s 2021-02-10T04:36:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[29319]: cluster 2021-02-10T04:36:34.207544+0000 2021-02-10T04:36:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[29319]: mon.a (mon.0) 1420 : cluster [DBG] osdmap e577: 8 total, 8 up, 8 in 2021-02-10T04:36:34.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[35961]: cluster 2021-02-10T04:36:33.209456+0000 mon.a (mon.0 2021-02-10T04:36:34.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[35961]: ) 1419 : cluster [DBG] osdmap e576: 8 total, 8 up, 8 in 2021-02-10T04:36:34.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[35961]: cluster 2021-02-10T04:36:33.620969+0000 osd.0 (osd.0) 72 : cluster [DBG] 2021-02-10T04:36:34.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[35961]: purged_snaps scrub starts 2021-02-10T04:36:34.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[35961]: cluster 2021-02-10T04:36: 2021-02-10T04:36:34.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[35961]: 33.621211+0000 osd.0 (osd.0) 73 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:36:34.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:36:34.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[35961]: 04:36:34.121033+0000 mgr.y (mgr.14140) 637 : cluster [DBG] pgmap v1014: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 408 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 6.9 MiB/s rd, 961 KiB/s wr, 10 op/s 2021-02-10T04:36:34.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[35961]: cluster 2021-02-10T04:36:34.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[35961]: 2021-02-10T04:36:34.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[35961]: 2021 2021-02-10T04:36:34.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[35961]: -02-10T04:36:34 2021-02-10T04:36:34.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:34 smithi085 conmon[35961]: .207544+0000 mon.a (mon.0) 1420 : cluster [DBG] osdmap e577: 8 total, 8 up, 8 in 2021-02-10T04:36:35.285 INFO:tasks.rados.rados.0.smithi085.stdout:2200: done (12 left) 2021-02-10T04:36:35.285 INFO:tasks.rados.rados.0.smithi085.stdout:2201: done (11 left) 2021-02-10T04:36:35.285 INFO:tasks.rados.rados.0.smithi085.stdout:2202: done (10 left) 2021-02-10T04:36:35.285 INFO:tasks.rados.rados.0.smithi085.stdout:2203: done (9 left) 2021-02-10T04:36:35.285 INFO:tasks.rados.rados.0.smithi085.stdout:2204: copy_from oid 42 from oid 23 current snap is 250 2021-02-10T04:36:35.285 INFO:tasks.rados.rados.0.smithi085.stdout:2205: write (excl) oid 43 current snap is 250 2021-02-10T04:36:35.286 INFO:tasks.rados.rados.0.smithi085.stdout:2205: seq_num 770 ranges {541390=493383,1592076=489269,2862175=79215} 2021-02-10T04:36:35.287 INFO:tasks.rados.rados.0.smithi085.stdout:2205: writing smithi08575397-43 from 541390 to 1034773 tid 1 2021-02-10T04:36:35.294 INFO:tasks.rados.rados.0.smithi085.stdout:2205: writing smithi08575397-43 from 1592076 to 2081345 tid 2 2021-02-10T04:36:35.300 INFO:tasks.rados.rados.0.smithi085.stdout:2205: writing smithi08575397-43 from 2862175 to 2941390 tid 3 2021-02-10T04:36:35.301 INFO:tasks.rados.rados.0.smithi085.stdout:2206: read oid 44 snap -1 2021-02-10T04:36:35.301 INFO:tasks.rados.rados.0.smithi085.stdout:2206: expect (ObjNum 760 snap 243 seq_num 760) 2021-02-10T04:36:35.301 INFO:tasks.rados.rados.0.smithi085.stdout:2207: snap_remove snap 233 2021-02-10T04:36:35.403 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:36:35.556 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:36:35.585 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:36:35.611 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:36:35 smithi085 conmon[41540]: debug 2021-02-10T04:36:35.325+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:36:35.694 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:36:35.799 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:36:35.828 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:36:36.144 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:36:35 smithi164 conmon[37263]: debug 2021-02-10T04:36:35.667+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:36:36.303 INFO:tasks.rados.rados.0.smithi085.stdout:2207: done (12 left) 2021-02-10T04:36:36.303 INFO:tasks.rados.rados.0.smithi085.stdout:2208: read oid 21 snap 239 2021-02-10T04:36:36.303 INFO:tasks.rados.rados.0.smithi085.stdout:2208: expect (ObjNum 431 snap 114 seq_num 431) 2021-02-10T04:36:36.303 INFO:tasks.rados.rados.0.smithi085.stdout:2209: copy_from oid 7 from oid 50 current snap is 250 2021-02-10T04:36:36.303 INFO:tasks.rados.rados.0.smithi085.stdout:2210: read oid 35 snap -1 2021-02-10T04:36:36.303 INFO:tasks.rados.rados.0.smithi085.stdout:2210: expect (ObjNum 578 snap 176 seq_num 578) 2021-02-10T04:36:36.304 INFO:tasks.rados.rados.0.smithi085.stdout:2211: read oid 20 snap -1 2021-02-10T04:36:36.304 INFO:tasks.rados.rados.0.smithi085.stdout:2211: expect (ObjNum 705 snap 221 seq_num 705) 2021-02-10T04:36:36.304 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:36:36.304 INFO:tasks.rados.rados.0.smithi085.stdout:2205: finishing write tid 1 to smithi08575397-43 2021-02-10T04:36:36.304 INFO:tasks.rados.rados.0.smithi085.stdout:2205: finishing write tid 2 to smithi08575397-43 2021-02-10T04:36:36.304 INFO:tasks.rados.rados.0.smithi085.stdout:2205: finishing write tid 3 to smithi08575397-43 2021-02-10T04:36:36.304 INFO:tasks.rados.rados.0.smithi085.stdout:2205: finishing write tid 4 to smithi08575397-43 2021-02-10T04:36:36.304 INFO:tasks.rados.rados.0.smithi085.stdout:2205: finishing write tid 5 to smithi08575397-43 2021-02-10T04:36:36.304 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 816 (ObjNum 770 snap 250 seq_num 770) dirty exists 2021-02-10T04:36:36.305 INFO:tasks.rados.rados.0.smithi085.stdout:2205: left oid 43 (ObjNum 770 snap 250 seq_num 770) 2021-02-10T04:36:36.305 INFO:tasks.rados.rados.0.smithi085.stdout:2206: expect (ObjNum 760 snap 243 seq_num 760) 2021-02-10T04:36:36.373 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:36 smithi085 conmon[29319]: cluster 2021-02-10T04:36:35.280865+0000 mon.a (mon.0) 1421 : cluster [DBG] osdmap e578: 8 total, 8 up, 8 in 2021-02-10T04:36:36.373 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:36 smithi085 conmon[29319]: cluster 2021-02-10T04:36:36.121517+0000 mgr.y (mgr.14140) 638 : cluster [DBG] pgmap v1017: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 412 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 1.8 MiB/s wr, 11 op/s 2021-02-10T04:36:36.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:36 smithi085 conmon[35961]: cluster 2021-02-10T04:36:35.280865+0000 2021-02-10T04:36:36.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:36 smithi085 conmon[35961]: mon.a (mon.0) 1421 : cluster [DBG] osdmap e578: 8 total, 8 up, 8 in 2021-02-10T04:36:36.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:36 smithi085 conmon[35961]: cluster 2021-02-10T04:36:36. 2021-02-10T04:36:36.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:36 smithi085 conmon[35961]: 121517+0000 mgr.y (mgr.14140) 638 : cluster [DBG] pgmap v1017: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 412 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 1.8 MiB/s wr, 11 op/s 2021-02-10T04:36:36.428 INFO:tasks.rados.rados.0.smithi085.stdout:2204: finishing copy_from to smithi08575397-42 2021-02-10T04:36:36.429 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 690 (ObjNum 722 snap 226 seq_num 722) dirty exists 2021-02-10T04:36:36.429 INFO:tasks.rados.rados.0.smithi085.stdout:2204: finishing copy_from racing read to smithi08575397-42 2021-02-10T04:36:36.429 INFO:tasks.rados.rados.0.smithi085.stdout:2208: expect (ObjNum 431 snap 114 seq_num 431) 2021-02-10T04:36:36.572 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:36:36.643 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:36:36 smithi164 conmon[44544]: debug 2021-02-10T04:36:36.299+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:36:36.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:36 smithi164 conmon[29288]: cluster 2021-02-10T04:36:35.280865+0000 mon.a 2021-02-10T04:36:36.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:36 smithi164 conmon[29288]: (mon.0) 1421 : cluster [DBG] osdmap e578: 8 total, 8 up, 8 in 2021-02-10T04:36:36.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:36 smithi164 conmon[29288]: cluster 2021-02-10T04:36:36.121517+0000 mgr.y (mgr.14140) 638 : cluster [DBG] pgmap v1017: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 412 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 1.8 MiB/s wr, 11 op/s 2021-02-10T04:36:36.675 INFO:tasks.rados.rados.0.smithi085.stdout:2210: expect (ObjNum 578 snap 176 seq_num 578) 2021-02-10T04:36:36.983 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:36:37.020 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:36:37.046 INFO:tasks.rados.rados.0.smithi085.stdout:2211: expect (ObjNum 705 snap 221 seq_num 705) 2021-02-10T04:36:37.303 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.7 to 1 (10000) 2021-02-10T04:36:37.306 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:37 smithi085 conmon[29319]: cluster 2021-02-10T04:36:36.293872+0000 2021-02-10T04:36:37.306 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:37 smithi085 conmon[29319]: mon.a (mon.0) 1422 : cluster [DBG] osdmap e579: 8 total, 8 up, 8 in 2021-02-10T04:36:37.306 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:37 smithi085 conmon[29319]: audit 2021-02-10T04:36:36.594586+0000 mon.a (mon.0) 1423 : audit [INF] 2021-02-10T04:36:37.306 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:37 smithi085 conmon[29319]: from='client.? 172.21.15.85:0/2509787274' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T04:36:37.306 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:37 smithi085 conmon[35961]: cluster 2021-02-10T04:36:36.293872 2021-02-10T04:36:37.306 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:37 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1422 : cluster [DBG] osdmap e579: 8 total, 8 up, 8 in 2021-02-10T04:36:37.306 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:37 smithi085 conmon[35961]: audit 2021-02-10T04:36:37.306 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:37 smithi085 conmon[35961]: 2021-02-10T04:36:36.594586+0000 2021-02-10T04:36:37.306 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:37 smithi085 conmon[35961]: mon.a (mon.0) 1423 : audit [INF] 2021-02-10T04:36:37.306 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:37 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/2509787274' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T04:36:37.307 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:36:37 smithi085 conmon[52836]: debug 2021-02-10T04:36:37.247+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:36:37.349 INFO:tasks.rados.rados.0.smithi085.stdout:2209: finishing copy_from to smithi08575397-7 2021-02-10T04:36:37.349 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 1117 (ObjNum 769 snap 249 seq_num 769) dirty exists 2021-02-10T04:36:37.349 INFO:tasks.rados.rados.0.smithi085.stdout:2209: finishing copy_from racing read to smithi08575397-7 2021-02-10T04:36:37.349 INFO:tasks.rados.rados.0.smithi085.stdout:2204: done (15 left) 2021-02-10T04:36:37.349 INFO:tasks.rados.rados.0.smithi085.stdout:2205: done (14 left) 2021-02-10T04:36:37.349 INFO:tasks.rados.rados.0.smithi085.stdout:2206: done (13 left) 2021-02-10T04:36:37.349 INFO:tasks.rados.rados.0.smithi085.stdout:2208: done (12 left) 2021-02-10T04:36:37.349 INFO:tasks.rados.rados.0.smithi085.stdout:2209: done (11 left) 2021-02-10T04:36:37.349 INFO:tasks.rados.rados.0.smithi085.stdout:2210: done (10 left) 2021-02-10T04:36:37.349 INFO:tasks.rados.rados.0.smithi085.stdout:2211: done (9 left) 2021-02-10T04:36:37.349 INFO:tasks.rados.rados.0.smithi085.stdout:2212: copy_from oid 26 from oid 13 current snap is 250 2021-02-10T04:36:37.350 INFO:tasks.rados.rados.0.smithi085.stdout:2213: write (excl) oid 1 current snap is 250 2021-02-10T04:36:37.350 INFO:tasks.rados.rados.0.smithi085.stdout:2213: seq_num 771 ranges {558197=568632,1650558=632927,2773245=184952} 2021-02-10T04:36:37.355 INFO:tasks.rados.rados.0.smithi085.stdout:2213: writing smithi08575397-1 from 558197 to 1126829 tid 1 2021-02-10T04:36:37.363 INFO:tasks.rados.rados.0.smithi085.stdout:2213: writing smithi08575397-1 from 1650558 to 2283485 tid 2 2021-02-10T04:36:37.367 INFO:tasks.rados.rados.0.smithi085.stdout:2213: writing smithi08575397-1 from 2773245 to 2958197 tid 3 2021-02-10T04:36:37.367 INFO:tasks.rados.rados.0.smithi085.stdout:2213: finishing write tid 1 to smithi08575397-1 2021-02-10T04:36:37.368 INFO:tasks.rados.rados.0.smithi085.stdout:2214: snap_remove snap 191 2021-02-10T04:36:37.536 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:36:37.560 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:36:37.560 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:36:37.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:37 smithi164 conmon[29288]: cluster 2021-02-10T04:36:36.293872+0000 mon.a 2021-02-10T04:36:37.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:37 smithi164 conmon[29288]: (mon.0) 1422 : cluster [DBG] osdmap e579: 8 total, 8 up, 8 in 2021-02-10T04:36:37.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:37 smithi164 conmon[29288]: audit 2021-02-10T04:36:36.594586+0000 mon.a (mon.0) 1423 : audit [INF] from='client.? 172.21.15.85:0/2509787274' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T04:36:37.667 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:36:37 smithi085 conmon[52836]: debug 2021-02-10T04:36:37.666+0000 7fba75fba700 -1 osd.3 pg_epoch: 580 pg[4.10( v 538'16 (0'0,538'16] local-lis/les=71/72 n=10 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,5,6] r=0 lpr=71 crt=538'16 lcod 538'15 mlcod 538'15 active+clean] scrubber pg(4.10) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6 (0x55feb3a14b00) 2021-02-10T04:36:38.184 INFO:tasks.rados.rados.0.smithi085.stdout:2214: done (11 left) 2021-02-10T04:36:38.184 INFO:tasks.rados.rados.0.smithi085.stdout:2215: write oid 42 current snap is 250 2021-02-10T04:36:38.184 INFO:tasks.rados.rados.0.smithi085.stdout:2213: finishing write tid 2 to smithi08575397-1 2021-02-10T04:36:38.184 INFO:tasks.rados.rados.0.smithi085.stdout:2213: finishing write tid 3 to smithi08575397-1 2021-02-10T04:36:38.184 INFO:tasks.rados.rados.0.smithi085.stdout:2213: finishing write tid 4 to smithi08575397-1 2021-02-10T04:36:38.184 INFO:tasks.rados.rados.0.smithi085.stdout:2213: finishing write tid 5 to smithi08575397-1 2021-02-10T04:36:38.184 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 1121 (ObjNum 771 snap 250 seq_num 771) dirty exists 2021-02-10T04:36:38.184 INFO:tasks.rados.rados.0.smithi085.stdout:2213: left oid 1 (ObjNum 771 snap 250 seq_num 771) 2021-02-10T04:36:38.184 INFO:tasks.rados.rados.0.smithi085.stdout:2212: finishing copy_from to smithi08575397-26 2021-02-10T04:36:38.184 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 748 (ObjNum 742 snap 238 seq_num 742) dirty exists 2021-02-10T04:36:38.184 INFO:tasks.rados.rados.0.smithi085.stdout:2212: finishing copy_from racing read to smithi08575397-26 2021-02-10T04:36:38.185 INFO:tasks.rados.rados.0.smithi085.stdout:2215: seq_num 772 ranges {575004=643881,1709040=492938,2917021=57983} 2021-02-10T04:36:38.191 INFO:tasks.rados.rados.0.smithi085.stdout:2215: writing smithi08575397-42 from 575004 to 1218885 tid 1 2021-02-10T04:36:38.198 INFO:tasks.rados.rados.0.smithi085.stdout:2215: writing smithi08575397-42 from 1709040 to 2201978 tid 2 2021-02-10T04:36:38.203 INFO:tasks.rados.rados.0.smithi085.stdout:2215: writing smithi08575397-42 from 2917021 to 2975004 tid 3 2021-02-10T04:36:38.203 INFO:tasks.rados.rados.0.smithi085.stdout:2215: finishing write tid 1 to smithi08575397-42 2021-02-10T04:36:38.203 INFO:tasks.rados.rados.0.smithi085.stdout:2212: done (11 left) 2021-02-10T04:36:38.203 INFO:tasks.rados.rados.0.smithi085.stdout:2213: done (10 left) 2021-02-10T04:36:38.203 INFO:tasks.rados.rados.0.smithi085.stdout:2216: read oid 7 snap -1 2021-02-10T04:36:38.203 INFO:tasks.rados.rados.0.smithi085.stdout:2216: expect (ObjNum 769 snap 249 seq_num 769) 2021-02-10T04:36:38.203 INFO:tasks.rados.rados.0.smithi085.stdout:2217: write oid 21 current snap is 250 2021-02-10T04:36:38.203 INFO:tasks.rados.rados.0.smithi085.stdout:2217: seq_num 773 ranges {591811=719130,1767522=636596,2828091=163720} 2021-02-10T04:36:38.210 INFO:tasks.rados.rados.0.smithi085.stdout:2217: writing smithi08575397-21 from 591811 to 1310941 tid 1 2021-02-10T04:36:38.217 INFO:tasks.rados.rados.0.smithi085.stdout:2217: writing smithi08575397-21 from 1767522 to 2404118 tid 2 2021-02-10T04:36:38.221 INFO:tasks.rados.rados.0.smithi085.stdout:2217: writing smithi08575397-21 from 2828091 to 2991811 tid 3 2021-02-10T04:36:38.222 INFO:tasks.rados.rados.0.smithi085.stdout:2218: rollback oid 10 current snap is 250 2021-02-10T04:36:38.222 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 10 to 200 2021-02-10T04:36:38.222 INFO:tasks.rados.rados.0.smithi085.stdout:2219: delete oid 13 current snap is 250 2021-02-10T04:36:38.222 INFO:tasks.rados.rados.0.smithi085.stdout:2215: finishing write tid 2 to smithi08575397-42 2021-02-10T04:36:38.222 INFO:tasks.rados.rados.0.smithi085.stdout:2215: finishing write tid 3 to smithi08575397-42 2021-02-10T04:36:38.222 INFO:tasks.rados.rados.0.smithi085.stdout:2215: finishing write tid 4 to smithi08575397-42 2021-02-10T04:36:38.222 INFO:tasks.rados.rados.0.smithi085.stdout:2216: expect (ObjNum 769 snap 249 seq_num 769) 2021-02-10T04:36:38.399 INFO:tasks.rados.rados.0.smithi085.stdout:2215: finishing write tid 5 to smithi08575397-42 2021-02-10T04:36:38.399 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 837 (ObjNum 772 snap 250 seq_num 772) dirty exists 2021-02-10T04:36:38.399 INFO:tasks.rados.rados.0.smithi085.stdout:2215: left oid 42 (ObjNum 772 snap 250 seq_num 772) 2021-02-10T04:36:38.399 INFO:tasks.rados.rados.0.smithi085.stdout:2215: done (13 left) 2021-02-10T04:36:38.400 INFO:tasks.rados.rados.0.smithi085.stdout:2216: done (12 left) 2021-02-10T04:36:38.400 INFO:tasks.rados.rados.0.smithi085.stdout:2219: done (11 left) 2021-02-10T04:36:38.400 INFO:tasks.rados.rados.0.smithi085.stdout:2220: read oid 14 snap 229 2021-02-10T04:36:38.400 INFO:tasks.rados.rados.0.smithi085.stdout:2220: expect deleted 2021-02-10T04:36:38.400 INFO:tasks.rados.rados.0.smithi085.stdout:2221: read oid 42 snap 229 2021-02-10T04:36:38.401 INFO:tasks.rados.rados.0.smithi085.stdout:2221: expect (ObjNum 646 snap 196 seq_num 646) 2021-02-10T04:36:38.401 INFO:tasks.rados.rados.0.smithi085.stdout:2222: write (excl) oid 11 current snap is 250 2021-02-10T04:36:38.401 INFO:tasks.rados.rados.0.smithi085.stdout:2222: seq_num 775 ranges {625425=469628,1768133=640265} 2021-02-10T04:36:38.405 INFO:tasks.rados.rados.0.smithi085.stdout:2222: writing smithi08575397-11 from 625425 to 1095053 tid 1 2021-02-10T04:36:38.413 INFO:tasks.rados.rados.0.smithi085.stdout:2222: writing smithi08575397-11 from 1768133 to 2408398 tid 2 2021-02-10T04:36:38.414 INFO:tasks.rados.rados.0.smithi085.stdout:2223: write oid 5 current snap is 250 2021-02-10T04:36:38.414 INFO:tasks.rados.rados.0.smithi085.stdout:2217: finishing write tid 1 to smithi08575397-21 2021-02-10T04:36:38.414 INFO:tasks.rados.rados.0.smithi085.stdout:2217: finishing write tid 2 to smithi08575397-21 2021-02-10T04:36:38.414 INFO:tasks.rados.rados.0.smithi085.stdout:2217: finishing write tid 3 to smithi08575397-21 2021-02-10T04:36:38.414 INFO:tasks.rados.rados.0.smithi085.stdout:2218: finishing rollback tid 1 to smithi08575397-10 2021-02-10T04:36:38.414 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 10 v 644 (ObjNum 1 snap 32576 seq_num 82800464) dirty dne 2021-02-10T04:36:38.414 INFO:tasks.rados.rados.0.smithi085.stdout:2217: finishing write tid 4 to smithi08575397-21 2021-02-10T04:36:38.415 INFO:tasks.rados.rados.0.smithi085.stdout:2217: finishing write tid 5 to smithi08575397-21 2021-02-10T04:36:38.415 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 1203 (ObjNum 773 snap 250 seq_num 773) dirty exists 2021-02-10T04:36:38.415 INFO:tasks.rados.rados.0.smithi085.stdout:2217: left oid 21 (ObjNum 773 snap 250 seq_num 773) 2021-02-10T04:36:38.415 INFO:tasks.rados.rados.0.smithi085.stdout:2223: seq_num 776 ranges {642232=661230,1942968=783923} 2021-02-10T04:36:38.421 INFO:tasks.rados.rados.0.smithi085.stdout:2223: writing smithi08575397-5 from 642232 to 1303462 tid 1 2021-02-10T04:36:38.431 INFO:tasks.rados.rados.0.smithi085.stdout:2223: writing smithi08575397-5 from 1942968 to 2726891 tid 2 2021-02-10T04:36:38.431 INFO:tasks.rados.rados.0.smithi085.stdout:2217: done (14 left) 2021-02-10T04:36:38.431 INFO:tasks.rados.rados.0.smithi085.stdout:2218: done (13 left) 2021-02-10T04:36:38.431 INFO:tasks.rados.rados.0.smithi085.stdout:2224: copy_from oid 31 from oid 13 current snap is 250 2021-02-10T04:36:38.431 INFO:tasks.rados.rados.0.smithi085.stdout:2221: expect (ObjNum 646 snap 196 seq_num 646) 2021-02-10T04:36:38.470 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:36:38 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:36:38.470 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:38 smithi164 conmon[29288]: audit 2021-02-10T04:36:37.294278+0000 mon.a (mon. 2021-02-10T04:36:38.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:38 smithi164 conmon[29288]: 0) 1424 : audit [INF] from='client.? 172.21.15.85:0/2509787274' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-02-10T04:36:38.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:38 smithi164 conmon[29288]: cluster 2021-02-10T04:36:37.294353+0000 mon.a (mon.0) 1425 : cluster [DBG] osdmap e580: 8 total, 8 up, 8 in 2021-02-10T04:36:38.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:38 smithi164 conmon[29288]: audit 2021-02-10T04:36:38.101299+0000 mon.a (mon.0) 1426 : 2021-02-10T04:36:38.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:38 smithi164 conmon[29288]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:36:38.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:38 smithi164 conmon[29288]: audit 2021-02-10T04:36 2021-02-10T04:36:38.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:38 smithi164 conmon[29288]: :38.101526+0000 mon.a (mon.0) 1427 : 2021-02-10T04:36:38.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:38 smithi164 conmon[29288]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:36:38.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:38 smithi164 conmon[29288]: cluster 2021-02-10T04:36 2021-02-10T04:36:38.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:38 smithi164 conmon[29288]: :38.122075+0000 mgr.y (mgr.14140) 639 2021-02-10T04:36:38.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:38 smithi164 conmon[29288]: : cluster [DBG] pgmap v1020: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 414 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 1.4 MiB/s wr, 7 op/s 2021-02-10T04:36:38.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:38 smithi164 conmon[29288]: audit 2021-02-10T04: 2021-02-10T04:36:38.471 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:38 smithi164 conmon[29288]: 36:38.156993+0000 mgr.y (mgr.14140) 2021-02-10T04:36:38.472 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:38 smithi164 conmon[29288]: 640 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:36:38.472 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:38 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:36:38.472 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:38 smithi164 conmon[29288]: 36:38.182470+0000 mon.a (mon.0) 2021-02-10T04:36:38.472 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:38 smithi164 conmon[29288]: 1428 : cluster [DBG] osdmap e581: 8 total, 8 up, 8 in 2021-02-10T04:36:38.605 INFO:tasks.rados.rados.0.smithi085.stdout:2222: finishing write tid 1 to smithi08575397-11 2021-02-10T04:36:38.605 INFO:tasks.rados.rados.0.smithi085.stdout:2220: done (13 left) 2021-02-10T04:36:38.605 INFO:tasks.rados.rados.0.smithi085.stdout:2221: done (12 left) 2021-02-10T04:36:38.605 INFO:tasks.rados.rados.0.smithi085.stdout:2225: delete oid 6 current snap is 250 2021-02-10T04:36:38.605 INFO:tasks.rados.rados.0.smithi085.stdout:2222: finishing write tid 2 to smithi08575397-11 2021-02-10T04:36:38.605 INFO:tasks.rados.rados.0.smithi085.stdout:2222: finishing write tid 3 to smithi08575397-11 2021-02-10T04:36:38.605 INFO:tasks.rados.rados.0.smithi085.stdout:2222: finishing write tid 4 to smithi08575397-11 2021-02-10T04:36:38.605 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 1207 (ObjNum 775 snap 250 seq_num 775) dirty exists 2021-02-10T04:36:38.605 INFO:tasks.rados.rados.0.smithi085.stdout:2222: left oid 11 (ObjNum 775 snap 250 seq_num 775) 2021-02-10T04:36:38.606 INFO:tasks.rados.rados.0.smithi085.stdout:2223: finishing write tid 1 to smithi08575397-5 2021-02-10T04:36:38.606 INFO:tasks.rados.rados.0.smithi085.stdout:2224: finishing copy_from to smithi08575397-31 2021-02-10T04:36:38.606 INFO:tasks.rados.rados.0.smithi085.stdout:2224: got expected ENOENT (src dne) 2021-02-10T04:36:38.606 INFO:tasks.rados.rados.0.smithi085.stdout:2224: finishing copy_from racing read to smithi08575397-31 2021-02-10T04:36:38.606 INFO:tasks.rados.rados.0.smithi085.stdout:2223: finishing write tid 2 to smithi08575397-5 2021-02-10T04:36:38.606 INFO:tasks.rados.rados.0.smithi085.stdout:2223: finishing write tid 3 to smithi08575397-5 2021-02-10T04:36:38.606 INFO:tasks.rados.rados.0.smithi085.stdout:2223: finishing write tid 4 to smithi08575397-5 2021-02-10T04:36:38.606 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 759 (ObjNum 776 snap 250 seq_num 776) dirty exists 2021-02-10T04:36:38.606 INFO:tasks.rados.rados.0.smithi085.stdout:2223: left oid 5 (ObjNum 776 snap 250 seq_num 776) 2021-02-10T04:36:38.607 INFO:tasks.rados.rados.0.smithi085.stdout:2222: done (12 left) 2021-02-10T04:36:38.608 INFO:tasks.rados.rados.0.smithi085.stdout:2223: done (11 left) 2021-02-10T04:36:38.608 INFO:tasks.rados.rados.0.smithi085.stdout:2224: done (10 left) 2021-02-10T04:36:38.608 INFO:tasks.rados.rados.0.smithi085.stdout:2225: done (9 left) 2021-02-10T04:36:38.608 INFO:tasks.rados.rados.0.smithi085.stdout:2226: read oid 16 snap 202 2021-02-10T04:36:38.608 INFO:tasks.rados.rados.0.smithi085.stdout:2226: expect (ObjNum 636 snap 196 seq_num 636) 2021-02-10T04:36:38.608 INFO:tasks.rados.rados.0.smithi085.stdout:2227: rollback oid 48 current snap is 250 2021-02-10T04:36:38.608 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 48 to 221 2021-02-10T04:36:38.608 INFO:tasks.rados.rados.0.smithi085.stdout:2228: rollback oid 21 current snap is 250 2021-02-10T04:36:38.608 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 21 to 249 2021-02-10T04:36:38.608 INFO:tasks.rados.rados.0.smithi085.stdout:2229: rollback oid 7 current snap is 250 2021-02-10T04:36:38.609 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 7 to 241 2021-02-10T04:36:38.609 INFO:tasks.rados.rados.0.smithi085.stdout:2230: write (excl) oid 22 current snap is 250 2021-02-10T04:36:38.609 INFO:tasks.rados.rados.0.smithi085.stdout:2230: seq_num 778 ranges {675846=411728,1543579=787592,2848853=226993} 2021-02-10T04:36:38.613 INFO:tasks.rados.rados.0.smithi085.stdout:2230: writing smithi08575397-22 from 675846 to 1087574 tid 1 2021-02-10T04:36:38.621 INFO:tasks.rados.rados.0.smithi085.stdout:2230: writing smithi08575397-22 from 1543579 to 2331171 tid 2 2021-02-10T04:36:38.626 INFO:tasks.rados.rados.0.smithi085.stdout:2230: writing smithi08575397-22 from 2848853 to 3075846 tid 3 2021-02-10T04:36:38.626 INFO:tasks.rados.rados.0.smithi085.stdout:2231: rollback oid 8 current snap is 250 2021-02-10T04:36:38.626 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 8 to 239 2021-02-10T04:36:38.626 INFO:tasks.rados.rados.0.smithi085.stdout:2227: finishing rollback tid 0 to smithi08575397-48 2021-02-10T04:36:38.627 INFO:tasks.rados.rados.0.smithi085.stdout:2227: finishing rollback tid 1 to smithi08575397-48 2021-02-10T04:36:38.627 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 839 (ObjNum 82790512 snap 22030 seq_num 0) dirty dne 2021-02-10T04:36:38.627 INFO:tasks.rados.rados.0.smithi085.stdout:2229: finishing rollback tid 0 to smithi08575397-7 2021-02-10T04:36:38.627 INFO:tasks.rados.rados.0.smithi085.stdout:2229: finishing rollback tid 1 to smithi08575397-7 2021-02-10T04:36:38.627 INFO:tasks.rados.rados.0.smithi085.stdout:2229: finishing rollback tid 2 to smithi08575397-7 2021-02-10T04:36:38.627 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 1120 (ObjNum 727 snap 230 seq_num 727) dirty exists 2021-02-10T04:36:38.627 INFO:tasks.rados.rados.0.smithi085.stdout:2226: expect (ObjNum 636 snap 196 seq_num 636) 2021-02-10T04:36:38.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[29319]: audit 2021-02-10T04:36:37.294278 2021-02-10T04:36:38.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1424 : audit [INF] from='client.? 172.21.15.85:0/2509787274' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-02-10T04:36:38.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:38.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[29319]: 02-10T04:36:37.294353+0000 mon.a (mon.0) 1425 : cluster [DBG] osdmap e580: 8 total, 8 up, 8 in 2021-02-10T04:36:38.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[29319]: audit 2021-02-10T04:36:38.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[29319]: 2021-02 2021-02-10T04:36:38.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[29319]: -10T04:36:38.101299+0000 mon.a 2021-02-10T04:36:38.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[29319]: (mon.0) 1426 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:36:38.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:36:38.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[29319]: 10T04:36:38.101526+0000 mon.a (mon.0 2021-02-10T04:36:38.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[29319]: ) 1427 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:36:38.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:38.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[29319]: -10T04:36:38.122075+0000 mgr.y (mgr.14140) 639 : cluster 2021-02-10T04:36:38.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[29319]: [DBG] pgmap v1020: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 414 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 1.4 MiB/s wr, 7 op/s 2021-02-10T04:36:38.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:36:38.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[29319]: -10T04:36:38.156993+0000 2021-02-10T04:36:38.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[29319]: mgr.y (mgr.14140) 640 : audit [DBG] 2021-02-10T04:36:38.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[29319]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:36:38.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[29319]: cluster 2021-02-10T04:36:38.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[29319]: 2021-02-10T04:36: 2021-02-10T04:36:38.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[29319]: 38.182470+0000 mon.a (mon.0) 2021-02-10T04:36:38.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[29319]: 1428 : cluster [DBG] osdmap e581: 8 total, 8 up, 8 in 2021-02-10T04:36:38.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[35961]: audit 2021-02-10T04:36:37.294278+0000 2021-02-10T04:36:38.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[35961]: mon.a (mon.0) 1424 : audit [INF] from='client.? 172.21.15.85:0/2509787274' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-02-10T04:36:38.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[35961]: cluster 2021-02-10T04:36:37.294353+0000 mon.a (mon. 2021-02-10T04:36:38.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[35961]: 0) 1425 : cluster [DBG] osdmap e580: 8 total, 8 up, 8 in 2021-02-10T04:36:38.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:36:38.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[35961]: 04:36:38.101299+0000 mon.a (mon.0) 1426 : audit [INF] 2021-02-10T04:36:38.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:36:38.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[35961]: audit 2021 2021-02-10T04:36:38.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[35961]: -02-10T04:36:38.101526+0000 mon.a (mon.0) 1427 2021-02-10T04:36:38.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[35961]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:36:38.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[35961]: cluster 2021-02-10T04:36:38.122075 2021-02-10T04:36:38.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 639 : cluster [DBG] 2021-02-10T04:36:38.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[35961]: pgmap v1020: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 414 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 1.4 MiB/s wr, 7 op/s 2021-02-10T04:36:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:36:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[35961]: 02-10T04:36:38.156993+0000 mgr.y (mgr.14140) 640 : 2021-02-10T04:36:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[35961]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:36:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[35961]: cluster 2021-02-10T04:36:38.182470 2021-02-10T04:36:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1428 : cluster [DBG] 2021-02-10T04:36:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:38 smithi085 conmon[35961]: osdmap e581: 8 total, 8 up, 8 in 2021-02-10T04:36:38.831 INFO:tasks.rados.rados.0.smithi085.stdout:2228: finishing rollback tid 0 to smithi08575397-21 2021-02-10T04:36:38.831 INFO:tasks.rados.rados.0.smithi085.stdout:2228: finishing rollback tid 1 to smithi08575397-21 2021-02-10T04:36:38.831 INFO:tasks.rados.rados.0.smithi085.stdout:2228: finishing rollback tid 2 to smithi08575397-21 2021-02-10T04:36:38.831 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 1210 (ObjNum 431 snap 114 seq_num 431) dirty exists 2021-02-10T04:36:38.831 INFO:tasks.rados.rados.0.smithi085.stdout:2230: finishing write tid 1 to smithi08575397-22 2021-02-10T04:36:38.831 INFO:tasks.rados.rados.0.smithi085.stdout:2231: finishing rollback tid 0 to smithi08575397-8 2021-02-10T04:36:38.832 INFO:tasks.rados.rados.0.smithi085.stdout:2230: finishing write tid 2 to smithi08575397-22 2021-02-10T04:36:38.832 INFO:tasks.rados.rados.0.smithi085.stdout:2231: finishing rollback tid 1 to smithi08575397-8 2021-02-10T04:36:38.832 INFO:tasks.rados.rados.0.smithi085.stdout:2230: finishing write tid 3 to smithi08575397-22 2021-02-10T04:36:38.832 INFO:tasks.rados.rados.0.smithi085.stdout:2230: finishing write tid 4 to smithi08575397-22 2021-02-10T04:36:38.832 INFO:tasks.rados.rados.0.smithi085.stdout:2231: finishing rollback tid 2 to smithi08575397-8 2021-02-10T04:36:38.832 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 1124 (ObjNum 743 snap 239 seq_num 743) dirty exists 2021-02-10T04:36:38.832 INFO:tasks.rados.rados.0.smithi085.stdout:2230: finishing write tid 5 to smithi08575397-22 2021-02-10T04:36:38.832 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 796 (ObjNum 778 snap 250 seq_num 778) dirty exists 2021-02-10T04:36:38.832 INFO:tasks.rados.rados.0.smithi085.stdout:2230: left oid 22 (ObjNum 778 snap 250 seq_num 778) 2021-02-10T04:36:38.832 INFO:tasks.rados.rados.0.smithi085.stdout:2226: done (14 left) 2021-02-10T04:36:38.832 INFO:tasks.rados.rados.0.smithi085.stdout:2227: done (13 left) 2021-02-10T04:36:38.832 INFO:tasks.rados.rados.0.smithi085.stdout:2228: done (12 left) 2021-02-10T04:36:38.832 INFO:tasks.rados.rados.0.smithi085.stdout:2229: done (11 left) 2021-02-10T04:36:38.833 INFO:tasks.rados.rados.0.smithi085.stdout:2230: done (10 left) 2021-02-10T04:36:38.833 INFO:tasks.rados.rados.0.smithi085.stdout:2231: done (9 left) 2021-02-10T04:36:38.833 INFO:tasks.rados.rados.0.smithi085.stdout:2232: write oid 12 current snap is 250 2021-02-10T04:36:38.833 INFO:tasks.rados.rados.0.smithi085.stdout:2232: seq_num 779 ranges {692653=486977,1602061=647603,2992629=100024} 2021-02-10T04:36:38.838 INFO:tasks.rados.rados.0.smithi085.stdout:2232: writing smithi08575397-12 from 692653 to 1179630 tid 1 2021-02-10T04:36:38.845 INFO:tasks.rados.rados.0.smithi085.stdout:2232: writing smithi08575397-12 from 1602061 to 2249664 tid 2 2021-02-10T04:36:38.852 INFO:tasks.rados.rados.0.smithi085.stdout:2232: writing smithi08575397-12 from 2992629 to 3092653 tid 3 2021-02-10T04:36:38.852 INFO:tasks.rados.rados.0.smithi085.stdout:2232: finishing write tid 1 to smithi08575397-12 2021-02-10T04:36:38.852 INFO:tasks.rados.rados.0.smithi085.stdout:2232: finishing write tid 2 to smithi08575397-12 2021-02-10T04:36:38.852 INFO:tasks.rados.rados.0.smithi085.stdout:2233: read oid 35 snap -1 2021-02-10T04:36:38.852 INFO:tasks.rados.rados.0.smithi085.stdout:2233: expect (ObjNum 578 snap 176 seq_num 578) 2021-02-10T04:36:38.852 INFO:tasks.rados.rados.0.smithi085.stdout:2234: copy_from oid 1 from oid 5 current snap is 250 2021-02-10T04:36:38.852 INFO:tasks.rados.rados.0.smithi085.stdout:2235: read oid 43 snap -1 2021-02-10T04:36:38.852 INFO:tasks.rados.rados.0.smithi085.stdout:2235: expect (ObjNum 770 snap 250 seq_num 770) 2021-02-10T04:36:38.852 INFO:tasks.rados.rados.0.smithi085.stdout:2236: snap_create 2021-02-10T04:36:38.856 INFO:tasks.rados.rados.0.smithi085.stdout:2232: finishing write tid 3 to smithi08575397-12 2021-02-10T04:36:38.856 INFO:tasks.rados.rados.0.smithi085.stdout:2232: finishing write tid 4 to smithi08575397-12 2021-02-10T04:36:38.857 INFO:tasks.rados.rados.0.smithi085.stdout:2232: finishing write tid 5 to smithi08575397-12 2021-02-10T04:36:38.857 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 843 (ObjNum 779 snap 250 seq_num 779) dirty exists 2021-02-10T04:36:38.857 INFO:tasks.rados.rados.0.smithi085.stdout:2232: left oid 12 (ObjNum 779 snap 250 seq_num 779) 2021-02-10T04:36:38.858 INFO:tasks.rados.rados.0.smithi085.stdout:2233: expect (ObjNum 578 snap 176 seq_num 578) 2021-02-10T04:36:39.212 INFO:tasks.rados.rados.0.smithi085.stdout:2235: expect (ObjNum 770 snap 250 seq_num 770) 2021-02-10T04:36:39.412 INFO:tasks.rados.rados.0.smithi085.stdout:2234: finishing copy_from to smithi08575397-1 2021-02-10T04:36:39.413 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 759 (ObjNum 776 snap 250 seq_num 776) dirty exists 2021-02-10T04:36:39.413 INFO:tasks.rados.rados.0.smithi085.stdout:2234: finishing copy_from racing read to smithi08575397-1 2021-02-10T04:36:39.413 INFO:tasks.rados.rados.0.smithi085.stdout:2232: done (13 left) 2021-02-10T04:36:39.413 INFO:tasks.rados.rados.0.smithi085.stdout:2233: done (12 left) 2021-02-10T04:36:39.413 INFO:tasks.rados.rados.0.smithi085.stdout:2234: done (11 left) 2021-02-10T04:36:39.413 INFO:tasks.rados.rados.0.smithi085.stdout:2235: done (10 left) 2021-02-10T04:36:39.413 INFO:tasks.rados.rados.0.smithi085.stdout:2236: done (9 left) 2021-02-10T04:36:39.413 INFO:tasks.rados.rados.0.smithi085.stdout:2237: snap_remove snap 235 2021-02-10T04:36:40.331 INFO:tasks.rados.rados.0.smithi085.stdout:2237: done (9 left) 2021-02-10T04:36:40.331 INFO:tasks.rados.rados.0.smithi085.stdout:2238: write (excl) oid 21 current snap is 251 2021-02-10T04:36:40.331 INFO:tasks.rados.rados.0.smithi085.stdout:2238: seq_num 780 ranges {709460=562226,2060543=791261} 2021-02-10T04:36:40.339 INFO:tasks.rados.rados.0.smithi085.stdout:2238: writing smithi08575397-21 from 709460 to 1271686 tid 1 2021-02-10T04:36:40.350 INFO:tasks.rados.rados.0.smithi085.stdout:2238: writing smithi08575397-21 from 2060543 to 2851804 tid 2 2021-02-10T04:36:40.350 INFO:tasks.rados.rados.0.smithi085.stdout:2239: rollback oid 46 current snap is 251 2021-02-10T04:36:40.350 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 46 to 221 2021-02-10T04:36:40.350 INFO:tasks.rados.rados.0.smithi085.stdout:2238: finishing write tid 1 to smithi08575397-21 2021-02-10T04:36:40.350 INFO:tasks.rados.rados.0.smithi085.stdout:2240: write oid 38 current snap is 251 2021-02-10T04:36:40.350 INFO:tasks.rados.rados.0.smithi085.stdout:2240: seq_num 781 ranges {726267=637475,2119025=651272} 2021-02-10T04:36:40.358 INFO:tasks.rados.rados.0.smithi085.stdout:2240: writing smithi08575397-38 from 726267 to 1363742 tid 1 2021-02-10T04:36:40.368 INFO:tasks.rados.rados.0.smithi085.stdout:2240: writing smithi08575397-38 from 2119025 to 2770297 tid 2 2021-02-10T04:36:40.368 INFO:tasks.rados.rados.0.smithi085.stdout:2241: copy_from oid 10 from oid 42 current snap is 251 2021-02-10T04:36:40.368 INFO:tasks.rados.rados.0.smithi085.stdout:2242: copy_from oid 20 from oid 16 current snap is 251 2021-02-10T04:36:40.368 INFO:tasks.rados.rados.0.smithi085.stdout:2243: read oid 2 snap -1 2021-02-10T04:36:40.368 INFO:tasks.rados.rados.0.smithi085.stdout:2243: expect (ObjNum 765 snap 246 seq_num 765) 2021-02-10T04:36:40.368 INFO:tasks.rados.rados.0.smithi085.stdout:2244: rollback oid 11 current snap is 251 2021-02-10T04:36:40.368 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 11 to 221 2021-02-10T04:36:40.368 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:36:40.369 INFO:tasks.rados.rados.0.smithi085.stdout:2239: finishing rollback tid 0 to smithi08575397-46 2021-02-10T04:36:40.369 INFO:tasks.rados.rados.0.smithi085.stdout:2239: finishing rollback tid 1 to smithi08575397-46 2021-02-10T04:36:40.369 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 1058 (ObjNum 83150640 snap 22030 seq_num 82306272) dirty dne 2021-02-10T04:36:40.369 INFO:tasks.rados.rados.0.smithi085.stdout:2238: finishing write tid 2 to smithi08575397-21 2021-02-10T04:36:40.369 INFO:tasks.rados.rados.0.smithi085.stdout:2239: done (15 left) 2021-02-10T04:36:40.369 INFO:tasks.rados.rados.0.smithi085.stdout:2245: rollback oid 23 current snap is 251 2021-02-10T04:36:40.369 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 23 to 247 2021-02-10T04:36:40.369 INFO:tasks.rados.rados.0.smithi085.stdout:2238: finishing write tid 3 to smithi08575397-21 2021-02-10T04:36:40.369 INFO:tasks.rados.rados.0.smithi085.stdout:2238: finishing write tid 4 to smithi08575397-21 2021-02-10T04:36:40.369 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 1214 (ObjNum 780 snap 251 seq_num 780) dirty exists 2021-02-10T04:36:40.369 INFO:tasks.rados.rados.0.smithi085.stdout:2238: left oid 21 (ObjNum 780 snap 251 seq_num 780) 2021-02-10T04:36:40.369 INFO:tasks.rados.rados.0.smithi085.stdout:2240: finishing write tid 1 to smithi08575397-38 2021-02-10T04:36:40.369 INFO:tasks.rados.rados.0.smithi085.stdout:2238: done (15 left) 2021-02-10T04:36:40.370 INFO:tasks.rados.rados.0.smithi085.stdout:2246: snap_create 2021-02-10T04:36:40.373 INFO:tasks.rados.rados.0.smithi085.stdout:2245: finishing rollback tid 0 to smithi08575397-23 2021-02-10T04:36:40.374 INFO:tasks.rados.rados.0.smithi085.stdout:2245: finishing rollback tid 1 to smithi08575397-23 2021-02-10T04:36:40.374 INFO:tasks.rados.rados.0.smithi085.stdout:2240: finishing write tid 2 to smithi08575397-38 2021-02-10T04:36:40.374 INFO:tasks.rados.rados.0.smithi085.stdout:2245: finishing rollback tid 2 to smithi08575397-23 2021-02-10T04:36:40.374 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 693 (ObjNum 722 snap 226 seq_num 722) dirty exists 2021-02-10T04:36:40.375 INFO:tasks.rados.rados.0.smithi085.stdout:2240: finishing write tid 3 to smithi08575397-38 2021-02-10T04:36:40.375 INFO:tasks.rados.rados.0.smithi085.stdout:2242: finishing copy_from to smithi08575397-20 2021-02-10T04:36:40.375 INFO:tasks.rados.rados.0.smithi085.stdout:2242: got expected ENOENT (src dne) 2021-02-10T04:36:40.375 INFO:tasks.rados.rados.0.smithi085.stdout:2240: finishing write tid 4 to smithi08575397-38 2021-02-10T04:36:40.376 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 1061 (ObjNum 781 snap 251 seq_num 781) dirty exists 2021-02-10T04:36:40.376 INFO:tasks.rados.rados.0.smithi085.stdout:2240: left oid 38 (ObjNum 781 snap 251 seq_num 781) 2021-02-10T04:36:40.376 INFO:tasks.rados.rados.0.smithi085.stdout:2242: finishing copy_from racing read to smithi08575397-20 2021-02-10T04:36:40.381 INFO:tasks.rados.rados.0.smithi085.stdout:2243: expect (ObjNum 765 snap 246 seq_num 765) 2021-02-10T04:36:40.604 INFO:tasks.rados.rados.0.smithi085.stdout:2244: finishing rollback tid 0 to smithi08575397-11 2021-02-10T04:36:40.604 INFO:tasks.rados.rados.0.smithi085.stdout:2244: finishing rollback tid 1 to smithi08575397-11 2021-02-10T04:36:40.604 INFO:tasks.rados.rados.0.smithi085.stdout:2244: finishing rollback tid 2 to smithi08575397-11 2021-02-10T04:36:40.604 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 1218 (ObjNum 681 snap 210 seq_num 681) dirty exists 2021-02-10T04:36:40.604 INFO:tasks.rados.rados.0.smithi085.stdout:2241: finishing copy_from to smithi08575397-10 2021-02-10T04:36:40.604 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 10 v 837 (ObjNum 772 snap 250 seq_num 772) dirty exists 2021-02-10T04:36:40.604 INFO:tasks.rados.rados.0.smithi085.stdout:2241: finishing copy_from racing read to smithi08575397-10 2021-02-10T04:36:40.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[29319]: cluster 2021-02-10T04:36:40.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[29319]: 2021-02-10T04:36:40.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[29319]: 2021 2021-02-10T04:36:40.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[29319]: - 2021-02-10T04:36:40.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[29319]: 02-10T04:36: 2021-02-10T04:36:40.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[29319]: 39.321693+0000 mon.a (mon.0) 1429 : cluster [DBG] osdmap e582: 8 total, 8 up, 8 in 2021-02-10T04:36:40.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:36:40.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[29319]: 36:39.721154+0000 osd.7 (osd.7) 54 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:36:40.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[29319]: cluster 2021-02-10T04:36:39.721420+0000 2021-02-10T04:36:40.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[29319]: osd.7 (osd.7) 55 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:36:40.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:36:40.616 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[29319]: T04:36:40.122648+0000 2021-02-10T04:36:40.616 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[29319]: mgr.y (mgr.14140) 641 : cluster [DBG] 2021-02-10T04:36:40.616 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[29319]: pgmap v1023: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 426 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 3.7 MiB/s wr, 26 op/s 2021-02-10T04:36:40.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: cluster 2021-02-10T04:36:40.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 2021-02-10T04:36:40.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 2021 2021-02-10T04:36:40.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: - 2021-02-10T04:36:40.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 02 2021-02-10T04:36:40.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: - 2021-02-10T04:36:40.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 10 2021-02-10T04:36:40.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: T 2021-02-10T04:36:40.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 04 2021-02-10T04:36:40.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: : 2021-02-10T04:36:40.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 36 2021-02-10T04:36:40.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: : 2021-02-10T04:36:40.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 39 2021-02-10T04:36:40.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: . 2021-02-10T04:36:40.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 321693 2021-02-10T04:36:40.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: +0000 2021-02-10T04:36:40.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 2021-02-10T04:36:40.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: mon.a 2021-02-10T04:36:40.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: ( 2021-02-10T04:36:40.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: mon 2021-02-10T04:36:40.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: . 2021-02-10T04:36:40.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 0 2021-02-10T04:36:40.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: ) 2021-02-10T04:36:40.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 1429 2021-02-10T04:36:40.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: : 2021-02-10T04:36:40.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: cluster 2021-02-10T04:36:40.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 2021-02-10T04:36:40.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: [DBG] 2021-02-10T04:36:40.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 2021-02-10T04:36:40.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: osdmap e582: 8 total, 8 up, 8 in 2021-02-10T04:36:40.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 2021-02-10T04:36:40.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: cluster 2021-02-10T04:36:40.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 2021-02-10T04:36:40.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 2021 2021-02-10T04:36:40.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: - 2021-02-10T04:36:40.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 02 2021-02-10T04:36:40.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: - 2021-02-10T04:36:40.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 10 2021-02-10T04:36:40.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: T 2021-02-10T04:36:40.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 04 2021-02-10T04:36:40.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: : 2021-02-10T04:36:40.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 36 2021-02-10T04:36:40.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: : 2021-02-10T04:36:40.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 39 2021-02-10T04:36:40.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: . 2021-02-10T04:36:40.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 721154 2021-02-10T04:36:40.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: +0000 2021-02-10T04:36:40.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 2021-02-10T04:36:40.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: osd.7 2021-02-10T04:36:40.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: ( 2021-02-10T04:36:40.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: osd 2021-02-10T04:36:40.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: .7) 54 : cluster 2021-02-10T04:36:40.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: [DBG] purged_snaps scrub starts 2021-02-10T04:36:40.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:40.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 02-10T04:36:39.721420+0000 osd.7 ( 2021-02-10T04:36:40.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: osd.7) 55 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:36:40.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: cluster 2021-02-10T04:36:40.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 2021-02-10T04:36:40.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 2021 2021-02-10T04:36:40.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: - 2021-02-10T04:36:40.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 02 2021-02-10T04:36:40.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: - 2021-02-10T04:36:40.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 10 2021-02-10T04:36:40.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: T 2021-02-10T04:36:40.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 04 2021-02-10T04:36:40.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: : 2021-02-10T04:36:40.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 36 2021-02-10T04:36:40.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: : 2021-02-10T04:36:40.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 40 2021-02-10T04:36:40.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: . 2021-02-10T04:36:40.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 122648 2021-02-10T04:36:40.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: +0000 2021-02-10T04:36:40.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 2021-02-10T04:36:40.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: mgr.y 2021-02-10T04:36:40.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: ( 2021-02-10T04:36:40.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: mgr 2021-02-10T04:36:40.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: . 2021-02-10T04:36:40.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 14140 2021-02-10T04:36:40.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: ) 2021-02-10T04:36:40.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 641 2021-02-10T04:36:40.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: : 2021-02-10T04:36:40.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: cluster 2021-02-10T04:36:40.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 2021-02-10T04:36:40.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: [DBG] 2021-02-10T04:36:40.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 2021-02-10T04:36:40.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: pgmap v1023: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 426 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 3.7 MiB/s wr, 26 op/s 2021-02-10T04:36:40.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:40 smithi085 conmon[35961]: 2021-02-10T04:36:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:40 smithi164 conmon[29288]: cluster 2021-02-10T04:36:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:40 smithi164 conmon[29288]: 2021-02-10T04:36:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:40 smithi164 conmon[29288]: 2021-02-10T04:36:39. 2021-02-10T04:36:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:40 smithi164 conmon[29288]: 321693+0000 mon.a (mon.0) 1429 : cluster [DBG] osdmap e582: 8 total, 8 up, 8 in 2021-02-10T04:36:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:40 smithi164 conmon[29288]: cluster 2021-02-10T04:36:39.721154 2021-02-10T04:36:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:40 smithi164 conmon[29288]: +0000 osd.7 (osd.7) 54 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:36:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:40 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:36:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:40 smithi164 conmon[29288]: :36:39.721420+0000 osd.7 (osd.7) 2021-02-10T04:36:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:40 smithi164 conmon[29288]: 55 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:36:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:40 smithi164 conmon[29288]: cluster 2021-02-10T04:36: 2021-02-10T04:36:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:40 smithi164 conmon[29288]: 40.122648+0000 mgr.y (mgr.14140) 641 : cluster [DBG] pgmap v1023: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 426 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 3.7 MiB/s wr, 26 op/s 2021-02-10T04:36:40.724 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:36:40.749 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:36:41.283 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.6 dump_historic_ops 2021-02-10T04:36:41.335 INFO:tasks.rados.rados.0.smithi085.stdout:2240: done (15 left) 2021-02-10T04:36:41.335 INFO:tasks.rados.rados.0.smithi085.stdout:2241: done (14 left) 2021-02-10T04:36:41.335 INFO:tasks.rados.rados.0.smithi085.stdout:2242: done (13 left) 2021-02-10T04:36:41.335 INFO:tasks.rados.rados.0.smithi085.stdout:2243: done (12 left) 2021-02-10T04:36:41.335 INFO:tasks.rados.rados.0.smithi085.stdout:2244: done (11 left) 2021-02-10T04:36:41.335 INFO:tasks.rados.rados.0.smithi085.stdout:2245: done (10 left) 2021-02-10T04:36:41.335 INFO:tasks.rados.rados.0.smithi085.stdout:2246: done (9 left) 2021-02-10T04:36:41.335 INFO:tasks.rados.rados.0.smithi085.stdout:2247: snap_create 2021-02-10T04:36:41.589 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:36:41.594 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:36:41.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:41 smithi164 conmon[29288]: cluster 2021-02-10T04:36: 2021-02-10T04:36:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:41 smithi164 conmon[29288]: 40.330685+0000 mon.a (mon.0) 1430 : cluster [DBG] osdmap e583: 8 total, 8 up, 8 in 2021-02-10T04:36:41.691 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[52836]: debug 2021-02-10T04:36:41.278+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:36:41.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: cluster 2021-02-10T04:36:41.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: 2021-02-10T04:36:41.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: 2021 2021-02-10T04:36:41.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: - 2021-02-10T04:36:41.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: 02 2021-02-10T04:36:41.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: - 2021-02-10T04:36:41.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: 10 2021-02-10T04:36:41.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: T 2021-02-10T04:36:41.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: 04 2021-02-10T04:36:41.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: : 2021-02-10T04:36:41.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: 36 2021-02-10T04:36:41.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: : 2021-02-10T04:36:41.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: 40 2021-02-10T04:36:41.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: . 2021-02-10T04:36:41.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: 330685 2021-02-10T04:36:41.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: +0000 2021-02-10T04:36:41.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: 2021-02-10T04:36:41.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: mon.a 2021-02-10T04:36:41.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: ( 2021-02-10T04:36:41.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: mon 2021-02-10T04:36:41.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: . 2021-02-10T04:36:41.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: 0 2021-02-10T04:36:41.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: ) 2021-02-10T04:36:41.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: 1430 2021-02-10T04:36:41.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: : 2021-02-10T04:36:41.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: cluster 2021-02-10T04:36:41.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: 2021-02-10T04:36:41.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: [DBG] 2021-02-10T04:36:41.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: 2021-02-10T04:36:41.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: osdmap e583: 8 total, 8 up, 8 in 2021-02-10T04:36:41.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[29319]: 2021-02-10T04:36:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[35961]: cluster 2021-02-10T04:36:40.330685+0000 mon.a ( 2021-02-10T04:36:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:41 smithi085 conmon[35961]: mon.0) 1430 : cluster [DBG] osdmap e583: 8 total, 8 up, 8 in 2021-02-10T04:36:42.345 INFO:tasks.rados.rados.0.smithi085.stdout:2247: done (9 left) 2021-02-10T04:36:42.346 INFO:tasks.rados.rados.0.smithi085.stdout:2248: read oid 31 snap 250 2021-02-10T04:36:42.346 INFO:tasks.rados.rados.0.smithi085.stdout:2248: expect (ObjNum 768 snap 249 seq_num 768) 2021-02-10T04:36:42.346 INFO:tasks.rados.rados.0.smithi085.stdout:2249: copy_from oid 22 from oid 2 current snap is 253 2021-02-10T04:36:42.346 INFO:tasks.rados.rados.0.smithi085.stdout:2250: snap_remove snap 246 2021-02-10T04:36:42.600 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:42 smithi164 conmon[29288]: cluster 2021-02-10T04:36:42.600 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:42 smithi164 conmon[29288]: 2021-02-10T04:36:42.600 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:42 smithi164 conmon[29288]: 2021 2021-02-10T04:36:42.600 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:42 smithi164 conmon[29288]: -02-10T04:36:41 2021-02-10T04:36:42.600 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:42 smithi164 conmon[29288]: .334199+0000 mon.a (mon.0) 1431 : cluster [DBG] osdmap e584: 8 total, 8 up, 8 in 2021-02-10T04:36:42.600 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:42 smithi164 conmon[29288]: cluster 2021-02-10T04:36:42.600 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:42 smithi164 conmon[29288]: 2021-02-10T04:36 2021-02-10T04:36:42.600 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:42 smithi164 conmon[29288]: :42.123119+0000 mgr.y (mgr.14140) 642 : cluster [DBG] pgmap v1026: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 430 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 4.1 MiB/s wr, 33 op/s 2021-02-10T04:36:42.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:42 smithi085 conmon[35961]: cluster 2021-02-10T04:36:42.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:42 smithi085 conmon[35961]: 2021-02-10T04:36:41.334199+0000 mon.a (mon.0) 2021-02-10T04:36:42.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:42 smithi085 conmon[35961]: 1431 : cluster [DBG] osdmap e584: 8 total, 8 up, 8 in 2021-02-10T04:36:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:42 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:42 smithi085 conmon[35961]: -10T04:36:42.123119+0000 mgr.y (mgr.14140) 642 : cluster 2021-02-10T04:36:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:42 smithi085 conmon[35961]: [DBG] pgmap v1026: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 430 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 4.1 MiB/s wr, 33 op/s 2021-02-10T04:36:42.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:42 smithi085 conmon[29319]: cluster 2021-02-10T04:36:41.334199+0000 mon.a 2021-02-10T04:36:42.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:42 smithi085 conmon[29319]: (mon.0) 1431 : cluster [DBG] osdmap e584: 8 total, 8 up, 8 in 2021-02-10T04:36:42.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:42 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:36:42.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:42 smithi085 conmon[29319]: :36:42.123119+0000 mgr.y (mgr.14140) 642 : cluster [DBG] pgmap v1026: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 430 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 4.1 MiB/s wr, 33 op/s 2021-02-10T04:36:43.345 INFO:tasks.rados.rados.0.smithi085.stdout:2250: done (11 left) 2021-02-10T04:36:43.345 INFO:tasks.rados.rados.0.smithi085.stdout:2251: read oid 13 snap -1 2021-02-10T04:36:43.345 INFO:tasks.rados.rados.0.smithi085.stdout:2251: expect deleted 2021-02-10T04:36:43.345 INFO:tasks.rados.rados.0.smithi085.stdout:2252: write oid 7 current snap is 253 2021-02-10T04:36:43.345 INFO:tasks.rados.rados.0.smithi085.stdout:2252: seq_num 782 ranges {743074=712724,2061154=794930} 2021-02-10T04:36:43.352 INFO:tasks.rados.rados.0.smithi085.stdout:2252: writing smithi08575397-7 from 743074 to 1455798 tid 1 2021-02-10T04:36:43.361 INFO:tasks.rados.rados.0.smithi085.stdout:2252: writing smithi08575397-7 from 2061154 to 2856084 tid 2 2021-02-10T04:36:43.361 INFO:tasks.rados.rados.0.smithi085.stdout:2253: read oid 38 snap -1 2021-02-10T04:36:43.361 INFO:tasks.rados.rados.0.smithi085.stdout:2253: expect (ObjNum 781 snap 251 seq_num 781) 2021-02-10T04:36:43.361 INFO:tasks.rados.rados.0.smithi085.stdout:2254: read oid 43 snap -1 2021-02-10T04:36:43.361 INFO:tasks.rados.rados.0.smithi085.stdout:2254: expect (ObjNum 770 snap 250 seq_num 770) 2021-02-10T04:36:43.361 INFO:tasks.rados.rados.0.smithi085.stdout:2255: read oid 28 snap -1 2021-02-10T04:36:43.361 INFO:tasks.rados.rados.0.smithi085.stdout:2255: expect (ObjNum 651 snap 197 seq_num 651) 2021-02-10T04:36:43.361 INFO:tasks.rados.rados.0.smithi085.stdout:2248: expect (ObjNum 768 snap 249 seq_num 768) 2021-02-10T04:36:43.591 INFO:tasks.rados.rados.0.smithi085.stdout:2249: finishing copy_from to smithi08575397-22 2021-02-10T04:36:43.591 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 677 (ObjNum 765 snap 246 seq_num 765) dirty exists 2021-02-10T04:36:43.591 INFO:tasks.rados.rados.0.smithi085.stdout:2249: finishing copy_from racing read to smithi08575397-22 2021-02-10T04:36:43.591 INFO:tasks.rados.rados.0.smithi085.stdout:2248: done (15 left) 2021-02-10T04:36:43.591 INFO:tasks.rados.rados.0.smithi085.stdout:2249: done (14 left) 2021-02-10T04:36:43.591 INFO:tasks.rados.rados.0.smithi085.stdout:2256: rollback oid 44 current snap is 253 2021-02-10T04:36:43.591 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 44 to 252 2021-02-10T04:36:43.591 INFO:tasks.rados.rados.0.smithi085.stdout:2252: finishing write tid 1 to smithi08575397-7 2021-02-10T04:36:43.592 INFO:tasks.rados.rados.0.smithi085.stdout:2255: expect (ObjNum 651 snap 197 seq_num 651) 2021-02-10T04:36:43.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:43 smithi164 conmon[29288]: cluster 2021-02-10T04:36:42.344556+0000 mon.a (mon 2021-02-10T04:36:43.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:43 smithi164 conmon[29288]: .0) 1432 : cluster [DBG] osdmap e585: 8 total, 8 up, 8 in 2021-02-10T04:36:43.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:43 smithi085 conmon[29319]: cluster 2021-02-10T04:36:42.344556+0000 mon.a 2021-02-10T04:36:43.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:43 smithi085 conmon[29319]: (mon.0) 1432 : cluster [DBG] osdmap e585: 8 total, 8 up, 8 in 2021-02-10T04:36:43.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:43 smithi085 conmon[35961]: cluster 2021-02-10T04:36:42.344556+0000 2021-02-10T04:36:43.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:43 smithi085 conmon[35961]: mon.a (mon.0) 1432 : cluster [DBG] osdmap e585: 8 total, 8 up, 8 in 2021-02-10T04:36:43.819 INFO:tasks.rados.rados.0.smithi085.stdout:2251: done (14 left) 2021-02-10T04:36:43.819 INFO:tasks.rados.rados.0.smithi085.stdout:2255: done (13 left) 2021-02-10T04:36:43.819 INFO:tasks.rados.rados.0.smithi085.stdout:2257: snap_create 2021-02-10T04:36:43.819 INFO:tasks.rados.rados.0.smithi085.stdout:2254: expect (ObjNum 770 snap 250 seq_num 770) 2021-02-10T04:36:44.013 INFO:tasks.rados.rados.0.smithi085.stdout:2252: finishing write tid 2 to smithi08575397-7 2021-02-10T04:36:44.013 INFO:tasks.rados.rados.0.smithi085.stdout:2252: finishing write tid 3 to smithi08575397-7 2021-02-10T04:36:44.013 INFO:tasks.rados.rados.0.smithi085.stdout:2252: finishing write tid 4 to smithi08575397-7 2021-02-10T04:36:44.013 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 1123 (ObjNum 782 snap 253 seq_num 782) dirty exists 2021-02-10T04:36:44.013 INFO:tasks.rados.rados.0.smithi085.stdout:2252: left oid 7 (ObjNum 782 snap 253 seq_num 782) 2021-02-10T04:36:44.013 INFO:tasks.rados.rados.0.smithi085.stdout:2253: expect (ObjNum 781 snap 251 seq_num 781) 2021-02-10T04:36:44.162 INFO:tasks.rados.rados.0.smithi085.stdout:2256: finishing rollback tid 0 to smithi08575397-44 2021-02-10T04:36:44.162 INFO:tasks.rados.rados.0.smithi085.stdout:2256: finishing rollback tid 1 to smithi08575397-44 2021-02-10T04:36:44.162 INFO:tasks.rados.rados.0.smithi085.stdout:2256: finishing rollback tid 2 to smithi08575397-44 2021-02-10T04:36:44.162 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 1086 (ObjNum 760 snap 243 seq_num 760) dirty exists 2021-02-10T04:36:44.360 INFO:tasks.rados.rados.0.smithi085.stdout:2252: done (13 left) 2021-02-10T04:36:44.361 INFO:tasks.rados.rados.0.smithi085.stdout:2253: done (12 left) 2021-02-10T04:36:44.361 INFO:tasks.rados.rados.0.smithi085.stdout:2254: done (11 left) 2021-02-10T04:36:44.361 INFO:tasks.rados.rados.0.smithi085.stdout:2256: done (10 left) 2021-02-10T04:36:44.361 INFO:tasks.rados.rados.0.smithi085.stdout:2257: done (9 left) 2021-02-10T04:36:44.361 INFO:tasks.rados.rados.0.smithi085.stdout:2258: snap_remove snap 250 2021-02-10T04:36:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:44 smithi164 conmon[29288]: cluster 2021-02-10T04:36:43.343449+0000 mon.a (mon.0) 1433 : cluster [DBG] osdmap e586: 8 total, 8 up, 8 in 2021-02-10T04:36:44.663 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:44 smithi164 conmon[29288]: cluster 2021-02-10T04:36:44.123688+0000 mgr.y (mgr.14140) 643 : cluster [DBG] pgmap v1029: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 441 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 2.9 MiB/s wr, 19 op/s 2021-02-10T04:36:44.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: cluster 2021-02-10T04:36:44.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: 2021-02-10T04:36:44.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: 2021 2021-02-10T04:36:44.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: - 2021-02-10T04:36:44.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: 02 2021-02-10T04:36:44.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: - 2021-02-10T04:36:44.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: 10 2021-02-10T04:36:44.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: T 2021-02-10T04:36:44.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: 04 2021-02-10T04:36:44.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: : 2021-02-10T04:36:44.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: 36:43.343449+0000 2021-02-10T04:36:44.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: mon.a (mon.0) 1433 : cluster [DBG] osdmap e586: 8 total, 8 up, 8 in 2021-02-10T04:36:44.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: cluster 2021-02-10T04:36:44.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: 2021-02-10T04:36:44.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: 2021 2021-02-10T04:36:44.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: - 2021-02-10T04:36:44.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: 02 2021-02-10T04:36:44.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: - 2021-02-10T04:36:44.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: 10 2021-02-10T04:36:44.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: T 2021-02-10T04:36:44.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: 04 2021-02-10T04:36:44.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: : 2021-02-10T04:36:44.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: 36 2021-02-10T04:36:44.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: : 2021-02-10T04:36:44.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: 44 2021-02-10T04:36:44.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: . 2021-02-10T04:36:44.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: 123688 2021-02-10T04:36:44.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: +0000 2021-02-10T04:36:44.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: 2021-02-10T04:36:44.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: mgr.y 2021-02-10T04:36:44.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: ( 2021-02-10T04:36:44.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: mgr 2021-02-10T04:36:44.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: . 2021-02-10T04:36:44.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: 14140 2021-02-10T04:36:44.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: ) 2021-02-10T04:36:44.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: 643 2021-02-10T04:36:44.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: : 2021-02-10T04:36:44.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: cluster 2021-02-10T04:36:44.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: 2021-02-10T04:36:44.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: [DBG] 2021-02-10T04:36:44.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: 2021-02-10T04:36:44.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: pgmap v1029: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 441 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 2.9 MiB/s wr, 19 op/s 2021-02-10T04:36:44.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[35961]: 2021-02-10T04:36:44.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[29319]: cluster 2021-02-10T04:36:43.343449+0000 mon.a (mon.0) 1433 : cluster [DBG] osdmap e586: 8 total, 8 up, 8 in 2021-02-10T04:36:44.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:44 smithi085 conmon[29319]: cluster 2021-02-10T04:36:44.123688+0000 mgr.y (mgr.14140) 643 : cluster [DBG] pgmap v1029: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 441 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 2.9 MiB/s wr, 19 op/s 2021-02-10T04:36:45.198 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:36:45.229 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:36:45.361 INFO:tasks.rados.rados.0.smithi085.stdout:2258: done (9 left) 2021-02-10T04:36:45.361 INFO:tasks.rados.rados.0.smithi085.stdout:2259: snap_remove snap 253 2021-02-10T04:36:45.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: cluster 2021-02-10T04:36:45.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 2021-02-10T04:36:45.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 2021 2021-02-10T04:36:45.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: - 2021-02-10T04:36:45.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 02 2021-02-10T04:36:45.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: - 2021-02-10T04:36:45.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 10 2021-02-10T04:36:45.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: T 2021-02-10T04:36:45.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 04 2021-02-10T04:36:45.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: : 2021-02-10T04:36:45.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 36 2021-02-10T04:36:45.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: : 2021-02-10T04:36:45.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 44 2021-02-10T04:36:45.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: . 2021-02-10T04:36:45.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 356854 2021-02-10T04:36:45.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: +0000 2021-02-10T04:36:45.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 2021-02-10T04:36:45.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: mon.a 2021-02-10T04:36:45.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: ( 2021-02-10T04:36:45.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: mon 2021-02-10T04:36:45.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: . 2021-02-10T04:36:45.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 0 2021-02-10T04:36:45.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: ) 2021-02-10T04:36:45.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 1434 2021-02-10T04:36:45.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: : 2021-02-10T04:36:45.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: cluster 2021-02-10T04:36:45.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 2021-02-10T04:36:45.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: [DBG] 2021-02-10T04:36:45.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 2021-02-10T04:36:45.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: osdmap e587: 8 total, 8 up, 8 in 2021-02-10T04:36:45.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 2021-02-10T04:36:45.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: cluster 2021-02-10T04:36:45.379 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 2021-02-10T04:36:45.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 2021 2021-02-10T04:36:45.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: - 2021-02-10T04:36:45.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 02 2021-02-10T04:36:45.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: - 2021-02-10T04:36:45.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 10 2021-02-10T04:36:45.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: T 2021-02-10T04:36:45.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 04 2021-02-10T04:36:45.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: : 2021-02-10T04:36:45.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 36 2021-02-10T04:36:45.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: : 2021-02-10T04:36:45.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 44 2021-02-10T04:36:45.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: . 2021-02-10T04:36:45.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 727066 2021-02-10T04:36:45.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: +0000 2021-02-10T04:36:45.381 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 2021-02-10T04:36:45.381 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: osd.4 2021-02-10T04:36:45.381 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: ( 2021-02-10T04:36:45.381 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: osd 2021-02-10T04:36:45.381 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: . 2021-02-10T04:36:45.381 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 4 2021-02-10T04:36:45.381 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: ) 2021-02-10T04:36:45.381 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 855 2021-02-10T04:36:45.381 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: : 2021-02-10T04:36:45.381 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: cluster 2021-02-10T04:36:45.381 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 2021-02-10T04:36:45.381 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: [DBG] 2021-02-10T04:36:45.381 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 2021-02-10T04:36:45.381 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: purged_snaps scrub starts 2021-02-10T04:36:45.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 2021-02-10T04:36:45.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: cluster 2021-02-10T04:36:45.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 2021-02-10T04:36:45.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 2021 2021-02-10T04:36:45.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: - 2021-02-10T04:36:45.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 02 2021-02-10T04:36:45.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: - 2021-02-10T04:36:45.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 10 2021-02-10T04:36:45.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: T 2021-02-10T04:36:45.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 04 2021-02-10T04:36:45.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: : 2021-02-10T04:36:45.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 36 2021-02-10T04:36:45.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: : 2021-02-10T04:36:45.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 44 2021-02-10T04:36:45.383 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: . 2021-02-10T04:36:45.383 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 727357 2021-02-10T04:36:45.383 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: +0000 2021-02-10T04:36:45.383 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 2021-02-10T04:36:45.383 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: osd.4 2021-02-10T04:36:45.383 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: ( 2021-02-10T04:36:45.383 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: osd 2021-02-10T04:36:45.383 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: . 2021-02-10T04:36:45.383 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 4 2021-02-10T04:36:45.383 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: ) 2021-02-10T04:36:45.383 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 856 2021-02-10T04:36:45.383 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: : 2021-02-10T04:36:45.383 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: cluster 2021-02-10T04:36:45.383 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 2021-02-10T04:36:45.384 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: [DBG] 2021-02-10T04:36:45.384 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 2021-02-10T04:36:45.384 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:36:45.384 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[35961]: 2021-02-10T04:36:45.384 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[45443]: debug 2021-02-10T04:36:45.368+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:36:45.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: cluster 2021-02-10T04:36:45.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 2021-02-10T04:36:45.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 2021 2021-02-10T04:36:45.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: - 2021-02-10T04:36:45.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 02 2021-02-10T04:36:45.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: - 2021-02-10T04:36:45.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 10 2021-02-10T04:36:45.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: T 2021-02-10T04:36:45.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 04 2021-02-10T04:36:45.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: : 2021-02-10T04:36:45.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 36 2021-02-10T04:36:45.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: : 2021-02-10T04:36:45.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 44 2021-02-10T04:36:45.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: . 2021-02-10T04:36:45.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 356854 2021-02-10T04:36:45.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: +0000 2021-02-10T04:36:45.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 2021-02-10T04:36:45.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: mon.a 2021-02-10T04:36:45.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: ( 2021-02-10T04:36:45.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: mon 2021-02-10T04:36:45.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: . 2021-02-10T04:36:45.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 0 2021-02-10T04:36:45.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: ) 2021-02-10T04:36:45.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 1434 2021-02-10T04:36:45.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: : 2021-02-10T04:36:45.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: cluster 2021-02-10T04:36:45.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 2021-02-10T04:36:45.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: [DBG] 2021-02-10T04:36:45.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 2021-02-10T04:36:45.387 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: osdmap e587: 8 total, 8 up, 8 in 2021-02-10T04:36:45.387 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 2021-02-10T04:36:45.387 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: cluster 2021-02-10T04:36:45.387 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 2021-02-10T04:36:45.387 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 2021 2021-02-10T04:36:45.387 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: - 2021-02-10T04:36:45.387 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 02 2021-02-10T04:36:45.387 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: - 2021-02-10T04:36:45.387 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 10 2021-02-10T04:36:45.387 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: T 2021-02-10T04:36:45.387 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 04 2021-02-10T04:36:45.387 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: : 2021-02-10T04:36:45.387 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 36 2021-02-10T04:36:45.387 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: : 2021-02-10T04:36:45.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 44 2021-02-10T04:36:45.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: . 2021-02-10T04:36:45.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 727066 2021-02-10T04:36:45.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: +0000 2021-02-10T04:36:45.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 2021-02-10T04:36:45.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: osd.4 2021-02-10T04:36:45.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: ( 2021-02-10T04:36:45.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: osd 2021-02-10T04:36:45.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: . 2021-02-10T04:36:45.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 4 2021-02-10T04:36:45.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: ) 2021-02-10T04:36:45.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 855 2021-02-10T04:36:45.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: : 2021-02-10T04:36:45.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: cluster 2021-02-10T04:36:45.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 2021-02-10T04:36:45.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: [DBG] 2021-02-10T04:36:45.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 2021-02-10T04:36:45.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: purged_snaps scrub starts 2021-02-10T04:36:45.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 2021-02-10T04:36:45.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: cluster 2021-02-10T04:36:45.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 2021-02-10T04:36:45.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 2021 2021-02-10T04:36:45.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: - 2021-02-10T04:36:45.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 02 2021-02-10T04:36:45.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: - 2021-02-10T04:36:45.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 10 2021-02-10T04:36:45.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: T 2021-02-10T04:36:45.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 04 2021-02-10T04:36:45.390 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: : 2021-02-10T04:36:45.390 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 36 2021-02-10T04:36:45.390 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: : 2021-02-10T04:36:45.390 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 44 2021-02-10T04:36:45.390 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: . 2021-02-10T04:36:45.390 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 727357 2021-02-10T04:36:45.390 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: +0000 2021-02-10T04:36:45.390 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 2021-02-10T04:36:45.390 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: osd.4 2021-02-10T04:36:45.390 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: ( 2021-02-10T04:36:45.390 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: osd 2021-02-10T04:36:45.390 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: . 2021-02-10T04:36:45.390 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 4 2021-02-10T04:36:45.390 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: ) 2021-02-10T04:36:45.391 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 856 2021-02-10T04:36:45.391 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: : 2021-02-10T04:36:45.391 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: cluster 2021-02-10T04:36:45.391 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 2021-02-10T04:36:45.391 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: [DBG] 2021-02-10T04:36:45.391 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 2021-02-10T04:36:45.391 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: purged_snaps scrub ok 2021-02-10T04:36:45.391 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:45 smithi085 conmon[29319]: 2021-02-10T04:36:45.431 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2021-02-10T04:36:45.431 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:36:45.443 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:36:45.549 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:36:45.583 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:36:45.586 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:45 smithi164 conmon[29288]: cluster 2021-02-10T04:36:45.586 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:45 smithi164 conmon[29288]: 2021-02-10T04:36:45.586 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:45 smithi164 conmon[29288]: 2021 2021-02-10T04:36:45.586 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:45 smithi164 conmon[29288]: - 2021-02-10T04:36:45.586 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:45 smithi164 conmon[29288]: 02 2021-02-10T04:36:45.586 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:45 smithi164 conmon[29288]: - 2021-02-10T04:36:45.586 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:45 smithi164 conmon[29288]: 10 2021-02-10T04:36:45.586 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:45 smithi164 conmon[29288]: T 2021-02-10T04:36:45.586 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:45 smithi164 conmon[29288]: 04 2021-02-10T04:36:45.586 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:45 smithi164 conmon[29288]: : 2021-02-10T04:36:45.586 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:45 smithi164 conmon[29288]: 36 2021-02-10T04:36:45.586 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:45 smithi164 conmon[29288]: : 2021-02-10T04:36:45.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:45 smithi164 conmon[29288]: 44 2021-02-10T04:36:45.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:45 smithi164 conmon[29288]: . 2021-02-10T04:36:45.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:45 smithi164 conmon[29288]: 356854 2021-02-10T04:36:45.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:45 smithi164 conmon[29288]: +0000 2021-02-10T04:36:45.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:45 smithi164 conmon[29288]: 2021-02-10T04:36:45.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:45 smithi164 conmon[29288]: mon.a (mon.0) 1434 : cluster [DBG] 2021-02-10T04:36:45.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:45 smithi164 conmon[29288]: osdmap e587: 8 total, 8 up, 8 in 2021-02-10T04:36:45.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:45 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:36:45.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:45 smithi164 conmon[29288]: 10T04:36:44.727066+0000 osd.4 (osd.4) 855 : cluster 2021-02-10T04:36:45.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:45 smithi164 conmon[29288]: [DBG] purged_snaps scrub starts 2021-02-10T04:36:45.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:45 smithi164 conmon[29288]: cluster 2021-02-10T04:36:44. 2021-02-10T04:36:45.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:45 smithi164 conmon[29288]: 727357+0000 osd.4 (osd.4) 856 : cluster [DBG] 2021-02-10T04:36:45.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:45 smithi164 conmon[29288]: purged_snaps scrub ok 2021-02-10T04:36:45.674 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:36:45.741 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:36:45.781 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:36:45.802 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:36:45.802 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:36:45.816 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:36:45.960 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:36:45 smithi164 conmon[40892]: debug 2021-02-10T04:36:45.960 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:36:45 smithi164 conmon[40892]: 2021-02-10T04:36:45.652+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:36:46.225 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:36:46.360 INFO:tasks.rados.rados.0.smithi085.stdout:2259: done (9 left) 2021-02-10T04:36:46.360 INFO:tasks.rados.rados.0.smithi085.stdout:2260: write (excl) oid 37 current snap is 254 2021-02-10T04:36:46.360 INFO:tasks.rados.rados.0.smithi085.stdout:2260: seq_num 783 ranges {759881=787973,2119636=654941} 2021-02-10T04:36:46.370 INFO:tasks.rados.rados.0.smithi085.stdout:2260: writing smithi08575397-37 from 759881 to 1547854 tid 1 2021-02-10T04:36:46.380 INFO:tasks.rados.rados.0.smithi085.stdout:2260: writing smithi08575397-37 from 2119636 to 2774577 tid 2 2021-02-10T04:36:46.380 INFO:tasks.rados.rados.0.smithi085.stdout:2261: read oid 28 snap -1 2021-02-10T04:36:46.380 INFO:tasks.rados.rados.0.smithi085.stdout:2261: expect (ObjNum 651 snap 197 seq_num 651) 2021-02-10T04:36:46.380 INFO:tasks.rados.rados.0.smithi085.stdout:2262: write oid 35 current snap is 254 2021-02-10T04:36:46.380 INFO:tasks.rados.rados.0.smithi085.stdout:2260: finishing write tid 1 to smithi08575397-37 2021-02-10T04:36:46.380 INFO:tasks.rados.rados.0.smithi085.stdout:2262: seq_num 784 ranges {776688=579575,1894471=798599} 2021-02-10T04:36:46.387 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:36:46 smithi164 conmon[37263]: debug 2021-02-10T04:36:45.998+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:36:46.388 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:46 smithi164 conmon[29288]: cluster 2021-02-10T04:36:45.358382 2021-02-10T04:36:46.388 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:46 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1435 : cluster [DBG] osdmap e588: 8 total, 8 up, 8 in 2021-02-10T04:36:46.388 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:46 smithi164 conmon[29288]: cluster 2021-02-10T04:36:46.124355+0000 mgr.y 2021-02-10T04:36:46.388 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:46 smithi164 conmon[29288]: (mgr.14140) 644 : cluster [DBG] pgmap v1032: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 449 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 3.3 MiB/s wr, 26 op/s 2021-02-10T04:36:46.388 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:46 smithi164 conmon[29288]: cluster 2021-02-10T04:36:46.359530+0000 mon.a (mon.0) 1436 : cluster [DBG] osdmap e589: 8 total, 8 up, 8 in 2021-02-10T04:36:46.388 INFO:tasks.rados.rados.0.smithi085.stdout:2262: writing smithi08575397-35 from 776688 to 1356263 tid 1 2021-02-10T04:36:46.397 INFO:tasks.rados.rados.0.smithi085.stdout:2262: writing smithi08575397-35 from 1894471 to 2693070 tid 2 2021-02-10T04:36:46.397 INFO:tasks.rados.rados.0.smithi085.stdout:2261: expect (ObjNum 651 snap 197 seq_num 651) 2021-02-10T04:36:46.603 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:36:46.636 INFO:tasks.rados.rados.0.smithi085.stdout:2260: finishing write tid 2 to smithi08575397-37 2021-02-10T04:36:46.636 INFO:tasks.rados.rados.0.smithi085.stdout:2260: finishing write tid 3 to smithi08575397-37 2021-02-10T04:36:46.636 INFO:tasks.rados.rados.0.smithi085.stdout:2261: done (11 left) 2021-02-10T04:36:46.636 INFO:tasks.rados.rados.0.smithi085.stdout:2263: read oid 50 snap -1 2021-02-10T04:36:46.636 INFO:tasks.rados.rados.0.smithi085.stdout:2263: expect (ObjNum 769 snap 249 seq_num 769) 2021-02-10T04:36:46.636 INFO:tasks.rados.rados.0.smithi085.stdout:2264: read oid 11 snap 247 2021-02-10T04:36:46.636 INFO:tasks.rados.rados.0.smithi085.stdout:2264: expect (ObjNum 744 snap 239 seq_num 744) 2021-02-10T04:36:46.636 INFO:tasks.rados.rados.0.smithi085.stdout:2265: snap_remove snap 234 2021-02-10T04:36:46.637 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:36:46.663 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:36:46.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:46 smithi085 conmon[29319]: cluster 2021-02-10T04:36:45 2021-02-10T04:36:46.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:46 smithi085 conmon[29319]: .358382+0000 mon.a (mon.0) 1435 : cluster [DBG] osdmap e588: 8 total, 8 up, 8 in 2021-02-10T04:36:46.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:46 smithi085 conmon[29319]: cluster 2021-02-10T04:36:46.124355+0000 mgr.y (mgr.14140) 644 : cluster [DBG] pgmap v1032: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 449 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 3.3 MiB/s wr, 26 op/s 2021-02-10T04:36:46.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:46 smithi085 conmon[29319]: cluster 2021-02-10T04:36:46.359530+0000 mon.a (mon.0) 1436 : cluster [DBG] osdmap e589: 8 total, 8 up, 8 in 2021-02-10T04:36:46.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:46 smithi085 conmon[35961]: cluster 2021-02-10T04:36:45.358382+0000 mon.a 2021-02-10T04:36:46.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:46 smithi085 conmon[35961]: (mon.0) 1435 : cluster [DBG] osdmap e588: 8 total, 8 up, 8 in 2021-02-10T04:36:46.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:46 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:36:46.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:46 smithi085 conmon[35961]: 10T04:36:46.124355+0000 mgr.y (mgr. 2021-02-10T04:36:46.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:46 smithi085 conmon[35961]: 14140) 644 : cluster [DBG] pgmap v1032: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 449 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 3.3 MiB/s wr, 26 op/s 2021-02-10T04:36:46.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:46 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:46.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:46 smithi085 conmon[35961]: -10T04:36:46.359530+0000 mon.a (mon.0 2021-02-10T04:36:46.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:46 smithi085 conmon[35961]: ) 1436 : cluster [DBG] osdmap e589: 8 total, 8 up, 8 in 2021-02-10T04:36:46.693 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:36:46.725 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:36:46.763 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:36:46.794 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:36:46.822 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:36:46.823 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:36:46.823 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:36:46.824 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:36:46.824 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:36:46.824 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:36:46.824 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:36:46.824 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:36:46.824 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:36:47.059 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:47 smithi085 conmon[49106]: debug 2021-02-10T04:36:47.388 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:47 smithi085 conmon[49106]: 2021-02-10T04:36:47.057+0000 7f77a3e94700 -1 osd.2 589 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:47.389 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:47 smithi085 conmon[49106]: 2021-02-10T04:36:47.397 INFO:tasks.rados.rados.0.smithi085.stdout:2265: done (13 left) 2021-02-10T04:36:47.397 INFO:tasks.rados.rados.0.smithi085.stdout:2266: snap_remove snap 221 2021-02-10T04:36:47.404 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is unset 2021-02-10T04:36:47.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:47 smithi164 conmon[29288]: audit 2021-02-10T04:36:46.567982+0000 mon.c ( 2021-02-10T04:36:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:47 smithi164 conmon[29288]: mon.1) 59 : audit [DBG] from='client.? 172.21.15.85:0/2365889063' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:36:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:47 smithi164 conmon[29288]: audit 2021-02-10T04:36:46.868978+0000 mon.a (mon.0) 1437 : audit [INF] from='client.? 172.21.15.85:0/3225558255' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:36:47.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:47 smithi164 conmon[33677]: debug 2021-02-10T04:36:47.151+0000 7fbec19a1700 -1 osd.4 589 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:47.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:47 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:36:47.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:47 smithi085 conmon[29319]: 02-10T04:36:46.567982+0000 mon.c (mon.1) 59 : audit [DBG] from='client.? 172.21.15.85:0/2365889063' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:36:47.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:47 smithi085 conmon[29319]: audit 2021-02-10T04:36:46.868978+0000 mon.a (mon.0) 1437 : audit [INF] from='client.? 172.21.15.85:0/3225558255' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:36:47.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:47 smithi085 conmon[35961]: audit 2021-02-10T04:36:46.567982 2021-02-10T04:36:47.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:47 smithi085 conmon[35961]: +0000 mon.c (mon.1) 59 : audit [DBG] from='client.? 172.21.15.85:0/2365889063' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:36:47.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:47 smithi085 conmon[35961]: audit 2021-02-10T04:36 2021-02-10T04:36:47.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:47 smithi085 conmon[35961]: :46.868978+0000 mon.a (mon.0) 1437 : audit [INF] from='client.? 172.21.15.85:0/3225558255' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:36:48.189 INFO:tasks.rados.rados.0.smithi085.stdout:2266: done (13 left) 2021-02-10T04:36:48.888 INFO:tasks.rados.rados.0.smithi085.stdout:2267: read oid 16 snap 229 2021-02-10T04:36:48.888 INFO:tasks.rados.rados.0.smithi085.stdout:2267: expect (ObjNum 671 snap 207 seq_num 671) 2021-02-10T04:36:48.888 INFO:tasks.rados.rados.0.smithi085.stdout:2268: snap_create 2021-02-10T04:36:48.888 INFO:tasks.rados.rados.0.smithi085.stdout:2260: finishing write tid 4 to smithi08575397-37 2021-02-10T04:36:48.888 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 1232 (ObjNum 783 snap 254 seq_num 783) dirty exists 2021-02-10T04:36:48.889 INFO:tasks.rados.rados.0.smithi085.stdout:2260: left oid 37 (ObjNum 783 snap 254 seq_num 783) 2021-02-10T04:36:48.889 INFO:tasks.rados.rados.0.smithi085.stdout:2262: finishing write tid 1 to smithi08575397-35 2021-02-10T04:36:48.889 INFO:tasks.rados.rados.0.smithi085.stdout:2262: finishing write tid 2 to smithi08575397-35 2021-02-10T04:36:48.889 INFO:tasks.rados.rados.0.smithi085.stdout:2262: finishing write tid 3 to smithi08575397-35 2021-02-10T04:36:48.889 INFO:tasks.rados.rados.0.smithi085.stdout:2262: finishing write tid 4 to smithi08575397-35 2021-02-10T04:36:48.889 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 35 v 840 (ObjNum 784 snap 254 seq_num 784) dirty exists 2021-02-10T04:36:48.889 INFO:tasks.rados.rados.0.smithi085.stdout:2262: left oid 35 (ObjNum 784 snap 254 seq_num 784) 2021-02-10T04:36:48.889 INFO:tasks.rados.rados.0.smithi085.stdout:2263: expect (ObjNum 769 snap 249 seq_num 769) 2021-02-10T04:36:48.892 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:36:48.892 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[33677]: debug 2021-02-10T04:36:48.134+0000 7fbec19a1700 -1 osd.4 590 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:48.892 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: cluster 2021-02-10T04:36:48.892 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: 2021-02-10T04:36:47. 2021-02-10T04:36:48.892 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: 058005+0000 osd.2 (osd.2) 1641 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:48.892 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: cluster 2021-02-10T04:36:47.058010+0000 2021-02-10T04:36:48.892 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: osd.2 (osd.2) 1642 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:48.892 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:48.892 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: -02-10T04:36:47.058014+0000 2021-02-10T04:36:48.892 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: osd.2 (osd.2) 1643 : cluster [WRN] 2021-02-10T04:36:48.892 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:48.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:36:48.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: :36:47.058018+0000 osd.2 (osd.2 2021-02-10T04:36:48.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: ) 1644 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:48.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: cluster 2021-02-10T04:36:48.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: 2021-02-10T04:36:47.058021 2021-02-10T04:36:48.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1645 : 2021-02-10T04:36:48.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:48.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: cluster 2021-02-10T04:36:48.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: 2021-02-10T04:36:48.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: 2021 2021-02-10T04:36:48.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: - 2021-02-10T04:36:48.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: 02 2021-02-10T04:36:48.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: -10T04:36:47. 2021-02-10T04:36:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: 058024+0000 osd.2 (osd.2) 1646 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: cluster 2021-02-10T04:36: 2021-02-10T04:36:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: 47.058029+0000 osd.2 (osd.2) 1647 : cluster [WRN] 2021-02-10T04:36:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:36:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: 04:36:47.058034+0000 osd.2 (osd. 2021-02-10T04:36:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: 2) 1648 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: -10T04:36:47. 2021-02-10T04:36:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: 058039+0000 osd.2 (osd.2) 1649 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: cluster 2021-02-10T04:36: 2021-02-10T04:36:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: 47.058044+0000 osd.2 (osd.2) 1650 : cluster 2021-02-10T04:36:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: cluster 2021-02-10T04:36:47.153680 2021-02-10T04:36:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 857 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: -02-10T04:36:47.153688+0000 2021-02-10T04:36:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: osd.4 (osd.4) 858 : cluster 2021-02-10T04:36:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: cluster 2021-02-10T04:36:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: 2021-02-10T04:36:47.153691+0000 osd.4 (osd.4 2021-02-10T04:36:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: ) 859 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: cluster 2021-02-10T04:36:47.153697+0000 osd.4 (osd.4 2021-02-10T04:36:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: ) 860 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: cluster 2021-02-10T04:36:47.387307+0000 2021-02-10T04:36:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: mon.a (mon.0) 1438 : cluster 2021-02-10T04:36:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:36:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: -02-10T04:36:47.387331+0000 2021-02-10T04:36:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: mon.a (mon.0) 1439 : cluster 2021-02-10T04:36:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: [INF] Cluster is now healthy 2021-02-10T04:36:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: audit 2021 2021-02-10T04:36:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: -02-10T04:36:47 2021-02-10T04:36:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: .396419+0000 mon.a (mon.0) 1440 2021-02-10T04:36:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: : audit [INF] from='client.? 172.21.15.85:0/3225558255' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:36:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:36:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: :36:47.396480+0000 mon.a (mon.0) 1441 : cluster 2021-02-10T04:36:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: [DBG] osdmap e590: 8 total, 8 up, 8 in 2021-02-10T04:36:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: cluster 2021-02-10T04:36:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: 2021-02-10T04:36:48.124970+0000 mgr.y (mgr.14140) 2021-02-10T04:36:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: 645 : cluster [DBG] pgmap v1035: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 449 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.4 MiB/s wr, 14 op/s 2021-02-10T04:36:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:36:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: :36:48.167072+0000 mgr.y (mgr.14140) 646 2021-02-10T04:36:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:36:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: cluster 2021-02-10T04:36 2021-02-10T04:36:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: :48.187990+0000 mon.a (mon.0) 1442 : cluster 2021-02-10T04:36:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:48 smithi164 conmon[29288]: [DBG] osdmap e591: 8 total, 8 up, 8 in 2021-02-10T04:36:48.897 INFO:tasks.rados.rados.0.smithi085.stdout:2264: expect (ObjNum 744 snap 239 seq_num 744) 2021-02-10T04:36:48.897 INFO:tasks.rados.rados.0.smithi085.stdout:2267: expect (ObjNum 671 snap 207 seq_num 671) 2021-02-10T04:36:48.897 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[49106]: debug 2021-02-10T04:36:48.101+0000 7f77a3e94700 -1 osd.2 590 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:48.898 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: cluster 2021-02-10T04:36:47.058005+0000 osd.2 (osd. 2021-02-10T04:36:48.898 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: 2) 1641 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:48.898 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:36:48.898 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: T04:36:47.058010+0000 osd.2 (osd.2) 1642 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:48.898 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:48.898 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: 02-10T04:36:47.058014+0000 osd.2 ( 2021-02-10T04:36:48.898 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: osd.2) 1643 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:48.898 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:36:48.898 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: 04:36:47.058018+0000 osd.2 (osd.2 2021-02-10T04:36:48.898 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: ) 1644 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:48.898 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: cluster 2021-02-10T04:36:48.899 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: 2021-02-10T04:36:48.899 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: 2021 2021-02-10T04:36:48.899 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: - 2021-02-10T04:36:48.899 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: 02 2021-02-10T04:36:48.899 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: -10T04:36:47. 2021-02-10T04:36:48.899 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: 058021+0000 osd.2 (osd.2) 1645 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:48.899 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: cluster 2021-02-10T04:36:48.899 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: 2021-02-10T04:36:47.058024+0000 2021-02-10T04:36:48.899 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: osd.2 (osd.2) 1646 : cluster [WRN] 2021-02-10T04:36:48.899 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:48.900 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: cluster 2021-02-10T04:36:47 2021-02-10T04:36:48.900 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: .058029+0000 osd.2 (osd.2) 1647 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:48.900 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: 2021-02-10T04:36:48.900 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: cluster 2021-02-10T04:36:48.900 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: 2021-02-10T04:36 2021-02-10T04:36:48.900 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: :47.058034+0000 osd.2 (osd.2) 1648 : cluster 2021-02-10T04:36:48.900 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:48.900 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:48.900 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: 02-10T04:36:47.058039+0000 osd.2 (osd 2021-02-10T04:36:48.900 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: .2) 1649 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:48.900 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: cluster 2021-02-10T04:36:48.900 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: 2021-02-10T04:36: 2021-02-10T04:36:48.900 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: 47.058044+0000 osd.2 (osd.2) 1650 : cluster 2021-02-10T04:36:48.900 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:48.901 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: cluster 2021-02-10T04:36:47.153680+0000 osd.4 2021-02-10T04:36:48.901 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: (osd.4) 857 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:48.901 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:36:48.901 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: :36:47.153688+0000 osd.4 (osd.4) 858 2021-02-10T04:36:48.901 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:48.901 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:36:48.901 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: 04:36:47.153691+0000 osd.4 (osd.4) 859 : cluster 2021-02-10T04:36:48.901 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:48.901 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: cluster 2021-02-10T04:36:47. 2021-02-10T04:36:48.901 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: 153697+0000 osd.4 (osd.4) 860 : cluster [WRN] 2021-02-10T04:36:48.901 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:48.901 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: cluster 2021-02-10T04:36:47 2021-02-10T04:36:48.901 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: .387307+0000 mon.a (mon.0) 1438 : cluster 2021-02-10T04:36:48.902 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:36:48.902 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: cluster 2021-02-10T04:36:47 2021-02-10T04:36:48.902 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: .387331+0000 mon.a (mon.0) 1439 : cluster 2021-02-10T04:36:48.902 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: [INF] Cluster is now healthy 2021-02-10T04:36:48.902 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: audit 2021-02-10T04:36:48.902 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: 2021-02-10T04:36:47.396419+0000 mon.a (mon. 2021-02-10T04:36:48.902 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: 0) 1440 : audit [INF] from='client.? 172.21.15.85:0/3225558255' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:36:48.902 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:36:48.902 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: :36:47.396480+0000 mon.a (mon.0) 2021-02-10T04:36:48.902 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: 1441 : cluster [DBG] osdmap e590: 8 total, 8 up, 8 in 2021-02-10T04:36:48.902 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:36:48.902 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: 10T04:36:48.124970+0000 mgr.y (mgr. 2021-02-10T04:36:48.902 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: 14140) 645 : cluster [DBG] pgmap v1035: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 449 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.4 MiB/s wr, 14 op/s 2021-02-10T04:36:48.902 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:36:48.903 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: T04:36:48.167072+0000 mgr.y (mgr. 2021-02-10T04:36:48.903 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: 14140) 646 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:36:48.903 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:36:48.903 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: 04:36:48.187990+0000 mon.a (mon.0) 2021-02-10T04:36:48.903 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[29319]: 1442 : cluster [DBG] osdmap e591: 8 total, 8 up, 8 in 2021-02-10T04:36:48.903 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: cluster 2021-02-10T04:36:47.058005 2021-02-10T04:36:48.903 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1641 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:48.903 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:48.903 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: -10T04:36:47.058010+0000 osd.2 (osd.2) 1642 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:48.903 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: 2021-02-10T04:36:48.903 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:48.903 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: T04:36:47.058014+0000 osd.2 (osd.2) 1643 : cluster 2021-02-10T04:36:48.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:48.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: cluster 2021-02-10T04:36:48.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: 2021-02-10T04:36:47.058018+0000 osd.2 (osd.2) 2021-02-10T04:36:48.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: 1644 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:48.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: cluster 2021-02-10T04:36:47. 2021-02-10T04:36:48.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: 058021+0000 osd.2 (osd.2) 1645 : cluster [WRN] 2021-02-10T04:36:48.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:48.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:48.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: 02-10T04:36:47.058024+0000 osd.2 (osd.2) 1646 2021-02-10T04:36:48.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:48.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: cluster 2021-02-10T04:36: 2021-02-10T04:36:48.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: 47.058029+0000 osd.2 (osd.2) 1647 : cluster 2021-02-10T04:36:48.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:48.905 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: cluster 2021-02-10T04:36:48.905 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: 2021-02-10T04:36:47.058034+0000 osd.2 (osd.2 2021-02-10T04:36:48.905 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: ) 1648 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:48.905 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: cluster 2021-02-10T04:36:47.058039+0000 osd.2 (osd.2) 1649 : cluster 2021-02-10T04:36:48.905 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:48.905 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: cluster 2021-02-10T04:36:48.905 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: 2021-02-10T04:36:47 2021-02-10T04:36:48.905 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: .058044+0000 osd.2 (osd. 2021-02-10T04:36:48.905 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: 2) 1650 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:48.905 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: 2021-02-10T04:36:48.906 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:36:48.906 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: 04:36:47.153680 2021-02-10T04:36:48.906 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 857 : 2021-02-10T04:36:48.906 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:48.906 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: cluster 2021-02-10T04:36:48.906 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: 2021-02-10T04:36:47.153688+0000 osd.4 (osd 2021-02-10T04:36:48.906 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: .4) 858 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:48.906 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:36:48.907 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: :36:47.153691+0000 osd.4 (osd.4) 2021-02-10T04:36:48.907 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: 859 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:48.907 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:48.907 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: T04:36:47.153697+0000 osd.4 (osd.4) 860 : cluster [WRN] 2021-02-10T04:36:48.907 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:48.907 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:48.907 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: -02-10T04:36:47.387307+0000 mon.a (mon.0) 1438 : cluster 2021-02-10T04:36:48.907 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:36:48.907 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:36:48.907 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: :36:47.387331+0000 mon.a (mon.0) 2021-02-10T04:36:48.907 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: 1439 : cluster [INF] Cluster is now healthy 2021-02-10T04:36:48.907 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:36:48.908 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: :36:47.396419+0000 mon.a (mon.0 2021-02-10T04:36:48.908 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: ) 1440 : audit [INF] from='client.? 172.21.15.85:0/3225558255' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:36:48.908 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:36:48.908 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: :36:47.396480+0000 mon.a (mon. 2021-02-10T04:36:48.908 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: 0) 1441 : cluster [DBG] osdmap e590: 8 total, 8 up, 8 in 2021-02-10T04:36:48.908 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:36:48.908 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: :36:48.124970+0000 mgr.y (mgr.14140) 2021-02-10T04:36:48.908 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: 645 : cluster [DBG] pgmap v1035: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 449 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 1.4 MiB/s wr, 14 op/s 2021-02-10T04:36:48.909 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:36:48.909 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: 36:48.167072+0000 mgr.y (mgr.14140) 646 2021-02-10T04:36:48.909 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:36:48.909 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: cluster 2021-02-10T04:36: 2021-02-10T04:36:48.909 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: 48.187990+0000 mon.a (mon.0) 1442 : 2021-02-10T04:36:48.909 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:48 smithi085 conmon[35961]: cluster [DBG] osdmap e591: 8 total, 8 up, 8 in 2021-02-10T04:36:48.909 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.7 dump_ops_in_flight 2021-02-10T04:36:49.010 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:36:49.044 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:36:49.137 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[33677]: debug 2021-02-10T04:36:49.088+0000 7fbec19a1700 -1 osd.4 591 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:49.137 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[44544]: debug 2021-02-10T04:36:49.135+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:36:49.186 INFO:tasks.rados.rados.0.smithi085.stdout:2260: done (14 left) 2021-02-10T04:36:49.187 INFO:tasks.rados.rados.0.smithi085.stdout:2262: done (13 left) 2021-02-10T04:36:49.187 INFO:tasks.rados.rados.0.smithi085.stdout:2263: done (12 left) 2021-02-10T04:36:49.187 INFO:tasks.rados.rados.0.smithi085.stdout:2264: done (11 left) 2021-02-10T04:36:49.187 INFO:tasks.rados.rados.0.smithi085.stdout:2267: done (10 left) 2021-02-10T04:36:49.187 INFO:tasks.rados.rados.0.smithi085.stdout:2268: done (9 left) 2021-02-10T04:36:49.187 INFO:tasks.rados.rados.0.smithi085.stdout:2269: read oid 36 snap 247 2021-02-10T04:36:49.187 INFO:tasks.rados.rados.0.smithi085.stdout:2269: expect deleted 2021-02-10T04:36:49.187 INFO:tasks.rados.rados.0.smithi085.stdout:2270: read oid 40 snap -1 2021-02-10T04:36:49.187 INFO:tasks.rados.rados.0.smithi085.stdout:2270: expect (ObjNum 756 snap 241 seq_num 756) 2021-02-10T04:36:49.187 INFO:tasks.rados.rados.0.smithi085.stdout:2271: delete oid 43 current snap is 255 2021-02-10T04:36:49.195 INFO:tasks.rados.rados.0.smithi085.stdout:2269: done (11 left) 2021-02-10T04:36:49.195 INFO:tasks.rados.rados.0.smithi085.stdout:2271: done (10 left) 2021-02-10T04:36:49.196 INFO:tasks.rados.rados.0.smithi085.stdout:2272: delete oid 37 current snap is 255 2021-02-10T04:36:49.199 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:36:49.199 INFO:tasks.rados.rados.0.smithi085.stdout:2270: expect (ObjNum 756 snap 241 seq_num 756) 2021-02-10T04:36:49.251 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:36:49.305 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:36:49.349 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:36:49.397 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[49106]: debug 2021-02-10T04:36:49.114+0000 7f77a3e94700 -1 osd.2 591 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:49.410 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: cluster 2021-02-10T04:36:49.411 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: 2021-02-10T04:36:49.411 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: 2021 2021-02-10T04:36:49.411 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: - 2021-02-10T04:36:49.411 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: 02-10T04:36:48. 2021-02-10T04:36:49.411 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: 102157+0000 osd.2 (osd.2) 1651 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:49.412 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: cluster 2021-02-10T04:36:49.412 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: 2021-02-10T04:36 2021-02-10T04:36:49.412 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: :48.102166+0000 osd.2 (osd.2 2021-02-10T04:36:49.412 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: ) 1652 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:49.412 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:49.412 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: 02-10T04:36:48.102172+0000 2021-02-10T04:36:49.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: osd.2 (osd.2) 1653 : 2021-02-10T04:36:49.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:49.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: cluster 2021-02-10T04:36:49.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:36:49.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: T04:36:48.102181+0000 2021-02-10T04:36:49.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: osd.2 (osd.2) 1654 : cluster 2021-02-10T04:36:49.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:49.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: cluster 2021-02-10T04:36:49.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:36:49.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: 04:36:48.102187+0000 2021-02-10T04:36:49.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: osd.2 (osd.2) 1655 : cluster 2021-02-10T04:36:49.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:49.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:49.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: -10T04:36:48.102193+0000 osd.2 (osd. 2021-02-10T04:36:49.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: 2) 1656 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:49.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:49.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: -02-10T04:36:48.102199+0000 2021-02-10T04:36:49.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: osd.2 (osd.2) 1657 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:49.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: 2021-02-10T04:36:49.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:49.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: 02-10T04:36:48.102205+0000 osd.2 (osd.2) 1658 2021-02-10T04:36:49.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:49.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:36:49.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: 10T04:36:48.102211+0000 osd.2 (osd 2021-02-10T04:36:49.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: .2) 1659 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:49.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: cluster 2021-02-10T04:36:49.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: 2021-02-10T04:36:49.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: 2021 2021-02-10T04:36:49.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: - 2021-02-10T04:36:49.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: 02-10T04:36:48. 2021-02-10T04:36:49.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: 102217+0000 osd.2 (osd.2) 1660 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:49.418 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: cluster 2021-02-10T04:36:48 2021-02-10T04:36:49.418 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: .136359+0000 osd.4 (osd.4) 861 : cluster 2021-02-10T04:36:49.418 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:49.418 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:36:49.418 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: :36:48.136369+0000 osd.4 (osd. 2021-02-10T04:36:49.418 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: 4) 862 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:49.419 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:49.419 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: -02-10T04:36:48.136375 2021-02-10T04:36:49.419 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 863 : cluster 2021-02-10T04:36:49.419 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:49.419 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:36:49.420 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: 10T04:36:48.136381+0000 osd.4 2021-02-10T04:36:49.420 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: (osd.4) 864 : cluster [WRN] 2021-02-10T04:36:49.420 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:49.420 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: cluster 2021-02-10T04:36:49 2021-02-10T04:36:49.420 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: .090056+0000 osd.4 (osd.4) 865 : 2021-02-10T04:36:49.420 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:49.420 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:36:49.421 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: 04:36:49.090067+0000 osd.4 ( 2021-02-10T04:36:49.421 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: osd.4) 866 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:49.421 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: 2021-02-10T04:36:49.421 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: cluster 2021-02-10T04:36 2021-02-10T04:36:49.421 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: :49.090073+0000 osd.4 (osd.4) 2021-02-10T04:36:49.422 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: 867 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:49.422 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:36:49.422 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: 10T04:36:49.090079+0000 2021-02-10T04:36:49.422 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: osd.4 (osd.4) 868 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:49.422 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: 2021-02-10T04:36:49.422 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: cluster 2021-02-10T04:36:49.423 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: 2021-02-10T04:36:49.187408+0000 mon.a (mon 2021-02-10T04:36:49.423 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[35961]: .0) 1443 : cluster [DBG] osdmap e592: 8 total, 8 up, 8 in 2021-02-10T04:36:49.423 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: cluster 2021-02-10T04:36:48.102157+0000 osd.2 (osd.2) 1651 : cluster 2021-02-10T04:36:49.423 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:49.423 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: cluster 2021-02-10T04:36:48.102166+0000 osd.2 (osd.2) 1652 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:49.424 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: cluster 2021-02-10T04:36:48.102172+0000 osd.2 (osd.2) 1653 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:49.424 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: cluster 2021-02-10T04:36:48.102181+0000 osd.2 (osd.2) 1654 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:49.424 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: cluster 2021-02-10T04:36:48.102187+0000 osd.2 (osd.2) 1655 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:49.424 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: cluster 2021-02-10T04:36:48.102193+0000 osd.2 (osd.2) 1656 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:49.424 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: cluster 2021-02-10T04:36:48.102199+0000 osd.2 (osd.2) 1657 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:49.425 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: cluster 2021-02-10T04:36:48.102205+0000 osd.2 (osd.2) 1658 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:49.425 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: cluster 2021-02-10T04:36:48.102211+0000 osd.2 (osd.2) 1659 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:49.425 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: cluster 2021-02-10T04:36:48.102217+0000 osd.2 (osd.2) 1660 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:49.425 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:49.425 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: 02-10T04:36:48.136359+0000 osd.4 (osd.4) 861 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:49.425 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: cluster 2021-02-10T04:36:48.136369+0000 osd.4 (osd.4) 862 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:49.426 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: cluster 2021-02-10T04:36:48.136375 2021-02-10T04:36:49.426 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 863 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:49.426 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: cluster 2021-02-10T04:36:48.136381 2021-02-10T04:36:49.426 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 864 : cluster [WRN] 2021-02-10T04:36:49.426 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:49.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:36:49.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: 36:49.090056+0000 osd.4 (osd.4) 2021-02-10T04:36:49.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: 865 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:49.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: cluster 2021-02-10T04:36 2021-02-10T04:36:49.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: :49.090067+0000 osd.4 ( 2021-02-10T04:36:49.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: osd.4) 866 : cluster 2021-02-10T04:36:49.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:49.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:49.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: -02-10T04:36:49.090073+0000 osd.4 (osd. 2021-02-10T04:36:49.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: 4) 867 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:49.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:49.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: -10T04:36:49.090079+0000 osd.4 2021-02-10T04:36:49.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: (osd.4) 868 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:49.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: 2021-02-10T04:36:49.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:36:49.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: 10T04:36:49.187408+0000 mon.a (mon.0) 2021-02-10T04:36:49.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:49 smithi085 conmon[29319]: 1443 : cluster [DBG] osdmap e592: 8 total, 8 up, 8 in 2021-02-10T04:36:49.431 INFO:tasks.rados.rados.0.smithi085.stdout:2270: done (10 left) 2021-02-10T04:36:49.431 INFO:tasks.rados.rados.0.smithi085.stdout:2272: done (9 left) 2021-02-10T04:36:49.431 INFO:tasks.rados.rados.0.smithi085.stdout:2273: read oid 42 snap -1 2021-02-10T04:36:49.431 INFO:tasks.rados.rados.0.smithi085.stdout:2273: expect (ObjNum 772 snap 250 seq_num 772) 2021-02-10T04:36:49.432 INFO:tasks.rados.rados.0.smithi085.stdout:2274: read oid 36 snap -1 2021-02-10T04:36:49.432 INFO:tasks.rados.rados.0.smithi085.stdout:2274: expect deleted 2021-02-10T04:36:49.432 INFO:tasks.rados.rados.0.smithi085.stdout:2275: delete oid 20 current snap is 255 2021-02-10T04:36:49.432 INFO:tasks.rados.rados.0.smithi085.stdout:2274: done (11 left) 2021-02-10T04:36:49.432 INFO:tasks.rados.rados.0.smithi085.stdout:2275: done (10 left) 2021-02-10T04:36:49.432 INFO:tasks.rados.rados.0.smithi085.stdout:2276: rollback oid 43 current snap is 255 2021-02-10T04:36:49.433 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 43 to 202 2021-02-10T04:36:49.433 INFO:tasks.rados.rados.0.smithi085.stdout:2277: write (excl) oid 46 current snap is 255 2021-02-10T04:36:49.433 INFO:tasks.rados.rados.0.smithi085.stdout:2277: seq_num 788 ranges {443916=480571,1612046=405937,2606730=495052} 2021-02-10T04:36:49.433 INFO:tasks.rados.rados.0.smithi085.stdout:2277: writing smithi08575397-46 from 443916 to 924487 tid 1 2021-02-10T04:36:49.433 INFO:tasks.rados.rados.0.smithi085.stdout:2277: writing smithi08575397-46 from 1612046 to 2017983 tid 2 2021-02-10T04:36:49.437 INFO:tasks.rados.rados.0.smithi085.stdout:2277: writing smithi08575397-46 from 2606730 to 3101782 tid 3 2021-02-10T04:36:49.437 INFO:tasks.rados.rados.0.smithi085.stdout:2278: snap_remove snap 217 2021-02-10T04:36:49.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: cluster 2021-02-10T04:36:48. 2021-02-10T04:36:49.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: 102157+0000 osd.2 (osd.2) 1651 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:49.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: cluster 2021-02-10T04:36:48.102166+0000 osd.2 (osd.2) 1652 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:49.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: cluster 2021-02-10T04:36:48.102172+0000 osd.2 (osd.2) 1653 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:49.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: cluster 2021-02-10T04:36 2021-02-10T04:36:49.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: :48.102181+0000 osd.2 (osd.2) 1654 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:49.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: cluster 2021-02-10T04:36:48.102187 2021-02-10T04:36:49.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1655 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:49.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: cluster 2021-02-10T04:36:49.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: 2021-02-10T04:36:48 2021-02-10T04:36:49.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: .102193+0000 osd.2 (osd.2 2021-02-10T04:36:49.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: ) 1656 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:49.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: 2021-02-10T04:36:49.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:49.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: -02-10T04:36:48.102199 2021-02-10T04:36:49.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1657 : 2021-02-10T04:36:49.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:49.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:49.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: -10T04:36:48.102205+0000 osd.2 ( 2021-02-10T04:36:49.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: osd.2) 1658 : cluster 2021-02-10T04:36:49.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:49.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:49.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: 02-10T04:36:48.102211+0000 osd.2 (osd.2 2021-02-10T04:36:49.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: ) 1659 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:49.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:49.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: -10T04:36:48.102217+0000 osd.2 (osd.2) 2021-02-10T04:36:49.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: 1660 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:49.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: cluster 2021-02-10T04:36:49.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: 2021-02-10T04:36:48 2021-02-10T04:36:49.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: .136359+0000 osd.4 (osd. 2021-02-10T04:36:49.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: 4) 861 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:49.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: 2021-02-10T04:36:49.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: cluster 2021-02-10T04:36:49.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: 2021-02-10T04:36: 2021-02-10T04:36:49.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: 48.136369+0000 osd.4 (osd.4) 862 : cluster 2021-02-10T04:36:49.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:49.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: cluster 2021-02-10T04:36: 2021-02-10T04:36:49.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: 48.136375+0000 osd.4 (osd.4) 863 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:49.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: 2021-02-10T04:36:49.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:49.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: -02-10T04:36:48.136381+0000 2021-02-10T04:36:49.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: osd.4 (osd.4) 864 : cluster [WRN] 2021-02-10T04:36:49.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:49.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:36:49.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: 10T04:36:49.090056 2021-02-10T04:36:49.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 865 : 2021-02-10T04:36:49.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:49.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:49.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: 02-10T04:36:49 2021-02-10T04:36:49.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: .090067+0000 osd.4 (osd.4) 866 : 2021-02-10T04:36:49.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:49.723 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:49.723 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: -10T04:36:49.090073 2021-02-10T04:36:49.723 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 867 : cluster 2021-02-10T04:36:49.723 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:49.723 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:36:49.723 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: T04:36:49.090079+0000 2021-02-10T04:36:49.723 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: osd.4 (osd.4) 868 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:49.723 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: 2021-02-10T04:36:49.723 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:49.723 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: 02-10T04:36:49.187408+0000 mon.a (mon.0 2021-02-10T04:36:49.723 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:49 smithi164 conmon[29288]: ) 1443 : cluster [DBG] osdmap e592: 8 total, 8 up, 8 in 2021-02-10T04:36:50.099 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[33677]: debug 2021-02-10T04:36:50.096+0000 7fbec19a1700 -1 osd.4 592 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:50.402 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[49106]: debug 2021-02-10T04:36:50.113+0000 7f77a3e94700 -1 osd.2 592 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:50.406 INFO:tasks.rados.rados.0.smithi085.stdout:2278: done (12 left) 2021-02-10T04:36:50.406 INFO:tasks.rados.rados.0.smithi085.stdout:2279: read oid 31 snap -1 2021-02-10T04:36:50.406 INFO:tasks.rados.rados.0.smithi085.stdout:2279: expect (ObjNum 768 snap 249 seq_num 768) 2021-02-10T04:36:50.406 INFO:tasks.rados.rados.0.smithi085.stdout:2280: write oid 50 current snap is 255 2021-02-10T04:36:50.406 INFO:tasks.rados.rados.0.smithi085.stdout:2280: seq_num 789 ranges {460723=555820,1670528=549595,2634153=626570} 2021-02-10T04:36:50.408 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: cluster 2021-02-10T04:36:49 2021-02-10T04:36:50.408 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: .116484+0000 osd.2 (osd.2) 1661 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:50.408 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: cluster 2021-02-10T04:36:49.116492+0000 osd.2 (osd.2) 1662 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:50.408 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:36:50.408 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: 04:36:49.116498+0000 osd.2 (osd.2) 1663 2021-02-10T04:36:50.408 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:50.409 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: cluster 2021-02-10T04:36:50.409 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: 2021-02-10T04:36:49.116502+0000 2021-02-10T04:36:50.409 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: osd.2 (osd.2) 1664 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:50.409 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:36:50.409 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: T04:36:49.116505+0000 osd.2 (osd.2) 1665 : cluster 2021-02-10T04:36:50.409 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:50.409 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:36:50.409 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: 36:49.116508+0000 osd.2 (osd.2) 1666 : 2021-02-10T04:36:50.409 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:50.409 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:36:50.409 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: 10T04:36:49.116512+0000 osd.2 (osd 2021-02-10T04:36:50.409 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: .2) 1667 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:50.410 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: cluster 2021-02-10T04:36:50.410 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: 2021-02-10T04:36 2021-02-10T04:36:50.410 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: :49.116516+0000 osd.2 (osd.2 2021-02-10T04:36:50.410 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: ) 1668 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:50.410 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:50.410 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: -02-10T04:36:49.116520+0000 2021-02-10T04:36:50.410 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: osd.2 (osd.2) 1669 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:50.410 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: 2021-02-10T04:36:50.410 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: cluster 2021-02-10T04:36:50.410 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: 2021-02-10T04:36:49.116523 2021-02-10T04:36:50.410 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1670 : cluster [WRN] 2021-02-10T04:36:50.410 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:50.411 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:36:50.411 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: :36:50.098434+0000 osd.4 ( 2021-02-10T04:36:50.411 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: osd.4) 869 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:50.411 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:50.411 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: -02-10T04:36:50.098443+0000 osd.4 ( 2021-02-10T04:36:50.411 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: osd.4) 870 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:50.411 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:50.411 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: -10T04:36:50.098449+0000 osd.4 (osd.4) 871 : cluster 2021-02-10T04:36:50.411 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:50.411 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:50.411 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: 02-10T04:36:50.098455+0000 osd.4 (osd.4) 872 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:50.411 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: cluster 2021-02-10T04:36:50.125526 2021-02-10T04:36:50.411 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 647 : cluster [DBG] pgmap v1038: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 446 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 689 KiB/s wr, 11 op/s 2021-02-10T04:36:50.412 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: cluster 2021-02-10T04:36:50.412 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: 2021-02-10T04:36:50.412 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: 2021-02-10T04:36:49. 2021-02-10T04:36:50.412 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: 116484+0000 osd.2 (osd.2) 1661 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:50.412 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: cluster 2021-02-10T04:36:49 2021-02-10T04:36:50.412 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: .116492+0000 osd.2 (osd.2) 1662 : cluster 2021-02-10T04:36:50.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:50.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: cluster 2021-02-10T04:36: 2021-02-10T04:36:50.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: 49.116498+0000 osd.2 (osd.2) 1663 : cluster [WRN] 2021-02-10T04:36:50.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:50.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:50.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: T04:36:49.116502+0000 osd.2 (osd.2) 1664 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:50.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: cluster 2021-02-10T04:36:49. 2021-02-10T04:36:50.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: 116505+0000 osd.2 (osd.2) 1665 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:50.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: cluster 2021-02-10T04:36 2021-02-10T04:36:50.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: :49.116508+0000 osd.2 (osd.2) 1666 : cluster [WRN] 2021-02-10T04:36:50.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:50.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:36:50.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: :36:49.116512+0000 osd.2 (osd.2) 1667 : cluster [WRN] 2021-02-10T04:36:50.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:50.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: cluster 2021-02-10T04:36: 2021-02-10T04:36:50.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: 49.116516+0000 osd.2 (osd.2) 1668 2021-02-10T04:36:50.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:50.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:36:50.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: 04:36:49.116520+0000 osd.2 (osd 2021-02-10T04:36:50.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: .2) 1669 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:50.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:50.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: -02-10T04:36:49. 2021-02-10T04:36:50.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: 116523+0000 osd.2 (osd.2) 1670 : cluster 2021-02-10T04:36:50.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:50.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: cluster 2021-02-10T04:36:50.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: 2021-02-10T04:36:50.098434+0000 osd.4 (osd.4) 869 : cluster 2021-02-10T04:36:50.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:50.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:50.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: -10T04:36:50.098443+0000 osd.4 (osd.4) 870 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:50.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: cluster 2021-02-10T04:36:50.098449+0000 osd.4 (osd.4 2021-02-10T04:36:50.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: ) 871 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:50.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: cluster 2021-02-10T04:36:50.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: 2021-02-10T04:36:50.098455+0000 osd.4 (osd.4) 872 2021-02-10T04:36:50.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:50.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:36:50.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: :36:50.125526+0000 mgr.y (mgr.14140 2021-02-10T04:36:50.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:50 smithi085 conmon[35961]: ) 647 : cluster [DBG] pgmap v1038: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 446 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 689 KiB/s wr, 11 op/s 2021-02-10T04:36:50.415 INFO:tasks.rados.rados.0.smithi085.stdout:2280: writing smithi08575397-50 from 460723 to 1016543 tid 1 2021-02-10T04:36:50.421 INFO:tasks.rados.rados.0.smithi085.stdout:2280: writing smithi08575397-50 from 1670528 to 2220123 tid 2 2021-02-10T04:36:50.428 INFO:tasks.rados.rados.0.smithi085.stdout:2280: writing smithi08575397-50 from 2634153 to 3260723 tid 3 2021-02-10T04:36:50.428 INFO:tasks.rados.rados.0.smithi085.stdout:2281: read oid 40 snap -1 2021-02-10T04:36:50.428 INFO:tasks.rados.rados.0.smithi085.stdout:2281: expect (ObjNum 756 snap 241 seq_num 756) 2021-02-10T04:36:50.429 INFO:tasks.rados.rados.0.smithi085.stdout:2282: snap_create 2021-02-10T04:36:50.429 INFO:tasks.rados.rados.0.smithi085.stdout:2276: finishing rollback tid 1 to smithi08575397-43 2021-02-10T04:36:50.429 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 1124 (ObjNum 82644416 snap 22030 seq_num 82647968) dirty dne 2021-02-10T04:36:50.429 INFO:tasks.rados.rados.0.smithi085.stdout:2273: expect (ObjNum 772 snap 250 seq_num 772) 2021-02-10T04:36:50.601 INFO:tasks.rados.rados.0.smithi085.stdout:2277: finishing write tid 1 to smithi08575397-46 2021-02-10T04:36:50.602 INFO:tasks.rados.rados.0.smithi085.stdout:2277: finishing write tid 2 to smithi08575397-46 2021-02-10T04:36:50.602 INFO:tasks.rados.rados.0.smithi085.stdout:2277: finishing write tid 3 to smithi08575397-46 2021-02-10T04:36:50.602 INFO:tasks.rados.rados.0.smithi085.stdout:2277: finishing write tid 4 to smithi08575397-46 2021-02-10T04:36:50.602 INFO:tasks.rados.rados.0.smithi085.stdout:2277: finishing write tid 5 to smithi08575397-46 2021-02-10T04:36:50.602 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 1065 (ObjNum 788 snap 255 seq_num 788) dirty exists 2021-02-10T04:36:50.603 INFO:tasks.rados.rados.0.smithi085.stdout:2277: left oid 46 (ObjNum 788 snap 255 seq_num 788) 2021-02-10T04:36:50.603 INFO:tasks.rados.rados.0.smithi085.stdout:2279: expect (ObjNum 768 snap 249 seq_num 768) 2021-02-10T04:36:50.662 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: cluster 2021-02-10T04:36:50.663 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2021-02-10T04:36:50.663 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2021 2021-02-10T04:36:50.663 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: - 2021-02-10T04:36:50.663 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 02 2021-02-10T04:36:50.663 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: - 2021-02-10T04:36:50.663 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 10T04:36:49 2021-02-10T04:36:50.663 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: .116484+0000 osd.2 (osd.2) 1661 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:50.663 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: cluster 2021-02-10T04:36:49.116492+0000 osd.2 (osd.2) 1662 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:50.663 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: cluster 2021-02-10T04:36:49.116498+0000 osd.2 (osd.2) 1663 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:50.663 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: cluster 2021-02-10T04:36:49.116502+0000 osd.2 ( 2021-02-10T04:36:50.663 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: osd.2) 1664 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:50.664 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:36:50.664 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: :36:49.116505+0000 osd.2 (osd.2) 2021-02-10T04:36:50.664 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 1665 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:50.664 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: cluster 2021-02-10T04:36:50.664 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2021-02-10T04:36:50.664 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2021 2021-02-10T04:36:50.664 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: - 2021-02-10T04:36:50.664 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 02 2021-02-10T04:36:50.664 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: - 2021-02-10T04:36:50.664 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 10 2021-02-10T04:36:50.664 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: T 2021-02-10T04:36:50.664 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 04 2021-02-10T04:36:50.664 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: : 2021-02-10T04:36:50.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 36 2021-02-10T04:36:50.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: : 2021-02-10T04:36:50.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 49 2021-02-10T04:36:50.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: . 2021-02-10T04:36:50.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 116508 2021-02-10T04:36:50.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: +0000 2021-02-10T04:36:50.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2021-02-10T04:36:50.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: osd.2 2021-02-10T04:36:50.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: ( 2021-02-10T04:36:50.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: osd 2021-02-10T04:36:50.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: . 2021-02-10T04:36:50.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2 2021-02-10T04:36:50.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: ) 2021-02-10T04:36:50.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 1666 2021-02-10T04:36:50.666 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: : 2021-02-10T04:36:50.666 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: cluster 2021-02-10T04:36:50.666 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2021-02-10T04:36:50.666 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: [WRN] 2021-02-10T04:36:50.666 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2021-02-10T04:36:50.666 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:50.666 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2021-02-10T04:36:50.666 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: cluster 2021-02-10T04:36:50.666 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2021-02-10T04:36:50.666 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2021 2021-02-10T04:36:50.666 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: - 2021-02-10T04:36:50.667 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 02 2021-02-10T04:36:50.667 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: - 2021-02-10T04:36:50.667 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 10 2021-02-10T04:36:50.667 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: T 2021-02-10T04:36:50.667 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 04 2021-02-10T04:36:50.667 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: : 2021-02-10T04:36:50.667 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 36 2021-02-10T04:36:50.667 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: : 2021-02-10T04:36:50.667 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 49 2021-02-10T04:36:50.667 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: . 2021-02-10T04:36:50.667 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 116512 2021-02-10T04:36:50.667 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: +0000 2021-02-10T04:36:50.667 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2021-02-10T04:36:50.667 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: osd.2 2021-02-10T04:36:50.668 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: ( 2021-02-10T04:36:50.668 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: osd 2021-02-10T04:36:50.668 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: . 2021-02-10T04:36:50.669 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2 2021-02-10T04:36:50.669 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: ) 2021-02-10T04:36:50.669 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 1667 2021-02-10T04:36:50.669 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: : 2021-02-10T04:36:50.669 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: cluster 2021-02-10T04:36:50.669 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2021-02-10T04:36:50.669 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: [WRN] 2021-02-10T04:36:50.669 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2021-02-10T04:36:50.669 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:50.669 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2021-02-10T04:36:50.669 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: cluster 2021-02-10T04:36:50.670 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2021-02-10T04:36:50.670 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2021 2021-02-10T04:36:50.670 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: - 2021-02-10T04:36:50.670 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 02 2021-02-10T04:36:50.670 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: - 2021-02-10T04:36:50.670 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 10 2021-02-10T04:36:50.670 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: T 2021-02-10T04:36:50.670 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 04 2021-02-10T04:36:50.670 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: : 2021-02-10T04:36:50.670 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 36 2021-02-10T04:36:50.670 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: : 2021-02-10T04:36:50.670 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 49 2021-02-10T04:36:50.670 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: . 2021-02-10T04:36:50.671 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 116516 2021-02-10T04:36:50.671 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: +0000 2021-02-10T04:36:50.671 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2021-02-10T04:36:50.671 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: osd.2 2021-02-10T04:36:50.671 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: ( 2021-02-10T04:36:50.671 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: osd 2021-02-10T04:36:50.671 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: . 2021-02-10T04:36:50.671 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2 2021-02-10T04:36:50.671 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: ) 2021-02-10T04:36:50.671 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 1668 2021-02-10T04:36:50.671 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: : 2021-02-10T04:36:50.671 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: cluster 2021-02-10T04:36:50.671 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2021-02-10T04:36:50.671 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: [WRN] 2021-02-10T04:36:50.672 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2021-02-10T04:36:50.672 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:50.672 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2021-02-10T04:36:50.672 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: cluster 2021-02-10T04:36:50.672 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2021-02-10T04:36:50.672 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2021 2021-02-10T04:36:50.672 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: - 2021-02-10T04:36:50.672 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 02 2021-02-10T04:36:50.672 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: - 2021-02-10T04:36:50.672 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 10 2021-02-10T04:36:50.672 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: T 2021-02-10T04:36:50.672 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 04 2021-02-10T04:36:50.672 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: : 2021-02-10T04:36:50.673 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 36 2021-02-10T04:36:50.673 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: : 2021-02-10T04:36:50.673 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 49 2021-02-10T04:36:50.673 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: . 2021-02-10T04:36:50.673 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 116520 2021-02-10T04:36:50.673 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: +0000 2021-02-10T04:36:50.673 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2021-02-10T04:36:50.673 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: osd.2 2021-02-10T04:36:50.673 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: ( 2021-02-10T04:36:50.673 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: osd 2021-02-10T04:36:50.673 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: . 2021-02-10T04:36:50.673 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2 2021-02-10T04:36:50.673 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: ) 2021-02-10T04:36:50.673 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 1669 2021-02-10T04:36:50.674 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: : 2021-02-10T04:36:50.674 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: cluster 2021-02-10T04:36:50.674 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2021-02-10T04:36:50.674 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: [WRN] 2021-02-10T04:36:50.674 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2021-02-10T04:36:50.674 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:50.674 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2021-02-10T04:36:50.674 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: cluster 2021-02-10T04:36:50.674 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:36:50.674 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 36:49.116523+0000 osd.2 (osd.2) 1670 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:50.674 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:36:50.674 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: T04:36:50.098434+0000 osd.4 (osd.4) 2021-02-10T04:36:50.675 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 869 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:50.675 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:36:50.675 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 10T04:36:50.098443+0000 osd.4 ( 2021-02-10T04:36:50.675 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: osd.4) 870 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:50.675 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:50.675 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: 02-10T04:36:50.098449+0000 osd.4 2021-02-10T04:36:50.675 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: (osd.4) 871 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:50.675 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:50.675 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: -10T04:36:50.098455+0000 osd.4 (osd.4 2021-02-10T04:36:50.675 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: ) 872 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:50.675 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:50.675 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: -10T04:36:50.125526+0000 mgr.y ( 2021-02-10T04:36:50.675 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:50 smithi164 conmon[29288]: mgr.14140) 647 : cluster [DBG] pgmap v1038: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 446 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 689 KiB/s wr, 11 op/s 2021-02-10T04:36:50.818 INFO:tasks.rados.rados.0.smithi085.stdout:2280: finishing write tid 1 to smithi08575397-50 2021-02-10T04:36:50.819 INFO:tasks.rados.rados.0.smithi085.stdout:2280: finishing write tid 2 to smithi08575397-50 2021-02-10T04:36:50.819 INFO:tasks.rados.rados.0.smithi085.stdout:2281: expect (ObjNum 756 snap 241 seq_num 756) 2021-02-10T04:36:51.022 INFO:tasks.rados.rados.0.smithi085.stdout:2280: finishing write tid 3 to smithi08575397-50 2021-02-10T04:36:51.022 INFO:tasks.rados.rados.0.smithi085.stdout:2280: finishing write tid 4 to smithi08575397-50 2021-02-10T04:36:51.022 INFO:tasks.rados.rados.0.smithi085.stdout:2280: finishing write tid 5 to smithi08575397-50 2021-02-10T04:36:51.022 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 1128 (ObjNum 789 snap 255 seq_num 789) dirty exists 2021-02-10T04:36:51.022 INFO:tasks.rados.rados.0.smithi085.stdout:2280: left oid 50 (ObjNum 789 snap 255 seq_num 789) 2021-02-10T04:36:51.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[33677]: debug 2021-02-10T04:36:51.079+0000 7fbec19a1700 -1 osd.4 593 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:51.409 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[49106]: debug 2021-02-10T04:36:51.409 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[49106]: 2021-02-10T04:36:51.114+0000 7f77a3e94700 -1 osd.2 593 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:51.409 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[49106]: 2021-02-10T04:36:51.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: cluster 2021-02-10T04:36:49.563399+0000 osd.1 (osd.1) 2021-02-10T04:36:51.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: 451 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:36:51.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: cluster 2021-02-10T04:36:49.563682+0000 osd.1 (osd.1 2021-02-10T04:36:51.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: ) 452 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:36:51.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:51.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: -10T04:36:50.115365+0000 osd.2 (osd.2) 1671 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:51.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:51.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: T04:36:50.115372+0000 osd.2 2021-02-10T04:36:51.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: (osd.2) 1672 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:51.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:36:51.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: 36:50.115377+0000 osd.2 (osd.2) 1673 : cluster [WRN] 2021-02-10T04:36:51.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:51.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:51.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: 02-10T04:36:50.115381+0000 osd.2 (osd.2) 1674 : cluster 2021-02-10T04:36:51.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:51.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:51.414 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: -10T04:36:50.115384+0000 osd.2 (osd.2) 2021-02-10T04:36:51.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: 1675 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:51.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:51.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: -10T04:36:50.115387+0000 osd.2 (osd.2 2021-02-10T04:36:51.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: ) 1676 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:51.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: cluster 2021-02-10T04:36:50.115391 2021-02-10T04:36:51.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1677 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:51.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:51.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: 02-10T04:36:50.115395+0000 osd.2 ( 2021-02-10T04:36:51.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: osd.2) 1678 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:51.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:51.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: 02-10T04:36:50.115398+0000 osd.2 (osd 2021-02-10T04:36:51.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: .2) 1679 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:51.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: cluster 2021-02-10T04:36:51.415 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: 2021-02-10T04:36:51.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: 2021 2021-02-10T04:36:51.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: -02-10T04:36: 2021-02-10T04:36:51.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: 50.115402+0000 osd.2 (osd.2) 1680 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:51.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: cluster 2021-02-10T04:36:51.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:36:51.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: 10T04:36:50.405521+0000 mon.a 2021-02-10T04:36:51.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: (mon.0) 1444 : cluster 2021-02-10T04:36:51.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: [DBG] osdmap e593: 8 total, 8 up, 8 in 2021-02-10T04:36:51.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:36:51.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: 36:51.081222+0000 osd.4 (osd.4) 873 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:51.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: cluster 2021-02-10T04:36:51. 2021-02-10T04:36:51.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: 081232+0000 osd.4 (osd.4) 874 : cluster [WRN] 2021-02-10T04:36:51.416 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:51.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:36:51.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: 10T04:36:51.081238+0000 osd.4 (osd.4) 875 2021-02-10T04:36:51.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:51.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:51.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: 02-10T04:36:51.081243+0000 osd.4 (osd.4) 2021-02-10T04:36:51.417 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[35961]: 876 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:51.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: cluster 2021-02-10T04:36:49.563399+0000 osd.1 (osd 2021-02-10T04:36:51.418 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: .1) 451 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:36:51.418 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: cluster 2021-02-10T04:36:49.563682+0000 osd.1 (osd.1) 452 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:36:51.418 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: cluster 2021-02-10T04:36:50.115365+0000 2021-02-10T04:36:51.418 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: osd.2 (osd.2) 1671 : cluster [WRN] 2021-02-10T04:36:51.418 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:51.418 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: cluster 2021-02-10T04:36:51.418 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:36:51.418 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: :36:50.115372+0000 osd.2 (osd.2) 1672 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:51.418 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: 2021-02-10T04:36:51.418 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:51.418 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: -02-10T04:36:50.115377+0000 2021-02-10T04:36:51.419 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: osd.2 (osd.2) 1673 : cluster [WRN] 2021-02-10T04:36:51.419 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:51.419 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: cluster 2021-02-10T04:36:50. 2021-02-10T04:36:51.419 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: 115381+0000 osd.2 (osd.2) 1674 : cluster 2021-02-10T04:36:51.419 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:51.419 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: cluster 2021-02-10T04:36 2021-02-10T04:36:51.419 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: :50.115384+0000 osd.2 (osd.2) 2021-02-10T04:36:51.419 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: 1675 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:51.419 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:36:51.419 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: T04:36:50.115387+0000 osd.2 (osd 2021-02-10T04:36:51.419 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: .2) 1676 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:51.420 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:51.420 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: -10T04:36:50.115391+0000 osd.2 2021-02-10T04:36:51.420 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: (osd.2) 1677 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:51.420 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: 2021-02-10T04:36:51.420 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:51.420 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: -02-10T04:36:50.115395+0000 osd.2 (osd 2021-02-10T04:36:51.420 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: .2) 1678 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:51.420 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:51.420 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: -10T04:36:50.115398+0000 osd.2 2021-02-10T04:36:51.420 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: (osd.2) 1679 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:51.420 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: 2021-02-10T04:36:51.420 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: cluster 2021-02-10T04:36:50.115402 2021-02-10T04:36:51.420 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1680 : cluster [WRN] 2021-02-10T04:36:51.421 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:51.421 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: cluster 2021-02-10T04:36:50 2021-02-10T04:36:51.421 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: .405521+0000 mon.a (mon.0) 1444 : cluster 2021-02-10T04:36:51.421 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: [DBG] osdmap e593: 8 total, 8 up, 8 in 2021-02-10T04:36:51.421 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: cluster 2021-02-10T04:36 2021-02-10T04:36:51.421 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: :51.081222+0000 osd.4 (osd.4) 2021-02-10T04:36:51.422 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: 873 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:51.422 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:36:51.422 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: 04:36:51.081232+0000 osd.4 (osd. 2021-02-10T04:36:51.422 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: 4) 874 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:51.423 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:51.423 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: -10T04:36:51.081238+0000 osd.4 2021-02-10T04:36:51.423 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: (osd.4) 875 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:51.423 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: 2021-02-10T04:36:51.423 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:51.423 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: -02-10T04:36:51.081243+0000 osd.4 (osd 2021-02-10T04:36:51.424 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:51 smithi085 conmon[29319]: .4) 876 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:51.425 INFO:tasks.rados.rados.0.smithi085.stdout:2273: done (15 left) 2021-02-10T04:36:51.425 INFO:tasks.rados.rados.0.smithi085.stdout:2276: done (14 left) 2021-02-10T04:36:51.425 INFO:tasks.rados.rados.0.smithi085.stdout:2277: done (13 left) 2021-02-10T04:36:51.425 INFO:tasks.rados.rados.0.smithi085.stdout:2279: done (12 left) 2021-02-10T04:36:51.426 INFO:tasks.rados.rados.0.smithi085.stdout:2280: done (11 left) 2021-02-10T04:36:51.426 INFO:tasks.rados.rados.0.smithi085.stdout:2281: done (10 left) 2021-02-10T04:36:51.426 INFO:tasks.rados.rados.0.smithi085.stdout:2282: done (9 left) 2021-02-10T04:36:51.426 INFO:tasks.rados.rados.0.smithi085.stdout:2283: write (excl) oid 13 current snap is 256 2021-02-10T04:36:51.426 INFO:tasks.rados.rados.0.smithi085.stdout:2283: seq_num 790 ranges {477530=631069,1612657=409606,2545223=517596} 2021-02-10T04:36:51.427 INFO:tasks.rados.rados.0.smithi085.stdout:2283: writing smithi08575397-13 from 477530 to 1108599 tid 1 2021-02-10T04:36:51.431 INFO:tasks.rados.rados.0.smithi085.stdout:2283: writing smithi08575397-13 from 1612657 to 2022263 tid 2 2021-02-10T04:36:51.438 INFO:tasks.rados.rados.0.smithi085.stdout:2283: writing smithi08575397-13 from 2545223 to 3062819 tid 3 2021-02-10T04:36:51.439 INFO:tasks.rados.rados.0.smithi085.stdout:2284: read oid 20 snap -1 2021-02-10T04:36:51.439 INFO:tasks.rados.rados.0.smithi085.stdout:2284: expect deleted 2021-02-10T04:36:51.439 INFO:tasks.rados.rados.0.smithi085.stdout:2285: write (excl) oid 35 current snap is 256 2021-02-10T04:36:51.439 INFO:tasks.rados.rados.0.smithi085.stdout:2283: finishing write tid 1 to smithi08575397-13 2021-02-10T04:36:51.439 INFO:tasks.rados.rados.0.smithi085.stdout:2283: finishing write tid 2 to smithi08575397-13 2021-02-10T04:36:51.440 INFO:tasks.rados.rados.0.smithi085.stdout:2285: seq_num 791 ranges {494337=422671,1387492=553264,2688999=445221} 2021-02-10T04:36:51.443 INFO:tasks.rados.rados.0.smithi085.stdout:2285: writing smithi08575397-35 from 494337 to 917008 tid 1 2021-02-10T04:36:51.450 INFO:tasks.rados.rados.0.smithi085.stdout:2285: writing smithi08575397-35 from 1387492 to 1940756 tid 2 2021-02-10T04:36:51.458 INFO:tasks.rados.rados.0.smithi085.stdout:2285: writing smithi08575397-35 from 2688999 to 3134220 tid 3 2021-02-10T04:36:51.458 INFO:tasks.rados.rados.0.smithi085.stdout:2286: copy_from oid 43 from oid 10 current snap is 256 2021-02-10T04:36:51.458 INFO:tasks.rados.rados.0.smithi085.stdout:2283: finishing write tid 3 to smithi08575397-13 2021-02-10T04:36:51.458 INFO:tasks.rados.rados.0.smithi085.stdout:2283: finishing write tid 4 to smithi08575397-13 2021-02-10T04:36:51.458 INFO:tasks.rados.rados.0.smithi085.stdout:2283: finishing write tid 5 to smithi08575397-13 2021-02-10T04:36:51.458 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 763 (ObjNum 790 snap 256 seq_num 790) dirty exists 2021-02-10T04:36:51.459 INFO:tasks.rados.rados.0.smithi085.stdout:2283: left oid 13 (ObjNum 790 snap 256 seq_num 790) 2021-02-10T04:36:51.459 INFO:tasks.rados.rados.0.smithi085.stdout:2285: finishing write tid 1 to smithi08575397-35 2021-02-10T04:36:51.459 INFO:tasks.rados.rados.0.smithi085.stdout:2285: finishing write tid 2 to smithi08575397-35 2021-02-10T04:36:51.459 INFO:tasks.rados.rados.0.smithi085.stdout:2283: done (12 left) 2021-02-10T04:36:51.459 INFO:tasks.rados.rados.0.smithi085.stdout:2284: done (11 left) 2021-02-10T04:36:51.459 INFO:tasks.rados.rados.0.smithi085.stdout:2287: rollback oid 50 current snap is 256 2021-02-10T04:36:51.460 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 50 to 184 2021-02-10T04:36:51.460 INFO:tasks.rados.rados.0.smithi085.stdout:2288: read oid 2 snap -1 2021-02-10T04:36:51.460 INFO:tasks.rados.rados.0.smithi085.stdout:2288: expect (ObjNum 765 snap 246 seq_num 765) 2021-02-10T04:36:51.460 INFO:tasks.rados.rados.0.smithi085.stdout:2289: write (excl) oid 28 current snap is 256 2021-02-10T04:36:51.461 INFO:tasks.rados.rados.0.smithi085.stdout:2289: seq_num 792 ranges {511144=497920,1445974=413275,2316422=656493} 2021-02-10T04:36:51.463 INFO:tasks.rados.rados.0.smithi085.stdout:2289: writing smithi08575397-28 from 511144 to 1009064 tid 1 2021-02-10T04:36:51.469 INFO:tasks.rados.rados.0.smithi085.stdout:2289: writing smithi08575397-28 from 1445974 to 1859249 tid 2 2021-02-10T04:36:51.476 INFO:tasks.rados.rados.0.smithi085.stdout:2289: writing smithi08575397-28 from 2316422 to 2972915 tid 3 2021-02-10T04:36:51.476 INFO:tasks.rados.rados.0.smithi085.stdout:2290: snap_remove snap 251 2021-02-10T04:36:51.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: cluster 2021-02-10T04:36:49.563399+0000 osd.1 2021-02-10T04:36:51.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: (osd.1) 451 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:36:51.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: cluster 2021-02-10T04:36:49.563682+0000 osd.1 (osd.1) 452 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:36:51.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:51.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: -02-10T04:36:50.115365+0000 osd.2 (osd.2) 1671 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:51.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: cluster 2021-02-10T04:36:50.115372 2021-02-10T04:36:51.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1672 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:51.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: 2021-02-10T04:36:51.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: cluster 2021-02-10T04:36:51.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: 2021-02-10T04:36:50.115377+0000 osd.2 (osd 2021-02-10T04:36:51.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: .2) 1673 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:51.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: cluster 2021-02-10T04:36:50.115381+0000 osd.2 (osd. 2021-02-10T04:36:51.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: 2) 1674 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:51.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:36:51.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: :36:50.115384+0000 osd.2 (osd.2 2021-02-10T04:36:51.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: ) 1675 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:51.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: cluster 2021-02-10T04:36 2021-02-10T04:36:51.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: :50.115387+0000 osd.2 (osd.2) 1676 : 2021-02-10T04:36:51.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:51.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: cluster 2021-02-10T04:36 2021-02-10T04:36:51.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: :50.115391+0000 osd.2 (osd.2) 1677 2021-02-10T04:36:51.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:51.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:36:51.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: 36:50.115395+0000 osd.2 (osd.2) 2021-02-10T04:36:51.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: 1678 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:51.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:36:51.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: 04:36:50.115398+0000 osd.2 (osd. 2021-02-10T04:36:51.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: 2) 1679 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:51.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:51.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: -10T04:36:50.115402+0000 osd.2 2021-02-10T04:36:51.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: (osd.2) 1680 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:51.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:51.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: -02-10T04:36:50.405521+0000 2021-02-10T04:36:51.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: mon.a (mon.0) 1444 : cluster [DBG] 2021-02-10T04:36:51.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: osdmap e593: 8 total, 8 up, 8 in 2021-02-10T04:36:51.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: cluster 2021-02-10T04:36:51.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: 2021-02-10T04:36:51.081222+0000 osd.4 ( 2021-02-10T04:36:51.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: osd.4) 873 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:51.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:51.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: -10T04:36:51.081232+0000 osd.4 2021-02-10T04:36:51.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: (osd.4) 874 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:51.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:51.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: -02-10T04:36:51.081238+0000 2021-02-10T04:36:51.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: osd.4 (osd.4) 875 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:51.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: 2021-02-10T04:36:51.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: cluster 2021-02-10T04:36:51.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: 2021-02-10T04:36:51.081243+0000 osd.4 (osd 2021-02-10T04:36:51.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:51 smithi164 conmon[29288]: .4) 876 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:52.176 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:36:52.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[33677]: debug 2021-02-10T04:36:52.073+0000 7fbec19a1700 -1 osd.4 594 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:52.424 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[49106]: debug 2021-02-10T04:36:52.007+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:36:52.424 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[49106]: debug 2021-02-10T04:36:52.088+0000 7f77a3e94700 -1 osd.2 594 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:52.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: cluster 2021-02-10T04:36:51.116230+0000 osd.2 (osd.2) 1681 2021-02-10T04:36:52.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:52.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: cluster 2021-02-10T04:36:51.116238+0000 osd.2 (osd.2) 1682 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:52.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: cluster 2021-02-10T04:36:51.116245+0000 osd.2 (osd.2) 1683 : cluster 2021-02-10T04:36:52.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:52.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: cluster 2021-02-10T04:36:51 2021-02-10T04:36:52.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: .116251+0000 osd.2 (osd.2) 1684 : cluster [WRN] 2021-02-10T04:36:52.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:52.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: cluster 2021-02-10T04:36:52.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: 2021-02-10T04:36:51.116260+0000 osd.2 (osd.2) 1685 2021-02-10T04:36:52.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:52.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: cluster 2021-02-10T04:36:51 2021-02-10T04:36:52.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: .116265+0000 osd.2 (osd.2) 1686 : cluster 2021-02-10T04:36:52.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:52.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: cluster 2021-02-10T04:36:51. 2021-02-10T04:36:52.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: 116270+0000 osd.2 (osd.2) 1687 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:52.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: 2021-02-10T04:36:52.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:52.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: -02-10T04:36:51.116279+0000 osd.2 (osd.2) 2021-02-10T04:36:52.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: 1688 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:52.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:36:52.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: :36:51.116285+0000 osd.2 (osd.2) 1689 2021-02-10T04:36:52.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:52.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: cluster 2021-02-10T04:36:51.116290 2021-02-10T04:36:52.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1690 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:52.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: 2021-02-10T04:36:52.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:52.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: -10T04:36:51.418649+0000 mon.a (mon.0) 1445 : cluster 2021-02-10T04:36:52.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: [DBG] osdmap e594: 8 total, 8 up, 8 in 2021-02-10T04:36:52.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:36:52.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: 36:52.075585+0000 osd.4 (osd.4) 2021-02-10T04:36:52.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: 877 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:52.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:36:52.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: :36:52.075600+0000 osd.4 (osd.4) 2021-02-10T04:36:52.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: 878 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:52.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: cluster 2021-02-10T04:36:52. 2021-02-10T04:36:52.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: 075608+0000 osd.4 (osd.4) 879 : cluster [WRN] 2021-02-10T04:36:52.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:52.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:36:52.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: :36:52.075615+0000 osd.4 2021-02-10T04:36:52.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: (osd.4) 880 : cluster [WRN] 2021-02-10T04:36:52.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:52.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:36:52.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: T04:36:52.126140+0000 mgr.y 2021-02-10T04:36:52.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: (mgr.14140) 648 : cluster [DBG] 2021-02-10T04:36:52.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[29319]: pgmap v1041: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 445 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 1.0 MiB/s wr, 14 op/s 2021-02-10T04:36:52.432 INFO:tasks.rados.rados.0.smithi085.stdout:2290: done (14 left) 2021-02-10T04:36:52.432 INFO:tasks.rados.rados.0.smithi085.stdout:2291: delete oid 13 current snap is 256 2021-02-10T04:36:52.432 INFO:tasks.rados.rados.0.smithi085.stdout:2287: finishing rollback tid 0 to smithi08575397-50 2021-02-10T04:36:52.432 INFO:tasks.rados.rados.0.smithi085.stdout:2285: finishing write tid 3 to smithi08575397-35 2021-02-10T04:36:52.432 INFO:tasks.rados.rados.0.smithi085.stdout:2287: finishing rollback tid 1 to smithi08575397-50 2021-02-10T04:36:52.432 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 1130 (ObjNum 1953066355 snap 942696808 seq_num 859125557) dirty dne 2021-02-10T04:36:52.432 INFO:tasks.rados.rados.0.smithi085.stdout:2288: expect (ObjNum 765 snap 246 seq_num 765) 2021-02-10T04:36:52.658 INFO:tasks.rados.rados.0.smithi085.stdout:2285: finishing write tid 4 to smithi08575397-35 2021-02-10T04:36:52.658 INFO:tasks.rados.rados.0.smithi085.stdout:2285: finishing write tid 5 to smithi08575397-35 2021-02-10T04:36:52.658 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 35 v 844 (ObjNum 791 snap 256 seq_num 791) dirty exists 2021-02-10T04:36:52.658 INFO:tasks.rados.rados.0.smithi085.stdout:2285: left oid 35 (ObjNum 791 snap 256 seq_num 791) 2021-02-10T04:36:52.658 INFO:tasks.rados.rados.0.smithi085.stdout:2289: finishing write tid 1 to smithi08575397-28 2021-02-10T04:36:52.659 INFO:tasks.rados.rados.0.smithi085.stdout:2289: finishing write tid 2 to smithi08575397-28 2021-02-10T04:36:52.659 INFO:tasks.rados.rados.0.smithi085.stdout:2289: finishing write tid 3 to smithi08575397-28 2021-02-10T04:36:52.659 INFO:tasks.rados.rados.0.smithi085.stdout:2289: finishing write tid 4 to smithi08575397-28 2021-02-10T04:36:52.659 INFO:tasks.rados.rados.0.smithi085.stdout:2289: finishing write tid 5 to smithi08575397-28 2021-02-10T04:36:52.659 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 926 (ObjNum 792 snap 256 seq_num 792) dirty exists 2021-02-10T04:36:52.659 INFO:tasks.rados.rados.0.smithi085.stdout:2289: left oid 28 (ObjNum 792 snap 256 seq_num 792) 2021-02-10T04:36:52.659 INFO:tasks.rados.rados.0.smithi085.stdout:2286: finishing copy_from to smithi08575397-43 2021-02-10T04:36:52.659 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 837 (ObjNum 772 snap 250 seq_num 772) dirty exists 2021-02-10T04:36:52.659 INFO:tasks.rados.rados.0.smithi085.stdout:2286: finishing copy_from racing read to smithi08575397-43 2021-02-10T04:36:52.659 INFO:tasks.rados.rados.0.smithi085.stdout:2285: done (14 left) 2021-02-10T04:36:52.659 INFO:tasks.rados.rados.0.smithi085.stdout:2286: done (13 left) 2021-02-10T04:36:52.659 INFO:tasks.rados.rados.0.smithi085.stdout:2287: done (12 left) 2021-02-10T04:36:52.660 INFO:tasks.rados.rados.0.smithi085.stdout:2288: done (11 left) 2021-02-10T04:36:52.660 INFO:tasks.rados.rados.0.smithi085.stdout:2289: done (10 left) 2021-02-10T04:36:52.660 INFO:tasks.rados.rados.0.smithi085.stdout:2291: done (9 left) 2021-02-10T04:36:52.660 INFO:tasks.rados.rados.0.smithi085.stdout:2292: read oid 20 snap -1 2021-02-10T04:36:52.660 INFO:tasks.rados.rados.0.smithi085.stdout:2292: expect deleted 2021-02-10T04:36:52.660 INFO:tasks.rados.rados.0.smithi085.stdout:2293: write (excl) oid 6 current snap is 256 2021-02-10T04:36:52.660 INFO:tasks.rados.rados.0.smithi085.stdout:2293: seq_num 794 ranges {544758=648418,1846585=416944,2771268=573490} 2021-02-10T04:36:52.665 INFO:tasks.rados.rados.0.smithi085.stdout:2293: writing smithi08575397-6 from 544758 to 1193176 tid 1 2021-02-10T04:36:52.671 INFO:tasks.rados.rados.0.smithi085.stdout:2293: writing smithi08575397-6 from 1846585 to 2263529 tid 2 2021-02-10T04:36:52.680 INFO:tasks.rados.rados.0.smithi085.stdout:2293: writing smithi08575397-6 from 2771268 to 3344758 tid 3 2021-02-10T04:36:52.680 INFO:tasks.rados.rados.0.smithi085.stdout:2293: finishing write tid 1 to smithi08575397-6 2021-02-10T04:36:52.680 INFO:tasks.rados.rados.0.smithi085.stdout:2293: finishing write tid 2 to smithi08575397-6 2021-02-10T04:36:52.680 INFO:tasks.rados.rados.0.smithi085.stdout:2292: done (10 left) 2021-02-10T04:36:52.681 INFO:tasks.rados.rados.0.smithi085.stdout:2294: copy_from oid 36 from oid 1 current snap is 256 2021-02-10T04:36:52.681 INFO:tasks.rados.rados.0.smithi085.stdout:2295: read oid 46 snap 190 2021-02-10T04:36:52.681 INFO:tasks.rados.rados.0.smithi085.stdout:2295: expect (ObjNum 521 snap 155 seq_num 521) 2021-02-10T04:36:52.681 INFO:tasks.rados.rados.0.smithi085.stdout:2296: read oid 5 snap -1 2021-02-10T04:36:52.681 INFO:tasks.rados.rados.0.smithi085.stdout:2296: expect (ObjNum 776 snap 250 seq_num 776) 2021-02-10T04:36:52.681 INFO:tasks.rados.rados.0.smithi085.stdout:2297: write (excl) oid 12 current snap is 256 2021-02-10T04:36:52.681 INFO:tasks.rados.rados.0.smithi085.stdout:2297: seq_num 795 ranges {561565=723667,1905067=560602,3082338=279227} 2021-02-10T04:36:52.687 INFO:tasks.rados.rados.0.smithi085.stdout:2297: writing smithi08575397-12 from 561565 to 1285232 tid 1 2021-02-10T04:36:52.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: cluster 2021-02-10T04:36:51.116230+0000 2021-02-10T04:36:52.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: osd.2 (osd.2) 1681 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:52.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: cluster 2021-02-10T04:36:51.116238 2021-02-10T04:36:52.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1682 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:52.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: cluster 2021-02-10T04:36:51.116245 2021-02-10T04:36:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1683 : cluster [WRN] 2021-02-10T04:36:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: cluster 2021-02-10T04:36:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: 2021-02-10T04:36:51.116251 2021-02-10T04:36:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1684 : cluster [WRN] 2021-02-10T04:36:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: cluster 2021-02-10T04:36:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: 2021-02-10T04:36:51.116260+0000 osd.2 (osd 2021-02-10T04:36:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: .2) 1685 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:36:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: 04:36:51.116265+0000 osd.2 (osd.2 2021-02-10T04:36:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: ) 1686 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:36:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: :36:51.116270+0000 osd.2 (osd.2 2021-02-10T04:36:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: ) 1687 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:36:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: 04:36:51.116279+0000 osd.2 (osd.2 2021-02-10T04:36:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: ) 1688 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:36:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: 04:36:51.116285+0000 osd.2 (osd.2 2021-02-10T04:36:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: ) 1689 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:36:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: 04:36:51.116290+0000 osd.2 (osd. 2021-02-10T04:36:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: 2) 1690 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: T04:36:51.418649+0000 mon.a (mon 2021-02-10T04:36:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: .0) 1445 : cluster [DBG] osdmap e594: 8 total, 8 up, 8 in 2021-02-10T04:36:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: -10T04:36:52.075585+0000 osd.4 (osd 2021-02-10T04:36:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: .4) 877 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: -02-10T04:36:52.075600+0000 2021-02-10T04:36:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: osd.4 (osd.4) 878 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: 2021-02-10T04:36:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: cluster 2021-02-10T04:36:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: 2021-02-10T04:36:52.075608+0000 osd.4 (osd 2021-02-10T04:36:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: .4) 879 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:52.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: -10T04:36:52.075615+0000 osd.4 ( 2021-02-10T04:36:52.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: osd.4) 880 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:52.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:52.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: -10T04:36:52.126140+0000 2021-02-10T04:36:52.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[35961]: mgr.y (mgr.14140) 648 : cluster [DBG] pgmap v1041: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 445 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 1.0 MiB/s wr, 14 op/s 2021-02-10T04:36:52.698 INFO:tasks.rados.rados.0.smithi085.stdout:2297: writing smithi08575397-12 from 1905067 to 2465669 tid 2 2021-02-10T04:36:52.700 INFO:tasks.rados.rados.0.smithi085.stdout:2297: writing smithi08575397-12 from 3082338 to 3361565 tid 3 2021-02-10T04:36:52.700 INFO:tasks.rados.rados.0.smithi085.stdout:2298: write oid 3 current snap is 256 2021-02-10T04:36:52.700 INFO:tasks.rados.rados.0.smithi085.stdout:2298: seq_num 796 ranges {578372=798916,1963549=420613,2826114=417934} 2021-02-10T04:36:52.709 INFO:tasks.rados.rados.0.smithi085.stdout:2298: writing smithi08575397-3 from 578372 to 1377288 tid 1 2021-02-10T04:36:52.715 INFO:tasks.rados.rados.0.smithi085.stdout:2298: writing smithi08575397-3 from 1963549 to 2384162 tid 2 2021-02-10T04:36:52.721 INFO:tasks.rados.rados.0.smithi085.stdout:2298: writing smithi08575397-3 from 2826114 to 3244048 tid 3 2021-02-10T04:36:52.721 INFO:tasks.rados.rados.0.smithi085.stdout:2299: read oid 28 snap -1 2021-02-10T04:36:52.721 INFO:tasks.rados.rados.0.smithi085.stdout:2299: expect (ObjNum 792 snap 256 seq_num 792) 2021-02-10T04:36:52.721 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:36:52.721 INFO:tasks.rados.rados.0.smithi085.stdout:2296: expect (ObjNum 776 snap 250 seq_num 776) 2021-02-10T04:36:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: cluster 2021-02-10T04:36:51.116230 2021-02-10T04:36:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1681 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: cluster 2021-02-10T04:36:51. 2021-02-10T04:36:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: 116238+0000 osd.2 (osd.2) 1682 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: cluster 2021-02-10T04:36: 2021-02-10T04:36:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: 51.116245+0000 osd.2 (osd.2) 1683 : 2021-02-10T04:36:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: cluster 2021-02-10T04:36:51 2021-02-10T04:36:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: .116251+0000 osd.2 (osd.2) 1684 : cluster 2021-02-10T04:36:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: cluster 2021-02-10T04:36:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: 2021-02-10T04:36:51.116260+0000 osd.2 ( 2021-02-10T04:36:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: osd.2) 1685 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:36:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: T04:36:51.116265+0000 osd.2 (osd 2021-02-10T04:36:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: .2) 1686 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: -10T04:36:51.116270+0000 osd.2 2021-02-10T04:36:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: (osd.2) 1687 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:36:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: T04:36:51.116279+0000 osd.2 (osd 2021-02-10T04:36:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: .2) 1688 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:36:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: 04:36:51.116285+0000 osd.2 (osd. 2021-02-10T04:36:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: 2) 1689 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:36:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: 04:36:51.116290+0000 osd.2 (osd 2021-02-10T04:36:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: .2) 1690 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: 02-10T04:36:51.418649+0000 2021-02-10T04:36:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: mon.a (mon.0) 1445 : cluster [DBG] 2021-02-10T04:36:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: osdmap e594: 8 total, 8 up, 8 in 2021-02-10T04:36:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: cluster 2021-02-10T04:36:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: 2021-02-10T04:36:52.075585+0000 osd.4 ( 2021-02-10T04:36:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: osd.4) 877 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:36:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: T04:36:52.075600+0000 osd.4 (osd 2021-02-10T04:36:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: .4) 878 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:36:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: 04:36:52.075608+0000 osd.4 (osd 2021-02-10T04:36:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: .4) 879 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:36:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: 10T04:36:52.075615+0000 osd.4 ( 2021-02-10T04:36:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: osd.4) 880 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:36:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: 10T04:36:52.126140+0000 mgr.y 2021-02-10T04:36:52.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:52 smithi164 conmon[29288]: (mgr.14140) 648 : cluster [DBG] pgmap v1041: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 445 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 1.0 MiB/s wr, 14 op/s 2021-02-10T04:36:52.933 INFO:tasks.rados.rados.0.smithi085.stdout:2295: expect (ObjNum 521 snap 155 seq_num 521) 2021-02-10T04:36:53.048 INFO:tasks.rados.rados.0.smithi085.stdout:2293: finishing write tid 3 to smithi08575397-6 2021-02-10T04:36:53.048 INFO:tasks.rados.rados.0.smithi085.stdout:2293: finishing write tid 4 to smithi08575397-6 2021-02-10T04:36:53.048 INFO:tasks.rados.rados.0.smithi085.stdout:2293: finishing write tid 5 to smithi08575397-6 2021-02-10T04:36:53.048 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 1069 (ObjNum 794 snap 256 seq_num 794) dirty exists 2021-02-10T04:36:53.048 INFO:tasks.rados.rados.0.smithi085.stdout:2293: left oid 6 (ObjNum 794 snap 256 seq_num 794) 2021-02-10T04:36:53.048 INFO:tasks.rados.rados.0.smithi085.stdout:2297: finishing write tid 1 to smithi08575397-12 2021-02-10T04:36:53.049 INFO:tasks.rados.rados.0.smithi085.stdout:2293: done (15 left) 2021-02-10T04:36:53.049 INFO:tasks.rados.rados.0.smithi085.stdout:2295: done (14 left) 2021-02-10T04:36:53.049 INFO:tasks.rados.rados.0.smithi085.stdout:2296: done (13 left) 2021-02-10T04:36:53.049 INFO:tasks.rados.rados.0.smithi085.stdout:2300: write (excl) oid 43 current snap is 256 2021-02-10T04:36:53.049 INFO:tasks.rados.rados.0.smithi085.stdout:2297: finishing write tid 2 to smithi08575397-12 2021-02-10T04:36:53.049 INFO:tasks.rados.rados.0.smithi085.stdout:2297: finishing write tid 3 to smithi08575397-12 2021-02-10T04:36:53.049 INFO:tasks.rados.rados.0.smithi085.stdout:2297: finishing write tid 4 to smithi08575397-12 2021-02-10T04:36:53.049 INFO:tasks.rados.rados.0.smithi085.stdout:2300: seq_num 797 ranges {595179=474165,1622031=564271,2737184=629206} 2021-02-10T04:36:53.054 INFO:tasks.rados.rados.0.smithi085.stdout:2300: writing smithi08575397-43 from 595179 to 1069344 tid 1 2021-02-10T04:36:53.061 INFO:tasks.rados.rados.0.smithi085.stdout:2300: writing smithi08575397-43 from 1622031 to 2186302 tid 2 2021-02-10T04:36:53.069 INFO:tasks.rados.rados.0.smithi085.stdout:2300: writing smithi08575397-43 from 2737184 to 3366390 tid 3 2021-02-10T04:36:53.069 INFO:tasks.rados.rados.0.smithi085.stdout:2301: write (excl) oid 31 current snap is 256 2021-02-10T04:36:53.069 INFO:tasks.rados.rados.0.smithi085.stdout:2294: finishing copy_from to smithi08575397-36 2021-02-10T04:36:53.070 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 759 (ObjNum 776 snap 250 seq_num 776) dirty exists 2021-02-10T04:36:53.070 INFO:tasks.rados.rados.0.smithi085.stdout:2297: finishing write tid 5 to smithi08575397-12 2021-02-10T04:36:53.070 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 847 (ObjNum 795 snap 256 seq_num 795) dirty exists 2021-02-10T04:36:53.070 INFO:tasks.rados.rados.0.smithi085.stdout:2297: left oid 12 (ObjNum 795 snap 256 seq_num 795) 2021-02-10T04:36:53.070 INFO:tasks.rados.rados.0.smithi085.stdout:2294: finishing copy_from racing read to smithi08575397-36 2021-02-10T04:36:53.070 INFO:tasks.rados.rados.0.smithi085.stdout:2298: finishing write tid 1 to smithi08575397-3 2021-02-10T04:36:53.070 INFO:tasks.rados.rados.0.smithi085.stdout:2298: finishing write tid 2 to smithi08575397-3 2021-02-10T04:36:53.070 INFO:tasks.rados.rados.0.smithi085.stdout:2299: expect (ObjNum 792 snap 256 seq_num 792) 2021-02-10T04:36:53.138 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:36:52 smithi085 conmon[52836]: debug 2021-02-10T04:36:52.692+0000 7fba71fb2700 -1 osd.3 pg_epoch: 595 pg[4.10( v 538'16 (0'0,538'16] local-lis/les=71/72 n=10 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,5,6] r=0 lpr=71 crt=538'16 lcod 538'15 mlcod 538'15 active+clean] scrubber pg(4.10) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6 (0x55fec03b5340) 2021-02-10T04:36:53.303 INFO:tasks.rados.rados.0.smithi085.stdout:2298: finishing write tid 3 to smithi08575397-3 2021-02-10T04:36:53.303 INFO:tasks.rados.rados.0.smithi085.stdout:2298: finishing write tid 4 to smithi08575397-3 2021-02-10T04:36:53.304 INFO:tasks.rados.rados.0.smithi085.stdout:2298: finishing write tid 5 to smithi08575397-3 2021-02-10T04:36:53.304 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 1128 (ObjNum 796 snap 256 seq_num 796) dirty exists 2021-02-10T04:36:53.304 INFO:tasks.rados.rados.0.smithi085.stdout:2298: left oid 3 (ObjNum 796 snap 256 seq_num 796) 2021-02-10T04:36:53.304 INFO:tasks.rados.rados.0.smithi085.stdout:2300: finishing write tid 1 to smithi08575397-43 2021-02-10T04:36:53.304 INFO:tasks.rados.rados.0.smithi085.stdout:2300: finishing write tid 2 to smithi08575397-43 2021-02-10T04:36:53.304 INFO:tasks.rados.rados.0.smithi085.stdout:2300: finishing write tid 3 to smithi08575397-43 2021-02-10T04:36:53.304 INFO:tasks.rados.rados.0.smithi085.stdout:2300: finishing write tid 4 to smithi08575397-43 2021-02-10T04:36:53.304 INFO:tasks.rados.rados.0.smithi085.stdout:2301: seq_num 798 ranges {611986=549414,1564160=424282,2764607=440478} 2021-02-10T04:36:53.310 INFO:tasks.rados.rados.0.smithi085.stdout:2301: writing smithi08575397-31 from 611986 to 1161400 tid 1 2021-02-10T04:36:53.315 INFO:tasks.rados.rados.0.smithi085.stdout:2301: writing smithi08575397-31 from 1564160 to 1988442 tid 2 2021-02-10T04:36:53.323 INFO:tasks.rados.rados.0.smithi085.stdout:2301: writing smithi08575397-31 from 2764607 to 3205085 tid 3 2021-02-10T04:36:53.323 INFO:tasks.rados.rados.0.smithi085.stdout:2294: done (14 left) 2021-02-10T04:36:53.324 INFO:tasks.rados.rados.0.smithi085.stdout:2297: done (13 left) 2021-02-10T04:36:53.324 INFO:tasks.rados.rados.0.smithi085.stdout:2298: done (12 left) 2021-02-10T04:36:53.324 INFO:tasks.rados.rados.0.smithi085.stdout:2299: done (11 left) 2021-02-10T04:36:53.324 INFO:tasks.rados.rados.0.smithi085.stdout:2302: rollback oid 2 current snap is 256 2021-02-10T04:36:53.324 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 2 to 239 2021-02-10T04:36:53.324 INFO:tasks.rados.rados.0.smithi085.stdout:2300: finishing write tid 5 to smithi08575397-43 2021-02-10T04:36:53.324 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 1132 (ObjNum 797 snap 256 seq_num 797) dirty exists 2021-02-10T04:36:53.324 INFO:tasks.rados.rados.0.smithi085.stdout:2300: left oid 43 (ObjNum 797 snap 256 seq_num 797) 2021-02-10T04:36:53.324 INFO:tasks.rados.rados.0.smithi085.stdout:2301: finishing write tid 1 to smithi08575397-31 2021-02-10T04:36:53.324 INFO:tasks.rados.rados.0.smithi085.stdout:2300: done (11 left) 2021-02-10T04:36:53.324 INFO:tasks.rados.rados.0.smithi085.stdout:2303: write (excl) oid 38 current snap is 256 2021-02-10T04:36:53.324 INFO:tasks.rados.rados.0.smithi085.stdout:2303: seq_num 799 ranges {628793=741016,2138995=567940,3192030=236763} 2021-02-10T04:36:53.332 INFO:tasks.rados.rados.0.smithi085.stdout:2303: writing smithi08575397-38 from 628793 to 1369809 tid 1 2021-02-10T04:36:53.340 INFO:tasks.rados.rados.0.smithi085.stdout:2303: writing smithi08575397-38 from 2138995 to 2706935 tid 2 2021-02-10T04:36:53.344 INFO:tasks.rados.rados.0.smithi085.stdout:2303: writing smithi08575397-38 from 3192030 to 3428793 tid 3 2021-02-10T04:36:53.345 INFO:tasks.rados.rados.0.smithi085.stdout:2304: write (excl) oid 12 current snap is 256 2021-02-10T04:36:53.345 INFO:tasks.rados.rados.0.smithi085.stdout:2304: seq_num 800 ranges {645600=416265,1797477=711598,3219453=226147} 2021-02-10T04:36:53.351 INFO:tasks.rados.rados.0.smithi085.stdout:2304: writing smithi08575397-12 from 645600 to 1061865 tid 1 2021-02-10T04:36:53.359 INFO:tasks.rados.rados.0.smithi085.stdout:2304: writing smithi08575397-12 from 1797477 to 2509075 tid 2 2021-02-10T04:36:53.365 INFO:tasks.rados.rados.0.smithi085.stdout:2304: writing smithi08575397-12 from 3219453 to 3445600 tid 3 2021-02-10T04:36:53.365 INFO:tasks.rados.rados.0.smithi085.stdout:2305: delete oid 44 current snap is 256 2021-02-10T04:36:53.376 INFO:tasks.rados.rados.0.smithi085.stdout:2304: finishing write tid 1 to smithi08575397-12 2021-02-10T04:36:53.376 INFO:tasks.rados.rados.0.smithi085.stdout:2301: finishing write tid 2 to smithi08575397-31 2021-02-10T04:36:53.377 INFO:tasks.rados.rados.0.smithi085.stdout:2303: finishing write tid 1 to smithi08575397-38 2021-02-10T04:36:53.377 INFO:tasks.rados.rados.0.smithi085.stdout:2301: finishing write tid 3 to smithi08575397-31 2021-02-10T04:36:53.377 INFO:tasks.rados.rados.0.smithi085.stdout:2302: finishing rollback tid 0 to smithi08575397-2 2021-02-10T04:36:53.377 INFO:tasks.rados.rados.0.smithi085.stdout:2302: finishing rollback tid 1 to smithi08575397-2 2021-02-10T04:36:53.378 INFO:tasks.rados.rados.0.smithi085.stdout:2302: finishing rollback tid 2 to smithi08575397-2 2021-02-10T04:36:53.378 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 847 (ObjNum 735 snap 234 seq_num 735) dirty exists 2021-02-10T04:36:53.378 INFO:tasks.rados.rados.0.smithi085.stdout:2301: finishing write tid 4 to smithi08575397-31 2021-02-10T04:36:53.378 INFO:tasks.rados.rados.0.smithi085.stdout:2301: finishing write tid 5 to smithi08575397-31 2021-02-10T04:36:53.378 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 800 (ObjNum 798 snap 256 seq_num 798) dirty exists 2021-02-10T04:36:53.378 INFO:tasks.rados.rados.0.smithi085.stdout:2301: left oid 31 (ObjNum 798 snap 256 seq_num 798) 2021-02-10T04:36:53.379 INFO:tasks.rados.rados.0.smithi085.stdout:2301: done (13 left) 2021-02-10T04:36:53.379 INFO:tasks.rados.rados.0.smithi085.stdout:2302: done (12 left) 2021-02-10T04:36:53.379 INFO:tasks.rados.rados.0.smithi085.stdout:2305: done (11 left) 2021-02-10T04:36:53.379 INFO:tasks.rados.rados.0.smithi085.stdout:2306: read oid 48 snap -1 2021-02-10T04:36:53.379 INFO:tasks.rados.rados.0.smithi085.stdout:2306: expect deleted 2021-02-10T04:36:53.379 INFO:tasks.rados.rados.0.smithi085.stdout:2307: delete oid 5 current snap is 256 2021-02-10T04:36:53.382 INFO:tasks.rados.rados.0.smithi085.stdout:2307: done (12 left) 2021-02-10T04:36:53.382 INFO:tasks.rados.rados.0.smithi085.stdout:2308: rollback oid 42 current snap is 256 2021-02-10T04:36:53.382 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 42 to 229 2021-02-10T04:36:53.382 INFO:tasks.rados.rados.0.smithi085.stdout:2303: finishing write tid 2 to smithi08575397-38 2021-02-10T04:36:53.382 INFO:tasks.rados.rados.0.smithi085.stdout:2309: read oid 5 snap -1 2021-02-10T04:36:53.382 INFO:tasks.rados.rados.0.smithi085.stdout:2309: expect deleted 2021-02-10T04:36:53.382 INFO:tasks.rados.rados.0.smithi085.stdout:2310: snap_remove snap 247 2021-02-10T04:36:53.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:53 smithi164 conmon[33677]: debug 2021-02-10T04:36:53.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:53 smithi164 conmon[33677]: 2021-02-10T04:36:53.120+0000 7fbec19a1700 -1 osd.4 595 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:53.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:53 smithi164 conmon[33677]: 2021-02-10T04:36:53.421 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[49106]: debug 2021-02-10T04:36:53.136+0000 7f77a3e94700 -1 osd.2 595 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:53.425 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: cluster 2021-02-10T04:36:52.089558+0000 osd.2 (osd. 2021-02-10T04:36:53.426 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: 2) 1691 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:53.426 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: cluster 2021-02-10T04:36:52.089589+0000 osd.2 (osd.2) 1692 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:53.426 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:53.426 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: -02-10T04:36:52.089596+0000 2021-02-10T04:36:53.426 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: osd.2 (osd.2) 1693 : 2021-02-10T04:36:53.426 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:53.426 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:53.426 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: -02-10T04:36:52.089603+0000 2021-02-10T04:36:53.426 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: osd.2 (osd.2) 1694 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:53.426 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: cluster 2021-02-10T04:36:52.089614 2021-02-10T04:36:53.426 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1695 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:53.426 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:53.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: -10T04:36:52.089621+0000 osd.2 (osd 2021-02-10T04:36:53.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: .2) 1696 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:53.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: cluster 2021-02-10T04:36:52 2021-02-10T04:36:53.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: .089627+0000 osd.2 (osd.2) 1697 : cluster [WRN] 2021-02-10T04:36:53.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:53.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:53.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: 02-10T04:36:52.089633+0000 osd.2 (osd.2) 1698 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:53.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:36:53.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: T04:36:52.089645+0000 osd.2 (osd.2) 1699 : cluster [WRN] 2021-02-10T04:36:53.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:53.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:36:53.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: T04:36:52.089650+0000 osd.2 (osd.2) 1700 : cluster [WRN] 2021-02-10T04:36:53.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:53.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:53.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: -02-10T04:36:52.415207+0000 mon.a (mon.0) 1446 : cluster [WRN] Health check failed: 14 slow ops, oldest one blocked for 442 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:36:53.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: cluster 2021-02-10T04:36 2021-02-10T04:36:53.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: :52.426229+0000 mon.a (mon.0) 1447 : cluster [DBG] 2021-02-10T04:36:53.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[29319]: osdmap e595: 8 total, 8 up, 8 in 2021-02-10T04:36:53.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: cluster 2021-02-10T04:36: 2021-02-10T04:36:53.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: 52.089558+0000 osd.2 (osd.2) 1691 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:53.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: cluster 2021-02-10T04:36 2021-02-10T04:36:53.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: :52.089589+0000 osd.2 (osd.2) 1692 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:53.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:36:53.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: :36:52.089596+0000 osd.2 (osd.2) 1693 : cluster 2021-02-10T04:36:53.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:53.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: cluster 2021-02-10T04:36:52.089603+0000 2021-02-10T04:36:53.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: osd.2 (osd.2) 1694 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:53.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:36:53.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: 36:52.089614+0000 osd.2 (osd.2) 1695 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:53.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: 2021-02-10T04:36:53.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:36:53.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: 10T04:36:52.089621+0000 osd.2 (osd.2) 1696 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:53.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:53.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: -02-10T04:36:52.089627+0000 osd.2 (osd.2) 1697 : cluster [WRN] 2021-02-10T04:36:53.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:53.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: cluster 2021-02-10T04:36:53.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: 2021-02-10T04:36:52.089633+0000 osd.2 (osd.2) 1698 : cluster [WRN] 2021-02-10T04:36:53.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:53.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: cluster 2021-02-10T04:36:53.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: 2021-02-10T04:36:52.089645+0000 osd.2 2021-02-10T04:36:53.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: (osd.2) 1699 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:53.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:36:53.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: 10T04:36:52.089650+0000 osd.2 (osd.2) 1700 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:53.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:36:53.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: 04:36:52.415207+0000 mon.a (mon 2021-02-10T04:36:53.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: .0) 1446 : cluster [WRN] Health check failed: 14 slow ops, oldest one blocked for 442 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:36:53.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:53.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: T04:36:52.426229+0000 mon.a (mon.0) 1447 2021-02-10T04:36:53.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:53 smithi085 conmon[35961]: : cluster [DBG] osdmap e595: 8 total, 8 up, 8 in 2021-02-10T04:36:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:53 smithi164 conmon[29288]: cluster 2021-02-10T04:36:52.089558+0000 osd.2 (osd.2) 1691 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:53 smithi164 conmon[29288]: cluster 2021-02-10T04:36:52.089589+0000 osd.2 (osd.2) 1692 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:53 smithi164 conmon[29288]: cluster 2021-02-10T04:36:52.089596+0000 osd.2 (osd.2) 1693 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:53 smithi164 conmon[29288]: cluster 2021-02-10T04:36:52.089603+0000 osd.2 (osd.2) 1694 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:53 smithi164 conmon[29288]: cluster 2021-02-10T04:36:52.089614+0000 osd.2 (osd.2) 1695 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:53 smithi164 conmon[29288]: cluster 2021-02-10T04:36:52.089621+0000 osd.2 (osd.2) 1696 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:53 smithi164 conmon[29288]: cluster 2021-02-10T04:36:52.089627+0000 osd.2 (osd.2) 1697 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:53 smithi164 conmon[29288]: cluster 2021-02-10T04:36:52.089633+0000 osd.2 (osd.2) 1698 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:53 smithi164 conmon[29288]: cluster 2021-02-10T04:36:52.089645+0000 osd.2 (osd.2) 1699 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:53 smithi164 conmon[29288]: cluster 2021-02-10T04:36:52.089650+0000 osd.2 (osd.2) 1700 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:53 smithi164 conmon[29288]: cluster 2021-02-10T04:36:52.415207+0000 mon.a (mon.0) 1446 : cluster [WRN] Health check failed: 14 slow ops, oldest one blocked for 442 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:36:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:53 smithi164 conmon[29288]: cluster 2021-02-10T04:36:52.426229+0000 mon.a (mon.0) 1447 : cluster [DBG] osdmap e595: 8 total, 8 up, 8 in 2021-02-10T04:36:54.196 INFO:tasks.rados.rados.0.smithi085.stdout:2310: done (14 left) 2021-02-10T04:36:54.196 INFO:tasks.rados.rados.0.smithi085.stdout:2311: copy_from oid 37 from oid 50 current snap is 256 2021-02-10T04:36:54.196 INFO:tasks.rados.rados.0.smithi085.stdout:2312: snap_create 2021-02-10T04:36:54.196 INFO:tasks.rados.rados.0.smithi085.stdout:2304: finishing write tid 2 to smithi08575397-12 2021-02-10T04:36:54.196 INFO:tasks.rados.rados.0.smithi085.stdout:2303: finishing write tid 3 to smithi08575397-38 2021-02-10T04:36:54.196 INFO:tasks.rados.rados.0.smithi085.stdout:2303: finishing write tid 4 to smithi08575397-38 2021-02-10T04:36:54.196 INFO:tasks.rados.rados.0.smithi085.stdout:2303: finishing write tid 5 to smithi08575397-38 2021-02-10T04:36:54.197 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 1073 (ObjNum 799 snap 256 seq_num 799) dirty exists 2021-02-10T04:36:54.197 INFO:tasks.rados.rados.0.smithi085.stdout:2303: left oid 38 (ObjNum 799 snap 256 seq_num 799) 2021-02-10T04:36:54.197 INFO:tasks.rados.rados.0.smithi085.stdout:2304: finishing write tid 3 to smithi08575397-12 2021-02-10T04:36:54.197 INFO:tasks.rados.rados.0.smithi085.stdout:2304: finishing write tid 4 to smithi08575397-12 2021-02-10T04:36:54.197 INFO:tasks.rados.rados.0.smithi085.stdout:2304: finishing write tid 5 to smithi08575397-12 2021-02-10T04:36:54.197 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 851 (ObjNum 800 snap 256 seq_num 800) dirty exists 2021-02-10T04:36:54.197 INFO:tasks.rados.rados.0.smithi085.stdout:2304: left oid 12 (ObjNum 800 snap 256 seq_num 800) 2021-02-10T04:36:54.197 INFO:tasks.rados.rados.0.smithi085.stdout:2308: finishing rollback tid 0 to smithi08575397-42 2021-02-10T04:36:54.197 INFO:tasks.rados.rados.0.smithi085.stdout:2308: finishing rollback tid 1 to smithi08575397-42 2021-02-10T04:36:54.197 INFO:tasks.rados.rados.0.smithi085.stdout:2308: finishing rollback tid 2 to smithi08575397-42 2021-02-10T04:36:54.197 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 854 (ObjNum 646 snap 196 seq_num 646) dirty exists 2021-02-10T04:36:54.198 INFO:tasks.rados.rados.0.smithi085.stdout:2311: finishing copy_from to smithi08575397-37 2021-02-10T04:36:54.198 INFO:tasks.rados.rados.0.smithi085.stdout:2311: got expected ENOENT (src dne) 2021-02-10T04:36:54.198 INFO:tasks.rados.rados.0.smithi085.stdout:2311: finishing copy_from racing read to smithi08575397-37 2021-02-10T04:36:54.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[33677]: debug 2021-02-10T04:36:54.091+0000 7fbec19a1700 -1 osd.4 595 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:54.428 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[49106]: debug 2021-02-10T04:36:54.428 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[49106]: 2021-02-10T04:36:54.095+0000 7f77a3e94700 -1 osd.2 595 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:54.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: cluster 2021-02-10T04:36:53.122501+0000 osd.4 2021-02-10T04:36:54.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: (osd.4) 881 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:54.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: cluster 2021-02-10T04:36:53.122510+0000 osd.4 (osd.4) 882 : cluster [WRN] 2021-02-10T04:36:54.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:54.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: cluster 2021-02-10T04:36: 2021-02-10T04:36:54.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: 53.122516+0000 osd.4 (osd.4) 883 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:54.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: 2021-02-10T04:36:54.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:36:54.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: 36:53.122521+0000 osd.4 (osd.4) 2021-02-10T04:36:54.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: 884 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:54.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: cluster 2021-02-10T04:36:54.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: 2021-02-10T04:36:53. 2021-02-10T04:36:54.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: 137552+0000 osd.2 (osd.2) 1701 : cluster 2021-02-10T04:36:54.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:54.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: cluster 2021-02-10T04:36:54.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: 2021-02-10T04:36:53 2021-02-10T04:36:54.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: .137564+0000 osd.2 (osd.2) 1702 : cluster 2021-02-10T04:36:54.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:54.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:36:54.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: 36:53.137574+0000 osd.2 (osd. 2021-02-10T04:36:54.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: 2) 1703 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:54.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:54.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: -02-10T04:36:53 2021-02-10T04:36:54.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: .137581+0000 osd.2 (osd.2) 1704 : cluster [WRN] 2021-02-10T04:36:54.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:54.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:36:54.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: 36:53.137586+0000 osd.2 (osd.2) 2021-02-10T04:36:54.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: 1705 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:54.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:36:54.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: 04:36:53.137593+0000 osd.2 (osd.2) 1706 2021-02-10T04:36:54.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:54.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: cluster 2021-02-10T04:36:54.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: 2021-02-10T04:36:53.137600+0000 osd.2 (osd.2) 1707 : cluster 2021-02-10T04:36:54.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:54.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:54.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: 02-10T04:36:53.137606+0000 osd.2 (osd.2) 1708 : cluster 2021-02-10T04:36:54.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:54.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: cluster 2021-02-10T04:36:54.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: 2021-02-10T04:36:53 2021-02-10T04:36:54.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: .137611+0000 osd.2 (osd.2) 2021-02-10T04:36:54.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: 1709 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:54.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:54.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: -02-10T04:36:53.137618+0000 2021-02-10T04:36:54.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: osd.2 (osd.2) 1710 : cluster [WRN] 2021-02-10T04:36:54.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:54.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: cluster 2021-02-10T04:36 2021-02-10T04:36:54.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: :54.093141+0000 osd.4 (osd.4) 885 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:54.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: 2021-02-10T04:36:54.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:54.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: -10T04:36:54.093148+0000 osd.4 (osd.4) 886 2021-02-10T04:36:54.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:54.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:36:54.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: cluster 2021-02-10T04:36:53.122501 2021-02-10T04:36:54.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 881 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:54.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:36:54.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: 04:36:53.122510+0000 osd.4 (osd.4) 882 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:54.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:54.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: -02-10T04:36:53.122516+0000 2021-02-10T04:36:54.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: osd.4 (osd.4) 883 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:54.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: 2021-02-10T04:36:54.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: cluster 2021-02-10T04:36:54.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: 2021-02-10T04:36:53.122521+0000 osd.4 (osd 2021-02-10T04:36:54.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: .4) 884 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:54.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: cluster 2021-02-10T04:36:53.137552+0000 osd.2 (osd.2) 1701 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:54.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: cluster 2021-02-10T04:36 2021-02-10T04:36:54.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: :53.137564+0000 osd.2 (osd.2) 1702 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:54.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: cluster 2021-02-10T04:36:53.137574 2021-02-10T04:36:54.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1703 : cluster [WRN] 2021-02-10T04:36:54.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:54.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:54.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: -10T04:36:53.137581+0000 osd.2 2021-02-10T04:36:54.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: (osd.2) 1704 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:54.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: T04:36:54.093152+0000 osd.4 (osd.4) 887 : cluster 2021-02-10T04:36:54.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:54.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: cluster 2021-02-10T04:36:54.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: 2021-02-10T04:36:54.093156+0000 osd.4 (osd.4) 2021-02-10T04:36:54.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: 888 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:54.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:54.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: 02-10T04:36 2021-02-10T04:36:54.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: :54.126895+0000 mgr.y (mgr. 2021-02-10T04:36:54.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: 14140) 649 : cluster [DBG] pgmap v1043: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 439 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 1.7 MiB/s wr, 21 op/s 2021-02-10T04:36:54.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: cluster 2021-02-10T04:36:54.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: 2021-02-10T04:36:54.195859+0000 2021-02-10T04:36:54.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[29319]: mon.a (mon.0) 1448 : cluster [DBG] osdmap e596: 8 total, 8 up, 8 in 2021-02-10T04:36:54.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: cluster 2021-02-10T04:36:54.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:36:54.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: 36:53.137586+0000 osd.2 (osd.2) 1705 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:54.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: 2021-02-10T04:36:54.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: cluster 2021-02-10T04:36:54.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: 2021-02-10T04:36:53.137593+0000 osd.2 (osd.2) 1706 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:54.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: cluster 2021-02-10T04:36:54.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: 2021-02-10T04:36:54.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: 2021 2021-02-10T04:36:54.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: - 2021-02-10T04:36:54.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: 02 2021-02-10T04:36:54.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: - 2021-02-10T04:36:54.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: 10 2021-02-10T04:36:54.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: T 2021-02-10T04:36:54.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: 04 2021-02-10T04:36:54.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: : 2021-02-10T04:36:54.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: 36 2021-02-10T04:36:54.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: : 2021-02-10T04:36:54.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: 53 2021-02-10T04:36:54.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: .137600+0000 osd.2 ( 2021-02-10T04:36:54.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: osd.2) 1707 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:54.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:54.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: 02-10T04:36:53.137606+0000 osd.2 (osd 2021-02-10T04:36:54.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: .2) 1708 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:54.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:54.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: 02-10T04:36:53.137611+0000 2021-02-10T04:36:54.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: osd.2 (osd.2) 1709 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:54.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: 2021-02-10T04:36:54.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: cluster 2021-02-10T04:36:54.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: 2021-02-10T04:36:53.137618+0000 osd.2 ( 2021-02-10T04:36:54.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: osd.2) 1710 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:54.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:54.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: -10T04:36:54.093141+0000 2021-02-10T04:36:54.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: osd.4 (osd.4) 885 : cluster [WRN] 2021-02-10T04:36:54.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:54.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: cluster 2021-02-10T04:36:54. 2021-02-10T04:36:54.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: 093148+0000 osd.4 (osd.4) 886 : cluster 2021-02-10T04:36:54.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:54.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: cluster 2021-02-10T04:36: 2021-02-10T04:36:54.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: 54.093152+0000 osd.4 (osd.4) 887 2021-02-10T04:36:54.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:54.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: 2021-02-10T04:36:54.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:54.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: -02-10T04:36:54.093156+0000 osd.4 (osd 2021-02-10T04:36:54.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: .4) 888 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:54.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:54.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: -10T04:36:54.126895+0000 mgr.y (mgr.14140) 649 2021-02-10T04:36:54.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: : cluster [DBG] pgmap v1043: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 439 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 1.7 MiB/s wr, 21 op/s 2021-02-10T04:36:54.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: cluster 2021-02-10T04:36:54.195859+0000 mon.a (mon.0) 1448 : cluster [DBG] 2021-02-10T04:36:54.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:54 smithi085 conmon[35961]: osdmap e596: 8 total, 8 up, 8 in 2021-02-10T04:36:54.738 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:36:54.739 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2021-02-10T04:36:54.739 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 0 2021-02-10T04:36:54.739 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:36:54.769 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: cluster 2021-02-10T04:36: 2021-02-10T04:36:54.769 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: 53.122501+0000 osd.4 (osd.4) 881 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:54.769 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:36:54.769 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: 04:36:53.122510+0000 osd.4 (osd.4) 882 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:54.769 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:36:54.769 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: :36:53.122516+0000 osd.4 (osd.4) 2021-02-10T04:36:54.770 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: 883 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:54.770 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:36:54.770 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: 36:53.122521+0000 osd.4 (osd.4) 884 : 2021-02-10T04:36:54.770 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:54.770 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:36:54.770 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: 36:53.137552+0000 osd.2 (osd.2) 1701 : 2021-02-10T04:36:54.770 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:54.770 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:36:54.770 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: 36:53.137564+0000 osd.2 (osd 2021-02-10T04:36:54.770 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: .2) 1702 : cluster [WRN] 2021-02-10T04:36:54.770 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:54.770 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:54.770 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: -02-10T04:36: 2021-02-10T04:36:54.771 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: 53.137574+0000 osd.2 (osd. 2021-02-10T04:36:54.771 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: 2) 1703 : cluster [WRN] 2021-02-10T04:36:54.771 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:54.771 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:36:54.771 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: 04:36:53. 2021-02-10T04:36:54.771 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: 137581+0000 osd.2 (osd.2) 1704 : cluster 2021-02-10T04:36:54.771 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:54.771 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:36:54.771 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: :36:53.137586+0000 osd.2 (osd.2 2021-02-10T04:36:54.771 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: ) 1705 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:54.771 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: cluster 2021-02-10T04:36:54.771 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: 2021-02-10T04:36:54.771 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: 2021 2021-02-10T04:36:54.772 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: -02-10T04:36:53. 2021-02-10T04:36:54.772 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: 137593+0000 osd.2 (osd.2) 1706 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:54.772 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:54.772 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: -10T04:36:53.137600+0000 osd.2 (osd.2 2021-02-10T04:36:54.772 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: ) 1707 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:54.772 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:36:54.772 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: 36:53.137606+0000 osd.2 (osd.2) 1708 2021-02-10T04:36:54.772 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:54.772 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: cluster 2021-02-10T04:36: 2021-02-10T04:36:54.772 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: 53.137611+0000 osd.2 (osd.2) 1709 : cluster 2021-02-10T04:36:54.772 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:54.772 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:54.772 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: 02-10T04:36:53.137618+0000 2021-02-10T04:36:54.773 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: osd.2 (osd.2) 1710 : cluster 2021-02-10T04:36:54.773 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:54.773 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:36:54.773 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: 04:36:54.093141+0000 osd.4 (osd.4 2021-02-10T04:36:54.773 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: ) 885 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:54.773 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:54.773 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: - 2021-02-10T04:36:54.773 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: 02-10T04:36: 2021-02-10T04:36:54.773 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: 54.093148+0000 osd.4 (osd.4) 886 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:54.773 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: cluster 2021-02-10T04:36:54.093152+0000 osd.4 (osd.4) 887 : cluster 2021-02-10T04:36:54.773 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:54.773 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: cluster 2021-02-10T04:36:54.093156 2021-02-10T04:36:54.773 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 888 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:54.774 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:36:54.774 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: :36:54.126895+0000 mgr.y (mgr.14140) 649 : 2021-02-10T04:36:54.774 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: cluster [DBG] pgmap v1043: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 439 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 1.7 MiB/s wr, 21 op/s 2021-02-10T04:36:54.774 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: cluster 2021-02-10T04:36:54.195859+0000 mon.a (mon 2021-02-10T04:36:54.774 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:54 smithi164 conmon[29288]: .0) 1448 : cluster [DBG] osdmap e596: 8 total, 8 up, 8 in 2021-02-10T04:36:55.078 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[33677]: debug 2021-02-10T04:36:55.079 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[33677]: 2021-02-10T04:36:55.075+0000 7fbec19a1700 -1 osd.4 595 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:55.201 INFO:tasks.rados.rados.0.smithi085.stdout:2303: done (15 left) 2021-02-10T04:36:55.201 INFO:tasks.rados.rados.0.smithi085.stdout:2304: done (14 left) 2021-02-10T04:36:55.201 INFO:tasks.rados.rados.0.smithi085.stdout:2306: done (13 left) 2021-02-10T04:36:55.201 INFO:tasks.rados.rados.0.smithi085.stdout:2308: done (12 left) 2021-02-10T04:36:55.201 INFO:tasks.rados.rados.0.smithi085.stdout:2309: done (11 left) 2021-02-10T04:36:55.201 INFO:tasks.rados.rados.0.smithi085.stdout:2311: done (10 left) 2021-02-10T04:36:55.201 INFO:tasks.rados.rados.0.smithi085.stdout:2312: done (9 left) 2021-02-10T04:36:55.201 INFO:tasks.rados.rados.0.smithi085.stdout:2313: read oid 3 snap 229 2021-02-10T04:36:55.201 INFO:tasks.rados.rados.0.smithi085.stdout:2313: expect (ObjNum 674 snap 208 seq_num 674) 2021-02-10T04:36:55.201 INFO:tasks.rados.rados.0.smithi085.stdout:2314: snap_create 2021-02-10T04:36:55.208 INFO:tasks.rados.rados.0.smithi085.stdout:2313: expect (ObjNum 674 snap 208 seq_num 674) 2021-02-10T04:36:55.432 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[49106]: debug 2021-02-10T04:36:55.143+0000 7f77a3e94700 -1 osd.2 596 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:55.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: cluster 2021-02-10T04:36 2021-02-10T04:36:55.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: :54.096557+0000 osd.2 (osd.2) 1711 2021-02-10T04:36:55.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:55.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:36:55.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: 36:54.096580+0000 osd.2 (osd.2) 2021-02-10T04:36:55.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: 1712 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:55.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:36:55.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: 04:36:54.096589+0000 osd.2 (osd. 2021-02-10T04:36:55.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: 2) 1713 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:55.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:55.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: -10T04:36:54. 2021-02-10T04:36:55.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: 096595+0000 osd.2 (osd.2) 1714 : cluster 2021-02-10T04:36:55.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:55.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:55.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: 02-10T04:36: 2021-02-10T04:36:55.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: 54.096600+0000 osd.2 (osd.2) 1715 : 2021-02-10T04:36:55.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:55.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:36:55.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: :36:54.096607+0000 osd.2 (osd. 2021-02-10T04:36:55.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: 2) 1716 : cluster 2021-02-10T04:36:55.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:55.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: cluster 2021-02-10T04:36 2021-02-10T04:36:55.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: :54.096612+0000 osd.2 (osd.2) 2021-02-10T04:36:55.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: 1717 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:55.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:55.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: T04:36:54.096619+0000 osd.2 (osd 2021-02-10T04:36:55.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: .2) 1718 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:55.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:55.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: -10T04:36:54.096630+0000 osd.2 2021-02-10T04:36:55.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: (osd.2) 1719 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:55.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:55.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: -02-10T04:36:54. 2021-02-10T04:36:55.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: 096637+0000 osd.2 2021-02-10T04:36:55.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: (osd.2) 1720 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:55.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:36:55.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: -10T04:36:55.077297+0000 osd.4 ( 2021-02-10T04:36:55.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: osd.4) 889 : cluster [WRN] 2021-02-10T04:36:55.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:55.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: cluster 2021-02-10T04:36:55 2021-02-10T04:36:55.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: .077306+0000 osd.4 (osd.4) 890 : 2021-02-10T04:36:55.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:55.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:36:55.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: 36:55.077312+0000 osd.4 (osd.4 2021-02-10T04:36:55.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: ) 891 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:55.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:55.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: T04:36:55.077322+0000 osd.4 (osd.4) 2021-02-10T04:36:55.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: 892 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:55.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:36:55.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: 10T04:36:55.199957+0000 mon.a 2021-02-10T04:36:55.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: (mon.0) 1449 : cluster [DBG] osdmap e597: 8 total, 8 up, 8 in 2021-02-10T04:36:55.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[35961]: 2021-02-10T04:36:55.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: cluster 2021-02-10T04:36:54.096557+0000 osd.2 (osd.2) 1711 2021-02-10T04:36:55.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:55.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: cluster 2021-02-10T04:36:54.096580+0000 osd.2 (osd.2) 1712 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:55.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:36:55.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: T04:36:54.096589+0000 osd.2 (osd.2) 1713 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:55.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: cluster 2021-02-10T04:36:55.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: 2021-02-10T04:36:54.096595+0000 osd.2 (osd.2) 1714 : cluster 2021-02-10T04:36:55.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:55.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:36:55.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: T04:36:54.096600+0000 osd.2 (osd.2) 1715 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:55.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:36:55.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: 36:54.096607+0000 osd.2 (osd.2) 2021-02-10T04:36:55.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: 1716 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:55.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: cluster 2021-02-10T04:36:54 2021-02-10T04:36:55.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: .096612+0000 osd.2 (osd.2) 1717 : cluster 2021-02-10T04:36:55.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:55.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: cluster 2021-02-10T04:36: 2021-02-10T04:36:55.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: 54.096619+0000 osd.2 (osd.2) 1718 2021-02-10T04:36:55.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:55.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: cluster 2021-02-10T04:36:54.096630+0000 2021-02-10T04:36:55.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: osd.2 (osd.2) 1719 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:55.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: 2021-02-10T04:36:55.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:55.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: -10T04 2021-02-10T04:36:55.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: :36:54.096637 2021-02-10T04:36:55.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1720 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:55.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: cluster 2021-02-10T04:36:55.077297 2021-02-10T04:36:55.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 889 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:55.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:55.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: 02-10T04:36:55.077306+0000 osd.4 (osd 2021-02-10T04:36:55.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: .4) 890 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:55.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:36:55.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: 04:36:55.077312+0000 osd.4 (osd. 2021-02-10T04:36:55.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: 4) 891 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:55.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: cluster 2021-02-10T04:36: 2021-02-10T04:36:55.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: 55.077322+0000 osd.4 (osd 2021-02-10T04:36:55.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: .4) 892 : cluster [WRN] 2021-02-10T04:36:55.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:55.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:55.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: -02-10T04:36 2021-02-10T04:36:55.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: :55.199957+0000 mon.a (mon 2021-02-10T04:36:55.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: .0) 1449 : cluster [DBG] 2021-02-10T04:36:55.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:55 smithi085 conmon[29319]: osdmap e597: 8 total, 8 up, 8 in 2021-02-10T04:36:55.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: cluster 2021-02-10T04:36:55.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: 2021-02-10T04:36:54.096557 2021-02-10T04:36:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1711 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: cluster 2021-02-10T04:36: 2021-02-10T04:36:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: 54.096580+0000 osd.2 (osd.2) 2021-02-10T04:36:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: 1712 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:36:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: T04:36:54.096589+0000 osd.2 ( 2021-02-10T04:36:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: osd.2) 1713 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: cluster 2021-02-10T04:36:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: 2021-02-10T04:36:54.096595+0000 2021-02-10T04:36:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: osd.2 (osd.2) 1714 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: cluster 2021-02-10T04:36:54. 2021-02-10T04:36:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: 096600+0000 osd.2 (osd.2) 1715 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:36:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: 36:54.096607+0000 osd.2 (osd.2) 1716 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: 2021-02-10T04:36:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: -10T04:36:54.096612+0000 osd.2 (osd.2) 1717 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:36:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: :36:54.096619+0000 osd.2 (osd.2) 1718 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: cluster 2021-02-10T04:36:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: 2021-02-10T04:36:54.096630+0000 osd.2 (osd.2) 1719 : cluster [WRN] 2021-02-10T04:36:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: -10T04:36:54.096637+0000 osd.2 (osd.2) 1720 : cluster 2021-02-10T04:36:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: cluster 2021-02-10T04:36:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: 2021-02-10T04:36:55.077297+0000 osd.4 (osd.4) 889 : cluster 2021-02-10T04:36:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: -10T04:36:55.077306+0000 osd.4 (osd.4) 890 : cluster [WRN] 2021-02-10T04:36:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: 02-10T04:36:55.077312+0000 osd.4 (osd.4) 891 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:36:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: 04:36:55.077322+0000 osd.4 (osd.4) 892 : cluster [WRN] 2021-02-10T04:36:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: -02-10T04:36:55.199957+0000 mon.a (mon.0) 1449 : cluster [DBG] osdmap e597: 8 total, 8 up, 8 in 2021-02-10T04:36:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:55 smithi164 conmon[29288]: 2021-02-10T04:36:56.199 INFO:tasks.rados.rados.0.smithi085.stdout:2313: done (10 left) 2021-02-10T04:36:56.200 INFO:tasks.rados.rados.0.smithi085.stdout:2314: done (9 left) 2021-02-10T04:36:56.200 INFO:tasks.rados.rados.0.smithi085.stdout:2315: snap_create 2021-02-10T04:36:56.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[33677]: debug 2021-02-10T04:36:56.031+0000 7fbec19a1700 -1 osd.4 597 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:56.439 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[49106]: debug 2021-02-10T04:36:56.152+0000 7f77a3e94700 -1 osd.2 597 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:56.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: cluster 2021-02-10T04:36:55. 2021-02-10T04:36:56.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: 144764+0000 osd.2 (osd.2) 1721 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:56.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: cluster 2021-02-10T04:36:55.144773+0000 osd.2 ( 2021-02-10T04:36:56.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: osd.2) 1722 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:56.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:56.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: -02-10T04:36:55.144789+0000 osd.2 (osd.2) 1723 : cluster 2021-02-10T04:36:56.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:56.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: cluster 2021-02-10T04:36:55. 2021-02-10T04:36:56.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: 144796+0000 osd.2 (osd.2) 1724 : cluster [WRN] 2021-02-10T04:36:56.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:56.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: cluster 2021-02-10T04:36:56.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: 2021-02-10T04:36:55.144801+0000 osd.2 (osd 2021-02-10T04:36:56.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: .2) 1725 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:56.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:56.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: -10T04:36:55.144822+0000 osd.2 (osd.2) 2021-02-10T04:36:56.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: 1726 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:56.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:56.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: 02-10T04:36:55.144828+0000 osd.2 ( 2021-02-10T04:36:56.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: osd.2) 1727 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:56.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:56.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: -10T04:36: 2021-02-10T04:36:56.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: 55.144834+0000 osd.2 (osd.2) 1728 : cluster 2021-02-10T04:36:56.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:56.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: cluster 2021-02-10T04:36:55 2021-02-10T04:36:56.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: .144840+0000 osd.2 (osd.2) 1729 : cluster 2021-02-10T04:36:56.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:56.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: cluster 2021-02-10T04:36:55. 2021-02-10T04:36:56.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: 144845+0000 osd.2 (osd.2) 1730 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:56.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: cluster 2021-02-10T04:36:56.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:36:56.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: :36:56.032748+0000 osd.4 (osd.4) 2021-02-10T04:36:56.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: 893 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:56.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:36:56.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: 10T04:36:56.032754+0000 osd.4 (osd 2021-02-10T04:36:56.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: .4) 894 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:56.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: cluster 2021-02-10T04:36:56. 2021-02-10T04:36:56.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: 032758+0000 osd.4 (osd.4) 895 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:56.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: cluster 2021-02-10T04:36:56 2021-02-10T04:36:56.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: .032766+0000 osd.4 (osd.4) 896 : cluster [WRN] 2021-02-10T04:36:56.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:56.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: cluster 2021-02-10T04:36:56.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: 2021-02-10T04:36:56.127656+0000 mgr.y (mgr.14140) 650 : cluster [DBG] 2021-02-10T04:36:56.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: pgmap v1046: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 457 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 4.4 MiB/s wr, 34 op/s 2021-02-10T04:36:56.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: cluster 2021-02-10T04:36:56.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: 2021-02-10T04:36:56.199005+0000 mon.a (mon.0) 1450 : cluster [DBG] 2021-02-10T04:36:56.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[29319]: osdmap e598: 8 total, 8 up, 8 in 2021-02-10T04:36:56.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: cluster 2021-02-10T04:36:55.144764 2021-02-10T04:36:56.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1721 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:56.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: cluster 2021-02-10T04:36:56.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: 2021-02-10T04:36:55.144773+0000 osd.2 2021-02-10T04:36:56.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: (osd.2) 1722 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:56.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: 2021-02-10T04:36:56.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: cluster 2021-02-10T04:36:55 2021-02-10T04:36:56.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: .144789+0000 osd.2 (osd.2) 1723 : cluster 2021-02-10T04:36:56.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:56.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:36:56.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: 36:55.144796+0000 osd.2 (osd.2) 2021-02-10T04:36:56.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: 1724 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:56.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:56.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: 02-10T04:36:55.144801+0000 2021-02-10T04:36:56.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: osd.2 (osd.2) 1725 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:56.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: 2021-02-10T04:36:56.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: cluster 2021-02-10T04:36: 2021-02-10T04:36:56.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: 55.144822+0000 osd.2 (osd.2) 1726 2021-02-10T04:36:56.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:56.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:36:56.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: :36:55.144828+0000 osd.2 (osd. 2021-02-10T04:36:56.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: 2) 1727 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:56.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:36:56.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: 10T04:36:55.144834+0000 osd.2 ( 2021-02-10T04:36:56.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: osd.2) 1728 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:56.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:56.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: -02-10T04:36:55.144840+0000 2021-02-10T04:36:56.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: osd.2 (osd.2) 1729 : cluster [WRN] 2021-02-10T04:36:56.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:56.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: cluster 2021-02-10T04:36:56.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: 2021-02-10T04:36:55. 2021-02-10T04:36:56.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: 144845+0000 osd.2 (osd.2) 1730 : cluster 2021-02-10T04:36:56.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:56.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:36:56.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: :36:56.032748+0000 osd.4 (osd.4 2021-02-10T04:36:56.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: ) 893 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:56.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: cluster 2021-02-10T04:36:56.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: 2021-02-10T04:36:56. 2021-02-10T04:36:56.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: 032754+0000 osd.4 (osd.4) 894 : cluster 2021-02-10T04:36:56.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:56.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:36:56.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: 04:36:56.032758+0000 osd.4 (osd 2021-02-10T04:36:56.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: .4) 895 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:56.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:56.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: -02-10T04:36:56.032766+0000 2021-02-10T04:36:56.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: osd.4 (osd.4) 896 : cluster [WRN] 2021-02-10T04:36:56.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:56.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: cluster 2021-02-10T04:36:56.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: 2021-02 2021-02-10T04:36:56.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: -10T04:36:56.127656+0000 2021-02-10T04:36:56.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: mgr.y (mgr.14140) 650 : cluster [DBG] 2021-02-10T04:36:56.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: pgmap v1046: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 457 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 4.4 MiB/s wr, 34 op/s 2021-02-10T04:36:56.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: cluster 2021-02-10T04:36:56.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: 2021-02-10T04:36:56.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: 2021 2021-02-10T04:36:56.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: - 2021-02-10T04:36:56.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: 02 2021-02-10T04:36:56.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: - 2021-02-10T04:36:56.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: 10 2021-02-10T04:36:56.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: T04:36:56.199005+0000 2021-02-10T04:36:56.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[35961]: mon.a (mon.0) 1450 : cluster [DBG] osdmap e598: 8 total, 8 up, 8 in 2021-02-10T04:36:56.706 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:36:56.751 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:36:56.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: cluster 2021-02-10T04:36:55.144764+0000 osd.2 (osd.2) 1721 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:56.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: cluster 2021-02-10T04:36:55.144773+0000 osd.2 (osd.2) 1722 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:56.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: cluster 2021-02-10T04:36:55.144789+0000 osd.2 (osd.2) 1723 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:56.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: cluster 2021-02-10T04:36:55.144796+0000 osd.2 (osd.2) 1724 : cluster [WRN] 2021-02-10T04:36:56.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:56.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: cluster 2021-02-10T04:36:56.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: 2021-02-10T04:36:55.144801+0000 osd.2 (osd 2021-02-10T04:36:56.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: .2) 1725 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:56.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:36:56.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: 10T04:36:55.144822+0000 osd.2 (osd 2021-02-10T04:36:56.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: .2) 1726 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:56.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: cluster 2021-02-10T04:36:55.144828+0000 osd.2 ( 2021-02-10T04:36:56.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: osd.2) 1727 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:56.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: cluster 2021-02-10T04:36:56.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: 2021-02-10T04:36:55.144834+0000 osd.2 (osd.2) 1728 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:56.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: cluster 2021-02-10T04:36:56.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: 2021-02-10T04:36:55.144840+0000 osd.2 (osd.2) 1729 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:56.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: cluster 2021-02-10T04:36:55.144845+0000 2021-02-10T04:36:56.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: osd.2 (osd.2) 1730 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:56.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: cluster 2021-02-10T04:36:56.032748+0000 osd.4 ( 2021-02-10T04:36:56.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: osd.4) 893 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:56.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: cluster 2021-02-10T04:36:56.032754+0000 osd.4 (osd 2021-02-10T04:36:56.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: .4) 894 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:56.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: cluster 2021-02-10T04:36:56.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: 2021-02-10T04:36:56.032758+0000 osd.4 (osd.4) 895 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:56.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: cluster 2021-02-10T04:36:56.032766+0000 2021-02-10T04:36:56.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: osd.4 (osd.4) 896 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:56.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: cluster 2021-02-10T04:36: 2021-02-10T04:36:56.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: 56.127656+0000 mgr.y (mgr.14140) 650 : cluster 2021-02-10T04:36:56.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: [DBG] pgmap v1046: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 457 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 4.4 MiB/s wr, 34 op/s 2021-02-10T04:36:56.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: cluster 2021-02-10T04:36: 2021-02-10T04:36:56.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: 56.199005+0000 mon.a (mon.0) 1450 : 2021-02-10T04:36:56.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:56 smithi164 conmon[29288]: cluster [DBG] osdmap e598: 8 total, 8 up, 8 in 2021-02-10T04:36:57.064 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:36:56 smithi085 conmon[45443]: debug 2021-02-10T04:36:56.989+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:36:57.117 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:36:57.204 INFO:tasks.rados.rados.0.smithi085.stdout:2315: done (9 left) 2021-02-10T04:36:57.204 INFO:tasks.rados.rados.0.smithi085.stdout:2316: snap_remove snap 238 2021-02-10T04:36:57.226 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:36:57.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[33677]: debug 2021-02-10T04:36:57.025+0000 7fbec19a1700 -1 osd.4 598 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:57.440 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[49106]: debug 2021-02-10T04:36:57.116+0000 7f77a3e94700 -1 osd.2 598 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:57.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: cluster 2021-02-10T04:36:57.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[35961]: cluster 2021-02-10T04:36:56.153715+0000 osd.2 (osd.2) 1731 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:57.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[35961]: cluster 2021-02-10T04:36:56.153726+0000 osd.2 (osd.2) 1732 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:57.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[35961]: cluster 2021-02-10T04:36:56.153733+0000 osd.2 (osd.2) 1733 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:57.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[35961]: cluster 2021-02-10T04:36:56.153738+0000 osd.2 (osd.2) 1734 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:57.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[35961]: cluster 2021-02-10T04:36:56.153749+0000 osd.2 (osd.2) 1735 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:57.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[35961]: cluster 2021-02-10T04:36:56.153753+0000 osd.2 (osd.2) 1736 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:57.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[35961]: cluster 2021-02-10T04:36:56.153757+0000 osd.2 (osd.2) 1737 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:57.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[35961]: cluster 2021-02-10T04:36:56.153762+0000 osd.2 (osd.2) 1738 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:57.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[35961]: cluster 2021-02-10T04:36:56.153768+0000 osd.2 (osd.2) 1739 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:57.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[35961]: cluster 2021-02-10T04:36:56.153780+0000 osd.2 (osd.2) 1740 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:57.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[35961]: cephadm 2021-02-10T04:36:56.631714+0000 mgr.y (mgr.14140) 651 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:36:57.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[35961]: cephadm 2021-02-10T04:36:56.632120+0000 mgr.y 2021-02-10T04:36:57.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[35961]: (mgr.14140) 652 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:36:57.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[35961]: cluster 2021-02-10T04:36:57.026304+0000 osd.4 (osd.4) 897 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:57.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[35961]: cluster 2021-02-10T04:36:57.026314+0000 osd.4 (osd.4) 898 2021-02-10T04:36:57.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:57.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[35961]: cluster 2021-02-10T04:36:57.026321+0000 osd.4 ( 2021-02-10T04:36:57.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[35961]: osd.4) 899 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:57.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[35961]: cluster 2021-02-10T04:36 2021-02-10T04:36:57.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[35961]: :57.026328+0000 osd.4 (osd.4) 900 2021-02-10T04:36:57.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:57.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:36:57.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[35961]: :36:57.200307+0000 mon.a (mon.0 2021-02-10T04:36:57.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[35961]: ) 1451 : cluster [DBG] osdmap e599: 8 total, 8 up, 8 in 2021-02-10T04:36:57.485 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:36:57.731 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: 2021-02-10T04:36 2021-02-10T04:36:57.732 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: :56.153715+0000 osd.2 (osd.2) 1731 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:57.733 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: cluster 2021-02-10T04:36:57.733 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: 2021- 2021-02-10T04:36:57.733 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: 02-10T04:36:56.153726 2021-02-10T04:36:57.734 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1732 : cluster 2021-02-10T04:36:57.734 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:57.734 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: cluster 2021-02-10T04:36:57.735 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:36:57.735 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: :36:56.153733+0000 osd.2 (osd.2) 1733 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:57.735 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: cluster 2021-02-10T04:36:57.735 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:36:57.737 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: T04:36:56.153738+0000 osd.2 (osd.2) 2021-02-10T04:36:57.737 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: 1734 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:57.738 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:57.738 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: 02-10T04:36:56.153749+0000 osd.2 2021-02-10T04:36:57.738 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: (osd.2) 1735 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:57.738 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: 2021-02-10T04:36:57.739 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:36:57.739 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: 10T04:36:56.153753 2021-02-10T04:36:57.739 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1736 2021-02-10T04:36:57.739 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:57.739 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: cluster 2021-02-10T04:36:57.739 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:36:57.739 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: 04:36:56.153757+0000 osd.2 2021-02-10T04:36:57.739 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: (osd.2) 1737 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:57.739 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: cluster 2021-02-10T04:36:57.740 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: 2021-02-10T04:36: 2021-02-10T04:36:57.740 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: 56.153762+0000 osd.2 (osd.2 2021-02-10T04:36:57.740 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: ) 1738 2021-02-10T04:36:57.740 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: : cluster 2021-02-10T04:36:57.740 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:57.740 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: cluster 2021-02-10T04:36:57.740 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: 2021-02-10T04:36:56.153768+0000 2021-02-10T04:36:57.740 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: osd.2 (osd.2) 1739 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:57.740 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: 2021-02-10T04:36:57.740 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: cluster 2021-02-10T04:36:57.741 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: 2021-02-10T04:36 2021-02-10T04:36:57.741 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: :56.153780+0000 osd.2 ( 2021-02-10T04:36:57.741 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: osd.2) 1740 : cluster [WRN] 2021-02-10T04:36:57.741 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:57.741 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: cephadm 2021 2021-02-10T04:36:57.741 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: -02-10T04:36:56.631714+0000 2021-02-10T04:36:57.741 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: mgr.y (mgr.14140) 651 : cephadm [INF] 2021-02-10T04:36:57.741 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: Refreshing smithi085 facts 2021-02-10T04:36:57.741 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: cephadm 2021-02- 2021-02-10T04:36:57.742 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: 10T04:36:56.632120+0000 mgr.y (mgr.14140) 652 : cephadm 2021-02-10T04:36:57.742 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: [INF] Refreshing smithi164 facts 2021-02-10T04:36:57.742 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:36:57.742 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: 04:36:57.026304+0000 osd.4 (osd.4) 897 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:57.742 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: cluster 2021-02-10T04:36:57.742 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:36:57.742 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: 36:57.026314+0000 osd.4 2021-02-10T04:36:57.742 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: (osd.4) 898 : cluster 2021-02-10T04:36:57.742 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:57.743 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: cluster 2021-02-10T04:36:57.743 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:36:57.743 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: :36:57.026321+0000 osd.4 2021-02-10T04:36:57.743 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: (osd.4) 899 : cluster 2021-02-10T04:36:57.743 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:57.743 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:57.743 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: -02-10T04:36:57 2021-02-10T04:36:57.743 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: .026328+0000 osd.4 (osd. 2021-02-10T04:36:57.744 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: 4) 900 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:57.744 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: 2021-02-10T04:36:57.744 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:36:57.744 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: 10T04:36:57.200307+0000 mon.a (mon. 2021-02-10T04:36:57.744 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[29319]: 0) 1451 : cluster [DBG] osdmap e599: 8 total, 8 up, 8 in 2021-02-10T04:36:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: cluster 2021-02-10T04:36:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:36:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: 36:56.153715+0000 osd.2 (osd.2) 1731 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:36:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: 10T04:36:56.153726+0000 osd.2 (osd. 2021-02-10T04:36:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: 2) 1732 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:36:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: 36:56.153733+0000 osd.2 (osd.2) 1733 : 2021-02-10T04:36:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: cluster 2021-02-10T04:36:56 2021-02-10T04:36:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: .153738+0000 osd.2 (osd.2) 1734 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: 2021-02-10T04:36:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: cluster 2021-02-10T04:36: 2021-02-10T04:36:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: 56.153749+0000 osd.2 (osd.2) 1735 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: 2021-02-10T04:36:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: 02-10T04:36:56.153753+0000 osd.2 (osd.2) 1736 : cluster [WRN] 2021-02-10T04:36:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: cluster 2021-02-10T04:36:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: 2021-02-10T04:36:56.153757+0000 osd.2 (osd. 2021-02-10T04:36:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: 2) 1737 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: -10T04:36:56.153762+0000 osd.2 2021-02-10T04:36:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: (osd.2) 1738 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: 02-10T04:36:56.153768+0000 osd.2 ( 2021-02-10T04:36:57.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: osd.2) 1739 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: 2021-02-10T04:36:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: -02-10T04:36:56.153780+0000 2021-02-10T04:36:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: osd.2 (osd.2) 2021-02-10T04:36:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: 1740 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: cephadm 2021-02- 2021-02-10T04:36:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: 10T04:36:56.631714+0000 mgr.y (mgr. 2021-02-10T04:36:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: 14140) 651 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:36:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: cephadm 2021-02 2021-02-10T04:36:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: -10T04:36:56.632120+0000 mgr.y (mgr.14140) 652 : cephadm [INF] 2021-02-10T04:36:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: Refreshing smithi164 facts 2021-02-10T04:36:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:36:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: T04:36:57.026304+0000 2021-02-10T04:36:57.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: osd.4 (osd.4) 897 : 2021-02-10T04:36:57.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:57.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:57.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: 02-10T04:36:57.026314+0000 2021-02-10T04:36:57.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: osd.4 (osd.4) 898 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:57.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: cluster 2021-02-10T04:36:57.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:36:57.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: :36:57.026321+0000 osd.4 (osd.4 2021-02-10T04:36:57.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: ) 899 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:57.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: cluster 2021-02-10T04:36:57.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: 2021-02-10T04:36:57 2021-02-10T04:36:57.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: .026328+0000 osd.4 (osd.4) 900 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:57.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:36:57.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: 10T04:36:57.200307+0000 mon.a (mon.0) 1451 : 2021-02-10T04:36:57.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[29288]: cluster [DBG] osdmap e599: 8 total, 8 up, 8 in 2021-02-10T04:36:57.899 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[33677]: debug 2021-02-10T04:36:57.899 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[33677]: 2021-02-10T04:36:57.595+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:36:57.899 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:57 smithi164 conmon[33677]: 2021-02-10T04:36:57.902 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:36:57.918 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:36:58.006 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:36:58.076 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[52836]: debug 2021-02-10T04:36:58.077 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:36:57 smithi085 conmon[52836]: 2021-02-10T04:36:57.723+0000 7fba75fba700 -1 osd.3 pg_epoch: 599 pg[4.10( v 538'16 (0'0,538'16] local-lis/les=71/72 n=10 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,5,6] r=0 lpr=71 crt=538'16 lcod 538'15 mlcod 538'15 active+clean] scrubber pg(4.10) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6 (0x55fec7c66840) 2021-02-10T04:36:58.139 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:36:58.178 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:36:58.210 INFO:tasks.rados.rados.0.smithi085.stdout:2316: done (9 left) 2021-02-10T04:36:58.210 INFO:tasks.rados.rados.0.smithi085.stdout:2317: read oid 46 snap -1 2021-02-10T04:36:58.210 INFO:tasks.rados.rados.0.smithi085.stdout:2317: expect (ObjNum 788 snap 255 seq_num 788) 2021-02-10T04:36:58.210 INFO:tasks.rados.rados.0.smithi085.stdout:2318: write oid 44 current snap is 259 2021-02-10T04:36:58.211 INFO:tasks.rados.rados.0.smithi085.stdout:2318: seq_num 803 ranges {696021=642012,1856570=575278,3185369=310652} 2021-02-10T04:36:58.216 INFO:tasks.rados.rados.0.smithi085.stdout:2318: writing smithi08575397-44 from 696021 to 1338033 tid 1 2021-02-10T04:36:58.223 INFO:tasks.rados.rados.0.smithi085.stdout:2318: writing smithi08575397-44 from 1856570 to 2431848 tid 2 2021-02-10T04:36:58.229 INFO:tasks.rados.rados.0.smithi085.stdout:2318: writing smithi08575397-44 from 3185369 to 3496021 tid 3 2021-02-10T04:36:58.229 INFO:tasks.rados.rados.0.smithi085.stdout:2319: read oid 42 snap 249 2021-02-10T04:36:58.229 INFO:tasks.rados.rados.0.smithi085.stdout:2319: expect (ObjNum 764 snap 246 seq_num 764) 2021-02-10T04:36:58.229 INFO:tasks.rados.rados.0.smithi085.stdout:2317: expect (ObjNum 788 snap 255 seq_num 788) 2021-02-10T04:36:58.295 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[33677]: debug 2021-02-10T04:36:58.015+0000 7fbec19a1700 -1 osd.4 599 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:58.296 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:36:58.312 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.7 dump_blocked_ops 2021-02-10T04:36:58.367 INFO:tasks.rados.rados.0.smithi085.stdout:2317: done (11 left) 2021-02-10T04:36:58.367 INFO:tasks.rados.rados.0.smithi085.stdout:2320: write (excl) oid 26 current snap is 259 2021-02-10T04:36:58.367 INFO:tasks.rados.rados.0.smithi085.stdout:2320: seq_num 804 ranges {712828=717261,1915052=718936,3212792=300036} 2021-02-10T04:36:58.372 INFO:tasks.rados.rados.0.smithi085.stdout:2320: writing smithi08575397-26 from 712828 to 1430089 tid 1 2021-02-10T04:36:58.379 INFO:tasks.rados.rados.0.smithi085.stdout:2320: writing smithi08575397-26 from 1915052 to 2633988 tid 2 2021-02-10T04:36:58.385 INFO:tasks.rados.rados.0.smithi085.stdout:2320: writing smithi08575397-26 from 3212792 to 3512828 tid 3 2021-02-10T04:36:58.385 INFO:tasks.rados.rados.0.smithi085.stdout:2321: snap_create 2021-02-10T04:36:58.385 INFO:tasks.rados.rados.0.smithi085.stdout:2318: finishing write tid 1 to smithi08575397-44 2021-02-10T04:36:58.385 INFO:tasks.rados.rados.0.smithi085.stdout:2318: finishing write tid 2 to smithi08575397-44 2021-02-10T04:36:58.386 INFO:tasks.rados.rados.0.smithi085.stdout:2318: finishing write tid 3 to smithi08575397-44 2021-02-10T04:36:58.386 INFO:tasks.rados.rados.0.smithi085.stdout:2318: finishing write tid 4 to smithi08575397-44 2021-02-10T04:36:58.386 INFO:tasks.rados.rados.0.smithi085.stdout:2318: finishing write tid 5 to smithi08575397-44 2021-02-10T04:36:58.386 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 1092 (ObjNum 803 snap 259 seq_num 803) dirty exists 2021-02-10T04:36:58.386 INFO:tasks.rados.rados.0.smithi085.stdout:2318: left oid 44 (ObjNum 803 snap 259 seq_num 803) 2021-02-10T04:36:58.386 INFO:tasks.rados.rados.0.smithi085.stdout:2319: expect (ObjNum 764 snap 246 seq_num 764) 2021-02-10T04:36:58.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[49106]: debug 2021-02-10T04:36:58.146+0000 7f77a3e94700 -1 osd.2 599 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:58.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[49106]: debug 2021-02-10T04:36:58.255+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:36:58.541 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:36:58.595 INFO:tasks.rados.rados.0.smithi085.stdout:2320: finishing write tid 1 to smithi08575397-26 2021-02-10T04:36:58.596 INFO:tasks.rados.rados.0.smithi085.stdout:2320: finishing write tid 2 to smithi08575397-26 2021-02-10T04:36:58.596 INFO:tasks.rados.rados.0.smithi085.stdout:2320: finishing write tid 3 to smithi08575397-26 2021-02-10T04:36:58.596 INFO:tasks.rados.rados.0.smithi085.stdout:2320: finishing write tid 4 to smithi08575397-26 2021-02-10T04:36:58.596 INFO:tasks.rados.rados.0.smithi085.stdout:2320: finishing write tid 5 to smithi08575397-26 2021-02-10T04:36:58.596 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 754 (ObjNum 804 snap 259 seq_num 804) dirty exists 2021-02-10T04:36:58.596 INFO:tasks.rados.rados.0.smithi085.stdout:2320: left oid 26 (ObjNum 804 snap 259 seq_num 804) 2021-02-10T04:36:58.628 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:36:58.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: cluster 2021-02-10T04:36:57.118111+0000 osd.2 ( 2021-02-10T04:36:58.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: osd.2) 1741 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:58.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: cluster 2021-02-10T04:36:57.118119+0000 osd.2 (osd.2) 1742 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:58.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: cluster 2021-02-10T04:36:57.118127+0000 osd.2 (osd.2) 1743 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: cluster 2021-02-10T04:36:57.118150+0000 osd.2 (osd.2) 1744 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: cluster 2021-02-10T04:36:57.118158+0000 osd.2 (osd.2 2021-02-10T04:36:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: ) 1745 : cluster [WRN] 2021-02-10T04:36:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: cluster 2021-02-10T04:36:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: 2021-02-10T04:36:57.118164+0000 osd.2 2021-02-10T04:36:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: (osd.2) 1746 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: -02-10T04:36:57.118170+0000 osd.2 2021-02-10T04:36:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: (osd.2) 1747 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:58.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: -02-10T04:36:57.118175+0000 osd.2 2021-02-10T04:36:58.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: (osd.2) 1748 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:58.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:58.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: 02-10T04:36:57.118181+0000 osd.2 ( 2021-02-10T04:36:58.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: osd.2) 1749 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:58.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:58.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: 02-10T04:36:57.118186+0000 osd.2 ( 2021-02-10T04:36:58.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: osd.2) 1750 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:58.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: audit 2021-02-10T04:36:58.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: 2021-02-10T04:36:57.625091 2021-02-10T04:36:58.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1452 : audit 2021-02-10T04:36:58.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:36:58.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:36:58.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: -10T04:36:57.876455+0000 mgr.y ( 2021-02-10T04:36:58.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: mgr.14140) 653 : audit [DBG] from='client.15186 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:36:58.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:36:58.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: -02-10T04:36:58.016130+0000 2021-02-10T04:36:58.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: osd.4 (osd.4) 901 : cluster [WRN] 2021-02-10T04:36:58.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:58.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: cluster 2021-02-10T04:36:58.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: 2021-02-10T04:36:58.016139+0000 osd.4 ( 2021-02-10T04:36:58.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: osd.4) 902 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:58.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:58.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: 02-10T04:36:58.016146+0000 2021-02-10T04:36:58.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: osd.4 (osd.4) 903 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:58.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: 2021-02-10T04:36:58.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: cluster 2021-02-10T04:36:58.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: 2021-02-10T04:36:58.016153+0000 osd.4 (osd 2021-02-10T04:36:58.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: .4) 904 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:58.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:36:58.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: 02-10T04:36:58.128321+0000 mgr.y 2021-02-10T04:36:58.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: (mgr.14140) 654 : cluster [DBG] pgmap v1049: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 460 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 4.0 MiB/s wr, 23 op/s 2021-02-10T04:36:58.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: audit 2021 2021-02-10T04:36:58.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: -02-10T04:36:58.177062+0000 2021-02-10T04:36:58.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: mgr.y (mgr.14140) 655 : audit [DBG] 2021-02-10T04:36:58.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:36:58.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: cluster 2021-02-10T04:36:58.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: 2021-02-10T04:36:58.203891+0000 mon.a ( 2021-02-10T04:36:58.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:58 smithi164 conmon[29288]: mon.0) 1453 : cluster [DBG] osdmap e600: 8 total, 8 up, 8 in 2021-02-10T04:36:58.810 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set noscrub 2021-02-10T04:36:58.834 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: cluster 2021-02-10T04:36:57.118111+0000 osd.2 (osd.2 2021-02-10T04:36:58.834 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: ) 1741 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:58.834 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: cluster 2021-02-10T04:36:57.118119+0000 osd.2 (osd.2) 1742 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:58.834 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: cluster 2021-02-10T04:36:57.118127+0000 osd.2 (osd.2) 1743 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:58.834 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: cluster 2021-02-10T04:36:57.118150+0000 osd.2 (osd.2) 1744 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:58.834 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: cluster 2021-02-10T04:36:57.118158+0000 osd.2 (osd.2) 1745 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:58.835 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: cluster 2021-02-10T04:36:57.118164+0000 osd.2 (osd.2) 1746 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:58.835 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: cluster 2021-02-10T04:36:57.118170+0000 osd.2 (osd.2) 1747 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:58.835 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: cluster 2021-02-10T04:36:57.118175+0000 osd.2 (osd.2) 1748 : 2021-02-10T04:36:58.835 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:58.835 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: cluster 2021-02-10T04:36:57.118181+0000 osd.2 (osd.2) 1749 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:58.835 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: cluster 2021-02-10T04:36:57.118186+0000 2021-02-10T04:36:58.835 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: osd.2 (osd.2) 1750 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:58.835 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: audit 2021-02-10T04:36:58.835 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: 2021-02-10T04:36:57.625091+0000 mon.a ( 2021-02-10T04:36:58.835 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: mon.0) 1452 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:36:58.836 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: audit 2021-02-10T 2021-02-10T04:36:58.836 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: 04:36:57.876455+0000 mgr.y (mgr.14140) 653 : audit 2021-02-10T04:36:58.836 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: [DBG] from='client.15186 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:36:58.836 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:58.836 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: 02-10T04:36:58.016130+0000 osd.4 (osd.4) 901 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:58.836 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: 2021-02-10T04:36:58.836 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:58.836 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: 02-10T04:36:58.016139+0000 osd.4 (osd.4) 902 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:58.836 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: 2021-02-10T04:36:58.836 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:58.836 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: -10T04:36:58.016146+0000 osd.4 (osd.4) 903 : cluster 2021-02-10T04:36:58.836 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:58.837 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:58.837 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: -02-10T04:36:58.016153+0000 osd.4 (osd.4) 904 : cluster [WRN] 2021-02-10T04:36:58.837 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:58.837 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:58.837 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: -02-10T04:36:58.128321+0000 mgr.y (mgr.14140) 654 : cluster [DBG] 2021-02-10T04:36:58.837 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: pgmap v1049: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 460 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 4.0 MiB/s wr, 23 op/s 2021-02-10T04:36:58.837 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:36:58.837 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: 02-10T04:36:58.177062+0000 mgr.y (mgr.14140) 655 : audit [DBG] 2021-02-10T04:36:58.837 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:36:58.837 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:58.837 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: 02-10T04:36:58.203891+0000 mon.a (mon.0) 1453 : cluster [DBG] osdmap e600: 8 total, 8 up, 8 in 2021-02-10T04:36:58.837 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[29319]: 2021-02-10T04:36:58.838 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: cluster 2021-02-10T04:36:58.838 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:36:58.838 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: 36:57.118111+0000 osd.2 (osd.2) 1741 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:58.838 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: cluster 2021-02-10T04:36:58.838 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: 2021-02-10T04:36:57.118119+0000 2021-02-10T04:36:58.838 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: osd.2 (osd.2) 1742 : cluster [WRN] 2021-02-10T04:36:58.839 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:58.839 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:58.839 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: -02-10T04:36:57.118127+0000 2021-02-10T04:36:58.839 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: osd.2 (osd.2) 1743 : cluster [WRN] 2021-02-10T04:36:58.839 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:58.839 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:58.839 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: T04:36:57.118150+0000 osd.2 (osd 2021-02-10T04:36:58.839 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: .2) 1744 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:58.839 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: cluster 2021-02-10T04:36:58.839 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:36:58.839 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: 36:57.118158+0000 osd.2 (osd.2) 1745 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:58.839 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: cluster 2021-02-10T04:36:57.118164 2021-02-10T04:36:58.839 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1746 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:58.840 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: cluster 2021-02-10T04:36:57.118170+0000 osd.2 (osd.2) 1747 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:58.840 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: cluster 2021-02-10T04:36:58.840 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: 2021-02-10T04:36:57.118175+0000 osd.2 (osd.2) 1748 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:58.840 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: cluster 2021-02-10T04:36:57.118181+0000 osd.2 (osd.2) 1749 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:58.840 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: cluster 2021-02-10T04:36:58.840 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:36:58.840 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: T04:36:57.118186+0000 osd.2 (osd.2) 1750 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:58.840 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: audit 2021-02-10T04:36: 2021-02-10T04:36:58.840 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: 57.625091+0000 mon.a (mon.0) 1452 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:36:58.840 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: audit 2021-02-10T04:36:57.876455+0000 mgr.y (mgr.14140) 653 : audit [DBG] from='client.15186 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:36:58.840 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:58.840 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: T04:36:58.016130+0000 osd.4 (osd.4) 901 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:58.840 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: cluster 2021-02-10T04:36:58.016139+0000 osd.4 (osd.4) 902 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:58.841 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: cluster 2021-02-10T04:36:58.016146+0000 osd.4 (osd.4) 903 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:58.841 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: cluster 2021-02-10T04:36:58.016153+0000 osd.4 (osd.4) 904 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:58.841 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: cluster 2021-02-10T04:36:58.128321+0000 mgr.y (mgr.14140) 654 : cluster [DBG] pgmap v1049: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 460 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 4.0 MiB/s wr, 23 op/s 2021-02-10T04:36:58.841 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:36:58.841 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: :36:58.177062+0000 mgr.y (mgr.14140) 655 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:36:58.841 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:58 smithi085 conmon[35961]: cluster 2021-02-10T04:36:58.203891+0000 mon.a (mon.0) 1453 : cluster [DBG] osdmap e600: 8 total, 8 up, 8 in 2021-02-10T04:36:58.847 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:36:59.107 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:36:59.149 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:36:59.204 INFO:tasks.rados.rados.0.smithi085.stdout:2318: done (12 left) 2021-02-10T04:36:59.204 INFO:tasks.rados.rados.0.smithi085.stdout:2319: done (11 left) 2021-02-10T04:36:59.204 INFO:tasks.rados.rados.0.smithi085.stdout:2320: done (10 left) 2021-02-10T04:36:59.204 INFO:tasks.rados.rados.0.smithi085.stdout:2321: done (9 left) 2021-02-10T04:36:59.204 INFO:tasks.rados.rados.0.smithi085.stdout:2322: snap_create 2021-02-10T04:36:59.279 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:36:59 smithi164 conmon[33677]: debug 2021-02-10T04:36:59.058+0000 7fbec19a1700 -1 osd.4 600 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:36:59.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[49106]: debug 2021-02-10T04:36:59.142+0000 7f77a3e94700 -1 osd.2 600 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:36:59.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:59 smithi164 conmon[29288]: cluster 2021-02-10T04:36:58.147735+0000 osd.2 (osd 2021-02-10T04:36:59.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:59 smithi164 conmon[29288]: .2) 1751 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:59.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:59 smithi164 conmon[29288]: cluster 2021-02-10T04:36:58.147744+0000 osd.2 (osd.2) 1752 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:59.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:59 smithi164 conmon[29288]: cluster 2021-02-10T04:36:58.147752+0000 osd.2 (osd.2) 1753 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:59.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:59 smithi164 conmon[29288]: cluster 2021-02-10T04:36:58.147758+0000 osd.2 (osd.2) 1754 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:59.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:59 smithi164 conmon[29288]: cluster 2021-02-10T04:36:58.147765+0000 osd.2 (osd.2) 1755 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:59.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:59 smithi164 conmon[29288]: cluster 2021-02-10T04:36:58.147771+0000 osd.2 (osd.2) 1756 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:59.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:59 smithi164 conmon[29288]: cluster 2021-02-10T04:36:58.147776+0000 osd.2 (osd.2) 1757 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:59.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:59 smithi164 conmon[29288]: cluster 2021-02-10T04:36:58.147783+0000 osd.2 (osd.2) 1758 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:59 smithi164 conmon[29288]: cluster 2021-02-10T04:36:58.147789+0000 osd.2 (osd.2) 1759 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:59 smithi164 conmon[29288]: cluster 2021-02-10T04:36:58.147794+0000 osd.2 (osd.2) 1760 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:59 smithi164 conmon[29288]: cluster 2021-02-10T04:36:58.622327+0000 mon.a (mon.0) 1454 : cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 446 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:36:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:59 smithi164 conmon[29288]: cluster 2021-02-10T04:36:59.058737+0000 osd.4 (osd.4) 905 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:59 smithi164 conmon[29288]: cluster 2021-02-10T04:36:59.058745+0000 osd.4 (osd.4) 906 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:59 smithi164 conmon[29288]: cluster 2021-02-10T04:36 2021-02-10T04:36:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:59 smithi164 conmon[29288]: :59.058751+0000 osd.4 (osd.4) 907 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:59 smithi164 conmon[29288]: cluster 2021-02-10T04:36:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:59 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:36:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:59 smithi164 conmon[29288]: :36:59.058757+0000 osd.4 (osd.4) 908 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:59 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:36:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:59 smithi164 conmon[29288]: -10T04:36:59.203940+0000 mon.a 2021-02-10T04:36:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:36:59 smithi164 conmon[29288]: (mon.0) 1455 : cluster [DBG] osdmap e601: 8 total, 8 up, 8 in 2021-02-10T04:36:59.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: cluster 2021-02-10T04:36:58.147735+0000 2021-02-10T04:36:59.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: osd.2 (osd.2) 1751 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:59.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: cluster 2021-02-10T04:36:58.147744+0000 osd.2 (osd.2) 1752 : cluster 2021-02-10T04:36:59.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:59.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:36:59.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: T04:36:58.147752+0000 osd.2 (osd.2) 1753 : cluster 2021-02-10T04:36:59.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:59.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: cluster 2021-02-10T04:36:58.147758 2021-02-10T04:36:59.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1754 : cluster 2021-02-10T04:36:59.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:59.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: cluster 2021-02-10T04:36:58 2021-02-10T04:36:59.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: .147765+0000 osd.2 (osd.2) 1755 : cluster 2021-02-10T04:36:59.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:59.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: cluster 2021-02-10T04:36: 2021-02-10T04:36:59.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: 58.147771+0000 osd.2 (osd.2) 1756 : 2021-02-10T04:36:59.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:59.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:36:59.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: 36:58.147776+0000 osd.2 (osd.2) 2021-02-10T04:36:59.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: 1757 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:59.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:36:59.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: 04:36:58.147783+0000 osd.2 (osd. 2021-02-10T04:36:59.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: 2) 1758 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:59.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:59.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: 02-10T04:36:58.147789+0000 osd.2 2021-02-10T04:36:59.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: (osd.2) 1759 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:59.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: 2021-02-10T04:36:59.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:36:59.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: -02-10T04:36:58.147794+0000 osd.2 (osd. 2021-02-10T04:36:59.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: 2) 1760 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:59.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:36:59.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: -10T04:36:58.622327+0000 2021-02-10T04:36:59.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: mon.a (mon.0) 1454 2021-02-10T04:36:59.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: : cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 446 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:36:59.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:36:59.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: :36:59.058737+0000 osd.4 (osd. 2021-02-10T04:36:59.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: 4) 905 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:59.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:36:59.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: 10T04:36:59.058745+0000 osd.4 (osd.4) 906 : 2021-02-10T04:36:59.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:59.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:36:59.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: 36:59.058751+0000 osd.4 (osd.4) 2021-02-10T04:36:59.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: 907 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:59.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:36:59.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: T04:36:59.058757+0000 osd.4 ( 2021-02-10T04:36:59.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: osd.4) 908 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:59.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:36:59.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: 02-10T04:36:59.203940+0000 2021-02-10T04:36:59.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: mon.a (mon.0) 1455 : cluster [DBG] 2021-02-10T04:36:59.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[29319]: osdmap e601: 8 total, 8 up, 8 in 2021-02-10T04:36:59.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: cluster 2021-02-10T04:36:59.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: 2021-02-10T04:36:59.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: 2021-02-10T04:36:58.147735+0000 2021-02-10T04:36:59.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: osd.2 (osd.2) 1751 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:36:59.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:59.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: -02-10T04:36:58.147744+0000 osd.2 (osd.2) 1752 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:36:59.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:59.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: T04:36:58.147752+0000 osd.2 (osd.2) 2021-02-10T04:36:59.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: 1753 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:36:59.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: cluster 2021-02-10T04:36:58.147758+0000 2021-02-10T04:36:59.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: osd.2 (osd.2) 1754 : cluster [WRN] 2021-02-10T04:36:59.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:36:59.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: cluster 2021-02-10T04:36: 2021-02-10T04:36:59.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: 58.147765+0000 2021-02-10T04:36:59.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: osd.2 (osd.2) 1755 : cluster [WRN] 2021-02-10T04:36:59.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:36:59.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:36:59.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: T04:36:58.147771+0000 osd.2 ( 2021-02-10T04:36:59.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: osd.2) 1756 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:36:59.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: cluster 2021-02-10T04:36:59.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:36:59.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: 36:58.147776+0000 osd.2 (osd.2 2021-02-10T04:36:59.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: ) 1757 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:36:59.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: cluster 2021-02-10T04:36:59.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: 2021-02-10T04:36:58 2021-02-10T04:36:59.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: .147783+0000 osd.2 (osd.2) 1758 : 2021-02-10T04:36:59.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:36:59.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:59.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: 02-10T04:36:58.147789+0000 osd.2 ( 2021-02-10T04:36:59.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: osd.2) 1759 2021-02-10T04:36:59.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:36:59.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: 2021-02-10T04:36:59.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:59.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: -02-10T04:36:58 2021-02-10T04:36:59.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: .147794+0000 osd.2 (osd.2) 2021-02-10T04:36:59.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: 1760 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:36:59.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:36:59.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: -02-10T04:36:58.622327 2021-02-10T04:36:59.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1454 : cluster 2021-02-10T04:36:59.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: [WRN] Health check update: 14 slow ops, oldest one blocked for 446 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:36:59.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: cluster 2021-02-10T04:36:59.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: 2021-02-10T04:36:59.058737+0000 2021-02-10T04:36:59.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: osd.4 (osd.4 2021-02-10T04:36:59.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: ) 905 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:36:59.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:59.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: 02-10T04:36:59.058745+0000 osd.4 2021-02-10T04:36:59.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: (osd.4) 906 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:36:59.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:36:59.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: 04:36:59.058751+0000 osd.4 (osd. 2021-02-10T04:36:59.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: 4) 907 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:36:59.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:36:59.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: 02-10T04:36:59.058757+0000 osd.4 ( 2021-02-10T04:36:59.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: osd.4) 908 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:36:59.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: cluster 2021-02-10T04:36: 2021-02-10T04:36:59.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: 59.203940+0000 mon.a (mon.0) 1455 : cluster [DBG] osdmap e601: 8 total, 8 up, 8 in 2021-02-10T04:36:59.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:36:59 smithi085 conmon[35961]: 2021-02-10T04:37:00.046 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[33677]: debug 2021-02-10T04:37:00.045+0000 7fbec19a1700 -1 osd.4 601 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:00.206 INFO:tasks.rados.rados.0.smithi085.stdout:2322: done (9 left) 2021-02-10T04:37:00.206 INFO:tasks.rados.rados.0.smithi085.stdout:2323: snap_create 2021-02-10T04:37:00.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[49106]: debug 2021-02-10T04:37:00.139+0000 7f77a3e94700 -1 osd.2 601 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:37:00.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: cluster 2021-02-10T04:36:59.144262+0000 osd.2 2021-02-10T04:37:00.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: (osd.2) 1761 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:00.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: cluster 2021-02-10T04:36:59.144269+0000 osd.2 (osd.2) 1762 : cluster [WRN] 2021-02-10T04:37:00.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:00.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: cluster 2021-02-10T04:36:59. 2021-02-10T04:37:00.791 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: 144273+0000 osd.2 (osd.2) 1763 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:00.791 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: cluster 2021-02-10T04:36:59.144281 2021-02-10T04:37:00.791 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1764 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:00.791 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: cluster 2021-02-10T04:37:00.791 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: 2021-02-10T04:36:59.144285+0000 2021-02-10T04:37:00.791 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: osd.2 (osd.2) 1765 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:00.791 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: cluster 2021-02-10T04:37:00.791 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: 2021-02-10T04:36:59.144291+0000 2021-02-10T04:37:00.791 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: osd.2 (osd.2) 1766 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:00.791 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: cluster 2021-02-10T04:37:00.791 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: 2021-02-10T04:36:59.144295+0000 2021-02-10T04:37:00.791 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: osd.2 (osd.2) 1767 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:00.791 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: cluster 2021-02-10T04:37:00.792 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:37:00.792 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: 36:59.144299+0000 osd.2 (osd.2) 1768 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:00.792 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: 2021-02-10T04:37:00.792 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:37:00.792 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: 10T04:36:59.144303+0000 osd.2 (osd.2) 1769 : 2021-02-10T04:37:00.792 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:00.792 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: cluster 2021-02-10T04:36 2021-02-10T04:37:00.792 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: :59.144306+0000 osd.2 (osd.2) 1770 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:00.792 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:37:00.792 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: 10T04:37:00.045763+0000 osd.4 (osd.4) 909 2021-02-10T04:37:00.792 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:00.792 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: cluster 2021-02-10T04:37: 2021-02-10T04:37:00.792 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: 00.045772+0000 osd.4 (osd.4) 910 : cluster [WRN] 2021-02-10T04:37:00.793 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:00.793 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: cluster 2021-02-10T04:37 2021-02-10T04:37:00.793 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: :00.045778+0000 osd.4 (osd.4) 911 : cluster 2021-02-10T04:37:00.793 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:00.793 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: cluster 2021-02-10T04:37:00.793 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: 2021-02-10T04:37:00 2021-02-10T04:37:00.793 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: .045784+0000 osd.4 (osd.4) 912 : cluster [WRN] 2021-02-10T04:37:00.793 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:00.793 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: cluster 2021-02-10T04:37:00.793 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: 2021-02-10T04:37:00.793 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: 2021 2021-02-10T04:37:00.793 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: - 2021-02-10T04:37:00.793 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: 02 2021-02-10T04:37:00.794 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: - 2021-02-10T04:37:00.794 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: 10 2021-02-10T04:37:00.794 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: T04:37:00.128958+0000 2021-02-10T04:37:00.794 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: mgr.y (mgr.14140) 656 : cluster [DBG] pgmap v1052: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 461 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 1.1 MiB/s wr, 5 op/s 2021-02-10T04:37:00.794 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:37:00.794 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: -10T04:37:00.205579+0000 mon.a (mon.0 2021-02-10T04:37:00.794 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[29319]: ) 1456 : cluster [DBG] osdmap e602: 8 total, 8 up, 8 in 2021-02-10T04:37:00.794 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: cluster 2021-02-10T04:37:00.794 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: 2021-02-10T04:37:00.794 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: 2021 2021-02-10T04:37:00.794 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: - 2021-02-10T04:37:00.795 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: 02 2021-02-10T04:37:00.795 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: - 2021-02-10T04:37:00.795 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: 10 2021-02-10T04:37:00.795 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: T 2021-02-10T04:37:00.795 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: 04 2021-02-10T04:37:00.795 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: : 2021-02-10T04:37:00.795 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: 36 2021-02-10T04:37:00.795 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: : 2021-02-10T04:37:00.795 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: 59 2021-02-10T04:37:00.795 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: .144262+0000 osd.2 2021-02-10T04:37:00.795 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: (osd.2) 1761 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:00.796 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: cluster 2021-02-10T04:37:00.796 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: 2021-02-10T04:36:59.144269+0000 osd.2 (osd.2) 1762 : cluster 2021-02-10T04:37:00.796 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:00.796 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: cluster 2021-02-10T04:36 2021-02-10T04:37:00.796 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: :59.144273+0000 osd.2 (osd.2) 1763 2021-02-10T04:37:00.796 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:00.796 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: cluster 2021-02-10T04:37:00.796 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:37:00.796 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: :36:59.144281+0000 osd.2 (osd.2 2021-02-10T04:37:00.796 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: ) 1764 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:00.796 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: cluster 2021-02-10T04:37:00.796 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: 2021-02-10T04:36:59.144285 2021-02-10T04:37:00.796 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1765 : cluster [WRN] 2021-02-10T04:37:00.796 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:00.797 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: cluster 2021-02-10T04:36:59. 2021-02-10T04:37:00.797 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: 144291+0000 osd.2 (osd.2) 1766 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:00.797 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: 2021-02-10T04:37:00.797 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: cluster 2021-02-10T04:37:00.797 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: 2021-02-10T04:36:59.144295+0000 osd.2 (osd 2021-02-10T04:37:00.797 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: .2) 1767 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:00.797 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: cluster 2021-02-10T04:36:59.144299+0000 2021-02-10T04:37:00.797 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: osd.2 (osd.2) 1768 : cluster [WRN] 2021-02-10T04:37:00.797 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:00.797 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:00.797 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: 02-10T04:36:59.144303+0000 osd.2 (osd.2) 1769 : cluster 2021-02-10T04:37:00.797 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:00.797 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:37:00.797 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: -10T04:36:59.144306+0000 osd.2 2021-02-10T04:37:00.798 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: (osd.2) 1770 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:00.798 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: 2021-02-10T04:37:00.798 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: cluster 2021-02-10T04:37: 2021-02-10T04:37:00.798 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: 00.045763+0000 osd.4 (osd.4) 909 : cluster 2021-02-10T04:37:00.798 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:00.798 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:37:00.798 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: 37:00.045772+0000 osd.4 (osd.4) 910 2021-02-10T04:37:00.798 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:00.798 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: cluster 2021-02-10T04:37 2021-02-10T04:37:00.798 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: :00.045778+0000 osd.4 (osd.4) 911 2021-02-10T04:37:00.798 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:00.798 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:37:00.798 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: 04:37:00.045784+0000 osd.4 (osd. 2021-02-10T04:37:00.799 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: 4) 912 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:00.799 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:37:00.799 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: 10T04:37:00.128958+0000 mgr.y ( 2021-02-10T04:37:00.799 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: mgr.14140) 656 : cluster [DBG] pgmap v1052: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 461 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 1.1 MiB/s wr, 5 op/s 2021-02-10T04:37:00.799 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:00.799 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: 02-10T04:37:00.205579+0000 mon.a 2021-02-10T04:37:00.799 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: (mon.0) 1456 : cluster [DBG] osdmap e602: 8 total, 8 up, 8 in 2021-02-10T04:37:00.799 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[35961]: 2021-02-10T04:37:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: cluster 2021-02-10T04:36:59.144262+0000 osd.2 (osd.2) 1761 : cluster 2021-02-10T04:37:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: cluster 2021-02-10T04:36:59.144269+0000 osd.2 (osd.2) 1762 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:37:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: 10T04:36:59.144273+0000 osd.2 (osd.2) 1763 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: cluster 2021-02-10T04:36:59. 2021-02-10T04:37:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: 144281+0000 osd.2 (osd.2) 1764 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: cluster 2021-02-10T04:36 2021-02-10T04:37:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: :59.144285+0000 osd.2 (osd 2021-02-10T04:37:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: .2) 1765 : cluster [WRN] 2021-02-10T04:37:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:37:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: T04:36:59.144291+0000 osd.2 (osd.2 2021-02-10T04:37:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: ) 1766 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:37:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: 04:36:59.144295+0000 osd.2 (osd.2 2021-02-10T04:37:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: ) 1767 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:37:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: :36:59.144299+0000 osd.2 (osd.2 2021-02-10T04:37:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: ) 1768 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:37:00.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: 10T04:36:59.144303+0000 osd.2 (osd 2021-02-10T04:37:00.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: .2) 1769 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:00.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:37:00.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: -10T04:36:59.144306+0000 osd.2 (osd. 2021-02-10T04:37:00.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: 2) 1770 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:00.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:37:00.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: -02-10T04:37:00.045763+0000 2021-02-10T04:37:00.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: osd.4 (osd.4) 909 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:00.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:37:00.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: -10T04:37:00. 2021-02-10T04:37:00.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: 045772+0000 osd.4 (osd.4) 2021-02-10T04:37:00.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: 910 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:00.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:37:00.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: -02-10T04:37: 2021-02-10T04:37:00.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: 00.045778+0000 osd.4 (osd 2021-02-10T04:37:00.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: .4) 911 : cluster [WRN] 2021-02-10T04:37:00.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:00.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: cluster 2021-02-10T04:37 2021-02-10T04:37:00.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: :00.045784+0000 osd.4 (osd.4) 912 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:00.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:37:00.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: :37:00.128958+0000 mgr.y (mgr.14140) 656 : 2021-02-10T04:37:00.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: cluster [DBG] pgmap v1052: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 461 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 1.1 MiB/s wr, 5 op/s 2021-02-10T04:37:00.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:37:00.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: 10T04:37:00.205579+0000 mon.a (mon 2021-02-10T04:37:00.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:00 smithi164 conmon[29288]: .0) 1456 : cluster [DBG] osdmap e602: 8 total, 8 up, 8 in 2021-02-10T04:37:01.141 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:37:00 smithi085 conmon[52836]: debug 2021-02-10T04:37:00.787+0000 7fba75fba700 -1 osd.3 pg_epoch: 602 pg[4.10( v 538'16 (0'0,538'16] local-lis/les=71/72 n=10 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,5,6] r=0 lpr=71 crt=538'16 lcod 538'15 mlcod 538'15 active+clean] scrubber pg(4.10) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6 (0x55fec097cc60) 2021-02-10T04:37:01.208 INFO:tasks.rados.rados.0.smithi085.stdout:2323: done (9 left) 2021-02-10T04:37:01.208 INFO:tasks.rados.rados.0.smithi085.stdout:2324: snap_remove snap 255 2021-02-10T04:37:01.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[33677]: debug 2021-02-10T04:37:01.093+0000 7fbec19a1700 -1 osd.4 602 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:01.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[49106]: debug 2021-02-10T04:37:01.139+0000 7f77a3e94700 -1 osd.2 602 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:37:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: cluster 2021-02-10T04:37:00.140907+0000 osd.2 (osd.2) 1771 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: cluster 2021-02-10T04:37:00.140915+0000 osd.2 (osd.2) 1772 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: cluster 2021-02-10T04:37:00.140921+0000 osd.2 (osd.2) 1773 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:37:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: 02-10T04:37:00.140927+0000 osd.2 (osd.2) 1774 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:37:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: -02-10T04:37:00.140935+0000 osd.2 (osd.2) 1775 : cluster [WRN] 2021-02-10T04:37:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:37:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: 02-10T04:37:00.140941+0000 osd.2 (osd.2) 1776 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: cluster 2021-02-10T04:37: 2021-02-10T04:37:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: 00.140948+0000 osd.2 (osd.2) 1777 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: 2021-02-10T04:37:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:37:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: -10T04:37:00.140955+0000 osd.2 (osd.2) 1778 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: 2021-02-10T04:37:01.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:37:01.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: 04:37:00.140961+0000 osd.2 (osd.2) 1779 : cluster [WRN] 2021-02-10T04:37:01.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:01.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: cluster 2021-02-10T04:37:01.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: 2021-02-10T04:37:00.140971+0000 osd.2 (osd.2 2021-02-10T04:37:01.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: ) 1780 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:01.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:37:01.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: T04:37:01.094010+0000 osd.4 (osd. 2021-02-10T04:37:01.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: 4) 913 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:01.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:37:01.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: :37:01.094018+0000 osd.4 (osd.4) 914 : 2021-02-10T04:37:01.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:01.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: cluster 2021-02-10T04:37:01.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: 2021-02-10T04:37:01.094026+0000 osd.4 (osd.4) 915 : 2021-02-10T04:37:01.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:01.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:37:01.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: 04:37:01.094031+0000 osd.4 (osd.4) 916 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:01.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:37:01.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:01 smithi164 conmon[29288]: -10T04:37:01.207456+0000 mon.a (mon.0) 1457 : cluster [DBG] osdmap e603: 8 total, 8 up, 8 in 2021-02-10T04:37:01.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: cluster 2021-02-10T04:37:00 2021-02-10T04:37:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: .140907+0000 osd.2 (osd.2) 1771 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: cluster 2021-02-10T04:37: 2021-02-10T04:37:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: 00.140915+0000 osd.2 (osd.2) 1772 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:37:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: -10T04:37:00.140921+0000 osd.2 2021-02-10T04:37:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: (osd.2) 1773 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: cluster 2021-02-10T04:37:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: 2021-02-10T04:37:00. 2021-02-10T04:37:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: 140927+0000 osd.2 (osd.2) 1774 : cluster [WRN] 2021-02-10T04:37:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:01.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:37:01.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: :37:00.140935+0000 osd.2 (osd. 2021-02-10T04:37:01.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: 2) 1775 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:01.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:37:01.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: -10T04:37:00.140941+0000 osd.2 2021-02-10T04:37:01.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: (osd.2) 1776 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:01.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: cluster 2021-02-10T04:37:00.140948+0000 osd.2 (osd.2) 1777 : cluster 2021-02-10T04:37:01.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:01.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:37:01.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: T04:37:00.140955+0000 osd.2 (osd.2) 1778 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:01.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: cluster 2021-02-10T04:37:00.140961+0000 2021-02-10T04:37:01.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: osd.2 (osd.2) 1779 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:01.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: cluster 2021-02-10T04:37:00.140971+0000 osd.2 (osd.2) 1780 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:01.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: cluster 2021-02-10T04:37:01 2021-02-10T04:37:01.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: .094010+0000 osd.4 (osd.4) 913 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:01.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:37:01.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: -10T04:37:01.094018+0000 osd.4 (osd.4) 914 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:01.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: cluster 2021-02-10T04:37:01.094026+0000 2021-02-10T04:37:01.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: osd.4 (osd.4) 915 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:01.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: cluster 2021-02-10T04:37:01.094031+0000 osd.4 (osd 2021-02-10T04:37:01.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: .4) 916 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:01.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: cluster 2021-02-10T04:37:01.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[29319]: 2021-02-10T04:37:01.207456+0000 mon.a (mon.0) 1457 : cluster [DBG] osdmap e603: 8 total, 8 up, 8 in 2021-02-10T04:37:01.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: cluster 2021-02-10T04:37:00.140907 2021-02-10T04:37:01.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1771 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:01.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:37:01.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: :37:00.140915+0000 osd.2 (osd.2) 1772 : 2021-02-10T04:37:01.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:01.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:01.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: -02-10T04:37:00.140921+0000 osd.2 2021-02-10T04:37:01.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: (osd.2) 1773 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:01.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: cluster 2021-02-10T04:37:00.140927+0000 osd.2 (osd.2 2021-02-10T04:37:01.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: ) 1774 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:01.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:01.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: -02-10T04:37:00.140935+0000 2021-02-10T04:37:01.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: osd.2 (osd.2) 1775 2021-02-10T04:37:01.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:01.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:01.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: 02-10T04:37:00.140941+0000 2021-02-10T04:37:01.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: osd.2 (osd.2) 1776 : cluster [WRN] 2021-02-10T04:37:01.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:01.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: cluster 2021-02-10T04:37:00 2021-02-10T04:37:01.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: .140948+0000 osd.2 (osd.2) 1777 : cluster 2021-02-10T04:37:01.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:01.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:37:01.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: 04:37:00.140955+0000 osd.2 (osd. 2021-02-10T04:37:01.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: 2) 1778 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:01.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:37:01.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: :37:00.140961+0000 osd.2 (osd.2) 1779 : cluster [WRN] 2021-02-10T04:37:01.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:01.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: cluster 2021-02-10T04:37:01.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: 2021-02-10T04:37:00.140971+0000 osd.2 (osd.2) 1780 2021-02-10T04:37:01.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:01.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: cluster 2021-02-10T04:37:01.094010+0000 osd.4 (osd.4) 913 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:01.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:37:01.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: 37:01.094018+0000 osd.4 (osd.4) 914 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:01.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: 2021-02-10T04:37:01.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:01.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: -02-10T04:37:01.094026+0000 osd.4 (osd.4) 915 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:01.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: 2021-02-10T04:37:01.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: cluster 2021-02-10T04:37 2021-02-10T04:37:01.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: :01.094031+0000 osd.4 ( 2021-02-10T04:37:01.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: osd.4) 916 : cluster 2021-02-10T04:37:01.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:01.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: cluster 2021-02-10T04:37:01.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: 2021-02-10T04:37:01.207456 2021-02-10T04:37:01.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1457 : cluster 2021-02-10T04:37:01.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:01 smithi085 conmon[35961]: [DBG] osdmap e603: 8 total, 8 up, 8 in 2021-02-10T04:37:02.215 INFO:tasks.rados.rados.0.smithi085.stdout:2324: done (9 left) 2021-02-10T04:37:02.215 INFO:tasks.rados.rados.0.smithi085.stdout:2325: snap_remove snap 241 2021-02-10T04:37:02.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[33677]: debug 2021-02-10T04:37:02.121+0000 7fbec19a1700 -1 osd.4 603 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:02.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[49106]: debug 2021-02-10T04:37:02.442 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[49106]: 2021-02-10T04:37:02.132+0000 7f77a3e94700 -1 osd.2 603 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:37:02.442 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[49106]: 2021-02-10T04:37:02.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: cluster 2021-02-10T04:37:01.140552+0000 osd.2 ( 2021-02-10T04:37:02.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: osd.2) 1781 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:02.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: cluster 2021-02-10T04:37:01.140560+0000 osd.2 (osd.2) 1782 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:02.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: 2021-02-10T04:37:02.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:37:02.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: 10T04:37:01.140567+0000 osd.2 (osd.2) 1783 : cluster 2021-02-10T04:37:02.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:02.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: cluster 2021-02-10T04:37 2021-02-10T04:37:02.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: :01.140578+0000 osd.2 (osd.2) 1784 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:02.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: cluster 2021-02-10T04:37 2021-02-10T04:37:02.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: :01.140584+0000 osd.2 (osd.2) 1785 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:02.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: cluster 2021-02-10T04:37:02.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:37:02.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: :37:01.140590+0000 osd.2 (osd 2021-02-10T04:37:02.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: .2) 1786 : cluster [WRN] 2021-02-10T04:37:02.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:02.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:37:02.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: -10T04:37:01.140605+0000 osd.2 (osd. 2021-02-10T04:37:02.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: 2) 1787 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:02.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: cluster 2021-02-10T04:37 2021-02-10T04:37:02.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: :01.140611+0000 osd.2 (osd.2) 1788 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:02.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:37:02.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: 02-10T04:37:01.140616+0000 osd.2 (osd.2) 1789 : cluster [WRN] 2021-02-10T04:37:02.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:02.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: cluster 2021-02-10T04:37 2021-02-10T04:37:02.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: :01.140621+0000 osd.2 (osd.2) 1790 : 2021-02-10T04:37:02.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:02.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:37:02.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: -10T04:37:02.129545+0000 mgr.y ( 2021-02-10T04:37:02.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: mgr.14140) 657 : cluster [DBG] pgmap v1055: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 457 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.6 MiB/s rd, 797 KiB/s wr, 6 op/s 2021-02-10T04:37:02.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:37:02.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: 02-10T04:37:02.210994+0000 mon.a ( 2021-02-10T04:37:02.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: mon.0) 1458 : cluster 2021-02-10T04:37:02.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:02 smithi164 conmon[29288]: [DBG] osdmap e604: 8 total, 8 up, 8 in 2021-02-10T04:37:02.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: cluster 2021-02-10T04:37:02.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: 2021-02-10T04:37:02.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: 2021 2021-02-10T04:37:02.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: - 2021-02-10T04:37:02.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: 02 2021-02-10T04:37:02.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: - 2021-02-10T04:37:02.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: 10T04:37:01.140552+0000 2021-02-10T04:37:02.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: osd.2 (osd.2) 1781 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:02.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: cluster 2021-02-10T04:37:02.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: 2021-02-10T04:37:01.140560+0000 osd.2 (osd.2) 1782 : cluster [WRN] 2021-02-10T04:37:02.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:02.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: cluster 2021-02-10T04:37:01.140567+0000 osd.2 (osd.2) 1783 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:02.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:37:02.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: T04:37:01.140578+0000 osd.2 (osd.2 2021-02-10T04:37:02.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: ) 1784 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:02.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:37:02.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: T04:37:01.140584+0000 osd.2 (osd. 2021-02-10T04:37:02.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: 2) 1785 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:02.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:02.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: 02-10T04:37:01.140590+0000 osd.2 2021-02-10T04:37:02.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: (osd.2) 1786 : cluster [WRN] 2021-02-10T04:37:02.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:02.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: cluster 2021-02-10T04:37:01. 2021-02-10T04:37:02.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: 140605+0000 osd.2 (osd.2) 1787 : cluster 2021-02-10T04:37:02.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:02.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: cluster 2021-02-10T04:37:02.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: 2021-02-10T04:37:02.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: 2021 2021-02-10T04:37:02.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: -02-10T04:37:01. 2021-02-10T04:37:02.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: 140611+0000 osd.2 (osd.2) 1788 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:02.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:37:02.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: 37:01.140616+0000 osd.2 (osd.2) 1789 : 2021-02-10T04:37:02.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:02.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: cluster 2021-02-10T04:37: 2021-02-10T04:37:02.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: 01.140621+0000 osd.2 (osd.2) 1790 : cluster [WRN] 2021-02-10T04:37:02.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:02.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: cluster 2021-02-10T04:37:02 2021-02-10T04:37:02.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: .129545+0000 mgr.y (mgr.14140) 657 : cluster [DBG] 2021-02-10T04:37:02.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: pgmap v1055: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 457 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.6 MiB/s rd, 797 KiB/s wr, 6 op/s 2021-02-10T04:37:02.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: cluster 2021-02-10T04:37:02.210994 2021-02-10T04:37:02.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1458 : cluster [DBG] osdmap e604: 8 total, 8 up, 8 in 2021-02-10T04:37:02.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: cluster 2021-02-10T04:37:02.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: 2021-02-10T04:37:02.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: 2021 2021-02-10T04:37:02.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: - 2021-02-10T04:37:02.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: 02 2021-02-10T04:37:02.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: - 2021-02-10T04:37:02.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: 10 2021-02-10T04:37:02.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: T04:37:01.140552+0000 2021-02-10T04:37:02.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: osd.2 (osd.2) 1781 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:02.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: cluster 2021-02-10T04:37:02.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:37:02.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: 37:01.140560+0000 osd.2 (osd.2) 1782 2021-02-10T04:37:02.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:02.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: cluster 2021-02-10T04:37:02.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:37:02.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: 10T04:37:01.140567+0000 osd.2 (osd.2 2021-02-10T04:37:02.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: ) 1783 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:02.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:37:02.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: -10T04:37:01.140578+0000 osd.2 (osd. 2021-02-10T04:37:02.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: 2) 1784 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:02.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:02.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: -02-10T04:37:01.140584 2021-02-10T04:37:02.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1785 : cluster [WRN] 2021-02-10T04:37:02.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:02.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: cluster 2021-02-10T04:37: 2021-02-10T04:37:02.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: 01.140590+0000 osd.2 (osd.2) 1786 : 2021-02-10T04:37:02.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:02.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:37:02.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: :37:01.140605+0000 osd.2 (osd.2) 2021-02-10T04:37:02.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: 1787 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:02.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:37:02.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: 37:01.140611+0000 osd.2 (osd.2) 1788 2021-02-10T04:37:02.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:02.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:37:02.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: 10T04:37:01.140616+0000 osd.2 (osd 2021-02-10T04:37:02.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: .2) 1789 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:02.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:02.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: -02-10T04:37:01.140621 2021-02-10T04:37:02.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1790 : cluster 2021-02-10T04:37:02.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:02.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: cluster 2021-02-10T04:37 2021-02-10T04:37:02.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: :02.129545+0000 mgr.y (mgr.14140) 657 2021-02-10T04:37:02.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: : cluster [DBG] pgmap v1055: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 457 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.6 MiB/s rd, 797 KiB/s wr, 6 op/s 2021-02-10T04:37:02.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: cluster 2021-02-10T04:37:02 2021-02-10T04:37:02.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: .210994+0000 mon.a (mon.0) 1458 : cluster [DBG] 2021-02-10T04:37:02.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:02 smithi085 conmon[35961]: osdmap e604: 8 total, 8 up, 8 in 2021-02-10T04:37:03.216 INFO:tasks.rados.rados.0.smithi085.stdout:2325: done (9 left) 2021-02-10T04:37:03.216 INFO:tasks.rados.rados.0.smithi085.stdout:2326: read oid 6 snap 249 2021-02-10T04:37:03.216 INFO:tasks.rados.rados.0.smithi085.stdout:2326: expect (ObjNum 729 snap 230 seq_num 729) 2021-02-10T04:37:03.216 INFO:tasks.rados.rados.0.smithi085.stdout:2327: copy_from oid 12 from oid 23 current snap is 262 2021-02-10T04:37:03.216 INFO:tasks.rados.rados.0.smithi085.stdout:2328: rollback oid 7 current snap is 262 2021-02-10T04:37:03.217 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 7 to 244 2021-02-10T04:37:03.217 INFO:tasks.rados.rados.0.smithi085.stdout:2329: read oid 28 snap -1 2021-02-10T04:37:03.217 INFO:tasks.rados.rados.0.smithi085.stdout:2329: expect (ObjNum 792 snap 256 seq_num 792) 2021-02-10T04:37:03.217 INFO:tasks.rados.rados.0.smithi085.stdout:2330: write (excl) oid 10 current snap is 262 2021-02-10T04:37:03.217 INFO:tasks.rados.rados.0.smithi085.stdout:2330: seq_num 805 ranges {729635=792510,1973534=578947,3240215=289420} 2021-02-10T04:37:03.223 INFO:tasks.rados.rados.0.smithi085.stdout:2330: writing smithi08575397-10 from 729635 to 1522145 tid 1 2021-02-10T04:37:03.230 INFO:tasks.rados.rados.0.smithi085.stdout:2330: writing smithi08575397-10 from 1973534 to 2552481 tid 2 2021-02-10T04:37:03.237 INFO:tasks.rados.rados.0.smithi085.stdout:2330: writing smithi08575397-10 from 3240215 to 3529635 tid 3 2021-02-10T04:37:03.237 INFO:tasks.rados.rados.0.smithi085.stdout:2331: copy_from oid 30 from oid 40 current snap is 262 2021-02-10T04:37:03.240 INFO:tasks.rados.rados.0.smithi085.stdout:2326: expect (ObjNum 729 snap 230 seq_num 729) 2021-02-10T04:37:03.336 INFO:tasks.rados.rados.0.smithi085.stdout:2329: expect (ObjNum 792 snap 256 seq_num 792) 2021-02-10T04:37:03.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[49106]: debug 2021-02-10T04:37:03.172+0000 7f77a3e94700 -1 osd.2 604 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:37:03.467 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[33677]: debug 2021-02-10T04:37:03.169+0000 7fbec19a1700 -1 osd.4 604 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:03.566 INFO:tasks.rados.rados.0.smithi085.stdout:2326: done (14 left) 2021-02-10T04:37:03.566 INFO:tasks.rados.rados.0.smithi085.stdout:2329: done (13 left) 2021-02-10T04:37:03.566 INFO:tasks.rados.rados.0.smithi085.stdout:2332: snap_create 2021-02-10T04:37:03.566 INFO:tasks.rados.rados.0.smithi085.stdout:2328: finishing rollback tid 0 to smithi08575397-7 2021-02-10T04:37:03.566 INFO:tasks.rados.rados.0.smithi085.stdout:2328: finishing rollback tid 1 to smithi08575397-7 2021-02-10T04:37:03.566 INFO:tasks.rados.rados.0.smithi085.stdout:2328: finishing rollback tid 2 to smithi08575397-7 2021-02-10T04:37:03.566 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 1135 (ObjNum 759 snap 243 seq_num 759) dirty exists 2021-02-10T04:37:03.567 INFO:tasks.rados.rados.0.smithi085.stdout:2327: finishing copy_from to smithi08575397-12 2021-02-10T04:37:03.567 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 693 (ObjNum 722 snap 226 seq_num 722) dirty exists 2021-02-10T04:37:03.567 INFO:tasks.rados.rados.0.smithi085.stdout:2327: finishing copy_from racing read to smithi08575397-12 2021-02-10T04:37:03.567 INFO:tasks.rados.rados.0.smithi085.stdout:2330: finishing write tid 1 to smithi08575397-10 2021-02-10T04:37:03.567 INFO:tasks.rados.rados.0.smithi085.stdout:2330: finishing write tid 2 to smithi08575397-10 2021-02-10T04:37:03.567 INFO:tasks.rados.rados.0.smithi085.stdout:2330: finishing write tid 3 to smithi08575397-10 2021-02-10T04:37:03.567 INFO:tasks.rados.rados.0.smithi085.stdout:2330: finishing write tid 4 to smithi08575397-10 2021-02-10T04:37:03.567 INFO:tasks.rados.rados.0.smithi085.stdout:2330: finishing write tid 5 to smithi08575397-10 2021-02-10T04:37:03.567 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 10 v 851 (ObjNum 805 snap 262 seq_num 805) dirty exists 2021-02-10T04:37:03.567 INFO:tasks.rados.rados.0.smithi085.stdout:2330: left oid 10 (ObjNum 805 snap 262 seq_num 805) 2021-02-10T04:37:03.567 INFO:tasks.rados.rados.0.smithi085.stdout:2331: finishing copy_from to smithi08575397-30 2021-02-10T04:37:03.567 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 784 (ObjNum 756 snap 241 seq_num 756) dirty exists 2021-02-10T04:37:03.567 INFO:tasks.rados.rados.0.smithi085.stdout:2331: finishing copy_from racing read to smithi08575397-30 2021-02-10T04:37:03.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: cluster 2021-02-10T04:37:03.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: 2021-02-10T04:37:03.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: 2021-02-10T04:37:02. 2021-02-10T04:37:03.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: 122401+0000 osd.4 (osd.4) 917 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:03.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:37:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: 04:37:02.122410+0000 osd.4 (osd.4) 918 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:37:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: -10T04:37:02. 2021-02-10T04:37:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: 122414+0000 osd.4 (osd.4) 919 : 2021-02-10T04:37:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:03.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:37:03.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: 02-10T04:37:02. 2021-02-10T04:37:03.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: 122417+0000 osd.4 (osd.4) 920 : cluster 2021-02-10T04:37:03.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:03.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:37:03.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: 10T04:37:02.133535+0000 2021-02-10T04:37:03.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: osd.2 (osd.2) 1791 : cluster [WRN] 2021-02-10T04:37:03.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:03.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:37:03.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: T04:37:02.133545+0000 osd.2 2021-02-10T04:37:03.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: (osd.2) 1792 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:03.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: 2021-02-10T04:37:03.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:37:03.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: T04:37:02.133553+0000 2021-02-10T04:37:03.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: osd.2 (osd.2) 1793 : 2021-02-10T04:37:03.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:03.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:37:03.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: 02-10T04:37:02.133559+0000 2021-02-10T04:37:03.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: osd.2 (osd.2) 1794 : cluster 2021-02-10T04:37:03.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:03.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: cluster 2021-02-10T04:37:03.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:37:03.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: 37:02.133565 2021-02-10T04:37:03.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1795 2021-02-10T04:37:03.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:03.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: cluster 2021-02-10T04:37:03.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:37:03.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: 37:02.133577+0000 osd.2 (osd 2021-02-10T04:37:03.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: .2) 1796 : cluster [WRN] 2021-02-10T04:37:03.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:03.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:37:03.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: 37:02.133583+0000 osd.2 (osd.2 2021-02-10T04:37:03.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: ) 1797 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:03.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:37:03.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: :37:02.133589+0000 osd.2 (osd. 2021-02-10T04:37:03.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: 2) 1798 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:03.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:37:03.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: T04:37:02.133595+0000 osd.2 (osd 2021-02-10T04:37:03.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: .2) 1799 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:03.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:37:03.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: T04:37:02.133601+0000 osd.2 (osd 2021-02-10T04:37:03.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: .2) 1800 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:03.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:37:03.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: 10T04:37:03.210315 2021-02-10T04:37:03.906 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1459 2021-02-10T04:37:03.906 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:03 smithi164 conmon[29288]: : cluster [DBG] osdmap e605: 8 total, 8 up, 8 in 2021-02-10T04:37:03.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: cluster 2021-02-10T04:37:03.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: 2021-02-10T04:37:03.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: 2021 2021-02-10T04:37:03.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: - 2021-02-10T04:37:03.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: 02 2021-02-10T04:37:03.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: - 2021-02-10T04:37:03.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: 10 2021-02-10T04:37:03.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: T 2021-02-10T04:37:03.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: 04 2021-02-10T04:37:03.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: : 2021-02-10T04:37:03.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: 37:02.122401+0000 2021-02-10T04:37:03.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: osd.4 (osd.4) 917 : cluster [WRN] 2021-02-10T04:37:03.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:03.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:37:03.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: -02-10T04:37:02.122410+0000 osd.4 2021-02-10T04:37:03.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: (osd.4) 918 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:03.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: 2021-02-10T04:37:03.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:37:03.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: 10T04:37:02.122414+0000 osd.4 ( 2021-02-10T04:37:03.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: osd.4) 919 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:03.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:03.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: 02-10T04:37:02.122417+0000 2021-02-10T04:37:03.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: osd.4 (osd.4) 920 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:03.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: 2021-02-10T04:37:03.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: cluster 2021-02-10T04:37:03.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: 2021-02-10T04:37:02.133535+0000 osd.2 (osd 2021-02-10T04:37:03.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: .2) 1791 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:03.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:37:03.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: 10T04:37:02.133545+0000 osd.2 2021-02-10T04:37:03.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: (osd.2) 1792 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:03.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:37:03.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: -02-10T04:37:02.133553+0000 2021-02-10T04:37:03.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: osd.2 (osd.2) 1793 : cluster [WRN] 2021-02-10T04:37:03.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:03.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: cluster 2021-02-10T04:37:03.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: 2021-02-10T04:37:02.133559+0000 osd.2 ( 2021-02-10T04:37:03.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: osd.2) 1794 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:03.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:03.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: 02-10T04:37:02.133565+0000 2021-02-10T04:37:03.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: osd.2 (osd.2) 1795 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:03.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: 2021-02-10T04:37:03.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: cluster 2021-02-10T04:37:03.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: 2021-02-10T04:37:02.133577+0000 osd.2 (osd. 2021-02-10T04:37:03.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: 2) 1796 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:03.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:37:03.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: 04:37:02.133583+0000 osd.2 (osd. 2021-02-10T04:37:03.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: 2) 1797 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:03.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:37:03.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: T04:37:02.133589+0000 osd.2 (osd 2021-02-10T04:37:03.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: .2) 1798 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:37:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: -10T04:37:02.133595+0000 2021-02-10T04:37:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: osd.2 (osd.2) 1799 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: 2021-02-10T04:37:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: cluster 2021-02-10T04:37:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: 2021-02-10T04:37:02.133601+0000 osd.2 (osd 2021-02-10T04:37:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: .2) 1800 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: 02-10T04:37:03.210315+0000 2021-02-10T04:37:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: mon.a (mon.0) 1459 : cluster [DBG] 2021-02-10T04:37:03.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[29319]: osdmap e605: 8 total, 8 up, 8 in 2021-02-10T04:37:03.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: cluster 2021-02-10T04:37:02.122401 2021-02-10T04:37:03.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 917 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:03.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:03.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: -02-10T04:37:02.122410+0000 osd.4 (osd.4) 918 : cluster 2021-02-10T04:37:03.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:03.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: cluster 2021-02-10T04:37:02 2021-02-10T04:37:03.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: .122414+0000 osd.4 (osd.4) 919 : cluster [WRN] 2021-02-10T04:37:03.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:03.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: cluster 2021-02-10T04:37 2021-02-10T04:37:03.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: :02.122417+0000 osd.4 (osd.4) 920 : cluster 2021-02-10T04:37:03.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:03.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: cluster 2021-02-10T04:37:02.133535 2021-02-10T04:37:03.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1791 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:03.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:03.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: 02-10T04:37:02.133545+0000 osd.2 ( 2021-02-10T04:37:03.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: osd.2) 1792 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:03.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:03.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: -02-10T04:37:02.133553+0000 2021-02-10T04:37:03.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: osd.2 (osd.2) 1793 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:03.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:03.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: 02-10T04:37:02.133559+0000 osd.2 (osd 2021-02-10T04:37:03.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: .2) 1794 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:03.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: cluster 2021-02-10T04:37:02 2021-02-10T04:37:03.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: .133565+0000 osd.2 (osd.2) 1795 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:03.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:37:03.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: 04:37:02.133577+0000 osd.2 (osd.2) 1796 : cluster 2021-02-10T04:37:03.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:03.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:03.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: -02-10T04:37:02.133583+0000 osd.2 (osd.2) 1797 : cluster [WRN] 2021-02-10T04:37:03.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:03.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:37:03.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: -10T04:37:02.133589+0000 osd.2 (osd.2) 1798 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:03.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:37:03.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: T04:37:02.133595+0000 osd.2 (osd.2 2021-02-10T04:37:03.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: ) 1799 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:03.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: cluster 2021-02-10T04:37 2021-02-10T04:37:03.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: :02.133601+0000 osd.2 (osd.2) 1800 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:03.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: 2021-02-10T04:37:03.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:03 smithi085 conmon[35961]: cluster 2021-02-10T04:37:03.210315+0000 mon.a (mon.0) 1459 : cluster [DBG] osdmap e605: 8 total, 8 up, 8 in 2021-02-10T04:37:04.134 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:37:04.172 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:37:04.382 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[49106]: debug 2021-02-10T04:37:04.382 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[49106]: 2021-02-10T04:37:04.198+0000 7f77a3e94700 -1 osd.2 605 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:37:04.382 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[49106]: 2021-02-10T04:37:04.416 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":1048,"stamp":"2021-02-10T04:36:57.201229+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":478734095,"num_objects":571,"num_object_clones":167,"num_object_copies":1713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":571,"num_whiteouts":7,"num_read":6616,"num_read_kb":4615726,"num_write":4298,"num_write_kb":1227693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":2080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8962,"ondisk_log_size":8962,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1203048,"kb_used_data":962536,"kb_used_omap":0,"kb_used_meta":237952,"kb_avail":748758168,"statfs":{"total":767960285184,"available":766728364032,"internally_reserved":0,"allocated":985636864,"data_stored":1382412468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":243662848},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,12],"upper_bound":1048576},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11735041,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":-1,"num_read":102,"num_read_kb":107562,"num_write":59,"num_write_kb":21272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.778277"},"pg_stats":[{"pgid":"5.9","version":"131'3","reported_seq":"584","reported_epoch":"595","state":"active+clean","last_fresh":"2021-02-10T04:36:52.442276+0000","last_change":"2021-02-10T04:33:57.871495+0000","last_active":"2021-02-10T04:36:52.442276+0000","last_peered":"2021-02-10T04:36:52.442276+0000","last_clean":"2021-02-10T04:36:52.442276+0000","last_became_active":"2021-02-10T04:29:07.145525+0000","last_became_peered":"2021-02-10T04:29:07.145525+0000","last_unstale":"2021-02-10T04:36:52.442276+0000","last_undegraded":"2021-02-10T04:36:52.442276+0000","last_fullsized":"2021-02-10T04:36:52.442276+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'3","last_scrub_stamp":"2021-02-10T04:33:57.871402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:33:57.871402+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"538'42","reported_seq":"642","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.419679+0000","last_change":"2021-02-10T04:33:55.585872+0000","last_active":"2021-02-10T04:36:51.419679+0000","last_peered":"2021-02-10T04:36:51.419679+0000","last_clean":"2021-02-10T04:36:51.419679+0000","last_became_active":"2021-02-10T04:25:49.002058+0000","last_became_peered":"2021-02-10T04:25:49.002058+0000","last_unstale":"2021-02-10T04:36:51.419679+0000","last_undegraded":"2021-02-10T04:36:51.419679+0000","last_fullsized":"2021-02-10T04:36:51.419679+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2021-02-10T04:33:55.585807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:55.585807+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":77,"num_read_kb":60,"num_write":43,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"585","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.199605+0000","last_change":"2021-02-10T04:29:34.787349+0000","last_active":"2021-02-10T04:36:54.199605+0000","last_peered":"2021-02-10T04:36:54.199605+0000","last_clean":"2021-02-10T04:36:54.199605+0000","last_became_active":"2021-02-10T04:25:36.344641+0000","last_became_peered":"2021-02-10T04:25:36.344641+0000","last_unstale":"2021-02-10T04:36:54.199605+0000","last_undegraded":"2021-02-10T04:36:54.199605+0000","last_fullsized":"2021-02-10T04:36:54.199605+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:29:34.787265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:34.787265+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"554","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.420775+0000","last_change":"2021-02-10T04:29:19.405593+0000","last_active":"2021-02-10T04:36:51.420775+0000","last_peered":"2021-02-10T04:36:51.420775+0000","last_clean":"2021-02-10T04:36:51.420775+0000","last_became_active":"2021-02-10T04:25:53.148090+0000","last_became_peered":"2021-02-10T04:25:53.148090+0000","last_unstale":"2021-02-10T04:36:51.420775+0000","last_undegraded":"2021-02-10T04:36:51.420775+0000","last_fullsized":"2021-02-10T04:36:51.420775+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:19.405525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:29:19.405525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"595'746","reported_seq":"1891","reported_epoch":"595","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:36:52.709342+0000","last_change":"2021-02-10T04:36:49.589137+0000","last_active":"2021-02-10T04:36:52.709342+0000","last_peered":"2021-02-10T04:36:52.709342+0000","last_clean":"2021-02-10T04:36:52.709342+0000","last_became_active":"2021-02-10T04:33:05.144488+0000","last_became_peered":"2021-02-10T04:33:05.144488+0000","last_unstale":"2021-02-10T04:36:52.709342+0000","last_undegraded":"2021-02-10T04:36:52.709342+0000","last_fullsized":"2021-02-10T04:36:52.709342+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'301","last_scrub_stamp":"2021-02-10T04:33:56.571925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:56.571925+0000","log_size":746,"ondisk_log_size":746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"stat_sum":{"num_bytes":57965325,"num_objects":26,"num_object_clones":21,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":1,"num_read":362,"num_read_kb":387176,"num_write":251,"num_write_kb":93172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"},{"start":"d4","length":"8"},{"start":"dd","length":"1"},{"start":"e1","length":"3"},{"start":"e9","length":"2"},{"start":"ed","length":"4"},{"start":"f2","length":"1"},{"start":"f4","length":"5"},{"start":"fb","length":"1"},{"start":"fd","length":"4"},{"start":"103","length":"1"},{"start":"106","length":"4"},{"start":"10c","length":"1"},{"start":"10f","length":"6"},{"start":"116","length":"2"},{"start":"11c","length":"3"},{"start":"121","length":"1"},{"start":"123","length":"2"},{"start":"126","length":"1"},{"start":"128","length":"5"},{"start":"12e","length":"3"},{"start":"133","length":"4"},{"start":"139","length":"2"},{"start":"13c","length":"3"},{"start":"140","length":"1"},{"start":"143","length":"1"},{"start":"146","length":"1"},{"start":"148","length":"1"},{"start":"14c","length":"1"},{"start":"14e","length":"1"},{"start":"155","length":"2"},{"start":"15a","length":"4"},{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"168","length":"2"},{"start":"16b","length":"1"},{"start":"16f","length":"5"},{"start":"178","length":"4"},{"start":"17f","length":"1"},{"start":"185","length":"2"},{"start":"18a","length":"2"},{"start":"18e","length":"3"},{"start":"194","length":"4"},{"start":"19b","length":"2"},{"start":"19e","length":"2"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a8","length":"1"},{"start":"1ab","length":"1"},{"start":"1ae","length":"1"},{"start":"1b1","length":"1"},{"start":"1b7","length":"1"},{"start":"1b9","length":"2"}]},{"pgid":"4.b","version":"538'26","reported_seq":"609","reported_epoch":"595","state":"active+clean","last_fresh":"2021-02-10T04:36:52.428160+0000","last_change":"2021-02-10T04:28:50.696234+0000","last_active":"2021-02-10T04:36:52.428160+0000","last_peered":"2021-02-10T04:36:52.428160+0000","last_clean":"2021-02-10T04:36:52.428160+0000","last_became_active":"2021-02-10T04:25:49.004346+0000","last_became_peered":"2021-02-10T04:25:49.004346+0000","last_unstale":"2021-02-10T04:36:52.428160+0000","last_undegraded":"2021-02-10T04:36:52.428160+0000","last_fullsized":"2021-02-10T04:36:52.428160+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'20","last_scrub_stamp":"2021-02-10T04:28:50.696124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:50.696124+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":53,"num_read_kb":36,"num_write":27,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"242","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.200818+0000","last_change":"2021-02-10T04:33:05.151251+0000","last_active":"2021-02-10T04:36:55.200818+0000","last_peered":"2021-02-10T04:36:55.200818+0000","last_clean":"2021-02-10T04:36:55.200818+0000","last_became_active":"2021-02-10T04:33:05.150882+0000","last_became_peered":"2021-02-10T04:33:05.150882+0000","last_unstale":"2021-02-10T04:36:55.200818+0000","last_undegraded":"2021-02-10T04:36:55.200818+0000","last_fullsized":"2021-02-10T04:36:55.200818+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:31.102558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:31.102558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"562","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.200651+0000","last_change":"2021-02-10T04:29:35.805411+0000","last_active":"2021-02-10T04:36:54.200651+0000","last_peered":"2021-02-10T04:36:54.200651+0000","last_clean":"2021-02-10T04:36:54.200651+0000","last_became_active":"2021-02-10T04:25:51.016118+0000","last_became_peered":"2021-02-10T04:25:51.016118+0000","last_unstale":"2021-02-10T04:36:54.200651+0000","last_undegraded":"2021-02-10T04:36:54.200651+0000","last_fullsized":"2021-02-10T04:36:54.200651+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:35.805326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.805326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"557","reported_epoch":"595","state":"active+clean","last_fresh":"2021-02-10T04:36:52.427625+0000","last_change":"2021-02-10T04:28:48.700480+0000","last_active":"2021-02-10T04:36:52.427625+0000","last_peered":"2021-02-10T04:36:52.427625+0000","last_clean":"2021-02-10T04:36:52.427625+0000","last_became_active":"2021-02-10T04:25:53.038249+0000","last_became_peered":"2021-02-10T04:25:53.038249+0000","last_unstale":"2021-02-10T04:36:52.427625+0000","last_undegraded":"2021-02-10T04:36:52.427625+0000","last_fullsized":"2021-02-10T04:36:52.427625+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:48.700418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:48.700418+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"259'74","reported_seq":"1785","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.199064+0000","last_change":"2021-02-10T04:36:54.199064+0000","last_active":"2021-02-10T04:36:54.199064+0000","last_peered":"2021-02-10T04:36:54.199064+0000","last_clean":"2021-02-10T04:36:54.199064+0000","last_became_active":"2021-02-10T04:28:25.155557+0000","last_became_peered":"2021-02-10T04:28:25.155557+0000","last_unstale":"2021-02-10T04:36:54.199064+0000","last_undegraded":"2021-02-10T04:36:54.199064+0000","last_fullsized":"2021-02-10T04:36:54.199064+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'62","last_scrub_stamp":"2021-02-10T04:30:10.905905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:10.905905+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2437048,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":48,"num_read_kb":56455,"num_write":38,"num_write_kb":14043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"},{"start":"d4","length":"8"},{"start":"dd","length":"1"},{"start":"e1","length":"3"},{"start":"e9","length":"2"},{"start":"ed","length":"4"},{"start":"f2","length":"1"},{"start":"f4","length":"5"},{"start":"fb","length":"1"},{"start":"fd","length":"4"},{"start":"103","length":"1"},{"start":"106","length":"4"},{"start":"10c","length":"1"},{"start":"10f","length":"6"},{"start":"116","length":"2"},{"start":"11c","length":"3"},{"start":"121","length":"1"},{"start":"123","length":"2"},{"start":"126","length":"1"},{"start":"128","length":"5"},{"start":"12e","length":"3"},{"start":"133","length":"4"},{"start":"139","length":"2"},{"start":"13c","length":"3"},{"start":"140","length":"1"},{"start":"143","length":"1"},{"start":"146","length":"1"},{"start":"148","length":"1"},{"start":"14c","length":"1"},{"start":"14e","length":"1"},{"start":"155","length":"2"},{"start":"15a","length":"4"},{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"168","length":"2"},{"start":"16b","length":"1"},{"start":"16f","length":"5"},{"start":"178","length":"4"},{"start":"17f","length":"1"},{"start":"185","length":"2"},{"start":"18a","length":"2"},{"start":"18e","length":"3"},{"start":"194","length":"4"},{"start":"19b","length":"5"},{"start":"1a1","length":"2"},{"start":"1a4","length":"1"},{"start":"1a8","length":"1"},{"start":"1ab","length":"1"},{"start":"1ae","length":"1"},{"start":"1b1","length":"1"},{"start":"1b7","length":"1"},{"start":"1b9","length":"2"},{"start":"1bc","length":"1"},{"start":"1bf","length":"2"},{"start":"1c8","length":"1"},{"start":"1d1","length":"1"},{"start":"1d5","length":"1"},{"start":"1d8","length":"2"},{"start":"1df","length":"1"},{"start":"1e1","length":"1"},{"start":"1e4","length":"1"}]},{"pgid":"4.a","version":"538'45","reported_seq":"638","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.422355+0000","last_change":"2021-02-10T04:29:26.671051+0000","last_active":"2021-02-10T04:36:51.422355+0000","last_peered":"2021-02-10T04:36:51.422355+0000","last_clean":"2021-02-10T04:36:51.422355+0000","last_became_active":"2021-02-10T04:25:49.015913+0000","last_became_peered":"2021-02-10T04:25:49.015913+0000","last_unstale":"2021-02-10T04:36:51.422355+0000","last_undegraded":"2021-02-10T04:36:51.422355+0000","last_fullsized":"2021-02-10T04:36:51.422355+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'29","last_scrub_stamp":"2021-02-10T04:29:26.670950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.670950+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":80,"num_read_kb":60,"num_write":46,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"62'2","reported_seq":"700","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.199971+0000","last_change":"2021-02-10T04:28:33.618291+0000","last_active":"2021-02-10T04:36:55.199971+0000","last_peered":"2021-02-10T04:36:55.199971+0000","last_clean":"2021-02-10T04:36:55.199971+0000","last_became_active":"2021-02-10T04:25:34.331199+0000","last_became_peered":"2021-02-10T04:25:34.331199+0000","last_unstale":"2021-02-10T04:36:55.199971+0000","last_undegraded":"2021-02-10T04:36:55.199971+0000","last_fullsized":"2021-02-10T04:36:55.199971+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2021-02-10T04:28:33.618170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:33.618170+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"557","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.199536+0000","last_change":"2021-02-10T04:28:37.645921+0000","last_active":"2021-02-10T04:36:54.199536+0000","last_peered":"2021-02-10T04:36:54.199536+0000","last_clean":"2021-02-10T04:36:54.199536+0000","last_became_active":"2021-02-10T04:25:51.025882+0000","last_became_peered":"2021-02-10T04:25:51.025882+0000","last_unstale":"2021-02-10T04:36:54.199536+0000","last_undegraded":"2021-02-10T04:36:54.199536+0000","last_fullsized":"2021-02-10T04:36:54.199536+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:37.645857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:37.645857+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"483","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.198342+0000","last_change":"2021-02-10T04:29:24.731524+0000","last_active":"2021-02-10T04:36:54.198342+0000","last_peered":"2021-02-10T04:36:54.198342+0000","last_clean":"2021-02-10T04:36:54.198342+0000","last_became_active":"2021-02-10T04:29:07.162878+0000","last_became_peered":"2021-02-10T04:29:07.162878+0000","last_unstale":"2021-02-10T04:36:54.198342+0000","last_undegraded":"2021-02-10T04:36:54.198342+0000","last_fullsized":"2021-02-10T04:36:54.198342+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.731458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.731458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"595'348","reported_seq":"2072","reported_epoch":"595","state":"active+clean+snaptrim_wait","last_fresh":"2021-02-10T04:36:53.591561+0000","last_change":"2021-02-10T04:36:53.591561+0000","last_active":"2021-02-10T04:36:53.591561+0000","last_peered":"2021-02-10T04:36:53.591561+0000","last_clean":"2021-02-10T04:36:53.591561+0000","last_became_active":"2021-02-10T04:28:25.150246+0000","last_became_peered":"2021-02-10T04:28:25.150246+0000","last_unstale":"2021-02-10T04:36:53.591561+0000","last_undegraded":"2021-02-10T04:36:53.591561+0000","last_fullsized":"2021-02-10T04:36:53.591561+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'63","last_scrub_stamp":"2021-02-10T04:29:39.700115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:29:39.700115+0000","log_size":348,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":29365265,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":2,"num_read":149,"num_read_kb":127190,"num_write":99,"num_write_kb":42259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"},{"start":"d4","length":"8"},{"start":"dd","length":"1"},{"start":"e1","length":"3"},{"start":"e9","length":"2"},{"start":"ed","length":"4"},{"start":"f2","length":"1"},{"start":"f4","length":"5"},{"start":"fb","length":"1"},{"start":"fd","length":"4"},{"start":"103","length":"1"},{"start":"106","length":"4"},{"start":"10c","length":"1"},{"start":"10f","length":"6"},{"start":"116","length":"2"},{"start":"11c","length":"3"},{"start":"121","length":"1"},{"start":"123","length":"2"},{"start":"126","length":"1"},{"start":"128","length":"5"},{"start":"12e","length":"3"},{"start":"133","length":"4"},{"start":"139","length":"2"},{"start":"13c","length":"3"},{"start":"140","length":"1"},{"start":"143","length":"1"},{"start":"146","length":"1"},{"start":"148","length":"1"},{"start":"14c","length":"1"},{"start":"14e","length":"1"},{"start":"155","length":"2"},{"start":"15a","length":"4"},{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"168","length":"2"},{"start":"16b","length":"1"},{"start":"16f","length":"5"},{"start":"178","length":"4"},{"start":"17f","length":"1"},{"start":"185","length":"2"},{"start":"18a","length":"2"},{"start":"18e","length":"3"},{"start":"194","length":"4"},{"start":"19b","length":"5"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a8","length":"1"},{"start":"1ab","length":"1"},{"start":"1ae","length":"1"},{"start":"1b1","length":"1"},{"start":"1b7","length":"1"},{"start":"1b9","length":"2"},{"start":"1bc","length":"1"},{"start":"1bf","length":"2"},{"start":"1c8","length":"1"},{"start":"1d1","length":"1"},{"start":"1d5","length":"1"},{"start":"1d8","length":"1"},{"start":"1df","length":"1"}]},{"pgid":"4.d","version":"538'47","reported_seq":"630","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.200958+0000","last_change":"2021-02-10T04:28:49.515083+0000","last_active":"2021-02-10T04:36:55.200958+0000","last_peered":"2021-02-10T04:36:55.200958+0000","last_clean":"2021-02-10T04:36:55.200958+0000","last_became_active":"2021-02-10T04:25:49.011699+0000","last_became_peered":"2021-02-10T04:25:49.011699+0000","last_unstale":"2021-02-10T04:36:55.200958+0000","last_undegraded":"2021-02-10T04:36:55.200958+0000","last_fullsized":"2021-02-10T04:36:55.200958+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'33","last_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:28:49.514984+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":62,"num_read_kb":44,"num_write":36,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"479","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.199727+0000","last_change":"2021-02-10T04:29:07.146266+0000","last_active":"2021-02-10T04:36:55.199727+0000","last_peered":"2021-02-10T04:36:55.199727+0000","last_clean":"2021-02-10T04:36:55.199727+0000","last_became_active":"2021-02-10T04:29:07.145616+0000","last_became_peered":"2021-02-10T04:29:07.145616+0000","last_unstale":"2021-02-10T04:36:55.199727+0000","last_undegraded":"2021-02-10T04:36:55.199727+0000","last_fullsized":"2021-02-10T04:36:55.199727+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:44.410817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:44.410817+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"242","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.200047+0000","last_change":"2021-02-10T04:33:05.147166+0000","last_active":"2021-02-10T04:36:55.200047+0000","last_peered":"2021-02-10T04:36:55.200047+0000","last_clean":"2021-02-10T04:36:55.200047+0000","last_became_active":"2021-02-10T04:33:05.146783+0000","last_became_peered":"2021-02-10T04:33:05.146783+0000","last_unstale":"2021-02-10T04:36:55.200047+0000","last_undegraded":"2021-02-10T04:36:55.200047+0000","last_fullsized":"2021-02-10T04:36:55.200047+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:35.958255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.958255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"555","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.199537+0000","last_change":"2021-02-10T04:28:32.623869+0000","last_active":"2021-02-10T04:36:54.199537+0000","last_peered":"2021-02-10T04:36:54.199537+0000","last_clean":"2021-02-10T04:36:54.199537+0000","last_became_active":"2021-02-10T04:25:53.038218+0000","last_became_peered":"2021-02-10T04:25:53.038218+0000","last_unstale":"2021-02-10T04:36:54.199537+0000","last_undegraded":"2021-02-10T04:36:54.199537+0000","last_fullsized":"2021-02-10T04:36:54.199537+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:32.623791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:32.623791+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"386'110","reported_seq":"1820","reported_epoch":"596","state":"active+clean+scrubbing","last_fresh":"2021-02-10T04:36:54.199420+0000","last_change":"2021-02-10T04:33:57.382223+0000","last_active":"2021-02-10T04:36:54.199420+0000","last_peered":"2021-02-10T04:36:54.199420+0000","last_clean":"2021-02-10T04:36:54.199420+0000","last_became_active":"2021-02-10T04:28:25.157931+0000","last_became_peered":"2021-02-10T04:28:25.157931+0000","last_unstale":"2021-02-10T04:36:54.199420+0000","last_undegraded":"2021-02-10T04:36:54.199420+0000","last_fullsized":"2021-02-10T04:36:54.199420+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":110,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":189,"stat_sum":{"num_bytes":12437781,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":78,"num_read_kb":76873,"num_write":73,"num_write_kb":20689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"497","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.419700+0000","last_change":"2021-02-10T04:29:37.336179+0000","last_active":"2021-02-10T04:36:51.419700+0000","last_peered":"2021-02-10T04:36:51.419700+0000","last_clean":"2021-02-10T04:36:51.419700+0000","last_became_active":"2021-02-10T04:28:49.046469+0000","last_became_peered":"2021-02-10T04:28:49.046469+0000","last_unstale":"2021-02-10T04:36:51.419700+0000","last_undegraded":"2021-02-10T04:36:51.419700+0000","last_fullsized":"2021-02-10T04:36:51.419700+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:37.336092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:37.336092+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"573","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.197134+0000","last_change":"2021-02-10T04:33:51.858490+0000","last_active":"2021-02-10T04:36:54.197134+0000","last_peered":"2021-02-10T04:36:54.197134+0000","last_clean":"2021-02-10T04:36:54.197134+0000","last_became_active":"2021-02-10T04:29:07.159274+0000","last_became_peered":"2021-02-10T04:29:07.159274+0000","last_unstale":"2021-02-10T04:36:54.197134+0000","last_undegraded":"2021-02-10T04:36:54.197134+0000","last_fullsized":"2021-02-10T04:36:54.197134+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:51.858386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:33:51.858386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"538'23","reported_seq":"597","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.200127+0000","last_change":"2021-02-10T04:28:41.427119+0000","last_active":"2021-02-10T04:36:55.200127+0000","last_peered":"2021-02-10T04:36:55.200127+0000","last_clean":"2021-02-10T04:36:55.200127+0000","last_became_active":"2021-02-10T04:25:49.001725+0000","last_became_peered":"2021-02-10T04:25:49.001725+0000","last_unstale":"2021-02-10T04:36:55.200127+0000","last_undegraded":"2021-02-10T04:36:55.200127+0000","last_fullsized":"2021-02-10T04:36:55.200127+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'13","last_scrub_stamp":"2021-02-10T04:28:41.427050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:41.427050+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":28,"num_write":23,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"245","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.202992+0000","last_change":"2021-02-10T04:33:05.168734+0000","last_active":"2021-02-10T04:36:54.202992+0000","last_peered":"2021-02-10T04:36:54.202992+0000","last_clean":"2021-02-10T04:36:54.202992+0000","last_became_active":"2021-02-10T04:33:05.168314+0000","last_became_peered":"2021-02-10T04:33:05.168314+0000","last_unstale":"2021-02-10T04:36:54.202992+0000","last_undegraded":"2021-02-10T04:36:54.202992+0000","last_fullsized":"2021-02-10T04:36:54.202992+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:18.107867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:18.107867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"58'2","reported_seq":"692","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.200561+0000","last_change":"2021-02-10T04:28:35.648670+0000","last_active":"2021-02-10T04:36:54.200561+0000","last_peered":"2021-02-10T04:36:54.200561+0000","last_clean":"2021-02-10T04:36:54.200561+0000","last_became_active":"2021-02-10T04:25:51.025268+0000","last_became_peered":"2021-02-10T04:25:51.025268+0000","last_unstale":"2021-02-10T04:36:54.200561+0000","last_undegraded":"2021-02-10T04:36:54.200561+0000","last_fullsized":"2021-02-10T04:36:54.200561+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:28:35.648602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:35.648602+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"557","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.202659+0000","last_change":"2021-02-10T04:29:38.873167+0000","last_active":"2021-02-10T04:36:55.202659+0000","last_peered":"2021-02-10T04:36:55.202659+0000","last_clean":"2021-02-10T04:36:55.202659+0000","last_became_active":"2021-02-10T04:25:53.043027+0000","last_became_peered":"2021-02-10T04:25:53.043027+0000","last_unstale":"2021-02-10T04:36:55.202659+0000","last_undegraded":"2021-02-10T04:36:55.202659+0000","last_fullsized":"2021-02-10T04:36:55.202659+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:38.873102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.873102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"595'528","reported_seq":"4226","reported_epoch":"595","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:36:53.594301+0000","last_change":"2021-02-10T04:36:53.591730+0000","last_active":"2021-02-10T04:36:53.594301+0000","last_peered":"2021-02-10T04:36:53.594301+0000","last_clean":"2021-02-10T04:36:53.594301+0000","last_became_active":"2021-02-10T04:28:25.149226+0000","last_became_peered":"2021-02-10T04:28:25.149226+0000","last_unstale":"2021-02-10T04:36:53.594301+0000","last_undegraded":"2021-02-10T04:36:53.594301+0000","last_fullsized":"2021-02-10T04:36:53.594301+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'236","last_scrub_stamp":"2021-02-10T04:33:50.351534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:50.351534+0000","log_size":528,"ondisk_log_size":528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"stat_sum":{"num_bytes":40925777,"num_objects":17,"num_object_clones":13,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":375,"num_read_kb":469640,"num_write":184,"num_write_kb":63750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"},{"start":"d4","length":"8"},{"start":"dd","length":"1"},{"start":"e1","length":"3"},{"start":"e9","length":"2"},{"start":"ed","length":"4"},{"start":"f2","length":"1"},{"start":"f4","length":"5"},{"start":"fb","length":"1"},{"start":"fd","length":"4"},{"start":"103","length":"1"},{"start":"106","length":"4"},{"start":"10c","length":"1"},{"start":"10f","length":"6"},{"start":"116","length":"2"},{"start":"11c","length":"3"},{"start":"121","length":"1"},{"start":"123","length":"2"},{"start":"126","length":"1"},{"start":"128","length":"5"},{"start":"12e","length":"3"},{"start":"133","length":"4"},{"start":"139","length":"2"},{"start":"13c","length":"3"},{"start":"140","length":"1"},{"start":"143","length":"1"},{"start":"146","length":"1"},{"start":"148","length":"1"},{"start":"14c","length":"1"},{"start":"14e","length":"1"},{"start":"155","length":"2"},{"start":"15a","length":"4"},{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"168","length":"2"},{"start":"16b","length":"1"},{"start":"16f","length":"5"},{"start":"178","length":"4"},{"start":"17f","length":"1"},{"start":"185","length":"2"},{"start":"18a","length":"2"},{"start":"18e","length":"3"},{"start":"194","length":"4"},{"start":"19b","length":"2"},{"start":"19e","length":"2"},{"start":"1a1","length":"2"},{"start":"1a4","length":"1"},{"start":"1a8","length":"1"},{"start":"1ab","length":"1"},{"start":"1ae","length":"1"},{"start":"1b1","length":"1"},{"start":"1b7","length":"1"},{"start":"1b9","length":"2"},{"start":"1bf","length":"1"}]},{"pgid":"1.a","version":"0'0","reported_seq":"497","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.420685+0000","last_change":"2021-02-10T04:29:29.384240+0000","last_active":"2021-02-10T04:36:51.420685+0000","last_peered":"2021-02-10T04:36:51.420685+0000","last_clean":"2021-02-10T04:36:51.420685+0000","last_became_active":"2021-02-10T04:28:49.050935+0000","last_became_peered":"2021-02-10T04:28:49.050935+0000","last_unstale":"2021-02-10T04:36:51.420685+0000","last_undegraded":"2021-02-10T04:36:51.420685+0000","last_fullsized":"2021-02-10T04:36:51.420685+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.384170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.384170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"538'29","reported_seq":"301","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.201355+0000","last_change":"2021-02-10T04:33:05.154998+0000","last_active":"2021-02-10T04:36:54.201355+0000","last_peered":"2021-02-10T04:36:54.201355+0000","last_clean":"2021-02-10T04:36:54.201355+0000","last_became_active":"2021-02-10T04:33:05.149026+0000","last_became_peered":"2021-02-10T04:33:05.149026+0000","last_unstale":"2021-02-10T04:36:54.201355+0000","last_undegraded":"2021-02-10T04:36:54.201355+0000","last_fullsized":"2021-02-10T04:36:54.201355+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'17","last_scrub_stamp":"2021-02-10T04:29:37.933206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:37.933206+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":49,"num_read_kb":36,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"305","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.204352+0000","last_change":"2021-02-10T04:33:05.169654+0000","last_active":"2021-02-10T04:36:54.204352+0000","last_peered":"2021-02-10T04:36:54.204352+0000","last_clean":"2021-02-10T04:36:54.204352+0000","last_became_active":"2021-02-10T04:33:05.169289+0000","last_became_peered":"2021-02-10T04:33:05.169289+0000","last_unstale":"2021-02-10T04:36:54.204352+0000","last_undegraded":"2021-02-10T04:36:54.204352+0000","last_fullsized":"2021-02-10T04:36:54.204352+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_clean_scrub_stamp":"2021-02-10T04:28:39.128781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"78'3","reported_seq":"695","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.202768+0000","last_change":"2021-02-10T04:28:43.509446+0000","last_active":"2021-02-10T04:36:55.202768+0000","last_peered":"2021-02-10T04:36:55.202768+0000","last_clean":"2021-02-10T04:36:55.202768+0000","last_became_active":"2021-02-10T04:25:51.025218+0000","last_became_peered":"2021-02-10T04:25:51.025218+0000","last_unstale":"2021-02-10T04:36:55.202768+0000","last_undegraded":"2021-02-10T04:36:55.202768+0000","last_fullsized":"2021-02-10T04:36:55.202768+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:28:43.509353+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:28:43.509353+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"554","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.422277+0000","last_change":"2021-02-10T04:28:45.481538+0000","last_active":"2021-02-10T04:36:51.422277+0000","last_peered":"2021-02-10T04:36:51.422277+0000","last_clean":"2021-02-10T04:36:51.422277+0000","last_became_active":"2021-02-10T04:25:53.042697+0000","last_became_peered":"2021-02-10T04:25:53.042697+0000","last_unstale":"2021-02-10T04:36:51.422277+0000","last_undegraded":"2021-02-10T04:36:51.422277+0000","last_fullsized":"2021-02-10T04:36:51.422277+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:45.481472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:28:45.481472+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"595'564","reported_seq":"3252","reported_epoch":"595","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:36:53.603561+0000","last_change":"2021-02-10T04:36:53.603561+0000","last_active":"2021-02-10T04:36:53.603561+0000","last_peered":"2021-02-10T04:36:53.603561+0000","last_clean":"2021-02-10T04:36:53.603561+0000","last_became_active":"2021-02-10T04:28:25.148500+0000","last_became_peered":"2021-02-10T04:28:25.148500+0000","last_unstale":"2021-02-10T04:36:53.603561+0000","last_undegraded":"2021-02-10T04:36:53.603561+0000","last_fullsized":"2021-02-10T04:36:53.603561+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'110","last_scrub_stamp":"2021-02-10T04:30:18.463782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:18.463782+0000","log_size":564,"ondisk_log_size":564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"stat_sum":{"num_bytes":37871602,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":374,"num_read_kb":465088,"num_write":215,"num_write_kb":84347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"},{"start":"d4","length":"8"},{"start":"dd","length":"1"},{"start":"e1","length":"3"},{"start":"e9","length":"2"},{"start":"ed","length":"4"},{"start":"f2","length":"1"},{"start":"f4","length":"5"},{"start":"fb","length":"1"},{"start":"fd","length":"4"},{"start":"103","length":"1"},{"start":"106","length":"4"},{"start":"10c","length":"1"},{"start":"10f","length":"6"},{"start":"116","length":"2"},{"start":"11c","length":"3"},{"start":"121","length":"1"},{"start":"123","length":"2"},{"start":"126","length":"1"},{"start":"128","length":"5"},{"start":"12e","length":"3"},{"start":"133","length":"4"},{"start":"139","length":"2"},{"start":"13c","length":"3"},{"start":"140","length":"1"},{"start":"143","length":"1"},{"start":"146","length":"1"},{"start":"148","length":"1"},{"start":"14c","length":"1"},{"start":"14e","length":"1"},{"start":"155","length":"2"},{"start":"15a","length":"4"},{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"168","length":"2"},{"start":"16b","length":"1"},{"start":"16f","length":"5"},{"start":"178","length":"4"},{"start":"17f","length":"1"},{"start":"185","length":"2"},{"start":"18a","length":"2"},{"start":"18e","length":"3"},{"start":"194","length":"4"},{"start":"19b","length":"5"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a8","length":"1"},{"start":"1ab","length":"1"},{"start":"1ae","length":"1"},{"start":"1b1","length":"1"},{"start":"1b7","length":"1"},{"start":"1b9","length":"2"},{"start":"1bc","length":"1"},{"start":"1bf","length":"2"}]},{"pgid":"4.e","version":"538'39","reported_seq":"625","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.200104+0000","last_change":"2021-02-10T04:28:11.693702+0000","last_active":"2021-02-10T04:36:55.200104+0000","last_peered":"2021-02-10T04:36:55.200104+0000","last_clean":"2021-02-10T04:36:55.200104+0000","last_became_active":"2021-02-10T04:25:49.011093+0000","last_became_peered":"2021-02-10T04:25:49.011093+0000","last_unstale":"2021-02-10T04:36:55.200104+0000","last_undegraded":"2021-02-10T04:36:55.200104+0000","last_fullsized":"2021-02-10T04:36:55.200104+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'31","last_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:28:11.693616+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":56,"num_read_kb":40,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"540","reported_epoch":"595","state":"active+clean","last_fresh":"2021-02-10T04:36:52.427267+0000","last_change":"2021-02-10T04:29:31.683314+0000","last_active":"2021-02-10T04:36:52.427267+0000","last_peered":"2021-02-10T04:36:52.427267+0000","last_clean":"2021-02-10T04:36:52.427267+0000","last_became_active":"2021-02-10T04:29:07.160593+0000","last_became_peered":"2021-02-10T04:29:07.160593+0000","last_unstale":"2021-02-10T04:36:52.427267+0000","last_undegraded":"2021-02-10T04:36:52.427267+0000","last_fullsized":"2021-02-10T04:36:52.427267+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:31.683213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.683213+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"561","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.423075+0000","last_change":"2021-02-10T04:30:10.487631+0000","last_active":"2021-02-10T04:36:51.423075+0000","last_peered":"2021-02-10T04:36:51.423075+0000","last_clean":"2021-02-10T04:36:51.423075+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:36:51.423075+0000","last_undegraded":"2021-02-10T04:36:51.423075+0000","last_fullsized":"2021-02-10T04:36:51.423075+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:10.487528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:30:10.487528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"555","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.202060+0000","last_change":"2021-02-10T04:33:55.849675+0000","last_active":"2021-02-10T04:36:54.202060+0000","last_peered":"2021-02-10T04:36:54.202060+0000","last_clean":"2021-02-10T04:36:54.202060+0000","last_became_active":"2021-02-10T04:25:53.148054+0000","last_became_peered":"2021-02-10T04:25:53.148054+0000","last_unstale":"2021-02-10T04:36:54.202060+0000","last_undegraded":"2021-02-10T04:36:54.202060+0000","last_fullsized":"2021-02-10T04:36:54.202060+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:55.849582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:33:55.849582+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"594'543","reported_seq":"2203","reported_epoch":"595","state":"active+clean+snaptrim_wait","last_fresh":"2021-02-10T04:36:53.603433+0000","last_change":"2021-02-10T04:36:53.603520+0000","last_active":"2021-02-10T04:36:53.603433+0000","last_peered":"2021-02-10T04:36:53.603433+0000","last_clean":"2021-02-10T04:36:53.603433+0000","last_became_active":"2021-02-10T04:28:25.152030+0000","last_became_peered":"2021-02-10T04:28:25.152030+0000","last_unstale":"2021-02-10T04:36:53.603433+0000","last_undegraded":"2021-02-10T04:36:53.603433+0000","last_fullsized":"2021-02-10T04:36:53.603433+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'283","last_scrub_stamp":"2021-02-10T04:33:53.834627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:53.834627+0000","log_size":543,"ondisk_log_size":543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"stat_sum":{"num_bytes":41477713,"num_objects":19,"num_object_clones":15,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":326,"num_read_kb":340700,"num_write":230,"num_write_kb":101637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"},{"start":"d4","length":"8"},{"start":"dd","length":"1"},{"start":"e1","length":"3"},{"start":"e9","length":"2"},{"start":"ed","length":"4"},{"start":"f2","length":"1"},{"start":"f4","length":"5"},{"start":"fb","length":"1"},{"start":"fd","length":"4"},{"start":"103","length":"1"},{"start":"106","length":"4"},{"start":"10c","length":"1"},{"start":"10f","length":"6"},{"start":"116","length":"2"},{"start":"11c","length":"3"},{"start":"121","length":"1"},{"start":"123","length":"2"},{"start":"126","length":"1"},{"start":"128","length":"5"},{"start":"12e","length":"3"},{"start":"133","length":"4"},{"start":"139","length":"2"},{"start":"13c","length":"3"},{"start":"140","length":"1"},{"start":"143","length":"1"},{"start":"146","length":"1"},{"start":"148","length":"1"},{"start":"14c","length":"1"},{"start":"14e","length":"1"},{"start":"155","length":"2"},{"start":"15a","length":"4"},{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"168","length":"2"},{"start":"16b","length":"1"},{"start":"16f","length":"5"},{"start":"178","length":"4"},{"start":"17f","length":"1"},{"start":"185","length":"2"},{"start":"18a","length":"2"},{"start":"18e","length":"3"},{"start":"194","length":"4"},{"start":"19b","length":"5"},{"start":"1a1","length":"2"},{"start":"1a4","length":"1"},{"start":"1a8","length":"1"},{"start":"1ab","length":"1"},{"start":"1ae","length":"1"},{"start":"1b1","length":"1"},{"start":"1b7","length":"1"},{"start":"1b9","length":"2"}]},{"pgid":"5.0","version":"158'3","reported_seq":"693","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.197350+0000","last_change":"2021-02-10T04:29:34.388978+0000","last_active":"2021-02-10T04:36:54.197350+0000","last_peered":"2021-02-10T04:36:54.197350+0000","last_clean":"2021-02-10T04:36:54.197350+0000","last_became_active":"2021-02-10T04:25:51.031906+0000","last_became_peered":"2021-02-10T04:25:51.031906+0000","last_unstale":"2021-02-10T04:36:54.197350+0000","last_undegraded":"2021-02-10T04:36:54.197350+0000","last_fullsized":"2021-02-10T04:36:54.197350+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'3","last_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_deep_scrub":"158'3","last_deep_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_clean_scrub_stamp":"2021-02-10T04:29:34.388912+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"538'40","reported_seq":"630","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.199182+0000","last_change":"2021-02-10T04:29:36.813190+0000","last_active":"2021-02-10T04:36:55.199182+0000","last_peered":"2021-02-10T04:36:55.199182+0000","last_clean":"2021-02-10T04:36:55.199182+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:36:55.199182+0000","last_undegraded":"2021-02-10T04:36:55.199182+0000","last_fullsized":"2021-02-10T04:36:55.199182+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:29:36.813083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.813083+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":72,"num_read_kb":52,"num_write":40,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"597","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.421497+0000","last_change":"2021-02-10T04:29:28.632178+0000","last_active":"2021-02-10T04:36:51.421497+0000","last_peered":"2021-02-10T04:36:51.421497+0000","last_clean":"2021-02-10T04:36:51.421497+0000","last_became_active":"2021-02-10T04:28:39.041833+0000","last_became_peered":"2021-02-10T04:28:39.041833+0000","last_unstale":"2021-02-10T04:36:51.421497+0000","last_undegraded":"2021-02-10T04:36:51.421497+0000","last_fullsized":"2021-02-10T04:36:51.421497+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.632085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.632085+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"481","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.424579+0000","last_change":"2021-02-10T04:29:24.658465+0000","last_active":"2021-02-10T04:36:51.424579+0000","last_peered":"2021-02-10T04:36:51.424579+0000","last_clean":"2021-02-10T04:36:51.424579+0000","last_became_active":"2021-02-10T04:29:07.148527+0000","last_became_peered":"2021-02-10T04:29:07.148527+0000","last_unstale":"2021-02-10T04:36:51.424579+0000","last_undegraded":"2021-02-10T04:36:51.424579+0000","last_fullsized":"2021-02-10T04:36:51.424579+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.658374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.658374+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"595'518","reported_seq":"2692","reported_epoch":"596","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:36:54.197451+0000","last_change":"2021-02-10T04:36:53.773635+0000","last_active":"2021-02-10T04:36:54.197451+0000","last_peered":"2021-02-10T04:36:54.197451+0000","last_clean":"2021-02-10T04:36:54.197451+0000","last_became_active":"2021-02-10T04:28:25.157306+0000","last_became_peered":"2021-02-10T04:28:25.157306+0000","last_unstale":"2021-02-10T04:36:54.197451+0000","last_undegraded":"2021-02-10T04:36:54.197451+0000","last_fullsized":"2021-02-10T04:36:54.197451+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":518,"ondisk_log_size":518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"stat_sum":{"num_bytes":21918629,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":312,"num_read_kb":299964,"num_write":204,"num_write_kb":60127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"},{"start":"d4","length":"8"},{"start":"dd","length":"1"},{"start":"e1","length":"3"},{"start":"e9","length":"2"},{"start":"ed","length":"4"},{"start":"f2","length":"1"},{"start":"f4","length":"5"},{"start":"fb","length":"1"},{"start":"fd","length":"4"},{"start":"103","length":"1"},{"start":"106","length":"4"},{"start":"10c","length":"1"},{"start":"10f","length":"6"},{"start":"116","length":"2"},{"start":"11c","length":"3"},{"start":"121","length":"1"},{"start":"123","length":"2"},{"start":"126","length":"1"},{"start":"128","length":"5"},{"start":"12e","length":"3"},{"start":"133","length":"4"},{"start":"139","length":"2"},{"start":"13c","length":"3"},{"start":"140","length":"1"},{"start":"143","length":"1"},{"start":"146","length":"1"},{"start":"148","length":"1"},{"start":"14c","length":"1"},{"start":"14e","length":"1"},{"start":"155","length":"2"},{"start":"15a","length":"4"},{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"168","length":"2"},{"start":"16b","length":"1"},{"start":"16f","length":"5"},{"start":"178","length":"4"},{"start":"17f","length":"1"},{"start":"185","length":"2"},{"start":"18a","length":"2"},{"start":"18e","length":"3"},{"start":"194","length":"4"},{"start":"19b","length":"2"},{"start":"19e","length":"2"},{"start":"1a1","length":"2"},{"start":"1a4","length":"1"},{"start":"1a8","length":"1"},{"start":"1ab","length":"1"},{"start":"1ae","length":"1"},{"start":"1b1","length":"1"},{"start":"1b7","length":"1"},{"start":"1b9","length":"2"},{"start":"1bc","length":"1"},{"start":"1bf","length":"2"},{"start":"1c8","length":"1"},{"start":"1d1","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":"499","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.202151+0000","last_change":"2021-02-10T04:30:12.452072+0000","last_active":"2021-02-10T04:36:54.202151+0000","last_peered":"2021-02-10T04:36:54.202151+0000","last_clean":"2021-02-10T04:36:54.202151+0000","last_became_active":"2021-02-10T04:28:49.048525+0000","last_became_peered":"2021-02-10T04:28:49.048525+0000","last_unstale":"2021-02-10T04:36:54.202151+0000","last_undegraded":"2021-02-10T04:36:54.202151+0000","last_fullsized":"2021-02-10T04:36:54.202151+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:12.452000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:30:12.452000+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"559","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.199364+0000","last_change":"2021-02-10T04:29:40.859352+0000","last_active":"2021-02-10T04:36:55.199364+0000","last_peered":"2021-02-10T04:36:55.199364+0000","last_clean":"2021-02-10T04:36:55.199364+0000","last_became_active":"2021-02-10T04:25:51.018706+0000","last_became_peered":"2021-02-10T04:25:51.018706+0000","last_unstale":"2021-02-10T04:36:55.199364+0000","last_undegraded":"2021-02-10T04:36:55.199364+0000","last_fullsized":"2021-02-10T04:36:55.199364+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:40.859274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:40.859274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"538'60","reported_seq":"661","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.197258+0000","last_change":"2021-02-10T04:29:28.398522+0000","last_active":"2021-02-10T04:36:54.197258+0000","last_peered":"2021-02-10T04:36:54.197258+0000","last_clean":"2021-02-10T04:36:54.197258+0000","last_became_active":"2021-02-10T04:25:48.997481+0000","last_became_peered":"2021-02-10T04:25:48.997481+0000","last_unstale":"2021-02-10T04:36:54.197258+0000","last_undegraded":"2021-02-10T04:36:54.197258+0000","last_fullsized":"2021-02-10T04:36:54.197258+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'48","last_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_deep_scrub":"152'48","last_deep_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.398429+0000","log_size":60,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":88,"num_read_kb":66,"num_write":48,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"252","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.423915+0000","last_change":"2021-02-10T04:33:05.142730+0000","last_active":"2021-02-10T04:36:51.423915+0000","last_peered":"2021-02-10T04:36:51.423915+0000","last_clean":"2021-02-10T04:36:51.423915+0000","last_became_active":"2021-02-10T04:33:05.141066+0000","last_became_peered":"2021-02-10T04:33:05.141066+0000","last_unstale":"2021-02-10T04:36:51.423915+0000","last_undegraded":"2021-02-10T04:36:51.423915+0000","last_fullsized":"2021-02-10T04:36:51.423915+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:28:35.010709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:35.010709+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":423,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"557","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.199635+0000","last_change":"2021-02-10T04:28:05.641668+0000","last_active":"2021-02-10T04:36:55.199635+0000","last_peered":"2021-02-10T04:36:55.199635+0000","last_clean":"2021-02-10T04:36:55.199635+0000","last_became_active":"2021-02-10T04:25:53.045085+0000","last_became_peered":"2021-02-10T04:25:53.045085+0000","last_unstale":"2021-02-10T04:36:55.199635+0000","last_undegraded":"2021-02-10T04:36:55.199635+0000","last_fullsized":"2021-02-10T04:36:55.199635+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:05.641577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:05.641577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"595'949","reported_seq":"2531","reported_epoch":"596","state":"active+clean+snaptrim_wait","last_fresh":"2021-02-10T04:36:54.203685+0000","last_change":"2021-02-10T04:36:53.803797+0000","last_active":"2021-02-10T04:36:54.203685+0000","last_peered":"2021-02-10T04:36:54.203685+0000","last_clean":"2021-02-10T04:36:54.203685+0000","last_became_active":"2021-02-10T04:28:25.161622+0000","last_became_peered":"2021-02-10T04:28:25.161622+0000","last_unstale":"2021-02-10T04:36:54.203685+0000","last_undegraded":"2021-02-10T04:36:54.203685+0000","last_fullsized":"2021-02-10T04:36:54.203685+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":949,"ondisk_log_size":949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"stat_sum":{"num_bytes":55279671,"num_objects":23,"num_object_clones":19,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":550,"num_read_kb":579880,"num_write":413,"num_write_kb":146341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"},{"start":"d4","length":"8"},{"start":"dd","length":"1"},{"start":"e1","length":"3"},{"start":"e9","length":"2"},{"start":"ed","length":"4"},{"start":"f2","length":"1"},{"start":"f4","length":"5"},{"start":"fb","length":"1"},{"start":"fd","length":"4"},{"start":"103","length":"1"},{"start":"106","length":"4"},{"start":"10c","length":"1"},{"start":"10f","length":"6"},{"start":"116","length":"2"},{"start":"11c","length":"3"},{"start":"121","length":"1"},{"start":"123","length":"2"},{"start":"126","length":"1"},{"start":"128","length":"5"},{"start":"12e","length":"3"},{"start":"133","length":"4"},{"start":"139","length":"2"},{"start":"13c","length":"3"},{"start":"140","length":"1"},{"start":"143","length":"1"},{"start":"146","length":"1"},{"start":"148","length":"1"},{"start":"14c","length":"1"},{"start":"14e","length":"1"},{"start":"155","length":"2"},{"start":"15a","length":"4"},{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"168","length":"2"},{"start":"16b","length":"1"},{"start":"16f","length":"5"},{"start":"178","length":"4"},{"start":"17f","length":"1"},{"start":"185","length":"2"},{"start":"18a","length":"2"},{"start":"18e","length":"3"},{"start":"194","length":"4"},{"start":"19b","length":"5"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"1"},{"start":"1b1","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":"500","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.200345+0000","last_change":"2021-02-10T04:29:33.839409+0000","last_active":"2021-02-10T04:36:55.200345+0000","last_peered":"2021-02-10T04:36:55.200345+0000","last_clean":"2021-02-10T04:36:55.200345+0000","last_became_active":"2021-02-10T04:28:49.045505+0000","last_became_peered":"2021-02-10T04:28:49.045505+0000","last_unstale":"2021-02-10T04:36:55.200345+0000","last_undegraded":"2021-02-10T04:36:55.200345+0000","last_fullsized":"2021-02-10T04:36:55.200345+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:33.839335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:33.839335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"627","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.422542+0000","last_change":"2021-02-10T04:33:52.827328+0000","last_active":"2021-02-10T04:36:51.422542+0000","last_peered":"2021-02-10T04:36:51.422542+0000","last_clean":"2021-02-10T04:36:51.422542+0000","last_became_active":"2021-02-10T04:25:51.025636+0000","last_became_peered":"2021-02-10T04:25:51.025636+0000","last_unstale":"2021-02-10T04:36:51.422542+0000","last_undegraded":"2021-02-10T04:36:51.422542+0000","last_fullsized":"2021-02-10T04:36:51.422542+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:52.827257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:33:52.827257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"538'37","reported_seq":"629","reported_epoch":"595","state":"active+clean","last_fresh":"2021-02-10T04:36:52.428050+0000","last_change":"2021-02-10T04:28:36.754064+0000","last_active":"2021-02-10T04:36:52.428050+0000","last_peered":"2021-02-10T04:36:52.428050+0000","last_clean":"2021-02-10T04:36:52.428050+0000","last_became_active":"2021-02-10T04:25:49.002937+0000","last_became_peered":"2021-02-10T04:25:49.002937+0000","last_unstale":"2021-02-10T04:36:52.428050+0000","last_undegraded":"2021-02-10T04:36:52.428050+0000","last_fullsized":"2021-02-10T04:36:52.428050+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'27","last_scrub_stamp":"2021-02-10T04:28:36.753994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:36.753994+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":72,"num_read_kb":53,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"493","reported_epoch":"595","state":"active+clean","last_fresh":"2021-02-10T04:36:52.428872+0000","last_change":"2021-02-10T04:29:35.711635+0000","last_active":"2021-02-10T04:36:52.428872+0000","last_peered":"2021-02-10T04:36:52.428872+0000","last_clean":"2021-02-10T04:36:52.428872+0000","last_became_active":"2021-02-10T04:29:07.161341+0000","last_became_peered":"2021-02-10T04:29:07.161341+0000","last_unstale":"2021-02-10T04:36:52.428872+0000","last_undegraded":"2021-02-10T04:36:52.428872+0000","last_fullsized":"2021-02-10T04:36:52.428872+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_deep_scrub":"53'2","last_deep_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.711557+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"239","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.420283+0000","last_change":"2021-02-10T04:33:05.141227+0000","last_active":"2021-02-10T04:36:51.420283+0000","last_peered":"2021-02-10T04:36:51.420283+0000","last_clean":"2021-02-10T04:36:51.420283+0000","last_became_active":"2021-02-10T04:33:05.140699+0000","last_became_peered":"2021-02-10T04:33:05.140699+0000","last_unstale":"2021-02-10T04:36:51.420283+0000","last_undegraded":"2021-02-10T04:36:51.420283+0000","last_fullsized":"2021-02-10T04:36:51.420283+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:26.978062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.978062+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"595'616","reported_seq":"2192","reported_epoch":"595","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:36:52.682702+0000","last_change":"2021-02-10T04:36:51.409929+0000","last_active":"2021-02-10T04:36:52.682702+0000","last_peered":"2021-02-10T04:36:52.682702+0000","last_clean":"2021-02-10T04:36:52.682702+0000","last_became_active":"2021-02-10T04:28:25.146766+0000","last_became_peered":"2021-02-10T04:28:25.146766+0000","last_unstale":"2021-02-10T04:36:52.682702+0000","last_undegraded":"2021-02-10T04:36:52.682702+0000","last_fullsized":"2021-02-10T04:36:52.682702+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'357","last_scrub_stamp":"2021-02-10T04:33:50.582002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:50.582002+0000","log_size":616,"ondisk_log_size":616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":29005001,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":266,"num_read_kb":239152,"num_write":245,"num_write_kb":103289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"},{"start":"d4","length":"8"},{"start":"dd","length":"1"},{"start":"e1","length":"3"},{"start":"e9","length":"2"},{"start":"ed","length":"4"},{"start":"f2","length":"1"},{"start":"f4","length":"5"},{"start":"fb","length":"1"},{"start":"fd","length":"4"},{"start":"103","length":"1"},{"start":"106","length":"4"},{"start":"10c","length":"1"},{"start":"10f","length":"6"},{"start":"116","length":"2"},{"start":"11c","length":"3"},{"start":"121","length":"1"},{"start":"123","length":"2"},{"start":"126","length":"1"},{"start":"128","length":"5"},{"start":"12e","length":"3"},{"start":"133","length":"4"},{"start":"139","length":"2"},{"start":"13c","length":"3"},{"start":"140","length":"1"},{"start":"143","length":"1"},{"start":"146","length":"1"},{"start":"148","length":"1"},{"start":"14c","length":"1"},{"start":"14e","length":"1"},{"start":"155","length":"2"},{"start":"15a","length":"4"},{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"168","length":"2"},{"start":"16b","length":"1"},{"start":"16f","length":"5"},{"start":"178","length":"4"},{"start":"17f","length":"1"},{"start":"185","length":"2"},{"start":"18a","length":"2"},{"start":"18e","length":"3"},{"start":"194","length":"4"},{"start":"19b","length":"2"},{"start":"19e","length":"2"},{"start":"1a1","length":"2"},{"start":"1a4","length":"1"},{"start":"1a8","length":"1"},{"start":"1ab","length":"1"},{"start":"1ae","length":"1"},{"start":"1b1","length":"1"},{"start":"1b7","length":"1"},{"start":"1b9","length":"2"},{"start":"1bc","length":"1"},{"start":"1bf","length":"2"},{"start":"1c8","length":"1"},{"start":"1d1","length":"1"},{"start":"1d5","length":"1"},{"start":"1d8","length":"1"},{"start":"1df","length":"1"},{"start":"1e4","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":"578","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.421120+0000","last_change":"2021-02-10T04:29:38.793837+0000","last_active":"2021-02-10T04:36:51.421120+0000","last_peered":"2021-02-10T04:36:51.421120+0000","last_clean":"2021-02-10T04:36:51.421120+0000","last_became_active":"2021-02-10T04:28:49.054638+0000","last_became_peered":"2021-02-10T04:28:49.054638+0000","last_unstale":"2021-02-10T04:36:51.421120+0000","last_undegraded":"2021-02-10T04:36:51.421120+0000","last_fullsized":"2021-02-10T04:36:51.421120+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:38.793760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.793760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"556","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.423200+0000","last_change":"2021-02-10T04:29:40.451948+0000","last_active":"2021-02-10T04:36:51.423200+0000","last_peered":"2021-02-10T04:36:51.423200+0000","last_clean":"2021-02-10T04:36:51.423200+0000","last_became_active":"2021-02-10T04:25:51.032201+0000","last_became_peered":"2021-02-10T04:25:51.032201+0000","last_unstale":"2021-02-10T04:36:51.423200+0000","last_undegraded":"2021-02-10T04:36:51.423200+0000","last_fullsized":"2021-02-10T04:36:51.423200+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:40.451832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:29:40.451832+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"538'28","reported_seq":"608","reported_epoch":"595","state":"active+clean","last_fresh":"2021-02-10T04:36:52.429327+0000","last_change":"2021-02-10T04:33:55.288118+0000","last_active":"2021-02-10T04:36:52.429327+0000","last_peered":"2021-02-10T04:36:52.429327+0000","last_clean":"2021-02-10T04:36:52.429327+0000","last_became_active":"2021-02-10T04:25:49.003521+0000","last_became_peered":"2021-02-10T04:25:49.003521+0000","last_unstale":"2021-02-10T04:36:52.429327+0000","last_undegraded":"2021-02-10T04:36:52.429327+0000","last_fullsized":"2021-02-10T04:36:52.429327+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'16","last_scrub_stamp":"2021-02-10T04:33:55.288033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:55.288033+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":34,"num_write":28,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"296","reported_epoch":"595","state":"active+clean","last_fresh":"2021-02-10T04:36:52.427573+0000","last_change":"2021-02-10T04:33:05.166788+0000","last_active":"2021-02-10T04:36:52.427573+0000","last_peered":"2021-02-10T04:36:52.427573+0000","last_clean":"2021-02-10T04:36:52.427573+0000","last_became_active":"2021-02-10T04:33:05.166368+0000","last_became_peered":"2021-02-10T04:33:05.166368+0000","last_unstale":"2021-02-10T04:36:52.427573+0000","last_undegraded":"2021-02-10T04:36:52.427573+0000","last_fullsized":"2021-02-10T04:36:52.427573+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:14.204533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:30:14.204533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"559","reported_epoch":"595","state":"active+clean","last_fresh":"2021-02-10T04:36:52.427461+0000","last_change":"2021-02-10T04:29:21.620573+0000","last_active":"2021-02-10T04:36:52.427461+0000","last_peered":"2021-02-10T04:36:52.427461+0000","last_clean":"2021-02-10T04:36:52.427461+0000","last_became_active":"2021-02-10T04:25:49.001782+0000","last_became_peered":"2021-02-10T04:25:49.001782+0000","last_unstale":"2021-02-10T04:36:52.427461+0000","last_undegraded":"2021-02-10T04:36:52.427461+0000","last_fullsized":"2021-02-10T04:36:52.427461+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:21.620482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:21.620482+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"239","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.421544+0000","last_change":"2021-02-10T04:33:05.153205+0000","last_active":"2021-02-10T04:36:51.421544+0000","last_peered":"2021-02-10T04:36:51.421544+0000","last_clean":"2021-02-10T04:36:51.421544+0000","last_became_active":"2021-02-10T04:33:05.152845+0000","last_became_peered":"2021-02-10T04:33:05.152845+0000","last_unstale":"2021-02-10T04:36:51.421544+0000","last_undegraded":"2021-02-10T04:36:51.421544+0000","last_fullsized":"2021-02-10T04:36:51.421544+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.098681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.098681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.5","version":"215'85","reported_seq":"1569","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.200855+0000","last_change":"2021-02-10T04:36:54.200855+0000","last_active":"2021-02-10T04:36:54.200855+0000","last_peered":"2021-02-10T04:36:54.200855+0000","last_clean":"2021-02-10T04:36:54.200855+0000","last_became_active":"2021-02-10T04:29:07.163659+0000","last_became_peered":"2021-02-10T04:29:07.163659+0000","last_unstale":"2021-02-10T04:36:54.200855+0000","last_undegraded":"2021-02-10T04:36:54.200855+0000","last_fullsized":"2021-02-10T04:36:54.200855+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2201750,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":44,"num_read_kb":45112,"num_write":42,"num_write_kb":10011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"},{"start":"d4","length":"8"},{"start":"dd","length":"1"},{"start":"e1","length":"3"},{"start":"e9","length":"2"},{"start":"ed","length":"4"},{"start":"f2","length":"1"},{"start":"f4","length":"5"},{"start":"fb","length":"1"},{"start":"fd","length":"4"},{"start":"103","length":"1"},{"start":"106","length":"4"},{"start":"10c","length":"1"},{"start":"10f","length":"6"},{"start":"116","length":"2"},{"start":"11c","length":"3"},{"start":"121","length":"1"},{"start":"123","length":"2"},{"start":"126","length":"1"},{"start":"128","length":"5"},{"start":"12e","length":"3"},{"start":"133","length":"4"},{"start":"139","length":"2"},{"start":"13c","length":"3"},{"start":"140","length":"1"},{"start":"143","length":"1"},{"start":"146","length":"1"},{"start":"148","length":"1"},{"start":"14c","length":"1"},{"start":"14e","length":"1"},{"start":"155","length":"2"},{"start":"15a","length":"4"},{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"168","length":"2"},{"start":"16b","length":"1"},{"start":"16f","length":"5"},{"start":"178","length":"4"},{"start":"17f","length":"1"},{"start":"185","length":"2"},{"start":"18a","length":"2"},{"start":"18e","length":"3"},{"start":"194","length":"4"},{"start":"19b","length":"5"},{"start":"1a1","length":"2"},{"start":"1a4","length":"1"},{"start":"1a8","length":"1"},{"start":"1ab","length":"1"},{"start":"1ae","length":"1"},{"start":"1b1","length":"1"},{"start":"1b7","length":"1"},{"start":"1b9","length":"2"},{"start":"1bc","length":"1"},{"start":"1bf","length":"2"},{"start":"1c8","length":"1"},{"start":"1d1","length":"1"},{"start":"1d5","length":"1"},{"start":"1d8","length":"2"},{"start":"1df","length":"1"},{"start":"1e1","length":"1"},{"start":"1e4","length":"1"}]},{"pgid":"1.3","version":"0'0","reported_seq":"318","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.423326+0000","last_change":"2021-02-10T04:33:05.141311+0000","last_active":"2021-02-10T04:36:51.423326+0000","last_peered":"2021-02-10T04:36:51.423326+0000","last_clean":"2021-02-10T04:36:51.423326+0000","last_became_active":"2021-02-10T04:33:05.140966+0000","last_became_peered":"2021-02-10T04:33:05.140966+0000","last_unstale":"2021-02-10T04:36:51.423326+0000","last_undegraded":"2021-02-10T04:36:51.423326+0000","last_fullsized":"2021-02-10T04:36:51.423326+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.928906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.928906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"483","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.199847+0000","last_change":"2021-02-10T04:29:29.805554+0000","last_active":"2021-02-10T04:36:54.199847+0000","last_peered":"2021-02-10T04:36:54.199847+0000","last_clean":"2021-02-10T04:36:54.199847+0000","last_became_active":"2021-02-10T04:29:07.148512+0000","last_became_peered":"2021-02-10T04:29:07.148512+0000","last_unstale":"2021-02-10T04:36:54.199847+0000","last_undegraded":"2021-02-10T04:36:54.199847+0000","last_fullsized":"2021-02-10T04:36:54.199847+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.805432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.805432+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"538'27","reported_seq":"605","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.422732+0000","last_change":"2021-02-10T04:33:56.837896+0000","last_active":"2021-02-10T04:36:51.422732+0000","last_peered":"2021-02-10T04:36:51.422732+0000","last_clean":"2021-02-10T04:36:51.422732+0000","last_became_active":"2021-02-10T04:25:49.002471+0000","last_became_peered":"2021-02-10T04:25:49.002471+0000","last_unstale":"2021-02-10T04:36:51.422732+0000","last_undegraded":"2021-02-10T04:36:51.422732+0000","last_fullsized":"2021-02-10T04:36:51.422732+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:33:56.837816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:56.837816+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"562","reported_epoch":"595","state":"active+clean","last_fresh":"2021-02-10T04:36:52.427351+0000","last_change":"2021-02-10T04:29:36.663903+0000","last_active":"2021-02-10T04:36:52.427351+0000","last_peered":"2021-02-10T04:36:52.427351+0000","last_clean":"2021-02-10T04:36:52.427351+0000","last_became_active":"2021-02-10T04:29:07.158075+0000","last_became_peered":"2021-02-10T04:29:07.158075+0000","last_unstale":"2021-02-10T04:36:52.427351+0000","last_undegraded":"2021-02-10T04:36:52.427351+0000","last_fullsized":"2021-02-10T04:36:52.427351+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.663840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.663840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"581","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.421546+0000","last_change":"2021-02-10T04:28:34.482961+0000","last_active":"2021-02-10T04:36:51.421546+0000","last_peered":"2021-02-10T04:36:51.421546+0000","last_clean":"2021-02-10T04:36:51.421546+0000","last_became_active":"2021-02-10T04:28:21.123027+0000","last_became_peered":"2021-02-10T04:28:21.123027+0000","last_unstale":"2021-02-10T04:36:51.421546+0000","last_undegraded":"2021-02-10T04:36:51.421546+0000","last_fullsized":"2021-02-10T04:36:51.421546+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:34.482906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:34.482906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"559","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.203069+0000","last_change":"2021-02-10T04:29:35.382459+0000","last_active":"2021-02-10T04:36:54.203069+0000","last_peered":"2021-02-10T04:36:54.203069+0000","last_clean":"2021-02-10T04:36:54.203069+0000","last_became_active":"2021-02-10T04:25:49.010614+0000","last_became_peered":"2021-02-10T04:25:49.010614+0000","last_unstale":"2021-02-10T04:36:54.203069+0000","last_undegraded":"2021-02-10T04:36:54.203069+0000","last_fullsized":"2021-02-10T04:36:54.203069+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:35.382341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.382341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"554","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.423278+0000","last_change":"2021-02-10T04:28:38.503398+0000","last_active":"2021-02-10T04:36:51.423278+0000","last_peered":"2021-02-10T04:36:51.423278+0000","last_clean":"2021-02-10T04:36:51.423278+0000","last_became_active":"2021-02-10T04:25:53.037506+0000","last_became_peered":"2021-02-10T04:25:53.037506+0000","last_unstale":"2021-02-10T04:36:51.423278+0000","last_undegraded":"2021-02-10T04:36:51.423278+0000","last_fullsized":"2021-02-10T04:36:51.423278+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:38.503309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:38.503309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"594'219","reported_seq":"1856","reported_epoch":"595","state":"active+clean+snaptrim_wait","last_fresh":"2021-02-10T04:36:53.591197+0000","last_change":"2021-02-10T04:36:53.591197+0000","last_active":"2021-02-10T04:36:53.591197+0000","last_peered":"2021-02-10T04:36:53.591197+0000","last_clean":"2021-02-10T04:36:53.591197+0000","last_became_active":"2021-02-10T04:28:25.150109+0000","last_became_peered":"2021-02-10T04:28:25.150109+0000","last_unstale":"2021-02-10T04:36:53.591197+0000","last_undegraded":"2021-02-10T04:36:53.591197+0000","last_fullsized":"2021-02-10T04:36:53.591197+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'36","last_scrub_stamp":"2021-02-10T04:30:13.863430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:13.863430+0000","log_size":219,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":8952994,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":161,"num_read_kb":158918,"num_write":96,"num_write_kb":32289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"},{"start":"d4","length":"8"},{"start":"dd","length":"1"},{"start":"e1","length":"3"},{"start":"e9","length":"2"},{"start":"ed","length":"4"},{"start":"f2","length":"1"},{"start":"f4","length":"5"},{"start":"fb","length":"1"},{"start":"fd","length":"4"},{"start":"103","length":"1"},{"start":"106","length":"4"},{"start":"10c","length":"1"},{"start":"10f","length":"6"},{"start":"116","length":"2"},{"start":"11c","length":"3"},{"start":"121","length":"1"},{"start":"123","length":"2"},{"start":"126","length":"1"},{"start":"128","length":"5"},{"start":"12e","length":"3"},{"start":"133","length":"4"},{"start":"139","length":"2"},{"start":"13c","length":"3"},{"start":"140","length":"1"},{"start":"143","length":"1"},{"start":"146","length":"1"},{"start":"148","length":"1"},{"start":"14c","length":"1"},{"start":"14e","length":"1"},{"start":"155","length":"2"},{"start":"15a","length":"4"},{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"168","length":"2"},{"start":"16b","length":"1"},{"start":"16f","length":"5"},{"start":"178","length":"4"},{"start":"17f","length":"1"},{"start":"185","length":"2"},{"start":"18a","length":"2"},{"start":"18e","length":"3"},{"start":"194","length":"4"},{"start":"19b","length":"5"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a8","length":"1"},{"start":"1ab","length":"1"},{"start":"1ae","length":"1"},{"start":"1b1","length":"1"},{"start":"1b7","length":"1"},{"start":"1b9","length":"2"},{"start":"1bc","length":"1"},{"start":"1bf","length":"2"},{"start":"1c8","length":"1"},{"start":"1d1","length":"1"},{"start":"1d5","length":"1"},{"start":"1d8","length":"1"},{"start":"1df","length":"1"}]},{"pgid":"5.6","version":"0'0","reported_seq":"557","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.199897+0000","last_change":"2021-02-10T04:29:23.779839+0000","last_active":"2021-02-10T04:36:54.199897+0000","last_peered":"2021-02-10T04:36:54.199897+0000","last_clean":"2021-02-10T04:36:54.199897+0000","last_became_active":"2021-02-10T04:25:51.025306+0000","last_became_peered":"2021-02-10T04:25:51.025306+0000","last_unstale":"2021-02-10T04:36:54.199897+0000","last_undegraded":"2021-02-10T04:36:54.199897+0000","last_fullsized":"2021-02-10T04:36:54.199897+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.779767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.779767+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"538'33","reported_seq":"327","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.423002+0000","last_change":"2021-02-10T04:33:05.147122+0000","last_active":"2021-02-10T04:36:51.423002+0000","last_peered":"2021-02-10T04:36:51.423002+0000","last_clean":"2021-02-10T04:36:51.423002+0000","last_became_active":"2021-02-10T04:33:05.146695+0000","last_became_peered":"2021-02-10T04:33:05.146695+0000","last_unstale":"2021-02-10T04:36:51.423002+0000","last_undegraded":"2021-02-10T04:36:51.423002+0000","last_fullsized":"2021-02-10T04:36:51.423002+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'29","last_scrub_stamp":"2021-02-10T04:29:25.974280+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.974280+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":38,"num_write":28,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"580","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.197890+0000","last_change":"2021-02-10T04:28:39.690493+0000","last_active":"2021-02-10T04:36:54.197890+0000","last_peered":"2021-02-10T04:36:54.197890+0000","last_clean":"2021-02-10T04:36:54.197890+0000","last_became_active":"2021-02-10T04:28:21.120083+0000","last_became_peered":"2021-02-10T04:28:21.120083+0000","last_unstale":"2021-02-10T04:36:54.197890+0000","last_undegraded":"2021-02-10T04:36:54.197890+0000","last_fullsized":"2021-02-10T04:36:54.197890+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:39.690436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:39.690436+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"494","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.198528+0000","last_change":"2021-02-10T04:33:05.142561+0000","last_active":"2021-02-10T04:36:55.198528+0000","last_peered":"2021-02-10T04:36:55.198528+0000","last_clean":"2021-02-10T04:36:55.198528+0000","last_became_active":"2021-02-10T04:33:05.141816+0000","last_became_peered":"2021-02-10T04:33:05.141816+0000","last_unstale":"2021-02-10T04:36:55.198528+0000","last_undegraded":"2021-02-10T04:36:55.198528+0000","last_fullsized":"2021-02-10T04:36:55.198528+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:28:50.992899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:50.992899+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":237,"num_read_kb":232,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"481","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.424433+0000","last_change":"2021-02-10T04:29:33.747527+0000","last_active":"2021-02-10T04:36:51.424433+0000","last_peered":"2021-02-10T04:36:51.424433+0000","last_clean":"2021-02-10T04:36:51.424433+0000","last_became_active":"2021-02-10T04:29:07.158869+0000","last_became_peered":"2021-02-10T04:29:07.158869+0000","last_unstale":"2021-02-10T04:36:51.424433+0000","last_undegraded":"2021-02-10T04:36:51.424433+0000","last_fullsized":"2021-02-10T04:36:51.424433+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:33.747448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:33.747448+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.4","version":"595'373","reported_seq":"2692","reported_epoch":"595","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:36:53.593114+0000","last_change":"2021-02-10T04:36:53.591258+0000","last_active":"2021-02-10T04:36:53.593114+0000","last_peered":"2021-02-10T04:36:53.593114+0000","last_clean":"2021-02-10T04:36:53.593114+0000","last_became_active":"2021-02-10T04:28:25.151037+0000","last_became_peered":"2021-02-10T04:28:25.151037+0000","last_unstale":"2021-02-10T04:36:53.593114+0000","last_undegraded":"2021-02-10T04:36:53.593114+0000","last_fullsized":"2021-02-10T04:36:53.593114+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'239","last_scrub_stamp":"2021-02-10T04:33:46.270478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:46.270478+0000","log_size":373,"ondisk_log_size":373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":17436311,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":202,"num_read_kb":175847,"num_write":187,"num_write_kb":68465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"},{"start":"d4","length":"8"},{"start":"dd","length":"1"},{"start":"e1","length":"3"},{"start":"e9","length":"2"},{"start":"ed","length":"4"},{"start":"f2","length":"1"},{"start":"f4","length":"5"},{"start":"fb","length":"1"},{"start":"fd","length":"4"},{"start":"103","length":"1"},{"start":"106","length":"4"},{"start":"10c","length":"1"},{"start":"10f","length":"6"},{"start":"116","length":"2"},{"start":"11c","length":"3"},{"start":"121","length":"1"},{"start":"123","length":"2"},{"start":"126","length":"1"},{"start":"128","length":"5"},{"start":"12e","length":"3"},{"start":"133","length":"4"},{"start":"139","length":"2"},{"start":"13c","length":"3"},{"start":"140","length":"1"},{"start":"143","length":"1"},{"start":"146","length":"1"},{"start":"148","length":"1"},{"start":"14c","length":"1"},{"start":"14e","length":"1"},{"start":"155","length":"2"},{"start":"15a","length":"4"},{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"168","length":"2"},{"start":"16b","length":"1"},{"start":"16f","length":"5"},{"start":"178","length":"4"},{"start":"17f","length":"1"},{"start":"185","length":"2"},{"start":"18a","length":"2"},{"start":"18e","length":"3"},{"start":"194","length":"4"},{"start":"19b","length":"2"},{"start":"19e","length":"2"},{"start":"1a1","length":"2"},{"start":"1a4","length":"1"},{"start":"1a8","length":"1"},{"start":"1ab","length":"1"},{"start":"1ae","length":"1"},{"start":"1b1","length":"1"},{"start":"1b7","length":"1"},{"start":"1b9","length":"2"},{"start":"1bc","length":"1"},{"start":"1bf","length":"2"},{"start":"1c8","length":"1"},{"start":"1d1","length":"1"},{"start":"1d5","length":"1"},{"start":"1d8","length":"1"},{"start":"1df","length":"1"}]},{"pgid":"1.2","version":"0'0","reported_seq":"572","reported_epoch":"595","state":"active+clean","last_fresh":"2021-02-10T04:36:52.427718+0000","last_change":"2021-02-10T04:29:29.683867+0000","last_active":"2021-02-10T04:36:52.427718+0000","last_peered":"2021-02-10T04:36:52.427718+0000","last_clean":"2021-02-10T04:36:52.427718+0000","last_became_active":"2021-02-10T04:29:07.159607+0000","last_became_peered":"2021-02-10T04:29:07.159607+0000","last_unstale":"2021-02-10T04:36:52.427718+0000","last_undegraded":"2021-02-10T04:36:52.427718+0000","last_fullsized":"2021-02-10T04:36:52.427718+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.683805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.683805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"557","reported_epoch":"595","state":"active+clean","last_fresh":"2021-02-10T04:36:52.428959+0000","last_change":"2021-02-10T04:28:43.740930+0000","last_active":"2021-02-10T04:36:52.428959+0000","last_peered":"2021-02-10T04:36:52.428959+0000","last_clean":"2021-02-10T04:36:52.428959+0000","last_became_active":"2021-02-10T04:25:51.027624+0000","last_became_peered":"2021-02-10T04:25:51.027624+0000","last_unstale":"2021-02-10T04:36:52.428959+0000","last_undegraded":"2021-02-10T04:36:52.428959+0000","last_fullsized":"2021-02-10T04:36:52.428959+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:43.740853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:28:43.740853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"538'43","reported_seq":"340","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.200079+0000","last_change":"2021-02-10T04:33:49.423179+0000","last_active":"2021-02-10T04:36:54.200079+0000","last_peered":"2021-02-10T04:36:54.200079+0000","last_clean":"2021-02-10T04:36:54.200079+0000","last_became_active":"2021-02-10T04:33:05.151701+0000","last_became_peered":"2021-02-10T04:33:05.151701+0000","last_unstale":"2021-02-10T04:36:54.200079+0000","last_undegraded":"2021-02-10T04:36:54.200079+0000","last_fullsized":"2021-02-10T04:36:54.200079+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'31","last_scrub_stamp":"2021-02-10T04:33:49.423083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:49.423083+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":82,"num_read_kb":58,"num_write":46,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"574","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.421092+0000","last_change":"2021-02-10T04:28:36.491041+0000","last_active":"2021-02-10T04:36:51.421092+0000","last_peered":"2021-02-10T04:36:51.421092+0000","last_clean":"2021-02-10T04:36:51.421092+0000","last_became_active":"2021-02-10T04:28:21.125541+0000","last_became_peered":"2021-02-10T04:28:21.125541+0000","last_unstale":"2021-02-10T04:36:51.421092+0000","last_undegraded":"2021-02-10T04:36:51.421092+0000","last_fullsized":"2021-02-10T04:36:51.421092+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:36.490961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:36.490961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"555","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.201622+0000","last_change":"2021-02-10T04:29:36.396044+0000","last_active":"2021-02-10T04:36:54.201622+0000","last_peered":"2021-02-10T04:36:54.201622+0000","last_clean":"2021-02-10T04:36:54.201622+0000","last_became_active":"2021-02-10T04:25:53.049487+0000","last_became_peered":"2021-02-10T04:25:53.049487+0000","last_unstale":"2021-02-10T04:36:54.201622+0000","last_undegraded":"2021-02-10T04:36:54.201622+0000","last_fullsized":"2021-02-10T04:36:54.201622+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.395943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.395943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"595'1240","reported_seq":"6344","reported_epoch":"595","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:36:52.647452+0000","last_change":"2021-02-10T04:36:52.644734+0000","last_active":"2021-02-10T04:36:52.647452+0000","last_peered":"2021-02-10T04:36:52.647452+0000","last_clean":"2021-02-10T04:36:52.647452+0000","last_became_active":"2021-02-10T04:28:25.147959+0000","last_became_peered":"2021-02-10T04:28:25.147959+0000","last_unstale":"2021-02-10T04:36:52.647452+0000","last_undegraded":"2021-02-10T04:36:52.647452+0000","last_fullsized":"2021-02-10T04:36:52.647452+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'223","last_scrub_stamp":"2021-02-10T04:30:14.478639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:14.478639+0000","log_size":1240,"ondisk_log_size":1240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":22,"stat_sum":{"num_bytes":68146071,"num_objects":35,"num_object_clones":28,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":2,"num_read":711,"num_read_kb":674440,"num_write":563,"num_write_kb":230603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"},{"start":"d4","length":"8"},{"start":"dd","length":"1"},{"start":"e1","length":"3"},{"start":"e9","length":"2"},{"start":"ed","length":"4"},{"start":"f2","length":"1"},{"start":"f4","length":"5"},{"start":"fb","length":"1"},{"start":"fd","length":"4"},{"start":"103","length":"1"},{"start":"106","length":"4"},{"start":"10c","length":"1"},{"start":"10f","length":"6"},{"start":"116","length":"2"},{"start":"11c","length":"3"},{"start":"121","length":"1"},{"start":"123","length":"2"},{"start":"126","length":"1"},{"start":"128","length":"5"},{"start":"12e","length":"3"},{"start":"133","length":"4"},{"start":"139","length":"2"},{"start":"13c","length":"3"},{"start":"140","length":"1"},{"start":"143","length":"1"},{"start":"146","length":"1"},{"start":"148","length":"1"},{"start":"14c","length":"1"},{"start":"14e","length":"1"},{"start":"155","length":"2"},{"start":"15a","length":"4"},{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"168","length":"2"},{"start":"16b","length":"1"},{"start":"16f","length":"5"},{"start":"178","length":"4"},{"start":"17f","length":"1"},{"start":"185","length":"2"},{"start":"18a","length":"2"},{"start":"18e","length":"3"},{"start":"194","length":"4"},{"start":"19b","length":"2"},{"start":"19e","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":"497","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.420190+0000","last_change":"2021-02-10T04:29:39.434301+0000","last_active":"2021-02-10T04:36:51.420190+0000","last_peered":"2021-02-10T04:36:51.420190+0000","last_clean":"2021-02-10T04:36:51.420190+0000","last_became_active":"2021-02-10T04:28:49.054267+0000","last_became_peered":"2021-02-10T04:28:49.054267+0000","last_unstale":"2021-02-10T04:36:51.420190+0000","last_undegraded":"2021-02-10T04:36:51.420190+0000","last_fullsized":"2021-02-10T04:36:51.420190+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:39.434210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:39.434210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"58'2","reported_seq":"692","reported_epoch":"595","state":"active+clean","last_fresh":"2021-02-10T04:36:52.825324+0000","last_change":"2021-02-10T04:28:33.656576+0000","last_active":"2021-02-10T04:36:52.825324+0000","last_peered":"2021-02-10T04:36:52.825324+0000","last_clean":"2021-02-10T04:36:52.825324+0000","last_became_active":"2021-02-10T04:25:51.026251+0000","last_became_peered":"2021-02-10T04:25:51.026251+0000","last_unstale":"2021-02-10T04:36:52.825324+0000","last_undegraded":"2021-02-10T04:36:52.825324+0000","last_fullsized":"2021-02-10T04:36:52.825324+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:28:33.656509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:33.656509+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"538'44","reported_seq":"329","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.423606+0000","last_change":"2021-02-10T04:33:05.142280+0000","last_active":"2021-02-10T04:36:51.423606+0000","last_peered":"2021-02-10T04:36:51.423606+0000","last_clean":"2021-02-10T04:36:51.423606+0000","last_became_active":"2021-02-10T04:33:05.141901+0000","last_became_peered":"2021-02-10T04:33:05.141901+0000","last_unstale":"2021-02-10T04:36:51.423606+0000","last_undegraded":"2021-02-10T04:36:51.423606+0000","last_fullsized":"2021-02-10T04:36:51.423606+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'34","last_scrub_stamp":"2021-02-10T04:28:49.959162+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:28:49.959162+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":60,"num_read_kb":42,"num_write":32,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"313","reported_epoch":"595","state":"active+clean","last_fresh":"2021-02-10T04:36:52.430092+0000","last_change":"2021-02-10T04:33:05.142215+0000","last_active":"2021-02-10T04:36:52.430092+0000","last_peered":"2021-02-10T04:36:52.430092+0000","last_clean":"2021-02-10T04:36:52.430092+0000","last_became_active":"2021-02-10T04:33:05.141852+0000","last_became_peered":"2021-02-10T04:33:05.141852+0000","last_unstale":"2021-02-10T04:36:52.430092+0000","last_undegraded":"2021-02-10T04:36:52.430092+0000","last_fullsized":"2021-02-10T04:36:52.430092+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:34.053323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:34.053323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"555","reported_epoch":"595","state":"active+clean","last_fresh":"2021-02-10T04:36:52.426718+0000","last_change":"2021-02-10T04:28:42.701993+0000","last_active":"2021-02-10T04:36:52.426718+0000","last_peered":"2021-02-10T04:36:52.426718+0000","last_clean":"2021-02-10T04:36:52.426718+0000","last_became_active":"2021-02-10T04:25:53.045215+0000","last_became_peered":"2021-02-10T04:25:53.045215+0000","last_unstale":"2021-02-10T04:36:52.426718+0000","last_undegraded":"2021-02-10T04:36:52.426718+0000","last_fullsized":"2021-02-10T04:36:52.426718+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:42.701924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:42.701924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"293'77","reported_seq":"1571","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.198928+0000","last_change":"2021-02-10T04:36:55.081218+0000","last_active":"2021-02-10T04:36:55.198928+0000","last_peered":"2021-02-10T04:36:55.198928+0000","last_clean":"2021-02-10T04:36:55.198928+0000","last_became_active":"2021-02-10T04:28:25.154434+0000","last_became_peered":"2021-02-10T04:28:25.154434+0000","last_unstale":"2021-02-10T04:36:55.198928+0000","last_undegraded":"2021-02-10T04:36:55.198928+0000","last_fullsized":"2021-02-10T04:36:55.198928+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7546343,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":74349,"num_write":37,"num_write_kb":13263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"},{"start":"d4","length":"8"},{"start":"dd","length":"1"},{"start":"e1","length":"3"},{"start":"e9","length":"2"},{"start":"ed","length":"4"},{"start":"f2","length":"1"},{"start":"f4","length":"5"},{"start":"fb","length":"1"},{"start":"fd","length":"4"},{"start":"103","length":"1"},{"start":"106","length":"4"},{"start":"10c","length":"1"},{"start":"10f","length":"6"},{"start":"116","length":"2"},{"start":"11c","length":"3"},{"start":"121","length":"1"},{"start":"123","length":"2"},{"start":"126","length":"1"},{"start":"128","length":"5"},{"start":"12e","length":"3"},{"start":"133","length":"4"},{"start":"139","length":"2"},{"start":"13c","length":"3"},{"start":"140","length":"1"},{"start":"143","length":"1"},{"start":"146","length":"1"},{"start":"148","length":"1"},{"start":"14c","length":"1"},{"start":"14e","length":"1"},{"start":"155","length":"2"},{"start":"15a","length":"4"},{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"168","length":"2"},{"start":"16b","length":"1"},{"start":"16f","length":"5"},{"start":"178","length":"4"},{"start":"17f","length":"1"},{"start":"185","length":"2"},{"start":"18a","length":"2"},{"start":"18e","length":"3"},{"start":"194","length":"4"},{"start":"19b","length":"5"},{"start":"1a1","length":"2"},{"start":"1a4","length":"1"},{"start":"1a8","length":"1"},{"start":"1ab","length":"1"},{"start":"1ae","length":"1"},{"start":"1b1","length":"1"},{"start":"1b7","length":"1"},{"start":"1b9","length":"2"},{"start":"1bc","length":"1"},{"start":"1bf","length":"2"},{"start":"1c8","length":"1"},{"start":"1d1","length":"1"},{"start":"1d5","length":"1"},{"start":"1d8","length":"2"},{"start":"1df","length":"1"},{"start":"1e1","length":"1"},{"start":"1e4","length":"1"}]},{"pgid":"1.7","version":"0'0","reported_seq":"578","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.421258+0000","last_change":"2021-02-10T04:29:39.829039+0000","last_active":"2021-02-10T04:36:51.421258+0000","last_peered":"2021-02-10T04:36:51.421258+0000","last_clean":"2021-02-10T04:36:51.421258+0000","last_became_active":"2021-02-10T04:28:49.053127+0000","last_became_peered":"2021-02-10T04:28:49.053127+0000","last_unstale":"2021-02-10T04:36:51.421258+0000","last_undegraded":"2021-02-10T04:36:51.421258+0000","last_fullsized":"2021-02-10T04:36:51.421258+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:39.828975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:39.828975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"559","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.198203+0000","last_change":"2021-02-10T04:29:20.757283+0000","last_active":"2021-02-10T04:36:54.198203+0000","last_peered":"2021-02-10T04:36:54.198203+0000","last_clean":"2021-02-10T04:36:54.198203+0000","last_became_active":"2021-02-10T04:25:51.023000+0000","last_became_peered":"2021-02-10T04:25:51.023000+0000","last_unstale":"2021-02-10T04:36:54.198203+0000","last_undegraded":"2021-02-10T04:36:54.198203+0000","last_fullsized":"2021-02-10T04:36:54.198203+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:20.757194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:20.757194+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"538'33","reported_seq":"629","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.199271+0000","last_change":"2021-02-10T04:28:47.476743+0000","last_active":"2021-02-10T04:36:55.199271+0000","last_peered":"2021-02-10T04:36:55.199271+0000","last_clean":"2021-02-10T04:36:55.199271+0000","last_became_active":"2021-02-10T04:25:49.013259+0000","last_became_peered":"2021-02-10T04:25:49.013259+0000","last_unstale":"2021-02-10T04:36:55.199271+0000","last_undegraded":"2021-02-10T04:36:55.199271+0000","last_fullsized":"2021-02-10T04:36:55.199271+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'27","last_scrub_stamp":"2021-02-10T04:28:47.476669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:47.476669+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":67,"num_read_kb":50,"num_write":35,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"679","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.203101+0000","last_change":"2021-02-10T04:33:49.761068+0000","last_active":"2021-02-10T04:36:55.203101+0000","last_peered":"2021-02-10T04:36:55.203101+0000","last_clean":"2021-02-10T04:36:55.203101+0000","last_became_active":"2021-02-10T04:28:39.041556+0000","last_became_peered":"2021-02-10T04:28:39.041556+0000","last_unstale":"2021-02-10T04:36:55.203101+0000","last_undegraded":"2021-02-10T04:36:55.203101+0000","last_fullsized":"2021-02-10T04:36:55.203101+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:33:49.760989+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:33:49.760989+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"555","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.203525+0000","last_change":"2021-02-10T04:29:37.364781+0000","last_active":"2021-02-10T04:36:54.203525+0000","last_peered":"2021-02-10T04:36:54.203525+0000","last_clean":"2021-02-10T04:36:54.203525+0000","last_became_active":"2021-02-10T04:25:53.047736+0000","last_became_peered":"2021-02-10T04:25:53.047736+0000","last_unstale":"2021-02-10T04:36:54.203525+0000","last_undegraded":"2021-02-10T04:36:54.203525+0000","last_fullsized":"2021-02-10T04:36:54.203525+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:37.364697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:37.364697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"595'855","reported_seq":"3414","reported_epoch":"597","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:36:55.207352+0000","last_change":"2021-02-10T04:36:53.803825+0000","last_active":"2021-02-10T04:36:55.207352+0000","last_peered":"2021-02-10T04:36:55.207352+0000","last_clean":"2021-02-10T04:36:55.207352+0000","last_became_active":"2021-02-10T04:28:25.157771+0000","last_became_peered":"2021-02-10T04:28:25.157771+0000","last_unstale":"2021-02-10T04:36:55.207352+0000","last_undegraded":"2021-02-10T04:36:55.207352+0000","last_fullsized":"2021-02-10T04:36:55.207352+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":855,"ondisk_log_size":855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"stat_sum":{"num_bytes":45741379,"num_objects":19,"num_object_clones":16,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":453,"num_read_kb":443342,"num_write":380,"num_write_kb":143234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"},{"start":"d4","length":"8"},{"start":"dd","length":"1"},{"start":"e1","length":"3"},{"start":"e9","length":"2"},{"start":"ed","length":"4"},{"start":"f2","length":"1"},{"start":"f4","length":"5"},{"start":"fb","length":"1"},{"start":"fd","length":"4"},{"start":"103","length":"1"},{"start":"106","length":"4"},{"start":"10c","length":"1"},{"start":"10f","length":"6"},{"start":"116","length":"2"},{"start":"11c","length":"3"},{"start":"121","length":"1"},{"start":"123","length":"2"},{"start":"126","length":"1"},{"start":"128","length":"5"},{"start":"12e","length":"3"},{"start":"133","length":"4"},{"start":"139","length":"2"},{"start":"13c","length":"3"},{"start":"140","length":"1"},{"start":"143","length":"1"},{"start":"146","length":"1"},{"start":"148","length":"1"},{"start":"14c","length":"1"},{"start":"14e","length":"1"},{"start":"155","length":"2"},{"start":"15a","length":"4"},{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"168","length":"2"},{"start":"16b","length":"1"},{"start":"16f","length":"5"},{"start":"178","length":"4"},{"start":"17f","length":"1"},{"start":"185","length":"2"},{"start":"18a","length":"2"},{"start":"18e","length":"3"},{"start":"194","length":"4"},{"start":"19b","length":"2"},{"start":"19e","length":"2"},{"start":"1a1","length":"2"},{"start":"1a4","length":"1"},{"start":"1a8","length":"1"},{"start":"1ab","length":"1"},{"start":"1ae","length":"1"},{"start":"1b1","length":"1"},{"start":"1b7","length":"1"},{"start":"1b9","length":"2"}]},{"pgid":"4.16","version":"538'34","reported_seq":"628","reported_epoch":"595","state":"active+clean","last_fresh":"2021-02-10T04:36:52.429579+0000","last_change":"2021-02-10T04:29:25.645462+0000","last_active":"2021-02-10T04:36:52.429579+0000","last_peered":"2021-02-10T04:36:52.429579+0000","last_clean":"2021-02-10T04:36:52.429579+0000","last_became_active":"2021-02-10T04:25:49.008942+0000","last_became_peered":"2021-02-10T04:25:49.008942+0000","last_unstale":"2021-02-10T04:36:52.429579+0000","last_undegraded":"2021-02-10T04:36:52.429579+0000","last_fullsized":"2021-02-10T04:36:52.429579+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'28","last_scrub_stamp":"2021-02-10T04:29:25.645396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.645396+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":71,"num_read_kb":49,"num_write":35,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"569","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.198123+0000","last_change":"2021-02-10T04:28:45.647998+0000","last_active":"2021-02-10T04:36:54.198123+0000","last_peered":"2021-02-10T04:36:54.198123+0000","last_clean":"2021-02-10T04:36:54.198123+0000","last_became_active":"2021-02-10T04:25:34.329694+0000","last_became_peered":"2021-02-10T04:25:34.329694+0000","last_unstale":"2021-02-10T04:36:54.198123+0000","last_undegraded":"2021-02-10T04:36:54.198123+0000","last_fullsized":"2021-02-10T04:36:54.198123+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:28:45.647916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:45.647916+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"557","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.201752+0000","last_change":"2021-02-10T04:28:33.321856+0000","last_active":"2021-02-10T04:36:54.201752+0000","last_peered":"2021-02-10T04:36:54.201752+0000","last_clean":"2021-02-10T04:36:54.201752+0000","last_became_active":"2021-02-10T04:25:51.030651+0000","last_became_peered":"2021-02-10T04:25:51.030651+0000","last_unstale":"2021-02-10T04:36:54.201752+0000","last_undegraded":"2021-02-10T04:36:54.201752+0000","last_fullsized":"2021-02-10T04:36:54.201752+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:33.321784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:33.321784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"555","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.201973+0000","last_change":"2021-02-10T04:29:28.776423+0000","last_active":"2021-02-10T04:36:54.201973+0000","last_peered":"2021-02-10T04:36:54.201973+0000","last_clean":"2021-02-10T04:36:54.201973+0000","last_became_active":"2021-02-10T04:25:53.035867+0000","last_became_peered":"2021-02-10T04:25:53.035867+0000","last_unstale":"2021-02-10T04:36:54.201973+0000","last_undegraded":"2021-02-10T04:36:54.201973+0000","last_fullsized":"2021-02-10T04:36:54.201973+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.776340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.776340+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"538'19","reported_seq":"615","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.201614+0000","last_change":"2021-02-10T04:28:11.311247+0000","last_active":"2021-02-10T04:36:54.201614+0000","last_peered":"2021-02-10T04:36:54.201614+0000","last_clean":"2021-02-10T04:36:54.201614+0000","last_became_active":"2021-02-10T04:26:39.003832+0000","last_became_peered":"2021-02-10T04:26:39.003832+0000","last_unstale":"2021-02-10T04:36:54.201614+0000","last_undegraded":"2021-02-10T04:36:54.201614+0000","last_fullsized":"2021-02-10T04:36:54.201614+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2021-02-10T04:28:11.311182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:11.311182+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":19,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"567","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.420432+0000","last_change":"2021-02-10T04:29:18.565481+0000","last_active":"2021-02-10T04:36:51.420432+0000","last_peered":"2021-02-10T04:36:51.420432+0000","last_clean":"2021-02-10T04:36:51.420432+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:36:51.420432+0000","last_undegraded":"2021-02-10T04:36:51.420432+0000","last_fullsized":"2021-02-10T04:36:51.420432+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:29:18.565413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:18.565413+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"556","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.422513+0000","last_change":"2021-02-10T04:29:18.429730+0000","last_active":"2021-02-10T04:36:51.422513+0000","last_peered":"2021-02-10T04:36:51.422513+0000","last_clean":"2021-02-10T04:36:51.422513+0000","last_became_active":"2021-02-10T04:25:51.031864+0000","last_became_peered":"2021-02-10T04:25:51.031864+0000","last_unstale":"2021-02-10T04:36:51.422513+0000","last_undegraded":"2021-02-10T04:36:51.422513+0000","last_fullsized":"2021-02-10T04:36:51.422513+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:18.429649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:18.429649+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"481","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.424309+0000","last_change":"2021-02-10T04:29:36.866949+0000","last_active":"2021-02-10T04:36:51.424309+0000","last_peered":"2021-02-10T04:36:51.424309+0000","last_clean":"2021-02-10T04:36:51.424309+0000","last_became_active":"2021-02-10T04:29:07.147249+0000","last_became_peered":"2021-02-10T04:29:07.147249+0000","last_unstale":"2021-02-10T04:36:51.424309+0000","last_undegraded":"2021-02-10T04:36:51.424309+0000","last_fullsized":"2021-02-10T04:36:51.424309+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.866833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.866833+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"538'36","reported_seq":"732","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.202305+0000","last_change":"2021-02-10T04:33:46.786034+0000","last_active":"2021-02-10T04:36:54.202305+0000","last_peered":"2021-02-10T04:36:54.202305+0000","last_clean":"2021-02-10T04:36:54.202305+0000","last_became_active":"2021-02-10T04:26:39.003971+0000","last_became_peered":"2021-02-10T04:26:39.003971+0000","last_unstale":"2021-02-10T04:36:54.202305+0000","last_undegraded":"2021-02-10T04:36:54.202305+0000","last_fullsized":"2021-02-10T04:36:54.202305+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:33:46.785905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:46.785905+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":36,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"84'7","reported_seq":"585","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.420983+0000","last_change":"2021-02-10T04:30:12.538670+0000","last_active":"2021-02-10T04:36:51.420983+0000","last_peered":"2021-02-10T04:36:51.420983+0000","last_clean":"2021-02-10T04:36:51.420983+0000","last_became_active":"2021-02-10T04:25:36.334366+0000","last_became_peered":"2021-02-10T04:25:36.334366+0000","last_unstale":"2021-02-10T04:36:51.420983+0000","last_undegraded":"2021-02-10T04:36:51.420983+0000","last_fullsized":"2021-02-10T04:36:51.420983+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'7","last_scrub_stamp":"2021-02-10T04:30:12.538549+0000","last_deep_scrub":"84'7","last_deep_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_clean_scrub_stamp":"2021-02-10T04:30:12.538549+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"58'2","reported_seq":"691","reported_epoch":"595","state":"active+clean","last_fresh":"2021-02-10T04:36:52.444456+0000","last_change":"2021-02-10T04:28:47.496534+0000","last_active":"2021-02-10T04:36:52.444456+0000","last_peered":"2021-02-10T04:36:52.444456+0000","last_clean":"2021-02-10T04:36:52.444456+0000","last_became_active":"2021-02-10T04:25:51.032328+0000","last_became_peered":"2021-02-10T04:25:51.032328+0000","last_unstale":"2021-02-10T04:36:52.444456+0000","last_undegraded":"2021-02-10T04:36:52.444456+0000","last_fullsized":"2021-02-10T04:36:52.444456+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:28:47.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:47.496468+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"555","reported_epoch":"595","state":"active+clean","last_fresh":"2021-02-10T04:36:52.429237+0000","last_change":"2021-02-10T04:28:45.697744+0000","last_active":"2021-02-10T04:36:52.429237+0000","last_peered":"2021-02-10T04:36:52.429237+0000","last_clean":"2021-02-10T04:36:52.429237+0000","last_became_active":"2021-02-10T04:25:53.045177+0000","last_became_peered":"2021-02-10T04:25:53.045177+0000","last_unstale":"2021-02-10T04:36:52.429237+0000","last_undegraded":"2021-02-10T04:36:52.429237+0000","last_fullsized":"2021-02-10T04:36:52.429237+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_clean_scrub_stamp":"2021-02-10T04:28:45.697677+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"538'25","reported_seq":"606","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.421983+0000","last_change":"2021-02-10T04:28:50.493712+0000","last_active":"2021-02-10T04:36:51.421983+0000","last_peered":"2021-02-10T04:36:51.421983+0000","last_clean":"2021-02-10T04:36:51.421983+0000","last_became_active":"2021-02-10T04:25:49.010890+0000","last_became_peered":"2021-02-10T04:25:49.010890+0000","last_unstale":"2021-02-10T04:36:51.421983+0000","last_undegraded":"2021-02-10T04:36:51.421983+0000","last_fullsized":"2021-02-10T04:36:51.421983+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2021-02-10T04:28:50.493641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:50.493641+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":35,"num_write":26,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"571","reported_epoch":"595","state":"active+clean","last_fresh":"2021-02-10T04:36:52.428118+0000","last_change":"2021-02-10T04:29:19.564845+0000","last_active":"2021-02-10T04:36:52.428118+0000","last_peered":"2021-02-10T04:36:52.428118+0000","last_clean":"2021-02-10T04:36:52.428118+0000","last_became_active":"2021-02-10T04:25:34.321096+0000","last_became_peered":"2021-02-10T04:25:34.321096+0000","last_unstale":"2021-02-10T04:36:52.428118+0000","last_undegraded":"2021-02-10T04:36:52.428118+0000","last_fullsized":"2021-02-10T04:36:52.428118+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:19.564725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:29:19.564725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"58'2","reported_seq":"692","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.202447+0000","last_change":"2021-02-10T04:29:38.343946+0000","last_active":"2021-02-10T04:36:54.202447+0000","last_peered":"2021-02-10T04:36:54.202447+0000","last_clean":"2021-02-10T04:36:54.202447+0000","last_became_active":"2021-02-10T04:25:51.018300+0000","last_became_peered":"2021-02-10T04:25:51.018300+0000","last_unstale":"2021-02-10T04:36:54.202447+0000","last_undegraded":"2021-02-10T04:36:54.202447+0000","last_fullsized":"2021-02-10T04:36:54.202447+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:29:38.343834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.343834+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"557","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.203305+0000","last_change":"2021-02-10T04:29:23.934149+0000","last_active":"2021-02-10T04:36:55.203305+0000","last_peered":"2021-02-10T04:36:55.203305+0000","last_clean":"2021-02-10T04:36:55.203305+0000","last_became_active":"2021-02-10T04:25:53.031782+0000","last_became_peered":"2021-02-10T04:25:53.031782+0000","last_unstale":"2021-02-10T04:36:55.203305+0000","last_undegraded":"2021-02-10T04:36:55.203305+0000","last_fullsized":"2021-02-10T04:36:55.203305+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.934078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.934078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"538'21","reported_seq":"289","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.422654+0000","last_change":"2021-02-10T04:33:05.141994+0000","last_active":"2021-02-10T04:36:51.422654+0000","last_peered":"2021-02-10T04:36:51.422654+0000","last_clean":"2021-02-10T04:36:51.422654+0000","last_became_active":"2021-02-10T04:33:05.141566+0000","last_became_peered":"2021-02-10T04:33:05.141566+0000","last_unstale":"2021-02-10T04:36:51.422654+0000","last_undegraded":"2021-02-10T04:36:51.422654+0000","last_fullsized":"2021-02-10T04:36:51.422654+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:30:11.170034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.170034+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"565","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.422096+0000","last_change":"2021-02-10T04:28:36.204091+0000","last_active":"2021-02-10T04:36:51.422096+0000","last_peered":"2021-02-10T04:36:51.422096+0000","last_clean":"2021-02-10T04:36:51.422096+0000","last_became_active":"2021-02-10T04:25:34.330070+0000","last_became_peered":"2021-02-10T04:25:34.330070+0000","last_unstale":"2021-02-10T04:36:51.422096+0000","last_undegraded":"2021-02-10T04:36:51.422096+0000","last_fullsized":"2021-02-10T04:36:51.422096+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:36.204007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:36.204007+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"557","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.203445+0000","last_change":"2021-02-10T04:28:36.311808+0000","last_active":"2021-02-10T04:36:54.203445+0000","last_peered":"2021-02-10T04:36:54.203445+0000","last_clean":"2021-02-10T04:36:54.203445+0000","last_became_active":"2021-02-10T04:25:51.029155+0000","last_became_peered":"2021-02-10T04:25:51.029155+0000","last_unstale":"2021-02-10T04:36:54.203445+0000","last_undegraded":"2021-02-10T04:36:54.203445+0000","last_fullsized":"2021-02-10T04:36:54.203445+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:36.311725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:28:36.311725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"560","reported_epoch":"595","state":"active+clean","last_fresh":"2021-02-10T04:36:52.427178+0000","last_change":"2021-02-10T04:33:49.338303+0000","last_active":"2021-02-10T04:36:52.427178+0000","last_peered":"2021-02-10T04:36:52.427178+0000","last_clean":"2021-02-10T04:36:52.427178+0000","last_became_active":"2021-02-10T04:25:53.039322+0000","last_became_peered":"2021-02-10T04:25:53.039322+0000","last_unstale":"2021-02-10T04:36:52.427178+0000","last_undegraded":"2021-02-10T04:36:52.427178+0000","last_fullsized":"2021-02-10T04:36:52.427178+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:49.338203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:33:49.338203+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"538'23","reported_seq":"595","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.199786+0000","last_change":"2021-02-10T04:27:42.487239+0000","last_active":"2021-02-10T04:36:55.199786+0000","last_peered":"2021-02-10T04:36:55.199786+0000","last_clean":"2021-02-10T04:36:55.199786+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:36:55.199786+0000","last_undegraded":"2021-02-10T04:36:55.199786+0000","last_fullsized":"2021-02-10T04:36:55.199786+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:27:42.487154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:42.487154+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"250","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.422064+0000","last_change":"2021-02-10T04:33:05.148470+0000","last_active":"2021-02-10T04:36:51.422064+0000","last_peered":"2021-02-10T04:36:51.422064+0000","last_clean":"2021-02-10T04:36:51.422064+0000","last_became_active":"2021-02-10T04:33:05.147956+0000","last_became_peered":"2021-02-10T04:33:05.147956+0000","last_unstale":"2021-02-10T04:36:51.422064+0000","last_undegraded":"2021-02-10T04:36:51.422064+0000","last_fullsized":"2021-02-10T04:36:51.422064+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:28:33.070726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:33.070726+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"239","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.423506+0000","last_change":"2021-02-10T04:33:05.150906+0000","last_active":"2021-02-10T04:36:51.423506+0000","last_peered":"2021-02-10T04:36:51.423506+0000","last_clean":"2021-02-10T04:36:51.423506+0000","last_became_active":"2021-02-10T04:33:05.150529+0000","last_became_peered":"2021-02-10T04:33:05.150529+0000","last_unstale":"2021-02-10T04:36:51.423506+0000","last_undegraded":"2021-02-10T04:36:51.423506+0000","last_fullsized":"2021-02-10T04:36:51.423506+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:33.026988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:29:33.026988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"555","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.204025+0000","last_change":"2021-02-10T04:29:24.421782+0000","last_active":"2021-02-10T04:36:54.204025+0000","last_peered":"2021-02-10T04:36:54.204025+0000","last_clean":"2021-02-10T04:36:54.204025+0000","last_became_active":"2021-02-10T04:25:53.043277+0000","last_became_peered":"2021-02-10T04:25:53.043277+0000","last_unstale":"2021-02-10T04:36:54.204025+0000","last_undegraded":"2021-02-10T04:36:54.204025+0000","last_fullsized":"2021-02-10T04:36:54.204025+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.421710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.421710+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"538'16","reported_seq":"697","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.197136+0000","last_change":"2021-02-10T04:36:52.693862+0000","last_active":"2021-02-10T04:36:54.197136+0000","last_peered":"2021-02-10T04:36:54.197136+0000","last_clean":"2021-02-10T04:36:54.197136+0000","last_became_active":"2021-02-10T04:26:39.005616+0000","last_became_peered":"2021-02-10T04:26:39.005616+0000","last_unstale":"2021-02-10T04:36:54.197136+0000","last_undegraded":"2021-02-10T04:36:54.197136+0000","last_fullsized":"2021-02-10T04:36:54.197136+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2021-02-10T04:27:48.284248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:48.284248+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":36,"num_read_kb":18,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"565","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.422683+0000","last_change":"2021-02-10T04:28:49.522178+0000","last_active":"2021-02-10T04:36:51.422683+0000","last_peered":"2021-02-10T04:36:51.422683+0000","last_clean":"2021-02-10T04:36:51.422683+0000","last_became_active":"2021-02-10T04:25:34.331787+0000","last_became_peered":"2021-02-10T04:25:34.331787+0000","last_unstale":"2021-02-10T04:36:51.422683+0000","last_undegraded":"2021-02-10T04:36:51.422683+0000","last_fullsized":"2021-02-10T04:36:51.422683+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:49.522124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:49.522124+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"556","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.420383+0000","last_change":"2021-02-10T04:29:23.645783+0000","last_active":"2021-02-10T04:36:51.420383+0000","last_peered":"2021-02-10T04:36:51.420383+0000","last_clean":"2021-02-10T04:36:51.420383+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:36:51.420383+0000","last_undegraded":"2021-02-10T04:36:51.420383+0000","last_fullsized":"2021-02-10T04:36:51.420383+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.645694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.645694+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"483","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.200785+0000","last_change":"2021-02-10T04:29:31.825006+0000","last_active":"2021-02-10T04:36:54.200785+0000","last_peered":"2021-02-10T04:36:54.200785+0000","last_clean":"2021-02-10T04:36:54.200785+0000","last_became_active":"2021-02-10T04:29:07.149764+0000","last_became_peered":"2021-02-10T04:29:07.149764+0000","last_unstale":"2021-02-10T04:36:54.200785+0000","last_undegraded":"2021-02-10T04:36:54.200785+0000","last_fullsized":"2021-02-10T04:36:54.200785+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:31.824925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.824925+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"538'29","reported_seq":"612","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.204476+0000","last_change":"2021-02-10T04:28:31.382776+0000","last_active":"2021-02-10T04:36:54.204476+0000","last_peered":"2021-02-10T04:36:54.204476+0000","last_clean":"2021-02-10T04:36:54.204476+0000","last_became_active":"2021-02-10T04:25:49.007049+0000","last_became_peered":"2021-02-10T04:25:49.007049+0000","last_unstale":"2021-02-10T04:36:54.204476+0000","last_undegraded":"2021-02-10T04:36:54.204476+0000","last_fullsized":"2021-02-10T04:36:54.204476+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2021-02-10T04:28:31.382671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:31.382671+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":29,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"674","reported_epoch":"595","state":"active+clean","last_fresh":"2021-02-10T04:36:53.255927+0000","last_change":"2021-02-10T04:36:53.256600+0000","last_active":"2021-02-10T04:36:53.255927+0000","last_peered":"2021-02-10T04:36:53.255927+0000","last_clean":"2021-02-10T04:36:53.255927+0000","last_became_active":"2021-02-10T04:25:35.328828+0000","last_became_peered":"2021-02-10T04:25:35.328828+0000","last_unstale":"2021-02-10T04:36:53.255927+0000","last_undegraded":"2021-02-10T04:36:53.255927+0000","last_fullsized":"2021-02-10T04:36:53.255927+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:28:35.196342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:35.196342+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"479","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.199356+0000","last_change":"2021-02-10T04:29:07.150253+0000","last_active":"2021-02-10T04:36:55.199356+0000","last_peered":"2021-02-10T04:36:55.199356+0000","last_clean":"2021-02-10T04:36:55.199356+0000","last_became_active":"2021-02-10T04:29:07.149867+0000","last_became_peered":"2021-02-10T04:29:07.149867+0000","last_unstale":"2021-02-10T04:36:55.199356+0000","last_undegraded":"2021-02-10T04:36:55.199356+0000","last_fullsized":"2021-02-10T04:36:55.199356+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:47.473024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:47.473024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"555","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.203825+0000","last_change":"2021-02-10T04:33:50.809533+0000","last_active":"2021-02-10T04:36:54.203825+0000","last_peered":"2021-02-10T04:36:54.203825+0000","last_clean":"2021-02-10T04:36:54.203825+0000","last_became_active":"2021-02-10T04:25:53.150845+0000","last_became_peered":"2021-02-10T04:25:53.150845+0000","last_unstale":"2021-02-10T04:36:54.203825+0000","last_undegraded":"2021-02-10T04:36:54.203825+0000","last_fullsized":"2021-02-10T04:36:54.203825+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:50.809468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_clean_scrub_stamp":"2021-02-10T04:33:50.809468+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"538'32","reported_seq":"621","reported_epoch":"595","state":"active+clean","last_fresh":"2021-02-10T04:36:52.429896+0000","last_change":"2021-02-10T04:29:38.663351+0000","last_active":"2021-02-10T04:36:52.429896+0000","last_peered":"2021-02-10T04:36:52.429896+0000","last_clean":"2021-02-10T04:36:52.429896+0000","last_became_active":"2021-02-10T04:25:49.009177+0000","last_became_peered":"2021-02-10T04:25:49.009177+0000","last_unstale":"2021-02-10T04:36:52.429896+0000","last_undegraded":"2021-02-10T04:36:52.429896+0000","last_fullsized":"2021-02-10T04:36:52.429896+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2021-02-10T04:29:38.663272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.663272+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":60,"num_read_kb":42,"num_write":32,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"574","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.419852+0000","last_change":"2021-02-10T04:28:37.515037+0000","last_active":"2021-02-10T04:36:51.419852+0000","last_peered":"2021-02-10T04:36:51.419852+0000","last_clean":"2021-02-10T04:36:51.419852+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:36:51.419852+0000","last_undegraded":"2021-02-10T04:36:51.419852+0000","last_fullsized":"2021-02-10T04:36:51.419852+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:37.514943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:37.514943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"58'2","reported_seq":"691","reported_epoch":"595","state":"active+clean","last_fresh":"2021-02-10T04:36:52.443245+0000","last_change":"2021-02-10T04:29:31.726022+0000","last_active":"2021-02-10T04:36:52.443245+0000","last_peered":"2021-02-10T04:36:52.443245+0000","last_clean":"2021-02-10T04:36:52.443245+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:36:52.443245+0000","last_undegraded":"2021-02-10T04:36:52.443245+0000","last_fullsized":"2021-02-10T04:36:52.443245+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:29:31.725932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.725932+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"481","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.421462+0000","last_change":"2021-02-10T04:29:26.437691+0000","last_active":"2021-02-10T04:36:51.421462+0000","last_peered":"2021-02-10T04:36:51.421462+0000","last_clean":"2021-02-10T04:36:51.421462+0000","last_became_active":"2021-02-10T04:29:07.150801+0000","last_became_peered":"2021-02-10T04:29:07.150801+0000","last_unstale":"2021-02-10T04:36:51.421462+0000","last_undegraded":"2021-02-10T04:36:51.421462+0000","last_fullsized":"2021-02-10T04:36:51.421462+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:26.437591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.437591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"538'43","reported_seq":"633","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.421752+0000","last_change":"2021-02-10T04:30:13.526264+0000","last_active":"2021-02-10T04:36:51.421752+0000","last_peered":"2021-02-10T04:36:51.421752+0000","last_clean":"2021-02-10T04:36:51.421752+0000","last_became_active":"2021-02-10T04:25:49.015760+0000","last_became_peered":"2021-02-10T04:25:49.015760+0000","last_unstale":"2021-02-10T04:36:51.421752+0000","last_undegraded":"2021-02-10T04:36:51.421752+0000","last_fullsized":"2021-02-10T04:36:51.421752+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'35","last_scrub_stamp":"2021-02-10T04:30:13.526170+0000","last_deep_scrub":"76'35","last_deep_scrub_stamp":"2021-02-10T04:26:58.198641+0000","last_clean_scrub_stamp":"2021-02-10T04:30:13.526170+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":62,"num_read_kb":42,"num_write":32,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"566","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.204260+0000","last_change":"2021-02-10T04:30:11.485059+0000","last_active":"2021-02-10T04:36:54.204260+0000","last_peered":"2021-02-10T04:36:54.204260+0000","last_clean":"2021-02-10T04:36:54.204260+0000","last_became_active":"2021-02-10T04:25:34.324981+0000","last_became_peered":"2021-02-10T04:25:34.324981+0000","last_unstale":"2021-02-10T04:36:54.204260+0000","last_undegraded":"2021-02-10T04:36:54.204260+0000","last_fullsized":"2021-02-10T04:36:54.204260+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:11.485004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.485004+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"557","reported_epoch":"595","state":"active+clean","last_fresh":"2021-02-10T04:36:52.429985+0000","last_change":"2021-02-10T04:29:23.589223+0000","last_active":"2021-02-10T04:36:52.429985+0000","last_peered":"2021-02-10T04:36:52.429985+0000","last_clean":"2021-02-10T04:36:52.429985+0000","last_became_active":"2021-02-10T04:25:51.022853+0000","last_became_peered":"2021-02-10T04:25:51.022853+0000","last_unstale":"2021-02-10T04:36:52.429985+0000","last_undegraded":"2021-02-10T04:36:52.429985+0000","last_fullsized":"2021-02-10T04:36:52.429985+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.589159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.589159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"239","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.422354+0000","last_change":"2021-02-10T04:33:05.142344+0000","last_active":"2021-02-10T04:36:51.422354+0000","last_peered":"2021-02-10T04:36:51.422354+0000","last_clean":"2021-02-10T04:36:51.422354+0000","last_became_active":"2021-02-10T04:33:05.141911+0000","last_became_peered":"2021-02-10T04:33:05.141911+0000","last_unstale":"2021-02-10T04:36:51.422354+0000","last_undegraded":"2021-02-10T04:36:51.422354+0000","last_fullsized":"2021-02-10T04:36:51.422354+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:25.075937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.075937+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"538'27","reported_seq":"602","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.200213+0000","last_change":"2021-02-10T04:28:46.680268+0000","last_active":"2021-02-10T04:36:54.200213+0000","last_peered":"2021-02-10T04:36:54.200213+0000","last_clean":"2021-02-10T04:36:54.200213+0000","last_became_active":"2021-02-10T04:25:49.010553+0000","last_became_peered":"2021-02-10T04:25:49.010553+0000","last_unstale":"2021-02-10T04:36:54.200213+0000","last_undegraded":"2021-02-10T04:36:54.200213+0000","last_fullsized":"2021-02-10T04:36:54.200213+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:28:46.680176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:46.680176+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"568","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.422437+0000","last_change":"2021-02-10T04:28:38.246850+0000","last_active":"2021-02-10T04:36:51.422437+0000","last_peered":"2021-02-10T04:36:51.422437+0000","last_clean":"2021-02-10T04:36:51.422437+0000","last_became_active":"2021-02-10T04:25:36.336442+0000","last_became_peered":"2021-02-10T04:25:36.336442+0000","last_unstale":"2021-02-10T04:36:51.422437+0000","last_undegraded":"2021-02-10T04:36:51.422437+0000","last_fullsized":"2021-02-10T04:36:51.422437+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:28:38.246768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:38.246768+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"242","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.200679+0000","last_change":"2021-02-10T04:33:05.147757+0000","last_active":"2021-02-10T04:36:55.200679+0000","last_peered":"2021-02-10T04:36:55.200679+0000","last_clean":"2021-02-10T04:36:55.200679+0000","last_became_active":"2021-02-10T04:33:05.146838+0000","last_became_peered":"2021-02-10T04:33:05.146838+0000","last_unstale":"2021-02-10T04:36:55.200679+0000","last_undegraded":"2021-02-10T04:36:55.200679+0000","last_fullsized":"2021-02-10T04:36:55.200679+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:22.078247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:22.078247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"557","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.202488+0000","last_change":"2021-02-10T04:29:31.903091+0000","last_active":"2021-02-10T04:36:55.202488+0000","last_peered":"2021-02-10T04:36:55.202488+0000","last_clean":"2021-02-10T04:36:55.202488+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:36:55.202488+0000","last_undegraded":"2021-02-10T04:36:55.202488+0000","last_fullsized":"2021-02-10T04:36:55.202488+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:31.902996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.902996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"538'28","reported_seq":"609","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.201016+0000","last_change":"2021-02-10T04:27:58.363322+0000","last_active":"2021-02-10T04:36:54.201016+0000","last_peered":"2021-02-10T04:36:54.201016+0000","last_clean":"2021-02-10T04:36:54.201016+0000","last_became_active":"2021-02-10T04:25:49.008997+0000","last_became_peered":"2021-02-10T04:25:49.008997+0000","last_unstale":"2021-02-10T04:36:54.201016+0000","last_undegraded":"2021-02-10T04:36:54.201016+0000","last_fullsized":"2021-02-10T04:36:54.201016+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'22","last_scrub_stamp":"2021-02-10T04:27:58.363248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:58.363248+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":40,"num_write":30,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"578","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.203266+0000","last_change":"2021-02-10T04:28:30.415204+0000","last_active":"2021-02-10T04:36:54.203266+0000","last_peered":"2021-02-10T04:36:54.203266+0000","last_clean":"2021-02-10T04:36:54.203266+0000","last_became_active":"2021-02-10T04:25:36.336893+0000","last_became_peered":"2021-02-10T04:25:36.336893+0000","last_unstale":"2021-02-10T04:36:54.203266+0000","last_undegraded":"2021-02-10T04:36:54.203266+0000","last_fullsized":"2021-02-10T04:36:54.203266+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:30.415134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:26:46.425288+0000","last_clean_scrub_stamp":"2021-02-10T04:28:30.415134+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"559","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.200260+0000","last_change":"2021-02-10T04:33:52.643172+0000","last_active":"2021-02-10T04:36:55.200260+0000","last_peered":"2021-02-10T04:36:55.200260+0000","last_clean":"2021-02-10T04:36:55.200260+0000","last_became_active":"2021-02-10T04:25:51.018872+0000","last_became_peered":"2021-02-10T04:25:51.018872+0000","last_unstale":"2021-02-10T04:36:55.200260+0000","last_undegraded":"2021-02-10T04:36:55.200260+0000","last_fullsized":"2021-02-10T04:36:55.200260+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:52.643099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:33:52.643099+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"555","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.201522+0000","last_change":"2021-02-10T04:29:26.355726+0000","last_active":"2021-02-10T04:36:54.201522+0000","last_peered":"2021-02-10T04:36:54.201522+0000","last_clean":"2021-02-10T04:36:54.201522+0000","last_became_active":"2021-02-10T04:25:53.147831+0000","last_became_peered":"2021-02-10T04:25:53.147831+0000","last_unstale":"2021-02-10T04:36:54.201522+0000","last_undegraded":"2021-02-10T04:36:54.201522+0000","last_fullsized":"2021-02-10T04:36:54.201522+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:26.355652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.355652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"538'31","reported_seq":"624","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.199266+0000","last_change":"2021-02-10T04:28:32.565457+0000","last_active":"2021-02-10T04:36:55.199266+0000","last_peered":"2021-02-10T04:36:55.199266+0000","last_clean":"2021-02-10T04:36:55.199266+0000","last_became_active":"2021-02-10T04:25:49.000911+0000","last_became_peered":"2021-02-10T04:25:49.000911+0000","last_unstale":"2021-02-10T04:36:55.199266+0000","last_undegraded":"2021-02-10T04:36:55.199266+0000","last_fullsized":"2021-02-10T04:36:55.199266+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2021-02-10T04:28:32.565374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:32.565374+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":62,"num_read_kb":46,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"481","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.421620+0000","last_change":"2021-02-10T04:33:48.646869+0000","last_active":"2021-02-10T04:36:51.421620+0000","last_peered":"2021-02-10T04:36:51.421620+0000","last_clean":"2021-02-10T04:36:51.421620+0000","last_became_active":"2021-02-10T04:29:07.151577+0000","last_became_peered":"2021-02-10T04:29:07.151577+0000","last_unstale":"2021-02-10T04:36:51.421620+0000","last_undegraded":"2021-02-10T04:36:51.421620+0000","last_fullsized":"2021-02-10T04:36:51.421620+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:48.646802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:33:48.646802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"556","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.421468+0000","last_change":"2021-02-10T04:28:46.464172+0000","last_active":"2021-02-10T04:36:51.421468+0000","last_peered":"2021-02-10T04:36:51.421468+0000","last_clean":"2021-02-10T04:36:51.421468+0000","last_became_active":"2021-02-10T04:25:51.025488+0000","last_became_peered":"2021-02-10T04:25:51.025488+0000","last_unstale":"2021-02-10T04:36:51.421468+0000","last_undegraded":"2021-02-10T04:36:51.421468+0000","last_fullsized":"2021-02-10T04:36:51.421468+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:46.464108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:46.464108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"555","reported_epoch":"595","state":"active+clean","last_fresh":"2021-02-10T04:36:52.427659+0000","last_change":"2021-02-10T04:28:44.719453+0000","last_active":"2021-02-10T04:36:52.427659+0000","last_peered":"2021-02-10T04:36:52.427659+0000","last_clean":"2021-02-10T04:36:52.427659+0000","last_became_active":"2021-02-10T04:25:53.039844+0000","last_became_peered":"2021-02-10T04:25:53.039844+0000","last_unstale":"2021-02-10T04:36:52.427659+0000","last_undegraded":"2021-02-10T04:36:52.427659+0000","last_fullsized":"2021-02-10T04:36:52.427659+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:44.719363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:44.719363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"538'14","reported_seq":"591","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.199692+0000","last_change":"2021-02-10T04:30:13.089576+0000","last_active":"2021-02-10T04:36:55.199692+0000","last_peered":"2021-02-10T04:36:55.199692+0000","last_clean":"2021-02-10T04:36:55.199692+0000","last_became_active":"2021-02-10T04:25:49.013196+0000","last_became_peered":"2021-02-10T04:25:49.013196+0000","last_unstale":"2021-02-10T04:36:55.199692+0000","last_undegraded":"2021-02-10T04:36:55.199692+0000","last_fullsized":"2021-02-10T04:36:55.199692+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2021-02-10T04:30:13.089494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:30:13.089494+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":15,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"481","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.424041+0000","last_change":"2021-02-10T04:30:11.537208+0000","last_active":"2021-02-10T04:36:51.424041+0000","last_peered":"2021-02-10T04:36:51.424041+0000","last_clean":"2021-02-10T04:36:51.424041+0000","last_became_active":"2021-02-10T04:29:07.157944+0000","last_became_peered":"2021-02-10T04:29:07.157944+0000","last_unstale":"2021-02-10T04:36:51.424041+0000","last_undegraded":"2021-02-10T04:36:51.424041+0000","last_fullsized":"2021-02-10T04:36:51.424041+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:11.537136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.537136+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"479","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.199398+0000","last_change":"2021-02-10T04:29:07.146984+0000","last_active":"2021-02-10T04:36:55.199398+0000","last_peered":"2021-02-10T04:36:55.199398+0000","last_clean":"2021-02-10T04:36:55.199398+0000","last_became_active":"2021-02-10T04:29:07.146525+0000","last_became_peered":"2021-02-10T04:29:07.146525+0000","last_unstale":"2021-02-10T04:36:55.199398+0000","last_undegraded":"2021-02-10T04:36:55.199398+0000","last_fullsized":"2021-02-10T04:36:55.199398+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:46.493066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:46.493066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"554","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.420051+0000","last_change":"2021-02-10T04:29:22.475613+0000","last_active":"2021-02-10T04:36:51.420051+0000","last_peered":"2021-02-10T04:36:51.420051+0000","last_clean":"2021-02-10T04:36:51.420051+0000","last_became_active":"2021-02-10T04:25:53.042601+0000","last_became_peered":"2021-02-10T04:25:53.042601+0000","last_unstale":"2021-02-10T04:36:51.420051+0000","last_undegraded":"2021-02-10T04:36:51.420051+0000","last_fullsized":"2021-02-10T04:36:51.420051+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:22.475504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:22.475504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"538'38","reported_seq":"624","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.199436+0000","last_change":"2021-02-10T04:28:35.520489+0000","last_active":"2021-02-10T04:36:55.199436+0000","last_peered":"2021-02-10T04:36:55.199436+0000","last_clean":"2021-02-10T04:36:55.199436+0000","last_became_active":"2021-02-10T04:25:49.010481+0000","last_became_peered":"2021-02-10T04:25:49.010481+0000","last_unstale":"2021-02-10T04:36:55.199436+0000","last_undegraded":"2021-02-10T04:36:55.199436+0000","last_fullsized":"2021-02-10T04:36:55.199436+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'32","last_scrub_stamp":"2021-02-10T04:28:35.520410+0000","last_deep_scrub":"78'32","last_deep_scrub_stamp":"2021-02-10T04:27:09.170428+0000","last_clean_scrub_stamp":"2021-02-10T04:28:35.520410+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":56,"num_read_kb":37,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"566","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.201515+0000","last_change":"2021-02-10T04:29:29.355356+0000","last_active":"2021-02-10T04:36:54.201515+0000","last_peered":"2021-02-10T04:36:54.201515+0000","last_clean":"2021-02-10T04:36:54.201515+0000","last_became_active":"2021-02-10T04:25:34.327162+0000","last_became_peered":"2021-02-10T04:25:34.327162+0000","last_unstale":"2021-02-10T04:36:54.201515+0000","last_undegraded":"2021-02-10T04:36:54.201515+0000","last_fullsized":"2021-02-10T04:36:54.201515+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.355257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.355257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"239","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.420260+0000","last_change":"2021-02-10T04:33:05.169577+0000","last_active":"2021-02-10T04:36:51.420260+0000","last_peered":"2021-02-10T04:36:51.420260+0000","last_clean":"2021-02-10T04:36:51.420260+0000","last_became_active":"2021-02-10T04:33:05.169159+0000","last_became_peered":"2021-02-10T04:33:05.169159+0000","last_unstale":"2021-02-10T04:36:51.420260+0000","last_undegraded":"2021-02-10T04:36:51.420260+0000","last_fullsized":"2021-02-10T04:36:51.420260+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:40.098239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:40.098239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"630","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.199185+0000","last_change":"2021-02-10T04:36:53.123742+0000","last_active":"2021-02-10T04:36:55.199185+0000","last_peered":"2021-02-10T04:36:55.199185+0000","last_clean":"2021-02-10T04:36:55.199185+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:36:55.199185+0000","last_undegraded":"2021-02-10T04:36:55.199185+0000","last_fullsized":"2021-02-10T04:36:55.199185+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:51.679515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:51.679515+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"538'56","reported_seq":"667","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.204544+0000","last_change":"2021-02-10T04:27:49.269619+0000","last_active":"2021-02-10T04:36:54.204544+0000","last_peered":"2021-02-10T04:36:54.204544+0000","last_clean":"2021-02-10T04:36:54.204544+0000","last_became_active":"2021-02-10T04:25:49.008451+0000","last_became_peered":"2021-02-10T04:25:49.008451+0000","last_unstale":"2021-02-10T04:36:54.204544+0000","last_undegraded":"2021-02-10T04:36:54.204544+0000","last_fullsized":"2021-02-10T04:36:54.204544+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'44","last_scrub_stamp":"2021-02-10T04:27:49.269539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:49.269539+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":107,"num_read_kb":84,"num_write":57,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"575","reported_epoch":"595","state":"active+clean","last_fresh":"2021-02-10T04:36:52.430266+0000","last_change":"2021-02-10T04:29:28.719237+0000","last_active":"2021-02-10T04:36:52.430266+0000","last_peered":"2021-02-10T04:36:52.430266+0000","last_clean":"2021-02-10T04:36:52.430266+0000","last_became_active":"2021-02-10T04:25:36.336112+0000","last_became_peered":"2021-02-10T04:25:36.336112+0000","last_unstale":"2021-02-10T04:36:52.430266+0000","last_undegraded":"2021-02-10T04:36:52.430266+0000","last_fullsized":"2021-02-10T04:36:52.430266+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.719147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.719147+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"554","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.198566+0000","last_change":"2021-02-10T04:27:49.628172+0000","last_active":"2021-02-10T04:36:55.198566+0000","last_peered":"2021-02-10T04:36:55.198566+0000","last_clean":"2021-02-10T04:36:55.198566+0000","last_became_active":"2021-02-10T04:25:51.029110+0000","last_became_peered":"2021-02-10T04:25:51.029110+0000","last_unstale":"2021-02-10T04:36:55.198566+0000","last_undegraded":"2021-02-10T04:36:55.198566+0000","last_fullsized":"2021-02-10T04:36:55.198566+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:49.628090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:49.628090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"555","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.203853+0000","last_change":"2021-02-10T04:29:25.382883+0000","last_active":"2021-02-10T04:36:54.203853+0000","last_peered":"2021-02-10T04:36:54.203853+0000","last_clean":"2021-02-10T04:36:54.203853+0000","last_became_active":"2021-02-10T04:25:53.150917+0000","last_became_peered":"2021-02-10T04:25:53.150917+0000","last_unstale":"2021-02-10T04:36:54.203853+0000","last_undegraded":"2021-02-10T04:36:54.203853+0000","last_fullsized":"2021-02-10T04:36:54.203853+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:25.382793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.382793+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.20","version":"0'0","reported_seq":"544","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.420414+0000","last_change":"2021-02-10T04:30:11.492843+0000","last_active":"2021-02-10T04:36:51.420414+0000","last_peered":"2021-02-10T04:36:51.420414+0000","last_clean":"2021-02-10T04:36:51.420414+0000","last_became_active":"2021-02-10T04:29:07.148937+0000","last_became_peered":"2021-02-10T04:29:07.148937+0000","last_unstale":"2021-02-10T04:36:51.420414+0000","last_undegraded":"2021-02-10T04:36:51.420414+0000","last_fullsized":"2021-02-10T04:36:51.420414+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:11.492788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:41.375276+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.492788+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":"597","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.198002+0000","last_change":"2021-02-10T04:29:21.708787+0000","last_active":"2021-02-10T04:36:54.198002+0000","last_peered":"2021-02-10T04:36:54.198002+0000","last_clean":"2021-02-10T04:36:54.198002+0000","last_became_active":"2021-02-10T04:28:39.045354+0000","last_became_peered":"2021-02-10T04:28:39.045354+0000","last_unstale":"2021-02-10T04:36:54.198002+0000","last_undegraded":"2021-02-10T04:36:54.198002+0000","last_fullsized":"2021-02-10T04:36:54.198002+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:21.708701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:21.708701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":"579","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.423606+0000","last_change":"2021-02-10T04:33:58.892594+0000","last_active":"2021-02-10T04:36:51.423606+0000","last_peered":"2021-02-10T04:36:51.423606+0000","last_clean":"2021-02-10T04:36:51.423606+0000","last_became_active":"2021-02-10T04:28:23.129832+0000","last_became_peered":"2021-02-10T04:28:23.129832+0000","last_unstale":"2021-02-10T04:36:51.423606+0000","last_undegraded":"2021-02-10T04:36:51.423606+0000","last_fullsized":"2021-02-10T04:36:51.423606+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:58.892504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:33:58.892504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":"520","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.203203+0000","last_change":"2021-02-10T04:28:48.409915+0000","last_active":"2021-02-10T04:36:54.203203+0000","last_peered":"2021-02-10T04:36:54.203203+0000","last_clean":"2021-02-10T04:36:54.203203+0000","last_became_active":"2021-02-10T04:28:23.129523+0000","last_became_peered":"2021-02-10T04:28:23.129523+0000","last_unstale":"2021-02-10T04:36:54.203203+0000","last_undegraded":"2021-02-10T04:36:54.203203+0000","last_fullsized":"2021-02-10T04:36:54.203203+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:48.409826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:48.409826+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":"523","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.421961+0000","last_change":"2021-02-10T04:34:01.895599+0000","last_active":"2021-02-10T04:36:51.421961+0000","last_peered":"2021-02-10T04:36:51.421961+0000","last_clean":"2021-02-10T04:36:51.421961+0000","last_became_active":"2021-02-10T04:28:23.126368+0000","last_became_peered":"2021-02-10T04:28:23.126368+0000","last_unstale":"2021-02-10T04:36:51.421961+0000","last_undegraded":"2021-02-10T04:36:51.421961+0000","last_fullsized":"2021-02-10T04:36:51.421961+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:34:01.895514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:34:01.895514+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.25","version":"53'2","reported_seq":"539","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.200425+0000","last_change":"2021-02-10T04:28:40.422198+0000","last_active":"2021-02-10T04:36:55.200425+0000","last_peered":"2021-02-10T04:36:55.200425+0000","last_clean":"2021-02-10T04:36:55.200425+0000","last_became_active":"2021-02-10T04:28:23.467770+0000","last_became_peered":"2021-02-10T04:28:23.467770+0000","last_unstale":"2021-02-10T04:36:55.200425+0000","last_undegraded":"2021-02-10T04:36:55.200425+0000","last_fullsized":"2021-02-10T04:36:55.200425+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:28:40.422115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:40.422115+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.26","version":"53'1","reported_seq":"509","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.199693+0000","last_change":"2021-02-10T04:29:22.801988+0000","last_active":"2021-02-10T04:36:54.199693+0000","last_peered":"2021-02-10T04:36:54.199693+0000","last_clean":"2021-02-10T04:36:54.199693+0000","last_became_active":"2021-02-10T04:28:39.043300+0000","last_became_peered":"2021-02-10T04:28:39.043300+0000","last_unstale":"2021-02-10T04:36:54.199693+0000","last_undegraded":"2021-02-10T04:36:54.199693+0000","last_fullsized":"2021-02-10T04:36:54.199693+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:29:22.801929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:22.801929+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":"550","reported_epoch":"597","state":"active+clean","last_fresh":"2021-02-10T04:36:55.202986+0000","last_change":"2021-02-10T04:29:07.146256+0000","last_active":"2021-02-10T04:36:55.202986+0000","last_peered":"2021-02-10T04:36:55.202986+0000","last_clean":"2021-02-10T04:36:55.202986+0000","last_became_active":"2021-02-10T04:29:07.145907+0000","last_became_peered":"2021-02-10T04:29:07.145907+0000","last_unstale":"2021-02-10T04:36:55.202986+0000","last_undegraded":"2021-02-10T04:36:55.202986+0000","last_fullsized":"2021-02-10T04:36:55.202986+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:39.388884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:39.388884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":"520","reported_epoch":"596","state":"active+clean","last_fresh":"2021-02-10T04:36:54.197496+0000","last_change":"2021-02-10T04:29:23.440451+0000","last_active":"2021-02-10T04:36:54.197496+0000","last_peered":"2021-02-10T04:36:54.197496+0000","last_clean":"2021-02-10T04:36:54.197496+0000","last_became_active":"2021-02-10T04:28:23.129469+0000","last_became_peered":"2021-02-10T04:28:23.129469+0000","last_unstale":"2021-02-10T04:36:54.197496+0000","last_undegraded":"2021-02-10T04:36:54.197496+0000","last_fullsized":"2021-02-10T04:36:54.197496+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.440391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.440391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":"518","reported_epoch":"594","state":"active+clean","last_fresh":"2021-02-10T04:36:51.420193+0000","last_change":"2021-02-10T04:28:45.220352+0000","last_active":"2021-02-10T04:36:51.420193+0000","last_peered":"2021-02-10T04:36:51.420193+0000","last_clean":"2021-02-10T04:36:51.420193+0000","last_became_active":"2021-02-10T04:28:23.126795+0000","last_became_peered":"2021-02-10T04:28:23.126795+0000","last_unstale":"2021-02-10T04:36:51.420193+0000","last_undegraded":"2021-02-10T04:36:51.420193+0000","last_fullsized":"2021-02-10T04:36:51.420193+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_clean_scrub_stamp":"2021-02-10T04:28:45.220258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":478708660,"num_objects":215,"num_object_clones":167,"num_object_copies":645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":215,"num_whiteouts":7,"num_read":4484,"num_read_kb":4614126,"num_write":3257,"num_write_kb":1227519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":982122496,"data_stored":1387014987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7845,"ondisk_log_size":7845,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":1522,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":17,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":5835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25,"ondisk_log_size":25,"up":126,"acting":126,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":237,"num_read_kb":232,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":1878,"num_read_kb":1351,"num_write":1020,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1068,"ondisk_log_size":1068,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748364938,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":92560,"kb_used_data":57996,"kb_used_omap":0,"kb_used_meta":34304,"kb_avail":93652592,"statfs":{"total":95995035648,"available":95900254208,"internally_reserved":0,"allocated":59387904,"data_stored":92910276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35127296},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:36:54 2021","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.752,"15min":1.2170000000000001},"min":{"1min":0.57199999999999995,"5min":0.495,"15min":0.441},"max":{"1min":7.4710000000000001,"5min":72.805999999999997,"15min":72.805999999999997},"last":0.70099999999999996},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.8300000000000001,"15min":1.27},"min":{"1min":0.52500000000000002,"5min":0.52000000000000002,"15min":0.50800000000000001},"max":{"1min":7.6970000000000001,"5min":72.721999999999994,"15min":72.721999999999994},"last":1.1479999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:36:54 2021","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.97399999999999998,"15min":1.0660000000000001},"min":{"1min":0.54700000000000004,"5min":0.52100000000000002,"15min":0.46700000000000003},"max":{"1min":2.2749999999999999,"5min":6.1239999999999997,"15min":41.966000000000001},"last":1.4339999999999999},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":1.129},"min":{"1min":0.64700000000000002,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.6279999999999999,"5min":3.5750000000000002,"15min":41.804000000000002},"last":0.997}]},{"osd":2,"last update":"Wed Feb 10 04:36:54 2021","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.9179999999999999,"15min":1.8680000000000001},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.496,"5min":48.174999999999997,"15min":125.714},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.2230000000000001,"5min":2.0659999999999998,"15min":1.9119999999999999},"min":{"1min":0.68799999999999994,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":4.4130000000000003,"5min":48.277999999999999,"15min":125.53700000000001},"last":0.86099999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:36:54 2021","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.879,"15min":2.5139999999999998},"min":{"1min":0.79800000000000004,"5min":0.72399999999999998,"15min":0.63600000000000001},"max":{"1min":4.1710000000000003,"5min":35.091000000000001,"15min":192.95500000000001},"last":1.081},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.873,"15min":2.5979999999999999},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.69999999999999996},"max":{"1min":6.5300000000000002,"5min":35.061999999999998,"15min":192.601},"last":1.173}]},{"osd":4,"last update":"Wed Feb 10 04:36:54 2021","interfaces":[{"interface":"back","average":{"1min":5.4989999999999997,"5min":2.133,"15min":1.421},"min":{"1min":0.75900000000000001,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":97.733999999999995,"5min":97.733999999999995,"15min":97.733999999999995},"last":0.78400000000000003},{"interface":"front","average":{"1min":5.6600000000000001,"5min":2.1899999999999999,"15min":1.492},"min":{"1min":0.73399999999999999,"5min":0.69599999999999995,"15min":0.64600000000000002},"max":{"1min":97.691000000000003,"5min":97.691000000000003,"15min":97.691000000000003},"last":1.2090000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:36:54 2021","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.613,"15min":1.351},"min":{"1min":0.83499999999999996,"5min":0.79100000000000004,"15min":0.624},"max":{"1min":2.5609999999999999,"5min":32.713999999999999,"15min":32.713999999999999},"last":2.0960000000000001},{"interface":"front","average":{"1min":1.706,"5min":1.786,"15min":1.4530000000000001},"min":{"1min":0.92900000000000005,"5min":0.753,"15min":0.753},"max":{"1min":6.6219999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":0.92900000000000005}]},{"osd":6,"last update":"Wed Feb 10 04:36:54 2021","interfaces":[{"interface":"back","average":{"1min":1.5680000000000001,"5min":1.8879999999999999,"15min":1.6459999999999999},"min":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.86899999999999999},"max":{"1min":3.218,"5min":32.616,"15min":32.616},"last":1.5309999999999999},{"interface":"front","average":{"1min":1.6699999999999999,"5min":1.825,"15min":1.599},"min":{"1min":0.95399999999999996,"5min":0.79500000000000004,"15min":0.72499999999999998},"max":{"1min":4.282,"5min":32.424999999999997,"15min":32.424999999999997},"last":1.2849999999999999}]}]},{"osd":6,"up_from":43,"seq":184683593870,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":175932,"kb_used_data":148788,"kb_used_omap":0,"kb_used_meta":27136,"kb_avail":93569220,"statfs":{"total":95995035648,"available":95814881280,"internally_reserved":0,"allocated":152358912,"data_stored":220691286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27787264},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:36:39 2021","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":1.2969999999999999,"15min":1.389},"min":{"1min":0.53000000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.282,"5min":40.652000000000001,"15min":64.164000000000001},"last":1.2729999999999999},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.833,"15min":1.6180000000000001},"min":{"1min":0.503,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.712,"5min":51.039999999999999,"15min":64.277000000000001},"last":1.4530000000000001}]},{"osd":1,"last update":"Wed Feb 10 04:36:39 2021","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.96699999999999997,"15min":1.1020000000000001},"min":{"1min":0.47699999999999998,"5min":0.443,"15min":0.443},"max":{"1min":2.3239999999999998,"5min":8.641,"15min":18.177},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.052,"15min":1.0900000000000001},"min":{"1min":0.60899999999999999,"5min":0.57799999999999996,"15min":0.52700000000000002},"max":{"1min":1.5169999999999999,"5min":2.4540000000000002,"15min":18.898},"last":1.2330000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:36:39 2021","interfaces":[{"interface":"back","average":{"1min":2.2040000000000002,"5min":1.4199999999999999,"15min":1.2509999999999999},"min":{"1min":0.72399999999999998,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":24.725000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":0.875},{"interface":"front","average":{"1min":2.0529999999999999,"5min":1.361,"15min":1.234},"min":{"1min":0.67300000000000004,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":24.611999999999998,"5min":24.611999999999998,"15min":24.611999999999998},"last":1.042}]},{"osd":3,"last update":"Wed Feb 10 04:36:39 2021","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.377,"15min":1.3080000000000001},"min":{"1min":0.754,"5min":0.754,"15min":0.66100000000000003},"max":{"1min":5.1630000000000003,"5min":5.1630000000000003,"15min":15.316000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.292,"15min":1.284},"min":{"1min":0.83599999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.9970000000000001,"5min":3.3300000000000001,"15min":15.35},"last":1.159}]},{"osd":4,"last update":"Wed Feb 10 04:36:39 2021","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.29,"15min":1.1699999999999999},"min":{"1min":0.621,"5min":0.621,"15min":0.58999999999999997},"max":{"1min":1.806,"5min":2.8090000000000002,"15min":3.4969999999999999},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.012,"15min":0.96999999999999997},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.57999999999999996},"max":{"1min":1.76,"5min":2.1560000000000001,"15min":2.1560000000000001},"last":0.90000000000000002}]},{"osd":5,"last update":"Wed Feb 10 04:36:39 2021","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2190000000000001,"15min":1.1240000000000001},"min":{"1min":0.75900000000000001,"5min":0.70899999999999996,"15min":0.64100000000000001},"max":{"1min":2.1429999999999998,"5min":3.629,"15min":5.3529999999999998},"last":1.113},{"interface":"front","average":{"1min":1.274,"5min":1.347,"15min":1.2929999999999999},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":2.286,"5min":2.532,"15min":2.532},"last":1.373}]},{"osd":7,"last update":"Wed Feb 10 04:36:02 2021","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.2450000000000001,"15min":1.1899999999999999},"min":{"1min":0.83999999999999997,"5min":0.76900000000000002,"15min":0.73599999999999999},"max":{"1min":4.2599999999999998,"5min":4.2599999999999998,"15min":4.2599999999999998},"last":1.01},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.482,"15min":1.466},"min":{"1min":1.0109999999999999,"5min":0.92700000000000005,"15min":0.89000000000000001},"max":{"1min":4.7699999999999996,"5min":4.7699999999999996,"15min":4.7699999999999996},"last":1.645}]}]},{"osd":1,"up_from":13,"seq":55834575007,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":163312,"kb_used_data":135976,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":93581840,"statfs":{"total":95995035648,"available":95827804160,"internally_reserved":0,"allocated":139239424,"data_stored":188995225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:36:11 2021","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.5780000000000001,"15min":1.1000000000000001},"min":{"1min":0.51500000000000001,"5min":0.502,"15min":0.36399999999999999},"max":{"1min":2.8079999999999998,"5min":40.539999999999999,"15min":40.539999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.651,"15min":1.177},"min":{"1min":0.53500000000000003,"5min":0.53200000000000003,"15min":0.41199999999999998},"max":{"1min":2.4020000000000001,"5min":40.831000000000003,"15min":40.831000000000003},"last":0.92500000000000004}]},{"osd":2,"last update":"Wed Feb 10 04:36:33 2021","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.5469999999999999,"15min":1.175},"min":{"1min":0.58899999999999997,"5min":0.53800000000000003,"15min":0.48699999999999999},"max":{"1min":2.2480000000000002,"5min":40.268000000000001,"15min":40.268000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":1.141,"5min":1.5800000000000001,"15min":1.2250000000000001},"min":{"1min":0.58299999999999996,"5min":0.55600000000000005,"15min":0.46000000000000002},"max":{"1min":3.77,"5min":40.651000000000003,"15min":40.651000000000003},"last":1.03}]},{"osd":3,"last update":"Wed Feb 10 04:36:48 2021","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.9430000000000001,"15min":1.3169999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.45200000000000001},"max":{"1min":2.5579999999999998,"5min":40.314,"15min":40.314},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.9390000000000001,"15min":1.3540000000000001},"min":{"1min":0.83299999999999996,"5min":0.66200000000000003,"15min":0.47199999999999998},"max":{"1min":1.8280000000000001,"5min":40.973999999999997,"15min":40.973999999999997},"last":1.0589999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:36:11 2021","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.23,"15min":1.198},"min":{"1min":0.76800000000000002,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":2.556,"5min":4.3739999999999997,"15min":4.3739999999999997},"last":1.298},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.349,"15min":1.2669999999999999},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.65700000000000003},"max":{"1min":2.7029999999999998,"5min":6.2869999999999999,"15min":6.2869999999999999},"last":1.1919999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:36:27 2021","interfaces":[{"interface":"back","average":{"1min":2.0539999999999998,"5min":1.484,"15min":1.3140000000000001},"min":{"1min":0.76600000000000001,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":14.539999999999999,"5min":14.539999999999999,"15min":14.539999999999999},"last":1.006},{"interface":"front","average":{"1min":2.0449999999999999,"5min":1.5900000000000001,"15min":1.5660000000000001},"min":{"1min":0.83799999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":14.564,"5min":14.564,"15min":14.564},"last":1.4370000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:36:38 2021","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.617,"15min":1.413},"min":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.80300000000000005},"max":{"1min":2.9369999999999998,"5min":12.395,"15min":12.395},"last":1.468},{"interface":"front","average":{"1min":1.387,"5min":1.5349999999999999,"15min":1.359},"min":{"1min":0.91500000000000004,"5min":0.80100000000000005,"15min":0.78500000000000003},"max":{"1min":5.4649999999999999,"5min":13.183,"15min":13.183},"last":1.397}]},{"osd":7,"last update":"Wed Feb 10 04:36:48 2021","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.534,"15min":1.573},"min":{"1min":1.026,"5min":1.026,"15min":0.83499999999999996},"max":{"1min":2.0550000000000002,"5min":3.6840000000000002,"15min":7.5540000000000003},"last":1.5569999999999999},{"interface":"front","average":{"1min":1.577,"5min":1.6200000000000001,"15min":1.581},"min":{"1min":1.1739999999999999,"5min":0.94899999999999995,"15min":0.94899999999999995},"max":{"1min":2.4969999999999999,"5min":6.1689999999999996,"15min":8.1020000000000003},"last":1.321}]}]},{"osd":0,"up_from":8,"seq":34359738533,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":176480,"kb_used_data":147164,"kb_used_omap":0,"kb_used_meta":29056,"kb_avail":93568672,"statfs":{"total":95995035648,"available":95814320128,"internally_reserved":0,"allocated":150695936,"data_stored":220467338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29753344},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 04:36:06 2021","interfaces":[{"interface":"back","average":{"1min":1.716,"5min":1.0589999999999999,"15min":0.85799999999999998},"min":{"1min":0.61499999999999999,"5min":0.48199999999999998,"15min":0.40799999999999997},"max":{"1min":14.781000000000001,"5min":14.781000000000001,"15min":14.781000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.91000000000000003,"15min":0.876},"min":{"1min":0.58099999999999996,"5min":0.53600000000000003,"15min":0.45000000000000001},"max":{"1min":1.8149999999999999,"5min":2.306,"15min":7.5549999999999997},"last":0.94999999999999996}]},{"osd":2,"last update":"Wed Feb 10 04:36:31 2021","interfaces":[{"interface":"back","average":{"1min":1.9239999999999999,"5min":1.3360000000000001,"15min":1.054},"min":{"1min":0.66600000000000004,"5min":0.55500000000000005,"15min":0.48099999999999998},"max":{"1min":16.254000000000001,"5min":16.254000000000001,"15min":16.254000000000001},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.865,"5min":1.2270000000000001,"15min":0.97999999999999998},"min":{"1min":0.58899999999999997,"5min":0.52800000000000002,"15min":0.495},"max":{"1min":16.829000000000001,"5min":16.829000000000001,"15min":16.829000000000001},"last":0.79900000000000004}]},{"osd":3,"last update":"Wed Feb 10 04:36:48 2021","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.256,"15min":1.4850000000000001},"min":{"1min":0.626,"5min":0.55200000000000005,"15min":0.46999999999999997},"max":{"1min":2.3660000000000001,"5min":4.9829999999999997,"15min":126.587},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.569,"5min":1.6299999999999999,"15min":1.7430000000000001},"min":{"1min":1.02,"5min":0.68600000000000005,"15min":0.54200000000000004},"max":{"1min":2.1600000000000001,"5min":16.66,"15min":126.88200000000001},"last":1.2490000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:36:01 2021","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.256,"15min":1.1899999999999999},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.8939999999999999,"5min":2.532,"15min":2.6440000000000001},"last":1.365},{"interface":"front","average":{"1min":1.472,"5min":1.3069999999999999,"15min":1.2509999999999999},"min":{"1min":0.75600000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":3.887,"5min":13.891,"15min":13.891},"last":1.0529999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:36:20 2021","interfaces":[{"interface":"back","average":{"1min":1.6259999999999999,"5min":1.4910000000000001,"15min":1.345},"min":{"1min":0.77800000000000002,"5min":0.76300000000000001,"15min":0.69599999999999995},"max":{"1min":3.9940000000000002,"5min":13.949999999999999,"15min":13.949999999999999},"last":1.341},{"interface":"front","average":{"1min":2.5070000000000001,"5min":1.823,"15min":1.496},"min":{"1min":1.1619999999999999,"5min":0.90900000000000003,"15min":0.81699999999999995},"max":{"1min":16.957000000000001,"5min":16.957000000000001,"15min":16.957000000000001},"last":1.3859999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:36:42 2021","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.929,"15min":1.597},"min":{"1min":0.996,"5min":0.88800000000000001,"15min":0.88800000000000001},"max":{"1min":2.8559999999999999,"5min":35.872999999999998,"15min":35.872999999999998},"last":1.22},{"interface":"front","average":{"1min":2.4300000000000002,"5min":2.194,"15min":1.671},"min":{"1min":1.3440000000000001,"5min":0.88200000000000001,"15min":0.83099999999999996},"max":{"1min":16.890000000000001,"5min":35.911000000000001,"15min":35.911000000000001},"last":1.587}]},{"osd":7,"last update":"Wed Feb 10 04:36:51 2021","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":1.4770000000000001,"15min":1.3480000000000001},"min":{"1min":1.0629999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":2.3679999999999999,"5min":2.3679999999999999,"15min":2.9249999999999998},"last":1.4490000000000001},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.728,"15min":1.579},"min":{"1min":1.0720000000000001,"5min":1.006,"15min":0.94999999999999996},"max":{"1min":1.8700000000000001,"5min":16.545000000000002,"15min":16.545000000000002},"last":1.157}]}]},{"osd":2,"up_from":18,"seq":77309411486,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":113652,"kb_used_data":90972,"kb_used_omap":0,"kb_used_meta":21184,"kb_avail":93631500,"statfs":{"total":95995035648,"available":95878656000,"internally_reserved":0,"allocated":93155328,"data_stored":123165245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21692416},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,8],"upper_bound":1048576},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:36:27 2021","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.448,"15min":0.99199999999999999},"min":{"1min":0.53700000000000003,"5min":0.51100000000000001,"15min":0.48699999999999999},"max":{"1min":6.1580000000000004,"5min":27.315000000000001,"15min":27.315000000000001},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.407,"5min":1.46,"15min":1.012},"min":{"1min":0.57199999999999995,"5min":0.52800000000000002,"15min":0.46000000000000002},"max":{"1min":5.9379999999999997,"5min":27.286000000000001,"15min":27.286000000000001},"last":1.2370000000000001}]},{"osd":1,"last update":"Wed Feb 10 04:36:27 2021","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":1.0980000000000001,"15min":0.95999999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.518,"5min":4.2240000000000002,"15min":5.0650000000000004},"last":1.5800000000000001},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.1859999999999999,"15min":0.89500000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":2.1110000000000002,"5min":10.093999999999999,"15min":10.093999999999999},"last":1.4279999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:36:49 2021","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.556,"15min":1.123},"min":{"1min":0.72599999999999998,"5min":0.59399999999999997,"15min":0.53500000000000003},"max":{"1min":5.6310000000000002,"5min":27.152999999999999,"15min":27.152999999999999},"last":2.0880000000000001},{"interface":"front","average":{"1min":1.6379999999999999,"5min":1.774,"15min":1.333},"min":{"1min":0.83999999999999997,"5min":0.78500000000000003,"15min":0.66600000000000004},"max":{"1min":5.532,"5min":27.454999999999998,"15min":27.454999999999998},"last":2.1930000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:35:59 2021","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":2.4670000000000001,"15min":1.6399999999999999},"min":{"1min":0.83499999999999996,"5min":0.752,"15min":0.72199999999999998},"max":{"1min":1.9490000000000001,"5min":106.42100000000001,"15min":106.42100000000001},"last":1.792},{"interface":"front","average":{"1min":1.093,"5min":1.282,"15min":1.1910000000000001},"min":{"1min":0.73899999999999999,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.736,"5min":9.7609999999999992,"15min":9.7609999999999992},"last":2.5990000000000002}]},{"osd":5,"last update":"Wed Feb 10 04:36:27 2021","interfaces":[{"interface":"back","average":{"1min":1.8260000000000001,"5min":1.6259999999999999,"15min":1.427},"min":{"1min":1.0229999999999999,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":5.125,"5min":5.125,"15min":5.125},"last":2.7709999999999999},{"interface":"front","average":{"1min":1.387,"5min":1.571,"15min":1.3340000000000001},"min":{"1min":0.83699999999999997,"5min":0.628,"15min":0.628},"max":{"1min":2.2629999999999999,"5min":21.879999999999999,"15min":21.879999999999999},"last":2.694}]},{"osd":6,"last update":"Wed Feb 10 04:36:39 2021","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":3.0640000000000001,"15min":1.9079999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":55.145000000000003,"5min":106.13200000000001,"15min":106.13200000000001},"last":2.548},{"interface":"front","average":{"1min":4.3390000000000004,"5min":3.1400000000000001,"15min":2.0179999999999998},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":55.101999999999997,"5min":106.568,"15min":106.568},"last":1.9079999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:35:52 2021","interfaces":[{"interface":"back","average":{"1min":1.71,"5min":2.9180000000000001,"15min":1.966},"min":{"1min":1.151,"5min":0.94099999999999995,"15min":0.94099999999999995},"max":{"1min":2.613,"5min":106.706,"15min":106.706},"last":1.99},{"interface":"front","average":{"1min":1.3,"5min":1.4710000000000001,"15min":1.335},"min":{"1min":1.03,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":2.113,"5min":9.9220000000000006,"15min":9.9220000000000006},"last":2.73}]}]},{"osd":3,"up_from":25,"seq":107374182552,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":257404,"kb_used_data":212856,"kb_used_omap":0,"kb_used_meta":44288,"kb_avail":93487748,"statfs":{"total":95995035648,"available":95731453952,"internally_reserved":0,"allocated":217964544,"data_stored":311340432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45350912},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:35:54 2021","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.0309999999999999,"15min":0.92800000000000005},"min":{"1min":0.55700000000000005,"5min":0.47599999999999998,"15min":0.46100000000000002},"max":{"1min":1.597,"5min":6.3869999999999996,"15min":17.233000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.046,"15min":1.079},"min":{"1min":0.58899999999999997,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":3.423,"5min":3.423,"15min":20.216999999999999},"last":0.62}]},{"osd":1,"last update":"Wed Feb 10 04:35:54 2021","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":1.089,"15min":0.99099999999999999},"min":{"1min":0.65700000000000003,"5min":0.57399999999999995,"15min":0.52800000000000002},"max":{"1min":1.4810000000000001,"5min":3.2629999999999999,"15min":5.9130000000000003},"last":0.876},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.995,"15min":0.93200000000000005},"min":{"1min":0.61099999999999999,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.5700000000000001,"5min":2.5699999999999998,"15min":5.4039999999999999},"last":0.71499999999999997}]},{"osd":2,"last update":"Wed Feb 10 04:35:54 2021","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.21,"15min":1.081},"min":{"1min":0.73199999999999998,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.9930000000000001,"5min":4.1210000000000004,"15min":4.649},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.1919999999999999,"15min":1.0449999999999999},"min":{"1min":0.76700000000000002,"5min":0.622,"15min":0.56999999999999995},"max":{"1min":1.764,"5min":3.0779999999999998,"15min":3.0779999999999998},"last":0.90100000000000002}]},{"osd":4,"last update":"Wed Feb 10 04:36:00 2021","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.3420000000000001,"15min":1.4079999999999999},"min":{"1min":0.80600000000000005,"5min":0.629,"15min":0.629},"max":{"1min":1.7110000000000001,"5min":3.351,"15min":24.882999999999999},"last":1.022},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.415,"15min":1.4279999999999999},"min":{"1min":0.79000000000000004,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.5329999999999999,"5min":4.0510000000000002,"15min":24.911999999999999},"last":0.84899999999999998}]},{"osd":5,"last update":"Wed Feb 10 04:36:24 2021","interfaces":[{"interface":"back","average":{"1min":2.2000000000000002,"5min":2.052,"15min":1.679},"min":{"1min":0.872,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":17.701000000000001,"5min":35.616999999999997,"15min":35.616999999999997},"last":1.6879999999999999},{"interface":"front","average":{"1min":2.274,"5min":2.1459999999999999,"15min":1.724},"min":{"1min":0.85999999999999999,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":17.361999999999998,"5min":35.661999999999999,"15min":35.661999999999999},"last":1.097}]},{"osd":6,"last update":"Wed Feb 10 04:36:44 2021","interfaces":[{"interface":"back","average":{"1min":2.976,"5min":2.3730000000000002,"15min":1.909},"min":{"1min":1.2829999999999999,"5min":0.96799999999999997,"15min":0.83699999999999997},"max":{"1min":17.882000000000001,"5min":35.518000000000001,"15min":35.518000000000001},"last":1.1619999999999999},{"interface":"front","average":{"1min":3.1240000000000001,"5min":2.956,"15min":2.097},"min":{"1min":1.282,"5min":0.92400000000000004,"15min":0.86599999999999999},"max":{"1min":17.015999999999998,"5min":53.965000000000003,"15min":53.965000000000003},"last":1.2470000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:35:54 2021","interfaces":[{"interface":"back","average":{"1min":1.5740000000000001,"5min":1.7030000000000001,"15min":1.7190000000000001},"min":{"1min":1.119,"5min":1.0609999999999999,"15min":0.99199999999999999},"max":{"1min":2.2949999999999999,"5min":4.3460000000000001,"15min":25.242999999999999},"last":1.232},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.7749999999999999,"15min":1.7569999999999999},"min":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"max":{"1min":2.1179999999999999,"5min":4.3239999999999998,"15min":25.286999999999999},"last":1.3899999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986326,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86812,"kb_used_data":67736,"kb_used_omap":0,"kb_used_meta":18816,"kb_avail":93658340,"statfs":{"total":95995035648,"available":95906140160,"internally_reserved":0,"allocated":69361664,"data_stored":80737100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19267584},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4],"upper_bound":1048576},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:36:03 2021","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":0.84899999999999998,"15min":1.131},"min":{"1min":0.54400000000000004,"5min":0.501,"15min":0.47099999999999997},"max":{"1min":3.4430000000000001,"5min":6.1059999999999999,"15min":62.273000000000003},"last":1.0980000000000001},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.91500000000000004,"15min":1.1619999999999999},"min":{"1min":0.53300000000000003,"5min":0.499,"15min":0.499},"max":{"1min":3.5739999999999998,"5min":6.2649999999999997,"15min":62.375999999999998},"last":1.196}]},{"osd":1,"last update":"Wed Feb 10 04:36:03 2021","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.85199999999999998,"15min":0.879},"min":{"1min":0.56999999999999995,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":3.3519999999999999,"5min":5.6699999999999999,"15min":5.6699999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.01,"15min":1.0409999999999999},"min":{"1min":0.74299999999999999,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.9830000000000001,"5min":2.9830000000000001,"15min":2.9830000000000001},"last":0.92500000000000004}]},{"osd":2,"last update":"Wed Feb 10 04:36:03 2021","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.0640000000000001,"15min":1.369},"min":{"1min":0.61899999999999999,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":2.8130000000000002,"5min":2.8130000000000002,"15min":62.521000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.167,"15min":1.4610000000000001},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.60699999999999998},"max":{"1min":3.1640000000000001,"5min":3.8919999999999999,"15min":62.572000000000003},"last":0.86099999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:36:03 2021","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.26,"15min":1.405},"min":{"1min":0.65900000000000003,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":3.6739999999999999,"5min":3.956,"15min":62.113},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.323,"5min":1.216,"15min":1.391},"min":{"1min":0.82499999999999996,"5min":0.72199999999999998,"15min":0.69899999999999995},"max":{"1min":3.9049999999999998,"5min":3.9049999999999998,"15min":62.146000000000001},"last":1.0289999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:36:21 2021","interfaces":[{"interface":"back","average":{"1min":1.998,"5min":1.448,"15min":1.149},"min":{"1min":0.748,"5min":0.71799999999999997,"15min":0.57499999999999996},"max":{"1min":12.773999999999999,"5min":12.773999999999999,"15min":12.773999999999999},"last":1.0029999999999999},{"interface":"front","average":{"1min":2.0430000000000001,"5min":1.349,"15min":1.099},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.624},"max":{"1min":13.038,"5min":13.038,"15min":13.038},"last":1.623}]},{"osd":6,"last update":"Wed Feb 10 04:36:33 2021","interfaces":[{"interface":"back","average":{"1min":2.488,"5min":1.6020000000000001,"15min":1.446},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.63100000000000001},"max":{"1min":20.001000000000001,"5min":20.001000000000001,"15min":29.686},"last":1.2609999999999999},{"interface":"front","average":{"1min":2.5979999999999999,"5min":1.738,"15min":1.5940000000000001},"min":{"1min":1.0940000000000001,"5min":0.83899999999999997,"15min":0.79000000000000004},"max":{"1min":20.292999999999999,"5min":20.292999999999999,"15min":29.510000000000002},"last":1.738}]},{"osd":7,"last update":"Wed Feb 10 04:36:56 2021","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.4099999999999999,"15min":1.3560000000000001},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.843,"5min":6.3440000000000003,"15min":6.3440000000000003},"last":1.548},{"interface":"front","average":{"1min":1.429,"5min":1.4750000000000001,"15min":1.4550000000000001},"min":{"1min":0.99199999999999999,"5min":0.88700000000000001,"15min":0.83099999999999996},"max":{"1min":2.4319999999999999,"5min":4.0780000000000003,"15min":4.0780000000000003},"last":1.6679999999999999}]}]},{"osd":5,"up_from":37,"seq":158913790098,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":136896,"kb_used_data":101048,"kb_used_omap":0,"kb_used_meta":35840,"kb_avail":93608256,"statfs":{"total":95995035648,"available":95854854144,"internally_reserved":0,"allocated":103473152,"data_stored":144105566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36700160},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:36:19 2021","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":2.1890000000000001,"15min":1.3939999999999999},"min":{"1min":0.53100000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.8260000000000001,"5min":104.04300000000001,"15min":104.04300000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.997,"5min":2.25,"15min":1.3779999999999999},"min":{"1min":0.52600000000000002,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.72,"5min":107.776,"15min":107.776},"last":1.1379999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:36:19 2021","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":2.0630000000000002,"15min":1.298},"min":{"1min":0.53700000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.4079999999999999,"5min":103.904,"15min":103.904},"last":1.085},{"interface":"front","average":{"1min":1.0109999999999999,"5min":2.1760000000000002,"15min":1.401},"min":{"1min":0.55300000000000005,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.3759999999999999,"5min":104.315,"15min":104.315},"last":0.85899999999999999}]},{"osd":2,"last update":"Wed Feb 10 04:36:19 2021","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":3.137,"15min":2.0259999999999998},"min":{"1min":0.63,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.895,"5min":103.65600000000001,"15min":103.65600000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":1.141,"5min":3.1890000000000001,"15min":1.9910000000000001},"min":{"1min":0.71199999999999997,"5min":0.63800000000000001,"15min":0.63200000000000001},"max":{"1min":1.74,"5min":104.136,"15min":104.136},"last":1.3029999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:36:19 2021","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":2.7829999999999999,"15min":2.0139999999999998},"min":{"1min":0.748,"5min":0.624,"15min":0.624},"max":{"1min":4.4530000000000003,"5min":106.18000000000001,"15min":106.18000000000001},"last":1.042},{"interface":"front","average":{"1min":1.306,"5min":2.7160000000000002,"15min":1.946},"min":{"1min":0.84899999999999998,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.9550000000000001,"5min":106.822,"15min":106.822},"last":1.2050000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:36:19 2021","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.1020000000000001,"15min":1.05},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.653,"5min":1.881,"15min":5.2290000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.361,"5min":1.2629999999999999,"15min":1.1910000000000001},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.60799999999999998},"max":{"1min":2.5129999999999999,"5min":2.5129999999999999,"15min":8.1449999999999996},"last":1.175}]},{"osd":6,"last update":"Wed Feb 10 04:36:37 2021","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.6890000000000001,"15min":1.431},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":2.8719999999999999,"5min":25.969000000000001,"15min":25.969000000000001},"last":1.25},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.5800000000000001,"15min":1.3049999999999999},"min":{"1min":0.82199999999999995,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.9780000000000002,"5min":25.902999999999999,"15min":25.902999999999999},"last":1.956}]},{"osd":7,"last update":"Wed Feb 10 04:35:56 2021","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.361,"15min":1.2749999999999999},"min":{"1min":0.878,"5min":0.84199999999999997,"15min":0.78400000000000003},"max":{"1min":4.6699999999999999,"5min":5.4770000000000003,"15min":5.4770000000000003},"last":1.274},{"interface":"front","average":{"1min":1.6339999999999999,"5min":1.583,"15min":1.49},"min":{"1min":1.137,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":2.4020000000000001,"5min":8.7840000000000007,"15min":8.7840000000000007},"last":1.3540000000000001}]}]}]}} 2021-02-10T04:37:04.425 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:37:04.479 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[33677]: debug 2021-02-10T04:37:04.179+0000 7fbec19a1700 -1 osd.4 605 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:04.479 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[44544]: debug 2021-02-10T04:37:04.479 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[44544]: 2021-02-10T04:37:04.266+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:37:04.479 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[44544]: 2021-02-10T04:37:04.480 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: cluster 2021-02-10T04:37:03.169817+0000 2021-02-10T04:37:04.480 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: osd.4 (osd.4) 921 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:04.480 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: cluster 2021-02-10T04:37:03.169827+0000 osd.4 ( 2021-02-10T04:37:04.480 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: osd.4) 922 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:04.480 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:37:04.480 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: -02-10T04:37:03.169833+0000 osd.4 (osd.4) 923 : cluster 2021-02-10T04:37:04.480 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:04.480 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:37:04.481 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: T04:37:03.169839+0000 osd.4 (osd.4 2021-02-10T04:37:04.481 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: ) 924 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:04.481 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:37:04.481 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: -10T04:37:03.173521+0000 osd.2 ( 2021-02-10T04:37:04.481 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: osd.2) 1801 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:04.481 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:37:04.481 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: 02-10T04:37:03.173530+0000 osd.2 2021-02-10T04:37:04.481 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: (osd.2) 1802 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:04.481 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:37:04.481 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: 02-10T04:37:03.173538+0000 osd.2 2021-02-10T04:37:04.481 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: (osd.2) 1803 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:04.481 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: cluster 2021-02-10T04:37:04.481 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: 2021-02-10T04:37 2021-02-10T04:37:04.482 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: :03.173544+0000 osd.2 (osd.2) 1804 : cluster 2021-02-10T04:37:04.482 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:04.482 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: cluster 2021-02-10T04:37 2021-02-10T04:37:04.482 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: :03.173550+0000 osd.2 (osd.2) 1805 : cluster 2021-02-10T04:37:04.482 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:04.482 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: cluster 2021-02-10T04:37:03. 2021-02-10T04:37:04.482 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: 173555+0000 osd.2 (osd.2) 1806 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:04.482 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: 2021-02-10T04:37:04.482 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: cluster 2021-02-10T04:37:04.482 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: 2021-02-10T04:37:03.173562+0000 osd.2 (osd. 2021-02-10T04:37:04.482 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: 2) 1807 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:04.482 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:37:04.482 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: -10T04:37:03.173572+0000 osd.2 (osd. 2021-02-10T04:37:04.483 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: 2) 1808 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:04.483 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:37:04.483 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: -10T04:37:03.173579+0000 osd.2 ( 2021-02-10T04:37:04.483 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: osd.2) 1809 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:04.483 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:37:04.483 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: 10T04:37:03.173584+0000 osd.2 (osd 2021-02-10T04:37:04.483 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: .2) 1810 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:04.483 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: cluster 2021-02-10T04:37:04.483 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:37:04.483 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: :37:04.130091+0000 mgr.y (mgr.14140 2021-02-10T04:37:04.483 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:04 smithi164 conmon[29288]: ) 658 : cluster [DBG] pgmap v1058: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 455 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.0 MiB/s rd, 5 op/s 2021-02-10T04:37:04.484 INFO:tasks.rados.rados.0.smithi085.stdout:2327: done (13 left) 2021-02-10T04:37:04.484 INFO:tasks.rados.rados.0.smithi085.stdout:2328: done (12 left) 2021-02-10T04:37:04.484 INFO:tasks.rados.rados.0.smithi085.stdout:2330: done (11 left) 2021-02-10T04:37:04.484 INFO:tasks.rados.rados.0.smithi085.stdout:2331: done (10 left) 2021-02-10T04:37:04.484 INFO:tasks.rados.rados.0.smithi085.stdout:2332: done (9 left) 2021-02-10T04:37:04.484 INFO:tasks.rados.rados.0.smithi085.stdout:2333: snap_create 2021-02-10T04:37:04.605 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:37:04.679 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: cluster 2021-02-10T04:37:04.679 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: 2021-02-10T04:37:04.679 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: 2021 2021-02-10T04:37:04.679 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: - 2021-02-10T04:37:04.679 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: 02 2021-02-10T04:37:04.679 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: - 2021-02-10T04:37:04.679 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: 10 2021-02-10T04:37:04.679 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: T 2021-02-10T04:37:04.679 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: 04 2021-02-10T04:37:04.680 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: : 2021-02-10T04:37:04.680 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: 37 2021-02-10T04:37:04.680 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: : 2021-02-10T04:37:04.680 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: 03 2021-02-10T04:37:04.680 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: . 2021-02-10T04:37:04.680 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: 169817 2021-02-10T04:37:04.680 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: +0000 2021-02-10T04:37:04.680 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: 2021-02-10T04:37:04.680 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: osd.4 2021-02-10T04:37:04.680 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: ( 2021-02-10T04:37:04.680 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: osd 2021-02-10T04:37:04.680 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: . 2021-02-10T04:37:04.680 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: 4 2021-02-10T04:37:04.680 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: ) 921 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:04.681 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: cluster 2021-02-10T04:37:04.681 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: 2021-02-10T04:37:03.169827+0000 2021-02-10T04:37:04.681 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: osd.4 (osd.4) 922 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:04.681 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: 2021-02-10T04:37:04.681 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: cluster 2021-02-10T04:37:04.681 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: 2021-02-10T04:37:03.169833+0000 osd.4 ( 2021-02-10T04:37:04.681 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: osd.4) 923 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:04.681 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:04.681 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: 02-10T04:37:03.169839+0000 2021-02-10T04:37:04.681 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: osd.4 (osd.4) 924 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:04.681 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: 2021-02-10T04:37:04.681 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: cluster 2021-02-10T04:37:04.681 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: 2021-02-10T04:37:03.173521+0000 2021-02-10T04:37:04.682 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: osd.2 (osd.2) 1801 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:04.682 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:37:04.682 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: T04:37:03.173530+0000 osd.2 (osd.2) 1802 : cluster 2021-02-10T04:37:04.682 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:04.682 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: cluster 2021-02-10T04:37:04.682 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: 2021-02-10T04:37:03. 2021-02-10T04:37:04.682 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: 173538+0000 osd.2 (osd.2) 1803 : cluster [WRN] 2021-02-10T04:37:04.682 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:04.682 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:37:04.682 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: T04:37:03.173544+0000 osd.2 (osd. 2021-02-10T04:37:04.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: cluster 2021-02-10T04:37:03.169817+0000 osd.4 ( 2021-02-10T04:37:04.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: osd.4) 921 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:04.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: cluster 2021-02-10T04:37:03.169827+0000 osd.4 (osd.4) 922 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:04.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: cluster 2021-02-10T04:37:03.169833+0000 osd.4 (osd.4) 923 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:04.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: cluster 2021-02-10T04:37:03.169839+0000 osd.4 (osd.4) 924 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:04.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: cluster 2021-02-10T04:37:04.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 2021-02-10T04:37:03.173521+0000 osd.2 (osd.2) 1801 : 2021-02-10T04:37:04.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:04.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:04.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 02-10T04:37:03.173530+0000 osd.2 (osd.2) 1802 : cluster [WRN] 2021-02-10T04:37:04.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:04.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: cluster 2021-02-10T04:37:04.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 2021-02-10T04:37:04.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 2021 2021-02-10T04:37:04.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: - 2021-02-10T04:37:04.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 02 2021-02-10T04:37:04.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: - 2021-02-10T04:37:04.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 10 2021-02-10T04:37:04.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: T 2021-02-10T04:37:04.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 04 2021-02-10T04:37:04.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: : 2021-02-10T04:37:04.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 37 2021-02-10T04:37:04.684 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: : 2021-02-10T04:37:04.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 03 2021-02-10T04:37:04.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: . 2021-02-10T04:37:04.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 173538 2021-02-10T04:37:04.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: +0000 2021-02-10T04:37:04.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 2021-02-10T04:37:04.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: osd.2 2021-02-10T04:37:04.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: ( 2021-02-10T04:37:04.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: osd 2021-02-10T04:37:04.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: .2) 1803 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:04.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 2021-02-10T04:37:04.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: cluster 2021-02-10T04:37:04.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 2021-02-10T04:37:03.173544+0000 osd.2 (osd. 2021-02-10T04:37:04.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 2) 1804 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:04.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: cluster 2021-02-10T04:37:04.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 2021-02-10T04:37:03.173550 2021-02-10T04:37:04.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1805 : cluster 2021-02-10T04:37:04.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:04.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:37:04.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 04:37:03.173555+0000 osd.2 (osd. 2021-02-10T04:37:04.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 2) 1806 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:04.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:37:04.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 04:37:03.173562+0000 osd.2 (osd.2) 2021-02-10T04:37:04.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 1807 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:04.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: cluster 2021-02-10T04:37:04.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 2021 2021-02-10T04:37:04.687 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: - 2021-02-10T04:37:04.688 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 02 2021-02-10T04:37:04.688 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: - 2021-02-10T04:37:04.688 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 10 2021-02-10T04:37:04.688 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: T 2021-02-10T04:37:04.688 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 04 2021-02-10T04:37:04.688 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: : 2021-02-10T04:37:04.688 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 37 2021-02-10T04:37:04.689 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: : 2021-02-10T04:37:04.689 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 03 2021-02-10T04:37:04.689 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: . 2021-02-10T04:37:04.689 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 173572 2021-02-10T04:37:04.689 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: +0000 2021-02-10T04:37:04.689 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 2021-02-10T04:37:04.689 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: osd.2 2021-02-10T04:37:04.689 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: ( 2021-02-10T04:37:04.689 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: osd 2021-02-10T04:37:04.689 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: . 2021-02-10T04:37:04.689 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 2 2021-02-10T04:37:04.689 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: ) 2021-02-10T04:37:04.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 1808 2021-02-10T04:37:04.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: : 2021-02-10T04:37:04.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: cluster 2021-02-10T04:37:04.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 2021-02-10T04:37:04.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: [WRN] 2021-02-10T04:37:04.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 2021-02-10T04:37:04.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:04.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: 2021-02-10T04:37:04.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:37:04.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: T04:37:03.173579+0000 osd.2 (osd.2) 1809 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:04.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:37:04.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: -02-10T04:37:03.173584+0000 2021-02-10T04:37:04.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: osd.2 (osd.2) 1810 : cluster [WRN] 2021-02-10T04:37:04.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:04.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: cluster 2021-02-10T04:37:04 2021-02-10T04:37:04.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: .130091+0000 mgr.y (mgr.14140) 658 : 2021-02-10T04:37:04.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[29319]: cluster [DBG] pgmap v1058: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 455 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.0 MiB/s rd, 5 op/s 2021-02-10T04:37:04.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: 2) 1804 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:04.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:04.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: 02-10T04:37:03.173550+0000 osd.2 (osd.2) 1805 2021-02-10T04:37:04.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:04.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: cluster 2021-02-10T04:37:04.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:37:04.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: :37:03.173555+0000 osd.2 (osd.2) 2021-02-10T04:37:04.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: 1806 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:04.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:04.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: -02-10T04:37:03.173562+0000 2021-02-10T04:37:04.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: osd.2 (osd.2) 1807 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:04.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: cluster 2021-02-10T04:37 2021-02-10T04:37:04.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: :03.173572+0000 osd.2 (osd.2) 1808 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:04.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:37:04.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: 04:37:03.173579+0000 2021-02-10T04:37:04.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: osd.2 (osd.2) 1809 : cluster 2021-02-10T04:37:04.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:04.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: cluster 2021-02-10T04:37:04.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: 2021-02-10T04:37:03 2021-02-10T04:37:04.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: .173584+0000 osd.2 (osd.2 2021-02-10T04:37:04.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: ) 1810 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:04.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:04.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: 02-10T04:37:04.130091+0000 mgr.y ( 2021-02-10T04:37:04.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:04 smithi085 conmon[35961]: mgr.14140) 658 : cluster [DBG] pgmap v1058: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 455 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.0 MiB/s rd, 5 op/s 2021-02-10T04:37:04.762 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:37:05.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[49106]: debug 2021-02-10T04:37:05.623 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[49106]: 2021-02-10T04:37:05.207+0000 7f77a3e94700 -1 osd.2 606 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:37:05.623 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[49106]: 2021-02-10T04:37:05.624 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[33677]: debug 2021-02-10T04:37:05.195+0000 7fbec19a1700 -1 osd.4 606 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:05.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: cluster 2021-02-10T04:37:05.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: 2021-02-10T04:37:05.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:37:05.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: :37:04.180618+0000 osd.4 (osd.4) 925 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:05.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: cluster 2021-02-10T04:37:04.180627+0000 osd.4 (osd.4) 926 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:05.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: cluster 2021-02-10T04:37:05.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: 2021-02-10 2021-02-10T04:37:05.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: T04:37:04.180633+0000 osd.4 (osd.4) 2021-02-10T04:37:05.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: 927 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:05.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: cluster 2021-02-10T04:37:05.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: 2021-02-10T04:37:04.180639 2021-02-10T04:37:05.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 928 : cluster [WRN] 2021-02-10T04:37:05.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:05.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: cluster 2021-02-10T04:37:04 2021-02-10T04:37:05.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: .199923+0000 osd.2 (osd.2) 1811 : cluster 2021-02-10T04:37:05.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:05.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: cluster 2021-02-10T04:37:04 2021-02-10T04:37:05.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: .199932+0000 osd.2 (osd.2) 1812 : 2021-02-10T04:37:05.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:05.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:37:05.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: :37:04.199938+0000 osd.2 (osd.2 2021-02-10T04:37:05.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: ) 1813 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:05.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: cluster 2021-02-10T04:37:05.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: 2021-02-10T04:37:04.199944+0000 2021-02-10T04:37:05.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: osd.2 (osd.2) 1814 : cluster [WRN] 2021-02-10T04:37:05.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:05.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:37:05.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: 02-10T04:37:04.199950+0000 osd.2 2021-02-10T04:37:05.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: (osd.2) 1815 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:05.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: 2021-02-10T04:37:05.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: cluster 2021-02-10T04:37: 2021-02-10T04:37:05.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: 04.199955+0000 osd.2 (osd.2) 1816 2021-02-10T04:37:05.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:05.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:37:05.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: 10T04:37:04.199961+0000 osd.2 ( 2021-02-10T04:37:05.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: osd.2) 1817 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:05.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:37:05.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: -02-10T04:37:04.199966+0000 2021-02-10T04:37:05.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: osd.2 (osd.2) 1818 : cluster [WRN] 2021-02-10T04:37:05.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:05.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: cluster 2021-02-10T04:37: 2021-02-10T04:37:05.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: 04.199972+0000 osd.2 (osd.2) 1819 2021-02-10T04:37:05.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:05.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:37:05.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: 10T04:37:04.199979+0000 osd.2 ( 2021-02-10T04:37:05.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: osd.2) 1820 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:05.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:37:05.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: 37:04.482074+0000 mon.a (mon.0) 1460 : cluster [DBG] 2021-02-10T04:37:05.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: osdmap e606: 8 total, 8 up, 8 in 2021-02-10T04:37:05.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: cluster 2021-02-10T04:37:05.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: 2021-02-10T04:37 2021-02-10T04:37:05.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: :04.740314+0000 osd.5 (osd.5) 69 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:37:05.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: cluster 2021-02-10T04:37:05.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: 2021-02-10T04:37: 2021-02-10T04:37:05.629 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: 04.740464+0000 osd.5 (osd.5) 2021-02-10T04:37:05.629 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:05 smithi164 conmon[29288]: 70 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:37:05.629 INFO:tasks.rados.rados.0.smithi085.stdout:2333: done (9 left) 2021-02-10T04:37:05.629 INFO:tasks.rados.rados.0.smithi085.stdout:2334: read oid 22 snap -1 2021-02-10T04:37:05.629 INFO:tasks.rados.rados.0.smithi085.stdout:2334: expect (ObjNum 765 snap 246 seq_num 765) 2021-02-10T04:37:05.629 INFO:tasks.rados.rados.0.smithi085.stdout:2335: read oid 14 snap -1 2021-02-10T04:37:05.629 INFO:tasks.rados.rados.0.smithi085.stdout:2335: expect (ObjNum 753 snap 241 seq_num 753) 2021-02-10T04:37:05.629 INFO:tasks.rados.rados.0.smithi085.stdout:2336: read oid 44 snap 244 2021-02-10T04:37:05.629 INFO:tasks.rados.rados.0.smithi085.stdout:2336: expect (ObjNum 760 snap 243 seq_num 760) 2021-02-10T04:37:05.629 INFO:tasks.rados.rados.0.smithi085.stdout:2337: copy_from oid 6 from oid 20 current snap is 264 2021-02-10T04:37:05.629 INFO:tasks.rados.rados.0.smithi085.stdout:2338: write (excl) oid 1 current snap is 264 2021-02-10T04:37:05.629 INFO:tasks.rados.rados.0.smithi085.stdout:2338: seq_num 806 ranges {746442=584112,1748369=722605,2983991=562451} 2021-02-10T04:37:05.629 INFO:tasks.rados.rados.0.smithi085.stdout:2338: writing smithi08575397-1 from 746442 to 1330554 tid 1 2021-02-10T04:37:05.630 INFO:tasks.rados.rados.0.smithi085.stdout:2338: writing smithi08575397-1 from 1748369 to 2470974 tid 2 2021-02-10T04:37:05.630 INFO:tasks.rados.rados.0.smithi085.stdout:2338: writing smithi08575397-1 from 2983991 to 3546442 tid 3 2021-02-10T04:37:05.630 INFO:tasks.rados.rados.0.smithi085.stdout:2339: write (excl) oid 2 current snap is 264 2021-02-10T04:37:05.630 INFO:tasks.rados.rados.0.smithi085.stdout:2339: seq_num 807 ranges {763249=659361,2090498=582616,3295061=268188} 2021-02-10T04:37:05.630 INFO:tasks.rados.rados.0.smithi085.stdout:2339: writing smithi08575397-2 from 763249 to 1422610 tid 1 2021-02-10T04:37:05.630 INFO:tasks.rados.rados.0.smithi085.stdout:2339: writing smithi08575397-2 from 2090498 to 2673114 tid 2 2021-02-10T04:37:05.630 INFO:tasks.rados.rados.0.smithi085.stdout:2339: writing smithi08575397-2 from 3295061 to 3563249 tid 3 2021-02-10T04:37:05.630 INFO:tasks.rados.rados.0.smithi085.stdout:2340: snap_remove snap 229 2021-02-10T04:37:05.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:37:05.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: 04:37:04.180618+0000 osd.4 (osd.4) 925 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:05.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:37:05.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: 10T04:37:04.180627+0000 osd.4 (osd 2021-02-10T04:37:05.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: .4) 926 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:05.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:05.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: -02-10T04:37:04.180633+0000 2021-02-10T04:37:05.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: osd.4 (osd.4) 927 : cluster [WRN] 2021-02-10T04:37:05.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:05.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: cluster 2021-02-10T04:37:05.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: 2021-02-10T04:37:04 2021-02-10T04:37:05.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: .180639+0000 osd.4 (osd.4) 928 : cluster 2021-02-10T04:37:05.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:05.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: cluster 2021-02-10T04:37:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: 2021-02-10T04:37:04. 2021-02-10T04:37:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: 199923+0000 osd.2 (osd.2) 1811 : cluster 2021-02-10T04:37:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:37:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: -10T04:37:04.199932+0000 osd.2 (osd. 2021-02-10T04:37:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: 2) 1812 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: cluster 2021-02-10T04:37:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: 2021 2021-02-10T04:37:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: - 2021-02-10T04:37:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: 02 2021-02-10T04:37:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: - 2021-02-10T04:37:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: 10 2021-02-10T04:37:05.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: T 2021-02-10T04:37:05.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: 04 2021-02-10T04:37:05.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: : 2021-02-10T04:37:05.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: 37 2021-02-10T04:37:05.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: : 2021-02-10T04:37:05.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: 04 2021-02-10T04:37:05.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: . 2021-02-10T04:37:05.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: 199938 2021-02-10T04:37:05.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: +0000 2021-02-10T04:37:05.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: 2021-02-10T04:37:05.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: osd.2 2021-02-10T04:37:05.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: ( 2021-02-10T04:37:05.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: osd 2021-02-10T04:37:05.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: . 2021-02-10T04:37:05.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: 2 2021-02-10T04:37:05.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: ) 2021-02-10T04:37:05.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: 1813 2021-02-10T04:37:05.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: : 2021-02-10T04:37:05.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: cluster 2021-02-10T04:37:05.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: 2021-02-10T04:37:05.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: [WRN] 2021-02-10T04:37:05.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: 2021-02-10T04:37:05.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:05.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: 2021-02-10T04:37:05.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: cluster 2021-02-10T04:37:05.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: 2021-02-10T04:37:05.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: 2021 2021-02-10T04:37:05.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: - 2021-02-10T04:37:05.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: 02 2021-02-10T04:37:05.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: - 2021-02-10T04:37:05.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: 10 2021-02-10T04:37:05.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: T04:37:04.199944+0000 osd.2 (osd.2) 1814 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:05.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: cluster 2021-02-10T04:37:04.199950+0000 osd.2 (osd.2) 1815 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:05.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: cluster 2021-02-10T04:37:04.199955+0000 osd.2 (osd.2) 1816 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:05.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: cluster 2021-02-10T04:37:04.199961+0000 osd.2 (osd.2) 1817 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:05.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:37:05.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: :37:04.199966+0000 osd.2 (osd.2) 1818 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:05.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: cluster 2021-02-10T04:37: 2021-02-10T04:37:05.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: 04.199972+0000 osd.2 (osd.2) 1819 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:05.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:37:05.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: :37:04.199979+0000 osd.2 (osd.2) 1820 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:05.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: cluster 2021-02-10T04:37:05.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: 2021-02-10T04:37:05.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: 2021 2021-02-10T04:37:05.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: - 2021-02-10T04:37:05.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: 02 2021-02-10T04:37:05.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: - 2021-02-10T04:37:05.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: 10 2021-02-10T04:37:05.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: T04:37:04.482074+0000 2021-02-10T04:37:05.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: mon.a (mon.0) 1460 : cluster [DBG] osdmap e606: 8 total, 8 up, 8 in 2021-02-10T04:37:05.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: cluster 2021-02-10T04:37:04. 2021-02-10T04:37:05.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: 740314+0000 osd.5 (osd.5) 69 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:37:05.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: 2021-02-10T04:37:05.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: cluster 2021-02-10T04:37:04.740464+0000 osd.5 ( 2021-02-10T04:37:05.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[35961]: osd.5) 70 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:37:05.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: cluster 2021-02-10T04:37:04.180618+0000 2021-02-10T04:37:05.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: osd.4 (osd.4) 925 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:05.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: cluster 2021-02-10T04:37:04.180627+0000 osd.4 (osd.4) 926 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:05.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: cluster 2021-02-10T04:37:04.180633+0000 osd.4 (osd.4) 927 : 2021-02-10T04:37:05.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:05.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: cluster 2021-02-10T04:37:04.180639+0000 osd.4 2021-02-10T04:37:05.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: (osd.4) 928 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:05.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: cluster 2021-02-10T04:37:04.199923+0000 osd.2 2021-02-10T04:37:05.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: (osd.2) 1811 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:05.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: cluster 2021-02-10T04:37:04.199932+0000 osd.2 (osd 2021-02-10T04:37:05.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: .2) 1812 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:05.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:05.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: 02-10T04:37:04.199938+0000 2021-02-10T04:37:05.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: osd.2 (osd.2) 1813 : cluster 2021-02-10T04:37:05.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:05.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:05.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: 02-10T04:37:04.199944+0000 2021-02-10T04:37:05.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: osd.2 (osd.2) 1814 : 2021-02-10T04:37:05.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:05.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: cluster 2021-02-10T04:37:05.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: 2021-02-10T04:37:04.199950+0000 2021-02-10T04:37:05.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: osd.2 (osd.2) 1815 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:05.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: 2021-02-10T04:37:05.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: cluster 2021-02-10T04:37:05.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:37:05.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: 10T04:37:04.199955+0000 osd.2 (osd.2) 1816 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:05.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: 2021-02-10T04:37:05.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: cluster 2021-02-10T04:37:05.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:37:05.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: T04:37:04.199961 2021-02-10T04:37:05.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1817 2021-02-10T04:37:05.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:05.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: 2021-02-10T04:37:05.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:37:05.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: -10T04:37 2021-02-10T04:37:05.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: :04.199966+0000 osd.2 (osd.2) 1818 2021-02-10T04:37:05.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:05.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: cluster 2021-02-10T04:37:05.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: 2021-02-10T04:37 2021-02-10T04:37:05.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: :04.199972+0000 osd.2 ( 2021-02-10T04:37:05.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: osd.2) 1819 : cluster [WRN] 2021-02-10T04:37:05.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:05.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:37:05.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: T04:37:04.199979+0000 osd.2 2021-02-10T04:37:05.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: (osd.2) 1820 : cluster 2021-02-10T04:37:05.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:05.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: cluster 2021-02-10T04:37:05.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: 2021-02-10T04:37: 2021-02-10T04:37:05.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: 04.482074+0000 mon.a (mon.0) 1460 2021-02-10T04:37:05.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: : cluster [DBG] osdmap e606: 8 total, 8 up, 8 in 2021-02-10T04:37:05.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:05.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: 02-10T04:37:04.740314+0000 2021-02-10T04:37:05.953 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: osd.5 (osd.5) 69 : 2021-02-10T04:37:05.953 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: cluster [DBG] purged_snaps scrub starts 2021-02-10T04:37:05.953 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: cluster 2021-02-10T04:37:05.953 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: 2021-02-10T04:37 2021-02-10T04:37:05.953 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: :04.740464+0000 osd.5 (osd 2021-02-10T04:37:05.953 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:05 smithi085 conmon[29319]: .5) 70 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:37:06.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[49106]: debug 2021-02-10T04:37:06.158+0000 7f77a3e94700 -1 osd.2 607 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:37:06.497 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[33677]: debug 2021-02-10T04:37:06.161+0000 7fbec19a1700 -1 osd.4 607 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:06.497 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: cluster 2021-02-10T04:37:06.506 INFO:tasks.rados.rados.0.smithi085.stdout:2336: expect (ObjNum 760 snap 243 seq_num 760) 2021-02-10T04:37:06.632 INFO:tasks.rados.rados.0.smithi085.stdout:2335: expect (ObjNum 753 snap 241 seq_num 753) 2021-02-10T04:37:06.752 INFO:tasks.rados.rados.0.smithi085.stdout:2334: expect (ObjNum 765 snap 246 seq_num 765) 2021-02-10T04:37:06.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: 2021-02-10T04:37:07.372 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: 2021 2021-02-10T04:37:07.372 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: - 2021-02-10T04:37:07.372 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: 02 2021-02-10T04:37:07.372 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: - 2021-02-10T04:37:07.373 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: 10 2021-02-10T04:37:07.373 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: T 2021-02-10T04:37:07.373 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: 04 2021-02-10T04:37:07.373 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: :37:05.196407+0000 osd.4 2021-02-10T04:37:07.373 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: (osd.4) 929 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:07.373 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:37:07.373 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: :37:05.196417+0000 osd.4 (osd.4) 930 : cluster 2021-02-10T04:37:07.373 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:07.373 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:37:07.373 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: 37:05.196422+0000 osd.4 (osd.4) 931 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:07.373 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:37:07.373 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: 04:37:05.196425+0000 osd.4 (osd.4) 932 2021-02-10T04:37:07.373 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:07.374 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:37:07.374 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: :37:05.209202+0000 osd.2 (osd.2) 2021-02-10T04:37:07.374 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: 1821 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:07.374 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:37:07.374 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: -10T04:37:05.209211+0000 osd.2 ( 2021-02-10T04:37:07.374 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: osd.2) 1822 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:07.374 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:37:07.374 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: -02-10T04:37: 2021-02-10T04:37:07.374 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: 05.209217+0000 osd.2 (osd 2021-02-10T04:37:07.374 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: .2) 1823 : cluster [WRN] 2021-02-10T04:37:07.374 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:07.374 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:37:07.375 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: 04:37:05.209223+0000 2021-02-10T04:37:07.375 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: osd.2 (osd.2) 1824 : cluster 2021-02-10T04:37:07.375 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:07.375 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: cluster 2021-02-10T04:37:07.375 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:37:07.375 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: :37:05.209229+0000 osd.2 (osd 2021-02-10T04:37:07.375 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: .2) 1825 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:07.375 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:37:07.375 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: -10T04:37:05.209237 2021-02-10T04:37:07.375 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1826 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:07.375 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: 2021-02-10T04:37:07.375 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:37:07.375 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: 02-10T04:37:05.209243+0000 osd.2 (osd.2 2021-02-10T04:37:07.375 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: ) 1827 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:07.376 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: cluster 2021-02-10T04:37:05 2021-02-10T04:37:07.376 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: .209248+0000 osd.2 (osd.2) 1828 : cluster 2021-02-10T04:37:07.376 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:07.376 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: cluster 2021-02-10T04:37:07.376 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: 2021-02-10T04:37:05.209255+0000 osd.2 (osd.2) 1829 : 2021-02-10T04:37:07.376 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:07.376 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:37:07.376 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: -02-10T04:37:05.209260+0000 osd.2 (osd.2) 1830 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:07.376 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: cluster 2021-02-10T04:37:05.479196 2021-02-10T04:37:07.376 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1461 : cluster [DBG] osdmap e607: 8 total, 8 up, 8 in 2021-02-10T04:37:07.376 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: cluster 2021-02-10T04:37:06.130820+0000 2021-02-10T04:37:07.376 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: mgr.y (mgr.14140) 659 : cluster [DBG] pgmap v1061: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 462 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.0 MiB/s wr, 10 op/s 2021-02-10T04:37:07.376 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: cluster 2021-02-10T04:37:06.160278+0000 2021-02-10T04:37:07.377 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: osd.2 (osd.2) 1831 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:07.377 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: 2021-02-10T04:37:07.377 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:37:07.377 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: 04:37:06.160287+0000 osd.2 (osd.2) 1832 2021-02-10T04:37:07.377 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:07.377 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:37:07.377 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: 02-10T04:37:06.160297+0000 osd.2 (osd. 2021-02-10T04:37:07.377 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: 2) 1833 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:07.377 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:37:07.377 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: -02-10T04:37:06.160303+0000 osd.2 (osd.2) 1834 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:07.377 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: cluster 2021-02-10T04:37:06.160309+0000 2021-02-10T04:37:07.377 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: osd.2 (osd.2) 1835 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:07.377 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: cluster 2021-02-10T04:37:06.160316+0000 osd.2 (osd.2 2021-02-10T04:37:07.378 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: ) 1836 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:07.378 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: cluster 2021-02-10T04:37: 2021-02-10T04:37:07.378 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: 06.160324+0000 osd.2 (osd.2) 1837 : cluster 2021-02-10T04:37:07.378 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:07.378 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: cluster 2021-02-10T04:37:06.160331 2021-02-10T04:37:07.378 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1838 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:07.378 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: 2021-02-10T04:37:07.378 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: cluster 2021-02-10T04:37:07.378 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: 2021-02-10T04:37:06.160337+0000 osd.2 ( 2021-02-10T04:37:07.378 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: osd.2) 1839 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:07.378 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:37:07.378 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: T04:37:06.160352+0000 osd.2 (osd.2 2021-02-10T04:37:07.378 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:06 smithi164 conmon[29288]: ) 1840 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:07.380 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: cluster 2021-02-10T04:37:07.380 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: 2021-02-10T04:37:07.380 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: 2021 2021-02-10T04:37:07.380 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: - 2021-02-10T04:37:07.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: 02 2021-02-10T04:37:07.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: - 2021-02-10T04:37:07.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: 10T04:37:05.196407 2021-02-10T04:37:07.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 929 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:07.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: cluster 2021-02-10T04:37:05.196417+0000 osd.4 2021-02-10T04:37:07.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: (osd.4) 930 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:07.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:37:07.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: T04:37:05.196422+0000 osd.4 (osd. 2021-02-10T04:37:07.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: 4) 931 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:07.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:37:07.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: 04:37:05.196425+0000 osd.4 (osd. 2021-02-10T04:37:07.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: 4) 932 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:07.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: cluster 2021-02-10T04:37:05 2021-02-10T04:37:07.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: .209202+0000 osd.2 (osd.2) 1821 : cluster [WRN] 2021-02-10T04:37:07.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:07.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:37:07.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: -10T04:37:05.209211+0000 osd.2 (osd.2) 1822 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:07.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:37:07.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: 04:37:05.209217+0000 osd.2 (osd. 2021-02-10T04:37:07.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: 2) 1823 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:07.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: cluster 2021-02-10T04:37:05.209223+0000 osd.2 (osd.2) 1824 : cluster 2021-02-10T04:37:07.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:07.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:37:07.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: 37:05.209229+0000 osd.2 (osd.2) 2021-02-10T04:37:07.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: 1825 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:07.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: cluster 2021-02-10T04:37 2021-02-10T04:37:07.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: :05.209237+0000 osd.2 (osd.2) 1826 2021-02-10T04:37:07.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:07.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:37:07.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: :37:05.209243+0000 osd.2 (osd.2) 2021-02-10T04:37:07.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: 1827 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:07.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:37:07.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: 04:37:05.209248+0000 osd.2 (osd.2 2021-02-10T04:37:07.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: ) 1828 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:07.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:37:07.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: 37:05.209255+0000 osd.2 (osd.2) 1829 : cluster 2021-02-10T04:37:07.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:07.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: cluster 2021-02-10T04:37: 2021-02-10T04:37:07.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: 05.209260+0000 osd.2 (osd.2) 1830 : cluster 2021-02-10T04:37:07.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:07.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:37:07.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: 10T04:37:05.479196+0000 2021-02-10T04:37:07.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: mon.a (mon.0) 1461 : cluster [DBG] osdmap e607: 8 total, 8 up, 8 in 2021-02-10T04:37:07.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: 2021-02-10T04:37:07.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: cluster 2021-02-10T04:37:07.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: 2021-02-10T04:37:06.130820+0000 mgr.y 2021-02-10T04:37:07.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: (mgr.14140) 659 : cluster [DBG] pgmap v1061: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 462 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.0 MiB/s wr, 10 op/s 2021-02-10T04:37:07.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: 2021-02-10T04:37:07.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:37:07.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: -10T04:37:06.160278+0000 osd.2 (osd.2 2021-02-10T04:37:07.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: ) 1831 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:07.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: cluster 2021-02-10T04:37 2021-02-10T04:37:07.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: :06.160287+0000 osd.2 (osd.2) 1832 : 2021-02-10T04:37:07.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:07.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: cluster 2021-02-10T04:37:06.160297+0000 osd.2 2021-02-10T04:37:07.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: (osd.2) 1833 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:07.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: cluster 2021-02-10T04:37:07.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: 2021-02-10T04:37:06.160303+0000 osd.2 (osd.2) 1834 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:07.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: 2021-02-10T04:37:07.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:07.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: 02-10T04:37:06.160309+0000 osd.2 (osd.2) 1835 : cluster [WRN] 2021-02-10T04:37:07.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:07.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:37:07.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: T04:37:06.160316+0000 osd.2 (osd.2) 1836 : cluster 2021-02-10T04:37:07.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:07.385 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: cluster 2021-02-10T04:37:07.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: 2021-02-10T04:37:06.160324+0000 osd.2 (osd.2 2021-02-10T04:37:07.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: ) 1837 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:07.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: cluster 2021-02-10T04:37 2021-02-10T04:37:07.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: :06.160331+0000 osd.2 (osd.2) 1838 : cluster 2021-02-10T04:37:07.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:07.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: cluster 2021-02-10T04:37 2021-02-10T04:37:07.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: :06.160337+0000 osd.2 (osd.2) 1839 : cluster 2021-02-10T04:37:07.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:07.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:37:07.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: 37:06.160352+0000 osd.2 (osd.2) 1840 2021-02-10T04:37:07.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:07.386 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: cluster 2021-02-10T04:37:07.387 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:37:07.387 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: :37:05.196407+0000 osd.4 (osd.4) 929 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:07.387 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:07.387 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: 02-10T04:37:05.196417+0000 2021-02-10T04:37:07.387 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: osd.4 (osd.4) 930 : cluster 2021-02-10T04:37:07.387 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:07.387 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: cluster 2021-02-10T04:37:07.387 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:37:07.387 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: :37:05.196422+0000 osd.4 (osd.4 2021-02-10T04:37:07.387 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: ) 931 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:07.387 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:37:07.387 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: -10T04:37:05.196425+0000 osd.4 ( 2021-02-10T04:37:07.387 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: osd.4) 932 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:07.388 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:07.388 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: -02-10T04:37:05.209202+0000 2021-02-10T04:37:07.388 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: osd.2 (osd.2) 1821 : cluster [WRN] 2021-02-10T04:37:07.388 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:07.388 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: cluster 2021-02-10T04:37:07.388 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:37:07.388 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: 37:05.209211+0000 osd.2 2021-02-10T04:37:07.388 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: (osd.2) 1822 : cluster 2021-02-10T04:37:07.388 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:07.388 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:07.388 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: 02-10T04:37:05 2021-02-10T04:37:07.388 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: .209217+0000 osd.2 (osd. 2021-02-10T04:37:07.388 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: 2) 1823 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:07.388 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: 2021-02-10T04:37:07.389 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: cluster 2021-02-10T04:37:07.389 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:37:07.389 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: 10T04:37:05.209223+0000 2021-02-10T04:37:07.389 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: osd.2 (osd.2) 1824 : 2021-02-10T04:37:07.389 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:07.389 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:07.389 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: -02-10T04:37:05 2021-02-10T04:37:07.389 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: .209229+0000 osd.2 (osd. 2021-02-10T04:37:07.389 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: 2) 1825 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:07.389 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: 2021-02-10T04:37:07.389 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: cluster 2021-02-10T04:37:05 2021-02-10T04:37:07.389 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: .209237+0000 osd.2 (osd.2) 1826 : 2021-02-10T04:37:07.389 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:07.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: cluster 2021-02-10T04:37:05.209243+0000 osd.2 2021-02-10T04:37:07.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: (osd.2) 1827 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:07.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: cluster 2021-02-10T04:37:05.209248+0000 2021-02-10T04:37:07.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: osd.2 (osd.2) 1828 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:07.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: cluster 2021-02-10T04:37:05. 2021-02-10T04:37:07.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: 209255+0000 osd.2 (osd.2) 1829 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:07.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: cluster 2021-02-10T04:37:05.209260+0000 2021-02-10T04:37:07.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: osd.2 (osd.2) 1830 : cluster [WRN] 2021-02-10T04:37:07.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:07.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: cluster 2021-02-10T04:37:05.479196+0000 2021-02-10T04:37:07.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: mon.a (mon.0) 1461 : cluster [DBG] osdmap e607: 8 total, 8 up, 8 in 2021-02-10T04:37:07.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:37:07.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: -10T04:37:06.130820+0000 mgr.y (mgr.14140) 659 : cluster [DBG] 2021-02-10T04:37:07.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: pgmap v1061: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 462 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.0 MiB/s wr, 10 op/s 2021-02-10T04:37:07.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:37:07.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: -10T04:37:06.160278+0000 osd.2 (osd 2021-02-10T04:37:07.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: .2) 1831 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:07.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:07.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: 02-10T04:37:06.160287+0000 osd.2 (osd 2021-02-10T04:37:07.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: .2) 1832 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:07.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:07.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: -02-10T04:37:06. 2021-02-10T04:37:07.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: 160297+0000 osd.2 (osd.2) 2021-02-10T04:37:07.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: 1833 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:07.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: cluster 2021-02-10T04:37:07.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:37:07.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: :37:06.160303+0000 osd.2 (osd.2) 1834 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:07.392 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: cluster 2021-02-10T04:37:07.392 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: 2021-02-10T04:37:07.392 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: 2021-02-10T04:37:06 2021-02-10T04:37:07.392 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: .160309+0000 osd.2 (osd.2) 1835 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:07.392 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:37:07.392 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: 37:06.160316+0000 osd.2 (osd.2) 2021-02-10T04:37:07.392 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: 1836 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:07.392 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:37:07.392 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: 04:37:06.160324+0000 osd.2 (osd.2) 1837 : 2021-02-10T04:37:07.392 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:07.392 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: cluster 2021-02-10T04:37:06 2021-02-10T04:37:07.392 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: .160331+0000 osd.2 (osd.2) 1838 : cluster 2021-02-10T04:37:07.392 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:07.393 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:37:07.393 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: 37:06.160337+0000 osd.2 (osd.2) 1839 : cluster 2021-02-10T04:37:07.393 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:07.393 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:07.393 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: 02-10T04:37:06.160352+0000 osd.2 (osd.2) 1840 : cluster 2021-02-10T04:37:07.393 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:06 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:07.393 INFO:tasks.rados.rados.0.smithi085.stdout:2337: finishing copy_from to smithi08575397-6 2021-02-10T04:37:07.393 INFO:tasks.rados.rados.0.smithi085.stdout:2337: got expected ENOENT (src dne) 2021-02-10T04:37:07.393 INFO:tasks.rados.rados.0.smithi085.stdout:2337: finishing copy_from racing read to smithi08575397-6 2021-02-10T04:37:07.393 INFO:tasks.rados.rados.0.smithi085.stdout:2334: done (15 left) 2021-02-10T04:37:07.393 INFO:tasks.rados.rados.0.smithi085.stdout:2335: done (14 left) 2021-02-10T04:37:07.393 INFO:tasks.rados.rados.0.smithi085.stdout:2336: done (13 left) 2021-02-10T04:37:07.393 INFO:tasks.rados.rados.0.smithi085.stdout:2337: done (12 left) 2021-02-10T04:37:07.394 INFO:tasks.rados.rados.0.smithi085.stdout:2340: done (11 left) 2021-02-10T04:37:07.394 INFO:tasks.rados.rados.0.smithi085.stdout:2341: delete oid 11 current snap is 264 2021-02-10T04:37:07.394 INFO:tasks.rados.rados.0.smithi085.stdout:2338: finishing write tid 1 to smithi08575397-1 2021-02-10T04:37:07.394 INFO:tasks.rados.rados.0.smithi085.stdout:2338: finishing write tid 2 to smithi08575397-1 2021-02-10T04:37:07.394 INFO:tasks.rados.rados.0.smithi085.stdout:2338: finishing write tid 3 to smithi08575397-1 2021-02-10T04:37:07.394 INFO:tasks.rados.rados.0.smithi085.stdout:2338: finishing write tid 4 to smithi08575397-1 2021-02-10T04:37:07.394 INFO:tasks.rados.rados.0.smithi085.stdout:2338: finishing write tid 5 to smithi08575397-1 2021-02-10T04:37:07.394 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 1134 (ObjNum 806 snap 264 seq_num 806) dirty exists 2021-02-10T04:37:07.394 INFO:tasks.rados.rados.0.smithi085.stdout:2338: left oid 1 (ObjNum 806 snap 264 seq_num 806) 2021-02-10T04:37:07.394 INFO:tasks.rados.rados.0.smithi085.stdout:2339: finishing write tid 1 to smithi08575397-2 2021-02-10T04:37:07.394 INFO:tasks.rados.rados.0.smithi085.stdout:2339: finishing write tid 2 to smithi08575397-2 2021-02-10T04:37:07.394 INFO:tasks.rados.rados.0.smithi085.stdout:2339: finishing write tid 3 to smithi08575397-2 2021-02-10T04:37:07.394 INFO:tasks.rados.rados.0.smithi085.stdout:2339: finishing write tid 4 to smithi08575397-2 2021-02-10T04:37:07.395 INFO:tasks.rados.rados.0.smithi085.stdout:2339: finishing write tid 5 to smithi08575397-2 2021-02-10T04:37:07.395 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 855 (ObjNum 807 snap 264 seq_num 807) dirty exists 2021-02-10T04:37:07.395 INFO:tasks.rados.rados.0.smithi085.stdout:2339: left oid 2 (ObjNum 807 snap 264 seq_num 807) 2021-02-10T04:37:07.395 INFO:tasks.rados.rados.0.smithi085.stdout:2338: done (11 left) 2021-02-10T04:37:07.395 INFO:tasks.rados.rados.0.smithi085.stdout:2339: done (10 left) 2021-02-10T04:37:07.395 INFO:tasks.rados.rados.0.smithi085.stdout:2341: done (9 left) 2021-02-10T04:37:07.395 INFO:tasks.rados.rados.0.smithi085.stdout:2342: snap_remove snap 257 2021-02-10T04:37:07.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[49106]: debug 2021-02-10T04:37:07.188+0000 7f77a3e94700 -1 osd.2 608 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:37:07.504 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[33677]: debug 2021-02-10T04:37:07.504 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[33677]: 2021-02-10T04:37:07.188+0000 7fbec19a1700 -1 osd.4 608 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:07.505 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[33677]: 2021-02-10T04:37:07.508 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: cluster 2021-02-10T04:37:07.508 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:37:07.508 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: 04:37:06.162672+0000 2021-02-10T04:37:07.508 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: 2021-02-10T04:37:07.508 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: osd.4 2021-02-10T04:37:07.508 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: ( 2021-02-10T04:37:07.508 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: osd 2021-02-10T04:37:07.508 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: .4) 933 : cluster 2021-02-10T04:37:07.508 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:07.508 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:37:07.508 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: -02-10T04:37:06.162679+0000 2021-02-10T04:37:07.508 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: osd.4 (osd.4) 934 : cluster [WRN] 2021-02-10T04:37:07.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:07.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: cluster 2021-02-10T04:37 2021-02-10T04:37:07.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: :06.162682+0000 osd.4 (osd.4) 935 2021-02-10T04:37:07.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:07.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:37:07.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: T04:37:06.162685+0000 osd.4 (osd 2021-02-10T04:37:07.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: .4) 936 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:07.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:37:07.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: 02-10T04:37:06.505971+0000 2021-02-10T04:37:07.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: mon.a (mon.0) 1462 : cluster [DBG] osdmap e608: 8 total, 8 up, 8 in 2021-02-10T04:37:07.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: 2021-02-10T04:37:07.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: cluster 2021-02-10T04:37:07.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: 2021-02-10T04:37:07.190346+0000 osd.2 ( 2021-02-10T04:37:07.510 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: osd.2) 1841 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:07.510 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:37:07.510 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: 02-10T04:37:07.190357+0000 osd.2 2021-02-10T04:37:07.510 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: (osd.2) 1842 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:07.510 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: 2021-02-10T04:37:07.510 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:37:07.510 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: -02-10T04:37:07.190366+0000 osd.2 (osd. 2021-02-10T04:37:07.510 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: 2) 1843 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:07.510 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: cluster 2021-02-10T04:37:07 2021-02-10T04:37:07.510 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: .190379+0000 osd.2 (osd.2) 1844 : cluster [WRN] 2021-02-10T04:37:07.510 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:07.510 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: cluster 2021-02-10T04:37:07. 2021-02-10T04:37:07.510 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: 190387+0000 osd.2 (osd.2) 1845 : cluster 2021-02-10T04:37:07.511 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:07.511 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: cluster 2021-02-10T04:37:07 2021-02-10T04:37:07.511 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: .190391+0000 osd.2 (osd.2) 1846 : cluster 2021-02-10T04:37:07.511 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:07.511 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: cluster 2021-02-10T04:37:07. 2021-02-10T04:37:07.511 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: 190395+0000 osd.2 (osd.2) 1847 : cluster 2021-02-10T04:37:07.511 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:07.511 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: cluster 2021-02-10T04:37:07.190400+0000 2021-02-10T04:37:07.511 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: osd.2 (osd.2) 1848 : cluster [WRN] 2021-02-10T04:37:07.511 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:07.511 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: cluster 2021-02-10T04:37:07. 2021-02-10T04:37:07.511 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: 190404+0000 osd.2 (osd.2) 1849 : cluster 2021-02-10T04:37:07.512 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:07.512 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: cluster 2021-02-10T04:37:07.512 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: 2021-02-10T04:37:07.190408+0000 osd.2 (osd 2021-02-10T04:37:07.512 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:07 smithi164 conmon[29288]: .2) 1850 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:07.517 INFO:tasks.rados.rados.0.smithi085.stdout:2342: done (9 left) 2021-02-10T04:37:07.518 INFO:tasks.rados.rados.0.smithi085.stdout:2343: read oid 48 snap -1 2021-02-10T04:37:07.518 INFO:tasks.rados.rados.0.smithi085.stdout:2343: expect deleted 2021-02-10T04:37:07.518 INFO:tasks.rados.rados.0.smithi085.stdout:2344: snap_remove snap 242 2021-02-10T04:37:07.626 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:37:07.663 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:37:07.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: cluster 2021-02-10T04:37:07.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: 2021-02-10T04:37:07.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: 2021 2021-02-10T04:37:07.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: - 2021-02-10T04:37:07.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: 02 2021-02-10T04:37:07.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: - 2021-02-10T04:37:07.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: 10 2021-02-10T04:37:07.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: T 2021-02-10T04:37:07.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: 04 2021-02-10T04:37:07.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: : 2021-02-10T04:37:07.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: 37 2021-02-10T04:37:07.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: : 2021-02-10T04:37:07.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: 06 2021-02-10T04:37:07.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: . 2021-02-10T04:37:07.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: 162672 2021-02-10T04:37:07.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: +0000 2021-02-10T04:37:07.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: 2021-02-10T04:37:07.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: osd.4 2021-02-10T04:37:07.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: ( 2021-02-10T04:37:07.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: osd 2021-02-10T04:37:07.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: . 2021-02-10T04:37:07.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: 4 2021-02-10T04:37:07.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: ) 2021-02-10T04:37:07.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: 933 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:07.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:07.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: 02-10T04:37:06.162679+0000 2021-02-10T04:37:07.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: osd.4 (osd.4) 934 : cluster 2021-02-10T04:37:07.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:07.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:37:07.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: 10T04:37:06.162682+0000 osd.4 (osd 2021-02-10T04:37:07.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: .4) 935 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:07.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:07.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: 02-10T04:37:06.162685+0000 osd.4 (osd 2021-02-10T04:37:07.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: .4) 936 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:07.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: cluster 2021-02-10T04:37:07.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: 2021-02-10T04:37:06.505971 2021-02-10T04:37:07.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1462 : cluster [DBG] 2021-02-10T04:37:07.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: osdmap e608: 8 total, 8 up, 8 in 2021-02-10T04:37:07.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:37:07.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: 04:37:07.190346+0000 osd.2 (osd.2 2021-02-10T04:37:07.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: ) 1841 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:07.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: cluster 2021-02-10T04:37:07.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:37:07.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: :37:07.190357+0000 osd.2 2021-02-10T04:37:07.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: (osd.2) 1842 : cluster 2021-02-10T04:37:07.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:07.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: cluster 2021-02-10T04:37:07.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: 2021-02-10T04:37:07.190366 2021-02-10T04:37:07.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1843 2021-02-10T04:37:07.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:07.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:07.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: 02-10T04:37:07.190379+0000 2021-02-10T04:37:07.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: osd.2 (osd.2) 1844 : 2021-02-10T04:37:07.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:07.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:07.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: 02-10T04:37:07.190387+0000 2021-02-10T04:37:07.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: osd.2 (osd.2) 1845 : cluster 2021-02-10T04:37:07.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:07.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:07.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: 02-10T04:37:07.190391+0000 osd.2 ( 2021-02-10T04:37:07.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: osd.2) 1846 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:07.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: 2021-02-10T04:37:07.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:37:07.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: 37:07.190395+0000 osd.2 (osd.2) 1847 : cluster [WRN] 2021-02-10T04:37:07.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:07.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:37:07.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: -10T04:37:07.190400+0000 osd.2 ( 2021-02-10T04:37:07.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: osd.2) 1848 : cluster [WRN] 2021-02-10T04:37:07.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:07.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: cluster 2021-02-10T04:37:07.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: 2021-02-10T04:37:07. 2021-02-10T04:37:07.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: 190404+0000 osd.2 (osd.2) 1849 2021-02-10T04:37:07.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:07.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:37:07.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[29319]: :37:07.190408+0000 osd.2 (osd.2) 1850 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:07.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: cluster 2021-02-10T04:37:06.162672+0000 2021-02-10T04:37:07.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: osd.4 (osd.4) 933 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:07.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: cluster 2021-02-10T04:37:06.162679 2021-02-10T04:37:07.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 934 : cluster [WRN] 2021-02-10T04:37:07.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:07.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:07.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: -02-10T04:37:06.162682+0000 2021-02-10T04:37:07.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: osd.4 (osd.4) 935 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:07.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: 2021-02-10T04:37:07.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: cluster 2021-02-10T04:37 2021-02-10T04:37:07.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: :06.162685+0000 osd.4 (osd.4) 936 2021-02-10T04:37:07.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:07.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:07.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: -02-10T04:37:06.505971+0000 2021-02-10T04:37:07.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: mon.a (mon.0) 1462 : cluster [DBG] osdmap e608: 8 total, 8 up, 8 in 2021-02-10T04:37:07.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: 2021-02-10T04:37:07.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:37:07.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: 37:07.190346+0000 osd.2 (osd.2 2021-02-10T04:37:07.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: ) 1841 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:07.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: cluster 2021-02-10T04:37:07.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:37:07.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: :37:07.190357+0000 osd.2 (osd.2) 1842 : cluster 2021-02-10T04:37:07.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:07.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:37:07.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: 04:37:07.190366+0000 osd.2 (osd. 2021-02-10T04:37:07.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: 2) 1843 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:07.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:07.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: 02-10T04:37:07.190379+0000 2021-02-10T04:37:07.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: osd.2 (osd.2) 1844 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:07.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:07.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: -02-10T04:37:07.190387+0000 2021-02-10T04:37:07.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: osd.2 (osd.2) 1845 : cluster [WRN] 2021-02-10T04:37:07.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:07.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: cluster 2021-02-10T04:37:07.190391+0000 osd.2 ( 2021-02-10T04:37:07.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: osd.2) 1846 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:07.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:37:07.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: -10T04:37:07.190395+0000 osd.2 2021-02-10T04:37:07.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: (osd.2) 1847 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:07.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:07.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: 02-10T04:37:07.190400+0000 osd.2 2021-02-10T04:37:07.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: (osd.2) 1848 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:07.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:07.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: -02-10T04:37:07.190404+0000 2021-02-10T04:37:07.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: osd.2 (osd.2) 1849 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:07.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:07.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: 02-10T04:37:07.190408+0000 2021-02-10T04:37:07.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: osd.2 (osd.2) 1850 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:07.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:07 smithi085 conmon[35961]: 2021-02-10T04:37:08.193 INFO:tasks.rados.rados.0.smithi085.stdout:2344: done (10 left) 2021-02-10T04:37:08.193 INFO:tasks.rados.rados.0.smithi085.stdout:2345: snap_create 2021-02-10T04:37:08.394 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:37:08.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[33677]: debug 2021-02-10T04:37:08.172+0000 7fbec19a1700 -1 osd.4 609 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:08.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[49106]: debug 2021-02-10T04:37:08.161+0000 7f77a3e94700 -1 osd.2 609 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:37:08.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: cluster 2021-02-10T04:37:08.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: 2021-02-10T04:37:08.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: 2021 2021-02-10T04:37:08.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: -02-10T04:37:07. 2021-02-10T04:37:08.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: 189307+0000 osd.4 (osd.4) 937 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:08.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:37:08.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: -02-10T04:37:07.189317+0000 osd.4 (osd.4) 938 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:08.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:37:08.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: 02-10T04:37:07.189323+0000 osd.4 (osd. 2021-02-10T04:37:08.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: 4) 939 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:08.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:37:08.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: T04:37:07.189329+0000 osd.4 (osd.4) 940 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:08.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:37:08.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: T04:37:07.516900+0000 mon.a (mon.0) 1463 : cluster 2021-02-10T04:37:08.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: [DBG] osdmap e609: 8 total, 8 up, 8 in 2021-02-10T04:37:08.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: cluster 2021-02-10T04:37:08.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: 2021-02-10T04:37:08.131535+0000 2021-02-10T04:37:08.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: mgr.y (mgr.14140) 660 : cluster 2021-02-10T04:37:08.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: [DBG] pgmap v1064: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 464 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2021-02-10T04:37:08.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: cluster 2021-02-10T04:37:08.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:37:08.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: 37:08.162662+0000 osd.2 (osd.2) 2021-02-10T04:37:08.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: 1851 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:08.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: cluster 2021-02-10T04:37:08.162687+0000 2021-02-10T04:37:08.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: osd.2 (osd.2) 1852 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:08.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:37:08.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: T04:37:08.162693+0000 osd.2 (osd.2 2021-02-10T04:37:08.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: ) 1853 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:08.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:37:08.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: :37:08.162702+0000 osd.2 (osd.2) 1854 2021-02-10T04:37:08.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:08.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: cluster 2021-02-10T04:37:08 2021-02-10T04:37:08.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: .162708+0000 osd.2 (osd.2) 1855 : cluster [WRN] 2021-02-10T04:37:08.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:08.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: cluster 2021-02-10T04:37:08.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: 2021-02-10T04:37:08.162713 2021-02-10T04:37:08.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1856 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:08.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: 2021-02-10T04:37:08.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:37:08.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: 37:08.162719+0000 osd.2 (osd.2) 1857 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:08.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: cluster 2021-02-10T04:37 2021-02-10T04:37:08.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: :08.162724+0000 osd.2 (osd.2) 1858 : cluster 2021-02-10T04:37:08.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:08.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: cluster 2021-02-10T04:37:08. 2021-02-10T04:37:08.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: 162729+0000 osd.2 (osd.2) 1859 : cluster [WRN] 2021-02-10T04:37:08.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:08.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: cluster 2021-02-10T04:37:08.162734+0000 osd.2 (osd.2) 2021-02-10T04:37:08.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: 1860 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:08.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: cluster 2021-02-10T04:37:08.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: 2021-02-10T04:37:08.186271+0000 mon.a (mon.0) 1464 : 2021-02-10T04:37:08.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 452 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:37:08.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: audit 2021-02-10T04:37: 2021-02-10T04:37:08.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: 08.187048+0000 mgr.y (mgr.14140) 661 : 2021-02-10T04:37:08.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:37:08.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: cluster 2021-02-10T04:37:08.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:37:08.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:08 smithi164 conmon[29288]: :37:08.191770+0000 mon.a (mon.0) 1465 : cluster [DBG] osdmap e610: 8 total, 8 up, 8 in 2021-02-10T04:37:08.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: cluster 2021-02-10T04:37:07. 2021-02-10T04:37:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: 189307+0000 osd.4 (osd.4) 937 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: 02-10T04:37:07.189317+0000 2021-02-10T04:37:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: osd.4 (osd.4) 938 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: 2021-02-10T04:37:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: cluster 2021-02-10T04:37:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: 2021-02-10T04:37:07.189323+0000 osd.4 (osd.4) 2021-02-10T04:37:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: 939 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: 02-10T04:37:07.189329+0000 osd.4 2021-02-10T04:37:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: (osd.4) 940 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:08.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:37:08.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: T04:37:07.516900+0000 mon.a (mon.0) 1463 : cluster 2021-02-10T04:37:08.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: [DBG] osdmap e609: 8 total, 8 up, 8 in 2021-02-10T04:37:08.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: cluster 2021-02-10T04:37:08.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: 2021-02-10T04:37:08.131535+0000 mgr.y (mgr.14140) 660 : cluster [DBG] 2021-02-10T04:37:08.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: pgmap v1064: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 464 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2021-02-10T04:37:08.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:37:08.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: -10T04:37:08.162662+0000 osd.2 (osd.2) 1851 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:08.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:37:08.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: 37:08. 2021-02-10T04:37:08.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: 162687+0000 osd.2 (osd.2) 1852 : 2021-02-10T04:37:08.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:08.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: cluster 2021-02-10T04:37:08.162693+0000 2021-02-10T04:37:08.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: osd.2 (osd.2) 1853 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:08.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: cluster 2021-02-10T04:37: 2021-02-10T04:37:08.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: 08.162702+0000 osd.2 (osd.2) 1854 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:08.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: cluster 2021-02-10T04:37:08.162708 2021-02-10T04:37:08.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1855 : cluster [WRN] 2021-02-10T04:37:08.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:08.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: cluster 2021-02-10T04:37:08.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: 2021-02-10T04:37:08.162713+0000 osd.2 (osd.2) 1856 : 2021-02-10T04:37:08.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:08.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:37:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: 10T04:37:08.162719+0000 osd.2 (osd 2021-02-10T04:37:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: .2) 1857 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: cluster 2021-02-10T04:37:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: 2021-02-10T04:37:08.162724 2021-02-10T04:37:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1858 : cluster [WRN] 2021-02-10T04:37:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:37:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: 04:37:08.162729+0000 osd.2 (osd. 2021-02-10T04:37:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: 2) 1859 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: -02-10T04:37:08.162734+0000 2021-02-10T04:37:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: osd.2 (osd.2) 1860 : cluster [WRN] 2021-02-10T04:37:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: cluster 2021-02-10T04:37 2021-02-10T04:37:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: :08.186271+0000 mon.a (mon.0) 1464 2021-02-10T04:37:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: : cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 452 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:37:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:37:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: 02-10T04:37:08.187048+0000 2021-02-10T04:37:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: mgr.y (mgr.14140) 661 : audit [DBG] 2021-02-10T04:37:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:37:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:37:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: 37:08.191770+0000 mon.a (mon.0) 2021-02-10T04:37:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[35961]: 1465 : cluster [DBG] osdmap e610: 8 total, 8 up, 8 in 2021-02-10T04:37:08.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: cluster 2021-02-10T04:37:08.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: 2021-02-10T04:37:08.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: 2021 2021-02-10T04:37:08.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: - 2021-02-10T04:37:08.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: 02 2021-02-10T04:37:08.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: - 2021-02-10T04:37:08.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: 10 2021-02-10T04:37:08.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: T 2021-02-10T04:37:08.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: 04 2021-02-10T04:37:08.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: :37:07.189307+0000 osd.4 ( 2021-02-10T04:37:08.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: osd.4) 937 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:08.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:37:08.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: 37:07.189317+0000 osd.4 (osd.4) 2021-02-10T04:37:08.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: 938 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:08.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: cluster 2021-02-10T04:37:08.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: 2021-02-10T04:37 2021-02-10T04:37:08.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: :07.189323+0000 osd.4 (osd.4) 2021-02-10T04:37:08.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: 939 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:08.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: cluster 2021-02-10T04:37:08.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: 2021-02-10T04:37:08.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: 2021 2021-02-10T04:37:08.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: -02-10T04:37 2021-02-10T04:37:08.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: :07.189329+0000 osd.4 (osd.4) 940 : cluster 2021-02-10T04:37:08.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:08.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: cluster 2021-02-10T04:37:07. 2021-02-10T04:37:08.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: 516900+0000 mon.a (mon.0) 1463 2021-02-10T04:37:08.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: : cluster [DBG] osdmap e609: 8 total, 8 up, 8 in 2021-02-10T04:37:08.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: cluster 2021-02-10T04:37:08.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: 2021-02-10T04:37:08.131535 2021-02-10T04:37:08.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 660 : cluster [DBG] pgmap v1064: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 464 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2021-02-10T04:37:08.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: cluster 2021-02-10T04:37: 2021-02-10T04:37:08.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: 08.162662+0000 osd.2 (osd.2) 1851 : cluster 2021-02-10T04:37:08.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:08.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: cluster 2021-02-10T04:37:08.162687 2021-02-10T04:37:08.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1852 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:08.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: cluster 2021-02-10T04:37:08.162693+0000 2021-02-10T04:37:08.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: osd.2 (osd.2) 1853 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:08.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: cluster 2021-02-10T04:37:08.162702+0000 2021-02-10T04:37:08.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: osd.2 (osd.2) 1854 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:08.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:37:08.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: :37:08.162708+0000 osd.2 (osd.2) 2021-02-10T04:37:08.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: 1855 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:08.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:37:08.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: -10T04:37:08.162713 2021-02-10T04:37:08.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1856 : cluster [WRN] 2021-02-10T04:37:08.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:08.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:37:08.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: 10T04:37:08.162719+0000 osd.2 ( 2021-02-10T04:37:08.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: osd.2) 1857 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:08.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: cluster 2021-02-10T04:37:08.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: 2021-02-10T04:37 2021-02-10T04:37:08.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: :08.162724+0000 osd.2 (osd.2) 1858 : cluster 2021-02-10T04:37:08.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:08.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:37:08.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: -02-10T04:37:08.162729+0000 2021-02-10T04:37:08.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: osd.2 (osd.2) 1859 : cluster [WRN] 2021-02-10T04:37:08.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:08.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: cluster 2021-02-10T04:37:08.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: 2021-02-10T04:37:08 2021-02-10T04:37:08.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: .162734+0000 osd.2 (osd.2) 1860 : 2021-02-10T04:37:08.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:08.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:37:08.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: -10T04:37:08.186271+0000 mon.a 2021-02-10T04:37:08.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: (mon.0) 1464 2021-02-10T04:37:08.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: : cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 452 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:37:08.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:37:08.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: 02-10T04:37:08.187048+0000 mgr.y (mgr.14140) 661 : audit 2021-02-10T04:37:08.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:37:08.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: cluster 2021-02-10T04:37:08.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: 2021-02 2021-02-10T04:37:08.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: -10T04:37:08.191770 2021-02-10T04:37:08.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1465 : cluster [DBG] 2021-02-10T04:37:08.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:08 smithi085 conmon[29319]: osdmap e610: 8 total, 8 up, 8 in 2021-02-10T04:37:09.200 INFO:tasks.rados.rados.0.smithi085.stdout:2343: done (10 left) 2021-02-10T04:37:09.200 INFO:tasks.rados.rados.0.smithi085.stdout:2345: done (9 left) 2021-02-10T04:37:09.200 INFO:tasks.rados.rados.0.smithi085.stdout:2346: write (excl) oid 14 current snap is 265 2021-02-10T04:37:09.200 INFO:tasks.rados.rados.0.smithi085.stdout:2346: seq_num 809 ranges {796863=409859,1807462=586285,3066260=530603} 2021-02-10T04:37:09.201 INFO:tasks.rados.rados.0.smithi085.stdout:2346: writing smithi08575397-14 from 796863 to 1206722 tid 1 2021-02-10T04:37:09.209 INFO:tasks.rados.rados.0.smithi085.stdout:2346: writing smithi08575397-14 from 1807462 to 2393747 tid 2 2021-02-10T04:37:09.217 INFO:tasks.rados.rados.0.smithi085.stdout:2346: writing smithi08575397-14 from 3066260 to 3596863 tid 3 2021-02-10T04:37:09.217 INFO:tasks.rados.rados.0.smithi085.stdout:2347: delete oid 22 current snap is 265 2021-02-10T04:37:09.217 INFO:tasks.rados.rados.0.smithi085.stdout:2346: finishing write tid 1 to smithi08575397-14 2021-02-10T04:37:09.217 INFO:tasks.rados.rados.0.smithi085.stdout:2346: finishing write tid 2 to smithi08575397-14 2021-02-10T04:37:09.222 INFO:tasks.rados.rados.0.smithi085.stdout:2346: finishing write tid 3 to smithi08575397-14 2021-02-10T04:37:09.222 INFO:tasks.rados.rados.0.smithi085.stdout:2346: finishing write tid 4 to smithi08575397-14 2021-02-10T04:37:09.222 INFO:tasks.rados.rados.0.smithi085.stdout:2346: finishing write tid 5 to smithi08575397-14 2021-02-10T04:37:09.222 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 858 (ObjNum 809 snap 265 seq_num 809) dirty exists 2021-02-10T04:37:09.222 INFO:tasks.rados.rados.0.smithi085.stdout:2346: left oid 14 (ObjNum 809 snap 265 seq_num 809) 2021-02-10T04:37:09.222 INFO:tasks.rados.rados.0.smithi085.stdout:2346: done (10 left) 2021-02-10T04:37:09.222 INFO:tasks.rados.rados.0.smithi085.stdout:2347: done (9 left) 2021-02-10T04:37:09.222 INFO:tasks.rados.rados.0.smithi085.stdout:2348: copy_from oid 20 from oid 7 current snap is 265 2021-02-10T04:37:09.222 INFO:tasks.rados.rados.0.smithi085.stdout:2349: rollback oid 14 current snap is 265 2021-02-10T04:37:09.222 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 14 to 200 2021-02-10T04:37:09.223 INFO:tasks.rados.rados.0.smithi085.stdout:2350: write oid 23 current snap is 265 2021-02-10T04:37:09.223 INFO:tasks.rados.rados.0.smithi085.stdout:2350: seq_num 811 ranges {430477=560357,1408073=473601,2488400=736073} 2021-02-10T04:37:09.228 INFO:tasks.rados.rados.0.smithi085.stdout:2350: writing smithi08575397-23 from 430477 to 990834 tid 1 2021-02-10T04:37:09.233 INFO:tasks.rados.rados.0.smithi085.stdout:2350: writing smithi08575397-23 from 1408073 to 1881674 tid 2 2021-02-10T04:37:09.243 INFO:tasks.rados.rados.0.smithi085.stdout:2350: writing smithi08575397-23 from 2488400 to 3224473 tid 3 2021-02-10T04:37:09.243 INFO:tasks.rados.rados.0.smithi085.stdout:2351: write (excl) oid 21 current snap is 265 2021-02-10T04:37:09.243 INFO:tasks.rados.rados.0.smithi085.stdout:2349: finishing rollback tid 0 to smithi08575397-14 2021-02-10T04:37:09.243 INFO:tasks.rados.rados.0.smithi085.stdout:2351: seq_num 812 ranges {447284=635606,1866555=733612,3315823=331461} 2021-02-10T04:37:09.248 INFO:tasks.rados.rados.0.smithi085.stdout:2351: writing smithi08575397-21 from 447284 to 1082890 tid 1 2021-02-10T04:37:09.258 INFO:tasks.rados.rados.0.smithi085.stdout:2351: writing smithi08575397-21 from 1866555 to 2600167 tid 2 2021-02-10T04:37:09.264 INFO:tasks.rados.rados.0.smithi085.stdout:2351: writing smithi08575397-21 from 3315823 to 3647284 tid 3 2021-02-10T04:37:09.265 INFO:tasks.rados.rados.0.smithi085.stdout:2352: delete oid 28 current snap is 265 2021-02-10T04:37:09.265 INFO:tasks.rados.rados.0.smithi085.stdout:2349: finishing rollback tid 1 to smithi08575397-14 2021-02-10T04:37:09.265 INFO:tasks.rados.rados.0.smithi085.stdout:2349: finishing rollback tid 2 to smithi08575397-14 2021-02-10T04:37:09.265 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 861 (ObjNum 651 snap 197 seq_num 651) dirty exists 2021-02-10T04:37:09.265 INFO:tasks.rados.rados.0.smithi085.stdout:2348: finishing copy_from to smithi08575397-20 2021-02-10T04:37:09.265 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 20 v 1135 (ObjNum 759 snap 243 seq_num 759) dirty exists 2021-02-10T04:37:09.265 INFO:tasks.rados.rados.0.smithi085.stdout:2350: finishing write tid 1 to smithi08575397-23 2021-02-10T04:37:09.265 INFO:tasks.rados.rados.0.smithi085.stdout:2348: finishing copy_from racing read to smithi08575397-20 2021-02-10T04:37:09.265 INFO:tasks.rados.rados.0.smithi085.stdout:2350: finishing write tid 2 to smithi08575397-23 2021-02-10T04:37:09.265 INFO:tasks.rados.rados.0.smithi085.stdout:2350: finishing write tid 3 to smithi08575397-23 2021-02-10T04:37:09.266 INFO:tasks.rados.rados.0.smithi085.stdout:2350: finishing write tid 4 to smithi08575397-23 2021-02-10T04:37:09.266 INFO:tasks.rados.rados.0.smithi085.stdout:2350: finishing write tid 5 to smithi08575397-23 2021-02-10T04:37:09.266 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 763 (ObjNum 811 snap 265 seq_num 811) dirty exists 2021-02-10T04:37:09.266 INFO:tasks.rados.rados.0.smithi085.stdout:2350: left oid 23 (ObjNum 811 snap 265 seq_num 811) 2021-02-10T04:37:09.266 INFO:tasks.rados.rados.0.smithi085.stdout:2351: finishing write tid 1 to smithi08575397-21 2021-02-10T04:37:09.266 INFO:tasks.rados.rados.0.smithi085.stdout:2351: finishing write tid 2 to smithi08575397-21 2021-02-10T04:37:09.267 INFO:tasks.rados.rados.0.smithi085.stdout:2348: done (13 left) 2021-02-10T04:37:09.268 INFO:tasks.rados.rados.0.smithi085.stdout:2349: done (12 left) 2021-02-10T04:37:09.268 INFO:tasks.rados.rados.0.smithi085.stdout:2350: done (11 left) 2021-02-10T04:37:09.268 INFO:tasks.rados.rados.0.smithi085.stdout:2352: done (10 left) 2021-02-10T04:37:09.268 INFO:tasks.rados.rados.0.smithi085.stdout:2353: copy_from oid 7 from oid 26 current snap is 265 2021-02-10T04:37:09.268 INFO:tasks.rados.rados.0.smithi085.stdout:2354: snap_remove snap 249 2021-02-10T04:37:09.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[33677]: debug 2021-02-10T04:37:09.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[33677]: 2021-02-10T04:37:09.135+0000 7fbec19a1700 -1 osd.4 610 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:09.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[49106]: debug 2021-02-10T04:37:09.167+0000 7f77a3e94700 -1 osd.2 610 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:37:09.777 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: cluster 2021-02-10T04:37:08.172829+0000 osd.4 (osd 2021-02-10T04:37:09.777 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: .4) 941 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:09.777 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: cluster 2021-02-10T04:37:08.172839+0000 osd.4 (osd.4) 942 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:09.777 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: cluster 2021-02-10T04:37:08.172846+0000 osd.4 (osd.4) 943 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:09.777 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: cluster 2021-02-10T04:37:08.172853+0000 osd.4 (osd.4) 944 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:09.777 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: cluster 2021-02-10T04:37:08.605121+0000 osd.2 (osd.2) 1861 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:37:09.778 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: cluster 2021-02-10T04:37:08.605361+0000 osd.2 2021-02-10T04:37:09.778 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: (osd.2) 1862 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:37:09.778 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: cluster 2021-02-10T04:37:09.169340+0000 osd.2 (osd 2021-02-10T04:37:09.778 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: .2) 1863 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:09.778 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:37:09.778 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: :37:09.169350+0000 osd.2 (osd.2) 1864 2021-02-10T04:37:09.778 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:09.778 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: cluster 2021-02-10T04:37:09.778 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: 2021-02-10T04:37:09.169357 2021-02-10T04:37:09.778 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1865 : cluster [WRN] 2021-02-10T04:37:09.778 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:09.778 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: cluster 2021-02-10T04:37:09.779 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: 2021 2021-02-10T04:37:09.779 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: - 2021-02-10T04:37:09.779 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: 02 2021-02-10T04:37:09.779 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: - 2021-02-10T04:37:09.779 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: 10 2021-02-10T04:37:09.779 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: T 2021-02-10T04:37:09.779 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: 04 2021-02-10T04:37:09.779 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: : 2021-02-10T04:37:09.779 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: 37:09.169364+0000 osd.2 ( 2021-02-10T04:37:09.779 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: osd.2) 1866 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:09.779 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:09.779 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: 02-10T04:37:09.169377+0000 osd.2 2021-02-10T04:37:09.779 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: (osd.2) 1867 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:09.780 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: 2021-02-10T04:37:09.780 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: cluster 2021-02-10T04:37 2021-02-10T04:37:09.780 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: :09.169383+0000 osd.2 (osd.2) 1868 2021-02-10T04:37:09.780 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:09.780 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:09.780 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: 02-10T04:37:09.169389+0000 2021-02-10T04:37:09.780 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: osd.2 (osd.2) 1869 : cluster [WRN] 2021-02-10T04:37:09.780 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:09.780 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: cluster 2021-02-10T04:37:09.169394+0000 2021-02-10T04:37:09.780 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: osd.2 (osd.2) 1870 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:09.780 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: 2021-02-10T04:37:09.780 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: cluster 2021-02-10T04:37: 2021-02-10T04:37:09.780 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: 09.169400+0000 osd.2 (osd.2) 1871 : 2021-02-10T04:37:09.780 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:09.781 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:37:09.781 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: T04:37:09.169406+0000 osd.2 (osd.2 2021-02-10T04:37:09.781 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: ) 1872 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:09.781 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:37:09.781 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: :37:09.194883+0000 mon.a (mon.0) 2021-02-10T04:37:09.781 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[29319]: 1466 : cluster [DBG] osdmap e611: 8 total, 8 up, 8 in 2021-02-10T04:37:09.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: cluster 2021-02-10T04:37:08.172829+0000 2021-02-10T04:37:09.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: osd.4 (osd.4) 941 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:09.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: cluster 2021-02-10T04:37:08. 2021-02-10T04:37:09.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: 172839+0000 osd.4 (osd.4) 942 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:09.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: cluster 2021-02-10T04:37:09.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: 2021-02-10T04:37:08.172846+0000 osd.4 (osd.4) 943 : 2021-02-10T04:37:09.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:09.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: cluster 2021-02-10T04:37:08.172853+0000 2021-02-10T04:37:09.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: osd.4 (osd.4) 944 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:09.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: cluster 2021-02-10T04:37:08.605121 2021-02-10T04:37:09.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1861 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:37:09.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:09.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: 02-10T04:37:08.605361+0000 osd.2 (osd. 2021-02-10T04:37:09.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: 2) 1862 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:37:09.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:37:09.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: :37:09.169340+0000 osd.2 (osd.2) 1863 : cluster 2021-02-10T04:37:09.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:09.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: cluster 2021-02-10T04:37:09.169350 2021-02-10T04:37:09.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1864 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:09.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:09.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: -02-10T04:37:09.169357+0000 osd.2 ( 2021-02-10T04:37:09.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: osd.2) 1865 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:09.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:37:09.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: 10T04:37:09.169364+0000 osd.2 (osd. 2021-02-10T04:37:09.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: 2) 1866 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:09.783 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:37:09.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: 10T04:37:09.169377+0000 osd.2 (osd. 2021-02-10T04:37:09.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: 2) 1867 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:09.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:37:09.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: :37:09.169383+0000 osd.2 (osd.2) 1868 2021-02-10T04:37:09.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:09.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: cluster 2021-02-10T04:37 2021-02-10T04:37:09.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: :09.169389+0000 osd.2 (osd.2) 1869 : cluster 2021-02-10T04:37:09.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:09.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: cluster 2021-02-10T04:37:09.169394 2021-02-10T04:37:09.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1870 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:09.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:09.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: 02-10T04:37:09.169400+0000 osd.2 ( 2021-02-10T04:37:09.784 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: osd.2) 1871 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:09.785 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:09.785 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: -02-10T04:37:09.169406+0000 osd.2 2021-02-10T04:37:09.785 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: (osd.2) 1872 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:09.785 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: 2021-02-10T04:37:09.785 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: cluster 2021-02-10T04:37: 2021-02-10T04:37:09.785 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: 09.194883+0000 mon.a (mon.0) 1466 : cluster 2021-02-10T04:37:09.785 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[35961]: [DBG] osdmap e611: 8 total, 8 up, 8 in 2021-02-10T04:37:09.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: cluster 2021-02-10T04:37:08.172829+0000 osd.4 (osd.4) 941 2021-02-10T04:37:09.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:09.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: cluster 2021-02-10T04:37:08.172839+0000 osd.4 (osd.4) 942 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:09.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: cluster 2021-02-10T04:37:08.172846+0000 osd.4 ( 2021-02-10T04:37:09.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: osd.4) 943 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:09.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: cluster 2021-02-10T04:37:08.172853+0000 osd.4 (osd.4 2021-02-10T04:37:09.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: ) 944 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:09.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:37:09.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: 37:08.605121+0000 osd.2 (osd.2) 1861 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:37:09.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: cluster 2021-02-10T04:37:09.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: 2021-02-10T04:37:08.605361+0000 osd.2 (osd.2) 1862 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:37:09.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: 2021-02-10T04:37:09.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:37:09.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: 04:37:09.169340+0000 osd.2 (osd.2) 1863 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:09.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: cluster 2021-02-10T04:37:09. 2021-02-10T04:37:09.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: 169350+0000 osd.2 (osd.2) 1864 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:09.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: cluster 2021-02-10T04:37:09.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: 2021-02-10T04:37:09.169357+0000 osd.2 (osd.2 2021-02-10T04:37:09.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: ) 1865 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:09.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: cluster 2021-02-10T04:37:09.169364+0000 osd.2 (osd. 2021-02-10T04:37:09.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: 2) 1866 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:09.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: cluster 2021-02-10T04:37:09.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: 2021-02-10T04:37:09.169377+0000 osd.2 (osd.2) 1867 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:09.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: 2021-02-10T04:37:09.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: cluster 2021-02-10T04:37 2021-02-10T04:37:09.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: :09.169383+0000 osd.2 (osd.2) 1868 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:09.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: cluster 2021-02-10T04:37:09.169389+0000 osd.2 ( 2021-02-10T04:37:09.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: osd.2) 1869 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:09.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: cluster 2021-02-10T04:37:09.169394+0000 2021-02-10T04:37:09.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: osd.2 (osd.2) 1870 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:09.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:37:09.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: 37:09.169400+0000 osd.2 (osd.2) 1871 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:09.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:37:09.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: 04:37:09.169406+0000 osd.2 (osd.2) 2021-02-10T04:37:09.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: 1872 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:09.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: cluster 2021-02-10T04:37:09.194883+0000 2021-02-10T04:37:09.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:09 smithi164 conmon[29288]: mon.a (mon.0) 1466 : cluster [DBG] osdmap e611: 8 total, 8 up, 8 in 2021-02-10T04:37:10.065 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:37:10.168 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[49106]: debug 2021-02-10T04:37:10.166+0000 7f77a3e94700 -1 osd.2 611 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:37:10.169 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:37:09 smithi085 conmon[52836]: debug 2021-02-10T04:37:09.775+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:37:10.188 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[33677]: debug 2021-02-10T04:37:10.172+0000 7fbec19a1700 -1 osd.4 611 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:10.196 INFO:tasks.rados.rados.0.smithi085.stdout:2351: finishing write tid 3 to smithi08575397-21 2021-02-10T04:37:10.197 INFO:tasks.rados.rados.0.smithi085.stdout:2351: finishing write tid 4 to smithi08575397-21 2021-02-10T04:37:10.197 INFO:tasks.rados.rados.0.smithi085.stdout:2351: finishing write tid 5 to smithi08575397-21 2021-02-10T04:37:10.197 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 1261 (ObjNum 812 snap 265 seq_num 812) dirty exists 2021-02-10T04:37:10.197 INFO:tasks.rados.rados.0.smithi085.stdout:2351: left oid 21 (ObjNum 812 snap 265 seq_num 812) 2021-02-10T04:37:10.197 INFO:tasks.rados.rados.0.smithi085.stdout:2353: finishing copy_from to smithi08575397-7 2021-02-10T04:37:10.197 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 754 (ObjNum 804 snap 259 seq_num 804) dirty exists 2021-02-10T04:37:10.197 INFO:tasks.rados.rados.0.smithi085.stdout:2353: finishing copy_from racing read to smithi08575397-7 2021-02-10T04:37:10.197 INFO:tasks.rados.rados.0.smithi085.stdout:2351: done (11 left) 2021-02-10T04:37:10.197 INFO:tasks.rados.rados.0.smithi085.stdout:2353: done (10 left) 2021-02-10T04:37:10.197 INFO:tasks.rados.rados.0.smithi085.stdout:2354: done (9 left) 2021-02-10T04:37:10.197 INFO:tasks.rados.rados.0.smithi085.stdout:2355: write oid 46 current snap is 265 2021-02-10T04:37:10.198 INFO:tasks.rados.rados.0.smithi085.stdout:2355: seq_num 814 ranges {480898=502457,1699872=737281,3087022=593876} 2021-02-10T04:37:10.202 INFO:tasks.rados.rados.0.smithi085.stdout:2355: writing smithi08575397-46 from 480898 to 983355 tid 1 2021-02-10T04:37:10.211 INFO:tasks.rados.rados.0.smithi085.stdout:2355: writing smithi08575397-46 from 1699872 to 2437153 tid 2 2021-02-10T04:37:10.220 INFO:tasks.rados.rados.0.smithi085.stdout:2355: writing smithi08575397-46 from 3087022 to 3680898 tid 3 2021-02-10T04:37:10.220 INFO:tasks.rados.rados.0.smithi085.stdout:2356: snap_remove snap 227 2021-02-10T04:37:10.582 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:37:10.582 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2021-02-10T04:37:10.582 INFO:teuthology.orchestra.run.smithi164.stdout: "complaint_time": 30, 2021-02-10T04:37:10.582 INFO:teuthology.orchestra.run.smithi164.stdout: "num_blocked_ops": 0 2021-02-10T04:37:10.582 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:37:10.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: cluster 2021-02-10T04:37:09.135956+0000 2021-02-10T04:37:10.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: osd.4 (osd.4) 945 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:10.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: cluster 2021-02-10T04:37:10.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:37:10.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: 37:09.135964+0000 osd.4 (osd.4) 946 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:10.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:37:10.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: -10T04:37:09.135970+0000 osd.4 (osd. 2021-02-10T04:37:10.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: 4) 947 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:10.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:37:10.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: 37:09.135976+0000 osd.4 (osd.4) 948 2021-02-10T04:37:10.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:10.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:37:10.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: 02-10T04:37:10 2021-02-10T04:37:10.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: .132184+0000 mgr.y (mgr. 2021-02-10T04:37:10.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: 14140) 662 : cluster [DBG] pgmap v1067: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 471 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.5 MiB/s rd, 2.0 MiB/s wr, 11 op/s 2021-02-10T04:37:10.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: 2021-02-10T04:37:10.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:37:10.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: 10T04:37:10.167878 2021-02-10T04:37:10.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1873 2021-02-10T04:37:10.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:10.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: cluster 2021-02-10T04:37:10.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: 2021-02-10T04:37 2021-02-10T04:37:10.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: :10.167886+0000 osd.2 (osd 2021-02-10T04:37:10.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: .2) 1874 : cluster [WRN] 2021-02-10T04:37:10.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:10.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: cluster 2021-02-10T04:37:10.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:37:10.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: 04:37:10.167893 2021-02-10T04:37:10.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1875 2021-02-10T04:37:10.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:10.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:37:10.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: 02-10T04:37:10.167898 2021-02-10T04:37:10.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1876 : cluster [WRN] 2021-02-10T04:37:10.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:10.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: cluster 2021-02-10T04:37:10 2021-02-10T04:37:10.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: .167904+0000 osd.2 (osd.2) 1877 : cluster 2021-02-10T04:37:10.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:10.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: cluster 2021-02-10T04:37:10 2021-02-10T04:37:10.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: .167912+0000 osd.2 (osd.2) 1878 2021-02-10T04:37:10.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:10.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:37:10.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: :37:10.167918+0000 2021-02-10T04:37:10.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: osd.2 (osd.2) 1879 : cluster [WRN] 2021-02-10T04:37:10.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:10.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:37:10.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: :37:10.167923+0000 osd.2 (osd.2) 2021-02-10T04:37:10.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: 1880 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:10.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: cluster 2021-02-10T04:37: 2021-02-10T04:37:10.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: 10.167930+0000 osd.2 (osd.2) 1881 : cluster 2021-02-10T04:37:10.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:10.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: cluster 2021-02-10T04:37:10 2021-02-10T04:37:10.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: .167935+0000 osd.2 (osd.2) 1882 : cluster [WRN] 2021-02-10T04:37:10.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:10.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: cluster 2021-02-10T04:37:10.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: 2021-02-10T04:37:10.195800+0000 mon.a (mon. 2021-02-10T04:37:10.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:10 smithi164 conmon[29288]: 0) 1467 : cluster [DBG] osdmap e612: 8 total, 8 up, 8 in 2021-02-10T04:37:10.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: cluster 2021-02-10T04:37:09.135956+0000 osd.4 (osd.4) 945 2021-02-10T04:37:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: cluster 2021-02-10T04:37:09.135964+0000 osd.4 (osd.4) 946 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: cluster 2021-02-10T04:37:09.135970+0000 osd.4 (osd.4) 947 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:37:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: 10T04:37:09.135976+0000 osd.4 (osd.4 2021-02-10T04:37:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: ) 948 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:37:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: T04:37:10.132184+0000 mgr.y (mgr.14140) 662 : cluster 2021-02-10T04:37:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: [DBG] pgmap v1067: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 471 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.5 MiB/s rd, 2.0 MiB/s wr, 11 op/s 2021-02-10T04:37:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: cluster 2021-02-10T04:37: 2021-02-10T04:37:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: 10.167878+0000 osd.2 (osd.2) 1873 : cluster 2021-02-10T04:37:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: cluster 2021-02-10T04:37:10 2021-02-10T04:37:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: .167886+0000 osd.2 (osd.2) 1874 : cluster [WRN] 2021-02-10T04:37:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: cluster 2021-02-10T04:37: 2021-02-10T04:37:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: 10.167893+0000 osd.2 (osd.2) 1875 : 2021-02-10T04:37:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: -02 2021-02-10T04:37:10.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: -10T04:37:10.167898+0000 osd.2 2021-02-10T04:37:10.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: (osd.2) 1876 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:10.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: cluster 2021-02-10T04:37:10. 2021-02-10T04:37:10.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: 167904+0000 osd.2 (osd.2) 2021-02-10T04:37:10.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: 1877 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:10.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: cluster 2021-02-10T04:37:10.167912+0000 osd.2 (osd 2021-02-10T04:37:10.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: .2) 1878 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:10.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:37:10.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: 04:37:10.167918+0000 osd.2 (osd.2 2021-02-10T04:37:10.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: ) 1879 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:10.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:37:10.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: :37:10.167923+0000 osd.2 (osd.2) 1880 2021-02-10T04:37:10.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:10.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: cluster 2021-02-10T04:37:10. 2021-02-10T04:37:10.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: 167930+0000 osd.2 (osd.2) 1881 : cluster [WRN] 2021-02-10T04:37:10.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:10.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: cluster 2021-02-10T04:37:10.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: 2021-02-10T04:37:10.167935+0000 osd.2 (osd.2) 2021-02-10T04:37:10.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: 1882 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:10.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:37:10.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: 04:37:10.195800+0000 mon.a (mon.0) 1467 : cluster 2021-02-10T04:37:10.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[35961]: [DBG] osdmap e612: 8 total, 8 up, 8 in 2021-02-10T04:37:10.945 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[52836]: debug 2021-02-10T04:37:10.721+0000 7fba71fb2700 -1 osd.3 pg_epoch: 612 pg[4.10( v 538'16 (0'0,538'16] local-lis/les=71/72 n=10 ec=55/55 lis/c=71/71 les/c/f=72/72/0 sis=71) [3,5,6] r=0 lpr=71 crt=538'16 lcod 538'15 mlcod 538'15 active+clean] scrubber pg(4.10) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6 (0x55fec06af080) 2021-02-10T04:37:10.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: cluster 2021-02-10T04:37:09.135956+0000 osd.4 (osd.4) 945 : 2021-02-10T04:37:10.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:10.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: cluster 2021-02-10T04:37:09.135964+0000 osd.4 (osd.4) 946 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:10.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:37:10.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: 37:09.135970+0000 osd.4 (osd.4) 947 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:10.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:37:10.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: -10T04:37:09.135976+0000 2021-02-10T04:37:10.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: osd.4 (osd.4) 948 : cluster [WRN] 2021-02-10T04:37:10.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:10.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:37:10.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: T04:37:10.132184+0000 mgr.y ( 2021-02-10T04:37:10.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: mgr.14140) 662 : cluster [DBG] pgmap v1067: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 471 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.5 MiB/s rd, 2.0 MiB/s wr, 11 op/s 2021-02-10T04:37:10.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:10.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: 02-10T04:37:10.167878+0000 2021-02-10T04:37:10.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: osd.2 (osd.2) 1873 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:10.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:37:10.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: 04:37:10.167886+0000 osd.2 (osd.2) 1874 : cluster 2021-02-10T04:37:10.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:10.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: cluster 2021-02-10T04:37:10.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: 2021-02-10T04:37:10.167893+0000 osd.2 (osd.2 2021-02-10T04:37:10.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: ) 1875 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:10.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: cluster 2021-02-10T04:37:10 2021-02-10T04:37:10.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: .167898+0000 osd.2 (osd.2) 1876 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:10.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: 2021-02-10T04:37:10.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:37:10.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: -10T04:37:10.167904+0000 osd.2 (osd.2) 1877 : cluster [WRN] 2021-02-10T04:37:10.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:10.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:37:10.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: -02-10T04:37:10.167912+0000 osd.2 (osd.2) 1878 : cluster 2021-02-10T04:37:10.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:10.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: cluster 2021-02-10T04:37:10.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: 2021-02-10T04:37:10.167918+0000 osd.2 (osd.2) 2021-02-10T04:37:10.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: 1879 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:10.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: cluster 2021-02-10T04:37: 2021-02-10T04:37:10.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: 10.167923+0000 osd.2 (osd.2) 1880 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:10.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:37:10.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: -02-10T04:37:10.167930+0000 osd.2 2021-02-10T04:37:10.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: (osd.2) 1881 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:10.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:37:10.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: T04:37:10.167935+0000 osd.2 (osd.2) 1882 : 2021-02-10T04:37:10.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:10.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: cluster 2021-02-10T04:37:10.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: 2021-02-10T04:37:10.195800+0000 mon.a (mon.0) 1467 : 2021-02-10T04:37:10.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:10 smithi085 conmon[29319]: cluster [DBG] osdmap e612: 8 total, 8 up, 8 in 2021-02-10T04:37:11.203 INFO:tasks.rados.rados.0.smithi085.stdout:2355: finishing write tid 1 to smithi08575397-46 2021-02-10T04:37:11.203 INFO:tasks.rados.rados.0.smithi085.stdout:2355: finishing write tid 2 to smithi08575397-46 2021-02-10T04:37:11.203 INFO:tasks.rados.rados.0.smithi085.stdout:2355: finishing write tid 3 to smithi08575397-46 2021-02-10T04:37:11.203 INFO:tasks.rados.rados.0.smithi085.stdout:2355: finishing write tid 4 to smithi08575397-46 2021-02-10T04:37:11.203 INFO:tasks.rados.rados.0.smithi085.stdout:2355: finishing write tid 5 to smithi08575397-46 2021-02-10T04:37:11.203 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 1077 (ObjNum 814 snap 265 seq_num 814) dirty exists 2021-02-10T04:37:11.203 INFO:tasks.rados.rados.0.smithi085.stdout:2355: left oid 46 (ObjNum 814 snap 265 seq_num 814) 2021-02-10T04:37:11.203 INFO:tasks.rados.rados.0.smithi085.stdout:2355: done (10 left) 2021-02-10T04:37:11.203 INFO:tasks.rados.rados.0.smithi085.stdout:2356: done (9 left) 2021-02-10T04:37:11.203 INFO:tasks.rados.rados.0.smithi085.stdout:2357: delete oid 28 current snap is 265 2021-02-10T04:37:11.214 INFO:tasks.rados.rados.0.smithi085.stdout:2357: done (9 left) 2021-02-10T04:37:11.214 INFO:tasks.rados.rados.0.smithi085.stdout:2358: read oid 44 snap -1 2021-02-10T04:37:11.214 INFO:tasks.rados.rados.0.smithi085.stdout:2358: expect (ObjNum 803 snap 259 seq_num 803) 2021-02-10T04:37:11.215 INFO:tasks.rados.rados.0.smithi085.stdout:2359: snap_remove snap 260 2021-02-10T04:37:11.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[33677]: debug 2021-02-10T04:37:11.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[33677]: 2021-02-10T04:37:11.215+0000 7fbec19a1700 -1 osd.4 613 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:11.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[33677]: 2021-02-10T04:37:11.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[49106]: debug 2021-02-10T04:37:11.184+0000 7f77a3e94700 -1 osd.2 612 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:37:11.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: cluster 2021-02-10T04:37:11.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: 2021- 2021-02-10T04:37:11.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: 02-10T04:37:10.172709+0000 osd.4 (osd.4) 949 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: cluster 2021-02-10T04:37:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: 2021-02-10T04:37:10. 2021-02-10T04:37:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: 172722+0000 osd.4 (osd.4) 950 : cluster 2021-02-10T04:37:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:37:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: -02-10T04:37:10.172728+0000 2021-02-10T04:37:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: osd.4 (osd.4) 951 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: 2021-02-10T04:37:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: cluster 2021-02-10T04:37 2021-02-10T04:37:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: :10.172735+0000 osd.4 (osd.4) 952 2021-02-10T04:37:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: cluster 2021-02-10T04:37:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: 2021-02-10T04:37:11. 2021-02-10T04:37:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: 185893+0000 osd.2 (osd.2) 1883 : cluster 2021-02-10T04:37:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:37:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: 37:11.185903+0000 osd.2 (osd.2) 2021-02-10T04:37:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: 1884 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: cluster 2021-02-10T04:37:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: 2021 2021-02-10T04:37:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: -02-10T04 2021-02-10T04:37:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: :37:11.185909+0000 osd.2 (osd.2) 1885 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:37:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: -10T04:37:11.185915+0000 osd.2 (osd.2) 1886 2021-02-10T04:37:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: cluster 2021-02-10T04:37:11.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:37:11.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: :37:11.185922+0000 osd.2 (osd.2 2021-02-10T04:37:11.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: ) 1887 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:11.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: cluster 2021-02-10T04:37:11.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: 2021-02-10T04:37:11. 2021-02-10T04:37:11.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: 185926+0000 osd.2 (osd.2) 1888 : cluster 2021-02-10T04:37:11.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:11.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:37:11.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: 02-10T04:37:11.185932+0000 osd.2 2021-02-10T04:37:11.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: (osd.2) 1889 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:11.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:37:11.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: 02-10T04:37:11.185936+0000 osd.2 (osd 2021-02-10T04:37:11.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: .2) 1890 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:11.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: cluster 2021-02-10T04:37:11.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:37:11.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: 37:11.185940+0000 osd.2 (osd.2) 2021-02-10T04:37:11.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: 1891 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:11.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: cluster 2021-02-10T04:37:11.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: 2021-02-10T04:37 2021-02-10T04:37:11.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: :11.185945+0000 osd.2 (osd.2) 1892 2021-02-10T04:37:11.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:11.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: cluster 2021-02-10T04:37:11.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: 2021-02-10T04:37:11. 2021-02-10T04:37:11.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: 201578+0000 mon.a (mon.0) 1468 : cluster 2021-02-10T04:37:11.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:11 smithi164 conmon[29288]: [DBG] osdmap e613: 8 total, 8 up, 8 in 2021-02-10T04:37:11.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: cluster 2021-02-10T04:37:11.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:37:11.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: :37:10.172709+0000 osd.4 (osd.4) 949 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:11.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: cluster 2021-02-10T04:37:11.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: 2021-02-10T04:37:10 2021-02-10T04:37:11.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: .172722+0000 osd.4 (osd.4) 950 : cluster 2021-02-10T04:37:11.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:11.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:37:11.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: -10T04:37:10.172728+0000 osd.4 2021-02-10T04:37:11.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: (osd.4) 951 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:11.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:11.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: -02-10T04:37:10.172735+0000 osd.4 2021-02-10T04:37:11.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: (osd.4) 952 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: 02-10T04:37:11.185893+0000 2021-02-10T04:37:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: osd.2 (osd.2) 1883 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: 2021-02-10T04:37:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: cluster 2021-02-10T04:37:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: 2021-02-10T04:37:11. 2021-02-10T04:37:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: 185903+0000 osd.2 (osd.2) 1884 : cluster 2021-02-10T04:37:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: cluster 2021-02-10T04:37:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: 2021-02-10T04:37:11.185909+0000 osd.2 (osd. 2021-02-10T04:37:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: 2) 1885 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:37:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: 04:37:11.185915+0000 osd.2 (osd.2 2021-02-10T04:37:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: ) 1886 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:11.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: cluster 2021-02-10T04:37:11.185922+0000 osd.2 ( 2021-02-10T04:37:11.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: osd.2) 1887 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:11.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: cluster 2021-02-10T04:37: 2021-02-10T04:37:11.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: 11.185926+0000 osd.2 (osd.2) 1888 : cluster 2021-02-10T04:37:11.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:11.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:11.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: -02-10T04:37:11.185932+0000 2021-02-10T04:37:11.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: osd.2 (osd.2) 1889 : cluster 2021-02-10T04:37:11.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:11.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: cluster 2021-02-10T04:37:11.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: 2021-02-10T04:37:11. 2021-02-10T04:37:11.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: 185936+0000 osd.2 (osd.2) 1890 : cluster 2021-02-10T04:37:11.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:11.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: cluster 2021-02-10T04:37:11.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:37:11.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: 10T04:37:11. 2021-02-10T04:37:11.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: 185940+0000 osd.2 (osd.2 2021-02-10T04:37:11.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: ) 1891 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:11.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: 2021-02-10T04:37:11.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:11.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: -02-10T04:37: 2021-02-10T04:37:11.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: 11.185945+0000 osd.2 ( 2021-02-10T04:37:11.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: osd.2) 1892 : cluster 2021-02-10T04:37:11.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:11.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:11.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: -02-10T04:37:11.201578+0000 mon.a 2021-02-10T04:37:11.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: (mon.0) 1468 : cluster [DBG] 2021-02-10T04:37:11.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[35961]: osdmap e613: 8 total, 8 up, 8 in 2021-02-10T04:37:11.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: cluster 2021-02-10T04:37:10. 2021-02-10T04:37:11.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: 172709+0000 osd.4 (osd.4) 949 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:11.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:37:11.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: -02-10T04:37:10.172722+0000 osd.4 (osd.4) 950 : cluster 2021-02-10T04:37:11.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:11.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:37:11.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: T04:37:10. 2021-02-10T04:37:11.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: 172728+0000 osd.4 (osd.4) 951 : 2021-02-10T04:37:11.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:11.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:37:11.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: -10T04:37:10. 2021-02-10T04:37:11.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: 172735+0000 osd.4 (osd.4) 952 : cluster [WRN] 2021-02-10T04:37:11.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:11.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:37:11.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: -02-10T04:37:11.185893+0000 2021-02-10T04:37:11.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: osd.2 (osd.2) 1883 : cluster [WRN] 2021-02-10T04:37:11.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:11.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: cluster 2021-02-10T04:37:11 2021-02-10T04:37:11.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: .185903+0000 osd.2 (osd.2) 1884 : 2021-02-10T04:37:11.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:11.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: 2021-02-10T04:37:11.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: cluster 2021-02-10T04:37:11 2021-02-10T04:37:11.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: .185909+0000 osd.2 (osd.2) 1885 2021-02-10T04:37:11.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:11.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:11.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: 02-10T04:37:11.185915+0000 osd.2 (osd. 2021-02-10T04:37:11.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: 2) 1886 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:11.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:11.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: 02-10T04:37:11.185922+0000 osd.2 2021-02-10T04:37:11.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: (osd.2) 2021-02-10T04:37:11.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: 1887 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:11.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:37:11.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: T04:37:11.185926+0000 osd.2 (osd 2021-02-10T04:37:11.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: .2) 1888 : cluster [WRN] 2021-02-10T04:37:11.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:11.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: cluster 2021-02-10T04:37:11.185932 2021-02-10T04:37:11.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: +0000 osd.2 (osd 2021-02-10T04:37:11.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: .2) 1889 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:11.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:11.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: 02-10T04:37:11.185936+0000 osd.2 2021-02-10T04:37:11.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: (osd.2) 1890 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:11.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: 2021-02-10T04:37:11.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:37:11.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: 04:37:11.185940+0000 osd.2 ( 2021-02-10T04:37:11.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: osd.2) 1891 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:11.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:37:11.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: -02-10T04:37:11.185945+0000 2021-02-10T04:37:11.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: osd.2 (osd.2) 1892 : cluster [WRN] 2021-02-10T04:37:11.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:11.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: cluster 2021-02-10T04:37:11. 2021-02-10T04:37:11.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: 201578+0000 mon.a (mon.0) 1468 : cluster 2021-02-10T04:37:11.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:11 smithi085 conmon[29319]: [DBG] osdmap e613: 8 total, 8 up, 8 in 2021-02-10T04:37:12.178 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:37:12.211 INFO:tasks.rados.rados.0.smithi085.stdout:2359: done (10 left) 2021-02-10T04:37:12.211 INFO:tasks.rados.rados.0.smithi085.stdout:2360: write (excl) oid 38 current snap is 265 2021-02-10T04:37:12.211 INFO:tasks.rados.rados.0.smithi085.stdout:2360: seq_num 816 ranges {514512=652955,1700483=740950,3025515=688997} 2021-02-10T04:37:12.213 INFO:tasks.rados.rados.0.smithi085.stdout:2360: writing smithi08575397-38 from 514512 to 1167467 tid 1 2021-02-10T04:37:12.222 INFO:tasks.rados.rados.0.smithi085.stdout:2360: writing smithi08575397-38 from 1700483 to 2441433 tid 2 2021-02-10T04:37:12.230 INFO:tasks.rados.rados.0.smithi085.stdout:2360: writing smithi08575397-38 from 3025515 to 3714512 tid 3 2021-02-10T04:37:12.231 INFO:tasks.rados.rados.0.smithi085.stdout:2361: write oid 11 current snap is 265 2021-02-10T04:37:12.231 INFO:tasks.rados.rados.0.smithi085.stdout:2361: seq_num 817 ranges {531319=728204,1758965=484608,2652938=520058} 2021-02-10T04:37:12.240 INFO:tasks.rados.rados.0.smithi085.stdout:2361: writing smithi08575397-11 from 531319 to 1259523 tid 1 2021-02-10T04:37:12.245 INFO:tasks.rados.rados.0.smithi085.stdout:2361: writing smithi08575397-11 from 1758965 to 2243573 tid 2 2021-02-10T04:37:12.251 INFO:tasks.rados.rados.0.smithi085.stdout:2361: writing smithi08575397-11 from 2652938 to 3172996 tid 3 2021-02-10T04:37:12.251 INFO:tasks.rados.rados.0.smithi085.stdout:2358: expect (ObjNum 803 snap 259 seq_num 803) 2021-02-10T04:37:12.409 INFO:tasks.rados.rados.0.smithi085.stdout:2358: done (11 left) 2021-02-10T04:37:12.409 INFO:tasks.rados.rados.0.smithi085.stdout:2362: copy_from oid 14 from oid 40 current snap is 265 2021-02-10T04:37:12.410 INFO:tasks.rados.rados.0.smithi085.stdout:2360: finishing write tid 1 to smithi08575397-38 2021-02-10T04:37:12.410 INFO:tasks.rados.rados.0.smithi085.stdout:2360: finishing write tid 2 to smithi08575397-38 2021-02-10T04:37:12.410 INFO:tasks.rados.rados.0.smithi085.stdout:2360: finishing write tid 3 to smithi08575397-38 2021-02-10T04:37:12.410 INFO:tasks.rados.rados.0.smithi085.stdout:2360: finishing write tid 4 to smithi08575397-38 2021-02-10T04:37:12.410 INFO:tasks.rados.rados.0.smithi085.stdout:2360: finishing write tid 5 to smithi08575397-38 2021-02-10T04:37:12.410 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 1081 (ObjNum 816 snap 265 seq_num 816) dirty exists 2021-02-10T04:37:12.410 INFO:tasks.rados.rados.0.smithi085.stdout:2360: left oid 38 (ObjNum 816 snap 265 seq_num 816) 2021-02-10T04:37:12.410 INFO:tasks.rados.rados.0.smithi085.stdout:2361: finishing write tid 1 to smithi08575397-11 2021-02-10T04:37:12.410 INFO:tasks.rados.rados.0.smithi085.stdout:2361: finishing write tid 2 to smithi08575397-11 2021-02-10T04:37:12.410 INFO:tasks.rados.rados.0.smithi085.stdout:2361: finishing write tid 3 to smithi08575397-11 2021-02-10T04:37:12.410 INFO:tasks.rados.rados.0.smithi085.stdout:2361: finishing write tid 4 to smithi08575397-11 2021-02-10T04:37:12.410 INFO:tasks.rados.rados.0.smithi085.stdout:2361: finishing write tid 5 to smithi08575397-11 2021-02-10T04:37:12.411 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 1269 (ObjNum 817 snap 265 seq_num 817) dirty exists 2021-02-10T04:37:12.411 INFO:tasks.rados.rados.0.smithi085.stdout:2361: left oid 11 (ObjNum 817 snap 265 seq_num 817) 2021-02-10T04:37:12.411 INFO:tasks.rados.rados.0.smithi085.stdout:2360: done (11 left) 2021-02-10T04:37:12.411 INFO:tasks.rados.rados.0.smithi085.stdout:2361: done (10 left) 2021-02-10T04:37:12.411 INFO:tasks.rados.rados.0.smithi085.stdout:2363: write (excl) oid 48 current snap is 265 2021-02-10T04:37:12.411 INFO:tasks.rados.rados.0.smithi085.stdout:2363: seq_num 818 ranges {548126=403453,1417447=744619,2680361=447683,3680450=67676} 2021-02-10T04:37:12.414 INFO:tasks.rados.rados.0.smithi085.stdout:2363: writing smithi08575397-48 from 548126 to 951579 tid 1 2021-02-10T04:37:12.422 INFO:tasks.rados.rados.0.smithi085.stdout:2363: writing smithi08575397-48 from 1417447 to 2162066 tid 2 2021-02-10T04:37:12.428 INFO:tasks.rados.rados.0.smithi085.stdout:2363: writing smithi08575397-48 from 2680361 to 3128044 tid 3 2021-02-10T04:37:12.432 INFO:tasks.rados.rados.0.smithi085.stdout:2363: writing smithi08575397-48 from 3680450 to 3748126 tid 4 2021-02-10T04:37:12.432 INFO:tasks.rados.rados.0.smithi085.stdout:2364: snap_remove snap 261 2021-02-10T04:37:12.432 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:37:12.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[49106]: debug 2021-02-10T04:37:12.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[49106]: 2021-02-10T04:37:12.138+0000 7f77a3e94700 -1 osd.2 613 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:37:12.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[49106]: 2021-02-10T04:37:12.525 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[33677]: debug 2021-02-10T04:37:12.208+0000 7fbec19a1700 -1 osd.4 614 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:12.893 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[40892]: debug 2021-02-10T04:37:12.894 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[40892]: 2021-02-10T04:37:12.523+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:37:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: cluster 2021-02-10T04:37:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: 2021-02-10T04:37:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: 2021-02-10T04:37:11 2021-02-10T04:37:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: .215815+0000 osd.4 (osd.4) 953 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:37:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: -02-10T04:37:11.215834+0000 osd.4 2021-02-10T04:37:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: (osd.4) 954 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:12.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:37:12.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: -10T04:37:11.215840+0000 osd.4 (osd 2021-02-10T04:37:12.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: .4) 955 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:12.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: cluster 2021-02-10T04:37:12.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: 2021-02-10T04:37:11.215847 2021-02-10T04:37:12.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 956 : cluster 2021-02-10T04:37:12.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:12.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: cluster 2021-02-10T04:37:12.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: 2021-02-10T04:37 2021-02-10T04:37:12.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: :12.132806+0000 mgr.y (mgr.14140) 663 2021-02-10T04:37:12.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: : cluster [DBG] pgmap v1070: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 474 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.7 MiB/s rd, 2.8 MiB/s wr, 12 op/s 2021-02-10T04:37:12.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:37:12.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: 02-10T04:37:12.140163+0000 osd.2 ( 2021-02-10T04:37:12.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: osd.2) 1893 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:12.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:37:12.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: 02-10T04:37:12.140171+0000 2021-02-10T04:37:12.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: osd.2 (osd.2) 1894 : cluster [WRN] 2021-02-10T04:37:12.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:12.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:37:12.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: -10T04:37:12.140178+0000 osd.2 2021-02-10T04:37:12.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: (osd.2) 1895 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:12.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:37:12.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: T04:37:12.140184+0000 osd.2 (osd. 2021-02-10T04:37:12.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: 2) 1896 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:12.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:37:12.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: 02-10T04:37:12.140189+0000 2021-02-10T04:37:12.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: osd.2 (osd.2) 1897 : cluster [WRN] 2021-02-10T04:37:12.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:12.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: cluster 2021-02-10T04:37 2021-02-10T04:37:12.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: :12.140194+0000 osd.2 (osd.2) 1898 : 2021-02-10T04:37:12.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:12.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: cluster 2021-02-10T04:37: 2021-02-10T04:37:12.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: 12.140198+0000 osd.2 (osd.2) 1899 : cluster 2021-02-10T04:37:12.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:12.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:37:12.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: T04:37:12.140201+0000 osd.2 (osd. 2021-02-10T04:37:12.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: 2) 1900 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:12.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: cluster 2021-02-10T04:37:12.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:37:12.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: 37:12.140204+0000 osd.2 (osd.2 2021-02-10T04:37:12.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: ) 1901 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:12.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:37:12.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: 02-10T04:37:12.140207+0000 osd.2 (osd 2021-02-10T04:37:12.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: .2) 1902 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:12.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:37:12.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: -02-10T04:37:12.202913 2021-02-10T04:37:12.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1469 : cluster [DBG] 2021-02-10T04:37:12.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:12 smithi164 conmon[29288]: osdmap e614: 8 total, 8 up, 8 in 2021-02-10T04:37:12.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: cluster 2021-02-10T04:37:11 2021-02-10T04:37:12.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: .215815+0000 osd.4 (osd.4) 953 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:12.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: cluster 2021-02-10T04:37:11.215834+0000 2021-02-10T04:37:12.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: osd.4 (osd.4) 954 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:12.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:37:12.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: -10T04:37:11. 2021-02-10T04:37:12.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: 215840+0000 osd.4 (osd.4 2021-02-10T04:37:12.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: ) 955 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:12.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:37:12.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: -10T04:37:11.215847+0000 osd.4 (osd 2021-02-10T04:37:12.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: .4) 956 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:12.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: cluster 2021-02-10T04:37:12.132806 2021-02-10T04:37:12.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 663 : cluster [DBG] pgmap v1070: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 474 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.7 MiB/s rd, 2.8 MiB/s wr, 12 op/s 2021-02-10T04:37:12.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:37:12.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: -02-10T04:37:12.140163+0000 osd.2 2021-02-10T04:37:12.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: (osd.2) 1893 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:12.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: cluster 2021-02-10T04:37:12.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: 2021-02-10T04:37:12.140171+0000 2021-02-10T04:37:12.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: osd.2 (osd.2) 1894 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:12.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: cluster 2021-02-10T04:37:12.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: 2021-02-10T04:37: 2021-02-10T04:37:12.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: 12.140178+0000 osd.2 (osd.2) 1895 : cluster 2021-02-10T04:37:12.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:12.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:37:12.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: -10T04:37:12. 2021-02-10T04:37:12.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: 140184+0000 osd.2 (osd.2) 1896 : cluster [WRN] 2021-02-10T04:37:12.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:12.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:37:12.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: 10T04:37:12.140189+0000 osd.2 (osd 2021-02-10T04:37:12.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: .2) 1897 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:12.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:37:12.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: -02-10T04:37:12.140194+0000 osd.2 2021-02-10T04:37:12.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: (osd.2) 1898 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:12.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:37:12.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: T04:37:12.140198+0000 osd.2 (osd.2 2021-02-10T04:37:12.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: ) 1899 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:12.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:37:12.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: 10T04:37:12.140201+0000 osd.2 (osd.2 2021-02-10T04:37:12.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: ) 1900 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:12.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: cluster 2021-02-10T04:37: 2021-02-10T04:37:12.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: 12.140204+0000 osd.2 (osd.2) 1901 : 2021-02-10T04:37:12.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:12.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: cluster 2021-02-10T04:37:12.140207+0000 2021-02-10T04:37:12.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: osd.2 (osd.2) 1902 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:12.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:37:12.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: -02-10T04:37:12.202913+0000 mon.a ( 2021-02-10T04:37:12.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: mon.0) 1469 : cluster [DBG] 2021-02-10T04:37:12.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[29319]: osdmap e614: 8 total, 8 up, 8 in 2021-02-10T04:37:12.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: cluster 2021-02-10T04:37: 2021-02-10T04:37:12.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: 11.215815+0000 osd.4 (osd.4) 953 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:12.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: cluster 2021-02-10T04:37:11.215834+0000 2021-02-10T04:37:12.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: osd.4 (osd.4) 954 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:12.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:37:12.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: -10T04:37:11.215840+0000 osd.4 ( 2021-02-10T04:37:12.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: osd.4) 955 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:12.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:37:12.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: 10T04:37:11.215847+0000 osd.4 (osd.4 2021-02-10T04:37:12.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: ) 956 : cluster [WRN] 2021-02-10T04:37:12.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:12.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:12.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: 02-10T04:37:12.132806+0000 mgr.y 2021-02-10T04:37:12.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: (mgr.14140) 663 : cluster [DBG] pgmap v1070: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 474 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.7 MiB/s rd, 2.8 MiB/s wr, 12 op/s 2021-02-10T04:37:12.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:37:12.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: -10T04:37:12.140163+0000 osd.2 (osd 2021-02-10T04:37:12.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: .2) 1893 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:12.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:37:12.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: :37:12.140171+0000 osd.2 (osd.2) 2021-02-10T04:37:12.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: 1894 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:12.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:37:12.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: 37:12.140178+0000 osd.2 (osd.2) 2021-02-10T04:37:12.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: 1895 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:12.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:37:12.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: :37:12.140184+0000 2021-02-10T04:37:12.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: osd.2 (osd.2) 1896 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:12.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:37:12.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: T 2021-02-10T04:37:12.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: 04:37:12.140189+0000 osd.2 (osd. 2021-02-10T04:37:12.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: 2) 1897 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:12.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:12.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: 02-10T04:37:12.140194 2021-02-10T04:37:12.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 2021-02-10T04:37:12.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: 1898 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:12.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:37:12.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: -10T04:37:12.140198+0000 osd.2 (osd 2021-02-10T04:37:12.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: .2) 1899 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:12.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:12.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: -02-10T04:37:12.140201 2021-02-10T04:37:12.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1900 : cluster 2021-02-10T04:37:12.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:12.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: cluster 2021-02-10T04:37 2021-02-10T04:37:12.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: :12.140204+0000 osd.2 (osd 2021-02-10T04:37:12.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: .2) 1901 : cluster 2021-02-10T04:37:12.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:12.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:37:12.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: 04:37:12. 2021-02-10T04:37:12.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: 140207+0000 osd.2 (osd.2) 1902 : cluster 2021-02-10T04:37:12.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:12.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: cluster 2021-02-10T04:37: 2021-02-10T04:37:12.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: 12.202913+0000 mon.a (mon.0) 1469 : cluster 2021-02-10T04:37:12.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:12 smithi085 conmon[35961]: [DBG] osdmap e614: 8 total, 8 up, 8 in 2021-02-10T04:37:12.956 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:37:13.197 INFO:tasks.rados.rados.0.smithi085.stdout:2362: finishing copy_from to smithi08575397-14 2021-02-10T04:37:13.198 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 784 (ObjNum 756 snap 241 seq_num 756) dirty exists 2021-02-10T04:37:13.198 INFO:tasks.rados.rados.0.smithi085.stdout:2362: finishing copy_from racing read to smithi08575397-14 2021-02-10T04:37:13.198 INFO:tasks.rados.rados.0.smithi085.stdout:2363: finishing write tid 1 to smithi08575397-48 2021-02-10T04:37:13.198 INFO:tasks.rados.rados.0.smithi085.stdout:2363: finishing write tid 2 to smithi08575397-48 2021-02-10T04:37:13.198 INFO:tasks.rados.rados.0.smithi085.stdout:2363: finishing write tid 3 to smithi08575397-48 2021-02-10T04:37:13.198 INFO:tasks.rados.rados.0.smithi085.stdout:2363: finishing write tid 4 to smithi08575397-48 2021-02-10T04:37:13.198 INFO:tasks.rados.rados.0.smithi085.stdout:2363: finishing write tid 5 to smithi08575397-48 2021-02-10T04:37:13.198 INFO:tasks.rados.rados.0.smithi085.stdout:2363: finishing write tid 6 to smithi08575397-48 2021-02-10T04:37:13.198 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 866 (ObjNum 818 snap 265 seq_num 818) dirty exists 2021-02-10T04:37:13.198 INFO:tasks.rados.rados.0.smithi085.stdout:2363: left oid 48 (ObjNum 818 snap 265 seq_num 818) 2021-02-10T04:37:13.198 INFO:tasks.rados.rados.0.smithi085.stdout:2362: done (11 left) 2021-02-10T04:37:13.199 INFO:tasks.rados.rados.0.smithi085.stdout:2363: done (10 left) 2021-02-10T04:37:13.199 INFO:tasks.rados.rados.0.smithi085.stdout:2364: done (9 left) 2021-02-10T04:37:13.199 INFO:tasks.rados.rados.0.smithi085.stdout:2365: copy_from oid 44 from oid 35 current snap is 265 2021-02-10T04:37:13.199 INFO:tasks.rados.rados.0.smithi085.stdout:2366: rollback oid 50 current snap is 265 2021-02-10T04:37:13.199 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 50 to 264 2021-02-10T04:37:13.199 INFO:tasks.rados.rados.0.smithi085.stdout:2367: copy_from oid 22 from oid 13 current snap is 265 2021-02-10T04:37:13.199 INFO:tasks.rados.rados.0.smithi085.stdout:2368: snap_create 2021-02-10T04:37:13.200 INFO:tasks.rados.rados.0.smithi085.stdout:2367: finishing copy_from to smithi08575397-22 2021-02-10T04:37:13.200 INFO:tasks.rados.rados.0.smithi085.stdout:2367: got expected ENOENT (src dne) 2021-02-10T04:37:13.200 INFO:tasks.rados.rados.0.smithi085.stdout:2367: finishing copy_from racing read to smithi08575397-22 2021-02-10T04:37:13.212 INFO:tasks.rados.rados.0.smithi085.stdout:2366: finishing rollback tid 1 to smithi08575397-50 2021-02-10T04:37:13.212 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 1130 (ObjNum 1 snap 0 seq_num 82799920) dirty dne 2021-02-10T04:37:13.253 INFO:tasks.rados.rados.0.smithi085.stdout:2365: finishing copy_from to smithi08575397-44 2021-02-10T04:37:13.253 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 844 (ObjNum 791 snap 256 seq_num 791) dirty exists 2021-02-10T04:37:13.253 INFO:tasks.rados.rados.0.smithi085.stdout:2365: finishing copy_from racing read to smithi08575397-44 2021-02-10T04:37:13.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[49106]: debug 2021-02-10T04:37:13.144+0000 7f77a3e94700 -1 osd.2 614 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:37:13.540 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:13 smithi164 conmon[33677]: debug 2021-02-10T04:37:13.182+0000 7fbec19a1700 -1 osd.4 614 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:13.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:13 smithi164 conmon[29288]: cluster 2021-02-10T04:37:12.209317 2021-02-10T04:37:13.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:13 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 957 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:13 smithi164 conmon[29288]: cluster 2021-02-10T04:37:12.209326+0000 osd.4 (osd.4) 958 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:13 smithi164 conmon[29288]: cluster 2021-02-10T04:37:12.209333+0000 osd.4 (osd.4) 959 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:13 smithi164 conmon[29288]: cluster 2021-02-10T04:37:12.209338+0000 osd.4 (osd.4) 960 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:13 smithi164 conmon[29288]: cluster 2021-02-10T04:37:13.145513+0000 osd.2 (osd.2) 1903 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:13 smithi164 conmon[29288]: cluster 2021-02-10T04:37:13.145522+0000 osd.2 (osd.2) 1904 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:13 smithi164 conmon[29288]: cluster 2021-02-10T04:37:13.145529+0000 osd.2 (osd.2) 1905 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:13 smithi164 conmon[29288]: cluster 2021-02-10T04:37:13.145535+0000 osd.2 (osd.2) 1906 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:13 smithi164 conmon[29288]: cluster 2021-02-10T04:37:13.145544+0000 osd.2 (osd.2) 1907 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:13 smithi164 conmon[29288]: cluster 2021-02-10T04:37:13.145550+0000 osd.2 (osd.2) 1908 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:13 smithi164 conmon[29288]: cluster 2021-02-10T04:37:13.145556+0000 osd.2 (osd.2) 1909 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:13 smithi164 conmon[29288]: cluster 2021-02-10T04:37:13.145563+0000 osd.2 (osd.2) 1910 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:13 smithi164 conmon[29288]: cluster 2021-02-10T04:37:13.145579+0000 osd.2 (osd.2) 1911 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:13 smithi164 conmon[29288]: cluster 2021-02-10T04:37:13.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:13 smithi164 conmon[29288]: 2021-02-10T04:37:13.145585+0000 osd.2 (osd.2) 1912 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:13.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:13 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:37:13.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:13 smithi164 conmon[29288]: -02-10T04:37: 2021-02-10T04:37:13.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:13 smithi164 conmon[29288]: 13.188414 2021-02-10T04:37:13.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:13 smithi164 conmon[29288]: +0000 mon.a (mon. 2021-02-10T04:37:13.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:13 smithi164 conmon[29288]: 0) 1470 : cluster [WRN] Health check update: 14 slow ops, oldest one blocked for 462 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:37:13.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:13 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:37:13.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:13 smithi164 conmon[29288]: 02-10T04:37:13.194016+0000 mon.a 2021-02-10T04:37:13.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:13 smithi164 conmon[29288]: (mon.0) 1471 : cluster [DBG] osdmap e615: 8 total, 8 up, 8 in 2021-02-10T04:37:13.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: cluster 2021-02-10T04:37:12.209317+0000 2021-02-10T04:37:13.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: osd.4 (osd.4) 957 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:13.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: cluster 2021-02-10T04:37: 2021-02-10T04:37:13.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: 12.209326+0000 osd.4 (osd.4) 958 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:13.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: cluster 2021-02-10T04:37:12 2021-02-10T04:37:13.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: .209333+0000 osd.4 (osd.4) 959 : cluster 2021-02-10T04:37:13.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:13.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:37:13.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: 04:37:12.209338+0000 osd.4 2021-02-10T04:37:13.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: (osd.4) 960 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:13.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: cluster 2021-02-10T04:37:13.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: 2021-02-10T04:37:13.145513 2021-02-10T04:37:13.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1903 : cluster [WRN] 2021-02-10T04:37:13.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:13.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:37:13.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: T04:37:13.145522+0000 osd.2 ( 2021-02-10T04:37:13.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: osd.2) 1904 : cluster 2021-02-10T04:37:13.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:13.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:37:13.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: -10T04:37:13.145529+0000 2021-02-10T04:37:13.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: osd.2 (osd.2) 1905 : cluster 2021-02-10T04:37:13.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:13.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: cluster 2021-02-10T04:37:13.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: 2021-02-10T04:37:13.145535 2021-02-10T04:37:13.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1906 : cluster [WRN] 2021-02-10T04:37:13.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:13.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:37:13.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: -10T04:37:13.145544+0000 2021-02-10T04:37:13.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: osd.2 (osd.2) 1907 : cluster [WRN] 2021-02-10T04:37:13.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:13.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:37:13.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: T04:37:13.145550+0000 osd.2 2021-02-10T04:37:13.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: (osd.2) 1908 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:13.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: 2021-02-10T04:37:13.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:37:13.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: 04:37:13.145556+0000 osd.2 (osd 2021-02-10T04:37:13.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: .2) 1909 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:13.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:13.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: 02-10T04:37:13.145563+0000 osd.2 (osd.2 2021-02-10T04:37:13.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: ) 1910 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:13.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:13.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: 02-10T04:37:13.145579+0000 2021-02-10T04:37:13.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: osd.2 (osd.2) 1911 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:13.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:13.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: 02-10T04:37:13.145585+0000 osd.2 (osd 2021-02-10T04:37:13.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: .2) 1912 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:13.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:13.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: 02-10T04:37:13.188414 2021-02-10T04:37:13.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1470 : cluster [WRN] 2021-02-10T04:37:13.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: Health check update: 14 slow ops, oldest one blocked for 462 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:37:13.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: cluster 2021-02-10T04:37:13.194016+0000 2021-02-10T04:37:13.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: mon.a (mon.0) 1471 : cluster 2021-02-10T04:37:13.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[29319]: [DBG] osdmap e615: 8 total, 8 up, 8 in 2021-02-10T04:37:13.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: cluster 2021-02-10T04:37:13.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:37:13.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: :37:12.209317+0000 osd.4 (osd.4) 957 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:13.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:13.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: 02-10T04:37:12.209326+0000 osd.4 ( 2021-02-10T04:37:13.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: osd.4) 958 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:13.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:37:13.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: 10T04:37:12.209333+0000 osd.4 (osd 2021-02-10T04:37:13.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: .4) 959 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:13.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:37:13.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: 04:37:12.209338+0000 osd.4 (osd.4 2021-02-10T04:37:13.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: ) 960 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:13.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: cluster 2021-02-10T04:37 2021-02-10T04:37:13.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: :13.145513+0000 osd.2 (osd.2) 2021-02-10T04:37:13.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: 1903 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:13.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: cluster 2021-02-10T04:37:13.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:37:13.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: 37:13.145522+0000 osd.2 2021-02-10T04:37:13.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: (osd.2) 1904 : cluster 2021-02-10T04:37:13.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:13.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: cluster 2021-02-10T04:37:13.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: 2021-02-10T04:37: 2021-02-10T04:37:13.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: 13.145529+0000 osd.2 (osd.2) 1905 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:13.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:13.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: -02-10T04:37:13.145535+0000 osd.2 (osd. 2021-02-10T04:37:13.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: 2) 1906 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:13.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:37:13.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: 10T04:37:13.145544+0000 osd.2 ( 2021-02-10T04:37:13.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: osd.2) 1907 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:13.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:13.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: -02-10T04:37:13.145550+0000 2021-02-10T04:37:13.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: osd.2 (osd.2) 1908 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:13.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:37:13.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: -10T04:37:13.145556+0000 osd.2 (osd 2021-02-10T04:37:13.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: .2) 1909 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:13.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:37:13.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: 37:13.145563+0000 osd.2 (osd.2) 1910 2021-02-10T04:37:13.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:13.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: cluster 2021-02-10T04:37:13.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: 2021-02-10T04:37:13.145579+0000 osd.2 (osd.2 2021-02-10T04:37:13.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: ) 1911 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:13.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:37:13.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: T04:37:13.145585+0000 2021-02-10T04:37:13.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: osd.2 (osd.2) 1912 2021-02-10T04:37:13.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:13.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: cluster 2021-02-10T04:37:13.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:37:13.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: 37:13.188414+0000 mon.a ( 2021-02-10T04:37:13.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: mon.0) 1470 : cluster 2021-02-10T04:37:13.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: [WRN] Health check update: 14 slow ops, oldest one blocked for 462 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:37:13.958 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:13.958 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: -02-10T04:37 2021-02-10T04:37:13.958 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: :13.194016+0000 mon.a (mon.0 2021-02-10T04:37:13.958 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:13 smithi085 conmon[35961]: ) 1471 : cluster [DBG] osdmap e615: 8 total, 8 up, 8 in 2021-02-10T04:37:14.195 INFO:tasks.rados.rados.0.smithi085.stdout:2365: done (12 left) 2021-02-10T04:37:14.196 INFO:tasks.rados.rados.0.smithi085.stdout:2366: done (11 left) 2021-02-10T04:37:14.196 INFO:tasks.rados.rados.0.smithi085.stdout:2367: done (10 left) 2021-02-10T04:37:14.196 INFO:tasks.rados.rados.0.smithi085.stdout:2368: done (9 left) 2021-02-10T04:37:14.196 INFO:tasks.rados.rados.0.smithi085.stdout:2369: copy_from oid 3 from oid 50 current snap is 266 2021-02-10T04:37:14.196 INFO:tasks.rados.rados.0.smithi085.stdout:2370: copy_from oid 8 from oid 43 current snap is 266 2021-02-10T04:37:14.196 INFO:tasks.rados.rados.0.smithi085.stdout:2371: snap_remove snap 262 2021-02-10T04:37:14.535 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[49106]: debug 2021-02-10T04:37:14.535 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[49106]: 2021-02-10T04:37:14.192+0000 7f77a3e94700 -1 osd.2 615 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:37:14.546 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[33677]: debug 2021-02-10T04:37:14.199+0000 7fbec19a1700 -1 osd.4 616 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:14.546 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: cluster 2021-02-10T04:37:13.183339+0000 osd.4 (osd.4) 961 : cluster [WRN] 2021-02-10T04:37:14.546 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:14.859 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: cluster 2021-02-10T04:37:14.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:37:14.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: 37:13.183339+0000 osd.4 (osd.4) 961 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:14.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: cluster 2021-02-10T04:37:13.183345+0000 osd.4 (osd 2021-02-10T04:37:14.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: .4) 962 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:14.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:37:14.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: -10T04:37:13.183349+0000 osd.4 2021-02-10T04:37:14.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: (osd.4) 963 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:14.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:14.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: 02-10T04:37:13.183352+0000 osd.4 ( 2021-02-10T04:37:14.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: osd.4) 964 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:14.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:14.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: -02-10T04:37:14.133638 2021-02-10T04:37:14.861 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 664 : cluster [DBG] 2021-02-10T04:37:14.861 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: pgmap v1073: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 484 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 3.6 MiB/s wr, 12 op/s 2021-02-10T04:37:14.861 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: cluster 2021-02-10T04:37 2021-02-10T04:37:14.861 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: :14.192567+0000 osd.2 (osd.2) 2021-02-10T04:37:14.861 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: 1913 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:14.861 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:37:14.861 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: 04:37:14.192585+0000 osd.2 (osd. 2021-02-10T04:37:14.861 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: 2) 1914 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:14.861 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:14.861 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: 02-10T04:37:14.192592+0000 2021-02-10T04:37:14.861 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: osd.2 (osd.2) 1915 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:14.861 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: 2021-02-10T04:37:14.861 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:14.862 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: 02-10T04:37:14.192598+0000 osd.2 (osd.2) 1916 2021-02-10T04:37:14.862 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:14.862 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:14.862 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: 02-10T04:37:14. 2021-02-10T04:37:14.862 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: 192603+0000 osd.2 (osd.2) 1917 : cluster 2021-02-10T04:37:14.862 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:14.862 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: cluster 2021-02-10T04:37: 2021-02-10T04:37:14.862 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: 14.192609+0000 osd.2 (osd.2) 1918 : cluster [WRN] 2021-02-10T04:37:14.862 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:14.862 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:37:14.862 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: 10T04:37:14.192616 2021-02-10T04:37:14.862 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: +0000 osd.2 (osd. 2021-02-10T04:37:14.862 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: 2) 1919 : cluster 2021-02-10T04:37:14.863 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:14.863 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:37:14.863 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: 37:14.192621+0000 osd.2 (osd.2) 1920 2021-02-10T04:37:14.863 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:14.863 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: cluster 2021-02-10T04:37 2021-02-10T04:37:14.863 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: :14.192627+0000 osd.2 (osd.2) 1921 2021-02-10T04:37:14.863 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:14.863 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: cluster 2021-02-10T04:37 2021-02-10T04:37:14.863 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: :14.192636+0000 osd.2 (osd.2) 1922 2021-02-10T04:37:14.863 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:14.863 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:37:14.863 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: :37:14.195940+0000 mon.a (mon. 2021-02-10T04:37:14.863 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[35961]: 0) 1472 : cluster [DBG] osdmap e616: 8 total, 8 up, 8 in 2021-02-10T04:37:14.864 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[29319]: cluster 2021-02-10T04:37:13.183339+0000 2021-02-10T04:37:14.864 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[29319]: osd.4 (osd.4) 961 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:14.864 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[29319]: cluster 2021-02-10T04:37:13.183345+0000 osd.4 (osd.4) 962 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:14.864 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[29319]: cluster 2021-02-10T04:37:13.183349+0000 osd.4 (osd.4) 963 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:14.864 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[29319]: cluster 2021-02-10T04:37:13.183352+0000 osd.4 (osd.4) 964 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:14.864 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[29319]: cluster 2021-02-10T04:37:14.133638+0000 mgr.y (mgr.14140) 664 : cluster [DBG] pgmap v1073: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 484 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 3.6 MiB/s wr, 12 op/s 2021-02-10T04:37:14.864 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[29319]: cluster 2021-02-10T04:37:14.192567+0000 osd.2 (osd.2) 1913 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:14.865 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[29319]: cluster 2021-02-10T04:37:14.192585+0000 osd.2 (osd.2) 1914 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:14.865 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[29319]: cluster 2021-02-10T04:37:14.192592+0000 osd.2 (osd.2) 1915 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:14.865 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[29319]: cluster 2021-02-10T04:37:14.192598+0000 osd.2 (osd.2) 1916 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:14.865 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[29319]: cluster 2021-02-10T04:37:14.192603+0000 osd.2 (osd.2) 1917 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:14.865 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[29319]: cluster 2021-02-10T04:37:14.192609+0000 osd.2 (osd.2) 1918 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:14.865 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[29319]: cluster 2021-02-10T04:37:14.192616+0000 osd.2 (osd. 2021-02-10T04:37:14.865 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[29319]: 2) 1919 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:14.865 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:37:14.865 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[29319]: 37:14.192621+0000 osd.2 (osd.2) 2021-02-10T04:37:14.865 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[29319]: 1920 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:14.865 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[29319]: cluster 2021-02-10T04:37:14.865 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[29319]: 2021-02-10T04:37: 2021-02-10T04:37:14.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[29319]: 14.192627+0000 osd.2 (osd.2) 1921 2021-02-10T04:37:14.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:14.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[29319]: cluster 2021-02-10T04:37:14.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:37:14.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[29319]: :37:14.192636+0000 osd.2 (osd. 2021-02-10T04:37:14.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[29319]: 2) 1922 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:14.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:37:14.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[29319]: -02-10T04:37:14.195940 2021-02-10T04:37:14.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1472 : cluster 2021-02-10T04:37:14.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:14 smithi085 conmon[29319]: [DBG] osdmap e616: 8 total, 8 up, 8 in 2021-02-10T04:37:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: cluster 2021-02-10T04:37:13.183345+0000 osd.4 (osd.4) 962 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: cluster 2021-02-10T04:37:13.183349+0000 osd.4 (osd.4) 963 : cluster 2021-02-10T04:37:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: cluster 2021-02-10T04:37:13.183352+0000 osd.4 (osd.4) 2021-02-10T04:37:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: 964 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:37:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: 04:37:14.133638+0000 mgr.y (mgr.14140) 664 2021-02-10T04:37:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: : cluster [DBG] pgmap v1073: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 484 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 3.6 MiB/s wr, 12 op/s 2021-02-10T04:37:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:37:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: 10T04:37:14.192567+0000 osd.2 (osd.2 2021-02-10T04:37:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: ) 1913 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: cluster 2021-02-10T04:37:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: 2021-02-10T04:37:14.192585+0000 2021-02-10T04:37:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: osd.2 (osd.2) 1914 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:37:14.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: 10T04:37:14.192592+0000 osd.2 (osd.2) 1915 2021-02-10T04:37:14.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:14.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: cluster 2021-02-10T04:37: 2021-02-10T04:37:14.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: 14.192598+0000 osd.2 (osd.2) 1916 : cluster 2021-02-10T04:37:14.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:14.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: cluster 2021-02-10T04:37:14.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: 2021-02-10T04:37:14.192603+0000 osd.2 (osd.2) 1917 : cluster [WRN] 2021-02-10T04:37:14.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:14.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: cluster 2021-02-10T04:37:14.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: 2021-02-10T04:37:14.192609+0000 osd.2 (osd.2) 1918 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:14.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:37:14.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: :37:14.192616+0000 osd.2 (osd.2) 1919 : cluster [WRN] 2021-02-10T04:37:14.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:14.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: cluster 2021-02-10T04:37:14.192621+0000 osd.2 ( 2021-02-10T04:37:14.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: osd.2) 1920 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:14.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:37:14.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: T04:37:14.192627+0000 osd.2 (osd.2) 1921 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:14.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: cluster 2021-02-10T04:37:14.192636+0000 osd.2 (osd.2) 2021-02-10T04:37:14.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: 1922 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:14.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: cluster 2021-02-10T04:37:14 2021-02-10T04:37:14.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:14 smithi164 conmon[29288]: .195940+0000 mon.a (mon.0) 1472 : cluster [DBG] osdmap e616: 8 total, 8 up, 8 in 2021-02-10T04:37:15.201 INFO:tasks.rados.rados.0.smithi085.stdout:2371: done (11 left) 2021-02-10T04:37:15.201 INFO:tasks.rados.rados.0.smithi085.stdout:2372: delete oid 37 current snap is 266 2021-02-10T04:37:15.201 INFO:tasks.rados.rados.0.smithi085.stdout:2369: finishing copy_from to smithi08575397-3 2021-02-10T04:37:15.201 INFO:tasks.rados.rados.0.smithi085.stdout:2369: got expected ENOENT (src dne) 2021-02-10T04:37:15.202 INFO:tasks.rados.rados.0.smithi085.stdout:2369: finishing copy_from racing read to smithi08575397-3 2021-02-10T04:37:15.202 INFO:tasks.rados.rados.0.smithi085.stdout:2370: finishing copy_from to smithi08575397-8 2021-02-10T04:37:15.202 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 1132 (ObjNum 797 snap 256 seq_num 797) dirty exists 2021-02-10T04:37:15.202 INFO:tasks.rados.rados.0.smithi085.stdout:2370: finishing copy_from racing read to smithi08575397-8 2021-02-10T04:37:15.206 INFO:tasks.rados.rados.0.smithi085.stdout:2369: done (11 left) 2021-02-10T04:37:15.206 INFO:tasks.rados.rados.0.smithi085.stdout:2370: done (10 left) 2021-02-10T04:37:15.206 INFO:tasks.rados.rados.0.smithi085.stdout:2372: done (9 left) 2021-02-10T04:37:15.206 INFO:tasks.rados.rados.0.smithi085.stdout:2373: read oid 5 snap -1 2021-02-10T04:37:15.206 INFO:tasks.rados.rados.0.smithi085.stdout:2373: expect deleted 2021-02-10T04:37:15.206 INFO:tasks.rados.rados.0.smithi085.stdout:2374: delete oid 26 current snap is 266 2021-02-10T04:37:15.210 INFO:tasks.rados.rados.0.smithi085.stdout:2373: done (10 left) 2021-02-10T04:37:15.210 INFO:tasks.rados.rados.0.smithi085.stdout:2374: done (9 left) 2021-02-10T04:37:15.210 INFO:tasks.rados.rados.0.smithi085.stdout:2375: rollback oid 31 current snap is 266 2021-02-10T04:37:15.210 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 31 to 254 2021-02-10T04:37:15.210 INFO:tasks.rados.rados.0.smithi085.stdout:2376: rollback oid 46 current snap is 266 2021-02-10T04:37:15.210 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 46 to 254 2021-02-10T04:37:15.211 INFO:tasks.rados.rados.0.smithi085.stdout:2377: write oid 48 current snap is 266 2021-02-10T04:37:15.211 INFO:tasks.rados.rados.0.smithi085.stdout:2377: seq_num 821 ranges {598547=629200,1876540=491946,3046277=681499} 2021-02-10T04:37:15.217 INFO:tasks.rados.rados.0.smithi085.stdout:2377: writing smithi08575397-48 from 598547 to 1227747 tid 1 2021-02-10T04:37:15.224 INFO:tasks.rados.rados.0.smithi085.stdout:2377: writing smithi08575397-48 from 1876540 to 2368486 tid 2 2021-02-10T04:37:15.232 INFO:tasks.rados.rados.0.smithi085.stdout:2377: writing smithi08575397-48 from 3046277 to 3727776 tid 3 2021-02-10T04:37:15.232 INFO:tasks.rados.rados.0.smithi085.stdout:2378: snap_create 2021-02-10T04:37:15.233 INFO:tasks.rados.rados.0.smithi085.stdout:2375: finishing rollback tid 0 to smithi08575397-31 2021-02-10T04:37:15.233 INFO:tasks.rados.rados.0.smithi085.stdout:2375: finishing rollback tid 1 to smithi08575397-31 2021-02-10T04:37:15.233 INFO:tasks.rados.rados.0.smithi085.stdout:2375: finishing rollback tid 2 to smithi08575397-31 2021-02-10T04:37:15.233 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 804 (ObjNum 768 snap 249 seq_num 768) dirty exists 2021-02-10T04:37:15.233 INFO:tasks.rados.rados.0.smithi085.stdout:2376: finishing rollback tid 0 to smithi08575397-46 2021-02-10T04:37:15.233 INFO:tasks.rados.rados.0.smithi085.stdout:2376: finishing rollback tid 1 to smithi08575397-46 2021-02-10T04:37:15.233 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 1083 (ObjNum 1215200384 snap 32576 seq_num 1215200384) dirty dne 2021-02-10T04:37:15.233 INFO:tasks.rados.rados.0.smithi085.stdout:2377: finishing write tid 1 to smithi08575397-48 2021-02-10T04:37:15.233 INFO:tasks.rados.rados.0.smithi085.stdout:2377: finishing write tid 2 to smithi08575397-48 2021-02-10T04:37:15.237 INFO:tasks.rados.rados.0.smithi085.stdout:2377: finishing write tid 3 to smithi08575397-48 2021-02-10T04:37:15.237 INFO:tasks.rados.rados.0.smithi085.stdout:2377: finishing write tid 4 to smithi08575397-48 2021-02-10T04:37:15.237 INFO:tasks.rados.rados.0.smithi085.stdout:2377: finishing write tid 5 to smithi08575397-48 2021-02-10T04:37:15.237 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 870 (ObjNum 821 snap 266 seq_num 821) dirty exists 2021-02-10T04:37:15.238 INFO:tasks.rados.rados.0.smithi085.stdout:2377: left oid 48 (ObjNum 821 snap 266 seq_num 821) 2021-02-10T04:37:15.339 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:37:15.369 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:37:15.401 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:37:15.437 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:37:15.480 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:37:15.514 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:37:15.546 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:37:15.547 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[49106]: debug 2021-02-10T04:37:15.236+0000 7f77a3e94700 -1 osd.2 617 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) 2021-02-10T04:37:15.547 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[33677]: debug 2021-02-10T04:37:15.547 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[33677]: 2021-02-10T04:37:15.169+0000 7fbec19a1700 -1 osd.4 616 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:15.549 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: cluster 2021-02-10T04:37:14. 2021-02-10T04:37:15.549 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: 200661+0000 osd.4 (osd.4) 965 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:15.550 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: cluster 2021-02-10T04:37:15.550 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:37:15.550 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: 04:37:14.200669+0000 osd.4 (osd 2021-02-10T04:37:15.550 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: .4) 966 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:15.550 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: cluster 2021-02-10T04:37:15.550 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:37:15.550 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: :37:14.200675+0000 osd.4 (osd.4) 967 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:15.550 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:37:15.550 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: 02-10T04:37:14.200681+0000 osd.4 ( 2021-02-10T04:37:15.550 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: osd.4) 968 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:15.550 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:37:15.550 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: -02-10T04:37:15.196974+0000 2021-02-10T04:37:15.551 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: mon.a (mon.0) 1473 : cluster [DBG] osdmap e617: 8 total, 8 up, 8 in 2021-02-10T04:37:15.551 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: 2021-02-10T04:37:15.551 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:37:15.551 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: T04:37:15.236938+0000 osd.2 (osd. 2021-02-10T04:37:15.551 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: 2) 1923 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:15.551 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:37:15.551 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: -10T04:37:15.236945+0000 osd.2 (osd.2) 2021-02-10T04:37:15.551 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: 1924 : cluster [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:15.551 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: cluster 2021-02-10T04:37:15.236951 2021-02-10T04:37:15.551 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1925 : cluster [WRN] 2021-02-10T04:37:15.551 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:15.551 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:37:15.551 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: 37:15.236957+0000 osd.2 (osd.2) 2021-02-10T04:37:15.552 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: 1926 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:15.552 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:37:15.552 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: 04:37:15.236963+0000 osd.2 (osd.2) 1927 : cluster 2021-02-10T04:37:15.552 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:15.552 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:37:15.552 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: 02-10T04:37:15.236969+0000 osd.2 (osd.2) 1928 : cluster 2021-02-10T04:37:15.552 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:15.552 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: 2021-02-10T04:37:15.553 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: cluster 2021-02-10T04:37:15.553 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: 2021-02-10 2021-02-10T04:37:15.553 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: T04:37:15.236976+0000 osd.2 ( 2021-02-10T04:37:15.553 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: osd.2) 1929 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:15.553 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: cluster 2021-02-10T04:37:15.553 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: 2021-02-10T04:37:15. 2021-02-10T04:37:15.553 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: 236980+0000 osd.2 (osd.2) 1930 : cluster 2021-02-10T04:37:15.553 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: [WRN] 2021-02-10T04:37:15.553 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:15.553 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: cluster 2021-02-10T04:37:15.553 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:37:15.553 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: 37:15.236983+0000 osd.2 (osd.2 2021-02-10T04:37:15.553 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: ) 1931 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:15.553 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:37:15.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: -02-10T04:37:15.236987+0000 osd.2 (osd 2021-02-10T04:37:15.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: .2) 1932 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:15.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: audit 2021-02-10T04:37:15.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:37:15.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: 37:15.300439+0000 2021-02-10T04:37:15.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: mon.a (mon.0) 1474 : audit [DBG] from='client.? 172.21.15.85:0/2672985359' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:37:15.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:15 smithi164 conmon[29288]: 2021-02-10T04:37:15.555 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: cluster 2021-02-10T04:37:15.555 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.555 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021 2021-02-10T04:37:15.555 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: - 2021-02-10T04:37:15.556 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 02-10T04:37:14. 2021-02-10T04:37:15.556 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 200661+0000 osd.4 (osd.4) 965 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:15.556 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: cluster 2021-02-10T04:37:15.556 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.556 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021 2021-02-10T04:37:15.556 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: - 2021-02-10T04:37:15.556 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 02 2021-02-10T04:37:15.556 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: - 2021-02-10T04:37:15.556 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 10 2021-02-10T04:37:15.556 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: T 2021-02-10T04:37:15.556 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 04 2021-02-10T04:37:15.556 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.557 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 37 2021-02-10T04:37:15.557 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.557 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 14.200669+0000 osd.4 ( 2021-02-10T04:37:15.557 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: osd.4) 966 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:15.557 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:37:15.557 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 10T04:37:14.200675+0000 osd.4 (osd.4) 967 2021-02-10T04:37:15.557 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:15.557 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.557 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: cluster 2021-02-10T04:37:15.557 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:37:15.557 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 10T04:37:14.200681+0000 osd.4 (osd.4) 2021-02-10T04:37:15.557 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 968 2021-02-10T04:37:15.557 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.558 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:15.558 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: cluster 2021-02-10T04:37:15.558 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.558 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021 2021-02-10T04:37:15.558 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: - 2021-02-10T04:37:15.558 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 02 2021-02-10T04:37:15.558 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: - 2021-02-10T04:37:15.558 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 10 2021-02-10T04:37:15.558 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: T 2021-02-10T04:37:15.558 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 04 2021-02-10T04:37:15.559 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: :37:15.196974+0000 2021-02-10T04:37:15.559 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: mon.a (mon.0) 1473 : cluster [DBG] osdmap e617: 8 total, 8 up, 8 in 2021-02-10T04:37:15.559 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: cluster 2021-02-10T04:37:15.559 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.559 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021 2021-02-10T04:37:15.559 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: - 2021-02-10T04:37:15.559 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: cluster 2021-02-10T04:37:14 2021-02-10T04:37:15.559 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: .200661+0000 osd.4 (osd.4) 965 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:15.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: cluster 2021-02-10T04:37:14.200669+0000 2021-02-10T04:37:15.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: osd.4 (osd.4) 966 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:15.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: cluster 2021-02-10T04:37:15.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: 2021-02-10T04:37:14.200675+0000 osd.4 2021-02-10T04:37:15.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: (osd.4) 967 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:15.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: cluster 2021-02-10T04:37:14.200681+0000 osd.4 (osd 2021-02-10T04:37:15.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: .4) 968 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:15.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: cluster 2021-02-10T04:37:15.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: 2021-02-10T04:37:15.196974+0000 mon.a (mon.0) 1473 : cluster 2021-02-10T04:37:15.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: [DBG] osdmap e617: 8 total, 8 up, 8 in 2021-02-10T04:37:15.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: cluster 2021-02-10T04:37:15.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: 2021-02-10T04:37:15.236938+0000 osd.2 (osd.2) 1923 : cluster [WRN] slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:15.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: 2021-02-10T04:37:15.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:37:15.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: 10T04:37:15.236945+0000 osd.2 (osd.2) 1924 : cluster 2021-02-10T04:37:15.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:15.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:37:15.561 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: 10T04:37:15.236951+0000 osd.2 (osd.2) 1925 : cluster [WRN] slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:15.562 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: cluster 2021-02-10T04:37:15.236957+0000 osd.2 ( 2021-02-10T04:37:15.562 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: osd.2) 1926 : cluster [WRN] slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:15.562 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: cluster 2021-02-10T04:37:15.562 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: 2021-02-10T04:37:15.236963+0000 osd.2 ( 2021-02-10T04:37:15.563 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: osd.2) 1927 : cluster [WRN] slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:15.563 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:37:15.563 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: T04:37:15.236969+0000 osd.2 (osd 2021-02-10T04:37:15.563 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: .2) 1928 : cluster [WRN] slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:15.563 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:15.563 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: 02-10T04:37:15.236976+0000 osd.2 2021-02-10T04:37:15.563 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: (osd.2) 1929 : cluster [WRN] slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:15.563 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:15.563 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: 02-10T04:37:15.236980+0000 osd.2 ( 2021-02-10T04:37:15.563 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: osd.2) 1930 : cluster [WRN] slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:15.563 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:15.563 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: 02-10T04:37:15.236983+0000 osd.2 ( 2021-02-10T04:37:15.563 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: osd.2) 1931 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:15.564 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:37:15.564 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: 10T04:37:15.236987+0000 osd.2 (osd.2) 1932 : cluster 2021-02-10T04:37:15.564 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:15.564 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:37:15.564 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: -10T04:37:15.300439+0000 mon.a ( 2021-02-10T04:37:15.564 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[35961]: mon.0) 1474 : audit [DBG] from='client.? 172.21.15.85:0/2672985359' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:37:15.574 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:37:15.578 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:37:15.578 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:37:15.578 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:37:15.578 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:37:15.578 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:37:15.578 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:37:15.578 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:37:15.578 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:37:15.622 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:37:15.625 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:37:15.909 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 02 2021-02-10T04:37:15.909 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: - 2021-02-10T04:37:15.909 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 10 2021-02-10T04:37:15.909 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: T 2021-02-10T04:37:15.909 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 04 2021-02-10T04:37:15.909 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.910 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 37 2021-02-10T04:37:15.910 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.910 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 15 2021-02-10T04:37:15.910 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: . 2021-02-10T04:37:15.910 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 236938 2021-02-10T04:37:15.910 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: +0000 2021-02-10T04:37:15.910 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.910 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: osd.2 2021-02-10T04:37:15.910 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: ( 2021-02-10T04:37:15.910 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: osd 2021-02-10T04:37:15.910 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: . 2021-02-10T04:37:15.910 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2 2021-02-10T04:37:15.910 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: ) 2021-02-10T04:37:15.911 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 1923 2021-02-10T04:37:15.911 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.911 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: cluster 2021-02-10T04:37:15.911 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.911 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: [WRN] 2021-02-10T04:37:15.911 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.911 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: slow request osd_op(client.14817.0:3882 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 627 in=54b] snapc 128=[128,126,124,123,11d,11c,117,116,111,109,108,107,106,f8,f2,f0,d8] ondisk+write+known_if_redirected e395) initiated 2021-02-10T04:33:32.700402+0000 currently started 2021-02-10T04:37:15.911 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.911 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: cluster 2021-02-10T04:37:15.911 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.911 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021 2021-02-10T04:37:15.911 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: - 2021-02-10T04:37:15.911 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 02 2021-02-10T04:37:15.912 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: - 2021-02-10T04:37:15.912 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 10 2021-02-10T04:37:15.912 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: T 2021-02-10T04:37:15.912 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 04 2021-02-10T04:37:15.912 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.912 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 37 2021-02-10T04:37:15.912 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.912 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 15 2021-02-10T04:37:15.912 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: . 2021-02-10T04:37:15.912 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 236945 2021-02-10T04:37:15.912 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: +0000 2021-02-10T04:37:15.912 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.912 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: osd.2 2021-02-10T04:37:15.913 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: ( 2021-02-10T04:37:15.913 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: osd 2021-02-10T04:37:15.913 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: . 2021-02-10T04:37:15.913 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2 2021-02-10T04:37:15.913 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: ) 2021-02-10T04:37:15.913 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 1924 2021-02-10T04:37:15.913 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.913 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: cluster 2021-02-10T04:37:15.913 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.913 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: [WRN] 2021-02-10T04:37:15.913 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.913 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: slow request osd_op(client.14817.0:3883 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e395) initiated 2021-02-10T04:33:32.700507+0000 currently delayed 2021-02-10T04:37:15.914 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.914 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: cluster 2021-02-10T04:37:15.914 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.914 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021 2021-02-10T04:37:15.914 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: - 2021-02-10T04:37:15.914 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 02 2021-02-10T04:37:15.914 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: - 2021-02-10T04:37:15.914 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 10 2021-02-10T04:37:15.914 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: T 2021-02-10T04:37:15.914 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 04 2021-02-10T04:37:15.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 37 2021-02-10T04:37:15.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 15 2021-02-10T04:37:15.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: . 2021-02-10T04:37:15.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 236951 2021-02-10T04:37:15.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: +0000 2021-02-10T04:37:15.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: osd.2 2021-02-10T04:37:15.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: ( 2021-02-10T04:37:15.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: osd 2021-02-10T04:37:15.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: . 2021-02-10T04:37:15.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2 2021-02-10T04:37:15.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: ) 2021-02-10T04:37:15.916 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 1925 2021-02-10T04:37:15.916 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.916 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: cluster 2021-02-10T04:37:15.916 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.916 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: [WRN] 2021-02-10T04:37:15.916 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.917 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1693 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 0 in=55b] snapc 75=[75,74] ondisk+write+known_if_redirected e215) initiated 2021-02-10T04:30:31.176628+0000 currently started 2021-02-10T04:37:15.917 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.917 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: cluster 2021-02-10T04:37:15.917 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.917 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021 2021-02-10T04:37:15.917 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: - 2021-02-10T04:37:15.917 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 02 2021-02-10T04:37:15.917 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: - 2021-02-10T04:37:15.917 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 10 2021-02-10T04:37:15.917 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: T 2021-02-10T04:37:15.917 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 04 2021-02-10T04:37:15.917 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.917 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 37 2021-02-10T04:37:15.918 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.918 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 15 2021-02-10T04:37:15.918 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: . 2021-02-10T04:37:15.918 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 236957 2021-02-10T04:37:15.918 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: +0000 2021-02-10T04:37:15.918 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.918 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: osd.2 2021-02-10T04:37:15.918 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: ( 2021-02-10T04:37:15.918 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: osd 2021-02-10T04:37:15.918 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: . 2021-02-10T04:37:15.918 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2 2021-02-10T04:37:15.918 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: ) 2021-02-10T04:37:15.918 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 1926 2021-02-10T04:37:15.919 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.919 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: cluster 2021-02-10T04:37:15.919 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.919 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: [WRN] 2021-02-10T04:37:15.919 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.919 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1694 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e215) initiated 2021-02-10T04:30:31.176747+0000 currently delayed 2021-02-10T04:37:15.919 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.919 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: cluster 2021-02-10T04:37:15.919 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.919 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021 2021-02-10T04:37:15.919 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: - 2021-02-10T04:37:15.919 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 02 2021-02-10T04:37:15.919 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: - 2021-02-10T04:37:15.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 10 2021-02-10T04:37:15.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: T 2021-02-10T04:37:15.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 04 2021-02-10T04:37:15.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 37 2021-02-10T04:37:15.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 15 2021-02-10T04:37:15.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: . 2021-02-10T04:37:15.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 236963 2021-02-10T04:37:15.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: +0000 2021-02-10T04:37:15.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: osd.2 2021-02-10T04:37:15.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: ( 2021-02-10T04:37:15.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: osd 2021-02-10T04:37:15.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: . 2021-02-10T04:37:15.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2 2021-02-10T04:37:15.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: ) 2021-02-10T04:37:15.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 1927 2021-02-10T04:37:15.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: cluster 2021-02-10T04:37:15.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: [WRN] 2021-02-10T04:37:15.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1164 7.5 7:adfc1887:::smithi08575397-25:head [copy-from ver 60 in=55b] snapc 4f=[4f,4e] ondisk+write+known_if_redirected e176) initiated 2021-02-10T04:29:52.087674+0000 currently started 2021-02-10T04:37:15.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: cluster 2021-02-10T04:37:15.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021 2021-02-10T04:37:15.922 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: - 2021-02-10T04:37:15.922 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 02 2021-02-10T04:37:15.922 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: - 2021-02-10T04:37:15.922 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 10 2021-02-10T04:37:15.922 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: T 2021-02-10T04:37:15.923 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 04 2021-02-10T04:37:15.923 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.923 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 37 2021-02-10T04:37:15.923 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.923 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 15 2021-02-10T04:37:15.923 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: . 2021-02-10T04:37:15.923 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 236969 2021-02-10T04:37:15.923 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: +0000 2021-02-10T04:37:15.924 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.924 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: osd.2 2021-02-10T04:37:15.924 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: ( 2021-02-10T04:37:15.924 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: osd 2021-02-10T04:37:15.924 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: . 2021-02-10T04:37:15.924 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2 2021-02-10T04:37:15.924 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: ) 2021-02-10T04:37:15.924 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 1928 2021-02-10T04:37:15.924 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.924 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: cluster 2021-02-10T04:37:15.924 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.924 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: [WRN] 2021-02-10T04:37:15.924 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.925 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1165 7.5 7:adfc1887:::smithi08575397-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e176) initiated 2021-02-10T04:29:52.087796+0000 currently delayed 2021-02-10T04:37:15.925 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.925 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: cluster 2021-02-10T04:37:15.925 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.925 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021 2021-02-10T04:37:15.925 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: - 2021-02-10T04:37:15.925 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 02 2021-02-10T04:37:15.925 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: - 2021-02-10T04:37:15.925 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 10 2021-02-10T04:37:15.925 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: T 2021-02-10T04:37:15.925 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 04 2021-02-10T04:37:15.925 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.925 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 37 2021-02-10T04:37:15.926 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.926 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 15 2021-02-10T04:37:15.926 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: . 2021-02-10T04:37:15.926 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 236976 2021-02-10T04:37:15.926 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: +0000 2021-02-10T04:37:15.926 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.926 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: osd.2 2021-02-10T04:37:15.926 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: ( 2021-02-10T04:37:15.926 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: osd 2021-02-10T04:37:15.926 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: . 2021-02-10T04:37:15.926 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2 2021-02-10T04:37:15.926 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: ) 2021-02-10T04:37:15.926 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 1929 2021-02-10T04:37:15.927 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.927 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: cluster 2021-02-10T04:37:15.927 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.927 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: [WRN] 2021-02-10T04:37:15.927 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.927 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1816 7.5 7:a5580f12:::smithi08575397-18:head [copy-from ver 121 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.071690+0000 currently started 2021-02-10T04:37:15.927 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.927 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: cluster 2021-02-10T04:37:15.927 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.927 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021 2021-02-10T04:37:15.927 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: - 2021-02-10T04:37:15.927 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 02 2021-02-10T04:37:15.927 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: - 2021-02-10T04:37:15.927 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 10 2021-02-10T04:37:15.928 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: T 2021-02-10T04:37:15.928 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 04 2021-02-10T04:37:15.928 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.928 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 37 2021-02-10T04:37:15.928 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.928 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 15 2021-02-10T04:37:15.928 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: . 2021-02-10T04:37:15.928 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 236980 2021-02-10T04:37:15.928 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: +0000 2021-02-10T04:37:15.928 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.928 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: osd.2 2021-02-10T04:37:15.928 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: ( 2021-02-10T04:37:15.928 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: osd 2021-02-10T04:37:15.929 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: . 2021-02-10T04:37:15.929 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2 2021-02-10T04:37:15.929 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: ) 2021-02-10T04:37:15.929 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 1930 2021-02-10T04:37:15.929 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.929 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: cluster 2021-02-10T04:37:15.929 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.929 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: [WRN] 2021-02-10T04:37:15.929 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.929 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1817 7.5 7:a5580f12:::smithi08575397-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.071824+0000 currently delayed 2021-02-10T04:37:15.929 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.929 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: cluster 2021-02-10T04:37:15.930 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.930 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021 2021-02-10T04:37:15.930 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: - 2021-02-10T04:37:15.930 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 02 2021-02-10T04:37:15.930 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: - 2021-02-10T04:37:15.930 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 10 2021-02-10T04:37:15.930 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: T 2021-02-10T04:37:15.930 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 04 2021-02-10T04:37:15.930 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.931 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 37 2021-02-10T04:37:15.931 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.931 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 15 2021-02-10T04:37:15.931 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: . 2021-02-10T04:37:15.931 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 236983 2021-02-10T04:37:15.931 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: +0000 2021-02-10T04:37:15.931 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.931 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: osd.2 2021-02-10T04:37:15.931 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: ( 2021-02-10T04:37:15.931 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: osd 2021-02-10T04:37:15.931 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: . 2021-02-10T04:37:15.931 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2 2021-02-10T04:37:15.932 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: ) 2021-02-10T04:37:15.932 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 1931 2021-02-10T04:37:15.932 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.932 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: cluster 2021-02-10T04:37:15.932 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.932 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: [WRN] 2021-02-10T04:37:15.932 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.932 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] ondisk+write+known_if_redirected e230) initiated 2021-02-10T04:30:46.083500+0000 currently started 2021-02-10T04:37:15.932 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.932 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: cluster 2021-02-10T04:37:15.932 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.932 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021 2021-02-10T04:37:15.932 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: - 2021-02-10T04:37:15.933 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 02 2021-02-10T04:37:15.933 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: - 2021-02-10T04:37:15.933 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 10 2021-02-10T04:37:15.933 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: T 2021-02-10T04:37:15.933 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 04 2021-02-10T04:37:15.933 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.933 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 37 2021-02-10T04:37:15.933 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.933 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 15 2021-02-10T04:37:15.933 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: . 2021-02-10T04:37:15.933 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 236987 2021-02-10T04:37:15.933 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: +0000 2021-02-10T04:37:15.933 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.934 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: osd.2 2021-02-10T04:37:15.934 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: ( 2021-02-10T04:37:15.934 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: osd 2021-02-10T04:37:15.934 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: . 2021-02-10T04:37:15.934 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2 2021-02-10T04:37:15.934 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: ) 2021-02-10T04:37:15.934 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 1932 2021-02-10T04:37:15.934 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.934 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: cluster 2021-02-10T04:37:15.934 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.934 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: [WRN] 2021-02-10T04:37:15.934 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.934 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e230) initiated 2021-02-10T04:30:46.083641+0000 currently delayed 2021-02-10T04:37:15.935 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.935 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: audit 2021-02-10T04:37:15.935 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.935 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021 2021-02-10T04:37:15.935 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: - 2021-02-10T04:37:15.935 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 02 2021-02-10T04:37:15.935 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: - 2021-02-10T04:37:15.935 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 10 2021-02-10T04:37:15.935 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: T 2021-02-10T04:37:15.935 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 04 2021-02-10T04:37:15.935 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.935 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 37 2021-02-10T04:37:15.935 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.935 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 15 2021-02-10T04:37:15.936 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: . 2021-02-10T04:37:15.936 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 300439 2021-02-10T04:37:15.936 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: +0000 2021-02-10T04:37:15.936 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.936 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: mon.a 2021-02-10T04:37:15.936 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: ( 2021-02-10T04:37:15.936 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: mon 2021-02-10T04:37:15.936 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: . 2021-02-10T04:37:15.936 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 0 2021-02-10T04:37:15.936 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: ) 2021-02-10T04:37:15.936 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 1474 2021-02-10T04:37:15.936 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: : 2021-02-10T04:37:15.936 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: audit 2021-02-10T04:37:15.937 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.937 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: [DBG] 2021-02-10T04:37:15.937 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:15.937 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: from='client.? 172.21.15.85:0/2672985359' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:37:15.937 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:15 smithi085 conmon[29319]: 2021-02-10T04:37:16.201 INFO:tasks.rados.rados.0.smithi085.stdout:2375: done (12 left) 2021-02-10T04:37:16.201 INFO:tasks.rados.rados.0.smithi085.stdout:2376: done (11 left) 2021-02-10T04:37:16.201 INFO:tasks.rados.rados.0.smithi085.stdout:2377: done (10 left) 2021-02-10T04:37:16.201 INFO:tasks.rados.rados.0.smithi085.stdout:2378: done (9 left) 2021-02-10T04:37:16.201 INFO:tasks.rados.rados.0.smithi085.stdout:2379: copy_from oid 44 from oid 38 current snap is 267 2021-02-10T04:37:16.201 INFO:tasks.rados.rados.0.smithi085.stdout:2380: snap_remove snap 239 2021-02-10T04:37:16.204 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is set 2021-02-10T04:37:16.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:16 smithi164 conmon[29288]: cluster 2021-02-10T04:37:15.169983+0000 osd.4 (osd.4) 969 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:16.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:16 smithi164 conmon[29288]: cluster 2021-02-10T04:37:15.169992+0000 osd.4 (osd.4) 970 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:16.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:16 smithi164 conmon[29288]: cluster 2021-02-10T04:37:15.169999+0000 osd.4 (osd 2021-02-10T04:37:16.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:16 smithi164 conmon[29288]: .4) 971 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:16.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:16 smithi164 conmon[29288]: cluster 2021-02-10T04:37:15.170004 2021-02-10T04:37:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:16 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 972 : cluster [WRN] 2021-02-10T04:37:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:16 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:16 smithi164 conmon[29288]: audit 2021-02-10T04: 2021-02-10T04:37:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:16 smithi164 conmon[29288]: 37:15.598631+0000 mgr.y (mgr.14140) 665 : 2021-02-10T04:37:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:16 smithi164 conmon[29288]: audit [DBG] from='client.25099 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:37:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:16 smithi164 conmon[29288]: audit 2021-02-10T04:37:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:16 smithi164 conmon[29288]: 2021-02-10T04:37:15.631164+0000 mon.b (mon.2) 38 : audit [INF] 2021-02-10T04:37:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:16 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/1269175109' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:37:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:16 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:37:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:16 smithi164 conmon[29288]: 04:37:15.631636+0000 mon.a (mon.0) 1475 : audit 2021-02-10T04:37:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:16 smithi164 conmon[29288]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:37:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:16 smithi164 conmon[29288]: cluster 2021-02-10T04:37: 2021-02-10T04:37:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:16 smithi164 conmon[29288]: 16.134394+0000 mgr.y (mgr.14140) 666 : 2021-02-10T04:37:16.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:16 smithi164 conmon[29288]: cluster [DBG] pgmap v1076: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 487 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 15 op/s 2021-02-10T04:37:16.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:16 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:37:16.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:16 smithi164 conmon[29288]: -02-10T04:37:16 2021-02-10T04:37:16.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:16 smithi164 conmon[29288]: .194231+0000 mon.a (mon.0) 2021-02-10T04:37:16.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:16 smithi164 conmon[29288]: 1476 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:37:16.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:16 smithi164 conmon[29288]: audit 2021 2021-02-10T04:37:16.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:16 smithi164 conmon[29288]: -02-10T04:37:16.198374+0000 mon.a 2021-02-10T04:37:16.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:16 smithi164 conmon[29288]: (mon.0) 1477 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:37:16.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:16 smithi164 conmon[29288]: 2021-02-10T04:37:16.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:16 smithi164 conmon[29288]: cluster 2021-02-10T04:37:16.198442 2021-02-10T04:37:16.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:16 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1478 : cluster [DBG] osdmap e618: 8 total, 8 up, 8 in 2021-02-10T04:37:16.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:16 smithi164 conmon[29288]: 2021-02-10T04:37:16.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:37:16.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[29319]: T04:37:15.169983+0000 osd.4 (osd.4) 969 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:16.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[29319]: cluster 2021-02-10T04:37:15.169992+0000 osd.4 (osd.4) 970 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:16.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[29319]: cluster 2021-02-10T04:37:15.169999+0000 osd.4 (osd.4) 971 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:16.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[29319]: cluster 2021-02-10T04:37:15.170004+0000 osd.4 (osd.4) 972 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:16.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[29319]: audit 2021-02-10T04:37:15.598631+0000 mgr.y (mgr.14140) 665 : audit [DBG] from='client.25099 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:37:16.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[29319]: audit 2021-02-10T04:37:15.631164+0000 mon.b (mon.2) 38 : audit [INF] from='client.? 172.21.15.85:0/1269175109' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:37:16.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[29319]: audit 2021-02-10T04:37:15.631636+0000 mon.a (mon.0) 1475 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:37:16.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[29319]: cluster 2021-02-10T04:37:16.134394+0000 mgr.y (mgr.14140) 666 : cluster [DBG] pgmap v1076: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 487 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 15 op/s 2021-02-10T04:37:16.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[29319]: cluster 2021-02-10T04:37:16.194231+0000 mon.a (mon.0) 1476 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:37:16.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[29319]: audit 2021-02-10T04:37:16.198374+0000 mon.a (mon.0) 1477 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:37:16.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[29319]: cluster 2021-02-10T04:37:16.198442+0000 mon.a (mon.0) 1478 : cluster [DBG] osdmap e618: 8 total, 8 up, 8 in 2021-02-10T04:37:16.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: cluster 2021-02-10T04:37:16.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: 2021-02-10T04:37:16.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: 2021 2021-02-10T04:37:16.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: - 2021-02-10T04:37:16.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: 02 2021-02-10T04:37:16.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: - 2021-02-10T04:37:16.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: 10 2021-02-10T04:37:16.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: T 2021-02-10T04:37:16.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: 04 2021-02-10T04:37:16.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: : 2021-02-10T04:37:16.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: 37 2021-02-10T04:37:16.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: : 2021-02-10T04:37:16.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: 15 2021-02-10T04:37:16.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: . 2021-02-10T04:37:16.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: 169983+0000 osd.4 (osd.4) 969 : 2021-02-10T04:37:16.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:16.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: cluster 2021-02-10T04:37:16.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: 2021-02-10T04:37:15.169992+0000 2021-02-10T04:37:16.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: osd.4 (osd.4) 970 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:16.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:37:16.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: :37:15.169999+0000 osd.4 (osd.4) 971 : 2021-02-10T04:37:16.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:16.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:37:16.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: 04:37:15.170004+0000 osd.4 (osd. 2021-02-10T04:37:16.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: 4) 972 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:16.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: audit 2021-02-10T04:37:16.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: 2021-02-10T04:37:15.598631+0000 mgr.y 2021-02-10T04:37:16.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: (mgr.14140) 665 : audit [DBG] from='client.25099 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:37:16.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:37:16.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: -10T04:37:15.631164+0000 mon.b ( 2021-02-10T04:37:16.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: mon.2) 38 : audit [INF] from='client.? 172.21.15.85:0/1269175109' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:37:16.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:37:16.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: T04:37:15.631636+0000 mon.a (mon.0 2021-02-10T04:37:16.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: ) 1475 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:37:16.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: cluster 2021-02-10T04:37:16.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: 2021-02-10T04:37:16.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: 2021 2021-02-10T04:37:16.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: - 2021-02-10T04:37:16.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: 02 2021-02-10T04:37:16.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: - 2021-02-10T04:37:16.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: 10 2021-02-10T04:37:16.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: T 2021-02-10T04:37:16.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: 04 2021-02-10T04:37:16.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: :37:16.134394+0000 mgr.y 2021-02-10T04:37:16.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: (mgr.14140) 666 : cluster [DBG] pgmap v1076: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 487 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 15 op/s 2021-02-10T04:37:16.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: cluster 2021-02-10T04:37:16.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: 2021-02-10T04:37:16.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:37:16.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: :37:16.194231+0000 mon.a (mon.0) 1476 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:37:16.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: audit 2021-02-10T04:37:16.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:37:16.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: 37:16.198374+0000 mon.a (mon.0) 1477 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:37:16.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:16.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: 02-10T04:37:16.198442+0000 mon.a (mon. 2021-02-10T04:37:16.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:16 smithi085 conmon[35961]: 0) 1478 : cluster [DBG] osdmap e618: 8 total, 8 up, 8 in 2021-02-10T04:37:17.200 INFO:tasks.rados.rados.0.smithi085.stdout:2380: done (10 left) 2021-02-10T04:37:17.200 INFO:tasks.rados.rados.0.smithi085.stdout:2381: delete oid 2 current snap is 267 2021-02-10T04:37:17.200 INFO:tasks.rados.rados.0.smithi085.stdout:2379: finishing copy_from to smithi08575397-44 2021-02-10T04:37:17.200 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 1081 (ObjNum 816 snap 265 seq_num 816) dirty exists 2021-02-10T04:37:17.200 INFO:tasks.rados.rados.0.smithi085.stdout:2379: finishing copy_from racing read to smithi08575397-44 2021-02-10T04:37:17.210 INFO:tasks.rados.rados.0.smithi085.stdout:2379: done (10 left) 2021-02-10T04:37:17.210 INFO:tasks.rados.rados.0.smithi085.stdout:2381: done (9 left) 2021-02-10T04:37:17.210 INFO:tasks.rados.rados.0.smithi085.stdout:2382: read oid 28 snap -1 2021-02-10T04:37:17.210 INFO:tasks.rados.rados.0.smithi085.stdout:2382: expect deleted 2021-02-10T04:37:17.210 INFO:tasks.rados.rados.0.smithi085.stdout:2383: delete oid 6 current snap is 267 2021-02-10T04:37:17.213 INFO:tasks.rados.rados.0.smithi085.stdout:2382: done (10 left) 2021-02-10T04:37:17.213 INFO:tasks.rados.rados.0.smithi085.stdout:2383: done (9 left) 2021-02-10T04:37:17.213 INFO:tasks.rados.rados.0.smithi085.stdout:2384: read oid 50 snap -1 2021-02-10T04:37:17.213 INFO:tasks.rados.rados.0.smithi085.stdout:2384: expect deleted 2021-02-10T04:37:17.213 INFO:tasks.rados.rados.0.smithi085.stdout:2385: delete oid 38 current snap is 267 2021-02-10T04:37:17.216 INFO:tasks.rados.rados.0.smithi085.stdout:2384: done (10 left) 2021-02-10T04:37:17.216 INFO:tasks.rados.rados.0.smithi085.stdout:2385: done (9 left) 2021-02-10T04:37:17.216 INFO:tasks.rados.rados.0.smithi085.stdout:2386: read oid 13 snap -1 2021-02-10T04:37:17.216 INFO:tasks.rados.rados.0.smithi085.stdout:2386: expect deleted 2021-02-10T04:37:17.216 INFO:tasks.rados.rados.0.smithi085.stdout:2387: snap_create 2021-02-10T04:37:18.406 INFO:tasks.rados.rados.0.smithi085.stdout:2386: done (10 left) 2021-02-10T04:37:18.406 INFO:tasks.rados.rados.0.smithi085.stdout:2387: done (9 left) 2021-02-10T04:37:18.406 INFO:tasks.rados.rados.0.smithi085.stdout:2388: write oid 38 current snap is 268 2021-02-10T04:37:18.406 INFO:tasks.rados.rados.0.smithi085.stdout:2388: seq_num 825 ranges {665775=646549,2110468=499284,3155969=442940} 2021-02-10T04:37:18.408 INFO:tasks.rados.rados.0.smithi085.stdout:2388: writing smithi08575397-38 from 665775 to 1312324 tid 1 2021-02-10T04:37:18.418 INFO:tasks.rados.rados.0.smithi085.stdout:2388: writing smithi08575397-38 from 2110468 to 2609752 tid 2 2021-02-10T04:37:18.425 INFO:tasks.rados.rados.0.smithi085.stdout:2388: writing smithi08575397-38 from 3155969 to 3598909 tid 3 2021-02-10T04:37:18.425 INFO:tasks.rados.rados.0.smithi085.stdout:2389: snap_create 2021-02-10T04:37:18.429 INFO:tasks.rados.rados.0.smithi085.stdout:2388: finishing write tid 1 to smithi08575397-38 2021-02-10T04:37:18.430 INFO:tasks.rados.rados.0.smithi085.stdout:2388: finishing write tid 2 to smithi08575397-38 2021-02-10T04:37:18.430 INFO:tasks.rados.rados.0.smithi085.stdout:2388: finishing write tid 3 to smithi08575397-38 2021-02-10T04:37:18.431 INFO:tasks.rados.rados.0.smithi085.stdout:2388: finishing write tid 4 to smithi08575397-38 2021-02-10T04:37:18.431 INFO:tasks.rados.rados.0.smithi085.stdout:2388: finishing write tid 5 to smithi08575397-38 2021-02-10T04:37:18.432 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 1089 (ObjNum 825 snap 268 seq_num 825) dirty exists 2021-02-10T04:37:18.432 INFO:tasks.rados.rados.0.smithi085.stdout:2388: left oid 38 (ObjNum 825 snap 268 seq_num 825) 2021-02-10T04:37:18.643 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:37:18 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:37:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:18 smithi164 conmon[29288]: cluster 2021-02-10T04:37:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:18 smithi164 conmon[29288]: 2021-02-10T04:37:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:18 smithi164 conmon[29288]: 2021 2021-02-10T04:37:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:18 smithi164 conmon[29288]: - 2021-02-10T04:37:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:18 smithi164 conmon[29288]: 02-10T04:37:17.200165 2021-02-10T04:37:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:18 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1479 : cluster [DBG] osdmap e619: 8 total, 8 up, 8 in 2021-02-10T04:37:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:18 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:37:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:18 smithi164 conmon[29288]: 04:37:18.135057+0000 mgr.y (mgr.14140) 2021-02-10T04:37:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:18 smithi164 conmon[29288]: 667 : cluster [DBG] pgmap v1079: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 489 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 8 op/s 2021-02-10T04:37:18.763 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:18 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:37:18.763 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:18 smithi085 conmon[29319]: 10T04:37:17.200165+0000 mon.a (mon.0) 1479 : cluster [DBG] osdmap e619: 8 total, 8 up, 8 in 2021-02-10T04:37:18.763 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:18 smithi085 conmon[29319]: cluster 2021-02-10T04:37:18.135057+0000 mgr.y (mgr.14140) 667 : cluster [DBG] pgmap v1079: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 489 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 8 op/s 2021-02-10T04:37:18.764 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:18 smithi085 conmon[35961]: cluster 2021-02-10T04:37:17.200165+0000 mon.a ( 2021-02-10T04:37:18.764 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:18 smithi085 conmon[35961]: mon.0) 1479 : cluster [DBG] osdmap e619: 8 total, 8 up, 8 in 2021-02-10T04:37:18.764 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:18 smithi085 conmon[35961]: cluster 2021-02-10T04:37:18.135057+0000 mgr.y (mgr.14140) 667 : cluster [DBG] pgmap v1079: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 489 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 8 op/s 2021-02-10T04:37:19.272 INFO:tasks.rados.rados.0.smithi085.stdout:2388: done (10 left) 2021-02-10T04:37:19.273 INFO:tasks.rados.rados.0.smithi085.stdout:2389: done (9 left) 2021-02-10T04:37:19.273 INFO:tasks.rados.rados.0.smithi085.stdout:2390: snap_remove snap 184 2021-02-10T04:37:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:19 smithi164 conmon[29288]: cluster 2021-02-10T04:37:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:19 smithi164 conmon[29288]: 2021-02-10T04:37:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:19 smithi164 conmon[29288]: 2021 2021-02-10T04:37:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:19 smithi164 conmon[29288]: - 2021-02-10T04:37:19.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:19 smithi164 conmon[29288]: 02 2021-02-10T04:37:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:19 smithi164 conmon[29288]: - 2021-02-10T04:37:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:19 smithi164 conmon[29288]: 10T04:37:18.196659+0000 mon.a 2021-02-10T04:37:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:19 smithi164 conmon[29288]: (mon.0) 1480 : cluster [WRN] Health check update: 10 slow ops, oldest one blocked for 443 sec, osd.2 has slow ops (SLOW_OPS) 2021-02-10T04:37:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:19 smithi164 conmon[29288]: audit 2021-02-10T04:37 2021-02-10T04:37:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:19 smithi164 conmon[29288]: :18.198046+0000 mgr.y (mgr.14140) 668 : 2021-02-10T04:37:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:19 smithi164 conmon[29288]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:37:19.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:19 smithi164 conmon[29288]: cluster 2021-02-10T04:37:18 2021-02-10T04:37:19.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:19 smithi164 conmon[29288]: .264134+0000 mon.a (mon.0) 1481 : cluster 2021-02-10T04:37:19.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:19 smithi164 conmon[29288]: [DBG] osdmap e620: 8 total, 8 up, 8 in 2021-02-10T04:37:19.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:19 smithi085 conmon[35961]: cluster 2021-02-10T04:37:18.196659 2021-02-10T04:37:19.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:19 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1480 : cluster [WRN] Health check update: 10 slow ops, oldest one blocked for 443 sec, osd.2 has slow ops (SLOW_OPS) 2021-02-10T04:37:19.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:19 smithi085 conmon[35961]: audit 2021-02-10T04:37:19.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:19 smithi085 conmon[35961]: 2021-02-10T04:37:18.198046+0000 mgr.y (mgr.14140) 2021-02-10T04:37:19.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:19 smithi085 conmon[35961]: 668 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:37:19.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:19 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:19.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:19 smithi085 conmon[35961]: -02-10T04:37:18.264134+0000 2021-02-10T04:37:19.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:19 smithi085 conmon[35961]: mon.a (mon.0) 1481 : cluster [DBG] 2021-02-10T04:37:19.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:19 smithi085 conmon[35961]: osdmap e620: 8 total, 8 up, 8 in 2021-02-10T04:37:19.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:19 smithi085 conmon[29319]: cluster 2021-02-10T04:37:19.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:19 smithi085 conmon[29319]: 2021-02-10T04:37:19.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:19 smithi085 conmon[29319]: 2021 2021-02-10T04:37:19.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:19 smithi085 conmon[29319]: - 2021-02-10T04:37:19.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:19 smithi085 conmon[29319]: 02-10T04:37:18.196659 2021-02-10T04:37:19.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:19 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1480 : cluster [WRN] Health check update: 10 slow ops, oldest one blocked for 443 sec, osd.2 has slow ops (SLOW_OPS) 2021-02-10T04:37:19.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:19 smithi085 conmon[29319]: audit 2021-02-10T04:37:19.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:19 smithi085 conmon[29319]: 2021-02-10T04:37:18 2021-02-10T04:37:19.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:19 smithi085 conmon[29319]: .198046+0000 mgr.y (mgr.14140) 2021-02-10T04:37:19.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:19 smithi085 conmon[29319]: 668 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:37:19.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:19 smithi085 conmon[29319]: cluster 2021-02-10T04:37:19.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:19 smithi085 conmon[29319]: 2021-02-10T04:37:18 2021-02-10T04:37:19.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:19 smithi085 conmon[29319]: .264134+0000 mon.a (mon 2021-02-10T04:37:19.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:19 smithi085 conmon[29319]: .0) 1481 : cluster [DBG] osdmap e620: 8 total, 8 up, 8 in 2021-02-10T04:37:20.276 INFO:tasks.rados.rados.0.smithi085.stdout:2390: done (9 left) 2021-02-10T04:37:20.276 INFO:tasks.rados.rados.0.smithi085.stdout:2391: delete oid 36 current snap is 269 2021-02-10T04:37:20.470 INFO:tasks.rados.rados.0.smithi085.stdout:2391: done (9 left) 2021-02-10T04:37:20.470 INFO:tasks.rados.rados.0.smithi085.stdout:2392: read oid 5 snap 214 2021-02-10T04:37:20.470 INFO:tasks.rados.rados.0.smithi085.stdout:2392: expect (ObjNum 686 snap 214 seq_num 686) 2021-02-10T04:37:20.470 INFO:tasks.rados.rados.0.smithi085.stdout:2393: read oid 7 snap 214 2021-02-10T04:37:20.470 INFO:tasks.rados.rados.0.smithi085.stdout:2393: expect (ObjNum 204 snap 44 seq_num 204) 2021-02-10T04:37:20.470 INFO:tasks.rados.rados.0.smithi085.stdout:2394: read oid 6 snap -1 2021-02-10T04:37:20.470 INFO:tasks.rados.rados.0.smithi085.stdout:2394: expect deleted 2021-02-10T04:37:20.470 INFO:tasks.rados.rados.0.smithi085.stdout:2395: read oid 44 snap -1 2021-02-10T04:37:20.470 INFO:tasks.rados.rados.0.smithi085.stdout:2395: expect (ObjNum 816 snap 265 seq_num 816) 2021-02-10T04:37:20.470 INFO:tasks.rados.rados.0.smithi085.stdout:2396: copy_from oid 36 from oid 21 current snap is 269 2021-02-10T04:37:20.470 INFO:tasks.rados.rados.0.smithi085.stdout:2397: snap_remove snap 258 2021-02-10T04:37:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:20 smithi164 conmon[29288]: cluster 2021-02-10T04:37:19.272320+0000 mon.a (mon 2021-02-10T04:37:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:20 smithi164 conmon[29288]: .0) 1482 : cluster [DBG] osdmap e621: 8 total, 8 up, 8 in 2021-02-10T04:37:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:20 smithi164 conmon[29288]: cluster 2021-02-10T04:37:20.135680+0000 mgr.y (mgr.14140) 669 : cluster [DBG] pgmap v1082: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 496 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 2.1 MiB/s wr, 3 op/s 2021-02-10T04:37:20.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:20 smithi085 conmon[35961]: cluster 2021-02-10T04:37:20.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:20 smithi085 conmon[35961]: 2021-02-10T04:37:20.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:20 smithi085 conmon[35961]: 2021 2021-02-10T04:37:20.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:20 smithi085 conmon[35961]: - 2021-02-10T04:37:20.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:20 smithi085 conmon[35961]: 02-10T04:37:19.272320 2021-02-10T04:37:20.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:20 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1482 : cluster [DBG] osdmap e621: 8 total, 8 up, 8 in 2021-02-10T04:37:20.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:20 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:37:20.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:20 smithi085 conmon[35961]: -10T04:37:20.135680+0000 mgr.y (mgr.14140 2021-02-10T04:37:20.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:20 smithi085 conmon[35961]: ) 669 : cluster [DBG] pgmap v1082: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 496 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 2.1 MiB/s wr, 3 op/s 2021-02-10T04:37:20.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:20 smithi085 conmon[29319]: cluster 2021-02-10T04:37:19.272320+0000 mon.a (mon 2021-02-10T04:37:20.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:20 smithi085 conmon[29319]: .0) 1482 : cluster [DBG] osdmap e621: 8 total, 8 up, 8 in 2021-02-10T04:37:20.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:20 smithi085 conmon[29319]: cluster 2021-02-10T04:37:20.135680+0000 mgr.y (mgr.14140) 669 : cluster [DBG] 2021-02-10T04:37:20.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:20 smithi085 conmon[29319]: pgmap v1082: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 496 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 2.1 MiB/s wr, 3 op/s 2021-02-10T04:37:20.754 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:37:20.840 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:37:21.078 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.7 dump_historic_ops 2021-02-10T04:37:21.285 INFO:tasks.rados.rados.0.smithi085.stdout:2397: done (14 left) 2021-02-10T04:37:21.286 INFO:tasks.rados.rados.0.smithi085.stdout:2398: write (excl) oid 3 current snap is 269 2021-02-10T04:37:21.286 INFO:tasks.rados.rados.0.smithi085.stdout:2393: expect (ObjNum 204 snap 44 seq_num 204) 2021-02-10T04:37:21.311 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:37:21.315 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:37:21 smithi164 conmon[44544]: debug 2021-02-10T04:37:21.113+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:37:21.316 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:21 smithi164 conmon[29288]: cluster 2021-02-10T04:37:20.275835+0000 mon.a (mon.0) 1483 : cluster [DBG] osdmap e622: 8 total, 8 up, 8 in 2021-02-10T04:37:21.417 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:37:21.535 INFO:tasks.rados.rados.0.smithi085.stdout:2395: expect (ObjNum 816 snap 265 seq_num 816) 2021-02-10T04:37:21.570 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:37:21.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:21 smithi085 conmon[29319]: cluster 2021-02-10T04:37:20.275835+0000 mon.a (mon.0) 1483 : cluster 2021-02-10T04:37:21.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:21 smithi085 conmon[29319]: [DBG] osdmap e622: 8 total, 8 up, 8 in 2021-02-10T04:37:21.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:21 smithi085 conmon[35961]: cluster 2021-02-10T04:37: 2021-02-10T04:37:21.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:21 smithi085 conmon[35961]: 20.275835+0000 mon.a (mon.0) 1483 : cluster [DBG] osdmap e622: 8 total, 8 up, 8 in 2021-02-10T04:37:21.761 INFO:tasks.rados.rados.0.smithi085.stdout:2392: expect (ObjNum 686 snap 214 seq_num 686) 2021-02-10T04:37:21.778 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:37:22.025 INFO:tasks.rados.rados.0.smithi085.stdout:2396: finishing copy_from to smithi08575397-36 2021-02-10T04:37:22.025 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 1261 (ObjNum 812 snap 265 seq_num 812) dirty exists 2021-02-10T04:37:22.025 INFO:tasks.rados.rados.0.smithi085.stdout:2396: finishing copy_from racing read to smithi08575397-36 2021-02-10T04:37:22.026 INFO:tasks.rados.rados.0.smithi085.stdout:2398: seq_num 827 ranges {699389=797047,2111079=502953,3094462=581837} 2021-02-10T04:37:22.034 INFO:tasks.rados.rados.0.smithi085.stdout:2398: writing smithi08575397-3 from 699389 to 1496436 tid 1 2021-02-10T04:37:22.041 INFO:tasks.rados.rados.0.smithi085.stdout:2398: writing smithi08575397-3 from 2111079 to 2614032 tid 2 2021-02-10T04:37:22.049 INFO:tasks.rados.rados.0.smithi085.stdout:2398: writing smithi08575397-3 from 3094462 to 3676299 tid 3 2021-02-10T04:37:22.049 INFO:tasks.rados.rados.0.smithi085.stdout:2398: finishing write tid 1 to smithi08575397-3 2021-02-10T04:37:22.049 INFO:tasks.rados.rados.0.smithi085.stdout:2398: finishing write tid 2 to smithi08575397-3 2021-02-10T04:37:22.049 INFO:tasks.rados.rados.0.smithi085.stdout:2392: done (14 left) 2021-02-10T04:37:22.049 INFO:tasks.rados.rados.0.smithi085.stdout:2393: done (13 left) 2021-02-10T04:37:22.049 INFO:tasks.rados.rados.0.smithi085.stdout:2394: done (12 left) 2021-02-10T04:37:22.049 INFO:tasks.rados.rados.0.smithi085.stdout:2395: done (11 left) 2021-02-10T04:37:22.050 INFO:tasks.rados.rados.0.smithi085.stdout:2396: done (10 left) 2021-02-10T04:37:22.050 INFO:tasks.rados.rados.0.smithi085.stdout:2399: read oid 20 snap -1 2021-02-10T04:37:22.050 INFO:tasks.rados.rados.0.smithi085.stdout:2399: expect (ObjNum 759 snap 243 seq_num 759) 2021-02-10T04:37:22.050 INFO:tasks.rados.rados.0.smithi085.stdout:2400: read oid 8 snap -1 2021-02-10T04:37:22.050 INFO:tasks.rados.rados.0.smithi085.stdout:2400: expect (ObjNum 797 snap 256 seq_num 797) 2021-02-10T04:37:22.050 INFO:tasks.rados.rados.0.smithi085.stdout:2401: read oid 46 snap 263 2021-02-10T04:37:22.050 INFO:tasks.rados.rados.0.smithi085.stdout:2401: expect (ObjNum 788 snap 255 seq_num 788) 2021-02-10T04:37:22.050 INFO:tasks.rados.rados.0.smithi085.stdout:2402: rollback oid 37 current snap is 269 2021-02-10T04:37:22.050 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 37 to 252 2021-02-10T04:37:22.050 INFO:tasks.rados.rados.0.smithi085.stdout:2403: write (excl) oid 14 current snap is 269 2021-02-10T04:37:22.050 INFO:tasks.rados.rados.0.smithi085.stdout:2403: seq_num 828 ranges {716196=472296,1769561=646611,3121885=793109} 2021-02-10T04:37:22.058 INFO:tasks.rados.rados.0.smithi085.stdout:2403: writing smithi08575397-14 from 716196 to 1188492 tid 1 2021-02-10T04:37:22.066 INFO:tasks.rados.rados.0.smithi085.stdout:2403: writing smithi08575397-14 from 1769561 to 2416172 tid 2 2021-02-10T04:37:22.076 INFO:tasks.rados.rados.0.smithi085.stdout:2403: writing smithi08575397-14 from 3121885 to 3914994 tid 3 2021-02-10T04:37:22.076 INFO:tasks.rados.rados.0.smithi085.stdout:2404: write oid 13 current snap is 269 2021-02-10T04:37:22.076 INFO:tasks.rados.rados.0.smithi085.stdout:2404: seq_num 829 ranges {733003=663898,1944396=506622,2865661=604381} 2021-02-10T04:37:22.084 INFO:tasks.rados.rados.0.smithi085.stdout:2404: writing smithi08575397-13 from 733003 to 1396901 tid 1 2021-02-10T04:37:22.091 INFO:tasks.rados.rados.0.smithi085.stdout:2404: writing smithi08575397-13 from 1944396 to 2451018 tid 2 2021-02-10T04:37:22.097 INFO:tasks.rados.rados.0.smithi085.stdout:2404: writing smithi08575397-13 from 2865661 to 3470042 tid 3 2021-02-10T04:37:22.098 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:37:22.098 INFO:tasks.rados.rados.0.smithi085.stdout:2399: expect (ObjNum 759 snap 243 seq_num 759) 2021-02-10T04:37:22.191 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:37:21 smithi085 conmon[41540]: debug 2021-02-10T04:37:22.191 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:37:21 smithi085 conmon[41540]: 2021-02-10T04:37:21.869+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:37:22.191 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:37:21 smithi085 conmon[41540]: 2021-02-10T04:37:22.240 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:37:22.302 INFO:tasks.rados.rados.0.smithi085.stdout:2401: expect (ObjNum 788 snap 255 seq_num 788) 2021-02-10T04:37:22.456 INFO:tasks.rados.rados.0.smithi085.stdout:2400: expect (ObjNum 797 snap 256 seq_num 797) 2021-02-10T04:37:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:22 smithi164 conmon[29288]: cluster 2021-02-10T04:37:21.282130+0000 mon.a (mon 2021-02-10T04:37:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:22 smithi164 conmon[29288]: .0) 1484 : cluster [DBG] osdmap e623: 8 total, 8 up, 8 in 2021-02-10T04:37:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:22 smithi164 conmon[29288]: cluster 2021-02-10T04:37:21.548418+0000 osd.6 (osd.6) 67 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:37:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:22 smithi164 conmon[29288]: cluster 2021-02-10T04:37 2021-02-10T04:37:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:22 smithi164 conmon[29288]: :21.548798+0000 osd.6 (osd.6) 68 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:37:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:22 smithi164 conmon[29288]: cluster 2021-02-10T04:37: 2021-02-10T04:37:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:22 smithi164 conmon[29288]: 22.136197+0000 mgr.y (mgr.14140) 670 : cluster 2021-02-10T04:37:22.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:22 smithi164 conmon[29288]: [DBG] pgmap v1085: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 498 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.1 MiB/s wr, 5 op/s 2021-02-10T04:37:22.682 INFO:tasks.rados.rados.0.smithi085.stdout:2402: finishing rollback tid 1 to smithi08575397-37 2021-02-10T04:37:22.683 INFO:tasks.rados.rados.0.smithi085.stdout:2402: finishing rollback tid 2 to smithi08575397-37 2021-02-10T04:37:22.683 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 1280 (ObjNum 761 snap 244 seq_num 761) dirty exists 2021-02-10T04:37:22.683 INFO:tasks.rados.rados.0.smithi085.stdout:2398: finishing write tid 3 to smithi08575397-3 2021-02-10T04:37:22.683 INFO:tasks.rados.rados.0.smithi085.stdout:2398: finishing write tid 4 to smithi08575397-3 2021-02-10T04:37:22.683 INFO:tasks.rados.rados.0.smithi085.stdout:2398: finishing write tid 5 to smithi08575397-3 2021-02-10T04:37:22.683 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 1139 (ObjNum 827 snap 269 seq_num 827) dirty exists 2021-02-10T04:37:22.683 INFO:tasks.rados.rados.0.smithi085.stdout:2398: left oid 3 (ObjNum 827 snap 269 seq_num 827) 2021-02-10T04:37:22.683 INFO:tasks.rados.rados.0.smithi085.stdout:2403: finishing write tid 1 to smithi08575397-14 2021-02-10T04:37:22.684 INFO:tasks.rados.rados.0.smithi085.stdout:2398: done (15 left) 2021-02-10T04:37:22.684 INFO:tasks.rados.rados.0.smithi085.stdout:2399: done (14 left) 2021-02-10T04:37:22.684 INFO:tasks.rados.rados.0.smithi085.stdout:2400: done (13 left) 2021-02-10T04:37:22.684 INFO:tasks.rados.rados.0.smithi085.stdout:2401: done (12 left) 2021-02-10T04:37:22.684 INFO:tasks.rados.rados.0.smithi085.stdout:2402: done (11 left) 2021-02-10T04:37:22.684 INFO:tasks.rados.rados.0.smithi085.stdout:2405: delete oid 50 current snap is 269 2021-02-10T04:37:22.684 INFO:tasks.rados.rados.0.smithi085.stdout:2403: finishing write tid 2 to smithi08575397-14 2021-02-10T04:37:22.684 INFO:tasks.rados.rados.0.smithi085.stdout:2403: finishing write tid 3 to smithi08575397-14 2021-02-10T04:37:22.684 INFO:tasks.rados.rados.0.smithi085.stdout:2403: finishing write tid 4 to smithi08575397-14 2021-02-10T04:37:22.684 INFO:tasks.rados.rados.0.smithi085.stdout:2403: finishing write tid 5 to smithi08575397-14 2021-02-10T04:37:22.684 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 874 (ObjNum 828 snap 269 seq_num 828) dirty exists 2021-02-10T04:37:22.684 INFO:tasks.rados.rados.0.smithi085.stdout:2403: left oid 14 (ObjNum 828 snap 269 seq_num 828) 2021-02-10T04:37:22.685 INFO:tasks.rados.rados.0.smithi085.stdout:2404: finishing write tid 1 to smithi08575397-13 2021-02-10T04:37:22.685 INFO:tasks.rados.rados.0.smithi085.stdout:2404: finishing write tid 2 to smithi08575397-13 2021-02-10T04:37:22.685 INFO:tasks.rados.rados.0.smithi085.stdout:2404: finishing write tid 3 to smithi08575397-13 2021-02-10T04:37:22.685 INFO:tasks.rados.rados.0.smithi085.stdout:2404: finishing write tid 4 to smithi08575397-13 2021-02-10T04:37:22.685 INFO:tasks.rados.rados.0.smithi085.stdout:2404: finishing write tid 5 to smithi08575397-13 2021-02-10T04:37:22.685 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 769 (ObjNum 829 snap 269 seq_num 829) dirty exists 2021-02-10T04:37:22.685 INFO:tasks.rados.rados.0.smithi085.stdout:2404: left oid 13 (ObjNum 829 snap 269 seq_num 829) 2021-02-10T04:37:22.686 INFO:tasks.rados.rados.0.smithi085.stdout:2403: done (11 left) 2021-02-10T04:37:22.686 INFO:tasks.rados.rados.0.smithi085.stdout:2404: done (10 left) 2021-02-10T04:37:22.686 INFO:tasks.rados.rados.0.smithi085.stdout:2405: done (9 left) 2021-02-10T04:37:22.686 INFO:tasks.rados.rados.0.smithi085.stdout:2406: read oid 8 snap -1 2021-02-10T04:37:22.686 INFO:tasks.rados.rados.0.smithi085.stdout:2406: expect (ObjNum 797 snap 256 seq_num 797) 2021-02-10T04:37:22.686 INFO:tasks.rados.rados.0.smithi085.stdout:2407: snap_remove snap 225 2021-02-10T04:37:22.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: cluster 2021-02-10T04:37:22.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: 2021-02-10T04:37:22.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: 2021 2021-02-10T04:37:22.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: - 2021-02-10T04:37:22.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: 02 2021-02-10T04:37:22.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: - 2021-02-10T04:37:22.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: 10 2021-02-10T04:37:22.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: T 2021-02-10T04:37:22.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: 04 2021-02-10T04:37:22.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: : 2021-02-10T04:37:22.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: 37 2021-02-10T04:37:22.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: : 2021-02-10T04:37:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: 21 2021-02-10T04:37:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: . 2021-02-10T04:37:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: 282130 2021-02-10T04:37:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: +0000 2021-02-10T04:37:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: 2021-02-10T04:37:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: mon.a 2021-02-10T04:37:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: ( 2021-02-10T04:37:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: mon 2021-02-10T04:37:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: . 2021-02-10T04:37:22.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: 0 2021-02-10T04:37:22.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: ) 2021-02-10T04:37:22.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: 1484 2021-02-10T04:37:22.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: : 2021-02-10T04:37:22.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: cluster 2021-02-10T04:37:22.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: 2021-02-10T04:37:22.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: [DBG] 2021-02-10T04:37:22.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: 2021-02-10T04:37:22.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: osdmap e623: 8 total, 8 up, 8 in 2021-02-10T04:37:22.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: 2021-02-10T04:37:22.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: cluster 2021-02-10T04:37:22.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: 2021-02-10T04:37:22.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: 2021 2021-02-10T04:37:22.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: - 2021-02-10T04:37:22.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: 02 2021-02-10T04:37:22.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: - 2021-02-10T04:37:22.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: 10 2021-02-10T04:37:22.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: T 2021-02-10T04:37:22.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: 04 2021-02-10T04:37:22.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: : 2021-02-10T04:37:22.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: 37 2021-02-10T04:37:22.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: :21.548418+0000 osd.6 ( 2021-02-10T04:37:22.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: osd.6) 67 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:37:22.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: cluster 2021-02-10T04:37:22.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: 2021-02-10T04:37:21.548798+0000 2021-02-10T04:37:22.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: osd.6 (osd.6) 68 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:37:22.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: 2021-02-10T04:37:22.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: cluster 2021-02-10T04:37 2021-02-10T04:37:22.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: :22.136197+0000 mgr.y (mgr.14140) 670 2021-02-10T04:37:22.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[35961]: : cluster [DBG] pgmap v1085: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 498 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.1 MiB/s wr, 5 op/s 2021-02-10T04:37:22.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[29319]: cluster 2021-02-10T04:37:22.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[29319]: 2021-02-10T04:37:22.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[29319]: 2021 2021-02-10T04:37:22.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[29319]: - 2021-02-10T04:37:22.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[29319]: 02 2021-02-10T04:37:22.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[29319]: - 2021-02-10T04:37:22.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[29319]: 10 2021-02-10T04:37:22.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[29319]: T 2021-02-10T04:37:22.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[29319]: 04 2021-02-10T04:37:22.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[29319]: : 2021-02-10T04:37:22.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[29319]: 37 2021-02-10T04:37:22.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[29319]: : 2021-02-10T04:37:22.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[29319]: 21.282130+0000 mon.a (mon.0 2021-02-10T04:37:22.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[29319]: ) 1484 : cluster [DBG] osdmap e623: 8 total, 8 up, 8 in 2021-02-10T04:37:22.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[29319]: cluster 2021-02-10T04:37:22.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:37:22.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[29319]: 04:37:21.548418+0000 osd.6 (osd. 2021-02-10T04:37:22.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[29319]: 6) 67 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:37:22.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:22.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[29319]: 02-10T04:37:21.548798+0000 osd.6 2021-02-10T04:37:22.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[29319]: (osd.6) 68 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:37:22.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:37:22.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[29319]: -10T04:37:22.136197+0000 mgr.y 2021-02-10T04:37:22.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:22 smithi085 conmon[29319]: (mgr.14140) 670 : cluster [DBG] pgmap v1085: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 498 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.1 MiB/s wr, 5 op/s 2021-02-10T04:37:23.289 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:23 smithi164 conmon[29288]: cluster 2021-02-10T04:37:23.289 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:23 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:37:23.289 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:23 smithi164 conmon[29288]: 37:22.278588+0000 mon.a (mon.0) 1485 : cluster [INF] Health check cleared: SLOW_OPS (was: 10 slow ops, oldest one blocked for 443 sec, osd.2 has slow ops) 2021-02-10T04:37:23.289 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:23 smithi164 conmon[29288]: cluster 2021-02-10T04:37:22. 2021-02-10T04:37:23.290 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:23 smithi164 conmon[29288]: 291825+0000 mon.a (mon.0) 1486 : cluster [DBG] osdmap e624: 8 total, 8 up, 8 in 2021-02-10T04:37:23.290 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:23 smithi164 conmon[29288]: 2021-02-10T04:37:23.305 INFO:tasks.rados.rados.0.smithi085.stdout:2406: expect (ObjNum 797 snap 256 seq_num 797) 2021-02-10T04:37:23.532 INFO:tasks.rados.rados.0.smithi085.stdout:2406: done (10 left) 2021-02-10T04:37:23.532 INFO:tasks.rados.rados.0.smithi085.stdout:2407: done (9 left) 2021-02-10T04:37:23.532 INFO:tasks.rados.rados.0.smithi085.stdout:2408: delete oid 42 current snap is 269 2021-02-10T04:37:23.538 INFO:tasks.rados.rados.0.smithi085.stdout:2408: done (9 left) 2021-02-10T04:37:23.538 INFO:tasks.rados.rados.0.smithi085.stdout:2409: snap_create 2021-02-10T04:37:23.606 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:37:23.661 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:37:23.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:23 smithi085 conmon[29319]: cluster 2021-02-10T04:37:23.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:23 smithi085 conmon[29319]: 2021-02-10T04:37:22 2021-02-10T04:37:23.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:23 smithi085 conmon[29319]: .278588+0000 mon.a (mon.0) 1485 : cluster [INF] Health check cleared: SLOW_OPS (was: 10 slow ops, oldest one blocked for 443 sec, osd.2 has slow ops) 2021-02-10T04:37:23.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:23 smithi085 conmon[29319]: cluster 2021-02-10T04:37:22.291825+0000 2021-02-10T04:37:23.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:23 smithi085 conmon[29319]: mon.a (mon.0) 1486 : cluster [DBG] osdmap e624: 8 total, 8 up, 8 in 2021-02-10T04:37:23.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:23 smithi085 conmon[35961]: cluster 2021-02-10T04:37:22. 2021-02-10T04:37:23.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:23 smithi085 conmon[35961]: 278588+0000 mon.a (mon.0) 1485 : cluster [INF] Health check cleared: SLOW_OPS (was: 10 slow ops, oldest one blocked for 443 sec, osd.2 has slow ops) 2021-02-10T04:37:23.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:23 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:37:23.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:23 smithi085 conmon[35961]: 37:22.291825+0000 mon.a (mon.0) 1486 : cluster [DBG] 2021-02-10T04:37:23.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:23 smithi085 conmon[35961]: osdmap e624: 8 total, 8 up, 8 in 2021-02-10T04:37:24.143 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:37:23 smithi164 conmon[44544]: debug 2021-02-10T04:37:23.807+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:37:24.307 INFO:tasks.rados.rados.0.smithi085.stdout:2409: done (9 left) 2021-02-10T04:37:24.308 INFO:tasks.rados.rados.0.smithi085.stdout:2410: write oid 7 current snap is 270 2021-02-10T04:37:24.308 INFO:tasks.rados.rados.0.smithi085.stdout:2410: seq_num 832 ranges {783424=489645,2003489=653949,3231577=554550} 2021-02-10T04:37:24.315 INFO:tasks.rados.rados.0.smithi085.stdout:2410: writing smithi08575397-7 from 783424 to 1273069 tid 1 2021-02-10T04:37:24.323 INFO:tasks.rados.rados.0.smithi085.stdout:2410: writing smithi08575397-7 from 2003489 to 2657438 tid 2 2021-02-10T04:37:24.330 INFO:tasks.rados.rados.0.smithi085.stdout:2410: writing smithi08575397-7 from 3231577 to 3786127 tid 3 2021-02-10T04:37:24.330 INFO:tasks.rados.rados.0.smithi085.stdout:2411: rollback oid 31 current snap is 270 2021-02-10T04:37:24.330 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 31 to 263 2021-02-10T04:37:24.331 INFO:tasks.rados.rados.0.smithi085.stdout:2410: finishing write tid 1 to smithi08575397-7 2021-02-10T04:37:24.331 INFO:tasks.rados.rados.0.smithi085.stdout:2410: finishing write tid 2 to smithi08575397-7 2021-02-10T04:37:24.331 INFO:tasks.rados.rados.0.smithi085.stdout:2412: read oid 11 snap -1 2021-02-10T04:37:24.331 INFO:tasks.rados.rados.0.smithi085.stdout:2412: expect (ObjNum 817 snap 265 seq_num 817) 2021-02-10T04:37:24.331 INFO:tasks.rados.rados.0.smithi085.stdout:2413: snap_remove snap 214 2021-02-10T04:37:24.552 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:37:24.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:24 smithi164 conmon[29288]: cluster 2021-02-10T04:37:23.299639 2021-02-10T04:37:24.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:24 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1487 : cluster [DBG] osdmap e625: 8 total, 8 up, 8 in 2021-02-10T04:37:24.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:24 smithi164 conmon[29288]: cluster 2021-02-10T04:37:24.136811+0000 mgr.y (mgr.14140) 671 : cluster [DBG] 2021-02-10T04:37:24.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:24 smithi164 conmon[29288]: pgmap v1088: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 498 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 1.3 MiB/s wr, 13 op/s 2021-02-10T04:37:24.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:24 smithi085 conmon[29319]: cluster 2021-02-10T04:37:23.299639+0000 2021-02-10T04:37:24.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:24 smithi085 conmon[29319]: mon.a (mon.0) 1487 : cluster [DBG] osdmap e625: 8 total, 8 up, 8 in 2021-02-10T04:37:24.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:24 smithi085 conmon[29319]: cluster 2021-02-10T04:37:24.136811+0000 mgr.y (mgr.14140) 671 : cluster [DBG] pgmap v1088: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 498 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 1.3 MiB/s wr, 13 op/s 2021-02-10T04:37:24.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:24 smithi085 conmon[35961]: cluster 2021-02-10T04:37:23.299639+0000 2021-02-10T04:37:24.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:24 smithi085 conmon[35961]: mon.a (mon.0) 1487 : cluster [DBG] osdmap e625: 8 total, 8 up, 8 in 2021-02-10T04:37:24.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:24 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:24.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:24 smithi085 conmon[35961]: -02-10T04:37:24.136811+0000 mgr.y (mgr.14140) 671 : cluster 2021-02-10T04:37:24.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:24 smithi085 conmon[35961]: [DBG] pgmap v1088: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 498 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 1.3 MiB/s wr, 13 op/s 2021-02-10T04:37:25.318 INFO:tasks.rados.rados.0.smithi085.stdout:2413: done (12 left) 2021-02-10T04:37:25.318 INFO:tasks.rados.rados.0.smithi085.stdout:2414: write oid 42 current snap is 270 2021-02-10T04:37:25.318 INFO:tasks.rados.rados.0.smithi085.stdout:2410: finishing write tid 3 to smithi08575397-7 2021-02-10T04:37:25.319 INFO:tasks.rados.rados.0.smithi085.stdout:2410: finishing write tid 4 to smithi08575397-7 2021-02-10T04:37:25.319 INFO:tasks.rados.rados.0.smithi085.stdout:2410: finishing write tid 5 to smithi08575397-7 2021-02-10T04:37:25.319 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 1143 (ObjNum 832 snap 270 seq_num 832) dirty exists 2021-02-10T04:37:25.319 INFO:tasks.rados.rados.0.smithi085.stdout:2410: left oid 7 (ObjNum 832 snap 270 seq_num 832) 2021-02-10T04:37:25.319 INFO:tasks.rados.rados.0.smithi085.stdout:2412: expect (ObjNum 817 snap 265 seq_num 817) 2021-02-10T04:37:25.487 INFO:tasks.rados.rados.0.smithi085.stdout:2411: finishing rollback tid 0 to smithi08575397-31 2021-02-10T04:37:25.487 INFO:tasks.rados.rados.0.smithi085.stdout:2411: finishing rollback tid 1 to smithi08575397-31 2021-02-10T04:37:25.487 INFO:tasks.rados.rados.0.smithi085.stdout:2411: finishing rollback tid 2 to smithi08575397-31 2021-02-10T04:37:25.487 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 807 (ObjNum 798 snap 256 seq_num 798) dirty exists 2021-02-10T04:37:25.487 INFO:tasks.rados.rados.0.smithi085.stdout:2414: seq_num 833 ranges {400231=564894,1661971=338260} 2021-02-10T04:37:25.493 INFO:tasks.rados.rados.0.smithi085.stdout:2414: writing smithi08575397-42 from 400231 to 965125 tid 1 2021-02-10T04:37:25.499 INFO:tasks.rados.rados.0.smithi085.stdout:2414: writing smithi08575397-42 from 1661971 to 2000231 tid 2 2021-02-10T04:37:25.499 INFO:tasks.rados.rados.0.smithi085.stdout:2410: done (12 left) 2021-02-10T04:37:25.500 INFO:tasks.rados.rados.0.smithi085.stdout:2411: done (11 left) 2021-02-10T04:37:25.500 INFO:tasks.rados.rados.0.smithi085.stdout:2412: done (10 left) 2021-02-10T04:37:25.500 INFO:tasks.rados.rados.0.smithi085.stdout:2415: copy_from oid 28 from oid 21 current snap is 270 2021-02-10T04:37:25.500 INFO:tasks.rados.rados.0.smithi085.stdout:2414: finishing write tid 1 to smithi08575397-42 2021-02-10T04:37:25.500 INFO:tasks.rados.rados.0.smithi085.stdout:2416: read oid 10 snap -1 2021-02-10T04:37:25.500 INFO:tasks.rados.rados.0.smithi085.stdout:2416: expect (ObjNum 805 snap 262 seq_num 805) 2021-02-10T04:37:25.500 INFO:tasks.rados.rados.0.smithi085.stdout:2417: rollback oid 44 current snap is 270 2021-02-10T04:37:25.500 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 44 to 252 2021-02-10T04:37:25.500 INFO:tasks.rados.rados.0.smithi085.stdout:2418: write (excl) oid 38 current snap is 270 2021-02-10T04:37:25.500 INFO:tasks.rados.rados.0.smithi085.stdout:2418: seq_num 834 ranges {417038=640143,1720453=296585} 2021-02-10T04:37:25.512 INFO:tasks.rados.rados.0.smithi085.stdout:2418: writing smithi08575397-38 from 417038 to 1057181 tid 1 2021-02-10T04:37:25.514 INFO:tasks.rados.rados.0.smithi085.stdout:2418: writing smithi08575397-38 from 1720453 to 2017038 tid 2 2021-02-10T04:37:25.514 INFO:tasks.rados.rados.0.smithi085.stdout:2419: read oid 16 snap -1 2021-02-10T04:37:25.514 INFO:tasks.rados.rados.0.smithi085.stdout:2419: expect deleted 2021-02-10T04:37:25.515 INFO:tasks.rados.rados.0.smithi085.stdout:2420: read oid 37 snap 190 2021-02-10T04:37:25.515 INFO:tasks.rados.rados.0.smithi085.stdout:2420: expect deleted 2021-02-10T04:37:25.515 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:37:25.515 INFO:tasks.rados.rados.0.smithi085.stdout:2414: finishing write tid 2 to smithi08575397-42 2021-02-10T04:37:25.515 INFO:tasks.rados.rados.0.smithi085.stdout:2414: finishing write tid 3 to smithi08575397-42 2021-02-10T04:37:25.515 INFO:tasks.rados.rados.0.smithi085.stdout:2414: finishing write tid 4 to smithi08575397-42 2021-02-10T04:37:25.515 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 878 (ObjNum 833 snap 270 seq_num 833) dirty exists 2021-02-10T04:37:25.515 INFO:tasks.rados.rados.0.smithi085.stdout:2414: left oid 42 (ObjNum 833 snap 270 seq_num 833) 2021-02-10T04:37:25.515 INFO:tasks.rados.rados.0.smithi085.stdout:2416: expect (ObjNum 805 snap 262 seq_num 805) 2021-02-10T04:37:25.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:25 smithi164 conmon[29288]: cluster 2021-02-10T04:37:24.310669 2021-02-10T04:37:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:25 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1488 : cluster [DBG] osdmap e626: 8 total, 8 up, 8 in 2021-02-10T04:37:25.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:25 smithi085 conmon[29319]: cluster 2021-02-10T04:37:24.310669 2021-02-10T04:37:25.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:25 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1488 : cluster [DBG] osdmap e626: 8 total, 8 up, 8 in 2021-02-10T04:37:25.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:25 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:37:25.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:25 smithi085 conmon[35961]: :37:24 2021-02-10T04:37:25.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:25 smithi085 conmon[35961]: .310669+0000 mon.a (mon.0) 1488 : cluster [DBG] osdmap e626: 8 total, 8 up, 8 in 2021-02-10T04:37:25.750 INFO:tasks.rados.rados.0.smithi085.stdout:2417: finishing rollback tid 0 to smithi08575397-44 2021-02-10T04:37:25.750 INFO:tasks.rados.rados.0.smithi085.stdout:2417: finishing rollback tid 1 to smithi08575397-44 2021-02-10T04:37:25.750 INFO:tasks.rados.rados.0.smithi085.stdout:2417: finishing rollback tid 2 to smithi08575397-44 2021-02-10T04:37:25.750 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 1138 (ObjNum 760 snap 243 seq_num 760) dirty exists 2021-02-10T04:37:25.750 INFO:tasks.rados.rados.0.smithi085.stdout:2414: done (15 left) 2021-02-10T04:37:25.750 INFO:tasks.rados.rados.0.smithi085.stdout:2416: done (14 left) 2021-02-10T04:37:25.750 INFO:tasks.rados.rados.0.smithi085.stdout:2417: done (13 left) 2021-02-10T04:37:25.750 INFO:tasks.rados.rados.0.smithi085.stdout:2421: write oid 5 current snap is 270 2021-02-10T04:37:25.750 INFO:tasks.rados.rados.0.smithi085.stdout:2418: finishing write tid 1 to smithi08575397-38 2021-02-10T04:37:25.750 INFO:tasks.rados.rados.0.smithi085.stdout:2418: finishing write tid 2 to smithi08575397-38 2021-02-10T04:37:25.751 INFO:tasks.rados.rados.0.smithi085.stdout:2418: finishing write tid 3 to smithi08575397-38 2021-02-10T04:37:25.751 INFO:tasks.rados.rados.0.smithi085.stdout:2418: finishing write tid 4 to smithi08575397-38 2021-02-10T04:37:25.751 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 1092 (ObjNum 834 snap 270 seq_num 834) dirty exists 2021-02-10T04:37:25.751 INFO:tasks.rados.rados.0.smithi085.stdout:2418: left oid 38 (ObjNum 834 snap 270 seq_num 834) 2021-02-10T04:37:25.751 INFO:tasks.rados.rados.0.smithi085.stdout:2415: finishing copy_from to smithi08575397-28 2021-02-10T04:37:25.751 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 1261 (ObjNum 812 snap 265 seq_num 812) dirty exists 2021-02-10T04:37:25.751 INFO:tasks.rados.rados.0.smithi085.stdout:2415: finishing copy_from racing read to smithi08575397-28 2021-02-10T04:37:25.751 INFO:tasks.rados.rados.0.smithi085.stdout:2421: seq_num 835 ranges {433845=715392,1662582=371263} 2021-02-10T04:37:25.757 INFO:tasks.rados.rados.0.smithi085.stdout:2421: writing smithi08575397-5 from 433845 to 1149237 tid 1 2021-02-10T04:37:25.762 INFO:tasks.rados.rados.0.smithi085.stdout:2421: writing smithi08575397-5 from 1662582 to 2033845 tid 2 2021-02-10T04:37:25.763 INFO:tasks.rados.rados.0.smithi085.stdout:2415: done (13 left) 2021-02-10T04:37:25.763 INFO:tasks.rados.rados.0.smithi085.stdout:2418: done (12 left) 2021-02-10T04:37:25.763 INFO:tasks.rados.rados.0.smithi085.stdout:2419: done (11 left) 2021-02-10T04:37:25.763 INFO:tasks.rados.rados.0.smithi085.stdout:2420: done (10 left) 2021-02-10T04:37:25.763 INFO:tasks.rados.rados.0.smithi085.stdout:2422: rollback oid 43 current snap is 270 2021-02-10T04:37:25.763 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 43 to 248 2021-02-10T04:37:25.763 INFO:tasks.rados.rados.0.smithi085.stdout:2421: finishing write tid 1 to smithi08575397-5 2021-02-10T04:37:25.763 INFO:tasks.rados.rados.0.smithi085.stdout:2423: snap_remove snap 200 2021-02-10T04:37:26.324 INFO:tasks.rados.rados.0.smithi085.stdout:2423: done (11 left) 2021-02-10T04:37:26.324 INFO:tasks.rados.rados.0.smithi085.stdout:2424: snap_create 2021-02-10T04:37:26.325 INFO:tasks.rados.rados.0.smithi085.stdout:2422: finishing rollback tid 0 to smithi08575397-43 2021-02-10T04:37:26.325 INFO:tasks.rados.rados.0.smithi085.stdout:2421: finishing write tid 2 to smithi08575397-5 2021-02-10T04:37:26.325 INFO:tasks.rados.rados.0.smithi085.stdout:2421: finishing write tid 3 to smithi08575397-5 2021-02-10T04:37:26.325 INFO:tasks.rados.rados.0.smithi085.stdout:2421: finishing write tid 4 to smithi08575397-5 2021-02-10T04:37:26.325 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 772 (ObjNum 835 snap 270 seq_num 835) dirty exists 2021-02-10T04:37:26.325 INFO:tasks.rados.rados.0.smithi085.stdout:2421: left oid 5 (ObjNum 835 snap 270 seq_num 835) 2021-02-10T04:37:26.325 INFO:tasks.rados.rados.0.smithi085.stdout:2422: finishing rollback tid 1 to smithi08575397-43 2021-02-10T04:37:26.325 INFO:tasks.rados.rados.0.smithi085.stdout:2422: finishing rollback tid 2 to smithi08575397-43 2021-02-10T04:37:26.325 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 1146 (ObjNum 748 snap 240 seq_num 748) dirty exists 2021-02-10T04:37:26.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:26 smithi085 conmon[29319]: cluster 2021-02-10T04:37:25.317394+0000 mon.a 2021-02-10T04:37:26.621 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:26 smithi085 conmon[29319]: (mon.0) 1489 : cluster [DBG] osdmap e627: 8 total, 8 up, 8 in 2021-02-10T04:37:26.621 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:26 smithi085 conmon[29319]: cluster 2021-02-10T04:37:26.137360+0000 mgr.y (mgr.14140) 672 : cluster [DBG] 2021-02-10T04:37:26.621 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:26 smithi085 conmon[29319]: pgmap v1091: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 491 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 2.6 MiB/s wr, 23 op/s 2021-02-10T04:37:26.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:26 smithi085 conmon[35961]: cluster 2021-02-10T04:37:26.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:26 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:37:26.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:26 smithi085 conmon[35961]: 37:25.317394+0000 mon.a (mon.0) 1489 : cluster [DBG] osdmap e627: 8 total, 8 up, 8 in 2021-02-10T04:37:26.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:26 smithi085 conmon[35961]: cluster 2021-02-10T04:37:26. 2021-02-10T04:37:26.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:26 smithi085 conmon[35961]: 137360+0000 mgr.y (mgr.14140) 672 : cluster [DBG] 2021-02-10T04:37:26.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:26 smithi085 conmon[35961]: pgmap v1091: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 491 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 2.6 MiB/s wr, 23 op/s 2021-02-10T04:37:26.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:26 smithi164 conmon[29288]: cluster 2021-02-10T04:37:26.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:26 smithi164 conmon[29288]: 2021-02-10T04:37:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:26 smithi164 conmon[29288]: 2021 2021-02-10T04:37:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:26 smithi164 conmon[29288]: -02-10T04:37:25.317394 2021-02-10T04:37:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:26 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1489 : cluster [DBG] osdmap e627: 8 total, 8 up, 8 in 2021-02-10T04:37:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:26 smithi164 conmon[29288]: cluster 2021-02-10T04:37:26. 2021-02-10T04:37:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:26 smithi164 conmon[29288]: 137360+0000 mgr.y (mgr.14140) 672 : cluster [DBG] 2021-02-10T04:37:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:26 smithi164 conmon[29288]: pgmap v1091: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 7 active+clean+snaptrim, 156 active+clean; 491 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 2.6 MiB/s wr, 23 op/s 2021-02-10T04:37:26.682 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":1075,"stamp":"2021-02-10T04:37:15.194559+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":507073657,"num_objects":581,"num_object_clones":177,"num_object_copies":1743,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":7,"num_read":6729,"num_read_kb":4713416,"num_write":4371,"num_write_kb":1262541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":2080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9265,"ondisk_log_size":9265,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1242832,"kb_used_data":1032976,"kb_used_omap":0,"kb_used_meta":209792,"kb_avail":748718384,"statfs":{"total":767960285184,"available":766687625216,"internally_reserved":0,"allocated":1057767424,"data_stored":1484278550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":214827008},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,12],"upper_bound":1048576},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9670442,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":-1,"num_read":17,"num_read_kb":4487,"num_write":26,"num_write_kb":12337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.992664"},"pg_stats":[{"pgid":"5.9","version":"131'3","reported_seq":"608","reported_epoch":"614","state":"active+clean","last_fresh":"2021-02-10T04:37:12.829405+0000","last_change":"2021-02-10T04:33:57.871495+0000","last_active":"2021-02-10T04:37:12.829405+0000","last_peered":"2021-02-10T04:37:12.829405+0000","last_clean":"2021-02-10T04:37:12.829405+0000","last_became_active":"2021-02-10T04:29:07.145525+0000","last_became_peered":"2021-02-10T04:29:07.145525+0000","last_unstale":"2021-02-10T04:37:12.829405+0000","last_undegraded":"2021-02-10T04:37:12.829405+0000","last_fullsized":"2021-02-10T04:37:12.829405+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'3","last_scrub_stamp":"2021-02-10T04:33:57.871402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:33:57.871402+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"538'42","reported_seq":"661","reported_epoch":"613","state":"active+clean","last_fresh":"2021-02-10T04:37:11.195501+0000","last_change":"2021-02-10T04:33:55.585872+0000","last_active":"2021-02-10T04:37:11.195501+0000","last_peered":"2021-02-10T04:37:11.195501+0000","last_clean":"2021-02-10T04:37:11.195501+0000","last_became_active":"2021-02-10T04:25:49.002058+0000","last_became_peered":"2021-02-10T04:25:49.002058+0000","last_unstale":"2021-02-10T04:37:11.195501+0000","last_undegraded":"2021-02-10T04:37:11.195501+0000","last_fullsized":"2021-02-10T04:37:11.195501+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2021-02-10T04:33:55.585807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:55.585807+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":77,"num_read_kb":60,"num_write":43,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"600","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.199993+0000","last_change":"2021-02-10T04:29:34.787349+0000","last_active":"2021-02-10T04:37:09.199993+0000","last_peered":"2021-02-10T04:37:09.199993+0000","last_clean":"2021-02-10T04:37:09.199993+0000","last_became_active":"2021-02-10T04:25:36.344641+0000","last_became_peered":"2021-02-10T04:25:36.344641+0000","last_unstale":"2021-02-10T04:37:09.199993+0000","last_undegraded":"2021-02-10T04:37:09.199993+0000","last_fullsized":"2021-02-10T04:37:09.199993+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:29:34.787265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:34.787265+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"573","reported_epoch":"613","state":"active+clean","last_fresh":"2021-02-10T04:37:11.196032+0000","last_change":"2021-02-10T04:29:19.405593+0000","last_active":"2021-02-10T04:37:11.196032+0000","last_peered":"2021-02-10T04:37:11.196032+0000","last_clean":"2021-02-10T04:37:11.196032+0000","last_became_active":"2021-02-10T04:25:53.148090+0000","last_became_peered":"2021-02-10T04:25:53.148090+0000","last_unstale":"2021-02-10T04:37:11.196032+0000","last_undegraded":"2021-02-10T04:37:11.196032+0000","last_fullsized":"2021-02-10T04:37:11.196032+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:19.405525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:29:19.405525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"614'800","reported_seq":"2003","reported_epoch":"614","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:37:12.438071+0000","last_change":"2021-02-10T04:37:11.629973+0000","last_active":"2021-02-10T04:37:12.438071+0000","last_peered":"2021-02-10T04:37:12.438071+0000","last_clean":"2021-02-10T04:37:12.438071+0000","last_became_active":"2021-02-10T04:33:05.144488+0000","last_became_peered":"2021-02-10T04:33:05.144488+0000","last_unstale":"2021-02-10T04:37:12.438071+0000","last_undegraded":"2021-02-10T04:37:12.438071+0000","last_fullsized":"2021-02-10T04:37:12.438071+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'301","last_scrub_stamp":"2021-02-10T04:33:56.571925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:56.571925+0000","log_size":800,"ondisk_log_size":800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"stat_sum":{"num_bytes":69646550,"num_objects":29,"num_object_clones":24,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":388,"num_read_kb":412612,"num_write":273,"num_write_kb":102343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"},{"start":"d4","length":"8"},{"start":"dd","length":"1"},{"start":"e1","length":"3"},{"start":"e9","length":"2"},{"start":"ed","length":"4"},{"start":"f2","length":"1"},{"start":"f4","length":"5"},{"start":"fb","length":"1"},{"start":"fd","length":"4"},{"start":"103","length":"1"},{"start":"106","length":"4"},{"start":"10c","length":"1"},{"start":"10f","length":"6"},{"start":"116","length":"2"},{"start":"11c","length":"3"},{"start":"121","length":"1"},{"start":"123","length":"2"},{"start":"126","length":"1"},{"start":"128","length":"5"},{"start":"12e","length":"3"},{"start":"133","length":"4"},{"start":"139","length":"2"},{"start":"13c","length":"3"},{"start":"140","length":"1"},{"start":"143","length":"1"},{"start":"146","length":"1"},{"start":"148","length":"1"},{"start":"14c","length":"1"},{"start":"14e","length":"1"},{"start":"155","length":"2"},{"start":"15a","length":"4"},{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"168","length":"2"},{"start":"16b","length":"1"},{"start":"16f","length":"5"},{"start":"178","length":"4"},{"start":"17f","length":"1"},{"start":"185","length":"2"},{"start":"18a","length":"2"},{"start":"18e","length":"3"},{"start":"194","length":"4"},{"start":"19b","length":"5"},{"start":"1a1","length":"2"},{"start":"1a4","length":"1"},{"start":"1a8","length":"1"},{"start":"1ab","length":"1"},{"start":"1ae","length":"1"},{"start":"1b1","length":"2"},{"start":"1b7","length":"1"},{"start":"1b9","length":"2"},{"start":"1bc","length":"1"},{"start":"1bf","length":"2"},{"start":"1c3","length":"1"},{"start":"1c8","length":"1"},{"start":"1cd","length":"1"}]},{"pgid":"4.b","version":"538'26","reported_seq":"629","reported_epoch":"615","state":"active+clean","last_fresh":"2021-02-10T04:37:13.196701+0000","last_change":"2021-02-10T04:28:50.696234+0000","last_active":"2021-02-10T04:37:13.196701+0000","last_peered":"2021-02-10T04:37:13.196701+0000","last_clean":"2021-02-10T04:37:13.196701+0000","last_became_active":"2021-02-10T04:25:49.004346+0000","last_became_peered":"2021-02-10T04:25:49.004346+0000","last_unstale":"2021-02-10T04:37:13.196701+0000","last_undegraded":"2021-02-10T04:37:13.196701+0000","last_fullsized":"2021-02-10T04:37:13.196701+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'20","last_scrub_stamp":"2021-02-10T04:28:50.696124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:50.696124+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":53,"num_read_kb":36,"num_write":27,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"257","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.201422+0000","last_change":"2021-02-10T04:33:05.151251+0000","last_active":"2021-02-10T04:37:10.201422+0000","last_peered":"2021-02-10T04:37:10.201422+0000","last_clean":"2021-02-10T04:37:10.201422+0000","last_became_active":"2021-02-10T04:33:05.150882+0000","last_became_peered":"2021-02-10T04:33:05.150882+0000","last_unstale":"2021-02-10T04:37:10.201422+0000","last_undegraded":"2021-02-10T04:37:10.201422+0000","last_fullsized":"2021-02-10T04:37:10.201422+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:31.102558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:31.102558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"577","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.199498+0000","last_change":"2021-02-10T04:29:35.805411+0000","last_active":"2021-02-10T04:37:09.199498+0000","last_peered":"2021-02-10T04:37:09.199498+0000","last_clean":"2021-02-10T04:37:09.199498+0000","last_became_active":"2021-02-10T04:25:51.016118+0000","last_became_peered":"2021-02-10T04:25:51.016118+0000","last_unstale":"2021-02-10T04:37:09.199498+0000","last_undegraded":"2021-02-10T04:37:09.199498+0000","last_fullsized":"2021-02-10T04:37:09.199498+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:35.805326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.805326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"577","reported_epoch":"615","state":"active+clean","last_fresh":"2021-02-10T04:37:13.196253+0000","last_change":"2021-02-10T04:28:48.700480+0000","last_active":"2021-02-10T04:37:13.196253+0000","last_peered":"2021-02-10T04:37:13.196253+0000","last_clean":"2021-02-10T04:37:13.196253+0000","last_became_active":"2021-02-10T04:25:53.038249+0000","last_became_peered":"2021-02-10T04:25:53.038249+0000","last_unstale":"2021-02-10T04:37:13.196253+0000","last_undegraded":"2021-02-10T04:37:13.196253+0000","last_fullsized":"2021-02-10T04:37:13.196253+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:48.700418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:48.700418+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"259'74","reported_seq":"1828","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.201546+0000","last_change":"2021-02-10T04:37:10.201886+0000","last_active":"2021-02-10T04:37:10.201546+0000","last_peered":"2021-02-10T04:37:10.201546+0000","last_clean":"2021-02-10T04:37:10.201546+0000","last_became_active":"2021-02-10T04:28:25.155557+0000","last_became_peered":"2021-02-10T04:28:25.155557+0000","last_unstale":"2021-02-10T04:37:10.201546+0000","last_undegraded":"2021-02-10T04:37:10.201546+0000","last_fullsized":"2021-02-10T04:37:10.201546+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'62","last_scrub_stamp":"2021-02-10T04:30:10.905905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:10.905905+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2437048,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":48,"num_read_kb":56455,"num_write":38,"num_write_kb":14043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"},{"start":"d4","length":"8"},{"start":"dd","length":"1"},{"start":"e1","length":"3"},{"start":"e9","length":"2"},{"start":"ed","length":"4"},{"start":"f2","length":"1"},{"start":"f4","length":"5"},{"start":"fb","length":"1"},{"start":"fd","length":"4"},{"start":"103","length":"1"},{"start":"106","length":"4"},{"start":"10c","length":"1"},{"start":"10f","length":"6"},{"start":"116","length":"2"},{"start":"11c","length":"3"},{"start":"121","length":"1"},{"start":"123","length":"2"},{"start":"126","length":"1"},{"start":"128","length":"5"},{"start":"12e","length":"3"},{"start":"133","length":"4"},{"start":"139","length":"2"},{"start":"13c","length":"3"},{"start":"140","length":"1"},{"start":"143","length":"1"},{"start":"146","length":"1"},{"start":"148","length":"1"},{"start":"14c","length":"1"},{"start":"14e","length":"1"},{"start":"155","length":"2"},{"start":"15a","length":"4"},{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"168","length":"2"},{"start":"16b","length":"1"},{"start":"16f","length":"5"},{"start":"178","length":"4"},{"start":"17f","length":"1"},{"start":"185","length":"2"},{"start":"18a","length":"2"},{"start":"18e","length":"3"},{"start":"194","length":"4"},{"start":"19b","length":"5"},{"start":"1a1","length":"2"},{"start":"1a4","length":"1"},{"start":"1a8","length":"1"},{"start":"1ab","length":"1"},{"start":"1ae","length":"1"},{"start":"1b1","length":"2"},{"start":"1b7","length":"1"},{"start":"1b9","length":"2"},{"start":"1bc","length":"1"},{"start":"1bf","length":"2"},{"start":"1c3","length":"1"},{"start":"1c8","length":"1"},{"start":"1cd","length":"1"},{"start":"1d0","length":"2"},{"start":"1d5","length":"1"},{"start":"1d8","length":"2"},{"start":"1dc","length":"1"},{"start":"1df","length":"1"},{"start":"1e1","length":"1"},{"start":"1e4","length":"1"},{"start":"1eb","length":"1"},{"start":"1ef","length":"1"}]},{"pgid":"4.a","version":"538'45","reported_seq":"658","reported_epoch":"614","state":"active+clean","last_fresh":"2021-02-10T04:37:12.201808+0000","last_change":"2021-02-10T04:29:26.671051+0000","last_active":"2021-02-10T04:37:12.201808+0000","last_peered":"2021-02-10T04:37:12.201808+0000","last_clean":"2021-02-10T04:37:12.201808+0000","last_became_active":"2021-02-10T04:25:49.015913+0000","last_became_peered":"2021-02-10T04:25:49.015913+0000","last_unstale":"2021-02-10T04:37:12.201808+0000","last_undegraded":"2021-02-10T04:37:12.201808+0000","last_fullsized":"2021-02-10T04:37:12.201808+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'29","last_scrub_stamp":"2021-02-10T04:29:26.670950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.670950+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":80,"num_read_kb":60,"num_write":46,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"62'2","reported_seq":"718","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.201023+0000","last_change":"2021-02-10T04:28:33.618291+0000","last_active":"2021-02-10T04:37:10.201023+0000","last_peered":"2021-02-10T04:37:10.201023+0000","last_clean":"2021-02-10T04:37:10.201023+0000","last_became_active":"2021-02-10T04:25:34.331199+0000","last_became_peered":"2021-02-10T04:25:34.331199+0000","last_unstale":"2021-02-10T04:37:10.201023+0000","last_undegraded":"2021-02-10T04:37:10.201023+0000","last_fullsized":"2021-02-10T04:37:10.201023+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2021-02-10T04:28:33.618170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:33.618170+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"572","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.199864+0000","last_change":"2021-02-10T04:28:37.645921+0000","last_active":"2021-02-10T04:37:09.199864+0000","last_peered":"2021-02-10T04:37:09.199864+0000","last_clean":"2021-02-10T04:37:09.199864+0000","last_became_active":"2021-02-10T04:25:51.025882+0000","last_became_peered":"2021-02-10T04:25:51.025882+0000","last_unstale":"2021-02-10T04:37:09.199864+0000","last_undegraded":"2021-02-10T04:37:09.199864+0000","last_fullsized":"2021-02-10T04:37:09.199864+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:37.645857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:37.645857+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"498","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.199141+0000","last_change":"2021-02-10T04:29:24.731524+0000","last_active":"2021-02-10T04:37:09.199141+0000","last_peered":"2021-02-10T04:37:09.199141+0000","last_clean":"2021-02-10T04:37:09.199141+0000","last_became_active":"2021-02-10T04:29:07.162878+0000","last_became_peered":"2021-02-10T04:29:07.162878+0000","last_unstale":"2021-02-10T04:37:09.199141+0000","last_undegraded":"2021-02-10T04:37:09.199141+0000","last_fullsized":"2021-02-10T04:37:09.199141+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.731458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.731458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"615'373","reported_seq":"2146","reported_epoch":"615","state":"active+clean+snaptrim_wait","last_fresh":"2021-02-10T04:37:13.639280+0000","last_change":"2021-02-10T04:37:13.639280+0000","last_active":"2021-02-10T04:37:13.639280+0000","last_peered":"2021-02-10T04:37:13.639280+0000","last_clean":"2021-02-10T04:37:13.639280+0000","last_became_active":"2021-02-10T04:28:25.150246+0000","last_became_peered":"2021-02-10T04:28:25.150246+0000","last_unstale":"2021-02-10T04:37:13.639280+0000","last_undegraded":"2021-02-10T04:37:13.639280+0000","last_fullsized":"2021-02-10T04:37:13.639280+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'63","last_scrub_stamp":"2021-02-10T04:29:39.700115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:29:39.700115+0000","log_size":373,"ondisk_log_size":373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"stat_sum":{"num_bytes":31819384,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":168,"num_read_kb":143666,"num_write":106,"num_write_kb":49662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"},{"start":"d4","length":"8"},{"start":"dd","length":"1"},{"start":"e1","length":"3"},{"start":"e9","length":"2"},{"start":"ed","length":"4"},{"start":"f2","length":"1"},{"start":"f4","length":"5"},{"start":"fb","length":"1"},{"start":"fd","length":"4"},{"start":"103","length":"1"},{"start":"106","length":"4"},{"start":"10c","length":"1"},{"start":"10f","length":"6"},{"start":"116","length":"2"},{"start":"11c","length":"3"},{"start":"121","length":"1"},{"start":"123","length":"2"},{"start":"126","length":"1"},{"start":"128","length":"5"},{"start":"12e","length":"3"},{"start":"133","length":"4"},{"start":"139","length":"2"},{"start":"13c","length":"3"},{"start":"140","length":"1"},{"start":"143","length":"1"},{"start":"146","length":"1"},{"start":"148","length":"1"},{"start":"14c","length":"1"},{"start":"14e","length":"1"},{"start":"155","length":"2"},{"start":"15a","length":"4"},{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"168","length":"2"},{"start":"16b","length":"1"},{"start":"16f","length":"5"},{"start":"178","length":"4"},{"start":"17f","length":"1"},{"start":"185","length":"2"},{"start":"18a","length":"2"},{"start":"18e","length":"3"},{"start":"194","length":"4"},{"start":"19b","length":"5"},{"start":"1a1","length":"2"},{"start":"1a4","length":"1"},{"start":"1a8","length":"1"},{"start":"1ab","length":"1"},{"start":"1ae","length":"2"},{"start":"1b1","length":"2"},{"start":"1b7","length":"1"},{"start":"1b9","length":"2"},{"start":"1bc","length":"1"},{"start":"1bf","length":"2"},{"start":"1c3","length":"1"},{"start":"1c8","length":"1"},{"start":"1cd","length":"1"},{"start":"1d1","length":"1"},{"start":"1d5","length":"1"},{"start":"1d8","length":"2"},{"start":"1df","length":"1"}]},{"pgid":"4.d","version":"538'47","reported_seq":"645","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.201563+0000","last_change":"2021-02-10T04:28:49.515083+0000","last_active":"2021-02-10T04:37:10.201563+0000","last_peered":"2021-02-10T04:37:10.201563+0000","last_clean":"2021-02-10T04:37:10.201563+0000","last_became_active":"2021-02-10T04:25:49.011699+0000","last_became_peered":"2021-02-10T04:25:49.011699+0000","last_unstale":"2021-02-10T04:37:10.201563+0000","last_undegraded":"2021-02-10T04:37:10.201563+0000","last_fullsized":"2021-02-10T04:37:10.201563+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'33","last_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:28:49.514984+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":62,"num_read_kb":44,"num_write":36,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"494","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.199235+0000","last_change":"2021-02-10T04:29:07.146266+0000","last_active":"2021-02-10T04:37:10.199235+0000","last_peered":"2021-02-10T04:37:10.199235+0000","last_clean":"2021-02-10T04:37:10.199235+0000","last_became_active":"2021-02-10T04:29:07.145616+0000","last_became_peered":"2021-02-10T04:29:07.145616+0000","last_unstale":"2021-02-10T04:37:10.199235+0000","last_undegraded":"2021-02-10T04:37:10.199235+0000","last_fullsized":"2021-02-10T04:37:10.199235+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:44.410817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:44.410817+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"257","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.200873+0000","last_change":"2021-02-10T04:33:05.147166+0000","last_active":"2021-02-10T04:37:10.200873+0000","last_peered":"2021-02-10T04:37:10.200873+0000","last_clean":"2021-02-10T04:37:10.200873+0000","last_became_active":"2021-02-10T04:33:05.146783+0000","last_became_peered":"2021-02-10T04:33:05.146783+0000","last_unstale":"2021-02-10T04:37:10.200873+0000","last_undegraded":"2021-02-10T04:37:10.200873+0000","last_fullsized":"2021-02-10T04:37:10.200873+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:35.958255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.958255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"570","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.199991+0000","last_change":"2021-02-10T04:28:32.623869+0000","last_active":"2021-02-10T04:37:09.199991+0000","last_peered":"2021-02-10T04:37:09.199991+0000","last_clean":"2021-02-10T04:37:09.199991+0000","last_became_active":"2021-02-10T04:25:53.038218+0000","last_became_peered":"2021-02-10T04:25:53.038218+0000","last_unstale":"2021-02-10T04:37:09.199991+0000","last_undegraded":"2021-02-10T04:37:09.199991+0000","last_fullsized":"2021-02-10T04:37:09.199991+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:32.623791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:32.623791+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"386'110","reported_seq":"1835","reported_epoch":"611","state":"active+clean+scrubbing","last_fresh":"2021-02-10T04:37:09.200526+0000","last_change":"2021-02-10T04:33:57.382223+0000","last_active":"2021-02-10T04:37:09.200526+0000","last_peered":"2021-02-10T04:37:09.200526+0000","last_clean":"2021-02-10T04:37:09.200526+0000","last_became_active":"2021-02-10T04:28:25.157931+0000","last_became_peered":"2021-02-10T04:28:25.157931+0000","last_unstale":"2021-02-10T04:37:09.200526+0000","last_undegraded":"2021-02-10T04:37:09.200526+0000","last_fullsized":"2021-02-10T04:37:09.200526+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":110,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":196,"stat_sum":{"num_bytes":12437781,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":78,"num_read_kb":76873,"num_write":73,"num_write_kb":20689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"516","reported_epoch":"613","state":"active+clean","last_fresh":"2021-02-10T04:37:11.195583+0000","last_change":"2021-02-10T04:29:37.336179+0000","last_active":"2021-02-10T04:37:11.195583+0000","last_peered":"2021-02-10T04:37:11.195583+0000","last_clean":"2021-02-10T04:37:11.195583+0000","last_became_active":"2021-02-10T04:28:49.046469+0000","last_became_peered":"2021-02-10T04:28:49.046469+0000","last_unstale":"2021-02-10T04:37:11.195583+0000","last_undegraded":"2021-02-10T04:37:11.195583+0000","last_fullsized":"2021-02-10T04:37:11.195583+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:37.336092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:37.336092+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"588","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.195772+0000","last_change":"2021-02-10T04:33:51.858490+0000","last_active":"2021-02-10T04:37:09.195772+0000","last_peered":"2021-02-10T04:37:09.195772+0000","last_clean":"2021-02-10T04:37:09.195772+0000","last_became_active":"2021-02-10T04:29:07.159274+0000","last_became_peered":"2021-02-10T04:29:07.159274+0000","last_unstale":"2021-02-10T04:37:09.195772+0000","last_undegraded":"2021-02-10T04:37:09.195772+0000","last_fullsized":"2021-02-10T04:37:09.195772+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:51.858386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:33:51.858386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"538'23","reported_seq":"612","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.200969+0000","last_change":"2021-02-10T04:28:41.427119+0000","last_active":"2021-02-10T04:37:10.200969+0000","last_peered":"2021-02-10T04:37:10.200969+0000","last_clean":"2021-02-10T04:37:10.200969+0000","last_became_active":"2021-02-10T04:25:49.001725+0000","last_became_peered":"2021-02-10T04:25:49.001725+0000","last_unstale":"2021-02-10T04:37:10.200969+0000","last_undegraded":"2021-02-10T04:37:10.200969+0000","last_fullsized":"2021-02-10T04:37:10.200969+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'13","last_scrub_stamp":"2021-02-10T04:28:41.427050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:41.427050+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":28,"num_write":23,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"260","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.197954+0000","last_change":"2021-02-10T04:33:05.168734+0000","last_active":"2021-02-10T04:37:09.197954+0000","last_peered":"2021-02-10T04:37:09.197954+0000","last_clean":"2021-02-10T04:37:09.197954+0000","last_became_active":"2021-02-10T04:33:05.168314+0000","last_became_peered":"2021-02-10T04:33:05.168314+0000","last_unstale":"2021-02-10T04:37:09.197954+0000","last_undegraded":"2021-02-10T04:37:09.197954+0000","last_fullsized":"2021-02-10T04:37:09.197954+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:18.107867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:18.107867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"58'2","reported_seq":"710","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.200145+0000","last_change":"2021-02-10T04:28:35.648670+0000","last_active":"2021-02-10T04:37:09.200145+0000","last_peered":"2021-02-10T04:37:09.200145+0000","last_clean":"2021-02-10T04:37:09.200145+0000","last_became_active":"2021-02-10T04:25:51.025268+0000","last_became_peered":"2021-02-10T04:25:51.025268+0000","last_unstale":"2021-02-10T04:37:09.200145+0000","last_undegraded":"2021-02-10T04:37:09.200145+0000","last_fullsized":"2021-02-10T04:37:09.200145+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:28:35.648602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:35.648602+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"572","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.201906+0000","last_change":"2021-02-10T04:29:38.873167+0000","last_active":"2021-02-10T04:37:10.201906+0000","last_peered":"2021-02-10T04:37:10.201906+0000","last_clean":"2021-02-10T04:37:10.201906+0000","last_became_active":"2021-02-10T04:25:53.043027+0000","last_became_peered":"2021-02-10T04:25:53.043027+0000","last_unstale":"2021-02-10T04:37:10.201906+0000","last_undegraded":"2021-02-10T04:37:10.201906+0000","last_fullsized":"2021-02-10T04:37:10.201906+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:38.873102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.873102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"615'558","reported_seq":"4301","reported_epoch":"615","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:37:13.642334+0000","last_change":"2021-02-10T04:37:13.640001+0000","last_active":"2021-02-10T04:37:13.642334+0000","last_peered":"2021-02-10T04:37:13.642334+0000","last_clean":"2021-02-10T04:37:13.642334+0000","last_became_active":"2021-02-10T04:28:25.149226+0000","last_became_peered":"2021-02-10T04:28:25.149226+0000","last_unstale":"2021-02-10T04:37:13.642334+0000","last_undegraded":"2021-02-10T04:37:13.642334+0000","last_fullsized":"2021-02-10T04:37:13.642334+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'236","last_scrub_stamp":"2021-02-10T04:33:50.351534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:50.351534+0000","log_size":558,"ondisk_log_size":558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"stat_sum":{"num_bytes":44983977,"num_objects":19,"num_object_clones":15,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":379,"num_read_kb":469642,"num_write":194,"num_write_kb":66848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"},{"start":"d4","length":"8"},{"start":"dd","length":"1"},{"start":"e1","length":"3"},{"start":"e9","length":"2"},{"start":"ed","length":"4"},{"start":"f2","length":"1"},{"start":"f4","length":"5"},{"start":"fb","length":"1"},{"start":"fd","length":"4"},{"start":"103","length":"1"},{"start":"106","length":"4"},{"start":"10c","length":"1"},{"start":"10f","length":"6"},{"start":"116","length":"2"},{"start":"11c","length":"3"},{"start":"121","length":"1"},{"start":"123","length":"2"},{"start":"126","length":"1"},{"start":"128","length":"5"},{"start":"12e","length":"3"},{"start":"133","length":"4"},{"start":"139","length":"2"},{"start":"13c","length":"3"},{"start":"140","length":"1"},{"start":"143","length":"1"},{"start":"146","length":"1"},{"start":"148","length":"1"},{"start":"14c","length":"1"},{"start":"14e","length":"1"},{"start":"155","length":"2"},{"start":"15a","length":"4"},{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"168","length":"2"},{"start":"16b","length":"1"},{"start":"16f","length":"5"},{"start":"178","length":"4"},{"start":"17f","length":"1"},{"start":"185","length":"2"},{"start":"18a","length":"2"},{"start":"18e","length":"3"},{"start":"194","length":"4"},{"start":"19b","length":"5"},{"start":"1a1","length":"2"},{"start":"1a4","length":"1"},{"start":"1a8","length":"1"},{"start":"1ab","length":"1"},{"start":"1ae","length":"1"},{"start":"1b1","length":"2"},{"start":"1b7","length":"1"},{"start":"1b9","length":"2"},{"start":"1bc","length":"1"},{"start":"1bf","length":"2"},{"start":"1c3","length":"1"},{"start":"1c8","length":"1"},{"start":"1cd","length":"1"}]},{"pgid":"1.a","version":"0'0","reported_seq":"516","reported_epoch":"613","state":"active+clean","last_fresh":"2021-02-10T04:37:11.195885+0000","last_change":"2021-02-10T04:29:29.384240+0000","last_active":"2021-02-10T04:37:11.195885+0000","last_peered":"2021-02-10T04:37:11.195885+0000","last_clean":"2021-02-10T04:37:11.195885+0000","last_became_active":"2021-02-10T04:28:49.050935+0000","last_became_peered":"2021-02-10T04:28:49.050935+0000","last_unstale":"2021-02-10T04:37:11.195885+0000","last_undegraded":"2021-02-10T04:37:11.195885+0000","last_fullsized":"2021-02-10T04:37:11.195885+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.384170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.384170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"538'29","reported_seq":"316","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.197844+0000","last_change":"2021-02-10T04:33:05.154998+0000","last_active":"2021-02-10T04:37:09.197844+0000","last_peered":"2021-02-10T04:37:09.197844+0000","last_clean":"2021-02-10T04:37:09.197844+0000","last_became_active":"2021-02-10T04:33:05.149026+0000","last_became_peered":"2021-02-10T04:33:05.149026+0000","last_unstale":"2021-02-10T04:37:09.197844+0000","last_undegraded":"2021-02-10T04:37:09.197844+0000","last_fullsized":"2021-02-10T04:37:09.197844+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'17","last_scrub_stamp":"2021-02-10T04:29:37.933206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:37.933206+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":49,"num_read_kb":36,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"320","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.199658+0000","last_change":"2021-02-10T04:33:05.169654+0000","last_active":"2021-02-10T04:37:09.199658+0000","last_peered":"2021-02-10T04:37:09.199658+0000","last_clean":"2021-02-10T04:37:09.199658+0000","last_became_active":"2021-02-10T04:33:05.169289+0000","last_became_peered":"2021-02-10T04:33:05.169289+0000","last_unstale":"2021-02-10T04:37:09.199658+0000","last_undegraded":"2021-02-10T04:37:09.199658+0000","last_fullsized":"2021-02-10T04:37:09.199658+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_clean_scrub_stamp":"2021-02-10T04:28:39.128781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"78'3","reported_seq":"713","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.201880+0000","last_change":"2021-02-10T04:28:43.509446+0000","last_active":"2021-02-10T04:37:10.201880+0000","last_peered":"2021-02-10T04:37:10.201880+0000","last_clean":"2021-02-10T04:37:10.201880+0000","last_became_active":"2021-02-10T04:25:51.025218+0000","last_became_peered":"2021-02-10T04:25:51.025218+0000","last_unstale":"2021-02-10T04:37:10.201880+0000","last_undegraded":"2021-02-10T04:37:10.201880+0000","last_fullsized":"2021-02-10T04:37:10.201880+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:28:43.509353+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:28:43.509353+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"573","reported_epoch":"613","state":"active+clean","last_fresh":"2021-02-10T04:37:11.196335+0000","last_change":"2021-02-10T04:28:45.481538+0000","last_active":"2021-02-10T04:37:11.196335+0000","last_peered":"2021-02-10T04:37:11.196335+0000","last_clean":"2021-02-10T04:37:11.196335+0000","last_became_active":"2021-02-10T04:25:53.042697+0000","last_became_peered":"2021-02-10T04:25:53.042697+0000","last_unstale":"2021-02-10T04:37:11.196335+0000","last_undegraded":"2021-02-10T04:37:11.196335+0000","last_fullsized":"2021-02-10T04:37:11.196335+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:45.481472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:28:45.481472+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"614'582","reported_seq":"3322","reported_epoch":"614","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:37:12.686580+0000","last_change":"2021-02-10T04:37:12.684453+0000","last_active":"2021-02-10T04:37:12.686580+0000","last_peered":"2021-02-10T04:37:12.686580+0000","last_clean":"2021-02-10T04:37:12.686580+0000","last_became_active":"2021-02-10T04:28:25.148500+0000","last_became_peered":"2021-02-10T04:28:25.148500+0000","last_unstale":"2021-02-10T04:37:12.686580+0000","last_undegraded":"2021-02-10T04:37:12.686580+0000","last_fullsized":"2021-02-10T04:37:12.686580+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'110","last_scrub_stamp":"2021-02-10T04:30:18.463782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:18.463782+0000","log_size":582,"ondisk_log_size":582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"stat_sum":{"num_bytes":36448576,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":1,"num_read":383,"num_read_kb":476256,"num_write":216,"num_write_kb":84347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"},{"start":"d4","length":"8"},{"start":"dd","length":"1"},{"start":"e1","length":"3"},{"start":"e9","length":"2"},{"start":"ed","length":"4"},{"start":"f2","length":"1"},{"start":"f4","length":"5"},{"start":"fb","length":"1"},{"start":"fd","length":"4"},{"start":"103","length":"1"},{"start":"106","length":"4"},{"start":"10c","length":"1"},{"start":"10f","length":"6"},{"start":"116","length":"2"},{"start":"11c","length":"3"},{"start":"121","length":"1"},{"start":"123","length":"2"},{"start":"126","length":"1"},{"start":"128","length":"5"},{"start":"12e","length":"3"},{"start":"133","length":"4"},{"start":"139","length":"2"},{"start":"13c","length":"3"},{"start":"140","length":"1"},{"start":"143","length":"1"},{"start":"146","length":"1"},{"start":"148","length":"1"},{"start":"14c","length":"1"},{"start":"14e","length":"1"},{"start":"155","length":"2"},{"start":"15a","length":"4"},{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"168","length":"2"},{"start":"16b","length":"1"},{"start":"16f","length":"5"},{"start":"178","length":"4"},{"start":"17f","length":"1"},{"start":"185","length":"2"},{"start":"18a","length":"2"},{"start":"18e","length":"3"},{"start":"194","length":"4"},{"start":"19b","length":"5"},{"start":"1a1","length":"2"},{"start":"1a4","length":"1"},{"start":"1a8","length":"1"},{"start":"1ab","length":"1"},{"start":"1ae","length":"1"},{"start":"1b1","length":"2"},{"start":"1b7","length":"1"},{"start":"1b9","length":"2"},{"start":"1bc","length":"1"},{"start":"1bf","length":"2"},{"start":"1c3","length":"1"},{"start":"1c8","length":"1"},{"start":"1cd","length":"1"},{"start":"1d0","length":"2"},{"start":"1d5","length":"1"}]},{"pgid":"4.e","version":"538'39","reported_seq":"640","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.201023+0000","last_change":"2021-02-10T04:28:11.693702+0000","last_active":"2021-02-10T04:37:10.201023+0000","last_peered":"2021-02-10T04:37:10.201023+0000","last_clean":"2021-02-10T04:37:10.201023+0000","last_became_active":"2021-02-10T04:25:49.011093+0000","last_became_peered":"2021-02-10T04:25:49.011093+0000","last_unstale":"2021-02-10T04:37:10.201023+0000","last_undegraded":"2021-02-10T04:37:10.201023+0000","last_fullsized":"2021-02-10T04:37:10.201023+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'31","last_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:28:11.693616+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":56,"num_read_kb":40,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"560","reported_epoch":"615","state":"active+clean","last_fresh":"2021-02-10T04:37:13.195064+0000","last_change":"2021-02-10T04:29:31.683314+0000","last_active":"2021-02-10T04:37:13.195064+0000","last_peered":"2021-02-10T04:37:13.195064+0000","last_clean":"2021-02-10T04:37:13.195064+0000","last_became_active":"2021-02-10T04:29:07.160593+0000","last_became_peered":"2021-02-10T04:29:07.160593+0000","last_unstale":"2021-02-10T04:37:13.195064+0000","last_undegraded":"2021-02-10T04:37:13.195064+0000","last_fullsized":"2021-02-10T04:37:13.195064+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:31.683213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.683213+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"580","reported_epoch":"613","state":"active+clean","last_fresh":"2021-02-10T04:37:11.197162+0000","last_change":"2021-02-10T04:30:10.487631+0000","last_active":"2021-02-10T04:37:11.197162+0000","last_peered":"2021-02-10T04:37:11.197162+0000","last_clean":"2021-02-10T04:37:11.197162+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:37:11.197162+0000","last_undegraded":"2021-02-10T04:37:11.197162+0000","last_fullsized":"2021-02-10T04:37:11.197162+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:10.487528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:30:10.487528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"570","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.199003+0000","last_change":"2021-02-10T04:33:55.849675+0000","last_active":"2021-02-10T04:37:09.199003+0000","last_peered":"2021-02-10T04:37:09.199003+0000","last_clean":"2021-02-10T04:37:09.199003+0000","last_became_active":"2021-02-10T04:25:53.148054+0000","last_became_peered":"2021-02-10T04:25:53.148054+0000","last_unstale":"2021-02-10T04:37:09.199003+0000","last_undegraded":"2021-02-10T04:37:09.199003+0000","last_fullsized":"2021-02-10T04:37:09.199003+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:55.849582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:33:55.849582+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"615'569","reported_seq":"2274","reported_epoch":"615","state":"active+clean+snaptrim_wait","last_fresh":"2021-02-10T04:37:13.198240+0000","last_change":"2021-02-10T04:37:12.684415+0000","last_active":"2021-02-10T04:37:13.198240+0000","last_peered":"2021-02-10T04:37:13.198240+0000","last_clean":"2021-02-10T04:37:13.198240+0000","last_became_active":"2021-02-10T04:28:25.152030+0000","last_became_peered":"2021-02-10T04:28:25.152030+0000","last_unstale":"2021-02-10T04:37:13.198240+0000","last_undegraded":"2021-02-10T04:37:13.198240+0000","last_fullsized":"2021-02-10T04:37:13.198240+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'283","last_scrub_stamp":"2021-02-10T04:33:53.834627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:53.834627+0000","log_size":569,"ondisk_log_size":569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"stat_sum":{"num_bytes":43346881,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":328,"num_read_kb":340701,"num_write":235,"num_write_kb":103464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"},{"start":"d4","length":"8"},{"start":"dd","length":"1"},{"start":"e1","length":"3"},{"start":"e9","length":"2"},{"start":"ed","length":"4"},{"start":"f2","length":"1"},{"start":"f4","length":"5"},{"start":"fb","length":"1"},{"start":"fd","length":"4"},{"start":"103","length":"1"},{"start":"106","length":"4"},{"start":"10c","length":"1"},{"start":"10f","length":"6"},{"start":"116","length":"2"},{"start":"11c","length":"3"},{"start":"121","length":"1"},{"start":"123","length":"2"},{"start":"126","length":"1"},{"start":"128","length":"5"},{"start":"12e","length":"3"},{"start":"133","length":"4"},{"start":"139","length":"2"},{"start":"13c","length":"3"},{"start":"140","length":"1"},{"start":"143","length":"1"},{"start":"146","length":"1"},{"start":"148","length":"1"},{"start":"14c","length":"1"},{"start":"14e","length":"1"},{"start":"155","length":"2"},{"start":"15a","length":"4"},{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"168","length":"2"},{"start":"16b","length":"1"},{"start":"16f","length":"5"},{"start":"178","length":"4"},{"start":"17f","length":"1"},{"start":"185","length":"2"},{"start":"18a","length":"2"},{"start":"18e","length":"3"},{"start":"194","length":"4"},{"start":"19b","length":"5"},{"start":"1a1","length":"2"},{"start":"1a4","length":"1"},{"start":"1a8","length":"1"},{"start":"1ab","length":"1"},{"start":"1ae","length":"1"},{"start":"1b1","length":"2"},{"start":"1b7","length":"1"},{"start":"1b9","length":"2"},{"start":"1bc","length":"1"},{"start":"1bf","length":"2"},{"start":"1c3","length":"1"},{"start":"1c8","length":"1"},{"start":"1cd","length":"1"}]},{"pgid":"5.0","version":"158'3","reported_seq":"711","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.195951+0000","last_change":"2021-02-10T04:29:34.388978+0000","last_active":"2021-02-10T04:37:09.195951+0000","last_peered":"2021-02-10T04:37:09.195951+0000","last_clean":"2021-02-10T04:37:09.195951+0000","last_became_active":"2021-02-10T04:25:51.031906+0000","last_became_peered":"2021-02-10T04:25:51.031906+0000","last_unstale":"2021-02-10T04:37:09.195951+0000","last_undegraded":"2021-02-10T04:37:09.195951+0000","last_fullsized":"2021-02-10T04:37:09.195951+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'3","last_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_deep_scrub":"158'3","last_deep_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_clean_scrub_stamp":"2021-02-10T04:29:34.388912+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"538'40","reported_seq":"645","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.199925+0000","last_change":"2021-02-10T04:29:36.813190+0000","last_active":"2021-02-10T04:37:10.199925+0000","last_peered":"2021-02-10T04:37:10.199925+0000","last_clean":"2021-02-10T04:37:10.199925+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:37:10.199925+0000","last_undegraded":"2021-02-10T04:37:10.199925+0000","last_fullsized":"2021-02-10T04:37:10.199925+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:29:36.813083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.813083+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":72,"num_read_kb":52,"num_write":40,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"617","reported_epoch":"614","state":"active+clean","last_fresh":"2021-02-10T04:37:12.200510+0000","last_change":"2021-02-10T04:29:28.632178+0000","last_active":"2021-02-10T04:37:12.200510+0000","last_peered":"2021-02-10T04:37:12.200510+0000","last_clean":"2021-02-10T04:37:12.200510+0000","last_became_active":"2021-02-10T04:28:39.041833+0000","last_became_peered":"2021-02-10T04:28:39.041833+0000","last_unstale":"2021-02-10T04:37:12.200510+0000","last_undegraded":"2021-02-10T04:37:12.200510+0000","last_fullsized":"2021-02-10T04:37:12.200510+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.632085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.632085+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"501","reported_epoch":"614","state":"active+clean","last_fresh":"2021-02-10T04:37:12.202265+0000","last_change":"2021-02-10T04:29:24.658465+0000","last_active":"2021-02-10T04:37:12.202265+0000","last_peered":"2021-02-10T04:37:12.202265+0000","last_clean":"2021-02-10T04:37:12.202265+0000","last_became_active":"2021-02-10T04:29:07.148527+0000","last_became_peered":"2021-02-10T04:29:07.148527+0000","last_unstale":"2021-02-10T04:37:12.202265+0000","last_undegraded":"2021-02-10T04:37:12.202265+0000","last_fullsized":"2021-02-10T04:37:12.202265+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.658374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.658374+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"611'535","reported_seq":"2763","reported_epoch":"611","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:37:09.807481+0000","last_change":"2021-02-10T04:37:09.804668+0000","last_active":"2021-02-10T04:37:09.807481+0000","last_peered":"2021-02-10T04:37:09.807481+0000","last_clean":"2021-02-10T04:37:09.807481+0000","last_became_active":"2021-02-10T04:28:25.157306+0000","last_became_peered":"2021-02-10T04:28:25.157306+0000","last_unstale":"2021-02-10T04:37:09.807481+0000","last_undegraded":"2021-02-10T04:37:09.807481+0000","last_fullsized":"2021-02-10T04:37:09.807481+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":535,"ondisk_log_size":535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":24448410,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":313,"num_read_kb":299965,"num_write":209,"num_write_kb":61857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"},{"start":"d4","length":"8"},{"start":"dd","length":"1"},{"start":"e1","length":"3"},{"start":"e9","length":"2"},{"start":"ed","length":"4"},{"start":"f2","length":"1"},{"start":"f4","length":"5"},{"start":"fb","length":"1"},{"start":"fd","length":"4"},{"start":"103","length":"1"},{"start":"106","length":"4"},{"start":"10c","length":"1"},{"start":"10f","length":"6"},{"start":"116","length":"2"},{"start":"11c","length":"3"},{"start":"121","length":"1"},{"start":"123","length":"2"},{"start":"126","length":"1"},{"start":"128","length":"5"},{"start":"12e","length":"3"},{"start":"133","length":"4"},{"start":"139","length":"2"},{"start":"13c","length":"3"},{"start":"140","length":"1"},{"start":"143","length":"1"},{"start":"146","length":"1"},{"start":"148","length":"1"},{"start":"14c","length":"1"},{"start":"14e","length":"1"},{"start":"155","length":"2"},{"start":"15a","length":"4"},{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"168","length":"2"},{"start":"16b","length":"1"},{"start":"16f","length":"5"},{"start":"178","length":"4"},{"start":"17f","length":"1"},{"start":"185","length":"2"},{"start":"18a","length":"2"},{"start":"18e","length":"3"},{"start":"194","length":"4"},{"start":"19b","length":"5"},{"start":"1a1","length":"2"},{"start":"1a4","length":"1"},{"start":"1a8","length":"1"},{"start":"1ab","length":"1"},{"start":"1ae","length":"1"},{"start":"1b1","length":"1"},{"start":"1b7","length":"1"},{"start":"1b9","length":"2"},{"start":"1bc","length":"1"},{"start":"1bf","length":"2"},{"start":"1c3","length":"1"},{"start":"1c8","length":"1"},{"start":"1cd","length":"1"},{"start":"1d1","length":"1"},{"start":"1d5","length":"1"},{"start":"1d8","length":"2"},{"start":"1df","length":"1"},{"start":"1e1","length":"1"},{"start":"1e4","length":"1"},{"start":"1eb","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":"514","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.199068+0000","last_change":"2021-02-10T04:30:12.452072+0000","last_active":"2021-02-10T04:37:09.199068+0000","last_peered":"2021-02-10T04:37:09.199068+0000","last_clean":"2021-02-10T04:37:09.199068+0000","last_became_active":"2021-02-10T04:28:49.048525+0000","last_became_peered":"2021-02-10T04:28:49.048525+0000","last_unstale":"2021-02-10T04:37:09.199068+0000","last_undegraded":"2021-02-10T04:37:09.199068+0000","last_fullsized":"2021-02-10T04:37:09.199068+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:12.452000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:30:12.452000+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"574","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.200242+0000","last_change":"2021-02-10T04:29:40.859352+0000","last_active":"2021-02-10T04:37:10.200242+0000","last_peered":"2021-02-10T04:37:10.200242+0000","last_clean":"2021-02-10T04:37:10.200242+0000","last_became_active":"2021-02-10T04:25:51.018706+0000","last_became_peered":"2021-02-10T04:25:51.018706+0000","last_unstale":"2021-02-10T04:37:10.200242+0000","last_undegraded":"2021-02-10T04:37:10.200242+0000","last_fullsized":"2021-02-10T04:37:10.200242+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:40.859274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:40.859274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"538'60","reported_seq":"676","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.195886+0000","last_change":"2021-02-10T04:29:28.398522+0000","last_active":"2021-02-10T04:37:09.195886+0000","last_peered":"2021-02-10T04:37:09.195886+0000","last_clean":"2021-02-10T04:37:09.195886+0000","last_became_active":"2021-02-10T04:25:48.997481+0000","last_became_peered":"2021-02-10T04:25:48.997481+0000","last_unstale":"2021-02-10T04:37:09.195886+0000","last_undegraded":"2021-02-10T04:37:09.195886+0000","last_fullsized":"2021-02-10T04:37:09.195886+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'48","last_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_deep_scrub":"152'48","last_deep_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.398429+0000","log_size":60,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":88,"num_read_kb":66,"num_write":48,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"272","reported_epoch":"614","state":"active+clean","last_fresh":"2021-02-10T04:37:12.200052+0000","last_change":"2021-02-10T04:33:05.142730+0000","last_active":"2021-02-10T04:37:12.200052+0000","last_peered":"2021-02-10T04:37:12.200052+0000","last_clean":"2021-02-10T04:37:12.200052+0000","last_became_active":"2021-02-10T04:33:05.141066+0000","last_became_peered":"2021-02-10T04:33:05.141066+0000","last_unstale":"2021-02-10T04:37:12.200052+0000","last_undegraded":"2021-02-10T04:37:12.200052+0000","last_fullsized":"2021-02-10T04:37:12.200052+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:28:35.010709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:35.010709+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":423,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"572","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.199430+0000","last_change":"2021-02-10T04:28:05.641668+0000","last_active":"2021-02-10T04:37:10.199430+0000","last_peered":"2021-02-10T04:37:10.199430+0000","last_clean":"2021-02-10T04:37:10.199430+0000","last_became_active":"2021-02-10T04:25:53.045085+0000","last_became_peered":"2021-02-10T04:25:53.045085+0000","last_unstale":"2021-02-10T04:37:10.199430+0000","last_undegraded":"2021-02-10T04:37:10.199430+0000","last_fullsized":"2021-02-10T04:37:10.199430+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:05.641577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:05.641577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"612'971","reported_seq":"2615","reported_epoch":"612","state":"active+clean+snaptrim_wait","last_fresh":"2021-02-10T04:37:10.200789+0000","last_change":"2021-02-10T04:37:09.835262+0000","last_active":"2021-02-10T04:37:10.200789+0000","last_peered":"2021-02-10T04:37:10.200789+0000","last_clean":"2021-02-10T04:37:10.200789+0000","last_became_active":"2021-02-10T04:28:25.161622+0000","last_became_peered":"2021-02-10T04:28:25.161622+0000","last_unstale":"2021-02-10T04:37:10.200789+0000","last_undegraded":"2021-02-10T04:37:10.200789+0000","last_fullsized":"2021-02-10T04:37:10.200789+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":971,"ondisk_log_size":971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"stat_sum":{"num_bytes":51264490,"num_objects":22,"num_object_clones":18,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":570,"num_read_kb":605060,"num_write":415,"num_write_kb":149475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"},{"start":"d4","length":"8"},{"start":"dd","length":"1"},{"start":"e1","length":"3"},{"start":"e9","length":"2"},{"start":"ed","length":"4"},{"start":"f2","length":"1"},{"start":"f4","length":"5"},{"start":"fb","length":"1"},{"start":"fd","length":"4"},{"start":"103","length":"1"},{"start":"106","length":"4"},{"start":"10c","length":"1"},{"start":"10f","length":"6"},{"start":"116","length":"2"},{"start":"11c","length":"3"},{"start":"121","length":"1"},{"start":"123","length":"2"},{"start":"126","length":"1"},{"start":"128","length":"5"},{"start":"12e","length":"3"},{"start":"133","length":"4"},{"start":"139","length":"2"},{"start":"13c","length":"3"},{"start":"140","length":"1"},{"start":"143","length":"1"},{"start":"146","length":"1"},{"start":"148","length":"1"},{"start":"14c","length":"1"},{"start":"14e","length":"1"},{"start":"155","length":"2"},{"start":"15a","length":"4"},{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"168","length":"2"},{"start":"16b","length":"1"},{"start":"16f","length":"5"},{"start":"178","length":"4"},{"start":"17f","length":"1"},{"start":"185","length":"2"},{"start":"18a","length":"2"},{"start":"18e","length":"3"},{"start":"194","length":"4"},{"start":"19b","length":"5"},{"start":"1a1","length":"2"},{"start":"1a4","length":"1"},{"start":"1a8","length":"1"},{"start":"1ab","length":"1"},{"start":"1ae","length":"1"},{"start":"1b1","length":"1"},{"start":"1b7","length":"1"},{"start":"1b9","length":"2"},{"start":"1bc","length":"1"},{"start":"1bf","length":"2"},{"start":"1c3","length":"1"},{"start":"1c8","length":"1"},{"start":"1cd","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":"515","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.200957+0000","last_change":"2021-02-10T04:29:33.839409+0000","last_active":"2021-02-10T04:37:10.200957+0000","last_peered":"2021-02-10T04:37:10.200957+0000","last_clean":"2021-02-10T04:37:10.200957+0000","last_became_active":"2021-02-10T04:28:49.045505+0000","last_became_peered":"2021-02-10T04:28:49.045505+0000","last_unstale":"2021-02-10T04:37:10.200957+0000","last_undegraded":"2021-02-10T04:37:10.200957+0000","last_fullsized":"2021-02-10T04:37:10.200957+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:33.839335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:33.839335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"647","reported_epoch":"614","state":"active+clean","last_fresh":"2021-02-10T04:37:12.201871+0000","last_change":"2021-02-10T04:33:52.827328+0000","last_active":"2021-02-10T04:37:12.201871+0000","last_peered":"2021-02-10T04:37:12.201871+0000","last_clean":"2021-02-10T04:37:12.201871+0000","last_became_active":"2021-02-10T04:25:51.025636+0000","last_became_peered":"2021-02-10T04:25:51.025636+0000","last_unstale":"2021-02-10T04:37:12.201871+0000","last_undegraded":"2021-02-10T04:37:12.201871+0000","last_fullsized":"2021-02-10T04:37:12.201871+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:52.827257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:33:52.827257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"538'37","reported_seq":"649","reported_epoch":"615","state":"active+clean","last_fresh":"2021-02-10T04:37:13.196485+0000","last_change":"2021-02-10T04:28:36.754064+0000","last_active":"2021-02-10T04:37:13.196485+0000","last_peered":"2021-02-10T04:37:13.196485+0000","last_clean":"2021-02-10T04:37:13.196485+0000","last_became_active":"2021-02-10T04:25:49.002937+0000","last_became_peered":"2021-02-10T04:25:49.002937+0000","last_unstale":"2021-02-10T04:37:13.196485+0000","last_undegraded":"2021-02-10T04:37:13.196485+0000","last_fullsized":"2021-02-10T04:37:13.196485+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'27","last_scrub_stamp":"2021-02-10T04:28:36.753994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:36.753994+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":72,"num_read_kb":53,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"513","reported_epoch":"615","state":"active+clean","last_fresh":"2021-02-10T04:37:13.197903+0000","last_change":"2021-02-10T04:29:35.711635+0000","last_active":"2021-02-10T04:37:13.197903+0000","last_peered":"2021-02-10T04:37:13.197903+0000","last_clean":"2021-02-10T04:37:13.197903+0000","last_became_active":"2021-02-10T04:29:07.161341+0000","last_became_peered":"2021-02-10T04:29:07.161341+0000","last_unstale":"2021-02-10T04:37:13.197903+0000","last_undegraded":"2021-02-10T04:37:13.197903+0000","last_fullsized":"2021-02-10T04:37:13.197903+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_deep_scrub":"53'2","last_deep_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.711557+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"259","reported_epoch":"614","state":"active+clean","last_fresh":"2021-02-10T04:37:12.201022+0000","last_change":"2021-02-10T04:33:05.141227+0000","last_active":"2021-02-10T04:37:12.201022+0000","last_peered":"2021-02-10T04:37:12.201022+0000","last_clean":"2021-02-10T04:37:12.201022+0000","last_became_active":"2021-02-10T04:33:05.140699+0000","last_became_peered":"2021-02-10T04:33:05.140699+0000","last_unstale":"2021-02-10T04:37:12.201022+0000","last_undegraded":"2021-02-10T04:37:12.201022+0000","last_fullsized":"2021-02-10T04:37:12.201022+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:26.978062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.978062+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"614'644","reported_seq":"2269","reported_epoch":"614","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:37:12.543172+0000","last_change":"2021-02-10T04:37:12.538608+0000","last_active":"2021-02-10T04:37:12.543172+0000","last_peered":"2021-02-10T04:37:12.543172+0000","last_clean":"2021-02-10T04:37:12.543172+0000","last_became_active":"2021-02-10T04:28:25.146766+0000","last_became_peered":"2021-02-10T04:28:25.146766+0000","last_unstale":"2021-02-10T04:37:12.543172+0000","last_undegraded":"2021-02-10T04:37:12.543172+0000","last_fullsized":"2021-02-10T04:37:12.543172+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'357","last_scrub_stamp":"2021-02-10T04:33:50.582002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:50.582002+0000","log_size":644,"ondisk_log_size":644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":27955370,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":2,"num_read":266,"num_read_kb":239152,"num_write":246,"num_write_kb":103289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"},{"start":"d4","length":"8"},{"start":"dd","length":"1"},{"start":"e1","length":"3"},{"start":"e9","length":"2"},{"start":"ed","length":"4"},{"start":"f2","length":"1"},{"start":"f4","length":"5"},{"start":"fb","length":"1"},{"start":"fd","length":"4"},{"start":"103","length":"1"},{"start":"106","length":"4"},{"start":"10c","length":"1"},{"start":"10f","length":"6"},{"start":"116","length":"2"},{"start":"11c","length":"3"},{"start":"121","length":"1"},{"start":"123","length":"2"},{"start":"126","length":"1"},{"start":"128","length":"5"},{"start":"12e","length":"3"},{"start":"133","length":"4"},{"start":"139","length":"2"},{"start":"13c","length":"3"},{"start":"140","length":"1"},{"start":"143","length":"1"},{"start":"146","length":"1"},{"start":"148","length":"1"},{"start":"14c","length":"1"},{"start":"14e","length":"1"},{"start":"155","length":"2"},{"start":"15a","length":"4"},{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"168","length":"2"},{"start":"16b","length":"1"},{"start":"16f","length":"5"},{"start":"178","length":"4"},{"start":"17f","length":"1"},{"start":"185","length":"2"},{"start":"18a","length":"2"},{"start":"18e","length":"3"},{"start":"194","length":"4"},{"start":"19b","length":"5"},{"start":"1a1","length":"2"},{"start":"1a4","length":"1"},{"start":"1a8","length":"1"},{"start":"1ab","length":"1"},{"start":"1ae","length":"1"},{"start":"1b1","length":"2"},{"start":"1b7","length":"1"},{"start":"1b9","length":"2"},{"start":"1bc","length":"1"},{"start":"1bf","length":"2"},{"start":"1c3","length":"1"},{"start":"1c8","length":"1"},{"start":"1cd","length":"1"},{"start":"1d0","length":"2"},{"start":"1d5","length":"1"},{"start":"1d8","length":"2"},{"start":"1dc","length":"1"},{"start":"1df","length":"1"},{"start":"1e1","length":"1"},{"start":"1e4","length":"1"},{"start":"1eb","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":"598","reported_epoch":"614","state":"active+clean","last_fresh":"2021-02-10T04:37:12.199778+0000","last_change":"2021-02-10T04:29:38.793837+0000","last_active":"2021-02-10T04:37:12.199778+0000","last_peered":"2021-02-10T04:37:12.199778+0000","last_clean":"2021-02-10T04:37:12.199778+0000","last_became_active":"2021-02-10T04:28:49.054638+0000","last_became_peered":"2021-02-10T04:28:49.054638+0000","last_unstale":"2021-02-10T04:37:12.199778+0000","last_undegraded":"2021-02-10T04:37:12.199778+0000","last_fullsized":"2021-02-10T04:37:12.199778+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:38.793760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.793760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"575","reported_epoch":"613","state":"active+clean","last_fresh":"2021-02-10T04:37:11.197305+0000","last_change":"2021-02-10T04:29:40.451948+0000","last_active":"2021-02-10T04:37:11.197305+0000","last_peered":"2021-02-10T04:37:11.197305+0000","last_clean":"2021-02-10T04:37:11.197305+0000","last_became_active":"2021-02-10T04:25:51.032201+0000","last_became_peered":"2021-02-10T04:25:51.032201+0000","last_unstale":"2021-02-10T04:37:11.197305+0000","last_undegraded":"2021-02-10T04:37:11.197305+0000","last_fullsized":"2021-02-10T04:37:11.197305+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:40.451832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:29:40.451832+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"538'28","reported_seq":"628","reported_epoch":"615","state":"active+clean","last_fresh":"2021-02-10T04:37:13.198425+0000","last_change":"2021-02-10T04:33:55.288118+0000","last_active":"2021-02-10T04:37:13.198425+0000","last_peered":"2021-02-10T04:37:13.198425+0000","last_clean":"2021-02-10T04:37:13.198425+0000","last_became_active":"2021-02-10T04:25:49.003521+0000","last_became_peered":"2021-02-10T04:25:49.003521+0000","last_unstale":"2021-02-10T04:37:13.198425+0000","last_undegraded":"2021-02-10T04:37:13.198425+0000","last_fullsized":"2021-02-10T04:37:13.198425+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'16","last_scrub_stamp":"2021-02-10T04:33:55.288033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:55.288033+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":34,"num_write":28,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"316","reported_epoch":"615","state":"active+clean","last_fresh":"2021-02-10T04:37:13.195232+0000","last_change":"2021-02-10T04:33:05.166788+0000","last_active":"2021-02-10T04:37:13.195232+0000","last_peered":"2021-02-10T04:37:13.195232+0000","last_clean":"2021-02-10T04:37:13.195232+0000","last_became_active":"2021-02-10T04:33:05.166368+0000","last_became_peered":"2021-02-10T04:33:05.166368+0000","last_unstale":"2021-02-10T04:37:13.195232+0000","last_undegraded":"2021-02-10T04:37:13.195232+0000","last_fullsized":"2021-02-10T04:37:13.195232+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:14.204533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:30:14.204533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"579","reported_epoch":"615","state":"active+clean","last_fresh":"2021-02-10T04:37:13.196127+0000","last_change":"2021-02-10T04:29:21.620573+0000","last_active":"2021-02-10T04:37:13.196127+0000","last_peered":"2021-02-10T04:37:13.196127+0000","last_clean":"2021-02-10T04:37:13.196127+0000","last_became_active":"2021-02-10T04:25:49.001782+0000","last_became_peered":"2021-02-10T04:25:49.001782+0000","last_unstale":"2021-02-10T04:37:13.196127+0000","last_undegraded":"2021-02-10T04:37:13.196127+0000","last_fullsized":"2021-02-10T04:37:13.196127+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:21.620482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:21.620482+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"258","reported_epoch":"613","state":"active+clean","last_fresh":"2021-02-10T04:37:11.197204+0000","last_change":"2021-02-10T04:33:05.153205+0000","last_active":"2021-02-10T04:37:11.197204+0000","last_peered":"2021-02-10T04:37:11.197204+0000","last_clean":"2021-02-10T04:37:11.197204+0000","last_became_active":"2021-02-10T04:33:05.152845+0000","last_became_peered":"2021-02-10T04:33:05.152845+0000","last_unstale":"2021-02-10T04:37:11.197204+0000","last_undegraded":"2021-02-10T04:37:11.197204+0000","last_fullsized":"2021-02-10T04:37:11.197204+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.098681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.098681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.5","version":"215'85","reported_seq":"1612","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.201710+0000","last_change":"2021-02-10T04:37:10.202069+0000","last_active":"2021-02-10T04:37:10.201710+0000","last_peered":"2021-02-10T04:37:10.201710+0000","last_clean":"2021-02-10T04:37:10.201710+0000","last_became_active":"2021-02-10T04:29:07.163659+0000","last_became_peered":"2021-02-10T04:29:07.163659+0000","last_unstale":"2021-02-10T04:37:10.201710+0000","last_undegraded":"2021-02-10T04:37:10.201710+0000","last_fullsized":"2021-02-10T04:37:10.201710+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2201750,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":44,"num_read_kb":45112,"num_write":42,"num_write_kb":10011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"},{"start":"d4","length":"8"},{"start":"dd","length":"1"},{"start":"e1","length":"3"},{"start":"e9","length":"2"},{"start":"ed","length":"4"},{"start":"f2","length":"1"},{"start":"f4","length":"5"},{"start":"fb","length":"1"},{"start":"fd","length":"4"},{"start":"103","length":"1"},{"start":"106","length":"4"},{"start":"10c","length":"1"},{"start":"10f","length":"6"},{"start":"116","length":"2"},{"start":"11c","length":"3"},{"start":"121","length":"1"},{"start":"123","length":"2"},{"start":"126","length":"1"},{"start":"128","length":"5"},{"start":"12e","length":"3"},{"start":"133","length":"4"},{"start":"139","length":"2"},{"start":"13c","length":"3"},{"start":"140","length":"1"},{"start":"143","length":"1"},{"start":"146","length":"1"},{"start":"148","length":"1"},{"start":"14c","length":"1"},{"start":"14e","length":"1"},{"start":"155","length":"2"},{"start":"15a","length":"4"},{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"168","length":"2"},{"start":"16b","length":"1"},{"start":"16f","length":"5"},{"start":"178","length":"4"},{"start":"17f","length":"1"},{"start":"185","length":"2"},{"start":"18a","length":"2"},{"start":"18e","length":"3"},{"start":"194","length":"4"},{"start":"19b","length":"5"},{"start":"1a1","length":"2"},{"start":"1a4","length":"1"},{"start":"1a8","length":"1"},{"start":"1ab","length":"1"},{"start":"1ae","length":"1"},{"start":"1b1","length":"2"},{"start":"1b7","length":"1"},{"start":"1b9","length":"2"},{"start":"1bc","length":"1"},{"start":"1bf","length":"2"},{"start":"1c3","length":"1"},{"start":"1c8","length":"1"},{"start":"1cd","length":"1"},{"start":"1d0","length":"2"},{"start":"1d5","length":"1"},{"start":"1d8","length":"2"},{"start":"1dc","length":"1"},{"start":"1df","length":"1"},{"start":"1e1","length":"1"},{"start":"1e4","length":"1"},{"start":"1eb","length":"1"},{"start":"1ef","length":"1"}]},{"pgid":"1.3","version":"0'0","reported_seq":"338","reported_epoch":"614","state":"active+clean","last_fresh":"2021-02-10T04:37:12.198275+0000","last_change":"2021-02-10T04:33:05.141311+0000","last_active":"2021-02-10T04:37:12.198275+0000","last_peered":"2021-02-10T04:37:12.198275+0000","last_clean":"2021-02-10T04:37:12.198275+0000","last_became_active":"2021-02-10T04:33:05.140966+0000","last_became_peered":"2021-02-10T04:33:05.140966+0000","last_unstale":"2021-02-10T04:37:12.198275+0000","last_undegraded":"2021-02-10T04:37:12.198275+0000","last_fullsized":"2021-02-10T04:37:12.198275+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.928906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.928906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"498","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.199972+0000","last_change":"2021-02-10T04:29:29.805554+0000","last_active":"2021-02-10T04:37:09.199972+0000","last_peered":"2021-02-10T04:37:09.199972+0000","last_clean":"2021-02-10T04:37:09.199972+0000","last_became_active":"2021-02-10T04:29:07.148512+0000","last_became_peered":"2021-02-10T04:29:07.148512+0000","last_unstale":"2021-02-10T04:37:09.199972+0000","last_undegraded":"2021-02-10T04:37:09.199972+0000","last_fullsized":"2021-02-10T04:37:09.199972+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.805432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.805432+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"538'27","reported_seq":"625","reported_epoch":"614","state":"active+clean","last_fresh":"2021-02-10T04:37:12.199094+0000","last_change":"2021-02-10T04:33:56.837896+0000","last_active":"2021-02-10T04:37:12.199094+0000","last_peered":"2021-02-10T04:37:12.199094+0000","last_clean":"2021-02-10T04:37:12.199094+0000","last_became_active":"2021-02-10T04:25:49.002471+0000","last_became_peered":"2021-02-10T04:25:49.002471+0000","last_unstale":"2021-02-10T04:37:12.199094+0000","last_undegraded":"2021-02-10T04:37:12.199094+0000","last_fullsized":"2021-02-10T04:37:12.199094+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:33:56.837816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:56.837816+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"582","reported_epoch":"615","state":"active+clean","last_fresh":"2021-02-10T04:37:13.195108+0000","last_change":"2021-02-10T04:29:36.663903+0000","last_active":"2021-02-10T04:37:13.195108+0000","last_peered":"2021-02-10T04:37:13.195108+0000","last_clean":"2021-02-10T04:37:13.195108+0000","last_became_active":"2021-02-10T04:29:07.158075+0000","last_became_peered":"2021-02-10T04:29:07.158075+0000","last_unstale":"2021-02-10T04:37:13.195108+0000","last_undegraded":"2021-02-10T04:37:13.195108+0000","last_fullsized":"2021-02-10T04:37:13.195108+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.663840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.663840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"600","reported_epoch":"613","state":"active+clean","last_fresh":"2021-02-10T04:37:11.196115+0000","last_change":"2021-02-10T04:28:34.482961+0000","last_active":"2021-02-10T04:37:11.196115+0000","last_peered":"2021-02-10T04:37:11.196115+0000","last_clean":"2021-02-10T04:37:11.196115+0000","last_became_active":"2021-02-10T04:28:21.123027+0000","last_became_peered":"2021-02-10T04:28:21.123027+0000","last_unstale":"2021-02-10T04:37:11.196115+0000","last_undegraded":"2021-02-10T04:37:11.196115+0000","last_fullsized":"2021-02-10T04:37:11.196115+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:34.482906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:34.482906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"574","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.198020+0000","last_change":"2021-02-10T04:29:35.382459+0000","last_active":"2021-02-10T04:37:09.198020+0000","last_peered":"2021-02-10T04:37:09.198020+0000","last_clean":"2021-02-10T04:37:09.198020+0000","last_became_active":"2021-02-10T04:25:49.010614+0000","last_became_peered":"2021-02-10T04:25:49.010614+0000","last_unstale":"2021-02-10T04:37:09.198020+0000","last_undegraded":"2021-02-10T04:37:09.198020+0000","last_fullsized":"2021-02-10T04:37:09.198020+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:35.382341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.382341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"573","reported_epoch":"613","state":"active+clean","last_fresh":"2021-02-10T04:37:11.197406+0000","last_change":"2021-02-10T04:28:38.503398+0000","last_active":"2021-02-10T04:37:11.197406+0000","last_peered":"2021-02-10T04:37:11.197406+0000","last_clean":"2021-02-10T04:37:11.197406+0000","last_became_active":"2021-02-10T04:25:53.037506+0000","last_became_peered":"2021-02-10T04:25:53.037506+0000","last_unstale":"2021-02-10T04:37:11.197406+0000","last_undegraded":"2021-02-10T04:37:11.197406+0000","last_fullsized":"2021-02-10T04:37:11.197406+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:38.503309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:38.503309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"615'232","reported_seq":"1925","reported_epoch":"615","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:37:13.634443+0000","last_change":"2021-02-10T04:37:13.632983+0000","last_active":"2021-02-10T04:37:13.634443+0000","last_peered":"2021-02-10T04:37:13.634443+0000","last_clean":"2021-02-10T04:37:13.634443+0000","last_became_active":"2021-02-10T04:28:25.150109+0000","last_became_peered":"2021-02-10T04:28:25.150109+0000","last_unstale":"2021-02-10T04:37:13.634443+0000","last_undegraded":"2021-02-10T04:37:13.634443+0000","last_fullsized":"2021-02-10T04:37:13.634443+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'36","last_scrub_stamp":"2021-02-10T04:30:13.863430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:13.863430+0000","log_size":232,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"stat_sum":{"num_bytes":11268031,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":163,"num_read_kb":158919,"num_write":101,"num_write_kb":33987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"},{"start":"d4","length":"8"},{"start":"dd","length":"1"},{"start":"e1","length":"3"},{"start":"e9","length":"2"},{"start":"ed","length":"4"},{"start":"f2","length":"1"},{"start":"f4","length":"5"},{"start":"fb","length":"1"},{"start":"fd","length":"4"},{"start":"103","length":"1"},{"start":"106","length":"4"},{"start":"10c","length":"1"},{"start":"10f","length":"6"},{"start":"116","length":"2"},{"start":"11c","length":"3"},{"start":"121","length":"1"},{"start":"123","length":"2"},{"start":"126","length":"1"},{"start":"128","length":"5"},{"start":"12e","length":"3"},{"start":"133","length":"4"},{"start":"139","length":"2"},{"start":"13c","length":"3"},{"start":"140","length":"1"},{"start":"143","length":"1"},{"start":"146","length":"1"},{"start":"148","length":"1"},{"start":"14c","length":"1"},{"start":"14e","length":"1"},{"start":"155","length":"2"},{"start":"15a","length":"4"},{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"168","length":"2"},{"start":"16b","length":"1"},{"start":"16f","length":"5"},{"start":"178","length":"4"},{"start":"17f","length":"1"},{"start":"185","length":"2"},{"start":"18a","length":"2"},{"start":"18e","length":"3"},{"start":"194","length":"4"},{"start":"19b","length":"5"},{"start":"1a1","length":"2"},{"start":"1a4","length":"1"},{"start":"1a8","length":"1"},{"start":"1ab","length":"1"},{"start":"1ae","length":"1"},{"start":"1b1","length":"2"},{"start":"1b7","length":"1"},{"start":"1b9","length":"2"},{"start":"1bc","length":"1"},{"start":"1bf","length":"2"},{"start":"1c3","length":"1"},{"start":"1c8","length":"1"},{"start":"1cd","length":"1"},{"start":"1d1","length":"1"},{"start":"1d5","length":"1"},{"start":"1d8","length":"2"},{"start":"1df","length":"1"},{"start":"1e1","length":"1"},{"start":"1e4","length":"1"},{"start":"1eb","length":"1"}]},{"pgid":"5.6","version":"0'0","reported_seq":"572","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.200379+0000","last_change":"2021-02-10T04:29:23.779839+0000","last_active":"2021-02-10T04:37:09.200379+0000","last_peered":"2021-02-10T04:37:09.200379+0000","last_clean":"2021-02-10T04:37:09.200379+0000","last_became_active":"2021-02-10T04:25:51.025306+0000","last_became_peered":"2021-02-10T04:25:51.025306+0000","last_unstale":"2021-02-10T04:37:09.200379+0000","last_undegraded":"2021-02-10T04:37:09.200379+0000","last_fullsized":"2021-02-10T04:37:09.200379+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.779767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.779767+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"538'33","reported_seq":"346","reported_epoch":"613","state":"active+clean","last_fresh":"2021-02-10T04:37:11.197079+0000","last_change":"2021-02-10T04:33:05.147122+0000","last_active":"2021-02-10T04:37:11.197079+0000","last_peered":"2021-02-10T04:37:11.197079+0000","last_clean":"2021-02-10T04:37:11.197079+0000","last_became_active":"2021-02-10T04:33:05.146695+0000","last_became_peered":"2021-02-10T04:33:05.146695+0000","last_unstale":"2021-02-10T04:37:11.197079+0000","last_undegraded":"2021-02-10T04:37:11.197079+0000","last_fullsized":"2021-02-10T04:37:11.197079+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'29","last_scrub_stamp":"2021-02-10T04:29:25.974280+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.974280+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":38,"num_write":28,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"595","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.199476+0000","last_change":"2021-02-10T04:28:39.690493+0000","last_active":"2021-02-10T04:37:09.199476+0000","last_peered":"2021-02-10T04:37:09.199476+0000","last_clean":"2021-02-10T04:37:09.199476+0000","last_became_active":"2021-02-10T04:28:21.120083+0000","last_became_peered":"2021-02-10T04:28:21.120083+0000","last_unstale":"2021-02-10T04:37:09.199476+0000","last_undegraded":"2021-02-10T04:37:09.199476+0000","last_fullsized":"2021-02-10T04:37:09.199476+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:39.690436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:39.690436+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"524","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.199444+0000","last_change":"2021-02-10T04:33:05.142561+0000","last_active":"2021-02-10T04:37:10.199444+0000","last_peered":"2021-02-10T04:37:10.199444+0000","last_clean":"2021-02-10T04:37:10.199444+0000","last_became_active":"2021-02-10T04:33:05.141816+0000","last_became_peered":"2021-02-10T04:33:05.141816+0000","last_unstale":"2021-02-10T04:37:10.199444+0000","last_undegraded":"2021-02-10T04:37:10.199444+0000","last_fullsized":"2021-02-10T04:37:10.199444+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:28:50.992899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:50.992899+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":252,"num_read_kb":247,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"501","reported_epoch":"614","state":"active+clean","last_fresh":"2021-02-10T04:37:12.199250+0000","last_change":"2021-02-10T04:29:33.747527+0000","last_active":"2021-02-10T04:37:12.199250+0000","last_peered":"2021-02-10T04:37:12.199250+0000","last_clean":"2021-02-10T04:37:12.199250+0000","last_became_active":"2021-02-10T04:29:07.158869+0000","last_became_peered":"2021-02-10T04:29:07.158869+0000","last_unstale":"2021-02-10T04:37:12.199250+0000","last_undegraded":"2021-02-10T04:37:12.199250+0000","last_fullsized":"2021-02-10T04:37:12.199250+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:33.747448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:33.747448+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.4","version":"613'386","reported_seq":"2758","reported_epoch":"615","state":"active+clean+snaptrim_wait","last_fresh":"2021-02-10T04:37:13.632947+0000","last_change":"2021-02-10T04:37:13.632947+0000","last_active":"2021-02-10T04:37:13.632947+0000","last_peered":"2021-02-10T04:37:13.632947+0000","last_clean":"2021-02-10T04:37:13.632947+0000","last_became_active":"2021-02-10T04:28:25.151037+0000","last_became_peered":"2021-02-10T04:28:25.151037+0000","last_unstale":"2021-02-10T04:37:13.632947+0000","last_undegraded":"2021-02-10T04:37:13.632947+0000","last_fullsized":"2021-02-10T04:37:13.632947+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'239","last_scrub_stamp":"2021-02-10T04:33:46.270478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:46.270478+0000","log_size":386,"ondisk_log_size":386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"stat_sum":{"num_bytes":17436311,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":213,"num_read_kb":195255,"num_write":188,"num_write_kb":68465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"},{"start":"d4","length":"8"},{"start":"dd","length":"1"},{"start":"e1","length":"3"},{"start":"e9","length":"2"},{"start":"ed","length":"4"},{"start":"f2","length":"1"},{"start":"f4","length":"5"},{"start":"fb","length":"1"},{"start":"fd","length":"4"},{"start":"103","length":"1"},{"start":"106","length":"4"},{"start":"10c","length":"1"},{"start":"10f","length":"6"},{"start":"116","length":"2"},{"start":"11c","length":"3"},{"start":"121","length":"1"},{"start":"123","length":"2"},{"start":"126","length":"1"},{"start":"128","length":"5"},{"start":"12e","length":"3"},{"start":"133","length":"4"},{"start":"139","length":"2"},{"start":"13c","length":"3"},{"start":"140","length":"1"},{"start":"143","length":"1"},{"start":"146","length":"1"},{"start":"148","length":"1"},{"start":"14c","length":"1"},{"start":"14e","length":"1"},{"start":"155","length":"2"},{"start":"15a","length":"4"},{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"168","length":"2"},{"start":"16b","length":"1"},{"start":"16f","length":"5"},{"start":"178","length":"4"},{"start":"17f","length":"1"},{"start":"185","length":"2"},{"start":"18a","length":"2"},{"start":"18e","length":"3"},{"start":"194","length":"4"},{"start":"19b","length":"5"},{"start":"1a1","length":"2"},{"start":"1a4","length":"1"},{"start":"1a8","length":"1"},{"start":"1ab","length":"1"},{"start":"1ae","length":"2"},{"start":"1b1","length":"2"},{"start":"1b7","length":"1"},{"start":"1b9","length":"2"},{"start":"1bc","length":"1"},{"start":"1bf","length":"2"},{"start":"1c3","length":"1"},{"start":"1c8","length":"1"},{"start":"1cd","length":"1"},{"start":"1d1","length":"1"},{"start":"1d5","length":"1"},{"start":"1d8","length":"2"},{"start":"1df","length":"1"},{"start":"1e1","length":"1"},{"start":"1e4","length":"1"}]},{"pgid":"1.2","version":"0'0","reported_seq":"592","reported_epoch":"615","state":"active+clean","last_fresh":"2021-02-10T04:37:13.196339+0000","last_change":"2021-02-10T04:29:29.683867+0000","last_active":"2021-02-10T04:37:13.196339+0000","last_peered":"2021-02-10T04:37:13.196339+0000","last_clean":"2021-02-10T04:37:13.196339+0000","last_became_active":"2021-02-10T04:29:07.159607+0000","last_became_peered":"2021-02-10T04:29:07.159607+0000","last_unstale":"2021-02-10T04:37:13.196339+0000","last_undegraded":"2021-02-10T04:37:13.196339+0000","last_fullsized":"2021-02-10T04:37:13.196339+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.683805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.683805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"577","reported_epoch":"615","state":"active+clean","last_fresh":"2021-02-10T04:37:13.197983+0000","last_change":"2021-02-10T04:28:43.740930+0000","last_active":"2021-02-10T04:37:13.197983+0000","last_peered":"2021-02-10T04:37:13.197983+0000","last_clean":"2021-02-10T04:37:13.197983+0000","last_became_active":"2021-02-10T04:25:51.027624+0000","last_became_peered":"2021-02-10T04:25:51.027624+0000","last_unstale":"2021-02-10T04:37:13.197983+0000","last_undegraded":"2021-02-10T04:37:13.197983+0000","last_fullsized":"2021-02-10T04:37:13.197983+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:43.740853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:28:43.740853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"538'43","reported_seq":"355","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.200135+0000","last_change":"2021-02-10T04:33:49.423179+0000","last_active":"2021-02-10T04:37:09.200135+0000","last_peered":"2021-02-10T04:37:09.200135+0000","last_clean":"2021-02-10T04:37:09.200135+0000","last_became_active":"2021-02-10T04:33:05.151701+0000","last_became_peered":"2021-02-10T04:33:05.151701+0000","last_unstale":"2021-02-10T04:37:09.200135+0000","last_undegraded":"2021-02-10T04:37:09.200135+0000","last_fullsized":"2021-02-10T04:37:09.200135+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'31","last_scrub_stamp":"2021-02-10T04:33:49.423083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:49.423083+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":82,"num_read_kb":58,"num_write":46,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"593","reported_epoch":"613","state":"active+clean","last_fresh":"2021-02-10T04:37:11.196686+0000","last_change":"2021-02-10T04:28:36.491041+0000","last_active":"2021-02-10T04:37:11.196686+0000","last_peered":"2021-02-10T04:37:11.196686+0000","last_clean":"2021-02-10T04:37:11.196686+0000","last_became_active":"2021-02-10T04:28:21.125541+0000","last_became_peered":"2021-02-10T04:28:21.125541+0000","last_unstale":"2021-02-10T04:37:11.196686+0000","last_undegraded":"2021-02-10T04:37:11.196686+0000","last_fullsized":"2021-02-10T04:37:11.196686+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:36.490961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:36.490961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"570","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.198019+0000","last_change":"2021-02-10T04:29:36.396044+0000","last_active":"2021-02-10T04:37:09.198019+0000","last_peered":"2021-02-10T04:37:09.198019+0000","last_clean":"2021-02-10T04:37:09.198019+0000","last_became_active":"2021-02-10T04:25:53.049487+0000","last_became_peered":"2021-02-10T04:25:53.049487+0000","last_unstale":"2021-02-10T04:37:09.198019+0000","last_undegraded":"2021-02-10T04:37:09.198019+0000","last_fullsized":"2021-02-10T04:37:09.198019+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.395943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.395943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"615'1273","reported_seq":"6461","reported_epoch":"615","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:37:13.641836+0000","last_change":"2021-02-10T04:37:11.633439+0000","last_active":"2021-02-10T04:37:13.641836+0000","last_peered":"2021-02-10T04:37:13.641836+0000","last_clean":"2021-02-10T04:37:13.641836+0000","last_became_active":"2021-02-10T04:28:25.147959+0000","last_became_peered":"2021-02-10T04:28:25.147959+0000","last_unstale":"2021-02-10T04:37:13.641836+0000","last_undegraded":"2021-02-10T04:37:13.641836+0000","last_fullsized":"2021-02-10T04:37:13.641836+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'223","last_scrub_stamp":"2021-02-10T04:30:14.478639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:14.478639+0000","log_size":1273,"ondisk_log_size":1273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":25,"stat_sum":{"num_bytes":71796600,"num_objects":36,"num_object_clones":29,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":2,"num_read":714,"num_read_kb":674442,"num_write":574,"num_write_kb":233959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"},{"start":"d4","length":"8"},{"start":"dd","length":"1"},{"start":"e1","length":"3"},{"start":"e9","length":"2"},{"start":"ed","length":"4"},{"start":"f2","length":"1"},{"start":"f4","length":"5"},{"start":"fb","length":"1"},{"start":"fd","length":"4"},{"start":"103","length":"1"},{"start":"106","length":"4"},{"start":"10c","length":"1"},{"start":"10f","length":"6"},{"start":"116","length":"2"},{"start":"11c","length":"3"},{"start":"121","length":"1"},{"start":"123","length":"2"},{"start":"126","length":"1"},{"start":"128","length":"5"},{"start":"12e","length":"3"},{"start":"133","length":"4"},{"start":"139","length":"2"},{"start":"13c","length":"3"},{"start":"140","length":"1"},{"start":"143","length":"1"},{"start":"146","length":"1"},{"start":"148","length":"1"},{"start":"14c","length":"1"},{"start":"14e","length":"1"},{"start":"155","length":"2"},{"start":"15a","length":"4"},{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"168","length":"2"},{"start":"16b","length":"1"},{"start":"16f","length":"5"},{"start":"178","length":"4"},{"start":"17f","length":"1"},{"start":"185","length":"2"},{"start":"18a","length":"2"},{"start":"18e","length":"3"},{"start":"194","length":"4"},{"start":"19b","length":"5"},{"start":"1a1","length":"2"},{"start":"1a4","length":"1"},{"start":"1a8","length":"1"},{"start":"1ab","length":"1"},{"start":"1ae","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":"516","reported_epoch":"613","state":"active+clean","last_fresh":"2021-02-10T04:37:11.196760+0000","last_change":"2021-02-10T04:29:39.434301+0000","last_active":"2021-02-10T04:37:11.196760+0000","last_peered":"2021-02-10T04:37:11.196760+0000","last_clean":"2021-02-10T04:37:11.196760+0000","last_became_active":"2021-02-10T04:28:49.054267+0000","last_became_peered":"2021-02-10T04:28:49.054267+0000","last_unstale":"2021-02-10T04:37:11.196760+0000","last_undegraded":"2021-02-10T04:37:11.196760+0000","last_fullsized":"2021-02-10T04:37:11.196760+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:39.434210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:39.434210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"58'2","reported_seq":"716","reported_epoch":"615","state":"active+clean","last_fresh":"2021-02-10T04:37:13.196799+0000","last_change":"2021-02-10T04:28:33.656576+0000","last_active":"2021-02-10T04:37:13.196799+0000","last_peered":"2021-02-10T04:37:13.196799+0000","last_clean":"2021-02-10T04:37:13.196799+0000","last_became_active":"2021-02-10T04:25:51.026251+0000","last_became_peered":"2021-02-10T04:25:51.026251+0000","last_unstale":"2021-02-10T04:37:13.196799+0000","last_undegraded":"2021-02-10T04:37:13.196799+0000","last_fullsized":"2021-02-10T04:37:13.196799+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:28:33.656509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:33.656509+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"538'44","reported_seq":"348","reported_epoch":"613","state":"active+clean","last_fresh":"2021-02-10T04:37:11.196488+0000","last_change":"2021-02-10T04:33:05.142280+0000","last_active":"2021-02-10T04:37:11.196488+0000","last_peered":"2021-02-10T04:37:11.196488+0000","last_clean":"2021-02-10T04:37:11.196488+0000","last_became_active":"2021-02-10T04:33:05.141901+0000","last_became_peered":"2021-02-10T04:33:05.141901+0000","last_unstale":"2021-02-10T04:37:11.196488+0000","last_undegraded":"2021-02-10T04:37:11.196488+0000","last_fullsized":"2021-02-10T04:37:11.196488+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'34","last_scrub_stamp":"2021-02-10T04:28:49.959162+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:28:49.959162+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":60,"num_read_kb":42,"num_write":32,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"333","reported_epoch":"615","state":"active+clean","last_fresh":"2021-02-10T04:37:13.197245+0000","last_change":"2021-02-10T04:33:05.142215+0000","last_active":"2021-02-10T04:37:13.197245+0000","last_peered":"2021-02-10T04:37:13.197245+0000","last_clean":"2021-02-10T04:37:13.197245+0000","last_became_active":"2021-02-10T04:33:05.141852+0000","last_became_peered":"2021-02-10T04:33:05.141852+0000","last_unstale":"2021-02-10T04:37:13.197245+0000","last_undegraded":"2021-02-10T04:37:13.197245+0000","last_fullsized":"2021-02-10T04:37:13.197245+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:34.053323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:34.053323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"575","reported_epoch":"615","state":"active+clean","last_fresh":"2021-02-10T04:37:13.195023+0000","last_change":"2021-02-10T04:28:42.701993+0000","last_active":"2021-02-10T04:37:13.195023+0000","last_peered":"2021-02-10T04:37:13.195023+0000","last_clean":"2021-02-10T04:37:13.195023+0000","last_became_active":"2021-02-10T04:25:53.045215+0000","last_became_peered":"2021-02-10T04:25:53.045215+0000","last_unstale":"2021-02-10T04:37:13.195023+0000","last_undegraded":"2021-02-10T04:37:13.195023+0000","last_fullsized":"2021-02-10T04:37:13.195023+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:42.701924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:42.701924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"293'77","reported_seq":"1618","reported_epoch":"613","state":"active+clean","last_fresh":"2021-02-10T04:37:11.214442+0000","last_change":"2021-02-10T04:37:11.214788+0000","last_active":"2021-02-10T04:37:11.214442+0000","last_peered":"2021-02-10T04:37:11.214442+0000","last_clean":"2021-02-10T04:37:11.214442+0000","last_became_active":"2021-02-10T04:28:25.154434+0000","last_became_peered":"2021-02-10T04:28:25.154434+0000","last_unstale":"2021-02-10T04:37:11.214442+0000","last_undegraded":"2021-02-10T04:37:11.214442+0000","last_fullsized":"2021-02-10T04:37:11.214442+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7546343,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":74349,"num_write":37,"num_write_kb":13263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"},{"start":"d4","length":"8"},{"start":"dd","length":"1"},{"start":"e1","length":"3"},{"start":"e9","length":"2"},{"start":"ed","length":"4"},{"start":"f2","length":"1"},{"start":"f4","length":"5"},{"start":"fb","length":"1"},{"start":"fd","length":"4"},{"start":"103","length":"1"},{"start":"106","length":"4"},{"start":"10c","length":"1"},{"start":"10f","length":"6"},{"start":"116","length":"2"},{"start":"11c","length":"3"},{"start":"121","length":"1"},{"start":"123","length":"2"},{"start":"126","length":"1"},{"start":"128","length":"5"},{"start":"12e","length":"3"},{"start":"133","length":"4"},{"start":"139","length":"2"},{"start":"13c","length":"3"},{"start":"140","length":"1"},{"start":"143","length":"1"},{"start":"146","length":"1"},{"start":"148","length":"1"},{"start":"14c","length":"1"},{"start":"14e","length":"1"},{"start":"155","length":"2"},{"start":"15a","length":"4"},{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"168","length":"2"},{"start":"16b","length":"1"},{"start":"16f","length":"5"},{"start":"178","length":"4"},{"start":"17f","length":"1"},{"start":"185","length":"2"},{"start":"18a","length":"2"},{"start":"18e","length":"3"},{"start":"194","length":"4"},{"start":"19b","length":"5"},{"start":"1a1","length":"2"},{"start":"1a4","length":"1"},{"start":"1a8","length":"1"},{"start":"1ab","length":"1"},{"start":"1ae","length":"2"},{"start":"1b1","length":"2"},{"start":"1b7","length":"1"},{"start":"1b9","length":"2"},{"start":"1bc","length":"1"},{"start":"1bf","length":"2"},{"start":"1c3","length":"1"},{"start":"1c8","length":"1"},{"start":"1cd","length":"1"},{"start":"1d0","length":"2"},{"start":"1d5","length":"1"},{"start":"1d8","length":"2"},{"start":"1dc","length":"1"},{"start":"1df","length":"1"},{"start":"1e1","length":"1"},{"start":"1e4","length":"1"},{"start":"1eb","length":"1"},{"start":"1ef","length":"1"}]},{"pgid":"1.7","version":"0'0","reported_seq":"598","reported_epoch":"614","state":"active+clean","last_fresh":"2021-02-10T04:37:12.200233+0000","last_change":"2021-02-10T04:29:39.829039+0000","last_active":"2021-02-10T04:37:12.200233+0000","last_peered":"2021-02-10T04:37:12.200233+0000","last_clean":"2021-02-10T04:37:12.200233+0000","last_became_active":"2021-02-10T04:28:49.053127+0000","last_became_peered":"2021-02-10T04:28:49.053127+0000","last_unstale":"2021-02-10T04:37:12.200233+0000","last_undegraded":"2021-02-10T04:37:12.200233+0000","last_fullsized":"2021-02-10T04:37:12.200233+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:39.828975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:39.828975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"574","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.199066+0000","last_change":"2021-02-10T04:29:20.757283+0000","last_active":"2021-02-10T04:37:09.199066+0000","last_peered":"2021-02-10T04:37:09.199066+0000","last_clean":"2021-02-10T04:37:09.199066+0000","last_became_active":"2021-02-10T04:25:51.023000+0000","last_became_peered":"2021-02-10T04:25:51.023000+0000","last_unstale":"2021-02-10T04:37:09.199066+0000","last_undegraded":"2021-02-10T04:37:09.199066+0000","last_fullsized":"2021-02-10T04:37:09.199066+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:20.757194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:20.757194+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"538'33","reported_seq":"644","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.200021+0000","last_change":"2021-02-10T04:28:47.476743+0000","last_active":"2021-02-10T04:37:10.200021+0000","last_peered":"2021-02-10T04:37:10.200021+0000","last_clean":"2021-02-10T04:37:10.200021+0000","last_became_active":"2021-02-10T04:25:49.013259+0000","last_became_peered":"2021-02-10T04:25:49.013259+0000","last_unstale":"2021-02-10T04:37:10.200021+0000","last_undegraded":"2021-02-10T04:37:10.200021+0000","last_fullsized":"2021-02-10T04:37:10.200021+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'27","last_scrub_stamp":"2021-02-10T04:28:47.476669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:47.476669+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":67,"num_read_kb":50,"num_write":35,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"694","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.201997+0000","last_change":"2021-02-10T04:33:49.761068+0000","last_active":"2021-02-10T04:37:10.201997+0000","last_peered":"2021-02-10T04:37:10.201997+0000","last_clean":"2021-02-10T04:37:10.201997+0000","last_became_active":"2021-02-10T04:28:39.041556+0000","last_became_peered":"2021-02-10T04:28:39.041556+0000","last_unstale":"2021-02-10T04:37:10.201997+0000","last_undegraded":"2021-02-10T04:37:10.201997+0000","last_fullsized":"2021-02-10T04:37:10.201997+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:33:49.760989+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:33:49.760989+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"570","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.198551+0000","last_change":"2021-02-10T04:29:37.364781+0000","last_active":"2021-02-10T04:37:09.198551+0000","last_peered":"2021-02-10T04:37:09.198551+0000","last_clean":"2021-02-10T04:37:09.198551+0000","last_became_active":"2021-02-10T04:25:53.047736+0000","last_became_peered":"2021-02-10T04:25:53.047736+0000","last_unstale":"2021-02-10T04:37:09.198551+0000","last_undegraded":"2021-02-10T04:37:09.198551+0000","last_fullsized":"2021-02-10T04:37:09.198551+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:37.364697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:37.364697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"611'879","reported_seq":"3478","reported_epoch":"611","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:37:09.838344+0000","last_change":"2021-02-10T04:37:09.835333+0000","last_active":"2021-02-10T04:37:09.838344+0000","last_peered":"2021-02-10T04:37:09.838344+0000","last_clean":"2021-02-10T04:37:09.838344+0000","last_became_active":"2021-02-10T04:28:25.157771+0000","last_became_peered":"2021-02-10T04:28:25.157771+0000","last_unstale":"2021-02-10T04:37:09.838344+0000","last_undegraded":"2021-02-10T04:37:09.838344+0000","last_fullsized":"2021-02-10T04:37:09.838344+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":879,"ondisk_log_size":879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"stat_sum":{"num_bytes":52010720,"num_objects":21,"num_object_clones":18,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":454,"num_read_kb":443342,"num_write":383,"num_write_kb":146665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"3"},{"start":"6b","length":"3"},{"start":"6f","length":"1"},{"start":"74","length":"2"},{"start":"78","length":"4"},{"start":"7e","length":"1"},{"start":"80","length":"4"},{"start":"87","length":"2"},{"start":"8b","length":"3"},{"start":"8f","length":"1"},{"start":"91","length":"2"},{"start":"97","length":"1"},{"start":"99","length":"1"},{"start":"9b","length":"2"},{"start":"9e","length":"1"},{"start":"a0","length":"1"},{"start":"a2","length":"1"},{"start":"a4","length":"1"},{"start":"a9","length":"3"},{"start":"b0","length":"1"},{"start":"b2","length":"2"},{"start":"b5","length":"2"},{"start":"b9","length":"1"},{"start":"bc","length":"3"},{"start":"c2","length":"1"},{"start":"c4","length":"2"},{"start":"c7","length":"1"},{"start":"c9","length":"1"},{"start":"cb","length":"2"},{"start":"cf","length":"1"},{"start":"d1","length":"1"},{"start":"d4","length":"8"},{"start":"dd","length":"1"},{"start":"e1","length":"3"},{"start":"e9","length":"2"},{"start":"ed","length":"4"},{"start":"f2","length":"1"},{"start":"f4","length":"5"},{"start":"fb","length":"1"},{"start":"fd","length":"4"},{"start":"103","length":"1"},{"start":"106","length":"4"},{"start":"10c","length":"1"},{"start":"10f","length":"6"},{"start":"116","length":"2"},{"start":"11c","length":"3"},{"start":"121","length":"1"},{"start":"123","length":"2"},{"start":"126","length":"1"},{"start":"128","length":"5"},{"start":"12e","length":"3"},{"start":"133","length":"4"},{"start":"139","length":"2"},{"start":"13c","length":"3"},{"start":"140","length":"1"},{"start":"143","length":"1"},{"start":"146","length":"1"},{"start":"148","length":"1"},{"start":"14c","length":"1"},{"start":"14e","length":"1"},{"start":"155","length":"2"},{"start":"15a","length":"4"},{"start":"161","length":"1"},{"start":"163","length":"1"},{"start":"168","length":"2"},{"start":"16b","length":"1"},{"start":"16f","length":"5"},{"start":"178","length":"4"},{"start":"17f","length":"1"},{"start":"185","length":"2"},{"start":"18a","length":"2"},{"start":"18e","length":"3"},{"start":"194","length":"4"},{"start":"19b","length":"5"},{"start":"1a1","length":"2"},{"start":"1a4","length":"1"},{"start":"1a8","length":"1"},{"start":"1ab","length":"1"},{"start":"1ae","length":"1"},{"start":"1b1","length":"1"},{"start":"1b7","length":"1"},{"start":"1b9","length":"2"},{"start":"1bc","length":"1"},{"start":"1bf","length":"2"},{"start":"1c3","length":"1"},{"start":"1c8","length":"1"},{"start":"1cd","length":"1"}]},{"pgid":"4.16","version":"538'34","reported_seq":"648","reported_epoch":"615","state":"active+clean","last_fresh":"2021-02-10T04:37:13.198720+0000","last_change":"2021-02-10T04:29:25.645462+0000","last_active":"2021-02-10T04:37:13.198720+0000","last_peered":"2021-02-10T04:37:13.198720+0000","last_clean":"2021-02-10T04:37:13.198720+0000","last_became_active":"2021-02-10T04:25:49.008942+0000","last_became_peered":"2021-02-10T04:25:49.008942+0000","last_unstale":"2021-02-10T04:37:13.198720+0000","last_undegraded":"2021-02-10T04:37:13.198720+0000","last_fullsized":"2021-02-10T04:37:13.198720+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'28","last_scrub_stamp":"2021-02-10T04:29:25.645396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.645396+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":71,"num_read_kb":49,"num_write":35,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"584","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.198970+0000","last_change":"2021-02-10T04:28:45.647998+0000","last_active":"2021-02-10T04:37:09.198970+0000","last_peered":"2021-02-10T04:37:09.198970+0000","last_clean":"2021-02-10T04:37:09.198970+0000","last_became_active":"2021-02-10T04:25:34.329694+0000","last_became_peered":"2021-02-10T04:25:34.329694+0000","last_unstale":"2021-02-10T04:37:09.198970+0000","last_undegraded":"2021-02-10T04:37:09.198970+0000","last_fullsized":"2021-02-10T04:37:09.198970+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:28:45.647916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:45.647916+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"572","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.198113+0000","last_change":"2021-02-10T04:28:33.321856+0000","last_active":"2021-02-10T04:37:09.198113+0000","last_peered":"2021-02-10T04:37:09.198113+0000","last_clean":"2021-02-10T04:37:09.198113+0000","last_became_active":"2021-02-10T04:25:51.030651+0000","last_became_peered":"2021-02-10T04:25:51.030651+0000","last_unstale":"2021-02-10T04:37:09.198113+0000","last_undegraded":"2021-02-10T04:37:09.198113+0000","last_fullsized":"2021-02-10T04:37:09.198113+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:33.321784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:33.321784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"570","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.199895+0000","last_change":"2021-02-10T04:29:28.776423+0000","last_active":"2021-02-10T04:37:09.199895+0000","last_peered":"2021-02-10T04:37:09.199895+0000","last_clean":"2021-02-10T04:37:09.199895+0000","last_became_active":"2021-02-10T04:25:53.035867+0000","last_became_peered":"2021-02-10T04:25:53.035867+0000","last_unstale":"2021-02-10T04:37:09.199895+0000","last_undegraded":"2021-02-10T04:37:09.199895+0000","last_fullsized":"2021-02-10T04:37:09.199895+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.776340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.776340+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"538'19","reported_seq":"630","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.197993+0000","last_change":"2021-02-10T04:28:11.311247+0000","last_active":"2021-02-10T04:37:09.197993+0000","last_peered":"2021-02-10T04:37:09.197993+0000","last_clean":"2021-02-10T04:37:09.197993+0000","last_became_active":"2021-02-10T04:26:39.003832+0000","last_became_peered":"2021-02-10T04:26:39.003832+0000","last_unstale":"2021-02-10T04:37:09.197993+0000","last_undegraded":"2021-02-10T04:37:09.197993+0000","last_fullsized":"2021-02-10T04:37:09.197993+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'13","last_scrub_stamp":"2021-02-10T04:28:11.311182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:11.311182+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":19,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"587","reported_epoch":"614","state":"active+clean","last_fresh":"2021-02-10T04:37:12.201284+0000","last_change":"2021-02-10T04:29:18.565481+0000","last_active":"2021-02-10T04:37:12.201284+0000","last_peered":"2021-02-10T04:37:12.201284+0000","last_clean":"2021-02-10T04:37:12.201284+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:37:12.201284+0000","last_undegraded":"2021-02-10T04:37:12.201284+0000","last_fullsized":"2021-02-10T04:37:12.201284+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:29:18.565413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:18.565413+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"575","reported_epoch":"613","state":"active+clean","last_fresh":"2021-02-10T04:37:11.196297+0000","last_change":"2021-02-10T04:29:18.429730+0000","last_active":"2021-02-10T04:37:11.196297+0000","last_peered":"2021-02-10T04:37:11.196297+0000","last_clean":"2021-02-10T04:37:11.196297+0000","last_became_active":"2021-02-10T04:25:51.031864+0000","last_became_peered":"2021-02-10T04:25:51.031864+0000","last_unstale":"2021-02-10T04:37:11.196297+0000","last_undegraded":"2021-02-10T04:37:11.196297+0000","last_fullsized":"2021-02-10T04:37:11.196297+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:18.429649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:18.429649+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"501","reported_epoch":"614","state":"active+clean","last_fresh":"2021-02-10T04:37:12.199597+0000","last_change":"2021-02-10T04:29:36.866949+0000","last_active":"2021-02-10T04:37:12.199597+0000","last_peered":"2021-02-10T04:37:12.199597+0000","last_clean":"2021-02-10T04:37:12.199597+0000","last_became_active":"2021-02-10T04:29:07.147249+0000","last_became_peered":"2021-02-10T04:29:07.147249+0000","last_unstale":"2021-02-10T04:37:12.199597+0000","last_undegraded":"2021-02-10T04:37:12.199597+0000","last_fullsized":"2021-02-10T04:37:12.199597+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.866833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.866833+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"538'36","reported_seq":"747","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.199190+0000","last_change":"2021-02-10T04:33:46.786034+0000","last_active":"2021-02-10T04:37:09.199190+0000","last_peered":"2021-02-10T04:37:09.199190+0000","last_clean":"2021-02-10T04:37:09.199190+0000","last_became_active":"2021-02-10T04:26:39.003971+0000","last_became_peered":"2021-02-10T04:26:39.003971+0000","last_unstale":"2021-02-10T04:37:09.199190+0000","last_undegraded":"2021-02-10T04:37:09.199190+0000","last_fullsized":"2021-02-10T04:37:09.199190+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:33:46.785905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:46.785905+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":36,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"84'7","reported_seq":"604","reported_epoch":"613","state":"active+clean","last_fresh":"2021-02-10T04:37:11.196114+0000","last_change":"2021-02-10T04:30:12.538670+0000","last_active":"2021-02-10T04:37:11.196114+0000","last_peered":"2021-02-10T04:37:11.196114+0000","last_clean":"2021-02-10T04:37:11.196114+0000","last_became_active":"2021-02-10T04:25:36.334366+0000","last_became_peered":"2021-02-10T04:25:36.334366+0000","last_unstale":"2021-02-10T04:37:11.196114+0000","last_undegraded":"2021-02-10T04:37:11.196114+0000","last_fullsized":"2021-02-10T04:37:11.196114+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'7","last_scrub_stamp":"2021-02-10T04:30:12.538549+0000","last_deep_scrub":"84'7","last_deep_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_clean_scrub_stamp":"2021-02-10T04:30:12.538549+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"58'2","reported_seq":"714","reported_epoch":"614","state":"active+clean","last_fresh":"2021-02-10T04:37:12.203544+0000","last_change":"2021-02-10T04:28:47.496534+0000","last_active":"2021-02-10T04:37:12.203544+0000","last_peered":"2021-02-10T04:37:12.203544+0000","last_clean":"2021-02-10T04:37:12.203544+0000","last_became_active":"2021-02-10T04:25:51.032328+0000","last_became_peered":"2021-02-10T04:25:51.032328+0000","last_unstale":"2021-02-10T04:37:12.203544+0000","last_undegraded":"2021-02-10T04:37:12.203544+0000","last_fullsized":"2021-02-10T04:37:12.203544+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:28:47.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:47.496468+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"575","reported_epoch":"615","state":"active+clean","last_fresh":"2021-02-10T04:37:13.198340+0000","last_change":"2021-02-10T04:28:45.697744+0000","last_active":"2021-02-10T04:37:13.198340+0000","last_peered":"2021-02-10T04:37:13.198340+0000","last_clean":"2021-02-10T04:37:13.198340+0000","last_became_active":"2021-02-10T04:25:53.045177+0000","last_became_peered":"2021-02-10T04:25:53.045177+0000","last_unstale":"2021-02-10T04:37:13.198340+0000","last_undegraded":"2021-02-10T04:37:13.198340+0000","last_fullsized":"2021-02-10T04:37:13.198340+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_clean_scrub_stamp":"2021-02-10T04:28:45.697677+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"538'25","reported_seq":"625","reported_epoch":"613","state":"active+clean","last_fresh":"2021-02-10T04:37:11.196012+0000","last_change":"2021-02-10T04:28:50.493712+0000","last_active":"2021-02-10T04:37:11.196012+0000","last_peered":"2021-02-10T04:37:11.196012+0000","last_clean":"2021-02-10T04:37:11.196012+0000","last_became_active":"2021-02-10T04:25:49.010890+0000","last_became_peered":"2021-02-10T04:25:49.010890+0000","last_unstale":"2021-02-10T04:37:11.196012+0000","last_undegraded":"2021-02-10T04:37:11.196012+0000","last_fullsized":"2021-02-10T04:37:11.196012+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2021-02-10T04:28:50.493641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:50.493641+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":35,"num_write":26,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"591","reported_epoch":"615","state":"active+clean","last_fresh":"2021-02-10T04:37:13.196617+0000","last_change":"2021-02-10T04:29:19.564845+0000","last_active":"2021-02-10T04:37:13.196617+0000","last_peered":"2021-02-10T04:37:13.196617+0000","last_clean":"2021-02-10T04:37:13.196617+0000","last_became_active":"2021-02-10T04:25:34.321096+0000","last_became_peered":"2021-02-10T04:25:34.321096+0000","last_unstale":"2021-02-10T04:37:13.196617+0000","last_undegraded":"2021-02-10T04:37:13.196617+0000","last_fullsized":"2021-02-10T04:37:13.196617+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:19.564725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:29:19.564725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"58'2","reported_seq":"710","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.199315+0000","last_change":"2021-02-10T04:29:38.343946+0000","last_active":"2021-02-10T04:37:09.199315+0000","last_peered":"2021-02-10T04:37:09.199315+0000","last_clean":"2021-02-10T04:37:09.199315+0000","last_became_active":"2021-02-10T04:25:51.018300+0000","last_became_peered":"2021-02-10T04:25:51.018300+0000","last_unstale":"2021-02-10T04:37:09.199315+0000","last_undegraded":"2021-02-10T04:37:09.199315+0000","last_fullsized":"2021-02-10T04:37:09.199315+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:29:38.343834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.343834+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"572","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.202151+0000","last_change":"2021-02-10T04:29:23.934149+0000","last_active":"2021-02-10T04:37:10.202151+0000","last_peered":"2021-02-10T04:37:10.202151+0000","last_clean":"2021-02-10T04:37:10.202151+0000","last_became_active":"2021-02-10T04:25:53.031782+0000","last_became_peered":"2021-02-10T04:25:53.031782+0000","last_unstale":"2021-02-10T04:37:10.202151+0000","last_undegraded":"2021-02-10T04:37:10.202151+0000","last_fullsized":"2021-02-10T04:37:10.202151+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.934078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.934078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"538'21","reported_seq":"309","reported_epoch":"614","state":"active+clean","last_fresh":"2021-02-10T04:37:12.201987+0000","last_change":"2021-02-10T04:33:05.141994+0000","last_active":"2021-02-10T04:37:12.201987+0000","last_peered":"2021-02-10T04:37:12.201987+0000","last_clean":"2021-02-10T04:37:12.201987+0000","last_became_active":"2021-02-10T04:33:05.141566+0000","last_became_peered":"2021-02-10T04:33:05.141566+0000","last_unstale":"2021-02-10T04:37:12.201987+0000","last_undegraded":"2021-02-10T04:37:12.201987+0000","last_fullsized":"2021-02-10T04:37:12.201987+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:30:11.170034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.170034+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"585","reported_epoch":"614","state":"active+clean","last_fresh":"2021-02-10T04:37:12.198883+0000","last_change":"2021-02-10T04:28:36.204091+0000","last_active":"2021-02-10T04:37:12.198883+0000","last_peered":"2021-02-10T04:37:12.198883+0000","last_clean":"2021-02-10T04:37:12.198883+0000","last_became_active":"2021-02-10T04:25:34.330070+0000","last_became_peered":"2021-02-10T04:25:34.330070+0000","last_unstale":"2021-02-10T04:37:12.198883+0000","last_undegraded":"2021-02-10T04:37:12.198883+0000","last_fullsized":"2021-02-10T04:37:12.198883+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:36.204007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:36.204007+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"572","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.198476+0000","last_change":"2021-02-10T04:28:36.311808+0000","last_active":"2021-02-10T04:37:09.198476+0000","last_peered":"2021-02-10T04:37:09.198476+0000","last_clean":"2021-02-10T04:37:09.198476+0000","last_became_active":"2021-02-10T04:25:51.029155+0000","last_became_peered":"2021-02-10T04:25:51.029155+0000","last_unstale":"2021-02-10T04:37:09.198476+0000","last_undegraded":"2021-02-10T04:37:09.198476+0000","last_fullsized":"2021-02-10T04:37:09.198476+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:36.311725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:28:36.311725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"580","reported_epoch":"615","state":"active+clean","last_fresh":"2021-02-10T04:37:13.194974+0000","last_change":"2021-02-10T04:33:49.338303+0000","last_active":"2021-02-10T04:37:13.194974+0000","last_peered":"2021-02-10T04:37:13.194974+0000","last_clean":"2021-02-10T04:37:13.194974+0000","last_became_active":"2021-02-10T04:25:53.039322+0000","last_became_peered":"2021-02-10T04:25:53.039322+0000","last_unstale":"2021-02-10T04:37:13.194974+0000","last_undegraded":"2021-02-10T04:37:13.194974+0000","last_fullsized":"2021-02-10T04:37:13.194974+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:49.338203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:33:49.338203+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"538'23","reported_seq":"610","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.200763+0000","last_change":"2021-02-10T04:27:42.487239+0000","last_active":"2021-02-10T04:37:10.200763+0000","last_peered":"2021-02-10T04:37:10.200763+0000","last_clean":"2021-02-10T04:37:10.200763+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:37:10.200763+0000","last_undegraded":"2021-02-10T04:37:10.200763+0000","last_fullsized":"2021-02-10T04:37:10.200763+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:27:42.487154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:42.487154+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"269","reported_epoch":"613","state":"active+clean","last_fresh":"2021-02-10T04:37:11.196104+0000","last_change":"2021-02-10T04:33:05.148470+0000","last_active":"2021-02-10T04:37:11.196104+0000","last_peered":"2021-02-10T04:37:11.196104+0000","last_clean":"2021-02-10T04:37:11.196104+0000","last_became_active":"2021-02-10T04:33:05.147956+0000","last_became_peered":"2021-02-10T04:33:05.147956+0000","last_unstale":"2021-02-10T04:37:11.196104+0000","last_undegraded":"2021-02-10T04:37:11.196104+0000","last_fullsized":"2021-02-10T04:37:11.196104+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:28:33.070726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:33.070726+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"258","reported_epoch":"613","state":"active+clean","last_fresh":"2021-02-10T04:37:11.196376+0000","last_change":"2021-02-10T04:33:05.150906+0000","last_active":"2021-02-10T04:37:11.196376+0000","last_peered":"2021-02-10T04:37:11.196376+0000","last_clean":"2021-02-10T04:37:11.196376+0000","last_became_active":"2021-02-10T04:33:05.150529+0000","last_became_peered":"2021-02-10T04:33:05.150529+0000","last_unstale":"2021-02-10T04:37:11.196376+0000","last_undegraded":"2021-02-10T04:37:11.196376+0000","last_fullsized":"2021-02-10T04:37:11.196376+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:33.026988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:29:33.026988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"570","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.196021+0000","last_change":"2021-02-10T04:29:24.421782+0000","last_active":"2021-02-10T04:37:09.196021+0000","last_peered":"2021-02-10T04:37:09.196021+0000","last_clean":"2021-02-10T04:37:09.196021+0000","last_became_active":"2021-02-10T04:25:53.043277+0000","last_became_peered":"2021-02-10T04:25:53.043277+0000","last_unstale":"2021-02-10T04:37:09.196021+0000","last_undegraded":"2021-02-10T04:37:09.196021+0000","last_fullsized":"2021-02-10T04:37:09.196021+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.421710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.421710+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"538'16","reported_seq":"724","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.195792+0000","last_change":"2021-02-10T04:37:07.799219+0000","last_active":"2021-02-10T04:37:09.195792+0000","last_peered":"2021-02-10T04:37:09.195792+0000","last_clean":"2021-02-10T04:37:09.195792+0000","last_became_active":"2021-02-10T04:26:39.005616+0000","last_became_peered":"2021-02-10T04:26:39.005616+0000","last_unstale":"2021-02-10T04:37:09.195792+0000","last_undegraded":"2021-02-10T04:37:09.195792+0000","last_fullsized":"2021-02-10T04:37:09.195792+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2021-02-10T04:27:48.284248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:48.284248+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":36,"num_read_kb":18,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"584","reported_epoch":"613","state":"active+clean","last_fresh":"2021-02-10T04:37:11.196769+0000","last_change":"2021-02-10T04:28:49.522178+0000","last_active":"2021-02-10T04:37:11.196769+0000","last_peered":"2021-02-10T04:37:11.196769+0000","last_clean":"2021-02-10T04:37:11.196769+0000","last_became_active":"2021-02-10T04:25:34.331787+0000","last_became_peered":"2021-02-10T04:25:34.331787+0000","last_unstale":"2021-02-10T04:37:11.196769+0000","last_undegraded":"2021-02-10T04:37:11.196769+0000","last_fullsized":"2021-02-10T04:37:11.196769+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:49.522124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:49.522124+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"576","reported_epoch":"614","state":"active+clean","last_fresh":"2021-02-10T04:37:12.201187+0000","last_change":"2021-02-10T04:29:23.645783+0000","last_active":"2021-02-10T04:37:12.201187+0000","last_peered":"2021-02-10T04:37:12.201187+0000","last_clean":"2021-02-10T04:37:12.201187+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:37:12.201187+0000","last_undegraded":"2021-02-10T04:37:12.201187+0000","last_fullsized":"2021-02-10T04:37:12.201187+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.645694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.645694+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"498","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.199559+0000","last_change":"2021-02-10T04:29:31.825006+0000","last_active":"2021-02-10T04:37:09.199559+0000","last_peered":"2021-02-10T04:37:09.199559+0000","last_clean":"2021-02-10T04:37:09.199559+0000","last_became_active":"2021-02-10T04:29:07.149764+0000","last_became_peered":"2021-02-10T04:29:07.149764+0000","last_unstale":"2021-02-10T04:37:09.199559+0000","last_undegraded":"2021-02-10T04:37:09.199559+0000","last_fullsized":"2021-02-10T04:37:09.199559+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:31.824925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.824925+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"538'29","reported_seq":"627","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.199791+0000","last_change":"2021-02-10T04:28:31.382776+0000","last_active":"2021-02-10T04:37:09.199791+0000","last_peered":"2021-02-10T04:37:09.199791+0000","last_clean":"2021-02-10T04:37:09.199791+0000","last_became_active":"2021-02-10T04:25:49.007049+0000","last_became_peered":"2021-02-10T04:25:49.007049+0000","last_unstale":"2021-02-10T04:37:09.199791+0000","last_undegraded":"2021-02-10T04:37:09.199791+0000","last_fullsized":"2021-02-10T04:37:09.199791+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2021-02-10T04:28:31.382671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:31.382671+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":29,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"710","reported_epoch":"614","state":"active+clean","last_fresh":"2021-02-10T04:37:12.246694+0000","last_change":"2021-02-10T04:37:12.246694+0000","last_active":"2021-02-10T04:37:12.246694+0000","last_peered":"2021-02-10T04:37:12.246694+0000","last_clean":"2021-02-10T04:37:12.246694+0000","last_became_active":"2021-02-10T04:25:35.328828+0000","last_became_peered":"2021-02-10T04:25:35.328828+0000","last_unstale":"2021-02-10T04:37:12.246694+0000","last_undegraded":"2021-02-10T04:37:12.246694+0000","last_fullsized":"2021-02-10T04:37:12.246694+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:28:35.196342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:35.196342+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"494","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.198987+0000","last_change":"2021-02-10T04:29:07.150253+0000","last_active":"2021-02-10T04:37:10.198987+0000","last_peered":"2021-02-10T04:37:10.198987+0000","last_clean":"2021-02-10T04:37:10.198987+0000","last_became_active":"2021-02-10T04:29:07.149867+0000","last_became_peered":"2021-02-10T04:29:07.149867+0000","last_unstale":"2021-02-10T04:37:10.198987+0000","last_undegraded":"2021-02-10T04:37:10.198987+0000","last_fullsized":"2021-02-10T04:37:10.198987+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:47.473024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:47.473024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"570","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.198811+0000","last_change":"2021-02-10T04:33:50.809533+0000","last_active":"2021-02-10T04:37:09.198811+0000","last_peered":"2021-02-10T04:37:09.198811+0000","last_clean":"2021-02-10T04:37:09.198811+0000","last_became_active":"2021-02-10T04:25:53.150845+0000","last_became_peered":"2021-02-10T04:25:53.150845+0000","last_unstale":"2021-02-10T04:37:09.198811+0000","last_undegraded":"2021-02-10T04:37:09.198811+0000","last_fullsized":"2021-02-10T04:37:09.198811+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:50.809468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_clean_scrub_stamp":"2021-02-10T04:33:50.809468+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"538'32","reported_seq":"641","reported_epoch":"615","state":"active+clean","last_fresh":"2021-02-10T04:37:13.198981+0000","last_change":"2021-02-10T04:29:38.663351+0000","last_active":"2021-02-10T04:37:13.198981+0000","last_peered":"2021-02-10T04:37:13.198981+0000","last_clean":"2021-02-10T04:37:13.198981+0000","last_became_active":"2021-02-10T04:25:49.009177+0000","last_became_peered":"2021-02-10T04:25:49.009177+0000","last_unstale":"2021-02-10T04:37:13.198981+0000","last_undegraded":"2021-02-10T04:37:13.198981+0000","last_fullsized":"2021-02-10T04:37:13.198981+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2021-02-10T04:29:38.663272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.663272+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":60,"num_read_kb":42,"num_write":32,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"593","reported_epoch":"613","state":"active+clean","last_fresh":"2021-02-10T04:37:11.195649+0000","last_change":"2021-02-10T04:28:37.515037+0000","last_active":"2021-02-10T04:37:11.195649+0000","last_peered":"2021-02-10T04:37:11.195649+0000","last_clean":"2021-02-10T04:37:11.195649+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:37:11.195649+0000","last_undegraded":"2021-02-10T04:37:11.195649+0000","last_fullsized":"2021-02-10T04:37:11.195649+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:37.514943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:37.514943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"58'2","reported_seq":"715","reported_epoch":"614","state":"active+clean","last_fresh":"2021-02-10T04:37:12.829362+0000","last_change":"2021-02-10T04:29:31.726022+0000","last_active":"2021-02-10T04:37:12.829362+0000","last_peered":"2021-02-10T04:37:12.829362+0000","last_clean":"2021-02-10T04:37:12.829362+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:37:12.829362+0000","last_undegraded":"2021-02-10T04:37:12.829362+0000","last_fullsized":"2021-02-10T04:37:12.829362+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:29:31.725932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.725932+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"500","reported_epoch":"613","state":"active+clean","last_fresh":"2021-02-10T04:37:11.197126+0000","last_change":"2021-02-10T04:29:26.437691+0000","last_active":"2021-02-10T04:37:11.197126+0000","last_peered":"2021-02-10T04:37:11.197126+0000","last_clean":"2021-02-10T04:37:11.197126+0000","last_became_active":"2021-02-10T04:29:07.150801+0000","last_became_peered":"2021-02-10T04:29:07.150801+0000","last_unstale":"2021-02-10T04:37:11.197126+0000","last_undegraded":"2021-02-10T04:37:11.197126+0000","last_fullsized":"2021-02-10T04:37:11.197126+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:26.437591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.437591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"538'43","reported_seq":"653","reported_epoch":"614","state":"active+clean","last_fresh":"2021-02-10T04:37:12.200777+0000","last_change":"2021-02-10T04:30:13.526264+0000","last_active":"2021-02-10T04:37:12.200777+0000","last_peered":"2021-02-10T04:37:12.200777+0000","last_clean":"2021-02-10T04:37:12.200777+0000","last_became_active":"2021-02-10T04:25:49.015760+0000","last_became_peered":"2021-02-10T04:25:49.015760+0000","last_unstale":"2021-02-10T04:37:12.200777+0000","last_undegraded":"2021-02-10T04:37:12.200777+0000","last_fullsized":"2021-02-10T04:37:12.200777+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'35","last_scrub_stamp":"2021-02-10T04:30:13.526170+0000","last_deep_scrub":"76'35","last_deep_scrub_stamp":"2021-02-10T04:26:58.198641+0000","last_clean_scrub_stamp":"2021-02-10T04:30:13.526170+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":62,"num_read_kb":42,"num_write":32,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"581","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.199537+0000","last_change":"2021-02-10T04:30:11.485059+0000","last_active":"2021-02-10T04:37:09.199537+0000","last_peered":"2021-02-10T04:37:09.199537+0000","last_clean":"2021-02-10T04:37:09.199537+0000","last_became_active":"2021-02-10T04:25:34.324981+0000","last_became_peered":"2021-02-10T04:25:34.324981+0000","last_unstale":"2021-02-10T04:37:09.199537+0000","last_undegraded":"2021-02-10T04:37:09.199537+0000","last_fullsized":"2021-02-10T04:37:09.199537+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:11.485004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.485004+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"577","reported_epoch":"615","state":"active+clean","last_fresh":"2021-02-10T04:37:13.199271+0000","last_change":"2021-02-10T04:29:23.589223+0000","last_active":"2021-02-10T04:37:13.199271+0000","last_peered":"2021-02-10T04:37:13.199271+0000","last_clean":"2021-02-10T04:37:13.199271+0000","last_became_active":"2021-02-10T04:25:51.022853+0000","last_became_peered":"2021-02-10T04:25:51.022853+0000","last_unstale":"2021-02-10T04:37:13.199271+0000","last_undegraded":"2021-02-10T04:37:13.199271+0000","last_fullsized":"2021-02-10T04:37:13.199271+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.589159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.589159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"258","reported_epoch":"613","state":"active+clean","last_fresh":"2021-02-10T04:37:11.196887+0000","last_change":"2021-02-10T04:33:05.142344+0000","last_active":"2021-02-10T04:37:11.196887+0000","last_peered":"2021-02-10T04:37:11.196887+0000","last_clean":"2021-02-10T04:37:11.196887+0000","last_became_active":"2021-02-10T04:33:05.141911+0000","last_became_peered":"2021-02-10T04:33:05.141911+0000","last_unstale":"2021-02-10T04:37:11.196887+0000","last_undegraded":"2021-02-10T04:37:11.196887+0000","last_fullsized":"2021-02-10T04:37:11.196887+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:25.075937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.075937+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"538'27","reported_seq":"617","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.200283+0000","last_change":"2021-02-10T04:28:46.680268+0000","last_active":"2021-02-10T04:37:09.200283+0000","last_peered":"2021-02-10T04:37:09.200283+0000","last_clean":"2021-02-10T04:37:09.200283+0000","last_became_active":"2021-02-10T04:25:49.010553+0000","last_became_peered":"2021-02-10T04:25:49.010553+0000","last_unstale":"2021-02-10T04:37:09.200283+0000","last_undegraded":"2021-02-10T04:37:09.200283+0000","last_fullsized":"2021-02-10T04:37:09.200283+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:28:46.680176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:46.680176+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"588","reported_epoch":"614","state":"active+clean","last_fresh":"2021-02-10T04:37:12.201860+0000","last_change":"2021-02-10T04:28:38.246850+0000","last_active":"2021-02-10T04:37:12.201860+0000","last_peered":"2021-02-10T04:37:12.201860+0000","last_clean":"2021-02-10T04:37:12.201860+0000","last_became_active":"2021-02-10T04:25:36.336442+0000","last_became_peered":"2021-02-10T04:25:36.336442+0000","last_unstale":"2021-02-10T04:37:12.201860+0000","last_undegraded":"2021-02-10T04:37:12.201860+0000","last_fullsized":"2021-02-10T04:37:12.201860+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:28:38.246768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:38.246768+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"257","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.201291+0000","last_change":"2021-02-10T04:33:05.147757+0000","last_active":"2021-02-10T04:37:10.201291+0000","last_peered":"2021-02-10T04:37:10.201291+0000","last_clean":"2021-02-10T04:37:10.201291+0000","last_became_active":"2021-02-10T04:33:05.146838+0000","last_became_peered":"2021-02-10T04:33:05.146838+0000","last_unstale":"2021-02-10T04:37:10.201291+0000","last_undegraded":"2021-02-10T04:37:10.201291+0000","last_fullsized":"2021-02-10T04:37:10.201291+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:22.078247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:22.078247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"572","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.201769+0000","last_change":"2021-02-10T04:29:31.903091+0000","last_active":"2021-02-10T04:37:10.201769+0000","last_peered":"2021-02-10T04:37:10.201769+0000","last_clean":"2021-02-10T04:37:10.201769+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:37:10.201769+0000","last_undegraded":"2021-02-10T04:37:10.201769+0000","last_fullsized":"2021-02-10T04:37:10.201769+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:31.902996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.902996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"538'28","reported_seq":"624","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.197582+0000","last_change":"2021-02-10T04:27:58.363322+0000","last_active":"2021-02-10T04:37:09.197582+0000","last_peered":"2021-02-10T04:37:09.197582+0000","last_clean":"2021-02-10T04:37:09.197582+0000","last_became_active":"2021-02-10T04:25:49.008997+0000","last_became_peered":"2021-02-10T04:25:49.008997+0000","last_unstale":"2021-02-10T04:37:09.197582+0000","last_undegraded":"2021-02-10T04:37:09.197582+0000","last_fullsized":"2021-02-10T04:37:09.197582+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'22","last_scrub_stamp":"2021-02-10T04:27:58.363248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:58.363248+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":40,"num_write":30,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"593","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.198318+0000","last_change":"2021-02-10T04:28:30.415204+0000","last_active":"2021-02-10T04:37:09.198318+0000","last_peered":"2021-02-10T04:37:09.198318+0000","last_clean":"2021-02-10T04:37:09.198318+0000","last_became_active":"2021-02-10T04:25:36.336893+0000","last_became_peered":"2021-02-10T04:25:36.336893+0000","last_unstale":"2021-02-10T04:37:09.198318+0000","last_undegraded":"2021-02-10T04:37:09.198318+0000","last_fullsized":"2021-02-10T04:37:09.198318+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:30.415134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:26:46.425288+0000","last_clean_scrub_stamp":"2021-02-10T04:28:30.415134+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"574","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.201083+0000","last_change":"2021-02-10T04:33:52.643172+0000","last_active":"2021-02-10T04:37:10.201083+0000","last_peered":"2021-02-10T04:37:10.201083+0000","last_clean":"2021-02-10T04:37:10.201083+0000","last_became_active":"2021-02-10T04:25:51.018872+0000","last_became_peered":"2021-02-10T04:25:51.018872+0000","last_unstale":"2021-02-10T04:37:10.201083+0000","last_undegraded":"2021-02-10T04:37:10.201083+0000","last_fullsized":"2021-02-10T04:37:10.201083+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:52.643099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:33:52.643099+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"570","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.197943+0000","last_change":"2021-02-10T04:29:26.355726+0000","last_active":"2021-02-10T04:37:09.197943+0000","last_peered":"2021-02-10T04:37:09.197943+0000","last_clean":"2021-02-10T04:37:09.197943+0000","last_became_active":"2021-02-10T04:25:53.147831+0000","last_became_peered":"2021-02-10T04:25:53.147831+0000","last_unstale":"2021-02-10T04:37:09.197943+0000","last_undegraded":"2021-02-10T04:37:09.197943+0000","last_fullsized":"2021-02-10T04:37:09.197943+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:26.355652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.355652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"538'31","reported_seq":"639","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.199202+0000","last_change":"2021-02-10T04:28:32.565457+0000","last_active":"2021-02-10T04:37:10.199202+0000","last_peered":"2021-02-10T04:37:10.199202+0000","last_clean":"2021-02-10T04:37:10.199202+0000","last_became_active":"2021-02-10T04:25:49.000911+0000","last_became_peered":"2021-02-10T04:25:49.000911+0000","last_unstale":"2021-02-10T04:37:10.199202+0000","last_undegraded":"2021-02-10T04:37:10.199202+0000","last_fullsized":"2021-02-10T04:37:10.199202+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2021-02-10T04:28:32.565374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:32.565374+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":62,"num_read_kb":46,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"500","reported_epoch":"613","state":"active+clean","last_fresh":"2021-02-10T04:37:11.197254+0000","last_change":"2021-02-10T04:33:48.646869+0000","last_active":"2021-02-10T04:37:11.197254+0000","last_peered":"2021-02-10T04:37:11.197254+0000","last_clean":"2021-02-10T04:37:11.197254+0000","last_became_active":"2021-02-10T04:29:07.151577+0000","last_became_peered":"2021-02-10T04:29:07.151577+0000","last_unstale":"2021-02-10T04:37:11.197254+0000","last_undegraded":"2021-02-10T04:37:11.197254+0000","last_fullsized":"2021-02-10T04:37:11.197254+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:48.646802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:33:48.646802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"575","reported_epoch":"613","state":"active+clean","last_fresh":"2021-02-10T04:37:11.196029+0000","last_change":"2021-02-10T04:28:46.464172+0000","last_active":"2021-02-10T04:37:11.196029+0000","last_peered":"2021-02-10T04:37:11.196029+0000","last_clean":"2021-02-10T04:37:11.196029+0000","last_became_active":"2021-02-10T04:25:51.025488+0000","last_became_peered":"2021-02-10T04:25:51.025488+0000","last_unstale":"2021-02-10T04:37:11.196029+0000","last_undegraded":"2021-02-10T04:37:11.196029+0000","last_fullsized":"2021-02-10T04:37:11.196029+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:46.464108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:46.464108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"575","reported_epoch":"615","state":"active+clean","last_fresh":"2021-02-10T04:37:13.195302+0000","last_change":"2021-02-10T04:28:44.719453+0000","last_active":"2021-02-10T04:37:13.195302+0000","last_peered":"2021-02-10T04:37:13.195302+0000","last_clean":"2021-02-10T04:37:13.195302+0000","last_became_active":"2021-02-10T04:25:53.039844+0000","last_became_peered":"2021-02-10T04:25:53.039844+0000","last_unstale":"2021-02-10T04:37:13.195302+0000","last_undegraded":"2021-02-10T04:37:13.195302+0000","last_fullsized":"2021-02-10T04:37:13.195302+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:44.719363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:44.719363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"538'14","reported_seq":"606","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.199186+0000","last_change":"2021-02-10T04:30:13.089576+0000","last_active":"2021-02-10T04:37:10.199186+0000","last_peered":"2021-02-10T04:37:10.199186+0000","last_clean":"2021-02-10T04:37:10.199186+0000","last_became_active":"2021-02-10T04:25:49.013196+0000","last_became_peered":"2021-02-10T04:25:49.013196+0000","last_unstale":"2021-02-10T04:37:10.199186+0000","last_undegraded":"2021-02-10T04:37:10.199186+0000","last_fullsized":"2021-02-10T04:37:10.199186+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2021-02-10T04:30:13.089494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:30:13.089494+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":15,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"501","reported_epoch":"614","state":"active+clean","last_fresh":"2021-02-10T04:37:12.199171+0000","last_change":"2021-02-10T04:30:11.537208+0000","last_active":"2021-02-10T04:37:12.199171+0000","last_peered":"2021-02-10T04:37:12.199171+0000","last_clean":"2021-02-10T04:37:12.199171+0000","last_became_active":"2021-02-10T04:29:07.157944+0000","last_became_peered":"2021-02-10T04:29:07.157944+0000","last_unstale":"2021-02-10T04:37:12.199171+0000","last_undegraded":"2021-02-10T04:37:12.199171+0000","last_fullsized":"2021-02-10T04:37:12.199171+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:11.537136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.537136+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"494","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.199292+0000","last_change":"2021-02-10T04:29:07.146984+0000","last_active":"2021-02-10T04:37:10.199292+0000","last_peered":"2021-02-10T04:37:10.199292+0000","last_clean":"2021-02-10T04:37:10.199292+0000","last_became_active":"2021-02-10T04:29:07.146525+0000","last_became_peered":"2021-02-10T04:29:07.146525+0000","last_unstale":"2021-02-10T04:37:10.199292+0000","last_undegraded":"2021-02-10T04:37:10.199292+0000","last_fullsized":"2021-02-10T04:37:10.199292+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:46.493066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:46.493066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"573","reported_epoch":"613","state":"active+clean","last_fresh":"2021-02-10T04:37:11.196737+0000","last_change":"2021-02-10T04:29:22.475613+0000","last_active":"2021-02-10T04:37:11.196737+0000","last_peered":"2021-02-10T04:37:11.196737+0000","last_clean":"2021-02-10T04:37:11.196737+0000","last_became_active":"2021-02-10T04:25:53.042601+0000","last_became_peered":"2021-02-10T04:25:53.042601+0000","last_unstale":"2021-02-10T04:37:11.196737+0000","last_undegraded":"2021-02-10T04:37:11.196737+0000","last_fullsized":"2021-02-10T04:37:11.196737+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:22.475504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:22.475504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"538'38","reported_seq":"639","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.199082+0000","last_change":"2021-02-10T04:28:35.520489+0000","last_active":"2021-02-10T04:37:10.199082+0000","last_peered":"2021-02-10T04:37:10.199082+0000","last_clean":"2021-02-10T04:37:10.199082+0000","last_became_active":"2021-02-10T04:25:49.010481+0000","last_became_peered":"2021-02-10T04:25:49.010481+0000","last_unstale":"2021-02-10T04:37:10.199082+0000","last_undegraded":"2021-02-10T04:37:10.199082+0000","last_fullsized":"2021-02-10T04:37:10.199082+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'32","last_scrub_stamp":"2021-02-10T04:28:35.520410+0000","last_deep_scrub":"78'32","last_deep_scrub_stamp":"2021-02-10T04:27:09.170428+0000","last_clean_scrub_stamp":"2021-02-10T04:28:35.520410+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":56,"num_read_kb":37,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"581","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.197925+0000","last_change":"2021-02-10T04:29:29.355356+0000","last_active":"2021-02-10T04:37:09.197925+0000","last_peered":"2021-02-10T04:37:09.197925+0000","last_clean":"2021-02-10T04:37:09.197925+0000","last_became_active":"2021-02-10T04:25:34.327162+0000","last_became_peered":"2021-02-10T04:25:34.327162+0000","last_unstale":"2021-02-10T04:37:09.197925+0000","last_undegraded":"2021-02-10T04:37:09.197925+0000","last_fullsized":"2021-02-10T04:37:09.197925+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.355257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.355257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"258","reported_epoch":"613","state":"active+clean","last_fresh":"2021-02-10T04:37:11.196834+0000","last_change":"2021-02-10T04:33:05.169577+0000","last_active":"2021-02-10T04:37:11.196834+0000","last_peered":"2021-02-10T04:37:11.196834+0000","last_clean":"2021-02-10T04:37:11.196834+0000","last_became_active":"2021-02-10T04:33:05.169159+0000","last_became_peered":"2021-02-10T04:33:05.169159+0000","last_unstale":"2021-02-10T04:37:11.196834+0000","last_undegraded":"2021-02-10T04:37:11.196834+0000","last_fullsized":"2021-02-10T04:37:11.196834+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:40.098239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:40.098239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"649","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.199170+0000","last_change":"2021-02-10T04:37:04.181509+0000","last_active":"2021-02-10T04:37:10.199170+0000","last_peered":"2021-02-10T04:37:10.199170+0000","last_clean":"2021-02-10T04:37:10.199170+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:37:10.199170+0000","last_undegraded":"2021-02-10T04:37:10.199170+0000","last_fullsized":"2021-02-10T04:37:10.199170+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:51.679515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:27:51.679515+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"538'56","reported_seq":"682","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.199902+0000","last_change":"2021-02-10T04:27:49.269619+0000","last_active":"2021-02-10T04:37:09.199902+0000","last_peered":"2021-02-10T04:37:09.199902+0000","last_clean":"2021-02-10T04:37:09.199902+0000","last_became_active":"2021-02-10T04:25:49.008451+0000","last_became_peered":"2021-02-10T04:25:49.008451+0000","last_unstale":"2021-02-10T04:37:09.199902+0000","last_undegraded":"2021-02-10T04:37:09.199902+0000","last_fullsized":"2021-02-10T04:37:09.199902+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'44","last_scrub_stamp":"2021-02-10T04:27:49.269539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:49.269539+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":107,"num_read_kb":84,"num_write":57,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"595","reported_epoch":"615","state":"active+clean","last_fresh":"2021-02-10T04:37:13.198339+0000","last_change":"2021-02-10T04:29:28.719237+0000","last_active":"2021-02-10T04:37:13.198339+0000","last_peered":"2021-02-10T04:37:13.198339+0000","last_clean":"2021-02-10T04:37:13.198339+0000","last_became_active":"2021-02-10T04:25:36.336112+0000","last_became_peered":"2021-02-10T04:25:36.336112+0000","last_unstale":"2021-02-10T04:37:13.198339+0000","last_undegraded":"2021-02-10T04:37:13.198339+0000","last_fullsized":"2021-02-10T04:37:13.198339+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.719147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.719147+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"569","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.198902+0000","last_change":"2021-02-10T04:27:49.628172+0000","last_active":"2021-02-10T04:37:10.198902+0000","last_peered":"2021-02-10T04:37:10.198902+0000","last_clean":"2021-02-10T04:37:10.198902+0000","last_became_active":"2021-02-10T04:25:51.029110+0000","last_became_peered":"2021-02-10T04:25:51.029110+0000","last_unstale":"2021-02-10T04:37:10.198902+0000","last_undegraded":"2021-02-10T04:37:10.198902+0000","last_fullsized":"2021-02-10T04:37:10.198902+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:49.628090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:49.628090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"570","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.198256+0000","last_change":"2021-02-10T04:29:25.382883+0000","last_active":"2021-02-10T04:37:09.198256+0000","last_peered":"2021-02-10T04:37:09.198256+0000","last_clean":"2021-02-10T04:37:09.198256+0000","last_became_active":"2021-02-10T04:25:53.150917+0000","last_became_peered":"2021-02-10T04:25:53.150917+0000","last_unstale":"2021-02-10T04:37:09.198256+0000","last_undegraded":"2021-02-10T04:37:09.198256+0000","last_fullsized":"2021-02-10T04:37:09.198256+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:25.382793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.382793+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.20","version":"0'0","reported_seq":"563","reported_epoch":"613","state":"active+clean","last_fresh":"2021-02-10T04:37:11.195851+0000","last_change":"2021-02-10T04:30:11.492843+0000","last_active":"2021-02-10T04:37:11.195851+0000","last_peered":"2021-02-10T04:37:11.195851+0000","last_clean":"2021-02-10T04:37:11.195851+0000","last_became_active":"2021-02-10T04:29:07.148937+0000","last_became_peered":"2021-02-10T04:29:07.148937+0000","last_unstale":"2021-02-10T04:37:11.195851+0000","last_undegraded":"2021-02-10T04:37:11.195851+0000","last_fullsized":"2021-02-10T04:37:11.195851+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:11.492788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:41.375276+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.492788+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":"612","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.199635+0000","last_change":"2021-02-10T04:29:21.708787+0000","last_active":"2021-02-10T04:37:09.199635+0000","last_peered":"2021-02-10T04:37:09.199635+0000","last_clean":"2021-02-10T04:37:09.199635+0000","last_became_active":"2021-02-10T04:28:39.045354+0000","last_became_peered":"2021-02-10T04:28:39.045354+0000","last_unstale":"2021-02-10T04:37:09.199635+0000","last_undegraded":"2021-02-10T04:37:09.199635+0000","last_fullsized":"2021-02-10T04:37:09.199635+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:21.708701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:21.708701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":"599","reported_epoch":"614","state":"active+clean","last_fresh":"2021-02-10T04:37:12.198165+0000","last_change":"2021-02-10T04:33:58.892594+0000","last_active":"2021-02-10T04:37:12.198165+0000","last_peered":"2021-02-10T04:37:12.198165+0000","last_clean":"2021-02-10T04:37:12.198165+0000","last_became_active":"2021-02-10T04:28:23.129832+0000","last_became_peered":"2021-02-10T04:28:23.129832+0000","last_unstale":"2021-02-10T04:37:12.198165+0000","last_undegraded":"2021-02-10T04:37:12.198165+0000","last_fullsized":"2021-02-10T04:37:12.198165+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:58.892504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:33:58.892504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":"535","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.198187+0000","last_change":"2021-02-10T04:28:48.409915+0000","last_active":"2021-02-10T04:37:09.198187+0000","last_peered":"2021-02-10T04:37:09.198187+0000","last_clean":"2021-02-10T04:37:09.198187+0000","last_became_active":"2021-02-10T04:28:23.129523+0000","last_became_peered":"2021-02-10T04:28:23.129523+0000","last_unstale":"2021-02-10T04:37:09.198187+0000","last_undegraded":"2021-02-10T04:37:09.198187+0000","last_fullsized":"2021-02-10T04:37:09.198187+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:48.409826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:48.409826+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":"543","reported_epoch":"614","state":"active+clean","last_fresh":"2021-02-10T04:37:12.198745+0000","last_change":"2021-02-10T04:34:01.895599+0000","last_active":"2021-02-10T04:37:12.198745+0000","last_peered":"2021-02-10T04:37:12.198745+0000","last_clean":"2021-02-10T04:37:12.198745+0000","last_became_active":"2021-02-10T04:28:23.126368+0000","last_became_peered":"2021-02-10T04:28:23.126368+0000","last_unstale":"2021-02-10T04:37:12.198745+0000","last_undegraded":"2021-02-10T04:37:12.198745+0000","last_fullsized":"2021-02-10T04:37:12.198745+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:34:01.895514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:34:01.895514+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.25","version":"53'2","reported_seq":"554","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.201051+0000","last_change":"2021-02-10T04:28:40.422198+0000","last_active":"2021-02-10T04:37:10.201051+0000","last_peered":"2021-02-10T04:37:10.201051+0000","last_clean":"2021-02-10T04:37:10.201051+0000","last_became_active":"2021-02-10T04:28:23.467770+0000","last_became_peered":"2021-02-10T04:28:23.467770+0000","last_unstale":"2021-02-10T04:37:10.201051+0000","last_undegraded":"2021-02-10T04:37:10.201051+0000","last_fullsized":"2021-02-10T04:37:10.201051+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:28:40.422115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:40.422115+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.26","version":"53'1","reported_seq":"524","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.200254+0000","last_change":"2021-02-10T04:29:22.801988+0000","last_active":"2021-02-10T04:37:09.200254+0000","last_peered":"2021-02-10T04:37:09.200254+0000","last_clean":"2021-02-10T04:37:09.200254+0000","last_became_active":"2021-02-10T04:28:39.043300+0000","last_became_peered":"2021-02-10T04:28:39.043300+0000","last_unstale":"2021-02-10T04:37:09.200254+0000","last_undegraded":"2021-02-10T04:37:09.200254+0000","last_fullsized":"2021-02-10T04:37:09.200254+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:29:22.801929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:22.801929+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":"565","reported_epoch":"612","state":"active+clean","last_fresh":"2021-02-10T04:37:10.201847+0000","last_change":"2021-02-10T04:29:07.146256+0000","last_active":"2021-02-10T04:37:10.201847+0000","last_peered":"2021-02-10T04:37:10.201847+0000","last_clean":"2021-02-10T04:37:10.201847+0000","last_became_active":"2021-02-10T04:29:07.145907+0000","last_became_peered":"2021-02-10T04:29:07.145907+0000","last_unstale":"2021-02-10T04:37:10.201847+0000","last_undegraded":"2021-02-10T04:37:10.201847+0000","last_fullsized":"2021-02-10T04:37:10.201847+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:39.388884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:39.388884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":"535","reported_epoch":"611","state":"active+clean","last_fresh":"2021-02-10T04:37:09.196035+0000","last_change":"2021-02-10T04:29:23.440451+0000","last_active":"2021-02-10T04:37:09.196035+0000","last_peered":"2021-02-10T04:37:09.196035+0000","last_clean":"2021-02-10T04:37:09.196035+0000","last_became_active":"2021-02-10T04:28:23.129469+0000","last_became_peered":"2021-02-10T04:28:23.129469+0000","last_unstale":"2021-02-10T04:37:09.196035+0000","last_undegraded":"2021-02-10T04:37:09.196035+0000","last_fullsized":"2021-02-10T04:37:09.196035+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.440391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.440391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":"538","reported_epoch":"614","state":"active+clean","last_fresh":"2021-02-10T04:37:12.197402+0000","last_change":"2021-02-10T04:28:45.220352+0000","last_active":"2021-02-10T04:37:12.197402+0000","last_peered":"2021-02-10T04:37:12.197402+0000","last_clean":"2021-02-10T04:37:12.197402+0000","last_became_active":"2021-02-10T04:28:23.126795+0000","last_became_peered":"2021-02-10T04:28:23.126795+0000","last_unstale":"2021-02-10T04:37:12.197402+0000","last_undegraded":"2021-02-10T04:37:12.197402+0000","last_fullsized":"2021-02-10T04:37:12.197402+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_clean_scrub_stamp":"2021-02-10T04:28:45.220258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":507048222,"num_objects":225,"num_object_clones":177,"num_object_copies":675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":7,"num_read":4582,"num_read_kb":4711801,"num_write":3330,"num_write_kb":1262367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1052934144,"data_stored":1482842396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8148,"ondisk_log_size":8148,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":1522,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":17,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":5835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25,"ondisk_log_size":25,"up":126,"acting":126,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":252,"num_read_kb":247,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":1878,"num_read_kb":1351,"num_write":1020,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1068,"ondisk_log_size":1068,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748364941,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":79976,"kb_used_data":61600,"kb_used_omap":0,"kb_used_meta":18368,"kb_avail":93665176,"statfs":{"total":95995035648,"available":95913140224,"internally_reserved":0,"allocated":63078400,"data_stored":100558478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":18808832},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:36:54 2021","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.752,"15min":1.2170000000000001},"min":{"1min":0.57199999999999995,"5min":0.495,"15min":0.441},"max":{"1min":7.4710000000000001,"5min":72.805999999999997,"15min":72.805999999999997},"last":1.3360000000000001},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.8300000000000001,"15min":1.27},"min":{"1min":0.52500000000000002,"5min":0.52000000000000002,"15min":0.50800000000000001},"max":{"1min":7.6970000000000001,"5min":72.721999999999994,"15min":72.721999999999994},"last":0.77400000000000002}]},{"osd":1,"last update":"Wed Feb 10 04:36:54 2021","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.97399999999999998,"15min":1.0660000000000001},"min":{"1min":0.54700000000000004,"5min":0.52100000000000002,"15min":0.46700000000000003},"max":{"1min":2.2749999999999999,"5min":6.1239999999999997,"15min":41.966000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":1.129},"min":{"1min":0.64700000000000002,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.6279999999999999,"5min":3.5750000000000002,"15min":41.804000000000002},"last":1.4470000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:36:54 2021","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.9179999999999999,"15min":1.8680000000000001},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.496,"5min":48.174999999999997,"15min":125.714},"last":1.554},{"interface":"front","average":{"1min":1.2230000000000001,"5min":2.0659999999999998,"15min":1.9119999999999999},"min":{"1min":0.68799999999999994,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":4.4130000000000003,"5min":48.277999999999999,"15min":125.53700000000001},"last":0.88800000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:36:54 2021","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.879,"15min":2.5139999999999998},"min":{"1min":0.79800000000000004,"5min":0.72399999999999998,"15min":0.63600000000000001},"max":{"1min":4.1710000000000003,"5min":35.091000000000001,"15min":192.95500000000001},"last":1.079},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.873,"15min":2.5979999999999999},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.69999999999999996},"max":{"1min":6.5300000000000002,"5min":35.061999999999998,"15min":192.601},"last":1.657}]},{"osd":4,"last update":"Wed Feb 10 04:36:54 2021","interfaces":[{"interface":"back","average":{"1min":5.4989999999999997,"5min":2.133,"15min":1.421},"min":{"1min":0.75900000000000001,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":97.733999999999995,"5min":97.733999999999995,"15min":97.733999999999995},"last":0.995},{"interface":"front","average":{"1min":5.6600000000000001,"5min":2.1899999999999999,"15min":1.492},"min":{"1min":0.73399999999999999,"5min":0.69599999999999995,"15min":0.64600000000000002},"max":{"1min":97.691000000000003,"5min":97.691000000000003,"15min":97.691000000000003},"last":1.1930000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:36:54 2021","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.613,"15min":1.351},"min":{"1min":0.83499999999999996,"5min":0.79100000000000004,"15min":0.624},"max":{"1min":2.5609999999999999,"5min":32.713999999999999,"15min":32.713999999999999},"last":1.7589999999999999},{"interface":"front","average":{"1min":1.706,"5min":1.786,"15min":1.4530000000000001},"min":{"1min":0.92900000000000005,"5min":0.753,"15min":0.753},"max":{"1min":6.6219999999999999,"5min":32.984000000000002,"15min":32.984000000000002},"last":1.526}]},{"osd":6,"last update":"Wed Feb 10 04:36:54 2021","interfaces":[{"interface":"back","average":{"1min":1.5680000000000001,"5min":1.8879999999999999,"15min":1.6459999999999999},"min":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.86899999999999999},"max":{"1min":3.218,"5min":32.616,"15min":32.616},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.6699999999999999,"5min":1.825,"15min":1.599},"min":{"1min":0.95399999999999996,"5min":0.79500000000000004,"15min":0.72499999999999998},"max":{"1min":4.282,"5min":32.424999999999997,"15min":32.424999999999997},"last":1.96}]}]},{"osd":6,"up_from":43,"seq":184683593874,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":186876,"kb_used_data":159028,"kb_used_omap":0,"kb_used_meta":27840,"kb_avail":93558276,"statfs":{"total":95995035648,"available":95803674624,"internally_reserved":0,"allocated":162844672,"data_stored":236607256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":28508160},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:36:39 2021","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":1.2969999999999999,"15min":1.389},"min":{"1min":0.53000000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.282,"5min":40.652000000000001,"15min":64.164000000000001},"last":1.3300000000000001},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.833,"15min":1.6180000000000001},"min":{"1min":0.503,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.712,"5min":51.039999999999999,"15min":64.277000000000001},"last":1.9119999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:36:39 2021","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.96699999999999997,"15min":1.1020000000000001},"min":{"1min":0.47699999999999998,"5min":0.443,"15min":0.443},"max":{"1min":2.3239999999999998,"5min":8.641,"15min":18.177},"last":1.1140000000000001},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.052,"15min":1.0900000000000001},"min":{"1min":0.60899999999999999,"5min":0.57799999999999996,"15min":0.52700000000000002},"max":{"1min":1.5169999999999999,"5min":2.4540000000000002,"15min":18.898},"last":1.145}]},{"osd":2,"last update":"Wed Feb 10 04:36:39 2021","interfaces":[{"interface":"back","average":{"1min":2.2040000000000002,"5min":1.4199999999999999,"15min":1.2509999999999999},"min":{"1min":0.72399999999999998,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":24.725000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":1.518},{"interface":"front","average":{"1min":2.0529999999999999,"5min":1.361,"15min":1.234},"min":{"1min":0.67300000000000004,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":24.611999999999998,"5min":24.611999999999998,"15min":24.611999999999998},"last":1.4419999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:36:39 2021","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.377,"15min":1.3080000000000001},"min":{"1min":0.754,"5min":0.754,"15min":0.66100000000000003},"max":{"1min":5.1630000000000003,"5min":5.1630000000000003,"15min":15.316000000000001},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.292,"15min":1.284},"min":{"1min":0.83599999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.9970000000000001,"5min":3.3300000000000001,"15min":15.35},"last":1.5660000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:36:39 2021","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.29,"15min":1.1699999999999999},"min":{"1min":0.621,"5min":0.621,"15min":0.58999999999999997},"max":{"1min":1.806,"5min":2.8090000000000002,"15min":3.4969999999999999},"last":1.982},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.012,"15min":0.96999999999999997},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.57999999999999996},"max":{"1min":1.76,"5min":2.1560000000000001,"15min":2.1560000000000001},"last":1.3879999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:36:39 2021","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2190000000000001,"15min":1.1240000000000001},"min":{"1min":0.75900000000000001,"5min":0.70899999999999996,"15min":0.64100000000000001},"max":{"1min":2.1429999999999998,"5min":3.629,"15min":5.3529999999999998},"last":1.4990000000000001},{"interface":"front","average":{"1min":1.274,"5min":1.347,"15min":1.2929999999999999},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":2.286,"5min":2.532,"15min":2.532},"last":1.661}]},{"osd":7,"last update":"Wed Feb 10 04:37:05 2021","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.27,"15min":1.202},"min":{"1min":0.89300000000000002,"5min":0.77800000000000002,"15min":0.73599999999999999},"max":{"1min":4.0999999999999996,"5min":4.2599999999999998,"15min":4.2599999999999998},"last":1.6870000000000001},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.4730000000000001,"15min":1.466},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":3.4990000000000001,"5min":4.7699999999999996,"15min":4.7699999999999996},"last":1.7729999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575011,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":172984,"kb_used_data":144944,"kb_used_omap":0,"kb_used_meta":28032,"kb_avail":93572168,"statfs":{"total":95995035648,"available":95817900032,"internally_reserved":0,"allocated":148422656,"data_stored":203726919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":28704768},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:36:11 2021","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.5780000000000001,"15min":1.1000000000000001},"min":{"1min":0.51500000000000001,"5min":0.502,"15min":0.36399999999999999},"max":{"1min":2.8079999999999998,"5min":40.539999999999999,"15min":40.539999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.651,"15min":1.177},"min":{"1min":0.53500000000000003,"5min":0.53200000000000003,"15min":0.41199999999999998},"max":{"1min":2.4020000000000001,"5min":40.831000000000003,"15min":40.831000000000003},"last":2.3130000000000002}]},{"osd":2,"last update":"Wed Feb 10 04:36:33 2021","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.5469999999999999,"15min":1.175},"min":{"1min":0.58899999999999997,"5min":0.53800000000000003,"15min":0.48699999999999999},"max":{"1min":2.2480000000000002,"5min":40.268000000000001,"15min":40.268000000000001},"last":1.8380000000000001},{"interface":"front","average":{"1min":1.141,"5min":1.5800000000000001,"15min":1.2250000000000001},"min":{"1min":0.58299999999999996,"5min":0.55600000000000005,"15min":0.46000000000000002},"max":{"1min":3.77,"5min":40.651000000000003,"15min":40.651000000000003},"last":0.84599999999999997}]},{"osd":3,"last update":"Wed Feb 10 04:36:48 2021","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.9430000000000001,"15min":1.3169999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.45200000000000001},"max":{"1min":2.5579999999999998,"5min":40.314,"15min":40.314},"last":0.995},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.9390000000000001,"15min":1.3540000000000001},"min":{"1min":0.83299999999999996,"5min":0.66200000000000003,"15min":0.47199999999999998},"max":{"1min":1.8280000000000001,"5min":40.973999999999997,"15min":40.973999999999997},"last":0.81000000000000005}]},{"osd":4,"last update":"Wed Feb 10 04:36:11 2021","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.23,"15min":1.198},"min":{"1min":0.76800000000000002,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":2.556,"5min":4.3739999999999997,"15min":4.3739999999999997},"last":6.3099999999999996},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.349,"15min":1.2669999999999999},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.65700000000000003},"max":{"1min":2.7029999999999998,"5min":6.2869999999999999,"15min":6.2869999999999999},"last":6.218}]},{"osd":5,"last update":"Wed Feb 10 04:36:27 2021","interfaces":[{"interface":"back","average":{"1min":2.0539999999999998,"5min":1.484,"15min":1.3140000000000001},"min":{"1min":0.76600000000000001,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":14.539999999999999,"5min":14.539999999999999,"15min":14.539999999999999},"last":0.96899999999999997},{"interface":"front","average":{"1min":2.0449999999999999,"5min":1.5900000000000001,"15min":1.5660000000000001},"min":{"1min":0.83799999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":14.564,"5min":14.564,"15min":14.564},"last":1.3460000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:36:38 2021","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.617,"15min":1.413},"min":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.80300000000000005},"max":{"1min":2.9369999999999998,"5min":12.395,"15min":12.395},"last":1.097},{"interface":"front","average":{"1min":1.387,"5min":1.5349999999999999,"15min":1.359},"min":{"1min":0.91500000000000004,"5min":0.80100000000000005,"15min":0.78500000000000003},"max":{"1min":5.4649999999999999,"5min":13.183,"15min":13.183},"last":1.2150000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:36:48 2021","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.534,"15min":1.573},"min":{"1min":1.026,"5min":1.026,"15min":0.83499999999999996},"max":{"1min":2.0550000000000002,"5min":3.6840000000000002,"15min":7.5540000000000003},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.577,"5min":1.6200000000000001,"15min":1.581},"min":{"1min":1.1739999999999999,"5min":0.94899999999999995,"15min":0.94899999999999995},"max":{"1min":2.4969999999999999,"5min":6.1689999999999996,"15min":8.1020000000000003},"last":1.2649999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738537,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":193208,"kb_used_data":163440,"kb_used_omap":0,"kb_used_meta":29760,"kb_avail":93551944,"statfs":{"total":95995035648,"available":95797190656,"internally_reserved":0,"allocated":167362560,"data_stored":240789238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":30474240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 04:37:06 2021","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0960000000000001,"15min":0.89200000000000002},"min":{"1min":0.54600000000000004,"5min":0.48199999999999998,"15min":0.40799999999999997},"max":{"1min":2.9039999999999999,"5min":14.781000000000001,"15min":14.781000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":1.0660000000000001,"5min":0.93700000000000006,"15min":0.90000000000000002},"min":{"1min":0.59199999999999997,"5min":0.58099999999999996,"15min":0.45000000000000001},"max":{"1min":1.9099999999999999,"5min":1.9099999999999999,"15min":7.5549999999999997},"last":1.109}]},{"osd":2,"last update":"Wed Feb 10 04:36:31 2021","interfaces":[{"interface":"back","average":{"1min":1.9239999999999999,"5min":1.3360000000000001,"15min":1.054},"min":{"1min":0.66600000000000004,"5min":0.55500000000000005,"15min":0.48099999999999998},"max":{"1min":16.254000000000001,"5min":16.254000000000001,"15min":16.254000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.865,"5min":1.2270000000000001,"15min":0.97999999999999998},"min":{"1min":0.58899999999999997,"5min":0.52800000000000002,"15min":0.495},"max":{"1min":16.829000000000001,"5min":16.829000000000001,"15min":16.829000000000001},"last":1.546}]},{"osd":3,"last update":"Wed Feb 10 04:36:48 2021","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.256,"15min":1.4850000000000001},"min":{"1min":0.626,"5min":0.55200000000000005,"15min":0.46999999999999997},"max":{"1min":2.3660000000000001,"5min":4.9829999999999997,"15min":126.587},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.569,"5min":1.6299999999999999,"15min":1.7430000000000001},"min":{"1min":1.02,"5min":0.68600000000000005,"15min":0.54200000000000004},"max":{"1min":2.1600000000000001,"5min":16.66,"15min":126.88200000000001},"last":2.1000000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:37:06 2021","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.2769999999999999,"15min":1.2010000000000001},"min":{"1min":0.85899999999999999,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":2.0510000000000002,"5min":2.0510000000000002,"15min":2.6440000000000001},"last":1.256},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.2070000000000001,"15min":1.2509999999999999},"min":{"1min":0.83399999999999996,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.669,"5min":3.887,"15min":13.891},"last":0.90500000000000003}]},{"osd":5,"last update":"Wed Feb 10 04:36:20 2021","interfaces":[{"interface":"back","average":{"1min":1.6259999999999999,"5min":1.4910000000000001,"15min":1.345},"min":{"1min":0.77800000000000002,"5min":0.76300000000000001,"15min":0.69599999999999995},"max":{"1min":3.9940000000000002,"5min":13.949999999999999,"15min":13.949999999999999},"last":1.032},{"interface":"front","average":{"1min":2.5070000000000001,"5min":1.823,"15min":1.496},"min":{"1min":1.1619999999999999,"5min":0.90900000000000003,"15min":0.81699999999999995},"max":{"1min":16.957000000000001,"5min":16.957000000000001,"15min":16.957000000000001},"last":1.1859999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:36:42 2021","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.929,"15min":1.597},"min":{"1min":0.996,"5min":0.88800000000000001,"15min":0.88800000000000001},"max":{"1min":2.8559999999999999,"5min":35.872999999999998,"15min":35.872999999999998},"last":1.234},{"interface":"front","average":{"1min":2.4300000000000002,"5min":2.194,"15min":1.671},"min":{"1min":1.3440000000000001,"5min":0.88200000000000001,"15min":0.83099999999999996},"max":{"1min":16.890000000000001,"5min":35.911000000000001,"15min":35.911000000000001},"last":1.3120000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:36:51 2021","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":1.4770000000000001,"15min":1.3480000000000001},"min":{"1min":1.0629999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":2.3679999999999999,"5min":2.3679999999999999,"15min":2.9249999999999998},"last":1.9910000000000001},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.728,"15min":1.579},"min":{"1min":1.0720000000000001,"5min":1.006,"15min":0.94999999999999996},"max":{"1min":1.8700000000000001,"5min":16.545000000000002,"15min":16.545000000000002},"last":1.339}]}]},{"osd":2,"up_from":18,"seq":77309411489,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":117440,"kb_used_data":95864,"kb_used_omap":0,"kb_used_meta":21568,"kb_avail":93627712,"statfs":{"total":95995035648,"available":95874777088,"internally_reserved":0,"allocated":98164736,"data_stored":128323565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":22085632},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,8],"upper_bound":1048576},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:36:27 2021","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.448,"15min":0.99199999999999999},"min":{"1min":0.53700000000000003,"5min":0.51100000000000001,"15min":0.48699999999999999},"max":{"1min":6.1580000000000004,"5min":27.315000000000001,"15min":27.315000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":1.407,"5min":1.46,"15min":1.012},"min":{"1min":0.57199999999999995,"5min":0.52800000000000002,"15min":0.46000000000000002},"max":{"1min":5.9379999999999997,"5min":27.286000000000001,"15min":27.286000000000001},"last":0.96499999999999997}]},{"osd":1,"last update":"Wed Feb 10 04:36:27 2021","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":1.0980000000000001,"15min":0.95999999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.518,"5min":4.2240000000000002,"15min":5.0650000000000004},"last":1.222},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.1859999999999999,"15min":0.89500000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":2.1110000000000002,"5min":10.093999999999999,"15min":10.093999999999999},"last":1.024}]},{"osd":3,"last update":"Wed Feb 10 04:36:49 2021","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.556,"15min":1.123},"min":{"1min":0.72599999999999998,"5min":0.59399999999999997,"15min":0.53500000000000003},"max":{"1min":5.6310000000000002,"5min":27.152999999999999,"15min":27.152999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":1.6379999999999999,"5min":1.774,"15min":1.333},"min":{"1min":0.83999999999999997,"5min":0.78500000000000003,"15min":0.66600000000000004},"max":{"1min":5.532,"5min":27.454999999999998,"15min":27.454999999999998},"last":1.9299999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:37:00 2021","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":2.4910000000000001,"15min":1.6599999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.5059999999999998,"5min":106.42100000000001,"15min":106.42100000000001},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.347,"5min":1.244,"15min":1.206},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.5990000000000002,"5min":5.4640000000000004,"15min":9.7609999999999992},"last":1.139}]},{"osd":5,"last update":"Wed Feb 10 04:36:27 2021","interfaces":[{"interface":"back","average":{"1min":1.8260000000000001,"5min":1.6259999999999999,"15min":1.427},"min":{"1min":1.0229999999999999,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":5.125,"5min":5.125,"15min":5.125},"last":2.1739999999999999},{"interface":"front","average":{"1min":1.387,"5min":1.571,"15min":1.3340000000000001},"min":{"1min":0.83699999999999997,"5min":0.628,"15min":0.628},"max":{"1min":2.2629999999999999,"5min":21.879999999999999,"15min":21.879999999999999},"last":1.369}]},{"osd":6,"last update":"Wed Feb 10 04:36:39 2021","interfaces":[{"interface":"back","average":{"1min":4.1769999999999996,"5min":3.0640000000000001,"15min":1.9079999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":55.145000000000003,"5min":106.13200000000001,"15min":106.13200000000001},"last":1.2549999999999999},{"interface":"front","average":{"1min":4.3390000000000004,"5min":3.1400000000000001,"15min":2.0179999999999998},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":55.101999999999997,"5min":106.568,"15min":106.568},"last":1.583}]},{"osd":7,"last update":"Wed Feb 10 04:36:56 2021","interfaces":[{"interface":"back","average":{"1min":2.0110000000000001,"5min":2.9790000000000001,"15min":2},"min":{"1min":1.1970000000000001,"5min":1.0960000000000001,"15min":0.94099999999999995},"max":{"1min":5.2439999999999998,"5min":106.706,"15min":106.706},"last":1.7949999999999999},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.4550000000000001,"15min":1.357},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":2.73,"5min":5.617,"15min":9.9220000000000006},"last":4.0819999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182555,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":265372,"kb_used_data":220372,"kb_used_omap":0,"kb_used_meta":44992,"kb_avail":93479780,"statfs":{"total":95995035648,"available":95723294720,"internally_reserved":0,"allocated":225660928,"data_stored":323060109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":46071808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:36:58 2021","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":1.0269999999999999,"15min":0.93999999999999995},"min":{"1min":0.60199999999999998,"5min":0.47599999999999998,"15min":0.46100000000000002},"max":{"1min":2.5990000000000002,"5min":6.3869999999999996,"15min":17.233000000000001},"last":0.88400000000000001},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.0960000000000001,"15min":1.097},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.52300000000000002},"max":{"1min":2.6989999999999998,"5min":3.423,"15min":20.216999999999999},"last":1.4390000000000001}]},{"osd":1,"last update":"Wed Feb 10 04:36:58 2021","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.0980000000000001,"15min":1.0069999999999999},"min":{"1min":0.70699999999999996,"5min":0.57399999999999995,"15min":0.52800000000000002},"max":{"1min":1.956,"5min":3.2629999999999999,"15min":5.9130000000000003},"last":1.3049999999999999},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.98699999999999999,"15min":0.93700000000000006},"min":{"1min":0.60399999999999998,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.74,"5min":2.5699999999999998,"15min":5.4039999999999999},"last":0.91700000000000004}]},{"osd":2,"last update":"Wed Feb 10 04:36:58 2021","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.212,"15min":1.0920000000000001},"min":{"1min":0.76500000000000001,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.8520000000000001,"5min":4.1210000000000004,"15min":4.649},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.194,"5min":1.1990000000000001,"15min":1.0589999999999999},"min":{"1min":0.73399999999999999,"5min":0.622,"15min":0.56999999999999995},"max":{"1min":1.873,"5min":3.0779999999999998,"15min":3.0779999999999998},"last":1.556}]},{"osd":4,"last update":"Wed Feb 10 04:37:00 2021","interfaces":[{"interface":"back","average":{"1min":2.774,"5min":1.6359999999999999,"15min":1.514},"min":{"1min":1.0149999999999999,"5min":0.629,"15min":0.629},"max":{"1min":30.751999999999999,"5min":30.751999999999999,"15min":30.751999999999999},"last":1.8779999999999999},{"interface":"front","average":{"1min":2.9689999999999999,"5min":1.7270000000000001,"15min":1.546},"min":{"1min":0.81299999999999994,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":30.928000000000001,"5min":30.928000000000001,"15min":30.928000000000001},"last":1.4690000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:36:24 2021","interfaces":[{"interface":"back","average":{"1min":2.2000000000000002,"5min":2.052,"15min":1.679},"min":{"1min":0.872,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":17.701000000000001,"5min":35.616999999999997,"15min":35.616999999999997},"last":2.4060000000000001},{"interface":"front","average":{"1min":2.274,"5min":2.1459999999999999,"15min":1.724},"min":{"1min":0.85999999999999999,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":17.361999999999998,"5min":35.661999999999999,"15min":35.661999999999999},"last":1.597}]},{"osd":6,"last update":"Wed Feb 10 04:36:44 2021","interfaces":[{"interface":"back","average":{"1min":2.976,"5min":2.3730000000000002,"15min":1.909},"min":{"1min":1.2829999999999999,"5min":0.96799999999999997,"15min":0.83699999999999997},"max":{"1min":17.882000000000001,"5min":35.518000000000001,"15min":35.518000000000001},"last":2.1509999999999998},{"interface":"front","average":{"1min":3.1240000000000001,"5min":2.956,"15min":2.097},"min":{"1min":1.282,"5min":0.92400000000000004,"15min":0.86599999999999999},"max":{"1min":17.015999999999998,"5min":53.965000000000003,"15min":53.965000000000003},"last":2.0569999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:36:58 2021","interfaces":[{"interface":"back","average":{"1min":1.6970000000000001,"5min":1.7310000000000001,"15min":1.732},"min":{"1min":1.214,"5min":1.0609999999999999,"15min":0.99199999999999999},"max":{"1min":4.1299999999999999,"5min":4.3460000000000001,"15min":25.242999999999999},"last":2.0150000000000001},{"interface":"front","average":{"1min":1.7649999999999999,"5min":1.788,"15min":1.7709999999999999},"min":{"1min":1.202,"5min":0.98299999999999998,"15min":0.98299999999999998},"max":{"1min":3.7610000000000001,"5min":4.3239999999999998,"15min":25.286999999999999},"last":1.9039999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986329,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94708,"kb_used_data":75564,"kb_used_omap":0,"kb_used_meta":19136,"kb_avail":93650444,"statfs":{"total":95995035648,"available":95898054656,"internally_reserved":0,"allocated":77377536,"data_stored":89278237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19595264},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4],"upper_bound":1048576},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:37:03 2021","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":0.95199999999999996,"15min":1.1499999999999999},"min":{"1min":0.56599999999999995,"5min":0.501,"15min":0.47099999999999997},"max":{"1min":10.946,"5min":10.946,"15min":62.273000000000003},"last":1.238},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.022,"15min":1.1899999999999999},"min":{"1min":0.53300000000000003,"5min":0.499,"15min":0.499},"max":{"1min":10.919,"5min":10.919,"15min":62.375999999999998},"last":0.82799999999999996}]},{"osd":1,"last update":"Wed Feb 10 04:37:03 2021","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.82999999999999996,"15min":0.86399999999999999},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.1240000000000001,"5min":5.6699999999999999,"15min":5.6699999999999999},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.95899999999999996,"5min":1.006,"15min":1.0369999999999999},"min":{"1min":0.72199999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.4670000000000001,"5min":2.9830000000000001,"15min":2.9830000000000001},"last":1.119}]},{"osd":2,"last update":"Wed Feb 10 04:37:03 2021","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0629999999999999,"15min":1.3680000000000001},"min":{"1min":0.61299999999999999,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.7250000000000001,"5min":2.8130000000000002,"15min":62.521000000000001},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.177,"15min":1.4630000000000001},"min":{"1min":0.73699999999999999,"5min":0.68100000000000005,"15min":0.60699999999999998},"max":{"1min":1.7969999999999999,"5min":3.8919999999999999,"15min":62.572000000000003},"last":1.4830000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:37:03 2021","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.252,"15min":1.4179999999999999},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":2.093,"5min":3.956,"15min":62.113},"last":1.6000000000000001},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.2030000000000001,"15min":1.3939999999999999},"min":{"1min":0.81899999999999995,"5min":0.72199999999999998,"15min":0.69899999999999995},"max":{"1min":1.625,"5min":3.9049999999999998,"15min":62.146000000000001},"last":1.764}]},{"osd":5,"last update":"Wed Feb 10 04:36:21 2021","interfaces":[{"interface":"back","average":{"1min":1.998,"5min":1.448,"15min":1.149},"min":{"1min":0.748,"5min":0.71799999999999997,"15min":0.57499999999999996},"max":{"1min":12.773999999999999,"5min":12.773999999999999,"15min":12.773999999999999},"last":2.2309999999999999},{"interface":"front","average":{"1min":2.0430000000000001,"5min":1.349,"15min":1.099},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.624},"max":{"1min":13.038,"5min":13.038,"15min":13.038},"last":1.655}]},{"osd":6,"last update":"Wed Feb 10 04:36:33 2021","interfaces":[{"interface":"back","average":{"1min":2.488,"5min":1.6020000000000001,"15min":1.446},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.63100000000000001},"max":{"1min":20.001000000000001,"5min":20.001000000000001,"15min":29.686},"last":2.331},{"interface":"front","average":{"1min":2.5979999999999999,"5min":1.738,"15min":1.5940000000000001},"min":{"1min":1.0940000000000001,"5min":0.83899999999999997,"15min":0.79000000000000004},"max":{"1min":20.292999999999999,"5min":20.292999999999999,"15min":29.510000000000002},"last":3.0489999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:36:56 2021","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.4099999999999999,"15min":1.3560000000000001},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.843,"5min":6.3440000000000003,"15min":6.3440000000000003},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.429,"5min":1.4750000000000001,"15min":1.4550000000000001},"min":{"1min":0.99199999999999999,"5min":0.88700000000000001,"15min":0.83099999999999996},"max":{"1min":2.4319999999999999,"5min":4.0780000000000003,"15min":4.0780000000000003},"last":1.7130000000000001}]}]},{"osd":5,"up_from":37,"seq":158913790102,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":132268,"kb_used_data":112164,"kb_used_omap":0,"kb_used_meta":20096,"kb_avail":93612884,"statfs":{"total":95995035648,"available":95859593216,"internally_reserved":0,"allocated":114855936,"data_stored":161934748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20578304},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:36:19 2021","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":2.1890000000000001,"15min":1.3939999999999999},"min":{"1min":0.53100000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.8260000000000001,"5min":104.04300000000001,"15min":104.04300000000001},"last":1.133},{"interface":"front","average":{"1min":0.997,"5min":2.25,"15min":1.3779999999999999},"min":{"1min":0.52600000000000002,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.72,"5min":107.776,"15min":107.776},"last":1.4670000000000001}]},{"osd":1,"last update":"Wed Feb 10 04:36:19 2021","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":2.0630000000000002,"15min":1.298},"min":{"1min":0.53700000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.4079999999999999,"5min":103.904,"15min":103.904},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.0109999999999999,"5min":2.1760000000000002,"15min":1.401},"min":{"1min":0.55300000000000005,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.3759999999999999,"5min":104.315,"15min":104.315},"last":0.66300000000000003}]},{"osd":2,"last update":"Wed Feb 10 04:36:19 2021","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":3.137,"15min":2.0259999999999998},"min":{"1min":0.63,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.895,"5min":103.65600000000001,"15min":103.65600000000001},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.141,"5min":3.1890000000000001,"15min":1.9910000000000001},"min":{"1min":0.71199999999999997,"5min":0.63800000000000001,"15min":0.63200000000000001},"max":{"1min":1.74,"5min":104.136,"15min":104.136},"last":1.2949999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:36:19 2021","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":2.7829999999999999,"15min":2.0139999999999998},"min":{"1min":0.748,"5min":0.624,"15min":0.624},"max":{"1min":4.4530000000000003,"5min":106.18000000000001,"15min":106.18000000000001},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.306,"5min":2.7160000000000002,"15min":1.946},"min":{"1min":0.84899999999999998,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.9550000000000001,"5min":106.822,"15min":106.822},"last":1.5269999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:36:19 2021","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.1020000000000001,"15min":1.05},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.653,"5min":1.881,"15min":5.2290000000000001},"last":0.91500000000000004},{"interface":"front","average":{"1min":1.361,"5min":1.2629999999999999,"15min":1.1910000000000001},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.60799999999999998},"max":{"1min":2.5129999999999999,"5min":2.5129999999999999,"15min":8.1449999999999996},"last":1.2110000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:36:37 2021","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.6890000000000001,"15min":1.431},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":2.8719999999999999,"5min":25.969000000000001,"15min":25.969000000000001},"last":1.6870000000000001},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.5800000000000001,"15min":1.3049999999999999},"min":{"1min":0.82199999999999995,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.9780000000000002,"5min":25.902999999999999,"15min":25.902999999999999},"last":1.877}]},{"osd":7,"last update":"Wed Feb 10 04:36:57 2021","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.3140000000000001,"15min":1.284},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.78400000000000003},"max":{"1min":2.746,"5min":4.6699999999999999,"15min":5.4770000000000003},"last":1.105},{"interface":"front","average":{"1min":1.5669999999999999,"5min":1.5409999999999999,"15min":1.504},"min":{"1min":0.96599999999999997,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":2.54,"5min":2.9729999999999999,"15min":8.7840000000000007},"last":1.4530000000000001}]}]}]}} 2021-02-10T04:37:26.691 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2021-02-10T04:37:26.692 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:37:26.692 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2021-02-10T04:37:26.692 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd out 2 2021-02-10T04:37:27.082 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:37:27.328 INFO:tasks.rados.rados.0.smithi085.stdout:2421: done (11 left) 2021-02-10T04:37:27.328 INFO:tasks.rados.rados.0.smithi085.stdout:2422: done (10 left) 2021-02-10T04:37:27.328 INFO:tasks.rados.rados.0.smithi085.stdout:2424: done (9 left) 2021-02-10T04:37:27.328 INFO:tasks.rados.rados.0.smithi085.stdout:2425: snap_remove snap 254 2021-02-10T04:37:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:27 smithi164 conmon[29288]: cluster 2021-02-10T04:37:26. 2021-02-10T04:37:28.911 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:27 smithi164 conmon[29288]: 322649+0000 mon.a (mon.0) 1490 : cluster [DBG] osdmap e628: 8 total, 8 up, 8 in 2021-02-10T04:37:28.912 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set nodeep-scrub 2021-02-10T04:37:28.913 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:37:28.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:27 smithi085 conmon[35961]: cluster 2021-02-10T04:37:26.322649+0000 2021-02-10T04:37:28.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:27 smithi085 conmon[35961]: mon.a (mon.0) 1490 : cluster [DBG] osdmap e628: 8 total, 8 up, 8 in 2021-02-10T04:37:28.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:28 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:37:28.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:28 smithi085 conmon[35961]: 04:37:27.324682+0000 mon.a (mon.0) 1491 : cluster [DBG] osdmap e629: 8 total, 8 up, 8 in 2021-02-10T04:37:28.917 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:28 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:37:28.917 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:28 smithi085 conmon[35961]: 10T04:37:28.137959+0000 mgr.y (mgr.14140) 673 : 2021-02-10T04:37:28.917 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:28 smithi085 conmon[35961]: cluster [DBG] pgmap v1094: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 478 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 2.2 MiB/s wr, 15 op/s 2021-02-10T04:37:28.917 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:27 smithi085 conmon[29319]: cluster 2021-02-10T04:37:28.917 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:27 smithi085 conmon[29319]: 2021-02-10T04:37:28.917 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:27 smithi085 conmon[29319]: 2021 2021-02-10T04:37:28.917 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:27 smithi085 conmon[29319]: - 2021-02-10T04:37:28.917 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:27 smithi085 conmon[29319]: 02 2021-02-10T04:37:28.917 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:27 smithi085 conmon[29319]: - 2021-02-10T04:37:28.917 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:27 smithi085 conmon[29319]: 10T04:37:26 2021-02-10T04:37:28.918 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:27 smithi085 conmon[29319]: .322649+0000 mon.a (mon.0) 1490 : cluster [DBG] osdmap e628: 8 total, 8 up, 8 in 2021-02-10T04:37:28.918 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:28 smithi085 conmon[29319]: cluster 2021-02-10T04:37:28.918 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:28 smithi085 conmon[29319]: 2021-02-10T04:37:28.918 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:28 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:37:28.918 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:28 smithi085 conmon[29319]: 37:27.324682+0000 mon.a (mon.0) 1491 : cluster [DBG] osdmap e629: 8 total, 8 up, 8 in 2021-02-10T04:37:28.918 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:28 smithi085 conmon[29319]: cluster 2021-02-10T04:37:28.918 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:28 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:37:28.918 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:28 smithi085 conmon[29319]: 10T04:37:28.137959+0000 mgr.y (mgr.14140) 673 : cluster [DBG] 2021-02-10T04:37:28.918 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:28 smithi085 conmon[29319]: pgmap v1094: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 478 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 2.2 MiB/s wr, 15 op/s 2021-02-10T04:37:28.918 INFO:tasks.rados.rados.0.smithi085.stdout:2425: done (9 left) 2021-02-10T04:37:28.918 INFO:tasks.rados.rados.0.smithi085.stdout:2426: read oid 6 snap -1 2021-02-10T04:37:28.919 INFO:tasks.rados.rados.0.smithi085.stdout:2426: expect deleted 2021-02-10T04:37:28.919 INFO:tasks.rados.rados.0.smithi085.stdout:2427: write (excl) oid 14 current snap is 271 2021-02-10T04:37:28.919 INFO:tasks.rados.rados.0.smithi085.stdout:2427: seq_num 836 ranges {450652=790641,1721064=329588} 2021-02-10T04:37:28.919 INFO:tasks.rados.rados.0.smithi085.stdout:2427: writing smithi08575397-14 from 450652 to 1241293 tid 1 2021-02-10T04:37:28.919 INFO:tasks.rados.rados.0.smithi085.stdout:2427: writing smithi08575397-14 from 1721064 to 2050652 tid 2 2021-02-10T04:37:28.919 INFO:tasks.rados.rados.0.smithi085.stdout:2428: snap_remove snap 202 2021-02-10T04:37:28.920 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:37:28 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:37:28.920 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:28 smithi164 conmon[29288]: cluster 2021-02-10T04:37:27.324682+0000 mon.a ( 2021-02-10T04:37:28.920 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:28 smithi164 conmon[29288]: mon.0) 1491 : cluster [DBG] osdmap e629: 8 total, 8 up, 8 in 2021-02-10T04:37:28.920 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:28 smithi164 conmon[29288]: cluster 2021-02-10T04:37:28.137959+0000 mgr.y (mgr.14140 2021-02-10T04:37:28.920 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:28 smithi164 conmon[29288]: ) 673 : cluster [DBG] pgmap v1094: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 478 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 2.2 MiB/s wr, 15 op/s 2021-02-10T04:37:28.940 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:37:28.940 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:37:29.021 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:37:29.055 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:37:29.254 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:37:29.265 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:37:29.392 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:37:29 smithi085 conmon[45443]: debug 2021-02-10T04:37:29.147+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:37:29.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:29 smithi164 conmon[29288]: audit 2021-02-10T04:37:28.208128+0000 mgr.y (mgr 2021-02-10T04:37:29.392 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:29 smithi164 conmon[29288]: .14140) 674 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:37:29.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:29 smithi164 conmon[29288]: cluster 2021-02-10T04:37:28.394807+0000 mon.a (mon.0) 1492 : cluster [DBG] osdmap e630: 8 total, 8 up, 8 in 2021-02-10T04:37:29.402 INFO:tasks.rados.rados.0.smithi085.stdout:2428: done (11 left) 2021-02-10T04:37:29.402 INFO:tasks.rados.rados.0.smithi085.stdout:2429: snap_create 2021-02-10T04:37:29.403 INFO:tasks.rados.rados.0.smithi085.stdout:2427: finishing write tid 1 to smithi08575397-14 2021-02-10T04:37:29.403 INFO:tasks.rados.rados.0.smithi085.stdout:2427: finishing write tid 2 to smithi08575397-14 2021-02-10T04:37:29.403 INFO:tasks.rados.rados.0.smithi085.stdout:2427: finishing write tid 3 to smithi08575397-14 2021-02-10T04:37:29.403 INFO:tasks.rados.rados.0.smithi085.stdout:2427: finishing write tid 4 to smithi08575397-14 2021-02-10T04:37:29.403 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 881 (ObjNum 836 snap 271 seq_num 836) dirty exists 2021-02-10T04:37:29.403 INFO:tasks.rados.rados.0.smithi085.stdout:2427: left oid 14 (ObjNum 836 snap 271 seq_num 836) 2021-02-10T04:37:29.540 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:37:29.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:29 smithi085 conmon[35961]: audit 2021-02-10T04:37: 2021-02-10T04:37:29.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:29 smithi085 conmon[35961]: 28.208128+0000 mgr.y (mgr.14140) 674 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:37:29.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:29 smithi085 conmon[35961]: cluster 2021-02-10T04:37:29.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:29 smithi085 conmon[35961]: 2021-02-10T04:37: 2021-02-10T04:37:29.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:29 smithi085 conmon[35961]: 28.394807+0000 mon.a (mon.0) 2021-02-10T04:37:29.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:29 smithi085 conmon[35961]: 1492 : cluster [DBG] osdmap e630: 8 total, 8 up, 8 in 2021-02-10T04:37:29.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:29 smithi085 conmon[29319]: audit 2021-02-10T04:37:29.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:29 smithi085 conmon[29319]: 2021-02-10T04:37: 2021-02-10T04:37:29.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:29 smithi085 conmon[29319]: 28.208128+0000 mgr.y (mgr.14140) 674 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:37:29.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:29 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:37:29.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:29 smithi085 conmon[29319]: 10T04:37:28.394807+0000 mon.a (mon.0) 1492 : 2021-02-10T04:37:29.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:29 smithi085 conmon[29319]: cluster [DBG] osdmap e630: 8 total, 8 up, 8 in 2021-02-10T04:37:30.409 INFO:tasks.rados.rados.0.smithi085.stdout:2426: done (11 left) 2021-02-10T04:37:30.409 INFO:tasks.rados.rados.0.smithi085.stdout:2427: done (10 left) 2021-02-10T04:37:30.409 INFO:tasks.rados.rados.0.smithi085.stdout:2429: done (9 left) 2021-02-10T04:37:30.409 INFO:tasks.rados.rados.0.smithi085.stdout:2430: write (excl) oid 7 current snap is 272 2021-02-10T04:37:30.409 INFO:tasks.rados.rados.0.smithi085.stdout:2430: seq_num 837 ranges {467459=582243,1495899=404945} 2021-02-10T04:37:30.412 INFO:tasks.rados.rados.0.smithi085.stdout:2430: writing smithi08575397-7 from 467459 to 1049702 tid 1 2021-02-10T04:37:30.417 INFO:tasks.rados.rados.0.smithi085.stdout:2430: writing smithi08575397-7 from 1495899 to 1900844 tid 2 2021-02-10T04:37:30.417 INFO:tasks.rados.rados.0.smithi085.stdout:2431: write oid 35 current snap is 272 2021-02-10T04:37:30.417 INFO:tasks.rados.rados.0.smithi085.stdout:2431: seq_num 838 ranges {484266=657492,1554381=529885} 2021-02-10T04:37:30.424 INFO:tasks.rados.rados.0.smithi085.stdout:2431: writing smithi08575397-35 from 484266 to 1141758 tid 1 2021-02-10T04:37:30.430 INFO:tasks.rados.rados.0.smithi085.stdout:2431: writing smithi08575397-35 from 1554381 to 2084266 tid 2 2021-02-10T04:37:30.430 INFO:tasks.rados.rados.0.smithi085.stdout:2432: delete oid 40 current snap is 272 2021-02-10T04:37:30.430 INFO:tasks.rados.rados.0.smithi085.stdout:2430: finishing write tid 1 to smithi08575397-7 2021-02-10T04:37:30.430 INFO:tasks.rados.rados.0.smithi085.stdout:2430: finishing write tid 2 to smithi08575397-7 2021-02-10T04:37:30.431 INFO:tasks.rados.rados.0.smithi085.stdout:2430: finishing write tid 3 to smithi08575397-7 2021-02-10T04:37:30.431 INFO:tasks.rados.rados.0.smithi085.stdout:2430: finishing write tid 4 to smithi08575397-7 2021-02-10T04:37:30.431 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 1149 (ObjNum 837 snap 272 seq_num 837) dirty exists 2021-02-10T04:37:30.431 INFO:tasks.rados.rados.0.smithi085.stdout:2430: left oid 7 (ObjNum 837 snap 272 seq_num 837) 2021-02-10T04:37:30.431 INFO:tasks.rados.rados.0.smithi085.stdout:2431: finishing write tid 1 to smithi08575397-35 2021-02-10T04:37:30.434 INFO:tasks.rados.rados.0.smithi085.stdout:2431: finishing write tid 2 to smithi08575397-35 2021-02-10T04:37:30.434 INFO:tasks.rados.rados.0.smithi085.stdout:2431: finishing write tid 3 to smithi08575397-35 2021-02-10T04:37:30.436 INFO:tasks.rados.rados.0.smithi085.stdout:2430: done (11 left) 2021-02-10T04:37:30.436 INFO:tasks.rados.rados.0.smithi085.stdout:2432: done (10 left) 2021-02-10T04:37:30.436 INFO:tasks.rados.rados.0.smithi085.stdout:2433: rollback oid 21 current snap is 272 2021-02-10T04:37:30.436 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 21 to 265 2021-02-10T04:37:30.436 INFO:tasks.rados.rados.0.smithi085.stdout:2431: finishing write tid 4 to smithi08575397-35 2021-02-10T04:37:30.436 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 35 v 859 (ObjNum 838 snap 272 seq_num 838) dirty exists 2021-02-10T04:37:30.436 INFO:tasks.rados.rados.0.smithi085.stdout:2431: left oid 35 (ObjNum 838 snap 272 seq_num 838) 2021-02-10T04:37:30.436 INFO:tasks.rados.rados.0.smithi085.stdout:2431: done (10 left) 2021-02-10T04:37:30.436 INFO:tasks.rados.rados.0.smithi085.stdout:2434: read oid 26 snap -1 2021-02-10T04:37:30.436 INFO:tasks.rados.rados.0.smithi085.stdout:2434: expect deleted 2021-02-10T04:37:30.436 INFO:tasks.rados.rados.0.smithi085.stdout:2435: read oid 2 snap -1 2021-02-10T04:37:30.437 INFO:tasks.rados.rados.0.smithi085.stdout:2435: expect deleted 2021-02-10T04:37:30.437 INFO:tasks.rados.rados.0.smithi085.stdout:2436: rollback oid 36 current snap is 272 2021-02-10T04:37:30.437 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 36 to 230 2021-02-10T04:37:30.437 INFO:tasks.rados.rados.0.smithi085.stdout:2437: snap_create 2021-02-10T04:37:30.440 INFO:tasks.rados.rados.0.smithi085.stdout:2433: finishing rollback tid 0 to smithi08575397-21 2021-02-10T04:37:30.440 INFO:tasks.rados.rados.0.smithi085.stdout:2436: finishing rollback tid 0 to smithi08575397-36 2021-02-10T04:37:30.441 INFO:tasks.rados.rados.0.smithi085.stdout:2433: finishing rollback tid 1 to smithi08575397-21 2021-02-10T04:37:30.441 INFO:tasks.rados.rados.0.smithi085.stdout:2433: finishing rollback tid 2 to smithi08575397-21 2021-02-10T04:37:30.441 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 1300 (ObjNum 812 snap 265 seq_num 812) dirty exists 2021-02-10T04:37:30.485 INFO:tasks.rados.rados.0.smithi085.stdout:2436: finishing rollback tid 1 to smithi08575397-36 2021-02-10T04:37:30.485 INFO:tasks.rados.rados.0.smithi085.stdout:2436: finishing rollback tid 2 to smithi08575397-36 2021-02-10T04:37:30.485 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 1264 (ObjNum 720 snap 226 seq_num 720) dirty exists 2021-02-10T04:37:30.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[29319]: cluster 2021-02-10T04:37:30.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[29319]: 2021-02-10T04:37:29.398626+0000 mon.a (mon.0) 1493 : cluster [DBG] osdmap e631: 8 total, 8 up, 8 in 2021-02-10T04:37:30.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:37:30.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[29319]: T04:37:29.448851+0000 osd.3 (osd.3) 72 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:37:30.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[29319]: cluster 2021-02-10T04:37 2021-02-10T04:37:30.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[29319]: :29.449191+0000 osd.3 (osd.3) 73 : cluster 2021-02-10T04:37:30.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[29319]: [DBG] purged_snaps scrub ok 2021-02-10T04:37:30.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[29319]: cluster 2021-02-10T04:37:30. 2021-02-10T04:37:30.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[29319]: 138480+0000 mgr.y (mgr.14140) 675 : cluster [DBG] 2021-02-10T04:37:30.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[29319]: pgmap v1097: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 465 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 1.3 MiB/s wr, 8 op/s 2021-02-10T04:37:30.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[35961]: cluster 2021-02-10T04:37:30.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[35961]: 2021-02-10T04:37:30.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[35961]: 2021 2021-02-10T04:37:30.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[35961]: - 2021-02-10T04:37:30.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[35961]: 02 2021-02-10T04:37:30.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[35961]: - 2021-02-10T04:37:30.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[35961]: 10 2021-02-10T04:37:30.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[35961]: T 2021-02-10T04:37:30.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[35961]: 04:37:29.398626+0000 mon.a ( 2021-02-10T04:37:30.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[35961]: mon.0) 1493 : cluster [DBG] osdmap e631: 8 total, 8 up, 8 in 2021-02-10T04:37:30.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:37:30.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[35961]: 37:29. 2021-02-10T04:37:30.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[35961]: 448851+0000 osd.3 (osd.3) 72 2021-02-10T04:37:30.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[35961]: : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:37:30.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[35961]: cluster 2021-02-10T04:37:30.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:37:30.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[35961]: 10T04:37:29.449191+0000 osd.3 (osd.3) 73 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:37:30.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:30.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[35961]: 02-10T04:37:30.138480+0000 2021-02-10T04:37:30.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[35961]: mgr.y (mgr.14140) 675 : cluster [DBG] pgmap v1097: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 465 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 1.3 MiB/s wr, 8 op/s 2021-02-10T04:37:30.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:30 smithi085 conmon[35961]: 2021-02-10T04:37:30.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:30 smithi164 conmon[29288]: cluster 2021-02-10T04:37:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:30 smithi164 conmon[29288]: 2021-02-10T04:37:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:30 smithi164 conmon[29288]: 2021 2021-02-10T04:37:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:30 smithi164 conmon[29288]: - 2021-02-10T04:37:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:30 smithi164 conmon[29288]: 02-10T04:37: 2021-02-10T04:37:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:30 smithi164 conmon[29288]: 29.398626+0000 mon.a (mon.0) 1493 : cluster [DBG] osdmap e631: 8 total, 8 up, 8 in 2021-02-10T04:37:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:30 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:37:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:30 smithi164 conmon[29288]: 02-10T04:37:29.448851+0000 osd.3 (osd.3 2021-02-10T04:37:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:30 smithi164 conmon[29288]: ) 72 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:37:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:30 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:37:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:30 smithi164 conmon[29288]: -10T04:37:29.449191+0000 osd.3 (osd 2021-02-10T04:37:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:30 smithi164 conmon[29288]: .3) 73 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:37:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:30 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:37:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:30 smithi164 conmon[29288]: 10T04:37:30.138480 2021-02-10T04:37:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:30 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 675 : cluster [DBG] 2021-02-10T04:37:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:30 smithi164 conmon[29288]: pgmap v1097: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 465 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 1.3 MiB/s wr, 8 op/s 2021-02-10T04:37:31.408 INFO:tasks.rados.rados.0.smithi085.stdout:2433: done (13 left) 2021-02-10T04:37:31.409 INFO:tasks.rados.rados.0.smithi085.stdout:2434: done (12 left) 2021-02-10T04:37:31.409 INFO:tasks.rados.rados.0.smithi085.stdout:2435: done (11 left) 2021-02-10T04:37:31.409 INFO:tasks.rados.rados.0.smithi085.stdout:2436: done (10 left) 2021-02-10T04:37:31.409 INFO:tasks.rados.rados.0.smithi085.stdout:2437: done (9 left) 2021-02-10T04:37:31.409 INFO:tasks.rados.rados.0.smithi085.stdout:2438: delete oid 48 current snap is 273 2021-02-10T04:37:31.418 INFO:tasks.rados.rados.0.smithi085.stdout:2438: done (9 left) 2021-02-10T04:37:31.419 INFO:tasks.rados.rados.0.smithi085.stdout:2439: copy_from oid 35 from oid 16 current snap is 273 2021-02-10T04:37:31.419 INFO:tasks.rados.rados.0.smithi085.stdout:2440: write (excl) oid 31 current snap is 273 2021-02-10T04:37:31.419 INFO:tasks.rados.rados.0.smithi085.stdout:2440: seq_num 841 ranges {534687=483239,1613474=412283} 2021-02-10T04:37:31.422 INFO:tasks.rados.rados.0.smithi085.stdout:2440: writing smithi08575397-31 from 534687 to 1017926 tid 1 2021-02-10T04:37:31.429 INFO:tasks.rados.rados.0.smithi085.stdout:2440: writing smithi08575397-31 from 1613474 to 2025757 tid 2 2021-02-10T04:37:31.429 INFO:tasks.rados.rados.0.smithi085.stdout:2441: write oid 1 current snap is 273 2021-02-10T04:37:31.429 INFO:tasks.rados.rados.0.smithi085.stdout:2441: seq_num 842 ranges {551494=558488,1671956=479538} 2021-02-10T04:37:31.435 INFO:tasks.rados.rados.0.smithi085.stdout:2441: writing smithi08575397-1 from 551494 to 1109982 tid 1 2021-02-10T04:37:31.441 INFO:tasks.rados.rados.0.smithi085.stdout:2441: writing smithi08575397-1 from 1671956 to 2151494 tid 2 2021-02-10T04:37:31.442 INFO:tasks.rados.rados.0.smithi085.stdout:2442: delete oid 5 current snap is 273 2021-02-10T04:37:31.442 INFO:tasks.rados.rados.0.smithi085.stdout:2440: finishing write tid 1 to smithi08575397-31 2021-02-10T04:37:31.442 INFO:tasks.rados.rados.0.smithi085.stdout:2440: finishing write tid 2 to smithi08575397-31 2021-02-10T04:37:31.442 INFO:tasks.rados.rados.0.smithi085.stdout:2440: finishing write tid 3 to smithi08575397-31 2021-02-10T04:37:31.442 INFO:tasks.rados.rados.0.smithi085.stdout:2440: finishing write tid 4 to smithi08575397-31 2021-02-10T04:37:31.442 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 810 (ObjNum 841 snap 273 seq_num 841) dirty exists 2021-02-10T04:37:31.442 INFO:tasks.rados.rados.0.smithi085.stdout:2440: left oid 31 (ObjNum 841 snap 273 seq_num 841) 2021-02-10T04:37:31.442 INFO:tasks.rados.rados.0.smithi085.stdout:2441: finishing write tid 1 to smithi08575397-1 2021-02-10T04:37:31.444 INFO:tasks.rados.rados.0.smithi085.stdout:2440: done (12 left) 2021-02-10T04:37:31.444 INFO:tasks.rados.rados.0.smithi085.stdout:2442: done (11 left) 2021-02-10T04:37:31.445 INFO:tasks.rados.rados.0.smithi085.stdout:2443: rollback oid 7 current snap is 273 2021-02-10T04:37:31.445 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 7 to 248 2021-02-10T04:37:31.445 INFO:tasks.rados.rados.0.smithi085.stdout:2444: snap_create 2021-02-10T04:37:31.446 INFO:tasks.rados.rados.0.smithi085.stdout:2441: finishing write tid 2 to smithi08575397-1 2021-02-10T04:37:31.447 INFO:tasks.rados.rados.0.smithi085.stdout:2441: finishing write tid 3 to smithi08575397-1 2021-02-10T04:37:31.447 INFO:tasks.rados.rados.0.smithi085.stdout:2441: finishing write tid 4 to smithi08575397-1 2021-02-10T04:37:31.447 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 1137 (ObjNum 842 snap 273 seq_num 842) dirty exists 2021-02-10T04:37:31.447 INFO:tasks.rados.rados.0.smithi085.stdout:2441: left oid 1 (ObjNum 842 snap 273 seq_num 842) 2021-02-10T04:37:31.449 INFO:tasks.rados.rados.0.smithi085.stdout:2443: finishing rollback tid 0 to smithi08575397-7 2021-02-10T04:37:31.450 INFO:tasks.rados.rados.0.smithi085.stdout:2443: finishing rollback tid 1 to smithi08575397-7 2021-02-10T04:37:31.450 INFO:tasks.rados.rados.0.smithi085.stdout:2443: finishing rollback tid 2 to smithi08575397-7 2021-02-10T04:37:31.450 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 1152 (ObjNum 759 snap 243 seq_num 759) dirty exists 2021-02-10T04:37:31.601 INFO:tasks.rados.rados.0.smithi085.stdout:2439: finishing copy_from to smithi08575397-35 2021-02-10T04:37:31.602 INFO:tasks.rados.rados.0.smithi085.stdout:2439: got expected ENOENT (src dne) 2021-02-10T04:37:31.602 INFO:tasks.rados.rados.0.smithi085.stdout:2439: finishing copy_from racing read to smithi08575397-35 2021-02-10T04:37:31.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:31 smithi085 conmon[29319]: cluster 2021-02-10T04:37:31.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:31 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:37:31.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:31 smithi085 conmon[29319]: :37:30.404868+0000 mon.a (mon.0) 1494 : cluster [DBG] osdmap e632: 8 total, 8 up, 8 in 2021-02-10T04:37:31.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:31 smithi085 conmon[35961]: cluster 2021-02-10T04:37:31.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:31 smithi085 conmon[35961]: 2021-02-10T04:37:31.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:31 smithi085 conmon[35961]: 2021-02-10T04:37:30 2021-02-10T04:37:31.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:31 smithi085 conmon[35961]: .404868+0000 mon.a (mon.0) 1494 : cluster [DBG] osdmap e632: 8 total, 8 up, 8 in 2021-02-10T04:37:31.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:31 smithi164 conmon[29288]: cluster 2021-02-10T04:37:30.404868+0000 mon.a ( 2021-02-10T04:37:31.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:31 smithi164 conmon[29288]: mon.0) 1494 : cluster [DBG] osdmap e632: 8 total, 8 up, 8 in 2021-02-10T04:37:32.415 INFO:tasks.rados.rados.0.smithi085.stdout:2439: done (12 left) 2021-02-10T04:37:32.415 INFO:tasks.rados.rados.0.smithi085.stdout:2441: done (11 left) 2021-02-10T04:37:32.415 INFO:tasks.rados.rados.0.smithi085.stdout:2443: done (10 left) 2021-02-10T04:37:32.415 INFO:tasks.rados.rados.0.smithi085.stdout:2444: done (9 left) 2021-02-10T04:37:32.415 INFO:tasks.rados.rados.0.smithi085.stdout:2445: snap_remove snap 268 2021-02-10T04:37:32.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:32 smithi085 conmon[29319]: cluster 2021-02-10T04:37:32.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:32 smithi085 conmon[29319]: 2021-02-10T04:37: 2021-02-10T04:37:32.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:32 smithi085 conmon[29319]: 31.408519+0000 mon.a (mon.0) 1495 : cluster [DBG] osdmap e633: 8 total, 8 up, 8 in 2021-02-10T04:37:32.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:32 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:32.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:32 smithi085 conmon[29319]: 02-10T04:37:32.138951+0000 mgr.y ( 2021-02-10T04:37:32.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:32 smithi085 conmon[29319]: mgr.14140) 676 : cluster [DBG] pgmap v1100: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 460 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 1.1 MiB/s wr, 8 op/s 2021-02-10T04:37:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:32 smithi085 conmon[35961]: cluster 2021-02-10T04:37:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:32 smithi085 conmon[35961]: 2021-02-10T04:37 2021-02-10T04:37:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:32 smithi085 conmon[35961]: :31.408519+0000 mon.a (mon.0) 1495 : cluster [DBG] osdmap e633: 8 total, 8 up, 8 in 2021-02-10T04:37:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:32 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:37:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:32 smithi085 conmon[35961]: T04:37:32.138951+0000 mgr.y 2021-02-10T04:37:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:32 smithi085 conmon[35961]: (mgr.14140) 676 : cluster [DBG] pgmap v1100: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 460 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 1.1 MiB/s wr, 8 op/s 2021-02-10T04:37:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:32 smithi164 conmon[29288]: cluster 2021-02-10T04:37:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:32 smithi164 conmon[29288]: 2021-02-10T04:37:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:32 smithi164 conmon[29288]: 2021 2021-02-10T04:37:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:32 smithi164 conmon[29288]: - 2021-02-10T04:37:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:32 smithi164 conmon[29288]: 02 2021-02-10T04:37:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:32 smithi164 conmon[29288]: - 2021-02-10T04:37:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:32 smithi164 conmon[29288]: 10 2021-02-10T04:37:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:32 smithi164 conmon[29288]: T04:37:31.408519+0000 mon.a 2021-02-10T04:37:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:32 smithi164 conmon[29288]: (mon.0) 1495 : cluster [DBG] osdmap e633: 8 total, 8 up, 8 in 2021-02-10T04:37:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:32 smithi164 conmon[29288]: cluster 2021-02-10T04:37: 2021-02-10T04:37:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:32 smithi164 conmon[29288]: 32.138951+0000 mgr.y (mgr.14140) 676 : cluster 2021-02-10T04:37:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:32 smithi164 conmon[29288]: [DBG] pgmap v1100: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 460 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.8 MiB/s rd, 1.1 MiB/s wr, 8 op/s 2021-02-10T04:37:33.205 INFO:tasks.rados.rados.0.smithi085.stdout:2445: done (9 left) 2021-02-10T04:37:33.205 INFO:tasks.rados.rados.0.smithi085.stdout:2446: rollback oid 20 current snap is 274 2021-02-10T04:37:33.205 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 20 to 271 2021-02-10T04:37:33.205 INFO:tasks.rados.rados.0.smithi085.stdout:2447: snap_create 2021-02-10T04:37:33.218 INFO:tasks.rados.rados.0.smithi085.stdout:2446: finishing rollback tid 0 to smithi08575397-20 2021-02-10T04:37:33.218 INFO:tasks.rados.rados.0.smithi085.stdout:2446: finishing rollback tid 1 to smithi08575397-20 2021-02-10T04:37:33.218 INFO:tasks.rados.rados.0.smithi085.stdout:2446: finishing rollback tid 2 to smithi08575397-20 2021-02-10T04:37:33.218 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 20 v 1141 (ObjNum 759 snap 243 seq_num 759) dirty exists 2021-02-10T04:37:33.575 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:37:33.609 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:37:33.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:33 smithi085 conmon[35961]: cluster 2021-02-10T04:37:32.414486+0000 mon.a (mon. 2021-02-10T04:37:33.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:33 smithi085 conmon[35961]: 0) 1496 : cluster [DBG] osdmap e634: 8 total, 8 up, 8 in 2021-02-10T04:37:33.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:33 smithi085 conmon[35961]: cluster 2021-02-10T04:37:33. 2021-02-10T04:37:33.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:33 smithi085 conmon[35961]: 204806+0000 mon.a (mon.0) 1497 : cluster [DBG] osdmap e635: 8 total, 8 up, 8 in 2021-02-10T04:37:33.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:33 smithi085 conmon[29319]: cluster 2021-02-10T04:37:32.414486+0000 2021-02-10T04:37:33.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:33 smithi085 conmon[29319]: mon.a (mon.0) 1496 : cluster [DBG] osdmap e634: 8 total, 8 up, 8 in 2021-02-10T04:37:33.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:33 smithi085 conmon[29319]: cluster 2021-02-10T04:37 2021-02-10T04:37:33.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:33 smithi085 conmon[29319]: :33.204806+0000 mon.a (mon.0) 1497 : cluster [DBG] osdmap e635: 8 total, 8 up, 8 in 2021-02-10T04:37:33.828 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:37:33.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:33 smithi164 conmon[29288]: cluster 2021-02-10T04:37:32.414486+0000 2021-02-10T04:37:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:33 smithi164 conmon[29288]: mon.a (mon.0) 1496 : cluster [DBG] osdmap e634: 8 total, 8 up, 8 in 2021-02-10T04:37:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:33 smithi164 conmon[29288]: cluster 2021-02-10T04:37:33.204806+0000 mon.a (mon.0) 1497 : cluster [DBG] osdmap e635: 8 total, 8 up, 8 in 2021-02-10T04:37:33.979 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:37:34.005 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:37:33 smithi085 conmon[52836]: debug 2021-02-10T04:37:33.732+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:37:34.010 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:37:34.178 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:37:34.212 INFO:tasks.rados.rados.0.smithi085.stdout:2446: done (10 left) 2021-02-10T04:37:34.212 INFO:tasks.rados.rados.0.smithi085.stdout:2447: done (9 left) 2021-02-10T04:37:34.212 INFO:tasks.rados.rados.0.smithi085.stdout:2448: write (excl) oid 31 current snap is 275 2021-02-10T04:37:34.212 INFO:tasks.rados.rados.0.smithi085.stdout:2448: seq_num 844 ranges {585108=425339,1788920=396188} 2021-02-10T04:37:34.212 INFO:tasks.rados.rados.0.smithi085.stdout:2448: writing smithi08575397-31 from 585108 to 1010447 tid 1 2021-02-10T04:37:34.215 INFO:tasks.rados.rados.0.smithi085.stdout:2448: writing smithi08575397-31 from 1788920 to 2185108 tid 2 2021-02-10T04:37:34.215 INFO:tasks.rados.rados.0.smithi085.stdout:2449: snap_create 2021-02-10T04:37:34.215 INFO:tasks.rados.rados.0.smithi085.stdout:2448: finishing write tid 1 to smithi08575397-31 2021-02-10T04:37:34.218 INFO:tasks.rados.rados.0.smithi085.stdout:2448: finishing write tid 2 to smithi08575397-31 2021-02-10T04:37:34.218 INFO:tasks.rados.rados.0.smithi085.stdout:2448: finishing write tid 3 to smithi08575397-31 2021-02-10T04:37:34.219 INFO:tasks.rados.rados.0.smithi085.stdout:2448: finishing write tid 4 to smithi08575397-31 2021-02-10T04:37:34.219 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 813 (ObjNum 844 snap 275 seq_num 844) dirty exists 2021-02-10T04:37:34.219 INFO:tasks.rados.rados.0.smithi085.stdout:2448: left oid 31 (ObjNum 844 snap 275 seq_num 844) 2021-02-10T04:37:34.289 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:37:34.320 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:37:34 smithi085 conmon[49106]: debug 2021-02-10T04:37:34.141+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:37:34.323 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:37:34.495 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:37:34.600 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:37:34.627 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:34 smithi085 conmon[29319]: cluster 2021-02-10T04:37:34.627 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:34 smithi085 conmon[29319]: 2021-02-10T04:37:34.627 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:34 smithi085 conmon[29319]: 2021-02-10T04:37:33. 2021-02-10T04:37:34.627 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:34 smithi085 conmon[29319]: 740342+0000 osd.0 (osd.0) 74 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:37:34.627 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:34 smithi085 conmon[29319]: cluster 2021-02-10T04:37:34.627 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:34 smithi085 conmon[29319]: 2021-02-10T04:37:33.740654 2021-02-10T04:37:34.627 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:34 smithi085 conmon[29319]: +0000 osd.0 (osd.0) 75 : cluster 2021-02-10T04:37:34.627 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:34 smithi085 conmon[29319]: [DBG] purged_snaps scrub ok 2021-02-10T04:37:34.627 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:34 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:37:34.627 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:34 smithi085 conmon[29319]: 37:34.139520+0000 mgr.y (mgr.14140 2021-02-10T04:37:34.627 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:34 smithi085 conmon[29319]: ) 677 : cluster [DBG] pgmap v1103: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 443 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 975 KiB/s wr, 9 op/s 2021-02-10T04:37:34.628 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:34 smithi085 conmon[29319]: cluster 2021-02-10T04:37:34.628 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:34 smithi085 conmon[29319]: 2021-02-10T04:37:34.202164 2021-02-10T04:37:34.628 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:34 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1498 : cluster [DBG] osdmap e636: 8 total, 8 up, 8 in 2021-02-10T04:37:34.628 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:34 smithi085 conmon[35961]: cluster 2021-02-10T04:37:33.740342+0000 osd.0 (osd.0) 74 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:37:34.628 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:34 smithi085 conmon[35961]: cluster 2021-02-10T04:37:33.740654+0000 osd.0 (osd.0) 75 : cluster [DBG] 2021-02-10T04:37:34.628 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:34 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:37:34.628 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:34 smithi085 conmon[35961]: cluster 2021-02-10T04:37:34.139520+0000 mgr.y (mgr 2021-02-10T04:37:34.628 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:34 smithi085 conmon[35961]: .14140) 677 : cluster [DBG] pgmap v1103: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 443 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 975 KiB/s wr, 9 op/s 2021-02-10T04:37:34.628 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:34 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:37:34.628 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:34 smithi085 conmon[35961]: 04:37:34.202164+0000 mon.a ( 2021-02-10T04:37:34.628 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:34 smithi085 conmon[35961]: mon.0) 1498 : cluster [DBG] osdmap e636: 8 total, 8 up, 8 in 2021-02-10T04:37:34.628 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:34 smithi085 conmon[35961]: 2021-02-10T04:37:34.629 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:37:34 smithi085 conmon[45443]: debug 2021-02-10T04:37:34.469+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:37:34.629 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:37:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:34 smithi164 conmon[29288]: cluster 2021-02-10T04:37:33.740342+0000 osd.0 (osd.0) 74 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:37:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:34 smithi164 conmon[29288]: cluster 2021-02-10T04:37:33.740654+0000 osd.0 (osd.0) 75 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:37:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:34 smithi164 conmon[29288]: cluster 2021-02-10T04:37:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:34 smithi164 conmon[29288]: 2021-02-10T04:37:34.139520+0000 mgr.y (mgr.14140) 677 : cluster [DBG] pgmap v1103: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 443 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 975 KiB/s wr, 9 op/s 2021-02-10T04:37:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:34 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:37:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:34 smithi164 conmon[29288]: 04:37:34.202164+0000 mon.a (mon.0) 1498 : cluster 2021-02-10T04:37:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:34 smithi164 conmon[29288]: [DBG] osdmap e636: 8 total, 8 up, 8 in 2021-02-10T04:37:35.203 INFO:tasks.rados.rados.0.smithi085.stdout:2448: done (10 left) 2021-02-10T04:37:35.203 INFO:tasks.rados.rados.0.smithi085.stdout:2449: done (9 left) 2021-02-10T04:37:35.204 INFO:tasks.rados.rados.0.smithi085.stdout:2450: delete oid 31 current snap is 276 2021-02-10T04:37:35.215 INFO:tasks.rados.rados.0.smithi085.stdout:2450: done (9 left) 2021-02-10T04:37:35.215 INFO:tasks.rados.rados.0.smithi085.stdout:2451: write oid 35 current snap is 276 2021-02-10T04:37:35.215 INFO:tasks.rados.rados.0.smithi085.stdout:2451: seq_num 846 ranges {618722=575837,1905884=312838} 2021-02-10T04:37:35.217 INFO:tasks.rados.rados.0.smithi085.stdout:2451: writing smithi08575397-35 from 618722 to 1194559 tid 1 2021-02-10T04:37:35.224 INFO:tasks.rados.rados.0.smithi085.stdout:2451: writing smithi08575397-35 from 1905884 to 2218722 tid 2 2021-02-10T04:37:35.225 INFO:tasks.rados.rados.0.smithi085.stdout:2451: finishing write tid 1 to smithi08575397-35 2021-02-10T04:37:35.225 INFO:tasks.rados.rados.0.smithi085.stdout:2452: write oid 31 current snap is 276 2021-02-10T04:37:35.225 INFO:tasks.rados.rados.0.smithi085.stdout:2452: seq_num 847 ranges {635529=651086,1848013=387516} 2021-02-10T04:37:35.233 INFO:tasks.rados.rados.0.smithi085.stdout:2452: writing smithi08575397-31 from 635529 to 1286615 tid 1 2021-02-10T04:37:35.240 INFO:tasks.rados.rados.0.smithi085.stdout:2452: writing smithi08575397-31 from 1848013 to 2235529 tid 2 2021-02-10T04:37:35.240 INFO:tasks.rados.rados.0.smithi085.stdout:2453: delete oid 28 current snap is 276 2021-02-10T04:37:35.240 INFO:tasks.rados.rados.0.smithi085.stdout:2451: finishing write tid 2 to smithi08575397-35 2021-02-10T04:37:35.240 INFO:tasks.rados.rados.0.smithi085.stdout:2451: finishing write tid 3 to smithi08575397-35 2021-02-10T04:37:35.240 INFO:tasks.rados.rados.0.smithi085.stdout:2451: finishing write tid 4 to smithi08575397-35 2021-02-10T04:37:35.240 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 35 v 862 (ObjNum 846 snap 276 seq_num 846) dirty exists 2021-02-10T04:37:35.240 INFO:tasks.rados.rados.0.smithi085.stdout:2451: left oid 35 (ObjNum 846 snap 276 seq_num 846) 2021-02-10T04:37:35.240 INFO:tasks.rados.rados.0.smithi085.stdout:2452: finishing write tid 1 to smithi08575397-31 2021-02-10T04:37:35.242 INFO:tasks.rados.rados.0.smithi085.stdout:2451: done (11 left) 2021-02-10T04:37:35.242 INFO:tasks.rados.rados.0.smithi085.stdout:2453: done (10 left) 2021-02-10T04:37:35.242 INFO:tasks.rados.rados.0.smithi085.stdout:2454: write oid 12 current snap is 276 2021-02-10T04:37:35.243 INFO:tasks.rados.rados.0.smithi085.stdout:2454: seq_num 849 ranges {669143=401584,1564977=426959} 2021-02-10T04:37:35.249 INFO:tasks.rados.rados.0.smithi085.stdout:2454: writing smithi08575397-12 from 669143 to 1070727 tid 1 2021-02-10T04:37:35.255 INFO:tasks.rados.rados.0.smithi085.stdout:2454: writing smithi08575397-12 from 1564977 to 1991936 tid 2 2021-02-10T04:37:35.255 INFO:tasks.rados.rados.0.smithi085.stdout:2452: finishing write tid 2 to smithi08575397-31 2021-02-10T04:37:35.255 INFO:tasks.rados.rados.0.smithi085.stdout:2452: finishing write tid 3 to smithi08575397-31 2021-02-10T04:37:35.255 INFO:tasks.rados.rados.0.smithi085.stdout:2452: finishing write tid 4 to smithi08575397-31 2021-02-10T04:37:35.255 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 817 (ObjNum 847 snap 276 seq_num 847) dirty exists 2021-02-10T04:37:35.255 INFO:tasks.rados.rados.0.smithi085.stdout:2452: left oid 31 (ObjNum 847 snap 276 seq_num 847) 2021-02-10T04:37:35.255 INFO:tasks.rados.rados.0.smithi085.stdout:2454: finishing write tid 1 to smithi08575397-12 2021-02-10T04:37:35.255 INFO:tasks.rados.rados.0.smithi085.stdout:2452: done (10 left) 2021-02-10T04:37:35.255 INFO:tasks.rados.rados.0.smithi085.stdout:2455: snap_remove snap 270 2021-02-10T04:37:35.686 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:37:35.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:37:35 smithi085 conmon[49106]: debug 2021-02-10T04:37:35.470+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:37:36.214 INFO:tasks.rados.rados.0.smithi085.stdout:2455: done (10 left) 2021-02-10T04:37:36.214 INFO:tasks.rados.rados.0.smithi085.stdout:2456: delete oid 1 current snap is 276 2021-02-10T04:37:36.214 INFO:tasks.rados.rados.0.smithi085.stdout:2454: finishing write tid 2 to smithi08575397-12 2021-02-10T04:37:36.214 INFO:tasks.rados.rados.0.smithi085.stdout:2454: finishing write tid 3 to smithi08575397-12 2021-02-10T04:37:36.214 INFO:tasks.rados.rados.0.smithi085.stdout:2454: finishing write tid 4 to smithi08575397-12 2021-02-10T04:37:36.214 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 885 (ObjNum 849 snap 276 seq_num 849) dirty exists 2021-02-10T04:37:36.214 INFO:tasks.rados.rados.0.smithi085.stdout:2454: left oid 12 (ObjNum 849 snap 276 seq_num 849) 2021-02-10T04:37:36.229 INFO:teuthology.orchestra.run.smithi085.stderr:marked out osd.2. 2021-02-10T04:37:36.230 INFO:tasks.rados.rados.0.smithi085.stdout:2454: done (10 left) 2021-02-10T04:37:36.230 INFO:tasks.rados.rados.0.smithi085.stdout:2456: done (9 left) 2021-02-10T04:37:36.230 INFO:tasks.rados.rados.0.smithi085.stdout:2457: write oid 37 current snap is 276 2021-02-10T04:37:36.230 INFO:tasks.rados.rados.0.smithi085.stdout:2457: seq_num 851 ranges {702757=552082,1681941=430628} 2021-02-10T04:37:36.237 INFO:tasks.rados.rados.0.smithi085.stdout:2457: writing smithi08575397-37 from 702757 to 1254839 tid 1 2021-02-10T04:37:36.243 INFO:tasks.rados.rados.0.smithi085.stdout:2457: writing smithi08575397-37 from 1681941 to 2112569 tid 2 2021-02-10T04:37:36.243 INFO:tasks.rados.rados.0.smithi085.stdout:2458: write (excl) oid 50 current snap is 276 2021-02-10T04:37:36.243 INFO:tasks.rados.rados.0.smithi085.stdout:2458: seq_num 852 ranges {719564=743684,2140423=179141} 2021-02-10T04:37:36.253 INFO:tasks.rados.rados.0.smithi085.stdout:2458: writing smithi08575397-50 from 719564 to 1463248 tid 1 2021-02-10T04:37:36.259 INFO:tasks.rados.rados.0.smithi085.stdout:2458: writing smithi08575397-50 from 2140423 to 2319564 tid 2 2021-02-10T04:37:36.259 INFO:tasks.rados.rados.0.smithi085.stdout:2459: snap_remove snap 248 2021-02-10T04:37:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:36 smithi085 conmon[29319]: debug 2021-02-10T04:37:36.074+0000 7f9e105d6700 -1 mon.a@0(leader).osd e637 definitely_dead 0 2021-02-10T04:37:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:36 smithi085 conmon[29319]: cluster 2021-02-10T04:37:35.202958+0000 mon.a 2021-02-10T04:37:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:36 smithi085 conmon[29319]: (mon.0) 1499 : cluster [DBG] osdmap e637: 8 total, 8 up, 8 in 2021-02-10T04:37:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:36 smithi085 conmon[29319]: audit 2021-02-10T04:37:36.075146+0000 mon.a (mon.0) 1500 : audit [INF] from='client.? 172.21.15.85:0/3294666876' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2021-02-10T04:37:36.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:36 smithi085 conmon[29319]: cluster 2021-02-10T04:37:36. 2021-02-10T04:37:36.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:36 smithi085 conmon[29319]: 075274+0000 mon.a (mon.0) 1501 : cluster [INF] Client client.admin marked osd.2 out, while it was still marked up 2021-02-10T04:37:36.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:36 smithi085 conmon[29319]: cluster 2021-02-10T04:37: 2021-02-10T04:37:36.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:36 smithi085 conmon[29319]: 36.140238+0000 mgr.y (mgr.14140) 678 : cluster [DBG] pgmap v1106: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 429 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 MiB/s wr, 11 op/s 2021-02-10T04:37:36.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:36 smithi085 conmon[35961]: cluster 2021-02-10T04:37:36.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:36 smithi085 conmon[35961]: 2021-02-10T04:37:36.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:36 smithi085 conmon[35961]: 2021-02-10T04:37:35. 2021-02-10T04:37:36.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:36 smithi085 conmon[35961]: 202958+0000 mon.a (mon.0) 1499 : cluster [DBG] osdmap e637: 8 total, 8 up, 8 in 2021-02-10T04:37:36.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:36 smithi085 conmon[35961]: audit 2021-02-10T04:37:36.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:36 smithi085 conmon[35961]: 2021-02-10T04:37:36.075146+0000 2021-02-10T04:37:36.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:36 smithi085 conmon[35961]: mon.a (mon.0) 1500 : audit [INF] from='client.? 172.21.15.85:0/3294666876' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2021-02-10T04:37:36.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:36 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:37:36.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:36 smithi085 conmon[35961]: 10T04:37:36.075274+0000 mon.a (mon.0) 2021-02-10T04:37:36.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:36 smithi085 conmon[35961]: 1501 : cluster [INF] Client client.admin marked osd.2 out, while it was still marked up 2021-02-10T04:37:36.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:36 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:36.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:36 smithi085 conmon[35961]: 02-10T04:37:36.140238+0000 2021-02-10T04:37:36.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:36 smithi085 conmon[35961]: mgr.y (mgr.14140) 678 : cluster [DBG] pgmap v1106: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 429 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 MiB/s wr, 11 op/s 2021-02-10T04:37:36.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: cluster 2021-02-10T04:37:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: 2021-02-10T04:37:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: 2021 2021-02-10T04:37:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: - 2021-02-10T04:37:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: 02 2021-02-10T04:37:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: - 2021-02-10T04:37:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: 10 2021-02-10T04:37:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: T 2021-02-10T04:37:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: 04 2021-02-10T04:37:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: : 2021-02-10T04:37:36.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: 37 2021-02-10T04:37:36.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: : 2021-02-10T04:37:36.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: 35 2021-02-10T04:37:36.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: . 2021-02-10T04:37:36.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: 202958 2021-02-10T04:37:36.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: +0000 2021-02-10T04:37:36.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: 2021-02-10T04:37:36.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: mon.a 2021-02-10T04:37:36.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: ( 2021-02-10T04:37:36.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: mon 2021-02-10T04:37:36.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: . 2021-02-10T04:37:36.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: 0 2021-02-10T04:37:36.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: ) 2021-02-10T04:37:36.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: 1499 2021-02-10T04:37:36.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: : 2021-02-10T04:37:36.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: cluster 2021-02-10T04:37:36.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: 2021-02-10T04:37:36.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: [DBG] 2021-02-10T04:37:36.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: 2021-02-10T04:37:36.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: osdmap e637: 8 total, 8 up, 8 in 2021-02-10T04:37:36.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: 2021-02-10T04:37:36.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: audit 2021-02-10T04:37:36.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: 2021-02-10T04:37:36.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: 2021 2021-02-10T04:37:36.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: - 2021-02-10T04:37:36.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: 02 2021-02-10T04:37:36.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: - 2021-02-10T04:37:36.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: 10 2021-02-10T04:37:36.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: T 2021-02-10T04:37:36.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: 04 2021-02-10T04:37:36.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: : 2021-02-10T04:37:36.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: 37 2021-02-10T04:37:36.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: : 2021-02-10T04:37:36.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: 36 2021-02-10T04:37:36.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: . 2021-02-10T04:37:36.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: 075146+0000 mon.a (mon.0) 2021-02-10T04:37:36.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: 1500 : audit [INF] from='client.? 172.21.15.85:0/3294666876' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2021-02-10T04:37:36.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:37:36.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: -10T04:37:36.075274+0000 2021-02-10T04:37:36.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: mon.a (mon.0) 1501 : cluster [INF] Client client.admin marked osd.2 out, while it was still marked up 2021-02-10T04:37:36.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: 2021-02-10T04:37:36.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:37:36.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: :37:36.140238+0000 mgr.y (mgr. 2021-02-10T04:37:36.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:36 smithi164 conmon[29288]: 14140) 678 : cluster [DBG] pgmap v1106: 168 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+scrubbing, 8 active+clean+snaptrim, 155 active+clean; 429 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 MiB/s wr, 11 op/s 2021-02-10T04:37:37.216 INFO:tasks.rados.rados.0.smithi085.stdout:2459: done (11 left) 2021-02-10T04:37:37.217 INFO:tasks.rados.rados.0.smithi085.stdout:2460: rollback oid 40 current snap is 276 2021-02-10T04:37:37.217 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 40 to 275 2021-02-10T04:37:37.217 INFO:tasks.rados.rados.0.smithi085.stdout:2457: finishing write tid 1 to smithi08575397-37 2021-02-10T04:37:37.217 INFO:tasks.rados.rados.0.smithi085.stdout:2457: finishing write tid 2 to smithi08575397-37 2021-02-10T04:37:37.217 INFO:tasks.rados.rados.0.smithi085.stdout:2457: finishing write tid 3 to smithi08575397-37 2021-02-10T04:37:37.217 INFO:tasks.rados.rados.0.smithi085.stdout:2457: finishing write tid 4 to smithi08575397-37 2021-02-10T04:37:37.217 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 1310 (ObjNum 851 snap 276 seq_num 851) dirty exists 2021-02-10T04:37:37.217 INFO:tasks.rados.rados.0.smithi085.stdout:2457: left oid 37 (ObjNum 851 snap 276 seq_num 851) 2021-02-10T04:37:37.217 INFO:tasks.rados.rados.0.smithi085.stdout:2458: finishing write tid 1 to smithi08575397-50 2021-02-10T04:37:37.217 INFO:tasks.rados.rados.0.smithi085.stdout:2458: finishing write tid 2 to smithi08575397-50 2021-02-10T04:37:37.217 INFO:tasks.rados.rados.0.smithi085.stdout:2458: finishing write tid 3 to smithi08575397-50 2021-02-10T04:37:37.218 INFO:tasks.rados.rados.0.smithi085.stdout:2458: finishing write tid 4 to smithi08575397-50 2021-02-10T04:37:37.218 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 1141 (ObjNum 852 snap 276 seq_num 852) dirty exists 2021-02-10T04:37:37.218 INFO:tasks.rados.rados.0.smithi085.stdout:2458: left oid 50 (ObjNum 852 snap 276 seq_num 852) 2021-02-10T04:37:37.218 INFO:tasks.rados.rados.0.smithi085.stdout:2457: done (11 left) 2021-02-10T04:37:37.218 INFO:tasks.rados.rados.0.smithi085.stdout:2458: done (10 left) 2021-02-10T04:37:37.218 INFO:tasks.rados.rados.0.smithi085.stdout:2461: snap_create 2021-02-10T04:37:37.248 INFO:tasks.rados.rados.0.smithi085.stdout:2460: finishing rollback tid 1 to smithi08575397-40 2021-02-10T04:37:37.249 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 1296 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-02-10T04:37:37.276 INFO:tasks.rados.rados.0.smithi085.stdout:430: finishing copy_from to smithi08575397-25 2021-02-10T04:37:37.276 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 25 v 60 (ObjNum 172 snap 37 seq_num 172) dirty exists 2021-02-10T04:37:37.276 INFO:tasks.rados.rados.0.smithi085.stdout:430: finishing copy_from racing read to smithi08575397-25 2021-02-10T04:37:37.485 INFO:tasks.rados.rados.0.smithi085.stdout:683: finishing copy_from to smithi08575397-18 2021-02-10T04:37:37.485 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 18 v 121 (ObjNum 266 snap 64 seq_num 266) dirty exists 2021-02-10T04:37:37.485 INFO:tasks.rados.rados.0.smithi085.stdout:683: finishing copy_from racing read to smithi08575397-18 2021-02-10T04:37:37.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:37 smithi164 conmon[29288]: audit 2021-02-10T04:37:36.209547+0000 mon.a ( 2021-02-10T04:37:37.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:37 smithi164 conmon[29288]: mon.0) 1502 : audit [INF] from='client.? 172.21.15.85:0/3294666876' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2021-02-10T04:37:37.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:37 smithi164 conmon[29288]: cluster 2021-02-10T04:37:36.209772+0000 mon.a (mon.0) 1503 : cluster [DBG] osdmap e638: 8 total, 8 up, 7 in 2021-02-10T04:37:37.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[29319]: audit 2021-02-10T04:37:36.209547+0000 mon.a (mon 2021-02-10T04:37:37.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[29319]: .0) 1502 : audit [INF] from='client.? 172.21.15.85:0/3294666876' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2021-02-10T04:37:37.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[29319]: cluster 2021-02-10T04:37:36.209772+0000 mon.a (mon.0) 1503 : cluster [DBG] osdmap e638: 8 total, 8 up, 7 in 2021-02-10T04:37:37.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: audit 2021-02-10T04:37:37.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: 2021-02-10T04:37:37.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: 2021 2021-02-10T04:37:37.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: - 2021-02-10T04:37:37.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: 02 2021-02-10T04:37:37.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: - 2021-02-10T04:37:37.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: 10 2021-02-10T04:37:37.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: T 2021-02-10T04:37:37.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: 04 2021-02-10T04:37:37.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: : 2021-02-10T04:37:37.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: 37 2021-02-10T04:37:37.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: : 2021-02-10T04:37:37.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: 36 2021-02-10T04:37:37.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: . 2021-02-10T04:37:37.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: 209547 2021-02-10T04:37:37.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: +0000 2021-02-10T04:37:37.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: 2021-02-10T04:37:37.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: mon.a 2021-02-10T04:37:37.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: ( 2021-02-10T04:37:37.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: mon 2021-02-10T04:37:37.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: . 2021-02-10T04:37:37.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: 0 2021-02-10T04:37:37.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: ) 2021-02-10T04:37:37.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: 1502 2021-02-10T04:37:37.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: : 2021-02-10T04:37:37.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: audit 2021-02-10T04:37:37.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: 2021-02-10T04:37:37.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: [INF] 2021-02-10T04:37:37.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: 2021-02-10T04:37:37.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/3294666876' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2021-02-10T04:37:37.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: 2021-02-10T04:37:37.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: cluster 2021-02-10T04:37:37.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: 2021-02-10T04:37:37.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: 2021 2021-02-10T04:37:37.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: - 2021-02-10T04:37:37.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: 02 2021-02-10T04:37:37.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: - 2021-02-10T04:37:37.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: 10 2021-02-10T04:37:37.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: T 2021-02-10T04:37:37.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: 04 2021-02-10T04:37:37.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: : 2021-02-10T04:37:37.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: 37 2021-02-10T04:37:37.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: : 2021-02-10T04:37:37.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: 36 2021-02-10T04:37:37.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: . 2021-02-10T04:37:37.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: 209772 2021-02-10T04:37:37.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: +0000 2021-02-10T04:37:37.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: 2021-02-10T04:37:37.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: mon.a 2021-02-10T04:37:37.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: ( 2021-02-10T04:37:37.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: mon 2021-02-10T04:37:37.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: . 2021-02-10T04:37:37.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: 0 2021-02-10T04:37:37.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: ) 2021-02-10T04:37:37.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: 1503 2021-02-10T04:37:37.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: : 2021-02-10T04:37:37.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: cluster 2021-02-10T04:37:37.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: 2021-02-10T04:37:37.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: [DBG] 2021-02-10T04:37:37.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: 2021-02-10T04:37:37.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: osdmap e638: 8 total, 8 up, 7 in 2021-02-10T04:37:37.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:37 smithi085 conmon[35961]: 2021-02-10T04:37:37.857 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.0 dump_ops_in_flight 2021-02-10T04:37:38.224 INFO:tasks.rados.rados.0.smithi085.stdout:430: done (10 left) 2021-02-10T04:37:38.224 INFO:tasks.rados.rados.0.smithi085.stdout:683: done (9 left) 2021-02-10T04:37:38.224 INFO:tasks.rados.rados.0.smithi085.stdout:2460: done (8 left) 2021-02-10T04:37:38.224 INFO:tasks.rados.rados.0.smithi085.stdout:2461: done (7 left) 2021-02-10T04:37:38.224 INFO:tasks.rados.rados.0.smithi085.stdout:2462: read oid 48 snap -1 2021-02-10T04:37:38.224 INFO:tasks.rados.rados.0.smithi085.stdout:2462: expect deleted 2021-02-10T04:37:38.224 INFO:tasks.rados.rados.0.smithi085.stdout:2463: delete oid 13 current snap is 277 2021-02-10T04:37:38.244 INFO:tasks.rados.rados.0.smithi085.stdout:2462: done (8 left) 2021-02-10T04:37:38.244 INFO:tasks.rados.rados.0.smithi085.stdout:2463: done (7 left) 2021-02-10T04:37:38.244 INFO:tasks.rados.rados.0.smithi085.stdout:2464: rollback oid 22 current snap is 277 2021-02-10T04:37:38.244 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 22 to 272 2021-02-10T04:37:38.244 INFO:tasks.rados.rados.0.smithi085.stdout:2465: read oid 35 snap -1 2021-02-10T04:37:38.244 INFO:tasks.rados.rados.0.smithi085.stdout:2465: expect (ObjNum 846 snap 276 seq_num 846) 2021-02-10T04:37:38.244 INFO:tasks.rados.rados.0.smithi085.stdout:2466: write (excl) oid 21 current snap is 277 2021-02-10T04:37:38.244 INFO:tasks.rados.rados.0.smithi085.stdout:2466: seq_num 854 ranges {753178=494182,1857387=495791} 2021-02-10T04:37:38.246 INFO:tasks.rados.rados.0.smithi085.stdout:2466: writing smithi08575397-21 from 753178 to 1247360 tid 1 2021-02-10T04:37:38.258 INFO:tasks.rados.rados.0.smithi085.stdout:2466: writing smithi08575397-21 from 1857387 to 2353178 tid 2 2021-02-10T04:37:38.265 INFO:tasks.rados.rados.0.smithi085.stdout:628: finishing copy_from to smithi08575397-15 2021-02-10T04:37:38.265 INFO:tasks.rados.rados.0.smithi085.stdout:628: got expected ENOENT (src dne) 2021-02-10T04:37:38.265 INFO:tasks.rados.rados.0.smithi085.stdout:628: finishing copy_from racing read to smithi08575397-15 2021-02-10T04:37:38.265 INFO:tasks.rados.rados.0.smithi085.stdout:2465: expect (ObjNum 846 snap 276 seq_num 846) 2021-02-10T04:37:38.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[29319]: cluster 2021-02-10T04:37:37.215023+0000 mon.a (mon 2021-02-10T04:37:38.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[29319]: .0) 1504 : cluster [DBG] osdmap e639: 8 total, 8 up, 7 in 2021-02-10T04:37:38.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[29319]: audit 2021-02-10T04:37:38.113368+0000 mon.a (mon.0) 1505 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:37:38.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[29319]: audit 2021-02-10T04:37:38.113595+0000 mon.a (mon.0) 1506 2021-02-10T04:37:38.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[29319]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:37:38.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[29319]: cluster 2021-02-10T04:37 2021-02-10T04:37:38.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[29319]: :38.141008+0000 mgr.y (mgr.14140) 2021-02-10T04:37:38.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[29319]: 679 : cluster [DBG] pgmap v1109: 168 pgs: 1 remapped+peering, 12 peering, 4 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 143 active+clean; 425 MiB data, 991 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 734 KiB/s wr, 7 op/s; 0 B/s, 0 objects/s recovering 2021-02-10T04:37:38.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:37:38.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[29319]: 37:38.214142+0000 mon.a (mon.0 2021-02-10T04:37:38.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[29319]: ) 1507 : cluster [DBG] osdmap e640: 8 total, 8 up, 7 in 2021-02-10T04:37:38.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: cluster 2021-02-10T04:37:38.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: 2021-02-10T04:37:38.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: 2021 2021-02-10T04:37:38.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: - 2021-02-10T04:37:38.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: 02 2021-02-10T04:37:38.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: - 2021-02-10T04:37:38.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: 10 2021-02-10T04:37:38.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: T 2021-02-10T04:37:38.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: 04 2021-02-10T04:37:38.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: : 2021-02-10T04:37:38.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: 37 2021-02-10T04:37:38.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: : 2021-02-10T04:37:38.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: 37 2021-02-10T04:37:38.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: . 2021-02-10T04:37:38.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: 215023 2021-02-10T04:37:38.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: +0000 2021-02-10T04:37:38.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: 2021-02-10T04:37:38.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: mon.a 2021-02-10T04:37:38.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: ( 2021-02-10T04:37:38.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: mon 2021-02-10T04:37:38.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: . 2021-02-10T04:37:38.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: 0) 1504 : cluster [DBG] osdmap e639: 8 total, 8 up, 7 in 2021-02-10T04:37:38.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: audit 2021-02-10T04:37:38.113368 2021-02-10T04:37:38.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1505 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:37:38.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:37:38.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: -10T04:37:38.113595+0000 mon.a (mon. 2021-02-10T04:37:38.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: 0) 1506 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:37:38.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:37:38.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: 37:38.141008+0000 mgr.y (mgr.14140) 679 2021-02-10T04:37:38.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: : cluster [DBG] pgmap v1109: 168 pgs: 1 remapped+peering, 12 peering, 4 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 143 active+clean; 425 MiB data, 991 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 734 KiB/s wr, 7 op/s; 0 B/s, 0 objects/s recovering 2021-02-10T04:37:38.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: cluster 2021-02-10T04:37:38 2021-02-10T04:37:38.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: .214142+0000 mon.a (mon.0) 1507 : cluster 2021-02-10T04:37:38.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:38 smithi085 conmon[35961]: [DBG] osdmap e640: 8 total, 8 up, 7 in 2021-02-10T04:37:38.469 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:37:38.487 INFO:tasks.rados.rados.0.smithi085.stdout:628: done (9 left) 2021-02-10T04:37:38.487 INFO:tasks.rados.rados.0.smithi085.stdout:2465: done (8 left) 2021-02-10T04:37:38.488 INFO:tasks.rados.rados.0.smithi085.stdout:2467: rollback oid 2 current snap is 277 2021-02-10T04:37:38.488 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 2 to 256 2021-02-10T04:37:38.488 INFO:tasks.rados.rados.0.smithi085.stdout:2464: finishing rollback tid 1 to smithi08575397-22 2021-02-10T04:37:38.488 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 801 (ObjNum 1 snap 32576 seq_num 1006676224) dirty dne 2021-02-10T04:37:38.488 INFO:tasks.rados.rados.0.smithi085.stdout:2466: finishing write tid 1 to smithi08575397-21 2021-02-10T04:37:38.488 INFO:tasks.rados.rados.0.smithi085.stdout:1482: finishing copy_from to smithi08575397-32 2021-02-10T04:37:38.488 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 32 v 627 (ObjNum 85 snap 11 seq_num 85) dirty exists 2021-02-10T04:37:38.488 INFO:tasks.rados.rados.0.smithi085.stdout:1482: finishing copy_from racing read to smithi08575397-32 2021-02-10T04:37:38.488 INFO:tasks.rados.rados.0.smithi085.stdout:2466: finishing write tid 2 to smithi08575397-21 2021-02-10T04:37:38.488 INFO:tasks.rados.rados.0.smithi085.stdout:2466: finishing write tid 3 to smithi08575397-21 2021-02-10T04:37:38.488 INFO:tasks.rados.rados.0.smithi085.stdout:2466: finishing write tid 4 to smithi08575397-21 2021-02-10T04:37:38.488 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 1315 (ObjNum 854 snap 277 seq_num 854) dirty exists 2021-02-10T04:37:38.488 INFO:tasks.rados.rados.0.smithi085.stdout:2466: left oid 21 (ObjNum 854 snap 277 seq_num 854) 2021-02-10T04:37:38.488 INFO:tasks.rados.rados.0.smithi085.stdout:242: finishing copy_from to smithi08575397-29 2021-02-10T04:37:38.489 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 29 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2021-02-10T04:37:38.489 INFO:tasks.rados.rados.0.smithi085.stdout:242: finishing copy_from racing read to smithi08575397-29 2021-02-10T04:37:38.489 INFO:tasks.rados.rados.0.smithi085.stdout:242: done (8 left) 2021-02-10T04:37:38.489 INFO:tasks.rados.rados.0.smithi085.stdout:1482: done (7 left) 2021-02-10T04:37:38.489 INFO:tasks.rados.rados.0.smithi085.stdout:2464: done (6 left) 2021-02-10T04:37:38.489 INFO:tasks.rados.rados.0.smithi085.stdout:2466: done (5 left) 2021-02-10T04:37:38.489 INFO:tasks.rados.rados.0.smithi085.stdout:2468: write oid 6 current snap is 277 2021-02-10T04:37:38.489 INFO:tasks.rados.rados.0.smithi085.stdout:2468: seq_num 855 ranges {769985=569431,1915869=437966} 2021-02-10T04:37:38.496 INFO:tasks.rados.rados.0.smithi085.stdout:2468: writing smithi08575397-6 from 769985 to 1339416 tid 1 2021-02-10T04:37:38.502 INFO:tasks.rados.rados.0.smithi085.stdout:2468: writing smithi08575397-6 from 1915869 to 2353835 tid 2 2021-02-10T04:37:38.502 INFO:tasks.rados.rados.0.smithi085.stdout:2467: finishing rollback tid 1 to smithi08575397-2 2021-02-10T04:37:38.503 INFO:tasks.rados.rados.0.smithi085.stdout:2467: finishing rollback tid 2 to smithi08575397-2 2021-02-10T04:37:38.503 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 864 (ObjNum 735 snap 234 seq_num 735) dirty exists 2021-02-10T04:37:38.503 INFO:tasks.rados.rados.0.smithi085.stdout:2468: finishing write tid 1 to smithi08575397-6 2021-02-10T04:37:38.503 INFO:tasks.rados.rados.0.smithi085.stdout:2467: done (5 left) 2021-02-10T04:37:38.503 INFO:tasks.rados.rados.0.smithi085.stdout:2469: copy_from oid 45 from oid 12 current snap is 277 2021-02-10T04:37:38.503 INFO:tasks.rados.rados.0.smithi085.stdout:2470: write (excl) oid 14 current snap is 277 2021-02-10T04:37:38.503 INFO:tasks.rados.rados.0.smithi085.stdout:2470: seq_num 856 ranges {786792=644680,1857998=528794} 2021-02-10T04:37:38.512 INFO:tasks.rados.rados.0.smithi085.stdout:2470: writing smithi08575397-14 from 786792 to 1431472 tid 1 2021-02-10T04:37:38.518 INFO:tasks.rados.rados.0.smithi085.stdout:2470: writing smithi08575397-14 from 1857998 to 2386792 tid 2 2021-02-10T04:37:38.518 INFO:tasks.rados.rados.0.smithi085.stdout:2471: snap_remove snap 190 2021-02-10T04:37:38.643 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:37:38 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:37:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:38 smithi164 conmon[29288]: cluster 2021-02-10T04:37:37.215023+0000 mon.a (mon 2021-02-10T04:37:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:38 smithi164 conmon[29288]: .0) 1504 : cluster [DBG] osdmap e639: 8 total, 8 up, 7 in 2021-02-10T04:37:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:38 smithi164 conmon[29288]: audit 2021-02-10T04:37:38.113368+0000 mon.a (mon.0) 1505 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:37:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:38 smithi164 conmon[29288]: audit 2021-02-10T04:37:38.113595+0000 2021-02-10T04:37:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:38 smithi164 conmon[29288]: mon.a (mon.0) 1506 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:37:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:38 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:37:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:38 smithi164 conmon[29288]: -10T04:37:38.141008+0000 mgr.y (mgr.14140) 679 2021-02-10T04:37:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:38 smithi164 conmon[29288]: : cluster [DBG] pgmap v1109: 168 pgs: 1 remapped+peering, 12 peering, 4 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 143 active+clean; 425 MiB data, 991 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 734 KiB/s wr, 7 op/s; 0 B/s, 0 objects/s recovering 2021-02-10T04:37:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:38 smithi164 conmon[29288]: cluster 2021-02-10T04:37:38.214142 2021-02-10T04:37:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:38 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1507 : cluster [DBG] osdmap e640: 8 total, 8 up, 7 in 2021-02-10T04:37:39.248 INFO:tasks.rados.rados.0.smithi085.stdout:2468: finishing write tid 2 to smithi08575397-6 2021-02-10T04:37:39.248 INFO:tasks.rados.rados.0.smithi085.stdout:2468: finishing write tid 3 to smithi08575397-6 2021-02-10T04:37:39.248 INFO:tasks.rados.rados.0.smithi085.stdout:2468: finishing write tid 4 to smithi08575397-6 2021-02-10T04:37:39.248 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 1095 (ObjNum 855 snap 277 seq_num 855) dirty exists 2021-02-10T04:37:39.248 INFO:tasks.rados.rados.0.smithi085.stdout:2468: left oid 6 (ObjNum 855 snap 277 seq_num 855) 2021-02-10T04:37:39.248 INFO:tasks.rados.rados.0.smithi085.stdout:2469: finishing copy_from to smithi08575397-45 2021-02-10T04:37:39.248 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 45 v 885 (ObjNum 849 snap 276 seq_num 849) dirty exists 2021-02-10T04:37:39.248 INFO:tasks.rados.rados.0.smithi085.stdout:2469: finishing copy_from racing read to smithi08575397-45 2021-02-10T04:37:39.248 INFO:tasks.rados.rados.0.smithi085.stdout:2470: finishing write tid 1 to smithi08575397-14 2021-02-10T04:37:39.249 INFO:tasks.rados.rados.0.smithi085.stdout:2470: finishing write tid 2 to smithi08575397-14 2021-02-10T04:37:39.249 INFO:tasks.rados.rados.0.smithi085.stdout:2470: finishing write tid 3 to smithi08575397-14 2021-02-10T04:37:39.249 INFO:tasks.rados.rados.0.smithi085.stdout:2470: finishing write tid 4 to smithi08575397-14 2021-02-10T04:37:39.249 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 888 (ObjNum 856 snap 277 seq_num 856) dirty exists 2021-02-10T04:37:39.249 INFO:tasks.rados.rados.0.smithi085.stdout:2470: left oid 14 (ObjNum 856 snap 277 seq_num 856) 2021-02-10T04:37:39.249 INFO:tasks.rados.rados.0.smithi085.stdout:2468: done (7 left) 2021-02-10T04:37:39.249 INFO:tasks.rados.rados.0.smithi085.stdout:2469: done (6 left) 2021-02-10T04:37:39.249 INFO:tasks.rados.rados.0.smithi085.stdout:2470: done (5 left) 2021-02-10T04:37:39.249 INFO:tasks.rados.rados.0.smithi085.stdout:2471: done (4 left) 2021-02-10T04:37:39.249 INFO:tasks.rados.rados.0.smithi085.stdout:2472: snap_remove snap 256 2021-02-10T04:37:39.565 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:39 smithi085 conmon[35961]: audit 2021-02-10T04:37:38.218869+0000 mgr.y (mgr.14140) 680 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:37:39.565 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:39 smithi085 conmon[35961]: cluster 2021-02-10T04:37:38.222882+0000 mon.a (mon.0) 1508 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2021-02-10T04:37:39.565 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:39 smithi085 conmon[29319]: audit 2021-02-10T04:37:39.565 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:39 smithi085 conmon[29319]: 2021-02-10T04:37:39.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:39 smithi085 conmon[29319]: 2021 2021-02-10T04:37:39.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:39 smithi085 conmon[29319]: -02-10T04:37:38 2021-02-10T04:37:39.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:39 smithi085 conmon[29319]: .218869+0000 mgr.y (mgr.14140) 680 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:37:39.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:39 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:37:39.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:39 smithi085 conmon[29319]: -02-10T04:37:38.222882+0000 2021-02-10T04:37:39.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:39 smithi085 conmon[29319]: mon.a (mon.0) 1508 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2021-02-10T04:37:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:39 smithi164 conmon[29288]: audit 2021-02-10T04:37:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:39 smithi164 conmon[29288]: 2021-02-10T04:37: 2021-02-10T04:37:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:39 smithi164 conmon[29288]: 38.218869+0000 mgr.y (mgr.14140) 680 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:37:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:39 smithi164 conmon[29288]: cluster 2021-02-10T04:37:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:39 smithi164 conmon[29288]: 2021-02-10T04:37:38.222882+0000 mon.a (mon 2021-02-10T04:37:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:39 smithi164 conmon[29288]: .0) 1508 : 2021-02-10T04:37:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:39 smithi164 conmon[29288]: cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2021-02-10T04:37:40.260 INFO:tasks.rados.rados.0.smithi085.stdout:2472: done (4 left) 2021-02-10T04:37:46.041 INFO:tasks.rados.rados.0.smithi085.stdout:2473: delete oid 31 current snap is 277 2021-02-10T04:37:46.054 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:40 smithi164 conmon[29288]: cluster 2021-02-10T04:37:39.247998 2021-02-10T04:37:46.054 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:40 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1509 : cluster [DBG] osdmap e641: 8 total, 8 up, 7 in 2021-02-10T04:37:46.054 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:40 smithi164 conmon[29288]: cluster 2021-02-10T04:37:39.944582+0000 osd.7 ( 2021-02-10T04:37:46.054 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:40 smithi164 conmon[29288]: osd.7) 56 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:37:46.054 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:40 smithi164 conmon[29288]: cluster 2021-02-10T04:37:46.055 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:40 smithi164 conmon[29288]: 2021-02-10T04:37:39.944869+0000 osd.7 (osd.7) 57 2021-02-10T04:37:46.055 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:40 smithi164 conmon[29288]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:37:46.055 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:40 smithi164 conmon[29288]: cluster 2021-02-10T04:37 2021-02-10T04:37:46.055 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:40 smithi164 conmon[29288]: :40.141794+0000 mgr.y (mgr.14140 2021-02-10T04:37:46.055 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:40 smithi164 conmon[29288]: ) 681 : cluster [DBG] pgmap v1112: 168 pgs: 1 remapped+peering, 12 peering, 5 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 142 active+clean; 422 MiB data, 1002 MiB used, 625 GiB / 626 GiB avail; 3.4 MiB/s wr, 8 op/s; 198 KiB/s, 8 objects/s recovering 2021-02-10T04:37:46.055 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:40 smithi164 conmon[29288]: 2021-02-10T04:37:46.055 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:41 smithi164 conmon[29288]: cluster 2021-02-10T04:37:40. 2021-02-10T04:37:46.055 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:41 smithi164 conmon[29288]: 258251+0000 mon.a (mon.0) 1510 : cluster [DBG] osdmap e642: 8 total, 8 up, 7 in 2021-02-10T04:37:46.055 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:42 smithi164 conmon[29288]: cluster 2021-02-10T04:37:41.259643+0000 mon.a ( 2021-02-10T04:37:46.055 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:42 smithi164 conmon[29288]: mon.0) 1511 : cluster [DBG] osdmap e643: 8 total, 8 up, 7 in 2021-02-10T04:37:46.055 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:42 smithi164 conmon[29288]: cluster 2021-02-10T04:37:42.142444+0000 mgr.y (mgr.14140) 682 : cluster [DBG] pgmap v1115: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 154 active+clean, 6 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 1 remapped+peering, 2 active+recovering+undersized+remapped; 416 MiB data, 1022 MiB used, 625 GiB / 626 GiB avail; 3.4 MiB/s wr, 8 op/s; 1/1674 objects degraded (0.060%); 51/1674 objects misplaced (3.047%); 475 KiB/s, 40 objects/s recovering 2021-02-10T04:37:46.055 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:43 smithi164 conmon[29288]: cluster 2021-02-10T04:37:46.055 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:43 smithi164 conmon[29288]: 2021-02-10T04:37 2021-02-10T04:37:46.055 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:43 smithi164 conmon[29288]: :42.263957+0000 mon.a (mon.0) 1512 : cluster [WRN] Health check failed: Degraded data redundancy: 1/1674 objects degraded (0.060%), 1 pg degraded (PG_DEGRADED) 2021-02-10T04:37:46.056 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:43 smithi164 conmon[29288]: cluster 2021-02-10T04:37:42 2021-02-10T04:37:46.056 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:43 smithi164 conmon[29288]: .278316+0000 mon.a (mon.0) 1513 : cluster [DBG] 2021-02-10T04:37:46.056 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:43 smithi164 conmon[29288]: osdmap e644: 8 total, 8 up, 7 in 2021-02-10T04:37:46.056 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:44 smithi164 conmon[29288]: cluster 2021-02-10T04:37:46.056 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:44 smithi164 conmon[29288]: 2021-02-10T04:37: 2021-02-10T04:37:46.056 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:44 smithi164 conmon[29288]: 43.305886+0000 mon.a (mon.0) 1514 : cluster [DBG] osdmap e645: 8 total, 8 up, 7 in 2021-02-10T04:37:46.056 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:44 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:37:46.056 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:44 smithi164 conmon[29288]: -10T04:37:44.143194+0000 mgr.y 2021-02-10T04:37:46.056 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:44 smithi164 conmon[29288]: (mgr.14140) 683 : cluster [DBG] pgmap v1118: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 154 active+clean, 6 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 2 active+recovering+undersized+remapped; 394 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 14 op/s; 1/1656 objects degraded (0.060%); 51/1656 objects misplaced (3.080%); 3.1 MiB/s, 35 objects/s recovering 2021-02-10T04:37:46.056 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:45 smithi164 conmon[29288]: cluster 2021-02-10T04:37:46.056 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:45 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:37:46.056 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:45 smithi164 conmon[29288]: :37:44.312687+0000 mon.a (mon.0) 1515 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2021-02-10T04:37:46.056 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:45 smithi164 conmon[29288]: cluster 2021-02-10T04:37:46.057 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:45 smithi164 conmon[29288]: 2021-02-10 2021-02-10T04:37:46.057 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:45 smithi164 conmon[29288]: T04:37:44.328648+0000 mon.a (mon.0) 1516 : cluster 2021-02-10T04:37:46.057 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:45 smithi164 conmon[29288]: [DBG] osdmap e646: 8 total, 8 up, 7 in 2021-02-10T04:37:46.057 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:45 smithi164 conmon[29288]: audit 2021-02-10T04:37:44.519178+0000 2021-02-10T04:37:46.057 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:45 smithi164 conmon[29288]: mon.c (mon.1) 60 : audit [DBG] from='client.? 172.21.15.85:0/3738548356' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:37:46.057 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:45 smithi164 conmon[29288]: audit 2021-02-10T04:37:44.853319 2021-02-10T04:37:46.057 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:45 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1517 : audit [INF] from='client.? 172.21.15.85:0/1726137331' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:37:46.057 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:45 smithi164 conmon[33677]: debug 2021-02-10T04:37:45.289+0000 7fbec19a1700 -1 osd.4 646 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:46.057 INFO:tasks.rados.rados.0.smithi085.stdout:2473: done (4 left) 2021-02-10T04:37:46.057 INFO:tasks.rados.rados.0.smithi085.stdout:2474: read oid 6 snap -1 2021-02-10T04:37:46.058 INFO:tasks.rados.rados.0.smithi085.stdout:2474: expect (ObjNum 855 snap 277 seq_num 855) 2021-02-10T04:37:46.058 INFO:tasks.rados.rados.0.smithi085.stdout:2475: delete oid 25 current snap is 277 2021-02-10T04:37:46.058 INFO:tasks.rados.rados.0.smithi085.stdout:2475: done (5 left) 2021-02-10T04:37:46.058 INFO:tasks.rados.rados.0.smithi085.stdout:2476: write (excl) oid 34 current snap is 277 2021-02-10T04:37:46.058 INFO:tasks.rados.rados.0.smithi085.stdout:2476: seq_num 859 ranges {437213=470427,1633444=728951} 2021-02-10T04:37:46.058 INFO:tasks.rados.rados.0.smithi085.stdout:2476: writing smithi08575397-34 from 437213 to 907640 tid 1 2021-02-10T04:37:46.058 INFO:tasks.rados.rados.0.smithi085.stdout:2476: writing smithi08575397-34 from 1633444 to 2362395 tid 2 2021-02-10T04:37:46.058 INFO:tasks.rados.rados.0.smithi085.stdout:2474: expect (ObjNum 855 snap 277 seq_num 855) 2021-02-10T04:37:46.058 INFO:tasks.rados.rados.0.smithi085.stdout:2476: finishing write tid 1 to smithi08575397-34 2021-02-10T04:37:46.058 INFO:tasks.rados.rados.0.smithi085.stdout:2476: finishing write tid 2 to smithi08575397-34 2021-02-10T04:37:46.058 INFO:tasks.rados.rados.0.smithi085.stdout:2476: finishing write tid 3 to smithi08575397-34 2021-02-10T04:37:46.058 INFO:tasks.rados.rados.0.smithi085.stdout:2476: finishing write tid 4 to smithi08575397-34 2021-02-10T04:37:46.058 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 34 v 777 (ObjNum 859 snap 277 seq_num 859) dirty exists 2021-02-10T04:37:46.059 INFO:tasks.rados.rados.0.smithi085.stdout:2476: left oid 34 (ObjNum 859 snap 277 seq_num 859) 2021-02-10T04:37:46.059 INFO:tasks.rados.rados.0.smithi085.stdout:2474: done (5 left) 2021-02-10T04:37:46.059 INFO:tasks.rados.rados.0.smithi085.stdout:2476: done (4 left) 2021-02-10T04:37:46.059 INFO:tasks.rados.rados.0.smithi085.stdout:2477: read oid 12 snap -1 2021-02-10T04:37:46.059 INFO:tasks.rados.rados.0.smithi085.stdout:2477: expect (ObjNum 849 snap 276 seq_num 849) 2021-02-10T04:37:46.059 INFO:tasks.rados.rados.0.smithi085.stdout:2478: write (excl) oid 28 current snap is 277 2021-02-10T04:37:46.059 INFO:tasks.rados.rados.0.smithi085.stdout:2478: seq_num 860 ranges {454020=662029,1691926=588962} 2021-02-10T04:37:46.059 INFO:tasks.rados.rados.0.smithi085.stdout:2478: writing smithi08575397-28 from 454020 to 1116049 tid 1 2021-02-10T04:37:46.059 INFO:tasks.rados.rados.0.smithi085.stdout:2478: writing smithi08575397-28 from 1691926 to 2280888 tid 2 2021-02-10T04:37:46.059 INFO:tasks.rados.rados.0.smithi085.stdout:2479: write oid 38 current snap is 277 2021-02-10T04:37:46.059 INFO:tasks.rados.rados.0.smithi085.stdout:2479: seq_num 861 ranges {470827=737278,1750408=720419} 2021-02-10T04:37:46.059 INFO:tasks.rados.rados.0.smithi085.stdout:2479: writing smithi08575397-38 from 470827 to 1208105 tid 1 2021-02-10T04:37:46.060 INFO:tasks.rados.rados.0.smithi085.stdout:2479: writing smithi08575397-38 from 1750408 to 2470827 tid 2 2021-02-10T04:37:46.060 INFO:tasks.rados.rados.0.smithi085.stdout:2480: write oid 32 current snap is 277 2021-02-10T04:37:46.060 INFO:tasks.rados.rados.0.smithi085.stdout:2477: expect (ObjNum 849 snap 276 seq_num 849) 2021-02-10T04:37:46.060 INFO:tasks.rados.rados.0.smithi085.stdout:2480: seq_num 862 ranges {487634=412527,1408890=592631} 2021-02-10T04:37:46.060 INFO:tasks.rados.rados.0.smithi085.stdout:2480: writing smithi08575397-32 from 487634 to 900161 tid 1 2021-02-10T04:37:46.060 INFO:tasks.rados.rados.0.smithi085.stdout:2480: writing smithi08575397-32 from 1408890 to 2001521 tid 2 2021-02-10T04:37:46.060 INFO:tasks.rados.rados.0.smithi085.stdout:2477: done (7 left) 2021-02-10T04:37:46.060 INFO:tasks.rados.rados.0.smithi085.stdout:2481: rollback oid 37 current snap is 277 2021-02-10T04:37:46.060 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 37 to 265 2021-02-10T04:37:46.060 INFO:tasks.rados.rados.0.smithi085.stdout:2478: finishing write tid 1 to smithi08575397-28 2021-02-10T04:37:46.060 INFO:tasks.rados.rados.0.smithi085.stdout:2478: finishing write tid 2 to smithi08575397-28 2021-02-10T04:37:46.060 INFO:tasks.rados.rados.0.smithi085.stdout:2478: finishing write tid 3 to smithi08575397-28 2021-02-10T04:37:46.061 INFO:tasks.rados.rados.0.smithi085.stdout:2478: finishing write tid 4 to smithi08575397-28 2021-02-10T04:37:46.061 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 1265 (ObjNum 860 snap 277 seq_num 860) dirty exists 2021-02-10T04:37:46.061 INFO:tasks.rados.rados.0.smithi085.stdout:2478: left oid 28 (ObjNum 860 snap 277 seq_num 860) 2021-02-10T04:37:46.061 INFO:tasks.rados.rados.0.smithi085.stdout:2479: finishing write tid 1 to smithi08575397-38 2021-02-10T04:37:46.061 INFO:tasks.rados.rados.0.smithi085.stdout:2479: finishing write tid 2 to smithi08575397-38 2021-02-10T04:37:46.061 INFO:tasks.rados.rados.0.smithi085.stdout:2478: done (7 left) 2021-02-10T04:37:46.061 INFO:tasks.rados.rados.0.smithi085.stdout:2482: snap_create 2021-02-10T04:37:46.061 INFO:tasks.rados.rados.0.smithi085.stdout:2479: finishing write tid 3 to smithi08575397-38 2021-02-10T04:37:46.061 INFO:tasks.rados.rados.0.smithi085.stdout:2479: finishing write tid 4 to smithi08575397-38 2021-02-10T04:37:46.061 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 1098 (ObjNum 861 snap 277 seq_num 861) dirty exists 2021-02-10T04:37:46.061 INFO:tasks.rados.rados.0.smithi085.stdout:2479: left oid 38 (ObjNum 861 snap 277 seq_num 861) 2021-02-10T04:37:46.061 INFO:tasks.rados.rados.0.smithi085.stdout:2480: finishing write tid 1 to smithi08575397-32 2021-02-10T04:37:46.061 INFO:tasks.rados.rados.0.smithi085.stdout:2481: finishing rollback tid 0 to smithi08575397-37 2021-02-10T04:37:46.062 INFO:tasks.rados.rados.0.smithi085.stdout:2481: finishing rollback tid 1 to smithi08575397-37 2021-02-10T04:37:46.062 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 1322 (ObjNum 82245184 snap 22030 seq_num 80674832) dirty dne 2021-02-10T04:37:46.062 INFO:tasks.rados.rados.0.smithi085.stdout:2480: finishing write tid 2 to smithi08575397-32 2021-02-10T04:37:46.062 INFO:tasks.rados.rados.0.smithi085.stdout:2480: finishing write tid 3 to smithi08575397-32 2021-02-10T04:37:46.062 INFO:tasks.rados.rados.0.smithi085.stdout:2480: finishing write tid 4 to smithi08575397-32 2021-02-10T04:37:46.062 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 32 v 630 (ObjNum 862 snap 277 seq_num 862) dirty exists 2021-02-10T04:37:46.062 INFO:tasks.rados.rados.0.smithi085.stdout:2480: left oid 32 (ObjNum 862 snap 277 seq_num 862) 2021-02-10T04:37:46.062 INFO:tasks.rados.rados.0.smithi085.stdout:2479: done (7 left) 2021-02-10T04:37:46.062 INFO:tasks.rados.rados.0.smithi085.stdout:2480: done (6 left) 2021-02-10T04:37:46.062 INFO:tasks.rados.rados.0.smithi085.stdout:2481: done (5 left) 2021-02-10T04:37:46.062 INFO:tasks.rados.rados.0.smithi085.stdout:2482: done (4 left) 2021-02-10T04:37:46.062 INFO:tasks.rados.rados.0.smithi085.stdout:2483: read oid 4 snap -1 2021-02-10T04:37:46.062 INFO:tasks.rados.rados.0.smithi085.stdout:2483: expect (ObjNum 8 snap 0 seq_num 8) 2021-02-10T04:37:46.063 INFO:tasks.rados.rados.0.smithi085.stdout:2484: copy_from oid 44 from oid 18 current snap is 278 2021-02-10T04:37:46.063 INFO:tasks.rados.rados.0.smithi085.stdout:2485: write (excl) oid 5 current snap is 278 2021-02-10T04:37:46.063 INFO:tasks.rados.rados.0.smithi085.stdout:2485: seq_num 863 ranges {504441=487776,1467372=736289} 2021-02-10T04:37:46.063 INFO:tasks.rados.rados.0.smithi085.stdout:2485: writing smithi08575397-5 from 504441 to 992217 tid 1 2021-02-10T04:37:46.063 INFO:tasks.rados.rados.0.smithi085.stdout:2485: writing smithi08575397-5 from 1467372 to 2203661 tid 2 2021-02-10T04:37:46.063 INFO:tasks.rados.rados.0.smithi085.stdout:2486: write (excl) oid 8 current snap is 278 2021-02-10T04:37:46.063 INFO:tasks.rados.rados.0.smithi085.stdout:2485: finishing write tid 1 to smithi08575397-5 2021-02-10T04:37:46.063 INFO:tasks.rados.rados.0.smithi085.stdout:2486: seq_num 864 ranges {521248=563025,1809501=596300} 2021-02-10T04:37:46.063 INFO:tasks.rados.rados.0.smithi085.stdout:2486: writing smithi08575397-8 from 521248 to 1084273 tid 1 2021-02-10T04:37:46.063 INFO:tasks.rados.rados.0.smithi085.stdout:2486: writing smithi08575397-8 from 1809501 to 2405801 tid 2 2021-02-10T04:37:46.063 INFO:tasks.rados.rados.0.smithi085.stdout:2487: write (excl) oid 14 current snap is 278 2021-02-10T04:37:46.063 INFO:tasks.rados.rados.0.smithi085.stdout:2487: seq_num 865 ranges {538055=638274,1867983=670072} 2021-02-10T04:37:46.063 INFO:tasks.rados.rados.0.smithi085.stdout:2487: writing smithi08575397-14 from 538055 to 1176329 tid 1 2021-02-10T04:37:46.064 INFO:tasks.rados.rados.0.smithi085.stdout:2487: writing smithi08575397-14 from 1867983 to 2538055 tid 2 2021-02-10T04:37:46.064 INFO:tasks.rados.rados.0.smithi085.stdout:2488: read oid 46 snap 266 2021-02-10T04:37:46.064 INFO:tasks.rados.rados.0.smithi085.stdout:2488: expect deleted 2021-02-10T04:37:46.064 INFO:tasks.rados.rados.0.smithi085.stdout:2489: write oid 20 current snap is 278 2021-02-10T04:37:46.064 INFO:tasks.rados.rados.0.smithi085.stdout:2489: seq_num 866 ranges {554862=713523,1926465=599969} 2021-02-10T04:37:46.064 INFO:tasks.rados.rados.0.smithi085.stdout:2489: writing smithi08575397-20 from 554862 to 1268385 tid 1 2021-02-10T04:37:46.064 INFO:tasks.rados.rados.0.smithi085.stdout:2489: writing smithi08575397-20 from 1926465 to 2526434 tid 2 2021-02-10T04:37:46.064 INFO:tasks.rados.rados.0.smithi085.stdout:2490: snap_create 2021-02-10T04:37:46.064 INFO:tasks.rados.rados.0.smithi085.stdout:2483: expect (ObjNum 8 snap 0 seq_num 8) 2021-02-10T04:37:46.064 INFO:tasks.rados.rados.0.smithi085.stdout:2485: finishing write tid 2 to smithi08575397-5 2021-02-10T04:37:46.064 INFO:tasks.rados.rados.0.smithi085.stdout:2485: finishing write tid 3 to smithi08575397-5 2021-02-10T04:37:46.064 INFO:tasks.rados.rados.0.smithi085.stdout:2486: finishing write tid 1 to smithi08575397-8 2021-02-10T04:37:46.064 INFO:tasks.rados.rados.0.smithi085.stdout:2485: finishing write tid 4 to smithi08575397-5 2021-02-10T04:37:46.065 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 780 (ObjNum 863 snap 278 seq_num 863) dirty exists 2021-02-10T04:37:46.065 INFO:tasks.rados.rados.0.smithi085.stdout:2485: left oid 5 (ObjNum 863 snap 278 seq_num 863) 2021-02-10T04:37:46.065 INFO:tasks.rados.rados.0.smithi085.stdout:2484: finishing copy_from to smithi08575397-44 2021-02-10T04:37:46.065 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 121 (ObjNum 266 snap 64 seq_num 266) dirty exists 2021-02-10T04:37:46.065 INFO:tasks.rados.rados.0.smithi085.stdout:2486: finishing write tid 2 to smithi08575397-8 2021-02-10T04:37:46.065 INFO:tasks.rados.rados.0.smithi085.stdout:2484: finishing copy_from racing read to smithi08575397-44 2021-02-10T04:37:46.065 INFO:tasks.rados.rados.0.smithi085.stdout:2486: finishing write tid 3 to smithi08575397-8 2021-02-10T04:37:46.065 INFO:tasks.rados.rados.0.smithi085.stdout:2486: finishing write tid 4 to smithi08575397-8 2021-02-10T04:37:46.065 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 1144 (ObjNum 864 snap 278 seq_num 864) dirty exists 2021-02-10T04:37:46.065 INFO:tasks.rados.rados.0.smithi085.stdout:2486: left oid 8 (ObjNum 864 snap 278 seq_num 864) 2021-02-10T04:37:46.065 INFO:tasks.rados.rados.0.smithi085.stdout:2487: finishing write tid 1 to smithi08575397-14 2021-02-10T04:37:46.065 INFO:tasks.rados.rados.0.smithi085.stdout:2487: finishing write tid 2 to smithi08575397-14 2021-02-10T04:37:46.066 INFO:tasks.rados.rados.0.smithi085.stdout:2487: finishing write tid 3 to smithi08575397-14 2021-02-10T04:37:46.066 INFO:tasks.rados.rados.0.smithi085.stdout:2487: finishing write tid 4 to smithi08575397-14 2021-02-10T04:37:46.066 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 891 (ObjNum 865 snap 278 seq_num 865) dirty exists 2021-02-10T04:37:46.066 INFO:tasks.rados.rados.0.smithi085.stdout:2487: left oid 14 (ObjNum 865 snap 278 seq_num 865) 2021-02-10T04:37:46.066 INFO:tasks.rados.rados.0.smithi085.stdout:2489: finishing write tid 1 to smithi08575397-20 2021-02-10T04:37:46.066 INFO:tasks.rados.rados.0.smithi085.stdout:2489: finishing write tid 2 to smithi08575397-20 2021-02-10T04:37:46.066 INFO:tasks.rados.rados.0.smithi085.stdout:2489: finishing write tid 3 to smithi08575397-20 2021-02-10T04:37:46.066 INFO:tasks.rados.rados.0.smithi085.stdout:2489: finishing write tid 4 to smithi08575397-20 2021-02-10T04:37:46.066 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 20 v 1144 (ObjNum 866 snap 278 seq_num 866) dirty exists 2021-02-10T04:37:46.066 INFO:tasks.rados.rados.0.smithi085.stdout:2489: left oid 20 (ObjNum 866 snap 278 seq_num 866) 2021-02-10T04:37:46.066 INFO:tasks.rados.rados.0.smithi085.stdout:2483: done (11 left) 2021-02-10T04:37:46.066 INFO:tasks.rados.rados.0.smithi085.stdout:2484: done (10 left) 2021-02-10T04:37:46.067 INFO:tasks.rados.rados.0.smithi085.stdout:2485: done (9 left) 2021-02-10T04:37:46.067 INFO:tasks.rados.rados.0.smithi085.stdout:2486: done (8 left) 2021-02-10T04:37:46.067 INFO:tasks.rados.rados.0.smithi085.stdout:2487: done (7 left) 2021-02-10T04:37:46.067 INFO:tasks.rados.rados.0.smithi085.stdout:2488: done (6 left) 2021-02-10T04:37:46.067 INFO:tasks.rados.rados.0.smithi085.stdout:2489: done (5 left) 2021-02-10T04:37:46.067 INFO:tasks.rados.rados.0.smithi085.stdout:2490: done (4 left) 2021-02-10T04:37:46.067 INFO:tasks.rados.rados.0.smithi085.stdout:2491: snap_create 2021-02-10T04:37:46.067 INFO:tasks.rados.rados.0.smithi085.stdout:2491: done (4 left) 2021-02-10T04:37:46.067 INFO:tasks.rados.rados.0.smithi085.stdout:2492: read oid 7 snap 276 2021-02-10T04:37:46.067 INFO:tasks.rados.rados.0.smithi085.stdout:2492: expect (ObjNum 759 snap 243 seq_num 759) 2021-02-10T04:37:46.067 INFO:tasks.rados.rados.0.smithi085.stdout:2493: read oid 42 snap -1 2021-02-10T04:37:46.067 INFO:tasks.rados.rados.0.smithi085.stdout:2493: expect (ObjNum 833 snap 270 seq_num 833) 2021-02-10T04:37:46.067 INFO:tasks.rados.rados.0.smithi085.stdout:2494: copy_from oid 3 from oid 34 current snap is 280 2021-02-10T04:37:46.068 INFO:tasks.rados.rados.0.smithi085.stdout:2495: write oid 25 current snap is 280 2021-02-10T04:37:46.068 INFO:tasks.rados.rados.0.smithi085.stdout:2495: seq_num 867 ranges {571669=505125,1701300=743627} 2021-02-10T04:37:46.068 INFO:tasks.rados.rados.0.smithi085.stdout:2495: writing smithi08575397-25 from 571669 to 1076794 tid 1 2021-02-10T04:37:46.068 INFO:tasks.rados.rados.0.smithi085.stdout:2495: writing smithi08575397-25 from 1701300 to 2444927 tid 2 2021-02-10T04:37:46.068 INFO:tasks.rados.rados.0.smithi085.stdout:2492: expect (ObjNum 759 snap 243 seq_num 759) 2021-02-10T04:37:46.068 INFO:tasks.rados.rados.0.smithi085.stdout:2495: finishing write tid 1 to smithi08575397-25 2021-02-10T04:37:46.068 INFO:tasks.rados.rados.0.smithi085.stdout:2493: expect (ObjNum 833 snap 270 seq_num 833) 2021-02-10T04:37:46.068 INFO:tasks.rados.rados.0.smithi085.stdout:2495: finishing write tid 2 to smithi08575397-25 2021-02-10T04:37:46.068 INFO:tasks.rados.rados.0.smithi085.stdout:2495: finishing write tid 3 to smithi08575397-25 2021-02-10T04:37:46.068 INFO:tasks.rados.rados.0.smithi085.stdout:2492: done (7 left) 2021-02-10T04:37:46.068 INFO:tasks.rados.rados.0.smithi085.stdout:2493: done (6 left) 2021-02-10T04:37:46.068 INFO:tasks.rados.rados.0.smithi085.stdout:2496: rollback oid 10 current snap is 280 2021-02-10T04:37:46.069 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 10 to 269 2021-02-10T04:37:46.069 INFO:tasks.rados.rados.0.smithi085.stdout:2495: finishing write tid 4 to smithi08575397-25 2021-02-10T04:37:46.069 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 25 v 125 (ObjNum 867 snap 280 seq_num 867) dirty exists 2021-02-10T04:37:46.069 INFO:tasks.rados.rados.0.smithi085.stdout:2495: left oid 25 (ObjNum 867 snap 280 seq_num 867) 2021-02-10T04:37:46.069 INFO:tasks.rados.rados.0.smithi085.stdout:2494: finishing copy_from to smithi08575397-3 2021-02-10T04:37:46.069 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 777 (ObjNum 859 snap 277 seq_num 859) dirty exists 2021-02-10T04:37:46.069 INFO:tasks.rados.rados.0.smithi085.stdout:2494: finishing copy_from racing read to smithi08575397-3 2021-02-10T04:37:46.069 INFO:tasks.rados.rados.0.smithi085.stdout:2494: done (6 left) 2021-02-10T04:37:46.069 INFO:tasks.rados.rados.0.smithi085.stdout:2495: done (5 left) 2021-02-10T04:37:46.069 INFO:tasks.rados.rados.0.smithi085.stdout:2497: write oid 26 current snap is 280 2021-02-10T04:37:46.069 INFO:tasks.rados.rados.0.smithi085.stdout:2497: seq_num 868 ranges {588476=580374,1643429=603638} 2021-02-10T04:37:46.069 INFO:tasks.rados.rados.0.smithi085.stdout:2497: writing smithi08575397-26 from 588476 to 1168850 tid 1 2021-02-10T04:37:46.070 INFO:tasks.rados.rados.0.smithi085.stdout:2497: writing smithi08575397-26 from 1643429 to 2247067 tid 2 2021-02-10T04:37:46.070 INFO:tasks.rados.rados.0.smithi085.stdout:2498: snap_remove snap 279 2021-02-10T04:37:46.070 INFO:tasks.rados.rados.0.smithi085.stdout:2498: done (6 left) 2021-02-10T04:37:46.070 INFO:tasks.rados.rados.0.smithi085.stdout:2499: copy_from oid 1 from oid 21 current snap is 280 2021-02-10T04:37:46.070 INFO:tasks.rados.rados.0.smithi085.stdout:2500: write oid 3 current snap is 280 2021-02-10T04:37:46.070 INFO:tasks.rados.rados.0.smithi085.stdout:2500: seq_num 869 ranges {605283=655623,1701911=747296} 2021-02-10T04:37:46.070 INFO:tasks.rados.rados.0.smithi085.stdout:2500: writing smithi08575397-3 from 605283 to 1260906 tid 1 2021-02-10T04:37:46.070 INFO:tasks.rados.rados.0.smithi085.stdout:2500: writing smithi08575397-3 from 1701911 to 2449207 tid 2 2021-02-10T04:37:46.070 INFO:tasks.rados.rados.0.smithi085.stdout:2496: finishing rollback tid 0 to smithi08575397-10 2021-02-10T04:37:46.070 INFO:tasks.rados.rados.0.smithi085.stdout:2496: finishing rollback tid 1 to smithi08575397-10 2021-02-10T04:37:46.070 INFO:tasks.rados.rados.0.smithi085.stdout:2496: finishing rollback tid 2 to smithi08575397-10 2021-02-10T04:37:46.070 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 10 v 867 (ObjNum 805 snap 262 seq_num 805) dirty exists 2021-02-10T04:37:46.071 INFO:tasks.rados.rados.0.smithi085.stdout:2497: finishing write tid 1 to smithi08575397-26 2021-02-10T04:37:46.071 INFO:tasks.rados.rados.0.smithi085.stdout:2497: finishing write tid 2 to smithi08575397-26 2021-02-10T04:37:46.071 INFO:tasks.rados.rados.0.smithi085.stdout:2497: finishing write tid 3 to smithi08575397-26 2021-02-10T04:37:46.071 INFO:tasks.rados.rados.0.smithi085.stdout:2497: finishing write tid 4 to smithi08575397-26 2021-02-10T04:37:46.071 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 758 (ObjNum 868 snap 280 seq_num 868) dirty exists 2021-02-10T04:37:46.071 INFO:tasks.rados.rados.0.smithi085.stdout:2497: left oid 26 (ObjNum 868 snap 280 seq_num 868) 2021-02-10T04:37:46.071 INFO:tasks.rados.rados.0.smithi085.stdout:2496: done (7 left) 2021-02-10T04:37:46.071 INFO:tasks.rados.rados.0.smithi085.stdout:2497: done (6 left) 2021-02-10T04:37:46.071 INFO:tasks.rados.rados.0.smithi085.stdout:2501: snap_create 2021-02-10T04:37:46.071 INFO:tasks.rados.rados.0.smithi085.stdout:2500: finishing write tid 1 to smithi08575397-3 2021-02-10T04:37:46.071 INFO:tasks.rados.rados.0.smithi085.stdout:2500: finishing write tid 2 to smithi08575397-3 2021-02-10T04:37:46.071 INFO:tasks.rados.rados.0.smithi085.stdout:2500: finishing write tid 3 to smithi08575397-3 2021-02-10T04:37:46.071 INFO:tasks.rados.rados.0.smithi085.stdout:2500: finishing write tid 4 to smithi08575397-3 2021-02-10T04:37:46.072 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 1155 (ObjNum 869 snap 280 seq_num 869) dirty exists 2021-02-10T04:37:46.072 INFO:tasks.rados.rados.0.smithi085.stdout:2500: left oid 3 (ObjNum 869 snap 280 seq_num 869) 2021-02-10T04:37:46.072 INFO:tasks.rados.rados.0.smithi085.stdout:2499: finishing copy_from to smithi08575397-1 2021-02-10T04:37:46.072 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 1315 (ObjNum 854 snap 277 seq_num 854) dirty exists 2021-02-10T04:37:46.072 INFO:tasks.rados.rados.0.smithi085.stdout:2499: finishing copy_from racing read to smithi08575397-1 2021-02-10T04:37:46.072 INFO:tasks.rados.rados.0.smithi085.stdout:2499: done (6 left) 2021-02-10T04:37:46.072 INFO:tasks.rados.rados.0.smithi085.stdout:2500: done (5 left) 2021-02-10T04:37:46.072 INFO:tasks.rados.rados.0.smithi085.stdout:2501: done (4 left) 2021-02-10T04:37:46.072 INFO:tasks.rados.rados.0.smithi085.stdout:2502: delete oid 35 current snap is 281 2021-02-10T04:37:46.072 INFO:tasks.rados.rados.0.smithi085.stdout:2502: done (4 left) 2021-02-10T04:37:46.072 INFO:tasks.rados.rados.0.smithi085.stdout:2503: read oid 43 snap 271 2021-02-10T04:37:46.072 INFO:tasks.rados.rados.0.smithi085.stdout:2503: expect (ObjNum 748 snap 240 seq_num 748) 2021-02-10T04:37:46.072 INFO:tasks.rados.rados.0.smithi085.stdout:2504: write (excl) oid 4 current snap is 281 2021-02-10T04:37:46.073 INFO:tasks.rados.rados.0.smithi085.stdout:2504: seq_num 871 ranges {638897=406121,1818875=750965} 2021-02-10T04:37:46.073 INFO:tasks.rados.rados.0.smithi085.stdout:2504: writing smithi08575397-4 from 638897 to 1045018 tid 1 2021-02-10T04:37:46.073 INFO:tasks.rados.rados.0.smithi085.stdout:2504: writing smithi08575397-4 from 1818875 to 2569840 tid 2 2021-02-10T04:37:46.073 INFO:tasks.rados.rados.0.smithi085.stdout:2504: finishing write tid 1 to smithi08575397-4 2021-02-10T04:37:46.073 INFO:tasks.rados.rados.0.smithi085.stdout:2505: read oid 49 snap 276 2021-02-10T04:37:46.073 INFO:tasks.rados.rados.0.smithi085.stdout:2505: expect (ObjNum 172 snap 37 seq_num 172) 2021-02-10T04:37:46.073 INFO:tasks.rados.rados.0.smithi085.stdout:2506: snap_remove snap 265 2021-02-10T04:37:46.073 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:40 smithi085 conmon[29319]: cluster 2021-02-10T04:37:39.247998+0000 mon.a (mon. 2021-02-10T04:37:46.073 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:40 smithi085 conmon[29319]: 0) 1509 : cluster [DBG] osdmap e641: 8 total, 8 up, 7 in 2021-02-10T04:37:46.073 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:40 smithi085 conmon[29319]: cluster 2021-02-10T04:37:39.944582+0000 osd.7 (osd.7) 56 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:37:46.073 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:40 smithi085 conmon[29319]: cluster 2021-02-10T04:37:39.944869+0000 2021-02-10T04:37:46.073 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:40 smithi085 conmon[29319]: osd.7 (osd.7) 57 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:37:46.074 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:40 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:46.074 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:40 smithi085 conmon[29319]: 02-10T04:37:40.141794+0000 mgr.y ( 2021-02-10T04:37:46.074 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:40 smithi085 conmon[29319]: mgr.14140) 681 : cluster [DBG] pgmap v1112: 168 pgs: 1 remapped+peering, 12 peering, 5 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 142 active+clean; 422 MiB data, 1002 MiB used, 625 GiB / 626 GiB avail; 3.4 MiB/s wr, 8 op/s; 198 KiB/s, 8 objects/s recovering 2021-02-10T04:37:46.074 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:41 smithi085 conmon[29319]: cluster 2021-02-10T04:37:40.258251+0000 mon.a (mon.0) 1510 : cluster [DBG] osdmap e642: 8 total, 8 up, 7 in 2021-02-10T04:37:46.074 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:42 smithi085 conmon[29319]: cluster 2021-02-10T04:37:46.074 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:42 smithi085 conmon[29319]: 2021-02-10T04:37:46.074 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:42 smithi085 conmon[29319]: 2021-02-10T04:37:41. 2021-02-10T04:37:46.074 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:42 smithi085 conmon[29319]: 259643+0000 mon.a (mon.0) 1511 : cluster [DBG] osdmap e643: 8 total, 8 up, 7 in 2021-02-10T04:37:46.074 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:42 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:37:46.074 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:42 smithi085 conmon[29319]: -02-10T04:37:42.142444+0000 2021-02-10T04:37:46.074 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:42 smithi085 conmon[29319]: mgr.y (mgr.14140) 682 : cluster [DBG] 2021-02-10T04:37:46.074 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:42 smithi085 conmon[29319]: pgmap v1115: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 154 active+clean, 6 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 1 remapped+peering, 2 active+recovering+undersized+remapped; 416 MiB data, 1022 MiB used, 625 GiB / 626 GiB avail; 3.4 MiB/s wr, 8 op/s; 1/1674 objects degraded (0.060%); 51/1674 objects misplaced (3.047%); 475 KiB/s, 40 objects/s recovering 2021-02-10T04:37:46.075 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:43 smithi085 conmon[29319]: cluster 2021-02-10T04:37:42.263957+0000 mon.a (mon.0) 1512 : cluster [WRN] Health check failed: Degraded data redundancy: 1/1674 objects degraded (0.060%), 1 pg degraded (PG_DEGRADED) 2021-02-10T04:37:46.075 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:43 smithi085 conmon[29319]: cluster 2021-02-10T04:37:42.278316+0000 mon.a (mon.0) 1513 : cluster [DBG] osdmap e644: 8 total, 8 up, 7 in 2021-02-10T04:37:46.075 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:44 smithi085 conmon[29319]: cluster 2021-02-10T04:37:43.305886+0000 mon.a (mon. 2021-02-10T04:37:46.075 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:44 smithi085 conmon[29319]: 0) 1514 : cluster [DBG] osdmap e645: 8 total, 8 up, 7 in 2021-02-10T04:37:46.075 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:44 smithi085 conmon[29319]: cluster 2021-02-10T04:37:44.143194+0000 mgr.y (mgr.14140) 683 : cluster [DBG] pgmap v1118: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 154 active+clean, 6 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 2 active+recovering+undersized+remapped; 394 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 14 op/s; 1/1656 objects degraded (0.060%); 51/1656 objects misplaced (3.080%); 3.1 MiB/s, 35 objects/s recovering 2021-02-10T04:37:46.075 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[29319]: cluster 2021-02-10T04:37:46.075 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[29319]: 2021-02-10T04:37:46.075 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[29319]: 2021 2021-02-10T04:37:46.075 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[29319]: - 2021-02-10T04:37:46.075 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[29319]: 02 2021-02-10T04:37:46.075 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[29319]: - 2021-02-10T04:37:46.075 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[29319]: 10 2021-02-10T04:37:46.075 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[29319]: T 2021-02-10T04:37:46.075 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[29319]: 04 2021-02-10T04:37:46.076 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[29319]: :37:44.312687+0000 mon.a (mon 2021-02-10T04:37:46.076 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[29319]: .0) 1515 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2021-02-10T04:37:46.076 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[29319]: cluster 2021-02-10T04:37:44 2021-02-10T04:37:46.076 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[29319]: .328648+0000 mon.a (mon.0) 1516 : 2021-02-10T04:37:46.076 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[29319]: cluster [DBG] osdmap e646: 8 total, 8 up, 7 in 2021-02-10T04:37:46.076 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:37:46.076 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[29319]: 02-10T04:37:44.519178+0000 mon.c 2021-02-10T04:37:46.076 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[29319]: (mon.1) 60 : audit [DBG] 2021-02-10T04:37:46.076 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[29319]: from='client.? 172.21.15.85:0/3738548356' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:37:46.076 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:37:46.076 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[29319]: 10T04:37:44.853319+0000 mon.a 2021-02-10T04:37:46.076 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[29319]: (mon.0) 1517 : audit [INF] from='client.? 172.21.15.85:0/1726137331' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:37:46.076 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:40 smithi085 conmon[35961]: cluster 2021-02-10T04:37:46.077 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:40 smithi085 conmon[35961]: 2021-02-10T04:37 2021-02-10T04:37:46.077 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:40 smithi085 conmon[35961]: :39.247998+0000 mon.a (mon.0) 1509 : cluster [DBG] osdmap e641: 8 total, 8 up, 7 in 2021-02-10T04:37:46.077 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:40 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:46.077 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:40 smithi085 conmon[35961]: 02-10T04:37:39.944582+0000 2021-02-10T04:37:46.077 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:40 smithi085 conmon[35961]: osd.7 (osd.7) 56 : cluster [DBG] 2021-02-10T04:37:46.077 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:40 smithi085 conmon[35961]: purged_snaps scrub starts 2021-02-10T04:37:46.077 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:40 smithi085 conmon[35961]: cluster 2021-02-10T04:37:46.077 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:40 smithi085 conmon[35961]: 2021-02-10T04:37:39.944869+0000 2021-02-10T04:37:46.077 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:40 smithi085 conmon[35961]: osd.7 (osd.7) 57 : cluster [DBG] 2021-02-10T04:37:46.077 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:40 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:37:46.077 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:40 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:37:46.077 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:40 smithi085 conmon[35961]: 37:40.141794+0000 mgr.y (mgr.14140 2021-02-10T04:37:46.078 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:40 smithi085 conmon[35961]: ) 681 : cluster [DBG] pgmap v1112: 168 pgs: 1 remapped+peering, 12 peering, 5 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 142 active+clean; 422 MiB data, 1002 MiB used, 625 GiB / 626 GiB avail; 3.4 MiB/s wr, 8 op/s; 198 KiB/s, 8 objects/s recovering 2021-02-10T04:37:46.078 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:41 smithi085 conmon[35961]: cluster 2021-02-10T04:37:40.258251+0000 mon.a 2021-02-10T04:37:46.078 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:41 smithi085 conmon[35961]: (mon.0) 1510 : cluster [DBG] osdmap e642: 8 total, 8 up, 7 in 2021-02-10T04:37:46.078 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:42 smithi085 conmon[35961]: cluster 2021-02-10T04:37:46.078 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:42 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:37:46.078 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:42 smithi085 conmon[35961]: :37:41.259643+0000 mon.a (mon.0) 1511 : cluster [DBG] osdmap e643: 8 total, 8 up, 7 in 2021-02-10T04:37:46.078 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:42 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:46.078 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:42 smithi085 conmon[35961]: 02-10T04:37:42.142444+0000 mgr.y 2021-02-10T04:37:46.078 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:42 smithi085 conmon[35961]: (mgr.14140) 682 : cluster [DBG] pgmap v1115: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 154 active+clean, 6 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 1 remapped+peering, 2 active+recovering+undersized+remapped; 416 MiB data, 1022 MiB used, 625 GiB / 626 GiB avail; 3.4 MiB/s wr, 8 op/s; 1/1674 objects degraded (0.060%); 51/1674 objects misplaced (3.047%); 475 KiB/s, 40 objects/s recovering 2021-02-10T04:37:46.078 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:42 smithi085 conmon[35961]: 2021-02-10T04:37:46.078 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:43 smithi085 conmon[35961]: cluster 2021-02-10T04:37:42.263957+0000 mon.a 2021-02-10T04:37:46.079 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:43 smithi085 conmon[35961]: (mon.0) 1512 : cluster [WRN] Health check failed: Degraded data redundancy: 1/1674 objects degraded (0.060%), 1 pg degraded (PG_DEGRADED) 2021-02-10T04:37:46.079 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:43 smithi085 conmon[35961]: cluster 2021-02-10T04:37:42.278316+0000 mon.a (mon.0) 1513 : cluster [DBG] osdmap e644: 8 total, 8 up, 7 in 2021-02-10T04:37:46.079 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:44 smithi085 conmon[35961]: cluster 2021-02-10T04:37:46.079 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:44 smithi085 conmon[35961]: 2021-02-10T04:37:43.305886+0000 2021-02-10T04:37:46.079 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:44 smithi085 conmon[35961]: mon.a (mon.0) 1514 : cluster [DBG] osdmap e645: 8 total, 8 up, 7 in 2021-02-10T04:37:46.079 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:44 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:46.079 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:44 smithi085 conmon[35961]: -02-10T04:37:44.143194+0000 mgr.y (mgr.14140) 683 : cluster [DBG] pgmap v1118: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 154 active+clean, 6 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 2 active+recovering+undersized+remapped; 394 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 14 op/s; 1/1656 objects degraded (0.060%); 51/1656 objects misplaced (3.080%); 3.1 MiB/s, 35 objects/s recovering 2021-02-10T04:37:46.079 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[35961]: cluster 2021-02-10T04:37:44. 2021-02-10T04:37:46.079 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[35961]: 312687+0000 mon.a (mon.0) 1515 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2021-02-10T04:37:46.079 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[35961]: cluster 2021-02-10T04:37:46.079 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[35961]: 2021-02-10T04:37:44.328648+0000 mon.a (mon.0) 2021-02-10T04:37:46.079 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[35961]: 1516 : cluster [DBG] osdmap e646: 8 total, 8 up, 7 in 2021-02-10T04:37:46.079 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:37:46.080 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[35961]: 10T04:37:44.519178+0000 mon.c ( 2021-02-10T04:37:46.080 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[35961]: mon.1) 60 : audit [DBG] from='client.? 172.21.15.85:0/3738548356' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:37:46.080 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:37:46.080 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[35961]: 02-10T04:37:44.853319+0000 2021-02-10T04:37:46.080 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[35961]: mon.a (mon.0) 1517 : audit [INF] from='client.? 172.21.15.85:0/1726137331' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:37:46.080 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:45 smithi085 conmon[35961]: 2021-02-10T04:37:46.081 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:37:46.081 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:37:46.081 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:37:46.081 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:37:46.081 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:37:46.081 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:37:46.082 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:37:46.082 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:37:46.082 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:37:46.082 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:37:46.082 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:37:46.082 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:37:46.082 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:37:46.082 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:37:46.082 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:37:46.082 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:37:46.082 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is set 2021-02-10T04:37:46.182 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:37:46.222 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:37:46.340 INFO:tasks.rados.rados.0.smithi085.stdout:2506: done (7 left) 2021-02-10T04:37:46.340 INFO:tasks.rados.rados.0.smithi085.stdout:2507: copy_from oid 18 from oid 36 current snap is 281 2021-02-10T04:37:46.340 INFO:tasks.rados.rados.0.smithi085.stdout:2508: snap_create 2021-02-10T04:37:46.340 INFO:tasks.rados.rados.0.smithi085.stdout:2503: expect (ObjNum 748 snap 240 seq_num 748) 2021-02-10T04:37:46.466 INFO:tasks.rados.rados.0.smithi085.stdout:2505: expect (ObjNum 172 snap 37 seq_num 172) 2021-02-10T04:37:46.569 INFO:tasks.rados.rados.0.smithi085.stdout:2504: finishing write tid 2 to smithi08575397-4 2021-02-10T04:37:46.569 INFO:tasks.rados.rados.0.smithi085.stdout:2504: finishing write tid 3 to smithi08575397-4 2021-02-10T04:37:46.569 INFO:tasks.rados.rados.0.smithi085.stdout:2504: finishing write tid 4 to smithi08575397-4 2021-02-10T04:37:46.569 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 4 v 761 (ObjNum 871 snap 281 seq_num 871) dirty exists 2021-02-10T04:37:46.569 INFO:tasks.rados.rados.0.smithi085.stdout:2504: left oid 4 (ObjNum 871 snap 281 seq_num 871) 2021-02-10T04:37:46.570 INFO:tasks.rados.rados.0.smithi085.stdout:2507: finishing copy_from to smithi08575397-18 2021-02-10T04:37:46.570 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 18 v 1264 (ObjNum 720 snap 226 seq_num 720) dirty exists 2021-02-10T04:37:46.570 INFO:tasks.rados.rados.0.smithi085.stdout:2507: finishing copy_from racing read to smithi08575397-18 2021-02-10T04:37:46.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[33677]: debug 2021-02-10T04:37:46.311+0000 7fbec19a1700 -1 osd.4 647 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: cluster 2021-02-10T04:37:45.291309+0000 2021-02-10T04:37:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: osd.4 (osd.4) 973 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: cluster 2021-02-10T04:37:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: 2021-02-10T04:37:45.291318+0000 osd.4 (osd.4) 2021-02-10T04:37:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: 974 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:37:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: 37:45.291325+0000 osd.4 (osd.4) 975 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: cluster 2021-02-10T04:37:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: 2021-02-10T04:37:45.291331+0000 osd.4 (osd.4) 2021-02-10T04:37:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: 976 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: cluster 2021-02-10T04:37:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: 2021-02-10T04:37:45.318763 2021-02-10T04:37:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1518 : cluster 2021-02-10T04:37:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:37:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: audit 2021-02-10T04:37:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: 2021-02-10T04:37:45.327759 2021-02-10T04:37:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: +0000 mon.a (mon.0 2021-02-10T04:37:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: ) 1519 : audit [INF] from='client.? 172.21.15.85:0/1726137331' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:37:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:37:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: 04:37:45.327826+0000 mon.a (mon.0 2021-02-10T04:37:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: ) 1520 : cluster [DBG] osdmap e647: 8 total, 8 up, 7 in 2021-02-10T04:37:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: cluster 2021-02-10T04:37:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: 2021-02 2021-02-10T04:37:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: -10T04:37:45. 2021-02-10T04:37:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: 482242+0000 osd.4 (osd.4 2021-02-10T04:37:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: ) 977 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:37:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:37:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: -10T04:37:45.482445+0000 osd.4 (osd 2021-02-10T04:37:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: .4) 978 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:37:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: cluster 2021-02-10T04:37:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: 2021-02-10T04:37:46.143875 2021-02-10T04:37:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:46 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 684 : cluster [DBG] pgmap v1121: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 154 active+clean, 6 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 402 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 6.1 MiB/s wr, 31 op/s; 2/1665 objects degraded (0.120%); 14/1665 objects misplaced (0.841%); 18 MiB/s, 49 objects/s recovering 2021-02-10T04:37:46.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[29319]: cluster 2021-02-10T04:37:45.291309+0000 2021-02-10T04:37:46.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[29319]: osd.4 (osd.4) 973 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:46.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[29319]: cluster 2021-02-10T04:37:45.291318+0000 osd.4 (osd.4) 974 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:46.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[29319]: cluster 2021-02-10T04:37:45.291325+0000 osd.4 (osd.4) 975 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:46.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[29319]: cluster 2021-02-10T04:37 2021-02-10T04:37:46.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[29319]: :45.291331+0000 osd.4 (osd.4) 976 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:46.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[29319]: cluster 2021-02-10T04:37: 2021-02-10T04:37:46.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[29319]: 45.318763+0000 mon.a (mon.0) 1518 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:37:46.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:37:46.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[29319]: 02-10T04:37:45.327759+0000 mon.a (mon.0) 1519 2021-02-10T04:37:46.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[29319]: : audit [INF] from='client.? 172.21.15.85:0/1726137331' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:37:46.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[29319]: cluster 2021-02-10T04:37:45. 2021-02-10T04:37:46.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[29319]: 327826+0000 mon.a (mon.0) 1520 : cluster 2021-02-10T04:37:46.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[29319]: [DBG] osdmap e647: 8 total, 8 up, 7 in 2021-02-10T04:37:46.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[29319]: cluster 2021-02-10T04:37:46.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[29319]: 2021-02-10T04:37 2021-02-10T04:37:46.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[29319]: :45.482242+0000 osd.4 (osd 2021-02-10T04:37:46.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[29319]: .4) 977 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:37:46.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[29319]: 2021-02-10T04:37:46.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[29319]: cluster 2021-02-10T04:37 2021-02-10T04:37:46.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[29319]: :45.482445+0000 osd.4 (osd.4) 978 : 2021-02-10T04:37:46.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[29319]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:37:46.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:37:46.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[29319]: :37:46.143875+0000 mgr.y (mgr.14140) 2021-02-10T04:37:46.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[29319]: 684 : cluster [DBG] pgmap v1121: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 154 active+clean, 6 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 402 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 6.1 MiB/s wr, 31 op/s; 2/1665 objects degraded (0.120%); 14/1665 objects misplaced (0.841%); 18 MiB/s, 49 objects/s recovering 2021-02-10T04:37:46.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[35961]: cluster 2021-02-10T04:37:45.291309+0000 2021-02-10T04:37:46.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[35961]: osd.4 (osd.4) 973 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:46.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[35961]: cluster 2021-02-10T04:37: 2021-02-10T04:37:46.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[35961]: 45.291318+0000 osd.4 (osd.4) 974 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:46.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:37:46.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[35961]: 10T04:37:45.291325+0000 osd.4 (osd 2021-02-10T04:37:46.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[35961]: .4) 975 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:46.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:37:46.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[35961]: 04:37:45.291331+0000 osd.4 ( 2021-02-10T04:37:46.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[35961]: osd.4) 976 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:46.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[35961]: cluster 2021-02-10T04:37:46.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[35961]: 2021-02-10T04:37:45. 2021-02-10T04:37:46.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[35961]: 318763+0000 mon.a (mon.0) 1518 : cluster [WRN] 2021-02-10T04:37:46.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[35961]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:37:46.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:37:46.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[35961]: 37:45.327759+0000 mon.a (mon.0) 1519 2021-02-10T04:37:46.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[35961]: : audit [INF] from='client.? 172.21.15.85:0/1726137331' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:37:46.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:37:46.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[35961]: 37:45.327826+0000 mon.a (mon.0) 1520 2021-02-10T04:37:46.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[35961]: : cluster [DBG] osdmap e647: 8 total, 8 up, 7 in 2021-02-10T04:37:46.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[35961]: cluster 2021-02-10T04:37: 2021-02-10T04:37:46.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[35961]: 45.482242+0000 osd.4 (osd 2021-02-10T04:37:46.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[35961]: .4) 977 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:37:46.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:37:46.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[35961]: -10T04:37:45.482445+0000 osd.4 (osd 2021-02-10T04:37:46.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[35961]: .4) 978 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:37:46.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:46.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[35961]: 02-10T04:37:46.143875+0000 mgr.y 2021-02-10T04:37:46.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[35961]: (mgr.14140) 684 : cluster [DBG] pgmap v1121: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 154 active+clean, 6 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 402 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 6.1 MiB/s wr, 31 op/s; 2/1665 objects degraded (0.120%); 14/1665 objects misplaced (0.841%); 18 MiB/s, 49 objects/s recovering 2021-02-10T04:37:47.093 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:37:47.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:37:46 smithi085 conmon[49106]: debug 2021-02-10T04:37:46.908+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:37:47.345 INFO:tasks.rados.rados.0.smithi085.stdout:2503: done (8 left) 2021-02-10T04:37:47.345 INFO:tasks.rados.rados.0.smithi085.stdout:2504: done (7 left) 2021-02-10T04:37:47.345 INFO:tasks.rados.rados.0.smithi085.stdout:2505: done (6 left) 2021-02-10T04:37:47.345 INFO:tasks.rados.rados.0.smithi085.stdout:2507: done (5 left) 2021-02-10T04:37:47.345 INFO:tasks.rados.rados.0.smithi085.stdout:2508: done (4 left) 2021-02-10T04:37:47.345 INFO:tasks.rados.rados.0.smithi085.stdout:2509: delete oid 14 current snap is 282 2021-02-10T04:37:47.350 INFO:tasks.rados.rados.0.smithi085.stdout:2509: done (4 left) 2021-02-10T04:37:47.351 INFO:tasks.rados.rados.0.smithi085.stdout:2510: copy_from oid 44 from oid 13 current snap is 282 2021-02-10T04:37:47.351 INFO:tasks.rados.rados.0.smithi085.stdout:2511: delete oid 3 current snap is 282 2021-02-10T04:37:47.353 INFO:tasks.rados.rados.0.smithi085.stdout:2510: finishing copy_from to smithi08575397-44 2021-02-10T04:37:47.353 INFO:tasks.rados.rados.0.smithi085.stdout:2510: got expected ENOENT (src dne) 2021-02-10T04:37:47.354 INFO:tasks.rados.rados.0.smithi085.stdout:2510: finishing copy_from racing read to smithi08575397-44 2021-02-10T04:37:47.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:47 smithi164 conmon[33677]: debug 2021-02-10T04:37:47.289+0000 7fbec19a1700 -1 osd.4 648 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:47 smithi164 conmon[29288]: cluster 2021-02-10T04:37:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:47 smithi164 conmon[29288]: 2021-02-10T04:37 2021-02-10T04:37:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:47 smithi164 conmon[29288]: :46.313438+0000 osd.4 (osd.4) 979 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:47.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:47 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:37:47.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:47 smithi164 conmon[29288]: -10T04:37:46.313447+0000 osd.4 (osd 2021-02-10T04:37:47.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:47 smithi164 conmon[29288]: .4) 980 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:47.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:47 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:37:47.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:47 smithi164 conmon[29288]: -10T04:37:46.313454+0000 osd.4 2021-02-10T04:37:47.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:47 smithi164 conmon[29288]: (osd.4) 981 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:47.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:47 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:37:47.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:47 smithi164 conmon[29288]: -10T04:37:46.313457+0000 osd.4 ( 2021-02-10T04:37:47.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:47 smithi164 conmon[29288]: osd.4) 982 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:47.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:47 smithi164 conmon[29288]: cluster 2021-02-10T04:37:47.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:47 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:37:47.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:47 smithi164 conmon[29288]: 04:37:46.332635+0000 mon.a (mon. 2021-02-10T04:37:47.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:47 smithi164 conmon[29288]: 0) 1521 : cluster [DBG] osdmap e648: 8 total, 8 up, 7 in 2021-02-10T04:37:47.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[29319]: cluster 2021-02-10T04:37:47.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[29319]: 2021-02 2021-02-10T04:37:47.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[29319]: -10T04:37:46.313438+0000 osd.4 (osd.4) 979 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:47.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[29319]: cluster 2021-02-10T04:37:47.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[29319]: 2021-02-10T04:37:46.313447+0000 osd.4 ( 2021-02-10T04:37:47.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[29319]: osd.4) 980 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:47.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[29319]: cluster 2021-02-10T04:37:46.313454+0000 osd.4 (osd. 2021-02-10T04:37:47.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[29319]: 4) 981 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:47.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[29319]: cluster 2021-02-10T04:37 2021-02-10T04:37:47.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[29319]: :46.313457+0000 osd.4 (osd.4) 982 2021-02-10T04:37:47.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:47.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:37:47.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[29319]: -10T04:37:46.332635+0000 mon.a (mon.0) 1521 : 2021-02-10T04:37:47.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[29319]: cluster [DBG] osdmap e648: 8 total, 8 up, 7 in 2021-02-10T04:37:47.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[35961]: cluster 2021-02-10T04:37:47.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[35961]: 2021-02-10T04:37:47.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:37:47.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[35961]: :37:46.313438+0000 osd.4 (osd.4) 979 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:47.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:47.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[35961]: 02-10T04:37:46.313447+0000 osd.4 (osd 2021-02-10T04:37:47.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[35961]: .4) 980 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:47.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[35961]: cluster 2021-02-10T04:37:47.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[35961]: 2021-02-10T04:37:46.313454+0000 2021-02-10T04:37:47.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[35961]: osd.4 (osd.4) 981 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:47.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:47.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[35961]: -02-10T04:37:46.313457+0000 2021-02-10T04:37:47.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[35961]: osd.4 (osd.4) 982 : cluster [WRN] 2021-02-10T04:37:47.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:47.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[35961]: cluster 2021-02-10T04:37:46 2021-02-10T04:37:47.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[35961]: .332635+0000 mon.a (mon.0) 1521 : 2021-02-10T04:37:47.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:47 smithi085 conmon[35961]: cluster [DBG] osdmap e648: 8 total, 8 up, 7 in 2021-02-10T04:37:48.219 INFO:tasks.rados.rados.0.smithi085.stdout:2510: done (5 left) 2021-02-10T04:37:48.220 INFO:tasks.rados.rados.0.smithi085.stdout:2511: done (4 left) 2021-02-10T04:37:48.220 INFO:tasks.rados.rados.0.smithi085.stdout:2512: delete oid 9 current snap is 282 2021-02-10T04:37:48.222 INFO:tasks.rados.rados.0.smithi085.stdout:2512: done (4 left) 2021-02-10T04:37:48.222 INFO:tasks.rados.rados.0.smithi085.stdout:2513: write (excl) oid 30 current snap is 282 2021-02-10T04:37:48.222 INFO:tasks.rados.rados.0.smithi085.stdout:2513: seq_num 875 ranges {706125=423470,1652803=758303} 2021-02-10T04:37:48.229 INFO:tasks.rados.rados.0.smithi085.stdout:2513: writing smithi08575397-30 from 706125 to 1129595 tid 1 2021-02-10T04:37:48.237 INFO:tasks.rados.rados.0.smithi085.stdout:2513: writing smithi08575397-30 from 1652803 to 2411106 tid 2 2021-02-10T04:37:48.238 INFO:tasks.rados.rados.0.smithi085.stdout:2514: write (excl) oid 9 current snap is 282 2021-02-10T04:37:48.238 INFO:tasks.rados.rados.0.smithi085.stdout:2513: finishing write tid 1 to smithi08575397-30 2021-02-10T04:37:48.238 INFO:tasks.rados.rados.0.smithi085.stdout:2514: seq_num 876 ranges {722932=498719,1994932=501961} 2021-02-10T04:37:48.246 INFO:tasks.rados.rados.0.smithi085.stdout:2514: writing smithi08575397-9 from 722932 to 1221651 tid 1 2021-02-10T04:37:48.254 INFO:tasks.rados.rados.0.smithi085.stdout:2514: writing smithi08575397-9 from 1994932 to 2496893 tid 2 2021-02-10T04:37:48.254 INFO:tasks.rados.rados.0.smithi085.stdout:2515: write (excl) oid 11 current snap is 282 2021-02-10T04:37:48.254 INFO:tasks.rados.rados.0.smithi085.stdout:2515: seq_num 877 ranges {739739=573968,2053414=686325} 2021-02-10T04:37:48.262 INFO:tasks.rados.rados.0.smithi085.stdout:2515: writing smithi08575397-11 from 739739 to 1313707 tid 1 2021-02-10T04:37:48.271 INFO:tasks.rados.rados.0.smithi085.stdout:2515: writing smithi08575397-11 from 2053414 to 2739739 tid 2 2021-02-10T04:37:48.271 INFO:tasks.rados.rados.0.smithi085.stdout:2516: delete oid 45 current snap is 282 2021-02-10T04:37:48.271 INFO:tasks.rados.rados.0.smithi085.stdout:2513: finishing write tid 2 to smithi08575397-30 2021-02-10T04:37:48.271 INFO:tasks.rados.rados.0.smithi085.stdout:2513: finishing write tid 3 to smithi08575397-30 2021-02-10T04:37:48.271 INFO:tasks.rados.rados.0.smithi085.stdout:2513: finishing write tid 4 to smithi08575397-30 2021-02-10T04:37:48.271 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 1101 (ObjNum 875 snap 282 seq_num 875) dirty exists 2021-02-10T04:37:48.271 INFO:tasks.rados.rados.0.smithi085.stdout:2513: left oid 30 (ObjNum 875 snap 282 seq_num 875) 2021-02-10T04:37:48.271 INFO:tasks.rados.rados.0.smithi085.stdout:2514: finishing write tid 1 to smithi08575397-9 2021-02-10T04:37:48.307 INFO:tasks.rados.rados.0.smithi085.stdout:2514: finishing write tid 2 to smithi08575397-9 2021-02-10T04:37:48.307 INFO:tasks.rados.rados.0.smithi085.stdout:2515: finishing write tid 1 to smithi08575397-11 2021-02-10T04:37:48.308 INFO:tasks.rados.rados.0.smithi085.stdout:2514: finishing write tid 3 to smithi08575397-9 2021-02-10T04:37:48.308 INFO:tasks.rados.rados.0.smithi085.stdout:2514: finishing write tid 4 to smithi08575397-9 2021-02-10T04:37:48.308 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 9 v 900 (ObjNum 876 snap 282 seq_num 876) dirty exists 2021-02-10T04:37:48.308 INFO:tasks.rados.rados.0.smithi085.stdout:2514: left oid 9 (ObjNum 876 snap 282 seq_num 876) 2021-02-10T04:37:48.312 INFO:tasks.rados.rados.0.smithi085.stdout:2513: done (7 left) 2021-02-10T04:37:48.312 INFO:tasks.rados.rados.0.smithi085.stdout:2514: done (6 left) 2021-02-10T04:37:48.312 INFO:tasks.rados.rados.0.smithi085.stdout:2516: done (5 left) 2021-02-10T04:37:48.313 INFO:tasks.rados.rados.0.smithi085.stdout:2517: write oid 40 current snap is 282 2021-02-10T04:37:48.313 INFO:tasks.rados.rados.0.smithi085.stdout:2517: seq_num 879 ranges {773353=724466,2170378=602975} 2021-02-10T04:37:48.322 INFO:tasks.rados.rados.0.smithi085.stdout:2517: writing smithi08575397-40 from 773353 to 1497819 tid 1 2021-02-10T04:37:48.330 INFO:tasks.rados.rados.0.smithi085.stdout:2517: writing smithi08575397-40 from 2170378 to 2773353 tid 2 2021-02-10T04:37:48.330 INFO:tasks.rados.rados.0.smithi085.stdout:2518: snap_create 2021-02-10T04:37:48.330 INFO:tasks.rados.rados.0.smithi085.stdout:2515: finishing write tid 2 to smithi08575397-11 2021-02-10T04:37:48.331 INFO:tasks.rados.rados.0.smithi085.stdout:2515: finishing write tid 3 to smithi08575397-11 2021-02-10T04:37:48.331 INFO:tasks.rados.rados.0.smithi085.stdout:2515: finishing write tid 4 to smithi08575397-11 2021-02-10T04:37:48.331 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 1336 (ObjNum 877 snap 282 seq_num 877) dirty exists 2021-02-10T04:37:48.331 INFO:tasks.rados.rados.0.smithi085.stdout:2515: left oid 11 (ObjNum 877 snap 282 seq_num 877) 2021-02-10T04:37:48.331 INFO:tasks.rados.rados.0.smithi085.stdout:2517: finishing write tid 1 to smithi08575397-40 2021-02-10T04:37:48.335 INFO:tasks.rados.rados.0.smithi085.stdout:2517: finishing write tid 2 to smithi08575397-40 2021-02-10T04:37:48.336 INFO:tasks.rados.rados.0.smithi085.stdout:2517: finishing write tid 3 to smithi08575397-40 2021-02-10T04:37:48.336 INFO:tasks.rados.rados.0.smithi085.stdout:2517: finishing write tid 4 to smithi08575397-40 2021-02-10T04:37:48.337 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 1339 (ObjNum 879 snap 282 seq_num 879) dirty exists 2021-02-10T04:37:48.337 INFO:tasks.rados.rados.0.smithi085.stdout:2517: left oid 40 (ObjNum 879 snap 282 seq_num 879) 2021-02-10T04:37:48.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:48 smithi164 conmon[33677]: debug 2021-02-10T04:37:48.301+0000 7fbec19a1700 -1 osd.4 650 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:48.644 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:37:48 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:37:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:48 smithi164 conmon[29288]: cluster 2021-02-10T04:37:47.290864+0000 osd.4 (osd. 2021-02-10T04:37:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:48 smithi164 conmon[29288]: 4) 983 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:48 smithi164 conmon[29288]: cluster 2021-02-10T04:37:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:48 smithi164 conmon[29288]: 2021-02-10T04:37:47.290876+0000 osd.4 (osd.4) 984 : cluster [WRN] 2021-02-10T04:37:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:48 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:48 smithi164 conmon[29288]: cluster 2021-02-10T04:37:47. 2021-02-10T04:37:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:48 smithi164 conmon[29288]: 290884+0000 osd.4 (osd.4) 985 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:48 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:37:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:48 smithi164 conmon[29288]: 04:37:47.290891+0000 osd.4 (osd.4) 986 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:48 smithi164 conmon[29288]: 2021-02-10T04:37:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:48 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:37:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:48 smithi164 conmon[29288]: -10T04:37:47.343821+0000 mon.a (mon.0) 1522 : cluster [DBG] 2021-02-10T04:37:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:48 smithi164 conmon[29288]: osdmap e649: 8 total, 8 up, 7 in 2021-02-10T04:37:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:48 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:37:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:48 smithi164 conmon[29288]: -02-10T04:37:48.144787+0000 mgr.y (mgr.14140) 685 : cluster 2021-02-10T04:37:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:48 smithi164 conmon[29288]: [DBG] pgmap v1124: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 155 active+clean, 6 active+clean+snaptrim, 4 active+clean+snaptrim_wait; 399 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 3.0 MiB/s wr, 19 op/s; 2/1665 objects degraded (0.120%); 14/1665 objects misplaced (0.841%); 16 MiB/s, 45 objects/s recovering 2021-02-10T04:37:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:48 smithi164 conmon[29288]: cluster 2021-02-10T04:37:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:48 smithi164 conmon[29288]: 2021-02-10T04:37:48.200030+0000 mon.a (mon.0) 1523 : cluster [WRN] 2021-02-10T04:37:48.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:48 smithi164 conmon[29288]: Health check update: Degraded data redundancy: 2/1665 objects degraded (0.120%), 1 pg degraded (PG_DEGRADED) 2021-02-10T04:37:48.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:48 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:37:48.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:48 smithi164 conmon[29288]: 10T04:37:48.205171+0000 mon.a (mon.0) 1524 : 2021-02-10T04:37:48.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:48 smithi164 conmon[29288]: cluster [DBG] osdmap e650: 8 total, 8 up, 7 in 2021-02-10T04:37:48.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[29319]: cluster 2021-02-10T04:37:47.290864+0000 osd.4 ( 2021-02-10T04:37:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[29319]: osd.4) 983 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[29319]: cluster 2021-02-10T04:37:47.290876+0000 osd.4 (osd.4) 984 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[29319]: cluster 2021-02-10T04:37:47.290884+0000 osd.4 (osd.4) 2021-02-10T04:37:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[29319]: 985 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[29319]: 02-10T04:37:47.290891+0000 osd.4 ( 2021-02-10T04:37:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[29319]: osd.4) 986 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[29319]: 02-10T04:37:47.343821+0000 mon.a 2021-02-10T04:37:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[29319]: (mon.0) 1522 : cluster [DBG] osdmap e649: 8 total, 8 up, 7 in 2021-02-10T04:37:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[29319]: 2021-02-10T04:37:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[29319]: cluster 2021-02-10T04:37:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[29319]: 2021 2021-02-10T04:37:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[29319]: - 2021-02-10T04:37:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[29319]: 02 2021-02-10T04:37:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[29319]: - 2021-02-10T04:37:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[29319]: 10 2021-02-10T04:37:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[29319]: T04:37:48.144787+0000 mgr.y ( 2021-02-10T04:37:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[29319]: mgr.14140) 685 : cluster [DBG] pgmap v1124: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 155 active+clean, 6 active+clean+snaptrim, 4 active+clean+snaptrim_wait; 399 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 3.0 MiB/s wr, 19 op/s; 2/1665 objects degraded (0.120%); 14/1665 objects misplaced (0.841%); 16 MiB/s, 45 objects/s recovering 2021-02-10T04:37:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[29319]: cluster 2021-02-10T04:37:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[29319]: 2021-02-10T04:37:48.200030+0000 mon.a (mon.0) 1523 : cluster 2021-02-10T04:37:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[29319]: [WRN] Health check update: Degraded data redundancy: 2/1665 objects degraded (0.120%), 1 pg degraded (PG_DEGRADED) 2021-02-10T04:37:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[29319]: cluster 2021-02-10T04:37: 2021-02-10T04:37:48.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[29319]: 48.205171+0000 mon.a (mon.0) 1524 : cluster [DBG] osdmap e650: 8 total, 8 up, 7 in 2021-02-10T04:37:48.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:37:48.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[35961]: T04:37:47.290864+0000 osd.4 (osd.4) 983 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:48.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[35961]: cluster 2021-02-10T04:37:47 2021-02-10T04:37:48.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[35961]: .290876+0000 osd.4 (osd.4) 984 : cluster [WRN] 2021-02-10T04:37:48.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:48.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[35961]: cluster 2021-02-10T04:37:47 2021-02-10T04:37:48.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[35961]: .290884+0000 osd.4 (osd.4) 985 : cluster 2021-02-10T04:37:48.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:48.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[35961]: cluster 2021-02-10T04:37:47 2021-02-10T04:37:48.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[35961]: .290891+0000 osd.4 (osd.4) 986 : cluster 2021-02-10T04:37:48.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:48.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[35961]: cluster 2021-02-10T04:37: 2021-02-10T04:37:48.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[35961]: 47.343821+0000 mon.a (mon.0) 1522 : 2021-02-10T04:37:48.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[35961]: cluster [DBG] osdmap e649: 8 total, 8 up, 7 in 2021-02-10T04:37:48.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:37:48.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[35961]: 37:48.144787+0000 mgr.y (mgr. 2021-02-10T04:37:48.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[35961]: 14140) 685 : cluster [DBG] pgmap v1124: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 155 active+clean, 6 active+clean+snaptrim, 4 active+clean+snaptrim_wait; 399 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 3.0 MiB/s wr, 19 op/s; 2/1665 objects degraded (0.120%); 14/1665 objects misplaced (0.841%); 16 MiB/s, 45 objects/s recovering 2021-02-10T04:37:48.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:37:48.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[35961]: -10T04:37:48.200030+0000 2021-02-10T04:37:48.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[35961]: mon.a (mon.0) 1523 : cluster [WRN] Health check update: Degraded data redundancy: 2/1665 objects degraded (0.120%), 1 pg degraded (PG_DEGRADED) 2021-02-10T04:37:48.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:48.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[35961]: 02-10T04:37:48.205171+0000 2021-02-10T04:37:48.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[35961]: mon.a (mon.0) 1524 : cluster [DBG] osdmap e650: 8 total, 8 up, 7 in 2021-02-10T04:37:48.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:48 smithi085 conmon[35961]: 2021-02-10T04:37:49.218 INFO:tasks.rados.rados.0.smithi085.stdout:2515: done (6 left) 2021-02-10T04:37:49.218 INFO:tasks.rados.rados.0.smithi085.stdout:2517: done (5 left) 2021-02-10T04:37:49.218 INFO:tasks.rados.rados.0.smithi085.stdout:2518: done (4 left) 2021-02-10T04:37:49.218 INFO:tasks.rados.rados.0.smithi085.stdout:2519: copy_from oid 26 from oid 11 current snap is 283 2021-02-10T04:37:49.219 INFO:tasks.rados.rados.0.smithi085.stdout:2520: copy_from oid 21 from oid 15 current snap is 283 2021-02-10T04:37:49.219 INFO:tasks.rados.rados.0.smithi085.stdout:2521: snap_create 2021-02-10T04:37:49.244 INFO:tasks.rados.rados.0.smithi085.stdout:2520: finishing copy_from to smithi08575397-21 2021-02-10T04:37:49.244 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 104 (ObjNum 217 snap 49 seq_num 217) dirty exists 2021-02-10T04:37:49.244 INFO:tasks.rados.rados.0.smithi085.stdout:2520: finishing copy_from racing read to smithi08575397-21 2021-02-10T04:37:49.377 INFO:tasks.rados.rados.0.smithi085.stdout:2519: finishing copy_from to smithi08575397-26 2021-02-10T04:37:49.377 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 1336 (ObjNum 877 snap 282 seq_num 877) dirty exists 2021-02-10T04:37:49.377 INFO:tasks.rados.rados.0.smithi085.stdout:2519: finishing copy_from racing read to smithi08575397-26 2021-02-10T04:37:49.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:49 smithi164 conmon[33677]: debug 2021-02-10T04:37:49.263+0000 7fbec19a1700 -1 osd.4 650 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:49 smithi164 conmon[29288]: audit 2021-02-10T04:37:48.230076+0000 mgr.y (mgr.14140) 686 2021-02-10T04:37:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:49 smithi164 conmon[29288]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:37:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:49 smithi164 conmon[29288]: cluster 2021-02-10T04:37:48.303353+0000 osd.4 (osd.4) 987 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:49 smithi164 conmon[29288]: cluster 2021-02-10T04:37:48.303363 2021-02-10T04:37:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:49 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 988 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:49 smithi164 conmon[29288]: cluster 2021-02-10T04:37:48.303370+0000 2021-02-10T04:37:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:49 smithi164 conmon[29288]: osd.4 (osd.4) 989 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:49 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:37:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:49 smithi164 conmon[29288]: 04:37:48.303381+0000 osd.4 (osd.4) 990 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:49 smithi164 conmon[29288]: cluster 2021-02-10T04:37:48.340964 2021-02-10T04:37:49.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:49 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1525 : cluster [WRN] 2021-02-10T04:37:49.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:49 smithi164 conmon[29288]: Health check failed: 4 slow ops, oldest one blocked for 497 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:37:49.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:49 smithi164 conmon[29288]: cluster 2021-02-10T04:37:49.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:49 smithi164 conmon[29288]: 2021-02-10T04:37:49.217269+0000 mon.a (mon.0) 1526 : cluster [DBG] osdmap e651: 8 total, 8 up, 7 in 2021-02-10T04:37:49.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[29319]: audit 2021-02-10T04:37:48.230076+0000 mgr.y (mgr.14140) 2021-02-10T04:37:49.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[29319]: 686 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:37:49.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[29319]: cluster 2021-02-10T04:37:48.303353+0000 osd.4 (osd.4) 987 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:49.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[29319]: cluster 2021-02-10T04:37:48.303363+0000 2021-02-10T04:37:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[29319]: osd.4 (osd.4) 988 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[29319]: cluster 2021-02-10T04:37:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[29319]: 2021-02-10T04:37:48.303370+0000 osd.4 ( 2021-02-10T04:37:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[29319]: osd.4) 989 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[29319]: cluster 2021-02-10T04:37:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[29319]: 2021-02-10T04:37:48.303381+0000 2021-02-10T04:37:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[29319]: osd.4 (osd.4) 990 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:37:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[29319]: -02-10T04:37:48.340964+0000 mon.a (mon.0) 1525 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 497 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:37:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[29319]: cluster 2021-02-10T04:37:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[29319]: 2021-02-10T04:37:49.217269+0000 mon.a (mon 2021-02-10T04:37:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[29319]: .0) 1526 : cluster [DBG] osdmap e651: 8 total, 8 up, 7 in 2021-02-10T04:37:49.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[35961]: audit 2021-02-10T04:37: 2021-02-10T04:37:49.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[35961]: 48.230076+0000 mgr.y (mgr.14140) 686 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:37:49.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[35961]: cluster 2021-02-10T04:37:48.303353 2021-02-10T04:37:49.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 987 : 2021-02-10T04:37:49.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:49.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:49.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[35961]: -02-10T04:37:48.303363 2021-02-10T04:37:49.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 988 2021-02-10T04:37:49.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:49.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:37:49.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[35961]: 10T04:37:48.303370+0000 osd.4 (osd.4) 989 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:49.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[35961]: 2021-02-10T04:37:49.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[35961]: cluster 2021-02-10T04:37:49.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[35961]: 2021-02-10T04:37:48.303381+0000 osd.4 2021-02-10T04:37:49.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[35961]: (osd.4) 990 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:49.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[35961]: cluster 2021-02-10T04:37:49.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[35961]: 2021-02-10T04:37:49.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[35961]: 2021-02-10T04:37 2021-02-10T04:37:49.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[35961]: :48.340964+0000 mon.a (mon.0) 1525 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 497 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:37:49.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[35961]: cluster 2021-02-10T04:37:49.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:37:49.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[35961]: T04:37:49.217269+0000 mon.a (mon.0) 1526 : cluster 2021-02-10T04:37:49.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:49 smithi085 conmon[35961]: [DBG] osdmap e651: 8 total, 8 up, 7 in 2021-02-10T04:37:50.218 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:37:50.219 INFO:tasks.rados.rados.0.smithi085.stdout:2519: done (6 left) 2021-02-10T04:37:50.219 INFO:tasks.rados.rados.0.smithi085.stdout:2520: done (5 left) 2021-02-10T04:37:50.219 INFO:tasks.rados.rados.0.smithi085.stdout:2521: done (4 left) 2021-02-10T04:37:50.219 INFO:tasks.rados.rados.0.smithi085.stdout:2522: snap_create 2021-02-10T04:37:50.253 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:37:50.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[35961]: cluster 2021-02-10T04:37:49.264946 2021-02-10T04:37:50.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 991 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:50.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[35961]: cluster 2021-02-10T04:37:49.264956+0000 osd.4 ( 2021-02-10T04:37:50.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[35961]: osd.4) 992 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:50.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[35961]: cluster 2021-02-10T04:37:50.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[35961]: 2021-02-10T04:37:49. 2021-02-10T04:37:50.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[35961]: 264963+0000 osd.4 (osd.4) 993 : cluster [WRN] 2021-02-10T04:37:50.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:50.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[35961]: cluster 2021-02-10T04:37:50.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:37:50.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[35961]: 37:49.264969+0000 osd.4 (osd.4) 994 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:50.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:50.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[35961]: 02-10T04:37:50.145612+0000 mgr.y (mgr. 2021-02-10T04:37:50.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[35961]: 14140) 687 : cluster [DBG] pgmap v1127: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 156 active+clean, 6 active+clean+snaptrim, 3 active+clean+snaptrim_wait; 406 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 4.2 MiB/s rd, 2.6 MiB/s wr, 12 op/s; 2/1671 objects degraded (0.120%); 14/1671 objects misplaced (0.838%) 2021-02-10T04:37:50.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[35961]: cluster 2021-02-10T04:37:50.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[35961]: 2021-02-10T04:37:50.217927 2021-02-10T04:37:50.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1527 : cluster [DBG] osdmap e652: 8 total, 8 up, 7 in 2021-02-10T04:37:50.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[35961]: 2021-02-10T04:37:50.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: cluster 2021-02-10T04:37:49.264946+0000 osd.4 ( 2021-02-10T04:37:50.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: osd.4) 991 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:50.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: cluster 2021-02-10T04:37:49.264956+0000 osd.4 (osd.4) 992 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:50.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:37:50.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: 37:49.264963+0000 osd.4 (osd 2021-02-10T04:37:50.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: .4) 993 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:50.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: cluster 2021-02-10T04:37:50.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: 2021 2021-02-10T04:37:50.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: - 2021-02-10T04:37:50.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: 02 2021-02-10T04:37:50.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: -10T04:37:49 2021-02-10T04:37:50.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: .264969+0000 osd.4 (osd.4) 994 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:50.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: cluster 2021-02-10T04:37:50.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: 2021-02-10T04:37:50.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: 2021 2021-02-10T04:37:50.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: - 2021-02-10T04:37:50.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: 02 2021-02-10T04:37:50.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: - 2021-02-10T04:37:50.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: 10 2021-02-10T04:37:50.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: T 2021-02-10T04:37:50.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: 04 2021-02-10T04:37:50.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: : 2021-02-10T04:37:50.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: 37 2021-02-10T04:37:50.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: : 2021-02-10T04:37:50.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: 50 2021-02-10T04:37:50.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: . 2021-02-10T04:37:50.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: 145612 2021-02-10T04:37:50.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: +0000 mgr.y (mgr. 2021-02-10T04:37:50.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: 14140) 687 : cluster [DBG] pgmap v1127: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 156 active+clean, 6 active+clean+snaptrim, 3 active+clean+snaptrim_wait; 406 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 4.2 MiB/s rd, 2.6 MiB/s wr, 12 op/s; 2/1671 objects degraded (0.120%); 14/1671 objects misplaced (0.838%) 2021-02-10T04:37:50.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:37:50.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: -02-10T04:37:50.217927+0000 mon.a ( 2021-02-10T04:37:50.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[29319]: mon.0) 1527 : cluster [DBG] osdmap e652: 8 total, 8 up, 7 in 2021-02-10T04:37:50.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:50 smithi164 conmon[33677]: debug 2021-02-10T04:37:50.255+0000 7fbec19a1700 -1 osd.4 652 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:50.928 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:50 smithi164 conmon[29288]: cluster 2021-02-10T04:37:49.264946+0000 osd.4 (osd.4 2021-02-10T04:37:50.929 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:50 smithi164 conmon[29288]: ) 991 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:50.930 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:50 smithi164 conmon[29288]: cluster 2021-02-10T04:37:49.264956+0000 osd.4 (osd.4) 992 2021-02-10T04:37:50.930 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:50 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:50.930 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:50 smithi164 conmon[29288]: cluster 2021-02-10T04:37:50.930 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:50 smithi164 conmon[29288]: 2021-02-10T04:37:49.264963+0000 2021-02-10T04:37:50.931 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:50 smithi164 conmon[29288]: osd.4 (osd.4) 993 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:50.931 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:50 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:37:50.931 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:50 smithi164 conmon[29288]: T04:37:49.264969+0000 osd.4 (osd. 2021-02-10T04:37:50.931 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:50 smithi164 conmon[29288]: 4) 994 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:50.931 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:50 smithi164 conmon[29288]: cluster 2021-02-10T04:37:50. 2021-02-10T04:37:50.931 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:50 smithi164 conmon[29288]: 145612+0000 mgr.y (mgr.14140) 687 : cluster 2021-02-10T04:37:50.932 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:50 smithi164 conmon[29288]: [DBG] pgmap v1127: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 156 active+clean, 6 active+clean+snaptrim, 3 active+clean+snaptrim_wait; 406 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 4.2 MiB/s rd, 2.6 MiB/s wr, 12 op/s; 2/1671 objects degraded (0.120%); 14/1671 objects misplaced (0.838%) 2021-02-10T04:37:50.932 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:50 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:37:50.933 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:50 smithi164 conmon[29288]: 10T04:37:50.217927+0000 mon.a (mon 2021-02-10T04:37:50.933 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:50 smithi164 conmon[29288]: .0) 1527 : cluster [DBG] osdmap e652: 8 total, 8 up, 7 in 2021-02-10T04:37:50.934 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:37:50.941 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:37:50 smithi085 conmon[45443]: debug 2021-02-10T04:37:50.597+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:37:51.081 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-02-10T04:37:51.082 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:37:51.082 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd dump -f json-pretty 2021-02-10T04:37:51.219 INFO:tasks.rados.rados.0.smithi085.stdout:2522: done (4 left) 2021-02-10T04:37:51.219 INFO:tasks.rados.rados.0.smithi085.stdout:2523: snap_create 2021-02-10T04:37:51.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:51 smithi085 conmon[29319]: cluster 2021-02-10T04:37:50. 2021-02-10T04:37:51.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:51 smithi085 conmon[29319]: 257024+0000 osd.4 (osd.4) 995 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:51.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:51 smithi085 conmon[29319]: cluster 2021-02-10T04:37:50.257031+0000 osd.4 (osd.4) 996 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:51.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:51 smithi085 conmon[29319]: cluster 2021-02-10T04:37:50.257036+0000 osd.4 (osd.4) 997 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:51.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:51 smithi085 conmon[29319]: cluster 2021-02-10T04:37:50.257040+0000 osd.4 (osd.4) 998 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:51.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:51 smithi085 conmon[29319]: cluster 2021-02-10T04:37:51.218788+0000 mon.a (mon.0) 1528 : cluster [DBG] osdmap e653: 8 total, 8 up, 7 in 2021-02-10T04:37:51.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:51 smithi085 conmon[35961]: cluster 2021-02-10T04:37:50.257024+0000 2021-02-10T04:37:51.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:51 smithi085 conmon[35961]: osd.4 (osd.4) 995 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:51.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:51 smithi085 conmon[35961]: cluster 2021-02-10T04:37:50 2021-02-10T04:37:51.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:51 smithi085 conmon[35961]: .257031+0000 osd.4 (osd.4) 996 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:51.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:51 smithi085 conmon[35961]: cluster 2021-02-10T04:37 2021-02-10T04:37:51.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:51 smithi085 conmon[35961]: :50.257036+0000 osd.4 (osd.4) 997 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:51.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:51 smithi085 conmon[35961]: cluster 2021-02-10T04:37: 2021-02-10T04:37:51.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:51 smithi085 conmon[35961]: 50.257040+0000 osd.4 (osd.4) 998 : cluster 2021-02-10T04:37:51.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:51 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:51.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:51 smithi085 conmon[35961]: cluster 2021-02-10T04:37:51.218788+0000 2021-02-10T04:37:51.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:51 smithi085 conmon[35961]: mon.a (mon.0) 1528 : cluster [DBG] osdmap e653: 8 total, 8 up, 7 in 2021-02-10T04:37:51.576 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:37:51.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:51 smithi164 conmon[33677]: debug 2021-02-10T04:37:51.281+0000 7fbec19a1700 -1 osd.4 653 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:51 smithi164 conmon[29288]: cluster 2021-02-10T04:37:50.257024+0000 osd.4 (osd.4) 2021-02-10T04:37:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:51 smithi164 conmon[29288]: 995 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:51 smithi164 conmon[29288]: cluster 2021-02-10T04:37:50.257031+0000 osd.4 (osd.4) 996 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:51 smithi164 conmon[29288]: cluster 2021-02-10T04:37:50 2021-02-10T04:37:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:51 smithi164 conmon[29288]: .257036+0000 osd.4 (osd.4) 997 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:51 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:37:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:51 smithi164 conmon[29288]: 02-10T04:37:50.257040+0000 osd.4 (osd.4) 998 : cluster 2021-02-10T04:37:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:51 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:51 smithi164 conmon[29288]: cluster 2021-02-10T04:37:51.218788 2021-02-10T04:37:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:51 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1528 : cluster 2021-02-10T04:37:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:51 smithi164 conmon[29288]: [DBG] osdmap e653: 8 total, 8 up, 7 in 2021-02-10T04:37:52.221 INFO:tasks.rados.rados.0.smithi085.stdout:2523: done (4 left) 2021-02-10T04:37:52.221 INFO:tasks.rados.rados.0.smithi085.stdout:2524: write oid 25 current snap is 286 2021-02-10T04:37:52.222 INFO:tasks.rados.rados.0.smithi085.stdout:2524: seq_num 880 ranges {790160=799715,2112507=509299} 2021-02-10T04:37:52.231 INFO:tasks.rados.rados.0.smithi085.stdout:2524: writing smithi08575397-25 from 790160 to 1589875 tid 1 2021-02-10T04:37:52.238 INFO:tasks.rados.rados.0.smithi085.stdout:2524: writing smithi08575397-25 from 2112507 to 2621806 tid 2 2021-02-10T04:37:52.238 INFO:tasks.rados.rados.0.smithi085.stdout:2524: finishing write tid 1 to smithi08575397-25 2021-02-10T04:37:52.238 INFO:tasks.rados.rados.0.smithi085.stdout:2525: snap_remove snap 263 2021-02-10T04:37:52.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:52 smithi164 conmon[33677]: debug 2021-02-10T04:37:52.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:52 smithi164 conmon[33677]: 2021-02-10T04:37:52.296+0000 7fbec19a1700 -1 osd.4 654 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:52.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:52 smithi164 conmon[33677]: 2021-02-10T04:37:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:52 smithi164 conmon[29288]: cluster 2021-02-10T04:37:50.553120+0000 osd.1 (osd.1) 453 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:37:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:52 smithi164 conmon[29288]: 2021-02-10T04:37:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:52 smithi164 conmon[29288]: cluster 2021-02-10T04:37:50.553660+0000 osd.1 (osd.1) 454 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:37:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:52 smithi164 conmon[29288]: cluster 2021-02-10T04:37:51.283198+0000 osd.4 (osd.4) 999 : cluster 2021-02-10T04:37:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:52 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:52.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:52 smithi164 conmon[29288]: cluster 2021-02-10T04:37: 2021-02-10T04:37:52.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:52 smithi164 conmon[29288]: 51.283217+0000 osd.4 (osd.4) 1000 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:52.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:52 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:37:52.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:52 smithi164 conmon[29288]: 02-10T04:37:51.283224+0000 osd.4 (osd 2021-02-10T04:37:52.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:52 smithi164 conmon[29288]: .4) 1001 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:52.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:52 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:37:52.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:52 smithi164 conmon[29288]: 04:37:51.283230+0000 osd.4 (osd.4) 1002 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:52.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:52 smithi164 conmon[29288]: 2021-02-10T04:37:52.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:52 smithi164 conmon[29288]: cluster 2021-02-10T04:37:52. 2021-02-10T04:37:52.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:52 smithi164 conmon[29288]: 146326+0000 mgr.y (mgr.14140) 688 : cluster [DBG] 2021-02-10T04:37:52.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:52 smithi164 conmon[29288]: pgmap v1130: 168 pgs: 156 active+clean, 8 active+clean+snaptrim, 4 active+clean+snaptrim_wait; 391 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 4.0 MiB/s rd, 2.9 MiB/s wr, 13 op/s; 7.0 MiB/s, 5 objects/s recovering 2021-02-10T04:37:52.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:52 smithi164 conmon[29288]: cluster 2021-02-10T04:37:52.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:52 smithi164 conmon[29288]: 2021-02-10T04:37:52.221268+0000 mon.a (mon. 2021-02-10T04:37:52.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:52 smithi164 conmon[29288]: 0) 1529 : cluster [DBG] osdmap e654: 8 total, 8 up, 7 in 2021-02-10T04:37:52.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[29319]: cluster 2021-02-10T04:37:50.553120+0000 osd.1 (osd 2021-02-10T04:37:52.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[29319]: .1) 453 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:37:52.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[29319]: cluster 2021-02-10T04:37:50.553660+0000 osd.1 (osd.1) 454 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:37:52.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[29319]: cluster 2021-02-10T04:37:51.283198 2021-02-10T04:37:52.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 999 : cluster [WRN] 2021-02-10T04:37:52.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:52.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:37:52.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[29319]: -02-10T04:37:51.283217+0000 2021-02-10T04:37:52.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[29319]: osd.4 (osd.4) 1000 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:52.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[29319]: 2021-02-10T04:37:52.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[29319]: cluster 2021-02-10T04:37:51. 2021-02-10T04:37:52.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[29319]: 283224+0000 osd.4 (osd.4) 1001 : cluster 2021-02-10T04:37:52.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:52.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:37:52.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[29319]: :37:51.283230+0000 osd.4 (osd.4 2021-02-10T04:37:52.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[29319]: ) 1002 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:52.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:52.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[29319]: 02-10T04:37:52.146326+0000 2021-02-10T04:37:52.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[29319]: mgr.y (mgr.14140) 688 : cluster [DBG] 2021-02-10T04:37:52.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[29319]: pgmap v1130: 168 pgs: 156 active+clean, 8 active+clean+snaptrim, 4 active+clean+snaptrim_wait; 391 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 4.0 MiB/s rd, 2.9 MiB/s wr, 13 op/s; 7.0 MiB/s, 5 objects/s recovering 2021-02-10T04:37:52.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[29319]: cluster 2021-02-10T04:37: 2021-02-10T04:37:52.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[29319]: 52.221268+0000 mon.a (mon.0) 1529 : 2021-02-10T04:37:52.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[29319]: cluster [DBG] osdmap e654: 8 total, 8 up, 7 in 2021-02-10T04:37:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[35961]: cluster 2021-02-10T04:37:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[35961]: 2021-02-10T04:37:50.553120+0000 osd.1 (osd.1) 453 : cluster [DBG] 2021-02-10T04:37:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[35961]: purged_snaps scrub starts 2021-02-10T04:37:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[35961]: cluster 2021-02-10T04:37:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[35961]: 2021-02-10T04:37:50.553660 2021-02-10T04:37:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 454 : cluster [DBG] 2021-02-10T04:37:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:37:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[35961]: cluster 2021-02-10T04:37:51.283198+0000 2021-02-10T04:37:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[35961]: osd.4 (osd.4) 999 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:37:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[35961]: :37:51.283217+0000 2021-02-10T04:37:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[35961]: osd.4 (osd.4) 1000 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:37:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[35961]: :37:51.283224+0000 osd.4 (osd.4) 1001 : cluster 2021-02-10T04:37:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[35961]: cluster 2021-02-10T04:37:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[35961]: 2021-02-10T04:37: 2021-02-10T04:37:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[35961]: 51.283230+0000 osd.4 (osd.4) 1002 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[35961]: 2021-02-10T04:37:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[35961]: cluster 2021-02-10T04:37:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:37:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[35961]: 10T04:37:52.146326+0000 mgr.y ( 2021-02-10T04:37:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[35961]: mgr.14140) 688 : cluster [DBG] pgmap v1130: 168 pgs: 156 active+clean, 8 active+clean+snaptrim, 4 active+clean+snaptrim_wait; 391 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 4.0 MiB/s rd, 2.9 MiB/s wr, 13 op/s; 7.0 MiB/s, 5 objects/s recovering 2021-02-10T04:37:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[35961]: -02-10T04:37:52.221268+0000 2021-02-10T04:37:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[35961]: mon.a (mon.0) 1529 : 2021-02-10T04:37:52.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:52 smithi085 conmon[35961]: cluster [DBG] osdmap e654: 8 total, 8 up, 7 in 2021-02-10T04:37:53.210 INFO:tasks.rados.rados.0.smithi085.stdout:2524: finishing write tid 2 to smithi08575397-25 2021-02-10T04:37:53.210 INFO:tasks.rados.rados.0.smithi085.stdout:2524: finishing write tid 3 to smithi08575397-25 2021-02-10T04:37:53.211 INFO:tasks.rados.rados.0.smithi085.stdout:2524: finishing write tid 4 to smithi08575397-25 2021-02-10T04:37:53.211 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 25 v 1267 (ObjNum 880 snap 286 seq_num 880) dirty exists 2021-02-10T04:37:53.211 INFO:tasks.rados.rados.0.smithi085.stdout:2524: left oid 25 (ObjNum 880 snap 286 seq_num 880) 2021-02-10T04:37:53.211 INFO:tasks.rados.rados.0.smithi085.stdout:2524: done (5 left) 2021-02-10T04:37:53.211 INFO:tasks.rados.rados.0.smithi085.stdout:2525: done (4 left) 2021-02-10T04:37:53.211 INFO:tasks.rados.rados.0.smithi085.stdout:2526: read oid 44 snap -1 2021-02-10T04:37:53.211 INFO:tasks.rados.rados.0.smithi085.stdout:2526: expect (ObjNum 266 snap 64 seq_num 266) 2021-02-10T04:37:53.211 INFO:tasks.rados.rados.0.smithi085.stdout:2527: read oid 10 snap 282 2021-02-10T04:37:53.211 INFO:tasks.rados.rados.0.smithi085.stdout:2527: expect (ObjNum 805 snap 262 seq_num 805) 2021-02-10T04:37:53.211 INFO:tasks.rados.rados.0.smithi085.stdout:2528: delete oid 48 current snap is 286 2021-02-10T04:37:53.218 INFO:tasks.rados.rados.0.smithi085.stdout:2527: expect (ObjNum 805 snap 262 seq_num 805) 2021-02-10T04:37:53.474 INFO:tasks.rados.rados.0.smithi085.stdout:2527: done (6 left) 2021-02-10T04:37:53.474 INFO:tasks.rados.rados.0.smithi085.stdout:2528: done (5 left) 2021-02-10T04:37:53.475 INFO:tasks.rados.rados.0.smithi085.stdout:2529: read oid 43 snap 284 2021-02-10T04:37:53.475 INFO:tasks.rados.rados.0.smithi085.stdout:2529: expect (ObjNum 748 snap 240 seq_num 748) 2021-02-10T04:37:53.475 INFO:tasks.rados.rados.0.smithi085.stdout:2530: snap_create 2021-02-10T04:37:53.475 INFO:tasks.rados.rados.0.smithi085.stdout:2526: expect (ObjNum 266 snap 64 seq_num 266) 2021-02-10T04:37:53.627 INFO:tasks.rados.rados.0.smithi085.stdout:2529: expect (ObjNum 748 snap 240 seq_num 748) 2021-02-10T04:37:53.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:53 smithi164 conmon[33677]: debug 2021-02-10T04:37:53.267+0000 7fbec19a1700 -1 osd.4 654 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:53 smithi164 conmon[29288]: cluster 2021-02-10T04:37:52.298300+0000 osd.4 2021-02-10T04:37:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:53 smithi164 conmon[29288]: (osd.4) 1003 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:53 smithi164 conmon[29288]: cluster 2021-02-10T04:37:52.298311 2021-02-10T04:37:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:53 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1004 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:53 smithi164 conmon[29288]: cluster 2021-02-10T04:37 2021-02-10T04:37:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:53 smithi164 conmon[29288]: :52.298318+0000 osd.4 (osd.4) 1005 : 2021-02-10T04:37:53.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:53 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:53.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:53 smithi164 conmon[29288]: cluster 2021-02-10T04:37 2021-02-10T04:37:53.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:53 smithi164 conmon[29288]: :52.298324+0000 osd.4 (osd.4) 2021-02-10T04:37:53.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:53 smithi164 conmon[29288]: 1006 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:53.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:53 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:37:53.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:53 smithi164 conmon[29288]: T04:37:52.357145+0000 mon.a (mon 2021-02-10T04:37:53.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:53 smithi164 conmon[29288]: .0) 1530 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/1671 objects degraded (0.120%), 1 pg degraded) 2021-02-10T04:37:53.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:53 smithi164 conmon[29288]: audit 2021 2021-02-10T04:37:53.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:53 smithi164 conmon[29288]: -02-10T04:37:53.076684+0000 2021-02-10T04:37:53.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:53 smithi164 conmon[29288]: mon.a (mon.0) 1531 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:37:53.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:53 smithi164 conmon[29288]: 2021-02-10T04:37:53.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:53 smithi164 conmon[29288]: cluster 2021-02-10T04:37:53.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:53 smithi164 conmon[29288]: 2021-02-10T04:37:53.208042+0000 mon.a 2021-02-10T04:37:53.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:53 smithi164 conmon[29288]: (mon.0) 1532 : cluster [DBG] osdmap e655: 8 total, 8 up, 7 in 2021-02-10T04:37:53.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[29319]: cluster 2021-02-10T04:37:52.298300+0000 2021-02-10T04:37:53.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[29319]: osd.4 (osd.4) 1003 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:53.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[29319]: cluster 2021-02-10T04:37:52.298311+0000 osd.4 (osd.4) 1004 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:53.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:37:53.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[29319]: 04:37:52.298318+0000 osd.4 (osd.4) 1005 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:53.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[29319]: cluster 2021-02-10T04:37:52.298324+0000 osd.4 (osd.4) 1006 : cluster 2021-02-10T04:37:53.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:53.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[29319]: cluster 2021-02-10T04:37:52 2021-02-10T04:37:53.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[29319]: .357145+0000 mon.a (mon.0) 1530 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/1671 objects degraded (0.120%), 1 pg degraded) 2021-02-10T04:37:53.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[29319]: audit 2021-02-10T04: 2021-02-10T04:37:53.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[29319]: 37:53.076684+0000 mon.a (mon.0) 1531 : 2021-02-10T04:37:53.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[29319]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:37:53.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:37:53.686 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[29319]: 37:53.208042+0000 mon.a (mon.0) 1532 : cluster [DBG] osdmap e655: 8 total, 8 up, 7 in 2021-02-10T04:37:53.686 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[35961]: cluster 2021-02-10T04:37:53.687 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:37:53.687 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[35961]: T04:37:52.298300+0000 osd.4 (osd.4) 1003 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:53.687 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:53.687 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[35961]: -02-10T04:37:52.298311+0000 osd.4 (osd.4 2021-02-10T04:37:53.687 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[35961]: ) 1004 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:53.687 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:53.687 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[35961]: -02-10T04:37:52.298318+0000 2021-02-10T04:37:53.687 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[35961]: osd.4 (osd.4) 1005 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:53.687 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[35961]: 2021-02-10T04:37:53.687 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:37:53.687 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[35961]: 37:52.298324+0000 osd.4 ( 2021-02-10T04:37:53.687 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[35961]: osd.4) 1006 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:53.688 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[35961]: 2021-02-10T04:37:53.688 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[35961]: cluster 2021-02-10T04:37:52.357145+0000 mon.a (mon.0) 1530 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/1671 objects degraded (0.120%), 1 pg degraded) 2021-02-10T04:37:53.688 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[35961]: 2021-02-10T04:37:53.688 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[35961]: audit 2021-02-10T04:37: 2021-02-10T04:37:53.688 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[35961]: 53.076684+0000 mon.a (mon.0) 1531 : 2021-02-10T04:37:53.688 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[35961]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:37:53.688 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:37:53.688 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[35961]: 37:53.208042+0000 mon.a (mon.0) 2021-02-10T04:37:53.688 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:53 smithi085 conmon[35961]: 1532 : cluster [DBG] osdmap e655: 8 total, 8 up, 7 in 2021-02-10T04:37:53.787 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:37:53.822 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:37:54.141 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:37:54.258 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:37:54.290 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:37:54 smithi085 conmon[45443]: debug 2021-02-10T04:37:53.999+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:37:54.306 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:37:54.384 INFO:tasks.rados.rados.0.smithi085.stdout:2526: done (6 left) 2021-02-10T04:37:54.384 INFO:tasks.rados.rados.0.smithi085.stdout:2529: done (5 left) 2021-02-10T04:37:54.384 INFO:tasks.rados.rados.0.smithi085.stdout:2530: done (4 left) 2021-02-10T04:37:54.384 INFO:tasks.rados.rados.0.smithi085.stdout:2531: read oid 46 snap 237 2021-02-10T04:37:54.384 INFO:tasks.rados.rados.0.smithi085.stdout:2531: expect (ObjNum 733 snap 234 seq_num 733) 2021-02-10T04:37:54.384 INFO:tasks.rados.rados.0.smithi085.stdout:2532: read oid 21 snap 285 2021-02-10T04:37:54.385 INFO:tasks.rados.rados.0.smithi085.stdout:2532: expect (ObjNum 217 snap 49 seq_num 217) 2021-02-10T04:37:54.385 INFO:tasks.rados.rados.0.smithi085.stdout:2533: write (excl) oid 13 current snap is 287 2021-02-10T04:37:54.385 INFO:tasks.rados.rados.0.smithi085.stdout:2533: seq_num 882 ranges {423774=666566,1545824=512968,2704609=119165} 2021-02-10T04:37:54.385 INFO:tasks.rados.rados.0.smithi085.stdout:2533: writing smithi08575397-13 from 423774 to 1090340 tid 1 2021-02-10T04:37:54.392 INFO:tasks.rados.rados.0.smithi085.stdout:2533: writing smithi08575397-13 from 1545824 to 2058792 tid 2 2021-02-10T04:37:54.397 INFO:tasks.rados.rados.0.smithi085.stdout:2533: writing smithi08575397-13 from 2704609 to 2823774 tid 3 2021-02-10T04:37:54.397 INFO:tasks.rados.rados.0.smithi085.stdout:2534: snap_remove snap 259 2021-02-10T04:37:54.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:54 smithi164 conmon[29288]: cluster 2021-02-10T04:37:53.269240+0000 osd.4 (osd.4) 1007 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:54.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:54 smithi164 conmon[29288]: cluster 2021-02-10T04:37:53.269249+0000 osd.4 (osd.4) 1008 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:54.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:54 smithi164 conmon[29288]: cluster 2021-02-10T04:37:53.269255+0000 osd.4 (osd.4) 1009 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:54.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:54 smithi164 conmon[29288]: cluster 2021-02-10T04:37:53.269262+0000 osd.4 (osd.4) 1010 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:54.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:54 smithi164 conmon[29288]: cluster 2021-02-10T04:37:54.147065+0000 mgr.y (mgr.14140) 689 : cluster [DBG] pgmap v1133: 168 pgs: 157 active+clean, 7 active+clean+snaptrim, 4 active+clean+snaptrim_wait; 396 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 7 op/s; 7.0 MiB/s, 5 objects/s recovering 2021-02-10T04:37:54.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:54 smithi164 conmon[33677]: debug 2021-02-10T04:37:54.236+0000 7fbec19a1700 -1 osd.4 655 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:54.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:54 smithi085 conmon[29319]: cluster 2021-02-10T04:37:53.269240+0000 osd.4 ( 2021-02-10T04:37:54.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:54 smithi085 conmon[29319]: osd.4) 1007 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:54.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:54 smithi085 conmon[29319]: cluster 2021-02-10T04:37:53.269249+0000 osd.4 (osd.4) 1008 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:54.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:54 smithi085 conmon[29319]: cluster 2021-02-10T04:37:53.269255+0000 osd.4 (osd.4) 1009 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:54.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:54 smithi085 conmon[29319]: cluster 2021-02-10T04:37:53.269262+0000 osd.4 (osd.4) 1010 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:54.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:54 smithi085 conmon[29319]: cluster 2021-02-10T04:37:54.147065+0000 mgr.y (mgr.14140) 689 : cluster [DBG] pgmap v1133: 168 pgs: 157 active+clean, 7 active+clean+snaptrim, 4 active+clean+snaptrim_wait; 396 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 7 op/s; 7.0 MiB/s, 5 objects/s recovering 2021-02-10T04:37:54.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:54 smithi085 conmon[35961]: cluster 2021-02-10T04:37:54.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:54 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:37:54.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:54 smithi085 conmon[35961]: T04:37:53.269240+0000 osd.4 (osd.4) 1007 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:54.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:54 smithi085 conmon[35961]: cluster 2021-02-10T04:37:54.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:54 smithi085 conmon[35961]: 2021-02-10T04:37:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:54 smithi085 conmon[35961]: 2021 2021-02-10T04:37:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:54 smithi085 conmon[35961]: - 2021-02-10T04:37:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:54 smithi085 conmon[35961]: 02-10T04:37 2021-02-10T04:37:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:54 smithi085 conmon[35961]: :53.269249+0000 osd.4 (osd.4) 1008 : cluster [WRN] 2021-02-10T04:37:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:54 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:54 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:37:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:54 smithi085 conmon[35961]: 10T04:37:53.269255+0000 osd.4 ( 2021-02-10T04:37:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:54 smithi085 conmon[35961]: osd.4) 1009 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:54 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:54 smithi085 conmon[35961]: 02-10T04: 2021-02-10T04:37:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:54 smithi085 conmon[35961]: 37:53.269262+0000 osd.4 (osd 2021-02-10T04:37:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:54 smithi085 conmon[35961]: .4) 1010 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:54 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:54 smithi085 conmon[35961]: 02-10T04:37:54.147065+0000 mgr.y (mgr. 2021-02-10T04:37:54.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:54 smithi085 conmon[35961]: 14140) 689 : cluster [DBG] pgmap v1133: 168 pgs: 157 active+clean, 7 active+clean+snaptrim, 4 active+clean+snaptrim_wait; 396 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 7 op/s; 7.0 MiB/s, 5 objects/s recovering 2021-02-10T04:37:55.038 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:37:55.038 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [], 2021-02-10T04:37:55.038 INFO:teuthology.orchestra.run.smithi085.stdout: "num_ops": 0 2021-02-10T04:37:55.038 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:37:55.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:55 smithi085 conmon[35961]: cluster 2021-02-10T04:37:54.237681+0000 2021-02-10T04:37:55.380 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:55 smithi085 conmon[35961]: osd.4 (osd.4) 1011 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:55.381 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:55 smithi085 conmon[35961]: cluster 2021-02-10T04:37:55.381 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:55 smithi085 conmon[35961]: 2021-02-10T04:37:54.237693+0000 osd.4 (osd.4) 1012 : cluster 2021-02-10T04:37:55.381 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:55 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:55.381 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:55 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:37:55.381 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:55 smithi085 conmon[35961]: T04:37:54.237701+0000 osd.4 ( 2021-02-10T04:37:55.381 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:55 smithi085 conmon[35961]: osd.4) 1013 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:55.381 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:55 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:37:55.381 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:55 smithi085 conmon[35961]: -10T04:37:54.237706+0000 osd.4 ( 2021-02-10T04:37:55.381 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:55 smithi085 conmon[35961]: osd.4) 1014 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:55.381 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:55 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:55.381 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:55 smithi085 conmon[35961]: 02-10T04:37:54.377327+0000 mon.a 2021-02-10T04:37:55.381 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:55 smithi085 conmon[35961]: (mon.0) 1533 : cluster [DBG] osdmap e656: 8 total, 8 up, 7 in 2021-02-10T04:37:55.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:55 smithi085 conmon[29319]: cluster 2021-02-10T04:37:54.237681 2021-02-10T04:37:55.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:55 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1011 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:55.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:55 smithi085 conmon[29319]: cluster 2021-02-10T04:37:54.237693+0000 osd.4 (osd.4) 1012 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:55.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:55 smithi085 conmon[29319]: cluster 2021-02-10T04:37:54.237701+0000 osd.4 (osd.4) 1013 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:55.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:55 smithi085 conmon[29319]: cluster 2021-02-10T04:37:54.237706+0000 osd.4 (osd.4) 1014 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:55.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:55 smithi085 conmon[29319]: cluster 2021-02-10T04:37:54.377327+0000 mon.a (mon.0) 1533 : cluster [DBG] osdmap e656: 8 total, 8 up, 7 in 2021-02-10T04:37:55.385 INFO:tasks.rados.rados.0.smithi085.stdout:2534: done (7 left) 2021-02-10T04:37:55.386 INFO:tasks.rados.rados.0.smithi085.stdout:2535: copy_from oid 36 from oid 8 current snap is 287 2021-02-10T04:37:55.386 INFO:tasks.rados.rados.0.smithi085.stdout:2531: expect (ObjNum 733 snap 234 seq_num 733) 2021-02-10T04:37:55.483 INFO:tasks.rados.rados.0.smithi085.stdout:2532: expect (ObjNum 217 snap 49 seq_num 217) 2021-02-10T04:37:55.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:55 smithi164 conmon[33677]: debug 2021-02-10T04:37:55.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:55 smithi164 conmon[33677]: 2021-02-10T04:37:55.270+0000 7fbec19a1700 -1 osd.4 656 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:55.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:55 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:37:55.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:55 smithi164 conmon[29288]: 04:37:54.237681+0000 osd.4 (osd.4) 1011 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:55.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:55 smithi164 conmon[29288]: cluster 2021-02-10T04:37:54.237693+0000 osd.4 2021-02-10T04:37:55.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:55 smithi164 conmon[29288]: (osd.4) 1012 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:55.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:55 smithi164 conmon[29288]: cluster 2021-02-10T04:37:55.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:55 smithi164 conmon[29288]: 2021-02-10T04:37:54.237701+0000 osd.4 (osd.4) 1013 : cluster [WRN] 2021-02-10T04:37:55.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:55 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:55.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:55 smithi164 conmon[29288]: cluster 2021-02-10T04:37:55.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:55 smithi164 conmon[29288]: 2021-02-10T04:37: 2021-02-10T04:37:55.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:55 smithi164 conmon[29288]: 54.237706+0000 osd.4 (osd.4) 1014 : cluster 2021-02-10T04:37:55.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:55 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:55.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:55 smithi164 conmon[29288]: cluster 2021-02-10T04:37:55.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:55 smithi164 conmon[29288]: 2021-02-10T04:37:54.377327+0000 2021-02-10T04:37:55.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:55 smithi164 conmon[29288]: mon.a (mon.0) 1533 : cluster [DBG] osdmap e656: 8 total, 8 up, 7 in 2021-02-10T04:37:55.766 INFO:tasks.rados.rados.0.smithi085.stdout:2533: finishing write tid 1 to smithi08575397-13 2021-02-10T04:37:55.767 INFO:tasks.rados.rados.0.smithi085.stdout:2533: finishing write tid 2 to smithi08575397-13 2021-02-10T04:37:55.767 INFO:tasks.rados.rados.0.smithi085.stdout:2533: finishing write tid 3 to smithi08575397-13 2021-02-10T04:37:55.767 INFO:tasks.rados.rados.0.smithi085.stdout:2533: finishing write tid 4 to smithi08575397-13 2021-02-10T04:37:55.767 INFO:tasks.rados.rados.0.smithi085.stdout:2533: finishing write tid 5 to smithi08575397-13 2021-02-10T04:37:55.767 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 784 (ObjNum 882 snap 287 seq_num 882) dirty exists 2021-02-10T04:37:55.767 INFO:tasks.rados.rados.0.smithi085.stdout:2533: left oid 13 (ObjNum 882 snap 287 seq_num 882) 2021-02-10T04:37:55.767 INFO:tasks.rados.rados.0.smithi085.stdout:2531: done (7 left) 2021-02-10T04:37:55.767 INFO:tasks.rados.rados.0.smithi085.stdout:2532: done (6 left) 2021-02-10T04:37:55.767 INFO:tasks.rados.rados.0.smithi085.stdout:2533: done (5 left) 2021-02-10T04:37:55.767 INFO:tasks.rados.rados.0.smithi085.stdout:2536: rollback oid 7 current snap is 287 2021-02-10T04:37:55.767 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 7 to 286 2021-02-10T04:37:55.767 INFO:tasks.rados.rados.0.smithi085.stdout:2535: finishing copy_from to smithi08575397-36 2021-02-10T04:37:55.768 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 1144 (ObjNum 864 snap 278 seq_num 864) dirty exists 2021-02-10T04:37:55.768 INFO:tasks.rados.rados.0.smithi085.stdout:2535: finishing copy_from racing read to smithi08575397-36 2021-02-10T04:37:55.768 INFO:tasks.rados.rados.0.smithi085.stdout:2535: done (5 left) 2021-02-10T04:37:55.768 INFO:tasks.rados.rados.0.smithi085.stdout:2537: delete oid 43 current snap is 287 2021-02-10T04:37:55.770 INFO:tasks.rados.rados.0.smithi085.stdout:2536: finishing rollback tid 0 to smithi08575397-7 2021-02-10T04:37:55.772 INFO:tasks.rados.rados.0.smithi085.stdout:2536: finishing rollback tid 1 to smithi08575397-7 2021-02-10T04:37:55.772 INFO:tasks.rados.rados.0.smithi085.stdout:2536: finishing rollback tid 2 to smithi08575397-7 2021-02-10T04:37:55.772 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 1159 (ObjNum 759 snap 243 seq_num 759) dirty exists 2021-02-10T04:37:55.774 INFO:tasks.rados.rados.0.smithi085.stdout:2536: done (5 left) 2021-02-10T04:37:55.774 INFO:tasks.rados.rados.0.smithi085.stdout:2537: done (4 left) 2021-02-10T04:37:55.774 INFO:tasks.rados.rados.0.smithi085.stdout:2538: write (excl) oid 43 current snap is 287 2021-02-10T04:37:55.774 INFO:tasks.rados.rados.0.smithi085.stdout:2538: seq_num 884 ranges {457388=417064,1546435=516637,2643102=214286} 2021-02-10T04:37:55.779 INFO:tasks.rados.rados.0.smithi085.stdout:2538: writing smithi08575397-43 from 457388 to 874452 tid 1 2021-02-10T04:37:55.787 INFO:tasks.rados.rados.0.smithi085.stdout:2538: writing smithi08575397-43 from 1546435 to 2063072 tid 2 2021-02-10T04:37:55.792 INFO:tasks.rados.rados.0.smithi085.stdout:2538: writing smithi08575397-43 from 2643102 to 2857388 tid 3 2021-02-10T04:37:55.793 INFO:tasks.rados.rados.0.smithi085.stdout:2539: rollback oid 36 current snap is 287 2021-02-10T04:37:55.793 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 36 to 275 2021-02-10T04:37:55.793 INFO:tasks.rados.rados.0.smithi085.stdout:2538: finishing write tid 1 to smithi08575397-43 2021-02-10T04:37:55.793 INFO:tasks.rados.rados.0.smithi085.stdout:2540: write oid 25 current snap is 287 2021-02-10T04:37:55.793 INFO:tasks.rados.rados.0.smithi085.stdout:2540: seq_num 885 ranges {474195=492313,1604917=660295,2670525=203670} 2021-02-10T04:37:55.799 INFO:tasks.rados.rados.0.smithi085.stdout:2540: writing smithi08575397-25 from 474195 to 966508 tid 1 2021-02-10T04:37:55.807 INFO:tasks.rados.rados.0.smithi085.stdout:2540: writing smithi08575397-25 from 1604917 to 2265212 tid 2 2021-02-10T04:37:55.812 INFO:tasks.rados.rados.0.smithi085.stdout:2540: writing smithi08575397-25 from 2670525 to 2874195 tid 3 2021-02-10T04:37:55.812 INFO:tasks.rados.rados.0.smithi085.stdout:2541: rollback oid 42 current snap is 287 2021-02-10T04:37:55.812 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 42 to 272 2021-02-10T04:37:55.812 INFO:tasks.rados.rados.0.smithi085.stdout:2538: finishing write tid 2 to smithi08575397-43 2021-02-10T04:37:55.812 INFO:tasks.rados.rados.0.smithi085.stdout:2538: finishing write tid 3 to smithi08575397-43 2021-02-10T04:37:55.813 INFO:tasks.rados.rados.0.smithi085.stdout:2538: finishing write tid 4 to smithi08575397-43 2021-02-10T04:37:55.813 INFO:tasks.rados.rados.0.smithi085.stdout:2538: finishing write tid 5 to smithi08575397-43 2021-02-10T04:37:55.813 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 1164 (ObjNum 884 snap 287 seq_num 884) dirty exists 2021-02-10T04:37:55.813 INFO:tasks.rados.rados.0.smithi085.stdout:2538: left oid 43 (ObjNum 884 snap 287 seq_num 884) 2021-02-10T04:37:55.813 INFO:tasks.rados.rados.0.smithi085.stdout:2540: finishing write tid 1 to smithi08575397-25 2021-02-10T04:37:55.813 INFO:tasks.rados.rados.0.smithi085.stdout:2538: done (7 left) 2021-02-10T04:37:55.813 INFO:tasks.rados.rados.0.smithi085.stdout:2542: write oid 38 current snap is 287 2021-02-10T04:37:55.813 INFO:tasks.rados.rados.0.smithi085.stdout:2539: finishing rollback tid 0 to smithi08575397-36 2021-02-10T04:37:55.813 INFO:tasks.rados.rados.0.smithi085.stdout:2539: finishing rollback tid 1 to smithi08575397-36 2021-02-10T04:37:55.813 INFO:tasks.rados.rados.0.smithi085.stdout:2539: finishing rollback tid 2 to smithi08575397-36 2021-02-10T04:37:55.813 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 1267 (ObjNum 720 snap 226 seq_num 720) dirty exists 2021-02-10T04:37:55.814 INFO:tasks.rados.rados.0.smithi085.stdout:2542: seq_num 886 ranges {491002=567562,1663399=520306,2814301=76701} 2021-02-10T04:37:55.818 INFO:tasks.rados.rados.0.smithi085.stdout:2542: writing smithi08575397-38 from 491002 to 1058564 tid 1 2021-02-10T04:37:55.825 INFO:tasks.rados.rados.0.smithi085.stdout:2542: writing smithi08575397-38 from 1663399 to 2183705 tid 2 2021-02-10T04:37:55.830 INFO:tasks.rados.rados.0.smithi085.stdout:2542: writing smithi08575397-38 from 2814301 to 2891002 tid 3 2021-02-10T04:37:55.830 INFO:tasks.rados.rados.0.smithi085.stdout:2539: done (7 left) 2021-02-10T04:37:55.830 INFO:tasks.rados.rados.0.smithi085.stdout:2543: snap_create 2021-02-10T04:37:55.830 INFO:tasks.rados.rados.0.smithi085.stdout:2540: finishing write tid 2 to smithi08575397-25 2021-02-10T04:37:55.830 INFO:tasks.rados.rados.0.smithi085.stdout:2541: finishing rollback tid 0 to smithi08575397-42 2021-02-10T04:37:55.830 INFO:tasks.rados.rados.0.smithi085.stdout:2541: finishing rollback tid 1 to smithi08575397-42 2021-02-10T04:37:55.831 INFO:tasks.rados.rados.0.smithi085.stdout:2540: finishing write tid 3 to smithi08575397-25 2021-02-10T04:37:55.831 INFO:tasks.rados.rados.0.smithi085.stdout:2540: finishing write tid 4 to smithi08575397-25 2021-02-10T04:37:55.831 INFO:tasks.rados.rados.0.smithi085.stdout:2541: finishing rollback tid 2 to smithi08575397-42 2021-02-10T04:37:55.831 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 919 (ObjNum 833 snap 270 seq_num 833) dirty exists 2021-02-10T04:37:55.831 INFO:tasks.rados.rados.0.smithi085.stdout:2540: finishing write tid 5 to smithi08575397-25 2021-02-10T04:37:55.831 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 25 v 1271 (ObjNum 885 snap 287 seq_num 885) dirty exists 2021-02-10T04:37:55.831 INFO:tasks.rados.rados.0.smithi085.stdout:2540: left oid 25 (ObjNum 885 snap 287 seq_num 885) 2021-02-10T04:37:55.831 INFO:tasks.rados.rados.0.smithi085.stdout:2542: finishing write tid 1 to smithi08575397-38 2021-02-10T04:37:55.831 INFO:tasks.rados.rados.0.smithi085.stdout:2542: finishing write tid 2 to smithi08575397-38 2021-02-10T04:37:55.833 INFO:tasks.rados.rados.0.smithi085.stdout:2542: finishing write tid 3 to smithi08575397-38 2021-02-10T04:37:55.833 INFO:tasks.rados.rados.0.smithi085.stdout:2542: finishing write tid 4 to smithi08575397-38 2021-02-10T04:37:55.834 INFO:tasks.rados.rados.0.smithi085.stdout:2542: finishing write tid 5 to smithi08575397-38 2021-02-10T04:37:55.834 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 1105 (ObjNum 886 snap 287 seq_num 886) dirty exists 2021-02-10T04:37:55.834 INFO:tasks.rados.rados.0.smithi085.stdout:2542: left oid 38 (ObjNum 886 snap 287 seq_num 886) 2021-02-10T04:37:55.910 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset noscrub 2021-02-10T04:37:55.934 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:37:56.265 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:37:56.287 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:37:56.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:56 smithi085 conmon[29319]: cluster 2021-02-10T04:37:55.271230+0000 2021-02-10T04:37:56.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:56 smithi085 conmon[29319]: osd.4 (osd.4) 1015 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:56.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:56 smithi085 conmon[29319]: cluster 2021-02-10T04:37:55.271240+0000 osd.4 (osd.4) 1016 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:56.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:56 smithi085 conmon[29319]: cluster 2021-02-10T04:37:55.271246+0000 osd.4 (osd.4) 1017 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:56.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:56 smithi085 conmon[29319]: cluster 2021-02-10T04:37:55.271252+0000 osd.4 (osd.4) 1018 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:56.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:56 smithi085 conmon[29319]: cluster 2021-02-10T04:37:55.385527+0000 mon.a (mon.0) 1534 : cluster [DBG] osdmap e657: 8 total, 8 up, 7 in 2021-02-10T04:37:56.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:56 smithi085 conmon[29319]: cluster 2021-02-10T04:37:56.147635+0000 2021-02-10T04:37:56.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:56 smithi085 conmon[29319]: mgr.y (mgr.14140) 690 : cluster [DBG] pgmap v1136: 168 pgs: 156 active+clean, 7 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 395 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 1.4 MiB/s wr, 13 op/s 2021-02-10T04:37:56.392 INFO:tasks.rados.rados.0.smithi085.stdout:2540: done (7 left) 2021-02-10T04:37:56.392 INFO:tasks.rados.rados.0.smithi085.stdout:2541: done (6 left) 2021-02-10T04:37:56.392 INFO:tasks.rados.rados.0.smithi085.stdout:2542: done (5 left) 2021-02-10T04:37:56.392 INFO:tasks.rados.rados.0.smithi085.stdout:2543: done (4 left) 2021-02-10T04:37:56.392 INFO:tasks.rados.rados.0.smithi085.stdout:2544: write (excl) oid 36 current snap is 288 2021-02-10T04:37:56.392 INFO:tasks.rados.rados.0.smithi085.stdout:2544: seq_num 887 ranges {507809=642811,1721881=663964} 2021-02-10T04:37:56.398 INFO:tasks.rados.rados.0.smithi085.stdout:2544: writing smithi08575397-36 from 507809 to 1150620 tid 1 2021-02-10T04:37:56.406 INFO:tasks.rados.rados.0.smithi085.stdout:2544: writing smithi08575397-36 from 1721881 to 2385845 tid 2 2021-02-10T04:37:56.406 INFO:tasks.rados.rados.0.smithi085.stdout:2545: snap_create 2021-02-10T04:37:56.406 INFO:tasks.rados.rados.0.smithi085.stdout:2544: finishing write tid 1 to smithi08575397-36 2021-02-10T04:37:56.410 INFO:tasks.rados.rados.0.smithi085.stdout:2544: finishing write tid 2 to smithi08575397-36 2021-02-10T04:37:56.410 INFO:tasks.rados.rados.0.smithi085.stdout:2544: finishing write tid 3 to smithi08575397-36 2021-02-10T04:37:56.411 INFO:tasks.rados.rados.0.smithi085.stdout:2544: finishing write tid 4 to smithi08575397-36 2021-02-10T04:37:56.411 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 1270 (ObjNum 887 snap 288 seq_num 887) dirty exists 2021-02-10T04:37:56.411 INFO:tasks.rados.rados.0.smithi085.stdout:2544: left oid 36 (ObjNum 887 snap 288 seq_num 887) 2021-02-10T04:37:56.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:56 smithi164 conmon[33677]: debug 2021-02-10T04:37:56.232+0000 7fbec19a1700 -1 osd.4 657 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:56.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:56 smithi164 conmon[29288]: cluster 2021-02-10T04:37:55.271230+0000 osd.4 (osd. 2021-02-10T04:37:56.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:56 smithi164 conmon[29288]: 4) 1015 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:56.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:56 smithi164 conmon[29288]: cluster 2021-02-10T04:37:55.271240+0000 osd.4 (osd.4) 1016 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:56.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:56 smithi164 conmon[29288]: cluster 2021-02-10T04:37:56.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:56 smithi164 conmon[29288]: 2021-02-10T04:37:55.271246+0000 osd.4 (osd 2021-02-10T04:37:56.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:56 smithi164 conmon[29288]: .4) 1017 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:56.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:56 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:37:56.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:56 smithi164 conmon[29288]: -02-10T04:37:55.271252+0000 osd.4 2021-02-10T04:37:56.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:56 smithi164 conmon[29288]: (osd.4) 1018 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:56.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:56 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:37:56.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:56 smithi164 conmon[29288]: -02-10T04:37:55.385527+0000 2021-02-10T04:37:56.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:56 smithi164 conmon[29288]: mon.a (mon.0) 1534 : cluster [DBG] osdmap e657: 8 total, 8 up, 7 in 2021-02-10T04:37:56.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:56 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:37:56.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:56 smithi164 conmon[29288]: -10T04:37:56.147635+0000 mgr.y (mgr.14140 2021-02-10T04:37:56.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:56 smithi164 conmon[29288]: ) 690 : cluster [DBG] pgmap v1136: 168 pgs: 156 active+clean, 7 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 395 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 1.4 MiB/s wr, 13 op/s 2021-02-10T04:37:56.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:56 smithi085 conmon[35961]: cluster 2021-02-10T04:37:55.271230+0000 osd.4 (osd.4) 1015 : 2021-02-10T04:37:56.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:56 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:56.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:56 smithi085 conmon[35961]: cluster 2021-02-10T04:37:55.271240+0000 osd.4 (osd.4) 1016 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:56.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:56 smithi085 conmon[35961]: cluster 2021-02-10T04:37:55.271246+0000 2021-02-10T04:37:56.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:56 smithi085 conmon[35961]: osd.4 (osd.4) 1017 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:56.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:56 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:37:56.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:56 smithi085 conmon[35961]: 02-10T04:37:55.271252+0000 osd.4 (osd 2021-02-10T04:37:56.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:56 smithi085 conmon[35961]: .4) 1018 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:56.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:56 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:37:56.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:56 smithi085 conmon[35961]: 10T04:37:55.385527+0000 2021-02-10T04:37:56.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:56 smithi085 conmon[35961]: mon.a (mon.0) 1534 : cluster 2021-02-10T04:37:56.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:56 smithi085 conmon[35961]: [DBG] osdmap e657: 8 total, 8 up, 7 in 2021-02-10T04:37:56.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:56 smithi085 conmon[35961]: cluster 2021-02-10T04:37:56.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:56 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:37:56.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:56 smithi085 conmon[35961]: 10T04:37:56. 2021-02-10T04:37:56.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:56 smithi085 conmon[35961]: 147635+0000 mgr.y (mgr.14140) 2021-02-10T04:37:56.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:56 smithi085 conmon[35961]: 690 : cluster [DBG] pgmap v1136: 168 pgs: 156 active+clean, 7 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 395 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 1.4 MiB/s wr, 13 op/s 2021-02-10T04:37:57.397 INFO:tasks.rados.rados.0.smithi085.stdout:2544: done (5 left) 2021-02-10T04:37:57.397 INFO:tasks.rados.rados.0.smithi085.stdout:2545: done (4 left) 2021-02-10T04:37:57.397 INFO:tasks.rados.rados.0.smithi085.stdout:2546: delete oid 22 current snap is 289 2021-02-10T04:37:57.401 INFO:tasks.rados.rados.0.smithi085.stdout:2546: done (4 left) 2021-02-10T04:37:57.401 INFO:tasks.rados.rados.0.smithi085.stdout:2547: copy_from oid 4 from oid 48 current snap is 289 2021-02-10T04:37:57.401 INFO:tasks.rados.rados.0.smithi085.stdout:2548: rollback oid 16 current snap is 289 2021-02-10T04:37:57.402 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 16 to 264 2021-02-10T04:37:57.402 INFO:tasks.rados.rados.0.smithi085.stdout:2549: rollback oid 29 current snap is 289 2021-02-10T04:37:57.402 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 29 to 276 2021-02-10T04:37:57.402 INFO:tasks.rados.rados.0.smithi085.stdout:2550: write (excl) oid 50 current snap is 289 2021-02-10T04:37:57.402 INFO:tasks.rados.rados.0.smithi085.stdout:2550: seq_num 889 ranges {541423=793309,2122492=667633} 2021-02-10T04:37:57.410 INFO:tasks.rados.rados.0.smithi085.stdout:2550: writing smithi08575397-50 from 541423 to 1334732 tid 1 2021-02-10T04:37:57.419 INFO:tasks.rados.rados.0.smithi085.stdout:2550: writing smithi08575397-50 from 2122492 to 2790125 tid 2 2021-02-10T04:37:57.419 INFO:tasks.rados.rados.0.smithi085.stdout:2551: rollback oid 1 current snap is 289 2021-02-10T04:37:57.419 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 1 to 237 2021-02-10T04:37:57.419 INFO:tasks.rados.rados.0.smithi085.stdout:2547: finishing copy_from to smithi08575397-4 2021-02-10T04:37:57.419 INFO:tasks.rados.rados.0.smithi085.stdout:2547: got expected ENOENT (src dne) 2021-02-10T04:37:57.419 INFO:tasks.rados.rados.0.smithi085.stdout:2547: finishing copy_from racing read to smithi08575397-4 2021-02-10T04:37:57.419 INFO:tasks.rados.rados.0.smithi085.stdout:2547: done (8 left) 2021-02-10T04:37:57.420 INFO:tasks.rados.rados.0.smithi085.stdout:2552: snap_remove snap 277 2021-02-10T04:37:57.603 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:57 smithi164 conmon[33677]: debug 2021-02-10T04:37:57.228+0000 7fbec19a1700 -1 osd.4 658 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:57.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:57 smithi164 conmon[29288]: cluster 2021-02-10T04:37:56.233235+0000 2021-02-10T04:37:57.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:57 smithi164 conmon[29288]: osd.4 (osd.4) 1019 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:57.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:57 smithi164 conmon[29288]: cluster 2021-02-10T04:37:56.233243+0000 osd.4 2021-02-10T04:37:57.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:57 smithi164 conmon[29288]: (osd.4) 1020 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:57.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:57 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:37:57.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:57 smithi164 conmon[29288]: 37:56.233250+0000 osd.4 (osd.4) 1021 2021-02-10T04:37:57.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:57 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:57.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:57 smithi164 conmon[29288]: cluster 2021-02-10T04:37:57.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:57 smithi164 conmon[29288]: 2021-02-10T04:37: 2021-02-10T04:37:57.605 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:57 smithi164 conmon[29288]: 56.233257+0000 osd.4 (osd.4) 1022 : cluster 2021-02-10T04:37:57.605 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:57 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:57.605 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:57 smithi164 conmon[29288]: cluster 2021-02-10T04:37:57.605 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:57 smithi164 conmon[29288]: 2021-02-10T04:37:56.390228+0000 2021-02-10T04:37:57.605 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:57 smithi164 conmon[29288]: mon.a (mon.0) 1535 : cluster [DBG] 2021-02-10T04:37:57.605 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:57 smithi164 conmon[29288]: osdmap e658: 8 total, 8 up, 7 in 2021-02-10T04:37:57.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:57 smithi085 conmon[29319]: cluster 2021-02-10T04:37:56.233235+0000 osd.4 (osd.4) 1019 : cluster 2021-02-10T04:37:57.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:57 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:57.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:57 smithi085 conmon[29319]: cluster 2021-02-10T04:37:56.233243+0000 osd.4 (osd.4) 1020 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:57.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:57 smithi085 conmon[29319]: cluster 2021-02-10T04:37:56.233250+0000 osd.4 (osd.4) 1021 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:57.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:57 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:37:57.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:57 smithi085 conmon[29319]: 04:37:56.233257+0000 osd.4 (osd.4) 1022 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:57.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:57 smithi085 conmon[29319]: cluster 2021-02-10T04:37:56.390228+0000 mon.a (mon.0) 1535 : cluster [DBG] osdmap e658: 8 total, 8 up, 7 in 2021-02-10T04:37:57.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:57 smithi085 conmon[35961]: cluster 2021-02-10T04:37:57.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:57 smithi085 conmon[35961]: 2021-02-10T04:37:57.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:57 smithi085 conmon[35961]: 2021 2021-02-10T04:37:57.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:57 smithi085 conmon[35961]: -02-10T04 2021-02-10T04:37:57.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:57 smithi085 conmon[35961]: :37:56.233235+0000 osd.4 (osd.4) 1019 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:57.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:57 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:37:57.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:57 smithi085 conmon[35961]: 10T04:37:56.233243+0000 osd.4 (osd. 2021-02-10T04:37:57.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:57 smithi085 conmon[35961]: 4) 1020 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:57.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:57 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:37:57.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:57 smithi085 conmon[35961]: -10T04:37:56.233250+0000 osd.4 2021-02-10T04:37:57.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:57 smithi085 conmon[35961]: (osd.4) 1021 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:57.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:57 smithi085 conmon[35961]: 2021-02-10T04:37:57.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:57 smithi085 conmon[35961]: cluster 2021-02-10T04:37:57.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:57 smithi085 conmon[35961]: 2021-02-10T04:37:56.233257+0000 osd.4 ( 2021-02-10T04:37:57.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:57 smithi085 conmon[35961]: osd.4) 1022 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:57.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:57 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:37:57.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:57 smithi085 conmon[35961]: 04:37:56.390228+0000 mon.a (mon. 2021-02-10T04:37:57.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:57 smithi085 conmon[35961]: 0) 1535 : cluster [DBG] osdmap e658: 8 total, 8 up, 7 in 2021-02-10T04:37:57.893 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:57 smithi164 conmon[33677]: debug 2021-02-10T04:37:57.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:57 smithi164 conmon[33677]: 2021-02-10T04:37:57.603+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:37:57.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:57 smithi164 conmon[33677]: 2021-02-10T04:37:58.101 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:37:58.213 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:37:58.214 INFO:tasks.rados.rados.0.smithi085.stdout:2552: done (8 left) 2021-02-10T04:37:58.214 INFO:tasks.rados.rados.0.smithi085.stdout:2553: snap_remove snap 282 2021-02-10T04:37:58.252 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:37:58.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[33677]: debug 2021-02-10T04:37:58.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[33677]: 2021-02-10T04:37:58.216+0000 7fbec19a1700 -1 osd.4 660 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:58.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[40892]: debug 2021-02-10T04:37:58.342+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:37:58.394 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:37:58.529 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:37:58.635 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:37:58.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: cluster 2021-02-10T04:37:57.229417+0000 osd.4 ( 2021-02-10T04:37:58.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: osd.4) 1023 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:58.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: cluster 2021-02-10T04:37:57.229425+0000 osd.4 (osd.4) 1024 : cluster 2021-02-10T04:37:58.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:58.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:37:58.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: 04:37:57.229432+0000 osd.4 (osd.4 2021-02-10T04:37:58.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: ) 1025 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:58.666 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:37:58.666 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: 02-10T04:37:57. 2021-02-10T04:37:58.666 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: 229437+0000 osd.4 (osd.4) 2021-02-10T04:37:58.666 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: 1026 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:58.666 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:37:58.666 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: -02-10T 2021-02-10T04:37:58.666 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: 04:37: 2021-02-10T04:37:58.666 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: 57.395273+0000 mon.a (mon.0) 1536 : 2021-02-10T04:37:58.666 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: cluster [DBG] osdmap e659: 8 total, 8 up, 7 in 2021-02-10T04:37:58.666 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:37:58.666 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: -10T04:37:57.635408+0000 mon.a (mon.0) 1537 2021-02-10T04:37:58.666 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:37:58.667 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: audit 2021-02-10T04:37 2021-02-10T04:37:58.667 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: :57.635659+0000 mon.a (mon.0) 1538 : audit 2021-02-10T04:37:58.667 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:37:58.667 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: cluster 2021-02-10T04:37:58. 2021-02-10T04:37:58.667 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: 148251+0000 mgr.y (mgr.14140) 691 : cluster [DBG] 2021-02-10T04:37:58.667 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: pgmap v1139: 168 pgs: 156 active+clean, 7 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 396 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 637 KiB/s wr, 12 op/s 2021-02-10T04:37:58.667 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:37:58.667 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: 02-10T04:37:58. 2021-02-10T04:37:58.667 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: 204864+0000 mon.a (mon.0) 2021-02-10T04:37:58.667 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: 1539 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 502 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:37:58.667 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:37:58.667 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: T04:37:58.211658+0000 mon.a (mon 2021-02-10T04:37:58.667 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[29288]: .0) 1540 : cluster [DBG] osdmap e660: 8 total, 8 up, 7 in 2021-02-10T04:37:58.668 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:37:58.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[35961]: cluster 2021-02-10T04:37:57.229417+0000 2021-02-10T04:37:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[35961]: osd.4 (osd.4) 1023 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:37:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[35961]: -10T04:37:57.229425+0000 osd.4 (osd.4) 1024 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:37:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[35961]: 10T04:37:57.229432+0000 osd.4 (osd.4 2021-02-10T04:37:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[35961]: ) 1025 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[35961]: cluster 2021-02-10T04:37:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[35961]: 2021-02-10T04:37:57.229437+0000 2021-02-10T04:37:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[35961]: osd.4 (osd.4) 1026 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[35961]: cluster 2021-02-10T04:37:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[35961]: 2021-02-10T04:37:57. 2021-02-10T04:37:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[35961]: 395273+0000 mon.a (mon.0) 1536 : cluster [DBG] 2021-02-10T04:37:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[35961]: osdmap e659: 8 total, 8 up, 7 in 2021-02-10T04:37:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:37:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[35961]: 04:37:57.635408+0000 mon.a (mon.0) 1537 : audit [DBG] 2021-02-10T04:37:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:37:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:37:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[35961]: :37:57.635659+0000 mon.a (mon.0) 1538 2021-02-10T04:37:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[35961]: : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:37:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:37:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[35961]: 37:58.148251+0000 mgr.y (mgr.14140) 691 : cluster [DBG] pgmap v1139: 168 pgs: 156 active+clean, 7 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 396 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 637 KiB/s wr, 12 op/s 2021-02-10T04:37:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[35961]: cluster 2021-02-10T04:37 2021-02-10T04:37:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[35961]: :58.204864+0000 mon.a (mon.0) 1539 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 502 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:37:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[35961]: cluster 2021-02-10T04:37:58.211658+0000 mon.a 2021-02-10T04:37:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[35961]: (mon.0) 1540 : cluster [DBG] osdmap e660: 8 total, 8 up, 7 in 2021-02-10T04:37:58.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: cluster 2021-02-10T04:37:58.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:37:58.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: T04:37:57.229417+0000 osd.4 (osd.4) 1023 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:58.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:58.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: 02-10T04:37:57.229425+0000 osd.4 (osd 2021-02-10T04:37:58.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: .4) 1024 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:58.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:58.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: 02-10T04:37:57.229432+0000 2021-02-10T04:37:58.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: osd.4 (osd.4) 1025 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:58.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: 2021-02-10T04:37:58.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:37:58.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: -02-10T04:37:57.229437+0000 osd.4 (osd. 2021-02-10T04:37:58.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: 4) 1026 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:58.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:37:58.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: 10T04:37:57.395273+0000 mon.a ( 2021-02-10T04:37:58.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: mon.0) 1536 : cluster [DBG] osdmap e659: 8 total, 8 up, 7 in 2021-02-10T04:37:58.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:37:58.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: 02-10T04:37:57.635408+0000 mon.a 2021-02-10T04:37:58.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: (mon.0) 1537 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:37:58.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: audit 2021 2021-02-10T04:37:58.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: -02-10T04:37:57.635659+0000 2021-02-10T04:37:58.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: mon.a (mon.0) 1538 : audit [DBG] 2021-02-10T04:37:58.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:37:58.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:37:58.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: 37:58.148251+0000 mgr.y (mgr.14140 2021-02-10T04:37:58.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: ) 691 : cluster [DBG] pgmap v1139: 168 pgs: 156 active+clean, 7 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 396 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 637 KiB/s wr, 12 op/s 2021-02-10T04:37:58.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:37:58.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: T04:37:58.204864+0000 mon.a ( 2021-02-10T04:37:58.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: mon.0) 1539 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 502 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:37:58.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:37:58.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: 02-10T04:37:58.211658+0000 mon.a (mon 2021-02-10T04:37:58.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:58 smithi085 conmon[29319]: .0) 1540 : cluster [DBG] osdmap e660: 8 total, 8 up, 7 in 2021-02-10T04:37:58.890 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:37:58.994 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:37:59.143 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:37:58 smithi164 conmon[40892]: debug 2021-02-10T04:37:58.745+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:37:59.215 INFO:tasks.rados.rados.0.smithi085.stdout:2553: done (8 left) 2021-02-10T04:37:59.215 INFO:tasks.rados.rados.0.smithi085.stdout:2554: snap_remove snap 244 2021-02-10T04:37:59.293 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:37:59.643 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[40892]: debug 2021-02-10T04:37:59.644 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[40892]: 2021-02-10T04:37:59.380+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:37:59.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[33677]: debug 2021-02-10T04:37:59.234+0000 7fbec19a1700 -1 osd.4 661 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:37:59.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: cluster 2021-02-10T04:37:59.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: 2021-02-10T04:37:59.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: 2021 2021-02-10T04:37:59.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: - 2021-02-10T04:37:59.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: 02 2021-02-10T04:37:59.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: - 2021-02-10T04:37:59.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: 10 2021-02-10T04:37:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: T 2021-02-10T04:37:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: 04 2021-02-10T04:37:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: : 2021-02-10T04:37:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: 37 2021-02-10T04:37:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: : 2021-02-10T04:37:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: 58 2021-02-10T04:37:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: . 2021-02-10T04:37:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: 217010 2021-02-10T04:37:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: +0000 2021-02-10T04:37:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: osd.4 (osd.4) 1027 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: 2021-02-10T04:37:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: cluster 2021-02-10T04:37:58.217020+0000 2021-02-10T04:37:59.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: osd.4 (osd.4) 1028 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:59.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: cluster 2021-02-10T04:37:59.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: 2021-02-10T04:37:58.217026+0000 osd.4 (osd.4) 1029 : cluster [WRN] 2021-02-10T04:37:59.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:59.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:37:59.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: :37:58.217032+0000 osd.4 (osd.4) 1030 2021-02-10T04:37:59.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:59.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: audit 2021-02-10T04:37:59.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: 2021-02-10T04:37 2021-02-10T04:37:59.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: :58.240008+0000 mgr.y (mgr.14140) 692 2021-02-10T04:37:59.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:37:59.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: cluster 2021-02-10T04:37:59.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:37:59.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: :37:59.215461+0000 mon.a (mon 2021-02-10T04:37:59.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:37:59 smithi164 conmon[29288]: .0) 1541 : cluster [DBG] osdmap e661: 8 total, 8 up, 7 in 2021-02-10T04:37:59.647 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:37:59.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[29319]: cluster 2021-02-10T04:37:59.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[29319]: 2021-02-10T04:37:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[29319]: 2021-02-10T04:37:58.217010+0000 osd.4 (osd.4) 1027 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[29319]: 2021-02-10T04:37:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[29319]: cluster 2021-02-10T04:37:58.217020 2021-02-10T04:37:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1028 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[29319]: cluster 2021-02-10T04:37:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[29319]: 2021-02-10T04:37:58.217026+0000 osd.4 (osd.4 2021-02-10T04:37:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[29319]: ) 1029 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:37:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[29319]: 10T04:37:58.217032+0000 osd.4 (osd.4) 2021-02-10T04:37:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[29319]: 1030 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:37:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[29319]: -10T04:37:58.240008+0000 2021-02-10T04:37:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[29319]: mgr.y (mgr.14140) 692 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:37:59.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[29319]: 2021-02-10T04:37:59.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:37:59.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[29319]: :37:59.215461+0000 mon.a (mon 2021-02-10T04:37:59.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[29319]: .0) 1541 : cluster [DBG] osdmap e661: 8 total, 8 up, 7 in 2021-02-10T04:37:59.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[35961]: cluster 2021-02-10T04:37:59.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[35961]: 2021-02-10T04:37:59.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[35961]: 2021-02-10T04:37: 2021-02-10T04:37:59.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[35961]: 58.217010+0000 osd.4 (osd.4) 1027 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:37:59.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:37:59.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[35961]: -10T04:37:58.217020+0000 osd.4 2021-02-10T04:37:59.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[35961]: (osd.4) 1028 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:37:59.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[35961]: 2021-02-10T04:37:59.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[35961]: cluster 2021-02-10T04:37 2021-02-10T04:37:59.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[35961]: :58.217026+0000 osd.4 (osd.4 2021-02-10T04:37:59.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[35961]: ) 1029 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:37:59.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:59.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[35961]: -02-10T04:37:58.217032 2021-02-10T04:37:59.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1030 : cluster [WRN] 2021-02-10T04:37:59.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:37:59.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:37:59.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[35961]: :37:58.240008+0000 mgr.y (mgr. 2021-02-10T04:37:59.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[35961]: 14140) 692 2021-02-10T04:37:59.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[35961]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:37:59.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:37:59.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[35961]: -02-10T04:37:59. 2021-02-10T04:37:59.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[35961]: 215461+0000 mon.a ( 2021-02-10T04:37:59.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[35961]: mon.0) 1541 : cluster [DBG] osdmap e661: 8 total, 8 up, 7 in 2021-02-10T04:37:59.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:37:59 smithi085 conmon[35961]: 2021-02-10T04:37:59.751 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:37:59.784 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:38:00.221 INFO:tasks.rados.rados.0.smithi085.stdout:2554: done (8 left) 2021-02-10T04:38:00.221 INFO:tasks.rados.rados.0.smithi085.stdout:2555: delete oid 11 current snap is 289 2021-02-10T04:38:00.221 INFO:tasks.rados.rados.0.smithi085.stdout:2549: finishing rollback tid 0 to smithi08575397-29 2021-02-10T04:38:00.221 INFO:tasks.rados.rados.0.smithi085.stdout:2549: finishing rollback tid 1 to smithi08575397-29 2021-02-10T04:38:00.221 INFO:tasks.rados.rados.0.smithi085.stdout:2549: finishing rollback tid 2 to smithi08575397-29 2021-02-10T04:38:00.221 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 29 v 633 (ObjNum 8 snap 0 seq_num 8) dirty exists 2021-02-10T04:38:00.222 INFO:tasks.rados.rados.0.smithi085.stdout:2548: finishing rollback tid 1 to smithi08575397-16 2021-02-10T04:38:00.222 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 1118 (ObjNum 71101440 snap 22030 seq_num 4000000) dirty dne 2021-02-10T04:38:00.222 INFO:tasks.rados.rados.0.smithi085.stdout:2550: finishing write tid 1 to smithi08575397-50 2021-02-10T04:38:00.222 INFO:tasks.rados.rados.0.smithi085.stdout:2550: finishing write tid 2 to smithi08575397-50 2021-02-10T04:38:00.222 INFO:tasks.rados.rados.0.smithi085.stdout:2550: finishing write tid 3 to smithi08575397-50 2021-02-10T04:38:00.222 INFO:tasks.rados.rados.0.smithi085.stdout:2550: finishing write tid 4 to smithi08575397-50 2021-02-10T04:38:00.222 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 1318 (ObjNum 889 snap 289 seq_num 889) dirty exists 2021-02-10T04:38:00.222 INFO:tasks.rados.rados.0.smithi085.stdout:2550: left oid 50 (ObjNum 889 snap 289 seq_num 889) 2021-02-10T04:38:00.222 INFO:tasks.rados.rados.0.smithi085.stdout:2551: finishing rollback tid 0 to smithi08575397-1 2021-02-10T04:38:00.222 INFO:tasks.rados.rados.0.smithi085.stdout:2551: finishing rollback tid 1 to smithi08575397-1 2021-02-10T04:38:00.222 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 1320 (ObjNum 82778384 snap 22030 seq_num 0) dirty dne 2021-02-10T04:38:00.226 INFO:tasks.rados.rados.0.smithi085.stdout:2548: done (8 left) 2021-02-10T04:38:00.226 INFO:tasks.rados.rados.0.smithi085.stdout:2549: done (7 left) 2021-02-10T04:38:00.226 INFO:tasks.rados.rados.0.smithi085.stdout:2550: done (6 left) 2021-02-10T04:38:00.226 INFO:tasks.rados.rados.0.smithi085.stdout:2551: done (5 left) 2021-02-10T04:38:00.226 INFO:tasks.rados.rados.0.smithi085.stdout:2555: done (4 left) 2021-02-10T04:38:00.226 INFO:tasks.rados.rados.0.smithi085.stdout:2556: read oid 48 snap -1 2021-02-10T04:38:00.226 INFO:tasks.rados.rados.0.smithi085.stdout:2556: expect deleted 2021-02-10T04:38:00.226 INFO:tasks.rados.rados.0.smithi085.stdout:2557: copy_from oid 50 from oid 1 current snap is 289 2021-02-10T04:38:00.227 INFO:tasks.rados.rados.0.smithi085.stdout:2558: read oid 28 snap -1 2021-02-10T04:38:00.227 INFO:tasks.rados.rados.0.smithi085.stdout:2558: expect (ObjNum 860 snap 277 seq_num 860) 2021-02-10T04:38:00.227 INFO:tasks.rados.rados.0.smithi085.stdout:2559: delete oid 43 current snap is 289 2021-02-10T04:38:00.233 INFO:tasks.rados.rados.0.smithi085.stdout:2556: done (7 left) 2021-02-10T04:38:00.233 INFO:tasks.rados.rados.0.smithi085.stdout:2559: done (6 left) 2021-02-10T04:38:00.233 INFO:tasks.rados.rados.0.smithi085.stdout:2560: write oid 43 current snap is 289 2021-02-10T04:38:00.233 INFO:tasks.rados.rados.0.smithi085.stdout:2560: seq_num 892 ranges {591844=735409,1897938=531313,2862486=129358} 2021-02-10T04:38:00.238 INFO:tasks.rados.rados.0.smithi085.stdout:2560: writing smithi08575397-43 from 591844 to 1327253 tid 1 2021-02-10T04:38:00.245 INFO:tasks.rados.rados.0.smithi085.stdout:2560: writing smithi08575397-43 from 1897938 to 2429251 tid 2 2021-02-10T04:38:00.249 INFO:tasks.rados.rados.0.smithi085.stdout:2560: writing smithi08575397-43 from 2862486 to 2991844 tid 3 2021-02-10T04:38:00.249 INFO:tasks.rados.rados.0.smithi085.stdout:2558: expect (ObjNum 860 snap 277 seq_num 860) 2021-02-10T04:38:00.364 INFO:tasks.rados.rados.0.smithi085.stdout:2557: finishing copy_from to smithi08575397-50 2021-02-10T04:38:00.364 INFO:tasks.rados.rados.0.smithi085.stdout:2557: got expected ENOENT (src dne) 2021-02-10T04:38:00.364 INFO:tasks.rados.rados.0.smithi085.stdout:2557: finishing copy_from racing read to smithi08575397-50 2021-02-10T04:38:00.364 INFO:tasks.rados.rados.0.smithi085.stdout:2560: finishing write tid 1 to smithi08575397-43 2021-02-10T04:38:00.364 INFO:tasks.rados.rados.0.smithi085.stdout:2557: done (6 left) 2021-02-10T04:38:00.364 INFO:tasks.rados.rados.0.smithi085.stdout:2558: done (5 left) 2021-02-10T04:38:00.364 INFO:tasks.rados.rados.0.smithi085.stdout:2561: rollback oid 2 current snap is 289 2021-02-10T04:38:00.365 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 2 to 286 2021-02-10T04:38:00.365 INFO:tasks.rados.rados.0.smithi085.stdout:2560: finishing write tid 2 to smithi08575397-43 2021-02-10T04:38:00.365 INFO:tasks.rados.rados.0.smithi085.stdout:2560: finishing write tid 3 to smithi08575397-43 2021-02-10T04:38:00.365 INFO:tasks.rados.rados.0.smithi085.stdout:2560: finishing write tid 4 to smithi08575397-43 2021-02-10T04:38:00.365 INFO:tasks.rados.rados.0.smithi085.stdout:2560: finishing write tid 5 to smithi08575397-43 2021-02-10T04:38:00.365 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 1169 (ObjNum 892 snap 289 seq_num 892) dirty exists 2021-02-10T04:38:00.365 INFO:tasks.rados.rados.0.smithi085.stdout:2560: left oid 43 (ObjNum 892 snap 289 seq_num 892) 2021-02-10T04:38:00.365 INFO:tasks.rados.rados.0.smithi085.stdout:2560: done (5 left) 2021-02-10T04:38:00.365 INFO:tasks.rados.rados.0.smithi085.stdout:2562: snap_create 2021-02-10T04:38:00.367 INFO:tasks.rados.rados.0.smithi085.stdout:2561: finishing rollback tid 0 to smithi08575397-2 2021-02-10T04:38:00.368 INFO:tasks.rados.rados.0.smithi085.stdout:2561: finishing rollback tid 1 to smithi08575397-2 2021-02-10T04:38:00.369 INFO:tasks.rados.rados.0.smithi085.stdout:2561: finishing rollback tid 2 to smithi08575397-2 2021-02-10T04:38:00.369 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 871 (ObjNum 735 snap 234 seq_num 735) dirty exists 2021-02-10T04:38:00.498 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:00 smithi164 conmon[29288]: cluster 2021-02-10T04:37:59.235364+0000 2021-02-10T04:38:00.498 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:00 smithi164 conmon[29288]: osd.4 (osd.4) 1031 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:00.498 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:00 smithi164 conmon[29288]: cluster 2021-02-10T04:37:59.235373+0000 osd.4 (osd.4) 1032 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:00.498 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:00 smithi164 conmon[29288]: cluster 2021-02-10T04:37:59.235380+0000 osd.4 (osd.4) 1033 : 2021-02-10T04:38:00.498 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:00 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:00.498 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:00 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:38:00.498 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:00 smithi164 conmon[29288]: 10T04:37:59.235386+0000 osd.4 (osd.4 2021-02-10T04:38:00.498 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:00 smithi164 conmon[29288]: ) 1034 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:00.499 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:00 smithi164 conmon[29288]: cluster 2021-02-10T04:38: 2021-02-10T04:38:00.499 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:00 smithi164 conmon[29288]: 00.148891+0000 mgr.y (mgr.14140) 693 : cluster [DBG] pgmap v1142: 168 pgs: 156 active+clean, 7 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 396 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.6 MiB/s rd, 1005 KiB/s wr, 9 op/s 2021-02-10T04:38:00.499 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:00 smithi164 conmon[29288]: 2021-02-10T04:38:00.499 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:00 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:38:00.499 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:00 smithi164 conmon[29288]: 10T04:38:00.220304+0000 mon.a (mon.0) 1542 : cluster 2021-02-10T04:38:00.499 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:00 smithi164 conmon[29288]: [DBG] osdmap e662: 8 total, 8 up, 7 in 2021-02-10T04:38:00.499 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:00 smithi164 conmon[33677]: debug 2021-02-10T04:38:00.499 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:00 smithi164 conmon[33677]: 2021-02-10T04:38:00.195+0000 7fbec19a1700 -1 osd.4 661 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:38:00.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:00 smithi085 conmon[29319]: cluster 2021-02-10T04:37:59.235364+0000 osd.4 (osd.4) 1031 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:00.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:00 smithi085 conmon[29319]: cluster 2021-02-10T04:37:59.235373+0000 osd.4 (osd.4) 1032 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:00.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:00 smithi085 conmon[29319]: cluster 2021-02-10T04:37:59.235380+0000 osd.4 (osd.4) 1033 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:00.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:00 smithi085 conmon[29319]: cluster 2021-02-10T04:37:59.235386+0000 osd.4 (osd.4) 1034 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:00.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:00 smithi085 conmon[29319]: cluster 2021-02-10T04:38:00.148891+0000 mgr.y (mgr.14140) 693 : cluster [DBG] pgmap v1142: 168 pgs: 156 active+clean, 7 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 396 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.6 MiB/s rd, 1005 KiB/s wr, 9 op/s 2021-02-10T04:38:00.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:00 smithi085 conmon[29319]: cluster 2021-02-10T04:38:00.220304+0000 mon.a (mon.0) 1542 : cluster [DBG] osdmap e662: 8 total, 8 up, 7 in 2021-02-10T04:38:00.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:00 smithi085 conmon[35961]: cluster 2021-02-10T04:37:59.235364+0000 osd.4 (osd.4) 1031 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:00.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:00 smithi085 conmon[35961]: cluster 2021-02-10T04:37:59.235373+0000 osd.4 (osd.4) 1032 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:00.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:00 smithi085 conmon[35961]: cluster 2021-02-10T04:37:59.235380+0000 osd.4 (osd.4) 1033 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:00.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:00 smithi085 conmon[35961]: cluster 2021-02-10T04:37:59.235386+0000 osd.4 (osd.4) 1034 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:00.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:00 smithi085 conmon[35961]: cluster 2021-02-10T04:38:00.148891+0000 mgr.y (mgr.14140) 693 : cluster [DBG] pgmap v1142: 168 pgs: 156 active+clean, 7 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 396 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.6 MiB/s rd, 1005 KiB/s wr, 9 op/s 2021-02-10T04:38:00.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:00 smithi085 conmon[35961]: cluster 2021-02-10T04:38:00.220304+0000 mon.a (mon.0) 1542 : cluster [DBG] osdmap e662: 8 total, 8 up, 7 in 2021-02-10T04:38:01.176 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:38:01.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:38:00 smithi085 conmon[49106]: debug 2021-02-10T04:38:00.898+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:38:01.218 INFO:tasks.rados.rados.0.smithi085.stdout:2561: done (5 left) 2021-02-10T04:38:01.218 INFO:tasks.rados.rados.0.smithi085.stdout:2562: done (4 left) 2021-02-10T04:38:01.218 INFO:tasks.rados.rados.0.smithi085.stdout:2563: read oid 37 snap -1 2021-02-10T04:38:01.219 INFO:tasks.rados.rados.0.smithi085.stdout:2563: expect deleted 2021-02-10T04:38:01.219 INFO:tasks.rados.rados.0.smithi085.stdout:2564: copy_from oid 23 from oid 15 current snap is 290 2021-02-10T04:38:01.219 INFO:tasks.rados.rados.0.smithi085.stdout:2565: rollback oid 22 current snap is 290 2021-02-10T04:38:01.219 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 22 to 275 2021-02-10T04:38:01.219 INFO:tasks.rados.rados.0.smithi085.stdout:2566: read oid 26 snap -1 2021-02-10T04:38:01.219 INFO:tasks.rados.rados.0.smithi085.stdout:2566: expect (ObjNum 877 snap 282 seq_num 877) 2021-02-10T04:38:01.219 INFO:tasks.rados.rados.0.smithi085.stdout:2567: read oid 40 snap -1 2021-02-10T04:38:01.219 INFO:tasks.rados.rados.0.smithi085.stdout:2567: expect (ObjNum 879 snap 282 seq_num 879) 2021-02-10T04:38:01.219 INFO:tasks.rados.rados.0.smithi085.stdout:2568: rollback oid 8 current snap is 290 2021-02-10T04:38:01.219 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 8 to 274 2021-02-10T04:38:01.219 INFO:tasks.rados.rados.0.smithi085.stdout:2569: snap_create 2021-02-10T04:38:01.233 INFO:tasks.rados.rados.0.smithi085.stdout:2566: expect (ObjNum 877 snap 282 seq_num 877) 2021-02-10T04:38:01.404 INFO:tasks.rados.rados.0.smithi085.stdout:2567: expect (ObjNum 879 snap 282 seq_num 879) 2021-02-10T04:38:01.532 INFO:tasks.rados.rados.0.smithi085.stdout:2565: finishing rollback tid 1 to smithi08575397-22 2021-02-10T04:38:01.532 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 801 (ObjNum 81058784 snap 22030 seq_num 1006635216) dirty dne 2021-02-10T04:38:01.532 INFO:tasks.rados.rados.0.smithi085.stdout:2568: finishing rollback tid 0 to smithi08575397-8 2021-02-10T04:38:01.532 INFO:tasks.rados.rados.0.smithi085.stdout:2568: finishing rollback tid 1 to smithi08575397-8 2021-02-10T04:38:01.532 INFO:tasks.rados.rados.0.smithi085.stdout:2568: finishing rollback tid 2 to smithi08575397-8 2021-02-10T04:38:01.532 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 1323 (ObjNum 797 snap 256 seq_num 797) dirty exists 2021-02-10T04:38:01.533 INFO:tasks.rados.rados.0.smithi085.stdout:2564: finishing copy_from to smithi08575397-23 2021-02-10T04:38:01.533 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 104 (ObjNum 217 snap 49 seq_num 217) dirty exists 2021-02-10T04:38:01.533 INFO:tasks.rados.rados.0.smithi085.stdout:2564: finishing copy_from racing read to smithi08575397-23 2021-02-10T04:38:01.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:01 smithi164 conmon[29288]: cluster 2021-02-10T04:38:00.196069+0000 osd.4 ( 2021-02-10T04:38:01.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:01 smithi164 conmon[29288]: osd.4) 1035 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:01.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:01 smithi164 conmon[29288]: cluster 2021-02-10T04:38:00.196081+0000 osd.4 (osd.4) 1036 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:01.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:01 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:38:01.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:01 smithi164 conmon[29288]: 10T04:38:00.196087+0000 osd.4 (osd.4 2021-02-10T04:38:01.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:01 smithi164 conmon[29288]: ) 1037 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:01.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:01 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:38:01.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:01 smithi164 conmon[29288]: -10T04:38:00.196095+0000 osd.4 (osd 2021-02-10T04:38:01.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:01 smithi164 conmon[29288]: .4) 1038 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:01.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:01 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:38:01.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:01 smithi164 conmon[29288]: -10T04:38:01.217234+0000 mon.a ( 2021-02-10T04:38:01.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:01 smithi164 conmon[29288]: mon.0) 1543 : cluster [DBG] osdmap e663: 8 total, 8 up, 7 in 2021-02-10T04:38:01.645 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:01 smithi164 conmon[33677]: debug 2021-02-10T04:38:01.194+0000 7fbec19a1700 -1 osd.4 662 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:38:01.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[29319]: cluster 2021-02-10T04:38:00.196069+0000 osd.4 (osd.4) 1035 : cluster [WRN] 2021-02-10T04:38:01.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:01.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[29319]: cluster 2021-02-10T04:38:00.196081+0000 osd.4 (osd.4) 1036 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:01.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[29319]: cluster 2021-02-10T04:38:01.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[29319]: 2021-02-10T04:38:00.196087+0000 osd.4 (osd.4) 1037 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:01.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[29319]: cluster 2021-02-10T04:38:01.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[29319]: 2021-02-10T04:38:00.196095+0000 osd.4 (osd.4) 1038 : 2021-02-10T04:38:01.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:01.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[29319]: cluster 2021-02-10T04:38: 2021-02-10T04:38:01.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[29319]: 01.217234+0000 mon.a (mon.0) 1543 : cluster 2021-02-10T04:38:01.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[29319]: [DBG] osdmap e663: 8 total, 8 up, 7 in 2021-02-10T04:38:01.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: cluster 2021-02-10T04:38:01.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 2021-02-10T04:38:01.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 2021 2021-02-10T04:38:01.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: - 2021-02-10T04:38:01.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 02 2021-02-10T04:38:01.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: - 2021-02-10T04:38:01.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 10 2021-02-10T04:38:01.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: T 2021-02-10T04:38:01.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 04 2021-02-10T04:38:01.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: : 2021-02-10T04:38:01.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 38 2021-02-10T04:38:01.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: : 2021-02-10T04:38:01.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 00 2021-02-10T04:38:01.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: . 2021-02-10T04:38:01.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 196069 2021-02-10T04:38:01.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: +0000 2021-02-10T04:38:01.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 2021-02-10T04:38:01.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: osd.4 2021-02-10T04:38:01.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: ( 2021-02-10T04:38:01.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: osd 2021-02-10T04:38:01.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: . 2021-02-10T04:38:01.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 4 2021-02-10T04:38:01.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: ) 2021-02-10T04:38:01.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 1035 2021-02-10T04:38:01.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: : 2021-02-10T04:38:01.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: cluster 2021-02-10T04:38:01.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 2021-02-10T04:38:01.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: [WRN] 2021-02-10T04:38:01.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 2021-02-10T04:38:01.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:01.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 2021-02-10T04:38:01.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: cluster 2021-02-10T04:38:01.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 2021-02-10T04:38:01.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 2021 2021-02-10T04:38:01.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: - 2021-02-10T04:38:01.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 02 2021-02-10T04:38:01.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: - 2021-02-10T04:38:01.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 10 2021-02-10T04:38:01.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: T 2021-02-10T04:38:01.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 04 2021-02-10T04:38:01.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: : 2021-02-10T04:38:01.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 38 2021-02-10T04:38:01.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: : 2021-02-10T04:38:01.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 00 2021-02-10T04:38:01.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: . 2021-02-10T04:38:01.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 196081 2021-02-10T04:38:01.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: +0000 2021-02-10T04:38:01.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 2021-02-10T04:38:01.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: osd.4 2021-02-10T04:38:01.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: ( 2021-02-10T04:38:01.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: osd 2021-02-10T04:38:01.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: . 2021-02-10T04:38:01.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 4 2021-02-10T04:38:01.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: ) 2021-02-10T04:38:01.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 1036 2021-02-10T04:38:01.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: : 2021-02-10T04:38:01.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: cluster 2021-02-10T04:38:01.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 2021-02-10T04:38:01.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: [WRN] 2021-02-10T04:38:01.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 2021-02-10T04:38:01.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:01.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 2021-02-10T04:38:01.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: cluster 2021-02-10T04:38:01.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 2021-02-10T04:38:01.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 2021 2021-02-10T04:38:01.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: - 2021-02-10T04:38:01.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 02 2021-02-10T04:38:01.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: - 2021-02-10T04:38:01.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 10 2021-02-10T04:38:01.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: T 2021-02-10T04:38:01.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 04 2021-02-10T04:38:01.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: :38:00.196087+0000 osd.4 ( 2021-02-10T04:38:01.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: osd.4) 1037 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:01.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: cluster 2021-02-10T04:38:00.196095+0000 2021-02-10T04:38:01.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: osd.4 (osd.4) 1038 : cluster [WRN] 2021-02-10T04:38:01.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:01.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:38:01.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: 04:38:01.217234+0000 mon.a (mon 2021-02-10T04:38:01.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: .0) 1543 : cluster [DBG] 2021-02-10T04:38:01.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:01 smithi085 conmon[35961]: osdmap e663: 8 total, 8 up, 7 in 2021-02-10T04:38:02.221 INFO:tasks.rados.rados.0.smithi085.stdout:2563: done (10 left) 2021-02-10T04:38:02.222 INFO:tasks.rados.rados.0.smithi085.stdout:2564: done (9 left) 2021-02-10T04:38:02.222 INFO:tasks.rados.rados.0.smithi085.stdout:2565: done (8 left) 2021-02-10T04:38:02.222 INFO:tasks.rados.rados.0.smithi085.stdout:2566: done (7 left) 2021-02-10T04:38:02.222 INFO:tasks.rados.rados.0.smithi085.stdout:2567: done (6 left) 2021-02-10T04:38:02.222 INFO:tasks.rados.rados.0.smithi085.stdout:2568: done (5 left) 2021-02-10T04:38:02.222 INFO:tasks.rados.rados.0.smithi085.stdout:2569: done (4 left) 2021-02-10T04:38:02.222 INFO:tasks.rados.rados.0.smithi085.stdout:2570: delete oid 30 current snap is 291 2021-02-10T04:38:02.232 INFO:tasks.rados.rados.0.smithi085.stdout:2570: done (4 left) 2021-02-10T04:38:02.232 INFO:tasks.rados.rados.0.smithi085.stdout:2571: rollback oid 1 current snap is 291 2021-02-10T04:38:02.232 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 1 to 273 2021-02-10T04:38:02.232 INFO:tasks.rados.rados.0.smithi085.stdout:2572: write oid 7 current snap is 291 2021-02-10T04:38:02.232 INFO:tasks.rados.rados.0.smithi085.stdout:2572: seq_num 894 ranges {625458=485907,1614902=534982,2917332=108126} 2021-02-10T04:38:02.236 INFO:tasks.rados.rados.0.smithi085.stdout:2572: writing smithi08575397-7 from 625458 to 1111365 tid 1 2021-02-10T04:38:02.240 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:02 smithi164 conmon[33677]: debug 2021-02-10T04:38:02.238+0000 7fbec19a1700 -1 osd.4 664 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:38:02.243 INFO:tasks.rados.rados.0.smithi085.stdout:2572: writing smithi08575397-7 from 1614902 to 2149884 tid 2 2021-02-10T04:38:02.249 INFO:tasks.rados.rados.0.smithi085.stdout:2572: writing smithi08575397-7 from 2917332 to 3025458 tid 3 2021-02-10T04:38:02.249 INFO:tasks.rados.rados.0.smithi085.stdout:2573: rollback oid 15 current snap is 291 2021-02-10T04:38:02.249 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 15 to 230 2021-02-10T04:38:02.249 INFO:tasks.rados.rados.0.smithi085.stdout:2571: finishing rollback tid 1 to smithi08575397-1 2021-02-10T04:38:02.249 INFO:tasks.rados.rados.0.smithi085.stdout:2571: finishing rollback tid 2 to smithi08575397-1 2021-02-10T04:38:02.249 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 1325 (ObjNum 842 snap 273 seq_num 842) dirty exists 2021-02-10T04:38:02.249 INFO:tasks.rados.rados.0.smithi085.stdout:2572: finishing write tid 1 to smithi08575397-7 2021-02-10T04:38:02.249 INFO:tasks.rados.rados.0.smithi085.stdout:2572: finishing write tid 2 to smithi08575397-7 2021-02-10T04:38:02.249 INFO:tasks.rados.rados.0.smithi085.stdout:2571: done (6 left) 2021-02-10T04:38:02.249 INFO:tasks.rados.rados.0.smithi085.stdout:2574: delete oid 4 current snap is 291 2021-02-10T04:38:02.254 INFO:tasks.rados.rados.0.smithi085.stdout:2572: finishing write tid 3 to smithi08575397-7 2021-02-10T04:38:02.254 INFO:tasks.rados.rados.0.smithi085.stdout:2574: done (6 left) 2021-02-10T04:38:02.255 INFO:tasks.rados.rados.0.smithi085.stdout:2575: read oid 48 snap -1 2021-02-10T04:38:02.255 INFO:tasks.rados.rados.0.smithi085.stdout:2575: expect deleted 2021-02-10T04:38:02.255 INFO:tasks.rados.rados.0.smithi085.stdout:2576: snap_remove snap 252 2021-02-10T04:38:02.543 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:38:02.579 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:38:02.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:02 smithi164 conmon[29288]: cluster 2021-02-10T04:38:01.195482+0000 osd.4 ( 2021-02-10T04:38:02.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:02 smithi164 conmon[29288]: osd.4) 1039 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:02.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:02 smithi164 conmon[29288]: cluster 2021-02-10T04:38:01.195490+0000 osd.4 (osd.4) 1040 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:02.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:02 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:38:02.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:02 smithi164 conmon[29288]: :38:01.195502+0000 osd.4 (osd.4) 1041 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:02.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:02 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:38:02.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:02 smithi164 conmon[29288]: 02-10T04:38:01.195509+0000 osd.4 (osd.4) 1042 : cluster [WRN] 2021-02-10T04:38:02.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:02 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:02.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:02 smithi164 conmon[29288]: cluster 2021-02-10T04:38:02.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:02 smithi164 conmon[29288]: 2021-02-10T04:38:02.149452+0000 mgr.y (mgr.14140) 694 2021-02-10T04:38:02.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:02 smithi164 conmon[29288]: : cluster [DBG] pgmap v1145: 168 pgs: 156 active+clean, 7 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 402 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 13 op/s 2021-02-10T04:38:02.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:02 smithi164 conmon[29288]: cluster 2021-02-10T04:38:02.221124 2021-02-10T04:38:02.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:02 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1544 : cluster [DBG] 2021-02-10T04:38:02.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:02 smithi164 conmon[29288]: osdmap e664: 8 total, 8 up, 7 in 2021-02-10T04:38:02.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[29319]: cluster 2021-02-10T04:38:01.195482+0000 osd.4 (osd.4) 1039 2021-02-10T04:38:02.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:02.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[29319]: cluster 2021-02-10T04:38:01.195490+0000 osd.4 (osd.4) 1040 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:02.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[29319]: cluster 2021-02-10T04:38:01.195502+0000 osd.4 (osd.4 2021-02-10T04:38:02.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[29319]: ) 1041 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:02.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[29319]: cluster 2021-02-10T04:38:01.195509+0000 osd.4 (osd 2021-02-10T04:38:02.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[29319]: .4) 1042 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:02.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:38:02.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[29319]: -02-10T04:38:02.149452+0000 mgr.y (mgr.14140) 694 : cluster 2021-02-10T04:38:02.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[29319]: [DBG] pgmap v1145: 168 pgs: 156 active+clean, 7 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 402 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 13 op/s 2021-02-10T04:38:02.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[29319]: cluster 2021-02-10T04:38:02.221124 2021-02-10T04:38:02.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1544 : cluster [DBG] osdmap e664: 8 total, 8 up, 7 in 2021-02-10T04:38:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: cluster 2021-02-10T04:38:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: 2021-02-10T04:38:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: 2021 2021-02-10T04:38:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: - 2021-02-10T04:38:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: 02 2021-02-10T04:38:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: - 2021-02-10T04:38:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: 10T04:38:01 2021-02-10T04:38:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: .195482+0000 osd.4 (osd.4) 1039 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: cluster 2021-02-10T04:38:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: 2021 2021-02-10T04:38:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: - 2021-02-10T04:38:02.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: 02 2021-02-10T04:38:02.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: - 2021-02-10T04:38:02.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: 10 2021-02-10T04:38:02.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: T 2021-02-10T04:38:02.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: 04 2021-02-10T04:38:02.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: : 2021-02-10T04:38:02.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: 38 2021-02-10T04:38:02.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: : 2021-02-10T04:38:02.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: 01 2021-02-10T04:38:02.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: . 2021-02-10T04:38:02.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: 195490 2021-02-10T04:38:02.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: +0000 2021-02-10T04:38:02.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: 2021-02-10T04:38:02.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: osd.4 (osd.4) 1040 : cluster 2021-02-10T04:38:02.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:02.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:38:02.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: 04:38:01.195502+0000 osd.4 (osd.4) 1041 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:02.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: cluster 2021-02-10T04:38:01.195509 2021-02-10T04:38:02.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1042 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:02.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: cluster 2021-02-10T04:38:02.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: 2021-02-10T04:38:02.149452+0000 mgr.y 2021-02-10T04:38:02.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: (mgr.14140) 694 : cluster [DBG] pgmap v1145: 168 pgs: 156 active+clean, 7 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 402 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 13 op/s 2021-02-10T04:38:02.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:38:02.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: 38:02.221124+0000 mon.a (mon.0) 1544 : cluster [DBG] 2021-02-10T04:38:02.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[35961]: osdmap e664: 8 total, 8 up, 7 in 2021-02-10T04:38:02.749 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.0 dump_blocked_ops 2021-02-10T04:38:02.875 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:38:02.981 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:38:03.015 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:38:02 smithi085 conmon[49106]: debug 2021-02-10T04:38:02.742+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:38:03.036 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:38:03.142 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:38:03.215 INFO:tasks.rados.rados.0.smithi085.stdout:2576: done (7 left) 2021-02-10T04:38:03.216 INFO:tasks.rados.rados.0.smithi085.stdout:2577: delete oid 26 current snap is 291 2021-02-10T04:38:03.217 INFO:tasks.rados.rados.0.smithi085.stdout:2572: finishing write tid 4 to smithi08575397-7 2021-02-10T04:38:03.217 INFO:tasks.rados.rados.0.smithi085.stdout:2573: finishing rollback tid 0 to smithi08575397-15 2021-02-10T04:38:03.218 INFO:tasks.rados.rados.0.smithi085.stdout:2572: finishing write tid 5 to smithi08575397-7 2021-02-10T04:38:03.218 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 1173 (ObjNum 894 snap 291 seq_num 894) dirty exists 2021-02-10T04:38:03.218 INFO:tasks.rados.rados.0.smithi085.stdout:2572: left oid 7 (ObjNum 894 snap 291 seq_num 894) 2021-02-10T04:38:03.218 INFO:tasks.rados.rados.0.smithi085.stdout:2573: finishing rollback tid 1 to smithi08575397-15 2021-02-10T04:38:03.218 INFO:tasks.rados.rados.0.smithi085.stdout:2573: finishing rollback tid 2 to smithi08575397-15 2021-02-10T04:38:03.218 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 15 v 636 (ObjNum 217 snap 49 seq_num 217) dirty exists 2021-02-10T04:38:03.245 INFO:tasks.rados.rados.0.smithi085.stdout:2572: done (7 left) 2021-02-10T04:38:03.245 INFO:tasks.rados.rados.0.smithi085.stdout:2573: done (6 left) 2021-02-10T04:38:03.245 INFO:tasks.rados.rados.0.smithi085.stdout:2575: done (5 left) 2021-02-10T04:38:03.245 INFO:tasks.rados.rados.0.smithi085.stdout:2577: done (4 left) 2021-02-10T04:38:03.246 INFO:tasks.rados.rados.0.smithi085.stdout:2578: snap_create 2021-02-10T04:38:03.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:03 smithi164 conmon[29288]: cluster 2021-02-10T04:38:02.238887+0000 osd.4 (osd.4 2021-02-10T04:38:03.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:03 smithi164 conmon[29288]: ) 1043 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:03.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:03 smithi164 conmon[29288]: cluster 2021-02-10T04:38:02.238893+0000 osd.4 (osd.4) 1044 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:03.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:03 smithi164 conmon[29288]: cluster 2021-02-10T04:38:02 2021-02-10T04:38:03.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:03 smithi164 conmon[29288]: .238912+0000 osd.4 (osd.4) 1045 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:03.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:03 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:38:03.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:03 smithi164 conmon[29288]: 04:38:02.238922+0000 osd.4 (osd.4) 1046 : 2021-02-10T04:38:03.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:03 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:03.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:03 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:38:03.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:03 smithi164 conmon[29288]: 38:03.207956+0000 mon.a (mon.0) 1545 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 512 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:38:03.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:03 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:38:03.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:03 smithi164 conmon[29288]: 10T04:38:03.213677+0000 mon.a (mon.0) 1546 : 2021-02-10T04:38:03.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:03 smithi164 conmon[29288]: cluster [DBG] osdmap e665: 8 total, 8 up, 7 in 2021-02-10T04:38:03.646 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:03 smithi164 conmon[33677]: debug 2021-02-10T04:38:03.260+0000 7fbec19a1700 -1 osd.4 665 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:38:03.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[29319]: cluster 2021-02-10T04:38:03.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[29319]: 2021-02-10T04:38:03.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[29319]: 2021 2021-02-10T04:38:03.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[29319]: - 2021-02-10T04:38:03.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[29319]: 02 2021-02-10T04:38:03.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[29319]: -10T04:38:02.238887+0000 2021-02-10T04:38:03.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[29319]: osd.4 (osd.4) 1043 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:03.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[29319]: cluster 2021-02-10T04:38:02.238893+0000 osd.4 ( 2021-02-10T04:38:03.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[29319]: osd.4) 1044 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:03.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:38:03.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[29319]: 04:38:02.238912+0000 osd.4 (osd 2021-02-10T04:38:03.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[29319]: .4) 1045 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:03.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[29319]: 2021-02-10T04:38:03.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:38:03.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[29319]: T04:38:02.238922+0000 2021-02-10T04:38:03.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[29319]: osd.4 (osd.4) 1046 : cluster 2021-02-10T04:38:03.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:03.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[29319]: cluster 2021-02-10T04:38:03.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[29319]: 2021-02-10T04:38:03.207956+0000 mon.a (mon.0) 1545 2021-02-10T04:38:03.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[29319]: : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 512 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:38:03.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:38:03.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[29319]: :38:03.213677+0000 mon.a (mon.0 2021-02-10T04:38:03.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[29319]: ) 1546 : cluster [DBG] osdmap e665: 8 total, 8 up, 7 in 2021-02-10T04:38:03.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[35961]: cluster 2021-02-10T04:38:02.238887 2021-02-10T04:38:03.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1043 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:03.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:38:03.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[35961]: 10T04:38:02 2021-02-10T04:38:03.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[35961]: .238893+0000 osd.4 (osd.4) 2021-02-10T04:38:03.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[35961]: 1044 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:03.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:38:03.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[35961]: 02-10T04:38:02.238912+0000 osd.4 (osd. 2021-02-10T04:38:03.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[35961]: 4) 1045 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:03.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[35961]: cluster 2021-02-10T04:38:02.238922+0000 osd.4 (osd 2021-02-10T04:38:03.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[35961]: .4) 1046 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:03.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:38:03.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[35961]: 38:03.207956+0000 mon.a ( 2021-02-10T04:38:03.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[35961]: mon.0) 1545 : cluster 2021-02-10T04:38:03.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[35961]: [WRN] Health check update: 4 slow ops, oldest one blocked for 512 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:38:03.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:38:03.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[35961]: -02-10T04:38 2021-02-10T04:38:03.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[35961]: :03.213677+0000 mon.a (mon 2021-02-10T04:38:03.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[35961]: .0) 1546 : cluster [DBG] osdmap e665: 8 total, 8 up, 7 in 2021-02-10T04:38:03.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:03 smithi085 conmon[35961]: 2021-02-10T04:38:04.227 INFO:tasks.rados.rados.0.smithi085.stdout:2578: done (4 left) 2021-02-10T04:38:08.054 INFO:tasks.rados.rados.0.smithi085.stdout:2579: snap_remove snap 285 2021-02-10T04:38:08.058 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:04 smithi164 conmon[29288]: cluster 2021-02-10T04:38:03.261127 2021-02-10T04:38:08.058 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:04 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1047 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:08.059 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:04 smithi164 conmon[29288]: cluster 2021-02-10T04:38:03.261132+0000 2021-02-10T04:38:08.059 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:04 smithi164 conmon[29288]: osd.4 (osd.4) 1048 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:08.059 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:04 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:38:08.059 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:04 smithi164 conmon[29288]: -02-10T04:38:03.261136+0000 osd.4 (osd.4) 1049 2021-02-10T04:38:08.059 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:04 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:08.059 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:04 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:38:08.059 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:04 smithi164 conmon[29288]: -02-10T04:38:03.261139+0000 osd.4 ( 2021-02-10T04:38:08.059 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:04 smithi164 conmon[29288]: osd.4) 1050 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:08.059 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:04 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:38:08.059 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:04 smithi164 conmon[29288]: :38:04.150023+0000 mgr.y (mgr.14140 2021-02-10T04:38:08.060 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:04 smithi164 conmon[29288]: ) 695 : cluster [DBG] pgmap v1148: 168 pgs: 155 active+clean, 8 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 397 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.0 MiB/s rd, 1.5 MiB/s wr, 11 op/s 2021-02-10T04:38:08.060 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:04 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:38:08.060 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:04 smithi164 conmon[29288]: :38:04.225725+0000 mon.a (mon.0) 2021-02-10T04:38:08.060 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:04 smithi164 conmon[29288]: 1547 : cluster [DBG] osdmap e666: 8 total, 8 up, 7 in 2021-02-10T04:38:08.060 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:05 smithi164 conmon[29288]: cluster 2021-02-10T04:38:04.304894+0000 osd.4 (osd.4) 1051 : 2021-02-10T04:38:08.060 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:05 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:08.060 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:05 smithi164 conmon[29288]: cluster 2021-02-10T04:38:04.304903+0000 osd.4 (osd.4) 1052 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:08.060 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:05 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:38:08.060 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:05 smithi164 conmon[29288]: 10T04:38:04.304913+0000 osd.4 (osd.4) 1053 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:08.060 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:05 smithi164 conmon[29288]: cluster 2021-02-10T04:38: 2021-02-10T04:38:08.060 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:05 smithi164 conmon[29288]: 04.304917+0000 osd.4 (osd.4) 1054 : 2021-02-10T04:38:08.060 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:05 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:08.061 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:05 smithi164 conmon[29288]: audit 2021-02-10T04:38: 2021-02-10T04:38:08.061 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:05 smithi164 conmon[29288]: 04.543684+0000 mon.a (mon.0) 1548 : 2021-02-10T04:38:08.061 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:05 smithi164 conmon[29288]: audit [DBG] from='client.? 172.21.15.85:0/2237760813' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-02-10T04:38:08.061 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:05 smithi164 conmon[29288]: cluster 2021-02-10T04:38 2021-02-10T04:38:08.061 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:05 smithi164 conmon[29288]: :04.785837+0000 osd.5 (osd.5) 2021-02-10T04:38:08.061 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:05 smithi164 conmon[29288]: 71 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:38:08.061 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:05 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:38:08.061 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:05 smithi164 conmon[29288]: 02-10T04:38:04.786840+0000 osd.5 (osd.5) 72 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:38:08.061 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:05 smithi164 conmon[29288]: cluster 2021-02-10T04:38:05.227648 2021-02-10T04:38:08.061 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:05 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1549 : cluster [DBG] osdmap e667: 8 total, 8 up, 7 in 2021-02-10T04:38:08.061 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:06 smithi164 conmon[29288]: cluster 2021-02-10T04:38:08.062 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:06 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:38:08.062 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:06 smithi164 conmon[29288]: 04:38:05.268631+0000 osd.4 (osd.4) 1055 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:08.062 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:06 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:38:08.062 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:06 smithi164 conmon[29288]: 02-10T04:38:05.268641+0000 osd.4 (osd.4 2021-02-10T04:38:08.062 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:06 smithi164 conmon[29288]: ) 1056 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:08.063 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:06 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:38:08.063 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:06 smithi164 conmon[29288]: 10T04:38:05.268647+0000 osd.4 (osd. 2021-02-10T04:38:08.063 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:06 smithi164 conmon[29288]: 4) 1057 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:08.063 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:06 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:38:08.063 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:06 smithi164 conmon[29288]: T04:38:05.268653+0000 osd.4 (osd.4 2021-02-10T04:38:08.063 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:06 smithi164 conmon[29288]: ) 1058 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:08.063 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:06 smithi164 conmon[29288]: cluster 2021-02-10T04:38:08.063 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:06 smithi164 conmon[29288]: 2021-02-10T04:38 2021-02-10T04:38:08.063 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:06 smithi164 conmon[29288]: :06.150625+0000 mgr.y (mgr.14140) 696 : cluster [DBG] pgmap v1151: 168 pgs: 155 active+clean, 8 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 392 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 9.1 MiB/s rd, 1.5 MiB/s wr, 15 op/s 2021-02-10T04:38:08.064 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:06 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:38:08.064 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:06 smithi164 conmon[29288]: -02-10T04:38:06.228261+0000 mon.a (mon.0) 1550 : cluster [DBG] osdmap e668: 8 total, 8 up, 7 in 2021-02-10T04:38:08.064 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:07 smithi164 conmon[29288]: cluster 2021-02-10T04:38:06.292180+0000 osd.4 2021-02-10T04:38:08.064 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:07 smithi164 conmon[29288]: (osd.4) 1059 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:08.064 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:07 smithi164 conmon[29288]: cluster 2021-02-10T04:38:06.292194+0000 osd.4 (osd.4) 1060 : 2021-02-10T04:38:08.064 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:07 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:08.064 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:07 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:38:08.064 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:07 smithi164 conmon[29288]: T04:38:06.292201+0000 osd.4 (osd.4) 1061 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:08.064 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:07 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:38:08.064 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:07 smithi164 conmon[29288]: 38:06.292205+0000 osd.4 (osd.4) 1062 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:08.064 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:07 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:38:08.065 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:07 smithi164 conmon[29288]: -02-10T04:38:07 2021-02-10T04:38:08.065 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:07 smithi164 conmon[29288]: .228375+0000 mon.a (mon.0) 1551 : 2021-02-10T04:38:08.065 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:07 smithi164 conmon[29288]: cluster [DBG] osdmap e669: 8 total, 8 up, 7 in 2021-02-10T04:38:08.065 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:04 smithi164 conmon[33677]: debug 2021-02-10T04:38:04.304+0000 7fbec19a1700 -1 osd.4 666 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:38:08.065 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:05 smithi164 conmon[33677]: debug 2021-02-10T04:38:05.267+0000 7fbec19a1700 -1 osd.4 667 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:38:08.065 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:06 smithi164 conmon[33677]: debug 2021-02-10T04:38:06.291+0000 7fbec19a1700 -1 osd.4 668 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:38:08.065 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:07 smithi164 conmon[33677]: debug 2021-02-10T04:38:07.248+0000 7fbec19a1700 -1 osd.4 669 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:38:08.065 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[35961]: cluster 2021-02-10T04:38:08.065 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[35961]: 2021-02-10T04:38:08.065 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[35961]: 2021 2021-02-10T04:38:08.066 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[35961]: - 2021-02-10T04:38:08.066 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[35961]: 02 2021-02-10T04:38:08.066 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[35961]: - 2021-02-10T04:38:08.066 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[35961]: 10 2021-02-10T04:38:08.066 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[35961]: T 2021-02-10T04:38:08.066 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[35961]: 04 2021-02-10T04:38:08.066 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[35961]: : 2021-02-10T04:38:08.066 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[35961]: 38 2021-02-10T04:38:08.066 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[35961]: :03.261127+0000 osd.4 (osd.4 2021-02-10T04:38:08.066 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[35961]: ) 1047 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:08.066 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[35961]: cluster 2021-02-10T04:38 2021-02-10T04:38:08.067 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[35961]: :03.261132+0000 osd.4 (osd.4) 1048 : 2021-02-10T04:38:08.067 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:08.067 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:38:08.067 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[35961]: :38:03.261136+0000 osd.4 (osd.4) 1049 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:08.067 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:38:08.067 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[35961]: -02-10T04:38:03.261139+0000 2021-02-10T04:38:08.067 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[35961]: osd.4 (osd.4) 1050 : 2021-02-10T04:38:08.067 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:08.067 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:38:08.067 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[35961]: T04:38:04.150023+0000 mgr.y ( 2021-02-10T04:38:08.067 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[35961]: mgr.14140) 695 : cluster [DBG] pgmap v1148: 168 pgs: 155 active+clean, 8 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 397 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.0 MiB/s rd, 1.5 MiB/s wr, 11 op/s 2021-02-10T04:38:08.068 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:38:08.068 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[35961]: 02-10T04:38:04.225725+0000 2021-02-10T04:38:08.068 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[35961]: mon.a (mon.0) 1547 : cluster [DBG] 2021-02-10T04:38:08.068 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[35961]: osdmap e666: 8 total, 8 up, 7 in 2021-02-10T04:38:08.068 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[35961]: cluster 2021-02-10T04:38:04.304894+0000 osd.4 ( 2021-02-10T04:38:08.068 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[35961]: osd.4) 1051 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:08.068 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[35961]: cluster 2021-02-10T04:38:04.304903+0000 osd.4 (osd.4) 1052 : cluster 2021-02-10T04:38:08.068 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:08.068 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:38:08.068 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[35961]: 38:04.304913+0000 osd.4 (osd.4) 1053 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:08.068 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:38:08.069 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[35961]: 02-10T04:38:04.304917+0000 osd.4 2021-02-10T04:38:08.069 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[35961]: (osd.4) 1054 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:08.069 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[35961]: audit 2021 2021-02-10T04:38:08.069 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[35961]: -02-10T04:38:04.543684+0000 2021-02-10T04:38:08.069 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[35961]: mon.a (mon.0) 1548 : audit [DBG] from='client.? 172.21.15.85:0/2237760813' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-02-10T04:38:08.069 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:38:08.069 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[35961]: 10T04:38:04.785837+0000 osd.5 ( 2021-02-10T04:38:08.069 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[35961]: osd.5) 71 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:38:08.069 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:38:08.069 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[35961]: 10T04:38:04.786840+0000 osd.5 (osd 2021-02-10T04:38:08.069 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[35961]: .5) 72 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:38:08.069 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:38:08.070 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[35961]: -02-10T04:38:05.227648+0000 2021-02-10T04:38:08.070 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[35961]: mon.a (mon.0) 1549 : cluster [DBG] 2021-02-10T04:38:08.070 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[35961]: osdmap e667: 8 total, 8 up, 7 in 2021-02-10T04:38:08.070 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[35961]: cluster 2021-02-10T04:38:08.070 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[35961]: 2021-02-10T04:38:08.070 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[35961]: 2021-02-10T04:38:05 2021-02-10T04:38:08.070 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[35961]: .268631+0000 osd.4 (osd.4) 1055 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:08.070 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[35961]: cluster 2021-02-10T04:38:08.070 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[35961]: 2021-02-10T04:38:05.268641+0000 osd.4 (osd.4) 1056 2021-02-10T04:38:08.071 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:08.071 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:38:08.071 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[35961]: -10T04:38:05. 2021-02-10T04:38:08.071 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[35961]: 268647+0000 osd.4 (osd.4) 2021-02-10T04:38:08.071 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[35961]: 1057 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:08.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:38:08.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[35961]: :38:05.268653+0000 osd.4 (osd.4) 1058 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:08.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[35961]: cluster 2021-02-10T04:38:06. 2021-02-10T04:38:08.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[35961]: 150625+0000 mgr.y (mgr.14140) 696 : cluster [DBG] pgmap v1151: 168 pgs: 155 active+clean, 8 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 392 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 9.1 MiB/s rd, 1.5 MiB/s wr, 15 op/s 2021-02-10T04:38:08.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[35961]: cluster 2021-02-10T04:38:06.228261+0000 mon.a (mon. 2021-02-10T04:38:08.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[35961]: 0) 1550 : cluster [DBG] osdmap e668: 8 total, 8 up, 7 in 2021-02-10T04:38:08.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[35961]: cluster 2021-02-10T04:38:06.292180+0000 2021-02-10T04:38:08.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[35961]: osd.4 (osd.4) 1059 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:08.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[35961]: cluster 2021-02-10T04:38:08.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[35961]: 2021-02-10T04:38 2021-02-10T04:38:08.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[35961]: :06.292194+0000 osd.4 (osd.4) 1060 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:08.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:38:08.073 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[35961]: 04:38:06.292201+0000 osd.4 (osd.4) 1061 : 2021-02-10T04:38:08.073 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:08.073 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[35961]: cluster 2021-02-10T04:38:06.292205+0000 osd.4 2021-02-10T04:38:08.073 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[35961]: (osd.4) 1062 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:08.073 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[35961]: cluster 2021-02-10T04:38:07 2021-02-10T04:38:08.073 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[35961]: .228375+0000 mon.a (mon.0) 1551 : cluster [DBG] 2021-02-10T04:38:08.073 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[35961]: osdmap e669: 8 total, 8 up, 7 in 2021-02-10T04:38:08.073 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[41540]: debug 2021-02-10T04:38:04.200+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:38:08.073 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:38:08.073 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:38:08.074 INFO:teuthology.orchestra.run.smithi085.stdout: "epoch": 666, 2021-02-10T04:38:08.074 INFO:teuthology.orchestra.run.smithi085.stdout: "fsid": "42224fce-6b57-11eb-8feb-001a4aab830c", 2021-02-10T04:38:08.074 INFO:teuthology.orchestra.run.smithi085.stdout: "created": "2021-02-10T04:20:57.933279+0000", 2021-02-10T04:38:08.074 INFO:teuthology.orchestra.run.smithi085.stdout: "modified": "2021-02-10T04:38:04.218561+0000", 2021-02-10T04:38:08.074 INFO:teuthology.orchestra.run.smithi085.stdout: "last_up_change": "2021-02-10T04:25:35.312604+0000", 2021-02-10T04:38:08.074 INFO:teuthology.orchestra.run.smithi085.stdout: "last_in_change": "2021-02-10T04:37:36.200492+0000", 2021-02-10T04:38:08.074 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T04:38:08.074 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_num": 5806080, 2021-02-10T04:38:08.074 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_set": [ 2021-02-10T04:38:08.074 INFO:teuthology.orchestra.run.smithi085.stdout: "nodeep-scrub", 2021-02-10T04:38:08.074 INFO:teuthology.orchestra.run.smithi085.stdout: "noscrub", 2021-02-10T04:38:08.074 INFO:teuthology.orchestra.run.smithi085.stdout: "pglog_hardlimit", 2021-02-10T04:38:08.075 INFO:teuthology.orchestra.run.smithi085.stdout: "purged_snapdirs", 2021-02-10T04:38:08.075 INFO:teuthology.orchestra.run.smithi085.stdout: "recovery_deletes", 2021-02-10T04:38:08.075 INFO:teuthology.orchestra.run.smithi085.stdout: "sortbitwise" 2021-02-10T04:38:08.075 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:38:08.075 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_version": 18, 2021-02-10T04:38:08.075 INFO:teuthology.orchestra.run.smithi085.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T04:38:08.075 INFO:teuthology.orchestra.run.smithi085.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T04:38:08.075 INFO:teuthology.orchestra.run.smithi085.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T04:38:08.075 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_snapshot": "", 2021-02-10T04:38:08.075 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_max": 7, 2021-02-10T04:38:08.075 INFO:teuthology.orchestra.run.smithi085.stdout: "max_osd": 8, 2021-02-10T04:38:08.075 INFO:teuthology.orchestra.run.smithi085.stdout: "require_min_compat_client": "luminous", 2021-02-10T04:38:08.075 INFO:teuthology.orchestra.run.smithi085.stdout: "min_compat_client": "luminous", 2021-02-10T04:38:08.076 INFO:teuthology.orchestra.run.smithi085.stdout: "require_osd_release": "quincy", 2021-02-10T04:38:08.076 INFO:teuthology.orchestra.run.smithi085.stdout: "pools": [ 2021-02-10T04:38:08.076 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.076 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 1, 2021-02-10T04:38:08.076 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "device_health_metrics", 2021-02-10T04:38:08.076 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:24:05.258028+0000", 2021-02-10T04:38:08.076 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:38:08.076 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:38:08.076 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:38:08.076 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:38:08.077 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:38:08.077 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:38:08.077 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:38:08.077 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:38:08.077 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:38:08.077 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:38:08.077 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:38:08.077 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:38:08.077 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 11, 2021-02-10T04:38:08.077 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 11, 2021-02-10T04:38:08.078 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 11, 2021-02-10T04:38:08.078 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 11, 2021-02-10T04:38:08.078 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 11, 2021-02-10T04:38:08.078 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:38:08.078 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:38:08.078 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:38:08.078 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:38:08.078 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:38:08.078 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:38:08.078 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:38:08.078 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.079 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "112", 2021-02-10T04:38:08.079 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:38:08.079 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:38:08.079 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "110", 2021-02-10T04:38:08.079 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:38:08.079 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:38:08.079 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:38:08.079 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:38:08.079 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:38:08.079 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:38:08.079 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:38:08.080 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:38:08.080 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:38:08.080 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:38:08.080 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:38:08.080 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:38:08.081 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:38:08.081 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:38:08.081 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:38:08.081 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:38:08.081 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:38:08.081 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:38:08.082 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:38:08.082 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:38:08.082 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:38:08.082 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:38:08.082 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:38:08.082 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.082 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:38:08.082 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:38:08.082 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:38:08.082 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:38:08.083 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:38:08.083 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:38:08.083 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:38:08.083 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:38:08.083 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:38:08.083 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:38:08.083 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:38:08.083 INFO:teuthology.orchestra.run.smithi085.stdout: "options": { 2021-02-10T04:38:08.084 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_min": 1 2021-02-10T04:38:08.084 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.084 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:38:08.084 INFO:teuthology.orchestra.run.smithi085.stdout: "mgr_devicehealth": {} 2021-02-10T04:38:08.084 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.084 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.084 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.084 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 2, 2021-02-10T04:38:08.084 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": ".rgw.root", 2021-02-10T04:38:08.084 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:32.688554+0000", 2021-02-10T04:38:08.085 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:38:08.085 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:38:08.085 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:38:08.085 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:38:08.085 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:38:08.085 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:38:08.085 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:38:08.085 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:38:08.085 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:38:08.085 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:38:08.085 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:38:08.085 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:38:08.085 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 42, 2021-02-10T04:38:08.086 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 42, 2021-02-10T04:38:08.086 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 42, 2021-02-10T04:38:08.086 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 42, 2021-02-10T04:38:08.086 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 42, 2021-02-10T04:38:08.086 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:38:08.086 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:38:08.086 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:38:08.086 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:38:08.086 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:38:08.086 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:38:08.086 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:38:08.086 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.086 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "91", 2021-02-10T04:38:08.087 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:38:08.087 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:38:08.087 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "89", 2021-02-10T04:38:08.087 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:38:08.087 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:38:08.087 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:38:08.087 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:38:08.087 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:38:08.087 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:38:08.087 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:38:08.087 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:38:08.087 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:38:08.087 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:38:08.088 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:38:08.088 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:38:08.088 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:38:08.088 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:38:08.088 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:38:08.088 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:38:08.088 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:38:08.088 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:38:08.088 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:38:08.088 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:38:08.089 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:38:08.089 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:38:08.089 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:38:08.089 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.089 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:38:08.089 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:38:08.089 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:38:08.089 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:38:08.089 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:38:08.089 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:38:08.089 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:38:08.090 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:38:08.090 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:38:08.090 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:38:08.090 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:38:08.090 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:38:08.090 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:38:08.090 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T04:38:08.090 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.090 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.090 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.090 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 3, 2021-02-10T04:38:08.090 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "iscsi", 2021-02-10T04:38:08.091 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:47.658613+0000", 2021-02-10T04:38:08.091 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:38:08.091 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:38:08.091 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:38:08.091 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:38:08.091 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:38:08.091 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:38:08.091 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:38:08.091 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:38:08.091 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:38:08.091 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:38:08.091 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:38:08.091 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:38:08.092 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 3, 2021-02-10T04:38:08.092 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 3, 2021-02-10T04:38:08.092 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 3, 2021-02-10T04:38:08.092 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 3, 2021-02-10T04:38:08.092 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 3, 2021-02-10T04:38:08.092 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:38:08.092 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:38:08.092 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:38:08.092 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:38:08.092 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:38:08.092 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:38:08.092 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:38:08.093 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.093 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "58", 2021-02-10T04:38:08.093 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:38:08.093 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:38:08.093 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T04:38:08.093 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:38:08.093 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:38:08.093 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:38:08.093 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:38:08.093 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:38:08.093 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:38:08.093 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:38:08.093 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:38:08.094 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:38:08.094 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:38:08.094 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:38:08.094 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:38:08.094 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:38:08.094 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:38:08.094 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:38:08.094 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:38:08.094 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:38:08.094 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:38:08.094 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:38:08.094 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:38:08.095 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:38:08.095 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:38:08.095 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:38:08.095 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.095 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:38:08.095 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:38:08.095 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:38:08.095 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:38:08.095 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:38:08.095 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:38:08.096 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:38:08.096 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:38:08.096 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:38:08.096 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:38:08.096 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:38:08.096 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:38:08.096 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:38:08.096 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd": {} 2021-02-10T04:38:08.096 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.096 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.096 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.096 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 4, 2021-02-10T04:38:08.097 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "zone.rgw.log", 2021-02-10T04:38:08.097 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:47.968727+0000", 2021-02-10T04:38:08.097 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:38:08.097 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:38:08.097 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:38:08.097 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:38:08.097 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:38:08.097 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:38:08.097 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:38:08.097 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:38:08.097 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:38:08.097 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:38:08.098 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:38:08.098 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:38:08.098 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 32, 2021-02-10T04:38:08.098 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 32, 2021-02-10T04:38:08.098 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 32, 2021-02-10T04:38:08.098 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 32, 2021-02-10T04:38:08.098 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 32, 2021-02-10T04:38:08.098 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:38:08.098 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:38:08.098 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:38:08.098 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:38:08.098 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:38:08.098 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:38:08.099 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:38:08.099 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.099 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "57", 2021-02-10T04:38:08.099 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:38:08.099 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:38:08.099 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T04:38:08.099 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:38:08.099 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:38:08.099 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:38:08.099 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:38:08.099 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:38:08.099 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:38:08.100 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:38:08.100 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:38:08.100 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:38:08.100 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:38:08.100 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:38:08.100 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:38:08.100 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:38:08.100 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:38:08.100 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:38:08.100 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:38:08.100 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:38:08.101 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:38:08.101 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:38:08.101 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:38:08.101 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:38:08.101 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:38:08.101 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:38:08.101 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.101 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:38:08.101 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:38:08.101 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:38:08.101 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:38:08.101 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:38:08.102 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:38:08.102 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:38:08.102 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:38:08.102 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:38:08.102 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:38:08.102 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:38:08.102 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:38:08.102 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:38:08.102 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T04:38:08.102 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.102 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.102 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.103 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 5, 2021-02-10T04:38:08.103 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "zone.rgw.control", 2021-02-10T04:38:08.103 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:49.278260+0000", 2021-02-10T04:38:08.103 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:38:08.103 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:38:08.103 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:38:08.103 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:38:08.103 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:38:08.103 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:38:08.103 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:38:08.103 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:38:08.103 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:38:08.103 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:38:08.104 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:38:08.104 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:38:08.104 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 32, 2021-02-10T04:38:08.104 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 32, 2021-02-10T04:38:08.104 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 32, 2021-02-10T04:38:08.104 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 32, 2021-02-10T04:38:08.104 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 32, 2021-02-10T04:38:08.104 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:38:08.104 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:38:08.104 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:38:08.104 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:38:08.104 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:38:08.104 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:38:08.104 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:38:08.105 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.105 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "59", 2021-02-10T04:38:08.105 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:38:08.105 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:38:08.105 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T04:38:08.105 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:38:08.105 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:38:08.105 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:38:08.105 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:38:08.105 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:38:08.105 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:38:08.105 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:38:08.105 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:38:08.106 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:38:08.106 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:38:08.106 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:38:08.106 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:38:08.106 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:38:08.106 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:38:08.106 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:38:08.106 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:38:08.106 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:38:08.106 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:38:08.107 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:38:08.107 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:38:08.107 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:38:08.107 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:38:08.107 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:38:08.107 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.107 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:38:08.107 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:38:08.107 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:38:08.108 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:38:08.108 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:38:08.108 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:38:08.108 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:38:08.108 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:38:08.108 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:38:08.108 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:38:08.108 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:38:08.108 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:38:08.108 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:38:08.108 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T04:38:08.108 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.109 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.109 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.109 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 6, 2021-02-10T04:38:08.109 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "zone.rgw.meta", 2021-02-10T04:38:08.109 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:51.792514+0000", 2021-02-10T04:38:08.109 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:38:08.109 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:38:08.109 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:38:08.109 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:38:08.109 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:38:08.109 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:38:08.109 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:38:08.110 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:38:08.110 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:38:08.110 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:38:08.110 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:38:08.110 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:38:08.110 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 32, 2021-02-10T04:38:08.110 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 32, 2021-02-10T04:38:08.110 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 32, 2021-02-10T04:38:08.110 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 32, 2021-02-10T04:38:08.110 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 32, 2021-02-10T04:38:08.110 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:38:08.110 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:38:08.111 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:38:08.111 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:38:08.111 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:38:08.111 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:38:08.111 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:38:08.111 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.111 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "62", 2021-02-10T04:38:08.111 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:38:08.111 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:38:08.111 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T04:38:08.111 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:38:08.111 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:38:08.112 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:38:08.112 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:38:08.112 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:38:08.112 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:38:08.112 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:38:08.112 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:38:08.112 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:38:08.112 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:38:08.112 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:38:08.112 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:38:08.112 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:38:08.112 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:38:08.113 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:38:08.113 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:38:08.113 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:38:08.113 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:38:08.113 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:38:08.113 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:38:08.113 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:38:08.113 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:38:08.113 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:38:08.113 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.113 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:38:08.113 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:38:08.114 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:38:08.114 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:38:08.114 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:38:08.114 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:38:08.114 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:38:08.114 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:38:08.114 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:38:08.114 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:38:08.114 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:38:08.114 INFO:teuthology.orchestra.run.smithi085.stdout: "options": { 2021-02-10T04:38:08.114 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_bias": 4, 2021-02-10T04:38:08.115 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_min": 8 2021-02-10T04:38:08.115 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.115 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:38:08.115 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T04:38:08.115 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.115 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.115 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.115 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 7, 2021-02-10T04:38:08.115 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "unique_pool_0", 2021-02-10T04:38:08.115 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:28:23.932391+0000", 2021-02-10T04:38:08.115 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 8193, 2021-02-10T04:38:08.116 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2021-02-10T04:38:08.116 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:38:08.116 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:38:08.116 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:38:08.116 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:38:08.116 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:38:08.116 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:38:08.116 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:38:08.116 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:38:08.117 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:38:08.117 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:38:08.117 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 16, 2021-02-10T04:38:08.117 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 16, 2021-02-10T04:38:08.117 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 16, 2021-02-10T04:38:08.117 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 16, 2021-02-10T04:38:08.117 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 16, 2021-02-10T04:38:08.117 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:38:08.117 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:38:08.117 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:38:08.117 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:38:08.117 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:38:08.117 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:38:08.118 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:38:08.118 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.118 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "666", 2021-02-10T04:38:08.118 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:38:08.118 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:38:08.118 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T04:38:08.118 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:38:08.118 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:38:08.118 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 561, 2021-02-10T04:38:08.118 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 666, 2021-02-10T04:38:08.119 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:38:08.119 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:38:08.119 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:38:08.119 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:38:08.119 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:38:08.119 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:38:08.119 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:38:08.119 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:38:08.119 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:38:08.119 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:38:08.119 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:38:08.119 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:38:08.120 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:38:08.120 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:38:08.120 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:38:08.120 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:38:08.120 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:38:08.120 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:38:08.120 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:38:08.120 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.120 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:38:08.120 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:38:08.120 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:38:08.121 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:38:08.121 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:38:08.121 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:38:08.121 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:38:08.121 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:38:08.121 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:38:08.121 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:38:08.121 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:38:08.121 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:38:08.121 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:38:08.121 INFO:teuthology.orchestra.run.smithi085.stdout: "rados": {} 2021-02-10T04:38:08.121 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.122 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.122 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:38:08.122 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T04:38:08.122 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.122 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 0, 2021-02-10T04:38:08.122 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "45d0bb5f-35fa-4596-964f-75abcee53bee", 2021-02-10T04:38:08.122 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:38:08.122 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:38:08.122 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:38:08.122 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:38:08.122 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:38:08.122 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:38:08.123 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 8, 2021-02-10T04:38:08.123 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 638, 2021-02-10T04:38:08.123 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:38:08.123 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:38:08.123 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:38:08.123 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.123 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.123 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.123 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6801", 2021-02-10T04:38:08.123 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.123 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.123 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.124 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.124 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:38:08.124 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.124 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.124 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.124 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6802", 2021-02-10T04:38:08.124 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.124 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.125 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.125 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.125 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:38:08.125 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.125 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.125 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.125 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6804", 2021-02-10T04:38:08.125 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.125 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.126 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.126 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.126 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:38:08.126 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.126 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.126 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.126 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6803", 2021-02-10T04:38:08.126 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.126 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.126 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.127 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.127 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6801/7", 2021-02-10T04:38:08.127 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6802/7", 2021-02-10T04:38:08.127 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6804/7", 2021-02-10T04:38:08.127 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6803/7", 2021-02-10T04:38:08.127 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:38:08.127 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:38:08.127 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:38:08.127 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.127 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.127 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.127 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 1, 2021-02-10T04:38:08.127 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "3421a5ce-872e-4b52-adf7-6aaa595871a4", 2021-02-10T04:38:08.128 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:38:08.128 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:38:08.128 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:38:08.128 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 0, 2021-02-10T04:38:08.128 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:38:08.128 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:38:08.128 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 13, 2021-02-10T04:38:08.128 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 130, 2021-02-10T04:38:08.128 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:38:08.128 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:38:08.128 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:38:08.128 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.129 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.129 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.129 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6805", 2021-02-10T04:38:08.129 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.129 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.129 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.129 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.129 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:38:08.129 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.129 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.129 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.129 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6806", 2021-02-10T04:38:08.129 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.130 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.130 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.130 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.130 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:38:08.130 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.130 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.130 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.130 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6808", 2021-02-10T04:38:08.130 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.130 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.130 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.131 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.131 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:38:08.131 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.131 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.131 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.131 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6807", 2021-02-10T04:38:08.131 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.131 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.131 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.131 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.131 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6805/7", 2021-02-10T04:38:08.131 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6806/7", 2021-02-10T04:38:08.131 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6808/7", 2021-02-10T04:38:08.132 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6807/7", 2021-02-10T04:38:08.132 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:38:08.132 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:38:08.132 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:38:08.132 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.132 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.132 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.132 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 2, 2021-02-10T04:38:08.132 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "96e1952a-1042-4316-9fc9-86813e88b912", 2021-02-10T04:38:08.132 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:38:08.132 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 0, 2021-02-10T04:38:08.132 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 0, 2021-02-10T04:38:08.133 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:38:08.133 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:38:08.133 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:38:08.133 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 18, 2021-02-10T04:38:08.133 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 367, 2021-02-10T04:38:08.133 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:38:08.133 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:38:08.133 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:38:08.133 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.133 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.133 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.133 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6809", 2021-02-10T04:38:08.133 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.133 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.134 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.134 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.134 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:38:08.134 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.134 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.134 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.134 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6810", 2021-02-10T04:38:08.134 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.134 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.134 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.134 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.134 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:38:08.135 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.135 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.135 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.135 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6812", 2021-02-10T04:38:08.135 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.135 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.135 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.135 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.135 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:38:08.135 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.135 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.135 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.135 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6811", 2021-02-10T04:38:08.136 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.136 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.136 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.136 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.136 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6809/7", 2021-02-10T04:38:08.136 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6810/7", 2021-02-10T04:38:08.136 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6812/7", 2021-02-10T04:38:08.136 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6811/7", 2021-02-10T04:38:08.136 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:38:08.136 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:38:08.136 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:38:08.136 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.137 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.137 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.137 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 3, 2021-02-10T04:38:08.137 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "e019f49f-5dd4-4104-9ced-94c710616642", 2021-02-10T04:38:08.137 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:38:08.137 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:38:08.137 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:38:08.137 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:38:08.137 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:38:08.137 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:38:08.137 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 25, 2021-02-10T04:38:08.137 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 649, 2021-02-10T04:38:08.138 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:38:08.138 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:38:08.138 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:38:08.138 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.138 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.138 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.138 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6813", 2021-02-10T04:38:08.138 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.138 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.138 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.138 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.138 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:38:08.139 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.139 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.139 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.139 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6814", 2021-02-10T04:38:08.139 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.139 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.139 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.139 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.139 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:38:08.139 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.139 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.139 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.139 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6816", 2021-02-10T04:38:08.140 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.140 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.140 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.140 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.140 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:38:08.140 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.140 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.140 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.140 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6815", 2021-02-10T04:38:08.140 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.140 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.140 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.140 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.140 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6813/7", 2021-02-10T04:38:08.141 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6814/7", 2021-02-10T04:38:08.141 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6816/7", 2021-02-10T04:38:08.141 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6815/7", 2021-02-10T04:38:08.141 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:38:08.141 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:38:08.141 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:38:08.141 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.141 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.141 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.141 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 4, 2021-02-10T04:38:08.141 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "91c8a80a-37b6-4c30-a32a-647b2e6d801f", 2021-02-10T04:38:08.142 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:38:08.142 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:38:08.142 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:38:08.142 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:38:08.142 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:38:08.142 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:38:08.142 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 31, 2021-02-10T04:38:08.142 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 638, 2021-02-10T04:38:08.142 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:38:08.142 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:38:08.143 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:38:08.143 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.143 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.143 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.143 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6800", 2021-02-10T04:38:08.143 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.143 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.143 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.143 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.143 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:38:08.144 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.144 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.144 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.144 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6801", 2021-02-10T04:38:08.144 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.144 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.144 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.144 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.144 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:38:08.144 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.145 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.145 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.145 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6803", 2021-02-10T04:38:08.145 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.145 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.145 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.145 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.145 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:38:08.145 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.145 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.145 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.145 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6802", 2021-02-10T04:38:08.145 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.145 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.146 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.146 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.146 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6800/7", 2021-02-10T04:38:08.146 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6801/7", 2021-02-10T04:38:08.146 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6803/7", 2021-02-10T04:38:08.146 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6802/7", 2021-02-10T04:38:08.146 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:38:08.146 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:38:08.146 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:38:08.146 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.146 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.146 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.147 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 5, 2021-02-10T04:38:08.147 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e", 2021-02-10T04:38:08.147 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:38:08.147 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:38:08.147 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:38:08.147 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:38:08.147 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:38:08.147 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:38:08.147 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 37, 2021-02-10T04:38:08.147 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 641, 2021-02-10T04:38:08.147 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:38:08.147 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:38:08.147 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:38:08.147 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.147 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.148 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.148 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6804", 2021-02-10T04:38:08.148 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.148 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.148 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.148 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.148 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:38:08.148 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.148 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.148 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.148 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6805", 2021-02-10T04:38:08.148 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.148 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.149 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.149 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.149 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:38:08.149 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.149 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.149 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.149 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6807", 2021-02-10T04:38:08.149 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.149 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.149 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.149 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.149 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:38:08.150 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.150 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.150 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.150 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6806", 2021-02-10T04:38:08.150 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.150 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.150 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.150 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.150 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6804/7", 2021-02-10T04:38:08.150 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6805/7", 2021-02-10T04:38:08.150 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6807/7", 2021-02-10T04:38:08.150 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6806/7", 2021-02-10T04:38:08.151 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:38:08.151 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:38:08.151 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:38:08.151 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.151 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.151 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.151 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 6, 2021-02-10T04:38:08.151 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "adc40430-3eeb-4003-a824-343021243077", 2021-02-10T04:38:08.151 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:38:08.151 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:38:08.151 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:38:08.151 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:38:08.152 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:38:08.152 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:38:08.152 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 43, 2021-02-10T04:38:08.152 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 638, 2021-02-10T04:38:08.152 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:38:08.152 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:38:08.152 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:38:08.152 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.152 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.152 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.152 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6808", 2021-02-10T04:38:08.152 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.153 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.153 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.153 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.153 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:38:08.153 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.153 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.153 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.153 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6809", 2021-02-10T04:38:08.153 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.153 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.153 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.153 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.154 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:38:08.154 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.154 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.154 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.154 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6811", 2021-02-10T04:38:08.154 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.154 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.154 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.154 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.154 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:38:08.154 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.154 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.155 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.155 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6810", 2021-02-10T04:38:08.155 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.155 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.155 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.155 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.155 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6808/7", 2021-02-10T04:38:08.155 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6809/7", 2021-02-10T04:38:08.155 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6811/7", 2021-02-10T04:38:08.155 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6810/7", 2021-02-10T04:38:08.155 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:38:08.155 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:38:08.155 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:38:08.156 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.156 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.156 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.156 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 7, 2021-02-10T04:38:08.156 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37", 2021-02-10T04:38:08.156 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:38:08.156 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:38:08.156 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:38:08.156 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 0, 2021-02-10T04:38:08.156 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:38:08.156 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:38:08.156 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 50, 2021-02-10T04:38:08.157 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 112, 2021-02-10T04:38:08.157 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:38:08.157 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:38:08.157 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:38:08.157 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.157 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.157 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.157 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6812", 2021-02-10T04:38:08.157 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.157 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.157 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.157 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.158 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:38:08.158 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.158 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.158 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.158 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6813", 2021-02-10T04:38:08.158 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.158 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.158 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.158 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.158 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:38:08.158 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.158 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.159 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.159 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6815", 2021-02-10T04:38:08.159 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.159 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.159 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.159 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.159 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:38:08.159 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:38:08.159 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.159 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:38:08.159 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6814", 2021-02-10T04:38:08.159 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:38:08.159 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.160 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.160 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.160 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6812/7", 2021-02-10T04:38:08.160 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6813/7", 2021-02-10T04:38:08.160 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6815/7", 2021-02-10T04:38:08.160 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6814/7", 2021-02-10T04:38:08.160 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:38:08.160 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:38:08.160 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:38:08.160 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.161 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.161 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:38:08.161 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_xinfo": [ 2021-02-10T04:38:08.161 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.161 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 0, 2021-02-10T04:38:08.161 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:38:08.161 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:38:08.161 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:38:08.161 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:38:08.161 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:38:08.161 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:37:33.740656+0000", 2021-02-10T04:38:08.161 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:38:08.162 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.162 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.162 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 1, 2021-02-10T04:38:08.162 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:38:08.162 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:38:08.162 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:38:08.162 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:38:08.162 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:38:08.162 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:37:50.553663+0000", 2021-02-10T04:38:08.162 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:38:08.162 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.163 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.163 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 2, 2021-02-10T04:38:08.163 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:38:08.163 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:38:08.163 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:38:08.163 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:38:08.163 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 65536, 2021-02-10T04:38:08.163 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:37:08.605364+0000", 2021-02-10T04:38:08.163 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:38:08.163 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.163 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.163 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 3, 2021-02-10T04:38:08.164 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:38:08.164 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:38:08.164 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:38:08.164 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:38:08.164 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:38:08.164 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:37:29.449193+0000", 2021-02-10T04:38:08.164 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:38:08.164 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.164 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.164 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 4, 2021-02-10T04:38:08.164 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:38:08.164 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:38:08.165 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:38:08.165 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:38:08.165 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:38:08.165 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:37:45.482448+0000", 2021-02-10T04:38:08.165 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:38:08.165 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.165 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.165 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 5, 2021-02-10T04:38:08.165 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:38:08.165 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:38:08.165 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:38:08.165 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:38:08.165 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:38:08.166 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:37:04.740466+0000", 2021-02-10T04:38:08.166 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:38:08.166 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.166 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.166 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 6, 2021-02-10T04:38:08.166 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:38:08.166 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:38:08.166 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:38:08.166 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:38:08.166 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:38:08.166 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:37:21.548801+0000", 2021-02-10T04:38:08.167 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:38:08.167 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.167 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.167 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 7, 2021-02-10T04:38:08.167 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:38:08.167 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:38:08.167 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:38:08.167 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:38:08.167 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:38:08.167 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:37:39.944872+0000", 2021-02-10T04:38:08.167 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:38:08.168 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.168 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:38:08.168 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_upmap": [], 2021-02-10T04:38:08.168 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_upmap_items": [ 2021-02-10T04:38:08.168 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.168 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "2.7", 2021-02-10T04:38:08.168 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:38:08.168 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.168 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:38:08.168 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 4 2021-02-10T04:38:08.168 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.169 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.169 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.169 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.169 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "2.21", 2021-02-10T04:38:08.169 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:38:08.169 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.169 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:38:08.169 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 1 2021-02-10T04:38:08.169 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.169 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.169 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.170 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.170 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.10", 2021-02-10T04:38:08.170 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:38:08.170 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.170 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 1, 2021-02-10T04:38:08.170 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 5 2021-02-10T04:38:08.170 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.170 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.170 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.170 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:38:08.170 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_temp": [], 2021-02-10T04:38:08.170 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_temp": [], 2021-02-10T04:38:08.171 INFO:teuthology.orchestra.run.smithi085.stdout: "blocklist": { 2021-02-10T04:38:08.171 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/801393150": "2021-02-11T04:21:37.851607+0000", 2021-02-10T04:38:08.171 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/3049671696": "2021-02-11T04:21:37.851607+0000", 2021-02-10T04:38:08.171 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/4056505602": "2021-02-11T04:21:20.700478+0000", 2021-02-10T04:38:08.171 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:6800/7": "2021-02-11T04:21:20.700478+0000", 2021-02-10T04:38:08.171 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/1879026062": "2021-02-11T04:21:20.700478+0000" 2021-02-10T04:38:08.171 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.171 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profiles": { 2021-02-10T04:38:08.171 INFO:teuthology.orchestra.run.smithi085.stdout: "default": { 2021-02-10T04:38:08.171 INFO:teuthology.orchestra.run.smithi085.stdout: "crush-failure-domain": "osd", 2021-02-10T04:38:08.171 INFO:teuthology.orchestra.run.smithi085.stdout: "k": "2", 2021-02-10T04:38:08.172 INFO:teuthology.orchestra.run.smithi085.stdout: "m": "1", 2021-02-10T04:38:08.172 INFO:teuthology.orchestra.run.smithi085.stdout: "plugin": "jerasure", 2021-02-10T04:38:08.172 INFO:teuthology.orchestra.run.smithi085.stdout: "ruleset-failure-domain": "osd", 2021-02-10T04:38:08.172 INFO:teuthology.orchestra.run.smithi085.stdout: "technique": "reed_sol_van" 2021-02-10T04:38:08.172 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.172 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.172 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps_queue": [ 2021-02-10T04:38:08.172 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.172 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 7, 2021-02-10T04:38:08.172 INFO:teuthology.orchestra.run.smithi085.stdout: "snaps": [ 2021-02-10T04:38:08.172 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.172 INFO:teuthology.orchestra.run.smithi085.stdout: "begin": 461, 2021-02-10T04:38:08.172 INFO:teuthology.orchestra.run.smithi085.stdout: "length": 1 2021-02-10T04:38:08.173 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.173 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.173 INFO:teuthology.orchestra.run.smithi085.stdout: "begin": 464, 2021-02-10T04:38:08.173 INFO:teuthology.orchestra.run.smithi085.stdout: "length": 2 2021-02-10T04:38:08.173 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.173 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.173 INFO:teuthology.orchestra.run.smithi085.stdout: "begin": 467, 2021-02-10T04:38:08.173 INFO:teuthology.orchestra.run.smithi085.stdout: "length": 1 2021-02-10T04:38:08.173 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.173 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.173 INFO:teuthology.orchestra.run.smithi085.stdout: "begin": 469, 2021-02-10T04:38:08.173 INFO:teuthology.orchestra.run.smithi085.stdout: "length": 1 2021-02-10T04:38:08.173 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.174 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.174 INFO:teuthology.orchestra.run.smithi085.stdout: "begin": 472, 2021-02-10T04:38:08.174 INFO:teuthology.orchestra.run.smithi085.stdout: "length": 2 2021-02-10T04:38:08.174 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.174 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.174 INFO:teuthology.orchestra.run.smithi085.stdout: "begin": 475, 2021-02-10T04:38:08.174 INFO:teuthology.orchestra.run.smithi085.stdout: "length": 2 2021-02-10T04:38:08.174 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.174 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.174 INFO:teuthology.orchestra.run.smithi085.stdout: "begin": 479, 2021-02-10T04:38:08.174 INFO:teuthology.orchestra.run.smithi085.stdout: "length": 1 2021-02-10T04:38:08.174 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.175 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.175 INFO:teuthology.orchestra.run.smithi085.stdout: "begin": 481, 2021-02-10T04:38:08.175 INFO:teuthology.orchestra.run.smithi085.stdout: "length": 2 2021-02-10T04:38:08.175 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.175 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.175 INFO:teuthology.orchestra.run.smithi085.stdout: "begin": 484, 2021-02-10T04:38:08.175 INFO:teuthology.orchestra.run.smithi085.stdout: "length": 1 2021-02-10T04:38:08.175 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.175 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.175 INFO:teuthology.orchestra.run.smithi085.stdout: "begin": 489, 2021-02-10T04:38:08.175 INFO:teuthology.orchestra.run.smithi085.stdout: "length": 1 2021-02-10T04:38:08.175 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.176 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.176 INFO:teuthology.orchestra.run.smithi085.stdout: "begin": 491, 2021-02-10T04:38:08.176 INFO:teuthology.orchestra.run.smithi085.stdout: "length": 1 2021-02-10T04:38:08.176 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.176 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.176 INFO:teuthology.orchestra.run.smithi085.stdout: "begin": 494, 2021-02-10T04:38:08.176 INFO:teuthology.orchestra.run.smithi085.stdout: "length": 3 2021-02-10T04:38:08.176 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.176 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.176 INFO:teuthology.orchestra.run.smithi085.stdout: "begin": 498, 2021-02-10T04:38:08.176 INFO:teuthology.orchestra.run.smithi085.stdout: "length": 3 2021-02-10T04:38:08.176 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.177 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.177 INFO:teuthology.orchestra.run.smithi085.stdout: "begin": 503, 2021-02-10T04:38:08.177 INFO:teuthology.orchestra.run.smithi085.stdout: "length": 2 2021-02-10T04:38:08.177 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.177 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.177 INFO:teuthology.orchestra.run.smithi085.stdout: "begin": 513, 2021-02-10T04:38:08.177 INFO:teuthology.orchestra.run.smithi085.stdout: "length": 1 2021-02-10T04:38:08.177 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.177 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.177 INFO:teuthology.orchestra.run.smithi085.stdout: "begin": 518, 2021-02-10T04:38:08.177 INFO:teuthology.orchestra.run.smithi085.stdout: "length": 1 2021-02-10T04:38:08.177 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.177 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.178 INFO:teuthology.orchestra.run.smithi085.stdout: "begin": 525, 2021-02-10T04:38:08.178 INFO:teuthology.orchestra.run.smithi085.stdout: "length": 1 2021-02-10T04:38:08.178 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.178 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.178 INFO:teuthology.orchestra.run.smithi085.stdout: "begin": 539, 2021-02-10T04:38:08.178 INFO:teuthology.orchestra.run.smithi085.stdout: "length": 1 2021-02-10T04:38:08.178 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.178 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.178 INFO:teuthology.orchestra.run.smithi085.stdout: "begin": 541, 2021-02-10T04:38:08.178 INFO:teuthology.orchestra.run.smithi085.stdout: "length": 1 2021-02-10T04:38:08.179 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:38:08.179 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:38:08.179 INFO:teuthology.orchestra.run.smithi085.stdout: "begin": 546, 2021-02-10T04:38:08.179 INFO:teuthology.orchestra.run.smithi085.stdout: "length": 1 2021-02-10T04:38:08.179 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.179 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:38:08.179 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.179 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:38:08.179 INFO:teuthology.orchestra.run.smithi085.stdout: "new_removed_snaps": [], 2021-02-10T04:38:08.179 INFO:teuthology.orchestra.run.smithi085.stdout: "new_purged_snaps": [], 2021-02-10T04:38:08.179 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_node_flags": {}, 2021-02-10T04:38:08.179 INFO:teuthology.orchestra.run.smithi085.stdout: "device_class_flags": {}, 2021-02-10T04:38:08.180 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_mode": { 2021-02-10T04:38:08.180 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_mode_enabled": false, 2021-02-10T04:38:08.180 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_bucket_count": 0, 2021-02-10T04:38:08.180 INFO:teuthology.orchestra.run.smithi085.stdout: "degraded_stretch_mode": 0, 2021-02-10T04:38:08.180 INFO:teuthology.orchestra.run.smithi085.stdout: "recovering_stretch_mode": 0, 2021-02-10T04:38:08.180 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_mode_bucket": 0 2021-02-10T04:38:08.180 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:38:08.180 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:38:08.180 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[29319]: cluster 2021-02-10T04:38:08.180 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[29319]: 2021-02-10T04:38:08.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[29319]: 2021 2021-02-10T04:38:08.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[29319]: - 2021-02-10T04:38:08.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[29319]: 02-10T04:38:03.261127+0000 2021-02-10T04:38:08.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[29319]: osd.4 (osd.4) 1047 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:08.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:38:08.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[29319]: :38:03.261132+0000 osd.4 (osd.4) 2021-02-10T04:38:08.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[29319]: 1048 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:08.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:38:08.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[29319]: 38:03.261136+0000 osd.4 (osd.4) 1049 2021-02-10T04:38:08.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:08.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:38:08.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[29319]: 04:38:03.261139+0000 osd.4 (osd. 2021-02-10T04:38:08.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[29319]: 4) 1050 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:08.182 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:38:08.182 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[29319]: -10T04:38:04.150023+0000 2021-02-10T04:38:08.182 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[29319]: mgr.y (mgr.14140) 695 : cluster [DBG] 2021-02-10T04:38:08.182 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[29319]: pgmap v1148: 168 pgs: 155 active+clean, 8 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 397 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.0 MiB/s rd, 1.5 MiB/s wr, 11 op/s 2021-02-10T04:38:08.182 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:38:08.182 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[29319]: -02-10T04:38:04.225725 2021-02-10T04:38:08.182 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1547 : cluster [DBG] 2021-02-10T04:38:08.182 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:04 smithi085 conmon[29319]: osdmap e666: 8 total, 8 up, 7 in 2021-02-10T04:38:08.182 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[29319]: cluster 2021-02-10T04:38:04.304894+0000 osd.4 (osd.4) 1051 2021-02-10T04:38:08.182 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:08.182 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[29319]: cluster 2021-02-10T04:38:08.183 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[29319]: 2021-02-10T04:38:04.304903+0000 osd.4 (osd.4) 1052 : 2021-02-10T04:38:08.183 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:08.183 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:38:08.183 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[29319]: -02-10T04:38:04.304913 2021-02-10T04:38:08.183 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1053 : cluster 2021-02-10T04:38:08.183 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:08.183 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:38:08.183 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[29319]: :38:04.304917+0000 osd.4 (osd. 2021-02-10T04:38:08.183 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[29319]: 4) 1054 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:08.183 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:38:08.183 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[29319]: T04:38:04.543684+0000 mon.a ( 2021-02-10T04:38:08.183 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[29319]: mon.0) 1548 : audit [DBG] from='client.? 172.21.15.85:0/2237760813' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-02-10T04:38:08.184 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:38:08.184 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[29319]: 02-10T04:38:04.785837+0000 2021-02-10T04:38:08.184 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[29319]: osd.5 (osd.5) 71 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:38:08.184 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[29319]: 2021-02-10T04:38:08.184 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[29319]: cluster 2021-02-10T04:38:08.184 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[29319]: 2021-02-10T04:38:04.786840+0000 osd.5 ( 2021-02-10T04:38:08.184 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[29319]: osd.5) 72 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:38:08.184 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:38:08.184 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[29319]: :38:05.227648+0000 mon.a (mon.0 2021-02-10T04:38:08.184 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:05 smithi085 conmon[29319]: ) 1549 : cluster [DBG] osdmap e667: 8 total, 8 up, 7 in 2021-02-10T04:38:08.185 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[29319]: cluster 2021-02-10T04:38:08.185 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[29319]: 2021-02-10T04:38:05.268631 2021-02-10T04:38:08.185 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1055 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:08.185 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:38:08.185 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[29319]: 10T04:38:05.268641+0000 osd.4 ( 2021-02-10T04:38:08.185 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[29319]: osd.4) 1056 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:08.185 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[29319]: cluster 2021-02-10T04:38 2021-02-10T04:38:08.185 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[29319]: :05.268647+0000 osd.4 (osd.4) 1057 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:08.185 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[29319]: cluster 2021-02-10T04:38:08.185 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:38:08.185 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[29319]: 04:38:05.268653+0000 osd.4 (osd.4) 1058 : cluster 2021-02-10T04:38:08.185 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:08.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[29319]: cluster 2021-02-10T04:38:06.150625+0000 mgr.y 2021-02-10T04:38:08.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[29319]: (mgr.14140) 696 : cluster [DBG] pgmap v1151: 168 pgs: 155 active+clean, 8 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 392 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 9.1 MiB/s rd, 1.5 MiB/s wr, 15 op/s 2021-02-10T04:38:08.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:38:08.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[29319]: 04:38:06.228261+0000 mon.a (mon.0 2021-02-10T04:38:08.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:06 smithi085 conmon[29319]: ) 1550 : cluster [DBG] osdmap e668: 8 total, 8 up, 7 in 2021-02-10T04:38:08.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[29319]: cluster 2021-02-10T04:38:08.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:38:08.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[29319]: 38:06.292180+0000 osd.4 (osd.4) 1059 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:08.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[29319]: cluster 2021-02-10T04:38:06 2021-02-10T04:38:08.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[29319]: .292194+0000 osd.4 (osd.4) 2021-02-10T04:38:08.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[29319]: 1060 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:08.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:38:08.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[29319]: -02-10T04:38: 2021-02-10T04:38:08.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[29319]: 06.292201+0000 osd.4 (osd 2021-02-10T04:38:08.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[29319]: .4) 1061 : cluster [WRN] 2021-02-10T04:38:08.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:08.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:38:08.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[29319]: 02-10T04:38:06.292205+0000 osd.4 (osd 2021-02-10T04:38:08.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[29319]: .4) 1062 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:08.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:38:08.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[29319]: -02-10T04:38:07 2021-02-10T04:38:08.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[29319]: .228375+0000 mon.a (mon.0) 1551 : 2021-02-10T04:38:08.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:07 smithi085 conmon[29319]: cluster [DBG] osdmap e669: 8 total, 8 up, 7 in 2021-02-10T04:38:08.188 INFO:tasks.rados.rados.0.smithi085.stdout:2579: done (4 left) 2021-02-10T04:38:08.188 INFO:tasks.rados.rados.0.smithi085.stdout:2580: read oid 34 snap 273 2021-02-10T04:38:08.188 INFO:tasks.rados.rados.0.smithi085.stdout:2580: expect deleted 2021-02-10T04:38:08.188 INFO:tasks.rados.rados.0.smithi085.stdout:2581: read oid 38 snap -1 2021-02-10T04:38:08.188 INFO:tasks.rados.rados.0.smithi085.stdout:2581: expect (ObjNum 886 snap 287 seq_num 886) 2021-02-10T04:38:08.188 INFO:tasks.rados.rados.0.smithi085.stdout:2582: rollback oid 16 current snap is 292 2021-02-10T04:38:08.188 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 16 to 291 2021-02-10T04:38:08.188 INFO:tasks.rados.rados.0.smithi085.stdout:2583: snap_create 2021-02-10T04:38:08.188 INFO:tasks.rados.rados.0.smithi085.stdout:2582: finishing rollback tid 1 to smithi08575397-16 2021-02-10T04:38:08.188 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 1118 (ObjNum 1953066355 snap 942696808 seq_num 859125557) dirty dne 2021-02-10T04:38:08.189 INFO:tasks.rados.rados.0.smithi085.stdout:2581: expect (ObjNum 886 snap 287 seq_num 886) 2021-02-10T04:38:08.189 INFO:tasks.rados.rados.0.smithi085.stdout:2580: done (7 left) 2021-02-10T04:38:08.189 INFO:tasks.rados.rados.0.smithi085.stdout:2581: done (6 left) 2021-02-10T04:38:08.189 INFO:tasks.rados.rados.0.smithi085.stdout:2582: done (5 left) 2021-02-10T04:38:08.189 INFO:tasks.rados.rados.0.smithi085.stdout:2583: done (4 left) 2021-02-10T04:38:08.189 INFO:tasks.rados.rados.0.smithi085.stdout:2584: snap_create 2021-02-10T04:38:08.189 INFO:tasks.rados.rados.0.smithi085.stdout:2584: done (4 left) 2021-02-10T04:38:08.189 INFO:tasks.rados.rados.0.smithi085.stdout:2585: delete oid 3 current snap is 294 2021-02-10T04:38:08.189 INFO:tasks.rados.rados.0.smithi085.stdout:2585: done (4 left) 2021-02-10T04:38:08.189 INFO:tasks.rados.rados.0.smithi085.stdout:2586: delete oid 5 current snap is 294 2021-02-10T04:38:08.189 INFO:tasks.rados.rados.0.smithi085.stdout:2586: done (4 left) 2021-02-10T04:38:08.189 INFO:tasks.rados.rados.0.smithi085.stdout:2587: snap_remove snap 267 2021-02-10T04:38:08.190 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:38:08.218 INFO:tasks.rados.rados.0.smithi085.stdout:2587: done (4 left) 2021-02-10T04:38:08.218 INFO:tasks.rados.rados.0.smithi085.stdout:2588: snap_create 2021-02-10T04:38:08.297 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:38:08 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:38:08.297 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:08 smithi164 conmon[33677]: debug 2021-02-10T04:38:08.296+0000 7fbec19a1700 -1 osd.4 670 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:38:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:08 smithi164 conmon[29288]: cluster 2021-02-10T04:38:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:08 smithi164 conmon[29288]: 2021-02-10T04:38:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:08 smithi164 conmon[29288]: 2021 2021-02-10T04:38:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:08 smithi164 conmon[29288]: -02-10T04:38:07.249493 2021-02-10T04:38:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:08 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1063 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:08 smithi164 conmon[29288]: cluster 2021-02-10T04:38 2021-02-10T04:38:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:08 smithi164 conmon[29288]: :07.249499+0000 osd.4 (osd.4) 1064 : cluster 2021-02-10T04:38:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:08 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:08 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:38:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:08 smithi164 conmon[29288]: -10T04:38:07.249503+0000 osd.4 (osd 2021-02-10T04:38:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:08 smithi164 conmon[29288]: .4) 1065 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:08 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:38:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:08 smithi164 conmon[29288]: -02-10T04:38:07.249507+0000 2021-02-10T04:38:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:08 smithi164 conmon[29288]: osd.4 (osd.4) 1066 : cluster [WRN] 2021-02-10T04:38:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:08 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:08 smithi164 conmon[29288]: cluster 2021-02-10T04:38:08.151165+0000 mgr.y (mgr.14140) 697 : cluster [DBG] pgmap v1154: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 388 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 6.1 MiB/s rd, 1.5 MiB/s wr, 10 op/s 2021-02-10T04:38:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:08 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:38:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:08 smithi164 conmon[29288]: -02-10T04:38:08.215872+0000 mon.a (mon.0) 1552 2021-02-10T04:38:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:08 smithi164 conmon[29288]: : cluster [DBG] osdmap e670: 8 total, 8 up, 7 in 2021-02-10T04:38:08.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[29319]: cluster 2021-02-10T04:38:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[29319]: 2021- 2021-02-10T04:38:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[29319]: 02-10T04:38:07.249493+0000 osd.4 (osd.4) 1063 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:38:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[29319]: :38:07.249499+0000 osd.4 (osd.4) 1064 : cluster [WRN] 2021-02-10T04:38:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:38:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[29319]: -02-10T04:38:07. 2021-02-10T04:38:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[29319]: 249503+0000 osd.4 (osd.4) 2021-02-10T04:38:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[29319]: 1065 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[29319]: cluster 2021-02-10T04:38:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[29319]: 2021-02-10T04:38:07.249507+0000 2021-02-10T04:38:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[29319]: osd.4 2021-02-10T04:38:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[29319]: (osd.4) 1066 : cluster 2021-02-10T04:38:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:38:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[29319]: -02-10T04:38:08.151165+0000 2021-02-10T04:38:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[29319]: mgr.y (mgr.14140) 697 : cluster [DBG] pgmap v1154: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 388 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 6.1 MiB/s rd, 1.5 MiB/s wr, 10 op/s 2021-02-10T04:38:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[29319]: cluster 2021-02-10T04:38:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[29319]: 2021-02-10T04:38:08.215872+0000 mon.a (mon.0 2021-02-10T04:38:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[29319]: ) 1552 : cluster [DBG] osdmap e670: 8 total, 8 up, 7 in 2021-02-10T04:38:08.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[35961]: cluster 2021-02-10T04:38: 2021-02-10T04:38:08.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[35961]: 07.249493+0000 osd.4 (osd.4) 1063 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:08.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[35961]: cluster 2021-02-10T04:38:08.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[35961]: 2021-02-10T04:38:07.249499 2021-02-10T04:38:08.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2021-02-10T04:38:08.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[35961]: 1064 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:08.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:38:08.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[35961]: 02-10T04:38:07.249503 2021-02-10T04:38:08.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2021-02-10T04:38:08.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[35961]: 1065 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:08.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:38:08.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[35961]: 02-10T04:38:07.249507+0000 2021-02-10T04:38:08.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[35961]: osd.4 (osd.4) 1066 : cluster 2021-02-10T04:38:08.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:38:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[35961]: 02-10T04:38:08.151165+0000 2021-02-10T04:38:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[35961]: mgr.y (mgr.14140) 697 : cluster [DBG] pgmap v1154: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 388 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 6.1 MiB/s rd, 1.5 MiB/s wr, 10 op/s 2021-02-10T04:38:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[35961]: 2021-02-10T04:38:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[35961]: cluster 2021-02-10T04:38:08 2021-02-10T04:38:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[35961]: .215872+0000 mon.a (mon.0) 1552 : cluster [DBG] 2021-02-10T04:38:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:08 smithi085 conmon[35961]: osdmap e670: 8 total, 8 up, 7 in 2021-02-10T04:38:09.222 INFO:tasks.rados.rados.0.smithi085.stdout:2588: done (4 left) 2021-02-10T04:38:09.222 INFO:tasks.rados.rados.0.smithi085.stdout:2589: write (excl) oid 10 current snap is 295 2021-02-10T04:38:09.222 INFO:tasks.rados.rados.0.smithi085.stdout:2589: seq_num 899 ranges {709493=578505,1907312=685978} 2021-02-10T04:38:09.227 INFO:tasks.rados.rados.0.smithi085.stdout:2589: writing smithi08575397-10 from 709493 to 1287998 tid 1 2021-02-10T04:38:09.235 INFO:tasks.rados.rados.0.smithi085.stdout:2589: writing smithi08575397-10 from 1907312 to 2593290 tid 2 2021-02-10T04:38:09.236 INFO:tasks.rados.rados.0.smithi085.stdout:2590: delete oid 7 current snap is 295 2021-02-10T04:38:09.236 INFO:tasks.rados.rados.0.smithi085.stdout:2589: finishing write tid 1 to smithi08575397-10 2021-02-10T04:38:09.241 INFO:tasks.rados.rados.0.smithi085.stdout:2590: done (5 left) 2021-02-10T04:38:09.241 INFO:tasks.rados.rados.0.smithi085.stdout:2591: delete oid 22 current snap is 295 2021-02-10T04:38:09.241 INFO:tasks.rados.rados.0.smithi085.stdout:2589: finishing write tid 2 to smithi08575397-10 2021-02-10T04:38:09.242 INFO:tasks.rados.rados.0.smithi085.stdout:2589: finishing write tid 3 to smithi08575397-10 2021-02-10T04:38:09.242 INFO:tasks.rados.rados.0.smithi085.stdout:2589: finishing write tid 4 to smithi08575397-10 2021-02-10T04:38:09.242 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 10 v 874 (ObjNum 899 snap 295 seq_num 899) dirty exists 2021-02-10T04:38:09.242 INFO:tasks.rados.rados.0.smithi085.stdout:2589: left oid 10 (ObjNum 899 snap 295 seq_num 899) 2021-02-10T04:38:09.246 INFO:tasks.rados.rados.0.smithi085.stdout:2589: done (5 left) 2021-02-10T04:38:09.246 INFO:tasks.rados.rados.0.smithi085.stdout:2591: done (4 left) 2021-02-10T04:38:09.246 INFO:tasks.rados.rados.0.smithi085.stdout:2592: snap_create 2021-02-10T04:38:09.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:09 smithi164 conmon[33677]: debug 2021-02-10T04:38:09.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:09 smithi164 conmon[33677]: 2021-02-10T04:38:09.325+0000 7fbec19a1700 -1 osd.4 671 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:38:09.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:09 smithi164 conmon[29288]: audit 2021-02-10T04:38:09.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:09 smithi164 conmon[29288]: 2021-02-10T04:38:09.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:09 smithi164 conmon[29288]: 2021 2021-02-10T04:38:09.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:09 smithi164 conmon[29288]: - 2021-02-10T04:38:09.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:09 smithi164 conmon[29288]: 02 2021-02-10T04:38:09.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:09 smithi164 conmon[29288]: -10T04:38:08.250031+0000 2021-02-10T04:38:09.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:09 smithi164 conmon[29288]: mgr.y (mgr.14140) 698 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:38:09.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:09 smithi164 conmon[29288]: cluster 2021-02-10T04:38:08.296839+0000 osd.4 ( 2021-02-10T04:38:09.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:09 smithi164 conmon[29288]: osd.4) 1067 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:09.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:09 smithi164 conmon[29288]: cluster 2021-02-10T04:38: 2021-02-10T04:38:09.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:09 smithi164 conmon[29288]: 08.296848+0000 osd.4 (osd.4) 1068 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:09.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:09 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:38:09.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:09 smithi164 conmon[29288]: -10T04:38:08.296854+0000 osd.4 (osd.4 2021-02-10T04:38:09.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:09 smithi164 conmon[29288]: ) 1069 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:09.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:09 smithi164 conmon[29288]: cluster 2021-02-10T04:38:08. 2021-02-10T04:38:09.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:09 smithi164 conmon[29288]: 296861+0000 osd.4 (osd.4) 1070 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:09.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:09 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:38:09.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:09 smithi164 conmon[29288]: :38:08.440014+0000 mon.a (mon.0) 1553 : cluster [WRN] 2021-02-10T04:38:09.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:09 smithi164 conmon[29288]: Health check update: 4 slow ops, oldest one blocked for 517 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:38:09.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:09 smithi164 conmon[29288]: cluster 2021-02-10T04:38:09.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:09 smithi164 conmon[29288]: 2021-02-10T04:38:09.219147+0000 mon.a (mon.0) 1554 : cluster [DBG] 2021-02-10T04:38:09.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:09 smithi164 conmon[29288]: osdmap e671: 8 total, 8 up, 7 in 2021-02-10T04:38:09.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[29319]: audit 2021-02-10T04:38:09.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[29319]: 2021-02-10T04:38:09.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[29319]: 2021 2021-02-10T04:38:09.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[29319]: - 2021-02-10T04:38:09.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[29319]: 02 2021-02-10T04:38:09.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[29319]: - 2021-02-10T04:38:09.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[29319]: 10 2021-02-10T04:38:09.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[29319]: T04:38:08. 2021-02-10T04:38:09.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[29319]: 250031+0000 mgr.y (mgr.14140) 698 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:38:09.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:38:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[29319]: 02-10T04:38:08.296839+0000 osd.4 (osd. 2021-02-10T04:38:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[29319]: 4) 1067 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:38:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[29319]: 02-10T04:38:08.296848+0000 2021-02-10T04:38:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[29319]: osd.4 (osd.4) 1068 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[29319]: 2021-02-10T04:38:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[29319]: cluster 2021-02-10T04:38:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[29319]: 2021-02-10T04:38:08.296854+0000 osd.4 ( 2021-02-10T04:38:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[29319]: osd.4) 1069 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:38:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[29319]: -02-10T04:38:08.296861+0000 2021-02-10T04:38:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[29319]: osd.4 (osd.4) 1070 : cluster [WRN] 2021-02-10T04:38:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[29319]: cluster 2021-02-10T04:38:08. 2021-02-10T04:38:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[29319]: 440014+0000 mon.a (mon.0) 1553 : cluster 2021-02-10T04:38:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[29319]: [WRN] Health check update: 4 slow ops, oldest one blocked for 517 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:38:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:38:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[29319]: 38:09.219147+0000 mon.a (mon.0) 2021-02-10T04:38:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[29319]: 1554 : cluster [DBG] osdmap e671: 8 total, 8 up, 7 in 2021-02-10T04:38:09.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:38:09.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[35961]: 10T04:38:08.250031+0000 mgr.y (mgr.14140) 698 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:38:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[35961]: cluster 2021-02-10T04:38:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:38:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[35961]: :38:08.296839+0000 osd.4 (osd.4 2021-02-10T04:38:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[35961]: ) 1067 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[35961]: cluster 2021-02-10T04:38:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[35961]: 2021-02-10T04:38:08.296848+0000 osd.4 2021-02-10T04:38:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[35961]: (osd.4) 1068 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[35961]: 2021-02-10T04:38:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[35961]: cluster 2021-02-10T04:38:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:38:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[35961]: :38:08.296854+0000 osd.4 (osd.4) 1069 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:09.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:38:09.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[35961]: 10T04:38:08.296861+0000 osd.4 (osd 2021-02-10T04:38:09.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[35961]: .4) 1070 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:09.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[35961]: cluster 2021-02-10T04:38:09.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[35961]: 2021-02-10T04:38:08.440014+0000 2021-02-10T04:38:09.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[35961]: mon.a (mon.0) 1553 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 517 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:38:09.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[35961]: 2021-02-10T04:38:09.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[35961]: cluster 2021-02-10T04:38:09.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[35961]: 2021-02-10T04:38:09.219147+0000 mon.a (mon 2021-02-10T04:38:09.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:09 smithi085 conmon[35961]: .0) 1554 : cluster [DBG] osdmap e671: 8 total, 8 up, 7 in 2021-02-10T04:38:10.226 INFO:tasks.rados.rados.0.smithi085.stdout:2592: done (4 left) 2021-02-10T04:38:10.226 INFO:tasks.rados.rados.0.smithi085.stdout:2593: snap_create 2021-02-10T04:38:10.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:10 smithi164 conmon[33677]: debug 2021-02-10T04:38:10.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:10 smithi164 conmon[33677]: 2021-02-10T04:38:10.302+0000 7fbec19a1700 -1 osd.4 672 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:38:10.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:10 smithi164 conmon[29288]: cluster 2021-02-10T04:38:09.326441+0000 osd.4 (osd.4 2021-02-10T04:38:10.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:10 smithi164 conmon[29288]: ) 1071 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:10.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:10 smithi164 conmon[29288]: cluster 2021-02-10T04:38:09.326449+0000 osd.4 (osd.4) 1072 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:10.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:10 smithi164 conmon[29288]: 2021-02-10T04:38:10.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:10 smithi164 conmon[29288]: cluster 2021-02-10T04:38: 2021-02-10T04:38:10.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:10 smithi164 conmon[29288]: 09.326455+0000 osd.4 (osd.4) 1073 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:10.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:10 smithi164 conmon[29288]: 2021-02-10T04:38:10.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:10 smithi164 conmon[29288]: cluster 2021-02-10T04:38:10.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:10 smithi164 conmon[29288]: 2021-02-10T04:38:09.326460+0000 2021-02-10T04:38:10.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:10 smithi164 conmon[29288]: osd.4 (osd.4) 1074 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:10.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:10 smithi164 conmon[29288]: cluster 2021-02-10T04:38 2021-02-10T04:38:10.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:10 smithi164 conmon[29288]: :10.151782+0000 mgr.y (mgr.14140) 699 : cluster [DBG] pgmap v1157: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 375 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 309 KiB/s wr, 2 op/s 2021-02-10T04:38:10.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:10 smithi164 conmon[29288]: 2021-02-10T04:38:10.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:10 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:38:10.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:10 smithi164 conmon[29288]: 38:10.223620+0000 mon.a (mon.0) 1555 : cluster [DBG] osdmap e672: 8 total, 8 up, 7 in 2021-02-10T04:38:10.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[35961]: cluster 2021-02-10T04:38:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[35961]: 2021-02-10T04:38:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[35961]: 2021-02-10T04:38:09 2021-02-10T04:38:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[35961]: .326441+0000 osd.4 (osd.4) 1071 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:38:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[35961]: -10T04:38:09.326449+0000 osd.4 (osd. 2021-02-10T04:38:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[35961]: 4) 1072 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:38:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[35961]: 38:09.326455+0000 osd.4 (osd.4) 1073 : cluster 2021-02-10T04:38:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:38:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[35961]: 02-10T04:38:09.326460+0000 osd.4 (osd.4) 1074 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[35961]: cluster 2021-02-10T04:38:10.151782+0000 mgr.y ( 2021-02-10T04:38:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[35961]: mgr.14140) 699 : cluster [DBG] pgmap v1157: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 375 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 309 KiB/s wr, 2 op/s 2021-02-10T04:38:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[35961]: cluster 2021-02-10T04:38 2021-02-10T04:38:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[35961]: :10.223620+0000 mon.a (mon.0) 1555 : cluster [DBG] osdmap e672: 8 total, 8 up, 7 in 2021-02-10T04:38:10.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[29319]: cluster 2021-02-10T04:38 2021-02-10T04:38:10.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[29319]: :09.326441+0000 osd.4 (osd.4) 1071 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:10.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[29319]: cluster 2021-02-10T04:38:09.326449+0000 osd.4 (osd.4) 1072 : 2021-02-10T04:38:10.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:10.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[29319]: cluster 2021-02-10T04:38:10.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[29319]: 2021-02-10T04:38:09.326455 2021-02-10T04:38:10.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1073 : cluster [WRN] 2021-02-10T04:38:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:38:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[29319]: -02-10T04:38:09.326460+0000 2021-02-10T04:38:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[29319]: osd.4 (osd.4) 1074 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[29319]: 2021-02-10T04:38:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[29319]: cluster 2021-02-10T04:38:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[29319]: 2021-02-10T04:38:10 2021-02-10T04:38:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[29319]: .151782+0000 mgr.y (mgr.14140) 699 : cluster [DBG] pgmap v1157: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 375 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 309 KiB/s wr, 2 op/s 2021-02-10T04:38:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:38:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[29319]: -10T04:38:10.223620 2021-02-10T04:38:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1555 : 2021-02-10T04:38:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:10 smithi085 conmon[29319]: cluster [DBG] osdmap e672: 8 total, 8 up, 7 in 2021-02-10T04:38:11.228 INFO:tasks.rados.rados.0.smithi085.stdout:2593: done (4 left) 2021-02-10T04:38:11.228 INFO:tasks.rados.rados.0.smithi085.stdout:2594: write oid 23 current snap is 297 2021-02-10T04:38:11.228 INFO:tasks.rados.rados.0.smithi085.stdout:2594: seq_num 902 ranges {759914=404252,1566405=433305,2504010=655904} 2021-02-10T04:38:11.235 INFO:tasks.rados.rados.0.smithi085.stdout:2594: writing smithi08575397-23 from 759914 to 1164166 tid 1 2021-02-10T04:38:11.241 INFO:tasks.rados.rados.0.smithi085.stdout:2594: writing smithi08575397-23 from 1566405 to 1999710 tid 2 2021-02-10T04:38:11.249 INFO:tasks.rados.rados.0.smithi085.stdout:2594: writing smithi08575397-23 from 2504010 to 3159914 tid 3 2021-02-10T04:38:11.249 INFO:tasks.rados.rados.0.smithi085.stdout:2595: snap_remove snap 264 2021-02-10T04:38:11.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:11 smithi164 conmon[29288]: cluster 2021-02-10T04:38:09.528614+0000 osd.2 ( 2021-02-10T04:38:11.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:11 smithi164 conmon[29288]: osd.2) 1933 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:38:11.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:11 smithi164 conmon[29288]: cluster 2021-02-10T04:38:09.529227+0000 osd.2 (osd.2) 1934 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:38:11.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:11 smithi164 conmon[29288]: cluster 2021-02-10T04:38:10. 2021-02-10T04:38:11.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:11 smithi164 conmon[29288]: 302832+0000 osd.4 (osd.4) 1075 : cluster [WRN] 2021-02-10T04:38:11.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:11 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:11.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:11 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:38:11.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:11 smithi164 conmon[29288]: 38:10.302842+0000 osd.4 (osd.4) 2021-02-10T04:38:11.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:11 smithi164 conmon[29288]: 1076 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:11.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:11 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:38:11.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:11 smithi164 conmon[29288]: 02-10T04:38:10.302853+0000 osd.4 2021-02-10T04:38:11.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:11 smithi164 conmon[29288]: (osd.4) 1077 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:11.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:11 smithi164 conmon[29288]: cluster 2021-02-10T04:38:11.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:11 smithi164 conmon[29288]: 2021-02-10T04:38:10.302860+0000 2021-02-10T04:38:11.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:11 smithi164 conmon[29288]: osd.4 (osd.4) 1078 : cluster 2021-02-10T04:38:11.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:11 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:11.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:11 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:38:11.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:11 smithi164 conmon[29288]: T04:38:10.648307+0000 mon.a (mon 2021-02-10T04:38:11.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:11 smithi164 conmon[29288]: .0) 1556 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:38:11.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:11 smithi164 conmon[29288]: cephadm 2021-02-10T04:38:11.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:11 smithi164 conmon[29288]: 2021-02-10T04:38:10.648921+0000 2021-02-10T04:38:11.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:11 smithi164 conmon[29288]: mgr.y (mgr.14140) 700 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:38:11.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:11 smithi164 conmon[29288]: 2021-02-10T04:38:11.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:11 smithi164 conmon[29288]: cluster 2021-02-10T04:38:11 2021-02-10T04:38:11.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:11 smithi164 conmon[29288]: .224458+0000 mon.a (mon.0) 1557 : 2021-02-10T04:38:11.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:11 smithi164 conmon[29288]: cluster [DBG] osdmap e673: 8 total, 8 up, 7 in 2021-02-10T04:38:11.646 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:11 smithi164 conmon[33677]: debug 2021-02-10T04:38:11.646 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:11 smithi164 conmon[33677]: 2021-02-10T04:38:11.305+0000 7fbec19a1700 -1 osd.4 673 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:38:11.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: cluster 2021-02-10T04:38:09 2021-02-10T04:38:11.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: .528614+0000 osd.2 (osd.2) 1933 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:38:11.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: cluster 2021-02-10T04:38:11.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: 2021-02-10T04:38:09.529227 2021-02-10T04:38:11.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1934 2021-02-10T04:38:11.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:38:11.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:38:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: 02-10T04:38:10.302832+0000 2021-02-10T04:38:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: osd.4 (osd.4) 1075 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: 2021-02-10T04:38:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: cluster 2021-02-10T04:38:10.302842+0000 osd.4 2021-02-10T04:38:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: (osd.4) 1076 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: cluster 2021-02-10T04:38:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: 2021 2021-02-10T04:38:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: -02-10T04:38:10.302853+0000 osd.4 2021-02-10T04:38:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: (osd.4) 1077 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: 2021-02-10T04:38:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: cluster 2021-02-10T04:38:11.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: 2021-02-10T04:38:11.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: 2021-02-10T04:38:10.302860+0000 osd.4 (osd. 2021-02-10T04:38:11.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: 4) 1078 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:11.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: audit 2021 2021-02-10T04:38:11.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: -02-10T04:38:10.648307 2021-02-10T04:38:11.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1556 : audit [INF] 2021-02-10T04:38:11.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:38:11.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: cephadm 2021-02-10T04:38:10 2021-02-10T04:38:11.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: .648921+0000 mgr.y (mgr.14140) 700 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:38:11.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: 2021-02-10T04:38:11.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:38:11.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: -02-10T04:38:11.224458 2021-02-10T04:38:11.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2021-02-10T04:38:11.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[35961]: 1557 : cluster [DBG] osdmap e673: 8 total, 8 up, 7 in 2021-02-10T04:38:11.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[29319]: cluster 2021-02-10T04:38:09.528614+0000 osd.2 ( 2021-02-10T04:38:11.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[29319]: osd.2) 1933 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:38:11.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[29319]: cluster 2021-02-10T04:38:09.529227+0000 osd.2 (osd.2) 1934 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:38:11.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:38:11.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[29319]: :38:10.302832+0000 osd.4 (osd.4) 1075 2021-02-10T04:38:11.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:11.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:38:11.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[29319]: 38:10.302842+0000 osd.4 (osd.4 2021-02-10T04:38:11.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[29319]: ) 1076 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:11.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[29319]: cluster 2021-02-10T04:38 2021-02-10T04:38:11.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[29319]: :10.302853+0000 osd.4 (osd.4) 1077 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:11.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:38:11.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[29319]: :38:10.302860+0000 osd.4 2021-02-10T04:38:11.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[29319]: (osd.4) 1078 : cluster [WRN] 2021-02-10T04:38:11.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:11.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[29319]: audit 2021-02-10T04:38:11.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[29319]: 2021-02-10T04:38:10.648307 2021-02-10T04:38:11.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1556 2021-02-10T04:38:11.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[29319]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:38:11.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[29319]: cephadm 2021 2021-02-10T04:38:11.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[29319]: -02-10T04:38:10.648921+0000 2021-02-10T04:38:11.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[29319]: mgr.y (mgr.14140) 700 : 2021-02-10T04:38:11.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[29319]: cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:38:11.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[29319]: cluster 2021-02-10T04:38:11.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:38:11.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[29319]: 10T04:38:11.224458+0000 mon.a (mon.0) 1557 2021-02-10T04:38:11.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:11 smithi085 conmon[29319]: : cluster [DBG] osdmap e673: 8 total, 8 up, 7 in 2021-02-10T04:38:12.222 INFO:tasks.rados.rados.0.smithi085.stdout:2594: finishing write tid 1 to smithi08575397-23 2021-02-10T04:38:12.222 INFO:tasks.rados.rados.0.smithi085.stdout:2594: finishing write tid 2 to smithi08575397-23 2021-02-10T04:38:12.222 INFO:tasks.rados.rados.0.smithi085.stdout:2594: finishing write tid 3 to smithi08575397-23 2021-02-10T04:38:12.222 INFO:tasks.rados.rados.0.smithi085.stdout:2594: finishing write tid 4 to smithi08575397-23 2021-02-10T04:38:12.222 INFO:tasks.rados.rados.0.smithi085.stdout:2594: finishing write tid 5 to smithi08575397-23 2021-02-10T04:38:12.222 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 1274 (ObjNum 902 snap 297 seq_num 902) dirty exists 2021-02-10T04:38:12.222 INFO:tasks.rados.rados.0.smithi085.stdout:2594: left oid 23 (ObjNum 902 snap 297 seq_num 902) 2021-02-10T04:38:12.222 INFO:tasks.rados.rados.0.smithi085.stdout:2594: done (5 left) 2021-02-10T04:38:12.222 INFO:tasks.rados.rados.0.smithi085.stdout:2595: done (4 left) 2021-02-10T04:38:12.222 INFO:tasks.rados.rados.0.smithi085.stdout:2596: write (excl) oid 28 current snap is 297 2021-02-10T04:38:12.223 INFO:tasks.rados.rados.0.smithi085.stdout:2596: seq_num 903 ranges {776721=479501,2024887=693316} 2021-02-10T04:38:12.229 INFO:tasks.rados.rados.0.smithi085.stdout:2596: writing smithi08575397-28 from 776721 to 1256222 tid 1 2021-02-10T04:38:12.238 INFO:tasks.rados.rados.0.smithi085.stdout:2596: writing smithi08575397-28 from 2024887 to 2718203 tid 2 2021-02-10T04:38:12.238 INFO:tasks.rados.rados.0.smithi085.stdout:2596: finishing write tid 1 to smithi08575397-28 2021-02-10T04:38:12.239 INFO:tasks.rados.rados.0.smithi085.stdout:2597: write oid 36 current snap is 297 2021-02-10T04:38:12.239 INFO:tasks.rados.rados.0.smithi085.stdout:2597: seq_num 904 ranges {793528=554750,2083369=436974,2958856=234672} 2021-02-10T04:38:12.247 INFO:tasks.rados.rados.0.smithi085.stdout:2597: writing smithi08575397-36 from 793528 to 1348278 tid 1 2021-02-10T04:38:12.254 INFO:tasks.rados.rados.0.smithi085.stdout:2597: writing smithi08575397-36 from 2083369 to 2520343 tid 2 2021-02-10T04:38:12.258 INFO:tasks.rados.rados.0.smithi085.stdout:2597: writing smithi08575397-36 from 2958856 to 3193528 tid 3 2021-02-10T04:38:12.258 INFO:tasks.rados.rados.0.smithi085.stdout:2598: write oid 45 current snap is 297 2021-02-10T04:38:12.259 INFO:tasks.rados.rados.0.smithi085.stdout:2596: finishing write tid 2 to smithi08575397-28 2021-02-10T04:38:12.259 INFO:tasks.rados.rados.0.smithi085.stdout:2596: finishing write tid 3 to smithi08575397-28 2021-02-10T04:38:12.259 INFO:tasks.rados.rados.0.smithi085.stdout:2596: finishing write tid 4 to smithi08575397-28 2021-02-10T04:38:12.259 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 1268 (ObjNum 903 snap 297 seq_num 903) dirty exists 2021-02-10T04:38:12.259 INFO:tasks.rados.rados.0.smithi085.stdout:2596: left oid 28 (ObjNum 903 snap 297 seq_num 903) 2021-02-10T04:38:12.259 INFO:tasks.rados.rados.0.smithi085.stdout:2597: finishing write tid 1 to smithi08575397-36 2021-02-10T04:38:12.259 INFO:tasks.rados.rados.0.smithi085.stdout:2597: finishing write tid 2 to smithi08575397-36 2021-02-10T04:38:12.259 INFO:tasks.rados.rados.0.smithi085.stdout:2598: seq_num 905 ranges {410335=746352,1741851=696985,3102632=107703} 2021-02-10T04:38:12.265 INFO:tasks.rados.rados.0.smithi085.stdout:2598: writing smithi08575397-45 from 410335 to 1156687 tid 1 2021-02-10T04:38:12.274 INFO:tasks.rados.rados.0.smithi085.stdout:2598: writing smithi08575397-45 from 1741851 to 2438836 tid 2 2021-02-10T04:38:12.279 INFO:tasks.rados.rados.0.smithi085.stdout:2598: writing smithi08575397-45 from 3102632 to 3210335 tid 3 2021-02-10T04:38:12.279 INFO:tasks.rados.rados.0.smithi085.stdout:2596: done (6 left) 2021-02-10T04:38:12.279 INFO:tasks.rados.rados.0.smithi085.stdout:2599: copy_from oid 21 from oid 29 current snap is 297 2021-02-10T04:38:12.279 INFO:tasks.rados.rados.0.smithi085.stdout:2597: finishing write tid 3 to smithi08575397-36 2021-02-10T04:38:12.279 INFO:tasks.rados.rados.0.smithi085.stdout:2597: finishing write tid 4 to smithi08575397-36 2021-02-10T04:38:12.279 INFO:tasks.rados.rados.0.smithi085.stdout:2597: finishing write tid 5 to smithi08575397-36 2021-02-10T04:38:12.279 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 1278 (ObjNum 904 snap 297 seq_num 904) dirty exists 2021-02-10T04:38:12.279 INFO:tasks.rados.rados.0.smithi085.stdout:2597: left oid 36 (ObjNum 904 snap 297 seq_num 904) 2021-02-10T04:38:12.279 INFO:tasks.rados.rados.0.smithi085.stdout:2598: finishing write tid 1 to smithi08575397-45 2021-02-10T04:38:12.279 INFO:tasks.rados.rados.0.smithi085.stdout:2598: finishing write tid 2 to smithi08575397-45 2021-02-10T04:38:12.279 INFO:tasks.rados.rados.0.smithi085.stdout:2597: done (6 left) 2021-02-10T04:38:12.280 INFO:tasks.rados.rados.0.smithi085.stdout:2600: snap_remove snap 296 2021-02-10T04:38:12.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:12 smithi164 conmon[33677]: debug 2021-02-10T04:38:12.338+0000 7fbec19a1700 -1 osd.4 674 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:38:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:12 smithi164 conmon[29288]: cluster 2021-02-10T04:38:11.306381+0000 osd.4 (osd.4) 1079 : cluster [WRN] 2021-02-10T04:38:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:12 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:12 smithi164 conmon[29288]: cluster 2021-02-10T04:38:11.306391+0000 osd.4 (osd.4) 1080 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:12 smithi164 conmon[29288]: cluster 2021-02-10T04:38 2021-02-10T04:38:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:12 smithi164 conmon[29288]: :11.306397+0000 osd.4 (osd.4) 1081 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:12 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:38:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:12 smithi164 conmon[29288]: :38:11.306403+0000 osd.4 (osd.4) 1082 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:12 smithi164 conmon[29288]: 2021-02-10T04:38:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:12 smithi164 conmon[29288]: cluster 2021-02-10T04:38:12 2021-02-10T04:38:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:12 smithi164 conmon[29288]: .152412+0000 mgr.y (mgr.14140) 701 : cluster [DBG] pgmap v1160: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 369 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 2.2 MiB/s rd, 309 KiB/s wr, 4 op/s 2021-02-10T04:38:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:12 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:38:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:12 smithi164 conmon[29288]: -10T04:38:12.221296+0000 mon.a (mon. 2021-02-10T04:38:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:12 smithi164 conmon[29288]: 0) 1558 : cluster [DBG] osdmap e674: 8 total, 8 up, 7 in 2021-02-10T04:38:12.807 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[35961]: cluster 2021-02-10T04:38:11.306381+0000 2021-02-10T04:38:12.807 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[35961]: osd.4 (osd.4) 1079 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:12.807 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[35961]: cluster 2021-02-10T04:38:11.306391 2021-02-10T04:38:12.807 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1080 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:12.807 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[35961]: cluster 2021-02-10T04:38:11.306397+0000 osd.4 (osd.4) 1081 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:12.807 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[35961]: cluster 2021-02-10T04:38:11.306403+0000 osd.4 (osd. 2021-02-10T04:38:12.807 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[35961]: 4) 1082 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:12.807 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[35961]: cluster 2021-02-10T04:38:12.807 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[35961]: 2021-02-10T04:38:12.807 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[35961]: 2021 2021-02-10T04:38:12.807 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[35961]: -02-10T04: 2021-02-10T04:38:12.808 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[35961]: 38:12.152412+0000 mgr.y (mgr.14140) 701 : cluster [DBG] pgmap v1160: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 369 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 2.2 MiB/s rd, 309 KiB/s wr, 4 op/s 2021-02-10T04:38:12.808 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:38:12.808 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[35961]: 02-10T04:38:12.221296+0000 mon.a (mon 2021-02-10T04:38:12.808 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[35961]: .0) 1558 : cluster [DBG] 2021-02-10T04:38:12.808 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[35961]: osdmap e674: 8 total, 8 up, 7 in 2021-02-10T04:38:12.808 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[29319]: cluster 2021-02-10T04:38:12.808 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[29319]: 2021-02-10T04:38:12.808 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[29319]: 2021-02-10T04:38:11.306381 2021-02-10T04:38:12.808 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1079 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:12.808 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[29319]: cluster 2021-02-10T04:38:11. 2021-02-10T04:38:12.808 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[29319]: 306391+0000 osd.4 (osd.4) 1080 : cluster 2021-02-10T04:38:12.809 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:12.809 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[29319]: cluster 2021-02-10T04:38:11.306397 2021-02-10T04:38:12.809 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1081 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:12.809 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[29319]: 2021-02-10T04:38:12.809 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[29319]: cluster 2021-02-10T04:38:12.809 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:38:12.809 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[29319]: 10T04:38:11.306403+0000 osd.4 (osd. 2021-02-10T04:38:12.809 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[29319]: 4) 1082 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:12.809 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[29319]: 2021-02-10T04:38:12.809 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:38:12.809 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[29319]: 02-10T04:38:12 2021-02-10T04:38:12.809 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[29319]: .152412+0000 mgr.y (mgr. 2021-02-10T04:38:12.809 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[29319]: 14140) 701 : cluster [DBG] pgmap v1160: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 369 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 2.2 MiB/s rd, 309 KiB/s wr, 4 op/s 2021-02-10T04:38:12.810 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[29319]: cluster 2021-02-10T04:38:12.810 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[29319]: 2021-02-10T04:38 2021-02-10T04:38:12.810 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[29319]: :12.221296+0000 mon.a (mon.0) 1558 : cluster [DBG] 2021-02-10T04:38:12.810 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:12 smithi085 conmon[29319]: osdmap e674: 8 total, 8 up, 7 in 2021-02-10T04:38:13.219 INFO:tasks.rados.rados.0.smithi085.stdout:2600: done (6 left) 2021-02-10T04:38:13.270 INFO:tasks.rados.rados.0.smithi085.stdout:2601: snap_create 2021-02-10T04:38:13.270 INFO:tasks.rados.rados.0.smithi085.stdout:2598: finishing write tid 3 to smithi08575397-45 2021-02-10T04:38:13.270 INFO:tasks.rados.rados.0.smithi085.stdout:2598: finishing write tid 4 to smithi08575397-45 2021-02-10T04:38:13.270 INFO:tasks.rados.rados.0.smithi085.stdout:2598: finishing write tid 5 to smithi08575397-45 2021-02-10T04:38:13.270 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 45 v 891 (ObjNum 905 snap 297 seq_num 905) dirty exists 2021-02-10T04:38:13.270 INFO:tasks.rados.rados.0.smithi085.stdout:2598: left oid 45 (ObjNum 905 snap 297 seq_num 905) 2021-02-10T04:38:13.270 INFO:tasks.rados.rados.0.smithi085.stdout:2599: finishing copy_from to smithi08575397-21 2021-02-10T04:38:13.270 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 633 (ObjNum 8 snap 0 seq_num 8) dirty exists 2021-02-10T04:38:13.271 INFO:tasks.rados.rados.0.smithi085.stdout:2599: finishing copy_from racing read to smithi08575397-21 2021-02-10T04:38:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:13 smithi164 conmon[29288]: cluster 2021-02-10T04:38:12.339583+0000 osd.4 ( 2021-02-10T04:38:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:13 smithi164 conmon[29288]: osd.4) 1083 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:13 smithi164 conmon[29288]: cluster 2021-02-10T04:38:12.339592+0000 osd.4 (osd.4) 1084 : cluster 2021-02-10T04:38:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:13 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:13 smithi164 conmon[29288]: cluster 2021-02-10T04:38:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:13 smithi164 conmon[29288]: 2021-02-10T04:38:12.339599+0000 2021-02-10T04:38:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:13 smithi164 conmon[29288]: osd.4 (osd.4) 1085 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:13 smithi164 conmon[29288]: 2021-02-10T04:38:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:13 smithi164 conmon[29288]: cluster 2021-02-10T04:38:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:13 smithi164 conmon[29288]: 2021-02-10T04:38:12.339605+0000 2021-02-10T04:38:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:13 smithi164 conmon[29288]: osd.4 (osd.4) 1086 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:13.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:13 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:38:13.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:13 smithi164 conmon[29288]: 04:38:13.217304+0000 mon.a (mon.0) 1559 : cluster [DBG] osdmap e675: 8 total, 8 up, 7 in 2021-02-10T04:38:13.645 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:13 smithi164 conmon[33677]: debug 2021-02-10T04:38:13.302+0000 7fbec19a1700 -1 osd.4 675 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:38:13.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:13 smithi085 conmon[35961]: cluster 2021-02-10T04:38:12.339583+0000 2021-02-10T04:38:13.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:13 smithi085 conmon[35961]: osd.4 (osd.4) 1083 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:13.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:13 smithi085 conmon[35961]: cluster 2021-02-10T04:38:12 2021-02-10T04:38:13.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:13 smithi085 conmon[35961]: .339592+0000 osd.4 (osd.4) 1084 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:13 smithi085 conmon[35961]: cluster 2021-02-10T04:38:12 2021-02-10T04:38:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:13 smithi085 conmon[35961]: .339599+0000 osd.4 (osd.4) 1085 : cluster [WRN] 2021-02-10T04:38:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:13 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:13 smithi085 conmon[35961]: cluster 2021-02-10T04:38:12.339605+0000 2021-02-10T04:38:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:13 smithi085 conmon[35961]: osd.4 (osd.4) 1086 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:13 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:38:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:13 smithi085 conmon[35961]: 02-10T04:38:13.217304+0000 mon.a (mon 2021-02-10T04:38:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:13 smithi085 conmon[35961]: .0) 1559 : cluster [DBG] osdmap e675: 8 total, 8 up, 7 in 2021-02-10T04:38:13.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:13 smithi085 conmon[29319]: cluster 2021-02-10T04:38:12.339583+0000 2021-02-10T04:38:13.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:13 smithi085 conmon[29319]: osd.4 (osd.4) 1083 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:13.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:13 smithi085 conmon[29319]: cluster 2021-02-10T04:38:12.339592+0000 osd.4 (osd.4) 1084 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:13.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:13 smithi085 conmon[29319]: cluster 2021-02-10T04:38:12.339599+0000 osd.4 (osd.4) 1085 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:13.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:13 smithi085 conmon[29319]: cluster 2021-02-10T04:38:12.339605+0000 osd.4 (osd.4) 1086 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:13.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:13 smithi085 conmon[29319]: cluster 2021-02-10T04:38:13.217304+0000 mon.a (mon.0) 1559 : cluster [DBG] osdmap e675: 8 total, 8 up, 7 in 2021-02-10T04:38:14.223 INFO:tasks.rados.rados.0.smithi085.stdout:2598: done (6 left) 2021-02-10T04:38:14.224 INFO:tasks.rados.rados.0.smithi085.stdout:2599: done (5 left) 2021-02-10T04:38:14.224 INFO:tasks.rados.rados.0.smithi085.stdout:2601: done (4 left) 2021-02-10T04:38:14.224 INFO:tasks.rados.rados.0.smithi085.stdout:2602: snap_remove snap 280 2021-02-10T04:38:14.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:14 smithi164 conmon[33677]: debug 2021-02-10T04:38:14.294+0000 7fbec19a1700 -1 osd.4 676 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:38:15.368 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:14 smithi164 conmon[29288]: cluster 2021-02-10T04:38:13.302786+0000 osd.4 ( 2021-02-10T04:38:15.368 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:14 smithi164 conmon[29288]: osd.4) 1087 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:15.368 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:14 smithi164 conmon[29288]: cluster 2021-02-10T04:38:13.302804+0000 osd.4 (osd.4) 1088 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:15.368 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:14 smithi164 conmon[29288]: cluster 2021-02-10T04:38: 2021-02-10T04:38:15.369 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:14 smithi164 conmon[29288]: 13.302811+0000 osd.4 (osd.4) 1089 : cluster 2021-02-10T04:38:15.369 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:14 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:15.369 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:14 smithi164 conmon[29288]: cluster 2021-02-10T04:38:15.369 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:14 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:38:15.369 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:14 smithi164 conmon[29288]: 38:13.302817+0000 osd.4 (osd.4) 2021-02-10T04:38:15.370 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:14 smithi164 conmon[29288]: 1090 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:15.370 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:14 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:38:15.370 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:14 smithi164 conmon[29288]: -02-10T04:38:14.153023+0000 mgr.y 2021-02-10T04:38:15.370 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:14 smithi164 conmon[29288]: (mgr.14140 2021-02-10T04:38:15.370 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:14 smithi164 conmon[29288]: ) 702 : cluster [DBG] pgmap v1163: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 365 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 2.2 MiB/s rd, 900 KiB/s wr, 5 op/s 2021-02-10T04:38:15.370 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:14 smithi164 conmon[29288]: 2021-02-10T04:38:15.371 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:14 smithi164 conmon[29288]: cluster 2021-02-10T04:38:15.371 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:14 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:38:15.371 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:14 smithi164 conmon[29288]: 38:14.224299+0000 mon.a (mon.0 2021-02-10T04:38:15.371 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:14 smithi164 conmon[29288]: ) 1560 : cluster [DBG] osdmap e676: 8 total, 8 up, 7 in 2021-02-10T04:38:15.375 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[29319]: cluster 2021-02-10T04:38:15.375 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:38:15.375 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[29319]: 04:38:13.302786+0000 osd.4 (osd.4) 1087 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:15.375 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[29319]: cluster 2021-02-10T04:38:15.375 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:38:15.375 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[29319]: 04:38:13.302804+0000 osd.4 (osd.4) 1088 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:15.376 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[29319]: cluster 2021-02-10T04:38:15.376 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[29319]: 2021-02-10T04:38:13.302811 2021-02-10T04:38:15.376 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1089 : cluster 2021-02-10T04:38:15.376 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:15.376 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:38:15.376 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[29319]: -10T04:38:13.302817+0000 osd.4 2021-02-10T04:38:15.376 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[29319]: (osd.4) 1090 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:15.377 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[29319]: 2021-02-10T04:38:15.377 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[29319]: cluster 2021-02-10T04:38:15.377 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[29319]: 2021-02-10T04:38:14.153023+0000 mgr.y 2021-02-10T04:38:15.377 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[29319]: (mgr.14140) 702 : cluster [DBG] pgmap v1163: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 365 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 2.2 MiB/s rd, 900 KiB/s wr, 5 op/s 2021-02-10T04:38:15.377 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[29319]: 2021-02-10T04:38:15.377 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:38:15.377 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[29319]: -10T04:38:14.224299+0000 mon.a (mon.0) 1560 : 2021-02-10T04:38:15.377 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[29319]: cluster [DBG] osdmap e676: 8 total, 8 up, 7 in 2021-02-10T04:38:15.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[35961]: cluster 2021-02-10T04:38:13. 2021-02-10T04:38:15.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[35961]: 302786+0000 osd.4 (osd.4) 1087 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:15.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:38:15.377 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[35961]: -10T04:38:13.302804+0000 osd.4 (osd.4) 1088 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:15.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:38:15.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[35961]: -02-10T04:38:13.302811+0000 2021-02-10T04:38:15.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[35961]: osd.4 (osd.4) 1089 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:15.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:38:15.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[35961]: -02-10T04:38:13.302817+0000 osd.4 2021-02-10T04:38:15.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[35961]: (osd.4) 1090 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:15.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:38:15.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[35961]: -02-10T04:38:14.153023+0000 mgr.y ( 2021-02-10T04:38:15.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[35961]: mgr.14140) 702 : cluster [DBG] pgmap v1163: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 365 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 2.2 MiB/s rd, 900 KiB/s wr, 5 op/s 2021-02-10T04:38:15.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:38:15.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[35961]: 02-10T04:38:14.224299+0000 2021-02-10T04:38:15.378 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:14 smithi085 conmon[35961]: mon.a (mon.0) 1560 : cluster [DBG] osdmap e676: 8 total, 8 up, 7 in 2021-02-10T04:38:15.378 INFO:tasks.rados.rados.0.smithi085.stdout:2602: done (4 left) 2021-02-10T04:38:15.379 INFO:tasks.rados.rados.0.smithi085.stdout:2603: write (excl) oid 31 current snap is 298 2021-02-10T04:38:15.379 INFO:tasks.rados.rados.0.smithi085.stdout:2603: seq_num 906 ranges {427142=421601,1400333=440643,2613702=450326} 2021-02-10T04:38:15.379 INFO:tasks.rados.rados.0.smithi085.stdout:2603: writing smithi08575397-31 from 427142 to 848743 tid 1 2021-02-10T04:38:15.379 INFO:tasks.rados.rados.0.smithi085.stdout:2603: writing smithi08575397-31 from 1400333 to 1840976 tid 2 2021-02-10T04:38:15.379 INFO:tasks.rados.rados.0.smithi085.stdout:2603: writing smithi08575397-31 from 2613702 to 3064028 tid 3 2021-02-10T04:38:15.379 INFO:tasks.rados.rados.0.smithi085.stdout:2603: finishing write tid 1 to smithi08575397-31 2021-02-10T04:38:15.379 INFO:tasks.rados.rados.0.smithi085.stdout:2603: finishing write tid 2 to smithi08575397-31 2021-02-10T04:38:15.379 INFO:tasks.rados.rados.0.smithi085.stdout:2604: copy_from oid 13 from oid 3 current snap is 298 2021-02-10T04:38:15.379 INFO:tasks.rados.rados.0.smithi085.stdout:2605: delete oid 6 current snap is 298 2021-02-10T04:38:15.379 INFO:tasks.rados.rados.0.smithi085.stdout:2605: done (6 left) 2021-02-10T04:38:15.379 INFO:tasks.rados.rados.0.smithi085.stdout:2606: copy_from oid 28 from oid 43 current snap is 298 2021-02-10T04:38:15.379 INFO:tasks.rados.rados.0.smithi085.stdout:2607: snap_create 2021-02-10T04:38:15.380 INFO:tasks.rados.rados.0.smithi085.stdout:2604: finishing copy_from to smithi08575397-13 2021-02-10T04:38:15.380 INFO:tasks.rados.rados.0.smithi085.stdout:2604: got expected ENOENT (src dne) 2021-02-10T04:38:15.380 INFO:tasks.rados.rados.0.smithi085.stdout:2604: finishing copy_from racing read to smithi08575397-13 2021-02-10T04:38:15.380 INFO:tasks.rados.rados.0.smithi085.stdout:2603: finishing write tid 3 to smithi08575397-31 2021-02-10T04:38:15.380 INFO:tasks.rados.rados.0.smithi085.stdout:2603: finishing write tid 4 to smithi08575397-31 2021-02-10T04:38:15.380 INFO:tasks.rados.rados.0.smithi085.stdout:2603: finishing write tid 5 to smithi08575397-31 2021-02-10T04:38:15.380 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 895 (ObjNum 906 snap 298 seq_num 906) dirty exists 2021-02-10T04:38:15.380 INFO:tasks.rados.rados.0.smithi085.stdout:2603: left oid 31 (ObjNum 906 snap 298 seq_num 906) 2021-02-10T04:38:15.380 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:38:14 smithi164 conmon[40892]: debug 2021-02-10T04:38:15.380 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:38:14 smithi164 conmon[40892]: 2021-02-10T04:38:14.813+0000 7f3032cc0700 -1 osd.6 pg_epoch: 676 pg[7.c( v 676'679 (0'0,676'679] local-lis/les=93/94 n=12 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [6,3,7] r=0 lpr=93 luod=674'677 crt=676'679 lcod 674'676 mlcod 674'676 active+clean+snaptrim trimq=[1fc~1,206~1,20d~1,21b~1,21d~1,222~1,225~1,238~1] ps=[1d3~1,1d5~1,1d8~2,1db~2,1df~1,1e1~2,1e4~1,1e9~1,1eb~1,1ee~3,1f2~3,1f7~2,201~1,205~1]] removing snap head 2021-02-10T04:38:15.380 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:38:14 smithi164 conmon[40892]: 2021-02-10T04:38:15.428 INFO:tasks.rados.rados.0.smithi085.stdout:2606: finishing copy_from to smithi08575397-28 2021-02-10T04:38:15.428 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 1169 (ObjNum 892 snap 289 seq_num 892) dirty exists 2021-02-10T04:38:15.428 INFO:tasks.rados.rados.0.smithi085.stdout:2606: finishing copy_from racing read to smithi08575397-28 2021-02-10T04:38:15.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:15 smithi164 conmon[29288]: cluster 2021-02-10T04:38:14.295357+0000 osd.4 (osd. 2021-02-10T04:38:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:15 smithi164 conmon[29288]: 4) 1091 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:15 smithi164 conmon[29288]: cluster 2021-02-10T04:38:14.295366+0000 osd.4 (osd.4) 1092 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:15 smithi164 conmon[29288]: cluster 2021-02-10T04:38:14.295373 2021-02-10T04:38:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:15 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1093 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:15 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:38:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:15 smithi164 conmon[29288]: 04:38:14.295380+0000 osd.4 (osd.4) 1094 : cluster [WRN] 2021-02-10T04:38:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:15 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:15 smithi164 conmon[29288]: cluster 2021-02-10T04:38:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:15 smithi164 conmon[29288]: 2021-02-10T04:38:15.219805+0000 mon.a (mon.0) 1561 : 2021-02-10T04:38:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:15 smithi164 conmon[29288]: cluster [DBG] osdmap e677: 8 total, 8 up, 7 in 2021-02-10T04:38:15.645 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:15 smithi164 conmon[33677]: debug 2021-02-10T04:38:15.292+0000 7fbec19a1700 -1 osd.4 677 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:38:15.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:15 smithi085 conmon[29319]: cluster 2021-02-10T04:38:14.295357+0000 osd.4 (osd. 2021-02-10T04:38:15.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:15 smithi085 conmon[29319]: 4) 1091 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:15.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:15 smithi085 conmon[29319]: cluster 2021-02-10T04:38:14.295366+0000 osd.4 (osd.4) 1092 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:15.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:15 smithi085 conmon[29319]: cluster 2021-02-10T04:38:14.295373+0000 2021-02-10T04:38:15.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:15 smithi085 conmon[29319]: osd.4 (osd.4) 1093 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:15.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:15 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:38:15.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:15 smithi085 conmon[29319]: 38:14.295380+0000 osd.4 (osd.4) 1094 2021-02-10T04:38:15.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:15 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:15.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:15 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:38:15.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:15 smithi085 conmon[29319]: 38:15.219805+0000 mon.a (mon.0) 1561 : 2021-02-10T04:38:15.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:15 smithi085 conmon[29319]: cluster [DBG] osdmap e677: 8 total, 8 up, 7 in 2021-02-10T04:38:15.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:15 smithi085 conmon[35961]: cluster 2021-02-10T04:38:15.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:15 smithi085 conmon[35961]: 2021-02-10T04:38:15.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:15 smithi085 conmon[35961]: 2021 2021-02-10T04:38:15.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:15 smithi085 conmon[35961]: -02-10T04:38:14.295357+0000 2021-02-10T04:38:15.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:15 smithi085 conmon[35961]: osd.4 (osd.4) 1091 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:15.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:15 smithi085 conmon[35961]: cluster 2021-02-10T04:38:14.295366+0000 osd.4 ( 2021-02-10T04:38:15.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:15 smithi085 conmon[35961]: osd.4) 1092 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:15.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:15 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:38:15.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:15 smithi085 conmon[35961]: T04:38:14.295373+0000 osd.4 (osd 2021-02-10T04:38:15.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:15 smithi085 conmon[35961]: .4) 1093 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:15.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:15 smithi085 conmon[35961]: cluster 2021-02-10T04:38:15.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:15 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:38:15.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:15 smithi085 conmon[35961]: :38:14.295380+0000 osd.4 (osd.4) 1094 : cluster 2021-02-10T04:38:15.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:15 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:15.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:15 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:38:15.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:15 smithi085 conmon[35961]: -02-10T04:38: 2021-02-10T04:38:15.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:15 smithi085 conmon[35961]: 15.219805+0000 mon.a (mon 2021-02-10T04:38:15.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:15 smithi085 conmon[35961]: .0) 1561 : cluster [DBG] 2021-02-10T04:38:15.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:15 smithi085 conmon[35961]: osdmap e677: 8 total, 8 up, 7 in 2021-02-10T04:38:16.224 INFO:tasks.rados.rados.0.smithi085.stdout:2603: done (7 left) 2021-02-10T04:38:16.224 INFO:tasks.rados.rados.0.smithi085.stdout:2604: done (6 left) 2021-02-10T04:38:16.225 INFO:tasks.rados.rados.0.smithi085.stdout:2606: done (5 left) 2021-02-10T04:38:16.225 INFO:tasks.rados.rados.0.smithi085.stdout:2607: done (4 left) 2021-02-10T04:38:16.225 INFO:tasks.rados.rados.0.smithi085.stdout:2608: delete oid 10 current snap is 299 2021-02-10T04:38:16.231 INFO:tasks.rados.rados.0.smithi085.stdout:2608: done (4 left) 2021-02-10T04:38:16.231 INFO:tasks.rados.rados.0.smithi085.stdout:2609: write oid 8 current snap is 299 2021-02-10T04:38:16.231 INFO:tasks.rados.rados.0.smithi085.stdout:2609: seq_num 909 ranges {477563=647348,1859426=587970,2979618=297945} 2021-02-10T04:38:16.237 INFO:tasks.rados.rados.0.smithi085.stdout:2609: writing smithi08575397-8 from 477563 to 1124911 tid 1 2021-02-10T04:38:16.245 INFO:tasks.rados.rados.0.smithi085.stdout:2609: writing smithi08575397-8 from 1859426 to 2447396 tid 2 2021-02-10T04:38:16.250 INFO:tasks.rados.rados.0.smithi085.stdout:2609: writing smithi08575397-8 from 2979618 to 3277563 tid 3 2021-02-10T04:38:16.251 INFO:tasks.rados.rados.0.smithi085.stdout:2610: rollback oid 46 current snap is 299 2021-02-10T04:38:16.251 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 46 to 287 2021-02-10T04:38:16.251 INFO:tasks.rados.rados.0.smithi085.stdout:2609: finishing write tid 1 to smithi08575397-8 2021-02-10T04:38:16.251 INFO:tasks.rados.rados.0.smithi085.stdout:2609: finishing write tid 2 to smithi08575397-8 2021-02-10T04:38:16.251 INFO:tasks.rados.rados.0.smithi085.stdout:2611: write oid 11 current snap is 299 2021-02-10T04:38:16.251 INFO:tasks.rados.rados.0.smithi085.stdout:2611: seq_num 910 ranges {494370=722597,1917908=447981,3007041=287329} 2021-02-10T04:38:16.260 INFO:tasks.rados.rados.0.smithi085.stdout:2611: writing smithi08575397-11 from 494370 to 1216967 tid 1 2021-02-10T04:38:16.267 INFO:tasks.rados.rados.0.smithi085.stdout:2611: writing smithi08575397-11 from 1917908 to 2365889 tid 2 2021-02-10T04:38:16.272 INFO:tasks.rados.rados.0.smithi085.stdout:2611: writing smithi08575397-11 from 3007041 to 3294370 tid 3 2021-02-10T04:38:16.273 INFO:tasks.rados.rados.0.smithi085.stdout:2612: write oid 28 current snap is 299 2021-02-10T04:38:16.273 INFO:tasks.rados.rados.0.smithi085.stdout:2609: finishing write tid 3 to smithi08575397-8 2021-02-10T04:38:16.273 INFO:tasks.rados.rados.0.smithi085.stdout:2612: seq_num 911 ranges {511177=797846,1976390=591639,3034464=276713} 2021-02-10T04:38:16.281 INFO:tasks.rados.rados.0.smithi085.stdout:2612: writing smithi08575397-28 from 511177 to 1309023 tid 1 2021-02-10T04:38:16.290 INFO:tasks.rados.rados.0.smithi085.stdout:2612: writing smithi08575397-28 from 1976390 to 2568029 tid 2 2021-02-10T04:38:16.294 INFO:tasks.rados.rados.0.smithi085.stdout:2612: writing smithi08575397-28 from 3034464 to 3311177 tid 3 2021-02-10T04:38:16.295 INFO:tasks.rados.rados.0.smithi085.stdout:2613: write (excl) oid 30 current snap is 299 2021-02-10T04:38:16.295 INFO:tasks.rados.rados.0.smithi085.stdout:2610: finishing rollback tid 1 to smithi08575397-46 2021-02-10T04:38:16.295 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 1083 (ObjNum 82847952 snap 22030 seq_num 1215200496) dirty dne 2021-02-10T04:38:16.295 INFO:tasks.rados.rados.0.smithi085.stdout:2609: finishing write tid 4 to smithi08575397-8 2021-02-10T04:38:16.295 INFO:tasks.rados.rados.0.smithi085.stdout:2609: finishing write tid 5 to smithi08575397-8 2021-02-10T04:38:16.295 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 1329 (ObjNum 909 snap 299 seq_num 909) dirty exists 2021-02-10T04:38:16.295 INFO:tasks.rados.rados.0.smithi085.stdout:2609: left oid 8 (ObjNum 909 snap 299 seq_num 909) 2021-02-10T04:38:16.295 INFO:tasks.rados.rados.0.smithi085.stdout:2611: finishing write tid 1 to smithi08575397-11 2021-02-10T04:38:16.295 INFO:tasks.rados.rados.0.smithi085.stdout:2611: finishing write tid 2 to smithi08575397-11 2021-02-10T04:38:16.296 INFO:tasks.rados.rados.0.smithi085.stdout:2611: finishing write tid 3 to smithi08575397-11 2021-02-10T04:38:16.296 INFO:tasks.rados.rados.0.smithi085.stdout:2611: finishing write tid 4 to smithi08575397-11 2021-02-10T04:38:16.296 INFO:tasks.rados.rados.0.smithi085.stdout:2611: finishing write tid 5 to smithi08575397-11 2021-02-10T04:38:16.296 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 1375 (ObjNum 910 snap 299 seq_num 910) dirty exists 2021-02-10T04:38:16.296 INFO:tasks.rados.rados.0.smithi085.stdout:2611: left oid 11 (ObjNum 910 snap 299 seq_num 910) 2021-02-10T04:38:16.296 INFO:tasks.rados.rados.0.smithi085.stdout:2612: finishing write tid 1 to smithi08575397-28 2021-02-10T04:38:16.296 INFO:tasks.rados.rados.0.smithi085.stdout:2612: finishing write tid 2 to smithi08575397-28 2021-02-10T04:38:16.296 INFO:tasks.rados.rados.0.smithi085.stdout:2613: seq_num 912 ranges {527984=473095,1634872=451650,2661887=634311} 2021-02-10T04:38:16.300 INFO:tasks.rados.rados.0.smithi085.stdout:2613: writing smithi08575397-30 from 527984 to 1001079 tid 1 2021-02-10T04:38:16.307 INFO:tasks.rados.rados.0.smithi085.stdout:2613: writing smithi08575397-30 from 1634872 to 2086522 tid 2 2021-02-10T04:38:16.315 INFO:tasks.rados.rados.0.smithi085.stdout:2613: writing smithi08575397-30 from 2661887 to 3296198 tid 3 2021-02-10T04:38:16.315 INFO:tasks.rados.rados.0.smithi085.stdout:2609: done (8 left) 2021-02-10T04:38:16.315 INFO:tasks.rados.rados.0.smithi085.stdout:2610: done (7 left) 2021-02-10T04:38:16.316 INFO:tasks.rados.rados.0.smithi085.stdout:2611: done (6 left) 2021-02-10T04:38:16.316 INFO:tasks.rados.rados.0.smithi085.stdout:2614: read oid 22 snap 274 2021-02-10T04:38:16.316 INFO:tasks.rados.rados.0.smithi085.stdout:2614: expect deleted 2021-02-10T04:38:16.316 INFO:tasks.rados.rados.0.smithi085.stdout:2615: rollback oid 43 current snap is 299 2021-02-10T04:38:16.316 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 43 to 290 2021-02-10T04:38:16.316 INFO:tasks.rados.rados.0.smithi085.stdout:2612: finishing write tid 3 to smithi08575397-28 2021-02-10T04:38:16.316 INFO:tasks.rados.rados.0.smithi085.stdout:2612: finishing write tid 4 to smithi08575397-28 2021-02-10T04:38:16.316 INFO:tasks.rados.rados.0.smithi085.stdout:2612: finishing write tid 5 to smithi08575397-28 2021-02-10T04:38:16.316 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 1272 (ObjNum 911 snap 299 seq_num 911) dirty exists 2021-02-10T04:38:16.316 INFO:tasks.rados.rados.0.smithi085.stdout:2612: left oid 28 (ObjNum 911 snap 299 seq_num 911) 2021-02-10T04:38:16.316 INFO:tasks.rados.rados.0.smithi085.stdout:2612: done (7 left) 2021-02-10T04:38:16.316 INFO:tasks.rados.rados.0.smithi085.stdout:2616: write (excl) oid 23 current snap is 299 2021-02-10T04:38:16.316 INFO:tasks.rados.rados.0.smithi085.stdout:2613: finishing write tid 1 to smithi08575397-30 2021-02-10T04:38:16.317 INFO:tasks.rados.rados.0.smithi085.stdout:2613: finishing write tid 2 to smithi08575397-30 2021-02-10T04:38:16.317 INFO:tasks.rados.rados.0.smithi085.stdout:2616: seq_num 913 ranges {544791=664697,1693354=595308,2689310=445583} 2021-02-10T04:38:16.323 INFO:tasks.rados.rados.0.smithi085.stdout:2616: writing smithi08575397-23 from 544791 to 1209488 tid 1 2021-02-10T04:38:16.329 INFO:tasks.rados.rados.0.smithi085.stdout:2616: writing smithi08575397-23 from 1693354 to 2288662 tid 2 2021-02-10T04:38:16.335 INFO:tasks.rados.rados.0.smithi085.stdout:2616: writing smithi08575397-23 from 2689310 to 3134893 tid 3 2021-02-10T04:38:16.335 INFO:tasks.rados.rados.0.smithi085.stdout:2617: write oid 32 current snap is 299 2021-02-10T04:38:16.335 INFO:tasks.rados.rados.0.smithi085.stdout:2615: finishing rollback tid 0 to smithi08575397-43 2021-02-10T04:38:16.335 INFO:tasks.rados.rados.0.smithi085.stdout:2615: finishing rollback tid 1 to smithi08575397-43 2021-02-10T04:38:16.335 INFO:tasks.rados.rados.0.smithi085.stdout:2613: finishing write tid 3 to smithi08575397-30 2021-02-10T04:38:16.335 INFO:tasks.rados.rados.0.smithi085.stdout:2615: finishing rollback tid 2 to smithi08575397-43 2021-02-10T04:38:16.336 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 1177 (ObjNum 892 snap 289 seq_num 892) dirty exists 2021-02-10T04:38:16.336 INFO:tasks.rados.rados.0.smithi085.stdout:2613: finishing write tid 4 to smithi08575397-30 2021-02-10T04:38:16.336 INFO:tasks.rados.rados.0.smithi085.stdout:2613: finishing write tid 5 to smithi08575397-30 2021-02-10T04:38:16.336 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 1111 (ObjNum 912 snap 299 seq_num 912) dirty exists 2021-02-10T04:38:16.336 INFO:tasks.rados.rados.0.smithi085.stdout:2613: left oid 30 (ObjNum 912 snap 299 seq_num 912) 2021-02-10T04:38:16.336 INFO:tasks.rados.rados.0.smithi085.stdout:2616: finishing write tid 1 to smithi08575397-23 2021-02-10T04:38:16.336 INFO:tasks.rados.rados.0.smithi085.stdout:2617: seq_num 914 ranges {561598=739946,1751836=455319,2716733=644865} 2021-02-10T04:38:16.343 INFO:tasks.rados.rados.0.smithi085.stdout:2617: writing smithi08575397-32 from 561598 to 1301544 tid 1 2021-02-10T04:38:16.348 INFO:tasks.rados.rados.0.smithi085.stdout:2617: writing smithi08575397-32 from 1751836 to 2207155 tid 2 2021-02-10T04:38:16.355 INFO:tasks.rados.rados.0.smithi085.stdout:2617: writing smithi08575397-32 from 2716733 to 3361598 tid 3 2021-02-10T04:38:16.356 INFO:tasks.rados.rados.0.smithi085.stdout:2613: done (8 left) 2021-02-10T04:38:16.356 INFO:tasks.rados.rados.0.smithi085.stdout:2614: done (7 left) 2021-02-10T04:38:16.356 INFO:tasks.rados.rados.0.smithi085.stdout:2615: done (6 left) 2021-02-10T04:38:16.356 INFO:tasks.rados.rados.0.smithi085.stdout:2618: read oid 25 snap -1 2021-02-10T04:38:16.356 INFO:tasks.rados.rados.0.smithi085.stdout:2618: expect (ObjNum 885 snap 287 seq_num 885) 2021-02-10T04:38:16.356 INFO:tasks.rados.rados.0.smithi085.stdout:2619: read oid 50 snap 294 2021-02-10T04:38:16.356 INFO:tasks.rados.rados.0.smithi085.stdout:2619: expect (ObjNum 889 snap 289 seq_num 889) 2021-02-10T04:38:16.356 INFO:tasks.rados.rados.0.smithi085.stdout:2620: snap_remove snap 275 2021-02-10T04:38:16.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:16 smithi164 conmon[33677]: debug 2021-02-10T04:38:16.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:16 smithi164 conmon[33677]: 2021-02-10T04:38:16.248+0000 7fbec19a1700 -1 osd.4 678 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:38:16.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:16 smithi164 conmon[33677]: 2021-02-10T04:38:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:16 smithi164 conmon[29288]: cluster 2021-02-10T04:38:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:16 smithi164 conmon[29288]: 2021-02-10T04:38:15.293380 2021-02-10T04:38:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:16 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1095 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:16 smithi164 conmon[29288]: cluster 2021-02-10T04:38: 2021-02-10T04:38:16.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:16 smithi164 conmon[29288]: 15.293386+0000 osd.4 (osd.4) 1096 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:16.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:16 smithi164 conmon[29288]: cluster 2021-02-10T04:38:16.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:16 smithi164 conmon[29288]: 2021-02-10T04:38:15.293390+0000 osd.4 2021-02-10T04:38:16.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:16 smithi164 conmon[29288]: (osd.4) 1097 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:16.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:16 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:38:16.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:16 smithi164 conmon[29288]: -02-10T04:38:15.293393+0000 osd.4 (osd.4) 1098 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:16.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:16 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:38:16.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:16 smithi164 conmon[29288]: 04:38:16.153601+0000 mgr.y 2021-02-10T04:38:16.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:16 smithi164 conmon[29288]: (mgr.14140) 703 : cluster [DBG] pgmap v1166: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 361 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 1.8 MiB/s wr, 7 op/s 2021-02-10T04:38:16.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:16 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:38:16.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:16 smithi164 conmon[29288]: 02-10T04:38:16.221359+0000 2021-02-10T04:38:16.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:16 smithi164 conmon[29288]: mon.a (mon.0) 1562 : cluster [DBG] osdmap e678: 8 total, 8 up, 7 in 2021-02-10T04:38:16.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[29319]: cluster 2021-02-10T04:38:15.293380+0000 osd.4 (osd. 2021-02-10T04:38:16.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[29319]: 4) 1095 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:16.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[29319]: cluster 2021-02-10T04:38:15.293386+0000 osd.4 (osd.4) 1096 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:16.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[29319]: cluster 2021-02-10T04:38:15.293390+0000 osd.4 2021-02-10T04:38:16.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[29319]: (osd.4) 1097 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:16.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:38:16.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[29319]: T04:38:15.293393+0000 2021-02-10T04:38:16.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[29319]: osd.4 (osd.4) 1098 : cluster 2021-02-10T04:38:16.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:16.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[29319]: cluster 2021-02-10T04:38:16. 2021-02-10T04:38:16.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[29319]: 153601+0000 mgr.y (mgr.14140) 703 : cluster 2021-02-10T04:38:16.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[29319]: [DBG] pgmap v1166: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 361 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 1.8 MiB/s wr, 7 op/s 2021-02-10T04:38:16.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:38:16.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[29319]: -10T04:38:16. 2021-02-10T04:38:16.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[29319]: 221359+0000 mon.a (mon.0 2021-02-10T04:38:16.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[29319]: ) 1562 : cluster [DBG] osdmap e678: 8 total, 8 up, 7 in 2021-02-10T04:38:16.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[35961]: cluster 2021-02-10T04:38:15.293380+0000 2021-02-10T04:38:16.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[35961]: osd.4 (osd.4) 1095 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:16.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[35961]: cluster 2021-02-10T04:38 2021-02-10T04:38:16.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[35961]: :15.293386+0000 osd.4 (osd.4) 1096 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:16.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:38:16.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[35961]: 04:38:15.293390+0000 osd.4 (osd. 2021-02-10T04:38:16.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[35961]: 4) 1097 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:16.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:38:16.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[35961]: -10T04:38:15.293393+0000 osd.4 2021-02-10T04:38:16.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[35961]: (osd.4) 1098 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:16.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:38:16.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[35961]: 02-10T04:38:16.153601+0000 2021-02-10T04:38:16.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[35961]: mgr.y (mgr.14140) 703 : cluster [DBG] pgmap v1166: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 361 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 1.8 MiB/s wr, 7 op/s 2021-02-10T04:38:16.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:38:16.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[35961]: -10T04:38:16.221359+0000 mon.a (mon. 2021-02-10T04:38:16.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:16 smithi085 conmon[35961]: 0) 1562 : cluster [DBG] osdmap e678: 8 total, 8 up, 7 in 2021-02-10T04:38:17.225 INFO:tasks.rados.rados.0.smithi085.stdout:2616: finishing write tid 2 to smithi08575397-23 2021-02-10T04:38:17.225 INFO:tasks.rados.rados.0.smithi085.stdout:2616: finishing write tid 3 to smithi08575397-23 2021-02-10T04:38:17.226 INFO:tasks.rados.rados.0.smithi085.stdout:2616: finishing write tid 4 to smithi08575397-23 2021-02-10T04:38:17.226 INFO:tasks.rados.rados.0.smithi085.stdout:2616: finishing write tid 5 to smithi08575397-23 2021-02-10T04:38:17.226 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 1282 (ObjNum 913 snap 299 seq_num 913) dirty exists 2021-02-10T04:38:17.226 INFO:tasks.rados.rados.0.smithi085.stdout:2616: left oid 23 (ObjNum 913 snap 299 seq_num 913) 2021-02-10T04:38:17.226 INFO:tasks.rados.rados.0.smithi085.stdout:2617: finishing write tid 1 to smithi08575397-32 2021-02-10T04:38:17.226 INFO:tasks.rados.rados.0.smithi085.stdout:2617: finishing write tid 2 to smithi08575397-32 2021-02-10T04:38:17.226 INFO:tasks.rados.rados.0.smithi085.stdout:2618: expect (ObjNum 885 snap 287 seq_num 885) 2021-02-10T04:38:17.441 INFO:tasks.rados.rados.0.smithi085.stdout:2619: expect (ObjNum 889 snap 289 seq_num 889) 2021-02-10T04:38:17.594 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:38:17.628 INFO:tasks.rados.rados.0.smithi085.stdout:2617: finishing write tid 3 to smithi08575397-32 2021-02-10T04:38:17.628 INFO:tasks.rados.rados.0.smithi085.stdout:2617: finishing write tid 4 to smithi08575397-32 2021-02-10T04:38:17.628 INFO:tasks.rados.rados.0.smithi085.stdout:2617: finishing write tid 5 to smithi08575397-32 2021-02-10T04:38:17.629 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 32 v 640 (ObjNum 914 snap 299 seq_num 914) dirty exists 2021-02-10T04:38:17.629 INFO:tasks.rados.rados.0.smithi085.stdout:2617: left oid 32 (ObjNum 914 snap 299 seq_num 914) 2021-02-10T04:38:17.629 INFO:tasks.rados.rados.0.smithi085.stdout:2616: done (8 left) 2021-02-10T04:38:17.629 INFO:tasks.rados.rados.0.smithi085.stdout:2617: done (7 left) 2021-02-10T04:38:17.629 INFO:tasks.rados.rados.0.smithi085.stdout:2618: done (6 left) 2021-02-10T04:38:17.629 INFO:tasks.rados.rados.0.smithi085.stdout:2619: done (5 left) 2021-02-10T04:38:17.630 INFO:tasks.rados.rados.0.smithi085.stdout:2620: done (4 left) 2021-02-10T04:38:17.630 INFO:tasks.rados.rados.0.smithi085.stdout:2621: read oid 34 snap -1 2021-02-10T04:38:17.630 INFO:tasks.rados.rados.0.smithi085.stdout:2621: expect (ObjNum 859 snap 277 seq_num 859) 2021-02-10T04:38:17.630 INFO:tasks.rados.rados.0.smithi085.stdout:2622: snap_create 2021-02-10T04:38:17.638 INFO:tasks.rados.rados.0.smithi085.stdout:2621: expect (ObjNum 859 snap 277 seq_num 859) 2021-02-10T04:38:17.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:17 smithi164 conmon[33677]: debug 2021-02-10T04:38:17.286+0000 7fbec19a1700 -1 osd.4 679 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:38:17.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:17 smithi164 conmon[29288]: cluster 2021-02-10T04:38:17.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:17 smithi164 conmon[29288]: 2021-02-10T04:38:16. 2021-02-10T04:38:17.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:17 smithi164 conmon[29288]: 248918+0000 osd.4 (osd.4) 1099 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:17.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:17 smithi164 conmon[29288]: cluster 2021-02-10T04:38:16.248930+0000 osd.4 (osd. 2021-02-10T04:38:17.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:17 smithi164 conmon[29288]: 4) 1100 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:17.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:17 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:38:17.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:17 smithi164 conmon[29288]: 04:38:16.248937+0000 osd.4 (osd.4) 1101 2021-02-10T04:38:17.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:17 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:17.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:17 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:38:17.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:17 smithi164 conmon[29288]: 02-10T04:38:16.248943+0000 osd.4 (osd. 2021-02-10T04:38:17.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:17 smithi164 conmon[29288]: 4) 1102 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:17.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:17 smithi164 conmon[29288]: cluster 2021-02-10T04:38:17.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:17 smithi164 conmon[29288]: 2021-02-10T04:38:17.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:17 smithi164 conmon[29288]: 2021 2021-02-10T04:38:17.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:17 smithi164 conmon[29288]: - 2021-02-10T04:38:17.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:17 smithi164 conmon[29288]: 02 2021-02-10T04:38:17.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:17 smithi164 conmon[29288]: - 2021-02-10T04:38:17.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:17 smithi164 conmon[29288]: 10T04:38:17.223071+0000 2021-02-10T04:38:17.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:17 smithi164 conmon[29288]: mon.a (mon.0) 1563 : cluster [DBG] osdmap e679: 8 total, 8 up, 7 in 2021-02-10T04:38:17.748 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:38:17.901 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:38:17.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:17 smithi085 conmon[29319]: cluster 2021-02-10T04:38:16.248918+0000 2021-02-10T04:38:17.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:17 smithi085 conmon[29319]: osd.4 (osd.4) 1099 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:17.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:17 smithi085 conmon[29319]: cluster 2021-02-10T04:38:16.248930+0000 osd.4 ( 2021-02-10T04:38:17.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:17 smithi085 conmon[29319]: osd.4) 1100 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:17.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:17 smithi085 conmon[29319]: cluster 2021-02-10T04:38:16 2021-02-10T04:38:17.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:17 smithi085 conmon[29319]: .248937+0000 osd.4 (osd.4) 1101 : cluster 2021-02-10T04:38:17.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:17 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:17.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:17 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:38:17.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:17 smithi085 conmon[29319]: 38:16.248943+0000 osd.4 (osd.4) 1102 : 2021-02-10T04:38:17.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:17 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:17.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:17 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:38:17.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:17 smithi085 conmon[29319]: 10T04:38:17.223071+0000 mon.a ( 2021-02-10T04:38:17.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:17 smithi085 conmon[29319]: mon.0) 1563 : cluster [DBG] osdmap e679: 8 total, 8 up, 7 in 2021-02-10T04:38:17.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:17 smithi085 conmon[35961]: cluster 2021-02-10T04:38:16.248918+0000 2021-02-10T04:38:17.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:17 smithi085 conmon[35961]: osd.4 (osd.4) 1099 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:17.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:17 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:38:17.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:17 smithi085 conmon[35961]: 04:38:16.248930+0000 osd.4 (osd.4) 1100 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:17.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:17 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:38:17.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:17 smithi085 conmon[35961]: 02-10T04:38:16.248937+0000 osd.4 2021-02-10T04:38:17.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:17 smithi085 conmon[35961]: (osd.4) 1101 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:17.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:17 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:38:17.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:17 smithi085 conmon[35961]: -10T04:38:16.248943+0000 osd.4 (osd.4) 2021-02-10T04:38:17.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:17 smithi085 conmon[35961]: 1102 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:17.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:17 smithi085 conmon[35961]: cluster 2021-02-10T04:38: 2021-02-10T04:38:17.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:17 smithi085 conmon[35961]: 17.223071+0000 mon.a (mon.0) 1563 : cluster 2021-02-10T04:38:17.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:17 smithi085 conmon[35961]: [DBG] osdmap e679: 8 total, 8 up, 7 in 2021-02-10T04:38:18.046 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:38:18.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:17 smithi164 conmon[33677]: debug 2021-02-10T04:38:17.801+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:38:18.147 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:38:18.227 INFO:tasks.rados.rados.0.smithi085.stdout:2621: done (5 left) 2021-02-10T04:38:18.227 INFO:tasks.rados.rados.0.smithi085.stdout:2622: done (4 left) 2021-02-10T04:38:18.227 INFO:tasks.rados.rados.0.smithi085.stdout:2623: copy_from oid 6 from oid 3 current snap is 300 2021-02-10T04:38:18.227 INFO:tasks.rados.rados.0.smithi085.stdout:2624: delete oid 15 current snap is 300 2021-02-10T04:38:18.231 INFO:tasks.rados.rados.0.smithi085.stdout:2623: finishing copy_from to smithi08575397-6 2021-02-10T04:38:18.231 INFO:tasks.rados.rados.0.smithi085.stdout:2623: got expected ENOENT (src dne) 2021-02-10T04:38:18.231 INFO:tasks.rados.rados.0.smithi085.stdout:2624: done (5 left) 2021-02-10T04:38:18.231 INFO:tasks.rados.rados.0.smithi085.stdout:2625: snap_remove snap 292 2021-02-10T04:38:18.645 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:38:18 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:38:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:18 smithi164 conmon[29288]: cluster 2021-02-10T04:38:17.286936 2021-02-10T04:38:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:18 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1103 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:18 smithi164 conmon[29288]: cluster 2021-02-10T04:38:17.286946+0000 osd.4 (osd.4 2021-02-10T04:38:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:18 smithi164 conmon[29288]: ) 1104 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:18.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:18 smithi164 conmon[29288]: cluster 2021-02-10T04:38:18.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:18 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:38:18.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:18 smithi164 conmon[29288]: : 2021-02-10T04:38:18.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:18 smithi164 conmon[29288]: 38:17.286954+0000 osd.4 2021-02-10T04:38:18.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:18 smithi164 conmon[29288]: (osd.4) 1105 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:18.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:18 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:38:18.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:18 smithi164 conmon[29288]: 04:38:17.286958+0000 osd.4 (osd.4 2021-02-10T04:38:18.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:18 smithi164 conmon[29288]: ) 1106 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:18.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:18 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:38:18.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:18 smithi164 conmon[29288]: 02-10T04:38:18. 2021-02-10T04:38:18.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:18 smithi164 conmon[29288]: 154152+0000 mgr.y (mgr. 2021-02-10T04:38:18.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:18 smithi164 conmon[29288]: 14140) 704 : cluster [DBG] pgmap v1169: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 360 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 2.6 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2021-02-10T04:38:18.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:18 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:38:18.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:18 smithi164 conmon[29288]: 02-10T04:38:18.213851+0000 2021-02-10T04:38:18.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:18 smithi164 conmon[29288]: mon.a (mon.0 2021-02-10T04:38:18.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:18 smithi164 conmon[29288]: ) 1564 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 522 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:38:18.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:18 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:38:18.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:18 smithi164 conmon[29288]: -10T04:38:18.218728+0000 mon.a (mon 2021-02-10T04:38:18.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:18 smithi164 conmon[29288]: .0) 1565 : cluster [DBG] osdmap e680: 8 total, 8 up, 7 in 2021-02-10T04:38:18.647 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:18 smithi164 conmon[33677]: debug 2021-02-10T04:38:18.322+0000 7fbec19a1700 -1 osd.4 680 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:38:18.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[29319]: cluster 2021-02-10T04:38:17.286936+0000 osd.4 (osd.4) 1103 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:18.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[29319]: cluster 2021-02-10T04:38:17.286946+0000 osd.4 (osd.4) 1104 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:18.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[29319]: cluster 2021-02-10T04:38:17.286954+0000 osd.4 (osd.4) 1105 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:18.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[29319]: cluster 2021-02-10T04:38:17.286958+0000 osd.4 (osd.4) 1106 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:18.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[29319]: cluster 2021-02-10T04:38:18.154152+0000 mgr.y (mgr.14140) 704 : cluster [DBG] pgmap v1169: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 360 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 2.6 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2021-02-10T04:38:18.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[29319]: cluster 2021-02-10T04:38:18.213851+0000 mon.a (mon.0) 1564 : cluster [WRN] 2021-02-10T04:38:18.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[29319]: Health check update: 4 slow ops, oldest one blocked for 522 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:38:18.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[29319]: cluster 2021-02-10T04:38:18.218728+0000 mon.a (mon.0) 1565 : cluster [DBG] osdmap e680: 8 total, 8 up, 7 in 2021-02-10T04:38:18.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[35961]: cluster 2021-02-10T04:38:18.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[35961]: 2021-02 2021-02-10T04:38:18.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[35961]: -10T04:38:17.286936+0000 osd.4 (osd.4) 1103 : cluster [WRN] 2021-02-10T04:38:18.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:18.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[35961]: cluster 2021-02-10T04:38:17 2021-02-10T04:38:18.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[35961]: .286946+0000 osd.4 (osd.4) 1104 : cluster [WRN] 2021-02-10T04:38:18.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:18.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:38:18.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[35961]: -10T04:38:17.286954+0000 osd.4 (osd.4) 1105 2021-02-10T04:38:18.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:18.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[35961]: cluster 2021-02-10T04:38:18.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[35961]: 2021-02 2021-02-10T04:38:18.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[35961]: -10T04:38:17.286958+0000 osd.4 (osd 2021-02-10T04:38:18.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[35961]: .4) 1106 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:18.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[35961]: cluster 2021-02-10T04:38:18.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[35961]: 2021-02-10T04:38:18. 2021-02-10T04:38:18.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[35961]: 154152+0000 mgr.y (mgr.14140) 704 : cluster 2021-02-10T04:38:18.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[35961]: [DBG] pgmap v1169: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 360 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 2.6 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2021-02-10T04:38:18.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[35961]: cluster 2021-02-10T04:38:18.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[35961]: 2021-02-10T04:38:18 2021-02-10T04:38:18.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[35961]: .213851+0000 mon.a (mon.0) 1564 : 2021-02-10T04:38:18.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[35961]: cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 522 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:38:18.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:38:18.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[35961]: 10T04:38:18.218728+0000 mon.a ( 2021-02-10T04:38:18.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:18 smithi085 conmon[35961]: mon.0) 1565 : cluster [DBG] osdmap e680: 8 total, 8 up, 7 in 2021-02-10T04:38:19.233 INFO:tasks.rados.rados.0.smithi085.stdout:2625: done (5 left) 2021-02-10T04:38:19.233 INFO:tasks.rados.rados.0.smithi085.stdout:2626: snap_create 2021-02-10T04:38:19.233 INFO:tasks.rados.rados.0.smithi085.stdout:2623: finishing copy_from racing read to smithi08575397-6 2021-02-10T04:38:19.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:19 smithi164 conmon[33677]: debug 2021-02-10T04:38:19.310+0000 7fbec19a1700 -1 osd.4 681 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:38:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:19 smithi164 conmon[29288]: audit 2021-02-10T04:38:18.260077+0000 mgr.y (mgr.14140) 705 : audit [DBG] 2021-02-10T04:38:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:19 smithi164 conmon[29288]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:38:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:19 smithi164 conmon[29288]: cluster 2021-02-10T04:38:18.322929+0000 osd.4 (osd.4) 1107 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:19 smithi164 conmon[29288]: cluster 2021-02-10T04:38:18.322939+0000 osd.4 2021-02-10T04:38:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:19 smithi164 conmon[29288]: (osd.4) 1108 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:19 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:38:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:19 smithi164 conmon[29288]: 02-10T04:38:18.322947+0000 osd.4 (osd.4) 1109 : cluster [WRN] 2021-02-10T04:38:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:19 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:19 smithi164 conmon[29288]: cluster 2021-02-10T04:38:18.322954+0000 2021-02-10T04:38:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:19 smithi164 conmon[29288]: osd.4 (osd.4) 1110 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:19 smithi164 conmon[29288]: cluster 2021-02-10T04:38:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:19 smithi164 conmon[29288]: 2021-02-10T04:38:19.228132 2021-02-10T04:38:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:19 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1566 : cluster [DBG] 2021-02-10T04:38:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:19 smithi164 conmon[29288]: osdmap e681: 8 total, 8 up, 7 in 2021-02-10T04:38:19.913 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[29319]: audit 2021-02-10T04:38:19.913 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[29319]: 2021- 2021-02-10T04:38:19.914 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[29319]: 02-10T04:38:18.260077+0000 mgr.y (mgr.14140) 705 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:38:19.914 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[29319]: 2021-02-10T04:38:19.914 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:38:19.914 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[29319]: -02-10T04:38:18.322929+0000 2021-02-10T04:38:19.914 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[29319]: osd.4 (osd.4) 1107 : cluster [WRN] 2021-02-10T04:38:19.914 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:19.914 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:38:19.914 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[29319]: :38:18.322939+0000 osd.4 (osd. 2021-02-10T04:38:19.914 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[29319]: 4) 1108 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:19.914 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[29319]: cluster 2021-02-10T04:38:19.914 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[29319]: 2021-02-10T04:38:19.914 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[29319]: 2021 2021-02-10T04:38:19.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[29319]: - 2021-02-10T04:38:19.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[29319]: 02 2021-02-10T04:38:19.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[29319]: -10T04:38:18.322947+0000 osd.4 (osd.4) 1109 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:19.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[29319]: cluster 2021-02-10T04:38:19.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[29319]: 2021-02-10T04:38:19.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[29319]: 2021 2021-02-10T04:38:19.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[29319]: - 2021-02-10T04:38:19.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[29319]: 02-10T04:38 2021-02-10T04:38:19.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[29319]: :18.322954+0000 osd.4 (osd.4) 1110 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:19.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[29319]: cluster 2021-02-10T04:38:19.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[29319]: 2021- 2021-02-10T04:38:19.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[29319]: 02-10T04:38:19.228132+0000 mon.a (mon 2021-02-10T04:38:19.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[29319]: .0) 1566 : cluster [DBG] osdmap e681: 8 total, 8 up, 7 in 2021-02-10T04:38:19.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[35961]: audit 2021-02-10T04:38:18. 2021-02-10T04:38:19.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[35961]: 260077+0000 mgr.y (mgr.14140) 705 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:38:19.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[35961]: cluster 2021-02-10T04:38: 2021-02-10T04:38:19.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[35961]: 18.322929+0000 osd.4 (osd.4) 1107 2021-02-10T04:38:19.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:19.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:38:19.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[35961]: T04:38:18.322939+0000 2021-02-10T04:38:19.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[35961]: osd.4 (osd.4) 1108 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:19.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[35961]: 2021-02-10T04:38:19.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:38:19.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[35961]: 02-10T04:38:18.322947+0000 2021-02-10T04:38:19.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[35961]: osd.4 (osd.4) 1109 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:19.917 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[35961]: 2021-02-10T04:38:19.917 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:38:19.917 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[35961]: 04:38:18.322954+0000 osd.4 (osd. 2021-02-10T04:38:19.917 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[35961]: 4) 1110 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:19.917 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[35961]: cluster 2021-02-10T04:38:19.917 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[35961]: 2021-02-10T04:38:19. 2021-02-10T04:38:19.917 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[35961]: 228132+0000 mon.a (mon.0) 1566 : cluster 2021-02-10T04:38:19.917 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[35961]: [DBG] osdmap e681: 8 total, 8 up, 7 in 2021-02-10T04:38:20.062 INFO:tasks.thrashosds.thrasher:j is {'epoch': 666, 'fsid': '42224fce-6b57-11eb-8feb-001a4aab830c', 'created': '2021-02-10T04:20:57.933279+0000', 'modified': '2021-02-10T04:38:04.218561+0000', 'last_up_change': '2021-02-10T04:25:35.312604+0000', 'last_in_change': '2021-02-10T04:37:36.200492+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-02-10T04:24:05.258028+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '110', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2021-02-10T04:25:32.688554+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 42, 'pg_placement_num': 42, 'pg_placement_num_target': 42, 'pg_num_target': 42, 'pg_num_pending': 42, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '89', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2021-02-10T04:25:47.658613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'zone.rgw.log', 'create_time': '2021-02-10T04:25:47.968727+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'zone.rgw.control', 'create_time': '2021-02-10T04:25:49.278260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'zone.rgw.meta', 'create_time': '2021-02-10T04:25:51.792514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 8}, 'application_metadata': {'rgw': {}}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2021-02-10T04:28:23.932391+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '666', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 561, 'snap_epoch': 666, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '45d0bb5f-35fa-4596-964f-75abcee53bee', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 638, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6801', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6802', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6804', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6803', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6801/7', 'cluster_addr': '172.21.15.85:6802/7', 'heartbeat_back_addr': '172.21.15.85:6804/7', 'heartbeat_front_addr': '172.21.15.85:6803/7', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '3421a5ce-872e-4b52-adf7-6aaa595871a4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 130, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6805', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6806', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6808', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6807', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6805/7', 'cluster_addr': '172.21.15.85:6806/7', 'heartbeat_back_addr': '172.21.15.85:6808/7', 'heartbeat_front_addr': '172.21.15.85:6807/7', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '96e1952a-1042-4316-9fc9-86813e88b912', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 367, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6809', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6810', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6812', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6811', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6809/7', 'cluster_addr': '172.21.15.85:6810/7', 'heartbeat_back_addr': '172.21.15.85:6812/7', 'heartbeat_front_addr': '172.21.15.85:6811/7', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'e019f49f-5dd4-4104-9ced-94c710616642', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 649, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6813', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6814', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6816', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6815', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6813/7', 'cluster_addr': '172.21.15.85:6814/7', 'heartbeat_back_addr': '172.21.15.85:6816/7', 'heartbeat_front_addr': '172.21.15.85:6815/7', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '91c8a80a-37b6-4c30-a32a-647b2e6d801f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 638, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6800', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6801', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6803', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6802', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6800/7', 'cluster_addr': '172.21.15.164:6801/7', 'heartbeat_back_addr': '172.21.15.164:6803/7', 'heartbeat_front_addr': '172.21.15.164:6802/7', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 641, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6804', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6805', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6807', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6806', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6804/7', 'cluster_addr': '172.21.15.164:6805/7', 'heartbeat_back_addr': '172.21.15.164:6807/7', 'heartbeat_front_addr': '172.21.15.164:6806/7', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'adc40430-3eeb-4003-a824-343021243077', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 43, 'up_thru': 638, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6808', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6809', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6811', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6810', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6808/7', 'cluster_addr': '172.21.15.164:6809/7', 'heartbeat_back_addr': '172.21.15.164:6811/7', 'heartbeat_front_addr': '172.21.15.164:6810/7', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 112, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6812', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6813', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6815', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6814', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6812/7', 'cluster_addr': '172.21.15.164:6813/7', 'heartbeat_back_addr': '172.21.15.164:6815/7', 'heartbeat_front_addr': '172.21.15.164:6814/7', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:37:33.740656+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:37:50.553663+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 65536, 'last_purged_snaps_scrub': '2021-02-10T04:37:08.605364+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:37:29.449193+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:37:45.482448+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:37:04.740466+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:37:21.548801+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:37:39.944872+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.7', 'mappings': [{'from': 7, 'to': 4}]}, {'pgid': '2.21', 'mappings': [{'from': 7, 'to': 1}]}, {'pgid': '4.10', 'mappings': [{'from': 1, 'to': 5}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.85:0/801393150': '2021-02-11T04:21:37.851607+0000', '172.21.15.85:0/3049671696': '2021-02-11T04:21:37.851607+0000', '172.21.15.85:0/4056505602': '2021-02-11T04:21:20.700478+0000', '172.21.15.85:6800/7': '2021-02-11T04:21:20.700478+0000', '172.21.15.85:0/1879026062': '2021-02-11T04:21:20.700478+0000'}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 461, 'length': 1}, {'begin': 464, 'length': 2}, {'begin': 467, 'length': 1}, {'begin': 469, 'length': 1}, {'begin': 472, 'length': 2}, {'begin': 475, 'length': 2}, {'begin': 479, 'length': 1}, {'begin': 481, 'length': 2}, {'begin': 484, 'length': 1}, {'begin': 489, 'length': 1}, {'begin': 491, 'length': 1}, {'begin': 494, 'length': 3}, {'begin': 498, 'length': 3}, {'begin': 503, 'length': 2}, {'begin': 513, 'length': 1}, {'begin': 518, 'length': 1}, {'begin': 525, 'length': 1}, {'begin': 539, 'length': 1}, {'begin': 541, 'length': 1}, {'begin': 546, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2021-02-10T04:38:20.063 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:38:20.063 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:38:20.168 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:38:20.191 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:38:19 smithi085 conmon[41540]: debug 2021-02-10T04:38:19.924+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:38:20.207 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:38:20.235 INFO:tasks.rados.rados.0.smithi085.stdout:2623: done (5 left) 2021-02-10T04:38:20.236 INFO:tasks.rados.rados.0.smithi085.stdout:2626: done (4 left) 2021-02-10T04:38:20.236 INFO:tasks.rados.rados.0.smithi085.stdout:2627: copy_from oid 28 from oid 6 current snap is 301 2021-02-10T04:38:20.236 INFO:tasks.rados.rados.0.smithi085.stdout:2628: rollback oid 1 current snap is 301 2021-02-10T04:38:20.236 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 1 to 284 2021-02-10T04:38:20.236 INFO:tasks.rados.rados.0.smithi085.stdout:2629: rollback oid 50 current snap is 301 2021-02-10T04:38:20.236 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 50 to 276 2021-02-10T04:38:20.236 INFO:tasks.rados.rados.0.smithi085.stdout:2630: read oid 15 snap -1 2021-02-10T04:38:20.236 INFO:tasks.rados.rados.0.smithi085.stdout:2630: expect deleted 2021-02-10T04:38:20.236 INFO:tasks.rados.rados.0.smithi085.stdout:2631: read oid 32 snap -1 2021-02-10T04:38:20.236 INFO:tasks.rados.rados.0.smithi085.stdout:2631: expect (ObjNum 914 snap 299 seq_num 914) 2021-02-10T04:38:20.236 INFO:tasks.rados.rados.0.smithi085.stdout:2632: snap_create 2021-02-10T04:38:20.241 INFO:tasks.rados.rados.0.smithi085.stdout:2628: finishing rollback tid 0 to smithi08575397-1 2021-02-10T04:38:20.245 INFO:tasks.rados.rados.0.smithi085.stdout:2628: finishing rollback tid 1 to smithi08575397-1 2021-02-10T04:38:20.245 INFO:tasks.rados.rados.0.smithi085.stdout:2628: finishing rollback tid 2 to smithi08575397-1 2021-02-10T04:38:20.245 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 1332 (ObjNum 854 snap 277 seq_num 854) dirty exists 2021-02-10T04:38:20.248 INFO:tasks.rados.rados.0.smithi085.stdout:2629: finishing rollback tid 0 to smithi08575397-50 2021-02-10T04:38:20.249 INFO:tasks.rados.rados.0.smithi085.stdout:2629: finishing rollback tid 1 to smithi08575397-50 2021-02-10T04:38:20.249 INFO:tasks.rados.rados.0.smithi085.stdout:2629: finishing rollback tid 2 to smithi08575397-50 2021-02-10T04:38:20.249 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 1335 (ObjNum 852 snap 276 seq_num 852) dirty exists 2021-02-10T04:38:20.251 INFO:tasks.rados.rados.0.smithi085.stdout:2627: finishing copy_from to smithi08575397-28 2021-02-10T04:38:20.251 INFO:tasks.rados.rados.0.smithi085.stdout:2627: got expected ENOENT (src dne) 2021-02-10T04:38:20.251 INFO:tasks.rados.rados.0.smithi085.stdout:2627: finishing copy_from racing read to smithi08575397-28 2021-02-10T04:38:20.259 INFO:tasks.rados.rados.0.smithi085.stdout:2631: expect (ObjNum 914 snap 299 seq_num 914) 2021-02-10T04:38:20.431 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:38:20.498 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:20 smithi085 conmon[29319]: cluster 2021-02-10T04:38:19.311685+0000 osd.4 (osd.4) 1111 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:20.498 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:20 smithi085 conmon[29319]: cluster 2021-02-10T04:38:19.311695+0000 osd.4 (osd.4) 1112 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:20.498 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:20 smithi085 conmon[29319]: cluster 2021-02-10T04:38:19.311704+0000 osd.4 (osd.4) 1113 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:20.498 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:20 smithi085 conmon[29319]: cluster 2021-02-10T04:38:19.311710+0000 osd.4 (osd.4) 1114 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:20.499 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:20 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:38:20.499 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:20 smithi085 conmon[29319]: -10T04:38:20.154775+0000 mgr.y (mgr.14140) 706 : cluster [DBG] pgmap v1172: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 360 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 7.3 MiB/s rd, 2.6 MiB/s wr, 16 op/s 2021-02-10T04:38:20.499 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:20 smithi085 conmon[29319]: cluster 2021-02-10T04:38:20.230302+0000 mon.a (mon.0) 1567 : cluster [DBG] osdmap e682: 8 total, 8 up, 7 in 2021-02-10T04:38:20.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:20 smithi164 conmon[33677]: debug 2021-02-10T04:38:20.270+0000 7fbec19a1700 -1 osd.4 682 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:38:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:20 smithi164 conmon[29288]: cluster 2021-02-10T04:38:19.311685+0000 osd.4 ( 2021-02-10T04:38:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:20 smithi164 conmon[29288]: osd.4) 1111 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:20 smithi164 conmon[29288]: cluster 2021-02-10T04:38:19.311695+0000 osd.4 (osd.4) 1112 : cluster [WRN] 2021-02-10T04:38:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:20 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:20 smithi164 conmon[29288]: cluster 2021-02-10T04:38:19.311704+0000 2021-02-10T04:38:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:20 smithi164 conmon[29288]: osd.4 (osd.4) 1113 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:20 smithi164 conmon[29288]: cluster 2021-02-10T04:38: 2021-02-10T04:38:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:20 smithi164 conmon[29288]: 19.311710+0000 osd.4 (osd.4) 1114 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:20 smithi164 conmon[29288]: 2021-02-10T04:38:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:20 smithi164 conmon[29288]: cluster 2021-02-10T04:38:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:20 smithi164 conmon[29288]: 2021-02-10T04:38:20.154775+0000 mgr.y (mgr. 2021-02-10T04:38:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:20 smithi164 conmon[29288]: 14140) 706 : cluster [DBG] pgmap v1172: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 360 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 7.3 MiB/s rd, 2.6 MiB/s wr, 16 op/s 2021-02-10T04:38:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:20 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:38:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:20 smithi164 conmon[29288]: 04:38:20.230302+0000 2021-02-10T04:38:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:20 smithi164 conmon[29288]: mon.a (mon.0) 1567 : cluster 2021-02-10T04:38:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:20 smithi164 conmon[29288]: [DBG] osdmap e682: 8 total, 8 up, 7 in 2021-02-10T04:38:20.654 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:38:20.793 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:38:20.824 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:20 smithi085 conmon[35961]: cluster 2021-02-10T04:38:19.311685 2021-02-10T04:38:20.824 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:20 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1111 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:20.825 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:20 smithi085 conmon[35961]: cluster 2021-02-10T04:38:20.825 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:20 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:38:20.825 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:20 smithi085 conmon[35961]: 10T04:38:19.311695+0000 osd.4 (osd.4) 1112 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:20.825 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:20 smithi085 conmon[35961]: cluster 2021-02-10T04:38:20.825 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:20 smithi085 conmon[35961]: 2021-02-10T04:38:20.825 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:20 smithi085 conmon[35961]: 2021 2021-02-10T04:38:20.825 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:20 smithi085 conmon[35961]: - 2021-02-10T04:38:20.825 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:20 smithi085 conmon[35961]: 02 2021-02-10T04:38:20.825 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:20 smithi085 conmon[35961]: - 2021-02-10T04:38:20.825 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:20 smithi085 conmon[35961]: 10 2021-02-10T04:38:20.825 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:20 smithi085 conmon[35961]: T04:38:19.311704 2021-02-10T04:38:20.825 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:20 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1113 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:20.826 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:20 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:38:20.826 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:20 smithi085 conmon[35961]: 38:19.311710+0000 osd.4 (osd.4) 2021-02-10T04:38:20.826 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:20 smithi085 conmon[35961]: 1114 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:20.826 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:20 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:38:20.826 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:20 smithi085 conmon[35961]: 10T04:38:20.154775+0000 mgr.y (mgr.14140) 706 : cluster [DBG] pgmap v1172: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 360 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 7.3 MiB/s rd, 2.6 MiB/s wr, 16 op/s 2021-02-10T04:38:20.826 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:20 smithi085 conmon[35961]: cluster 2021-02-10T04:38:20 2021-02-10T04:38:20.826 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:20 smithi085 conmon[35961]: .230302+0000 mon.a (mon.0) 1567 : cluster 2021-02-10T04:38:20.826 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:20 smithi085 conmon[35961]: [DBG] osdmap e682: 8 total, 8 up, 7 in 2021-02-10T04:38:20.826 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:38:20 smithi085 conmon[45443]: debug 2021-02-10T04:38:20.562+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:38:20.833 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:38:21.147 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:38:21.173 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[52836]: debug 2021-02-10T04:38:21.173 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[52836]: 2021-02-10T04:38:21.029+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:38:21.173 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[52836]: 2021-02-10T04:38:21.211 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:38:21.232 INFO:tasks.rados.rados.0.smithi085.stdout:2627: done (9 left) 2021-02-10T04:38:21.232 INFO:tasks.rados.rados.0.smithi085.stdout:2628: done (8 left) 2021-02-10T04:38:21.233 INFO:tasks.rados.rados.0.smithi085.stdout:2629: done (7 left) 2021-02-10T04:38:21.233 INFO:tasks.rados.rados.0.smithi085.stdout:2630: done (6 left) 2021-02-10T04:38:21.233 INFO:tasks.rados.rados.0.smithi085.stdout:2631: done (5 left) 2021-02-10T04:38:21.233 INFO:tasks.rados.rados.0.smithi085.stdout:2632: done (4 left) 2021-02-10T04:38:21.233 INFO:tasks.rados.rados.0.smithi085.stdout:2633: read oid 3 snap -1 2021-02-10T04:38:21.233 INFO:tasks.rados.rados.0.smithi085.stdout:2633: expect deleted 2021-02-10T04:38:21.233 INFO:tasks.rados.rados.0.smithi085.stdout:2634: delete oid 25 current snap is 302 2021-02-10T04:38:21.242 INFO:tasks.rados.rados.0.smithi085.stdout:2633: done (5 left) 2021-02-10T04:38:21.242 INFO:tasks.rados.rados.0.smithi085.stdout:2634: done (4 left) 2021-02-10T04:38:21.242 INFO:tasks.rados.rados.0.smithi085.stdout:2635: write (excl) oid 48 current snap is 302 2021-02-10T04:38:21.242 INFO:tasks.rados.rados.0.smithi085.stdout:2635: seq_num 917 ranges {612019=565693,1810929=602646,3082649=329370} 2021-02-10T04:38:21.245 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is unset 2021-02-10T04:38:21.252 INFO:tasks.rados.rados.0.smithi085.stdout:2635: writing smithi08575397-48 from 612019 to 1177712 tid 1 2021-02-10T04:38:21.259 INFO:tasks.rados.rados.0.smithi085.stdout:2635: writing smithi08575397-48 from 1810929 to 2413575 tid 2 2021-02-10T04:38:21.265 INFO:tasks.rados.rados.0.smithi085.stdout:2635: writing smithi08575397-48 from 3082649 to 3412019 tid 3 2021-02-10T04:38:21.265 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:38:21.265 INFO:tasks.rados.rados.0.smithi085.stdout:2636: rollback oid 45 current snap is 302 2021-02-10T04:38:21.265 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 45 to 290 2021-02-10T04:38:21.265 INFO:tasks.rados.rados.0.smithi085.stdout:2637: delete oid 16 current snap is 302 2021-02-10T04:38:21.267 INFO:tasks.rados.rados.0.smithi085.stdout:2635: finishing write tid 1 to smithi08575397-48 2021-02-10T04:38:21.267 INFO:tasks.rados.rados.0.smithi085.stdout:2635: finishing write tid 2 to smithi08575397-48 2021-02-10T04:38:21.270 INFO:tasks.rados.rados.0.smithi085.stdout:2635: finishing write tid 3 to smithi08575397-48 2021-02-10T04:38:21.270 INFO:tasks.rados.rados.0.smithi085.stdout:2635: finishing write tid 4 to smithi08575397-48 2021-02-10T04:38:21.270 INFO:tasks.rados.rados.0.smithi085.stdout:2637: done (6 left) 2021-02-10T04:38:21.270 INFO:tasks.rados.rados.0.smithi085.stdout:2638: snap_remove snap 283 2021-02-10T04:38:21.277 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:38:21.278 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:38:21.310 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:38:21.316 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:38:21.353 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:38:21.390 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:38:21.419 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:38:21.449 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:38:21.450 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:38:21.450 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:38:21.450 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:38:21.450 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:38:21.451 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:38:21.451 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:38:21.451 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:38:21.451 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:38:21.479 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:38:21.479 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [], 2021-02-10T04:38:21.479 INFO:teuthology.orchestra.run.smithi085.stdout: "complaint_time": 30, 2021-02-10T04:38:21.479 INFO:teuthology.orchestra.run.smithi085.stdout: "num_blocked_ops": 0 2021-02-10T04:38:21.479 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:38:21.729 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[49106]: debug 2021-02-10T04:38:21.517+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:38:21.731 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[29319]: cluster 2021-02-10T04:38:20.270732 2021-02-10T04:38:21.731 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1115 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:21.731 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[29319]: cluster 2021-02-10T04:38:20.270742+0000 osd.4 (osd.4) 1116 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:21.731 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[29319]: cluster 2021-02-10T04:38:20.270750+0000 osd.4 (osd.4) 1117 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:21.731 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[29319]: cluster 2021-02-10T04:38:20.270755+0000 osd.4 (osd.4) 1118 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:21.732 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[29319]: audit 2021-02-10T04:38:21.161639+0000 mon.a (mon.0) 1568 : audit [DBG] from='client.? 172.21.15.85:0/1401093359' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:38:21.732 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[29319]: audit 2021-02-10T04:38:21.184764+0000 mon.c (mon.1) 61 : audit [INF] from='client.? 172.21.15.85:0/1113271903' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:38:21.732 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[29319]: audit 2021-02-10T04:38:21.185183+0000 mon.a (mon.0) 1569 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:38:21.732 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[29319]: cluster 2021-02-10T04:38:21.226352+0000 mon.a (mon.0) 1570 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:38:21.732 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[29319]: audit 2021-02-10T04:38:21.231473+0000 mon.a (mon.0) 1571 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:38:21.732 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[29319]: cluster 2021-02-10T04:38:21.231553+0000 mon.a (mon.0) 1572 : cluster [DBG] osdmap e683: 8 total, 8 up, 7 in 2021-02-10T04:38:21.732 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: cluster 2021-02-10T04:38:21.732 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: 2021-02-10T04:38:20. 2021-02-10T04:38:21.732 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: 270732+0000 osd.4 (osd.4) 1115 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:21.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: cluster 2021-02-10T04:38:21.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: 2021-02-10T04:38:20.270742 2021-02-10T04:38:21.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1116 : cluster [WRN] 2021-02-10T04:38:21.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:21.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: cluster 2021-02-10T04:38:20. 2021-02-10T04:38:21.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: 270750+0000 osd.4 (osd.4) 1117 : cluster [WRN] 2021-02-10T04:38:21.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:21.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: cluster 2021-02-10T04:38:21.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: 2021-02-10T04:38:20.270755+0000 osd.4 (osd.4) 1118 2021-02-10T04:38:21.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:21.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: audit 2021-02-10T04:38:21. 2021-02-10T04:38:21.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: 161639+0000 mon.a (mon.0) 1568 : audit [DBG] 2021-02-10T04:38:21.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/1401093359' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:38:21.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: audit 2021-02-10T04:38:21.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: 2021-02-10T04:38:21.184764+0000 mon.c (mon. 2021-02-10T04:38:21.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: 1) 61 : audit [INF] from='client.? 172.21.15.85:0/1113271903' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:38:21.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: audit 2021-02-10T04:38: 2021-02-10T04:38:21.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: 21.185183+0000 mon.a (mon.0) 1569 : audit 2021-02-10T04:38:21.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:38:21.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: cluster 2021-02-10T04:38:21.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: 2021-02-10T04:38:21.226352+0000 mon.a (mon. 2021-02-10T04:38:21.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: 0) 1570 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:38:21.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:38:21.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: 38:21.231473+0000 mon.a (mon.0) 1571 2021-02-10T04:38:21.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:38:21.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:38:21.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: T04:38:21.231553+0000 mon.a (mon.0) 1572 : cluster [DBG] 2021-02-10T04:38:21.735 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:21 smithi085 conmon[35961]: osdmap e683: 8 total, 8 up, 7 in 2021-02-10T04:38:21.745 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:38:21.851 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[33677]: debug 2021-02-10T04:38:21.285+0000 7fbec19a1700 -1 osd.4 683 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:38:21.851 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[29288]: cluster 2021-02-10T04:38:20.270732+0000 osd.4 (osd.4) 1115 2021-02-10T04:38:21.851 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:21.851 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[29288]: cluster 2021-02-10T04:38:20.270742+0000 osd.4 (osd.4) 1116 : cluster 2021-02-10T04:38:21.851 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:21.851 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[29288]: cluster 2021-02-10T04:38:21.851 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[29288]: 2021-02-10T04:38:20. 2021-02-10T04:38:21.851 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[29288]: 270750+0000 osd.4 (osd.4) 1117 : cluster 2021-02-10T04:38:21.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:21.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:38:21.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[29288]: 04:38:20.270755+0000 2021-02-10T04:38:21.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[29288]: osd.4 (osd.4) 1118 : cluster 2021-02-10T04:38:21.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:21.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:38:21.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[29288]: -10T04:38:21.161639+0000 mon.a ( 2021-02-10T04:38:21.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[29288]: mon.0) 1568 : audit [DBG] from='client.? 172.21.15.85:0/1401093359' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:38:21.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:38:21.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[29288]: 02-10T04:38:21.184764+0000 mon.c 2021-02-10T04:38:21.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[29288]: (mon.1) 61 : audit [INF] from='client.? 172.21.15.85:0/1113271903' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:38:21.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[29288]: 2021-02-10T04:38:21.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:38:21.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[29288]: 04:38:21.185183+0000 mon.a (mon.0) 1569 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:38:21.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[29288]: cluster 2021-02-10T04:38:21.226352+0000 mon.a (mon. 2021-02-10T04:38:21.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[29288]: 0) 1570 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:38:21.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[29288]: audit 2021-02-10T04:38:21.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[29288]: 2021-02-10T04:38:21.231473+0000 mon.a (mon.0) 1571 : audit [INF] 2021-02-10T04:38:21.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[29288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:38:21.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[29288]: cluster 2021-02-10T04:38: 2021-02-10T04:38:21.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[29288]: 21.231553+0000 mon.a (mon 2021-02-10T04:38:21.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[29288]: .0) 1572 : cluster [DBG] 2021-02-10T04:38:21.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[29288]: osdmap e683: 8 total, 8 up, 7 in 2021-02-10T04:38:21.855 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:38:21.892 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:38:22.109 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:38:22.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:21 smithi164 conmon[33677]: debug 2021-02-10T04:38:21.973+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:38:22.215 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:38:22.233 INFO:tasks.rados.rados.0.smithi085.stdout:2638: done (6 left) 2021-02-10T04:38:22.233 INFO:tasks.rados.rados.0.smithi085.stdout:2639: copy_from oid 38 from oid 1 current snap is 302 2021-02-10T04:38:22.234 INFO:tasks.rados.rados.0.smithi085.stdout:2640: delete oid 28 current snap is 302 2021-02-10T04:38:22.237 INFO:tasks.rados.rados.0.smithi085.stdout:2636: finishing rollback tid 0 to smithi08575397-45 2021-02-10T04:38:22.237 INFO:tasks.rados.rados.0.smithi085.stdout:2635: finishing write tid 5 to smithi08575397-48 2021-02-10T04:38:22.237 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 957 (ObjNum 917 snap 302 seq_num 917) dirty exists 2021-02-10T04:38:22.237 INFO:tasks.rados.rados.0.smithi085.stdout:2635: left oid 48 (ObjNum 917 snap 302 seq_num 917) 2021-02-10T04:38:22.237 INFO:tasks.rados.rados.0.smithi085.stdout:2636: finishing rollback tid 1 to smithi08575397-45 2021-02-10T04:38:22.237 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 45 v 897 (ObjNum 1953066355 snap 942696808 seq_num 859125557) dirty dne 2021-02-10T04:38:22.245 INFO:tasks.rados.rados.0.smithi085.stdout:2635: done (7 left) 2021-02-10T04:38:22.245 INFO:tasks.rados.rados.0.smithi085.stdout:2636: done (6 left) 2021-02-10T04:38:22.245 INFO:tasks.rados.rados.0.smithi085.stdout:2640: done (5 left) 2021-02-10T04:38:22.245 INFO:tasks.rados.rados.0.smithi085.stdout:2641: write oid 37 current snap is 302 2021-02-10T04:38:22.245 INFO:tasks.rados.rados.0.smithi085.stdout:2641: seq_num 920 ranges {662440=507793,1702728=749973,2881271=557193} 2021-02-10T04:38:22.250 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:38:22.251 INFO:tasks.rados.rados.0.smithi085.stdout:2641: writing smithi08575397-37 from 662440 to 1170233 tid 1 2021-02-10T04:38:22.260 INFO:tasks.rados.rados.0.smithi085.stdout:2641: writing smithi08575397-37 from 1702728 to 2452701 tid 2 2021-02-10T04:38:22.266 INFO:tasks.rados.rados.0.smithi085.stdout:2641: writing smithi08575397-37 from 2881271 to 3438464 tid 3 2021-02-10T04:38:22.267 INFO:tasks.rados.rados.0.smithi085.stdout:2641: finishing write tid 1 to smithi08575397-37 2021-02-10T04:38:22.267 INFO:tasks.rados.rados.0.smithi085.stdout:2642: snap_create 2021-02-10T04:38:22.272 INFO:tasks.rados.rados.0.smithi085.stdout:2641: finishing write tid 2 to smithi08575397-37 2021-02-10T04:38:22.273 INFO:tasks.rados.rados.0.smithi085.stdout:2641: finishing write tid 3 to smithi08575397-37 2021-02-10T04:38:22.273 INFO:tasks.rados.rados.0.smithi085.stdout:2641: finishing write tid 4 to smithi08575397-37 2021-02-10T04:38:22.273 INFO:tasks.rados.rados.0.smithi085.stdout:2641: finishing write tid 5 to smithi08575397-37 2021-02-10T04:38:22.273 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 1386 (ObjNum 920 snap 302 seq_num 920) dirty exists 2021-02-10T04:38:22.273 INFO:tasks.rados.rados.0.smithi085.stdout:2641: left oid 37 (ObjNum 920 snap 302 seq_num 920) 2021-02-10T04:38:22.278 INFO:tasks.rados.rados.0.smithi085.stdout:2639: finishing copy_from to smithi08575397-38 2021-02-10T04:38:22.278 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 1332 (ObjNum 854 snap 277 seq_num 854) dirty exists 2021-02-10T04:38:22.279 INFO:tasks.rados.rados.0.smithi085.stdout:2639: finishing copy_from racing read to smithi08575397-38 2021-02-10T04:38:22.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:22 smithi164 conmon[29288]: cluster 2021-02-10T04:38:21.285872+0000 2021-02-10T04:38:22.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:22 smithi164 conmon[29288]: osd.4 (osd.4) 1119 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:22.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:22 smithi164 conmon[29288]: cluster 2021-02-10T04:38:21.285883+0000 osd.4 (osd.4) 1120 : cluster [WRN] 2021-02-10T04:38:22.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:22 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:22.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:22 smithi164 conmon[29288]: cluster 2021-02-10T04:38:21 2021-02-10T04:38:22.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:22 smithi164 conmon[29288]: .285889+0000 osd.4 (osd.4) 1121 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:22.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:22 smithi164 conmon[29288]: cluster 2021-02-10T04:38:21.285895 2021-02-10T04:38:22.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:22 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1122 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:22.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:22 smithi164 conmon[29288]: cluster 2021-02-10T04:38 2021-02-10T04:38:22.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:22 smithi164 conmon[29288]: :21.961466+0000 osd.3 (osd.3 2021-02-10T04:38:22.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:22 smithi164 conmon[29288]: ) 74 : cluster [DBG] 4.19 scrub ok 2021-02-10T04:38:22.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:22 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:38:22.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:22 smithi164 conmon[29288]: 02-10T 2021-02-10T04:38:22.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:22 smithi164 conmon[29288]: 04 2021-02-10T04:38:22.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:22 smithi164 conmon[29288]: :38:22.155385 2021-02-10T04:38:22.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:22 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 707 : cluster 2021-02-10T04:38:22.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:22 smithi164 conmon[29288]: [DBG] pgmap v1175: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 359 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 4.6 MiB/s rd, 2.9 MiB/s wr, 16 op/s 2021-02-10T04:38:22.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:22 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:38:22.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:22 smithi164 conmon[29288]: -02-10T04:38:22.232358+0000 mon.a (mon.0) 1573 : cluster [DBG] osdmap e684: 8 total, 8 up, 7 in 2021-02-10T04:38:22.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[29319]: cluster 2021-02-10T04:38: 2021-02-10T04:38:22.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[29319]: 21.285872+0000 osd.4 (osd.4) 1119 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:22.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[29319]: cluster 2021-02-10T04:38:21.285883+0000 osd.4 (osd.4) 1120 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:22.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[29319]: cluster 2021-02-10T04:38 2021-02-10T04:38:22.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[29319]: :21.285889 2021-02-10T04:38:22.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1121 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:22.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[29319]: cluster 2021-02-10T04:38:22.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:38:22.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[29319]: 10T04:38:21.285895+0000 osd.4 ( 2021-02-10T04:38:22.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[29319]: osd.4) 1122 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:22.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:38:22.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[29319]: 02-10T04:38:21.961466+0000 2021-02-10T04:38:22.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[29319]: osd.3 (osd.3) 74 : 2021-02-10T04:38:22.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[29319]: cluster [DBG] 4.19 scrub ok 2021-02-10T04:38:22.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:38:22.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[29319]: 02-10T04:38:22.155385+0000 mgr.y 2021-02-10T04:38:22.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[29319]: (mgr.14140) 707 : cluster [DBG] pgmap v1175: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 359 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 4.6 MiB/s rd, 2.9 MiB/s wr, 16 op/s 2021-02-10T04:38:22.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[29319]: cluster 2021-02-10T04:38:22.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[29319]: 2021-02-10T04:38:22.232358+0000 2021-02-10T04:38:22.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[29319]: mon.a (mon.0) 1573 : cluster [DBG] 2021-02-10T04:38:22.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[29319]: osdmap e684: 8 total, 8 up, 7 in 2021-02-10T04:38:22.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[35961]: cluster 2021-02-10T04:38:21.285872 2021-02-10T04:38:22.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1119 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:38:22.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[35961]: cluster 2021-02-10T04:38:21.285883+0000 osd.4 (osd.4) 1120 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:38:22.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[35961]: cluster 2021-02-10T04:38:21.285889+0000 osd.4 (osd.4) 1121 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:38:22.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[35961]: cluster 2021-02-10T04:38:21.285895+0000 2021-02-10T04:38:22.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[35961]: osd.4 (osd.4) 1122 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:38:22.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:38:22.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[35961]: :38:21.961466+0000 osd.3 (osd.3) 74 : cluster [DBG] 4.19 scrub ok 2021-02-10T04:38:22.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[35961]: cluster 2021-02-10T04:38:22.155385+0000 mgr.y (mgr.14140) 707 : cluster [DBG] pgmap v1175: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 359 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 4.6 MiB/s rd, 2.9 MiB/s wr, 16 op/s 2021-02-10T04:38:22.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[35961]: 2021-02-10T04:38:22.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[35961]: cluster 2021-02-10T04:38 2021-02-10T04:38:22.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:22 smithi085 conmon[35961]: :22.232358+0000 mon.a (mon.0) 1573 : cluster [DBG] osdmap e684: 8 total, 8 up, 7 in 2021-02-10T04:38:23.224 INFO:tasks.rados.rados.0.smithi085.stdout:2639: done (6 left) 2021-02-10T04:38:23.224 INFO:tasks.rados.rados.0.smithi085.stdout:2641: done (5 left) 2021-02-10T04:38:23.224 INFO:tasks.rados.rados.0.smithi085.stdout:2642: done (4 left) 2021-02-10T04:38:23.225 INFO:tasks.rados.rados.0.smithi085.stdout:2643: write oid 29 current snap is 303 2021-02-10T04:38:23.225 INFO:tasks.rados.rados.0.smithi085.stdout:2643: seq_num 921 ranges {679247=583042,2044857=609984,3192341=286906} 2021-02-10T04:38:23.232 INFO:tasks.rados.rados.0.smithi085.stdout:2643: writing smithi08575397-29 from 679247 to 1262289 tid 1 2021-02-10T04:38:23.242 INFO:tasks.rados.rados.0.smithi085.stdout:2643: writing smithi08575397-29 from 2044857 to 2654841 tid 2 2021-02-10T04:38:23.248 INFO:tasks.rados.rados.0.smithi085.stdout:2643: writing smithi08575397-29 from 3192341 to 3479247 tid 3 2021-02-10T04:38:23.248 INFO:tasks.rados.rados.0.smithi085.stdout:2644: snap_create 2021-02-10T04:38:23.248 INFO:tasks.rados.rados.0.smithi085.stdout:2643: finishing write tid 1 to smithi08575397-29 2021-02-10T04:38:23.250 INFO:tasks.rados.rados.0.smithi085.stdout:2643: finishing write tid 2 to smithi08575397-29 2021-02-10T04:38:23.252 INFO:tasks.rados.rados.0.smithi085.stdout:2643: finishing write tid 3 to smithi08575397-29 2021-02-10T04:38:23.252 INFO:tasks.rados.rados.0.smithi085.stdout:2643: finishing write tid 4 to smithi08575397-29 2021-02-10T04:38:23.252 INFO:tasks.rados.rados.0.smithi085.stdout:2643: finishing write tid 5 to smithi08575397-29 2021-02-10T04:38:23.252 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 29 v 645 (ObjNum 921 snap 303 seq_num 921) dirty exists 2021-02-10T04:38:23.253 INFO:tasks.rados.rados.0.smithi085.stdout:2643: left oid 29 (ObjNum 921 snap 303 seq_num 921) 2021-02-10T04:38:23.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:23 smithi164 conmon[29288]: cluster 2021-02-10T04:38:22.024022+0000 osd.6 (osd.6 2021-02-10T04:38:23.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:23 smithi164 conmon[29288]: ) 69 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:38:23.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:23 smithi164 conmon[29288]: cluster 2021-02-10T04:38:23.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:23 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:38:23.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:23 smithi164 conmon[29288]: 38:22.024835+0000 osd.6 (osd.6) 70 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:38:23.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:23 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:38:23.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:23 smithi164 conmon[29288]: -10T04:38:22.136284+0000 osd.6 (osd.6 2021-02-10T04:38:23.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:23 smithi164 conmon[29288]: ) 71 : cluster [DBG] 2.29 scrub ok 2021-02-10T04:38:23.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:23 smithi164 conmon[29288]: cluster 2021-02-10T04:38:23.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:23 smithi164 conmon[29288]: 2021-02 2021-02-10T04:38:23.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:23 smithi164 conmon[29288]: -10T04:38:22.919370+0000 osd.3 2021-02-10T04:38:23.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:23 smithi164 conmon[29288]: (osd.3) 75 : cluster [DBG] 4.10 scrub ok 2021-02-10T04:38:23.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:23 smithi164 conmon[29288]: cluster 2021-02-10T04:38:23.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:23 smithi164 conmon[29288]: 2021-02-10T04:38:23.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:23 smithi164 conmon[29288]: 2021-02-10T04:38 2021-02-10T04:38:23.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:23 smithi164 conmon[29288]: :23.215507+0000 mon.a (mon.0) 1574 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 532 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:38:23.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:23 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:38:23.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:23 smithi164 conmon[29288]: 02-10T04:38:23.223104+0000 mon.a ( 2021-02-10T04:38:23.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:23 smithi164 conmon[29288]: mon.0) 1575 : cluster [DBG] osdmap e685: 8 total, 8 up, 7 in 2021-02-10T04:38:23.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[35961]: cluster 2021-02-10T04:38:22.024022+0000 2021-02-10T04:38:23.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[35961]: osd.6 (osd.6) 69 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:38:23.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[35961]: cluster 2021-02-10T04:38:22. 2021-02-10T04:38:23.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[35961]: 024835+0000 osd.6 (osd.6) 70 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:38:23.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[35961]: cluster 2021-02-10T04:38: 2021-02-10T04:38:23.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[35961]: 22.136284+0000 osd.6 (osd.6) 71 : cluster [DBG] 2.29 scrub ok 2021-02-10T04:38:23.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[35961]: cluster 2021-02-10T04:38:23.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[35961]: 2021-02-10T04:38:22.919370+0000 osd.3 (osd.3) 75 : cluster [DBG] 4.10 scrub ok 2021-02-10T04:38:23.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[35961]: cluster 2021-02-10T04:38:23.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[35961]: 2021-02-10T04:38:23.215507+0000 mon.a (mon.0 2021-02-10T04:38:23.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[35961]: ) 1574 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 532 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:38:23.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:38:23.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[35961]: T04:38:23.223104+0000 mon.a (mon.0 2021-02-10T04:38:23.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[35961]: ) 1575 : cluster [DBG] osdmap e685: 8 total, 8 up, 7 in 2021-02-10T04:38:23.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[29319]: cluster 2021-02-10T04:38:22.024022+0000 osd.6 ( 2021-02-10T04:38:23.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[29319]: osd.6) 69 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:38:23.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[29319]: cluster 2021-02-10T04:38:22.024835+0000 osd.6 (osd.6) 70 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:38:23.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[29319]: cluster 2021-02-10T04:38:22.136284+0000 2021-02-10T04:38:23.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[29319]: osd.6 (osd 2021-02-10T04:38:23.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[29319]: .6) 71 : cluster 2021-02-10T04:38:23.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[29319]: [DBG] 2.29 scrub ok 2021-02-10T04:38:23.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:38:23.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[29319]: 02-10T04:38:22.919370 2021-02-10T04:38:23.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[29319]: +0000 osd.3 (osd.3) 75 : cluster [DBG] 2021-02-10T04:38:23.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[29319]: 4.10 scrub ok 2021-02-10T04:38:23.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:38:23.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[29319]: 04:38:23.215507+0000 mon.a 2021-02-10T04:38:23.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[29319]: (mon.0) 1574 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 532 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:38:23.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[29319]: 2021-02-10T04:38:23.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[29319]: cluster 2021-02-10T04:38 2021-02-10T04:38:23.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[29319]: :23.223104+0000 mon.a (mon.0) 1575 2021-02-10T04:38:23.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:23 smithi085 conmon[29319]: : cluster [DBG] osdmap e685: 8 total, 8 up, 7 in 2021-02-10T04:38:24.236 INFO:tasks.rados.rados.0.smithi085.stdout:2643: done (5 left) 2021-02-10T04:38:24.236 INFO:tasks.rados.rados.0.smithi085.stdout:2644: done (4 left) 2021-02-10T04:38:24.236 INFO:tasks.rados.rados.0.smithi085.stdout:2645: write (excl) oid 10 current snap is 304 2021-02-10T04:38:24.236 INFO:tasks.rados.rados.0.smithi085.stdout:2645: seq_num 922 ranges {696054=658291,2103339=753642} 2021-02-10T04:38:24.244 INFO:tasks.rados.rados.0.smithi085.stdout:2645: writing smithi08575397-10 from 696054 to 1354345 tid 1 2021-02-10T04:38:24.253 INFO:tasks.rados.rados.0.smithi085.stdout:2645: writing smithi08575397-10 from 2103339 to 2856981 tid 2 2021-02-10T04:38:24.253 INFO:tasks.rados.rados.0.smithi085.stdout:2646: write oid 32 current snap is 304 2021-02-10T04:38:24.253 INFO:tasks.rados.rados.0.smithi085.stdout:2645: finishing write tid 1 to smithi08575397-10 2021-02-10T04:38:24.253 INFO:tasks.rados.rados.0.smithi085.stdout:2646: seq_num 923 ranges {712861=733540,2161821=613653,3247187=265674} 2021-02-10T04:38:24.261 INFO:tasks.rados.rados.0.smithi085.stdout:2646: writing smithi08575397-32 from 712861 to 1446401 tid 1 2021-02-10T04:38:24.269 INFO:tasks.rados.rados.0.smithi085.stdout:2646: writing smithi08575397-32 from 2161821 to 2775474 tid 2 2021-02-10T04:38:24.274 INFO:tasks.rados.rados.0.smithi085.stdout:2646: writing smithi08575397-32 from 3247187 to 3512861 tid 3 2021-02-10T04:38:24.274 INFO:tasks.rados.rados.0.smithi085.stdout:2645: finishing write tid 2 to smithi08575397-10 2021-02-10T04:38:24.274 INFO:tasks.rados.rados.0.smithi085.stdout:2645: finishing write tid 3 to smithi08575397-10 2021-02-10T04:38:24.274 INFO:tasks.rados.rados.0.smithi085.stdout:2645: finishing write tid 4 to smithi08575397-10 2021-02-10T04:38:24.274 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 10 v 878 (ObjNum 922 snap 304 seq_num 922) dirty exists 2021-02-10T04:38:24.274 INFO:tasks.rados.rados.0.smithi085.stdout:2645: left oid 10 (ObjNum 922 snap 304 seq_num 922) 2021-02-10T04:38:24.275 INFO:tasks.rados.rados.0.smithi085.stdout:2646: finishing write tid 1 to smithi08575397-32 2021-02-10T04:38:24.275 INFO:tasks.rados.rados.0.smithi085.stdout:2645: done (5 left) 2021-02-10T04:38:24.275 INFO:tasks.rados.rados.0.smithi085.stdout:2647: snap_create 2021-02-10T04:38:24.275 INFO:tasks.rados.rados.0.smithi085.stdout:2646: finishing write tid 2 to smithi08575397-32 2021-02-10T04:38:24.276 INFO:tasks.rados.rados.0.smithi085.stdout:2646: finishing write tid 3 to smithi08575397-32 2021-02-10T04:38:24.277 INFO:tasks.rados.rados.0.smithi085.stdout:2646: finishing write tid 4 to smithi08575397-32 2021-02-10T04:38:24.278 INFO:tasks.rados.rados.0.smithi085.stdout:2646: finishing write tid 5 to smithi08575397-32 2021-02-10T04:38:24.278 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 32 v 649 (ObjNum 923 snap 304 seq_num 923) dirty exists 2021-02-10T04:38:24.278 INFO:tasks.rados.rados.0.smithi085.stdout:2646: left oid 32 (ObjNum 923 snap 304 seq_num 923) 2021-02-10T04:38:24.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:24 smithi164 conmon[29288]: cluster 2021-02-10T04:38:23.183451+0000 osd.0 2021-02-10T04:38:24.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:24 smithi164 conmon[29288]: (osd.0) 76 : cluster [DBG] 7.5 scrub ok 2021-02-10T04:38:24.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:24 smithi164 conmon[29288]: cluster 2021-02-10T04:38:23.319012+0000 osd.4 (osd.4) 1123 : cluster [DBG] 6.1a scrub ok 2021-02-10T04:38:24.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:24 smithi164 conmon[29288]: cluster 2021-02-10T04:38:24.155869+0000 mgr.y (mgr.14140) 708 : cluster [DBG] pgmap v1178: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 365 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 3.1 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2021-02-10T04:38:24.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:24 smithi164 conmon[29288]: cluster 2021-02-10T04:38:24.235618+0000 mon.a (mon.0) 1576 : cluster [DBG] osdmap e686: 8 total, 8 up, 7 in 2021-02-10T04:38:24.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:24 smithi085 conmon[29319]: cluster 2021-02-10T04:38:23.183451+0000 osd.0 ( 2021-02-10T04:38:24.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:24 smithi085 conmon[29319]: osd.0) 76 : cluster [DBG] 7.5 scrub ok 2021-02-10T04:38:24.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:24 smithi085 conmon[29319]: cluster 2021-02-10T04:38:23.319012+0000 osd.4 (osd.4) 1123 : cluster [DBG] 6.1a scrub ok 2021-02-10T04:38:24.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:24 smithi085 conmon[29319]: cluster 2021-02-10T04:38:24.155869+0000 mgr.y (mgr.14140) 708 : cluster [DBG] pgmap v1178: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 365 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 3.1 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2021-02-10T04:38:24.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:24 smithi085 conmon[29319]: cluster 2021-02-10T04:38:24.235618+0000 mon.a (mon.0) 1576 : cluster [DBG] osdmap e686: 8 total, 8 up, 7 in 2021-02-10T04:38:24.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:24 smithi085 conmon[35961]: cluster 2021-02-10T04:38:24.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:24 smithi085 conmon[35961]: 2021-02-10T04:38:24.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:24 smithi085 conmon[35961]: 2021 2021-02-10T04:38:24.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:24 smithi085 conmon[35961]: - 2021-02-10T04:38:24.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:24 smithi085 conmon[35961]: 02 2021-02-10T04:38:24.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:24 smithi085 conmon[35961]: - 2021-02-10T04:38:24.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:24 smithi085 conmon[35961]: 10 2021-02-10T04:38:24.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:24 smithi085 conmon[35961]: T04:38:23.183451+0000 osd.0 2021-02-10T04:38:24.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:24 smithi085 conmon[35961]: (osd.0) 76 : cluster [DBG] 7.5 scrub ok 2021-02-10T04:38:24.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:24 smithi085 conmon[35961]: cluster 2021-02-10T04:38:24.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:24 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:38:24.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:24 smithi085 conmon[35961]: 04:38:23.319012+0000 osd.4 (osd.4) 1123 : 2021-02-10T04:38:24.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:24 smithi085 conmon[35961]: cluster [DBG] 6.1a scrub ok 2021-02-10T04:38:24.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:24 smithi085 conmon[35961]: cluster 2021-02-10T04:38:24.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:24 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:38:24.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:24 smithi085 conmon[35961]: 38:24.155869+0000 mgr.y (mgr.14140) 708 : 2021-02-10T04:38:24.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:24 smithi085 conmon[35961]: cluster [DBG] pgmap v1178: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 365 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 3.1 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2021-02-10T04:38:24.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:24 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:38:24.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:24 smithi085 conmon[35961]: T04:38:24.235618+0000 mon.a (mon 2021-02-10T04:38:24.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:24 smithi085 conmon[35961]: .0) 1576 : cluster [DBG] osdmap e686: 8 total, 8 up, 7 in 2021-02-10T04:38:25.242 INFO:tasks.rados.rados.0.smithi085.stdout:2646: done (5 left) 2021-02-10T04:38:25.242 INFO:tasks.rados.rados.0.smithi085.stdout:2647: done (4 left) 2021-02-10T04:38:25.242 INFO:tasks.rados.rados.0.smithi085.stdout:2648: snap_remove snap 297 2021-02-10T04:38:25.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:25 smithi164 conmon[29288]: cluster 2021-02-10T04:38:25.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:25 smithi164 conmon[29288]: 2021-02-10T04:38:25.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:25 smithi164 conmon[29288]: 2021 2021-02-10T04:38:25.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:25 smithi164 conmon[29288]: -02-10T04: 2021-02-10T04:38:25.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:25 smithi164 conmon[29288]: 38:24.163197+0000 osd.5 (osd.5) 73 : cluster [DBG] 2.c scrub ok 2021-02-10T04:38:25.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:25 smithi164 conmon[29288]: cluster 2021-02-10T04:38:25.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:25 smithi164 conmon[29288]: 2021-02-10T04:38:25.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:25 smithi164 conmon[29288]: 2021 2021-02-10T04:38:25.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:25 smithi164 conmon[29288]: - 2021-02-10T04:38:25.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:25 smithi164 conmon[29288]: 02 2021-02-10T04:38:25.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:25 smithi164 conmon[29288]: - 2021-02-10T04:38:25.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:25 smithi164 conmon[29288]: 10T04:38:25 2021-02-10T04:38:25.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:25 smithi164 conmon[29288]: .234489+0000 mon.a (mon.0) 1577 : cluster [DBG] osdmap e687: 8 total, 8 up, 7 in 2021-02-10T04:38:25.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[29319]: cluster 2021-02-10T04:38:25.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[29319]: 2021-02-10T04:38:25.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[29319]: 2021 2021-02-10T04:38:25.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[29319]: - 2021-02-10T04:38:25.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[29319]: 02 2021-02-10T04:38:25.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[29319]: - 2021-02-10T04:38:25.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[29319]: 10 2021-02-10T04:38:25.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[29319]: T 2021-02-10T04:38:25.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[29319]: 04 2021-02-10T04:38:25.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[29319]: :38:24.163197+0000 osd.5 2021-02-10T04:38:25.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[29319]: (osd.5) 73 : cluster [DBG] 2.c scrub ok 2021-02-10T04:38:25.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[29319]: cluster 2021-02-10T04:38:25.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[29319]: 2021- 2021-02-10T04:38:25.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[29319]: 02-10T04:38:25 2021-02-10T04:38:25.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[29319]: .234489+0000 mon.a ( 2021-02-10T04:38:25.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[29319]: mon.0) 1577 2021-02-10T04:38:25.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[29319]: : cluster [DBG] osdmap e687: 8 total, 8 up, 7 in 2021-02-10T04:38:25.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[35961]: cluster 2021-02-10T04:38:25.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[35961]: 2021-02-10T04:38:25.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[35961]: 2021 2021-02-10T04:38:25.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[35961]: - 2021-02-10T04:38:25.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[35961]: 02 2021-02-10T04:38:25.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[35961]: - 2021-02-10T04:38:25.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[35961]: 10 2021-02-10T04:38:25.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[35961]: T 2021-02-10T04:38:25.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[35961]: 04 2021-02-10T04:38:25.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[35961]: : 2021-02-10T04:38:25.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[35961]: 38 2021-02-10T04:38:25.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[35961]: : 2021-02-10T04:38:25.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[35961]: 24 2021-02-10T04:38:25.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[35961]: . 2021-02-10T04:38:25.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[35961]: 163197 2021-02-10T04:38:25.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[35961]: +0000 2021-02-10T04:38:25.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[35961]: 2021-02-10T04:38:25.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[35961]: osd.5 2021-02-10T04:38:25.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[35961]: ( 2021-02-10T04:38:25.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[35961]: osd 2021-02-10T04:38:25.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[35961]: . 2021-02-10T04:38:25.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[35961]: 5 2021-02-10T04:38:25.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[35961]: ) 2021-02-10T04:38:25.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[35961]: 73 : cluster [DBG] 2.c scrub ok 2021-02-10T04:38:25.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:38:25.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[35961]: -02-10T04:38:25.234489+0000 mon.a ( 2021-02-10T04:38:25.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:25 smithi085 conmon[35961]: mon.0) 1577 : cluster [DBG] osdmap e687: 8 total, 8 up, 7 in 2021-02-10T04:38:26.241 INFO:tasks.rados.rados.0.smithi085.stdout:2648: done (4 left) 2021-02-10T04:38:29.704 INFO:tasks.rados.rados.0.smithi085.stdout:2649: snap_create 2021-02-10T04:38:29.715 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:26 smithi164 conmon[29288]: cluster 2021-02-10T04:38:25.860240+0000 osd.3 2021-02-10T04:38:29.715 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:26 smithi164 conmon[29288]: (osd.3) 76 : cluster [DBG] 4.1d scrub ok 2021-02-10T04:38:29.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:26 smithi164 conmon[29288]: cluster 2021-02-10T04:38:26.124799+0000 osd.0 (osd.0) 77 : cluster [DBG] 6.0 scrub ok 2021-02-10T04:38:29.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:26 smithi164 conmon[29288]: cluster 2021-02-10T04:38:26.156330+0000 2021-02-10T04:38:29.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:26 smithi164 conmon[29288]: mgr.y (mgr.14140) 709 : cluster [DBG] pgmap v1181: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 367 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 3.1 MiB/s rd, 1.7 MiB/s wr, 11 op/s 2021-02-10T04:38:29.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:26 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:38:29.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:26 smithi164 conmon[29288]: 38:26.241206+0000 mon.a (mon.0) 1578 : cluster 2021-02-10T04:38:29.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:26 smithi164 conmon[29288]: [DBG] osdmap e688: 8 total, 8 up, 7 in 2021-02-10T04:38:29.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:27 smithi164 conmon[29288]: cluster 2021-02-10T04:38:26.127280+0000 osd.6 (osd.6) 72 : cluster [DBG] 2.14 scrub ok 2021-02-10T04:38:29.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:27 smithi164 conmon[29288]: cluster 2021-02-10T04:38:26.845191+0000 osd.3 (osd.3) 77 : cluster [DBG] 4.17 scrub ok 2021-02-10T04:38:29.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:27 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:38:29.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:27 smithi164 conmon[29288]: -02-10T04:38:27.092691+0000 osd.0 (osd.0) 78 : cluster [DBG] 5.3 scrub ok 2021-02-10T04:38:29.717 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:27 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:38:29.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:27 smithi164 conmon[29288]: -10T04:38:27.237675+0000 mon.a 2021-02-10T04:38:29.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:27 smithi164 conmon[29288]: (mon.0) 2021-02-10T04:38:29.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:27 smithi164 conmon[29288]: 1579 : cluster [DBG] osdmap e689: 8 total, 8 up, 7 in 2021-02-10T04:38:29.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:28 smithi164 conmon[29288]: cluster 2021-02-10T04:38:27. 2021-02-10T04:38:29.718 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:28 smithi164 conmon[29288]: 225482+0000 osd.5 (osd.5) 74 : cluster [DBG] 7.e scrub ok 2021-02-10T04:38:29.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:28 smithi164 conmon[29288]: cluster 2021-02-10T04:38:27.895732+0000 osd.3 (osd.3) 78 : cluster [DBG] 7.3 scrub ok 2021-02-10T04:38:29.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:28 smithi164 conmon[29288]: cluster 2021-02-10T04:38:28.156937+0000 mgr.y ( 2021-02-10T04:38:29.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:28 smithi164 conmon[29288]: mgr.14140) 710 : cluster [DBG] pgmap v1184: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 370 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 1.6 MiB/s wr, 5 op/s 2021-02-10T04:38:29.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:28 smithi164 conmon[29288]: cluster 2021-02-10T04:38: 2021-02-10T04:38:29.719 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:28 smithi164 conmon[29288]: 28.222599+0000 mon.a (mon.0) 1580 : cluster [DBG] osdmap e690: 8 total, 8 up, 7 in 2021-02-10T04:38:29.720 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:38:28 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:38:29.720 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:26 smithi085 conmon[29319]: cluster 2021-02-10T04:38:25.860240+0000 osd.3 (osd.3) 76 2021-02-10T04:38:29.720 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:26 smithi085 conmon[29319]: : cluster [DBG] 4.1d scrub ok 2021-02-10T04:38:29.720 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:26 smithi085 conmon[29319]: cluster 2021-02-10T04:38:26.124799+0000 osd.0 (osd.0) 77 : cluster [DBG] 6.0 scrub ok 2021-02-10T04:38:29.721 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:26 smithi085 conmon[29319]: cluster 2021-02-10T04:38:26.156330+0000 mgr.y ( 2021-02-10T04:38:29.721 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:26 smithi085 conmon[29319]: mgr.14140) 709 : cluster [DBG] pgmap v1181: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 367 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 3.1 MiB/s rd, 1.7 MiB/s wr, 11 op/s 2021-02-10T04:38:29.721 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:26 smithi085 conmon[29319]: cluster 2021-02-10T04:38:29.721 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:26 smithi085 conmon[29319]: 2021-02-10T04:38:26.241206 2021-02-10T04:38:29.721 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:26 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1578 : cluster [DBG] 2021-02-10T04:38:29.721 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:26 smithi085 conmon[29319]: osdmap e688: 8 total, 8 up, 7 in 2021-02-10T04:38:29.721 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[29319]: cluster 2021-02-10T04:38:26.127280+0000 osd.6 ( 2021-02-10T04:38:29.721 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[29319]: osd.6) 72 : cluster [DBG] 2.14 scrub ok 2021-02-10T04:38:29.721 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[29319]: cluster 2021-02-10T04:38:26.845191+0000 osd.3 (osd.3) 77 : cluster [DBG] 4.17 scrub ok 2021-02-10T04:38:29.721 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[29319]: cluster 2021-02-10T04:38:27.092691+0000 osd.0 ( 2021-02-10T04:38:29.721 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[29319]: osd.0) 78 : cluster [DBG] 5.3 scrub ok 2021-02-10T04:38:29.721 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:38:29.721 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[29319]: 04:38:27.237675+0000 2021-02-10T04:38:29.722 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[29319]: mon.a (mon.0) 1579 : cluster [DBG] osdmap e689: 8 total, 8 up, 7 in 2021-02-10T04:38:29.722 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[29319]: 2021-02-10T04:38:29.722 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[29319]: cluster 2021-02-10T04:38:27.225482+0000 osd.5 (osd.5) 74 : cluster [DBG] 2021-02-10T04:38:29.722 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[29319]: 7.e scrub ok 2021-02-10T04:38:29.722 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[29319]: cluster 2021-02-10T04:38:27.895732+0000 osd.3 (osd.3) 78 : cluster [DBG] 7.3 scrub ok 2021-02-10T04:38:29.722 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[29319]: cluster 2021-02-10T04:38:29.722 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[29319]: 2021-02-10T04:38:28.156937+0000 mgr.y (mgr.14140) 710 : cluster [DBG] pgmap v1184: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 370 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 1.6 MiB/s wr, 5 op/s 2021-02-10T04:38:29.722 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:38:29.723 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[29319]: -10T04:38:28. 2021-02-10T04:38:29.723 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[29319]: 222599+0000 mon.a ( 2021-02-10T04:38:29.723 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[29319]: mon.0) 1580 : cluster [DBG] osdmap e690: 8 total, 8 up, 7 in 2021-02-10T04:38:29.723 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:26 smithi085 conmon[35961]: cluster 2021-02-10T04:38:29.723 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:26 smithi085 conmon[35961]: 2021-02-10T04:38:29.723 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:26 smithi085 conmon[35961]: 2021 2021-02-10T04:38:29.724 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:26 smithi085 conmon[35961]: - 2021-02-10T04:38:29.724 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:26 smithi085 conmon[35961]: 02 2021-02-10T04:38:29.724 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:26 smithi085 conmon[35961]: - 2021-02-10T04:38:29.724 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:26 smithi085 conmon[35961]: 10T04:38:25 2021-02-10T04:38:29.724 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:26 smithi085 conmon[35961]: .860240+0000 osd.3 (osd.3) 76 : cluster [DBG] 4.1d scrub ok 2021-02-10T04:38:29.724 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:26 smithi085 conmon[35961]: cluster 2021-02-10T04:38:29.725 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:26 smithi085 conmon[35961]: 2021-02-10T04:38:29.725 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:26 smithi085 conmon[35961]: 2021-02-10T04:38:26 2021-02-10T04:38:29.725 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:26 smithi085 conmon[35961]: .124799+0000 osd.0 (osd.0) 77 : cluster [DBG] 6.0 scrub ok 2021-02-10T04:38:29.725 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:26 smithi085 conmon[35961]: cluster 2021-02-10T04:38:29.725 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:26 smithi085 conmon[35961]: 2021-02-10T04:38: 2021-02-10T04:38:29.725 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:26 smithi085 conmon[35961]: 26.156330+0000 mgr.y (mgr.14140) 709 : cluster [DBG] pgmap v1181: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 367 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 3.1 MiB/s rd, 1.7 MiB/s wr, 11 op/s 2021-02-10T04:38:29.726 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:26 smithi085 conmon[35961]: 2021-02-10T04:38:29.726 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:26 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:38:29.726 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:26 smithi085 conmon[35961]: 38:26.241206+0000 mon.a (mon 2021-02-10T04:38:29.726 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:26 smithi085 conmon[35961]: .0) 1578 : cluster [DBG] osdmap e688: 8 total, 8 up, 7 in 2021-02-10T04:38:29.726 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[35961]: cluster 2021-02-10T04:38:29.726 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[35961]: 2021-02-10T04:38:29.727 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[35961]: 2021 2021-02-10T04:38:29.727 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[35961]: - 2021-02-10T04:38:29.727 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[35961]: 02 2021-02-10T04:38:29.728 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[35961]: - 2021-02-10T04:38:29.728 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[35961]: 10 2021-02-10T04:38:29.728 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[35961]: T 2021-02-10T04:38:29.728 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[35961]: 04 2021-02-10T04:38:29.728 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[35961]: : 2021-02-10T04:38:29.729 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[35961]: 38 2021-02-10T04:38:29.729 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[35961]: :26.127280+0000 osd.6 (osd. 2021-02-10T04:38:29.729 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[35961]: 6) 72 : cluster [DBG] 2.14 scrub ok 2021-02-10T04:38:29.729 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[35961]: cluster 2021-02-10T04:38:29.729 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:38:29.729 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[35961]: 04:38:26.845191+0000 osd.3 (osd.3) 77 : cluster [DBG] 4.17 scrub ok 2021-02-10T04:38:29.730 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:38:29.730 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[35961]: -02-10T04:38:27.092691 2021-02-10T04:38:29.730 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[35961]: +0000 osd.0 (osd.0) 78 : cluster [DBG] 2021-02-10T04:38:29.730 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[35961]: 5.3 scrub ok 2021-02-10T04:38:29.730 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:38:29.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[35961]: 04:38:27.237675+0000 mon.a (mon.0 2021-02-10T04:38:29.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[35961]: ) 1579 : cluster [DBG] 2021-02-10T04:38:29.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:27 smithi085 conmon[35961]: osdmap e689: 8 total, 8 up, 7 in 2021-02-10T04:38:29.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[35961]: cluster 2021-02-10T04:38:29.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[35961]: 2021-02-10T04:38:29.732 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[35961]: 2021 2021-02-10T04:38:29.732 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[35961]: - 2021-02-10T04:38:29.732 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[35961]: 02 2021-02-10T04:38:29.732 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[35961]: - 2021-02-10T04:38:29.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[35961]: 10T04:38:27.225482+0000 2021-02-10T04:38:29.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[35961]: osd.5 (osd.5) 74 : cluster [DBG] 7.e scrub ok 2021-02-10T04:38:29.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[35961]: cluster 2021-02-10T04:38:29.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[35961]: 2021-02-10T04:38:29.733 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[35961]: 2021 2021-02-10T04:38:29.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[35961]: - 2021-02-10T04:38:29.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[35961]: 02 2021-02-10T04:38:29.734 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[35961]: - 2021-02-10T04:38:29.735 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[35961]: 10 2021-02-10T04:38:29.735 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[35961]: T 2021-02-10T04:38:29.735 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[35961]: 04:38:27.895732+0000 osd.3 (osd. 2021-02-10T04:38:29.735 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[35961]: 3) 78 : cluster [DBG] 7.3 scrub ok 2021-02-10T04:38:29.735 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[35961]: cluster 2021-02-10T04:38:29.735 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[35961]: 2021-02-10T04:38: 2021-02-10T04:38:29.735 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[35961]: 28.156937+0000 mgr.y (mgr 2021-02-10T04:38:29.735 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[35961]: .14140) 710 : cluster [DBG] pgmap v1184: 168 pgs: 154 active+clean, 8 active+clean+snaptrim, 6 active+clean+snaptrim_wait; 370 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 1.6 MiB/s wr, 5 op/s 2021-02-10T04:38:29.736 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[35961]: 2021-02-10T04:38:29.736 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[35961]: cluster 2021-02-10T04:38: 2021-02-10T04:38:29.736 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[35961]: 28.222599+0000 mon.a (mon.0) 1580 : 2021-02-10T04:38:29.736 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:28 smithi085 conmon[35961]: cluster [DBG] osdmap e690: 8 total, 8 up, 7 in 2021-02-10T04:38:29.736 INFO:tasks.rados.rados.0.smithi085.stdout:2649: done (4 left) 2021-02-10T04:38:29.736 INFO:tasks.rados.rados.0.smithi085.stdout:2650: write oid 7 current snap is 306 2021-02-10T04:38:29.736 INFO:tasks.rados.rados.0.smithi085.stdout:2650: seq_num 924 ranges {729668=408789,1820303=757311,3274610=255058} 2021-02-10T04:38:29.736 INFO:tasks.rados.rados.0.smithi085.stdout:2650: writing smithi08575397-7 from 729668 to 1138457 tid 1 2021-02-10T04:38:29.736 INFO:tasks.rados.rados.0.smithi085.stdout:2650: writing smithi08575397-7 from 1820303 to 2577614 tid 2 2021-02-10T04:38:29.736 INFO:tasks.rados.rados.0.smithi085.stdout:2650: writing smithi08575397-7 from 3274610 to 3529668 tid 3 2021-02-10T04:38:29.737 INFO:tasks.rados.rados.0.smithi085.stdout:2651: snap_create 2021-02-10T04:38:29.737 INFO:tasks.rados.rados.0.smithi085.stdout:2650: finishing write tid 1 to smithi08575397-7 2021-02-10T04:38:29.737 INFO:tasks.rados.rados.0.smithi085.stdout:2650: finishing write tid 2 to smithi08575397-7 2021-02-10T04:38:29.737 INFO:tasks.rados.rados.0.smithi085.stdout:2650: finishing write tid 3 to smithi08575397-7 2021-02-10T04:38:29.737 INFO:tasks.rados.rados.0.smithi085.stdout:2650: finishing write tid 4 to smithi08575397-7 2021-02-10T04:38:29.737 INFO:tasks.rados.rados.0.smithi085.stdout:2650: finishing write tid 5 to smithi08575397-7 2021-02-10T04:38:29.737 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 1181 (ObjNum 924 snap 306 seq_num 924) dirty exists 2021-02-10T04:38:29.737 INFO:tasks.rados.rados.0.smithi085.stdout:2650: left oid 7 (ObjNum 924 snap 306 seq_num 924) 2021-02-10T04:38:29.737 INFO:tasks.rados.rados.0.smithi085.stdout:2650: done (5 left) 2021-02-10T04:38:29.737 INFO:tasks.rados.rados.0.smithi085.stdout:2651: done (4 left) 2021-02-10T04:38:29.737 INFO:tasks.rados.rados.0.smithi085.stdout:2652: delete oid 14 current snap is 307 2021-02-10T04:38:29.737 INFO:tasks.rados.rados.0.smithi085.stdout:2652: done (4 left) 2021-02-10T04:38:29.737 INFO:tasks.rados.rados.0.smithi085.stdout:2653: snap_remove snap 304 2021-02-10T04:38:29.738 INFO:tasks.rados.rados.0.smithi085.stdout:2653: done (4 left) 2021-02-10T04:38:29.738 INFO:tasks.rados.rados.0.smithi085.stdout:2654: rollback oid 30 current snap is 307 2021-02-10T04:38:29.738 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 30 to 303 2021-02-10T04:38:29.738 INFO:tasks.rados.rados.0.smithi085.stdout:2655: rollback oid 37 current snap is 307 2021-02-10T04:38:29.738 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 37 to 289 2021-02-10T04:38:29.738 INFO:tasks.rados.rados.0.smithi085.stdout:2656: write oid 40 current snap is 307 2021-02-10T04:38:29.738 INFO:tasks.rados.rados.0.smithi085.stdout:2656: seq_num 926 ranges {763282=559287,1820914=760980,3213103=350179} 2021-02-10T04:38:29.738 INFO:tasks.rados.rados.0.smithi085.stdout:2656: writing smithi08575397-40 from 763282 to 1322569 tid 1 2021-02-10T04:38:29.738 INFO:tasks.rados.rados.0.smithi085.stdout:2656: writing smithi08575397-40 from 1820914 to 2581894 tid 2 2021-02-10T04:38:29.738 INFO:tasks.rados.rados.0.smithi085.stdout:2656: writing smithi08575397-40 from 3213103 to 3563282 tid 3 2021-02-10T04:38:29.738 INFO:tasks.rados.rados.0.smithi085.stdout:2657: rollback oid 11 current snap is 307 2021-02-10T04:38:29.738 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 11 to 294 2021-02-10T04:38:29.739 INFO:tasks.rados.rados.0.smithi085.stdout:2655: finishing rollback tid 0 to smithi08575397-37 2021-02-10T04:38:29.739 INFO:tasks.rados.rados.0.smithi085.stdout:2655: finishing rollback tid 1 to smithi08575397-37 2021-02-10T04:38:29.739 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 1393 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-02-10T04:38:29.739 INFO:tasks.rados.rados.0.smithi085.stdout:2655: done (7 left) 2021-02-10T04:38:29.739 INFO:tasks.rados.rados.0.smithi085.stdout:2658: write (excl) oid 29 current snap is 307 2021-02-10T04:38:29.739 INFO:tasks.rados.rados.0.smithi085.stdout:2654: finishing rollback tid 0 to smithi08575397-30 2021-02-10T04:38:29.739 INFO:tasks.rados.rados.0.smithi085.stdout:2658: seq_num 927 ranges {780089=634536,1879396=620991,3240526=339563} 2021-02-10T04:38:29.739 INFO:tasks.rados.rados.0.smithi085.stdout:2658: writing smithi08575397-29 from 780089 to 1414625 tid 1 2021-02-10T04:38:29.739 INFO:tasks.rados.rados.0.smithi085.stdout:2658: writing smithi08575397-29 from 1879396 to 2500387 tid 2 2021-02-10T04:38:29.739 INFO:tasks.rados.rados.0.smithi085.stdout:2658: writing smithi08575397-29 from 3240526 to 3580089 tid 3 2021-02-10T04:38:29.740 INFO:tasks.rados.rados.0.smithi085.stdout:2659: snap_remove snap 302 2021-02-10T04:38:29.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:29 smithi164 conmon[29288]: audit 2021-02-10T04:38:28.270097+0000 2021-02-10T04:38:29.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:29 smithi164 conmon[29288]: mgr.y (mgr.14140) 711 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:38:29.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:29 smithi164 conmon[29288]: cluster 2021-02-10T04:38:28.525215+0000 mon.a (mon.0) 1581 : cluster [INF] 2021-02-10T04:38:29.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:29 smithi164 conmon[29288]: Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 532 sec, osd.4 has slow ops) 2021-02-10T04:38:29.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:29 smithi164 conmon[29288]: cluster 2021-02-10T04:38: 2021-02-10T04:38:29.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:29 smithi164 conmon[29288]: 29.239920+0000 mon.a (mon.0) 1582 : cluster [DBG] osdmap e691: 8 total, 8 up, 7 in 2021-02-10T04:38:29.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:29 smithi164 conmon[29288]: 2021-02-10T04:38:29.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:29 smithi085 conmon[29319]: audit 2021-02-10T04:38:28.270097+0000 mgr.y (mgr. 2021-02-10T04:38:29.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:29 smithi085 conmon[29319]: 14140) 711 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:38:29.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:29 smithi085 conmon[29319]: cluster 2021-02-10T04:38:28.525215+0000 mon.a (mon.0) 1581 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 532 sec, osd.4 has slow ops) 2021-02-10T04:38:29.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:29 smithi085 conmon[29319]: cluster 2021-02-10T04:38:29.239920+0000 mon.a (mon 2021-02-10T04:38:29.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:29 smithi085 conmon[29319]: .0) 1582 : cluster [DBG] osdmap e691: 8 total, 8 up, 7 in 2021-02-10T04:38:29.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:29 smithi085 conmon[35961]: audit 2021-02-10T04:38:28. 2021-02-10T04:38:29.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:29 smithi085 conmon[35961]: 270097+0000 mgr.y (mgr.14140) 711 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:38:29.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:29 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:38:29.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:29 smithi085 conmon[35961]: 04:38:28.525215+0000 mon.a (mon.0) 1581 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 532 sec, osd.4 has slow ops) 2021-02-10T04:38:29.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:29 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:38:29.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:29 smithi085 conmon[35961]: -02-10T04:38:29.239920+0000 mon.a (mon 2021-02-10T04:38:29.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:29 smithi085 conmon[35961]: .0) 1582 : cluster [DBG] osdmap e691: 8 total, 8 up, 7 in 2021-02-10T04:38:30.236 INFO:tasks.rados.rados.0.smithi085.stdout:2659: done (8 left) 2021-02-10T04:38:30.237 INFO:tasks.rados.rados.0.smithi085.stdout:2660: rollback oid 48 current snap is 307 2021-02-10T04:38:30.237 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 48 to 287 2021-02-10T04:38:30.237 INFO:tasks.rados.rados.0.smithi085.stdout:2654: finishing rollback tid 1 to smithi08575397-30 2021-02-10T04:38:30.237 INFO:tasks.rados.rados.0.smithi085.stdout:2654: finishing rollback tid 2 to smithi08575397-30 2021-02-10T04:38:30.237 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 1335 (ObjNum 912 snap 299 seq_num 912) dirty exists 2021-02-10T04:38:30.237 INFO:tasks.rados.rados.0.smithi085.stdout:2656: finishing write tid 1 to smithi08575397-40 2021-02-10T04:38:30.237 INFO:tasks.rados.rados.0.smithi085.stdout:2656: finishing write tid 2 to smithi08575397-40 2021-02-10T04:38:30.237 INFO:tasks.rados.rados.0.smithi085.stdout:2656: finishing write tid 3 to smithi08575397-40 2021-02-10T04:38:30.237 INFO:tasks.rados.rados.0.smithi085.stdout:2656: finishing write tid 4 to smithi08575397-40 2021-02-10T04:38:30.237 INFO:tasks.rados.rados.0.smithi085.stdout:2656: finishing write tid 5 to smithi08575397-40 2021-02-10T04:38:30.237 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 1398 (ObjNum 926 snap 307 seq_num 926) dirty exists 2021-02-10T04:38:30.238 INFO:tasks.rados.rados.0.smithi085.stdout:2656: left oid 40 (ObjNum 926 snap 307 seq_num 926) 2021-02-10T04:38:30.238 INFO:tasks.rados.rados.0.smithi085.stdout:2657: finishing rollback tid 0 to smithi08575397-11 2021-02-10T04:38:30.238 INFO:tasks.rados.rados.0.smithi085.stdout:2657: finishing rollback tid 1 to smithi08575397-11 2021-02-10T04:38:30.238 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 1401 (ObjNum 82830800 snap 22030 seq_num 1215200448) dirty dne 2021-02-10T04:38:30.238 INFO:tasks.rados.rados.0.smithi085.stdout:2658: finishing write tid 1 to smithi08575397-29 2021-02-10T04:38:30.238 INFO:tasks.rados.rados.0.smithi085.stdout:2658: finishing write tid 2 to smithi08575397-29 2021-02-10T04:38:30.238 INFO:tasks.rados.rados.0.smithi085.stdout:2658: finishing write tid 3 to smithi08575397-29 2021-02-10T04:38:30.238 INFO:tasks.rados.rados.0.smithi085.stdout:2658: finishing write tid 4 to smithi08575397-29 2021-02-10T04:38:30.238 INFO:tasks.rados.rados.0.smithi085.stdout:2658: finishing write tid 5 to smithi08575397-29 2021-02-10T04:38:30.238 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 29 v 653 (ObjNum 927 snap 307 seq_num 927) dirty exists 2021-02-10T04:38:30.238 INFO:tasks.rados.rados.0.smithi085.stdout:2658: left oid 29 (ObjNum 927 snap 307 seq_num 927) 2021-02-10T04:38:30.238 INFO:tasks.rados.rados.0.smithi085.stdout:2654: done (8 left) 2021-02-10T04:38:30.239 INFO:tasks.rados.rados.0.smithi085.stdout:2656: done (7 left) 2021-02-10T04:38:30.239 INFO:tasks.rados.rados.0.smithi085.stdout:2657: done (6 left) 2021-02-10T04:38:30.239 INFO:tasks.rados.rados.0.smithi085.stdout:2658: done (5 left) 2021-02-10T04:38:30.239 INFO:tasks.rados.rados.0.smithi085.stdout:2661: delete oid 38 current snap is 307 2021-02-10T04:38:30.243 INFO:tasks.rados.rados.0.smithi085.stdout:2660: finishing rollback tid 0 to smithi08575397-48 2021-02-10T04:38:30.244 INFO:tasks.rados.rados.0.smithi085.stdout:2660: finishing rollback tid 1 to smithi08575397-48 2021-02-10T04:38:30.244 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 975 (ObjNum 1 snap 32576 seq_num 82830288) dirty dne 2021-02-10T04:38:30.247 INFO:tasks.rados.rados.0.smithi085.stdout:2660: done (5 left) 2021-02-10T04:38:30.247 INFO:tasks.rados.rados.0.smithi085.stdout:2661: done (4 left) 2021-02-10T04:38:30.247 INFO:tasks.rados.rados.0.smithi085.stdout:2662: read oid 5 snap -1 2021-02-10T04:38:30.247 INFO:tasks.rados.rados.0.smithi085.stdout:2662: expect deleted 2021-02-10T04:38:30.247 INFO:tasks.rados.rados.0.smithi085.stdout:2663: copy_from oid 23 from oid 7 current snap is 307 2021-02-10T04:38:30.248 INFO:tasks.rados.rados.0.smithi085.stdout:2664: rollback oid 28 current snap is 307 2021-02-10T04:38:30.248 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 28 to 298 2021-02-10T04:38:30.248 INFO:tasks.rados.rados.0.smithi085.stdout:2665: snap_create 2021-02-10T04:38:30.251 INFO:tasks.rados.rados.0.smithi085.stdout:2664: finishing rollback tid 1 to smithi08575397-28 2021-02-10T04:38:30.253 INFO:tasks.rados.rados.0.smithi085.stdout:2664: finishing rollback tid 2 to smithi08575397-28 2021-02-10T04:38:30.253 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 1275 (ObjNum 892 snap 289 seq_num 892) dirty exists 2021-02-10T04:38:30.274 INFO:tasks.rados.rados.0.smithi085.stdout:2663: finishing copy_from to smithi08575397-23 2021-02-10T04:38:30.275 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 1181 (ObjNum 924 snap 306 seq_num 924) dirty exists 2021-02-10T04:38:30.275 INFO:tasks.rados.rados.0.smithi085.stdout:2663: finishing copy_from racing read to smithi08575397-23 2021-02-10T04:38:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:30 smithi164 conmon[29288]: cluster 2021-02-10T04:38:30.157450+0000 mgr.y ( 2021-02-10T04:38:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:30 smithi164 conmon[29288]: mgr.14140) 712 : cluster [DBG] pgmap v1187: 168 pgs: 155 active+clean, 8 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 368 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 756 KiB/s wr, 4 op/s 2021-02-10T04:38:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:30 smithi164 conmon[29288]: cluster 2021-02-10T04:38:30.236090+0000 mon.a (mon.0) 1583 : 2021-02-10T04:38:30.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:30 smithi164 conmon[29288]: cluster [DBG] osdmap e692: 8 total, 8 up, 7 in 2021-02-10T04:38:30.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:30 smithi085 conmon[29319]: cluster 2021-02-10T04:38:30.157450+0000 mgr.y ( 2021-02-10T04:38:30.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:30 smithi085 conmon[29319]: mgr.14140) 712 : cluster [DBG] pgmap v1187: 168 pgs: 155 active+clean, 8 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 368 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 756 KiB/s wr, 4 op/s 2021-02-10T04:38:30.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:30 smithi085 conmon[29319]: cluster 2021-02-10T04:38:30.236090+0000 mon.a (mon.0) 1583 : cluster [DBG] osdmap e692: 8 total, 8 up, 7 in 2021-02-10T04:38:30.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:30 smithi085 conmon[35961]: cluster 2021-02-10T04:38:30.157450 2021-02-10T04:38:30.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:30 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 712 : cluster [DBG] pgmap v1187: 168 pgs: 155 active+clean, 8 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 368 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 756 KiB/s wr, 4 op/s 2021-02-10T04:38:30.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:30 smithi085 conmon[35961]: cluster 2021-02-10T04:38:30. 2021-02-10T04:38:30.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:30 smithi085 conmon[35961]: 236090+0000 mon.a (mon.0) 1583 : cluster [DBG] osdmap e692: 8 total, 8 up, 7 in 2021-02-10T04:38:30.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:30 smithi085 conmon[35961]: 2021-02-10T04:38:31.239 INFO:tasks.rados.rados.0.smithi085.stdout:2662: done (7 left) 2021-02-10T04:38:31.239 INFO:tasks.rados.rados.0.smithi085.stdout:2663: done (6 left) 2021-02-10T04:38:31.239 INFO:tasks.rados.rados.0.smithi085.stdout:2664: done (5 left) 2021-02-10T04:38:31.239 INFO:tasks.rados.rados.0.smithi085.stdout:2665: done (4 left) 2021-02-10T04:38:31.239 INFO:tasks.rados.rados.0.smithi085.stdout:2666: snap_remove snap 307 2021-02-10T04:38:31.353 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:38:31.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[49106]: debug 2021-02-10T04:38:31.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[49106]: 2021-02-10T04:38:31.176+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:38:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:31 smithi164 conmon[29288]: cluster 2021-02-10T04:38:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:31 smithi164 conmon[29288]: 2021-02-10T04:38:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:31 smithi164 conmon[29288]: 2021 2021-02-10T04:38:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:31 smithi164 conmon[29288]: - 2021-02-10T04:38:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:31 smithi164 conmon[29288]: 02 2021-02-10T04:38:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:31 smithi164 conmon[29288]: - 2021-02-10T04:38:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:31 smithi164 conmon[29288]: 10 2021-02-10T04:38:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:31 smithi164 conmon[29288]: T 2021-02-10T04:38:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:31 smithi164 conmon[29288]: 04 2021-02-10T04:38:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:31 smithi164 conmon[29288]: : 2021-02-10T04:38:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:31 smithi164 conmon[29288]: 38:30.276204+0000 osd.3 (osd.3) 79 : 2021-02-10T04:38:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:31 smithi164 conmon[29288]: cluster [DBG] purged_snaps scrub starts 2021-02-10T04:38:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:31 smithi164 conmon[29288]: cluster 2021-02-10T04:38:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:31 smithi164 conmon[29288]: 2021-02-10T04:38:30.277252+0000 osd.3 (osd.3) 80 : cluster 2021-02-10T04:38:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:31 smithi164 conmon[29288]: [DBG] purged_snaps scrub ok 2021-02-10T04:38:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:31 smithi164 conmon[29288]: cluster 2021-02-10T04:38:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:31 smithi164 conmon[29288]: 2021-02-10T04:38:31.121035+0000 osd.0 (osd.0) 79 2021-02-10T04:38:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:31 smithi164 conmon[29288]: : cluster [DBG] 5.b scrub ok 2021-02-10T04:38:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:31 smithi164 conmon[29288]: cluster 2021-02-10T04:38:31.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:31 smithi164 conmon[29288]: 2021-02-10T04:38:31.238369+0000 mon.a (mon.0) 1584 2021-02-10T04:38:31.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:31 smithi164 conmon[29288]: : cluster [DBG] osdmap e693: 8 total, 8 up, 7 in 2021-02-10T04:38:31.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: cluster 2021-02-10T04:38:31.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: 2021-02-10T04:38:31.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: 2021 2021-02-10T04:38:31.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: - 2021-02-10T04:38:31.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: 02 2021-02-10T04:38:31.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: - 2021-02-10T04:38:31.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: 10 2021-02-10T04:38:31.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: T 2021-02-10T04:38:31.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: 04 2021-02-10T04:38:31.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: : 2021-02-10T04:38:31.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: 38 2021-02-10T04:38:31.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: : 2021-02-10T04:38:31.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: 30 2021-02-10T04:38:31.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: . 2021-02-10T04:38:31.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: 276204 2021-02-10T04:38:31.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: +0000 2021-02-10T04:38:31.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: 2021-02-10T04:38:31.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: osd.3 2021-02-10T04:38:31.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: ( 2021-02-10T04:38:31.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: osd.3) 79 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:38:31.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: cluster 2021-02-10T04:38:31.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: 2021 2021-02-10T04:38:31.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: - 2021-02-10T04:38:31.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: 02-10T04:38: 2021-02-10T04:38:31.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: 30.277252+0000 osd.3 (osd.3) 80 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:38:31.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: cluster 2021-02-10T04:38:31.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: 2021-02-10T04:38:31.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: 2021-02-10T04:38 2021-02-10T04:38:31.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: :31.121035+0000 osd.0 (osd.0) 79 : cluster [DBG] 5.b scrub ok 2021-02-10T04:38:31.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:38:31.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: 02-10T04:38:31.238369+0000 mon.a (mon 2021-02-10T04:38:31.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[35961]: .0) 1584 : cluster [DBG] osdmap e693: 8 total, 8 up, 7 in 2021-02-10T04:38:31.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[29319]: cluster 2021-02-10T04:38:30.276204+0000 osd.3 (osd.3 2021-02-10T04:38:31.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[29319]: ) 79 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:38:31.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[29319]: cluster 2021-02-10T04:38:30.277252+0000 osd.3 (osd.3) 80 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:38:31.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[29319]: cluster 2021-02-10T04:38:31.121035+0000 osd.0 2021-02-10T04:38:31.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[29319]: (osd.0) 79 : cluster [DBG] 5.b scrub ok 2021-02-10T04:38:31.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[29319]: cluster 2021-02-10T04:38:31.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[29319]: 2021-02-10T04:38 2021-02-10T04:38:31.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[29319]: :31.238369+0000 mon.a (mon.0) 2021-02-10T04:38:31.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:31 smithi085 conmon[29319]: 1584 : cluster [DBG] osdmap e693: 8 total, 8 up, 7 in 2021-02-10T04:38:32.247 INFO:tasks.rados.rados.0.smithi085.stdout:2666: done (4 left) 2021-02-10T04:38:32.247 INFO:tasks.rados.rados.0.smithi085.stdout:2667: read oid 37 snap 295 2021-02-10T04:38:32.247 INFO:tasks.rados.rados.0.smithi085.stdout:2667: expect deleted 2021-02-10T04:38:32.247 INFO:tasks.rados.rados.0.smithi085.stdout:2668: write oid 48 current snap is 308 2021-02-10T04:38:32.247 INFO:tasks.rados.rados.0.smithi085.stdout:2668: seq_num 929 ranges {413703=501387,1596360=624660,2895372=691347} 2021-02-10T04:38:32.247 INFO:tasks.rados.rados.0.smithi085.stdout:2668: writing smithi08575397-48 from 413703 to 915090 tid 1 2021-02-10T04:38:32.255 INFO:tasks.rados.rados.0.smithi085.stdout:2668: writing smithi08575397-48 from 1596360 to 2221020 tid 2 2021-02-10T04:38:32.265 INFO:tasks.rados.rados.0.smithi085.stdout:2668: writing smithi08575397-48 from 2895372 to 3586719 tid 3 2021-02-10T04:38:32.265 INFO:tasks.rados.rados.0.smithi085.stdout:2668: finishing write tid 1 to smithi08575397-48 2021-02-10T04:38:32.265 INFO:tasks.rados.rados.0.smithi085.stdout:2668: finishing write tid 2 to smithi08575397-48 2021-02-10T04:38:32.265 INFO:tasks.rados.rados.0.smithi085.stdout:2667: done (5 left) 2021-02-10T04:38:32.265 INFO:tasks.rados.rados.0.smithi085.stdout:2669: read oid 12 snap -1 2021-02-10T04:38:32.265 INFO:tasks.rados.rados.0.smithi085.stdout:2669: expect (ObjNum 849 snap 276 seq_num 849) 2021-02-10T04:38:32.265 INFO:tasks.rados.rados.0.smithi085.stdout:2670: snap_create 2021-02-10T04:38:32.275 INFO:tasks.rados.rados.0.smithi085.stdout:2669: expect (ObjNum 849 snap 276 seq_num 849) 2021-02-10T04:38:32.425 INFO:tasks.rados.rados.0.smithi085.stdout:2668: finishing write tid 3 to smithi08575397-48 2021-02-10T04:38:32.426 INFO:tasks.rados.rados.0.smithi085.stdout:2668: finishing write tid 4 to smithi08575397-48 2021-02-10T04:38:32.426 INFO:tasks.rados.rados.0.smithi085.stdout:2668: finishing write tid 5 to smithi08575397-48 2021-02-10T04:38:32.426 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 979 (ObjNum 929 snap 308 seq_num 929) dirty exists 2021-02-10T04:38:32.426 INFO:tasks.rados.rados.0.smithi085.stdout:2668: left oid 48 (ObjNum 929 snap 308 seq_num 929) 2021-02-10T04:38:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:32 smithi164 conmon[29288]: cluster 2021-02-10T04:38:31.166582+0000 2021-02-10T04:38:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:32 smithi164 conmon[29288]: osd.6 (osd.6) 73 : cluster [DBG] 2.17 scrub ok 2021-02-10T04:38:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:32 smithi164 conmon[29288]: cluster 2021-02-10T04:38:31.211647+0000 osd.5 (osd.5) 75 : cluster [DBG] 2021-02-10T04:38:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:32 smithi164 conmon[29288]: 5.d scrub ok 2021-02-10T04:38:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:32 smithi164 conmon[29288]: cluster 2021-02-10T04:38:32 2021-02-10T04:38:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:32 smithi164 conmon[29288]: .145507+0000 osd.0 (osd.0) 80 : cluster [DBG] 4.b scrub ok 2021-02-10T04:38:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:32 smithi164 conmon[29288]: cluster 2021-02-10T04:38 2021-02-10T04:38:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:32 smithi164 conmon[29288]: :32.158249+0000 mgr.y (mgr.14140) 713 : cluster 2021-02-10T04:38:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:32 smithi164 conmon[29288]: [DBG] pgmap v1190: 168 pgs: 155 active+clean, 8 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 371 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 347 KiB/s wr, 3 op/s 2021-02-10T04:38:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:32 smithi164 conmon[29288]: cluster 2021-02-10T04:38:32.239549 2021-02-10T04:38:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:32 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1585 : cluster [DBG] osdmap e694: 8 total, 8 up, 7 in 2021-02-10T04:38:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:32 smithi164 conmon[29288]: 2021-02-10T04:38:32.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[35961]: cluster 2021-02-10T04:38:31.166582 2021-02-10T04:38:32.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[35961]: +0000 osd.6 (osd.6) 73 : cluster [DBG] 2.17 scrub ok 2021-02-10T04:38:32.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[35961]: cluster 2021-02-10T04:38:31.211647 2021-02-10T04:38:32.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[35961]: +0000 osd.5 (osd.5) 75 : cluster [DBG] 5.d scrub ok 2021-02-10T04:38:32.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[35961]: 2021-02-10T04:38:32.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:38:32.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[35961]: -02-10T04:38:32.145507+0000 osd.0 (osd.0) 2021-02-10T04:38:32.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[35961]: 80 : cluster [DBG] 4.b scrub ok 2021-02-10T04:38:32.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[35961]: cluster 2021-02-10T04:38:32.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[35961]: 2021-02-10T04:38:32 2021-02-10T04:38:32.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[35961]: .158249+0000 mgr.y (mgr.14140) 713 : 2021-02-10T04:38:32.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[35961]: cluster [DBG] pgmap v1190: 168 pgs: 155 active+clean, 8 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 371 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 347 KiB/s wr, 3 op/s 2021-02-10T04:38:32.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[35961]: cluster 2021-02-10T04:38:32.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:38:32.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[35961]: 04:38:32.239549+0000 mon.a ( 2021-02-10T04:38:32.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[35961]: mon.0) 1585 : cluster [DBG] 2021-02-10T04:38:32.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[35961]: osdmap e694: 8 total, 8 up, 7 in 2021-02-10T04:38:32.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[29319]: cluster 2021-02-10T04:38:32.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:38:32.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[29319]: 04:38:31.166582+0000 osd.6 (osd.6) 73 : cluster [DBG] 2.17 scrub ok 2021-02-10T04:38:32.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:38:32.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[29319]: -10T04:38:31.211647+0000 osd.5 (osd.5) 2021-02-10T04:38:32.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[29319]: 75 : cluster [DBG] 5.d scrub ok 2021-02-10T04:38:32.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:38:32.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[29319]: -02-10T04:38:32.145507+0000 2021-02-10T04:38:32.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[29319]: osd.0 (osd.0) 80 : cluster [DBG] 4.b scrub ok 2021-02-10T04:38:32.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[29319]: 2021-02-10T04:38:32.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:38:32.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[29319]: 10T04:38:32.158249+0000 mgr.y (mgr.14140) 713 : cluster [DBG] pgmap v1190: 168 pgs: 155 active+clean, 8 active+clean+snaptrim, 5 active+clean+snaptrim_wait; 371 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 347 KiB/s wr, 3 op/s 2021-02-10T04:38:32.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:38:32.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[29319]: 02-10T04:38:32.239549+0000 mon.a (mon. 2021-02-10T04:38:32.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:32 smithi085 conmon[29319]: 0) 1585 : cluster [DBG] osdmap e694: 8 total, 8 up, 7 in 2021-02-10T04:38:33.228 INFO:tasks.rados.rados.0.smithi085.stdout:2668: done (6 left) 2021-02-10T04:38:33.228 INFO:tasks.rados.rados.0.smithi085.stdout:2669: done (5 left) 2021-02-10T04:38:33.228 INFO:tasks.rados.rados.0.smithi085.stdout:2670: done (4 left) 2021-02-10T04:38:33.228 INFO:tasks.rados.rados.0.smithi085.stdout:2671: read oid 38 snap -1 2021-02-10T04:38:33.229 INFO:tasks.rados.rados.0.smithi085.stdout:2671: expect deleted 2021-02-10T04:38:33.229 INFO:tasks.rados.rados.0.smithi085.stdout:2672: rollback oid 21 current snap is 309 2021-02-10T04:38:33.229 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 21 to 306 2021-02-10T04:38:33.229 INFO:tasks.rados.rados.0.smithi085.stdout:2673: snap_create 2021-02-10T04:38:33.241 INFO:tasks.rados.rados.0.smithi085.stdout:2672: finishing rollback tid 0 to smithi08575397-21 2021-02-10T04:38:33.243 INFO:tasks.rados.rados.0.smithi085.stdout:2672: finishing rollback tid 1 to smithi08575397-21 2021-02-10T04:38:33.244 INFO:tasks.rados.rados.0.smithi085.stdout:2672: finishing rollback tid 2 to smithi08575397-21 2021-02-10T04:38:33.244 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 1411 (ObjNum 8 snap 0 seq_num 8) dirty exists 2021-02-10T04:38:33.569 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:38:33.673 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:38:33.796 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:33 smithi085 conmon[29319]: cluster 2021-02-10T04:38:32.193231+0000 2021-02-10T04:38:33.796 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:33 smithi085 conmon[29319]: osd.6 (osd.6) 74 : cluster [DBG] 2.1a scrub ok 2021-02-10T04:38:33.796 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:33 smithi085 conmon[29319]: cluster 2021-02-10T04:38:32.221649 2021-02-10T04:38:33.796 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:33 smithi085 conmon[29319]: +0000 osd.4 (osd. 2021-02-10T04:38:33.796 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:33 smithi085 conmon[29319]: 4) 1124 : cluster [DBG] 4.13 scrub ok 2021-02-10T04:38:33.797 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:33 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:38:33.797 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:33 smithi085 conmon[29319]: 02-10T04:38:32.875495+0000 osd.3 (osd 2021-02-10T04:38:33.797 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:33 smithi085 conmon[29319]: .3) 81 : cluster [DBG] 6.f scrub ok 2021-02-10T04:38:33.797 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:33 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:38:33.797 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:33 smithi085 conmon[29319]: 02-10T04:38:33.226707+0000 mon.a ( 2021-02-10T04:38:33.797 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:33 smithi085 conmon[29319]: mon.0) 1586 : cluster [DBG] osdmap e695: 8 total, 8 up, 7 in 2021-02-10T04:38:33.797 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:33 smithi085 conmon[35961]: cluster 2021-02-10T04:38:32.193231+0000 2021-02-10T04:38:33.797 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:33 smithi085 conmon[35961]: osd.6 (osd.6) 74 : cluster [DBG] 2.1a scrub ok 2021-02-10T04:38:33.797 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:33 smithi085 conmon[35961]: cluster 2021-02-10T04:38: 2021-02-10T04:38:33.797 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:33 smithi085 conmon[35961]: 32.221649+0000 osd.4 (osd.4) 1124 : cluster [DBG] 4.13 scrub ok 2021-02-10T04:38:33.797 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:33 smithi085 conmon[35961]: cluster 2021-02-10T04:38:33.797 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:33 smithi085 conmon[35961]: 2021-02-10T04:38:32.875495+0000 osd.3 (osd.3) 81 : cluster [DBG] 2021-02-10T04:38:33.798 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:33 smithi085 conmon[35961]: 6.f scrub ok 2021-02-10T04:38:33.798 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:33 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:38:33.798 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:33 smithi085 conmon[35961]: 02-10T04:38:33.226707+0000 mon.a (mon.0) 1586 : 2021-02-10T04:38:33.798 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:33 smithi085 conmon[35961]: cluster [DBG] osdmap e695: 8 total, 8 up, 7 in 2021-02-10T04:38:33.893 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:33 smithi164 conmon[33677]: debug 2021-02-10T04:38:33.736+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:38:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:33 smithi164 conmon[29288]: cluster 2021-02-10T04:38:32.193231 2021-02-10T04:38:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:33 smithi164 conmon[29288]: +0000 osd.6 (osd.6) 74 : cluster [DBG] 2.1a scrub ok 2021-02-10T04:38:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:33 smithi164 conmon[29288]: cluster 2021-02-10T04:38:32.221649+0000 osd.4 (osd.4) 1124 : cluster [DBG] 2021-02-10T04:38:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:33 smithi164 conmon[29288]: 4.13 scrub ok 2021-02-10T04:38:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:33 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:38:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:33 smithi164 conmon[29288]: :38:32.875495+0000 osd.3 (osd.3) 81 : cluster [DBG] 2021-02-10T04:38:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:33 smithi164 conmon[29288]: 6.f scrub ok 2021-02-10T04:38:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:33 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:38:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:33 smithi164 conmon[29288]: 04:38:33.226707+0000 mon.a (mon. 2021-02-10T04:38:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:33 smithi164 conmon[29288]: 0) 1586 : cluster [DBG] osdmap e695: 8 total, 8 up, 7 in 2021-02-10T04:38:34.007 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:38:34.113 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:38:34.146 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:38:34.233 INFO:tasks.rados.rados.0.smithi085.stdout:2671: done (6 left) 2021-02-10T04:38:34.233 INFO:tasks.rados.rados.0.smithi085.stdout:2672: done (5 left) 2021-02-10T04:38:34.233 INFO:tasks.rados.rados.0.smithi085.stdout:2673: done (4 left) 2021-02-10T04:38:34.233 INFO:tasks.rados.rados.0.smithi085.stdout:2674: write oid 2 current snap is 310 2021-02-10T04:38:34.233 INFO:tasks.rados.rados.0.smithi085.stdout:2674: seq_num 930 ranges {430510=576636,1654842=768318,2922795=502619} 2021-02-10T04:38:34.233 INFO:tasks.rados.rados.0.smithi085.stdout:2674: writing smithi08575397-2 from 430510 to 1007146 tid 1 2021-02-10T04:38:34.241 INFO:tasks.rados.rados.0.smithi085.stdout:2674: writing smithi08575397-2 from 1654842 to 2423160 tid 2 2021-02-10T04:38:34.248 INFO:tasks.rados.rados.0.smithi085.stdout:2674: writing smithi08575397-2 from 2922795 to 3425414 tid 3 2021-02-10T04:38:34.248 INFO:tasks.rados.rados.0.smithi085.stdout:2675: write (excl) oid 46 current snap is 310 2021-02-10T04:38:34.248 INFO:tasks.rados.rados.0.smithi085.stdout:2674: finishing write tid 1 to smithi08575397-2 2021-02-10T04:38:34.248 INFO:tasks.rados.rados.0.smithi085.stdout:2675: seq_num 931 ranges {447317=651885,1713324=628329,3066571=580746} 2021-02-10T04:38:34.255 INFO:tasks.rados.rados.0.smithi085.stdout:2675: writing smithi08575397-46 from 447317 to 1099202 tid 1 2021-02-10T04:38:34.262 INFO:tasks.rados.rados.0.smithi085.stdout:2675: writing smithi08575397-46 from 1713324 to 2341653 tid 2 2021-02-10T04:38:34.271 INFO:tasks.rados.rados.0.smithi085.stdout:2675: writing smithi08575397-46 from 3066571 to 3647317 tid 3 2021-02-10T04:38:34.271 INFO:tasks.rados.rados.0.smithi085.stdout:2676: write oid 18 current snap is 310 2021-02-10T04:38:34.271 INFO:tasks.rados.rados.0.smithi085.stdout:2676: seq_num 932 ranges {464124=727134,1771806=771987,2977641=641516} 2021-02-10T04:38:34.278 INFO:tasks.rados.rados.0.smithi085.stdout:2676: writing smithi08575397-18 from 464124 to 1191258 tid 1 2021-02-10T04:38:34.287 INFO:tasks.rados.rados.0.smithi085.stdout:2676: writing smithi08575397-18 from 1771806 to 2543793 tid 2 2021-02-10T04:38:34.294 INFO:tasks.rados.rados.0.smithi085.stdout:2676: writing smithi08575397-18 from 2977641 to 3619157 tid 3 2021-02-10T04:38:34.294 INFO:tasks.rados.rados.0.smithi085.stdout:2677: snap_remove snap 294 2021-02-10T04:38:34.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[44544]: debug 2021-02-10T04:38:34.224+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:38:34.542 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:38:34.646 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:38:34.680 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:38:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: cluster 2021-02-10T04:38:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: 2021-02 2021-02-10T04:38:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: -10T04:38:33.199514+0000 osd.4 (osd.4) 1125 : cluster [DBG] 5.18 scrub ok 2021-02-10T04:38:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: cluster 2021-02-10T04:38:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: 2021-02-10T04:38:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: 2021 2021-02-10T04:38:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: - 2021-02-10T04:38:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: 02 2021-02-10T04:38:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: - 2021-02-10T04:38:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: 10 2021-02-10T04:38:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: T 2021-02-10T04:38:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: 04 2021-02-10T04:38:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: : 2021-02-10T04:38:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: 38 2021-02-10T04:38:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: : 2021-02-10T04:38:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: 33 2021-02-10T04:38:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: . 2021-02-10T04:38:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: 243513 2021-02-10T04:38:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: +0000 2021-02-10T04:38:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: 2021-02-10T04:38:34.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: osd.6 2021-02-10T04:38:34.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: (osd.6) 75 : cluster [DBG] 2021-02-10T04:38:34.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: 3.0 scrub ok 2021-02-10T04:38:34.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: cluster 2021-02-10T04:38:34.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: 2021-02-10T04:38:33.835936+0000 osd.0 (osd.0) 81 2021-02-10T04:38:34.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:38:34.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:38:34.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: 10T04:38:33.837013+0000 osd.0 (osd. 2021-02-10T04:38:34.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: 0) 82 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:38:34.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:38:34.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: 04:38:34.159031+0000 mgr.y (mgr.14140) 2021-02-10T04:38:34.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: 714 : cluster [DBG] pgmap v1193: 168 pgs: 158 active+clean, 6 active+clean+snaptrim, 4 active+clean+snaptrim_wait; 378 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2021-02-10T04:38:34.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:38:34.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: 38:34.225434+0000 mon.a (mon.0) 2021-02-10T04:38:34.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[29288]: 1587 : cluster [DBG] osdmap e696: 8 total, 8 up, 7 in 2021-02-10T04:38:34.897 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[33677]: debug 2021-02-10T04:38:34.897 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[33677]: 2021-02-10T04:38:34.766+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:38:34.897 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:34 smithi164 conmon[33677]: 2021-02-10T04:38:34.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[35961]: cluster 2021-02-10T04:38:33 2021-02-10T04:38:34.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[35961]: .199514+0000 osd.4 (osd.4) 1125 : cluster [DBG] 5.18 scrub ok 2021-02-10T04:38:34.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:38:34.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[35961]: T04:38:33.243513+0000 osd.6 2021-02-10T04:38:34.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[35961]: (osd.6) 75 : cluster [DBG] 3.0 scrub ok 2021-02-10T04:38:34.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[35961]: 2021-02-10T04:38:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:38:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[35961]: 04:38:33.835936+0000 osd.0 ( 2021-02-10T04:38:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[35961]: osd.0) 81 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:38:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[35961]: cluster 2021-02-10T04:38:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[35961]: 2021-02-10T04:38: 2021-02-10T04:38:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[35961]: 33.837013+0000 osd.0 (osd.0) 82 2021-02-10T04:38:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[35961]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:38:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:38:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[35961]: 02-10T04:38:34 2021-02-10T04:38:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[35961]: .159031+0000 mgr.y (mgr.14140 2021-02-10T04:38:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[35961]: ) 714 : cluster [DBG] pgmap v1193: 168 pgs: 158 active+clean, 6 active+clean+snaptrim, 4 active+clean+snaptrim_wait; 378 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2021-02-10T04:38:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[35961]: cluster 2021-02-10T04:38:34.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[35961]: 2021-02-10T04:38 2021-02-10T04:38:34.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[35961]: :34.225434+0000 mon.a (mon.0) 1587 2021-02-10T04:38:34.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[35961]: : cluster [DBG] osdmap e696: 8 total, 8 up, 7 in 2021-02-10T04:38:34.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[29319]: cluster 2021-02-10T04:38:33.199514+0000 osd.4 (osd.4) 1125 : 2021-02-10T04:38:34.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[29319]: cluster [DBG] 5.18 scrub ok 2021-02-10T04:38:34.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[29319]: cluster 2021-02-10T04:38:33.243513+0000 osd.6 (osd.6) 75 : cluster [DBG] 3.0 scrub ok 2021-02-10T04:38:34.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[29319]: cluster 2021-02-10T04:38:33 2021-02-10T04:38:34.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[29319]: .835936+0000 osd.0 (osd.0) 81 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:38:34.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[29319]: cluster 2021-02-10T04:38:34.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[29319]: 2021-02-10T04:38:33.837013+0000 osd.0 (osd.0 2021-02-10T04:38:34.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[29319]: ) 82 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:38:34.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[29319]: cluster 2021-02-10T04:38:34.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[29319]: 2021-02-10T04:38:34.159031+0000 mgr.y (mgr.14140) 714 : cluster 2021-02-10T04:38:34.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[29319]: [DBG] pgmap v1193: 168 pgs: 158 active+clean, 6 active+clean+snaptrim, 4 active+clean+snaptrim_wait; 378 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2021-02-10T04:38:34.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[29319]: cluster 2021-02-10T04:38:34 2021-02-10T04:38:34.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[29319]: .225434+0000 mon.a (mon.0) 1587 : 2021-02-10T04:38:34.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:34 smithi085 conmon[29319]: cluster [DBG] osdmap e696: 8 total, 8 up, 7 in 2021-02-10T04:38:35.174 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:38:35.230 INFO:tasks.rados.rados.0.smithi085.stdout:2674: finishing write tid 2 to smithi08575397-2 2021-02-10T04:38:35.230 INFO:tasks.rados.rados.0.smithi085.stdout:2674: finishing write tid 3 to smithi08575397-2 2021-02-10T04:38:35.230 INFO:tasks.rados.rados.0.smithi085.stdout:2674: finishing write tid 4 to smithi08575397-2 2021-02-10T04:38:35.230 INFO:tasks.rados.rados.0.smithi085.stdout:2674: finishing write tid 5 to smithi08575397-2 2021-02-10T04:38:35.230 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 882 (ObjNum 930 snap 310 seq_num 930) dirty exists 2021-02-10T04:38:35.230 INFO:tasks.rados.rados.0.smithi085.stdout:2674: left oid 2 (ObjNum 930 snap 310 seq_num 930) 2021-02-10T04:38:35.230 INFO:tasks.rados.rados.0.smithi085.stdout:2675: finishing write tid 1 to smithi08575397-46 2021-02-10T04:38:35.231 INFO:tasks.rados.rados.0.smithi085.stdout:2675: finishing write tid 2 to smithi08575397-46 2021-02-10T04:38:35.231 INFO:tasks.rados.rados.0.smithi085.stdout:2675: finishing write tid 3 to smithi08575397-46 2021-02-10T04:38:35.231 INFO:tasks.rados.rados.0.smithi085.stdout:2675: finishing write tid 4 to smithi08575397-46 2021-02-10T04:38:35.231 INFO:tasks.rados.rados.0.smithi085.stdout:2675: finishing write tid 5 to smithi08575397-46 2021-02-10T04:38:35.231 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 1340 (ObjNum 931 snap 310 seq_num 931) dirty exists 2021-02-10T04:38:35.231 INFO:tasks.rados.rados.0.smithi085.stdout:2675: left oid 46 (ObjNum 931 snap 310 seq_num 931) 2021-02-10T04:38:35.231 INFO:tasks.rados.rados.0.smithi085.stdout:2676: finishing write tid 1 to smithi08575397-18 2021-02-10T04:38:35.231 INFO:tasks.rados.rados.0.smithi085.stdout:2676: finishing write tid 2 to smithi08575397-18 2021-02-10T04:38:35.231 INFO:tasks.rados.rados.0.smithi085.stdout:2676: finishing write tid 3 to smithi08575397-18 2021-02-10T04:38:35.231 INFO:tasks.rados.rados.0.smithi085.stdout:2676: finishing write tid 4 to smithi08575397-18 2021-02-10T04:38:35.231 INFO:tasks.rados.rados.0.smithi085.stdout:2676: finishing write tid 5 to smithi08575397-18 2021-02-10T04:38:35.231 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 18 v 1276 (ObjNum 932 snap 310 seq_num 932) dirty exists 2021-02-10T04:38:35.232 INFO:tasks.rados.rados.0.smithi085.stdout:2676: left oid 18 (ObjNum 932 snap 310 seq_num 932) 2021-02-10T04:38:35.232 INFO:tasks.rados.rados.0.smithi085.stdout:2674: done (7 left) 2021-02-10T04:38:35.232 INFO:tasks.rados.rados.0.smithi085.stdout:2675: done (6 left) 2021-02-10T04:38:35.232 INFO:tasks.rados.rados.0.smithi085.stdout:2676: done (5 left) 2021-02-10T04:38:35.232 INFO:tasks.rados.rados.0.smithi085.stdout:2677: done (4 left) 2021-02-10T04:38:35.232 INFO:tasks.rados.rados.0.smithi085.stdout:2678: read oid 6 snap -1 2021-02-10T04:38:35.232 INFO:tasks.rados.rados.0.smithi085.stdout:2678: expect deleted 2021-02-10T04:38:35.232 INFO:tasks.rados.rados.0.smithi085.stdout:2679: read oid 46 snap -1 2021-02-10T04:38:35.232 INFO:tasks.rados.rados.0.smithi085.stdout:2679: expect (ObjNum 931 snap 310 seq_num 931) 2021-02-10T04:38:35.232 INFO:tasks.rados.rados.0.smithi085.stdout:2680: copy_from oid 29 from oid 25 current snap is 310 2021-02-10T04:38:35.232 INFO:tasks.rados.rados.0.smithi085.stdout:2681: read oid 13 snap -1 2021-02-10T04:38:35.232 INFO:tasks.rados.rados.0.smithi085.stdout:2681: expect (ObjNum 882 snap 287 seq_num 882) 2021-02-10T04:38:35.232 INFO:tasks.rados.rados.0.smithi085.stdout:2682: delete oid 36 current snap is 310 2021-02-10T04:38:35.240 INFO:tasks.rados.rados.0.smithi085.stdout:2680: finishing copy_from to smithi08575397-29 2021-02-10T04:38:35.241 INFO:tasks.rados.rados.0.smithi085.stdout:2680: got expected ENOENT (src dne) 2021-02-10T04:38:35.241 INFO:tasks.rados.rados.0.smithi085.stdout:2678: done (8 left) 2021-02-10T04:38:35.241 INFO:tasks.rados.rados.0.smithi085.stdout:2682: done (7 left) 2021-02-10T04:38:35.241 INFO:tasks.rados.rados.0.smithi085.stdout:2683: rollback oid 40 current snap is 310 2021-02-10T04:38:35.241 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 40 to 301 2021-02-10T04:38:35.241 INFO:tasks.rados.rados.0.smithi085.stdout:2684: read oid 9 snap -1 2021-02-10T04:38:35.241 INFO:tasks.rados.rados.0.smithi085.stdout:2684: expect (ObjNum 876 snap 282 seq_num 876) 2021-02-10T04:38:35.241 INFO:tasks.rados.rados.0.smithi085.stdout:2685: rollback oid 12 current snap is 310 2021-02-10T04:38:35.241 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 12 to 293 2021-02-10T04:38:35.241 INFO:tasks.rados.rados.0.smithi085.stdout:2686: read oid 38 snap 290 2021-02-10T04:38:35.241 INFO:tasks.rados.rados.0.smithi085.stdout:2686: expect (ObjNum 886 snap 287 seq_num 886) 2021-02-10T04:38:35.241 INFO:tasks.rados.rados.0.smithi085.stdout:2687: rollback oid 6 current snap is 310 2021-02-10T04:38:35.241 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 6 to 309 2021-02-10T04:38:35.242 INFO:tasks.rados.rados.0.smithi085.stdout:2688: copy_from oid 45 from oid 36 current snap is 310 2021-02-10T04:38:35.242 INFO:tasks.rados.rados.0.smithi085.stdout:2689: read oid 23 snap 306 2021-02-10T04:38:35.242 INFO:tasks.rados.rados.0.smithi085.stdout:2689: expect (ObjNum 913 snap 299 seq_num 913) 2021-02-10T04:38:35.242 INFO:tasks.rados.rados.0.smithi085.stdout:2690: delete oid 2 current snap is 310 2021-02-10T04:38:35.242 INFO:tasks.rados.rados.0.smithi085.stdout:2679: expect (ObjNum 931 snap 310 seq_num 931) 2021-02-10T04:38:35.279 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:38:35.320 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:38:35.409 INFO:tasks.rados.rados.0.smithi085.stdout:2679: done (14 left) 2021-02-10T04:38:35.409 INFO:tasks.rados.rados.0.smithi085.stdout:2690: done (13 left) 2021-02-10T04:38:35.410 INFO:tasks.rados.rados.0.smithi085.stdout:2691: write (excl) oid 32 current snap is 310 2021-02-10T04:38:35.410 INFO:tasks.rados.rados.0.smithi085.stdout:2691: seq_num 935 ranges {514545=552881,1830899=519314,2943557=475332} 2021-02-10T04:38:35.415 INFO:tasks.rados.rados.0.smithi085.stdout:2691: writing smithi08575397-32 from 514545 to 1067426 tid 1 2021-02-10T04:38:35.423 INFO:tasks.rados.rados.0.smithi085.stdout:2691: writing smithi08575397-32 from 1830899 to 2350213 tid 2 2021-02-10T04:38:35.430 INFO:tasks.rados.rados.0.smithi085.stdout:2691: writing smithi08575397-32 from 2943557 to 3418889 tid 3 2021-02-10T04:38:35.430 INFO:tasks.rados.rados.0.smithi085.stdout:2692: read oid 34 snap -1 2021-02-10T04:38:35.431 INFO:tasks.rados.rados.0.smithi085.stdout:2692: expect (ObjNum 859 snap 277 seq_num 859) 2021-02-10T04:38:35.431 INFO:tasks.rados.rados.0.smithi085.stdout:2693: write (excl) oid 8 current snap is 310 2021-02-10T04:38:35.431 INFO:tasks.rados.rados.0.smithi085.stdout:2680: finishing copy_from racing read to smithi08575397-29 2021-02-10T04:38:35.431 INFO:tasks.rados.rados.0.smithi085.stdout:2693: seq_num 936 ranges {531352=744483,2005734=779325,3487333=244019} 2021-02-10T04:38:35.438 INFO:tasks.rados.rados.0.smithi085.stdout:2693: writing smithi08575397-8 from 531352 to 1275835 tid 1 2021-02-10T04:38:35.446 INFO:tasks.rados.rados.0.smithi085.stdout:2693: writing smithi08575397-8 from 2005734 to 2785059 tid 2 2021-02-10T04:38:35.452 INFO:tasks.rados.rados.0.smithi085.stdout:2693: writing smithi08575397-8 from 3487333 to 3731352 tid 3 2021-02-10T04:38:35.452 INFO:tasks.rados.rados.0.smithi085.stdout:2680: done (15 left) 2021-02-10T04:38:35.452 INFO:tasks.rados.rados.0.smithi085.stdout:2694: copy_from oid 31 from oid 35 current snap is 310 2021-02-10T04:38:35.452 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 16 2021-02-10T04:38:35.452 INFO:tasks.rados.rados.0.smithi085.stdout:2681: expect (ObjNum 882 snap 287 seq_num 882) 2021-02-10T04:38:35.560 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:35 smithi164 conmon[29288]: cluster 2021-02-10T04:38:34 2021-02-10T04:38:35.560 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:35 smithi164 conmon[29288]: .289487+0000 osd.5 (osd.5) 76 : cluster [DBG] 2.2 scrub ok 2021-02-10T04:38:35.560 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:35 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:38:35.561 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:35 smithi164 conmon[29288]: 10T04:38:35.227564+0000 mon.a (mon.0 2021-02-10T04:38:35.561 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:35 smithi164 conmon[29288]: ) 1588 : cluster [DBG] osdmap e697: 8 total, 8 up, 7 in 2021-02-10T04:38:35.569 INFO:tasks.rados.rados.0.smithi085.stdout:2681: done (15 left) 2021-02-10T04:38:35.569 INFO:tasks.rados.rados.0.smithi085.stdout:2695: rollback oid 15 current snap is 310 2021-02-10T04:38:35.569 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 15 to 237 2021-02-10T04:38:35.569 INFO:tasks.rados.rados.0.smithi085.stdout:2689: expect (ObjNum 913 snap 299 seq_num 913) 2021-02-10T04:38:35.652 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:35 smithi085 conmon[35961]: cluster 2021-02-10T04:38:34.289487+0000 osd.5 2021-02-10T04:38:35.652 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:35 smithi085 conmon[35961]: (osd.5) 76 : cluster [DBG] 2.2 scrub ok 2021-02-10T04:38:35.652 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:35 smithi085 conmon[35961]: cluster 2021-02-10T04:38 2021-02-10T04:38:35.652 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:35 smithi085 conmon[35961]: :35.227564+0000 mon.a (mon.0) 1588 : cluster [DBG] osdmap e697: 8 total, 8 up, 7 in 2021-02-10T04:38:35.652 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:35 smithi085 conmon[29319]: cluster 2021-02-10T04:38:34.289487 2021-02-10T04:38:35.652 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:35 smithi085 conmon[29319]: +0000 osd.5 (osd.5) 76 : cluster [DBG] 2.2 scrub ok 2021-02-10T04:38:35.652 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:35 smithi085 conmon[29319]: cluster 2021-02-10T04:38:35.227564+0000 mon.a (mon.0) 1588 : cluster [DBG] osdmap e697: 8 total, 8 up, 7 in 2021-02-10T04:38:35.654 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.0 dump_historic_ops 2021-02-10T04:38:35.793 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:38:35.899 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:38:35.918 INFO:tasks.rados.rados.0.smithi085.stdout:2689: done (15 left) 2021-02-10T04:38:35.919 INFO:tasks.rados.rados.0.smithi085.stdout:2696: snap_create 2021-02-10T04:38:35.919 INFO:tasks.rados.rados.0.smithi085.stdout:2688: finishing copy_from to smithi08575397-45 2021-02-10T04:38:35.919 INFO:tasks.rados.rados.0.smithi085.stdout:2688: got expected ENOENT (src dne) 2021-02-10T04:38:35.919 INFO:tasks.rados.rados.0.smithi085.stdout:2688: finishing copy_from racing read to smithi08575397-45 2021-02-10T04:38:35.919 INFO:tasks.rados.rados.0.smithi085.stdout:2686: expect (ObjNum 886 snap 287 seq_num 886) 2021-02-10T04:38:35.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:38:35 smithi085 conmon[49106]: debug 2021-02-10T04:38:35.764+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:38:35.960 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:38:36.029 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:38:36.127 INFO:tasks.rados.rados.0.smithi085.stdout:2684: expect (ObjNum 876 snap 282 seq_num 876) 2021-02-10T04:38:36.152 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:38:36.203 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:38:36 smithi085 conmon[49106]: debug 2021-02-10T04:38:36.071+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:38:36.259 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:38:36.262 INFO:tasks.rados.rados.0.smithi085.stdout:2685: finishing rollback tid 0 to smithi08575397-12 2021-02-10T04:38:36.263 INFO:tasks.rados.rados.0.smithi085.stdout:2685: finishing rollback tid 1 to smithi08575397-12 2021-02-10T04:38:36.263 INFO:tasks.rados.rados.0.smithi085.stdout:2685: finishing rollback tid 2 to smithi08575397-12 2021-02-10T04:38:36.263 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 983 (ObjNum 849 snap 276 seq_num 849) dirty exists 2021-02-10T04:38:36.263 INFO:tasks.rados.rados.0.smithi085.stdout:2683: finishing rollback tid 0 to smithi08575397-40 2021-02-10T04:38:36.263 INFO:tasks.rados.rados.0.smithi085.stdout:2683: finishing rollback tid 1 to smithi08575397-40 2021-02-10T04:38:36.263 INFO:tasks.rados.rados.0.smithi085.stdout:2687: finishing rollback tid 1 to smithi08575397-6 2021-02-10T04:38:36.263 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 1107 (ObjNum 82861440 snap 22030 seq_num 0) dirty dne 2021-02-10T04:38:36.263 INFO:tasks.rados.rados.0.smithi085.stdout:2683: finishing rollback tid 2 to smithi08575397-40 2021-02-10T04:38:36.263 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 1417 (ObjNum 879 snap 282 seq_num 879) dirty exists 2021-02-10T04:38:36.263 INFO:tasks.rados.rados.0.smithi085.stdout:2691: finishing write tid 1 to smithi08575397-32 2021-02-10T04:38:36.263 INFO:tasks.rados.rados.0.smithi085.stdout:2691: finishing write tid 2 to smithi08575397-32 2021-02-10T04:38:36.264 INFO:tasks.rados.rados.0.smithi085.stdout:2692: expect (ObjNum 859 snap 277 seq_num 859) 2021-02-10T04:38:36.294 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:38:36.375 INFO:tasks.rados.rados.0.smithi085.stdout:2691: finishing write tid 3 to smithi08575397-32 2021-02-10T04:38:36.375 INFO:tasks.rados.rados.0.smithi085.stdout:2691: finishing write tid 4 to smithi08575397-32 2021-02-10T04:38:36.375 INFO:tasks.rados.rados.0.smithi085.stdout:2691: finishing write tid 5 to smithi08575397-32 2021-02-10T04:38:36.375 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 32 v 657 (ObjNum 935 snap 310 seq_num 935) dirty exists 2021-02-10T04:38:36.376 INFO:tasks.rados.rados.0.smithi085.stdout:2691: left oid 32 (ObjNum 935 snap 310 seq_num 935) 2021-02-10T04:38:36.376 INFO:tasks.rados.rados.0.smithi085.stdout:2693: finishing write tid 1 to smithi08575397-8 2021-02-10T04:38:36.376 INFO:tasks.rados.rados.0.smithi085.stdout:2693: finishing write tid 2 to smithi08575397-8 2021-02-10T04:38:36.376 INFO:tasks.rados.rados.0.smithi085.stdout:2694: finishing copy_from to smithi08575397-31 2021-02-10T04:38:36.376 INFO:tasks.rados.rados.0.smithi085.stdout:2694: got expected ENOENT (src dne) 2021-02-10T04:38:36.376 INFO:tasks.rados.rados.0.smithi085.stdout:2694: finishing copy_from racing read to smithi08575397-31 2021-02-10T04:38:36.376 INFO:tasks.rados.rados.0.smithi085.stdout:2693: finishing write tid 3 to smithi08575397-8 2021-02-10T04:38:36.376 INFO:tasks.rados.rados.0.smithi085.stdout:2693: finishing write tid 4 to smithi08575397-8 2021-02-10T04:38:36.376 INFO:tasks.rados.rados.0.smithi085.stdout:2693: finishing write tid 5 to smithi08575397-8 2021-02-10T04:38:36.376 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 1339 (ObjNum 936 snap 310 seq_num 936) dirty exists 2021-02-10T04:38:36.376 INFO:tasks.rados.rados.0.smithi085.stdout:2693: left oid 8 (ObjNum 936 snap 310 seq_num 936) 2021-02-10T04:38:36.376 INFO:tasks.rados.rados.0.smithi085.stdout:2695: finishing rollback tid 1 to smithi08575397-15 2021-02-10T04:38:36.376 INFO:tasks.rados.rados.0.smithi085.stdout:2695: finishing rollback tid 2 to smithi08575397-15 2021-02-10T04:38:36.377 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 15 v 659 (ObjNum 217 snap 49 seq_num 217) dirty exists 2021-02-10T04:38:36.571 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:36 smithi085 conmon[29319]: cluster 2021-02-10T04:38:36.577 INFO:tasks.rados.rados.0.smithi085.stdout:2683: done (15 left) 2021-02-10T04:38:36.577 INFO:tasks.rados.rados.0.smithi085.stdout:2684: done (14 left) 2021-02-10T04:38:36.577 INFO:tasks.rados.rados.0.smithi085.stdout:2685: done (13 left) 2021-02-10T04:38:36.578 INFO:tasks.rados.rados.0.smithi085.stdout:2686: done (12 left) 2021-02-10T04:38:36.578 INFO:tasks.rados.rados.0.smithi085.stdout:2687: done (11 left) 2021-02-10T04:38:36.578 INFO:tasks.rados.rados.0.smithi085.stdout:2688: done (10 left) 2021-02-10T04:38:36.578 INFO:tasks.rados.rados.0.smithi085.stdout:2691: done (9 left) 2021-02-10T04:38:36.578 INFO:tasks.rados.rados.0.smithi085.stdout:2692: done (8 left) 2021-02-10T04:38:36.579 INFO:tasks.rados.rados.0.smithi085.stdout:2693: done (7 left) 2021-02-10T04:38:36.579 INFO:tasks.rados.rados.0.smithi085.stdout:2694: done (6 left) 2021-02-10T04:38:36.579 INFO:tasks.rados.rados.0.smithi085.stdout:2695: done (5 left) 2021-02-10T04:38:36.579 INFO:tasks.rados.rados.0.smithi085.stdout:2696: done (4 left) 2021-02-10T04:38:36.579 INFO:tasks.rados.rados.0.smithi085.stdout:2697: rollback oid 21 current snap is 311 2021-02-10T04:38:36.579 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 21 to 276 2021-02-10T04:38:36.580 INFO:tasks.rados.rados.0.smithi085.stdout:2698: copy_from oid 31 from oid 48 current snap is 311 2021-02-10T04:38:36.580 INFO:tasks.rados.rados.0.smithi085.stdout:2699: read oid 34 snap -1 2021-02-10T04:38:36.580 INFO:tasks.rados.rados.0.smithi085.stdout:2699: expect (ObjNum 859 snap 277 seq_num 859) 2021-02-10T04:38:36.580 INFO:tasks.rados.rados.0.smithi085.stdout:2700: rollback oid 46 current snap is 311 2021-02-10T04:38:36.580 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 46 to 299 2021-02-10T04:38:36.580 INFO:tasks.rados.rados.0.smithi085.stdout:2701: delete oid 13 current snap is 311 2021-02-10T04:38:36.585 INFO:tasks.rados.rados.0.smithi085.stdout:2699: expect (ObjNum 859 snap 277 seq_num 859) 2021-02-10T04:38:36.651 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:38:36.697 INFO:tasks.rados.rados.0.smithi085.stdout:2699: done (8 left) 2021-02-10T04:38:36.697 INFO:tasks.rados.rados.0.smithi085.stdout:2701: done (7 left) 2021-02-10T04:38:36.697 INFO:tasks.rados.rados.0.smithi085.stdout:2702: rollback oid 25 current snap is 311 2021-02-10T04:38:36.697 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 25 to 287 2021-02-10T04:38:36.697 INFO:tasks.rados.rados.0.smithi085.stdout:2703: delete oid 16 current snap is 311 2021-02-10T04:38:36.698 INFO:tasks.rados.rados.0.smithi085.stdout:2700: finishing rollback tid 0 to smithi08575397-46 2021-02-10T04:38:36.698 INFO:tasks.rados.rados.0.smithi085.stdout:2697: finishing rollback tid 0 to smithi08575397-21 2021-02-10T04:38:36.698 INFO:tasks.rados.rados.0.smithi085.stdout:2697: finishing rollback tid 1 to smithi08575397-21 2021-02-10T04:38:36.698 INFO:tasks.rados.rados.0.smithi085.stdout:2697: finishing rollback tid 2 to smithi08575397-21 2021-02-10T04:38:36.698 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 1425 (ObjNum 812 snap 265 seq_num 812) dirty exists 2021-02-10T04:38:36.698 INFO:tasks.rados.rados.0.smithi085.stdout:2700: finishing rollback tid 1 to smithi08575397-46 2021-02-10T04:38:36.698 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 1342 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-02-10T04:38:36.698 INFO:tasks.rados.rados.0.smithi085.stdout:2698: finishing copy_from to smithi08575397-31 2021-02-10T04:38:36.698 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 979 (ObjNum 929 snap 308 seq_num 929) dirty exists 2021-02-10T04:38:36.698 INFO:tasks.rados.rados.0.smithi085.stdout:2698: finishing copy_from racing read to smithi08575397-31 2021-02-10T04:38:36.699 INFO:tasks.rados.rados.0.smithi085.stdout:2697: done (8 left) 2021-02-10T04:38:36.699 INFO:tasks.rados.rados.0.smithi085.stdout:2698: done (7 left) 2021-02-10T04:38:36.700 INFO:tasks.rados.rados.0.smithi085.stdout:2700: done (6 left) 2021-02-10T04:38:36.700 INFO:tasks.rados.rados.0.smithi085.stdout:2703: done (5 left) 2021-02-10T04:38:36.700 INFO:tasks.rados.rados.0.smithi085.stdout:2704: write (excl) oid 29 current snap is 311 2021-02-10T04:38:36.700 INFO:tasks.rados.rados.0.smithi085.stdout:2704: seq_num 939 ranges {581773=570230,1664827=526652,2653249=636773} 2021-02-10T04:38:36.706 INFO:tasks.rados.rados.0.smithi085.stdout:2704: writing smithi08575397-29 from 581773 to 1152003 tid 1 2021-02-10T04:38:36.713 INFO:tasks.rados.rados.0.smithi085.stdout:2704: writing smithi08575397-29 from 1664827 to 2191479 tid 2 2021-02-10T04:38:36.720 INFO:tasks.rados.rados.0.smithi085.stdout:2704: writing smithi08575397-29 from 2653249 to 3290022 tid 3 2021-02-10T04:38:36.721 INFO:tasks.rados.rados.0.smithi085.stdout:2702: finishing rollback tid 1 to smithi08575397-25 2021-02-10T04:38:36.721 INFO:tasks.rados.rados.0.smithi085.stdout:2702: finishing rollback tid 2 to smithi08575397-25 2021-02-10T04:38:36.722 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 25 v 1278 (ObjNum 885 snap 287 seq_num 885) dirty exists 2021-02-10T04:38:36.722 INFO:tasks.rados.rados.0.smithi085.stdout:2704: finishing write tid 1 to smithi08575397-29 2021-02-10T04:38:36.722 INFO:tasks.rados.rados.0.smithi085.stdout:2704: finishing write tid 2 to smithi08575397-29 2021-02-10T04:38:36.722 INFO:tasks.rados.rados.0.smithi085.stdout:2702: done (5 left) 2021-02-10T04:38:36.722 INFO:tasks.rados.rados.0.smithi085.stdout:2705: write (excl) oid 37 current snap is 311 2021-02-10T04:38:36.722 INFO:tasks.rados.rados.0.smithi085.stdout:2705: seq_num 940 ranges {598580=645479,1723309=786663,3080672=448045} 2021-02-10T04:38:36.729 INFO:tasks.rados.rados.0.smithi085.stdout:2705: writing smithi08575397-37 from 598580 to 1244059 tid 1 2021-02-10T04:38:36.737 INFO:tasks.rados.rados.0.smithi085.stdout:2705: writing smithi08575397-37 from 1723309 to 2509972 tid 2 2021-02-10T04:38:36.744 INFO:tasks.rados.rados.0.smithi085.stdout:2705: writing smithi08575397-37 from 3080672 to 3528717 tid 3 2021-02-10T04:38:36.744 INFO:tasks.rados.rados.0.smithi085.stdout:2704: finishing write tid 3 to smithi08575397-29 2021-02-10T04:38:36.744 INFO:tasks.rados.rados.0.smithi085.stdout:2704: finishing write tid 4 to smithi08575397-29 2021-02-10T04:38:36.744 INFO:tasks.rados.rados.0.smithi085.stdout:2704: finishing write tid 5 to smithi08575397-29 2021-02-10T04:38:36.744 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 29 v 663 (ObjNum 939 snap 311 seq_num 939) dirty exists 2021-02-10T04:38:36.744 INFO:tasks.rados.rados.0.smithi085.stdout:2704: left oid 29 (ObjNum 939 snap 311 seq_num 939) 2021-02-10T04:38:36.744 INFO:tasks.rados.rados.0.smithi085.stdout:2705: finishing write tid 1 to smithi08575397-37 2021-02-10T04:38:36.744 INFO:tasks.rados.rados.0.smithi085.stdout:2704: done (5 left) 2021-02-10T04:38:36.745 INFO:tasks.rados.rados.0.smithi085.stdout:2706: snap_remove snap 276 2021-02-10T04:38:36.757 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:38:36.800 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:38:36.863 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:36 smithi085 conmon[29319]: 2021-02-10T04:38:36.863 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:36 smithi085 conmon[29319]: 2021 2021-02-10T04:38:36.863 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:36 smithi085 conmon[29319]: - 2021-02-10T04:38:36.863 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:36 smithi085 conmon[29319]: 02 2021-02-10T04:38:36.863 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:36 smithi085 conmon[29319]: - 2021-02-10T04:38:36.863 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:36 smithi085 conmon[29319]: 10 2021-02-10T04:38:36.864 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:36 smithi085 conmon[29319]: T 2021-02-10T04:38:36.864 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:36 smithi085 conmon[29319]: 04:38:36.138015+0000 osd.0 (osd 2021-02-10T04:38:36.864 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:36 smithi085 conmon[29319]: .0) 83 : cluster [DBG] 4.3 scrub ok 2021-02-10T04:38:36.864 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:36 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:38:36.864 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:36 smithi085 conmon[29319]: -02-10T04:38:36. 2021-02-10T04:38:36.864 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:36 smithi085 conmon[29319]: 161065+0000 mgr.y (mgr.14140) 715 2021-02-10T04:38:36.864 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:36 smithi085 conmon[29319]: : cluster [DBG] pgmap v1196: 168 pgs: 163 active+clean, 5 active+clean+snaptrim; 393 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.7 MiB/s rd, 3.4 MiB/s wr, 15 op/s 2021-02-10T04:38:36.864 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:38:36 smithi085 conmon[45443]: debug 2021-02-10T04:38:36.580+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:38:36.864 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:36 smithi085 conmon[35961]: cluster 2021-02-10T04:38:36.138015+0000 osd.0 (osd 2021-02-10T04:38:36.864 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:36 smithi085 conmon[35961]: .0) 83 : cluster [DBG] 4.3 scrub ok 2021-02-10T04:38:36.864 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:36 smithi085 conmon[35961]: cluster 2021-02-10T04:38:36.161065+0000 mgr.y (mgr.14140) 715 : cluster 2021-02-10T04:38:36.865 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:36 smithi085 conmon[35961]: [DBG] pgmap v1196: 168 pgs: 163 active+clean, 5 active+clean+snaptrim; 393 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.7 MiB/s rd, 3.4 MiB/s wr, 15 op/s 2021-02-10T04:38:36.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:36 smithi164 conmon[29288]: cluster 2021-02-10T04:38:36.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:36 smithi164 conmon[29288]: 2021-02 2021-02-10T04:38:36.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:36 smithi164 conmon[29288]: -10T04:38:36.138015+0000 osd.0 (osd.0) 83 : cluster [DBG] 4.3 scrub ok 2021-02-10T04:38:36.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:36 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:38:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:36 smithi164 conmon[29288]: -10T04:38:36.161065+0000 mgr.y (mgr.14140) 2021-02-10T04:38:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:36 smithi164 conmon[29288]: 715 : cluster [DBG] pgmap v1196: 168 pgs: 163 active+clean, 5 active+clean+snaptrim; 393 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.7 MiB/s rd, 3.4 MiB/s wr, 15 op/s 2021-02-10T04:38:36.987 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:38:36.990 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:38:37.423 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:38:37.441 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[45443]: debug 2021-02-10T04:38:37.361+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:38:37.584 INFO:tasks.rados.rados.0.smithi085.stdout:2706: done (5 left) 2021-02-10T04:38:37.584 INFO:tasks.rados.rados.0.smithi085.stdout:2707: copy_from oid 13 from oid 38 current snap is 311 2021-02-10T04:38:37.584 INFO:tasks.rados.rados.0.smithi085.stdout:2705: finishing write tid 2 to smithi08575397-37 2021-02-10T04:38:37.584 INFO:tasks.rados.rados.0.smithi085.stdout:2705: finishing write tid 3 to smithi08575397-37 2021-02-10T04:38:37.584 INFO:tasks.rados.rados.0.smithi085.stdout:2705: finishing write tid 4 to smithi08575397-37 2021-02-10T04:38:37.585 INFO:tasks.rados.rados.0.smithi085.stdout:2705: finishing write tid 5 to smithi08575397-37 2021-02-10T04:38:37.585 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 1430 (ObjNum 940 snap 311 seq_num 940) dirty exists 2021-02-10T04:38:37.585 INFO:tasks.rados.rados.0.smithi085.stdout:2705: left oid 37 (ObjNum 940 snap 311 seq_num 940) 2021-02-10T04:38:37.585 INFO:tasks.rados.rados.0.smithi085.stdout:2705: done (5 left) 2021-02-10T04:38:37.585 INFO:tasks.rados.rados.0.smithi085.stdout:2708: copy_from oid 22 from oid 21 current snap is 311 2021-02-10T04:38:37.585 INFO:tasks.rados.rados.0.smithi085.stdout:2709: snap_remove snap 287 2021-02-10T04:38:37.708 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:38:37.738 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset nodeep-scrub 2021-02-10T04:38:37.766 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[29319]: cluster 2021-02-10T04:38:37.766 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:38:37.766 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[29319]: :38:36.295592+0000 osd.6 (osd.6) 76 : cluster [DBG] 2.11 scrub ok 2021-02-10T04:38:37.767 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:38:37.767 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[29319]: 02-10T04:38:36.309356+0000 2021-02-10T04:38:37.767 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[29319]: osd.5 (osd.5) 77 : cluster [DBG] 2021-02-10T04:38:37.767 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[29319]: 2.15 scrub ok 2021-02-10T04:38:37.767 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:38:37.767 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[29319]: -02-10T04:38:36. 2021-02-10T04:38:37.767 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[29319]: 576761+0000 mon.a (mon.0) 2021-02-10T04:38:37.767 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[29319]: 1589 : cluster [DBG] osdmap e698: 8 total, 8 up, 7 in 2021-02-10T04:38:37.767 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[29319]: audit 2021-02-10T04:38:37.767 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[29319]: 2021-02-10T04:38:36.962428+0000 2021-02-10T04:38:37.767 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[29319]: mgr.y (mgr.14140) 716 : audit [DBG] 2021-02-10T04:38:37.767 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[29319]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:38:37.767 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:38:37.768 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[29319]: 38:37.121648+0000 osd.0 (osd.0) 2021-02-10T04:38:37.768 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[29319]: 84 : cluster [DBG] 6.18 scrub ok 2021-02-10T04:38:37.768 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[35961]: cluster 2021-02-10T04:38:37.768 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:38:37.768 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[35961]: 10T04:38:36.295592+0000 osd.6 (osd.6) 76 : cluster [DBG] 2.11 scrub ok 2021-02-10T04:38:37.768 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[35961]: 2021-02-10T04:38:37.768 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[35961]: cluster 2021-02-10T04:38:37.768 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:38:37.768 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[35961]: :38:36.309356+0000 osd.5 (osd. 2021-02-10T04:38:37.768 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[35961]: 5) 77 : cluster [DBG] 2.15 scrub ok 2021-02-10T04:38:37.768 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:38:37.769 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[35961]: 10T04:38:36.576761+0000 mon.a (mon. 2021-02-10T04:38:37.769 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[35961]: 0) 1589 : cluster [DBG] osdmap e698: 8 total, 8 up, 7 in 2021-02-10T04:38:37.769 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[35961]: audit 2021-02-10T04:38:37.769 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[35961]: 2021-02-10T04:38:36.962428+0000 2021-02-10T04:38:37.769 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[35961]: mgr.y (mgr.14140) 716 : audit [DBG] 2021-02-10T04:38:37.769 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[35961]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:38:37.769 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:37 smithi085 conmon[35961]: cluster 2021-02-10T04:38:37.121648+0000 osd.0 (osd.0) 84 : cluster [DBG] 6.18 scrub ok 2021-02-10T04:38:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:37 smithi164 conmon[29288]: cluster 2021-02-10T04:38:36.295592+0000 osd.6 ( 2021-02-10T04:38:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:37 smithi164 conmon[29288]: osd.6) 76 : cluster [DBG] 2.11 scrub ok 2021-02-10T04:38:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:37 smithi164 conmon[29288]: cluster 2021-02-10T04:38:36.309356+0000 osd.5 (osd.5) 2021-02-10T04:38:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:37 smithi164 conmon[29288]: 77 : cluster [DBG] 2.15 scrub ok 2021-02-10T04:38:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:37 smithi164 conmon[29288]: cluster 2021-02-10T04:38 2021-02-10T04:38:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:37 smithi164 conmon[29288]: :36.576761+0000 mon.a (mon.0) 2021-02-10T04:38:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:37 smithi164 conmon[29288]: 1589 : cluster [DBG] osdmap e698: 8 total, 8 up, 7 in 2021-02-10T04:38:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:37 smithi164 conmon[29288]: audit 2021-02-10T04:38:36. 2021-02-10T04:38:37.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:37 smithi164 conmon[29288]: 962428+0000 mgr.y (mgr.14140) 716 : audit [DBG] 2021-02-10T04:38:37.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:37 smithi164 conmon[29288]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:38:37.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:37 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:38:37.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:37 smithi164 conmon[29288]: :38:37.121648+0000 osd.0 (osd.0) 84 2021-02-10T04:38:37.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:37 smithi164 conmon[29288]: : cluster [DBG] 6.18 scrub ok 2021-02-10T04:38:38.035 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:38:38.265 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:38:38.265 INFO:tasks.rados.rados.0.smithi085.stdout:2709: done (6 left) 2021-02-10T04:38:38.266 INFO:tasks.rados.rados.0.smithi085.stdout:2710: delete oid 20 current snap is 311 2021-02-10T04:38:38.266 INFO:tasks.rados.rados.0.smithi085.stdout:2707: finishing copy_from to smithi08575397-13 2021-02-10T04:38:38.266 INFO:tasks.rados.rados.0.smithi085.stdout:2707: got expected ENOENT (src dne) 2021-02-10T04:38:38.266 INFO:tasks.rados.rados.0.smithi085.stdout:2707: finishing copy_from racing read to smithi08575397-13 2021-02-10T04:38:38.266 INFO:tasks.rados.rados.0.smithi085.stdout:2708: finishing copy_from to smithi08575397-22 2021-02-10T04:38:38.266 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 1425 (ObjNum 812 snap 265 seq_num 812) dirty exists 2021-02-10T04:38:38.266 INFO:tasks.rados.rados.0.smithi085.stdout:2708: finishing copy_from racing read to smithi08575397-22 2021-02-10T04:38:38.266 INFO:tasks.rados.rados.0.smithi085.stdout:2707: done (6 left) 2021-02-10T04:38:38.266 INFO:tasks.rados.rados.0.smithi085.stdout:2708: done (5 left) 2021-02-10T04:38:38.266 INFO:tasks.rados.rados.0.smithi085.stdout:2710: done (4 left) 2021-02-10T04:38:38.266 INFO:tasks.rados.rados.0.smithi085.stdout:2711: snap_create 2021-02-10T04:38:38.586 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:38:38.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: cluster 2021-02-10T04:38:37.218962+0000 osd.4 ( 2021-02-10T04:38:38.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: osd.4) 1126 : cluster [DBG] 2.27 scrub ok 2021-02-10T04:38:38.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: cluster 2021-02-10T04:38:37.582783+0000 mon.a (mon.0) 1590 : cluster [DBG] osdmap e699: 8 total, 8 up, 7 in 2021-02-10T04:38:38.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:38:38.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: T04:38:38.056068+0000 mon.a (mon. 2021-02-10T04:38:38.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: 0) 1591 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2021-02-10T04:38:38.587 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:38:38.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: 02-10T04:38:38.056360+0000 mon.a ( 2021-02-10T04:38:38.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: mon.0) 1592 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.6", "id": [5, 6]}]: dispatch 2021-02-10T04:38:38.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:38:38.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: -10T04:38:38.056596+0000 mon.a ( 2021-02-10T04:38:38.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: mon.0) 1593 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.8", "id": [1, 3]}]: dispatch 2021-02-10T04:38:38.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: audit 2021-02-10T04:38:38.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: 2021-02-10T04:38:38.056845+0000 2021-02-10T04:38:38.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: mon.a (mon.0) 1594 : audit [INF] 2021-02-10T04:38:38.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [7, 3]}]: dispatch 2021-02-10T04:38:38.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: audit 2021-02-10T04:38:38 2021-02-10T04:38:38.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: .057123+0000 mon.a 2021-02-10T04:38:38.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: (mon.0) 1595 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.18", "id": [4, 3]}]: dispatch 2021-02-10T04:38:38.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: audit 2021-02-10T04:38:38.589 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: 2021-02-10T04:38:38.057320 2021-02-10T04:38:38.589 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1596 : audit [INF] 2021-02-10T04:38:38.589 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1b", "id": [0, 3]}]: dispatch 2021-02-10T04:38:38.589 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: audit 2021-02-10T04: 2021-02-10T04:38:38.589 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: 38:38.057746+0000 mon.a (mon.0) 2021-02-10T04:38:38.589 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: 1597 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1c", "id": [3, 0]}]: dispatch 2021-02-10T04:38:38.589 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:38:38.589 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: T04:38:38.121668+0000 mon.a (mon 2021-02-10T04:38:38.589 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: .0) 1598 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:38:38.589 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: audit 2021 2021-02-10T04:38:38.589 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: -02-10T04:38:38.122031+0000 2021-02-10T04:38:38.589 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: mon.a (mon.0) 1599 : audit [INF] 2021-02-10T04:38:38.589 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:38:38.590 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: cluster 2021-02-10T04:38:38.590 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: 2021-02-10T04:38: 2021-02-10T04:38:38.590 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: 38.161661+0000 mgr.y (mgr.14140) 717 2021-02-10T04:38:38.590 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: : cluster [DBG] pgmap v1199: 168 pgs: 1 active+clean+snaptrim_wait, 161 active+clean, 6 active+clean+snaptrim; 400 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 8.1 MiB/s rd, 3.0 MiB/s wr, 18 op/s 2021-02-10T04:38:38.590 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:38:38.590 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: 10T04:38:38.226368+0000 mon.a (mon.0) 1600 : 2021-02-10T04:38:38.590 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2021-02-10T04:38:38.590 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: audit 2021-02-10T04:38:38.591 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:38:38.591 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: :38:38.226505+0000 mon.a (mon.0) 1601 : audit [INF] 2021-02-10T04:38:38.591 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.6", "id": [5, 6]}]': finished 2021-02-10T04:38:38.591 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: audit 2021-02-10T04:38 2021-02-10T04:38:38.591 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: :38.226626+0000 mon.a (mon.0) 1602 : 2021-02-10T04:38:38.591 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.8", "id": [1, 3]}]': finished 2021-02-10T04:38:38.591 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: audit 2021-02-10T04:38 2021-02-10T04:38:38.591 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: :38.226703+0000 mon.a (mon.0) 1603 : audit 2021-02-10T04:38:38.591 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [7, 3]}]': finished 2021-02-10T04:38:38.591 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: audit 2021-02-10T04: 2021-02-10T04:38:38.591 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: 38:38.226793+0000 mon.a (mon.0) 1604 2021-02-10T04:38:38.591 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.18", "id": [4, 3]}]': finished 2021-02-10T04:38:38.591 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: audit 2021-02-10T04:38: 2021-02-10T04:38:38.592 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: 38.226865+0000 mon.a (mon.0) 1605 : audit 2021-02-10T04:38:38.592 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1b", "id": [0, 3]}]': finished 2021-02-10T04:38:38.592 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:38:38.592 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: 10T04:38:38.226986+0000 mon.a (mon 2021-02-10T04:38:38.592 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: .0) 1606 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1c", "id": [3, 0]}]': finished 2021-02-10T04:38:38.592 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: cluster 2021-02-10T04:38:38.592 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: 2021-02-10T04:38:38. 2021-02-10T04:38:38.592 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: 227077+0000 mon.a (mon.0) 1607 : cluster 2021-02-10T04:38:38.592 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:38 smithi164 conmon[29288]: [DBG] osdmap e700: 8 total, 8 up, 7 in 2021-02-10T04:38:38.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: cluster 2021-02-10T04:38:37.218962+0000 osd.4 (osd.4) 1126 : cluster [DBG] 2.27 scrub ok 2021-02-10T04:38:38.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: cluster 2021-02-10T04:38:37.582783+0000 mon.a (mon.0) 1590 : cluster [DBG] osdmap e699: 8 total, 8 up, 7 in 2021-02-10T04:38:38.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: audit 2021-02-10T04:38:38.056068+0000 mon.a (mon.0) 1591 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2021-02-10T04:38:38.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: audit 2021-02-10T04:38:38.056360+0000 mon.a (mon.0) 1592 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.6", "id": [5, 6]}]: dispatch 2021-02-10T04:38:38.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: audit 2021-02-10T04:38:38.056596+0000 mon.a (mon.0) 1593 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.8", "id": [1, 3]}]: dispatch 2021-02-10T04:38:38.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: audit 2021-02-10T04:38:38.056845+0000 mon.a (mon.0) 1594 : audit [INF] 2021-02-10T04:38:38.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [7, 3]}]: dispatch 2021-02-10T04:38:38.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: audit 2021-02-10T04:38:38.057123+0000 mon.a (mon.0) 1595 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.18", "id": [4, 3]}]: dispatch 2021-02-10T04:38:38.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: audit 2021-02-10T04:38:38.057320+0000 2021-02-10T04:38:38.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: mon.a (mon.0) 1596 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1b", "id": [0, 3]}]: dispatch 2021-02-10T04:38:38.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: audit 2021-02-10T04: 2021-02-10T04:38:38.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: 38:38.057746+0000 mon.a (mon.0 2021-02-10T04:38:38.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: ) 1597 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1c", "id": [3, 0]}]: dispatch 2021-02-10T04:38:38.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:38:38.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: 10T04:38:38.121668+0000 mon.a ( 2021-02-10T04:38:38.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: mon.0) 1598 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:38:38.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:38:38.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: 02-10T04:38:38.122031+0000 mon.a (mon.0) 1599 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:38:38.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:38:38.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: :38:38.161661+0000 mgr.y (mgr.14140 2021-02-10T04:38:38.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: ) 717 : cluster [DBG] pgmap v1199: 168 pgs: 1 active+clean+snaptrim_wait, 161 active+clean, 6 active+clean+snaptrim; 400 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 8.1 MiB/s rd, 3.0 MiB/s wr, 18 op/s 2021-02-10T04:38:38.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: audit 2021-02-10T04:38:38.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:38:38.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: 38:38.226368+0000 mon.a (mon.0) 1600 : audit 2021-02-10T04:38:38.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2021-02-10T04:38:38.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:38:38.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: T04:38:38.226505+0000 mon.a (mon.0) 1601 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.6", "id": [5, 6]}]': finished 2021-02-10T04:38:38.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:38:38.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: 02-10T04:38:38.226626+0000 mon.a ( 2021-02-10T04:38:38.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: mon.0) 1602 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.8", "id": [1, 3]}]': finished 2021-02-10T04:38:38.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: audit 2021-02-10T04:38:38.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: 2021-02-10T04:38:38. 2021-02-10T04:38:38.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: 226703+0000 mon.a (mon.0) 1603 : audit [INF] 2021-02-10T04:38:38.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [7, 3]}]': finished 2021-02-10T04:38:38.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: audit 2021-02-10T04:38:38.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: 2021-02-10T04:38:38. 2021-02-10T04:38:38.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: 226793+0000 mon.a (mon.0) 1604 : audit 2021-02-10T04:38:38.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.18", "id": [4, 3]}]': finished 2021-02-10T04:38:38.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: audit 2021-02-10T04:38 2021-02-10T04:38:38.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: :38.226865+0000 mon.a (mon.0) 1605 : audit [INF] 2021-02-10T04:38:38.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1b", "id": [0, 3]}]': finished 2021-02-10T04:38:38.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:38:38.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: 10T04:38:38.226986+0000 mon.a (mon.0) 1606 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1c", "id": [3, 0]}]': finished 2021-02-10T04:38:38.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:38:38.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: 38:38.227077+0000 mon.a (mon.0) 2021-02-10T04:38:38.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[29319]: 1607 : cluster [DBG] osdmap e700: 8 total, 8 up, 7 in 2021-02-10T04:38:38.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: cluster 2021-02-10T04:38:37.218962+0000 osd.4 (osd 2021-02-10T04:38:38.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: .4) 1126 : cluster [DBG] 2.27 scrub ok 2021-02-10T04:38:38.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:38:38.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: 10T04:38:37.582783+0000 mon.a (mon.0) 1590 : 2021-02-10T04:38:38.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: cluster [DBG] osdmap e699: 8 total, 8 up, 7 in 2021-02-10T04:38:38.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:38:38.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: -10T04:38:38.056068+0000 mon.a (mon.0) 1591 2021-02-10T04:38:38.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2021-02-10T04:38:38.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:38:38.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: -10T04:38:38.056360+0000 mon.a (mon.0 2021-02-10T04:38:38.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: ) 1592 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.6", "id": [5, 6]}]: dispatch 2021-02-10T04:38:38.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:38:38.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: -10T04:38:38.056596+0000 mon.a (mon.0) 2021-02-10T04:38:38.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: 1593 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.8", "id": [1, 3]}]: dispatch 2021-02-10T04:38:38.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:38:38.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: T04:38:38.056845+0000 mon.a (mon.0) 1594 : audit 2021-02-10T04:38:38.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [7, 3]}]: dispatch 2021-02-10T04:38:38.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:38:38.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: 10T04:38:38.057123+0000 mon.a (mon.0 2021-02-10T04:38:38.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: ) 1595 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.18", "id": [4, 3]}]: dispatch 2021-02-10T04:38:38.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:38:38.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: 10T04:38:38.057320+0000 mon.a (mon.0) 1596 2021-02-10T04:38:38.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1b", "id": [0, 3]}]: dispatch 2021-02-10T04:38:38.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:38:38.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: -10T04:38:38.057746+0000 mon.a (mon.0 2021-02-10T04:38:38.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: ) 1597 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1c", "id": [3, 0]}]: dispatch 2021-02-10T04:38:38.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:38:38.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: -10T04:38:38.121668+0000 mon.a 2021-02-10T04:38:38.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: (mon.0) 1598 : audit 2021-02-10T04:38:38.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:38:38.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:38:38.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: T04:38:38.122031+0000 mon.a (mon.0) 1599 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:38:38.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:38:38.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: -10T04:38:38.161661+0000 mgr.y (mgr.14140) 2021-02-10T04:38:38.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: 717 : cluster [DBG] pgmap v1199: 168 pgs: 1 active+clean+snaptrim_wait, 161 active+clean, 6 active+clean+snaptrim; 400 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 8.1 MiB/s rd, 3.0 MiB/s wr, 18 op/s 2021-02-10T04:38:38.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:38:38.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: -10T04:38:38.226368+0000 mon.a (mon.0) 2021-02-10T04:38:38.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: 1600 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2021-02-10T04:38:38.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:38:38.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: -10T04:38:38.226505+0000 mon.a (mon.0) 1601 : 2021-02-10T04:38:38.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.6", "id": [5, 6]}]': finished 2021-02-10T04:38:38.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: audit 2021 2021-02-10T04:38:38.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: -02-10T04:38:38.226626+0000 mon.a (mon 2021-02-10T04:38:38.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: .0) 1602 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.8", "id": [1, 3]}]': finished 2021-02-10T04:38:38.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:38:38.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: -10T04:38:38.226703+0000 mon.a (mon.0) 2021-02-10T04:38:38.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: 1603 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [7, 3]}]': finished 2021-02-10T04:38:38.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:38:38.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: 02-10T04:38:38.226793+0000 mon.a (mon.0 2021-02-10T04:38:38.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: ) 1604 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.18", "id": [4, 3]}]': finished 2021-02-10T04:38:38.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: audit 2021-02-10T04:38:38.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: 2021-02-10T04:38:38.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: 2021 2021-02-10T04:38:38.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: - 2021-02-10T04:38:38.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: 02-10T04:38:38.226865 2021-02-10T04:38:38.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1605 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1b", "id": [0, 3]}]': finished 2021-02-10T04:38:38.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:38:38.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: -10T04:38:38.226986+0000 mon.a (mon.0) 2021-02-10T04:38:38.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: 1606 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1c", "id": [3, 0]}]': finished 2021-02-10T04:38:38.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:38:38.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: -10T04:38:38.227077+0000 mon.a (mon.0) 2021-02-10T04:38:38.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:38 smithi085 conmon[35961]: 1607 : cluster [DBG] osdmap e700: 8 total, 8 up, 7 in 2021-02-10T04:38:39.235 INFO:tasks.rados.rados.0.smithi085.stdout:2711: done (4 left) 2021-02-10T04:38:39.235 INFO:tasks.rados.rados.0.smithi085.stdout:2712: rollback oid 5 current snap is 312 2021-02-10T04:38:39.235 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 5 to 266 2021-02-10T04:38:39.235 INFO:tasks.rados.rados.0.smithi085.stdout:2713: snap_create 2021-02-10T04:38:39.244 INFO:tasks.rados.rados.0.smithi085.stdout:2712: finishing rollback tid 1 to smithi08575397-5 2021-02-10T04:38:39.245 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 785 (ObjNum 1953066355 snap 942696808 seq_num 859125557) dirty dne 2021-02-10T04:38:39.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:39 smithi164 conmon[29288]: cluster 2021-02-10T04:38:38. 2021-02-10T04:38:39.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:39 smithi164 conmon[29288]: 163219+0000 osd.0 (osd.0) 85 : cluster [DBG] 2.10 scrub ok 2021-02-10T04:38:39.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:39 smithi164 conmon[29288]: audit 2021-02-10T04:38:38.280006+0000 mgr.y (mgr.14140) 718 : 2021-02-10T04:38:39.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:39 smithi164 conmon[29288]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:38:39.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:39 smithi164 conmon[29288]: cluster 2021-02-10T04:38: 2021-02-10T04:38:39.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:39 smithi164 conmon[29288]: 39.235325+0000 mon.a (mon.0) 1608 : cluster [DBG] osdmap e701: 8 total, 8 up, 7 in 2021-02-10T04:38:39.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:39 smithi085 conmon[29319]: cluster 2021-02-10T04:38:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:39 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:38:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:39 smithi085 conmon[29319]: 10T04:38:38.163219+0000 osd.0 (osd.0) 85 : cluster [DBG] 2.10 scrub ok 2021-02-10T04:38:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:39 smithi085 conmon[29319]: audit 2021-02-10T04:38:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:39 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:38:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:39 smithi085 conmon[29319]: :38:38.280006+0000 mgr.y (mgr.14140) 718 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:38:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:39 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:38:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:39 smithi085 conmon[29319]: 10T04:38:39.235325+0000 mon.a (mon. 2021-02-10T04:38:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:39 smithi085 conmon[29319]: 0) 1608 : cluster [DBG] osdmap e701: 8 total, 8 up, 7 in 2021-02-10T04:38:39.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:39 smithi085 conmon[35961]: cluster 2021-02-10T04:38:38. 2021-02-10T04:38:39.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:39 smithi085 conmon[35961]: 163219+0000 osd.0 (osd.0) 85 : cluster [DBG] 2.10 scrub ok 2021-02-10T04:38:39.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:39 smithi085 conmon[35961]: audit 2021 2021-02-10T04:38:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:39 smithi085 conmon[35961]: -02-10T04:38:38.280006+0000 mgr.y 2021-02-10T04:38:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:39 smithi085 conmon[35961]: (mgr.14140) 718 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:38:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:39 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:38:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:39 smithi085 conmon[35961]: 04:38:39.235325+0000 mon.a (mon.0) 1608 : cluster [DBG] osdmap e701: 8 total, 8 up, 7 in 2021-02-10T04:38:40.236 INFO:tasks.rados.rados.0.smithi085.stdout:2712: done (5 left) 2021-02-10T04:38:40.236 INFO:tasks.rados.rados.0.smithi085.stdout:2713: done (4 left) 2021-02-10T04:38:40.236 INFO:tasks.rados.rados.0.smithi085.stdout:2714: snap_remove snap 295 2021-02-10T04:38:40.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:40 smithi164 conmon[29288]: cluster 2021-02-10T04:38:39.191905 2021-02-10T04:38:40.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:40 smithi164 conmon[29288]: +0000 osd.0 (osd.0) 86 : cluster [DBG] 6.16 scrub ok 2021-02-10T04:38:40.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:40 smithi164 conmon[29288]: cluster 2021-02-10T04:38:40.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:40 smithi164 conmon[29288]: 2021-02-10T04:38:40.162251+0000 2021-02-10T04:38:40.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:40 smithi164 conmon[29288]: mgr.y (mgr.14140) 719 : cluster 2021-02-10T04:38:40.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:40 smithi164 conmon[29288]: [DBG] pgmap v1202: 168 pgs: 1 peering, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 415 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 8.1 MiB/s rd, 3.4 MiB/s wr, 21 op/s; 223 B/s, 2 objects/s recovering 2021-02-10T04:38:40.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:40 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:38:40.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:40 smithi164 conmon[29288]: 02-10T04:38:40. 2021-02-10T04:38:40.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:40 smithi164 conmon[29288]: 235195+0000 mon.a (mon.0 2021-02-10T04:38:40.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:40 smithi164 conmon[29288]: ) 1609 : cluster [DBG] osdmap e702: 8 total, 8 up, 7 in 2021-02-10T04:38:40.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:40 smithi085 conmon[29319]: cluster 2021-02-10T04:38:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:40 smithi085 conmon[29319]: 2021-02-10T04:38:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:40 smithi085 conmon[29319]: 2021-02-10T04:38:39.191905+0000 osd.0 2021-02-10T04:38:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:40 smithi085 conmon[29319]: (osd.0) 86 : cluster [DBG] 6.16 scrub ok 2021-02-10T04:38:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:40 smithi085 conmon[29319]: cluster 2021-02-10T04:38: 2021-02-10T04:38:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:40 smithi085 conmon[29319]: 40.162251+0000 mgr.y (mgr.14140) 719 : cluster [DBG] pgmap v1202: 168 pgs: 1 peering, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 415 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 8.1 MiB/s rd, 3.4 MiB/s wr, 21 op/s; 223 B/s, 2 objects/s recovering 2021-02-10T04:38:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:40 smithi085 conmon[29319]: 2021-02-10T04:38:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:40 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:38:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:40 smithi085 conmon[29319]: 10T04:38:40.235195+0000 mon.a (mon.0) 1609 : cluster [DBG] osdmap e702: 8 total, 8 up, 7 in 2021-02-10T04:38:40.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:40 smithi085 conmon[35961]: cluster 2021-02-10T04:38:40.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:40 smithi085 conmon[35961]: 2021- 2021-02-10T04:38:40.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:40 smithi085 conmon[35961]: 02-10T04:38:39.191905+0000 osd.0 (osd.0) 86 : cluster 2021-02-10T04:38:40.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:40 smithi085 conmon[35961]: [DBG] 6.16 scrub ok 2021-02-10T04:38:40.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:40 smithi085 conmon[35961]: cluster 2021-02-10T04:38:40.162251+0000 mgr.y (mgr 2021-02-10T04:38:40.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:40 smithi085 conmon[35961]: .14140) 719 : cluster [DBG] pgmap v1202: 168 pgs: 1 peering, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 415 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 8.1 MiB/s rd, 3.4 MiB/s wr, 21 op/s; 223 B/s, 2 objects/s recovering 2021-02-10T04:38:40.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:40 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:38:40.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:40 smithi085 conmon[35961]: -02-10T04:38:40.235195+0000 2021-02-10T04:38:40.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:40 smithi085 conmon[35961]: mon.a (mon.0) 1609 : cluster [DBG] osdmap e702: 8 total, 8 up, 7 in 2021-02-10T04:38:40.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:40 smithi085 conmon[35961]: 2021-02-10T04:38:41.239 INFO:tasks.rados.rados.0.smithi085.stdout:2714: done (4 left) 2021-02-10T04:38:41.239 INFO:tasks.rados.rados.0.smithi085.stdout:2715: read oid 30 snap -1 2021-02-10T04:38:41.240 INFO:tasks.rados.rados.0.smithi085.stdout:2715: expect (ObjNum 912 snap 299 seq_num 912) 2021-02-10T04:38:41.240 INFO:tasks.rados.rados.0.smithi085.stdout:2716: rollback oid 11 current snap is 313 2021-02-10T04:38:41.240 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 11 to 271 2021-02-10T04:38:41.240 INFO:tasks.rados.rados.0.smithi085.stdout:2717: write oid 4 current snap is 313 2021-02-10T04:38:41.240 INFO:tasks.rados.rados.0.smithi085.stdout:2717: seq_num 942 ranges {632194=795977,2123920=790332,3419165=413029} 2021-02-10T04:38:41.247 INFO:tasks.rados.rados.0.smithi085.stdout:2717: writing smithi08575397-4 from 632194 to 1428171 tid 1 2021-02-10T04:38:41.273 INFO:tasks.rados.rados.0.smithi085.stdout:2717: writing smithi08575397-4 from 2123920 to 2914252 tid 2 2021-02-10T04:38:41.274 INFO:tasks.rados.rados.0.smithi085.stdout:2717: writing smithi08575397-4 from 3419165 to 3832194 tid 3 2021-02-10T04:38:41.274 INFO:tasks.rados.rados.0.smithi085.stdout:2718: write (excl) oid 32 current snap is 313 2021-02-10T04:38:41.274 INFO:tasks.rados.rados.0.smithi085.stdout:2718: seq_num 943 ranges {649001=587579,1898755=533990,3162941=686060} 2021-02-10T04:38:41.274 INFO:tasks.rados.rados.0.smithi085.stdout:2718: writing smithi08575397-32 from 649001 to 1236580 tid 1 2021-02-10T04:38:41.277 INFO:tasks.rados.rados.0.smithi085.stdout:2718: writing smithi08575397-32 from 1898755 to 2432745 tid 2 2021-02-10T04:38:41.285 INFO:tasks.rados.rados.0.smithi085.stdout:2718: writing smithi08575397-32 from 3162941 to 3849001 tid 3 2021-02-10T04:38:41.286 INFO:tasks.rados.rados.0.smithi085.stdout:2719: copy_from oid 9 from oid 1 current snap is 313 2021-02-10T04:38:41.286 INFO:tasks.rados.rados.0.smithi085.stdout:2716: finishing rollback tid 1 to smithi08575397-11 2021-02-10T04:38:41.286 INFO:tasks.rados.rados.0.smithi085.stdout:2715: expect (ObjNum 912 snap 299 seq_num 912) 2021-02-10T04:38:41.419 INFO:tasks.rados.rados.0.smithi085.stdout:2716: finishing rollback tid 2 to smithi08575397-11 2021-02-10T04:38:41.419 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 1436 (ObjNum 817 snap 265 seq_num 817) dirty exists 2021-02-10T04:38:41.419 INFO:tasks.rados.rados.0.smithi085.stdout:2717: finishing write tid 1 to smithi08575397-4 2021-02-10T04:38:41.420 INFO:tasks.rados.rados.0.smithi085.stdout:2717: finishing write tid 2 to smithi08575397-4 2021-02-10T04:38:41.420 INFO:tasks.rados.rados.0.smithi085.stdout:2715: done (8 left) 2021-02-10T04:38:41.420 INFO:tasks.rados.rados.0.smithi085.stdout:2716: done (7 left) 2021-02-10T04:38:41.420 INFO:tasks.rados.rados.0.smithi085.stdout:2720: read oid 10 snap -1 2021-02-10T04:38:41.420 INFO:tasks.rados.rados.0.smithi085.stdout:2720: expect (ObjNum 922 snap 304 seq_num 922) 2021-02-10T04:38:41.420 INFO:tasks.rados.rados.0.smithi085.stdout:2721: write (excl) oid 6 current snap is 313 2021-02-10T04:38:41.420 INFO:tasks.rados.rados.0.smithi085.stdout:2717: finishing write tid 3 to smithi08575397-4 2021-02-10T04:38:41.420 INFO:tasks.rados.rados.0.smithi085.stdout:2717: finishing write tid 4 to smithi08575397-4 2021-02-10T04:38:41.420 INFO:tasks.rados.rados.0.smithi085.stdout:2717: finishing write tid 5 to smithi08575397-4 2021-02-10T04:38:41.420 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 4 v 1342 (ObjNum 942 snap 313 seq_num 942) dirty exists 2021-02-10T04:38:41.420 INFO:tasks.rados.rados.0.smithi085.stdout:2717: left oid 4 (ObjNum 942 snap 313 seq_num 942) 2021-02-10T04:38:41.420 INFO:tasks.rados.rados.0.smithi085.stdout:2718: finishing write tid 1 to smithi08575397-32 2021-02-10T04:38:41.420 INFO:tasks.rados.rados.0.smithi085.stdout:2718: finishing write tid 2 to smithi08575397-32 2021-02-10T04:38:41.421 INFO:tasks.rados.rados.0.smithi085.stdout:2718: finishing write tid 3 to smithi08575397-32 2021-02-10T04:38:41.421 INFO:tasks.rados.rados.0.smithi085.stdout:2718: finishing write tid 4 to smithi08575397-32 2021-02-10T04:38:41.421 INFO:tasks.rados.rados.0.smithi085.stdout:2718: finishing write tid 5 to smithi08575397-32 2021-02-10T04:38:41.421 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 32 v 667 (ObjNum 943 snap 313 seq_num 943) dirty exists 2021-02-10T04:38:41.421 INFO:tasks.rados.rados.0.smithi085.stdout:2718: left oid 32 (ObjNum 943 snap 313 seq_num 943) 2021-02-10T04:38:41.421 INFO:tasks.rados.rados.0.smithi085.stdout:2719: finishing copy_from to smithi08575397-9 2021-02-10T04:38:41.421 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 9 v 1332 (ObjNum 854 snap 277 seq_num 854) dirty exists 2021-02-10T04:38:41.421 INFO:tasks.rados.rados.0.smithi085.stdout:2719: finishing copy_from racing read to smithi08575397-9 2021-02-10T04:38:41.421 INFO:tasks.rados.rados.0.smithi085.stdout:2721: seq_num 944 ranges {665808=662828,1957237=677648,3074011=609486} 2021-02-10T04:38:41.425 INFO:tasks.rados.rados.0.smithi085.stdout:2721: writing smithi08575397-6 from 665808 to 1328636 tid 1 2021-02-10T04:38:41.434 INFO:tasks.rados.rados.0.smithi085.stdout:2721: writing smithi08575397-6 from 1957237 to 2634885 tid 2 2021-02-10T04:38:41.441 INFO:tasks.rados.rados.0.smithi085.stdout:2721: writing smithi08575397-6 from 3074011 to 3683497 tid 3 2021-02-10T04:38:41.441 INFO:tasks.rados.rados.0.smithi085.stdout:2717: done (8 left) 2021-02-10T04:38:41.442 INFO:tasks.rados.rados.0.smithi085.stdout:2718: done (7 left) 2021-02-10T04:38:41.442 INFO:tasks.rados.rados.0.smithi085.stdout:2719: done (6 left) 2021-02-10T04:38:41.442 INFO:tasks.rados.rados.0.smithi085.stdout:2722: read oid 15 snap 269 2021-02-10T04:38:41.442 INFO:tasks.rados.rados.0.smithi085.stdout:2722: expect (ObjNum 217 snap 49 seq_num 217) 2021-02-10T04:38:41.442 INFO:tasks.rados.rados.0.smithi085.stdout:2723: read oid 18 snap -1 2021-02-10T04:38:41.442 INFO:tasks.rados.rados.0.smithi085.stdout:2723: expect (ObjNum 932 snap 310 seq_num 932) 2021-02-10T04:38:41.445 INFO:tasks.rados.rados.0.smithi085.stdout:2720: expect (ObjNum 922 snap 304 seq_num 922) 2021-02-10T04:38:41.604 INFO:tasks.rados.rados.0.smithi085.stdout:2721: finishing write tid 1 to smithi08575397-6 2021-02-10T04:38:41.604 INFO:tasks.rados.rados.0.smithi085.stdout:2721: finishing write tid 2 to smithi08575397-6 2021-02-10T04:38:41.604 INFO:tasks.rados.rados.0.smithi085.stdout:2720: done (7 left) 2021-02-10T04:38:41.604 INFO:tasks.rados.rados.0.smithi085.stdout:2724: delete oid 46 current snap is 313 2021-02-10T04:38:41.604 INFO:tasks.rados.rados.0.smithi085.stdout:2723: expect (ObjNum 932 snap 310 seq_num 932) 2021-02-10T04:38:41.869 INFO:tasks.rados.rados.0.smithi085.stdout:2721: finishing write tid 3 to smithi08575397-6 2021-02-10T04:38:41.870 INFO:tasks.rados.rados.0.smithi085.stdout:2721: finishing write tid 4 to smithi08575397-6 2021-02-10T04:38:41.870 INFO:tasks.rados.rados.0.smithi085.stdout:2721: finishing write tid 5 to smithi08575397-6 2021-02-10T04:38:41.870 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 1346 (ObjNum 944 snap 313 seq_num 944) dirty exists 2021-02-10T04:38:41.870 INFO:tasks.rados.rados.0.smithi085.stdout:2721: left oid 6 (ObjNum 944 snap 313 seq_num 944) 2021-02-10T04:38:41.870 INFO:tasks.rados.rados.0.smithi085.stdout:2721: done (7 left) 2021-02-10T04:38:41.870 INFO:tasks.rados.rados.0.smithi085.stdout:2723: done (6 left) 2021-02-10T04:38:41.870 INFO:tasks.rados.rados.0.smithi085.stdout:2724: done (5 left) 2021-02-10T04:38:41.870 INFO:tasks.rados.rados.0.smithi085.stdout:2725: copy_from oid 23 from oid 35 current snap is 313 2021-02-10T04:38:41.870 INFO:tasks.rados.rados.0.smithi085.stdout:2722: expect (ObjNum 217 snap 49 seq_num 217) 2021-02-10T04:38:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:41 smithi164 conmon[29288]: cluster 2021-02-10T04:38:40. 2021-02-10T04:38:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:41 smithi164 conmon[29288]: 414597+0000 osd.6 (osd.6) 77 : cluster [DBG] 6.1 scrub ok 2021-02-10T04:38:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:41 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:38:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:41 smithi164 conmon[29288]: -10T04:38:40.930204+0000 osd.7 (osd.7) 58 : cluster 2021-02-10T04:38:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:41 smithi164 conmon[29288]: [DBG] purged_snaps scrub starts 2021-02-10T04:38:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:41 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:38:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:41 smithi164 conmon[29288]: 02-10T04:38:40.931018+0000 osd.7 2021-02-10T04:38:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:41 smithi164 conmon[29288]: (osd.7) 59 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:38:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:41 smithi164 conmon[29288]: 2021-02-10T04:38:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:41 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:38:41.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:41 smithi164 conmon[29288]: -10T04:38:41.238178+0000 2021-02-10T04:38:41.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:41 smithi164 conmon[29288]: mon.a (mon.0) 1610 : cluster [DBG] osdmap e703: 8 total, 8 up, 7 in 2021-02-10T04:38:41.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:41 smithi164 conmon[29288]: 2021-02-10T04:38:41.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:41 smithi085 conmon[29319]: cluster 2021-02-10T04:38:40.414597+0000 osd.6 (osd.6) 77 : cluster [DBG] 6.1 scrub ok 2021-02-10T04:38:41.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:41 smithi085 conmon[29319]: cluster 2021-02-10T04:38:40.930204+0000 osd.7 (osd.7) 58 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:38:41.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:41 smithi085 conmon[29319]: cluster 2021-02-10T04:38:41.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:41 smithi085 conmon[29319]: 2021-02-10T04:38:40.931018+0000 osd.7 (osd.7) 59 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:38:41.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:41 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:38:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:41 smithi085 conmon[29319]: 02-10T04:38:41.238178+0000 2021-02-10T04:38:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:41 smithi085 conmon[29319]: mon.a (mon.0) 1610 : cluster [DBG] osdmap e703: 8 total, 8 up, 7 in 2021-02-10T04:38:41.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:41 smithi085 conmon[35961]: cluster 2021-02-10T04:38:41.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:41 smithi085 conmon[35961]: 2021-02-10T04:38:41.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:41 smithi085 conmon[35961]: 2021-02-10T04:38:40. 2021-02-10T04:38:41.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:41 smithi085 conmon[35961]: 414597+0000 osd.6 (osd.6) 77 : cluster [DBG] 6.1 scrub ok 2021-02-10T04:38:41.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:41 smithi085 conmon[35961]: cluster 2021-02-10T04:38:40 2021-02-10T04:38:41.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:41 smithi085 conmon[35961]: .930204+0000 osd.7 (osd.7) 58 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:38:41.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:41 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:38:41.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:41 smithi085 conmon[35961]: 02-10T04:38:40.931018+0000 osd.7 (osd.7) 2021-02-10T04:38:41.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:41 smithi085 conmon[35961]: 59 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:38:41.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:41 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:38:41.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:41 smithi085 conmon[35961]: 04:38:41.238178+0000 mon.a (mon 2021-02-10T04:38:41.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:41 smithi085 conmon[35961]: .0) 1610 : cluster [DBG] osdmap e703: 8 total, 8 up, 7 in 2021-02-10T04:38:42.166 INFO:tasks.rados.rados.0.smithi085.stdout:2725: finishing copy_from to smithi08575397-23 2021-02-10T04:38:42.167 INFO:tasks.rados.rados.0.smithi085.stdout:2725: got expected ENOENT (src dne) 2021-02-10T04:38:42.167 INFO:tasks.rados.rados.0.smithi085.stdout:2725: finishing copy_from racing read to smithi08575397-23 2021-02-10T04:38:42.167 INFO:tasks.rados.rados.0.smithi085.stdout:2722: done (5 left) 2021-02-10T04:38:42.167 INFO:tasks.rados.rados.0.smithi085.stdout:2725: done (4 left) 2021-02-10T04:38:42.167 INFO:tasks.rados.rados.0.smithi085.stdout:2726: delete oid 35 current snap is 313 2021-02-10T04:38:42.169 INFO:tasks.rados.rados.0.smithi085.stdout:2726: done (4 left) 2021-02-10T04:38:42.170 INFO:tasks.rados.rados.0.smithi085.stdout:2727: write (excl) oid 46 current snap is 313 2021-02-10T04:38:42.170 INFO:tasks.rados.rados.0.smithi085.stdout:2727: seq_num 947 ranges {716229=488575,1616330=541328,2756280=559655} 2021-02-10T04:38:42.177 INFO:tasks.rados.rados.0.smithi085.stdout:2727: writing smithi08575397-46 from 716229 to 1204804 tid 1 2021-02-10T04:38:42.185 INFO:tasks.rados.rados.0.smithi085.stdout:2727: writing smithi08575397-46 from 1616330 to 2157658 tid 2 2021-02-10T04:38:42.192 INFO:tasks.rados.rados.0.smithi085.stdout:2727: writing smithi08575397-46 from 2756280 to 3315935 tid 3 2021-02-10T04:38:42.192 INFO:tasks.rados.rados.0.smithi085.stdout:2728: delete oid 1 current snap is 313 2021-02-10T04:38:42.193 INFO:tasks.rados.rados.0.smithi085.stdout:2727: finishing write tid 1 to smithi08575397-46 2021-02-10T04:38:42.196 INFO:tasks.rados.rados.0.smithi085.stdout:2727: finishing write tid 2 to smithi08575397-46 2021-02-10T04:38:42.196 INFO:tasks.rados.rados.0.smithi085.stdout:2728: done (5 left) 2021-02-10T04:38:42.196 INFO:tasks.rados.rados.0.smithi085.stdout:2729: read oid 16 snap -1 2021-02-10T04:38:42.197 INFO:tasks.rados.rados.0.smithi085.stdout:2729: expect deleted 2021-02-10T04:38:42.197 INFO:tasks.rados.rados.0.smithi085.stdout:2730: write oid 34 current snap is 313 2021-02-10T04:38:42.197 INFO:tasks.rados.rados.0.smithi085.stdout:2730: seq_num 949 ranges {749843=639073,2133294=544997,3211126=582199} 2021-02-10T04:38:42.206 INFO:tasks.rados.rados.0.smithi085.stdout:2730: writing smithi08575397-34 from 749843 to 1388916 tid 1 2021-02-10T04:38:42.214 INFO:tasks.rados.rados.0.smithi085.stdout:2730: writing smithi08575397-34 from 2133294 to 2678291 tid 2 2021-02-10T04:38:42.221 INFO:tasks.rados.rados.0.smithi085.stdout:2730: writing smithi08575397-34 from 3211126 to 3793325 tid 3 2021-02-10T04:38:42.221 INFO:tasks.rados.rados.0.smithi085.stdout:2731: write (excl) oid 14 current snap is 313 2021-02-10T04:38:42.222 INFO:tasks.rados.rados.0.smithi085.stdout:2727: finishing write tid 3 to smithi08575397-46 2021-02-10T04:38:42.222 INFO:tasks.rados.rados.0.smithi085.stdout:2727: finishing write tid 4 to smithi08575397-46 2021-02-10T04:38:42.222 INFO:tasks.rados.rados.0.smithi085.stdout:2727: finishing write tid 5 to smithi08575397-46 2021-02-10T04:38:42.222 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 1350 (ObjNum 947 snap 313 seq_num 947) dirty exists 2021-02-10T04:38:42.222 INFO:tasks.rados.rados.0.smithi085.stdout:2727: left oid 46 (ObjNum 947 snap 313 seq_num 947) 2021-02-10T04:38:42.222 INFO:tasks.rados.rados.0.smithi085.stdout:2730: finishing write tid 1 to smithi08575397-34 2021-02-10T04:38:42.222 INFO:tasks.rados.rados.0.smithi085.stdout:2731: seq_num 950 ranges {766650=714322,2075423=688655,3522196=444454} 2021-02-10T04:38:42.230 INFO:tasks.rados.rados.0.smithi085.stdout:2731: writing smithi08575397-14 from 766650 to 1480972 tid 1 2021-02-10T04:38:42.238 INFO:tasks.rados.rados.0.smithi085.stdout:2731: writing smithi08575397-14 from 2075423 to 2764078 tid 2 2021-02-10T04:38:42.246 INFO:tasks.rados.rados.0.smithi085.stdout:2731: writing smithi08575397-14 from 3522196 to 3966650 tid 3 2021-02-10T04:38:42.246 INFO:tasks.rados.rados.0.smithi085.stdout:2727: done (7 left) 2021-02-10T04:38:42.246 INFO:tasks.rados.rados.0.smithi085.stdout:2729: done (6 left) 2021-02-10T04:38:42.246 INFO:tasks.rados.rados.0.smithi085.stdout:2732: read oid 20 snap 293 2021-02-10T04:38:42.246 INFO:tasks.rados.rados.0.smithi085.stdout:2732: expect (ObjNum 866 snap 278 seq_num 866) 2021-02-10T04:38:42.246 INFO:tasks.rados.rados.0.smithi085.stdout:2733: rollback oid 25 current snap is 313 2021-02-10T04:38:42.246 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 25 to 305 2021-02-10T04:38:42.246 INFO:tasks.rados.rados.0.smithi085.stdout:2730: finishing write tid 2 to smithi08575397-34 2021-02-10T04:38:42.247 INFO:tasks.rados.rados.0.smithi085.stdout:2730: finishing write tid 3 to smithi08575397-34 2021-02-10T04:38:42.247 INFO:tasks.rados.rados.0.smithi085.stdout:2730: finishing write tid 4 to smithi08575397-34 2021-02-10T04:38:42.247 INFO:tasks.rados.rados.0.smithi085.stdout:2730: finishing write tid 5 to smithi08575397-34 2021-02-10T04:38:42.247 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 34 v 790 (ObjNum 949 snap 313 seq_num 949) dirty exists 2021-02-10T04:38:42.247 INFO:tasks.rados.rados.0.smithi085.stdout:2730: left oid 34 (ObjNum 949 snap 313 seq_num 949) 2021-02-10T04:38:42.247 INFO:tasks.rados.rados.0.smithi085.stdout:2731: finishing write tid 1 to smithi08575397-14 2021-02-10T04:38:42.247 INFO:tasks.rados.rados.0.smithi085.stdout:2731: finishing write tid 2 to smithi08575397-14 2021-02-10T04:38:42.247 INFO:tasks.rados.rados.0.smithi085.stdout:2730: done (7 left) 2021-02-10T04:38:42.247 INFO:tasks.rados.rados.0.smithi085.stdout:2734: rollback oid 49 current snap is 313 2021-02-10T04:38:42.247 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 49 to 303 2021-02-10T04:38:42.247 INFO:tasks.rados.rados.0.smithi085.stdout:2735: write (excl) oid 2 current snap is 313 2021-02-10T04:38:42.247 INFO:tasks.rados.rados.0.smithi085.stdout:2735: seq_num 951 ranges {783457=505924,1850258=548666,2865972=721096} 2021-02-10T04:38:42.253 INFO:tasks.rados.rados.0.smithi085.stdout:2735: writing smithi08575397-2 from 783457 to 1289381 tid 1 2021-02-10T04:38:42.260 INFO:tasks.rados.rados.0.smithi085.stdout:2735: writing smithi08575397-2 from 1850258 to 2398924 tid 2 2021-02-10T04:38:42.267 INFO:tasks.rados.rados.0.smithi085.stdout:2735: writing smithi08575397-2 from 2865972 to 3587068 tid 3 2021-02-10T04:38:42.268 INFO:tasks.rados.rados.0.smithi085.stdout:2736: write (excl) oid 18 current snap is 313 2021-02-10T04:38:42.268 INFO:tasks.rados.rados.0.smithi085.stdout:2736: seq_num 952 ranges {400264=581173,1508740=491524} 2021-02-10T04:38:42.274 INFO:tasks.rados.rados.0.smithi085.stdout:2736: writing smithi08575397-18 from 400264 to 981437 tid 1 2021-02-10T04:38:42.280 INFO:tasks.rados.rados.0.smithi085.stdout:2736: writing smithi08575397-18 from 1508740 to 2000264 tid 2 2021-02-10T04:38:42.280 INFO:tasks.rados.rados.0.smithi085.stdout:2737: delete oid 36 current snap is 313 2021-02-10T04:38:42.280 INFO:tasks.rados.rados.0.smithi085.stdout:2731: finishing write tid 3 to smithi08575397-14 2021-02-10T04:38:42.280 INFO:tasks.rados.rados.0.smithi085.stdout:2732: expect (ObjNum 866 snap 278 seq_num 866) 2021-02-10T04:38:42.515 INFO:tasks.rados.rados.0.smithi085.stdout:2731: finishing write tid 4 to smithi08575397-14 2021-02-10T04:38:42.515 INFO:tasks.rados.rados.0.smithi085.stdout:2733: finishing rollback tid 0 to smithi08575397-25 2021-02-10T04:38:42.515 INFO:tasks.rados.rados.0.smithi085.stdout:2731: finishing write tid 5 to smithi08575397-14 2021-02-10T04:38:42.515 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 1336 (ObjNum 950 snap 313 seq_num 950) dirty exists 2021-02-10T04:38:42.515 INFO:tasks.rados.rados.0.smithi085.stdout:2731: left oid 14 (ObjNum 950 snap 313 seq_num 950) 2021-02-10T04:38:42.516 INFO:tasks.rados.rados.0.smithi085.stdout:2733: finishing rollback tid 1 to smithi08575397-25 2021-02-10T04:38:42.516 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 25 v 1280 (ObjNum 83213808 snap 22030 seq_num 82788224) dirty dne 2021-02-10T04:38:42.516 INFO:tasks.rados.rados.0.smithi085.stdout:2734: finishing rollback tid 0 to smithi08575397-49 2021-02-10T04:38:42.516 INFO:tasks.rados.rados.0.smithi085.stdout:2731: done (10 left) 2021-02-10T04:38:42.516 INFO:tasks.rados.rados.0.smithi085.stdout:2732: done (9 left) 2021-02-10T04:38:42.516 INFO:tasks.rados.rados.0.smithi085.stdout:2733: done (8 left) 2021-02-10T04:38:42.516 INFO:tasks.rados.rados.0.smithi085.stdout:2737: done (7 left) 2021-02-10T04:38:42.516 INFO:tasks.rados.rados.0.smithi085.stdout:2738: read oid 44 snap -1 2021-02-10T04:38:42.516 INFO:tasks.rados.rados.0.smithi085.stdout:2738: expect (ObjNum 266 snap 64 seq_num 266) 2021-02-10T04:38:42.516 INFO:tasks.rados.rados.0.smithi085.stdout:2739: delete oid 45 current snap is 313 2021-02-10T04:38:42.516 INFO:tasks.rados.rados.0.smithi085.stdout:2734: finishing rollback tid 1 to smithi08575397-49 2021-02-10T04:38:42.516 INFO:tasks.rados.rados.0.smithi085.stdout:2734: finishing rollback tid 2 to smithi08575397-49 2021-02-10T04:38:42.516 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 49 v 886 (ObjNum 172 snap 37 seq_num 172) dirty exists 2021-02-10T04:38:42.517 INFO:tasks.rados.rados.0.smithi085.stdout:2735: finishing write tid 1 to smithi08575397-2 2021-02-10T04:38:42.517 INFO:tasks.rados.rados.0.smithi085.stdout:2735: finishing write tid 2 to smithi08575397-2 2021-02-10T04:38:42.517 INFO:tasks.rados.rados.0.smithi085.stdout:2735: finishing write tid 3 to smithi08575397-2 2021-02-10T04:38:42.517 INFO:tasks.rados.rados.0.smithi085.stdout:2735: finishing write tid 4 to smithi08575397-2 2021-02-10T04:38:42.517 INFO:tasks.rados.rados.0.smithi085.stdout:2735: finishing write tid 5 to smithi08575397-2 2021-02-10T04:38:42.517 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 890 (ObjNum 951 snap 313 seq_num 951) dirty exists 2021-02-10T04:38:42.517 INFO:tasks.rados.rados.0.smithi085.stdout:2735: left oid 2 (ObjNum 951 snap 313 seq_num 951) 2021-02-10T04:38:42.517 INFO:tasks.rados.rados.0.smithi085.stdout:2736: finishing write tid 1 to smithi08575397-18 2021-02-10T04:38:42.517 INFO:tasks.rados.rados.0.smithi085.stdout:2736: finishing write tid 2 to smithi08575397-18 2021-02-10T04:38:42.517 INFO:tasks.rados.rados.0.smithi085.stdout:2736: finishing write tid 3 to smithi08575397-18 2021-02-10T04:38:42.517 INFO:tasks.rados.rados.0.smithi085.stdout:2736: finishing write tid 4 to smithi08575397-18 2021-02-10T04:38:42.517 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 18 v 1283 (ObjNum 952 snap 313 seq_num 952) dirty exists 2021-02-10T04:38:42.517 INFO:tasks.rados.rados.0.smithi085.stdout:2736: left oid 18 (ObjNum 952 snap 313 seq_num 952) 2021-02-10T04:38:42.518 INFO:tasks.rados.rados.0.smithi085.stdout:2738: expect (ObjNum 266 snap 64 seq_num 266) 2021-02-10T04:38:42.670 INFO:tasks.rados.rados.0.smithi085.stdout:2734: done (8 left) 2021-02-10T04:38:42.670 INFO:tasks.rados.rados.0.smithi085.stdout:2735: done (7 left) 2021-02-10T04:38:42.670 INFO:tasks.rados.rados.0.smithi085.stdout:2736: done (6 left) 2021-02-10T04:38:42.670 INFO:tasks.rados.rados.0.smithi085.stdout:2738: done (5 left) 2021-02-10T04:38:42.670 INFO:tasks.rados.rados.0.smithi085.stdout:2739: done (4 left) 2021-02-10T04:38:42.671 INFO:tasks.rados.rados.0.smithi085.stdout:2740: copy_from oid 9 from oid 32 current snap is 313 2021-02-10T04:38:42.671 INFO:tasks.rados.rados.0.smithi085.stdout:2741: snap_create 2021-02-10T04:38:42.692 INFO:tasks.rados.rados.0.smithi085.stdout:2740: finishing copy_from to smithi08575397-9 2021-02-10T04:38:42.692 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 9 v 667 (ObjNum 943 snap 313 seq_num 943) dirty exists 2021-02-10T04:38:42.692 INFO:tasks.rados.rados.0.smithi085.stdout:2740: finishing copy_from racing read to smithi08575397-9 2021-02-10T04:38:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:42 smithi164 conmon[29288]: cluster 2021-02-10T04:38:42.162809+0000 mgr.y (mgr.14140) 720 : cluster [DBG] pgmap v1205: 168 pgs: 1 active+clean+scrubbing, 1 peering, 6 active+clean+snaptrim_wait, 152 active+clean, 8 active+clean+snaptrim; 415 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 8.9 MiB/s rd, 2.6 MiB/s wr, 15 op/s; 223 B/s, 2 objects/s recovering 2021-02-10T04:38:42.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:42 smithi085 conmon[29319]: cluster 2021-02-10T04:38:42.162809+0000 mgr.y (mgr.14140) 720 : cluster 2021-02-10T04:38:42.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:42 smithi085 conmon[29319]: [DBG] pgmap v1205: 168 pgs: 1 active+clean+scrubbing, 1 peering, 6 active+clean+snaptrim_wait, 152 active+clean, 8 active+clean+snaptrim; 415 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 8.9 MiB/s rd, 2.6 MiB/s wr, 15 op/s; 223 B/s, 2 objects/s recovering 2021-02-10T04:38:42.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:42 smithi085 conmon[35961]: cluster 2021-02-10T04:38:42.162809+0000 2021-02-10T04:38:42.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:42 smithi085 conmon[35961]: mgr.y (mgr.14140) 720 : cluster [DBG] pgmap v1205: 168 pgs: 1 active+clean+scrubbing, 1 peering, 6 active+clean+snaptrim_wait, 152 active+clean, 8 active+clean+snaptrim; 415 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 8.9 MiB/s rd, 2.6 MiB/s wr, 15 op/s; 223 B/s, 2 objects/s recovering 2021-02-10T04:38:43.230 INFO:tasks.rados.rados.0.smithi085.stdout:2740: done (5 left) 2021-02-10T04:38:43.230 INFO:tasks.rados.rados.0.smithi085.stdout:2741: done (4 left) 2021-02-10T04:38:43.230 INFO:tasks.rados.rados.0.smithi085.stdout:2742: read oid 42 snap -1 2021-02-10T04:38:43.230 INFO:tasks.rados.rados.0.smithi085.stdout:2742: expect (ObjNum 833 snap 270 seq_num 833) 2021-02-10T04:38:43.230 INFO:tasks.rados.rados.0.smithi085.stdout:2743: snap_remove snap 284 2021-02-10T04:38:43.299 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:38:43.397 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:38:43.441 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:38:43 smithi085 conmon[41540]: debug 2021-02-10T04:38:43.189+0000 7fb94bb63700 -1 osd.0 pg_epoch: 703 pg[5.5( empty local-lis/les=57/58 n=0 ec=57/57 lis/c=57/57 les/c/f=58/58/0 sis=57) [0,1,4] r=0 lpr=57 crt=0'0 mlcod 0'0 active+clean] scrubber pg(5.5) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x555d32b594a0) 2021-02-10T04:38:43.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:43 smithi164 conmon[33677]: debug 2021-02-10T04:38:43.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:43 smithi164 conmon[33677]: 2021-02-10T04:38:43.484+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:38:43.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:43 smithi164 conmon[33677]: 2021-02-10T04:38:43.949 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:38:44.055 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:38:44.079 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:38:44.232 INFO:tasks.rados.rados.0.smithi085.stdout:2743: done (5 left) 2021-02-10T04:38:44.232 INFO:tasks.rados.rados.0.smithi085.stdout:2744: delete oid 31 current snap is 314 2021-02-10T04:38:44.232 INFO:tasks.rados.rados.0.smithi085.stdout:2742: expect (ObjNum 833 snap 270 seq_num 833) 2021-02-10T04:38:44.234 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:38:44 smithi164 conmon[44544]: debug 2021-02-10T04:38:44.141+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:38:44.406 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:38:44.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:44 smithi085 conmon[29319]: cluster 2021-02-10T04:38:43.228298 2021-02-10T04:38:44.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:44 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1611 : cluster [DBG] osdmap e704: 8 total, 8 up, 7 in 2021-02-10T04:38:44.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:44 smithi085 conmon[29319]: cluster 2021-02-10T04:38:44.163298+0000 2021-02-10T04:38:44.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:44 smithi085 conmon[29319]: mgr.y (mgr.14140) 721 : cluster [DBG] pgmap v1207: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 429 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.2 MiB/s wr, 15 op/s; 181 B/s, 2 objects/s recovering 2021-02-10T04:38:44.441 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:38:44 smithi085 conmon[41540]: debug 2021-02-10T04:38:44.156+0000 7fb947b5b700 -1 osd.0 pg_epoch: 704 pg[5.5( empty local-lis/les=57/58 n=0 ec=57/57 lis/c=57/57 les/c/f=58/58/0 sis=57) [0,1,4] r=0 lpr=57 crt=0'0 mlcod 0'0 active+clean] scrubber pg(5.5) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x555d32bb3600) 2021-02-10T04:38:44.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:44 smithi085 conmon[35961]: cluster 2021-02-10T04:38:43.228298+0000 2021-02-10T04:38:44.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:44 smithi085 conmon[35961]: mon.a (mon.0) 1611 : cluster [DBG] osdmap e704: 8 total, 8 up, 7 in 2021-02-10T04:38:44.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:44 smithi085 conmon[35961]: cluster 2021-02-10T04:38:44.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:44 smithi085 conmon[35961]: 2021-02-10T04:38:44.163298+0000 2021-02-10T04:38:44.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:44 smithi085 conmon[35961]: mgr.y (mgr.14140) 721 : 2021-02-10T04:38:44.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:44 smithi085 conmon[35961]: cluster [DBG] pgmap v1207: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 429 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.2 MiB/s wr, 15 op/s; 181 B/s, 2 objects/s recovering 2021-02-10T04:38:44.462 INFO:tasks.rados.rados.0.smithi085.stdout:2742: done (5 left) 2021-02-10T04:38:44.462 INFO:tasks.rados.rados.0.smithi085.stdout:2744: done (4 left) 2021-02-10T04:38:44.462 INFO:tasks.rados.rados.0.smithi085.stdout:2745: copy_from oid 15 from oid 6 current snap is 314 2021-02-10T04:38:44.463 INFO:tasks.rados.rados.0.smithi085.stdout:2746: rollback oid 1 current snap is 314 2021-02-10T04:38:44.463 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 1 to 237 2021-02-10T04:38:44.463 INFO:tasks.rados.rados.0.smithi085.stdout:2747: rollback oid 42 current snap is 314 2021-02-10T04:38:44.463 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 42 to 290 2021-02-10T04:38:44.463 INFO:tasks.rados.rados.0.smithi085.stdout:2748: snap_remove snap 289 2021-02-10T04:38:44.511 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:38:44.542 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:44 smithi164 conmon[29288]: cluster 2021-02-10T04:38:44.542 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:44 smithi164 conmon[29288]: 2021-02-10T04:38:44.542 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:44 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:38:44.542 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:44 smithi164 conmon[29288]: 38:43.228298+0000 mon.a (mon.0) 1611 : cluster [DBG] osdmap e704: 8 total, 8 up, 7 in 2021-02-10T04:38:44.542 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:44 smithi164 conmon[29288]: cluster 2021-02-10T04:38:44.542 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:44 smithi164 conmon[29288]: 2021-02-10T04:38:44.542 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:44 smithi164 conmon[29288]: 2021 2021-02-10T04:38:44.542 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:44 smithi164 conmon[29288]: -02-10T04 2021-02-10T04:38:44.542 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:44 smithi164 conmon[29288]: :38:44.163298+0000 mgr.y (mgr.14140) 721 : cluster [DBG] 2021-02-10T04:38:44.543 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:44 smithi164 conmon[29288]: pgmap v1207: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 429 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.2 MiB/s wr, 15 op/s; 181 B/s, 2 objects/s recovering 2021-02-10T04:38:44.545 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:38:44.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:38:44 smithi164 conmon[33677]: debug 2021-02-10T04:38:44.668+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:38:44.911 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:38:44.994 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":1197,"stamp":"2021-02-10T04:38:36.570564+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":412051212,"num_objects":549,"num_object_clones":143,"num_object_copies":1647,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":549,"num_whiteouts":14,"num_read":6943,"num_read_kb":5059050,"num_write":4817,"num_write_kb":1419490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":411,"num_bytes_recovered":108948019,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10690,"ondisk_log_size":10690,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":1122084,"kb_used_data":891628,"kb_used_omap":0,"kb_used_meta":230400,"kb_avail":655093980,"statfs":{"total":671965249536,"available":670816235520,"internally_reserved":0,"allocated":913027072,"data_stored":1200407526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":235929600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4],"upper_bound":2097152},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22659574,"num_objects":10,"num_object_clones":10,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":23,"num_read_kb":10964,"num_write":41,"num_write_kb":14133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.333425"},"pg_stats":[{"pgid":"5.9","version":"131'3","reported_seq":"704","reported_epoch":"694","state":"active+clean","last_fresh":"2021-02-10T04:38:32.843257+0000","last_change":"2021-02-10T04:33:57.871495+0000","last_active":"2021-02-10T04:38:32.843257+0000","last_peered":"2021-02-10T04:38:32.843257+0000","last_clean":"2021-02-10T04:38:32.843257+0000","last_became_active":"2021-02-10T04:29:07.145525+0000","last_became_peered":"2021-02-10T04:29:07.145525+0000","last_unstale":"2021-02-10T04:38:32.843257+0000","last_undegraded":"2021-02-10T04:38:32.843257+0000","last_fullsized":"2021-02-10T04:38:32.843257+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'3","last_scrub_stamp":"2021-02-10T04:33:57.871402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:33:57.871402+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"538'42","reported_seq":"741","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.243625+0000","last_change":"2021-02-10T04:33:55.585872+0000","last_active":"2021-02-10T04:38:31.243625+0000","last_peered":"2021-02-10T04:38:31.243625+0000","last_clean":"2021-02-10T04:38:31.243625+0000","last_became_active":"2021-02-10T04:25:49.002058+0000","last_became_peered":"2021-02-10T04:25:49.002058+0000","last_unstale":"2021-02-10T04:38:31.243625+0000","last_undegraded":"2021-02-10T04:38:31.243625+0000","last_fullsized":"2021-02-10T04:38:31.243625+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2021-02-10T04:33:55.585807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:55.585807+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":77,"num_read_kb":60,"num_write":43,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"96","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.231895+0000","last_change":"2021-02-10T04:37:37.973234+0000","last_active":"2021-02-10T04:38:33.231895+0000","last_peered":"2021-02-10T04:38:33.231895+0000","last_clean":"2021-02-10T04:38:33.231895+0000","last_became_active":"2021-02-10T04:37:37.246903+0000","last_became_peered":"2021-02-10T04:37:37.246903+0000","last_unstale":"2021-02-10T04:38:33.231895+0000","last_undegraded":"2021-02-10T04:38:33.231895+0000","last_fullsized":"2021-02-10T04:38:33.231895+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:29:34.787265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:34.787265+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"653","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.244160+0000","last_change":"2021-02-10T04:29:19.405593+0000","last_active":"2021-02-10T04:38:31.244160+0000","last_peered":"2021-02-10T04:38:31.244160+0000","last_clean":"2021-02-10T04:38:31.244160+0000","last_became_active":"2021-02-10T04:25:53.148090+0000","last_became_peered":"2021-02-10T04:25:53.148090+0000","last_unstale":"2021-02-10T04:38:31.244160+0000","last_undegraded":"2021-02-10T04:38:31.244160+0000","last_fullsized":"2021-02-10T04:38:31.244160+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:19.405525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:29:19.405525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"694'979","reported_seq":"2448","reported_epoch":"694","state":"active+clean","last_fresh":"2021-02-10T04:38:32.271578+0000","last_change":"2021-02-10T04:38:32.241021+0000","last_active":"2021-02-10T04:38:32.271578+0000","last_peered":"2021-02-10T04:38:32.271578+0000","last_clean":"2021-02-10T04:38:32.271578+0000","last_became_active":"2021-02-10T04:33:05.144488+0000","last_became_peered":"2021-02-10T04:33:05.144488+0000","last_unstale":"2021-02-10T04:38:32.271578+0000","last_undegraded":"2021-02-10T04:38:32.271578+0000","last_fullsized":"2021-02-10T04:38:32.271578+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'301","last_scrub_stamp":"2021-02-10T04:33:56.571925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:56.571925+0000","log_size":979,"ondisk_log_size":979,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":42131415,"num_objects":19,"num_object_clones":14,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":432,"num_read_kb":440883,"num_write":325,"num_write_kb":115408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1eb","length":"1"},{"start":"1ee","length":"3"},{"start":"1f2","length":"3"},{"start":"1f7","length":"2"},{"start":"1fc","length":"1"},{"start":"201","length":"1"},{"start":"205","length":"2"},{"start":"20d","length":"1"},{"start":"215","length":"1"},{"start":"21b","length":"1"},{"start":"21d","length":"1"},{"start":"21f","length":"1"},{"start":"222","length":"2"},{"start":"225","length":"1"},{"start":"233","length":"1"},{"start":"238","length":"1"},{"start":"23b","length":"1"},{"start":"244","length":"1"},{"start":"246","length":"1"},{"start":"24c","length":"1"}]},{"pgid":"4.b","version":"538'26","reported_seq":"714","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.227308+0000","last_change":"2021-02-10T04:38:32.145595+0000","last_active":"2021-02-10T04:38:33.227308+0000","last_peered":"2021-02-10T04:38:33.227308+0000","last_clean":"2021-02-10T04:38:33.227308+0000","last_became_active":"2021-02-10T04:25:49.004346+0000","last_became_peered":"2021-02-10T04:25:49.004346+0000","last_unstale":"2021-02-10T04:38:33.227308+0000","last_undegraded":"2021-02-10T04:38:33.227308+0000","last_fullsized":"2021-02-10T04:38:33.227308+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'26","last_scrub_stamp":"2021-02-10T04:38:32.145511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:32.145511+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":53,"num_read_kb":36,"num_write":27,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"337","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.237522+0000","last_change":"2021-02-10T04:33:05.151251+0000","last_active":"2021-02-10T04:38:30.237522+0000","last_peered":"2021-02-10T04:38:30.237522+0000","last_clean":"2021-02-10T04:38:30.237522+0000","last_became_active":"2021-02-10T04:33:05.150882+0000","last_became_peered":"2021-02-10T04:33:05.150882+0000","last_unstale":"2021-02-10T04:38:30.237522+0000","last_undegraded":"2021-02-10T04:38:30.237522+0000","last_fullsized":"2021-02-10T04:38:30.237522+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:31.102558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:31.102558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"66","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.237777+0000","last_change":"2021-02-10T04:37:37.244470+0000","last_active":"2021-02-10T04:38:30.237777+0000","last_peered":"2021-02-10T04:38:30.237777+0000","last_clean":"2021-02-10T04:38:30.237777+0000","last_became_active":"2021-02-10T04:37:37.242485+0000","last_became_peered":"2021-02-10T04:37:37.242485+0000","last_unstale":"2021-02-10T04:38:30.237777+0000","last_undegraded":"2021-02-10T04:38:30.237777+0000","last_fullsized":"2021-02-10T04:38:30.237777+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:35.805326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.805326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"666","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.226742+0000","last_change":"2021-02-10T04:37:37.254421+0000","last_active":"2021-02-10T04:38:33.226742+0000","last_peered":"2021-02-10T04:38:33.226742+0000","last_clean":"2021-02-10T04:38:33.226742+0000","last_became_active":"2021-02-10T04:37:37.254038+0000","last_became_peered":"2021-02-10T04:37:37.254038+0000","last_unstale":"2021-02-10T04:38:33.226742+0000","last_undegraded":"2021-02-10T04:38:33.226742+0000","last_fullsized":"2021-02-10T04:38:33.226742+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:48.700418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:48.700418+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"259'74","reported_seq":"994","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.237923+0000","last_change":"2021-02-10T04:38:30.237923+0000","last_active":"2021-02-10T04:38:30.237923+0000","last_peered":"2021-02-10T04:38:30.237923+0000","last_clean":"2021-02-10T04:38:30.237923+0000","last_became_active":"2021-02-10T04:37:37.246533+0000","last_became_peered":"2021-02-10T04:37:37.246533+0000","last_unstale":"2021-02-10T04:38:30.237923+0000","last_undegraded":"2021-02-10T04:38:30.237923+0000","last_fullsized":"2021-02-10T04:38:30.237923+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'62","last_scrub_stamp":"2021-02-10T04:30:10.905905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:10.905905+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2437048,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":48,"num_read_kb":56455,"num_write":38,"num_write_kb":14043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1134570,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1eb","length":"1"},{"start":"1ee","length":"3"},{"start":"1f2","length":"3"},{"start":"1f7","length":"2"},{"start":"1fc","length":"1"},{"start":"201","length":"1"},{"start":"205","length":"2"},{"start":"20d","length":"1"},{"start":"215","length":"1"},{"start":"21b","length":"1"},{"start":"21d","length":"1"},{"start":"21f","length":"1"},{"start":"222","length":"2"},{"start":"225","length":"1"},{"start":"233","length":"1"},{"start":"238","length":"1"},{"start":"23b","length":"1"},{"start":"244","length":"1"},{"start":"246","length":"1"}]},{"pgid":"4.a","version":"538'45","reported_seq":"738","reported_epoch":"694","state":"active+clean","last_fresh":"2021-02-10T04:38:32.243813+0000","last_change":"2021-02-10T04:29:26.671051+0000","last_active":"2021-02-10T04:38:32.243813+0000","last_peered":"2021-02-10T04:38:32.243813+0000","last_clean":"2021-02-10T04:38:32.243813+0000","last_became_active":"2021-02-10T04:25:49.015913+0000","last_became_peered":"2021-02-10T04:25:49.015913+0000","last_unstale":"2021-02-10T04:38:32.243813+0000","last_undegraded":"2021-02-10T04:38:32.243813+0000","last_fullsized":"2021-02-10T04:38:32.243813+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'29","last_scrub_stamp":"2021-02-10T04:29:26.670950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.670950+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":80,"num_read_kb":60,"num_write":46,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"642'3","reported_seq":"109","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.245063+0000","last_change":"2021-02-10T04:38:24.163263+0000","last_active":"2021-02-10T04:38:31.245063+0000","last_peered":"2021-02-10T04:38:31.245063+0000","last_clean":"2021-02-10T04:38:31.245063+0000","last_became_active":"2021-02-10T04:37:37.234230+0000","last_became_peered":"2021-02-10T04:37:37.234230+0000","last_unstale":"2021-02-10T04:38:31.245063+0000","last_undegraded":"2021-02-10T04:38:31.245063+0000","last_fullsized":"2021-02-10T04:38:31.245063+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'3","last_scrub_stamp":"2021-02-10T04:38:24.163200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:24.163200+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"74","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.227147+0000","last_change":"2021-02-10T04:38:31.121100+0000","last_active":"2021-02-10T04:38:33.227147+0000","last_peered":"2021-02-10T04:38:33.227147+0000","last_clean":"2021-02-10T04:38:33.227147+0000","last_became_active":"2021-02-10T04:37:37.230818+0000","last_became_peered":"2021-02-10T04:37:37.230818+0000","last_unstale":"2021-02-10T04:38:33.227147+0000","last_undegraded":"2021-02-10T04:38:33.227147+0000","last_fullsized":"2021-02-10T04:38:33.227147+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:31.121038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:38:31.121038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"72","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.235031+0000","last_change":"2021-02-10T04:37:37.245491+0000","last_active":"2021-02-10T04:38:33.235031+0000","last_peered":"2021-02-10T04:38:33.235031+0000","last_clean":"2021-02-10T04:38:33.235031+0000","last_became_active":"2021-02-10T04:37:37.245113+0000","last_became_peered":"2021-02-10T04:37:37.245113+0000","last_unstale":"2021-02-10T04:38:33.235031+0000","last_undegraded":"2021-02-10T04:38:33.235031+0000","last_fullsized":"2021-02-10T04:38:33.235031+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.731458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.731458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"685'437","reported_seq":"2468","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.226042+0000","last_change":"2021-02-10T04:38:32.245516+0000","last_active":"2021-02-10T04:38:33.226042+0000","last_peered":"2021-02-10T04:38:33.226042+0000","last_clean":"2021-02-10T04:38:33.226042+0000","last_became_active":"2021-02-10T04:28:25.150246+0000","last_became_peered":"2021-02-10T04:28:25.150246+0000","last_unstale":"2021-02-10T04:38:33.226042+0000","last_undegraded":"2021-02-10T04:38:33.226042+0000","last_fullsized":"2021-02-10T04:38:33.226042+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'63","last_scrub_stamp":"2021-02-10T04:29:39.700115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:29:39.700115+0000","log_size":437,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18339808,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":196,"num_read_kb":178611,"num_write":116,"num_write_kb":56986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"201","length":"1"},{"start":"205","length":"2"},{"start":"20d","length":"1"},{"start":"215","length":"1"},{"start":"21b","length":"1"},{"start":"21d","length":"1"},{"start":"21f","length":"1"},{"start":"222","length":"2"},{"start":"225","length":"1"},{"start":"233","length":"1"},{"start":"238","length":"1"},{"start":"23b","length":"1"},{"start":"244","length":"1"},{"start":"246","length":"1"},{"start":"24c","length":"1"}]},{"pgid":"4.d","version":"538'47","reported_seq":"760","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.237669+0000","last_change":"2021-02-10T04:37:37.821860+0000","last_active":"2021-02-10T04:38:30.237669+0000","last_peered":"2021-02-10T04:38:30.237669+0000","last_clean":"2021-02-10T04:38:30.237669+0000","last_became_active":"2021-02-10T04:37:37.237215+0000","last_became_peered":"2021-02-10T04:37:37.237215+0000","last_unstale":"2021-02-10T04:38:30.237669+0000","last_undegraded":"2021-02-10T04:38:30.237669+0000","last_fullsized":"2021-02-10T04:38:30.237669+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'33","last_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:28:49.514984+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":62,"num_read_kb":44,"num_write":36,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"574","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.236197+0000","last_change":"2021-02-10T04:29:07.146266+0000","last_active":"2021-02-10T04:38:30.236197+0000","last_peered":"2021-02-10T04:38:30.236197+0000","last_clean":"2021-02-10T04:38:30.236197+0000","last_became_active":"2021-02-10T04:29:07.145616+0000","last_became_peered":"2021-02-10T04:29:07.145616+0000","last_unstale":"2021-02-10T04:38:30.236197+0000","last_undegraded":"2021-02-10T04:38:30.236197+0000","last_fullsized":"2021-02-10T04:38:30.236197+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:44.410817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:44.410817+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"337","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.236624+0000","last_change":"2021-02-10T04:33:05.147166+0000","last_active":"2021-02-10T04:38:30.236624+0000","last_peered":"2021-02-10T04:38:30.236624+0000","last_clean":"2021-02-10T04:38:30.236624+0000","last_became_active":"2021-02-10T04:33:05.146783+0000","last_became_peered":"2021-02-10T04:33:05.146783+0000","last_unstale":"2021-02-10T04:38:30.236624+0000","last_undegraded":"2021-02-10T04:38:30.236624+0000","last_fullsized":"2021-02-10T04:38:30.236624+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:35.958255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.958255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"74","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.234007+0000","last_change":"2021-02-10T04:38:32.875574+0000","last_active":"2021-02-10T04:38:33.234007+0000","last_peered":"2021-02-10T04:38:33.234007+0000","last_clean":"2021-02-10T04:38:33.234007+0000","last_became_active":"2021-02-10T04:37:37.251479+0000","last_became_peered":"2021-02-10T04:37:37.251479+0000","last_unstale":"2021-02-10T04:38:33.234007+0000","last_undegraded":"2021-02-10T04:38:33.234007+0000","last_fullsized":"2021-02-10T04:38:33.234007+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:32.875498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:38:32.875498+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"694'200","reported_seq":"2742","reported_epoch":"694","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:38:32.346674+0000","last_change":"2021-02-10T04:38:32.346674+0000","last_active":"2021-02-10T04:38:32.346674+0000","last_peered":"2021-02-10T04:38:32.346674+0000","last_clean":"2021-02-10T04:38:32.346674+0000","last_became_active":"2021-02-10T04:37:40.267794+0000","last_became_peered":"2021-02-10T04:37:40.267794+0000","last_unstale":"2021-02-10T04:38:32.346674+0000","last_undegraded":"2021-02-10T04:38:32.346674+0000","last_fullsized":"2021-02-10T04:38:32.346674+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"688'187","last_scrub_stamp":"2021-02-10T04:38:27.225488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:38:27.225488+0000","log_size":200,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":17509946,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":95,"num_read_kb":89568,"num_write":104,"num_write_kb":33486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":11919937,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1eb","length":"1"},{"start":"1ee","length":"3"},{"start":"1f2","length":"3"},{"start":"1f7","length":"2"},{"start":"1fc","length":"1"},{"start":"201","length":"1"},{"start":"205","length":"2"},{"start":"20d","length":"1"},{"start":"215","length":"1"},{"start":"21b","length":"1"},{"start":"21d","length":"1"},{"start":"21f","length":"1"},{"start":"222","length":"2"},{"start":"225","length":"1"},{"start":"233","length":"1"},{"start":"238","length":"1"},{"start":"23b","length":"1"}]},{"pgid":"1.8","version":"0'0","reported_seq":"605","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.243540+0000","last_change":"2021-02-10T04:37:37.246395+0000","last_active":"2021-02-10T04:38:31.243540+0000","last_peered":"2021-02-10T04:38:31.243540+0000","last_clean":"2021-02-10T04:38:31.243540+0000","last_became_active":"2021-02-10T04:37:37.246004+0000","last_became_peered":"2021-02-10T04:37:37.246004+0000","last_unstale":"2021-02-10T04:38:31.243540+0000","last_undegraded":"2021-02-10T04:38:31.243540+0000","last_fullsized":"2021-02-10T04:38:31.243540+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:37.336092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:37.336092+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"681","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.231312+0000","last_change":"2021-02-10T04:37:37.259938+0000","last_active":"2021-02-10T04:38:33.231312+0000","last_peered":"2021-02-10T04:38:33.231312+0000","last_clean":"2021-02-10T04:38:33.231312+0000","last_became_active":"2021-02-10T04:37:37.259532+0000","last_became_peered":"2021-02-10T04:37:37.259532+0000","last_unstale":"2021-02-10T04:38:33.231312+0000","last_undegraded":"2021-02-10T04:38:33.231312+0000","last_fullsized":"2021-02-10T04:38:33.231312+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:51.858386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:33:51.858386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"538'23","reported_seq":"692","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.236727+0000","last_change":"2021-02-10T04:28:41.427119+0000","last_active":"2021-02-10T04:38:30.236727+0000","last_peered":"2021-02-10T04:38:30.236727+0000","last_clean":"2021-02-10T04:38:30.236727+0000","last_became_active":"2021-02-10T04:25:49.001725+0000","last_became_peered":"2021-02-10T04:25:49.001725+0000","last_unstale":"2021-02-10T04:38:30.236727+0000","last_undegraded":"2021-02-10T04:38:30.236727+0000","last_fullsized":"2021-02-10T04:38:30.236727+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'13","last_scrub_stamp":"2021-02-10T04:28:41.427050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:41.427050+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":28,"num_write":23,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"344","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.232936+0000","last_change":"2021-02-10T04:33:05.168734+0000","last_active":"2021-02-10T04:38:33.232936+0000","last_peered":"2021-02-10T04:38:33.232936+0000","last_clean":"2021-02-10T04:38:33.232936+0000","last_became_active":"2021-02-10T04:33:05.168314+0000","last_became_peered":"2021-02-10T04:33:05.168314+0000","last_unstale":"2021-02-10T04:38:33.232936+0000","last_undegraded":"2021-02-10T04:38:33.232936+0000","last_fullsized":"2021-02-10T04:38:33.232936+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:18.107867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:18.107867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"639'3","reported_seq":"97","reported_epoch":"694","state":"active+clean","last_fresh":"2021-02-10T04:38:32.242320+0000","last_change":"2021-02-10T04:38:31.211769+0000","last_active":"2021-02-10T04:38:32.242320+0000","last_peered":"2021-02-10T04:38:32.242320+0000","last_clean":"2021-02-10T04:38:32.242320+0000","last_became_active":"2021-02-10T04:37:37.233738+0000","last_became_peered":"2021-02-10T04:37:37.233738+0000","last_unstale":"2021-02-10T04:38:32.242320+0000","last_undegraded":"2021-02-10T04:38:32.242320+0000","last_fullsized":"2021-02-10T04:38:32.242320+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"639'3","last_scrub_stamp":"2021-02-10T04:38:31.211662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:38:31.211662+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"661","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.238800+0000","last_change":"2021-02-10T04:37:37.236009+0000","last_active":"2021-02-10T04:38:30.238800+0000","last_peered":"2021-02-10T04:38:30.238800+0000","last_clean":"2021-02-10T04:38:30.238800+0000","last_became_active":"2021-02-10T04:37:37.233109+0000","last_became_peered":"2021-02-10T04:37:37.233109+0000","last_unstale":"2021-02-10T04:38:30.238800+0000","last_undegraded":"2021-02-10T04:38:30.238800+0000","last_fullsized":"2021-02-10T04:38:30.238800+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:38.873102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.873102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"696'652","reported_seq":"4628","reported_epoch":"696","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:38:34.252904+0000","last_change":"2021-02-10T04:38:33.880281+0000","last_active":"2021-02-10T04:38:34.252904+0000","last_peered":"2021-02-10T04:38:34.252904+0000","last_clean":"2021-02-10T04:38:34.252904+0000","last_became_active":"2021-02-10T04:28:25.149226+0000","last_became_peered":"2021-02-10T04:28:25.149226+0000","last_unstale":"2021-02-10T04:38:34.252904+0000","last_undegraded":"2021-02-10T04:38:34.252904+0000","last_fullsized":"2021-02-10T04:38:34.252904+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'236","last_scrub_stamp":"2021-02-10T04:33:50.351534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:50.351534+0000","log_size":652,"ondisk_log_size":652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"stat_sum":{"num_bytes":30628452,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":1,"num_read":421,"num_read_kb":511192,"num_write":223,"num_write_kb":73298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"201","length":"1"},{"start":"205","length":"2"},{"start":"20d","length":"1"},{"start":"215","length":"1"}]},{"pgid":"1.a","version":"0'0","reported_seq":"605","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.244011+0000","last_change":"2021-02-10T04:37:37.232619+0000","last_active":"2021-02-10T04:38:31.244011+0000","last_peered":"2021-02-10T04:38:31.244011+0000","last_clean":"2021-02-10T04:38:31.244011+0000","last_became_active":"2021-02-10T04:37:37.229648+0000","last_became_peered":"2021-02-10T04:37:37.229648+0000","last_unstale":"2021-02-10T04:38:31.244011+0000","last_undegraded":"2021-02-10T04:38:31.244011+0000","last_fullsized":"2021-02-10T04:38:31.244011+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.384170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.384170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"538'29","reported_seq":"400","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.233190+0000","last_change":"2021-02-10T04:33:05.154998+0000","last_active":"2021-02-10T04:38:33.233190+0000","last_peered":"2021-02-10T04:38:33.233190+0000","last_clean":"2021-02-10T04:38:33.233190+0000","last_became_active":"2021-02-10T04:33:05.149026+0000","last_became_peered":"2021-02-10T04:33:05.149026+0000","last_unstale":"2021-02-10T04:38:33.233190+0000","last_undegraded":"2021-02-10T04:38:33.233190+0000","last_fullsized":"2021-02-10T04:38:33.233190+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'17","last_scrub_stamp":"2021-02-10T04:29:37.933206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:37.933206+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":49,"num_read_kb":36,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"404","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.231846+0000","last_change":"2021-02-10T04:33:05.169654+0000","last_active":"2021-02-10T04:38:33.231846+0000","last_peered":"2021-02-10T04:38:33.231846+0000","last_clean":"2021-02-10T04:38:33.231846+0000","last_became_active":"2021-02-10T04:33:05.169289+0000","last_became_peered":"2021-02-10T04:33:05.169289+0000","last_unstale":"2021-02-10T04:38:33.231846+0000","last_undegraded":"2021-02-10T04:38:33.231846+0000","last_fullsized":"2021-02-10T04:38:33.231846+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_clean_scrub_stamp":"2021-02-10T04:28:39.128781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"78'3","reported_seq":"809","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.238748+0000","last_change":"2021-02-10T04:28:43.509446+0000","last_active":"2021-02-10T04:38:30.238748+0000","last_peered":"2021-02-10T04:38:30.238748+0000","last_clean":"2021-02-10T04:38:30.238748+0000","last_became_active":"2021-02-10T04:25:51.025218+0000","last_became_peered":"2021-02-10T04:25:51.025218+0000","last_unstale":"2021-02-10T04:38:30.238748+0000","last_undegraded":"2021-02-10T04:38:30.238748+0000","last_fullsized":"2021-02-10T04:38:30.238748+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:28:43.509353+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:28:43.509353+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"653","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.244165+0000","last_change":"2021-02-10T04:28:45.481538+0000","last_active":"2021-02-10T04:38:31.244165+0000","last_peered":"2021-02-10T04:38:31.244165+0000","last_clean":"2021-02-10T04:38:31.244165+0000","last_became_active":"2021-02-10T04:25:53.042697+0000","last_became_peered":"2021-02-10T04:25:53.042697+0000","last_unstale":"2021-02-10T04:38:31.244165+0000","last_undegraded":"2021-02-10T04:38:31.244165+0000","last_fullsized":"2021-02-10T04:38:31.244165+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:45.481472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:28:45.481472+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"688'700","reported_seq":"3773","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.226096+0000","last_change":"2021-02-10T04:38:32.847429+0000","last_active":"2021-02-10T04:38:33.226096+0000","last_peered":"2021-02-10T04:38:33.226096+0000","last_clean":"2021-02-10T04:38:33.226096+0000","last_became_active":"2021-02-10T04:28:25.148500+0000","last_became_peered":"2021-02-10T04:28:25.148500+0000","last_unstale":"2021-02-10T04:38:33.226096+0000","last_undegraded":"2021-02-10T04:38:33.226096+0000","last_fullsized":"2021-02-10T04:38:33.226096+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'110","last_scrub_stamp":"2021-02-10T04:30:18.463782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:18.463782+0000","log_size":700,"ondisk_log_size":700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22400367,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":393,"num_read_kb":476261,"num_write":248,"num_write_kb":92055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"201","length":"1"},{"start":"205","length":"2"},{"start":"20d","length":"1"},{"start":"215","length":"1"},{"start":"21b","length":"1"},{"start":"21d","length":"1"},{"start":"21f","length":"1"},{"start":"222","length":"2"},{"start":"225","length":"1"},{"start":"233","length":"1"},{"start":"238","length":"1"},{"start":"23b","length":"1"},{"start":"244","length":"1"},{"start":"246","length":"1"},{"start":"24c","length":"1"}]},{"pgid":"4.e","version":"538'39","reported_seq":"720","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.237901+0000","last_change":"2021-02-10T04:28:11.693702+0000","last_active":"2021-02-10T04:38:30.237901+0000","last_peered":"2021-02-10T04:38:30.237901+0000","last_clean":"2021-02-10T04:38:30.237901+0000","last_became_active":"2021-02-10T04:25:49.011093+0000","last_became_peered":"2021-02-10T04:25:49.011093+0000","last_unstale":"2021-02-10T04:38:30.237901+0000","last_undegraded":"2021-02-10T04:38:30.237901+0000","last_fullsized":"2021-02-10T04:38:30.237901+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'31","last_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:28:11.693616+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":56,"num_read_kb":40,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"640","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.225860+0000","last_change":"2021-02-10T04:29:31.683314+0000","last_active":"2021-02-10T04:38:33.225860+0000","last_peered":"2021-02-10T04:38:33.225860+0000","last_clean":"2021-02-10T04:38:33.225860+0000","last_became_active":"2021-02-10T04:29:07.160593+0000","last_became_peered":"2021-02-10T04:29:07.160593+0000","last_unstale":"2021-02-10T04:38:33.225860+0000","last_undegraded":"2021-02-10T04:38:33.225860+0000","last_fullsized":"2021-02-10T04:38:33.225860+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:31.683213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.683213+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"660","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.244439+0000","last_change":"2021-02-10T04:30:10.487631+0000","last_active":"2021-02-10T04:38:31.244439+0000","last_peered":"2021-02-10T04:38:31.244439+0000","last_clean":"2021-02-10T04:38:31.244439+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:38:31.244439+0000","last_undegraded":"2021-02-10T04:38:31.244439+0000","last_fullsized":"2021-02-10T04:38:31.244439+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:10.487528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:30:10.487528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"654","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.232202+0000","last_change":"2021-02-10T04:33:55.849675+0000","last_active":"2021-02-10T04:38:33.232202+0000","last_peered":"2021-02-10T04:38:33.232202+0000","last_clean":"2021-02-10T04:38:33.232202+0000","last_became_active":"2021-02-10T04:25:53.148054+0000","last_became_peered":"2021-02-10T04:25:53.148054+0000","last_unstale":"2021-02-10T04:38:33.232202+0000","last_undegraded":"2021-02-10T04:38:33.232202+0000","last_fullsized":"2021-02-10T04:38:33.232202+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:55.849582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:33:55.849582+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"692'692","reported_seq":"2684","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.226020+0000","last_change":"2021-02-10T04:38:32.848736+0000","last_active":"2021-02-10T04:38:33.226020+0000","last_peered":"2021-02-10T04:38:33.226020+0000","last_clean":"2021-02-10T04:38:33.226020+0000","last_became_active":"2021-02-10T04:28:25.152030+0000","last_became_peered":"2021-02-10T04:28:25.152030+0000","last_unstale":"2021-02-10T04:38:33.226020+0000","last_undegraded":"2021-02-10T04:38:33.226020+0000","last_fullsized":"2021-02-10T04:38:33.226020+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'283","last_scrub_stamp":"2021-02-10T04:33:53.834627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:53.834627+0000","log_size":692,"ondisk_log_size":692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39184129,"num_objects":17,"num_object_clones":13,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":366,"num_read_kb":382730,"num_write":268,"num_write_kb":115055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"201","length":"1"},{"start":"205","length":"2"},{"start":"20d","length":"1"},{"start":"215","length":"1"},{"start":"21b","length":"1"},{"start":"21d","length":"1"},{"start":"21f","length":"1"},{"start":"222","length":"2"},{"start":"225","length":"1"},{"start":"233","length":"1"},{"start":"238","length":"1"},{"start":"23b","length":"1"},{"start":"244","length":"1"},{"start":"246","length":"1"},{"start":"24c","length":"1"}]},{"pgid":"5.0","version":"158'3","reported_seq":"812","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.234931+0000","last_change":"2021-02-10T04:29:34.388978+0000","last_active":"2021-02-10T04:38:33.234931+0000","last_peered":"2021-02-10T04:38:33.234931+0000","last_clean":"2021-02-10T04:38:33.234931+0000","last_became_active":"2021-02-10T04:25:51.031906+0000","last_became_peered":"2021-02-10T04:25:51.031906+0000","last_unstale":"2021-02-10T04:38:33.234931+0000","last_undegraded":"2021-02-10T04:38:33.234931+0000","last_fullsized":"2021-02-10T04:38:33.234931+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'3","last_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_deep_scrub":"158'3","last_deep_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_clean_scrub_stamp":"2021-02-10T04:29:34.388912+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"538'40","reported_seq":"725","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.236191+0000","last_change":"2021-02-10T04:29:36.813190+0000","last_active":"2021-02-10T04:38:30.236191+0000","last_peered":"2021-02-10T04:38:30.236191+0000","last_clean":"2021-02-10T04:38:30.236191+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:38:30.236191+0000","last_undegraded":"2021-02-10T04:38:30.236191+0000","last_fullsized":"2021-02-10T04:38:30.236191+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:29:36.813083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.813083+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":72,"num_read_kb":52,"num_write":40,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"706","reported_epoch":"694","state":"active+clean","last_fresh":"2021-02-10T04:38:32.245471+0000","last_change":"2021-02-10T04:37:37.245911+0000","last_active":"2021-02-10T04:38:32.245471+0000","last_peered":"2021-02-10T04:38:32.245471+0000","last_clean":"2021-02-10T04:38:32.245471+0000","last_became_active":"2021-02-10T04:37:37.245391+0000","last_became_peered":"2021-02-10T04:37:37.245391+0000","last_unstale":"2021-02-10T04:38:32.245471+0000","last_undegraded":"2021-02-10T04:38:32.245471+0000","last_fullsized":"2021-02-10T04:38:32.245471+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.632085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.632085+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"590","reported_epoch":"694","state":"active+clean","last_fresh":"2021-02-10T04:38:32.244391+0000","last_change":"2021-02-10T04:37:37.235415+0000","last_active":"2021-02-10T04:38:32.244391+0000","last_peered":"2021-02-10T04:38:32.244391+0000","last_clean":"2021-02-10T04:38:32.244391+0000","last_became_active":"2021-02-10T04:37:37.235082+0000","last_became_peered":"2021-02-10T04:37:37.235082+0000","last_unstale":"2021-02-10T04:38:32.244391+0000","last_undegraded":"2021-02-10T04:38:32.244391+0000","last_fullsized":"2021-02-10T04:38:32.244391+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.658374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.658374+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"692'640","reported_seq":"3230","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.480073+0000","last_change":"2021-02-10T04:38:33.480073+0000","last_active":"2021-02-10T04:38:33.480073+0000","last_peered":"2021-02-10T04:38:33.480073+0000","last_clean":"2021-02-10T04:38:33.480073+0000","last_became_active":"2021-02-10T04:37:45.352779+0000","last_became_peered":"2021-02-10T04:37:45.352779+0000","last_unstale":"2021-02-10T04:38:33.480073+0000","last_undegraded":"2021-02-10T04:38:33.480073+0000","last_fullsized":"2021-02-10T04:38:33.480073+0000","mapping_epoch":646,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":647,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":640,"ondisk_log_size":640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31994185,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":324,"num_read_kb":299969,"num_write":237,"num_write_kb":80495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":16580116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"201","length":"1"},{"start":"205","length":"2"},{"start":"20d","length":"1"},{"start":"215","length":"1"},{"start":"21b","length":"1"},{"start":"21d","length":"1"},{"start":"21f","length":"1"},{"start":"222","length":"2"},{"start":"225","length":"1"},{"start":"233","length":"1"},{"start":"238","length":"1"},{"start":"23b","length":"1"},{"start":"244","length":"1"},{"start":"246","length":"1"},{"start":"24c","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":"598","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.232274+0000","last_change":"2021-02-10T04:30:12.452072+0000","last_active":"2021-02-10T04:38:33.232274+0000","last_peered":"2021-02-10T04:38:33.232274+0000","last_clean":"2021-02-10T04:38:33.232274+0000","last_became_active":"2021-02-10T04:28:49.048525+0000","last_became_peered":"2021-02-10T04:28:49.048525+0000","last_unstale":"2021-02-10T04:38:33.232274+0000","last_undegraded":"2021-02-10T04:38:33.232274+0000","last_fullsized":"2021-02-10T04:38:33.232274+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:12.452000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:30:12.452000+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"654","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.236335+0000","last_change":"2021-02-10T04:29:40.859352+0000","last_active":"2021-02-10T04:38:30.236335+0000","last_peered":"2021-02-10T04:38:30.236335+0000","last_clean":"2021-02-10T04:38:30.236335+0000","last_became_active":"2021-02-10T04:25:51.018706+0000","last_became_peered":"2021-02-10T04:25:51.018706+0000","last_unstale":"2021-02-10T04:38:30.236335+0000","last_undegraded":"2021-02-10T04:38:30.236335+0000","last_fullsized":"2021-02-10T04:38:30.236335+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:40.859274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:40.859274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"538'60","reported_seq":"760","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.234784+0000","last_change":"2021-02-10T04:29:28.398522+0000","last_active":"2021-02-10T04:38:33.234784+0000","last_peered":"2021-02-10T04:38:33.234784+0000","last_clean":"2021-02-10T04:38:33.234784+0000","last_became_active":"2021-02-10T04:25:48.997481+0000","last_became_peered":"2021-02-10T04:25:48.997481+0000","last_unstale":"2021-02-10T04:38:33.234784+0000","last_undegraded":"2021-02-10T04:38:33.234784+0000","last_fullsized":"2021-02-10T04:38:33.234784+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'48","last_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_deep_scrub":"152'48","last_deep_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.398429+0000","log_size":60,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":88,"num_read_kb":66,"num_write":48,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"352","reported_epoch":"694","state":"active+clean","last_fresh":"2021-02-10T04:38:32.245012+0000","last_change":"2021-02-10T04:33:05.142730+0000","last_active":"2021-02-10T04:38:32.245012+0000","last_peered":"2021-02-10T04:38:32.245012+0000","last_clean":"2021-02-10T04:38:32.245012+0000","last_became_active":"2021-02-10T04:33:05.141066+0000","last_became_peered":"2021-02-10T04:33:05.141066+0000","last_unstale":"2021-02-10T04:38:32.245012+0000","last_undegraded":"2021-02-10T04:38:32.245012+0000","last_fullsized":"2021-02-10T04:38:32.245012+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:28:35.010709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:35.010709+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":423,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"661","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.237117+0000","last_change":"2021-02-10T04:37:37.245330+0000","last_active":"2021-02-10T04:38:30.237117+0000","last_peered":"2021-02-10T04:38:30.237117+0000","last_clean":"2021-02-10T04:38:30.237117+0000","last_became_active":"2021-02-10T04:37:37.244859+0000","last_became_peered":"2021-02-10T04:37:37.244859+0000","last_unstale":"2021-02-10T04:38:30.237117+0000","last_undegraded":"2021-02-10T04:38:30.237117+0000","last_fullsized":"2021-02-10T04:38:30.237117+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:05.641577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:05.641577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"696'1113","reported_seq":"3262","reported_epoch":"696","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:38:34.275992+0000","last_change":"2021-02-10T04:38:33.481456+0000","last_active":"2021-02-10T04:38:34.275992+0000","last_peered":"2021-02-10T04:38:34.275992+0000","last_clean":"2021-02-10T04:38:34.275992+0000","last_became_active":"2021-02-10T04:37:45.352022+0000","last_became_peered":"2021-02-10T04:37:45.352022+0000","last_unstale":"2021-02-10T04:38:34.275992+0000","last_undegraded":"2021-02-10T04:38:34.275992+0000","last_fullsized":"2021-02-10T04:38:34.275992+0000","mapping_epoch":646,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":647,"parent":"0.0","parent_split_bits":0,"last_scrub":"687'1095","last_scrub_stamp":"2021-02-10T04:38:27.895743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:38:27.895743+0000","log_size":1113,"ondisk_log_size":1113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"stat_sum":{"num_bytes":39624705,"num_objects":19,"num_object_clones":15,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":2,"num_read":621,"num_read_kb":642341,"num_write":470,"num_write_kb":165626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":35873876,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"201","length":"1"},{"start":"205","length":"2"},{"start":"20d","length":"1"},{"start":"215","length":"1"},{"start":"21b","length":"1"},{"start":"21d","length":"1"},{"start":"21f","length":"1"},{"start":"222","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":"604","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.236627+0000","last_change":"2021-02-10T04:37:37.233535+0000","last_active":"2021-02-10T04:38:30.236627+0000","last_peered":"2021-02-10T04:38:30.236627+0000","last_clean":"2021-02-10T04:38:30.236627+0000","last_became_active":"2021-02-10T04:37:37.233158+0000","last_became_peered":"2021-02-10T04:37:37.233158+0000","last_unstale":"2021-02-10T04:38:30.236627+0000","last_undegraded":"2021-02-10T04:38:30.236627+0000","last_fullsized":"2021-02-10T04:38:30.236627+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:33.839335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:33.839335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"727","reported_epoch":"694","state":"active+clean","last_fresh":"2021-02-10T04:38:32.243894+0000","last_change":"2021-02-10T04:33:52.827328+0000","last_active":"2021-02-10T04:38:32.243894+0000","last_peered":"2021-02-10T04:38:32.243894+0000","last_clean":"2021-02-10T04:38:32.243894+0000","last_became_active":"2021-02-10T04:25:51.025636+0000","last_became_peered":"2021-02-10T04:25:51.025636+0000","last_unstale":"2021-02-10T04:38:32.243894+0000","last_undegraded":"2021-02-10T04:38:32.243894+0000","last_fullsized":"2021-02-10T04:38:32.243894+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:52.827257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:33:52.827257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"538'37","reported_seq":"729","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.227069+0000","last_change":"2021-02-10T04:28:36.754064+0000","last_active":"2021-02-10T04:38:33.227069+0000","last_peered":"2021-02-10T04:38:33.227069+0000","last_clean":"2021-02-10T04:38:33.227069+0000","last_became_active":"2021-02-10T04:25:49.002937+0000","last_became_peered":"2021-02-10T04:25:49.002937+0000","last_unstale":"2021-02-10T04:38:33.227069+0000","last_undegraded":"2021-02-10T04:38:33.227069+0000","last_fullsized":"2021-02-10T04:38:33.227069+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'27","last_scrub_stamp":"2021-02-10T04:28:36.753994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:36.753994+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":72,"num_read_kb":53,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"593","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.229373+0000","last_change":"2021-02-10T04:29:35.711635+0000","last_active":"2021-02-10T04:38:33.229373+0000","last_peered":"2021-02-10T04:38:33.229373+0000","last_clean":"2021-02-10T04:38:33.229373+0000","last_became_active":"2021-02-10T04:29:07.161341+0000","last_became_peered":"2021-02-10T04:29:07.161341+0000","last_unstale":"2021-02-10T04:38:33.229373+0000","last_undegraded":"2021-02-10T04:38:33.229373+0000","last_fullsized":"2021-02-10T04:38:33.229373+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_deep_scrub":"53'2","last_deep_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.711557+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"348","reported_epoch":"694","state":"active+clean","last_fresh":"2021-02-10T04:38:32.242992+0000","last_change":"2021-02-10T04:37:37.235246+0000","last_active":"2021-02-10T04:38:32.242992+0000","last_peered":"2021-02-10T04:38:32.242992+0000","last_clean":"2021-02-10T04:38:32.242992+0000","last_became_active":"2021-02-10T04:37:37.234868+0000","last_became_peered":"2021-02-10T04:37:37.234868+0000","last_unstale":"2021-02-10T04:38:32.242992+0000","last_undegraded":"2021-02-10T04:38:32.242992+0000","last_fullsized":"2021-02-10T04:38:32.242992+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:26.978062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.978062+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"684'707","reported_seq":"2581","reported_epoch":"694","state":"active+clean","last_fresh":"2021-02-10T04:38:32.244039+0000","last_change":"2021-02-10T04:38:32.244246+0000","last_active":"2021-02-10T04:38:32.244039+0000","last_peered":"2021-02-10T04:38:32.244039+0000","last_clean":"2021-02-10T04:38:32.244039+0000","last_became_active":"2021-02-10T04:28:25.146766+0000","last_became_peered":"2021-02-10T04:28:25.146766+0000","last_unstale":"2021-02-10T04:38:32.244039+0000","last_undegraded":"2021-02-10T04:38:32.244039+0000","last_fullsized":"2021-02-10T04:38:32.244039+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'357","last_scrub_stamp":"2021-02-10T04:33:50.582002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:50.582002+0000","log_size":707,"ondisk_log_size":707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15749281,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":294,"num_read_kb":263357,"num_write":271,"num_write_kb":109724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1eb","length":"1"},{"start":"1ee","length":"3"},{"start":"1f2","length":"3"},{"start":"1f7","length":"2"},{"start":"1fc","length":"1"},{"start":"201","length":"1"},{"start":"205","length":"2"},{"start":"20d","length":"1"},{"start":"215","length":"1"},{"start":"21b","length":"1"},{"start":"21d","length":"1"},{"start":"21f","length":"1"},{"start":"222","length":"2"},{"start":"225","length":"1"},{"start":"233","length":"1"},{"start":"238","length":"1"},{"start":"23b","length":"1"},{"start":"244","length":"1"},{"start":"246","length":"1"},{"start":"24c","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":"678","reported_epoch":"694","state":"active+clean","last_fresh":"2021-02-10T04:38:32.245273+0000","last_change":"2021-02-10T04:29:38.793837+0000","last_active":"2021-02-10T04:38:32.245273+0000","last_peered":"2021-02-10T04:38:32.245273+0000","last_clean":"2021-02-10T04:38:32.245273+0000","last_became_active":"2021-02-10T04:28:49.054638+0000","last_became_peered":"2021-02-10T04:28:49.054638+0000","last_unstale":"2021-02-10T04:38:32.245273+0000","last_undegraded":"2021-02-10T04:38:32.245273+0000","last_fullsized":"2021-02-10T04:38:32.245273+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:38.793760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.793760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"655","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.244593+0000","last_change":"2021-02-10T04:29:40.451948+0000","last_active":"2021-02-10T04:38:31.244593+0000","last_peered":"2021-02-10T04:38:31.244593+0000","last_clean":"2021-02-10T04:38:31.244593+0000","last_became_active":"2021-02-10T04:25:51.032201+0000","last_became_peered":"2021-02-10T04:25:51.032201+0000","last_unstale":"2021-02-10T04:38:31.244593+0000","last_undegraded":"2021-02-10T04:38:31.244593+0000","last_fullsized":"2021-02-10T04:38:31.244593+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:40.451832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:29:40.451832+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"538'28","reported_seq":"739","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.228200+0000","last_change":"2021-02-10T04:37:38.566112+0000","last_active":"2021-02-10T04:38:33.228200+0000","last_peered":"2021-02-10T04:38:33.228200+0000","last_clean":"2021-02-10T04:38:33.228200+0000","last_became_active":"2021-02-10T04:37:37.247141+0000","last_became_peered":"2021-02-10T04:37:37.247141+0000","last_unstale":"2021-02-10T04:38:33.228200+0000","last_undegraded":"2021-02-10T04:38:33.228200+0000","last_fullsized":"2021-02-10T04:38:33.228200+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'16","last_scrub_stamp":"2021-02-10T04:33:55.288033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:55.288033+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":34,"num_write":28,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"396","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.226023+0000","last_change":"2021-02-10T04:33:05.166788+0000","last_active":"2021-02-10T04:38:33.226023+0000","last_peered":"2021-02-10T04:38:33.226023+0000","last_clean":"2021-02-10T04:38:33.226023+0000","last_became_active":"2021-02-10T04:33:05.166368+0000","last_became_peered":"2021-02-10T04:33:05.166368+0000","last_unstale":"2021-02-10T04:38:33.226023+0000","last_undegraded":"2021-02-10T04:38:33.226023+0000","last_fullsized":"2021-02-10T04:38:33.226023+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:14.204533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:30:14.204533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"659","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.226518+0000","last_change":"2021-02-10T04:29:21.620573+0000","last_active":"2021-02-10T04:38:33.226518+0000","last_peered":"2021-02-10T04:38:33.226518+0000","last_clean":"2021-02-10T04:38:33.226518+0000","last_became_active":"2021-02-10T04:25:49.001782+0000","last_became_peered":"2021-02-10T04:25:49.001782+0000","last_unstale":"2021-02-10T04:38:33.226518+0000","last_undegraded":"2021-02-10T04:38:33.226518+0000","last_fullsized":"2021-02-10T04:38:33.226518+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:21.620482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:21.620482+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"338","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.244895+0000","last_change":"2021-02-10T04:33:05.153205+0000","last_active":"2021-02-10T04:38:31.244895+0000","last_peered":"2021-02-10T04:38:31.244895+0000","last_clean":"2021-02-10T04:38:31.244895+0000","last_became_active":"2021-02-10T04:33:05.152845+0000","last_became_peered":"2021-02-10T04:33:05.152845+0000","last_unstale":"2021-02-10T04:38:31.244895+0000","last_undegraded":"2021-02-10T04:38:31.244895+0000","last_fullsized":"2021-02-10T04:38:31.244895+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.098681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.098681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.5","version":"696'144","reported_seq":"727","reported_epoch":"696","state":"active+clean","last_fresh":"2021-02-10T04:38:34.298256+0000","last_change":"2021-02-10T04:38:32.243212+0000","last_active":"2021-02-10T04:38:34.298256+0000","last_peered":"2021-02-10T04:38:34.298256+0000","last_clean":"2021-02-10T04:38:34.298256+0000","last_became_active":"2021-02-10T04:37:37.247692+0000","last_became_peered":"2021-02-10T04:37:37.247692+0000","last_unstale":"2021-02-10T04:38:34.298256+0000","last_undegraded":"2021-02-10T04:38:34.298256+0000","last_fullsized":"2021-02-10T04:38:34.298256+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"683'135","last_scrub_stamp":"2021-02-10T04:38:23.183462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:38:23.183462+0000","log_size":144,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15487273,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":60,"num_read_kb":55297,"num_write":65,"num_write_kb":23217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":807386,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"201","length":"1"},{"start":"205","length":"2"},{"start":"20d","length":"1"},{"start":"215","length":"1"},{"start":"21b","length":"1"},{"start":"21d","length":"1"},{"start":"21f","length":"1"},{"start":"222","length":"2"},{"start":"225","length":"1"},{"start":"233","length":"1"},{"start":"238","length":"1"},{"start":"23b","length":"1"},{"start":"244","length":"1"},{"start":"246","length":"1"},{"start":"24c","length":"1"}]},{"pgid":"1.3","version":"0'0","reported_seq":"418","reported_epoch":"694","state":"active+clean","last_fresh":"2021-02-10T04:38:32.243815+0000","last_change":"2021-02-10T04:33:05.141311+0000","last_active":"2021-02-10T04:38:32.243815+0000","last_peered":"2021-02-10T04:38:32.243815+0000","last_clean":"2021-02-10T04:38:32.243815+0000","last_became_active":"2021-02-10T04:33:05.140966+0000","last_became_peered":"2021-02-10T04:33:05.140966+0000","last_unstale":"2021-02-10T04:38:32.243815+0000","last_undegraded":"2021-02-10T04:38:32.243815+0000","last_fullsized":"2021-02-10T04:38:32.243815+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.928906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.928906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"70","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.245269+0000","last_change":"2021-02-10T04:37:37.235036+0000","last_active":"2021-02-10T04:38:31.245269+0000","last_peered":"2021-02-10T04:38:31.245269+0000","last_clean":"2021-02-10T04:38:31.245269+0000","last_became_active":"2021-02-10T04:37:37.234644+0000","last_became_peered":"2021-02-10T04:37:37.234644+0000","last_unstale":"2021-02-10T04:38:31.245269+0000","last_undegraded":"2021-02-10T04:38:31.245269+0000","last_fullsized":"2021-02-10T04:38:31.245269+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.805432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.805432+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"538'27","reported_seq":"705","reported_epoch":"694","state":"active+clean","last_fresh":"2021-02-10T04:38:32.244835+0000","last_change":"2021-02-10T04:33:56.837896+0000","last_active":"2021-02-10T04:38:32.244835+0000","last_peered":"2021-02-10T04:38:32.244835+0000","last_clean":"2021-02-10T04:38:32.244835+0000","last_became_active":"2021-02-10T04:25:49.002471+0000","last_became_peered":"2021-02-10T04:25:49.002471+0000","last_unstale":"2021-02-10T04:38:32.244835+0000","last_undegraded":"2021-02-10T04:38:32.244835+0000","last_fullsized":"2021-02-10T04:38:32.244835+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:33:56.837816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:56.837816+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"662","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.225927+0000","last_change":"2021-02-10T04:29:36.663903+0000","last_active":"2021-02-10T04:38:33.225927+0000","last_peered":"2021-02-10T04:38:33.225927+0000","last_clean":"2021-02-10T04:38:33.225927+0000","last_became_active":"2021-02-10T04:29:07.158075+0000","last_became_peered":"2021-02-10T04:29:07.158075+0000","last_unstale":"2021-02-10T04:38:33.225927+0000","last_undegraded":"2021-02-10T04:38:33.225927+0000","last_fullsized":"2021-02-10T04:38:33.225927+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.663840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.663840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"689","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.244310+0000","last_change":"2021-02-10T04:37:37.246321+0000","last_active":"2021-02-10T04:38:31.244310+0000","last_peered":"2021-02-10T04:38:31.244310+0000","last_clean":"2021-02-10T04:38:31.244310+0000","last_became_active":"2021-02-10T04:37:37.245931+0000","last_became_peered":"2021-02-10T04:37:37.245931+0000","last_unstale":"2021-02-10T04:38:31.244310+0000","last_undegraded":"2021-02-10T04:38:31.244310+0000","last_fullsized":"2021-02-10T04:38:31.244310+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:34.482906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:34.482906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"658","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.233014+0000","last_change":"2021-02-10T04:29:35.382459+0000","last_active":"2021-02-10T04:38:33.233014+0000","last_peered":"2021-02-10T04:38:33.233014+0000","last_clean":"2021-02-10T04:38:33.233014+0000","last_became_active":"2021-02-10T04:25:49.010614+0000","last_became_peered":"2021-02-10T04:25:49.010614+0000","last_unstale":"2021-02-10T04:38:33.233014+0000","last_undegraded":"2021-02-10T04:38:33.233014+0000","last_fullsized":"2021-02-10T04:38:33.233014+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:35.382341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.382341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"653","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.244675+0000","last_change":"2021-02-10T04:28:38.503398+0000","last_active":"2021-02-10T04:38:31.244675+0000","last_peered":"2021-02-10T04:38:31.244675+0000","last_clean":"2021-02-10T04:38:31.244675+0000","last_became_active":"2021-02-10T04:25:53.037506+0000","last_became_peered":"2021-02-10T04:25:53.037506+0000","last_unstale":"2021-02-10T04:38:31.244675+0000","last_undegraded":"2021-02-10T04:38:31.244675+0000","last_fullsized":"2021-02-10T04:38:31.244675+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:38.503309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:38.503309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"687'285","reported_seq":"2230","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.228479+0000","last_change":"2021-02-10T04:38:32.244566+0000","last_active":"2021-02-10T04:38:33.228479+0000","last_peered":"2021-02-10T04:38:33.228479+0000","last_clean":"2021-02-10T04:38:33.228479+0000","last_became_active":"2021-02-10T04:28:25.150109+0000","last_became_peered":"2021-02-10T04:28:25.150109+0000","last_unstale":"2021-02-10T04:38:33.228479+0000","last_undegraded":"2021-02-10T04:38:33.228479+0000","last_fullsized":"2021-02-10T04:38:33.228479+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'36","last_scrub_stamp":"2021-02-10T04:30:13.863430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:13.863430+0000","log_size":285,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10806661,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":2,"num_read":186,"num_read_kb":180058,"num_write":113,"num_write_kb":38951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"201","length":"1"},{"start":"205","length":"2"},{"start":"20d","length":"1"},{"start":"215","length":"1"},{"start":"21b","length":"1"},{"start":"21d","length":"1"},{"start":"21f","length":"1"},{"start":"222","length":"2"},{"start":"225","length":"1"},{"start":"233","length":"1"},{"start":"238","length":"1"},{"start":"23b","length":"1"},{"start":"244","length":"1"},{"start":"246","length":"1"},{"start":"24c","length":"1"}]},{"pgid":"5.6","version":"0'0","reported_seq":"67","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.244623+0000","last_change":"2021-02-10T04:37:37.230965+0000","last_active":"2021-02-10T04:38:31.244623+0000","last_peered":"2021-02-10T04:38:31.244623+0000","last_clean":"2021-02-10T04:38:31.244623+0000","last_became_active":"2021-02-10T04:37:37.230604+0000","last_became_peered":"2021-02-10T04:37:37.230604+0000","last_unstale":"2021-02-10T04:38:31.244623+0000","last_undegraded":"2021-02-10T04:38:31.244623+0000","last_fullsized":"2021-02-10T04:38:31.244623+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.779767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.779767+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"538'33","reported_seq":"426","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.244325+0000","last_change":"2021-02-10T04:33:05.147122+0000","last_active":"2021-02-10T04:38:31.244325+0000","last_peered":"2021-02-10T04:38:31.244325+0000","last_clean":"2021-02-10T04:38:31.244325+0000","last_became_active":"2021-02-10T04:33:05.146695+0000","last_became_peered":"2021-02-10T04:33:05.146695+0000","last_unstale":"2021-02-10T04:38:31.244325+0000","last_undegraded":"2021-02-10T04:38:31.244325+0000","last_fullsized":"2021-02-10T04:38:31.244325+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'29","last_scrub_stamp":"2021-02-10T04:29:25.974280+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.974280+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":38,"num_write":28,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"128","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.231773+0000","last_change":"2021-02-10T04:37:37.258992+0000","last_active":"2021-02-10T04:38:33.231773+0000","last_peered":"2021-02-10T04:38:33.231773+0000","last_clean":"2021-02-10T04:38:33.231773+0000","last_became_active":"2021-02-10T04:37:37.257600+0000","last_became_peered":"2021-02-10T04:37:37.257600+0000","last_unstale":"2021-02-10T04:38:33.231773+0000","last_undegraded":"2021-02-10T04:38:33.231773+0000","last_fullsized":"2021-02-10T04:38:33.231773+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:39.690436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:39.690436+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"156","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.243713+0000","last_change":"2021-02-10T04:38:33.243623+0000","last_active":"2021-02-10T04:38:33.243713+0000","last_peered":"2021-02-10T04:38:33.243713+0000","last_clean":"2021-02-10T04:38:33.243713+0000","last_became_active":"2021-02-10T04:37:37.249591+0000","last_became_peered":"2021-02-10T04:37:37.249591+0000","last_unstale":"2021-02-10T04:38:33.243713+0000","last_undegraded":"2021-02-10T04:38:33.243713+0000","last_fullsized":"2021-02-10T04:38:33.243713+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:38:33.243519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:33.243519+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":63,"num_read_kb":58,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"581","reported_epoch":"694","state":"active+clean","last_fresh":"2021-02-10T04:38:32.244930+0000","last_change":"2021-02-10T04:29:33.747527+0000","last_active":"2021-02-10T04:38:32.244930+0000","last_peered":"2021-02-10T04:38:32.244930+0000","last_clean":"2021-02-10T04:38:32.244930+0000","last_became_active":"2021-02-10T04:29:07.158869+0000","last_became_peered":"2021-02-10T04:29:07.158869+0000","last_unstale":"2021-02-10T04:38:32.244930+0000","last_undegraded":"2021-02-10T04:38:32.244930+0000","last_fullsized":"2021-02-10T04:38:32.244930+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:33.747448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:33.747448+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.4","version":"692'443","reported_seq":"3068","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.228709+0000","last_change":"2021-02-10T04:38:32.244310+0000","last_active":"2021-02-10T04:38:33.228709+0000","last_peered":"2021-02-10T04:38:33.228709+0000","last_clean":"2021-02-10T04:38:33.228709+0000","last_became_active":"2021-02-10T04:28:25.151037+0000","last_became_peered":"2021-02-10T04:28:25.151037+0000","last_unstale":"2021-02-10T04:38:33.228709+0000","last_undegraded":"2021-02-10T04:38:33.228709+0000","last_fullsized":"2021-02-10T04:38:33.228709+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'239","last_scrub_stamp":"2021-02-10T04:33:46.270478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:46.270478+0000","log_size":443,"ondisk_log_size":443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19256283,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":231,"num_read_kb":209644,"num_write":206,"num_write_kb":78948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"201","length":"1"},{"start":"205","length":"2"},{"start":"20d","length":"1"},{"start":"215","length":"1"},{"start":"21b","length":"1"},{"start":"21d","length":"1"},{"start":"21f","length":"1"},{"start":"222","length":"2"},{"start":"225","length":"1"},{"start":"233","length":"1"},{"start":"238","length":"1"},{"start":"23b","length":"1"},{"start":"244","length":"1"},{"start":"246","length":"1"},{"start":"24c","length":"1"}]},{"pgid":"1.2","version":"0'0","reported_seq":"672","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.227538+0000","last_change":"2021-02-10T04:29:29.683867+0000","last_active":"2021-02-10T04:38:33.227538+0000","last_peered":"2021-02-10T04:38:33.227538+0000","last_clean":"2021-02-10T04:38:33.227538+0000","last_became_active":"2021-02-10T04:29:07.159607+0000","last_became_peered":"2021-02-10T04:29:07.159607+0000","last_unstale":"2021-02-10T04:38:33.227538+0000","last_undegraded":"2021-02-10T04:38:33.227538+0000","last_fullsized":"2021-02-10T04:38:33.227538+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.683805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.683805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"657","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.229410+0000","last_change":"2021-02-10T04:28:43.740930+0000","last_active":"2021-02-10T04:38:33.229410+0000","last_peered":"2021-02-10T04:38:33.229410+0000","last_clean":"2021-02-10T04:38:33.229410+0000","last_became_active":"2021-02-10T04:25:51.027624+0000","last_became_peered":"2021-02-10T04:25:51.027624+0000","last_unstale":"2021-02-10T04:38:33.229410+0000","last_undegraded":"2021-02-10T04:38:33.229410+0000","last_fullsized":"2021-02-10T04:38:33.229410+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:43.740853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:28:43.740853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"538'43","reported_seq":"378","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.232629+0000","last_change":"2021-02-10T04:37:38.636607+0000","last_active":"2021-02-10T04:38:33.232629+0000","last_peered":"2021-02-10T04:38:33.232629+0000","last_clean":"2021-02-10T04:38:33.232629+0000","last_became_active":"2021-02-10T04:37:37.244344+0000","last_became_peered":"2021-02-10T04:37:37.244344+0000","last_unstale":"2021-02-10T04:38:33.232629+0000","last_undegraded":"2021-02-10T04:38:33.232629+0000","last_fullsized":"2021-02-10T04:38:33.232629+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'31","last_scrub_stamp":"2021-02-10T04:33:49.423083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:49.423083+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":81,"num_read_kb":57,"num_write":46,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"673","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.244295+0000","last_change":"2021-02-10T04:28:36.491041+0000","last_active":"2021-02-10T04:38:31.244295+0000","last_peered":"2021-02-10T04:38:31.244295+0000","last_clean":"2021-02-10T04:38:31.244295+0000","last_became_active":"2021-02-10T04:28:21.125541+0000","last_became_peered":"2021-02-10T04:28:21.125541+0000","last_unstale":"2021-02-10T04:38:31.244295+0000","last_undegraded":"2021-02-10T04:38:31.244295+0000","last_fullsized":"2021-02-10T04:38:31.244295+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:36.490961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:36.490961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"654","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.233017+0000","last_change":"2021-02-10T04:29:36.396044+0000","last_active":"2021-02-10T04:38:33.233017+0000","last_peered":"2021-02-10T04:38:33.233017+0000","last_clean":"2021-02-10T04:38:33.233017+0000","last_became_active":"2021-02-10T04:25:53.049487+0000","last_became_peered":"2021-02-10T04:25:53.049487+0000","last_unstale":"2021-02-10T04:38:33.233017+0000","last_undegraded":"2021-02-10T04:38:33.233017+0000","last_fullsized":"2021-02-10T04:38:33.233017+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.395943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.395943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"695'1413","reported_seq":"6926","reported_epoch":"695","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:38:33.885475+0000","last_change":"2021-02-10T04:38:33.885475+0000","last_active":"2021-02-10T04:38:33.885475+0000","last_peered":"2021-02-10T04:38:33.885475+0000","last_clean":"2021-02-10T04:38:33.885475+0000","last_became_active":"2021-02-10T04:28:25.147959+0000","last_became_peered":"2021-02-10T04:28:25.147959+0000","last_unstale":"2021-02-10T04:38:33.885475+0000","last_undegraded":"2021-02-10T04:38:33.885475+0000","last_fullsized":"2021-02-10T04:38:33.885475+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'223","last_scrub_stamp":"2021-02-10T04:30:14.478639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:14.478639+0000","log_size":1413,"ondisk_log_size":1413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"stat_sum":{"num_bytes":56246060,"num_objects":27,"num_object_clones":20,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":3,"num_read":750,"num_read_kb":707954,"num_write":620,"num_write_kb":248896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"201","length":"1"},{"start":"205","length":"2"},{"start":"20d","length":"1"},{"start":"215","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":"605","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.243776+0000","last_change":"2021-02-10T04:37:37.219806+0000","last_active":"2021-02-10T04:38:31.243776+0000","last_peered":"2021-02-10T04:38:31.243776+0000","last_clean":"2021-02-10T04:38:31.243776+0000","last_became_active":"2021-02-10T04:37:37.219422+0000","last_became_peered":"2021-02-10T04:37:37.219422+0000","last_unstale":"2021-02-10T04:38:31.243776+0000","last_undegraded":"2021-02-10T04:38:31.243776+0000","last_fullsized":"2021-02-10T04:38:31.243776+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:39.434210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:39.434210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"58'2","reported_seq":"817","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.227388+0000","last_change":"2021-02-10T04:38:27.092813+0000","last_active":"2021-02-10T04:38:33.227388+0000","last_peered":"2021-02-10T04:38:33.227388+0000","last_clean":"2021-02-10T04:38:33.227388+0000","last_became_active":"2021-02-10T04:25:51.026251+0000","last_became_peered":"2021-02-10T04:25:51.026251+0000","last_unstale":"2021-02-10T04:38:33.227388+0000","last_undegraded":"2021-02-10T04:38:33.227388+0000","last_fullsized":"2021-02-10T04:38:33.227388+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:38:27.092697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:38:27.092697+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"538'44","reported_seq":"428","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.244617+0000","last_change":"2021-02-10T04:33:05.142280+0000","last_active":"2021-02-10T04:38:31.244617+0000","last_peered":"2021-02-10T04:38:31.244617+0000","last_clean":"2021-02-10T04:38:31.244617+0000","last_became_active":"2021-02-10T04:33:05.141901+0000","last_became_peered":"2021-02-10T04:33:05.141901+0000","last_unstale":"2021-02-10T04:38:31.244617+0000","last_undegraded":"2021-02-10T04:38:31.244617+0000","last_fullsized":"2021-02-10T04:38:31.244617+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'34","last_scrub_stamp":"2021-02-10T04:28:49.959162+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:28:49.959162+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":60,"num_read_kb":42,"num_write":32,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"422","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.227793+0000","last_change":"2021-02-10T04:37:37.247740+0000","last_active":"2021-02-10T04:38:33.227793+0000","last_peered":"2021-02-10T04:38:33.227793+0000","last_clean":"2021-02-10T04:38:33.227793+0000","last_became_active":"2021-02-10T04:37:37.247405+0000","last_became_peered":"2021-02-10T04:37:37.247405+0000","last_unstale":"2021-02-10T04:38:33.227793+0000","last_undegraded":"2021-02-10T04:38:33.227793+0000","last_fullsized":"2021-02-10T04:38:33.227793+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:34.053323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:34.053323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"669","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.225828+0000","last_change":"2021-02-10T04:38:26.124897+0000","last_active":"2021-02-10T04:38:33.225828+0000","last_peered":"2021-02-10T04:38:33.225828+0000","last_clean":"2021-02-10T04:38:33.225828+0000","last_became_active":"2021-02-10T04:37:37.254482+0000","last_became_peered":"2021-02-10T04:37:37.254482+0000","last_unstale":"2021-02-10T04:38:33.225828+0000","last_undegraded":"2021-02-10T04:38:33.225828+0000","last_fullsized":"2021-02-10T04:38:33.225828+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:26.124805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:38:26.124805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"293'77","reported_seq":"1847","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.236601+0000","last_change":"2021-02-10T04:38:30.236601+0000","last_active":"2021-02-10T04:38:30.236601+0000","last_peered":"2021-02-10T04:38:30.236601+0000","last_clean":"2021-02-10T04:38:30.236601+0000","last_became_active":"2021-02-10T04:28:25.154434+0000","last_became_peered":"2021-02-10T04:28:25.154434+0000","last_unstale":"2021-02-10T04:38:30.236601+0000","last_undegraded":"2021-02-10T04:38:30.236601+0000","last_fullsized":"2021-02-10T04:38:30.236601+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7546343,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":74349,"num_write":37,"num_write_kb":13263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1eb","length":"1"},{"start":"1ee","length":"3"},{"start":"1f2","length":"3"},{"start":"1f7","length":"2"},{"start":"1fc","length":"1"},{"start":"201","length":"1"},{"start":"205","length":"2"},{"start":"20d","length":"1"},{"start":"215","length":"1"},{"start":"21b","length":"1"},{"start":"21d","length":"1"},{"start":"21f","length":"1"},{"start":"222","length":"2"},{"start":"225","length":"1"},{"start":"233","length":"1"},{"start":"238","length":"1"},{"start":"23b","length":"1"},{"start":"244","length":"1"},{"start":"246","length":"1"}]},{"pgid":"1.7","version":"0'0","reported_seq":"678","reported_epoch":"694","state":"active+clean","last_fresh":"2021-02-10T04:38:32.245395+0000","last_change":"2021-02-10T04:29:39.829039+0000","last_active":"2021-02-10T04:38:32.245395+0000","last_peered":"2021-02-10T04:38:32.245395+0000","last_clean":"2021-02-10T04:38:32.245395+0000","last_became_active":"2021-02-10T04:28:49.053127+0000","last_became_peered":"2021-02-10T04:28:49.053127+0000","last_unstale":"2021-02-10T04:38:32.245395+0000","last_undegraded":"2021-02-10T04:38:32.245395+0000","last_fullsized":"2021-02-10T04:38:32.245395+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:39.828975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:39.828975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"69","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.226095+0000","last_change":"2021-02-10T04:37:37.235271+0000","last_active":"2021-02-10T04:38:33.226095+0000","last_peered":"2021-02-10T04:38:33.226095+0000","last_clean":"2021-02-10T04:38:33.226095+0000","last_became_active":"2021-02-10T04:37:37.234985+0000","last_became_peered":"2021-02-10T04:37:37.234985+0000","last_unstale":"2021-02-10T04:38:33.226095+0000","last_undegraded":"2021-02-10T04:38:33.226095+0000","last_fullsized":"2021-02-10T04:38:33.226095+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:20.757194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:20.757194+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"538'33","reported_seq":"724","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.236235+0000","last_change":"2021-02-10T04:28:47.476743+0000","last_active":"2021-02-10T04:38:30.236235+0000","last_peered":"2021-02-10T04:38:30.236235+0000","last_clean":"2021-02-10T04:38:30.236235+0000","last_became_active":"2021-02-10T04:25:49.013259+0000","last_became_peered":"2021-02-10T04:25:49.013259+0000","last_unstale":"2021-02-10T04:38:30.236235+0000","last_undegraded":"2021-02-10T04:38:30.236235+0000","last_fullsized":"2021-02-10T04:38:30.236235+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'27","last_scrub_stamp":"2021-02-10T04:28:47.476669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:47.476669+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":67,"num_read_kb":50,"num_write":35,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"787","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.238371+0000","last_change":"2021-02-10T04:37:37.351533+0000","last_active":"2021-02-10T04:38:30.238371+0000","last_peered":"2021-02-10T04:38:30.238371+0000","last_clean":"2021-02-10T04:38:30.238371+0000","last_became_active":"2021-02-10T04:37:37.238701+0000","last_became_peered":"2021-02-10T04:37:37.238701+0000","last_unstale":"2021-02-10T04:38:30.238371+0000","last_undegraded":"2021-02-10T04:38:30.238371+0000","last_fullsized":"2021-02-10T04:38:30.238371+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:33:49.760989+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:33:49.760989+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"654","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.233671+0000","last_change":"2021-02-10T04:29:37.364781+0000","last_active":"2021-02-10T04:38:33.233671+0000","last_peered":"2021-02-10T04:38:33.233671+0000","last_clean":"2021-02-10T04:38:33.233671+0000","last_became_active":"2021-02-10T04:25:53.047736+0000","last_became_peered":"2021-02-10T04:25:53.047736+0000","last_unstale":"2021-02-10T04:38:33.233671+0000","last_undegraded":"2021-02-10T04:38:33.233671+0000","last_fullsized":"2021-02-10T04:38:33.233671+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:37.364697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:37.364697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"695'1014","reported_seq":"4107","reported_epoch":"695","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:38:33.494072+0000","last_change":"2021-02-10T04:38:33.491322+0000","last_active":"2021-02-10T04:38:33.494072+0000","last_peered":"2021-02-10T04:38:33.494072+0000","last_clean":"2021-02-10T04:38:33.494072+0000","last_became_active":"2021-02-10T04:37:48.215401+0000","last_became_peered":"2021-02-10T04:37:48.215401+0000","last_unstale":"2021-02-10T04:38:33.494072+0000","last_undegraded":"2021-02-10T04:38:33.494072+0000","last_fullsized":"2021-02-10T04:38:33.494072+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":1014,"ondisk_log_size":1014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"stat_sum":{"num_bytes":42683398,"num_objects":19,"num_object_clones":16,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":502,"num_read_kb":488962,"num_write":434,"num_write_kb":159864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":42623785,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"201","length":"1"},{"start":"205","length":"2"},{"start":"20d","length":"1"},{"start":"215","length":"1"},{"start":"21b","length":"1"},{"start":"21d","length":"1"},{"start":"21f","length":"1"},{"start":"222","length":"2"},{"start":"225","length":"1"}]},{"pgid":"4.16","version":"538'34","reported_seq":"728","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.228634+0000","last_change":"2021-02-10T04:29:25.645462+0000","last_active":"2021-02-10T04:38:33.228634+0000","last_peered":"2021-02-10T04:38:33.228634+0000","last_clean":"2021-02-10T04:38:33.228634+0000","last_became_active":"2021-02-10T04:25:49.008942+0000","last_became_peered":"2021-02-10T04:25:49.008942+0000","last_unstale":"2021-02-10T04:38:33.228634+0000","last_undegraded":"2021-02-10T04:38:33.228634+0000","last_fullsized":"2021-02-10T04:38:33.228634+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'28","last_scrub_stamp":"2021-02-10T04:29:25.645396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.645396+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":71,"num_read_kb":49,"num_write":35,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"82","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.225942+0000","last_change":"2021-02-10T04:37:37.313885+0000","last_active":"2021-02-10T04:38:33.225942+0000","last_peered":"2021-02-10T04:38:33.225942+0000","last_clean":"2021-02-10T04:38:33.225942+0000","last_became_active":"2021-02-10T04:37:37.234649+0000","last_became_peered":"2021-02-10T04:37:37.234649+0000","last_unstale":"2021-02-10T04:38:33.225942+0000","last_undegraded":"2021-02-10T04:38:33.225942+0000","last_fullsized":"2021-02-10T04:38:33.225942+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:28:45.647916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:45.647916+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":247,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"656","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.234071+0000","last_change":"2021-02-10T04:28:33.321856+0000","last_active":"2021-02-10T04:38:33.234071+0000","last_peered":"2021-02-10T04:38:33.234071+0000","last_clean":"2021-02-10T04:38:33.234071+0000","last_became_active":"2021-02-10T04:25:51.030651+0000","last_became_peered":"2021-02-10T04:25:51.030651+0000","last_unstale":"2021-02-10T04:38:33.234071+0000","last_undegraded":"2021-02-10T04:38:33.234071+0000","last_fullsized":"2021-02-10T04:38:33.234071+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:33.321784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:33.321784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"66","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.236818+0000","last_change":"2021-02-10T04:37:37.238450+0000","last_active":"2021-02-10T04:38:30.236818+0000","last_peered":"2021-02-10T04:38:30.236818+0000","last_clean":"2021-02-10T04:38:30.236818+0000","last_became_active":"2021-02-10T04:37:37.238049+0000","last_became_peered":"2021-02-10T04:37:37.238049+0000","last_unstale":"2021-02-10T04:38:30.236818+0000","last_undegraded":"2021-02-10T04:38:30.236818+0000","last_fullsized":"2021-02-10T04:38:30.236818+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.776340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.776340+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"538'19","reported_seq":"740","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.233838+0000","last_change":"2021-02-10T04:38:26.845289+0000","last_active":"2021-02-10T04:38:33.233838+0000","last_peered":"2021-02-10T04:38:33.233838+0000","last_clean":"2021-02-10T04:38:33.233838+0000","last_became_active":"2021-02-10T04:37:37.247396+0000","last_became_peered":"2021-02-10T04:37:37.247396+0000","last_unstale":"2021-02-10T04:38:33.233838+0000","last_undegraded":"2021-02-10T04:38:33.233838+0000","last_fullsized":"2021-02-10T04:38:33.233838+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'19","last_scrub_stamp":"2021-02-10T04:38:26.845195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:26.845195+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":19,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"667","reported_epoch":"694","state":"active+clean","last_fresh":"2021-02-10T04:38:32.243196+0000","last_change":"2021-02-10T04:29:18.565481+0000","last_active":"2021-02-10T04:38:32.243196+0000","last_peered":"2021-02-10T04:38:32.243196+0000","last_clean":"2021-02-10T04:38:32.243196+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:38:32.243196+0000","last_undegraded":"2021-02-10T04:38:32.243196+0000","last_fullsized":"2021-02-10T04:38:32.243196+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:29:18.565413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:18.565413+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"655","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.244324+0000","last_change":"2021-02-10T04:29:18.429730+0000","last_active":"2021-02-10T04:38:31.244324+0000","last_peered":"2021-02-10T04:38:31.244324+0000","last_clean":"2021-02-10T04:38:31.244324+0000","last_became_active":"2021-02-10T04:25:51.031864+0000","last_became_peered":"2021-02-10T04:25:51.031864+0000","last_unstale":"2021-02-10T04:38:31.244324+0000","last_undegraded":"2021-02-10T04:38:31.244324+0000","last_fullsized":"2021-02-10T04:38:31.244324+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:18.429649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:18.429649+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"581","reported_epoch":"694","state":"active+clean","last_fresh":"2021-02-10T04:38:32.245017+0000","last_change":"2021-02-10T04:29:36.866949+0000","last_active":"2021-02-10T04:38:32.245017+0000","last_peered":"2021-02-10T04:38:32.245017+0000","last_clean":"2021-02-10T04:38:32.245017+0000","last_became_active":"2021-02-10T04:29:07.147249+0000","last_became_peered":"2021-02-10T04:29:07.147249+0000","last_unstale":"2021-02-10T04:38:32.245017+0000","last_undegraded":"2021-02-10T04:38:32.245017+0000","last_fullsized":"2021-02-10T04:38:32.245017+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.866833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.866833+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"538'36","reported_seq":"864","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.232398+0000","last_change":"2021-02-10T04:37:38.541093+0000","last_active":"2021-02-10T04:38:33.232398+0000","last_peered":"2021-02-10T04:38:33.232398+0000","last_clean":"2021-02-10T04:38:33.232398+0000","last_became_active":"2021-02-10T04:37:37.244921+0000","last_became_peered":"2021-02-10T04:37:37.244921+0000","last_unstale":"2021-02-10T04:38:33.232398+0000","last_undegraded":"2021-02-10T04:38:33.232398+0000","last_fullsized":"2021-02-10T04:38:33.232398+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:33:46.785905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:46.785905+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":36,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"84'7","reported_seq":"684","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.244227+0000","last_change":"2021-02-10T04:30:12.538670+0000","last_active":"2021-02-10T04:38:31.244227+0000","last_peered":"2021-02-10T04:38:31.244227+0000","last_clean":"2021-02-10T04:38:31.244227+0000","last_became_active":"2021-02-10T04:25:36.334366+0000","last_became_peered":"2021-02-10T04:25:36.334366+0000","last_unstale":"2021-02-10T04:38:31.244227+0000","last_undegraded":"2021-02-10T04:38:31.244227+0000","last_fullsized":"2021-02-10T04:38:31.244227+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'7","last_scrub_stamp":"2021-02-10T04:30:12.538549+0000","last_deep_scrub":"84'7","last_deep_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_clean_scrub_stamp":"2021-02-10T04:30:12.538549+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"58'2","reported_seq":"810","reported_epoch":"694","state":"active+clean","last_fresh":"2021-02-10T04:38:32.242195+0000","last_change":"2021-02-10T04:28:47.496534+0000","last_active":"2021-02-10T04:38:32.242195+0000","last_peered":"2021-02-10T04:38:32.242195+0000","last_clean":"2021-02-10T04:38:32.242195+0000","last_became_active":"2021-02-10T04:25:51.032328+0000","last_became_peered":"2021-02-10T04:25:51.032328+0000","last_unstale":"2021-02-10T04:38:32.242195+0000","last_undegraded":"2021-02-10T04:38:32.242195+0000","last_fullsized":"2021-02-10T04:38:32.242195+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:28:47.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:47.496468+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"655","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.228118+0000","last_change":"2021-02-10T04:28:45.697744+0000","last_active":"2021-02-10T04:38:33.228118+0000","last_peered":"2021-02-10T04:38:33.228118+0000","last_clean":"2021-02-10T04:38:33.228118+0000","last_became_active":"2021-02-10T04:25:53.045177+0000","last_became_peered":"2021-02-10T04:25:53.045177+0000","last_unstale":"2021-02-10T04:38:33.228118+0000","last_undegraded":"2021-02-10T04:38:33.228118+0000","last_fullsized":"2021-02-10T04:38:33.228118+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_clean_scrub_stamp":"2021-02-10T04:28:45.697677+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"538'25","reported_seq":"705","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.243976+0000","last_change":"2021-02-10T04:28:50.493712+0000","last_active":"2021-02-10T04:38:31.243976+0000","last_peered":"2021-02-10T04:38:31.243976+0000","last_clean":"2021-02-10T04:38:31.243976+0000","last_became_active":"2021-02-10T04:25:49.010890+0000","last_became_peered":"2021-02-10T04:25:49.010890+0000","last_unstale":"2021-02-10T04:38:31.243976+0000","last_undegraded":"2021-02-10T04:38:31.243976+0000","last_fullsized":"2021-02-10T04:38:31.243976+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2021-02-10T04:28:50.493641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:50.493641+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":35,"num_write":26,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"671","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.227230+0000","last_change":"2021-02-10T04:29:19.564845+0000","last_active":"2021-02-10T04:38:33.227230+0000","last_peered":"2021-02-10T04:38:33.227230+0000","last_clean":"2021-02-10T04:38:33.227230+0000","last_became_active":"2021-02-10T04:25:34.321096+0000","last_became_peered":"2021-02-10T04:25:34.321096+0000","last_unstale":"2021-02-10T04:38:33.227230+0000","last_undegraded":"2021-02-10T04:38:33.227230+0000","last_fullsized":"2021-02-10T04:38:33.227230+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:19.564725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:29:19.564725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"639'3","reported_seq":"824","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.232552+0000","last_change":"2021-02-10T04:37:37.922249+0000","last_active":"2021-02-10T04:38:33.232552+0000","last_peered":"2021-02-10T04:38:33.232552+0000","last_clean":"2021-02-10T04:38:33.232552+0000","last_became_active":"2021-02-10T04:37:37.245428+0000","last_became_peered":"2021-02-10T04:37:37.245428+0000","last_unstale":"2021-02-10T04:38:33.232552+0000","last_undegraded":"2021-02-10T04:38:33.232552+0000","last_fullsized":"2021-02-10T04:38:33.232552+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:29:38.343834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.343834+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"661","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.238634+0000","last_change":"2021-02-10T04:37:37.238692+0000","last_active":"2021-02-10T04:38:30.238634+0000","last_peered":"2021-02-10T04:38:30.238634+0000","last_clean":"2021-02-10T04:38:30.238634+0000","last_became_active":"2021-02-10T04:37:37.238360+0000","last_became_peered":"2021-02-10T04:37:37.238360+0000","last_unstale":"2021-02-10T04:38:30.238634+0000","last_undegraded":"2021-02-10T04:38:30.238634+0000","last_fullsized":"2021-02-10T04:38:30.238634+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.934078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.934078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"538'21","reported_seq":"420","reported_epoch":"694","state":"active+clean","last_fresh":"2021-02-10T04:38:32.243981+0000","last_change":"2021-02-10T04:37:38.590027+0000","last_active":"2021-02-10T04:38:32.243981+0000","last_peered":"2021-02-10T04:38:32.243981+0000","last_clean":"2021-02-10T04:38:32.243981+0000","last_became_active":"2021-02-10T04:37:37.249007+0000","last_became_peered":"2021-02-10T04:37:37.249007+0000","last_unstale":"2021-02-10T04:38:32.243981+0000","last_undegraded":"2021-02-10T04:38:32.243981+0000","last_fullsized":"2021-02-10T04:38:32.243981+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:30:11.170034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.170034+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"670","reported_epoch":"694","state":"active+clean","last_fresh":"2021-02-10T04:38:32.244520+0000","last_change":"2021-02-10T04:38:26.127398+0000","last_active":"2021-02-10T04:38:32.244520+0000","last_peered":"2021-02-10T04:38:32.244520+0000","last_clean":"2021-02-10T04:38:32.244520+0000","last_became_active":"2021-02-10T04:25:34.330070+0000","last_became_peered":"2021-02-10T04:25:34.330070+0000","last_unstale":"2021-02-10T04:38:32.244520+0000","last_undegraded":"2021-02-10T04:38:32.244520+0000","last_fullsized":"2021-02-10T04:38:32.244520+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:26.127288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:26.127288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"656","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.233595+0000","last_change":"2021-02-10T04:28:36.311808+0000","last_active":"2021-02-10T04:38:33.233595+0000","last_peered":"2021-02-10T04:38:33.233595+0000","last_clean":"2021-02-10T04:38:33.233595+0000","last_became_active":"2021-02-10T04:25:51.029155+0000","last_became_peered":"2021-02-10T04:25:51.029155+0000","last_unstale":"2021-02-10T04:38:33.233595+0000","last_undegraded":"2021-02-10T04:38:33.233595+0000","last_fullsized":"2021-02-10T04:38:33.233595+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:36.311725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:28:36.311725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"660","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.225777+0000","last_change":"2021-02-10T04:33:49.338303+0000","last_active":"2021-02-10T04:38:33.225777+0000","last_peered":"2021-02-10T04:38:33.225777+0000","last_clean":"2021-02-10T04:38:33.225777+0000","last_became_active":"2021-02-10T04:25:53.039322+0000","last_became_peered":"2021-02-10T04:25:53.039322+0000","last_unstale":"2021-02-10T04:38:33.225777+0000","last_undegraded":"2021-02-10T04:38:33.225777+0000","last_fullsized":"2021-02-10T04:38:33.225777+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:49.338203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:33:49.338203+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"538'23","reported_seq":"690","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.236294+0000","last_change":"2021-02-10T04:27:42.487239+0000","last_active":"2021-02-10T04:38:30.236294+0000","last_peered":"2021-02-10T04:38:30.236294+0000","last_clean":"2021-02-10T04:38:30.236294+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:38:30.236294+0000","last_undegraded":"2021-02-10T04:38:30.236294+0000","last_fullsized":"2021-02-10T04:38:30.236294+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:27:42.487154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:27:42.487154+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"349","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.244032+0000","last_change":"2021-02-10T04:33:05.148470+0000","last_active":"2021-02-10T04:38:31.244032+0000","last_peered":"2021-02-10T04:38:31.244032+0000","last_clean":"2021-02-10T04:38:31.244032+0000","last_became_active":"2021-02-10T04:33:05.147956+0000","last_became_peered":"2021-02-10T04:33:05.147956+0000","last_unstale":"2021-02-10T04:38:31.244032+0000","last_undegraded":"2021-02-10T04:38:31.244032+0000","last_fullsized":"2021-02-10T04:38:31.244032+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:28:33.070726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:33.070726+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"338","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.244503+0000","last_change":"2021-02-10T04:33:05.150906+0000","last_active":"2021-02-10T04:38:31.244503+0000","last_peered":"2021-02-10T04:38:31.244503+0000","last_clean":"2021-02-10T04:38:31.244503+0000","last_became_active":"2021-02-10T04:33:05.150529+0000","last_became_peered":"2021-02-10T04:33:05.150529+0000","last_unstale":"2021-02-10T04:38:31.244503+0000","last_undegraded":"2021-02-10T04:38:31.244503+0000","last_fullsized":"2021-02-10T04:38:31.244503+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:33.026988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:29:33.026988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"654","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.231516+0000","last_change":"2021-02-10T04:29:24.421782+0000","last_active":"2021-02-10T04:38:33.231516+0000","last_peered":"2021-02-10T04:38:33.231516+0000","last_clean":"2021-02-10T04:38:33.231516+0000","last_became_active":"2021-02-10T04:25:53.043277+0000","last_became_peered":"2021-02-10T04:25:53.043277+0000","last_unstale":"2021-02-10T04:38:33.231516+0000","last_undegraded":"2021-02-10T04:38:33.231516+0000","last_fullsized":"2021-02-10T04:38:33.231516+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.421710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.421710+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"538'16","reported_seq":"817","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.234637+0000","last_change":"2021-02-10T04:38:22.919477+0000","last_active":"2021-02-10T04:38:33.234637+0000","last_peered":"2021-02-10T04:38:33.234637+0000","last_clean":"2021-02-10T04:38:33.234637+0000","last_became_active":"2021-02-10T04:26:39.005616+0000","last_became_peered":"2021-02-10T04:26:39.005616+0000","last_unstale":"2021-02-10T04:38:33.234637+0000","last_undegraded":"2021-02-10T04:38:33.234637+0000","last_fullsized":"2021-02-10T04:38:33.234637+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'16","last_scrub_stamp":"2021-02-10T04:38:22.919377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:22.919377+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":36,"num_read_kb":18,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"673","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.244542+0000","last_change":"2021-02-10T04:37:37.230226+0000","last_active":"2021-02-10T04:38:31.244542+0000","last_peered":"2021-02-10T04:38:31.244542+0000","last_clean":"2021-02-10T04:38:31.244542+0000","last_became_active":"2021-02-10T04:37:37.229852+0000","last_became_peered":"2021-02-10T04:37:37.229852+0000","last_unstale":"2021-02-10T04:38:31.244542+0000","last_undegraded":"2021-02-10T04:38:31.244542+0000","last_fullsized":"2021-02-10T04:38:31.244542+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:49.522124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:49.522124+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"656","reported_epoch":"694","state":"active+clean","last_fresh":"2021-02-10T04:38:32.243116+0000","last_change":"2021-02-10T04:29:23.645783+0000","last_active":"2021-02-10T04:38:32.243116+0000","last_peered":"2021-02-10T04:38:32.243116+0000","last_clean":"2021-02-10T04:38:32.243116+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:38:32.243116+0000","last_undegraded":"2021-02-10T04:38:32.243116+0000","last_fullsized":"2021-02-10T04:38:32.243116+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.645694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.645694+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"69","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.236586+0000","last_change":"2021-02-10T04:37:37.244700+0000","last_active":"2021-02-10T04:38:30.236586+0000","last_peered":"2021-02-10T04:38:30.236586+0000","last_clean":"2021-02-10T04:38:30.236586+0000","last_became_active":"2021-02-10T04:37:37.244082+0000","last_became_peered":"2021-02-10T04:37:37.244082+0000","last_unstale":"2021-02-10T04:38:30.236586+0000","last_undegraded":"2021-02-10T04:38:30.236586+0000","last_fullsized":"2021-02-10T04:38:30.236586+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:31.824925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.824925+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.11","version":"538'29","reported_seq":"711","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.231975+0000","last_change":"2021-02-10T04:28:31.382776+0000","last_active":"2021-02-10T04:38:33.231975+0000","last_peered":"2021-02-10T04:38:33.231975+0000","last_clean":"2021-02-10T04:38:33.231975+0000","last_became_active":"2021-02-10T04:25:49.007049+0000","last_became_peered":"2021-02-10T04:25:49.007049+0000","last_unstale":"2021-02-10T04:38:33.231975+0000","last_undegraded":"2021-02-10T04:38:33.231975+0000","last_fullsized":"2021-02-10T04:38:33.231975+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2021-02-10T04:28:31.382671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:31.382671+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":29,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"810","reported_epoch":"694","state":"active+clean","last_fresh":"2021-02-10T04:38:32.245604+0000","last_change":"2021-02-10T04:38:31.166652+0000","last_active":"2021-02-10T04:38:32.245604+0000","last_peered":"2021-02-10T04:38:32.245604+0000","last_clean":"2021-02-10T04:38:32.245604+0000","last_became_active":"2021-02-10T04:37:37.228840+0000","last_became_peered":"2021-02-10T04:37:37.228840+0000","last_unstale":"2021-02-10T04:38:32.245604+0000","last_undegraded":"2021-02-10T04:38:32.245604+0000","last_fullsized":"2021-02-10T04:38:32.245604+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:38:31.166586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:31.166586+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"574","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.235946+0000","last_change":"2021-02-10T04:29:07.150253+0000","last_active":"2021-02-10T04:38:30.235946+0000","last_peered":"2021-02-10T04:38:30.235946+0000","last_clean":"2021-02-10T04:38:30.235946+0000","last_became_active":"2021-02-10T04:29:07.149867+0000","last_became_peered":"2021-02-10T04:29:07.149867+0000","last_unstale":"2021-02-10T04:38:30.235946+0000","last_undegraded":"2021-02-10T04:38:30.235946+0000","last_fullsized":"2021-02-10T04:38:30.235946+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:47.473024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:47.473024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"654","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.235694+0000","last_change":"2021-02-10T04:33:50.809533+0000","last_active":"2021-02-10T04:38:33.235694+0000","last_peered":"2021-02-10T04:38:33.235694+0000","last_clean":"2021-02-10T04:38:33.235694+0000","last_became_active":"2021-02-10T04:25:53.150845+0000","last_became_peered":"2021-02-10T04:25:53.150845+0000","last_unstale":"2021-02-10T04:38:33.235694+0000","last_undegraded":"2021-02-10T04:38:33.235694+0000","last_fullsized":"2021-02-10T04:38:33.235694+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:50.809468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_clean_scrub_stamp":"2021-02-10T04:33:50.809468+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"538'32","reported_seq":"721","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.229320+0000","last_change":"2021-02-10T04:29:38.663351+0000","last_active":"2021-02-10T04:38:33.229320+0000","last_peered":"2021-02-10T04:38:33.229320+0000","last_clean":"2021-02-10T04:38:33.229320+0000","last_became_active":"2021-02-10T04:25:49.009177+0000","last_became_peered":"2021-02-10T04:25:49.009177+0000","last_unstale":"2021-02-10T04:38:33.229320+0000","last_undegraded":"2021-02-10T04:38:33.229320+0000","last_fullsized":"2021-02-10T04:38:33.229320+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2021-02-10T04:29:38.663272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.663272+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":60,"num_read_kb":42,"num_write":32,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"673","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.243619+0000","last_change":"2021-02-10T04:28:37.515037+0000","last_active":"2021-02-10T04:38:31.243619+0000","last_peered":"2021-02-10T04:38:31.243619+0000","last_clean":"2021-02-10T04:38:31.243619+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:38:31.243619+0000","last_undegraded":"2021-02-10T04:38:31.243619+0000","last_fullsized":"2021-02-10T04:38:31.243619+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:37.514943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:37.514943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"58'2","reported_seq":"811","reported_epoch":"694","state":"active+clean","last_fresh":"2021-02-10T04:38:32.843204+0000","last_change":"2021-02-10T04:29:31.726022+0000","last_active":"2021-02-10T04:38:32.843204+0000","last_peered":"2021-02-10T04:38:32.843204+0000","last_clean":"2021-02-10T04:38:32.843204+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:38:32.843204+0000","last_undegraded":"2021-02-10T04:38:32.843204+0000","last_fullsized":"2021-02-10T04:38:32.843204+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:29:31.725932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.725932+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"589","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.244806+0000","last_change":"2021-02-10T04:37:37.247706+0000","last_active":"2021-02-10T04:38:31.244806+0000","last_peered":"2021-02-10T04:38:31.244806+0000","last_clean":"2021-02-10T04:38:31.244806+0000","last_became_active":"2021-02-10T04:37:37.247139+0000","last_became_peered":"2021-02-10T04:37:37.247139+0000","last_unstale":"2021-02-10T04:38:31.244806+0000","last_undegraded":"2021-02-10T04:38:31.244806+0000","last_fullsized":"2021-02-10T04:38:31.244806+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:26.437591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.437591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"538'43","reported_seq":"733","reported_epoch":"694","state":"active+clean","last_fresh":"2021-02-10T04:38:32.245647+0000","last_change":"2021-02-10T04:30:13.526264+0000","last_active":"2021-02-10T04:38:32.245647+0000","last_peered":"2021-02-10T04:38:32.245647+0000","last_clean":"2021-02-10T04:38:32.245647+0000","last_became_active":"2021-02-10T04:25:49.015760+0000","last_became_peered":"2021-02-10T04:25:49.015760+0000","last_unstale":"2021-02-10T04:38:32.245647+0000","last_undegraded":"2021-02-10T04:38:32.245647+0000","last_fullsized":"2021-02-10T04:38:32.245647+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'35","last_scrub_stamp":"2021-02-10T04:30:13.526170+0000","last_deep_scrub":"76'35","last_deep_scrub_stamp":"2021-02-10T04:26:58.198641+0000","last_clean_scrub_stamp":"2021-02-10T04:30:13.526170+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":62,"num_read_kb":42,"num_write":32,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"665","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.231698+0000","last_change":"2021-02-10T04:30:11.485059+0000","last_active":"2021-02-10T04:38:33.231698+0000","last_peered":"2021-02-10T04:38:33.231698+0000","last_clean":"2021-02-10T04:38:33.231698+0000","last_became_active":"2021-02-10T04:25:34.324981+0000","last_became_peered":"2021-02-10T04:25:34.324981+0000","last_unstale":"2021-02-10T04:38:33.231698+0000","last_undegraded":"2021-02-10T04:38:33.231698+0000","last_fullsized":"2021-02-10T04:38:33.231698+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:11.485004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.485004+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"666","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.229381+0000","last_change":"2021-02-10T04:37:37.248748+0000","last_active":"2021-02-10T04:38:33.229381+0000","last_peered":"2021-02-10T04:38:33.229381+0000","last_clean":"2021-02-10T04:38:33.229381+0000","last_became_active":"2021-02-10T04:37:37.246998+0000","last_became_peered":"2021-02-10T04:37:37.246998+0000","last_unstale":"2021-02-10T04:38:33.229381+0000","last_undegraded":"2021-02-10T04:38:33.229381+0000","last_fullsized":"2021-02-10T04:38:33.229381+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.589159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.589159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"338","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.244737+0000","last_change":"2021-02-10T04:33:05.142344+0000","last_active":"2021-02-10T04:38:31.244737+0000","last_peered":"2021-02-10T04:38:31.244737+0000","last_clean":"2021-02-10T04:38:31.244737+0000","last_became_active":"2021-02-10T04:33:05.141911+0000","last_became_peered":"2021-02-10T04:33:05.141911+0000","last_unstale":"2021-02-10T04:38:31.244737+0000","last_undegraded":"2021-02-10T04:38:31.244737+0000","last_fullsized":"2021-02-10T04:38:31.244737+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:25.075937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.075937+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"538'27","reported_seq":"626","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.232476+0000","last_change":"2021-02-10T04:37:39.100758+0000","last_active":"2021-02-10T04:38:33.232476+0000","last_peered":"2021-02-10T04:38:33.232476+0000","last_clean":"2021-02-10T04:38:33.232476+0000","last_became_active":"2021-02-10T04:37:37.257293+0000","last_became_peered":"2021-02-10T04:37:37.257293+0000","last_unstale":"2021-02-10T04:38:33.232476+0000","last_undegraded":"2021-02-10T04:38:33.232476+0000","last_fullsized":"2021-02-10T04:38:33.232476+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:28:46.680176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:46.680176+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"673","reported_epoch":"694","state":"active+clean","last_fresh":"2021-02-10T04:38:32.243861+0000","last_change":"2021-02-10T04:38:32.193334+0000","last_active":"2021-02-10T04:38:32.243861+0000","last_peered":"2021-02-10T04:38:32.243861+0000","last_clean":"2021-02-10T04:38:32.243861+0000","last_became_active":"2021-02-10T04:25:36.336442+0000","last_became_peered":"2021-02-10T04:25:36.336442+0000","last_unstale":"2021-02-10T04:38:32.243861+0000","last_undegraded":"2021-02-10T04:38:32.243861+0000","last_fullsized":"2021-02-10T04:38:32.243861+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:38:32.193240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:32.193240+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"337","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.237348+0000","last_change":"2021-02-10T04:33:05.147757+0000","last_active":"2021-02-10T04:38:30.237348+0000","last_peered":"2021-02-10T04:38:30.237348+0000","last_clean":"2021-02-10T04:38:30.237348+0000","last_became_active":"2021-02-10T04:33:05.146838+0000","last_became_peered":"2021-02-10T04:33:05.146838+0000","last_unstale":"2021-02-10T04:38:30.237348+0000","last_undegraded":"2021-02-10T04:38:30.237348+0000","last_fullsized":"2021-02-10T04:38:30.237348+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:22.078247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:22.078247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"652","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.238013+0000","last_change":"2021-02-10T04:29:31.903091+0000","last_active":"2021-02-10T04:38:30.238013+0000","last_peered":"2021-02-10T04:38:30.238013+0000","last_clean":"2021-02-10T04:38:30.238013+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:38:30.238013+0000","last_undegraded":"2021-02-10T04:38:30.238013+0000","last_fullsized":"2021-02-10T04:38:30.238013+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:31.902996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.902996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"538'28","reported_seq":"713","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.231787+0000","last_change":"2021-02-10T04:38:25.860322+0000","last_active":"2021-02-10T04:38:33.231787+0000","last_peered":"2021-02-10T04:38:33.231787+0000","last_clean":"2021-02-10T04:38:33.231787+0000","last_became_active":"2021-02-10T04:25:49.008997+0000","last_became_peered":"2021-02-10T04:25:49.008997+0000","last_unstale":"2021-02-10T04:38:33.231787+0000","last_undegraded":"2021-02-10T04:38:33.231787+0000","last_fullsized":"2021-02-10T04:38:33.231787+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'28","last_scrub_stamp":"2021-02-10T04:38:25.860247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:25.860247+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":40,"num_write":30,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"677","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.233418+0000","last_change":"2021-02-10T04:28:30.415204+0000","last_active":"2021-02-10T04:38:33.233418+0000","last_peered":"2021-02-10T04:38:33.233418+0000","last_clean":"2021-02-10T04:38:33.233418+0000","last_became_active":"2021-02-10T04:25:36.336893+0000","last_became_peered":"2021-02-10T04:25:36.336893+0000","last_unstale":"2021-02-10T04:38:33.233418+0000","last_undegraded":"2021-02-10T04:38:33.233418+0000","last_fullsized":"2021-02-10T04:38:33.233418+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:30.415134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:26:46.425288+0000","last_clean_scrub_stamp":"2021-02-10T04:28:30.415134+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"663","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.236904+0000","last_change":"2021-02-10T04:37:37.243679+0000","last_active":"2021-02-10T04:38:30.236904+0000","last_peered":"2021-02-10T04:38:30.236904+0000","last_clean":"2021-02-10T04:38:30.236904+0000","last_became_active":"2021-02-10T04:37:37.243275+0000","last_became_peered":"2021-02-10T04:37:37.243275+0000","last_unstale":"2021-02-10T04:38:30.236904+0000","last_undegraded":"2021-02-10T04:38:30.236904+0000","last_fullsized":"2021-02-10T04:38:30.236904+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:52.643099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:33:52.643099+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"654","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.233407+0000","last_change":"2021-02-10T04:29:26.355726+0000","last_active":"2021-02-10T04:38:33.233407+0000","last_peered":"2021-02-10T04:38:33.233407+0000","last_clean":"2021-02-10T04:38:33.233407+0000","last_became_active":"2021-02-10T04:25:53.147831+0000","last_became_peered":"2021-02-10T04:25:53.147831+0000","last_unstale":"2021-02-10T04:38:33.233407+0000","last_undegraded":"2021-02-10T04:38:33.233407+0000","last_fullsized":"2021-02-10T04:38:33.233407+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:26.355652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.355652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"538'31","reported_seq":"719","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.236829+0000","last_change":"2021-02-10T04:28:32.565457+0000","last_active":"2021-02-10T04:38:30.236829+0000","last_peered":"2021-02-10T04:38:30.236829+0000","last_clean":"2021-02-10T04:38:30.236829+0000","last_became_active":"2021-02-10T04:25:49.000911+0000","last_became_peered":"2021-02-10T04:25:49.000911+0000","last_unstale":"2021-02-10T04:38:30.236829+0000","last_undegraded":"2021-02-10T04:38:30.236829+0000","last_fullsized":"2021-02-10T04:38:30.236829+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2021-02-10T04:28:32.565374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:32.565374+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":62,"num_read_kb":46,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"589","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.244974+0000","last_change":"2021-02-10T04:37:37.242121+0000","last_active":"2021-02-10T04:38:31.244974+0000","last_peered":"2021-02-10T04:38:31.244974+0000","last_clean":"2021-02-10T04:38:31.244974+0000","last_became_active":"2021-02-10T04:37:37.241751+0000","last_became_peered":"2021-02-10T04:37:37.241751+0000","last_unstale":"2021-02-10T04:38:31.244974+0000","last_undegraded":"2021-02-10T04:38:31.244974+0000","last_fullsized":"2021-02-10T04:38:31.244974+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:48.646802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:33:48.646802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"655","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.244219+0000","last_change":"2021-02-10T04:28:46.464172+0000","last_active":"2021-02-10T04:38:31.244219+0000","last_peered":"2021-02-10T04:38:31.244219+0000","last_clean":"2021-02-10T04:38:31.244219+0000","last_became_active":"2021-02-10T04:25:51.025488+0000","last_became_peered":"2021-02-10T04:25:51.025488+0000","last_unstale":"2021-02-10T04:38:31.244219+0000","last_undegraded":"2021-02-10T04:38:31.244219+0000","last_fullsized":"2021-02-10T04:38:31.244219+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:46.464108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:46.464108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"655","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.226104+0000","last_change":"2021-02-10T04:28:44.719453+0000","last_active":"2021-02-10T04:38:33.226104+0000","last_peered":"2021-02-10T04:38:33.226104+0000","last_clean":"2021-02-10T04:38:33.226104+0000","last_became_active":"2021-02-10T04:25:53.039844+0000","last_became_peered":"2021-02-10T04:25:53.039844+0000","last_unstale":"2021-02-10T04:38:33.226104+0000","last_undegraded":"2021-02-10T04:38:33.226104+0000","last_fullsized":"2021-02-10T04:38:33.226104+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:44.719363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:44.719363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"538'14","reported_seq":"686","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.236113+0000","last_change":"2021-02-10T04:30:13.089576+0000","last_active":"2021-02-10T04:38:30.236113+0000","last_peered":"2021-02-10T04:38:30.236113+0000","last_clean":"2021-02-10T04:38:30.236113+0000","last_became_active":"2021-02-10T04:25:49.013196+0000","last_became_peered":"2021-02-10T04:25:49.013196+0000","last_unstale":"2021-02-10T04:38:30.236113+0000","last_undegraded":"2021-02-10T04:38:30.236113+0000","last_fullsized":"2021-02-10T04:38:30.236113+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2021-02-10T04:30:13.089494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:30:13.089494+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":15,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"581","reported_epoch":"694","state":"active+clean","last_fresh":"2021-02-10T04:38:32.244901+0000","last_change":"2021-02-10T04:30:11.537208+0000","last_active":"2021-02-10T04:38:32.244901+0000","last_peered":"2021-02-10T04:38:32.244901+0000","last_clean":"2021-02-10T04:38:32.244901+0000","last_became_active":"2021-02-10T04:29:07.157944+0000","last_became_peered":"2021-02-10T04:29:07.157944+0000","last_unstale":"2021-02-10T04:38:32.244901+0000","last_undegraded":"2021-02-10T04:38:32.244901+0000","last_fullsized":"2021-02-10T04:38:32.244901+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:11.537136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.537136+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"574","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.236973+0000","last_change":"2021-02-10T04:29:07.146984+0000","last_active":"2021-02-10T04:38:30.236973+0000","last_peered":"2021-02-10T04:38:30.236973+0000","last_clean":"2021-02-10T04:38:30.236973+0000","last_became_active":"2021-02-10T04:29:07.146525+0000","last_became_peered":"2021-02-10T04:29:07.146525+0000","last_unstale":"2021-02-10T04:38:30.236973+0000","last_undegraded":"2021-02-10T04:38:30.236973+0000","last_fullsized":"2021-02-10T04:38:30.236973+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:46.493066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:46.493066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"653","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.243912+0000","last_change":"2021-02-10T04:29:22.475613+0000","last_active":"2021-02-10T04:38:31.243912+0000","last_peered":"2021-02-10T04:38:31.243912+0000","last_clean":"2021-02-10T04:38:31.243912+0000","last_became_active":"2021-02-10T04:25:53.042601+0000","last_became_peered":"2021-02-10T04:25:53.042601+0000","last_unstale":"2021-02-10T04:38:31.243912+0000","last_undegraded":"2021-02-10T04:38:31.243912+0000","last_fullsized":"2021-02-10T04:38:31.243912+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:22.475504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:22.475504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"538'38","reported_seq":"719","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.235856+0000","last_change":"2021-02-10T04:28:35.520489+0000","last_active":"2021-02-10T04:38:30.235856+0000","last_peered":"2021-02-10T04:38:30.235856+0000","last_clean":"2021-02-10T04:38:30.235856+0000","last_became_active":"2021-02-10T04:25:49.010481+0000","last_became_peered":"2021-02-10T04:25:49.010481+0000","last_unstale":"2021-02-10T04:38:30.235856+0000","last_undegraded":"2021-02-10T04:38:30.235856+0000","last_fullsized":"2021-02-10T04:38:30.235856+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'32","last_scrub_stamp":"2021-02-10T04:28:35.520410+0000","last_deep_scrub":"78'32","last_deep_scrub_stamp":"2021-02-10T04:27:09.170428+0000","last_clean_scrub_stamp":"2021-02-10T04:28:35.520410+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":56,"num_read_kb":37,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"665","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.233010+0000","last_change":"2021-02-10T04:29:29.355356+0000","last_active":"2021-02-10T04:38:33.233010+0000","last_peered":"2021-02-10T04:38:33.233010+0000","last_clean":"2021-02-10T04:38:33.233010+0000","last_became_active":"2021-02-10T04:25:34.327162+0000","last_became_peered":"2021-02-10T04:25:34.327162+0000","last_unstale":"2021-02-10T04:38:33.233010+0000","last_undegraded":"2021-02-10T04:38:33.233010+0000","last_fullsized":"2021-02-10T04:38:33.233010+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.355257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.355257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"338","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.244101+0000","last_change":"2021-02-10T04:33:05.169577+0000","last_active":"2021-02-10T04:38:31.244101+0000","last_peered":"2021-02-10T04:38:31.244101+0000","last_clean":"2021-02-10T04:38:31.244101+0000","last_became_active":"2021-02-10T04:33:05.169159+0000","last_became_peered":"2021-02-10T04:33:05.169159+0000","last_unstale":"2021-02-10T04:38:31.244101+0000","last_undegraded":"2021-02-10T04:38:31.244101+0000","last_fullsized":"2021-02-10T04:38:31.244101+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:40.098239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:40.098239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"738","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.236735+0000","last_change":"2021-02-10T04:38:23.319098+0000","last_active":"2021-02-10T04:38:30.236735+0000","last_peered":"2021-02-10T04:38:30.236735+0000","last_clean":"2021-02-10T04:38:30.236735+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:38:30.236735+0000","last_undegraded":"2021-02-10T04:38:30.236735+0000","last_fullsized":"2021-02-10T04:38:30.236735+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:23.319023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:38:23.319023+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"538'56","reported_seq":"814","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.231994+0000","last_change":"2021-02-10T04:38:21.961539+0000","last_active":"2021-02-10T04:38:33.231994+0000","last_peered":"2021-02-10T04:38:33.231994+0000","last_clean":"2021-02-10T04:38:33.231994+0000","last_became_active":"2021-02-10T04:37:37.266384+0000","last_became_peered":"2021-02-10T04:37:37.266384+0000","last_unstale":"2021-02-10T04:38:33.231994+0000","last_undegraded":"2021-02-10T04:38:33.231994+0000","last_fullsized":"2021-02-10T04:38:33.231994+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":641,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'56","last_scrub_stamp":"2021-02-10T04:38:21.961470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:21.961470+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":107,"num_read_kb":84,"num_write":57,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":890,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"675","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.228149+0000","last_change":"2021-02-10T04:29:28.719237+0000","last_active":"2021-02-10T04:38:33.228149+0000","last_peered":"2021-02-10T04:38:33.228149+0000","last_clean":"2021-02-10T04:38:33.228149+0000","last_became_active":"2021-02-10T04:25:36.336112+0000","last_became_peered":"2021-02-10T04:25:36.336112+0000","last_unstale":"2021-02-10T04:38:33.228149+0000","last_undegraded":"2021-02-10T04:38:33.228149+0000","last_fullsized":"2021-02-10T04:38:33.228149+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.719147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.719147+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"649","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.236151+0000","last_change":"2021-02-10T04:27:49.628172+0000","last_active":"2021-02-10T04:38:30.236151+0000","last_peered":"2021-02-10T04:38:30.236151+0000","last_clean":"2021-02-10T04:38:30.236151+0000","last_became_active":"2021-02-10T04:25:51.029110+0000","last_became_peered":"2021-02-10T04:25:51.029110+0000","last_unstale":"2021-02-10T04:38:30.236151+0000","last_undegraded":"2021-02-10T04:38:30.236151+0000","last_fullsized":"2021-02-10T04:38:30.236151+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:27:49.628090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:27:49.628090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"654","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.233344+0000","last_change":"2021-02-10T04:29:25.382883+0000","last_active":"2021-02-10T04:38:33.233344+0000","last_peered":"2021-02-10T04:38:33.233344+0000","last_clean":"2021-02-10T04:38:33.233344+0000","last_became_active":"2021-02-10T04:25:53.150917+0000","last_became_peered":"2021-02-10T04:25:53.150917+0000","last_unstale":"2021-02-10T04:38:33.233344+0000","last_undegraded":"2021-02-10T04:38:33.233344+0000","last_fullsized":"2021-02-10T04:38:33.233344+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:25.382793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.382793+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.20","version":"0'0","reported_seq":"643","reported_epoch":"693","state":"active+clean","last_fresh":"2021-02-10T04:38:31.243777+0000","last_change":"2021-02-10T04:30:11.492843+0000","last_active":"2021-02-10T04:38:31.243777+0000","last_peered":"2021-02-10T04:38:31.243777+0000","last_clean":"2021-02-10T04:38:31.243777+0000","last_became_active":"2021-02-10T04:29:07.148937+0000","last_became_peered":"2021-02-10T04:29:07.148937+0000","last_unstale":"2021-02-10T04:38:31.243777+0000","last_undegraded":"2021-02-10T04:38:31.243777+0000","last_fullsized":"2021-02-10T04:38:31.243777+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:11.492788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:41.375276+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.492788+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":"158","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.226328+0000","last_change":"2021-02-10T04:37:37.247286+0000","last_active":"2021-02-10T04:38:33.226328+0000","last_peered":"2021-02-10T04:38:33.226328+0000","last_clean":"2021-02-10T04:38:33.226328+0000","last_became_active":"2021-02-10T04:37:37.246865+0000","last_became_peered":"2021-02-10T04:37:37.246865+0000","last_unstale":"2021-02-10T04:38:33.226328+0000","last_undegraded":"2021-02-10T04:38:33.226328+0000","last_fullsized":"2021-02-10T04:38:33.226328+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:21.708701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:21.708701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":"679","reported_epoch":"694","state":"active+clean","last_fresh":"2021-02-10T04:38:32.243627+0000","last_change":"2021-02-10T04:33:58.892594+0000","last_active":"2021-02-10T04:38:32.243627+0000","last_peered":"2021-02-10T04:38:32.243627+0000","last_clean":"2021-02-10T04:38:32.243627+0000","last_became_active":"2021-02-10T04:28:23.129832+0000","last_became_peered":"2021-02-10T04:28:23.129832+0000","last_unstale":"2021-02-10T04:38:32.243627+0000","last_undegraded":"2021-02-10T04:38:32.243627+0000","last_fullsized":"2021-02-10T04:38:32.243627+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:58.892504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:33:58.892504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":"619","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.233266+0000","last_change":"2021-02-10T04:28:48.409915+0000","last_active":"2021-02-10T04:38:33.233266+0000","last_peered":"2021-02-10T04:38:33.233266+0000","last_clean":"2021-02-10T04:38:33.233266+0000","last_became_active":"2021-02-10T04:28:23.129523+0000","last_became_peered":"2021-02-10T04:28:23.129523+0000","last_unstale":"2021-02-10T04:38:33.233266+0000","last_undegraded":"2021-02-10T04:38:33.233266+0000","last_fullsized":"2021-02-10T04:38:33.233266+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:48.409826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:48.409826+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":"623","reported_epoch":"694","state":"active+clean","last_fresh":"2021-02-10T04:38:32.244713+0000","last_change":"2021-02-10T04:34:01.895599+0000","last_active":"2021-02-10T04:38:32.244713+0000","last_peered":"2021-02-10T04:38:32.244713+0000","last_clean":"2021-02-10T04:38:32.244713+0000","last_became_active":"2021-02-10T04:28:23.126368+0000","last_became_peered":"2021-02-10T04:28:23.126368+0000","last_unstale":"2021-02-10T04:38:32.244713+0000","last_undegraded":"2021-02-10T04:38:32.244713+0000","last_fullsized":"2021-02-10T04:38:32.244713+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:34:01.895514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:34:01.895514+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.25","version":"53'2","reported_seq":"634","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.236724+0000","last_change":"2021-02-10T04:28:40.422198+0000","last_active":"2021-02-10T04:38:30.236724+0000","last_peered":"2021-02-10T04:38:30.236724+0000","last_clean":"2021-02-10T04:38:30.236724+0000","last_became_active":"2021-02-10T04:28:23.467770+0000","last_became_peered":"2021-02-10T04:28:23.467770+0000","last_unstale":"2021-02-10T04:38:30.236724+0000","last_undegraded":"2021-02-10T04:38:30.236724+0000","last_fullsized":"2021-02-10T04:38:30.236724+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:28:40.422115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:40.422115+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.26","version":"53'1","reported_seq":"79","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.237259+0000","last_change":"2021-02-10T04:37:37.233712+0000","last_active":"2021-02-10T04:38:30.237259+0000","last_peered":"2021-02-10T04:38:30.237259+0000","last_clean":"2021-02-10T04:38:30.237259+0000","last_became_active":"2021-02-10T04:37:37.231081+0000","last_became_peered":"2021-02-10T04:37:37.231081+0000","last_unstale":"2021-02-10T04:38:30.237259+0000","last_undegraded":"2021-02-10T04:38:30.237259+0000","last_fullsized":"2021-02-10T04:38:30.237259+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:29:22.801929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:22.801929+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":423,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":"645","reported_epoch":"692","state":"active+clean","last_fresh":"2021-02-10T04:38:30.238216+0000","last_change":"2021-02-10T04:29:07.146256+0000","last_active":"2021-02-10T04:38:30.238216+0000","last_peered":"2021-02-10T04:38:30.238216+0000","last_clean":"2021-02-10T04:38:30.238216+0000","last_became_active":"2021-02-10T04:29:07.145907+0000","last_became_peered":"2021-02-10T04:29:07.145907+0000","last_unstale":"2021-02-10T04:38:30.238216+0000","last_undegraded":"2021-02-10T04:38:30.238216+0000","last_fullsized":"2021-02-10T04:38:30.238216+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:39.388884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:39.388884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":"619","reported_epoch":"695","state":"active+clean","last_fresh":"2021-02-10T04:38:33.235112+0000","last_change":"2021-02-10T04:29:23.440451+0000","last_active":"2021-02-10T04:38:33.235112+0000","last_peered":"2021-02-10T04:38:33.235112+0000","last_clean":"2021-02-10T04:38:33.235112+0000","last_became_active":"2021-02-10T04:28:23.129469+0000","last_became_peered":"2021-02-10T04:28:23.129469+0000","last_unstale":"2021-02-10T04:38:33.235112+0000","last_undegraded":"2021-02-10T04:38:33.235112+0000","last_fullsized":"2021-02-10T04:38:33.235112+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.440391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.440391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":"623","reported_epoch":"694","state":"active+clean","last_fresh":"2021-02-10T04:38:32.242904+0000","last_change":"2021-02-10T04:38:22.136377+0000","last_active":"2021-02-10T04:38:32.242904+0000","last_peered":"2021-02-10T04:38:32.242904+0000","last_clean":"2021-02-10T04:38:32.242904+0000","last_became_active":"2021-02-10T04:28:23.126795+0000","last_became_peered":"2021-02-10T04:28:23.126795+0000","last_unstale":"2021-02-10T04:38:32.242904+0000","last_undegraded":"2021-02-10T04:38:32.242904+0000","last_fullsized":"2021-02-10T04:38:32.242904+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:22.136290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_clean_scrub_stamp":"2021-02-10T04:38:22.136290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":412025354,"num_objects":192,"num_object_clones":143,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":14,"num_read":4992,"num_read_kb":5057631,"num_write":3775,"num_write_kb":1419315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":291,"num_bytes_recovered":108939670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":935747584,"data_stored":1247207361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9570,"ondisk_log_size":9570,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":1945,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":18,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":6565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":126,"acting":126,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":63,"num_read_kb":58,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":1876,"num_read_kb":1349,"num_write":1020,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":7184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":75722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1068,"ondisk_log_size":1068,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21,"ondisk_log_size":21,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748364958,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":109672,"kb_used_data":84128,"kb_used_omap":0,"kb_used_meta":25536,"kb_avail":93635480,"statfs":{"total":95995035648,"available":95882731520,"internally_reserved":0,"allocated":86147072,"data_stored":113967184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26148864},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:37:55 2021","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":1.698,"15min":1.214},"min":{"1min":0.58399999999999996,"5min":0.495,"15min":0.441},"max":{"1min":1.677,"5min":72.805999999999997,"15min":72.805999999999997},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.8169999999999999,"15min":1.3009999999999999},"min":{"1min":0.61199999999999999,"5min":0.52000000000000002,"15min":0.50800000000000001},"max":{"1min":6.4269999999999996,"5min":72.721999999999994,"15min":72.721999999999994},"last":2.5139999999999998}]},{"osd":1,"last update":"Wed Feb 10 04:37:55 2021","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":0.97899999999999998,"15min":1.077},"min":{"1min":0.58099999999999996,"5min":0.54600000000000004,"15min":0.46700000000000003},"max":{"1min":2.4750000000000001,"5min":6.1239999999999997,"15min":41.966000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":1.1040000000000001,"5min":0.96799999999999997,"15min":1.135},"min":{"1min":0.59399999999999997,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.714,"5min":3.5750000000000002,"15min":41.804000000000002},"last":0.98899999999999999}]},{"osd":2,"last update":"Wed Feb 10 04:37:55 2021","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.927,"15min":1.8720000000000001},"min":{"1min":0.72099999999999997,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.554,"5min":48.174999999999997,"15min":125.714},"last":1.196},{"interface":"front","average":{"1min":1.347,"5min":2.0960000000000001,"15min":1.9370000000000001},"min":{"1min":0.88300000000000001,"5min":0.64700000000000002,"15min":0.59099999999999997},"max":{"1min":2.1629999999999998,"5min":48.277999999999999,"15min":125.53700000000001},"last":3.883}]},{"osd":3,"last update":"Wed Feb 10 04:37:55 2021","interfaces":[{"interface":"back","average":{"1min":1.581,"5min":1.6879999999999999,"15min":2.5510000000000002},"min":{"1min":0.79100000000000004,"5min":0.72399999999999998,"15min":0.63600000000000001},"max":{"1min":5.3819999999999997,"5min":35.091000000000001,"15min":192.95500000000001},"last":2.3319999999999999},{"interface":"front","average":{"1min":1.302,"5min":1.659,"15min":2.6080000000000001},"min":{"1min":0.97199999999999998,"5min":0.72199999999999998,"15min":0.69999999999999996},"max":{"1min":1.8180000000000001,"5min":35.061999999999998,"15min":192.601},"last":1.1000000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:37:55 2021","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":2.1949999999999998,"15min":1.476},"min":{"1min":0.72199999999999998,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":6.0209999999999999,"5min":97.733999999999995,"15min":97.733999999999995},"last":4.7539999999999996},{"interface":"front","average":{"1min":1.782,"5min":2.2320000000000002,"15min":1.5369999999999999},"min":{"1min":0.76900000000000002,"5min":0.69599999999999995,"15min":0.64600000000000002},"max":{"1min":5.5789999999999997,"5min":97.691000000000003,"15min":97.691000000000003},"last":3.0270000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:37:55 2021","interfaces":[{"interface":"back","average":{"1min":1.867,"5min":1.4610000000000001,"15min":1.399},"min":{"1min":0.82499999999999996,"5min":0.81299999999999994,"15min":0.624},"max":{"1min":7.0010000000000003,"5min":7.0010000000000003,"15min":32.713999999999999},"last":1.2969999999999999},{"interface":"front","average":{"1min":2.681,"5min":1.756,"15min":1.552},"min":{"1min":0.98399999999999999,"5min":0.85299999999999998,"15min":0.753},"max":{"1min":6.9740000000000002,"5min":6.9740000000000002,"15min":32.984000000000002},"last":3.992}]},{"osd":6,"last update":"Wed Feb 10 04:37:55 2021","interfaces":[{"interface":"back","average":{"1min":2.258,"5min":1.712,"15min":1.7130000000000001},"min":{"1min":1.234,"5min":0.96199999999999997,"15min":0.86899999999999999},"max":{"1min":6.5439999999999996,"5min":6.5439999999999996,"15min":32.616},"last":4.0940000000000003},{"interface":"front","average":{"1min":2.0800000000000001,"5min":1.6479999999999999,"15min":1.6559999999999999},"min":{"1min":0.997,"5min":0.79500000000000004,"15min":0.72499999999999998},"max":{"1min":5.9089999999999998,"5min":6.3730000000000002,"15min":32.424999999999997},"last":2.9199999999999999}]}]},{"osd":6,"up_from":43,"seq":184683593890,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":182976,"kb_used_data":147192,"kb_used_omap":0,"kb_used_meta":35776,"kb_avail":93562176,"statfs":{"total":95995035648,"available":95807668224,"internally_reserved":0,"allocated":150724608,"data_stored":190034036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36634624},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:37:42 2021","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.3640000000000001,"15min":1.4019999999999999},"min":{"1min":0.44700000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":5.1890000000000001,"5min":40.652000000000001,"15min":64.164000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":3.7109999999999999,"5min":2.3980000000000001,"15min":1.7969999999999999},"min":{"1min":0.53800000000000003,"5min":0.503,"15min":0.47399999999999998},"max":{"1min":28.167000000000002,"5min":51.039999999999999,"15min":64.277000000000001},"last":1.9179999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:37:42 2021","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.98199999999999998,"15min":1.099},"min":{"1min":0.56499999999999995,"5min":0.443,"15min":0.443},"max":{"1min":1.79,"5min":8.641,"15min":18.177},"last":0.78700000000000003},{"interface":"front","average":{"1min":1.238,"5min":1.0920000000000001,"15min":1.107},"min":{"1min":0.65200000000000002,"5min":0.59799999999999998,"15min":0.52700000000000002},"max":{"1min":6.0960000000000001,"5min":6.0960000000000001,"15min":18.898},"last":3.3239999999999998}]},{"osd":2,"last update":"Wed Feb 10 04:37:42 2021","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.474,"15min":1.272},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":7.8140000000000001,"5min":24.725000000000001,"15min":24.725000000000001},"last":2.101},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.415,"15min":1.254},"min":{"1min":0.77700000000000002,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":3.343,"5min":24.611999999999998,"15min":24.611999999999998},"last":1.1379999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:37:42 2021","interfaces":[{"interface":"back","average":{"1min":2.0990000000000002,"5min":1.516,"15min":1.3720000000000001},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":7.4130000000000003,"5min":7.4130000000000003,"15min":15.316000000000001},"last":2.2170000000000001},{"interface":"front","average":{"1min":2.2559999999999998,"5min":1.494,"15min":1.361},"min":{"1min":1.03,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":11.968,"5min":11.968,"15min":15.35},"last":1.3660000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:37:42 2021","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":1.3360000000000001,"15min":1.214},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.58999999999999997},"max":{"1min":7.5389999999999997,"5min":7.5389999999999997,"15min":7.5389999999999997},"last":2.3370000000000002},{"interface":"front","average":{"1min":1.038,"5min":1.024,"15min":0.97999999999999998},"min":{"1min":0.68899999999999995,"5min":0.64900000000000002,"15min":0.57999999999999996},"max":{"1min":2.4020000000000001,"5min":2.4020000000000001,"15min":2.4020000000000001},"last":3.2869999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:37:42 2021","interfaces":[{"interface":"back","average":{"1min":1.946,"5min":1.3779999999999999,"15min":1.1879999999999999},"min":{"1min":0.71599999999999997,"5min":0.71199999999999997,"15min":0.64100000000000001},"max":{"1min":11.380000000000001,"5min":11.380000000000001,"15min":11.380000000000001},"last":4.6369999999999996},{"interface":"front","average":{"1min":2.2549999999999999,"5min":1.5269999999999999,"15min":1.369},"min":{"1min":0.90200000000000002,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":11.538,"5min":11.538,"15min":11.538},"last":2.1669999999999998}]},{"osd":7,"last update":"Wed Feb 10 04:38:09 2021","interfaces":[{"interface":"back","average":{"1min":1.589,"5min":1.3460000000000001,"15min":1.236},"min":{"1min":0.84999999999999998,"5min":0.77800000000000002,"15min":0.73599999999999999},"max":{"1min":3.6989999999999998,"5min":4.2599999999999998,"15min":4.2599999999999998},"last":4.5279999999999996},{"interface":"front","average":{"1min":1.845,"5min":1.5600000000000001,"15min":1.496},"min":{"1min":1.006,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":3.8149999999999999,"5min":4.7699999999999996,"15min":4.7699999999999996},"last":1.542}]}]},{"osd":1,"up_from":13,"seq":55834575027,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":198764,"kb_used_data":162212,"kb_used_omap":0,"kb_used_meta":36544,"kb_avail":93546388,"statfs":{"total":95995035648,"available":95791501312,"internally_reserved":0,"allocated":166105088,"data_stored":215550792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37421056},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:38:14 2021","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.591,"15min":1.1699999999999999},"min":{"1min":0.55600000000000005,"5min":0.51400000000000001,"15min":0.36399999999999999},"max":{"1min":4.8360000000000003,"5min":40.539999999999999,"15min":40.539999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.722,"15min":1.26},"min":{"1min":0.56499999999999995,"5min":0.53200000000000003,"15min":0.41199999999999998},"max":{"1min":3.8580000000000001,"5min":40.831000000000003,"15min":40.831000000000003},"last":1.72}]},{"osd":2,"last update":"Wed Feb 10 04:37:34 2021","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.534,"15min":1.1990000000000001},"min":{"1min":0.61799999999999999,"5min":0.53800000000000003,"15min":0.48699999999999999},"max":{"1min":1.8380000000000001,"5min":40.268000000000001,"15min":40.268000000000001},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.111,"5min":1.573,"15min":1.2470000000000001},"min":{"1min":0.498,"5min":0.498,"15min":0.46000000000000002},"max":{"1min":1.8759999999999999,"5min":40.651000000000003,"15min":40.651000000000003},"last":0.91100000000000003}]},{"osd":3,"last update":"Wed Feb 10 04:37:48 2021","interfaces":[{"interface":"back","average":{"1min":1.6819999999999999,"5min":1.8580000000000001,"15min":1.369},"min":{"1min":0.81100000000000005,"5min":0.64000000000000001,"15min":0.45200000000000001},"max":{"1min":6.2210000000000001,"5min":40.314,"15min":40.314},"last":1.0609999999999999},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.9219999999999999,"15min":1.393},"min":{"1min":0.66500000000000004,"5min":0.66200000000000003,"15min":0.47199999999999998},"max":{"1min":4.7670000000000003,"5min":40.973999999999997,"15min":40.973999999999997},"last":1.607}]},{"osd":4,"last update":"Wed Feb 10 04:38:14 2021","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.2509999999999999,"15min":1.222},"min":{"1min":0.79800000000000004,"5min":0.746,"15min":0.64000000000000001},"max":{"1min":2.1909999999999998,"5min":6.3099999999999996,"15min":6.3099999999999996},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3799999999999999,"15min":1.3009999999999999},"min":{"1min":0.97799999999999998,"5min":0.70999999999999996,"15min":0.65700000000000003},"max":{"1min":2.5939999999999999,"5min":6.218,"15min":6.2869999999999999},"last":1.8280000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:38:29 2021","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.9419999999999999,"15min":1.4870000000000001},"min":{"1min":0.73399999999999999,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":3.2040000000000002,"5min":48.686999999999998,"15min":48.686999999999998},"last":1.325},{"interface":"front","average":{"1min":1.5600000000000001,"5min":2.129,"15min":1.7190000000000001},"min":{"1min":0.93000000000000005,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.9420000000000002,"5min":48.905000000000001,"15min":48.905000000000001},"last":1.2350000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:37:40 2021","interfaces":[{"interface":"back","average":{"1min":3.7650000000000001,"5min":2.0369999999999999,"15min":1.5780000000000001},"min":{"1min":1.097,"5min":0.83099999999999996,"15min":0.80300000000000005},"max":{"1min":48.878999999999998,"5min":48.878999999999998,"15min":48.878999999999998},"last":1.9059999999999999},{"interface":"front","average":{"1min":3.6110000000000002,"5min":1.931,"15min":1.512},"min":{"1min":0.93300000000000005,"5min":0.80100000000000005,"15min":0.78500000000000003},"max":{"1min":49.011000000000003,"5min":49.011000000000003,"15min":49.011000000000003},"last":1.4830000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:37:48 2021","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.5189999999999999,"15min":1.5569999999999999},"min":{"1min":1.0469999999999999,"5min":1.026,"15min":0.83499999999999996},"max":{"1min":2.2930000000000001,"5min":3.6840000000000002,"15min":7.5540000000000003},"last":1.369},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.615,"15min":1.5720000000000001},"min":{"1min":1.2010000000000001,"5min":0.94899999999999995,"15min":0.94899999999999995},"max":{"1min":2.8149999999999999,"5min":6.1689999999999996,"15min":8.1020000000000003},"last":2.0230000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738553,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":196332,"kb_used_data":158436,"kb_used_omap":0,"kb_used_meta":37888,"kb_avail":93548820,"statfs":{"total":95995035648,"available":95793991680,"internally_reserved":0,"allocated":162238464,"data_stored":225131739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":38797312},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 04:38:08 2021","interfaces":[{"interface":"back","average":{"1min":2.274,"5min":1.381,"15min":1.006},"min":{"1min":0.53800000000000003,"5min":0.48199999999999998,"15min":0.40799999999999997},"max":{"1min":28.946999999999999,"5min":28.946999999999999,"15min":28.946999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":1.0029999999999999,"5min":0.97099999999999997,"15min":0.92000000000000004},"min":{"1min":0.63600000000000001,"5min":0.58099999999999996,"15min":0.45000000000000001},"max":{"1min":1.8919999999999999,"5min":1.9099999999999999,"15min":7.5549999999999997},"last":0.755}]},{"osd":2,"last update":"Wed Feb 10 04:38:34 2021","interfaces":[{"interface":"back","average":{"1min":2.4710000000000001,"5min":1.657,"15min":1.204},"min":{"1min":0.59599999999999997,"5min":0.55500000000000005,"15min":0.48099999999999998},"max":{"1min":25.984999999999999,"5min":25.984999999999999,"15min":25.984999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.331,"5min":1.2909999999999999,"15min":1.044},"min":{"1min":0.624,"5min":0.52800000000000002,"15min":0.495},"max":{"1min":5.4930000000000003,"5min":16.829000000000001,"15min":16.829000000000001},"last":0.98899999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:37:51 2021","interfaces":[{"interface":"back","average":{"1min":2.5329999999999999,"5min":1.542,"15min":1.599},"min":{"1min":0.63900000000000001,"5min":0.55200000000000005,"15min":0.46999999999999997},"max":{"1min":24.565000000000001,"5min":24.565000000000001,"15min":126.587},"last":1.506},{"interface":"front","average":{"1min":3.2839999999999998,"5min":2.0339999999999998,"15min":1.8919999999999999},"min":{"1min":0.90100000000000002,"5min":0.68600000000000005,"15min":0.54200000000000004},"max":{"1min":29.814,"5min":29.814,"15min":126.88200000000001},"last":1.206}]},{"osd":4,"last update":"Wed Feb 10 04:38:08 2021","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.3260000000000001,"15min":1.2170000000000001},"min":{"1min":0.83299999999999996,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":2.782,"5min":2.782,"15min":2.782},"last":0.92300000000000004},{"interface":"front","average":{"1min":1.843,"5min":1.3540000000000001,"15min":1.296},"min":{"1min":0.88300000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":14.672000000000001,"5min":14.672000000000001,"15min":14.672000000000001},"last":1.1619999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:38:24 2021","interfaces":[{"interface":"back","average":{"1min":2.0299999999999998,"5min":1.5349999999999999,"15min":1.3979999999999999},"min":{"1min":0.83999999999999997,"5min":0.76300000000000001,"15min":0.69599999999999995},"max":{"1min":13.808999999999999,"5min":13.808999999999999,"15min":13.949999999999999},"last":1.048},{"interface":"front","average":{"1min":2.9079999999999999,"5min":2.0699999999999998,"15min":1.631},"min":{"1min":0.93899999999999995,"5min":0.93300000000000005,"15min":0.81699999999999995},"max":{"1min":28.827999999999999,"5min":28.827999999999999,"15min":28.827999999999999},"last":1.446}]},{"osd":6,"last update":"Wed Feb 10 04:37:43 2021","interfaces":[{"interface":"back","average":{"1min":2.4289999999999998,"5min":2.1320000000000001,"15min":1.6719999999999999},"min":{"1min":1.101,"5min":0.88800000000000001,"15min":0.88800000000000001},"max":{"1min":16.861999999999998,"5min":35.872999999999998,"15min":35.872999999999998},"last":1.6919999999999999},{"interface":"front","average":{"1min":3.468,"5min":2.5710000000000002,"15min":1.8109999999999999},"min":{"1min":0.96299999999999997,"5min":0.88200000000000001,"15min":0.83099999999999996},"max":{"1min":29.696000000000002,"5min":35.911000000000001,"15min":35.911000000000001},"last":1.5580000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:37:51 2021","interfaces":[{"interface":"back","average":{"1min":1.831,"5min":1.5649999999999999,"15min":1.3859999999999999},"min":{"1min":1.1140000000000001,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":5.6050000000000004,"5min":5.6050000000000004,"15min":5.6050000000000004},"last":1.222},{"interface":"front","average":{"1min":2.3109999999999999,"5min":1.8839999999999999,"15min":1.6419999999999999},"min":{"1min":1.214,"5min":1.006,"15min":0.94999999999999996},"max":{"1min":13.968999999999999,"5min":16.545000000000002,"15min":16.545000000000002},"last":1.8169999999999999}]}]},{"osd":2,"up_from":0,"seq":77309411506,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":25,"seq":107374182572,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":238232,"kb_used_data":196112,"kb_used_omap":0,"kb_used_meta":42112,"kb_avail":93506920,"statfs":{"total":95995035648,"available":95751086080,"internally_reserved":0,"allocated":200818688,"data_stored":265315898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43122688},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:37:58 2021","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.0529999999999999,"15min":0.96999999999999997},"min":{"1min":0.502,"5min":0.502,"15min":0.46100000000000002},"max":{"1min":3.4649999999999999,"5min":4.5979999999999999,"15min":17.233000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.1220000000000001,"15min":1.1200000000000001},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52300000000000002},"max":{"1min":2.3980000000000001,"5min":3.423,"15min":20.216999999999999},"last":0.64300000000000002}]},{"osd":1,"last update":"Wed Feb 10 04:37:58 2021","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.1719999999999999,"15min":1.0489999999999999},"min":{"1min":0.73099999999999998,"5min":0.65700000000000003,"15min":0.52800000000000002},"max":{"1min":8.9190000000000005,"5min":8.9190000000000005,"15min":8.9190000000000005},"last":0.753},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.044,"15min":0.96699999999999997},"min":{"1min":0.65000000000000002,"5min":0.56299999999999994,"15min":0.52100000000000002},"max":{"1min":4.6769999999999996,"5min":4.6769999999999996,"15min":5.4039999999999999},"last":0.57699999999999996}]},{"osd":2,"last update":"Wed Feb 10 04:37:58 2021","interfaces":[{"interface":"back","average":{"1min":1.5309999999999999,"5min":1.2689999999999999,"15min":1.131},"min":{"1min":0.70299999999999996,"5min":0.63500000000000001,"15min":0.56799999999999995},"max":{"1min":5.024,"5min":5.024,"15min":5.024},"last":0.80000000000000004},{"interface":"front","average":{"1min":1.399,"5min":1.27,"15min":1.087},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.56999999999999995},"max":{"1min":3.0670000000000002,"5min":3.0779999999999998,"15min":3.0779999999999998},"last":0.92100000000000004}]},{"osd":4,"last update":"Wed Feb 10 04:38:03 2021","interfaces":[{"interface":"back","average":{"1min":1.5409999999999999,"5min":1.6910000000000001,"15min":1.538},"min":{"1min":0.85099999999999998,"5min":0.80600000000000005,"15min":0.629},"max":{"1min":2.8199999999999998,"5min":30.751999999999999,"15min":30.751999999999999},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.7569999999999999,"15min":1.573},"min":{"1min":1.0509999999999999,"5min":0.78600000000000003,"15min":0.65700000000000003},"max":{"1min":2.9590000000000001,"5min":30.928000000000001,"15min":30.928000000000001},"last":0.97799999999999998}]},{"osd":5,"last update":"Wed Feb 10 04:38:27 2021","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.804,"15min":1.7689999999999999},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.77400000000000002},"max":{"1min":2.9569999999999999,"5min":17.701000000000001,"15min":35.616999999999997},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.6339999999999999,"5min":1.9379999999999999,"15min":1.8200000000000001},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.77200000000000002},"max":{"1min":2.6429999999999998,"5min":17.361999999999998,"15min":35.661999999999999},"last":0.90700000000000003}]},{"osd":6,"last update":"Wed Feb 10 04:37:45 2021","interfaces":[{"interface":"back","average":{"1min":1.8120000000000001,"5min":2.427,"15min":1.9359999999999999},"min":{"1min":1.1619999999999999,"5min":0.96799999999999997,"15min":0.83699999999999997},"max":{"1min":3.1829999999999998,"5min":35.518000000000001,"15min":35.518000000000001},"last":1.7350000000000001},{"interface":"front","average":{"1min":1.9850000000000001,"5min":3.044,"15min":2.1320000000000001},"min":{"1min":1.2390000000000001,"5min":0.92400000000000004,"15min":0.86599999999999999},"max":{"1min":5.3730000000000002,"5min":53.965000000000003,"15min":53.965000000000003},"last":1.111}]},{"osd":7,"last update":"Wed Feb 10 04:37:58 2021","interfaces":[{"interface":"back","average":{"1min":1.905,"5min":1.7629999999999999,"15min":1.7589999999999999},"min":{"1min":1.143,"5min":1.0609999999999999,"15min":0.99199999999999999},"max":{"1min":3.2999999999999998,"5min":4.3460000000000001,"15min":25.242999999999999},"last":1.3939999999999999},{"interface":"front","average":{"1min":2.0760000000000001,"5min":1.851,"15min":1.806},"min":{"1min":1.24,"5min":0.98299999999999998,"15min":0.98299999999999998},"max":{"1min":5.2549999999999999,"5min":5.2549999999999999,"15min":25.286999999999999},"last":1.048}]}]},{"osd":4,"up_from":31,"seq":133143986345,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87588,"kb_used_data":62236,"kb_used_omap":0,"kb_used_meta":25344,"kb_avail":93657564,"statfs":{"total":95995035648,"available":95905345536,"internally_reserved":0,"allocated":63729664,"data_stored":75421097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25952256},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4],"upper_bound":2097152},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:38:04 2021","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.93799999999999994,"15min":1.1419999999999999},"min":{"1min":0.53100000000000003,"5min":0.502,"15min":0.47099999999999997},"max":{"1min":1.615,"5min":10.946,"15min":62.273000000000003},"last":43.939999999999998},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.94599999999999995,"15min":1.1799999999999999},"min":{"1min":0.61499999999999999,"5min":0.499,"15min":0.499},"max":{"1min":1.5609999999999999,"5min":10.919,"15min":62.375999999999998},"last":43.829999999999998}]},{"osd":1,"last update":"Wed Feb 10 04:38:04 2021","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.77400000000000002,"15min":0.85799999999999998},"min":{"1min":0.53100000000000003,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.913,"5min":3.3519999999999999,"15min":5.6699999999999999},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.028,"15min":1.038},"min":{"1min":0.752,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.8100000000000001,"5min":2.9830000000000001,"15min":2.9830000000000001},"last":1.3400000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:38:04 2021","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.081,"15min":1.367},"min":{"1min":0.64700000000000002,"5min":0.61299999999999999,"15min":0.56499999999999995},"max":{"1min":1.929,"5min":2.8130000000000002,"15min":62.521000000000001},"last":1.2},{"interface":"front","average":{"1min":1.133,"5min":1.1479999999999999,"15min":1.4610000000000001},"min":{"1min":0.73499999999999999,"5min":0.68100000000000005,"15min":0.60699999999999998},"max":{"1min":1.698,"5min":3.1640000000000001,"15min":62.572000000000003},"last":0.99099999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:38:04 2021","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.24,"15min":1.4379999999999999},"min":{"1min":0.93000000000000005,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":2.0289999999999999,"5min":3.6739999999999999,"15min":62.113},"last":1.5940000000000001},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.218,"15min":1.4079999999999999},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.69899999999999995},"max":{"1min":2.7160000000000002,"5min":3.9049999999999998,"15min":62.146000000000001},"last":1.2669999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:38:22 2021","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.472,"15min":1.224},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.57499999999999996},"max":{"1min":2.0659999999999998,"5min":12.773999999999999,"15min":12.773999999999999},"last":1.708},{"interface":"front","average":{"1min":1.173,"5min":1.3440000000000001,"15min":1.1539999999999999},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.624},"max":{"1min":2,"5min":13.038,"15min":13.038},"last":1.665}]},{"osd":6,"last update":"Wed Feb 10 04:37:38 2021","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.625,"15min":1.4730000000000001},"min":{"1min":0.96799999999999997,"5min":0.70299999999999996,"15min":0.63100000000000001},"max":{"1min":2.423,"5min":20.001000000000001,"15min":29.686},"last":2.0990000000000002},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.7649999999999999,"15min":1.621},"min":{"1min":1.0700000000000001,"5min":0.83899999999999997,"15min":0.79000000000000004},"max":{"1min":3.0489999999999999,"5min":20.292999999999999,"15min":29.510000000000002},"last":1.8899999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:37:58 2021","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.4350000000000001,"15min":1.365},"min":{"1min":1.002,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.998,"5min":6.3440000000000003,"15min":6.3440000000000003},"last":1.827},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.4870000000000001,"15min":1.466},"min":{"1min":1.1220000000000001,"5min":0.88700000000000001,"15min":0.83099999999999996},"max":{"1min":2.0259999999999998,"5min":4.0780000000000003,"15min":4.0780000000000003},"last":2.1549999999999998}]}]},{"osd":5,"up_from":37,"seq":158913790118,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":108520,"kb_used_data":81312,"kb_used_omap":0,"kb_used_meta":27200,"kb_avail":93636632,"statfs":{"total":95995035648,"available":95883911168,"internally_reserved":0,"allocated":83263488,"data_stored":114986780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27852800},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:38:22 2021","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":0.95499999999999996,"15min":1.391},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.40500000000000003},"max":{"1min":2.5550000000000002,"5min":2.5550000000000002,"15min":104.04300000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":1.069,"5min":0.92700000000000005,"15min":1.385},"min":{"1min":0.495,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":107.776},"last":1.208}]},{"osd":1,"last update":"Wed Feb 10 04:38:22 2021","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.86499999999999999,"15min":1.2869999999999999},"min":{"1min":0.56000000000000005,"5min":0.504,"15min":0.504},"max":{"1min":1.97,"5min":3.1269999999999998,"15min":103.904},"last":0.60999999999999999},{"interface":"front","average":{"1min":1.081,"5min":1.0109999999999999,"15min":1.409},"min":{"1min":0.67100000000000004,"5min":0.55300000000000005,"15min":0.50800000000000001},"max":{"1min":2.3170000000000002,"5min":3.9910000000000001,"15min":104.315},"last":1.0960000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:38:22 2021","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.9730000000000001,"15min":2.0430000000000001},"min":{"1min":0.71399999999999997,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.784,"5min":99.271000000000001,"15min":103.65600000000001},"last":1.0349999999999999},{"interface":"front","average":{"1min":1.2649999999999999,"5min":2.0110000000000001,"15min":2.0139999999999998},"min":{"1min":0.67800000000000005,"5min":0.63500000000000001,"15min":0.63200000000000001},"max":{"1min":2.6469999999999998,"5min":99.561999999999998,"15min":104.136},"last":0.72299999999999998}]},{"osd":3,"last update":"Wed Feb 10 04:38:22 2021","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.621,"15min":2.052},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.624},"max":{"1min":2.4380000000000002,"5min":27.295999999999999,"15min":106.18000000000001},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.5449999999999999,"15min":1.98},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.64500000000000002},"max":{"1min":2.992,"5min":27.503,"15min":106.822},"last":0.90100000000000002}]},{"osd":4,"last update":"Wed Feb 10 04:38:22 2021","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.1519999999999999,"15min":1.0980000000000001},"min":{"1min":0.81899999999999995,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":2.7719999999999998,"5min":2.7719999999999998,"15min":5.2290000000000001},"last":1.385},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.351,"15min":1.2569999999999999},"min":{"1min":0.86799999999999999,"5min":0.73699999999999999,"15min":0.60799999999999998},"max":{"1min":2.9830000000000001,"5min":2.9830000000000001,"15min":8.1449999999999996},"last":1.2909999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:37:37 2021","interfaces":[{"interface":"back","average":{"1min":1.762,"5min":1.72,"15min":1.474},"min":{"1min":1.153,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":2.9990000000000001,"5min":25.969000000000001,"15min":25.969000000000001},"last":2.1280000000000001},{"interface":"front","average":{"1min":1.45,"5min":1.5840000000000001,"15min":1.3320000000000001},"min":{"1min":0.875,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.056,"5min":25.902999999999999,"15min":25.902999999999999},"last":1.3129999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:38:01 2021","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.333,"15min":1.3},"min":{"1min":1.0529999999999999,"5min":0.80800000000000005,"15min":0.78400000000000003},"max":{"1min":2.028,"5min":4.6699999999999999,"15min":5.4770000000000003},"last":1.4670000000000001},{"interface":"front","average":{"1min":1.591,"5min":1.544,"15min":1.52},"min":{"1min":1.1040000000000001,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":2.8839999999999999,"5min":2.8839999999999999,"15min":8.7840000000000007},"last":1.4970000000000001}]}]}]}} 2021-02-10T04:38:45.001 INFO:tasks.thrashosds.thrasher:Setting 4.9 to [1, 6, 5, 0, 3, 7] 2021-02-10T04:38:45.001 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '4.9', '1', '6', '5', '0', '3', '7'] 2021-02-10T04:38:45.001 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pg-upmap-items 4.9 1 6 5 0 3 7 2021-02-10T04:38:45.017 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:38:45.045 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:38:45.235 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:45 smithi085 conmon[29319]: cluster 2021-02-10T04:38:44.232009+0000 2021-02-10T04:38:45.236 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:45 smithi085 conmon[29319]: mon.a (mon.0) 1612 : cluster [DBG] osdmap e705: 8 total, 8 up, 7 in 2021-02-10T04:38:45.238 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:38:45 smithi164 conmon[40892]: debug 2021-02-10T04:38:45.116+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:38:45.238 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:45 smithi164 conmon[29288]: cluster 2021-02-10T04:38:45.253 INFO:tasks.rados.rados.0.smithi085.stdout:2748: done (7 left) 2021-02-10T04:38:45.253 INFO:tasks.rados.rados.0.smithi085.stdout:2749: write (excl) oid 22 current snap is 314 2021-02-10T04:38:45.254 INFO:tasks.rados.rados.0.smithi085.stdout:2749: seq_num 956 ranges {467492=482169,1626315=441177} 2021-02-10T04:38:45.254 INFO:tasks.rados.rados.0.smithi085.stdout:2749: writing smithi08575397-22 from 467492 to 949661 tid 1 2021-02-10T04:38:45.262 INFO:tasks.rados.rados.0.smithi085.stdout:2749: writing smithi08575397-22 from 1626315 to 2067492 tid 2 2021-02-10T04:38:45.262 INFO:tasks.rados.rados.0.smithi085.stdout:2750: read oid 32 snap -1 2021-02-10T04:38:45.262 INFO:tasks.rados.rados.0.smithi085.stdout:2750: expect (ObjNum 943 snap 313 seq_num 943) 2021-02-10T04:38:45.262 INFO:tasks.rados.rados.0.smithi085.stdout:2746: finishing rollback tid 1 to smithi08575397-1 2021-02-10T04:38:45.263 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 1340 (ObjNum 82149952 snap 22030 seq_num 82875760) dirty dne 2021-02-10T04:38:45.263 INFO:tasks.rados.rados.0.smithi085.stdout:2747: finishing rollback tid 0 to smithi08575397-42 2021-02-10T04:38:45.263 INFO:tasks.rados.rados.0.smithi085.stdout:2747: finishing rollback tid 1 to smithi08575397-42 2021-02-10T04:38:45.263 INFO:tasks.rados.rados.0.smithi085.stdout:2747: finishing rollback tid 2 to smithi08575397-42 2021-02-10T04:38:45.263 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 1339 (ObjNum 833 snap 270 seq_num 833) dirty exists 2021-02-10T04:38:45.264 INFO:tasks.rados.rados.0.smithi085.stdout:2745: finishing copy_from to smithi08575397-15 2021-02-10T04:38:45.264 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 15 v 1346 (ObjNum 944 snap 313 seq_num 944) dirty exists 2021-02-10T04:38:45.264 INFO:tasks.rados.rados.0.smithi085.stdout:2745: finishing copy_from racing read to smithi08575397-15 2021-02-10T04:38:45.264 INFO:tasks.rados.rados.0.smithi085.stdout:2749: finishing write tid 1 to smithi08575397-22 2021-02-10T04:38:45.264 INFO:tasks.rados.rados.0.smithi085.stdout:2745: done (8 left) 2021-02-10T04:38:45.264 INFO:tasks.rados.rados.0.smithi085.stdout:2746: done (7 left) 2021-02-10T04:38:45.265 INFO:tasks.rados.rados.0.smithi085.stdout:2747: done (6 left) 2021-02-10T04:38:45.265 INFO:tasks.rados.rados.0.smithi085.stdout:2751: snap_remove snap 271 2021-02-10T04:38:45.357 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:38:45.362 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:38:45.463 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:38:45.498 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:38:45.505 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:45 smithi085 conmon[35961]: cluster 2021-02-10T04:38:45.505 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:45 smithi085 conmon[35961]: 2021-02-10T04:38:45.506 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:45 smithi085 conmon[35961]: 2021 2021-02-10T04:38:45.506 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:45 smithi085 conmon[35961]: - 2021-02-10T04:38:45.506 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:45 smithi085 conmon[35961]: 02 2021-02-10T04:38:45.506 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:45 smithi085 conmon[35961]: - 2021-02-10T04:38:45.506 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:45 smithi085 conmon[35961]: 10 2021-02-10T04:38:45.506 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:45 smithi085 conmon[35961]: T 2021-02-10T04:38:45.507 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:45 smithi085 conmon[35961]: 04 2021-02-10T04:38:45.507 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:45 smithi085 conmon[35961]: : 2021-02-10T04:38:45.507 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:45 smithi085 conmon[35961]: 38 2021-02-10T04:38:45.507 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:45 smithi085 conmon[35961]: : 2021-02-10T04:38:45.507 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:45 smithi085 conmon[35961]: 44 2021-02-10T04:38:45.507 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:45 smithi085 conmon[35961]: . 2021-02-10T04:38:45.508 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:45 smithi085 conmon[35961]: 232009 2021-02-10T04:38:45.508 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:45 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2021-02-10T04:38:45.508 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:45 smithi085 conmon[35961]: 1612 : cluster [DBG] osdmap e705: 8 total, 8 up, 7 in 2021-02-10T04:38:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:45 smithi164 conmon[29288]: 2021-02-10T04:38:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:45 smithi164 conmon[29288]: 2021 2021-02-10T04:38:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:45 smithi164 conmon[29288]: - 2021-02-10T04:38:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:45 smithi164 conmon[29288]: 02 2021-02-10T04:38:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:45 smithi164 conmon[29288]: - 2021-02-10T04:38:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:45 smithi164 conmon[29288]: 10 2021-02-10T04:38:45.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:45 smithi164 conmon[29288]: T04:38:44. 2021-02-10T04:38:45.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:45 smithi164 conmon[29288]: 232009+0000 mon.a (mon.0) 1612 : cluster [DBG] osdmap e705: 8 total, 8 up, 7 in 2021-02-10T04:38:45.783 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:38:45.889 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:38:45.931 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:38:46.034 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:38:45 smithi085 conmon[41540]: debug 2021-02-10T04:38:46.034 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:38:45 smithi085 conmon[41540]: 2021-02-10T04:38:45.757+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:38:46.034 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:38:45 smithi085 conmon[41540]: 2021-02-10T04:38:46.256 INFO:tasks.rados.rados.0.smithi085.stdout:2751: done (6 left) 2021-02-10T04:38:46.256 INFO:tasks.rados.rados.0.smithi085.stdout:2752: rollback oid 43 current snap is 314 2021-02-10T04:38:46.256 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 43 to 311 2021-02-10T04:38:46.256 INFO:tasks.rados.rados.0.smithi085.stdout:2749: finishing write tid 2 to smithi08575397-22 2021-02-10T04:38:46.257 INFO:tasks.rados.rados.0.smithi085.stdout:2749: finishing write tid 3 to smithi08575397-22 2021-02-10T04:38:46.257 INFO:tasks.rados.rados.0.smithi085.stdout:2749: finishing write tid 4 to smithi08575397-22 2021-02-10T04:38:46.257 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 1429 (ObjNum 956 snap 314 seq_num 956) dirty exists 2021-02-10T04:38:46.257 INFO:tasks.rados.rados.0.smithi085.stdout:2749: left oid 22 (ObjNum 956 snap 314 seq_num 956) 2021-02-10T04:38:46.257 INFO:tasks.rados.rados.0.smithi085.stdout:2750: expect (ObjNum 943 snap 313 seq_num 943) 2021-02-10T04:38:46.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[49106]: debug 2021-02-10T04:38:46.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[49106]: 2021-02-10T04:38:46.127+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:38:46.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[49106]: 2021-02-10T04:38:46.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[29319]: cluster 2021-02-10T04:38:46.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[29319]: 2021-02-10T04:38:46.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[29319]: 2021-02-10T04:38 2021-02-10T04:38:46.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[29319]: :45.246936+0000 mon.a (mon.0) 1613 : cluster [DBG] osdmap e706: 8 total, 8 up, 7 in 2021-02-10T04:38:46.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:38:46.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[29319]: -10T04:38:45.490096+0000 osd.4 (osd.4) 1127 : cluster [DBG] 2021-02-10T04:38:46.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[29319]: purged_snaps scrub starts 2021-02-10T04:38:46.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[29319]: cluster 2021-02-10T04:38: 2021-02-10T04:38:46.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[29319]: 45.494632+0000 osd.4 (osd.4) 1128 2021-02-10T04:38:46.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[29319]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:38:46.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:38:46.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[29319]: T04:38:46.163747+0000 mgr.y ( 2021-02-10T04:38:46.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[29319]: mgr.14140) 722 : cluster [DBG] pgmap v1210: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 444 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.9 MiB/s wr, 23 op/s 2021-02-10T04:38:46.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: cluster 2021-02-10T04:38:46.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: 2021-02-10T04:38:46.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: 2021 2021-02-10T04:38:46.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: - 2021-02-10T04:38:46.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: 02 2021-02-10T04:38:46.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: - 2021-02-10T04:38:46.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: 10 2021-02-10T04:38:46.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: T 2021-02-10T04:38:46.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: 04 2021-02-10T04:38:46.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: : 2021-02-10T04:38:46.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: 38 2021-02-10T04:38:46.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: : 2021-02-10T04:38:46.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: 45 2021-02-10T04:38:46.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: . 2021-02-10T04:38:46.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: 246936 2021-02-10T04:38:46.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: +0000 2021-02-10T04:38:46.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: 2021-02-10T04:38:46.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: mon.a 2021-02-10T04:38:46.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: ( 2021-02-10T04:38:46.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: mon 2021-02-10T04:38:46.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: . 2021-02-10T04:38:46.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: 0 2021-02-10T04:38:46.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: ) 2021-02-10T04:38:46.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: 1613 2021-02-10T04:38:46.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: : 2021-02-10T04:38:46.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: cluster 2021-02-10T04:38:46.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: 2021-02-10T04:38:46.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: [DBG] 2021-02-10T04:38:46.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: 2021-02-10T04:38:46.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: osdmap e706: 8 total, 8 up, 7 in 2021-02-10T04:38:46.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: 2021-02-10T04:38:46.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: cluster 2021-02-10T04:38:46.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: 2021-02-10T04:38:46.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: 2021 2021-02-10T04:38:46.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: - 2021-02-10T04:38:46.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: 02 2021-02-10T04:38:46.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: - 2021-02-10T04:38:46.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: 10T04:38:45. 2021-02-10T04:38:46.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: 490096+0000 osd.4 (osd.4) 1127 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:38:46.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: cluster 2021-02-10T04:38:46.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: 2021-02 2021-02-10T04:38:46.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: -10T04:38:45.494632+0000 osd.4 (osd.4 2021-02-10T04:38:46.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: ) 1128 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:38:46.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: cluster 2021-02-10T04:38:46.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: 2021-02 2021-02-10T04:38:46.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:46 smithi085 conmon[35961]: -10T04:38:46.163747+0000 mgr.y (mgr.14140) 722 : cluster [DBG] pgmap v1210: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 444 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.9 MiB/s wr, 23 op/s 2021-02-10T04:38:46.624 INFO:tasks.rados.rados.0.smithi085.stdout:2749: done (6 left) 2021-02-10T04:38:46.624 INFO:tasks.rados.rados.0.smithi085.stdout:2750: done (5 left) 2021-02-10T04:38:46.624 INFO:tasks.rados.rados.0.smithi085.stdout:2753: snap_create 2021-02-10T04:38:46.624 INFO:tasks.rados.rados.0.smithi085.stdout:2752: finishing rollback tid 0 to smithi08575397-43 2021-02-10T04:38:46.624 INFO:tasks.rados.rados.0.smithi085.stdout:2752: finishing rollback tid 1 to smithi08575397-43 2021-02-10T04:38:46.625 INFO:tasks.rados.rados.0.smithi085.stdout:2752: finishing rollback tid 2 to smithi08575397-43 2021-02-10T04:38:46.625 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 1184 (ObjNum 892 snap 289 seq_num 892) dirty exists 2021-02-10T04:38:46.643 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[37263]: debug 2021-02-10T04:38:46.643 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[37263]: 2021-02-10T04:38:46.289+0000 7f1c671e2700 -1 osd.5 pg_epoch: 707 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x556653511760) 2021-02-10T04:38:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: cluster 2021-02-10T04:38:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 2021-02-10T04:38:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 2021 2021-02-10T04:38:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: - 2021-02-10T04:38:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 02 2021-02-10T04:38:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: - 2021-02-10T04:38:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 10 2021-02-10T04:38:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: T 2021-02-10T04:38:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 04 2021-02-10T04:38:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: : 2021-02-10T04:38:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 38 2021-02-10T04:38:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: : 2021-02-10T04:38:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 45 2021-02-10T04:38:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: . 2021-02-10T04:38:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 246936 2021-02-10T04:38:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: +0000 2021-02-10T04:38:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 2021-02-10T04:38:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: mon.a 2021-02-10T04:38:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: ( 2021-02-10T04:38:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: mon 2021-02-10T04:38:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: . 2021-02-10T04:38:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 0 2021-02-10T04:38:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: ) 2021-02-10T04:38:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 1613 2021-02-10T04:38:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: : 2021-02-10T04:38:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: cluster 2021-02-10T04:38:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 2021-02-10T04:38:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: [DBG] 2021-02-10T04:38:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 2021-02-10T04:38:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: osdmap e706: 8 total, 8 up, 7 in 2021-02-10T04:38:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 2021-02-10T04:38:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: cluster 2021-02-10T04:38:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 2021-02-10T04:38:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 2021 2021-02-10T04:38:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: - 2021-02-10T04:38:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 02 2021-02-10T04:38:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: - 2021-02-10T04:38:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 10 2021-02-10T04:38:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: T 2021-02-10T04:38:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 04 2021-02-10T04:38:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: : 2021-02-10T04:38:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 38 2021-02-10T04:38:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: : 2021-02-10T04:38:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 45 2021-02-10T04:38:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: . 2021-02-10T04:38:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 490096 2021-02-10T04:38:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: +0000 2021-02-10T04:38:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 2021-02-10T04:38:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: osd.4 2021-02-10T04:38:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: ( 2021-02-10T04:38:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: osd 2021-02-10T04:38:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: . 2021-02-10T04:38:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 4 2021-02-10T04:38:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: ) 2021-02-10T04:38:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 1127 2021-02-10T04:38:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: : 2021-02-10T04:38:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: cluster 2021-02-10T04:38:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 2021-02-10T04:38:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: [DBG] 2021-02-10T04:38:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 2021-02-10T04:38:46.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: purged_snaps scrub starts 2021-02-10T04:38:46.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 2021-02-10T04:38:46.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: cluster 2021-02-10T04:38:46.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:38:46.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 04:38:45.494632+0000 osd.4 (osd.4) 1128 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:38:46.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:38:46.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 02-10T04:38:46.163747+0000 mgr.y (mgr. 2021-02-10T04:38:46.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:46 smithi164 conmon[29288]: 14140) 722 : cluster [DBG] pgmap v1210: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 444 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.9 MiB/s wr, 23 op/s 2021-02-10T04:38:47.264 INFO:tasks.rados.rados.0.smithi085.stdout:2752: done (5 left) 2021-02-10T04:38:47.265 INFO:tasks.rados.rados.0.smithi085.stdout:2753: done (4 left) 2021-02-10T04:38:47.265 INFO:tasks.rados.rados.0.smithi085.stdout:2754: delete oid 48 current snap is 315 2021-02-10T04:38:47.272 INFO:tasks.rados.rados.0.smithi085.stdout:2754: done (4 left) 2021-02-10T04:38:47.272 INFO:tasks.rados.rados.0.smithi085.stdout:2755: read oid 31 snap 306 2021-02-10T04:38:47.273 INFO:tasks.rados.rados.0.smithi085.stdout:2755: expect (ObjNum 906 snap 298 seq_num 906) 2021-02-10T04:38:47.273 INFO:tasks.rados.rados.0.smithi085.stdout:2756: read oid 34 snap 299 2021-02-10T04:38:47.273 INFO:tasks.rados.rados.0.smithi085.stdout:2756: expect (ObjNum 859 snap 277 seq_num 859) 2021-02-10T04:38:47.273 INFO:tasks.rados.rados.0.smithi085.stdout:2757: snap_remove snap 269 2021-02-10T04:38:47.563 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:47 smithi085 conmon[35961]: cluster 2021-02-10T04:38:46.255971+0000 mon.a (mon 2021-02-10T04:38:47.563 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:47 smithi085 conmon[35961]: .0) 1614 : cluster [DBG] osdmap e707: 8 total, 8 up, 7 in 2021-02-10T04:38:47.563 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:47 smithi085 conmon[29319]: cluster 2021-02-10T04:38:46.255971+0000 mon.a (mon.0) 1614 : cluster [DBG] 2021-02-10T04:38:47.563 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:47 smithi085 conmon[29319]: osdmap e707: 8 total, 8 up, 7 in 2021-02-10T04:38:47.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:47 smithi164 conmon[29288]: cluster 2021-02-10T04:38:46.255971+0000 mon.a (mon.0) 1614 : cluster [DBG] osdmap e707: 8 total, 8 up, 7 in 2021-02-10T04:38:48.234 INFO:tasks.rados.rados.0.smithi085.stdout:2757: done (6 left) 2021-02-10T04:38:48.241 INFO:tasks.rados.rados.0.smithi085.stdout:2758: write oid 22 current snap is 315 2021-02-10T04:38:48.242 INFO:tasks.rados.rados.0.smithi085.stdout:2758: seq_num 958 ranges {501106=749020,1743279=357827} 2021-02-10T04:38:48.242 INFO:tasks.rados.rados.0.smithi085.stdout:2758: writing smithi08575397-22 from 501106 to 1250126 tid 1 2021-02-10T04:38:48.247 INFO:tasks.rados.rados.0.smithi085.stdout:2758: writing smithi08575397-22 from 1743279 to 2101106 tid 2 2021-02-10T04:38:48.248 INFO:tasks.rados.rados.0.smithi085.stdout:2759: read oid 32 snap -1 2021-02-10T04:38:48.248 INFO:tasks.rados.rados.0.smithi085.stdout:2759: expect (ObjNum 943 snap 313 seq_num 943) 2021-02-10T04:38:48.248 INFO:tasks.rados.rados.0.smithi085.stdout:2760: write (excl) oid 28 current snap is 315 2021-02-10T04:38:48.248 INFO:tasks.rados.rados.0.smithi085.stdout:2760: seq_num 959 ranges {517913=424269,1401761=446989} 2021-02-10T04:38:48.253 INFO:tasks.rados.rados.0.smithi085.stdout:2760: writing smithi08575397-28 from 517913 to 942182 tid 1 2021-02-10T04:38:48.259 INFO:tasks.rados.rados.0.smithi085.stdout:2760: writing smithi08575397-28 from 1401761 to 1848750 tid 2 2021-02-10T04:38:48.260 INFO:tasks.rados.rados.0.smithi085.stdout:2761: read oid 37 snap 299 2021-02-10T04:38:48.260 INFO:tasks.rados.rados.0.smithi085.stdout:2761: expect deleted 2021-02-10T04:38:48.260 INFO:tasks.rados.rados.0.smithi085.stdout:2762: write oid 7 current snap is 315 2021-02-10T04:38:48.260 INFO:tasks.rados.rados.0.smithi085.stdout:2762: seq_num 960 ranges {534720=499518,1460243=674477} 2021-02-10T04:38:48.267 INFO:tasks.rados.rados.0.smithi085.stdout:2762: writing smithi08575397-7 from 534720 to 1034238 tid 1 2021-02-10T04:38:48.275 INFO:tasks.rados.rados.0.smithi085.stdout:2762: writing smithi08575397-7 from 1460243 to 2134720 tid 2 2021-02-10T04:38:48.276 INFO:tasks.rados.rados.0.smithi085.stdout:2763: read oid 16 snap -1 2021-02-10T04:38:48.276 INFO:tasks.rados.rados.0.smithi085.stdout:2763: expect deleted 2021-02-10T04:38:48.276 INFO:tasks.rados.rados.0.smithi085.stdout:2764: write oid 5 current snap is 315 2021-02-10T04:38:48.276 INFO:tasks.rados.rados.0.smithi085.stdout:2756: expect (ObjNum 859 snap 277 seq_num 859) 2021-02-10T04:38:48.389 INFO:tasks.rados.rados.0.smithi085.stdout:2755: expect (ObjNum 906 snap 298 seq_num 906) 2021-02-10T04:38:48.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:48 smithi085 conmon[29319]: cluster 2021-02-10T04:38:47.264299+0000 mon.a ( 2021-02-10T04:38:48.519 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:48 smithi085 conmon[29319]: mon.0) 1615 : cluster [DBG] osdmap e708: 8 total, 8 up, 7 in 2021-02-10T04:38:48.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:48 smithi085 conmon[29319]: cluster 2021-02-10T04:38:48.164299+0000 mgr.y (mgr.14140) 723 : cluster [DBG] pgmap v1213: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 448 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 20 MiB/s rd, 2.9 MiB/s wr, 26 op/s 2021-02-10T04:38:48.520 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:48 smithi085 conmon[29319]: cluster 2021-02-10T04:38:48.230028+0000 mon.a (mon.0) 1616 : cluster [DBG] osdmap e709: 8 total, 8 up, 7 in 2021-02-10T04:38:48.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:48 smithi085 conmon[35961]: cluster 2021-02-10T04:38:47.264299 2021-02-10T04:38:48.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:48 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1615 : cluster [DBG] osdmap e708: 8 total, 8 up, 7 in 2021-02-10T04:38:48.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:48 smithi085 conmon[35961]: cluster 2021-02-10T04:38:48 2021-02-10T04:38:48.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:48 smithi085 conmon[35961]: .164299+0000 mgr.y (mgr.14140) 723 : cluster [DBG] 2021-02-10T04:38:48.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:48 smithi085 conmon[35961]: pgmap v1213: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 448 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 20 MiB/s rd, 2.9 MiB/s wr, 26 op/s 2021-02-10T04:38:48.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:48 smithi085 conmon[35961]: cluster 2021-02-10T04:38:48 2021-02-10T04:38:48.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:48 smithi085 conmon[35961]: .230028+0000 mon.a (mon.0) 1616 : cluster 2021-02-10T04:38:48.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:48 smithi085 conmon[35961]: [DBG] osdmap e709: 8 total, 8 up, 7 in 2021-02-10T04:38:48.533 INFO:tasks.rados.rados.0.smithi085.stdout:2764: seq_num 961 ranges {551527=574767,1918725=232802} 2021-02-10T04:38:48.540 INFO:tasks.rados.rados.0.smithi085.stdout:2764: writing smithi08575397-5 from 551527 to 1126294 tid 1 2021-02-10T04:38:48.546 INFO:tasks.rados.rados.0.smithi085.stdout:2764: writing smithi08575397-5 from 1918725 to 2151527 tid 2 2021-02-10T04:38:48.546 INFO:tasks.rados.rados.0.smithi085.stdout:2755: done (12 left) 2021-02-10T04:38:48.546 INFO:tasks.rados.rados.0.smithi085.stdout:2756: done (11 left) 2021-02-10T04:38:48.546 INFO:tasks.rados.rados.0.smithi085.stdout:2765: write oid 15 current snap is 315 2021-02-10T04:38:48.546 INFO:tasks.rados.rados.0.smithi085.stdout:2759: expect (ObjNum 943 snap 313 seq_num 943) 2021-02-10T04:38:48.644 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:38:48 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:38:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:48 smithi164 conmon[29288]: cluster 2021-02-10T04:38:47.264299+0000 2021-02-10T04:38:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:48 smithi164 conmon[29288]: mon.a (mon.0) 1615 : cluster [DBG] osdmap e708: 8 total, 8 up, 7 in 2021-02-10T04:38:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:48 smithi164 conmon[29288]: cluster 2021-02-10T04:38:48.164299+0000 mgr.y (mgr.14140) 723 : cluster 2021-02-10T04:38:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:48 smithi164 conmon[29288]: [DBG] pgmap v1213: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 448 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 20 MiB/s rd, 2.9 MiB/s wr, 26 op/s 2021-02-10T04:38:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:48 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:38:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:48 smithi164 conmon[29288]: -02-10T04:38:48.230028+0000 mon.a ( 2021-02-10T04:38:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:48 smithi164 conmon[29288]: mon.0) 1616 : cluster [DBG] osdmap e709: 8 total, 8 up, 7 in 2021-02-10T04:38:48.893 INFO:tasks.rados.rados.0.smithi085.stdout:2758: finishing write tid 1 to smithi08575397-22 2021-02-10T04:38:48.893 INFO:tasks.rados.rados.0.smithi085.stdout:2758: finishing write tid 2 to smithi08575397-22 2021-02-10T04:38:48.893 INFO:tasks.rados.rados.0.smithi085.stdout:2758: finishing write tid 3 to smithi08575397-22 2021-02-10T04:38:48.893 INFO:tasks.rados.rados.0.smithi085.stdout:2758: finishing write tid 4 to smithi08575397-22 2021-02-10T04:38:48.893 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 1432 (ObjNum 958 snap 315 seq_num 958) dirty exists 2021-02-10T04:38:48.893 INFO:tasks.rados.rados.0.smithi085.stdout:2758: left oid 22 (ObjNum 958 snap 315 seq_num 958) 2021-02-10T04:38:48.893 INFO:tasks.rados.rados.0.smithi085.stdout:2760: finishing write tid 1 to smithi08575397-28 2021-02-10T04:38:48.894 INFO:tasks.rados.rados.0.smithi085.stdout:2762: finishing write tid 1 to smithi08575397-7 2021-02-10T04:38:48.894 INFO:tasks.rados.rados.0.smithi085.stdout:2762: finishing write tid 2 to smithi08575397-7 2021-02-10T04:38:48.894 INFO:tasks.rados.rados.0.smithi085.stdout:2762: finishing write tid 3 to smithi08575397-7 2021-02-10T04:38:48.894 INFO:tasks.rados.rados.0.smithi085.stdout:2762: finishing write tid 4 to smithi08575397-7 2021-02-10T04:38:48.894 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 1187 (ObjNum 960 snap 315 seq_num 960) dirty exists 2021-02-10T04:38:48.894 INFO:tasks.rados.rados.0.smithi085.stdout:2762: left oid 7 (ObjNum 960 snap 315 seq_num 960) 2021-02-10T04:38:48.894 INFO:tasks.rados.rados.0.smithi085.stdout:2760: finishing write tid 2 to smithi08575397-28 2021-02-10T04:38:48.894 INFO:tasks.rados.rados.0.smithi085.stdout:2760: finishing write tid 3 to smithi08575397-28 2021-02-10T04:38:48.894 INFO:tasks.rados.rados.0.smithi085.stdout:2760: finishing write tid 4 to smithi08575397-28 2021-02-10T04:38:48.894 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 1278 (ObjNum 959 snap 315 seq_num 959) dirty exists 2021-02-10T04:38:48.894 INFO:tasks.rados.rados.0.smithi085.stdout:2760: left oid 28 (ObjNum 959 snap 315 seq_num 959) 2021-02-10T04:38:48.894 INFO:tasks.rados.rados.0.smithi085.stdout:2764: finishing write tid 1 to smithi08575397-5 2021-02-10T04:38:48.895 INFO:tasks.rados.rados.0.smithi085.stdout:2764: finishing write tid 2 to smithi08575397-5 2021-02-10T04:38:48.895 INFO:tasks.rados.rados.0.smithi085.stdout:2764: finishing write tid 3 to smithi08575397-5 2021-02-10T04:38:48.895 INFO:tasks.rados.rados.0.smithi085.stdout:2764: finishing write tid 4 to smithi08575397-5 2021-02-10T04:38:48.895 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 793 (ObjNum 961 snap 315 seq_num 961) dirty exists 2021-02-10T04:38:48.895 INFO:tasks.rados.rados.0.smithi085.stdout:2764: left oid 5 (ObjNum 961 snap 315 seq_num 961) 2021-02-10T04:38:48.895 INFO:tasks.rados.rados.0.smithi085.stdout:2765: seq_num 962 ranges {568334=650016,1860854=307480} 2021-02-10T04:38:48.900 INFO:tasks.rados.rados.0.smithi085.stdout:2765: writing smithi08575397-15 from 568334 to 1218350 tid 1 2021-02-10T04:38:48.906 INFO:tasks.rados.rados.0.smithi085.stdout:2765: writing smithi08575397-15 from 1860854 to 2168334 tid 2 2021-02-10T04:38:48.906 INFO:tasks.rados.rados.0.smithi085.stdout:2765: finishing write tid 1 to smithi08575397-15 2021-02-10T04:38:48.906 INFO:tasks.rados.rados.0.smithi085.stdout:2758: done (11 left) 2021-02-10T04:38:48.906 INFO:tasks.rados.rados.0.smithi085.stdout:2759: done (10 left) 2021-02-10T04:38:48.906 INFO:tasks.rados.rados.0.smithi085.stdout:2760: done (9 left) 2021-02-10T04:38:48.906 INFO:tasks.rados.rados.0.smithi085.stdout:2761: done (8 left) 2021-02-10T04:38:48.906 INFO:tasks.rados.rados.0.smithi085.stdout:2762: done (7 left) 2021-02-10T04:38:48.906 INFO:tasks.rados.rados.0.smithi085.stdout:2763: done (6 left) 2021-02-10T04:38:48.907 INFO:tasks.rados.rados.0.smithi085.stdout:2764: done (5 left) 2021-02-10T04:38:48.907 INFO:tasks.rados.rados.0.smithi085.stdout:2766: delete oid 35 current snap is 315 2021-02-10T04:38:48.908 INFO:tasks.rados.rados.0.smithi085.stdout:2766: done (5 left) 2021-02-10T04:38:48.908 INFO:tasks.rados.rados.0.smithi085.stdout:2767: write (excl) oid 36 current snap is 315 2021-02-10T04:38:48.909 INFO:tasks.rados.rados.0.smithi085.stdout:2767: seq_num 964 ranges {601948=400514,1577818=624130} 2021-02-10T04:38:48.914 INFO:tasks.rados.rados.0.smithi085.stdout:2767: writing smithi08575397-36 from 601948 to 1002462 tid 1 2021-02-10T04:38:48.922 INFO:tasks.rados.rados.0.smithi085.stdout:2767: writing smithi08575397-36 from 1577818 to 2201948 tid 2 2021-02-10T04:38:48.922 INFO:tasks.rados.rados.0.smithi085.stdout:2768: write (excl) oid 48 current snap is 315 2021-02-10T04:38:48.923 INFO:tasks.rados.rados.0.smithi085.stdout:2765: finishing write tid 2 to smithi08575397-15 2021-02-10T04:38:48.923 INFO:tasks.rados.rados.0.smithi085.stdout:2768: seq_num 965 ranges {618755=475763,1636300=457996} 2021-02-10T04:38:48.929 INFO:tasks.rados.rados.0.smithi085.stdout:2768: writing smithi08575397-48 from 618755 to 1094518 tid 1 2021-02-10T04:38:48.936 INFO:tasks.rados.rados.0.smithi085.stdout:2768: writing smithi08575397-48 from 1636300 to 2094296 tid 2 2021-02-10T04:38:48.936 INFO:tasks.rados.rados.0.smithi085.stdout:2769: copy_from oid 4 from oid 49 current snap is 315 2021-02-10T04:38:48.936 INFO:tasks.rados.rados.0.smithi085.stdout:2765: finishing write tid 3 to smithi08575397-15 2021-02-10T04:38:48.936 INFO:tasks.rados.rados.0.smithi085.stdout:2765: finishing write tid 4 to smithi08575397-15 2021-02-10T04:38:48.937 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 15 v 1349 (ObjNum 962 snap 315 seq_num 962) dirty exists 2021-02-10T04:38:48.937 INFO:tasks.rados.rados.0.smithi085.stdout:2765: left oid 15 (ObjNum 962 snap 315 seq_num 962) 2021-02-10T04:38:48.937 INFO:tasks.rados.rados.0.smithi085.stdout:2767: finishing write tid 1 to smithi08575397-36 2021-02-10T04:38:48.937 INFO:tasks.rados.rados.0.smithi085.stdout:2767: finishing write tid 2 to smithi08575397-36 2021-02-10T04:38:48.937 INFO:tasks.rados.rados.0.smithi085.stdout:2767: finishing write tid 3 to smithi08575397-36 2021-02-10T04:38:48.937 INFO:tasks.rados.rados.0.smithi085.stdout:2767: finishing write tid 4 to smithi08575397-36 2021-02-10T04:38:48.937 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 1286 (ObjNum 964 snap 315 seq_num 964) dirty exists 2021-02-10T04:38:48.937 INFO:tasks.rados.rados.0.smithi085.stdout:2767: left oid 36 (ObjNum 964 snap 315 seq_num 964) 2021-02-10T04:38:48.938 INFO:tasks.rados.rados.0.smithi085.stdout:2768: finishing write tid 1 to smithi08575397-48 2021-02-10T04:38:48.938 INFO:tasks.rados.rados.0.smithi085.stdout:2765: done (7 left) 2021-02-10T04:38:48.938 INFO:tasks.rados.rados.0.smithi085.stdout:2767: done (6 left) 2021-02-10T04:38:48.938 INFO:tasks.rados.rados.0.smithi085.stdout:2770: write oid 22 current snap is 315 2021-02-10T04:38:48.938 INFO:tasks.rados.rados.0.smithi085.stdout:2770: seq_num 966 ranges {635562=667365,2094782=140780} 2021-02-10T04:38:48.943 INFO:tasks.rados.rados.0.smithi085.stdout:2770: writing smithi08575397-22 from 635562 to 1302927 tid 1 2021-02-10T04:38:48.949 INFO:tasks.rados.rados.0.smithi085.stdout:2770: writing smithi08575397-22 from 2094782 to 2235562 tid 2 2021-02-10T04:38:48.949 INFO:tasks.rados.rados.0.smithi085.stdout:2771: read oid 18 snap -1 2021-02-10T04:38:48.949 INFO:tasks.rados.rados.0.smithi085.stdout:2771: expect (ObjNum 952 snap 313 seq_num 952) 2021-02-10T04:38:48.949 INFO:tasks.rados.rados.0.smithi085.stdout:2772: snap_create 2021-02-10T04:38:48.949 INFO:tasks.rados.rados.0.smithi085.stdout:2768: finishing write tid 2 to smithi08575397-48 2021-02-10T04:38:48.949 INFO:tasks.rados.rados.0.smithi085.stdout:2768: finishing write tid 3 to smithi08575397-48 2021-02-10T04:38:48.949 INFO:tasks.rados.rados.0.smithi085.stdout:2768: finishing write tid 4 to smithi08575397-48 2021-02-10T04:38:48.949 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 1343 (ObjNum 965 snap 315 seq_num 965) dirty exists 2021-02-10T04:38:48.949 INFO:tasks.rados.rados.0.smithi085.stdout:2768: left oid 48 (ObjNum 965 snap 315 seq_num 965) 2021-02-10T04:38:48.949 INFO:tasks.rados.rados.0.smithi085.stdout:2770: finishing write tid 1 to smithi08575397-22 2021-02-10T04:38:48.952 INFO:tasks.rados.rados.0.smithi085.stdout:2770: finishing write tid 2 to smithi08575397-22 2021-02-10T04:38:48.953 INFO:tasks.rados.rados.0.smithi085.stdout:2770: finishing write tid 3 to smithi08575397-22 2021-02-10T04:38:48.953 INFO:tasks.rados.rados.0.smithi085.stdout:2770: finishing write tid 4 to smithi08575397-22 2021-02-10T04:38:48.953 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 1435 (ObjNum 966 snap 315 seq_num 966) dirty exists 2021-02-10T04:38:48.953 INFO:tasks.rados.rados.0.smithi085.stdout:2770: left oid 22 (ObjNum 966 snap 315 seq_num 966) 2021-02-10T04:38:48.954 INFO:tasks.rados.rados.0.smithi085.stdout:2771: expect (ObjNum 952 snap 313 seq_num 952) 2021-02-10T04:38:49.133 INFO:tasks.rados.rados.0.smithi085.stdout:2769: finishing copy_from to smithi08575397-4 2021-02-10T04:38:49.133 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 4 v 886 (ObjNum 172 snap 37 seq_num 172) dirty exists 2021-02-10T04:38:49.133 INFO:tasks.rados.rados.0.smithi085.stdout:2769: finishing copy_from racing read to smithi08575397-4 2021-02-10T04:38:49.338 INFO:tasks.rados.rados.0.smithi085.stdout:2768: done (8 left) 2021-02-10T04:38:49.338 INFO:tasks.rados.rados.0.smithi085.stdout:2769: done (7 left) 2021-02-10T04:38:49.338 INFO:tasks.rados.rados.0.smithi085.stdout:2770: done (6 left) 2021-02-10T04:38:49.338 INFO:tasks.rados.rados.0.smithi085.stdout:2771: done (5 left) 2021-02-10T04:38:49.338 INFO:tasks.rados.rados.0.smithi085.stdout:2772: done (4 left) 2021-02-10T04:38:49.338 INFO:tasks.rados.rados.0.smithi085.stdout:2773: write (excl) oid 11 current snap is 316 2021-02-10T04:38:49.338 INFO:tasks.rados.rados.0.smithi085.stdout:2773: seq_num 967 ranges {652369=742614,2153264=99105} 2021-02-10T04:38:49.346 INFO:tasks.rados.rados.0.smithi085.stdout:2773: writing smithi08575397-11 from 652369 to 1394983 tid 1 2021-02-10T04:38:49.352 INFO:tasks.rados.rados.0.smithi085.stdout:2773: writing smithi08575397-11 from 2153264 to 2252369 tid 2 2021-02-10T04:38:49.352 INFO:tasks.rados.rados.0.smithi085.stdout:2774: rollback oid 46 current snap is 316 2021-02-10T04:38:49.352 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 46 to 288 2021-02-10T04:38:49.352 INFO:tasks.rados.rados.0.smithi085.stdout:2775: write oid 21 current snap is 316 2021-02-10T04:38:49.352 INFO:tasks.rados.rados.0.smithi085.stdout:2775: seq_num 968 ranges {669176=417863,1811746=457430} 2021-02-10T04:38:49.359 INFO:tasks.rados.rados.0.smithi085.stdout:2775: writing smithi08575397-21 from 669176 to 1087039 tid 1 2021-02-10T04:38:49.367 INFO:tasks.rados.rados.0.smithi085.stdout:2775: writing smithi08575397-21 from 1811746 to 2269176 tid 2 2021-02-10T04:38:49.367 INFO:tasks.rados.rados.0.smithi085.stdout:2776: delete oid 4 current snap is 316 2021-02-10T04:38:49.367 INFO:tasks.rados.rados.0.smithi085.stdout:2773: finishing write tid 1 to smithi08575397-11 2021-02-10T04:38:49.367 INFO:tasks.rados.rados.0.smithi085.stdout:2774: finishing rollback tid 0 to smithi08575397-46 2021-02-10T04:38:49.367 INFO:tasks.rados.rados.0.smithi085.stdout:2774: finishing rollback tid 1 to smithi08575397-46 2021-02-10T04:38:49.367 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 1352 (ObjNum 71101512 snap 22030 seq_num 1) dirty dne 2021-02-10T04:38:49.367 INFO:tasks.rados.rados.0.smithi085.stdout:2773: finishing write tid 2 to smithi08575397-11 2021-02-10T04:38:49.367 INFO:tasks.rados.rados.0.smithi085.stdout:2773: finishing write tid 3 to smithi08575397-11 2021-02-10T04:38:49.367 INFO:tasks.rados.rados.0.smithi085.stdout:2773: finishing write tid 4 to smithi08575397-11 2021-02-10T04:38:49.368 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 1452 (ObjNum 967 snap 316 seq_num 967) dirty exists 2021-02-10T04:38:49.368 INFO:tasks.rados.rados.0.smithi085.stdout:2773: left oid 11 (ObjNum 967 snap 316 seq_num 967) 2021-02-10T04:38:49.368 INFO:tasks.rados.rados.0.smithi085.stdout:2775: finishing write tid 1 to smithi08575397-21 2021-02-10T04:38:49.371 INFO:tasks.rados.rados.0.smithi085.stdout:2773: done (7 left) 2021-02-10T04:38:49.371 INFO:tasks.rados.rados.0.smithi085.stdout:2774: done (6 left) 2021-02-10T04:38:49.371 INFO:tasks.rados.rados.0.smithi085.stdout:2776: done (5 left) 2021-02-10T04:38:49.371 INFO:tasks.rados.rados.0.smithi085.stdout:2777: copy_from oid 43 from oid 13 current snap is 316 2021-02-10T04:38:49.371 INFO:tasks.rados.rados.0.smithi085.stdout:2778: read oid 20 snap -1 2021-02-10T04:38:49.371 INFO:tasks.rados.rados.0.smithi085.stdout:2778: expect deleted 2021-02-10T04:38:49.371 INFO:tasks.rados.rados.0.smithi085.stdout:2779: delete oid 3 current snap is 316 2021-02-10T04:38:49.373 INFO:tasks.rados.rados.0.smithi085.stdout:2777: finishing copy_from to smithi08575397-43 2021-02-10T04:38:49.373 INFO:tasks.rados.rados.0.smithi085.stdout:2777: got expected ENOENT (src dne) 2021-02-10T04:38:49.373 INFO:tasks.rados.rados.0.smithi085.stdout:2777: finishing copy_from racing read to smithi08575397-43 2021-02-10T04:38:49.376 INFO:tasks.rados.rados.0.smithi085.stdout:2775: finishing write tid 2 to smithi08575397-21 2021-02-10T04:38:49.376 INFO:tasks.rados.rados.0.smithi085.stdout:2775: finishing write tid 3 to smithi08575397-21 2021-02-10T04:38:49.376 INFO:tasks.rados.rados.0.smithi085.stdout:2775: finishing write tid 4 to smithi08575397-21 2021-02-10T04:38:49.376 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 1456 (ObjNum 968 snap 316 seq_num 968) dirty exists 2021-02-10T04:38:49.376 INFO:tasks.rados.rados.0.smithi085.stdout:2775: left oid 21 (ObjNum 968 snap 316 seq_num 968) 2021-02-10T04:38:49.378 INFO:tasks.rados.rados.0.smithi085.stdout:2775: done (7 left) 2021-02-10T04:38:49.378 INFO:tasks.rados.rados.0.smithi085.stdout:2777: done (6 left) 2021-02-10T04:38:49.378 INFO:tasks.rados.rados.0.smithi085.stdout:2778: done (5 left) 2021-02-10T04:38:49.378 INFO:tasks.rados.rados.0.smithi085.stdout:2779: done (4 left) 2021-02-10T04:38:49.378 INFO:tasks.rados.rados.0.smithi085.stdout:2780: rollback oid 9 current snap is 316 2021-02-10T04:38:49.378 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 9 to 273 2021-02-10T04:38:49.378 INFO:tasks.rados.rados.0.smithi085.stdout:2781: read oid 21 snap -1 2021-02-10T04:38:49.378 INFO:tasks.rados.rados.0.smithi085.stdout:2781: expect (ObjNum 968 snap 316 seq_num 968) 2021-02-10T04:38:49.378 INFO:tasks.rados.rados.0.smithi085.stdout:2782: delete oid 45 current snap is 316 2021-02-10T04:38:49.384 INFO:tasks.rados.rados.0.smithi085.stdout:2780: finishing rollback tid 0 to smithi08575397-9 2021-02-10T04:38:49.384 INFO:tasks.rados.rados.0.smithi085.stdout:2780: finishing rollback tid 1 to smithi08575397-9 2021-02-10T04:38:49.385 INFO:tasks.rados.rados.0.smithi085.stdout:2780: finishing rollback tid 2 to smithi08575397-9 2021-02-10T04:38:49.385 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 9 v 1346 (ObjNum 85 snap 11 seq_num 85) dirty exists 2021-02-10T04:38:49.386 INFO:tasks.rados.rados.0.smithi085.stdout:2781: expect (ObjNum 968 snap 316 seq_num 968) 2021-02-10T04:38:49.618 INFO:tasks.rados.rados.0.smithi085.stdout:2780: done (6 left) 2021-02-10T04:38:49.618 INFO:tasks.rados.rados.0.smithi085.stdout:2781: done (5 left) 2021-02-10T04:38:49.618 INFO:tasks.rados.rados.0.smithi085.stdout:2782: done (4 left) 2021-02-10T04:38:49.618 INFO:tasks.rados.rados.0.smithi085.stdout:2783: write oid 13 current snap is 316 2021-02-10T04:38:49.618 INFO:tasks.rados.rados.0.smithi085.stdout:2783: seq_num 972 ranges {736404=718859,1929321=407083} 2021-02-10T04:38:49.627 INFO:tasks.rados.rados.0.smithi085.stdout:2783: writing smithi08575397-13 from 736404 to 1455263 tid 1 2021-02-10T04:38:49.632 INFO:tasks.rados.rados.0.smithi085.stdout:2783: writing smithi08575397-13 from 1929321 to 2336404 tid 2 2021-02-10T04:38:49.633 INFO:tasks.rados.rados.0.smithi085.stdout:2783: finishing write tid 1 to smithi08575397-13 2021-02-10T04:38:49.633 INFO:tasks.rados.rados.0.smithi085.stdout:2784: write (excl) oid 38 current snap is 316 2021-02-10T04:38:49.633 INFO:tasks.rados.rados.0.smithi085.stdout:2784: seq_num 973 ranges {753211=794108,1987803=365408} 2021-02-10T04:38:49.642 INFO:tasks.rados.rados.0.smithi085.stdout:2784: writing smithi08575397-38 from 753211 to 1547319 tid 1 2021-02-10T04:38:49.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:49 smithi164 conmon[29288]: audit 2021-02-10T04:38:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:49 smithi164 conmon[29288]: 2021-02-10T04:38:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:49 smithi164 conmon[29288]: 2021 2021-02-10T04:38:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:49 smithi164 conmon[29288]: - 2021-02-10T04:38:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:49 smithi164 conmon[29288]: 02-10T04:38: 2021-02-10T04:38:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:49 smithi164 conmon[29288]: 48.294674+0000 mgr.y (mgr.14140) 724 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:38:49.647 INFO:tasks.rados.rados.0.smithi085.stdout:2784: writing smithi08575397-38 from 1987803 to 2353211 tid 2 2021-02-10T04:38:49.648 INFO:tasks.rados.rados.0.smithi085.stdout:2785: delete oid 48 current snap is 316 2021-02-10T04:38:49.648 INFO:tasks.rados.rados.0.smithi085.stdout:2783: finishing write tid 2 to smithi08575397-13 2021-02-10T04:38:49.648 INFO:tasks.rados.rados.0.smithi085.stdout:2783: finishing write tid 3 to smithi08575397-13 2021-02-10T04:38:49.648 INFO:tasks.rados.rados.0.smithi085.stdout:2783: finishing write tid 4 to smithi08575397-13 2021-02-10T04:38:49.648 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 796 (ObjNum 972 snap 316 seq_num 972) dirty exists 2021-02-10T04:38:49.648 INFO:tasks.rados.rados.0.smithi085.stdout:2783: left oid 13 (ObjNum 972 snap 316 seq_num 972) 2021-02-10T04:38:49.648 INFO:tasks.rados.rados.0.smithi085.stdout:2784: finishing write tid 1 to smithi08575397-38 2021-02-10T04:38:49.651 INFO:tasks.rados.rados.0.smithi085.stdout:2783: done (6 left) 2021-02-10T04:38:49.651 INFO:tasks.rados.rados.0.smithi085.stdout:2785: done (5 left) 2021-02-10T04:38:49.651 INFO:tasks.rados.rados.0.smithi085.stdout:2786: write (excl) oid 44 current snap is 316 2021-02-10T04:38:49.651 INFO:tasks.rados.rados.0.smithi085.stdout:2786: seq_num 975 ranges {786825=660959,2104767=282058} 2021-02-10T04:38:49.659 INFO:tasks.rados.rados.0.smithi085.stdout:2786: writing smithi08575397-44 from 786825 to 1447784 tid 1 2021-02-10T04:38:49.665 INFO:tasks.rados.rados.0.smithi085.stdout:2786: writing smithi08575397-44 from 2104767 to 2386825 tid 2 2021-02-10T04:38:49.665 INFO:tasks.rados.rados.0.smithi085.stdout:2787: write (excl) oid 43 current snap is 316 2021-02-10T04:38:49.665 INFO:tasks.rados.rados.0.smithi085.stdout:2784: finishing write tid 2 to smithi08575397-38 2021-02-10T04:38:49.665 INFO:tasks.rados.rados.0.smithi085.stdout:2787: seq_num 976 ranges {403632=736208,1763249=619999} 2021-02-10T04:38:49.673 INFO:tasks.rados.rados.0.smithi085.stdout:2787: writing smithi08575397-43 from 403632 to 1139840 tid 1 2021-02-10T04:38:49.681 INFO:tasks.rados.rados.0.smithi085.stdout:2787: writing smithi08575397-43 from 1763249 to 2383248 tid 2 2021-02-10T04:38:49.681 INFO:tasks.rados.rados.0.smithi085.stdout:2788: write (excl) oid 25 current snap is 316 2021-02-10T04:38:49.681 INFO:tasks.rados.rados.0.smithi085.stdout:2784: finishing write tid 3 to smithi08575397-38 2021-02-10T04:38:49.681 INFO:tasks.rados.rados.0.smithi085.stdout:2784: finishing write tid 4 to smithi08575397-38 2021-02-10T04:38:49.681 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 1355 (ObjNum 973 snap 316 seq_num 973) dirty exists 2021-02-10T04:38:49.681 INFO:tasks.rados.rados.0.smithi085.stdout:2784: left oid 38 (ObjNum 973 snap 316 seq_num 973) 2021-02-10T04:38:49.681 INFO:tasks.rados.rados.0.smithi085.stdout:2786: finishing write tid 1 to smithi08575397-44 2021-02-10T04:38:49.682 INFO:tasks.rados.rados.0.smithi085.stdout:2788: seq_num 977 ranges {420439=411457,1421731=480010} 2021-02-10T04:38:49.685 INFO:tasks.rados.rados.0.smithi085.stdout:2788: writing smithi08575397-25 from 420439 to 831896 tid 1 2021-02-10T04:38:49.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:49 smithi085 conmon[29319]: audit 2021-02-10T04:38:48.294674+0000 mgr.y ( 2021-02-10T04:38:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:49 smithi085 conmon[29319]: mgr.14140) 724 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:38:49.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:49 smithi085 conmon[35961]: audit 2021-02-10T04:38:48.294674+0000 mgr.y (mgr.14140 2021-02-10T04:38:49.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:49 smithi085 conmon[35961]: ) 724 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:38:49.693 INFO:tasks.rados.rados.0.smithi085.stdout:2788: writing smithi08575397-25 from 1421731 to 1901741 tid 2 2021-02-10T04:38:49.693 INFO:tasks.rados.rados.0.smithi085.stdout:2784: done (7 left) 2021-02-10T04:38:49.693 INFO:tasks.rados.rados.0.smithi085.stdout:2789: read oid 8 snap -1 2021-02-10T04:38:49.694 INFO:tasks.rados.rados.0.smithi085.stdout:2789: expect (ObjNum 936 snap 310 seq_num 936) 2021-02-10T04:38:49.694 INFO:tasks.rados.rados.0.smithi085.stdout:2790: snap_create 2021-02-10T04:38:49.694 INFO:tasks.rados.rados.0.smithi085.stdout:2786: finishing write tid 2 to smithi08575397-44 2021-02-10T04:38:49.694 INFO:tasks.rados.rados.0.smithi085.stdout:2786: finishing write tid 3 to smithi08575397-44 2021-02-10T04:38:49.694 INFO:tasks.rados.rados.0.smithi085.stdout:2786: finishing write tid 4 to smithi08575397-44 2021-02-10T04:38:49.694 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 1148 (ObjNum 975 snap 316 seq_num 975) dirty exists 2021-02-10T04:38:49.694 INFO:tasks.rados.rados.0.smithi085.stdout:2786: left oid 44 (ObjNum 975 snap 316 seq_num 975) 2021-02-10T04:38:49.694 INFO:tasks.rados.rados.0.smithi085.stdout:2787: finishing write tid 1 to smithi08575397-43 2021-02-10T04:38:49.695 INFO:tasks.rados.rados.0.smithi085.stdout:2787: finishing write tid 2 to smithi08575397-43 2021-02-10T04:38:49.695 INFO:tasks.rados.rados.0.smithi085.stdout:2787: finishing write tid 3 to smithi08575397-43 2021-02-10T04:38:49.695 INFO:tasks.rados.rados.0.smithi085.stdout:2787: finishing write tid 4 to smithi08575397-43 2021-02-10T04:38:49.695 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 1190 (ObjNum 976 snap 316 seq_num 976) dirty exists 2021-02-10T04:38:49.695 INFO:tasks.rados.rados.0.smithi085.stdout:2787: left oid 43 (ObjNum 976 snap 316 seq_num 976) 2021-02-10T04:38:49.695 INFO:tasks.rados.rados.0.smithi085.stdout:2788: finishing write tid 1 to smithi08575397-25 2021-02-10T04:38:49.696 INFO:tasks.rados.rados.0.smithi085.stdout:2788: finishing write tid 2 to smithi08575397-25 2021-02-10T04:38:49.697 INFO:tasks.rados.rados.0.smithi085.stdout:2788: finishing write tid 3 to smithi08575397-25 2021-02-10T04:38:49.697 INFO:tasks.rados.rados.0.smithi085.stdout:2788: finishing write tid 4 to smithi08575397-25 2021-02-10T04:38:49.697 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 25 v 1286 (ObjNum 977 snap 316 seq_num 977) dirty exists 2021-02-10T04:38:49.697 INFO:tasks.rados.rados.0.smithi085.stdout:2788: left oid 25 (ObjNum 977 snap 316 seq_num 977) 2021-02-10T04:38:49.705 INFO:tasks.rados.rados.0.smithi085.stdout:2789: expect (ObjNum 936 snap 310 seq_num 936) 2021-02-10T04:38:50.191 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:38:52.664 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:50 smithi164 conmon[29288]: cluster 2021-02-10T04:38:49.338143+0000 mon.a (mon.0) 1617 : cluster [DBG] osdmap e710: 8 total, 8 up, 7 in 2021-02-10T04:38:52.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:50 smithi164 conmon[29288]: cluster 2021-02-10T04:38:50.164845+0000 mgr.y (mgr.14140) 725 : cluster [DBG] pgmap v1216: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 453 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 2.9 MiB/s wr, 23 op/s 2021-02-10T04:38:52.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:51 smithi164 conmon[29288]: cluster 2021-02-10T04:38:50.341633+0000 mon.a (mon.0) 1618 : 2021-02-10T04:38:52.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:51 smithi164 conmon[29288]: cluster [DBG] osdmap e711: 8 total, 8 up, 7 in 2021-02-10T04:38:52.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:52 smithi164 conmon[29288]: cluster 2021-02-10T04:38:51.066202+0000 osd.1 ( 2021-02-10T04:38:52.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:52 smithi164 conmon[29288]: osd.1) 455 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:38:52.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:52 smithi164 conmon[29288]: cluster 2021-02-10T04:38:51.067624+0000 osd.1 (osd.1) 456 : cluster [DBG] 2021-02-10T04:38:52.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:52 smithi164 conmon[29288]: purged_snaps scrub ok 2021-02-10T04:38:52.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:52 smithi164 conmon[29288]: cluster 2021-02-10T04:38:51.341121+0000 2021-02-10T04:38:52.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:52 smithi164 conmon[29288]: mon.a (mon.0) 1619 : cluster [DBG] osdmap e712: 8 total, 8 up, 7 in 2021-02-10T04:38:52.665 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:52 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:38:52.666 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:52 smithi164 conmon[29288]: 10T04:38:52.165338+0000 mgr.y (mgr.14140) 726 : 2021-02-10T04:38:52.666 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:52 smithi164 conmon[29288]: cluster [DBG] pgmap v1219: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 455 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.6 MiB/s rd, 3.1 MiB/s wr, 22 op/s 2021-02-10T04:38:52.666 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:38:52 smithi164 conmon[37263]: debug 2021-02-10T04:38:52.327+0000 7f1c6b1ea700 -1 osd.5 pg_epoch: 712 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x5566533aa840) 2021-02-10T04:38:52.666 INFO:tasks.rados.rados.0.smithi085.stdout:2786: done (8 left) 2021-02-10T04:38:52.666 INFO:tasks.rados.rados.0.smithi085.stdout:2787: done (7 left) 2021-02-10T04:38:52.666 INFO:tasks.rados.rados.0.smithi085.stdout:2788: done (6 left) 2021-02-10T04:38:52.666 INFO:tasks.rados.rados.0.smithi085.stdout:2789: done (5 left) 2021-02-10T04:38:52.666 INFO:tasks.rados.rados.0.smithi085.stdout:2790: done (4 left) 2021-02-10T04:38:52.666 INFO:tasks.rados.rados.0.smithi085.stdout:2791: rollback oid 20 current snap is 317 2021-02-10T04:38:52.666 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 20 to 315 2021-02-10T04:38:52.666 INFO:tasks.rados.rados.0.smithi085.stdout:2792: read oid 42 snap -1 2021-02-10T04:38:52.667 INFO:tasks.rados.rados.0.smithi085.stdout:2792: expect (ObjNum 833 snap 270 seq_num 833) 2021-02-10T04:38:52.667 INFO:tasks.rados.rados.0.smithi085.stdout:2793: copy_from oid 49 from oid 37 current snap is 317 2021-02-10T04:38:52.667 INFO:tasks.rados.rados.0.smithi085.stdout:2794: rollback oid 12 current snap is 317 2021-02-10T04:38:52.667 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 12 to 291 2021-02-10T04:38:52.667 INFO:tasks.rados.rados.0.smithi085.stdout:2795: delete oid 34 current snap is 317 2021-02-10T04:38:52.667 INFO:tasks.rados.rados.0.smithi085.stdout:2791: finishing rollback tid 1 to smithi08575397-20 2021-02-10T04:38:52.667 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 20 v 1145 (ObjNum 82255976 snap 22030 seq_num 82255976) dirty dne 2021-02-10T04:38:52.667 INFO:tasks.rados.rados.0.smithi085.stdout:2792: expect (ObjNum 833 snap 270 seq_num 833) 2021-02-10T04:38:52.667 INFO:tasks.rados.rados.0.smithi085.stdout:2791: done (8 left) 2021-02-10T04:38:52.667 INFO:tasks.rados.rados.0.smithi085.stdout:2792: done (7 left) 2021-02-10T04:38:52.667 INFO:tasks.rados.rados.0.smithi085.stdout:2795: done (6 left) 2021-02-10T04:38:52.667 INFO:tasks.rados.rados.0.smithi085.stdout:2796: rollback oid 40 current snap is 317 2021-02-10T04:38:52.667 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 40 to 313 2021-02-10T04:38:52.668 INFO:tasks.rados.rados.0.smithi085.stdout:2794: finishing rollback tid 0 to smithi08575397-12 2021-02-10T04:38:52.668 INFO:tasks.rados.rados.0.smithi085.stdout:2794: finishing rollback tid 1 to smithi08575397-12 2021-02-10T04:38:52.668 INFO:tasks.rados.rados.0.smithi085.stdout:2794: finishing rollback tid 2 to smithi08575397-12 2021-02-10T04:38:52.668 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 1350 (ObjNum 849 snap 276 seq_num 849) dirty exists 2021-02-10T04:38:52.668 INFO:tasks.rados.rados.0.smithi085.stdout:2794: done (6 left) 2021-02-10T04:38:52.668 INFO:tasks.rados.rados.0.smithi085.stdout:2797: delete oid 12 current snap is 317 2021-02-10T04:38:52.668 INFO:tasks.rados.rados.0.smithi085.stdout:2797: done (6 left) 2021-02-10T04:38:52.668 INFO:tasks.rados.rados.0.smithi085.stdout:2798: delete oid 10 current snap is 317 2021-02-10T04:38:52.668 INFO:tasks.rados.rados.0.smithi085.stdout:2793: finishing copy_from to smithi08575397-49 2021-02-10T04:38:52.668 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 49 v 1430 (ObjNum 940 snap 311 seq_num 940) dirty exists 2021-02-10T04:38:52.668 INFO:tasks.rados.rados.0.smithi085.stdout:2793: finishing copy_from racing read to smithi08575397-49 2021-02-10T04:38:52.668 INFO:tasks.rados.rados.0.smithi085.stdout:2796: finishing rollback tid 0 to smithi08575397-40 2021-02-10T04:38:52.668 INFO:tasks.rados.rados.0.smithi085.stdout:2796: finishing rollback tid 1 to smithi08575397-40 2021-02-10T04:38:52.669 INFO:tasks.rados.rados.0.smithi085.stdout:2793: done (6 left) 2021-02-10T04:38:52.669 INFO:tasks.rados.rados.0.smithi085.stdout:2798: done (5 left) 2021-02-10T04:38:52.669 INFO:tasks.rados.rados.0.smithi085.stdout:2799: write (excl) oid 42 current snap is 317 2021-02-10T04:38:52.669 INFO:tasks.rados.rados.0.smithi085.stdout:2796: finishing rollback tid 2 to smithi08575397-40 2021-02-10T04:38:52.669 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 1464 (ObjNum 879 snap 282 seq_num 879) dirty exists 2021-02-10T04:38:52.669 INFO:tasks.rados.rados.0.smithi085.stdout:2799: seq_num 981 ranges {487667=428806,1655659=770995} 2021-02-10T04:38:52.669 INFO:tasks.rados.rados.0.smithi085.stdout:2799: writing smithi08575397-42 from 487667 to 916473 tid 1 2021-02-10T04:38:52.669 INFO:tasks.rados.rados.0.smithi085.stdout:2799: writing smithi08575397-42 from 1655659 to 2426654 tid 2 2021-02-10T04:38:52.669 INFO:tasks.rados.rados.0.smithi085.stdout:2796: done (5 left) 2021-02-10T04:38:52.669 INFO:tasks.rados.rados.0.smithi085.stdout:2800: write (excl) oid 2 current snap is 317 2021-02-10T04:38:52.669 INFO:tasks.rados.rados.0.smithi085.stdout:2799: finishing write tid 1 to smithi08575397-42 2021-02-10T04:38:52.669 INFO:tasks.rados.rados.0.smithi085.stdout:2800: seq_num 982 ranges {504474=504055,1714141=631006} 2021-02-10T04:38:52.669 INFO:tasks.rados.rados.0.smithi085.stdout:2800: writing smithi08575397-2 from 504474 to 1008529 tid 1 2021-02-10T04:38:52.670 INFO:tasks.rados.rados.0.smithi085.stdout:2800: writing smithi08575397-2 from 1714141 to 2345147 tid 2 2021-02-10T04:38:52.670 INFO:tasks.rados.rados.0.smithi085.stdout:2801: snap_create 2021-02-10T04:38:52.670 INFO:tasks.rados.rados.0.smithi085.stdout:2799: finishing write tid 2 to smithi08575397-42 2021-02-10T04:38:52.670 INFO:tasks.rados.rados.0.smithi085.stdout:2799: finishing write tid 3 to smithi08575397-42 2021-02-10T04:38:52.670 INFO:tasks.rados.rados.0.smithi085.stdout:2799: finishing write tid 4 to smithi08575397-42 2021-02-10T04:38:52.670 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 1354 (ObjNum 981 snap 317 seq_num 981) dirty exists 2021-02-10T04:38:52.670 INFO:tasks.rados.rados.0.smithi085.stdout:2799: left oid 42 (ObjNum 981 snap 317 seq_num 981) 2021-02-10T04:38:52.670 INFO:tasks.rados.rados.0.smithi085.stdout:2800: finishing write tid 1 to smithi08575397-2 2021-02-10T04:38:52.670 INFO:tasks.rados.rados.0.smithi085.stdout:2800: finishing write tid 2 to smithi08575397-2 2021-02-10T04:38:52.670 INFO:tasks.rados.rados.0.smithi085.stdout:2800: finishing write tid 3 to smithi08575397-2 2021-02-10T04:38:52.670 INFO:tasks.rados.rados.0.smithi085.stdout:2800: finishing write tid 4 to smithi08575397-2 2021-02-10T04:38:52.670 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 1434 (ObjNum 982 snap 317 seq_num 982) dirty exists 2021-02-10T04:38:52.670 INFO:tasks.rados.rados.0.smithi085.stdout:2800: left oid 2 (ObjNum 982 snap 317 seq_num 982) 2021-02-10T04:38:52.671 INFO:tasks.rados.rados.0.smithi085.stdout:2799: done (6 left) 2021-02-10T04:38:52.671 INFO:tasks.rados.rados.0.smithi085.stdout:2800: done (5 left) 2021-02-10T04:38:52.671 INFO:tasks.rados.rados.0.smithi085.stdout:2801: done (4 left) 2021-02-10T04:38:52.671 INFO:tasks.rados.rados.0.smithi085.stdout:2802: delete oid 44 current snap is 318 2021-02-10T04:38:52.671 INFO:tasks.rados.rados.0.smithi085.stdout:2802: done (4 left) 2021-02-10T04:38:52.671 INFO:tasks.rados.rados.0.smithi085.stdout:2803: copy_from oid 21 from oid 15 current snap is 318 2021-02-10T04:38:52.671 INFO:tasks.rados.rados.0.smithi085.stdout:2804: snap_create 2021-02-10T04:38:52.671 INFO:tasks.rados.rados.0.smithi085.stdout:2803: finishing copy_from to smithi08575397-21 2021-02-10T04:38:52.671 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 1349 (ObjNum 962 snap 315 seq_num 962) dirty exists 2021-02-10T04:38:52.671 INFO:tasks.rados.rados.0.smithi085.stdout:2803: finishing copy_from racing read to smithi08575397-21 2021-02-10T04:38:52.671 INFO:tasks.rados.rados.0.smithi085.stdout:2803: done (5 left) 2021-02-10T04:38:52.671 INFO:tasks.rados.rados.0.smithi085.stdout:2804: done (4 left) 2021-02-10T04:38:52.671 INFO:tasks.rados.rados.0.smithi085.stdout:2805: snap_remove snap 315 2021-02-10T04:38:52.672 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:50 smithi085 conmon[29319]: cluster 2021-02-10T04:38:49.338143+0000 mon.a 2021-02-10T04:38:52.672 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:50 smithi085 conmon[29319]: (mon.0) 1617 : cluster [DBG] osdmap e710: 8 total, 8 up, 7 in 2021-02-10T04:38:52.672 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:50 smithi085 conmon[29319]: cluster 2021-02-10T04:38:50.164845+0000 mgr.y (mgr.14140) 725 : cluster [DBG] pgmap v1216: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 453 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 2.9 MiB/s wr, 23 op/s 2021-02-10T04:38:52.672 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:51 smithi085 conmon[29319]: cluster 2021-02-10T04:38:52.672 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:51 smithi085 conmon[29319]: 2021-02-10T04:38:52.672 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:51 smithi085 conmon[29319]: 2021 2021-02-10T04:38:52.672 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:51 smithi085 conmon[29319]: - 2021-02-10T04:38:52.672 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:51 smithi085 conmon[29319]: 02 2021-02-10T04:38:52.672 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:51 smithi085 conmon[29319]: - 2021-02-10T04:38:52.672 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:51 smithi085 conmon[29319]: 10T04:38:50.341633+0000 2021-02-10T04:38:52.672 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:51 smithi085 conmon[29319]: mon.a (mon.0) 1618 : cluster [DBG] osdmap e711: 8 total, 8 up, 7 in 2021-02-10T04:38:52.672 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:50 smithi085 conmon[35961]: cluster 2021-02-10T04:38:49.338143+0000 mon.a 2021-02-10T04:38:52.673 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:50 smithi085 conmon[35961]: (mon.0) 1617 : cluster [DBG] osdmap e710: 8 total, 8 up, 7 in 2021-02-10T04:38:52.673 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:50 smithi085 conmon[35961]: cluster 2021-02-10T04:38:50.164845 2021-02-10T04:38:52.673 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:50 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 725 : cluster [DBG] pgmap v1216: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 453 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 2.9 MiB/s wr, 23 op/s 2021-02-10T04:38:52.673 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:51 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:38:52.673 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:51 smithi085 conmon[35961]: T04:38:50.341633+0000 mon.a (mon.0) 1618 : cluster [DBG] osdmap e711: 8 total, 8 up, 7 in 2021-02-10T04:38:52.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:52 smithi085 conmon[35961]: cluster 2021-02-10T04:38:51.066202 2021-02-10T04:38:52.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:52 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 455 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:38:52.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:52 smithi085 conmon[35961]: cluster 2021-02-10T04:38:51.067624 2021-02-10T04:38:52.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:52 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 456 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:38:52.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:52 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:38:52.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:52 smithi085 conmon[35961]: :38:51.341121+0000 mon.a (mon.0) 1619 : 2021-02-10T04:38:52.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:52 smithi085 conmon[35961]: cluster [DBG] osdmap e712: 8 total, 8 up, 7 in 2021-02-10T04:38:52.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:52 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:38:52.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:52 smithi085 conmon[35961]: :38:52.165338+0000 mgr.y (mgr.14140 2021-02-10T04:38:52.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:52 smithi085 conmon[35961]: ) 726 : cluster [DBG] pgmap v1219: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 455 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.6 MiB/s rd, 3.1 MiB/s wr, 22 op/s 2021-02-10T04:38:52.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:52 smithi085 conmon[29319]: cluster 2021-02-10T04:38:51.066202+0000 osd.1 (osd 2021-02-10T04:38:52.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:52 smithi085 conmon[29319]: .1) 455 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:38:52.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:52 smithi085 conmon[29319]: cluster 2021-02-10T04:38:51.067624+0000 osd.1 (osd.1) 456 : cluster 2021-02-10T04:38:52.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:52 smithi085 conmon[29319]: [DBG] purged_snaps scrub ok 2021-02-10T04:38:52.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:52 smithi085 conmon[29319]: cluster 2021-02-10T04:38:51.341121+0000 mon.a (mon.0) 1619 : cluster [DBG] osdmap e712: 8 total, 8 up, 7 in 2021-02-10T04:38:52.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:52 smithi085 conmon[29319]: cluster 2021-02-10T04:38:52.165338+0000 mgr.y (mgr.14140) 726 : cluster [DBG] pgmap v1219: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 455 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.6 MiB/s rd, 3.1 MiB/s wr, 22 op/s 2021-02-10T04:38:52.774 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:38:52.811 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:38:53.237 INFO:tasks.rados.rados.0.smithi085.stdout:2805: done (4 left) 2021-02-10T04:38:53.237 INFO:tasks.rados.rados.0.smithi085.stdout:2806: read oid 23 snap 316 2021-02-10T04:38:53.237 INFO:tasks.rados.rados.0.smithi085.stdout:2806: expect (ObjNum 924 snap 306 seq_num 924) 2021-02-10T04:38:53.237 INFO:tasks.rados.rados.0.smithi085.stdout:2807: delete oid 5 current snap is 319 2021-02-10T04:38:53.246 INFO:tasks.rados.rados.0.smithi085.stdout:2807: done (5 left) 2021-02-10T04:38:53.246 INFO:tasks.rados.rados.0.smithi085.stdout:2808: delete oid 13 current snap is 319 2021-02-10T04:38:53.253 INFO:tasks.rados.rados.0.smithi085.stdout:2806: expect (ObjNum 924 snap 306 seq_num 924) 2021-02-10T04:38:53.407 INFO:tasks.rados.rados.0.smithi085.stdout:2806: done (5 left) 2021-02-10T04:38:53.407 INFO:tasks.rados.rados.0.smithi085.stdout:2808: done (4 left) 2021-02-10T04:38:53.407 INFO:tasks.rados.rados.0.smithi085.stdout:2809: snap_remove snap 274 2021-02-10T04:38:53.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:53 smithi164 conmon[29288]: cluster 2021-02-10T04:38:52.357525 2021-02-10T04:38:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:53 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1620 : cluster [DBG] osdmap e713: 8 total, 8 up, 7 in 2021-02-10T04:38:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:53 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:38:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:53 smithi164 conmon[29288]: 10T04:38:52.497890+0000 mon.a (mon.0) 1621 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:38:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:53 smithi164 conmon[29288]: cephadm 2021-02- 2021-02-10T04:38:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:53 smithi164 conmon[29288]: 10T04:38:52.498923+0000 mgr.y (mgr.14140) 727 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:38:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:53 smithi164 conmon[29288]: 2021-02-10T04:38:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:53 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:38:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:53 smithi164 conmon[29288]: -10T04:38:53.091852+0000 mon.a (mon.0) 1622 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:38:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:53 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:38:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:53 smithi164 conmon[29288]: 02-10T04:38:53. 2021-02-10T04:38:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:53 smithi164 conmon[29288]: 235552+0000 mon.a (mon.0 2021-02-10T04:38:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:53 smithi164 conmon[29288]: ) 1623 : cluster [DBG] osdmap e714: 8 total, 8 up, 7 in 2021-02-10T04:38:53.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[29319]: cluster 2021-02-10T04:38:52.357525+0000 mon.a (mon. 2021-02-10T04:38:53.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[29319]: 0) 1620 : cluster [DBG] osdmap e713: 8 total, 8 up, 7 in 2021-02-10T04:38:53.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[29319]: audit 2021-02-10T04:38:52.497890+0000 mon.a (mon.0) 1621 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:38:53.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[29319]: cephadm 2021-02-10T04:38:52.498923+0000 mgr.y (mgr.14140) 727 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:38:53.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[29319]: audit 2021-02-10T04:38:53.091852+0000 mon.a (mon.0) 1622 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:38:53.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[29319]: cluster 2021-02-10T04:38:53.235552+0000 mon.a (mon.0) 1623 : cluster [DBG] osdmap e714: 8 total, 8 up, 7 in 2021-02-10T04:38:53.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: cluster 2021-02-10T04:38:53.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: 2021-02-10T04:38:53.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: 2021 2021-02-10T04:38:53.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: - 2021-02-10T04:38:53.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: 02 2021-02-10T04:38:53.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: - 2021-02-10T04:38:53.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: 10 2021-02-10T04:38:53.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: T 2021-02-10T04:38:53.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: 04 2021-02-10T04:38:53.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: : 2021-02-10T04:38:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: 38 2021-02-10T04:38:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: : 2021-02-10T04:38:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: 52 2021-02-10T04:38:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: . 2021-02-10T04:38:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: 357525+0000 2021-02-10T04:38:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: mon.a (mon.0) 1620 : cluster [DBG] osdmap e713: 8 total, 8 up, 7 in 2021-02-10T04:38:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: audit 2021-02-10T04:38:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:38:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: 10T04:38: 2021-02-10T04:38:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: 52.497890+0000 2021-02-10T04:38:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: mon.a (mon.0) 1621 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:38:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: cephadm 2021- 2021-02-10T04:38:53.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: 02-10T04:38:52.498923+0000 mgr.y (mgr.14140) 727 : cephadm [INF] 2021-02-10T04:38:53.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: Refreshing smithi085 facts 2021-02-10T04:38:53.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: audit 2021-02-10T04:38:53.091852+0000 mon.a (mon.0) 1622 : 2021-02-10T04:38:53.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:38:53.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: cluster 2021-02-10T04:38:53.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: 2021 2021-02-10T04:38:53.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: - 2021-02-10T04:38:53.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: 02 2021-02-10T04:38:53.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: - 2021-02-10T04:38:53.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: 10T04:38:53.235552+0000 2021-02-10T04:38:53.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:53 smithi085 conmon[35961]: mon.a (mon.0) 1623 : cluster [DBG] osdmap e714: 8 total, 8 up, 7 in 2021-02-10T04:38:54.363 INFO:tasks.rados.rados.0.smithi085.stdout:2809: done (4 left) 2021-02-10T04:38:54.363 INFO:tasks.rados.rados.0.smithi085.stdout:2810: snap_create 2021-02-10T04:38:54.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:54 smithi164 conmon[29288]: cluster 2021-02-10T04:38:54.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:54 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:38:54.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:54 smithi164 conmon[29288]: 04:38:54.165896+0000 mgr.y (mgr.14140) 728 : cluster [DBG] pgmap v1222: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 468 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 3.3 MiB/s wr, 29 op/s 2021-02-10T04:38:54.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:54 smithi085 conmon[29319]: cluster 2021-02-10T04:38:54.165896+0000 mgr.y (mgr.14140) 728 : cluster [DBG] pgmap v1222: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 468 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 3.3 MiB/s wr, 29 op/s 2021-02-10T04:38:54.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:54 smithi085 conmon[35961]: cluster 2021-02-10T04:38:54.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:54 smithi085 conmon[35961]: 2021-02-10T04:38:54.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:54 smithi085 conmon[35961]: 2021 2021-02-10T04:38:54.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:54 smithi085 conmon[35961]: - 2021-02-10T04:38:54.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:54 smithi085 conmon[35961]: 02 2021-02-10T04:38:54.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:54 smithi085 conmon[35961]: - 2021-02-10T04:38:54.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:54 smithi085 conmon[35961]: 10 2021-02-10T04:38:54.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:54 smithi085 conmon[35961]: T 2021-02-10T04:38:54.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:54 smithi085 conmon[35961]: 04 2021-02-10T04:38:54.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:54 smithi085 conmon[35961]: : 2021-02-10T04:38:54.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:54 smithi085 conmon[35961]: 38 2021-02-10T04:38:54.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:54 smithi085 conmon[35961]: : 2021-02-10T04:38:54.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:54 smithi085 conmon[35961]: 54 2021-02-10T04:38:54.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:54 smithi085 conmon[35961]: .165896+0000 mgr.y (mgr.14140 2021-02-10T04:38:54.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:54 smithi085 conmon[35961]: ) 728 : cluster [DBG] pgmap v1222: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 468 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 3.3 MiB/s wr, 29 op/s 2021-02-10T04:38:55.173 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:38:55.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:38:54 smithi085 conmon[52836]: debug 2021-02-10T04:38:55.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:38:54 smithi085 conmon[52836]: 2021-02-10T04:38:54.887+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:38:55.192 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:38:54 smithi085 conmon[52836]: 2021-02-10T04:38:55.368 INFO:tasks.rados.rados.0.smithi085.stdout:2810: done (4 left) 2021-02-10T04:38:55.368 INFO:tasks.rados.rados.0.smithi085.stdout:2811: read oid 18 snap -1 2021-02-10T04:38:55.368 INFO:tasks.rados.rados.0.smithi085.stdout:2811: expect (ObjNum 952 snap 313 seq_num 952) 2021-02-10T04:38:55.372 INFO:tasks.rados.rados.0.smithi085.stdout:2812: write oid 32 current snap is 320 2021-02-10T04:38:55.372 INFO:tasks.rados.rados.0.smithi085.stdout:2812: seq_num 986 ranges {571702=405051,1431716=638344,2560365=11337} 2021-02-10T04:38:55.374 INFO:tasks.rados.rados.0.smithi085.stdout:2812: writing smithi08575397-32 from 571702 to 976753 tid 1 2021-02-10T04:38:55.381 INFO:tasks.rados.rados.0.smithi085.stdout:2812: writing smithi08575397-32 from 1431716 to 2070060 tid 2 2021-02-10T04:38:55.384 INFO:tasks.rados.rados.0.smithi085.stdout:2812: writing smithi08575397-32 from 2560365 to 2571702 tid 3 2021-02-10T04:38:55.384 INFO:tasks.rados.rados.0.smithi085.stdout:2813: snap_remove snap 273 2021-02-10T04:38:55.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:55 smithi164 conmon[29288]: cluster 2021-02-10T04:38:54.363483+0000 2021-02-10T04:38:55.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:55 smithi164 conmon[29288]: mon.a (mon.0) 1624 : cluster [DBG] osdmap e715: 8 total, 8 up, 7 in 2021-02-10T04:38:55.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:55 smithi085 conmon[35961]: cluster 2021-02-10T04:38:54.363483 2021-02-10T04:38:55.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:55 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1624 : cluster [DBG] osdmap e715: 8 total, 8 up, 7 in 2021-02-10T04:38:55.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:55 smithi085 conmon[29319]: cluster 2021-02-10T04:38:55.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:55 smithi085 conmon[29319]: 2021-02-10T04:38:54 2021-02-10T04:38:55.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:55 smithi085 conmon[29319]: .363483+0000 mon.a (mon.0) 1624 : cluster [DBG] osdmap e715: 8 total, 8 up, 7 in 2021-02-10T04:38:56.392 INFO:tasks.rados.rados.0.smithi085.stdout:2813: done (6 left) 2021-02-10T04:38:56.393 INFO:tasks.rados.rados.0.smithi085.stdout:2814: snap_create 2021-02-10T04:38:56.393 INFO:tasks.rados.rados.0.smithi085.stdout:2811: expect (ObjNum 952 snap 313 seq_num 952) 2021-02-10T04:38:56.578 INFO:tasks.rados.rados.0.smithi085.stdout:2812: finishing write tid 1 to smithi08575397-32 2021-02-10T04:38:56.578 INFO:tasks.rados.rados.0.smithi085.stdout:2812: finishing write tid 2 to smithi08575397-32 2021-02-10T04:38:56.578 INFO:tasks.rados.rados.0.smithi085.stdout:2812: finishing write tid 3 to smithi08575397-32 2021-02-10T04:38:56.578 INFO:tasks.rados.rados.0.smithi085.stdout:2812: finishing write tid 4 to smithi08575397-32 2021-02-10T04:38:56.578 INFO:tasks.rados.rados.0.smithi085.stdout:2812: finishing write tid 5 to smithi08575397-32 2021-02-10T04:38:56.578 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 32 v 1353 (ObjNum 986 snap 320 seq_num 986) dirty exists 2021-02-10T04:38:56.578 INFO:tasks.rados.rados.0.smithi085.stdout:2812: left oid 32 (ObjNum 986 snap 320 seq_num 986) 2021-02-10T04:38:56.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:56 smithi164 conmon[29288]: cluster 2021-02-10T04:38:55.367190+0000 mon.a (mon.0 2021-02-10T04:38:56.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:56 smithi164 conmon[29288]: ) 1625 : cluster [DBG] osdmap e716: 8 total, 8 up, 7 in 2021-02-10T04:38:56.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:56 smithi164 conmon[29288]: cluster 2021-02-10T04:38:56.166331+0000 2021-02-10T04:38:56.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:56 smithi164 conmon[29288]: mgr.y (mgr.14140) 729 : cluster [DBG] pgmap v1225: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 470 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 16 MiB/s rd, 3.3 MiB/s wr, 27 op/s 2021-02-10T04:38:56.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:56 smithi085 conmon[35961]: cluster 2021-02-10T04:38:55.367190 2021-02-10T04:38:56.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:56 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1625 : cluster [DBG] osdmap e716: 8 total, 8 up, 7 in 2021-02-10T04:38:56.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:56 smithi085 conmon[35961]: cluster 2021-02-10T04:38:56.166331+0000 2021-02-10T04:38:56.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:56 smithi085 conmon[35961]: mgr.y (mgr.14140) 729 : cluster [DBG] pgmap v1225: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 470 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 16 MiB/s rd, 3.3 MiB/s wr, 27 op/s 2021-02-10T04:38:56.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:56 smithi085 conmon[29319]: cluster 2021-02-10T04:38:56.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:56 smithi085 conmon[29319]: 2021-02-10T04:38:56.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:56 smithi085 conmon[29319]: 2021-02-10T04:38: 2021-02-10T04:38:56.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:56 smithi085 conmon[29319]: 55.367190+0000 mon.a (mon.0) 1625 : cluster [DBG] osdmap e716: 8 total, 8 up, 7 in 2021-02-10T04:38:56.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:56 smithi085 conmon[29319]: cluster 2021-02-10T04:38:56.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:56 smithi085 conmon[29319]: 2021-02-10T04:38:56.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:56 smithi085 conmon[29319]: 2021 2021-02-10T04:38:56.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:56 smithi085 conmon[29319]: - 2021-02-10T04:38:56.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:56 smithi085 conmon[29319]: 02-10T04:38:56.166331 2021-02-10T04:38:56.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:56 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 729 : cluster [DBG] pgmap v1225: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 470 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 16 MiB/s rd, 3.3 MiB/s wr, 27 op/s 2021-02-10T04:38:56.903 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:38:56.903 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:38:57.001 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:38:57.040 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:38:57.392 INFO:tasks.rados.rados.0.smithi085.stdout:2811: done (6 left) 2021-02-10T04:38:57.392 INFO:tasks.rados.rados.0.smithi085.stdout:2812: done (5 left) 2021-02-10T04:38:57.393 INFO:tasks.rados.rados.0.smithi085.stdout:2814: done (4 left) 2021-02-10T04:38:57.393 INFO:tasks.rados.rados.0.smithi085.stdout:2815: snap_create 2021-02-10T04:38:57.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:57 smithi085 conmon[35961]: cluster 2021-02-10T04:38:56.392623+0000 mon.a (mon 2021-02-10T04:38:57.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:57 smithi085 conmon[35961]: .0) 1626 : cluster [DBG] osdmap e717: 8 total, 8 up, 7 in 2021-02-10T04:38:57.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:57 smithi085 conmon[29319]: cluster 2021-02-10T04:38:56.392623+0000 mon.a (mon.0) 1626 : cluster [DBG] osdmap e717: 8 total, 8 up, 7 in 2021-02-10T04:38:57.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:57 smithi164 conmon[29288]: cluster 2021-02-10T04:38:56.392623+0000 2021-02-10T04:38:57.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:57 smithi164 conmon[29288]: mon.a (mon.0) 1626 : cluster [DBG] osdmap e717: 8 total, 8 up, 7 in 2021-02-10T04:38:58.235 INFO:tasks.rados.rados.0.smithi085.stdout:2815: done (4 left) 2021-02-10T04:38:58.236 INFO:tasks.rados.rados.0.smithi085.stdout:2816: copy_from oid 42 from oid 44 current snap is 322 2021-02-10T04:38:58.236 INFO:tasks.rados.rados.0.smithi085.stdout:2817: read oid 45 snap -1 2021-02-10T04:38:58.236 INFO:tasks.rados.rados.0.smithi085.stdout:2817: expect deleted 2021-02-10T04:38:58.236 INFO:tasks.rados.rados.0.smithi085.stdout:2818: read oid 4 snap -1 2021-02-10T04:38:58.236 INFO:tasks.rados.rados.0.smithi085.stdout:2818: expect deleted 2021-02-10T04:38:58.236 INFO:tasks.rados.rados.0.smithi085.stdout:2819: read oid 32 snap -1 2021-02-10T04:38:58.236 INFO:tasks.rados.rados.0.smithi085.stdout:2819: expect (ObjNum 986 snap 320 seq_num 986) 2021-02-10T04:38:58.236 INFO:tasks.rados.rados.0.smithi085.stdout:2820: read oid 3 snap -1 2021-02-10T04:38:58.236 INFO:tasks.rados.rados.0.smithi085.stdout:2820: expect deleted 2021-02-10T04:38:58.236 INFO:tasks.rados.rados.0.smithi085.stdout:2821: rollback oid 46 current snap is 322 2021-02-10T04:38:58.236 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 46 to 272 2021-02-10T04:38:58.236 INFO:tasks.rados.rados.0.smithi085.stdout:2822: snap_create 2021-02-10T04:38:58.248 INFO:tasks.rados.rados.0.smithi085.stdout:2816: finishing copy_from to smithi08575397-42 2021-02-10T04:38:58.248 INFO:tasks.rados.rados.0.smithi085.stdout:2816: got expected ENOENT (src dne) 2021-02-10T04:38:58.248 INFO:tasks.rados.rados.0.smithi085.stdout:2816: finishing copy_from racing read to smithi08575397-42 2021-02-10T04:38:58.248 INFO:tasks.rados.rados.0.smithi085.stdout:2821: finishing rollback tid 1 to smithi08575397-46 2021-02-10T04:38:58.249 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 1352 (ObjNum 71096184 snap 22030 seq_num 1) dirty dne 2021-02-10T04:38:58.252 INFO:tasks.rados.rados.0.smithi085.stdout:2819: expect (ObjNum 986 snap 320 seq_num 986) 2021-02-10T04:38:58.644 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:38:58 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:38:58.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:58 smithi164 conmon[29288]: cluster 2021-02-10T04:38:57.391819+0000 mon.a (mon.0) 1627 : cluster 2021-02-10T04:38:58.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:58 smithi164 conmon[29288]: [DBG] osdmap e718: 8 total, 8 up, 7 in 2021-02-10T04:38:58.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:58 smithi164 conmon[29288]: cluster 2021-02-10T04:38:58.166927+0000 mgr.y (mgr.14140) 730 : cluster [DBG] pgmap v1228: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 471 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 3.4 MiB/s rd, 1.4 MiB/s wr, 6 op/s 2021-02-10T04:38:58.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:58 smithi164 conmon[29288]: cluster 2021-02-10T04:38:58. 2021-02-10T04:38:58.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:58 smithi164 conmon[29288]: 232871+0000 mon.a (mon.0) 1628 : cluster [DBG] osdmap e719: 8 total, 8 up, 7 in 2021-02-10T04:38:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[29319]: cluster 2021-02-10T04:38:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[29319]: 2021-02-10T04:38:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[29319]: 2021 2021-02-10T04:38:58.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[29319]: - 2021-02-10T04:38:58.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[29319]: 02 2021-02-10T04:38:58.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[29319]: - 2021-02-10T04:38:58.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[29319]: 10 2021-02-10T04:38:58.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[29319]: T 2021-02-10T04:38:58.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[29319]: 04:38:57.391819 2021-02-10T04:38:58.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1627 : cluster [DBG] osdmap e718: 8 total, 8 up, 7 in 2021-02-10T04:38:58.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:38:58.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[29319]: 02-10T04:38:58.166927+0000 mgr.y (mgr. 2021-02-10T04:38:58.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[29319]: 14140) 730 : cluster [DBG] pgmap v1228: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 471 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 3.4 MiB/s rd, 1.4 MiB/s wr, 6 op/s 2021-02-10T04:38:58.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:38:58.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[29319]: 04:38:58.232871+0000 mon.a (mon.0) 1628 : cluster 2021-02-10T04:38:58.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[29319]: [DBG] osdmap e719: 8 total, 8 up, 7 in 2021-02-10T04:38:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[35961]: cluster 2021-02-10T04:38:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[35961]: 2021-02-10T04:38:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[35961]: 2021 2021-02-10T04:38:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[35961]: - 2021-02-10T04:38:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[35961]: 02 2021-02-10T04:38:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[35961]: - 2021-02-10T04:38:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[35961]: 10 2021-02-10T04:38:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[35961]: T 2021-02-10T04:38:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[35961]: 04:38:57.391819+0000 2021-02-10T04:38:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[35961]: mon.a (mon.0) 1627 : cluster [DBG] osdmap e718: 8 total, 8 up, 7 in 2021-02-10T04:38:58.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:38:58.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[35961]: -02-10T04:38:58.166927+0000 2021-02-10T04:38:58.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[35961]: mgr.y (mgr.14140) 730 : cluster [DBG] pgmap v1228: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 471 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 3.4 MiB/s rd, 1.4 MiB/s wr, 6 op/s 2021-02-10T04:38:58.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[35961]: cluster 2021-02-10T04:38: 2021-02-10T04:38:58.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[35961]: 58.232871+0000 mon.a (mon.0) 1628 : cluster 2021-02-10T04:38:58.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:58 smithi085 conmon[35961]: [DBG] osdmap e719: 8 total, 8 up, 7 in 2021-02-10T04:38:59.234 INFO:tasks.rados.rados.0.smithi085.stdout:2816: done (10 left) 2021-02-10T04:38:59.234 INFO:tasks.rados.rados.0.smithi085.stdout:2817: done (9 left) 2021-02-10T04:38:59.235 INFO:tasks.rados.rados.0.smithi085.stdout:2818: done (8 left) 2021-02-10T04:38:59.235 INFO:tasks.rados.rados.0.smithi085.stdout:2819: done (7 left) 2021-02-10T04:38:59.235 INFO:tasks.rados.rados.0.smithi085.stdout:2820: done (6 left) 2021-02-10T04:38:59.235 INFO:tasks.rados.rados.0.smithi085.stdout:2821: done (5 left) 2021-02-10T04:38:59.235 INFO:tasks.rados.rados.0.smithi085.stdout:2822: done (4 left) 2021-02-10T04:38:59.235 INFO:tasks.rados.rados.0.smithi085.stdout:2823: snap_create 2021-02-10T04:38:59.399 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:38:59 smithi085 conmon[52836]: debug 2021-02-10T04:38:59.397+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:38:59.460 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:38:59.528 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight 2021-02-10T04:38:59.569 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:38:59.601 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:38:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:59 smithi085 conmon[29319]: audit 2021-02-10T04:38:58.305010+0000 mgr.y (mgr.14140) 2021-02-10T04:38:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:59 smithi085 conmon[29319]: 731 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:38:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:38:59 smithi085 conmon[29319]: cluster 2021-02-10T04:38:59.234516+0000 mon.a (mon.0) 1629 : cluster [DBG] osdmap e720: 8 total, 8 up, 7 in 2021-02-10T04:38:59.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:59 smithi085 conmon[35961]: audit 2021-02-10T04:38:58.305010 2021-02-10T04:38:59.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:59 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 731 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:38:59.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:59 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:38:59.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:59 smithi085 conmon[35961]: :38:59.234516+0000 mon.a (mon.0 2021-02-10T04:38:59.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:38:59 smithi085 conmon[35961]: ) 1629 : cluster [DBG] osdmap e720: 8 total, 8 up, 7 in 2021-02-10T04:38:59.843 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:38:59.893 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:38:59 smithi164 conmon[37263]: debug 2021-02-10T04:38:59.438+0000 7f1c671e2700 -1 osd.5 pg_epoch: 720 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x556655936000) 2021-02-10T04:38:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:59 smithi164 conmon[29288]: audit 2021-02-10T04:38:58.305010+0000 mgr.y (mgr.14140) 731 2021-02-10T04:38:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:59 smithi164 conmon[29288]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:38:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:38:59 smithi164 conmon[29288]: cluster 2021-02-10T04:38:59.234516+0000 mon.a (mon.0) 1629 : cluster [DBG] osdmap e720: 8 total, 8 up, 7 in 2021-02-10T04:39:00.241 INFO:tasks.rados.rados.0.smithi085.stdout:2823: done (4 left) 2021-02-10T04:39:00.475 INFO:tasks.rados.rados.0.smithi085.stdout:2824: write (excl) oid 6 current snap is 324 2021-02-10T04:39:00.475 INFO:tasks.rados.rados.0.smithi085.stdout:2824: seq_num 987 ranges {588509=480300,1773845=782002} 2021-02-10T04:39:00.477 INFO:tasks.rados.rados.0.smithi085.stdout:2824: writing smithi08575397-6 from 588509 to 1068809 tid 1 2021-02-10T04:39:00.477 INFO:tasks.rados.rados.0.smithi085.stdout:2824: writing smithi08575397-6 from 1773845 to 2555847 tid 2 2021-02-10T04:39:00.477 INFO:tasks.rados.rados.0.smithi085.stdout:2825: snap_remove snap 319 2021-02-10T04:39:00.477 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:39:00.478 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:39:00 smithi085 conmon[52836]: debug 2021-02-10T04:39:00.190+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:00.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:00 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:39:00.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:00 smithi085 conmon[29319]: 04:39:00.167550+0000 mgr.y (mgr.14140) 732 : cluster [DBG] pgmap v1231: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 470 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.4 MiB/s rd, 258 KiB/s wr, 5 op/s 2021-02-10T04:39:00.478 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:00 smithi085 conmon[29319]: cluster 2021-02-10T04:39:00.235609+0000 mon.a (mon.0) 1630 : cluster [DBG] osdmap e721: 8 total, 8 up, 7 in 2021-02-10T04:39:00.478 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:00 smithi085 conmon[35961]: cluster 2021-02-10T04:39:00.167550 2021-02-10T04:39:00.478 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:00 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 732 : cluster [DBG] pgmap v1231: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 470 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.4 MiB/s rd, 258 KiB/s wr, 5 op/s 2021-02-10T04:39:00.478 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:00 smithi085 conmon[35961]: cluster 2021-02-10T04:39:00.479 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:00 smithi085 conmon[35961]: 2021-02-10T04:39:00.235609+0000 mon.a (mon. 2021-02-10T04:39:00.479 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:00 smithi085 conmon[35961]: 0) 1630 : cluster [DBG] osdmap e721: 8 total, 8 up, 7 in 2021-02-10T04:39:00.479 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:39:00 smithi085 conmon[41540]: debug 2021-02-10T04:39:00.263+0000 7fb94bb63700 -1 osd.0 pg_epoch: 721 pg[5.5( empty local-lis/les=57/58 n=0 ec=57/57 lis/c=57/57 les/c/f=58/58/0 sis=57) [0,1,4] r=0 lpr=57 crt=0'0 mlcod 0'0 active+clean] scrubber pg(5.5) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x555d32bf8000) 2021-02-10T04:39:00.578 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:39:00.607 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:39:00.682 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:00 smithi164 conmon[29288]: cluster 2021-02-10T04:39:00.167550+0000 mgr.y (mgr.14140) 732 : cluster [DBG] 2021-02-10T04:39:00.683 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:00 smithi164 conmon[29288]: pgmap v1231: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 470 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.4 MiB/s rd, 258 KiB/s wr, 5 op/s 2021-02-10T04:39:00.683 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:00 smithi164 conmon[29288]: cluster 2021-02-10T04:39:00.235609+0000 mon.a (mon.0) 1630 : cluster [DBG] osdmap e721: 8 total, 8 up, 7 in 2021-02-10T04:39:00.772 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:39:00.804 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:39:00.835 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:39:00.870 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:39:00.903 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:39:00.924 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:39:00.947 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:39:00.980 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:39:01.013 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:39:01.018 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:39:01.018 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:39:01.018 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:39:01.018 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:39:01.018 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:39:01.018 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:39:01.019 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:39:01.019 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:39:01.025 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:39:01.062 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:39:01.143 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:39:00 smithi164 conmon[40892]: debug 2021-02-10T04:39:00.681+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:01.241 INFO:tasks.rados.rados.0.smithi085.stdout:2825: done (5 left) 2021-02-10T04:39:01.241 INFO:tasks.rados.rados.0.smithi085.stdout:2826: snap_remove snap 320 2021-02-10T04:39:01.247 INFO:teuthology.orchestra.run.smithi085.stderr:set 4.9 pg_upmap_items mapping to [1->6,5->0,3->7] 2021-02-10T04:39:01.249 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is unset 2021-02-10T04:39:01.416 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[45443]: debug 2021-02-10T04:39:01.416 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[45443]: 2021-02-10T04:39:01.238+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:01.417 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[45443]: 2021-02-10T04:39:01.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[29319]: audit 2021-02-10T04:39:01.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:39:01.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[29319]: 39:00.736361+0000 mon.c (mon.1) 62 : audit [DBG] from='client.? 172.21.15.85:0/598757740' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:39:01.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:39:01.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[29319]: T04:39:00.953491+0000 mon.c (mon.1) 2021-02-10T04:39:01.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[29319]: 63 : audit [INF] from='client.? 172.21.15.85:0/2372737071' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:39:01.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[29319]: audit 2021-02-10T04:39:00. 2021-02-10T04:39:01.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[29319]: 953994+0000 mon.a (mon.0) 1631 : audit [INF] 2021-02-10T04:39:01.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[29319]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:39:01.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[29319]: audit 2021-02-10T04:39:01.417 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[29319]: 2021-02-10T04:39:00.967058+0000 mon.c ( 2021-02-10T04:39:01.418 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[29319]: mon.1) 64 : audit [INF] from='client.? 172.21.15.85:0/3149241954' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.9", "id": [1, 6, 5, 0, 3, 7]}]: dispatch 2021-02-10T04:39:01.418 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:39:01.418 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[29319]: T04:39:00.967643+0000 mon.a (mon. 2021-02-10T04:39:01.418 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[29319]: 0) 1632 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.9", "id": [1, 6, 5, 0, 3, 7]}]: dispatch 2021-02-10T04:39:01.418 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:39:01.418 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[29319]: 39:01.232933+0000 mon.a (mon.0) 1633 : cluster 2021-02-10T04:39:01.418 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[29319]: [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:39:01.418 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:39:01.418 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[29319]: 02-10T04:39:01.232973+0000 mon.a (mon.0) 1634 : 2021-02-10T04:39:01.418 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[29319]: cluster [INF] Cluster is now healthy 2021-02-10T04:39:01.418 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[29319]: audit 2021-02-10T04:39:01.418 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[29319]: 2021-02-10T04:39:01.239149+0000 mon.a (mon.0) 1635 : audit [INF] 2021-02-10T04:39:01.418 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[29319]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:39:01.419 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[29319]: audit 2021 2021-02-10T04:39:01.419 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[29319]: -02-10T04:39:01.239284+0000 mon.a (mon.0 2021-02-10T04:39:01.419 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[29319]: ) 1636 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.9", "id": [1, 6, 5, 0, 3, 7]}]': finished 2021-02-10T04:39:01.419 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[29319]: cluster 2021-02-10T04:39:01.239409 2021-02-10T04:39:01.419 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1637 : cluster [DBG] osdmap e722: 8 total, 8 up, 7 in 2021-02-10T04:39:01.509 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:39:01.620 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:39:01.652 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:39:01.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: audit 2021-02-10T04:39:01.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: 2021-02-10T04:39:01.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: 2021 2021-02-10T04:39:01.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: - 2021-02-10T04:39:01.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: 02 2021-02-10T04:39:01.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: - 2021-02-10T04:39:01.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: 10 2021-02-10T04:39:01.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: T 2021-02-10T04:39:01.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: 04 2021-02-10T04:39:01.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: : 2021-02-10T04:39:01.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: 39 2021-02-10T04:39:01.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: : 2021-02-10T04:39:01.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: 00 2021-02-10T04:39:01.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: . 2021-02-10T04:39:01.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: 736361 2021-02-10T04:39:01.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: +0000 2021-02-10T04:39:01.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: mon.c (mon.1) 62 : audit [DBG] 2021-02-10T04:39:01.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/598757740' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:39:01.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: audit 2021-02-10T04:39:01.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: 2021-02 2021-02-10T04:39:01.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: -10T04:39:00.953491+0000 mon.c ( 2021-02-10T04:39:01.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: mon.1) 63 : audit [INF] from='client.? 172.21.15.85:0/2372737071' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:39:01.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: audit 2021-02-10T04:39:01.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:39:01.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: 04:39:00.953994+0000 mon.a (mon.0) 1631 : audit 2021-02-10T04:39:01.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:39:01.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:39:01.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: 10T04:39:00.967058+0000 mon.c (mon 2021-02-10T04:39:01.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: .1) 64 : audit [INF] from='client.? 172.21.15.85:0/3149241954' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.9", "id": [1, 6, 5, 0, 3, 7]}]: dispatch 2021-02-10T04:39:01.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: audit 2021 2021-02-10T04:39:01.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: -02-10T04:39:00.967643+0000 mon.a (mon. 2021-02-10T04:39:01.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: 0) 1632 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.9", "id": [1, 6, 5, 0, 3, 7]}]: dispatch 2021-02-10T04:39:01.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:39:01.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: T04:39:01.232933 2021-02-10T04:39:01.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1633 2021-02-10T04:39:01.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:39:01.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: cluster 2021-02-10T04:39:01.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:39:01.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: 10T04:39:01.232973+0000 mon.a (mon.0 2021-02-10T04:39:01.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: ) 1634 : cluster 2021-02-10T04:39:01.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: [INF] Cluster is now healthy 2021-02-10T04:39:01.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: audit 2021-02-10T04:39:01.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: 2021-02-10T04:39:01.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: 2021 2021-02-10T04:39:01.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: - 2021-02-10T04:39:01.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: 02 2021-02-10T04:39:01.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: - 2021-02-10T04:39:01.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: 10T04:39:01.239149 2021-02-10T04:39:01.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1635 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:39:01.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: audit 2021-02-10T04:39:01.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: 2021-02-10T04:39:01.239284+0000 2021-02-10T04:39:01.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: mon.a (mon.0) 1636 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.9", "id": [1, 6, 5, 0, 3, 7]}]': finished 2021-02-10T04:39:01.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: 2021-02-10T04:39:01.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: cluster 2021-02-10T04:39:01.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: 2021-02-10T04:39:01.239409+0000 mon.a 2021-02-10T04:39:01.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: (mon.0 2021-02-10T04:39:01.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:01 smithi085 conmon[35961]: ) 1637 : cluster [DBG] osdmap e722: 8 total, 8 up, 7 in 2021-02-10T04:39:01.726 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:01 smithi164 conmon[29288]: audit 2021-02-10T04:39:00.736361+0000 2021-02-10T04:39:01.726 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:01 smithi164 conmon[29288]: mon.c (mon.1) 62 : audit [DBG] from='client.? 172.21.15.85:0/598757740' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:39:01.726 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:01 smithi164 conmon[29288]: audit 2021-02-10T04:39:00.953491+0000 mon.c (mon.1) 63 : audit [INF] from='client.? 172.21.15.85:0/2372737071' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:39:01.727 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:01 smithi164 conmon[29288]: audit 2021-02-10T04:39:00.953994+0000 mon.a (mon.0) 1631 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:39:01.727 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:01 smithi164 conmon[29288]: audit 2021-02-10T04:39:00.967058+0000 2021-02-10T04:39:01.727 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:01 smithi164 conmon[29288]: mon.c (mon.1) 64 : audit [INF] from='client.? 172.21.15.85:0/3149241954' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.9", "id": [1, 6, 5, 0, 3, 7]}]: dispatch 2021-02-10T04:39:01.727 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:01 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:39:01.727 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:01 smithi164 conmon[29288]: -10T04:39:00.967643+0000 mon.a (mon.0) 1632 : audit 2021-02-10T04:39:01.727 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:01 smithi164 conmon[29288]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.9", "id": [1, 6, 5, 0, 3, 7]}]: dispatch 2021-02-10T04:39:01.727 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:01 smithi164 conmon[29288]: cluster 2021-02-10T04:39:01 2021-02-10T04:39:01.727 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:01 smithi164 conmon[29288]: .232933+0000 mon.a (mon.0) 1633 : cluster 2021-02-10T04:39:01.727 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:01 smithi164 conmon[29288]: [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:39:01.727 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:01 smithi164 conmon[29288]: cluster 2021-02-10T04:39 2021-02-10T04:39:01.727 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:01 smithi164 conmon[29288]: :01.232973+0000 mon.a (mon.0) 1634 : cluster 2021-02-10T04:39:01.727 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:01 smithi164 conmon[29288]: [INF] Cluster is now healthy 2021-02-10T04:39:01.727 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:01 smithi164 conmon[29288]: audit 2021-02-10T04:39:01.239149+0000 2021-02-10T04:39:01.728 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:01 smithi164 conmon[29288]: mon.a (mon.0) 1635 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:39:01.728 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:01 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:39:01.728 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:01 smithi164 conmon[29288]: 02-10T04:39:01.239284+0000 2021-02-10T04:39:01.728 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:01 smithi164 conmon[29288]: mon.a (mon.0) 1636 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.9", "id": [1, 6, 5, 0, 3, 7]}]': finished 2021-02-10T04:39:01.728 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:01 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:39:01.728 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:01 smithi164 conmon[29288]: -10T04:39:01.239409+0000 mon.a (mon 2021-02-10T04:39:01.728 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:01 smithi164 conmon[29288]: .0) 1637 : cluster [DBG] osdmap e722: 8 total, 8 up, 7 in 2021-02-10T04:39:02.094 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:39:02.098 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:01 smithi164 conmon[33677]: debug 2021-02-10T04:39:01.724+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:02.098 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:01 smithi164 conmon[33677]: debug 2021-02-10T04:39:01.833+0000 7fbec19a1700 -1 osd.4 722 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:02.200 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:39:02.234 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:39:02.239 INFO:tasks.rados.rados.0.smithi085.stdout:2826: done (5 left) 2021-02-10T04:39:02.239 INFO:tasks.rados.rados.0.smithi085.stdout:2827: snap_remove snap 286 2021-02-10T04:39:02.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:39:02 smithi164 conmon[44544]: debug 2021-02-10T04:39:02.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:39:02 smithi164 conmon[44544]: 2021-02-10T04:39:02.324+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:02.396 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:39:02 smithi164 conmon[44544]: 2021-02-10T04:39:02.618 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:39:02.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[35961]: cluster 2021-02-10T04:39:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:39:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[35961]: T04:39:01.833959+0000 osd.4 (osd.4) 1129 : cluster [WRN] 2021-02-10T04:39:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[35961]: cluster 2021-02-10T04:39:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[35961]: 2021-02-10T04:39:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:39:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[35961]: 39:01.833968+0000 osd.4 (osd.4) 1130 : cluster [WRN] 2021-02-10T04:39:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:39:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[35961]: -02-10T04:39:01.833974+0000 2021-02-10T04:39:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[35961]: osd.4 (osd.4) 1131 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[35961]: cluster 2021-02-10T04:39:01.833980 2021-02-10T04:39:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1132 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:39:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[35961]: 02-10T04:39:02.168180+0000 mgr.y (mgr. 2021-02-10T04:39:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[35961]: 14140) 733 : cluster [DBG] pgmap v1234: 168 pgs: 1 active+clean+scrubbing, 1 peering, 6 active+clean+snaptrim_wait, 152 active+clean, 8 active+clean+snaptrim; 470 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.4 MiB/s rd, 2 op/s 2021-02-10T04:39:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:39:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[35961]: -02-10T04:39:02.238604+0000 mon.a 2021-02-10T04:39:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[35961]: (mon.0) 1638 : cluster [DBG] 2021-02-10T04:39:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[35961]: osdmap e723: 8 total, 8 up, 7 in 2021-02-10T04:39:02.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[29319]: cluster 2021-02-10T04:39:02.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[29319]: 2021-02-10T04:39:02.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[29319]: 2021 2021-02-10T04:39:02.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[29319]: - 2021-02-10T04:39:02.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[29319]: 02 2021-02-10T04:39:02.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[29319]: -10T04:39:01.833959 2021-02-10T04:39:02.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1129 : cluster 2021-02-10T04:39:02.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:02.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:39:02.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[29319]: 02-10T04:39:01.833968+0000 osd.4 (osd. 2021-02-10T04:39:02.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[29319]: 4) 1130 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:02.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:39:02.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[29319]: 10T04:39:01.833974+0000 osd.4 (osd 2021-02-10T04:39:02.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[29319]: .4) 1131 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:02.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:39:02.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[29319]: 02-10T04:39:01.833980+0000 osd.4 ( 2021-02-10T04:39:02.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[29319]: osd.4) 1132 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:02.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:39:02.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[29319]: 02-10T04:39:02.168180+0000 2021-02-10T04:39:02.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[29319]: mgr.y (mgr.14140) 733 : cluster [DBG] pgmap v1234: 168 pgs: 1 active+clean+scrubbing, 1 peering, 6 active+clean+snaptrim_wait, 152 active+clean, 8 active+clean+snaptrim; 470 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.4 MiB/s rd, 2 op/s 2021-02-10T04:39:02.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[29319]: 2021-02-10T04:39:02.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[29319]: cluster 2021-02-10T04:39:02.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[29319]: 2021-02-10T04:39:02.238604+0000 2021-02-10T04:39:02.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[29319]: mon.a (mon.0) 1638 : 2021-02-10T04:39:02.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:02 smithi085 conmon[29319]: cluster [DBG] osdmap e723: 8 total, 8 up, 7 in 2021-02-10T04:39:02.724 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:39:02.764 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:39:02.797 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:02 smithi164 conmon[29288]: cluster 2021-02-10T04:39:01.833959+0000 osd.4 (osd.4) 2021-02-10T04:39:02.797 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:02 smithi164 conmon[29288]: 1129 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:02.798 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:02 smithi164 conmon[29288]: cluster 2021-02-10T04:39:01.833968+0000 osd.4 (osd.4) 1130 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:02.798 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:02 smithi164 conmon[29288]: cluster 2021-02-10T04:39:01.833974+0000 osd.4 ( 2021-02-10T04:39:02.798 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:02 smithi164 conmon[29288]: osd.4) 1131 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:02.798 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:02 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:39:02.798 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:02 smithi164 conmon[29288]: 04:39:01.833980+0000 osd.4 (osd.4) 1132 : cluster [WRN] 2021-02-10T04:39:02.799 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:02 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:02.799 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:02 smithi164 conmon[29288]: cluster 2021-02-10T04:39:02. 2021-02-10T04:39:02.799 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:02 smithi164 conmon[29288]: 168180+0000 mgr.y (mgr.14140) 733 : cluster 2021-02-10T04:39:02.799 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:02 smithi164 conmon[29288]: [DBG] pgmap v1234: 168 pgs: 1 active+clean+scrubbing, 1 peering, 6 active+clean+snaptrim_wait, 152 active+clean, 8 active+clean+snaptrim; 470 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.4 MiB/s rd, 2 op/s 2021-02-10T04:39:02.799 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:02 smithi164 conmon[29288]: cluster 2021-02-10T04:39:02. 2021-02-10T04:39:02.799 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:02 smithi164 conmon[29288]: 238604+0000 mon.a (mon.0) 1638 : cluster 2021-02-10T04:39:02.799 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:02 smithi164 conmon[29288]: [DBG] osdmap e723: 8 total, 8 up, 7 in 2021-02-10T04:39:03.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:02 smithi164 conmon[33677]: debug 2021-02-10T04:39:03.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:02 smithi164 conmon[33677]: 2021-02-10T04:39:02.795+0000 7fbec19a1700 -1 osd.4 723 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:03.249 INFO:tasks.rados.rados.0.smithi085.stdout:2827: done (5 left) 2021-02-10T04:39:03.249 INFO:tasks.rados.rados.0.smithi085.stdout:2828: read oid 12 snap -1 2021-02-10T04:39:03.249 INFO:tasks.rados.rados.0.smithi085.stdout:2828: expect deleted 2021-02-10T04:39:03.249 INFO:tasks.rados.rados.0.smithi085.stdout:2829: snap_create 2021-02-10T04:39:03.249 INFO:tasks.rados.rados.0.smithi085.stdout:2824: finishing write tid 1 to smithi08575397-6 2021-02-10T04:39:03.249 INFO:tasks.rados.rados.0.smithi085.stdout:2824: finishing write tid 2 to smithi08575397-6 2021-02-10T04:39:03.249 INFO:tasks.rados.rados.0.smithi085.stdout:2824: finishing write tid 3 to smithi08575397-6 2021-02-10T04:39:03.249 INFO:tasks.rados.rados.0.smithi085.stdout:2824: finishing write tid 4 to smithi08575397-6 2021-02-10T04:39:03.250 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 1358 (ObjNum 987 snap 324 seq_num 987) dirty exists 2021-02-10T04:39:03.250 INFO:tasks.rados.rados.0.smithi085.stdout:2824: left oid 6 (ObjNum 987 snap 324 seq_num 987) 2021-02-10T04:39:03.426 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:03 smithi085 conmon[29319]: cluster 2021-02-10T04:39:02.413857+0000 2021-02-10T04:39:03.426 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:03 smithi085 conmon[29319]: mon.a (mon.0) 1639 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 572 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:39:03.426 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:03 smithi085 conmon[29319]: cluster 2021-02-10T04:39:02.796354+0000 osd.4 (osd.4) 1133 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:03.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:03 smithi085 conmon[29319]: cluster 2021-02-10T04:39:02.796363+0000 osd.4 (osd.4) 1134 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:03.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:03 smithi085 conmon[29319]: cluster 2021-02-10T04:39:02.796370+0000 osd.4 (osd.4) 1135 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:03.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:03 smithi085 conmon[29319]: cluster 2021-02-10T04:39:02.796382+0000 osd.4 (osd.4) 1136 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:03.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:03 smithi085 conmon[29319]: cluster 2021-02-10T04:39:03.248023+0000 mon.a (mon.0) 1640 : cluster [DBG] osdmap e724: 8 total, 8 up, 7 in 2021-02-10T04:39:03.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:03 smithi085 conmon[35961]: cluster 2021-02-10T04:39:03.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:03 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:39:03.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:03 smithi085 conmon[35961]: 39:02.413857+0000 mon.a (mon.0) 1639 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 572 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:39:03.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:03 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:39:03.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:03 smithi085 conmon[35961]: -02-10T04:39:02.796354+0000 2021-02-10T04:39:03.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:03 smithi085 conmon[35961]: osd.4 (osd.4) 1133 : cluster [WRN] 2021-02-10T04:39:03.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:03 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:03.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:03 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:39:03.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:03 smithi085 conmon[35961]: 10T04:39:02.796363+0000 osd.4 2021-02-10T04:39:03.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:03 smithi085 conmon[35961]: (osd.4) 1134 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:03.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:03 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:39:03.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:03 smithi085 conmon[35961]: T04:39:02.796370+0000 osd.4 (osd.4 2021-02-10T04:39:03.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:03 smithi085 conmon[35961]: ) 1135 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:03.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:03 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:39:03.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:03 smithi085 conmon[35961]: 02-10T04:39:02.796382+0000 2021-02-10T04:39:03.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:03 smithi085 conmon[35961]: osd.4 (osd.4) 1136 : cluster [WRN] 2021-02-10T04:39:03.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:03 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:03.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:03 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:39:03.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:03 smithi085 conmon[35961]: -02-10T04:39:03.248023 2021-02-10T04:39:03.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:03 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1640 : cluster 2021-02-10T04:39:03.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:03 smithi085 conmon[35961]: [DBG] osdmap e724: 8 total, 8 up, 7 in 2021-02-10T04:39:03.814 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:03 smithi164 conmon[29288]: cluster 2021-02-10T04:39:02.413857+0000 mon.a (mon.0) 1639 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 572 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:39:03.814 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:03 smithi164 conmon[29288]: cluster 2021-02-10T04:39:02.796354+0000 osd.4 (osd.4) 1133 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:03.815 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:03 smithi164 conmon[29288]: cluster 2021-02-10T04:39:02.796363+0000 osd.4 2021-02-10T04:39:03.815 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:03 smithi164 conmon[29288]: (osd.4) 1134 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:03.815 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:03 smithi164 conmon[29288]: cluster 2021-02-10T04:39:02 2021-02-10T04:39:03.815 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:03 smithi164 conmon[29288]: .796370+0000 osd.4 (osd.4) 1135 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:03.815 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:03 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:39:03.815 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:03 smithi164 conmon[29288]: 04:39:02.796382+0000 osd.4 (osd.4) 1136 : cluster 2021-02-10T04:39:03.815 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:03 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:03.815 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:03 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:39:03.815 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:03 smithi164 conmon[29288]: -02-10T04:39:03.248023+0000 mon.a (mon.0) 1640 : cluster [DBG] 2021-02-10T04:39:03.816 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:03 smithi164 conmon[29288]: osdmap e724: 8 total, 8 up, 7 in 2021-02-10T04:39:04.143 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:03 smithi164 conmon[33677]: debug 2021-02-10T04:39:04.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:03 smithi164 conmon[33677]: 2021-02-10T04:39:03.813+0000 7fbec19a1700 -1 osd.4 724 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:04.250 INFO:tasks.rados.rados.0.smithi085.stdout:2824: done (6 left) 2021-02-10T04:39:04.250 INFO:tasks.rados.rados.0.smithi085.stdout:2828: done (5 left) 2021-02-10T04:39:04.250 INFO:tasks.rados.rados.0.smithi085.stdout:2829: done (4 left) 2021-02-10T04:39:04.250 INFO:tasks.rados.rados.0.smithi085.stdout:2830: read oid 48 snap 291 2021-02-10T04:39:04.250 INFO:tasks.rados.rados.0.smithi085.stdout:2830: expect deleted 2021-02-10T04:39:04.250 INFO:tasks.rados.rados.0.smithi085.stdout:2831: read oid 15 snap -1 2021-02-10T04:39:04.250 INFO:tasks.rados.rados.0.smithi085.stdout:2831: expect (ObjNum 962 snap 315 seq_num 962) 2021-02-10T04:39:04.251 INFO:tasks.rados.rados.0.smithi085.stdout:2832: copy_from oid 40 from oid 26 current snap is 325 2021-02-10T04:39:04.251 INFO:tasks.rados.rados.0.smithi085.stdout:2833: write oid 45 current snap is 325 2021-02-10T04:39:04.251 INFO:tasks.rados.rados.0.smithi085.stdout:2833: seq_num 988 ranges {605316=555549,1832327=642013} 2021-02-10T04:39:04.258 INFO:tasks.rados.rados.0.smithi085.stdout:2833: writing smithi08575397-45 from 605316 to 1160865 tid 1 2021-02-10T04:39:04.267 INFO:tasks.rados.rados.0.smithi085.stdout:2833: writing smithi08575397-45 from 1832327 to 2474340 tid 2 2021-02-10T04:39:04.267 INFO:tasks.rados.rados.0.smithi085.stdout:2834: read oid 3 snap -1 2021-02-10T04:39:04.267 INFO:tasks.rados.rados.0.smithi085.stdout:2834: expect deleted 2021-02-10T04:39:04.268 INFO:tasks.rados.rados.0.smithi085.stdout:2835: delete oid 6 current snap is 325 2021-02-10T04:39:04.268 INFO:tasks.rados.rados.0.smithi085.stdout:2832: finishing copy_from to smithi08575397-40 2021-02-10T04:39:04.268 INFO:tasks.rados.rados.0.smithi085.stdout:2832: got expected ENOENT (src dne) 2021-02-10T04:39:04.268 INFO:tasks.rados.rados.0.smithi085.stdout:2832: finishing copy_from racing read to smithi08575397-40 2021-02-10T04:39:04.268 INFO:tasks.rados.rados.0.smithi085.stdout:2831: expect (ObjNum 962 snap 315 seq_num 962) 2021-02-10T04:39:04.389 INFO:tasks.rados.rados.0.smithi085.stdout:2833: finishing write tid 1 to smithi08575397-45 2021-02-10T04:39:04.389 INFO:tasks.rados.rados.0.smithi085.stdout:2833: finishing write tid 2 to smithi08575397-45 2021-02-10T04:39:04.389 INFO:tasks.rados.rados.0.smithi085.stdout:2833: finishing write tid 3 to smithi08575397-45 2021-02-10T04:39:04.389 INFO:tasks.rados.rados.0.smithi085.stdout:2833: finishing write tid 4 to smithi08575397-45 2021-02-10T04:39:04.389 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 45 v 1438 (ObjNum 988 snap 325 seq_num 988) dirty exists 2021-02-10T04:39:04.389 INFO:tasks.rados.rados.0.smithi085.stdout:2833: left oid 45 (ObjNum 988 snap 325 seq_num 988) 2021-02-10T04:39:04.389 INFO:tasks.rados.rados.0.smithi085.stdout:2830: done (9 left) 2021-02-10T04:39:04.389 INFO:tasks.rados.rados.0.smithi085.stdout:2831: done (8 left) 2021-02-10T04:39:04.389 INFO:tasks.rados.rados.0.smithi085.stdout:2832: done (7 left) 2021-02-10T04:39:04.389 INFO:tasks.rados.rados.0.smithi085.stdout:2833: done (6 left) 2021-02-10T04:39:04.390 INFO:tasks.rados.rados.0.smithi085.stdout:2834: done (5 left) 2021-02-10T04:39:04.390 INFO:tasks.rados.rados.0.smithi085.stdout:2835: done (4 left) 2021-02-10T04:39:04.390 INFO:tasks.rados.rados.0.smithi085.stdout:2836: copy_from oid 15 from oid 1 current snap is 325 2021-02-10T04:39:04.390 INFO:tasks.rados.rados.0.smithi085.stdout:2837: snap_create 2021-02-10T04:39:04.453 INFO:tasks.rados.rados.0.smithi085.stdout:2836: finishing copy_from to smithi08575397-15 2021-02-10T04:39:04.453 INFO:tasks.rados.rados.0.smithi085.stdout:2836: got expected ENOENT (src dne) 2021-02-10T04:39:04.453 INFO:tasks.rados.rados.0.smithi085.stdout:2836: finishing copy_from racing read to smithi08575397-15 2021-02-10T04:39:04.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[49106]: debug 2021-02-10T04:39:04.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[49106]: 2021-02-10T04:39:04.552+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:04.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[29319]: cluster 2021-02-10T04:39:04.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[29319]: 2021-02-10T04:39:04.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[29319]: 2021-02-10T04:39:03.814201 2021-02-10T04:39:04.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1137 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:04.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[29319]: cluster 2021-02-10T04:39:03.814209+0000 osd.4 (osd 2021-02-10T04:39:04.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[29319]: .4) 1138 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:04.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[29319]: cluster 2021-02-10T04:39:04.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[29319]: 2021-02-10T04:39:03.814215+0000 osd.4 (osd.4) 1139 2021-02-10T04:39:04.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:04.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:39:04.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[29319]: :39:03.814221+0000 osd.4 (osd.4 2021-02-10T04:39:04.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[29319]: ) 1140 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:04.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[29319]: cluster 2021-02-10T04:39:04.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[29319]: 2021-02-10T04:39:04.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[29319]: 2021 2021-02-10T04:39:04.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[29319]: - 2021-02-10T04:39:04.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[29319]: 02-10T04:39:04.168682+0000 mgr.y (mgr 2021-02-10T04:39:04.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[29319]: .14140) 734 : cluster [DBG] pgmap v1237: 168 pgs: 1 active+clean+scrubbing, 1 peering, 5 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 470 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 3.7 MiB/s rd, 3 op/s 2021-02-10T04:39:04.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:39:04.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[29319]: :39:04.249507+0000 mon.a (mon.0 2021-02-10T04:39:04.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[29319]: ) 1641 : cluster [DBG] osdmap e725: 8 total, 8 up, 7 in 2021-02-10T04:39:04.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[35961]: cluster 2021-02-10T04:39:03. 2021-02-10T04:39:04.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[35961]: 814201+0000 osd.4 (osd.4) 1137 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:04.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[35961]: cluster 2021-02-10T04:39:03.814209+0000 osd.4 (osd 2021-02-10T04:39:04.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[35961]: .4) 1138 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:04.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[35961]: cluster 2021-02-10T04:39:04.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[35961]: 2021-02-10T04:39:03.814215+0000 osd.4 (osd.4) 1139 : cluster 2021-02-10T04:39:04.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:04.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:39:04.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[35961]: -10T04:39:03.814221+0000 2021-02-10T04:39:04.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[35961]: osd.4 (osd.4) 1140 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:04.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[35961]: 2021-02-10T04:39:04.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[35961]: cluster 2021-02-10T04:39: 2021-02-10T04:39:04.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[35961]: 04.168682+0000 mgr.y (mgr.14140) 734 : cluster 2021-02-10T04:39:04.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[35961]: [DBG] pgmap v1237: 168 pgs: 1 active+clean+scrubbing, 1 peering, 5 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 470 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 3.7 MiB/s rd, 3 op/s 2021-02-10T04:39:04.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:39:04.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[35961]: 10T04:39:04.249507+0000 mon.a 2021-02-10T04:39:04.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[35961]: (mon.0) 1641 : cluster [DBG] osdmap e725: 8 total, 8 up, 7 in 2021-02-10T04:39:04.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:04 smithi085 conmon[35961]: 2021-02-10T04:39:04.809 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:39:04 smithi164 conmon[37263]: debug 2021-02-10T04:39:04.403+0000 7f1c671e2700 -1 osd.5 pg_epoch: 725 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x556652f9d340) 2021-02-10T04:39:04.810 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:04 smithi164 conmon[29288]: cluster 2021-02-10T04:39:03.814201+0000 osd.4 (osd.4) 1137 : cluster 2021-02-10T04:39:04.810 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:04 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:04.811 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:04 smithi164 conmon[29288]: cluster 2021-02-10T04:39:03.814209+0000 osd.4 (osd.4) 1138 : cluster [WRN] 2021-02-10T04:39:04.811 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:04 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:04.811 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:04 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:39:04.811 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:04 smithi164 conmon[29288]: 04:39:03.814215+0000 osd.4 (osd.4) 1139 : cluster [WRN] 2021-02-10T04:39:04.811 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:04 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:04.811 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:04 smithi164 conmon[29288]: cluster 2021-02-10T04:39: 2021-02-10T04:39:04.811 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:04 smithi164 conmon[29288]: 03.814221+0000 osd.4 (osd.4) 1140 : cluster 2021-02-10T04:39:04.812 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:04 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:04.812 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:04 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:39:04.812 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:04 smithi164 conmon[29288]: -02-10T04:39:04.168682+0000 mgr.y (mgr.14140) 2021-02-10T04:39:04.812 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:04 smithi164 conmon[29288]: 734 : cluster [DBG] pgmap v1237: 168 pgs: 1 active+clean+scrubbing, 1 peering, 5 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 470 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 3.7 MiB/s rd, 3 op/s 2021-02-10T04:39:04.812 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:04 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:39:04.812 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:04 smithi164 conmon[29288]: :39:04.249507+0000 mon.a (mon.0) 1641 2021-02-10T04:39:04.812 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:04 smithi164 conmon[29288]: : cluster [DBG] osdmap e725: 8 total, 8 up, 7 in 2021-02-10T04:39:04.812 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:04 smithi164 conmon[33677]: debug 2021-02-10T04:39:04.809+0000 7fbec19a1700 -1 osd.4 725 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:05.147 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:39:05.253 INFO:tasks.rados.rados.0.smithi085.stdout:2836: done (5 left) 2021-02-10T04:39:05.254 INFO:tasks.rados.rados.0.smithi085.stdout:2837: done (4 left) 2021-02-10T04:39:05.254 INFO:tasks.rados.rados.0.smithi085.stdout:2838: read oid 6 snap -1 2021-02-10T04:39:05.254 INFO:tasks.rados.rados.0.smithi085.stdout:2838: expect deleted 2021-02-10T04:39:05.254 INFO:tasks.rados.rados.0.smithi085.stdout:2839: delete oid 25 current snap is 326 2021-02-10T04:39:05.263 INFO:tasks.rados.rados.0.smithi085.stdout:2838: done (5 left) 2021-02-10T04:39:05.263 INFO:tasks.rados.rados.0.smithi085.stdout:2839: done (4 left) 2021-02-10T04:39:05.263 INFO:tasks.rados.rados.0.smithi085.stdout:2840: write oid 36 current snap is 326 2021-02-10T04:39:05.263 INFO:tasks.rados.rados.0.smithi085.stdout:2840: seq_num 991 ranges {655737=497649,1607773=789340} 2021-02-10T04:39:05.270 INFO:tasks.rados.rados.0.smithi085.stdout:2840: writing smithi08575397-36 from 655737 to 1153386 tid 1 2021-02-10T04:39:05.278 INFO:tasks.rados.rados.0.smithi085.stdout:2840: writing smithi08575397-36 from 1607773 to 2397113 tid 2 2021-02-10T04:39:05.278 INFO:tasks.rados.rados.0.smithi085.stdout:2840: finishing write tid 1 to smithi08575397-36 2021-02-10T04:39:05.278 INFO:tasks.rados.rados.0.smithi085.stdout:2841: snap_remove snap 290 2021-02-10T04:39:05.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[29319]: cluster 2021-02-10T04:39:04.809708+0000 2021-02-10T04:39:05.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[29319]: osd.4 (osd.4) 1141 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:05.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[29319]: cluster 2021-02-10T04:39:04.809714+0000 osd.4 (osd.4) 1142 2021-02-10T04:39:05.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:05.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[29319]: cluster 2021-02-10T04:39:05.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[29319]: 2021-02-10T04:39:04.809729+0000 osd.4 ( 2021-02-10T04:39:05.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[29319]: osd.4) 1143 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:05.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:39:05.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[29319]: -02-10T04:39:04.809734 2021-02-10T04:39:05.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1144 : cluster [WRN] 2021-02-10T04:39:05.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:05.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[29319]: cluster 2021-02-10T04:39 2021-02-10T04:39:05.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[29319]: :05.252959+0000 mon.a (mon.0) 1642 2021-02-10T04:39:05.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[29319]: : cluster [DBG] osdmap e726: 8 total, 8 up, 7 in 2021-02-10T04:39:05.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[35961]: cluster 2021-02-10T04:39:04.809708+0000 2021-02-10T04:39:05.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[35961]: osd.4 (osd.4) 1141 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:05.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[35961]: cluster 2021-02-10T04:39:05.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[35961]: 2021 2021-02-10T04:39:05.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[35961]: - 2021-02-10T04:39:05.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[35961]: 02 2021-02-10T04:39:05.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[35961]: -10T04:39:04.809714 2021-02-10T04:39:05.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1142 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:05.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:39:05.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[35961]: T04:39:04.809729+0000 osd.4 (osd.4 2021-02-10T04:39:05.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[35961]: ) 1143 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:05.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[35961]: cluster 2021-02-10T04:39:05.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[35961]: 2021-02-10T04:39:04.809734 2021-02-10T04:39:05.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1144 : cluster [WRN] 2021-02-10T04:39:05.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:05.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:39:05.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[35961]: 39:05.252959+0000 mon.a (mon.0) 1642 2021-02-10T04:39:05.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:05 smithi085 conmon[35961]: : cluster [DBG] osdmap e726: 8 total, 8 up, 7 in 2021-02-10T04:39:05.777 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:05 smithi164 conmon[29288]: cluster 2021-02-10T04:39:04.809708+0000 osd.4 ( 2021-02-10T04:39:05.777 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:05 smithi164 conmon[29288]: osd.4) 1141 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:05.777 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:05 smithi164 conmon[29288]: cluster 2021-02-10T04:39:04.809714+0000 osd.4 (osd.4) 1142 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:05.777 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:05 smithi164 conmon[29288]: cluster 2021-02-10T04:39:04.809729+0000 osd.4 (osd.4) 2021-02-10T04:39:05.777 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:05 smithi164 conmon[29288]: 1143 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:05.777 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:05 smithi164 conmon[29288]: cluster 2021-02-10T04:39:05.777 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:05 smithi164 conmon[29288]: 2021-02-10T04:39:04.809734+0000 osd.4 (osd.4) 1144 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:05.777 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:05 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:39:05.777 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:05 smithi164 conmon[29288]: :39:05.252959+0000 mon.a (mon.0) 1642 : 2021-02-10T04:39:05.778 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:05 smithi164 conmon[29288]: cluster [DBG] osdmap e726: 8 total, 8 up, 7 in 2021-02-10T04:39:05.778 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:05 smithi164 conmon[33677]: debug 2021-02-10T04:39:05.778 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:05 smithi164 conmon[33677]: 2021-02-10T04:39:05.775+0000 7fbec19a1700 -1 osd.4 726 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:05.778 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:05 smithi164 conmon[33677]: 2021-02-10T04:39:06.253 INFO:tasks.rados.rados.0.smithi085.stdout:2840: finishing write tid 2 to smithi08575397-36 2021-02-10T04:39:06.253 INFO:tasks.rados.rados.0.smithi085.stdout:2840: finishing write tid 3 to smithi08575397-36 2021-02-10T04:39:06.254 INFO:tasks.rados.rados.0.smithi085.stdout:2840: finishing write tid 4 to smithi08575397-36 2021-02-10T04:39:06.254 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 1289 (ObjNum 991 snap 326 seq_num 991) dirty exists 2021-02-10T04:39:06.254 INFO:tasks.rados.rados.0.smithi085.stdout:2840: left oid 36 (ObjNum 991 snap 326 seq_num 991) 2021-02-10T04:39:06.254 INFO:tasks.rados.rados.0.smithi085.stdout:2840: done (5 left) 2021-02-10T04:39:06.254 INFO:tasks.rados.rados.0.smithi085.stdout:2841: done (4 left) 2021-02-10T04:39:06.254 INFO:tasks.rados.rados.0.smithi085.stdout:2842: read oid 13 snap -1 2021-02-10T04:39:06.254 INFO:tasks.rados.rados.0.smithi085.stdout:2842: expect deleted 2021-02-10T04:39:06.254 INFO:tasks.rados.rados.0.smithi085.stdout:2843: rollback oid 25 current snap is 326 2021-02-10T04:39:06.254 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 25 to 316 2021-02-10T04:39:06.254 INFO:tasks.rados.rados.0.smithi085.stdout:2844: copy_from oid 35 from oid 32 current snap is 326 2021-02-10T04:39:06.254 INFO:tasks.rados.rados.0.smithi085.stdout:2845: copy_from oid 40 from oid 5 current snap is 326 2021-02-10T04:39:06.254 INFO:tasks.rados.rados.0.smithi085.stdout:2846: snap_remove snap 312 2021-02-10T04:39:06.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[29319]: cluster 2021-02-10T04:39:05.623478 2021-02-10T04:39:06.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[29319]: +0000 osd.5 (osd.5) 78 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:39:06.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[29319]: cluster 2021-02-10T04:39:05.624122+0000 osd.5 (osd.5) 79 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:39:06.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[29319]: cluster 2021-02-10T04:39:05.775694+0000 osd.4 (osd.4 2021-02-10T04:39:06.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[29319]: ) 1145 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:06.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[29319]: cluster 2021-02-10T04:39:06.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[29319]: 2021-02-10T04:39:05.775709+0000 osd.4 (osd.4) 1146 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:06.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:39:06.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[29319]: 02-10T04:39: 2021-02-10T04:39:06.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[29319]: 05.775716+0000 osd.4 (osd.4) 1147 : cluster 2021-02-10T04:39:06.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:06.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:39:06.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[29319]: 10T04:39:05.775729+0000 osd.4 ( 2021-02-10T04:39:06.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[29319]: osd.4) 1148 : cluster [WRN] 2021-02-10T04:39:06.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:06.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[29319]: cluster 2021-02-10T04:39:06.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[29319]: 2021-02-10T04:39:06.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[29319]: 2021-02-10T04:39:06.169130+0000 2021-02-10T04:39:06.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[29319]: mgr.y (mgr.14140) 735 : cluster [DBG] pgmap v1240: 168 pgs: 1 active+clean+scrubbing, 1 peering, 5 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 471 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 3.7 MiB/s rd, 308 KiB/s wr, 5 op/s 2021-02-10T04:39:06.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[29319]: cluster 2021-02-10T04:39:06.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[29319]: 2021-02-10T04:39:06.252463+0000 mon.a (mon 2021-02-10T04:39:06.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[29319]: .0) 1643 : cluster [DBG] osdmap e727: 8 total, 8 up, 7 in 2021-02-10T04:39:06.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: cluster 2021-02-10T04:39:06.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: 2021-02-10T04:39:06.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: 2021 2021-02-10T04:39:06.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: - 2021-02-10T04:39:06.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: 02-10T04:39:05.623478+0000 osd.5 (osd.5) 78 2021-02-10T04:39:06.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:39:06.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: cluster 2021-02-10T04:39:06.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: 2021-02-10T04:39:06.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: 2021 2021-02-10T04:39:06.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: - 2021-02-10T04:39:06.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: 02 2021-02-10T04:39:06.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: - 2021-02-10T04:39:06.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: 10 2021-02-10T04:39:06.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: T 2021-02-10T04:39:06.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: 04 2021-02-10T04:39:06.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: : 2021-02-10T04:39:06.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: 39 2021-02-10T04:39:06.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: : 2021-02-10T04:39:06.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: 05 2021-02-10T04:39:06.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: . 2021-02-10T04:39:06.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: 624122 2021-02-10T04:39:06.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: +0000 2021-02-10T04:39:06.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: 2021-02-10T04:39:06.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: osd.5 (osd.5) 2021-02-10T04:39:06.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: 79 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:39:06.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: cluster 2021-02-10T04:39:06.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: 2021-02-10T04:39:05 2021-02-10T04:39:06.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: .775694+0000 osd.4 (osd.4) 1145 : 2021-02-10T04:39:06.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:06.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:39:06.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: 04:39:05.775709+0000 osd.4 (osd. 2021-02-10T04:39:06.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: 4) 1146 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:06.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: cluster 2021-02-10T04:39:05.775716+0000 2021-02-10T04:39:06.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: osd.4 (osd.4) 1147 : cluster [WRN] 2021-02-10T04:39:06.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:06.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: cluster 2021-02-10T04:39:05.775729 2021-02-10T04:39:06.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1148 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:06.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:39:06.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: T04:39:06.169130+0000 mgr.y (mgr 2021-02-10T04:39:06.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: .14140) 735 : cluster [DBG] pgmap v1240: 168 pgs: 1 active+clean+scrubbing, 1 peering, 5 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 471 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 3.7 MiB/s rd, 308 KiB/s wr, 5 op/s 2021-02-10T04:39:06.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:39:06.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: -02-10T04:39:06.252463+0000 mon.a (mon.0) 1643 2021-02-10T04:39:06.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:06 smithi085 conmon[35961]: : cluster [DBG] osdmap e727: 8 total, 8 up, 7 in 2021-02-10T04:39:06.799 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:06 smithi164 conmon[33677]: debug 2021-02-10T04:39:06.797+0000 7fbec19a1700 -1 osd.4 727 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:06.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:06 smithi164 conmon[29288]: cluster 2021-02-10T04:39:05.623478+0000 osd.5 (osd.5) 78 : cluster [DBG] 2021-02-10T04:39:06.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:06 smithi164 conmon[29288]: purged_snaps scrub starts 2021-02-10T04:39:06.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:06 smithi164 conmon[29288]: cluster 2021-02-10T04:39:05.624122+0000 osd.5 (osd.5) 79 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:39:06.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:06 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:39:06.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:06 smithi164 conmon[29288]: -10T04:39:05.775694+0000 osd.4 (osd.4) 1145 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:06.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:06 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:39:06.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:06 smithi164 conmon[29288]: 02-10T04:39:05.775709+0000 osd.4 2021-02-10T04:39:06.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:06 smithi164 conmon[29288]: (osd.4) 1146 : cluster [WRN] 2021-02-10T04:39:06.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:06 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:06.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:06 smithi164 conmon[29288]: cluster 2021-02-10T04:39:05.775716 2021-02-10T04:39:06.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:06 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1147 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:06.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:06 smithi164 conmon[29288]: 2021-02-10T04:39:06.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:06 smithi164 conmon[29288]: cluster 2021-02-10T04:39:06.801 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:06 smithi164 conmon[29288]: 2021-02-10T04:39:05.775729+0000 osd.4 (osd. 2021-02-10T04:39:06.801 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:06 smithi164 conmon[29288]: 4) 1148 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:06.801 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:06 smithi164 conmon[29288]: cluster 2021-02-10T04:39: 2021-02-10T04:39:06.801 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:06 smithi164 conmon[29288]: 06.169130+0000 mgr.y (mgr.14140) 735 : cluster [DBG] 2021-02-10T04:39:06.801 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:06 smithi164 conmon[29288]: pgmap v1240: 168 pgs: 1 active+clean+scrubbing, 1 peering, 5 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 471 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 3.7 MiB/s rd, 308 KiB/s wr, 5 op/s 2021-02-10T04:39:06.801 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:06 smithi164 conmon[29288]: cluster 2021-02-10T04:39:06.252463 2021-02-10T04:39:06.801 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:06 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1643 : cluster [DBG] osdmap e727: 8 total, 8 up, 7 in 2021-02-10T04:39:06.801 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:06 smithi164 conmon[29288]: 2021-02-10T04:39:07.256 INFO:tasks.rados.rados.0.smithi085.stdout:2843: finishing rollback tid 1 to smithi08575397-25 2021-02-10T04:39:07.256 INFO:tasks.rados.rados.0.smithi085.stdout:2843: finishing rollback tid 2 to smithi08575397-25 2021-02-10T04:39:07.256 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 25 v 1289 (ObjNum 977 snap 316 seq_num 977) dirty exists 2021-02-10T04:39:07.256 INFO:tasks.rados.rados.0.smithi085.stdout:2845: finishing copy_from to smithi08575397-40 2021-02-10T04:39:07.256 INFO:tasks.rados.rados.0.smithi085.stdout:2845: got expected ENOENT (src dne) 2021-02-10T04:39:07.256 INFO:tasks.rados.rados.0.smithi085.stdout:2845: finishing copy_from racing read to smithi08575397-40 2021-02-10T04:39:07.257 INFO:tasks.rados.rados.0.smithi085.stdout:2844: finishing copy_from to smithi08575397-35 2021-02-10T04:39:07.257 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 35 v 1353 (ObjNum 986 snap 320 seq_num 986) dirty exists 2021-02-10T04:39:07.257 INFO:tasks.rados.rados.0.smithi085.stdout:2844: finishing copy_from racing read to smithi08575397-35 2021-02-10T04:39:07.257 INFO:tasks.rados.rados.0.smithi085.stdout:2842: done (8 left) 2021-02-10T04:39:07.257 INFO:tasks.rados.rados.0.smithi085.stdout:2843: done (7 left) 2021-02-10T04:39:07.257 INFO:tasks.rados.rados.0.smithi085.stdout:2844: done (6 left) 2021-02-10T04:39:07.257 INFO:tasks.rados.rados.0.smithi085.stdout:2845: done (5 left) 2021-02-10T04:39:07.257 INFO:tasks.rados.rados.0.smithi085.stdout:2846: done (4 left) 2021-02-10T04:39:07.257 INFO:tasks.rados.rados.0.smithi085.stdout:2847: copy_from oid 43 from oid 42 current snap is 326 2021-02-10T04:39:07.257 INFO:tasks.rados.rados.0.smithi085.stdout:2848: read oid 7 snap -1 2021-02-10T04:39:07.257 INFO:tasks.rados.rados.0.smithi085.stdout:2848: expect (ObjNum 960 snap 315 seq_num 960) 2021-02-10T04:39:07.257 INFO:tasks.rados.rados.0.smithi085.stdout:2849: read oid 11 snap 309 2021-02-10T04:39:07.257 INFO:tasks.rados.rados.0.smithi085.stdout:2849: expect deleted 2021-02-10T04:39:07.258 INFO:tasks.rados.rados.0.smithi085.stdout:2850: rollback oid 46 current snap is 326 2021-02-10T04:39:07.258 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 46 to 318 2021-02-10T04:39:07.258 INFO:tasks.rados.rados.0.smithi085.stdout:2851: delete oid 8 current snap is 326 2021-02-10T04:39:07.260 INFO:tasks.rados.rados.0.smithi085.stdout:2849: done (8 left) 2021-02-10T04:39:07.260 INFO:tasks.rados.rados.0.smithi085.stdout:2851: done (7 left) 2021-02-10T04:39:07.260 INFO:tasks.rados.rados.0.smithi085.stdout:2852: rollback oid 5 current snap is 326 2021-02-10T04:39:07.260 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 5 to 291 2021-02-10T04:39:07.260 INFO:tasks.rados.rados.0.smithi085.stdout:2853: snap_remove snap 305 2021-02-10T04:39:07.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:07 smithi085 conmon[29319]: cluster 2021-02-10T04:39:06.798174+0000 osd.4 (osd.4) 1149 : cluster 2021-02-10T04:39:07.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:07 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:07.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:07 smithi085 conmon[29319]: cluster 2021-02-10T04:39:06.798183+0000 osd.4 (osd.4) 1150 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:07.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:07 smithi085 conmon[29319]: cluster 2021-02-10T04:39:06.798190+0000 osd.4 (osd. 2021-02-10T04:39:07.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:07 smithi085 conmon[29319]: 4) 1151 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:07.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:07 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:39:07.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:07 smithi085 conmon[29319]: :39:06.798195+0000 osd.4 (osd. 2021-02-10T04:39:07.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:07 smithi085 conmon[29319]: 4) 1152 : cluster [WRN] 2021-02-10T04:39:07.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:07 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:07.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:07 smithi085 conmon[29319]: cluster 2021-02-10T04:39:07.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:07 smithi085 conmon[29319]: 2021-02-10T04:39: 2021-02-10T04:39:07.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:07 smithi085 conmon[29319]: 07.254500+0000 mon.a (mon.0) 1644 : 2021-02-10T04:39:07.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:07 smithi085 conmon[29319]: cluster [DBG] osdmap e728: 8 total, 8 up, 7 in 2021-02-10T04:39:07.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:07 smithi085 conmon[35961]: cluster 2021-02-10T04:39: 2021-02-10T04:39:07.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:07 smithi085 conmon[35961]: 06.798174+0000 osd.4 (osd.4) 1149 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:07.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:07 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:39:07.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:07 smithi085 conmon[35961]: -02-10T04:39:06.798183+0000 osd.4 (osd.4) 1150 : cluster [WRN] 2021-02-10T04:39:07.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:07 smithi085 conmon[35961]: 2021-02-10T04:39:07.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:07 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:07.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:07 smithi085 conmon[35961]: cluster 2021-02-10T04:39:07.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:07 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:39:07.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:07 smithi085 conmon[35961]: 39:06.798190+0000 osd.4 ( 2021-02-10T04:39:07.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:07 smithi085 conmon[35961]: osd.4) 1151 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:07.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:07 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:39:07.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:07 smithi085 conmon[35961]: 02-10T04:39:06 2021-02-10T04:39:07.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:07 smithi085 conmon[35961]: .798195+0000 osd.4 (osd.4) 2021-02-10T04:39:07.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:07 smithi085 conmon[35961]: 1152 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:07.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:07 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:39:07.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:07 smithi085 conmon[35961]: T04:39:07.254500+0000 mon.a (mon. 2021-02-10T04:39:07.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:07 smithi085 conmon[35961]: 0) 1644 : cluster [DBG] osdmap e728: 8 total, 8 up, 7 in 2021-02-10T04:39:07.843 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:07 smithi164 conmon[29288]: cluster 2021-02-10T04:39:06.798174+0000 2021-02-10T04:39:07.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:07 smithi164 conmon[29288]: osd.4 (osd.4) 1149 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:07.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:07 smithi164 conmon[29288]: cluster 2021-02-10T04:39:06.798183+0000 osd.4 ( 2021-02-10T04:39:07.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:07 smithi164 conmon[29288]: osd.4) 1150 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:07.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:07 smithi164 conmon[29288]: cluster 2021-02-10T04:39:07.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:07 smithi164 conmon[29288]: 2021-02-10T04:39:06.798190+0000 osd.4 (osd.4) 1151 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:07.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:07 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:39:07.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:07 smithi164 conmon[29288]: -02-10T04:39:06.798195 2021-02-10T04:39:07.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:07 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1152 : cluster 2021-02-10T04:39:07.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:07 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:07.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:07 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:39:07.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:07 smithi164 conmon[29288]: 02-10T04:39:07.254500+0000 mon.a ( 2021-02-10T04:39:07.845 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:07 smithi164 conmon[29288]: mon.0) 1644 : cluster [DBG] osdmap e728: 8 total, 8 up, 7 in 2021-02-10T04:39:08.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:07 smithi164 conmon[33677]: debug 2021-02-10T04:39:08.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:07 smithi164 conmon[33677]: 2021-02-10T04:39:07.842+0000 7fbec19a1700 -1 osd.4 728 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:08.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:07 smithi164 conmon[33677]: 2021-02-10T04:39:08.240 INFO:tasks.rados.rados.0.smithi085.stdout:2853: done (8 left) 2021-02-10T04:39:08.240 INFO:tasks.rados.rados.0.smithi085.stdout:2854: delete oid 1 current snap is 326 2021-02-10T04:39:08.240 INFO:tasks.rados.rados.0.smithi085.stdout:2848: expect (ObjNum 960 snap 315 seq_num 960) 2021-02-10T04:39:08.351 INFO:tasks.rados.rados.0.smithi085.stdout:2850: finishing rollback tid 1 to smithi08575397-46 2021-02-10T04:39:08.351 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 1352 (ObjNum 82292512 snap 22030 seq_num 1215200496) dirty dne 2021-02-10T04:39:08.351 INFO:tasks.rados.rados.0.smithi085.stdout:2852: finishing rollback tid 1 to smithi08575397-5 2021-02-10T04:39:08.351 INFO:tasks.rados.rados.0.smithi085.stdout:2852: finishing rollback tid 2 to smithi08575397-5 2021-02-10T04:39:08.351 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 801 (ObjNum 863 snap 278 seq_num 863) dirty exists 2021-02-10T04:39:08.352 INFO:tasks.rados.rados.0.smithi085.stdout:2847: finishing copy_from to smithi08575397-43 2021-02-10T04:39:08.352 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 1354 (ObjNum 981 snap 317 seq_num 981) dirty exists 2021-02-10T04:39:08.352 INFO:tasks.rados.rados.0.smithi085.stdout:2847: finishing copy_from racing read to smithi08575397-43 2021-02-10T04:39:08.353 INFO:tasks.rados.rados.0.smithi085.stdout:2847: done (8 left) 2021-02-10T04:39:08.353 INFO:tasks.rados.rados.0.smithi085.stdout:2848: done (7 left) 2021-02-10T04:39:08.353 INFO:tasks.rados.rados.0.smithi085.stdout:2850: done (6 left) 2021-02-10T04:39:08.353 INFO:tasks.rados.rados.0.smithi085.stdout:2852: done (5 left) 2021-02-10T04:39:08.353 INFO:tasks.rados.rados.0.smithi085.stdout:2854: done (4 left) 2021-02-10T04:39:08.353 INFO:tasks.rados.rados.0.smithi085.stdout:2855: write oid 5 current snap is 326 2021-02-10T04:39:08.353 INFO:tasks.rados.rados.0.smithi085.stdout:2855: seq_num 994 ranges {706158=723396,2183219=522939} 2021-02-10T04:39:08.361 INFO:tasks.rados.rados.0.smithi085.stdout:2855: writing smithi08575397-5 from 706158 to 1429554 tid 1 2021-02-10T04:39:08.370 INFO:tasks.rados.rados.0.smithi085.stdout:2855: writing smithi08575397-5 from 2183219 to 2706158 tid 2 2021-02-10T04:39:08.370 INFO:tasks.rados.rados.0.smithi085.stdout:2856: rollback oid 42 current snap is 326 2021-02-10T04:39:08.371 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 42 to 303 2021-02-10T04:39:08.371 INFO:tasks.rados.rados.0.smithi085.stdout:2855: finishing write tid 1 to smithi08575397-5 2021-02-10T04:39:08.371 INFO:tasks.rados.rados.0.smithi085.stdout:2857: copy_from oid 36 from oid 1 current snap is 326 2021-02-10T04:39:08.371 INFO:tasks.rados.rados.0.smithi085.stdout:2858: snap_remove snap 230 2021-02-10T04:39:08.644 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:39:08 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:39:08.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:39:08 smithi164 conmon[37263]: debug 2021-02-10T04:39:08.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:39:08 smithi164 conmon[37263]: 2021-02-10T04:39:08.330+0000 7f1c671e2700 -1 osd.5 pg_epoch: 729 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55665307b760) 2021-02-10T04:39:08.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:39:08 smithi164 conmon[37263]: 2021-02-10T04:39:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:08 smithi164 conmon[29288]: cluster 2021-02-10T04:39:07.843088+0000 osd.4 ( 2021-02-10T04:39:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:08 smithi164 conmon[29288]: osd.4) 1153 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:08 smithi164 conmon[29288]: cluster 2021-02-10T04:39:07.843105+0000 osd.4 (osd.4) 1154 : cluster [WRN] 2021-02-10T04:39:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:08 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:08 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:39:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:08 smithi164 conmon[29288]: :39:07.843112+0000 osd.4 (osd.4) 1155 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:08 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:39:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:08 smithi164 conmon[29288]: T04:39:07.843118+0000 osd.4 (osd.4) 2021-02-10T04:39:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:08 smithi164 conmon[29288]: 1156 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:08 smithi164 conmon[29288]: cluster 2021-02-10T04:39: 2021-02-10T04:39:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:08 smithi164 conmon[29288]: 08.169979+0000 mgr.y (mgr.14140) 736 : cluster [DBG] pgmap v1243: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 473 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 2.1 MiB/s rd, 308 KiB/s wr, 4 op/s; 172 B/s, 2 objects/s recovering 2021-02-10T04:39:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:08 smithi164 conmon[29288]: 2021-02-10T04:39:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:08 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:39:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:08 smithi164 conmon[29288]: -10T04:39:08.238578+0000 mon.a (mon.0) 1645 : cluster [DBG] 2021-02-10T04:39:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:08 smithi164 conmon[29288]: osdmap e729: 8 total, 8 up, 7 in 2021-02-10T04:39:08.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:08 smithi085 conmon[29319]: cluster 2021-02-10T04:39:07.843088+0000 2021-02-10T04:39:08.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:08 smithi085 conmon[29319]: osd.4 (osd.4) 1153 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:08.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:08 smithi085 conmon[29319]: cluster 2021-02-10T04:39:07.843105+0000 osd.4 (osd.4) 1154 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:08.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:08 smithi085 conmon[29319]: cluster 2021-02-10T04:39:07.843112+0000 osd.4 (osd.4) 1155 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:08.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:08 smithi085 conmon[29319]: cluster 2021-02-10T04:39:07.843118+0000 osd.4 (osd.4) 1156 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:08.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:08 smithi085 conmon[29319]: cluster 2021-02-10T04:39:08.169979+0000 mgr.y (mgr.14140) 736 : cluster [DBG] pgmap v1243: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 473 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 2.1 MiB/s rd, 308 KiB/s wr, 4 op/s; 172 B/s, 2 objects/s recovering 2021-02-10T04:39:08.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:08 smithi085 conmon[29319]: cluster 2021-02-10T04:39 2021-02-10T04:39:08.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:08 smithi085 conmon[29319]: :08.238578+0000 mon.a (mon.0) 2021-02-10T04:39:08.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:08 smithi085 conmon[29319]: 1645 : cluster [DBG] osdmap e729: 8 total, 8 up, 7 in 2021-02-10T04:39:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:08 smithi085 conmon[35961]: cluster 2021-02-10T04:39:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:08 smithi085 conmon[35961]: 2021-02-10T04:39:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:08 smithi085 conmon[35961]: 2021-02-10T04:39:07 2021-02-10T04:39:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:08 smithi085 conmon[35961]: .843088+0000 osd.4 (osd.4) 1153 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:08.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:08 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:39:08.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:08 smithi085 conmon[35961]: :39:07.843105+0000 osd.4 (osd.4) 1154 : 2021-02-10T04:39:08.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:08 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:08.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:08 smithi085 conmon[35961]: cluster 2021-02-10T04:39: 2021-02-10T04:39:08.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:08 smithi085 conmon[35961]: 07.843112+0000 osd.4 (osd.4) 1155 : cluster 2021-02-10T04:39:08.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:08 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:08.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:08 smithi085 conmon[35961]: cluster 2021-02-10T04:39:07 2021-02-10T04:39:08.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:08 smithi085 conmon[35961]: .843118+0000 osd.4 (osd.4) 1156 : cluster [WRN] 2021-02-10T04:39:08.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:08 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:08.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:08 smithi085 conmon[35961]: cluster 2021-02-10T04:39: 2021-02-10T04:39:08.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:08 smithi085 conmon[35961]: 08.169979+0000 mgr.y (mgr.14140) 736 : cluster 2021-02-10T04:39:08.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:08 smithi085 conmon[35961]: [DBG] pgmap v1243: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 473 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 2.1 MiB/s rd, 308 KiB/s wr, 4 op/s; 172 B/s, 2 objects/s recovering 2021-02-10T04:39:08.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:08 smithi085 conmon[35961]: cluster 2021-02-10T04:39 2021-02-10T04:39:08.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:08 smithi085 conmon[35961]: :08.238578+0000 mon.a (mon.0) 1645 : cluster 2021-02-10T04:39:08.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:08 smithi085 conmon[35961]: [DBG] osdmap e729: 8 total, 8 up, 7 in 2021-02-10T04:39:09.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:08 smithi164 conmon[33677]: debug 2021-02-10T04:39:08.849+0000 7fbec19a1700 -1 osd.4 728 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:09.246 INFO:tasks.rados.rados.0.smithi085.stdout:2858: done (7 left) 2021-02-10T04:39:09.247 INFO:tasks.rados.rados.0.smithi085.stdout:2859: write (excl) oid 9 current snap is 326 2021-02-10T04:39:09.247 INFO:tasks.rados.rados.0.smithi085.stdout:2859: seq_num 995 ranges {722965=798645,2125348=597617} 2021-02-10T04:39:09.256 INFO:tasks.rados.rados.0.smithi085.stdout:2859: writing smithi08575397-9 from 722965 to 1521610 tid 1 2021-02-10T04:39:09.264 INFO:tasks.rados.rados.0.smithi085.stdout:2859: writing smithi08575397-9 from 2125348 to 2722965 tid 2 2021-02-10T04:39:09.265 INFO:tasks.rados.rados.0.smithi085.stdout:2857: finishing copy_from to smithi08575397-36 2021-02-10T04:39:09.265 INFO:tasks.rados.rados.0.smithi085.stdout:2857: got expected ENOENT (src dne) 2021-02-10T04:39:09.265 INFO:tasks.rados.rados.0.smithi085.stdout:2857: finishing copy_from racing read to smithi08575397-36 2021-02-10T04:39:09.265 INFO:tasks.rados.rados.0.smithi085.stdout:2855: finishing write tid 2 to smithi08575397-5 2021-02-10T04:39:09.265 INFO:tasks.rados.rados.0.smithi085.stdout:2855: finishing write tid 3 to smithi08575397-5 2021-02-10T04:39:09.265 INFO:tasks.rados.rados.0.smithi085.stdout:2856: finishing rollback tid 0 to smithi08575397-42 2021-02-10T04:39:09.266 INFO:tasks.rados.rados.0.smithi085.stdout:2855: finishing write tid 4 to smithi08575397-5 2021-02-10T04:39:09.266 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 804 (ObjNum 994 snap 326 seq_num 994) dirty exists 2021-02-10T04:39:09.266 INFO:tasks.rados.rados.0.smithi085.stdout:2855: left oid 5 (ObjNum 994 snap 326 seq_num 994) 2021-02-10T04:39:09.266 INFO:tasks.rados.rados.0.smithi085.stdout:2856: finishing rollback tid 1 to smithi08575397-42 2021-02-10T04:39:09.266 INFO:tasks.rados.rados.0.smithi085.stdout:2856: finishing rollback tid 2 to smithi08575397-42 2021-02-10T04:39:09.266 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 1357 (ObjNum 833 snap 270 seq_num 833) dirty exists 2021-02-10T04:39:09.266 INFO:tasks.rados.rados.0.smithi085.stdout:2859: finishing write tid 1 to smithi08575397-9 2021-02-10T04:39:09.266 INFO:tasks.rados.rados.0.smithi085.stdout:2855: done (7 left) 2021-02-10T04:39:09.266 INFO:tasks.rados.rados.0.smithi085.stdout:2856: done (6 left) 2021-02-10T04:39:09.266 INFO:tasks.rados.rados.0.smithi085.stdout:2857: done (5 left) 2021-02-10T04:39:09.266 INFO:tasks.rados.rados.0.smithi085.stdout:2860: read oid 35 snap -1 2021-02-10T04:39:09.266 INFO:tasks.rados.rados.0.smithi085.stdout:2860: expect (ObjNum 986 snap 320 seq_num 986) 2021-02-10T04:39:09.267 INFO:tasks.rados.rados.0.smithi085.stdout:2861: read oid 50 snap -1 2021-02-10T04:39:09.267 INFO:tasks.rados.rados.0.smithi085.stdout:2861: expect (ObjNum 852 snap 276 seq_num 852) 2021-02-10T04:39:09.267 INFO:tasks.rados.rados.0.smithi085.stdout:2862: snap_create 2021-02-10T04:39:09.275 INFO:tasks.rados.rados.0.smithi085.stdout:2860: expect (ObjNum 986 snap 320 seq_num 986) 2021-02-10T04:39:09.496 INFO:tasks.rados.rados.0.smithi085.stdout:2861: expect (ObjNum 852 snap 276 seq_num 852) 2021-02-10T04:39:09.601 INFO:tasks.rados.rados.0.smithi085.stdout:2859: finishing write tid 2 to smithi08575397-9 2021-02-10T04:39:09.602 INFO:tasks.rados.rados.0.smithi085.stdout:2859: finishing write tid 3 to smithi08575397-9 2021-02-10T04:39:09.602 INFO:tasks.rados.rados.0.smithi085.stdout:2859: finishing write tid 4 to smithi08575397-9 2021-02-10T04:39:09.602 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 9 v 1360 (ObjNum 995 snap 326 seq_num 995) dirty exists 2021-02-10T04:39:09.602 INFO:tasks.rados.rados.0.smithi085.stdout:2859: left oid 9 (ObjNum 995 snap 326 seq_num 995) 2021-02-10T04:39:09.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:39:09 smithi164 conmon[37263]: debug 2021-02-10T04:39:09.320+0000 7f1c671e2700 -1 osd.5 pg_epoch: 730 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x556650695080) 2021-02-10T04:39:09.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:09 smithi164 conmon[29288]: audit 2021-02-10T04:39:08.315049+0000 2021-02-10T04:39:09.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:09 smithi164 conmon[29288]: mgr.y (mgr.14140) 737 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:39:09.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:09 smithi164 conmon[29288]: cluster 2021-02-10T04:39:08.434022+0000 mon.a (mon.0) 1646 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 577 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:39:09.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:09 smithi164 conmon[29288]: cluster 2021-02-10T04:39:08.849924+0000 osd.4 (osd.4) 1157 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:09.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:09 smithi164 conmon[29288]: 2021-02-10T04:39:09.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:09 smithi164 conmon[29288]: cluster 2021-02-10T04:39:09.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:09 smithi164 conmon[29288]: 2021-02-10T04:39:08.849929+0000 osd.4 (osd.4) 1158 2021-02-10T04:39:09.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:09 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:09.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:09 smithi164 conmon[29288]: cluster 2021-02-10T04:39:09.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:09 smithi164 conmon[29288]: 2021-02-10T04:39:08.849933+0000 osd.4 (osd.4) 1159 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:09.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:09 smithi164 conmon[29288]: 2021-02-10T04:39:09.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:09 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:39:09.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:09 smithi164 conmon[29288]: -10T04:39:08.849936+0000 osd.4 (osd.4) 2021-02-10T04:39:09.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:09 smithi164 conmon[29288]: 1160 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:09.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:09 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:39:09.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:09 smithi164 conmon[29288]: 10T04:39:09.246786+0000 mon.a 2021-02-10T04:39:09.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:09 smithi164 conmon[29288]: (mon.0) 1647 : cluster [DBG] osdmap e730: 8 total, 8 up, 7 in 2021-02-10T04:39:09.817 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[29319]: audit 2021-02-10T04:39:08.315049+0000 2021-02-10T04:39:09.817 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[29319]: mgr.y (mgr.14140) 737 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:39:09.817 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[29319]: cluster 2021-02-10T04:39:08.434022+0000 mon.a (mon.0) 2021-02-10T04:39:09.817 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[29319]: 1646 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 577 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:39:09.818 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:39:09.818 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[29319]: -02-10T04:39:08.849924+0000 osd.4 2021-02-10T04:39:09.818 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[29319]: (osd.4) 1157 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:09.818 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:39:09.818 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[29319]: -10T04:39:08.849929+0000 osd.4 ( 2021-02-10T04:39:09.818 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[29319]: osd.4) 1158 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:09.818 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:39:09.818 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[29319]: :39:08.849933+0000 osd.4 (osd.4 2021-02-10T04:39:09.818 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[29319]: ) 1159 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:09.818 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[29319]: cluster 2021-02-10T04:39 2021-02-10T04:39:09.818 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[29319]: :08.849936+0000 osd.4 (osd.4) 1160 2021-02-10T04:39:09.819 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:09.819 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[29319]: cluster 2021-02-10T04:39 2021-02-10T04:39:09.819 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[29319]: :09.246786+0000 mon.a ( 2021-02-10T04:39:09.819 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[29319]: mon.0) 1647 : cluster [DBG] 2021-02-10T04:39:09.819 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[29319]: osdmap e730: 8 total, 8 up, 7 in 2021-02-10T04:39:09.819 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: audit 2021-02-10T04:39:09.819 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: 2021-02-10T04:39:09.819 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: 2021 2021-02-10T04:39:09.819 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: -02-10T04:39:08.315049 2021-02-10T04:39:09.819 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 737 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:39:09.819 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: cluster 2021-02-10T04:39:09.820 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:39:09.820 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: 10T04:39:08.434022+0000 2021-02-10T04:39:09.820 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: mon.a (mon.0) 1646 : cluster [WRN] 2021-02-10T04:39:09.820 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: Health check update: 4 slow ops, oldest one blocked for 577 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:39:09.820 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: cluster 2021-02-10T04:39:09.820 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: 2021- 2021-02-10T04:39:09.820 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: 02-10T04:39:08.849924+0000 2021-02-10T04:39:09.820 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: osd.4 (osd.4) 1157 : cluster [WRN] 2021-02-10T04:39:09.820 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:09.820 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: cluster 2021-02-10T04:39:09.820 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: 2021- 2021-02-10T04:39:09.820 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: 02-10T04:39:08.849929+0000 2021-02-10T04:39:09.820 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: osd.4 (osd.4) 1158 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:09.821 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: 2021-02-10T04:39:09.821 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: cluster 2021-02-10T04:39:09.821 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:39:09.821 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: 04:39:08.849933+0000 osd.4 (osd.4) 1159 2021-02-10T04:39:09.821 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:09.821 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: cluster 2021-02-10T04:39:09.821 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: 2021- 2021-02-10T04:39:09.821 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: 02-10T04:39:08.849936+0000 osd.4 2021-02-10T04:39:09.821 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: (osd.4) 1160 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:09.821 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: cluster 2021-02-10T04:39:09.821 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:39:09.821 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: 10T04:39:09.246786+0000 mon.a (mon. 2021-02-10T04:39:09.821 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:09 smithi085 conmon[35961]: 0) 1647 : cluster [DBG] osdmap e730: 8 total, 8 up, 7 in 2021-02-10T04:39:09.923 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:39:10.100 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:09 smithi164 conmon[33677]: debug 2021-02-10T04:39:09.845+0000 7fbec19a1700 -1 osd.4 730 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:10.103 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:39:10.249 INFO:tasks.rados.rados.0.smithi085.stdout:2859: done (7 left) 2021-02-10T04:39:10.249 INFO:tasks.rados.rados.0.smithi085.stdout:2860: done (6 left) 2021-02-10T04:39:10.249 INFO:tasks.rados.rados.0.smithi085.stdout:2861: done (5 left) 2021-02-10T04:39:10.250 INFO:tasks.rados.rados.0.smithi085.stdout:2862: done (4 left) 2021-02-10T04:39:10.250 INFO:tasks.rados.rados.0.smithi085.stdout:2863: write (excl) oid 46 current snap is 327 2021-02-10T04:39:10.250 INFO:tasks.rados.rados.0.smithi085.stdout:2863: seq_num 996 ranges {739772=590247,1900183=540336} 2021-02-10T04:39:10.257 INFO:tasks.rados.rados.0.smithi085.stdout:2863: writing smithi08575397-46 from 739772 to 1330019 tid 1 2021-02-10T04:39:10.264 INFO:tasks.rados.rados.0.smithi085.stdout:2863: writing smithi08575397-46 from 1900183 to 2440519 tid 2 2021-02-10T04:39:10.265 INFO:tasks.rados.rados.0.smithi085.stdout:2864: rollback oid 40 current snap is 327 2021-02-10T04:39:10.265 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 40 to 299 2021-02-10T04:39:10.265 INFO:tasks.rados.rados.0.smithi085.stdout:2865: rollback oid 1 current snap is 327 2021-02-10T04:39:10.265 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 1 to 313 2021-02-10T04:39:10.265 INFO:tasks.rados.rados.0.smithi085.stdout:2866: snap_remove snap 300 2021-02-10T04:39:10.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:39:10 smithi164 conmon[37263]: debug 2021-02-10T04:39:10.284+0000 7f1c6b1ea700 -1 osd.5 pg_epoch: 731 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x556655fa69a0) 2021-02-10T04:39:10.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:39:10 smithi164 conmon[40892]: debug 2021-02-10T04:39:10.174+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:10.523 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:39:10.629 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:39:10.687 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:39:10.763 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:10 smithi085 conmon[29319]: cluster 2021-02-10T04:39:09.846628+0000 osd.4 2021-02-10T04:39:10.763 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:10 smithi085 conmon[29319]: (osd.4) 1161 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:10.763 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:10 smithi085 conmon[29319]: cluster 2021-02-10T04:39:09.846637+0000 osd.4 (osd.4) 1162 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:10.763 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:10 smithi085 conmon[29319]: cluster 2021-02-10T04:39:10.763 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:10 smithi085 conmon[29319]: 2021-02-10T04:39:09.846644+0000 2021-02-10T04:39:10.763 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:10 smithi085 conmon[29319]: osd.4 (osd.4) 1163 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:10.763 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:10 smithi085 conmon[29319]: 2021-02-10T04:39:10.763 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:10 smithi085 conmon[29319]: cluster 2021-02-10T04:39:09.846650+0000 osd.4 (osd.4) 1164 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:10.763 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:10 smithi085 conmon[29319]: cluster 2021-02-10T04:39:10.170548+0000 mgr.y (mgr.14140) 738 : cluster [DBG] pgmap v1246: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 480 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 5.7 MiB/s rd, 920 KiB/s wr, 9 op/s; 172 B/s, 2 objects/s recovering 2021-02-10T04:39:10.763 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:10 smithi085 conmon[29319]: cluster 2021-02-10T04:39:10.248704+0000 mon.a (mon.0) 1648 : cluster [DBG] osdmap e731: 8 total, 8 up, 7 in 2021-02-10T04:39:10.764 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:39:10 smithi085 conmon[45443]: debug 2021-02-10T04:39:10.758+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:10.764 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:10 smithi085 conmon[35961]: cluster 2021-02-10T04:39:09.846628 2021-02-10T04:39:10.764 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:10 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1161 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:10.764 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:10 smithi085 conmon[35961]: cluster 2021-02-10T04:39:10.764 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:10 smithi085 conmon[35961]: 2021-02-10T04:39:09.846637+0000 osd.4 (osd.4) 1162 : 2021-02-10T04:39:10.764 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:10 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:10.764 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:10 smithi085 conmon[35961]: cluster 2021-02-10T04:39 2021-02-10T04:39:10.764 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:10 smithi085 conmon[35961]: :09.846644+0000 osd.4 (osd. 2021-02-10T04:39:10.764 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:10 smithi085 conmon[35961]: 4) 1163 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:10.764 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:10 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:39:10.764 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:10 smithi085 conmon[35961]: 04:39:09.846650+0000 osd.4 (osd. 2021-02-10T04:39:10.765 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:10 smithi085 conmon[35961]: 4) 1164 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:10.765 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:10 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:39:10.765 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:10 smithi085 conmon[35961]: 10T04:39:10.170548+0000 mgr.y 2021-02-10T04:39:10.765 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:10 smithi085 conmon[35961]: (mgr.14140) 738 : cluster [DBG] pgmap v1246: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 480 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 5.7 MiB/s rd, 920 KiB/s wr, 9 op/s; 172 B/s, 2 objects/s recovering 2021-02-10T04:39:10.765 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:10 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:39:10.765 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:10 smithi085 conmon[35961]: 02-10T04:39:10.248704+0000 mon.a (mon 2021-02-10T04:39:10.765 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:10 smithi085 conmon[35961]: .0) 1648 : cluster [DBG] osdmap e731: 8 total, 8 up, 7 in 2021-02-10T04:39:10.849 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:10 smithi164 conmon[29288]: cluster 2021-02-10T04:39:09.846628+0000 osd.4 (osd.4) 1161 2021-02-10T04:39:10.849 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:10 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:10.849 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:10 smithi164 conmon[29288]: cluster 2021-02-10T04:39:09.846637+0000 osd.4 (osd.4) 1162 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:10.849 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:10 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:39:10.850 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:10 smithi164 conmon[29288]: -02-10T04:39:09.846644+0000 osd.4 (osd.4) 1163 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:10.850 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:10 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:39:10.850 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:10 smithi164 conmon[29288]: -02-10T04:39: 2021-02-10T04:39:10.850 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:10 smithi164 conmon[29288]: 09.846650+0000 osd.4 (osd.4) 1164 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:10.850 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:10 smithi164 conmon[29288]: 2021-02-10T04:39:10.850 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:10 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:39:10.850 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:10 smithi164 conmon[29288]: 39:10.170548+0000 mgr.y (mgr.14140) 738 : 2021-02-10T04:39:10.850 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:10 smithi164 conmon[29288]: cluster [DBG] pgmap v1246: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 480 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 5.7 MiB/s rd, 920 KiB/s wr, 9 op/s; 172 B/s, 2 objects/s recovering 2021-02-10T04:39:10.850 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:10 smithi164 conmon[29288]: cluster 2021-02-10T04:39:10.248704+0000 mon.a (mon 2021-02-10T04:39:10.850 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:10 smithi164 conmon[29288]: .0) 1648 : cluster [DBG] osdmap e731: 8 total, 8 up, 7 in 2021-02-10T04:39:10.921 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:39:11.143 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:10 smithi164 conmon[33677]: debug 2021-02-10T04:39:10.848+0000 7fbec19a1700 -1 osd.4 731 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:11.257 INFO:tasks.rados.rados.0.smithi085.stdout:2866: done (7 left) 2021-02-10T04:39:11.257 INFO:tasks.rados.rados.0.smithi085.stdout:2867: write oid 28 current snap is 327 2021-02-10T04:39:11.257 INFO:tasks.rados.rados.0.smithi085.stdout:2867: seq_num 997 ranges {756579=665496,1958665=400347} 2021-02-10T04:39:11.258 INFO:tasks.rados.rados.0.smithi085.stdout:2867: writing smithi08575397-28 from 756579 to 1422075 tid 1 2021-02-10T04:39:11.265 INFO:tasks.rados.rados.0.smithi085.stdout:2867: writing smithi08575397-28 from 1958665 to 2359012 tid 2 2021-02-10T04:39:11.265 INFO:tasks.rados.rados.0.smithi085.stdout:2868: delete oid 42 current snap is 327 2021-02-10T04:39:11.265 INFO:tasks.rados.rados.0.smithi085.stdout:2863: finishing write tid 1 to smithi08575397-46 2021-02-10T04:39:11.265 INFO:tasks.rados.rados.0.smithi085.stdout:2865: finishing rollback tid 1 to smithi08575397-1 2021-02-10T04:39:11.265 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 1340 (ObjNum 0 snap 0 seq_num 80674832) dirty dne 2021-02-10T04:39:11.265 INFO:tasks.rados.rados.0.smithi085.stdout:2864: finishing rollback tid 0 to smithi08575397-40 2021-02-10T04:39:11.265 INFO:tasks.rados.rados.0.smithi085.stdout:2863: finishing write tid 2 to smithi08575397-46 2021-02-10T04:39:11.266 INFO:tasks.rados.rados.0.smithi085.stdout:2864: finishing rollback tid 1 to smithi08575397-40 2021-02-10T04:39:11.266 INFO:tasks.rados.rados.0.smithi085.stdout:2864: finishing rollback tid 2 to smithi08575397-40 2021-02-10T04:39:11.266 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 1498 (ObjNum 879 snap 282 seq_num 879) dirty exists 2021-02-10T04:39:11.266 INFO:tasks.rados.rados.0.smithi085.stdout:2863: finishing write tid 3 to smithi08575397-46 2021-02-10T04:39:11.266 INFO:tasks.rados.rados.0.smithi085.stdout:2863: finishing write tid 4 to smithi08575397-46 2021-02-10T04:39:11.266 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 1362 (ObjNum 996 snap 327 seq_num 996) dirty exists 2021-02-10T04:39:11.266 INFO:tasks.rados.rados.0.smithi085.stdout:2863: left oid 46 (ObjNum 996 snap 327 seq_num 996) 2021-02-10T04:39:11.266 INFO:tasks.rados.rados.0.smithi085.stdout:2867: finishing write tid 1 to smithi08575397-28 2021-02-10T04:39:11.268 INFO:tasks.rados.rados.0.smithi085.stdout:2863: done (8 left) 2021-02-10T04:39:11.268 INFO:tasks.rados.rados.0.smithi085.stdout:2864: done (7 left) 2021-02-10T04:39:11.268 INFO:tasks.rados.rados.0.smithi085.stdout:2865: done (6 left) 2021-02-10T04:39:11.269 INFO:tasks.rados.rados.0.smithi085.stdout:2868: done (5 left) 2021-02-10T04:39:11.269 INFO:tasks.rados.rados.0.smithi085.stdout:2869: read oid 15 snap -1 2021-02-10T04:39:11.269 INFO:tasks.rados.rados.0.smithi085.stdout:2869: expect (ObjNum 962 snap 315 seq_num 962) 2021-02-10T04:39:11.269 INFO:tasks.rados.rados.0.smithi085.stdout:2870: write oid 14 current snap is 327 2021-02-10T04:39:11.269 INFO:tasks.rados.rados.0.smithi085.stdout:2870: seq_num 999 ranges {790193=415994,1959276=404016} 2021-02-10T04:39:11.276 INFO:tasks.rados.rados.0.smithi085.stdout:2870: writing smithi08575397-14 from 790193 to 1206187 tid 1 2021-02-10T04:39:11.284 INFO:tasks.rados.rados.0.smithi085.stdout:2870: writing smithi08575397-14 from 1959276 to 2363292 tid 2 2021-02-10T04:39:11.471 INFO:tasks.rados.rados.0.smithi085.stdout:2871: delete oid 50 current snap is 327 2021-02-10T04:39:11.471 INFO:tasks.rados.rados.0.smithi085.stdout:2867: finishing write tid 2 to smithi08575397-28 2021-02-10T04:39:11.471 INFO:tasks.rados.rados.0.smithi085.stdout:2867: finishing write tid 3 to smithi08575397-28 2021-02-10T04:39:11.471 INFO:tasks.rados.rados.0.smithi085.stdout:2867: finishing write tid 4 to smithi08575397-28 2021-02-10T04:39:11.471 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 1281 (ObjNum 997 snap 327 seq_num 997) dirty exists 2021-02-10T04:39:11.471 INFO:tasks.rados.rados.0.smithi085.stdout:2867: left oid 28 (ObjNum 997 snap 327 seq_num 997) 2021-02-10T04:39:11.471 INFO:tasks.rados.rados.0.smithi085.stdout:2869: expect (ObjNum 962 snap 315 seq_num 962) 2021-02-10T04:39:11.473 INFO:tasks.rados.rados.0.smithi085.stdout:2870: finishing write tid 1 to smithi08575397-14 2021-02-10T04:39:11.473 INFO:tasks.rados.rados.0.smithi085.stdout:2867: done (7 left) 2021-02-10T04:39:11.473 INFO:tasks.rados.rados.0.smithi085.stdout:2869: done (6 left) 2021-02-10T04:39:11.473 INFO:tasks.rados.rados.0.smithi085.stdout:2871: done (5 left) 2021-02-10T04:39:11.473 INFO:tasks.rados.rados.0.smithi085.stdout:2872: read oid 32 snap -1 2021-02-10T04:39:11.473 INFO:tasks.rados.rados.0.smithi085.stdout:2872: expect (ObjNum 986 snap 320 seq_num 986) 2021-02-10T04:39:11.473 INFO:tasks.rados.rados.0.smithi085.stdout:2873: copy_from oid 28 from oid 18 current snap is 327 2021-02-10T04:39:11.473 INFO:tasks.rados.rados.0.smithi085.stdout:2870: finishing write tid 2 to smithi08575397-14 2021-02-10T04:39:11.473 INFO:tasks.rados.rados.0.smithi085.stdout:2870: finishing write tid 3 to smithi08575397-14 2021-02-10T04:39:11.473 INFO:tasks.rados.rados.0.smithi085.stdout:2870: finishing write tid 4 to smithi08575397-14 2021-02-10T04:39:11.473 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 1364 (ObjNum 999 snap 327 seq_num 999) dirty exists 2021-02-10T04:39:11.473 INFO:tasks.rados.rados.0.smithi085.stdout:2870: left oid 14 (ObjNum 999 snap 327 seq_num 999) 2021-02-10T04:39:11.474 INFO:tasks.rados.rados.0.smithi085.stdout:2870: done (6 left) 2021-02-10T04:39:11.474 INFO:tasks.rados.rados.0.smithi085.stdout:2874: read oid 16 snap -1 2021-02-10T04:39:11.474 INFO:tasks.rados.rados.0.smithi085.stdout:2874: expect deleted 2021-02-10T04:39:11.474 INFO:tasks.rados.rados.0.smithi085.stdout:2875: rollback oid 12 current snap is 327 2021-02-10T04:39:11.474 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 12 to 299 2021-02-10T04:39:11.474 INFO:tasks.rados.rados.0.smithi085.stdout:2876: delete oid 40 current snap is 327 2021-02-10T04:39:11.474 INFO:tasks.rados.rados.0.smithi085.stdout:2873: finishing copy_from to smithi08575397-28 2021-02-10T04:39:11.474 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 1283 (ObjNum 952 snap 313 seq_num 952) dirty exists 2021-02-10T04:39:11.474 INFO:tasks.rados.rados.0.smithi085.stdout:2873: finishing copy_from racing read to smithi08575397-28 2021-02-10T04:39:11.474 INFO:tasks.rados.rados.0.smithi085.stdout:2872: expect (ObjNum 986 snap 320 seq_num 986) 2021-02-10T04:39:11.630 INFO:tasks.rados.rados.0.smithi085.stdout:2872: done (8 left) 2021-02-10T04:39:11.630 INFO:tasks.rados.rados.0.smithi085.stdout:2873: done (7 left) 2021-02-10T04:39:11.630 INFO:tasks.rados.rados.0.smithi085.stdout:2874: done (6 left) 2021-02-10T04:39:11.630 INFO:tasks.rados.rados.0.smithi085.stdout:2876: done (5 left) 2021-02-10T04:39:11.630 INFO:tasks.rados.rados.0.smithi085.stdout:2877: rollback oid 34 current snap is 327 2021-02-10T04:39:11.630 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 34 to 278 2021-02-10T04:39:11.630 INFO:tasks.rados.rados.0.smithi085.stdout:2875: finishing rollback tid 1 to smithi08575397-12 2021-02-10T04:39:11.630 INFO:tasks.rados.rados.0.smithi085.stdout:2875: finishing rollback tid 2 to smithi08575397-12 2021-02-10T04:39:11.631 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 1366 (ObjNum 849 snap 276 seq_num 849) dirty exists 2021-02-10T04:39:11.631 INFO:tasks.rados.rados.0.smithi085.stdout:2875: done (5 left) 2021-02-10T04:39:11.631 INFO:tasks.rados.rados.0.smithi085.stdout:2878: delete oid 45 current snap is 327 2021-02-10T04:39:11.632 INFO:tasks.rados.rados.0.smithi085.stdout:2878: done (5 left) 2021-02-10T04:39:11.632 INFO:tasks.rados.rados.0.smithi085.stdout:2879: snap_create 2021-02-10T04:39:11.633 INFO:tasks.rados.rados.0.smithi085.stdout:2877: finishing rollback tid 1 to smithi08575397-34 2021-02-10T04:39:11.634 INFO:tasks.rados.rados.0.smithi085.stdout:2877: finishing rollback tid 2 to smithi08575397-34 2021-02-10T04:39:11.634 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 34 v 806 (ObjNum 859 snap 277 seq_num 859) dirty exists 2021-02-10T04:39:11.645 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:39:11 smithi164 conmon[40892]: debug 2021-02-10T04:39:11.278+0000 7f30354c5700 -1 osd.6 pg_epoch: 732 pg[7.7( v 732'1500 (0'0,732'1500] local-lis/les=93/94 n=31 ec=93/93 lis/c=93/93 les/c/f=94/94/0 sis=93) [6,0,3] r=0 lpr=93 luod=731'1498 crt=732'1500 lcod 731'1497 mlcod 731'1497 active+clean+snaptrim trimq=[1b6~1,22f~1,241~1,246~1,248~1,24c~1,255~1,25d~1,263~1,265~1] ps=[20a~1,210~1,212~1,214~1,223~3,227~1,229~1,22e~1,233~1,236~3,23b~1,244~1]] removing snap head 2021-02-10T04:39:11.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:11 smithi164 conmon[29288]: cluster 2021-02-10T04:39:10.414868+0000 osd.2 (osd.2) 1935 2021-02-10T04:39:11.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:11 smithi164 conmon[29288]: : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:39:11.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:11 smithi164 conmon[29288]: cluster 2021-02-10T04:39:10.415704+0000 osd.2 (osd.2) 1936 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:39:11.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:11 smithi164 conmon[29288]: cluster 2021-02-10T04:39:10.849049+0000 osd.4 (osd.4 2021-02-10T04:39:11.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:11 smithi164 conmon[29288]: ) 1165 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:11.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:11 smithi164 conmon[29288]: cluster 2021-02-10T04:39:10.849056+0000 osd.4 2021-02-10T04:39:11.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:11 smithi164 conmon[29288]: (osd.4) 1166 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:11.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:11 smithi164 conmon[29288]: cluster 2021-02-10T04:39:10 2021-02-10T04:39:11.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:11 smithi164 conmon[29288]: .849065+0000 osd.4 (osd.4) 1167 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:11.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:11 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:39:11.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:11 smithi164 conmon[29288]: :39:10.849070+0000 osd.4 (osd.4) 1168 : cluster [WRN] 2021-02-10T04:39:11.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:11 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:11.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:11 smithi164 conmon[29288]: cluster 2021-02-10T04:39:11.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:11 smithi164 conmon[29288]: 2021-02-10T04:39:11.249363+0000 mon.a (mon.0) 1649 : cluster 2021-02-10T04:39:11.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:11 smithi164 conmon[29288]: [DBG] osdmap e732: 8 total, 8 up, 7 in 2021-02-10T04:39:11.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[29319]: cluster 2021-02-10T04:39: 2021-02-10T04:39:11.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[29319]: 10.414868+0000 osd.2 (osd.2) 1935 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:39:11.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[29319]: cluster 2021-02-10T04:39:10.415704 2021-02-10T04:39:11.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 1936 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:39:11.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[29319]: cluster 2021-02-10T04:39:10. 2021-02-10T04:39:11.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[29319]: 849049+0000 osd.4 (osd.4) 1165 : cluster [WRN] 2021-02-10T04:39:11.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:11.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:39:11.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[29319]: -02-10T04:39:10.849056+0000 osd.4 (osd.4) 2021-02-10T04:39:11.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[29319]: 1166 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:11.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[29319]: cluster 2021-02-10T04:39:10 2021-02-10T04:39:11.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[29319]: .849065+0000 osd.4 (osd.4) 1167 : cluster 2021-02-10T04:39:11.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:11.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[29319]: cluster 2021-02-10T04:39: 2021-02-10T04:39:11.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[29319]: 10.849070+0000 osd.4 (osd.4) 1168 : cluster 2021-02-10T04:39:11.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:11.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[29319]: cluster 2021-02-10T04:39:11.249363 2021-02-10T04:39:11.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1649 : cluster [DBG] 2021-02-10T04:39:11.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[29319]: osdmap e732: 8 total, 8 up, 7 in 2021-02-10T04:39:11.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[35961]: cluster 2021-02-10T04:39:10.414868+0000 osd.2 2021-02-10T04:39:11.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[35961]: (osd.2) 1935 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:39:11.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[35961]: cluster 2021-02-10T04:39:10. 2021-02-10T04:39:11.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[35961]: 415704+0000 osd.2 (osd.2) 1936 : cluster 2021-02-10T04:39:11.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[35961]: [DBG] purged_snaps scrub ok 2021-02-10T04:39:11.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:39:11.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[35961]: :39:10.849049+0000 osd.4 (osd. 2021-02-10T04:39:11.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[35961]: 4) 1165 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:11.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:39:11.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[35961]: 10T04:39:10.849056+0000 osd.4 (osd.4) 2021-02-10T04:39:11.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[35961]: 1166 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:11.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:39:11.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[35961]: 02-10T04:39:10.849065+0000 2021-02-10T04:39:11.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[35961]: osd.4 (osd.4) 1167 : cluster 2021-02-10T04:39:11.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:11.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:39:11.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[35961]: 39:10.849070+0000 osd.4 (osd.4) 1168 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:11.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:39:11.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[35961]: T04:39:11.249363+0000 mon.a (mon.0) 1649 : cluster [DBG] 2021-02-10T04:39:11.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:11 smithi085 conmon[35961]: osdmap e732: 8 total, 8 up, 7 in 2021-02-10T04:39:12.143 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:11 smithi164 conmon[33677]: debug 2021-02-10T04:39:11.842+0000 7fbec19a1700 -1 osd.4 732 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:12.464 INFO:tasks.rados.rados.0.smithi085.stdout:2877: done (5 left) 2021-02-10T04:39:12.464 INFO:tasks.rados.rados.0.smithi085.stdout:2879: done (4 left) 2021-02-10T04:39:12.464 INFO:tasks.rados.rados.0.smithi085.stdout:2880: delete oid 1 current snap is 328 2021-02-10T04:39:12.468 INFO:tasks.rados.rados.0.smithi085.stdout:2880: done (4 left) 2021-02-10T04:39:12.468 INFO:tasks.rados.rados.0.smithi085.stdout:2881: snap_create 2021-02-10T04:39:12.643 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:39:12 smithi164 conmon[37263]: debug 2021-02-10T04:39:12.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:39:12 smithi164 conmon[37263]: 2021-02-10T04:39:12.214+0000 7f1c671e2700 -1 osd.5 pg_epoch: 732 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x5566508aa420) 2021-02-10T04:39:12.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:39:12 smithi164 conmon[37263]: 2021-02-10T04:39:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:12 smithi164 conmon[29288]: cluster 2021-02-10T04:39:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:12 smithi164 conmon[29288]: 2021-02-10T04:39:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:12 smithi164 conmon[29288]: 2021-02-10T04:39:11. 2021-02-10T04:39:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:12 smithi164 conmon[29288]: 843444+0000 osd.4 (osd.4) 1169 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:12 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:39:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:12 smithi164 conmon[29288]: 39:11.843454+0000 osd.4 (osd.4) 1170 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:12 smithi164 conmon[29288]: 2021-02-10T04:39:12.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:12 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:39:12.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:12 smithi164 conmon[29288]: T04:39:11.843461+0000 osd.4 (osd.4 2021-02-10T04:39:12.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:12 smithi164 conmon[29288]: ) 1171 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:12.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:12 smithi164 conmon[29288]: cluster 2021-02-10T04:39:12.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:12 smithi164 conmon[29288]: 2021 2021-02-10T04:39:12.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:12 smithi164 conmon[29288]: - 2021-02-10T04:39:12.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:12 smithi164 conmon[29288]: 02 2021-02-10T04:39:12.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:12 smithi164 conmon[29288]: - 2021-02-10T04:39:12.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:12 smithi164 conmon[29288]: 10T04:39:11.843467 2021-02-10T04:39:12.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:12 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1172 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:12.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:12 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:39:12.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:12 smithi164 conmon[29288]: -10T04:39:12.171151+0000 mgr.y (mgr 2021-02-10T04:39:12.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:12 smithi164 conmon[29288]: .14140) 739 : cluster [DBG] pgmap v1249: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 484 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.6 MiB/s rd, 1.8 MiB/s wr, 10 op/s 2021-02-10T04:39:12.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[29319]: cluster 2021-02-10T04:39:12.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[29319]: 2021-02-10T04:39:12.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[29319]: 2021 2021-02-10T04:39:12.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[29319]: - 2021-02-10T04:39:12.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[29319]: 02 2021-02-10T04:39:12.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[29319]: - 2021-02-10T04:39:12.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[29319]: 10 2021-02-10T04:39:12.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[29319]: T 2021-02-10T04:39:12.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[29319]: 04 2021-02-10T04:39:12.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[29319]: : 2021-02-10T04:39:12.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[29319]: 39 2021-02-10T04:39:12.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[29319]: : 2021-02-10T04:39:12.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[29319]: 11.843444+0000 osd.4 2021-02-10T04:39:12.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[29319]: (osd.4) 1169 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:12.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[29319]: cluster 2021-02-10T04:39:12.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:39:12.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[29319]: 39:11.843454+0000 osd.4 (osd.4 2021-02-10T04:39:12.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[29319]: ) 1170 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:12.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:39:12.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[29319]: -02-10T04:39:11.843461+0000 2021-02-10T04:39:12.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[29319]: osd.4 (osd.4) 1171 : cluster [WRN] 2021-02-10T04:39:12.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:12.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:39:12.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[29319]: -02-10T04:39:11.843467 2021-02-10T04:39:12.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1172 : cluster [WRN] 2021-02-10T04:39:12.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:12.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[29319]: cluster 2021-02-10T04:39:12.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[29319]: 2021-02-10T04:39:12.171151+0000 mgr.y ( 2021-02-10T04:39:12.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[29319]: mgr.14140) 739 : cluster [DBG] pgmap v1249: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 484 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.6 MiB/s rd, 1.8 MiB/s wr, 10 op/s 2021-02-10T04:39:12.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[35961]: cluster 2021-02-10T04:39:11.843444+0000 2021-02-10T04:39:12.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[35961]: osd.4 (osd.4) 1169 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:12.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[35961]: cluster 2021-02-10T04:39:12.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[35961]: 2021-02-10T04:39:11.843454+0000 osd.4 (osd.4) 1170 : 2021-02-10T04:39:12.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:12.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:39:12.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[35961]: T04:39:11.843461+0000 osd.4 (osd. 2021-02-10T04:39:12.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[35961]: 4) 1171 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:12.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:39:12.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[35961]: -10T04:39:11.843467+0000 osd.4 2021-02-10T04:39:12.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[35961]: (osd.4) 1172 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:12.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[35961]: 2021-02-10T04:39:12.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[35961]: cluster 2021-02-10T04:39:12.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[35961]: 2021-02-10T04:39:12.171151+0000 mgr.y ( 2021-02-10T04:39:12.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:12 smithi085 conmon[35961]: mgr.14140) 739 : cluster [DBG] pgmap v1249: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 484 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.6 MiB/s rd, 1.8 MiB/s wr, 10 op/s 2021-02-10T04:39:13.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:12 smithi164 conmon[33677]: debug 2021-02-10T04:39:13.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:12 smithi164 conmon[33677]: 2021-02-10T04:39:12.886+0000 7fbec19a1700 -1 osd.4 733 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:13.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:12 smithi164 conmon[33677]: 2021-02-10T04:39:13.245 INFO:tasks.rados.rados.0.smithi085.stdout:2881: done (4 left) 2021-02-10T04:39:13.245 INFO:tasks.rados.rados.0.smithi085.stdout:2882: rollback oid 34 current snap is 329 2021-02-10T04:39:13.246 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 34 to 322 2021-02-10T04:39:13.246 INFO:tasks.rados.rados.0.smithi085.stdout:2883: write (excl) oid 40 current snap is 329 2021-02-10T04:39:13.246 INFO:tasks.rados.rados.0.smithi085.stdout:2883: seq_num 1004 ranges {474228=508592,1451686=555012,2704920=169308} 2021-02-10T04:39:13.251 INFO:tasks.rados.rados.0.smithi085.stdout:2883: writing smithi08575397-40 from 474228 to 982820 tid 1 2021-02-10T04:39:13.258 INFO:tasks.rados.rados.0.smithi085.stdout:2883: writing smithi08575397-40 from 1451686 to 2006698 tid 2 2021-02-10T04:39:13.263 INFO:tasks.rados.rados.0.smithi085.stdout:2883: writing smithi08575397-40 from 2704920 to 2874228 tid 3 2021-02-10T04:39:13.264 INFO:tasks.rados.rados.0.smithi085.stdout:2884: write oid 32 current snap is 329 2021-02-10T04:39:13.264 INFO:tasks.rados.rados.0.smithi085.stdout:2884: seq_num 1005 ranges {491035=583841,1510168=698670,2732343=158692} 2021-02-10T04:39:13.270 INFO:tasks.rados.rados.0.smithi085.stdout:2884: writing smithi08575397-32 from 491035 to 1074876 tid 1 2021-02-10T04:39:13.277 INFO:tasks.rados.rados.0.smithi085.stdout:2884: writing smithi08575397-32 from 1510168 to 2208838 tid 2 2021-02-10T04:39:13.283 INFO:tasks.rados.rados.0.smithi085.stdout:2884: writing smithi08575397-32 from 2732343 to 2891035 tid 3 2021-02-10T04:39:13.283 INFO:tasks.rados.rados.0.smithi085.stdout:2885: read oid 37 snap -1 2021-02-10T04:39:13.283 INFO:tasks.rados.rados.0.smithi085.stdout:2885: expect (ObjNum 940 snap 311 seq_num 940) 2021-02-10T04:39:13.283 INFO:tasks.rados.rados.0.smithi085.stdout:2886: copy_from oid 29 from oid 44 current snap is 329 2021-02-10T04:39:13.283 INFO:tasks.rados.rados.0.smithi085.stdout:2882: finishing rollback tid 0 to smithi08575397-34 2021-02-10T04:39:13.283 INFO:tasks.rados.rados.0.smithi085.stdout:2882: finishing rollback tid 1 to smithi08575397-34 2021-02-10T04:39:13.283 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 34 v 808 (ObjNum 1 snap 0 seq_num 82679272) dirty dne 2021-02-10T04:39:13.283 INFO:tasks.rados.rados.0.smithi085.stdout:2883: finishing write tid 1 to smithi08575397-40 2021-02-10T04:39:13.283 INFO:tasks.rados.rados.0.smithi085.stdout:2882: done (8 left) 2021-02-10T04:39:13.283 INFO:tasks.rados.rados.0.smithi085.stdout:2887: delete oid 21 current snap is 329 2021-02-10T04:39:13.284 INFO:tasks.rados.rados.0.smithi085.stdout:2883: finishing write tid 2 to smithi08575397-40 2021-02-10T04:39:13.284 INFO:tasks.rados.rados.0.smithi085.stdout:2883: finishing write tid 3 to smithi08575397-40 2021-02-10T04:39:13.284 INFO:tasks.rados.rados.0.smithi085.stdout:2883: finishing write tid 4 to smithi08575397-40 2021-02-10T04:39:13.284 INFO:tasks.rados.rados.0.smithi085.stdout:2883: finishing write tid 5 to smithi08575397-40 2021-02-10T04:39:13.284 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 1507 (ObjNum 1004 snap 329 seq_num 1004) dirty exists 2021-02-10T04:39:13.284 INFO:tasks.rados.rados.0.smithi085.stdout:2883: left oid 40 (ObjNum 1004 snap 329 seq_num 1004) 2021-02-10T04:39:13.285 INFO:tasks.rados.rados.0.smithi085.stdout:2884: finishing write tid 1 to smithi08575397-32 2021-02-10T04:39:13.287 INFO:tasks.rados.rados.0.smithi085.stdout:2884: finishing write tid 2 to smithi08575397-32 2021-02-10T04:39:13.289 INFO:tasks.rados.rados.0.smithi085.stdout:2884: finishing write tid 3 to smithi08575397-32 2021-02-10T04:39:13.289 INFO:tasks.rados.rados.0.smithi085.stdout:2884: finishing write tid 4 to smithi08575397-32 2021-02-10T04:39:13.290 INFO:tasks.rados.rados.0.smithi085.stdout:2884: finishing write tid 5 to smithi08575397-32 2021-02-10T04:39:13.290 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 32 v 1357 (ObjNum 1005 snap 329 seq_num 1005) dirty exists 2021-02-10T04:39:13.290 INFO:tasks.rados.rados.0.smithi085.stdout:2884: left oid 32 (ObjNum 1005 snap 329 seq_num 1005) 2021-02-10T04:39:13.291 INFO:tasks.rados.rados.0.smithi085.stdout:2886: finishing copy_from to smithi08575397-29 2021-02-10T04:39:13.291 INFO:tasks.rados.rados.0.smithi085.stdout:2886: got expected ENOENT (src dne) 2021-02-10T04:39:13.292 INFO:tasks.rados.rados.0.smithi085.stdout:2886: finishing copy_from racing read to smithi08575397-29 2021-02-10T04:39:13.292 INFO:tasks.rados.rados.0.smithi085.stdout:2885: expect (ObjNum 940 snap 311 seq_num 940) 2021-02-10T04:39:13.468 INFO:tasks.rados.rados.0.smithi085.stdout:2883: done (8 left) 2021-02-10T04:39:13.469 INFO:tasks.rados.rados.0.smithi085.stdout:2884: done (7 left) 2021-02-10T04:39:13.469 INFO:tasks.rados.rados.0.smithi085.stdout:2885: done (6 left) 2021-02-10T04:39:13.469 INFO:tasks.rados.rados.0.smithi085.stdout:2886: done (5 left) 2021-02-10T04:39:13.470 INFO:tasks.rados.rados.0.smithi085.stdout:2887: done (4 left) 2021-02-10T04:39:13.470 INFO:tasks.rados.rados.0.smithi085.stdout:2888: write oid 20 current snap is 329 2021-02-10T04:39:13.471 INFO:tasks.rados.rados.0.smithi085.stdout:2888: seq_num 1007 ranges {524649=734339,1910779=702339} 2021-02-10T04:39:13.474 INFO:tasks.rados.rados.0.smithi085.stdout:2888: writing smithi08575397-20 from 524649 to 1258988 tid 1 2021-02-10T04:39:13.483 INFO:tasks.rados.rados.0.smithi085.stdout:2888: writing smithi08575397-20 from 1910779 to 2613118 tid 2 2021-02-10T04:39:13.484 INFO:tasks.rados.rados.0.smithi085.stdout:2889: read oid 43 snap 306 2021-02-10T04:39:13.484 INFO:tasks.rados.rados.0.smithi085.stdout:2889: expect (ObjNum 892 snap 289 seq_num 892) 2021-02-10T04:39:13.484 INFO:tasks.rados.rados.0.smithi085.stdout:2890: read oid 38 snap 288 2021-02-10T04:39:13.484 INFO:tasks.rados.rados.0.smithi085.stdout:2890: expect (ObjNum 886 snap 287 seq_num 886) 2021-02-10T04:39:13.485 INFO:tasks.rados.rados.0.smithi085.stdout:2891: snap_create 2021-02-10T04:39:13.485 INFO:tasks.rados.rados.0.smithi085.stdout:2888: finishing write tid 1 to smithi08575397-20 2021-02-10T04:39:13.488 INFO:tasks.rados.rados.0.smithi085.stdout:2888: finishing write tid 2 to smithi08575397-20 2021-02-10T04:39:13.488 INFO:tasks.rados.rados.0.smithi085.stdout:2888: finishing write tid 3 to smithi08575397-20 2021-02-10T04:39:13.489 INFO:tasks.rados.rados.0.smithi085.stdout:2888: finishing write tid 4 to smithi08575397-20 2021-02-10T04:39:13.489 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 20 v 1152 (ObjNum 1007 snap 329 seq_num 1007) dirty exists 2021-02-10T04:39:13.489 INFO:tasks.rados.rados.0.smithi085.stdout:2888: left oid 20 (ObjNum 1007 snap 329 seq_num 1007) 2021-02-10T04:39:13.492 INFO:tasks.rados.rados.0.smithi085.stdout:2889: expect (ObjNum 892 snap 289 seq_num 892) 2021-02-10T04:39:13.627 INFO:tasks.rados.rados.0.smithi085.stdout:2890: expect (ObjNum 886 snap 287 seq_num 886) 2021-02-10T04:39:13.789 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[29319]: cluster 2021-02-10T04:39:13.789 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[29319]: 2021- 2021-02-10T04:39:13.789 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[29319]: 02-10T04:39:12.463124+0000 mon.a (mon.0) 1650 : cluster [DBG] osdmap e733: 8 total, 8 up, 7 in 2021-02-10T04:39:13.789 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[29319]: 2021-02-10T04:39:13.789 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:39:13.789 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[29319]: 10T04:39:12.886846+0000 osd.4 ( 2021-02-10T04:39:13.789 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[29319]: osd.4) 1173 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:13.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:39:13.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[29319]: -10T04:39:12.886852+0000 osd.4 (osd. 2021-02-10T04:39:13.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[29319]: 4) 1174 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:13.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:39:13.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[29319]: -02-10T04:39:12.886856+0000 2021-02-10T04:39:13.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[29319]: osd.4 (osd.4) 1175 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:13.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:39:13.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[29319]: -02-10T04:39:12.886859+0000 osd.4 (osd 2021-02-10T04:39:13.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[29319]: .4) 1176 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:13.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[29319]: cluster 2021-02-10T04:39:13.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[29319]: 2021-02-10T04:39:13.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[29319]: 2021 2021-02-10T04:39:13.791 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[29319]: - 2021-02-10T04:39:13.791 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[29319]: 02 2021-02-10T04:39:13.791 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[29319]: -10T04:39: 2021-02-10T04:39:13.791 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[29319]: 13.244753+0000 mon.a (mon.0) 1651 : cluster [DBG] osdmap e734: 8 total, 8 up, 7 in 2021-02-10T04:39:13.791 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[35961]: cluster 2021-02-10T04:39:12.463124 2021-02-10T04:39:13.791 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1650 : cluster [DBG] osdmap e733: 8 total, 8 up, 7 in 2021-02-10T04:39:13.791 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[35961]: cluster 2021-02-10T04:39:12 2021-02-10T04:39:13.791 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[35961]: .886846+0000 osd.4 (osd.4) 1173 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:13.791 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:39:13.791 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[35961]: :39:12.886852+0000 osd.4 (osd.4) 2021-02-10T04:39:13.791 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[35961]: 1174 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:13.791 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[35961]: cluster 2021-02-10T04:39:12 2021-02-10T04:39:13.792 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[35961]: .886856+0000 osd.4 (osd.4) 1175 : cluster 2021-02-10T04:39:13.792 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:13.792 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[35961]: cluster 2021-02-10T04:39:12.886859 2021-02-10T04:39:13.792 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1176 : cluster [WRN] 2021-02-10T04:39:13.792 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:13.792 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[35961]: cluster 2021-02-10T04:39:13. 2021-02-10T04:39:13.792 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[35961]: 244753+0000 mon.a (mon.0) 1651 : cluster 2021-02-10T04:39:13.792 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:13 smithi085 conmon[35961]: [DBG] osdmap e734: 8 total, 8 up, 7 in 2021-02-10T04:39:13.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:13 smithi164 conmon[29288]: cluster 2021-02-10T04:39:12.463124+0000 mon.a (mon.0) 1650 : cluster [DBG] 2021-02-10T04:39:13.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:13 smithi164 conmon[29288]: osdmap e733: 8 total, 8 up, 7 in 2021-02-10T04:39:13.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:13 smithi164 conmon[29288]: cluster 2021-02-10T04:39:12.886846+0000 osd.4 (osd.4) 1173 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:13.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:13 smithi164 conmon[29288]: cluster 2021-02-10T04:39 2021-02-10T04:39:13.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:13 smithi164 conmon[29288]: :12.886852+0000 osd.4 (osd.4) 1174 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:13.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:13 smithi164 conmon[29288]: cluster 2021-02-10T04:39:13.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:13 smithi164 conmon[29288]: 2021-02-10T04:39:12.886856+0000 osd.4 (osd.4) 1175 : cluster [WRN] 2021-02-10T04:39:13.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:13 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:13 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:39:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:13 smithi164 conmon[29288]: T04:39:12.886859+0000 osd.4 (osd.4) 1176 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:13 smithi164 conmon[29288]: cluster 2021-02-10T04:39:13.244753+0000 mon.a (mon.0 2021-02-10T04:39:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:13 smithi164 conmon[29288]: ) 1651 : cluster [DBG] osdmap e734: 8 total, 8 up, 7 in 2021-02-10T04:39:14.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:13 smithi164 conmon[33677]: debug 2021-02-10T04:39:13.898+0000 7fbec19a1700 -1 osd.4 734 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:14.477 INFO:tasks.rados.rados.0.smithi085.stdout:2888: done (7 left) 2021-02-10T04:39:14.478 INFO:tasks.rados.rados.0.smithi085.stdout:2889: done (6 left) 2021-02-10T04:39:14.478 INFO:tasks.rados.rados.0.smithi085.stdout:2890: done (5 left) 2021-02-10T04:39:14.478 INFO:tasks.rados.rados.0.smithi085.stdout:2891: done (4 left) 2021-02-10T04:39:14.478 INFO:tasks.rados.rados.0.smithi085.stdout:2892: read oid 50 snap -1 2021-02-10T04:39:14.478 INFO:tasks.rados.rados.0.smithi085.stdout:2892: expect deleted 2021-02-10T04:39:14.478 INFO:tasks.rados.rados.0.smithi085.stdout:2893: snap_create 2021-02-10T04:39:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:14 smithi164 conmon[29288]: cluster 2021-02-10T04:39:13.898904+0000 2021-02-10T04:39:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:14 smithi164 conmon[29288]: osd.4 (osd.4) 1177 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:14 smithi164 conmon[29288]: cluster 2021-02-10T04:39:13.898913+0000 osd.4 (osd.4) 1178 : cluster 2021-02-10T04:39:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:14 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:14 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:39:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:14 smithi164 conmon[29288]: 39:13.898920+0000 osd.4 (osd.4) 1179 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:14 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:39:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:14 smithi164 conmon[29288]: 39:13.898926+0000 osd.4 (osd.4) 1180 2021-02-10T04:39:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:14 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:14 smithi164 conmon[29288]: cluster 2021-02-10T04:39:14 2021-02-10T04:39:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:14 smithi164 conmon[29288]: .171630+0000 mgr.y (mgr.14140) 740 : cluster [DBG] pgmap v1252: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 479 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.0 MiB/s wr, 22 op/s 2021-02-10T04:39:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:14 smithi164 conmon[29288]: 2021-02-10T04:39:14.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[29319]: cluster 2021-02-10T04:39:14.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[29319]: 2021-02-10T04:39: 2021-02-10T04:39:14.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[29319]: 13.898904+0000 osd.4 (osd.4) 1177 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:14.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[29319]: cluster 2021-02-10T04:39:13. 2021-02-10T04:39:14.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[29319]: 898913+0000 osd.4 (osd.4) 1178 : cluster [WRN] 2021-02-10T04:39:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[29319]: cluster 2021-02-10T04:39:13.898920 2021-02-10T04:39:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1179 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:39:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[29319]: -02-10T04:39:13.898926+0000 2021-02-10T04:39:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[29319]: osd.4 (osd.4) 1180 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[29319]: cluster 2021-02-10T04:39:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[29319]: 2021-02-10T04:39:14.171630 2021-02-10T04:39:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 740 : cluster [DBG] 2021-02-10T04:39:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[29319]: pgmap v1252: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 479 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.0 MiB/s wr, 22 op/s 2021-02-10T04:39:14.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: cluster 2021-02-10T04:39:14.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 2021-02-10T04:39:14.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 2021 2021-02-10T04:39:14.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: - 2021-02-10T04:39:14.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 02 2021-02-10T04:39:14.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: - 2021-02-10T04:39:14.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 10 2021-02-10T04:39:14.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: T 2021-02-10T04:39:14.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 04 2021-02-10T04:39:14.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: : 2021-02-10T04:39:14.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 39 2021-02-10T04:39:14.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: : 2021-02-10T04:39:14.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 13 2021-02-10T04:39:14.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: . 2021-02-10T04:39:14.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 898904 2021-02-10T04:39:14.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: +0000 2021-02-10T04:39:14.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 2021-02-10T04:39:14.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: osd.4 2021-02-10T04:39:14.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: ( 2021-02-10T04:39:14.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: osd 2021-02-10T04:39:14.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: . 2021-02-10T04:39:14.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 4 2021-02-10T04:39:14.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: ) 2021-02-10T04:39:14.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 1177 2021-02-10T04:39:14.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: : 2021-02-10T04:39:14.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: cluster 2021-02-10T04:39:14.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 2021-02-10T04:39:14.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: [WRN] 2021-02-10T04:39:14.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 2021-02-10T04:39:14.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:14.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 2021-02-10T04:39:14.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: cluster 2021-02-10T04:39:14.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 2021-02-10T04:39:14.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 2021 2021-02-10T04:39:14.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: - 2021-02-10T04:39:14.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 02 2021-02-10T04:39:14.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: - 2021-02-10T04:39:14.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 10 2021-02-10T04:39:14.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: T 2021-02-10T04:39:14.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 04 2021-02-10T04:39:14.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: : 2021-02-10T04:39:14.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 39 2021-02-10T04:39:14.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: : 2021-02-10T04:39:14.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 13 2021-02-10T04:39:14.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: . 2021-02-10T04:39:14.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 898913 2021-02-10T04:39:14.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: +0000 2021-02-10T04:39:14.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 2021-02-10T04:39:14.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: osd.4 2021-02-10T04:39:14.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: ( 2021-02-10T04:39:14.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: osd 2021-02-10T04:39:14.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: . 2021-02-10T04:39:14.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 4 2021-02-10T04:39:14.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: ) 2021-02-10T04:39:14.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 1178 2021-02-10T04:39:14.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: : 2021-02-10T04:39:14.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: cluster 2021-02-10T04:39:14.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 2021-02-10T04:39:14.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: [WRN] 2021-02-10T04:39:14.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 2021-02-10T04:39:14.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:14.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 2021-02-10T04:39:14.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: cluster 2021-02-10T04:39:14.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 2021-02-10T04:39:14.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 2021 2021-02-10T04:39:14.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: - 2021-02-10T04:39:14.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 02 2021-02-10T04:39:14.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: - 2021-02-10T04:39:14.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 10 2021-02-10T04:39:14.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: T 2021-02-10T04:39:14.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 04:39:13.898920+0000 osd.4 (osd.4) 1179 : cluster 2021-02-10T04:39:14.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:14.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: cluster 2021-02-10T04:39:13.898926+0000 2021-02-10T04:39:14.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: osd.4 (osd.4) 1180 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:14.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 2021-02-10T04:39:14.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: cluster 2021-02-10T04:39: 2021-02-10T04:39:14.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: 14.171630+0000 mgr.y (mgr.14140) 740 2021-02-10T04:39:14.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:14 smithi085 conmon[35961]: : cluster [DBG] pgmap v1252: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 479 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.0 MiB/s wr, 22 op/s 2021-02-10T04:39:15.249 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:14 smithi164 conmon[33677]: debug 2021-02-10T04:39:14.903+0000 7fbec19a1700 -1 osd.4 735 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:15.483 INFO:tasks.rados.rados.0.smithi085.stdout:2892: done (5 left) 2021-02-10T04:39:15.483 INFO:tasks.rados.rados.0.smithi085.stdout:2893: done (4 left) 2021-02-10T04:39:15.483 INFO:tasks.rados.rados.0.smithi085.stdout:2894: delete oid 26 current snap is 331 2021-02-10T04:39:15.488 INFO:tasks.rados.rados.0.smithi085.stdout:2894: done (4 left) 2021-02-10T04:39:15.488 INFO:tasks.rados.rados.0.smithi085.stdout:2895: read oid 49 snap -1 2021-02-10T04:39:15.488 INFO:tasks.rados.rados.0.smithi085.stdout:2895: expect (ObjNum 940 snap 311 seq_num 940) 2021-02-10T04:39:15.488 INFO:tasks.rados.rados.0.smithi085.stdout:2896: write oid 45 current snap is 331 2021-02-10T04:39:15.488 INFO:tasks.rados.rados.0.smithi085.stdout:2896: seq_num 1009 ranges {558263=484837,1627743=706008} 2021-02-10T04:39:15.494 INFO:tasks.rados.rados.0.smithi085.stdout:2896: writing smithi08575397-45 from 558263 to 1043100 tid 1 2021-02-10T04:39:15.503 INFO:tasks.rados.rados.0.smithi085.stdout:2896: writing smithi08575397-45 from 1627743 to 2333751 tid 2 2021-02-10T04:39:15.503 INFO:tasks.rados.rados.0.smithi085.stdout:2897: read oid 37 snap -1 2021-02-10T04:39:15.503 INFO:tasks.rados.rados.0.smithi085.stdout:2897: expect (ObjNum 940 snap 311 seq_num 940) 2021-02-10T04:39:15.503 INFO:tasks.rados.rados.0.smithi085.stdout:2898: snap_remove snap 317 2021-02-10T04:39:15.643 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:39:15 smithi164 conmon[37263]: debug 2021-02-10T04:39:15.248+0000 7f1c6b1ea700 -1 osd.5 pg_epoch: 735 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x5566536c7e40) 2021-02-10T04:39:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:15 smithi164 conmon[29288]: cluster 2021-02-10T04:39:14.472862+0000 2021-02-10T04:39:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:15 smithi164 conmon[29288]: mon.a (mon.0) 1652 : cluster [DBG] osdmap e735: 8 total, 8 up, 7 in 2021-02-10T04:39:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:15 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:39:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:15 smithi164 conmon[29288]: 39:14.904369+0000 osd.4 (osd.4) 1181 : cluster [WRN] 2021-02-10T04:39:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:15 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:15 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:39:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:15 smithi164 conmon[29288]: T04:39:14.904375+0000 osd.4 (osd 2021-02-10T04:39:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:15 smithi164 conmon[29288]: .4) 1182 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:15 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:39:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:15 smithi164 conmon[29288]: -10T04:39:14.904379+0000 osd.4 ( 2021-02-10T04:39:15.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:15 smithi164 conmon[29288]: osd.4) 1183 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:15.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:15 smithi164 conmon[29288]: cluster 2021-02-10T04:39:15.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:15 smithi164 conmon[29288]: 2021-02-10T04:39:14.904382 2021-02-10T04:39:15.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:15 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1184 : cluster [WRN] 2021-02-10T04:39:15.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:15 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:15.737 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:15 smithi085 conmon[29319]: cluster 2021-02-10T04:39:14.472862+0000 mon.a ( 2021-02-10T04:39:15.737 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:15 smithi085 conmon[29319]: mon.0) 1652 : cluster [DBG] osdmap e735: 8 total, 8 up, 7 in 2021-02-10T04:39:15.737 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:15 smithi085 conmon[29319]: cluster 2021-02-10T04:39:14.904369+0000 osd.4 (osd.4) 1181 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:15.737 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:15 smithi085 conmon[29319]: cluster 2021-02-10T04:39:14.904375+0000 osd.4 (osd.4) 1182 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:15.738 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:15 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:39:15.738 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:15 smithi085 conmon[29319]: 10T04:39:14.904379+0000 osd.4 (osd.4) 1183 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:15.738 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:15 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:39:15.738 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:15 smithi085 conmon[29319]: 02-10T04:39:14.904382+0000 osd.4 2021-02-10T04:39:15.738 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:15 smithi085 conmon[29319]: (osd.4) 1184 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:15.738 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:15 smithi085 conmon[35961]: cluster 2021-02-10T04:39:14. 2021-02-10T04:39:15.738 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:15 smithi085 conmon[35961]: 472862+0000 mon.a (mon.0) 1652 : cluster [DBG] osdmap e735: 8 total, 8 up, 7 in 2021-02-10T04:39:15.738 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:15 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:39:15.739 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:15 smithi085 conmon[35961]: -10T04:39:14.904369+0000 osd.4 (osd.4) 1181 : 2021-02-10T04:39:15.739 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:15 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:15.739 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:15 smithi085 conmon[35961]: cluster 2021-02-10T04:39: 2021-02-10T04:39:15.739 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:15 smithi085 conmon[35961]: 14.904375+0000 osd.4 (osd.4) 1182 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:15.739 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:15 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:39:15.739 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:15 smithi085 conmon[35961]: 04:39 2021-02-10T04:39:15.739 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:15 smithi085 conmon[35961]: :14.904379+0000 2021-02-10T04:39:15.739 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:15 smithi085 conmon[35961]: osd.4 (osd.4) 1183 : cluster 2021-02-10T04:39:15.739 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:15 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:15.739 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:15 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:39:15.739 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:15 smithi085 conmon[35961]: -02-10T04:39:14.904382+0000 2021-02-10T04:39:15.739 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:15 smithi085 conmon[35961]: osd.4 (osd.4) 1184 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:15.740 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:15 smithi085 conmon[35961]: 2021-02-10T04:39:15.837 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:39:15.892 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:39:16.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:15 smithi164 conmon[33677]: debug 2021-02-10T04:39:15.874+0000 7fbec19a1700 -1 osd.4 736 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:16.154 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:39:16.261 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:39:16.306 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[49106]: debug 2021-02-10T04:39:16.102+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:16.325 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:39:16.502 INFO:tasks.rados.rados.0.smithi085.stdout:2898: done (7 left) 2021-02-10T04:39:16.503 INFO:tasks.rados.rados.0.smithi085.stdout:2899: snap_create 2021-02-10T04:39:16.503 INFO:tasks.rados.rados.0.smithi085.stdout:2895: expect (ObjNum 940 snap 311 seq_num 940) 2021-02-10T04:39:16.643 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:39:16 smithi164 conmon[44544]: debug 2021-02-10T04:39:16.418+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:16 smithi164 conmon[29288]: cluster 2021-02-10T04:39:15.482423+0000 mon.a (mon. 2021-02-10T04:39:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:16 smithi164 conmon[29288]: 0) 1653 : cluster [DBG] osdmap e736: 8 total, 8 up, 7 in 2021-02-10T04:39:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:16 smithi164 conmon[29288]: cluster 2021-02-10T04:39:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:16 smithi164 conmon[29288]: 2021-02-10T04:39 2021-02-10T04:39:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:16 smithi164 conmon[29288]: :15.874868+0000 osd.4 (osd.4) 1185 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:16 smithi164 conmon[29288]: cluster 2021-02-10T04:39:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:16 smithi164 conmon[29288]: 2021-02-10T04:39:15.874879+0000 2021-02-10T04:39:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:16 smithi164 conmon[29288]: osd.4 (osd.4) 1186 : cluster [WRN] 2021-02-10T04:39:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:16 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:16 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:39:16.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:16 smithi164 conmon[29288]: 39:15.874883+0000 osd.4 (osd.4) 2021-02-10T04:39:16.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:16 smithi164 conmon[29288]: 1187 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:16.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:16 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:39:16.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:16 smithi164 conmon[29288]: 02-10T04:39:15.874887+0000 osd.4 2021-02-10T04:39:16.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:16 smithi164 conmon[29288]: (osd.4) 1188 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:16.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:16 smithi164 conmon[29288]: cluster 2021-02-10T04:39:16.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:16 smithi164 conmon[29288]: 2021-02-10T04:39:16.172114+0000 2021-02-10T04:39:16.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:16 smithi164 conmon[29288]: mgr.y (mgr.14140) 741 : cluster [DBG] 2021-02-10T04:39:16.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:16 smithi164 conmon[29288]: pgmap v1255: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 482 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 16 MiB/s rd, 2.5 MiB/s wr, 27 op/s 2021-02-10T04:39:16.679 INFO:tasks.rados.rados.0.smithi085.stdout:2896: finishing write tid 1 to smithi08575397-45 2021-02-10T04:39:16.680 INFO:tasks.rados.rados.0.smithi085.stdout:2897: expect (ObjNum 940 snap 311 seq_num 940) 2021-02-10T04:39:16.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[35961]: cluster 2021-02-10T04:39:15.482423 2021-02-10T04:39:16.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1653 : cluster [DBG] osdmap e736: 8 total, 8 up, 7 in 2021-02-10T04:39:16.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[35961]: cluster 2021-02-10T04:39:16.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[35961]: 2021-02-10T04:39:15.874868+0000 osd.4 (osd.4) 1185 2021-02-10T04:39:16.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:16.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:39:16.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[35961]: 10T04:39:15.874879+0000 osd.4 (osd 2021-02-10T04:39:16.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[35961]: .4) 1186 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:16.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[35961]: cluster 2021-02-10T04:39:16.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[35961]: 2021 2021-02-10T04:39:16.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[35961]: -02-10T04:39:15 2021-02-10T04:39:16.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[35961]: .874883+0000 osd.4 (osd.4) 1187 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:16.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[35961]: cluster 2021-02-10T04:39:16.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:39:16.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[35961]: 10T04:39:15.874887+0000 osd.4 (osd.4) 1188 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:16.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[35961]: 2021-02-10T04:39:16.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:39:16.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[35961]: 02-10T04:39:16.172114+0000 mgr.y (mgr.14140) 741 : cluster [DBG] pgmap v1255: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 482 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 16 MiB/s rd, 2.5 MiB/s wr, 27 op/s 2021-02-10T04:39:16.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: cluster 2021-02-10T04:39:16.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 2021-02-10T04:39:16.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 2021 2021-02-10T04:39:16.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: - 2021-02-10T04:39:16.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 02 2021-02-10T04:39:16.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: - 2021-02-10T04:39:16.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 10 2021-02-10T04:39:16.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: T 2021-02-10T04:39:16.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 04 2021-02-10T04:39:16.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: : 2021-02-10T04:39:16.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 39 2021-02-10T04:39:16.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: :15.482423+0000 2021-02-10T04:39:16.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: mon.a (mon.0) 1653 : cluster [DBG] osdmap e736: 8 total, 8 up, 7 in 2021-02-10T04:39:16.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: cluster 2021-02-10T04:39:16.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 2021-02-10T04:39:16.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 2021 2021-02-10T04:39:16.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: - 2021-02-10T04:39:16.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 02 2021-02-10T04:39:16.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: - 2021-02-10T04:39:16.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 10 2021-02-10T04:39:16.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: T 2021-02-10T04:39:16.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 04 2021-02-10T04:39:16.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: : 2021-02-10T04:39:16.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 39 2021-02-10T04:39:16.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: : 2021-02-10T04:39:16.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 15 2021-02-10T04:39:16.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: . 2021-02-10T04:39:16.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 874868 2021-02-10T04:39:16.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: +0000 2021-02-10T04:39:16.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 2021-02-10T04:39:16.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: osd.4 2021-02-10T04:39:16.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: ( 2021-02-10T04:39:16.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: osd 2021-02-10T04:39:16.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: . 2021-02-10T04:39:16.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 4 2021-02-10T04:39:16.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: ) 2021-02-10T04:39:16.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 1185 2021-02-10T04:39:16.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: : 2021-02-10T04:39:16.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: cluster 2021-02-10T04:39:16.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 2021-02-10T04:39:16.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: [WRN] 2021-02-10T04:39:16.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 2021-02-10T04:39:16.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:16.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 2021-02-10T04:39:16.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: cluster 2021-02-10T04:39:16.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 2021-02-10T04:39:16.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 2021 2021-02-10T04:39:16.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: - 2021-02-10T04:39:16.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 02 2021-02-10T04:39:16.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: - 2021-02-10T04:39:16.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 10 2021-02-10T04:39:16.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: T 2021-02-10T04:39:16.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 04 2021-02-10T04:39:16.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: : 2021-02-10T04:39:16.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 39 2021-02-10T04:39:16.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: : 2021-02-10T04:39:16.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 15 2021-02-10T04:39:16.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: . 2021-02-10T04:39:16.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 874879 2021-02-10T04:39:16.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: +0000 2021-02-10T04:39:16.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 2021-02-10T04:39:16.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: osd.4 2021-02-10T04:39:16.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: ( 2021-02-10T04:39:16.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: osd 2021-02-10T04:39:16.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: . 2021-02-10T04:39:16.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 4 2021-02-10T04:39:16.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: ) 2021-02-10T04:39:16.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 1186 2021-02-10T04:39:16.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: : 2021-02-10T04:39:16.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: cluster 2021-02-10T04:39:16.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 2021-02-10T04:39:16.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: [WRN] 2021-02-10T04:39:16.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 2021-02-10T04:39:16.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:16.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: 2021-02-10T04:39:16.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: cluster 2021-02-10T04:39 2021-02-10T04:39:16.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: :15.874883+0000 osd.4 (osd.4) 1187 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:16.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: cluster 2021-02-10T04:39:15.874887 2021-02-10T04:39:16.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1188 : cluster [WRN] 2021-02-10T04:39:16.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:16.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: cluster 2021-02-10T04:39:16.172114 2021-02-10T04:39:16.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 741 : cluster [DBG] 2021-02-10T04:39:16.704 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[29319]: pgmap v1255: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 482 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 16 MiB/s rd, 2.5 MiB/s wr, 27 op/s 2021-02-10T04:39:16.725 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:39:16.727 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:39:16.727 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [], 2021-02-10T04:39:16.727 INFO:teuthology.orchestra.run.smithi085.stdout: "num_ops": 0 2021-02-10T04:39:16.728 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:39:16.832 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:39:16.858 INFO:tasks.rados.rados.0.smithi085.stdout:2896: finishing write tid 2 to smithi08575397-45 2021-02-10T04:39:16.858 INFO:tasks.rados.rados.0.smithi085.stdout:2896: finishing write tid 3 to smithi08575397-45 2021-02-10T04:39:16.859 INFO:tasks.rados.rados.0.smithi085.stdout:2896: finishing write tid 4 to smithi08575397-45 2021-02-10T04:39:16.859 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 45 v 1442 (ObjNum 1009 snap 331 seq_num 1009) dirty exists 2021-02-10T04:39:16.859 INFO:tasks.rados.rados.0.smithi085.stdout:2896: left oid 45 (ObjNum 1009 snap 331 seq_num 1009) 2021-02-10T04:39:16.871 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:39:17.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:16 smithi164 conmon[33677]: debug 2021-02-10T04:39:17.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:16 smithi164 conmon[33677]: 2021-02-10T04:39:16.859+0000 7fbec19a1700 -1 osd.4 737 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:17.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:16 smithi164 conmon[33677]: 2021-02-10T04:39:17.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:39:16 smithi085 conmon[49106]: debug 2021-02-10T04:39:16.963+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:17.201 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:39:17.500 INFO:tasks.rados.rados.0.smithi085.stdout:2895: done (7 left) 2021-02-10T04:39:17.500 INFO:tasks.rados.rados.0.smithi085.stdout:2896: done (6 left) 2021-02-10T04:39:17.501 INFO:tasks.rados.rados.0.smithi085.stdout:2897: done (5 left) 2021-02-10T04:39:17.501 INFO:tasks.rados.rados.0.smithi085.stdout:2899: done (4 left) 2021-02-10T04:39:17.501 INFO:tasks.rados.rados.0.smithi085.stdout:2900: rollback oid 5 current snap is 332 2021-02-10T04:39:17.501 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 5 to 298 2021-02-10T04:39:17.501 INFO:tasks.rados.rados.0.smithi085.stdout:2901: snap_create 2021-02-10T04:39:17.508 INFO:tasks.rados.rados.0.smithi085.stdout:2900: finishing rollback tid 0 to smithi08575397-5 2021-02-10T04:39:17.508 INFO:tasks.rados.rados.0.smithi085.stdout:2900: finishing rollback tid 1 to smithi08575397-5 2021-02-10T04:39:17.509 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 810 (ObjNum 82754208 snap 22030 seq_num 83204912) dirty dne 2021-02-10T04:39:17.845 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:17 smithi164 conmon[33677]: debug 2021-02-10T04:39:17.843+0000 7fbec19a1700 -1 osd.4 738 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:17.846 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:17 smithi164 conmon[29288]: cluster 2021-02-10T04:39:16.493990+0000 mon.a 2021-02-10T04:39:17.846 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:17 smithi164 conmon[29288]: (mon.0) 1654 : cluster [DBG] osdmap e737: 8 total, 8 up, 7 in 2021-02-10T04:39:17.846 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:17 smithi164 conmon[29288]: cluster 2021-02-10T04:39:16.859956+0000 osd.4 (osd.4) 1189 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:17.846 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:17 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:39:17.846 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:17 smithi164 conmon[29288]: 04:39:16.859984+0000 osd.4 (osd.4) 1190 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:17.846 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:17 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:39:17.846 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:17 smithi164 conmon[29288]: 10T04:39:16.859997+0000 osd.4 (osd.4) 1191 2021-02-10T04:39:17.846 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:17 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:17.846 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:17 smithi164 conmon[29288]: cluster 2021-02-10T04:39:16. 2021-02-10T04:39:17.846 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:17 smithi164 conmon[29288]: 860004+0000 osd.4 (osd.4) 1192 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:17.846 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:17 smithi164 conmon[29288]: 2021-02-10T04:39:17.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:17 smithi085 conmon[29319]: cluster 2021-02-10T04:39:16.493990+0000 mon.a (mon 2021-02-10T04:39:17.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:17 smithi085 conmon[29319]: .0) 1654 : cluster [DBG] osdmap e737: 8 total, 8 up, 7 in 2021-02-10T04:39:17.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:17 smithi085 conmon[29319]: cluster 2021-02-10T04:39:16.859956+0000 osd.4 (osd.4) 1189 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:17.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:17 smithi085 conmon[29319]: cluster 2021-02-10T04:39:16.859984+0000 osd.4 (osd.4) 1190 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:17.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:17 smithi085 conmon[29319]: cluster 2021-02-10T04:39:16.859997 2021-02-10T04:39:17.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:17 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1191 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:17.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:17 smithi085 conmon[29319]: cluster 2021-02-10T04:39:16.860004 2021-02-10T04:39:17.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:17 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1192 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:17.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:17 smithi085 conmon[35961]: cluster 2021-02-10T04:39:16. 2021-02-10T04:39:17.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:17 smithi085 conmon[35961]: 493990+0000 mon.a (mon.0) 1654 : cluster [DBG] osdmap e737: 8 total, 8 up, 7 in 2021-02-10T04:39:17.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:17 smithi085 conmon[35961]: cluster 2021-02-10T04:39: 2021-02-10T04:39:17.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:17 smithi085 conmon[35961]: 16.859956+0000 osd.4 (osd.4) 1189 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:17.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:17 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:39:17.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:17 smithi085 conmon[35961]: -02-10T04:39:16.859984+0000 2021-02-10T04:39:17.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:17 smithi085 conmon[35961]: osd.4 (osd.4) 1190 : cluster 2021-02-10T04:39:17.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:17 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:17.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:17 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:39:17.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:17 smithi085 conmon[35961]: 02-10T04:39:16.859997+0000 2021-02-10T04:39:17.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:17 smithi085 conmon[35961]: osd.4 (osd.4) 1191 : cluster 2021-02-10T04:39:17.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:17 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:17.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:17 smithi085 conmon[35961]: cluster 2021-02-10T04:39:17.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:17 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:39:17.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:17 smithi085 conmon[35961]: 39:16.860004+0000 osd.4 2021-02-10T04:39:17.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:17 smithi085 conmon[35961]: (osd.4) 1192 : cluster 2021-02-10T04:39:17.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:17 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:18.045 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set noscrub 2021-02-10T04:39:18.070 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:39:18.249 INFO:tasks.rados.rados.0.smithi085.stdout:2900: done (5 left) 2021-02-10T04:39:18.249 INFO:tasks.rados.rados.0.smithi085.stdout:2901: done (4 left) 2021-02-10T04:39:18.250 INFO:tasks.rados.rados.0.smithi085.stdout:2902: delete oid 9 current snap is 333 2021-02-10T04:39:18.255 INFO:tasks.rados.rados.0.smithi085.stdout:2902: done (4 left) 2021-02-10T04:39:18.256 INFO:tasks.rados.rados.0.smithi085.stdout:2903: read oid 16 snap 308 2021-02-10T04:39:18.256 INFO:tasks.rados.rados.0.smithi085.stdout:2903: expect deleted 2021-02-10T04:39:18.256 INFO:tasks.rados.rados.0.smithi085.stdout:2904: snap_remove snap 327 2021-02-10T04:39:18.450 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:39:18.470 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:39:18.644 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:39:18 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:39:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:18 smithi164 conmon[29288]: cluster 2021-02-10T04:39:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:18 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:39:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:18 smithi164 conmon[29288]: :39:17.499618+0000 mon.a (mon.0) 1655 : cluster [DBG] osdmap e738: 8 total, 8 up, 7 in 2021-02-10T04:39:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:18 smithi164 conmon[29288]: cluster 2021-02-10T04:39:17.844371+0000 osd.4 ( 2021-02-10T04:39:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:18 smithi164 conmon[29288]: osd.4) 1193 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:18 smithi164 conmon[29288]: cluster 2021-02-10T04:39:17.844377+0000 osd.4 (osd.4) 1194 : cluster 2021-02-10T04:39:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:18 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:18 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:39:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:18 smithi164 conmon[29288]: -10T04:39:17.844380+0000 osd.4 (osd.4) 1195 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:18 smithi164 conmon[29288]: cluster 2021-02-10T04:39:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:18 smithi164 conmon[29288]: 2021-02-10T04:39:17.844384+0000 2021-02-10T04:39:18.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:18 smithi164 conmon[29288]: osd.4 (osd.4) 1196 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:18.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:18 smithi164 conmon[29288]: cluster 2021-02-10T04:39:18.172859+0000 2021-02-10T04:39:18.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:18 smithi164 conmon[29288]: mgr.y (mgr.14140) 742 : cluster [DBG] pgmap v1258: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 484 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 5.3 MiB/s rd, 1.7 MiB/s wr, 11 op/s 2021-02-10T04:39:18.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:18 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:39:18.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:18 smithi164 conmon[29288]: 10T04:39:18.236429+0000 mon.a (mon.0) 1656 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 582 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:39:18.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:18 smithi164 conmon[29288]: cluster 2021-02-10T04:39:18.247892+0000 2021-02-10T04:39:18.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:18 smithi164 conmon[29288]: mon.a (mon.0) 1657 : cluster [DBG] 2021-02-10T04:39:18.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:18 smithi164 conmon[29288]: osdmap e739: 8 total, 8 up, 7 in 2021-02-10T04:39:18.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[29319]: cluster 2021-02-10T04:39:18.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[29319]: 2021-02-10T04:39:18.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[29319]: 2021-02-10T04:39:17 2021-02-10T04:39:18.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[29319]: .499618+0000 mon.a (mon.0) 1655 : cluster [DBG] osdmap e738: 8 total, 8 up, 7 in 2021-02-10T04:39:18.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:39:18.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[29319]: -10T04:39:17.844371+0000 osd.4 2021-02-10T04:39:18.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[29319]: (osd.4) 1193 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:18.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:39:18.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[29319]: 02-10T04:39:17.844377+0000 osd.4 2021-02-10T04:39:18.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[29319]: (osd.4) 1194 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:18.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:39:18.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[29319]: -10T04:39:17.844380+0000 osd.4 2021-02-10T04:39:18.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[29319]: (osd.4) 1195 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:18.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:39:18.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[29319]: 10T04:39:17.844384+0000 osd.4 (osd 2021-02-10T04:39:18.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[29319]: .4) 1196 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:18.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:39:18.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[29319]: 10T04:39:18.172859+0000 mgr.y (mgr.14140) 742 : cluster [DBG] pgmap v1258: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 484 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 5.3 MiB/s rd, 1.7 MiB/s wr, 11 op/s 2021-02-10T04:39:18.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[29319]: cluster 2021-02-10T04:39:18. 2021-02-10T04:39:18.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[29319]: 236429+0000 mon.a (mon.0) 1656 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 582 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:39:18.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[29319]: cluster 2021-02-10T04:39:18 2021-02-10T04:39:18.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[29319]: .247892+0000 mon.a (mon.0) 1657 : cluster 2021-02-10T04:39:18.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[29319]: [DBG] osdmap e739: 8 total, 8 up, 7 in 2021-02-10T04:39:18.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[35961]: cluster 2021-02-10T04:39:17.499618 2021-02-10T04:39:18.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1655 : cluster [DBG] osdmap e738: 8 total, 8 up, 7 in 2021-02-10T04:39:18.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[35961]: cluster 2021-02-10T04:39: 2021-02-10T04:39:18.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[35961]: 17.844371+0000 osd.4 (osd.4) 1193 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:18.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[35961]: cluster 2021-02-10T04:39:18.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[35961]: 2021-02-10T04:39:17.844377+0000 2021-02-10T04:39:18.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[35961]: osd.4 (osd.4) 1194 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:18.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:39:18.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[35961]: -10T04:39:17.844380+0000 osd.4 (osd 2021-02-10T04:39:18.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[35961]: .4) 1195 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:18.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:39:18.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[35961]: :39:17.844384+0000 osd.4 (osd.4) 1196 2021-02-10T04:39:18.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:18.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[35961]: cluster 2021-02-10T04:39 2021-02-10T04:39:18.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[35961]: :18.172859+0000 mgr.y (mgr.14140) 742 : 2021-02-10T04:39:18.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[35961]: cluster [DBG] pgmap v1258: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 484 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 5.3 MiB/s rd, 1.7 MiB/s wr, 11 op/s 2021-02-10T04:39:18.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:39:18.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[35961]: -02-10T04:39:18.236429+0000 2021-02-10T04:39:18.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[35961]: mon.a (mon.0) 1656 : cluster 2021-02-10T04:39:18.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[35961]: [WRN] Health check update: 4 slow ops, oldest one blocked for 582 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:39:18.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:18 smithi085 conmon[35961]: cluster 2021-02-10T04:39:18.247892+0000 mon.a (mon.0) 1657 : cluster [DBG] osdmap e739: 8 total, 8 up, 7 in 2021-02-10T04:39:19.143 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:18 smithi164 conmon[33677]: debug 2021-02-10T04:39:18.816+0000 7fbec19a1700 -1 osd.4 739 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:19.270 INFO:tasks.rados.rados.0.smithi085.stdout:2904: done (5 left) 2021-02-10T04:39:19.270 INFO:tasks.rados.rados.0.smithi085.stdout:2905: write oid 26 current snap is 333 2021-02-10T04:39:19.270 INFO:tasks.rados.rados.0.smithi085.stdout:2905: seq_num 1011 ranges {591877=635335,1628354=709677} 2021-02-10T04:39:19.271 INFO:tasks.rados.rados.0.smithi085.stdout:2905: writing smithi08575397-26 from 591877 to 1227212 tid 1 2021-02-10T04:39:19.277 INFO:tasks.rados.rados.0.smithi085.stdout:2905: writing smithi08575397-26 from 1628354 to 2338031 tid 2 2021-02-10T04:39:19.278 INFO:tasks.rados.rados.0.smithi085.stdout:2906: copy_from oid 46 from oid 32 current snap is 333 2021-02-10T04:39:19.278 INFO:tasks.rados.rados.0.smithi085.stdout:2905: finishing write tid 1 to smithi08575397-26 2021-02-10T04:39:19.278 INFO:tasks.rados.rados.0.smithi085.stdout:2903: done (6 left) 2021-02-10T04:39:19.278 INFO:tasks.rados.rados.0.smithi085.stdout:2907: write (excl) oid 6 current snap is 333 2021-02-10T04:39:19.278 INFO:tasks.rados.rados.0.smithi085.stdout:2907: seq_num 1012 ranges {608684=426937,1803189=569688,2924304=84380} 2021-02-10T04:39:19.284 INFO:tasks.rados.rados.0.smithi085.stdout:2907: writing smithi08575397-6 from 608684 to 1035621 tid 1 2021-02-10T04:39:19.296 INFO:tasks.rados.rados.0.smithi085.stdout:2907: writing smithi08575397-6 from 1803189 to 2372877 tid 2 2021-02-10T04:39:19.300 INFO:tasks.rados.rados.0.smithi085.stdout:2907: writing smithi08575397-6 from 2924304 to 3008684 tid 3 2021-02-10T04:39:19.300 INFO:tasks.rados.rados.0.smithi085.stdout:2908: delete oid 38 current snap is 333 2021-02-10T04:39:19.300 INFO:tasks.rados.rados.0.smithi085.stdout:2905: finishing write tid 2 to smithi08575397-26 2021-02-10T04:39:19.301 INFO:tasks.rados.rados.0.smithi085.stdout:2905: finishing write tid 3 to smithi08575397-26 2021-02-10T04:39:19.301 INFO:tasks.rados.rados.0.smithi085.stdout:2905: finishing write tid 4 to smithi08575397-26 2021-02-10T04:39:19.301 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 1346 (ObjNum 1011 snap 333 seq_num 1011) dirty exists 2021-02-10T04:39:19.301 INFO:tasks.rados.rados.0.smithi085.stdout:2905: left oid 26 (ObjNum 1011 snap 333 seq_num 1011) 2021-02-10T04:39:19.301 INFO:tasks.rados.rados.0.smithi085.stdout:2906: finishing copy_from to smithi08575397-46 2021-02-10T04:39:19.301 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 1357 (ObjNum 1005 snap 329 seq_num 1005) dirty exists 2021-02-10T04:39:19.301 INFO:tasks.rados.rados.0.smithi085.stdout:2906: finishing copy_from racing read to smithi08575397-46 2021-02-10T04:39:19.301 INFO:tasks.rados.rados.0.smithi085.stdout:2907: finishing write tid 1 to smithi08575397-6 2021-02-10T04:39:19.302 INFO:tasks.rados.rados.0.smithi085.stdout:2907: finishing write tid 2 to smithi08575397-6 2021-02-10T04:39:19.303 INFO:tasks.rados.rados.0.smithi085.stdout:2907: finishing write tid 3 to smithi08575397-6 2021-02-10T04:39:19.305 INFO:tasks.rados.rados.0.smithi085.stdout:2907: finishing write tid 4 to smithi08575397-6 2021-02-10T04:39:19.306 INFO:tasks.rados.rados.0.smithi085.stdout:2907: finishing write tid 5 to smithi08575397-6 2021-02-10T04:39:19.306 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 1366 (ObjNum 1012 snap 333 seq_num 1012) dirty exists 2021-02-10T04:39:19.306 INFO:tasks.rados.rados.0.smithi085.stdout:2907: left oid 6 (ObjNum 1012 snap 333 seq_num 1012) 2021-02-10T04:39:19.307 INFO:tasks.rados.rados.0.smithi085.stdout:2905: done (7 left) 2021-02-10T04:39:19.307 INFO:tasks.rados.rados.0.smithi085.stdout:2906: done (6 left) 2021-02-10T04:39:19.307 INFO:tasks.rados.rados.0.smithi085.stdout:2907: done (5 left) 2021-02-10T04:39:19.307 INFO:tasks.rados.rados.0.smithi085.stdout:2908: done (4 left) 2021-02-10T04:39:19.307 INFO:tasks.rados.rados.0.smithi085.stdout:2909: snap_create 2021-02-10T04:39:19.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:19 smithi164 conmon[29288]: audit 2021-02-10T04:39:19.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:19 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:39:19.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:19 smithi164 conmon[29288]: :39:18.325120+0000 mgr.y (mgr.14140) 743 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:39:19.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:19 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:39:19.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:19 smithi164 conmon[29288]: -10T04:39:18.816932+0000 osd.4 (osd.4 2021-02-10T04:39:19.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:19 smithi164 conmon[29288]: ) 1197 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:19.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:19 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:39:19.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:19 smithi164 conmon[29288]: -10T04:39:18.816940+0000 osd.4 ( 2021-02-10T04:39:19.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:19 smithi164 conmon[29288]: osd.4) 1198 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:19.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:19 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:39:19.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:19 smithi164 conmon[29288]: 02-10T04:39:18.816947+0000 osd.4 2021-02-10T04:39:19.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:19 smithi164 conmon[29288]: (osd.4) 1199 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:19.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:19 smithi164 conmon[29288]: cluster 2021-02-10T04:39:19.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:19 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:39:19.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:19 smithi164 conmon[29288]: 39:18.816953+0000 osd.4 (osd.4) 1200 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:19.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:19 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:39:19.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:19 smithi164 conmon[29288]: 39:19.262466+0000 mon.a (mon.0) 1658 2021-02-10T04:39:19.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:19 smithi164 conmon[29288]: : cluster [DBG] osdmap e740: 8 total, 8 up, 7 in 2021-02-10T04:39:19.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[35961]: audit 2021-02-10T04:39:18.325120+0000 mgr.y 2021-02-10T04:39:19.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[35961]: (mgr.14140) 743 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:39:19.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[35961]: cluster 2021-02-10T04:39:19.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[35961]: 2021-02-10T04:39:18. 2021-02-10T04:39:19.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[35961]: 816932+0000 osd.4 (osd.4 2021-02-10T04:39:19.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[35961]: ) 1197 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:19.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:39:19.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[35961]: 02-10T04:39:18.816940+0000 2021-02-10T04:39:19.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[35961]: osd.4 (osd.4) 1198 : cluster 2021-02-10T04:39:19.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:19.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:39:19.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[35961]: -02-10T04:39:18 2021-02-10T04:39:19.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[35961]: .816947+0000 osd.4 (osd. 2021-02-10T04:39:19.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[35961]: 4) 1199 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:19.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[35961]: 2021-02-10T04:39:19.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:39:19.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[35961]: -02-10T 2021-02-10T04:39:19.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[35961]: 04:39:18.816953 2021-02-10T04:39:19.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2021-02-10T04:39:19.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[35961]: 1200 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:19.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:39:19.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[35961]: -02-10T04:39:19.262466+0000 2021-02-10T04:39:19.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[35961]: mon.a (mon.0) 1658 : cluster [DBG] osdmap e740: 8 total, 8 up, 7 in 2021-02-10T04:39:19.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[29319]: audit 2021-02-10T04:39:18.325120+0000 mgr.y ( 2021-02-10T04:39:19.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[29319]: mgr.14140) 743 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:39:19.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[29319]: cluster 2021-02-10T04:39:18.816932+0000 osd.4 (osd.4) 1197 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:19.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:39:19.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[29319]: 04:39:18.816940+0000 osd.4 (osd.4) 1198 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:19.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[29319]: cluster 2021-02-10T04:39:19.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[29319]: 2021-02-10T04:39:18.816947+0000 osd.4 (osd.4) 2021-02-10T04:39:19.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[29319]: 1199 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:19.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[29319]: cluster 2021-02-10T04:39: 2021-02-10T04:39:19.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[29319]: 18.816953+0000 osd.4 (osd.4) 1200 2021-02-10T04:39:19.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:19.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[29319]: cluster 2021-02-10T04:39 2021-02-10T04:39:19.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[29319]: :19.262466+0000 mon.a (mon.0) 1658 : cluster 2021-02-10T04:39:19.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:19 smithi085 conmon[29319]: [DBG] osdmap e740: 8 total, 8 up, 7 in 2021-02-10T04:39:20.143 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:19 smithi164 conmon[33677]: debug 2021-02-10T04:39:20.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:19 smithi164 conmon[33677]: 2021-02-10T04:39:19.850+0000 7fbec19a1700 -1 osd.4 740 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:20.258 INFO:tasks.rados.rados.0.smithi085.stdout:2909: done (4 left) 2021-02-10T04:39:20.258 INFO:tasks.rados.rados.0.smithi085.stdout:2910: delete oid 6 current snap is 334 2021-02-10T04:39:20.266 INFO:tasks.rados.rados.0.smithi085.stdout:2910: done (4 left) 2021-02-10T04:39:20.267 INFO:tasks.rados.rados.0.smithi085.stdout:2911: read oid 22 snap -1 2021-02-10T04:39:20.267 INFO:tasks.rados.rados.0.smithi085.stdout:2911: expect (ObjNum 966 snap 315 seq_num 966) 2021-02-10T04:39:20.267 INFO:tasks.rados.rados.0.smithi085.stdout:2912: snap_remove snap 301 2021-02-10T04:39:20.413 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:39:20.448 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:39:20.669 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[29319]: cluster 2021-02-10T04:39:20.669 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[29319]: 2021-02-10T04:39:20.670 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[29319]: 2021 2021-02-10T04:39:20.670 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[29319]: - 2021-02-10T04:39:20.670 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[29319]: 02 2021-02-10T04:39:20.670 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[29319]: - 2021-02-10T04:39:20.670 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[29319]: 10 2021-02-10T04:39:20.670 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[29319]: T 2021-02-10T04:39:20.670 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[29319]: 04 2021-02-10T04:39:20.670 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[29319]: : 2021-02-10T04:39:20.670 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[29319]: 39:19.851150+0000 2021-02-10T04:39:20.670 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[29319]: osd.4 (osd.4) 1201 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:20.670 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:39:20.670 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[29319]: -02-10T04:39:19.851156+0000 osd.4 (osd.4 2021-02-10T04:39:20.670 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[29319]: ) 1202 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:20.671 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[29319]: cluster 2021-02-10T04:39:20.671 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[29319]: 2021-02-10T04:39:19.851160 2021-02-10T04:39:20.671 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1203 2021-02-10T04:39:20.671 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:20.671 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:39:20.671 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[29319]: 02-10T04:39:19.851163+0000 osd.4 2021-02-10T04:39:20.671 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[29319]: (osd.4) 1204 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:20.671 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:39:20.671 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[29319]: 02-10T04:39:20.173399+0000 mgr.y 2021-02-10T04:39:20.671 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[29319]: (mgr.14140) 744 : cluster [DBG] pgmap v1261: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 489 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.1 MiB/s rd, 972 KiB/s wr, 9 op/s 2021-02-10T04:39:20.671 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[29319]: cluster 2021-02-10T04:39:20.671 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[29319]: 2021-02-10T04:39:20 2021-02-10T04:39:20.671 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[29319]: .257918+0000 mon.a (mon.0 2021-02-10T04:39:20.672 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[29319]: ) 1659 : cluster [DBG] osdmap e741: 8 total, 8 up, 7 in 2021-02-10T04:39:20.672 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[29319]: 2021-02-10T04:39:20.672 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[35961]: cluster 2021-02-10T04:39:19.851150 2021-02-10T04:39:20.672 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1201 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:20.672 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[35961]: cluster 2021-02-10T04:39:19.851156+0000 osd.4 (osd.4) 1202 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:20.672 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:39:20.672 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[35961]: 04:39:19.851160+0000 osd.4 (osd.4) 1203 : 2021-02-10T04:39:20.672 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:20.672 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:39:20.672 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[35961]: 04:39:19.851163+0000 osd.4 (osd.4) 1204 : cluster [WRN] 2021-02-10T04:39:20.672 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:20.672 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[35961]: cluster 2021-02-10T04:39:20. 2021-02-10T04:39:20.673 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[35961]: 173399+0000 mgr.y (mgr.14140) 744 : cluster [DBG] 2021-02-10T04:39:20.673 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[35961]: pgmap v1261: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 489 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.1 MiB/s rd, 972 KiB/s wr, 9 op/s 2021-02-10T04:39:20.673 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[35961]: cluster 2021-02-10T04:39:20 2021-02-10T04:39:20.673 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[35961]: .257918+0000 mon.a (mon.0) 1659 : cluster [DBG] 2021-02-10T04:39:20.673 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[35961]: osdmap e741: 8 total, 8 up, 7 in 2021-02-10T04:39:20.745 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:39:20.777 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.1 dump_blocked_ops 2021-02-10T04:39:20.811 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:20 smithi164 conmon[29288]: cluster 2021-02-10T04:39:19.851150+0000 2021-02-10T04:39:20.811 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:20 smithi164 conmon[29288]: osd.4 (osd.4) 1201 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:20.811 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:20 smithi164 conmon[29288]: cluster 2021-02-10T04:39:19.851156+0000 osd.4 (osd.4) 1202 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:20.812 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:20 smithi164 conmon[29288]: cluster 2021-02-10T04:39:19.851160+0000 osd.4 ( 2021-02-10T04:39:20.812 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:20 smithi164 conmon[29288]: osd.4) 1203 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:20.812 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:20 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:39:20.812 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:20 smithi164 conmon[29288]: -02-10T04:39:19.851163+0000 osd.4 2021-02-10T04:39:20.812 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:20 smithi164 conmon[29288]: (osd.4) 1204 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:20.812 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:20 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:39:20.812 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:20 smithi164 conmon[29288]: T04:39:20.173399+0000 mgr.y (mgr.14140) 744 : cluster [DBG] pgmap v1261: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 489 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.1 MiB/s rd, 972 KiB/s wr, 9 op/s 2021-02-10T04:39:20.812 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:20 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:39:20.812 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:20 smithi164 conmon[29288]: 02-10T04:39:20.257918+0000 2021-02-10T04:39:20.812 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:20 smithi164 conmon[29288]: mon.a (mon.0) 1659 : cluster [DBG] osdmap e741: 8 total, 8 up, 7 in 2021-02-10T04:39:20.852 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:39:20.922 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:39:20 smithi085 conmon[52836]: debug 2021-02-10T04:39:20.707+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:20.973 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-02-10T04:39:20.973 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 0 1 2021-02-10T04:39:21.142 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:39:21.143 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:20 smithi164 conmon[33677]: debug 2021-02-10T04:39:20.809+0000 7fbec19a1700 -1 osd.4 741 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:21.272 INFO:tasks.rados.rados.0.smithi085.stdout:2912: done (5 left) 2021-02-10T04:39:21.272 INFO:tasks.rados.rados.0.smithi085.stdout:2913: write (excl) oid 2 current snap is 334 2021-02-10T04:39:21.272 INFO:tasks.rados.rados.0.smithi085.stdout:2913: seq_num 1015 ranges {659105=652684,1862282=717015} 2021-02-10T04:39:21.276 INFO:tasks.rados.rados.0.smithi085.stdout:2913: writing smithi08575397-2 from 659105 to 1311789 tid 1 2021-02-10T04:39:21.285 INFO:tasks.rados.rados.0.smithi085.stdout:2913: writing smithi08575397-2 from 1862282 to 2579297 tid 2 2021-02-10T04:39:21.285 INFO:tasks.rados.rados.0.smithi085.stdout:2911: expect (ObjNum 966 snap 315 seq_num 966) 2021-02-10T04:39:21.308 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:39:21.319 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:39:21.499 INFO:tasks.rados.rados.0.smithi085.stdout:2913: finishing write tid 1 to smithi08575397-2 2021-02-10T04:39:21.500 INFO:tasks.rados.rados.0.smithi085.stdout:2913: finishing write tid 2 to smithi08575397-2 2021-02-10T04:39:21.500 INFO:tasks.rados.rados.0.smithi085.stdout:2911: done (5 left) 2021-02-10T04:39:21.500 INFO:tasks.rados.rados.0.smithi085.stdout:2914: write oid 25 current snap is 334 2021-02-10T04:39:21.500 INFO:tasks.rados.rados.0.smithi085.stdout:2913: finishing write tid 3 to smithi08575397-2 2021-02-10T04:39:21.500 INFO:tasks.rados.rados.0.smithi085.stdout:2913: finishing write tid 4 to smithi08575397-2 2021-02-10T04:39:21.500 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 1437 (ObjNum 1015 snap 334 seq_num 1015) dirty exists 2021-02-10T04:39:21.500 INFO:tasks.rados.rados.0.smithi085.stdout:2913: left oid 2 (ObjNum 1015 snap 334 seq_num 1015) 2021-02-10T04:39:21.500 INFO:tasks.rados.rados.0.smithi085.stdout:2914: seq_num 1016 ranges {675912=727933,1920764=460673,2801290=274622} 2021-02-10T04:39:21.507 INFO:tasks.rados.rados.0.smithi085.stdout:2914: writing smithi08575397-25 from 675912 to 1403845 tid 1 2021-02-10T04:39:21.514 INFO:tasks.rados.rados.0.smithi085.stdout:2914: writing smithi08575397-25 from 1920764 to 2381437 tid 2 2021-02-10T04:39:21.521 INFO:tasks.rados.rados.0.smithi085.stdout:2914: writing smithi08575397-25 from 2801290 to 3075912 tid 3 2021-02-10T04:39:21.521 INFO:tasks.rados.rados.0.smithi085.stdout:2914: finishing write tid 1 to smithi08575397-25 2021-02-10T04:39:21.521 INFO:tasks.rados.rados.0.smithi085.stdout:2914: finishing write tid 2 to smithi08575397-25 2021-02-10T04:39:21.521 INFO:tasks.rados.rados.0.smithi085.stdout:2913: done (5 left) 2021-02-10T04:39:21.521 INFO:tasks.rados.rados.0.smithi085.stdout:2915: read oid 26 snap 281 2021-02-10T04:39:21.521 INFO:tasks.rados.rados.0.smithi085.stdout:2915: expect (ObjNum 868 snap 280 seq_num 868) 2021-02-10T04:39:21.521 INFO:tasks.rados.rados.0.smithi085.stdout:2916: read oid 9 snap -1 2021-02-10T04:39:21.521 INFO:tasks.rados.rados.0.smithi085.stdout:2916: expect deleted 2021-02-10T04:39:21.522 INFO:tasks.rados.rados.0.smithi085.stdout:2917: write (excl) oid 7 current snap is 334 2021-02-10T04:39:21.522 INFO:tasks.rados.rados.0.smithi085.stdout:2917: seq_num 1017 ranges {692719=403182,1579246=720684,2828713=264006} 2021-02-10T04:39:21.528 INFO:tasks.rados.rados.0.smithi085.stdout:2917: writing smithi08575397-7 from 692719 to 1095901 tid 1 2021-02-10T04:39:21.537 INFO:tasks.rados.rados.0.smithi085.stdout:2917: writing smithi08575397-7 from 1579246 to 2299930 tid 2 2021-02-10T04:39:21.542 INFO:tasks.rados.rados.0.smithi085.stdout:2917: writing smithi08575397-7 from 2828713 to 3092719 tid 3 2021-02-10T04:39:21.542 INFO:tasks.rados.rados.0.smithi085.stdout:2918: read oid 21 snap -1 2021-02-10T04:39:21.542 INFO:tasks.rados.rados.0.smithi085.stdout:2918: expect deleted 2021-02-10T04:39:21.542 INFO:tasks.rados.rados.0.smithi085.stdout:2919: read oid 40 snap -1 2021-02-10T04:39:21.543 INFO:tasks.rados.rados.0.smithi085.stdout:2919: expect (ObjNum 1004 snap 329 seq_num 1004) 2021-02-10T04:39:21.543 INFO:tasks.rados.rados.0.smithi085.stdout:2920: snap_remove snap 314 2021-02-10T04:39:21.643 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:39:21 smithi164 conmon[44544]: debug 2021-02-10T04:39:21.388+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:21 smithi164 conmon[29288]: cluster 2021-02-10T04:39:20.809921 2021-02-10T04:39:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:21 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1205 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:21 smithi164 conmon[29288]: cluster 2021-02-10T04:39:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:21 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:39:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:21 smithi164 conmon[29288]: :39:20.809929+0000 osd.4 (osd.4) 1206 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:21 smithi164 conmon[29288]: cluster 2021-02-10T04:39:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:21 smithi164 conmon[29288]: 2021-02-10T04:39:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:21 smithi164 conmon[29288]: 2021 2021-02-10T04:39:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:21 smithi164 conmon[29288]: - 2021-02-10T04:39:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:21 smithi164 conmon[29288]: 02-10T04:39:20 2021-02-10T04:39:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:21 smithi164 conmon[29288]: .809936+0000 osd.4 (osd.4) 1207 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:21 smithi164 conmon[29288]: cluster 2021-02-10T04:39:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:21 smithi164 conmon[29288]: 2021-02-10T04:39:20.809943+0000 2021-02-10T04:39:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:21 smithi164 conmon[29288]: osd.4 (osd.4) 1208 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:21 smithi164 conmon[29288]: 2021-02-10T04:39:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:21 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:39:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:21 smithi164 conmon[29288]: T04:39:21.266213+0000 2021-02-10T04:39:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:21 smithi164 conmon[29288]: mon.a (mon.0) 1660 : cluster [DBG] 2021-02-10T04:39:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:21 smithi164 conmon[29288]: osdmap e742: 8 total, 8 up, 7 in 2021-02-10T04:39:21.666 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:39:21.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:21 smithi085 conmon[29319]: cluster 2021-02-10T04:39:20.809921+0000 osd.4 (osd.4) 1205 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:21.879 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:21 smithi085 conmon[29319]: 2021-02-10T04:39:21.879 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:21 smithi085 conmon[29319]: cluster 2021-02-10T04:39:20.809929+0000 osd.4 (osd.4) 1206 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:21.879 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:21 smithi085 conmon[29319]: cluster 2021-02-10T04:39:20.809936+0000 osd.4 (osd.4) 1207 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:21.879 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:21 smithi085 conmon[29319]: cluster 2021-02-10T04:39:20.809943+0000 osd.4 (osd.4) 1208 : cluster [WRN] 2021-02-10T04:39:21.879 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:21 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:21.879 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:21 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:39:21.879 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:21 smithi085 conmon[29319]: 02-10T04:39:21.266213+0000 mon.a ( 2021-02-10T04:39:21.879 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:21 smithi085 conmon[29319]: mon.0) 1660 : cluster [DBG] osdmap e742: 8 total, 8 up, 7 in 2021-02-10T04:39:21.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:21 smithi085 conmon[35961]: cluster 2021-02-10T04:39:20.809921+0000 osd.4 ( 2021-02-10T04:39:21.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:21 smithi085 conmon[35961]: osd.4) 1205 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:21.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:21 smithi085 conmon[35961]: cluster 2021-02-10T04:39:20.809929+0000 2021-02-10T04:39:21.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:21 smithi085 conmon[35961]: osd.4 (osd.4) 1206 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:21.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:21 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:39:21.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:21 smithi085 conmon[35961]: 02-10T04:39:20.809936+0000 osd.4 2021-02-10T04:39:21.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:21 smithi085 conmon[35961]: (osd.4) 1207 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:21.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:21 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:39:21.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:21 smithi085 conmon[35961]: 10T04:39:20.809943+0000 osd.4 (osd. 2021-02-10T04:39:21.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:21 smithi085 conmon[35961]: 4) 1208 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:21.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:21 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:39:21.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:21 smithi085 conmon[35961]: T04:39:21.266213+0000 mon.a (mon. 2021-02-10T04:39:21.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:21 smithi085 conmon[35961]: 0) 1660 : cluster [DBG] osdmap e742: 8 total, 8 up, 7 in 2021-02-10T04:39:21.881 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:39:21 smithi085 conmon[41540]: debug 2021-02-10T04:39:21.349+0000 7fb94bb63700 -1 osd.0 pg_epoch: 742 pg[5.5( empty local-lis/les=57/58 n=0 ec=57/57 lis/c=57/57 les/c/f=58/58/0 sis=57) [0,1,4] r=0 lpr=57 crt=0'0 mlcod 0'0 active+clean] scrubber pg(5.5) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x555d327d2420) 2021-02-10T04:39:21.882 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:39:21.914 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:39:21.999 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:21 smithi164 conmon[33677]: debug 2021-02-10T04:39:21.999 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:21 smithi164 conmon[33677]: 2021-02-10T04:39:21.809+0000 7fbec19a1700 -1 osd.4 742 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:21.999 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:21 smithi164 conmon[33677]: 2021-02-10T04:39:22.262 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:39:22.265 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:39:21 smithi164 conmon[37263]: debug 2021-02-10T04:39:21.996+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:22.367 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:39:22.399 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:39:22.524 INFO:tasks.rados.rados.0.smithi085.stdout:2920: done (10 left) 2021-02-10T04:39:22.524 INFO:tasks.rados.rados.0.smithi085.stdout:2921: rollback oid 20 current snap is 334 2021-02-10T04:39:22.524 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 20 to 309 2021-02-10T04:39:22.524 INFO:tasks.rados.rados.0.smithi085.stdout:2922: snap_remove snap 288 2021-02-10T04:39:22.526 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:22 smithi164 conmon[29288]: cluster 2021-02-10T04:39:21.809943+0000 osd.4 (osd.4) 2021-02-10T04:39:22.526 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:22 smithi164 conmon[29288]: 1209 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:22.526 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:22 smithi164 conmon[29288]: cluster 2021-02-10T04:39:21.809952+0000 osd.4 2021-02-10T04:39:22.526 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:22 smithi164 conmon[29288]: (osd.4) 1210 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:22.526 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:22 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:39:22.527 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:22 smithi164 conmon[29288]: 10T04:39:21.809960+0000 osd.4 ( 2021-02-10T04:39:22.527 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:22 smithi164 conmon[29288]: osd.4) 1211 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:22.527 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:22 smithi164 conmon[29288]: cluster 2021-02-10T04:39:22.527 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:22 smithi164 conmon[29288]: 2021-02-10T04:39: 2021-02-10T04:39:22.527 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:22 smithi164 conmon[29288]: 21.809966+0000 osd.4 (osd.4) 1212 2021-02-10T04:39:22.527 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:22 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:22.527 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:22 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:39:22.527 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:22 smithi164 conmon[29288]: -02-10T04:39:22 2021-02-10T04:39:22.527 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:22 smithi164 conmon[29288]: .174018+0000 mgr.y (mgr. 2021-02-10T04:39:22.527 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:22 smithi164 conmon[29288]: 14140) 745 : cluster [DBG] pgmap v1264: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 495 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 4.1 MiB/s rd, 1.6 MiB/s wr, 9 op/s 2021-02-10T04:39:22.527 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:22 smithi164 conmon[29288]: 2021-02-10T04:39:22.527 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:22 smithi164 conmon[33677]: debug 2021-02-10T04:39:22.472+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:22.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:22 smithi085 conmon[29319]: cluster 2021-02-10T04:39:21. 2021-02-10T04:39:22.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:22 smithi085 conmon[29319]: 809943+0000 osd.4 (osd.4) 1209 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:22.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:22 smithi085 conmon[29319]: cluster 2021-02-10T04:39:21.809952+0000 osd.4 (osd.4) 1210 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:22 smithi085 conmon[29319]: cluster 2021-02-10T04:39:21.809960+0000 osd.4 (osd.4) 1211 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:22 smithi085 conmon[29319]: cluster 2021-02-10T04:39:21.809966+0000 osd.4 2021-02-10T04:39:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:22 smithi085 conmon[29319]: (osd.4) 1212 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:22 smithi085 conmon[29319]: cluster 2021-02-10T04:39:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:22 smithi085 conmon[29319]: 2021-02-10T04:39:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:22 smithi085 conmon[29319]: 2021 2021-02-10T04:39:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:22 smithi085 conmon[29319]: - 2021-02-10T04:39:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:22 smithi085 conmon[29319]: 02 2021-02-10T04:39:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:22 smithi085 conmon[29319]: - 2021-02-10T04:39:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:22 smithi085 conmon[29319]: 10 2021-02-10T04:39:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:22 smithi085 conmon[29319]: T 2021-02-10T04:39:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:22 smithi085 conmon[29319]: 04:39:22.174018 2021-02-10T04:39:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:22 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 745 : cluster [DBG] pgmap v1264: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 495 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 4.1 MiB/s rd, 1.6 MiB/s wr, 9 op/s 2021-02-10T04:39:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:22 smithi085 conmon[35961]: cluster 2021-02-10T04:39:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:22 smithi085 conmon[35961]: 2021-02-10T04:39:21.809943+0000 osd.4 2021-02-10T04:39:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:22 smithi085 conmon[35961]: (osd.4) 1209 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:22 smithi085 conmon[35961]: cluster 2021-02-10T04:39:21.809952+0000 osd.4 (osd.4) 1210 : cluster 2021-02-10T04:39:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:22 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:22 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:39:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:22 smithi085 conmon[35961]: 39:21.809960+0000 osd.4 (osd.4) 1211 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:22 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:39:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:22 smithi085 conmon[35961]: -02-10T04:39:21.809966+0000 osd.4 (osd. 2021-02-10T04:39:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:22 smithi085 conmon[35961]: 4) 1212 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:22 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:39:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:22 smithi085 conmon[35961]: 39:22.174018+0000 mgr.y (mgr.14140) 745 : 2021-02-10T04:39:22.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:22 smithi085 conmon[35961]: cluster [DBG] pgmap v1264: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 495 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 4.1 MiB/s rd, 1.6 MiB/s wr, 9 op/s 2021-02-10T04:39:22.849 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:39:22.854 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:22 smithi164 conmon[33677]: debug 2021-02-10T04:39:22.771+0000 7fbec19a1700 -1 osd.4 743 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:22.956 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:39:22.991 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:39:23.144 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:39:23 smithi164 conmon[37263]: debug 2021-02-10T04:39:23.144 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:39:23 smithi164 conmon[37263]: 2021-02-10T04:39:23.083+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:23.144 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:39:23 smithi164 conmon[37263]: 2021-02-10T04:39:23.249 INFO:tasks.rados.rados.0.smithi085.stdout:2922: done (11 left) 2021-02-10T04:39:23.249 INFO:tasks.rados.rados.0.smithi085.stdout:2923: copy_from oid 1 from oid 10 current snap is 334 2021-02-10T04:39:23.249 INFO:tasks.rados.rados.0.smithi085.stdout:2924: rollback oid 44 current snap is 334 2021-02-10T04:39:23.249 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 44 to 309 2021-02-10T04:39:23.250 INFO:tasks.rados.rados.0.smithi085.stdout:2925: snap_remove snap 323 2021-02-10T04:39:23.383 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:39:23.488 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:39:23.518 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:39:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:23 smithi164 conmon[29288]: cluster 2021-02-10T04:39:22.520346+0000 mon.a (mon. 2021-02-10T04:39:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:23 smithi164 conmon[29288]: 0) 1661 : cluster [DBG] osdmap e743: 8 total, 8 up, 7 in 2021-02-10T04:39:23.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:23 smithi164 conmon[29288]: cluster 2021-02-10T04:39:22.772606+0000 osd.4 (osd.4) 1213 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:23 smithi164 conmon[29288]: cluster 2021-02-10T04:39:22.772617+0000 osd.4 (osd.4) 1214 : 2021-02-10T04:39:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:23 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:23 smithi164 conmon[29288]: cluster 2021-02-10T04:39:22. 2021-02-10T04:39:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:23 smithi164 conmon[29288]: 772626+0000 osd.4 (osd.4) 1215 : cluster [WRN] 2021-02-10T04:39:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:23 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:23.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:23 smithi164 conmon[29288]: cluster 2021-02-10T04:39:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:23 smithi164 conmon[29288]: 2021-02-10T04:39:22.772633+0000 osd.4 (osd.4) 1216 : cluster 2021-02-10T04:39:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:23 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:23 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:39:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:23 smithi164 conmon[29288]: -02-10T04:39:23.238795+0000 mon.a (mon.0 2021-02-10T04:39:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:23 smithi164 conmon[29288]: ) 1662 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 592 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:39:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:23 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:39:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:23 smithi164 conmon[29288]: 04:39:23.247739+0000 mon.a (mon. 2021-02-10T04:39:23.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:23 smithi164 conmon[29288]: 0) 1663 : cluster [DBG] osdmap e744: 8 total, 8 up, 7 in 2021-02-10T04:39:23.648 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:23 smithi164 conmon[33677]: debug 2021-02-10T04:39:23.648 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:23 smithi164 conmon[33677]: 2021-02-10T04:39:23.599+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:23.833 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:39:23.939 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:39:23.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[29319]: cluster 2021-02-10T04:39:22 2021-02-10T04:39:23.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[29319]: .520346+0000 mon.a (mon.0) 1661 : cluster [DBG] osdmap e743: 8 total, 8 up, 7 in 2021-02-10T04:39:23.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[29319]: cluster 2021-02-10T04:39:22.772606+0000 2021-02-10T04:39:23.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[29319]: osd.4 (osd.4) 1213 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:23.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[29319]: cluster 2021-02-10T04:39 2021-02-10T04:39:23.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[29319]: :22.772617+0000 osd.4 (osd.4) 1214 : 2021-02-10T04:39:23.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:23.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:39:23.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[29319]: 10T04:39:22.772626+0000 osd.4 (osd 2021-02-10T04:39:23.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[29319]: .4) 1215 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:23.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[29319]: cluster 2021-02-10T04:39:23.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:39:23.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[29319]: :39:22.772633+0000 osd.4 (osd.4) 2021-02-10T04:39:23.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[29319]: 1216 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:23.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:39:23.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[29319]: -10T04:39:23.238795+0000 mon.a (mon 2021-02-10T04:39:23.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[29319]: .0) 2021-02-10T04:39:23.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[29319]: 1662 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 592 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:39:23.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[29319]: cluster 2021-02-10T04:39:23.247739 2021-02-10T04:39:23.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1663 : cluster [DBG] osdmap e744: 8 total, 8 up, 7 in 2021-02-10T04:39:23.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:39:23.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[35961]: 04:39:22.520346+0000 mon.a (mon.0) 1661 : cluster [DBG] osdmap e743: 8 total, 8 up, 7 in 2021-02-10T04:39:23.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:39:23.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[35961]: -10T04:39:22.772606+0000 osd.4 (osd.4) 1213 2021-02-10T04:39:23.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:23.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:39:23.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[35961]: T04:39:22.772617+0000 osd.4 (osd.4) 1214 : cluster 2021-02-10T04:39:23.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:23.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:39:23.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[35961]: -10T04:39:22.772626+0000 osd.4 (osd.4) 1215 : cluster 2021-02-10T04:39:23.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:23.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:39:23.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[35961]: 02-10T04:39:22.772633+0000 osd.4 (osd. 2021-02-10T04:39:23.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[35961]: 4) 1216 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:23.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:39:23.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[35961]: -10T04:39:23.238795+0000 mon.a (mon.0) 1662 2021-02-10T04:39:23.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[35961]: : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 592 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:39:23.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[35961]: cluster 2021-02-10T04:39:23.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[35961]: 2021-02-10T04:39:23.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:39:23.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[35961]: :39:23.247739+0000 mon.a (mon.0) 1663 : cluster [DBG] osdmap e744: 8 total, 8 up, 7 in 2021-02-10T04:39:23.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:23 smithi085 conmon[35961]: 2021-02-10T04:39:23.971 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:39:24.144 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:39:24 smithi164 conmon[44544]: debug 2021-02-10T04:39:24.044+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:24.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:23 smithi164 conmon[33677]: debug 2021-02-10T04:39:23.803+0000 7fbec19a1700 -1 osd.4 744 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:24.251 INFO:tasks.rados.rados.0.smithi085.stdout:2925: done (13 left) 2021-02-10T04:39:24.251 INFO:tasks.rados.rados.0.smithi085.stdout:2926: snap_remove snap 293 2021-02-10T04:39:24.367 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:39:24.472 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:39:24.501 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:39:24 smithi164 conmon[37263]: debug 2021-02-10T04:39:24.175+0000 7f1c671e2700 -1 osd.5 pg_epoch: 744 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x5566508ab1e0) 2021-02-10T04:39:24.505 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:39:24.691 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[41540]: debug 2021-02-10T04:39:24.330+0000 7fb947b5b700 -1 osd.0 pg_epoch: 745 pg[5.5( empty local-lis/les=57/58 n=0 ec=57/57 lis/c=57/57 les/c/f=58/58/0 sis=57) [0,1,4] r=0 lpr=57 crt=0'0 mlcod 0'0 active+clean] scrubber pg(5.5) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x555d32bd8f20) 2021-02-10T04:39:24.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[29319]: cluster 2021-02-10T04:39:22.874629+0000 osd.6 (osd 2021-02-10T04:39:24.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[29319]: .6) 78 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:39:24.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[29319]: cluster 2021-02-10T04:39:22.875921+0000 osd.6 (osd.6) 79 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:39:24.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[29319]: cluster 2021-02-10T04:39:23.803769+0000 osd.4 (osd.4) 1217 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:24.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[29319]: cluster 2021-02-10T04:39:23.803780+0000 osd.4 (osd.4) 1218 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:24.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[29319]: cluster 2021-02-10T04:39:23.803786+0000 osd.4 (osd.4) 1219 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:24.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[29319]: cluster 2021-02-10T04:39:23.803792+0000 osd.4 (osd.4) 1220 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:24.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[29319]: cluster 2021-02-10T04:39:24.174653+0000 mgr.y (mgr.14140) 746 : cluster [DBG] pgmap v1267: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 495 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 6.2 MiB/s rd, 970 KiB/s wr, 8 op/s 2021-02-10T04:39:24.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[29319]: cluster 2021-02-10T04:39:24.250743+0000 mon.a (mon.0) 1664 : cluster [DBG] osdmap e745: 8 total, 8 up, 7 in 2021-02-10T04:39:24.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[35961]: cluster 2021-02-10T04:39:22.874629+0000 osd.6 (osd.6) 78 : 2021-02-10T04:39:24.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[35961]: cluster [DBG] purged_snaps scrub starts 2021-02-10T04:39:24.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[35961]: cluster 2021-02-10T04:39:24.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[35961]: 2021-02-10T04:39:22.875921+0000 osd.6 (osd.6) 79 : 2021-02-10T04:39:24.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[35961]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:39:24.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[35961]: cluster 2021-02-10T04:39: 2021-02-10T04:39:24.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[35961]: 23.803769+0000 osd.4 (osd.4) 1217 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:24.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[35961]: 2021-02-10T04:39:24.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:39:24.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[35961]: 02-10T04:39:23.803780+0000 osd.4 (osd.4) 1218 : cluster [WRN] 2021-02-10T04:39:24.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:24.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:39:24.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[35961]: 02-10T04:39:23.803786+0000 osd.4 (osd.4) 1219 : 2021-02-10T04:39:24.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:24.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:39:24.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[35961]: -10T04:39:23.803792+0000 osd.4 (osd.4 2021-02-10T04:39:24.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[35961]: ) 1220 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:24.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:39:24.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[35961]: -02-10T04:39:24 2021-02-10T04:39:24.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[35961]: .174653+0000 mgr.y (mgr.14140) 746 : 2021-02-10T04:39:24.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[35961]: cluster [DBG] pgmap v1267: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 495 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 6.2 MiB/s rd, 970 KiB/s wr, 8 op/s 2021-02-10T04:39:24.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[35961]: cluster 2021-02-10T04:39:24.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[35961]: 2021-02-10T04:39:24.250743+0000 mon.a 2021-02-10T04:39:24.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[35961]: (mon.0) 1664 : 2021-02-10T04:39:24.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:24 smithi085 conmon[35961]: cluster [DBG] osdmap e745: 8 total, 8 up, 7 in 2021-02-10T04:39:24.743 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:39:24.801 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:39:24 smithi164 conmon[37263]: debug 2021-02-10T04:39:24.588+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:24.804 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:24 smithi164 conmon[29288]: cluster 2021-02-10T04:39:22.874629+0000 osd.6 (osd. 2021-02-10T04:39:24.804 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:24 smithi164 conmon[29288]: 6) 78 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:39:24.804 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:24 smithi164 conmon[29288]: cluster 2021-02-10T04:39:22.875921+0000 osd.6 (osd.6) 79 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:39:24.804 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:24 smithi164 conmon[29288]: cluster 2021-02-10T04:39:23.803769+0000 osd.4 (osd.4) 1217 : 2021-02-10T04:39:24.804 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:24 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:24.804 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:24 smithi164 conmon[29288]: cluster 2021-02-10T04:39:23 2021-02-10T04:39:24.804 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:24 smithi164 conmon[29288]: .803780+0000 osd.4 (osd.4) 1218 : cluster 2021-02-10T04:39:24.804 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:24 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:24.804 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:24 smithi164 conmon[29288]: cluster 2021-02-10T04:39:23.803786 2021-02-10T04:39:24.804 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:24 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1219 : cluster [WRN] 2021-02-10T04:39:24.804 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:24 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:24.804 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:24 smithi164 conmon[29288]: cluster 2021-02-10T04:39:23.803792 2021-02-10T04:39:24.805 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:24 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1220 : cluster [WRN] 2021-02-10T04:39:24.805 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:24 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:24.805 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:24 smithi164 conmon[29288]: cluster 2021-02-10T04:39:24.174653 2021-02-10T04:39:24.805 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:24 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 746 : cluster 2021-02-10T04:39:24.805 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:24 smithi164 conmon[29288]: [DBG] pgmap v1267: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 495 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 6.2 MiB/s rd, 970 KiB/s wr, 8 op/s 2021-02-10T04:39:24.805 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:24 smithi164 conmon[29288]: cluster 2021-02-10T04:39: 2021-02-10T04:39:24.805 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:24 smithi164 conmon[29288]: 24.250743+0000 mon.a (mon.0) 1664 2021-02-10T04:39:24.805 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:24 smithi164 conmon[29288]: : cluster [DBG] osdmap e745: 8 total, 8 up, 7 in 2021-02-10T04:39:24.805 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:24 smithi164 conmon[33677]: debug 2021-02-10T04:39:24.848 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:39:24.883 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:39:25.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:24 smithi164 conmon[33677]: 2021-02-10T04:39:24.800+0000 7fbec19a1700 -1 osd.4 745 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:25.248 INFO:tasks.rados.rados.0.smithi085.stdout:2926: done (13 left) 2021-02-10T04:39:25.249 INFO:tasks.rados.rados.0.smithi085.stdout:2927: rollback oid 2 current snap is 334 2021-02-10T04:39:25.249 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 2 to 311 2021-02-10T04:39:25.249 INFO:tasks.rados.rados.0.smithi085.stdout:2928: snap_remove snap 318 2021-02-10T04:39:25.895 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:25 smithi164 conmon[33677]: debug 2021-02-10T04:39:25.755+0000 7fbec19a1700 -1 osd.4 746 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:25.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:25 smithi164 conmon[29288]: cluster 2021-02-10T04:39:24.801624 2021-02-10T04:39:25.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:25 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1221 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:25.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:25 smithi164 conmon[29288]: cluster 2021-02-10T04:39:24.801633+0000 2021-02-10T04:39:25.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:25 smithi164 conmon[29288]: osd.4 (osd.4) 1222 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:25.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:25 smithi164 conmon[29288]: cluster 2021-02-10T04:39:24. 2021-02-10T04:39:25.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:25 smithi164 conmon[29288]: 801639+0000 osd.4 (osd.4) 1223 : cluster [WRN] 2021-02-10T04:39:25.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:25 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:25.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:25 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:39:25.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:25 smithi164 conmon[29288]: 02-10T04:39:24.801646+0000 2021-02-10T04:39:25.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:25 smithi164 conmon[29288]: osd.4 (osd.4) 1224 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:25.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:25 smithi164 conmon[29288]: cluster 2021-02-10T04:39:25.247556+0000 2021-02-10T04:39:25.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:25 smithi164 conmon[29288]: mon.a (mon.0) 1665 : cluster [DBG] osdmap e746: 8 total, 8 up, 7 in 2021-02-10T04:39:25.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:25 smithi085 conmon[29319]: cluster 2021-02-10T04:39:24.801624+0000 osd.4 (osd.4) 1221 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:25.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:25 smithi085 conmon[29319]: cluster 2021-02-10T04:39:24.801633+0000 osd.4 (osd.4) 1222 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:25.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:25 smithi085 conmon[29319]: cluster 2021-02-10T04:39:24.801639+0000 osd.4 (osd.4) 1223 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:25.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:25 smithi085 conmon[29319]: cluster 2021-02-10T04:39:24.801646+0000 osd.4 (osd.4) 1224 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:25.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:25 smithi085 conmon[29319]: cluster 2021-02-10T04:39:25.247556+0000 mon.a (mon.0) 1665 : cluster [DBG] osdmap e746: 8 total, 8 up, 7 in 2021-02-10T04:39:25.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:25 smithi085 conmon[35961]: cluster 2021-02-10T04:39:24.801624+0000 osd.4 (osd.4) 1221 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:25.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:25 smithi085 conmon[35961]: cluster 2021-02-10T04:39:24.801633+0000 osd.4 (osd.4) 1222 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:25.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:25 smithi085 conmon[35961]: cluster 2021-02-10T04:39:24.801639+0000 osd.4 (osd.4) 1223 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:25.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:25 smithi085 conmon[35961]: cluster 2021-02-10T04:39:24.801646+0000 osd.4 (osd.4) 1224 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:25.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:25 smithi085 conmon[35961]: cluster 2021-02-10T04:39:25.247556+0000 mon.a (mon.0) 1665 : cluster [DBG] osdmap e746: 8 total, 8 up, 7 in 2021-02-10T04:39:26.250 INFO:tasks.rados.rados.0.smithi085.stdout:2928: done (14 left) 2021-02-10T04:39:26.251 INFO:tasks.rados.rados.0.smithi085.stdout:2929: snap_create 2021-02-10T04:39:26.251 INFO:tasks.rados.rados.0.smithi085.stdout:2915: expect (ObjNum 868 snap 280 seq_num 868) 2021-02-10T04:39:26.362 INFO:tasks.rados.rados.0.smithi085.stdout:2914: finishing write tid 3 to smithi08575397-25 2021-02-10T04:39:26.362 INFO:tasks.rados.rados.0.smithi085.stdout:2914: finishing write tid 4 to smithi08575397-25 2021-02-10T04:39:26.362 INFO:tasks.rados.rados.0.smithi085.stdout:2914: finishing write tid 5 to smithi08575397-25 2021-02-10T04:39:26.362 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 25 v 1293 (ObjNum 1016 snap 334 seq_num 1016) dirty exists 2021-02-10T04:39:26.362 INFO:tasks.rados.rados.0.smithi085.stdout:2914: left oid 25 (ObjNum 1016 snap 334 seq_num 1016) 2021-02-10T04:39:26.363 INFO:tasks.rados.rados.0.smithi085.stdout:2917: finishing write tid 1 to smithi08575397-7 2021-02-10T04:39:26.363 INFO:tasks.rados.rados.0.smithi085.stdout:2917: finishing write tid 2 to smithi08575397-7 2021-02-10T04:39:26.363 INFO:tasks.rados.rados.0.smithi085.stdout:2917: finishing write tid 3 to smithi08575397-7 2021-02-10T04:39:26.363 INFO:tasks.rados.rados.0.smithi085.stdout:2917: finishing write tid 4 to smithi08575397-7 2021-02-10T04:39:26.363 INFO:tasks.rados.rados.0.smithi085.stdout:2917: finishing write tid 5 to smithi08575397-7 2021-02-10T04:39:26.363 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 1358 (ObjNum 1017 snap 334 seq_num 1017) dirty exists 2021-02-10T04:39:26.363 INFO:tasks.rados.rados.0.smithi085.stdout:2917: left oid 7 (ObjNum 1017 snap 334 seq_num 1017) 2021-02-10T04:39:26.363 INFO:tasks.rados.rados.0.smithi085.stdout:2919: expect (ObjNum 1004 snap 329 seq_num 1004) 2021-02-10T04:39:26.494 INFO:tasks.rados.rados.0.smithi085.stdout:2921: finishing rollback tid 0 to smithi08575397-20 2021-02-10T04:39:26.494 INFO:tasks.rados.rados.0.smithi085.stdout:2921: finishing rollback tid 1 to smithi08575397-20 2021-02-10T04:39:26.495 INFO:tasks.rados.rados.0.smithi085.stdout:2921: finishing rollback tid 2 to smithi08575397-20 2021-02-10T04:39:26.495 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 20 v 1155 (ObjNum 866 snap 278 seq_num 866) dirty exists 2021-02-10T04:39:26.495 INFO:tasks.rados.rados.0.smithi085.stdout:2924: finishing rollback tid 1 to smithi08575397-44 2021-02-10T04:39:26.495 INFO:tasks.rados.rados.0.smithi085.stdout:2924: finishing rollback tid 2 to smithi08575397-44 2021-02-10T04:39:26.495 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 1157 (ObjNum 266 snap 64 seq_num 266) dirty exists 2021-02-10T04:39:26.495 INFO:tasks.rados.rados.0.smithi085.stdout:2923: finishing copy_from to smithi08575397-1 2021-02-10T04:39:26.495 INFO:tasks.rados.rados.0.smithi085.stdout:2923: got expected ENOENT (src dne) 2021-02-10T04:39:26.495 INFO:tasks.rados.rados.0.smithi085.stdout:2923: finishing copy_from racing read to smithi08575397-1 2021-02-10T04:39:26.495 INFO:tasks.rados.rados.0.smithi085.stdout:2927: finishing rollback tid 0 to smithi08575397-2 2021-02-10T04:39:26.495 INFO:tasks.rados.rados.0.smithi085.stdout:2927: finishing rollback tid 1 to smithi08575397-2 2021-02-10T04:39:26.495 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 1439 (ObjNum 82954032 snap 22030 seq_num 82907328) dirty dne 2021-02-10T04:39:26.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[29319]: cluster 2021-02-10T04:39:25.755856+0000 osd.4 (osd 2021-02-10T04:39:26.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[29319]: .4) 1225 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:26.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[29319]: cluster 2021-02-10T04:39:25.755863+0000 osd.4 (osd.4) 1226 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:26.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[29319]: cluster 2021-02-10T04:39:25.755866+0000 osd.4 (osd.4) 1227 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:26.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:39:26.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[29319]: T04:39:25.755870+0000 osd.4 (osd.4) 1228 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:26.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[29319]: cluster 2021-02-10T04:39:26.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[29319]: 2021-02-10T04:39:26. 2021-02-10T04:39:26.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[29319]: 175120+0000 mgr.y (mgr.14140) 747 : cluster 2021-02-10T04:39:26.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[29319]: [DBG] pgmap v1270: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 501 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 9.0 MiB/s rd, 1.0 MiB/s wr, 14 op/s 2021-02-10T04:39:26.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[29319]: cluster 2021-02-10T04:39:26.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[29319]: 2021-02-10T04:39:26. 2021-02-10T04:39:26.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[29319]: 248507+0000 mon.a (mon.0) 1666 : 2021-02-10T04:39:26.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[29319]: cluster [DBG] osdmap e747: 8 total, 8 up, 7 in 2021-02-10T04:39:26.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[35961]: cluster 2021-02-10T04:39:25.755856+0000 2021-02-10T04:39:26.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[35961]: osd.4 (osd.4) 1225 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:26.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[35961]: cluster 2021-02-10T04:39:25.755863 2021-02-10T04:39:26.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1226 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:26.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:39:26.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[35961]: 04:39:25.755866+0000 osd.4 (osd 2021-02-10T04:39:26.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[35961]: .4) 1227 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:26.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:39:26.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[35961]: 02-10T04:39:25.755870+0000 osd.4 2021-02-10T04:39:26.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[35961]: (osd.4) 1228 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:26.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[35961]: cluster 2021-02-10T04:39:26.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[35961]: 2021-02-10T04:39:26. 2021-02-10T04:39:26.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[35961]: 175120+0000 mgr.y (mgr.14140) 747 : 2021-02-10T04:39:26.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[35961]: cluster [DBG] pgmap v1270: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 501 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 9.0 MiB/s rd, 1.0 MiB/s wr, 14 op/s 2021-02-10T04:39:26.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:39:26.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[35961]: 04:39:26.248507+0000 mon.a (mon. 2021-02-10T04:39:26.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[35961]: 0) 1666 : cluster [DBG] osdmap e747: 8 total, 8 up, 7 in 2021-02-10T04:39:26.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:26 smithi164 conmon[33677]: debug 2021-02-10T04:39:26.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:26 smithi164 conmon[33677]: 2021-02-10T04:39:26.781+0000 7fbec19a1700 -1 osd.4 747 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:26.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:26 smithi164 conmon[29288]: cluster 2021-02-10T04:39:25.755856+0000 osd.4 (osd.4) 2021-02-10T04:39:26.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:26 smithi164 conmon[29288]: 1225 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:26.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:26 smithi164 conmon[29288]: cluster 2021-02-10T04:39:25.755863+0000 osd.4 (osd.4) 1226 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:26.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:26 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:39:26.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:26 smithi164 conmon[29288]: 04:39:25.755866+0000 osd.4 (osd.4) 1227 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:26.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:26 smithi164 conmon[29288]: cluster 2021-02-10T04:39 2021-02-10T04:39:26.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:26 smithi164 conmon[29288]: :25.755870+0000 osd.4 (osd.4) 1228 : 2021-02-10T04:39:26.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:26 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:26.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:26 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:39:26.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:26 smithi164 conmon[29288]: 39:26.175120+0000 mgr.y (mgr.14140 2021-02-10T04:39:26.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:26 smithi164 conmon[29288]: ) 747 : cluster [DBG] pgmap v1270: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 501 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 9.0 MiB/s rd, 1.0 MiB/s wr, 14 op/s 2021-02-10T04:39:26.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:26 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:39:26.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:26 smithi164 conmon[29288]: 10T04:39:26.248507+0000 mon.a ( 2021-02-10T04:39:26.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:26 smithi164 conmon[29288]: mon.0) 1666 : cluster [DBG] osdmap e747: 8 total, 8 up, 7 in 2021-02-10T04:39:27.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:39:26 smithi085 conmon[49106]: debug 2021-02-10T04:39:26.918+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:27.252 INFO:tasks.rados.rados.0.smithi085.stdout:2914: done (14 left) 2021-02-10T04:39:27.253 INFO:tasks.rados.rados.0.smithi085.stdout:2915: done (13 left) 2021-02-10T04:39:27.253 INFO:tasks.rados.rados.0.smithi085.stdout:2916: done (12 left) 2021-02-10T04:39:27.253 INFO:tasks.rados.rados.0.smithi085.stdout:2917: done (11 left) 2021-02-10T04:39:27.253 INFO:tasks.rados.rados.0.smithi085.stdout:2918: done (10 left) 2021-02-10T04:39:27.253 INFO:tasks.rados.rados.0.smithi085.stdout:2919: done (9 left) 2021-02-10T04:39:27.253 INFO:tasks.rados.rados.0.smithi085.stdout:2921: done (8 left) 2021-02-10T04:39:27.253 INFO:tasks.rados.rados.0.smithi085.stdout:2923: done (7 left) 2021-02-10T04:39:27.253 INFO:tasks.rados.rados.0.smithi085.stdout:2924: done (6 left) 2021-02-10T04:39:27.253 INFO:tasks.rados.rados.0.smithi085.stdout:2927: done (5 left) 2021-02-10T04:39:27.253 INFO:tasks.rados.rados.0.smithi085.stdout:2929: done (4 left) 2021-02-10T04:39:27.253 INFO:tasks.rados.rados.0.smithi085.stdout:2930: snap_remove snap 313 2021-02-10T04:39:27.285 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:39:27.798 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: cluster 2021-02-10T04:39:27.798 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: 2021-02-10T04:39:27.798 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: 2021 2021-02-10T04:39:27.798 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: - 2021-02-10T04:39:27.798 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: 02 2021-02-10T04:39:27.798 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: - 2021-02-10T04:39:27.798 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: 10 2021-02-10T04:39:27.798 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: T 2021-02-10T04:39:27.798 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: 04 2021-02-10T04:39:27.798 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: : 2021-02-10T04:39:27.799 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: 39 2021-02-10T04:39:27.799 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: : 2021-02-10T04:39:27.799 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: 26 2021-02-10T04:39:27.799 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: . 2021-02-10T04:39:27.799 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: 781784 2021-02-10T04:39:27.799 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: +0000 2021-02-10T04:39:27.799 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: 2021-02-10T04:39:27.799 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: osd.4 2021-02-10T04:39:27.799 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: ( 2021-02-10T04:39:27.799 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: osd 2021-02-10T04:39:27.799 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: .4) 1229 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:27.799 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:39:27.799 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: 10T04:39:26.781794+0000 osd.4 (osd.4) 1230 : cluster 2021-02-10T04:39:27.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:27.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:39:27.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: 02-10T04:39:26.781818+0000 2021-02-10T04:39:27.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: osd.4 (osd.4) 1231 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:27.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:39:27.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: T04:39:26.781825+0000 osd.4 (osd.4 2021-02-10T04:39:27.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: ) 1232 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:27.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:39:27.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: -10T04:39:27.249955+0000 mon.a ( 2021-02-10T04:39:27.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[29288]: mon.0) 1667 : cluster [DBG] osdmap e748: 8 total, 8 up, 7 in 2021-02-10T04:39:27.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[35961]: cluster 2021-02-10T04:39:27.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[35961]: 2021-02-10T04:39:27.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[35961]: 2021 2021-02-10T04:39:27.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[35961]: -02-10T04:39:26. 2021-02-10T04:39:27.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[35961]: 781784+0000 osd.4 (osd.4) 1229 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:27.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[35961]: cluster 2021-02-10T04:39:27.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[35961]: 2021-02-10T04:39:27.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[35961]: 2021 2021-02-10T04:39:27.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[35961]: - 2021-02-10T04:39:27.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[35961]: 02 2021-02-10T04:39:27.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[35961]: - 2021-02-10T04:39:27.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[35961]: 10 2021-02-10T04:39:27.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[35961]: T 2021-02-10T04:39:27.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[35961]: 04 2021-02-10T04:39:27.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[35961]: : 2021-02-10T04:39:27.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[35961]: 39 2021-02-10T04:39:27.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[35961]: : 2021-02-10T04:39:27.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[35961]: 26.781794+0000 osd.4 ( 2021-02-10T04:39:27.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[35961]: osd.4) 1230 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:27.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[35961]: cluster 2021-02-10T04:39:27.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[35961]: 2021-02-10T04:39:27.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[35961]: 2021-02-10T04:39:26 2021-02-10T04:39:27.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[35961]: .781818+0000 osd.4 (osd.4) 1231 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:27.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[35961]: cluster 2021-02-10T04:39:27.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[35961]: 2021-02-10T04:39:27.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[35961]: 2021 2021-02-10T04:39:27.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[35961]: -02-10T04:39 2021-02-10T04:39:27.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[35961]: :26.781825+0000 osd.4 (osd.4) 1232 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:27.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:39:27.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[35961]: 39:27.249955+0000 mon.a (mon.0) 1667 : cluster [DBG] osdmap e748: 8 total, 8 up, 7 in 2021-02-10T04:39:27.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[29319]: cluster 2021-02-10T04:39:27.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[29319]: 2021-02-10T04:39:27.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[29319]: 2021 2021-02-10T04:39:27.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[29319]: - 2021-02-10T04:39:27.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[29319]: 02 2021-02-10T04:39:27.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[29319]: - 2021-02-10T04:39:27.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[29319]: 10 2021-02-10T04:39:27.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[29319]: T 2021-02-10T04:39:27.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[29319]: 04 2021-02-10T04:39:27.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[29319]: : 2021-02-10T04:39:27.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[29319]: 39 2021-02-10T04:39:27.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[29319]: : 2021-02-10T04:39:27.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[29319]: 26.781784+0000 osd.4 (osd.4) 2021-02-10T04:39:27.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[29319]: 1229 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:27.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:39:27.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[29319]: 02-10T04:39:26.781794+0000 2021-02-10T04:39:27.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[29319]: osd.4 (osd.4) 1230 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:27.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[29319]: cluster 2021-02-10T04:39:27.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[29319]: 2021-02-10T04:39:27.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[29319]: 2021 2021-02-10T04:39:27.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[29319]: -02-10T04:39: 2021-02-10T04:39:27.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[29319]: 26.781818+0000 osd.4 (osd.4) 1231 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:27.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[29319]: cluster 2021-02-10T04:39:27.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[29319]: 2021-02-10T04:39 2021-02-10T04:39:27.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[29319]: :26.781825+0000 osd.4 (osd.4) 1232 : cluster 2021-02-10T04:39:27.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:27.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[29319]: cluster 2021-02-10T04:39:27.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[29319]: 2021-02-10T04:39:27.249955+0000 2021-02-10T04:39:27.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:27 smithi085 conmon[29319]: mon.a (mon.0) 1667 : cluster [DBG] osdmap e748: 8 total, 8 up, 7 in 2021-02-10T04:39:28.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[33677]: debug 2021-02-10T04:39:28.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[33677]: 2021-02-10T04:39:27.796+0000 7fbec19a1700 -1 osd.4 748 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:28.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:27 smithi164 conmon[33677]: 2021-02-10T04:39:28.250 INFO:tasks.rados.rados.0.smithi085.stdout:2930: done (4 left) 2021-02-10T04:39:28.250 INFO:tasks.rados.rados.0.smithi085.stdout:2931: rollback oid 40 current snap is 335 2021-02-10T04:39:28.251 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 40 to 322 2021-02-10T04:39:28.251 INFO:tasks.rados.rados.0.smithi085.stdout:2932: read oid 50 snap -1 2021-02-10T04:39:28.251 INFO:tasks.rados.rados.0.smithi085.stdout:2932: expect deleted 2021-02-10T04:39:28.251 INFO:tasks.rados.rados.0.smithi085.stdout:2933: read oid 29 snap -1 2021-02-10T04:39:28.251 INFO:tasks.rados.rados.0.smithi085.stdout:2933: expect (ObjNum 939 snap 311 seq_num 939) 2021-02-10T04:39:28.251 INFO:tasks.rados.rados.0.smithi085.stdout:2934: rollback oid 6 current snap is 335 2021-02-10T04:39:28.251 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 6 to 237 2021-02-10T04:39:28.251 INFO:tasks.rados.rados.0.smithi085.stdout:2935: delete oid 49 current snap is 335 2021-02-10T04:39:28.264 INFO:tasks.rados.rados.0.smithi085.stdout:2932: done (8 left) 2021-02-10T04:39:28.264 INFO:tasks.rados.rados.0.smithi085.stdout:2935: done (7 left) 2021-02-10T04:39:28.264 INFO:tasks.rados.rados.0.smithi085.stdout:2936: write oid 38 current snap is 335 2021-02-10T04:39:28.264 INFO:tasks.rados.rados.0.smithi085.stdout:2936: seq_num 1019 ranges {726333=670033,2096210=724353} 2021-02-10T04:39:28.271 INFO:tasks.rados.rados.0.smithi085.stdout:2936: writing smithi08575397-38 from 726333 to 1396366 tid 1 2021-02-10T04:39:28.280 INFO:tasks.rados.rados.0.smithi085.stdout:2936: writing smithi08575397-38 from 2096210 to 2820563 tid 2 2021-02-10T04:39:28.281 INFO:tasks.rados.rados.0.smithi085.stdout:2937: rollback oid 2 current snap is 335 2021-02-10T04:39:28.281 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 2 to 316 2021-02-10T04:39:28.281 INFO:tasks.rados.rados.0.smithi085.stdout:2934: finishing rollback tid 1 to smithi08575397-6 2021-02-10T04:39:28.281 INFO:tasks.rados.rados.0.smithi085.stdout:2931: finishing rollback tid 0 to smithi08575397-40 2021-02-10T04:39:28.281 INFO:tasks.rados.rados.0.smithi085.stdout:2931: finishing rollback tid 1 to smithi08575397-40 2021-02-10T04:39:28.281 INFO:tasks.rados.rados.0.smithi085.stdout:2934: finishing rollback tid 2 to smithi08575397-6 2021-02-10T04:39:28.281 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 1370 (ObjNum 729 snap 230 seq_num 729) dirty exists 2021-02-10T04:39:28.281 INFO:tasks.rados.rados.0.smithi085.stdout:2931: finishing rollback tid 2 to smithi08575397-40 2021-02-10T04:39:28.281 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 1533 (ObjNum 879 snap 282 seq_num 879) dirty exists 2021-02-10T04:39:28.281 INFO:tasks.rados.rados.0.smithi085.stdout:2933: expect (ObjNum 939 snap 311 seq_num 939) 2021-02-10T04:39:28.562 INFO:tasks.rados.rados.0.smithi085.stdout:2936: finishing write tid 1 to smithi08575397-38 2021-02-10T04:39:28.563 INFO:tasks.rados.rados.0.smithi085.stdout:2936: finishing write tid 2 to smithi08575397-38 2021-02-10T04:39:28.563 INFO:tasks.rados.rados.0.smithi085.stdout:2937: finishing rollback tid 1 to smithi08575397-2 2021-02-10T04:39:28.563 INFO:tasks.rados.rados.0.smithi085.stdout:2936: finishing write tid 3 to smithi08575397-38 2021-02-10T04:39:28.563 INFO:tasks.rados.rados.0.smithi085.stdout:2936: finishing write tid 4 to smithi08575397-38 2021-02-10T04:39:28.563 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 1373 (ObjNum 1019 snap 335 seq_num 1019) dirty exists 2021-02-10T04:39:28.563 INFO:tasks.rados.rados.0.smithi085.stdout:2936: left oid 38 (ObjNum 1019 snap 335 seq_num 1019) 2021-02-10T04:39:28.564 INFO:tasks.rados.rados.0.smithi085.stdout:2937: finishing rollback tid 2 to smithi08575397-2 2021-02-10T04:39:28.564 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 1442 (ObjNum 951 snap 313 seq_num 951) dirty exists 2021-02-10T04:39:28.564 INFO:tasks.rados.rados.0.smithi085.stdout:2931: done (8 left) 2021-02-10T04:39:28.564 INFO:tasks.rados.rados.0.smithi085.stdout:2933: done (7 left) 2021-02-10T04:39:28.564 INFO:tasks.rados.rados.0.smithi085.stdout:2934: done (6 left) 2021-02-10T04:39:28.564 INFO:tasks.rados.rados.0.smithi085.stdout:2936: done (5 left) 2021-02-10T04:39:28.565 INFO:tasks.rados.rados.0.smithi085.stdout:2937: done (4 left) 2021-02-10T04:39:28.565 INFO:tasks.rados.rados.0.smithi085.stdout:2938: snap_remove snap 322 2021-02-10T04:39:28.645 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:39:28 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:39:28.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:28 smithi164 conmon[29288]: cluster 2021-02-10T04:39:27.797154+0000 osd.4 (osd.4) 1233 : cluster [WRN] 2021-02-10T04:39:28.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:28 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:28.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:28 smithi164 conmon[29288]: cluster 2021-02-10T04:39:27.797166+0000 osd.4 (osd.4) 1234 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:28.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:28 smithi164 conmon[29288]: cluster 2021-02-10T04:39:27.797173+0000 osd.4 ( 2021-02-10T04:39:28.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:28 smithi164 conmon[29288]: osd.4) 1235 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:28.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:28 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:39:28.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:28 smithi164 conmon[29288]: 04:39:27.797180+0000 osd.4 (osd.4) 2021-02-10T04:39:28.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:28 smithi164 conmon[29288]: 1236 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:28.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:28 smithi164 conmon[29288]: cluster 2021-02-10T04:39: 2021-02-10T04:39:28.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:28 smithi164 conmon[29288]: 28.175778+0000 mgr.y (mgr.14140) 748 : cluster [DBG] 2021-02-10T04:39:28.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:28 smithi164 conmon[29288]: pgmap v1273: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 501 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 2.8 MiB/s rd, 1.0 MiB/s wr, 8 op/s 2021-02-10T04:39:28.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:28 smithi164 conmon[29288]: cluster 2021-02-10T04:39:28.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:28 smithi164 conmon[29288]: 2021-02-10T04:39:28.248279+0000 mon.a (mon 2021-02-10T04:39:28.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:28 smithi164 conmon[29288]: .0) 1668 : cluster [DBG] osdmap e749: 8 total, 8 up, 7 in 2021-02-10T04:39:28.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[35961]: cluster 2021-02-10T04:39:27.797154 2021-02-10T04:39:28.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1233 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:28.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[35961]: cluster 2021-02-10T04:39:27.797166+0000 osd.4 ( 2021-02-10T04:39:28.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[35961]: osd.4) 1234 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:28.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[35961]: 2021-02-10T04:39:28.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:39:28.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[35961]: T04:39:27.797173+0000 osd.4 (osd 2021-02-10T04:39:28.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[35961]: .4) 1235 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:28.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[35961]: cluster 2021-02-10T04:39:28.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[35961]: 2021-02-10T04:39:27.797180 2021-02-10T04:39:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1236 : cluster 2021-02-10T04:39:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:39:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[35961]: T04:39:28. 2021-02-10T04:39:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[35961]: 175778+0000 mgr.y (mgr.14140) 748 : cluster [DBG] pgmap v1273: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 501 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 2.8 MiB/s rd, 1.0 MiB/s wr, 8 op/s 2021-02-10T04:39:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:39:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[35961]: 02-10T04:39:28.248279+0000 2021-02-10T04:39:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[35961]: mon.a (mon.0) 1668 : cluster [DBG] 2021-02-10T04:39:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[35961]: osdmap e749: 8 total, 8 up, 7 in 2021-02-10T04:39:28.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[29319]: cluster 2021-02-10T04:39:27.797154+0000 2021-02-10T04:39:28.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[29319]: osd.4 (osd.4) 1233 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:28.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[29319]: cluster 2021-02-10T04:39:28.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[29319]: 2021-02-10T04:39:28.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[29319]: 2021 2021-02-10T04:39:28.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[29319]: - 2021-02-10T04:39:28.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[29319]: 02-10T04 2021-02-10T04:39:28.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[29319]: :39:27.797166+0000 osd.4 (osd.4) 1234 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:28.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[29319]: cluster 2021-02-10T04:39:28.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[29319]: 2021-02-10T04:39:28.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[29319]: 2021 2021-02-10T04:39:28.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[29319]: -02-10T04:39: 2021-02-10T04:39:28.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[29319]: 27.797173+0000 osd.4 (osd.4) 1235 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:28.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[29319]: cluster 2021-02-10T04:39:28.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:39:28.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[29319]: 04:39:27.797180+0000 osd.4 2021-02-10T04:39:28.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[29319]: (osd.4) 1236 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:28.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:39:28.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[29319]: -02-10T04:39:28.175778+0000 mgr.y 2021-02-10T04:39:28.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[29319]: (mgr.14140) 748 : cluster [DBG] pgmap v1273: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 501 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 2.8 MiB/s rd, 1.0 MiB/s wr, 8 op/s 2021-02-10T04:39:28.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[29319]: cluster 2021-02-10T04:39:28.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[29319]: 2021 2021-02-10T04:39:28.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[29319]: -02-10T04:39: 2021-02-10T04:39:28.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:28 smithi085 conmon[29319]: 28.248279+0000 mon.a (mon.0) 1668 : cluster [DBG] osdmap e749: 8 total, 8 up, 7 in 2021-02-10T04:39:29.109 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:28 smithi164 conmon[33677]: debug 2021-02-10T04:39:28.810+0000 7fbec19a1700 -1 osd.4 749 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:29.393 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:39:29 smithi164 conmon[37263]: debug 2021-02-10T04:39:29.107+0000 7f1c6b1ea700 -1 osd.5 pg_epoch: 749 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x556655936000) 2021-02-10T04:39:29.563 INFO:tasks.rados.rados.0.smithi085.stdout:2938: done (4 left) 2021-02-10T04:39:29.563 INFO:tasks.rados.rados.0.smithi085.stdout:2939: write (excl) oid 32 current snap is 335 2021-02-10T04:39:29.563 INFO:tasks.rados.rados.0.smithi085.stdout:2939: seq_num 1020 ranges {743140=745282,2154692=468011} 2021-02-10T04:39:29.572 INFO:tasks.rados.rados.0.smithi085.stdout:2939: writing smithi08575397-32 from 743140 to 1488422 tid 1 2021-02-10T04:39:29.580 INFO:tasks.rados.rados.0.smithi085.stdout:2939: writing smithi08575397-32 from 2154692 to 2622703 tid 2 2021-02-10T04:39:29.581 INFO:tasks.rados.rados.0.smithi085.stdout:2940: write oid 40 current snap is 335 2021-02-10T04:39:29.581 INFO:tasks.rados.rados.0.smithi085.stdout:2940: seq_num 1021 ranges {759947=420531,1813174=728022} 2021-02-10T04:39:29.587 INFO:tasks.rados.rados.0.smithi085.stdout:2940: writing smithi08575397-40 from 759947 to 1180478 tid 1 2021-02-10T04:39:29.596 INFO:tasks.rados.rados.0.smithi085.stdout:2940: writing smithi08575397-40 from 1813174 to 2541196 tid 2 2021-02-10T04:39:29.597 INFO:tasks.rados.rados.0.smithi085.stdout:2941: write (excl) oid 15 current snap is 335 2021-02-10T04:39:29.597 INFO:tasks.rados.rados.0.smithi085.stdout:2939: finishing write tid 1 to smithi08575397-32 2021-02-10T04:39:29.597 INFO:tasks.rados.rados.0.smithi085.stdout:2941: seq_num 1022 ranges {776754=495780,1871656=471680,2965828=210926} 2021-02-10T04:39:29.604 INFO:tasks.rados.rados.0.smithi085.stdout:2941: writing smithi08575397-15 from 776754 to 1272534 tid 1 2021-02-10T04:39:29.611 INFO:tasks.rados.rados.0.smithi085.stdout:2941: writing smithi08575397-15 from 1871656 to 2343336 tid 2 2021-02-10T04:39:29.618 INFO:tasks.rados.rados.0.smithi085.stdout:2941: writing smithi08575397-15 from 2965828 to 3176754 tid 3 2021-02-10T04:39:29.618 INFO:tasks.rados.rados.0.smithi085.stdout:2942: write (excl) oid 35 current snap is 335 2021-02-10T04:39:29.618 INFO:tasks.rados.rados.0.smithi085.stdout:2939: finishing write tid 2 to smithi08575397-32 2021-02-10T04:39:29.618 INFO:tasks.rados.rados.0.smithi085.stdout:2939: finishing write tid 3 to smithi08575397-32 2021-02-10T04:39:29.618 INFO:tasks.rados.rados.0.smithi085.stdout:2939: finishing write tid 4 to smithi08575397-32 2021-02-10T04:39:29.618 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 32 v 1360 (ObjNum 1020 snap 335 seq_num 1020) dirty exists 2021-02-10T04:39:29.618 INFO:tasks.rados.rados.0.smithi085.stdout:2939: left oid 32 (ObjNum 1020 snap 335 seq_num 1020) 2021-02-10T04:39:29.618 INFO:tasks.rados.rados.0.smithi085.stdout:2940: finishing write tid 1 to smithi08575397-40 2021-02-10T04:39:29.618 INFO:tasks.rados.rados.0.smithi085.stdout:2940: finishing write tid 2 to smithi08575397-40 2021-02-10T04:39:29.618 INFO:tasks.rados.rados.0.smithi085.stdout:2940: finishing write tid 3 to smithi08575397-40 2021-02-10T04:39:29.619 INFO:tasks.rados.rados.0.smithi085.stdout:2940: finishing write tid 4 to smithi08575397-40 2021-02-10T04:39:29.619 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 1540 (ObjNum 1021 snap 335 seq_num 1021) dirty exists 2021-02-10T04:39:29.619 INFO:tasks.rados.rados.0.smithi085.stdout:2940: left oid 40 (ObjNum 1021 snap 335 seq_num 1021) 2021-02-10T04:39:29.619 INFO:tasks.rados.rados.0.smithi085.stdout:2941: finishing write tid 1 to smithi08575397-15 2021-02-10T04:39:29.619 INFO:tasks.rados.rados.0.smithi085.stdout:2941: finishing write tid 2 to smithi08575397-15 2021-02-10T04:39:29.619 INFO:tasks.rados.rados.0.smithi085.stdout:2942: seq_num 1023 ranges {793561=571029,1813785=731691,2993251=200310} 2021-02-10T04:39:29.626 INFO:tasks.rados.rados.0.smithi085.stdout:2942: writing smithi08575397-35 from 793561 to 1364590 tid 1 2021-02-10T04:39:29.634 INFO:tasks.rados.rados.0.smithi085.stdout:2942: writing smithi08575397-35 from 1813785 to 2545476 tid 2 2021-02-10T04:39:29.639 INFO:tasks.rados.rados.0.smithi085.stdout:2942: writing smithi08575397-35 from 2993251 to 3193561 tid 3 2021-02-10T04:39:29.639 INFO:tasks.rados.rados.0.smithi085.stdout:2939: done (7 left) 2021-02-10T04:39:29.639 INFO:tasks.rados.rados.0.smithi085.stdout:2940: done (6 left) 2021-02-10T04:39:29.639 INFO:tasks.rados.rados.0.smithi085.stdout:2943: write oid 7 current snap is 335 2021-02-10T04:39:29.639 INFO:tasks.rados.rados.0.smithi085.stdout:2941: finishing write tid 3 to smithi08575397-15 2021-02-10T04:39:29.639 INFO:tasks.rados.rados.0.smithi085.stdout:2941: finishing write tid 4 to smithi08575397-15 2021-02-10T04:39:29.639 INFO:tasks.rados.rados.0.smithi085.stdout:2941: finishing write tid 5 to smithi08575397-15 2021-02-10T04:39:29.639 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 15 v 1364 (ObjNum 1022 snap 335 seq_num 1022) dirty exists 2021-02-10T04:39:29.639 INFO:tasks.rados.rados.0.smithi085.stdout:2941: left oid 15 (ObjNum 1022 snap 335 seq_num 1022) 2021-02-10T04:39:29.639 INFO:tasks.rados.rados.0.smithi085.stdout:2942: finishing write tid 1 to smithi08575397-35 2021-02-10T04:39:29.640 INFO:tasks.rados.rados.0.smithi085.stdout:2942: finishing write tid 2 to smithi08575397-35 2021-02-10T04:39:29.640 INFO:tasks.rados.rados.0.smithi085.stdout:2943: seq_num 1024 ranges {410368=646278,1472267=475349,2504321=405600} 2021-02-10T04:39:29.645 INFO:tasks.rados.rados.0.smithi085.stdout:2943: writing smithi08575397-7 from 410368 to 1056646 tid 1 2021-02-10T04:39:29.651 INFO:tasks.rados.rados.0.smithi085.stdout:2943: writing smithi08575397-7 from 1472267 to 1947616 tid 2 2021-02-10T04:39:29.658 INFO:tasks.rados.rados.0.smithi085.stdout:2943: writing smithi08575397-7 from 2504321 to 2909921 tid 3 2021-02-10T04:39:29.658 INFO:tasks.rados.rados.0.smithi085.stdout:2941: done (6 left) 2021-02-10T04:39:29.658 INFO:tasks.rados.rados.0.smithi085.stdout:2944: read oid 5 snap -1 2021-02-10T04:39:29.659 INFO:tasks.rados.rados.0.smithi085.stdout:2944: expect deleted 2021-02-10T04:39:29.659 INFO:tasks.rados.rados.0.smithi085.stdout:2945: write (excl) oid 30 current snap is 335 2021-02-10T04:39:29.659 INFO:tasks.rados.rados.0.smithi085.stdout:2942: finishing write tid 3 to smithi08575397-35 2021-02-10T04:39:29.659 INFO:tasks.rados.rados.0.smithi085.stdout:2942: finishing write tid 4 to smithi08575397-35 2021-02-10T04:39:29.659 INFO:tasks.rados.rados.0.smithi085.stdout:2942: finishing write tid 5 to smithi08575397-35 2021-02-10T04:39:29.659 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 35 v 1446 (ObjNum 1023 snap 335 seq_num 1023) dirty exists 2021-02-10T04:39:29.659 INFO:tasks.rados.rados.0.smithi085.stdout:2942: left oid 35 (ObjNum 1023 snap 335 seq_num 1023) 2021-02-10T04:39:29.659 INFO:tasks.rados.rados.0.smithi085.stdout:2943: finishing write tid 1 to smithi08575397-7 2021-02-10T04:39:29.659 INFO:tasks.rados.rados.0.smithi085.stdout:2943: finishing write tid 2 to smithi08575397-7 2021-02-10T04:39:29.659 INFO:tasks.rados.rados.0.smithi085.stdout:2945: seq_num 1025 ranges {427175=721527,1930749=735360} 2021-02-10T04:39:29.666 INFO:tasks.rados.rados.0.smithi085.stdout:2945: writing smithi08575397-30 from 427175 to 1148702 tid 1 2021-02-10T04:39:29.676 INFO:tasks.rados.rados.0.smithi085.stdout:2945: writing smithi08575397-30 from 1930749 to 2666109 tid 2 2021-02-10T04:39:29.676 INFO:tasks.rados.rados.0.smithi085.stdout:2942: done (7 left) 2021-02-10T04:39:29.676 INFO:tasks.rados.rados.0.smithi085.stdout:2946: delete oid 48 current snap is 335 2021-02-10T04:39:29.677 INFO:tasks.rados.rados.0.smithi085.stdout:2943: finishing write tid 3 to smithi08575397-7 2021-02-10T04:39:29.677 INFO:tasks.rados.rados.0.smithi085.stdout:2943: finishing write tid 4 to smithi08575397-7 2021-02-10T04:39:29.677 INFO:tasks.rados.rados.0.smithi085.stdout:2943: finishing write tid 5 to smithi08575397-7 2021-02-10T04:39:29.677 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 1362 (ObjNum 1024 snap 335 seq_num 1024) dirty exists 2021-02-10T04:39:29.677 INFO:tasks.rados.rados.0.smithi085.stdout:2943: left oid 7 (ObjNum 1024 snap 335 seq_num 1024) 2021-02-10T04:39:29.677 INFO:tasks.rados.rados.0.smithi085.stdout:2945: finishing write tid 1 to smithi08575397-30 2021-02-10T04:39:29.682 INFO:tasks.rados.rados.0.smithi085.stdout:2943: done (7 left) 2021-02-10T04:39:29.682 INFO:tasks.rados.rados.0.smithi085.stdout:2944: done (6 left) 2021-02-10T04:39:29.682 INFO:tasks.rados.rados.0.smithi085.stdout:2946: done (5 left) 2021-02-10T04:39:29.682 INFO:tasks.rados.rados.0.smithi085.stdout:2947: copy_from oid 26 from oid 3 current snap is 335 2021-02-10T04:39:29.682 INFO:tasks.rados.rados.0.smithi085.stdout:2948: write (excl) oid 10 current snap is 335 2021-02-10T04:39:29.682 INFO:tasks.rados.rados.0.smithi085.stdout:2948: seq_num 1027 ranges {460789=588378,1764066=739029,3219296=41493} 2021-02-10T04:39:29.688 INFO:tasks.rados.rados.0.smithi085.stdout:2948: writing smithi08575397-10 from 460789 to 1049167 tid 1 2021-02-10T04:39:29.698 INFO:tasks.rados.rados.0.smithi085.stdout:2948: writing smithi08575397-10 from 1764066 to 2503095 tid 2 2021-02-10T04:39:29.703 INFO:tasks.rados.rados.0.smithi085.stdout:2948: writing smithi08575397-10 from 3219296 to 3260789 tid 3 2021-02-10T04:39:29.703 INFO:tasks.rados.rados.0.smithi085.stdout:2949: read oid 49 snap 291 2021-02-10T04:39:29.703 INFO:tasks.rados.rados.0.smithi085.stdout:2949: expect (ObjNum 172 snap 37 seq_num 172) 2021-02-10T04:39:29.703 INFO:tasks.rados.rados.0.smithi085.stdout:2950: snap_remove snap 334 2021-02-10T04:39:29.849 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:29 smithi164 conmon[29288]: audit 2021-02-10T04:39:29.850 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:29 smithi164 conmon[29288]: 2021-02-10T04:39:29.850 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:29 smithi164 conmon[29288]: 2021-02-10T04:39:28.335797+0000 2021-02-10T04:39:29.850 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:29 smithi164 conmon[29288]: mgr.y (mgr.14140) 749 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:39:29.850 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:29 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:39:29.850 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:29 smithi164 conmon[29288]: -02-10T04:39:28.543804+0000 mon.a (mon.0) 1669 : 2021-02-10T04:39:29.850 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:29 smithi164 conmon[29288]: cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 597 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:39:29.850 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:29 smithi164 conmon[29288]: cluster 2021-02-10T04:39:29.850 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:29 smithi164 conmon[29288]: 2021-02-10T04:39:28.811120 2021-02-10T04:39:29.850 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:29 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1237 : cluster 2021-02-10T04:39:29.850 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:29 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:29.850 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:29 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:39:29.851 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:29 smithi164 conmon[29288]: 39:28.811131+0000 osd.4 (osd.4) 1238 2021-02-10T04:39:29.851 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:29 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:29.851 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:29 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:39:29.851 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:29 smithi164 conmon[29288]: :39:28.811138+0000 osd.4 (osd.4 2021-02-10T04:39:29.851 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:29 smithi164 conmon[29288]: ) 1239 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:29.851 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:29 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:39:29.851 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:29 smithi164 conmon[29288]: T04:39:28.811145+0000 osd.4 (osd 2021-02-10T04:39:29.851 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:29 smithi164 conmon[29288]: .4) 1240 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:29.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:29 smithi085 conmon[35961]: audit 2021-02-10T04:39:28.335797+0000 2021-02-10T04:39:29.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:29 smithi085 conmon[35961]: mgr.y (mgr.14140) 749 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:39:29.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:29 smithi085 conmon[35961]: cluster 2021-02-10T04:39:28.543804+0000 2021-02-10T04:39:29.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:29 smithi085 conmon[35961]: mon.a (mon.0) 1669 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 597 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:39:29.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:29 smithi085 conmon[35961]: cluster 2021-02-10T04:39: 2021-02-10T04:39:29.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:29 smithi085 conmon[35961]: 28.811120+0000 osd.4 (osd.4) 1237 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:29.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:29 smithi085 conmon[35961]: cluster 2021-02-10T04:39:28.811131+0000 osd.4 (osd.4) 1238 : 2021-02-10T04:39:29.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:29 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:29.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:29 smithi085 conmon[35961]: cluster 2021-02-10T04:39:28 2021-02-10T04:39:29.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:29 smithi085 conmon[35961]: .811138+0000 osd.4 (osd.4) 1239 : cluster [WRN] 2021-02-10T04:39:29.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:29 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:29.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:29 smithi085 conmon[35961]: cluster 2021-02-10T04:39:29.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:29 smithi085 conmon[35961]: 2021-02-10T04:39:28.811145+0000 osd.4 (osd 2021-02-10T04:39:29.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:29 smithi085 conmon[35961]: .4) 1240 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:29 smithi085 conmon[29319]: audit 2021-02-10T04:39:28.335797 2021-02-10T04:39:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:29 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 749 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:39:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:29 smithi085 conmon[29319]: cluster 2021-02-10T04:39:28.543804+0000 mon.a (mon.0) 1669 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 597 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:39:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:29 smithi085 conmon[29319]: cluster 2021-02-10T04:39:28.811120+0000 osd.4 (osd.4) 1237 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:29 smithi085 conmon[29319]: cluster 2021-02-10T04:39 2021-02-10T04:39:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:29 smithi085 conmon[29319]: :28.811131+0000 osd.4 (osd.4) 1238 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:29 smithi085 conmon[29319]: cluster 2021-02-10T04:39:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:29 smithi085 conmon[29319]: 2021-02-10T04:39:28.811138+0000 osd.4 (osd.4) 1239 : cluster [WRN] 2021-02-10T04:39:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:29 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:29.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:29 smithi085 conmon[29319]: cluster 2021-02-10T04:39:28.811145+0000 osd.4 (osd.4) 1240 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:30.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:29 smithi164 conmon[33677]: debug 2021-02-10T04:39:30.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:29 smithi164 conmon[33677]: 2021-02-10T04:39:29.845+0000 7fbec19a1700 -1 osd.4 750 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:30.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:29 smithi164 conmon[33677]: 2021-02-10T04:39:30.577 INFO:tasks.rados.rados.0.smithi085.stdout:2950: done (8 left) 2021-02-10T04:39:30.577 INFO:tasks.rados.rados.0.smithi085.stdout:2951: rollback oid 18 current snap is 335 2021-02-10T04:39:30.578 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 18 to 321 2021-02-10T04:39:30.578 INFO:tasks.rados.rados.0.smithi085.stdout:2945: finishing write tid 2 to smithi08575397-30 2021-02-10T04:39:30.578 INFO:tasks.rados.rados.0.smithi085.stdout:2945: finishing write tid 3 to smithi08575397-30 2021-02-10T04:39:30.578 INFO:tasks.rados.rados.0.smithi085.stdout:2945: finishing write tid 4 to smithi08575397-30 2021-02-10T04:39:30.578 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 1376 (ObjNum 1025 snap 335 seq_num 1025) dirty exists 2021-02-10T04:39:30.578 INFO:tasks.rados.rados.0.smithi085.stdout:2945: left oid 30 (ObjNum 1025 snap 335 seq_num 1025) 2021-02-10T04:39:30.578 INFO:tasks.rados.rados.0.smithi085.stdout:2947: finishing copy_from to smithi08575397-26 2021-02-10T04:39:30.578 INFO:tasks.rados.rados.0.smithi085.stdout:2947: got expected ENOENT (src dne) 2021-02-10T04:39:30.578 INFO:tasks.rados.rados.0.smithi085.stdout:2947: finishing copy_from racing read to smithi08575397-26 2021-02-10T04:39:30.578 INFO:tasks.rados.rados.0.smithi085.stdout:2948: finishing write tid 1 to smithi08575397-10 2021-02-10T04:39:30.578 INFO:tasks.rados.rados.0.smithi085.stdout:2945: done (8 left) 2021-02-10T04:39:30.579 INFO:tasks.rados.rados.0.smithi085.stdout:2947: done (7 left) 2021-02-10T04:39:30.579 INFO:tasks.rados.rados.0.smithi085.stdout:2952: write (excl) oid 6 current snap is 335 2021-02-10T04:39:30.579 INFO:tasks.rados.rados.0.smithi085.stdout:2948: finishing write tid 2 to smithi08575397-10 2021-02-10T04:39:30.579 INFO:tasks.rados.rados.0.smithi085.stdout:2948: finishing write tid 3 to smithi08575397-10 2021-02-10T04:39:30.579 INFO:tasks.rados.rados.0.smithi085.stdout:2948: finishing write tid 4 to smithi08575397-10 2021-02-10T04:39:30.579 INFO:tasks.rados.rados.0.smithi085.stdout:2952: seq_num 1028 ranges {477596=663627,1706195=482687,2614013=567041} 2021-02-10T04:39:30.584 INFO:tasks.rados.rados.0.smithi085.stdout:2952: writing smithi08575397-6 from 477596 to 1141223 tid 1 2021-02-10T04:39:30.591 INFO:tasks.rados.rados.0.smithi085.stdout:2952: writing smithi08575397-6 from 1706195 to 2188882 tid 2 2021-02-10T04:39:30.598 INFO:tasks.rados.rados.0.smithi085.stdout:2952: writing smithi08575397-6 from 2614013 to 3181054 tid 3 2021-02-10T04:39:30.598 INFO:tasks.rados.rados.0.smithi085.stdout:2953: rollback oid 29 current snap is 335 2021-02-10T04:39:30.598 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 29 to 331 2021-02-10T04:39:30.598 INFO:tasks.rados.rados.0.smithi085.stdout:2954: snap_create 2021-02-10T04:39:30.598 INFO:tasks.rados.rados.0.smithi085.stdout:2948: finishing write tid 5 to smithi08575397-10 2021-02-10T04:39:30.598 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 10 v 1450 (ObjNum 1027 snap 335 seq_num 1027) dirty exists 2021-02-10T04:39:30.598 INFO:tasks.rados.rados.0.smithi085.stdout:2948: left oid 10 (ObjNum 1027 snap 335 seq_num 1027) 2021-02-10T04:39:30.599 INFO:tasks.rados.rados.0.smithi085.stdout:2949: expect (ObjNum 172 snap 37 seq_num 172) 2021-02-10T04:39:30.721 INFO:tasks.rados.rados.0.smithi085.stdout:2951: finishing rollback tid 0 to smithi08575397-18 2021-02-10T04:39:30.722 INFO:tasks.rados.rados.0.smithi085.stdout:2951: finishing rollback tid 1 to smithi08575397-18 2021-02-10T04:39:30.722 INFO:tasks.rados.rados.0.smithi085.stdout:2951: finishing rollback tid 2 to smithi08575397-18 2021-02-10T04:39:30.722 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 18 v 1296 (ObjNum 952 snap 313 seq_num 952) dirty exists 2021-02-10T04:39:30.722 INFO:tasks.rados.rados.0.smithi085.stdout:2952: finishing write tid 1 to smithi08575397-6 2021-02-10T04:39:30.722 INFO:tasks.rados.rados.0.smithi085.stdout:2952: finishing write tid 2 to smithi08575397-6 2021-02-10T04:39:30.722 INFO:tasks.rados.rados.0.smithi085.stdout:2953: finishing rollback tid 0 to smithi08575397-29 2021-02-10T04:39:30.722 INFO:tasks.rados.rados.0.smithi085.stdout:2953: finishing rollback tid 1 to smithi08575397-29 2021-02-10T04:39:30.722 INFO:tasks.rados.rados.0.smithi085.stdout:2953: finishing rollback tid 2 to smithi08575397-29 2021-02-10T04:39:30.722 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 29 v 1367 (ObjNum 939 snap 311 seq_num 939) dirty exists 2021-02-10T04:39:30.722 INFO:tasks.rados.rados.0.smithi085.stdout:2952: finishing write tid 3 to smithi08575397-6 2021-02-10T04:39:30.723 INFO:tasks.rados.rados.0.smithi085.stdout:2952: finishing write tid 4 to smithi08575397-6 2021-02-10T04:39:30.723 INFO:tasks.rados.rados.0.smithi085.stdout:2952: finishing write tid 5 to smithi08575397-6 2021-02-10T04:39:30.723 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 1380 (ObjNum 1028 snap 335 seq_num 1028) dirty exists 2021-02-10T04:39:30.723 INFO:tasks.rados.rados.0.smithi085.stdout:2952: left oid 6 (ObjNum 1028 snap 335 seq_num 1028) 2021-02-10T04:39:30.881 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:30 smithi164 conmon[29288]: cluster 2021-02-10T04:39:29.563550+0000 mon.a (mon.0 2021-02-10T04:39:30.881 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:30 smithi164 conmon[29288]: ) 1670 : cluster [DBG] osdmap e750: 8 total, 8 up, 7 in 2021-02-10T04:39:30.881 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:30 smithi164 conmon[29288]: cluster 2021-02-10T04:39:29.846399+0000 osd.4 (osd.4) 1241 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:30.882 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:30 smithi164 conmon[29288]: cluster 2021-02-10T04:39:29.846409+0000 osd.4 (osd.4) 1242 : cluster 2021-02-10T04:39:30.882 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:30 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:30.882 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:30 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:39:30.882 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:30 smithi164 conmon[29288]: -02-10T04:39:29.846416+0000 osd.4 (osd.4) 1243 : cluster 2021-02-10T04:39:30.882 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:30 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:30.882 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:30 smithi164 conmon[29288]: cluster 2021-02-10T04:39:30.882 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:30 smithi164 conmon[29288]: 2021-02-10T04:39:29.846422+0000 osd.4 (osd.4) 1244 : 2021-02-10T04:39:30.882 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:30 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:30.882 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:30 smithi164 conmon[29288]: cluster 2021-02-10T04:39:30. 2021-02-10T04:39:30.882 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:30 smithi164 conmon[29288]: 176508+0000 mgr.y (mgr.14140) 750 : cluster [DBG] pgmap v1276: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 515 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 12 op/s 2021-02-10T04:39:30.882 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:30 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:39:30.882 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:30 smithi164 conmon[29288]: 39:30.429304+0000 osd.3 (osd.3) 82 : cluster 2021-02-10T04:39:30.883 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:30 smithi164 conmon[29288]: [DBG] purged_snaps scrub starts 2021-02-10T04:39:30.883 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:30 smithi164 conmon[29288]: cluster 2021-02-10T04:39:30. 2021-02-10T04:39:30.883 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:30 smithi164 conmon[29288]: 430597+0000 osd.3 (osd.3) 83 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:39:30.883 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:30 smithi164 conmon[29288]: 2021-02-10T04:39:30.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: cluster 2021-02-10T04:39:30.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: 2021-02-10T04:39:30.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: 2021 2021-02-10T04:39:30.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: - 2021-02-10T04:39:30.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: 02-10T04:39:29. 2021-02-10T04:39:30.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: 563550+0000 mon.a (mon.0) 1670 : cluster [DBG] osdmap e750: 8 total, 8 up, 7 in 2021-02-10T04:39:30.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: cluster 2021-02-10T04:39:30.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: 2021-02-10T04:39:29. 2021-02-10T04:39:30.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: 846399+0000 osd.4 (osd.4) 2021-02-10T04:39:30.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: 1241 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:30.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:39:30.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: -10T04:39:29.846409+0000 osd.4 (osd 2021-02-10T04:39:30.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: .4) 1242 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:30.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: cluster 2021-02-10T04:39:30.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: 2021-02-10T04:39 2021-02-10T04:39:30.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: :29.846416+0000 osd.4 (osd.4) 2021-02-10T04:39:30.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: 1243 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:30.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:39:30.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: -02-10T04:39:29.846422 2021-02-10T04:39:30.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1244 : cluster 2021-02-10T04:39:30.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:30.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:39:30.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: T04:39:30.176508+0000 mgr.y (mgr 2021-02-10T04:39:30.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: .14140) 750 : cluster [DBG] pgmap v1276: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 515 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 12 op/s 2021-02-10T04:39:30.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: cluster 2021-02-10T04:39:30.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:39:30.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: 39:30.429304+0000 osd.3 (osd.3) 82 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:39:30.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: cluster 2021-02-10T04:39:30.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:39:30.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: :39:30.430597+0000 osd.3 (osd.3) 83 : cluster [DBG] 2021-02-10T04:39:30.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:39:30.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[29319]: cluster 2021-02-10T04:39:30.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[29319]: 2021-02-10T04:39:30.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[29319]: 2021-02-10T04:39:29.563550+0000 2021-02-10T04:39:30.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[29319]: mon.a (mon.0) 1670 : cluster [DBG] osdmap e750: 8 total, 8 up, 7 in 2021-02-10T04:39:30.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:39:30.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[29319]: 39:29.846399+0000 osd.4 (osd.4) 1241 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:30.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[29319]: cluster 2021-02-10T04:39:30.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[29319]: 2021-02-10T04:39:29.846409 2021-02-10T04:39:30.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1242 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:30.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[29319]: cluster 2021-02-10T04:39: 2021-02-10T04:39:30.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[29319]: 29.846416+0000 osd.4 (osd.4) 1243 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:30.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[29319]: cluster 2021-02-10T04:39:30.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:39:30.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[29319]: 04:39:29.846422+0000 osd.4 (osd.4) 1244 2021-02-10T04:39:30.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:30.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[29319]: cluster 2021-02-10T04:39:30.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[29319]: 2021-02-10T04:39:30.176508 2021-02-10T04:39:30.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 750 : cluster 2021-02-10T04:39:30.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[29319]: [DBG] pgmap v1276: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 515 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 12 op/s 2021-02-10T04:39:30.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:39:30.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[29319]: 04:39:30.429304+0000 osd.3 (osd.3 2021-02-10T04:39:30.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[29319]: ) 82 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:39:30.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:39:30.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[29319]: 02-10T04:39:30.430597+0000 2021-02-10T04:39:30.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[29319]: osd.3 (osd.3) 83 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:39:30.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:30 smithi085 conmon[29319]: 2021-02-10T04:39:31.143 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:39:31 smithi164 conmon[37263]: debug 2021-02-10T04:39:31.119+0000 7f1c6b1ea700 -1 osd.5 pg_epoch: 751 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55665699f600) 2021-02-10T04:39:31.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:30 smithi164 conmon[33677]: debug 2021-02-10T04:39:30.878+0000 7fbec19a1700 -1 osd.4 751 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:31.487 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:39:31.583 INFO:tasks.rados.rados.0.smithi085.stdout:2948: done (9 left) 2021-02-10T04:39:31.584 INFO:tasks.rados.rados.0.smithi085.stdout:2949: done (8 left) 2021-02-10T04:39:31.584 INFO:tasks.rados.rados.0.smithi085.stdout:2951: done (7 left) 2021-02-10T04:39:31.584 INFO:tasks.rados.rados.0.smithi085.stdout:2952: done (6 left) 2021-02-10T04:39:31.584 INFO:tasks.rados.rados.0.smithi085.stdout:2953: done (5 left) 2021-02-10T04:39:31.584 INFO:tasks.rados.rados.0.smithi085.stdout:2954: done (4 left) 2021-02-10T04:39:31.584 INFO:tasks.rados.rados.0.smithi085.stdout:2955: read oid 48 snap -1 2021-02-10T04:39:31.584 INFO:tasks.rados.rados.0.smithi085.stdout:2955: expect deleted 2021-02-10T04:39:31.584 INFO:tasks.rados.rados.0.smithi085.stdout:2956: delete oid 5 current snap is 336 2021-02-10T04:39:31.591 INFO:tasks.rados.rados.0.smithi085.stdout:2955: done (5 left) 2021-02-10T04:39:31.591 INFO:tasks.rados.rados.0.smithi085.stdout:2956: done (4 left) 2021-02-10T04:39:31.591 INFO:tasks.rados.rados.0.smithi085.stdout:2957: rollback oid 32 current snap is 336 2021-02-10T04:39:31.591 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 32 to 310 2021-02-10T04:39:31.591 INFO:tasks.rados.rados.0.smithi085.stdout:2958: snap_create 2021-02-10T04:39:31.595 INFO:tasks.rados.rados.0.smithi085.stdout:2957: finishing rollback tid 0 to smithi08575397-32 2021-02-10T04:39:31.595 INFO:tasks.rados.rados.0.smithi085.stdout:2957: finishing rollback tid 1 to smithi08575397-32 2021-02-10T04:39:31.596 INFO:tasks.rados.rados.0.smithi085.stdout:2957: finishing rollback tid 2 to smithi08575397-32 2021-02-10T04:39:31.596 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 32 v 1370 (ObjNum 935 snap 310 seq_num 935) dirty exists 2021-02-10T04:39:31.690 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:31 smithi085 conmon[35961]: cluster 2021-02-10T04:39:31.690 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:31 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:39:31.690 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:31 smithi085 conmon[35961]: 10T04:39:30.575365+0000 mon.a (mon.0) 1671 : cluster [DBG] osdmap e751: 8 total, 8 up, 7 in 2021-02-10T04:39:31.690 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:31 smithi085 conmon[35961]: cluster 2021-02-10T04:39:31.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:31 smithi085 conmon[35961]: 2021-02 2021-02-10T04:39:31.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:31 smithi085 conmon[35961]: -10T04:39:30.879451+0000 osd.4 (osd.4 2021-02-10T04:39:31.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:31 smithi085 conmon[35961]: ) 1245 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:31.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:31 smithi085 conmon[35961]: cluster 2021-02-10T04:39:31.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:31 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:39:31.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:31 smithi085 conmon[35961]: T04:39:30.879460+0000 osd.4 (osd.4) 1246 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:31.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:31 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:39:31.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:31 smithi085 conmon[35961]: -02-10T04:39:30.879468+0000 osd.4 (osd.4) 1247 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:31.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:31 smithi085 conmon[35961]: cluster 2021-02-10T04:39:30.879474 2021-02-10T04:39:31.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:31 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1248 : cluster [WRN] 2021-02-10T04:39:31.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:31 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:31 smithi085 conmon[29319]: cluster 2021-02-10T04:39:30.575365+0000 mon.a (mon.0) 1671 : cluster [DBG] osdmap e751: 8 total, 8 up, 7 in 2021-02-10T04:39:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:31 smithi085 conmon[29319]: cluster 2021-02-10T04:39:30.879451+0000 osd.4 (osd.4) 1245 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:31 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:39:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:31 smithi085 conmon[29319]: 10T04:39:30.879460+0000 osd.4 (osd. 2021-02-10T04:39:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:31 smithi085 conmon[29319]: 4) 1246 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:31 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:39:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:31 smithi085 conmon[29319]: 02-10T04:39:30.879468+0000 osd.4 2021-02-10T04:39:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:31 smithi085 conmon[29319]: (osd.4) 1247 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:31 smithi085 conmon[29319]: 2021-02-10T04:39:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:31 smithi085 conmon[29319]: cluster 2021-02-10T04:39:30.879474+0000 2021-02-10T04:39:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:31 smithi085 conmon[29319]: osd.4 (osd.4) 1248 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:31.763 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:39:31.826 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:31 smithi164 conmon[29288]: cluster 2021-02-10T04:39:30.575365+0000 mon.a ( 2021-02-10T04:39:31.826 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:31 smithi164 conmon[29288]: mon.0) 1671 : cluster [DBG] osdmap e751: 8 total, 8 up, 7 in 2021-02-10T04:39:31.826 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:31 smithi164 conmon[29288]: cluster 2021-02-10T04:39:30 2021-02-10T04:39:31.826 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:31 smithi164 conmon[29288]: .879451+0000 osd.4 (osd.4) 1245 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:31.826 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:31 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:39:31.827 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:31 smithi164 conmon[29288]: -02-10T04:39:30.879460+0000 2021-02-10T04:39:31.827 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:31 smithi164 conmon[29288]: osd.4 (osd.4) 1246 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:31.827 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:31 smithi164 conmon[29288]: cluster 2021-02-10T04:39:31.827 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:31 smithi164 conmon[29288]: 2021-02-10T04:39:30.879468+0000 2021-02-10T04:39:31.827 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:31 smithi164 conmon[29288]: osd.4 (osd.4) 1247 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:31.827 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:31 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:39:31.827 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:31 smithi164 conmon[29288]: 02-10T04:39:30.879474+0000 osd.4 ( 2021-02-10T04:39:31.827 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:31 smithi164 conmon[29288]: osd.4) 1248 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:32.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:39:31 smithi164 conmon[33677]: debug 2021-02-10T04:39:31.912+0000 7fbec19a1700 -1 osd.4 752 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:39:32.144 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:39:31 smithi164 conmon[40892]: debug 2021-02-10T04:39:31.825+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:32.381 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:39:32.487 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:39:32.545 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:39:32.594 INFO:tasks.rados.rados.0.smithi085.stdout:2957: done (5 left) 2021-02-10T04:39:32.594 INFO:tasks.rados.rados.0.smithi085.stdout:2958: done (4 left) 2021-02-10T04:39:32.594 INFO:tasks.rados.rados.0.smithi085.stdout:2959: read oid 23 snap -1 2021-02-10T04:39:32.594 INFO:tasks.rados.rados.0.smithi085.stdout:2959: expect (ObjNum 924 snap 306 seq_num 924) 2021-02-10T04:39:32.594 INFO:tasks.rados.rados.0.smithi085.stdout:2960: snap_create 2021-02-10T04:39:32.602 INFO:tasks.rados.rados.0.smithi085.stdout:2959: expect (ObjNum 924 snap 306 seq_num 924) 2021-02-10T04:39:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:32 smithi164 conmon[29288]: cluster 2021-02-10T04:39:37.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:32 smithi164 conmon[29288]: 2021-02-10T04:39:31 2021-02-10T04:39:37.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:32 smithi164 conmon[29288]: .579294+0000 mon.a (mon.0) 1672 : cluster [DBG] osdmap e752: 8 total, 8 up, 7 in 2021-02-10T04:39:37.607 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:32 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:39:37.607 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:32 smithi164 conmon[29288]: -02-10T04:39:31.913471+0000 osd.4 (osd.4) 1249 : cluster [WRN] 2021-02-10T04:39:37.607 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:32 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:37.607 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:32 smithi164 conmon[29288]: cluster 2021-02-10T04:39:37.607 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:32 smithi164 conmon[29288]: 2021-02-10T04:39:31.913481+0000 osd.4 (osd. 2021-02-10T04:39:37.607 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:32 smithi164 conmon[29288]: 4) 1250 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:37.607 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:32 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:39:37.607 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:32 smithi164 conmon[29288]: -10T04:39:31.913488+0000 osd.4 (osd. 2021-02-10T04:39:37.607 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:32 smithi164 conmon[29288]: 4) 1251 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:37.607 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:32 smithi164 conmon[29288]: cluster 2021-02-10T04:39:31 2021-02-10T04:39:37.607 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:32 smithi164 conmon[29288]: .913496+0000 osd.4 (osd.4) 1252 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:37.607 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:32 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:39:37.608 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:32 smithi164 conmon[29288]: 10T04:39:32.177080+0000 mgr.y (mgr.14140) 751 2021-02-10T04:39:37.608 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:32 smithi164 conmon[29288]: : cluster [DBG] pgmap v1279: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 515 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 12 op/s 2021-02-10T04:39:37.612 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:39:37.612 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:39:37.612 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:39:37.612 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:39:37.613 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:39:37.613 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:39:37.613 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:39:37.613 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:39:37.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:32 smithi085 conmon[29319]: cluster 2021-02-10T04:39:31 2021-02-10T04:39:37.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:32 smithi085 conmon[29319]: .579294+0000 mon.a (mon.0) 1672 : cluster [DBG] osdmap e752: 8 total, 8 up, 7 in 2021-02-10T04:39:37.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:32 smithi085 conmon[29319]: cluster 2021-02-10T04:39:31.913471+0000 osd.4 (osd.4) 1249 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:37.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:32 smithi085 conmon[29319]: cluster 2021-02-10T04:39:31.913481+0000 osd.4 (osd.4) 1250 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:37.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:32 smithi085 conmon[29319]: cluster 2021-02-10T04:39:31.913488+0000 osd.4 (osd.4) 1251 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:37.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:32 smithi085 conmon[29319]: cluster 2021-02-10T04:39:31.913496+0000 osd.4 (osd.4) 1252 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:37.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:32 smithi085 conmon[29319]: cluster 2021-02-10T04:39:32.177080+0000 mgr.y (mgr.14140) 751 : cluster [DBG] pgmap v1279: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 515 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 12 op/s 2021-02-10T04:39:37.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:33 smithi085 conmon[29319]: cluster 2021-02-10T04:39:32.591600+0000 mon.a (mon.0) 1673 : 2021-02-10T04:39:37.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:33 smithi085 conmon[29319]: cluster [DBG] osdmap e753: 8 total, 8 up, 7 in 2021-02-10T04:39:37.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:33 smithi085 conmon[29319]: audit 2021-02-10T04:39:32.681762+0000 mon.b (mon.2) 39 : audit [DBG] from='client.? 172.21.15.85:0/3369693159' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:39:37.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:33 smithi085 conmon[29319]: audit 2021-02-10T04:39:33.005904+0000 mon.a (mon.0) 1674 : audit [INF] from='client.? 172.21.15.85:0/2859887596' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:39:37.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[29319]: cluster 2021-02-10T04:39:33.582793+0000 mon.a ( 2021-02-10T04:39:37.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[29319]: mon.0) 1675 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:39:37.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[29319]: audit 2021-02-10T04:39:33.597648+0000 mon.a (mon.0 2021-02-10T04:39:37.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[29319]: ) 1676 : audit 2021-02-10T04:39:37.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[29319]: [INF] from='client.? 172.21.15.85:0/2859887596' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:39:37.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:39:37.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[29319]: -10T04:39:33.597753+0000 mon.a (mon.0) 1677 : cluster [DBG] osdmap e754: 8 total, 8 up, 7 in 2021-02-10T04:39:37.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[29319]: cluster 2021-02-10T04:39: 2021-02-10T04:39:37.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[29319]: 33.901985+0000 osd.0 (osd.0) 87 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:39:37.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[29319]: cluster 2021-02-10T04:39:37.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[29319]: 2021-02-10T04:39:33.903511+0000 osd.0 (osd. 2021-02-10T04:39:37.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[29319]: 0) 88 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:39:37.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:39:37.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[29319]: 02-10T04:39:34.177728+0000 2021-02-10T04:39:37.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[29319]: mgr.y (mgr.14140) 752 : 2021-02-10T04:39:37.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[29319]: cluster [DBG] pgmap v1282: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 521 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 865 KiB/s wr, 8 op/s 2021-02-10T04:39:37.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:35 smithi085 conmon[29319]: cluster 2021-02-10T04:39:34.601878+0000 mon.a (mon. 2021-02-10T04:39:37.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:35 smithi085 conmon[29319]: 0) 1678 : cluster [DBG] osdmap e755: 8 total, 8 up, 7 in 2021-02-10T04:39:37.616 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:36 smithi085 conmon[29319]: cluster 2021-02-10T04:39:35.603600+0000 mon.a (mon.0) 1679 : cluster 2021-02-10T04:39:37.616 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:36 smithi085 conmon[29319]: [DBG] osdmap e756: 8 total, 8 up, 7 in 2021-02-10T04:39:37.616 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:36 smithi085 conmon[29319]: cluster 2021-02-10T04:39:36.178171+0000 mgr.y (mgr 2021-02-10T04:39:37.616 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:36 smithi085 conmon[29319]: .14140) 753 : cluster [DBG] pgmap v1285: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 518 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 6.9 MiB/s rd, 1.3 MiB/s wr, 12 op/s 2021-02-10T04:39:37.616 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:39:32 smithi085 conmon[41540]: debug 2021-02-10T04:39:37.616 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:39:32 smithi085 conmon[41540]: 2021-02-10T04:39:32.693+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:37.616 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:39:32 smithi085 conmon[41540]: 2021-02-10T04:39:37.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:32 smithi085 conmon[35961]: cluster 2021-02-10T04:39:31. 2021-02-10T04:39:37.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:32 smithi085 conmon[35961]: 579294+0000 mon.a (mon.0) 1672 : cluster [DBG] osdmap e752: 8 total, 8 up, 7 in 2021-02-10T04:39:37.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:32 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:39:37.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:32 smithi085 conmon[35961]: 02-10T04:39:31.913471+0000 osd.4 (osd.4) 1249 2021-02-10T04:39:37.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:32 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:39:37.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:32 smithi085 conmon[35961]: cluster 2021-02-10T04:39 2021-02-10T04:39:37.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:32 smithi085 conmon[35961]: :31.913481+0000 osd.4 (osd.4) 2021-02-10T04:39:37.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:32 smithi085 conmon[35961]: 1250 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:39:37.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:32 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:39:37.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:32 smithi085 conmon[35961]: 02-10T04:39:31.913488+0000 2021-02-10T04:39:37.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:32 smithi085 conmon[35961]: osd.4 (osd.4) 1251 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:39:37.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:32 smithi085 conmon[35961]: 2021-02-10T04:39:37.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:32 smithi085 conmon[35961]: cluster 2021-02-10T04:39:37.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:32 smithi085 conmon[35961]: 2021-02-10T04:39:31.913496+0000 osd.4 ( 2021-02-10T04:39:37.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:32 smithi085 conmon[35961]: osd.4) 1252 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:39:37.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:32 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:39:37.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:32 smithi085 conmon[35961]: -02-10T04:39:32.177080+0000 2021-02-10T04:39:37.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:32 smithi085 conmon[35961]: mgr.y (mgr.14140) 751 : cluster [DBG] 2021-02-10T04:39:37.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:32 smithi085 conmon[35961]: pgmap v1279: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 515 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 2.1 MiB/s rd, 1.7 MiB/s wr, 12 op/s 2021-02-10T04:39:37.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:33 smithi085 conmon[35961]: cluster 2021-02-10T04:39:32.591600 2021-02-10T04:39:37.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:33 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1673 : cluster [DBG] osdmap e753: 8 total, 8 up, 7 in 2021-02-10T04:39:37.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:33 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:39:37.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:33 smithi085 conmon[35961]: -10T04:39:32.681762+0000 mon.b (mon.2) 39 : audit 2021-02-10T04:39:37.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:33 smithi085 conmon[35961]: [DBG] from='client.? 172.21.15.85:0/3369693159' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:39:37.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:33 smithi085 conmon[35961]: audit 2021-02-10T04:39 2021-02-10T04:39:37.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:33 smithi085 conmon[35961]: :33.005904+0000 mon.a (mon.0) 1674 : audit [INF] 2021-02-10T04:39:37.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:33 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/2859887596' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:39:37.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[35961]: cluster 2021-02-10T04:39:37.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[35961]: 2021-02-10T04:39:37.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:39:37.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[35961]: :39:33.582793+0000 mon.a (mon.0) 1675 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:39:37.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[35961]: audit 2021-02-10T04:39:37.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[35961]: 2021-02-10T04:39:33. 2021-02-10T04:39:37.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[35961]: 597648+0000 mon.a (mon.0) 1676 : audit [INF] from='client.? 172.21.15.85:0/2859887596' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:39:37.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[35961]: cluster 2021-02-10T04:39:33.597753 2021-02-10T04:39:37.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1677 : cluster [DBG] osdmap e754: 8 total, 8 up, 7 in 2021-02-10T04:39:37.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[35961]: cluster 2021-02-10T04:39:37.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[35961]: 2021-02-10T04:39:33.901985+0000 osd.0 (osd 2021-02-10T04:39:37.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[35961]: .0) 87 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:39:37.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[35961]: cluster 2021-02-10T04:39:33. 2021-02-10T04:39:37.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[35961]: 903511+0000 osd.0 (osd.0) 88 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:39:37.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[35961]: cluster 2021-02-10T04:39:34.177728 2021-02-10T04:39:37.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 752 : cluster [DBG] pgmap v1282: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 521 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 865 KiB/s wr, 8 op/s 2021-02-10T04:39:37.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:34 smithi085 conmon[35961]: 2021-02-10T04:39:37.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:35 smithi085 conmon[35961]: cluster 2021-02-10T04:39:34.601878+0000 2021-02-10T04:39:37.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:35 smithi085 conmon[35961]: mon.a (mon.0) 1678 : cluster [DBG] osdmap e755: 8 total, 8 up, 7 in 2021-02-10T04:39:37.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:36 smithi085 conmon[35961]: cluster 2021-02-10T04:39:35.603600+0000 2021-02-10T04:39:37.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:36 smithi085 conmon[35961]: mon.a (mon.0) 1679 : cluster [DBG] osdmap e756: 8 total, 8 up, 7 in 2021-02-10T04:39:37.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:36 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:39:37.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:36 smithi085 conmon[35961]: :39:36.178171+0000 mgr.y (mgr.14140) 753 : cluster [DBG] pgmap v1285: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 518 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 6.9 MiB/s rd, 1.3 MiB/s wr, 12 op/s 2021-02-10T04:39:37.620 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:39:37.620 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:39:37.620 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:39:37.620 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:39:37.620 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:39:37.620 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:39:37.620 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:39:37.620 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:39:37.621 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:39:37.621 INFO:tasks.rados.rados.0.smithi085.stdout:2959: done (5 left) 2021-02-10T04:39:37.621 INFO:tasks.rados.rados.0.smithi085.stdout:2960: done (4 left) 2021-02-10T04:39:37.621 INFO:tasks.rados.rados.0.smithi085.stdout:2961: snap_remove snap 281 2021-02-10T04:39:37.621 INFO:tasks.rados.rados.0.smithi085.stdout:2961: done (4 left) 2021-02-10T04:39:37.621 INFO:tasks.rados.rados.0.smithi085.stdout:2962: snap_create 2021-02-10T04:39:37.621 INFO:tasks.rados.rados.0.smithi085.stdout:2962: done (4 left) 2021-02-10T04:39:37.621 INFO:tasks.rados.rados.0.smithi085.stdout:2963: delete oid 45 current snap is 339 2021-02-10T04:39:37.621 INFO:tasks.rados.rados.0.smithi085.stdout:2963: done (4 left) 2021-02-10T04:39:37.621 INFO:tasks.rados.rados.0.smithi085.stdout:2964: snap_create 2021-02-10T04:39:37.621 INFO:tasks.rados.rados.0.smithi085.stdout:2964: done (4 left) 2021-02-10T04:39:37.621 INFO:tasks.rados.rados.0.smithi085.stdout:2965: read oid 42 snap -1 2021-02-10T04:39:37.622 INFO:tasks.rados.rados.0.smithi085.stdout:2965: expect deleted 2021-02-10T04:39:37.622 INFO:tasks.rados.rados.0.smithi085.stdout:2966: read oid 36 snap -1 2021-02-10T04:39:37.622 INFO:tasks.rados.rados.0.smithi085.stdout:2966: expect (ObjNum 991 snap 326 seq_num 991) 2021-02-10T04:39:37.622 INFO:tasks.rados.rados.0.smithi085.stdout:2967: copy_from oid 45 from oid 43 current snap is 340 2021-02-10T04:39:37.622 INFO:tasks.rados.rados.0.smithi085.stdout:2968: write (excl) oid 9 current snap is 340 2021-02-10T04:39:37.622 INFO:tasks.rados.rados.0.smithi085.stdout:2968: seq_num 1031 ranges {528017=489374,1481641=630014,2696282=400857} 2021-02-10T04:39:37.622 INFO:tasks.rados.rados.0.smithi085.stdout:2968: writing smithi08575397-9 from 528017 to 1017391 tid 1 2021-02-10T04:39:37.622 INFO:tasks.rados.rados.0.smithi085.stdout:2968: writing smithi08575397-9 from 1481641 to 2111655 tid 2 2021-02-10T04:39:37.622 INFO:tasks.rados.rados.0.smithi085.stdout:2968: writing smithi08575397-9 from 2696282 to 3097139 tid 3 2021-02-10T04:39:37.622 INFO:tasks.rados.rados.0.smithi085.stdout:2969: rollback oid 50 current snap is 340 2021-02-10T04:39:37.622 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 50 to 298 2021-02-10T04:39:37.622 INFO:tasks.rados.rados.0.smithi085.stdout:2968: finishing write tid 1 to smithi08575397-9 2021-02-10T04:39:37.622 INFO:tasks.rados.rados.0.smithi085.stdout:2965: done (8 left) 2021-02-10T04:39:37.623 INFO:tasks.rados.rados.0.smithi085.stdout:2970: read oid 28 snap -1 2021-02-10T04:39:37.623 INFO:tasks.rados.rados.0.smithi085.stdout:2970: expect (ObjNum 952 snap 313 seq_num 952) 2021-02-10T04:39:37.623 INFO:tasks.rados.rados.0.smithi085.stdout:2971: write oid 21 current snap is 340 2021-02-10T04:39:37.623 INFO:tasks.rados.rados.0.smithi085.stdout:2971: seq_num 1032 ranges {544824=564623,1823770=490025,3007352=337472} 2021-02-10T04:39:37.623 INFO:tasks.rados.rados.0.smithi085.stdout:2971: writing smithi08575397-21 from 544824 to 1109447 tid 1 2021-02-10T04:39:37.623 INFO:tasks.rados.rados.0.smithi085.stdout:2971: writing smithi08575397-21 from 1823770 to 2313795 tid 2 2021-02-10T04:39:37.623 INFO:tasks.rados.rados.0.smithi085.stdout:2971: writing smithi08575397-21 from 3007352 to 3344824 tid 3 2021-02-10T04:39:37.623 INFO:tasks.rados.rados.0.smithi085.stdout:2972: delete oid 44 current snap is 340 2021-02-10T04:39:37.623 INFO:tasks.rados.rados.0.smithi085.stdout:2966: expect (ObjNum 991 snap 326 seq_num 991) 2021-02-10T04:39:37.623 INFO:tasks.rados.rados.0.smithi085.stdout:2968: finishing write tid 2 to smithi08575397-9 2021-02-10T04:39:37.623 INFO:tasks.rados.rados.0.smithi085.stdout:2969: finishing rollback tid 1 to smithi08575397-50 2021-02-10T04:39:37.623 INFO:tasks.rados.rados.0.smithi085.stdout:2969: finishing rollback tid 2 to smithi08575397-50 2021-02-10T04:39:37.623 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 1344 (ObjNum 889 snap 289 seq_num 889) dirty exists 2021-02-10T04:39:37.623 INFO:tasks.rados.rados.0.smithi085.stdout:2966: done (10 left) 2021-02-10T04:39:37.624 INFO:tasks.rados.rados.0.smithi085.stdout:2969: done (9 left) 2021-02-10T04:39:37.624 INFO:tasks.rados.rados.0.smithi085.stdout:2972: done (8 left) 2021-02-10T04:39:37.624 INFO:tasks.rados.rados.0.smithi085.stdout:2973: snap_remove snap 329 2021-02-10T04:39:37.624 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is set 2021-02-10T04:39:37.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:33 smithi164 conmon[29288]: cluster 2021-02-10T04:39:37.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:33 smithi164 conmon[29288]: 2021-02-10T04:39:37.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:33 smithi164 conmon[29288]: 2021 2021-02-10T04:39:37.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:33 smithi164 conmon[29288]: - 2021-02-10T04:39:37.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:33 smithi164 conmon[29288]: 02-10T04:39 2021-02-10T04:39:37.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:33 smithi164 conmon[29288]: :32.591600+0000 mon.a (mon.0) 1673 : cluster [DBG] osdmap e753: 8 total, 8 up, 7 in 2021-02-10T04:39:37.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:33 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:39:37.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:33 smithi164 conmon[29288]: 02-10T04:39:32.681762+0000 mon.b (mon.2 2021-02-10T04:39:37.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:33 smithi164 conmon[29288]: ) 39 : audit [DBG] from='client.? 172.21.15.85:0/3369693159' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:39:37.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:33 smithi164 conmon[29288]: audit 2021 2021-02-10T04:39:37.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:33 smithi164 conmon[29288]: -02-10T04:39:33.005904+0000 mon.a 2021-02-10T04:39:37.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:33 smithi164 conmon[29288]: (mon.0) 1674 : audit [INF] from='client.? 172.21.15.85:0/2859887596' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:39:37.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:34 smithi164 conmon[29288]: cluster 2021-02-10T04:39:33.582793+0000 mon.a (mon.0) 1675 2021-02-10T04:39:37.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:34 smithi164 conmon[29288]: : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:39:37.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:34 smithi164 conmon[29288]: audit 2021-02-10T04:39:33.597648+0000 mon.a (mon.0) 1676 : audit [INF] from='client.? 172.21.15.85:0/2859887596' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:39:37.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:34 smithi164 conmon[29288]: cluster 2021-02-10T04:39:33.597753+0000 mon.a (mon.0) 2021-02-10T04:39:37.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:34 smithi164 conmon[29288]: 1677 : cluster [DBG] osdmap e754: 8 total, 8 up, 7 in 2021-02-10T04:39:37.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:34 smithi164 conmon[29288]: cluster 2021-02-10T04:39:37.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:34 smithi164 conmon[29288]: 2021-02-10 2021-02-10T04:39:37.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:34 smithi164 conmon[29288]: T04:39:33.901985+0000 osd.0 (osd.0) 2021-02-10T04:39:37.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:34 smithi164 conmon[29288]: 87 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:39:37.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:34 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:39:37.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:34 smithi164 conmon[29288]: -02-10T04:39: 2021-02-10T04:39:37.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:34 smithi164 conmon[29288]: 33.903511+0000 osd.0 (osd.0) 2021-02-10T04:39:37.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:34 smithi164 conmon[29288]: 88 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:39:37.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:34 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:39:37.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:34 smithi164 conmon[29288]: 02-10T04:39:34.177728+0000 mgr.y (mgr. 2021-02-10T04:39:37.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:34 smithi164 conmon[29288]: 14140) 752 : cluster [DBG] pgmap v1282: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 521 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 865 KiB/s wr, 8 op/s 2021-02-10T04:39:37.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:35 smithi164 conmon[29288]: cluster 2021-02-10T04:39:34.601878+0000 mon.a (mon.0) 1678 : cluster [DBG] 2021-02-10T04:39:37.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:35 smithi164 conmon[29288]: osdmap e755: 8 total, 8 up, 7 in 2021-02-10T04:39:37.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:36 smithi164 conmon[29288]: cluster 2021-02-10T04:39:35.603600+0000 mon.a 2021-02-10T04:39:37.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:36 smithi164 conmon[29288]: (mon.0) 1679 : cluster [DBG] osdmap e756: 8 total, 8 up, 7 in 2021-02-10T04:39:37.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:36 smithi164 conmon[29288]: cluster 2021-02-10T04:39:36.178171+0000 mgr.y (mgr.14140) 753 : cluster [DBG] 2021-02-10T04:39:37.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:36 smithi164 conmon[29288]: pgmap v1285: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 518 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 6.9 MiB/s rd, 1.3 MiB/s wr, 12 op/s 2021-02-10T04:39:37.627 INFO:tasks.rados.rados.0.smithi085.stdout:2973: done (8 left) 2021-02-10T04:39:37.627 INFO:tasks.rados.rados.0.smithi085.stdout:2974: read oid 15 snap 298 2021-02-10T04:39:37.627 INFO:tasks.rados.rados.0.smithi085.stdout:2974: expect (ObjNum 217 snap 49 seq_num 217) 2021-02-10T04:39:37.627 INFO:tasks.rados.rados.0.smithi085.stdout:2975: rollback oid 2 current snap is 340 2021-02-10T04:39:37.627 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 2 to 331 2021-02-10T04:39:37.627 INFO:tasks.rados.rados.0.smithi085.stdout:2976: rollback oid 37 current snap is 340 2021-02-10T04:39:37.627 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 37 to 237 2021-02-10T04:39:37.628 INFO:tasks.rados.rados.0.smithi085.stdout:2977: write oid 48 current snap is 340 2021-02-10T04:39:37.628 INFO:tasks.rados.rados.0.smithi085.stdout:2977: seq_num 1034 ranges {578438=431474,1657087=493694,2778551=599887} 2021-02-10T04:39:37.628 INFO:tasks.rados.rados.0.smithi085.stdout:2977: writing smithi08575397-48 from 578438 to 1009912 tid 1 2021-02-10T04:39:37.634 INFO:tasks.rados.rados.0.smithi085.stdout:2977: writing smithi08575397-48 from 1657087 to 2150781 tid 2 2021-02-10T04:39:37.641 INFO:tasks.rados.rados.0.smithi085.stdout:2977: writing smithi08575397-48 from 2778551 to 3378438 tid 3 2021-02-10T04:39:37.641 INFO:tasks.rados.rados.0.smithi085.stdout:2978: copy_from oid 1 from oid 40 current snap is 340 2021-02-10T04:39:37.642 INFO:tasks.rados.rados.0.smithi085.stdout:2970: expect (ObjNum 952 snap 313 seq_num 952) 2021-02-10T04:39:37.829 INFO:tasks.rados.rados.0.smithi085.stdout:2968: finishing write tid 3 to smithi08575397-9 2021-02-10T04:39:37.830 INFO:tasks.rados.rados.0.smithi085.stdout:2968: finishing write tid 4 to smithi08575397-9 2021-02-10T04:39:37.830 INFO:tasks.rados.rados.0.smithi085.stdout:2968: finishing write tid 5 to smithi08575397-9 2021-02-10T04:39:37.830 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 9 v 1371 (ObjNum 1031 snap 340 seq_num 1031) dirty exists 2021-02-10T04:39:37.830 INFO:tasks.rados.rados.0.smithi085.stdout:2968: left oid 9 (ObjNum 1031 snap 340 seq_num 1031) 2021-02-10T04:39:37.830 INFO:tasks.rados.rados.0.smithi085.stdout:2971: finishing write tid 1 to smithi08575397-21 2021-02-10T04:39:37.830 INFO:tasks.rados.rados.0.smithi085.stdout:2971: finishing write tid 2 to smithi08575397-21 2021-02-10T04:39:37.830 INFO:tasks.rados.rados.0.smithi085.stdout:2971: finishing write tid 3 to smithi08575397-21 2021-02-10T04:39:37.830 INFO:tasks.rados.rados.0.smithi085.stdout:2971: finishing write tid 4 to smithi08575397-21 2021-02-10T04:39:37.830 INFO:tasks.rados.rados.0.smithi085.stdout:2971: finishing write tid 5 to smithi08575397-21 2021-02-10T04:39:37.830 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 1548 (ObjNum 1032 snap 340 seq_num 1032) dirty exists 2021-02-10T04:39:37.831 INFO:tasks.rados.rados.0.smithi085.stdout:2971: left oid 21 (ObjNum 1032 snap 340 seq_num 1032) 2021-02-10T04:39:37.831 INFO:tasks.rados.rados.0.smithi085.stdout:2967: finishing copy_from to smithi08575397-45 2021-02-10T04:39:37.831 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 45 v 1354 (ObjNum 981 snap 317 seq_num 981) dirty exists 2021-02-10T04:39:37.831 INFO:tasks.rados.rados.0.smithi085.stdout:2967: finishing copy_from racing read to smithi08575397-45 2021-02-10T04:39:37.831 INFO:tasks.rados.rados.0.smithi085.stdout:2976: finishing rollback tid 0 to smithi08575397-37 2021-02-10T04:39:37.831 INFO:tasks.rados.rados.0.smithi085.stdout:2975: finishing rollback tid 0 to smithi08575397-2 2021-02-10T04:39:37.831 INFO:tasks.rados.rados.0.smithi085.stdout:2975: finishing rollback tid 1 to smithi08575397-2 2021-02-10T04:39:37.831 INFO:tasks.rados.rados.0.smithi085.stdout:2975: finishing rollback tid 2 to smithi08575397-2 2021-02-10T04:39:37.831 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 1453 (ObjNum 982 snap 317 seq_num 982) dirty exists 2021-02-10T04:39:37.831 INFO:tasks.rados.rados.0.smithi085.stdout:2976: finishing rollback tid 1 to smithi08575397-37 2021-02-10T04:39:37.831 INFO:tasks.rados.rados.0.smithi085.stdout:2976: finishing rollback tid 2 to smithi08575397-37 2021-02-10T04:39:37.831 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 1556 (ObjNum 646 snap 196 seq_num 646) dirty exists 2021-02-10T04:39:37.831 INFO:tasks.rados.rados.0.smithi085.stdout:2974: expect (ObjNum 217 snap 49 seq_num 217) 2021-02-10T04:39:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:37 smithi164 conmon[29288]: cluster 2021-02-10T04:39:36.613379+0000 mon.a (mon.0) 1680 : cluster 2021-02-10T04:39:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:37 smithi164 conmon[29288]: [DBG] osdmap e757: 8 total, 8 up, 7 in 2021-02-10T04:39:37.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:37 smithi085 conmon[29319]: cluster 2021-02-10T04:39:36.613379+0000 mon.a (mon.0) 1680 : cluster [DBG] osdmap e757: 8 total, 8 up, 7 in 2021-02-10T04:39:37.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:37 smithi085 conmon[35961]: cluster 2021-02-10T04:39:37.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:37 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:39:37.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:37 smithi085 conmon[35961]: 10T04:39:36.613379+0000 mon.a (mon.0) 1680 : cluster [DBG] osdmap e757: 8 total, 8 up, 7 in 2021-02-10T04:39:38.126 INFO:tasks.rados.rados.0.smithi085.stdout:2977: finishing write tid 1 to smithi08575397-48 2021-02-10T04:39:38.127 INFO:tasks.rados.rados.0.smithi085.stdout:2977: finishing write tid 2 to smithi08575397-48 2021-02-10T04:39:38.127 INFO:tasks.rados.rados.0.smithi085.stdout:2977: finishing write tid 3 to smithi08575397-48 2021-02-10T04:39:38.127 INFO:tasks.rados.rados.0.smithi085.stdout:2977: finishing write tid 4 to smithi08575397-48 2021-02-10T04:39:38.127 INFO:tasks.rados.rados.0.smithi085.stdout:2977: finishing write tid 5 to smithi08575397-48 2021-02-10T04:39:38.127 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 1375 (ObjNum 1034 snap 340 seq_num 1034) dirty exists 2021-02-10T04:39:38.127 INFO:tasks.rados.rados.0.smithi085.stdout:2977: left oid 48 (ObjNum 1034 snap 340 seq_num 1034) 2021-02-10T04:39:38.127 INFO:tasks.rados.rados.0.smithi085.stdout:2978: finishing copy_from to smithi08575397-1 2021-02-10T04:39:38.127 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 1540 (ObjNum 1021 snap 335 seq_num 1021) dirty exists 2021-02-10T04:39:38.127 INFO:tasks.rados.rados.0.smithi085.stdout:2978: finishing copy_from racing read to smithi08575397-1 2021-02-10T04:39:38.127 INFO:tasks.rados.rados.0.smithi085.stdout:2967: done (12 left) 2021-02-10T04:39:38.127 INFO:tasks.rados.rados.0.smithi085.stdout:2968: done (11 left) 2021-02-10T04:39:38.127 INFO:tasks.rados.rados.0.smithi085.stdout:2970: done (10 left) 2021-02-10T04:39:38.128 INFO:tasks.rados.rados.0.smithi085.stdout:2971: done (9 left) 2021-02-10T04:39:38.128 INFO:tasks.rados.rados.0.smithi085.stdout:2974: done (8 left) 2021-02-10T04:39:38.128 INFO:tasks.rados.rados.0.smithi085.stdout:2975: done (7 left) 2021-02-10T04:39:38.128 INFO:tasks.rados.rados.0.smithi085.stdout:2976: done (6 left) 2021-02-10T04:39:38.128 INFO:tasks.rados.rados.0.smithi085.stdout:2977: done (5 left) 2021-02-10T04:39:38.128 INFO:tasks.rados.rados.0.smithi085.stdout:2978: done (4 left) 2021-02-10T04:39:38.128 INFO:tasks.rados.rados.0.smithi085.stdout:2979: read oid 11 snap 306 2021-02-10T04:39:38.128 INFO:tasks.rados.rados.0.smithi085.stdout:2979: expect (ObjNum 910 snap 299 seq_num 910) 2021-02-10T04:39:38.128 INFO:tasks.rados.rados.0.smithi085.stdout:2980: delete oid 13 current snap is 340 2021-02-10T04:39:38.129 INFO:tasks.rados.rados.0.smithi085.stdout:2980: done (5 left) 2021-02-10T04:39:38.129 INFO:tasks.rados.rados.0.smithi085.stdout:2981: delete oid 31 current snap is 340 2021-02-10T04:39:38.134 INFO:tasks.rados.rados.0.smithi085.stdout:2979: expect (ObjNum 910 snap 299 seq_num 910) 2021-02-10T04:39:38.288 INFO:tasks.rados.rados.0.smithi085.stdout:2979: done (5 left) 2021-02-10T04:39:38.289 INFO:tasks.rados.rados.0.smithi085.stdout:2981: done (4 left) 2021-02-10T04:39:38.289 INFO:tasks.rados.rados.0.smithi085.stdout:2982: rollback oid 5 current snap is 340 2021-02-10T04:39:38.289 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 5 to 299 2021-02-10T04:39:38.289 INFO:tasks.rados.rados.0.smithi085.stdout:2983: delete oid 32 current snap is 340 2021-02-10T04:39:38.337 INFO:tasks.rados.rados.0.smithi085.stdout:2983: done (5 left) 2021-02-10T04:39:38.337 INFO:tasks.rados.rados.0.smithi085.stdout:2984: copy_from oid 38 from oid 30 current snap is 340 2021-02-10T04:39:38.338 INFO:tasks.rados.rados.0.smithi085.stdout:2985: write (excl) oid 9 current snap is 340 2021-02-10T04:39:38.338 INFO:tasks.rados.rados.0.smithi085.stdout:2985: seq_num 1038 ranges {645666=732470,2174662=501032,3288243=157423} 2021-02-10T04:39:38.346 INFO:tasks.rados.rados.0.smithi085.stdout:2985: writing smithi08575397-9 from 645666 to 1378136 tid 1 2021-02-10T04:39:38.354 INFO:tasks.rados.rados.0.smithi085.stdout:2985: writing smithi08575397-9 from 2174662 to 2675694 tid 2 2021-02-10T04:39:38.358 INFO:tasks.rados.rados.0.smithi085.stdout:2985: writing smithi08575397-9 from 3288243 to 3445666 tid 3 2021-02-10T04:39:38.358 INFO:tasks.rados.rados.0.smithi085.stdout:2986: read oid 11 snap -1 2021-02-10T04:39:38.359 INFO:tasks.rados.rados.0.smithi085.stdout:2986: expect (ObjNum 967 snap 316 seq_num 967) 2021-02-10T04:39:38.359 INFO:tasks.rados.rados.0.smithi085.stdout:2987: snap_remove snap 316 2021-02-10T04:39:38.624 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:39:38 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:39:38.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:38 smithi164 conmon[29288]: cluster 2021-02-10T04:39:37.619528+0000 2021-02-10T04:39:38.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:38 smithi164 conmon[29288]: mon.a (mon.0) 1681 : cluster [DBG] osdmap e758: 8 total, 8 up, 7 in 2021-02-10T04:39:38.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:38 smithi164 conmon[29288]: audit 2021-02-10T04:39:38.127364+0000 2021-02-10T04:39:38.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:38 smithi164 conmon[29288]: mon.a (mon.0) 1682 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:39:38.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:38 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:39:38.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:38 smithi164 conmon[29288]: 04:39:38.127552+0000 mon.a (mon.0 2021-02-10T04:39:38.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:38 smithi164 conmon[29288]: ) 1683 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:39:38.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:38 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:39:38.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:38 smithi164 conmon[29288]: 02-10T04:39:38.178828+0000 2021-02-10T04:39:38.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:38 smithi164 conmon[29288]: mgr.y (mgr.14140) 754 : cluster 2021-02-10T04:39:38.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:38 smithi164 conmon[29288]: [DBG] pgmap v1288: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 524 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 7.5 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2021-02-10T04:39:38.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:38 smithi164 conmon[29288]: cluster 2021-02-10T04:39 2021-02-10T04:39:38.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:38 smithi164 conmon[29288]: :38.244474+0000 mon.a (mon.0) 2021-02-10T04:39:38.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:38 smithi164 conmon[29288]: 1684 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 602 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:39:38.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:38 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:39:38.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:38 smithi164 conmon[29288]: 04:39:38.249980+0000 mon.a (mon. 2021-02-10T04:39:38.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:38 smithi164 conmon[29288]: 0) 1685 : cluster [DBG] osdmap e759: 8 total, 8 up, 7 in 2021-02-10T04:39:38.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[29319]: cluster 2021-02-10T04:39:37.619528+0000 mon.a 2021-02-10T04:39:38.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[29319]: (mon.0) 1681 : cluster [DBG] osdmap e758: 8 total, 8 up, 7 in 2021-02-10T04:39:38.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[29319]: audit 2021-02-10T04:39:38.127364+0000 mon.a (mon.0) 1682 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:39:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[29319]: audit 2021-02-10T04:39:38.127552+0000 2021-02-10T04:39:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[29319]: mon.a (mon.0) 1683 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:39:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[29319]: cluster 2021-02-10T04:39: 2021-02-10T04:39:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[29319]: 38.178828+0000 mgr.y (mgr.14140) 754 : cluster [DBG] 2021-02-10T04:39:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[29319]: pgmap v1288: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 524 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 7.5 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2021-02-10T04:39:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[29319]: cluster 2021-02-10T04:39 2021-02-10T04:39:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[29319]: :38.244474+0000 mon.a (mon.0) 1684 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 602 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:39:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:39:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[29319]: 10T04:39:38.249980+0000 2021-02-10T04:39:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[29319]: mon.a (mon.0) 1685 : cluster [DBG] 2021-02-10T04:39:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[29319]: osdmap e759: 8 total, 8 up, 7 in 2021-02-10T04:39:38.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:39:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[35961]: 39:37.619528+0000 mon.a (mon.0) 1681 : cluster [DBG] osdmap e758: 8 total, 8 up, 7 in 2021-02-10T04:39:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[35961]: audit 2021-02-10T04:39:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:39:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[35961]: 10T04:39:38.127364 2021-02-10T04:39:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2021-02-10T04:39:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[35961]: 1682 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:39:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[35961]: audit 2021 2021-02-10T04:39:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[35961]: -02-10T04:39:38.127552+0000 2021-02-10T04:39:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[35961]: mon.a (mon.0) 1683 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:39:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[35961]: cluster 2021-02-10T04:39:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[35961]: 2021-02-10T04:39:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:39:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[35961]: :39:38.178828+0000 mgr.y (mgr.14140) 754 : cluster [DBG] pgmap v1288: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 524 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 7.5 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2021-02-10T04:39:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[35961]: 2021-02-10T04:39:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[35961]: cluster 2021-02-10T04:39:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[35961]: 2021-02-10T04:39:38.244474 2021-02-10T04:39:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1684 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 602 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:39:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[35961]: cluster 2021-02-10T04:39:38 2021-02-10T04:39:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:38 smithi085 conmon[35961]: .249980+0000 mon.a (mon.0) 1685 : cluster [DBG] osdmap e759: 8 total, 8 up, 7 in 2021-02-10T04:39:39.257 INFO:tasks.rados.rados.0.smithi085.stdout:2987: done (8 left) 2021-02-10T04:39:39.257 INFO:tasks.rados.rados.0.smithi085.stdout:2988: write (excl) oid 16 current snap is 340 2021-02-10T04:39:39.257 INFO:tasks.rados.rados.0.smithi085.stdout:2982: finishing rollback tid 1 to smithi08575397-5 2021-02-10T04:39:39.257 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 810 (ObjNum 82895248 snap 22030 seq_num 0) dirty dne 2021-02-10T04:39:39.257 INFO:tasks.rados.rados.0.smithi085.stdout:2985: finishing write tid 1 to smithi08575397-9 2021-02-10T04:39:39.257 INFO:tasks.rados.rados.0.smithi085.stdout:2984: finishing copy_from to smithi08575397-38 2021-02-10T04:39:39.257 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 1376 (ObjNum 1025 snap 335 seq_num 1025) dirty exists 2021-02-10T04:39:39.257 INFO:tasks.rados.rados.0.smithi085.stdout:2985: finishing write tid 2 to smithi08575397-9 2021-02-10T04:39:39.258 INFO:tasks.rados.rados.0.smithi085.stdout:2984: finishing copy_from racing read to smithi08575397-38 2021-02-10T04:39:39.258 INFO:tasks.rados.rados.0.smithi085.stdout:2985: finishing write tid 3 to smithi08575397-9 2021-02-10T04:39:39.258 INFO:tasks.rados.rados.0.smithi085.stdout:2985: finishing write tid 4 to smithi08575397-9 2021-02-10T04:39:39.258 INFO:tasks.rados.rados.0.smithi085.stdout:2985: finishing write tid 5 to smithi08575397-9 2021-02-10T04:39:39.258 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 9 v 1379 (ObjNum 1038 snap 340 seq_num 1038) dirty exists 2021-02-10T04:39:39.258 INFO:tasks.rados.rados.0.smithi085.stdout:2985: left oid 9 (ObjNum 1038 snap 340 seq_num 1038) 2021-02-10T04:39:39.258 INFO:tasks.rados.rados.0.smithi085.stdout:2986: expect (ObjNum 967 snap 316 seq_num 967) 2021-02-10T04:39:39.403 INFO:tasks.rados.rados.0.smithi085.stdout:2988: seq_num 1039 ranges {662473=407719,1833144=644690,3199313=263160} 2021-02-10T04:39:39.409 INFO:tasks.rados.rados.0.smithi085.stdout:2988: writing smithi08575397-16 from 662473 to 1070192 tid 1 2021-02-10T04:39:39.419 INFO:tasks.rados.rados.0.smithi085.stdout:2988: writing smithi08575397-16 from 1833144 to 2477834 tid 2 2021-02-10T04:39:39.425 INFO:tasks.rados.rados.0.smithi085.stdout:2988: writing smithi08575397-16 from 3199313 to 3462473 tid 3 2021-02-10T04:39:39.425 INFO:tasks.rados.rados.0.smithi085.stdout:2982: done (8 left) 2021-02-10T04:39:39.426 INFO:tasks.rados.rados.0.smithi085.stdout:2984: done (7 left) 2021-02-10T04:39:39.426 INFO:tasks.rados.rados.0.smithi085.stdout:2985: done (6 left) 2021-02-10T04:39:39.426 INFO:tasks.rados.rados.0.smithi085.stdout:2986: done (5 left) 2021-02-10T04:39:39.426 INFO:tasks.rados.rados.0.smithi085.stdout:2989: delete oid 38 current snap is 340 2021-02-10T04:39:39.426 INFO:tasks.rados.rados.0.smithi085.stdout:2988: finishing write tid 1 to smithi08575397-16 2021-02-10T04:39:39.427 INFO:tasks.rados.rados.0.smithi085.stdout:2988: finishing write tid 2 to smithi08575397-16 2021-02-10T04:39:39.428 INFO:tasks.rados.rados.0.smithi085.stdout:2989: done (5 left) 2021-02-10T04:39:39.429 INFO:tasks.rados.rados.0.smithi085.stdout:2990: snap_create 2021-02-10T04:39:39.429 INFO:tasks.rados.rados.0.smithi085.stdout:2988: finishing write tid 3 to smithi08575397-16 2021-02-10T04:39:39.430 INFO:tasks.rados.rados.0.smithi085.stdout:2988: finishing write tid 4 to smithi08575397-16 2021-02-10T04:39:39.430 INFO:tasks.rados.rados.0.smithi085.stdout:2988: finishing write tid 5 to smithi08575397-16 2021-02-10T04:39:39.430 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 1562 (ObjNum 1039 snap 340 seq_num 1039) dirty exists 2021-02-10T04:39:39.430 INFO:tasks.rados.rados.0.smithi085.stdout:2988: left oid 16 (ObjNum 1039 snap 340 seq_num 1039) 2021-02-10T04:39:39.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:39 smithi164 conmon[29288]: audit 2021-02-10T04:39:38.346078+0000 mgr.y (mgr.14140) 755 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:39:39.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:39 smithi164 conmon[29288]: 2021-02-10T04:39:39.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:39 smithi164 conmon[29288]: cluster 2021-02-10T04:39:38.616499+0000 mon.a (mon.0) 1686 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 602 sec, osd.4 has slow ops) 2021-02-10T04:39:39.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:39 smithi164 conmon[29288]: cluster 2021-02-10T04:39:39.260631+0000 mon.a (mon.0) 2021-02-10T04:39:39.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:39 smithi164 conmon[29288]: 1687 : cluster [DBG] osdmap e760: 8 total, 8 up, 7 in 2021-02-10T04:39:39.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:39 smithi085 conmon[29319]: audit 2021-02-10T04:39:39.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:39 smithi085 conmon[29319]: 2021-02-10T04:39:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:39 smithi085 conmon[29319]: 2021 2021-02-10T04:39:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:39 smithi085 conmon[29319]: - 2021-02-10T04:39:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:39 smithi085 conmon[29319]: 02 2021-02-10T04:39:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:39 smithi085 conmon[29319]: - 2021-02-10T04:39:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:39 smithi085 conmon[29319]: 10 2021-02-10T04:39:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:39 smithi085 conmon[29319]: T04:39:38.346078+0000 mgr.y ( 2021-02-10T04:39:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:39 smithi085 conmon[29319]: mgr.14140) 755 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:39:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:39 smithi085 conmon[29319]: cluster 2021-02-10T04:39:38.616499+0000 mon.a 2021-02-10T04:39:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:39 smithi085 conmon[29319]: (mon.0) 1686 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 602 sec, osd.4 has slow ops) 2021-02-10T04:39:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:39 smithi085 conmon[29319]: cluster 2021-02-10T04:39:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:39 smithi085 conmon[29319]: 2021-02-10T04:39:39.260631+0000 mon.a (mon.0 2021-02-10T04:39:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:39 smithi085 conmon[29319]: ) 2021-02-10T04:39:39.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:39 smithi085 conmon[29319]: 1687 : cluster [DBG] osdmap e760: 8 total, 8 up, 7 in 2021-02-10T04:39:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:39 smithi085 conmon[35961]: audit 2021-02-10T04:39:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:39 smithi085 conmon[35961]: 2021-02-10T04:39:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:39 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:39:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:39 smithi085 conmon[35961]: :39:38.346078+0000 mgr.y (mgr.14140) 755 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:39:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:39 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:39:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:39 smithi085 conmon[35961]: -10T04:39:38.616499+0000 mon.a (mon.0) 2021-02-10T04:39:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:39 smithi085 conmon[35961]: 1686 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 602 sec, osd.4 has slow ops) 2021-02-10T04:39:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:39 smithi085 conmon[35961]: cluster 2021-02-10T04:39:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:39 smithi085 conmon[35961]: 2021-02-10T04:39:39.260631 2021-02-10T04:39:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:39 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1687 : cluster 2021-02-10T04:39:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:39 smithi085 conmon[35961]: [DBG] osdmap e760: 8 total, 8 up, 7 in 2021-02-10T04:39:40.259 INFO:tasks.rados.rados.0.smithi085.stdout:2988: done (5 left) 2021-02-10T04:39:40.259 INFO:tasks.rados.rados.0.smithi085.stdout:2990: done (4 left) 2021-02-10T04:39:40.259 INFO:tasks.rados.rados.0.smithi085.stdout:2991: read oid 11 snap -1 2021-02-10T04:39:40.259 INFO:tasks.rados.rados.0.smithi085.stdout:2991: expect (ObjNum 967 snap 316 seq_num 967) 2021-02-10T04:39:40.259 INFO:tasks.rados.rados.0.smithi085.stdout:2992: delete oid 1 current snap is 341 2021-02-10T04:39:40.266 INFO:tasks.rados.rados.0.smithi085.stdout:2992: done (5 left) 2021-02-10T04:39:40.266 INFO:tasks.rados.rados.0.smithi085.stdout:2993: copy_from oid 36 from oid 32 current snap is 341 2021-02-10T04:39:40.266 INFO:tasks.rados.rados.0.smithi085.stdout:2994: read oid 8 snap -1 2021-02-10T04:39:40.267 INFO:tasks.rados.rados.0.smithi085.stdout:2994: expect deleted 2021-02-10T04:39:40.267 INFO:tasks.rados.rados.0.smithi085.stdout:2995: read oid 28 snap 309 2021-02-10T04:39:40.267 INFO:tasks.rados.rados.0.smithi085.stdout:2995: expect (ObjNum 892 snap 289 seq_num 892) 2021-02-10T04:39:40.267 INFO:tasks.rados.rados.0.smithi085.stdout:2996: rollback oid 16 current snap is 341 2021-02-10T04:39:40.267 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 16 to 272 2021-02-10T04:39:40.267 INFO:tasks.rados.rados.0.smithi085.stdout:2997: delete oid 4 current snap is 341 2021-02-10T04:39:40.268 INFO:tasks.rados.rados.0.smithi085.stdout:2991: expect (ObjNum 967 snap 316 seq_num 967) 2021-02-10T04:39:40.406 INFO:tasks.rados.rados.0.smithi085.stdout:2996: finishing rollback tid 0 to smithi08575397-16 2021-02-10T04:39:41.571 INFO:tasks.rados.rados.0.smithi085.stdout:2996: finishing rollback tid 1 to smithi08575397-16 2021-02-10T04:39:41.571 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 1565 (ObjNum 82780112 snap 22030 seq_num 0) dirty dne 2021-02-10T04:39:41.571 INFO:tasks.rados.rados.0.smithi085.stdout:2995: expect (ObjNum 892 snap 289 seq_num 892) 2021-02-10T04:39:41.571 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:39:41.573 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:40 smithi164 conmon[29288]: cluster 2021-02-10T04:39:40.179589+0000 mgr.y (mgr.14140 2021-02-10T04:39:41.573 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:40 smithi164 conmon[29288]: ) 756 : cluster [DBG] pgmap v1291: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 530 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.2 MiB/s wr, 20 op/s 2021-02-10T04:39:41.573 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:40 smithi164 conmon[29288]: cluster 2021-02-10T04:39:40.258117+0000 mon.a (mon.0) 1688 : cluster [DBG] osdmap e761: 8 total, 8 up, 7 in 2021-02-10T04:39:41.574 INFO:tasks.rados.rados.0.smithi085.stdout:2993: finishing copy_from to smithi08575397-36 2021-02-10T04:39:41.574 INFO:tasks.rados.rados.0.smithi085.stdout:2993: got expected ENOENT (src dne) 2021-02-10T04:39:41.574 INFO:tasks.rados.rados.0.smithi085.stdout:2993: finishing copy_from racing read to smithi08575397-36 2021-02-10T04:39:41.574 INFO:tasks.rados.rados.0.smithi085.stdout:2991: done (9 left) 2021-02-10T04:39:41.574 INFO:tasks.rados.rados.0.smithi085.stdout:2993: done (8 left) 2021-02-10T04:39:41.574 INFO:tasks.rados.rados.0.smithi085.stdout:2994: done (7 left) 2021-02-10T04:39:41.574 INFO:tasks.rados.rados.0.smithi085.stdout:2995: done (6 left) 2021-02-10T04:39:41.574 INFO:tasks.rados.rados.0.smithi085.stdout:2996: done (5 left) 2021-02-10T04:39:41.574 INFO:tasks.rados.rados.0.smithi085.stdout:2997: done (4 left) 2021-02-10T04:39:41.574 INFO:tasks.rados.rados.0.smithi085.stdout:2998: snap_remove snap 311 2021-02-10T04:39:41.574 INFO:tasks.rados.rados.0.smithi085.stdout:2998: done (4 left) 2021-02-10T04:39:41.574 INFO:tasks.rados.rados.0.smithi085.stdout:2999: delete oid 16 current snap is 341 2021-02-10T04:39:41.575 INFO:tasks.rados.rados.0.smithi085.stdout:2999: done (4 left) 2021-02-10T04:39:41.575 INFO:tasks.rados.rados.0.smithi085.stdout:3000: snap_remove snap 309 2021-02-10T04:39:41.575 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:40 smithi085 conmon[29319]: cluster 2021-02-10T04:39:40.179589+0000 mgr.y ( 2021-02-10T04:39:41.575 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:40 smithi085 conmon[29319]: mgr.14140) 756 : cluster [DBG] pgmap v1291: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 530 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.2 MiB/s wr, 20 op/s 2021-02-10T04:39:41.575 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:40 smithi085 conmon[29319]: cluster 2021-02-10T04:39:40.258117+0000 mon.a (mon.0) 1688 : cluster [DBG] osdmap e761: 8 total, 8 up, 7 in 2021-02-10T04:39:41.575 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:40 smithi085 conmon[35961]: cluster 2021-02-10T04:39:40 2021-02-10T04:39:41.575 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:40 smithi085 conmon[35961]: .179589+0000 mgr.y (mgr.14140) 756 : cluster [DBG] pgmap v1291: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 530 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.2 MiB/s wr, 20 op/s 2021-02-10T04:39:41.575 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:40 smithi085 conmon[35961]: cluster 2021-02-10T04:39:40.258117+0000 mon.a ( 2021-02-10T04:39:41.575 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:40 smithi085 conmon[35961]: mon.0) 1688 : cluster [DBG] osdmap e761: 8 total, 8 up, 7 in 2021-02-10T04:39:41.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:41 smithi085 conmon[29319]: cluster 2021-02-10T04:39:41.086849+0000 osd.7 2021-02-10T04:39:41.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:41 smithi085 conmon[29319]: (osd.7) 60 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:39:41.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:41 smithi085 conmon[29319]: cluster 2021-02-10T04:39:41.087828+0000 osd.7 (osd.7) 61 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:39:41.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:41 smithi085 conmon[29319]: cluster 2021-02-10T04:39:41.264376+0000 mon.a (mon.0) 1689 : cluster [DBG] osdmap e762: 8 total, 8 up, 7 in 2021-02-10T04:39:41.635 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:41 smithi085 conmon[35961]: cluster 2021-02-10T04:39:41.086849+0000 2021-02-10T04:39:41.635 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:41 smithi085 conmon[35961]: osd.7 (osd.7) 60 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:39:41.635 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:41 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:39:41.635 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:41 smithi085 conmon[35961]: -02-10T04:39:41.087828+0000 osd.7 (osd.7) 2021-02-10T04:39:41.636 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:41 smithi085 conmon[35961]: 61 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:39:41.636 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:41 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:39:41.636 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:41 smithi085 conmon[35961]: 02-10T04:39:41.264376+0000 mon.a (mon 2021-02-10T04:39:41.636 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:41 smithi085 conmon[35961]: .0) 1689 : cluster [DBG] osdmap e762: 8 total, 8 up, 7 in 2021-02-10T04:39:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:41 smithi164 conmon[29288]: cluster 2021-02-10T04:39:41.086849+0000 osd.7 (osd.7) 60 : 2021-02-10T04:39:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:41 smithi164 conmon[29288]: cluster [DBG] purged_snaps scrub starts 2021-02-10T04:39:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:41 smithi164 conmon[29288]: cluster 2021-02-10T04:39:41.087828+0000 osd.7 (osd.7) 61 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:39:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:41 smithi164 conmon[29288]: cluster 2021-02-10T04:39:41.264376+0000 2021-02-10T04:39:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:41 smithi164 conmon[29288]: mon.a (mon.0) 1689 : cluster [DBG] osdmap e762: 8 total, 8 up, 7 in 2021-02-10T04:39:42.219 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:39:42.263 INFO:tasks.rados.rados.0.smithi085.stdout:3000: done (4 left) 2021-02-10T04:39:42.263 INFO:tasks.rados.rados.0.smithi085.stdout:3001: rollback oid 23 current snap is 341 2021-02-10T04:39:42.263 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 23 to 337 2021-02-10T04:39:42.264 INFO:tasks.rados.rados.0.smithi085.stdout:3002: write (excl) oid 43 current snap is 341 2021-02-10T04:39:42.264 INFO:tasks.rados.rados.0.smithi085.stdout:3002: seq_num 1044 ranges {746508=500317,2009201=795686,3220075=326433} 2021-02-10T04:39:42.271 INFO:tasks.rados.rados.0.smithi085.stdout:3002: writing smithi08575397-43 from 746508 to 1246825 tid 1 2021-02-10T04:39:42.282 INFO:tasks.rados.rados.0.smithi085.stdout:3002: writing smithi08575397-43 from 2009201 to 2804887 tid 2 2021-02-10T04:39:42.287 INFO:tasks.rados.rados.0.smithi085.stdout:3002: writing smithi08575397-43 from 3220075 to 3546508 tid 3 2021-02-10T04:39:42.287 INFO:tasks.rados.rados.0.smithi085.stdout:3001: finishing rollback tid 0 to smithi08575397-23 2021-02-10T04:39:42.287 INFO:tasks.rados.rados.0.smithi085.stdout:3001: finishing rollback tid 1 to smithi08575397-23 2021-02-10T04:39:42.287 INFO:tasks.rados.rados.0.smithi085.stdout:3003: write oid 7 current snap is 341 2021-02-10T04:39:42.287 INFO:tasks.rados.rados.0.smithi085.stdout:3003: seq_num 1045 ranges {763315=575566,2067683=655697,3247498=315817} 2021-02-10T04:39:42.296 INFO:tasks.rados.rados.0.smithi085.stdout:3003: writing smithi08575397-7 from 763315 to 1338881 tid 1 2021-02-10T04:39:42.304 INFO:tasks.rados.rados.0.smithi085.stdout:3003: writing smithi08575397-7 from 2067683 to 2723380 tid 2 2021-02-10T04:39:42.310 INFO:tasks.rados.rados.0.smithi085.stdout:3003: writing smithi08575397-7 from 3247498 to 3563315 tid 3 2021-02-10T04:39:42.310 INFO:tasks.rados.rados.0.smithi085.stdout:3001: finishing rollback tid 2 to smithi08575397-23 2021-02-10T04:39:42.310 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 1292 (ObjNum 924 snap 306 seq_num 924) dirty exists 2021-02-10T04:39:42.310 INFO:tasks.rados.rados.0.smithi085.stdout:3002: finishing write tid 1 to smithi08575397-43 2021-02-10T04:39:42.310 INFO:tasks.rados.rados.0.smithi085.stdout:3002: finishing write tid 2 to smithi08575397-43 2021-02-10T04:39:42.310 INFO:tasks.rados.rados.0.smithi085.stdout:3002: finishing write tid 3 to smithi08575397-43 2021-02-10T04:39:42.310 INFO:tasks.rados.rados.0.smithi085.stdout:3002: finishing write tid 4 to smithi08575397-43 2021-02-10T04:39:42.311 INFO:tasks.rados.rados.0.smithi085.stdout:3002: finishing write tid 5 to smithi08575397-43 2021-02-10T04:39:42.311 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 1366 (ObjNum 1044 snap 341 seq_num 1044) dirty exists 2021-02-10T04:39:42.311 INFO:tasks.rados.rados.0.smithi085.stdout:3002: left oid 43 (ObjNum 1044 snap 341 seq_num 1044) 2021-02-10T04:39:42.311 INFO:tasks.rados.rados.0.smithi085.stdout:3003: finishing write tid 1 to smithi08575397-7 2021-02-10T04:39:42.311 INFO:tasks.rados.rados.0.smithi085.stdout:3003: finishing write tid 2 to smithi08575397-7 2021-02-10T04:39:42.311 INFO:tasks.rados.rados.0.smithi085.stdout:3001: done (6 left) 2021-02-10T04:39:42.311 INFO:tasks.rados.rados.0.smithi085.stdout:3002: done (5 left) 2021-02-10T04:39:42.311 INFO:tasks.rados.rados.0.smithi085.stdout:3004: copy_from oid 36 from oid 34 current snap is 341 2021-02-10T04:39:42.311 INFO:tasks.rados.rados.0.smithi085.stdout:3005: snap_create 2021-02-10T04:39:42.315 INFO:tasks.rados.rados.0.smithi085.stdout:3003: finishing write tid 3 to smithi08575397-7 2021-02-10T04:39:42.315 INFO:tasks.rados.rados.0.smithi085.stdout:3003: finishing write tid 4 to smithi08575397-7 2021-02-10T04:39:42.316 INFO:tasks.rados.rados.0.smithi085.stdout:3003: finishing write tid 5 to smithi08575397-7 2021-02-10T04:39:42.316 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 1370 (ObjNum 1045 snap 341 seq_num 1045) dirty exists 2021-02-10T04:39:42.316 INFO:tasks.rados.rados.0.smithi085.stdout:3003: left oid 7 (ObjNum 1045 snap 341 seq_num 1045) 2021-02-10T04:39:42.317 INFO:tasks.rados.rados.0.smithi085.stdout:3004: finishing copy_from to smithi08575397-36 2021-02-10T04:39:42.317 INFO:tasks.rados.rados.0.smithi085.stdout:3004: got expected ENOENT (src dne) 2021-02-10T04:39:42.317 INFO:tasks.rados.rados.0.smithi085.stdout:3004: finishing copy_from racing read to smithi08575397-36 2021-02-10T04:39:42.583 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:39:42.636 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:39:42 smithi085 conmon[49106]: debug 2021-02-10T04:39:42.348+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:42.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:42 smithi085 conmon[29319]: cluster 2021-02-10T04:39:42.180194+0000 2021-02-10T04:39:42.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:42 smithi085 conmon[29319]: mgr.y (mgr.14140) 757 : cluster [DBG] pgmap v1294: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 530 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 9.2 MiB/s rd, 2.4 MiB/s wr, 15 op/s 2021-02-10T04:39:42.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:42 smithi085 conmon[29319]: cluster 2021-02-10T04:39:42.261746+0000 mon.a (mon.0) 1690 : cluster [DBG] osdmap e763: 8 total, 8 up, 7 in 2021-02-10T04:39:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:42 smithi164 conmon[29288]: cluster 2021-02-10T04:39:42.180194 2021-02-10T04:39:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:42 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 757 : cluster [DBG] pgmap v1294: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 530 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 9.2 MiB/s rd, 2.4 MiB/s wr, 15 op/s 2021-02-10T04:39:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:42 smithi164 conmon[29288]: cluster 2021-02-10T04:39:42.261746+0000 2021-02-10T04:39:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:42 smithi164 conmon[29288]: mon.a (mon.0) 1690 : cluster [DBG] osdmap e763: 8 total, 8 up, 7 in 2021-02-10T04:39:42.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:42 smithi085 conmon[35961]: cluster 2021-02-10T04:39:42.180194+0000 mgr.y ( 2021-02-10T04:39:42.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:42 smithi085 conmon[35961]: mgr.14140) 757 : cluster [DBG] pgmap v1294: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 530 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 9.2 MiB/s rd, 2.4 MiB/s wr, 15 op/s 2021-02-10T04:39:42.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:42 smithi085 conmon[35961]: cluster 2021-02-10T04:39:42.261746+0000 mon.a ( 2021-02-10T04:39:42.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:42 smithi085 conmon[35961]: mon.0) 1690 : cluster [DBG] osdmap e763: 8 total, 8 up, 7 in 2021-02-10T04:39:43.066 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:39:43.137 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:39:43.268 INFO:tasks.rados.rados.0.smithi085.stdout:3003: done (6 left) 2021-02-10T04:39:43.268 INFO:tasks.rados.rados.0.smithi085.stdout:3004: done (5 left) 2021-02-10T04:39:43.268 INFO:tasks.rados.rados.0.smithi085.stdout:3005: done (4 left) 2021-02-10T04:39:43.269 INFO:tasks.rados.rados.0.smithi085.stdout:3006: copy_from oid 44 from oid 22 current snap is 342 2021-02-10T04:39:43.269 INFO:tasks.rados.rados.0.smithi085.stdout:3007: snap_remove snap 237 2021-02-10T04:39:43.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:39:43 smithi164 conmon[37263]: debug 2021-02-10T04:39:43.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:39:43 smithi164 conmon[37263]: 2021-02-10T04:39:43.238+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:43.605 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:39:43.710 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:39:43.835 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:39:44.273 INFO:tasks.rados.rados.0.smithi085.stdout:3007: done (5 left) 2021-02-10T04:39:44.273 INFO:tasks.rados.rados.0.smithi085.stdout:3008: snap_create 2021-02-10T04:39:44.273 INFO:tasks.rados.rados.0.smithi085.stdout:3006: finishing copy_from to smithi08575397-44 2021-02-10T04:39:44.273 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 1435 (ObjNum 966 snap 315 seq_num 966) dirty exists 2021-02-10T04:39:44.273 INFO:tasks.rados.rados.0.smithi085.stdout:3006: finishing copy_from racing read to smithi08575397-44 2021-02-10T04:39:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:44 smithi164 conmon[29288]: cluster 2021-02-10T04:39:43.267763+0000 2021-02-10T04:39:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:44 smithi164 conmon[29288]: mon.a (mon.0) 1691 : cluster [DBG] osdmap e764: 8 total, 8 up, 7 in 2021-02-10T04:39:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:44 smithi164 conmon[29288]: cluster 2021-02-10T04:39:44.180692+0000 mgr.y (mgr.14140) 758 : cluster [DBG] 2021-02-10T04:39:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:44 smithi164 conmon[29288]: pgmap v1297: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 523 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 1.7 MiB/s rd, 661 KiB/s wr, 7 op/s 2021-02-10T04:39:44.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:44 smithi085 conmon[29319]: cluster 2021-02-10T04:39:43.267763+0000 2021-02-10T04:39:44.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:44 smithi085 conmon[29319]: mon.a (mon.0) 1691 : cluster [DBG] osdmap e764: 8 total, 8 up, 7 in 2021-02-10T04:39:44.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:44 smithi085 conmon[29319]: cluster 2021-02-10T04:39:44.180692+0000 mgr.y (mgr.14140) 758 : cluster [DBG] pgmap v1297: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 523 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 1.7 MiB/s rd, 661 KiB/s wr, 7 op/s 2021-02-10T04:39:44.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:44 smithi085 conmon[35961]: cluster 2021-02-10T04:39:43.267763+0000 mon.a (mon.0) 2021-02-10T04:39:44.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:44 smithi085 conmon[35961]: 1691 : cluster [DBG] osdmap e764: 8 total, 8 up, 7 in 2021-02-10T04:39:44.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:44 smithi085 conmon[35961]: cluster 2021-02-10T04:39:44.180692+0000 mgr.y (mgr.14140) 758 : 2021-02-10T04:39:44.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:44 smithi085 conmon[35961]: cluster [DBG] pgmap v1297: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 523 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 1.7 MiB/s rd, 661 KiB/s wr, 7 op/s 2021-02-10T04:39:45.275 INFO:tasks.rados.rados.0.smithi085.stdout:3006: done (5 left) 2021-02-10T04:39:45.276 INFO:tasks.rados.rados.0.smithi085.stdout:3008: done (4 left) 2021-02-10T04:39:45.276 INFO:tasks.rados.rados.0.smithi085.stdout:3009: read oid 18 snap 326 2021-02-10T04:39:45.276 INFO:tasks.rados.rados.0.smithi085.stdout:3009: expect (ObjNum 952 snap 313 seq_num 952) 2021-02-10T04:39:45.276 INFO:tasks.rados.rados.0.smithi085.stdout:3010: rollback oid 34 current snap is 343 2021-02-10T04:39:45.276 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 34 to 272 2021-02-10T04:39:45.276 INFO:tasks.rados.rados.0.smithi085.stdout:3011: write (excl) oid 5 current snap is 343 2021-02-10T04:39:45.276 INFO:tasks.rados.rados.0.smithi085.stdout:3011: seq_num 1046 ranges {780122=650815,2126165=799355} 2021-02-10T04:39:45.284 INFO:tasks.rados.rados.0.smithi085.stdout:3011: writing smithi08575397-5 from 780122 to 1430937 tid 1 2021-02-10T04:39:45.293 INFO:tasks.rados.rados.0.smithi085.stdout:3011: writing smithi08575397-5 from 2126165 to 2925520 tid 2 2021-02-10T04:39:45.293 INFO:tasks.rados.rados.0.smithi085.stdout:3012: snap_create 2021-02-10T04:39:45.293 INFO:tasks.rados.rados.0.smithi085.stdout:3009: expect (ObjNum 952 snap 313 seq_num 952) 2021-02-10T04:39:45.454 INFO:tasks.rados.rados.0.smithi085.stdout:3010: finishing rollback tid 1 to smithi08575397-34 2021-02-10T04:39:45.454 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 34 v 808 (ObjNum 1215201120 snap 32576 seq_num 1215201120) dirty dne 2021-02-10T04:39:45.454 INFO:tasks.rados.rados.0.smithi085.stdout:3011: finishing write tid 1 to smithi08575397-5 2021-02-10T04:39:45.454 INFO:tasks.rados.rados.0.smithi085.stdout:3011: finishing write tid 2 to smithi08575397-5 2021-02-10T04:39:45.454 INFO:tasks.rados.rados.0.smithi085.stdout:3011: finishing write tid 3 to smithi08575397-5 2021-02-10T04:39:45.454 INFO:tasks.rados.rados.0.smithi085.stdout:3011: finishing write tid 4 to smithi08575397-5 2021-02-10T04:39:45.454 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 813 (ObjNum 1046 snap 343 seq_num 1046) dirty exists 2021-02-10T04:39:45.454 INFO:tasks.rados.rados.0.smithi085.stdout:3011: left oid 5 (ObjNum 1046 snap 343 seq_num 1046) 2021-02-10T04:39:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:45 smithi164 conmon[29288]: cluster 2021-02-10T04:39:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:45 smithi164 conmon[29288]: 2021-02-10T04:39:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:45 smithi164 conmon[29288]: 2021 2021-02-10T04:39:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:45 smithi164 conmon[29288]: - 2021-02-10T04:39:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:45 smithi164 conmon[29288]: 02 2021-02-10T04:39:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:45 smithi164 conmon[29288]: -10T04:39: 2021-02-10T04:39:45.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:45 smithi164 conmon[29288]: 44.273006+0000 mon.a (mon.0) 1692 : cluster [DBG] osdmap e765: 8 total, 8 up, 7 in 2021-02-10T04:39:45.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:45 smithi085 conmon[29319]: cluster 2021-02-10T04:39:44.273006+0000 mon.a (mon.0) 2021-02-10T04:39:45.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:45 smithi085 conmon[29319]: 1692 : cluster [DBG] osdmap e765: 8 total, 8 up, 7 in 2021-02-10T04:39:45.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:45 smithi085 conmon[35961]: cluster 2021-02-10T04:39:45.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:45 smithi085 conmon[35961]: 2021-02-10T04:39:45.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:45 smithi085 conmon[35961]: 2021 2021-02-10T04:39:45.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:45 smithi085 conmon[35961]: -02-10T04:39: 2021-02-10T04:39:45.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:45 smithi085 conmon[35961]: 44.273006+0000 mon.a (mon.0) 1692 : cluster [DBG] osdmap e765: 8 total, 8 up, 7 in 2021-02-10T04:39:45.866 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:39:45.972 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:39:46.002 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:39:46.114 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:39:45 smithi085 conmon[49106]: debug 2021-02-10T04:39:45.774+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:46.190 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:39:46.190 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:39:46.288 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:46 smithi164 conmon[29288]: cluster 2021-02-10T04:39:46.288 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:46 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:39:46.289 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:46 smithi164 conmon[29288]: :39:45.273875+0000 mon.a (mon.0) 1693 : cluster [DBG] osdmap e766: 8 total, 8 up, 7 in 2021-02-10T04:39:46.289 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:46 smithi164 conmon[29288]: cluster 2021-02-10T04:39:46.181141 2021-02-10T04:39:46.289 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:46 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 759 : cluster [DBG] pgmap v1300: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 526 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 3.9 MiB/s rd, 1.9 MiB/s wr, 14 op/s 2021-02-10T04:39:46.289 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:39:46 smithi164 conmon[44544]: debug 2021-02-10T04:39:46.078+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:46.291 INFO:tasks.rados.rados.0.smithi085.stdout:3009: done (7 left) 2021-02-10T04:39:46.291 INFO:tasks.rados.rados.0.smithi085.stdout:3010: done (6 left) 2021-02-10T04:39:46.291 INFO:tasks.rados.rados.0.smithi085.stdout:3011: done (5 left) 2021-02-10T04:39:46.291 INFO:tasks.rados.rados.0.smithi085.stdout:3012: done (4 left) 2021-02-10T04:39:46.291 INFO:tasks.rados.rados.0.smithi085.stdout:3013: delete oid 5 current snap is 344 2021-02-10T04:39:46.300 INFO:tasks.rados.rados.0.smithi085.stdout:3013: done (4 left) 2021-02-10T04:39:46.300 INFO:tasks.rados.rados.0.smithi085.stdout:3014: delete oid 12 current snap is 344 2021-02-10T04:39:46.302 INFO:tasks.rados.rados.0.smithi085.stdout:3014: done (4 left) 2021-02-10T04:39:46.302 INFO:tasks.rados.rados.0.smithi085.stdout:3015: copy_from oid 25 from oid 32 current snap is 344 2021-02-10T04:39:46.303 INFO:tasks.rados.rados.0.smithi085.stdout:3016: snap_remove snap 330 2021-02-10T04:39:46.354 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:39:46.410 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:46 smithi085 conmon[29319]: cluster 2021-02-10T04:39:45.273875+0000 mon.a (mon.0) 1693 : cluster [DBG] osdmap e766: 8 total, 8 up, 7 in 2021-02-10T04:39:46.410 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:46 smithi085 conmon[29319]: cluster 2021-02-10T04:39:46.181141+0000 mgr.y (mgr.14140) 759 : cluster [DBG] pgmap v1300: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 526 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 3.9 MiB/s rd, 1.9 MiB/s wr, 14 op/s 2021-02-10T04:39:46.410 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:46 smithi085 conmon[35961]: cluster 2021-02-10T04:39:45.273875+0000 2021-02-10T04:39:46.411 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:46 smithi085 conmon[35961]: mon.a (mon.0) 1693 : cluster [DBG] osdmap e766: 8 total, 8 up, 7 in 2021-02-10T04:39:46.411 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:46 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:39:46.411 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:46 smithi085 conmon[35961]: 02-10T04:39:46.181141+0000 mgr.y (mgr.14140) 759 : cluster 2021-02-10T04:39:46.411 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:46 smithi085 conmon[35961]: [DBG] pgmap v1300: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 526 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 3.9 MiB/s rd, 1.9 MiB/s wr, 14 op/s 2021-02-10T04:39:46.459 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:39:46.487 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:39:46.691 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:39:46 smithi085 conmon[41540]: debug 2021-02-10T04:39:46.572+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:46.823 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:39:47.298 INFO:tasks.rados.rados.0.smithi085.stdout:3016: done (5 left) 2021-02-10T04:39:47.298 INFO:tasks.rados.rados.0.smithi085.stdout:3017: read oid 37 snap -1 2021-02-10T04:39:47.298 INFO:tasks.rados.rados.0.smithi085.stdout:3017: expect (ObjNum 646 snap 196 seq_num 646) 2021-02-10T04:39:47.299 INFO:tasks.rados.rados.0.smithi085.stdout:3018: snap_remove snap 321 2021-02-10T04:39:47.303 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.0 to 1 (10000) 2021-02-10T04:39:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:47 smithi164 conmon[29288]: cluster 2021-02-10T04:39:46.290303+0000 mon.a (mon.0) 1694 : cluster [DBG] 2021-02-10T04:39:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:47 smithi164 conmon[29288]: osdmap e767: 8 total, 8 up, 7 in 2021-02-10T04:39:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:47 smithi164 conmon[29288]: cluster 2021-02-10T04:39:46.442828+0000 osd.4 (osd.4) 1253 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:39:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:47 smithi164 conmon[29288]: cluster 2021-02-10T04:39:46.443257+0000 osd.4 (osd.4) 1254 2021-02-10T04:39:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:47 smithi164 conmon[29288]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:39:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:47 smithi164 conmon[29288]: audit 2021-02-10T04:39:46.623707+0000 2021-02-10T04:39:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:47 smithi164 conmon[29288]: mon.c (mon.1) 65 : audit [INF] from='client.? 172.21.15.85:0/3895037977' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T04:39:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:47 smithi164 conmon[29288]: audit 2021-02-10T04:39:46.624130+0000 2021-02-10T04:39:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:47 smithi164 conmon[29288]: mon.a (mon.0) 1695 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T04:39:47.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[29319]: cluster 2021-02-10T04:39:47.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[29319]: 2021-02-10T04:39:47.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[29319]: 2021 2021-02-10T04:39:47.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[29319]: - 2021-02-10T04:39:47.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[29319]: 02 2021-02-10T04:39:47.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[29319]: -10T04:39: 2021-02-10T04:39:47.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[29319]: 46.290303+0000 mon.a (mon.0) 1694 : cluster [DBG] 2021-02-10T04:39:47.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[29319]: osdmap e767: 8 total, 8 up, 7 in 2021-02-10T04:39:47.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[29319]: cluster 2021-02-10T04:39:47.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[29319]: 2021-02-10T04:39: 2021-02-10T04:39:47.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[29319]: 46.442828+0000 osd.4 (osd.4) 1253 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:39:47.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:39:47.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[29319]: :39:46.443257+0000 osd.4 (osd.4) 1254 : cluster [DBG] 2021-02-10T04:39:47.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[29319]: purged_snaps scrub ok 2021-02-10T04:39:47.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:39:47.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[29319]: 10T04:39:46.623707+0000 mon.c (mon.1) 65 2021-02-10T04:39:47.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[29319]: : audit [INF] from='client.? 172.21.15.85:0/3895037977' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T04:39:47.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:39:47.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[29319]: -10T04:39:46.624130+0000 mon.a (mon 2021-02-10T04:39:47.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[29319]: .0) 1695 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T04:39:47.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[35961]: cluster 2021-02-10T04:39:47.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[35961]: 2021-02-10T04:39:47.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[35961]: 2021 2021-02-10T04:39:47.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[35961]: - 2021-02-10T04:39:47.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[35961]: 02 2021-02-10T04:39:47.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[35961]: - 2021-02-10T04:39:47.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[35961]: 10 2021-02-10T04:39:47.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[35961]: T04:39:46.290303+0000 mon.a (mon.0 2021-02-10T04:39:47.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[35961]: ) 1694 : cluster [DBG] osdmap e767: 8 total, 8 up, 7 in 2021-02-10T04:39:47.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[35961]: cluster 2021-02-10T04:39:46.442828+0000 osd.4 (osd.4) 1253 : cluster 2021-02-10T04:39:47.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[35961]: [DBG] purged_snaps scrub starts 2021-02-10T04:39:47.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[35961]: cluster 2021-02-10T04:39:46. 2021-02-10T04:39:47.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[35961]: 443257+0000 osd.4 (osd 2021-02-10T04:39:47.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[35961]: .4) 1254 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:39:47.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[35961]: audit 2021-02-10T04:39:46 2021-02-10T04:39:47.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[35961]: .623707+0000 mon.c (mon.1 2021-02-10T04:39:47.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[35961]: ) 65 : audit [INF] from='client.? 172.21.15.85:0/3895037977' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T04:39:47.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[35961]: audit 2021-02-10T04:39 2021-02-10T04:39:47.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[35961]: :46.624130+0000 mon.a (mon.0) 1695 : audit 2021-02-10T04:39:47.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:47 smithi085 conmon[35961]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T04:39:47.830 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set nodeep-scrub 2021-02-10T04:39:47.848 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:39:48.257 INFO:tasks.rados.rados.0.smithi085.stdout:3018: done (6 left) 2021-02-10T04:39:48.257 INFO:tasks.rados.rados.0.smithi085.stdout:3019: write oid 4 current snap is 344 2021-02-10T04:39:48.257 INFO:tasks.rados.rados.0.smithi085.stdout:3019: seq_num 1049 ranges {430543=476562,1385258=663035,2840837=669165} 2021-02-10T04:39:48.262 INFO:tasks.rados.rados.0.smithi085.stdout:3019: writing smithi08575397-4 from 430543 to 907105 tid 1 2021-02-10T04:39:48.269 INFO:tasks.rados.rados.0.smithi085.stdout:3019: writing smithi08575397-4 from 1385258 to 2048293 tid 2 2021-02-10T04:39:48.279 INFO:tasks.rados.rados.0.smithi085.stdout:3019: writing smithi08575397-4 from 2840837 to 3510002 tid 3 2021-02-10T04:39:48.288 INFO:tasks.rados.rados.0.smithi085.stdout:3015: finishing copy_from to smithi08575397-25 2021-02-10T04:39:48.288 INFO:tasks.rados.rados.0.smithi085.stdout:3015: got expected ENOENT (src dne) 2021-02-10T04:39:48.288 INFO:tasks.rados.rados.0.smithi085.stdout:3015: finishing copy_from racing read to smithi08575397-25 2021-02-10T04:39:48.288 INFO:tasks.rados.rados.0.smithi085.stdout:3017: expect (ObjNum 646 snap 196 seq_num 646) 2021-02-10T04:39:48.442 INFO:tasks.rados.rados.0.smithi085.stdout:3019: finishing write tid 1 to smithi08575397-4 2021-02-10T04:39:48.443 INFO:tasks.rados.rados.0.smithi085.stdout:3019: finishing write tid 2 to smithi08575397-4 2021-02-10T04:39:48.443 INFO:tasks.rados.rados.0.smithi085.stdout:3015: done (6 left) 2021-02-10T04:39:48.443 INFO:tasks.rados.rados.0.smithi085.stdout:3017: done (5 left) 2021-02-10T04:39:48.443 INFO:tasks.rados.rados.0.smithi085.stdout:3020: write oid 32 current snap is 344 2021-02-10T04:39:48.443 INFO:tasks.rados.rados.0.smithi085.stdout:3019: finishing write tid 3 to smithi08575397-4 2021-02-10T04:39:48.443 INFO:tasks.rados.rados.0.smithi085.stdout:3019: finishing write tid 4 to smithi08575397-4 2021-02-10T04:39:48.443 INFO:tasks.rados.rados.0.smithi085.stdout:3019: finishing write tid 5 to smithi08575397-4 2021-02-10T04:39:48.443 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 4 v 1350 (ObjNum 1049 snap 344 seq_num 1049) dirty exists 2021-02-10T04:39:48.443 INFO:tasks.rados.rados.0.smithi085.stdout:3019: left oid 4 (ObjNum 1049 snap 344 seq_num 1049) 2021-02-10T04:39:48.443 INFO:tasks.rados.rados.0.smithi085.stdout:3020: seq_num 1050 ranges {447350=668164,1560093=406693,2584613=480437} 2021-02-10T04:39:48.449 INFO:tasks.rados.rados.0.smithi085.stdout:3020: writing smithi08575397-32 from 447350 to 1115514 tid 1 2021-02-10T04:39:48.454 INFO:tasks.rados.rados.0.smithi085.stdout:3020: writing smithi08575397-32 from 1560093 to 1966786 tid 2 2021-02-10T04:39:48.462 INFO:tasks.rados.rados.0.smithi085.stdout:3020: writing smithi08575397-32 from 2584613 to 3065050 tid 3 2021-02-10T04:39:48.462 INFO:tasks.rados.rados.0.smithi085.stdout:3020: finishing write tid 1 to smithi08575397-32 2021-02-10T04:39:48.462 INFO:tasks.rados.rados.0.smithi085.stdout:3020: finishing write tid 2 to smithi08575397-32 2021-02-10T04:39:48.463 INFO:tasks.rados.rados.0.smithi085.stdout:3019: done (5 left) 2021-02-10T04:39:48.463 INFO:tasks.rados.rados.0.smithi085.stdout:3021: read oid 9 snap -1 2021-02-10T04:39:48.463 INFO:tasks.rados.rados.0.smithi085.stdout:3021: expect (ObjNum 1038 snap 340 seq_num 1038) 2021-02-10T04:39:48.463 INFO:tasks.rados.rados.0.smithi085.stdout:3022: read oid 42 snap -1 2021-02-10T04:39:48.463 INFO:tasks.rados.rados.0.smithi085.stdout:3022: expect deleted 2021-02-10T04:39:48.463 INFO:tasks.rados.rados.0.smithi085.stdout:3023: read oid 13 snap 272 2021-02-10T04:39:48.463 INFO:tasks.rados.rados.0.smithi085.stdout:3023: expect (ObjNum 829 snap 269 seq_num 829) 2021-02-10T04:39:48.463 INFO:tasks.rados.rados.0.smithi085.stdout:3024: read oid 31 snap -1 2021-02-10T04:39:48.463 INFO:tasks.rados.rados.0.smithi085.stdout:3024: expect deleted 2021-02-10T04:39:48.463 INFO:tasks.rados.rados.0.smithi085.stdout:3025: copy_from oid 22 from oid 2 current snap is 344 2021-02-10T04:39:48.463 INFO:tasks.rados.rados.0.smithi085.stdout:3026: snap_create 2021-02-10T04:39:48.483 INFO:tasks.rados.rados.0.smithi085.stdout:3023: expect (ObjNum 829 snap 269 seq_num 829) 2021-02-10T04:39:48.580 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:39:48.584 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:39:48.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:48 smithi164 conmon[29288]: audit 2021-02-10T04:39:47.297305+0000 mon.a ( 2021-02-10T04:39:48.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:48 smithi164 conmon[29288]: mon.0) 1696 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2021-02-10T04:39:48.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:48 smithi164 conmon[29288]: cluster 2021-02-10T04:39:47.297382+0000 mon.a 2021-02-10T04:39:48.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:48 smithi164 conmon[29288]: (mon.0) 1697 : cluster [DBG] osdmap e768: 8 total, 8 up, 7 in 2021-02-10T04:39:48.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:48 smithi164 conmon[29288]: cluster 2021-02-10T04:39: 2021-02-10T04:39:48.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:48 smithi164 conmon[29288]: 48.181727+0000 mgr.y (mgr.14140) 760 : cluster [DBG] pgmap v1303: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 522 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 8 op/s 2021-02-10T04:39:48.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:48 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:39:48.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:48 smithi164 conmon[29288]: 10T04:39:48.252964+0000 mon.a (mon.0) 1698 : cluster 2021-02-10T04:39:48.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:48 smithi164 conmon[29288]: [DBG] osdmap e769: 8 total, 8 up, 7 in 2021-02-10T04:39:48.649 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:39:48 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:39:48.654 INFO:tasks.rados.rados.0.smithi085.stdout:3025: finishing copy_from to smithi08575397-22 2021-02-10T04:39:48.654 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 1453 (ObjNum 982 snap 317 seq_num 982) dirty exists 2021-02-10T04:39:48.654 INFO:tasks.rados.rados.0.smithi085.stdout:3025: finishing copy_from racing read to smithi08575397-22 2021-02-10T04:39:48.654 INFO:tasks.rados.rados.0.smithi085.stdout:3021: expect (ObjNum 1038 snap 340 seq_num 1038) 2021-02-10T04:39:48.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:48 smithi085 conmon[29319]: audit 2021-02-10T04:39:47.297305 2021-02-10T04:39:48.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:48 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1696 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2021-02-10T04:39:48.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:48 smithi085 conmon[29319]: cluster 2021-02-10T04:39:47.297382+0000 mon.a (mon.0) 1697 : cluster [DBG] osdmap e768: 8 total, 8 up, 7 in 2021-02-10T04:39:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:48 smithi085 conmon[29319]: cluster 2021-02-10T04:39:48.181727+0000 mgr.y (mgr.14140) 760 : cluster [DBG] pgmap v1303: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 522 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 8 op/s 2021-02-10T04:39:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:48 smithi085 conmon[29319]: cluster 2021-02-10T04:39:48.252964+0000 mon.a (mon.0) 1698 : cluster [DBG] osdmap e769: 8 total, 8 up, 7 in 2021-02-10T04:39:48.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:48 smithi085 conmon[35961]: audit 2021-02-10T04:39:47.297305+0000 2021-02-10T04:39:48.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:48 smithi085 conmon[35961]: mon.a (mon.0) 1696 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2021-02-10T04:39:48.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:48 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:39:48.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:48 smithi085 conmon[35961]: 10T04:39:47.297382+0000 mon.a (mon.0) 1697 : cluster [DBG] osdmap e768: 8 total, 8 up, 7 in 2021-02-10T04:39:48.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:48 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:39:48.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:48 smithi085 conmon[35961]: 39:48.181727+0000 mgr.y (mgr.14140) 760 2021-02-10T04:39:48.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:48 smithi085 conmon[35961]: : cluster [DBG] pgmap v1303: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 522 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 8 op/s 2021-02-10T04:39:48.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:48 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:39:48.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:48 smithi085 conmon[35961]: 39:48.252964+0000 mon.a (mon.0) 1698 2021-02-10T04:39:48.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:48 smithi085 conmon[35961]: : cluster [DBG] osdmap e769: 8 total, 8 up, 7 in 2021-02-10T04:39:48.846 INFO:tasks.rados.rados.0.smithi085.stdout:3020: finishing write tid 3 to smithi08575397-32 2021-02-10T04:39:48.846 INFO:tasks.rados.rados.0.smithi085.stdout:3020: finishing write tid 4 to smithi08575397-32 2021-02-10T04:39:48.846 INFO:tasks.rados.rados.0.smithi085.stdout:3020: finishing write tid 5 to smithi08575397-32 2021-02-10T04:39:48.846 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 32 v 1375 (ObjNum 1050 snap 344 seq_num 1050) dirty exists 2021-02-10T04:39:48.847 INFO:tasks.rados.rados.0.smithi085.stdout:3020: left oid 32 (ObjNum 1050 snap 344 seq_num 1050) 2021-02-10T04:39:49.348 INFO:tasks.rados.rados.0.smithi085.stdout:3020: done (10 left) 2021-02-10T04:39:49.519 INFO:tasks.rados.rados.0.smithi085.stdout:3021: done (9 left) 2021-02-10T04:39:49.519 INFO:tasks.rados.rados.0.smithi085.stdout:3022: done (8 left) 2021-02-10T04:39:49.520 INFO:tasks.rados.rados.0.smithi085.stdout:3023: done (7 left) 2021-02-10T04:39:49.520 INFO:tasks.rados.rados.0.smithi085.stdout:3024: done (6 left) 2021-02-10T04:39:49.520 INFO:tasks.rados.rados.0.smithi085.stdout:3025: done (5 left) 2021-02-10T04:39:49.520 INFO:tasks.rados.rados.0.smithi085.stdout:3026: done (4 left) 2021-02-10T04:39:49.520 INFO:tasks.rados.rados.0.smithi085.stdout:3027: rollback oid 36 current snap is 345 2021-02-10T04:39:49.520 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 36 to 303 2021-02-10T04:39:49.521 INFO:tasks.rados.rados.0.smithi085.stdout:3028: snap_remove snap 341 2021-02-10T04:39:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:49 smithi164 conmon[29288]: audit 2021-02-10T04:39:48.356952 2021-02-10T04:39:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:49 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 761 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:39:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:49 smithi085 conmon[29319]: audit 2021-02-10T04:39:48.356952 2021-02-10T04:39:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:49 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 761 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:39:49.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:49 smithi085 conmon[35961]: audit 2021-02-10T04:39:48.356952 2021-02-10T04:39:49.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:49 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 761 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:39:50.352 INFO:tasks.rados.rados.0.smithi085.stdout:3028: done (5 left) 2021-02-10T04:39:50.352 INFO:tasks.rados.rados.0.smithi085.stdout:3029: write oid 1 current snap is 345 2021-02-10T04:39:50.352 INFO:tasks.rados.rados.0.smithi085.stdout:3027: finishing rollback tid 0 to smithi08575397-36 2021-02-10T04:39:50.352 INFO:tasks.rados.rados.0.smithi085.stdout:3029: seq_num 1051 ranges {464157=743413,1618575=666704,3012036=408062} 2021-02-10T04:39:50.360 INFO:tasks.rados.rados.0.smithi085.stdout:3029: writing smithi08575397-1 from 464157 to 1207570 tid 1 2021-02-10T04:39:50.367 INFO:tasks.rados.rados.0.smithi085.stdout:3029: writing smithi08575397-1 from 1618575 to 2285279 tid 2 2021-02-10T04:39:50.374 INFO:tasks.rados.rados.0.smithi085.stdout:3029: writing smithi08575397-1 from 3012036 to 3420098 tid 3 2021-02-10T04:39:50.374 INFO:tasks.rados.rados.0.smithi085.stdout:3030: snap_remove snap 336 2021-02-10T04:39:50.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:50 smithi164 conmon[29288]: cluster 2021-02-10T04:39:49.348465+0000 2021-02-10T04:39:50.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:50 smithi164 conmon[29288]: mon.a (mon.0) 1699 : cluster [DBG] osdmap e770: 8 total, 8 up, 7 in 2021-02-10T04:39:50.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:50 smithi164 conmon[29288]: cluster 2021-02-10T04:39:50.182492+0000 mgr.y (mgr.14140) 762 : cluster [DBG] 2021-02-10T04:39:50.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:50 smithi164 conmon[29288]: pgmap v1306: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 492 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 4.6 MiB/s rd, 1.4 MiB/s wr, 10 op/s 2021-02-10T04:39:50.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:50 smithi085 conmon[29319]: cluster 2021-02-10T04:39:50.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:50 smithi085 conmon[29319]: 2021-02-10T04:39:50.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:50 smithi085 conmon[29319]: 2021 2021-02-10T04:39:50.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:50 smithi085 conmon[29319]: - 2021-02-10T04:39:50.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:50 smithi085 conmon[29319]: 02-10T04:39:49. 2021-02-10T04:39:50.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:50 smithi085 conmon[29319]: 348465+0000 mon.a (mon.0) 1699 : cluster [DBG] osdmap e770: 8 total, 8 up, 7 in 2021-02-10T04:39:50.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:50 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:39:50.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:50 smithi085 conmon[29319]: 02-10T04:39:50.182492+0000 mgr.y 2021-02-10T04:39:50.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:50 smithi085 conmon[29319]: (mgr.14140) 762 : cluster 2021-02-10T04:39:50.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:50 smithi085 conmon[29319]: [DBG] pgmap v1306: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 492 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 4.6 MiB/s rd, 1.4 MiB/s wr, 10 op/s 2021-02-10T04:39:50.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:50 smithi085 conmon[35961]: cluster 2021-02-10T04:39:50.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:50 smithi085 conmon[35961]: 2021-02-10T04:39: 2021-02-10T04:39:50.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:50 smithi085 conmon[35961]: 49.348465+0000 mon.a (mon.0) 1699 : cluster [DBG] osdmap e770: 8 total, 8 up, 7 in 2021-02-10T04:39:50.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:50 smithi085 conmon[35961]: cluster 2021-02-10T04:39: 2021-02-10T04:39:50.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:50 smithi085 conmon[35961]: 50.182492+0000 mgr.y (mgr.14140) 762 : cluster [DBG] 2021-02-10T04:39:50.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:50 smithi085 conmon[35961]: pgmap v1306: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 492 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 4.6 MiB/s rd, 1.4 MiB/s wr, 10 op/s 2021-02-10T04:39:51.368 INFO:tasks.rados.rados.0.smithi085.stdout:3030: done (6 left) 2021-02-10T04:39:51.368 INFO:tasks.rados.rados.0.smithi085.stdout:3031: read oid 12 snap -1 2021-02-10T04:39:51.368 INFO:tasks.rados.rados.0.smithi085.stdout:3031: expect deleted 2021-02-10T04:39:51.368 INFO:tasks.rados.rados.0.smithi085.stdout:3032: snap_remove snap 331 2021-02-10T04:39:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:51 smithi164 conmon[29288]: cluster 2021-02-10T04:39:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:51 smithi164 conmon[29288]: 2021-02-10T04:39:50.351176 2021-02-10T04:39:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:51 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1700 : cluster [DBG] osdmap e771: 8 total, 8 up, 7 in 2021-02-10T04:39:51.652 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:39:51.682 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:51 smithi085 conmon[29319]: cluster 2021-02-10T04:39:50.351176+0000 2021-02-10T04:39:51.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:51 smithi085 conmon[29319]: mon.a (mon.0) 1700 : cluster [DBG] osdmap e771: 8 total, 8 up, 7 in 2021-02-10T04:39:51.683 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:51 smithi085 conmon[35961]: cluster 2021-02-10T04:39:50.351176+0000 2021-02-10T04:39:51.683 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:51 smithi085 conmon[35961]: mon.a (mon.0) 1700 : cluster [DBG] osdmap e771: 8 total, 8 up, 7 in 2021-02-10T04:39:51.686 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:39:52.371 INFO:tasks.rados.rados.0.smithi085.stdout:3032: done (7 left) 2021-02-10T04:39:52.372 INFO:tasks.rados.rados.0.smithi085.stdout:3033: snap_remove snap 272 2021-02-10T04:39:52.372 INFO:tasks.rados.rados.0.smithi085.stdout:3027: finishing rollback tid 1 to smithi08575397-36 2021-02-10T04:39:52.372 INFO:tasks.rados.rados.0.smithi085.stdout:3027: finishing rollback tid 2 to smithi08575397-36 2021-02-10T04:39:52.372 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 1295 (ObjNum 904 snap 297 seq_num 904) dirty exists 2021-02-10T04:39:52.372 INFO:tasks.rados.rados.0.smithi085.stdout:3029: finishing write tid 1 to smithi08575397-1 2021-02-10T04:39:52.372 INFO:tasks.rados.rados.0.smithi085.stdout:3029: finishing write tid 2 to smithi08575397-1 2021-02-10T04:39:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:52 smithi164 conmon[29288]: cluster 2021-02-10T04:39:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:52 smithi164 conmon[29288]: 2021-02-10T04:39:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:52 smithi164 conmon[29288]: 2021 2021-02-10T04:39:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:52 smithi164 conmon[29288]: - 2021-02-10T04:39:52.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:52 smithi164 conmon[29288]: 02 2021-02-10T04:39:52.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:52 smithi164 conmon[29288]: - 2021-02-10T04:39:52.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:52 smithi164 conmon[29288]: 10 2021-02-10T04:39:52.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:52 smithi164 conmon[29288]: T 2021-02-10T04:39:52.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:52 smithi164 conmon[29288]: 04:39:51.367395+0000 mon.a ( 2021-02-10T04:39:52.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:52 smithi164 conmon[29288]: mon.0) 1701 : cluster [DBG] osdmap e772: 8 total, 8 up, 7 in 2021-02-10T04:39:52.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:52 smithi164 conmon[29288]: cluster 2021-02-10T04:39:52 2021-02-10T04:39:52.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:52 smithi164 conmon[29288]: .183165+0000 mgr.y (mgr.14140) 763 : cluster [DBG] 2021-02-10T04:39:52.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:52 smithi164 conmon[29288]: pgmap v1309: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 492 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.6 MiB/s rd, 1.0 MiB/s wr, 7 op/s 2021-02-10T04:39:52.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:52 smithi085 conmon[29319]: cluster 2021-02-10T04:39:52.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:52 smithi085 conmon[29319]: 2021-02-10T04:39:52.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:52 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:39:52.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:52 smithi085 conmon[29319]: :39:51.367395+0000 mon.a (mon.0) 1701 : cluster [DBG] osdmap e772: 8 total, 8 up, 7 in 2021-02-10T04:39:52.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:52 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:39:52.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:52 smithi085 conmon[29319]: 02-10T04:39:52.183165+0000 mgr.y (mgr. 2021-02-10T04:39:52.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:52 smithi085 conmon[29319]: 14140) 763 : cluster [DBG] pgmap v1309: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 492 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.6 MiB/s rd, 1.0 MiB/s wr, 7 op/s 2021-02-10T04:39:52.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:52 smithi085 conmon[35961]: cluster 2021-02-10T04:39:52.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:52 smithi085 conmon[35961]: 2021-02-10T04:39:52.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:52 smithi085 conmon[35961]: 2021 2021-02-10T04:39:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:52 smithi085 conmon[35961]: - 2021-02-10T04:39:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:52 smithi085 conmon[35961]: 02 2021-02-10T04:39:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:52 smithi085 conmon[35961]: - 2021-02-10T04:39:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:52 smithi085 conmon[35961]: 10 2021-02-10T04:39:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:52 smithi085 conmon[35961]: T 2021-02-10T04:39:52.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:52 smithi085 conmon[35961]: 04 2021-02-10T04:39:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:52 smithi085 conmon[35961]: :39:51.367395+0000 mon.a (mon. 2021-02-10T04:39:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:52 smithi085 conmon[35961]: 0) 1701 : cluster [DBG] osdmap e772: 8 total, 8 up, 7 in 2021-02-10T04:39:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:52 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:39:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:52 smithi085 conmon[35961]: -02-10T04:39:52.183165+0000 mgr.y (mgr. 2021-02-10T04:39:52.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:52 smithi085 conmon[35961]: 14140) 763 : cluster [DBG] pgmap v1309: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 492 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.6 MiB/s rd, 1.0 MiB/s wr, 7 op/s 2021-02-10T04:39:53.373 INFO:tasks.rados.rados.0.smithi085.stdout:3027: done (7 left) 2021-02-10T04:39:53.373 INFO:tasks.rados.rados.0.smithi085.stdout:3033: done (6 left) 2021-02-10T04:39:53.373 INFO:tasks.rados.rados.0.smithi085.stdout:3034: write oid 44 current snap is 345 2021-02-10T04:39:53.373 INFO:tasks.rados.rados.0.smithi085.stdout:3029: finishing write tid 3 to smithi08575397-1 2021-02-10T04:39:53.373 INFO:tasks.rados.rados.0.smithi085.stdout:3029: finishing write tid 4 to smithi08575397-1 2021-02-10T04:39:53.374 INFO:tasks.rados.rados.0.smithi085.stdout:3029: finishing write tid 5 to smithi08575397-1 2021-02-10T04:39:53.374 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 1545 (ObjNum 1051 snap 345 seq_num 1051) dirty exists 2021-02-10T04:39:53.374 INFO:tasks.rados.rados.0.smithi085.stdout:3029: left oid 1 (ObjNum 1051 snap 345 seq_num 1051) 2021-02-10T04:39:53.374 INFO:tasks.rados.rados.0.smithi085.stdout:3034: seq_num 1052 ranges {480964=418662,1560704=410362,2523106=619334} 2021-02-10T04:39:53.378 INFO:tasks.rados.rados.0.smithi085.stdout:3034: writing smithi08575397-44 from 480964 to 899626 tid 1 2021-02-10T04:39:53.393 INFO:tasks.rados.rados.0.smithi085.stdout:3034: writing smithi08575397-44 from 1560704 to 1971066 tid 2 2021-02-10T04:39:53.401 INFO:tasks.rados.rados.0.smithi085.stdout:3034: writing smithi08575397-44 from 2523106 to 3142440 tid 3 2021-02-10T04:39:53.401 INFO:tasks.rados.rados.0.smithi085.stdout:3029: done (6 left) 2021-02-10T04:39:53.401 INFO:tasks.rados.rados.0.smithi085.stdout:3031: done (5 left) 2021-02-10T04:39:53.401 INFO:tasks.rados.rados.0.smithi085.stdout:3035: rollback oid 35 current snap is 345 2021-02-10T04:39:53.401 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 35 to 328 2021-02-10T04:39:53.401 INFO:tasks.rados.rados.0.smithi085.stdout:3036: copy_from oid 5 from oid 34 current snap is 345 2021-02-10T04:39:53.401 INFO:tasks.rados.rados.0.smithi085.stdout:3037: copy_from oid 23 from oid 8 current snap is 345 2021-02-10T04:39:53.402 INFO:tasks.rados.rados.0.smithi085.stdout:3034: finishing write tid 1 to smithi08575397-44 2021-02-10T04:39:53.402 INFO:tasks.rados.rados.0.smithi085.stdout:3038: snap_create 2021-02-10T04:39:53.406 INFO:tasks.rados.rados.0.smithi085.stdout:3037: finishing copy_from to smithi08575397-23 2021-02-10T04:39:53.406 INFO:tasks.rados.rados.0.smithi085.stdout:3037: got expected ENOENT (src dne) 2021-02-10T04:39:53.406 INFO:tasks.rados.rados.0.smithi085.stdout:3037: finishing copy_from racing read to smithi08575397-23 2021-02-10T04:39:53.444 INFO:tasks.rados.rados.0.smithi085.stdout:3035: finishing rollback tid 0 to smithi08575397-35 2021-02-10T04:39:53.447 INFO:tasks.rados.rados.0.smithi085.stdout:3035: finishing rollback tid 1 to smithi08575397-35 2021-02-10T04:39:53.447 INFO:tasks.rados.rados.0.smithi085.stdout:3035: finishing rollback tid 2 to smithi08575397-35 2021-02-10T04:39:53.447 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 35 v 1456 (ObjNum 986 snap 320 seq_num 986) dirty exists 2021-02-10T04:39:53.448 INFO:tasks.rados.rados.0.smithi085.stdout:3034: finishing write tid 2 to smithi08575397-44 2021-02-10T04:39:53.448 INFO:tasks.rados.rados.0.smithi085.stdout:3034: finishing write tid 3 to smithi08575397-44 2021-02-10T04:39:53.448 INFO:tasks.rados.rados.0.smithi085.stdout:3034: finishing write tid 4 to smithi08575397-44 2021-02-10T04:39:53.448 INFO:tasks.rados.rados.0.smithi085.stdout:3034: finishing write tid 5 to smithi08575397-44 2021-02-10T04:39:53.448 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 1439 (ObjNum 1052 snap 345 seq_num 1052) dirty exists 2021-02-10T04:39:53.448 INFO:tasks.rados.rados.0.smithi085.stdout:3034: left oid 44 (ObjNum 1052 snap 345 seq_num 1052) 2021-02-10T04:39:53.459 INFO:tasks.rados.rados.0.smithi085.stdout:3036: finishing copy_from to smithi08575397-5 2021-02-10T04:39:53.459 INFO:tasks.rados.rados.0.smithi085.stdout:3036: got expected ENOENT (src dne) 2021-02-10T04:39:53.459 INFO:tasks.rados.rados.0.smithi085.stdout:3036: finishing copy_from racing read to smithi08575397-5 2021-02-10T04:39:53.574 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:39:53.581 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.1 dump_historic_ops 2021-02-10T04:39:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:53 smithi164 conmon[29288]: cluster 2021-02-10T04:39:51.538283+0000 osd.1 (osd.1) 457 : cluster 2021-02-10T04:39:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:53 smithi164 conmon[29288]: [DBG] purged_snaps scrub starts 2021-02-10T04:39:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:53 smithi164 conmon[29288]: cluster 2021-02-10T04:39:51.539516+0000 osd.1 (osd.1) 458 2021-02-10T04:39:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:53 smithi164 conmon[29288]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:39:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:53 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:39:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:53 smithi164 conmon[29288]: -10T04:39:52.369116+0000 mon.a (mon.0) 1702 : cluster [DBG] 2021-02-10T04:39:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:53 smithi164 conmon[29288]: osdmap e773: 8 total, 8 up, 7 in 2021-02-10T04:39:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:53 smithi164 conmon[29288]: cephadm 2021- 2021-02-10T04:39:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:53 smithi164 conmon[29288]: 02-10T04:39:53.097904+0000 mgr.y (mgr.14140) 764 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:39:53.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:53 smithi164 conmon[29288]: cephadm 2021-02-10T04:39:53.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:53 smithi164 conmon[29288]: 2021-02-10T04:39:53.098456+0000 mgr.y (mgr.14140) 765 : 2021-02-10T04:39:53.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:53 smithi164 conmon[29288]: cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:39:53.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:53 smithi164 conmon[29288]: cluster 2021-02-10T04:39 2021-02-10T04:39:53.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:53 smithi164 conmon[29288]: :53. 2021-02-10T04:39:53.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:53 smithi164 conmon[29288]: 371744+0000 mon.a (mon.0) 1703 : cluster [DBG] osdmap e774: 8 total, 8 up, 7 in 2021-02-10T04:39:53.665 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:53 smithi085 conmon[29319]: cluster 2021-02-10T04:39:51.538283+0000 osd.1 (osd 2021-02-10T04:39:53.665 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:53 smithi085 conmon[29319]: .1) 457 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:39:53.666 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:53 smithi085 conmon[29319]: cluster 2021-02-10T04:39:51.539516+0000 osd.1 (osd.1) 458 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:39:53.666 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:53 smithi085 conmon[29319]: cluster 2021-02-10T04:39:52.369116+0000 mon.a (mon.0) 1702 : cluster [DBG] osdmap e773: 8 total, 8 up, 7 in 2021-02-10T04:39:53.666 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:53 smithi085 conmon[29319]: cephadm 2021-02-10T04:39:53.097904+0000 mgr.y (mgr.14140) 764 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:39:53.666 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:53 smithi085 conmon[29319]: cephadm 2021-02-10T04:39:53.098456+0000 mgr.y (mgr.14140) 765 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:39:53.666 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:53 smithi085 conmon[29319]: cluster 2021-02-10T04:39:53.371744+0000 mon.a (mon.0) 1703 : cluster [DBG] osdmap e774: 8 total, 8 up, 7 in 2021-02-10T04:39:53.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:53 smithi085 conmon[35961]: cluster 2021-02-10T04:39:51.538283+0000 osd.1 ( 2021-02-10T04:39:53.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:53 smithi085 conmon[35961]: osd.1) 457 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:39:53.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:53 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:39:53.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:53 smithi085 conmon[35961]: :39:51.539516+0000 osd.1 (osd.1) 458 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:39:53.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:53 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:39:53.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:53 smithi085 conmon[35961]: 10T04:39:52.369116+0000 mon.a (mon.0) 1702 2021-02-10T04:39:53.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:53 smithi085 conmon[35961]: : cluster [DBG] osdmap e773: 8 total, 8 up, 7 in 2021-02-10T04:39:53.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:53 smithi085 conmon[35961]: cephadm 2021- 2021-02-10T04:39:53.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:53 smithi085 conmon[35961]: 02-10T04:39:53.097904+0000 mgr.y (mgr.14140 2021-02-10T04:39:53.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:53 smithi085 conmon[35961]: ) 764 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:39:53.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:53 smithi085 conmon[35961]: cephadm 2021-02 2021-02-10T04:39:53.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:53 smithi085 conmon[35961]: -10T04:39:53.098456+0000 mgr.y (mgr. 2021-02-10T04:39:53.669 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:53 smithi085 conmon[35961]: 14140) 765 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:39:53.669 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:53 smithi085 conmon[35961]: cluster 2021-02-10T04:39:53.669 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:53 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:39:53.669 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:53 smithi085 conmon[35961]: T04:39:53.371744+0000 mon.a (mon 2021-02-10T04:39:53.669 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:53 smithi085 conmon[35961]: .0) 1703 : cluster [DBG] osdmap e774: 8 total, 8 up, 7 in 2021-02-10T04:39:53.670 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:39:53 smithi085 conmon[45443]: debug 2021-02-10T04:39:53.471+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:53.732 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:39:53.741 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 1 1 2021-02-10T04:39:53.767 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:39:53.970 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:39:54.031 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:39:54.122 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:39:54.137 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:39:54.173 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:39:54.191 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:39:53 smithi085 conmon[41540]: debug 2021-02-10T04:39:53.976+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:54.391 INFO:tasks.rados.rados.0.smithi085.stdout:3034: done (8 left) 2021-02-10T04:39:54.392 INFO:tasks.rados.rados.0.smithi085.stdout:3035: done (7 left) 2021-02-10T04:39:54.392 INFO:tasks.rados.rados.0.smithi085.stdout:3036: done (6 left) 2021-02-10T04:39:54.392 INFO:tasks.rados.rados.0.smithi085.stdout:3037: done (5 left) 2021-02-10T04:39:54.392 INFO:tasks.rados.rados.0.smithi085.stdout:3038: done (4 left) 2021-02-10T04:39:54.392 INFO:tasks.rados.rados.0.smithi085.stdout:3039: read oid 21 snap -1 2021-02-10T04:39:54.392 INFO:tasks.rados.rados.0.smithi085.stdout:3039: expect (ObjNum 1032 snap 340 seq_num 1032) 2021-02-10T04:39:54.392 INFO:tasks.rados.rados.0.smithi085.stdout:3040: write oid 9 current snap is 346 2021-02-10T04:39:54.392 INFO:tasks.rados.rados.0.smithi085.stdout:3040: seq_num 1053 ranges {497771=493911,1619186=554020,2950529=430606} 2021-02-10T04:39:54.397 INFO:tasks.rados.rados.0.smithi085.stdout:3040: writing smithi08575397-9 from 497771 to 991682 tid 1 2021-02-10T04:39:54.405 INFO:tasks.rados.rados.0.smithi085.stdout:3040: writing smithi08575397-9 from 1619186 to 2173206 tid 2 2021-02-10T04:39:54.412 INFO:tasks.rados.rados.0.smithi085.stdout:3040: writing smithi08575397-9 from 2950529 to 3381135 tid 3 2021-02-10T04:39:54.412 INFO:tasks.rados.rados.0.smithi085.stdout:3041: write oid 50 current snap is 346 2021-02-10T04:39:54.413 INFO:tasks.rados.rados.0.smithi085.stdout:3041: seq_num 1054 ranges {514578=569160,1677668=414031,2577952=641878} 2021-02-10T04:39:54.419 INFO:tasks.rados.rados.0.smithi085.stdout:3041: writing smithi08575397-50 from 514578 to 1083738 tid 1 2021-02-10T04:39:54.425 INFO:tasks.rados.rados.0.smithi085.stdout:3041: writing smithi08575397-50 from 1677668 to 2091699 tid 2 2021-02-10T04:39:54.432 INFO:tasks.rados.rados.0.smithi085.stdout:3041: writing smithi08575397-50 from 2577952 to 3219830 tid 3 2021-02-10T04:39:54.432 INFO:tasks.rados.rados.0.smithi085.stdout:3042: read oid 23 snap -1 2021-02-10T04:39:54.432 INFO:tasks.rados.rados.0.smithi085.stdout:3042: expect (ObjNum 924 snap 306 seq_num 924) 2021-02-10T04:39:54.432 INFO:tasks.rados.rados.0.smithi085.stdout:3043: snap_remove snap 299 2021-02-10T04:39:54.643 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:39:54 smithi164 conmon[40892]: debug 2021-02-10T04:39:54.250+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:54.646 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:39:54.752 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:39:54.792 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:39:54.899 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[29319]: audit 2021-02-10T04:39:54.899 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[29319]: 2021-02-10T04:39:54.900 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[29319]: 2021-02-10T04:39:53.843453+0000 2021-02-10T04:39:54.900 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[29319]: mon.a (mon.0) 1704 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:39:54.900 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:39:54.900 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[29319]: :39:54.183811+0000 mgr.y (mgr.14140) 766 : cluster 2021-02-10T04:39:54.900 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[29319]: [DBG] pgmap v1312: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 486 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 7.7 MiB/s rd, 824 KiB/s wr, 8 op/s 2021-02-10T04:39:54.900 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[29319]: cluster 2021-02-10T04:39:54.391430+0000 2021-02-10T04:39:54.900 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[29319]: mon.a (mon.0) 1705 : cluster [DBG] osdmap e775: 8 total, 8 up, 7 in 2021-02-10T04:39:54.900 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: audit 2021-02-10T04:39:54.900 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: 2021-02-10T04:39:54.900 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: 2021 2021-02-10T04:39:54.900 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: - 2021-02-10T04:39:54.900 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: 02 2021-02-10T04:39:54.901 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: - 2021-02-10T04:39:54.901 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: 10 2021-02-10T04:39:54.901 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: T 2021-02-10T04:39:54.901 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: 04 2021-02-10T04:39:54.901 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: : 2021-02-10T04:39:54.901 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: 39 2021-02-10T04:39:54.901 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: : 2021-02-10T04:39:54.901 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: 53 2021-02-10T04:39:54.901 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: . 2021-02-10T04:39:54.901 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: 843453 2021-02-10T04:39:54.901 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: +0000 2021-02-10T04:39:54.901 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: 2021-02-10T04:39:54.901 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: mon.a 2021-02-10T04:39:54.902 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: ( 2021-02-10T04:39:54.902 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: mon 2021-02-10T04:39:54.902 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: . 2021-02-10T04:39:54.902 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: 0 2021-02-10T04:39:54.902 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: ) 2021-02-10T04:39:54.902 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: 1704 2021-02-10T04:39:54.902 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:39:54.902 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:39:54.902 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: :39:54.183811+0000 mgr.y (mgr.14140) 766 : cluster 2021-02-10T04:39:54.902 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: [DBG] pgmap v1312: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 486 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 7.7 MiB/s rd, 824 KiB/s wr, 8 op/s 2021-02-10T04:39:54.902 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: cluster 2021-02-10T04:39 2021-02-10T04:39:54.902 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: :54.391430+0000 mon.a (mon.0) 1705 2021-02-10T04:39:54.902 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[35961]: : cluster [DBG] osdmap e775: 8 total, 8 up, 7 in 2021-02-10T04:39:55.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:54 smithi164 conmon[29288]: audit 2021-02-10T04:39:53.843453+0000 mon.a (mon.0) 1704 : 2021-02-10T04:39:55.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:54 smithi164 conmon[29288]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:39:55.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:54 smithi164 conmon[29288]: cluster 2021-02-10T04:39:54.183811+0000 mgr.y (mgr.14140) 766 : cluster [DBG] pgmap v1312: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 486 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 7.7 MiB/s rd, 824 KiB/s wr, 8 op/s 2021-02-10T04:39:55.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:54 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:39:55.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:54 smithi164 conmon[29288]: :39:54.391430+0000 mon.a (mon.0) 1705 : cluster [DBG] osdmap e775: 8 total, 8 up, 7 in 2021-02-10T04:39:55.188 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:39:55.191 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:39:54 smithi085 conmon[45443]: debug 2021-02-10T04:39:54.918+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:39:55.395 INFO:tasks.rados.rados.0.smithi085.stdout:3043: done (8 left) 2021-02-10T04:39:55.396 INFO:tasks.rados.rados.0.smithi085.stdout:3044: write oid 14 current snap is 346 2021-02-10T04:39:55.396 INFO:tasks.rados.rados.0.smithi085.stdout:3044: seq_num 1055 ranges {531385=644409,1736150=557689,3005375=569503} 2021-02-10T04:39:55.403 INFO:tasks.rados.rados.0.smithi085.stdout:3044: writing smithi08575397-14 from 531385 to 1175794 tid 1 2021-02-10T04:39:55.409 INFO:tasks.rados.rados.0.smithi085.stdout:3044: writing smithi08575397-14 from 1736150 to 2293839 tid 2 2021-02-10T04:39:55.418 INFO:tasks.rados.rados.0.smithi085.stdout:3044: writing smithi08575397-14 from 3005375 to 3574878 tid 3 2021-02-10T04:39:55.418 INFO:tasks.rados.rados.0.smithi085.stdout:3045: read oid 13 snap 332 2021-02-10T04:39:55.418 INFO:tasks.rados.rados.0.smithi085.stdout:3045: expect deleted 2021-02-10T04:39:55.418 INFO:tasks.rados.rados.0.smithi085.stdout:3046: read oid 6 snap -1 2021-02-10T04:39:55.418 INFO:tasks.rados.rados.0.smithi085.stdout:3046: expect (ObjNum 1028 snap 335 seq_num 1028) 2021-02-10T04:39:55.418 INFO:tasks.rados.rados.0.smithi085.stdout:3047: read oid 32 snap 345 2021-02-10T04:39:55.418 INFO:tasks.rados.rados.0.smithi085.stdout:3047: expect (ObjNum 1050 snap 344 seq_num 1050) 2021-02-10T04:39:55.418 INFO:tasks.rados.rados.0.smithi085.stdout:3040: finishing write tid 1 to smithi08575397-9 2021-02-10T04:39:55.419 INFO:tasks.rados.rados.0.smithi085.stdout:3040: finishing write tid 2 to smithi08575397-9 2021-02-10T04:39:55.419 INFO:tasks.rados.rados.0.smithi085.stdout:3039: expect (ObjNum 1032 snap 340 seq_num 1032) 2021-02-10T04:39:55.563 INFO:tasks.rados.rados.0.smithi085.stdout:3048: read oid 16 snap -1 2021-02-10T04:39:55.563 INFO:tasks.rados.rados.0.smithi085.stdout:3048: expect deleted 2021-02-10T04:39:55.563 INFO:tasks.rados.rados.0.smithi085.stdout:3039: done (12 left) 2021-02-10T04:39:55.563 INFO:tasks.rados.rados.0.smithi085.stdout:3049: snap_remove snap 298 2021-02-10T04:39:56.409 INFO:tasks.rados.rados.0.smithi085.stdout:3049: done (12 left) 2021-02-10T04:39:56.409 INFO:tasks.rados.rados.0.smithi085.stdout:3050: read oid 8 snap -1 2021-02-10T04:39:56.409 INFO:tasks.rados.rados.0.smithi085.stdout:3050: expect deleted 2021-02-10T04:39:56.411 INFO:tasks.rados.rados.0.smithi085.stdout:3051: copy_from oid 28 from oid 3 current snap is 346 2021-02-10T04:39:56.411 INFO:tasks.rados.rados.0.smithi085.stdout:3052: copy_from oid 43 from oid 2 current snap is 346 2021-02-10T04:39:56.411 INFO:tasks.rados.rados.0.smithi085.stdout:3053: write oid 30 current snap is 346 2021-02-10T04:39:56.411 INFO:tasks.rados.rados.0.smithi085.stdout:3053: seq_num 1056 ranges {548192=719658,1678279=417700,2516445=780775} 2021-02-10T04:39:56.417 INFO:tasks.rados.rados.0.smithi085.stdout:3053: writing smithi08575397-30 from 548192 to 1267850 tid 1 2021-02-10T04:39:56.422 INFO:tasks.rados.rados.0.smithi085.stdout:3053: writing smithi08575397-30 from 1678279 to 2095979 tid 2 2021-02-10T04:39:56.430 INFO:tasks.rados.rados.0.smithi085.stdout:3053: writing smithi08575397-30 from 2516445 to 3297220 tid 3 2021-02-10T04:39:56.430 INFO:tasks.rados.rados.0.smithi085.stdout:3040: finishing write tid 3 to smithi08575397-9 2021-02-10T04:39:56.430 INFO:tasks.rados.rados.0.smithi085.stdout:3040: finishing write tid 4 to smithi08575397-9 2021-02-10T04:39:56.430 INFO:tasks.rados.rados.0.smithi085.stdout:3040: finishing write tid 5 to smithi08575397-9 2021-02-10T04:39:56.430 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 9 v 1384 (ObjNum 1053 snap 346 seq_num 1053) dirty exists 2021-02-10T04:39:56.430 INFO:tasks.rados.rados.0.smithi085.stdout:3040: left oid 9 (ObjNum 1053 snap 346 seq_num 1053) 2021-02-10T04:39:56.431 INFO:tasks.rados.rados.0.smithi085.stdout:3041: finishing write tid 1 to smithi08575397-50 2021-02-10T04:39:56.431 INFO:tasks.rados.rados.0.smithi085.stdout:3042: expect (ObjNum 924 snap 306 seq_num 924) 2021-02-10T04:39:56.589 INFO:tasks.rados.rados.0.smithi085.stdout:3041: finishing write tid 2 to smithi08575397-50 2021-02-10T04:39:56.590 INFO:tasks.rados.rados.0.smithi085.stdout:3041: finishing write tid 3 to smithi08575397-50 2021-02-10T04:39:56.590 INFO:tasks.rados.rados.0.smithi085.stdout:3041: finishing write tid 4 to smithi08575397-50 2021-02-10T04:39:56.590 INFO:tasks.rados.rados.0.smithi085.stdout:3041: finishing write tid 5 to smithi08575397-50 2021-02-10T04:39:56.590 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 1549 (ObjNum 1054 snap 346 seq_num 1054) dirty exists 2021-02-10T04:39:56.590 INFO:tasks.rados.rados.0.smithi085.stdout:3041: left oid 50 (ObjNum 1054 snap 346 seq_num 1054) 2021-02-10T04:39:56.590 INFO:tasks.rados.rados.0.smithi085.stdout:3044: finishing write tid 1 to smithi08575397-14 2021-02-10T04:39:56.590 INFO:tasks.rados.rados.0.smithi085.stdout:3044: finishing write tid 2 to smithi08575397-14 2021-02-10T04:39:56.590 INFO:tasks.rados.rados.0.smithi085.stdout:3040: done (15 left) 2021-02-10T04:39:56.590 INFO:tasks.rados.rados.0.smithi085.stdout:3041: done (14 left) 2021-02-10T04:39:56.591 INFO:tasks.rados.rados.0.smithi085.stdout:3042: done (13 left) 2021-02-10T04:39:56.591 INFO:tasks.rados.rados.0.smithi085.stdout:3054: snap_remove snap 324 2021-02-10T04:39:56.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:56 smithi085 conmon[29319]: cluster 2021-02-10T04:39:56.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:56 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:39:56.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:56 smithi085 conmon[29319]: 10T04:39:55.395121+0000 mon.a (mon.0) 1706 : cluster [DBG] osdmap e776: 8 total, 8 up, 7 in 2021-02-10T04:39:56.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:56 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:39:56.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:56 smithi085 conmon[29319]: T04:39:56.184252+0000 mgr.y (mgr.14140) 767 2021-02-10T04:39:56.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:56 smithi085 conmon[29319]: : cluster [DBG] pgmap v1315: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 472 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2021-02-10T04:39:56.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:56 smithi085 conmon[35961]: cluster 2021-02-10T04:39:56.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:56 smithi085 conmon[35961]: 2021-02-10T04:39:56.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:56 smithi085 conmon[35961]: 2021 2021-02-10T04:39:56.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:56 smithi085 conmon[35961]: - 2021-02-10T04:39:56.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:56 smithi085 conmon[35961]: 02 2021-02-10T04:39:56.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:56 smithi085 conmon[35961]: - 2021-02-10T04:39:56.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:56 smithi085 conmon[35961]: 10 2021-02-10T04:39:56.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:56 smithi085 conmon[35961]: T 2021-02-10T04:39:56.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:56 smithi085 conmon[35961]: 04 2021-02-10T04:39:56.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:56 smithi085 conmon[35961]: :39:55.395121+0000 mon.a (mon.0) 1706 : cluster [DBG] osdmap e776: 8 total, 8 up, 7 in 2021-02-10T04:39:56.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:56 smithi085 conmon[35961]: cluster 2021-02-10T04:39:56.184252+0000 2021-02-10T04:39:56.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:56 smithi085 conmon[35961]: mgr.y (mgr.14140) 767 : cluster [DBG] pgmap v1315: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 472 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2021-02-10T04:39:56.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:56 smithi164 conmon[29288]: cluster 2021-02-10T04:39:55.395121+0000 mon.a (mon.0) 1706 2021-02-10T04:39:56.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:56 smithi164 conmon[29288]: : cluster [DBG] osdmap e776: 8 total, 8 up, 7 in 2021-02-10T04:39:56.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:56 smithi164 conmon[29288]: cluster 2021-02-10T04:39:56.184252+0000 mgr.y (mgr.14140) 767 : cluster [DBG] pgmap v1315: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 472 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2021-02-10T04:39:57.408 INFO:tasks.rados.rados.0.smithi085.stdout:3054: done (13 left) 2021-02-10T04:39:57.408 INFO:tasks.rados.rados.0.smithi085.stdout:3055: read oid 21 snap -1 2021-02-10T04:39:57.408 INFO:tasks.rados.rados.0.smithi085.stdout:3055: expect (ObjNum 1032 snap 340 seq_num 1032) 2021-02-10T04:39:57.408 INFO:tasks.rados.rados.0.smithi085.stdout:3056: rollback oid 20 current snap is 346 2021-02-10T04:39:57.408 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 20 to 344 2021-02-10T04:39:57.408 INFO:tasks.rados.rados.0.smithi085.stdout:3057: snap_create 2021-02-10T04:39:57.408 INFO:tasks.rados.rados.0.smithi085.stdout:3044: finishing write tid 3 to smithi08575397-14 2021-02-10T04:39:57.408 INFO:tasks.rados.rados.0.smithi085.stdout:3046: expect (ObjNum 1028 snap 335 seq_num 1028) 2021-02-10T04:39:57.599 INFO:tasks.rados.rados.0.smithi085.stdout:3044: finishing write tid 4 to smithi08575397-14 2021-02-10T04:39:57.599 INFO:tasks.rados.rados.0.smithi085.stdout:3047: expect (ObjNum 1050 snap 344 seq_num 1050) 2021-02-10T04:39:57.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:57 smithi085 conmon[29319]: cluster 2021-02-10T04:39:56.402163+0000 2021-02-10T04:39:57.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:57 smithi085 conmon[29319]: mon.a (mon.0) 1707 : cluster [DBG] osdmap e777: 8 total, 8 up, 7 in 2021-02-10T04:39:57.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:57 smithi085 conmon[35961]: cluster 2021-02-10T04:39:57.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:57 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:39:57.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:57 smithi085 conmon[35961]: 39:56.402163+0000 mon.a (mon.0) 1707 : cluster [DBG] osdmap e777: 8 total, 8 up, 7 in 2021-02-10T04:39:57.759 INFO:tasks.rados.rados.0.smithi085.stdout:3044: finishing write tid 5 to smithi08575397-14 2021-02-10T04:39:57.759 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 1388 (ObjNum 1055 snap 346 seq_num 1055) dirty exists 2021-02-10T04:39:57.759 INFO:tasks.rados.rados.0.smithi085.stdout:3044: left oid 14 (ObjNum 1055 snap 346 seq_num 1055) 2021-02-10T04:39:57.759 INFO:tasks.rados.rados.0.smithi085.stdout:3053: finishing write tid 1 to smithi08575397-30 2021-02-10T04:39:57.759 INFO:tasks.rados.rados.0.smithi085.stdout:3052: finishing copy_from to smithi08575397-43 2021-02-10T04:39:57.759 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 1453 (ObjNum 982 snap 317 seq_num 982) dirty exists 2021-02-10T04:39:57.760 INFO:tasks.rados.rados.0.smithi085.stdout:3052: finishing copy_from racing read to smithi08575397-43 2021-02-10T04:39:57.760 INFO:tasks.rados.rados.0.smithi085.stdout:3053: finishing write tid 2 to smithi08575397-30 2021-02-10T04:39:57.760 INFO:tasks.rados.rados.0.smithi085.stdout:3053: finishing write tid 3 to smithi08575397-30 2021-02-10T04:39:57.760 INFO:tasks.rados.rados.0.smithi085.stdout:3053: finishing write tid 4 to smithi08575397-30 2021-02-10T04:39:57.760 INFO:tasks.rados.rados.0.smithi085.stdout:3053: finishing write tid 5 to smithi08575397-30 2021-02-10T04:39:57.760 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 1385 (ObjNum 1056 snap 346 seq_num 1056) dirty exists 2021-02-10T04:39:57.760 INFO:tasks.rados.rados.0.smithi085.stdout:3053: left oid 30 (ObjNum 1056 snap 346 seq_num 1056) 2021-02-10T04:39:57.760 INFO:tasks.rados.rados.0.smithi085.stdout:3051: finishing copy_from to smithi08575397-28 2021-02-10T04:39:57.760 INFO:tasks.rados.rados.0.smithi085.stdout:3051: got expected ENOENT (src dne) 2021-02-10T04:39:57.760 INFO:tasks.rados.rados.0.smithi085.stdout:3051: finishing copy_from racing read to smithi08575397-28 2021-02-10T04:39:57.760 INFO:tasks.rados.rados.0.smithi085.stdout:3055: expect (ObjNum 1032 snap 340 seq_num 1032) 2021-02-10T04:39:57.881 INFO:tasks.rados.rados.0.smithi085.stdout:3056: finishing rollback tid 0 to smithi08575397-20 2021-02-10T04:39:57.881 INFO:tasks.rados.rados.0.smithi085.stdout:3056: finishing rollback tid 1 to smithi08575397-20 2021-02-10T04:39:57.881 INFO:tasks.rados.rados.0.smithi085.stdout:3056: finishing rollback tid 2 to smithi08575397-20 2021-02-10T04:39:57.881 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 20 v 1442 (ObjNum 866 snap 278 seq_num 866) dirty exists 2021-02-10T04:39:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:57 smithi164 conmon[29288]: cluster 2021-02-10T04:39:56.402163+0000 mon.a 2021-02-10T04:39:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:57 smithi164 conmon[29288]: (mon.0) 1707 : cluster [DBG] osdmap e777: 8 total, 8 up, 7 in 2021-02-10T04:39:58.258 INFO:tasks.rados.rados.0.smithi085.stdout:3044: done (15 left) 2021-02-10T04:39:58.258 INFO:tasks.rados.rados.0.smithi085.stdout:3045: done (14 left) 2021-02-10T04:39:58.258 INFO:tasks.rados.rados.0.smithi085.stdout:3046: done (13 left) 2021-02-10T04:39:58.258 INFO:tasks.rados.rados.0.smithi085.stdout:3047: done (12 left) 2021-02-10T04:39:58.258 INFO:tasks.rados.rados.0.smithi085.stdout:3048: done (11 left) 2021-02-10T04:39:58.258 INFO:tasks.rados.rados.0.smithi085.stdout:3050: done (10 left) 2021-02-10T04:39:58.258 INFO:tasks.rados.rados.0.smithi085.stdout:3051: done (9 left) 2021-02-10T04:39:58.258 INFO:tasks.rados.rados.0.smithi085.stdout:3052: done (8 left) 2021-02-10T04:39:58.258 INFO:tasks.rados.rados.0.smithi085.stdout:3053: done (7 left) 2021-02-10T04:39:58.258 INFO:tasks.rados.rados.0.smithi085.stdout:3055: done (6 left) 2021-02-10T04:39:58.259 INFO:tasks.rados.rados.0.smithi085.stdout:3056: done (5 left) 2021-02-10T04:39:58.259 INFO:tasks.rados.rados.0.smithi085.stdout:3057: done (4 left) 2021-02-10T04:39:58.259 INFO:tasks.rados.rados.0.smithi085.stdout:3058: snap_create 2021-02-10T04:39:58.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:58 smithi164 conmon[29288]: cluster 2021-02-10T04:39:57.406279+0000 mon.a (mon 2021-02-10T04:39:58.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:58 smithi164 conmon[29288]: .0) 1708 : cluster [DBG] osdmap e778: 8 total, 8 up, 7 in 2021-02-10T04:39:58.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:58 smithi164 conmon[29288]: cluster 2021-02-10T04:39:58. 2021-02-10T04:39:58.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:58 smithi164 conmon[29288]: 184915+0000 mgr.y (mgr.14140) 768 : cluster [DBG] pgmap v1318: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 470 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 5.8 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2021-02-10T04:39:58.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:58 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:39:58.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:58 smithi164 conmon[29288]: T04:39:58.255988+0000 mon.a (mon.0) 1709 : cluster [DBG] 2021-02-10T04:39:58.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:58 smithi164 conmon[29288]: osdmap e779: 8 total, 8 up, 7 in 2021-02-10T04:39:58.645 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:39:58 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:39:58.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:58 smithi085 conmon[29319]: cluster 2021-02-10T04:39:58.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:58 smithi085 conmon[29319]: 2021-02 2021-02-10T04:39:58.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:58 smithi085 conmon[29319]: -10T04:39:57.406279+0000 mon.a (mon.0) 1708 : cluster [DBG] osdmap e778: 8 total, 8 up, 7 in 2021-02-10T04:39:58.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:58 smithi085 conmon[29319]: cluster 2021-02-10T04:39:58.184915+0000 mgr.y (mgr.14140) 768 : cluster [DBG] pgmap v1318: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 470 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 5.8 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2021-02-10T04:39:58.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:58 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:39:58.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:58 smithi085 conmon[29319]: 02-10T04:39:58.255988+0000 2021-02-10T04:39:58.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:58 smithi085 conmon[29319]: mon.a (mon.0) 1709 : cluster [DBG] osdmap e779: 8 total, 8 up, 7 in 2021-02-10T04:39:58.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:58 smithi085 conmon[29319]: 2021-02-10T04:39:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:58 smithi085 conmon[35961]: cluster 2021-02-10T04:39: 2021-02-10T04:39:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:58 smithi085 conmon[35961]: 57.406279+0000 mon.a (mon.0) 1708 : cluster [DBG] osdmap e778: 8 total, 8 up, 7 in 2021-02-10T04:39:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:58 smithi085 conmon[35961]: cluster 2021-02-10T04:39:58.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:58 smithi085 conmon[35961]: 2021-02-10T04:39:58.184915+0000 mgr.y (mgr.14140) 768 : cluster [DBG] 2021-02-10T04:39:58.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:58 smithi085 conmon[35961]: pgmap v1318: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 470 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 5.8 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2021-02-10T04:39:58.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:58 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:39:58.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:58 smithi085 conmon[35961]: 10T04:39:58.255988+0000 mon.a ( 2021-02-10T04:39:58.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:58 smithi085 conmon[35961]: mon.0) 1709 : cluster [DBG] osdmap e779: 8 total, 8 up, 7 in 2021-02-10T04:39:59.258 INFO:tasks.rados.rados.0.smithi085.stdout:3058: done (4 left) 2021-02-10T04:39:59.258 INFO:tasks.rados.rados.0.smithi085.stdout:3059: delete oid 31 current snap is 348 2021-02-10T04:39:59.266 INFO:tasks.rados.rados.0.smithi085.stdout:3059: done (4 left) 2021-02-10T04:39:59.266 INFO:tasks.rados.rados.0.smithi085.stdout:3060: write oid 3 current snap is 348 2021-02-10T04:39:59.266 INFO:tasks.rados.rados.0.smithi085.stdout:3060: seq_num 1058 ranges {581806=586509,1911596=421369,3087644=403319} 2021-02-10T04:39:59.273 INFO:tasks.rados.rados.0.smithi085.stdout:3060: writing smithi08575397-3 from 581806 to 1168315 tid 1 2021-02-10T04:39:59.280 INFO:tasks.rados.rados.0.smithi085.stdout:3060: writing smithi08575397-3 from 1911596 to 2332965 tid 2 2021-02-10T04:39:59.287 INFO:tasks.rados.rados.0.smithi085.stdout:3060: writing smithi08575397-3 from 3087644 to 3490963 tid 3 2021-02-10T04:39:59.287 INFO:tasks.rados.rados.0.smithi085.stdout:3061: write (excl) oid 45 current snap is 348 2021-02-10T04:39:59.287 INFO:tasks.rados.rados.0.smithi085.stdout:3060: finishing write tid 1 to smithi08575397-3 2021-02-10T04:39:59.287 INFO:tasks.rados.rados.0.smithi085.stdout:3061: seq_num 1059 ranges {598613=661758,1970078=565027,3115067=614591} 2021-02-10T04:39:59.295 INFO:tasks.rados.rados.0.smithi085.stdout:3061: writing smithi08575397-45 from 598613 to 1260371 tid 1 2021-02-10T04:39:59.302 INFO:tasks.rados.rados.0.smithi085.stdout:3061: writing smithi08575397-45 from 1970078 to 2535105 tid 2 2021-02-10T04:39:59.310 INFO:tasks.rados.rados.0.smithi085.stdout:3061: writing smithi08575397-45 from 3115067 to 3729658 tid 3 2021-02-10T04:39:59.310 INFO:tasks.rados.rados.0.smithi085.stdout:3062: write (excl) oid 34 current snap is 348 2021-02-10T04:39:59.310 INFO:tasks.rados.rados.0.smithi085.stdout:3062: seq_num 1060 ranges {615420=737007,1912207=425038,3026137=542216} 2021-02-10T04:39:59.319 INFO:tasks.rados.rados.0.smithi085.stdout:3062: writing smithi08575397-34 from 615420 to 1352427 tid 1 2021-02-10T04:39:59.325 INFO:tasks.rados.rados.0.smithi085.stdout:3062: writing smithi08575397-34 from 1912207 to 2337245 tid 2 2021-02-10T04:39:59.333 INFO:tasks.rados.rados.0.smithi085.stdout:3062: writing smithi08575397-34 from 3026137 to 3568353 tid 3 2021-02-10T04:39:59.333 INFO:tasks.rados.rados.0.smithi085.stdout:3063: copy_from oid 14 from oid 22 current snap is 348 2021-02-10T04:39:59.333 INFO:tasks.rados.rados.0.smithi085.stdout:3060: finishing write tid 2 to smithi08575397-3 2021-02-10T04:39:59.333 INFO:tasks.rados.rados.0.smithi085.stdout:3060: finishing write tid 3 to smithi08575397-3 2021-02-10T04:39:59.333 INFO:tasks.rados.rados.0.smithi085.stdout:3060: finishing write tid 4 to smithi08575397-3 2021-02-10T04:39:59.333 INFO:tasks.rados.rados.0.smithi085.stdout:3060: finishing write tid 5 to smithi08575397-3 2021-02-10T04:39:59.334 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 1457 (ObjNum 1058 snap 348 seq_num 1058) dirty exists 2021-02-10T04:39:59.334 INFO:tasks.rados.rados.0.smithi085.stdout:3060: left oid 3 (ObjNum 1058 snap 348 seq_num 1058) 2021-02-10T04:39:59.334 INFO:tasks.rados.rados.0.smithi085.stdout:3060: done (7 left) 2021-02-10T04:39:59.334 INFO:tasks.rados.rados.0.smithi085.stdout:3064: write oid 37 current snap is 348 2021-02-10T04:39:59.334 INFO:tasks.rados.rados.0.smithi085.stdout:3061: finishing write tid 1 to smithi08575397-45 2021-02-10T04:39:59.334 INFO:tasks.rados.rados.0.smithi085.stdout:3061: finishing write tid 2 to smithi08575397-45 2021-02-10T04:39:59.334 INFO:tasks.rados.rados.0.smithi085.stdout:3061: finishing write tid 3 to smithi08575397-45 2021-02-10T04:39:59.334 INFO:tasks.rados.rados.0.smithi085.stdout:3061: finishing write tid 4 to smithi08575397-45 2021-02-10T04:39:59.334 INFO:tasks.rados.rados.0.smithi085.stdout:3064: seq_num 1061 ranges {632227=412256,1570689=568696,2653560=753488} 2021-02-10T04:39:59.339 INFO:tasks.rados.rados.0.smithi085.stdout:3064: writing smithi08575397-37 from 632227 to 1044483 tid 1 2021-02-10T04:39:59.346 INFO:tasks.rados.rados.0.smithi085.stdout:3064: writing smithi08575397-37 from 1570689 to 2139385 tid 2 2021-02-10T04:39:59.355 INFO:tasks.rados.rados.0.smithi085.stdout:3064: writing smithi08575397-37 from 2653560 to 3407048 tid 3 2021-02-10T04:39:59.355 INFO:tasks.rados.rados.0.smithi085.stdout:3065: read oid 16 snap -1 2021-02-10T04:39:59.355 INFO:tasks.rados.rados.0.smithi085.stdout:3065: expect deleted 2021-02-10T04:39:59.355 INFO:tasks.rados.rados.0.smithi085.stdout:3066: delete oid 30 current snap is 348 2021-02-10T04:39:59.355 INFO:tasks.rados.rados.0.smithi085.stdout:3061: finishing write tid 5 to smithi08575397-45 2021-02-10T04:39:59.355 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 45 v 1457 (ObjNum 1059 snap 348 seq_num 1059) dirty exists 2021-02-10T04:39:59.355 INFO:tasks.rados.rados.0.smithi085.stdout:3061: left oid 45 (ObjNum 1059 snap 348 seq_num 1059) 2021-02-10T04:39:59.355 INFO:tasks.rados.rados.0.smithi085.stdout:3062: finishing write tid 1 to smithi08575397-34 2021-02-10T04:39:59.355 INFO:tasks.rados.rados.0.smithi085.stdout:3062: finishing write tid 2 to smithi08575397-34 2021-02-10T04:39:59.355 INFO:tasks.rados.rados.0.smithi085.stdout:3062: finishing write tid 3 to smithi08575397-34 2021-02-10T04:39:59.356 INFO:tasks.rados.rados.0.smithi085.stdout:3062: finishing write tid 4 to smithi08575397-34 2021-02-10T04:39:59.356 INFO:tasks.rados.rados.0.smithi085.stdout:3062: finishing write tid 5 to smithi08575397-34 2021-02-10T04:39:59.356 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 34 v 823 (ObjNum 1060 snap 348 seq_num 1060) dirty exists 2021-02-10T04:39:59.356 INFO:tasks.rados.rados.0.smithi085.stdout:3062: left oid 34 (ObjNum 1060 snap 348 seq_num 1060) 2021-02-10T04:39:59.356 INFO:tasks.rados.rados.0.smithi085.stdout:3064: finishing write tid 1 to smithi08575397-37 2021-02-10T04:39:59.356 INFO:tasks.rados.rados.0.smithi085.stdout:3064: finishing write tid 2 to smithi08575397-37 2021-02-10T04:39:59.356 INFO:tasks.rados.rados.0.smithi085.stdout:3063: finishing copy_from to smithi08575397-14 2021-02-10T04:39:59.356 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 1453 (ObjNum 982 snap 317 seq_num 982) dirty exists 2021-02-10T04:39:59.356 INFO:tasks.rados.rados.0.smithi085.stdout:3063: finishing copy_from racing read to smithi08575397-14 2021-02-10T04:39:59.358 INFO:tasks.rados.rados.0.smithi085.stdout:3061: done (9 left) 2021-02-10T04:39:59.358 INFO:tasks.rados.rados.0.smithi085.stdout:3062: done (8 left) 2021-02-10T04:39:59.358 INFO:tasks.rados.rados.0.smithi085.stdout:3063: done (7 left) 2021-02-10T04:39:59.358 INFO:tasks.rados.rados.0.smithi085.stdout:3066: done (6 left) 2021-02-10T04:39:59.359 INFO:tasks.rados.rados.0.smithi085.stdout:3067: copy_from oid 45 from oid 5 current snap is 348 2021-02-10T04:39:59.359 INFO:tasks.rados.rados.0.smithi085.stdout:3068: copy_from oid 46 from oid 50 current snap is 348 2021-02-10T04:39:59.359 INFO:tasks.rados.rados.0.smithi085.stdout:3069: snap_create 2021-02-10T04:39:59.360 INFO:tasks.rados.rados.0.smithi085.stdout:3067: finishing copy_from to smithi08575397-45 2021-02-10T04:39:59.360 INFO:tasks.rados.rados.0.smithi085.stdout:3067: got expected ENOENT (src dne) 2021-02-10T04:39:59.360 INFO:tasks.rados.rados.0.smithi085.stdout:3067: finishing copy_from racing read to smithi08575397-45 2021-02-10T04:39:59.360 INFO:tasks.rados.rados.0.smithi085.stdout:3064: finishing write tid 3 to smithi08575397-37 2021-02-10T04:39:59.361 INFO:tasks.rados.rados.0.smithi085.stdout:3064: finishing write tid 4 to smithi08575397-37 2021-02-10T04:39:59.362 INFO:tasks.rados.rados.0.smithi085.stdout:3064: finishing write tid 5 to smithi08575397-37 2021-02-10T04:39:59.362 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 1595 (ObjNum 1061 snap 348 seq_num 1061) dirty exists 2021-02-10T04:39:59.362 INFO:tasks.rados.rados.0.smithi085.stdout:3064: left oid 37 (ObjNum 1061 snap 348 seq_num 1061) 2021-02-10T04:39:59.384 INFO:tasks.rados.rados.0.smithi085.stdout:3068: finishing copy_from to smithi08575397-46 2021-02-10T04:39:59.385 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 1549 (ObjNum 1054 snap 346 seq_num 1054) dirty exists 2021-02-10T04:39:59.385 INFO:tasks.rados.rados.0.smithi085.stdout:3068: finishing copy_from racing read to smithi08575397-46 2021-02-10T04:39:59.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:59 smithi085 conmon[29319]: audit 2021-02-10T04:39:58.367042+0000 mgr.y 2021-02-10T04:39:59.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:59 smithi085 conmon[29319]: (mgr.14140) 769 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:39:59.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:39:59 smithi085 conmon[29319]: cluster 2021-02-10T04:39:59.258182+0000 mon.a (mon.0) 1710 : cluster [DBG] osdmap e780: 8 total, 8 up, 7 in 2021-02-10T04:39:59.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:59 smithi085 conmon[35961]: audit 2021-02-10T04:39:59.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:59 smithi085 conmon[35961]: 2021-02-10T04:39:59.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:59 smithi085 conmon[35961]: 2021 2021-02-10T04:39:59.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:59 smithi085 conmon[35961]: - 2021-02-10T04:39:59.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:59 smithi085 conmon[35961]: 02 2021-02-10T04:39:59.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:59 smithi085 conmon[35961]: -10T04:39:58.367042+0000 2021-02-10T04:39:59.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:59 smithi085 conmon[35961]: mgr.y (mgr.14140) 769 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:39:59.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:59 smithi085 conmon[35961]: cluster 2021-02-10T04:39:59.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:59 smithi085 conmon[35961]: 2021-02-10T04:39:59.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:59 smithi085 conmon[35961]: 2021 2021-02-10T04:39:59.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:59 smithi085 conmon[35961]: - 2021-02-10T04:39:59.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:59 smithi085 conmon[35961]: 02 2021-02-10T04:39:59.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:59 smithi085 conmon[35961]: - 2021-02-10T04:39:59.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:59 smithi085 conmon[35961]: 10T04:39:59. 2021-02-10T04:39:59.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:39:59 smithi085 conmon[35961]: 258182+0000 mon.a (mon.0) 1710 : cluster [DBG] osdmap e780: 8 total, 8 up, 7 in 2021-02-10T04:39:59.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:59 smithi164 conmon[29288]: audit 2021-02-10T04:39:58.367042+0000 2021-02-10T04:39:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:59 smithi164 conmon[29288]: mgr.y (mgr.14140) 769 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:39:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:59 smithi164 conmon[29288]: cluster 2021-02-10T04:39:59.258182+0000 2021-02-10T04:39:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:39:59 smithi164 conmon[29288]: mon.a (mon.0) 1710 : cluster [DBG] osdmap e780: 8 total, 8 up, 7 in 2021-02-10T04:40:00.259 INFO:tasks.rados.rados.0.smithi085.stdout:3064: done (8 left) 2021-02-10T04:40:00.259 INFO:tasks.rados.rados.0.smithi085.stdout:3065: done (7 left) 2021-02-10T04:40:00.260 INFO:tasks.rados.rados.0.smithi085.stdout:3067: done (6 left) 2021-02-10T04:40:00.260 INFO:tasks.rados.rados.0.smithi085.stdout:3068: done (5 left) 2021-02-10T04:40:00.260 INFO:tasks.rados.rados.0.smithi085.stdout:3069: done (4 left) 2021-02-10T04:40:00.260 INFO:tasks.rados.rados.0.smithi085.stdout:3070: copy_from oid 8 from oid 15 current snap is 349 2021-02-10T04:40:00.260 INFO:tasks.rados.rados.0.smithi085.stdout:3071: write (excl) oid 10 current snap is 349 2021-02-10T04:40:00.260 INFO:tasks.rados.rados.0.smithi085.stdout:3071: seq_num 1063 ranges {665841=562754,1687653=572365,2708406=776032} 2021-02-10T04:40:00.266 INFO:tasks.rados.rados.0.smithi085.stdout:3071: writing smithi08575397-10 from 665841 to 1228595 tid 1 2021-02-10T04:40:00.272 INFO:tasks.rados.rados.0.smithi085.stdout:3071: writing smithi08575397-10 from 1687653 to 2260018 tid 2 2021-02-10T04:40:00.280 INFO:tasks.rados.rados.0.smithi085.stdout:3071: writing smithi08575397-10 from 2708406 to 3484438 tid 3 2021-02-10T04:40:00.281 INFO:tasks.rados.rados.0.smithi085.stdout:3072: read oid 14 snap -1 2021-02-10T04:40:00.281 INFO:tasks.rados.rados.0.smithi085.stdout:3072: expect (ObjNum 982 snap 317 seq_num 982) 2021-02-10T04:40:00.281 INFO:tasks.rados.rados.0.smithi085.stdout:3073: copy_from oid 49 from oid 9 current snap is 349 2021-02-10T04:40:00.281 INFO:tasks.rados.rados.0.smithi085.stdout:3071: finishing write tid 1 to smithi08575397-10 2021-02-10T04:40:00.281 INFO:tasks.rados.rados.0.smithi085.stdout:3074: write oid 42 current snap is 349 2021-02-10T04:40:00.281 INFO:tasks.rados.rados.0.smithi085.stdout:3071: finishing write tid 2 to smithi08575397-10 2021-02-10T04:40:00.281 INFO:tasks.rados.rados.0.smithi085.stdout:3074: seq_num 1064 ranges {682648=638003,2029782=432376,3019476=587304} 2021-02-10T04:40:00.288 INFO:tasks.rados.rados.0.smithi085.stdout:3074: writing smithi08575397-42 from 682648 to 1320651 tid 1 2021-02-10T04:40:00.294 INFO:tasks.rados.rados.0.smithi085.stdout:3074: writing smithi08575397-42 from 2029782 to 2462158 tid 2 2021-02-10T04:40:00.301 INFO:tasks.rados.rados.0.smithi085.stdout:3074: writing smithi08575397-42 from 3019476 to 3606780 tid 3 2021-02-10T04:40:00.301 INFO:tasks.rados.rados.0.smithi085.stdout:3075: snap_remove snap 332 2021-02-10T04:40:00.327 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:40:00.357 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:40:00.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:00 smithi164 conmon[29288]: cluster 2021-02-10T04:40:00.000086+0000 2021-02-10T04:40:00.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:00 smithi164 conmon[29288]: mon.a (mon.0) 1711 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set 2021-02-10T04:40:00.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:00 smithi164 conmon[29288]: cluster 2021-02-10T04:40:00.000110+0000 mon.a (mon.0) 1712 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2021-02-10T04:40:00.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:00 smithi164 conmon[29288]: cluster 2021-02-10T04:40:00.185642+0000 mgr.y (mgr.14140) 770 : cluster [DBG] pgmap v1321: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 475 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.4 MiB/s wr, 19 op/s 2021-02-10T04:40:00.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:00 smithi164 conmon[29288]: cluster 2021-02-10T04:40:00.257801+0000 mon.a (mon.0) 1713 : cluster [DBG] osdmap e781: 8 total, 8 up, 7 in 2021-02-10T04:40:00.645 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:00 smithi164 conmon[33677]: debug 2021-02-10T04:40:00.444+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:40:00.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:00 smithi085 conmon[29319]: cluster 2021-02-10T04:40:00.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:00 smithi085 conmon[29319]: 2021-02-10T04:40:00.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:00 smithi085 conmon[29319]: 2021 2021-02-10T04:40:00.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:00 smithi085 conmon[29319]: - 2021-02-10T04:40:00.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:00 smithi085 conmon[29319]: 02-10T04:40:00.000086 2021-02-10T04:40:00.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:00 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1711 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set 2021-02-10T04:40:00.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:00 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:40:00.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:00 smithi085 conmon[29319]: -10T04:40:00.000110+0000 mon.a (mon.0 2021-02-10T04:40:00.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:00 smithi085 conmon[29319]: ) 1712 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2021-02-10T04:40:00.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:00 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:40:00.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:00 smithi085 conmon[29319]: 02-10T04:40:00.185642+0000 mgr.y (mgr. 2021-02-10T04:40:00.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:00 smithi085 conmon[29319]: 14140) 770 : cluster [DBG] pgmap v1321: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 475 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.4 MiB/s wr, 19 op/s 2021-02-10T04:40:00.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:00 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:40:00.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:00 smithi085 conmon[29319]: -10T04:40: 2021-02-10T04:40:00.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:00 smithi085 conmon[29319]: 00.257801+0000 mon.a (mon.0) 1713 : 2021-02-10T04:40:00.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:00 smithi085 conmon[29319]: cluster [DBG] osdmap e781: 8 total, 8 up, 7 in 2021-02-10T04:40:00.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:00 smithi085 conmon[35961]: cluster 2021-02-10T04:40:00.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:00 smithi085 conmon[35961]: 2021-02-10T04:40 2021-02-10T04:40:00.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:00 smithi085 conmon[35961]: :00.000086+0000 mon.a (mon.0) 1711 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set 2021-02-10T04:40:00.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:00 smithi085 conmon[35961]: cluster 2021-02-10T04:40:00.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:00 smithi085 conmon[35961]: 2021-02-10T04:40:00.000110 2021-02-10T04:40:00.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:00 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1712 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2021-02-10T04:40:00.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:00 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:40:00.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:00 smithi085 conmon[35961]: 04:40:00.185642+0000 mgr.y (mgr.14140 2021-02-10T04:40:00.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:00 smithi085 conmon[35961]: ) 770 : cluster [DBG] pgmap v1321: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 475 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.4 MiB/s wr, 19 op/s 2021-02-10T04:40:00.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:00 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:40:00.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:00 smithi085 conmon[35961]: 04:40:00.257801+0000 mon.a (mon.0 2021-02-10T04:40:00.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:00 smithi085 conmon[35961]: ) 1713 : cluster [DBG] osdmap e781: 8 total, 8 up, 7 in 2021-02-10T04:40:00.825 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:40:00.931 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:40:01.764 INFO:tasks.rados.rados.0.smithi085.stdout:3075: done (9 left) 2021-02-10T04:40:01.764 INFO:tasks.rados.rados.0.smithi085.stdout:3076: snap_create 2021-02-10T04:40:01.764 INFO:tasks.rados.rados.0.smithi085.stdout:3070: finishing copy_from to smithi08575397-8 2021-02-10T04:40:01.764 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 1364 (ObjNum 1022 snap 335 seq_num 1022) dirty exists 2021-02-10T04:40:01.764 INFO:tasks.rados.rados.0.smithi085.stdout:3070: finishing copy_from racing read to smithi08575397-8 2021-02-10T04:40:01.764 INFO:tasks.rados.rados.0.smithi085.stdout:3071: finishing write tid 3 to smithi08575397-10 2021-02-10T04:40:01.764 INFO:tasks.rados.rados.0.smithi085.stdout:3071: finishing write tid 4 to smithi08575397-10 2021-02-10T04:40:01.764 INFO:tasks.rados.rados.0.smithi085.stdout:3071: finishing write tid 5 to smithi08575397-10 2021-02-10T04:40:01.764 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 10 v 1460 (ObjNum 1063 snap 349 seq_num 1063) dirty exists 2021-02-10T04:40:01.764 INFO:tasks.rados.rados.0.smithi085.stdout:3071: left oid 10 (ObjNum 1063 snap 349 seq_num 1063) 2021-02-10T04:40:01.765 INFO:tasks.rados.rados.0.smithi085.stdout:3072: expect (ObjNum 982 snap 317 seq_num 982) 2021-02-10T04:40:01.765 INFO:tasks.rados.rados.0.smithi085.stdout:3074: finishing write tid 1 to smithi08575397-42 2021-02-10T04:40:01.765 INFO:tasks.rados.rados.0.smithi085.stdout:3074: finishing write tid 2 to smithi08575397-42 2021-02-10T04:40:01.765 INFO:tasks.rados.rados.0.smithi085.stdout:3074: finishing write tid 3 to smithi08575397-42 2021-02-10T04:40:01.765 INFO:tasks.rados.rados.0.smithi085.stdout:3074: finishing write tid 4 to smithi08575397-42 2021-02-10T04:40:01.765 INFO:tasks.rados.rados.0.smithi085.stdout:3074: finishing write tid 5 to smithi08575397-42 2021-02-10T04:40:01.765 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 1457 (ObjNum 1064 snap 349 seq_num 1064) dirty exists 2021-02-10T04:40:01.765 INFO:tasks.rados.rados.0.smithi085.stdout:3074: left oid 42 (ObjNum 1064 snap 349 seq_num 1064) 2021-02-10T04:40:01.765 INFO:tasks.rados.rados.0.smithi085.stdout:3073: finishing copy_from to smithi08575397-49 2021-02-10T04:40:01.765 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 49 v 1384 (ObjNum 1053 snap 346 seq_num 1053) dirty exists 2021-02-10T04:40:01.765 INFO:tasks.rados.rados.0.smithi085.stdout:3073: finishing copy_from racing read to smithi08575397-49 2021-02-10T04:40:01.796 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:40:02.144 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:40:01 smithi164 conmon[44544]: debug 2021-02-10T04:40:01.883+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:40:02.218 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:40:02.262 INFO:tasks.rados.rados.0.smithi085.stdout:3070: done (9 left) 2021-02-10T04:40:02.262 INFO:tasks.rados.rados.0.smithi085.stdout:3071: done (8 left) 2021-02-10T04:40:02.262 INFO:tasks.rados.rados.0.smithi085.stdout:3072: done (7 left) 2021-02-10T04:40:02.262 INFO:tasks.rados.rados.0.smithi085.stdout:3073: done (6 left) 2021-02-10T04:40:02.262 INFO:tasks.rados.rados.0.smithi085.stdout:3074: done (5 left) 2021-02-10T04:40:02.262 INFO:tasks.rados.rados.0.smithi085.stdout:3076: done (4 left) 2021-02-10T04:40:02.262 INFO:tasks.rados.rados.0.smithi085.stdout:3077: read oid 44 snap 303 2021-02-10T04:40:02.263 INFO:tasks.rados.rados.0.smithi085.stdout:3077: expect (ObjNum 266 snap 64 seq_num 266) 2021-02-10T04:40:02.263 INFO:tasks.rados.rados.0.smithi085.stdout:3078: read oid 6 snap -1 2021-02-10T04:40:02.263 INFO:tasks.rados.rados.0.smithi085.stdout:3078: expect (ObjNum 1028 snap 335 seq_num 1028) 2021-02-10T04:40:02.263 INFO:tasks.rados.rados.0.smithi085.stdout:3079: read oid 16 snap -1 2021-02-10T04:40:02.263 INFO:tasks.rados.rados.0.smithi085.stdout:3079: expect deleted 2021-02-10T04:40:02.263 INFO:tasks.rados.rados.0.smithi085.stdout:3080: delete oid 28 current snap is 350 2021-02-10T04:40:02.272 INFO:tasks.rados.rados.0.smithi085.stdout:3079: done (7 left) 2021-02-10T04:40:02.272 INFO:tasks.rados.rados.0.smithi085.stdout:3080: done (6 left) 2021-02-10T04:40:02.272 INFO:tasks.rados.rados.0.smithi085.stdout:3081: snap_remove snap 339 2021-02-10T04:40:02.323 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:40:02.357 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:40:02.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:02 smithi164 conmon[29288]: cluster 2021-02-10T04:40:01.262106+0000 mon.a ( 2021-02-10T04:40:02.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:02 smithi164 conmon[29288]: mon.0) 1714 : cluster [DBG] osdmap e782: 8 total, 8 up, 7 in 2021-02-10T04:40:02.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:02 smithi164 conmon[29288]: cluster 2021-02-10T04:40:02.186215+0000 mgr.y (mgr.14140) 771 : cluster [DBG] pgmap v1324: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 475 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.7 MiB/s wr, 14 op/s 2021-02-10T04:40:02.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[29319]: cluster 2021-02-10T04:40:02.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[29319]: 2021-02-10T04:40:02.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[29319]: 2021 2021-02-10T04:40:02.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[29319]: - 2021-02-10T04:40:02.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[29319]: 02 2021-02-10T04:40:02.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[29319]: - 2021-02-10T04:40:02.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[29319]: 10 2021-02-10T04:40:02.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[29319]: T 2021-02-10T04:40:02.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[29319]: 04 2021-02-10T04:40:02.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[29319]: :40:01.262106 2021-02-10T04:40:02.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1714 : cluster [DBG] osdmap e782: 8 total, 8 up, 7 in 2021-02-10T04:40:02.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:40:02.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[29319]: 02-10T04:40:02.186215+0000 mgr.y (mgr 2021-02-10T04:40:02.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[29319]: .14140) 771 : cluster [DBG] pgmap v1324: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 475 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.7 MiB/s wr, 14 op/s 2021-02-10T04:40:02.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: cluster 2021-02-10T04:40:02.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: 2021-02-10T04:40:02.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: 2021 2021-02-10T04:40:02.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: - 2021-02-10T04:40:02.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: 02 2021-02-10T04:40:02.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: - 2021-02-10T04:40:02.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: 10 2021-02-10T04:40:02.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: T 2021-02-10T04:40:02.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: 04 2021-02-10T04:40:02.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: : 2021-02-10T04:40:02.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: 40 2021-02-10T04:40:02.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: : 2021-02-10T04:40:02.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: 01 2021-02-10T04:40:02.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: . 2021-02-10T04:40:02.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: 262106 2021-02-10T04:40:02.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: +0000 2021-02-10T04:40:02.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: 2021-02-10T04:40:02.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: mon.a 2021-02-10T04:40:02.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: ( 2021-02-10T04:40:02.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: mon 2021-02-10T04:40:02.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: . 2021-02-10T04:40:02.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: 0 2021-02-10T04:40:02.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: ) 2021-02-10T04:40:02.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: 1714 2021-02-10T04:40:02.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: : 2021-02-10T04:40:02.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: cluster 2021-02-10T04:40:02.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: 2021-02-10T04:40:02.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: [DBG] 2021-02-10T04:40:02.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: 2021-02-10T04:40:02.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: osdmap e782: 8 total, 8 up, 7 in 2021-02-10T04:40:02.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: 2021-02-10T04:40:02.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: cluster 2021-02-10T04:40:02.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: 2021-02-10T04:40:02.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: 2021 2021-02-10T04:40:02.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: - 2021-02-10T04:40:02.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: 02 2021-02-10T04:40:02.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: - 2021-02-10T04:40:02.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: 10 2021-02-10T04:40:02.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: T 2021-02-10T04:40:02.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: 04 2021-02-10T04:40:02.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: : 2021-02-10T04:40:02.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: 40 2021-02-10T04:40:02.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: : 2021-02-10T04:40:02.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: 02 2021-02-10T04:40:02.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: . 2021-02-10T04:40:02.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: 186215 2021-02-10T04:40:02.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: +0000 2021-02-10T04:40:02.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: 2021-02-10T04:40:02.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: mgr.y 2021-02-10T04:40:02.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: ( 2021-02-10T04:40:02.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: mgr 2021-02-10T04:40:02.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: . 2021-02-10T04:40:02.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: 14140 2021-02-10T04:40:02.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: ) 2021-02-10T04:40:02.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: 771 2021-02-10T04:40:02.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: : 2021-02-10T04:40:02.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: cluster 2021-02-10T04:40:02.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: 2021-02-10T04:40:02.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: [DBG] 2021-02-10T04:40:02.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: 2021-02-10T04:40:02.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: pgmap v1324: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 475 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.7 MiB/s wr, 14 op/s 2021-02-10T04:40:02.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:02 smithi085 conmon[35961]: 2021-02-10T04:40:03.269 INFO:tasks.rados.rados.0.smithi085.stdout:3081: done (6 left) 2021-02-10T04:40:03.269 INFO:tasks.rados.rados.0.smithi085.stdout:3082: write oid 35 current snap is 350 2021-02-10T04:40:03.270 INFO:tasks.rados.rados.0.smithi085.stdout:3082: seq_num 1066 ranges {716262=504854,1863099=719692,3074322=726201} 2021-02-10T04:40:03.278 INFO:tasks.rados.rados.0.smithi085.stdout:3082: writing smithi08575397-35 from 716262 to 1221116 tid 1 2021-02-10T04:40:03.285 INFO:tasks.rados.rados.0.smithi085.stdout:3082: writing smithi08575397-35 from 1863099 to 2582791 tid 2 2021-02-10T04:40:03.293 INFO:tasks.rados.rados.0.smithi085.stdout:3082: writing smithi08575397-35 from 3074322 to 3800523 tid 3 2021-02-10T04:40:03.293 INFO:tasks.rados.rados.0.smithi085.stdout:3083: snap_create 2021-02-10T04:40:03.293 INFO:tasks.rados.rados.0.smithi085.stdout:3077: expect (ObjNum 266 snap 64 seq_num 266) 2021-02-10T04:40:03.452 INFO:tasks.rados.rados.0.smithi085.stdout:3078: expect (ObjNum 1028 snap 335 seq_num 1028) 2021-02-10T04:40:03.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:03 smithi164 conmon[29288]: cluster 2021-02-10T04:40:02.260650+0000 mon.a (mon.0 2021-02-10T04:40:03.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:03 smithi164 conmon[29288]: ) 1715 : cluster [DBG] osdmap e783: 8 total, 8 up, 7 in 2021-02-10T04:40:03.665 INFO:tasks.rados.rados.0.smithi085.stdout:3082: finishing write tid 1 to smithi08575397-35 2021-02-10T04:40:03.665 INFO:tasks.rados.rados.0.smithi085.stdout:3082: finishing write tid 2 to smithi08575397-35 2021-02-10T04:40:03.665 INFO:tasks.rados.rados.0.smithi085.stdout:3082: finishing write tid 3 to smithi08575397-35 2021-02-10T04:40:03.665 INFO:tasks.rados.rados.0.smithi085.stdout:3082: finishing write tid 4 to smithi08575397-35 2021-02-10T04:40:03.665 INFO:tasks.rados.rados.0.smithi085.stdout:3082: finishing write tid 5 to smithi08575397-35 2021-02-10T04:40:03.665 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 35 v 1464 (ObjNum 1066 snap 350 seq_num 1066) dirty exists 2021-02-10T04:40:03.665 INFO:tasks.rados.rados.0.smithi085.stdout:3082: left oid 35 (ObjNum 1066 snap 350 seq_num 1066) 2021-02-10T04:40:03.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:03 smithi085 conmon[29319]: cluster 2021-02-10T04:40:02.260650+0000 mon.a (mon.0) 1715 : cluster [DBG] osdmap e783: 8 total, 8 up, 7 in 2021-02-10T04:40:03.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:03 smithi085 conmon[35961]: cluster 2021-02-10T04:40:02.260650+0000 mon.a (mon.0) 1715 : cluster [DBG] osdmap e783: 8 total, 8 up, 7 in 2021-02-10T04:40:04.282 INFO:tasks.rados.rados.0.smithi085.stdout:3077: done (7 left) 2021-02-10T04:40:04.282 INFO:tasks.rados.rados.0.smithi085.stdout:3078: done (6 left) 2021-02-10T04:40:04.282 INFO:tasks.rados.rados.0.smithi085.stdout:3082: done (5 left) 2021-02-10T04:40:04.282 INFO:tasks.rados.rados.0.smithi085.stdout:3083: done (4 left) 2021-02-10T04:40:04.283 INFO:tasks.rados.rados.0.smithi085.stdout:3084: read oid 37 snap -1 2021-02-10T04:40:04.283 INFO:tasks.rados.rados.0.smithi085.stdout:3084: expect (ObjNum 1061 snap 348 seq_num 1061) 2021-02-10T04:40:04.283 INFO:tasks.rados.rados.0.smithi085.stdout:3085: copy_from oid 43 from oid 30 current snap is 351 2021-02-10T04:40:04.283 INFO:tasks.rados.rados.0.smithi085.stdout:3086: snap_remove snap 326 2021-02-10T04:40:04.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:04 smithi164 conmon[29288]: cluster 2021-02-10T04:40:04.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:04 smithi164 conmon[29288]: 2021-02-10T04:40:04.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:04 smithi164 conmon[29288]: 2021 2021-02-10T04:40:04.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:04 smithi164 conmon[29288]: - 2021-02-10T04:40:04.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:04 smithi164 conmon[29288]: 02 2021-02-10T04:40:04.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:04 smithi164 conmon[29288]: - 2021-02-10T04:40:04.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:04 smithi164 conmon[29288]: 10 2021-02-10T04:40:04.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:04 smithi164 conmon[29288]: T 2021-02-10T04:40:04.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:04 smithi164 conmon[29288]: 04 2021-02-10T04:40:04.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:04 smithi164 conmon[29288]: : 2021-02-10T04:40:04.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:04 smithi164 conmon[29288]: 40 2021-02-10T04:40:04.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:04 smithi164 conmon[29288]: : 2021-02-10T04:40:04.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:04 smithi164 conmon[29288]: 03 2021-02-10T04:40:04.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:04 smithi164 conmon[29288]: . 2021-02-10T04:40:04.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:04 smithi164 conmon[29288]: 267776 2021-02-10T04:40:04.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:04 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1716 : 2021-02-10T04:40:04.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:04 smithi164 conmon[29288]: cluster [DBG] osdmap e784: 8 total, 8 up, 7 in 2021-02-10T04:40:04.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:04 smithi164 conmon[29288]: cluster 2021-02-10T04:40:04.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:04 smithi164 conmon[29288]: 2021-02-10T04:40:04.186874 2021-02-10T04:40:04.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:04 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 772 : cluster 2021-02-10T04:40:04.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:04 smithi164 conmon[29288]: [DBG] pgmap v1327: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 489 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 10 op/s 2021-02-10T04:40:04.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:04 smithi085 conmon[35961]: cluster 2021-02-10T04:40:03.267776 2021-02-10T04:40:04.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:04 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1716 : cluster [DBG] osdmap e784: 8 total, 8 up, 7 in 2021-02-10T04:40:04.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:04 smithi085 conmon[35961]: cluster 2021-02-10T04:40 2021-02-10T04:40:04.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:04 smithi085 conmon[35961]: :04.186874+0000 mgr.y (mgr.14140) 772 2021-02-10T04:40:04.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:04 smithi085 conmon[35961]: : cluster [DBG] pgmap v1327: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 489 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 10 op/s 2021-02-10T04:40:04.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:04 smithi085 conmon[29319]: cluster 2021-02-10T04:40:04.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:04 smithi085 conmon[29319]: 2021-02-10T04:40:04.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:04 smithi085 conmon[29319]: 2021-02-10T04:40:03.267776+0000 mon.a ( 2021-02-10T04:40:04.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:04 smithi085 conmon[29319]: mon.0) 1716 : cluster [DBG] osdmap e784: 8 total, 8 up, 7 in 2021-02-10T04:40:04.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:04 smithi085 conmon[29319]: cluster 2021-02-10T04:40:04. 2021-02-10T04:40:04.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:04 smithi085 conmon[29319]: 186874+0000 mgr.y (mgr.14140) 772 : cluster [DBG] pgmap v1327: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 489 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 10 op/s 2021-02-10T04:40:05.282 INFO:tasks.rados.rados.0.smithi085.stdout:3086: done (6 left) 2021-02-10T04:40:05.282 INFO:tasks.rados.rados.0.smithi085.stdout:3087: delete oid 1 current snap is 351 2021-02-10T04:40:05.282 INFO:tasks.rados.rados.0.smithi085.stdout:3085: finishing copy_from to smithi08575397-43 2021-02-10T04:40:05.282 INFO:tasks.rados.rados.0.smithi085.stdout:3085: got expected ENOENT (src dne) 2021-02-10T04:40:05.282 INFO:tasks.rados.rados.0.smithi085.stdout:3085: finishing copy_from racing read to smithi08575397-43 2021-02-10T04:40:05.282 INFO:tasks.rados.rados.0.smithi085.stdout:3084: expect (ObjNum 1061 snap 348 seq_num 1061) 2021-02-10T04:40:05.544 INFO:tasks.rados.rados.0.smithi085.stdout:3084: done (6 left) 2021-02-10T04:40:05.544 INFO:tasks.rados.rados.0.smithi085.stdout:3085: done (5 left) 2021-02-10T04:40:05.544 INFO:tasks.rados.rados.0.smithi085.stdout:3087: done (4 left) 2021-02-10T04:40:05.544 INFO:tasks.rados.rados.0.smithi085.stdout:3088: snap_create 2021-02-10T04:40:05.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:05 smithi164 conmon[29288]: cluster 2021-02-10T04:40:04.281553+0000 mon.a (mon.0) 1717 : cluster [DBG] osdmap e785: 8 total, 8 up, 7 in 2021-02-10T04:40:05.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:05 smithi164 conmon[29288]: 2021-02-10T04:40:05.663 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:05 smithi085 conmon[29319]: cluster 2021-02-10T04:40:05.663 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:05 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:40:05.663 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:05 smithi085 conmon[29319]: 10T04:40:04.281553+0000 mon.a (mon.0) 1717 : 2021-02-10T04:40:05.663 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:05 smithi085 conmon[29319]: cluster [DBG] osdmap e785: 8 total, 8 up, 7 in 2021-02-10T04:40:05.663 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:05 smithi085 conmon[35961]: cluster 2021-02-10T04:40:04.281553+0000 2021-02-10T04:40:05.663 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:05 smithi085 conmon[35961]: mon.a (mon.0) 1717 : cluster [DBG] osdmap e785: 8 total, 8 up, 7 in 2021-02-10T04:40:06.299 INFO:tasks.rados.rados.0.smithi085.stdout:3088: done (4 left) 2021-02-10T04:40:06.299 INFO:tasks.rados.rados.0.smithi085.stdout:3089: read oid 18 snap -1 2021-02-10T04:40:06.299 INFO:tasks.rados.rados.0.smithi085.stdout:3089: expect (ObjNum 952 snap 313 seq_num 952) 2021-02-10T04:40:06.299 INFO:tasks.rados.rados.0.smithi085.stdout:3090: read oid 1 snap -1 2021-02-10T04:40:06.299 INFO:tasks.rados.rados.0.smithi085.stdout:3090: expect deleted 2021-02-10T04:40:06.299 INFO:tasks.rados.rados.0.smithi085.stdout:3091: write oid 11 current snap is 352 2021-02-10T04:40:06.299 INFO:tasks.rados.rados.0.smithi085.stdout:3091: seq_num 1068 ranges {749876=655352,1863710=723361,3012815=748745} 2021-02-10T04:40:06.307 INFO:tasks.rados.rados.0.smithi085.stdout:3091: writing smithi08575397-11 from 749876 to 1405228 tid 1 2021-02-10T04:40:06.314 INFO:tasks.rados.rados.0.smithi085.stdout:3091: writing smithi08575397-11 from 1863710 to 2587071 tid 2 2021-02-10T04:40:06.322 INFO:tasks.rados.rados.0.smithi085.stdout:3091: writing smithi08575397-11 from 3012815 to 3761560 tid 3 2021-02-10T04:40:06.322 INFO:tasks.rados.rados.0.smithi085.stdout:3092: rollback oid 43 current snap is 352 2021-02-10T04:40:06.322 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 43 to 344 2021-02-10T04:40:06.322 INFO:tasks.rados.rados.0.smithi085.stdout:3089: expect (ObjNum 952 snap 313 seq_num 952) 2021-02-10T04:40:06.429 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:40:06.441 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:40:06 smithi085 conmon[52836]: debug 2021-02-10T04:40:06.442 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:40:06 smithi085 conmon[52836]: 2021-02-10T04:40:06.193+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:40:06.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:06 smithi085 conmon[29319]: cluster 2021-02-10T04:40:05.281593+0000 mon.a (mon.0) 1718 : cluster [DBG] osdmap e786: 8 total, 8 up, 7 in 2021-02-10T04:40:06.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:06 smithi085 conmon[29319]: cluster 2021-02-10T04:40:05.709217+0000 osd.5 (osd.5) 80 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:40:06.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:06 smithi085 conmon[29319]: cluster 2021-02-10T04:40:05.710051+0000 osd.5 (osd.5) 81 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:40:06.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:06 smithi085 conmon[29319]: cluster 2021-02-10T04:40:06.187370+0000 mgr.y (mgr.14140) 2021-02-10T04:40:06.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:06 smithi085 conmon[29319]: 773 : cluster [DBG] pgmap v1330: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 491 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 4.8 MiB/s wr, 19 op/s 2021-02-10T04:40:06.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:06 smithi085 conmon[35961]: cluster 2021-02-10T04:40:05.281593+0000 2021-02-10T04:40:06.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:06 smithi085 conmon[35961]: mon.a (mon.0) 1718 : cluster [DBG] osdmap e786: 8 total, 8 up, 7 in 2021-02-10T04:40:06.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:06 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:40:06.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:06 smithi085 conmon[35961]: :40:05.709217+0000 osd.5 (osd.5) 80 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:40:06.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:06 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:40:06.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:06 smithi085 conmon[35961]: 04:40:05.710051+0000 osd.5 (osd.5) 2021-02-10T04:40:06.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:06 smithi085 conmon[35961]: 81 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:40:06.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:06 smithi085 conmon[35961]: cluster 2021-02-10T04:40 2021-02-10T04:40:06.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:06 smithi085 conmon[35961]: :06.187370+0000 mgr.y (mgr.14140) 773 : cluster 2021-02-10T04:40:06.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:06 smithi085 conmon[35961]: [DBG] pgmap v1330: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 491 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 4.8 MiB/s wr, 19 op/s 2021-02-10T04:40:06.481 INFO:tasks.rados.rados.0.smithi085.stdout:3091: finishing write tid 1 to smithi08575397-11 2021-02-10T04:40:06.481 INFO:tasks.rados.rados.0.smithi085.stdout:3091: finishing write tid 2 to smithi08575397-11 2021-02-10T04:40:06.481 INFO:tasks.rados.rados.0.smithi085.stdout:3092: finishing rollback tid 0 to smithi08575397-43 2021-02-10T04:40:06.481 INFO:tasks.rados.rados.0.smithi085.stdout:3092: finishing rollback tid 1 to smithi08575397-43 2021-02-10T04:40:06.481 INFO:tasks.rados.rados.0.smithi085.stdout:3091: finishing write tid 3 to smithi08575397-11 2021-02-10T04:40:06.482 INFO:tasks.rados.rados.0.smithi085.stdout:3092: finishing rollback tid 2 to smithi08575397-43 2021-02-10T04:40:06.482 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 1460 (ObjNum 1044 snap 341 seq_num 1044) dirty exists 2021-02-10T04:40:06.482 INFO:tasks.rados.rados.0.smithi085.stdout:3091: finishing write tid 4 to smithi08575397-11 2021-02-10T04:40:06.482 INFO:tasks.rados.rados.0.smithi085.stdout:3089: done (7 left) 2021-02-10T04:40:06.482 INFO:tasks.rados.rados.0.smithi085.stdout:3090: done (6 left) 2021-02-10T04:40:06.482 INFO:tasks.rados.rados.0.smithi085.stdout:3092: done (5 left) 2021-02-10T04:40:06.482 INFO:tasks.rados.rados.0.smithi085.stdout:3093: read oid 15 snap -1 2021-02-10T04:40:06.482 INFO:tasks.rados.rados.0.smithi085.stdout:3093: expect (ObjNum 1022 snap 335 seq_num 1022) 2021-02-10T04:40:06.482 INFO:tasks.rados.rados.0.smithi085.stdout:3094: read oid 21 snap -1 2021-02-10T04:40:06.482 INFO:tasks.rados.rados.0.smithi085.stdout:3094: expect (ObjNum 1032 snap 340 seq_num 1032) 2021-02-10T04:40:06.482 INFO:tasks.rados.rados.0.smithi085.stdout:3095: snap_create 2021-02-10T04:40:06.483 INFO:tasks.rados.rados.0.smithi085.stdout:3091: finishing write tid 5 to smithi08575397-11 2021-02-10T04:40:06.483 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 1608 (ObjNum 1068 snap 352 seq_num 1068) dirty exists 2021-02-10T04:40:06.483 INFO:tasks.rados.rados.0.smithi085.stdout:3091: left oid 11 (ObjNum 1068 snap 352 seq_num 1068) 2021-02-10T04:40:06.493 INFO:tasks.rados.rados.0.smithi085.stdout:3093: expect (ObjNum 1022 snap 335 seq_num 1022) 2021-02-10T04:40:06.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:06 smithi164 conmon[29288]: cluster 2021-02-10T04:40:05.281593 2021-02-10T04:40:06.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:06 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1718 : cluster [DBG] osdmap e786: 8 total, 8 up, 7 in 2021-02-10T04:40:06.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:06 smithi164 conmon[29288]: cluster 2021-02-10T04:40:05.709217+0000 osd.5 (osd.5) 80 : cluster 2021-02-10T04:40:06.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:06 smithi164 conmon[29288]: [DBG] purged_snaps scrub starts 2021-02-10T04:40:06.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:06 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:40:06.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:06 smithi164 conmon[29288]: 02-10T04:40:05.710051+0000 osd.5 (osd.5) 2021-02-10T04:40:06.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:06 smithi164 conmon[29288]: 81 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:40:06.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:06 smithi164 conmon[29288]: cluster 2021-02-10T04:40:06.187370+0000 2021-02-10T04:40:06.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:06 smithi164 conmon[29288]: mgr.y (mgr.14140) 773 : cluster [DBG] pgmap v1330: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 491 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 4.8 MiB/s wr, 19 op/s 2021-02-10T04:40:06.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:06 smithi164 conmon[29288]: 2021-02-10T04:40:06.712 INFO:tasks.rados.rados.0.smithi085.stdout:3094: expect (ObjNum 1032 snap 340 seq_num 1032) 2021-02-10T04:40:07.306 INFO:tasks.rados.rados.0.smithi085.stdout:3091: done (7 left) 2021-02-10T04:40:07.306 INFO:tasks.rados.rados.0.smithi085.stdout:3093: done (6 left) 2021-02-10T04:40:07.306 INFO:tasks.rados.rados.0.smithi085.stdout:3094: done (5 left) 2021-02-10T04:40:07.306 INFO:tasks.rados.rados.0.smithi085.stdout:3095: done (4 left) 2021-02-10T04:40:07.306 INFO:tasks.rados.rados.0.smithi085.stdout:3096: snap_create 2021-02-10T04:40:07.593 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:40:07.622 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:07 smithi164 conmon[29288]: cluster 2021-02-10T04:40:07.622 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:07 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:40:07.622 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:07 smithi164 conmon[29288]: 40:06.297054+0000 mon.a (mon.0) 1719 : cluster [DBG] osdmap e787: 8 total, 8 up, 7 in 2021-02-10T04:40:07.625 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:40:07.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:07 smithi085 conmon[35961]: cluster 2021-02-10T04:40:06 2021-02-10T04:40:07.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:07 smithi085 conmon[35961]: .297054+0000 mon.a (mon.0) 1719 : cluster [DBG] osdmap e787: 8 total, 8 up, 7 in 2021-02-10T04:40:07.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:07 smithi085 conmon[29319]: cluster 2021-02-10T04:40:06.297054+0000 mon.a ( 2021-02-10T04:40:07.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:07 smithi085 conmon[29319]: mon.0) 1719 : cluster [DBG] osdmap e787: 8 total, 8 up, 7 in 2021-02-10T04:40:07.893 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:40:07 smithi164 conmon[37263]: debug 2021-02-10T04:40:07.705+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:40:07.974 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:40:08.080 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:40:08.114 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:40:08.269 INFO:tasks.rados.rados.0.smithi085.stdout:3096: done (4 left) 2021-02-10T04:40:08.269 INFO:tasks.rados.rados.0.smithi085.stdout:3097: write (excl) oid 14 current snap is 354 2021-02-10T04:40:08.270 INFO:tasks.rados.rados.0.smithi085.stdout:3097: seq_num 1069 ranges {766683=730601,1922192=583372,3156591=676370} 2021-02-10T04:40:08.273 INFO:tasks.rados.rados.0.smithi085.stdout:3097: writing smithi08575397-14 from 766683 to 1497284 tid 1 2021-02-10T04:40:08.280 INFO:tasks.rados.rados.0.smithi085.stdout:3097: writing smithi08575397-14 from 1922192 to 2505564 tid 2 2021-02-10T04:40:08.291 INFO:tasks.rados.rados.0.smithi085.stdout:3097: writing smithi08575397-14 from 3156591 to 3832961 tid 3 2021-02-10T04:40:08.291 INFO:tasks.rados.rados.0.smithi085.stdout:3098: snap_create 2021-02-10T04:40:08.361 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:40:08 smithi164 conmon[44544]: debug 2021-02-10T04:40:08.192+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:40:08.428 INFO:tasks.rados.rados.0.smithi085.stdout:3097: finishing write tid 1 to smithi08575397-14 2021-02-10T04:40:08.432 INFO:tasks.rados.rados.0.smithi085.stdout:3097: finishing write tid 2 to smithi08575397-14 2021-02-10T04:40:08.435 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:40:08.441 INFO:tasks.rados.rados.0.smithi085.stdout:3097: finishing write tid 3 to smithi08575397-14 2021-02-10T04:40:08.441 INFO:tasks.rados.rados.0.smithi085.stdout:3097: finishing write tid 4 to smithi08575397-14 2021-02-10T04:40:08.442 INFO:tasks.rados.rados.0.smithi085.stdout:3097: finishing write tid 5 to smithi08575397-14 2021-02-10T04:40:08.442 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 1461 (ObjNum 1069 snap 354 seq_num 1069) dirty exists 2021-02-10T04:40:08.442 INFO:tasks.rados.rados.0.smithi085.stdout:3097: left oid 14 (ObjNum 1069 snap 354 seq_num 1069) 2021-02-10T04:40:08.542 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:40:08.581 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:40:08.644 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:40:08 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:40:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:08 smithi164 conmon[29288]: cluster 2021-02-10T04:40:07.303403+0000 mon.a (mon.0) 2021-02-10T04:40:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:08 smithi164 conmon[29288]: 1720 : cluster [DBG] osdmap e788: 8 total, 8 up, 7 in 2021-02-10T04:40:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:08 smithi164 conmon[29288]: cluster 2021-02-10T04:40:08.187939+0000 mgr.y (mgr.14140) 774 : cluster [DBG] pgmap v1333: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 488 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 9.5 MiB/s rd, 1.8 MiB/s wr, 10 op/s 2021-02-10T04:40:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:08 smithi164 conmon[29288]: cluster 2021-02-10T04:40:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:08 smithi164 conmon[29288]: 2021-02-10T04:40:08.259746+0000 mon.a (mon.0) 1721 : cluster [DBG] osdmap e789: 8 total, 8 up, 7 in 2021-02-10T04:40:08.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:08 smithi085 conmon[29319]: cluster 2021-02-10T04:40:07.303403+0000 mon.a (mon.0 2021-02-10T04:40:08.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:08 smithi085 conmon[29319]: ) 1720 : cluster [DBG] osdmap e788: 8 total, 8 up, 7 in 2021-02-10T04:40:08.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:08 smithi085 conmon[29319]: cluster 2021-02-10T04:40:08.187939+0000 mgr.y (mgr.14140) 774 : cluster [DBG] pgmap v1333: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 488 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 9.5 MiB/s rd, 1.8 MiB/s wr, 10 op/s 2021-02-10T04:40:08.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:08 smithi085 conmon[29319]: cluster 2021-02-10T04:40:08.259746+0000 mon.a (mon.0) 1721 : cluster [DBG] osdmap e789: 8 total, 8 up, 7 in 2021-02-10T04:40:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:08 smithi085 conmon[35961]: cluster 2021-02-10T04:40:07.303403+0000 mon.a (mon. 2021-02-10T04:40:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:08 smithi085 conmon[35961]: 0) 1720 : cluster [DBG] osdmap e788: 8 total, 8 up, 7 in 2021-02-10T04:40:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:08 smithi085 conmon[35961]: cluster 2021-02-10T04:40:08.187939+0000 mgr.y (mgr.14140) 774 : cluster [DBG] pgmap v1333: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 488 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 9.5 MiB/s rd, 1.8 MiB/s wr, 10 op/s 2021-02-10T04:40:08.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:08 smithi085 conmon[35961]: cluster 2021-02-10T04:40:08.259746+0000 mon.a (mon.0) 1721 : cluster [DBG] osdmap e789: 8 total, 8 up, 7 in 2021-02-10T04:40:08.921 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:40:09.027 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:40:09.063 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:40:09.191 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:40:08 smithi085 conmon[41540]: debug 2021-02-10T04:40:08.864+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:40:09.204 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:40:09.204 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:40:09.216 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:40:09.252 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:40:09.275 INFO:tasks.rados.rados.0.smithi085.stdout:3097: done (5 left) 2021-02-10T04:40:09.275 INFO:tasks.rados.rados.0.smithi085.stdout:3098: done (4 left) 2021-02-10T04:40:09.275 INFO:tasks.rados.rados.0.smithi085.stdout:3099: read oid 50 snap 353 2021-02-10T04:40:09.275 INFO:tasks.rados.rados.0.smithi085.stdout:3099: expect (ObjNum 1054 snap 346 seq_num 1054) 2021-02-10T04:40:09.275 INFO:tasks.rados.rados.0.smithi085.stdout:3100: write oid 28 current snap is 355 2021-02-10T04:40:09.275 INFO:tasks.rados.rados.0.smithi085.stdout:3100: seq_num 1070 ranges {783490=405850,1980674=727030,3184014=487642} 2021-02-10T04:40:09.282 INFO:tasks.rados.rados.0.smithi085.stdout:3100: writing smithi08575397-28 from 783490 to 1189340 tid 1 2021-02-10T04:40:09.293 INFO:tasks.rados.rados.0.smithi085.stdout:3100: writing smithi08575397-28 from 1980674 to 2707704 tid 2 2021-02-10T04:40:09.300 INFO:tasks.rados.rados.0.smithi085.stdout:3100: writing smithi08575397-28 from 3184014 to 3671656 tid 3 2021-02-10T04:40:09.301 INFO:tasks.rados.rados.0.smithi085.stdout:3099: expect (ObjNum 1054 snap 346 seq_num 1054) 2021-02-10T04:40:09.308 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:40:09.350 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:40:09.363 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:40:09 smithi164 conmon[44544]: debug 2021-02-10T04:40:09.142+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:40:09.388 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:40:09.409 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:40:09.416 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:40:09.455 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:40:09.486 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:40:09.487 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:40:09.487 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:40:09.487 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:40:09.488 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:40:09.488 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:40:09.488 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:40:09.488 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:40:09.488 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:40:09.514 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:40:09.516 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: audit 2021-02-10T04:40:09.516 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 2021-02-10T04:40:09.516 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 2021 2021-02-10T04:40:09.516 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: - 2021-02-10T04:40:09.516 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 02 2021-02-10T04:40:09.516 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: - 2021-02-10T04:40:09.516 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 10 2021-02-10T04:40:09.516 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: T 2021-02-10T04:40:09.516 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 04 2021-02-10T04:40:09.516 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: : 2021-02-10T04:40:09.516 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 40 2021-02-10T04:40:09.517 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: : 2021-02-10T04:40:09.517 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 08 2021-02-10T04:40:09.517 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: . 2021-02-10T04:40:09.517 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 377278 2021-02-10T04:40:09.517 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: +0000 2021-02-10T04:40:09.517 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 2021-02-10T04:40:09.517 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: mgr.y 2021-02-10T04:40:09.517 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: ( 2021-02-10T04:40:09.517 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: mgr 2021-02-10T04:40:09.517 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: . 2021-02-10T04:40:09.517 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 14140 2021-02-10T04:40:09.517 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: ) 2021-02-10T04:40:09.518 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 775 2021-02-10T04:40:09.518 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: : 2021-02-10T04:40:09.518 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: audit 2021-02-10T04:40:09.518 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 2021-02-10T04:40:09.518 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: [DBG] 2021-02-10T04:40:09.518 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 2021-02-10T04:40:09.518 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:40:09.518 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 2021-02-10T04:40:09.518 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: audit 2021-02-10T04:40:09.518 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 2021-02-10T04:40:09.518 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 2021 2021-02-10T04:40:09.518 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: - 2021-02-10T04:40:09.518 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 02 2021-02-10T04:40:09.519 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: - 2021-02-10T04:40:09.519 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 10 2021-02-10T04:40:09.519 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: T 2021-02-10T04:40:09.519 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 04 2021-02-10T04:40:09.519 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: : 2021-02-10T04:40:09.519 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 40 2021-02-10T04:40:09.519 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: : 2021-02-10T04:40:09.519 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 09 2021-02-10T04:40:09.519 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: . 2021-02-10T04:40:09.519 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 172022 2021-02-10T04:40:09.519 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: +0000 2021-02-10T04:40:09.519 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 2021-02-10T04:40:09.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: mon.a 2021-02-10T04:40:09.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: ( 2021-02-10T04:40:09.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: mon 2021-02-10T04:40:09.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: . 2021-02-10T04:40:09.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 0 2021-02-10T04:40:09.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: ) 2021-02-10T04:40:09.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 1722 2021-02-10T04:40:09.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: : 2021-02-10T04:40:09.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: audit 2021-02-10T04:40:09.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 2021-02-10T04:40:09.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: [DBG] 2021-02-10T04:40:09.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 2021-02-10T04:40:09.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/2226632181' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:40:09.520 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 2021-02-10T04:40:09.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: cluster 2021-02-10T04:40:09.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 2021-02-10T04:40:09.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 2021 2021-02-10T04:40:09.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: - 2021-02-10T04:40:09.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 02 2021-02-10T04:40:09.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: - 2021-02-10T04:40:09.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 10 2021-02-10T04:40:09.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: T 2021-02-10T04:40:09.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 04 2021-02-10T04:40:09.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: : 2021-02-10T04:40:09.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 40 2021-02-10T04:40:09.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: : 2021-02-10T04:40:09.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 09 2021-02-10T04:40:09.521 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: . 2021-02-10T04:40:09.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 273739 2021-02-10T04:40:09.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: +0000 2021-02-10T04:40:09.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 2021-02-10T04:40:09.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: mon.a 2021-02-10T04:40:09.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: ( 2021-02-10T04:40:09.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: mon 2021-02-10T04:40:09.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: . 2021-02-10T04:40:09.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 0 2021-02-10T04:40:09.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: ) 2021-02-10T04:40:09.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 1723 2021-02-10T04:40:09.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: : 2021-02-10T04:40:09.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: cluster 2021-02-10T04:40:09.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 2021-02-10T04:40:09.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: [DBG] 2021-02-10T04:40:09.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 2021-02-10T04:40:09.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: osdmap e790: 8 total, 8 up, 7 in 2021-02-10T04:40:09.523 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[35961]: 2021-02-10T04:40:09.524 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[29319]: audit 2021-02-10T04:40:08.377278+0000 2021-02-10T04:40:09.524 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[29319]: mgr.y (mgr.14140) 775 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:40:09.524 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[29319]: audit 2021-02-10T04:40:09.172022+0000 mon.a (mon.0) 1722 : audit [DBG] from='client.? 172.21.15.85:0/2226632181' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:40:09.524 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[29319]: cluster 2021-02-10T04:40:09.273739+0000 mon.a (mon.0) 1723 : cluster [DBG] osdmap e790: 8 total, 8 up, 7 in 2021-02-10T04:40:09.539 INFO:tasks.rados.rados.0.smithi085.stdout:3099: done (5 left) 2021-02-10T04:40:09.539 INFO:tasks.rados.rados.0.smithi085.stdout:3101: rollback oid 35 current snap is 355 2021-02-10T04:40:09.539 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 35 to 335 2021-02-10T04:40:09.539 INFO:tasks.rados.rados.0.smithi085.stdout:3102: rollback oid 1 current snap is 355 2021-02-10T04:40:09.540 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 1 to 325 2021-02-10T04:40:09.540 INFO:tasks.rados.rados.0.smithi085.stdout:3103: write oid 13 current snap is 355 2021-02-10T04:40:09.540 INFO:tasks.rados.rados.0.smithi085.stdout:3103: seq_num 1071 ranges {400297=481099,1639156=361141} 2021-02-10T04:40:09.545 INFO:tasks.rados.rados.0.smithi085.stdout:3103: writing smithi08575397-13 from 400297 to 881396 tid 1 2021-02-10T04:40:09.556 INFO:tasks.rados.rados.0.smithi085.stdout:3103: writing smithi08575397-13 from 1639156 to 2000297 tid 2 2021-02-10T04:40:09.556 INFO:tasks.rados.rados.0.smithi085.stdout:3104: rollback oid 8 current snap is 355 2021-02-10T04:40:09.556 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 8 to 310 2021-02-10T04:40:09.556 INFO:tasks.rados.rados.0.smithi085.stdout:3100: finishing write tid 1 to smithi08575397-28 2021-02-10T04:40:09.556 INFO:tasks.rados.rados.0.smithi085.stdout:3100: finishing write tid 2 to smithi08575397-28 2021-02-10T04:40:09.556 INFO:tasks.rados.rados.0.smithi085.stdout:3100: finishing write tid 3 to smithi08575397-28 2021-02-10T04:40:09.556 INFO:tasks.rados.rados.0.smithi085.stdout:3100: finishing write tid 4 to smithi08575397-28 2021-02-10T04:40:09.556 INFO:tasks.rados.rados.0.smithi085.stdout:3100: finishing write tid 5 to smithi08575397-28 2021-02-10T04:40:09.556 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 1288 (ObjNum 1070 snap 355 seq_num 1070) dirty exists 2021-02-10T04:40:09.556 INFO:tasks.rados.rados.0.smithi085.stdout:3100: left oid 28 (ObjNum 1070 snap 355 seq_num 1070) 2021-02-10T04:40:09.556 INFO:tasks.rados.rados.0.smithi085.stdout:3102: finishing rollback tid 1 to smithi08575397-1 2021-02-10T04:40:09.556 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 1550 (ObjNum 82948560 snap 22030 seq_num 82920912) dirty dne 2021-02-10T04:40:09.557 INFO:tasks.rados.rados.0.smithi085.stdout:3101: finishing rollback tid 0 to smithi08575397-35 2021-02-10T04:40:09.557 INFO:tasks.rados.rados.0.smithi085.stdout:3101: finishing rollback tid 1 to smithi08575397-35 2021-02-10T04:40:09.557 INFO:tasks.rados.rados.0.smithi085.stdout:3101: finishing rollback tid 2 to smithi08575397-35 2021-02-10T04:40:09.557 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 35 v 1467 (ObjNum 1023 snap 335 seq_num 1023) dirty exists 2021-02-10T04:40:09.557 INFO:tasks.rados.rados.0.smithi085.stdout:3103: finishing write tid 1 to smithi08575397-13 2021-02-10T04:40:09.557 INFO:tasks.rados.rados.0.smithi085.stdout:3100: done (8 left) 2021-02-10T04:40:09.557 INFO:tasks.rados.rados.0.smithi085.stdout:3101: done (7 left) 2021-02-10T04:40:09.557 INFO:tasks.rados.rados.0.smithi085.stdout:3102: done (6 left) 2021-02-10T04:40:09.557 INFO:tasks.rados.rados.0.smithi085.stdout:3105: read oid 9 snap -1 2021-02-10T04:40:09.557 INFO:tasks.rados.rados.0.smithi085.stdout:3105: expect (ObjNum 1053 snap 346 seq_num 1053) 2021-02-10T04:40:09.557 INFO:tasks.rados.rados.0.smithi085.stdout:3106: snap_create 2021-02-10T04:40:09.561 INFO:tasks.rados.rados.0.smithi085.stdout:3104: finishing rollback tid 0 to smithi08575397-8 2021-02-10T04:40:09.561 INFO:tasks.rados.rados.0.smithi085.stdout:3104: finishing rollback tid 1 to smithi08575397-8 2021-02-10T04:40:09.562 INFO:tasks.rados.rados.0.smithi085.stdout:3104: finishing rollback tid 2 to smithi08575397-8 2021-02-10T04:40:09.562 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 1553 (ObjNum 936 snap 310 seq_num 936) dirty exists 2021-02-10T04:40:09.563 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:40:09.569 INFO:tasks.rados.rados.0.smithi085.stdout:3105: expect (ObjNum 1053 snap 346 seq_num 1053) 2021-02-10T04:40:09.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:09 smithi164 conmon[29288]: audit 2021-02-10T04:40:08.377278+0000 mgr.y (mgr.14140) 775 : audit 2021-02-10T04:40:09.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:09 smithi164 conmon[29288]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:40:09.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:09 smithi164 conmon[29288]: audit 2021-02-10T04:40:09.172022+0000 mon.a (mon.0) 1722 : audit [DBG] from='client.? 172.21.15.85:0/2226632181' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:40:09.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:09 smithi164 conmon[29288]: cluster 2021-02-10T04:40:09.273739+0000 mon.a (mon 2021-02-10T04:40:09.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:09 smithi164 conmon[29288]: .0) 1723 : cluster [DBG] osdmap e790: 8 total, 8 up, 7 in 2021-02-10T04:40:09.844 INFO:tasks.rados.rados.0.smithi085.stdout:3103: finishing write tid 2 to smithi08575397-13 2021-02-10T04:40:09.845 INFO:tasks.rados.rados.0.smithi085.stdout:3103: finishing write tid 3 to smithi08575397-13 2021-02-10T04:40:09.845 INFO:tasks.rados.rados.0.smithi085.stdout:3103: finishing write tid 4 to smithi08575397-13 2021-02-10T04:40:09.845 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 830 (ObjNum 1071 snap 355 seq_num 1071) dirty exists 2021-02-10T04:40:09.845 INFO:tasks.rados.rados.0.smithi085.stdout:3103: left oid 13 (ObjNum 1071 snap 355 seq_num 1071) 2021-02-10T04:40:09.933 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:40:09.941 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:40:09 smithi085 conmon[41540]: debug 2021-02-10T04:40:09.641+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:40:10.369 INFO:tasks.rados.rados.0.smithi085.stdout:3103: done (7 left) 2021-02-10T04:40:10.369 INFO:tasks.rados.rados.0.smithi085.stdout:3104: done (6 left) 2021-02-10T04:40:10.369 INFO:tasks.rados.rados.0.smithi085.stdout:3105: done (5 left) 2021-02-10T04:40:10.370 INFO:tasks.rados.rados.0.smithi085.stdout:3106: done (4 left) 2021-02-10T04:40:10.370 INFO:tasks.rados.rados.0.smithi085.stdout:3107: rollback oid 43 current snap is 356 2021-02-10T04:40:10.370 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 43 to 278 2021-02-10T04:40:10.375 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:10 smithi164 conmon[33677]: debug 2021-02-10T04:40:10.375 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:10 smithi164 conmon[33677]: 2021-02-10T04:40:10.090+0000 7fbec19a1700 -1 osd.4 790 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:10.375 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:10 smithi164 conmon[29288]: audit 2021-02-10T04:40:09.505404+0000 mon.b (mon.2) 40 : audit [INF] 2021-02-10T04:40:10.375 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:10 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/3399307948' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-02-10T04:40:10.375 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:10 smithi164 conmon[29288]: audit 2021-02-10T04:40:09.505868+0000 mon.a (mon.0) 1724 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-02-10T04:40:10.375 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:10 smithi164 conmon[29288]: audit 2021-02-10T04:40:09.518511+0000 mon.b (mon.2) 41 : audit [INF] from='client.? 172.21.15.85:0/4286754252' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:40:10.375 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:10 smithi164 conmon[29288]: audit 2021-02-10T04:40:09.519129+0000 mon.a (mon.0) 1725 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:40:10.375 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:10 smithi164 conmon[29288]: cluster 2021-02-10T04:40:10. 2021-02-10T04:40:10.376 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:10 smithi164 conmon[29288]: 091662+0000 osd.4 (osd.4) 1255 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:10.376 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:10 smithi164 conmon[29288]: 2021-02-10T04:40:10.376 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:10 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:40:10.376 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:10 smithi164 conmon[29288]: 02-10T04:40:10.091672+0000 osd.4 (osd.4) 1256 : 2021-02-10T04:40:10.376 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:10 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:10.376 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:10 smithi164 conmon[29288]: cluster 2021-02-10T04:40:10.376 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:10 smithi164 conmon[29288]: 2021-02-10T04:40:10.091678+0000 osd.4 (osd.4) 1257 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:10.376 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:10 smithi164 conmon[29288]: cluster 2021-02-10T04:40 2021-02-10T04:40:10.376 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:10 smithi164 conmon[29288]: :10.091684+0000 osd.4 (osd.4) 1258 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:10.376 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:10 smithi164 conmon[29288]: cluster 2021-02-10T04:40: 2021-02-10T04:40:10.376 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:10 smithi164 conmon[29288]: 10.188698+0000 mgr.y (mgr.14140) 776 : cluster [DBG] pgmap v1336: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 494 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 916 KiB/s wr, 13 op/s 2021-02-10T04:40:10.379 INFO:tasks.rados.rados.0.smithi085.stdout:3108: write (excl) oid 12 current snap is 356 2021-02-10T04:40:10.379 INFO:tasks.rados.rados.0.smithi085.stdout:3108: seq_num 1072 ranges {417104=672701,1813991=203113} 2021-02-10T04:40:10.379 INFO:tasks.rados.rados.0.smithi085.stdout:3108: writing smithi08575397-12 from 417104 to 1089805 tid 1 2021-02-10T04:40:10.379 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is set 2021-02-10T04:40:10.382 INFO:tasks.rados.rados.0.smithi085.stdout:3108: writing smithi08575397-12 from 1813991 to 2017104 tid 2 2021-02-10T04:40:10.382 INFO:tasks.rados.rados.0.smithi085.stdout:3109: rollback oid 49 current snap is 356 2021-02-10T04:40:10.383 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 49 to 308 2021-02-10T04:40:10.383 INFO:tasks.rados.rados.0.smithi085.stdout:3108: finishing write tid 1 to smithi08575397-12 2021-02-10T04:40:10.383 INFO:tasks.rados.rados.0.smithi085.stdout:3110: write oid 8 current snap is 356 2021-02-10T04:40:10.383 INFO:tasks.rados.rados.0.smithi085.stdout:3110: seq_num 1073 ranges {433911=747950,1756120=277791} 2021-02-10T04:40:10.387 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.1 to 1 (10000) 2021-02-10T04:40:10.390 INFO:tasks.rados.rados.0.smithi085.stdout:3110: writing smithi08575397-8 from 433911 to 1181861 tid 1 2021-02-10T04:40:10.396 INFO:tasks.rados.rados.0.smithi085.stdout:3110: writing smithi08575397-8 from 1756120 to 2033911 tid 2 2021-02-10T04:40:10.396 INFO:tasks.rados.rados.0.smithi085.stdout:3111: snap_create 2021-02-10T04:40:10.396 INFO:tasks.rados.rados.0.smithi085.stdout:3107: finishing rollback tid 0 to smithi08575397-43 2021-02-10T04:40:10.396 INFO:tasks.rados.rados.0.smithi085.stdout:3108: finishing write tid 2 to smithi08575397-12 2021-02-10T04:40:10.396 INFO:tasks.rados.rados.0.smithi085.stdout:3108: finishing write tid 3 to smithi08575397-12 2021-02-10T04:40:10.396 INFO:tasks.rados.rados.0.smithi085.stdout:3107: finishing rollback tid 1 to smithi08575397-43 2021-02-10T04:40:10.396 INFO:tasks.rados.rados.0.smithi085.stdout:3107: finishing rollback tid 2 to smithi08575397-43 2021-02-10T04:40:10.396 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 1463 (ObjNum 748 snap 240 seq_num 748) dirty exists 2021-02-10T04:40:10.397 INFO:tasks.rados.rados.0.smithi085.stdout:3108: finishing write tid 4 to smithi08575397-12 2021-02-10T04:40:10.397 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 1464 (ObjNum 1072 snap 356 seq_num 1072) dirty exists 2021-02-10T04:40:10.397 INFO:tasks.rados.rados.0.smithi085.stdout:3108: left oid 12 (ObjNum 1072 snap 356 seq_num 1072) 2021-02-10T04:40:10.397 INFO:tasks.rados.rados.0.smithi085.stdout:3109: finishing rollback tid 0 to smithi08575397-49 2021-02-10T04:40:10.397 INFO:tasks.rados.rados.0.smithi085.stdout:3109: finishing rollback tid 1 to smithi08575397-49 2021-02-10T04:40:10.397 INFO:tasks.rados.rados.0.smithi085.stdout:3109: finishing rollback tid 2 to smithi08575397-49 2021-02-10T04:40:10.397 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 49 v 1470 (ObjNum 172 snap 37 seq_num 172) dirty exists 2021-02-10T04:40:10.398 INFO:tasks.rados.rados.0.smithi085.stdout:3110: finishing write tid 1 to smithi08575397-8 2021-02-10T04:40:10.398 INFO:tasks.rados.rados.0.smithi085.stdout:3110: finishing write tid 2 to smithi08575397-8 2021-02-10T04:40:10.399 INFO:tasks.rados.rados.0.smithi085.stdout:3110: finishing write tid 3 to smithi08575397-8 2021-02-10T04:40:10.399 INFO:tasks.rados.rados.0.smithi085.stdout:3110: finishing write tid 4 to smithi08575397-8 2021-02-10T04:40:10.399 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 1556 (ObjNum 1073 snap 356 seq_num 1073) dirty exists 2021-02-10T04:40:10.399 INFO:tasks.rados.rados.0.smithi085.stdout:3110: left oid 8 (ObjNum 1073 snap 356 seq_num 1073) 2021-02-10T04:40:10.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: audit 2021-02-10T04:40:09.505404+0000 mon.b ( 2021-02-10T04:40:10.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: mon.2) 40 : audit [INF] from='client.? 172.21.15.85:0/3399307948' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-02-10T04:40:10.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: audit 2021-02-10T04:40:09.505868+0000 mon.a (mon.0) 2021-02-10T04:40:10.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: 1724 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-02-10T04:40:10.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: audit 2021-02-10T04:40 2021-02-10T04:40:10.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: :09.518511+0000 mon.b (mon.2) 41 2021-02-10T04:40:10.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: : audit [INF] from='client.? 172.21.15.85:0/4286754252' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:40:10.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: audit 2021-02-10T04:40:10.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: 2021-02-10T04:40:09 2021-02-10T04:40:10.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: .519129+0000 mon.a (mon.0) 1725 : 2021-02-10T04:40:10.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:40:10.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: cluster 2021-02-10T04:40:10.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: 2021-02-10T04:40:10 2021-02-10T04:40:10.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: .091662+0000 osd.4 (osd.4) 1255 : 2021-02-10T04:40:10.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:10.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: cluster 2021-02-10T04:40:10.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: 2021-02-10T04:40:10. 2021-02-10T04:40:10.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: 091672+0000 osd.4 (osd.4) 1256 : cluster 2021-02-10T04:40:10.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:10.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:40:10.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: -02-10T04:40:10.091678 2021-02-10T04:40:10.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1257 : cluster 2021-02-10T04:40:10.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:10.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:40:10.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: -10T04:40:10.091684+0000 2021-02-10T04:40:10.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: osd.4 (osd.4) 1258 : cluster [WRN] 2021-02-10T04:40:10.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:10.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: cluster 2021-02-10T04:40:10.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:40:10.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: :40:10.188698+0000 mgr.y (mgr. 2021-02-10T04:40:10.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[29319]: 14140) 776 : cluster [DBG] pgmap v1336: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 494 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 916 KiB/s wr, 13 op/s 2021-02-10T04:40:10.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: audit 2021-02-10T04:40:10.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021-02-10T04:40:10.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021 2021-02-10T04:40:10.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: - 2021-02-10T04:40:10.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 02 2021-02-10T04:40:10.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: - 2021-02-10T04:40:10.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 10 2021-02-10T04:40:10.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: T 2021-02-10T04:40:10.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 04 2021-02-10T04:40:10.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: : 2021-02-10T04:40:10.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 40 2021-02-10T04:40:10.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: : 2021-02-10T04:40:10.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 09 2021-02-10T04:40:10.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: . 2021-02-10T04:40:10.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 505404 2021-02-10T04:40:10.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: +0000 2021-02-10T04:40:10.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021-02-10T04:40:10.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: mon.b 2021-02-10T04:40:10.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: ( 2021-02-10T04:40:10.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: mon 2021-02-10T04:40:10.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: . 2021-02-10T04:40:10.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2 2021-02-10T04:40:10.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: ) 2021-02-10T04:40:10.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 40 2021-02-10T04:40:10.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: : 2021-02-10T04:40:10.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: audit 2021-02-10T04:40:10.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021-02-10T04:40:10.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: [INF] 2021-02-10T04:40:10.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021-02-10T04:40:10.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/3399307948' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-02-10T04:40:10.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021-02-10T04:40:10.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: audit 2021-02-10T04:40:10.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021-02-10T04:40:10.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021 2021-02-10T04:40:10.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: - 2021-02-10T04:40:10.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 02 2021-02-10T04:40:10.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: - 2021-02-10T04:40:10.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 10 2021-02-10T04:40:10.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: T 2021-02-10T04:40:10.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 04 2021-02-10T04:40:10.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: : 2021-02-10T04:40:10.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 40 2021-02-10T04:40:10.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: : 2021-02-10T04:40:10.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 09 2021-02-10T04:40:10.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: . 2021-02-10T04:40:10.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 505868 2021-02-10T04:40:10.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: +0000 2021-02-10T04:40:10.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021-02-10T04:40:10.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: mon.a 2021-02-10T04:40:10.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: ( 2021-02-10T04:40:10.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: mon 2021-02-10T04:40:10.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: . 2021-02-10T04:40:10.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 0 2021-02-10T04:40:10.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: ) 2021-02-10T04:40:10.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 1724 2021-02-10T04:40:10.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: : 2021-02-10T04:40:10.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: audit 2021-02-10T04:40:10.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021-02-10T04:40:10.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: [INF] 2021-02-10T04:40:10.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021-02-10T04:40:10.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-02-10T04:40:10.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021-02-10T04:40:10.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: audit 2021-02-10T04:40:10.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021-02-10T04:40:10.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021 2021-02-10T04:40:10.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: - 2021-02-10T04:40:10.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 02 2021-02-10T04:40:10.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: - 2021-02-10T04:40:10.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 10 2021-02-10T04:40:10.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: T 2021-02-10T04:40:10.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 04 2021-02-10T04:40:10.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: : 2021-02-10T04:40:10.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 40 2021-02-10T04:40:10.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: : 2021-02-10T04:40:10.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 09 2021-02-10T04:40:10.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: . 2021-02-10T04:40:10.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 518511 2021-02-10T04:40:10.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: +0000 2021-02-10T04:40:10.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021-02-10T04:40:10.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: mon.b 2021-02-10T04:40:10.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: ( 2021-02-10T04:40:10.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: mon 2021-02-10T04:40:10.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: . 2021-02-10T04:40:10.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2 2021-02-10T04:40:10.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: ) 2021-02-10T04:40:10.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 41 2021-02-10T04:40:10.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: : 2021-02-10T04:40:10.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: audit 2021-02-10T04:40:10.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021-02-10T04:40:10.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: [INF] 2021-02-10T04:40:10.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021-02-10T04:40:10.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/4286754252' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:40:10.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021-02-10T04:40:10.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: audit 2021-02-10T04:40:10.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021-02-10T04:40:10.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021 2021-02-10T04:40:10.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: - 2021-02-10T04:40:10.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 02 2021-02-10T04:40:10.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: - 2021-02-10T04:40:10.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 10 2021-02-10T04:40:10.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: T 2021-02-10T04:40:10.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 04 2021-02-10T04:40:10.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: : 2021-02-10T04:40:10.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 40 2021-02-10T04:40:10.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: : 2021-02-10T04:40:10.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 09 2021-02-10T04:40:10.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: . 2021-02-10T04:40:10.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 519129 2021-02-10T04:40:10.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: +0000 2021-02-10T04:40:10.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021-02-10T04:40:10.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: mon.a 2021-02-10T04:40:10.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: ( 2021-02-10T04:40:10.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: mon 2021-02-10T04:40:10.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: . 2021-02-10T04:40:10.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 0 2021-02-10T04:40:10.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: ) 2021-02-10T04:40:10.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 1725 2021-02-10T04:40:10.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: : 2021-02-10T04:40:10.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: audit 2021-02-10T04:40:10.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021-02-10T04:40:10.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: [INF] 2021-02-10T04:40:10.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021-02-10T04:40:10.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:40:10.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021-02-10T04:40:10.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: cluster 2021-02-10T04:40:10.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021-02-10T04:40:10.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021 2021-02-10T04:40:10.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: - 2021-02-10T04:40:10.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 02 2021-02-10T04:40:10.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: - 2021-02-10T04:40:10.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 10 2021-02-10T04:40:10.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: T 2021-02-10T04:40:10.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 04 2021-02-10T04:40:10.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: : 2021-02-10T04:40:10.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 40 2021-02-10T04:40:10.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: : 2021-02-10T04:40:10.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 10 2021-02-10T04:40:10.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: . 2021-02-10T04:40:10.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 091662 2021-02-10T04:40:10.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: +0000 2021-02-10T04:40:10.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021-02-10T04:40:10.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: osd.4 2021-02-10T04:40:10.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: ( 2021-02-10T04:40:10.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: osd 2021-02-10T04:40:10.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: . 2021-02-10T04:40:10.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 4 2021-02-10T04:40:10.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: ) 2021-02-10T04:40:10.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 1255 2021-02-10T04:40:10.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: : 2021-02-10T04:40:10.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: cluster 2021-02-10T04:40:10.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021-02-10T04:40:10.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: [WRN] 2021-02-10T04:40:10.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021-02-10T04:40:10.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:10.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021-02-10T04:40:10.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: cluster 2021-02-10T04:40:10.707 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021-02-10T04:40:10.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021 2021-02-10T04:40:10.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: - 2021-02-10T04:40:10.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 02 2021-02-10T04:40:10.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: - 2021-02-10T04:40:10.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 10 2021-02-10T04:40:10.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: T 2021-02-10T04:40:10.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 04 2021-02-10T04:40:10.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: : 2021-02-10T04:40:10.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 40:10.091672+0000 osd.4 (osd. 2021-02-10T04:40:10.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 4) 1256 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:10.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: cluster 2021-02-10T04:40:10.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: 2021-02-10T04:40:10.091678+0000 osd.4 (osd 2021-02-10T04:40:10.708 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: .4) 1257 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:10.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:40:10.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: -10T04:40:10.091684+0000 osd.4 (osd.4) 1258 : 2021-02-10T04:40:10.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:10.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:40:10.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: -02-10T04:40:10 2021-02-10T04:40:10.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: .188698+0000 mgr.y (mgr.14140 2021-02-10T04:40:10.709 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:10 smithi085 conmon[35961]: ) 776 : cluster [DBG] pgmap v1336: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 494 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 916 KiB/s wr, 13 op/s 2021-02-10T04:40:11.389 INFO:tasks.rados.rados.0.smithi085.stdout:3107: done (8 left) 2021-02-10T04:40:11.389 INFO:tasks.rados.rados.0.smithi085.stdout:3108: done (7 left) 2021-02-10T04:40:11.389 INFO:tasks.rados.rados.0.smithi085.stdout:3109: done (6 left) 2021-02-10T04:40:11.390 INFO:tasks.rados.rados.0.smithi085.stdout:3110: done (5 left) 2021-02-10T04:40:11.390 INFO:tasks.rados.rados.0.smithi085.stdout:3111: done (4 left) 2021-02-10T04:40:11.390 INFO:tasks.rados.rados.0.smithi085.stdout:3112: snap_remove snap 328 2021-02-10T04:40:11.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:11 smithi164 conmon[33677]: debug 2021-02-10T04:40:11.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:11 smithi164 conmon[33677]: 2021-02-10T04:40:11.128+0000 7fbec19a1700 -1 osd.4 791 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:11.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:11 smithi164 conmon[33677]: 2021-02-10T04:40:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:11 smithi164 conmon[29288]: cluster 2021-02-10T04:40:10.363097+0000 mon.a 2021-02-10T04:40:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:11 smithi164 conmon[29288]: (mon.0) 1726 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:40:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:11 smithi164 conmon[29288]: audit 2021-02-10T04:40:10.367999+0000 mon.a (mon.0) 1727 : audit 2021-02-10T04:40:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:11 smithi164 conmon[29288]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-02-10T04:40:11.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:11 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:40:11.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:11 smithi164 conmon[29288]: 04:40:10.368155+0000 mon.a (mon.0) 1728 : 2021-02-10T04:40:11.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:11 smithi164 conmon[29288]: audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:40:11.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:11 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:40:11.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:11 smithi164 conmon[29288]: 04:40:10.368343+0000 mon.a (mon 2021-02-10T04:40:11.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:11 smithi164 conmon[29288]: .0) 1729 : cluster [DBG] osdmap e791: 8 total, 8 up, 7 in 2021-02-10T04:40:11.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:11 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:40:11.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:11 smithi164 conmon[29288]: 02-10T04:40:10.437389+0000 osd.2 2021-02-10T04:40:11.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:11 smithi164 conmon[29288]: (osd.2) 1937 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:40:11.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:11 smithi164 conmon[29288]: cluster 2021-02-10T04:40 2021-02-10T04:40:11.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:11 smithi164 conmon[29288]: :10.437951+0000 osd.2 (osd.2) 1938 2021-02-10T04:40:11.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:11 smithi164 conmon[29288]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:40:11.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:11 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:40:11.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:11 smithi164 conmon[29288]: 02-10T04:40:11.129099+0000 osd.4 (osd.4) 1259 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:11.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:11 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:40:11.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:11 smithi164 conmon[29288]: 10T04:40:11.129108+0000 osd.4 2021-02-10T04:40:11.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:11 smithi164 conmon[29288]: (osd.4) 1260 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:11.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:11 smithi164 conmon[29288]: 2021-02-10T04:40:11.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:11 smithi164 conmon[29288]: cluster 2021-02-10T04:40:11. 2021-02-10T04:40:11.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:11 smithi164 conmon[29288]: 129114+0000 osd.4 (osd.4) 1261 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:11.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:11 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:40:11.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:11 smithi164 conmon[29288]: -10T04:40:11.129120+0000 osd.4 ( 2021-02-10T04:40:11.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:11 smithi164 conmon[29288]: osd.4) 1262 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:11.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[35961]: cluster 2021-02-10T04:40:10.363097 2021-02-10T04:40:11.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1726 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:40:11.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[35961]: audit 2021-02-10T04:40:10 2021-02-10T04:40:11.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[35961]: .367999+0000 mon.a (mon.0) 1727 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-02-10T04:40:11.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:40:11.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[35961]: :40:10.368155+0000 mon.a (mon.0 2021-02-10T04:40:11.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[35961]: ) 1728 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:40:11.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:40:11.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[35961]: -10T04:40:10.368343+0000 mon.a ( 2021-02-10T04:40:11.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[35961]: mon.0) 1729 : cluster [DBG] osdmap e791: 8 total, 8 up, 7 in 2021-02-10T04:40:11.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[35961]: cluster 2021-02-10T04:40:11.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[35961]: 2021-02-10T04:40:10.437389 2021-02-10T04:40:11.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1937 : cluster [DBG] 2021-02-10T04:40:11.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[35961]: purged_snaps scrub starts 2021-02-10T04:40:11.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:40:11.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[35961]: 40:10.437951+0000 osd.2 (osd.2) 2021-02-10T04:40:11.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[35961]: 1938 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:40:11.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:40:11.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[35961]: -02-10T04:40:11.129099+0000 2021-02-10T04:40:11.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[35961]: osd.4 (osd.4) 1259 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:11.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[35961]: 2021-02-10T04:40:11.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[35961]: cluster 2021-02-10T04:40:11.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[35961]: 2021-02-10T04:40:11.129108+0000 osd.4 (osd.4) 1260 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:11.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[35961]: 2021-02-10T04:40:11.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[35961]: cluster 2021-02-10T04:40:11.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[35961]: 2021-02-10T04:40:11.129114+0000 osd.4 2021-02-10T04:40:11.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[35961]: (osd.4) 1261 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:11.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:40:11.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[35961]: 02-10T04:40:11.129120+0000 osd.4 2021-02-10T04:40:11.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[35961]: (osd.4) 1262 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:11.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: cluster 2021-02-10T04:40:10.363097 2021-02-10T04:40:11.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1726 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:40:11.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: audit 2021-02-10T04:40:10.367999+0000 mon.a (mon.0) 1727 : 2021-02-10T04:40:11.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-02-10T04:40:11.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: audit 2021-02-10T04:40:11.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: 2021-02-10T04:40:10.368155+0000 mon.a (mon.0 2021-02-10T04:40:11.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: ) 1728 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:40:11.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:40:11.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: 10T04:40:10.368343+0000 mon.a ( 2021-02-10T04:40:11.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: mon.0) 1729 : cluster [DBG] osdmap e791: 8 total, 8 up, 7 in 2021-02-10T04:40:11.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:40:11.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: -10T04:40:10.437389+0000 osd.2 2021-02-10T04:40:11.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: (osd.2) 1937 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:40:11.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:40:11.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: 02-10T04:40:10.437951+0000 osd.2 ( 2021-02-10T04:40:11.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: osd.2) 1938 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:40:11.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:40:11.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: 02-10T04:40:11.129099+0000 osd.4 2021-02-10T04:40:11.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: (osd.4) 1259 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:11.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:40:11.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: -02-10T04:40: 2021-02-10T04:40:11.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: 11.129108+0000 osd.4 (osd 2021-02-10T04:40:11.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: .4) 1260 : cluster [WRN] 2021-02-10T04:40:11.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:11.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:40:11.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: 02-10T04:40: 2021-02-10T04:40:11.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: 11.129114+0000 osd.4 2021-02-10T04:40:11.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: (osd.4) 1261 : cluster [WRN] 2021-02-10T04:40:11.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:11.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: cluster 2021-02-10T04:40 2021-02-10T04:40:11.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:11 smithi085 conmon[29319]: :11.129120+0000 osd.4 (osd.4) 1262 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:12.392 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:12 smithi164 conmon[33677]: debug 2021-02-10T04:40:12.130+0000 7fbec19a1700 -1 osd.4 792 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:13.398 INFO:tasks.rados.rados.0.smithi085.stdout:3112: done (4 left) 2021-02-10T04:40:13.398 INFO:tasks.rados.rados.0.smithi085.stdout:3113: write (excl) oid 43 current snap is 357 2021-02-10T04:40:13.398 INFO:tasks.rados.rados.0.smithi085.stdout:3113: seq_num 1074 ranges {450718=423199,1414602=636116} 2021-02-10T04:40:13.399 INFO:tasks.rados.rados.0.smithi085.stdout:3113: writing smithi08575397-43 from 450718 to 873917 tid 1 2021-02-10T04:40:13.399 INFO:tasks.rados.rados.0.smithi085.stdout:3113: writing smithi08575397-43 from 1414602 to 2050718 tid 2 2021-02-10T04:40:13.399 INFO:tasks.rados.rados.0.smithi085.stdout:3114: rollback oid 36 current snap is 357 2021-02-10T04:40:13.399 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 36 to 340 2021-02-10T04:40:13.399 INFO:tasks.rados.rados.0.smithi085.stdout:3115: snap_remove snap 335 2021-02-10T04:40:13.399 INFO:tasks.rados.rados.0.smithi085.stdout:3115: done (6 left) 2021-02-10T04:40:13.399 INFO:tasks.rados.rados.0.smithi085.stdout:3116: copy_from oid 28 from oid 30 current snap is 357 2021-02-10T04:40:13.399 INFO:tasks.rados.rados.0.smithi085.stdout:3113: finishing write tid 1 to smithi08575397-43 2021-02-10T04:40:13.399 INFO:tasks.rados.rados.0.smithi085.stdout:3113: finishing write tid 2 to smithi08575397-43 2021-02-10T04:40:13.399 INFO:tasks.rados.rados.0.smithi085.stdout:3113: finishing write tid 3 to smithi08575397-43 2021-02-10T04:40:13.400 INFO:tasks.rados.rados.0.smithi085.stdout:3113: finishing write tid 4 to smithi08575397-43 2021-02-10T04:40:13.400 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 1466 (ObjNum 1074 snap 357 seq_num 1074) dirty exists 2021-02-10T04:40:13.400 INFO:tasks.rados.rados.0.smithi085.stdout:3113: left oid 43 (ObjNum 1074 snap 357 seq_num 1074) 2021-02-10T04:40:13.400 INFO:tasks.rados.rados.0.smithi085.stdout:3114: finishing rollback tid 0 to smithi08575397-36 2021-02-10T04:40:13.400 INFO:tasks.rados.rados.0.smithi085.stdout:3114: finishing rollback tid 1 to smithi08575397-36 2021-02-10T04:40:13.400 INFO:tasks.rados.rados.0.smithi085.stdout:3114: finishing rollback tid 2 to smithi08575397-36 2021-02-10T04:40:13.400 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 1298 (ObjNum 991 snap 326 seq_num 991) dirty exists 2021-02-10T04:40:13.400 INFO:tasks.rados.rados.0.smithi085.stdout:3113: done (6 left) 2021-02-10T04:40:13.400 INFO:tasks.rados.rados.0.smithi085.stdout:3114: done (5 left) 2021-02-10T04:40:13.400 INFO:tasks.rados.rados.0.smithi085.stdout:3117: write (excl) oid 16 current snap is 357 2021-02-10T04:40:13.400 INFO:tasks.rados.rados.0.smithi085.stdout:3117: seq_num 1075 ranges {467525=498448,1473084=594379} 2021-02-10T04:40:13.400 INFO:tasks.rados.rados.0.smithi085.stdout:3117: writing smithi08575397-16 from 467525 to 965973 tid 1 2021-02-10T04:40:13.400 INFO:tasks.rados.rados.0.smithi085.stdout:3117: writing smithi08575397-16 from 1473084 to 2067463 tid 2 2021-02-10T04:40:13.400 INFO:tasks.rados.rados.0.smithi085.stdout:3118: delete oid 37 current snap is 357 2021-02-10T04:40:13.401 INFO:tasks.rados.rados.0.smithi085.stdout:3116: finishing copy_from to smithi08575397-28 2021-02-10T04:40:13.401 INFO:tasks.rados.rados.0.smithi085.stdout:3116: got expected ENOENT (src dne) 2021-02-10T04:40:13.401 INFO:tasks.rados.rados.0.smithi085.stdout:3116: finishing copy_from racing read to smithi08575397-28 2021-02-10T04:40:13.401 INFO:tasks.rados.rados.0.smithi085.stdout:3117: finishing write tid 1 to smithi08575397-16 2021-02-10T04:40:13.401 INFO:tasks.rados.rados.0.smithi085.stdout:3117: finishing write tid 2 to smithi08575397-16 2021-02-10T04:40:13.401 INFO:tasks.rados.rados.0.smithi085.stdout:3117: finishing write tid 3 to smithi08575397-16 2021-02-10T04:40:13.401 INFO:tasks.rados.rados.0.smithi085.stdout:3117: finishing write tid 4 to smithi08575397-16 2021-02-10T04:40:13.401 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 1617 (ObjNum 1075 snap 357 seq_num 1075) dirty exists 2021-02-10T04:40:13.401 INFO:tasks.rados.rados.0.smithi085.stdout:3117: left oid 16 (ObjNum 1075 snap 357 seq_num 1075) 2021-02-10T04:40:13.401 INFO:tasks.rados.rados.0.smithi085.stdout:3116: done (6 left) 2021-02-10T04:40:13.401 INFO:tasks.rados.rados.0.smithi085.stdout:3117: done (5 left) 2021-02-10T04:40:13.401 INFO:tasks.rados.rados.0.smithi085.stdout:3118: done (4 left) 2021-02-10T04:40:13.401 INFO:tasks.rados.rados.0.smithi085.stdout:3119: snap_create 2021-02-10T04:40:13.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[29319]: cluster 2021-02-10T04:40:13.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[29319]: 2021-02-10T04:40:13.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[29319]: 2021 2021-02-10T04:40:13.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[29319]: - 2021-02-10T04:40:13.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[29319]: 02 2021-02-10T04:40:13.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[29319]: - 2021-02-10T04:40:13.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[29319]: 10T04:40:11. 2021-02-10T04:40:13.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[29319]: 387651+0000 mon.a (mon.0) 1730 : cluster [DBG] osdmap e792: 8 total, 8 up, 7 in 2021-02-10T04:40:13.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:40:13.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[29319]: 10T04:40:12.130984+0000 osd.4 (osd.4) 1263 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:13.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[29319]: cluster 2021-02-10T04:40:12.130993+0000 osd.4 (osd.4) 2021-02-10T04:40:13.402 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[29319]: 1264 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:13.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:40:13.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[29319]: 40:12.130999+0000 osd.4 (osd.4) 1265 2021-02-10T04:40:13.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:13.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:40:13.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[29319]: 10T04:40:12.131007+0000 osd.4 (osd.4 2021-02-10T04:40:13.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[29319]: ) 1266 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:13.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[29319]: cluster 2021-02-10T04:40:13.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[29319]: 2021-02-10T04:40:12 2021-02-10T04:40:13.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[29319]: .189289+0000 mgr.y (mgr.14140) 777 : 2021-02-10T04:40:13.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[29319]: cluster [DBG] pgmap v1339: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 494 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 8.9 MiB/s rd, 915 KiB/s wr, 11 op/s 2021-02-10T04:40:13.403 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[35961]: cluster 2021-02-10T04:40:13.404 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:40:13.404 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[35961]: 40:11.387651+0000 mon.a (mon.0) 1730 : cluster [DBG] osdmap e792: 8 total, 8 up, 7 in 2021-02-10T04:40:13.404 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:40:13.404 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[35961]: -02-10T04:40:12.130984+0000 2021-02-10T04:40:13.404 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[35961]: osd.4 (osd.4) 1263 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:13.404 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:40:13.404 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[35961]: -10T04:40:12.130993+0000 osd.4 (osd. 2021-02-10T04:40:13.404 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[35961]: 4) 1264 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:13.404 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:40:13.405 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[35961]: -02-10T04:40:12.130999+0000 2021-02-10T04:40:13.405 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[35961]: osd.4 (osd.4) 1265 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:13.405 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[35961]: 2021-02-10T04:40:13.405 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[35961]: cluster 2021-02-10T04:40 2021-02-10T04:40:13.405 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[35961]: :12.131007+0000 osd.4 (osd.4) 1266 : 2021-02-10T04:40:13.405 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:13.405 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[35961]: cluster 2021-02-10T04:40:13.405 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[35961]: 2021-02-10T04:40:12.189289+0000 mgr.y (mgr.14140) 777 2021-02-10T04:40:13.405 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:12 smithi085 conmon[35961]: : cluster [DBG] pgmap v1339: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 494 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 8.9 MiB/s rd, 915 KiB/s wr, 11 op/s 2021-02-10T04:40:13.405 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:12 smithi164 conmon[29288]: cluster 2021-02-10T04:40:11.387651+0000 2021-02-10T04:40:13.405 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:12 smithi164 conmon[29288]: mon.a (mon.0) 1730 : cluster [DBG] osdmap e792: 8 total, 8 up, 7 in 2021-02-10T04:40:13.406 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:12 smithi164 conmon[29288]: cluster 2021-02-10T04:40:12.130984+0000 osd.4 (osd.4) 1263 : 2021-02-10T04:40:13.406 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:12 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:13.406 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:12 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:40:13.406 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:12 smithi164 conmon[29288]: T04:40:12.130993+0000 2021-02-10T04:40:13.406 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:12 smithi164 conmon[29288]: osd.4 (osd.4) 1264 2021-02-10T04:40:13.406 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:12 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:13.406 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:12 smithi164 conmon[29288]: cluster 2021-02-10T04:40:12.130999 2021-02-10T04:40:13.406 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:12 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1265 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:13.406 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:12 smithi164 conmon[29288]: cluster 2021-02-10T04:40:13.406 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:12 smithi164 conmon[29288]: 2021-02-10T04:40:12.131007 2021-02-10T04:40:13.406 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:12 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1266 : cluster [WRN] 2021-02-10T04:40:13.406 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:12 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:13.407 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:12 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:40:13.407 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:12 smithi164 conmon[29288]: :40:12.189289+0000 mgr.y (mgr.14140 2021-02-10T04:40:13.407 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:12 smithi164 conmon[29288]: ) 777 : cluster [DBG] pgmap v1339: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 494 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 8.9 MiB/s rd, 915 KiB/s wr, 11 op/s 2021-02-10T04:40:13.407 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:13 smithi164 conmon[29288]: cluster 2021-02-10T04:40:12.382817+0000 mon.a (mon 2021-02-10T04:40:13.407 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:13 smithi164 conmon[29288]: .0) 1731 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 643 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:40:13.407 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:13 smithi164 conmon[29288]: cluster 2021-02-10T04:40:12.394065+0000 mon.a (mon.0) 1732 : cluster [DBG] osdmap e793: 8 total, 8 up, 7 in 2021-02-10T04:40:13.407 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:13 smithi164 conmon[29288]: 2021-02-10T04:40:13.407 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:13 smithi164 conmon[29288]: cluster 2021-02-10T04:40:13.407 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:13 smithi164 conmon[29288]: 2021 2021-02-10T04:40:13.407 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:13 smithi164 conmon[29288]: - 2021-02-10T04:40:13.407 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:13 smithi164 conmon[29288]: 02-10T04:40:13. 2021-02-10T04:40:13.407 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:13 smithi164 conmon[29288]: 098339+0000 osd.4 (osd.4) 1267 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:13.407 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:13 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:40:13.407 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:13 smithi164 conmon[29288]: -02-10T04:40:13.098349+0000 osd.4 (osd. 2021-02-10T04:40:13.408 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:13 smithi164 conmon[29288]: 4) 1268 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:13.408 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:13 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:40:13.408 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:13 smithi164 conmon[29288]: -10T04:40:13.098355+0000 osd.4 (osd. 2021-02-10T04:40:13.408 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:13 smithi164 conmon[29288]: 4) 1269 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:13.408 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:13 smithi164 conmon[29288]: cluster 2021-02-10T04:40:13.408 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:13 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:40:13.408 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:13 smithi164 conmon[29288]: 40:13.098361+0000 osd.4 (osd.4) 1270 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:13.408 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:13 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:40:13.408 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:13 smithi164 conmon[29288]: 02-10T04:40:13.261789+0000 mon.a (mon 2021-02-10T04:40:13.408 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:13 smithi164 conmon[29288]: .0) 1733 : cluster [DBG] osdmap e794: 8 total, 8 up, 7 in 2021-02-10T04:40:13.408 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:13 smithi164 conmon[33677]: debug 2021-02-10T04:40:13.097+0000 7fbec19a1700 -1 osd.4 793 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:13.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[29319]: cluster 2021-02-10T04:40:12.382817+0000 mon.a (mon.0) 1731 : 2021-02-10T04:40:13.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[29319]: cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 643 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:40:13.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[29319]: cluster 2021-02-10T04:40:12.394065+0000 mon.a (mon.0) 1732 : cluster [DBG] osdmap e793: 8 total, 8 up, 7 in 2021-02-10T04:40:13.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[29319]: cluster 2021-02-10T04:40:13.098339+0000 osd.4 (osd.4) 1267 : 2021-02-10T04:40:13.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:13.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:40:13.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[29319]: T04:40:13.098349+0000 osd.4 ( 2021-02-10T04:40:13.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[29319]: osd.4) 1268 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:13.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[29319]: cluster 2021-02-10T04:40:13.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[29319]: 2021-02-10T04:40:13. 2021-02-10T04:40:13.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[29319]: 098355+0000 osd.4 (osd.4) 1269 : 2021-02-10T04:40:13.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:13.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:40:13.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[29319]: 10T04:40:13.098361+0000 osd.4 2021-02-10T04:40:13.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[29319]: (osd.4) 1270 : cluster 2021-02-10T04:40:13.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:13.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:40:13.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[29319]: -02-10T04:40:13.261789+0000 mon.a 2021-02-10T04:40:13.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[29319]: (mon.0) 1733 : cluster [DBG] osdmap e794: 8 total, 8 up, 7 in 2021-02-10T04:40:13.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[35961]: cluster 2021-02-10T04:40:13.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[35961]: 2021-02-10T04:40: 2021-02-10T04:40:13.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[35961]: 12.382817+0000 mon.a (mon.0) 1731 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 643 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:40:13.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:40:13.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[35961]: 04:40:12.394065+0000 mon.a (mon. 2021-02-10T04:40:13.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[35961]: 0) 1732 : cluster [DBG] osdmap e793: 8 total, 8 up, 7 in 2021-02-10T04:40:13.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:40:13.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[35961]: -02-10T04:40:13.098339+0000 osd.4 ( 2021-02-10T04:40:13.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[35961]: osd.4) 1267 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:13.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:40:13.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[35961]: 40:13.098349+0000 osd.4 (osd.4) 1268 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:13.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[35961]: 2021-02-10T04:40:13.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:40:13.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[35961]: -10T04:40:13.098355+0000 osd.4 (osd.4) 1269 : cluster 2021-02-10T04:40:13.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:13.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[35961]: cluster 2021-02-10T04:40:13 2021-02-10T04:40:13.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[35961]: .098361+0000 osd.4 (osd.4) 1270 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:13.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:40:13.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[35961]: -10T04:40:13.261789+0000 mon.a (mon.0 2021-02-10T04:40:13.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:13 smithi085 conmon[35961]: ) 1733 : cluster [DBG] osdmap e794: 8 total, 8 up, 7 in 2021-02-10T04:40:14.280 INFO:tasks.rados.rados.0.smithi085.stdout:3119: done (4 left) 2021-02-10T04:40:14.280 INFO:tasks.rados.rados.0.smithi085.stdout:3120: snap_create 2021-02-10T04:40:14.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:14 smithi164 conmon[33677]: debug 2021-02-10T04:40:14.120+0000 7fbec19a1700 -1 osd.4 794 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:14.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:14 smithi085 conmon[29319]: cluster 2021-02-10T04:40:14.121191+0000 osd.4 (osd.4) 1271 : 2021-02-10T04:40:14.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:14 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:14.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:14 smithi085 conmon[29319]: cluster 2021-02-10T04:40:14.121201+0000 osd.4 (osd.4) 1272 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:14.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:14 smithi085 conmon[29319]: cluster 2021-02-10T04:40:14.121207+0000 osd.4 ( 2021-02-10T04:40:14.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:14 smithi085 conmon[29319]: osd.4) 1273 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:14.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:14 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:40:14.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:14 smithi085 conmon[29319]: -10T04:40:14.121213+0000 osd.4 ( 2021-02-10T04:40:14.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:14 smithi085 conmon[29319]: osd.4) 1274 : cluster [WRN] 2021-02-10T04:40:14.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:14 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:14.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:14 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:40:14.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:14 smithi085 conmon[29319]: 40:14.189865+0000 mgr.y (mgr 2021-02-10T04:40:14.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:14 smithi085 conmon[29319]: .14140) 778 : cluster [DBG] pgmap v1342: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 504 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 8.3 MiB/s rd, 1.4 MiB/s wr, 13 op/s 2021-02-10T04:40:14.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:14 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:40:14.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:14 smithi085 conmon[29319]: 02-10T04:40:14.276528+0000 mon.a (mon.0) 1734 : cluster [DBG] 2021-02-10T04:40:14.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:14 smithi085 conmon[29319]: osdmap e795: 8 total, 8 up, 7 in 2021-02-10T04:40:14.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:14 smithi085 conmon[35961]: cluster 2021-02-10T04:40:14.121191 2021-02-10T04:40:14.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:14 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1271 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:14.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:14 smithi085 conmon[35961]: cluster 2021-02-10T04:40:14.121201+0000 osd.4 (osd.4 2021-02-10T04:40:14.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:14 smithi085 conmon[35961]: ) 1272 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:14.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:14 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:40:14.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:14 smithi085 conmon[35961]: :40:14.121207+0000 osd.4 (osd.4) 1273 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:14.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:14 smithi085 conmon[35961]: cluster 2021-02-10T04:40:14.121213+0000 2021-02-10T04:40:14.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:14 smithi085 conmon[35961]: osd.4 (osd.4) 1274 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:14.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:14 smithi085 conmon[35961]: cluster 2021-02-10T04:40:14.189865+0000 mgr.y 2021-02-10T04:40:14.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:14 smithi085 conmon[35961]: (mgr.14140) 778 : cluster [DBG] pgmap v1342: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 504 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 8.3 MiB/s rd, 1.4 MiB/s wr, 13 op/s 2021-02-10T04:40:14.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:14 smithi085 conmon[35961]: cluster 2021-02-10T04:40:14.276528+0000 mon.a (mon 2021-02-10T04:40:14.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:14 smithi085 conmon[35961]: .0) 1734 : cluster [DBG] osdmap e795: 8 total, 8 up, 7 in 2021-02-10T04:40:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:14 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:40:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:14 smithi164 conmon[29288]: 02-10T04:40:14.121191+0000 osd.4 (osd.4) 1271 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:14 smithi164 conmon[29288]: cluster 2021-02-10T04:40:14.121201+0000 osd.4 (osd.4) 1272 2021-02-10T04:40:14.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:14 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:14.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:14 smithi164 conmon[29288]: cluster 2021-02-10T04:40:14.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:14 smithi164 conmon[29288]: 2021-02-10T04:40: 2021-02-10T04:40:14.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:14 smithi164 conmon[29288]: 14.121207+0000 osd.4 (osd. 2021-02-10T04:40:14.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:14 smithi164 conmon[29288]: 4) 1273 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:14.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:14 smithi164 conmon[29288]: 2021-02-10T04:40:14.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:14 smithi164 conmon[29288]: cluster 2021-02-10T04:40:14.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:14 smithi164 conmon[29288]: 2021-02-10T04:40:14.121213 2021-02-10T04:40:14.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:14 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1274 : cluster 2021-02-10T04:40:14.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:14 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:14.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:14 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:40:14.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:14 smithi164 conmon[29288]: T04:40:14.189865+0000 mgr.y (mgr 2021-02-10T04:40:14.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:14 smithi164 conmon[29288]: .14140) 778 : cluster [DBG] 2021-02-10T04:40:14.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:14 smithi164 conmon[29288]: pgmap v1342: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 504 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 8.3 MiB/s rd, 1.4 MiB/s wr, 13 op/s 2021-02-10T04:40:14.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:14 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:40:14.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:14 smithi164 conmon[29288]: T04:40:14.276528+0000 mon.a (mon 2021-02-10T04:40:14.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:14 smithi164 conmon[29288]: .0) 1734 : cluster [DBG] 2021-02-10T04:40:14.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:14 smithi164 conmon[29288]: osdmap e795: 8 total, 8 up, 7 in 2021-02-10T04:40:15.278 INFO:tasks.rados.rados.0.smithi085.stdout:3120: done (4 left) 2021-02-10T04:40:15.278 INFO:tasks.rados.rados.0.smithi085.stdout:3121: write oid 1 current snap is 359 2021-02-10T04:40:15.279 INFO:tasks.rados.rados.0.smithi085.stdout:3121: seq_num 1077 ranges {501139=648946,1873695=227444} 2021-02-10T04:40:15.284 INFO:tasks.rados.rados.0.smithi085.stdout:3121: writing smithi08575397-1 from 501139 to 1150085 tid 1 2021-02-10T04:40:15.291 INFO:tasks.rados.rados.0.smithi085.stdout:3121: writing smithi08575397-1 from 1873695 to 2101139 tid 2 2021-02-10T04:40:15.291 INFO:tasks.rados.rados.0.smithi085.stdout:3122: write (excl) oid 40 current snap is 359 2021-02-10T04:40:15.291 INFO:tasks.rados.rados.0.smithi085.stdout:3121: finishing write tid 1 to smithi08575397-1 2021-02-10T04:40:15.291 INFO:tasks.rados.rados.0.smithi085.stdout:3122: seq_num 1078 ranges {517946=724195,1932177=185769} 2021-02-10T04:40:15.298 INFO:tasks.rados.rados.0.smithi085.stdout:3122: writing smithi08575397-40 from 517946 to 1242141 tid 1 2021-02-10T04:40:15.304 INFO:tasks.rados.rados.0.smithi085.stdout:3122: writing smithi08575397-40 from 1932177 to 2117946 tid 2 2021-02-10T04:40:15.304 INFO:tasks.rados.rados.0.smithi085.stdout:3123: copy_from oid 28 from oid 6 current snap is 359 2021-02-10T04:40:15.304 INFO:tasks.rados.rados.0.smithi085.stdout:3121: finishing write tid 2 to smithi08575397-1 2021-02-10T04:40:15.304 INFO:tasks.rados.rados.0.smithi085.stdout:3121: finishing write tid 3 to smithi08575397-1 2021-02-10T04:40:15.304 INFO:tasks.rados.rados.0.smithi085.stdout:3121: finishing write tid 4 to smithi08575397-1 2021-02-10T04:40:15.304 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 1559 (ObjNum 1077 snap 359 seq_num 1077) dirty exists 2021-02-10T04:40:15.305 INFO:tasks.rados.rados.0.smithi085.stdout:3121: left oid 1 (ObjNum 1077 snap 359 seq_num 1077) 2021-02-10T04:40:15.305 INFO:tasks.rados.rados.0.smithi085.stdout:3121: done (6 left) 2021-02-10T04:40:15.305 INFO:tasks.rados.rados.0.smithi085.stdout:3124: delete oid 35 current snap is 359 2021-02-10T04:40:15.305 INFO:tasks.rados.rados.0.smithi085.stdout:3122: finishing write tid 1 to smithi08575397-40 2021-02-10T04:40:15.307 INFO:tasks.rados.rados.0.smithi085.stdout:3124: done (6 left) 2021-02-10T04:40:15.307 INFO:tasks.rados.rados.0.smithi085.stdout:3125: write oid 5 current snap is 359 2021-02-10T04:40:15.308 INFO:tasks.rados.rados.0.smithi085.stdout:3125: seq_num 1080 ranges {551560=591046,1765494=386066} 2021-02-10T04:40:15.315 INFO:tasks.rados.rados.0.smithi085.stdout:3125: writing smithi08575397-5 from 551560 to 1142606 tid 1 2021-02-10T04:40:15.322 INFO:tasks.rados.rados.0.smithi085.stdout:3125: writing smithi08575397-5 from 1765494 to 2151560 tid 2 2021-02-10T04:40:15.322 INFO:tasks.rados.rados.0.smithi085.stdout:3126: copy_from oid 34 from oid 44 current snap is 359 2021-02-10T04:40:15.322 INFO:tasks.rados.rados.0.smithi085.stdout:3122: finishing write tid 2 to smithi08575397-40 2021-02-10T04:40:15.322 INFO:tasks.rados.rados.0.smithi085.stdout:3122: finishing write tid 3 to smithi08575397-40 2021-02-10T04:40:15.322 INFO:tasks.rados.rados.0.smithi085.stdout:3122: finishing write tid 4 to smithi08575397-40 2021-02-10T04:40:15.322 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 1627 (ObjNum 1078 snap 359 seq_num 1078) dirty exists 2021-02-10T04:40:15.322 INFO:tasks.rados.rados.0.smithi085.stdout:3122: left oid 40 (ObjNum 1078 snap 359 seq_num 1078) 2021-02-10T04:40:15.322 INFO:tasks.rados.rados.0.smithi085.stdout:3123: finishing copy_from to smithi08575397-28 2021-02-10T04:40:15.322 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 1380 (ObjNum 1028 snap 335 seq_num 1028) dirty exists 2021-02-10T04:40:15.322 INFO:tasks.rados.rados.0.smithi085.stdout:3123: finishing copy_from racing read to smithi08575397-28 2021-02-10T04:40:15.323 INFO:tasks.rados.rados.0.smithi085.stdout:3125: finishing write tid 1 to smithi08575397-5 2021-02-10T04:40:15.323 INFO:tasks.rados.rados.0.smithi085.stdout:3122: done (7 left) 2021-02-10T04:40:15.323 INFO:tasks.rados.rados.0.smithi085.stdout:3123: done (6 left) 2021-02-10T04:40:15.323 INFO:tasks.rados.rados.0.smithi085.stdout:3127: read oid 7 snap -1 2021-02-10T04:40:15.323 INFO:tasks.rados.rados.0.smithi085.stdout:3127: expect (ObjNum 1045 snap 341 seq_num 1045) 2021-02-10T04:40:15.323 INFO:tasks.rados.rados.0.smithi085.stdout:3128: write oid 42 current snap is 359 2021-02-10T04:40:15.323 INFO:tasks.rados.rados.0.smithi085.stdout:3128: seq_num 1081 ranges {568367=666295,1707623=460744} 2021-02-10T04:40:15.329 INFO:tasks.rados.rados.0.smithi085.stdout:3128: writing smithi08575397-42 from 568367 to 1234662 tid 1 2021-02-10T04:40:15.335 INFO:tasks.rados.rados.0.smithi085.stdout:3128: writing smithi08575397-42 from 1707623 to 2168367 tid 2 2021-02-10T04:40:15.335 INFO:tasks.rados.rados.0.smithi085.stdout:3129: copy_from oid 12 from oid 43 current snap is 359 2021-02-10T04:40:15.335 INFO:tasks.rados.rados.0.smithi085.stdout:3125: finishing write tid 2 to smithi08575397-5 2021-02-10T04:40:15.335 INFO:tasks.rados.rados.0.smithi085.stdout:3125: finishing write tid 3 to smithi08575397-5 2021-02-10T04:40:15.336 INFO:tasks.rados.rados.0.smithi085.stdout:3125: finishing write tid 4 to smithi08575397-5 2021-02-10T04:40:15.336 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 835 (ObjNum 1080 snap 359 seq_num 1080) dirty exists 2021-02-10T04:40:15.336 INFO:tasks.rados.rados.0.smithi085.stdout:3125: left oid 5 (ObjNum 1080 snap 359 seq_num 1080) 2021-02-10T04:40:15.336 INFO:tasks.rados.rados.0.smithi085.stdout:3127: expect (ObjNum 1045 snap 341 seq_num 1045) 2021-02-10T04:40:15.615 INFO:tasks.rados.rados.0.smithi085.stdout:3128: finishing write tid 1 to smithi08575397-42 2021-02-10T04:40:15.616 INFO:tasks.rados.rados.0.smithi085.stdout:3128: finishing write tid 2 to smithi08575397-42 2021-02-10T04:40:15.616 INFO:tasks.rados.rados.0.smithi085.stdout:3128: finishing write tid 3 to smithi08575397-42 2021-02-10T04:40:15.616 INFO:tasks.rados.rados.0.smithi085.stdout:3128: finishing write tid 4 to smithi08575397-42 2021-02-10T04:40:15.616 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 1467 (ObjNum 1081 snap 359 seq_num 1081) dirty exists 2021-02-10T04:40:15.616 INFO:tasks.rados.rados.0.smithi085.stdout:3128: left oid 42 (ObjNum 1081 snap 359 seq_num 1081) 2021-02-10T04:40:15.616 INFO:tasks.rados.rados.0.smithi085.stdout:3129: finishing copy_from to smithi08575397-12 2021-02-10T04:40:15.616 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 1466 (ObjNum 1074 snap 357 seq_num 1074) dirty exists 2021-02-10T04:40:15.616 INFO:tasks.rados.rados.0.smithi085.stdout:3129: finishing copy_from racing read to smithi08575397-12 2021-02-10T04:40:15.616 INFO:tasks.rados.rados.0.smithi085.stdout:3126: finishing copy_from to smithi08575397-34 2021-02-10T04:40:15.616 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 34 v 1439 (ObjNum 1052 snap 345 seq_num 1052) dirty exists 2021-02-10T04:40:15.616 INFO:tasks.rados.rados.0.smithi085.stdout:3126: finishing copy_from racing read to smithi08575397-34 2021-02-10T04:40:15.617 INFO:tasks.rados.rados.0.smithi085.stdout:3125: done (8 left) 2021-02-10T04:40:15.617 INFO:tasks.rados.rados.0.smithi085.stdout:3126: done (7 left) 2021-02-10T04:40:15.617 INFO:tasks.rados.rados.0.smithi085.stdout:3127: done (6 left) 2021-02-10T04:40:15.617 INFO:tasks.rados.rados.0.smithi085.stdout:3128: done (5 left) 2021-02-10T04:40:15.617 INFO:tasks.rados.rados.0.smithi085.stdout:3129: done (4 left) 2021-02-10T04:40:15.617 INFO:tasks.rados.rados.0.smithi085.stdout:3130: write (excl) oid 32 current snap is 359 2021-02-10T04:40:15.617 INFO:tasks.rados.rados.0.smithi085.stdout:3130: seq_num 1082 ranges {585174=741544,1766105=419069} 2021-02-10T04:40:15.623 INFO:tasks.rados.rados.0.smithi085.stdout:3130: writing smithi08575397-32 from 585174 to 1326718 tid 1 2021-02-10T04:40:15.629 INFO:tasks.rados.rados.0.smithi085.stdout:3130: writing smithi08575397-32 from 1766105 to 2185174 tid 2 2021-02-10T04:40:15.629 INFO:tasks.rados.rados.0.smithi085.stdout:3131: snap_create 2021-02-10T04:40:15.631 INFO:tasks.rados.rados.0.smithi085.stdout:3130: finishing write tid 1 to smithi08575397-32 2021-02-10T04:40:15.637 INFO:tasks.rados.rados.0.smithi085.stdout:3130: finishing write tid 2 to smithi08575397-32 2021-02-10T04:40:15.637 INFO:tasks.rados.rados.0.smithi085.stdout:3130: finishing write tid 3 to smithi08575397-32 2021-02-10T04:40:15.637 INFO:tasks.rados.rados.0.smithi085.stdout:3130: finishing write tid 4 to smithi08575397-32 2021-02-10T04:40:15.637 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 32 v 1378 (ObjNum 1082 snap 359 seq_num 1082) dirty exists 2021-02-10T04:40:15.637 INFO:tasks.rados.rados.0.smithi085.stdout:3130: left oid 32 (ObjNum 1082 snap 359 seq_num 1082) 2021-02-10T04:40:15.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:15 smithi164 conmon[33677]: debug 2021-02-10T04:40:15.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:15 smithi164 conmon[33677]: 2021-02-10T04:40:15.162+0000 7fbec19a1700 -1 osd.4 795 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:16.288 INFO:tasks.rados.rados.0.smithi085.stdout:3130: done (5 left) 2021-02-10T04:40:16.288 INFO:tasks.rados.rados.0.smithi085.stdout:3131: done (4 left) 2021-02-10T04:40:16.288 INFO:tasks.rados.rados.0.smithi085.stdout:3132: read oid 23 snap 358 2021-02-10T04:40:16.288 INFO:tasks.rados.rados.0.smithi085.stdout:3132: expect (ObjNum 924 snap 306 seq_num 924) 2021-02-10T04:40:16.288 INFO:tasks.rados.rados.0.smithi085.stdout:3133: delete oid 40 current snap is 360 2021-02-10T04:40:16.293 INFO:tasks.rados.rados.0.smithi085.stdout:3132: expect (ObjNum 924 snap 306 seq_num 924) 2021-02-10T04:40:16.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[33677]: debug 2021-02-10T04:40:16.140+0000 7fbec19a1700 -1 osd.4 796 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:16.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:40:16.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[29288]: :40:15.163010+0000 osd.4 (osd.4) 1275 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:16.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[29288]: cluster 2021-02-10T04:40:15.163018+0000 osd.4 (osd.4) 1276 : cluster [WRN] 2021-02-10T04:40:16.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:16.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[29288]: cluster 2021-02-10T04:40:15. 2021-02-10T04:40:16.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[29288]: 163026+0000 osd.4 (osd.4) 1277 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:16.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[29288]: cluster 2021-02-10T04:40:15 2021-02-10T04:40:16.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[29288]: .163032+0000 osd.4 (osd.4) 1278 : cluster 2021-02-10T04:40:16.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:16.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[29288]: cluster 2021-02-10T04:40: 2021-02-10T04:40:16.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[29288]: 15.277243+0000 mon.a (mon.0) 1735 : 2021-02-10T04:40:16.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[29288]: cluster [DBG] osdmap e796: 8 total, 8 up, 7 in 2021-02-10T04:40:16.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[29288]: cluster 2021-02-10T04:40 2021-02-10T04:40:16.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[29288]: :16.141387+0000 osd.4 (osd.4) 1279 2021-02-10T04:40:16.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:16.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[29288]: cluster 2021-02-10T04:40:16 2021-02-10T04:40:16.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[29288]: .141399+0000 osd.4 (osd.4) 1280 : cluster [WRN] 2021-02-10T04:40:16.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:16.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[29288]: cluster 2021-02-10T04:40:16.141405 2021-02-10T04:40:16.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1281 : cluster [WRN] 2021-02-10T04:40:16.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:16.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[29288]: cluster 2021-02-10T04:40:16.141408 2021-02-10T04:40:16.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1282 : cluster 2021-02-10T04:40:16.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:16.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[29288]: cluster 2021-02-10T04:40:16 2021-02-10T04:40:16.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[29288]: . 2021-02-10T04:40:16.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[29288]: 190295 2021-02-10T04:40:16.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[29288]: +0000 2021-02-10T04:40:16.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[29288]: mgr.y (mgr.14140) 779 : cluster 2021-02-10T04:40:16.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:16 smithi164 conmon[29288]: [DBG] pgmap v1345: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 510 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 8.3 MiB/s rd, 1.6 MiB/s wr, 16 op/s 2021-02-10T04:40:16.451 INFO:tasks.rados.rados.0.smithi085.stdout:3132: done (5 left) 2021-02-10T04:40:16.451 INFO:tasks.rados.rados.0.smithi085.stdout:3133: done (4 left) 2021-02-10T04:40:16.451 INFO:tasks.rados.rados.0.smithi085.stdout:3134: rollback oid 48 current snap is 360 2021-02-10T04:40:16.451 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 48 to 278 2021-02-10T04:40:16.451 INFO:tasks.rados.rados.0.smithi085.stdout:3135: copy_from oid 13 from oid 22 current snap is 360 2021-02-10T04:40:16.451 INFO:tasks.rados.rados.0.smithi085.stdout:3136: snap_create 2021-02-10T04:40:16.454 INFO:tasks.rados.rados.0.smithi085.stdout:3134: finishing rollback tid 0 to smithi08575397-48 2021-02-10T04:40:16.455 INFO:tasks.rados.rados.0.smithi085.stdout:3134: finishing rollback tid 1 to smithi08575397-48 2021-02-10T04:40:16.455 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 1469 (ObjNum 1215200272 snap 32576 seq_num 83182448) dirty dne 2021-02-10T04:40:16.465 INFO:tasks.rados.rados.0.smithi085.stdout:3135: finishing copy_from to smithi08575397-13 2021-02-10T04:40:16.465 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 1453 (ObjNum 982 snap 317 seq_num 982) dirty exists 2021-02-10T04:40:16.465 INFO:tasks.rados.rados.0.smithi085.stdout:3135: finishing copy_from racing read to smithi08575397-13 2021-02-10T04:40:16.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[35961]: cluster 2021-02-10T04:40:16.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[35961]: 2021-02-10T04:40:16.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[35961]: 2021-02-10T04:40:15 2021-02-10T04:40:16.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[35961]: .163010+0000 osd.4 (osd.4) 1275 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:16.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[35961]: cluster 2021-02-10T04:40:15.163018+0000 osd.4 (osd.4) 1276 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:16.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[35961]: cluster 2021-02-10T04:40:15.163026+0000 osd.4 (osd.4) 1277 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:16.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[35961]: cluster 2021-02-10T04:40:15.163032+0000 osd.4 (osd.4) 1278 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:16.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[35961]: cluster 2021-02-10T04:40:15.277243+0000 mon.a (mon.0) 1735 : cluster [DBG] osdmap e796: 8 total, 8 up, 7 in 2021-02-10T04:40:16.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[35961]: cluster 2021-02-10T04:40:16.141387+0000 osd.4 (osd.4) 1279 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:16.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[35961]: cluster 2021-02-10T04:40:16.141399 2021-02-10T04:40:16.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1280 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:16.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[35961]: cluster 2021-02-10T04:40:16.141405+0000 osd.4 (osd.4) 1281 : cluster [WRN] 2021-02-10T04:40:16.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:16.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:40:16.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[35961]: T04:40:16.141408+0000 osd.4 (osd.4) 1282 : cluster [WRN] 2021-02-10T04:40:16.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:16.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[35961]: cluster 2021-02-10T04:40:16.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[35961]: 2021-02-10T04:40:16.190295+0000 mgr.y (mgr.14140) 2021-02-10T04:40:16.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[35961]: 779 : cluster [DBG] pgmap v1345: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 510 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 8.3 MiB/s rd, 1.6 MiB/s wr, 16 op/s 2021-02-10T04:40:16.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: cluster 2021-02-10T04:40:16.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:40:16.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: :40:15.163010+0000 osd.4 (osd.4) 1275 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:16.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:40:16.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: :40:15.163018+0000 osd.4 (osd.4) 1276 : 2021-02-10T04:40:16.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:16.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: cluster 2021-02-10T04:40 2021-02-10T04:40:16.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: :15.163026+0000 osd.4 (osd.4) 1277 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:16.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: cluster 2021-02-10T04:40:15.163032 2021-02-10T04:40:16.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1278 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:16.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: 2021-02-10T04:40:16.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: cluster 2021-02-10T04:40:16.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: 2021-02-10T04:40:15.277243 2021-02-10T04:40:16.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1735 : cluster [DBG] 2021-02-10T04:40:16.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: osdmap e796: 8 total, 8 up, 7 in 2021-02-10T04:40:16.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: cluster 2021-02-10T04:40:16.141387+0000 2021-02-10T04:40:16.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: osd.4 (osd.4) 1279 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:16.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: 2021-02-10T04:40:16.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: cluster 2021-02-10T04:40:16.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: 2021-02-10T04:40:16.141399+0000 2021-02-10T04:40:16.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: osd.4 (osd.4) 1280 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:16.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:40:16.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: -02-10T04:40:16.141405+0000 osd.4 2021-02-10T04:40:16.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: (osd.4) 1281 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:16.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:40:16.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: 02-10T04:40:16.141408+0000 osd.4 2021-02-10T04:40:16.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: (osd.4) 1282 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:16.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: cluster 2021-02-10T04:40:16.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: 2021-02-10T04:40:16. 2021-02-10T04:40:16.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: 190295+0000 mgr.y (mgr.14140) 779 : cluster [DBG] 2021-02-10T04:40:16.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:16 smithi085 conmon[29319]: pgmap v1345: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 510 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 8.3 MiB/s rd, 1.6 MiB/s wr, 16 op/s 2021-02-10T04:40:17.287 INFO:tasks.rados.rados.0.smithi085.stdout:3134: done (6 left) 2021-02-10T04:40:17.287 INFO:tasks.rados.rados.0.smithi085.stdout:3135: done (5 left) 2021-02-10T04:40:17.288 INFO:tasks.rados.rados.0.smithi085.stdout:3136: done (4 left) 2021-02-10T04:40:17.288 INFO:tasks.rados.rados.0.smithi085.stdout:3137: read oid 48 snap -1 2021-02-10T04:40:17.288 INFO:tasks.rados.rados.0.smithi085.stdout:3137: expect deleted 2021-02-10T04:40:17.288 INFO:tasks.rados.rados.0.smithi085.stdout:3138: read oid 49 snap -1 2021-02-10T04:40:17.288 INFO:tasks.rados.rados.0.smithi085.stdout:3138: expect (ObjNum 172 snap 37 seq_num 172) 2021-02-10T04:40:17.288 INFO:tasks.rados.rados.0.smithi085.stdout:3139: copy_from oid 15 from oid 13 current snap is 361 2021-02-10T04:40:17.288 INFO:tasks.rados.rados.0.smithi085.stdout:3140: rollback oid 43 current snap is 361 2021-02-10T04:40:17.288 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 43 to 353 2021-02-10T04:40:17.288 INFO:tasks.rados.rados.0.smithi085.stdout:3141: read oid 50 snap 337 2021-02-10T04:40:17.288 INFO:tasks.rados.rados.0.smithi085.stdout:3141: expect deleted 2021-02-10T04:40:17.288 INFO:tasks.rados.rados.0.smithi085.stdout:3142: snap_remove snap 310 2021-02-10T04:40:17.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:17 smithi164 conmon[29288]: cluster 2021-02-10T04:40:16.287137+0000 mon.a (mon.0 2021-02-10T04:40:17.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:17 smithi164 conmon[29288]: ) 1736 : cluster [DBG] osdmap e797: 8 total, 8 up, 7 in 2021-02-10T04:40:17.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:17 smithi164 conmon[33677]: debug 2021-02-10T04:40:17.177+0000 7fbec19a1700 -1 osd.4 796 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:17.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:17 smithi085 conmon[29319]: cluster 2021-02-10T04:40:16.287137+0000 mon.a (mon. 2021-02-10T04:40:17.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:17 smithi085 conmon[29319]: 0) 1736 : cluster [DBG] osdmap e797: 8 total, 8 up, 7 in 2021-02-10T04:40:17.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:17 smithi085 conmon[35961]: cluster 2021-02-10T04:40:16.287137+0000 mon.a 2021-02-10T04:40:17.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:17 smithi085 conmon[35961]: (mon.0) 1736 : cluster [DBG] osdmap e797: 8 total, 8 up, 7 in 2021-02-10T04:40:17.934 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:40:17.965 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:40:18.263 INFO:tasks.rados.rados.0.smithi085.stdout:3142: done (9 left) 2021-02-10T04:40:18.263 INFO:tasks.rados.rados.0.smithi085.stdout:3143: delete oid 16 current snap is 361 2021-02-10T04:40:18.263 INFO:tasks.rados.rados.0.smithi085.stdout:3140: finishing rollback tid 0 to smithi08575397-43 2021-02-10T04:40:18.264 INFO:tasks.rados.rados.0.smithi085.stdout:3140: finishing rollback tid 1 to smithi08575397-43 2021-02-10T04:40:18.264 INFO:tasks.rados.rados.0.smithi085.stdout:3140: finishing rollback tid 2 to smithi08575397-43 2021-02-10T04:40:18.264 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 1469 (ObjNum 1044 snap 341 seq_num 1044) dirty exists 2021-02-10T04:40:18.264 INFO:tasks.rados.rados.0.smithi085.stdout:3138: expect (ObjNum 172 snap 37 seq_num 172) 2021-02-10T04:40:18.289 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:40:18 smithi164 conmon[44544]: debug 2021-02-10T04:40:18.038+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:40:18.289 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:18 smithi164 conmon[33677]: debug 2021-02-10T04:40:18.289 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:18 smithi164 conmon[33677]: 2021-02-10T04:40:18.203+0000 7fbec19a1700 -1 osd.4 798 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:18.307 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:40:18.396 INFO:tasks.rados.rados.0.smithi085.stdout:3139: finishing copy_from to smithi08575397-15 2021-02-10T04:40:18.396 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 15 v 1453 (ObjNum 982 snap 317 seq_num 982) dirty exists 2021-02-10T04:40:18.396 INFO:tasks.rados.rados.0.smithi085.stdout:3139: finishing copy_from racing read to smithi08575397-15 2021-02-10T04:40:18.397 INFO:tasks.rados.rados.0.smithi085.stdout:3137: done (9 left) 2021-02-10T04:40:18.397 INFO:tasks.rados.rados.0.smithi085.stdout:3138: done (8 left) 2021-02-10T04:40:18.397 INFO:tasks.rados.rados.0.smithi085.stdout:3139: done (7 left) 2021-02-10T04:40:18.397 INFO:tasks.rados.rados.0.smithi085.stdout:3140: done (6 left) 2021-02-10T04:40:18.397 INFO:tasks.rados.rados.0.smithi085.stdout:3141: done (5 left) 2021-02-10T04:40:18.397 INFO:tasks.rados.rados.0.smithi085.stdout:3143: done (4 left) 2021-02-10T04:40:18.397 INFO:tasks.rados.rados.0.smithi085.stdout:3144: read oid 34 snap 278 2021-02-10T04:40:18.397 INFO:tasks.rados.rados.0.smithi085.stdout:3144: expect (ObjNum 859 snap 277 seq_num 859) 2021-02-10T04:40:18.397 INFO:tasks.rados.rados.0.smithi085.stdout:3145: copy_from oid 12 from oid 42 current snap is 361 2021-02-10T04:40:18.397 INFO:tasks.rados.rados.0.smithi085.stdout:3146: rollback oid 22 current snap is 361 2021-02-10T04:40:18.397 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 22 to 348 2021-02-10T04:40:18.397 INFO:tasks.rados.rados.0.smithi085.stdout:3147: copy_from oid 7 from oid 46 current snap is 361 2021-02-10T04:40:18.398 INFO:tasks.rados.rados.0.smithi085.stdout:3148: rollback oid 15 current snap is 361 2021-02-10T04:40:18.398 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 15 to 306 2021-02-10T04:40:18.398 INFO:tasks.rados.rados.0.smithi085.stdout:3149: rollback oid 37 current snap is 361 2021-02-10T04:40:18.398 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 37 to 357 2021-02-10T04:40:18.398 INFO:tasks.rados.rados.0.smithi085.stdout:3150: delete oid 8 current snap is 361 2021-02-10T04:40:18.402 INFO:tasks.rados.rados.0.smithi085.stdout:3146: finishing rollback tid 0 to smithi08575397-22 2021-02-10T04:40:18.402 INFO:tasks.rados.rados.0.smithi085.stdout:3146: finishing rollback tid 1 to smithi08575397-22 2021-02-10T04:40:18.402 INFO:tasks.rados.rados.0.smithi085.stdout:3146: finishing rollback tid 2 to smithi08575397-22 2021-02-10T04:40:18.402 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 1460 (ObjNum 982 snap 317 seq_num 982) dirty exists 2021-02-10T04:40:18.403 INFO:tasks.rados.rados.0.smithi085.stdout:3146: done (10 left) 2021-02-10T04:40:18.403 INFO:tasks.rados.rados.0.smithi085.stdout:3150: done (9 left) 2021-02-10T04:40:18.403 INFO:tasks.rados.rados.0.smithi085.stdout:3151: copy_from oid 13 from oid 43 current snap is 361 2021-02-10T04:40:18.403 INFO:tasks.rados.rados.0.smithi085.stdout:3152: snap_create 2021-02-10T04:40:18.403 INFO:tasks.rados.rados.0.smithi085.stdout:3144: expect (ObjNum 859 snap 277 seq_num 859) 2021-02-10T04:40:18.412 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:40:18.447 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:40:18.509 INFO:tasks.rados.rados.0.smithi085.stdout:3149: finishing rollback tid 1 to smithi08575397-37 2021-02-10T04:40:18.510 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 1619 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-02-10T04:40:18.510 INFO:tasks.rados.rados.0.smithi085.stdout:3148: finishing rollback tid 0 to smithi08575397-15 2021-02-10T04:40:18.510 INFO:tasks.rados.rados.0.smithi085.stdout:3148: finishing rollback tid 1 to smithi08575397-15 2021-02-10T04:40:18.510 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 15 v 1455 (ObjNum 82689008 snap 22030 seq_num 82349504) dirty dne 2021-02-10T04:40:18.510 INFO:tasks.rados.rados.0.smithi085.stdout:3145: finishing copy_from to smithi08575397-12 2021-02-10T04:40:18.510 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 1467 (ObjNum 1081 snap 359 seq_num 1081) dirty exists 2021-02-10T04:40:18.510 INFO:tasks.rados.rados.0.smithi085.stdout:3145: finishing copy_from racing read to smithi08575397-12 2021-02-10T04:40:18.510 INFO:tasks.rados.rados.0.smithi085.stdout:3147: finishing copy_from to smithi08575397-7 2021-02-10T04:40:18.510 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 1549 (ObjNum 1054 snap 346 seq_num 1054) dirty exists 2021-02-10T04:40:18.510 INFO:tasks.rados.rados.0.smithi085.stdout:3147: finishing copy_from racing read to smithi08575397-7 2021-02-10T04:40:18.524 INFO:tasks.rados.rados.0.smithi085.stdout:3151: finishing copy_from to smithi08575397-13 2021-02-10T04:40:18.524 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 1469 (ObjNum 1044 snap 341 seq_num 1044) dirty exists 2021-02-10T04:40:18.524 INFO:tasks.rados.rados.0.smithi085.stdout:3151: finishing copy_from racing read to smithi08575397-13 2021-02-10T04:40:18.643 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:40:18 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:40:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:18 smithi164 conmon[29288]: cluster 2021-02-10T04:40:17.178664+0000 osd.4 (osd.4) 2021-02-10T04:40:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:18 smithi164 conmon[29288]: 1283 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:18 smithi164 conmon[29288]: cluster 2021-02-10T04:40:17.178675+0000 osd.4 (osd.4) 1284 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:18 smithi164 conmon[29288]: cluster 2021-02-10T04:40:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:18 smithi164 conmon[29288]: 2021-02-10T04:40:17.178682+0000 osd.4 (osd.4) 1285 : cluster 2021-02-10T04:40:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:18 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:18 smithi164 conmon[29288]: cluster 2021-02-10T04:40:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:18 smithi164 conmon[29288]: 2021-02-10T04:40:17. 2021-02-10T04:40:18.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:18 smithi164 conmon[29288]: 178689+0000 osd.4 (osd.4) 1286 : cluster [WRN] 2021-02-10T04:40:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:18 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:18 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:40:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:18 smithi164 conmon[29288]: T04:40:17.287058+0000 mon.a ( 2021-02-10T04:40:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:18 smithi164 conmon[29288]: mon.0) 1737 : cluster [DBG] 2021-02-10T04:40:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:18 smithi164 conmon[29288]: osdmap e798: 8 total, 8 up, 7 in 2021-02-10T04:40:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:18 smithi164 conmon[29288]: cluster 2021-02-10T04:40:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:18 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:40:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:18 smithi164 conmon[29288]: 40:18.190955+0000 mgr.y ( 2021-02-10T04:40:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:18 smithi164 conmon[29288]: mgr.14140) 780 : cluster [DBG] 2021-02-10T04:40:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:18 smithi164 conmon[29288]: pgmap v1348: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 517 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 3.6 MiB/s wr, 9 op/s 2021-02-10T04:40:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:18 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:40:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:18 smithi164 conmon[29288]: 02-10T04:40:18.261833+0000 2021-02-10T04:40:18.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:18 smithi164 conmon[29288]: mon.a (mon.0) 1738 : cluster [DBG] osdmap e799: 8 total, 8 up, 7 in 2021-02-10T04:40:18.693 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[45443]: debug 2021-02-10T04:40:18.532+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:40:18.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[35961]: cluster 2021-02-10T04:40:17.178664+0000 osd.4 (osd 2021-02-10T04:40:18.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[35961]: .4) 1283 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:18.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[35961]: cluster 2021-02-10T04:40:17.178675+0000 osd.4 (osd.4) 1284 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:18.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[35961]: 2021-02-10T04:40:18.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:40:18.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[35961]: 04:40:17.178682+0000 osd.4 (osd.4) 1285 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:18.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[35961]: 2021-02-10T04:40:18.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[35961]: cluster 2021-02-10T04:40:18.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[35961]: 2021-02-10T04:40:17.178689+0000 osd.4 (osd 2021-02-10T04:40:18.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[35961]: .4) 1286 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:18.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:40:18.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[35961]: 10T04:40:17.287058+0000 mon.a (mon 2021-02-10T04:40:18.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[35961]: .0) 1737 : cluster [DBG] osdmap e798: 8 total, 8 up, 7 in 2021-02-10T04:40:18.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:40:18.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[35961]: -10T04:40:18.190955+0000 mgr.y (mgr 2021-02-10T04:40:18.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[35961]: .14140) 780 : cluster [DBG] pgmap v1348: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 517 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 3.6 MiB/s wr, 9 op/s 2021-02-10T04:40:18.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:40:18.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[35961]: :40:18.261833+0000 mon.a (mon.0) 1738 2021-02-10T04:40:18.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[35961]: : cluster [DBG] osdmap e799: 8 total, 8 up, 7 in 2021-02-10T04:40:18.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[29319]: cluster 2021-02-10T04:40:17.178664+0000 osd.4 (osd 2021-02-10T04:40:18.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[29319]: .4) 1283 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:18.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[29319]: cluster 2021-02-10T04:40:17.178675+0000 osd.4 (osd.4) 1284 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:18.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[29319]: cluster 2021-02-10T04:40:17.178682+0000 osd.4 (osd.4) 1285 2021-02-10T04:40:18.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:18.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:40:18.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[29319]: 40:17.178689+0000 osd.4 (osd.4) 2021-02-10T04:40:18.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[29319]: 1286 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:18.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:40:18.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[29319]: :40:17.287058+0000 mon.a (mon.0 2021-02-10T04:40:18.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[29319]: ) 1737 : cluster [DBG] osdmap e798: 8 total, 8 up, 7 in 2021-02-10T04:40:18.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:40:18.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[29319]: -10T04:40:18.190955+0000 mgr.y ( 2021-02-10T04:40:18.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[29319]: mgr.14140) 780 : cluster [DBG] pgmap v1348: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 517 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 3.6 MiB/s wr, 9 op/s 2021-02-10T04:40:18.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:40:18.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[29319]: -10T04:40:18.261833+0000 mon.a 2021-02-10T04:40:18.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:18 smithi085 conmon[29319]: (mon.0) 1738 : cluster [DBG] osdmap e799: 8 total, 8 up, 7 in 2021-02-10T04:40:18.827 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:40:19.302 INFO:tasks.rados.rados.0.smithi085.stdout:3144: done (10 left) 2021-02-10T04:40:19.302 INFO:tasks.rados.rados.0.smithi085.stdout:3145: done (9 left) 2021-02-10T04:40:19.302 INFO:tasks.rados.rados.0.smithi085.stdout:3147: done (8 left) 2021-02-10T04:40:19.302 INFO:tasks.rados.rados.0.smithi085.stdout:3148: done (7 left) 2021-02-10T04:40:19.303 INFO:tasks.rados.rados.0.smithi085.stdout:3149: done (6 left) 2021-02-10T04:40:19.303 INFO:tasks.rados.rados.0.smithi085.stdout:3151: done (5 left) 2021-02-10T04:40:19.303 INFO:tasks.rados.rados.0.smithi085.stdout:3152: done (4 left) 2021-02-10T04:40:19.303 INFO:tasks.rados.rados.0.smithi085.stdout:3153: copy_from oid 50 from oid 30 current snap is 362 2021-02-10T04:40:19.303 INFO:tasks.rados.rados.0.smithi085.stdout:3154: read oid 7 snap -1 2021-02-10T04:40:19.303 INFO:tasks.rados.rados.0.smithi085.stdout:3154: expect (ObjNum 1054 snap 346 seq_num 1054) 2021-02-10T04:40:19.303 INFO:tasks.rados.rados.0.smithi085.stdout:3155: copy_from oid 46 from oid 5 current snap is 362 2021-02-10T04:40:19.303 INFO:tasks.rados.rados.0.smithi085.stdout:3156: rollback oid 14 current snap is 362 2021-02-10T04:40:19.303 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 14 to 308 2021-02-10T04:40:19.303 INFO:tasks.rados.rados.0.smithi085.stdout:3157: rollback oid 22 current snap is 362 2021-02-10T04:40:19.303 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 22 to 355 2021-02-10T04:40:19.303 INFO:tasks.rados.rados.0.smithi085.stdout:3158: snap_remove snap 353 2021-02-10T04:40:19.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:19 smithi164 conmon[33677]: debug 2021-02-10T04:40:19.179+0000 7fbec19a1700 -1 osd.4 799 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:19 smithi164 conmon[29288]: cluster 2021-02-10T04:40:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:19 smithi164 conmon[29288]: 2021-02-10T04:40:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:19 smithi164 conmon[29288]: 2021 2021-02-10T04:40:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:19 smithi164 conmon[29288]: -02-10T04:40:18.204645+0000 2021-02-10T04:40:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:19 smithi164 conmon[29288]: osd.4 (osd.4) 1287 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:19 smithi164 conmon[29288]: cluster 2021-02-10T04:40:18 2021-02-10T04:40:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:19 smithi164 conmon[29288]: .204655+0000 osd.4 (osd.4) 1288 : cluster 2021-02-10T04:40:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:19 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:19 smithi164 conmon[29288]: cluster 2021-02-10T04:40:18 2021-02-10T04:40:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:19 smithi164 conmon[29288]: .204661+0000 osd.4 (osd.4) 1289 : cluster [WRN] 2021-02-10T04:40:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:19 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:19 smithi164 conmon[29288]: cluster 2021-02-10T04:40: 2021-02-10T04:40:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:19 smithi164 conmon[29288]: 18.204666+0000 osd.4 (osd.4) 1290 : cluster 2021-02-10T04:40:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:19 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:19 smithi164 conmon[29288]: cluster 2021-02-10T04:40:18. 2021-02-10T04:40:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:19 smithi164 conmon[29288]: 285866+0000 mon.a (mon.0) 1739 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 647 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:40:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:19 smithi164 conmon[29288]: audit 2021-02-10T04:40:18.387236+0000 2021-02-10T04:40:19.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:19 smithi164 conmon[29288]: mgr.y (mgr.14140) 781 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:40:19.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[35961]: cluster 2021-02-10T04:40:18.204645+0000 osd.4 2021-02-10T04:40:19.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[35961]: (osd.4) 1287 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:19.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:40:19.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[35961]: 02-10T04:40:18.204655+0000 osd.4 (osd. 2021-02-10T04:40:19.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[35961]: 4) 1288 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:19.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[35961]: cluster 2021-02-10T04:40:19.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:40:19.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[35961]: T04:40:18.204661+0000 osd.4 (osd 2021-02-10T04:40:19.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[35961]: .4) 1289 : cluster [WRN] 2021-02-10T04:40:19.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:19.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[35961]: cluster 2021-02-10T04:40:19.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:40:19.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[35961]: :40:18.204666+0000 osd.4 (osd.4) 1290 : 2021-02-10T04:40:19.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:19.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[35961]: cluster 2021-02-10T04:40:19.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:40:19.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[35961]: 04:40:18.285866+0000 mon.a (mon.0) 1739 : cluster [WRN] 2021-02-10T04:40:19.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[35961]: Health check update: 4 slow ops, oldest one blocked for 647 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:40:19.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:40:19.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[35961]: -10T04:40:18.387236+0000 mgr.y (mgr.14140) 781 : 2021-02-10T04:40:19.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[35961]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:40:19.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[29319]: cluster 2021-02-10T04:40:19.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[29319]: 2021-02-10T04:40:19.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[29319]: 2021-02-10T04:40:18.204645 2021-02-10T04:40:19.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1287 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:19.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:40:19.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[29319]: 40:18.204655+0000 osd.4 (osd.4) 1288 : 2021-02-10T04:40:19.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:19.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:40:19.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[29319]: 40:18.204661+0000 osd.4 (osd.4) 1289 : 2021-02-10T04:40:19.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:19.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[29319]: cluster 2021-02-10T04:40 2021-02-10T04:40:19.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[29319]: :18.204666+0000 osd.4 (osd.4) 2021-02-10T04:40:19.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[29319]: 1290 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:19.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:40:19.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[29319]: :40:18.285866+0000 mon.a (mon.0) 2021-02-10T04:40:19.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[29319]: 1739 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 647 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:40:19.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[29319]: audit 2021-02-10T04:40:19.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[29319]: 2021-02-10T04:40:18.387236+0000 mgr.y 2021-02-10T04:40:19.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:19 smithi085 conmon[29319]: (mgr.14140) 781 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:40:20.305 INFO:tasks.rados.rados.0.smithi085.stdout:3158: done (9 left) 2021-02-10T04:40:20.305 INFO:tasks.rados.rados.0.smithi085.stdout:3159: snap_create 2021-02-10T04:40:20.305 INFO:tasks.rados.rados.0.smithi085.stdout:3156: finishing rollback tid 0 to smithi08575397-14 2021-02-10T04:40:20.305 INFO:tasks.rados.rados.0.smithi085.stdout:3156: finishing rollback tid 1 to smithi08575397-14 2021-02-10T04:40:20.305 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 1471 (ObjNum 82848608 snap 22030 seq_num 82463456) dirty dne 2021-02-10T04:40:20.305 INFO:tasks.rados.rados.0.smithi085.stdout:3153: finishing copy_from to smithi08575397-50 2021-02-10T04:40:20.305 INFO:tasks.rados.rados.0.smithi085.stdout:3153: got expected ENOENT (src dne) 2021-02-10T04:40:20.305 INFO:tasks.rados.rados.0.smithi085.stdout:3153: finishing copy_from racing read to smithi08575397-50 2021-02-10T04:40:20.305 INFO:tasks.rados.rados.0.smithi085.stdout:3157: finishing rollback tid 0 to smithi08575397-22 2021-02-10T04:40:20.305 INFO:tasks.rados.rados.0.smithi085.stdout:3154: expect (ObjNum 1054 snap 346 seq_num 1054) 2021-02-10T04:40:20.550 INFO:tasks.rados.rados.0.smithi085.stdout:3157: finishing rollback tid 1 to smithi08575397-22 2021-02-10T04:40:20.550 INFO:tasks.rados.rados.0.smithi085.stdout:3157: finishing rollback tid 2 to smithi08575397-22 2021-02-10T04:40:20.550 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 1463 (ObjNum 982 snap 317 seq_num 982) dirty exists 2021-02-10T04:40:20.550 INFO:tasks.rados.rados.0.smithi085.stdout:3155: finishing copy_from to smithi08575397-46 2021-02-10T04:40:20.550 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 835 (ObjNum 1080 snap 359 seq_num 1080) dirty exists 2021-02-10T04:40:20.550 INFO:tasks.rados.rados.0.smithi085.stdout:3155: finishing copy_from racing read to smithi08575397-46 2021-02-10T04:40:20.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:20 smithi164 conmon[33677]: debug 2021-02-10T04:40:20.188+0000 7fbec19a1700 -1 osd.4 800 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:20 smithi164 conmon[29288]: cluster 2021-02-10T04:40:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:20 smithi164 conmon[29288]: 2021- 2021-02-10T04:40:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:20 smithi164 conmon[29288]: 02-10T04:40:19.180019+0000 osd.4 (osd.4) 1291 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:20 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:40:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:20 smithi164 conmon[29288]: 10T04:40:19.180028+0000 osd.4 (osd.4) 2021-02-10T04:40:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:20 smithi164 conmon[29288]: 1292 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:20 smithi164 conmon[29288]: cluster 2021-02-10T04:40:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:20 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:40:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:20 smithi164 conmon[29288]: 40:19.180034+0000 osd.4 (osd.4) 2021-02-10T04:40:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:20 smithi164 conmon[29288]: 1293 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:20 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:40:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:20 smithi164 conmon[29288]: -10T04:40:19.180040+0000 osd.4 ( 2021-02-10T04:40:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:20 smithi164 conmon[29288]: osd.4) 1294 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:20 smithi164 conmon[29288]: cluster 2021-02-10T04:40:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:20 smithi164 conmon[29288]: 2021-02-10T04:40:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:20 smithi164 conmon[29288]: 2021 2021-02-10T04:40:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:20 smithi164 conmon[29288]: -02-10T04 2021-02-10T04:40:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:20 smithi164 conmon[29288]: :40:19.302445+0000 mon.a (mon.0) 1740 : cluster 2021-02-10T04:40:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:20 smithi164 conmon[29288]: [DBG] osdmap e800: 8 total, 8 up, 7 in 2021-02-10T04:40:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:20 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:40:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:20 smithi164 conmon[29288]: -02-10T04:40:20.191669+0000 2021-02-10T04:40:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:20 smithi164 conmon[29288]: mgr.y (mgr.14140) 782 : cluster [DBG] pgmap v1351: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 529 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 5.9 MiB/s wr, 22 op/s 2021-02-10T04:40:20.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:20 smithi164 conmon[29288]: 2021-02-10T04:40:20.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: cluster 2021-02-10T04:40:20.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:40:20.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: T04:40:19.180019+0000 osd.4 (osd.4) 1291 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:20.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: 2021-02-10T04:40:20.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: cluster 2021-02-10T04:40:20.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: 2021-02-10T04:40:20.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: 2021 2021-02-10T04:40:20.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: - 2021-02-10T04:40:20.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: 02 2021-02-10T04:40:20.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: - 2021-02-10T04:40:20.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: 10 2021-02-10T04:40:20.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: T 2021-02-10T04:40:20.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: 04 2021-02-10T04:40:20.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: : 2021-02-10T04:40:20.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: 40 2021-02-10T04:40:20.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: : 2021-02-10T04:40:20.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: 19 2021-02-10T04:40:20.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: . 2021-02-10T04:40:20.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: 180028 2021-02-10T04:40:20.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: +0000 2021-02-10T04:40:20.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: 2021-02-10T04:40:20.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: osd.4 (osd.4) 1292 2021-02-10T04:40:20.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:20.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:40:20.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: -10T04:40:19.180034+0000 osd.4 (osd.4) 2021-02-10T04:40:20.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: 1293 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:20.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: cluster 2021-02-10T04:40:20.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: 2021-02-10T04:40:19.180040+0000 2021-02-10T04:40:20.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: osd.4 (osd.4) 1294 : cluster [WRN] 2021-02-10T04:40:20.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:20.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:40:20.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: 02-10T04:40:19.302445+0000 mon.a (mon. 2021-02-10T04:40:20.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: 0) 1740 : cluster [DBG] osdmap e800: 8 total, 8 up, 7 in 2021-02-10T04:40:20.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: cluster 2021-02-10T04:40:20.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: 2021-02 2021-02-10T04:40:20.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: -10T04:40:20.191669+0000 mgr.y (mgr.14140 2021-02-10T04:40:20.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[35961]: ) 782 : cluster [DBG] pgmap v1351: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 529 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 5.9 MiB/s wr, 22 op/s 2021-02-10T04:40:20.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: cluster 2021-02-10T04:40:19.180019+0000 osd.4 (osd. 2021-02-10T04:40:20.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: 4) 1291 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:20.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: cluster 2021-02-10T04:40:20.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: 2021-02-10T04:40:20.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: 2021-02-10T04:40:19.180028 2021-02-10T04:40:20.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1292 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:20.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: cluster 2021-02-10T04:40:20.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: 2021-02-10T04:40:20.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: 2021 2021-02-10T04:40:20.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: - 2021-02-10T04:40:20.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: 02 2021-02-10T04:40:20.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: - 2021-02-10T04:40:20.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: 10 2021-02-10T04:40:20.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: T 2021-02-10T04:40:20.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: 04 2021-02-10T04:40:20.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: : 2021-02-10T04:40:20.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: 40 2021-02-10T04:40:20.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: : 2021-02-10T04:40:20.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: 19 2021-02-10T04:40:20.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: . 2021-02-10T04:40:20.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: 180034 2021-02-10T04:40:20.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: +0000 2021-02-10T04:40:20.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: 2021-02-10T04:40:20.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: osd.4 2021-02-10T04:40:20.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: ( 2021-02-10T04:40:20.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: osd 2021-02-10T04:40:20.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: . 2021-02-10T04:40:20.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: 4 2021-02-10T04:40:20.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: ) 2021-02-10T04:40:20.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: 1293 2021-02-10T04:40:20.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: : 2021-02-10T04:40:20.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: cluster 2021-02-10T04:40:20.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:20.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:40:20.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: 02-10T04:40:19.180040+0000 osd.4 (osd.4) 2021-02-10T04:40:20.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: 1294 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:20.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: cluster 2021-02-10T04:40:20.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: 2021-02-10T04:40:20.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: 2021 2021-02-10T04:40:20.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: - 2021-02-10T04:40:20.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: 02-10T04:40:19. 2021-02-10T04:40:20.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: 302445+0000 mon.a (mon.0) 1740 : cluster [DBG] osdmap e800: 8 total, 8 up, 7 in 2021-02-10T04:40:20.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:40:20.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: 10T04:40:20.191669+0000 mgr.y (mgr.14140 2021-02-10T04:40:20.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:20 smithi085 conmon[29319]: ) 782 : cluster [DBG] pgmap v1351: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 529 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 5.9 MiB/s wr, 22 op/s 2021-02-10T04:40:21.307 INFO:tasks.rados.rados.0.smithi085.stdout:3153: done (9 left) 2021-02-10T04:40:21.307 INFO:tasks.rados.rados.0.smithi085.stdout:3154: done (8 left) 2021-02-10T04:40:21.307 INFO:tasks.rados.rados.0.smithi085.stdout:3155: done (7 left) 2021-02-10T04:40:21.307 INFO:tasks.rados.rados.0.smithi085.stdout:3156: done (6 left) 2021-02-10T04:40:21.307 INFO:tasks.rados.rados.0.smithi085.stdout:3157: done (5 left) 2021-02-10T04:40:21.307 INFO:tasks.rados.rados.0.smithi085.stdout:3159: done (4 left) 2021-02-10T04:40:21.307 INFO:tasks.rados.rados.0.smithi085.stdout:3160: snap_create 2021-02-10T04:40:21.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:21 smithi164 conmon[33677]: debug 2021-02-10T04:40:21.214+0000 7fbec19a1700 -1 osd.4 801 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:21 smithi164 conmon[29288]: cluster 2021-02-10T04:40:20.189115+0000 osd.4 ( 2021-02-10T04:40:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:21 smithi164 conmon[29288]: osd.4) 1295 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:21 smithi164 conmon[29288]: cluster 2021-02-10T04:40:20.189145+0000 osd.4 (osd.4) 1296 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:21 smithi164 conmon[29288]: cluster 2021-02-10T04:40:20. 2021-02-10T04:40:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:21 smithi164 conmon[29288]: 189152+0000 osd.4 (osd.4) 1297 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:21 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:40:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:21 smithi164 conmon[29288]: T04:40:20.189157+0000 osd.4 (osd.4) 1298 : cluster 2021-02-10T04:40:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:21 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:21 smithi164 conmon[29288]: cluster 2021-02-10T04:40:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:21 smithi164 conmon[29288]: 2021-02-10T04:40:20.303361+0000 mon.a (mon.0) 1741 : 2021-02-10T04:40:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:21 smithi164 conmon[29288]: cluster [DBG] osdmap e801: 8 total, 8 up, 7 in 2021-02-10T04:40:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:21 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:40:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:21 smithi164 conmon[29288]: 40:21.306452+0000 mon.a (mon.0) 1742 : cluster 2021-02-10T04:40:21.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:21 smithi164 conmon[29288]: [DBG] osdmap e802: 8 total, 8 up, 7 in 2021-02-10T04:40:21.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[35961]: cluster 2021-02-10T04:40:20.189115+0000 osd.4 (osd.4 2021-02-10T04:40:21.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[35961]: ) 1295 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:21.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[35961]: cluster 2021-02-10T04:40:20.189145+0000 2021-02-10T04:40:21.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[35961]: osd.4 (osd.4) 1296 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:21.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:40:21.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[35961]: -10T04:40:20.189152+0000 osd.4 (osd.4) 1297 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:21.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[35961]: cluster 2021-02-10T04:40 2021-02-10T04:40:21.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[35961]: :20.189157+0000 osd.4 (osd.4) 1298 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:21.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:40:21.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[35961]: T04:40:20.303361+0000 2021-02-10T04:40:21.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[35961]: mon.a (mon.0) 1741 : cluster [DBG] 2021-02-10T04:40:21.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[35961]: osdmap e801: 8 total, 8 up, 7 in 2021-02-10T04:40:21.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[35961]: cluster 2021-02-10T04:40:21.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[35961]: 2021-02-10T04:40:21.306452+0000 mon.a (mon.0 2021-02-10T04:40:21.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[35961]: ) 1742 : cluster [DBG] osdmap e802: 8 total, 8 up, 7 in 2021-02-10T04:40:21.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[29319]: cluster 2021-02-10T04:40:21.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:40:21.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[29319]: :40:20.189115+0000 osd.4 (osd.4) 1295 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:21.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:40:21.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[29319]: -10T04:40:20.189145+0000 osd.4 (osd 2021-02-10T04:40:21.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[29319]: .4) 1296 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:21.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[29319]: cluster 2021-02-10T04:40:21.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[29319]: 2021-02-10T04:40:21.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[29319]: 2021 2021-02-10T04:40:21.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[29319]: -02-10T04:40:20.189152+0000 2021-02-10T04:40:21.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[29319]: osd.4 (osd.4) 1297 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:21.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:40:21.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[29319]: 10T04:40:20.189157+0000 osd.4 (osd 2021-02-10T04:40:21.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[29319]: .4) 1298 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:21.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:40:21.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[29319]: -02-10T04:40:20.303361+0000 2021-02-10T04:40:21.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[29319]: mon.a (mon.0) 1741 : cluster [DBG] 2021-02-10T04:40:21.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[29319]: osdmap e801: 8 total, 8 up, 7 in 2021-02-10T04:40:21.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[29319]: cluster 2021-02-10T04:40 2021-02-10T04:40:21.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[29319]: :21.306452+0000 mon.a (mon.0) 1742 : 2021-02-10T04:40:21.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:21 smithi085 conmon[29319]: cluster [DBG] osdmap e802: 8 total, 8 up, 7 in 2021-02-10T04:40:22.307 INFO:tasks.rados.rados.0.smithi085.stdout:3160: done (4 left) 2021-02-10T04:40:22.307 INFO:tasks.rados.rados.0.smithi085.stdout:3161: snap_remove snap 363 2021-02-10T04:40:22.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:22 smithi164 conmon[33677]: debug 2021-02-10T04:40:22.218+0000 7fbec19a1700 -1 osd.4 802 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:22.832 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:22 smithi164 conmon[29288]: cluster 2021-02-10T04:40:21.215256+0000 osd.4 (osd.4 2021-02-10T04:40:22.833 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:22 smithi164 conmon[29288]: ) 1299 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:22.833 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:22 smithi164 conmon[29288]: cluster 2021-02-10T04:40:21.215265+0000 osd.4 (osd.4) 1300 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:22.833 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:22 smithi164 conmon[29288]: cluster 2021-02-10T04:40:21.215277 2021-02-10T04:40:22.833 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:22 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1301 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:22.833 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:22 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:40:22.833 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:22 smithi164 conmon[29288]: 02-10T04:40:21.215283+0000 osd.4 (osd.4) 1302 : 2021-02-10T04:40:22.833 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:22 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:22.833 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:22 smithi164 conmon[29288]: cluster 2021-02-10T04:40:22. 2021-02-10T04:40:22.833 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:22 smithi164 conmon[29288]: 192405+0000 mgr.y (mgr.14140) 783 : cluster [DBG] 2021-02-10T04:40:22.833 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:22 smithi164 conmon[29288]: pgmap v1354: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 529 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 2.6 MiB/s wr, 15 op/s 2021-02-10T04:40:22.833 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:22 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:40:22.833 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:22 smithi164 conmon[29288]: 40:22.306926+0000 mon.a (mon 2021-02-10T04:40:22.833 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:22 smithi164 conmon[29288]: .0) 1743 : cluster [DBG] osdmap e803: 8 total, 8 up, 7 in 2021-02-10T04:40:22.834 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[35961]: cluster 2021-02-10T04:40:21.215256+0000 2021-02-10T04:40:22.834 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[35961]: osd.4 (osd.4) 1299 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:22.834 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:40:22.834 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[35961]: :40:21.215265+0000 osd.4 2021-02-10T04:40:22.835 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[35961]: (osd.4) 1300 : cluster 2021-02-10T04:40:22.835 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:22.835 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[35961]: cluster 2021-02-10T04:40: 2021-02-10T04:40:22.835 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[35961]: 21.215277+0000 osd.4 (osd.4) 1301 : cluster 2021-02-10T04:40:22.835 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:22.835 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:40:22.835 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[35961]: 40:21.215283+0000 osd.4 (osd.4 2021-02-10T04:40:22.835 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[35961]: ) 1302 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:22.835 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:40:22.835 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[35961]: T04:40:22.192405+0000 mgr.y (mgr 2021-02-10T04:40:22.835 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[35961]: .14140) 783 : cluster [DBG] pgmap v1354: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 529 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 2.6 MiB/s wr, 15 op/s 2021-02-10T04:40:22.835 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:40:22.835 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[35961]: T04:40:22.306926+0000 mon.a (mon.0) 1743 : cluster [DBG] osdmap e803: 8 total, 8 up, 7 in 2021-02-10T04:40:22.836 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[29319]: cluster 2021-02-10T04:40:22.836 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[29319]: 2021-02-10T04:40:22.836 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[29319]: 2021 2021-02-10T04:40:22.836 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[29319]: - 2021-02-10T04:40:22.836 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[29319]: 02 2021-02-10T04:40:22.836 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[29319]: -10T04:40: 2021-02-10T04:40:22.836 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[29319]: 21.215256+0000 osd.4 (osd.4) 1299 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:22.836 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:40:22.836 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[29319]: 02-10T04:40:21.215265+0000 osd.4 (osd. 2021-02-10T04:40:22.836 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[29319]: 4) 1300 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:22.836 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[29319]: cluster 2021-02-10T04:40:22.836 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[29319]: 2021-02-10T04:40:21.215277+0000 2021-02-10T04:40:22.837 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[29319]: osd.4 (osd.4) 1301 : cluster [WRN] 2021-02-10T04:40:22.837 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:22.837 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:40:22.837 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[29319]: 04:40:21.215283+0000 osd.4 (osd. 2021-02-10T04:40:22.837 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[29319]: 4) 1302 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:22.837 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:40:22.837 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[29319]: -02-10T04:40:22.192405+0000 2021-02-10T04:40:22.837 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[29319]: mgr.y (mgr.14140) 783 : cluster [DBG] 2021-02-10T04:40:22.837 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[29319]: pgmap v1354: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 529 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 2.6 MiB/s wr, 15 op/s 2021-02-10T04:40:22.837 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[29319]: cluster 2021-02-10T04:40:22.837 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[29319]: 2021-02-10T04:40:22.306926 2021-02-10T04:40:22.837 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1743 : cluster 2021-02-10T04:40:22.837 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:22 smithi085 conmon[29319]: [DBG] osdmap e803: 8 total, 8 up, 7 in 2021-02-10T04:40:23.356 INFO:tasks.rados.rados.0.smithi085.stdout:3161: done (4 left) 2021-02-10T04:40:23.357 INFO:tasks.rados.rados.0.smithi085.stdout:3162: read oid 10 snap -1 2021-02-10T04:40:23.357 INFO:tasks.rados.rados.0.smithi085.stdout:3162: expect (ObjNum 1063 snap 349 seq_num 1063) 2021-02-10T04:40:23.357 INFO:tasks.rados.rados.0.smithi085.stdout:3163: rollback oid 18 current snap is 364 2021-02-10T04:40:23.357 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 18 to 325 2021-02-10T04:40:23.357 INFO:tasks.rados.rados.0.smithi085.stdout:3164: delete oid 16 current snap is 364 2021-02-10T04:40:23.379 INFO:tasks.rados.rados.0.smithi085.stdout:3162: expect (ObjNum 1063 snap 349 seq_num 1063) 2021-02-10T04:40:23.590 INFO:tasks.rados.rados.0.smithi085.stdout:3162: done (6 left) 2021-02-10T04:40:23.590 INFO:tasks.rados.rados.0.smithi085.stdout:3164: done (5 left) 2021-02-10T04:40:23.590 INFO:tasks.rados.rados.0.smithi085.stdout:3165: read oid 9 snap -1 2021-02-10T04:40:23.590 INFO:tasks.rados.rados.0.smithi085.stdout:3165: expect (ObjNum 1053 snap 346 seq_num 1053) 2021-02-10T04:40:23.590 INFO:tasks.rados.rados.0.smithi085.stdout:3166: snap_remove snap 355 2021-02-10T04:40:23.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:23 smithi164 conmon[33677]: debug 2021-02-10T04:40:23.234+0000 7fbec19a1700 -1 osd.4 803 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:23 smithi164 conmon[29288]: cluster 2021-02-10T04:40:22.219081+0000 osd.4 (osd. 2021-02-10T04:40:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:23 smithi164 conmon[29288]: 4) 1303 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:23 smithi164 conmon[29288]: cluster 2021-02-10T04:40:22.219093+0000 osd.4 (osd.4) 1304 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:23 smithi164 conmon[29288]: cluster 2021-02-10T04:40:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:23 smithi164 conmon[29288]: 2021-02-10T04:40:22.219099+0000 osd.4 (osd.4) 1305 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:23 smithi164 conmon[29288]: cluster 2021-02-10T04:40:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:23 smithi164 conmon[29288]: 2021-02-10T04:40:22.219106 2021-02-10T04:40:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:23 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1306 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:23.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:23 smithi164 conmon[29288]: 2021-02-10T04:40:23.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:23 smithi085 conmon[29319]: cluster 2021-02-10T04:40:22.219081+0000 osd.4 (osd 2021-02-10T04:40:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:23 smithi085 conmon[29319]: .4) 1303 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:23 smithi085 conmon[29319]: cluster 2021-02-10T04:40:22.219093+0000 osd.4 (osd.4) 1304 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:23 smithi085 conmon[29319]: cluster 2021-02-10T04:40:22.219099+0000 osd.4 (osd.4) 1305 : cluster 2021-02-10T04:40:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:23 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:23 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:40:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:23 smithi085 conmon[29319]: 02-10T04:40:22.219106+0000 osd.4 (osd.4) 1306 2021-02-10T04:40:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:23 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:23.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:23 smithi085 conmon[35961]: cluster 2021-02-10T04:40:22.219081+0000 osd.4 ( 2021-02-10T04:40:23.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:23 smithi085 conmon[35961]: osd.4) 1303 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:23 smithi085 conmon[35961]: cluster 2021-02-10T04:40:22.219093+0000 osd.4 2021-02-10T04:40:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:23 smithi085 conmon[35961]: (osd.4) 1304 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:23 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:40:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:23 smithi085 conmon[35961]: -10T04:40:22.219099+0000 osd.4 (osd.4) 1305 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:23 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:40:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:23 smithi085 conmon[35961]: 10T04:40:22.219106+0000 osd.4 (osd.4 2021-02-10T04:40:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:23 smithi085 conmon[35961]: ) 1306 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:23.894 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:40:24.375 INFO:tasks.rados.rados.0.smithi085.stdout:3166: done (6 left) 2021-02-10T04:40:24.375 INFO:tasks.rados.rados.0.smithi085.stdout:3167: snap_remove snap 361 2021-02-10T04:40:24.379 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:40:24.542 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:24 smithi164 conmon[33677]: debug 2021-02-10T04:40:24.230+0000 7fbec19a1700 -1 osd.4 804 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:24.542 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:40:24 smithi164 conmon[40892]: debug 2021-02-10T04:40:24.542 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:40:24 smithi164 conmon[40892]: 2021-02-10T04:40:24.539+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:40:24.542 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:40:24 smithi164 conmon[40892]: 2021-02-10T04:40:24.543 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:24 smithi164 conmon[29288]: cluster 2021-02-10T04:40:23.224751+0000 osd.6 (osd. 2021-02-10T04:40:24.543 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:24 smithi164 conmon[29288]: 6) 80 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:40:24.543 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:24 smithi164 conmon[29288]: cluster 2021-02-10T04:40:23.226078+0000 osd.6 (osd.6) 81 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:40:24.543 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:24 smithi164 conmon[29288]: cluster 2021-02-10T04:40:23.235662+0000 osd.4 (osd.4) 1307 : 2021-02-10T04:40:24.543 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:24 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:24.543 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:24 smithi164 conmon[29288]: cluster 2021-02-10T04:40:23 2021-02-10T04:40:24.543 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:24 smithi164 conmon[29288]: .235671+0000 osd.4 (osd.4) 1308 : cluster [WRN] 2021-02-10T04:40:24.543 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:24 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:24.543 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:24 smithi164 conmon[29288]: cluster 2021-02-10T04:40:24.543 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:24 smithi164 conmon[29288]: 2021-02-10T04:40:23.235681+0000 osd.4 ( 2021-02-10T04:40:24.543 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:24 smithi164 conmon[29288]: osd.4) 1309 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:24.543 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:24 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:40:24.544 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:24 smithi164 conmon[29288]: T04:40:23.235688+0000 osd.4 (osd. 2021-02-10T04:40:24.544 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:24 smithi164 conmon[29288]: 4) 1310 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:24.544 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:24 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:40:24.544 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:24 smithi164 conmon[29288]: 04:40:23.354933+0000 mon.a (mon. 2021-02-10T04:40:24.544 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:24 smithi164 conmon[29288]: 0) 1744 : cluster [DBG] osdmap e804: 8 total, 8 up, 7 in 2021-02-10T04:40:24.544 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:24 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:40:24.544 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:24 smithi164 conmon[29288]: 04:40:24.193018+0000 mgr.y (mgr.14140) 2021-02-10T04:40:24.544 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:24 smithi164 conmon[29288]: 784 : cluster [DBG] pgmap v1357: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 527 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 5 op/s 2021-02-10T04:40:24.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[35961]: cluster 2021-02-10T04:40:23.224751 2021-02-10T04:40:24.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[35961]: +0000 osd.6 (osd.6) 80 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:40:24.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[35961]: cluster 2021-02-10T04:40:23.226078+0000 2021-02-10T04:40:24.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[35961]: osd.6 (osd.6) 81 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:40:24.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:40:24.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[35961]: -02-10T04:40:23.235662+0000 osd.4 (osd. 2021-02-10T04:40:24.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[35961]: 4) 1307 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:24.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:40:24.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[35961]: -10T04:40:23.235671+0000 osd.4 (osd 2021-02-10T04:40:24.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[35961]: .4) 1308 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:24.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[35961]: cluster 2021-02-10T04:40:24.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[35961]: 2021-02-10T04:40:24.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[35961]: 2021 2021-02-10T04:40:24.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[35961]: - 2021-02-10T04:40:24.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[35961]: 02 2021-02-10T04:40:24.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[35961]: - 2021-02-10T04:40:24.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[35961]: 10T04:40:23.235681+0000 2021-02-10T04:40:24.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[35961]: osd.4 (osd.4) 1309 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:24.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:40:24.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[35961]: -10T04:40:23.235688+0000 osd.4 ( 2021-02-10T04:40:24.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[35961]: osd.4) 1310 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:24.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:40:24.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[35961]: -02-10T04:40:23.354933+0000 mon.a 2021-02-10T04:40:24.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[35961]: (mon.0) 1744 : cluster [DBG] osdmap e804: 8 total, 8 up, 7 in 2021-02-10T04:40:24.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:40:24.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[35961]: 10T04:40:24.193018+0000 mgr.y (mgr 2021-02-10T04:40:24.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[35961]: .14140) 784 : cluster [DBG] pgmap v1357: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 527 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 5 op/s 2021-02-10T04:40:24.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[29319]: cluster 2021-02-10T04:40:23.224751+0000 osd.6 (osd.6) 2021-02-10T04:40:24.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[29319]: 80 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:40:24.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[29319]: cluster 2021-02-10T04:40:23.226078+0000 osd.6 (osd.6) 81 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:40:24.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[29319]: cluster 2021-02-10T04:40:23.235662+0000 osd.4 (osd.4) 1307 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:24.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[29319]: cluster 2021-02-10T04:40:24.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[29319]: 2021-02-10T04:40:23.235671+0000 osd.4 (osd.4) 1308 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:24.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[29319]: cluster 2021-02-10T04:40: 2021-02-10T04:40:24.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[29319]: 23.235681+0000 osd.4 (osd.4) 1309 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:24.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:40:24.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[29319]: 04:40:23.235688+0000 osd.4 (osd.4) 2021-02-10T04:40:24.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[29319]: 1310 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:24.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[29319]: cluster 2021-02-10T04:40:24.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[29319]: 2021 2021-02-10T04:40:24.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[29319]: - 2021-02-10T04:40:24.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[29319]: 02-10T04:40:23.354933+0000 mon.a (mon.0 2021-02-10T04:40:24.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[29319]: ) 1744 : cluster [DBG] osdmap e804: 8 total, 8 up, 7 in 2021-02-10T04:40:24.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:40:24.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[29319]: :40:24.193018+0000 mgr.y (mgr.14140) 784 2021-02-10T04:40:24.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:24 smithi085 conmon[29319]: : cluster [DBG] pgmap v1357: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 527 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 5 op/s 2021-02-10T04:40:24.906 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:40:25.013 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:40:25.050 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:40:25.380 INFO:tasks.rados.rados.0.smithi085.stdout:3167: done (6 left) 2021-02-10T04:40:25.380 INFO:tasks.rados.rados.0.smithi085.stdout:3168: write (excl) oid 46 current snap is 364 2021-02-10T04:40:25.380 INFO:tasks.rados.rados.0.smithi085.stdout:3168: seq_num 1087 ranges {669209=717789,1942162=327047} 2021-02-10T04:40:25.388 INFO:tasks.rados.rados.0.smithi085.stdout:3168: writing smithi08575397-46 from 669209 to 1386998 tid 1 2021-02-10T04:40:25.394 INFO:tasks.rados.rados.0.smithi085.stdout:3168: writing smithi08575397-46 from 1942162 to 2269209 tid 2 2021-02-10T04:40:25.394 INFO:tasks.rados.rados.0.smithi085.stdout:3169: delete oid 26 current snap is 364 2021-02-10T04:40:25.394 INFO:tasks.rados.rados.0.smithi085.stdout:3163: finishing rollback tid 0 to smithi08575397-18 2021-02-10T04:40:25.395 INFO:tasks.rados.rados.0.smithi085.stdout:3163: finishing rollback tid 1 to smithi08575397-18 2021-02-10T04:40:25.395 INFO:tasks.rados.rados.0.smithi085.stdout:3163: finishing rollback tid 2 to smithi08575397-18 2021-02-10T04:40:25.395 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 18 v 1299 (ObjNum 952 snap 313 seq_num 952) dirty exists 2021-02-10T04:40:25.395 INFO:tasks.rados.rados.0.smithi085.stdout:3165: expect (ObjNum 1053 snap 346 seq_num 1053) 2021-02-10T04:40:25.646 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[33677]: debug 2021-02-10T04:40:25.251+0000 7fbec19a1700 -1 osd.4 805 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:25.647 INFO:tasks.rados.rados.0.smithi085.stdout:3168: finishing write tid 1 to smithi08575397-46 2021-02-10T04:40:25.647 INFO:tasks.rados.rados.0.smithi085.stdout:3168: finishing write tid 2 to smithi08575397-46 2021-02-10T04:40:25.648 INFO:tasks.rados.rados.0.smithi085.stdout:3163: done (7 left) 2021-02-10T04:40:25.648 INFO:tasks.rados.rados.0.smithi085.stdout:3165: done (6 left) 2021-02-10T04:40:25.648 INFO:tasks.rados.rados.0.smithi085.stdout:3169: done (5 left) 2021-02-10T04:40:25.648 INFO:tasks.rados.rados.0.smithi085.stdout:3170: read oid 50 snap -1 2021-02-10T04:40:25.648 INFO:tasks.rados.rados.0.smithi085.stdout:3170: expect (ObjNum 1054 snap 346 seq_num 1054) 2021-02-10T04:40:25.648 INFO:tasks.rados.rados.0.smithi085.stdout:3171: write (excl) oid 49 current snap is 364 2021-02-10T04:40:25.648 INFO:tasks.rados.rados.0.smithi085.stdout:3168: finishing write tid 3 to smithi08575397-46 2021-02-10T04:40:25.648 INFO:tasks.rados.rados.0.smithi085.stdout:3168: finishing write tid 4 to smithi08575397-46 2021-02-10T04:40:25.648 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 1552 (ObjNum 1087 snap 364 seq_num 1087) dirty exists 2021-02-10T04:40:25.648 INFO:tasks.rados.rados.0.smithi085.stdout:3168: left oid 46 (ObjNum 1087 snap 364 seq_num 1087) 2021-02-10T04:40:25.648 INFO:tasks.rados.rados.0.smithi085.stdout:3171: seq_num 1089 ranges {702823=584640,2059126=243697} 2021-02-10T04:40:25.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: cluster 2021-02-10T04:40:25.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: 2021-02-10T04:40:25.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: 2021 2021-02-10T04:40:25.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: - 2021-02-10T04:40:25.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: 02 2021-02-10T04:40:25.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: - 2021-02-10T04:40:25.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: 10 2021-02-10T04:40:25.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: T 2021-02-10T04:40:25.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: 04 2021-02-10T04:40:25.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: : 2021-02-10T04:40:25.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: 40 2021-02-10T04:40:25.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: : 2021-02-10T04:40:25.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: 24 2021-02-10T04:40:25.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: . 2021-02-10T04:40:25.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: 231632 2021-02-10T04:40:25.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: +0000 2021-02-10T04:40:25.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: 2021-02-10T04:40:25.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: osd.4 2021-02-10T04:40:25.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: ( 2021-02-10T04:40:25.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: osd 2021-02-10T04:40:25.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: . 2021-02-10T04:40:25.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: 4 2021-02-10T04:40:25.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: ) 2021-02-10T04:40:25.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: 1311 2021-02-10T04:40:25.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: : 2021-02-10T04:40:25.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: cluster 2021-02-10T04:40:25.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: 2021-02-10T04:40:25.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: [WRN] 2021-02-10T04:40:25.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: 2021-02-10T04:40:25.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:25.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: 2021-02-10T04:40:25.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: cluster 2021-02-10T04:40:25.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: 2021-02-10T04:40:25.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: 2021 2021-02-10T04:40:25.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: - 2021-02-10T04:40:25.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: 02 2021-02-10T04:40:25.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: -10T04:40:24.231642+0000 2021-02-10T04:40:25.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: osd.4 (osd.4) 1312 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:25.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: cluster 2021-02-10T04:40:25.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: 2021-02-10T04:40 2021-02-10T04:40:25.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: :24.231652+0000 osd.4 (osd.4) 1313 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:25.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: cluster 2021-02-10T04:40:25.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:40:25.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: :40:24.231662+0000 osd.4 (osd.4 2021-02-10T04:40:25.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: ) 1314 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:25.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: cluster 2021-02-10T04:40:25.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: 2021-02-10T04:40:24.375433+0000 2021-02-10T04:40:25.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:25 smithi164 conmon[29288]: mon.a (mon.0) 1745 : cluster [DBG] osdmap e805: 8 total, 8 up, 7 in 2021-02-10T04:40:25.655 INFO:tasks.rados.rados.0.smithi085.stdout:3171: writing smithi08575397-49 from 702823 to 1287463 tid 1 2021-02-10T04:40:25.661 INFO:tasks.rados.rados.0.smithi085.stdout:3171: writing smithi08575397-49 from 2059126 to 2302823 tid 2 2021-02-10T04:40:25.662 INFO:tasks.rados.rados.0.smithi085.stdout:3170: expect (ObjNum 1054 snap 346 seq_num 1054) 2021-02-10T04:40:25.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:25 smithi085 conmon[29319]: cluster 2021-02-10T04:40:24.231632 2021-02-10T04:40:25.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:25 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1311 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:25.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:25 smithi085 conmon[29319]: cluster 2021-02-10T04:40:24.231642+0000 osd.4 (osd.4) 1312 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:25.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:25 smithi085 conmon[29319]: cluster 2021-02-10T04:40:24.231652+0000 osd.4 (osd.4) 1313 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:25.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:25 smithi085 conmon[29319]: cluster 2021-02-10T04:40:24.231662+0000 osd.4 (osd.4) 1314 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:25.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:25 smithi085 conmon[29319]: cluster 2021-02-10T04:40:24.375433+0000 mon.a (mon.0) 1745 : cluster [DBG] osdmap e805: 8 total, 8 up, 7 in 2021-02-10T04:40:25.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:25 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:40:25.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:25 smithi085 conmon[35961]: T04:40:24.231632+0000 osd.4 (osd.4) 1311 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:25.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:25 smithi085 conmon[35961]: cluster 2021-02-10T04:40:24.231642+0000 osd.4 (osd.4) 1312 : cluster 2021-02-10T04:40:25.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:25 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:25.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:25 smithi085 conmon[35961]: cluster 2021-02-10T04:40:24.231652 2021-02-10T04:40:25.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:25 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1313 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:25.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:25 smithi085 conmon[35961]: cluster 2021-02-10T04:40:25.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:25 smithi085 conmon[35961]: 2021-02-10T04:40:24.231662 2021-02-10T04:40:25.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:25 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1314 : cluster [WRN] 2021-02-10T04:40:25.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:25 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:25.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:25 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:40:25.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:25 smithi085 conmon[35961]: 04:40:24.375433+0000 mon.a (mon. 2021-02-10T04:40:25.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:25 smithi085 conmon[35961]: 0) 1745 : cluster [DBG] osdmap e805: 8 total, 8 up, 7 in 2021-02-10T04:40:25.801 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.2 dump_ops_in_flight 2021-02-10T04:40:26.396 INFO:tasks.rados.rados.0.smithi085.stdout:3171: finishing write tid 1 to smithi08575397-49 2021-02-10T04:40:26.396 INFO:tasks.rados.rados.0.smithi085.stdout:3171: finishing write tid 2 to smithi08575397-49 2021-02-10T04:40:26.396 INFO:tasks.rados.rados.0.smithi085.stdout:3171: finishing write tid 3 to smithi08575397-49 2021-02-10T04:40:26.396 INFO:tasks.rados.rados.0.smithi085.stdout:3168: done (6 left) 2021-02-10T04:40:26.396 INFO:tasks.rados.rados.0.smithi085.stdout:3170: done (5 left) 2021-02-10T04:40:26.396 INFO:tasks.rados.rados.0.smithi085.stdout:3172: read oid 1 snap -1 2021-02-10T04:40:26.396 INFO:tasks.rados.rados.0.smithi085.stdout:3172: expect (ObjNum 1077 snap 359 seq_num 1077) 2021-02-10T04:40:26.397 INFO:tasks.rados.rados.0.smithi085.stdout:3173: copy_from oid 21 from oid 44 current snap is 364 2021-02-10T04:40:26.397 INFO:tasks.rados.rados.0.smithi085.stdout:3171: finishing write tid 4 to smithi08575397-49 2021-02-10T04:40:26.397 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 49 v 1474 (ObjNum 1089 snap 364 seq_num 1089) dirty exists 2021-02-10T04:40:26.397 INFO:tasks.rados.rados.0.smithi085.stdout:3171: left oid 49 (ObjNum 1089 snap 364 seq_num 1089) 2021-02-10T04:40:26.397 INFO:tasks.rados.rados.0.smithi085.stdout:3171: done (6 left) 2021-02-10T04:40:26.397 INFO:tasks.rados.rados.0.smithi085.stdout:3174: delete oid 23 current snap is 364 2021-02-10T04:40:26.397 INFO:tasks.rados.rados.0.smithi085.stdout:3174: done (6 left) 2021-02-10T04:40:26.397 INFO:tasks.rados.rados.0.smithi085.stdout:3175: read oid 14 snap -1 2021-02-10T04:40:26.397 INFO:tasks.rados.rados.0.smithi085.stdout:3175: expect deleted 2021-02-10T04:40:26.398 INFO:tasks.rados.rados.0.smithi085.stdout:3176: copy_from oid 35 from oid 26 current snap is 364 2021-02-10T04:40:26.398 INFO:tasks.rados.rados.0.smithi085.stdout:3177: write oid 29 current snap is 364 2021-02-10T04:40:26.399 INFO:tasks.rados.rados.0.smithi085.stdout:3177: seq_num 1091 ranges {736437=735138,2176090=160347} 2021-02-10T04:40:26.399 INFO:tasks.rados.rados.0.smithi085.stdout:3177: writing smithi08575397-29 from 736437 to 1471575 tid 1 2021-02-10T04:40:26.399 INFO:tasks.rados.rados.0.smithi085.stdout:3177: writing smithi08575397-29 from 2176090 to 2336437 tid 2 2021-02-10T04:40:26.400 INFO:tasks.rados.rados.0.smithi085.stdout:3178: rollback oid 40 current snap is 364 2021-02-10T04:40:26.400 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 40 to 278 2021-02-10T04:40:26.400 INFO:tasks.rados.rados.0.smithi085.stdout:3179: rollback oid 36 current snap is 364 2021-02-10T04:40:26.402 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 36 to 343 2021-02-10T04:40:26.402 INFO:tasks.rados.rados.0.smithi085.stdout:3176: finishing copy_from to smithi08575397-35 2021-02-10T04:40:26.402 INFO:tasks.rados.rados.0.smithi085.stdout:3176: got expected ENOENT (src dne) 2021-02-10T04:40:26.403 INFO:tasks.rados.rados.0.smithi085.stdout:3176: finishing copy_from racing read to smithi08575397-35 2021-02-10T04:40:26.403 INFO:tasks.rados.rados.0.smithi085.stdout:3172: expect (ObjNum 1077 snap 359 seq_num 1077) 2021-02-10T04:40:26.403 INFO:tasks.rados.rados.0.smithi085.stdout:3172: done (10 left) 2021-02-10T04:40:26.403 INFO:tasks.rados.rados.0.smithi085.stdout:3175: done (9 left) 2021-02-10T04:40:26.404 INFO:tasks.rados.rados.0.smithi085.stdout:3176: done (8 left) 2021-02-10T04:40:26.404 INFO:tasks.rados.rados.0.smithi085.stdout:3180: read oid 32 snap -1 2021-02-10T04:40:26.404 INFO:tasks.rados.rados.0.smithi085.stdout:3180: expect (ObjNum 1082 snap 359 seq_num 1082) 2021-02-10T04:40:26.404 INFO:tasks.rados.rados.0.smithi085.stdout:3181: snap_create 2021-02-10T04:40:26.405 INFO:tasks.rados.rados.0.smithi085.stdout:3179: finishing rollback tid 0 to smithi08575397-36 2021-02-10T04:40:26.405 INFO:tasks.rados.rados.0.smithi085.stdout:3179: finishing rollback tid 1 to smithi08575397-36 2021-02-10T04:40:26.405 INFO:tasks.rados.rados.0.smithi085.stdout:3179: finishing rollback tid 2 to smithi08575397-36 2021-02-10T04:40:26.406 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 1302 (ObjNum 991 snap 326 seq_num 991) dirty exists 2021-02-10T04:40:26.406 INFO:tasks.rados.rados.0.smithi085.stdout:3177: finishing write tid 1 to smithi08575397-29 2021-02-10T04:40:26.407 INFO:tasks.rados.rados.0.smithi085.stdout:3177: finishing write tid 2 to smithi08575397-29 2021-02-10T04:40:26.407 INFO:tasks.rados.rados.0.smithi085.stdout:3177: finishing write tid 3 to smithi08575397-29 2021-02-10T04:40:26.407 INFO:tasks.rados.rados.0.smithi085.stdout:3177: finishing write tid 4 to smithi08575397-29 2021-02-10T04:40:26.408 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 29 v 1458 (ObjNum 1091 snap 364 seq_num 1091) dirty exists 2021-02-10T04:40:26.408 INFO:tasks.rados.rados.0.smithi085.stdout:3177: left oid 29 (ObjNum 1091 snap 364 seq_num 1091) 2021-02-10T04:40:26.408 INFO:tasks.rados.rados.0.smithi085.stdout:3178: finishing rollback tid 1 to smithi08575397-40 2021-02-10T04:40:26.409 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 1631 (ObjNum 82698080 snap 22030 seq_num 83051520) dirty dne 2021-02-10T04:40:26.409 INFO:tasks.rados.rados.0.smithi085.stdout:3173: finishing copy_from to smithi08575397-21 2021-02-10T04:40:26.409 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 1439 (ObjNum 1052 snap 345 seq_num 1052) dirty exists 2021-02-10T04:40:26.409 INFO:tasks.rados.rados.0.smithi085.stdout:3173: finishing copy_from racing read to smithi08575397-21 2021-02-10T04:40:26.410 INFO:tasks.rados.rados.0.smithi085.stdout:3180: expect (ObjNum 1082 snap 359 seq_num 1082) 2021-02-10T04:40:26.410 INFO:tasks.rados.rados.0.smithi085.stdout:3173: done (9 left) 2021-02-10T04:40:26.410 INFO:tasks.rados.rados.0.smithi085.stdout:3177: done (8 left) 2021-02-10T04:40:26.410 INFO:tasks.rados.rados.0.smithi085.stdout:3178: done (7 left) 2021-02-10T04:40:26.410 INFO:tasks.rados.rados.0.smithi085.stdout:3179: done (6 left) 2021-02-10T04:40:26.411 INFO:tasks.rados.rados.0.smithi085.stdout:3180: done (5 left) 2021-02-10T04:40:26.411 INFO:tasks.rados.rados.0.smithi085.stdout:3181: done (4 left) 2021-02-10T04:40:26.411 INFO:tasks.rados.rados.0.smithi085.stdout:3182: delete oid 3 current snap is 365 2021-02-10T04:40:26.414 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 3 1 2021-02-10T04:40:26.415 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:40:26.416 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:40:25 smithi085 conmon[41540]: debug 2021-02-10T04:40:26.416 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:40:25 smithi085 conmon[41540]: 2021-02-10T04:40:25.951+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:40:26.416 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:40:25 smithi085 conmon[41540]: 2021-02-10T04:40:26.416 INFO:tasks.rados.rados.0.smithi085.stdout:3182: done (4 left) 2021-02-10T04:40:26.416 INFO:tasks.rados.rados.0.smithi085.stdout:3183: snap_create 2021-02-10T04:40:26.516 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:40:26.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:26 smithi164 conmon[29288]: cluster 2021-02-10T04:40:25.252399+0000 osd.4 (osd. 2021-02-10T04:40:26.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:26 smithi164 conmon[29288]: 4) 1315 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:26.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:26 smithi164 conmon[29288]: cluster 2021-02-10T04:40:25.252407+0000 osd.4 (osd.4) 1316 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:26.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:26 smithi164 conmon[29288]: cluster 2021-02-10T04:40:25.252411+0000 osd.4 (osd. 2021-02-10T04:40:26.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:26 smithi164 conmon[29288]: 4) 1317 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:26.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:26 smithi164 conmon[29288]: cluster 2021-02-10T04:40:26.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:26 smithi164 conmon[29288]: 2021-02-10T04:40:25.252414+0000 2021-02-10T04:40:26.557 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:26 smithi164 conmon[29288]: osd.4 (osd.4) 1318 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:26.557 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:26 smithi164 conmon[29288]: cluster 2021-02-10T04:40:25.376362+0000 mon.a (mon. 2021-02-10T04:40:26.557 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:26 smithi164 conmon[29288]: 0) 1746 : cluster [DBG] osdmap e806: 8 total, 8 up, 7 in 2021-02-10T04:40:26.557 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:26 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:40:26.557 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:26 smithi164 conmon[29288]: :40:26.193564+0000 mgr.y (mgr.14140) 785 : cluster [DBG] 2021-02-10T04:40:26.557 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:26 smithi164 conmon[29288]: pgmap v1360: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 521 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 3.4 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2021-02-10T04:40:26.557 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:26 smithi164 conmon[33677]: debug 2021-02-10T04:40:26.251+0000 7fbec19a1700 -1 osd.4 806 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:26.561 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:40:26.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[29319]: cluster 2021-02-10T04:40:25.252399+0000 osd.4 (osd.4 2021-02-10T04:40:26.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[29319]: ) 1315 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:26.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[29319]: cluster 2021-02-10T04:40:25.252407+0000 osd.4 (osd.4) 1316 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:26.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:40:26.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[29319]: 10T04:40:25.252411+0000 osd.4 (osd.4) 1317 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:26.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:40:26.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[29319]: 40:25.252414+0000 osd.4 (osd.4) 2021-02-10T04:40:26.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[29319]: 1318 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:26.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[29319]: cluster 2021-02-10T04:40: 2021-02-10T04:40:26.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[29319]: 25.376362+0000 mon.a (mon.0) 1746 : cluster 2021-02-10T04:40:26.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[29319]: [DBG] osdmap e806: 8 total, 8 up, 7 in 2021-02-10T04:40:26.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[29319]: cluster 2021-02-10T04:40:26 2021-02-10T04:40:26.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[29319]: .193564+0000 mgr.y (mgr.14140) 785 : cluster 2021-02-10T04:40:26.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[29319]: [DBG] pgmap v1360: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 521 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 3.4 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2021-02-10T04:40:26.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: cluster 2021-02-10T04:40:26.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: 2021-02-10T04:40:26.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: 2021 2021-02-10T04:40:26.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: - 2021-02-10T04:40:26.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: 02 2021-02-10T04:40:26.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: - 2021-02-10T04:40:26.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: 10 2021-02-10T04:40:26.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: T04:40:25.252399+0000 osd.4 2021-02-10T04:40:26.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: (osd.4) 1315 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:26.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:40:26.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: 02-10T04:40:25.252407+0000 osd.4 (osd. 2021-02-10T04:40:26.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: 4) 1316 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:26.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: cluster 2021-02-10T04:40:26.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: 2021-02-10T04:40:26.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: 2021 2021-02-10T04:40:26.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: - 2021-02-10T04:40:26.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: 02 2021-02-10T04:40:26.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: - 2021-02-10T04:40:26.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: 10 2021-02-10T04:40:26.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: T 2021-02-10T04:40:26.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: 04 2021-02-10T04:40:26.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: : 2021-02-10T04:40:26.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: 40 2021-02-10T04:40:26.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: : 2021-02-10T04:40:26.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: 25 2021-02-10T04:40:26.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: . 2021-02-10T04:40:26.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: 252411 2021-02-10T04:40:26.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1317 : 2021-02-10T04:40:26.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:26.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:40:26.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: -10T04:40:25.252414+0000 osd.4 (osd.4 2021-02-10T04:40:26.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: ) 1318 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:26.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: cluster 2021-02-10T04:40:26.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:40:26.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: 04:40:25.376362+0000 mon.a (mon.0) 1746 : cluster [DBG] osdmap e806: 8 total, 8 up, 7 in 2021-02-10T04:40:26.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: cluster 2021-02-10T04:40:26 2021-02-10T04:40:26.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:26 smithi085 conmon[35961]: .193564+0000 mgr.y (mgr.14140) 785 : cluster [DBG] pgmap v1360: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 521 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 3.4 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2021-02-10T04:40:26.765 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:40:26.766 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:40:26.893 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:40:26 smithi164 conmon[40892]: debug 2021-02-10T04:40:26.894 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:40:26 smithi164 conmon[40892]: 2021-02-10T04:40:26.636+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:40:26.894 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:40:26 smithi164 conmon[40892]: 2021-02-10T04:40:27.031 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:40:27.137 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:40:27.174 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:40:27.267 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[45443]: debug 2021-02-10T04:40:27.260+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:40:27.293 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:40:27.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:27 smithi164 conmon[33677]: debug 2021-02-10T04:40:27.223+0000 7fbec19a1700 -1 osd.4 807 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:27.399 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:40:27.408 INFO:tasks.rados.rados.0.smithi085.stdout:3183: done (4 left) 2021-02-10T04:40:27.409 INFO:tasks.rados.rados.0.smithi085.stdout:3184: rollback oid 22 current snap is 366 2021-02-10T04:40:27.409 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 22 to 342 2021-02-10T04:40:27.409 INFO:tasks.rados.rados.0.smithi085.stdout:3185: rollback oid 42 current snap is 366 2021-02-10T04:40:27.409 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 42 to 362 2021-02-10T04:40:27.409 INFO:tasks.rados.rados.0.smithi085.stdout:3186: snap_create 2021-02-10T04:40:27.418 INFO:tasks.rados.rados.0.smithi085.stdout:3184: finishing rollback tid 0 to smithi08575397-22 2021-02-10T04:40:27.418 INFO:tasks.rados.rados.0.smithi085.stdout:3184: finishing rollback tid 1 to smithi08575397-22 2021-02-10T04:40:27.419 INFO:tasks.rados.rados.0.smithi085.stdout:3184: finishing rollback tid 2 to smithi08575397-22 2021-02-10T04:40:27.419 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 1466 (ObjNum 966 snap 315 seq_num 966) dirty exists 2021-02-10T04:40:27.419 INFO:tasks.rados.rados.0.smithi085.stdout:3185: finishing rollback tid 0 to smithi08575397-42 2021-02-10T04:40:27.419 INFO:tasks.rados.rados.0.smithi085.stdout:3185: finishing rollback tid 1 to smithi08575397-42 2021-02-10T04:40:27.419 INFO:tasks.rados.rados.0.smithi085.stdout:3185: finishing rollback tid 2 to smithi08575397-42 2021-02-10T04:40:27.419 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 1474 (ObjNum 1081 snap 359 seq_num 1081) dirty exists 2021-02-10T04:40:27.438 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:40:27.568 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:40:27.573 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[35961]: cluster 2021-02-10T04:40:26.251862+0000 2021-02-10T04:40:27.573 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[35961]: osd.4 (osd.4) 1319 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:27.573 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[35961]: cluster 2021-02-10T04:40:27.573 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[35961]: 2021-02-10T04:40:26.251873 2021-02-10T04:40:27.574 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1320 : cluster [WRN] 2021-02-10T04:40:27.574 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:27.574 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[35961]: cluster 2021-02-10T04:40:27.574 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[35961]: 2021-02-10T04:40:27.574 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[35961]: 2021 2021-02-10T04:40:27.574 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[35961]: -02-10T04:40: 2021-02-10T04:40:27.574 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[35961]: 26.251878+0000 osd.4 (osd.4) 1321 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:27.574 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:40:27.574 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[35961]: T04:40:26.251884+0000 osd.4 (osd.4) 1322 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:27.574 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:40:27.574 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[35961]: 10T04:40:26.395165+0000 mon.a (mon.0) 1747 : cluster [DBG] osdmap e807: 8 total, 8 up, 7 in 2021-02-10T04:40:27.574 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[52836]: debug 2021-02-10T04:40:27.514+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:40:27.575 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[29319]: cluster 2021-02-10T04:40:26.251862+0000 2021-02-10T04:40:27.575 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[29319]: osd.4 (osd.4) 1319 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:27.575 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[29319]: cluster 2021-02-10T04:40:26.251873+0000 osd.4 (osd.4) 1320 : 2021-02-10T04:40:27.575 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:27.575 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[29319]: cluster 2021-02-10T04:40:26.251878 2021-02-10T04:40:27.575 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1321 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:27.575 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[29319]: cluster 2021-02-10T04:40:27.575 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[29319]: 2021-02-10T04:40:26 2021-02-10T04:40:27.575 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[29319]: .251884+0000 2021-02-10T04:40:27.575 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[29319]: osd.4 (osd.4) 1322 : cluster 2021-02-10T04:40:27.575 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:27.575 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[29319]: cluster 2021-02-10T04:40:27.576 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[29319]: 2021-02-10T04:40:26. 2021-02-10T04:40:27.576 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[29319]: 395165+0000 mon.a (mon.0) 1747 : cluster 2021-02-10T04:40:27.576 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[29319]: [DBG] osdmap e807: 8 total, 8 up, 7 in 2021-02-10T04:40:27.576 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:27 smithi085 conmon[29319]: 2021-02-10T04:40:27.674 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:40:27.706 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:27 smithi164 conmon[29288]: cluster 2021-02-10T04:40:27.707 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:27 smithi164 conmon[29288]: 2021-02-10T04:40:27.707 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:27 smithi164 conmon[29288]: 2021-02-10T04:40:26.251862 2021-02-10T04:40:27.707 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:27 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1319 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:27.707 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:27 smithi164 conmon[29288]: cluster 2021-02-10T04:40:26 2021-02-10T04:40:27.707 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:27 smithi164 conmon[29288]: .251873+0000 osd.4 (osd.4) 1320 : cluster [WRN] 2021-02-10T04:40:27.707 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:27 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:27.707 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:27 smithi164 conmon[29288]: cluster 2021-02-10T04:40:26.251878 2021-02-10T04:40:27.707 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:27 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1321 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:27.707 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:27 smithi164 conmon[29288]: 2021-02-10T04:40:27.707 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:27 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:40:27.707 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:27 smithi164 conmon[29288]: 02-10T04:40:26.251884+0000 osd.4 (osd.4) 1322 : cluster 2021-02-10T04:40:27.707 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:27 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:27.708 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:27 smithi164 conmon[29288]: cluster 2021-02-10T04:40:27.708 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:27 smithi164 conmon[29288]: 2021-02-10T04:40:26.395165+0000 mon.a ( 2021-02-10T04:40:27.708 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:27 smithi164 conmon[29288]: mon.0) 1747 : cluster [DBG] osdmap e807: 8 total, 8 up, 7 in 2021-02-10T04:40:27.709 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:40:28.132 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:40:28.137 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:40:27 smithi164 conmon[44544]: debug 2021-02-10T04:40:27.811+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:40:28.238 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:40:28.270 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:40:28.389 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:28 smithi164 conmon[33677]: debug 2021-02-10T04:40:28.389 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:28 smithi164 conmon[33677]: 2021-02-10T04:40:28.259+0000 7fbec19a1700 -1 osd.4 808 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:28.389 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:28 smithi164 conmon[33677]: 2021-02-10T04:40:28.414 INFO:tasks.rados.rados.0.smithi085.stdout:3184: done (6 left) 2021-02-10T04:40:28.414 INFO:tasks.rados.rados.0.smithi085.stdout:3185: done (5 left) 2021-02-10T04:40:28.414 INFO:tasks.rados.rados.0.smithi085.stdout:3186: done (4 left) 2021-02-10T04:40:28.414 INFO:tasks.rados.rados.0.smithi085.stdout:3187: rollback oid 10 current snap is 367 2021-02-10T04:40:28.414 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 10 to 303 2021-02-10T04:40:28.414 INFO:tasks.rados.rados.0.smithi085.stdout:3188: delete oid 21 current snap is 367 2021-02-10T04:40:28.416 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:40:28.416 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset noscrub 2021-02-10T04:40:28.420 INFO:tasks.rados.rados.0.smithi085.stdout:3187: finishing rollback tid 0 to smithi08575397-10 2021-02-10T04:40:28.420 INFO:tasks.rados.rados.0.smithi085.stdout:3188: done (5 left) 2021-02-10T04:40:28.420 INFO:tasks.rados.rados.0.smithi085.stdout:3189: read oid 34 snap -1 2021-02-10T04:40:28.420 INFO:tasks.rados.rados.0.smithi085.stdout:3189: expect (ObjNum 1052 snap 345 seq_num 1052) 2021-02-10T04:40:28.420 INFO:tasks.rados.rados.0.smithi085.stdout:3190: read oid 3 snap -1 2021-02-10T04:40:28.420 INFO:tasks.rados.rados.0.smithi085.stdout:3190: expect deleted 2021-02-10T04:40:28.421 INFO:tasks.rados.rados.0.smithi085.stdout:3191: delete oid 30 current snap is 367 2021-02-10T04:40:28.421 INFO:tasks.rados.rados.0.smithi085.stdout:3187: finishing rollback tid 1 to smithi08575397-10 2021-02-10T04:40:28.421 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 10 v 1476 (ObjNum 1953066355 snap 942696808 seq_num 859125557) dirty dne 2021-02-10T04:40:28.424 INFO:tasks.rados.rados.0.smithi085.stdout:3187: done (7 left) 2021-02-10T04:40:28.424 INFO:tasks.rados.rados.0.smithi085.stdout:3190: done (6 left) 2021-02-10T04:40:28.424 INFO:tasks.rados.rados.0.smithi085.stdout:3191: done (5 left) 2021-02-10T04:40:28.424 INFO:tasks.rados.rados.0.smithi085.stdout:3192: rollback oid 6 current snap is 367 2021-02-10T04:40:28.424 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 6 to 337 2021-02-10T04:40:28.424 INFO:tasks.rados.rados.0.smithi085.stdout:3193: snap_remove snap 340 2021-02-10T04:40:28.555 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:40:28.645 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:40:28 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:40:28.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:28 smithi164 conmon[29288]: cluster 2021-02-10T04:40:27.223988 2021-02-10T04:40:28.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:28 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1323 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:28.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:28 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:40:28.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:28 smithi164 conmon[29288]: :40:27.223994+0000 osd.4 (osd.4) 1324 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:28.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:28 smithi164 conmon[29288]: cluster 2021-02-10T04:40:27 2021-02-10T04:40:28.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:28 smithi164 conmon[29288]: .223998+0000 osd.4 (osd.4) 1325 : cluster 2021-02-10T04:40:28.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:28 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:28.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:28 smithi164 conmon[29288]: cluster 2021-02-10T04:40:27 2021-02-10T04:40:28.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:28 smithi164 conmon[29288]: .224024+0000 osd.4 (osd.4) 1326 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:28.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:28 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:40:28.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:28 smithi164 conmon[29288]: T04:40:27.405379+0000 mon.a (mon. 2021-02-10T04:40:28.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:28 smithi164 conmon[29288]: 0) 1748 : cluster [DBG] osdmap e808: 8 total, 8 up, 7 in 2021-02-10T04:40:28.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:28 smithi164 conmon[29288]: cluster 2021-02-10T04:40 2021-02-10T04:40:28.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:28 smithi164 conmon[29288]: :28.194085+0000 mgr.y (mgr.14140) 786 : cluster [DBG] pgmap v1363: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 520 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 7.6 MiB/s rd, 219 KiB/s wr, 8 op/s 2021-02-10T04:40:28.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:28 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:40:28.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:28 smithi164 conmon[29288]: T04:40:28.260436+0000 mon.a (mon.0 2021-02-10T04:40:28.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:28 smithi164 conmon[29288]: ) 1749 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 652 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:40:28.647 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:40:28 smithi164 conmon[44544]: debug 2021-02-10T04:40:28.388+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:40:28.662 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:40:28.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[29319]: cluster 2021-02-10T04:40:27.223988+0000 osd.4 ( 2021-02-10T04:40:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[29319]: osd.4) 1323 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[29319]: cluster 2021-02-10T04:40:27.223994+0000 osd.4 (osd.4) 1324 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[29319]: cluster 2021-02-10T04:40:27.223998+0000 osd.4 (osd.4) 1325 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[29319]: cluster 2021-02-10T04:40:27.224024+0000 osd.4 (osd.4) 1326 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[29319]: cluster 2021-02-10T04:40:27.405379+0000 mon.a (mon.0) 1748 : cluster [DBG] osdmap e808: 8 total, 8 up, 7 in 2021-02-10T04:40:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:40:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[29319]: -10T04:40:28.194085+0000 mgr.y (mgr.14140) 786 : cluster [DBG] pgmap v1363: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 520 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 7.6 MiB/s rd, 219 KiB/s wr, 8 op/s 2021-02-10T04:40:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[29319]: cluster 2021-02-10T04:40:28.260436+0000 mon.a (mon.0) 1749 2021-02-10T04:40:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[29319]: : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 652 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:40:28.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: cluster 2021-02-10T04:40:27.223988 2021-02-10T04:40:28.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1323 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:40:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: 04:40:27.223994+0000 osd.4 (osd.4) 1324 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:40:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: -02-10T04:40:27.223998+0000 2021-02-10T04:40:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: osd.4 (osd.4) 1325 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: 2021-02-10T04:40:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:40:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: 40:27.224024+0000 osd.4 (osd.4 2021-02-10T04:40:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: ) 1326 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: cluster 2021-02-10T04:40:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: 2021 2021-02-10T04:40:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: - 2021-02-10T04:40:28.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: 02 2021-02-10T04:40:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: - 2021-02-10T04:40:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: 10 2021-02-10T04:40:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: T 2021-02-10T04:40:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: 04 2021-02-10T04:40:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: : 2021-02-10T04:40:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: 40 2021-02-10T04:40:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: : 2021-02-10T04:40:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: 27 2021-02-10T04:40:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: . 2021-02-10T04:40:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: 405379 2021-02-10T04:40:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: +0000 2021-02-10T04:40:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: 2021-02-10T04:40:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: mon.a 2021-02-10T04:40:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: ( 2021-02-10T04:40:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: mon 2021-02-10T04:40:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: . 2021-02-10T04:40:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: 0 2021-02-10T04:40:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: ) 2021-02-10T04:40:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: 1748 : cluster [DBG] osdmap e808: 8 total, 8 up, 7 in 2021-02-10T04:40:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: cluster 2021-02-10T04:40:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: 2021-02-10T04:40:28.194085+0000 mgr.y (mgr.14140) 786 : cluster [DBG] pgmap v1363: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 520 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 7.6 MiB/s rd, 219 KiB/s wr, 8 op/s 2021-02-10T04:40:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: cluster 2021-02-10T04:40 2021-02-10T04:40:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: :28.260436+0000 mon.a (mon.0) 1749 : cluster 2021-02-10T04:40:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:28 smithi085 conmon[35961]: [WRN] Health check update: 4 slow ops, oldest one blocked for 652 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:40:28.698 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:40:28.703 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:40:28.707 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:40:29.034 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:40:29.039 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:40:28 smithi164 conmon[44544]: debug 2021-02-10T04:40:29.039 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:40:28 smithi164 conmon[44544]: 2021-02-10T04:40:28.773+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:40:29.040 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:40:28 smithi164 conmon[44544]: 2021-02-10T04:40:29.139 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:40:29.171 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:40:29.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:29 smithi164 conmon[33677]: debug 2021-02-10T04:40:29.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:29 smithi164 conmon[33677]: 2021-02-10T04:40:29.209+0000 7fbec19a1700 -1 osd.4 809 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:29.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:29 smithi164 conmon[33677]: 2021-02-10T04:40:29.431 INFO:tasks.rados.rados.0.smithi085.stdout:3193: done (6 left) 2021-02-10T04:40:29.431 INFO:tasks.rados.rados.0.smithi085.stdout:3194: read oid 1 snap 346 2021-02-10T04:40:29.431 INFO:tasks.rados.rados.0.smithi085.stdout:3194: expect (ObjNum 1051 snap 345 seq_num 1051) 2021-02-10T04:40:29.431 INFO:tasks.rados.rados.0.smithi085.stdout:3195: copy_from oid 10 from oid 22 current snap is 367 2021-02-10T04:40:29.431 INFO:tasks.rados.rados.0.smithi085.stdout:3196: read oid 2 snap 354 2021-02-10T04:40:29.432 INFO:tasks.rados.rados.0.smithi085.stdout:3196: expect (ObjNum 982 snap 317 seq_num 982) 2021-02-10T04:40:29.432 INFO:tasks.rados.rados.0.smithi085.stdout:3197: snap_create 2021-02-10T04:40:29.432 INFO:tasks.rados.rados.0.smithi085.stdout:3192: finishing rollback tid 0 to smithi08575397-6 2021-02-10T04:40:29.432 INFO:tasks.rados.rados.0.smithi085.stdout:3192: finishing rollback tid 1 to smithi08575397-6 2021-02-10T04:40:29.432 INFO:tasks.rados.rados.0.smithi085.stdout:3192: finishing rollback tid 2 to smithi08575397-6 2021-02-10T04:40:29.432 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 1555 (ObjNum 1028 snap 335 seq_num 1028) dirty exists 2021-02-10T04:40:29.432 INFO:tasks.rados.rados.0.smithi085.stdout:3189: expect (ObjNum 1052 snap 345 seq_num 1052) 2021-02-10T04:40:29.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[29319]: cluster 2021-02-10T04:40:28.259894+0000 osd.4 (osd.4) 1327 : cluster 2021-02-10T04:40:29.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:29.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[29319]: cluster 2021-02-10T04:40:28.259906+0000 osd.4 (osd.4) 1328 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:29.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[29319]: cluster 2021-02-10T04:40:28.259913+0000 osd.4 (osd.4) 1329 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:29.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[29319]: cluster 2021-02-10T04:40:28.259919+0000 osd.4 (osd.4) 1330 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:29.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[29319]: audit 2021-02-10T04:40:28.397034+0000 2021-02-10T04:40:29.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[29319]: mgr.y (mgr.14140) 787 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:40:29.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[29319]: cluster 2021-02-10T04:40:28.410438+0000 2021-02-10T04:40:29.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[29319]: mon.a (mon.0) 1750 : cluster [DBG] osdmap e809: 8 total, 8 up, 7 in 2021-02-10T04:40:29.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[35961]: cluster 2021-02-10T04:40:28.259894+0000 2021-02-10T04:40:29.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[35961]: osd.4 (osd.4) 1327 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:29.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:40:29.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[35961]: -10T04:40:28.259906+0000 osd.4 (osd.4) 1328 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:29.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[35961]: 2021-02-10T04:40:29.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[35961]: cluster 2021-02-10T04:40:29.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[35961]: 2021 2021-02-10T04:40:29.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[35961]: - 2021-02-10T04:40:29.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[35961]: 02 2021-02-10T04:40:29.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[35961]: -10T04:40:28.259913+0000 2021-02-10T04:40:29.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[35961]: osd.4 (osd.4) 1329 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:29.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:40:29.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[35961]: 02-10T04:40:28 2021-02-10T04:40:29.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[35961]: .259919+0000 osd.4 (osd.4) 1330 : 2021-02-10T04:40:29.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:29.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:40:29.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[35961]: 02-10T04:40:28 2021-02-10T04:40:29.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[35961]: .397034+0000 mgr.y (mgr.14140) 787 : audit [DBG] 2021-02-10T04:40:29.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[35961]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:40:29.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:40:29.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[35961]: -02-10T04:40:28.410438+0000 mon.a (mon.0) 1750 : cluster 2021-02-10T04:40:29.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:29 smithi085 conmon[35961]: [DBG] osdmap e809: 8 total, 8 up, 7 in 2021-02-10T04:40:29.723 INFO:tasks.rados.rados.0.smithi085.stdout:3196: expect (ObjNum 982 snap 317 seq_num 982) 2021-02-10T04:40:29.741 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:40:29.746 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:29 smithi164 conmon[29288]: cluster 2021-02-10T04:40:28.259894+0000 2021-02-10T04:40:29.746 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:29 smithi164 conmon[29288]: osd.4 (osd.4) 1327 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:29.746 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:29 smithi164 conmon[29288]: cluster 2021-02-10T04:40:28.259906+0000 osd.4 (osd.4) 1328 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:29.747 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:29 smithi164 conmon[29288]: cluster 2021-02-10T04:40:28.259913+0000 osd.4 (osd.4) 1329 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:29.747 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:29 smithi164 conmon[29288]: cluster 2021-02-10T04:40 2021-02-10T04:40:29.747 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:29 smithi164 conmon[29288]: :28.259919+0000 osd.4 (osd.4) 1330 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:29.747 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:29 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:40:29.747 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:29 smithi164 conmon[29288]: T04:40:28.397034+0000 mgr.y (mgr.14140) 787 : audit 2021-02-10T04:40:29.747 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:29 smithi164 conmon[29288]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:40:29.747 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:29 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:40:29.747 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:29 smithi164 conmon[29288]: 02-10T04:40:28.410438+0000 mon.a (mon.0) 1750 : cluster [DBG] osdmap e809: 8 total, 8 up, 7 in 2021-02-10T04:40:29.748 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:29 smithi164 conmon[33677]: debug 2021-02-10T04:40:29.513+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:40:29.849 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:40:29.873 INFO:tasks.rados.rados.0.smithi085.stdout:3194: expect (ObjNum 1051 snap 345 seq_num 1051) 2021-02-10T04:40:29.879 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:40:30.033 INFO:tasks.rados.rados.0.smithi085.stdout:3195: finishing copy_from to smithi08575397-10 2021-02-10T04:40:30.033 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 10 v 1466 (ObjNum 966 snap 315 seq_num 966) dirty exists 2021-02-10T04:40:30.033 INFO:tasks.rados.rados.0.smithi085.stdout:3195: finishing copy_from racing read to smithi08575397-10 2021-02-10T04:40:30.427 INFO:tasks.rados.rados.0.smithi085.stdout:3189: done (9 left) 2021-02-10T04:40:30.427 INFO:tasks.rados.rados.0.smithi085.stdout:3192: done (8 left) 2021-02-10T04:40:30.428 INFO:tasks.rados.rados.0.smithi085.stdout:3194: done (7 left) 2021-02-10T04:40:30.428 INFO:tasks.rados.rados.0.smithi085.stdout:3195: done (6 left) 2021-02-10T04:40:30.428 INFO:tasks.rados.rados.0.smithi085.stdout:3196: done (5 left) 2021-02-10T04:40:30.428 INFO:tasks.rados.rados.0.smithi085.stdout:3197: done (4 left) 2021-02-10T04:40:30.428 INFO:tasks.rados.rados.0.smithi085.stdout:3198: snap_remove snap 366 2021-02-10T04:40:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:30 smithi164 conmon[29288]: cluster 2021-02-10T04:40:29.210340+0000 osd.4 (osd 2021-02-10T04:40:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:30 smithi164 conmon[29288]: .4) 1331 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:30 smithi164 conmon[29288]: cluster 2021-02-10T04:40:29.210363+0000 osd.4 (osd.4) 1332 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:30 smithi164 conmon[29288]: cluster 2021-02-10T04:40:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:30 smithi164 conmon[29288]: 2021-02-10T04:40:29.210366+0000 osd.4 (osd.4) 1333 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:30 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:40:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:30 smithi164 conmon[29288]: -10T04:40:29.210370+0000 osd.4 (osd.4 2021-02-10T04:40:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:30 smithi164 conmon[29288]: ) 1334 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:30 smithi164 conmon[29288]: cluster 2021-02-10T04:40:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:30 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:40:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:30 smithi164 conmon[29288]: :40:29.427364+0000 mon.a (mon. 2021-02-10T04:40:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:30 smithi164 conmon[29288]: 0) 1751 : cluster [DBG] osdmap e810: 8 total, 8 up, 7 in 2021-02-10T04:40:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:30 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:40:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:30 smithi164 conmon[29288]: -02-10T04:40:30.194814+0000 mgr.y 2021-02-10T04:40:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:30 smithi164 conmon[29288]: (mgr.14140) 788 : cluster [DBG] 2021-02-10T04:40:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:30 smithi164 conmon[29288]: pgmap v1366: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 530 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 2.1 MiB/s wr, 21 op/s 2021-02-10T04:40:30.645 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:30 smithi164 conmon[33677]: debug 2021-02-10T04:40:30.252+0000 7fbec19a1700 -1 osd.4 810 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:30.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[29319]: cluster 2021-02-10T04:40:29.210340+0000 2021-02-10T04:40:30.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[29319]: osd.4 (osd.4) 1331 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:30.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[29319]: cluster 2021-02-10T04:40:29.210363+0000 osd.4 (osd.4) 1332 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:30.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:40:30.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[29319]: T04:40:29.210366+0000 osd.4 (osd.4) 1333 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:30.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[29319]: cluster 2021-02-10T04:40:29. 2021-02-10T04:40:30.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[29319]: 210370+0000 osd.4 (osd.4) 1334 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:30.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:40:30.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[29319]: T04:40:29.427364+0000 mon.a (mon. 2021-02-10T04:40:30.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[29319]: 0) 1751 : cluster [DBG] osdmap e810: 8 total, 8 up, 7 in 2021-02-10T04:40:30.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:40:30.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[29319]: -10T04:40:30.194814+0000 mgr.y (mgr 2021-02-10T04:40:30.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[29319]: .14140) 788 : cluster [DBG] pgmap v1366: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 530 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 2.1 MiB/s wr, 21 op/s 2021-02-10T04:40:30.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[35961]: cluster 2021-02-10T04:40:29.210340+0000 2021-02-10T04:40:30.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[35961]: osd.4 (osd.4) 1331 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:30.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:40:30.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[35961]: -02-10T04:40:29.210363+0000 osd.4 (osd.4) 1332 : cluster [WRN] 2021-02-10T04:40:30.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:30.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:40:30.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[35961]: T04:40:29.210366+0000 osd.4 (osd. 2021-02-10T04:40:30.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[35961]: 4) 1333 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:30.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[35961]: cluster 2021-02-10T04:40:30.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[35961]: 2021-02-10T04:40:29.210370+0000 2021-02-10T04:40:30.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[35961]: osd.4 (osd.4) 1334 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:30.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[35961]: 2021-02-10T04:40:30.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:40:30.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[35961]: :40:29.427364+0000 mon.a (mon. 2021-02-10T04:40:30.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[35961]: 0) 1751 : cluster [DBG] osdmap e810: 8 total, 8 up, 7 in 2021-02-10T04:40:30.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[35961]: cluster 2021-02-10T04:40:30.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[35961]: 2021-02-10T04:40:30.194814 2021-02-10T04:40:30.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 788 : cluster [DBG] 2021-02-10T04:40:30.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:30 smithi085 conmon[35961]: pgmap v1366: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 530 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 2.1 MiB/s wr, 21 op/s 2021-02-10T04:40:31.439 INFO:tasks.rados.rados.0.smithi085.stdout:3198: done (4 left) 2021-02-10T04:40:31.439 INFO:tasks.rados.rados.0.smithi085.stdout:3199: snap_remove snap 303 2021-02-10T04:40:31.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:31 smithi164 conmon[33677]: debug 2021-02-10T04:40:31.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:31 smithi164 conmon[33677]: 2021-02-10T04:40:31.275+0000 7fbec19a1700 -1 osd.4 811 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:31 smithi164 conmon[29288]: cluster 2021-02-10T04:40:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:31 smithi164 conmon[29288]: 2021-02-10T04:40:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:31 smithi164 conmon[29288]: 2021 2021-02-10T04:40:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:31 smithi164 conmon[29288]: - 2021-02-10T04:40:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:31 smithi164 conmon[29288]: 02-10T04:40:30.254094 2021-02-10T04:40:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:31 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1335 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:31 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:40:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:31 smithi164 conmon[29288]: 10T04:40:30.254106+0000 osd.4 ( 2021-02-10T04:40:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:31 smithi164 conmon[29288]: osd.4) 1336 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:31 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:40:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:31 smithi164 conmon[29288]: -02-10T04:40:30.254114+0000 2021-02-10T04:40:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:31 smithi164 conmon[29288]: osd.4 (osd.4) 1337 : cluster [WRN] 2021-02-10T04:40:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:31 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:31 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:40:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:31 smithi164 conmon[29288]: 04:40:30.254122+0000 osd.4 (osd. 2021-02-10T04:40:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:31 smithi164 conmon[29288]: 4) 1338 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:31 smithi164 conmon[29288]: cluster 2021-02-10T04:40:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:31 smithi164 conmon[29288]: 2021-02-10T04:40:30. 2021-02-10T04:40:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:31 smithi164 conmon[29288]: 427256+0000 mon.a (mon.0) 1752 : cluster 2021-02-10T04:40:31.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:31 smithi164 conmon[29288]: [DBG] osdmap e811: 8 total, 8 up, 7 in 2021-02-10T04:40:31.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:31 smithi085 conmon[29319]: cluster 2021-02-10T04:40:30.254094+0000 osd.4 ( 2021-02-10T04:40:31.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:31 smithi085 conmon[29319]: osd.4) 1335 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:31 smithi085 conmon[29319]: cluster 2021-02-10T04:40:30.254106+0000 osd.4 (osd.4) 1336 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:31 smithi085 conmon[29319]: 2021-02-10T04:40:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:31 smithi085 conmon[29319]: cluster 2021-02-10T04:40:30.254114+0000 2021-02-10T04:40:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:31 smithi085 conmon[29319]: osd.4 (osd.4) 1337 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:31 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:40:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:31 smithi085 conmon[29319]: 04:40:30.254122+0000 osd.4 (osd.4) 1338 : 2021-02-10T04:40:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:31 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:31 smithi085 conmon[29319]: cluster 2021-02-10T04:40:30.427256+0000 mon.a (mon. 2021-02-10T04:40:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:31 smithi085 conmon[29319]: 0) 1752 : cluster [DBG] osdmap e811: 8 total, 8 up, 7 in 2021-02-10T04:40:31.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:31 smithi085 conmon[35961]: cluster 2021-02-10T04:40:31.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:31 smithi085 conmon[35961]: 2021-02-10T04:40:31.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:31 smithi085 conmon[35961]: 2021 2021-02-10T04:40:31.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:31 smithi085 conmon[35961]: - 2021-02-10T04:40:31.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:31 smithi085 conmon[35961]: 02 2021-02-10T04:40:31.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:31 smithi085 conmon[35961]: -10T04:40: 2021-02-10T04:40:31.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:31 smithi085 conmon[35961]: 30.254094+0000 osd.4 (osd.4) 1335 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:31.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:31 smithi085 conmon[35961]: cluster 2021-02-10T04:40:31.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:31 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:40:31.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:31 smithi085 conmon[35961]: 40:30.254106+0000 osd.4 (osd.4) 1336 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:31.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:31 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:40:31.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:31 smithi085 conmon[35961]: 02-10T04:40:30.254114+0000 osd.4 (osd. 2021-02-10T04:40:31.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:31 smithi085 conmon[35961]: 4) 1337 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:31.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:31 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:40:31.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:31 smithi085 conmon[35961]: 02-10T04:40:30.254122+0000 osd.4 ( 2021-02-10T04:40:31.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:31 smithi085 conmon[35961]: osd.4) 1338 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:31.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:31 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:40:31.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:31 smithi085 conmon[35961]: -10T04:40:30.427256+0000 mon.a (mon.0) 1752 : cluster 2021-02-10T04:40:31.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:31 smithi085 conmon[35961]: [DBG] osdmap e811: 8 total, 8 up, 7 in 2021-02-10T04:40:32.434 INFO:tasks.rados.rados.0.smithi085.stdout:3199: done (4 left) 2021-02-10T04:40:32.434 INFO:tasks.rados.rados.0.smithi085.stdout:3200: rollback oid 40 current snap is 368 2021-02-10T04:40:32.434 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 40 to 266 2021-02-10T04:40:32.434 INFO:tasks.rados.rados.0.smithi085.stdout:3201: copy_from oid 28 from oid 44 current snap is 368 2021-02-10T04:40:32.434 INFO:tasks.rados.rados.0.smithi085.stdout:3202: snap_remove snap 338 2021-02-10T04:40:32.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:32 smithi164 conmon[33677]: debug 2021-02-10T04:40:32.314+0000 7fbec19a1700 -1 osd.4 812 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:32 smithi164 conmon[29288]: cluster 2021-02-10T04:40:31.276716+0000 osd.4 (osd.4) 1339 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:32 smithi164 conmon[29288]: cluster 2021-02-10T04:40:31.276739+0000 osd.4 (osd.4) 1340 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:32 smithi164 conmon[29288]: cluster 2021-02-10T04:40:31.276746 2021-02-10T04:40:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:32 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1341 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:32 smithi164 conmon[29288]: cluster 2021-02-10T04:40:31.276753+0000 osd.4 2021-02-10T04:40:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:32 smithi164 conmon[29288]: (osd.4) 1342 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:32 smithi164 conmon[29288]: cluster 2021-02-10T04:40:31. 2021-02-10T04:40:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:32 smithi164 conmon[29288]: 434554+0000 mon.a (mon.0) 1753 : cluster [DBG] osdmap e812: 8 total, 8 up, 7 in 2021-02-10T04:40:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:32 smithi164 conmon[29288]: cluster 2021-02-10T04:40:31.436825+0000 osd.3 2021-02-10T04:40:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:32 smithi164 conmon[29288]: (osd.3) 84 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:40:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:32 smithi164 conmon[29288]: cluster 2021-02-10T04:40:31.441092+0000 osd.3 (osd.3) 2021-02-10T04:40:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:32 smithi164 conmon[29288]: 85 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:40:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:32 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:40:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:32 smithi164 conmon[29288]: -02-10T04:40:32.195414+0000 mgr.y (mgr.14140) 789 : cluster [DBG] pgmap v1369: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 530 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 7.9 MiB/s rd, 1.9 MiB/s wr, 15 op/s 2021-02-10T04:40:32.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:32 smithi164 conmon[29288]: cluster 2021-02-10T04:40:32.431618+0000 2021-02-10T04:40:32.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:32 smithi164 conmon[29288]: mon.a (mon.0) 1754 : cluster [DBG] osdmap e813: 8 total, 8 up, 7 in 2021-02-10T04:40:32.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[35961]: cluster 2021-02-10T04:40:31. 2021-02-10T04:40:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[35961]: 276716+0000 osd.4 (osd.4) 1339 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:40:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[35961]: 02-10T04:40:31.276739+0000 osd.4 (osd.4) 1340 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[35961]: 2021-02-10T04:40:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[35961]: cluster 2021-02-10T04:40:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[35961]: 2021-02-10T04:40:31.276746+0000 osd.4 (osd. 2021-02-10T04:40:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[35961]: 4) 1341 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:40:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[35961]: :40:31.276753+0000 osd.4 (osd.4) 1342 : 2021-02-10T04:40:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[35961]: cluster 2021-02-10T04:40:31.434554 2021-02-10T04:40:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1753 : cluster [DBG] 2021-02-10T04:40:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[35961]: osdmap e812: 8 total, 8 up, 7 in 2021-02-10T04:40:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[35961]: cluster 2021-02-10T04:40:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[35961]: 2021-02-10T04:40 2021-02-10T04:40:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[35961]: :31.436825+0000 osd.3 (osd.3) 84 : cluster 2021-02-10T04:40:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[35961]: [DBG] purged_snaps scrub starts 2021-02-10T04:40:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:40:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[35961]: 40:31.441092 2021-02-10T04:40:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[35961]: +0000 osd.3 (osd.3) 85 : cluster [DBG] 2021-02-10T04:40:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:40:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[35961]: cluster 2021-02-10T04:40:32 2021-02-10T04:40:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[35961]: .195414+0000 mgr.y (mgr. 2021-02-10T04:40:32.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[35961]: 14140) 789 : cluster [DBG] pgmap v1369: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 530 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 7.9 MiB/s rd, 1.9 MiB/s wr, 15 op/s 2021-02-10T04:40:32.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:40:32.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[35961]: 02-10T04:40:32.431618+0000 2021-02-10T04:40:32.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[35961]: mon.a (mon.0) 1754 : cluster 2021-02-10T04:40:32.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[35961]: [DBG] osdmap e813: 8 total, 8 up, 7 in 2021-02-10T04:40:32.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[29319]: cluster 2021-02-10T04:40:31.276716+0000 osd.4 (osd.4) 1339 : cluster [WRN] 2021-02-10T04:40:32.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:32.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[29319]: cluster 2021-02-10T04:40:31.276739+0000 osd.4 (osd.4) 1340 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:32.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[29319]: cluster 2021-02-10T04:40:32.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[29319]: 2021-02-10T04:40:31.276746+0000 osd.4 (osd.4) 1341 : 2021-02-10T04:40:32.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:32.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[29319]: cluster 2021-02-10T04:40:32.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[29319]: 2021-02-10T04:40:31.276753+0000 osd.4 (osd.4) 1342 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:32.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[29319]: cluster 2021-02-10T04:40:32.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[29319]: 2021-02-10T04:40:31.434554+0000 2021-02-10T04:40:32.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[29319]: mon.a (mon.0) 1753 : cluster [DBG] 2021-02-10T04:40:32.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[29319]: osdmap e812: 8 total, 8 up, 7 in 2021-02-10T04:40:32.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:40:32.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[29319]: :40:31.436825+0000 osd.3 (osd.3 2021-02-10T04:40:32.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[29319]: ) 84 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:40:32.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:40:32.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[29319]: -02-10T04:40:31. 2021-02-10T04:40:32.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[29319]: 441092+0000 osd.3 (osd.3) 85 : cluster 2021-02-10T04:40:32.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[29319]: [DBG] purged_snaps scrub ok 2021-02-10T04:40:32.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:40:32.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[29319]: 10T04:40:32.195414+0000 mgr.y (mgr 2021-02-10T04:40:32.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[29319]: .14140) 789 : cluster [DBG] pgmap v1369: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 530 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 7.9 MiB/s rd, 1.9 MiB/s wr, 15 op/s 2021-02-10T04:40:32.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[29319]: cluster 2021-02-10T04:40:32.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[29319]: 2021-02-10T04:40 2021-02-10T04:40:32.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[29319]: :32.431618+0000 mon.a (mon.0) 1754 2021-02-10T04:40:32.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[29319]: : cluster [DBG] osdmap e813: 8 total, 8 up, 7 in 2021-02-10T04:40:33.042 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:40:33.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:40:32 smithi085 conmon[49106]: debug 2021-02-10T04:40:32.760+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:40:33.434 INFO:tasks.rados.rados.0.smithi085.stdout:3202: done (6 left) 2021-02-10T04:40:33.434 INFO:tasks.rados.rados.0.smithi085.stdout:3203: snap_remove snap 362 2021-02-10T04:40:33.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:33 smithi164 conmon[33677]: debug 2021-02-10T04:40:33.350+0000 7fbec19a1700 -1 osd.4 813 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:33.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:33 smithi164 conmon[29288]: cluster 2021-02-10T04:40:32.316640+0000 osd.4 (osd.4) 1343 2021-02-10T04:40:33.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:33 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:33.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:33 smithi164 conmon[29288]: cluster 2021-02-10T04:40:32.316650+0000 osd.4 (osd.4) 1344 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:33 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:40:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:33 smithi164 conmon[29288]: :40:32.316656+0000 osd.4 (osd.4) 1345 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:33 smithi164 conmon[29288]: cluster 2021-02-10T04:40:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:33 smithi164 conmon[29288]: 2021-02-10T04:40:32.316663+0000 osd.4 (osd.4) 1346 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:33 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:40:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:33 smithi164 conmon[29288]: T04:40:33.261696+0000 mon.a (mon.0) 1755 : cluster [WRN] 2021-02-10T04:40:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:33 smithi164 conmon[29288]: Health check update: 4 slow ops, oldest one blocked for 662 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:40:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:33 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:40:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:33 smithi164 conmon[29288]: 10T04:40:33.432621+0000 mon.a (mon.0) 1756 : cluster [DBG] osdmap e814: 8 total, 8 up, 7 in 2021-02-10T04:40:33.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[29319]: cluster 2021-02-10T04:40:32.316640+0000 2021-02-10T04:40:33.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[29319]: osd.4 (osd.4) 1343 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:33.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[29319]: cluster 2021-02-10T04:40:32.316650+0000 osd.4 (osd.4) 1344 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:33.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[29319]: cluster 2021-02-10T04:40:32.316656+0000 osd.4 (osd.4) 2021-02-10T04:40:33.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[29319]: 1345 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:33.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:40:33.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[29319]: 04:40:32.316663+0000 osd.4 (osd.4) 1346 : 2021-02-10T04:40:33.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:33.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[29319]: cluster 2021-02-10T04:40:33.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[29319]: 2021-02-10T04:40:33.261696+0000 2021-02-10T04:40:33.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[29319]: mon.a (mon.0) 1755 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 662 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:40:33.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:40:33.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[29319]: 04:40:33.432621+0000 mon.a (mon. 2021-02-10T04:40:33.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[29319]: 0) 1756 : cluster [DBG] osdmap e814: 8 total, 8 up, 7 in 2021-02-10T04:40:33.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[35961]: cluster 2021-02-10T04:40:33.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[35961]: 2021-02-10T04:40:33.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[35961]: 2021 2021-02-10T04:40:33.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[35961]: -02-10T04:40:32.316640 2021-02-10T04:40:33.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1343 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:33.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[35961]: cluster 2021-02-10T04:40:32.316650+0000 osd.4 (osd.4) 1344 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:33.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[35961]: 2021-02-10T04:40:33.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[35961]: cluster 2021-02-10T04:40:32. 2021-02-10T04:40:33.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[35961]: 316656+0000 osd.4 (osd.4) 1345 : cluster [WRN] 2021-02-10T04:40:33.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:33.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:40:33.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[35961]: -02-10T04:40:32.316663 2021-02-10T04:40:33.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1346 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:33.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[35961]: 2021-02-10T04:40:33.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[35961]: cluster 2021-02-10T04:40: 2021-02-10T04:40:33.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[35961]: 33.261696+0000 mon.a (mon.0) 1755 : cluster 2021-02-10T04:40:33.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[35961]: [WRN] Health check update: 4 slow ops, oldest one blocked for 662 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:40:33.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:40:33.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[35961]: 04:40:33.432621+0000 mon.a (mon.0 2021-02-10T04:40:33.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:33 smithi085 conmon[35961]: ) 1756 : cluster [DBG] osdmap e814: 8 total, 8 up, 7 in 2021-02-10T04:40:34.436 INFO:tasks.rados.rados.0.smithi085.stdout:3203: done (6 left) 2021-02-10T04:40:34.436 INFO:tasks.rados.rados.0.smithi085.stdout:3204: snap_remove snap 306 2021-02-10T04:40:34.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:34 smithi164 conmon[33677]: debug 2021-02-10T04:40:34.392+0000 7fbec19a1700 -1 osd.4 814 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:35.032 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:34 smithi164 conmon[29288]: cluster 2021-02-10T04:40:33.351751 2021-02-10T04:40:35.032 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:34 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1347 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:35.032 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:34 smithi164 conmon[29288]: cluster 2021-02-10T04:40:33.351765+0000 osd.4 (osd.4) 1348 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:35.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:34 smithi164 conmon[29288]: cluster 2021-02-10T04:40:33.351777+0000 osd.4 (osd.4) 1349 : cluster 2021-02-10T04:40:35.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:34 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:35.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:34 smithi164 conmon[29288]: cluster 2021-02-10T04:40:35.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:34 smithi164 conmon[29288]: 2021-02-10T04:40:33.351784+0000 osd.4 (osd.4) 1350 : 2021-02-10T04:40:35.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:34 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:35.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:34 smithi164 conmon[29288]: cluster 2021-02-10T04:40:35.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:34 smithi164 conmon[29288]: 2021-02-10T04:40:34.112711+0000 osd.0 (osd 2021-02-10T04:40:35.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:34 smithi164 conmon[29288]: .0) 89 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:40:35.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:34 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:40:35.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:34 smithi164 conmon[29288]: T04:40:34.113862+0000 osd.0 (osd.0) 2021-02-10T04:40:35.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:34 smithi164 conmon[29288]: 90 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:40:35.034 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:34 smithi164 conmon[29288]: cluster 2021-02-10T04:40:35.034 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:34 smithi164 conmon[29288]: 2021-02-10T04:40:34.195961+0000 mgr.y (mgr. 2021-02-10T04:40:35.034 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:34 smithi164 conmon[29288]: 14140) 790 : cluster [DBG] pgmap v1372: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 529 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 7.0 MiB/s rd, 0 B/s wr, 5 op/s 2021-02-10T04:40:35.034 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:34 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:40:35.034 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:34 smithi164 conmon[29288]: 04:40:34.435122+0000 mon.a (mon. 2021-02-10T04:40:35.034 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:34 smithi164 conmon[29288]: 0) 1757 : cluster [DBG] osdmap e815: 8 total, 8 up, 7 in 2021-02-10T04:40:35.035 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[35961]: cluster 2021-02-10T04:40: 2021-02-10T04:40:35.035 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[35961]: 33.351751+0000 osd.4 (osd.4) 1347 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:35.036 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:40:35.036 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[35961]: T04:40:33.351765+0000 osd.4 (osd.4) 2021-02-10T04:40:35.036 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[35961]: 1348 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:35.036 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:40:35.036 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[35961]: 02-10T04:40:33.351777 2021-02-10T04:40:35.036 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[35961]: +0000 osd.4 2021-02-10T04:40:35.036 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[35961]: (osd.4) 1349 2021-02-10T04:40:35.036 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:35.036 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:40:35.036 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[35961]: -02-10T04:40:33.351784+0000 2021-02-10T04:40:35.036 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[35961]: osd.4 (osd.4) 2021-02-10T04:40:35.037 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[35961]: 1350 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:35.037 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:40:35.037 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[35961]: T04:40:34.112711+0000 osd.0 (osd.0) 89 : cluster 2021-02-10T04:40:35.037 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[35961]: [DBG] purged_snaps scrub starts 2021-02-10T04:40:35.037 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:40:35.037 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[35961]: :40:34.113862+0000 2021-02-10T04:40:35.037 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[35961]: osd.0 (osd.0) 90 : 2021-02-10T04:40:35.037 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[35961]: cluster [DBG] 2021-02-10T04:40:35.037 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:40:35.037 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[35961]: cluster 2021-02-10T04:40:35.037 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:40:35.038 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[35961]: :40:34.195961+0000 mgr.y (mgr.14140 2021-02-10T04:40:35.038 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[35961]: ) 790 : cluster [DBG] pgmap v1372: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 529 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 7.0 MiB/s rd, 0 B/s wr, 5 op/s 2021-02-10T04:40:35.038 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[35961]: cluster 2021-02-10T04:40:35.038 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[35961]: 2021-02-10T04:40:34.435122+0000 mon.a 2021-02-10T04:40:35.038 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[35961]: (mon.0) 1757 : cluster [DBG] 2021-02-10T04:40:35.038 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[35961]: osdmap e815: 8 total, 8 up, 7 in 2021-02-10T04:40:35.038 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: cluster 2021-02-10T04:40:35.038 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: 2021-02-10T04:40:35.038 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: 2021 2021-02-10T04:40:35.038 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: - 2021-02-10T04:40:35.038 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: 02-10T04:40:33.351751 2021-02-10T04:40:35.039 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1347 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:35.039 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: cluster 2021-02-10T04:40:35.039 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: 2021-02-10T04:40:33.351765 2021-02-10T04:40:35.039 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1348 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:35.039 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: 2021-02-10T04:40:35.039 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: cluster 2021-02-10T04:40:33. 2021-02-10T04:40:35.039 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: 351777+0000 osd.4 (osd.4) 1349 : cluster [WRN] 2021-02-10T04:40:35.039 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:35.039 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: cluster 2021-02-10T04:40:35.039 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: 2021-02-10T04:40:33.351784+0000 osd.4 (osd. 2021-02-10T04:40:35.039 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: 4) 1350 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:35.040 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:40:35.040 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: 10T04:40:34.112711+0000 osd.0 ( 2021-02-10T04:40:35.040 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: osd.0) 89 : cluster [DBG] 2021-02-10T04:40:35.040 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: purged_snaps scrub starts 2021-02-10T04:40:35.040 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: cluster 2021-02-10T04:40:35.040 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: 2021-02-10T04:40 2021-02-10T04:40:35.040 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: :34.113862+0000 osd.0 (osd.0) 2021-02-10T04:40:35.040 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: 90 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:40:35.040 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:40:35.041 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: -10T04:40:34.195961+0000 mgr.y 2021-02-10T04:40:35.041 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: (mgr.14140) 790 : cluster [DBG] pgmap v1372: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 529 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 7.0 MiB/s rd, 0 B/s wr, 5 op/s 2021-02-10T04:40:35.041 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: 2021-02-10T04:40:35.041 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:40:35.041 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: -02-10T04:40:34.435122+0000 2021-02-10T04:40:35.041 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:34 smithi085 conmon[29319]: mon.a (mon.0) 1757 : cluster [DBG] osdmap e815: 8 total, 8 up, 7 in 2021-02-10T04:40:35.437 INFO:tasks.rados.rados.0.smithi085.stdout:3204: done (6 left) 2021-02-10T04:40:37.547 INFO:tasks.rados.rados.0.smithi085.stdout:3205: write oid 38 current snap is 368 2021-02-10T04:40:37.548 INFO:tasks.rados.rados.0.smithi085.stdout:3205: seq_num 1095 ranges {403665=752487,1610018=514716} 2021-02-10T04:40:37.553 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:35 smithi164 conmon[29288]: cluster 2021-02-10T04:40:34.394491+0000 2021-02-10T04:40:37.553 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:35 smithi164 conmon[29288]: osd.4 (osd.4) 1351 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:37.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:35 smithi164 conmon[29288]: cluster 2021-02-10T04:40:34.394500+0000 2021-02-10T04:40:37.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:35 smithi164 conmon[29288]: osd.4 (osd.4) 1352 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:37.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:35 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:40:37.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:35 smithi164 conmon[29288]: 10T04:40:34.394507+0000 osd.4 ( 2021-02-10T04:40:37.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:35 smithi164 conmon[29288]: osd.4) 1353 : cluster [WRN] 2021-02-10T04:40:37.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:35 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:37.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:35 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:40:37.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:35 smithi164 conmon[29288]: 10T04:40:34.394515+0000 osd.4 (osd.4) 1354 : cluster [WRN] 2021-02-10T04:40:37.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:35 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:37.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:35 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:40:37.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:35 smithi164 conmon[29288]: 04:40:35.435744+0000 mon.a (mon 2021-02-10T04:40:37.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:35 smithi164 conmon[29288]: .0) 1758 : cluster [DBG] osdmap e816: 8 total, 8 up, 7 in 2021-02-10T04:40:37.555 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:36 smithi164 conmon[29288]: cluster 2021-02-10T04:40:37.555 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:36 smithi164 conmon[29288]: 2021-02-10T04:40:37.555 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:36 smithi164 conmon[29288]: 2021-02-10T04:40: 2021-02-10T04:40:37.555 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:36 smithi164 conmon[29288]: 35.374090+0000 osd.4 (osd.4) 1355 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:37.555 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:36 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:40:37.555 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:36 smithi164 conmon[29288]: -10T04:40:35.374100 2021-02-10T04:40:37.555 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:36 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1356 2021-02-10T04:40:37.555 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:36 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:37.555 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:36 smithi164 conmon[29288]: cluster 2021-02-10T04:40:37.555 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:36 smithi164 conmon[29288]: 2021-02-10T04:40: 2021-02-10T04:40:37.555 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:36 smithi164 conmon[29288]: 35.374103+0000 osd.4 (osd. 2021-02-10T04:40:37.555 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:36 smithi164 conmon[29288]: 4) 1357 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:37.555 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:36 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:40:37.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:36 smithi164 conmon[29288]: 04:40:35.374107+0000 osd.4 (osd. 2021-02-10T04:40:37.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:36 smithi164 conmon[29288]: 4) 1358 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:37.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:36 smithi164 conmon[29288]: cluster 2021-02-10T04:40:37.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:36 smithi164 conmon[29288]: 2021-02 2021-02-10T04:40:37.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:36 smithi164 conmon[29288]: -10T04:40:36.196509 2021-02-10T04:40:37.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:36 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 2021-02-10T04:40:37.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:36 smithi164 conmon[29288]: 791 : cluster [DBG] pgmap v1375: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 533 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 7.0 MiB/s rd, 898 KiB/s wr, 6 op/s 2021-02-10T04:40:37.556 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:35 smithi164 conmon[33677]: debug 2021-02-10T04:40:35.372+0000 7fbec19a1700 -1 osd.4 815 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:37.556 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:36 smithi164 conmon[33677]: debug 2021-02-10T04:40:36.397+0000 7fbec19a1700 -1 osd.4 816 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:37.556 INFO:tasks.rados.rados.0.smithi085.stdout:3205: writing smithi08575397-38 from 403665 to 1156152 tid 1 2021-02-10T04:40:37.556 INFO:tasks.rados.rados.0.smithi085.stdout:3205: writing smithi08575397-38 from 1610018 to 2124734 tid 2 2021-02-10T04:40:37.557 INFO:tasks.rados.rados.0.smithi085.stdout:3206: write (excl) oid 3 current snap is 368 2021-02-10T04:40:37.557 INFO:tasks.rados.rados.0.smithi085.stdout:3200: finishing rollback tid 1 to smithi08575397-40 2021-02-10T04:40:37.557 INFO:tasks.rados.rados.0.smithi085.stdout:3206: seq_num 1096 ranges {420472=427736,1268500=658374,2347953=72519} 2021-02-10T04:40:37.557 INFO:tasks.rados.rados.0.smithi085.stdout:3206: writing smithi08575397-3 from 420472 to 848208 tid 1 2021-02-10T04:40:37.557 INFO:tasks.rados.rados.0.smithi085.stdout:3206: writing smithi08575397-3 from 1268500 to 1926874 tid 2 2021-02-10T04:40:37.557 INFO:tasks.rados.rados.0.smithi085.stdout:3206: writing smithi08575397-3 from 2347953 to 2420472 tid 3 2021-02-10T04:40:37.557 INFO:tasks.rados.rados.0.smithi085.stdout:3207: read oid 34 snap -1 2021-02-10T04:40:37.557 INFO:tasks.rados.rados.0.smithi085.stdout:3207: expect (ObjNum 1052 snap 345 seq_num 1052) 2021-02-10T04:40:37.557 INFO:tasks.rados.rados.0.smithi085.stdout:3208: write oid 31 current snap is 368 2021-02-10T04:40:37.557 INFO:tasks.rados.rados.0.smithi085.stdout:3200: finishing rollback tid 2 to smithi08575397-40 2021-02-10T04:40:37.557 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 1664 (ObjNum 756 snap 241 seq_num 756) dirty exists 2021-02-10T04:40:37.557 INFO:tasks.rados.rados.0.smithi085.stdout:3201: finishing copy_from to smithi08575397-28 2021-02-10T04:40:37.557 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 1439 (ObjNum 1052 snap 345 seq_num 1052) dirty exists 2021-02-10T04:40:37.558 INFO:tasks.rados.rados.0.smithi085.stdout:3201: finishing copy_from racing read to smithi08575397-28 2021-02-10T04:40:37.558 INFO:tasks.rados.rados.0.smithi085.stdout:3205: finishing write tid 1 to smithi08575397-38 2021-02-10T04:40:37.558 INFO:tasks.rados.rados.0.smithi085.stdout:3208: seq_num 1097 ranges {437279=502985,1610629=518385} 2021-02-10T04:40:37.558 INFO:tasks.rados.rados.0.smithi085.stdout:3208: writing smithi08575397-31 from 437279 to 940264 tid 1 2021-02-10T04:40:37.558 INFO:tasks.rados.rados.0.smithi085.stdout:3208: writing smithi08575397-31 from 1610629 to 2129014 tid 2 2021-02-10T04:40:37.558 INFO:tasks.rados.rados.0.smithi085.stdout:3200: done (9 left) 2021-02-10T04:40:37.558 INFO:tasks.rados.rados.0.smithi085.stdout:3201: done (8 left) 2021-02-10T04:40:37.558 INFO:tasks.rados.rados.0.smithi085.stdout:3209: write (excl) oid 6 current snap is 368 2021-02-10T04:40:37.558 INFO:tasks.rados.rados.0.smithi085.stdout:3205: finishing write tid 2 to smithi08575397-38 2021-02-10T04:40:37.558 INFO:tasks.rados.rados.0.smithi085.stdout:3209: seq_num 1098 ranges {454086=578234,1669111=662043} 2021-02-10T04:40:37.558 INFO:tasks.rados.rados.0.smithi085.stdout:3209: writing smithi08575397-6 from 454086 to 1032320 tid 1 2021-02-10T04:40:37.558 INFO:tasks.rados.rados.0.smithi085.stdout:3209: writing smithi08575397-6 from 1669111 to 2331154 tid 2 2021-02-10T04:40:37.558 INFO:tasks.rados.rados.0.smithi085.stdout:3210: snap_create 2021-02-10T04:40:37.559 INFO:tasks.rados.rados.0.smithi085.stdout:3205: finishing write tid 3 to smithi08575397-38 2021-02-10T04:40:37.559 INFO:tasks.rados.rados.0.smithi085.stdout:3205: finishing write tid 4 to smithi08575397-38 2021-02-10T04:40:37.559 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 1558 (ObjNum 1095 snap 368 seq_num 1095) dirty exists 2021-02-10T04:40:37.559 INFO:tasks.rados.rados.0.smithi085.stdout:3205: left oid 38 (ObjNum 1095 snap 368 seq_num 1095) 2021-02-10T04:40:37.559 INFO:tasks.rados.rados.0.smithi085.stdout:3206: finishing write tid 1 to smithi08575397-3 2021-02-10T04:40:37.559 INFO:tasks.rados.rados.0.smithi085.stdout:3206: finishing write tid 2 to smithi08575397-3 2021-02-10T04:40:37.559 INFO:tasks.rados.rados.0.smithi085.stdout:3206: finishing write tid 3 to smithi08575397-3 2021-02-10T04:40:37.559 INFO:tasks.rados.rados.0.smithi085.stdout:3206: finishing write tid 4 to smithi08575397-3 2021-02-10T04:40:37.559 INFO:tasks.rados.rados.0.smithi085.stdout:3206: finishing write tid 5 to smithi08575397-3 2021-02-10T04:40:37.559 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 1554 (ObjNum 1096 snap 368 seq_num 1096) dirty exists 2021-02-10T04:40:37.559 INFO:tasks.rados.rados.0.smithi085.stdout:3206: left oid 3 (ObjNum 1096 snap 368 seq_num 1096) 2021-02-10T04:40:37.559 INFO:tasks.rados.rados.0.smithi085.stdout:3207: expect (ObjNum 1052 snap 345 seq_num 1052) 2021-02-10T04:40:37.559 INFO:tasks.rados.rados.0.smithi085.stdout:3208: finishing write tid 1 to smithi08575397-31 2021-02-10T04:40:37.560 INFO:tasks.rados.rados.0.smithi085.stdout:3208: finishing write tid 2 to smithi08575397-31 2021-02-10T04:40:37.560 INFO:tasks.rados.rados.0.smithi085.stdout:3208: finishing write tid 3 to smithi08575397-31 2021-02-10T04:40:37.560 INFO:tasks.rados.rados.0.smithi085.stdout:3208: finishing write tid 4 to smithi08575397-31 2021-02-10T04:40:37.560 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 1469 (ObjNum 1097 snap 368 seq_num 1097) dirty exists 2021-02-10T04:40:37.560 INFO:tasks.rados.rados.0.smithi085.stdout:3208: left oid 31 (ObjNum 1097 snap 368 seq_num 1097) 2021-02-10T04:40:37.560 INFO:tasks.rados.rados.0.smithi085.stdout:3209: finishing write tid 1 to smithi08575397-6 2021-02-10T04:40:37.560 INFO:tasks.rados.rados.0.smithi085.stdout:3209: finishing write tid 2 to smithi08575397-6 2021-02-10T04:40:37.560 INFO:tasks.rados.rados.0.smithi085.stdout:3209: finishing write tid 3 to smithi08575397-6 2021-02-10T04:40:37.560 INFO:tasks.rados.rados.0.smithi085.stdout:3209: finishing write tid 4 to smithi08575397-6 2021-02-10T04:40:37.560 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 1561 (ObjNum 1098 snap 368 seq_num 1098) dirty exists 2021-02-10T04:40:37.560 INFO:tasks.rados.rados.0.smithi085.stdout:3209: left oid 6 (ObjNum 1098 snap 368 seq_num 1098) 2021-02-10T04:40:37.560 INFO:tasks.rados.rados.0.smithi085.stdout:3205: done (9 left) 2021-02-10T04:40:37.561 INFO:tasks.rados.rados.0.smithi085.stdout:3206: done (8 left) 2021-02-10T04:40:37.561 INFO:tasks.rados.rados.0.smithi085.stdout:3207: done (7 left) 2021-02-10T04:40:37.561 INFO:tasks.rados.rados.0.smithi085.stdout:3208: done (6 left) 2021-02-10T04:40:37.561 INFO:tasks.rados.rados.0.smithi085.stdout:3209: done (5 left) 2021-02-10T04:40:37.561 INFO:tasks.rados.rados.0.smithi085.stdout:3210: done (4 left) 2021-02-10T04:40:37.561 INFO:tasks.rados.rados.0.smithi085.stdout:3211: read oid 28 snap -1 2021-02-10T04:40:37.561 INFO:tasks.rados.rados.0.smithi085.stdout:3211: expect (ObjNum 1052 snap 345 seq_num 1052) 2021-02-10T04:40:37.561 INFO:tasks.rados.rados.0.smithi085.stdout:3212: read oid 15 snap -1 2021-02-10T04:40:37.561 INFO:tasks.rados.rados.0.smithi085.stdout:3212: expect deleted 2021-02-10T04:40:37.561 INFO:tasks.rados.rados.0.smithi085.stdout:3213: copy_from oid 1 from oid 46 current snap is 369 2021-02-10T04:40:37.561 INFO:tasks.rados.rados.0.smithi085.stdout:3214: read oid 9 snap -1 2021-02-10T04:40:37.561 INFO:tasks.rados.rados.0.smithi085.stdout:3214: expect (ObjNum 1053 snap 346 seq_num 1053) 2021-02-10T04:40:37.561 INFO:tasks.rados.rados.0.smithi085.stdout:3215: copy_from oid 2 from oid 34 current snap is 369 2021-02-10T04:40:37.562 INFO:tasks.rados.rados.0.smithi085.stdout:3216: write (excl) oid 40 current snap is 369 2021-02-10T04:40:37.562 INFO:tasks.rados.rados.0.smithi085.stdout:3216: seq_num 1099 ranges {470893=653483,1727593=522054} 2021-02-10T04:40:37.562 INFO:tasks.rados.rados.0.smithi085.stdout:3216: writing smithi08575397-40 from 470893 to 1124376 tid 1 2021-02-10T04:40:37.562 INFO:tasks.rados.rados.0.smithi085.stdout:3216: writing smithi08575397-40 from 1727593 to 2249647 tid 2 2021-02-10T04:40:37.562 INFO:tasks.rados.rados.0.smithi085.stdout:3217: write (excl) oid 38 current snap is 369 2021-02-10T04:40:37.562 INFO:tasks.rados.rados.0.smithi085.stdout:3217: seq_num 1100 ranges {487700=728732,1786075=665712} 2021-02-10T04:40:37.562 INFO:tasks.rados.rados.0.smithi085.stdout:3217: writing smithi08575397-38 from 487700 to 1216432 tid 1 2021-02-10T04:40:37.562 INFO:tasks.rados.rados.0.smithi085.stdout:3217: writing smithi08575397-38 from 1786075 to 2451787 tid 2 2021-02-10T04:40:37.562 INFO:tasks.rados.rados.0.smithi085.stdout:3211: expect (ObjNum 1052 snap 345 seq_num 1052) 2021-02-10T04:40:37.562 INFO:tasks.rados.rados.0.smithi085.stdout:3214: expect (ObjNum 1053 snap 346 seq_num 1053) 2021-02-10T04:40:37.562 INFO:tasks.rados.rados.0.smithi085.stdout:3216: finishing write tid 1 to smithi08575397-40 2021-02-10T04:40:37.562 INFO:tasks.rados.rados.0.smithi085.stdout:3211: done (10 left) 2021-02-10T04:40:37.562 INFO:tasks.rados.rados.0.smithi085.stdout:3212: done (9 left) 2021-02-10T04:40:37.563 INFO:tasks.rados.rados.0.smithi085.stdout:3214: done (8 left) 2021-02-10T04:40:37.563 INFO:tasks.rados.rados.0.smithi085.stdout:3218: copy_from oid 50 from oid 35 current snap is 369 2021-02-10T04:40:37.563 INFO:tasks.rados.rados.0.smithi085.stdout:3219: read oid 36 snap -1 2021-02-10T04:40:37.563 INFO:tasks.rados.rados.0.smithi085.stdout:3219: expect (ObjNum 991 snap 326 seq_num 991) 2021-02-10T04:40:37.563 INFO:tasks.rados.rados.0.smithi085.stdout:3220: snap_create 2021-02-10T04:40:37.563 INFO:tasks.rados.rados.0.smithi085.stdout:3216: finishing write tid 2 to smithi08575397-40 2021-02-10T04:40:37.563 INFO:tasks.rados.rados.0.smithi085.stdout:3216: finishing write tid 3 to smithi08575397-40 2021-02-10T04:40:37.563 INFO:tasks.rados.rados.0.smithi085.stdout:3216: finishing write tid 4 to smithi08575397-40 2021-02-10T04:40:37.563 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 1670 (ObjNum 1099 snap 369 seq_num 1099) dirty exists 2021-02-10T04:40:37.563 INFO:tasks.rados.rados.0.smithi085.stdout:3216: left oid 40 (ObjNum 1099 snap 369 seq_num 1099) 2021-02-10T04:40:37.563 INFO:tasks.rados.rados.0.smithi085.stdout:3217: finishing write tid 1 to smithi08575397-38 2021-02-10T04:40:37.563 INFO:tasks.rados.rados.0.smithi085.stdout:3215: finishing copy_from to smithi08575397-2 2021-02-10T04:40:37.563 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 1439 (ObjNum 1052 snap 345 seq_num 1052) dirty exists 2021-02-10T04:40:37.564 INFO:tasks.rados.rados.0.smithi085.stdout:3215: finishing copy_from racing read to smithi08575397-2 2021-02-10T04:40:37.564 INFO:tasks.rados.rados.0.smithi085.stdout:3217: finishing write tid 2 to smithi08575397-38 2021-02-10T04:40:37.564 INFO:tasks.rados.rados.0.smithi085.stdout:3217: finishing write tid 3 to smithi08575397-38 2021-02-10T04:40:37.564 INFO:tasks.rados.rados.0.smithi085.stdout:3217: finishing write tid 4 to smithi08575397-38 2021-02-10T04:40:37.564 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 1564 (ObjNum 1100 snap 369 seq_num 1100) dirty exists 2021-02-10T04:40:37.564 INFO:tasks.rados.rados.0.smithi085.stdout:3217: left oid 38 (ObjNum 1100 snap 369 seq_num 1100) 2021-02-10T04:40:37.564 INFO:tasks.rados.rados.0.smithi085.stdout:3213: finishing copy_from to smithi08575397-1 2021-02-10T04:40:37.564 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 1552 (ObjNum 1087 snap 364 seq_num 1087) dirty exists 2021-02-10T04:40:37.564 INFO:tasks.rados.rados.0.smithi085.stdout:3213: finishing copy_from racing read to smithi08575397-1 2021-02-10T04:40:37.564 INFO:tasks.rados.rados.0.smithi085.stdout:3218: finishing copy_from to smithi08575397-50 2021-02-10T04:40:37.564 INFO:tasks.rados.rados.0.smithi085.stdout:3218: got expected ENOENT (src dne) 2021-02-10T04:40:37.564 INFO:tasks.rados.rados.0.smithi085.stdout:3218: finishing copy_from racing read to smithi08575397-50 2021-02-10T04:40:37.564 INFO:tasks.rados.rados.0.smithi085.stdout:3219: expect (ObjNum 991 snap 326 seq_num 991) 2021-02-10T04:40:37.565 INFO:tasks.rados.rados.0.smithi085.stdout:3213: done (10 left) 2021-02-10T04:40:37.565 INFO:tasks.rados.rados.0.smithi085.stdout:3215: done (9 left) 2021-02-10T04:40:37.565 INFO:tasks.rados.rados.0.smithi085.stdout:3216: done (8 left) 2021-02-10T04:40:37.565 INFO:tasks.rados.rados.0.smithi085.stdout:3217: done (7 left) 2021-02-10T04:40:37.565 INFO:tasks.rados.rados.0.smithi085.stdout:3218: done (6 left) 2021-02-10T04:40:37.565 INFO:tasks.rados.rados.0.smithi085.stdout:3219: done (5 left) 2021-02-10T04:40:37.565 INFO:tasks.rados.rados.0.smithi085.stdout:3220: done (4 left) 2021-02-10T04:40:37.565 INFO:tasks.rados.rados.0.smithi085.stdout:3221: snap_create 2021-02-10T04:40:37.565 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[29319]: cluster 2021-02-10T04:40:37.565 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[29319]: 2021-02-10T04:40:37.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[29319]: 2021 2021-02-10T04:40:37.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[29319]: -02-10T04:40:34.394491 2021-02-10T04:40:37.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1351 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:37.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[29319]: cluster 2021-02-10T04:40:34 2021-02-10T04:40:37.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[29319]: .394500+0000 osd.4 (osd.4) 1352 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:37.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[29319]: 2021-02-10T04:40:37.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[29319]: cluster 2021-02-10T04:40:37.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:40:37.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[29319]: :40:34.394507+0000 osd.4 (osd 2021-02-10T04:40:37.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[29319]: .4) 1353 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:37.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:40:37.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[29319]: 10T04:40:34.394515+0000 osd.4 ( 2021-02-10T04:40:37.567 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[29319]: osd.4) 1354 : cluster [WRN] 2021-02-10T04:40:37.567 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:37.567 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:40:37.567 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[29319]: 02-10T04:40:35.435744+0000 mon.a 2021-02-10T04:40:37.567 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[29319]: (mon.0) 1758 : cluster [DBG] osdmap e816: 8 total, 8 up, 7 in 2021-02-10T04:40:37.567 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:36 smithi085 conmon[29319]: cluster 2021-02-10T04:40:35.374090+0000 osd.4 (osd.4) 1355 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:37.567 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:36 smithi085 conmon[29319]: cluster 2021-02-10T04:40:35.374100+0000 osd.4 (osd.4) 1356 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:37.567 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:36 smithi085 conmon[29319]: cluster 2021-02-10T04:40:35.374103+0000 osd.4 (osd.4) 1357 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:37.567 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:36 smithi085 conmon[29319]: cluster 2021-02-10T04:40:35.374107+0000 osd.4 (osd.4) 1358 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:37.567 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:36 smithi085 conmon[29319]: cluster 2021-02-10T04:40:36.196509+0000 mgr.y (mgr.14140) 791 : cluster [DBG] pgmap v1375: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 533 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 7.0 MiB/s rd, 898 KiB/s wr, 6 op/s 2021-02-10T04:40:37.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[35961]: cluster 2021-02-10T04:40:34.394491+0000 osd.4 (osd.4) 1351 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:37.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[35961]: cluster 2021-02-10T04:40:34.394500+0000 osd.4 (osd.4) 1352 : 2021-02-10T04:40:37.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:37.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[35961]: cluster 2021-02-10T04:40:34.394507 2021-02-10T04:40:37.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1353 : cluster [WRN] 2021-02-10T04:40:37.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:37.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[35961]: cluster 2021-02-10T04:40:37.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[35961]: 2021-02-10T04:40:34.394515+0000 osd.4 (osd.4 2021-02-10T04:40:37.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[35961]: ) 1354 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:37.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:40:37.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[35961]: 04:40:35.435744+0000 mon.a (mon. 2021-02-10T04:40:37.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:35 smithi085 conmon[35961]: 0) 1758 : cluster [DBG] osdmap e816: 8 total, 8 up, 7 in 2021-02-10T04:40:37.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:36 smithi085 conmon[35961]: cluster 2021-02-10T04:40:37.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:36 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:40:37.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:36 smithi085 conmon[35961]: 10T04:40:35.374090+0000 osd.4 (osd.4) 1355 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:37.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:36 smithi085 conmon[35961]: cluster 2021-02-10T04:40:37.569 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:36 smithi085 conmon[35961]: 2021-02 2021-02-10T04:40:37.569 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:36 smithi085 conmon[35961]: -10T04:40:35.374100+0000 osd.4 (osd.4 2021-02-10T04:40:37.569 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:36 smithi085 conmon[35961]: ) 1356 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:37.569 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:36 smithi085 conmon[35961]: cluster 2021-02-10T04:40:37.569 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:36 smithi085 conmon[35961]: 2021-02-10T04:40:35.374103+0000 2021-02-10T04:40:37.569 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:36 smithi085 conmon[35961]: osd.4 (osd.4) 1357 : cluster [WRN] 2021-02-10T04:40:37.569 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:36 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:37.569 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:36 smithi085 conmon[35961]: cluster 2021-02-10T04:40: 2021-02-10T04:40:37.569 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:36 smithi085 conmon[35961]: 35.374107+0000 osd.4 (osd.4) 1358 : 2021-02-10T04:40:37.569 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:36 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:37.569 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:36 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:40:37.569 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:36 smithi085 conmon[35961]: :40:36.196509+0000 mgr.y (mgr. 2021-02-10T04:40:37.569 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:36 smithi085 conmon[35961]: 14140) 791 : cluster [DBG] pgmap v1375: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 533 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 7.0 MiB/s rd, 898 KiB/s wr, 6 op/s 2021-02-10T04:40:37.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:37 smithi164 conmon[29288]: cluster 2021-02-10T04:40:36.399447 2021-02-10T04:40:37.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:37 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1359 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:37.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:37 smithi164 conmon[29288]: cluster 2021-02-10T04:40:37.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:37 smithi164 conmon[29288]: 2021 2021-02-10T04:40:37.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:37 smithi164 conmon[29288]: -02-10T04:40:36 2021-02-10T04:40:37.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:37 smithi164 conmon[29288]: .399457+0000 osd.4 (osd.4) 1360 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:37.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:37 smithi164 conmon[29288]: cluster 2021-02-10T04:40: 2021-02-10T04:40:37.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:37 smithi164 conmon[29288]: 36.399461+0000 osd.4 (osd.4) 1361 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:37.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:37 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:40:37.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:37 smithi164 conmon[29288]: -02-10T04:40:36.399464+0000 osd.4 (osd 2021-02-10T04:40:37.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:37 smithi164 conmon[29288]: .4) 1362 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:37.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:37 smithi164 conmon[29288]: cluster 2021-02-10T04:40:36.466904+0000 mon.a (mon.0) 1759 : cluster 2021-02-10T04:40:37.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:37 smithi164 conmon[29288]: [DBG] osdmap e817: 8 total, 8 up, 7 in 2021-02-10T04:40:37.645 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:37 smithi164 conmon[33677]: debug 2021-02-10T04:40:37.362+0000 7fbec19a1700 -1 osd.4 817 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:37.671 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:40:37.709 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:40:37.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:37 smithi085 conmon[29319]: cluster 2021-02-10T04:40:36.399447+0000 osd.4 ( 2021-02-10T04:40:37.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:37 smithi085 conmon[29319]: osd.4) 1359 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:37.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:37 smithi085 conmon[29319]: cluster 2021-02-10T04:40:36.399457+0000 osd.4 (osd.4) 1360 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:37.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:37 smithi085 conmon[29319]: cluster 2021-02-10T04:40:36.399461+0000 osd.4 (osd.4) 1361 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:37.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:37 smithi085 conmon[29319]: cluster 2021-02-10T04:40:36.399464+0000 osd.4 (osd.4) 1362 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:37.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:37 smithi085 conmon[29319]: cluster 2021-02-10T04:40:36.466904+0000 mon.a (mon.0) 1759 : cluster [DBG] osdmap e817: 8 total, 8 up, 7 in 2021-02-10T04:40:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:37 smithi085 conmon[35961]: cluster 2021-02-10T04:40:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:37 smithi085 conmon[35961]: 2021-02-10T04:40:37.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:37 smithi085 conmon[35961]: 2021 2021-02-10T04:40:37.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:37 smithi085 conmon[35961]: -02-10T04:40:36. 2021-02-10T04:40:37.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:37 smithi085 conmon[35961]: 399447+0000 osd.4 (osd.4) 1359 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:37.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:37 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:40:37.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:37 smithi085 conmon[35961]: 40:36.399457+0000 osd.4 (osd.4) 1360 : 2021-02-10T04:40:37.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:37 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:37.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:37 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:40:37.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:37 smithi085 conmon[35961]: 04:40:36.399461+0000 osd.4 (osd.4 2021-02-10T04:40:37.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:37 smithi085 conmon[35961]: ) 1361 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:37.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:37 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:40:37.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:37 smithi085 conmon[35961]: :40:36.399464+0000 osd.4 (osd.4) 2021-02-10T04:40:37.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:37 smithi085 conmon[35961]: 1362 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:37.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:37 smithi085 conmon[35961]: cluster 2021-02-10T04:40:37.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:37 smithi085 conmon[35961]: 2021-02-10T04:40:36 2021-02-10T04:40:37.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:37 smithi085 conmon[35961]: .466904+0000 mon.a (mon.0) 1759 : cluster 2021-02-10T04:40:37.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:37 smithi085 conmon[35961]: [DBG] osdmap e817: 8 total, 8 up, 7 in 2021-02-10T04:40:38.278 INFO:tasks.rados.rados.0.smithi085.stdout:3221: done (4 left) 2021-02-10T04:40:38.278 INFO:tasks.rados.rados.0.smithi085.stdout:3222: write oid 26 current snap is 371 2021-02-10T04:40:38.278 INFO:tasks.rados.rados.0.smithi085.stdout:3222: seq_num 1101 ranges {504507=403981,1328204=409370,2252362=252145} 2021-02-10T04:40:38.281 INFO:tasks.rados.rados.0.smithi085.stdout:3222: writing smithi08575397-26 from 504507 to 908488 tid 1 2021-02-10T04:40:38.286 INFO:tasks.rados.rados.0.smithi085.stdout:3222: writing smithi08575397-26 from 1328204 to 1737574 tid 2 2021-02-10T04:40:38.291 INFO:tasks.rados.rados.0.smithi085.stdout:3222: writing smithi08575397-26 from 2252362 to 2504507 tid 3 2021-02-10T04:40:38.291 INFO:tasks.rados.rados.0.smithi085.stdout:3222: finishing write tid 1 to smithi08575397-26 2021-02-10T04:40:38.291 INFO:tasks.rados.rados.0.smithi085.stdout:3223: read oid 4 snap -1 2021-02-10T04:40:38.291 INFO:tasks.rados.rados.0.smithi085.stdout:3223: expect (ObjNum 1049 snap 344 seq_num 1049) 2021-02-10T04:40:38.291 INFO:tasks.rados.rados.0.smithi085.stdout:3224: snap_create 2021-02-10T04:40:38.291 INFO:tasks.rados.rados.0.smithi085.stdout:3222: finishing write tid 2 to smithi08575397-26 2021-02-10T04:40:38.304 INFO:tasks.rados.rados.0.smithi085.stdout:3223: expect (ObjNum 1049 snap 344 seq_num 1049) 2021-02-10T04:40:38.462 INFO:tasks.rados.rados.0.smithi085.stdout:3222: finishing write tid 3 to smithi08575397-26 2021-02-10T04:40:38.463 INFO:tasks.rados.rados.0.smithi085.stdout:3222: finishing write tid 4 to smithi08575397-26 2021-02-10T04:40:38.463 INFO:tasks.rados.rados.0.smithi085.stdout:3222: finishing write tid 5 to smithi08575397-26 2021-02-10T04:40:38.463 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 1355 (ObjNum 1101 snap 371 seq_num 1101) dirty exists 2021-02-10T04:40:38.463 INFO:tasks.rados.rados.0.smithi085.stdout:3222: left oid 26 (ObjNum 1101 snap 371 seq_num 1101) 2021-02-10T04:40:38.559 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:40:38.559 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [], 2021-02-10T04:40:38.559 INFO:teuthology.orchestra.run.smithi085.stdout: "num_ops": 0 2021-02-10T04:40:38.559 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:40:38.644 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:40:38.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[33677]: debug 2021-02-10T04:40:38.312+0000 7fbec19a1700 -1 osd.4 818 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[29288]: cluster 2021-02-10T04:40:37.364123+0000 osd.4 (osd.4) 2021-02-10T04:40:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[29288]: 1363 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[29288]: cluster 2021-02-10T04:40:37.364133+0000 osd.4 (osd.4) 1364 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[29288]: cluster 2021-02-10T04:40:37.364139+0000 osd.4 (osd.4) 1365 : cluster 2021-02-10T04:40:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[29288]: cluster 2021-02-10T04:40 2021-02-10T04:40:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[29288]: :37.364145+0000 osd.4 (osd. 2021-02-10T04:40:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[29288]: 4) 1366 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:40:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[29288]: 02-10T04:40:37.475698 2021-02-10T04:40:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1760 2021-02-10T04:40:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[29288]: : cluster [DBG] osdmap e818: 8 total, 8 up, 7 in 2021-02-10T04:40:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[29288]: audit 2021-02-10T04:40:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:40:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[29288]: 40:38.132727+0000 mon.a ( 2021-02-10T04:40:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[29288]: mon.0) 1761 : audit [INF] 2021-02-10T04:40:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:40:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:40:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[29288]: -10T04:40:38. 2021-02-10T04:40:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[29288]: 132912+0000 mon.a (mon.0) 2021-02-10T04:40:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[29288]: 1762 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:40:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:40:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[29288]: -10T04:40:38.197096 2021-02-10T04:40:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 792 2021-02-10T04:40:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[29288]: : cluster [DBG] pgmap v1378: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 527 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 8.6 MiB/s rd, 898 KiB/s wr, 4 op/s 2021-02-10T04:40:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:40:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[29288]: -02-10T04:40: 2021-02-10T04:40:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[29288]: 38.274644+0000 mon.a (mon. 2021-02-10T04:40:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[29288]: 0) 1763 : cluster [DBG] osdmap e819: 8 total, 8 up, 7 in 2021-02-10T04:40:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:38 smithi164 conmon[29288]: 2021-02-10T04:40:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[29319]: cluster 2021-02-10T04:40:37.364123+0000 2021-02-10T04:40:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[29319]: osd.4 (osd.4) 1363 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[29319]: cluster 2021-02-10T04:40:37.364133+0000 osd.4 (osd.4) 1364 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[29319]: cluster 2021-02-10T04:40:37.364139+0000 osd.4 (osd.4) 1365 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:40:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[29319]: -10T04:40:37.364145+0000 osd.4 (osd.4) 1366 : cluster [WRN] 2021-02-10T04:40:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:40:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[29319]: :40:37.475698+0000 mon.a (mon.0) 2021-02-10T04:40:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[29319]: 1760 : cluster [DBG] osdmap e818: 8 total, 8 up, 7 in 2021-02-10T04:40:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[29319]: audit 2021-02-10T04:40:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:40:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[29319]: :40:38.132727+0000 mon.a (mon. 2021-02-10T04:40:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[29319]: 0) 1761 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:40:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[29319]: audit 2021-02-10T04:40:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[29319]: 2021-02-10T04:40 2021-02-10T04:40:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[29319]: :38.132912+0000 mon.a (mon.0) 2021-02-10T04:40:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[29319]: 1762 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:40:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:40:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[29319]: 02-10T04:40:38.197096+0000 mgr.y 2021-02-10T04:40:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[29319]: (mgr.14140) 792 : cluster [DBG] pgmap v1378: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 527 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 8.6 MiB/s rd, 898 KiB/s wr, 4 op/s 2021-02-10T04:40:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:40:38.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[29319]: 04:40:38.274644+0000 mon.a (mon.0) 1763 : 2021-02-10T04:40:38.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[29319]: cluster [DBG] osdmap e819: 8 total, 8 up, 7 in 2021-02-10T04:40:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: cluster 2021-02-10T04:40:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: 2021-02-10T04:40:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: 2021 2021-02-10T04:40:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: - 2021-02-10T04:40:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: 02 2021-02-10T04:40:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: - 2021-02-10T04:40:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: 10 2021-02-10T04:40:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: T 2021-02-10T04:40:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: 04:40:37.364123+0000 2021-02-10T04:40:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: osd.4 (osd.4) 1363 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: cluster 2021-02-10T04:40:37. 2021-02-10T04:40:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: 364133+0000 osd.4 (osd.4) 1364 : cluster 2021-02-10T04:40:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: cluster 2021-02-10T04:40:37.364139+0000 2021-02-10T04:40:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: osd.4 (osd.4) 1365 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: 2021-02-10T04:40:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: cluster 2021-02-10T04:40:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: 2021-02-10T04:40 2021-02-10T04:40:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: :37.364145+0000 osd.4 (osd.4) 2021-02-10T04:40:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: 1366 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: cluster 2021-02-10T04:40:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: 2021 2021-02-10T04:40:38.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: - 2021-02-10T04:40:38.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: 02 2021-02-10T04:40:38.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: - 2021-02-10T04:40:38.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: 10 2021-02-10T04:40:38.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: T 2021-02-10T04:40:38.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: 04 2021-02-10T04:40:38.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: : 2021-02-10T04:40:38.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: 40 2021-02-10T04:40:38.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: : 2021-02-10T04:40:38.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: 37 2021-02-10T04:40:38.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: .475698+0000 mon.a 2021-02-10T04:40:38.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: (mon.0) 1760 : cluster [DBG] osdmap e818: 8 total, 8 up, 7 in 2021-02-10T04:40:38.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: audit 2021-02-10T04:40:38.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: 2021-02-10T04:40:38.132727+0000 mon.a (mon.0) 1761 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:40:38.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: audit 2021-02-10T04:40:38.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: 2021-02-10T04:40:38.132912+0000 mon.a (mon.0) 1762 : audit 2021-02-10T04:40:38.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:40:38.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:40:38.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: 40:38.197096+0000 mgr.y (mgr.14140) 2021-02-10T04:40:38.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: 792 : cluster [DBG] pgmap v1378: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 527 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 8.6 MiB/s rd, 898 KiB/s wr, 4 op/s 2021-02-10T04:40:38.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:40:38.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: T04:40:38.274644 2021-02-10T04:40:38.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1763 : cluster [DBG] 2021-02-10T04:40:38.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:38 smithi085 conmon[35961]: osdmap e819: 8 total, 8 up, 7 in 2021-02-10T04:40:39.284 INFO:tasks.rados.rados.0.smithi085.stdout:3222: done (6 left) 2021-02-10T04:40:39.284 INFO:tasks.rados.rados.0.smithi085.stdout:3223: done (5 left) 2021-02-10T04:40:39.284 INFO:tasks.rados.rados.0.smithi085.stdout:3224: done (4 left) 2021-02-10T04:40:39.284 INFO:tasks.rados.rados.0.smithi085.stdout:3225: copy_from oid 8 from oid 3 current snap is 372 2021-02-10T04:40:39.284 INFO:tasks.rados.rados.0.smithi085.stdout:3226: copy_from oid 11 from oid 50 current snap is 372 2021-02-10T04:40:39.284 INFO:tasks.rados.rados.0.smithi085.stdout:3227: read oid 46 snap -1 2021-02-10T04:40:39.284 INFO:tasks.rados.rados.0.smithi085.stdout:3227: expect (ObjNum 1087 snap 364 seq_num 1087) 2021-02-10T04:40:39.284 INFO:tasks.rados.rados.0.smithi085.stdout:3228: snap_create 2021-02-10T04:40:39.288 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.3 to 1 (10000) 2021-02-10T04:40:39.290 INFO:tasks.rados.rados.0.smithi085.stdout:3227: expect (ObjNum 1087 snap 364 seq_num 1087) 2021-02-10T04:40:39.412 INFO:tasks.rados.rados.0.smithi085.stdout:3226: finishing copy_from to smithi08575397-11 2021-02-10T04:40:39.413 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 1549 (ObjNum 1054 snap 346 seq_num 1054) dirty exists 2021-02-10T04:40:39.413 INFO:tasks.rados.rados.0.smithi085.stdout:3226: finishing copy_from racing read to smithi08575397-11 2021-02-10T04:40:39.413 INFO:tasks.rados.rados.0.smithi085.stdout:3225: finishing copy_from to smithi08575397-8 2021-02-10T04:40:39.413 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 1554 (ObjNum 1096 snap 368 seq_num 1096) dirty exists 2021-02-10T04:40:39.413 INFO:tasks.rados.rados.0.smithi085.stdout:3225: finishing copy_from racing read to smithi08575397-8 2021-02-10T04:40:39.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:39 smithi164 conmon[33677]: debug 2021-02-10T04:40:39.311+0000 7fbec19a1700 -1 osd.4 819 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:39 smithi164 conmon[29288]: cluster 2021-02-10T04:40:38.314594+0000 2021-02-10T04:40:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:39 smithi164 conmon[29288]: osd.4 (osd.4) 1367 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:39 smithi164 conmon[29288]: cluster 2021-02-10T04:40:38.314604+0000 osd.4 (osd.4) 1368 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:39 smithi164 conmon[29288]: cluster 2021-02-10T04:40:38.314611+0000 2021-02-10T04:40:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:39 smithi164 conmon[29288]: osd.4 (osd.4) 1369 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:39 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:40:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:39 smithi164 conmon[29288]: 10T04:40:38.314618+0000 osd.4 (osd.4) 2021-02-10T04:40:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:39 smithi164 conmon[29288]: 1370 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:39.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:39 smithi164 conmon[29288]: audit 2021-02-10T04:40:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:39 smithi164 conmon[29288]: 2021-02 2021-02-10T04:40:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:39 smithi164 conmon[29288]: -10T04:40:38.407052+0000 mgr.y 2021-02-10T04:40:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:39 smithi164 conmon[29288]: (mgr.14140) 793 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:40:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:39 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:40:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:39 smithi164 conmon[29288]: 02-10T04:40:38.473112+0000 mon.a (mon. 2021-02-10T04:40:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:39 smithi164 conmon[29288]: 0) 1764 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 667 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:40:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:39 smithi164 conmon[29288]: audit 2021-02-10T04:40:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:39 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:40:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:39 smithi164 conmon[29288]: 40:38.817200+0000 mon.a (mon.0) 2021-02-10T04:40:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:39 smithi164 conmon[29288]: 1765 : audit [INF] from='client.? 172.21.15.85:0/2667609075' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-02-10T04:40:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:39 smithi164 conmon[29288]: audit 2021-02-10T04:40:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:39 smithi164 conmon[29288]: 2021-02-10T04:40:39.283800+0000 2021-02-10T04:40:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:39 smithi164 conmon[29288]: mon.a (mon.0) 1766 : audit [INF] 2021-02-10T04:40:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:39 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/2667609075' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2021-02-10T04:40:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:39 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:40:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:39 smithi164 conmon[29288]: 40:39.283858+0000 mon.a (mon.0) 2021-02-10T04:40:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:39 smithi164 conmon[29288]: 1767 : cluster [DBG] osdmap e820: 8 total, 8 up, 7 in 2021-02-10T04:40:39.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[29319]: cluster 2021-02-10T04:40:38.314594+0000 osd.4 (osd.4 2021-02-10T04:40:39.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[29319]: ) 1367 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:39.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[29319]: cluster 2021-02-10T04:40:38.314604+0000 osd.4 (osd.4) 1368 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:39.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[29319]: cluster 2021-02-10T04:40:38.314611+0000 osd.4 (osd.4) 1369 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:39.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[29319]: cluster 2021-02-10T04:40:38.314618+0000 osd.4 (osd.4 2021-02-10T04:40:39.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[29319]: ) 1370 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:39.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[29319]: audit 2021-02-10T04:40 2021-02-10T04:40:39.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[29319]: :38.407052+0000 mgr.y (mgr.14140) 793 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:40:39.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:40:39.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[29319]: -10T04:40:38.473112+0000 mon.a (mon.0) 2021-02-10T04:40:39.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[29319]: 1764 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 667 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:40:39.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:40:39.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[29319]: 10T04:40:38.817200+0000 mon.a (mon.0) 1765 : audit 2021-02-10T04:40:39.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[29319]: [INF] from='client.? 172.21.15.85:0/2667609075' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-02-10T04:40:39.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[29319]: audit 2021-02-10T04: 2021-02-10T04:40:39.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[29319]: 40:39.283800+0000 mon.a (mon.0) 1766 2021-02-10T04:40:39.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[29319]: : audit [INF] from='client.? 172.21.15.85:0/2667609075' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2021-02-10T04:40:39.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:40:39.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[29319]: -10T04:40:39.283858+0000 mon.a 2021-02-10T04:40:39.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[29319]: (mon.0) 1767 : cluster [DBG] osdmap e820: 8 total, 8 up, 7 in 2021-02-10T04:40:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[35961]: cluster 2021-02-10T04:40:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[35961]: 2021-02-10T04:40:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[35961]: 2021-02-10T04:40:38. 2021-02-10T04:40:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[35961]: 314594+0000 osd.4 (osd.4) 1367 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[35961]: cluster 2021-02-10T04:40 2021-02-10T04:40:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[35961]: :38.314604+0000 osd.4 (osd.4) 1368 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[35961]: 2021-02-10T04:40:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:40:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[35961]: -02-10T04:40:38.314611+0000 2021-02-10T04:40:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[35961]: osd.4 (osd.4) 1369 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:40:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[35961]: 04:40:38.314618+0000 osd.4 (osd.4) 1370 : cluster [WRN] 2021-02-10T04:40:39.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:39.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[35961]: audit 2021-02-10T04:40:38.407052+0000 mgr.y (mgr.14140) 793 : 2021-02-10T04:40:39.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[35961]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:40:39.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[35961]: cluster 2021-02-10T04:40:38 2021-02-10T04:40:39.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[35961]: .473112+0000 mon.a (mon.0) 1764 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 667 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:40:39.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[35961]: audit 2021-02-10T04:40:38.817200 2021-02-10T04:40:39.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1765 : audit [INF] from='client.? 172.21.15.85:0/2667609075' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-02-10T04:40:39.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[35961]: audit 2021-02-10T04:40 2021-02-10T04:40:39.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[35961]: :39.283800+0000 mon.a (mon.0) 1766 : audit [INF] 2021-02-10T04:40:39.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/2667609075' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2021-02-10T04:40:39.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[35961]: cluster 2021-02-10T04:40:39. 2021-02-10T04:40:39.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[35961]: 283858+0000 mon.a (mon.0) 1767 : cluster [DBG] 2021-02-10T04:40:39.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:39 smithi085 conmon[35961]: osdmap e820: 8 total, 8 up, 7 in 2021-02-10T04:40:40.285 INFO:tasks.rados.rados.0.smithi085.stdout:3225: done (7 left) 2021-02-10T04:40:40.285 INFO:tasks.rados.rados.0.smithi085.stdout:3226: done (6 left) 2021-02-10T04:40:40.285 INFO:tasks.rados.rados.0.smithi085.stdout:3227: done (5 left) 2021-02-10T04:40:40.285 INFO:tasks.rados.rados.0.smithi085.stdout:3228: done (4 left) 2021-02-10T04:40:40.285 INFO:tasks.rados.rados.0.smithi085.stdout:3229: snap_remove snap 358 2021-02-10T04:40:40.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:40 smithi164 conmon[33677]: debug 2021-02-10T04:40:40.289+0000 7fbec19a1700 -1 osd.4 821 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:40 smithi164 conmon[29288]: cluster 2021-02-10T04:40:39. 2021-02-10T04:40:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:40 smithi164 conmon[29288]: 313181+0000 osd.4 (osd.4) 1371 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:40 smithi164 conmon[29288]: cluster 2021-02-10T04:40:39.313189 2021-02-10T04:40:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:40 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1372 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:40 smithi164 conmon[29288]: cluster 2021-02-10T04:40:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:40 smithi164 conmon[29288]: 2021-02-10T04:40:39.313195+0000 osd.4 (osd.4 2021-02-10T04:40:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:40 smithi164 conmon[29288]: ) 1373 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:40 smithi164 conmon[29288]: cluster 2021-02-10T04:40:40.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:40 smithi164 conmon[29288]: 2021-02-10T04:40 2021-02-10T04:40:40.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:40 smithi164 conmon[29288]: :39.313199+0000 osd.4 (osd. 2021-02-10T04:40:40.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:40 smithi164 conmon[29288]: 4) 1374 : cluster [WRN] 2021-02-10T04:40:40.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:40 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:40.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:40 smithi164 conmon[29288]: cluster 2021-02-10T04:40 2021-02-10T04:40:40.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:40 smithi164 conmon[29288]: :40.197972+0000 mgr.y (mgr.14140) 794 : cluster [DBG] 2021-02-10T04:40:40.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:40 smithi164 conmon[29288]: pgmap v1381: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 535 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 21 MiB/s rd, 3.4 MiB/s wr, 25 op/s 2021-02-10T04:40:40.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:40 smithi164 conmon[29288]: cluster 2021-02-10T04:40:40.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:40 smithi164 conmon[29288]: 2021-02-10T04:40:40.284715+0000 mon.a (mon.0) 1768 : cluster [DBG] 2021-02-10T04:40:40.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:40 smithi164 conmon[29288]: osdmap e821: 8 total, 8 up, 7 in 2021-02-10T04:40:40.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[29319]: cluster 2021-02-10T04:40:40.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[29319]: 2021-02-10T04:40:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[29319]: 2021 2021-02-10T04:40:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[29319]: -02-10T04:40:39 2021-02-10T04:40:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[29319]: .313181+0000 osd.4 (osd.4) 1371 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:40:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[29319]: -02-10T04:40:39.313189+0000 osd.4 2021-02-10T04:40:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[29319]: (osd.4) 1372 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:40:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[29319]: 02-10T04:40:39.313195+0000 osd.4 ( 2021-02-10T04:40:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[29319]: osd.4) 1373 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[29319]: 2021-02-10T04:40:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:40:40.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[29319]: -10T04:40:39.313199+0000 osd.4 ( 2021-02-10T04:40:40.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[29319]: osd.4) 1374 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:40.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[29319]: cluster 2021-02-10T04:40:40.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[29319]: 2021-02-10T04:40:40.197972 2021-02-10T04:40:40.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 794 : cluster [DBG] pgmap v1381: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 535 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 21 MiB/s rd, 3.4 MiB/s wr, 25 op/s 2021-02-10T04:40:40.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:40:40.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[29319]: 02-10T04:40:40.284715+0000 mon.a 2021-02-10T04:40:40.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[29319]: (mon.0) 1768 : cluster [DBG] osdmap e821: 8 total, 8 up, 7 in 2021-02-10T04:40:40.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[35961]: cluster 2021-02-10T04:40:39.313181+0000 2021-02-10T04:40:40.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[35961]: osd.4 (osd.4) 1371 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:40.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[35961]: cluster 2021-02-10T04:40:40.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[35961]: 2021-02-10T04:40:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[35961]: 2021 2021-02-10T04:40:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[35961]: - 2021-02-10T04:40:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[35961]: 02 2021-02-10T04:40:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[35961]: - 2021-02-10T04:40:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[35961]: 10 2021-02-10T04:40:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[35961]: T 2021-02-10T04:40:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[35961]: 04:40:39.313189+0000 osd.4 2021-02-10T04:40:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[35961]: (osd.4) 1372 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:40:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[35961]: 40:39.313195+0000 osd.4 (osd.4) 1373 : cluster 2021-02-10T04:40:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[35961]: cluster 2021-02-10T04:40:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[35961]: 2021-02-10T04:40:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[35961]: 2021-02-10T04:40: 2021-02-10T04:40:40.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[35961]: 39.313199+0000 osd.4 (osd.4) 1374 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:40.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:40:40.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[35961]: 10T04:40:40.197972 2021-02-10T04:40:40.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 794 2021-02-10T04:40:40.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[35961]: : cluster [DBG] pgmap v1381: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 535 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 21 MiB/s rd, 3.4 MiB/s wr, 25 op/s 2021-02-10T04:40:40.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[35961]: cluster 2021-02-10T04:40:40.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:40:40.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:40 smithi085 conmon[35961]: :40:40.284715+0000 mon.a (mon.0) 1768 : cluster [DBG] osdmap e821: 8 total, 8 up, 7 in 2021-02-10T04:40:41.287 INFO:tasks.rados.rados.0.smithi085.stdout:3229: done (4 left) 2021-02-10T04:40:41.287 INFO:tasks.rados.rados.0.smithi085.stdout:3230: snap_remove snap 364 2021-02-10T04:40:41.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:41 smithi164 conmon[33677]: debug 2021-02-10T04:40:41.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:41 smithi164 conmon[33677]: 2021-02-10T04:40:41.271+0000 7fbec19a1700 -1 osd.4 821 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:41 smithi164 conmon[29288]: cluster 2021-02-10T04:40:40.291132+0000 osd.4 (osd.4) 1375 : cluster 2021-02-10T04:40:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:41 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:41 smithi164 conmon[29288]: cluster 2021-02-10T04:40:40.291142+0000 osd.4 (osd.4) 1376 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:41 smithi164 conmon[29288]: cluster 2021-02-10T04:40:40.291147+0000 osd.4 2021-02-10T04:40:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:41 smithi164 conmon[29288]: (osd.4) 1377 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:41 smithi164 conmon[29288]: 2021-02-10T04:40:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:41 smithi164 conmon[29288]: cluster 2021-02-10T04:40:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:41 smithi164 conmon[29288]: 2021-02-10T04:40:40.291153+0000 osd.4 (osd.4) 2021-02-10T04:40:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:41 smithi164 conmon[29288]: 1378 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:41 smithi164 conmon[29288]: cluster 2021-02-10T04:40:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:41 smithi164 conmon[29288]: 2021-02-10T04:40:41. 2021-02-10T04:40:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:41 smithi164 conmon[29288]: 286450+0000 mon.a (mon.0) 1769 : cluster [DBG] 2021-02-10T04:40:41.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:41 smithi164 conmon[29288]: osdmap e822: 8 total, 8 up, 7 in 2021-02-10T04:40:41.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[29319]: cluster 2021-02-10T04:40:40.291132+0000 2021-02-10T04:40:41.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[29319]: osd.4 (osd.4) 1375 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[29319]: cluster 2021-02-10T04:40:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:40:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[29319]: 10T04:40:40.291142+0000 osd.4 (osd 2021-02-10T04:40:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[29319]: .4) 1376 : cluster [WRN] 2021-02-10T04:40:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:40:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[29319]: -02-10T04:40:40 2021-02-10T04:40:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[29319]: .291147+0000 osd.4 (osd.4) 1377 : cluster [WRN] 2021-02-10T04:40:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:40:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[29319]: T04:40:40.291153+0000 osd.4 (osd.4) 1378 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:41.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:40:41.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[29319]: -02-10T04:40:41.286450+0000 mon.a (mon.0) 1769 : cluster [DBG] osdmap e822: 8 total, 8 up, 7 in 2021-02-10T04:40:41.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: cluster 2021-02-10T04:40:41.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: 2021-02-10T04:40:41.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: 2021 2021-02-10T04:40:41.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: - 2021-02-10T04:40:41.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: 02 2021-02-10T04:40:41.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: - 2021-02-10T04:40:41.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: 10 2021-02-10T04:40:41.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: T 2021-02-10T04:40:41.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: 04 2021-02-10T04:40:41.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: : 2021-02-10T04:40:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: 40 2021-02-10T04:40:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: :40.291132+0000 osd.4 (osd.4) 1375 : cluster 2021-02-10T04:40:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: cluster 2021-02-10T04:40:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: 2021-02-10T04:40:40.291142+0000 osd.4 (osd.4) 1376 : cluster 2021-02-10T04:40:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:40:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: T04:40:40.291147+0000 osd.4 (osd. 2021-02-10T04:40:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: 4) 1377 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:40:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: -10T04:40:40.291153+0000 osd.4 2021-02-10T04:40:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: (osd.4) 1378 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: cluster 2021-02-10T04:40:41.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: 2021-02-10T04:40:41.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: 2021 2021-02-10T04:40:41.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: - 2021-02-10T04:40:41.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: 02 2021-02-10T04:40:41.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: - 2021-02-10T04:40:41.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: 10 2021-02-10T04:40:41.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: T 2021-02-10T04:40:41.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: 04 2021-02-10T04:40:41.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: :40:41.286450+0000 mon.a ( 2021-02-10T04:40:41.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:41 smithi085 conmon[35961]: mon.0) 1769 : cluster [DBG] osdmap e822: 8 total, 8 up, 7 in 2021-02-10T04:40:42.288 INFO:tasks.rados.rados.0.smithi085.stdout:3230: done (4 left) 2021-02-10T04:40:42.288 INFO:tasks.rados.rados.0.smithi085.stdout:3231: delete oid 25 current snap is 373 2021-02-10T04:40:42.297 INFO:tasks.rados.rados.0.smithi085.stdout:3231: done (4 left) 2021-02-10T04:40:42.298 INFO:tasks.rados.rados.0.smithi085.stdout:3232: read oid 9 snap -1 2021-02-10T04:40:42.298 INFO:tasks.rados.rados.0.smithi085.stdout:3232: expect (ObjNum 1053 snap 346 seq_num 1053) 2021-02-10T04:40:42.298 INFO:tasks.rados.rados.0.smithi085.stdout:3233: read oid 1 snap -1 2021-02-10T04:40:42.298 INFO:tasks.rados.rados.0.smithi085.stdout:3233: expect (ObjNum 1087 snap 364 seq_num 1087) 2021-02-10T04:40:42.298 INFO:tasks.rados.rados.0.smithi085.stdout:3234: write (excl) oid 20 current snap is 373 2021-02-10T04:40:42.298 INFO:tasks.rados.rados.0.smithi085.stdout:3234: seq_num 1103 ranges {538121=670832,1961521=413039} 2021-02-10T04:40:42.304 INFO:tasks.rados.rados.0.smithi085.stdout:3234: writing smithi08575397-20 from 538121 to 1208953 tid 1 2021-02-10T04:40:42.312 INFO:tasks.rados.rados.0.smithi085.stdout:3234: writing smithi08575397-20 from 1961521 to 2374560 tid 2 2021-02-10T04:40:42.312 INFO:tasks.rados.rados.0.smithi085.stdout:3235: write oid 8 current snap is 373 2021-02-10T04:40:42.312 INFO:tasks.rados.rados.0.smithi085.stdout:3235: seq_num 1104 ranges {554928=746081,2020003=534925} 2021-02-10T04:40:42.320 INFO:tasks.rados.rados.0.smithi085.stdout:3235: writing smithi08575397-8 from 554928 to 1301009 tid 1 2021-02-10T04:40:42.328 INFO:tasks.rados.rados.0.smithi085.stdout:3235: writing smithi08575397-8 from 2020003 to 2554928 tid 2 2021-02-10T04:40:42.328 INFO:tasks.rados.rados.0.smithi085.stdout:3236: rollback oid 43 current snap is 373 2021-02-10T04:40:42.328 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 43 to 356 2021-02-10T04:40:42.328 INFO:tasks.rados.rados.0.smithi085.stdout:3232: expect (ObjNum 1053 snap 346 seq_num 1053) 2021-02-10T04:40:42.600 INFO:tasks.rados.rados.0.smithi085.stdout:3233: expect (ObjNum 1087 snap 364 seq_num 1087) 2021-02-10T04:40:42.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:42 smithi164 conmon[33677]: debug 2021-02-10T04:40:42.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:42 smithi164 conmon[33677]: 2021-02-10T04:40:42.262+0000 7fbec19a1700 -1 osd.4 822 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:42.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:42 smithi164 conmon[29288]: cluster 2021-02-10T04:40:41.273181+0000 osd.4 (osd.4) 1379 : cluster [WRN] 2021-02-10T04:40:42.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:42 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:42.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:42 smithi164 conmon[29288]: cluster 2021-02-10T04:40:41.273191+0000 osd.4 (osd.4) 1380 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:42.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:42 smithi164 conmon[29288]: cluster 2021-02-10T04:40:41.273197+0000 osd.4 (osd.4) 1381 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:42 smithi164 conmon[29288]: cluster 2021-02-10T04:40:41.273202+0000 osd.4 (osd.4) 2021-02-10T04:40:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:42 smithi164 conmon[29288]: 1382 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:42 smithi164 conmon[29288]: cluster 2021-02-10T04:40:42 2021-02-10T04:40:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:42 smithi164 conmon[29288]: .029787+0000 osd.7 (osd.7) 62 : cluster 2021-02-10T04:40:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:42 smithi164 conmon[29288]: [DBG] purged_snaps scrub starts 2021-02-10T04:40:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:42 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:40:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:42 smithi164 conmon[29288]: 02-10T04:40:42.030575+0000 osd.7 (osd 2021-02-10T04:40:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:42 smithi164 conmon[29288]: .7) 63 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:40:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:42 smithi164 conmon[29288]: cluster 2021-02-10T04:40:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:42 smithi164 conmon[29288]: 2021-02-10T04:40:42.199797+0000 2021-02-10T04:40:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:42 smithi164 conmon[29288]: mgr.y (mgr.14140) 795 : cluster [DBG] pgmap v1384: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 535 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 3.4 MiB/s wr, 20 op/s 2021-02-10T04:40:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:42 smithi164 conmon[29288]: cluster 2021-02-10T04:40:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:42 smithi164 conmon[29288]: 2021-02-10T04:40 2021-02-10T04:40:42.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:42 smithi164 conmon[29288]: :42.287728+0000 mon.a (mon.0) 1770 : cluster 2021-02-10T04:40:42.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:42 smithi164 conmon[29288]: [DBG] osdmap e823: 8 total, 8 up, 7 in 2021-02-10T04:40:42.720 INFO:tasks.rados.rados.0.smithi085.stdout:3234: finishing write tid 1 to smithi08575397-20 2021-02-10T04:40:42.720 INFO:tasks.rados.rados.0.smithi085.stdout:3234: finishing write tid 2 to smithi08575397-20 2021-02-10T04:40:42.720 INFO:tasks.rados.rados.0.smithi085.stdout:3234: finishing write tid 3 to smithi08575397-20 2021-02-10T04:40:42.721 INFO:tasks.rados.rados.0.smithi085.stdout:3234: finishing write tid 4 to smithi08575397-20 2021-02-10T04:40:42.721 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 20 v 1445 (ObjNum 1103 snap 373 seq_num 1103) dirty exists 2021-02-10T04:40:42.721 INFO:tasks.rados.rados.0.smithi085.stdout:3234: left oid 20 (ObjNum 1103 snap 373 seq_num 1103) 2021-02-10T04:40:42.721 INFO:tasks.rados.rados.0.smithi085.stdout:3235: finishing write tid 1 to smithi08575397-8 2021-02-10T04:40:42.721 INFO:tasks.rados.rados.0.smithi085.stdout:3232: done (8 left) 2021-02-10T04:40:42.721 INFO:tasks.rados.rados.0.smithi085.stdout:3233: done (7 left) 2021-02-10T04:40:42.721 INFO:tasks.rados.rados.0.smithi085.stdout:3234: done (6 left) 2021-02-10T04:40:42.721 INFO:tasks.rados.rados.0.smithi085.stdout:3237: snap_remove snap 365 2021-02-10T04:40:42.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: cluster 2021-02-10T04:40:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: 2021-02-10T04:40:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: 2021 2021-02-10T04:40:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: - 2021-02-10T04:40:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: 02 2021-02-10T04:40:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: - 2021-02-10T04:40:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: 10 2021-02-10T04:40:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: T 2021-02-10T04:40:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: 04 2021-02-10T04:40:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: :40:41.273181+0000 2021-02-10T04:40:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: osd.4 (osd.4) 1379 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:40:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: T04:40:41.273191+0000 osd.4 (osd.4) 2021-02-10T04:40:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: 1380 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:40:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: :40:41.273197+0000 osd.4 (osd.4 2021-02-10T04:40:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: ) 1381 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:40:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: -02-10T04:40:41.273202+0000 2021-02-10T04:40:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: osd.4 (osd.4) 1382 : cluster [WRN] 2021-02-10T04:40:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: cluster 2021-02-10T04:40:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: 2021-02-10T04:40:42.029787+0000 osd.7 ( 2021-02-10T04:40:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: osd.7) 62 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:40:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:40:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: -02-10T04:40:42.030575+0000 osd.7 2021-02-10T04:40:42.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: (osd.7) 63 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:40:42.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: 2021-02-10T04:40:42.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: cluster 2021-02-10T04:40:42.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: 2021-02-10T04:40: 2021-02-10T04:40:42.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: 42.199797+0000 mgr.y (mgr.14140) 795 2021-02-10T04:40:42.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: : cluster [DBG] pgmap v1384: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 535 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 3.4 MiB/s wr, 20 op/s 2021-02-10T04:40:42.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:40:42.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: 10T04:40:42.287728+0000 mon.a (mon.0 2021-02-10T04:40:42.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[29319]: ) 1770 : cluster [DBG] osdmap e823: 8 total, 8 up, 7 in 2021-02-10T04:40:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: cluster 2021-02-10T04:40:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: 2021-02-10T04:40:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: 2021 2021-02-10T04:40:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: - 2021-02-10T04:40:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: 02 2021-02-10T04:40:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: - 2021-02-10T04:40:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: 10 2021-02-10T04:40:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: T 2021-02-10T04:40:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: 04 2021-02-10T04:40:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: : 2021-02-10T04:40:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: 40 2021-02-10T04:40:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: : 2021-02-10T04:40:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: 41 2021-02-10T04:40:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: . 2021-02-10T04:40:42.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: 273181 2021-02-10T04:40:42.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: +0000 2021-02-10T04:40:42.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: 2021-02-10T04:40:42.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: osd.4 2021-02-10T04:40:42.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: ( 2021-02-10T04:40:42.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: osd 2021-02-10T04:40:42.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: . 2021-02-10T04:40:42.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: 4 2021-02-10T04:40:42.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: ) 2021-02-10T04:40:42.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: 1379 2021-02-10T04:40:42.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: : 2021-02-10T04:40:42.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:42.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:40:42.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: 10T04:40:41.273191+0000 osd.4 (osd.4) 2021-02-10T04:40:42.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: 1380 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:42.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:40:42.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: T04:40:41.273197+0000 osd.4 (osd 2021-02-10T04:40:42.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: .4) 1381 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:42.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:40:42.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: 02-10T04:40:41.273202+0000 2021-02-10T04:40:42.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: osd.4 (osd.4) 1382 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:42.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:40:42.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: 02-10T04:40:42.029787+0000 osd.7 (osd.7) 62 : cluster [DBG] 2021-02-10T04:40:42.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: purged_snaps scrub starts 2021-02-10T04:40:42.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: cluster 2021-02-10T04:40: 2021-02-10T04:40:42.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: 42.030575+0000 osd.7 (osd.7) 63 : 2021-02-10T04:40:42.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:40:42.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:40:42.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: 04:40:42.199797+0000 mgr.y (mgr 2021-02-10T04:40:42.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: .14140) 795 : cluster [DBG] pgmap v1384: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 535 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 3.4 MiB/s wr, 20 op/s 2021-02-10T04:40:42.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:40:42.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: 02-10T04:40:42.287728+0000 2021-02-10T04:40:42.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: mon.a (mon.0) 1770 : 2021-02-10T04:40:42.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:42 smithi085 conmon[35961]: cluster [DBG] osdmap e823: 8 total, 8 up, 7 in 2021-02-10T04:40:43.497 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:43 smithi164 conmon[33677]: debug 2021-02-10T04:40:43.497 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:43 smithi164 conmon[33677]: 2021-02-10T04:40:43.214+0000 7fbec19a1700 -1 osd.4 823 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:43.497 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:43 smithi164 conmon[33677]: 2021-02-10T04:40:43.505 INFO:tasks.rados.rados.0.smithi085.stdout:3237: done (6 left) 2021-02-10T04:40:43.506 INFO:tasks.rados.rados.0.smithi085.stdout:3238: snap_remove snap 348 2021-02-10T04:40:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:43 smithi164 conmon[29288]: cluster 2021-02-10T04:40:42.263764+0000 osd.4 (osd.4) 1383 2021-02-10T04:40:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:43 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:43 smithi164 conmon[29288]: cluster 2021-02-10T04:40:42.263773+0000 osd.4 (osd.4) 1384 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:43 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:40:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:43 smithi164 conmon[29288]: -10T04:40:42.263780+0000 osd.4 (osd.4) 1385 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:43 smithi164 conmon[29288]: cluster 2021-02-10T04:40:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:43 smithi164 conmon[29288]: 2021-02-10T04:40:42.263786+0000 osd.4 (osd.4) 1386 2021-02-10T04:40:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:43 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:43.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[29319]: cluster 2021-02-10T04:40:43.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[29319]: 2021-02-10T04:40:43.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[29319]: 2021 2021-02-10T04:40:43.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[29319]: - 2021-02-10T04:40:43.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[29319]: 02 2021-02-10T04:40:43.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[29319]: -10T04:40: 2021-02-10T04:40:43.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[29319]: 42.263764+0000 osd.4 (osd.4) 1383 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:43.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[29319]: cluster 2021-02-10T04:40:43.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[29319]: 2021-02-10T04:40 2021-02-10T04:40:43.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[29319]: :42.263773+0000 osd.4 (osd.4) 2021-02-10T04:40:43.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[29319]: 1384 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:43.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[29319]: cluster 2021-02-10T04:40:43.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[29319]: 2021-02-10T04:40:42.263780 2021-02-10T04:40:43.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1385 : cluster 2021-02-10T04:40:43.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:43.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:40:43.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[29319]: 10T04:40:42.263786+0000 osd.4 (osd.4 2021-02-10T04:40:43.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[29319]: ) 1386 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[35961]: cluster 2021-02-10T04:40:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[35961]: 2021-02-10T04:40:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[35961]: 2021-02-10T04:40:42 2021-02-10T04:40:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[35961]: .263764+0000 osd.4 (osd.4) 1383 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[35961]: cluster 2021-02-10T04:40:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[35961]: 2021-02-10T04:40:42. 2021-02-10T04:40:43.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[35961]: 263773+0000 osd.4 (osd.4) 1384 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:43.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:40:43.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[35961]: T04:40:42.263780+0000 osd.4 (osd 2021-02-10T04:40:43.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[35961]: .4) 1385 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:43.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:40:43.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[35961]: -02-10T04:40:42.263786+0000 2021-02-10T04:40:43.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[35961]: osd.4 (osd.4) 1386 : cluster [WRN] 2021-02-10T04:40:43.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:43 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:44.510 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:44 smithi164 conmon[33677]: debug 2021-02-10T04:40:44.510 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:44 smithi164 conmon[33677]: 2021-02-10T04:40:44.170+0000 7fbec19a1700 -1 osd.4 824 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:44.511 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:44 smithi164 conmon[33677]: 2021-02-10T04:40:44.517 INFO:tasks.rados.rados.0.smithi085.stdout:3238: done (6 left) 2021-02-10T04:40:44.517 INFO:tasks.rados.rados.0.smithi085.stdout:3239: delete oid 15 current snap is 373 2021-02-10T04:40:44.517 INFO:tasks.rados.rados.0.smithi085.stdout:3236: finishing rollback tid 0 to smithi08575397-43 2021-02-10T04:40:44.517 INFO:tasks.rados.rados.0.smithi085.stdout:3236: finishing rollback tid 1 to smithi08575397-43 2021-02-10T04:40:44.517 INFO:tasks.rados.rados.0.smithi085.stdout:3236: finishing rollback tid 2 to smithi08575397-43 2021-02-10T04:40:44.517 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 1557 (ObjNum 748 snap 240 seq_num 748) dirty exists 2021-02-10T04:40:44.517 INFO:tasks.rados.rados.0.smithi085.stdout:3235: finishing write tid 2 to smithi08575397-8 2021-02-10T04:40:44.517 INFO:tasks.rados.rados.0.smithi085.stdout:3235: finishing write tid 3 to smithi08575397-8 2021-02-10T04:40:44.517 INFO:tasks.rados.rados.0.smithi085.stdout:3235: finishing write tid 4 to smithi08575397-8 2021-02-10T04:40:44.517 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 1563 (ObjNum 1104 snap 373 seq_num 1104) dirty exists 2021-02-10T04:40:44.518 INFO:tasks.rados.rados.0.smithi085.stdout:3235: left oid 8 (ObjNum 1104 snap 373 seq_num 1104) 2021-02-10T04:40:44.520 INFO:tasks.rados.rados.0.smithi085.stdout:3235: done (6 left) 2021-02-10T04:40:44.520 INFO:tasks.rados.rados.0.smithi085.stdout:3236: done (5 left) 2021-02-10T04:40:44.520 INFO:tasks.rados.rados.0.smithi085.stdout:3239: done (4 left) 2021-02-10T04:40:44.521 INFO:tasks.rados.rados.0.smithi085.stdout:3240: snap_remove snap 345 2021-02-10T04:40:44.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:44 smithi164 conmon[29288]: cluster 2021-02-10T04:40:45.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:44 smithi164 conmon[29288]: 2021-02-10T04:40 2021-02-10T04:40:45.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:44 smithi164 conmon[29288]: :43.216312+0000 osd.4 (osd.4) 1387 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:45.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:44 smithi164 conmon[29288]: cluster 2021-02-10T04:40 2021-02-10T04:40:45.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:44 smithi164 conmon[29288]: :43.216327+0000 osd.4 (osd.4) 1388 : cluster [WRN] 2021-02-10T04:40:45.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:44 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:45.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:44 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:40:45.509 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:44 smithi164 conmon[29288]: 40:43.216334+0000 osd.4 (osd.4) 2021-02-10T04:40:45.510 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:44 smithi164 conmon[29288]: 1389 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:45.510 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:44 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:40:45.510 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:44 smithi164 conmon[29288]: -10T04:40:43.216339+0000 osd.4 ( 2021-02-10T04:40:45.510 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:44 smithi164 conmon[29288]: osd.4) 1390 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:45.510 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:44 smithi164 conmon[29288]: cluster 2021-02-10T04:40:45.510 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:44 smithi164 conmon[29288]: 2021-02-10T04:40:43.505885+0000 2021-02-10T04:40:45.510 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:44 smithi164 conmon[29288]: mon.a (mon.0) 1771 : cluster [DBG] osdmap e824: 8 total, 8 up, 7 in 2021-02-10T04:40:45.510 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:44 smithi164 conmon[29288]: 2021-02-10T04:40:45.510 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:44 smithi164 conmon[29288]: cluster 2021-02-10T04:40 2021-02-10T04:40:45.511 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:44 smithi164 conmon[29288]: :44.200469+0000 mgr.y (mgr.14140) 796 2021-02-10T04:40:45.511 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:44 smithi164 conmon[29288]: : cluster [DBG] pgmap v1387: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 538 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 6.9 MiB/s rd, 1.8 MiB/s wr, 7 op/s 2021-02-10T04:40:45.512 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[29319]: cluster 2021-02-10T04:40:43.216312+0000 osd.4 ( 2021-02-10T04:40:45.512 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[29319]: osd.4) 1387 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:45.512 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[29319]: cluster 2021-02-10T04:40:43.216327+0000 osd.4 (osd.4) 1388 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:45.512 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:40:45.512 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[29319]: -02-10T04:40:43.216334+0000 osd.4 2021-02-10T04:40:45.512 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[29319]: (osd.4) 1389 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:45.512 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[29319]: cluster 2021-02-10T04:40:45.512 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[29319]: 2021-02-10T04:40:43.216339+0000 osd.4 2021-02-10T04:40:45.512 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[29319]: (osd.4) 1390 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:45.513 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[29319]: 2021-02-10T04:40:45.513 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[29319]: cluster 2021-02-10T04:40:45.513 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[29319]: 2021-02-10T04:40:43.505885+0000 mon.a (mon 2021-02-10T04:40:45.513 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[29319]: .0) 1771 : cluster [DBG] osdmap e824: 8 total, 8 up, 7 in 2021-02-10T04:40:45.513 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:40:45.513 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[29319]: 02-10T04:40:44.200469+0000 2021-02-10T04:40:45.513 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[29319]: mgr.y (mgr.14140) 796 : cluster [DBG] 2021-02-10T04:40:45.513 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[29319]: pgmap v1387: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 538 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 6.9 MiB/s rd, 1.8 MiB/s wr, 7 op/s 2021-02-10T04:40:45.513 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[35961]: cluster 2021-02-10T04:40:43.216312+0000 osd.4 2021-02-10T04:40:45.513 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[35961]: (osd.4) 1387 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:45.513 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[35961]: cluster 2021-02-10T04:40:43.216327+0000 2021-02-10T04:40:45.513 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[35961]: osd.4 (osd.4) 1388 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:45.513 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[35961]: cluster 2021-02-10T04:40:43 2021-02-10T04:40:45.514 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[35961]: .216334+0000 osd.4 (osd.4) 1389 : cluster 2021-02-10T04:40:45.514 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:45.514 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[35961]: cluster 2021-02-10T04:40:43 2021-02-10T04:40:45.514 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[35961]: .216339+0000 osd.4 (osd.4) 1390 : cluster 2021-02-10T04:40:45.514 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:45.514 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[35961]: cluster 2021-02-10T04:40:43.505885 2021-02-10T04:40:45.514 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1771 : cluster [DBG] 2021-02-10T04:40:45.514 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[35961]: osdmap e824: 8 total, 8 up, 7 in 2021-02-10T04:40:45.514 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[35961]: cluster 2021-02-10T04:40:44.200469 2021-02-10T04:40:45.514 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 796 : cluster [DBG] 2021-02-10T04:40:45.514 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:44 smithi085 conmon[35961]: pgmap v1387: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 538 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 6.9 MiB/s rd, 1.8 MiB/s wr, 7 op/s 2021-02-10T04:40:45.515 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:45 smithi164 conmon[33677]: debug 2021-02-10T04:40:45.175+0000 7fbec19a1700 -1 osd.4 825 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:45.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:45 smithi164 conmon[29288]: cluster 2021-02-10T04:40:45.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:45 smithi164 conmon[29288]: 2021-02-10T04:40:44 2021-02-10T04:40:45.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:45 smithi164 conmon[29288]: .172657+0000 osd.4 (osd.4) 1391 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:45.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:45 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:40:45.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:45 smithi164 conmon[29288]: -02-10T04:40:44.172669+0000 2021-02-10T04:40:45.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:45 smithi164 conmon[29288]: osd.4 (osd.4) 1392 : cluster [WRN] 2021-02-10T04:40:45.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:45 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:45.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:45 smithi164 conmon[29288]: cluster 2021-02-10T04:40: 2021-02-10T04:40:45.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:45 smithi164 conmon[29288]: 44.172676+0000 osd.4 (osd.4) 1393 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:45.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:45 smithi164 conmon[29288]: 2021-02-10T04:40:45.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:45 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:40:45.515 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:45 smithi164 conmon[29288]: 02-10T04:40:44.172686+0000 osd.4 (osd.4) 1394 : cluster 2021-02-10T04:40:45.516 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:45 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:45.516 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:45 smithi164 conmon[29288]: cluster 2021-02-10T04:40 2021-02-10T04:40:45.516 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:45 smithi164 conmon[29288]: :44.512674+0000 mon.a (mon.0) 1772 : 2021-02-10T04:40:45.516 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:45 smithi164 conmon[29288]: cluster [DBG] osdmap e825: 8 total, 8 up, 7 in 2021-02-10T04:40:45.524 INFO:tasks.rados.rados.0.smithi085.stdout:3240: done (4 left) 2021-02-10T04:40:45.524 INFO:tasks.rados.rados.0.smithi085.stdout:3241: read oid 42 snap 356 2021-02-10T04:40:45.524 INFO:tasks.rados.rados.0.smithi085.stdout:3241: expect (ObjNum 1064 snap 349 seq_num 1064) 2021-02-10T04:40:45.524 INFO:tasks.rados.rados.0.smithi085.stdout:3242: delete oid 48 current snap is 373 2021-02-10T04:40:45.534 INFO:tasks.rados.rados.0.smithi085.stdout:3241: expect (ObjNum 1064 snap 349 seq_num 1064) 2021-02-10T04:40:45.698 INFO:tasks.rados.rados.0.smithi085.stdout:3241: done (5 left) 2021-02-10T04:40:45.699 INFO:tasks.rados.rados.0.smithi085.stdout:3242: done (4 left) 2021-02-10T04:40:45.699 INFO:tasks.rados.rados.0.smithi085.stdout:3243: rollback oid 6 current snap is 373 2021-02-10T04:40:45.699 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 6 to 367 2021-02-10T04:40:45.699 INFO:tasks.rados.rados.0.smithi085.stdout:3244: read oid 12 snap -1 2021-02-10T04:40:45.699 INFO:tasks.rados.rados.0.smithi085.stdout:3244: expect (ObjNum 1081 snap 359 seq_num 1081) 2021-02-10T04:40:45.699 INFO:tasks.rados.rados.0.smithi085.stdout:3245: read oid 15 snap -1 2021-02-10T04:40:45.699 INFO:tasks.rados.rados.0.smithi085.stdout:3245: expect deleted 2021-02-10T04:40:45.699 INFO:tasks.rados.rados.0.smithi085.stdout:3246: read oid 3 snap -1 2021-02-10T04:40:45.699 INFO:tasks.rados.rados.0.smithi085.stdout:3246: expect (ObjNum 1096 snap 368 seq_num 1096) 2021-02-10T04:40:45.699 INFO:tasks.rados.rados.0.smithi085.stdout:3247: read oid 29 snap 266 2021-02-10T04:40:45.699 INFO:tasks.rados.rados.0.smithi085.stdout:3247: expect (ObjNum 8 snap 0 seq_num 8) 2021-02-10T04:40:45.700 INFO:tasks.rados.rados.0.smithi085.stdout:3248: read oid 38 snap 337 2021-02-10T04:40:45.700 INFO:tasks.rados.rados.0.smithi085.stdout:3248: expect (ObjNum 1019 snap 335 seq_num 1019) 2021-02-10T04:40:45.700 INFO:tasks.rados.rados.0.smithi085.stdout:3249: read oid 48 snap 372 2021-02-10T04:40:45.700 INFO:tasks.rados.rados.0.smithi085.stdout:3249: expect deleted 2021-02-10T04:40:45.700 INFO:tasks.rados.rados.0.smithi085.stdout:3250: snap_create 2021-02-10T04:40:45.700 INFO:tasks.rados.rados.0.smithi085.stdout:3246: expect (ObjNum 1096 snap 368 seq_num 1096) 2021-02-10T04:40:45.809 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:45 smithi085 conmon[35961]: cluster 2021-02-10T04:40:44.172657 2021-02-10T04:40:45.809 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:45 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1391 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:45.809 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:45 smithi085 conmon[35961]: cluster 2021-02-10T04:40:44.172669+0000 2021-02-10T04:40:45.809 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:45 smithi085 conmon[35961]: osd.4 (osd.4) 1392 : cluster [WRN] 2021-02-10T04:40:45.809 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:45 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:45.809 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:45 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:40:45.809 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:45 smithi085 conmon[35961]: 04:40:44.172676+0000 osd.4 (osd.4) 2021-02-10T04:40:45.809 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:45 smithi085 conmon[35961]: 1393 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:45.809 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:45 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:40:45.809 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:45 smithi085 conmon[35961]: :40:44.172686+0000 osd.4 (osd.4 2021-02-10T04:40:45.810 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:45 smithi085 conmon[35961]: ) 1394 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:45.810 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:45 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:40:45.810 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:45 smithi085 conmon[35961]: T04:40:44.512674+0000 mon.a (mon. 2021-02-10T04:40:45.810 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:45 smithi085 conmon[35961]: 0) 1772 : cluster [DBG] osdmap e825: 8 total, 8 up, 7 in 2021-02-10T04:40:45.810 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:45 smithi085 conmon[29319]: cluster 2021-02-10T04:40:44.172657 2021-02-10T04:40:45.810 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:45 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1391 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:45.810 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:45 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:40:45.810 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:45 smithi085 conmon[29319]: T04:40:44.172669+0000 osd.4 (osd.4) 1392 : 2021-02-10T04:40:45.810 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:45 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:45.811 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:45 smithi085 conmon[29319]: cluster 2021-02-10T04:40:45.811 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:45 smithi085 conmon[29319]: 2021-02-10T04:40:44.172676+0000 2021-02-10T04:40:45.811 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:45 smithi085 conmon[29319]: osd.4 (osd.4) 1393 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:45.811 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:45 smithi085 conmon[29319]: 2021-02-10T04:40:45.811 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:45 smithi085 conmon[29319]: cluster 2021-02-10T04:40: 2021-02-10T04:40:45.811 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:45 smithi085 conmon[29319]: 44.172686+0000 osd.4 (osd.4) 1394 : cluster 2021-02-10T04:40:45.811 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:45 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:45.811 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:45 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:40:45.811 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:45 smithi085 conmon[29319]: 02-10T04:40:44. 2021-02-10T04:40:45.811 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:45 smithi085 conmon[29319]: 512674+0000 mon.a (mon.0) 2021-02-10T04:40:45.811 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:45 smithi085 conmon[29319]: 1772 : cluster [DBG] osdmap e825: 8 total, 8 up, 7 in 2021-02-10T04:40:45.812 INFO:tasks.rados.rados.0.smithi085.stdout:3244: expect (ObjNum 1081 snap 359 seq_num 1081) 2021-02-10T04:40:45.936 INFO:tasks.rados.rados.0.smithi085.stdout:3243: finishing rollback tid 0 to smithi08575397-6 2021-02-10T04:40:45.937 INFO:tasks.rados.rados.0.smithi085.stdout:3243: finishing rollback tid 1 to smithi08575397-6 2021-02-10T04:40:45.937 INFO:tasks.rados.rados.0.smithi085.stdout:3248: expect (ObjNum 1019 snap 335 seq_num 1019) 2021-02-10T04:40:46.085 INFO:tasks.rados.rados.0.smithi085.stdout:3243: finishing rollback tid 2 to smithi08575397-6 2021-02-10T04:40:46.085 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 1567 (ObjNum 1028 snap 335 seq_num 1028) dirty exists 2021-02-10T04:40:46.085 INFO:tasks.rados.rados.0.smithi085.stdout:3247: expect (ObjNum 8 snap 0 seq_num 8) 2021-02-10T04:40:46.148 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[41540]: debug 2021-02-10T04:40:46.146+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:40:46.407 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:40:46.518 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:46 smithi164 conmon[33677]: debug 2021-02-10T04:40:46.182+0000 7fbec19a1700 -1 osd.4 826 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:46.526 INFO:tasks.rados.rados.0.smithi085.stdout:3243: done (11 left) 2021-02-10T04:40:46.526 INFO:tasks.rados.rados.0.smithi085.stdout:3244: done (10 left) 2021-02-10T04:40:46.526 INFO:tasks.rados.rados.0.smithi085.stdout:3245: done (9 left) 2021-02-10T04:40:46.527 INFO:tasks.rados.rados.0.smithi085.stdout:3246: done (8 left) 2021-02-10T04:40:46.527 INFO:tasks.rados.rados.0.smithi085.stdout:3247: done (7 left) 2021-02-10T04:40:46.527 INFO:tasks.rados.rados.0.smithi085.stdout:3248: done (6 left) 2021-02-10T04:40:46.527 INFO:tasks.rados.rados.0.smithi085.stdout:3249: done (5 left) 2021-02-10T04:40:46.527 INFO:tasks.rados.rados.0.smithi085.stdout:3250: done (4 left) 2021-02-10T04:40:46.528 INFO:tasks.rados.rados.0.smithi085.stdout:3251: read oid 25 snap 369 2021-02-10T04:40:46.528 INFO:tasks.rados.rados.0.smithi085.stdout:3251: expect (ObjNum 1016 snap 334 seq_num 1016) 2021-02-10T04:40:46.528 INFO:tasks.rados.rados.0.smithi085.stdout:3252: write (excl) oid 49 current snap is 374 2021-02-10T04:40:46.528 INFO:tasks.rados.rados.0.smithi085.stdout:3252: seq_num 1107 ranges {605349=571828,1679096=420377} 2021-02-10T04:40:46.533 INFO:tasks.rados.rados.0.smithi085.stdout:3252: writing smithi08575397-49 from 605349 to 1177177 tid 1 2021-02-10T04:40:46.539 INFO:tasks.rados.rados.0.smithi085.stdout:3252: writing smithi08575397-49 from 1679096 to 2099473 tid 2 2021-02-10T04:40:46.539 INFO:tasks.rados.rados.0.smithi085.stdout:3253: read oid 18 snap 371 2021-02-10T04:40:46.540 INFO:tasks.rados.rados.0.smithi085.stdout:3253: expect (ObjNum 952 snap 313 seq_num 952) 2021-02-10T04:40:46.540 INFO:tasks.rados.rados.0.smithi085.stdout:3254: write (excl) oid 22 current snap is 374 2021-02-10T04:40:46.540 INFO:tasks.rados.rados.0.smithi085.stdout:3251: expect (ObjNum 1016 snap 334 seq_num 1016) 2021-02-10T04:40:46.690 INFO:tasks.rados.rados.0.smithi085.stdout:3252: finishing write tid 1 to smithi08575397-49 2021-02-10T04:40:46.690 INFO:tasks.rados.rados.0.smithi085.stdout:3254: seq_num 1108 ranges {622156=647077,1737578=680388} 2021-02-10T04:40:46.696 INFO:tasks.rados.rados.0.smithi085.stdout:3254: writing smithi08575397-22 from 622156 to 1269233 tid 1 2021-02-10T04:40:46.704 INFO:tasks.rados.rados.0.smithi085.stdout:3254: writing smithi08575397-22 from 1737578 to 2417966 tid 2 2021-02-10T04:40:46.704 INFO:tasks.rados.rados.0.smithi085.stdout:3251: done (7 left) 2021-02-10T04:40:46.704 INFO:tasks.rados.rados.0.smithi085.stdout:3255: snap_create 2021-02-10T04:40:46.704 INFO:tasks.rados.rados.0.smithi085.stdout:3253: expect (ObjNum 952 snap 313 seq_num 952) 2021-02-10T04:40:46.883 INFO:tasks.rados.rados.0.smithi085.stdout:3252: finishing write tid 2 to smithi08575397-49 2021-02-10T04:40:46.884 INFO:tasks.rados.rados.0.smithi085.stdout:3252: finishing write tid 3 to smithi08575397-49 2021-02-10T04:40:46.884 INFO:tasks.rados.rados.0.smithi085.stdout:3252: finishing write tid 4 to smithi08575397-49 2021-02-10T04:40:46.884 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 49 v 1479 (ObjNum 1107 snap 374 seq_num 1107) dirty exists 2021-02-10T04:40:46.884 INFO:tasks.rados.rados.0.smithi085.stdout:3252: left oid 49 (ObjNum 1107 snap 374 seq_num 1107) 2021-02-10T04:40:46.884 INFO:tasks.rados.rados.0.smithi085.stdout:3254: finishing write tid 1 to smithi08575397-22 2021-02-10T04:40:46.884 INFO:tasks.rados.rados.0.smithi085.stdout:3254: finishing write tid 2 to smithi08575397-22 2021-02-10T04:40:46.884 INFO:tasks.rados.rados.0.smithi085.stdout:3254: finishing write tid 3 to smithi08575397-22 2021-02-10T04:40:46.884 INFO:tasks.rados.rados.0.smithi085.stdout:3254: finishing write tid 4 to smithi08575397-22 2021-02-10T04:40:46.884 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 1472 (ObjNum 1108 snap 374 seq_num 1108) dirty exists 2021-02-10T04:40:46.884 INFO:tasks.rados.rados.0.smithi085.stdout:3254: left oid 22 (ObjNum 1108 snap 374 seq_num 1108) 2021-02-10T04:40:46.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:46 smithi164 conmon[29288]: cluster 2021-02-10T04:40:45.177315+0000 osd.4 (osd.4) 1395 : cluster 2021-02-10T04:40:46.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:46 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:46.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:46 smithi164 conmon[29288]: cluster 2021-02-10T04:40:45.177324+0000 osd.4 (osd.4) 1396 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:46.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:46 smithi164 conmon[29288]: cluster 2021-02-10T04:40:45.177332+0000 osd.4 (osd.4 2021-02-10T04:40:46.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:46 smithi164 conmon[29288]: ) 1397 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:46.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:46 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:40:46.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:46 smithi164 conmon[29288]: 40:45.177338+0000 osd.4 (osd.4) 1398 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:46.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:46 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:40:46.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:46 smithi164 conmon[29288]: -10T04:40:45.523504+0000 mon.a (mon.0 2021-02-10T04:40:46.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:46 smithi164 conmon[29288]: ) 1773 : cluster [DBG] osdmap e826: 8 total, 8 up, 7 in 2021-02-10T04:40:46.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:46 smithi164 conmon[29288]: cluster 2021-02-10T04:40:46.201055+0000 mgr.y (mgr.14140) 797 : cluster [DBG] pgmap v1390: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 541 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 6.9 MiB/s rd, 2.0 MiB/s wr, 10 op/s 2021-02-10T04:40:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[29319]: cluster 2021-02-10T04:40:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[29319]: 2021-02-10T04:40:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[29319]: 2021-02-10T04:40:45 2021-02-10T04:40:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[29319]: .177315+0000 osd.4 (osd.4) 1395 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:40:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[29319]: -02-10T04:40:45.177324+0000 osd.4 2021-02-10T04:40:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[29319]: (osd.4) 1396 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[29319]: cluster 2021-02-10T04:40:45.177332+0000 osd.4 (osd.4 2021-02-10T04:40:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[29319]: ) 1397 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:40:46.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[29319]: 10T04:40:45.177338+0000 osd.4 (osd.4) 1398 : cluster 2021-02-10T04:40:46.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:46.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[29319]: cluster 2021-02-10T04:40 2021-02-10T04:40:46.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[29319]: :45.523504+0000 mon.a (mon.0) 1773 : cluster [DBG] 2021-02-10T04:40:46.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[29319]: osdmap e826: 8 total, 8 up, 7 in 2021-02-10T04:40:46.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[29319]: cluster 2021-02-10T04:40:46 2021-02-10T04:40:46.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[29319]: .201055+0000 mgr.y (mgr.14140) 797 : 2021-02-10T04:40:46.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[29319]: cluster [DBG] pgmap v1390: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 541 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 6.9 MiB/s rd, 2.0 MiB/s wr, 10 op/s 2021-02-10T04:40:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[35961]: cluster 2021-02-10T04:40:45. 2021-02-10T04:40:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[35961]: 177315+0000 osd.4 (osd.4) 1395 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[35961]: cluster 2021-02-10T04:40:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[35961]: 2021-02-10T04:40:45.177324+0000 osd.4 (osd.4) 1396 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:46.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:40:46.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[35961]: 02-10T04: 2021-02-10T04:40:46.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[35961]: 40:45.177332+0000 osd.4 (osd.4) 2021-02-10T04:40:46.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[35961]: 1397 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:46.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:40:46.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[35961]: 10T04:40:45.177338+0000 osd.4 2021-02-10T04:40:46.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[35961]: (osd.4) 2021-02-10T04:40:46.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[35961]: 1398 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:46.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:40:46.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[35961]: T04:40:45.523504+0000 mon.a (mon 2021-02-10T04:40:46.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[35961]: .0) 1773 : cluster [DBG] osdmap e826: 8 total, 8 up, 7 in 2021-02-10T04:40:46.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:40:46.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[35961]: -02-10T04:40:46.201055+0000 2021-02-10T04:40:46.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[35961]: mgr.y (mgr.14140) 797 : cluster [DBG] 2021-02-10T04:40:46.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:46 smithi085 conmon[35961]: pgmap v1390: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 541 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 6.9 MiB/s rd, 2.0 MiB/s wr, 10 op/s 2021-02-10T04:40:47.525 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:47 smithi164 conmon[33677]: debug 2021-02-10T04:40:47.160+0000 7fbec19a1700 -1 osd.4 827 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:47.533 INFO:tasks.rados.rados.0.smithi085.stdout:3252: done (7 left) 2021-02-10T04:40:47.533 INFO:tasks.rados.rados.0.smithi085.stdout:3253: done (6 left) 2021-02-10T04:40:47.533 INFO:tasks.rados.rados.0.smithi085.stdout:3254: done (5 left) 2021-02-10T04:40:47.533 INFO:tasks.rados.rados.0.smithi085.stdout:3255: done (4 left) 2021-02-10T04:40:47.533 INFO:tasks.rados.rados.0.smithi085.stdout:3256: write (excl) oid 9 current snap is 375 2021-02-10T04:40:47.534 INFO:tasks.rados.rados.0.smithi085.stdout:3256: seq_num 1109 ranges {638963=722326,2079707=424046} 2021-02-10T04:40:47.542 INFO:tasks.rados.rados.0.smithi085.stdout:3256: writing smithi08575397-9 from 638963 to 1361289 tid 1 2021-02-10T04:40:47.549 INFO:tasks.rados.rados.0.smithi085.stdout:3256: writing smithi08575397-9 from 2079707 to 2503753 tid 2 2021-02-10T04:40:47.549 INFO:tasks.rados.rados.0.smithi085.stdout:3257: read oid 4 snap -1 2021-02-10T04:40:47.549 INFO:tasks.rados.rados.0.smithi085.stdout:3257: expect (ObjNum 1049 snap 344 seq_num 1049) 2021-02-10T04:40:47.549 INFO:tasks.rados.rados.0.smithi085.stdout:3258: snap_create 2021-02-10T04:40:47.549 INFO:tasks.rados.rados.0.smithi085.stdout:3256: finishing write tid 1 to smithi08575397-9 2021-02-10T04:40:47.556 INFO:tasks.rados.rados.0.smithi085.stdout:3256: finishing write tid 2 to smithi08575397-9 2021-02-10T04:40:47.556 INFO:tasks.rados.rados.0.smithi085.stdout:3256: finishing write tid 3 to smithi08575397-9 2021-02-10T04:40:47.556 INFO:tasks.rados.rados.0.smithi085.stdout:3256: finishing write tid 4 to smithi08575397-9 2021-02-10T04:40:47.556 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 9 v 1477 (ObjNum 1109 snap 375 seq_num 1109) dirty exists 2021-02-10T04:40:47.556 INFO:tasks.rados.rados.0.smithi085.stdout:3256: left oid 9 (ObjNum 1109 snap 375 seq_num 1109) 2021-02-10T04:40:47.557 INFO:tasks.rados.rados.0.smithi085.stdout:3257: expect (ObjNum 1049 snap 344 seq_num 1049) 2021-02-10T04:40:47.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:47 smithi164 conmon[29288]: cluster 2021-02-10T04:40:46.182971+0000 osd.4 ( 2021-02-10T04:40:47.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:47 smithi164 conmon[29288]: osd.4) 1399 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:47.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:47 smithi164 conmon[29288]: cluster 2021-02-10T04:40:46.182982+0000 osd.4 (osd.4 2021-02-10T04:40:47.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:47 smithi164 conmon[29288]: ) 1400 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:47.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:47 smithi164 conmon[29288]: cluster 2021-02-10T04:40:47.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:47 smithi164 conmon[29288]: 2021-02-10T04:40:46.182988+0000 osd.4 (osd.4) 1401 : cluster 2021-02-10T04:40:47.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:47 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:47.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:47 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:40:47.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:47 smithi164 conmon[29288]: :40:46.183013+0000 osd.4 (osd.4) 2021-02-10T04:40:47.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:47 smithi164 conmon[29288]: 1402 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:47.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:47 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:40:47.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:47 smithi164 conmon[29288]: :40:46.525186+0000 mon.a (mon.0) 1774 : cluster 2021-02-10T04:40:47.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:47 smithi164 conmon[29288]: [DBG] osdmap e827: 8 total, 8 up, 7 in 2021-02-10T04:40:47.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:47 smithi164 conmon[29288]: cluster 2021-02-10T04:40:47 2021-02-10T04:40:47.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:47 smithi164 conmon[29288]: .015939+0000 osd.4 (osd.4) 1403 : cluster 2021-02-10T04:40:47.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:47 smithi164 conmon[29288]: [DBG] purged_snaps scrub starts 2021-02-10T04:40:47.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:47 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:40:47.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:47 smithi164 conmon[29288]: :40:47.016573+0000 osd.4 (osd.4) 2021-02-10T04:40:47.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:47 smithi164 conmon[29288]: 1404 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:40:47.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[29319]: cluster 2021-02-10T04:40:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[29319]: 2021-02-10T04:40:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[29319]: 2021-02-10T04:40: 2021-02-10T04:40:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[29319]: 46.182971+0000 osd.4 (osd.4) 1399 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[29319]: cluster 2021-02-10T04:40:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:40:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[29319]: 04:40:46.182982+0000 osd.4 (osd 2021-02-10T04:40:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[29319]: .4) 1400 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:40:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[29319]: -02-10T04:40:46.182988+0000 osd.4 (osd. 2021-02-10T04:40:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[29319]: 4) 1401 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:40:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[29319]: -02-10T04:40:46.183013 2021-02-10T04:40:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1402 : cluster [WRN] 2021-02-10T04:40:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[29319]: cluster 2021-02-10T04:40: 2021-02-10T04:40:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[29319]: 46.525186+0000 mon.a 2021-02-10T04:40:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[29319]: (mon.0) 1774 : cluster [DBG] osdmap e827: 8 total, 8 up, 7 in 2021-02-10T04:40:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:40:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[29319]: -02-10T04:40:47.015939 2021-02-10T04:40:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1403 : cluster [DBG] 2021-02-10T04:40:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[29319]: purged_snaps scrub starts 2021-02-10T04:40:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[29319]: cluster 2021-02-10T04:40: 2021-02-10T04:40:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[29319]: 47.016573+0000 osd.4 (osd.4) 1404 : cluster [DBG] 2021-02-10T04:40:47.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[29319]: purged_snaps scrub ok 2021-02-10T04:40:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[35961]: cluster 2021-02-10T04:40:46.182971+0000 2021-02-10T04:40:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[35961]: osd.4 (osd.4) 1399 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[35961]: cluster 2021-02-10T04:40:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[35961]: 2021-02-10T04:40: 2021-02-10T04:40:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[35961]: 46.182982+0000 osd.4 (osd.4) 1400 : 2021-02-10T04:40:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[35961]: cluster 2021-02-10T04:40:46.182988+0000 2021-02-10T04:40:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[35961]: osd.4 (osd.4) 1401 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:40:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[35961]: -10T04:40:46.183013+0000 osd.4 ( 2021-02-10T04:40:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[35961]: osd.4) 1402 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:47.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:40:47.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[35961]: -10T04:40:46.525186+0000 mon.a 2021-02-10T04:40:47.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[35961]: (mon.0) 1774 : cluster [DBG] osdmap e827: 8 total, 8 up, 7 in 2021-02-10T04:40:47.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:40:47.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[35961]: -02-10T04:40:47.015939+0000 2021-02-10T04:40:47.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[35961]: osd.4 (osd.4) 1403 : cluster [DBG] 2021-02-10T04:40:47.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[35961]: purged_snaps scrub starts 2021-02-10T04:40:47.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[35961]: cluster 2021-02-10T04:40:47.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[35961]: 2021-02-10T04:40:47.016573+0000 osd.4 2021-02-10T04:40:47.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:47 smithi085 conmon[35961]: (osd.4) 1404 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:40:48.272 INFO:tasks.rados.rados.0.smithi085.stdout:3256: done (6 left) 2021-02-10T04:40:48.272 INFO:tasks.rados.rados.0.smithi085.stdout:3257: done (5 left) 2021-02-10T04:40:48.272 INFO:tasks.rados.rados.0.smithi085.stdout:3258: done (4 left) 2021-02-10T04:40:48.273 INFO:tasks.rados.rados.0.smithi085.stdout:3259: read oid 6 snap -1 2021-02-10T04:40:48.273 INFO:tasks.rados.rados.0.smithi085.stdout:3259: expect (ObjNum 1028 snap 335 seq_num 1028) 2021-02-10T04:40:48.273 INFO:tasks.rados.rados.0.smithi085.stdout:3260: delete oid 7 current snap is 376 2021-02-10T04:40:48.284 INFO:tasks.rados.rados.0.smithi085.stdout:3260: done (5 left) 2021-02-10T04:40:48.285 INFO:tasks.rados.rados.0.smithi085.stdout:3261: snap_create 2021-02-10T04:40:48.285 INFO:tasks.rados.rados.0.smithi085.stdout:3259: expect (ObjNum 1028 snap 335 seq_num 1028) 2021-02-10T04:40:48.418 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:48 smithi164 conmon[33677]: debug 2021-02-10T04:40:48.160+0000 7fbec19a1700 -1 osd.4 828 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:48.418 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:40:48 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:40:48.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[29319]: cluster 2021-02-10T04:40:47.161362+0000 osd.4 (osd 2021-02-10T04:40:48.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[29319]: .4) 1405 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:48.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[29319]: cluster 2021-02-10T04:40:47.161371+0000 osd.4 (osd.4) 1406 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:48.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[29319]: cluster 2021-02-10T04:40:48.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[29319]: 2021-02-10T04:40:47.161376+0000 osd.4 (osd.4) 1407 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:48.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[29319]: cluster 2021-02-10T04:40:48.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[29319]: 2021-02-10T04:40:47.161381+0000 osd.4 (osd.4) 2021-02-10T04:40:48.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[29319]: 1408 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:48.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:40:48.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[29319]: -10T04:40:47.532280+0000 mon.a ( 2021-02-10T04:40:48.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[29319]: mon.0) 1775 : cluster [DBG] osdmap e828: 8 total, 8 up, 7 in 2021-02-10T04:40:48.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[29319]: cluster 2021-02-10T04:40:48.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[29319]: 2021-02-10T04:40:48.201683+0000 mgr.y (mgr 2021-02-10T04:40:48.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[29319]: .14140) 798 : cluster [DBG] pgmap v1393: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 542 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 7.3 MiB/s rd, 545 KiB/s wr, 9 op/s 2021-02-10T04:40:48.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:40:48.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[29319]: 04:40:48.266592+0000 mon.a (mon.0) 2021-02-10T04:40:48.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[29319]: 1776 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 672 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:40:48.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:40:48.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[29319]: -10T04:40:48.271326+0000 mon.a ( 2021-02-10T04:40:48.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[29319]: mon.0) 1777 : cluster [DBG] osdmap e829: 8 total, 8 up, 7 in 2021-02-10T04:40:48.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[35961]: cluster 2021-02-10T04:40:48.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[35961]: 2021-02-10T04:40:48.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[35961]: 2021 2021-02-10T04:40:48.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[35961]: - 2021-02-10T04:40:48.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[35961]: 02-10T04:40 2021-02-10T04:40:48.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[35961]: :47.161362+0000 osd.4 (osd.4) 1405 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:48.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:40:48.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[35961]: 02-10T04:40:47.161371+0000 osd.4 (osd.4) 1406 : 2021-02-10T04:40:48.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:48.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[35961]: cluster 2021-02-10T04:40:47.161376 2021-02-10T04:40:48.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1407 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:48.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:40:48.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[35961]: -10T04:40:47.161381+0000 osd.4 (osd.4) 1408 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:48.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[35961]: cluster 2021-02-10T04:40:48.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[35961]: 2021-02-10T04:40:47.532280+0000 mon.a 2021-02-10T04:40:48.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[35961]: (mon.0) 1775 : cluster [DBG] osdmap e828: 8 total, 8 up, 7 in 2021-02-10T04:40:48.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[35961]: 2021-02-10T04:40:48.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[35961]: cluster 2021-02-10T04:40:48.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[35961]: 2021-02-10T04:40:48.201683 2021-02-10T04:40:48.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 798 : cluster [DBG] pgmap v1393: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 542 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 7.3 MiB/s rd, 545 KiB/s wr, 9 op/s 2021-02-10T04:40:48.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[35961]: cluster 2021-02-10T04:40 2021-02-10T04:40:48.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[35961]: :48.266592+0000 mon.a (mon.0) 1776 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 672 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:40:48.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:48 smithi085 conmon[35961]: cluster 2021-02-10T04:40:48.271326+0000 mon.a (mon.0) 1777 : cluster [DBG] osdmap e829: 8 total, 8 up, 7 in 2021-02-10T04:40:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:48 smithi164 conmon[29288]: cluster 2021-02-10T04:40:47.161362+0000 osd.4 (osd.4) 1405 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:48 smithi164 conmon[29288]: cluster 2021-02-10T04:40:47.161371+0000 osd.4 (osd.4) 1406 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:48 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:40:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:48 smithi164 conmon[29288]: 10T04:40:47.161376+0000 osd.4 (osd.4) 1407 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:48 smithi164 conmon[29288]: cluster 2021-02-10T04:40:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:48 smithi164 conmon[29288]: 2021-02-10 2021-02-10T04:40:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:48 smithi164 conmon[29288]: T04:40:47.161381+0000 2021-02-10T04:40:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:48 smithi164 conmon[29288]: osd.4 (osd.4) 1408 : 2021-02-10T04:40:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:48 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:48 smithi164 conmon[29288]: cluster 2021-02-10T04:40:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:48 smithi164 conmon[29288]: 2021-02 2021-02-10T04:40:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:48 smithi164 conmon[29288]: -10T04:40:47.532280 2021-02-10T04:40:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:48 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2021-02-10T04:40:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:48 smithi164 conmon[29288]: 1775 : cluster [DBG] osdmap e828: 8 total, 8 up, 7 in 2021-02-10T04:40:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:48 smithi164 conmon[29288]: cluster 2021-02-10T04:40:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:48 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:40:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:48 smithi164 conmon[29288]: 40:48.201683+0000 mgr.y (mgr.14140) 798 2021-02-10T04:40:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:48 smithi164 conmon[29288]: : cluster [DBG] pgmap v1393: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 542 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 7.3 MiB/s rd, 545 KiB/s wr, 9 op/s 2021-02-10T04:40:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:48 smithi164 conmon[29288]: cluster 2021-02-10T04:40:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:48 smithi164 conmon[29288]: 2021-02-10T04:40:48 2021-02-10T04:40:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:48 smithi164 conmon[29288]: .266592+0000 mon.a (mon.0) 1776 : 2021-02-10T04:40:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:48 smithi164 conmon[29288]: cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 672 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:40:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:48 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:40:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:48 smithi164 conmon[29288]: -02-10T04:40:48 2021-02-10T04:40:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:48 smithi164 conmon[29288]: .271326+0000 mon.a (mon.0 2021-02-10T04:40:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:48 smithi164 conmon[29288]: ) 1777 : cluster [DBG] osdmap e829: 8 total, 8 up, 7 in 2021-02-10T04:40:49.283 INFO:tasks.rados.rados.0.smithi085.stdout:3259: done (5 left) 2021-02-10T04:40:49.283 INFO:tasks.rados.rados.0.smithi085.stdout:3261: done (4 left) 2021-02-10T04:40:49.283 INFO:tasks.rados.rados.0.smithi085.stdout:3262: delete oid 18 current snap is 377 2021-02-10T04:40:49.291 INFO:tasks.rados.rados.0.smithi085.stdout:3262: done (4 left) 2021-02-10T04:40:49.291 INFO:tasks.rados.rados.0.smithi085.stdout:3263: read oid 43 snap -1 2021-02-10T04:40:49.291 INFO:tasks.rados.rados.0.smithi085.stdout:3263: expect (ObjNum 748 snap 240 seq_num 748) 2021-02-10T04:40:49.291 INFO:tasks.rados.rados.0.smithi085.stdout:3264: copy_from oid 18 from oid 29 current snap is 377 2021-02-10T04:40:49.292 INFO:tasks.rados.rados.0.smithi085.stdout:3265: write (excl) oid 45 current snap is 377 2021-02-10T04:40:49.292 INFO:tasks.rados.rados.0.smithi085.stdout:3265: seq_num 1112 ranges {689384=664426,1971506=687726} 2021-02-10T04:40:49.299 INFO:tasks.rados.rados.0.smithi085.stdout:3265: writing smithi08575397-45 from 689384 to 1353810 tid 1 2021-02-10T04:40:49.307 INFO:tasks.rados.rados.0.smithi085.stdout:3265: writing smithi08575397-45 from 1971506 to 2659232 tid 2 2021-02-10T04:40:49.307 INFO:tasks.rados.rados.0.smithi085.stdout:3266: rollback oid 15 current snap is 377 2021-02-10T04:40:49.307 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 15 to 356 2021-02-10T04:40:49.307 INFO:tasks.rados.rados.0.smithi085.stdout:3263: expect (ObjNum 748 snap 240 seq_num 748) 2021-02-10T04:40:49.455 INFO:tasks.rados.rados.0.smithi085.stdout:3264: finishing copy_from to smithi08575397-18 2021-02-10T04:40:49.455 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 18 v 1458 (ObjNum 1091 snap 364 seq_num 1091) dirty exists 2021-02-10T04:40:49.455 INFO:tasks.rados.rados.0.smithi085.stdout:3264: finishing copy_from racing read to smithi08575397-18 2021-02-10T04:40:49.455 INFO:tasks.rados.rados.0.smithi085.stdout:3265: finishing write tid 1 to smithi08575397-45 2021-02-10T04:40:49.455 INFO:tasks.rados.rados.0.smithi085.stdout:3265: finishing write tid 2 to smithi08575397-45 2021-02-10T04:40:49.455 INFO:tasks.rados.rados.0.smithi085.stdout:3265: finishing write tid 3 to smithi08575397-45 2021-02-10T04:40:49.455 INFO:tasks.rados.rados.0.smithi085.stdout:3266: finishing rollback tid 1 to smithi08575397-15 2021-02-10T04:40:49.455 INFO:tasks.rados.rados.0.smithi085.stdout:3265: finishing write tid 4 to smithi08575397-45 2021-02-10T04:40:49.455 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 45 v 1475 (ObjNum 1112 snap 377 seq_num 1112) dirty exists 2021-02-10T04:40:49.455 INFO:tasks.rados.rados.0.smithi085.stdout:3265: left oid 45 (ObjNum 1112 snap 377 seq_num 1112) 2021-02-10T04:40:49.456 INFO:tasks.rados.rados.0.smithi085.stdout:3266: finishing rollback tid 2 to smithi08575397-15 2021-02-10T04:40:49.456 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 15 v 1460 (ObjNum 1022 snap 335 seq_num 1022) dirty exists 2021-02-10T04:40:49.456 INFO:tasks.rados.rados.0.smithi085.stdout:3263: done (7 left) 2021-02-10T04:40:49.456 INFO:tasks.rados.rados.0.smithi085.stdout:3264: done (6 left) 2021-02-10T04:40:49.456 INFO:tasks.rados.rados.0.smithi085.stdout:3265: done (5 left) 2021-02-10T04:40:49.456 INFO:tasks.rados.rados.0.smithi085.stdout:3266: done (4 left) 2021-02-10T04:40:49.456 INFO:tasks.rados.rados.0.smithi085.stdout:3267: write oid 28 current snap is 377 2021-02-10T04:40:49.456 INFO:tasks.rados.rados.0.smithi085.stdout:3267: seq_num 1113 ranges {706191=739675,1913635=431384} 2021-02-10T04:40:49.463 INFO:tasks.rados.rados.0.smithi085.stdout:3267: writing smithi08575397-28 from 706191 to 1445866 tid 1 2021-02-10T04:40:49.469 INFO:tasks.rados.rados.0.smithi085.stdout:3267: writing smithi08575397-28 from 1913635 to 2345019 tid 2 2021-02-10T04:40:49.469 INFO:tasks.rados.rados.0.smithi085.stdout:3268: snap_create 2021-02-10T04:40:49.469 INFO:tasks.rados.rados.0.smithi085.stdout:3267: finishing write tid 1 to smithi08575397-28 2021-02-10T04:40:49.472 INFO:tasks.rados.rados.0.smithi085.stdout:3267: finishing write tid 2 to smithi08575397-28 2021-02-10T04:40:49.473 INFO:tasks.rados.rados.0.smithi085.stdout:3267: finishing write tid 3 to smithi08575397-28 2021-02-10T04:40:49.473 INFO:tasks.rados.rados.0.smithi085.stdout:3267: finishing write tid 4 to smithi08575397-28 2021-02-10T04:40:49.473 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 1442 (ObjNum 1113 snap 377 seq_num 1113) dirty exists 2021-02-10T04:40:49.473 INFO:tasks.rados.rados.0.smithi085.stdout:3267: left oid 28 (ObjNum 1113 snap 377 seq_num 1113) 2021-02-10T04:40:49.541 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:49 smithi164 conmon[33677]: debug 2021-02-10T04:40:49.162+0000 7fbec19a1700 -1 osd.4 829 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:49 smithi164 conmon[29288]: cluster 2021-02-10T04:40:48.161326+0000 osd.4 (osd.4) 1409 2021-02-10T04:40:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:49 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:49 smithi164 conmon[29288]: cluster 2021-02-10T04:40:48.161336+0000 osd.4 (osd.4) 1410 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:49 smithi164 conmon[29288]: cluster 2021-02-10T04:40:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:49 smithi164 conmon[29288]: 2021-02-10T04:40:48.161344+0000 osd.4 (osd.4) 1411 : cluster [WRN] 2021-02-10T04:40:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:49 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:49 smithi164 conmon[29288]: cluster 2021-02-10T04:40:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:49 smithi164 conmon[29288]: 2021-02- 2021-02-10T04:40:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:49 smithi164 conmon[29288]: 10T04:40:48.161351+0000 osd.4 (osd 2021-02-10T04:40:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:49 smithi164 conmon[29288]: .4) 1412 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:49 smithi164 conmon[29288]: 2021-02-10T04:40:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:49 smithi164 conmon[29288]: audit 2021-02-10T04:40:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:49 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:40:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:49 smithi164 conmon[29288]: :40:48.417083+0000 mgr.y 2021-02-10T04:40:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:49 smithi164 conmon[29288]: (mgr.14140) 799 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:40:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:49 smithi164 conmon[29288]: 2021-02-10T04:40:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:49 smithi164 conmon[29288]: cluster 2021-02-10T04:40 2021-02-10T04:40:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:49 smithi164 conmon[29288]: :49.281012+0000 mon.a (mon.0) 1778 2021-02-10T04:40:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:49 smithi164 conmon[29288]: : cluster [DBG] osdmap e830: 8 total, 8 up, 7 in 2021-02-10T04:40:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[29319]: cluster 2021-02-10T04:40:48.161326+0000 osd.4 (osd.4) 1409 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[29319]: cluster 2021-02-10T04:40:48.161336+0000 osd.4 (osd.4) 1410 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[29319]: cluster 2021-02-10T04:40:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:40:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[29319]: 40:48.161344 2021-02-10T04:40:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2021-02-10T04:40:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[29319]: 1411 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:40:49.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[29319]: 02-10T04:40:48.161351+0000 osd.4 2021-02-10T04:40:49.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[29319]: (osd.4) 1412 : cluster 2021-02-10T04:40:49.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:49.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:40:49.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[29319]: 02-10T04:40:48.417083+0000 mgr.y (mgr 2021-02-10T04:40:49.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[29319]: .14140) 799 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:40:49.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:40:49.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[29319]: 02-10T04:40:49.281012+0000 mon.a ( 2021-02-10T04:40:49.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[29319]: mon.0) 1778 : cluster [DBG] osdmap e830: 8 total, 8 up, 7 in 2021-02-10T04:40:49.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[35961]: cluster 2021-02-10T04:40:49.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[35961]: 2021-02-10T04:40:49.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[35961]: 2021 2021-02-10T04:40:49.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[35961]: - 2021-02-10T04:40:49.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[35961]: 02 2021-02-10T04:40:49.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[35961]: -10T04:40:48.161326+0000 2021-02-10T04:40:49.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[35961]: osd.4 (osd.4) 1409 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:49.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[35961]: cluster 2021-02-10T04:40:48.161336+0000 osd.4 2021-02-10T04:40:49.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[35961]: (osd.4) 1410 : cluster [WRN] 2021-02-10T04:40:49.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:49.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[35961]: cluster 2021-02-10T04:40:48 2021-02-10T04:40:49.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[35961]: .161344+0000 osd.4 (osd.4) 1411 : cluster 2021-02-10T04:40:49.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:49.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:40:49.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[35961]: 10T04:40:48. 2021-02-10T04:40:49.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[35961]: 161351+0000 osd.4 (osd.4) 1412 : cluster 2021-02-10T04:40:49.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:49.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:40:49.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[35961]: 04:40:48.417083+0000 mgr.y (mgr.14140 2021-02-10T04:40:49.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[35961]: ) 799 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:40:49.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:40:49.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[35961]: -02-10T04:40:49.281012+0000 2021-02-10T04:40:49.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[35961]: mon.a (mon.0) 1778 : cluster [DBG] 2021-02-10T04:40:49.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:49 smithi085 conmon[35961]: osdmap e830: 8 total, 8 up, 7 in 2021-02-10T04:40:50.282 INFO:tasks.rados.rados.0.smithi085.stdout:3267: done (5 left) 2021-02-10T04:40:50.282 INFO:tasks.rados.rados.0.smithi085.stdout:3268: done (4 left) 2021-02-10T04:40:50.283 INFO:tasks.rados.rados.0.smithi085.stdout:3269: read oid 31 snap 367 2021-02-10T04:40:50.283 INFO:tasks.rados.rados.0.smithi085.stdout:3269: expect deleted 2021-02-10T04:40:50.283 INFO:tasks.rados.rados.0.smithi085.stdout:3270: write oid 21 current snap is 378 2021-02-10T04:40:50.283 INFO:tasks.rados.rados.0.smithi085.stdout:3270: seq_num 1114 ranges {722998=414924,1572117=575042} 2021-02-10T04:40:50.288 INFO:tasks.rados.rados.0.smithi085.stdout:3270: writing smithi08575397-21 from 722998 to 1137922 tid 1 2021-02-10T04:40:50.295 INFO:tasks.rados.rados.0.smithi085.stdout:3270: writing smithi08575397-21 from 1572117 to 2147159 tid 2 2021-02-10T04:40:50.295 INFO:tasks.rados.rados.0.smithi085.stdout:3271: rollback oid 49 current snap is 378 2021-02-10T04:40:50.295 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 49 to 374 2021-02-10T04:40:50.296 INFO:tasks.rados.rados.0.smithi085.stdout:3270: finishing write tid 1 to smithi08575397-21 2021-02-10T04:40:50.296 INFO:tasks.rados.rados.0.smithi085.stdout:3269: done (6 left) 2021-02-10T04:40:50.296 INFO:tasks.rados.rados.0.smithi085.stdout:3272: read oid 30 snap -1 2021-02-10T04:40:50.296 INFO:tasks.rados.rados.0.smithi085.stdout:3272: expect deleted 2021-02-10T04:40:50.296 INFO:tasks.rados.rados.0.smithi085.stdout:3273: copy_from oid 14 from oid 31 current snap is 378 2021-02-10T04:40:50.296 INFO:tasks.rados.rados.0.smithi085.stdout:3274: write (excl) oid 45 current snap is 378 2021-02-10T04:40:50.296 INFO:tasks.rados.rados.0.smithi085.stdout:3274: seq_num 1115 ranges {739805=490173,1630599=435053,2636284=103521} 2021-02-10T04:40:50.303 INFO:tasks.rados.rados.0.smithi085.stdout:3274: writing smithi08575397-45 from 739805 to 1229978 tid 1 2021-02-10T04:40:50.308 INFO:tasks.rados.rados.0.smithi085.stdout:3274: writing smithi08575397-45 from 1630599 to 2065652 tid 2 2021-02-10T04:40:50.312 INFO:tasks.rados.rados.0.smithi085.stdout:3274: writing smithi08575397-45 from 2636284 to 2739805 tid 3 2021-02-10T04:40:50.313 INFO:tasks.rados.rados.0.smithi085.stdout:3275: write oid 26 current snap is 378 2021-02-10T04:40:50.313 INFO:tasks.rados.rados.0.smithi085.stdout:3275: seq_num 1116 ranges {756612=565422,2089081=578711} 2021-02-10T04:40:50.321 INFO:tasks.rados.rados.0.smithi085.stdout:3275: writing smithi08575397-26 from 756612 to 1322034 tid 1 2021-02-10T04:40:50.330 INFO:tasks.rados.rados.0.smithi085.stdout:3275: writing smithi08575397-26 from 2089081 to 2667792 tid 2 2021-02-10T04:40:50.330 INFO:tasks.rados.rados.0.smithi085.stdout:3276: read oid 4 snap 368 2021-02-10T04:40:50.330 INFO:tasks.rados.rados.0.smithi085.stdout:3276: expect (ObjNum 1049 snap 344 seq_num 1049) 2021-02-10T04:40:50.330 INFO:tasks.rados.rados.0.smithi085.stdout:3277: write oid 48 current snap is 378 2021-02-10T04:40:50.330 INFO:tasks.rados.rados.0.smithi085.stdout:3271: finishing rollback tid 0 to smithi08575397-49 2021-02-10T04:40:50.330 INFO:tasks.rados.rados.0.smithi085.stdout:3271: finishing rollback tid 1 to smithi08575397-49 2021-02-10T04:40:50.330 INFO:tasks.rados.rados.0.smithi085.stdout:3271: finishing rollback tid 2 to smithi08575397-49 2021-02-10T04:40:50.331 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 49 v 1482 (ObjNum 1107 snap 374 seq_num 1107) dirty exists 2021-02-10T04:40:50.331 INFO:tasks.rados.rados.0.smithi085.stdout:3270: finishing write tid 2 to smithi08575397-21 2021-02-10T04:40:50.331 INFO:tasks.rados.rados.0.smithi085.stdout:3270: finishing write tid 3 to smithi08575397-21 2021-02-10T04:40:50.331 INFO:tasks.rados.rados.0.smithi085.stdout:3277: seq_num 1117 ranges {773419=640671,2147563=438722} 2021-02-10T04:40:50.338 INFO:tasks.rados.rados.0.smithi085.stdout:3277: writing smithi08575397-48 from 773419 to 1414090 tid 1 2021-02-10T04:40:50.345 INFO:tasks.rados.rados.0.smithi085.stdout:3277: writing smithi08575397-48 from 2147563 to 2586285 tid 2 2021-02-10T04:40:50.346 INFO:tasks.rados.rados.0.smithi085.stdout:3271: done (11 left) 2021-02-10T04:40:50.346 INFO:tasks.rados.rados.0.smithi085.stdout:3272: done (10 left) 2021-02-10T04:40:50.346 INFO:tasks.rados.rados.0.smithi085.stdout:3278: copy_from oid 43 from oid 15 current snap is 378 2021-02-10T04:40:50.350 INFO:tasks.rados.rados.0.smithi085.stdout:3270: finishing write tid 4 to smithi08575397-21 2021-02-10T04:40:50.350 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 1693 (ObjNum 1114 snap 378 seq_num 1114) dirty exists 2021-02-10T04:40:50.350 INFO:tasks.rados.rados.0.smithi085.stdout:3270: left oid 21 (ObjNum 1114 snap 378 seq_num 1114) 2021-02-10T04:40:50.351 INFO:tasks.rados.rados.0.smithi085.stdout:3274: finishing write tid 1 to smithi08575397-45 2021-02-10T04:40:50.351 INFO:tasks.rados.rados.0.smithi085.stdout:3273: finishing copy_from to smithi08575397-14 2021-02-10T04:40:50.351 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 1469 (ObjNum 1097 snap 368 seq_num 1097) dirty exists 2021-02-10T04:40:50.351 INFO:tasks.rados.rados.0.smithi085.stdout:3273: finishing copy_from racing read to smithi08575397-14 2021-02-10T04:40:50.351 INFO:tasks.rados.rados.0.smithi085.stdout:3274: finishing write tid 2 to smithi08575397-45 2021-02-10T04:40:50.351 INFO:tasks.rados.rados.0.smithi085.stdout:3274: finishing write tid 3 to smithi08575397-45 2021-02-10T04:40:50.352 INFO:tasks.rados.rados.0.smithi085.stdout:3274: finishing write tid 4 to smithi08575397-45 2021-02-10T04:40:50.352 INFO:tasks.rados.rados.0.smithi085.stdout:3274: finishing write tid 5 to smithi08575397-45 2021-02-10T04:40:50.352 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 45 v 1479 (ObjNum 1115 snap 378 seq_num 1115) dirty exists 2021-02-10T04:40:50.352 INFO:tasks.rados.rados.0.smithi085.stdout:3274: left oid 45 (ObjNum 1115 snap 378 seq_num 1115) 2021-02-10T04:40:50.352 INFO:tasks.rados.rados.0.smithi085.stdout:3275: finishing write tid 1 to smithi08575397-26 2021-02-10T04:40:50.352 INFO:tasks.rados.rados.0.smithi085.stdout:3275: finishing write tid 2 to smithi08575397-26 2021-02-10T04:40:50.353 INFO:tasks.rados.rados.0.smithi085.stdout:3276: expect (ObjNum 1049 snap 344 seq_num 1049) 2021-02-10T04:40:50.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:50 smithi164 conmon[33677]: debug 2021-02-10T04:40:50.132+0000 7fbec19a1700 -1 osd.4 830 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:50.515 INFO:tasks.rados.rados.0.smithi085.stdout:3275: finishing write tid 3 to smithi08575397-26 2021-02-10T04:40:50.515 INFO:tasks.rados.rados.0.smithi085.stdout:3275: finishing write tid 4 to smithi08575397-26 2021-02-10T04:40:50.515 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 1358 (ObjNum 1116 snap 378 seq_num 1116) dirty exists 2021-02-10T04:40:50.516 INFO:tasks.rados.rados.0.smithi085.stdout:3275: left oid 26 (ObjNum 1116 snap 378 seq_num 1116) 2021-02-10T04:40:50.516 INFO:tasks.rados.rados.0.smithi085.stdout:3277: finishing write tid 1 to smithi08575397-48 2021-02-10T04:40:50.516 INFO:tasks.rados.rados.0.smithi085.stdout:3270: done (10 left) 2021-02-10T04:40:50.516 INFO:tasks.rados.rados.0.smithi085.stdout:3273: done (9 left) 2021-02-10T04:40:50.516 INFO:tasks.rados.rados.0.smithi085.stdout:3274: done (8 left) 2021-02-10T04:40:50.516 INFO:tasks.rados.rados.0.smithi085.stdout:3275: done (7 left) 2021-02-10T04:40:50.516 INFO:tasks.rados.rados.0.smithi085.stdout:3276: done (6 left) 2021-02-10T04:40:50.516 INFO:tasks.rados.rados.0.smithi085.stdout:3279: read oid 25 snap -1 2021-02-10T04:40:50.516 INFO:tasks.rados.rados.0.smithi085.stdout:3279: expect deleted 2021-02-10T04:40:50.516 INFO:tasks.rados.rados.0.smithi085.stdout:3280: copy_from oid 28 from oid 36 current snap is 378 2021-02-10T04:40:50.516 INFO:tasks.rados.rados.0.smithi085.stdout:3277: finishing write tid 2 to smithi08575397-48 2021-02-10T04:40:50.516 INFO:tasks.rados.rados.0.smithi085.stdout:3277: finishing write tid 3 to smithi08575397-48 2021-02-10T04:40:50.516 INFO:tasks.rados.rados.0.smithi085.stdout:3277: finishing write tid 4 to smithi08575397-48 2021-02-10T04:40:50.517 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 1480 (ObjNum 1117 snap 378 seq_num 1117) dirty exists 2021-02-10T04:40:50.517 INFO:tasks.rados.rados.0.smithi085.stdout:3277: left oid 48 (ObjNum 1117 snap 378 seq_num 1117) 2021-02-10T04:40:50.517 INFO:tasks.rados.rados.0.smithi085.stdout:3277: done (7 left) 2021-02-10T04:40:50.517 INFO:tasks.rados.rados.0.smithi085.stdout:3281: snap_create 2021-02-10T04:40:50.532 INFO:tasks.rados.rados.0.smithi085.stdout:3280: finishing copy_from to smithi08575397-28 2021-02-10T04:40:50.532 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 1302 (ObjNum 991 snap 326 seq_num 991) dirty exists 2021-02-10T04:40:50.532 INFO:tasks.rados.rados.0.smithi085.stdout:3280: finishing copy_from racing read to smithi08575397-28 2021-02-10T04:40:50.825 INFO:tasks.rados.rados.0.smithi085.stdout:3278: finishing copy_from to smithi08575397-43 2021-02-10T04:40:50.825 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 1460 (ObjNum 1022 snap 335 seq_num 1022) dirty exists 2021-02-10T04:40:50.825 INFO:tasks.rados.rados.0.smithi085.stdout:3278: finishing copy_from racing read to smithi08575397-43 2021-02-10T04:40:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:50 smithi164 conmon[29288]: cluster 2021-02-10T04:40:49.163253+0000 osd.4 2021-02-10T04:40:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:50 smithi164 conmon[29288]: (osd.4) 1413 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:50 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:40:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:50 smithi164 conmon[29288]: 04:40:49.163259+0000 osd.4 (osd.4) 1414 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:50 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:40:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:50 smithi164 conmon[29288]: 02-10T04:40:49.163263+0000 osd.4 (osd.4 2021-02-10T04:40:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:50 smithi164 conmon[29288]: ) 1415 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:50 smithi164 conmon[29288]: cluster 2021-02-10T04:40:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:50 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:40:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:50 smithi164 conmon[29288]: 40:49.163267+0000 osd.4 (osd.4) 1416 2021-02-10T04:40:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:50 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:50 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:40:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:50 smithi164 conmon[29288]: 02-10T04:40:50.132903+0000 osd.4 2021-02-10T04:40:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:50 smithi164 conmon[29288]: (osd.4) 1417 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:50 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:40:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:50 smithi164 conmon[29288]: -10T04:40:50.132912+0000 osd.4 (osd.4 2021-02-10T04:40:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:50 smithi164 conmon[29288]: ) 1418 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:50 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:40:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:50 smithi164 conmon[29288]: 04:40:50.132918+0000 osd.4 (osd.4) 2021-02-10T04:40:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:50 smithi164 conmon[29288]: 1419 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:50 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:40:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:50 smithi164 conmon[29288]: :40:50.132924+0000 osd.4 (osd.4) 2021-02-10T04:40:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:50 smithi164 conmon[29288]: 1420 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:50.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:50 smithi164 conmon[29288]: cluster 2021-02-10T04:40:50.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:50 smithi164 conmon[29288]: 2021-02-10T04:40:50.202407+0000 2021-02-10T04:40:50.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:50 smithi164 conmon[29288]: mgr.y (mgr.14140) 800 : cluster [DBG] pgmap v1396: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 550 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 26 MiB/s rd, 1.7 MiB/s wr, 29 op/s 2021-02-10T04:40:50.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:50 smithi164 conmon[29288]: 2021-02-10T04:40:50.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:50 smithi164 conmon[29288]: cluster 2021-02-10T04:40:50.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:50 smithi164 conmon[29288]: 2021-02-10T04:40:50.280717+0000 mon.a (mon.0) 1779 : cluster [DBG] osdmap e831: 8 total, 8 up, 7 in 2021-02-10T04:40:50.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: cluster 2021-02-10T04:40:50.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: 2021-02-10T04:40:50.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: 2021 2021-02-10T04:40:50.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: - 2021-02-10T04:40:50.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: 02 2021-02-10T04:40:50.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: - 2021-02-10T04:40:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: 10 2021-02-10T04:40:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: T 2021-02-10T04:40:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: 04 2021-02-10T04:40:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: : 2021-02-10T04:40:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: 40 2021-02-10T04:40:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: : 2021-02-10T04:40:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: 49 2021-02-10T04:40:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: . 2021-02-10T04:40:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: 163253 2021-02-10T04:40:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1413 : cluster 2021-02-10T04:40:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:40:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: 10T04:40:49.163259+0000 osd.4 (osd.4) 1414 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: 2021-02-10T04:40:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: cluster 2021-02-10T04:40:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: 2021-02-10T04:40:49.163263+0000 osd.4 (osd 2021-02-10T04:40:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: .4) 1415 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:40:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: 04:40:49.163267+0000 osd.4 (osd.4 2021-02-10T04:40:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: ) 1416 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:40:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: :40:50.132903+0000 osd.4 (osd.4 2021-02-10T04:40:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: ) 1417 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: cluster 2021-02-10T04:40:50.132912 2021-02-10T04:40:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1418 : cluster [WRN] 2021-02-10T04:40:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:40:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: 10T04:40:50. 2021-02-10T04:40:50.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: 132918+0000 osd.4 (osd.4 2021-02-10T04:40:50.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: ) 1419 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:50.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: 2021-02-10T04:40:50.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:40:50.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: T04:40:50. 2021-02-10T04:40:50.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: 132924+0000 osd.4 (osd.4 2021-02-10T04:40:50.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: ) 1420 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:50.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: 2021-02-10T04:40:50.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:40:50.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: -10T04:40: 2021-02-10T04:40:50.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: 50.202407+0000 mgr.y (mgr 2021-02-10T04:40:50.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: .14140) 800 : cluster [DBG] 2021-02-10T04:40:50.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: pgmap v1396: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 550 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 26 MiB/s rd, 1.7 MiB/s wr, 29 op/s 2021-02-10T04:40:50.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:40:50.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: -02-10T04:40 2021-02-10T04:40:50.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: :50.280717+0000 mon.a ( 2021-02-10T04:40:50.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: mon.0) 1779 : cluster 2021-02-10T04:40:50.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[29319]: [DBG] osdmap e831: 8 total, 8 up, 7 in 2021-02-10T04:40:50.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[35961]: cluster 2021-02-10T04:40:49.163253+0000 osd.4 (osd.4) 1413 : 2021-02-10T04:40:50.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:50.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[35961]: cluster 2021-02-10T04:40:49.163259+0000 osd.4 (osd.4) 1414 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:50.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[35961]: cluster 2021-02-10T04:40:50.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[35961]: 2021-02-10T04:40:49.163263+0000 osd.4 (osd.4) 1415 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:50.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[35961]: cluster 2021-02-10T04:40:49.163267+0000 2021-02-10T04:40:50.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[35961]: osd.4 (osd.4) 1416 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:50.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[35961]: cluster 2021-02-10T04:40:50.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[35961]: 2021-02-10T04:40:50.132903+0000 osd.4 (osd.4) 1417 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:50.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[35961]: cluster 2021-02-10T04:40:50.132912+0000 osd.4 (osd.4) 1418 : cluster [WRN] 2021-02-10T04:40:50.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:50.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[35961]: cluster 2021-02-10T04:40:50.132918+0000 2021-02-10T04:40:50.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[35961]: osd.4 (osd.4) 1419 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:50.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[35961]: cluster 2021-02-10T04:40:50.132924+0000 osd.4 (osd. 2021-02-10T04:40:50.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[35961]: 4) 1420 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:50.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[35961]: cluster 2021-02-10T04:40:50.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:40:50.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[35961]: :40:50.202407+0000 mgr.y (mgr.14140) 800 : cluster [DBG] pgmap v1396: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 550 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 26 MiB/s rd, 1.7 MiB/s wr, 29 op/s 2021-02-10T04:40:50.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:40:50.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:50 smithi085 conmon[35961]: 02-10T04:40:50.280717+0000 mon.a (mon.0) 1779 : cluster [DBG] osdmap e831: 8 total, 8 up, 7 in 2021-02-10T04:40:51.287 INFO:tasks.rados.rados.0.smithi085.stdout:3278: done (7 left) 2021-02-10T04:40:51.287 INFO:tasks.rados.rados.0.smithi085.stdout:3279: done (6 left) 2021-02-10T04:40:51.288 INFO:tasks.rados.rados.0.smithi085.stdout:3280: done (5 left) 2021-02-10T04:40:51.288 INFO:tasks.rados.rados.0.smithi085.stdout:3281: done (4 left) 2021-02-10T04:40:51.288 INFO:tasks.rados.rados.0.smithi085.stdout:3282: delete oid 6 current snap is 379 2021-02-10T04:40:51.292 INFO:tasks.rados.rados.0.smithi085.stdout:3282: done (4 left) 2021-02-10T04:40:51.292 INFO:tasks.rados.rados.0.smithi085.stdout:3283: read oid 29 snap -1 2021-02-10T04:40:51.292 INFO:tasks.rados.rados.0.smithi085.stdout:3283: expect (ObjNum 1091 snap 364 seq_num 1091) 2021-02-10T04:40:51.292 INFO:tasks.rados.rados.0.smithi085.stdout:3284: write (excl) oid 44 current snap is 379 2021-02-10T04:40:51.292 INFO:tasks.rados.rados.0.smithi085.stdout:3284: seq_num 1119 ranges {407033=507522,1464527=442391,2345976=461057} 2021-02-10T04:40:51.298 INFO:tasks.rados.rados.0.smithi085.stdout:3284: writing smithi08575397-44 from 407033 to 914555 tid 1 2021-02-10T04:40:51.305 INFO:tasks.rados.rados.0.smithi085.stdout:3284: writing smithi08575397-44 from 1464527 to 1906918 tid 2 2021-02-10T04:40:51.311 INFO:tasks.rados.rados.0.smithi085.stdout:3284: writing smithi08575397-44 from 2345976 to 2807033 tid 3 2021-02-10T04:40:51.311 INFO:tasks.rados.rados.0.smithi085.stdout:3285: delete oid 12 current snap is 379 2021-02-10T04:40:51.311 INFO:tasks.rados.rados.0.smithi085.stdout:3283: expect (ObjNum 1091 snap 364 seq_num 1091) 2021-02-10T04:40:51.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:51 smithi164 conmon[33677]: debug 2021-02-10T04:40:51.083+0000 7fbec19a1700 -1 osd.4 831 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:51.462 INFO:tasks.rados.rados.0.smithi085.stdout:3284: finishing write tid 1 to smithi08575397-44 2021-02-10T04:40:51.462 INFO:tasks.rados.rados.0.smithi085.stdout:3284: finishing write tid 2 to smithi08575397-44 2021-02-10T04:40:51.462 INFO:tasks.rados.rados.0.smithi085.stdout:3284: finishing write tid 3 to smithi08575397-44 2021-02-10T04:40:51.462 INFO:tasks.rados.rados.0.smithi085.stdout:3284: finishing write tid 4 to smithi08575397-44 2021-02-10T04:40:51.462 INFO:tasks.rados.rados.0.smithi085.stdout:3284: finishing write tid 5 to smithi08575397-44 2021-02-10T04:40:51.462 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 1449 (ObjNum 1119 snap 379 seq_num 1119) dirty exists 2021-02-10T04:40:51.462 INFO:tasks.rados.rados.0.smithi085.stdout:3284: left oid 44 (ObjNum 1119 snap 379 seq_num 1119) 2021-02-10T04:40:51.463 INFO:tasks.rados.rados.0.smithi085.stdout:3283: done (6 left) 2021-02-10T04:40:51.463 INFO:tasks.rados.rados.0.smithi085.stdout:3284: done (5 left) 2021-02-10T04:40:51.463 INFO:tasks.rados.rados.0.smithi085.stdout:3285: done (4 left) 2021-02-10T04:40:51.463 INFO:tasks.rados.rados.0.smithi085.stdout:3286: read oid 22 snap 266 2021-02-10T04:40:51.463 INFO:tasks.rados.rados.0.smithi085.stdout:3286: expect deleted 2021-02-10T04:40:51.463 INFO:tasks.rados.rados.0.smithi085.stdout:3287: snap_create 2021-02-10T04:40:51.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:51 smithi164 conmon[29288]: cluster 2021-02-10T04:40:51.084217+0000 osd.4 (osd.4) 1421 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:51.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:51 smithi164 conmon[29288]: cluster 2021-02-10T04:40:51.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:51 smithi164 conmon[29288]: 2021-02-10T04:40:51.084225+0000 osd.4 (osd.4) 1422 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:51.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:51 smithi164 conmon[29288]: cluster 2021-02-10T04:40:51.084231+0000 osd.4 (osd.4) 1423 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:51.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:51 smithi164 conmon[29288]: cluster 2021-02-10T04:40 2021-02-10T04:40:51.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:51 smithi164 conmon[29288]: :51.084238+0000 osd.4 (osd.4) 1424 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:51.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:51 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:40:51.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:51 smithi164 conmon[29288]: 02-10T04:40:51.286427 2021-02-10T04:40:51.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:51 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1780 2021-02-10T04:40:51.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:51 smithi164 conmon[29288]: : cluster [DBG] osdmap e832: 8 total, 8 up, 7 in 2021-02-10T04:40:51.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[29319]: cluster 2021-02-10T04:40:51.084217+0000 osd.4 (osd 2021-02-10T04:40:51.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[29319]: .4) 1421 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:51.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[29319]: cluster 2021-02-10T04:40:51.084225+0000 osd.4 (osd.4) 1422 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:51.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[29319]: cluster 2021-02-10T04:40:51.084231+0000 osd.4 (osd.4) 1423 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:51.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[29319]: cluster 2021-02-10T04:40:51. 2021-02-10T04:40:51.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[29319]: 084238+0000 osd.4 (osd.4) 1424 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:51.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[29319]: cluster 2021-02-10T04:40:51.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:40:51.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[29319]: 40:51.286427+0000 mon.a (mon.0) 1780 2021-02-10T04:40:51.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[29319]: : cluster [DBG] osdmap e832: 8 total, 8 up, 7 in 2021-02-10T04:40:51.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[35961]: cluster 2021-02-10T04:40:51.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[35961]: 2021-02-10T04:40:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[35961]: 2021 2021-02-10T04:40:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[35961]: - 2021-02-10T04:40:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[35961]: 02 2021-02-10T04:40:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[35961]: - 2021-02-10T04:40:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[35961]: 10 2021-02-10T04:40:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[35961]: T 2021-02-10T04:40:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[35961]: 04 2021-02-10T04:40:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[35961]: : 2021-02-10T04:40:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[35961]: 40 2021-02-10T04:40:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[35961]: : 2021-02-10T04:40:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[35961]: 51 2021-02-10T04:40:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[35961]: .084217+0000 osd.4 (osd.4) 1421 2021-02-10T04:40:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[35961]: cluster 2021-02-10T04:40:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[35961]: 2021-02-10T04:40:51.084225 2021-02-10T04:40:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1422 : 2021-02-10T04:40:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[35961]: cluster 2021-02-10T04:40:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:40:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[35961]: :40:51.084231+0000 osd.4 (osd.4) 1423 : cluster [WRN] 2021-02-10T04:40:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[35961]: cluster 2021-02-10T04:40:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[35961]: 2021-02-10T04:40:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[35961]: 2021-02-10T04:40:51 2021-02-10T04:40:51.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[35961]: .084238+0000 osd.4 (osd.4) 1424 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:51.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[35961]: cluster 2021-02-10T04:40:51.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:40:51.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:51 smithi085 conmon[35961]: :40:51.286427+0000 mon.a (mon.0) 1780 : cluster [DBG] osdmap e832: 8 total, 8 up, 7 in 2021-02-10T04:40:52.287 INFO:tasks.rados.rados.0.smithi085.stdout:3286: done (5 left) 2021-02-10T04:40:52.287 INFO:tasks.rados.rados.0.smithi085.stdout:3287: done (4 left) 2021-02-10T04:40:52.287 INFO:tasks.rados.rados.0.smithi085.stdout:3288: copy_from oid 28 from oid 20 current snap is 380 2021-02-10T04:40:52.287 INFO:tasks.rados.rados.0.smithi085.stdout:3289: write (excl) oid 14 current snap is 380 2021-02-10T04:40:52.287 INFO:tasks.rados.rados.0.smithi085.stdout:3289: seq_num 1121 ranges {440647=658020,1581491=446060,2800822=39825} 2021-02-10T04:40:52.293 INFO:tasks.rados.rados.0.smithi085.stdout:3289: writing smithi08575397-14 from 440647 to 1098667 tid 1 2021-02-10T04:40:52.300 INFO:tasks.rados.rados.0.smithi085.stdout:3289: writing smithi08575397-14 from 1581491 to 2027551 tid 2 2021-02-10T04:40:52.304 INFO:tasks.rados.rados.0.smithi085.stdout:3289: writing smithi08575397-14 from 2800822 to 2840647 tid 3 2021-02-10T04:40:52.305 INFO:tasks.rados.rados.0.smithi085.stdout:3290: snap_remove snap 376 2021-02-10T04:40:52.305 INFO:tasks.rados.rados.0.smithi085.stdout:3289: finishing write tid 1 to smithi08575397-14 2021-02-10T04:40:52.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:52 smithi164 conmon[33677]: debug 2021-02-10T04:40:52.124+0000 7fbec19a1700 -1 osd.4 832 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:52.817 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:40:52.850 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[29319]: cluster 2021-02-10T04:40:52.850 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[29319]: 2021-02-10T04:40 2021-02-10T04:40:52.850 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[29319]: :52.125013+0000 osd.4 (osd.4) 1425 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:52.850 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[29319]: cluster 2021-02-10T04:40 2021-02-10T04:40:52.851 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[29319]: :52.125025+0000 osd.4 (osd.4) 1426 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:52.851 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[29319]: 2021-02-10T04:40:52.851 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:40:52.851 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[29319]: 40:52.125031+0000 osd.4 (osd 2021-02-10T04:40:52.851 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[29319]: .4) 1427 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:52.851 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[29319]: cluster 2021-02-10T04:40:52.125038 2021-02-10T04:40:52.851 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1428 : cluster [WRN] 2021-02-10T04:40:52.851 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:52.851 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:40:52.851 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[29319]: -02-10T04:40:52.203128+0000 mgr.y (mgr.14140) 801 2021-02-10T04:40:52.852 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[29319]: : cluster [DBG] pgmap v1399: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 550 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 19 MiB/s rd, 1.4 MiB/s wr, 21 op/s 2021-02-10T04:40:52.852 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:40:52.852 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[29319]: 04:40:52.258329+0000 osd.1 (osd.1 2021-02-10T04:40:52.852 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[29319]: ) 459 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:40:52.852 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:40:52.852 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[29319]: T04:40:52.259619+0000 osd.1 (osd 2021-02-10T04:40:52.852 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[29319]: .1) 460 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:40:52.852 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:40:52.852 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[29319]: :40:52.285756+0000 mon.a (mon.0) 1781 : cluster 2021-02-10T04:40:52.852 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[29319]: [DBG] osdmap e833: 8 total, 8 up, 7 in 2021-02-10T04:40:52.852 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:40:52.853 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[35961]: 04:40:52.125013+0000 osd.4 (osd.4) 1425 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:52.853 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:40:52.853 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[35961]: :40:52.125025+0000 osd.4 (osd.4) 1426 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:52.853 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[35961]: cluster 2021-02-10T04:40:52.853 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[35961]: 2021-02-10T04:40 2021-02-10T04:40:52.853 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[35961]: :52.125031+0000 osd.4 (osd.4) 1427 : cluster [WRN] 2021-02-10T04:40:52.853 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:52.853 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:40:52.853 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[35961]: -02-10T04:40:52.125038+0000 osd.4 ( 2021-02-10T04:40:52.853 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[35961]: osd.4) 1428 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:52.853 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:40:52.853 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[35961]: T04:40:52.203128+0000 mgr.y (mgr 2021-02-10T04:40:52.854 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[35961]: .14140) 801 : cluster [DBG] pgmap v1399: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 550 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 19 MiB/s rd, 1.4 MiB/s wr, 21 op/s 2021-02-10T04:40:52.854 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:40:52.854 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[35961]: :40:52.258329+0000 osd.1 (osd.1) 459 : cluster 2021-02-10T04:40:52.854 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[35961]: [DBG] purged_snaps scrub starts 2021-02-10T04:40:52.854 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[35961]: cluster 2021-02-10T04:40:52.854 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[35961]: 2021-02-10T04:40:52.259619+0000 osd.1 (osd.1) 2021-02-10T04:40:52.854 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[35961]: 460 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:40:52.854 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:40:52.854 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[35961]: 02-10T04:40:52.285756+0000 mon.a (mon 2021-02-10T04:40:52.854 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:52 smithi085 conmon[35961]: .0) 1781 : cluster [DBG] osdmap e833: 8 total, 8 up, 7 in 2021-02-10T04:40:52.855 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:40:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:52 smithi164 conmon[29288]: cluster 2021-02-10T04:40:52.125013+0000 osd.4 (osd. 2021-02-10T04:40:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:52 smithi164 conmon[29288]: 4) 1425 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:52 smithi164 conmon[29288]: cluster 2021-02-10T04:40:52.125025+0000 osd.4 (osd.4) 1426 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:52 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:40:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:52 smithi164 conmon[29288]: -10T04:40:52.125031+0000 osd.4 (osd.4) 1427 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:52 smithi164 conmon[29288]: cluster 2021-02-10T04:40:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:52 smithi164 conmon[29288]: 2021-02-10T04:40:52.125038+0000 osd.4 (osd. 2021-02-10T04:40:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:52 smithi164 conmon[29288]: 4) 1428 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:52 smithi164 conmon[29288]: cluster 2021-02-10T04:40:52.203128+0000 2021-02-10T04:40:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:52 smithi164 conmon[29288]: mgr.y (mgr.14140) 801 : cluster [DBG] pgmap v1399: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 550 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 19 MiB/s rd, 1.4 MiB/s wr, 21 op/s 2021-02-10T04:40:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:52 smithi164 conmon[29288]: cluster 2021-02-10T04:40:52 2021-02-10T04:40:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:52 smithi164 conmon[29288]: .258329+0000 osd.1 (osd.1) 459 : cluster 2021-02-10T04:40:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:52 smithi164 conmon[29288]: [DBG] purged_snaps scrub starts 2021-02-10T04:40:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:52 smithi164 conmon[29288]: cluster 2021-02-10T04:40:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:52 smithi164 conmon[29288]: 2021-02-10T04:40:52.259619+0000 osd.1 (osd. 2021-02-10T04:40:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:52 smithi164 conmon[29288]: 1) 460 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:40:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:52 smithi164 conmon[29288]: cluster 2021-02-10T04:40:52.285756+0000 2021-02-10T04:40:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:52 smithi164 conmon[29288]: mon.a (mon.0) 1781 : cluster [DBG] osdmap e833: 8 total, 8 up, 7 in 2021-02-10T04:40:53.160 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.2 dump_blocked_ops 2021-02-10T04:40:53.260 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 4 1 2021-02-10T04:40:53.298 INFO:tasks.rados.rados.0.smithi085.stdout:3290: done (6 left) 2021-02-10T04:40:53.299 INFO:tasks.rados.rados.0.smithi085.stdout:3291: delete oid 13 current snap is 380 2021-02-10T04:40:53.299 INFO:tasks.rados.rados.0.smithi085.stdout:3289: finishing write tid 2 to smithi08575397-14 2021-02-10T04:40:53.299 INFO:tasks.rados.rados.0.smithi085.stdout:3289: finishing write tid 3 to smithi08575397-14 2021-02-10T04:40:53.299 INFO:tasks.rados.rados.0.smithi085.stdout:3289: finishing write tid 4 to smithi08575397-14 2021-02-10T04:40:53.299 INFO:tasks.rados.rados.0.smithi085.stdout:3289: finishing write tid 5 to smithi08575397-14 2021-02-10T04:40:53.299 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 1485 (ObjNum 1121 snap 380 seq_num 1121) dirty exists 2021-02-10T04:40:53.299 INFO:tasks.rados.rados.0.smithi085.stdout:3289: left oid 14 (ObjNum 1121 snap 380 seq_num 1121) 2021-02-10T04:40:53.299 INFO:tasks.rados.rados.0.smithi085.stdout:3288: finishing copy_from to smithi08575397-28 2021-02-10T04:40:53.299 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 1445 (ObjNum 1103 snap 373 seq_num 1103) dirty exists 2021-02-10T04:40:53.299 INFO:tasks.rados.rados.0.smithi085.stdout:3288: finishing copy_from racing read to smithi08575397-28 2021-02-10T04:40:53.307 INFO:tasks.rados.rados.0.smithi085.stdout:3288: done (6 left) 2021-02-10T04:40:53.307 INFO:tasks.rados.rados.0.smithi085.stdout:3289: done (5 left) 2021-02-10T04:40:53.307 INFO:tasks.rados.rados.0.smithi085.stdout:3291: done (4 left) 2021-02-10T04:40:53.307 INFO:tasks.rados.rados.0.smithi085.stdout:3292: read oid 16 snap -1 2021-02-10T04:40:53.307 INFO:tasks.rados.rados.0.smithi085.stdout:3292: expect deleted 2021-02-10T04:40:53.307 INFO:tasks.rados.rados.0.smithi085.stdout:3293: write oid 9 current snap is 380 2021-02-10T04:40:53.308 INFO:tasks.rados.rados.0.smithi085.stdout:3293: seq_num 1123 ranges {474261=408518,1582102=449729,2739315=134946} 2021-02-10T04:40:53.312 INFO:tasks.rados.rados.0.smithi085.stdout:3293: writing smithi08575397-9 from 474261 to 882779 tid 1 2021-02-10T04:40:53.320 INFO:tasks.rados.rados.0.smithi085.stdout:3293: writing smithi08575397-9 from 1582102 to 2031831 tid 2 2021-02-10T04:40:53.324 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:40:53.326 INFO:tasks.rados.rados.0.smithi085.stdout:3293: writing smithi08575397-9 from 2739315 to 2874261 tid 3 2021-02-10T04:40:53.326 INFO:tasks.rados.rados.0.smithi085.stdout:3294: write oid 42 current snap is 380 2021-02-10T04:40:53.326 INFO:tasks.rados.rados.0.smithi085.stdout:3294: seq_num 1124 ranges {491068=483767,1640584=593387,2650385=240683} 2021-02-10T04:40:53.332 INFO:tasks.rados.rados.0.smithi085.stdout:3294: writing smithi08575397-42 from 491068 to 974835 tid 1 2021-02-10T04:40:53.340 INFO:tasks.rados.rados.0.smithi085.stdout:3294: writing smithi08575397-42 from 1640584 to 2233971 tid 2 2021-02-10T04:40:53.344 INFO:tasks.rados.rados.0.smithi085.stdout:3294: writing smithi08575397-42 from 2650385 to 2891068 tid 3 2021-02-10T04:40:53.345 INFO:tasks.rados.rados.0.smithi085.stdout:3292: done (6 left) 2021-02-10T04:40:53.345 INFO:tasks.rados.rados.0.smithi085.stdout:3295: write oid 36 current snap is 380 2021-02-10T04:40:53.345 INFO:tasks.rados.rados.0.smithi085.stdout:3295: seq_num 1125 ranges {507875=559016,1699066=737045} 2021-02-10T04:40:53.352 INFO:tasks.rados.rados.0.smithi085.stdout:3295: writing smithi08575397-36 from 507875 to 1066891 tid 1 2021-02-10T04:40:53.362 INFO:tasks.rados.rados.0.smithi085.stdout:3295: writing smithi08575397-36 from 1699066 to 2436111 tid 2 2021-02-10T04:40:53.363 INFO:tasks.rados.rados.0.smithi085.stdout:3296: delete oid 49 current snap is 380 2021-02-10T04:40:53.380 INFO:tasks.rados.rados.0.smithi085.stdout:3293: finishing write tid 1 to smithi08575397-9 2021-02-10T04:40:53.382 INFO:tasks.rados.rados.0.smithi085.stdout:3293: finishing write tid 2 to smithi08575397-9 2021-02-10T04:40:53.382 INFO:tasks.rados.rados.0.smithi085.stdout:3293: finishing write tid 3 to smithi08575397-9 2021-02-10T04:40:53.383 INFO:tasks.rados.rados.0.smithi085.stdout:3293: finishing write tid 4 to smithi08575397-9 2021-02-10T04:40:53.383 INFO:tasks.rados.rados.0.smithi085.stdout:3295: finishing write tid 1 to smithi08575397-36 2021-02-10T04:40:53.383 INFO:tasks.rados.rados.0.smithi085.stdout:3295: finishing write tid 2 to smithi08575397-36 2021-02-10T04:40:53.383 INFO:tasks.rados.rados.0.smithi085.stdout:3295: finishing write tid 3 to smithi08575397-36 2021-02-10T04:40:53.383 INFO:tasks.rados.rados.0.smithi085.stdout:3295: finishing write tid 4 to smithi08575397-36 2021-02-10T04:40:53.383 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 1305 (ObjNum 1125 snap 380 seq_num 1125) dirty exists 2021-02-10T04:40:53.383 INFO:tasks.rados.rados.0.smithi085.stdout:3295: left oid 36 (ObjNum 1125 snap 380 seq_num 1125) 2021-02-10T04:40:53.386 INFO:tasks.rados.rados.0.smithi085.stdout:3293: finishing write tid 5 to smithi08575397-9 2021-02-10T04:40:53.386 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 9 v 1489 (ObjNum 1123 snap 380 seq_num 1123) dirty exists 2021-02-10T04:40:53.386 INFO:tasks.rados.rados.0.smithi085.stdout:3293: left oid 9 (ObjNum 1123 snap 380 seq_num 1123) 2021-02-10T04:40:53.386 INFO:tasks.rados.rados.0.smithi085.stdout:3294: finishing write tid 1 to smithi08575397-42 2021-02-10T04:40:53.386 INFO:tasks.rados.rados.0.smithi085.stdout:3294: finishing write tid 2 to smithi08575397-42 2021-02-10T04:40:53.386 INFO:tasks.rados.rados.0.smithi085.stdout:3294: finishing write tid 3 to smithi08575397-42 2021-02-10T04:40:53.386 INFO:tasks.rados.rados.0.smithi085.stdout:3294: finishing write tid 4 to smithi08575397-42 2021-02-10T04:40:53.387 INFO:tasks.rados.rados.0.smithi085.stdout:3294: finishing write tid 5 to smithi08575397-42 2021-02-10T04:40:53.387 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 1493 (ObjNum 1124 snap 380 seq_num 1124) dirty exists 2021-02-10T04:40:53.387 INFO:tasks.rados.rados.0.smithi085.stdout:3294: left oid 42 (ObjNum 1124 snap 380 seq_num 1124) 2021-02-10T04:40:53.390 INFO:tasks.rados.rados.0.smithi085.stdout:3293: done (7 left) 2021-02-10T04:40:53.390 INFO:tasks.rados.rados.0.smithi085.stdout:3294: done (6 left) 2021-02-10T04:40:53.390 INFO:tasks.rados.rados.0.smithi085.stdout:3295: done (5 left) 2021-02-10T04:40:53.390 INFO:tasks.rados.rados.0.smithi085.stdout:3296: done (4 left) 2021-02-10T04:40:53.391 INFO:tasks.rados.rados.0.smithi085.stdout:3297: snap_create 2021-02-10T04:40:53.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:53 smithi164 conmon[33677]: debug 2021-02-10T04:40:53.121+0000 7fbec19a1700 -1 osd.4 833 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:53.443 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[45443]: debug 2021-02-10T04:40:53.277+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:40:53.464 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:40:53.539 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:40:53.628 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:40:53.858 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[29319]: cluster 2021-02-10T04:40:53.121987+0000 osd.4 (osd.4) 1429 : 2021-02-10T04:40:53.858 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:53.858 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[29319]: cluster 2021-02-10T04:40:53.121997+0000 osd.4 (osd.4) 1430 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:53.858 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[29319]: cluster 2021-02-10T04:40:53.122003+0000 osd.4 (osd.4) 1431 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:53.858 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[29319]: cluster 2021-02-10T04:40:53.122009+0000 osd.4 (osd.4) 1432 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:53.858 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[29319]: cluster 2021-02-10T04:40:53.268470+0000 mon.a (mon.0) 1782 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 683 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:40:53.858 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[29319]: cluster 2021-02-10T04:40:53.292732+0000 mon.a (mon.0) 1783 : cluster [DBG] osdmap e834: 8 total, 8 up, 7 in 2021-02-10T04:40:53.859 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[35961]: cluster 2021-02-10T04:40:53.859 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[35961]: 2021-02-10T04:40:53.859 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[35961]: 2021 2021-02-10T04:40:53.859 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[35961]: - 2021-02-10T04:40:53.859 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[35961]: 02 2021-02-10T04:40:53.859 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[35961]: - 2021-02-10T04:40:53.859 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[35961]: 10 2021-02-10T04:40:53.859 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[35961]: T04:40:53.121987+0000 osd.4 ( 2021-02-10T04:40:53.859 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[35961]: osd.4) 1429 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:53.859 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[35961]: cluster 2021-02-10T04:40:53.121997+0000 osd.4 2021-02-10T04:40:53.859 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[35961]: (osd.4) 1430 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:53.859 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:40:53.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[35961]: -02-10T04:40:53.122003+0000 2021-02-10T04:40:53.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[35961]: osd.4 (osd. 2021-02-10T04:40:53.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[35961]: 4) 1431 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:53.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:40:53.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[35961]: -10T04:40:53.122009+0000 osd.4 2021-02-10T04:40:53.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[35961]: (osd.4) 1432 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:53.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:40:53.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[35961]: 40:53.268470+0000 mon.a (mon.0) 1782 2021-02-10T04:40:53.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[35961]: : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 683 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:40:53.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:40:53.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[35961]: -02-10T 2021-02-10T04:40:53.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[35961]: 04:40:53.292732+0000 mon.a (mon.0) 2021-02-10T04:40:53.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:53 smithi085 conmon[35961]: 1783 : cluster [DBG] osdmap e834: 8 total, 8 up, 7 in 2021-02-10T04:40:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:53 smithi164 conmon[29288]: cluster 2021-02-10T04:40:53.121987+0000 osd.4 2021-02-10T04:40:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:53 smithi164 conmon[29288]: (osd.4) 1429 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:53 smithi164 conmon[29288]: cluster 2021-02-10T04:40:53.121997+0000 osd.4 (osd.4) 1430 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:53 smithi164 conmon[29288]: cluster 2021-02-10T04:40:53.122003+0000 osd.4 (osd.4) 1431 : cluster 2021-02-10T04:40:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:53 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:53 smithi164 conmon[29288]: cluster 2021-02-10T04:40:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:53 smithi164 conmon[29288]: 2021-02-10T04:40:53. 2021-02-10T04:40:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:53 smithi164 conmon[29288]: 122009+0000 osd.4 (osd.4) 1432 : 2021-02-10T04:40:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:53 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:53 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:40:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:53 smithi164 conmon[29288]: -02-10T04:40:53 2021-02-10T04:40:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:53 smithi164 conmon[29288]: .268470+0000 mon.a (mon.0 2021-02-10T04:40:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:53 smithi164 conmon[29288]: ) 1782 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 683 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:40:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:53 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:40:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:53 smithi164 conmon[29288]: -10T04:40:53.292732+0000 2021-02-10T04:40:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:53 smithi164 conmon[29288]: mon.a (mon.0) 1783 : cluster [DBG] osdmap e834: 8 total, 8 up, 7 in 2021-02-10T04:40:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:53 smithi164 conmon[29288]: 2021-02-10T04:40:54.296 INFO:tasks.rados.rados.0.smithi085.stdout:3297: done (4 left) 2021-02-10T04:40:54.296 INFO:tasks.rados.rados.0.smithi085.stdout:3298: write (excl) oid 3 current snap is 381 2021-02-10T04:40:54.296 INFO:tasks.rados.rados.0.smithi085.stdout:3298: seq_num 1127 ranges {541489=425867,1416030=740714,2732654=208835} 2021-02-10T04:40:54.302 INFO:tasks.rados.rados.0.smithi085.stdout:3298: writing smithi08575397-3 from 541489 to 967356 tid 1 2021-02-10T04:40:54.313 INFO:tasks.rados.rados.0.smithi085.stdout:3298: writing smithi08575397-3 from 1416030 to 2156744 tid 2 2021-02-10T04:40:54.317 INFO:tasks.rados.rados.0.smithi085.stdout:3298: writing smithi08575397-3 from 2732654 to 2941489 tid 3 2021-02-10T04:40:54.317 INFO:tasks.rados.rados.0.smithi085.stdout:3299: read oid 30 snap -1 2021-02-10T04:40:54.317 INFO:tasks.rados.rados.0.smithi085.stdout:3299: expect deleted 2021-02-10T04:40:54.317 INFO:tasks.rados.rados.0.smithi085.stdout:3300: delete oid 35 current snap is 381 2021-02-10T04:40:54.325 INFO:tasks.rados.rados.0.smithi085.stdout:3299: done (6 left) 2021-02-10T04:40:54.325 INFO:tasks.rados.rados.0.smithi085.stdout:3300: done (5 left) 2021-02-10T04:40:54.325 INFO:tasks.rados.rados.0.smithi085.stdout:3301: read oid 36 snap 349 2021-02-10T04:40:54.325 INFO:tasks.rados.rados.0.smithi085.stdout:3301: expect (ObjNum 904 snap 297 seq_num 904) 2021-02-10T04:40:54.325 INFO:tasks.rados.rados.0.smithi085.stdout:3302: snap_remove snap 291 2021-02-10T04:40:54.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:54 smithi164 conmon[33677]: debug 2021-02-10T04:40:54.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:54 smithi164 conmon[33677]: 2021-02-10T04:40:54.138+0000 7fbec19a1700 -1 osd.4 834 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:40:54.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:40:54 smithi164 conmon[33677]: 2021-02-10T04:40:54.631 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:40:54.649 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:40:54.664 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:40:54.698 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:40:54.732 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:40:54.768 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:40:54.806 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:40:54.838 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:40:54.868 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:40:54.869 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:40:54.869 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:40:54.870 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:40:54.870 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:40:54.870 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:40:54.870 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:40:54.870 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:40:54.870 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:40:54.891 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: cephadm 2021-02-10T04:40:54.892 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:40:54.892 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: :40:53.851552+0000 mgr.y (mgr.14140) 802 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:40:54.892 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: cephadm 2021-02-10T04:40:54.892 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: 2021-02-10T04:40:53 2021-02-10T04:40:54.892 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: .852081+0000 mgr.y (mgr.14140) 803 : cephadm 2021-02-10T04:40:54.892 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: [INF] Refreshing smithi164 facts 2021-02-10T04:40:54.892 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: cluster 2021-02-10T04:40:54.892 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: 2021-02-10T04:40:54 2021-02-10T04:40:54.892 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: .139281+0000 osd.4 (osd.4) 1433 : 2021-02-10T04:40:54.892 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:54.892 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: cluster 2021-02-10T04:40:54.892 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:40:54.893 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: 04:40:54.139291+0000 osd.4 (osd 2021-02-10T04:40:54.893 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: .4) 1434 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:54.893 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:40:54.893 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: 02-10T04:40:54.139298+0000 osd.4 2021-02-10T04:40:54.893 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: (osd.4) 1435 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:54.893 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: 2021-02-10T04:40:54.893 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:40:54.893 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: T04:40:54.139303+0000 osd.4 2021-02-10T04:40:54.893 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: (osd.4) 1436 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:54.893 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: 2021-02-10T04:40:54.893 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: cluster 2021-02-10T04:40:54.893 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: 2021-02-10T04:40:54 2021-02-10T04:40:54.893 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: .203754+0000 mgr.y (mgr.14140) 804 : 2021-02-10T04:40:54.893 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: cluster [DBG] pgmap v1402: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim_wait, 156 active+clean, 7 active+clean+snaptrim; 565 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 12 op/s 2021-02-10T04:40:54.894 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:40:54.894 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: -02-10T04:40:54.292448 2021-02-10T04:40:54.894 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1784 : cluster [DBG] 2021-02-10T04:40:54.894 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[29319]: osdmap e835: 8 total, 8 up, 7 in 2021-02-10T04:40:54.894 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[35961]: cephadm 2021-02-10T04:40:53.851552+0000 mgr.y (mgr.14140) 802 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:40:54.894 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[35961]: cephadm 2021-02-10T04:40:53.852081+0000 mgr.y (mgr.14140) 803 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:40:54.894 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[35961]: cluster 2021-02-10T04:40:54.139281+0000 osd.4 (osd.4) 1433 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:54.894 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[35961]: cluster 2021-02-10T04:40:54.139291+0000 osd.4 (osd.4) 1434 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:54.894 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[35961]: cluster 2021-02-10T04:40:54.139298+0000 osd.4 (osd.4) 1435 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:54.894 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[35961]: cluster 2021-02-10T04:40:54.139303+0000 osd.4 (osd.4) 1436 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:54.894 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[35961]: cluster 2021-02-10T04:40:54.203754+0000 mgr.y (mgr.14140) 804 : cluster [DBG] pgmap v1402: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim_wait, 156 active+clean, 7 active+clean+snaptrim; 565 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 12 op/s 2021-02-10T04:40:54.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[35961]: cluster 2021-02-10T04:40:54.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:54 smithi085 conmon[35961]: 2021-02-10T04:40:54.292448+0000 mon.a (mon.0) 1784 : cluster [DBG] osdmap e835: 8 total, 8 up, 7 in 2021-02-10T04:40:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:54 smithi164 conmon[29288]: cephadm 2021-02-10T04:40:53.851552+0000 mgr.y ( 2021-02-10T04:40:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:54 smithi164 conmon[29288]: mgr.14140) 802 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:40:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:54 smithi164 conmon[29288]: cephadm 2021-02-10T04:40:53.852081+0000 mgr.y (mgr.14140) 803 : 2021-02-10T04:40:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:54 smithi164 conmon[29288]: cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:40:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:54 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:40:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:54 smithi164 conmon[29288]: 40:54.139281+0000 osd.4 (osd.4) 1433 : 2021-02-10T04:40:54.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:54 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:40:54.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:54 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:40:54.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:54 smithi164 conmon[29288]: 02-10T04:40:54.139291+0000 osd.4 ( 2021-02-10T04:40:54.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:54 smithi164 conmon[29288]: osd.4) 1434 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:40:54.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:54 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:40:54.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:54 smithi164 conmon[29288]: -02-10T04:40:54.139298+0000 2021-02-10T04:40:54.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:54 smithi164 conmon[29288]: osd.4 (osd.4) 1435 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:40:54.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:54 smithi164 conmon[29288]: 2021-02-10T04:40:54.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:54 smithi164 conmon[29288]: cluster 2021-02-10T04:40:54 2021-02-10T04:40:54.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:54 smithi164 conmon[29288]: .139303+0000 osd.4 (osd.4) 1436 : cluster 2021-02-10T04:40:54.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:54 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:40:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:54 smithi164 conmon[29288]: cluster 2021-02-10T04:40:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:54 smithi164 conmon[29288]: 2021-02-10T04:40:54.203754+0000 mgr.y (mgr.14140) 804 : 2021-02-10T04:40:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:54 smithi164 conmon[29288]: cluster [DBG] pgmap v1402: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim_wait, 156 active+clean, 7 active+clean+snaptrim; 565 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 12 op/s 2021-02-10T04:40:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:54 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:40:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:54 smithi164 conmon[29288]: 40:54.292448+0000 mon.a (mon.0) 2021-02-10T04:40:54.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:54 smithi164 conmon[29288]: 1784 : cluster [DBG] osdmap e835: 8 total, 8 up, 7 in 2021-02-10T04:40:55.310 INFO:tasks.rados.rados.0.smithi085.stdout:3302: done (6 left) 2021-02-10T04:40:58.158 INFO:tasks.rados.rados.0.smithi085.stdout:3303: copy_from oid 42 from oid 13 current snap is 381 2021-02-10T04:40:58.158 INFO:tasks.rados.rados.0.smithi085.stdout:3304: write (excl) oid 37 current snap is 381 2021-02-10T04:40:58.158 INFO:tasks.rados.rados.0.smithi085.stdout:3304: seq_num 1129 ranges {575103=576365,1932994=744383} 2021-02-10T04:40:58.165 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:40:58.165 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:40:55 smithi164 conmon[44544]: debug 2021-02-10T04:40:55.150+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:40:58.165 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:55 smithi164 conmon[29288]: audit 2021-02-10T04:40:54.601199+0000 mon.c (mon.1) 66 : audit [DBG] 2021-02-10T04:40:58.166 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:55 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/782076477' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:40:58.166 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:55 smithi164 conmon[29288]: audit 2021-02-10T04:40:54.910009+0000 mon.a (mon.0) 1785 : audit [INF] from='client.? 172.21.15.85:0/795330126' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:40:58.166 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:55 smithi164 conmon[29288]: audit 2021-02-10T04:40:55.135157+0000 mon.a (mon.0) 1786 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:40:58.166 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:55 smithi164 conmon[29288]: cluster 2021-02-10T04:40:58.166 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:55 smithi164 conmon[29288]: 2021-02-10T04:40:55.297338+0000 mon.a ( 2021-02-10T04:40:58.166 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:55 smithi164 conmon[29288]: mon.0) 1787 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:40:58.166 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:55 smithi164 conmon[29288]: audit 2021-02-10T04:40:55.303785+0000 2021-02-10T04:40:58.166 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:55 smithi164 conmon[29288]: mon.a (mon.0) 1788 : audit [INF] from='client.? 172.21.15.85:0/795330126' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:40:58.166 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:55 smithi164 conmon[29288]: cluster 2021-02-10T04:40:55 2021-02-10T04:40:58.166 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:55 smithi164 conmon[29288]: .303847+0000 mon.a (mon.0) 1789 : cluster [DBG] osdmap e836: 8 total, 8 up, 7 in 2021-02-10T04:40:58.166 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:56 smithi164 conmon[29288]: cluster 2021-02-10T04:40:58.166 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:56 smithi164 conmon[29288]: 2021-02-10T04:40:58.167 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:56 smithi164 conmon[29288]: 2021 2021-02-10T04:40:58.167 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:56 smithi164 conmon[29288]: - 2021-02-10T04:40:58.167 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:56 smithi164 conmon[29288]: 02 2021-02-10T04:40:58.167 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:56 smithi164 conmon[29288]: -10T04:40:56 2021-02-10T04:40:58.167 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:56 smithi164 conmon[29288]: .204294+0000 mgr.y (mgr.14140) 805 : cluster [DBG] pgmap v1405: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim_wait, 157 active+clean, 7 active+clean+snaptrim; 577 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 8.0 MiB/s rd, 5.2 MiB/s wr, 24 op/s 2021-02-10T04:40:58.167 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:57 smithi164 conmon[29288]: cluster 2021-02-10T04:40:58.167 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:57 smithi164 conmon[29288]: 2021-02-10T04:40:58.167 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:57 smithi164 conmon[29288]: 2021 2021-02-10T04:40:58.167 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:57 smithi164 conmon[29288]: - 2021-02-10T04:40:58.167 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:57 smithi164 conmon[29288]: 02 2021-02-10T04:40:58.167 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:57 smithi164 conmon[29288]: - 2021-02-10T04:40:58.167 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:57 smithi164 conmon[29288]: 10 2021-02-10T04:40:58.167 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:57 smithi164 conmon[29288]: T 2021-02-10T04:40:58.167 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:57 smithi164 conmon[29288]: 04 2021-02-10T04:40:58.168 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:57 smithi164 conmon[29288]: : 2021-02-10T04:40:58.168 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:57 smithi164 conmon[29288]: 40:56.582110+0000 mon.a (mon. 2021-02-10T04:40:58.168 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:57 smithi164 conmon[29288]: 0) 1790 : cluster [DBG] osdmap e837: 8 total, 8 up, 7 in 2021-02-10T04:40:58.168 INFO:tasks.rados.rados.0.smithi085.stdout:3304: writing smithi08575397-37 from 575103 to 1151468 tid 1 2021-02-10T04:40:58.168 INFO:tasks.rados.rados.0.smithi085.stdout:3304: writing smithi08575397-37 from 1932994 to 2677377 tid 2 2021-02-10T04:40:58.168 INFO:tasks.rados.rados.0.smithi085.stdout:3301: expect (ObjNum 904 snap 297 seq_num 904) 2021-02-10T04:40:58.168 INFO:tasks.rados.rados.0.smithi085.stdout:3298: finishing write tid 1 to smithi08575397-3 2021-02-10T04:40:58.168 INFO:tasks.rados.rados.0.smithi085.stdout:3298: finishing write tid 2 to smithi08575397-3 2021-02-10T04:40:58.168 INFO:tasks.rados.rados.0.smithi085.stdout:3298: finishing write tid 3 to smithi08575397-3 2021-02-10T04:40:58.168 INFO:tasks.rados.rados.0.smithi085.stdout:3298: finishing write tid 4 to smithi08575397-3 2021-02-10T04:40:58.168 INFO:tasks.rados.rados.0.smithi085.stdout:3298: finishing write tid 5 to smithi08575397-3 2021-02-10T04:40:58.168 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 1562 (ObjNum 1127 snap 381 seq_num 1127) dirty exists 2021-02-10T04:40:58.169 INFO:tasks.rados.rados.0.smithi085.stdout:3298: left oid 3 (ObjNum 1127 snap 381 seq_num 1127) 2021-02-10T04:40:58.169 INFO:tasks.rados.rados.0.smithi085.stdout:3304: finishing write tid 1 to smithi08575397-37 2021-02-10T04:40:58.169 INFO:tasks.rados.rados.0.smithi085.stdout:3304: finishing write tid 2 to smithi08575397-37 2021-02-10T04:40:58.169 INFO:tasks.rados.rados.0.smithi085.stdout:3304: finishing write tid 3 to smithi08575397-37 2021-02-10T04:40:58.169 INFO:tasks.rados.rados.0.smithi085.stdout:3304: finishing write tid 4 to smithi08575397-37 2021-02-10T04:40:58.169 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 1704 (ObjNum 1129 snap 381 seq_num 1129) dirty exists 2021-02-10T04:40:58.169 INFO:tasks.rados.rados.0.smithi085.stdout:3304: left oid 37 (ObjNum 1129 snap 381 seq_num 1129) 2021-02-10T04:40:58.169 INFO:tasks.rados.rados.0.smithi085.stdout:3303: finishing copy_from to smithi08575397-42 2021-02-10T04:40:58.169 INFO:tasks.rados.rados.0.smithi085.stdout:3303: got expected ENOENT (src dne) 2021-02-10T04:40:58.169 INFO:tasks.rados.rados.0.smithi085.stdout:3303: finishing copy_from racing read to smithi08575397-42 2021-02-10T04:40:58.169 INFO:tasks.rados.rados.0.smithi085.stdout:3298: done (7 left) 2021-02-10T04:40:58.169 INFO:tasks.rados.rados.0.smithi085.stdout:3301: done (6 left) 2021-02-10T04:40:58.169 INFO:tasks.rados.rados.0.smithi085.stdout:3303: done (5 left) 2021-02-10T04:40:58.170 INFO:tasks.rados.rados.0.smithi085.stdout:3304: done (4 left) 2021-02-10T04:40:58.170 INFO:tasks.rados.rados.0.smithi085.stdout:3305: rollback oid 13 current snap is 381 2021-02-10T04:40:58.170 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 13 to 337 2021-02-10T04:40:58.170 INFO:tasks.rados.rados.0.smithi085.stdout:3306: write (excl) oid 34 current snap is 381 2021-02-10T04:40:58.170 INFO:tasks.rados.rados.0.smithi085.stdout:3306: seq_num 1130 ranges {591910=651614,1875123=604394} 2021-02-10T04:40:58.170 INFO:tasks.rados.rados.0.smithi085.stdout:3306: writing smithi08575397-34 from 591910 to 1243524 tid 1 2021-02-10T04:40:58.170 INFO:tasks.rados.rados.0.smithi085.stdout:3306: writing smithi08575397-34 from 1875123 to 2479517 tid 2 2021-02-10T04:40:58.170 INFO:tasks.rados.rados.0.smithi085.stdout:3307: rollback oid 43 current snap is 381 2021-02-10T04:40:58.170 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 43 to 373 2021-02-10T04:40:58.170 INFO:tasks.rados.rados.0.smithi085.stdout:3305: finishing rollback tid 1 to smithi08575397-13 2021-02-10T04:40:58.170 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 1470 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-02-10T04:40:58.170 INFO:tasks.rados.rados.0.smithi085.stdout:3306: finishing write tid 1 to smithi08575397-34 2021-02-10T04:40:58.170 INFO:tasks.rados.rados.0.smithi085.stdout:3305: done (6 left) 2021-02-10T04:40:58.171 INFO:tasks.rados.rados.0.smithi085.stdout:3308: rollback oid 36 current snap is 381 2021-02-10T04:40:58.171 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 36 to 350 2021-02-10T04:40:58.171 INFO:tasks.rados.rados.0.smithi085.stdout:3309: rollback oid 8 current snap is 381 2021-02-10T04:40:58.171 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 8 to 346 2021-02-10T04:40:58.171 INFO:tasks.rados.rados.0.smithi085.stdout:3310: write (excl) oid 35 current snap is 381 2021-02-10T04:40:58.171 INFO:tasks.rados.rados.0.smithi085.stdout:3310: seq_num 1131 ranges {608717=726863,1933605=748052} 2021-02-10T04:40:58.171 INFO:tasks.rados.rados.0.smithi085.stdout:3310: writing smithi08575397-35 from 608717 to 1335580 tid 1 2021-02-10T04:40:58.171 INFO:tasks.rados.rados.0.smithi085.stdout:3310: writing smithi08575397-35 from 1933605 to 2681657 tid 2 2021-02-10T04:40:58.171 INFO:tasks.rados.rados.0.smithi085.stdout:3311: delete oid 18 current snap is 381 2021-02-10T04:40:58.171 INFO:tasks.rados.rados.0.smithi085.stdout:3309: finishing rollback tid 0 to smithi08575397-8 2021-02-10T04:40:58.171 INFO:tasks.rados.rados.0.smithi085.stdout:3309: finishing rollback tid 1 to smithi08575397-8 2021-02-10T04:40:58.171 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 1565 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-02-10T04:40:58.171 INFO:tasks.rados.rados.0.smithi085.stdout:3306: finishing write tid 2 to smithi08575397-34 2021-02-10T04:40:58.172 INFO:tasks.rados.rados.0.smithi085.stdout:3306: finishing write tid 3 to smithi08575397-34 2021-02-10T04:40:58.172 INFO:tasks.rados.rados.0.smithi085.stdout:3306: finishing write tid 4 to smithi08575397-34 2021-02-10T04:40:58.172 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 34 v 1473 (ObjNum 1130 snap 381 seq_num 1130) dirty exists 2021-02-10T04:40:58.172 INFO:tasks.rados.rados.0.smithi085.stdout:3306: left oid 34 (ObjNum 1130 snap 381 seq_num 1130) 2021-02-10T04:40:58.172 INFO:tasks.rados.rados.0.smithi085.stdout:3307: finishing rollback tid 0 to smithi08575397-43 2021-02-10T04:40:58.172 INFO:tasks.rados.rados.0.smithi085.stdout:3307: finishing rollback tid 1 to smithi08575397-43 2021-02-10T04:40:58.172 INFO:tasks.rados.rados.0.smithi085.stdout:3307: finishing rollback tid 2 to smithi08575397-43 2021-02-10T04:40:58.172 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 1565 (ObjNum 748 snap 240 seq_num 748) dirty exists 2021-02-10T04:40:58.172 INFO:tasks.rados.rados.0.smithi085.stdout:3308: finishing rollback tid 0 to smithi08575397-36 2021-02-10T04:40:58.172 INFO:tasks.rados.rados.0.smithi085.stdout:3308: finishing rollback tid 1 to smithi08575397-36 2021-02-10T04:40:58.172 INFO:tasks.rados.rados.0.smithi085.stdout:3308: finishing rollback tid 2 to smithi08575397-36 2021-02-10T04:40:58.172 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 1308 (ObjNum 904 snap 297 seq_num 904) dirty exists 2021-02-10T04:40:58.172 INFO:tasks.rados.rados.0.smithi085.stdout:3310: finishing write tid 1 to smithi08575397-35 2021-02-10T04:40:58.172 INFO:tasks.rados.rados.0.smithi085.stdout:3306: done (9 left) 2021-02-10T04:40:58.173 INFO:tasks.rados.rados.0.smithi085.stdout:3307: done (8 left) 2021-02-10T04:40:58.173 INFO:tasks.rados.rados.0.smithi085.stdout:3308: done (7 left) 2021-02-10T04:40:58.173 INFO:tasks.rados.rados.0.smithi085.stdout:3309: done (6 left) 2021-02-10T04:40:58.173 INFO:tasks.rados.rados.0.smithi085.stdout:3311: done (5 left) 2021-02-10T04:40:58.173 INFO:tasks.rados.rados.0.smithi085.stdout:3312: snap_remove snap 378 2021-02-10T04:40:58.173 INFO:tasks.rados.rados.0.smithi085.stdout:3310: finishing write tid 2 to smithi08575397-35 2021-02-10T04:40:58.173 INFO:tasks.rados.rados.0.smithi085.stdout:3310: finishing write tid 3 to smithi08575397-35 2021-02-10T04:40:58.173 INFO:tasks.rados.rados.0.smithi085.stdout:3310: finishing write tid 4 to smithi08575397-35 2021-02-10T04:40:58.173 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 35 v 1486 (ObjNum 1131 snap 381 seq_num 1131) dirty exists 2021-02-10T04:40:58.173 INFO:tasks.rados.rados.0.smithi085.stdout:3310: left oid 35 (ObjNum 1131 snap 381 seq_num 1131) 2021-02-10T04:40:58.173 INFO:tasks.rados.rados.0.smithi085.stdout:3310: done (5 left) 2021-02-10T04:40:58.173 INFO:tasks.rados.rados.0.smithi085.stdout:3312: done (4 left) 2021-02-10T04:40:58.173 INFO:tasks.rados.rados.0.smithi085.stdout:3313: read oid 1 snap -1 2021-02-10T04:40:58.174 INFO:tasks.rados.rados.0.smithi085.stdout:3313: expect (ObjNum 1087 snap 364 seq_num 1087) 2021-02-10T04:40:58.174 INFO:tasks.rados.rados.0.smithi085.stdout:3314: read oid 16 snap -1 2021-02-10T04:40:58.174 INFO:tasks.rados.rados.0.smithi085.stdout:3314: expect deleted 2021-02-10T04:40:58.174 INFO:tasks.rados.rados.0.smithi085.stdout:3315: copy_from oid 40 from oid 26 current snap is 381 2021-02-10T04:40:58.174 INFO:tasks.rados.rados.0.smithi085.stdout:3316: snap_remove snap 379 2021-02-10T04:40:58.174 INFO:tasks.rados.rados.0.smithi085.stdout:3316: done (7 left) 2021-02-10T04:40:58.174 INFO:tasks.rados.rados.0.smithi085.stdout:3317: read oid 14 snap -1 2021-02-10T04:40:58.174 INFO:tasks.rados.rados.0.smithi085.stdout:3317: expect (ObjNum 1121 snap 380 seq_num 1121) 2021-02-10T04:40:58.174 INFO:tasks.rados.rados.0.smithi085.stdout:3318: rollback oid 30 current snap is 381 2021-02-10T04:40:58.174 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 30 to 343 2021-02-10T04:40:58.174 INFO:tasks.rados.rados.0.smithi085.stdout:3319: delete oid 25 current snap is 381 2021-02-10T04:40:58.174 INFO:tasks.rados.rados.0.smithi085.stdout:3313: expect (ObjNum 1087 snap 364 seq_num 1087) 2021-02-10T04:40:58.174 INFO:tasks.rados.rados.0.smithi085.stdout:3315: finishing copy_from to smithi08575397-40 2021-02-10T04:40:58.175 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 1358 (ObjNum 1116 snap 378 seq_num 1116) dirty exists 2021-02-10T04:40:58.175 INFO:tasks.rados.rados.0.smithi085.stdout:3315: finishing copy_from racing read to smithi08575397-40 2021-02-10T04:40:58.175 INFO:tasks.rados.rados.0.smithi085.stdout:3318: finishing rollback tid 1 to smithi08575397-30 2021-02-10T04:40:58.175 INFO:tasks.rados.rados.0.smithi085.stdout:3318: finishing rollback tid 2 to smithi08575397-30 2021-02-10T04:40:58.175 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 1570 (ObjNum 1025 snap 335 seq_num 1025) dirty exists 2021-02-10T04:40:58.175 INFO:tasks.rados.rados.0.smithi085.stdout:3317: expect (ObjNum 1121 snap 380 seq_num 1121) 2021-02-10T04:40:58.175 INFO:tasks.rados.rados.0.smithi085.stdout:3313: done (9 left) 2021-02-10T04:40:58.175 INFO:tasks.rados.rados.0.smithi085.stdout:3314: done (8 left) 2021-02-10T04:40:58.175 INFO:tasks.rados.rados.0.smithi085.stdout:3315: done (7 left) 2021-02-10T04:40:58.175 INFO:tasks.rados.rados.0.smithi085.stdout:3317: done (6 left) 2021-02-10T04:40:58.175 INFO:tasks.rados.rados.0.smithi085.stdout:3318: done (5 left) 2021-02-10T04:40:58.175 INFO:tasks.rados.rados.0.smithi085.stdout:3319: done (4 left) 2021-02-10T04:40:58.175 INFO:tasks.rados.rados.0.smithi085.stdout:3320: write oid 21 current snap is 381 2021-02-10T04:40:58.175 INFO:tasks.rados.rados.0.smithi085.stdout:3320: seq_num 1134 ranges {659138=668963,2109051=611732} 2021-02-10T04:40:58.176 INFO:tasks.rados.rados.0.smithi085.stdout:3320: writing smithi08575397-21 from 659138 to 1328101 tid 1 2021-02-10T04:40:58.176 INFO:tasks.rados.rados.0.smithi085.stdout:3320: writing smithi08575397-21 from 2109051 to 2720783 tid 2 2021-02-10T04:40:58.176 INFO:tasks.rados.rados.0.smithi085.stdout:3321: delete oid 28 current snap is 381 2021-02-10T04:40:58.176 INFO:tasks.rados.rados.0.smithi085.stdout:3320: finishing write tid 1 to smithi08575397-21 2021-02-10T04:40:58.176 INFO:tasks.rados.rados.0.smithi085.stdout:3321: done (5 left) 2021-02-10T04:40:58.176 INFO:tasks.rados.rados.0.smithi085.stdout:3322: write (excl) oid 1 current snap is 381 2021-02-10T04:40:58.176 INFO:tasks.rados.rados.0.smithi085.stdout:3322: seq_num 1136 ranges {692752=419461,1826015=615401,2979461=113291} 2021-02-10T04:40:58.176 INFO:tasks.rados.rados.0.smithi085.stdout:3322: writing smithi08575397-1 from 692752 to 1112213 tid 1 2021-02-10T04:40:58.176 INFO:tasks.rados.rados.0.smithi085.stdout:3322: writing smithi08575397-1 from 1826015 to 2441416 tid 2 2021-02-10T04:40:58.176 INFO:tasks.rados.rados.0.smithi085.stdout:3322: writing smithi08575397-1 from 2979461 to 3092752 tid 3 2021-02-10T04:40:58.176 INFO:tasks.rados.rados.0.smithi085.stdout:3323: snap_create 2021-02-10T04:40:58.176 INFO:tasks.rados.rados.0.smithi085.stdout:3320: finishing write tid 2 to smithi08575397-21 2021-02-10T04:40:58.176 INFO:tasks.rados.rados.0.smithi085.stdout:3320: finishing write tid 3 to smithi08575397-21 2021-02-10T04:40:58.176 INFO:tasks.rados.rados.0.smithi085.stdout:3320: finishing write tid 4 to smithi08575397-21 2021-02-10T04:40:58.177 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 1714 (ObjNum 1134 snap 381 seq_num 1134) dirty exists 2021-02-10T04:40:58.177 INFO:tasks.rados.rados.0.smithi085.stdout:3320: left oid 21 (ObjNum 1134 snap 381 seq_num 1134) 2021-02-10T04:40:58.177 INFO:tasks.rados.rados.0.smithi085.stdout:3322: finishing write tid 1 to smithi08575397-1 2021-02-10T04:40:58.177 INFO:tasks.rados.rados.0.smithi085.stdout:3322: finishing write tid 2 to smithi08575397-1 2021-02-10T04:40:58.177 INFO:tasks.rados.rados.0.smithi085.stdout:3322: finishing write tid 3 to smithi08575397-1 2021-02-10T04:40:58.177 INFO:tasks.rados.rados.0.smithi085.stdout:3322: finishing write tid 4 to smithi08575397-1 2021-02-10T04:40:58.177 INFO:tasks.rados.rados.0.smithi085.stdout:3322: finishing write tid 5 to smithi08575397-1 2021-02-10T04:40:58.177 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 1569 (ObjNum 1136 snap 381 seq_num 1136) dirty exists 2021-02-10T04:40:58.177 INFO:tasks.rados.rados.0.smithi085.stdout:3322: left oid 1 (ObjNum 1136 snap 381 seq_num 1136) 2021-02-10T04:40:58.177 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is unset 2021-02-10T04:40:58.177 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:55 smithi085 conmon[29319]: audit 2021-02-10T04:40:54 2021-02-10T04:40:58.178 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:55 smithi085 conmon[29319]: .601199+0000 mon.c (mon.1) 66 : audit [DBG] from='client.? 172.21.15.85:0/782076477' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:40:58.178 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:55 smithi085 conmon[29319]: audit 2021-02-10T04:40:54.910009+0000 mon.a (mon.0) 1785 : audit [INF] from='client.? 172.21.15.85:0/795330126' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:40:58.178 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:55 smithi085 conmon[29319]: audit 2021-02-10T04:40:55.135157+0000 mon.a (mon.0) 1786 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:40:58.178 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:55 smithi085 conmon[29319]: cluster 2021-02-10T04:40:55.297338+0000 mon.a (mon.0) 1787 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:40:58.178 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:55 smithi085 conmon[29319]: audit 2021-02-10T04:40:55.303785+0000 2021-02-10T04:40:58.178 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:55 smithi085 conmon[29319]: mon.a (mon.0) 1788 : audit [INF] from='client.? 172.21.15.85:0/795330126' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:40:58.178 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:55 smithi085 conmon[29319]: cluster 2021-02-10T04:40:58.178 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:55 smithi085 conmon[29319]: 2021-02-10T04:40:55.303847+0000 mon.a (mon.0) 2021-02-10T04:40:58.178 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:55 smithi085 conmon[29319]: 1789 : cluster [DBG] osdmap e836: 8 total, 8 up, 7 in 2021-02-10T04:40:58.178 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:56 smithi085 conmon[29319]: cluster 2021-02-10T04:40:56.204294+0000 mgr.y 2021-02-10T04:40:58.178 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:56 smithi085 conmon[29319]: (mgr.14140) 805 : cluster [DBG] pgmap v1405: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim_wait, 157 active+clean, 7 active+clean+snaptrim; 577 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 8.0 MiB/s rd, 5.2 MiB/s wr, 24 op/s 2021-02-10T04:40:58.178 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:57 smithi085 conmon[29319]: cluster 2021-02-10T04:40:58.179 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:57 smithi085 conmon[29319]: 2021-02-10T04:40:58.179 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:57 smithi085 conmon[29319]: 2021-02-10T04:40:56.582110 2021-02-10T04:40:58.179 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:57 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1790 : cluster [DBG] osdmap e837: 8 total, 8 up, 7 in 2021-02-10T04:40:58.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:55 smithi085 conmon[35961]: audit 2021-02-10T04:40:54.601199 2021-02-10T04:40:58.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:55 smithi085 conmon[35961]: +0000 mon.c (mon.1) 66 : audit [DBG] from='client.? 172.21.15.85:0/782076477' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:40:58.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:55 smithi085 conmon[35961]: audit 2021-02-10T04:40:58.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:55 smithi085 conmon[35961]: 2021-02-10T04:40:54.910009+0000 mon.a (mon 2021-02-10T04:40:58.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:55 smithi085 conmon[35961]: .0) 1785 : audit [INF] from='client.? 172.21.15.85:0/795330126' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:40:58.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:55 smithi085 conmon[35961]: 2021-02-10T04:40:58.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:55 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:40:58.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:55 smithi085 conmon[35961]: 02-10T04:40:55 2021-02-10T04:40:58.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:55 smithi085 conmon[35961]: .135157+0000 mon.a (mon. 2021-02-10T04:40:58.179 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:55 smithi085 conmon[35961]: 0) 1786 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:40:58.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:55 smithi085 conmon[35961]: 2021-02-10T04:40:58.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:55 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:40:58.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:55 smithi085 conmon[35961]: 40:55.297338+0000 mon.a (mon.0) 1787 2021-02-10T04:40:58.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:55 smithi085 conmon[35961]: : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:40:58.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:55 smithi085 conmon[35961]: audit 2021-02-10T04:40 2021-02-10T04:40:58.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:55 smithi085 conmon[35961]: :55.303785+0000 mon.a (mon.0) 1788 : audit 2021-02-10T04:40:58.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:55 smithi085 conmon[35961]: [INF] from='client.? 172.21.15.85:0/795330126' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:40:58.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:55 smithi085 conmon[35961]: cluster 2021-02-10T04:40:55. 2021-02-10T04:40:58.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:55 smithi085 conmon[35961]: 303847+0000 mon.a (mon.0) 1789 : cluster [DBG] 2021-02-10T04:40:58.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:55 smithi085 conmon[35961]: osdmap e836: 8 total, 8 up, 7 in 2021-02-10T04:40:58.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:56 smithi085 conmon[35961]: cluster 2021-02-10T04:40:56. 2021-02-10T04:40:58.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:56 smithi085 conmon[35961]: 204294+0000 mgr.y (mgr.14140) 805 : cluster [DBG] pgmap v1405: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim_wait, 157 active+clean, 7 active+clean+snaptrim; 577 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 8.0 MiB/s rd, 5.2 MiB/s wr, 24 op/s 2021-02-10T04:40:58.180 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:57 smithi085 conmon[35961]: cluster 2021-02-10T04:40:56 2021-02-10T04:40:58.181 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:57 smithi085 conmon[35961]: .582110+0000 mon.a (mon.0) 1790 : cluster [DBG] osdmap e837: 8 total, 8 up, 7 in 2021-02-10T04:40:58.267 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:40:58.306 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:40:58.606 INFO:tasks.rados.rados.0.smithi085.stdout:3320: done (6 left) 2021-02-10T04:40:58.606 INFO:tasks.rados.rados.0.smithi085.stdout:3322: done (5 left) 2021-02-10T04:40:58.606 INFO:tasks.rados.rados.0.smithi085.stdout:3323: done (4 left) 2021-02-10T04:40:58.607 INFO:tasks.rados.rados.0.smithi085.stdout:3324: delete oid 43 current snap is 382 2021-02-10T04:40:58.618 INFO:tasks.rados.rados.0.smithi085.stdout:3324: done (4 left) 2021-02-10T04:40:58.618 INFO:tasks.rados.rados.0.smithi085.stdout:3325: write (excl) oid 23 current snap is 382 2021-02-10T04:40:58.618 INFO:tasks.rados.rados.0.smithi085.stdout:3325: seq_num 1138 ranges {726366=569959,1826626=502717,2801601=324765} 2021-02-10T04:40:58.623 INFO:tasks.rados.rados.0.smithi085.stdout:3325: writing smithi08575397-23 from 726366 to 1296325 tid 1 2021-02-10T04:40:58.630 INFO:tasks.rados.rados.0.smithi085.stdout:3325: writing smithi08575397-23 from 1826626 to 2329343 tid 2 2021-02-10T04:40:58.635 INFO:tasks.rados.rados.0.smithi085.stdout:3325: writing smithi08575397-23 from 2801601 to 3126366 tid 3 2021-02-10T04:40:58.635 INFO:tasks.rados.rados.0.smithi085.stdout:3326: snap_remove snap 350 2021-02-10T04:40:58.894 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:40:58 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:40:58.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:58 smithi164 conmon[29288]: cluster 2021-02-10T04:40:58.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:58 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:40:58.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:58 smithi164 conmon[29288]: 40:57.585111+0000 mon.a (mon.0) 1791 : cluster [DBG] osdmap e838: 8 total, 8 up, 7 in 2021-02-10T04:40:58.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:58 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:40:58.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:58 smithi164 conmon[29288]: 40:58.204860+0000 mgr.y (mgr. 2021-02-10T04:40:58.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:58 smithi164 conmon[29288]: 14140) 806 : cluster [DBG] pgmap v1408: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim_wait, 155 active+clean, 8 active+clean+snaptrim; 581 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 8.5 MiB/s rd, 4.1 MiB/s wr, 19 op/s 2021-02-10T04:40:58.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:58 smithi085 conmon[35961]: cluster 2021-02-10T04:40:58.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:58 smithi085 conmon[35961]: 2021-02-10T04:40:58.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:58 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:40:58.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:58 smithi085 conmon[35961]: :40:57.585111+0000 mon.a (mon.0) 1791 : cluster [DBG] osdmap e838: 8 total, 8 up, 7 in 2021-02-10T04:40:58.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:58 smithi085 conmon[35961]: cluster 2021-02-10T04:40:58.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:58 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:40:58.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:58 smithi085 conmon[35961]: 04:40:58.204860+0000 mgr.y (mgr.14140) 806 2021-02-10T04:40:58.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:58 smithi085 conmon[35961]: : cluster [DBG] pgmap v1408: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim_wait, 155 active+clean, 8 active+clean+snaptrim; 581 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 8.5 MiB/s rd, 4.1 MiB/s wr, 19 op/s 2021-02-10T04:40:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:58 smithi085 conmon[29319]: cluster 2021-02-10T04:40:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:58 smithi085 conmon[29319]: 2021-02-10T04:40:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:58 smithi085 conmon[29319]: 2021 2021-02-10T04:40:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:58 smithi085 conmon[29319]: - 2021-02-10T04:40:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:58 smithi085 conmon[29319]: 02-10T04:40:57. 2021-02-10T04:40:58.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:58 smithi085 conmon[29319]: 585111+0000 mon.a (mon.0) 1791 : cluster [DBG] osdmap e838: 8 total, 8 up, 7 in 2021-02-10T04:40:58.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:58 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:40:58.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:58 smithi085 conmon[29319]: 04:40:58.204860+0000 mgr.y (mgr.14140) 806 : cluster [DBG] pgmap v1408: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim_wait, 155 active+clean, 8 active+clean+snaptrim; 581 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 8.5 MiB/s rd, 4.1 MiB/s wr, 19 op/s 2021-02-10T04:40:59.607 INFO:tasks.rados.rados.0.smithi085.stdout:3325: finishing write tid 1 to smithi08575397-23 2021-02-10T04:40:59.607 INFO:tasks.rados.rados.0.smithi085.stdout:3325: finishing write tid 2 to smithi08575397-23 2021-02-10T04:40:59.607 INFO:tasks.rados.rados.0.smithi085.stdout:3325: finishing write tid 3 to smithi08575397-23 2021-02-10T04:40:59.608 INFO:tasks.rados.rados.0.smithi085.stdout:3325: finishing write tid 4 to smithi08575397-23 2021-02-10T04:40:59.608 INFO:tasks.rados.rados.0.smithi085.stdout:3325: finishing write tid 5 to smithi08575397-23 2021-02-10T04:40:59.608 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 1312 (ObjNum 1138 snap 382 seq_num 1138) dirty exists 2021-02-10T04:40:59.608 INFO:tasks.rados.rados.0.smithi085.stdout:3325: left oid 23 (ObjNum 1138 snap 382 seq_num 1138) 2021-02-10T04:40:59.608 INFO:tasks.rados.rados.0.smithi085.stdout:3325: done (5 left) 2021-02-10T04:40:59.608 INFO:tasks.rados.rados.0.smithi085.stdout:3326: done (4 left) 2021-02-10T04:40:59.608 INFO:tasks.rados.rados.0.smithi085.stdout:3327: read oid 50 snap -1 2021-02-10T04:40:59.608 INFO:tasks.rados.rados.0.smithi085.stdout:3327: expect (ObjNum 1054 snap 346 seq_num 1054) 2021-02-10T04:40:59.608 INFO:tasks.rados.rados.0.smithi085.stdout:3328: read oid 29 snap -1 2021-02-10T04:40:59.608 INFO:tasks.rados.rados.0.smithi085.stdout:3328: expect (ObjNum 1091 snap 364 seq_num 1091) 2021-02-10T04:40:59.608 INFO:tasks.rados.rados.0.smithi085.stdout:3329: copy_from oid 44 from oid 3 current snap is 382 2021-02-10T04:40:59.609 INFO:tasks.rados.rados.0.smithi085.stdout:3330: write (excl) oid 14 current snap is 382 2021-02-10T04:40:59.609 INFO:tasks.rados.rados.0.smithi085.stdout:3330: seq_num 1139 ranges {743173=645208,1885108=762728} 2021-02-10T04:40:59.615 INFO:tasks.rados.rados.0.smithi085.stdout:3330: writing smithi08575397-14 from 743173 to 1388381 tid 1 2021-02-10T04:40:59.624 INFO:tasks.rados.rados.0.smithi085.stdout:3330: writing smithi08575397-14 from 1885108 to 2647836 tid 2 2021-02-10T04:40:59.624 INFO:tasks.rados.rados.0.smithi085.stdout:3331: snap_create 2021-02-10T04:40:59.626 INFO:tasks.rados.rados.0.smithi085.stdout:3328: expect (ObjNum 1091 snap 364 seq_num 1091) 2021-02-10T04:40:59.783 INFO:tasks.rados.rados.0.smithi085.stdout:3327: expect (ObjNum 1054 snap 346 seq_num 1054) 2021-02-10T04:40:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:59 smithi164 conmon[29288]: audit 2021-02-10T04:40:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:59 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:40:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:59 smithi164 conmon[29288]: 04:40:58.427085+0000 mgr.y (mgr.14140) 807 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:40:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:59 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:40:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:59 smithi164 conmon[29288]: 02-10T04:40:58.589104+0000 mon.a (mon. 2021-02-10T04:40:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:59 smithi164 conmon[29288]: 0) 1792 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 683 sec, osd.4 has slow ops) 2021-02-10T04:40:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:59 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:40:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:59 smithi164 conmon[29288]: 02-10T04:40:58.606452+0000 mon.a 2021-02-10T04:40:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:40:59 smithi164 conmon[29288]: (mon.0) 1793 : cluster [DBG] osdmap e839: 8 total, 8 up, 7 in 2021-02-10T04:40:59.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:59 smithi085 conmon[29319]: audit 2021-02-10T04:40:58.427085+0000 mgr.y (mgr.14140) 807 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:40:59.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:59 smithi085 conmon[29319]: cluster 2021-02-10T04:40:58.589104+0000 mon.a (mon.0) 1792 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 683 sec, osd.4 has slow ops) 2021-02-10T04:40:59.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:59 smithi085 conmon[29319]: cluster 2021-02-10T04:40:58 2021-02-10T04:40:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:40:59 smithi085 conmon[29319]: .606452+0000 mon.a (mon.0) 1793 : cluster [DBG] osdmap e839: 8 total, 8 up, 7 in 2021-02-10T04:40:59.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:59 smithi085 conmon[35961]: audit 2021-02-10T04:40:58. 2021-02-10T04:40:59.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:59 smithi085 conmon[35961]: 427085+0000 mgr.y (mgr.14140) 807 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:40:59.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:59 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:40:59.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:59 smithi085 conmon[35961]: -10T04:40:58.589104+0000 mon.a (mon.0) 1792 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 683 sec, osd.4 has slow ops) 2021-02-10T04:40:59.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:59 smithi085 conmon[35961]: 2021-02-10T04:40:59.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:59 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:40:59.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:59 smithi085 conmon[35961]: 04:40:58.606452+0000 mon.a (mon. 2021-02-10T04:40:59.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:59 smithi085 conmon[35961]: 0) 1793 2021-02-10T04:40:59.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:40:59 smithi085 conmon[35961]: : cluster [DBG] osdmap e839: 8 total, 8 up, 7 in 2021-02-10T04:41:00.036 INFO:tasks.rados.rados.0.smithi085.stdout:3330: finishing write tid 1 to smithi08575397-14 2021-02-10T04:41:00.036 INFO:tasks.rados.rados.0.smithi085.stdout:3330: finishing write tid 2 to smithi08575397-14 2021-02-10T04:41:00.036 INFO:tasks.rados.rados.0.smithi085.stdout:3330: finishing write tid 3 to smithi08575397-14 2021-02-10T04:41:00.036 INFO:tasks.rados.rados.0.smithi085.stdout:3330: finishing write tid 4 to smithi08575397-14 2021-02-10T04:41:00.036 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 1496 (ObjNum 1139 snap 382 seq_num 1139) dirty exists 2021-02-10T04:41:00.036 INFO:tasks.rados.rados.0.smithi085.stdout:3330: left oid 14 (ObjNum 1139 snap 382 seq_num 1139) 2021-02-10T04:41:00.036 INFO:tasks.rados.rados.0.smithi085.stdout:3329: finishing copy_from to smithi08575397-44 2021-02-10T04:41:00.036 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 1562 (ObjNum 1127 snap 381 seq_num 1127) dirty exists 2021-02-10T04:41:00.036 INFO:tasks.rados.rados.0.smithi085.stdout:3329: finishing copy_from racing read to smithi08575397-44 2021-02-10T04:41:00.613 INFO:tasks.rados.rados.0.smithi085.stdout:3327: done (8 left) 2021-02-10T04:41:00.614 INFO:tasks.rados.rados.0.smithi085.stdout:3328: done (7 left) 2021-02-10T04:41:00.614 INFO:tasks.rados.rados.0.smithi085.stdout:3329: done (6 left) 2021-02-10T04:41:00.614 INFO:tasks.rados.rados.0.smithi085.stdout:3330: done (5 left) 2021-02-10T04:41:00.614 INFO:tasks.rados.rados.0.smithi085.stdout:3331: done (4 left) 2021-02-10T04:41:00.614 INFO:tasks.rados.rados.0.smithi085.stdout:3332: copy_from oid 6 from oid 23 current snap is 383 2021-02-10T04:41:00.614 INFO:tasks.rados.rados.0.smithi085.stdout:3333: delete oid 38 current snap is 383 2021-02-10T04:41:00.620 INFO:tasks.rados.rados.0.smithi085.stdout:3333: done (5 left) 2021-02-10T04:41:00.620 INFO:tasks.rados.rados.0.smithi085.stdout:3334: read oid 32 snap 369 2021-02-10T04:41:00.620 INFO:tasks.rados.rados.0.smithi085.stdout:3334: expect (ObjNum 1082 snap 359 seq_num 1082) 2021-02-10T04:41:00.620 INFO:tasks.rados.rados.0.smithi085.stdout:3335: rollback oid 44 current snap is 383 2021-02-10T04:41:00.620 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 44 to 349 2021-02-10T04:41:00.620 INFO:tasks.rados.rados.0.smithi085.stdout:3336: delete oid 35 current snap is 383 2021-02-10T04:41:00.626 INFO:tasks.rados.rados.0.smithi085.stdout:3335: finishing rollback tid 0 to smithi08575397-44 2021-02-10T04:41:00.628 INFO:tasks.rados.rados.0.smithi085.stdout:3335: finishing rollback tid 1 to smithi08575397-44 2021-02-10T04:41:00.628 INFO:tasks.rados.rados.0.smithi085.stdout:3334: expect (ObjNum 1082 snap 359 seq_num 1082) 2021-02-10T04:41:00.755 INFO:tasks.rados.rados.0.smithi085.stdout:3334: done (7 left) 2021-02-10T04:41:00.755 INFO:tasks.rados.rados.0.smithi085.stdout:3336: done (6 left) 2021-02-10T04:41:00.755 INFO:tasks.rados.rados.0.smithi085.stdout:3337: snap_create 2021-02-10T04:41:00.755 INFO:tasks.rados.rados.0.smithi085.stdout:3335: finishing rollback tid 2 to smithi08575397-44 2021-02-10T04:41:00.756 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 1565 (ObjNum 1052 snap 345 seq_num 1052) dirty exists 2021-02-10T04:41:00.756 INFO:tasks.rados.rados.0.smithi085.stdout:3332: finishing copy_from to smithi08575397-6 2021-02-10T04:41:00.756 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 1312 (ObjNum 1138 snap 382 seq_num 1138) dirty exists 2021-02-10T04:41:00.756 INFO:tasks.rados.rados.0.smithi085.stdout:3332: finishing copy_from racing read to smithi08575397-6 2021-02-10T04:41:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:00 smithi164 conmon[29288]: cluster 2021-02-10T04:41:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:00 smithi164 conmon[29288]: 2021-02-10T04:41:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:00 smithi164 conmon[29288]: 2021 2021-02-10T04:41:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:00 smithi164 conmon[29288]: - 2021-02-10T04:41:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:00 smithi164 conmon[29288]: 02 2021-02-10T04:41:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:00 smithi164 conmon[29288]: - 2021-02-10T04:41:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:00 smithi164 conmon[29288]: 10T04:40:59.605723+0000 mon.a 2021-02-10T04:41:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:00 smithi164 conmon[29288]: (mon.0) 1794 : cluster [DBG] osdmap e840: 8 total, 8 up, 7 in 2021-02-10T04:41:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:00 smithi164 conmon[29288]: cluster 2021-02-10T04:41:00 2021-02-10T04:41:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:00 smithi164 conmon[29288]: .205609+0000 mgr.y (mgr.14140) 808 : cluster [DBG] pgmap v1411: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 584 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 6.0 MiB/s rd, 3.8 MiB/s wr, 22 op/s 2021-02-10T04:41:00.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[35961]: cluster 2021-02-10T04:40:59.605723+0000 mon.a 2021-02-10T04:41:00.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[35961]: (mon.0) 1794 : cluster [DBG] osdmap e840: 8 total, 8 up, 7 in 2021-02-10T04:41:00.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[35961]: cluster 2021-02-10T04:41:00.205609+0000 mgr.y 2021-02-10T04:41:00.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[35961]: (mgr.14140) 808 : cluster [DBG] pgmap v1411: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 584 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 6.0 MiB/s rd, 3.8 MiB/s wr, 22 op/s 2021-02-10T04:41:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: cluster 2021-02-10T04:41:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: 2021-02-10T04:41:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: 2021 2021-02-10T04:41:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: - 2021-02-10T04:41:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: 02 2021-02-10T04:41:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: - 2021-02-10T04:41:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: 10 2021-02-10T04:41:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: T 2021-02-10T04:41:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: 04 2021-02-10T04:41:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: : 2021-02-10T04:41:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: 40 2021-02-10T04:41:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: : 2021-02-10T04:41:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: 59 2021-02-10T04:41:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: . 2021-02-10T04:41:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: 605723 2021-02-10T04:41:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: +0000 2021-02-10T04:41:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: 2021-02-10T04:41:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: mon.a 2021-02-10T04:41:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: ( 2021-02-10T04:41:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: mon 2021-02-10T04:41:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: . 2021-02-10T04:41:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: 0 2021-02-10T04:41:00.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: ) 2021-02-10T04:41:00.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: 1794 2021-02-10T04:41:00.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: : 2021-02-10T04:41:00.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: cluster 2021-02-10T04:41:00.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: 2021-02-10T04:41:00.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: [DBG] 2021-02-10T04:41:00.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: 2021-02-10T04:41:00.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: osdmap e840: 8 total, 8 up, 7 in 2021-02-10T04:41:00.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: 2021-02-10T04:41:00.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: cluster 2021-02-10T04:41:00.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:41:00.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:00 smithi085 conmon[29319]: 10T04:41:00.205609+0000 mgr.y (mgr.14140) 808 : cluster [DBG] pgmap v1411: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 584 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 6.0 MiB/s rd, 3.8 MiB/s wr, 22 op/s 2021-02-10T04:41:01.624 INFO:tasks.rados.rados.0.smithi085.stdout:3332: done (6 left) 2021-02-10T04:41:01.625 INFO:tasks.rados.rados.0.smithi085.stdout:3335: done (5 left) 2021-02-10T04:41:01.625 INFO:tasks.rados.rados.0.smithi085.stdout:3337: done (4 left) 2021-02-10T04:41:01.625 INFO:tasks.rados.rados.0.smithi085.stdout:3338: snap_create 2021-02-10T04:41:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:01 smithi164 conmon[29288]: cluster 2021-02-10T04:41:00.613193+0000 mon.a (mon.0) 1795 : cluster 2021-02-10T04:41:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:01 smithi164 conmon[29288]: [DBG] osdmap e841: 8 total, 8 up, 7 in 2021-02-10T04:41:01.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:01 smithi085 conmon[35961]: cluster 2021-02-10T04:41:00.613193+0000 2021-02-10T04:41:01.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:01 smithi085 conmon[35961]: mon.a (mon.0) 1795 : cluster [DBG] osdmap e841: 8 total, 8 up, 7 in 2021-02-10T04:41:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:01 smithi085 conmon[29319]: cluster 2021-02-10T04:41:00.613193+0000 mon.a (mon.0) 1795 : cluster [DBG] osdmap e841: 8 total, 8 up, 7 in 2021-02-10T04:41:02.624 INFO:tasks.rados.rados.0.smithi085.stdout:3338: done (4 left) 2021-02-10T04:41:02.624 INFO:tasks.rados.rados.0.smithi085.stdout:3339: write oid 48 current snap is 385 2021-02-10T04:41:02.624 INFO:tasks.rados.rados.0.smithi085.stdout:3339: seq_num 1142 ranges {793594=587308,2060554=510055} 2021-02-10T04:41:02.632 INFO:tasks.rados.rados.0.smithi085.stdout:3339: writing smithi08575397-48 from 793594 to 1380902 tid 1 2021-02-10T04:41:02.639 INFO:tasks.rados.rados.0.smithi085.stdout:3339: writing smithi08575397-48 from 2060554 to 2570609 tid 2 2021-02-10T04:41:02.640 INFO:tasks.rados.rados.0.smithi085.stdout:3340: write (excl) oid 40 current snap is 385 2021-02-10T04:41:02.640 INFO:tasks.rados.rados.0.smithi085.stdout:3340: seq_num 1143 ranges {410401=662557,1719036=770066,2938716=271685} 2021-02-10T04:41:02.646 INFO:tasks.rados.rados.0.smithi085.stdout:3340: writing smithi08575397-40 from 410401 to 1072958 tid 1 2021-02-10T04:41:02.654 INFO:tasks.rados.rados.0.smithi085.stdout:3340: writing smithi08575397-40 from 1719036 to 2489102 tid 2 2021-02-10T04:41:02.659 INFO:tasks.rados.rados.0.smithi085.stdout:3340: writing smithi08575397-40 from 2938716 to 3210401 tid 3 2021-02-10T04:41:02.659 INFO:tasks.rados.rados.0.smithi085.stdout:3341: write (excl) oid 28 current snap is 385 2021-02-10T04:41:02.659 INFO:tasks.rados.rados.0.smithi085.stdout:3339: finishing write tid 1 to smithi08575397-48 2021-02-10T04:41:02.659 INFO:tasks.rados.rados.0.smithi085.stdout:3341: seq_num 1144 ranges {427208=737806,1777518=513724,2966139=261069} 2021-02-10T04:41:02.666 INFO:tasks.rados.rados.0.smithi085.stdout:3341: writing smithi08575397-28 from 427208 to 1165014 tid 1 2021-02-10T04:41:02.673 INFO:tasks.rados.rados.0.smithi085.stdout:3341: writing smithi08575397-28 from 1777518 to 2291242 tid 2 2021-02-10T04:41:02.679 INFO:tasks.rados.rados.0.smithi085.stdout:3341: writing smithi08575397-28 from 2966139 to 3227208 tid 3 2021-02-10T04:41:02.679 INFO:tasks.rados.rados.0.smithi085.stdout:3342: write (excl) oid 34 current snap is 385 2021-02-10T04:41:02.679 INFO:tasks.rados.rados.0.smithi085.stdout:3339: finishing write tid 2 to smithi08575397-48 2021-02-10T04:41:02.679 INFO:tasks.rados.rados.0.smithi085.stdout:3339: finishing write tid 3 to smithi08575397-48 2021-02-10T04:41:02.679 INFO:tasks.rados.rados.0.smithi085.stdout:3339: finishing write tid 4 to smithi08575397-48 2021-02-10T04:41:02.679 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 1499 (ObjNum 1142 snap 385 seq_num 1142) dirty exists 2021-02-10T04:41:02.679 INFO:tasks.rados.rados.0.smithi085.stdout:3339: left oid 48 (ObjNum 1142 snap 385 seq_num 1142) 2021-02-10T04:41:02.679 INFO:tasks.rados.rados.0.smithi085.stdout:3340: finishing write tid 1 to smithi08575397-40 2021-02-10T04:41:02.679 INFO:tasks.rados.rados.0.smithi085.stdout:3342: seq_num 1145 ranges {444015=413055,1436000=773735,2709915=534100} 2021-02-10T04:41:02.683 INFO:tasks.rados.rados.0.smithi085.stdout:3342: writing smithi08575397-34 from 444015 to 857070 tid 1 2021-02-10T04:41:02.692 INFO:tasks.rados.rados.0.smithi085.stdout:3342: writing smithi08575397-34 from 1436000 to 2209735 tid 2 2021-02-10T04:41:02.699 INFO:tasks.rados.rados.0.smithi085.stdout:3342: writing smithi08575397-34 from 2709915 to 3244015 tid 3 2021-02-10T04:41:02.699 INFO:tasks.rados.rados.0.smithi085.stdout:3339: done (7 left) 2021-02-10T04:41:02.699 INFO:tasks.rados.rados.0.smithi085.stdout:3343: read oid 48 snap -1 2021-02-10T04:41:02.699 INFO:tasks.rados.rados.0.smithi085.stdout:3343: expect (ObjNum 1142 snap 385 seq_num 1142) 2021-02-10T04:41:02.699 INFO:tasks.rados.rados.0.smithi085.stdout:3344: snap_create 2021-02-10T04:41:02.700 INFO:tasks.rados.rados.0.smithi085.stdout:3340: finishing write tid 2 to smithi08575397-40 2021-02-10T04:41:02.700 INFO:tasks.rados.rados.0.smithi085.stdout:3340: finishing write tid 3 to smithi08575397-40 2021-02-10T04:41:02.700 INFO:tasks.rados.rados.0.smithi085.stdout:3340: finishing write tid 4 to smithi08575397-40 2021-02-10T04:41:02.700 INFO:tasks.rados.rados.0.smithi085.stdout:3340: finishing write tid 5 to smithi08575397-40 2021-02-10T04:41:02.700 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 1725 (ObjNum 1143 snap 385 seq_num 1143) dirty exists 2021-02-10T04:41:02.700 INFO:tasks.rados.rados.0.smithi085.stdout:3340: left oid 40 (ObjNum 1143 snap 385 seq_num 1143) 2021-02-10T04:41:02.700 INFO:tasks.rados.rados.0.smithi085.stdout:3341: finishing write tid 1 to smithi08575397-28 2021-02-10T04:41:02.700 INFO:tasks.rados.rados.0.smithi085.stdout:3341: finishing write tid 2 to smithi08575397-28 2021-02-10T04:41:02.700 INFO:tasks.rados.rados.0.smithi085.stdout:3341: finishing write tid 3 to smithi08575397-28 2021-02-10T04:41:02.700 INFO:tasks.rados.rados.0.smithi085.stdout:3341: finishing write tid 4 to smithi08575397-28 2021-02-10T04:41:02.700 INFO:tasks.rados.rados.0.smithi085.stdout:3341: finishing write tid 5 to smithi08575397-28 2021-02-10T04:41:02.700 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 1450 (ObjNum 1144 snap 385 seq_num 1144) dirty exists 2021-02-10T04:41:02.700 INFO:tasks.rados.rados.0.smithi085.stdout:3341: left oid 28 (ObjNum 1144 snap 385 seq_num 1144) 2021-02-10T04:41:02.701 INFO:tasks.rados.rados.0.smithi085.stdout:3342: finishing write tid 1 to smithi08575397-34 2021-02-10T04:41:02.701 INFO:tasks.rados.rados.0.smithi085.stdout:3342: finishing write tid 2 to smithi08575397-34 2021-02-10T04:41:02.706 INFO:tasks.rados.rados.0.smithi085.stdout:3342: finishing write tid 3 to smithi08575397-34 2021-02-10T04:41:02.708 INFO:tasks.rados.rados.0.smithi085.stdout:3343: expect (ObjNum 1142 snap 385 seq_num 1142) 2021-02-10T04:41:02.893 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:41:02 smithi164 conmon[37263]: debug 2021-02-10T04:41:02.814+0000 7f1c671e2700 -1 osd.5 pg_epoch: 843 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x556650668160) 2021-02-10T04:41:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:02 smithi164 conmon[29288]: cluster 2021-02-10T04:41:01.623659+0000 mon.a ( 2021-02-10T04:41:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:02 smithi164 conmon[29288]: mon.0) 1796 : cluster [DBG] osdmap e842: 8 total, 8 up, 7 in 2021-02-10T04:41:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:02 smithi164 conmon[29288]: cluster 2021-02-10T04:41:02 2021-02-10T04:41:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:02 smithi164 conmon[29288]: .206223+0000 mgr.y (mgr.14140) 809 : cluster [DBG] pgmap v1414: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 584 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 14 op/s 2021-02-10T04:41:02.908 INFO:tasks.rados.rados.0.smithi085.stdout:3342: finishing write tid 4 to smithi08575397-34 2021-02-10T04:41:02.908 INFO:tasks.rados.rados.0.smithi085.stdout:3342: finishing write tid 5 to smithi08575397-34 2021-02-10T04:41:02.909 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 34 v 1477 (ObjNum 1145 snap 385 seq_num 1145) dirty exists 2021-02-10T04:41:02.909 INFO:tasks.rados.rados.0.smithi085.stdout:3342: left oid 34 (ObjNum 1145 snap 385 seq_num 1145) 2021-02-10T04:41:02.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:02 smithi085 conmon[29319]: cluster 2021-02-10T04:41:01.623659+0000 2021-02-10T04:41:02.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:02 smithi085 conmon[29319]: mon.a (mon.0) 1796 : cluster [DBG] osdmap e842: 8 total, 8 up, 7 in 2021-02-10T04:41:02.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:02 smithi085 conmon[29319]: cluster 2021-02-10T04:41:02.206223+0000 mgr.y (mgr.14140) 809 : cluster [DBG] pgmap v1414: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 584 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 14 op/s 2021-02-10T04:41:02.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:02 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:41:02.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:02 smithi085 conmon[35961]: T04:41:01.623659+0000 mon.a (mon.0) 1796 : cluster [DBG] osdmap e842: 8 total, 8 up, 7 in 2021-02-10T04:41:02.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:02 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:41:02.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:02 smithi085 conmon[35961]: 02-10T04:41:02.206223+0000 mgr.y 2021-02-10T04:41:02.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:02 smithi085 conmon[35961]: (mgr.14140) 809 : cluster [DBG] pgmap v1414: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 584 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 14 op/s 2021-02-10T04:41:03.649 INFO:tasks.rados.rados.0.smithi085.stdout:3340: done (8 left) 2021-02-10T04:41:03.649 INFO:tasks.rados.rados.0.smithi085.stdout:3341: done (7 left) 2021-02-10T04:41:03.649 INFO:tasks.rados.rados.0.smithi085.stdout:3342: done (6 left) 2021-02-10T04:41:03.650 INFO:tasks.rados.rados.0.smithi085.stdout:3343: done (5 left) 2021-02-10T04:41:03.650 INFO:tasks.rados.rados.0.smithi085.stdout:3344: done (4 left) 2021-02-10T04:41:03.650 INFO:tasks.rados.rados.0.smithi085.stdout:3345: rollback oid 36 current snap is 386 2021-02-10T04:41:03.650 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 36 to 370 2021-02-10T04:41:03.650 INFO:tasks.rados.rados.0.smithi085.stdout:3346: copy_from oid 26 from oid 43 current snap is 386 2021-02-10T04:41:03.650 INFO:tasks.rados.rados.0.smithi085.stdout:3347: snap_create 2021-02-10T04:41:03.658 INFO:tasks.rados.rados.0.smithi085.stdout:3345: finishing rollback tid 0 to smithi08575397-36 2021-02-10T04:41:03.658 INFO:tasks.rados.rados.0.smithi085.stdout:3345: finishing rollback tid 1 to smithi08575397-36 2021-02-10T04:41:03.659 INFO:tasks.rados.rados.0.smithi085.stdout:3345: finishing rollback tid 2 to smithi08575397-36 2021-02-10T04:41:03.659 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 1315 (ObjNum 991 snap 326 seq_num 991) dirty exists 2021-02-10T04:41:03.799 INFO:tasks.rados.rados.0.smithi085.stdout:3346: finishing copy_from to smithi08575397-26 2021-02-10T04:41:03.799 INFO:tasks.rados.rados.0.smithi085.stdout:3346: got expected ENOENT (src dne) 2021-02-10T04:41:03.799 INFO:tasks.rados.rados.0.smithi085.stdout:3346: finishing copy_from racing read to smithi08575397-26 2021-02-10T04:41:03.894 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:41:03 smithi164 conmon[37263]: debug 2021-02-10T04:41:03.810+0000 7f1c671e2700 -1 osd.5 pg_epoch: 844 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x5566534ba9a0) 2021-02-10T04:41:03.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:03 smithi164 conmon[29288]: cluster 2021-02-10T04:41:02. 2021-02-10T04:41:03.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:03 smithi164 conmon[29288]: 622179+0000 mon.a (mon.0) 1797 : cluster [DBG] osdmap e843: 8 total, 8 up, 7 in 2021-02-10T04:41:03.941 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:41:03 smithi085 conmon[52836]: debug 2021-02-10T04:41:03.861+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:41:03.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:03 smithi085 conmon[29319]: cluster 2021-02-10T04:41:02.622179+0000 2021-02-10T04:41:03.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:03 smithi085 conmon[29319]: mon.a (mon.0) 1797 : cluster [DBG] osdmap e843: 8 total, 8 up, 7 in 2021-02-10T04:41:03.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:03 smithi085 conmon[35961]: cluster 2021-02-10T04:41:02.622179+0000 2021-02-10T04:41:03.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:03 smithi085 conmon[35961]: mon.a (mon.0) 1797 : cluster [DBG] osdmap e843: 8 total, 8 up, 7 in 2021-02-10T04:41:04.121 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:41:04.652 INFO:tasks.rados.rados.0.smithi085.stdout:3345: done (6 left) 2021-02-10T04:41:04.652 INFO:tasks.rados.rados.0.smithi085.stdout:3346: done (5 left) 2021-02-10T04:41:04.652 INFO:tasks.rados.rados.0.smithi085.stdout:3347: done (4 left) 2021-02-10T04:41:04.653 INFO:tasks.rados.rados.0.smithi085.stdout:3348: snap_create 2021-02-10T04:41:04.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:04 smithi085 conmon[35961]: cluster 2021-02-10T04:41:03.649374+0000 mon.a 2021-02-10T04:41:04.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:04 smithi085 conmon[35961]: (mon.0) 1798 : cluster [DBG] osdmap e844: 8 total, 8 up, 7 in 2021-02-10T04:41:04.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:04 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:41:04.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:04 smithi085 conmon[35961]: 41:04.206846+0000 mgr.y (mgr.14140) 810 : cluster [DBG] pgmap v1417: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 590 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 10 MiB/s rd, 1.4 MiB/s wr, 15 op/s 2021-02-10T04:41:04.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:04 smithi085 conmon[29319]: cluster 2021-02-10T04:41:03.649374+0000 mon.a (mon.0) 1798 2021-02-10T04:41:04.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:04 smithi085 conmon[29319]: : cluster [DBG] osdmap e844: 8 total, 8 up, 7 in 2021-02-10T04:41:04.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:04 smithi085 conmon[29319]: cluster 2021-02-10T04:41:04.206846+0000 mgr.y (mgr.14140) 810 : 2021-02-10T04:41:04.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:04 smithi085 conmon[29319]: cluster [DBG] pgmap v1417: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 590 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 10 MiB/s rd, 1.4 MiB/s wr, 15 op/s 2021-02-10T04:41:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:04 smithi164 conmon[29288]: cluster 2021-02-10T04:41:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:04 smithi164 conmon[29288]: 2021-02-10T04:41:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:04 smithi164 conmon[29288]: 2021 2021-02-10T04:41:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:04 smithi164 conmon[29288]: - 2021-02-10T04:41:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:04 smithi164 conmon[29288]: 02 2021-02-10T04:41:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:04 smithi164 conmon[29288]: - 2021-02-10T04:41:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:04 smithi164 conmon[29288]: 10 2021-02-10T04:41:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:04 smithi164 conmon[29288]: T04:41:03.649374+0000 mon.a 2021-02-10T04:41:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:04 smithi164 conmon[29288]: (mon.0) 1798 : cluster [DBG] osdmap e844: 8 total, 8 up, 7 in 2021-02-10T04:41:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:04 smithi164 conmon[29288]: cluster 2021-02-10T04:41:04.206846+0000 mgr.y (mgr.14140) 2021-02-10T04:41:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:04 smithi164 conmon[29288]: 810 : cluster [DBG] pgmap v1417: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 590 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 10 MiB/s rd, 1.4 MiB/s wr, 15 op/s 2021-02-10T04:41:05.145 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:41:04 smithi164 conmon[37263]: debug 2021-02-10T04:41:04.794+0000 7f1c671e2700 -1 osd.5 pg_epoch: 845 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x556656232840) 2021-02-10T04:41:05.652 INFO:tasks.rados.rados.0.smithi085.stdout:3348: done (4 left) 2021-02-10T04:41:05.652 INFO:tasks.rados.rados.0.smithi085.stdout:3349: read oid 29 snap 381 2021-02-10T04:41:05.653 INFO:tasks.rados.rados.0.smithi085.stdout:3349: expect (ObjNum 1091 snap 364 seq_num 1091) 2021-02-10T04:41:05.653 INFO:tasks.rados.rados.0.smithi085.stdout:3350: snap_remove snap 383 2021-02-10T04:41:05.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:05 smithi085 conmon[35961]: cluster 2021-02-10T04:41:04.650772+0000 2021-02-10T04:41:05.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:05 smithi085 conmon[35961]: mon.a (mon.0) 1799 : cluster [DBG] osdmap e845: 8 total, 8 up, 7 in 2021-02-10T04:41:05.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:05 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:41:05.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:05 smithi085 conmon[35961]: 02-10T04:41:05.652089+0000 mon.a (mon.0) 1800 : cluster 2021-02-10T04:41:05.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:05 smithi085 conmon[35961]: [DBG] osdmap e846: 8 total, 8 up, 7 in 2021-02-10T04:41:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:05 smithi085 conmon[29319]: cluster 2021-02-10T04:41:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:05 smithi085 conmon[29319]: 2021-02-10T04:41:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:05 smithi085 conmon[29319]: 2021 2021-02-10T04:41:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:05 smithi085 conmon[29319]: - 2021-02-10T04:41:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:05 smithi085 conmon[29319]: 02 2021-02-10T04:41:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:05 smithi085 conmon[29319]: - 2021-02-10T04:41:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:05 smithi085 conmon[29319]: 10 2021-02-10T04:41:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:05 smithi085 conmon[29319]: T 2021-02-10T04:41:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:05 smithi085 conmon[29319]: 04 2021-02-10T04:41:05.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:05 smithi085 conmon[29319]: : 2021-02-10T04:41:05.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:05 smithi085 conmon[29319]: 41:04.650772+0000 2021-02-10T04:41:05.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:05 smithi085 conmon[29319]: mon.a (mon.0) 1799 : cluster [DBG] osdmap e845: 8 total, 8 up, 7 in 2021-02-10T04:41:05.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:05 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:41:05.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:05 smithi085 conmon[29319]: 02-10T04:41:05.652089+0000 mon.a (mon. 2021-02-10T04:41:05.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:05 smithi085 conmon[29319]: 0) 1800 : cluster [DBG] osdmap e846: 8 total, 8 up, 7 in 2021-02-10T04:41:06.143 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:41:05 smithi164 conmon[37263]: debug 2021-02-10T04:41:05.788+0000 7f1c671e2700 -1 osd.5 pg_epoch: 846 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55665011e6e0) 2021-02-10T04:41:06.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:05 smithi164 conmon[29288]: cluster 2021-02-10T04:41:04.650772+0000 mon.a ( 2021-02-10T04:41:06.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:05 smithi164 conmon[29288]: mon.0) 1799 : cluster [DBG] osdmap e845: 8 total, 8 up, 7 in 2021-02-10T04:41:06.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:05 smithi164 conmon[29288]: cluster 2021-02-10T04:41:05.652089+0000 mon.a (mon.0) 1800 : cluster [DBG] osdmap e846: 8 total, 8 up, 7 in 2021-02-10T04:41:06.662 INFO:tasks.rados.rados.0.smithi085.stdout:3350: done (5 left) 2021-02-10T04:41:06.662 INFO:tasks.rados.rados.0.smithi085.stdout:3351: snap_remove snap 374 2021-02-10T04:41:06.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:06 smithi085 conmon[29319]: cluster 2021-02-10T04:41:06.207342+0000 mgr.y (mgr.14140) 2021-02-10T04:41:07.256 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:06 smithi085 conmon[29319]: 811 : cluster [DBG] pgmap v1420: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim_wait, 155 active+clean, 8 active+clean+snaptrim; 602 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 10 MiB/s rd, 2.5 MiB/s wr, 20 op/s 2021-02-10T04:41:07.256 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:06 smithi085 conmon[35961]: cluster 2021-02-10T04:41:07.257 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:06 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:41:07.257 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:06 smithi085 conmon[35961]: 41:06.207342+0000 mgr.y (mgr.14140) 811 : cluster [DBG] pgmap v1420: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim_wait, 155 active+clean, 8 active+clean+snaptrim; 602 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 10 MiB/s rd, 2.5 MiB/s wr, 20 op/s 2021-02-10T04:41:07.257 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:06 smithi164 conmon[29288]: cluster 2021-02-10T04:41:07.258 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:06 smithi164 conmon[29288]: 2021-02-10T04:41:07.258 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:06 smithi164 conmon[29288]: 2021 2021-02-10T04:41:07.258 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:06 smithi164 conmon[29288]: - 2021-02-10T04:41:07.258 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:06 smithi164 conmon[29288]: 02 2021-02-10T04:41:07.258 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:06 smithi164 conmon[29288]: -10T04:41:06.207342+0000 mgr.y (mgr.14140 2021-02-10T04:41:07.258 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:06 smithi164 conmon[29288]: ) 811 : cluster [DBG] pgmap v1420: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim_wait, 155 active+clean, 8 active+clean+snaptrim; 602 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 10 MiB/s rd, 2.5 MiB/s wr, 20 op/s 2021-02-10T04:41:07.656 INFO:tasks.rados.rados.0.smithi085.stdout:3351: done (5 left) 2021-02-10T04:41:07.656 INFO:tasks.rados.rados.0.smithi085.stdout:3352: snap_create 2021-02-10T04:41:07.656 INFO:tasks.rados.rados.0.smithi085.stdout:3349: expect (ObjNum 1091 snap 364 seq_num 1091) 2021-02-10T04:41:07.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:07 smithi085 conmon[35961]: cluster 2021-02-10T04:41:07.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:07 smithi085 conmon[35961]: 2021-02-10T04:41:07.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:07 smithi085 conmon[35961]: 2021 2021-02-10T04:41:07.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:07 smithi085 conmon[35961]: - 2021-02-10T04:41:07.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:07 smithi085 conmon[35961]: 02-10T04:41:06.126164+0000 osd.5 (osd 2021-02-10T04:41:07.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:07 smithi085 conmon[35961]: .5) 82 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:41:07.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:07 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:41:07.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:07 smithi085 conmon[35961]: 04:41:06.127122+0000 osd.5 (osd.5) 2021-02-10T04:41:07.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:07 smithi085 conmon[35961]: 83 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:41:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:07 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:41:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:07 smithi085 conmon[35961]: 02-10T04:41:06 2021-02-10T04:41:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:07 smithi085 conmon[35961]: .659373+0000 mon.a (mon.0) 1801 : cluster [DBG] osdmap e847: 8 total, 8 up, 7 in 2021-02-10T04:41:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:07 smithi085 conmon[35961]: 2021-02-10T04:41:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:07 smithi085 conmon[35961]: cluster 2021-02-10T04:41: 2021-02-10T04:41:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:07 smithi085 conmon[35961]: 07.655575+0000 mon.a (mon.0) 1802 2021-02-10T04:41:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:07 smithi085 conmon[35961]: : cluster [DBG] osdmap e848: 8 total, 8 up, 7 in 2021-02-10T04:41:07.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:07 smithi085 conmon[29319]: cluster 2021-02-10T04:41:06.126164+0000 2021-02-10T04:41:07.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:07 smithi085 conmon[29319]: osd.5 (osd.5) 82 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:41:07.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:07 smithi085 conmon[29319]: cluster 2021-02-10T04:41:06.127122+0000 osd.5 (osd.5) 83 : cluster 2021-02-10T04:41:07.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:07 smithi085 conmon[29319]: [DBG] purged_snaps scrub ok 2021-02-10T04:41:07.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:07 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:41:07.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:07 smithi085 conmon[29319]: 10T04:41:06.659373+0000 mon.a (mon.0) 2021-02-10T04:41:07.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:07 smithi085 conmon[29319]: 1801 : cluster [DBG] osdmap e847: 8 total, 8 up, 7 in 2021-02-10T04:41:07.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:07 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:41:07.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:07 smithi085 conmon[29319]: 10T04:41:07.655575+0000 mon.a (mon 2021-02-10T04:41:07.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:07 smithi085 conmon[29319]: .0) 1802 : cluster [DBG] osdmap e848: 8 total, 8 up, 7 in 2021-02-10T04:41:08.144 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:41:07 smithi164 conmon[37263]: debug 2021-02-10T04:41:07.839+0000 7f1c671e2700 -1 osd.5 pg_epoch: 848 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x5566534bbb80) 2021-02-10T04:41:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:07 smithi164 conmon[29288]: cluster 2021-02-10T04:41:06.126164+0000 osd.5 (osd.5) 82 : cluster 2021-02-10T04:41:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:07 smithi164 conmon[29288]: [DBG] purged_snaps scrub starts 2021-02-10T04:41:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:07 smithi164 conmon[29288]: cluster 2021-02-10T04:41:06.127122+0000 osd.5 (osd.5) 83 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:41:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:07 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:41:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:07 smithi164 conmon[29288]: :41:06.659373+0000 mon.a (mon.0) 1801 : cluster [DBG] osdmap e847: 8 total, 8 up, 7 in 2021-02-10T04:41:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:07 smithi164 conmon[29288]: cluster 2021-02-10T04:41:07.655575 2021-02-10T04:41:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:07 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1802 : cluster [DBG] osdmap e848: 8 total, 8 up, 7 in 2021-02-10T04:41:08.280 INFO:tasks.rados.rados.0.smithi085.stdout:3349: done (5 left) 2021-02-10T04:41:08.281 INFO:tasks.rados.rados.0.smithi085.stdout:3352: done (4 left) 2021-02-10T04:41:08.281 INFO:tasks.rados.rados.0.smithi085.stdout:3353: read oid 1 snap -1 2021-02-10T04:41:08.281 INFO:tasks.rados.rados.0.smithi085.stdout:3353: expect (ObjNum 1136 snap 381 seq_num 1136) 2021-02-10T04:41:08.281 INFO:tasks.rados.rados.0.smithi085.stdout:3354: copy_from oid 14 from oid 4 current snap is 389 2021-02-10T04:41:08.281 INFO:tasks.rados.rados.0.smithi085.stdout:3355: write (excl) oid 11 current snap is 389 2021-02-10T04:41:08.281 INFO:tasks.rados.rados.0.smithi085.stdout:3355: seq_num 1146 ranges {460822=488304,1378129=517393,2504632=405962} 2021-02-10T04:41:08.286 INFO:tasks.rados.rados.0.smithi085.stdout:3355: writing smithi08575397-11 from 460822 to 949126 tid 1 2021-02-10T04:41:08.293 INFO:tasks.rados.rados.0.smithi085.stdout:3355: writing smithi08575397-11 from 1378129 to 1895522 tid 2 2021-02-10T04:41:08.300 INFO:tasks.rados.rados.0.smithi085.stdout:3355: writing smithi08575397-11 from 2504632 to 2910594 tid 3 2021-02-10T04:41:08.300 INFO:tasks.rados.rados.0.smithi085.stdout:3353: expect (ObjNum 1136 snap 381 seq_num 1136) 2021-02-10T04:41:08.518 INFO:tasks.rados.rados.0.smithi085.stdout:3355: finishing write tid 1 to smithi08575397-11 2021-02-10T04:41:08.518 INFO:tasks.rados.rados.0.smithi085.stdout:3355: finishing write tid 2 to smithi08575397-11 2021-02-10T04:41:08.519 INFO:tasks.rados.rados.0.smithi085.stdout:3355: finishing write tid 3 to smithi08575397-11 2021-02-10T04:41:08.519 INFO:tasks.rados.rados.0.smithi085.stdout:3355: finishing write tid 4 to smithi08575397-11 2021-02-10T04:41:08.519 INFO:tasks.rados.rados.0.smithi085.stdout:3355: finishing write tid 5 to smithi08575397-11 2021-02-10T04:41:08.519 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 1738 (ObjNum 1146 snap 389 seq_num 1146) dirty exists 2021-02-10T04:41:08.519 INFO:tasks.rados.rados.0.smithi085.stdout:3355: left oid 11 (ObjNum 1146 snap 389 seq_num 1146) 2021-02-10T04:41:08.519 INFO:tasks.rados.rados.0.smithi085.stdout:3353: done (6 left) 2021-02-10T04:41:08.519 INFO:tasks.rados.rados.0.smithi085.stdout:3355: done (5 left) 2021-02-10T04:41:08.519 INFO:tasks.rados.rados.0.smithi085.stdout:3356: snap_create 2021-02-10T04:41:08.550 INFO:tasks.rados.rados.0.smithi085.stdout:3354: finishing copy_from to smithi08575397-14 2021-02-10T04:41:08.550 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 1350 (ObjNum 1049 snap 344 seq_num 1049) dirty exists 2021-02-10T04:41:08.550 INFO:tasks.rados.rados.0.smithi085.stdout:3354: finishing copy_from racing read to smithi08575397-14 2021-02-10T04:41:08.728 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:41:08.763 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:41:08.893 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:41:08 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:41:08.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:08 smithi164 conmon[29288]: cluster 2021-02-10T04:41:08.208024+0000 mgr.y (mgr.14140) 812 : cluster [DBG] pgmap v1423: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim_wait, 155 active+clean, 8 active+clean+snaptrim; 602 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 2.2 MiB/s rd, 1.1 MiB/s wr, 6 op/s 2021-02-10T04:41:08.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:08 smithi164 conmon[29288]: cluster 2021-02-10T04:41:08.278233+0000 mon.a (mon.0) 1803 : cluster [DBG] osdmap e849: 8 total, 8 up, 7 in 2021-02-10T04:41:08.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:08 smithi085 conmon[29319]: cluster 2021-02-10T04:41:08.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:08 smithi085 conmon[29319]: 2021-02 2021-02-10T04:41:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:08 smithi085 conmon[29319]: -10T04:41:08.208024+0000 mgr.y (mgr.14140) 812 : cluster [DBG] pgmap v1423: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim_wait, 155 active+clean, 8 active+clean+snaptrim; 602 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 2.2 MiB/s rd, 1.1 MiB/s wr, 6 op/s 2021-02-10T04:41:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:08 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:41:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:08 smithi085 conmon[29319]: 10T04:41:08.278233+0000 mon.a (mon.0 2021-02-10T04:41:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:08 smithi085 conmon[29319]: ) 1803 : cluster [DBG] osdmap e849: 8 total, 8 up, 7 in 2021-02-10T04:41:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:08 smithi085 conmon[35961]: cluster 2021-02-10T04:41:08.208024+0000 mgr.y (mgr 2021-02-10T04:41:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:08 smithi085 conmon[35961]: .14140) 812 : cluster [DBG] pgmap v1423: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim_wait, 155 active+clean, 8 active+clean+snaptrim; 602 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 2.2 MiB/s rd, 1.1 MiB/s wr, 6 op/s 2021-02-10T04:41:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:08 smithi085 conmon[35961]: cluster 2021-02-10T04:41 2021-02-10T04:41:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:08 smithi085 conmon[35961]: :08.278233+0000 mon.a (mon.0) 1803 : cluster [DBG] osdmap e849: 8 total, 8 up, 7 in 2021-02-10T04:41:09.293 INFO:tasks.rados.rados.0.smithi085.stdout:3354: done (5 left) 2021-02-10T04:41:09.293 INFO:tasks.rados.rados.0.smithi085.stdout:3356: done (4 left) 2021-02-10T04:41:09.293 INFO:tasks.rados.rados.0.smithi085.stdout:3357: snap_remove snap 357 2021-02-10T04:41:09.394 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:41:09.683 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:41:09 smithi085 conmon[52836]: debug 2021-02-10T04:41:09.685 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:41:09 smithi085 conmon[52836]: 2021-02-10T04:41:09.195+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:41:09.687 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:09 smithi085 conmon[29319]: audit 2021-02-10T04:41:09.688 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:09 smithi085 conmon[29319]: 2021-02-10T04:41:09.689 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:09 smithi085 conmon[29319]: 2021 2021-02-10T04:41:09.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:09 smithi085 conmon[29319]: - 2021-02-10T04:41:09.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:09 smithi085 conmon[29319]: 02 2021-02-10T04:41:09.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:09 smithi085 conmon[29319]: - 2021-02-10T04:41:09.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:09 smithi085 conmon[29319]: 10 2021-02-10T04:41:09.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:09 smithi085 conmon[29319]: T04:41:08. 2021-02-10T04:41:09.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:09 smithi085 conmon[29319]: 438020+0000 mgr.y (mgr.14140) 813 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:41:09.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:09 smithi085 conmon[29319]: cluster 2021-02-10T04:41:09.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:09 smithi085 conmon[29319]: 2021-02-10T04:41: 2021-02-10T04:41:09.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:09 smithi085 conmon[29319]: 09.287993+0000 mon.a (mon.0) 1804 : cluster [DBG] osdmap e850: 8 total, 8 up, 7 in 2021-02-10T04:41:09.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:09 smithi085 conmon[35961]: audit 2021-02-10T04:41:08. 2021-02-10T04:41:09.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:09 smithi085 conmon[35961]: 438020+0000 mgr.y (mgr.14140) 813 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:41:09.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:09 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:41:09.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:09 smithi085 conmon[35961]: -02-10T04:41:09.287993+0000 mon.a (mon.0) 1804 : cluster [DBG] 2021-02-10T04:41:09.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:09 smithi085 conmon[35961]: osdmap e850: 8 total, 8 up, 7 in 2021-02-10T04:41:10.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:09 smithi164 conmon[29288]: audit 2021-02-10T04:41:08.438020+0000 mgr.y (mgr.14140) 813 2021-02-10T04:41:10.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:09 smithi164 conmon[29288]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:41:10.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:09 smithi164 conmon[29288]: cluster 2021-02-10T04:41:09.287993+0000 mon.a (mon.0) 1804 : cluster [DBG] osdmap e850: 8 total, 8 up, 7 in 2021-02-10T04:41:10.291 INFO:tasks.rados.rados.0.smithi085.stdout:3357: done (4 left) 2021-02-10T04:41:10.291 INFO:tasks.rados.rados.0.smithi085.stdout:3358: delete oid 32 current snap is 390 2021-02-10T04:41:10.297 INFO:tasks.rados.rados.0.smithi085.stdout:3358: done (4 left) 2021-02-10T04:41:10.297 INFO:tasks.rados.rados.0.smithi085.stdout:3359: write (excl) oid 28 current snap is 390 2021-02-10T04:41:10.297 INFO:tasks.rados.rados.0.smithi085.stdout:3359: seq_num 1148 ranges {494436=755155,2011446=521062,3075831=218605} 2021-02-10T04:41:10.305 INFO:tasks.rados.rados.0.smithi085.stdout:3359: writing smithi08575397-28 from 494436 to 1249591 tid 1 2021-02-10T04:41:10.313 INFO:tasks.rados.rados.0.smithi085.stdout:3359: writing smithi08575397-28 from 2011446 to 2532508 tid 2 2021-02-10T04:41:10.318 INFO:tasks.rados.rados.0.smithi085.stdout:3359: writing smithi08575397-28 from 3075831 to 3294436 tid 3 2021-02-10T04:41:10.318 INFO:tasks.rados.rados.0.smithi085.stdout:3360: read oid 14 snap 266 2021-02-10T04:41:10.318 INFO:tasks.rados.rados.0.smithi085.stdout:3360: expect (ObjNum 756 snap 241 seq_num 756) 2021-02-10T04:41:10.318 INFO:tasks.rados.rados.0.smithi085.stdout:3361: read oid 34 snap -1 2021-02-10T04:41:10.318 INFO:tasks.rados.rados.0.smithi085.stdout:3361: expect (ObjNum 1145 snap 385 seq_num 1145) 2021-02-10T04:41:10.318 INFO:tasks.rados.rados.0.smithi085.stdout:3359: finishing write tid 1 to smithi08575397-28 2021-02-10T04:41:10.318 INFO:tasks.rados.rados.0.smithi085.stdout:3359: finishing write tid 2 to smithi08575397-28 2021-02-10T04:41:10.319 INFO:tasks.rados.rados.0.smithi085.stdout:3362: copy_from oid 13 from oid 12 current snap is 390 2021-02-10T04:41:10.319 INFO:tasks.rados.rados.0.smithi085.stdout:3363: rollback oid 40 current snap is 390 2021-02-10T04:41:10.319 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 40 to 278 2021-02-10T04:41:10.319 INFO:tasks.rados.rados.0.smithi085.stdout:3364: copy_from oid 5 from oid 50 current snap is 390 2021-02-10T04:41:10.319 INFO:tasks.rados.rados.0.smithi085.stdout:3365: copy_from oid 2 from oid 16 current snap is 390 2021-02-10T04:41:10.321 INFO:tasks.rados.rados.0.smithi085.stdout:3366: snap_create 2021-02-10T04:41:10.321 INFO:tasks.rados.rados.0.smithi085.stdout:3359: finishing write tid 3 to smithi08575397-28 2021-02-10T04:41:10.321 INFO:tasks.rados.rados.0.smithi085.stdout:3359: finishing write tid 4 to smithi08575397-28 2021-02-10T04:41:10.326 INFO:tasks.rados.rados.0.smithi085.stdout:3359: finishing write tid 5 to smithi08575397-28 2021-02-10T04:41:10.326 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 1454 (ObjNum 1148 snap 390 seq_num 1148) dirty exists 2021-02-10T04:41:10.326 INFO:tasks.rados.rados.0.smithi085.stdout:3359: left oid 28 (ObjNum 1148 snap 390 seq_num 1148) 2021-02-10T04:41:10.326 INFO:tasks.rados.rados.0.smithi085.stdout:3363: finishing rollback tid 0 to smithi08575397-40 2021-02-10T04:41:10.326 INFO:tasks.rados.rados.0.smithi085.stdout:3363: finishing rollback tid 1 to smithi08575397-40 2021-02-10T04:41:10.326 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 1743 (ObjNum 1075066000 snap 32576 seq_num 1073744080) dirty dne 2021-02-10T04:41:10.328 INFO:tasks.rados.rados.0.smithi085.stdout:3360: expect (ObjNum 756 snap 241 seq_num 756) 2021-02-10T04:41:10.532 INFO:tasks.rados.rados.0.smithi085.stdout:3361: expect (ObjNum 1145 snap 385 seq_num 1145) 2021-02-10T04:41:10.804 INFO:tasks.rados.rados.0.smithi085.stdout:3362: finishing copy_from to smithi08575397-13 2021-02-10T04:41:10.804 INFO:tasks.rados.rados.0.smithi085.stdout:3362: got expected ENOENT (src dne) 2021-02-10T04:41:10.804 INFO:tasks.rados.rados.0.smithi085.stdout:3362: finishing copy_from racing read to smithi08575397-13 2021-02-10T04:41:10.805 INFO:tasks.rados.rados.0.smithi085.stdout:3365: finishing copy_from to smithi08575397-2 2021-02-10T04:41:10.805 INFO:tasks.rados.rados.0.smithi085.stdout:3365: got expected ENOENT (src dne) 2021-02-10T04:41:10.805 INFO:tasks.rados.rados.0.smithi085.stdout:3365: finishing copy_from racing read to smithi08575397-2 2021-02-10T04:41:10.805 INFO:tasks.rados.rados.0.smithi085.stdout:3364: finishing copy_from to smithi08575397-5 2021-02-10T04:41:10.805 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 1549 (ObjNum 1054 snap 346 seq_num 1054) dirty exists 2021-02-10T04:41:10.805 INFO:tasks.rados.rados.0.smithi085.stdout:3364: finishing copy_from racing read to smithi08575397-5 2021-02-10T04:41:10.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:10 smithi085 conmon[29319]: cluster 2021-02-10T04:41:10.208832+0000 mgr.y (mgr. 2021-02-10T04:41:10.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:10 smithi085 conmon[29319]: 14140) 814 : cluster [DBG] pgmap v1426: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 604 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 6.7 MiB/s rd, 345 KiB/s wr, 7 op/s 2021-02-10T04:41:10.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:10 smithi085 conmon[29319]: cluster 2021-02-10T04:41:10.290261+0000 mon.a (mon.0) 1805 : cluster [DBG] osdmap e851: 8 total, 8 up, 7 in 2021-02-10T04:41:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:10 smithi085 conmon[35961]: cluster 2021-02-10T04:41: 2021-02-10T04:41:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:10 smithi085 conmon[35961]: 10.208832+0000 mgr.y (mgr.14140) 814 : cluster [DBG] pgmap v1426: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 604 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 6.7 MiB/s rd, 345 KiB/s wr, 7 op/s 2021-02-10T04:41:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:10 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:41:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:10 smithi085 conmon[35961]: 04:41:10.290261+0000 mon.a ( 2021-02-10T04:41:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:10 smithi085 conmon[35961]: mon.0) 1805 : cluster [DBG] 2021-02-10T04:41:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:10 smithi085 conmon[35961]: osdmap e851: 8 total, 8 up, 7 in 2021-02-10T04:41:10.992 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:41:11.040 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:41:11.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:10 smithi164 conmon[29288]: cluster 2021-02-10T04:41: 2021-02-10T04:41:11.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:10 smithi164 conmon[29288]: 10.208832+0000 mgr.y (mgr.14140) 814 : cluster [DBG] pgmap v1426: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 604 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 6.7 MiB/s rd, 345 KiB/s wr, 7 op/s 2021-02-10T04:41:11.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:10 smithi164 conmon[29288]: cluster 2021-02-10T04:41:10. 2021-02-10T04:41:11.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:10 smithi164 conmon[29288]: 290261+0000 mon.a (mon.0) 1805 : cluster [DBG] osdmap e851: 8 total, 8 up, 7 in 2021-02-10T04:41:11.208 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:41:11.294 INFO:tasks.rados.rados.0.smithi085.stdout:3359: done (11 left) 2021-02-10T04:41:11.294 INFO:tasks.rados.rados.0.smithi085.stdout:3360: done (10 left) 2021-02-10T04:41:11.294 INFO:tasks.rados.rados.0.smithi085.stdout:3361: done (9 left) 2021-02-10T04:41:11.294 INFO:tasks.rados.rados.0.smithi085.stdout:3362: done (8 left) 2021-02-10T04:41:11.294 INFO:tasks.rados.rados.0.smithi085.stdout:3363: done (7 left) 2021-02-10T04:41:11.294 INFO:tasks.rados.rados.0.smithi085.stdout:3364: done (6 left) 2021-02-10T04:41:11.295 INFO:tasks.rados.rados.0.smithi085.stdout:3365: done (5 left) 2021-02-10T04:41:11.295 INFO:tasks.rados.rados.0.smithi085.stdout:3366: done (4 left) 2021-02-10T04:41:11.295 INFO:tasks.rados.rados.0.smithi085.stdout:3367: snap_create 2021-02-10T04:41:11.309 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:41:11.341 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:41:11.441 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:41:11 smithi085 conmon[41540]: debug 2021-02-10T04:41:11.136+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:41:12.143 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:41:11 smithi164 conmon[40892]: debug 2021-02-10T04:41:11.694+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:41:12.207 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:41:12.207 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:41:12.232 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:41:12.303 INFO:tasks.rados.rados.0.smithi085.stdout:3367: done (4 left) 2021-02-10T04:41:12.303 INFO:tasks.rados.rados.0.smithi085.stdout:3368: write (excl) oid 2 current snap is 392 2021-02-10T04:41:12.303 INFO:tasks.rados.rados.0.smithi085.stdout:3368: seq_num 1149 ranges {511243=430404,1669928=664720,3103254=207989} 2021-02-10T04:41:12.303 INFO:tasks.rados.rados.0.smithi085.stdout:3368: writing smithi08575397-2 from 511243 to 941647 tid 1 2021-02-10T04:41:12.312 INFO:tasks.rados.rados.0.smithi085.stdout:3368: writing smithi08575397-2 from 1669928 to 2334648 tid 2 2021-02-10T04:41:12.319 INFO:tasks.rados.rados.0.smithi085.stdout:3368: writing smithi08575397-2 from 3103254 to 3311243 tid 3 2021-02-10T04:41:12.319 INFO:tasks.rados.rados.0.smithi085.stdout:3368: finishing write tid 1 to smithi08575397-2 2021-02-10T04:41:12.319 INFO:tasks.rados.rados.0.smithi085.stdout:3368: finishing write tid 2 to smithi08575397-2 2021-02-10T04:41:12.319 INFO:tasks.rados.rados.0.smithi085.stdout:3369: copy_from oid 49 from oid 13 current snap is 392 2021-02-10T04:41:12.319 INFO:tasks.rados.rados.0.smithi085.stdout:3370: write (excl) oid 43 current snap is 392 2021-02-10T04:41:12.319 INFO:tasks.rados.rados.0.smithi085.stdout:3370: seq_num 1150 ranges {528050=505653,1612057=524731,2614324=567403} 2021-02-10T04:41:12.325 INFO:tasks.rados.rados.0.smithi085.stdout:3370: writing smithi08575397-43 from 528050 to 1033703 tid 1 2021-02-10T04:41:12.333 INFO:tasks.rados.rados.0.smithi085.stdout:3370: writing smithi08575397-43 from 1612057 to 2136788 tid 2 2021-02-10T04:41:12.337 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:41:12.340 INFO:tasks.rados.rados.0.smithi085.stdout:3370: writing smithi08575397-43 from 2614324 to 3181727 tid 3 2021-02-10T04:41:12.340 INFO:tasks.rados.rados.0.smithi085.stdout:3369: finishing copy_from to smithi08575397-49 2021-02-10T04:41:12.340 INFO:tasks.rados.rados.0.smithi085.stdout:3369: got expected ENOENT (src dne) 2021-02-10T04:41:12.340 INFO:tasks.rados.rados.0.smithi085.stdout:3369: finishing copy_from racing read to smithi08575397-49 2021-02-10T04:41:12.340 INFO:tasks.rados.rados.0.smithi085.stdout:3368: finishing write tid 3 to smithi08575397-2 2021-02-10T04:41:12.340 INFO:tasks.rados.rados.0.smithi085.stdout:3368: finishing write tid 4 to smithi08575397-2 2021-02-10T04:41:12.340 INFO:tasks.rados.rados.0.smithi085.stdout:3368: finishing write tid 5 to smithi08575397-2 2021-02-10T04:41:12.340 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 1491 (ObjNum 1149 snap 392 seq_num 1149) dirty exists 2021-02-10T04:41:12.340 INFO:tasks.rados.rados.0.smithi085.stdout:3368: left oid 2 (ObjNum 1149 snap 392 seq_num 1149) 2021-02-10T04:41:12.340 INFO:tasks.rados.rados.0.smithi085.stdout:3370: finishing write tid 1 to smithi08575397-43 2021-02-10T04:41:12.341 INFO:tasks.rados.rados.0.smithi085.stdout:3370: finishing write tid 2 to smithi08575397-43 2021-02-10T04:41:12.341 INFO:tasks.rados.rados.0.smithi085.stdout:3368: done (6 left) 2021-02-10T04:41:12.341 INFO:tasks.rados.rados.0.smithi085.stdout:3369: done (5 left) 2021-02-10T04:41:12.341 INFO:tasks.rados.rados.0.smithi085.stdout:3371: snap_create 2021-02-10T04:41:12.343 INFO:tasks.rados.rados.0.smithi085.stdout:3370: finishing write tid 3 to smithi08575397-43 2021-02-10T04:41:12.344 INFO:tasks.rados.rados.0.smithi085.stdout:3370: finishing write tid 4 to smithi08575397-43 2021-02-10T04:41:12.344 INFO:tasks.rados.rados.0.smithi085.stdout:3370: finishing write tid 5 to smithi08575397-43 2021-02-10T04:41:12.344 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 1570 (ObjNum 1150 snap 392 seq_num 1150) dirty exists 2021-02-10T04:41:12.344 INFO:tasks.rados.rados.0.smithi085.stdout:3370: left oid 43 (ObjNum 1150 snap 392 seq_num 1150) 2021-02-10T04:41:12.368 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:41:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:12 smithi164 conmon[29288]: cluster 2021-02-10T04:41:11.080072 2021-02-10T04:41:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:12 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1939 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:41:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:12 smithi164 conmon[29288]: cluster 2021-02-10T04:41:11.080742+0000 osd.2 (osd.2) 1940 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:41:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:12 smithi164 conmon[29288]: cluster 2021-02-10T04:41:11.291950+0000 mon.a (mon.0) 1806 : cluster [DBG] osdmap e852: 8 total, 8 up, 7 in 2021-02-10T04:41:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:12 smithi164 conmon[29288]: cluster 2021-02-10T04:41:12.209413+0000 2021-02-10T04:41:12.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:12 smithi164 conmon[29288]: mgr.y (mgr.14140) 815 : cluster [DBG] pgmap v1429: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 604 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 4.4 MiB/s rd, 345 KiB/s wr, 5 op/s 2021-02-10T04:41:12.645 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:12 smithi164 conmon[33677]: debug 2021-02-10T04:41:12.645 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:12 smithi164 conmon[33677]: 2021-02-10T04:41:12.451+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:41:12.645 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:12 smithi164 conmon[33677]: 2021-02-10T04:41:12.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:12 smithi085 conmon[29319]: cluster 2021-02-10T04:41:11.080072+0000 osd.2 (osd.2) 1939 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:41:12.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:12 smithi085 conmon[29319]: cluster 2021-02-10T04:41:11.080742+0000 osd.2 (osd.2) 1940 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:41:12.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:12 smithi085 conmon[29319]: cluster 2021-02-10T04:41:11.291950+0000 mon.a (mon.0) 1806 : cluster [DBG] osdmap e852: 8 total, 8 up, 7 in 2021-02-10T04:41:12.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:12 smithi085 conmon[29319]: cluster 2021-02-10T04:41:12.209413+0000 mgr.y (mgr.14140) 815 : cluster [DBG] pgmap v1429: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 604 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 4.4 MiB/s rd, 345 KiB/s wr, 5 op/s 2021-02-10T04:41:12.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:12 smithi085 conmon[35961]: cluster 2021-02-10T04:41:11.080072+0000 osd.2 (osd.2) 1939 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:41:12.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:12 smithi085 conmon[35961]: cluster 2021-02-10T04:41:11.080742+0000 osd.2 (osd.2) 1940 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:41:12.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:12 smithi085 conmon[35961]: cluster 2021-02-10T04:41:11.291950+0000 mon.a (mon.0) 1806 : cluster [DBG] osdmap e852: 8 total, 8 up, 7 in 2021-02-10T04:41:12.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:12 smithi085 conmon[35961]: cluster 2021-02-10T04:41:12.209413+0000 mgr.y (mgr.14140) 815 : cluster [DBG] pgmap v1429: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 604 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 4.4 MiB/s rd, 345 KiB/s wr, 5 op/s 2021-02-10T04:41:12.698 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:41:12.805 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:41:12.838 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:41:12.991 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:41:13.073 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset nodeep-scrub 2021-02-10T04:41:13.143 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:41:12 smithi164 conmon[37263]: debug 2021-02-10T04:41:12.910+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:41:13.187 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:41:13.293 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:41:13.311 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:13 smithi085 conmon[29319]: cluster 2021-02-10T04:41:13.325 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:41:13.325 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:41:13.380 INFO:tasks.rados.rados.0.smithi085.stdout:3370: done (5 left) 2021-02-10T04:41:13.381 INFO:tasks.rados.rados.0.smithi085.stdout:3371: done (4 left) 2021-02-10T04:41:13.381 INFO:tasks.rados.rados.0.smithi085.stdout:3372: rollback oid 15 current snap is 393 2021-02-10T04:41:13.381 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 15 to 372 2021-02-10T04:41:13.381 INFO:tasks.rados.rados.0.smithi085.stdout:3373: copy_from oid 7 from oid 14 current snap is 393 2021-02-10T04:41:13.381 INFO:tasks.rados.rados.0.smithi085.stdout:3374: read oid 2 snap 346 2021-02-10T04:41:13.381 INFO:tasks.rados.rados.0.smithi085.stdout:3374: expect (ObjNum 982 snap 317 seq_num 982) 2021-02-10T04:41:13.381 INFO:tasks.rados.rados.0.smithi085.stdout:3375: snap_create 2021-02-10T04:41:13.387 INFO:tasks.rados.rados.0.smithi085.stdout:3374: expect (ObjNum 982 snap 317 seq_num 982) 2021-02-10T04:41:13.403 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.4 to 1 (10000) 2021-02-10T04:41:13.419 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:41:13.535 INFO:tasks.rados.rados.0.smithi085.stdout:3372: finishing rollback tid 0 to smithi08575397-15 2021-02-10T04:41:13.536 INFO:tasks.rados.rados.0.smithi085.stdout:3372: finishing rollback tid 1 to smithi08575397-15 2021-02-10T04:41:13.536 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 15 v 1463 (ObjNum 83033760 snap 22030 seq_num 0) dirty dne 2021-02-10T04:41:13.615 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:41:13.621 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:41:13 smithi164 conmon[44544]: debug 2021-02-10T04:41:13.399+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:41:13.621 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:13 smithi164 conmon[29288]: cluster 2021-02-10T04:41:12.293316+0000 mon.a (mon.0) 1807 : 2021-02-10T04:41:13.621 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:13 smithi164 conmon[29288]: cluster [DBG] osdmap e853: 8 total, 8 up, 7 in 2021-02-10T04:41:13.621 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:13 smithi164 conmon[29288]: audit 2021-02-10T04:41:12.480321+0000 mon.a (mon.0) 1808 : audit 2021-02-10T04:41:13.621 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:13 smithi164 conmon[29288]: [INF] from='client.? 172.21.15.85:0/1401302564' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-02-10T04:41:13.652 INFO:tasks.rados.rados.0.smithi085.stdout:3373: finishing copy_from to smithi08575397-7 2021-02-10T04:41:13.652 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 1350 (ObjNum 1049 snap 344 seq_num 1049) dirty exists 2021-02-10T04:41:13.653 INFO:tasks.rados.rados.0.smithi085.stdout:3373: finishing copy_from racing read to smithi08575397-7 2021-02-10T04:41:13.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:13 smithi085 conmon[29319]: 2021-02-10T04:41:13.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:13 smithi085 conmon[29319]: 2021 2021-02-10T04:41:13.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:13 smithi085 conmon[29319]: - 2021-02-10T04:41:13.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:13 smithi085 conmon[29319]: 02 2021-02-10T04:41:13.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:13 smithi085 conmon[29319]: - 2021-02-10T04:41:13.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:13 smithi085 conmon[29319]: 10 2021-02-10T04:41:13.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:13 smithi085 conmon[29319]: T 2021-02-10T04:41:13.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:13 smithi085 conmon[29319]: 04:41:12.293316 2021-02-10T04:41:13.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:13 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1807 : cluster [DBG] osdmap e853: 8 total, 8 up, 7 in 2021-02-10T04:41:13.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:13 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:41:13.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:13 smithi085 conmon[29319]: 10T04:41:12.480321+0000 mon.a (mon.0) 1808 : 2021-02-10T04:41:13.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:13 smithi085 conmon[29319]: audit [INF] from='client.? 172.21.15.85:0/1401302564' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-02-10T04:41:13.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:13 smithi085 conmon[35961]: cluster 2021-02-10T04:41:12.293316+0000 2021-02-10T04:41:13.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:13 smithi085 conmon[35961]: mon.a (mon.0) 1807 : cluster [DBG] osdmap e853: 8 total, 8 up, 7 in 2021-02-10T04:41:13.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:13 smithi085 conmon[35961]: audit 2021-02-10T04:41:12.480321+0000 mon.a (mon.0) 2021-02-10T04:41:13.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:13 smithi085 conmon[35961]: 1808 : audit [INF] from='client.? 172.21.15.85:0/1401302564' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-02-10T04:41:13.721 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:41:13.752 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:41:13.893 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:41:13 smithi164 conmon[37263]: debug 2021-02-10T04:41:13.838+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:41:14.190 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:41:14.296 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:41:14.328 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:41:14.391 INFO:tasks.rados.rados.0.smithi085.stdout:3372: done (7 left) 2021-02-10T04:41:14.391 INFO:tasks.rados.rados.0.smithi085.stdout:3373: done (6 left) 2021-02-10T04:41:14.391 INFO:tasks.rados.rados.0.smithi085.stdout:3374: done (5 left) 2021-02-10T04:41:14.392 INFO:tasks.rados.rados.0.smithi085.stdout:3375: done (4 left) 2021-02-10T04:41:14.392 INFO:tasks.rados.rados.0.smithi085.stdout:3376: delete oid 4 current snap is 394 2021-02-10T04:41:14.398 INFO:tasks.rados.rados.0.smithi085.stdout:3376: done (4 left) 2021-02-10T04:41:14.398 INFO:tasks.rados.rados.0.smithi085.stdout:3377: write (excl) oid 50 current snap is 394 2021-02-10T04:41:14.398 INFO:tasks.rados.rados.0.smithi085.stdout:3377: seq_num 1152 ranges {561664=656151,1729021=528400,2669170=692494} 2021-02-10T04:41:14.405 INFO:tasks.rados.rados.0.smithi085.stdout:3377: writing smithi08575397-50 from 561664 to 1217815 tid 1 2021-02-10T04:41:14.412 INFO:tasks.rados.rados.0.smithi085.stdout:3377: writing smithi08575397-50 from 1729021 to 2257421 tid 2 2021-02-10T04:41:14.420 INFO:tasks.rados.rados.0.smithi085.stdout:3377: writing smithi08575397-50 from 2669170 to 3361664 tid 3 2021-02-10T04:41:14.420 INFO:tasks.rados.rados.0.smithi085.stdout:3378: copy_from oid 1 from oid 30 current snap is 394 2021-02-10T04:41:14.420 INFO:tasks.rados.rados.0.smithi085.stdout:3377: finishing write tid 1 to smithi08575397-50 2021-02-10T04:41:14.420 INFO:tasks.rados.rados.0.smithi085.stdout:3377: finishing write tid 2 to smithi08575397-50 2021-02-10T04:41:14.420 INFO:tasks.rados.rados.0.smithi085.stdout:3379: snap_create 2021-02-10T04:41:14.425 INFO:tasks.rados.rados.0.smithi085.stdout:3377: finishing write tid 3 to smithi08575397-50 2021-02-10T04:41:14.425 INFO:tasks.rados.rados.0.smithi085.stdout:3377: finishing write tid 4 to smithi08575397-50 2021-02-10T04:41:14.425 INFO:tasks.rados.rados.0.smithi085.stdout:3377: finishing write tid 5 to smithi08575397-50 2021-02-10T04:41:14.425 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 1573 (ObjNum 1152 snap 394 seq_num 1152) dirty exists 2021-02-10T04:41:14.425 INFO:tasks.rados.rados.0.smithi085.stdout:3377: left oid 50 (ObjNum 1152 snap 394 seq_num 1152) 2021-02-10T04:41:14.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:14 smithi164 conmon[29288]: audit 2021-02-10T04:41:13.376687+0000 mon.a (mon.0) 1809 : audit [INF] 2021-02-10T04:41:14.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:14 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/1401302564' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-02-10T04:41:14.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:14 smithi164 conmon[29288]: cluster 2021-02-10T04:41:13.376751+0000 mon.a (mon.0) 1810 : cluster [DBG] osdmap e854: 8 total, 8 up, 7 in 2021-02-10T04:41:14.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:14 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:41:14.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:14 smithi164 conmon[29288]: T04:41:14.210040+0000 mgr.y (mgr.14140) 816 : cluster [DBG] pgmap v1432: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 155 active+clean, 7 active+clean+snaptrim; 611 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 7.1 MiB/s rd, 1.8 MiB/s wr, 7 op/s 2021-02-10T04:41:14.645 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:14 smithi164 conmon[33677]: debug 2021-02-10T04:41:14.421+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:41:14.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:14 smithi085 conmon[35961]: audit 2021-02-10T04:41:13.376687+0000 mon.a (mon.0 2021-02-10T04:41:14.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:14 smithi085 conmon[35961]: ) 1809 : audit [INF] from='client.? 172.21.15.85:0/1401302564' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-02-10T04:41:14.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:14 smithi085 conmon[35961]: cluster 2021-02-10T04:41:13.376751+0000 mon.a (mon.0) 1810 : cluster [DBG] 2021-02-10T04:41:14.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:14 smithi085 conmon[35961]: osdmap e854: 8 total, 8 up, 7 in 2021-02-10T04:41:14.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:14 smithi085 conmon[35961]: cluster 2021-02-10T04:41:14.210040+0000 2021-02-10T04:41:14.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:14 smithi085 conmon[35961]: mgr.y (mgr.14140) 816 : cluster [DBG] pgmap v1432: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 155 active+clean, 7 active+clean+snaptrim; 611 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 7.1 MiB/s rd, 1.8 MiB/s wr, 7 op/s 2021-02-10T04:41:14.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:14 smithi085 conmon[29319]: audit 2021-02-10T04:41:13.376687+0000 mon.a (mon.0 2021-02-10T04:41:14.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:14 smithi085 conmon[29319]: ) 1809 : audit [INF] from='client.? 172.21.15.85:0/1401302564' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-02-10T04:41:14.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:14 smithi085 conmon[29319]: cluster 2021-02-10T04:41:13.376751+0000 mon.a (mon.0) 1810 : cluster [DBG] osdmap e854: 8 total, 8 up, 7 in 2021-02-10T04:41:14.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:14 smithi085 conmon[29319]: cluster 2021-02-10T04:41:14.210040+0000 mgr.y ( 2021-02-10T04:41:14.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:14 smithi085 conmon[29319]: mgr.14140) 816 : cluster [DBG] pgmap v1432: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 155 active+clean, 7 active+clean+snaptrim; 611 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 7.1 MiB/s rd, 1.8 MiB/s wr, 7 op/s 2021-02-10T04:41:14.698 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:41:14.725 INFO:tasks.rados.rados.0.smithi085.stdout:3378: finishing copy_from to smithi08575397-1 2021-02-10T04:41:14.725 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 1570 (ObjNum 1025 snap 335 seq_num 1025) dirty exists 2021-02-10T04:41:14.725 INFO:tasks.rados.rados.0.smithi085.stdout:3378: finishing copy_from racing read to smithi08575397-1 2021-02-10T04:41:14.803 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:41:14.838 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:41:15.394 INFO:tasks.rados.rados.0.smithi085.stdout:3377: done (6 left) 2021-02-10T04:41:15.394 INFO:tasks.rados.rados.0.smithi085.stdout:3378: done (5 left) 2021-02-10T04:41:15.394 INFO:tasks.rados.rados.0.smithi085.stdout:3379: done (4 left) 2021-02-10T04:41:15.394 INFO:tasks.rados.rados.0.smithi085.stdout:3380: delete oid 45 current snap is 395 2021-02-10T04:41:15.404 INFO:tasks.rados.rados.0.smithi085.stdout:3380: done (4 left) 2021-02-10T04:41:15.404 INFO:tasks.rados.rados.0.smithi085.stdout:3381: delete oid 36 current snap is 395 2021-02-10T04:41:15.406 INFO:tasks.rados.rados.0.smithi085.stdout:3381: done (4 left) 2021-02-10T04:41:15.406 INFO:tasks.rados.rados.0.smithi085.stdout:3382: snap_create 2021-02-10T04:41:15.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:15 smithi164 conmon[29288]: cluster 2021-02-10T04:41:14.391242+0000 2021-02-10T04:41:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:15 smithi164 conmon[29288]: mon.a (mon.0) 1811 : cluster [DBG] osdmap e855: 8 total, 8 up, 7 in 2021-02-10T04:41:15.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:15 smithi085 conmon[35961]: cluster 2021-02-10T04:41:14. 2021-02-10T04:41:15.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:15 smithi085 conmon[35961]: 391242+0000 mon.a (mon.0) 1811 : cluster [DBG] osdmap e855: 8 total, 8 up, 7 in 2021-02-10T04:41:15.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:15 smithi085 conmon[29319]: cluster 2021-02-10T04:41:15.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:15 smithi085 conmon[29319]: 2021-02-10T04:41 2021-02-10T04:41:15.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:15 smithi085 conmon[29319]: :14.391242+0000 mon.a (mon.0) 1811 : cluster [DBG] osdmap e855: 8 total, 8 up, 7 in 2021-02-10T04:41:16.393 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:41:15 smithi164 conmon[37263]: debug 2021-02-10T04:41:15.913+0000 7f1c671e2700 -1 osd.5 pg_epoch: 856 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55665699e6e0) 2021-02-10T04:41:16.404 INFO:tasks.rados.rados.0.smithi085.stdout:3382: done (4 left) 2021-02-10T04:41:16.404 INFO:tasks.rados.rados.0.smithi085.stdout:3383: write oid 44 current snap is 396 2021-02-10T04:41:16.404 INFO:tasks.rados.rados.0.smithi085.stdout:3383: seq_num 1155 ranges {612085=481898,1788114=675727,3035086=376999} 2021-02-10T04:41:16.410 INFO:tasks.rados.rados.0.smithi085.stdout:3383: writing smithi08575397-44 from 612085 to 1093983 tid 1 2021-02-10T04:41:16.419 INFO:tasks.rados.rados.0.smithi085.stdout:3383: writing smithi08575397-44 from 1788114 to 2463841 tid 2 2021-02-10T04:41:16.425 INFO:tasks.rados.rados.0.smithi085.stdout:3383: writing smithi08575397-44 from 3035086 to 3412085 tid 3 2021-02-10T04:41:16.425 INFO:tasks.rados.rados.0.smithi085.stdout:3383: finishing write tid 1 to smithi08575397-44 2021-02-10T04:41:16.425 INFO:tasks.rados.rados.0.smithi085.stdout:3383: finishing write tid 2 to smithi08575397-44 2021-02-10T04:41:16.425 INFO:tasks.rados.rados.0.smithi085.stdout:3384: write (excl) oid 21 current snap is 396 2021-02-10T04:41:16.425 INFO:tasks.rados.rados.0.smithi085.stdout:3384: seq_num 1156 ranges {628892=673500,1962949=535738,3178862=250030} 2021-02-10T04:41:16.433 INFO:tasks.rados.rados.0.smithi085.stdout:3384: writing smithi08575397-21 from 628892 to 1302392 tid 1 2021-02-10T04:41:16.441 INFO:tasks.rados.rados.0.smithi085.stdout:3384: writing smithi08575397-21 from 1962949 to 2498687 tid 2 2021-02-10T04:41:16.447 INFO:tasks.rados.rados.0.smithi085.stdout:3384: writing smithi08575397-21 from 3178862 to 3428892 tid 3 2021-02-10T04:41:16.447 INFO:tasks.rados.rados.0.smithi085.stdout:3385: copy_from oid 9 from oid 48 current snap is 396 2021-02-10T04:41:16.448 INFO:tasks.rados.rados.0.smithi085.stdout:3383: finishing write tid 3 to smithi08575397-44 2021-02-10T04:41:16.448 INFO:tasks.rados.rados.0.smithi085.stdout:3383: finishing write tid 4 to smithi08575397-44 2021-02-10T04:41:16.448 INFO:tasks.rados.rados.0.smithi085.stdout:3383: finishing write tid 5 to smithi08575397-44 2021-02-10T04:41:16.448 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 1569 (ObjNum 1155 snap 396 seq_num 1155) dirty exists 2021-02-10T04:41:16.448 INFO:tasks.rados.rados.0.smithi085.stdout:3383: left oid 44 (ObjNum 1155 snap 396 seq_num 1155) 2021-02-10T04:41:16.448 INFO:tasks.rados.rados.0.smithi085.stdout:3384: finishing write tid 1 to smithi08575397-21 2021-02-10T04:41:16.448 INFO:tasks.rados.rados.0.smithi085.stdout:3384: finishing write tid 2 to smithi08575397-21 2021-02-10T04:41:16.448 INFO:tasks.rados.rados.0.smithi085.stdout:3383: done (6 left) 2021-02-10T04:41:16.448 INFO:tasks.rados.rados.0.smithi085.stdout:3386: read oid 50 snap -1 2021-02-10T04:41:16.448 INFO:tasks.rados.rados.0.smithi085.stdout:3386: expect (ObjNum 1152 snap 394 seq_num 1152) 2021-02-10T04:41:16.449 INFO:tasks.rados.rados.0.smithi085.stdout:3387: rollback oid 2 current snap is 396 2021-02-10T04:41:16.449 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 2 to 375 2021-02-10T04:41:16.449 INFO:tasks.rados.rados.0.smithi085.stdout:3388: write oid 12 current snap is 396 2021-02-10T04:41:16.449 INFO:tasks.rados.rados.0.smithi085.stdout:3388: seq_num 1157 ranges {645699=748749,2021431=679396,3206285=239414} 2021-02-10T04:41:16.457 INFO:tasks.rados.rados.0.smithi085.stdout:3388: writing smithi08575397-12 from 645699 to 1394448 tid 1 2021-02-10T04:41:16.466 INFO:tasks.rados.rados.0.smithi085.stdout:3388: writing smithi08575397-12 from 2021431 to 2700827 tid 2 2021-02-10T04:41:16.471 INFO:tasks.rados.rados.0.smithi085.stdout:3388: writing smithi08575397-12 from 3206285 to 3445699 tid 3 2021-02-10T04:41:16.471 INFO:tasks.rados.rados.0.smithi085.stdout:3389: snap_create 2021-02-10T04:41:16.473 INFO:tasks.rados.rados.0.smithi085.stdout:3387: finishing rollback tid 0 to smithi08575397-2 2021-02-10T04:41:16.473 INFO:tasks.rados.rados.0.smithi085.stdout:3387: finishing rollback tid 1 to smithi08575397-2 2021-02-10T04:41:16.473 INFO:tasks.rados.rados.0.smithi085.stdout:3387: finishing rollback tid 2 to smithi08575397-2 2021-02-10T04:41:16.473 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 1494 (ObjNum 1052 snap 345 seq_num 1052) dirty exists 2021-02-10T04:41:16.473 INFO:tasks.rados.rados.0.smithi085.stdout:3386: expect (ObjNum 1152 snap 394 seq_num 1152) 2021-02-10T04:41:16.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:16 smithi085 conmon[29319]: cluster 2021-02-10T04:41:16.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:16 smithi085 conmon[29319]: 2021-02-10T04:41:16.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:16 smithi085 conmon[29319]: 2021 2021-02-10T04:41:16.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:16 smithi085 conmon[29319]: - 2021-02-10T04:41:16.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:16 smithi085 conmon[29319]: 02 2021-02-10T04:41:16.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:16 smithi085 conmon[29319]: - 2021-02-10T04:41:16.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:16 smithi085 conmon[29319]: 10 2021-02-10T04:41:16.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:16 smithi085 conmon[29319]: T 2021-02-10T04:41:16.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:16 smithi085 conmon[29319]: 04 2021-02-10T04:41:16.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:16 smithi085 conmon[29319]: : 2021-02-10T04:41:16.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:16 smithi085 conmon[29319]: 41 2021-02-10T04:41:16.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:16 smithi085 conmon[29319]: : 2021-02-10T04:41:16.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:16 smithi085 conmon[29319]: 15 2021-02-10T04:41:16.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:16 smithi085 conmon[29319]: . 2021-02-10T04:41:16.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:16 smithi085 conmon[29319]: 393547 2021-02-10T04:41:16.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:16 smithi085 conmon[29319]: +0000 2021-02-10T04:41:16.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:16 smithi085 conmon[29319]: mon.a (mon.0) 1812 : cluster 2021-02-10T04:41:16.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:16 smithi085 conmon[29319]: [DBG] osdmap e856: 8 total, 8 up, 7 in 2021-02-10T04:41:16.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:16 smithi085 conmon[29319]: cluster 2021-02-10T04:41:16.210523 2021-02-10T04:41:16.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:16 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 817 : cluster [DBG] pgmap v1435: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim_wait, 156 active+clean, 7 active+clean+snaptrim; 620 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 8.1 MiB/s rd, 3.7 MiB/s wr, 15 op/s 2021-02-10T04:41:16.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:16 smithi085 conmon[29319]: 2021-02-10T04:41:16.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:16 smithi085 conmon[35961]: cluster 2021-02-10T04:41:15.393547+0000 mon.a 2021-02-10T04:41:16.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:16 smithi085 conmon[35961]: (mon.0) 1812 : cluster [DBG] osdmap e856: 8 total, 8 up, 7 in 2021-02-10T04:41:16.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:16 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:41:16.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:16 smithi085 conmon[35961]: 04:41:16.210523+0000 mgr.y (mgr.14140) 817 : cluster [DBG] pgmap v1435: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim_wait, 156 active+clean, 7 active+clean+snaptrim; 620 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 8.1 MiB/s rd, 3.7 MiB/s wr, 15 op/s 2021-02-10T04:41:16.741 INFO:tasks.rados.rados.0.smithi085.stdout:3384: finishing write tid 3 to smithi08575397-21 2021-02-10T04:41:16.741 INFO:tasks.rados.rados.0.smithi085.stdout:3384: finishing write tid 4 to smithi08575397-21 2021-02-10T04:41:16.741 INFO:tasks.rados.rados.0.smithi085.stdout:3384: finishing write tid 5 to smithi08575397-21 2021-02-10T04:41:16.742 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 1756 (ObjNum 1156 snap 396 seq_num 1156) dirty exists 2021-02-10T04:41:16.742 INFO:tasks.rados.rados.0.smithi085.stdout:3384: left oid 21 (ObjNum 1156 snap 396 seq_num 1156) 2021-02-10T04:41:16.742 INFO:tasks.rados.rados.0.smithi085.stdout:3388: finishing write tid 1 to smithi08575397-12 2021-02-10T04:41:16.742 INFO:tasks.rados.rados.0.smithi085.stdout:3388: finishing write tid 2 to smithi08575397-12 2021-02-10T04:41:16.742 INFO:tasks.rados.rados.0.smithi085.stdout:3388: finishing write tid 3 to smithi08575397-12 2021-02-10T04:41:16.743 INFO:tasks.rados.rados.0.smithi085.stdout:3388: finishing write tid 4 to smithi08575397-12 2021-02-10T04:41:16.743 INFO:tasks.rados.rados.0.smithi085.stdout:3388: finishing write tid 5 to smithi08575397-12 2021-02-10T04:41:16.743 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 1503 (ObjNum 1157 snap 396 seq_num 1157) dirty exists 2021-02-10T04:41:16.743 INFO:tasks.rados.rados.0.smithi085.stdout:3388: left oid 12 (ObjNum 1157 snap 396 seq_num 1157) 2021-02-10T04:41:16.743 INFO:tasks.rados.rados.0.smithi085.stdout:3385: finishing copy_from to smithi08575397-9 2021-02-10T04:41:16.744 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 9 v 1499 (ObjNum 1142 snap 385 seq_num 1142) dirty exists 2021-02-10T04:41:16.744 INFO:tasks.rados.rados.0.smithi085.stdout:3385: finishing copy_from racing read to smithi08575397-9 2021-02-10T04:41:16.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:16 smithi164 conmon[29288]: cluster 2021-02-10T04:41:15.393547+0000 2021-02-10T04:41:16.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:16 smithi164 conmon[29288]: mon.a (mon.0) 1812 : cluster [DBG] osdmap e856: 8 total, 8 up, 7 in 2021-02-10T04:41:16.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:16 smithi164 conmon[29288]: cluster 2021-02-10T04:41:16.210523+0000 mgr.y (mgr.14140) 817 : cluster [DBG] pgmap v1435: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim_wait, 156 active+clean, 7 active+clean+snaptrim; 620 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 8.1 MiB/s rd, 3.7 MiB/s wr, 15 op/s 2021-02-10T04:41:17.410 INFO:tasks.rados.rados.0.smithi085.stdout:3384: done (9 left) 2021-02-10T04:41:17.410 INFO:tasks.rados.rados.0.smithi085.stdout:3385: done (8 left) 2021-02-10T04:41:17.410 INFO:tasks.rados.rados.0.smithi085.stdout:3386: done (7 left) 2021-02-10T04:41:17.410 INFO:tasks.rados.rados.0.smithi085.stdout:3387: done (6 left) 2021-02-10T04:41:17.410 INFO:tasks.rados.rados.0.smithi085.stdout:3388: done (5 left) 2021-02-10T04:41:17.410 INFO:tasks.rados.rados.0.smithi085.stdout:3389: done (4 left) 2021-02-10T04:41:17.410 INFO:tasks.rados.rados.0.smithi085.stdout:3390: delete oid 29 current snap is 397 2021-02-10T04:41:17.414 INFO:tasks.rados.rados.0.smithi085.stdout:3390: done (4 left) 2021-02-10T04:41:17.415 INFO:tasks.rados.rados.0.smithi085.stdout:3391: read oid 14 snap -1 2021-02-10T04:41:17.415 INFO:tasks.rados.rados.0.smithi085.stdout:3391: expect (ObjNum 1049 snap 344 seq_num 1049) 2021-02-10T04:41:17.415 INFO:tasks.rados.rados.0.smithi085.stdout:3392: write oid 11 current snap is 397 2021-02-10T04:41:17.415 INFO:tasks.rados.rados.0.smithi085.stdout:3392: seq_num 1159 ranges {679313=499247,1622042=683065,2744778=701557} 2021-02-10T04:41:17.422 INFO:tasks.rados.rados.0.smithi085.stdout:3392: writing smithi08575397-11 from 679313 to 1178560 tid 1 2021-02-10T04:41:17.429 INFO:tasks.rados.rados.0.smithi085.stdout:3392: writing smithi08575397-11 from 1622042 to 2305107 tid 2 2021-02-10T04:41:17.436 INFO:tasks.rados.rados.0.smithi085.stdout:3392: writing smithi08575397-11 from 2744778 to 3446335 tid 3 2021-02-10T04:41:17.436 INFO:tasks.rados.rados.0.smithi085.stdout:3393: read oid 12 snap -1 2021-02-10T04:41:17.437 INFO:tasks.rados.rados.0.smithi085.stdout:3393: expect (ObjNum 1157 snap 396 seq_num 1157) 2021-02-10T04:41:17.437 INFO:tasks.rados.rados.0.smithi085.stdout:3394: write oid 25 current snap is 397 2021-02-10T04:41:17.437 INFO:tasks.rados.rados.0.smithi085.stdout:3394: seq_num 1160 ranges {696120=574496,1680524=543076,2888554=512829} 2021-02-10T04:41:17.444 INFO:tasks.rados.rados.0.smithi085.stdout:3394: writing smithi08575397-25 from 696120 to 1270616 tid 1 2021-02-10T04:41:17.450 INFO:tasks.rados.rados.0.smithi085.stdout:3394: writing smithi08575397-25 from 1680524 to 2223600 tid 2 2021-02-10T04:41:17.458 INFO:tasks.rados.rados.0.smithi085.stdout:3394: writing smithi08575397-25 from 2888554 to 3401383 tid 3 2021-02-10T04:41:17.458 INFO:tasks.rados.rados.0.smithi085.stdout:3395: snap_create 2021-02-10T04:41:17.458 INFO:tasks.rados.rados.0.smithi085.stdout:3391: expect (ObjNum 1049 snap 344 seq_num 1049) 2021-02-10T04:41:17.608 INFO:tasks.rados.rados.0.smithi085.stdout:3392: finishing write tid 1 to smithi08575397-11 2021-02-10T04:41:17.608 INFO:tasks.rados.rados.0.smithi085.stdout:3392: finishing write tid 2 to smithi08575397-11 2021-02-10T04:41:17.608 INFO:tasks.rados.rados.0.smithi085.stdout:3392: finishing write tid 3 to smithi08575397-11 2021-02-10T04:41:17.609 INFO:tasks.rados.rados.0.smithi085.stdout:3392: finishing write tid 4 to smithi08575397-11 2021-02-10T04:41:17.609 INFO:tasks.rados.rados.0.smithi085.stdout:3392: finishing write tid 5 to smithi08575397-11 2021-02-10T04:41:17.609 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 1761 (ObjNum 1159 snap 397 seq_num 1159) dirty exists 2021-02-10T04:41:17.609 INFO:tasks.rados.rados.0.smithi085.stdout:3392: left oid 11 (ObjNum 1159 snap 397 seq_num 1159) 2021-02-10T04:41:17.609 INFO:tasks.rados.rados.0.smithi085.stdout:3393: expect (ObjNum 1157 snap 396 seq_num 1157) 2021-02-10T04:41:17.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:17 smithi085 conmon[29319]: cluster 2021-02-10T04:41:16.403070+0000 mon.a (mon. 2021-02-10T04:41:17.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:17 smithi085 conmon[29319]: 0) 1813 : cluster [DBG] osdmap e857: 8 total, 8 up, 7 in 2021-02-10T04:41:17.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:17 smithi085 conmon[35961]: cluster 2021-02-10T04:41:16.403070 2021-02-10T04:41:17.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:17 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1813 : cluster [DBG] osdmap e857: 8 total, 8 up, 7 in 2021-02-10T04:41:17.757 INFO:tasks.rados.rados.0.smithi085.stdout:3394: finishing write tid 1 to smithi08575397-25 2021-02-10T04:41:17.757 INFO:tasks.rados.rados.0.smithi085.stdout:3394: finishing write tid 2 to smithi08575397-25 2021-02-10T04:41:17.757 INFO:tasks.rados.rados.0.smithi085.stdout:3394: finishing write tid 3 to smithi08575397-25 2021-02-10T04:41:17.757 INFO:tasks.rados.rados.0.smithi085.stdout:3394: finishing write tid 4 to smithi08575397-25 2021-02-10T04:41:17.757 INFO:tasks.rados.rados.0.smithi085.stdout:3394: finishing write tid 5 to smithi08575397-25 2021-02-10T04:41:17.757 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 25 v 1463 (ObjNum 1160 snap 397 seq_num 1160) dirty exists 2021-02-10T04:41:17.757 INFO:tasks.rados.rados.0.smithi085.stdout:3394: left oid 25 (ObjNum 1160 snap 397 seq_num 1160) 2021-02-10T04:41:17.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:17 smithi164 conmon[29288]: cluster 2021-02-10T04:41:16.403070+0000 mon.a (mon.0) 2021-02-10T04:41:17.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:17 smithi164 conmon[29288]: 1813 : cluster [DBG] osdmap e857: 8 total, 8 up, 7 in 2021-02-10T04:41:18.393 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:41:17 smithi164 conmon[37263]: debug 2021-02-10T04:41:17.906+0000 7f1c6b1ea700 -1 osd.5 pg_epoch: 858 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x556650b77080) 2021-02-10T04:41:18.413 INFO:tasks.rados.rados.0.smithi085.stdout:3391: done (8 left) 2021-02-10T04:41:18.413 INFO:tasks.rados.rados.0.smithi085.stdout:3392: done (7 left) 2021-02-10T04:41:18.413 INFO:tasks.rados.rados.0.smithi085.stdout:3393: done (6 left) 2021-02-10T04:41:18.414 INFO:tasks.rados.rados.0.smithi085.stdout:3394: done (5 left) 2021-02-10T04:41:18.414 INFO:tasks.rados.rados.0.smithi085.stdout:3395: done (4 left) 2021-02-10T04:41:18.414 INFO:tasks.rados.rados.0.smithi085.stdout:3396: rollback oid 12 current snap is 398 2021-02-10T04:41:18.414 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 12 to 377 2021-02-10T04:41:18.414 INFO:tasks.rados.rados.0.smithi085.stdout:3397: write (excl) oid 18 current snap is 398 2021-02-10T04:41:18.414 INFO:tasks.rados.rados.0.smithi085.stdout:3397: seq_num 1161 ranges {712927=649745,2139006=686734} 2021-02-10T04:41:18.421 INFO:tasks.rados.rados.0.smithi085.stdout:3397: writing smithi08575397-18 from 712927 to 1362672 tid 1 2021-02-10T04:41:18.430 INFO:tasks.rados.rados.0.smithi085.stdout:3397: writing smithi08575397-18 from 2139006 to 2825740 tid 2 2021-02-10T04:41:18.430 INFO:tasks.rados.rados.0.smithi085.stdout:3398: copy_from oid 36 from oid 14 current snap is 398 2021-02-10T04:41:18.431 INFO:tasks.rados.rados.0.smithi085.stdout:3396: finishing rollback tid 0 to smithi08575397-12 2021-02-10T04:41:18.431 INFO:tasks.rados.rados.0.smithi085.stdout:3396: finishing rollback tid 1 to smithi08575397-12 2021-02-10T04:41:18.431 INFO:tasks.rados.rados.0.smithi085.stdout:3396: finishing rollback tid 2 to smithi08575397-12 2021-02-10T04:41:18.431 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 1506 (ObjNum 1081 snap 359 seq_num 1081) dirty exists 2021-02-10T04:41:18.431 INFO:tasks.rados.rados.0.smithi085.stdout:3397: finishing write tid 1 to smithi08575397-18 2021-02-10T04:41:18.431 INFO:tasks.rados.rados.0.smithi085.stdout:3396: done (6 left) 2021-02-10T04:41:18.431 INFO:tasks.rados.rados.0.smithi085.stdout:3399: write oid 6 current snap is 398 2021-02-10T04:41:18.431 INFO:tasks.rados.rados.0.smithi085.stdout:3399: seq_num 1162 ranges {729734=724994,2197488=430392,3227047=302687} 2021-02-10T04:41:18.439 INFO:tasks.rados.rados.0.smithi085.stdout:3399: writing smithi08575397-6 from 729734 to 1454728 tid 1 2021-02-10T04:41:18.446 INFO:tasks.rados.rados.0.smithi085.stdout:3399: writing smithi08575397-6 from 2197488 to 2627880 tid 2 2021-02-10T04:41:18.451 INFO:tasks.rados.rados.0.smithi085.stdout:3399: writing smithi08575397-6 from 3227047 to 3529734 tid 3 2021-02-10T04:41:18.451 INFO:tasks.rados.rados.0.smithi085.stdout:3400: snap_remove snap 368 2021-02-10T04:41:18.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:18 smithi085 conmon[29319]: cluster 2021-02-10T04:41:17.407540+0000 mon.a (mon 2021-02-10T04:41:18.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:18 smithi085 conmon[29319]: .0) 1814 : cluster [DBG] osdmap e858: 8 total, 8 up, 7 in 2021-02-10T04:41:18.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:18 smithi085 conmon[29319]: cluster 2021-02-10T04:41:18.211211+0000 mgr.y (mgr.14140) 818 : cluster [DBG] pgmap v1438: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim_wait, 158 active+clean, 6 active+clean+snaptrim; 626 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 7.0 MiB/s rd, 3.1 MiB/s wr, 15 op/s 2021-02-10T04:41:18.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:18 smithi085 conmon[35961]: cluster 2021-02-10T04:41:18.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:18 smithi085 conmon[35961]: 2021-02-10T04:41:17.407540+0000 2021-02-10T04:41:18.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:18 smithi085 conmon[35961]: mon.a (mon.0) 1814 : cluster [DBG] osdmap e858: 8 total, 8 up, 7 in 2021-02-10T04:41:18.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:18 smithi085 conmon[35961]: cluster 2021-02-10T04:41:18.211211+0000 2021-02-10T04:41:18.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:18 smithi085 conmon[35961]: mgr.y (mgr.14140) 818 : cluster [DBG] pgmap v1438: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim_wait, 158 active+clean, 6 active+clean+snaptrim; 626 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 7.0 MiB/s rd, 3.1 MiB/s wr, 15 op/s 2021-02-10T04:41:18.894 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:41:18 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:41:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:18 smithi164 conmon[29288]: cluster 2021-02-10T04:41:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:18 smithi164 conmon[29288]: 2021-02-10T04:41:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:18 smithi164 conmon[29288]: 2021 2021-02-10T04:41:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:18 smithi164 conmon[29288]: - 2021-02-10T04:41:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:18 smithi164 conmon[29288]: 02-10T04:41:17.407540 2021-02-10T04:41:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:18 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1814 : cluster [DBG] osdmap e858: 8 total, 8 up, 7 in 2021-02-10T04:41:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:18 smithi164 conmon[29288]: cluster 2021-02-10T04:41:18.211211 2021-02-10T04:41:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:18 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 818 : cluster [DBG] 2021-02-10T04:41:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:18 smithi164 conmon[29288]: pgmap v1438: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim_wait, 158 active+clean, 6 active+clean+snaptrim; 626 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 7.0 MiB/s rd, 3.1 MiB/s wr, 15 op/s 2021-02-10T04:41:19.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:19 smithi085 conmon[29319]: cluster 2021-02-10T04:41:18 2021-02-10T04:41:19.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:19 smithi085 conmon[29319]: .412207+0000 mon.a (mon.0) 1815 : cluster [DBG] osdmap e859: 8 total, 8 up, 7 in 2021-02-10T04:41:19.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:19 smithi085 conmon[29319]: audit 2021-02-10T04:41:18.448028+0000 mgr.y (mgr.14140 2021-02-10T04:41:19.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:19 smithi085 conmon[29319]: ) 819 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:41:19.412 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:19 smithi085 conmon[35961]: cluster 2021-02-10T04:41:18.412207+0000 2021-02-10T04:41:19.412 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:19 smithi085 conmon[35961]: mon.a (mon.0) 1815 : cluster [DBG] osdmap e859: 8 total, 8 up, 7 in 2021-02-10T04:41:19.412 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:19 smithi085 conmon[35961]: audit 2021-02-10T04:41:19.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:19 smithi085 conmon[35961]: 2021-02-10T04:41:18.448028+0000 mgr.y (mgr 2021-02-10T04:41:19.413 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:19 smithi085 conmon[35961]: .14140) 819 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:41:19.421 INFO:tasks.rados.rados.0.smithi085.stdout:3400: done (7 left) 2021-02-10T04:41:19.421 INFO:tasks.rados.rados.0.smithi085.stdout:3401: write oid 7 current snap is 398 2021-02-10T04:41:19.421 INFO:tasks.rados.rados.0.smithi085.stdout:3401: seq_num 1163 ranges {746541=400243,1739617=690403,3138117=408424} 2021-02-10T04:41:19.427 INFO:tasks.rados.rados.0.smithi085.stdout:3401: writing smithi08575397-7 from 746541 to 1146784 tid 1 2021-02-10T04:41:19.435 INFO:tasks.rados.rados.0.smithi085.stdout:3401: writing smithi08575397-7 from 1739617 to 2430020 tid 2 2021-02-10T04:41:19.442 INFO:tasks.rados.rados.0.smithi085.stdout:3401: writing smithi08575397-7 from 3138117 to 3546541 tid 3 2021-02-10T04:41:19.442 INFO:tasks.rados.rados.0.smithi085.stdout:3402: copy_from oid 31 from oid 38 current snap is 398 2021-02-10T04:41:19.442 INFO:tasks.rados.rados.0.smithi085.stdout:3397: finishing write tid 2 to smithi08575397-18 2021-02-10T04:41:19.443 INFO:tasks.rados.rados.0.smithi085.stdout:3397: finishing write tid 3 to smithi08575397-18 2021-02-10T04:41:19.443 INFO:tasks.rados.rados.0.smithi085.stdout:3397: finishing write tid 4 to smithi08575397-18 2021-02-10T04:41:19.443 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 18 v 1466 (ObjNum 1161 snap 398 seq_num 1161) dirty exists 2021-02-10T04:41:19.443 INFO:tasks.rados.rados.0.smithi085.stdout:3397: left oid 18 (ObjNum 1161 snap 398 seq_num 1161) 2021-02-10T04:41:19.443 INFO:tasks.rados.rados.0.smithi085.stdout:3399: finishing write tid 1 to smithi08575397-6 2021-02-10T04:41:19.443 INFO:tasks.rados.rados.0.smithi085.stdout:3399: finishing write tid 2 to smithi08575397-6 2021-02-10T04:41:19.443 INFO:tasks.rados.rados.0.smithi085.stdout:3399: finishing write tid 3 to smithi08575397-6 2021-02-10T04:41:19.443 INFO:tasks.rados.rados.0.smithi085.stdout:3399: finishing write tid 4 to smithi08575397-6 2021-02-10T04:41:19.443 INFO:tasks.rados.rados.0.smithi085.stdout:3399: finishing write tid 5 to smithi08575397-6 2021-02-10T04:41:19.443 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 1575 (ObjNum 1162 snap 398 seq_num 1162) dirty exists 2021-02-10T04:41:19.443 INFO:tasks.rados.rados.0.smithi085.stdout:3399: left oid 6 (ObjNum 1162 snap 398 seq_num 1162) 2021-02-10T04:41:19.443 INFO:tasks.rados.rados.0.smithi085.stdout:3398: finishing copy_from to smithi08575397-36 2021-02-10T04:41:19.443 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 1350 (ObjNum 1049 snap 344 seq_num 1049) dirty exists 2021-02-10T04:41:19.444 INFO:tasks.rados.rados.0.smithi085.stdout:3398: finishing copy_from racing read to smithi08575397-36 2021-02-10T04:41:19.444 INFO:tasks.rados.rados.0.smithi085.stdout:3401: finishing write tid 1 to smithi08575397-7 2021-02-10T04:41:19.444 INFO:tasks.rados.rados.0.smithi085.stdout:3401: finishing write tid 2 to smithi08575397-7 2021-02-10T04:41:19.444 INFO:tasks.rados.rados.0.smithi085.stdout:3397: done (8 left) 2021-02-10T04:41:19.444 INFO:tasks.rados.rados.0.smithi085.stdout:3398: done (7 left) 2021-02-10T04:41:19.444 INFO:tasks.rados.rados.0.smithi085.stdout:3399: done (6 left) 2021-02-10T04:41:19.444 INFO:tasks.rados.rados.0.smithi085.stdout:3403: snap_create 2021-02-10T04:41:19.444 INFO:tasks.rados.rados.0.smithi085.stdout:3402: finishing copy_from to smithi08575397-31 2021-02-10T04:41:19.445 INFO:tasks.rados.rados.0.smithi085.stdout:3402: got expected ENOENT (src dne) 2021-02-10T04:41:19.445 INFO:tasks.rados.rados.0.smithi085.stdout:3402: finishing copy_from racing read to smithi08575397-31 2021-02-10T04:41:19.446 INFO:tasks.rados.rados.0.smithi085.stdout:3401: finishing write tid 3 to smithi08575397-7 2021-02-10T04:41:19.446 INFO:tasks.rados.rados.0.smithi085.stdout:3401: finishing write tid 4 to smithi08575397-7 2021-02-10T04:41:19.447 INFO:tasks.rados.rados.0.smithi085.stdout:3401: finishing write tid 5 to smithi08575397-7 2021-02-10T04:41:19.447 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 1574 (ObjNum 1163 snap 398 seq_num 1163) dirty exists 2021-02-10T04:41:19.447 INFO:tasks.rados.rados.0.smithi085.stdout:3401: left oid 7 (ObjNum 1163 snap 398 seq_num 1163) 2021-02-10T04:41:19.691 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:41:19 smithi085 conmon[52836]: debug 2021-02-10T04:41:19.658+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:41:19.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:19 smithi164 conmon[29288]: cluster 2021-02-10T04:41:18.412207+0000 mon.a ( 2021-02-10T04:41:19.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:19 smithi164 conmon[29288]: mon.0) 1815 : cluster [DBG] osdmap e859: 8 total, 8 up, 7 in 2021-02-10T04:41:19.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:19 smithi164 conmon[29288]: audit 2021-02-10T04:41:18.448028+0000 mgr.y (mgr.14140) 819 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:41:19.910 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:41:20.427 INFO:tasks.rados.rados.0.smithi085.stdout:3401: done (6 left) 2021-02-10T04:41:20.428 INFO:tasks.rados.rados.0.smithi085.stdout:3402: done (5 left) 2021-02-10T04:41:20.428 INFO:tasks.rados.rados.0.smithi085.stdout:3403: done (4 left) 2021-02-10T04:41:20.428 INFO:tasks.rados.rados.0.smithi085.stdout:3404: write (excl) oid 42 current snap is 399 2021-02-10T04:41:20.428 INFO:tasks.rados.rados.0.smithi085.stdout:3404: seq_num 1164 ranges {763348=591845,1914452=434061,2881893=674270} 2021-02-10T04:41:20.434 INFO:tasks.rados.rados.0.smithi085.stdout:3404: writing smithi08575397-42 from 763348 to 1355193 tid 1 2021-02-10T04:41:20.440 INFO:tasks.rados.rados.0.smithi085.stdout:3404: writing smithi08575397-42 from 1914452 to 2348513 tid 2 2021-02-10T04:41:20.448 INFO:tasks.rados.rados.0.smithi085.stdout:3404: writing smithi08575397-42 from 2881893 to 3556163 tid 3 2021-02-10T04:41:20.448 INFO:tasks.rados.rados.0.smithi085.stdout:3405: copy_from oid 35 from oid 6 current snap is 399 2021-02-10T04:41:20.448 INFO:tasks.rados.rados.0.smithi085.stdout:3404: finishing write tid 1 to smithi08575397-42 2021-02-10T04:41:20.448 INFO:tasks.rados.rados.0.smithi085.stdout:3404: finishing write tid 2 to smithi08575397-42 2021-02-10T04:41:20.449 INFO:tasks.rados.rados.0.smithi085.stdout:3406: snap_remove snap 343 2021-02-10T04:41:20.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[29319]: cluster 2021-02-10T04:41:20.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[29319]: 2021-02-10T04:41:20.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[29319]: 2021 2021-02-10T04:41:20.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[29319]: - 2021-02-10T04:41:20.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[29319]: 02 2021-02-10T04:41:20.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[29319]: - 2021-02-10T04:41:20.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[29319]: 10 2021-02-10T04:41:20.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[29319]: T 2021-02-10T04:41:20.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[29319]: 04 2021-02-10T04:41:20.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[29319]: : 2021-02-10T04:41:20.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[29319]: 41 2021-02-10T04:41:20.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[29319]: : 2021-02-10T04:41:20.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[29319]: 19.420652+0000 mon.a 2021-02-10T04:41:20.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[29319]: (mon.0) 1816 : cluster [DBG] osdmap e860: 8 total, 8 up, 7 in 2021-02-10T04:41:20.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:41:20.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[29319]: 02-10T04:41:20.212081+0000 mgr.y (mgr. 2021-02-10T04:41:20.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[29319]: 14140) 820 : cluster [DBG] pgmap v1441: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim_wait, 158 active+clean, 5 active+clean+snaptrim; 649 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 8.8 MiB/s rd, 5.8 MiB/s wr, 24 op/s 2021-02-10T04:41:20.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: cluster 2021-02-10T04:41:20.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: 2021-02-10T04:41:20.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: 2021 2021-02-10T04:41:20.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: - 2021-02-10T04:41:20.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: 02 2021-02-10T04:41:20.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: - 2021-02-10T04:41:20.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: 10 2021-02-10T04:41:20.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: T 2021-02-10T04:41:20.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: 04 2021-02-10T04:41:20.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: : 2021-02-10T04:41:20.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: 41 2021-02-10T04:41:20.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: : 2021-02-10T04:41:20.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: 19 2021-02-10T04:41:20.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: . 2021-02-10T04:41:20.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: 420652 2021-02-10T04:41:20.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: +0000 2021-02-10T04:41:20.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: 2021-02-10T04:41:20.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: mon.a 2021-02-10T04:41:20.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: ( 2021-02-10T04:41:20.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: mon 2021-02-10T04:41:20.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: . 2021-02-10T04:41:20.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: 0 2021-02-10T04:41:20.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: ) 2021-02-10T04:41:20.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: 1816 2021-02-10T04:41:20.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: : 2021-02-10T04:41:20.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: cluster 2021-02-10T04:41:20.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: 2021-02-10T04:41:20.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: [DBG] 2021-02-10T04:41:20.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: 2021-02-10T04:41:20.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: osdmap e860: 8 total, 8 up, 7 in 2021-02-10T04:41:20.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: 2021-02-10T04:41:20.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: cluster 2021-02-10T04:41:20 2021-02-10T04:41:20.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: .212081+0000 mgr.y (mgr.14140) 820 : cluster [DBG] pgmap v1441: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim_wait, 158 active+clean, 5 active+clean+snaptrim; 649 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 8.8 MiB/s rd, 5.8 MiB/s wr, 24 op/s 2021-02-10T04:41:20.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:20 smithi085 conmon[35961]: 2021-02-10T04:41:20.798 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:20 smithi164 conmon[29288]: cluster 2021-02-10T04:41:20.798 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:20 smithi164 conmon[29288]: 2021-02-10T04:41:20.798 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:20 smithi164 conmon[29288]: 2021 2021-02-10T04:41:20.799 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:20 smithi164 conmon[29288]: - 2021-02-10T04:41:20.799 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:20 smithi164 conmon[29288]: 02-10T04:41 2021-02-10T04:41:20.799 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:20 smithi164 conmon[29288]: :19.420652+0000 mon.a (mon.0) 1816 : cluster [DBG] osdmap e860: 8 total, 8 up, 7 in 2021-02-10T04:41:20.799 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:20 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:41:20.799 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:20 smithi164 conmon[29288]: 02-10T04:41:20.212081+0000 mgr.y (mgr.14140) 2021-02-10T04:41:20.799 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:20 smithi164 conmon[29288]: 820 : cluster 2021-02-10T04:41:20.799 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:20 smithi164 conmon[29288]: [DBG] pgmap v1441: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim_wait, 158 active+clean, 5 active+clean+snaptrim; 649 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 8.8 MiB/s rd, 5.8 MiB/s wr, 24 op/s 2021-02-10T04:41:21.144 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:41:20 smithi164 conmon[37263]: debug 2021-02-10T04:41:21.144 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:41:20 smithi164 conmon[37263]: 2021-02-10T04:41:20.797+0000 7f1c6b1ea700 -1 osd.5 pg_epoch: 861 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55665319a6e0) 2021-02-10T04:41:21.432 INFO:tasks.rados.rados.0.smithi085.stdout:3406: done (6 left) 2021-02-10T04:41:21.433 INFO:tasks.rados.rados.0.smithi085.stdout:3407: write oid 9 current snap is 399 2021-02-10T04:41:21.433 INFO:tasks.rados.rados.0.smithi085.stdout:3404: finishing write tid 3 to smithi08575397-42 2021-02-10T04:41:21.433 INFO:tasks.rados.rados.0.smithi085.stdout:3407: seq_num 1165 ranges {780155=667094,1972934=694072,3309316=270839} 2021-02-10T04:41:21.441 INFO:tasks.rados.rados.0.smithi085.stdout:3407: writing smithi08575397-9 from 780155 to 1447249 tid 1 2021-02-10T04:41:21.449 INFO:tasks.rados.rados.0.smithi085.stdout:3407: writing smithi08575397-9 from 1972934 to 2667006 tid 2 2021-02-10T04:41:21.455 INFO:tasks.rados.rados.0.smithi085.stdout:3407: writing smithi08575397-9 from 3309316 to 3580155 tid 3 2021-02-10T04:41:21.456 INFO:tasks.rados.rados.0.smithi085.stdout:3404: finishing write tid 4 to smithi08575397-42 2021-02-10T04:41:21.456 INFO:tasks.rados.rados.0.smithi085.stdout:3404: finishing write tid 5 to smithi08575397-42 2021-02-10T04:41:21.456 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 1510 (ObjNum 1164 snap 399 seq_num 1164) dirty exists 2021-02-10T04:41:21.456 INFO:tasks.rados.rados.0.smithi085.stdout:3404: left oid 42 (ObjNum 1164 snap 399 seq_num 1164) 2021-02-10T04:41:21.456 INFO:tasks.rados.rados.0.smithi085.stdout:3405: finishing copy_from to smithi08575397-35 2021-02-10T04:41:21.456 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 35 v 1575 (ObjNum 1162 snap 398 seq_num 1162) dirty exists 2021-02-10T04:41:21.456 INFO:tasks.rados.rados.0.smithi085.stdout:3405: finishing copy_from racing read to smithi08575397-35 2021-02-10T04:41:21.457 INFO:tasks.rados.rados.0.smithi085.stdout:3407: finishing write tid 1 to smithi08575397-9 2021-02-10T04:41:21.457 INFO:tasks.rados.rados.0.smithi085.stdout:3407: finishing write tid 2 to smithi08575397-9 2021-02-10T04:41:21.457 INFO:tasks.rados.rados.0.smithi085.stdout:3404: done (6 left) 2021-02-10T04:41:21.457 INFO:tasks.rados.rados.0.smithi085.stdout:3405: done (5 left) 2021-02-10T04:41:21.457 INFO:tasks.rados.rados.0.smithi085.stdout:3408: read oid 48 snap -1 2021-02-10T04:41:21.457 INFO:tasks.rados.rados.0.smithi085.stdout:3408: expect (ObjNum 1142 snap 385 seq_num 1142) 2021-02-10T04:41:21.457 INFO:tasks.rados.rados.0.smithi085.stdout:3409: rollback oid 42 current snap is 399 2021-02-10T04:41:21.457 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 42 to 325 2021-02-10T04:41:21.458 INFO:tasks.rados.rados.0.smithi085.stdout:3410: rollback oid 16 current snap is 399 2021-02-10T04:41:21.458 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 16 to 325 2021-02-10T04:41:21.458 INFO:tasks.rados.rados.0.smithi085.stdout:3411: read oid 10 snap -1 2021-02-10T04:41:21.458 INFO:tasks.rados.rados.0.smithi085.stdout:3411: expect (ObjNum 966 snap 315 seq_num 966) 2021-02-10T04:41:21.458 INFO:tasks.rados.rados.0.smithi085.stdout:3412: read oid 29 snap -1 2021-02-10T04:41:21.458 INFO:tasks.rados.rados.0.smithi085.stdout:3412: expect deleted 2021-02-10T04:41:21.458 INFO:tasks.rados.rados.0.smithi085.stdout:3413: rollback oid 7 current snap is 399 2021-02-10T04:41:21.458 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 7 to 359 2021-02-10T04:41:21.458 INFO:tasks.rados.rados.0.smithi085.stdout:3414: snap_create 2021-02-10T04:41:21.460 INFO:tasks.rados.rados.0.smithi085.stdout:3411: expect (ObjNum 966 snap 315 seq_num 966) 2021-02-10T04:41:21.665 INFO:tasks.rados.rados.0.smithi085.stdout:3410: finishing rollback tid 1 to smithi08575397-16 2021-02-10T04:41:21.666 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 1635 (ObjNum 83335616 snap 22030 seq_num 83122816) dirty dne 2021-02-10T04:41:21.666 INFO:tasks.rados.rados.0.smithi085.stdout:3413: finishing rollback tid 0 to smithi08575397-7 2021-02-10T04:41:21.666 INFO:tasks.rados.rados.0.smithi085.stdout:3407: finishing write tid 3 to smithi08575397-9 2021-02-10T04:41:21.666 INFO:tasks.rados.rados.0.smithi085.stdout:3407: finishing write tid 4 to smithi08575397-9 2021-02-10T04:41:21.666 INFO:tasks.rados.rados.0.smithi085.stdout:3407: finishing write tid 5 to smithi08575397-9 2021-02-10T04:41:21.666 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 9 v 1514 (ObjNum 1165 snap 399 seq_num 1165) dirty exists 2021-02-10T04:41:21.666 INFO:tasks.rados.rados.0.smithi085.stdout:3407: left oid 9 (ObjNum 1165 snap 399 seq_num 1165) 2021-02-10T04:41:21.666 INFO:tasks.rados.rados.0.smithi085.stdout:3413: finishing rollback tid 1 to smithi08575397-7 2021-02-10T04:41:21.666 INFO:tasks.rados.rados.0.smithi085.stdout:3408: expect (ObjNum 1142 snap 385 seq_num 1142) 2021-02-10T04:41:21.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:21 smithi085 conmon[29319]: cluster 2021-02-10T04:41:20.425094+0000 mon.a ( 2021-02-10T04:41:21.685 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:21 smithi085 conmon[29319]: mon.0) 1817 : cluster [DBG] osdmap e861: 8 total, 8 up, 7 in 2021-02-10T04:41:21.685 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:21 smithi085 conmon[35961]: cluster 2021-02-10T04:41:20.425094+0000 2021-02-10T04:41:21.685 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:21 smithi085 conmon[35961]: mon.a (mon.0) 1817 : cluster [DBG] osdmap e861: 8 total, 8 up, 7 in 2021-02-10T04:41:21.744 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:41:21.774 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.2 dump_historic_ops 2021-02-10T04:41:21.792 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:41:21.869 INFO:tasks.rados.rados.0.smithi085.stdout:3413: finishing rollback tid 2 to smithi08575397-7 2021-02-10T04:41:21.870 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 1577 (ObjNum 1045 snap 341 seq_num 1045) dirty exists 2021-02-10T04:41:21.870 INFO:tasks.rados.rados.0.smithi085.stdout:3409: finishing rollback tid 0 to smithi08575397-42 2021-02-10T04:41:21.870 INFO:tasks.rados.rados.0.smithi085.stdout:3409: finishing rollback tid 1 to smithi08575397-42 2021-02-10T04:41:21.870 INFO:tasks.rados.rados.0.smithi085.stdout:3409: finishing rollback tid 2 to smithi08575397-42 2021-02-10T04:41:21.870 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 1517 (ObjNum 981 snap 317 seq_num 981) dirty exists 2021-02-10T04:41:21.873 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 5 1 2021-02-10T04:41:21.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:21 smithi164 conmon[29288]: cluster 2021-02-10T04:41:20.425094+0000 mon.a (mon.0) 2021-02-10T04:41:21.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:21 smithi164 conmon[29288]: 1817 : cluster [DBG] osdmap e861: 8 total, 8 up, 7 in 2021-02-10T04:41:22.020 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:41:22.106 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:41:22.127 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:41:22.161 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:41:22.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:41:21 smithi085 conmon[52836]: debug 2021-02-10T04:41:22.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:41:21 smithi085 conmon[52836]: 2021-02-10T04:41:21.948+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:41:22.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:41:21 smithi085 conmon[52836]: 2021-02-10T04:41:22.206 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:41:22.434 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:22 smithi164 conmon[29288]: cluster 2021-02-10T04:41:21.432039+0000 mon.a (mon.0 2021-02-10T04:41:22.435 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:22 smithi164 conmon[29288]: ) 1818 : cluster [DBG] osdmap e862: 8 total, 8 up, 7 in 2021-02-10T04:41:22.435 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:22 smithi164 conmon[29288]: cluster 2021-02-10T04:41:22.212785+0000 mgr.y (mgr.14140) 821 : cluster [DBG] pgmap v1444: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim_wait, 158 active+clean, 5 active+clean+snaptrim; 649 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 2.9 MiB/s rd, 4.7 MiB/s wr, 17 op/s 2021-02-10T04:41:22.435 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:41:22 smithi164 conmon[44544]: debug 2021-02-10T04:41:22.327+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:41:22.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:22 smithi085 conmon[35961]: cluster 2021-02-10T04:41:21.432039+0000 mon.a ( 2021-02-10T04:41:22.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:22 smithi085 conmon[35961]: mon.0) 1818 : cluster [DBG] osdmap e862: 8 total, 8 up, 7 in 2021-02-10T04:41:22.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:22 smithi085 conmon[35961]: cluster 2021-02-10T04:41:22. 2021-02-10T04:41:22.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:22 smithi085 conmon[35961]: 212785+0000 mgr.y (mgr.14140) 821 : cluster [DBG] pgmap v1444: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim_wait, 158 active+clean, 5 active+clean+snaptrim; 649 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 2.9 MiB/s rd, 4.7 MiB/s wr, 17 op/s 2021-02-10T04:41:22.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:22 smithi085 conmon[29319]: cluster 2021-02-10T04:41:21.432039+0000 mon.a 2021-02-10T04:41:22.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:22 smithi085 conmon[29319]: (mon.0) 1818 : cluster [DBG] osdmap e862: 8 total, 8 up, 7 in 2021-02-10T04:41:22.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:22 smithi085 conmon[29319]: cluster 2021-02-10T04:41:22.212785+0000 mgr.y (mgr.14140) 821 : cluster [DBG] pgmap v1444: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim_wait, 158 active+clean, 5 active+clean+snaptrim; 649 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 2.9 MiB/s rd, 4.7 MiB/s wr, 17 op/s 2021-02-10T04:41:22.445 INFO:tasks.rados.rados.0.smithi085.stdout:3407: done (11 left) 2021-02-10T04:41:22.445 INFO:tasks.rados.rados.0.smithi085.stdout:3408: done (10 left) 2021-02-10T04:41:22.445 INFO:tasks.rados.rados.0.smithi085.stdout:3409: done (9 left) 2021-02-10T04:41:22.445 INFO:tasks.rados.rados.0.smithi085.stdout:3410: done (8 left) 2021-02-10T04:41:22.446 INFO:tasks.rados.rados.0.smithi085.stdout:3411: done (7 left) 2021-02-10T04:41:22.446 INFO:tasks.rados.rados.0.smithi085.stdout:3412: done (6 left) 2021-02-10T04:41:22.446 INFO:tasks.rados.rados.0.smithi085.stdout:3413: done (5 left) 2021-02-10T04:41:22.446 INFO:tasks.rados.rados.0.smithi085.stdout:3414: done (4 left) 2021-02-10T04:41:22.446 INFO:tasks.rados.rados.0.smithi085.stdout:3415: read oid 7 snap -1 2021-02-10T04:41:22.446 INFO:tasks.rados.rados.0.smithi085.stdout:3415: expect (ObjNum 1045 snap 341 seq_num 1045) 2021-02-10T04:41:22.449 INFO:tasks.rados.rados.0.smithi085.stdout:3416: write oid 32 current snap is 400 2021-02-10T04:41:22.449 INFO:tasks.rados.rados.0.smithi085.stdout:3416: seq_num 1166 ranges {796962=742343,2031416=437730,2936739=413167} 2021-02-10T04:41:22.453 INFO:tasks.rados.rados.0.smithi085.stdout:3416: writing smithi08575397-32 from 796962 to 1539305 tid 1 2021-02-10T04:41:22.459 INFO:tasks.rados.rados.0.smithi085.stdout:3416: writing smithi08575397-32 from 2031416 to 2469146 tid 2 2021-02-10T04:41:22.465 INFO:tasks.rados.rados.0.smithi085.stdout:3416: writing smithi08575397-32 from 2936739 to 3349906 tid 3 2021-02-10T04:41:22.466 INFO:tasks.rados.rados.0.smithi085.stdout:3417: copy_from oid 8 from oid 6 current snap is 400 2021-02-10T04:41:22.466 INFO:tasks.rados.rados.0.smithi085.stdout:3415: expect (ObjNum 1045 snap 341 seq_num 1045) 2021-02-10T04:41:22.664 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:41:22.762 INFO:tasks.rados.rados.0.smithi085.stdout:3416: finishing write tid 1 to smithi08575397-32 2021-02-10T04:41:22.762 INFO:tasks.rados.rados.0.smithi085.stdout:3416: finishing write tid 2 to smithi08575397-32 2021-02-10T04:41:22.762 INFO:tasks.rados.rados.0.smithi085.stdout:3416: finishing write tid 3 to smithi08575397-32 2021-02-10T04:41:22.762 INFO:tasks.rados.rados.0.smithi085.stdout:3415: done (6 left) 2021-02-10T04:41:22.762 INFO:tasks.rados.rados.0.smithi085.stdout:3418: rollback oid 48 current snap is 400 2021-02-10T04:41:22.762 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 48 to 393 2021-02-10T04:41:22.762 INFO:tasks.rados.rados.0.smithi085.stdout:3416: finishing write tid 4 to smithi08575397-32 2021-02-10T04:41:22.762 INFO:tasks.rados.rados.0.smithi085.stdout:3416: finishing write tid 5 to smithi08575397-32 2021-02-10T04:41:22.762 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 32 v 1468 (ObjNum 1166 snap 400 seq_num 1166) dirty exists 2021-02-10T04:41:22.762 INFO:tasks.rados.rados.0.smithi085.stdout:3416: left oid 32 (ObjNum 1166 snap 400 seq_num 1166) 2021-02-10T04:41:22.763 INFO:tasks.rados.rados.0.smithi085.stdout:3417: finishing copy_from to smithi08575397-8 2021-02-10T04:41:22.763 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 1575 (ObjNum 1162 snap 398 seq_num 1162) dirty exists 2021-02-10T04:41:22.763 INFO:tasks.rados.rados.0.smithi085.stdout:3417: finishing copy_from racing read to smithi08575397-8 2021-02-10T04:41:22.763 INFO:tasks.rados.rados.0.smithi085.stdout:3416: done (6 left) 2021-02-10T04:41:22.763 INFO:tasks.rados.rados.0.smithi085.stdout:3417: done (5 left) 2021-02-10T04:41:22.763 INFO:tasks.rados.rados.0.smithi085.stdout:3419: read oid 7 snap 369 2021-02-10T04:41:22.763 INFO:tasks.rados.rados.0.smithi085.stdout:3419: expect (ObjNum 1054 snap 346 seq_num 1054) 2021-02-10T04:41:22.763 INFO:tasks.rados.rados.0.smithi085.stdout:3420: snap_remove snap 359 2021-02-10T04:41:22.770 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:41:22.802 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:41:23.113 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:41:23.118 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:41:22 smithi164 conmon[44544]: debug 2021-02-10T04:41:22.932+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:41:23.220 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:41:23.250 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:41:23.292 INFO:tasks.rados.rados.0.smithi085.stdout:3420: done (6 left) 2021-02-10T04:41:23.292 INFO:tasks.rados.rados.0.smithi085.stdout:3421: copy_from oid 14 from oid 22 current snap is 400 2021-02-10T04:41:23.292 INFO:tasks.rados.rados.0.smithi085.stdout:3422: read oid 43 snap -1 2021-02-10T04:41:23.292 INFO:tasks.rados.rados.0.smithi085.stdout:3422: expect (ObjNum 1150 snap 392 seq_num 1150) 2021-02-10T04:41:23.292 INFO:tasks.rados.rados.0.smithi085.stdout:3423: snap_create 2021-02-10T04:41:23.297 INFO:tasks.rados.rados.0.smithi085.stdout:3418: finishing rollback tid 0 to smithi08575397-48 2021-02-10T04:41:23.297 INFO:tasks.rados.rados.0.smithi085.stdout:3418: finishing rollback tid 1 to smithi08575397-48 2021-02-10T04:41:23.297 INFO:tasks.rados.rados.0.smithi085.stdout:3418: finishing rollback tid 2 to smithi08575397-48 2021-02-10T04:41:23.297 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 1520 (ObjNum 1142 snap 385 seq_num 1142) dirty exists 2021-02-10T04:41:23.297 INFO:tasks.rados.rados.0.smithi085.stdout:3419: expect (ObjNum 1054 snap 346 seq_num 1054) 2021-02-10T04:41:23.393 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:23 smithi164 conmon[33677]: debug 2021-02-10T04:41:23.331+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:41:23.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:23 smithi085 conmon[35961]: cluster 2021-02-10T04:41:22.437976+0000 2021-02-10T04:41:23.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:23 smithi085 conmon[35961]: mon.a (mon.0) 1819 : cluster [DBG] osdmap e863: 8 total, 8 up, 7 in 2021-02-10T04:41:23.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:23 smithi085 conmon[35961]: cluster 2021-02-10T04:41:23.290594+0000 mon.a (mon.0) 1820 : cluster [DBG] osdmap e864: 8 total, 8 up, 7 in 2021-02-10T04:41:23.437 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:23 smithi085 conmon[29319]: cluster 2021-02-10T04:41:23.437 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:23 smithi085 conmon[29319]: 2021-02-10T04:41:23.437 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:23 smithi085 conmon[29319]: 2021 2021-02-10T04:41:23.437 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:23 smithi085 conmon[29319]: - 2021-02-10T04:41:23.437 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:23 smithi085 conmon[29319]: 02-10T04:41:22.437976 2021-02-10T04:41:23.437 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:23 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1819 : cluster [DBG] osdmap e863: 8 total, 8 up, 7 in 2021-02-10T04:41:23.437 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:23 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:41:23.437 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:23 smithi085 conmon[29319]: -10T04:41:23.290594+0000 mon.a (mon.0) 1820 : 2021-02-10T04:41:23.437 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:23 smithi085 conmon[29319]: cluster [DBG] osdmap e864: 8 total, 8 up, 7 in 2021-02-10T04:41:23.521 INFO:tasks.rados.rados.0.smithi085.stdout:3422: expect (ObjNum 1150 snap 392 seq_num 1150) 2021-02-10T04:41:23.659 INFO:tasks.rados.rados.0.smithi085.stdout:3421: finishing copy_from to smithi08575397-14 2021-02-10T04:41:23.659 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 1472 (ObjNum 1108 snap 374 seq_num 1108) dirty exists 2021-02-10T04:41:23.659 INFO:tasks.rados.rados.0.smithi085.stdout:3421: finishing copy_from racing read to smithi08575397-14 2021-02-10T04:41:23.742 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:41:23.748 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:23 smithi164 conmon[29288]: cluster 2021-02-10T04:41:23.748 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:23 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:41:23.748 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:23 smithi164 conmon[29288]: 41:22.437976+0000 mon.a (mon.0) 1819 : cluster [DBG] osdmap e863: 8 total, 8 up, 7 in 2021-02-10T04:41:23.748 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:23 smithi164 conmon[29288]: cluster 2021-02-10T04:41:23.748 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:23 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:41:23.748 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:23 smithi164 conmon[29288]: 41:23.290594+0000 mon.a (mon.0) 2021-02-10T04:41:23.748 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:23 smithi164 conmon[29288]: 1820 : cluster [DBG] osdmap e864: 8 total, 8 up, 7 in 2021-02-10T04:41:23.848 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:41:23.888 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:41:24.297 INFO:tasks.rados.rados.0.smithi085.stdout:3418: done (8 left) 2021-02-10T04:41:24.297 INFO:tasks.rados.rados.0.smithi085.stdout:3419: done (7 left) 2021-02-10T04:41:24.297 INFO:tasks.rados.rados.0.smithi085.stdout:3421: done (6 left) 2021-02-10T04:41:24.297 INFO:tasks.rados.rados.0.smithi085.stdout:3422: done (5 left) 2021-02-10T04:41:24.297 INFO:tasks.rados.rados.0.smithi085.stdout:3423: done (4 left) 2021-02-10T04:41:24.297 INFO:tasks.rados.rados.0.smithi085.stdout:3424: copy_from oid 50 from oid 44 current snap is 401 2021-02-10T04:41:24.303 INFO:tasks.rados.rados.0.smithi085.stdout:3425: write (excl) oid 10 current snap is 401 2021-02-10T04:41:24.303 INFO:tasks.rados.rados.0.smithi085.stdout:3425: seq_num 1167 ranges {413769=417592,1573545=697741,2847809=624439} 2021-02-10T04:41:24.303 INFO:tasks.rados.rados.0.smithi085.stdout:3425: writing smithi08575397-10 from 413769 to 831361 tid 1 2021-02-10T04:41:24.310 INFO:tasks.rados.rados.0.smithi085.stdout:3425: writing smithi08575397-10 from 1573545 to 2271286 tid 2 2021-02-10T04:41:24.317 INFO:tasks.rados.rados.0.smithi085.stdout:3425: writing smithi08575397-10 from 2847809 to 3472248 tid 3 2021-02-10T04:41:24.318 INFO:tasks.rados.rados.0.smithi085.stdout:3425: finishing write tid 1 to smithi08575397-10 2021-02-10T04:41:24.318 INFO:tasks.rados.rados.0.smithi085.stdout:3425: finishing write tid 2 to smithi08575397-10 2021-02-10T04:41:24.318 INFO:tasks.rados.rados.0.smithi085.stdout:3426: delete oid 3 current snap is 401 2021-02-10T04:41:24.323 INFO:tasks.rados.rados.0.smithi085.stdout:3426: done (6 left) 2021-02-10T04:41:24.323 INFO:tasks.rados.rados.0.smithi085.stdout:3427: rollback oid 31 current snap is 401 2021-02-10T04:41:24.323 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 31 to 396 2021-02-10T04:41:24.323 INFO:tasks.rados.rados.0.smithi085.stdout:3428: delete oid 28 current snap is 401 2021-02-10T04:41:24.324 INFO:tasks.rados.rados.0.smithi085.stdout:3425: finishing write tid 3 to smithi08575397-10 2021-02-10T04:41:24.325 INFO:tasks.rados.rados.0.smithi085.stdout:3425: finishing write tid 4 to smithi08575397-10 2021-02-10T04:41:24.326 INFO:tasks.rados.rados.0.smithi085.stdout:3425: finishing write tid 5 to smithi08575397-10 2021-02-10T04:41:24.326 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 10 v 1579 (ObjNum 1167 snap 401 seq_num 1167) dirty exists 2021-02-10T04:41:24.326 INFO:tasks.rados.rados.0.smithi085.stdout:3425: left oid 10 (ObjNum 1167 snap 401 seq_num 1167) 2021-02-10T04:41:24.328 INFO:tasks.rados.rados.0.smithi085.stdout:3427: finishing rollback tid 0 to smithi08575397-31 2021-02-10T04:41:24.331 INFO:tasks.rados.rados.0.smithi085.stdout:3427: finishing rollback tid 1 to smithi08575397-31 2021-02-10T04:41:24.331 INFO:tasks.rados.rados.0.smithi085.stdout:3427: finishing rollback tid 2 to smithi08575397-31 2021-02-10T04:41:24.331 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 1483 (ObjNum 1097 snap 368 seq_num 1097) dirty exists 2021-02-10T04:41:24.332 INFO:tasks.rados.rados.0.smithi085.stdout:3425: done (7 left) 2021-02-10T04:41:24.332 INFO:tasks.rados.rados.0.smithi085.stdout:3427: done (6 left) 2021-02-10T04:41:24.332 INFO:tasks.rados.rados.0.smithi085.stdout:3428: done (5 left) 2021-02-10T04:41:24.332 INFO:tasks.rados.rados.0.smithi085.stdout:3429: snap_remove snap 377 2021-02-10T04:41:24.626 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:41:24.671 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:24 smithi085 conmon[35961]: cluster 2021-02-10T04:41 2021-02-10T04:41:24.671 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:24 smithi085 conmon[35961]: :23.275522+0000 osd.6 (osd.6) 82 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:41:24.671 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:24 smithi085 conmon[35961]: cluster 2021-02-10T04:41:24.672 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:24 smithi085 conmon[35961]: 2021-02-10T04:41 2021-02-10T04:41:24.672 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:24 smithi085 conmon[35961]: :23.276897+0000 osd.6 (osd.6) 83 : 2021-02-10T04:41:24.672 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:24 smithi085 conmon[35961]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:41:24.672 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:24 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:41:24.672 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:24 smithi085 conmon[35961]: -10T04:41:24.213413+0000 mgr.y ( 2021-02-10T04:41:24.672 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:24 smithi085 conmon[35961]: mgr.14140) 822 : cluster [DBG] pgmap v1447: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 661 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 10 op/s 2021-02-10T04:41:24.672 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:24 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:41:24.672 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:24 smithi085 conmon[35961]: -10T04:41:24.297546+0000 mon.a 2021-02-10T04:41:24.672 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:24 smithi085 conmon[35961]: (mon.0) 1821 : cluster [DBG] osdmap e865: 8 total, 8 up, 7 in 2021-02-10T04:41:24.673 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:24 smithi085 conmon[29319]: cluster 2021-02-10T04:41:24.673 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:24 smithi085 conmon[29319]: 2021- 2021-02-10T04:41:24.673 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:24 smithi085 conmon[29319]: 02-10T04:41:23.275522+0000 osd.6 (osd.6) 82 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:41:24.673 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:24 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:41:24.673 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:24 smithi085 conmon[29319]: -10T04:41:23.276897+0000 osd.6 (osd.6) 83 : 2021-02-10T04:41:24.673 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:24 smithi085 conmon[29319]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:41:24.673 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:24 smithi085 conmon[29319]: cluster 2021-02-10T04:41:24.673 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:24 smithi085 conmon[29319]: 2021-02-10T04:41:24.213413+0000 2021-02-10T04:41:24.673 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:24 smithi085 conmon[29319]: mgr.y (mgr.14140) 822 : cluster [DBG] pgmap v1447: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 661 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 10 op/s 2021-02-10T04:41:24.673 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:24 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:41:24.674 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:24 smithi085 conmon[29319]: 10T04:41:24.297546+0000 mon.a (mon 2021-02-10T04:41:24.674 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:24 smithi085 conmon[29319]: .0) 1821 : cluster [DBG] osdmap e865: 8 total, 8 up, 7 in 2021-02-10T04:41:24.674 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:41:24 smithi085 conmon[45443]: debug 2021-02-10T04:41:24.272+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:41:24.778 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:24 smithi164 conmon[29288]: cluster 2021-02-10T04:41:23.275522+0000 osd.6 ( 2021-02-10T04:41:24.779 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:24 smithi164 conmon[29288]: osd.6) 82 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:41:24.779 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:24 smithi164 conmon[29288]: cluster 2021-02-10T04:41:23.276897+0000 osd.6 (osd.6) 83 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:41:24.779 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:24 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:41:24.779 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:24 smithi164 conmon[29288]: 10T04:41:24. 2021-02-10T04:41:24.779 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:24 smithi164 conmon[29288]: 213413+0000 mgr.y (mgr.14140) 2021-02-10T04:41:24.779 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:24 smithi164 conmon[29288]: 822 : cluster [DBG] pgmap v1447: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 661 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 10 op/s 2021-02-10T04:41:24.779 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:24 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:41:24.779 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:24 smithi164 conmon[29288]: 02-10T04:41:24.297546 2021-02-10T04:41:24.779 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:24 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1821 : 2021-02-10T04:41:24.779 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:24 smithi164 conmon[29288]: cluster [DBG] osdmap e865: 8 total, 8 up, 7 in 2021-02-10T04:41:25.143 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:41:24 smithi164 conmon[37263]: debug 2021-02-10T04:41:25.144 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:41:24 smithi164 conmon[37263]: 2021-02-10T04:41:24.777+0000 7f1c6b1ea700 -1 osd.5 pg_epoch: 865 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x5566533aa9a0) 2021-02-10T04:41:25.305 INFO:tasks.rados.rados.0.smithi085.stdout:3424: finishing copy_from to smithi08575397-50 2021-02-10T04:41:25.305 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 1569 (ObjNum 1155 snap 396 seq_num 1155) dirty exists 2021-02-10T04:41:25.305 INFO:tasks.rados.rados.0.smithi085.stdout:3424: finishing copy_from racing read to smithi08575397-50 2021-02-10T04:41:25.305 INFO:tasks.rados.rados.0.smithi085.stdout:3424: done (5 left) 2021-02-10T04:41:25.306 INFO:tasks.rados.rados.0.smithi085.stdout:3429: done (4 left) 2021-02-10T04:41:25.306 INFO:tasks.rados.rados.0.smithi085.stdout:3430: read oid 3 snap -1 2021-02-10T04:41:25.306 INFO:tasks.rados.rados.0.smithi085.stdout:3430: expect deleted 2021-02-10T04:41:25.306 INFO:tasks.rados.rados.0.smithi085.stdout:3431: delete oid 15 current snap is 401 2021-02-10T04:41:25.309 INFO:tasks.rados.rados.0.smithi085.stdout:3431: done (5 left) 2021-02-10T04:41:25.309 INFO:tasks.rados.rados.0.smithi085.stdout:3432: read oid 21 snap 390 2021-02-10T04:41:25.310 INFO:tasks.rados.rados.0.smithi085.stdout:3432: expect (ObjNum 1134 snap 381 seq_num 1134) 2021-02-10T04:41:25.310 INFO:tasks.rados.rados.0.smithi085.stdout:3433: read oid 36 snap -1 2021-02-10T04:41:25.310 INFO:tasks.rados.rados.0.smithi085.stdout:3433: expect (ObjNum 1049 snap 344 seq_num 1049) 2021-02-10T04:41:25.310 INFO:tasks.rados.rados.0.smithi085.stdout:3434: snap_remove snap 375 2021-02-10T04:41:26.306 INFO:tasks.rados.rados.0.smithi085.stdout:3434: done (7 left) 2021-02-10T04:41:26.306 INFO:tasks.rados.rados.0.smithi085.stdout:3435: snap_remove snap 373 2021-02-10T04:41:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:26 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:41:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:26 smithi164 conmon[29288]: 41:25.301937+0000 mon.a (mon.0) 1822 : cluster [DBG] osdmap e866: 8 total, 8 up, 7 in 2021-02-10T04:41:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:26 smithi164 conmon[29288]: cluster 2021-02-10T04:41:26.213956+0000 mgr.y 2021-02-10T04:41:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:26 smithi164 conmon[29288]: (mgr.14140) 823 : cluster [DBG] pgmap v1450: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 670 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 3.3 MiB/s wr, 21 op/s 2021-02-10T04:41:26.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:26 smithi085 conmon[29319]: cluster 2021-02-10T04:41:25.301937+0000 mon.a ( 2021-02-10T04:41:26.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:26 smithi085 conmon[29319]: mon.0) 1822 : cluster [DBG] osdmap e866: 8 total, 8 up, 7 in 2021-02-10T04:41:26.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:26 smithi085 conmon[29319]: cluster 2021-02-10T04:41:26.213956+0000 mgr.y (mgr.14140) 823 : cluster [DBG] pgmap v1450: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 670 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 3.3 MiB/s wr, 21 op/s 2021-02-10T04:41:26.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:26 smithi085 conmon[35961]: cluster 2021-02-10T04:41:25.301937 2021-02-10T04:41:26.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:26 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1822 : cluster [DBG] osdmap e866: 8 total, 8 up, 7 in 2021-02-10T04:41:26.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:26 smithi085 conmon[35961]: cluster 2021-02-10T04:41:26.213956+0000 mgr.y (mgr.14140) 823 : cluster [DBG] pgmap v1450: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 670 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 3.3 MiB/s wr, 21 op/s 2021-02-10T04:41:27.143 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:41:26 smithi164 conmon[37263]: debug 2021-02-10T04:41:26.825+0000 7f1c6b1ea700 -1 osd.5 pg_epoch: 867 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x556655936f20) 2021-02-10T04:41:27.314 INFO:tasks.rados.rados.0.smithi085.stdout:3432: expect (ObjNum 1134 snap 381 seq_num 1134) 2021-02-10T04:41:27.498 INFO:tasks.rados.rados.0.smithi085.stdout:3432: done (7 left) 2021-02-10T04:41:27.498 INFO:tasks.rados.rados.0.smithi085.stdout:3435: done (6 left) 2021-02-10T04:41:27.498 INFO:tasks.rados.rados.0.smithi085.stdout:3436: snap_remove snap 398 2021-02-10T04:41:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:27 smithi164 conmon[29288]: cluster 2021-02-10T04:41:26.305657+0000 mon.a 2021-02-10T04:41:27.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:27 smithi164 conmon[29288]: (mon.0) 1823 : cluster [DBG] osdmap e867: 8 total, 8 up, 7 in 2021-02-10T04:41:27.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[35961]: cluster 2021-02-10T04:41:26 2021-02-10T04:41:27.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[35961]: .305657+0000 mon.a (mon.0) 1823 : cluster [DBG] osdmap e867: 8 total, 8 up, 7 in 2021-02-10T04:41:27.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: cluster 2021-02-10T04:41:27.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: 2021-02-10T04:41:27.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: 2021 2021-02-10T04:41:27.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: - 2021-02-10T04:41:27.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: 02 2021-02-10T04:41:27.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: - 2021-02-10T04:41:27.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: 10 2021-02-10T04:41:27.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: T 2021-02-10T04:41:27.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: 04 2021-02-10T04:41:27.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: : 2021-02-10T04:41:27.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: 41 2021-02-10T04:41:27.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: : 2021-02-10T04:41:27.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: 26 2021-02-10T04:41:27.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: . 2021-02-10T04:41:27.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: 305657 2021-02-10T04:41:27.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: +0000 2021-02-10T04:41:27.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: 2021-02-10T04:41:27.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: mon.a 2021-02-10T04:41:27.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: ( 2021-02-10T04:41:27.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: mon 2021-02-10T04:41:27.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: . 2021-02-10T04:41:27.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: 0 2021-02-10T04:41:27.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: ) 2021-02-10T04:41:27.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: 1823 2021-02-10T04:41:27.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: : 2021-02-10T04:41:27.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: cluster 2021-02-10T04:41:27.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: 2021-02-10T04:41:27.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: [DBG] 2021-02-10T04:41:27.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: 2021-02-10T04:41:27.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: osdmap e867: 8 total, 8 up, 7 in 2021-02-10T04:41:27.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:27 smithi085 conmon[29319]: 2021-02-10T04:41:28.289 INFO:tasks.rados.rados.0.smithi085.stdout:3436: done (6 left) 2021-02-10T04:41:28.289 INFO:tasks.rados.rados.0.smithi085.stdout:3437: write (excl) oid 26 current snap is 401 2021-02-10T04:41:28.289 INFO:tasks.rados.rados.0.smithi085.stdout:3437: seq_num 1171 ranges {480997=434941,1407473=705079,2557501=785880} 2021-02-10T04:41:28.295 INFO:tasks.rados.rados.0.smithi085.stdout:3437: writing smithi08575397-26 from 480997 to 915938 tid 1 2021-02-10T04:41:28.302 INFO:tasks.rados.rados.0.smithi085.stdout:3437: writing smithi08575397-26 from 1407473 to 2112552 tid 2 2021-02-10T04:41:28.310 INFO:tasks.rados.rados.0.smithi085.stdout:3437: writing smithi08575397-26 from 2557501 to 3343381 tid 3 2021-02-10T04:41:28.311 INFO:tasks.rados.rados.0.smithi085.stdout:3438: rollback oid 8 current snap is 401 2021-02-10T04:41:28.311 INFO:tasks.rados.rados.0.smithi085.stdout:3433: expect (ObjNum 1049 snap 344 seq_num 1049) 2021-02-10T04:41:28.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:28 smithi085 conmon[29319]: cluster 2021-02-10T04:41:28.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:28 smithi085 conmon[29319]: 2021-02-10T04:41:28.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:28 smithi085 conmon[29319]: 2021 2021-02-10T04:41:28.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:28 smithi085 conmon[29319]: - 2021-02-10T04:41:28.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:28 smithi085 conmon[29319]: 02-10T04:41:27.312778 2021-02-10T04:41:28.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:28 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1824 : cluster [DBG] osdmap e868: 8 total, 8 up, 7 in 2021-02-10T04:41:28.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:28 smithi085 conmon[29319]: cluster 2021-02-10T04:41:28.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:28 smithi085 conmon[29319]: 2021-02-10T04:41:28.214632+0000 mgr.y 2021-02-10T04:41:28.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:28 smithi085 conmon[29319]: (mgr.14140) 824 : cluster [DBG] 2021-02-10T04:41:28.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:28 smithi085 conmon[29319]: pgmap v1453: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 669 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 1.9 MiB/s wr, 12 op/s 2021-02-10T04:41:28.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:28 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:41:28.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:28 smithi085 conmon[29319]: :41:28.288091+0000 mon.a (mon 2021-02-10T04:41:28.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:28 smithi085 conmon[29319]: .0) 1825 : cluster [DBG] osdmap e869: 8 total, 8 up, 7 in 2021-02-10T04:41:28.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:28 smithi085 conmon[35961]: cluster 2021-02-10T04:41:27.312778+0000 2021-02-10T04:41:28.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:28 smithi085 conmon[35961]: mon.a (mon.0) 1824 : cluster [DBG] osdmap e868: 8 total, 8 up, 7 in 2021-02-10T04:41:28.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:28 smithi085 conmon[35961]: cluster 2021-02-10T04:41:28.214632+0000 mgr.y (mgr. 2021-02-10T04:41:28.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:28 smithi085 conmon[35961]: 14140) 824 : cluster [DBG] pgmap v1453: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 669 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 1.9 MiB/s wr, 12 op/s 2021-02-10T04:41:28.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:28 smithi085 conmon[35961]: cluster 2021-02-10T04:41:28.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:28 smithi085 conmon[35961]: 2021-02-10T04:41:28.288091+0000 mon.a (mon. 2021-02-10T04:41:28.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:28 smithi085 conmon[35961]: 0) 1825 : cluster [DBG] osdmap e869: 8 total, 8 up, 7 in 2021-02-10T04:41:28.460 INFO:tasks.rados.rados.0.smithi085.stdout:3437: finishing write tid 1 to smithi08575397-26 2021-02-10T04:41:28.461 INFO:tasks.rados.rados.0.smithi085.stdout:3437: finishing write tid 2 to smithi08575397-26 2021-02-10T04:41:28.461 INFO:tasks.rados.rados.0.smithi085.stdout:3437: finishing write tid 3 to smithi08575397-26 2021-02-10T04:41:28.461 INFO:tasks.rados.rados.0.smithi085.stdout:3437: finishing write tid 4 to smithi08575397-26 2021-02-10T04:41:28.461 INFO:tasks.rados.rados.0.smithi085.stdout:3437: finishing write tid 5 to smithi08575397-26 2021-02-10T04:41:28.461 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 1363 (ObjNum 1171 snap 401 seq_num 1171) dirty exists 2021-02-10T04:41:28.461 INFO:tasks.rados.rados.0.smithi085.stdout:3437: left oid 26 (ObjNum 1171 snap 401 seq_num 1171) 2021-02-10T04:41:28.461 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 8 to 347 2021-02-10T04:41:28.461 INFO:tasks.rados.rados.0.smithi085.stdout:3430: done (7 left) 2021-02-10T04:41:28.461 INFO:tasks.rados.rados.0.smithi085.stdout:3433: done (6 left) 2021-02-10T04:41:28.461 INFO:tasks.rados.rados.0.smithi085.stdout:3437: done (5 left) 2021-02-10T04:41:28.461 INFO:tasks.rados.rados.0.smithi085.stdout:3439: write (excl) oid 2 current snap is 401 2021-02-10T04:41:28.461 INFO:tasks.rados.rados.0.smithi085.stdout:3439: seq_num 1172 ranges {497804=510190,1465955=448737,2584924=597152} 2021-02-10T04:41:28.466 INFO:tasks.rados.rados.0.smithi085.stdout:3439: writing smithi08575397-2 from 497804 to 1007994 tid 1 2021-02-10T04:41:28.472 INFO:tasks.rados.rados.0.smithi085.stdout:3439: writing smithi08575397-2 from 1465955 to 1914692 tid 2 2021-02-10T04:41:28.480 INFO:tasks.rados.rados.0.smithi085.stdout:3439: writing smithi08575397-2 from 2584924 to 3182076 tid 3 2021-02-10T04:41:28.480 INFO:tasks.rados.rados.0.smithi085.stdout:3440: delete oid 7 current snap is 401 2021-02-10T04:41:28.480 INFO:tasks.rados.rados.0.smithi085.stdout:3438: finishing rollback tid 0 to smithi08575397-8 2021-02-10T04:41:28.480 INFO:tasks.rados.rados.0.smithi085.stdout:3438: finishing rollback tid 1 to smithi08575397-8 2021-02-10T04:41:28.480 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 1577 (ObjNum 1953066355 snap 942696808 seq_num 859125557) dirty dne 2021-02-10T04:41:28.480 INFO:tasks.rados.rados.0.smithi085.stdout:3439: finishing write tid 1 to smithi08575397-2 2021-02-10T04:41:28.481 INFO:tasks.rados.rados.0.smithi085.stdout:3439: finishing write tid 2 to smithi08575397-2 2021-02-10T04:41:28.484 INFO:tasks.rados.rados.0.smithi085.stdout:3438: done (6 left) 2021-02-10T04:41:28.484 INFO:tasks.rados.rados.0.smithi085.stdout:3440: done (5 left) 2021-02-10T04:41:28.484 INFO:tasks.rados.rados.0.smithi085.stdout:3441: delete oid 32 current snap is 401 2021-02-10T04:41:28.484 INFO:tasks.rados.rados.0.smithi085.stdout:3439: finishing write tid 3 to smithi08575397-2 2021-02-10T04:41:28.484 INFO:tasks.rados.rados.0.smithi085.stdout:3439: finishing write tid 4 to smithi08575397-2 2021-02-10T04:41:28.485 INFO:tasks.rados.rados.0.smithi085.stdout:3439: finishing write tid 5 to smithi08575397-2 2021-02-10T04:41:28.485 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 1583 (ObjNum 1172 snap 401 seq_num 1172) dirty exists 2021-02-10T04:41:28.485 INFO:tasks.rados.rados.0.smithi085.stdout:3439: left oid 2 (ObjNum 1172 snap 401 seq_num 1172) 2021-02-10T04:41:28.485 INFO:tasks.rados.rados.0.smithi085.stdout:3439: done (5 left) 2021-02-10T04:41:28.485 INFO:tasks.rados.rados.0.smithi085.stdout:3441: done (4 left) 2021-02-10T04:41:28.485 INFO:tasks.rados.rados.0.smithi085.stdout:3442: write oid 43 current snap is 401 2021-02-10T04:41:28.485 INFO:tasks.rados.rados.0.smithi085.stdout:3442: seq_num 1175 ranges {548225=735937,1925048=596064,2950840=547321} 2021-02-10T04:41:28.492 INFO:tasks.rados.rados.0.smithi085.stdout:3442: writing smithi08575397-43 from 548225 to 1284162 tid 1 2021-02-10T04:41:28.500 INFO:tasks.rados.rados.0.smithi085.stdout:3442: writing smithi08575397-43 from 1925048 to 2521112 tid 2 2021-02-10T04:41:28.506 INFO:tasks.rados.rados.0.smithi085.stdout:3442: writing smithi08575397-43 from 2950840 to 3498161 tid 3 2021-02-10T04:41:28.506 INFO:tasks.rados.rados.0.smithi085.stdout:3443: write oid 7 current snap is 401 2021-02-10T04:41:28.506 INFO:tasks.rados.rados.0.smithi085.stdout:3442: finishing write tid 1 to smithi08575397-43 2021-02-10T04:41:28.506 INFO:tasks.rados.rados.0.smithi085.stdout:3443: seq_num 1176 ranges {565032=411186,1583530=456075,2578263=758593} 2021-02-10T04:41:28.513 INFO:tasks.rados.rados.0.smithi085.stdout:3443: writing smithi08575397-7 from 565032 to 976218 tid 1 2021-02-10T04:41:28.519 INFO:tasks.rados.rados.0.smithi085.stdout:3443: writing smithi08575397-7 from 1583530 to 2039605 tid 2 2021-02-10T04:41:28.528 INFO:tasks.rados.rados.0.smithi085.stdout:3443: writing smithi08575397-7 from 2578263 to 3336856 tid 3 2021-02-10T04:41:28.529 INFO:tasks.rados.rados.0.smithi085.stdout:3444: rollback oid 40 current snap is 401 2021-02-10T04:41:28.529 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 40 to 369 2021-02-10T04:41:28.529 INFO:tasks.rados.rados.0.smithi085.stdout:3442: finishing write tid 2 to smithi08575397-43 2021-02-10T04:41:28.529 INFO:tasks.rados.rados.0.smithi085.stdout:3442: finishing write tid 3 to smithi08575397-43 2021-02-10T04:41:28.529 INFO:tasks.rados.rados.0.smithi085.stdout:3442: finishing write tid 4 to smithi08575397-43 2021-02-10T04:41:28.530 INFO:tasks.rados.rados.0.smithi085.stdout:3442: finishing write tid 5 to smithi08575397-43 2021-02-10T04:41:28.530 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 1583 (ObjNum 1175 snap 401 seq_num 1175) dirty exists 2021-02-10T04:41:28.530 INFO:tasks.rados.rados.0.smithi085.stdout:3442: left oid 43 (ObjNum 1175 snap 401 seq_num 1175) 2021-02-10T04:41:28.530 INFO:tasks.rados.rados.0.smithi085.stdout:3443: finishing write tid 1 to smithi08575397-7 2021-02-10T04:41:28.530 INFO:tasks.rados.rados.0.smithi085.stdout:3443: finishing write tid 2 to smithi08575397-7 2021-02-10T04:41:28.530 INFO:tasks.rados.rados.0.smithi085.stdout:3442: done (6 left) 2021-02-10T04:41:28.531 INFO:tasks.rados.rados.0.smithi085.stdout:3445: snap_create 2021-02-10T04:41:28.534 INFO:tasks.rados.rados.0.smithi085.stdout:3443: finishing write tid 3 to smithi08575397-7 2021-02-10T04:41:28.534 INFO:tasks.rados.rados.0.smithi085.stdout:3443: finishing write tid 4 to smithi08575397-7 2021-02-10T04:41:28.534 INFO:tasks.rados.rados.0.smithi085.stdout:3444: finishing rollback tid 1 to smithi08575397-40 2021-02-10T04:41:28.534 INFO:tasks.rados.rados.0.smithi085.stdout:3443: finishing write tid 5 to smithi08575397-7 2021-02-10T04:41:28.535 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 1587 (ObjNum 1176 snap 401 seq_num 1176) dirty exists 2021-02-10T04:41:28.535 INFO:tasks.rados.rados.0.smithi085.stdout:3443: left oid 7 (ObjNum 1176 snap 401 seq_num 1176) 2021-02-10T04:41:28.535 INFO:tasks.rados.rados.0.smithi085.stdout:3444: finishing rollback tid 2 to smithi08575397-40 2021-02-10T04:41:28.536 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 1776 (ObjNum 1099 snap 369 seq_num 1099) dirty exists 2021-02-10T04:41:28.644 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:41:28 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:41:28.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:28 smithi164 conmon[29288]: cluster 2021-02-10T04:41:27.312778+0000 mon.a ( 2021-02-10T04:41:28.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:28 smithi164 conmon[29288]: mon.0) 1824 : cluster [DBG] osdmap e868: 8 total, 8 up, 7 in 2021-02-10T04:41:28.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:28 smithi164 conmon[29288]: cluster 2021-02-10T04:41:28.214632+0000 mgr.y (mgr.14140) 824 : cluster [DBG] pgmap v1453: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 669 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 1.9 MiB/s wr, 12 op/s 2021-02-10T04:41:28.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:28 smithi164 conmon[29288]: cluster 2021-02-10T04:41:28.288091+0000 mon.a (mon.0 2021-02-10T04:41:28.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:28 smithi164 conmon[29288]: ) 1825 : cluster [DBG] osdmap e869: 8 total, 8 up, 7 in 2021-02-10T04:41:29.329 INFO:tasks.rados.rados.0.smithi085.stdout:3443: done (6 left) 2021-02-10T04:41:29.329 INFO:tasks.rados.rados.0.smithi085.stdout:3444: done (5 left) 2021-02-10T04:41:29.329 INFO:tasks.rados.rados.0.smithi085.stdout:3445: done (4 left) 2021-02-10T04:41:29.329 INFO:tasks.rados.rados.0.smithi085.stdout:3446: delete oid 13 current snap is 402 2021-02-10T04:41:29.334 INFO:tasks.rados.rados.0.smithi085.stdout:3446: done (4 left) 2021-02-10T04:41:29.334 INFO:tasks.rados.rados.0.smithi085.stdout:3447: rollback oid 11 current snap is 402 2021-02-10T04:41:29.334 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 11 to 380 2021-02-10T04:41:29.334 INFO:tasks.rados.rados.0.smithi085.stdout:3448: snap_remove snap 325 2021-02-10T04:41:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: audit 2021-02-10T04:41:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: 2021-02-10T04:41:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: 2021 2021-02-10T04:41:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: - 2021-02-10T04:41:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: 02 2021-02-10T04:41:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: - 2021-02-10T04:41:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: 10 2021-02-10T04:41:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: T 2021-02-10T04:41:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: 04 2021-02-10T04:41:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: : 2021-02-10T04:41:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: 41 2021-02-10T04:41:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: : 2021-02-10T04:41:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: 28 2021-02-10T04:41:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: . 2021-02-10T04:41:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: 458014 2021-02-10T04:41:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: +0000 2021-02-10T04:41:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: 2021-02-10T04:41:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: mgr.y 2021-02-10T04:41:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: ( 2021-02-10T04:41:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: mgr 2021-02-10T04:41:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: . 2021-02-10T04:41:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: 14140 2021-02-10T04:41:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: ) 2021-02-10T04:41:29.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: 825 2021-02-10T04:41:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: : 2021-02-10T04:41:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: audit 2021-02-10T04:41:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: 2021-02-10T04:41:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: [DBG] 2021-02-10T04:41:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: 2021-02-10T04:41:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:41:29.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:29 smithi164 conmon[29288]: 2021-02-10T04:41:29.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:29 smithi085 conmon[29319]: audit 2021-02-10T04:41:29.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:29 smithi085 conmon[29319]: 2021-02 2021-02-10T04:41:29.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:29 smithi085 conmon[29319]: -10T04:41:28.458014+0000 mgr.y (mgr.14140) 825 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:41:29.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:29 smithi085 conmon[35961]: audit 2021-02-10T04:41:28.458014 2021-02-10T04:41:29.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:29 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 825 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:41:30.344 INFO:tasks.rados.rados.0.smithi085.stdout:3447: finishing rollback tid 0 to smithi08575397-11 2021-02-10T04:41:30.345 INFO:tasks.rados.rados.0.smithi085.stdout:3447: finishing rollback tid 1 to smithi08575397-11 2021-02-10T04:41:30.345 INFO:tasks.rados.rados.0.smithi085.stdout:3447: finishing rollback tid 2 to smithi08575397-11 2021-02-10T04:41:30.345 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 1780 (ObjNum 1054 snap 346 seq_num 1054) dirty exists 2021-02-10T04:41:30.345 INFO:tasks.rados.rados.0.smithi085.stdout:3447: done (5 left) 2021-02-10T04:41:30.345 INFO:tasks.rados.rados.0.smithi085.stdout:3448: done (4 left) 2021-02-10T04:41:30.345 INFO:tasks.rados.rados.0.smithi085.stdout:3449: snap_remove snap 384 2021-02-10T04:41:30.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:30 smithi164 conmon[29288]: cluster 2021-02-10T04:41:29.329012+0000 2021-02-10T04:41:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:30 smithi164 conmon[29288]: mon.a (mon.0) 1826 : cluster [DBG] osdmap e870: 8 total, 8 up, 7 in 2021-02-10T04:41:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:30 smithi164 conmon[29288]: cluster 2021-02-10T04:41:30.215425+0000 mgr.y (mgr 2021-02-10T04:41:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:30 smithi164 conmon[29288]: .14140) 826 : cluster [DBG] pgmap v1456: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 684 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 6.7 MiB/s rd, 3.1 MiB/s wr, 14 op/s 2021-02-10T04:41:30.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:30 smithi085 conmon[35961]: cluster 2021-02-10T04:41:29. 2021-02-10T04:41:30.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:30 smithi085 conmon[35961]: 329012+0000 mon.a (mon.0) 1826 : cluster [DBG] osdmap e870: 8 total, 8 up, 7 in 2021-02-10T04:41:30.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:30 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:41:30.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:30 smithi085 conmon[35961]: -02-10T04:41:30.215425+0000 2021-02-10T04:41:30.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:30 smithi085 conmon[35961]: mgr.y (mgr.14140) 826 : cluster [DBG] pgmap v1456: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 684 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 6.7 MiB/s rd, 3.1 MiB/s wr, 14 op/s 2021-02-10T04:41:30.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:30 smithi085 conmon[35961]: 2021-02-10T04:41:30.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:30 smithi085 conmon[29319]: cluster 2021-02-10T04:41:30.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:30 smithi085 conmon[29319]: 2021-02-10T04:41:30.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:30 smithi085 conmon[29319]: 2021 2021-02-10T04:41:30.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:30 smithi085 conmon[29319]: - 2021-02-10T04:41:30.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:30 smithi085 conmon[29319]: 02 2021-02-10T04:41:30.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:30 smithi085 conmon[29319]: - 2021-02-10T04:41:30.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:30 smithi085 conmon[29319]: 10 2021-02-10T04:41:30.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:30 smithi085 conmon[29319]: T 2021-02-10T04:41:30.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:30 smithi085 conmon[29319]: 04 2021-02-10T04:41:30.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:30 smithi085 conmon[29319]: : 2021-02-10T04:41:30.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:30 smithi085 conmon[29319]: 41 2021-02-10T04:41:30.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:30 smithi085 conmon[29319]: : 2021-02-10T04:41:30.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:30 smithi085 conmon[29319]: 29 2021-02-10T04:41:30.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:30 smithi085 conmon[29319]: . 2021-02-10T04:41:30.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:30 smithi085 conmon[29319]: 329012+0000 mon.a (mon.0) 1826 2021-02-10T04:41:30.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:30 smithi085 conmon[29319]: : cluster [DBG] osdmap e870: 8 total, 8 up, 7 in 2021-02-10T04:41:30.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:30 smithi085 conmon[29319]: cluster 2021-02-10T04:41: 2021-02-10T04:41:30.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:30 smithi085 conmon[29319]: 30.215425+0000 mgr.y (mgr. 2021-02-10T04:41:30.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:30 smithi085 conmon[29319]: 14140) 826 : cluster [DBG] pgmap v1456: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 684 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 6.7 MiB/s rd, 3.1 MiB/s wr, 14 op/s 2021-02-10T04:41:31.343 INFO:tasks.rados.rados.0.smithi085.stdout:3449: done (4 left) 2021-02-10T04:41:31.343 INFO:tasks.rados.rados.0.smithi085.stdout:3450: rollback oid 22 current snap is 402 2021-02-10T04:41:31.343 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 22 to 391 2021-02-10T04:41:31.343 INFO:tasks.rados.rados.0.smithi085.stdout:3451: write oid 13 current snap is 402 2021-02-10T04:41:31.344 INFO:tasks.rados.rados.0.smithi085.stdout:3451: seq_num 1178 ranges {598646=561684,1700494=459744,2633109=781137} 2021-02-10T04:41:31.350 INFO:tasks.rados.rados.0.smithi085.stdout:3451: writing smithi08575397-13 from 598646 to 1160330 tid 1 2021-02-10T04:41:31.356 INFO:tasks.rados.rados.0.smithi085.stdout:3451: writing smithi08575397-13 from 1700494 to 2160238 tid 2 2021-02-10T04:41:31.364 INFO:tasks.rados.rados.0.smithi085.stdout:3451: writing smithi08575397-13 from 2633109 to 3414246 tid 3 2021-02-10T04:41:31.364 INFO:tasks.rados.rados.0.smithi085.stdout:3452: snap_create 2021-02-10T04:41:31.364 INFO:tasks.rados.rados.0.smithi085.stdout:3450: finishing rollback tid 0 to smithi08575397-22 2021-02-10T04:41:31.364 INFO:tasks.rados.rados.0.smithi085.stdout:3450: finishing rollback tid 1 to smithi08575397-22 2021-02-10T04:41:31.364 INFO:tasks.rados.rados.0.smithi085.stdout:3450: finishing rollback tid 2 to smithi08575397-22 2021-02-10T04:41:31.364 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 1486 (ObjNum 1108 snap 374 seq_num 1108) dirty exists 2021-02-10T04:41:31.364 INFO:tasks.rados.rados.0.smithi085.stdout:3451: finishing write tid 1 to smithi08575397-13 2021-02-10T04:41:31.364 INFO:tasks.rados.rados.0.smithi085.stdout:3451: finishing write tid 2 to smithi08575397-13 2021-02-10T04:41:31.369 INFO:tasks.rados.rados.0.smithi085.stdout:3451: finishing write tid 3 to smithi08575397-13 2021-02-10T04:41:31.369 INFO:tasks.rados.rados.0.smithi085.stdout:3451: finishing write tid 4 to smithi08575397-13 2021-02-10T04:41:31.369 INFO:tasks.rados.rados.0.smithi085.stdout:3451: finishing write tid 5 to smithi08575397-13 2021-02-10T04:41:31.369 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 1553 (ObjNum 1178 snap 402 seq_num 1178) dirty exists 2021-02-10T04:41:31.370 INFO:tasks.rados.rados.0.smithi085.stdout:3451: left oid 13 (ObjNum 1178 snap 402 seq_num 1178) 2021-02-10T04:41:31.544 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:41:31.576 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:41:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:31 smithi164 conmon[29288]: cluster 2021-02-10T04:41:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:31 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:41:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:31 smithi164 conmon[29288]: 04:41:30.343627+0000 mon.a (mon.0) 1827 : cluster [DBG] osdmap e871: 8 total, 8 up, 7 in 2021-02-10T04:41:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:31 smithi164 conmon[29288]: cluster 2021-02-10T04:41:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:31 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:41:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:31 smithi164 conmon[29288]: 41:31.342634+0000 mon.a (mon.0) 1828 : cluster [DBG] osdmap e872: 8 total, 8 up, 7 in 2021-02-10T04:41:31.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:31 smithi085 conmon[29319]: cluster 2021-02-10T04:41:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:31 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:41:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:31 smithi085 conmon[29319]: 41:30.343627+0000 mon.a (mon.0) 1827 : cluster [DBG] osdmap e871: 8 total, 8 up, 7 in 2021-02-10T04:41:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:31 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:41:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:31 smithi085 conmon[29319]: :41:31.342634+0000 mon.a (mon.0) 1828 2021-02-10T04:41:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:31 smithi085 conmon[29319]: : cluster [DBG] osdmap e872: 8 total, 8 up, 7 in 2021-02-10T04:41:31.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:31 smithi085 conmon[35961]: cluster 2021-02-10T04:41:30.343627+0000 mon.a (mon.0 2021-02-10T04:41:31.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:31 smithi085 conmon[35961]: ) 1827 : cluster [DBG] osdmap e871: 8 total, 8 up, 7 in 2021-02-10T04:41:31.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:31 smithi085 conmon[35961]: cluster 2021-02-10T04:41:31.342634+0000 mon.a (mon.0) 2021-02-10T04:41:31.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:31 smithi085 conmon[35961]: 1828 : cluster [DBG] osdmap e872: 8 total, 8 up, 7 in 2021-02-10T04:41:32.321 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:41:32.356 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:41:32 smithi085 conmon[49106]: debug 2021-02-10T04:41:32.058+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:41:32.358 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:32 smithi085 conmon[29319]: cluster 2021-02-10T04:41:31.508306+0000 osd.3 (osd.3) 86 : 2021-02-10T04:41:32.358 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:32 smithi085 conmon[29319]: cluster [DBG] purged_snaps scrub starts 2021-02-10T04:41:32.358 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:32 smithi085 conmon[29319]: cluster 2021-02-10T04:41:31.509540+0000 osd.3 (osd.3) 87 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:41:32.358 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:32 smithi085 conmon[29319]: cluster 2021-02-10T04:41:32.216104 2021-02-10T04:41:32.358 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:32 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 827 : cluster [DBG] 2021-02-10T04:41:32.358 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:32 smithi085 conmon[29319]: pgmap v1459: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 684 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 6.7 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2021-02-10T04:41:32.359 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:32 smithi085 conmon[35961]: cluster 2021-02-10T04:41:31.508306+0000 2021-02-10T04:41:32.360 INFO:tasks.rados.rados.0.smithi085.stdout:3450: done (6 left) 2021-02-10T04:41:32.360 INFO:tasks.rados.rados.0.smithi085.stdout:3451: done (5 left) 2021-02-10T04:41:32.360 INFO:tasks.rados.rados.0.smithi085.stdout:3452: done (4 left) 2021-02-10T04:41:32.360 INFO:tasks.rados.rados.0.smithi085.stdout:3453: delete oid 25 current snap is 403 2021-02-10T04:41:32.365 INFO:tasks.rados.rados.0.smithi085.stdout:3453: done (4 left) 2021-02-10T04:41:32.365 INFO:tasks.rados.rados.0.smithi085.stdout:3454: write (excl) oid 32 current snap is 403 2021-02-10T04:41:32.365 INFO:tasks.rados.rados.0.smithi085.stdout:3454: seq_num 1180 ranges {632260=428535,1817458=463413,2687955=520034,3709324=122936} 2021-02-10T04:41:32.371 INFO:tasks.rados.rados.0.smithi085.stdout:3454: writing smithi08575397-32 from 632260 to 1060795 tid 1 2021-02-10T04:41:32.379 INFO:tasks.rados.rados.0.smithi085.stdout:3454: writing smithi08575397-32 from 1817458 to 2280871 tid 2 2021-02-10T04:41:32.385 INFO:tasks.rados.rados.0.smithi085.stdout:3454: writing smithi08575397-32 from 2687955 to 3207989 tid 3 2021-02-10T04:41:32.389 INFO:tasks.rados.rados.0.smithi085.stdout:3454: writing smithi08575397-32 from 3709324 to 3832260 tid 4 2021-02-10T04:41:32.389 INFO:tasks.rados.rados.0.smithi085.stdout:3455: write (excl) oid 3 current snap is 403 2021-02-10T04:41:32.389 INFO:tasks.rados.rados.0.smithi085.stdout:3454: finishing write tid 1 to smithi08575397-32 2021-02-10T04:41:32.389 INFO:tasks.rados.rados.0.smithi085.stdout:3455: seq_num 1181 ranges {649067=503784,1875940=607071,3115378=731306} 2021-02-10T04:41:32.395 INFO:tasks.rados.rados.0.smithi085.stdout:3455: writing smithi08575397-3 from 649067 to 1152851 tid 1 2021-02-10T04:41:32.403 INFO:tasks.rados.rados.0.smithi085.stdout:3455: writing smithi08575397-3 from 1875940 to 2483011 tid 2 2021-02-10T04:41:32.412 INFO:tasks.rados.rados.0.smithi085.stdout:3455: writing smithi08575397-3 from 3115378 to 3846684 tid 3 2021-02-10T04:41:32.412 INFO:tasks.rados.rados.0.smithi085.stdout:3456: snap_create 2021-02-10T04:41:32.412 INFO:tasks.rados.rados.0.smithi085.stdout:3454: finishing write tid 2 to smithi08575397-32 2021-02-10T04:41:32.413 INFO:tasks.rados.rados.0.smithi085.stdout:3454: finishing write tid 3 to smithi08575397-32 2021-02-10T04:41:32.413 INFO:tasks.rados.rados.0.smithi085.stdout:3454: finishing write tid 4 to smithi08575397-32 2021-02-10T04:41:32.413 INFO:tasks.rados.rados.0.smithi085.stdout:3454: finishing write tid 5 to smithi08575397-32 2021-02-10T04:41:32.413 INFO:tasks.rados.rados.0.smithi085.stdout:3454: finishing write tid 6 to smithi08575397-32 2021-02-10T04:41:32.413 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 32 v 1474 (ObjNum 1180 snap 403 seq_num 1180) dirty exists 2021-02-10T04:41:32.413 INFO:tasks.rados.rados.0.smithi085.stdout:3454: left oid 32 (ObjNum 1180 snap 403 seq_num 1180) 2021-02-10T04:41:32.413 INFO:tasks.rados.rados.0.smithi085.stdout:3455: finishing write tid 1 to smithi08575397-3 2021-02-10T04:41:32.413 INFO:tasks.rados.rados.0.smithi085.stdout:3455: finishing write tid 2 to smithi08575397-3 2021-02-10T04:41:32.416 INFO:tasks.rados.rados.0.smithi085.stdout:3455: finishing write tid 3 to smithi08575397-3 2021-02-10T04:41:32.416 INFO:tasks.rados.rados.0.smithi085.stdout:3455: finishing write tid 4 to smithi08575397-3 2021-02-10T04:41:32.416 INFO:tasks.rados.rados.0.smithi085.stdout:3455: finishing write tid 5 to smithi08575397-3 2021-02-10T04:41:32.417 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 1591 (ObjNum 1181 snap 403 seq_num 1181) dirty exists 2021-02-10T04:41:32.417 INFO:tasks.rados.rados.0.smithi085.stdout:3455: left oid 3 (ObjNum 1181 snap 403 seq_num 1181) 2021-02-10T04:41:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:32 smithi164 conmon[29288]: cluster 2021-02-10T04:41:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:32 smithi164 conmon[29288]: 2021-02-10T04:41:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:32 smithi164 conmon[29288]: 2021 2021-02-10T04:41:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:32 smithi164 conmon[29288]: - 2021-02-10T04:41:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:32 smithi164 conmon[29288]: 02-10T04:41:31.508306 2021-02-10T04:41:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:32 smithi164 conmon[29288]: +0000 osd.3 (osd.3) 86 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:41:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:32 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:41:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:32 smithi164 conmon[29288]: 41:31.509540+0000 osd.3 (osd.3) 87 : cluster [DBG] 2021-02-10T04:41:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:32 smithi164 conmon[29288]: purged_snaps scrub ok 2021-02-10T04:41:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:32 smithi164 conmon[29288]: cluster 2021-02-10T04:41:32 2021-02-10T04:41:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:32 smithi164 conmon[29288]: .216104+0000 2021-02-10T04:41:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:32 smithi164 conmon[29288]: mgr.y (mgr.14140 2021-02-10T04:41:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:32 smithi164 conmon[29288]: ) 827 : cluster [DBG] pgmap v1459: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 684 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 6.7 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2021-02-10T04:41:32.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:32 smithi085 conmon[35961]: osd.3 (osd.3) 86 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:41:32.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:32 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:41:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:32 smithi085 conmon[35961]: T04:41:31.509540+0000 osd.3 (osd.3) 87 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:41:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:32 smithi085 conmon[35961]: cluster 2021-02-10T04:41 2021-02-10T04:41:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:32 smithi085 conmon[35961]: :32.216104+0000 mgr.y (mgr.14140) 827 : cluster [DBG] pgmap v1459: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 684 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 6.7 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2021-02-10T04:41:33.408 INFO:tasks.rados.rados.0.smithi085.stdout:3454: done (6 left) 2021-02-10T04:41:33.408 INFO:tasks.rados.rados.0.smithi085.stdout:3455: done (5 left) 2021-02-10T04:41:33.409 INFO:tasks.rados.rados.0.smithi085.stdout:3456: done (4 left) 2021-02-10T04:41:33.409 INFO:tasks.rados.rados.0.smithi085.stdout:3457: rollback oid 3 current snap is 404 2021-02-10T04:41:33.409 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 3 to 360 2021-02-10T04:41:33.409 INFO:tasks.rados.rados.0.smithi085.stdout:3458: snap_remove snap 392 2021-02-10T04:41:33.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:33 smithi164 conmon[29288]: cluster 2021-02-10T04:41:32.359694+0000 mon.a (mon.0) 1829 2021-02-10T04:41:33.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:33 smithi164 conmon[29288]: : cluster [DBG] osdmap e873: 8 total, 8 up, 7 in 2021-02-10T04:41:33.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:33 smithi085 conmon[29319]: cluster 2021-02-10T04:41:32.359694+0000 mon.a 2021-02-10T04:41:33.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:33 smithi085 conmon[29319]: (mon.0) 1829 : cluster [DBG] osdmap e873: 8 total, 8 up, 7 in 2021-02-10T04:41:33.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:33 smithi085 conmon[35961]: cluster 2021-02-10T04:41:32.359694+0000 2021-02-10T04:41:33.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:33 smithi085 conmon[35961]: mon.a (mon.0) 1829 : cluster [DBG] osdmap e873: 8 total, 8 up, 7 in 2021-02-10T04:41:34.403 INFO:tasks.rados.rados.0.smithi085.stdout:3457: finishing rollback tid 0 to smithi08575397-3 2021-02-10T04:41:34.403 INFO:tasks.rados.rados.0.smithi085.stdout:3457: finishing rollback tid 1 to smithi08575397-3 2021-02-10T04:41:34.404 INFO:tasks.rados.rados.0.smithi085.stdout:3457: finishing rollback tid 2 to smithi08575397-3 2021-02-10T04:41:34.404 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 1594 (ObjNum 1058 snap 348 seq_num 1058) dirty exists 2021-02-10T04:41:34.404 INFO:tasks.rados.rados.0.smithi085.stdout:3457: done (5 left) 2021-02-10T04:41:34.404 INFO:tasks.rados.rados.0.smithi085.stdout:3458: done (4 left) 2021-02-10T04:41:34.404 INFO:tasks.rados.rados.0.smithi085.stdout:3459: snap_create 2021-02-10T04:41:34.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:34 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:41:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:34 smithi085 conmon[29319]: -10T04:41:33.407623+0000 mon.a (mon.0) 1830 : cluster [DBG] osdmap e874: 8 total, 8 up, 7 in 2021-02-10T04:41:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:34 smithi085 conmon[29319]: cluster 2021-02-10T04:41:34.216800+0000 mgr.y (mgr.14140) 828 2021-02-10T04:41:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:34 smithi085 conmon[29319]: : cluster [DBG] pgmap v1462: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 686 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 815 KiB/s wr, 6 op/s 2021-02-10T04:41:34.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:34 smithi085 conmon[35961]: cluster 2021-02-10T04:41:33.407623 2021-02-10T04:41:34.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:34 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1830 : cluster [DBG] osdmap e874: 8 total, 8 up, 7 in 2021-02-10T04:41:34.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:34 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:41:34.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:34 smithi085 conmon[35961]: 41:34.216800+0000 mgr.y (mgr.14140) 828 : cluster [DBG] pgmap v1462: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 686 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 815 KiB/s wr, 6 op/s 2021-02-10T04:41:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:34 smithi164 conmon[29288]: cluster 2021-02-10T04:41:33.407623+0000 mon.a (mon 2021-02-10T04:41:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:34 smithi164 conmon[29288]: .0) 1830 : cluster [DBG] osdmap e874: 8 total, 8 up, 7 in 2021-02-10T04:41:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:34 smithi164 conmon[29288]: cluster 2021-02-10T04:41:34.216800+0000 mgr.y (mgr.14140) 828 : cluster [DBG] pgmap v1462: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 686 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 815 KiB/s wr, 6 op/s 2021-02-10T04:41:35.406 INFO:tasks.rados.rados.0.smithi085.stdout:3459: done (4 left) 2021-02-10T04:41:35.406 INFO:tasks.rados.rados.0.smithi085.stdout:3460: write (excl) oid 38 current snap is 405 2021-02-10T04:41:35.406 INFO:tasks.rados.rados.0.smithi085.stdout:3460: seq_num 1182 ranges {665874=579033,1934422=467082,3142801=542578} 2021-02-10T04:41:35.414 INFO:tasks.rados.rados.0.smithi085.stdout:3460: writing smithi08575397-38 from 665874 to 1244907 tid 1 2021-02-10T04:41:35.421 INFO:tasks.rados.rados.0.smithi085.stdout:3460: writing smithi08575397-38 from 1934422 to 2401504 tid 2 2021-02-10T04:41:35.429 INFO:tasks.rados.rados.0.smithi085.stdout:3460: writing smithi08575397-38 from 3142801 to 3685379 tid 3 2021-02-10T04:41:35.429 INFO:tasks.rados.rados.0.smithi085.stdout:3461: delete oid 2 current snap is 405 2021-02-10T04:41:35.429 INFO:tasks.rados.rados.0.smithi085.stdout:3460: finishing write tid 1 to smithi08575397-38 2021-02-10T04:41:35.429 INFO:tasks.rados.rados.0.smithi085.stdout:3460: finishing write tid 2 to smithi08575397-38 2021-02-10T04:41:35.433 INFO:tasks.rados.rados.0.smithi085.stdout:3461: done (5 left) 2021-02-10T04:41:35.433 INFO:tasks.rados.rados.0.smithi085.stdout:3462: read oid 46 snap -1 2021-02-10T04:41:35.433 INFO:tasks.rados.rados.0.smithi085.stdout:3462: expect (ObjNum 1087 snap 364 seq_num 1087) 2021-02-10T04:41:35.433 INFO:tasks.rados.rados.0.smithi085.stdout:3463: delete oid 5 current snap is 405 2021-02-10T04:41:35.438 INFO:tasks.rados.rados.0.smithi085.stdout:3463: done (6 left) 2021-02-10T04:41:35.438 INFO:tasks.rados.rados.0.smithi085.stdout:3464: snap_remove snap 349 2021-02-10T04:41:35.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:35 smithi085 conmon[29319]: cluster 2021-02-10T04:41:35.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:35 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:41:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:35 smithi085 conmon[29319]: :41:34.403648+0000 mon.a (mon.0) 1831 : cluster [DBG] osdmap e875: 8 total, 8 up, 7 in 2021-02-10T04:41:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:35 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:41:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:35 smithi085 conmon[29319]: 02-10T04:41:34.997940+0000 osd.0 (osd.0) 2021-02-10T04:41:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:35 smithi085 conmon[29319]: 91 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:41:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:35 smithi085 conmon[29319]: cluster 2021-02-10T04:41 2021-02-10T04:41:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:35 smithi085 conmon[29319]: :34.999267+0000 osd.0 (osd.0) 92 : cluster 2021-02-10T04:41:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:35 smithi085 conmon[29319]: [DBG] purged_snaps scrub ok 2021-02-10T04:41:35.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:35 smithi085 conmon[35961]: cluster 2021-02-10T04:41:35.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:35 smithi085 conmon[35961]: 2021-02-10T04:41:35.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:35 smithi085 conmon[35961]: 2021 2021-02-10T04:41:35.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:35 smithi085 conmon[35961]: - 2021-02-10T04:41:35.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:35 smithi085 conmon[35961]: 02 2021-02-10T04:41:35.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:35 smithi085 conmon[35961]: - 2021-02-10T04:41:35.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:35 smithi085 conmon[35961]: 10 2021-02-10T04:41:35.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:35 smithi085 conmon[35961]: T04:41:34.403648+0000 2021-02-10T04:41:35.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:35 smithi085 conmon[35961]: mon.a (mon.0) 1831 : cluster [DBG] osdmap e875: 8 total, 8 up, 7 in 2021-02-10T04:41:35.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:35 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:41:35.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:35 smithi085 conmon[35961]: :41:34.997940+0000 osd.0 (osd.0) 2021-02-10T04:41:35.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:35 smithi085 conmon[35961]: 91 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:41:35.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:35 smithi085 conmon[35961]: cluster 2021-02-10T04:41:35.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:35 smithi085 conmon[35961]: 2021-02-10T04:41:34.999267+0000 2021-02-10T04:41:35.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:35 smithi085 conmon[35961]: osd.0 (osd.0) 92 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:41:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:35 smithi164 conmon[29288]: cluster 2021-02-10T04:41:34.403648+0000 mon.a (mon.0) 1831 : cluster [DBG] 2021-02-10T04:41:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:35 smithi164 conmon[29288]: osdmap e875: 8 total, 8 up, 7 in 2021-02-10T04:41:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:35 smithi164 conmon[29288]: cluster 2021-02-10T04:41:34.997940+0000 osd.0 (osd.0) 91 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:41:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:35 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:41:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:35 smithi164 conmon[29288]: 04:41:34.999267+0000 osd.0 (osd.0) 92 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:41:36.421 INFO:tasks.rados.rados.0.smithi085.stdout:3464: done (6 left) 2021-02-10T04:41:36.422 INFO:tasks.rados.rados.0.smithi085.stdout:3465: snap_create 2021-02-10T04:41:36.422 INFO:tasks.rados.rados.0.smithi085.stdout:3462: expect (ObjNum 1087 snap 364 seq_num 1087) 2021-02-10T04:41:36.538 INFO:tasks.rados.rados.0.smithi085.stdout:3460: finishing write tid 3 to smithi08575397-38 2021-02-10T04:41:36.538 INFO:tasks.rados.rados.0.smithi085.stdout:3460: finishing write tid 4 to smithi08575397-38 2021-02-10T04:41:36.538 INFO:tasks.rados.rados.0.smithi085.stdout:3460: finishing write tid 5 to smithi08575397-38 2021-02-10T04:41:36.538 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 1579 (ObjNum 1182 snap 405 seq_num 1182) dirty exists 2021-02-10T04:41:36.538 INFO:tasks.rados.rados.0.smithi085.stdout:3460: left oid 38 (ObjNum 1182 snap 405 seq_num 1182) 2021-02-10T04:41:36.689 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:41:36.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:36 smithi085 conmon[29319]: cluster 2021-02-10T04:41:35.405679+0000 mon.a (mon.0) 2021-02-10T04:41:36.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:36 smithi085 conmon[29319]: 1832 : cluster [DBG] osdmap e876: 8 total, 8 up, 7 in 2021-02-10T04:41:36.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:36 smithi085 conmon[29319]: cluster 2021-02-10T04:41:36.217294+0000 mgr.y (mgr.14140) 829 : cluster [DBG] pgmap v1465: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 683 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.2 MiB/s wr, 8 op/s 2021-02-10T04:41:36.690 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:36 smithi085 conmon[35961]: cluster 2021-02-10T04:41:36.690 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:36 smithi085 conmon[35961]: 2021-02-10T04:41:35 2021-02-10T04:41:36.690 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:36 smithi085 conmon[35961]: .405679+0000 mon.a (mon.0) 1832 : cluster [DBG] osdmap e876: 8 total, 8 up, 7 in 2021-02-10T04:41:36.690 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:36 smithi085 conmon[35961]: cluster 2021-02-10T04:41 2021-02-10T04:41:36.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:36 smithi085 conmon[35961]: :36.217294+0000 mgr.y (mgr.14140) 829 : cluster [DBG] pgmap v1465: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 683 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.2 MiB/s wr, 8 op/s 2021-02-10T04:41:36.760 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:41:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:36 smithi164 conmon[29288]: cluster 2021-02-10T04:41:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:36 smithi164 conmon[29288]: 2021-02-10T04:41:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:36 smithi164 conmon[29288]: 2021-02-10T04:41:35.405679 2021-02-10T04:41:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:36 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1832 : cluster [DBG] osdmap e876: 8 total, 8 up, 7 in 2021-02-10T04:41:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:36 smithi164 conmon[29288]: cluster 2021-02-10T04:41: 2021-02-10T04:41:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:36 smithi164 conmon[29288]: 36.217294+0000 mgr.y (mgr.14140) 829 : cluster [DBG] pgmap v1465: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 683 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.2 MiB/s wr, 8 op/s 2021-02-10T04:41:36.933 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:41:36.941 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:41:36 smithi085 conmon[52836]: debug 2021-02-10T04:41:36.941 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:41:36 smithi085 conmon[52836]: 2021-02-10T04:41:36.898+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:41:36.941 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:41:36 smithi085 conmon[52836]: 2021-02-10T04:41:37.039 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:41:37.074 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:41:37.349 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:41:37.354 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[41540]: debug 2021-02-10T04:41:37.355 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[41540]: 2021-02-10T04:41:37.284+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:41:37.393 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:41:36 smithi164 conmon[37263]: debug 2021-02-10T04:41:36.946+0000 7f1c671e2700 -1 osd.5 pg_epoch: 877 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x556656232420) 2021-02-10T04:41:37.427 INFO:tasks.rados.rados.0.smithi085.stdout:3460: done (6 left) 2021-02-10T04:41:37.427 INFO:tasks.rados.rados.0.smithi085.stdout:3462: done (5 left) 2021-02-10T04:41:37.428 INFO:tasks.rados.rados.0.smithi085.stdout:3465: done (4 left) 2021-02-10T04:41:37.428 INFO:tasks.rados.rados.0.smithi085.stdout:3466: write (excl) oid 8 current snap is 406 2021-02-10T04:41:37.428 INFO:tasks.rados.rados.0.smithi085.stdout:3466: seq_num 1185 ranges {716295=404780,1593515=614409,2708717=492747,3640519=275776} 2021-02-10T04:41:37.433 INFO:tasks.rados.rados.0.smithi085.stdout:3466: writing smithi08575397-8 from 716295 to 1121075 tid 1 2021-02-10T04:41:37.441 INFO:tasks.rados.rados.0.smithi085.stdout:3466: writing smithi08575397-8 from 1593515 to 2207924 tid 2 2021-02-10T04:41:37.448 INFO:tasks.rados.rados.0.smithi085.stdout:3466: writing smithi08575397-8 from 2708717 to 3201464 tid 3 2021-02-10T04:41:37.454 INFO:tasks.rados.rados.0.smithi085.stdout:3466: writing smithi08575397-8 from 3640519 to 3916295 tid 4 2021-02-10T04:41:37.454 INFO:tasks.rados.rados.0.smithi085.stdout:3467: write oid 48 current snap is 406 2021-02-10T04:41:37.454 INFO:tasks.rados.rados.0.smithi085.stdout:3466: finishing write tid 1 to smithi08575397-8 2021-02-10T04:41:37.454 INFO:tasks.rados.rados.0.smithi085.stdout:3467: seq_num 1186 ranges {733102=596382,1768350=758067,3136140=704019} 2021-02-10T04:41:37.456 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:41:37.463 INFO:tasks.rados.rados.0.smithi085.stdout:3467: writing smithi08575397-48 from 733102 to 1329484 tid 1 2021-02-10T04:41:37.471 INFO:tasks.rados.rados.0.smithi085.stdout:3467: writing smithi08575397-48 from 1768350 to 2526417 tid 2 2021-02-10T04:41:37.480 INFO:tasks.rados.rados.0.smithi085.stdout:3467: writing smithi08575397-48 from 3136140 to 3840159 tid 3 2021-02-10T04:41:37.480 INFO:tasks.rados.rados.0.smithi085.stdout:3468: write oid 46 current snap is 406 2021-02-10T04:41:37.480 INFO:tasks.rados.rados.0.smithi085.stdout:3466: finishing write tid 2 to smithi08575397-8 2021-02-10T04:41:37.480 INFO:tasks.rados.rados.0.smithi085.stdout:3468: seq_num 1187 ranges {749909=671631,2110479=618078,3163563=515291} 2021-02-10T04:41:37.487 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:41:37.488 INFO:tasks.rados.rados.0.smithi085.stdout:3468: writing smithi08575397-46 from 749909 to 1421540 tid 1 2021-02-10T04:41:37.496 INFO:tasks.rados.rados.0.smithi085.stdout:3468: writing smithi08575397-46 from 2110479 to 2728557 tid 2 2021-02-10T04:41:37.503 INFO:tasks.rados.rados.0.smithi085.stdout:3468: writing smithi08575397-46 from 3163563 to 3678854 tid 3 2021-02-10T04:41:37.503 INFO:tasks.rados.rados.0.smithi085.stdout:3469: write oid 4 current snap is 406 2021-02-10T04:41:37.503 INFO:tasks.rados.rados.0.smithi085.stdout:3469: seq_num 1188 ranges {766716=746880,2168961=761736,3474633=442916} 2021-02-10T04:41:37.513 INFO:tasks.rados.rados.0.smithi085.stdout:3469: writing smithi08575397-4 from 766716 to 1513596 tid 1 2021-02-10T04:41:37.521 INFO:tasks.rados.rados.0.smithi085.stdout:3469: writing smithi08575397-4 from 2168961 to 2930697 tid 2 2021-02-10T04:41:37.527 INFO:tasks.rados.rados.0.smithi085.stdout:3469: writing smithi08575397-4 from 3474633 to 3917549 tid 3 2021-02-10T04:41:37.528 INFO:tasks.rados.rados.0.smithi085.stdout:3470: snap_remove snap 367 2021-02-10T04:41:37.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[29319]: cluster 2021-02-10T04:41:36.421453 2021-02-10T04:41:37.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1833 : cluster [DBG] osdmap e877: 8 total, 8 up, 7 in 2021-02-10T04:41:37.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: cluster 2021-02-10T04:41:37.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: 2021-02-10T04:41:37.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: 2021 2021-02-10T04:41:37.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: - 2021-02-10T04:41:37.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: 02 2021-02-10T04:41:37.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: - 2021-02-10T04:41:37.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: 10 2021-02-10T04:41:37.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: T 2021-02-10T04:41:37.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: 04 2021-02-10T04:41:37.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: : 2021-02-10T04:41:37.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: 41 2021-02-10T04:41:37.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: : 2021-02-10T04:41:37.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: 36 2021-02-10T04:41:37.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: . 2021-02-10T04:41:37.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: 421453 2021-02-10T04:41:37.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: +0000 2021-02-10T04:41:37.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: 2021-02-10T04:41:37.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: mon.a 2021-02-10T04:41:37.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: ( 2021-02-10T04:41:37.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: mon 2021-02-10T04:41:37.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: . 2021-02-10T04:41:37.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: 0 2021-02-10T04:41:37.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: ) 2021-02-10T04:41:37.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: 1833 2021-02-10T04:41:37.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: : 2021-02-10T04:41:37.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: cluster 2021-02-10T04:41:37.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: 2021-02-10T04:41:37.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: [DBG] 2021-02-10T04:41:37.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: 2021-02-10T04:41:37.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: osdmap e877: 8 total, 8 up, 7 in 2021-02-10T04:41:37.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:37 smithi085 conmon[35961]: 2021-02-10T04:41:37.746 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:37 smithi164 conmon[29288]: cluster 2021-02-10T04:41:36.421453+0000 2021-02-10T04:41:37.746 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:37 smithi164 conmon[29288]: mon.a (mon.0) 1833 : cluster [DBG] osdmap e877: 8 total, 8 up, 7 in 2021-02-10T04:41:37.936 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:41:38.042 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:41:38.068 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:41:37 smithi164 conmon[44544]: debug 2021-02-10T04:41:37.743+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:41:38.071 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:41:38.296 INFO:tasks.rados.rados.0.smithi085.stdout:3470: done (8 left) 2021-02-10T04:41:38.296 INFO:tasks.rados.rados.0.smithi085.stdout:3471: delete oid 45 current snap is 406 2021-02-10T04:41:38.296 INFO:tasks.rados.rados.0.smithi085.stdout:3466: finishing write tid 3 to smithi08575397-8 2021-02-10T04:41:38.296 INFO:tasks.rados.rados.0.smithi085.stdout:3466: finishing write tid 4 to smithi08575397-8 2021-02-10T04:41:38.297 INFO:tasks.rados.rados.0.smithi085.stdout:3466: finishing write tid 5 to smithi08575397-8 2021-02-10T04:41:38.297 INFO:tasks.rados.rados.0.smithi085.stdout:3466: finishing write tid 6 to smithi08575397-8 2021-02-10T04:41:38.297 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 1582 (ObjNum 1185 snap 406 seq_num 1185) dirty exists 2021-02-10T04:41:38.297 INFO:tasks.rados.rados.0.smithi085.stdout:3466: left oid 8 (ObjNum 1185 snap 406 seq_num 1185) 2021-02-10T04:41:38.297 INFO:tasks.rados.rados.0.smithi085.stdout:3467: finishing write tid 1 to smithi08575397-48 2021-02-10T04:41:38.297 INFO:tasks.rados.rados.0.smithi085.stdout:3467: finishing write tid 2 to smithi08575397-48 2021-02-10T04:41:38.297 INFO:tasks.rados.rados.0.smithi085.stdout:3467: finishing write tid 3 to smithi08575397-48 2021-02-10T04:41:38.297 INFO:tasks.rados.rados.0.smithi085.stdout:3467: finishing write tid 4 to smithi08575397-48 2021-02-10T04:41:38.297 INFO:tasks.rados.rados.0.smithi085.stdout:3467: finishing write tid 5 to smithi08575397-48 2021-02-10T04:41:38.297 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 1524 (ObjNum 1186 snap 406 seq_num 1186) dirty exists 2021-02-10T04:41:38.297 INFO:tasks.rados.rados.0.smithi085.stdout:3467: left oid 48 (ObjNum 1186 snap 406 seq_num 1186) 2021-02-10T04:41:38.297 INFO:tasks.rados.rados.0.smithi085.stdout:3468: finishing write tid 1 to smithi08575397-46 2021-02-10T04:41:38.297 INFO:tasks.rados.rados.0.smithi085.stdout:3468: finishing write tid 2 to smithi08575397-46 2021-02-10T04:41:38.298 INFO:tasks.rados.rados.0.smithi085.stdout:3468: finishing write tid 3 to smithi08575397-46 2021-02-10T04:41:38.298 INFO:tasks.rados.rados.0.smithi085.stdout:3468: finishing write tid 4 to smithi08575397-46 2021-02-10T04:41:38.298 INFO:tasks.rados.rados.0.smithi085.stdout:3468: finishing write tid 5 to smithi08575397-46 2021-02-10T04:41:38.298 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 1583 (ObjNum 1187 snap 406 seq_num 1187) dirty exists 2021-02-10T04:41:38.298 INFO:tasks.rados.rados.0.smithi085.stdout:3468: left oid 46 (ObjNum 1187 snap 406 seq_num 1187) 2021-02-10T04:41:38.298 INFO:tasks.rados.rados.0.smithi085.stdout:3469: finishing write tid 1 to smithi08575397-4 2021-02-10T04:41:38.298 INFO:tasks.rados.rados.0.smithi085.stdout:3469: finishing write tid 2 to smithi08575397-4 2021-02-10T04:41:38.298 INFO:tasks.rados.rados.0.smithi085.stdout:3469: finishing write tid 3 to smithi08575397-4 2021-02-10T04:41:38.298 INFO:tasks.rados.rados.0.smithi085.stdout:3469: finishing write tid 4 to smithi08575397-4 2021-02-10T04:41:38.298 INFO:tasks.rados.rados.0.smithi085.stdout:3469: finishing write tid 5 to smithi08575397-4 2021-02-10T04:41:38.298 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 4 v 1367 (ObjNum 1188 snap 406 seq_num 1188) dirty exists 2021-02-10T04:41:38.298 INFO:tasks.rados.rados.0.smithi085.stdout:3469: left oid 4 (ObjNum 1188 snap 406 seq_num 1188) 2021-02-10T04:41:38.300 INFO:tasks.rados.rados.0.smithi085.stdout:3466: done (8 left) 2021-02-10T04:41:38.301 INFO:tasks.rados.rados.0.smithi085.stdout:3467: done (7 left) 2021-02-10T04:41:38.301 INFO:tasks.rados.rados.0.smithi085.stdout:3468: done (6 left) 2021-02-10T04:41:38.301 INFO:tasks.rados.rados.0.smithi085.stdout:3469: done (5 left) 2021-02-10T04:41:38.301 INFO:tasks.rados.rados.0.smithi085.stdout:3471: done (4 left) 2021-02-10T04:41:38.301 INFO:tasks.rados.rados.0.smithi085.stdout:3472: read oid 43 snap -1 2021-02-10T04:41:38.301 INFO:tasks.rados.rados.0.smithi085.stdout:3472: expect (ObjNum 1175 snap 401 seq_num 1175) 2021-02-10T04:41:38.301 INFO:tasks.rados.rados.0.smithi085.stdout:3473: snap_remove snap 404 2021-02-10T04:41:38.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:41:38 smithi164 conmon[44544]: debug 2021-02-10T04:41:38.148+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:41:38.413 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:41:38.424 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:41:38.436 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:41:38.436 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:41:38.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:38 smithi085 conmon[35961]: cluster 2021-02-10T04:41:37.423995+0000 2021-02-10T04:41:38.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:38 smithi085 conmon[35961]: mon.a (mon.0) 1834 : cluster [DBG] osdmap e878: 8 total, 8 up, 7 in 2021-02-10T04:41:38.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:38 smithi085 conmon[35961]: audit 2021-02-10T04:41: 2021-02-10T04:41:38.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:38 smithi085 conmon[35961]: 38.139672+0000 mon.a (mon.0) 1835 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:41:38.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:38 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:41:38.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:38 smithi085 conmon[35961]: 04:41:38.139932+0000 mon.a (mon.0 2021-02-10T04:41:38.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:38 smithi085 conmon[35961]: ) 1836 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:41:38.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:38 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:41:38.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:38 smithi085 conmon[35961]: -10T04:41:38.217936+0000 mgr.y 2021-02-10T04:41:38.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:38 smithi085 conmon[35961]: (mgr.14140) 830 : cluster [DBG] pgmap v1468: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 679 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 953 KiB/s wr, 4 op/s 2021-02-10T04:41:38.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:38 smithi085 conmon[35961]: 2021-02-10T04:41:38.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:38 smithi085 conmon[35961]: cluster 2021-02-10T04:41:38.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:38 smithi085 conmon[35961]: 2021-02-10T04:41:38.292102+0000 mon.a (mon 2021-02-10T04:41:38.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:38 smithi085 conmon[35961]: .0) 1837 : cluster [DBG] osdmap e879: 8 total, 8 up, 7 in 2021-02-10T04:41:38.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:38 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:41:38.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:38 smithi085 conmon[35961]: -10T04:41:38.366611+0000 mon.c 2021-02-10T04:41:38.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:38 smithi085 conmon[35961]: (mon.1) 67 : audit [DBG] from='client.? 172.21.15.85:0/2141558857' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:41:38.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:38 smithi085 conmon[35961]: 2021-02-10T04:41:38.450 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:41:38.484 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:41:38.529 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:41:38.530 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:41:38.570 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:41:38.588 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:41:38.604 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:41:38.670 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:41:38.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:38 smithi085 conmon[29319]: cluster 2021-02-10T04:41:37. 2021-02-10T04:41:38.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:38 smithi085 conmon[29319]: 423995+0000 mon.a (mon.0) 1834 : cluster [DBG] osdmap e878: 8 total, 8 up, 7 in 2021-02-10T04:41:38.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:38 smithi085 conmon[29319]: audit 2021-02-10T04:41:38.139672+0000 mon.a (mon.0) 1835 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:41:38.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:38 smithi085 conmon[29319]: audit 2021-02-10T04:41:38.139932+0000 mon.a (mon.0) 1836 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:41:38.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:38 smithi085 conmon[29319]: cluster 2021-02-10T04:41:38.217936+0000 mgr.y (mgr.14140) 830 : cluster [DBG] pgmap v1468: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 679 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 953 KiB/s wr, 4 op/s 2021-02-10T04:41:38.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:38 smithi085 conmon[29319]: cluster 2021-02-10T04:41:38.292102+0000 mon.a (mon.0) 2021-02-10T04:41:38.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:38 smithi085 conmon[29319]: 1837 : cluster [DBG] osdmap e879: 8 total, 8 up, 7 in 2021-02-10T04:41:38.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:38 smithi085 conmon[29319]: audit 2021-02-10T04:41:38.366611+0000 mon.c (mon.1) 67 : audit [DBG] 2021-02-10T04:41:38.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:38 smithi085 conmon[29319]: from='client.? 172.21.15.85:0/2141558857' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:41:38.702 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:41:38.703 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:41:38.703 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:41:38.703 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:41:38.704 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:41:38.704 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:41:38.704 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:41:38.704 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:41:38.704 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:41:38.883 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:41:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:38 smithi164 conmon[29288]: cluster 2021-02-10T04:41:37.423995+0000 mon.a (mon.0 2021-02-10T04:41:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:38 smithi164 conmon[29288]: ) 1834 : cluster [DBG] osdmap e878: 8 total, 8 up, 7 in 2021-02-10T04:41:38.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:38 smithi164 conmon[29288]: audit 2021-02-10T04:41:38.139672+0000 mon.a (mon.0) 1835 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:41:38.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:38 smithi164 conmon[29288]: 2021-02-10T04:41:38.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:38 smithi164 conmon[29288]: audit 2021-02-10T04:41: 2021-02-10T04:41:38.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:38 smithi164 conmon[29288]: 38.139932+0000 mon.a (mon.0) 1836 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:41:38.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:38 smithi164 conmon[29288]: cluster 2021-02-10T04:41:38.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:38 smithi164 conmon[29288]: 2021-02-10T04:41:38.217936 2021-02-10T04:41:38.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:38 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 830 : cluster [DBG] 2021-02-10T04:41:38.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:38 smithi164 conmon[29288]: pgmap v1468: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 679 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 953 KiB/s wr, 4 op/s 2021-02-10T04:41:38.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:38 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:41:38.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:38 smithi164 conmon[29288]: 41:38.292102+0000 mon.a (mon.0 2021-02-10T04:41:38.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:38 smithi164 conmon[29288]: ) 1837 : cluster [DBG] osdmap e879: 8 total, 8 up, 7 in 2021-02-10T04:41:38.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:38 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:41:38.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:38 smithi164 conmon[29288]: 02-10T04:41:38.366611+0000 2021-02-10T04:41:38.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:38 smithi164 conmon[29288]: mon.c (mon.1) 67 : audit [DBG] from='client.? 172.21.15.85:0/2141558857' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:41:38.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:38 smithi164 conmon[29288]: 2021-02-10T04:41:38.897 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:41:38 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:41:39.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:41:38 smithi085 conmon[49106]: debug 2021-02-10T04:41:38.696+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:41:39.305 INFO:tasks.rados.rados.0.smithi085.stdout:3473: done (5 left) 2021-02-10T04:41:39.305 INFO:tasks.rados.rados.0.smithi085.stdout:3474: write oid 36 current snap is 406 2021-02-10T04:41:39.305 INFO:tasks.rados.rados.0.smithi085.stdout:3474: seq_num 1190 ranges {400330=497378,1485925=514405} 2021-02-10T04:41:39.310 INFO:tasks.rados.rados.0.smithi085.stdout:3474: writing smithi08575397-36 from 400330 to 897708 tid 1 2021-02-10T04:41:39.318 INFO:tasks.rados.rados.0.smithi085.stdout:3474: writing smithi08575397-36 from 1485925 to 2000330 tid 2 2021-02-10T04:41:39.318 INFO:tasks.rados.rados.0.smithi085.stdout:3472: expect (ObjNum 1175 snap 401 seq_num 1175) 2021-02-10T04:41:39.326 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.5 to 1 (10000) 2021-02-10T04:41:39.334 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is unset 2021-02-10T04:41:39.521 INFO:tasks.rados.rados.0.smithi085.stdout:3472: done (5 left) 2021-02-10T04:41:39.545 INFO:tasks.rados.rados.0.smithi085.stdout:3475: rollback oid 42 current snap is 406 2021-02-10T04:41:39.545 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 42 to 344 2021-02-10T04:41:39.545 INFO:tasks.rados.rados.0.smithi085.stdout:3476: snap_create 2021-02-10T04:41:39.545 INFO:tasks.rados.rados.0.smithi085.stdout:3474: finishing write tid 1 to smithi08575397-36 2021-02-10T04:41:39.545 INFO:tasks.rados.rados.0.smithi085.stdout:3474: finishing write tid 2 to smithi08575397-36 2021-02-10T04:41:39.546 INFO:tasks.rados.rados.0.smithi085.stdout:3474: finishing write tid 3 to smithi08575397-36 2021-02-10T04:41:39.546 INFO:tasks.rados.rados.0.smithi085.stdout:3474: finishing write tid 4 to smithi08575397-36 2021-02-10T04:41:39.546 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 1353 (ObjNum 1190 snap 406 seq_num 1190) dirty exists 2021-02-10T04:41:39.546 INFO:tasks.rados.rados.0.smithi085.stdout:3474: left oid 36 (ObjNum 1190 snap 406 seq_num 1190) 2021-02-10T04:41:39.546 INFO:tasks.rados.rados.0.smithi085.stdout:3475: finishing rollback tid 0 to smithi08575397-42 2021-02-10T04:41:39.547 INFO:tasks.rados.rados.0.smithi085.stdout:3475: finishing rollback tid 1 to smithi08575397-42 2021-02-10T04:41:39.547 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 1526 (ObjNum 0 snap 22030 seq_num 83071344) dirty dne 2021-02-10T04:41:39.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[33677]: debug 2021-02-10T04:41:39.320+0000 7fbec19a1700 -1 osd.4 879 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:41:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[29288]: audit 2021-02-10T04:41:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:41:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[29288]: 41:38.468054+0000 mgr.y (mgr.14140) 831 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:41:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:41:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[29288]: 04:41:38.683705+0000 mon.c (mon.1) 68 2021-02-10T04:41:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[29288]: : audit [INF] from='client.? 172.21.15.85:0/4198892379' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:41:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:41:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[29288]: :41:38.684875+0000 mon.a (mon. 2021-02-10T04:41:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[29288]: 0) 1838 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:41:39.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:41:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[29288]: 02-10T04:41:38. 2021-02-10T04:41:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[29288]: 690889+0000 mon.a (mon.0) 2021-02-10T04:41:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[29288]: 1839 : audit [INF] from='client.? 172.21.15.85:0/2861616405' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-02-10T04:41:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:41:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[29288]: 10T04:41:39.298979+0000 mon.a (mon. 2021-02-10T04:41:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[29288]: 0) 1840 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:41:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:41:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[29288]: -10T04:41:39.299018+0000 mon.a 2021-02-10T04:41:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[29288]: (mon.0) 1841 : cluster [INF] Cluster is now healthy 2021-02-10T04:41:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:41:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[29288]: T04:41:39.309474+0000 mon.a ( 2021-02-10T04:41:39.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[29288]: mon.0) 1842 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:41:39.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:41:39.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[29288]: -10T04:41:39.309659+0000 2021-02-10T04:41:39.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[29288]: mon.a (mon.0) 1843 : audit [INF] from='client.? 172.21.15.85:0/2861616405' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-02-10T04:41:39.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[29288]: cluster 2021-02-10T04:41:39.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[29288]: 2021-02-10T04:41 2021-02-10T04:41:39.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[29288]: :39.309812+0000 mon.a (mon. 2021-02-10T04:41:39.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[29288]: 0) 1844 : cluster [DBG] osdmap e880: 8 total, 8 up, 7 in 2021-02-10T04:41:39.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:39 smithi164 conmon[29288]: 2021-02-10T04:41:39.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[29319]: audit 2021-02-10T04:41:38.468054+0000 mgr.y 2021-02-10T04:41:39.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[29319]: (mgr.14140) 831 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:41:39.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[29319]: audit 2021-02-10T04:41:38.683705+0000 mon.c (mon.1) 68 : audit [INF] from='client.? 172.21.15.85:0/4198892379' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:41:39.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[29319]: audit 2021-02-10T04:41:38.684875+0000 mon.a (mon.0) 1838 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:41:39.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[29319]: audit 2021-02-10T04:41:38.690889+0000 mon.a (mon.0) 1839 : audit [INF] from='client.? 172.21.15.85:0/2861616405' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-02-10T04:41:39.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[29319]: cluster 2021-02-10T04:41:39. 2021-02-10T04:41:39.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[29319]: 298979+0000 mon.a (mon.0) 1840 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:41:39.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:41:39.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[29319]: 10T04:41:39.299018+0000 mon.a 2021-02-10T04:41:39.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[29319]: (mon.0) 1841 : cluster [INF] Cluster is now healthy 2021-02-10T04:41:39.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:41:39.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[29319]: -10T04:41:39.309474+0000 2021-02-10T04:41:39.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[29319]: mon.a (mon.0) 1842 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:41:39.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[29319]: 2021-02-10T04:41:39.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:41:39.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[29319]: 02-10T04:41:39.309659+0000 mon.a ( 2021-02-10T04:41:39.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[29319]: mon.0) 1843 : audit [INF] 2021-02-10T04:41:39.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[29319]: from='client.? 172.21.15.85:0/2861616405' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-02-10T04:41:39.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:41:39.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[29319]: -02-10T04:41 2021-02-10T04:41:39.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[29319]: :39.309812+0000 mon.a (mon. 2021-02-10T04:41:39.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[29319]: 0) 1844 : cluster [DBG] osdmap e880: 8 total, 8 up, 7 in 2021-02-10T04:41:39.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[29319]: 2021-02-10T04:41:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[35961]: audit 2021-02-10T04:41:38.468054+0000 2021-02-10T04:41:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[35961]: mgr.y (mgr.14140) 831 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:41:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[35961]: audit 2021-02-10T04:41:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[35961]: 2021-02-10T04:41:38.683705+0000 mon.c (mon.1) 68 : audit [INF] from='client.? 172.21.15.85:0/4198892379' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:41:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[35961]: 2021-02-10T04:41:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[35961]: audit 2021-02-10T04:41 2021-02-10T04:41:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[35961]: :38.684875+0000 mon.a (mon.0) 1838 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:41:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[35961]: audit 2021-02-10T04:41:38 2021-02-10T04:41:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[35961]: .690889+0000 mon.a (mon.0) 1839 : audit [INF] from='client.? 172.21.15.85:0/2861616405' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-02-10T04:41:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:41:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[35961]: -10T04:41:39.298979+0000 mon.a (mon.0) 1840 : 2021-02-10T04:41:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[35961]: cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:41:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[35961]: cluster 2021-02-10T04:41:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[35961]: 2021-02-10T04:41:39.299018+0000 mon.a (mon.0) 1841 : cluster [INF] 2021-02-10T04:41:39.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[35961]: Cluster is now healthy 2021-02-10T04:41:39.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[35961]: audit 2021 2021-02-10T04:41:39.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[35961]: -02-10T04:41:39.309474+0000 mon.a (mon.0) 1842 : audit [INF] 2021-02-10T04:41:39.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[35961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:41:39.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[35961]: audit 2021-02-10T04:41:39.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[35961]: 2021-02-10T04:41:39.309659+0000 mon.a (mon.0) 1843 : audit [INF] 2021-02-10T04:41:39.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/2861616405' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-02-10T04:41:39.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:41:39.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[35961]: -10T04:41:39.309812+0000 mon.a (mon.0) 1844 : cluster [DBG] osdmap e880: 8 total, 8 up, 7 in 2021-02-10T04:41:39.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:39 smithi085 conmon[35961]: 2021-02-10T04:41:40.453 INFO:tasks.rados.rados.0.smithi085.stdout:3474: done (6 left) 2021-02-10T04:41:40.453 INFO:tasks.rados.rados.0.smithi085.stdout:3475: done (5 left) 2021-02-10T04:41:40.453 INFO:tasks.rados.rados.0.smithi085.stdout:3476: done (4 left) 2021-02-10T04:41:40.453 INFO:tasks.rados.rados.0.smithi085.stdout:3477: read oid 7 snap -1 2021-02-10T04:41:40.453 INFO:tasks.rados.rados.0.smithi085.stdout:3477: expect (ObjNum 1176 snap 401 seq_num 1176) 2021-02-10T04:41:40.457 INFO:tasks.rados.rados.0.smithi085.stdout:3478: write oid 16 current snap is 407 2021-02-10T04:41:40.457 INFO:tasks.rados.rados.0.smithi085.stdout:3478: seq_num 1191 ranges {417137=572627,1428054=589083} 2021-02-10T04:41:40.459 INFO:tasks.rados.rados.0.smithi085.stdout:3478: writing smithi08575397-16 from 417137 to 989764 tid 1 2021-02-10T04:41:40.468 INFO:tasks.rados.rados.0.smithi085.stdout:3478: writing smithi08575397-16 from 1428054 to 2017137 tid 2 2021-02-10T04:41:40.468 INFO:tasks.rados.rados.0.smithi085.stdout:3479: snap_create 2021-02-10T04:41:40.469 INFO:tasks.rados.rados.0.smithi085.stdout:3477: expect (ObjNum 1176 snap 401 seq_num 1176) 2021-02-10T04:41:40.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:40 smithi164 conmon[33677]: debug 2021-02-10T04:41:40.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:40 smithi164 conmon[33677]: 2021-02-10T04:41:40.316+0000 7fbec19a1700 -1 osd.4 880 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:41:40.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:40 smithi164 conmon[33677]: 2021-02-10T04:41:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:40 smithi164 conmon[29288]: cluster 2021-02-10T04:41:39.322222+0000 osd.4 (osd. 2021-02-10T04:41:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:40 smithi164 conmon[29288]: 4) 1437 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:40 smithi164 conmon[29288]: cluster 2021-02-10T04:41:39.322228+0000 osd.4 (osd.4) 1438 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:40 smithi164 conmon[29288]: cluster 2021-02-10T04:41:39.322232+0000 osd.4 (osd.4) 1439 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:40 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:41:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:40 smithi164 conmon[29288]: 10T04:41:39.322236+0000 osd.4 (osd. 2021-02-10T04:41:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:40 smithi164 conmon[29288]: 4) 1440 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:40 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:41:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:40 smithi164 conmon[29288]: 04:41:40.218748+0000 mgr.y (mgr.14140) 832 : 2021-02-10T04:41:40.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:40 smithi164 conmon[29288]: cluster [DBG] pgmap v1471: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 690 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 4.0 MiB/s rd, 2.4 MiB/s wr, 15 op/s 2021-02-10T04:41:40.650 INFO:tasks.rados.rados.0.smithi085.stdout:3478: finishing write tid 1 to smithi08575397-16 2021-02-10T04:41:40.650 INFO:tasks.rados.rados.0.smithi085.stdout:3478: finishing write tid 2 to smithi08575397-16 2021-02-10T04:41:40.650 INFO:tasks.rados.rados.0.smithi085.stdout:3478: finishing write tid 3 to smithi08575397-16 2021-02-10T04:41:40.650 INFO:tasks.rados.rados.0.smithi085.stdout:3478: finishing write tid 4 to smithi08575397-16 2021-02-10T04:41:40.650 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 1801 (ObjNum 1191 snap 407 seq_num 1191) dirty exists 2021-02-10T04:41:40.650 INFO:tasks.rados.rados.0.smithi085.stdout:3478: left oid 16 (ObjNum 1191 snap 407 seq_num 1191) 2021-02-10T04:41:40.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[29319]: cluster 2021-02-10T04:41:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[29319]: 2021-02-10T04:41:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:41:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[29319]: :41:39.322222+0000 osd.4 (osd.4) 1437 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[29319]: cluster 2021-02-10T04:41:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[29319]: 2021- 2021-02-10T04:41:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[29319]: 02-10T04:41:39.322228+0000 osd.4 (osd.4 2021-02-10T04:41:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[29319]: ) 1438 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:41:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[29319]: -02-10T04:41:39.322232+0000 2021-02-10T04:41:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[29319]: osd.4 (osd.4) 1439 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[29319]: 2021-02-10T04:41:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[29319]: cluster 2021-02-10T04:41:40.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[29319]: 2021-02-10T04:41:39.322236+0000 osd.4 ( 2021-02-10T04:41:40.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[29319]: osd.4) 1440 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:40.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:41:40.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[29319]: -02-10T04:41:40.218748+0000 2021-02-10T04:41:40.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[29319]: mgr.y (mgr.14140) 2021-02-10T04:41:40.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[29319]: 832 : cluster [DBG] pgmap v1471: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 690 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 4.0 MiB/s rd, 2.4 MiB/s wr, 15 op/s 2021-02-10T04:41:40.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[35961]: cluster 2021-02-10T04:41:40.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[35961]: 2021-02-10T04:41:40.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[35961]: 2021 2021-02-10T04:41:40.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[35961]: - 2021-02-10T04:41:40.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[35961]: 02 2021-02-10T04:41:40.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[35961]: -10T04:41:39 2021-02-10T04:41:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[35961]: .322222+0000 osd.4 (osd.4) 1437 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[35961]: cluster 2021-02-10T04:41:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[35961]: 2021-02-10T04:41:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:41:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[35961]: 04:41:39.322228+0000 osd.4 (osd.4) 1438 : cluster 2021-02-10T04:41:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[35961]: cluster 2021-02-10T04:41 2021-02-10T04:41:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[35961]: :39.322232+0000 osd.4 (osd.4) 1439 2021-02-10T04:41:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:41:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[35961]: T04:41:39.322236+0000 osd.4 (osd 2021-02-10T04:41:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[35961]: .4) 1440 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:41:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[35961]: 04:41:40.218748+0000 mgr.y (mgr 2021-02-10T04:41:40.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:40 smithi085 conmon[35961]: .14140) 832 : cluster [DBG] pgmap v1471: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 690 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 4.0 MiB/s rd, 2.4 MiB/s wr, 15 op/s 2021-02-10T04:41:41.461 INFO:tasks.rados.rados.0.smithi085.stdout:3477: done (6 left) 2021-02-10T04:41:41.461 INFO:tasks.rados.rados.0.smithi085.stdout:3478: done (5 left) 2021-02-10T04:41:41.461 INFO:tasks.rados.rados.0.smithi085.stdout:3479: done (4 left) 2021-02-10T04:41:41.462 INFO:tasks.rados.rados.0.smithi085.stdout:3480: read oid 48 snap 380 2021-02-10T04:41:41.462 INFO:tasks.rados.rados.0.smithi085.stdout:3480: expect (ObjNum 1117 snap 378 seq_num 1117) 2021-02-10T04:41:41.462 INFO:tasks.rados.rados.0.smithi085.stdout:3481: read oid 37 snap -1 2021-02-10T04:41:41.462 INFO:tasks.rados.rados.0.smithi085.stdout:3481: expect (ObjNum 1129 snap 381 seq_num 1129) 2021-02-10T04:41:41.462 INFO:tasks.rados.rados.0.smithi085.stdout:3482: delete oid 9 current snap is 408 2021-02-10T04:41:41.475 INFO:tasks.rados.rados.0.smithi085.stdout:3481: expect (ObjNum 1129 snap 381 seq_num 1129) 2021-02-10T04:41:41.608 INFO:tasks.rados.rados.0.smithi085.stdout:3481: done (6 left) 2021-02-10T04:41:41.609 INFO:tasks.rados.rados.0.smithi085.stdout:3482: done (5 left) 2021-02-10T04:41:41.609 INFO:tasks.rados.rados.0.smithi085.stdout:3483: rollback oid 6 current snap is 408 2021-02-10T04:41:41.609 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 6 to 387 2021-02-10T04:41:41.609 INFO:tasks.rados.rados.0.smithi085.stdout:3480: expect (ObjNum 1117 snap 378 seq_num 1117) 2021-02-10T04:41:41.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:41 smithi164 conmon[33677]: debug 2021-02-10T04:41:41.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:41 smithi164 conmon[33677]: 2021-02-10T04:41:41.323+0000 7fbec19a1700 -1 osd.4 881 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:41:41.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:41 smithi164 conmon[33677]: 2021-02-10T04:41:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:41 smithi164 conmon[29288]: cluster 2021-02-10T04:41:40.318028 2021-02-10T04:41:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:41 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1441 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:41 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:41:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:41 smithi164 conmon[29288]: :41:40.318038+0000 osd.4 (osd.4) 1442 : cluster 2021-02-10T04:41:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:41 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:41 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:41:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:41 smithi164 conmon[29288]: 04:41:40.318045+0000 2021-02-10T04:41:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:41 smithi164 conmon[29288]: osd.4 (osd.4) 1443 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:41 smithi164 conmon[29288]: 2021-02-10T04:41:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:41 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:41:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:41 smithi164 conmon[29288]: -02-10T04:41:40.318050+0000 2021-02-10T04:41:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:41 smithi164 conmon[29288]: osd.4 (osd.4 2021-02-10T04:41:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:41 smithi164 conmon[29288]: ) 1444 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:41 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:41:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:41 smithi164 conmon[29288]: -02-10T04:41:40.452283+0000 2021-02-10T04:41:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:41 smithi164 conmon[29288]: mon.a (mon.0) 1845 : cluster [DBG] osdmap e881: 8 total, 8 up, 7 in 2021-02-10T04:41:41.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:41 smithi164 conmon[29288]: 2021-02-10T04:41:41.732 INFO:tasks.rados.rados.0.smithi085.stdout:3480: done (5 left) 2021-02-10T04:41:41.732 INFO:tasks.rados.rados.0.smithi085.stdout:3484: read oid 30 snap -1 2021-02-10T04:41:41.732 INFO:tasks.rados.rados.0.smithi085.stdout:3484: expect (ObjNum 1025 snap 335 seq_num 1025) 2021-02-10T04:41:41.732 INFO:tasks.rados.rados.0.smithi085.stdout:3485: write oid 12 current snap is 408 2021-02-10T04:41:41.732 INFO:tasks.rados.rados.0.smithi085.stdout:3485: seq_num 1193 ranges {450751=723125,1945018=105733} 2021-02-10T04:41:41.738 INFO:tasks.rados.rados.0.smithi085.stdout:3485: writing smithi08575397-12 from 450751 to 1173876 tid 1 2021-02-10T04:41:41.744 INFO:tasks.rados.rados.0.smithi085.stdout:3485: writing smithi08575397-12 from 1945018 to 2050751 tid 2 2021-02-10T04:41:41.744 INFO:tasks.rados.rados.0.smithi085.stdout:3486: copy_from oid 5 from oid 31 current snap is 408 2021-02-10T04:41:41.745 INFO:tasks.rados.rados.0.smithi085.stdout:3483: finishing rollback tid 0 to smithi08575397-6 2021-02-10T04:41:41.745 INFO:tasks.rados.rados.0.smithi085.stdout:3483: finishing rollback tid 1 to smithi08575397-6 2021-02-10T04:41:41.745 INFO:tasks.rados.rados.0.smithi085.stdout:3483: finishing rollback tid 2 to smithi08575397-6 2021-02-10T04:41:41.745 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 1586 (ObjNum 1138 snap 382 seq_num 1138) dirty exists 2021-02-10T04:41:41.745 INFO:tasks.rados.rados.0.smithi085.stdout:3484: expect (ObjNum 1025 snap 335 seq_num 1025) 2021-02-10T04:41:41.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[29319]: cluster 2021-02-10T04:41:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[29319]: 2021-02-10T04:41:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[29319]: 2021 2021-02-10T04:41:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[29319]: -02-10T04:41:40. 2021-02-10T04:41:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[29319]: 318028+0000 osd.4 (osd.4) 1441 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:41:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[29319]: T04:41:40.318038+0000 osd.4 (osd. 2021-02-10T04:41:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[29319]: 4) 1442 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[29319]: cluster 2021-02-10T04:41:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[29319]: 2021-02-10T04:41 2021-02-10T04:41:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[29319]: :40.318045+0000 osd.4 (osd 2021-02-10T04:41:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[29319]: .4) 1443 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[29319]: cluster 2021-02-10T04:41:41.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[29319]: 2021-02-10T04:41:40.318050 2021-02-10T04:41:41.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1444 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:41.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[29319]: 2021-02-10T04:41:41.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:41:41.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[29319]: 04:41:40.452283+0000 mon.a (mon.0 2021-02-10T04:41:41.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[29319]: ) 1845 : cluster [DBG] osdmap e881: 8 total, 8 up, 7 in 2021-02-10T04:41:41.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[35961]: cluster 2021-02-10T04:41:41.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:41:41.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[35961]: :41:40.318028+0000 osd.4 (osd.4) 1441 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:41.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[35961]: cluster 2021-02-10T04:41:41.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[35961]: 2021-02-10T04:41:41.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:41:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[35961]: :41:40.318038+0000 osd.4 (osd.4) 1442 : cluster 2021-02-10T04:41:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:41:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[35961]: 02-10T04:41:40.318045+0000 2021-02-10T04:41:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[35961]: osd.4 (osd.4) 1443 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[35961]: 2021-02-10T04:41:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[35961]: cluster 2021-02-10T04:41 2021-02-10T04:41:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[35961]: :40.318050+0000 osd.4 (osd.4) 2021-02-10T04:41:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[35961]: 1444 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:41:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[35961]: T04:41:40.452283+0000 mon.a (mon 2021-02-10T04:41:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:41 smithi085 conmon[35961]: .0) 1845 : cluster [DBG] osdmap e881: 8 total, 8 up, 7 in 2021-02-10T04:41:41.953 INFO:tasks.rados.rados.0.smithi085.stdout:3483: done (7 left) 2021-02-10T04:41:41.954 INFO:tasks.rados.rados.0.smithi085.stdout:3484: done (6 left) 2021-02-10T04:41:41.954 INFO:tasks.rados.rados.0.smithi085.stdout:3487: delete oid 1 current snap is 408 2021-02-10T04:41:41.954 INFO:tasks.rados.rados.0.smithi085.stdout:3485: finishing write tid 1 to smithi08575397-12 2021-02-10T04:41:41.954 INFO:tasks.rados.rados.0.smithi085.stdout:3485: finishing write tid 2 to smithi08575397-12 2021-02-10T04:41:41.954 INFO:tasks.rados.rados.0.smithi085.stdout:3485: finishing write tid 3 to smithi08575397-12 2021-02-10T04:41:41.954 INFO:tasks.rados.rados.0.smithi085.stdout:3485: finishing write tid 4 to smithi08575397-12 2021-02-10T04:41:41.954 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 1530 (ObjNum 1193 snap 408 seq_num 1193) dirty exists 2021-02-10T04:41:41.954 INFO:tasks.rados.rados.0.smithi085.stdout:3485: left oid 12 (ObjNum 1193 snap 408 seq_num 1193) 2021-02-10T04:41:41.954 INFO:tasks.rados.rados.0.smithi085.stdout:3486: finishing copy_from to smithi08575397-5 2021-02-10T04:41:41.954 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 1483 (ObjNum 1097 snap 368 seq_num 1097) dirty exists 2021-02-10T04:41:41.954 INFO:tasks.rados.rados.0.smithi085.stdout:3486: finishing copy_from racing read to smithi08575397-5 2021-02-10T04:41:41.958 INFO:tasks.rados.rados.0.smithi085.stdout:3485: done (6 left) 2021-02-10T04:41:41.958 INFO:tasks.rados.rados.0.smithi085.stdout:3486: done (5 left) 2021-02-10T04:41:41.958 INFO:tasks.rados.rados.0.smithi085.stdout:3487: done (4 left) 2021-02-10T04:41:41.958 INFO:tasks.rados.rados.0.smithi085.stdout:3488: read oid 25 snap -1 2021-02-10T04:41:41.958 INFO:tasks.rados.rados.0.smithi085.stdout:3488: expect deleted 2021-02-10T04:41:41.958 INFO:tasks.rados.rados.0.smithi085.stdout:3489: write (excl) oid 35 current snap is 408 2021-02-10T04:41:41.958 INFO:tasks.rados.rados.0.smithi085.stdout:3489: seq_num 1195 ranges {484365=589976,1661982=422383} 2021-02-10T04:41:41.964 INFO:tasks.rados.rados.0.smithi085.stdout:3489: writing smithi08575397-35 from 484365 to 1074341 tid 1 2021-02-10T04:41:41.971 INFO:tasks.rados.rados.0.smithi085.stdout:3489: writing smithi08575397-35 from 1661982 to 2084365 tid 2 2021-02-10T04:41:41.971 INFO:tasks.rados.rados.0.smithi085.stdout:3489: finishing write tid 1 to smithi08575397-35 2021-02-10T04:41:41.971 INFO:tasks.rados.rados.0.smithi085.stdout:3490: read oid 4 snap -1 2021-02-10T04:41:41.971 INFO:tasks.rados.rados.0.smithi085.stdout:3490: expect (ObjNum 1188 snap 406 seq_num 1188) 2021-02-10T04:41:41.971 INFO:tasks.rados.rados.0.smithi085.stdout:3488: done (6 left) 2021-02-10T04:41:41.971 INFO:tasks.rados.rados.0.smithi085.stdout:3491: delete oid 23 current snap is 408 2021-02-10T04:41:41.974 INFO:tasks.rados.rados.0.smithi085.stdout:3491: done (6 left) 2021-02-10T04:41:41.974 INFO:tasks.rados.rados.0.smithi085.stdout:3492: snap_create 2021-02-10T04:41:41.978 INFO:tasks.rados.rados.0.smithi085.stdout:3489: finishing write tid 2 to smithi08575397-35 2021-02-10T04:41:41.979 INFO:tasks.rados.rados.0.smithi085.stdout:3490: expect (ObjNum 1188 snap 406 seq_num 1188) 2021-02-10T04:41:42.137 INFO:tasks.rados.rados.0.smithi085.stdout:3489: finishing write tid 3 to smithi08575397-35 2021-02-10T04:41:42.138 INFO:tasks.rados.rados.0.smithi085.stdout:3489: finishing write tid 4 to smithi08575397-35 2021-02-10T04:41:42.138 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 35 v 1587 (ObjNum 1195 snap 408 seq_num 1195) dirty exists 2021-02-10T04:41:42.138 INFO:tasks.rados.rados.0.smithi085.stdout:3489: left oid 35 (ObjNum 1195 snap 408 seq_num 1195) 2021-02-10T04:41:42.468 INFO:tasks.rados.rados.0.smithi085.stdout:3489: done (6 left) 2021-02-10T04:41:42.468 INFO:tasks.rados.rados.0.smithi085.stdout:3490: done (5 left) 2021-02-10T04:41:42.468 INFO:tasks.rados.rados.0.smithi085.stdout:3492: done (4 left) 2021-02-10T04:41:42.469 INFO:tasks.rados.rados.0.smithi085.stdout:3493: rollback oid 23 current snap is 409 2021-02-10T04:41:42.469 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 23 to 354 2021-02-10T04:41:42.469 INFO:tasks.rados.rados.0.smithi085.stdout:3494: write (excl) oid 50 current snap is 409 2021-02-10T04:41:42.469 INFO:tasks.rados.rados.0.smithi085.stdout:3494: seq_num 1197 ranges {517979=740474,1778946=339033} 2021-02-10T04:41:42.476 INFO:tasks.rados.rados.0.smithi085.stdout:3494: writing smithi08575397-50 from 517979 to 1258453 tid 1 2021-02-10T04:41:42.481 INFO:tasks.rados.rados.0.smithi085.stdout:3494: writing smithi08575397-50 from 1778946 to 2117979 tid 2 2021-02-10T04:41:42.486 INFO:tasks.rados.rados.0.smithi085.stdout:3495: write (excl) oid 15 current snap is 409 2021-02-10T04:41:42.486 INFO:tasks.rados.rados.0.smithi085.stdout:3493: finishing rollback tid 1 to smithi08575397-23 2021-02-10T04:41:42.486 INFO:tasks.rados.rados.0.smithi085.stdout:3495: seq_num 1198 ranges {534786=415723,1437428=697358} 2021-02-10T04:41:42.488 INFO:tasks.rados.rados.0.smithi085.stdout:3495: writing smithi08575397-15 from 534786 to 950509 tid 1 2021-02-10T04:41:42.497 INFO:tasks.rados.rados.0.smithi085.stdout:3495: writing smithi08575397-15 from 1437428 to 2134786 tid 2 2021-02-10T04:41:42.497 INFO:tasks.rados.rados.0.smithi085.stdout:3496: read oid 13 snap -1 2021-02-10T04:41:42.497 INFO:tasks.rados.rados.0.smithi085.stdout:3496: expect (ObjNum 1178 snap 402 seq_num 1178) 2021-02-10T04:41:42.497 INFO:tasks.rados.rados.0.smithi085.stdout:3497: rollback oid 46 current snap is 409 2021-02-10T04:41:42.497 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 46 to 278 2021-02-10T04:41:42.497 INFO:tasks.rados.rados.0.smithi085.stdout:3493: finishing rollback tid 2 to smithi08575397-23 2021-02-10T04:41:42.497 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 1356 (ObjNum 924 snap 306 seq_num 924) dirty exists 2021-02-10T04:41:42.497 INFO:tasks.rados.rados.0.smithi085.stdout:3493: done (8 left) 2021-02-10T04:41:42.498 INFO:tasks.rados.rados.0.smithi085.stdout:3498: read oid 25 snap -1 2021-02-10T04:41:42.498 INFO:tasks.rados.rados.0.smithi085.stdout:3498: expect deleted 2021-02-10T04:41:42.498 INFO:tasks.rados.rados.0.smithi085.stdout:3499: read oid 4 snap 395 2021-02-10T04:41:42.498 INFO:tasks.rados.rados.0.smithi085.stdout:3499: expect deleted 2021-02-10T04:41:42.498 INFO:tasks.rados.rados.0.smithi085.stdout:3500: snap_remove snap 352 2021-02-10T04:41:42.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:42 smithi164 conmon[33677]: debug 2021-02-10T04:41:42.329+0000 7fbec19a1700 -1 osd.4 882 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:41:42.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:42 smithi164 conmon[29288]: cluster 2021-02-10T04:41:41.325181+0000 osd.4 (osd.4) 1445 : cluster 2021-02-10T04:41:42.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:42 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:42.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:42 smithi164 conmon[29288]: cluster 2021-02-10T04:41:41.325190+0000 osd.4 (osd.4) 1446 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:42.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:42 smithi164 conmon[29288]: cluster 2021-02-10T04:41:41.325213+0000 osd.4 2021-02-10T04:41:42.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:42 smithi164 conmon[29288]: (osd.4) 1447 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:42.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:42 smithi164 conmon[29288]: cluster 2021-02-10T04:41:41 2021-02-10T04:41:42.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:42 smithi164 conmon[29288]: .325218+0000 osd.4 (osd.4) 1448 2021-02-10T04:41:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:42 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:42 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:41:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:42 smithi164 conmon[29288]: 02-10T04:41:41.460978+0000 mon.a (mon.0) 1846 2021-02-10T04:41:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:42 smithi164 conmon[29288]: : cluster [DBG] osdmap e882: 8 total, 8 up, 7 in 2021-02-10T04:41:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:42 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:41:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:42 smithi164 conmon[29288]: 02-10T04:41:42. 2021-02-10T04:41:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:42 smithi164 conmon[29288]: 100019+0000 osd.7 (osd.7) 64 2021-02-10T04:41:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:42 smithi164 conmon[29288]: : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:41:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:42 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:41:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:42 smithi164 conmon[29288]: 04:41:42.100422+0000 osd.7 (osd. 2021-02-10T04:41:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:42 smithi164 conmon[29288]: 7) 65 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:41:42.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:42 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:41:42.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:42 smithi164 conmon[29288]: 04: 2021-02-10T04:41:42.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:42 smithi164 conmon[29288]: 41:42.219423 2021-02-10T04:41:42.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:42 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 2021-02-10T04:41:42.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:42 smithi164 conmon[29288]: 833 : cluster [DBG] pgmap v1474: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 690 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 4.0 MiB/s rd, 1.9 MiB/s wr, 13 op/s 2021-02-10T04:41:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[29319]: cluster 2021-02-10T04:41:41.325181+0000 osd.4 ( 2021-02-10T04:41:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[29319]: osd.4) 1445 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[29319]: cluster 2021-02-10T04:41:41.325190+0000 osd.4 (osd.4) 1446 : cluster 2021-02-10T04:41:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:41:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[29319]: 04:41:41.325213+0000 osd.4 (osd.4) 1447 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:41:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[29319]: 02-10T04:41:41.325218+0000 osd.4 2021-02-10T04:41:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[29319]: (osd.4) 1448 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:41:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[29319]: -02-10T04:41:41.460978+0000 2021-02-10T04:41:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[29319]: mon.a (mon.0) 1846 : cluster [DBG] osdmap e882: 8 total, 8 up, 7 in 2021-02-10T04:41:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[29319]: 2021-02-10T04:41:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[29319]: cluster 2021-02-10T04:41:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[29319]: 2021-02-10T04:41:42.100019+0000 osd.7 2021-02-10T04:41:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[29319]: (osd.7) 64 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:41:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:41:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[29319]: -02-10T04:41:42.100422+0000 2021-02-10T04:41:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[29319]: osd.7 (osd.7) 65 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:41:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:41:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[29319]: -02-10T04:41:42.219423+0000 2021-02-10T04:41:42.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[29319]: mgr.y (mgr.14140) 833 : cluster 2021-02-10T04:41:42.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[29319]: [DBG] pgmap v1474: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 690 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 4.0 MiB/s rd, 1.9 MiB/s wr, 13 op/s 2021-02-10T04:41:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[35961]: cluster 2021-02-10T04:41:41.325181+0000 2021-02-10T04:41:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[35961]: osd.4 (osd.4) 1445 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[35961]: cluster 2021-02-10T04:41:41 2021-02-10T04:41:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[35961]: .325190+0000 osd.4 (osd.4) 1446 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[35961]: cluster 2021-02-10T04:41:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[35961]: 2021-02-10T04:41:41 2021-02-10T04:41:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[35961]: .325213+0000 osd.4 (osd.4) 1447 : cluster 2021-02-10T04:41:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[35961]: cluster 2021-02-10T04:41 2021-02-10T04:41:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[35961]: :41.325218+0000 osd.4 (osd.4) 1448 2021-02-10T04:41:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:41:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[35961]: 41:41.460978+0000 mon.a (mon.0 2021-02-10T04:41:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[35961]: ) 1846 : cluster [DBG] osdmap e882: 8 total, 8 up, 7 in 2021-02-10T04:41:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:41:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[35961]: -02-10T04:41:42.100019+0000 2021-02-10T04:41:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[35961]: osd.7 (osd.7) 64 : cluster [DBG] 2021-02-10T04:41:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[35961]: purged_snaps scrub starts 2021-02-10T04:41:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[35961]: cluster 2021-02-10T04:41:42 2021-02-10T04:41:42.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[35961]: .100422+0000 osd.7 (osd.7) 65 : 2021-02-10T04:41:42.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[35961]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:41:42.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:41:42.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[35961]: 04:41:42.219423+0000 mgr.y (mgr. 2021-02-10T04:41:42.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:42 smithi085 conmon[35961]: 14140) 833 : cluster [DBG] pgmap v1474: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 690 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 4.0 MiB/s rd, 1.9 MiB/s wr, 13 op/s 2021-02-10T04:41:43.474 INFO:tasks.rados.rados.0.smithi085.stdout:3500: done (10 left) 2021-02-10T04:41:43.474 INFO:tasks.rados.rados.0.smithi085.stdout:3501: rollback oid 42 current snap is 409 2021-02-10T04:41:43.474 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 42 to 396 2021-02-10T04:41:43.474 INFO:tasks.rados.rados.0.smithi085.stdout:3494: finishing write tid 1 to smithi08575397-50 2021-02-10T04:41:43.474 INFO:tasks.rados.rados.0.smithi085.stdout:3494: finishing write tid 2 to smithi08575397-50 2021-02-10T04:41:43.474 INFO:tasks.rados.rados.0.smithi085.stdout:3494: finishing write tid 3 to smithi08575397-50 2021-02-10T04:41:43.474 INFO:tasks.rados.rados.0.smithi085.stdout:3494: finishing write tid 4 to smithi08575397-50 2021-02-10T04:41:43.475 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 1586 (ObjNum 1197 snap 409 seq_num 1197) dirty exists 2021-02-10T04:41:43.475 INFO:tasks.rados.rados.0.smithi085.stdout:3494: left oid 50 (ObjNum 1197 snap 409 seq_num 1197) 2021-02-10T04:41:43.475 INFO:tasks.rados.rados.0.smithi085.stdout:3495: finishing write tid 1 to smithi08575397-15 2021-02-10T04:41:43.475 INFO:tasks.rados.rados.0.smithi085.stdout:3495: finishing write tid 2 to smithi08575397-15 2021-02-10T04:41:43.475 INFO:tasks.rados.rados.0.smithi085.stdout:3495: finishing write tid 3 to smithi08575397-15 2021-02-10T04:41:43.475 INFO:tasks.rados.rados.0.smithi085.stdout:3497: finishing rollback tid 0 to smithi08575397-46 2021-02-10T04:41:43.475 INFO:tasks.rados.rados.0.smithi085.stdout:3497: finishing rollback tid 1 to smithi08575397-46 2021-02-10T04:41:43.475 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 1588 (ObjNum 83003472 snap 22030 seq_num 0) dirty dne 2021-02-10T04:41:43.475 INFO:tasks.rados.rados.0.smithi085.stdout:3495: finishing write tid 4 to smithi08575397-15 2021-02-10T04:41:43.475 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 15 v 1477 (ObjNum 1198 snap 409 seq_num 1198) dirty exists 2021-02-10T04:41:43.475 INFO:tasks.rados.rados.0.smithi085.stdout:3495: left oid 15 (ObjNum 1198 snap 409 seq_num 1198) 2021-02-10T04:41:43.475 INFO:tasks.rados.rados.0.smithi085.stdout:3496: expect (ObjNum 1178 snap 402 seq_num 1178) 2021-02-10T04:41:43.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:43 smithi164 conmon[33677]: debug 2021-02-10T04:41:43.367+0000 7fbec19a1700 -1 osd.4 883 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:41:43.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:43 smithi164 conmon[29288]: cluster 2021-02-10T04:41:42.331494+0000 osd.4 ( 2021-02-10T04:41:43.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:43 smithi164 conmon[29288]: osd.4) 1449 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:43.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:43 smithi164 conmon[29288]: cluster 2021-02-10T04:41:42.331499+0000 osd.4 (osd.4) 1450 2021-02-10T04:41:43.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:43 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:43.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:43 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:41:43.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:43 smithi164 conmon[29288]: -10T04:41: 2021-02-10T04:41:43.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:43 smithi164 conmon[29288]: 42.331503+0000 osd.4 (osd. 2021-02-10T04:41:43.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:43 smithi164 conmon[29288]: 4) 1451 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:43.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:43 smithi164 conmon[29288]: 2021-02-10T04:41:43.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:43 smithi164 conmon[29288]: cluster 2021-02-10T04:41:43.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:43 smithi164 conmon[29288]: 2021-02-10T04:41:42.331512 2021-02-10T04:41:43.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:43 smithi164 conmon[29288]: +0000 2021-02-10T04:41:43.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:43 smithi164 conmon[29288]: osd.4 (osd.4) 1452 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:43.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:43 smithi164 conmon[29288]: cluster 2021-02-10T04:41:43.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:43 smithi164 conmon[29288]: 2021-02-10T04:41:42.457887+0000 mon.a (mon.0) 1847 2021-02-10T04:41:43.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:43 smithi164 conmon[29288]: : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 732 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:41:43.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:43 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:41:43.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:43 smithi164 conmon[29288]: 10T04:41:42.467543+0000 mon.a (mon.0) 1848 2021-02-10T04:41:43.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:43 smithi164 conmon[29288]: : cluster [DBG] osdmap e883: 8 total, 8 up, 7 in 2021-02-10T04:41:43.646 INFO:tasks.rados.rados.0.smithi085.stdout:3494: done (10 left) 2021-02-10T04:41:43.646 INFO:tasks.rados.rados.0.smithi085.stdout:3495: done (9 left) 2021-02-10T04:41:43.646 INFO:tasks.rados.rados.0.smithi085.stdout:3496: done (8 left) 2021-02-10T04:41:43.646 INFO:tasks.rados.rados.0.smithi085.stdout:3497: done (7 left) 2021-02-10T04:41:43.646 INFO:tasks.rados.rados.0.smithi085.stdout:3498: done (6 left) 2021-02-10T04:41:43.647 INFO:tasks.rados.rados.0.smithi085.stdout:3499: done (5 left) 2021-02-10T04:41:43.647 INFO:tasks.rados.rados.0.smithi085.stdout:3502: write oid 25 current snap is 409 2021-02-10T04:41:43.647 INFO:tasks.rados.rados.0.smithi085.stdout:3501: finishing rollback tid 1 to smithi08575397-42 2021-02-10T04:41:43.647 INFO:tasks.rados.rados.0.smithi085.stdout:3501: finishing rollback tid 2 to smithi08575397-42 2021-02-10T04:41:43.647 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 1532 (ObjNum 1124 snap 380 seq_num 1124) dirty exists 2021-02-10T04:41:43.647 INFO:tasks.rados.rados.0.smithi085.stdout:3502: seq_num 1199 ranges {551593=490972,1779557=372036} 2021-02-10T04:41:43.652 INFO:tasks.rados.rados.0.smithi085.stdout:3502: writing smithi08575397-25 from 551593 to 1042565 tid 1 2021-02-10T04:41:43.659 INFO:tasks.rados.rados.0.smithi085.stdout:3502: writing smithi08575397-25 from 1779557 to 2151593 tid 2 2021-02-10T04:41:43.659 INFO:tasks.rados.rados.0.smithi085.stdout:3501: done (5 left) 2021-02-10T04:41:43.659 INFO:tasks.rados.rados.0.smithi085.stdout:3503: read oid 40 snap 337 2021-02-10T04:41:43.659 INFO:tasks.rados.rados.0.smithi085.stdout:3503: expect (ObjNum 1021 snap 335 seq_num 1021) 2021-02-10T04:41:43.659 INFO:tasks.rados.rados.0.smithi085.stdout:3504: snap_remove snap 388 2021-02-10T04:41:43.904 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[41540]: debug 2021-02-10T04:41:43.902+0000 7fb94bb63700 -1 osd.0 pg_epoch: 884 pg[5.5( empty local-lis/les=57/58 n=0 ec=57/57 lis/c=57/57 les/c/f=58/58/0 sis=57) [0,1,4] r=0 lpr=57 crt=0'0 mlcod 0'0 active+clean] scrubber pg(5.5) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x555d327ef600) 2021-02-10T04:41:43.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[35961]: cluster 2021-02-10T04:41 2021-02-10T04:41:43.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[35961]: :42.331494+0000 osd.4 (osd.4) 1449 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:43.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[35961]: cluster 2021-02-10T04:41:43.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:41:43.905 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[35961]: :41:42.331499+0000 osd.4 (osd.4) 1450 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:43.905 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[35961]: cluster 2021-02-10T04:41:43.905 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:41:43.905 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[35961]: 04:41:42.331503+0000 osd.4 (osd.4) 1451 : cluster [WRN] 2021-02-10T04:41:43.905 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:43.905 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[35961]: cluster 2021-02-10T04:41:43.905 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[35961]: 2021-02-10T04:41:43.905 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[35961]: 2021-02-10T04:41 2021-02-10T04:41:43.905 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[35961]: :42.331512+0000 osd.4 (osd.4) 1452 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:43.905 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:41:43.905 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[35961]: -10T04:41:42.457887+0000 mon.a (mon.0) 1847 : 2021-02-10T04:41:43.905 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[35961]: cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 732 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:41:43.905 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[35961]: cluster 2021-02-10T04:41:43.906 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:41:43.906 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[35961]: 04:41:42.467543+0000 mon.a (mon.0) 1848 : cluster [DBG] osdmap e883: 8 total, 8 up, 7 in 2021-02-10T04:41:43.906 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[35961]: 2021-02-10T04:41:43.906 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[29319]: cluster 2021-02-10T04:41:42.331494+0000 osd.4 ( 2021-02-10T04:41:43.906 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[29319]: osd.4) 1449 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:43.906 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[29319]: cluster 2021-02-10T04:41:42.331499+0000 osd.4 (osd.4) 1450 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:43.906 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[29319]: cluster 2021-02-10T04:41:42.331503+0000 osd.4 (osd.4 2021-02-10T04:41:43.907 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[29319]: ) 1451 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:43.907 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[29319]: cluster 2021-02-10T04:41: 2021-02-10T04:41:43.907 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[29319]: 42.331512+0000 osd.4 (osd.4) 1452 : cluster 2021-02-10T04:41:43.907 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:43.907 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[29319]: cluster 2021-02-10T04:41:42.457887 2021-02-10T04:41:43.907 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1847 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 732 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:41:43.907 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[29319]: 2021-02-10T04:41:43.907 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[29319]: cluster 2021-02-10T04:41:43.907 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[29319]: 2021-02-10T04:41:42.467543+0000 mon.a (mon.0) 2021-02-10T04:41:43.908 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:43 smithi085 conmon[29319]: 1848 : cluster [DBG] osdmap e883: 8 total, 8 up, 7 in 2021-02-10T04:41:44.480 INFO:tasks.rados.rados.0.smithi085.stdout:3504: done (6 left) 2021-02-10T04:41:44.480 INFO:tasks.rados.rados.0.smithi085.stdout:3505: write oid 38 current snap is 409 2021-02-10T04:41:44.481 INFO:tasks.rados.rados.0.smithi085.stdout:3502: finishing write tid 1 to smithi08575397-25 2021-02-10T04:41:44.481 INFO:tasks.rados.rados.0.smithi085.stdout:3505: seq_num 1200 ranges {568400=566221,1838039=330361} 2021-02-10T04:41:44.487 INFO:tasks.rados.rados.0.smithi085.stdout:3505: writing smithi08575397-38 from 568400 to 1134621 tid 1 2021-02-10T04:41:44.494 INFO:tasks.rados.rados.0.smithi085.stdout:3505: writing smithi08575397-38 from 1838039 to 2168400 tid 2 2021-02-10T04:41:44.494 INFO:tasks.rados.rados.0.smithi085.stdout:3506: write oid 1 current snap is 409 2021-02-10T04:41:44.494 INFO:tasks.rados.rados.0.smithi085.stdout:3502: finishing write tid 2 to smithi08575397-25 2021-02-10T04:41:44.494 INFO:tasks.rados.rados.0.smithi085.stdout:3502: finishing write tid 3 to smithi08575397-25 2021-02-10T04:41:44.494 INFO:tasks.rados.rados.0.smithi085.stdout:3502: finishing write tid 4 to smithi08575397-25 2021-02-10T04:41:44.494 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 25 v 1470 (ObjNum 1199 snap 409 seq_num 1199) dirty exists 2021-02-10T04:41:44.495 INFO:tasks.rados.rados.0.smithi085.stdout:3502: left oid 25 (ObjNum 1199 snap 409 seq_num 1199) 2021-02-10T04:41:44.495 INFO:tasks.rados.rados.0.smithi085.stdout:3503: expect (ObjNum 1021 snap 335 seq_num 1021) 2021-02-10T04:41:44.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[33677]: debug 2021-02-10T04:41:44.332+0000 7fbec19a1700 -1 osd.4 884 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:41:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: cluster 2021-02-10T04:41:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: 2021-02-10T04:41:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: 2021 2021-02-10T04:41:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: - 2021-02-10T04:41:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: 02 2021-02-10T04:41:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: - 2021-02-10T04:41:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: 10 2021-02-10T04:41:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: T 2021-02-10T04:41:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: 04 2021-02-10T04:41:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: : 2021-02-10T04:41:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: 41 2021-02-10T04:41:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: : 2021-02-10T04:41:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: 43 2021-02-10T04:41:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: . 2021-02-10T04:41:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: 369258 2021-02-10T04:41:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: +0000 2021-02-10T04:41:44.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: 2021-02-10T04:41:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: osd.4 2021-02-10T04:41:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: ( 2021-02-10T04:41:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: osd 2021-02-10T04:41:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: . 2021-02-10T04:41:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: 4 2021-02-10T04:41:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: ) 2021-02-10T04:41:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: 1453 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: cluster 2021-02-10T04:41:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: 2021-02-10T04:41:43 2021-02-10T04:41:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: .369267+0000 osd.4 ( 2021-02-10T04:41:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: osd.4) 1454 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:41:44.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: -02-10T04:41:43.369274+0000 2021-02-10T04:41:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: osd.4 (osd.4) 1455 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: 2021-02-10T04:41:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: cluster 2021-02-10T04:41:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: 2021-02-10T04:41:43.369279+0000 osd.4 (osd 2021-02-10T04:41:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: .4) 1456 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:41:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: -10T04:41:43.473635+0000 mon.a ( 2021-02-10T04:41:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: mon.0) 1849 : cluster [DBG] osdmap e884: 8 total, 8 up, 7 in 2021-02-10T04:41:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:41:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: 02-10T04:41:44.220075+0000 mgr.y 2021-02-10T04:41:44.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:44 smithi164 conmon[29288]: (mgr.14140) 834 : cluster [DBG] pgmap v1477: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 698 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 9.7 MiB/s rd, 1.6 MiB/s wr, 16 op/s 2021-02-10T04:41:44.677 INFO:tasks.rados.rados.0.smithi085.stdout:3506: seq_num 1201 ranges {585207=641470,1896521=288686} 2021-02-10T04:41:44.684 INFO:tasks.rados.rados.0.smithi085.stdout:3506: writing smithi08575397-1 from 585207 to 1226677 tid 1 2021-02-10T04:41:44.690 INFO:tasks.rados.rados.0.smithi085.stdout:3506: writing smithi08575397-1 from 1896521 to 2185207 tid 2 2021-02-10T04:41:44.690 INFO:tasks.rados.rados.0.smithi085.stdout:3502: done (7 left) 2021-02-10T04:41:44.690 INFO:tasks.rados.rados.0.smithi085.stdout:3503: done (6 left) 2021-02-10T04:41:44.691 INFO:tasks.rados.rados.0.smithi085.stdout:3507: write oid 7 current snap is 409 2021-02-10T04:41:44.691 INFO:tasks.rados.rados.0.smithi085.stdout:3507: seq_num 1202 ranges {602014=433072,1671356=530658} 2021-02-10T04:41:44.696 INFO:tasks.rados.rados.0.smithi085.stdout:3507: writing smithi08575397-7 from 602014 to 1035086 tid 1 2021-02-10T04:41:44.704 INFO:tasks.rados.rados.0.smithi085.stdout:3507: writing smithi08575397-7 from 1671356 to 2202014 tid 2 2021-02-10T04:41:44.704 INFO:tasks.rados.rados.0.smithi085.stdout:3508: snap_remove snap 402 2021-02-10T04:41:44.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[29319]: cluster 2021-02-10T04:41:44.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:41:44.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[29319]: T04:41:43.369258+0000 osd.4 (osd.4) 2021-02-10T04:41:44.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[29319]: 1453 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:44.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[29319]: cluster 2021-02-10T04:41:44.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[29319]: 2021-02-10T04:41:44.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[29319]: 2021 2021-02-10T04:41:44.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[29319]: - 2021-02-10T04:41:44.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[29319]: 02 2021-02-10T04:41:44.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[29319]: - 2021-02-10T04:41:44.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[29319]: 10 2021-02-10T04:41:44.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[29319]: T 2021-02-10T04:41:44.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[29319]: 04:41:43.369267+0000 osd.4 (osd 2021-02-10T04:41:44.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[29319]: .4) 1454 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:44.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:41:44.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[29319]: -02-10T04:41:43.369274+0000 osd.4 (osd.4) 1455 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:44.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:41:44.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[29319]: :41:43.369279+0000 osd.4 (osd.4) 1456 : cluster [WRN] 2021-02-10T04:41:44.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:44.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:41:44.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[29319]: 04:41:43.473635+0000 mon.a (mon.0) 1849 : cluster [DBG] osdmap e884: 8 total, 8 up, 7 in 2021-02-10T04:41:44.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[29319]: cluster 2021-02-10T04:41:44.220075+0000 2021-02-10T04:41:44.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[29319]: mgr.y (mgr.14140) 834 : cluster [DBG] pgmap v1477: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 698 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 9.7 MiB/s rd, 1.6 MiB/s wr, 16 op/s 2021-02-10T04:41:44.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[35961]: cluster 2021-02-10T04:41:43.369258+0000 2021-02-10T04:41:44.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[35961]: osd.4 (osd.4) 1453 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:44.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:41:44.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[35961]: -10T04:41:43.369267+0000 osd.4 (osd. 2021-02-10T04:41:44.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[35961]: 4) 1454 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:44.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:41:44.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[35961]: 10T04:41:43.369274+0000 osd.4 (osd.4) 1455 : cluster 2021-02-10T04:41:44.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:44.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:41:44.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[35961]: :41:43.369279+0000 osd.4 (osd.4) 1456 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:44.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[35961]: cluster 2021-02-10T04:41:44.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[35961]: 2021 2021-02-10T04:41:44.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[35961]: -02-10T04:41:43. 2021-02-10T04:41:44.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[35961]: 473635+0000 mon.a (mon.0) 1849 : cluster [DBG] osdmap e884: 8 total, 8 up, 7 in 2021-02-10T04:41:44.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:41:44.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[35961]: 41:44.220075+0000 mgr.y (mgr.14140) 834 : cluster 2021-02-10T04:41:44.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:44 smithi085 conmon[35961]: [DBG] pgmap v1477: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 698 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 9.7 MiB/s rd, 1.6 MiB/s wr, 16 op/s 2021-02-10T04:41:45.494 INFO:tasks.rados.rados.0.smithi085.stdout:3508: done (7 left) 2021-02-10T04:41:45.494 INFO:tasks.rados.rados.0.smithi085.stdout:3509: read oid 31 snap -1 2021-02-10T04:41:45.494 INFO:tasks.rados.rados.0.smithi085.stdout:3509: expect (ObjNum 1097 snap 368 seq_num 1097) 2021-02-10T04:41:45.494 INFO:tasks.rados.rados.0.smithi085.stdout:3510: write (excl) oid 25 current snap is 409 2021-02-10T04:41:45.494 INFO:tasks.rados.rados.0.smithi085.stdout:3505: finishing write tid 1 to smithi08575397-38 2021-02-10T04:41:45.494 INFO:tasks.rados.rados.0.smithi085.stdout:3510: seq_num 1203 ranges {618821=508321,1613485=531077} 2021-02-10T04:41:45.501 INFO:tasks.rados.rados.0.smithi085.stdout:3510: writing smithi08575397-25 from 618821 to 1127142 tid 1 2021-02-10T04:41:45.508 INFO:tasks.rados.rados.0.smithi085.stdout:3510: writing smithi08575397-25 from 1613485 to 2144562 tid 2 2021-02-10T04:41:45.509 INFO:tasks.rados.rados.0.smithi085.stdout:3511: read oid 50 snap -1 2021-02-10T04:41:45.509 INFO:tasks.rados.rados.0.smithi085.stdout:3511: expect (ObjNum 1197 snap 409 seq_num 1197) 2021-02-10T04:41:45.509 INFO:tasks.rados.rados.0.smithi085.stdout:3512: rollback oid 15 current snap is 409 2021-02-10T04:41:45.509 INFO:tasks.rados.rados.0.smithi085.stdout:3505: finishing write tid 2 to smithi08575397-38 2021-02-10T04:41:45.509 INFO:tasks.rados.rados.0.smithi085.stdout:3505: finishing write tid 3 to smithi08575397-38 2021-02-10T04:41:45.509 INFO:tasks.rados.rados.0.smithi085.stdout:3505: finishing write tid 4 to smithi08575397-38 2021-02-10T04:41:45.509 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 1591 (ObjNum 1200 snap 409 seq_num 1200) dirty exists 2021-02-10T04:41:45.509 INFO:tasks.rados.rados.0.smithi085.stdout:3505: left oid 38 (ObjNum 1200 snap 409 seq_num 1200) 2021-02-10T04:41:45.509 INFO:tasks.rados.rados.0.smithi085.stdout:3506: finishing write tid 1 to smithi08575397-1 2021-02-10T04:41:45.509 INFO:tasks.rados.rados.0.smithi085.stdout:3506: finishing write tid 2 to smithi08575397-1 2021-02-10T04:41:45.509 INFO:tasks.rados.rados.0.smithi085.stdout:3506: finishing write tid 3 to smithi08575397-1 2021-02-10T04:41:45.509 INFO:tasks.rados.rados.0.smithi085.stdout:3506: finishing write tid 4 to smithi08575397-1 2021-02-10T04:41:45.509 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 1589 (ObjNum 1201 snap 409 seq_num 1201) dirty exists 2021-02-10T04:41:45.510 INFO:tasks.rados.rados.0.smithi085.stdout:3506: left oid 1 (ObjNum 1201 snap 409 seq_num 1201) 2021-02-10T04:41:45.510 INFO:tasks.rados.rados.0.smithi085.stdout:3507: finishing write tid 1 to smithi08575397-7 2021-02-10T04:41:45.510 INFO:tasks.rados.rados.0.smithi085.stdout:3507: finishing write tid 2 to smithi08575397-7 2021-02-10T04:41:45.510 INFO:tasks.rados.rados.0.smithi085.stdout:3507: finishing write tid 3 to smithi08575397-7 2021-02-10T04:41:45.510 INFO:tasks.rados.rados.0.smithi085.stdout:3507: finishing write tid 4 to smithi08575397-7 2021-02-10T04:41:45.510 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 1597 (ObjNum 1202 snap 409 seq_num 1202) dirty exists 2021-02-10T04:41:45.510 INFO:tasks.rados.rados.0.smithi085.stdout:3507: left oid 7 (ObjNum 1202 snap 409 seq_num 1202) 2021-02-10T04:41:45.510 INFO:tasks.rados.rados.0.smithi085.stdout:3509: expect (ObjNum 1097 snap 368 seq_num 1097) 2021-02-10T04:41:45.625 INFO:tasks.rados.rados.0.smithi085.stdout:3510: finishing write tid 1 to smithi08575397-25 2021-02-10T04:41:45.625 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 15 to 408 2021-02-10T04:41:45.625 INFO:tasks.rados.rados.0.smithi085.stdout:3510: finishing write tid 2 to smithi08575397-25 2021-02-10T04:41:45.626 INFO:tasks.rados.rados.0.smithi085.stdout:3510: finishing write tid 3 to smithi08575397-25 2021-02-10T04:41:45.626 INFO:tasks.rados.rados.0.smithi085.stdout:3510: finishing write tid 4 to smithi08575397-25 2021-02-10T04:41:45.626 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 25 v 1473 (ObjNum 1203 snap 409 seq_num 1203) dirty exists 2021-02-10T04:41:45.626 INFO:tasks.rados.rados.0.smithi085.stdout:3510: left oid 25 (ObjNum 1203 snap 409 seq_num 1203) 2021-02-10T04:41:45.626 INFO:tasks.rados.rados.0.smithi085.stdout:3511: expect (ObjNum 1197 snap 409 seq_num 1197) 2021-02-10T04:41:45.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:45 smithi164 conmon[33677]: debug 2021-02-10T04:41:45.350+0000 7fbec19a1700 -1 osd.4 885 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:41:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:45 smithi164 conmon[29288]: cluster 2021-02-10T04:41:44.333059+0000 osd.4 (osd.4) 1457 : cluster [WRN] 2021-02-10T04:41:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:45 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:45 smithi164 conmon[29288]: cluster 2021-02-10T04:41:44.333070+0000 osd.4 (osd.4) 1458 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:45 smithi164 conmon[29288]: cluster 2021-02-10T04:41 2021-02-10T04:41:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:45 smithi164 conmon[29288]: :44.333077+0000 osd.4 (osd.4) 1459 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:45.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:45 smithi164 conmon[29288]: cluster 2021-02-10T04:41:45.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:45 smithi164 conmon[29288]: 2021-02-10T04:41:44 2021-02-10T04:41:45.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:45 smithi164 conmon[29288]: .333083+0000 osd.4 (osd.4 2021-02-10T04:41:45.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:45 smithi164 conmon[29288]: ) 1460 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:45.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:45 smithi164 conmon[29288]: cluster 2021-02-10T04:41:44.479813+0000 mon.a (mon.0 2021-02-10T04:41:45.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:45 smithi164 conmon[29288]: ) 1850 : cluster [DBG] osdmap e885: 8 total, 8 up, 7 in 2021-02-10T04:41:45.810 INFO:tasks.rados.rados.0.smithi085.stdout:3505: done (10 left) 2021-02-10T04:41:45.810 INFO:tasks.rados.rados.0.smithi085.stdout:3506: done (9 left) 2021-02-10T04:41:45.810 INFO:tasks.rados.rados.0.smithi085.stdout:3507: done (8 left) 2021-02-10T04:41:45.810 INFO:tasks.rados.rados.0.smithi085.stdout:3509: done (7 left) 2021-02-10T04:41:45.811 INFO:tasks.rados.rados.0.smithi085.stdout:3510: done (6 left) 2021-02-10T04:41:45.811 INFO:tasks.rados.rados.0.smithi085.stdout:3511: done (5 left) 2021-02-10T04:41:45.811 INFO:tasks.rados.rados.0.smithi085.stdout:3513: read oid 4 snap -1 2021-02-10T04:41:45.811 INFO:tasks.rados.rados.0.smithi085.stdout:3513: expect (ObjNum 1188 snap 406 seq_num 1188) 2021-02-10T04:41:45.811 INFO:tasks.rados.rados.0.smithi085.stdout:3514: delete oid 30 current snap is 409 2021-02-10T04:41:45.811 INFO:tasks.rados.rados.0.smithi085.stdout:3512: finishing rollback tid 0 to smithi08575397-15 2021-02-10T04:41:45.811 INFO:tasks.rados.rados.0.smithi085.stdout:3512: finishing rollback tid 1 to smithi08575397-15 2021-02-10T04:41:45.811 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 15 v 1479 (ObjNum 0 snap 7 seq_num 3) dirty dne 2021-02-10T04:41:45.814 INFO:tasks.rados.rados.0.smithi085.stdout:3512: done (6 left) 2021-02-10T04:41:45.814 INFO:tasks.rados.rados.0.smithi085.stdout:3514: done (5 left) 2021-02-10T04:41:45.814 INFO:tasks.rados.rados.0.smithi085.stdout:3515: write (excl) oid 20 current snap is 409 2021-02-10T04:41:45.814 INFO:tasks.rados.rados.0.smithi085.stdout:3515: seq_num 1205 ranges {652435=658819,1730449=521986} 2021-02-10T04:41:45.822 INFO:tasks.rados.rados.0.smithi085.stdout:3515: writing smithi08575397-20 from 652435 to 1311254 tid 1 2021-02-10T04:41:45.828 INFO:tasks.rados.rados.0.smithi085.stdout:3515: writing smithi08575397-20 from 1730449 to 2252435 tid 2 2021-02-10T04:41:45.829 INFO:tasks.rados.rados.0.smithi085.stdout:3516: snap_create 2021-02-10T04:41:45.829 INFO:tasks.rados.rados.0.smithi085.stdout:3513: expect (ObjNum 1188 snap 406 seq_num 1188) 2021-02-10T04:41:45.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:45 smithi085 conmon[29319]: cluster 2021-02-10T04:41:44.333059+0000 osd.4 (osd 2021-02-10T04:41:45.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:45 smithi085 conmon[29319]: .4) 1457 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:45 smithi085 conmon[29319]: cluster 2021-02-10T04:41:44.333070+0000 osd.4 (osd.4) 1458 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:45 smithi085 conmon[29319]: cluster 2021-02-10T04:41:44.333077+0000 osd.4 (osd.4) 1459 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:45 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:41:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:45 smithi085 conmon[29319]: -02-10T04:41:44.333083+0000 osd.4 2021-02-10T04:41:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:45 smithi085 conmon[29319]: (osd.4) 1460 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:45 smithi085 conmon[29319]: 2021-02-10T04:41:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:45 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:41:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:45 smithi085 conmon[29319]: :41:44.479813+0000 mon.a (mon.0) 1850 : 2021-02-10T04:41:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:45 smithi085 conmon[29319]: cluster [DBG] osdmap e885: 8 total, 8 up, 7 in 2021-02-10T04:41:45.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:45 smithi085 conmon[35961]: cluster 2021-02-10T04:41:44.333059+0000 osd.4 (osd.4) 1457 : 2021-02-10T04:41:45.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:45 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:45.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:45 smithi085 conmon[35961]: cluster 2021-02-10T04:41:44.333070+0000 osd.4 (osd.4) 1458 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:45.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:45 smithi085 conmon[35961]: cluster 2021-02-10T04:41:44.333077+0000 osd.4 (osd.4) 1459 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:45.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:45 smithi085 conmon[35961]: cluster 2021-02-10T04:41:44.333083+0000 2021-02-10T04:41:45.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:45 smithi085 conmon[35961]: osd.4 (osd.4) 1460 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:45.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:45 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:41:45.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:45 smithi085 conmon[35961]: 02-10T04:41:44.479813+0000 mon.a ( 2021-02-10T04:41:45.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:45 smithi085 conmon[35961]: mon.0) 1850 : cluster [DBG] osdmap e885: 8 total, 8 up, 7 in 2021-02-10T04:41:46.014 INFO:tasks.rados.rados.0.smithi085.stdout:3515: finishing write tid 1 to smithi08575397-20 2021-02-10T04:41:46.014 INFO:tasks.rados.rados.0.smithi085.stdout:3515: finishing write tid 2 to smithi08575397-20 2021-02-10T04:41:46.015 INFO:tasks.rados.rados.0.smithi085.stdout:3515: finishing write tid 3 to smithi08575397-20 2021-02-10T04:41:46.015 INFO:tasks.rados.rados.0.smithi085.stdout:3515: finishing write tid 4 to smithi08575397-20 2021-02-10T04:41:46.015 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 20 v 1572 (ObjNum 1205 snap 409 seq_num 1205) dirty exists 2021-02-10T04:41:46.015 INFO:tasks.rados.rados.0.smithi085.stdout:3515: left oid 20 (ObjNum 1205 snap 409 seq_num 1205) 2021-02-10T04:41:46.499 INFO:tasks.rados.rados.0.smithi085.stdout:3513: done (6 left) 2021-02-10T04:41:46.499 INFO:tasks.rados.rados.0.smithi085.stdout:3515: done (5 left) 2021-02-10T04:41:46.499 INFO:tasks.rados.rados.0.smithi085.stdout:3516: done (4 left) 2021-02-10T04:41:46.500 INFO:tasks.rados.rados.0.smithi085.stdout:3517: read oid 40 snap 356 2021-02-10T04:41:46.500 INFO:tasks.rados.rados.0.smithi085.stdout:3517: expect (ObjNum 1021 snap 335 seq_num 1021) 2021-02-10T04:41:46.500 INFO:tasks.rados.rados.0.smithi085.stdout:3518: write (excl) oid 20 current snap is 410 2021-02-10T04:41:46.500 INFO:tasks.rados.rados.0.smithi085.stdout:3518: seq_num 1206 ranges {669242=734068,2188931=80311} 2021-02-10T04:41:46.507 INFO:tasks.rados.rados.0.smithi085.stdout:3518: writing smithi08575397-20 from 669242 to 1403310 tid 1 2021-02-10T04:41:46.513 INFO:tasks.rados.rados.0.smithi085.stdout:3518: writing smithi08575397-20 from 2188931 to 2269242 tid 2 2021-02-10T04:41:46.513 INFO:tasks.rados.rados.0.smithi085.stdout:3519: write (excl) oid 26 current snap is 410 2021-02-10T04:41:46.513 INFO:tasks.rados.rados.0.smithi085.stdout:3517: expect (ObjNum 1021 snap 335 seq_num 1021) 2021-02-10T04:41:46.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[33677]: debug 2021-02-10T04:41:46.348+0000 7fbec19a1700 -1 osd.4 886 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:41:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: cluster 2021-02-10T04:41:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: 2021-02-10T04:41:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: 2021 2021-02-10T04:41:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: - 2021-02-10T04:41:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: 02 2021-02-10T04:41:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: - 2021-02-10T04:41:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: 10 2021-02-10T04:41:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: T 2021-02-10T04:41:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: 04 2021-02-10T04:41:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: : 2021-02-10T04:41:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: 41 2021-02-10T04:41:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: : 2021-02-10T04:41:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: 45 2021-02-10T04:41:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: . 2021-02-10T04:41:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: 350746 2021-02-10T04:41:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: +0000 2021-02-10T04:41:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: 2021-02-10T04:41:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: osd.4 2021-02-10T04:41:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: ( 2021-02-10T04:41:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: osd 2021-02-10T04:41:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: . 2021-02-10T04:41:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: 4 2021-02-10T04:41:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: ) 2021-02-10T04:41:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: 1461 2021-02-10T04:41:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: : 2021-02-10T04:41:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: cluster 2021-02-10T04:41:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: 2021-02-10T04:41:45. 2021-02-10T04:41:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: 350757+0000 osd.4 (osd.4) 1462 2021-02-10T04:41:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: cluster 2021-02-10T04:41:45.350763+0000 osd.4 (osd.4 2021-02-10T04:41:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: ) 1463 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:46.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:41:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: T04:41 2021-02-10T04:41:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: :45.350769+0000 osd.4 ( 2021-02-10T04:41:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: osd.4) 1464 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:41:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: :41:45.493171+0000 mon.a (mon.0) 2021-02-10T04:41:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: 1851 : cluster [DBG] osdmap e886: 8 total, 8 up, 7 in 2021-02-10T04:41:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:41:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: 10T04:41:46.220602+0000 mgr.y (mgr.14140 2021-02-10T04:41:46.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:46 smithi164 conmon[29288]: ) 835 : cluster [DBG] pgmap v1480: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 707 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 21 MiB/s rd, 2.5 MiB/s wr, 29 op/s 2021-02-10T04:41:46.686 INFO:tasks.rados.rados.0.smithi085.stdout:3518: finishing write tid 1 to smithi08575397-20 2021-02-10T04:41:46.686 INFO:tasks.rados.rados.0.smithi085.stdout:3518: finishing write tid 2 to smithi08575397-20 2021-02-10T04:41:46.686 INFO:tasks.rados.rados.0.smithi085.stdout:3518: finishing write tid 3 to smithi08575397-20 2021-02-10T04:41:46.687 INFO:tasks.rados.rados.0.smithi085.stdout:3518: finishing write tid 4 to smithi08575397-20 2021-02-10T04:41:46.687 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 20 v 1575 (ObjNum 1206 snap 410 seq_num 1206) dirty exists 2021-02-10T04:41:46.687 INFO:tasks.rados.rados.0.smithi085.stdout:3518: left oid 20 (ObjNum 1206 snap 410 seq_num 1206) 2021-02-10T04:41:46.687 INFO:tasks.rados.rados.0.smithi085.stdout:3519: seq_num 1207 ranges {686049=409317,1847413=438636} 2021-02-10T04:41:46.693 INFO:tasks.rados.rados.0.smithi085.stdout:3519: writing smithi08575397-26 from 686049 to 1095366 tid 1 2021-02-10T04:41:46.701 INFO:tasks.rados.rados.0.smithi085.stdout:3519: writing smithi08575397-26 from 1847413 to 2286049 tid 2 2021-02-10T04:41:46.702 INFO:tasks.rados.rados.0.smithi085.stdout:3519: finishing write tid 1 to smithi08575397-26 2021-02-10T04:41:46.702 INFO:tasks.rados.rados.0.smithi085.stdout:3517: done (6 left) 2021-02-10T04:41:46.702 INFO:tasks.rados.rados.0.smithi085.stdout:3518: done (5 left) 2021-02-10T04:41:46.702 INFO:tasks.rados.rados.0.smithi085.stdout:3520: copy_from oid 34 from oid 46 current snap is 410 2021-02-10T04:41:46.702 INFO:tasks.rados.rados.0.smithi085.stdout:3521: snap_remove snap 397 2021-02-10T04:41:46.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[29319]: cluster 2021-02-10T04:41:45.350746+0000 osd.4 ( 2021-02-10T04:41:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[29319]: osd.4) 1461 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[29319]: cluster 2021-02-10T04:41:45.350757+0000 osd.4 (osd.4) 1462 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:41:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[29319]: 02-10T04:41:45.350763+0000 osd.4 (osd.4) 1463 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:41:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[29319]: 41:45.350769+0000 osd.4 (osd.4) 1464 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[29319]: cluster 2021-02-10T04:41:45.493171+0000 mon.a 2021-02-10T04:41:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[29319]: (mon.0) 1851 : cluster [DBG] 2021-02-10T04:41:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[29319]: osdmap e886: 8 total, 8 up, 7 in 2021-02-10T04:41:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[29319]: cluster 2021-02-10T04:41:46.220602+0000 mgr.y (mgr 2021-02-10T04:41:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[29319]: .14140) 835 : cluster [DBG] pgmap v1480: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 707 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 21 MiB/s rd, 2.5 MiB/s wr, 29 op/s 2021-02-10T04:41:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[35961]: cluster 2021-02-10T04:41:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[35961]: 2021-02-10T04:41:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[35961]: 2021 2021-02-10T04:41:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[35961]: -02-10T04:41:45.350746 2021-02-10T04:41:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1461 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[35961]: cluster 2021-02-10T04:41:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[35961]: 2021-02-10T04:41:45.350757+0000 osd.4 ( 2021-02-10T04:41:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[35961]: osd.4) 1462 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:41:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[35961]: T04:41:45.350763+0000 osd.4 (osd.4 2021-02-10T04:41:46.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[35961]: ) 1463 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:46.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:41:46.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[35961]: 41:45.350769+0000 osd.4 (osd.4) 1464 : 2021-02-10T04:41:46.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:46.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:41:46.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[35961]: T04:41:45.493171+0000 mon.a 2021-02-10T04:41:46.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[35961]: (mon.0) 1851 : cluster [DBG] osdmap e886: 8 total, 8 up, 7 in 2021-02-10T04:41:46.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:41:46.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[35961]: -10T04:41:46.220602+0000 mgr.y (mgr. 2021-02-10T04:41:46.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:46 smithi085 conmon[35961]: 14140) 835 : cluster [DBG] pgmap v1480: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 707 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 21 MiB/s rd, 2.5 MiB/s wr, 29 op/s 2021-02-10T04:41:47.505 INFO:tasks.rados.rados.0.smithi085.stdout:3521: done (6 left) 2021-02-10T04:41:47.505 INFO:tasks.rados.rados.0.smithi085.stdout:3522: write (excl) oid 6 current snap is 410 2021-02-10T04:41:47.505 INFO:tasks.rados.rados.0.smithi085.stdout:3520: finishing copy_from to smithi08575397-34 2021-02-10T04:41:47.505 INFO:tasks.rados.rados.0.smithi085.stdout:3520: got expected ENOENT (src dne) 2021-02-10T04:41:47.505 INFO:tasks.rados.rados.0.smithi085.stdout:3520: finishing copy_from racing read to smithi08575397-34 2021-02-10T04:41:47.506 INFO:tasks.rados.rados.0.smithi085.stdout:3519: finishing write tid 2 to smithi08575397-26 2021-02-10T04:41:47.506 INFO:tasks.rados.rados.0.smithi085.stdout:3519: finishing write tid 3 to smithi08575397-26 2021-02-10T04:41:47.506 INFO:tasks.rados.rados.0.smithi085.stdout:3519: finishing write tid 4 to smithi08575397-26 2021-02-10T04:41:47.506 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 1370 (ObjNum 1207 snap 410 seq_num 1207) dirty exists 2021-02-10T04:41:47.506 INFO:tasks.rados.rados.0.smithi085.stdout:3519: left oid 26 (ObjNum 1207 snap 410 seq_num 1207) 2021-02-10T04:41:47.506 INFO:tasks.rados.rados.0.smithi085.stdout:3522: seq_num 1208 ranges {702856=484566,1789542=513314} 2021-02-10T04:41:47.509 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:41:47.512 INFO:tasks.rados.rados.0.smithi085.stdout:3522: writing smithi08575397-6 from 702856 to 1187422 tid 1 2021-02-10T04:41:47.519 INFO:tasks.rados.rados.0.smithi085.stdout:3522: writing smithi08575397-6 from 1789542 to 2302856 tid 2 2021-02-10T04:41:47.520 INFO:tasks.rados.rados.0.smithi085.stdout:3519: done (6 left) 2021-02-10T04:41:47.520 INFO:tasks.rados.rados.0.smithi085.stdout:3520: done (5 left) 2021-02-10T04:41:47.520 INFO:tasks.rados.rados.0.smithi085.stdout:3523: copy_from oid 49 from oid 31 current snap is 410 2021-02-10T04:41:47.520 INFO:tasks.rados.rados.0.smithi085.stdout:3522: finishing write tid 1 to smithi08575397-6 2021-02-10T04:41:47.520 INFO:tasks.rados.rados.0.smithi085.stdout:3524: copy_from oid 26 from oid 30 current snap is 410 2021-02-10T04:41:47.520 INFO:tasks.rados.rados.0.smithi085.stdout:3525: read oid 1 snap -1 2021-02-10T04:41:47.520 INFO:tasks.rados.rados.0.smithi085.stdout:3525: expect (ObjNum 1201 snap 409 seq_num 1201) 2021-02-10T04:41:47.520 INFO:tasks.rados.rados.0.smithi085.stdout:3526: delete oid 46 current snap is 410 2021-02-10T04:41:47.527 INFO:tasks.rados.rados.0.smithi085.stdout:3525: expect (ObjNum 1201 snap 409 seq_num 1201) 2021-02-10T04:41:47.544 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:41:47.624 INFO:tasks.rados.rados.0.smithi085.stdout:3525: done (8 left) 2021-02-10T04:41:47.624 INFO:tasks.rados.rados.0.smithi085.stdout:3526: done (7 left) 2021-02-10T04:41:47.624 INFO:tasks.rados.rados.0.smithi085.stdout:3527: write (excl) oid 34 current snap is 410 2021-02-10T04:41:47.624 INFO:tasks.rados.rados.0.smithi085.stdout:3527: seq_num 1210 ranges {736470=751417,2022859=313611} 2021-02-10T04:41:47.628 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:41:47 smithi164 conmon[40892]: debug 2021-02-10T04:41:47.627+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:41:47.629 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:47 smithi164 conmon[29288]: cluster 2021-02-10T04:41:46.349336+0000 2021-02-10T04:41:47.629 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:47 smithi164 conmon[29288]: osd.4 (osd.4) 1465 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:47.629 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:47 smithi164 conmon[29288]: cluster 2021-02-10T04:41:46.349345+0000 osd.4 (osd. 2021-02-10T04:41:47.629 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:47 smithi164 conmon[29288]: 4) 1466 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:47.629 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:47 smithi164 conmon[29288]: cluster 2021-02-10T04:41:46.349353 2021-02-10T04:41:47.629 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:47 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1467 : cluster [WRN] 2021-02-10T04:41:47.629 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:47 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:47.629 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:47 smithi164 conmon[29288]: cluster 2021-02-10T04:41:47.629 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:47 smithi164 conmon[29288]: 2021-02- 2021-02-10T04:41:47.629 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:47 smithi164 conmon[29288]: 10T04:41:46.349359+0000 osd.4 (osd. 2021-02-10T04:41:47.630 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:47 smithi164 conmon[29288]: 4) 1468 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:47.630 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:47 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:41:47.630 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:47 smithi164 conmon[29288]: 02-10T04:41:46.497850+0000 mon.a ( 2021-02-10T04:41:47.630 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:47 smithi164 conmon[29288]: mon.0) 1852 : cluster [DBG] osdmap e887: 8 total, 8 up, 7 in 2021-02-10T04:41:47.630 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:47 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:41:47.630 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:47 smithi164 conmon[29288]: 02-10T04:41:47.046930+0000 osd.4 ( 2021-02-10T04:41:47.630 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:47 smithi164 conmon[29288]: osd.4) 1469 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:41:47.630 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:47 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:41:47.630 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:47 smithi164 conmon[29288]: -02-10T04:41:47.047613+0000 2021-02-10T04:41:47.630 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:47 smithi164 conmon[29288]: osd.4 (osd.4) 1470 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:41:47.630 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:47 smithi164 conmon[33677]: debug 2021-02-10T04:41:47.370+0000 7fbec19a1700 -1 osd.4 887 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:41:47.632 INFO:tasks.rados.rados.0.smithi085.stdout:3527: writing smithi08575397-34 from 736470 to 1487887 tid 1 2021-02-10T04:41:47.638 INFO:tasks.rados.rados.0.smithi085.stdout:3527: writing smithi08575397-34 from 2022859 to 2336470 tid 2 2021-02-10T04:41:47.638 INFO:tasks.rados.rados.0.smithi085.stdout:3528: snap_remove snap 389 2021-02-10T04:41:47.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[29319]: cluster 2021-02-10T04:41:46.349336+0000 osd.4 2021-02-10T04:41:47.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[29319]: (osd.4) 1465 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:47.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[29319]: cluster 2021-02-10T04:41:46.349345+0000 osd.4 (osd.4) 1466 : cluster [WRN] 2021-02-10T04:41:47.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:47.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[29319]: cluster 2021-02-10T04:41:46. 2021-02-10T04:41:47.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[29319]: 349353+0000 osd.4 (osd.4) 1467 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:47.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:41:47.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[29319]: 41:46.349359+0000 osd.4 (osd.4) 1468 : 2021-02-10T04:41:47.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:47.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[29319]: cluster 2021-02-10T04:41:46 2021-02-10T04:41:47.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[29319]: .497850+0000 mon.a (mon.0) 1852 : cluster [DBG] 2021-02-10T04:41:47.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[29319]: osdmap e887: 8 total, 8 up, 7 in 2021-02-10T04:41:47.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[29319]: cluster 2021-02-10T04:41:47.046930 2021-02-10T04:41:47.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1469 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:41:47.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[29319]: 2021-02-10T04:41:47.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[29319]: cluster 2021-02-10T04:41:47.047613 2021-02-10T04:41:47.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1470 : cluster [DBG] 2021-02-10T04:41:47.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[29319]: purged_snaps scrub ok 2021-02-10T04:41:47.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[35961]: cluster 2021-02-10T04:41:47.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:41:47.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[35961]: 04:41:46.349336+0000 osd.4 (osd.4) 1465 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:47.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[35961]: cluster 2021-02-10T04:41:47.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[35961]: 2021-02-10T04:41:46.349345+0000 osd.4 2021-02-10T04:41:47.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[35961]: (osd.4) 1466 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:47.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[35961]: cluster 2021-02-10T04:41:47.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[35961]: 2021-02-10T04:41:46.349353+0000 osd.4 ( 2021-02-10T04:41:47.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[35961]: osd.4) 1467 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:47.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:41:47.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[35961]: 02-10T04:41:46.349359+0000 osd.4 ( 2021-02-10T04:41:47.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[35961]: osd.4) 1468 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:47.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:41:47.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[35961]: T04:41:46.497850+0000 mon.a (mon. 2021-02-10T04:41:47.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[35961]: 0) 1852 : cluster [DBG] osdmap e887: 8 total, 8 up, 7 in 2021-02-10T04:41:47.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[35961]: cluster 2021-02-10T04:41:47.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[35961]: 2021-02-10T04:41:47. 2021-02-10T04:41:47.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[35961]: 046930+0000 osd.4 (osd.4) 1469 : cluster [DBG] 2021-02-10T04:41:47.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[35961]: purged_snaps scrub starts 2021-02-10T04:41:47.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[35961]: cluster 2021-02-10T04:41:47.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:41:47.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[35961]: 04:41:47.047613+0000 osd.4 (osd.4) 2021-02-10T04:41:47.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:47 smithi085 conmon[35961]: 1470 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:41:48.086 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:41:48.191 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:41:48.223 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:41:48.304 INFO:tasks.rados.rados.0.smithi085.stdout:3528: done (8 left) 2021-02-10T04:41:48.305 INFO:tasks.rados.rados.0.smithi085.stdout:3529: read oid 32 snap -1 2021-02-10T04:41:48.305 INFO:tasks.rados.rados.0.smithi085.stdout:3529: expect (ObjNum 1180 snap 403 seq_num 1180) 2021-02-10T04:41:48.305 INFO:tasks.rados.rados.0.smithi085.stdout:3530: rollback oid 4 current snap is 410 2021-02-10T04:41:48.305 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 4 to 266 2021-02-10T04:41:48.305 INFO:tasks.rados.rados.0.smithi085.stdout:3531: read oid 29 snap -1 2021-02-10T04:41:48.305 INFO:tasks.rados.rados.0.smithi085.stdout:3531: expect deleted 2021-02-10T04:41:48.305 INFO:tasks.rados.rados.0.smithi085.stdout:3532: write oid 44 current snap is 410 2021-02-10T04:41:48.305 INFO:tasks.rados.rados.0.smithi085.stdout:3532: seq_num 1211 ranges {753277=426666,1681341=545753} 2021-02-10T04:41:48.312 INFO:tasks.rados.rados.0.smithi085.stdout:3532: writing smithi08575397-44 from 753277 to 1179943 tid 1 2021-02-10T04:41:48.325 INFO:tasks.rados.rados.0.smithi085.stdout:3532: writing smithi08575397-44 from 1681341 to 2227094 tid 2 2021-02-10T04:41:48.325 INFO:tasks.rados.rados.0.smithi085.stdout:3533: write oid 12 current snap is 410 2021-02-10T04:41:48.325 INFO:tasks.rados.rados.0.smithi085.stdout:3533: seq_num 1212 ranges {770084=501915,2023470=346614} 2021-02-10T04:41:48.335 INFO:tasks.rados.rados.0.smithi085.stdout:3533: writing smithi08575397-12 from 770084 to 1271999 tid 1 2021-02-10T04:41:48.343 INFO:tasks.rados.rados.0.smithi085.stdout:3533: writing smithi08575397-12 from 2023470 to 2370084 tid 2 2021-02-10T04:41:48.343 INFO:tasks.rados.rados.0.smithi085.stdout:3522: finishing write tid 2 to smithi08575397-6 2021-02-10T04:41:48.343 INFO:tasks.rados.rados.0.smithi085.stdout:3522: finishing write tid 3 to smithi08575397-6 2021-02-10T04:41:48.343 INFO:tasks.rados.rados.0.smithi085.stdout:3522: finishing write tid 4 to smithi08575397-6 2021-02-10T04:41:48.344 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 1595 (ObjNum 1208 snap 410 seq_num 1208) dirty exists 2021-02-10T04:41:48.344 INFO:tasks.rados.rados.0.smithi085.stdout:3522: left oid 6 (ObjNum 1208 snap 410 seq_num 1208) 2021-02-10T04:41:48.344 INFO:tasks.rados.rados.0.smithi085.stdout:3524: finishing copy_from to smithi08575397-26 2021-02-10T04:41:48.344 INFO:tasks.rados.rados.0.smithi085.stdout:3524: got expected ENOENT (src dne) 2021-02-10T04:41:48.344 INFO:tasks.rados.rados.0.smithi085.stdout:3524: finishing copy_from racing read to smithi08575397-26 2021-02-10T04:41:48.344 INFO:tasks.rados.rados.0.smithi085.stdout:3527: finishing write tid 1 to smithi08575397-34 2021-02-10T04:41:48.345 INFO:tasks.rados.rados.0.smithi085.stdout:3527: finishing write tid 2 to smithi08575397-34 2021-02-10T04:41:48.345 INFO:tasks.rados.rados.0.smithi085.stdout:3527: finishing write tid 3 to smithi08575397-34 2021-02-10T04:41:48.345 INFO:tasks.rados.rados.0.smithi085.stdout:3527: finishing write tid 4 to smithi08575397-34 2021-02-10T04:41:48.345 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 34 v 1557 (ObjNum 1210 snap 410 seq_num 1210) dirty exists 2021-02-10T04:41:48.345 INFO:tasks.rados.rados.0.smithi085.stdout:3527: left oid 34 (ObjNum 1210 snap 410 seq_num 1210) 2021-02-10T04:41:48.345 INFO:tasks.rados.rados.0.smithi085.stdout:3523: finishing copy_from to smithi08575397-49 2021-02-10T04:41:48.345 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 49 v 1483 (ObjNum 1097 snap 368 seq_num 1097) dirty exists 2021-02-10T04:41:48.346 INFO:tasks.rados.rados.0.smithi085.stdout:3523: finishing copy_from racing read to smithi08575397-49 2021-02-10T04:41:48.346 INFO:tasks.rados.rados.0.smithi085.stdout:3530: finishing rollback tid 0 to smithi08575397-4 2021-02-10T04:41:48.346 INFO:tasks.rados.rados.0.smithi085.stdout:3529: expect (ObjNum 1180 snap 403 seq_num 1180) 2021-02-10T04:41:48.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:41:47 smithi164 conmon[37263]: debug 2021-02-10T04:41:48.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:41:47 smithi164 conmon[37263]: 2021-02-10T04:41:47.924+0000 7f1c671e2700 -1 osd.5 pg_epoch: 888 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55665319a6e0) 2021-02-10T04:41:48.507 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[35961]: cluster 2021-02-10T04:41:47. 2021-02-10T04:41:48.507 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[35961]: 370855+0000 osd.4 (osd.4) 1471 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:48.508 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[35961]: cluster 2021-02-10T04:41:48.508 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[35961]: 2021-02-10T04:41:47.370865+0000 osd.4 (osd.4) 1472 : cluster 2021-02-10T04:41:48.508 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:48.508 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[35961]: cluster 2021-02-10T04:41:47 2021-02-10T04:41:48.508 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[35961]: .370871+0000 osd.4 (osd.4) 1473 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:48.508 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:41:48.508 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[35961]: :41:47.370878+0000 osd.4 (osd 2021-02-10T04:41:48.508 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[35961]: .4) 1474 : cluster [WRN] 2021-02-10T04:41:48.508 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:48.508 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[35961]: cluster 2021-02-10T04:41:48.508 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[35961]: 2021-02-10T04:41:47.502110 2021-02-10T04:41:48.508 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1853 : cluster [DBG] osdmap e888: 8 total, 8 up, 7 in 2021-02-10T04:41:48.508 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[35961]: 2021-02-10T04:41:48.509 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[35961]: cluster 2021-02-10T04:41: 2021-02-10T04:41:48.509 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[35961]: 48.221236+0000 mgr.y (mgr.14140) 836 : cluster 2021-02-10T04:41:48.509 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[35961]: [DBG] pgmap v1483: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 709 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2021-02-10T04:41:48.509 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[35961]: cluster 2021-02-10T04:41 2021-02-10T04:41:48.509 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[35961]: :48.295126+0000 mon.a (mon.0) 2021-02-10T04:41:48.509 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[35961]: 1854 : cluster [DBG] osdmap e889: 8 total, 8 up, 7 in 2021-02-10T04:41:48.509 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[41540]: debug 2021-02-10T04:41:48.509 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[41540]: 2021-02-10T04:41:48.303+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:41:48.509 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[41540]: 2021-02-10T04:41:48.516 INFO:tasks.rados.rados.0.smithi085.stdout:3522: done (12 left) 2021-02-10T04:41:48.516 INFO:tasks.rados.rados.0.smithi085.stdout:3523: done (11 left) 2021-02-10T04:41:48.516 INFO:tasks.rados.rados.0.smithi085.stdout:3524: done (10 left) 2021-02-10T04:41:48.516 INFO:tasks.rados.rados.0.smithi085.stdout:3527: done (9 left) 2021-02-10T04:41:48.516 INFO:tasks.rados.rados.0.smithi085.stdout:3529: done (8 left) 2021-02-10T04:41:48.516 INFO:tasks.rados.rados.0.smithi085.stdout:3534: snap_remove snap 400 2021-02-10T04:41:48.772 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:41:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:48 smithi164 conmon[29288]: cluster 2021-02-10T04:41:47.370855+0000 osd.4 (osd.4) 2021-02-10T04:41:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:48 smithi164 conmon[29288]: 1471 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:48 smithi164 conmon[29288]: cluster 2021-02-10T04:41:47.370865+0000 osd.4 (osd.4) 1472 : cluster [WRN] 2021-02-10T04:41:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:48 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:48 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:41:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:48 smithi164 conmon[29288]: 02-10T04:41:47.370871+0000 osd.4 (osd.4) 1473 : cluster 2021-02-10T04:41:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:48 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:48 smithi164 conmon[29288]: cluster 2021-02-10T04:41:47.370878+0000 osd.4 (osd.4) 1474 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:48 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:41:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:48 smithi164 conmon[29288]: 41:47.502110+0000 mon.a (mon.0) 2021-02-10T04:41:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:48 smithi164 conmon[29288]: 1853 : cluster [DBG] osdmap e888: 8 total, 8 up, 7 in 2021-02-10T04:41:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:48 smithi164 conmon[29288]: cluster 2021-02-10T04:41:48.221236+0000 mgr.y ( 2021-02-10T04:41:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:48 smithi164 conmon[29288]: mgr.14140) 836 : cluster [DBG] pgmap v1483: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 709 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2021-02-10T04:41:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:48 smithi164 conmon[29288]: cluster 2021-02-10T04:41:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:48 smithi164 conmon[29288]: 2021-02-10T04:41:48.295126 2021-02-10T04:41:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:48 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1854 : cluster 2021-02-10T04:41:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:48 smithi164 conmon[29288]: [DBG] osdmap e889: 8 total, 8 up, 7 in 2021-02-10T04:41:48.897 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:48 smithi164 conmon[33677]: debug 2021-02-10T04:41:48.416+0000 7fbec19a1700 -1 osd.4 889 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:41:48.897 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:41:48 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:41:48.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[29319]: cluster 2021-02-10T04:41:47.370855+0000 osd.4 2021-02-10T04:41:48.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[29319]: (osd.4) 1471 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:48.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[29319]: cluster 2021-02-10T04:41:47.370865+0000 osd.4 (osd.4) 1472 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:48.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[29319]: cluster 2021-02-10T04:41:47.370871+0000 osd.4 (osd.4) 1473 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:48.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[29319]: cluster 2021-02-10T04:41:47.370878+0000 osd.4 (osd.4) 1474 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:48.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[29319]: cluster 2021-02-10T04:41:47.502110+0000 2021-02-10T04:41:48.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[29319]: mon.a (mon.0) 1853 : cluster [DBG] osdmap e888: 8 total, 8 up, 7 in 2021-02-10T04:41:48.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[29319]: cluster 2021-02-10T04:41:48.221236+0000 mgr.y 2021-02-10T04:41:48.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[29319]: (mgr.14140) 836 : cluster [DBG] pgmap v1483: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 709 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2021-02-10T04:41:48.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[29319]: cluster 2021-02-10T04:41:48.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[29319]: 2021 2021-02-10T04:41:48.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[29319]: -02-10T04:41:48 2021-02-10T04:41:48.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:48 smithi085 conmon[29319]: .295126+0000 mon.a (mon.0) 1854 : cluster [DBG] osdmap e889: 8 total, 8 up, 7 in 2021-02-10T04:41:49.510 INFO:tasks.rados.rados.0.smithi085.stdout:3534: done (8 left) 2021-02-10T04:41:49.510 INFO:tasks.rados.rados.0.smithi085.stdout:3535: write oid 11 current snap is 410 2021-02-10T04:41:49.510 INFO:tasks.rados.rados.0.smithi085.stdout:3535: seq_num 1213 ranges {786891=577164,2081952=304939} 2021-02-10T04:41:49.518 INFO:tasks.rados.rados.0.smithi085.stdout:3535: writing smithi08575397-11 from 786891 to 1364055 tid 1 2021-02-10T04:41:49.525 INFO:tasks.rados.rados.0.smithi085.stdout:3535: writing smithi08575397-11 from 2081952 to 2386891 tid 2 2021-02-10T04:41:49.525 INFO:tasks.rados.rados.0.smithi085.stdout:3536: delete oid 40 current snap is 410 2021-02-10T04:41:49.525 INFO:tasks.rados.rados.0.smithi085.stdout:3530: finishing rollback tid 1 to smithi08575397-4 2021-02-10T04:41:49.526 INFO:tasks.rados.rados.0.smithi085.stdout:3530: finishing rollback tid 2 to smithi08575397-4 2021-02-10T04:41:49.526 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 4 v 1373 (ObjNum 8 snap 0 seq_num 8) dirty exists 2021-02-10T04:41:49.526 INFO:tasks.rados.rados.0.smithi085.stdout:3532: finishing write tid 1 to smithi08575397-44 2021-02-10T04:41:49.526 INFO:tasks.rados.rados.0.smithi085.stdout:3532: finishing write tid 2 to smithi08575397-44 2021-02-10T04:41:49.526 INFO:tasks.rados.rados.0.smithi085.stdout:3532: finishing write tid 3 to smithi08575397-44 2021-02-10T04:41:49.526 INFO:tasks.rados.rados.0.smithi085.stdout:3532: finishing write tid 4 to smithi08575397-44 2021-02-10T04:41:49.526 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 1578 (ObjNum 1211 snap 410 seq_num 1211) dirty exists 2021-02-10T04:41:49.526 INFO:tasks.rados.rados.0.smithi085.stdout:3532: left oid 44 (ObjNum 1211 snap 410 seq_num 1211) 2021-02-10T04:41:49.526 INFO:tasks.rados.rados.0.smithi085.stdout:3533: finishing write tid 1 to smithi08575397-12 2021-02-10T04:41:49.526 INFO:tasks.rados.rados.0.smithi085.stdout:3533: finishing write tid 2 to smithi08575397-12 2021-02-10T04:41:49.526 INFO:tasks.rados.rados.0.smithi085.stdout:3533: finishing write tid 3 to smithi08575397-12 2021-02-10T04:41:49.526 INFO:tasks.rados.rados.0.smithi085.stdout:3533: finishing write tid 4 to smithi08575397-12 2021-02-10T04:41:49.526 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 1535 (ObjNum 1212 snap 410 seq_num 1212) dirty exists 2021-02-10T04:41:49.527 INFO:tasks.rados.rados.0.smithi085.stdout:3533: left oid 12 (ObjNum 1212 snap 410 seq_num 1212) 2021-02-10T04:41:49.529 INFO:tasks.rados.rados.0.smithi085.stdout:3535: finishing write tid 1 to smithi08575397-11 2021-02-10T04:41:49.529 INFO:tasks.rados.rados.0.smithi085.stdout:3535: finishing write tid 2 to smithi08575397-11 2021-02-10T04:41:49.531 INFO:tasks.rados.rados.0.smithi085.stdout:3535: finishing write tid 3 to smithi08575397-11 2021-02-10T04:41:49.532 INFO:tasks.rados.rados.0.smithi085.stdout:3535: finishing write tid 4 to smithi08575397-11 2021-02-10T04:41:49.532 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 1817 (ObjNum 1213 snap 410 seq_num 1213) dirty exists 2021-02-10T04:41:49.532 INFO:tasks.rados.rados.0.smithi085.stdout:3535: left oid 11 (ObjNum 1213 snap 410 seq_num 1213) 2021-02-10T04:41:49.532 INFO:tasks.rados.rados.0.smithi085.stdout:3530: done (9 left) 2021-02-10T04:41:49.532 INFO:tasks.rados.rados.0.smithi085.stdout:3531: done (8 left) 2021-02-10T04:41:49.532 INFO:tasks.rados.rados.0.smithi085.stdout:3532: done (7 left) 2021-02-10T04:41:49.533 INFO:tasks.rados.rados.0.smithi085.stdout:3533: done (6 left) 2021-02-10T04:41:49.533 INFO:tasks.rados.rados.0.smithi085.stdout:3535: done (5 left) 2021-02-10T04:41:49.533 INFO:tasks.rados.rados.0.smithi085.stdout:3536: done (4 left) 2021-02-10T04:41:49.533 INFO:tasks.rados.rados.0.smithi085.stdout:3537: snap_create 2021-02-10T04:41:49.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:49 smithi164 conmon[29288]: cluster 2021-02-10T04:41:48.417475+0000 2021-02-10T04:41:49.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:49 smithi164 conmon[29288]: osd.4 (osd.4) 1475 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:49.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:49 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:41:49.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:49 smithi164 conmon[29288]: 04:41:48.417481+0000 osd.4 (osd.4) 1476 : cluster 2021-02-10T04:41:49.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:49 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:49.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:49 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:41:49.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:49 smithi164 conmon[29288]: -10T04:41:48.417485+0000 osd.4 ( 2021-02-10T04:41:49.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:49 smithi164 conmon[29288]: osd.4) 1477 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:49.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:49 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:41:49.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:49 smithi164 conmon[29288]: -10T04:41:48.417488+0000 2021-02-10T04:41:49.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:49 smithi164 conmon[29288]: osd.4 (osd.4) 1478 : cluster 2021-02-10T04:41:49.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:49 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:49.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:49 smithi164 conmon[29288]: audit 2021-02-10T04:41:49.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:49 smithi164 conmon[29288]: 2021-02-10T04:41:48.478031+0000 mgr.y 2021-02-10T04:41:49.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:49 smithi164 conmon[29288]: (mgr.14140) 837 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:41:49.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:49 smithi164 conmon[29288]: cluster 2021-02-10T04:41:49.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:49 smithi164 conmon[29288]: 2021-02-10T04:41:49.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:49 smithi164 conmon[29288]: 2021 2021-02-10T04:41:49.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:49 smithi164 conmon[29288]: -02-10 2021-02-10T04:41:49.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:49 smithi164 conmon[29288]: T04:41:48.498983+0000 mon.a (mon.0) 1855 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 737 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:41:49.647 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:49 smithi164 conmon[33677]: debug 2021-02-10T04:41:49.647 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:49 smithi164 conmon[33677]: 2021-02-10T04:41:49.377+0000 7fbec19a1700 -1 osd.4 889 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:41:49.647 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:49 smithi164 conmon[33677]: 2021-02-10T04:41:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[29319]: cluster 2021-02-10T04:41:48.417475+0000 osd.4 (osd.4) 1475 : cluster 2021-02-10T04:41:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[29319]: cluster 2021-02-10T04:41:48.417481+0000 osd.4 (osd.4) 1476 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[29319]: cluster 2021-02-10T04:41 2021-02-10T04:41:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[29319]: :48.417485+0000 osd.4 (osd.4) 1477 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[29319]: cluster 2021-02-10T04:41:48. 2021-02-10T04:41:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[29319]: 417488+0000 osd.4 (osd.4) 1478 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[29319]: audit 2021-02-10T04:41:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[29319]: 2021-02-10T04:41:48.478031+0000 mgr.y (mgr.14140) 837 : 2021-02-10T04:41:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[29319]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:41:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:41:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[29319]: -02-10T04:41:48.498983+0000 mon.a (mon.0 2021-02-10T04:41:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[29319]: ) 1855 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 737 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:41:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: cluster 2021-02-10T04:41:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: 2021-02-10T04:41:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: 2021 2021-02-10T04:41:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: - 2021-02-10T04:41:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: 02 2021-02-10T04:41:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: - 2021-02-10T04:41:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: 10 2021-02-10T04:41:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: T 2021-02-10T04:41:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: 04 2021-02-10T04:41:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: : 2021-02-10T04:41:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: 41 2021-02-10T04:41:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: :48.417475+0000 osd.4 (osd.4 2021-02-10T04:41:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: ) 1475 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:41:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: 41:48.417481+0000 osd.4 (osd.4 2021-02-10T04:41:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: ) 1476 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:41:49.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: -02-10T04:41:48.417485 2021-02-10T04:41:49.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1477 : cluster [WRN] 2021-02-10T04:41:49.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:49.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:41:49.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: 04:41:48.417488+0000 osd.4 ( 2021-02-10T04:41:49.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: osd.4) 1478 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:49.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: 2021-02-10T04:41:49.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: audit 2021-02-10T04:41:49.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: 2021-02-10T04:41:48.478031+0000 2021-02-10T04:41:49.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: mgr.y (mgr.14140) 837 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:41:49.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: 2021-02-10T04:41:49.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: cluster 2021-02-10T04:41 2021-02-10T04:41:49.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: :48.498983+0000 mon.a (mon.0) 2021-02-10T04:41:49.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:49 smithi085 conmon[35961]: 1855 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 737 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:41:50.393 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:41:49 smithi164 conmon[37263]: debug 2021-02-10T04:41:49.912+0000 7f1c671e2700 -1 osd.5 pg_epoch: 890 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x556655fa7b80) 2021-02-10T04:41:50.534 INFO:tasks.rados.rados.0.smithi085.stdout:3537: done (4 left) 2021-02-10T04:41:50.534 INFO:tasks.rados.rados.0.smithi085.stdout:3538: delete oid 25 current snap is 411 2021-02-10T04:41:50.540 INFO:tasks.rados.rados.0.smithi085.stdout:3538: done (4 left) 2021-02-10T04:41:50.540 INFO:tasks.rados.rados.0.smithi085.stdout:3539: read oid 5 snap -1 2021-02-10T04:41:50.541 INFO:tasks.rados.rados.0.smithi085.stdout:3539: expect (ObjNum 1097 snap 368 seq_num 1097) 2021-02-10T04:41:50.541 INFO:tasks.rados.rados.0.smithi085.stdout:3540: rollback oid 40 current snap is 411 2021-02-10T04:41:50.541 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 40 to 337 2021-02-10T04:41:50.541 INFO:tasks.rados.rados.0.smithi085.stdout:3541: read oid 11 snap -1 2021-02-10T04:41:50.541 INFO:tasks.rados.rados.0.smithi085.stdout:3541: expect (ObjNum 1213 snap 410 seq_num 1213) 2021-02-10T04:41:50.541 INFO:tasks.rados.rados.0.smithi085.stdout:3542: write oid 32 current snap is 411 2021-02-10T04:41:50.541 INFO:tasks.rados.rados.0.smithi085.stdout:3542: seq_num 1216 ranges {437312=402911,1341045=696749} 2021-02-10T04:41:50.545 INFO:tasks.rados.rados.0.smithi085.stdout:3542: writing smithi08575397-32 from 437312 to 840223 tid 1 2021-02-10T04:41:50.552 INFO:tasks.rados.rados.0.smithi085.stdout:3542: writing smithi08575397-32 from 1341045 to 2037794 tid 2 2021-02-10T04:41:50.552 INFO:tasks.rados.rados.0.smithi085.stdout:3543: copy_from oid 48 from oid 28 current snap is 411 2021-02-10T04:41:50.553 INFO:tasks.rados.rados.0.smithi085.stdout:3544: snap_remove snap 278 2021-02-10T04:41:50.896 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:50 smithi164 conmon[33677]: debug 2021-02-10T04:41:50.896 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:50 smithi164 conmon[33677]: 2021-02-10T04:41:50.394+0000 7fbec19a1700 -1 osd.4 890 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:41:50.896 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:50 smithi164 conmon[33677]: 2021-02-10T04:41:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:50 smithi164 conmon[29288]: cluster 2021-02-10T04:41:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:50 smithi164 conmon[29288]: 2021-02-10T04:41:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:50 smithi164 conmon[29288]: 2021 2021-02-10T04:41:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:50 smithi164 conmon[29288]: - 2021-02-10T04:41:50.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:50 smithi164 conmon[29288]: 02 2021-02-10T04:41:50.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:50 smithi164 conmon[29288]: - 2021-02-10T04:41:50.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:50 smithi164 conmon[29288]: 10 2021-02-10T04:41:50.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:50 smithi164 conmon[29288]: T 2021-02-10T04:41:50.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:50 smithi164 conmon[29288]: 04:41:49.378364+0000 osd.4 ( 2021-02-10T04:41:50.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:50 smithi164 conmon[29288]: osd.4) 1479 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:50.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:50 smithi164 conmon[29288]: cluster 2021-02-10T04:41:49. 2021-02-10T04:41:50.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:50 smithi164 conmon[29288]: 378372+0000 osd.4 (osd.4) 1480 : cluster [WRN] 2021-02-10T04:41:50.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:50 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:50.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:50 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:41:50.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:50 smithi164 conmon[29288]: -02-10T04:41:49.378378+0000 osd.4 2021-02-10T04:41:50.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:50 smithi164 conmon[29288]: (osd.4) 1481 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:50.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:50 smithi164 conmon[29288]: cluster 2021-02-10T04:41:50.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:50 smithi164 conmon[29288]: 2021-02-10T04:41 2021-02-10T04:41:50.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:50 smithi164 conmon[29288]: :49.378384+0000 osd.4 (osd.4) 1482 : cluster 2021-02-10T04:41:50.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:50 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:50.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:50 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:41:50.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:50 smithi164 conmon[29288]: 04:41:49.510371+0000 mon.a (mon.0) 2021-02-10T04:41:50.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:50 smithi164 conmon[29288]: 1856 : cluster [DBG] osdmap e890: 8 total, 8 up, 7 in 2021-02-10T04:41:50.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:50 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:41:50.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:50 smithi164 conmon[29288]: 10T04:41:50. 2021-02-10T04:41:50.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:50 smithi164 conmon[29288]: 222000+0000 mgr.y (mgr. 2021-02-10T04:41:50.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:50 smithi164 conmon[29288]: 14140) 838 : cluster [DBG] pgmap v1486: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 719 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 2.5 MiB/s wr, 25 op/s 2021-02-10T04:41:50.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[29319]: cluster 2021-02-10T04:41:49.378364+0000 osd.4 2021-02-10T04:41:50.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[29319]: (osd.4) 1479 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:50.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[29319]: cluster 2021-02-10T04:41:49.378372+0000 osd.4 (osd.4) 1480 : cluster [WRN] 2021-02-10T04:41:50.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:50.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[29319]: cluster 2021-02-10T04:41 2021-02-10T04:41:50.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[29319]: :49.378378+0000 osd.4 (osd.4) 1481 : 2021-02-10T04:41:50.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:50.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[29319]: cluster 2021-02-10T04:41:50.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[29319]: 2021-02-10T04:41:49.378384 2021-02-10T04:41:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1482 : cluster [WRN] 2021-02-10T04:41:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:41:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[29319]: -02-10T04:41:49.510371+0000 2021-02-10T04:41:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[29319]: mon.a (mon.0) 1856 : cluster [DBG] 2021-02-10T04:41:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[29319]: osdmap e890: 8 total, 8 up, 7 in 2021-02-10T04:41:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:41:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[29319]: -10T04:41:50.222000+0000 mgr.y 2021-02-10T04:41:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[29319]: (mgr.14140) 838 : cluster [DBG] pgmap v1486: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 719 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 2.5 MiB/s wr, 25 op/s 2021-02-10T04:41:50.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[29319]: 2021-02-10T04:41:50.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[35961]: cluster 2021-02-10T04:41:50.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[35961]: 2021-02-10T04:41 2021-02-10T04:41:50.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[35961]: :49.378364+0000 osd.4 (osd.4) 1479 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:50.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:41:50.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[35961]: -02-10T04:41:49.378372+0000 osd.4 (osd.4) 1480 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:50.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[35961]: 2021-02-10T04:41:50.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[35961]: cluster 2021-02-10T04:41:50.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[35961]: 2021-02-10T04:41:49.378378+0000 osd.4 (osd.4 2021-02-10T04:41:50.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[35961]: ) 1481 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:50.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[35961]: cluster 2021-02-10T04:41 2021-02-10T04:41:50.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[35961]: :49.378384+0000 osd.4 (osd.4) 1482 : cluster 2021-02-10T04:41:50.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:50.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[35961]: cluster 2021-02-10T04:41:49.510371 2021-02-10T04:41:50.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1856 : cluster [DBG] osdmap e890: 8 total, 8 up, 7 in 2021-02-10T04:41:50.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:41:50.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[35961]: -02-10T04:41:50.222000+0000 mgr.y 2021-02-10T04:41:50.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:50 smithi085 conmon[35961]: (mgr.14140) 838 : cluster [DBG] pgmap v1486: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 719 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 2.5 MiB/s wr, 25 op/s 2021-02-10T04:41:51.544 INFO:tasks.rados.rados.0.smithi085.stdout:3544: done (9 left) 2021-02-10T04:41:51.544 INFO:tasks.rados.rados.0.smithi085.stdout:3545: copy_from oid 6 from oid 43 current snap is 411 2021-02-10T04:41:51.544 INFO:tasks.rados.rados.0.smithi085.stdout:3539: expect (ObjNum 1097 snap 368 seq_num 1097) 2021-02-10T04:41:51.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:51 smithi164 conmon[33677]: debug 2021-02-10T04:41:51.390+0000 7fbec19a1700 -1 osd.4 891 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:41:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:51 smithi164 conmon[29288]: cluster 2021-02-10T04:41:50.395293+0000 osd.4 ( 2021-02-10T04:41:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:51 smithi164 conmon[29288]: osd.4) 1483 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:51 smithi164 conmon[29288]: cluster 2021-02-10T04:41:50.395306+0000 osd.4 (osd.4) 1484 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:51 smithi164 conmon[29288]: cluster 2021-02-10T04:41:50.395313+0000 osd.4 ( 2021-02-10T04:41:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:51 smithi164 conmon[29288]: osd.4) 1485 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:51 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:41:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:51 smithi164 conmon[29288]: 02-10T04:41:50.395319+0000 osd.4 ( 2021-02-10T04:41:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:51 smithi164 conmon[29288]: osd.4) 1486 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:51 smithi164 conmon[29288]: cluster 2021-02-10T04:41:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:51 smithi164 conmon[29288]: 2021-02-10T04:41:50.533227+0000 2021-02-10T04:41:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:51 smithi164 conmon[29288]: mon.a (mon.0) 1857 : cluster [DBG] 2021-02-10T04:41:51.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:51 smithi164 conmon[29288]: osdmap e891: 8 total, 8 up, 7 in 2021-02-10T04:41:51.653 INFO:tasks.rados.rados.0.smithi085.stdout:3539: done (9 left) 2021-02-10T04:41:51.653 INFO:tasks.rados.rados.0.smithi085.stdout:3546: copy_from oid 5 from oid 23 current snap is 411 2021-02-10T04:41:51.653 INFO:tasks.rados.rados.0.smithi085.stdout:3542: finishing write tid 1 to smithi08575397-32 2021-02-10T04:41:51.653 INFO:tasks.rados.rados.0.smithi085.stdout:3540: finishing rollback tid 1 to smithi08575397-40 2021-02-10T04:41:51.653 INFO:tasks.rados.rados.0.smithi085.stdout:3540: finishing rollback tid 2 to smithi08575397-40 2021-02-10T04:41:51.653 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 1825 (ObjNum 1021 snap 335 seq_num 1021) dirty exists 2021-02-10T04:41:51.653 INFO:tasks.rados.rados.0.smithi085.stdout:3543: finishing copy_from to smithi08575397-48 2021-02-10T04:41:51.653 INFO:tasks.rados.rados.0.smithi085.stdout:3543: got expected ENOENT (src dne) 2021-02-10T04:41:51.653 INFO:tasks.rados.rados.0.smithi085.stdout:3543: finishing copy_from racing read to smithi08575397-48 2021-02-10T04:41:51.653 INFO:tasks.rados.rados.0.smithi085.stdout:3541: expect (ObjNum 1213 snap 410 seq_num 1213) 2021-02-10T04:41:51.813 INFO:tasks.rados.rados.0.smithi085.stdout:3542: finishing write tid 2 to smithi08575397-32 2021-02-10T04:41:51.813 INFO:tasks.rados.rados.0.smithi085.stdout:3542: finishing write tid 3 to smithi08575397-32 2021-02-10T04:41:51.813 INFO:tasks.rados.rados.0.smithi085.stdout:3542: finishing write tid 4 to smithi08575397-32 2021-02-10T04:41:51.813 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 32 v 1482 (ObjNum 1216 snap 411 seq_num 1216) dirty exists 2021-02-10T04:41:51.813 INFO:tasks.rados.rados.0.smithi085.stdout:3542: left oid 32 (ObjNum 1216 snap 411 seq_num 1216) 2021-02-10T04:41:51.813 INFO:tasks.rados.rados.0.smithi085.stdout:3540: done (9 left) 2021-02-10T04:41:51.813 INFO:tasks.rados.rados.0.smithi085.stdout:3541: done (8 left) 2021-02-10T04:41:51.813 INFO:tasks.rados.rados.0.smithi085.stdout:3542: done (7 left) 2021-02-10T04:41:51.814 INFO:tasks.rados.rados.0.smithi085.stdout:3543: done (6 left) 2021-02-10T04:41:51.814 INFO:tasks.rados.rados.0.smithi085.stdout:3547: delete oid 40 current snap is 411 2021-02-10T04:41:51.814 INFO:tasks.rados.rados.0.smithi085.stdout:3545: finishing copy_from to smithi08575397-6 2021-02-10T04:41:51.814 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 1583 (ObjNum 1175 snap 401 seq_num 1175) dirty exists 2021-02-10T04:41:51.814 INFO:tasks.rados.rados.0.smithi085.stdout:3545: finishing copy_from racing read to smithi08575397-6 2021-02-10T04:41:51.814 INFO:tasks.rados.rados.0.smithi085.stdout:3546: finishing copy_from to smithi08575397-5 2021-02-10T04:41:51.814 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 1356 (ObjNum 924 snap 306 seq_num 924) dirty exists 2021-02-10T04:41:51.814 INFO:tasks.rados.rados.0.smithi085.stdout:3546: finishing copy_from racing read to smithi08575397-5 2021-02-10T04:41:51.815 INFO:tasks.rados.rados.0.smithi085.stdout:3545: done (6 left) 2021-02-10T04:41:51.815 INFO:tasks.rados.rados.0.smithi085.stdout:3546: done (5 left) 2021-02-10T04:41:51.815 INFO:tasks.rados.rados.0.smithi085.stdout:3547: done (4 left) 2021-02-10T04:41:51.815 INFO:tasks.rados.rados.0.smithi085.stdout:3548: snap_remove snap 371 2021-02-10T04:41:51.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:51 smithi085 conmon[29319]: cluster 2021-02-10T04:41:51.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:51 smithi085 conmon[29319]: 2021-02-10T04:41:50 2021-02-10T04:41:51.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:51 smithi085 conmon[29319]: .395293+0000 osd.4 (osd.4) 1483 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:51.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:51 smithi085 conmon[29319]: cluster 2021-02-10T04:41:50.395306+0000 osd.4 ( 2021-02-10T04:41:51.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:51 smithi085 conmon[29319]: osd.4) 1484 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:51.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:51 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:41:51.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:51 smithi085 conmon[29319]: 04:41:50.395313+0000 osd.4 (osd.4) 1485 : 2021-02-10T04:41:51.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:51 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:51.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:51 smithi085 conmon[29319]: cluster 2021-02-10T04:41:51.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:51 smithi085 conmon[29319]: 2021-02-10T04:41:50.395319+0000 osd.4 (osd.4) 1486 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:51.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:51 smithi085 conmon[29319]: cluster 2021-02-10T04:41:50.533227+0000 mon.a (mon.0) 1857 : cluster [DBG] osdmap e891: 8 total, 8 up, 7 in 2021-02-10T04:41:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:51 smithi085 conmon[35961]: cluster 2021-02-10T04:41:50.395293+0000 2021-02-10T04:41:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:51 smithi085 conmon[35961]: osd.4 (osd.4) 1483 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:51 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:41:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:51 smithi085 conmon[35961]: 41:50.395306+0000 osd.4 (osd.4) 1484 : cluster 2021-02-10T04:41:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:51 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:51 smithi085 conmon[35961]: cluster 2021-02-10T04:41:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:51 smithi085 conmon[35961]: 2021-02-10T04:41:50.395313+0000 osd.4 2021-02-10T04:41:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:51 smithi085 conmon[35961]: (osd.4) 1485 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:51 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:41:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:51 smithi085 conmon[35961]: 02-10T04:41:50.395319+0000 osd.4 2021-02-10T04:41:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:51 smithi085 conmon[35961]: (osd.4) 1486 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:51 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:41:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:51 smithi085 conmon[35961]: 02-10T04:41:50.533227+0000 2021-02-10T04:41:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:51 smithi085 conmon[35961]: mon.a (mon.0) 1857 : cluster [DBG] osdmap e891: 8 total, 8 up, 7 in 2021-02-10T04:41:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:51 smithi085 conmon[35961]: 2021-02-10T04:41:52.551 INFO:tasks.rados.rados.0.smithi085.stdout:3548: done (4 left) 2021-02-10T04:41:52.551 INFO:tasks.rados.rados.0.smithi085.stdout:3549: snap_create 2021-02-10T04:41:52.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:52 smithi164 conmon[33677]: debug 2021-02-10T04:41:52.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:52 smithi164 conmon[33677]: 2021-02-10T04:41:52.417+0000 7fbec19a1700 -1 osd.4 892 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:41:52.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:52 smithi164 conmon[33677]: 2021-02-10T04:41:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:52 smithi164 conmon[29288]: cluster 2021-02-10T04:41:51.391411+0000 osd.4 ( 2021-02-10T04:41:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:52 smithi164 conmon[29288]: osd.4) 1487 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:52 smithi164 conmon[29288]: cluster 2021-02-10T04:41:51.391423 2021-02-10T04:41:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:52 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1488 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:52 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:41:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:52 smithi164 conmon[29288]: 02-10T04:41:51.391431+0000 osd.4 (osd 2021-02-10T04:41:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:52 smithi164 conmon[29288]: .4) 1489 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:52 smithi164 conmon[29288]: cluster 2021-02-10T04:41:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:52 smithi164 conmon[29288]: 2021-02-10T04:41: 2021-02-10T04:41:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:52 smithi164 conmon[29288]: 51.391437+0000 osd.4 (osd.4) 1490 : cluster 2021-02-10T04:41:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:52 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:52 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:41:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:52 smithi164 conmon[29288]: 02-10T04:41:51.543297+0000 mon.a ( 2021-02-10T04:41:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:52 smithi164 conmon[29288]: mon.0) 1858 : cluster [DBG] osdmap e892: 8 total, 8 up, 7 in 2021-02-10T04:41:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:52 smithi164 conmon[29288]: cluster 2021-02-10T04:41:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:52 smithi164 conmon[29288]: 2021-02-10T04:41:52. 2021-02-10T04:41:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:52 smithi164 conmon[29288]: 222647+0000 mgr.y (mgr.14140) 839 : cluster [DBG] 2021-02-10T04:41:52.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:52 smithi164 conmon[29288]: pgmap v1489: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 719 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 2.2 MiB/s wr, 23 op/s 2021-02-10T04:41:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[29319]: cluster 2021-02-10T04:41:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[29319]: 2021-02-10T04:41:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[29319]: 2021-02-10T04:41:51.391411 2021-02-10T04:41:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1487 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[29319]: cluster 2021-02-10T04:41:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[29319]: 2021-02-10T04:41:51.391423+0000 osd.4 2021-02-10T04:41:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[29319]: (osd.4) 1488 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[29319]: cluster 2021-02-10T04:41:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[29319]: 2021-02-10T04:41:51.391431 2021-02-10T04:41:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1489 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[29319]: 2021-02-10T04:41:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[29319]: cluster 2021-02-10T04:41:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:41:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[29319]: 41:51.391437+0000 osd.4 2021-02-10T04:41:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[29319]: (osd.4) 1490 : cluster 2021-02-10T04:41:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[29319]: cluster 2021-02-10T04:41:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[29319]: 2021-02-10T04:41 2021-02-10T04:41:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[29319]: :51.543297+0000 mon.a (mon.0) 2021-02-10T04:41:52.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[29319]: 1858 : cluster [DBG] osdmap e892: 8 total, 8 up, 7 in 2021-02-10T04:41:52.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:41:52.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[29319]: 02-10T04:41:52.222647+0000 mgr.y 2021-02-10T04:41:52.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[29319]: (mgr.14140) 839 : cluster [DBG] pgmap v1489: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 719 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 2.2 MiB/s wr, 23 op/s 2021-02-10T04:41:52.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[35961]: cluster 2021-02-10T04:41:52.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[35961]: 2021-02-10T04:41:52.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[35961]: 2021-02-10T04:41:51. 2021-02-10T04:41:52.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[35961]: 391411+0000 osd.4 (osd.4) 1487 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:52.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[35961]: cluster 2021-02-10T04:41:51.391423 2021-02-10T04:41:52.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1488 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:52.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:41:52.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[35961]: -10T04:41:51.391431+0000 osd.4 (osd 2021-02-10T04:41:52.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[35961]: .4) 1489 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:52.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:41:52.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[35961]: 04:41:51.391437+0000 osd.4 (osd.4 2021-02-10T04:41:52.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[35961]: ) 1490 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:52.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:41:52.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[35961]: T04:41:51.543297+0000 mon.a (mon.0) 1858 : cluster [DBG] osdmap e892: 8 total, 8 up, 7 in 2021-02-10T04:41:52.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[35961]: cluster 2021-02-10T04:41:52.222647+0000 2021-02-10T04:41:52.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[35961]: mgr.y (mgr.14140) 839 : cluster [DBG] pgmap v1489: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 719 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 2.2 MiB/s wr, 23 op/s 2021-02-10T04:41:52.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:52 smithi085 conmon[35961]: 2021-02-10T04:41:53.553 INFO:tasks.rados.rados.0.smithi085.stdout:3549: done (4 left) 2021-02-10T04:41:53.553 INFO:tasks.rados.rados.0.smithi085.stdout:3550: write oid 38 current snap is 412 2021-02-10T04:41:53.554 INFO:tasks.rados.rados.0.smithi085.stdout:3550: seq_num 1218 ranges {470926=669762,1574362=700418} 2021-02-10T04:41:53.559 INFO:tasks.rados.rados.0.smithi085.stdout:3550: writing smithi08575397-38 from 470926 to 1140688 tid 1 2021-02-10T04:41:53.560 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[29319]: cluster 2021-02-10T04:41:53.560 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[29319]: 2021-02-10T04:41 2021-02-10T04:41:53.560 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[29319]: :52.418329+0000 osd.4 (osd.4) 1491 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:53.560 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:41:53.560 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[29319]: 04:41:52.418339+0000 osd.4 (osd.4 2021-02-10T04:41:53.560 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[29319]: ) 1492 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:53.561 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[29319]: cluster 2021-02-10T04:41:53.561 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[29319]: 2021-02-10T04:41:52.418346 2021-02-10T04:41:53.561 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1493 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:53.561 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[29319]: 2021-02-10T04:41:53.561 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[29319]: cluster 2021-02-10T04:41: 2021-02-10T04:41:53.561 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[29319]: 52.418353+0000 osd.4 (osd.4) 1494 : 2021-02-10T04:41:53.561 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:53.561 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:41:53.561 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[29319]: T04:41:52.550640+0000 mon.a ( 2021-02-10T04:41:53.561 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[29319]: mon.0) 1859 : cluster [DBG] osdmap e893: 8 total, 8 up, 7 in 2021-02-10T04:41:53.562 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:41:53.562 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[29319]: 02-10T04:41:52.822469+0000 osd.1 ( 2021-02-10T04:41:53.562 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[29319]: osd.1) 461 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:41:53.562 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[29319]: cluster 2021-02-10T04:41:53.562 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[29319]: 2021-02-10T04:41:52.823366+0000 2021-02-10T04:41:53.562 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[29319]: osd.1 (osd.1) 462 : cluster [DBG] 2021-02-10T04:41:53.562 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[29319]: purged_snaps scrub ok 2021-02-10T04:41:53.567 INFO:tasks.rados.rados.0.smithi085.stdout:3550: writing smithi08575397-38 from 1574362 to 2274780 tid 2 2021-02-10T04:41:53.567 INFO:tasks.rados.rados.0.smithi085.stdout:3551: write (excl) oid 40 current snap is 412 2021-02-10T04:41:53.567 INFO:tasks.rados.rados.0.smithi085.stdout:3551: seq_num 1219 ranges {487733=745011,1632844=560429} 2021-02-10T04:41:53.574 INFO:tasks.rados.rados.0.smithi085.stdout:3551: writing smithi08575397-40 from 487733 to 1232744 tid 1 2021-02-10T04:41:53.581 INFO:tasks.rados.rados.0.smithi085.stdout:3551: writing smithi08575397-40 from 1632844 to 2193273 tid 2 2021-02-10T04:41:53.581 INFO:tasks.rados.rados.0.smithi085.stdout:3552: read oid 23 snap -1 2021-02-10T04:41:53.581 INFO:tasks.rados.rados.0.smithi085.stdout:3552: expect (ObjNum 924 snap 306 seq_num 924) 2021-02-10T04:41:53.581 INFO:tasks.rados.rados.0.smithi085.stdout:3553: snap_remove snap 387 2021-02-10T04:41:53.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:53 smithi164 conmon[33677]: debug 2021-02-10T04:41:53.379+0000 7fbec19a1700 -1 osd.4 893 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:41:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:53 smithi164 conmon[29288]: cluster 2021-02-10T04:41:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:53 smithi164 conmon[29288]: 2021-02-10T04:41:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:53 smithi164 conmon[29288]: 2021 2021-02-10T04:41:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:53 smithi164 conmon[29288]: - 2021-02-10T04:41:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:53 smithi164 conmon[29288]: 02 2021-02-10T04:41:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:53 smithi164 conmon[29288]: -10T04:41:52.418329 2021-02-10T04:41:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:53 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1491 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:53 smithi164 conmon[29288]: cluster 2021-02-10T04:41:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:53 smithi164 conmon[29288]: 2021-02-10T04:41:52.418339+0000 osd.4 2021-02-10T04:41:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:53 smithi164 conmon[29288]: (osd.4) 1492 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:53 smithi164 conmon[29288]: 2021-02-10T04:41:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:53 smithi164 conmon[29288]: cluster 2021-02-10T04:41:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:53 smithi164 conmon[29288]: 2021-02-10T04:41:52.418346 2021-02-10T04:41:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:53 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1493 : cluster [WRN] 2021-02-10T04:41:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:53 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:53 smithi164 conmon[29288]: cluster 2021-02-10T04:41: 2021-02-10T04:41:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:53 smithi164 conmon[29288]: 52.418353+0000 osd.4 (osd.4) 2021-02-10T04:41:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:53 smithi164 conmon[29288]: 1494 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:53 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:41:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:53 smithi164 conmon[29288]: 41:52.550640+0000 mon.a (mon.0) 1859 : cluster 2021-02-10T04:41:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:53 smithi164 conmon[29288]: [DBG] osdmap e893: 8 total, 8 up, 7 in 2021-02-10T04:41:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:53 smithi164 conmon[29288]: cluster 2021-02-10T04:41:52 2021-02-10T04:41:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:53 smithi164 conmon[29288]: .822469+0000 osd.1 (osd.1) 461 : cluster [DBG] 2021-02-10T04:41:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:53 smithi164 conmon[29288]: purged_snaps scrub starts 2021-02-10T04:41:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:53 smithi164 conmon[29288]: cluster 2021-02-10T04:41:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:53 smithi164 conmon[29288]: 2021-02-10T04:41:52.823366+0000 osd.1 (osd 2021-02-10T04:41:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:53 smithi164 conmon[29288]: .1) 462 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:41:53.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[35961]: cluster 2021-02-10T04:41:52.418329+0000 osd.4 2021-02-10T04:41:53.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[35961]: (osd.4) 1491 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:53.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[35961]: cluster 2021-02-10T04:41:52.418339+0000 osd.4 (osd.4 2021-02-10T04:41:53.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[35961]: ) 1492 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:53.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:41:53.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[35961]: 04:41:52.418346+0000 osd.4 (osd.4) 1493 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:53.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[35961]: 2021-02-10T04:41:53.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:41:53.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[35961]: 10T04:41:52.418353+0000 osd.4 (osd.4) 1494 : cluster 2021-02-10T04:41:53.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:53.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[35961]: cluster 2021-02-10T04:41:53.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[35961]: 2021-02-10T04:41:52.550640+0000 mon.a (mon 2021-02-10T04:41:53.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[35961]: .0) 1859 : cluster [DBG] osdmap e893: 8 total, 8 up, 7 in 2021-02-10T04:41:53.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[35961]: cluster 2021-02-10T04:41:53.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[35961]: 2021-02-10T04:41:52.822469+0000 osd.1 (osd.1) 2021-02-10T04:41:53.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[35961]: 461 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:41:53.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[35961]: cluster 2021-02-10T04:41:52.823366+0000 osd.1 (osd 2021-02-10T04:41:53.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:53 smithi085 conmon[35961]: .1) 462 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:41:54.428 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:41:54.520 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:41:54.585 INFO:tasks.rados.rados.0.smithi085.stdout:3553: done (7 left) 2021-02-10T04:41:54.586 INFO:tasks.rados.rados.0.smithi085.stdout:3554: snap_remove snap 346 2021-02-10T04:41:54.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[29319]: cluster 2021-02-10T04:41:53.379840+0000 osd.4 (osd.4) 1495 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:54.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[29319]: cluster 2021-02-10T04:41:53.379845+0000 osd.4 (osd.4) 1496 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:54.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[29319]: cluster 2021-02-10T04:41:53.379849+0000 osd.4 (osd.4) 1497 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:54.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[29319]: cluster 2021-02-10T04:41:53.379852+0000 osd.4 (osd.4) 1498 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:54.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[29319]: cluster 2021-02-10T04:41:53.552262+0000 mon.a (mon.0) 1860 : cluster [DBG] osdmap e894: 8 total, 8 up, 7 in 2021-02-10T04:41:54.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[29319]: cluster 2021-02-10T04:41:54.223321+0000 mgr.y (mgr.14140) 840 : cluster [DBG] pgmap v1492: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 724 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 6.6 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2021-02-10T04:41:54.596 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: cluster 2021-02-10T04:41:54.596 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: 2021-02-10T04:41:54.596 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: 2021 2021-02-10T04:41:54.596 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: - 2021-02-10T04:41:54.596 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: 02 2021-02-10T04:41:54.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: - 2021-02-10T04:41:54.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: 10 2021-02-10T04:41:54.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: T 2021-02-10T04:41:54.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: 04 2021-02-10T04:41:54.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: : 2021-02-10T04:41:54.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: 41 2021-02-10T04:41:54.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: : 2021-02-10T04:41:54.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: 53 2021-02-10T04:41:54.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: . 2021-02-10T04:41:54.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: 379840 2021-02-10T04:41:54.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: +0000 2021-02-10T04:41:54.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: 2021-02-10T04:41:54.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: osd.4 2021-02-10T04:41:54.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: ( 2021-02-10T04:41:54.598 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: osd 2021-02-10T04:41:54.598 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: . 2021-02-10T04:41:54.598 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: 4 2021-02-10T04:41:54.598 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: ) 2021-02-10T04:41:54.598 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: 1495 2021-02-10T04:41:54.598 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: : 2021-02-10T04:41:54.598 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: cluster 2021-02-10T04:41:54.598 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: 2021-02-10T04:41:54.598 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:54.598 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: cluster 2021-02-10T04:41:53.379845+0000 osd.4 (osd.4) 1496 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:54.598 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: cluster 2021-02-10T04:41:53.379849+0000 osd.4 (osd.4) 1497 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:54.598 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: cluster 2021-02-10T04:41:54.599 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: 2021-02-10T04:41:53.379852+0000 osd.4 (osd.4) 1498 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:54.599 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: 2021-02-10T04:41:54.599 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:41:54.599 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: -02-10T04:41:53.552262+0000 mon.a (mon.0) 2021-02-10T04:41:54.599 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: 1860 : cluster [DBG] osdmap e894: 8 total, 8 up, 7 in 2021-02-10T04:41:54.599 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: cluster 2021-02-10T04:41:54.223321 2021-02-10T04:41:54.599 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 840 : cluster [DBG] pgmap v1492: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 724 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 6.6 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2021-02-10T04:41:54.599 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[35961]: 2021-02-10T04:41:54.600 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.3 dump_ops_in_flight 2021-02-10T04:41:54.645 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:54 smithi164 conmon[33677]: debug 2021-02-10T04:41:54.348+0000 7fbec19a1700 -1 osd.4 894 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:41:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:54 smithi164 conmon[29288]: cluster 2021-02-10T04:41:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:54 smithi164 conmon[29288]: 2021-02 2021-02-10T04:41:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:54 smithi164 conmon[29288]: -10T04:41:53.379840+0000 osd.4 (osd.4) 1495 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:54 smithi164 conmon[29288]: cluster 2021-02-10T04:41:53 2021-02-10T04:41:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:54 smithi164 conmon[29288]: .379845+0000 osd.4 (osd.4) 1496 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:54 smithi164 conmon[29288]: cluster 2021-02-10T04:41:53.379849+0000 osd.4 ( 2021-02-10T04:41:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:54 smithi164 conmon[29288]: osd.4) 1497 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:54.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:54 smithi164 conmon[29288]: cluster 2021-02-10T04:41:54.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:54 smithi164 conmon[29288]: 2021-02-10T04:41:53.379852+0000 osd.4 (osd.4) 1498 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:54.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:54 smithi164 conmon[29288]: cluster 2021-02-10T04:41:53.552262+0000 mon.a (mon.0 2021-02-10T04:41:54.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:54 smithi164 conmon[29288]: ) 1860 : cluster [DBG] osdmap e894: 8 total, 8 up, 7 in 2021-02-10T04:41:54.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:54 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:41:54.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:54 smithi164 conmon[29288]: 04:41:54.223321+0000 mgr.y (mgr.14140 2021-02-10T04:41:54.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:54 smithi164 conmon[29288]: ) 840 : cluster [DBG] pgmap v1492: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 724 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 6.6 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2021-02-10T04:41:54.725 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 6 1 2021-02-10T04:41:54.838 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:41:54.931 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:41:54 smithi085 conmon[49106]: debug 2021-02-10T04:41:54.802+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:41:54.944 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:41:54.967 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:41:55.183 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:41:55.183 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:41:55.183 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:41:55.192 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:41:55 smithi085 conmon[52836]: debug 2021-02-10T04:41:55.074+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:41:55.284 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:41:55.319 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:41:55.580 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: cluster 2021-02-10T04:41:55.581 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: 2021-02-10T04:41:55.581 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: 2021 2021-02-10T04:41:55.581 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: - 2021-02-10T04:41:55.581 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: 02 2021-02-10T04:41:55.581 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: - 2021-02-10T04:41:55.581 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: 10 2021-02-10T04:41:55.581 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: T 2021-02-10T04:41:55.581 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: 04 2021-02-10T04:41:55.581 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: : 2021-02-10T04:41:55.581 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: 41 2021-02-10T04:41:55.581 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: : 2021-02-10T04:41:55.581 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: 54 2021-02-10T04:41:55.581 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: . 2021-02-10T04:41:55.582 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: 349331 2021-02-10T04:41:55.582 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: +0000 2021-02-10T04:41:55.582 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: 2021-02-10T04:41:55.582 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: osd.4 2021-02-10T04:41:55.582 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: (osd.4) 1499 : cluster [WRN] 2021-02-10T04:41:55.582 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:55.582 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:41:55.582 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: -02-10T04:41:54. 2021-02-10T04:41:55.582 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: 349341+0000 osd.4 (osd.4) 1500 : cluster 2021-02-10T04:41:55.582 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:55.582 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:41:55.582 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: -10T04:41:54.349349+0000 2021-02-10T04:41:55.582 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: osd.4 (osd.4) 1501 : cluster [WRN] 2021-02-10T04:41:55.583 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:55.583 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:41:55.583 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: 10T04:41:54.349356+0000 2021-02-10T04:41:55.583 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: osd.4 (osd.4) 1502 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:55.583 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: 2021-02-10T04:41:55.583 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:41:55.583 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: 04:41:54.576904+0000 mon.a (mon 2021-02-10T04:41:55.583 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: .0) 1861 : cluster [DBG] osdmap e895: 8 total, 8 up, 7 in 2021-02-10T04:41:55.583 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: cephadm 2021 2021-02-10T04:41:55.583 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: -02-10T04:41:55 2021-02-10T04:41:55.583 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: .141967+0000 mgr.y (mgr.14140) 841 : 2021-02-10T04:41:55.583 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[29288]: cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:41:55.584 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[44544]: debug 2021-02-10T04:41:55.420+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:41:55.584 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[33677]: debug 2021-02-10T04:41:55.584 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:55 smithi164 conmon[33677]: 2021-02-10T04:41:55.383+0000 7fbec19a1700 -1 osd.4 895 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:41:55.594 INFO:tasks.rados.rados.0.smithi085.stdout:3554: done (7 left) 2021-02-10T04:41:55.595 INFO:tasks.rados.rados.0.smithi085.stdout:3555: read oid 31 snap -1 2021-02-10T04:41:55.595 INFO:tasks.rados.rados.0.smithi085.stdout:3555: expect (ObjNum 1097 snap 368 seq_num 1097) 2021-02-10T04:41:55.595 INFO:tasks.rados.rados.0.smithi085.stdout:3556: read oid 35 snap 399 2021-02-10T04:41:55.595 INFO:tasks.rados.rados.0.smithi085.stdout:3556: expect (ObjNum 1162 snap 398 seq_num 1162) 2021-02-10T04:41:55.595 INFO:tasks.rados.rados.0.smithi085.stdout:3557: snap_remove snap 342 2021-02-10T04:41:55.821 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:41:55.831 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:55 smithi085 conmon[35961]: cluster 2021-02-10T04:41:54.349331+0000 2021-02-10T04:41:55.831 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:55 smithi085 conmon[35961]: osd.4 (osd.4) 1499 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:55.831 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:55 smithi085 conmon[35961]: cluster 2021-02-10T04:41:54.349341 2021-02-10T04:41:55.831 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:55 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1500 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:55.831 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:55 smithi085 conmon[35961]: cluster 2021-02-10T04:41:54. 2021-02-10T04:41:55.831 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:55 smithi085 conmon[35961]: 349349+0000 osd.4 (osd.4) 1501 : cluster [WRN] 2021-02-10T04:41:55.831 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:55 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:55.831 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:55 smithi085 conmon[35961]: cluster 2021-02-10T04:41:54 2021-02-10T04:41:55.831 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:55 smithi085 conmon[35961]: .349356+0000 osd.4 (osd.4) 1502 : cluster 2021-02-10T04:41:55.831 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:55 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:55.831 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:55 smithi085 conmon[35961]: cluster 2021-02-10T04:41:54.576904 2021-02-10T04:41:55.832 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:55 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1861 : cluster [DBG] osdmap e895: 8 total, 8 up, 7 in 2021-02-10T04:41:55.832 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:55 smithi085 conmon[35961]: cephadm 2021-02- 2021-02-10T04:41:55.832 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:55 smithi085 conmon[35961]: 10T04:41:55.141967+0000 mgr.y (mgr 2021-02-10T04:41:55.832 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:55 smithi085 conmon[35961]: .14140) 841 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:41:55.832 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:55 smithi085 conmon[29319]: cluster 2021-02-10T04:41:54.349331+0000 osd.4 (osd.4) 2021-02-10T04:41:55.832 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:55 smithi085 conmon[29319]: 1499 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:55.832 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:55 smithi085 conmon[29319]: cluster 2021-02-10T04:41:54.349341+0000 osd.4 (osd.4) 1500 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:55.832 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:55 smithi085 conmon[29319]: cluster 2021-02-10T04:41:54.349349+0000 osd.4 (osd.4) 1501 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:55.832 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:55 smithi085 conmon[29319]: cluster 2021-02-10T04:41:54.349356+0000 osd.4 (osd.4) 1502 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:55.832 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:55 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:41:55.832 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:55 smithi085 conmon[29319]: 10T04:41:54.576904+0000 mon.a (mon.0) 1861 : cluster [DBG] osdmap e895: 8 total, 8 up, 7 in 2021-02-10T04:41:55.833 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:55 smithi085 conmon[29319]: cephadm 2021-02-10T04:41:55.141967+0000 mgr.y 2021-02-10T04:41:55.833 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:55 smithi085 conmon[29319]: (mgr.14140) 841 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:41:55.927 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:41:55.969 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:41:56.385 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:41:56.441 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[52836]: debug 2021-02-10T04:41:56.130+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:41:56.601 INFO:tasks.rados.rados.0.smithi085.stdout:3557: done (9 left) 2021-02-10T04:41:56.601 INFO:tasks.rados.rados.0.smithi085.stdout:3558: read oid 18 snap -1 2021-02-10T04:41:56.601 INFO:tasks.rados.rados.0.smithi085.stdout:3558: expect (ObjNum 1161 snap 398 seq_num 1161) 2021-02-10T04:41:56.601 INFO:tasks.rados.rados.0.smithi085.stdout:3559: write (excl) oid 32 current snap is 412 2021-02-10T04:41:56.602 INFO:tasks.rados.rados.0.smithi085.stdout:3559: seq_num 1220 ranges {504540=420260,1574973=704087} 2021-02-10T04:41:56.602 INFO:tasks.rados.rados.0.smithi085.stdout:3559: writing smithi08575397-32 from 504540 to 924800 tid 1 2021-02-10T04:41:56.610 INFO:tasks.rados.rados.0.smithi085.stdout:3559: writing smithi08575397-32 from 1574973 to 2279060 tid 2 2021-02-10T04:41:56.611 INFO:tasks.rados.rados.0.smithi085.stdout:3550: finishing write tid 1 to smithi08575397-38 2021-02-10T04:41:56.611 INFO:tasks.rados.rados.0.smithi085.stdout:3560: snap_remove snap 337 2021-02-10T04:41:56.809 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set noscrub 2021-02-10T04:41:56.827 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:56 smithi164 conmon[29288]: cluster 2021-02-10T04:41:55.384640+0000 osd.4 (osd.4) 1503 : cluster 2021-02-10T04:41:56.827 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:56 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:56.827 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:56 smithi164 conmon[29288]: cluster 2021-02-10T04:41:55.384652+0000 osd.4 (osd.4) 1504 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:56.827 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:56 smithi164 conmon[29288]: cluster 2021-02-10T04:41:55.384659+0000 osd.4 (osd.4) 1505 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:56.827 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:56 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:41:56.827 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:56 smithi164 conmon[29288]: 41:55.384665+0000 osd.4 (osd.4) 1506 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:56.828 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:56 smithi164 conmon[29288]: cluster 2021-02-10T04:41:55.592061+0000 mon.a (mon.0) 1862 : cluster [DBG] 2021-02-10T04:41:56.828 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:56 smithi164 conmon[29288]: osdmap e896: 8 total, 8 up, 7 in 2021-02-10T04:41:56.828 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:56 smithi164 conmon[29288]: audit 2021-02-10T04:41:55.740278+0000 mon.a (mon.0) 1863 : audit 2021-02-10T04:41:56.828 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:56 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:41:56.828 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:56 smithi164 conmon[29288]: cephadm 2021-02-10T 2021-02-10T04:41:56.828 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:56 smithi164 conmon[29288]: 04:41:55.741166+0000 mgr.y (mgr.14140) 842 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:41:56.828 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:56 smithi164 conmon[29288]: cluster 2021-02-10T04:41:56.223856 2021-02-10T04:41:56.828 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:56 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 843 : cluster [DBG] pgmap v1495: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 722 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 3.0 MiB/s wr, 18 op/s 2021-02-10T04:41:56.828 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:56 smithi164 conmon[29288]: audit 2021-02-10T04:41 2021-02-10T04:41:56.828 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:56 smithi164 conmon[29288]: :56.302925+0000 mon.a (mon.0) 1864 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:41:56.828 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:56 smithi164 conmon[33677]: debug 2021-02-10T04:41:56.425+0000 7fbec19a1700 -1 osd.4 896 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:41:56.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[35961]: cluster 2021-02-10T04:41:56.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[35961]: 2021-02-10T04:41:56.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[35961]: 2021-02-10T04:41:55 2021-02-10T04:41:56.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[35961]: .384640+0000 osd.4 (osd.4) 1503 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:56.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[35961]: cluster 2021-02-10T04:41: 2021-02-10T04:41:56.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[35961]: 55.384652+0000 osd.4 (osd.4) 1504 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:56.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[35961]: cluster 2021-02-10T04:41:56.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[35961]: 2021-02-10T04:41:55.384659+0000 osd.4 (osd.4) 1505 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:56.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[35961]: cluster 2021-02-10T04:41:55.384665+0000 2021-02-10T04:41:56.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[35961]: osd.4 (osd.4) 1506 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:56.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[35961]: cluster 2021-02-10T04:41:56.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[35961]: 2021-02-10T04:41:55 2021-02-10T04:41:56.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[35961]: .592061+0000 mon.a (mon.0) 1862 : cluster 2021-02-10T04:41:56.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[35961]: [DBG] osdmap e896: 8 total, 8 up, 7 in 2021-02-10T04:41:56.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:41:56.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[35961]: 04:41:55.740278+0000 mon.a (mon.0) 1863 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:41:56.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[35961]: cephadm 2021-02-10T04:41:55.741166+0000 2021-02-10T04:41:56.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[35961]: mgr.y (mgr.14140) 842 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:41:56.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:41:56.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[35961]: -02-10T04:41:56.223856+0000 mgr.y (mgr.14140) 843 : cluster [DBG] pgmap v1495: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 722 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 3.0 MiB/s wr, 18 op/s 2021-02-10T04:41:56.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[35961]: audit 2021 2021-02-10T04:41:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[35961]: -02-10T04:41:56.302925+0000 2021-02-10T04:41:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[35961]: mon.a (mon.0) 1864 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:41:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: cluster 2021-02-10T04:41:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: 2021-02-10T04:41:55. 2021-02-10T04:41:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: 384640+0000 osd.4 (osd.4) 1503 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: cluster 2021-02-10T04:41:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: 2021-02-10T04:41:55.384652+0000 osd.4 (osd.4) 1504 : cluster [WRN] 2021-02-10T04:41:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: cluster 2021-02-10T04:41:55. 2021-02-10T04:41:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: 384659+0000 osd.4 (osd.4) 1505 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:56.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:41:56.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: -02-10T04:41:55.384665+0000 osd.4 2021-02-10T04:41:56.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: (osd.4) 1506 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:56.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:41:56.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: -02-10T04:41:55.592061+0000 mon.a (mon.0) 1862 : cluster [DBG] osdmap e896: 8 total, 8 up, 7 in 2021-02-10T04:41:56.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: audit 2021-02-10T04:41:56.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: 2021-02-10T04:41:56.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: 2021 2021-02-10T04:41:56.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: - 2021-02-10T04:41:56.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: 02 2021-02-10T04:41:56.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: - 2021-02-10T04:41:56.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: 10 2021-02-10T04:41:56.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: T 2021-02-10T04:41:56.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: 04 2021-02-10T04:41:56.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: : 2021-02-10T04:41:56.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: 41 2021-02-10T04:41:56.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: : 2021-02-10T04:41:56.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: 55 2021-02-10T04:41:56.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: . 2021-02-10T04:41:56.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: 740278 2021-02-10T04:41:56.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: +0000 2021-02-10T04:41:56.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: mon.a (mon.0) 1863 : 2021-02-10T04:41:56.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:41:56.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: cephadm 2021-02-10T04:41:55.741166+0000 2021-02-10T04:41:56.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: mgr.y (mgr.14140) 842 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:41:56.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:41:56.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: T04:41:56.223856+0000 mgr.y (mgr.14140) 2021-02-10T04:41:56.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: 843 : cluster [DBG] pgmap v1495: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 722 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 3.0 MiB/s wr, 18 op/s 2021-02-10T04:41:56.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:41:56.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: T04:41:56.302925+0000 mon.a (mon.0) 1864 : 2021-02-10T04:41:56.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:56 smithi085 conmon[29319]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:41:56.952 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:41:57.091 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:41:57.144 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:41:56 smithi164 conmon[37263]: debug 2021-02-10T04:41:56.825+0000 7f1c671e2700 -1 osd.5 pg_epoch: 897 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55665320b1e0) 2021-02-10T04:41:57.268 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:41:57.597 INFO:tasks.rados.rados.0.smithi085.stdout:3560: done (11 left) 2021-02-10T04:41:57.597 INFO:tasks.rados.rados.0.smithi085.stdout:3561: read oid 43 snap 360 2021-02-10T04:41:57.597 INFO:tasks.rados.rados.0.smithi085.stdout:3561: expect (ObjNum 1074 snap 357 seq_num 1074) 2021-02-10T04:41:57.597 INFO:tasks.rados.rados.0.smithi085.stdout:3562: snap_create 2021-02-10T04:41:57.597 INFO:tasks.rados.rados.0.smithi085.stdout:3550: finishing write tid 2 to smithi08575397-38 2021-02-10T04:41:57.597 INFO:tasks.rados.rados.0.smithi085.stdout:3550: finishing write tid 3 to smithi08575397-38 2021-02-10T04:41:57.598 INFO:tasks.rados.rados.0.smithi085.stdout:3550: finishing write tid 4 to smithi08575397-38 2021-02-10T04:41:57.598 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 1598 (ObjNum 1218 snap 412 seq_num 1218) dirty exists 2021-02-10T04:41:57.598 INFO:tasks.rados.rados.0.smithi085.stdout:3550: left oid 38 (ObjNum 1218 snap 412 seq_num 1218) 2021-02-10T04:41:57.598 INFO:tasks.rados.rados.0.smithi085.stdout:3551: finishing write tid 1 to smithi08575397-40 2021-02-10T04:41:57.598 INFO:tasks.rados.rados.0.smithi085.stdout:3552: expect (ObjNum 924 snap 306 seq_num 924) 2021-02-10T04:41:57.748 INFO:tasks.rados.rados.0.smithi085.stdout:3551: finishing write tid 2 to smithi08575397-40 2021-02-10T04:41:57.748 INFO:tasks.rados.rados.0.smithi085.stdout:3551: finishing write tid 3 to smithi08575397-40 2021-02-10T04:41:57.749 INFO:tasks.rados.rados.0.smithi085.stdout:3551: finishing write tid 4 to smithi08575397-40 2021-02-10T04:41:57.749 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 1833 (ObjNum 1219 snap 412 seq_num 1219) dirty exists 2021-02-10T04:41:57.749 INFO:tasks.rados.rados.0.smithi085.stdout:3551: left oid 40 (ObjNum 1219 snap 412 seq_num 1219) 2021-02-10T04:41:57.749 INFO:tasks.rados.rados.0.smithi085.stdout:3555: expect (ObjNum 1097 snap 368 seq_num 1097) 2021-02-10T04:41:57.841 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:57 smithi164 conmon[29288]: cluster 2021-02-10T04:41:56.426245 2021-02-10T04:41:57.841 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:57 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1507 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:57.841 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:57 smithi164 conmon[29288]: cluster 2021-02-10T04:41:56. 2021-02-10T04:41:57.842 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:57 smithi164 conmon[29288]: 426251+0000 osd.4 (osd.4) 1508 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:57.842 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:57 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:41:57.842 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:57 smithi164 conmon[29288]: T04:41:56.426255+0000 osd.4 (osd.4 2021-02-10T04:41:57.842 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:57 smithi164 conmon[29288]: ) 1509 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:57.842 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:57 smithi164 conmon[29288]: cluster 2021-02-10T04:41:57.842 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:57 smithi164 conmon[29288]: 2021-02-10T04:41:57.842 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:57 smithi164 conmon[29288]: 2021 2021-02-10T04:41:57.842 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:57 smithi164 conmon[29288]: -02-10T04:41:56.426258+0000 2021-02-10T04:41:57.842 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:57 smithi164 conmon[29288]: osd.4 (osd.4) 1510 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:57.842 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:57 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:41:57.842 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:57 smithi164 conmon[29288]: 10T04:41:56.594656+0000 mon.a (mon.0) 1865 : cluster [DBG] osdmap e897: 8 total, 8 up, 7 in 2021-02-10T04:41:57.842 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:57 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:41:57.843 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:57 smithi164 conmon[29288]: 02-10T04:41:57.591906+0000 mon.a (mon.0) 1866 : cluster [DBG] osdmap e898: 8 total, 8 up, 7 in 2021-02-10T04:41:57.843 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:57 smithi164 conmon[29288]: 2021-02-10T04:41:57.843 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:57 smithi164 conmon[33677]: debug 2021-02-10T04:41:57.407+0000 7fbec19a1700 -1 osd.4 897 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:41:57.855 INFO:tasks.rados.rados.0.smithi085.stdout:3556: expect (ObjNum 1162 snap 398 seq_num 1162) 2021-02-10T04:41:57.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[29319]: cluster 2021-02-10T04:41:57.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[29319]: 2021-02-10T04:41:57.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[29319]: 2021-02-10T04:41:56. 2021-02-10T04:41:57.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[29319]: 426245+0000 osd.4 (osd.4) 1507 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:57.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:41:57.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[29319]: T04:41:56.426251+0000 osd.4 (osd 2021-02-10T04:41:57.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[29319]: .4) 1508 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:57.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[29319]: cluster 2021-02-10T04:41:57.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:41:57.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[29319]: 04:41:56.426255+0000 osd.4 (osd.4) 1509 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:57.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[29319]: cluster 2021-02-10T04:41:57.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[29319]: 2021-02-10T04:41: 2021-02-10T04:41:57.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[29319]: 56.426258+0000 osd.4 (osd.4 2021-02-10T04:41:57.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[29319]: ) 1510 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:57.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:41:57.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[29319]: 02-10T04:41:56.594656+0000 2021-02-10T04:41:57.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[29319]: mon.a 2021-02-10T04:41:57.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[29319]: (mon.0 2021-02-10T04:41:57.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[29319]: ) 1865 : cluster [DBG] osdmap e897: 8 total, 8 up, 7 in 2021-02-10T04:41:57.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:41:57.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[29319]: 02-10T04:41:57.591906+0000 2021-02-10T04:41:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[29319]: mon.a (mon.0) 1866 : 2021-02-10T04:41:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[29319]: cluster [DBG] osdmap e898: 8 total, 8 up, 7 in 2021-02-10T04:41:57.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[35961]: cluster 2021-02-10T04:41:56. 2021-02-10T04:41:57.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[35961]: 426245+0000 osd.4 (osd.4) 1507 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:57.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[35961]: cluster 2021-02-10T04:41 2021-02-10T04:41:57.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[35961]: :56.426251+0000 osd.4 (osd.4) 1508 2021-02-10T04:41:57.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:57.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:41:57.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[35961]: :41:56.426255+0000 osd.4 (osd.4 2021-02-10T04:41:57.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[35961]: ) 1509 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:57.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:41:57.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[35961]: 04:41:56.426258+0000 osd.4 (osd. 2021-02-10T04:41:57.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[35961]: 4) 1510 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:57.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:41:57.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[35961]: 10T04:41:56.594656+0000 mon.a ( 2021-02-10T04:41:57.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[35961]: mon.0) 1865 : cluster [DBG] osdmap e897: 8 total, 8 up, 7 in 2021-02-10T04:41:57.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:41:57.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[35961]: -02-10T04:41:57.591906+0000 2021-02-10T04:41:57.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[35961]: mon.a (mon.0) 1866 : cluster [DBG] osdmap e898: 8 total, 8 up, 7 in 2021-02-10T04:41:57.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:57 smithi085 conmon[35961]: 2021-02-10T04:41:58.063 INFO:tasks.rados.rados.0.smithi085.stdout:3558: expect (ObjNum 1161 snap 398 seq_num 1161) 2021-02-10T04:41:58.143 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:41:57 smithi164 conmon[37263]: debug 2021-02-10T04:41:57.840+0000 7f1c671e2700 -1 osd.5 pg_epoch: 898 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x5566505ea2c0) 2021-02-10T04:41:58.213 INFO:tasks.rados.rados.0.smithi085.stdout:3559: finishing write tid 1 to smithi08575397-32 2021-02-10T04:41:58.213 INFO:tasks.rados.rados.0.smithi085.stdout:3559: finishing write tid 2 to smithi08575397-32 2021-02-10T04:41:58.213 INFO:tasks.rados.rados.0.smithi085.stdout:3559: finishing write tid 3 to smithi08575397-32 2021-02-10T04:41:58.213 INFO:tasks.rados.rados.0.smithi085.stdout:3559: finishing write tid 4 to smithi08575397-32 2021-02-10T04:41:58.213 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 32 v 1485 (ObjNum 1220 snap 412 seq_num 1220) dirty exists 2021-02-10T04:41:58.213 INFO:tasks.rados.rados.0.smithi085.stdout:3559: left oid 32 (ObjNum 1220 snap 412 seq_num 1220) 2021-02-10T04:41:58.213 INFO:tasks.rados.rados.0.smithi085.stdout:3561: expect (ObjNum 1074 snap 357 seq_num 1074) 2021-02-10T04:41:58.593 INFO:tasks.rados.rados.0.smithi085.stdout:3550: done (12 left) 2021-02-10T04:41:58.593 INFO:tasks.rados.rados.0.smithi085.stdout:3551: done (11 left) 2021-02-10T04:41:58.593 INFO:tasks.rados.rados.0.smithi085.stdout:3552: done (10 left) 2021-02-10T04:41:58.593 INFO:tasks.rados.rados.0.smithi085.stdout:3555: done (9 left) 2021-02-10T04:41:58.593 INFO:tasks.rados.rados.0.smithi085.stdout:3556: done (8 left) 2021-02-10T04:41:58.593 INFO:tasks.rados.rados.0.smithi085.stdout:3558: done (7 left) 2021-02-10T04:41:58.593 INFO:tasks.rados.rados.0.smithi085.stdout:3559: done (6 left) 2021-02-10T04:41:58.594 INFO:tasks.rados.rados.0.smithi085.stdout:3561: done (5 left) 2021-02-10T04:41:58.594 INFO:tasks.rados.rados.0.smithi085.stdout:3562: done (4 left) 2021-02-10T04:41:58.594 INFO:tasks.rados.rados.0.smithi085.stdout:3563: delete oid 44 current snap is 413 2021-02-10T04:41:58.597 INFO:tasks.rados.rados.0.smithi085.stdout:3563: done (4 left) 2021-02-10T04:41:58.597 INFO:tasks.rados.rados.0.smithi085.stdout:3564: read oid 7 snap -1 2021-02-10T04:41:58.598 INFO:tasks.rados.rados.0.smithi085.stdout:3564: expect (ObjNum 1202 snap 409 seq_num 1202) 2021-02-10T04:41:58.598 INFO:tasks.rados.rados.0.smithi085.stdout:3565: snap_create 2021-02-10T04:41:58.604 INFO:tasks.rados.rados.0.smithi085.stdout:3564: expect (ObjNum 1202 snap 409 seq_num 1202) 2021-02-10T04:41:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:58 smithi164 conmon[29288]: cluster 2021-02-10T04:41:57.407786+0000 osd.4 (osd. 2021-02-10T04:41:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:58 smithi164 conmon[29288]: 4) 1511 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:58 smithi164 conmon[29288]: cluster 2021-02-10T04:41:57.407796+0000 osd.4 (osd.4) 1512 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:58 smithi164 conmon[29288]: cluster 2021-02-10T04:41:57.407802+0000 osd.4 (osd.4) 1513 : 2021-02-10T04:41:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:58 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:58 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:41:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:58 smithi164 conmon[29288]: 02-10T04:41:57.407808+0000 osd.4 (osd.4) 1514 : 2021-02-10T04:41:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:58 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:58 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:41:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:58 smithi164 conmon[29288]: -10T04:41:58.224475+0000 mgr.y (mgr. 2021-02-10T04:41:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:58 smithi164 conmon[29288]: 14140) 844 : cluster [DBG] pgmap v1498: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 719 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 5.0 MiB/s rd, 1.5 MiB/s wr, 6 op/s 2021-02-10T04:41:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:58 smithi164 conmon[29288]: cluster 2021-02-10T04:41:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:58 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:41:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:58 smithi164 conmon[29288]: :41:58.291996+0000 mon.a (mon.0) 1867 : cluster [WRN] 2021-02-10T04:41:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:58 smithi164 conmon[29288]: Health check update: 4 slow ops, oldest one blocked for 742 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:41:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:58 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:41:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:58 smithi164 conmon[29288]: -10T04:41:58.593230+0000 mon.a ( 2021-02-10T04:41:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:58 smithi164 conmon[29288]: mon.0) 1868 : cluster [DBG] osdmap e899: 8 total, 8 up, 7 in 2021-02-10T04:41:58.897 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:58 smithi164 conmon[33677]: debug 2021-02-10T04:41:58.397+0000 7fbec19a1700 -1 osd.4 898 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:41:58.897 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:41:58 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:41:58.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[29319]: cluster 2021-02-10T04:41:57.407786+0000 osd.4 ( 2021-02-10T04:41:58.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[29319]: osd.4) 1511 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[29319]: cluster 2021-02-10T04:41:57.407796+0000 osd.4 (osd.4) 1512 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[29319]: cluster 2021-02-10T04:41:57.407802+0000 osd.4 (osd.4) 1513 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[29319]: cluster 2021-02-10T04:41:57.407808+0000 osd.4 (osd.4) 1514 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[29319]: cluster 2021-02-10T04:41:58.224475+0000 mgr.y (mgr.14140) 844 : cluster [DBG] pgmap v1498: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 719 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 5.0 MiB/s rd, 1.5 MiB/s wr, 6 op/s 2021-02-10T04:41:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[29319]: cluster 2021-02-10T04:41:58.291996+0000 mon.a (mon.0) 1867 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 742 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:41:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[29319]: cluster 2021-02-10T04:41:58.593230+0000 mon.a (mon.0) 1868 : cluster [DBG] osdmap e899: 8 total, 8 up, 7 in 2021-02-10T04:41:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[35961]: cluster 2021-02-10T04:41:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[35961]: 2021-02-10T04:41:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[35961]: 2021 2021-02-10T04:41:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[35961]: -02-10T04:41:57.407786 2021-02-10T04:41:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1511 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[35961]: cluster 2021-02-10T04:41:57. 2021-02-10T04:41:58.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[35961]: 407796+0000 osd.4 (osd.4) 1512 : cluster [WRN] 2021-02-10T04:41:58.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:58.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[35961]: cluster 2021-02-10T04:41:58.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[35961]: 2021-02-10T04:41:57.407802+0000 osd.4 (osd.4) 2021-02-10T04:41:58.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[35961]: 1513 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:58.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:41:58.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[35961]: - 2021-02-10T04:41:58.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[35961]: 02 2021-02-10T04:41:58.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[35961]: -10T04:41:57.407808 2021-02-10T04:41:58.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1514 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:58.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:41:58.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[35961]: 04:41:58.224475+0000 mgr.y (mgr. 2021-02-10T04:41:58.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[35961]: 14140) 844 : cluster [DBG] pgmap v1498: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 719 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 5.0 MiB/s rd, 1.5 MiB/s wr, 6 op/s 2021-02-10T04:41:58.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:41:58.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[35961]: 41:58.291996+0000 mon.a (mon.0) 1867 : cluster 2021-02-10T04:41:58.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[35961]: [WRN] Health check update: 4 slow ops, oldest one blocked for 742 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:41:58.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[35961]: cluster 2021-02-10T04:41:58.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[35961]: 2021-02-10T04:41:58.593230+0000 mon.a (mon.0) 1868 2021-02-10T04:41:58.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:58 smithi085 conmon[35961]: : cluster [DBG] osdmap e899: 8 total, 8 up, 7 in 2021-02-10T04:41:59.595 INFO:tasks.rados.rados.0.smithi085.stdout:3564: done (5 left) 2021-02-10T04:41:59.595 INFO:tasks.rados.rados.0.smithi085.stdout:3565: done (4 left) 2021-02-10T04:41:59.595 INFO:tasks.rados.rados.0.smithi085.stdout:3566: write (excl) oid 25 current snap is 414 2021-02-10T04:41:59.595 INFO:tasks.rados.rados.0.smithi085.stdout:3566: seq_num 1222 ranges {538154=570758,1691937=707756} 2021-02-10T04:41:59.601 INFO:tasks.rados.rados.0.smithi085.stdout:3566: writing smithi08575397-25 from 538154 to 1108912 tid 1 2021-02-10T04:41:59.609 INFO:tasks.rados.rados.0.smithi085.stdout:3566: writing smithi08575397-25 from 1691937 to 2399693 tid 2 2021-02-10T04:41:59.609 INFO:tasks.rados.rados.0.smithi085.stdout:3567: write (excl) oid 12 current snap is 414 2021-02-10T04:41:59.610 INFO:tasks.rados.rados.0.smithi085.stdout:3566: finishing write tid 1 to smithi08575397-25 2021-02-10T04:41:59.610 INFO:tasks.rados.rados.0.smithi085.stdout:3567: seq_num 1223 ranges {554961=646007,1750419=451414} 2021-02-10T04:41:59.617 INFO:tasks.rados.rados.0.smithi085.stdout:3567: writing smithi08575397-12 from 554961 to 1200968 tid 1 2021-02-10T04:41:59.623 INFO:tasks.rados.rados.0.smithi085.stdout:3567: writing smithi08575397-12 from 1750419 to 2201833 tid 2 2021-02-10T04:41:59.623 INFO:tasks.rados.rados.0.smithi085.stdout:3568: snap_remove snap 409 2021-02-10T04:41:59.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:41:59 smithi164 conmon[33677]: debug 2021-02-10T04:41:59.408+0000 7fbec19a1700 -1 osd.4 899 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:41:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:59 smithi164 conmon[29288]: cluster 2021-02-10T04:41:58.397909+0000 osd.4 ( 2021-02-10T04:41:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:59 smithi164 conmon[29288]: osd.4) 1515 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:59 smithi164 conmon[29288]: cluster 2021-02-10T04:41:58.397917+0000 osd.4 (osd.4) 1516 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:59 smithi164 conmon[29288]: cluster 2021-02-10T04:41:58.397925+0000 osd.4 (osd 2021-02-10T04:41:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:59 smithi164 conmon[29288]: .4) 1517 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:59 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:41:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:59 smithi164 conmon[29288]: -02-10T04:41:58.397931+0000 osd.4 2021-02-10T04:41:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:59 smithi164 conmon[29288]: (osd.4) 1518 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:59 smithi164 conmon[29288]: audit 2021 2021-02-10T04:41:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:59 smithi164 conmon[29288]: -02-10T04:41:58.488021+0000 2021-02-10T04:41:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:59 smithi164 conmon[29288]: mgr.y (mgr.14140) 845 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:41:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:59 smithi164 conmon[29288]: 2021-02-10T04:41:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:59 smithi164 conmon[29288]: cluster 2021-02-10T04:41:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:59 smithi164 conmon[29288]: 2021-02-10T04:41:59.593731+0000 mon.a (mon.0) 2021-02-10T04:41:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:41:59 smithi164 conmon[29288]: 1869 : cluster [DBG] osdmap e900: 8 total, 8 up, 7 in 2021-02-10T04:41:59.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[29319]: cluster 2021-02-10T04:41:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:41:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[29319]: 41:58.397909+0000 osd.4 (osd.4) 1515 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[29319]: cluster 2021-02-10T04:41: 2021-02-10T04:41:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[29319]: 58.397917+0000 osd.4 (osd.4) 1516 : cluster 2021-02-10T04:41:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:41:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[29319]: 41:58.397925+0000 osd.4 (osd.4) 1517 2021-02-10T04:41:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[29319]: cluster 2021-02-10T04:41: 2021-02-10T04:41:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[29319]: 58.397931+0000 osd.4 (osd. 2021-02-10T04:41:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[29319]: 4) 1518 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[29319]: 2021-02-10T04:41:59.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:41:59.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[29319]: :41:58.488021+0000 mgr.y (mgr. 2021-02-10T04:41:59.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[29319]: 14140) 845 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:41:59.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:41:59.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[29319]: :41:59.593731+0000 mon.a (mon.0 2021-02-10T04:41:59.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[29319]: ) 1869 : cluster [DBG] osdmap e900: 8 total, 8 up, 7 in 2021-02-10T04:41:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[35961]: cluster 2021-02-10T04:41:58.397909+0000 2021-02-10T04:41:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[35961]: osd.4 (osd.4) 1515 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:41:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[35961]: cluster 2021-02-10T04:41:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[35961]: 2021-02-10T04:41:58.397917+0000 osd.4 (osd.4) 1516 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:41:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:41:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[35961]: :41:58.397925+0000 osd.4 (osd.4) 1517 : cluster [WRN] 2021-02-10T04:41:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:41:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:41:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[35961]: -10T04:41:58.397931+0000 osd.4 (osd.4) 1518 : cluster [WRN] 2021-02-10T04:41:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:41:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[35961]: audit 2021-02-10T04:41:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[35961]: 2021-02-10T04:41:58.488021+0000 mgr.y (mgr.14140) 845 2021-02-10T04:41:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[35961]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:41:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[35961]: cluster 2021-02-10T04:41:59 2021-02-10T04:41:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[35961]: .593731+0000 mon.a (mon.0) 1869 : cluster [DBG] 2021-02-10T04:41:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:41:59 smithi085 conmon[35961]: osdmap e900: 8 total, 8 up, 7 in 2021-02-10T04:42:00.636 INFO:tasks.rados.rados.0.smithi085.stdout:3568: done (6 left) 2021-02-10T04:42:00.636 INFO:tasks.rados.rados.0.smithi085.stdout:3569: snap_remove snap 351 2021-02-10T04:42:00.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:00 smithi164 conmon[33677]: debug 2021-02-10T04:42:00.415+0000 7fbec19a1700 -1 osd.4 900 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:42:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:00 smithi164 conmon[29288]: cluster 2021-02-10T04:41:59.408987+0000 osd.4 (osd.4) 1519 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:00 smithi164 conmon[29288]: cluster 2021-02-10T04:41:59.408996+0000 osd.4 (osd.4) 1520 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:00 smithi164 conmon[29288]: cluster 2021-02-10T04:41:59.409002+0000 osd.4 (osd.4) 1521 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:00 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:42:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:00 smithi164 conmon[29288]: 41:59.409009+0000 osd.4 (osd.4) 1522 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:00 smithi164 conmon[29288]: cluster 2021-02-10T04:42:00.225161+0000 mgr.y (mgr. 2021-02-10T04:42:00.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:00 smithi164 conmon[29288]: 14140) 846 : cluster [DBG] pgmap v1501: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 703 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 587 KiB/s wr, 14 op/s 2021-02-10T04:42:00.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[35961]: cluster 2021-02-10T04:41:59. 2021-02-10T04:42:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[35961]: 408987+0000 osd.4 (osd.4) 1519 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[35961]: cluster 2021-02-10T04:41:59.408996+0000 osd.4 (osd.4) 1520 : cluster 2021-02-10T04:42:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[35961]: cluster 2021-02-10T04:42:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[35961]: 2021-02-10T04:41:59.409002 2021-02-10T04:42:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1521 : cluster [WRN] 2021-02-10T04:42:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:42:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[35961]: 04:41:59.409009+0000 2021-02-10T04:42:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[35961]: osd.4 (osd.4) 1522 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[35961]: cluster 2021-02-10T04:42:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[35961]: 2021-02-10T04:42 2021-02-10T04:42:00.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[35961]: :00.225161+0000 mgr.y (mgr.14140 2021-02-10T04:42:00.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[35961]: ) 846 : cluster [DBG] pgmap v1501: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 703 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 587 KiB/s wr, 14 op/s 2021-02-10T04:42:00.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[35961]: 2021-02-10T04:42:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[29319]: cluster 2021-02-10T04:42:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[29319]: 2021-02-10T04:42:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[29319]: 2021-02-10T04:41:59. 2021-02-10T04:42:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[29319]: 408987+0000 osd.4 (osd.4) 1519 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:42:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[29319]: -02-10T04:41:59.408996+0000 osd.4 2021-02-10T04:42:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[29319]: (osd.4) 1520 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:42:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[29319]: -02-10T04:41:59.409002+0000 osd.4 (osd. 2021-02-10T04:42:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[29319]: 4) 1521 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:00.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[29319]: cluster 2021-02-10T04:42:00.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[29319]: 2021-02-10T04:41:59.409009+0000 2021-02-10T04:42:00.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[29319]: osd.4 (osd.4) 1522 : 2021-02-10T04:42:00.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:00.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:42:00.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[29319]: 10T04:42:00.225161+0000 mgr.y (mgr.14140) 2021-02-10T04:42:00.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:00 smithi085 conmon[29319]: 846 : cluster [DBG] pgmap v1501: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 703 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 587 KiB/s wr, 14 op/s 2021-02-10T04:42:01.634 INFO:tasks.rados.rados.0.smithi085.stdout:3569: done (6 left) 2021-02-10T04:42:01.635 INFO:tasks.rados.rados.0.smithi085.stdout:3570: read oid 40 snap -1 2021-02-10T04:42:01.635 INFO:tasks.rados.rados.0.smithi085.stdout:3570: expect (ObjNum 1219 snap 412 seq_num 1219) 2021-02-10T04:42:01.636 INFO:tasks.rados.rados.0.smithi085.stdout:3571: write oid 22 current snap is 414 2021-02-10T04:42:01.636 INFO:tasks.rados.rados.0.smithi085.stdout:3571: seq_num 1224 ranges {571768=437609,1808901=711425} 2021-02-10T04:42:01.640 INFO:tasks.rados.rados.0.smithi085.stdout:3571: writing smithi08575397-22 from 571768 to 1009377 tid 1 2021-02-10T04:42:01.650 INFO:tasks.rados.rados.0.smithi085.stdout:3571: writing smithi08575397-22 from 1808901 to 2520326 tid 2 2021-02-10T04:42:01.650 INFO:tasks.rados.rados.0.smithi085.stdout:3572: snap_create 2021-02-10T04:42:01.650 INFO:tasks.rados.rados.0.smithi085.stdout:3566: finishing write tid 2 to smithi08575397-25 2021-02-10T04:42:01.650 INFO:tasks.rados.rados.0.smithi085.stdout:3566: finishing write tid 3 to smithi08575397-25 2021-02-10T04:42:01.650 INFO:tasks.rados.rados.0.smithi085.stdout:3566: finishing write tid 4 to smithi08575397-25 2021-02-10T04:42:01.650 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 25 v 1477 (ObjNum 1222 snap 414 seq_num 1222) dirty exists 2021-02-10T04:42:01.650 INFO:tasks.rados.rados.0.smithi085.stdout:3566: left oid 25 (ObjNum 1222 snap 414 seq_num 1222) 2021-02-10T04:42:01.651 INFO:tasks.rados.rados.0.smithi085.stdout:3567: finishing write tid 1 to smithi08575397-12 2021-02-10T04:42:01.651 INFO:tasks.rados.rados.0.smithi085.stdout:3567: finishing write tid 2 to smithi08575397-12 2021-02-10T04:42:01.651 INFO:tasks.rados.rados.0.smithi085.stdout:3567: finishing write tid 3 to smithi08575397-12 2021-02-10T04:42:01.651 INFO:tasks.rados.rados.0.smithi085.stdout:3567: finishing write tid 4 to smithi08575397-12 2021-02-10T04:42:01.651 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 1538 (ObjNum 1223 snap 414 seq_num 1223) dirty exists 2021-02-10T04:42:01.651 INFO:tasks.rados.rados.0.smithi085.stdout:3567: left oid 12 (ObjNum 1223 snap 414 seq_num 1223) 2021-02-10T04:42:01.651 INFO:tasks.rados.rados.0.smithi085.stdout:3570: expect (ObjNum 1219 snap 412 seq_num 1219) 2021-02-10T04:42:01.754 INFO:tasks.rados.rados.0.smithi085.stdout:3571: finishing write tid 1 to smithi08575397-22 2021-02-10T04:42:01.754 INFO:tasks.rados.rados.0.smithi085.stdout:3571: finishing write tid 2 to smithi08575397-22 2021-02-10T04:42:01.755 INFO:tasks.rados.rados.0.smithi085.stdout:3571: finishing write tid 3 to smithi08575397-22 2021-02-10T04:42:01.755 INFO:tasks.rados.rados.0.smithi085.stdout:3571: finishing write tid 4 to smithi08575397-22 2021-02-10T04:42:01.755 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 1489 (ObjNum 1224 snap 414 seq_num 1224) dirty exists 2021-02-10T04:42:01.755 INFO:tasks.rados.rados.0.smithi085.stdout:3571: left oid 22 (ObjNum 1224 snap 414 seq_num 1224) 2021-02-10T04:42:01.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:01 smithi164 conmon[33677]: debug 2021-02-10T04:42:01.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:01 smithi164 conmon[33677]: 2021-02-10T04:42:01.445+0000 7fbec19a1700 -1 osd.4 901 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:42:01.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:01 smithi164 conmon[33677]: 2021-02-10T04:42:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:01 smithi164 conmon[29288]: cluster 2021-02-10T04:42:00.416012+0000 osd.4 ( 2021-02-10T04:42:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:01 smithi164 conmon[29288]: osd.4) 1523 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:01 smithi164 conmon[29288]: cluster 2021-02-10T04:42:00.416021+0000 osd.4 (osd.4) 1524 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:01 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:42:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:01 smithi164 conmon[29288]: -02-10T04:42:00.416027+0000 osd.4 (osd.4) 1525 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:01 smithi164 conmon[29288]: cluster 2021-02-10T04:42:00 2021-02-10T04:42:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:01 smithi164 conmon[29288]: .416033+0000 osd.4 (osd.4) 1526 2021-02-10T04:42:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:01 smithi164 conmon[29288]: : cluster [WRN] 2021-02-10T04:42:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:01 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:01 smithi164 conmon[29288]: cluster 2021-02-10T04:42: 2021-02-10T04:42:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:01 smithi164 conmon[29288]: 00.632047+0000 mon.a (mon.0) 1870 2021-02-10T04:42:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:01 smithi164 conmon[29288]: : cluster [DBG] osdmap e901: 8 total, 8 up, 7 in 2021-02-10T04:42:01.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:01 smithi085 conmon[35961]: cluster 2021-02-10T04:42:00.416012+0000 osd.4 (osd.4) 2021-02-10T04:42:01.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:01 smithi085 conmon[35961]: 1523 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:01.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:01 smithi085 conmon[35961]: cluster 2021-02-10T04:42:00 2021-02-10T04:42:01.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:01 smithi085 conmon[35961]: .416021+0000 osd.4 (osd.4) 1524 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:01.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:01 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:42:01.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:01 smithi085 conmon[35961]: 04:42:00.416027+0000 osd.4 (osd.4) 1525 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:01.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:01 smithi085 conmon[35961]: cluster 2021-02-10T04:42:00.416033 2021-02-10T04:42:01.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:01 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1526 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:01.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:01 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:42:01.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:01 smithi085 conmon[35961]: 10T04:42:00.632047+0000 mon.a (mon.0) 1870 2021-02-10T04:42:01.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:01 smithi085 conmon[35961]: : cluster [DBG] osdmap e901: 8 total, 8 up, 7 in 2021-02-10T04:42:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:01 smithi085 conmon[29319]: cluster 2021-02-10T04:42:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:01 smithi085 conmon[29319]: 2021-02-10T04:42:00. 2021-02-10T04:42:01.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:01 smithi085 conmon[29319]: 416012+0000 osd.4 (osd.4) 1523 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:01.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:01 smithi085 conmon[29319]: cluster 2021-02-10T04:42:00.416021+0000 osd.4 ( 2021-02-10T04:42:01.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:01 smithi085 conmon[29319]: osd.4) 1524 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:01.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:01 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:42:01.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:01 smithi085 conmon[29319]: 02-10T04:42:00.416027+0000 osd.4 2021-02-10T04:42:01.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:01 smithi085 conmon[29319]: (osd.4) 1525 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:01.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:01 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:42:01.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:01 smithi085 conmon[29319]: 02-10T04:42:00.416033+0000 osd.4 (osd 2021-02-10T04:42:01.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:01 smithi085 conmon[29319]: .4) 1526 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:01.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:01 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:42:01.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:01 smithi085 conmon[29319]: :42:00.632047+0000 mon.a (mon.0) 1870 : cluster 2021-02-10T04:42:01.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:01 smithi085 conmon[29319]: [DBG] osdmap e901: 8 total, 8 up, 7 in 2021-02-10T04:42:02.263 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:42:02.296 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:02.644 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[44544]: debug 2021-02-10T04:42:02.381+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:02.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[33677]: debug 2021-02-10T04:42:02.403+0000 7fbec19a1700 -1 osd.4 902 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:42:02.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: cluster 2021-02-10T04:42:02.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: 2021-02-10T04:42:02.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: 2021 2021-02-10T04:42:02.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: - 2021-02-10T04:42:02.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: 02 2021-02-10T04:42:02.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: - 2021-02-10T04:42:02.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: 10 2021-02-10T04:42:02.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: T 2021-02-10T04:42:02.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: 04 2021-02-10T04:42:02.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: : 2021-02-10T04:42:02.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: 42 2021-02-10T04:42:02.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: : 2021-02-10T04:42:02.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: 01 2021-02-10T04:42:02.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: . 2021-02-10T04:42:02.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: 446002 2021-02-10T04:42:02.652 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: +0000 2021-02-10T04:42:02.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: 2021-02-10T04:42:02.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: osd.4 2021-02-10T04:42:02.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: ( 2021-02-10T04:42:02.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: osd 2021-02-10T04:42:02.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: . 2021-02-10T04:42:02.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: 4 2021-02-10T04:42:02.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: ) 2021-02-10T04:42:02.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: 1527 2021-02-10T04:42:02.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: : 2021-02-10T04:42:02.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: cluster 2021-02-10T04:42:02.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:02.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: cluster 2021-02-10T04:42:02.653 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: 2021-02 2021-02-10T04:42:02.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: -10T04:42:01.446008+0000 osd.4 (osd.4) 2021-02-10T04:42:02.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: 1528 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:02.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: cluster 2021-02-10T04:42:02.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: 2021-02-10T04:42:01.446011 2021-02-10T04:42:02.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1529 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:02.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: cluster 2021-02-10T04:42:02.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:42:02.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: 42 2021-02-10T04:42:02.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: :01.446015+0000 osd.4 ( 2021-02-10T04:42:02.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: osd.4) 1530 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:02.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: cluster 2021-02-10T04:42:02.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: 2021-02-10T04:42:02.654 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: 2021 2021-02-10T04:42:02.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: - 2021-02-10T04:42:02.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: 02 2021-02-10T04:42:02.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: - 2021-02-10T04:42:02.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: 10 2021-02-10T04:42:02.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: T04:42:01 2021-02-10T04:42:02.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: .633352+0000 mon.a (mon.0) 1871 : cluster [DBG] osdmap e902: 8 total, 8 up, 7 in 2021-02-10T04:42:02.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: 2021-02-10T04:42:02.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: cluster 2021-02-10T04:42:02.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: 2021-02-10T04:42 2021-02-10T04:42:02.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: :02.225773+0000 mgr.y (mgr.14140) 847 2021-02-10T04:42:02.655 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:02 smithi164 conmon[29288]: : cluster [DBG] pgmap v1504: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 703 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 312 KiB/s wr, 13 op/s 2021-02-10T04:42:02.656 INFO:tasks.rados.rados.0.smithi085.stdout:3566: done (8 left) 2021-02-10T04:42:02.656 INFO:tasks.rados.rados.0.smithi085.stdout:3567: done (7 left) 2021-02-10T04:42:02.656 INFO:tasks.rados.rados.0.smithi085.stdout:3570: done (6 left) 2021-02-10T04:42:02.656 INFO:tasks.rados.rados.0.smithi085.stdout:3571: done (5 left) 2021-02-10T04:42:02.656 INFO:tasks.rados.rados.0.smithi085.stdout:3572: done (4 left) 2021-02-10T04:42:02.656 INFO:tasks.rados.rados.0.smithi085.stdout:3573: delete oid 28 current snap is 415 2021-02-10T04:42:02.662 INFO:tasks.rados.rados.0.smithi085.stdout:3573: done (4 left) 2021-02-10T04:42:02.662 INFO:tasks.rados.rados.0.smithi085.stdout:3574: read oid 8 snap -1 2021-02-10T04:42:02.663 INFO:tasks.rados.rados.0.smithi085.stdout:3574: expect (ObjNum 1185 snap 406 seq_num 1185) 2021-02-10T04:42:02.663 INFO:tasks.rados.rados.0.smithi085.stdout:3575: delete oid 4 current snap is 415 2021-02-10T04:42:02.676 INFO:tasks.rados.rados.0.smithi085.stdout:3574: expect (ObjNum 1185 snap 406 seq_num 1185) 2021-02-10T04:42:02.844 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: cluster 2021-02-10T04:42:02.844 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: 2021-02-10T04:42:02.844 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: 2021 2021-02-10T04:42:02.844 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: - 2021-02-10T04:42:02.844 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: 02 2021-02-10T04:42:02.844 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: - 2021-02-10T04:42:02.844 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: 10 2021-02-10T04:42:02.845 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: T 2021-02-10T04:42:02.845 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: 04 2021-02-10T04:42:02.845 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: : 2021-02-10T04:42:02.845 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: 42 2021-02-10T04:42:02.845 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: : 2021-02-10T04:42:02.845 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: 01 2021-02-10T04:42:02.845 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: .446002+0000 osd.4 ( 2021-02-10T04:42:02.845 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: osd.4) 1527 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:02.845 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:42:02.845 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: 04:42:01.446008+0000 osd.4 (osd.4) 2021-02-10T04:42:02.845 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: 1528 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:02.845 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: cluster 2021-02-10T04:42:02.846 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: 2021-02-10T04:42:01.446011+0000 2021-02-10T04:42:02.846 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: osd.4 (osd.4) 1529 : cluster [WRN] 2021-02-10T04:42:02.846 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:02.846 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:42:02.846 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: :42:01.446015+0000 osd.4 (osd. 2021-02-10T04:42:02.846 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: 4) 1530 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:02.846 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: cluster 2021-02-10T04:42:02.846 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: 2021-02-10T04:42:01.633352+0000 2021-02-10T04:42:02.846 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: mon.a (mon.0) 1871 : cluster [DBG] 2021-02-10T04:42:02.846 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: osdmap e902: 8 total, 8 up, 7 in 2021-02-10T04:42:02.846 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:42:02.847 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: :42:02.225773+0000 mgr.y (mgr.14140 2021-02-10T04:42:02.847 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[35961]: ) 847 : cluster [DBG] pgmap v1504: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 703 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 312 KiB/s wr, 13 op/s 2021-02-10T04:42:02.847 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[29319]: cluster 2021-02-10T04:42:01.446002+0000 osd.4 2021-02-10T04:42:02.847 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[29319]: (osd.4) 1527 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:02.847 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[29319]: cluster 2021-02-10T04:42:01.446008+0000 osd.4 (osd.4) 1528 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:02.847 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[29319]: cluster 2021-02-10T04:42:01.446011+0000 osd.4 (osd.4) 1529 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:02.847 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[29319]: cluster 2021-02-10T04:42:01.446015+0000 osd.4 (osd.4) 1530 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:02.847 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[29319]: cluster 2021-02-10T04:42:01.633352 2021-02-10T04:42:02.848 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1871 : cluster [DBG] osdmap e902: 8 total, 8 up, 7 in 2021-02-10T04:42:02.848 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[29319]: cluster 2021-02-10T04:42:02.225773+0000 mgr.y (mgr 2021-02-10T04:42:02.848 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:02 smithi085 conmon[29319]: .14140) 847 : cluster [DBG] pgmap v1504: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 703 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 312 KiB/s wr, 13 op/s 2021-02-10T04:42:02.869 INFO:tasks.rados.rados.0.smithi085.stdout:3574: done (5 left) 2021-02-10T04:42:02.869 INFO:tasks.rados.rados.0.smithi085.stdout:3575: done (4 left) 2021-02-10T04:42:02.869 INFO:tasks.rados.rados.0.smithi085.stdout:3576: rollback oid 49 current snap is 415 2021-02-10T04:42:02.869 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 49 to 410 2021-02-10T04:42:02.869 INFO:tasks.rados.rados.0.smithi085.stdout:3577: copy_from oid 50 from oid 13 current snap is 415 2021-02-10T04:42:02.869 INFO:tasks.rados.rados.0.smithi085.stdout:3578: snap_remove snap 360 2021-02-10T04:42:02.891 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:42:02.996 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:42:03.025 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:03.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:03 smithi164 conmon[33677]: debug 2021-02-10T04:42:03.100+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:03.449 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:42:03.554 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:42:03.585 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:03.657 INFO:tasks.rados.rados.0.smithi085.stdout:3578: done (6 left) 2021-02-10T04:42:03.657 INFO:tasks.rados.rados.0.smithi085.stdout:3579: rollback oid 37 current snap is 415 2021-02-10T04:42:03.658 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 37 to 401 2021-02-10T04:42:03.658 INFO:tasks.rados.rados.0.smithi085.stdout:3576: finishing rollback tid 0 to smithi08575397-49 2021-02-10T04:42:03.658 INFO:tasks.rados.rados.0.smithi085.stdout:3576: finishing rollback tid 1 to smithi08575397-49 2021-02-10T04:42:03.658 INFO:tasks.rados.rados.0.smithi085.stdout:3576: finishing rollback tid 2 to smithi08575397-49 2021-02-10T04:42:03.658 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 49 v 1590 (ObjNum 1097 snap 368 seq_num 1097) dirty exists 2021-02-10T04:42:03.658 INFO:tasks.rados.rados.0.smithi085.stdout:3577: finishing copy_from to smithi08575397-50 2021-02-10T04:42:03.658 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 1553 (ObjNum 1178 snap 402 seq_num 1178) dirty exists 2021-02-10T04:42:03.658 INFO:tasks.rados.rados.0.smithi085.stdout:3577: finishing copy_from racing read to smithi08575397-50 2021-02-10T04:42:03.658 INFO:tasks.rados.rados.0.smithi085.stdout:3576: done (6 left) 2021-02-10T04:42:03.658 INFO:tasks.rados.rados.0.smithi085.stdout:3577: done (5 left) 2021-02-10T04:42:03.658 INFO:tasks.rados.rados.0.smithi085.stdout:3580: copy_from oid 43 from oid 44 current snap is 415 2021-02-10T04:42:03.659 INFO:tasks.rados.rados.0.smithi085.stdout:3581: write (excl) oid 32 current snap is 415 2021-02-10T04:42:03.659 INFO:tasks.rados.rados.0.smithi085.stdout:3581: seq_num 1227 ranges {622189=663356,1984347=458752} 2021-02-10T04:42:03.665 INFO:tasks.rados.rados.0.smithi085.stdout:3581: writing smithi08575397-32 from 622189 to 1285545 tid 1 2021-02-10T04:42:03.672 INFO:tasks.rados.rados.0.smithi085.stdout:3581: writing smithi08575397-32 from 1984347 to 2443099 tid 2 2021-02-10T04:42:03.672 INFO:tasks.rados.rados.0.smithi085.stdout:3582: rollback oid 48 current snap is 415 2021-02-10T04:42:03.673 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 48 to 266 2021-02-10T04:42:03.673 INFO:tasks.rados.rados.0.smithi085.stdout:3580: finishing copy_from to smithi08575397-43 2021-02-10T04:42:03.673 INFO:tasks.rados.rados.0.smithi085.stdout:3580: got expected ENOENT (src dne) 2021-02-10T04:42:03.673 INFO:tasks.rados.rados.0.smithi085.stdout:3580: finishing copy_from racing read to smithi08575397-43 2021-02-10T04:42:03.673 INFO:tasks.rados.rados.0.smithi085.stdout:3580: done (7 left) 2021-02-10T04:42:03.673 INFO:tasks.rados.rados.0.smithi085.stdout:3583: write (excl) oid 1 current snap is 415 2021-02-10T04:42:03.673 INFO:tasks.rados.rados.0.smithi085.stdout:3583: seq_num 1228 ranges {638996=738605,1926476=712520} 2021-02-10T04:42:03.678 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:03 smithi164 conmon[29288]: cluster 2021-02-10T04:42:02.404132+0000 osd.4 (osd.4 2021-02-10T04:42:03.678 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:03 smithi164 conmon[29288]: ) 1531 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:03.678 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:03 smithi164 conmon[29288]: cluster 2021-02-10T04:42:02.404141+0000 osd.4 (osd.4) 1532 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:03.678 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:03 smithi164 conmon[29288]: cluster 2021-02-10T04:42:02.404145+0000 osd.4 (osd.4) 1533 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:03.678 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:03 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:42:03.678 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:03 smithi164 conmon[29288]: 42:02.404148+0000 osd.4 (osd.4) 1534 2021-02-10T04:42:03.679 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:03 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:03.679 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:03 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:42:03.679 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:03 smithi164 conmon[29288]: 42:02.648680+0000 mon.a (mon.0) 2021-02-10T04:42:03.679 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:03 smithi164 conmon[29288]: 1872 : cluster [DBG] osdmap e903: 8 total, 8 up, 7 in 2021-02-10T04:42:03.679 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:03 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:42:03.679 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:03 smithi164 conmon[29288]: -02-10T04:42:03.293623+0000 mon.a (mon.0) 1873 : cluster 2021-02-10T04:42:03.679 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:03 smithi164 conmon[29288]: [WRN] Health check update: 4 slow ops, oldest one blocked for 752 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:42:03.679 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:03 smithi164 conmon[33677]: debug 2021-02-10T04:42:03.405+0000 7fbec19a1700 -1 osd.4 903 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:42:03.683 INFO:tasks.rados.rados.0.smithi085.stdout:3583: writing smithi08575397-1 from 638996 to 1377601 tid 1 2021-02-10T04:42:03.692 INFO:tasks.rados.rados.0.smithi085.stdout:3583: writing smithi08575397-1 from 1926476 to 2638996 tid 2 2021-02-10T04:42:03.693 INFO:tasks.rados.rados.0.smithi085.stdout:3584: snap_remove snap 411 2021-02-10T04:42:03.927 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:03 smithi085 conmon[29319]: cluster 2021-02-10T04:42:02.404132+0000 osd.4 (osd.4 2021-02-10T04:42:03.927 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:03 smithi085 conmon[29319]: ) 1531 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:03.928 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:03 smithi085 conmon[29319]: cluster 2021-02-10T04:42:02.404141+0000 osd.4 (osd.4) 1532 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:03.928 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:03 smithi085 conmon[29319]: cluster 2021-02-10T04:42:02.404145+0000 osd.4 (osd.4) 1533 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:03.928 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:03 smithi085 conmon[29319]: cluster 2021-02-10T04:42:02.404148+0000 osd.4 (osd.4) 1534 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:03.928 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:03 smithi085 conmon[29319]: cluster 2021-02-10T04:42:02.648680+0000 mon.a (mon.0) 1872 : cluster [DBG] osdmap e903: 8 total, 8 up, 7 in 2021-02-10T04:42:03.928 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:03 smithi085 conmon[29319]: cluster 2021-02-10T04:42:03.293623+0000 mon.a (mon.0) 1873 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 752 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:42:03.928 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:03 smithi085 conmon[35961]: cluster 2021-02-10T04:42:02.404132+0000 osd.4 2021-02-10T04:42:03.928 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:03 smithi085 conmon[35961]: (osd.4) 1531 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:03.928 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:03 smithi085 conmon[35961]: cluster 2021-02-10T04:42:02.404141+0000 osd.4 (osd.4) 1532 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:03.928 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:03 smithi085 conmon[35961]: cluster 2021-02-10T04:42:02.404145+0000 osd.4 (osd.4) 1533 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:03.928 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:03 smithi085 conmon[35961]: cluster 2021-02-10T04:42:02.404148+0000 osd.4 (osd.4) 1534 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:03.929 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:03 smithi085 conmon[35961]: cluster 2021-02-10T04:42:02.648680+0000 mon.a (mon.0) 1872 : cluster [DBG] osdmap e903: 8 total, 8 up, 7 in 2021-02-10T04:42:03.929 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:03 smithi085 conmon[35961]: cluster 2021-02-10T04:42:03.293623+0000 mon.a (mon.0) 1873 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 752 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:42:03.975 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:42:03.978 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:42:03 smithi164 conmon[40892]: debug 2021-02-10T04:42:03.676+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:04.080 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:42:04.119 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:04.667 INFO:tasks.rados.rados.0.smithi085.stdout:3584: done (8 left) 2021-02-10T04:42:04.667 INFO:tasks.rados.rados.0.smithi085.stdout:3585: rollback oid 36 current snap is 415 2021-02-10T04:42:04.667 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 36 to 354 2021-02-10T04:42:04.667 INFO:tasks.rados.rados.0.smithi085.stdout:3586: read oid 46 snap -1 2021-02-10T04:42:04.667 INFO:tasks.rados.rados.0.smithi085.stdout:3586: expect deleted 2021-02-10T04:42:04.667 INFO:tasks.rados.rados.0.smithi085.stdout:3587: snap_create 2021-02-10T04:42:04.668 INFO:tasks.rados.rados.0.smithi085.stdout:3581: finishing write tid 1 to smithi08575397-32 2021-02-10T04:42:04.668 INFO:tasks.rados.rados.0.smithi085.stdout:3581: finishing write tid 2 to smithi08575397-32 2021-02-10T04:42:04.668 INFO:tasks.rados.rados.0.smithi085.stdout:3581: finishing write tid 3 to smithi08575397-32 2021-02-10T04:42:04.668 INFO:tasks.rados.rados.0.smithi085.stdout:3581: finishing write tid 4 to smithi08575397-32 2021-02-10T04:42:04.668 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 32 v 1488 (ObjNum 1227 snap 415 seq_num 1227) dirty exists 2021-02-10T04:42:04.668 INFO:tasks.rados.rados.0.smithi085.stdout:3581: left oid 32 (ObjNum 1227 snap 415 seq_num 1227) 2021-02-10T04:42:04.668 INFO:tasks.rados.rados.0.smithi085.stdout:3582: finishing rollback tid 0 to smithi08575397-48 2021-02-10T04:42:04.668 INFO:tasks.rados.rados.0.smithi085.stdout:3582: finishing rollback tid 1 to smithi08575397-48 2021-02-10T04:42:04.668 INFO:tasks.rados.rados.0.smithi085.stdout:3582: finishing rollback tid 2 to smithi08575397-48 2021-02-10T04:42:04.668 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 1541 (ObjNum 821 snap 266 seq_num 821) dirty exists 2021-02-10T04:42:04.668 INFO:tasks.rados.rados.0.smithi085.stdout:3579: finishing rollback tid 0 to smithi08575397-37 2021-02-10T04:42:04.668 INFO:tasks.rados.rados.0.smithi085.stdout:3579: finishing rollback tid 1 to smithi08575397-37 2021-02-10T04:42:04.668 INFO:tasks.rados.rados.0.smithi085.stdout:3579: finishing rollback tid 2 to smithi08575397-37 2021-02-10T04:42:04.668 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 37 v 1851 (ObjNum 1129 snap 381 seq_num 1129) dirty exists 2021-02-10T04:42:04.669 INFO:tasks.rados.rados.0.smithi085.stdout:3583: finishing write tid 1 to smithi08575397-1 2021-02-10T04:42:04.669 INFO:tasks.rados.rados.0.smithi085.stdout:3583: finishing write tid 2 to smithi08575397-1 2021-02-10T04:42:04.669 INFO:tasks.rados.rados.0.smithi085.stdout:3583: finishing write tid 3 to smithi08575397-1 2021-02-10T04:42:04.669 INFO:tasks.rados.rados.0.smithi085.stdout:3583: finishing write tid 4 to smithi08575397-1 2021-02-10T04:42:04.669 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 1592 (ObjNum 1228 snap 415 seq_num 1228) dirty exists 2021-02-10T04:42:04.669 INFO:tasks.rados.rados.0.smithi085.stdout:3583: left oid 1 (ObjNum 1228 snap 415 seq_num 1228) 2021-02-10T04:42:04.676 INFO:tasks.rados.rados.0.smithi085.stdout:3585: finishing rollback tid 0 to smithi08575397-36 2021-02-10T04:42:04.677 INFO:tasks.rados.rados.0.smithi085.stdout:3585: finishing rollback tid 1 to smithi08575397-36 2021-02-10T04:42:04.678 INFO:tasks.rados.rados.0.smithi085.stdout:3585: finishing rollback tid 2 to smithi08575397-36 2021-02-10T04:42:04.679 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 1359 (ObjNum 904 snap 297 seq_num 904) dirty exists 2021-02-10T04:42:04.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:04 smithi164 conmon[33677]: debug 2021-02-10T04:42:04.421+0000 7fbec19a1700 -1 osd.4 904 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:42:04.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:04 smithi164 conmon[29288]: cluster 2021-02-10T04:42:03.406368+0000 osd.4 (osd. 2021-02-10T04:42:04.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:04 smithi164 conmon[29288]: 4) 1535 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:04.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:04 smithi164 conmon[29288]: cluster 2021-02-10T04:42:03.406377+0000 osd.4 (osd.4) 1536 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:04 smithi164 conmon[29288]: cluster 2021-02-10T04:42:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:04 smithi164 conmon[29288]: 2021-02-10T04:42:03.406384+0000 2021-02-10T04:42:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:04 smithi164 conmon[29288]: osd.4 (osd.4) 1537 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:04 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:42:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:04 smithi164 conmon[29288]: -10T04:42:03.406390+0000 osd.4 (osd. 2021-02-10T04:42:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:04 smithi164 conmon[29288]: 4) 1538 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:04 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:42:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:04 smithi164 conmon[29288]: 02-10T04:42:03.657668+0000 2021-02-10T04:42:04.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:04 smithi164 conmon[29288]: mon.a (mon.0) 1874 : cluster [DBG] osdmap e904: 8 total, 8 up, 7 in 2021-02-10T04:42:04.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[29319]: cluster 2021-02-10T04:42:04.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[29319]: 2021-02-10T04:42:04.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[29319]: 2021 2021-02-10T04:42:04.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[29319]: -02-10T04:42:03. 2021-02-10T04:42:04.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[29319]: 406368+0000 osd.4 (osd.4) 1535 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:04.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[29319]: cluster 2021-02-10T04:42:03.406377+0000 2021-02-10T04:42:04.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[29319]: osd.4 (osd.4) 1536 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:04.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[29319]: cluster 2021-02-10T04:42: 2021-02-10T04:42:04.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[29319]: 03.406384+0000 osd.4 (osd.4) 1537 : cluster [WRN] 2021-02-10T04:42:04.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:04.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[29319]: cluster 2021-02-10T04:42 2021-02-10T04:42:04.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[29319]: :03.406390+0000 osd.4 (osd.4) 1538 : 2021-02-10T04:42:04.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:04.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:42:04.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[29319]: 02-10T04:42:03.657668+0000 mon.a (mon.0) 2021-02-10T04:42:04.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[29319]: 1874 : cluster [DBG] osdmap e904: 8 total, 8 up, 7 in 2021-02-10T04:42:04.943 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[45443]: debug 2021-02-10T04:42:04.661+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:04.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[35961]: cluster 2021-02-10T04:42:04.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[35961]: 2021-02-10T04:42:04.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[35961]: 2021-02-10T04:42:03. 2021-02-10T04:42:04.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[35961]: 406368+0000 osd.4 (osd.4) 1535 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:04.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:42:04.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[35961]: 04:42:03.406377+0000 osd.4 (osd.4) 2021-02-10T04:42:04.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[35961]: 1536 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:04.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:42:04.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[35961]: 04:42:03.406384+0000 2021-02-10T04:42:04.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[35961]: osd.4 (osd.4) 1537 : 2021-02-10T04:42:04.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:04.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:42:04.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[35961]: 04:42:03.406390+0000 2021-02-10T04:42:04.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[35961]: osd.4 (osd.4) 1538 : 2021-02-10T04:42:04.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:04.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[35961]: cluster 2021-02-10T04:42:04.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[35961]: 2021-02-10T04:42: 2021-02-10T04:42:04.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[35961]: 03.657668+0000 mon.a (mon 2021-02-10T04:42:04.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:04 smithi085 conmon[35961]: .0) 1874 : cluster [DBG] osdmap e904: 8 total, 8 up, 7 in 2021-02-10T04:42:05.013 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:42:05.664 INFO:tasks.rados.rados.0.smithi085.stdout:3579: done (10 left) 2021-02-10T04:42:05.664 INFO:tasks.rados.rados.0.smithi085.stdout:3581: done (9 left) 2021-02-10T04:42:05.665 INFO:tasks.rados.rados.0.smithi085.stdout:3582: done (8 left) 2021-02-10T04:42:05.665 INFO:tasks.rados.rados.0.smithi085.stdout:3583: done (7 left) 2021-02-10T04:42:05.665 INFO:tasks.rados.rados.0.smithi085.stdout:3585: done (6 left) 2021-02-10T04:42:05.665 INFO:tasks.rados.rados.0.smithi085.stdout:3586: done (5 left) 2021-02-10T04:42:05.665 INFO:tasks.rados.rados.0.smithi085.stdout:3587: done (4 left) 2021-02-10T04:42:05.665 INFO:tasks.rados.rados.0.smithi085.stdout:3588: rollback oid 7 current snap is 416 2021-02-10T04:42:05.666 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 7 to 347 2021-02-10T04:42:05.666 INFO:tasks.rados.rados.0.smithi085.stdout:3589: copy_from oid 50 from oid 26 current snap is 416 2021-02-10T04:42:05.666 INFO:tasks.rados.rados.0.smithi085.stdout:3590: snap_remove snap 344 2021-02-10T04:42:05.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:05 smithi164 conmon[33677]: debug 2021-02-10T04:42:05.458+0000 7fbec19a1700 -1 osd.4 905 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:42:05.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:05 smithi164 conmon[29288]: cluster 2021-02-10T04:42:04.226436 2021-02-10T04:42:05.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:05 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 848 : cluster [DBG] pgmap v1507: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 707 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 4.0 MiB/s rd, 1.8 MiB/s wr, 10 op/s 2021-02-10T04:42:05.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:05 smithi164 conmon[29288]: cluster 2021-02-10T04:42:04.422363+0000 2021-02-10T04:42:05.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:05 smithi164 conmon[29288]: osd.4 (osd.4) 1539 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:05.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:05 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:42:05.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:05 smithi164 conmon[29288]: 02-10T04:42:04.422372+0000 osd.4 2021-02-10T04:42:05.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:05 smithi164 conmon[29288]: (osd.4) 1540 : cluster [WRN] 2021-02-10T04:42:05.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:05 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:05.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:05 smithi164 conmon[29288]: cluster 2021-02-10T04:42:04 2021-02-10T04:42:05.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:05 smithi164 conmon[29288]: .422378+0000 osd.4 (osd.4) 1541 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:05.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:05 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:42:05.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:05 smithi164 conmon[29288]: 04:42:04.422384+0000 osd.4 (osd.4 2021-02-10T04:42:05.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:05 smithi164 conmon[29288]: ) 1542 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:05.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:05 smithi164 conmon[29288]: cluster 2021-02-10T04:42: 2021-02-10T04:42:05.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:05 smithi164 conmon[29288]: 04.665349+0000 mon.a (mon.0) 1875 2021-02-10T04:42:05.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:05 smithi164 conmon[29288]: : cluster [DBG] osdmap e905: 8 total, 8 up, 7 in 2021-02-10T04:42:05.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[29319]: cluster 2021-02-10T04:42:04.226436+0000 mgr.y ( 2021-02-10T04:42:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[29319]: mgr.14140) 848 : cluster [DBG] pgmap v1507: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 707 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 4.0 MiB/s rd, 1.8 MiB/s wr, 10 op/s 2021-02-10T04:42:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[29319]: cluster 2021-02-10T04:42:04.422363+0000 osd.4 (osd.4) 1539 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:42:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[29319]: T04:42:04.422372+0000 osd.4 (osd.4) 1540 : cluster [WRN] 2021-02-10T04:42:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[29319]: cluster 2021-02-10T04:42:04.422378+0000 2021-02-10T04:42:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[29319]: osd.4 (osd.4) 1541 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[29319]: cluster 2021-02-10T04:42:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[29319]: 2021-02-10T04:42 2021-02-10T04:42:05.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[29319]: :04.422384+0000 osd.4 (osd.4) 1542 2021-02-10T04:42:05.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:05.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[29319]: cluster 2021-02-10T04:42:05.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[29319]: 2021-02-10T04:42 2021-02-10T04:42:05.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[29319]: :04.665349+0000 mon.a (mon.0) 1875 2021-02-10T04:42:05.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[29319]: : cluster [DBG] osdmap e905: 8 total, 8 up, 7 in 2021-02-10T04:42:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[35961]: cluster 2021-02-10T04:42:04.226436 2021-02-10T04:42:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 848 : cluster [DBG] pgmap v1507: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 707 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 4.0 MiB/s rd, 1.8 MiB/s wr, 10 op/s 2021-02-10T04:42:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:42:05.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[35961]: 42:04.422363+0000 osd.4 (osd.4) 1539 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:05.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:42:05.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[35961]: 02-10T04:42:04.422372+0000 osd.4 2021-02-10T04:42:05.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[35961]: (osd.4) 1540 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:05.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:42:05.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[35961]: 42:04.422378+0000 osd.4 (osd.4) 1541 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:05.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[35961]: cluster 2021-02-10T04:42:05.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[35961]: 2021-02-10T04:42:05.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[35961]: 2021 2021-02-10T04:42:05.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[35961]: - 2021-02-10T04:42:05.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[35961]: 02-10T04:42:04.422384+0000 2021-02-10T04:42:05.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[35961]: osd.4 (osd.4) 1542 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:05.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[35961]: cluster 2021-02-10T04:42 2021-02-10T04:42:05.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[35961]: :04.665349+0000 mon.a (mon.0) 1875 : cluster 2021-02-10T04:42:05.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:05 smithi085 conmon[35961]: [DBG] osdmap e905: 8 total, 8 up, 7 in 2021-02-10T04:42:06.666 INFO:tasks.rados.rados.0.smithi085.stdout:3590: done (6 left) 2021-02-10T04:42:06.667 INFO:tasks.rados.rados.0.smithi085.stdout:3591: rollback oid 21 current snap is 416 2021-02-10T04:42:06.667 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 21 to 391 2021-02-10T04:42:06.667 INFO:tasks.rados.rados.0.smithi085.stdout:3588: finishing rollback tid 0 to smithi08575397-7 2021-02-10T04:42:06.667 INFO:tasks.rados.rados.0.smithi085.stdout:3588: finishing rollback tid 1 to smithi08575397-7 2021-02-10T04:42:06.667 INFO:tasks.rados.rados.0.smithi085.stdout:3588: finishing rollback tid 2 to smithi08575397-7 2021-02-10T04:42:06.667 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 7 v 1600 (ObjNum 1045 snap 341 seq_num 1045) dirty exists 2021-02-10T04:42:06.667 INFO:tasks.rados.rados.0.smithi085.stdout:3589: finishing copy_from to smithi08575397-50 2021-02-10T04:42:06.667 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 1370 (ObjNum 1207 snap 410 seq_num 1207) dirty exists 2021-02-10T04:42:06.667 INFO:tasks.rados.rados.0.smithi085.stdout:3589: finishing copy_from racing read to smithi08575397-50 2021-02-10T04:42:06.667 INFO:tasks.rados.rados.0.smithi085.stdout:3588: done (6 left) 2021-02-10T04:42:06.667 INFO:tasks.rados.rados.0.smithi085.stdout:3589: done (5 left) 2021-02-10T04:42:06.667 INFO:tasks.rados.rados.0.smithi085.stdout:3592: delete oid 29 current snap is 416 2021-02-10T04:42:06.672 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:06 smithi164 conmon[33677]: debug 2021-02-10T04:42:06.414+0000 7fbec19a1700 -1 osd.4 906 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:42:06.682 INFO:tasks.rados.rados.0.smithi085.stdout:3592: done (5 left) 2021-02-10T04:42:06.682 INFO:tasks.rados.rados.0.smithi085.stdout:3593: write oid 1 current snap is 416 2021-02-10T04:42:06.682 INFO:tasks.rados.rados.0.smithi085.stdout:3593: seq_num 1230 ranges {672610=489103,1643440=722432} 2021-02-10T04:42:06.684 INFO:tasks.rados.rados.0.smithi085.stdout:3593: writing smithi08575397-1 from 672610 to 1161713 tid 1 2021-02-10T04:42:06.693 INFO:tasks.rados.rados.0.smithi085.stdout:3593: writing smithi08575397-1 from 1643440 to 2365872 tid 2 2021-02-10T04:42:06.694 INFO:tasks.rados.rados.0.smithi085.stdout:3594: write oid 5 current snap is 416 2021-02-10T04:42:06.694 INFO:tasks.rados.rados.0.smithi085.stdout:3591: finishing rollback tid 0 to smithi08575397-21 2021-02-10T04:42:06.694 INFO:tasks.rados.rados.0.smithi085.stdout:3594: seq_num 1231 ranges {689417=564352,1701922=466090} 2021-02-10T04:42:06.702 INFO:tasks.rados.rados.0.smithi085.stdout:3594: writing smithi08575397-5 from 689417 to 1253769 tid 1 2021-02-10T04:42:06.708 INFO:tasks.rados.rados.0.smithi085.stdout:3594: writing smithi08575397-5 from 1701922 to 2168012 tid 2 2021-02-10T04:42:06.709 INFO:tasks.rados.rados.0.smithi085.stdout:3595: read oid 50 snap 354 2021-02-10T04:42:06.709 INFO:tasks.rados.rados.0.smithi085.stdout:3595: expect (ObjNum 1054 snap 346 seq_num 1054) 2021-02-10T04:42:06.709 INFO:tasks.rados.rados.0.smithi085.stdout:3591: finishing rollback tid 1 to smithi08575397-21 2021-02-10T04:42:06.709 INFO:tasks.rados.rados.0.smithi085.stdout:3591: finishing rollback tid 2 to smithi08575397-21 2021-02-10T04:42:06.709 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 1855 (ObjNum 1134 snap 381 seq_num 1134) dirty exists 2021-02-10T04:42:06.709 INFO:tasks.rados.rados.0.smithi085.stdout:3593: finishing write tid 1 to smithi08575397-1 2021-02-10T04:42:06.709 INFO:tasks.rados.rados.0.smithi085.stdout:3593: finishing write tid 2 to smithi08575397-1 2021-02-10T04:42:06.709 INFO:tasks.rados.rados.0.smithi085.stdout:3593: finishing write tid 3 to smithi08575397-1 2021-02-10T04:42:06.709 INFO:tasks.rados.rados.0.smithi085.stdout:3593: finishing write tid 4 to smithi08575397-1 2021-02-10T04:42:06.709 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 1595 (ObjNum 1230 snap 416 seq_num 1230) dirty exists 2021-02-10T04:42:06.709 INFO:tasks.rados.rados.0.smithi085.stdout:3593: left oid 1 (ObjNum 1230 snap 416 seq_num 1230) 2021-02-10T04:42:06.709 INFO:tasks.rados.rados.0.smithi085.stdout:3596: read oid 42 snap -1 2021-02-10T04:42:06.710 INFO:tasks.rados.rados.0.smithi085.stdout:3596: expect (ObjNum 1124 snap 380 seq_num 1124) 2021-02-10T04:42:06.710 INFO:tasks.rados.rados.0.smithi085.stdout:3591: done (8 left) 2021-02-10T04:42:06.710 INFO:tasks.rados.rados.0.smithi085.stdout:3593: done (7 left) 2021-02-10T04:42:06.710 INFO:tasks.rados.rados.0.smithi085.stdout:3597: snap_remove snap 412 2021-02-10T04:42:06.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[35961]: cluster 2021-02-10T04:42:05.459245+0000 osd.4 2021-02-10T04:42:06.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[35961]: (osd.4) 1543 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:06.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[35961]: cluster 2021-02-10T04:42 2021-02-10T04:42:06.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[35961]: :05.459256+0000 osd.4 (osd.4) 1544 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:06.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[35961]: cluster 2021-02-10T04:42:05.459262+0000 osd.4 (osd.4 2021-02-10T04:42:06.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[35961]: ) 1545 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:06.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[35961]: cluster 2021-02-10T04:42:05 2021-02-10T04:42:06.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[35961]: .459268+0000 osd.4 (osd.4) 1546 : cluster 2021-02-10T04:42:06.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:06.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[35961]: cluster 2021-02-10T04:42:06.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[35961]: 2021-02-10T04:42:05.663140+0000 mon.a (mon 2021-02-10T04:42:06.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[35961]: .0) 1876 : cluster [DBG] osdmap e906: 8 total, 8 up, 7 in 2021-02-10T04:42:06.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:42:06.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[35961]: 04:42:06.665029+0000 mon.a (mon.0) 2021-02-10T04:42:06.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[35961]: 1877 : cluster [DBG] osdmap e907: 8 total, 8 up, 7 in 2021-02-10T04:42:06.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: cluster 2021-02-10T04:42:06.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: 2021-02-10T04:42:06.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: 2021 2021-02-10T04:42:06.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: - 2021-02-10T04:42:06.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: 02 2021-02-10T04:42:06.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: - 2021-02-10T04:42:06.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: 10 2021-02-10T04:42:06.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: T 2021-02-10T04:42:06.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: 04 2021-02-10T04:42:06.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: : 2021-02-10T04:42:06.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: 42:05.459245+0000 osd.4 2021-02-10T04:42:06.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: (osd.4) 1543 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:06.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:42:06.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: 04:42:05.459256+0000 osd.4 (osd.4 2021-02-10T04:42:06.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: ) 1544 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:06.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: cluster 2021-02-10T04:42:06.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: 2021-02-10T04:42:06.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: 2021 2021-02-10T04:42:06.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: - 2021-02-10T04:42:06.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: 02 2021-02-10T04:42:06.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: - 2021-02-10T04:42:06.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: 10 2021-02-10T04:42:06.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: T 2021-02-10T04:42:06.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: 04 2021-02-10T04:42:06.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: : 2021-02-10T04:42:06.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: 42 2021-02-10T04:42:06.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: : 2021-02-10T04:42:06.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: 05 2021-02-10T04:42:06.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: . 2021-02-10T04:42:06.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: 459262 2021-02-10T04:42:06.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: +0000 2021-02-10T04:42:06.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: 2021-02-10T04:42:06.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: osd.4 2021-02-10T04:42:06.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: ( 2021-02-10T04:42:06.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: osd 2021-02-10T04:42:06.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: . 2021-02-10T04:42:06.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: 4 2021-02-10T04:42:06.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: ) 2021-02-10T04:42:06.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: 1545 2021-02-10T04:42:06.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: : 2021-02-10T04:42:06.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: cluster 2021-02-10T04:42:06.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: 2021-02-10T04:42:06.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: [WRN] 2021-02-10T04:42:06.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: 2021-02-10T04:42:06.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:06.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: 2021-02-10T04:42:06.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: cluster 2021-02-10T04:42:06.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: 2021-02-10T04:42:06.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: 2021 2021-02-10T04:42:06.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: - 2021-02-10T04:42:06.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: 02 2021-02-10T04:42:06.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: - 2021-02-10T04:42:06.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: 10 2021-02-10T04:42:06.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: T 2021-02-10T04:42:06.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: 04:42:05.459268+0000 osd.4 (osd 2021-02-10T04:42:06.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: .4) 1546 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:06.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:42:06.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: 02-10T04:42:05.663140+0000 mon.a (mon.0) 1876 : cluster [DBG] 2021-02-10T04:42:06.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: osdmap e906: 8 total, 8 up, 7 in 2021-02-10T04:42:06.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: cluster 2021-02-10T04:42:06.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: 2021-02-10T04:42:06.665029+0000 mon.a (mon. 2021-02-10T04:42:06.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:06 smithi085 conmon[29319]: 0) 1877 : cluster [DBG] osdmap e907: 8 total, 8 up, 7 in 2021-02-10T04:42:07.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:06 smithi164 conmon[29288]: cluster 2021-02-10T04:42:05. 2021-02-10T04:42:07.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:06 smithi164 conmon[29288]: 459245+0000 osd.4 (osd.4) 1543 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:07.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:06 smithi164 conmon[29288]: cluster 2021-02-10T04:42:05.459256+0000 osd.4 (osd.4) 1544 : 2021-02-10T04:42:07.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:06 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:07.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:06 smithi164 conmon[29288]: cluster 2021-02-10T04:42:05.459262 2021-02-10T04:42:07.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:06 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1545 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:07.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:06 smithi164 conmon[29288]: cluster 2021-02-10T04:42:07.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:06 smithi164 conmon[29288]: 2021-02-10T04:42:05.459268+0000 osd.4 (osd.4) 1546 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:07.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:06 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:42:07.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:06 smithi164 conmon[29288]: -10T04:42:05.663140+0000 mon.a 2021-02-10T04:42:07.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:06 smithi164 conmon[29288]: (mon.0) 1876 : cluster [DBG] osdmap e906: 8 total, 8 up, 7 in 2021-02-10T04:42:07.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:06 smithi164 conmon[29288]: 2021-02-10T04:42:07.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:06 smithi164 conmon[29288]: cluster 2021-02-10T04:42 2021-02-10T04:42:07.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:06 smithi164 conmon[29288]: :06.665029+0000 mon.a (mon 2021-02-10T04:42:07.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:06 smithi164 conmon[29288]: .0) 1877 : cluster [DBG] osdmap e907: 8 total, 8 up, 7 in 2021-02-10T04:42:07.508 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:42:07.509 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [], 2021-02-10T04:42:07.509 INFO:teuthology.orchestra.run.smithi085.stdout: "num_ops": 0 2021-02-10T04:42:07.509 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:42:07.687 INFO:tasks.rados.rados.0.smithi085.stdout:3597: done (7 left) 2021-02-10T04:42:07.687 INFO:tasks.rados.rados.0.smithi085.stdout:3598: write (excl) oid 12 current snap is 416 2021-02-10T04:42:07.687 INFO:tasks.rados.rados.0.smithi085.stdout:3594: finishing write tid 1 to smithi08575397-5 2021-02-10T04:42:07.688 INFO:tasks.rados.rados.0.smithi085.stdout:3594: finishing write tid 2 to smithi08575397-5 2021-02-10T04:42:07.688 INFO:tasks.rados.rados.0.smithi085.stdout:3594: finishing write tid 3 to smithi08575397-5 2021-02-10T04:42:07.688 INFO:tasks.rados.rados.0.smithi085.stdout:3595: expect (ObjNum 1054 snap 346 seq_num 1054) 2021-02-10T04:42:07.689 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.6 to 1 (10000) 2021-02-10T04:42:07.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:07 smithi164 conmon[33677]: debug 2021-02-10T04:42:07.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:07 smithi164 conmon[33677]: 2021-02-10T04:42:07.454+0000 7fbec19a1700 -1 osd.4 907 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:42:07.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:07 smithi164 conmon[33677]: 2021-02-10T04:42:07.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:07 smithi164 conmon[29288]: cluster 2021-02-10T04:42:06.226915+0000 mgr.y (mgr. 2021-02-10T04:42:07.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:07 smithi164 conmon[29288]: 14140) 849 : cluster [DBG] pgmap v1510: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 706 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 4.0 MiB/s rd, 1.8 MiB/s wr, 11 op/s 2021-02-10T04:42:07.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:07 smithi164 conmon[29288]: cluster 2021-02-10T04:42:06.414797+0000 osd.4 (osd.4) 1547 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:07.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:07 smithi164 conmon[29288]: cluster 2021-02-10T04:42:06.414805+0000 osd.4 (osd.4) 1548 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:07.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:07 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:42:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:07 smithi164 conmon[29288]: 02-10T04:42:06.414812+0000 osd.4 ( 2021-02-10T04:42:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:07 smithi164 conmon[29288]: osd.4) 1549 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:07 smithi164 conmon[29288]: cluster 2021-02-10T04:42:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:07 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:42:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:07 smithi164 conmon[29288]: 04:42:06.414817+0000 osd.4 (osd.4) 1550 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:07 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:42:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:07 smithi164 conmon[29288]: 02-10T04:42:06.512874+0000 osd.5 2021-02-10T04:42:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:07 smithi164 conmon[29288]: (osd.5) 84 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:42:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:07 smithi164 conmon[29288]: cluster 2021-02-10T04:42:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:07 smithi164 conmon[29288]: 2021-02-10T04:42:06.513758+0000 2021-02-10T04:42:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:07 smithi164 conmon[29288]: osd.5 (osd.5) 85 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:42:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:07 smithi164 conmon[29288]: 2021-02-10T04:42:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:07 smithi164 conmon[29288]: audit 2021-02-10T04:42:07 2021-02-10T04:42:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:07 smithi164 conmon[29288]: .122838+0000 mon.a (mon.0) 1878 : audit 2021-02-10T04:42:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:07 smithi164 conmon[29288]: [INF] from='client.? 172.21.15.85:0/919248963' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T04:42:07.933 INFO:tasks.rados.rados.0.smithi085.stdout:3598: seq_num 1232 ranges {706224=755954,2160404=545820} 2021-02-10T04:42:07.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[35961]: cluster 2021-02-10T04:42:06. 2021-02-10T04:42:07.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[35961]: 226915+0000 mgr.y (mgr.14140) 849 : cluster [DBG] pgmap v1510: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 706 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 4.0 MiB/s rd, 1.8 MiB/s wr, 11 op/s 2021-02-10T04:42:07.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[35961]: cluster 2021-02-10T04:42:07.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[35961]: 2021-02-10T04:42:06.414797+0000 osd.4 (osd.4) 1547 : cluster 2021-02-10T04:42:07.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:07.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:42:07.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[35961]: 42:06.414805+0000 osd.4 (osd 2021-02-10T04:42:07.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[35961]: .4) 1548 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:07.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:42:07.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[35961]: 02-10T04:42:06.414812+0000 2021-02-10T04:42:07.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[35961]: osd.4 (osd.4) 1549 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:07.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[35961]: 2021-02-10T04:42:07.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[35961]: cluster 2021-02-10T04:42:07.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[35961]: 2021-02-10T04:42:06.414817+0000 osd.4 2021-02-10T04:42:07.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[35961]: (osd.4) 1550 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:42:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[35961]: -10T04:42: 2021-02-10T04:42:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[35961]: 06.512874+0000 osd.5 (osd.5) 84 : 2021-02-10T04:42:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[35961]: cluster [DBG] purged_snaps scrub starts 2021-02-10T04:42:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:42:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[35961]: 04:42:06.513758+0000 osd.5 (osd. 2021-02-10T04:42:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[35961]: 5) 85 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:42:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:42:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[35961]: -10T04:42:07.122838+0000 mon.a 2021-02-10T04:42:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[35961]: (mon.0) 1878 : audit [INF] from='client.? 172.21.15.85:0/919248963' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T04:42:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[35961]: 2021-02-10T04:42:07.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[29319]: cluster 2021-02-10T04:42:07.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[29319]: 2021-02 2021-02-10T04:42:07.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[29319]: -10T04:42:06.226915+0000 mgr.y (mgr.14140) 849 : cluster [DBG] pgmap v1510: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 706 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 4.0 MiB/s rd, 1.8 MiB/s wr, 11 op/s 2021-02-10T04:42:07.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[29319]: 2021-02-10T04:42:07.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[29319]: cluster 2021-02-10T04:42:07.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:42:07.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[29319]: :42:06.414797+0000 osd.4 (osd 2021-02-10T04:42:07.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[29319]: .4) 1547 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:07.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:42:07.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[29319]: 02-10T04:42:06.414805+0000 osd.4 2021-02-10T04:42:07.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[29319]: (osd.4) 1548 : cluster [WRN] 2021-02-10T04:42:07.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:07.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:42:07.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[29319]: 04:42:06.414812+0000 osd.4 (osd. 2021-02-10T04:42:07.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[29319]: 4) 1549 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:07.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:42:07.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[29319]: 02-10T04:42:06.414817+0000 2021-02-10T04:42:07.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[29319]: osd.4 (osd.4) 1550 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:07.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[29319]: 2021-02-10T04:42:07.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[29319]: cluster 2021-02-10T04:42:07.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[29319]: 2021-02-10T04:42:06.512874+0000 osd.5 ( 2021-02-10T04:42:07.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[29319]: osd.5) 84 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:42:07.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:42:07.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[29319]: 02-10T04:42:06.513758+0000 osd.5 2021-02-10T04:42:07.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[29319]: (osd.5) 85 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:42:07.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[29319]: 2021-02-10T04:42:07.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[29319]: audit 2021-02-10T04:42:07.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[29319]: 2021-02-10T04:42:07.122838+0000 mon.a 2021-02-10T04:42:07.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[29319]: (mon.0) 1878 : audit [INF] from='client.? 172.21.15.85:0/919248963' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T04:42:07.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:07 smithi085 conmon[29319]: 2021-02-10T04:42:07.946 INFO:tasks.rados.rados.0.smithi085.stdout:3598: writing smithi08575397-12 from 706224 to 1462178 tid 1 2021-02-10T04:42:07.950 INFO:tasks.rados.rados.0.smithi085.stdout:3598: writing smithi08575397-12 from 2160404 to 2706224 tid 2 2021-02-10T04:42:07.950 INFO:tasks.rados.rados.0.smithi085.stdout:3595: done (7 left) 2021-02-10T04:42:07.950 INFO:tasks.rados.rados.0.smithi085.stdout:3599: rollback oid 29 current snap is 416 2021-02-10T04:42:07.950 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 29 to 266 2021-02-10T04:42:07.950 INFO:tasks.rados.rados.0.smithi085.stdout:3596: expect (ObjNum 1124 snap 380 seq_num 1124) 2021-02-10T04:42:08.141 INFO:tasks.rados.rados.0.smithi085.stdout:3594: finishing write tid 4 to smithi08575397-5 2021-02-10T04:42:08.141 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 1560 (ObjNum 1231 snap 416 seq_num 1231) dirty exists 2021-02-10T04:42:08.141 INFO:tasks.rados.rados.0.smithi085.stdout:3594: left oid 5 (ObjNum 1231 snap 416 seq_num 1231) 2021-02-10T04:42:08.141 INFO:tasks.rados.rados.0.smithi085.stdout:3594: done (7 left) 2021-02-10T04:42:08.141 INFO:tasks.rados.rados.0.smithi085.stdout:3596: done (6 left) 2021-02-10T04:42:08.141 INFO:tasks.rados.rados.0.smithi085.stdout:3600: copy_from oid 20 from oid 42 current snap is 416 2021-02-10T04:42:08.141 INFO:tasks.rados.rados.0.smithi085.stdout:3598: finishing write tid 1 to smithi08575397-12 2021-02-10T04:42:08.141 INFO:tasks.rados.rados.0.smithi085.stdout:3599: finishing rollback tid 1 to smithi08575397-29 2021-02-10T04:42:08.141 INFO:tasks.rados.rados.0.smithi085.stdout:3599: finishing rollback tid 2 to smithi08575397-29 2021-02-10T04:42:08.141 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 29 v 1490 (ObjNum 8 snap 0 seq_num 8) dirty exists 2021-02-10T04:42:08.142 INFO:tasks.rados.rados.0.smithi085.stdout:3598: finishing write tid 2 to smithi08575397-12 2021-02-10T04:42:08.142 INFO:tasks.rados.rados.0.smithi085.stdout:3598: finishing write tid 3 to smithi08575397-12 2021-02-10T04:42:08.142 INFO:tasks.rados.rados.0.smithi085.stdout:3598: finishing write tid 4 to smithi08575397-12 2021-02-10T04:42:08.142 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 1544 (ObjNum 1232 snap 416 seq_num 1232) dirty exists 2021-02-10T04:42:08.142 INFO:tasks.rados.rados.0.smithi085.stdout:3598: left oid 12 (ObjNum 1232 snap 416 seq_num 1232) 2021-02-10T04:42:08.142 INFO:tasks.rados.rados.0.smithi085.stdout:3598: done (6 left) 2021-02-10T04:42:08.142 INFO:tasks.rados.rados.0.smithi085.stdout:3599: done (5 left) 2021-02-10T04:42:08.142 INFO:tasks.rados.rados.0.smithi085.stdout:3601: copy_from oid 12 from oid 16 current snap is 416 2021-02-10T04:42:08.142 INFO:tasks.rados.rados.0.smithi085.stdout:3602: snap_remove snap 408 2021-02-10T04:42:08.301 INFO:tasks.rados.rados.0.smithi085.stdout:3602: done (6 left) 2021-02-10T04:42:08.301 INFO:tasks.rados.rados.0.smithi085.stdout:3601: finishing copy_from to smithi08575397-12 2021-02-10T04:42:08.302 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 1801 (ObjNum 1191 snap 407 seq_num 1191) dirty exists 2021-02-10T04:42:08.302 INFO:tasks.rados.rados.0.smithi085.stdout:3601: finishing copy_from racing read to smithi08575397-12 2021-02-10T04:42:08.302 INFO:tasks.rados.rados.0.smithi085.stdout:3603: read oid 3 snap 354 2021-02-10T04:42:08.302 INFO:tasks.rados.rados.0.smithi085.stdout:3603: expect (ObjNum 1058 snap 348 seq_num 1058) 2021-02-10T04:42:08.302 INFO:tasks.rados.rados.0.smithi085.stdout:3601: done (6 left) 2021-02-10T04:42:08.302 INFO:tasks.rados.rados.0.smithi085.stdout:3604: delete oid 28 current snap is 416 2021-02-10T04:42:08.304 INFO:tasks.rados.rados.0.smithi085.stdout:3604: done (6 left) 2021-02-10T04:42:08.304 INFO:tasks.rados.rados.0.smithi085.stdout:3605: snap_remove snap 405 2021-02-10T04:42:08.441 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[41540]: debug 2021-02-10T04:42:08.123+0000 7fb94bb63700 -1 osd.0 pg_epoch: 908 pg[5.5( empty local-lis/les=57/58 n=0 ec=57/57 lis/c=57/57 les/c/f=58/58/0 sis=57) [0,1,4] r=0 lpr=57 crt=0'0 mlcod 0'0 active+clean] scrubber pg(5.5) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x555d327fc2c0) 2021-02-10T04:42:08.690 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:08 smithi164 conmon[33677]: debug 2021-02-10T04:42:08.690 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:08 smithi164 conmon[33677]: 2021-02-10T04:42:08.439+0000 7fbec19a1700 -1 osd.4 909 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:42:08.690 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:08 smithi164 conmon[33677]: 2021-02-10T04:42:08.691 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:08 smithi164 conmon[29288]: cluster 2021-02-10T04:42:08.691 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:08 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:42:08.691 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:08 smithi164 conmon[29288]: :42:07.454855+0000 osd.4 (osd.4) 1551 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:08.691 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:08 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:42:08.691 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:08 smithi164 conmon[29288]: -02-10T04:42:07. 2021-02-10T04:42:08.691 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:08 smithi164 conmon[29288]: 454863+0000 osd.4 (osd.4) 1552 2021-02-10T04:42:08.692 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:08 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:08.692 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:08 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:42:08.692 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:08 smithi164 conmon[29288]: T04:42:07.454869+0000 osd.4 (osd. 2021-02-10T04:42:08.692 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:08 smithi164 conmon[29288]: 4) 1553 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:08.692 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:08 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:42:08.692 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:08 smithi164 conmon[29288]: 02-10T04:42:07.454874+0000 osd.4 (osd 2021-02-10T04:42:08.692 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:08 smithi164 conmon[29288]: .4) 1554 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:08.692 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:08 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:42:08.692 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:08 smithi164 conmon[29288]: :42:07.685955+0000 mon.a (mon.0) 2021-02-10T04:42:08.692 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:08 smithi164 conmon[29288]: 1879 : audit [INF] from='client.? 172.21.15.85:0/919248963' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2021-02-10T04:42:08.692 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:08 smithi164 conmon[29288]: cluster 2021-02-10T04:42:08.692 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:08 smithi164 conmon[29288]: 2021-02-10T04:42 2021-02-10T04:42:08.692 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:08 smithi164 conmon[29288]: :07.686777+0000 mon.a (mon 2021-02-10T04:42:08.692 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:08 smithi164 conmon[29288]: .0) 1880 : cluster [DBG] osdmap e908: 8 total, 8 up, 7 in 2021-02-10T04:42:08.692 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:08 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:42:08.693 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:08 smithi164 conmon[29288]: -10T04:42:08.300036+0000 mon.a (mon.0) 2021-02-10T04:42:08.693 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:08 smithi164 conmon[29288]: 1881 : cluster [DBG] osdmap e909: 8 total, 8 up, 7 in 2021-02-10T04:42:08.693 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:42:08 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:42:08.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[35961]: cluster 2021-02-10T04:42:07.454855+0000 osd.4 2021-02-10T04:42:08.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[35961]: (osd.4) 1551 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:08.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[35961]: cluster 2021-02-10T04:42:07. 2021-02-10T04:42:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[35961]: 454863+0000 osd.4 (osd.4) 1552 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[35961]: cluster 2021-02-10T04:42:07. 2021-02-10T04:42:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[35961]: 454869+0000 osd.4 (osd.4) 1553 : cluster [WRN] 2021-02-10T04:42:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[35961]: cluster 2021-02-10T04:42:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[35961]: 2021-02-10T04:42:07.454874+0000 osd.4 2021-02-10T04:42:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[35961]: (osd.4) 1554 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:42:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[35961]: 02-10T04:42:07.685955+0000 mon.a (mon 2021-02-10T04:42:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[35961]: .0) 1879 : audit [INF] from='client.? 172.21.15.85:0/919248963' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2021-02-10T04:42:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:42:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[35961]: -02-10T04:42:07.686777+0000 2021-02-10T04:42:08.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[35961]: mon.a (mon.0) 2021-02-10T04:42:08.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[35961]: 1880 : cluster [DBG] osdmap e908: 8 total, 8 up, 7 in 2021-02-10T04:42:08.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[35961]: cluster 2021-02-10T04:42:08.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:42:08.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[35961]: 04:42:08.300036+0000 mon.a (mon.0 2021-02-10T04:42:08.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[35961]: ) 1881 : cluster [DBG] osdmap e909: 8 total, 8 up, 7 in 2021-02-10T04:42:08.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[29319]: cluster 2021-02-10T04:42:07. 2021-02-10T04:42:08.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[29319]: 454855+0000 osd.4 (osd.4) 1551 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:08.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[29319]: cluster 2021-02-10T04:42:07.454863+0000 osd.4 (osd.4) 1552 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:08.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[29319]: cluster 2021-02-10T04:42:07 2021-02-10T04:42:08.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[29319]: .454869+0000 osd.4 (osd.4) 1553 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:08.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:42:08.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[29319]: -10T04:42:07.454874+0000 osd.4 (osd 2021-02-10T04:42:08.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[29319]: .4) 1554 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:08.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:42:08.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[29319]: -10T04:42:07.685955+0000 2021-02-10T04:42:08.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[29319]: mon.a (mon.0) 1879 : audit [INF] from='client.? 172.21.15.85:0/919248963' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2021-02-10T04:42:08.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[29319]: cluster 2021-02-10T04:42:08.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[29319]: 2021-02-10T04:42: 2021-02-10T04:42:08.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[29319]: 07.686777+0000 mon.a (mon.0) 2021-02-10T04:42:08.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[29319]: 1880 : cluster [DBG] osdmap e908: 8 total, 8 up, 7 in 2021-02-10T04:42:08.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[29319]: cluster 2021-02-10T04:42:08.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[29319]: 2021-02-10T04:42: 2021-02-10T04:42:08.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[29319]: 08.300036+0000 mon.a (mon.0) 1881 2021-02-10T04:42:08.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:08 smithi085 conmon[29319]: : cluster [DBG] osdmap e909: 8 total, 8 up, 7 in 2021-02-10T04:42:09.313 INFO:tasks.rados.rados.0.smithi085.stdout:3605: done (6 left) 2021-02-10T04:42:09.313 INFO:tasks.rados.rados.0.smithi085.stdout:3606: delete oid 5 current snap is 416 2021-02-10T04:42:09.313 INFO:tasks.rados.rados.0.smithi085.stdout:3603: expect (ObjNum 1058 snap 348 seq_num 1058) 2021-02-10T04:42:09.488 INFO:tasks.rados.rados.0.smithi085.stdout:3600: finishing copy_from to smithi08575397-20 2021-02-10T04:42:09.488 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 20 v 1532 (ObjNum 1124 snap 380 seq_num 1124) dirty exists 2021-02-10T04:42:09.488 INFO:tasks.rados.rados.0.smithi085.stdout:3600: finishing copy_from racing read to smithi08575397-20 2021-02-10T04:42:09.489 INFO:tasks.rados.rados.0.smithi085.stdout:3600: done (6 left) 2021-02-10T04:42:09.489 INFO:tasks.rados.rados.0.smithi085.stdout:3603: done (5 left) 2021-02-10T04:42:09.489 INFO:tasks.rados.rados.0.smithi085.stdout:3606: done (4 left) 2021-02-10T04:42:09.489 INFO:tasks.rados.rados.0.smithi085.stdout:3607: write (excl) oid 31 current snap is 416 2021-02-10T04:42:09.489 INFO:tasks.rados.rados.0.smithi085.stdout:3607: seq_num 1235 ranges {756645=581701,1935850=473428} 2021-02-10T04:42:09.496 INFO:tasks.rados.rados.0.smithi085.stdout:3607: writing smithi08575397-31 from 756645 to 1338346 tid 1 2021-02-10T04:42:09.503 INFO:tasks.rados.rados.0.smithi085.stdout:3607: writing smithi08575397-31 from 1935850 to 2409278 tid 2 2021-02-10T04:42:09.503 INFO:tasks.rados.rados.0.smithi085.stdout:3607: finishing write tid 1 to smithi08575397-31 2021-02-10T04:42:09.503 INFO:tasks.rados.rados.0.smithi085.stdout:3608: delete oid 23 current snap is 416 2021-02-10T04:42:09.506 INFO:tasks.rados.rados.0.smithi085.stdout:3608: done (5 left) 2021-02-10T04:42:09.506 INFO:tasks.rados.rados.0.smithi085.stdout:3609: write (excl) oid 21 current snap is 416 2021-02-10T04:42:09.506 INFO:tasks.rados.rados.0.smithi085.stdout:3609: seq_num 1237 ranges {790259=732199,1936461=477097} 2021-02-10T04:42:09.516 INFO:tasks.rados.rados.0.smithi085.stdout:3609: writing smithi08575397-21 from 790259 to 1522458 tid 1 2021-02-10T04:42:09.522 INFO:tasks.rados.rados.0.smithi085.stdout:3609: writing smithi08575397-21 from 1936461 to 2413558 tid 2 2021-02-10T04:42:09.522 INFO:tasks.rados.rados.0.smithi085.stdout:3610: snap_create 2021-02-10T04:42:09.522 INFO:tasks.rados.rados.0.smithi085.stdout:3607: finishing write tid 2 to smithi08575397-31 2021-02-10T04:42:09.522 INFO:tasks.rados.rados.0.smithi085.stdout:3607: finishing write tid 3 to smithi08575397-31 2021-02-10T04:42:09.522 INFO:tasks.rados.rados.0.smithi085.stdout:3607: finishing write tid 4 to smithi08575397-31 2021-02-10T04:42:09.522 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 1492 (ObjNum 1235 snap 416 seq_num 1235) dirty exists 2021-02-10T04:42:09.522 INFO:tasks.rados.rados.0.smithi085.stdout:3607: left oid 31 (ObjNum 1235 snap 416 seq_num 1235) 2021-02-10T04:42:09.523 INFO:tasks.rados.rados.0.smithi085.stdout:3609: finishing write tid 1 to smithi08575397-21 2021-02-10T04:42:09.528 INFO:tasks.rados.rados.0.smithi085.stdout:3609: finishing write tid 2 to smithi08575397-21 2021-02-10T04:42:09.528 INFO:tasks.rados.rados.0.smithi085.stdout:3609: finishing write tid 3 to smithi08575397-21 2021-02-10T04:42:09.528 INFO:tasks.rados.rados.0.smithi085.stdout:3609: finishing write tid 4 to smithi08575397-21 2021-02-10T04:42:09.528 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 1862 (ObjNum 1237 snap 416 seq_num 1237) dirty exists 2021-02-10T04:42:09.528 INFO:tasks.rados.rados.0.smithi085.stdout:3609: left oid 21 (ObjNum 1237 snap 416 seq_num 1237) 2021-02-10T04:42:09.895 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:09 smithi164 conmon[33677]: debug 2021-02-10T04:42:09.447+0000 7fbec19a1700 -1 osd.4 910 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:42:09.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:09 smithi164 conmon[29288]: cluster 2021-02-10T04:42:08.227595+0000 mgr.y 2021-02-10T04:42:09.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:09 smithi164 conmon[29288]: (mgr.14140) 850 : cluster [DBG] pgmap v1513: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 714 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 3.2 MiB/s rd, 844 KiB/s wr, 8 op/s 2021-02-10T04:42:09.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:09 smithi164 conmon[29288]: cluster 2021-02-10T04:42:08.440207+0000 osd.4 (osd.4) 1555 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:09.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:09 smithi164 conmon[29288]: cluster 2021-02-10T04:42:08.440217+0000 2021-02-10T04:42:09.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:09 smithi164 conmon[29288]: osd.4 (osd.4) 1556 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:09.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:09 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:42:09.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:09 smithi164 conmon[29288]: 02-10T04:42:08.440224+0000 osd.4 2021-02-10T04:42:09.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:09 smithi164 conmon[29288]: (osd.4) 1557 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:09.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:09 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:42:09.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:09 smithi164 conmon[29288]: 02-10T04:42:08.440230+0000 osd.4 2021-02-10T04:42:09.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:09 smithi164 conmon[29288]: (osd.4) 1558 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:09.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:09 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:42:09.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:09 smithi164 conmon[29288]: 02-10T04:42:08.498068+0000 mgr.y 2021-02-10T04:42:09.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:09 smithi164 conmon[29288]: (mgr.14140) 851 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:42:09.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:09 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:42:09.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:09 smithi164 conmon[29288]: -10T04:42:08.683332+0000 mon.a ( 2021-02-10T04:42:09.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:09 smithi164 conmon[29288]: mon.0) 1882 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 757 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:42:09.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:09 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:42:09.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:09 smithi164 conmon[29288]: -02-10T04:42:09.313373+0000 2021-02-10T04:42:09.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:09 smithi164 conmon[29288]: mon.a (mon.0) 1883 : cluster [DBG] osdmap e910: 8 total, 8 up, 7 in 2021-02-10T04:42:09.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:09 smithi164 conmon[29288]: 2021-02-10T04:42:09.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[35961]: cluster 2021-02-10T04:42:09.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:42:09.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[35961]: 42:08.227595+0000 mgr.y (mgr.14140) 850 : cluster [DBG] pgmap v1513: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 714 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 3.2 MiB/s rd, 844 KiB/s wr, 8 op/s 2021-02-10T04:42:09.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[35961]: cluster 2021-02-10T04:42 2021-02-10T04:42:09.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[35961]: :08.440207+0000 osd.4 (osd.4) 1555 : cluster 2021-02-10T04:42:09.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:09.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:42:09.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[35961]: :42:08.440217+0000 osd.4 (osd.4) 2021-02-10T04:42:09.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[35961]: 1556 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:09.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:42:09.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[35961]: -10T04:42:08.440224+0000 osd.4 ( 2021-02-10T04:42:09.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[35961]: osd.4) 1557 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:09.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:42:09.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[35961]: -02-10T04:42:08.440230+0000 2021-02-10T04:42:09.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[35961]: osd.4 (osd.4) 1558 : cluster [WRN] 2021-02-10T04:42:09.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:09.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[35961]: audit 2021-02-10T04:42:08 2021-02-10T04:42:09.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[35961]: .498068+0000 mgr.y (mgr.14140) 851 : 2021-02-10T04:42:09.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[35961]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:42:09.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:42:09.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[35961]: 04:42:08.683332+0000 mon.a ( 2021-02-10T04:42:09.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[35961]: mon.0) 1882 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 757 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:42:09.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:42:09.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[35961]: 02-10T04:42:09.313373+0000 mon.a (mon.0) 1883 : 2021-02-10T04:42:09.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[35961]: cluster [DBG] osdmap e910: 8 total, 8 up, 7 in 2021-02-10T04:42:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[29319]: cluster 2021-02-10T04:42:08.227595+0000 mgr.y (mgr.14140) 2021-02-10T04:42:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[29319]: 850 : cluster [DBG] pgmap v1513: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 714 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 3.2 MiB/s rd, 844 KiB/s wr, 8 op/s 2021-02-10T04:42:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[29319]: cluster 2021-02-10T04:42:08.440207+0000 osd.4 (osd.4) 1555 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:42:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[29319]: -02-10T04:42:08.440217+0000 osd.4 (osd.4) 1556 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:42:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[29319]: :42:08.440224+0000 osd.4 (osd.4 2021-02-10T04:42:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[29319]: ) 1557 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:42:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[29319]: 10T04:42:08.440230+0000 osd.4 ( 2021-02-10T04:42:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[29319]: osd.4) 1558 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:42:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[29319]: 02-10T04:42:08.498068+0000 2021-02-10T04:42:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[29319]: mgr.y (mgr.14140) 851 : audit 2021-02-10T04:42:09.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[29319]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:42:09.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:42:09.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[29319]: -02- 2021-02-10T04:42:09.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[29319]: 10T04:42:08.683332+0000 mon.a ( 2021-02-10T04:42:09.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[29319]: mon.0) 1882 : cluster [WRN] 2021-02-10T04:42:09.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[29319]: Health check update: 4 slow ops, oldest one blocked for 757 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:42:09.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:42:09.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[29319]: 02-10T04:42:09.313373+0000 2021-02-10T04:42:09.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[29319]: mon.a (mon.0) 1883 : cluster [DBG] 2021-02-10T04:42:09.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:09 smithi085 conmon[29319]: osdmap e910: 8 total, 8 up, 7 in 2021-02-10T04:42:10.310 INFO:tasks.rados.rados.0.smithi085.stdout:3607: done (6 left) 2021-02-10T04:42:10.311 INFO:tasks.rados.rados.0.smithi085.stdout:3609: done (5 left) 2021-02-10T04:42:10.311 INFO:tasks.rados.rados.0.smithi085.stdout:3610: done (4 left) 2021-02-10T04:42:10.311 INFO:tasks.rados.rados.0.smithi085.stdout:3611: write (excl) oid 42 current snap is 417 2021-02-10T04:42:10.311 INFO:tasks.rados.rados.0.smithi085.stdout:3611: seq_num 1238 ranges {407066=407448,1594943=620755} 2021-02-10T04:42:10.315 INFO:tasks.rados.rados.0.smithi085.stdout:3611: writing smithi08575397-42 from 407066 to 814514 tid 1 2021-02-10T04:42:10.325 INFO:tasks.rados.rados.0.smithi085.stdout:3611: writing smithi08575397-42 from 1594943 to 2215698 tid 2 2021-02-10T04:42:10.325 INFO:tasks.rados.rados.0.smithi085.stdout:3612: read oid 28 snap 372 2021-02-10T04:42:10.325 INFO:tasks.rados.rados.0.smithi085.stdout:3612: expect (ObjNum 1052 snap 345 seq_num 1052) 2021-02-10T04:42:10.325 INFO:tasks.rados.rados.0.smithi085.stdout:3613: write oid 25 current snap is 417 2021-02-10T04:42:10.325 INFO:tasks.rados.rados.0.smithi085.stdout:3611: finishing write tid 1 to smithi08575397-42 2021-02-10T04:42:10.326 INFO:tasks.rados.rados.0.smithi085.stdout:3613: seq_num 1239 ranges {423873=482697,1653425=480766,2691441=132432} 2021-02-10T04:42:10.331 INFO:tasks.rados.rados.0.smithi085.stdout:3613: writing smithi08575397-25 from 423873 to 906570 tid 1 2021-02-10T04:42:10.339 INFO:tasks.rados.rados.0.smithi085.stdout:3613: writing smithi08575397-25 from 1653425 to 2134191 tid 2 2021-02-10T04:42:10.343 INFO:tasks.rados.rados.0.smithi085.stdout:3613: writing smithi08575397-25 from 2691441 to 2823873 tid 3 2021-02-10T04:42:10.351 INFO:tasks.rados.rados.0.smithi085.stdout:3612: expect (ObjNum 1052 snap 345 seq_num 1052) 2021-02-10T04:42:10.634 INFO:tasks.rados.rados.0.smithi085.stdout:3611: finishing write tid 2 to smithi08575397-42 2021-02-10T04:42:10.634 INFO:tasks.rados.rados.0.smithi085.stdout:3611: finishing write tid 3 to smithi08575397-42 2021-02-10T04:42:10.634 INFO:tasks.rados.rados.0.smithi085.stdout:3611: finishing write tid 4 to smithi08575397-42 2021-02-10T04:42:10.634 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 1804 (ObjNum 1238 snap 417 seq_num 1238) dirty exists 2021-02-10T04:42:10.634 INFO:tasks.rados.rados.0.smithi085.stdout:3611: left oid 42 (ObjNum 1238 snap 417 seq_num 1238) 2021-02-10T04:42:10.634 INFO:tasks.rados.rados.0.smithi085.stdout:3613: finishing write tid 1 to smithi08575397-25 2021-02-10T04:42:10.634 INFO:tasks.rados.rados.0.smithi085.stdout:3613: finishing write tid 2 to smithi08575397-25 2021-02-10T04:42:10.634 INFO:tasks.rados.rados.0.smithi085.stdout:3613: finishing write tid 3 to smithi08575397-25 2021-02-10T04:42:10.634 INFO:tasks.rados.rados.0.smithi085.stdout:3613: finishing write tid 4 to smithi08575397-25 2021-02-10T04:42:10.635 INFO:tasks.rados.rados.0.smithi085.stdout:3611: done (6 left) 2021-02-10T04:42:10.635 INFO:tasks.rados.rados.0.smithi085.stdout:3612: done (5 left) 2021-02-10T04:42:10.635 INFO:tasks.rados.rados.0.smithi085.stdout:3614: write (excl) oid 50 current snap is 417 2021-02-10T04:42:10.635 INFO:tasks.rados.rados.0.smithi085.stdout:3614: seq_num 1240 ranges {440680=674299,1711907=624424} 2021-02-10T04:42:10.640 INFO:tasks.rados.rados.0.smithi085.stdout:3614: writing smithi08575397-50 from 440680 to 1114979 tid 1 2021-02-10T04:42:10.648 INFO:tasks.rados.rados.0.smithi085.stdout:3614: writing smithi08575397-50 from 1711907 to 2336331 tid 2 2021-02-10T04:42:10.648 INFO:tasks.rados.rados.0.smithi085.stdout:3615: snap_remove snap 415 2021-02-10T04:42:10.689 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:42:10.720 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:10 smithi164 conmon[33677]: debug 2021-02-10T04:42:10.469+0000 7fbec19a1700 -1 osd.4 911 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:42:10.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:10 smithi164 conmon[29288]: cluster 2021-02-10T04:42:09.448629+0000 osd.4 (osd 2021-02-10T04:42:10.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:10 smithi164 conmon[29288]: .4) 1559 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:10.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:10 smithi164 conmon[29288]: cluster 2021-02-10T04:42:09.448638+0000 osd.4 (osd.4) 1560 : cluster 2021-02-10T04:42:10.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:10 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:10.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:10 smithi164 conmon[29288]: cluster 2021-02-10T04:42:09 2021-02-10T04:42:10.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:10 smithi164 conmon[29288]: .448643+0000 osd.4 (osd.4) 1561 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:10.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:10 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:42:10.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:10 smithi164 conmon[29288]: 10T04:42:09.448649+0000 osd.4 (osd. 2021-02-10T04:42:10.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:10 smithi164 conmon[29288]: 4) 1562 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:10.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:10 smithi164 conmon[29288]: cluster 2021-02-10T04:42 2021-02-10T04:42:10.722 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:10 smithi164 conmon[29288]: :10.310163+0000 mon.a (mon.0) 1884 : cluster [DBG] osdmap e911: 8 total, 8 up, 7 in 2021-02-10T04:42:10.723 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:10.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[35961]: cluster 2021-02-10T04:42:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:42:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[35961]: 42:09.448629+0000 osd.4 (osd.4) 1559 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[35961]: cluster 2021-02-10T04:42:09.448638+0000 osd.4 (osd. 2021-02-10T04:42:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[35961]: 4) 1560 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[35961]: cluster 2021-02-10T04:42: 2021-02-10T04:42:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[35961]: 09.448643+0000 osd.4 (osd.4) 1561 : cluster [WRN] 2021-02-10T04:42:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[35961]: cluster 2021-02-10T04:42:09.448649+0000 2021-02-10T04:42:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[35961]: osd.4 (osd.4) 1562 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[35961]: 2021-02-10T04:42:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:42:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[35961]: 04:42:10.310163+0000 mon.a (mon.0) 2021-02-10T04:42:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[35961]: 1884 : cluster [DBG] osdmap e911: 8 total, 8 up, 7 in 2021-02-10T04:42:10.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[29319]: cluster 2021-02-10T04:42:10.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[29319]: 2021-02-10T04:42:10.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[29319]: 2021-02-10T04:42:09.448629 2021-02-10T04:42:10.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1559 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:10.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[29319]: cluster 2021-02-10T04:42:10.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[29319]: 2021-02-10T04:42:10.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[29319]: 2021-02-10T04:42 2021-02-10T04:42:10.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[29319]: :09.448638+0000 osd.4 (osd.4) 1560 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:10.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:42:10.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[29319]: -10T04:42 2021-02-10T04:42:10.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[29319]: :09.448643+0000 osd.4 (osd.4) 1561 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[29319]: cluster 2021-02-10T04:42:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[29319]: 2021-02-10T04:42:09 2021-02-10T04:42:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[29319]: .448649+0000 osd.4 (osd 2021-02-10T04:42:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[29319]: .4) 1562 : cluster [WRN] 2021-02-10T04:42:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[29319]: cluster 2021-02-10T04:42:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[29319]: 2021 2021-02-10T04:42:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[29319]: -02-10T 2021-02-10T04:42:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[29319]: 04:42:10.310163+0000 2021-02-10T04:42:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[29319]: mon.a (mon.0) 1884 2021-02-10T04:42:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:10 smithi085 conmon[29319]: : cluster [DBG] osdmap e911: 8 total, 8 up, 7 in 2021-02-10T04:42:11.144 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:42:10 smithi164 conmon[37263]: debug 2021-02-10T04:42:10.798+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:11.313 INFO:tasks.rados.rados.0.smithi085.stdout:3615: done (6 left) 2021-02-10T04:42:11.314 INFO:tasks.rados.rados.0.smithi085.stdout:3616: rollback oid 48 current snap is 417 2021-02-10T04:42:11.314 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 48 to 385 2021-02-10T04:42:11.314 INFO:tasks.rados.rados.0.smithi085.stdout:3613: finishing write tid 5 to smithi08575397-25 2021-02-10T04:42:11.314 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 25 v 1481 (ObjNum 1239 snap 417 seq_num 1239) dirty exists 2021-02-10T04:42:11.314 INFO:tasks.rados.rados.0.smithi085.stdout:3613: left oid 25 (ObjNum 1239 snap 417 seq_num 1239) 2021-02-10T04:42:11.314 INFO:tasks.rados.rados.0.smithi085.stdout:3614: finishing write tid 1 to smithi08575397-50 2021-02-10T04:42:11.314 INFO:tasks.rados.rados.0.smithi085.stdout:3614: finishing write tid 2 to smithi08575397-50 2021-02-10T04:42:11.314 INFO:tasks.rados.rados.0.smithi085.stdout:3614: finishing write tid 3 to smithi08575397-50 2021-02-10T04:42:11.314 INFO:tasks.rados.rados.0.smithi085.stdout:3614: finishing write tid 4 to smithi08575397-50 2021-02-10T04:42:11.315 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 1598 (ObjNum 1240 snap 417 seq_num 1240) dirty exists 2021-02-10T04:42:11.315 INFO:tasks.rados.rados.0.smithi085.stdout:3614: left oid 50 (ObjNum 1240 snap 417 seq_num 1240) 2021-02-10T04:42:11.315 INFO:tasks.rados.rados.0.smithi085.stdout:3613: done (6 left) 2021-02-10T04:42:11.315 INFO:tasks.rados.rados.0.smithi085.stdout:3614: done (5 left) 2021-02-10T04:42:11.315 INFO:tasks.rados.rados.0.smithi085.stdout:3617: snap_create 2021-02-10T04:42:11.319 INFO:tasks.rados.rados.0.smithi085.stdout:3616: finishing rollback tid 0 to smithi08575397-48 2021-02-10T04:42:11.319 INFO:tasks.rados.rados.0.smithi085.stdout:3616: finishing rollback tid 1 to smithi08575397-48 2021-02-10T04:42:11.319 INFO:tasks.rados.rados.0.smithi085.stdout:3616: finishing rollback tid 2 to smithi08575397-48 2021-02-10T04:42:11.320 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 1807 (ObjNum 1142 snap 385 seq_num 1142) dirty exists 2021-02-10T04:42:11.345 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:42:11.451 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:42:11.478 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:11.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:11 smithi164 conmon[33677]: debug 2021-02-10T04:42:11.501+0000 7fbec19a1700 -1 osd.4 911 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:42:11.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:11 smithi164 conmon[33677]: debug 2021-02-10T04:42:11.548+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:11.810 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:42:11.915 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:42:11.947 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:12.017 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:11 smithi085 conmon[29319]: cluster 2021-02-10T04:42:10.228495+0000 2021-02-10T04:42:12.018 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:11 smithi085 conmon[29319]: mgr.y (mgr.14140) 852 : cluster [DBG] pgmap v1516: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 722 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 9.8 MiB/s rd, 2.3 MiB/s wr, 16 op/s 2021-02-10T04:42:12.018 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:11 smithi085 conmon[29319]: cluster 2021-02-10T04:42:10.469697+0000 osd.4 (osd.4) 1563 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:12.018 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:11 smithi085 conmon[29319]: cluster 2021-02-10T04:42:12.018 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:11 smithi085 conmon[29319]: 2021-02-10T04:42:10.469704+0000 osd.4 (osd.4) 1564 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:12.018 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:11 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:42:12.018 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:11 smithi085 conmon[29319]: 04:42:10.469709+0000 osd.4 (osd.4) 1565 : 2021-02-10T04:42:12.018 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:11 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:12.018 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:11 smithi085 conmon[29319]: cluster 2021-02-10T04:42 2021-02-10T04:42:12.018 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:11 smithi085 conmon[29319]: :10.469715+0000 osd.4 (osd.4) 1566 : cluster 2021-02-10T04:42:12.018 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:11 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:12.018 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:11 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:42:12.019 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:11 smithi085 conmon[29319]: 04:42:11.313259+0000 mon.a (mon.0 2021-02-10T04:42:12.019 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:11 smithi085 conmon[29319]: ) 1885 : cluster [DBG] osdmap e912: 8 total, 8 up, 7 in 2021-02-10T04:42:12.019 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:11 smithi085 conmon[35961]: cluster 2021-02-10T04:42:10 2021-02-10T04:42:12.019 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:11 smithi085 conmon[35961]: .228495+0000 mgr.y (mgr.14140) 852 : cluster [DBG] pgmap v1516: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 722 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 9.8 MiB/s rd, 2.3 MiB/s wr, 16 op/s 2021-02-10T04:42:12.019 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:11 smithi085 conmon[35961]: cluster 2021-02-10T04:42:10.469697+0000 osd.4 ( 2021-02-10T04:42:12.019 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:11 smithi085 conmon[35961]: osd.4) 1563 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:12.019 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:11 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:42:12.019 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:11 smithi085 conmon[35961]: -10T04:42:10.469704+0000 osd.4 (osd. 2021-02-10T04:42:12.019 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:11 smithi085 conmon[35961]: 4) 1564 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:12.020 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:11 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:42:12.020 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:11 smithi085 conmon[35961]: 04:42:10.469709+0000 osd.4 (osd.4) 1565 : cluster [WRN] 2021-02-10T04:42:12.020 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:11 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:12.020 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:11 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:42:12.020 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:11 smithi085 conmon[35961]: :42:10.469715+0000 osd.4 (osd.4) 1566 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:12.020 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:11 smithi085 conmon[35961]: cluster 2021-02-10T04:42:12.020 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:11 smithi085 conmon[35961]: 2021-02-10T04:42:11.313259+0000 mon.a (mon.0) 1885 : cluster [DBG] 2021-02-10T04:42:12.020 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:11 smithi085 conmon[35961]: osdmap e912: 8 total, 8 up, 7 in 2021-02-10T04:42:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:11 smithi164 conmon[29288]: cluster 2021-02-10T04:42:10.228495+0000 mgr.y ( 2021-02-10T04:42:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:11 smithi164 conmon[29288]: mgr.14140) 852 : cluster [DBG] pgmap v1516: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 722 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 9.8 MiB/s rd, 2.3 MiB/s wr, 16 op/s 2021-02-10T04:42:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:11 smithi164 conmon[29288]: cluster 2021-02-10T04:42:10.469697+0000 osd.4 (osd.4) 1563 : cluster [WRN] 2021-02-10T04:42:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:11 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:11 smithi164 conmon[29288]: cluster 2021-02-10T04:42: 2021-02-10T04:42:12.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:11 smithi164 conmon[29288]: 10.469704+0000 osd.4 (osd.4) 1564 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:12.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:11 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:42:12.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:11 smithi164 conmon[29288]: -10T04:42:10.469709+0000 osd.4 (osd.4 2021-02-10T04:42:12.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:11 smithi164 conmon[29288]: ) 1565 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:12.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:11 smithi164 conmon[29288]: cluster 2021-02-10T04:42:12.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:11 smithi164 conmon[29288]: 2021-02-10T04:42:10.469715 2021-02-10T04:42:12.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:11 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1566 : cluster [WRN] 2021-02-10T04:42:12.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:11 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:12.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:11 smithi164 conmon[29288]: cluster 2021-02-10T04:42: 2021-02-10T04:42:12.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:11 smithi164 conmon[29288]: 11.313259+0000 mon.a (mon.0) 1885 2021-02-10T04:42:12.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:11 smithi164 conmon[29288]: : cluster [DBG] osdmap e912: 8 total, 8 up, 7 in 2021-02-10T04:42:12.288 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:42:12.313 INFO:tasks.rados.rados.0.smithi085.stdout:3616: done (5 left) 2021-02-10T04:42:12.313 INFO:tasks.rados.rados.0.smithi085.stdout:3617: done (4 left) 2021-02-10T04:42:12.313 INFO:tasks.rados.rados.0.smithi085.stdout:3618: read oid 48 snap -1 2021-02-10T04:42:12.313 INFO:tasks.rados.rados.0.smithi085.stdout:3618: expect (ObjNum 1142 snap 385 seq_num 1142) 2021-02-10T04:42:12.313 INFO:tasks.rados.rados.0.smithi085.stdout:3619: write oid 13 current snap is 418 2021-02-10T04:42:12.313 INFO:tasks.rados.rados.0.smithi085.stdout:3619: seq_num 1241 ranges {457487=749548,1770389=484435,2746287=111200} 2021-02-10T04:42:12.320 INFO:tasks.rados.rados.0.smithi085.stdout:3619: writing smithi08575397-13 from 457487 to 1207035 tid 1 2021-02-10T04:42:12.326 INFO:tasks.rados.rados.0.smithi085.stdout:3619: writing smithi08575397-13 from 1770389 to 2254824 tid 2 2021-02-10T04:42:12.330 INFO:tasks.rados.rados.0.smithi085.stdout:3619: writing smithi08575397-13 from 2746287 to 2857487 tid 3 2021-02-10T04:42:12.330 INFO:tasks.rados.rados.0.smithi085.stdout:3620: rollback oid 34 current snap is 418 2021-02-10T04:42:12.331 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 34 to 396 2021-02-10T04:42:12.331 INFO:tasks.rados.rados.0.smithi085.stdout:3618: expect (ObjNum 1142 snap 385 seq_num 1142) 2021-02-10T04:42:12.441 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[45443]: debug 2021-02-10T04:42:12.441 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[45443]: 2021-02-10T04:42:12.016+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:12.528 INFO:tasks.rados.rados.0.smithi085.stdout:3618: done (6 left) 2021-02-10T04:42:12.528 INFO:tasks.rados.rados.0.smithi085.stdout:3621: snap_create 2021-02-10T04:42:12.528 INFO:tasks.rados.rados.0.smithi085.stdout:3619: finishing write tid 1 to smithi08575397-13 2021-02-10T04:42:12.528 INFO:tasks.rados.rados.0.smithi085.stdout:3619: finishing write tid 2 to smithi08575397-13 2021-02-10T04:42:12.528 INFO:tasks.rados.rados.0.smithi085.stdout:3619: finishing write tid 3 to smithi08575397-13 2021-02-10T04:42:12.528 INFO:tasks.rados.rados.0.smithi085.stdout:3619: finishing write tid 4 to smithi08575397-13 2021-02-10T04:42:12.528 INFO:tasks.rados.rados.0.smithi085.stdout:3619: finishing write tid 5 to smithi08575397-13 2021-02-10T04:42:12.529 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 1565 (ObjNum 1241 snap 418 seq_num 1241) dirty exists 2021-02-10T04:42:12.529 INFO:tasks.rados.rados.0.smithi085.stdout:3619: left oid 13 (ObjNum 1241 snap 418 seq_num 1241) 2021-02-10T04:42:12.529 INFO:tasks.rados.rados.0.smithi085.stdout:3620: finishing rollback tid 0 to smithi08575397-34 2021-02-10T04:42:12.529 INFO:tasks.rados.rados.0.smithi085.stdout:3620: finishing rollback tid 1 to smithi08575397-34 2021-02-10T04:42:12.529 INFO:tasks.rados.rados.0.smithi085.stdout:3620: finishing rollback tid 2 to smithi08575397-34 2021-02-10T04:42:12.529 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 34 v 1568 (ObjNum 1145 snap 385 seq_num 1145) dirty exists 2021-02-10T04:42:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:12 smithi164 conmon[29288]: cluster 2021-02-10T04:42:11.501783+0000 osd.4 (osd.4) 1567 : cluster [WRN] 2021-02-10T04:42:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:12 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:12 smithi164 conmon[29288]: cluster 2021-02-10T04:42:11.501792+0000 osd.4 (osd.4) 1568 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:12 smithi164 conmon[29288]: cluster 2021-02-10T04:42:11.501799+0000 osd.4 (osd.4) 1569 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:12 smithi164 conmon[29288]: cluster 2021-02-10T04:42:11.501805+0000 osd.4 (osd.4) 1570 : 2021-02-10T04:42:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:12 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:12 smithi164 conmon[29288]: cluster 2021-02-10T04:42:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:12 smithi164 conmon[29288]: 2021-02-10T04:42:11.936426+0000 osd.2 (osd.2) 1941 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:42:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:12 smithi164 conmon[29288]: 2021-02-10T04:42:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:12 smithi164 conmon[29288]: cluster 2021-02-10T04:42:12.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:12 smithi164 conmon[29288]: 2021-02-10T04:42:11.937073+0000 osd.2 (osd 2021-02-10T04:42:12.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:12 smithi164 conmon[29288]: .2) 1942 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:42:12.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:12 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:42:12.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:12 smithi164 conmon[29288]: 02-10T04:42:12 2021-02-10T04:42:12.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:12 smithi164 conmon[29288]: .312248+0000 mon.a (mon.0) 1886 : cluster 2021-02-10T04:42:12.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:12 smithi164 conmon[29288]: [DBG] osdmap e913: 8 total, 8 up, 7 in 2021-02-10T04:42:12.896 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:12 smithi164 conmon[33677]: debug 2021-02-10T04:42:12.896 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:12 smithi164 conmon[33677]: 2021-02-10T04:42:12.543+0000 7fbec19a1700 -1 osd.4 913 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:42:12.896 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:12 smithi164 conmon[33677]: 2021-02-10T04:42:13.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[29319]: cluster 2021-02-10T04:42:13.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[29319]: 2021-02-10T04:42:13.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[29319]: 2021 2021-02-10T04:42:13.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[29319]: -02-10T04: 2021-02-10T04:42:13.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[29319]: 42:11.501783+0000 osd.4 (osd.4) 1567 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:13.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[29319]: cluster 2021-02-10T04:42:11 2021-02-10T04:42:13.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[29319]: .501792+0000 osd.4 (osd.4) 1568 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:13.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[29319]: cluster 2021-02-10T04:42:11. 2021-02-10T04:42:13.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[29319]: 501799+0000 osd.4 (osd.4) 1569 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:13.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:42:13.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[29319]: 10T04:42:11.501805+0000 osd.4 (osd.4) 1570 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:13.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:42:13.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[29319]: T04:42:11.936426+0000 osd.2 (osd.2) 1941 : cluster [DBG] 2021-02-10T04:42:13.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[29319]: purged_snaps scrub starts 2021-02-10T04:42:13.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:42:13.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[29319]: -10T04:42:11.937073+0000 osd.2 (osd.2) 1942 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:42:13.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[29319]: 2021-02-10T04:42:13.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:42:13.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[29319]: -10T04:42:12.312248+0000 mon.a (mon.0) 1886 2021-02-10T04:42:13.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[29319]: : cluster [DBG] osdmap e913: 8 total, 8 up, 7 in 2021-02-10T04:42:13.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: cluster 2021-02-10T04:42:13.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: 2021-02-10T04:42:13.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: 2021-02-10T04:42: 2021-02-10T04:42:13.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: 11.501783+0000 osd.4 (osd.4) 1567 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:13.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:42:13.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: 02-10T04:42:11.501792+0000 osd.4 (osd 2021-02-10T04:42:13.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: .4) 1568 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:13.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: cluster 2021-02-10T04:42:13.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: 2021-02-10T04:42:11.501799+0000 osd.4 ( 2021-02-10T04:42:13.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: osd.4) 1569 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:13.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: 2021-02-10T04:42:13.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: cluster 2021-02-10T04:42:13.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: 2021 2021-02-10T04:42:13.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: - 2021-02-10T04:42:13.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: 02 2021-02-10T04:42:13.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: - 2021-02-10T04:42:13.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: 10 2021-02-10T04:42:13.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: T 2021-02-10T04:42:13.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: 04 2021-02-10T04:42:13.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: : 2021-02-10T04:42:13.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: 42:11.501805+0000 osd.4 ( 2021-02-10T04:42:13.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: osd.4) 1570 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:13.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:42:13.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: 02-10T04:42:11.936426+0000 osd.2 (osd.2) 2021-02-10T04:42:13.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: 1941 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:42:13.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: cluster 2021-02-10T04:42:13.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:42:13.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: T04:42:11.937073+0000 osd.2 (osd.2 2021-02-10T04:42:13.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: ) 1942 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:42:13.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:42:13.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: -02-10T04:42:12.312248+0000 mon.a ( 2021-02-10T04:42:13.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:12 smithi085 conmon[35961]: mon.0) 1886 : cluster [DBG] osdmap e913: 8 total, 8 up, 7 in 2021-02-10T04:42:13.316 INFO:tasks.rados.rados.0.smithi085.stdout:3619: done (6 left) 2021-02-10T04:42:13.317 INFO:tasks.rados.rados.0.smithi085.stdout:3620: done (5 left) 2021-02-10T04:42:13.317 INFO:tasks.rados.rados.0.smithi085.stdout:3621: done (4 left) 2021-02-10T04:42:13.317 INFO:tasks.rados.rados.0.smithi085.stdout:3622: read oid 30 snap -1 2021-02-10T04:42:13.317 INFO:tasks.rados.rados.0.smithi085.stdout:3622: expect deleted 2021-02-10T04:42:13.317 INFO:tasks.rados.rados.0.smithi085.stdout:3623: rollback oid 21 current snap is 419 2021-02-10T04:42:13.317 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 21 to 386 2021-02-10T04:42:13.317 INFO:tasks.rados.rados.0.smithi085.stdout:3624: read oid 26 snap 333 2021-02-10T04:42:13.317 INFO:tasks.rados.rados.0.smithi085.stdout:3624: expect (ObjNum 1011 snap 333 seq_num 1011) 2021-02-10T04:42:13.317 INFO:tasks.rados.rados.0.smithi085.stdout:3625: copy_from oid 40 from oid 16 current snap is 419 2021-02-10T04:42:13.317 INFO:tasks.rados.rados.0.smithi085.stdout:3626: read oid 36 snap -1 2021-02-10T04:42:13.317 INFO:tasks.rados.rados.0.smithi085.stdout:3626: expect (ObjNum 904 snap 297 seq_num 904) 2021-02-10T04:42:13.317 INFO:tasks.rados.rados.0.smithi085.stdout:3627: read oid 4 snap -1 2021-02-10T04:42:13.318 INFO:tasks.rados.rados.0.smithi085.stdout:3627: expect deleted 2021-02-10T04:42:13.318 INFO:tasks.rados.rados.0.smithi085.stdout:3628: read oid 29 snap -1 2021-02-10T04:42:13.318 INFO:tasks.rados.rados.0.smithi085.stdout:3628: expect (ObjNum 8 snap 0 seq_num 8) 2021-02-10T04:42:13.318 INFO:tasks.rados.rados.0.smithi085.stdout:3629: delete oid 15 current snap is 419 2021-02-10T04:42:13.332 INFO:tasks.rados.rados.0.smithi085.stdout:3623: finishing rollback tid 0 to smithi08575397-21 2021-02-10T04:42:13.334 INFO:tasks.rados.rados.0.smithi085.stdout:3624: expect (ObjNum 1011 snap 333 seq_num 1011) 2021-02-10T04:42:13.476 INFO:tasks.rados.rados.0.smithi085.stdout:3622: done (11 left) 2021-02-10T04:42:13.476 INFO:tasks.rados.rados.0.smithi085.stdout:3624: done (10 left) 2021-02-10T04:42:13.476 INFO:tasks.rados.rados.0.smithi085.stdout:3627: done (9 left) 2021-02-10T04:42:13.476 INFO:tasks.rados.rados.0.smithi085.stdout:3629: done (8 left) 2021-02-10T04:42:13.477 INFO:tasks.rados.rados.0.smithi085.stdout:3630: snap_remove snap 395 2021-02-10T04:42:13.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:13 smithi164 conmon[33677]: debug 2021-02-10T04:42:13.541+0000 7fbec19a1700 -1 osd.4 913 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:42:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:13 smithi164 conmon[29288]: cluster 2021-02-10T04:42:12.229121+0000 mgr.y ( 2021-02-10T04:42:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:13 smithi164 conmon[29288]: mgr.14140) 853 : cluster [DBG] pgmap v1519: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 722 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 6.6 MiB/s rd, 1.5 MiB/s wr, 8 op/s 2021-02-10T04:42:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:13 smithi164 conmon[29288]: cluster 2021-02-10T04:42:12.544473+0000 osd.4 (osd.4) 1571 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:13 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:42:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:13 smithi164 conmon[29288]: 04:42:12.544483+0000 osd.4 (osd.4) 1572 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:13.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:13 smithi164 conmon[29288]: cluster 2021-02-10T04:42: 2021-02-10T04:42:13.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:13 smithi164 conmon[29288]: 12.544490+0000 osd.4 (osd.4) 1573 : 2021-02-10T04:42:13.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:13 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:13.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:13 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:42:13.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:13 smithi164 conmon[29288]: 10T04:42:12.544496+0000 osd.4 (osd. 2021-02-10T04:42:13.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:13 smithi164 conmon[29288]: 4) 1574 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:13.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:13 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:42:13.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:13 smithi164 conmon[29288]: -10T04:42:13.315698+0000 mon.a (mon 2021-02-10T04:42:13.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:13 smithi164 conmon[29288]: .0) 1887 : cluster [DBG] osdmap e914: 8 total, 8 up, 7 in 2021-02-10T04:42:14.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[29319]: cluster 2021-02-10T04:42:12.229121+0000 mgr.y ( 2021-02-10T04:42:14.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[29319]: mgr.14140) 853 : cluster [DBG] pgmap v1519: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 722 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 6.6 MiB/s rd, 1.5 MiB/s wr, 8 op/s 2021-02-10T04:42:14.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[29319]: cluster 2021-02-10T04:42:12.544473+0000 osd.4 (osd.4) 1571 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:14.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[29319]: 2021-02-10T04:42:14.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[29319]: cluster 2021-02-10T04:42 2021-02-10T04:42:14.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[29319]: :12.544483+0000 osd.4 (osd.4) 1572 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:14.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:42:14.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[29319]: -02-10T04:42:12. 2021-02-10T04:42:14.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[29319]: 544490+0000 osd.4 (osd.4) 1573 : cluster 2021-02-10T04:42:14.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:14.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[29319]: cluster 2021-02-10T04:42:12 2021-02-10T04:42:14.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[29319]: .544496+0000 osd.4 (osd.4) 1574 : 2021-02-10T04:42:14.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:14.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:42:14.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[29319]: -02-10T04:42:13.315698+0000 2021-02-10T04:42:14.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[29319]: mon.a (mon.0) 1887 : cluster [DBG] 2021-02-10T04:42:14.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[29319]: osdmap e914: 8 total, 8 up, 7 in 2021-02-10T04:42:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[35961]: cluster 2021-02-10T04:42:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[35961]: 2021-02-10T04:42 2021-02-10T04:42:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[35961]: :12.229121+0000 mgr.y (mgr.14140) 853 : cluster [DBG] pgmap v1519: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 722 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 6.6 MiB/s rd, 1.5 MiB/s wr, 8 op/s 2021-02-10T04:42:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[35961]: cluster 2021-02-10T04:42:12 2021-02-10T04:42:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[35961]: .544473+0000 osd.4 (osd.4) 1571 : cluster [WRN] 2021-02-10T04:42:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[35961]: cluster 2021-02-10T04:42 2021-02-10T04:42:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[35961]: :12.544483+0000 osd.4 (osd.4) 1572 : cluster 2021-02-10T04:42:14.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:14.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[35961]: cluster 2021-02-10T04:42 2021-02-10T04:42:14.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[35961]: :12.544490+0000 osd.4 (osd.4) 1573 : 2021-02-10T04:42:14.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:14.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[35961]: cluster 2021-02-10T04:42:12. 2021-02-10T04:42:14.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[35961]: 544496+0000 osd.4 (osd.4) 1574 : cluster 2021-02-10T04:42:14.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:14.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[35961]: cluster 2021-02-10T04:42 2021-02-10T04:42:14.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[35961]: :13.315698+0000 mon.a (mon.0) 2021-02-10T04:42:14.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:13 smithi085 conmon[35961]: 1887 : cluster [DBG] osdmap e914: 8 total, 8 up, 7 in 2021-02-10T04:42:14.317 INFO:tasks.rados.rados.0.smithi085.stdout:3630: done (8 left) 2021-02-10T04:42:14.317 INFO:tasks.rados.rados.0.smithi085.stdout:3631: write (excl) oid 50 current snap is 419 2021-02-10T04:42:14.317 INFO:tasks.rados.rados.0.smithi085.stdout:3623: finishing rollback tid 1 to smithi08575397-21 2021-02-10T04:42:14.317 INFO:tasks.rados.rados.0.smithi085.stdout:3623: finishing rollback tid 2 to smithi08575397-21 2021-02-10T04:42:14.317 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 1872 (ObjNum 1134 snap 381 seq_num 1134) dirty exists 2021-02-10T04:42:14.317 INFO:tasks.rados.rados.0.smithi085.stdout:3628: expect (ObjNum 8 snap 0 seq_num 8) 2021-02-10T04:42:14.407 INFO:tasks.rados.rados.0.smithi085.stdout:3626: expect (ObjNum 904 snap 297 seq_num 904) 2021-02-10T04:42:14.665 INFO:tasks.rados.rados.0.smithi085.stdout:3625: finishing copy_from to smithi08575397-40 2021-02-10T04:42:14.666 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 1801 (ObjNum 1191 snap 407 seq_num 1191) dirty exists 2021-02-10T04:42:14.666 INFO:tasks.rados.rados.0.smithi085.stdout:3625: finishing copy_from racing read to smithi08575397-40 2021-02-10T04:42:14.666 INFO:tasks.rados.rados.0.smithi085.stdout:3631: seq_num 1243 ranges {491101=500046,1487353=488104,2401133=489968} 2021-02-10T04:42:14.671 INFO:tasks.rados.rados.0.smithi085.stdout:3631: writing smithi08575397-50 from 491101 to 991147 tid 1 2021-02-10T04:42:14.679 INFO:tasks.rados.rados.0.smithi085.stdout:3631: writing smithi08575397-50 from 1487353 to 1975457 tid 2 2021-02-10T04:42:14.685 INFO:tasks.rados.rados.0.smithi085.stdout:3631: writing smithi08575397-50 from 2401133 to 2891101 tid 3 2021-02-10T04:42:14.686 INFO:tasks.rados.rados.0.smithi085.stdout:3631: finishing write tid 1 to smithi08575397-50 2021-02-10T04:42:14.686 INFO:tasks.rados.rados.0.smithi085.stdout:3631: finishing write tid 2 to smithi08575397-50 2021-02-10T04:42:14.686 INFO:tasks.rados.rados.0.smithi085.stdout:3623: done (8 left) 2021-02-10T04:42:14.686 INFO:tasks.rados.rados.0.smithi085.stdout:3625: done (7 left) 2021-02-10T04:42:14.686 INFO:tasks.rados.rados.0.smithi085.stdout:3626: done (6 left) 2021-02-10T04:42:14.686 INFO:tasks.rados.rados.0.smithi085.stdout:3628: done (5 left) 2021-02-10T04:42:14.686 INFO:tasks.rados.rados.0.smithi085.stdout:3632: write oid 5 current snap is 419 2021-02-10T04:42:14.687 INFO:tasks.rados.rados.0.smithi085.stdout:3632: seq_num 1244 ranges {507908=575295,1829482=631762} 2021-02-10T04:42:14.694 INFO:tasks.rados.rados.0.smithi085.stdout:3632: writing smithi08575397-5 from 507908 to 1083203 tid 1 2021-02-10T04:42:14.704 INFO:tasks.rados.rados.0.smithi085.stdout:3632: writing smithi08575397-5 from 1829482 to 2461244 tid 2 2021-02-10T04:42:14.704 INFO:tasks.rados.rados.0.smithi085.stdout:3633: rollback oid 25 current snap is 419 2021-02-10T04:42:14.704 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 25 to 369 2021-02-10T04:42:14.704 INFO:tasks.rados.rados.0.smithi085.stdout:3631: finishing write tid 3 to smithi08575397-50 2021-02-10T04:42:14.704 INFO:tasks.rados.rados.0.smithi085.stdout:3631: finishing write tid 4 to smithi08575397-50 2021-02-10T04:42:14.704 INFO:tasks.rados.rados.0.smithi085.stdout:3631: finishing write tid 5 to smithi08575397-50 2021-02-10T04:42:14.704 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 1602 (ObjNum 1243 snap 419 seq_num 1243) dirty exists 2021-02-10T04:42:14.705 INFO:tasks.rados.rados.0.smithi085.stdout:3631: left oid 50 (ObjNum 1243 snap 419 seq_num 1243) 2021-02-10T04:42:14.705 INFO:tasks.rados.rados.0.smithi085.stdout:3632: finishing write tid 1 to smithi08575397-5 2021-02-10T04:42:14.705 INFO:tasks.rados.rados.0.smithi085.stdout:3631: done (6 left) 2021-02-10T04:42:14.705 INFO:tasks.rados.rados.0.smithi085.stdout:3634: read oid 40 snap -1 2021-02-10T04:42:14.705 INFO:tasks.rados.rados.0.smithi085.stdout:3634: expect (ObjNum 1191 snap 407 seq_num 1191) 2021-02-10T04:42:14.705 INFO:tasks.rados.rados.0.smithi085.stdout:3635: read oid 48 snap -1 2021-02-10T04:42:14.705 INFO:tasks.rados.rados.0.smithi085.stdout:3635: expect (ObjNum 1142 snap 385 seq_num 1142) 2021-02-10T04:42:14.705 INFO:tasks.rados.rados.0.smithi085.stdout:3636: copy_from oid 26 from oid 46 current snap is 419 2021-02-10T04:42:14.705 INFO:tasks.rados.rados.0.smithi085.stdout:3637: snap_create 2021-02-10T04:42:14.710 INFO:tasks.rados.rados.0.smithi085.stdout:3636: finishing copy_from to smithi08575397-26 2021-02-10T04:42:14.710 INFO:tasks.rados.rados.0.smithi085.stdout:3636: got expected ENOENT (src dne) 2021-02-10T04:42:14.710 INFO:tasks.rados.rados.0.smithi085.stdout:3636: finishing copy_from racing read to smithi08575397-26 2021-02-10T04:42:14.711 INFO:tasks.rados.rados.0.smithi085.stdout:3633: finishing rollback tid 0 to smithi08575397-25 2021-02-10T04:42:14.711 INFO:tasks.rados.rados.0.smithi085.stdout:3633: finishing rollback tid 1 to smithi08575397-25 2021-02-10T04:42:14.711 INFO:tasks.rados.rados.0.smithi085.stdout:3633: finishing rollback tid 2 to smithi08575397-25 2021-02-10T04:42:14.711 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 25 v 1484 (ObjNum 1016 snap 334 seq_num 1016) dirty exists 2021-02-10T04:42:14.715 INFO:tasks.rados.rados.0.smithi085.stdout:3634: expect (ObjNum 1191 snap 407 seq_num 1191) 2021-02-10T04:42:14.809 INFO:tasks.rados.rados.0.smithi085.stdout:3635: expect (ObjNum 1142 snap 385 seq_num 1142) 2021-02-10T04:42:14.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:14 smithi164 conmon[33677]: debug 2021-02-10T04:42:14.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:14 smithi164 conmon[33677]: 2021-02-10T04:42:14.522+0000 7fbec19a1700 -1 osd.4 915 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:42:14.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:14 smithi164 conmon[33677]: 2021-02-10T04:42:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:14 smithi164 conmon[29288]: cluster 2021-02-10T04:42:13.542184+0000 osd.4 (osd. 2021-02-10T04:42:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:14 smithi164 conmon[29288]: 4) 1575 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:14 smithi164 conmon[29288]: cluster 2021-02-10T04:42:13.542194+0000 osd.4 (osd.4) 1576 : cluster [WRN] 2021-02-10T04:42:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:14 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:14 smithi164 conmon[29288]: cluster 2021-02-10T04:42:13 2021-02-10T04:42:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:14 smithi164 conmon[29288]: .542202+0000 osd.4 (osd.4) 2021-02-10T04:42:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:14 smithi164 conmon[29288]: 1577 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:14 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:42:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:14 smithi164 conmon[29288]: -10T04:42:13.542208+0000 osd.4 (osd 2021-02-10T04:42:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:14 smithi164 conmon[29288]: .4) 1578 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:14 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:42:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:14 smithi164 conmon[29288]: 02-10T04:42:14. 2021-02-10T04:42:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:14 smithi164 conmon[29288]: 317035+0000 mon.a (mon.0) 1888 : cluster [DBG] 2021-02-10T04:42:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:14 smithi164 conmon[29288]: osdmap e915: 8 total, 8 up, 7 in 2021-02-10T04:42:15.005 INFO:tasks.rados.rados.0.smithi085.stdout:3632: finishing write tid 2 to smithi08575397-5 2021-02-10T04:42:15.005 INFO:tasks.rados.rados.0.smithi085.stdout:3632: finishing write tid 3 to smithi08575397-5 2021-02-10T04:42:15.006 INFO:tasks.rados.rados.0.smithi085.stdout:3632: finishing write tid 4 to smithi08575397-5 2021-02-10T04:42:15.006 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 1571 (ObjNum 1244 snap 419 seq_num 1244) dirty exists 2021-02-10T04:42:15.006 INFO:tasks.rados.rados.0.smithi085.stdout:3632: left oid 5 (ObjNum 1244 snap 419 seq_num 1244) 2021-02-10T04:42:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:14 smithi085 conmon[29319]: cluster 2021-02-10T04:42:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:14 smithi085 conmon[29319]: 2021-02-10T04:42:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:14 smithi085 conmon[29319]: 2021-02-10T04:42:13.542184+0000 2021-02-10T04:42:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:14 smithi085 conmon[29319]: osd.4 (osd.4) 1575 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:14 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:42:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:14 smithi085 conmon[29319]: 02-10T04:42:13.542194+0000 osd.4 (osd.4) 1576 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:14 smithi085 conmon[29319]: cluster 2021-02-10T04:42:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:14 smithi085 conmon[29319]: 2021-02-10T04:42:13.542202+0000 osd.4 (osd.4) 2021-02-10T04:42:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:14 smithi085 conmon[29319]: 1577 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:14 smithi085 conmon[29319]: cluster 2021-02-10T04:42:15.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:14 smithi085 conmon[29319]: 2021-02-10T04:42:13.542208+0000 osd.4 (osd.4) 2021-02-10T04:42:15.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:14 smithi085 conmon[29319]: 1578 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:15.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:14 smithi085 conmon[29319]: cluster 2021-02-10T04:42 2021-02-10T04:42:15.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:14 smithi085 conmon[29319]: :14.317035+0000 mon.a (mon.0) 1888 : cluster [DBG] osdmap e915: 8 total, 8 up, 7 in 2021-02-10T04:42:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:14 smithi085 conmon[35961]: cluster 2021-02-10T04:42:13.542184 2021-02-10T04:42:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:14 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1575 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:14 smithi085 conmon[35961]: cluster 2021-02-10T04:42:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:14 smithi085 conmon[35961]: 2021-02-10T04:42:13.542194+0000 osd.4 (osd.4) 1576 : cluster 2021-02-10T04:42:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:14 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:14 smithi085 conmon[35961]: cluster 2021-02-10T04:42: 2021-02-10T04:42:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:14 smithi085 conmon[35961]: 13.542202+0000 osd.4 (osd.4) 1577 : cluster 2021-02-10T04:42:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:14 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:15.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:14 smithi085 conmon[35961]: cluster 2021-02-10T04:42: 2021-02-10T04:42:15.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:14 smithi085 conmon[35961]: 13.542208+0000 osd.4 (osd.4) 1578 : cluster 2021-02-10T04:42:15.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:14 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:15.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:14 smithi085 conmon[35961]: cluster 2021-02-10T04:42:14 2021-02-10T04:42:15.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:14 smithi085 conmon[35961]: .317035+0000 mon.a (mon.0) 1888 : cluster [DBG] 2021-02-10T04:42:15.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:14 smithi085 conmon[35961]: osdmap e915: 8 total, 8 up, 7 in 2021-02-10T04:42:15.318 INFO:tasks.rados.rados.0.smithi085.stdout:3632: done (9 left) 2021-02-10T04:42:15.319 INFO:tasks.rados.rados.0.smithi085.stdout:3633: done (8 left) 2021-02-10T04:42:15.319 INFO:tasks.rados.rados.0.smithi085.stdout:3634: done (7 left) 2021-02-10T04:42:15.319 INFO:tasks.rados.rados.0.smithi085.stdout:3635: done (6 left) 2021-02-10T04:42:15.319 INFO:tasks.rados.rados.0.smithi085.stdout:3636: done (5 left) 2021-02-10T04:42:15.319 INFO:tasks.rados.rados.0.smithi085.stdout:3637: done (4 left) 2021-02-10T04:42:15.319 INFO:tasks.rados.rados.0.smithi085.stdout:3638: delete oid 30 current snap is 420 2021-02-10T04:42:15.324 INFO:tasks.rados.rados.0.smithi085.stdout:3638: done (4 left) 2021-02-10T04:42:15.324 INFO:tasks.rados.rados.0.smithi085.stdout:3639: read oid 30 snap -1 2021-02-10T04:42:15.325 INFO:tasks.rados.rados.0.smithi085.stdout:3639: expect deleted 2021-02-10T04:42:15.325 INFO:tasks.rados.rados.0.smithi085.stdout:3640: read oid 36 snap 406 2021-02-10T04:42:15.325 INFO:tasks.rados.rados.0.smithi085.stdout:3640: expect (ObjNum 1190 snap 406 seq_num 1190) 2021-02-10T04:42:15.325 INFO:tasks.rados.rados.0.smithi085.stdout:3641: write oid 50 current snap is 420 2021-02-10T04:42:15.325 INFO:tasks.rados.rados.0.smithi085.stdout:3641: seq_num 1246 ranges {541522=725793,1946446=635431} 2021-02-10T04:42:15.332 INFO:tasks.rados.rados.0.smithi085.stdout:3641: writing smithi08575397-50 from 541522 to 1267315 tid 1 2021-02-10T04:42:15.341 INFO:tasks.rados.rados.0.smithi085.stdout:3641: writing smithi08575397-50 from 1946446 to 2581877 tid 2 2021-02-10T04:42:15.341 INFO:tasks.rados.rados.0.smithi085.stdout:3642: read oid 23 snap -1 2021-02-10T04:42:15.341 INFO:tasks.rados.rados.0.smithi085.stdout:3642: expect deleted 2021-02-10T04:42:15.341 INFO:tasks.rados.rados.0.smithi085.stdout:3643: write (excl) oid 16 current snap is 420 2021-02-10T04:42:15.341 INFO:tasks.rados.rados.0.smithi085.stdout:3640: expect (ObjNum 1190 snap 406 seq_num 1190) 2021-02-10T04:42:15.454 INFO:tasks.rados.rados.0.smithi085.stdout:3643: seq_num 1247 ranges {558329=517395,1721281=779089} 2021-02-10T04:42:15.460 INFO:tasks.rados.rados.0.smithi085.stdout:3643: writing smithi08575397-16 from 558329 to 1075724 tid 1 2021-02-10T04:42:15.469 INFO:tasks.rados.rados.0.smithi085.stdout:3643: writing smithi08575397-16 from 1721281 to 2500370 tid 2 2021-02-10T04:42:15.470 INFO:tasks.rados.rados.0.smithi085.stdout:3639: done (8 left) 2021-02-10T04:42:15.470 INFO:tasks.rados.rados.0.smithi085.stdout:3640: done (7 left) 2021-02-10T04:42:15.470 INFO:tasks.rados.rados.0.smithi085.stdout:3644: copy_from oid 38 from oid 29 current snap is 420 2021-02-10T04:42:15.470 INFO:tasks.rados.rados.0.smithi085.stdout:3641: finishing write tid 1 to smithi08575397-50 2021-02-10T04:42:15.471 INFO:tasks.rados.rados.0.smithi085.stdout:3641: finishing write tid 2 to smithi08575397-50 2021-02-10T04:42:15.471 INFO:tasks.rados.rados.0.smithi085.stdout:3641: finishing write tid 3 to smithi08575397-50 2021-02-10T04:42:15.471 INFO:tasks.rados.rados.0.smithi085.stdout:3641: finishing write tid 4 to smithi08575397-50 2021-02-10T04:42:15.471 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 1605 (ObjNum 1246 snap 420 seq_num 1246) dirty exists 2021-02-10T04:42:15.471 INFO:tasks.rados.rados.0.smithi085.stdout:3641: left oid 50 (ObjNum 1246 snap 420 seq_num 1246) 2021-02-10T04:42:15.471 INFO:tasks.rados.rados.0.smithi085.stdout:3643: finishing write tid 1 to smithi08575397-16 2021-02-10T04:42:15.471 INFO:tasks.rados.rados.0.smithi085.stdout:3641: done (7 left) 2021-02-10T04:42:15.471 INFO:tasks.rados.rados.0.smithi085.stdout:3642: done (6 left) 2021-02-10T04:42:15.471 INFO:tasks.rados.rados.0.smithi085.stdout:3645: copy_from oid 13 from oid 30 current snap is 420 2021-02-10T04:42:15.471 INFO:tasks.rados.rados.0.smithi085.stdout:3646: write oid 15 current snap is 420 2021-02-10T04:42:15.471 INFO:tasks.rados.rados.0.smithi085.stdout:3646: seq_num 1248 ranges {575136=592644,1663410=639100,2705542=269594} 2021-02-10T04:42:15.477 INFO:tasks.rados.rados.0.smithi085.stdout:3646: writing smithi08575397-15 from 575136 to 1167780 tid 1 2021-02-10T04:42:15.484 INFO:tasks.rados.rados.0.smithi085.stdout:3646: writing smithi08575397-15 from 1663410 to 2302510 tid 2 2021-02-10T04:42:15.489 INFO:tasks.rados.rados.0.smithi085.stdout:3646: writing smithi08575397-15 from 2705542 to 2975136 tid 3 2021-02-10T04:42:15.489 INFO:tasks.rados.rados.0.smithi085.stdout:3645: finishing copy_from to smithi08575397-13 2021-02-10T04:42:15.489 INFO:tasks.rados.rados.0.smithi085.stdout:3645: got expected ENOENT (src dne) 2021-02-10T04:42:15.489 INFO:tasks.rados.rados.0.smithi085.stdout:3647: read oid 14 snap -1 2021-02-10T04:42:15.489 INFO:tasks.rados.rados.0.smithi085.stdout:3647: expect (ObjNum 1108 snap 374 seq_num 1108) 2021-02-10T04:42:15.490 INFO:tasks.rados.rados.0.smithi085.stdout:3648: write oid 2 current snap is 420 2021-02-10T04:42:15.490 INFO:tasks.rados.rados.0.smithi085.stdout:3645: finishing copy_from racing read to smithi08575397-13 2021-02-10T04:42:15.490 INFO:tasks.rados.rados.0.smithi085.stdout:3643: finishing write tid 2 to smithi08575397-16 2021-02-10T04:42:15.490 INFO:tasks.rados.rados.0.smithi085.stdout:3643: finishing write tid 3 to smithi08575397-16 2021-02-10T04:42:15.490 INFO:tasks.rados.rados.0.smithi085.stdout:3643: finishing write tid 4 to smithi08575397-16 2021-02-10T04:42:15.490 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 1882 (ObjNum 1247 snap 420 seq_num 1247) dirty exists 2021-02-10T04:42:15.490 INFO:tasks.rados.rados.0.smithi085.stdout:3643: left oid 16 (ObjNum 1247 snap 420 seq_num 1247) 2021-02-10T04:42:15.490 INFO:tasks.rados.rados.0.smithi085.stdout:3646: finishing write tid 1 to smithi08575397-15 2021-02-10T04:42:15.490 INFO:tasks.rados.rados.0.smithi085.stdout:3648: seq_num 1249 ranges {591943=667893,1721892=782758} 2021-02-10T04:42:15.496 INFO:tasks.rados.rados.0.smithi085.stdout:3648: writing smithi08575397-2 from 591943 to 1259836 tid 1 2021-02-10T04:42:15.504 INFO:tasks.rados.rados.0.smithi085.stdout:3648: writing smithi08575397-2 from 1721892 to 2504650 tid 2 2021-02-10T04:42:15.504 INFO:tasks.rados.rados.0.smithi085.stdout:3643: done (9 left) 2021-02-10T04:42:15.504 INFO:tasks.rados.rados.0.smithi085.stdout:3645: done (8 left) 2021-02-10T04:42:15.504 INFO:tasks.rados.rados.0.smithi085.stdout:3649: snap_create 2021-02-10T04:42:15.504 INFO:tasks.rados.rados.0.smithi085.stdout:3646: finishing write tid 2 to smithi08575397-15 2021-02-10T04:42:15.505 INFO:tasks.rados.rados.0.smithi085.stdout:3647: expect (ObjNum 1108 snap 374 seq_num 1108) 2021-02-10T04:42:15.599 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:42:15.633 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:15.759 INFO:tasks.rados.rados.0.smithi085.stdout:3646: finishing write tid 3 to smithi08575397-15 2021-02-10T04:42:15.760 INFO:tasks.rados.rados.0.smithi085.stdout:3646: finishing write tid 4 to smithi08575397-15 2021-02-10T04:42:15.760 INFO:tasks.rados.rados.0.smithi085.stdout:3646: finishing write tid 5 to smithi08575397-15 2021-02-10T04:42:15.760 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 15 v 1494 (ObjNum 1248 snap 420 seq_num 1248) dirty exists 2021-02-10T04:42:15.760 INFO:tasks.rados.rados.0.smithi085.stdout:3646: left oid 15 (ObjNum 1248 snap 420 seq_num 1248) 2021-02-10T04:42:15.760 INFO:tasks.rados.rados.0.smithi085.stdout:3648: finishing write tid 1 to smithi08575397-2 2021-02-10T04:42:15.760 INFO:tasks.rados.rados.0.smithi085.stdout:3648: finishing write tid 2 to smithi08575397-2 2021-02-10T04:42:15.760 INFO:tasks.rados.rados.0.smithi085.stdout:3648: finishing write tid 3 to smithi08575397-2 2021-02-10T04:42:15.760 INFO:tasks.rados.rados.0.smithi085.stdout:3648: finishing write tid 4 to smithi08575397-2 2021-02-10T04:42:15.760 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 1593 (ObjNum 1249 snap 420 seq_num 1249) dirty exists 2021-02-10T04:42:15.760 INFO:tasks.rados.rados.0.smithi085.stdout:3648: left oid 2 (ObjNum 1249 snap 420 seq_num 1249) 2021-02-10T04:42:15.760 INFO:tasks.rados.rados.0.smithi085.stdout:3644: finishing copy_from to smithi08575397-38 2021-02-10T04:42:15.760 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 1490 (ObjNum 8 snap 0 seq_num 8) dirty exists 2021-02-10T04:42:15.761 INFO:tasks.rados.rados.0.smithi085.stdout:3644: finishing copy_from racing read to smithi08575397-38 2021-02-10T04:42:15.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:15 smithi164 conmon[29288]: cluster 2021-02-10T04:42:15.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:15 smithi164 conmon[29288]: 2021-02-10 2021-02-10T04:42:15.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:15 smithi164 conmon[29288]: T04:42:14.229716+0000 mgr.y (mgr.14140) 854 : cluster [DBG] pgmap v1522: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 728 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 3.3 MiB/s rd, 2.4 MiB/s wr, 13 op/s 2021-02-10T04:42:15.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:15 smithi164 conmon[29288]: cluster 2021-02-10T04:42:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:15 smithi164 conmon[29288]: 2021-02-10T04:42:14.523658+0000 osd.4 (osd.4) 1579 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:15 smithi164 conmon[29288]: 2021-02-10T04:42:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:15 smithi164 conmon[29288]: cluster 2021-02-10T04:42:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:15 smithi164 conmon[29288]: 2021-02-10T04:42:14.523669 2021-02-10T04:42:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:15 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1580 : cluster [WRN] 2021-02-10T04:42:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:15 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:15 smithi164 conmon[29288]: cluster 2021-02-10T04:42:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:15 smithi164 conmon[29288]: 2021-02-10T04:42:14.523676+0000 2021-02-10T04:42:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:15 smithi164 conmon[29288]: osd.4 (osd.4) 1581 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:15 smithi164 conmon[29288]: 2021-02-10T04:42:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:15 smithi164 conmon[29288]: cluster 2021-02-10T04:42:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:15 smithi164 conmon[29288]: 2021-02-10T04:42:14.523682+0000 2021-02-10T04:42:15.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:15 smithi164 conmon[29288]: osd.4 (osd.4) 1582 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:15.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:15 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:42:15.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:15 smithi164 conmon[29288]: 10T04:42:15.317542+0000 mon.a (mon. 2021-02-10T04:42:15.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:15 smithi164 conmon[29288]: 0) 1889 : cluster [DBG] osdmap e916: 8 total, 8 up, 7 in 2021-02-10T04:42:15.896 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:42:15 smithi164 conmon[44544]: debug 2021-02-10T04:42:15.730+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:15.896 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:15 smithi164 conmon[33677]: debug 2021-02-10T04:42:15.896 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:15 smithi164 conmon[33677]: 2021-02-10T04:42:15.493+0000 7fbec19a1700 -1 osd.4 916 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:42:15.896 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:15 smithi164 conmon[33677]: 2021-02-10T04:42:15.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: cluster 2021-02-10T04:42:14.229716+0000 2021-02-10T04:42:15.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: mgr.y (mgr.14140) 854 : cluster [DBG] pgmap v1522: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 728 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 3.3 MiB/s rd, 2.4 MiB/s wr, 13 op/s 2021-02-10T04:42:15.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: cluster 2021-02-10T04:42:15.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 2021 2021-02-10T04:42:15.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: - 2021-02-10T04:42:15.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 02 2021-02-10T04:42:15.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: - 2021-02-10T04:42:15.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 10 2021-02-10T04:42:15.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: T 2021-02-10T04:42:15.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 04 2021-02-10T04:42:15.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: : 2021-02-10T04:42:15.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 42 2021-02-10T04:42:15.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: : 2021-02-10T04:42:15.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 14 2021-02-10T04:42:15.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: . 2021-02-10T04:42:15.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 523658 2021-02-10T04:42:15.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: +0000 2021-02-10T04:42:15.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 2021-02-10T04:42:15.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: osd.4 2021-02-10T04:42:15.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: ( 2021-02-10T04:42:15.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: osd 2021-02-10T04:42:15.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: . 2021-02-10T04:42:15.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 4 2021-02-10T04:42:15.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: ) 2021-02-10T04:42:15.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 1579 2021-02-10T04:42:15.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: : 2021-02-10T04:42:15.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: cluster 2021-02-10T04:42:15.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 2021-02-10T04:42:15.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: [WRN] 2021-02-10T04:42:15.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 2021-02-10T04:42:15.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:15.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 2021-02-10T04:42:15.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: cluster 2021-02-10T04:42:15.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 2021-02-10T04:42:15.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 2021 2021-02-10T04:42:15.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: - 2021-02-10T04:42:15.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 02 2021-02-10T04:42:15.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: -10T04:42:14. 2021-02-10T04:42:15.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 523669+0000 osd.4 (osd.4) 1580 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:15.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:42:15.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 02-10T04:42:14.523676+0000 osd.4 (osd. 2021-02-10T04:42:15.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 4) 1581 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:15.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: cluster 2021-02-10T04:42 2021-02-10T04:42:15.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: :14.523682+0000 osd.4 (osd.4) 1582 : cluster [WRN] 2021-02-10T04:42:15.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:15.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: cluster 2021-02-10T04:42:15.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 2021 2021-02-10T04:42:15.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: - 2021-02-10T04:42:15.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 02 2021-02-10T04:42:15.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: - 2021-02-10T04:42:15.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 10 2021-02-10T04:42:15.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: T 2021-02-10T04:42:15.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 04 2021-02-10T04:42:15.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: : 2021-02-10T04:42:15.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 42 2021-02-10T04:42:15.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: : 2021-02-10T04:42:15.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 15 2021-02-10T04:42:15.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: . 2021-02-10T04:42:15.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 317542 2021-02-10T04:42:15.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: +0000 2021-02-10T04:42:15.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 2021-02-10T04:42:15.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: mon.a 2021-02-10T04:42:15.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: ( 2021-02-10T04:42:15.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: mon 2021-02-10T04:42:15.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: . 2021-02-10T04:42:15.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 0 2021-02-10T04:42:15.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: ) 2021-02-10T04:42:15.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 1889 2021-02-10T04:42:15.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: : 2021-02-10T04:42:15.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: cluster 2021-02-10T04:42:15.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 2021-02-10T04:42:15.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: [DBG] 2021-02-10T04:42:15.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 2021-02-10T04:42:15.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: osdmap e916: 8 total, 8 up, 7 in 2021-02-10T04:42:15.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[29319]: 2021-02-10T04:42:15.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: cluster 2021-02-10T04:42:15.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: 2021-02-10T04:42:15.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: 2021 2021-02-10T04:42:15.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: - 2021-02-10T04:42:15.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: 02 2021-02-10T04:42:15.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: - 2021-02-10T04:42:15.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: 10 2021-02-10T04:42:15.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: T 2021-02-10T04:42:15.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: 04 2021-02-10T04:42:15.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: : 2021-02-10T04:42:15.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: 42 2021-02-10T04:42:15.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: : 2021-02-10T04:42:15.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: 14 2021-02-10T04:42:15.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: . 2021-02-10T04:42:15.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: 229716 2021-02-10T04:42:15.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: +0000 2021-02-10T04:42:15.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: mgr.y (mgr.14140) 854 : cluster 2021-02-10T04:42:15.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: [DBG] pgmap v1522: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 728 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 3.3 MiB/s rd, 2.4 MiB/s wr, 13 op/s 2021-02-10T04:42:15.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: cluster 2021-02-10T04:42:15.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: 2021-02-10T04:42:14.523658+0000 osd.4 (osd.4) 1579 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:15.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: cluster 2021-02-10T04:42:14. 2021-02-10T04:42:15.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: 523669+0000 osd.4 (osd.4) 1580 : cluster [WRN] 2021-02-10T04:42:15.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:15.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: cluster 2021-02-10T04:42: 2021-02-10T04:42:15.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: 14.523676+0000 osd.4 (osd.4) 1581 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:15.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: 2021-02-10T04:42:15.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: cluster 2021-02-10T04:42:15.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: 2021-02-10T04:42:14.523682+0000 osd.4 (osd.4) 1582 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:15.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:42:15.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: -02-10T04:42:15.317542+0000 mon.a (mon.0) 1889 2021-02-10T04:42:15.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:15 smithi085 conmon[35961]: : cluster [DBG] osdmap e916: 8 total, 8 up, 7 in 2021-02-10T04:42:16.141 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:42:16.247 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:42:16.280 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:16.320 INFO:tasks.rados.rados.0.smithi085.stdout:3644: done (8 left) 2021-02-10T04:42:16.321 INFO:tasks.rados.rados.0.smithi085.stdout:3646: done (7 left) 2021-02-10T04:42:16.321 INFO:tasks.rados.rados.0.smithi085.stdout:3647: done (6 left) 2021-02-10T04:42:16.321 INFO:tasks.rados.rados.0.smithi085.stdout:3648: done (5 left) 2021-02-10T04:42:16.321 INFO:tasks.rados.rados.0.smithi085.stdout:3649: done (4 left) 2021-02-10T04:42:16.321 INFO:tasks.rados.rados.0.smithi085.stdout:3650: delete oid 36 current snap is 421 2021-02-10T04:42:16.327 INFO:tasks.rados.rados.0.smithi085.stdout:3650: done (4 left) 2021-02-10T04:42:16.327 INFO:tasks.rados.rados.0.smithi085.stdout:3651: rollback oid 11 current snap is 421 2021-02-10T04:42:16.327 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 11 to 370 2021-02-10T04:42:16.327 INFO:tasks.rados.rados.0.smithi085.stdout:3652: copy_from oid 3 from oid 16 current snap is 421 2021-02-10T04:42:16.327 INFO:tasks.rados.rados.0.smithi085.stdout:3653: read oid 34 snap 401 2021-02-10T04:42:16.327 INFO:tasks.rados.rados.0.smithi085.stdout:3653: expect (ObjNum 1145 snap 385 seq_num 1145) 2021-02-10T04:42:16.327 INFO:tasks.rados.rados.0.smithi085.stdout:3654: rollback oid 4 current snap is 421 2021-02-10T04:42:16.327 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 4 to 380 2021-02-10T04:42:16.327 INFO:tasks.rados.rados.0.smithi085.stdout:3655: copy_from oid 12 from oid 6 current snap is 421 2021-02-10T04:42:16.327 INFO:tasks.rados.rados.0.smithi085.stdout:3656: read oid 40 snap -1 2021-02-10T04:42:16.327 INFO:tasks.rados.rados.0.smithi085.stdout:3656: expect (ObjNum 1191 snap 407 seq_num 1191) 2021-02-10T04:42:16.327 INFO:tasks.rados.rados.0.smithi085.stdout:3657: delete oid 20 current snap is 421 2021-02-10T04:42:16.336 INFO:tasks.rados.rados.0.smithi085.stdout:3657: done (10 left) 2021-02-10T04:42:16.336 INFO:tasks.rados.rados.0.smithi085.stdout:3658: read oid 23 snap -1 2021-02-10T04:42:16.336 INFO:tasks.rados.rados.0.smithi085.stdout:3658: expect deleted 2021-02-10T04:42:16.336 INFO:tasks.rados.rados.0.smithi085.stdout:3659: copy_from oid 9 from oid 42 current snap is 421 2021-02-10T04:42:16.336 INFO:tasks.rados.rados.0.smithi085.stdout:3660: read oid 7 snap -1 2021-02-10T04:42:16.336 INFO:tasks.rados.rados.0.smithi085.stdout:3660: expect (ObjNum 1045 snap 341 seq_num 1045) 2021-02-10T04:42:16.336 INFO:tasks.rados.rados.0.smithi085.stdout:3661: write (excl) oid 44 current snap is 421 2021-02-10T04:42:16.336 INFO:tasks.rados.rados.0.smithi085.stdout:3661: seq_num 1252 ranges {642364=493640,1897338=646438} 2021-02-10T04:42:16.343 INFO:tasks.rados.rados.0.smithi085.stdout:3661: writing smithi08575397-44 from 642364 to 1136004 tid 1 2021-02-10T04:42:16.352 INFO:tasks.rados.rados.0.smithi085.stdout:3661: writing smithi08575397-44 from 1897338 to 2543776 tid 2 2021-02-10T04:42:16.365 INFO:tasks.rados.rados.0.smithi085.stdout:3654: finishing rollback tid 1 to smithi08575397-4 2021-02-10T04:42:16.365 INFO:tasks.rados.rados.0.smithi085.stdout:3654: finishing rollback tid 2 to smithi08575397-4 2021-02-10T04:42:16.365 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 4 v 1376 (ObjNum 1049 snap 344 seq_num 1049) dirty exists 2021-02-10T04:42:16.365 INFO:tasks.rados.rados.0.smithi085.stdout:3660: expect (ObjNum 1045 snap 341 seq_num 1045) 2021-02-10T04:42:16.393 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:42:16 smithi164 conmon[37263]: debug 2021-02-10T04:42:16.110+0000 7f1c6b1ea700 -1 osd.5 pg_epoch: 916 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x5566534bba20) 2021-02-10T04:42:16.601 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:42:16.625 INFO:tasks.rados.rados.0.smithi085.stdout:3653: expect (ObjNum 1145 snap 385 seq_num 1145) 2021-02-10T04:42:16.691 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[52836]: debug 2021-02-10T04:42:16.382+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:16.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:16 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:42:16.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:16 smithi164 conmon[29288]: 42:15.494338+0000 osd.4 (osd.4) 1583 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:16.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:16 smithi164 conmon[29288]: cluster 2021-02-10T04:42:15.494346+0000 2021-02-10T04:42:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:16 smithi164 conmon[29288]: osd.4 (osd.4) 1584 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:16 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:42:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:16 smithi164 conmon[29288]: 42:15.494353+0000 osd.4 (osd.4) 2021-02-10T04:42:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:16 smithi164 conmon[29288]: 1585 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:16 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:42:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:16 smithi164 conmon[29288]: -02-10T04:42:15.494359 2021-02-10T04:42:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:16 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1586 : cluster [WRN] 2021-02-10T04:42:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:16 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:16 smithi164 conmon[29288]: cluster 2021-02-10T04:42: 2021-02-10T04:42:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:16 smithi164 conmon[29288]: 16.319015+0000 mon.a (mon.0) 1890 : cluster 2021-02-10T04:42:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:16 smithi164 conmon[29288]: [DBG] osdmap e917: 8 total, 8 up, 7 in 2021-02-10T04:42:16.895 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:16 smithi164 conmon[33677]: debug 2021-02-10T04:42:16.518+0000 7fbec19a1700 -1 osd.4 917 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:42:16.914 INFO:tasks.rados.rados.0.smithi085.stdout:3656: expect (ObjNum 1191 snap 407 seq_num 1191) 2021-02-10T04:42:16.997 INFO:tasks.rados.rados.0.smithi085.stdout:3651: finishing rollback tid 0 to smithi08575397-11 2021-02-10T04:42:16.997 INFO:tasks.rados.rados.0.smithi085.stdout:3651: finishing rollback tid 1 to smithi08575397-11 2021-02-10T04:42:16.997 INFO:tasks.rados.rados.0.smithi085.stdout:3651: finishing rollback tid 2 to smithi08575397-11 2021-02-10T04:42:16.997 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 1886 (ObjNum 1068 snap 352 seq_num 1068) dirty exists 2021-02-10T04:42:16.998 INFO:tasks.rados.rados.0.smithi085.stdout:3661: finishing write tid 1 to smithi08575397-44 2021-02-10T04:42:16.998 INFO:tasks.rados.rados.0.smithi085.stdout:3651: done (13 left) 2021-02-10T04:42:16.998 INFO:tasks.rados.rados.0.smithi085.stdout:3653: done (12 left) 2021-02-10T04:42:16.998 INFO:tasks.rados.rados.0.smithi085.stdout:3654: done (11 left) 2021-02-10T04:42:16.998 INFO:tasks.rados.rados.0.smithi085.stdout:3656: done (10 left) 2021-02-10T04:42:16.998 INFO:tasks.rados.rados.0.smithi085.stdout:3658: done (9 left) 2021-02-10T04:42:16.998 INFO:tasks.rados.rados.0.smithi085.stdout:3660: done (8 left) 2021-02-10T04:42:16.998 INFO:tasks.rados.rados.0.smithi085.stdout:3662: snap_create 2021-02-10T04:42:16.998 INFO:tasks.rados.rados.0.smithi085.stdout:3655: finishing copy_from to smithi08575397-12 2021-02-10T04:42:16.998 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 1583 (ObjNum 1175 snap 401 seq_num 1175) dirty exists 2021-02-10T04:42:16.998 INFO:tasks.rados.rados.0.smithi085.stdout:3655: finishing copy_from racing read to smithi08575397-12 2021-02-10T04:42:16.998 INFO:tasks.rados.rados.0.smithi085.stdout:3661: finishing write tid 2 to smithi08575397-44 2021-02-10T04:42:16.999 INFO:tasks.rados.rados.0.smithi085.stdout:3661: finishing write tid 3 to smithi08575397-44 2021-02-10T04:42:16.999 INFO:tasks.rados.rados.0.smithi085.stdout:3661: finishing write tid 4 to smithi08575397-44 2021-02-10T04:42:16.999 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 1583 (ObjNum 1252 snap 421 seq_num 1252) dirty exists 2021-02-10T04:42:16.999 INFO:tasks.rados.rados.0.smithi085.stdout:3661: left oid 44 (ObjNum 1252 snap 421 seq_num 1252) 2021-02-10T04:42:16.999 INFO:tasks.rados.rados.0.smithi085.stdout:3652: finishing copy_from to smithi08575397-3 2021-02-10T04:42:16.999 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 1882 (ObjNum 1247 snap 420 seq_num 1247) dirty exists 2021-02-10T04:42:16.999 INFO:tasks.rados.rados.0.smithi085.stdout:3652: finishing copy_from racing read to smithi08575397-3 2021-02-10T04:42:16.999 INFO:tasks.rados.rados.0.smithi085.stdout:3659: finishing copy_from to smithi08575397-9 2021-02-10T04:42:16.999 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 9 v 1804 (ObjNum 1238 snap 417 seq_num 1238) dirty exists 2021-02-10T04:42:16.999 INFO:tasks.rados.rados.0.smithi085.stdout:3659: finishing copy_from racing read to smithi08575397-9 2021-02-10T04:42:17.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: cluster 2021-02-10T04:42:17.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: 2021-02-10T04:42:17.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: 2021 2021-02-10T04:42:17.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: - 2021-02-10T04:42:17.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: 02 2021-02-10T04:42:17.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: - 2021-02-10T04:42:17.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: 10 2021-02-10T04:42:17.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: T 2021-02-10T04:42:17.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: 04 2021-02-10T04:42:17.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: : 2021-02-10T04:42:17.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: 42 2021-02-10T04:42:17.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: : 2021-02-10T04:42:17.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: 15 2021-02-10T04:42:17.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: . 2021-02-10T04:42:17.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: 494338 2021-02-10T04:42:17.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: +0000 2021-02-10T04:42:17.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: 2021-02-10T04:42:17.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: osd.4 2021-02-10T04:42:17.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: ( 2021-02-10T04:42:17.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: osd 2021-02-10T04:42:17.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: . 2021-02-10T04:42:17.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: 4 2021-02-10T04:42:17.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: ) 2021-02-10T04:42:17.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: 1583 2021-02-10T04:42:17.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: : 2021-02-10T04:42:17.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: cluster 2021-02-10T04:42:17.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: 2021-02-10T04:42:17.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: [WRN] 2021-02-10T04:42:17.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:17.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:42:17.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: -10T04:42:15.494346+0000 osd.4 (osd.4) 2021-02-10T04:42:17.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: 1584 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:17.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:42:17.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: 02-10T04:42:15.494353+0000 osd.4 (osd.4) 1585 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:17.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:42:17.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: -10T04:42:15.494359+0000 osd.4 (osd.4 2021-02-10T04:42:17.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: ) 1586 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:17.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:42:17.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: 02-10T04:42:16.319015+0000 mon.a (mon 2021-02-10T04:42:17.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: .0) 2021-02-10T04:42:17.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[35961]: 1890 : cluster [DBG] osdmap e917: 8 total, 8 up, 7 in 2021-02-10T04:42:17.198 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[29319]: cluster 2021-02-10T04:42:15.494338+0000 osd.4 ( 2021-02-10T04:42:17.198 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[29319]: osd.4) 1583 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:17.198 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[29319]: cluster 2021-02-10T04:42:15.494346+0000 2021-02-10T04:42:17.199 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[29319]: osd.4 (osd.4) 1584 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:17.199 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[29319]: cluster 2021-02-10T04:42:17.199 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[29319]: 2021-02-10T04:42:15. 2021-02-10T04:42:17.199 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[29319]: 494353+0000 osd.4 (osd.4) 1585 : cluster [WRN] 2021-02-10T04:42:17.199 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:17.199 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:42:17.199 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[29319]: -10T04:42:15.494359+0000 osd.4 (osd.4) 1586 : 2021-02-10T04:42:17.199 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:17.199 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:42:17.199 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[29319]: -10T04:42:16.319015+0000 mon.a ( 2021-02-10T04:42:17.199 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:16 smithi085 conmon[29319]: mon.0) 1890 : cluster [DBG] osdmap e917: 8 total, 8 up, 7 in 2021-02-10T04:42:17.724 INFO:tasks.rados.rados.0.smithi085.stdout:3652: done (8 left) 2021-02-10T04:42:17.724 INFO:tasks.rados.rados.0.smithi085.stdout:3655: done (7 left) 2021-02-10T04:42:17.724 INFO:tasks.rados.rados.0.smithi085.stdout:3659: done (6 left) 2021-02-10T04:42:17.724 INFO:tasks.rados.rados.0.smithi085.stdout:3661: done (5 left) 2021-02-10T04:42:17.724 INFO:tasks.rados.rados.0.smithi085.stdout:3662: done (4 left) 2021-02-10T04:42:17.724 INFO:tasks.rados.rados.0.smithi085.stdout:3663: rollback oid 8 current snap is 422 2021-02-10T04:42:17.725 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 8 to 401 2021-02-10T04:42:17.725 INFO:tasks.rados.rados.0.smithi085.stdout:3664: write (excl) oid 28 current snap is 422 2021-02-10T04:42:17.725 INFO:tasks.rados.rados.0.smithi085.stdout:3664: seq_num 1253 ranges {659171=568889,1839467=790096} 2021-02-10T04:42:17.731 INFO:tasks.rados.rados.0.smithi085.stdout:3664: writing smithi08575397-28 from 659171 to 1228060 tid 1 2021-02-10T04:42:17.741 INFO:tasks.rados.rados.0.smithi085.stdout:3664: writing smithi08575397-28 from 1839467 to 2629563 tid 2 2021-02-10T04:42:17.741 INFO:tasks.rados.rados.0.smithi085.stdout:3665: copy_from oid 12 from oid 13 current snap is 422 2021-02-10T04:42:17.741 INFO:tasks.rados.rados.0.smithi085.stdout:3663: finishing rollback tid 0 to smithi08575397-8 2021-02-10T04:42:17.741 INFO:tasks.rados.rados.0.smithi085.stdout:3663: finishing rollback tid 1 to smithi08575397-8 2021-02-10T04:42:17.741 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 1607 (ObjNum 1953066355 snap 942696808 seq_num 859125557) dirty dne 2021-02-10T04:42:17.741 INFO:tasks.rados.rados.0.smithi085.stdout:3664: finishing write tid 1 to smithi08575397-28 2021-02-10T04:42:17.741 INFO:tasks.rados.rados.0.smithi085.stdout:3663: done (6 left) 2021-02-10T04:42:17.741 INFO:tasks.rados.rados.0.smithi085.stdout:3666: snap_remove snap 385 2021-02-10T04:42:17.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:17 smithi164 conmon[33677]: debug 2021-02-10T04:42:17.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:17 smithi164 conmon[33677]: 2021-02-10T04:42:17.536+0000 7fbec19a1700 -1 osd.4 917 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:42:17.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:17 smithi164 conmon[33677]: 2021-02-10T04:42:17.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:17 smithi164 conmon[29288]: cluster 2021-02-10T04:42:16.230165+0000 2021-02-10T04:42:17.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:17 smithi164 conmon[29288]: mgr.y (mgr.14140) 855 : cluster [DBG] pgmap v1525: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 724 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 2.6 MiB/s wr, 22 op/s 2021-02-10T04:42:17.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:17 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:42:17.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:17 smithi164 conmon[29288]: 04:42:16.519092+0000 osd.4 (osd.4) 1587 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:17.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:17 smithi164 conmon[29288]: 2021-02-10T04:42:17.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:17 smithi164 conmon[29288]: cluster 2021-02-10T04:42:17.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:17 smithi164 conmon[29288]: 2021-02-10T04:42:16.519098+0000 2021-02-10T04:42:17.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:17 smithi164 conmon[29288]: osd.4 (osd.4) 1588 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:17.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:17 smithi164 conmon[29288]: 2021-02-10T04:42:17.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:17 smithi164 conmon[29288]: cluster 2021-02-10T04:42: 2021-02-10T04:42:17.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:17 smithi164 conmon[29288]: 16.519102+0000 osd.4 (osd.4) 1589 : cluster 2021-02-10T04:42:17.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:17 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:17.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:17 smithi164 conmon[29288]: cluster 2021-02-10T04:42:17.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:17 smithi164 conmon[29288]: 2021-02-10T04:42:16.519107+0000 2021-02-10T04:42:17.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:17 smithi164 conmon[29288]: osd.4 (osd.4) 1590 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:18.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[29319]: cluster 2021-02-10T04:42:18.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[29319]: 2021-02-10T04:42 2021-02-10T04:42:18.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[29319]: :16.230165+0000 mgr.y (mgr.14140) 855 : cluster [DBG] pgmap v1525: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 724 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 2.6 MiB/s wr, 22 op/s 2021-02-10T04:42:18.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[29319]: cluster 2021-02-10T04:42:16.519092+0000 osd.4 ( 2021-02-10T04:42:18.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[29319]: osd.4) 1587 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:18.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:42:18.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[29319]: T04:42:16.519098+0000 osd.4 (osd.4) 1588 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:18.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[29319]: cluster 2021-02-10T04:42:16. 2021-02-10T04:42:18.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[29319]: 519102+0000 osd.4 (osd.4) 1589 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:18.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:42:18.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[29319]: T04:42:16.519107+0000 osd.4 (osd.4) 1590 2021-02-10T04:42:18.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:18.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: cluster 2021-02-10T04:42:18.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: 2021-02-10T04:42:18.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: 2021 2021-02-10T04:42:18.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: - 2021-02-10T04:42:18.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: 02 2021-02-10T04:42:18.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: - 2021-02-10T04:42:18.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: 10 2021-02-10T04:42:18.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: T 2021-02-10T04:42:18.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: 04 2021-02-10T04:42:18.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: : 2021-02-10T04:42:18.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: 42 2021-02-10T04:42:18.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: : 2021-02-10T04:42:18.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: 16 2021-02-10T04:42:18.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: . 2021-02-10T04:42:18.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: 230165 2021-02-10T04:42:18.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: +0000 2021-02-10T04:42:18.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: 2021-02-10T04:42:18.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: mgr.y 2021-02-10T04:42:18.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: ( 2021-02-10T04:42:18.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: mgr 2021-02-10T04:42:18.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: . 2021-02-10T04:42:18.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: 14140 2021-02-10T04:42:18.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: ) 2021-02-10T04:42:18.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: 855 2021-02-10T04:42:18.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: : cluster [DBG] pgmap v1525: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 724 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 2.6 MiB/s wr, 22 op/s 2021-02-10T04:42:18.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: cluster 2021-02-10T04:42:16.519092+0000 osd.4 (osd.4) 1587 : cluster 2021-02-10T04:42:18.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:18.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: cluster 2021-02-10T04:42:16. 2021-02-10T04:42:18.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: 519098+0000 osd.4 (osd.4) 1588 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:18.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:42:18.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: -02-10T04:42: 2021-02-10T04:42:18.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: 16.519102+0000 osd.4 (osd. 2021-02-10T04:42:18.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: 4) 1589 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:18.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:42:18.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: :42:16.519107+0000 osd.4 (osd.4) 2021-02-10T04:42:18.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:17 smithi085 conmon[35961]: 1590 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:18.746 INFO:tasks.rados.rados.0.smithi085.stdout:3664: finishing write tid 2 to smithi08575397-28 2021-02-10T04:42:18.746 INFO:tasks.rados.rados.0.smithi085.stdout:3664: finishing write tid 3 to smithi08575397-28 2021-02-10T04:42:18.746 INFO:tasks.rados.rados.0.smithi085.stdout:3664: finishing write tid 4 to smithi08575397-28 2021-02-10T04:42:18.746 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 1458 (ObjNum 1253 snap 422 seq_num 1253) dirty exists 2021-02-10T04:42:18.747 INFO:tasks.rados.rados.0.smithi085.stdout:3664: left oid 28 (ObjNum 1253 snap 422 seq_num 1253) 2021-02-10T04:42:18.747 INFO:tasks.rados.rados.0.smithi085.stdout:3665: finishing copy_from to smithi08575397-12 2021-02-10T04:42:18.747 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 1565 (ObjNum 1241 snap 418 seq_num 1241) dirty exists 2021-02-10T04:42:18.747 INFO:tasks.rados.rados.0.smithi085.stdout:3665: finishing copy_from racing read to smithi08575397-12 2021-02-10T04:42:18.747 INFO:tasks.rados.rados.0.smithi085.stdout:3664: done (6 left) 2021-02-10T04:42:18.747 INFO:tasks.rados.rados.0.smithi085.stdout:3665: done (5 left) 2021-02-10T04:42:18.747 INFO:tasks.rados.rados.0.smithi085.stdout:3666: done (4 left) 2021-02-10T04:42:18.747 INFO:tasks.rados.rados.0.smithi085.stdout:3667: rollback oid 6 current snap is 422 2021-02-10T04:42:18.747 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 6 to 413 2021-02-10T04:42:18.747 INFO:tasks.rados.rados.0.smithi085.stdout:3668: read oid 37 snap 406 2021-02-10T04:42:18.747 INFO:tasks.rados.rados.0.smithi085.stdout:3668: expect (ObjNum 1129 snap 381 seq_num 1129) 2021-02-10T04:42:18.747 INFO:tasks.rados.rados.0.smithi085.stdout:3669: delete oid 23 current snap is 422 2021-02-10T04:42:18.750 INFO:tasks.rados.rados.0.smithi085.stdout:3669: done (6 left) 2021-02-10T04:42:18.750 INFO:tasks.rados.rados.0.smithi085.stdout:3670: read oid 42 snap -1 2021-02-10T04:42:18.751 INFO:tasks.rados.rados.0.smithi085.stdout:3670: expect (ObjNum 1238 snap 417 seq_num 1238) 2021-02-10T04:42:18.751 INFO:tasks.rados.rados.0.smithi085.stdout:3671: read oid 4 snap -1 2021-02-10T04:42:18.751 INFO:tasks.rados.rados.0.smithi085.stdout:3671: expect (ObjNum 1049 snap 344 seq_num 1049) 2021-02-10T04:42:18.751 INFO:tasks.rados.rados.0.smithi085.stdout:3672: read oid 7 snap 413 2021-02-10T04:42:18.751 INFO:tasks.rados.rados.0.smithi085.stdout:3672: expect (ObjNum 1202 snap 409 seq_num 1202) 2021-02-10T04:42:18.751 INFO:tasks.rados.rados.0.smithi085.stdout:3673: delete oid 8 current snap is 422 2021-02-10T04:42:18.755 INFO:tasks.rados.rados.0.smithi085.stdout:3672: expect (ObjNum 1202 snap 409 seq_num 1202) 2021-02-10T04:42:18.883 INFO:tasks.rados.rados.0.smithi085.stdout:3672: done (9 left) 2021-02-10T04:42:18.883 INFO:tasks.rados.rados.0.smithi085.stdout:3673: done (8 left) 2021-02-10T04:42:18.883 INFO:tasks.rados.rados.0.smithi085.stdout:3674: copy_from oid 43 from oid 5 current snap is 422 2021-02-10T04:42:18.883 INFO:tasks.rados.rados.0.smithi085.stdout:3667: finishing rollback tid 0 to smithi08575397-6 2021-02-10T04:42:18.883 INFO:tasks.rados.rados.0.smithi085.stdout:3667: finishing rollback tid 1 to smithi08575397-6 2021-02-10T04:42:18.883 INFO:tasks.rados.rados.0.smithi085.stdout:3667: finishing rollback tid 2 to smithi08575397-6 2021-02-10T04:42:18.883 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 1601 (ObjNum 1175 snap 401 seq_num 1175) dirty exists 2021-02-10T04:42:18.883 INFO:tasks.rados.rados.0.smithi085.stdout:3671: expect (ObjNum 1049 snap 344 seq_num 1049) 2021-02-10T04:42:18.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:18 smithi164 conmon[33677]: debug 2021-02-10T04:42:18.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:18 smithi164 conmon[33677]: 2021-02-10T04:42:18.547+0000 7fbec19a1700 -1 osd.4 918 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:42:18.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:18 smithi164 conmon[33677]: 2021-02-10T04:42:18.894 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:42:18 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:42:18.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:18 smithi164 conmon[29288]: cluster 2021-02-10T04:42:17.537456+0000 osd.4 ( 2021-02-10T04:42:18.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:18 smithi164 conmon[29288]: osd.4) 1591 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:18 smithi164 conmon[29288]: cluster 2021-02-10T04:42:17.537464+0000 osd.4 (osd.4) 1592 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:18 smithi164 conmon[29288]: cluster 2021-02-10T04:42:17 2021-02-10T04:42:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:18 smithi164 conmon[29288]: .537472+0000 osd.4 (osd.4) 1593 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:18 smithi164 conmon[29288]: 2021-02-10T04:42:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:18 smithi164 conmon[29288]: cluster 2021-02-10T04:42:17 2021-02-10T04:42:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:18 smithi164 conmon[29288]: .537478+0000 osd.4 (osd.4) 1594 : cluster [WRN] 2021-02-10T04:42:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:18 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:18 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:42:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:18 smithi164 conmon[29288]: 42:17.723098+0000 mon.a (mon.0) 1891 2021-02-10T04:42:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:18 smithi164 conmon[29288]: : cluster [DBG] osdmap e918: 8 total, 8 up, 7 in 2021-02-10T04:42:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:18 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:42:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:18 smithi164 conmon[29288]: -10T04:42:18.298450+0000 mon.a (mon 2021-02-10T04:42:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:18 smithi164 conmon[29288]: .0) 1892 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 762 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:42:19.048 INFO:tasks.rados.rados.0.smithi085.stdout:3668: expect (ObjNum 1129 snap 381 seq_num 1129) 2021-02-10T04:42:19.155 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[41540]: debug 2021-02-10T04:42:19.155 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[41540]: 2021-02-10T04:42:19.154+0000 7fb94bb63700 -1 osd.0 pg_epoch: 919 pg[5.5( empty local-lis/les=57/58 n=0 ec=57/57 lis/c=57/57 les/c/f=58/58/0 sis=57) [0,1,4] r=0 lpr=57 crt=0'0 mlcod 0'0 active+clean] scrubber pg(5.5) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x555d327c8b00) 2021-02-10T04:42:19.156 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:18 smithi085 conmon[29319]: cluster 2021-02-10T04:42:17.537456+0000 osd.4 (osd.4) 1591 2021-02-10T04:42:19.156 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:18 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:19.156 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:18 smithi085 conmon[29319]: cluster 2021-02-10T04:42:17.537464+0000 osd.4 (osd.4) 1592 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:19.156 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:18 smithi085 conmon[29319]: cluster 2021-02-10T04:42:17.537472+0000 osd.4 (osd.4) 1593 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:19.156 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:18 smithi085 conmon[29319]: cluster 2021-02-10T04:42:17.537478+0000 2021-02-10T04:42:19.156 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:18 smithi085 conmon[29319]: osd.4 (osd.4) 1594 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:19.156 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:18 smithi085 conmon[29319]: cluster 2021-02-10T04:42:17. 2021-02-10T04:42:19.156 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:18 smithi085 conmon[29319]: 723098+0000 mon.a (mon.0) 1891 : cluster [DBG] osdmap e918: 8 total, 8 up, 7 in 2021-02-10T04:42:19.156 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:18 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:42:19.156 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:18 smithi085 conmon[29319]: -02-10T04:42:18.298450+0000 mon.a (mon. 2021-02-10T04:42:19.156 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:18 smithi085 conmon[29319]: 0) 1892 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 762 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:42:19.157 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:18 smithi085 conmon[35961]: cluster 2021-02-10T04:42:17.537456+0000 osd.4 ( 2021-02-10T04:42:19.157 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:18 smithi085 conmon[35961]: osd.4) 1591 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:19.157 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:18 smithi085 conmon[35961]: cluster 2021-02-10T04:42:17.537464+0000 2021-02-10T04:42:19.157 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:18 smithi085 conmon[35961]: osd.4 (osd.4) 1592 : 2021-02-10T04:42:19.157 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:18 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:19.157 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:18 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:42:19.157 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:18 smithi085 conmon[35961]: -02-10T04:42:17.537472+0000 2021-02-10T04:42:19.157 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:18 smithi085 conmon[35961]: osd.4 (osd.4) 1593 2021-02-10T04:42:19.157 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:18 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:19.157 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:18 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:42:19.157 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:18 smithi085 conmon[35961]: -02-10T04:42:17.537478+0000 osd.4 (osd.4) 1594 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:19.157 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:18 smithi085 conmon[35961]: 2021-02-10T04:42:19.157 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:18 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:42:19.157 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:18 smithi085 conmon[35961]: 04:42:17.723098+0000 mon.a (mon. 2021-02-10T04:42:19.158 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:18 smithi085 conmon[35961]: 0) 1891 : cluster [DBG] osdmap e918: 8 total, 8 up, 7 in 2021-02-10T04:42:19.158 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:18 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:42:19.158 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:18 smithi085 conmon[35961]: -02-10T04:42:18.298450+0000 mon.a 2021-02-10T04:42:19.158 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:18 smithi085 conmon[35961]: (mon.0) 2021-02-10T04:42:19.158 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:18 smithi085 conmon[35961]: 1892 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 762 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:42:19.182 INFO:tasks.rados.rados.0.smithi085.stdout:3670: expect (ObjNum 1238 snap 417 seq_num 1238) 2021-02-10T04:42:19.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:42:19 smithi164 conmon[37263]: debug 2021-02-10T04:42:19.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:42:19 smithi164 conmon[37263]: 2021-02-10T04:42:19.053+0000 7f1c671e2700 -1 osd.5 pg_epoch: 919 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x5566534ea6e0) 2021-02-10T04:42:19.410 INFO:tasks.rados.rados.0.smithi085.stdout:3667: done (8 left) 2021-02-10T04:42:19.411 INFO:tasks.rados.rados.0.smithi085.stdout:3668: done (7 left) 2021-02-10T04:42:19.411 INFO:tasks.rados.rados.0.smithi085.stdout:3670: done (6 left) 2021-02-10T04:42:19.411 INFO:tasks.rados.rados.0.smithi085.stdout:3671: done (5 left) 2021-02-10T04:42:19.411 INFO:tasks.rados.rados.0.smithi085.stdout:3675: read oid 40 snap -1 2021-02-10T04:42:19.411 INFO:tasks.rados.rados.0.smithi085.stdout:3675: expect (ObjNum 1191 snap 407 seq_num 1191) 2021-02-10T04:42:19.411 INFO:tasks.rados.rados.0.smithi085.stdout:3676: copy_from oid 9 from oid 21 current snap is 422 2021-02-10T04:42:19.411 INFO:tasks.rados.rados.0.smithi085.stdout:3677: read oid 50 snap -1 2021-02-10T04:42:19.411 INFO:tasks.rados.rados.0.smithi085.stdout:3677: expect (ObjNum 1246 snap 420 seq_num 1246) 2021-02-10T04:42:19.411 INFO:tasks.rados.rados.0.smithi085.stdout:3678: delete oid 12 current snap is 422 2021-02-10T04:42:19.416 INFO:tasks.rados.rados.0.smithi085.stdout:3678: done (8 left) 2021-02-10T04:42:19.416 INFO:tasks.rados.rados.0.smithi085.stdout:3679: snap_create 2021-02-10T04:42:19.421 INFO:tasks.rados.rados.0.smithi085.stdout:3675: expect (ObjNum 1191 snap 407 seq_num 1191) 2021-02-10T04:42:19.503 INFO:tasks.rados.rados.0.smithi085.stdout:3677: expect (ObjNum 1246 snap 420 seq_num 1246) 2021-02-10T04:42:19.757 INFO:tasks.rados.rados.0.smithi085.stdout:3676: finishing copy_from to smithi08575397-9 2021-02-10T04:42:19.757 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 9 v 1872 (ObjNum 1134 snap 381 seq_num 1134) dirty exists 2021-02-10T04:42:19.757 INFO:tasks.rados.rados.0.smithi085.stdout:3676: finishing copy_from racing read to smithi08575397-9 2021-02-10T04:42:19.757 INFO:tasks.rados.rados.0.smithi085.stdout:3675: done (8 left) 2021-02-10T04:42:19.757 INFO:tasks.rados.rados.0.smithi085.stdout:3676: done (7 left) 2021-02-10T04:42:19.757 INFO:tasks.rados.rados.0.smithi085.stdout:3677: done (6 left) 2021-02-10T04:42:19.758 INFO:tasks.rados.rados.0.smithi085.stdout:3679: done (5 left) 2021-02-10T04:42:19.758 INFO:tasks.rados.rados.0.smithi085.stdout:3680: snap_create 2021-02-10T04:42:19.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:19 smithi164 conmon[33677]: debug 2021-02-10T04:42:19.570+0000 7fbec19a1700 -1 osd.4 919 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:42:19.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:19 smithi164 conmon[29288]: cluster 2021-02-10T04:42:18.230994+0000 mgr.y (mgr.14140) 856 : cluster [DBG] pgmap v1528: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 735 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 20 MiB/s rd, 3.2 MiB/s wr, 21 op/s 2021-02-10T04:42:19.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:19 smithi164 conmon[29288]: audit 2021-02-10T04:42:18.508070+0000 mgr.y (mgr.14140) 857 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:42:19.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:19 smithi164 conmon[29288]: cluster 2021-02-10T04:42:18.548028+0000 osd.4 2021-02-10T04:42:19.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:19 smithi164 conmon[29288]: (osd.4) 1595 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:19.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:19 smithi164 conmon[29288]: cluster 2021-02-10T04:42:19.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:19 smithi164 conmon[29288]: 2021-02-10T04:42: 2021-02-10T04:42:19.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:19 smithi164 conmon[29288]: 18.548039+0000 osd.4 (osd.4) 1596 : 2021-02-10T04:42:19.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:19 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:19.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:19 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:42:19.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:19 smithi164 conmon[29288]: -10T04:42:18.548046+0000 2021-02-10T04:42:19.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:19 smithi164 conmon[29288]: osd.4 (osd.4) 1597 : cluster [WRN] 2021-02-10T04:42:19.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:19 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:19.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:19 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:42:19.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:19 smithi164 conmon[29288]: 04:42:18.548052+0000 osd.4 (osd 2021-02-10T04:42:19.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:19 smithi164 conmon[29288]: .4) 1598 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:19.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:19 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:42:19.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:19 smithi164 conmon[29288]: -02-10T04:42:18 2021-02-10T04:42:19.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:19 smithi164 conmon[29288]: .743771+0000 mon.a (mon.0) 1893 : 2021-02-10T04:42:19.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:19 smithi164 conmon[29288]: cluster [DBG] osdmap e919: 8 total, 8 up, 7 in 2021-02-10T04:42:19.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:42:19.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[29319]: -10T04:42:18.230994+0000 mgr.y (mgr.14140) 856 : cluster [DBG] pgmap v1528: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 735 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 20 MiB/s rd, 3.2 MiB/s wr, 21 op/s 2021-02-10T04:42:19.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[29319]: audit 2021-02-10T04:42:19.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[29319]: 2021-02-10T04:42:18.508070+0000 mgr.y (mgr.14140) 857 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:42:19.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[29319]: cluster 2021-02-10T04:42:18.548028+0000 osd.4 (osd.4) 1595 2021-02-10T04:42:19.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:19.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[29319]: cluster 2021-02-10T04:42:19.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[29319]: 2021-02-10T04:42:19.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[29319]: 2021-02-10T04:42: 2021-02-10T04:42:19.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[29319]: 18.548039+0000 osd.4 (osd.4) 1596 2021-02-10T04:42:19.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:19.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[29319]: cluster 2021-02-10T04:42:18.548046+0000 osd.4 (osd.4 2021-02-10T04:42:19.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[29319]: ) 1597 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:19.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[29319]: cluster 2021-02-10T04:42:19.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[29319]: 2021-02-10T04:42:18. 2021-02-10T04:42:19.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[29319]: 548052+0000 osd.4 (osd.4) 1598 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:19.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[29319]: cluster 2021-02-10T04:42:19.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:42:19.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[29319]: T04:42:18.743771+0000 mon.a (mon.0 2021-02-10T04:42:19.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[29319]: ) 1893 : cluster [DBG] osdmap e919: 8 total, 8 up, 7 in 2021-02-10T04:42:19.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[35961]: cluster 2021-02-10T04:42:18.230994+0000 mgr.y 2021-02-10T04:42:19.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[35961]: (mgr.14140) 856 : cluster [DBG] pgmap v1528: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 735 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 20 MiB/s rd, 3.2 MiB/s wr, 21 op/s 2021-02-10T04:42:19.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[35961]: audit 2021-02-10T04:42:18. 2021-02-10T04:42:19.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[35961]: 508070+0000 mgr.y (mgr.14140) 857 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:42:19.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[35961]: cluster 2021-02-10T04:42:18 2021-02-10T04:42:19.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[35961]: .548028+0000 osd.4 (osd.4) 1595 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:19.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[35961]: 2021-02-10T04:42:19.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:42:19.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[35961]: -02-10T04:42:18.548039+0000 osd.4 (osd.4) 1596 : cluster [WRN] 2021-02-10T04:42:19.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:19.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[35961]: cluster 2021-02-10T04:42:19.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[35961]: 2021-02-10T04:42:18.548046+0000 osd.4 (osd.4) 1597 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:19.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[35961]: cluster 2021-02-10T04:42:19.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[35961]: 2021-02-10T04:42:18.548052+0000 osd.4 (osd.4) 1598 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:19.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[35961]: 2021-02-10T04:42:19.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:42:19.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[35961]: 02-10T04:42:18.743771+0000 mon.a (mon.0) 1893 : cluster [DBG] 2021-02-10T04:42:19.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:19 smithi085 conmon[35961]: osdmap e919: 8 total, 8 up, 7 in 2021-02-10T04:42:20.760 INFO:tasks.rados.rados.0.smithi085.stdout:3680: done (5 left) 2021-02-10T04:42:20.760 INFO:tasks.rados.rados.0.smithi085.stdout:3681: snap_remove snap 414 2021-02-10T04:42:20.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:20 smithi164 conmon[33677]: debug 2021-02-10T04:42:20.545+0000 7fbec19a1700 -1 osd.4 920 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:42:20.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:20 smithi164 conmon[29288]: cluster 2021-02-10T04:42:19.571412+0000 osd.4 (osd.4 2021-02-10T04:42:20.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:20 smithi164 conmon[29288]: ) 1599 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:20.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:20 smithi164 conmon[29288]: cluster 2021-02-10T04:42:19.571423+0000 osd.4 (osd.4) 1600 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:20.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:20 smithi164 conmon[29288]: cluster 2021-02-10T04:42:19.571429+0000 osd.4 (osd.4) 1601 : 2021-02-10T04:42:20.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:20 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:20.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:20 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:42:20.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:20 smithi164 conmon[29288]: 10T04:42:19.571437+0000 osd.4 (osd.4) 1602 : cluster [WRN] 2021-02-10T04:42:20.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:20 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:20.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:20 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:42:20.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:20 smithi164 conmon[29288]: -02-10T04:42:19.756341 2021-02-10T04:42:20.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:20 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1894 : 2021-02-10T04:42:20.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:20 smithi164 conmon[29288]: cluster [DBG] osdmap e920: 8 total, 8 up, 7 in 2021-02-10T04:42:21.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:20 smithi085 conmon[29319]: cluster 2021-02-10T04:42:19.571412+0000 osd.4 (osd.4) 1599 : cluster [WRN] 2021-02-10T04:42:21.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:20 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:21.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:20 smithi085 conmon[29319]: cluster 2021-02-10T04:42:19.571423+0000 osd.4 (osd.4) 1600 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:21.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:20 smithi085 conmon[29319]: cluster 2021-02-10T04:42:19.571429+0000 osd.4 (osd.4) 1601 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:21.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:20 smithi085 conmon[29319]: cluster 2021-02-10T04:42:19.571437+0000 osd.4 (osd.4) 1602 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:21.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:20 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:42:21.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:20 smithi085 conmon[29319]: 42:19.756341+0000 mon.a (mon 2021-02-10T04:42:21.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:20 smithi085 conmon[29319]: .0) 1894 : cluster [DBG] osdmap e920: 8 total, 8 up, 7 in 2021-02-10T04:42:21.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:20 smithi085 conmon[29319]: 2021-02-10T04:42:21.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:20 smithi085 conmon[35961]: cluster 2021-02-10T04:42:19.571412+0000 2021-02-10T04:42:21.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:20 smithi085 conmon[35961]: osd.4 (osd.4) 1599 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:21.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:20 smithi085 conmon[35961]: cluster 2021-02-10T04:42:19.571423+0000 osd.4 (osd.4) 1600 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:21.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:20 smithi085 conmon[35961]: cluster 2021-02-10T04:42:19.571429+0000 osd.4 (osd.4) 1601 : cluster [WRN] 2021-02-10T04:42:21.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:20 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:21.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:20 smithi085 conmon[35961]: cluster 2021-02-10T04:42:19. 2021-02-10T04:42:21.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:20 smithi085 conmon[35961]: 571437+0000 osd.4 (osd.4) 1602 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:21.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:20 smithi085 conmon[35961]: cluster 2021-02-10T04:42:19.756341+0000 mon.a (mon.0) 1894 : cluster [DBG] osdmap e920: 8 total, 8 up, 7 in 2021-02-10T04:42:21.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:20 smithi085 conmon[35961]: 2021-02-10T04:42:21.536 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:42:21.570 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:21.704 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:42:21.758 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:42:21 smithi164 conmon[40892]: debug 2021-02-10T04:42:21.639+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:21.758 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:21 smithi164 conmon[33677]: debug 2021-02-10T04:42:21.758 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:21 smithi164 conmon[33677]: 2021-02-10T04:42:21.505+0000 7fbec19a1700 -1 osd.4 921 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:42:21.758 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:21 smithi164 conmon[33677]: 2021-02-10T04:42:21.769 INFO:tasks.rados.rados.0.smithi085.stdout:3681: done (5 left) 2021-02-10T04:42:21.769 INFO:tasks.rados.rados.0.smithi085.stdout:3682: read oid 37 snap -1 2021-02-10T04:42:21.770 INFO:tasks.rados.rados.0.smithi085.stdout:3682: expect (ObjNum 1129 snap 381 seq_num 1129) 2021-02-10T04:42:21.770 INFO:tasks.rados.rados.0.smithi085.stdout:3683: snap_create 2021-02-10T04:42:21.770 INFO:tasks.rados.rados.0.smithi085.stdout:3674: finishing copy_from to smithi08575397-43 2021-02-10T04:42:21.770 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 1571 (ObjNum 1244 snap 419 seq_num 1244) dirty exists 2021-02-10T04:42:21.770 INFO:tasks.rados.rados.0.smithi085.stdout:3674: finishing copy_from racing read to smithi08575397-43 2021-02-10T04:42:21.778 INFO:tasks.rados.rados.0.smithi085.stdout:3682: expect (ObjNum 1129 snap 381 seq_num 1129) 2021-02-10T04:42:21.808 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:42:21.840 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:21.905 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 7 1 2021-02-10T04:42:21.941 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.3 dump_blocked_ops 2021-02-10T04:42:22.019 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:42:22.094 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[29319]: cluster 2021-02-10T04:42:20.231852+0000 mgr.y (mgr.14140) 858 : cluster [DBG] pgmap v1531: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 759 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 29 MiB/s rd, 6.0 MiB/s wr, 40 op/s 2021-02-10T04:42:22.094 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[29319]: cluster 2021-02-10T04:42:20.546546+0000 osd.4 (osd.4) 1603 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:22.094 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[29319]: cluster 2021-02-10T04:42:20.546556+0000 osd.4 (osd.4) 1604 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:22.094 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[29319]: cluster 2021-02-10T04:42:20.546563+0000 osd.4 (osd.4) 1605 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:22.094 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[29319]: cluster 2021-02-10T04:42:20.546569+0000 osd.4 (osd.4) 1606 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:22.094 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[29319]: cluster 2021-02-10T04:42:20.759617+0000 mon.a (mon.0) 1895 : cluster [DBG] osdmap e921: 8 total, 8 up, 7 in 2021-02-10T04:42:22.095 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[35961]: cluster 2021-02-10T04:42:22.095 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[35961]: 2021-02-10T04:42:22.095 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[35961]: 2021 2021-02-10T04:42:22.095 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[35961]: - 2021-02-10T04:42:22.095 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[35961]: 02 2021-02-10T04:42:22.095 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[35961]: - 2021-02-10T04:42:22.095 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[35961]: 10 2021-02-10T04:42:22.096 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[35961]: T 2021-02-10T04:42:22.096 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[35961]: 04 2021-02-10T04:42:22.096 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[35961]: : 2021-02-10T04:42:22.096 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[35961]: 42 2021-02-10T04:42:22.096 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[35961]: :20.231852+0000 mgr.y (mgr.14140 2021-02-10T04:42:22.096 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[35961]: ) 858 : cluster [DBG] pgmap v1531: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 759 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 29 MiB/s rd, 6.0 MiB/s wr, 40 op/s 2021-02-10T04:42:22.096 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[35961]: cluster 2021-02-10T04:42:20 2021-02-10T04:42:22.096 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[35961]: .546546+0000 osd.4 (osd.4) 1603 : cluster 2021-02-10T04:42:22.096 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:22.096 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[35961]: cluster 2021-02-10T04:42:20 2021-02-10T04:42:22.096 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[35961]: .546556+0000 osd.4 (osd.4) 1604 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:22.096 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:42:22.097 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[35961]: -10T04:42:20.546563+0000 osd.4 (osd. 2021-02-10T04:42:22.097 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[35961]: 4) 1605 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:22.097 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[35961]: cluster 2021-02-10T04:42:20.546569+0000 osd.4 ( 2021-02-10T04:42:22.097 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[35961]: osd.4) 1606 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:22.097 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[35961]: cluster 2021-02-10T04:42:20.759617 2021-02-10T04:42:22.097 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1895 : cluster [DBG] osdmap e921: 8 total, 8 up, 7 in 2021-02-10T04:42:22.097 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[41540]: debug 2021-02-10T04:42:22.097 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[41540]: 2021-02-10T04:42:21.948+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:22.097 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:42:21 smithi085 conmon[41540]: 2021-02-10T04:42:22.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:21 smithi164 conmon[29288]: cluster 2021-02-10T04:42:20. 2021-02-10T04:42:22.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:21 smithi164 conmon[29288]: 231852+0000 mgr.y (mgr.14140) 858 : cluster [DBG] pgmap v1531: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 759 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 29 MiB/s rd, 6.0 MiB/s wr, 40 op/s 2021-02-10T04:42:22.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:21 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:42:22.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:21 smithi164 conmon[29288]: -02-10T04:42:20.546546+0000 osd.4 (osd.4) 1603 : cluster [WRN] 2021-02-10T04:42:22.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:21 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:22.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:21 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:42:22.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:21 smithi164 conmon[29288]: :42:20.546556+0000 osd.4 (osd.4) 1604 2021-02-10T04:42:22.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:21 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:22.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:21 smithi164 conmon[29288]: cluster 2021-02-10T04:42:20.546563+0000 osd.4 (osd.4 2021-02-10T04:42:22.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:21 smithi164 conmon[29288]: ) 1605 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:22.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:21 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:42:22.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:21 smithi164 conmon[29288]: -02-10T04:42:20.546569+0000 osd.4 (osd.4) 1606 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:22.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:21 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:42:22.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:21 smithi164 conmon[29288]: T04:42:20.759617+0000 mon.a (mon.0) 1895 : cluster 2021-02-10T04:42:22.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:21 smithi164 conmon[29288]: [DBG] osdmap e921: 8 total, 8 up, 7 in 2021-02-10T04:42:22.193 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:42:22.263 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:22.342 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:42:22.352 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:42:22.412 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:42:22 smithi085 conmon[49106]: debug 2021-02-10T04:42:22.410+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:22.454 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:42:22.558 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:42:22.591 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:22.721 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:42:22.724 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:42:22 smithi164 conmon[37263]: debug 2021-02-10T04:42:22.682+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:22.724 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:22 smithi164 conmon[33677]: debug 2021-02-10T04:42:22.465+0000 7fbec19a1700 -1 osd.4 922 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:42:22.767 INFO:tasks.rados.rados.0.smithi085.stdout:3674: done (6 left) 2021-02-10T04:42:22.767 INFO:tasks.rados.rados.0.smithi085.stdout:3682: done (5 left) 2021-02-10T04:42:22.767 INFO:tasks.rados.rados.0.smithi085.stdout:3683: done (4 left) 2021-02-10T04:42:22.767 INFO:tasks.rados.rados.0.smithi085.stdout:3684: delete oid 42 current snap is 425 2021-02-10T04:42:22.773 INFO:tasks.rados.rados.0.smithi085.stdout:3684: done (4 left) 2021-02-10T04:42:22.773 INFO:tasks.rados.rados.0.smithi085.stdout:3685: write (excl) oid 45 current snap is 425 2021-02-10T04:42:22.774 INFO:tasks.rados.rados.0.smithi085.stdout:3685: seq_num 1258 ranges {743206=661487,2131877=541092} 2021-02-10T04:42:22.782 INFO:tasks.rados.rados.0.smithi085.stdout:3685: writing smithi08575397-45 from 743206 to 1404693 tid 1 2021-02-10T04:42:22.790 INFO:tasks.rados.rados.0.smithi085.stdout:3685: writing smithi08575397-45 from 2131877 to 2672969 tid 2 2021-02-10T04:42:22.790 INFO:tasks.rados.rados.0.smithi085.stdout:3686: snap_create 2021-02-10T04:42:22.790 INFO:tasks.rados.rados.0.smithi085.stdout:3685: finishing write tid 1 to smithi08575397-45 2021-02-10T04:42:22.793 INFO:tasks.rados.rados.0.smithi085.stdout:3685: finishing write tid 2 to smithi08575397-45 2021-02-10T04:42:22.794 INFO:tasks.rados.rados.0.smithi085.stdout:3685: finishing write tid 3 to smithi08575397-45 2021-02-10T04:42:22.794 INFO:tasks.rados.rados.0.smithi085.stdout:3685: finishing write tid 4 to smithi08575397-45 2021-02-10T04:42:22.794 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 45 v 1495 (ObjNum 1258 snap 425 seq_num 1258) dirty exists 2021-02-10T04:42:22.794 INFO:tasks.rados.rados.0.smithi085.stdout:3685: left oid 45 (ObjNum 1258 snap 425 seq_num 1258) 2021-02-10T04:42:22.826 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:42:22.859 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:22.972 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:42:22.977 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:22 smithi164 conmon[29288]: cluster 2021-02-10T04:42:22.977 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:22 smithi164 conmon[29288]: 2021-02-10T04:42 2021-02-10T04:42:22.977 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:22 smithi164 conmon[29288]: :21.506191+0000 osd.4 (osd.4) 1607 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:22.977 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:22 smithi164 conmon[29288]: cluster 2021-02-10T04:42:21.506203 2021-02-10T04:42:22.977 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:22 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1608 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:22.977 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:22 smithi164 conmon[29288]: 2021-02-10T04:42:22.978 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:22 smithi164 conmon[29288]: cluster 2021-02-10T04:42:21.506211+0000 2021-02-10T04:42:22.978 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:22 smithi164 conmon[29288]: osd.4 (osd.4) 1609 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:22.978 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:22 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:42:22.978 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:22 smithi164 conmon[29288]: 04:42:21.506217+0000 osd.4 (osd.4) 2021-02-10T04:42:22.978 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:22 smithi164 conmon[29288]: 1610 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:22.978 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:22 smithi164 conmon[29288]: cluster 2021-02-10T04:42 2021-02-10T04:42:22.978 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:22 smithi164 conmon[29288]: :21.768887+0000 mon.a (mon.0) 1896 : cluster 2021-02-10T04:42:22.978 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:22 smithi164 conmon[29288]: [DBG] osdmap e922: 8 total, 8 up, 7 in 2021-02-10T04:42:22.978 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:42:22 smithi164 conmon[44544]: debug 2021-02-10T04:42:22.979 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:42:22 smithi164 conmon[44544]: 2021-02-10T04:42:22.938+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:22.979 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:42:22 smithi164 conmon[44544]: 2021-02-10T04:42:23.077 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:42:23.107 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:23.116 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:22 smithi085 conmon[29319]: cluster 2021-02-10T04:42:21.506191 2021-02-10T04:42:23.116 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:22 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1607 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:23.116 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:22 smithi085 conmon[29319]: cluster 2021-02-10T04:42:21.506203+0000 osd.4 (osd.4) 1608 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:23.116 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:22 smithi085 conmon[29319]: cluster 2021-02-10T04:42:21.506211+0000 osd.4 (osd.4) 1609 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:23.116 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:22 smithi085 conmon[29319]: cluster 2021-02-10T04:42:21.506217+0000 osd.4 (osd.4) 1610 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:23.116 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:22 smithi085 conmon[29319]: cluster 2021-02-10T04:42:21. 2021-02-10T04:42:23.116 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:22 smithi085 conmon[29319]: 768887+0000 mon.a (mon.0) 1896 : cluster [DBG] osdmap e922: 8 total, 8 up, 7 in 2021-02-10T04:42:23.116 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:22 smithi085 conmon[35961]: cluster 2021-02-10T04:42:21. 2021-02-10T04:42:23.116 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:22 smithi085 conmon[35961]: 506191+0000 osd.4 (osd.4) 1607 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:23.116 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:22 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:42:23.117 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:22 smithi085 conmon[35961]: 02-10T04:42:21.506203+0000 osd.4 (osd.4) 1608 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:23.117 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:22 smithi085 conmon[35961]: 2021-02-10T04:42:23.117 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:22 smithi085 conmon[35961]: cluster 2021-02-10T04:42 2021-02-10T04:42:23.117 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:22 smithi085 conmon[35961]: :21.506211+0000 osd.4 (osd.4) 1609 2021-02-10T04:42:23.117 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:22 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:23.117 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:22 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:42:23.117 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:22 smithi085 conmon[35961]: 04:42:21.506217+0000 osd.4 (osd.4 2021-02-10T04:42:23.117 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:22 smithi085 conmon[35961]: ) 1610 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:23.117 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:22 smithi085 conmon[35961]: cluster 2021-02-10T04:42:23.117 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:22 smithi085 conmon[35961]: 2021-02-10T04:42:21 2021-02-10T04:42:23.117 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:22 smithi085 conmon[35961]: .768887+0000 mon.a (mon.0) 1896 : cluster 2021-02-10T04:42:23.117 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:22 smithi085 conmon[35961]: [DBG] osdmap e922: 8 total, 8 up, 7 in 2021-02-10T04:42:23.245 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:42:23.250 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:23 smithi164 conmon[33677]: debug 2021-02-10T04:42:23.251 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:23 smithi164 conmon[33677]: 2021-02-10T04:42:23.198+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:23.251 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:23 smithi164 conmon[33677]: 2021-02-10T04:42:23.314 INFO:tasks.rados.rados.0.smithi085.stdout:3685: done (5 left) 2021-02-10T04:42:23.315 INFO:tasks.rados.rados.0.smithi085.stdout:3686: done (4 left) 2021-02-10T04:42:23.315 INFO:tasks.rados.rados.0.smithi085.stdout:3687: snap_create 2021-02-10T04:42:23.351 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:42:23.383 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:23.512 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:42:23.515 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:42:23 smithi164 conmon[40892]: debug 2021-02-10T04:42:23.469+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:23.517 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:23 smithi164 conmon[33677]: debug 2021-02-10T04:42:23.433+0000 7fbec19a1700 -1 osd.4 924 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:42:23.617 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:42:23.642 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:23.762 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:42:23.774 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:42:23 smithi164 conmon[40892]: debug 2021-02-10T04:42:23.719+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:23.774 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:23 smithi164 conmon[29288]: cluster 2021-02-10T04:42:22.233121+0000 mgr.y 2021-02-10T04:42:23.774 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:23 smithi164 conmon[29288]: (mgr.14140) 859 : cluster [DBG] pgmap v1534: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 759 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 18 MiB/s rd, 3.1 MiB/s wr, 28 op/s 2021-02-10T04:42:23.775 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:23 smithi164 conmon[29288]: cluster 2021-02-10T04:42:22.465779+0000 osd.4 (osd.4) 1611 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:23.775 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:23 smithi164 conmon[29288]: cluster 2021-02-10T04:42:22.465787+0000 osd.4 (osd.4 2021-02-10T04:42:23.775 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:23 smithi164 conmon[29288]: ) 1612 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:23.775 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:23 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:42:23.775 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:23 smithi164 conmon[29288]: 10T04:42:22.465792+0000 osd.4 (osd.4 2021-02-10T04:42:23.775 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:23 smithi164 conmon[29288]: ) 1613 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:23.775 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:23 smithi164 conmon[29288]: cluster 2021-02-10T04:42:22. 2021-02-10T04:42:23.775 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:23 smithi164 conmon[29288]: 465796+0000 osd.4 (osd.4) 1614 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:23.775 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:23 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:42:23.775 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:23 smithi164 conmon[29288]: T04:42:22.766794+0000 mon.a (mon.0) 1897 : cluster 2021-02-10T04:42:23.775 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:23 smithi164 conmon[29288]: [DBG] osdmap e923: 8 total, 8 up, 7 in 2021-02-10T04:42:23.775 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:23 smithi164 conmon[29288]: cluster 2021-02-10T04:42:23.300639+0000 2021-02-10T04:42:23.775 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:23 smithi164 conmon[29288]: mon.a (mon.0) 1898 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 772 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:42:23.776 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:23 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:42:23.776 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:23 smithi164 conmon[29288]: -10T04:42:23.313647+0000 mon.a (mon.0) 1899 : cluster [DBG] osdmap e924: 8 total, 8 up, 7 in 2021-02-10T04:42:23.776 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:23 smithi164 conmon[29288]: cluster 2021-02-10T04:42:23.776 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:23 smithi164 conmon[29288]: 2021-02-10T04:42:23.510102+0000 osd.6 (osd.6) 84 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:42:23.776 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:23 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:42:23.776 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:23 smithi164 conmon[29288]: 10T04:42:23.510794+0000 osd.6 (osd.6) 85 : cluster 2021-02-10T04:42:23.776 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:23 smithi164 conmon[29288]: [DBG] purged_snaps scrub ok 2021-02-10T04:42:23.776 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:23 smithi164 conmon[29288]: audit 2021-02-10T04:42:23.758353+0000 mon.c (mon.1) 69 : audit [DBG] 2021-02-10T04:42:23.776 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:23 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/2067440624' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:42:23.796 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:42:23.824 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:42:23.860 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:42:23.867 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:42:23.898 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:42:23.907 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:23.931 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:42:23.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: cluster 2021-02-10T04:42:23.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: 2021-02-10T04:42:23.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: 2021 2021-02-10T04:42:23.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: - 2021-02-10T04:42:23.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: 02 2021-02-10T04:42:23.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: -10T04:42:22 2021-02-10T04:42:23.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: .233121+0000 mgr.y (mgr.14140) 859 : cluster [DBG] pgmap v1534: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 759 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 18 MiB/s rd, 3.1 MiB/s wr, 28 op/s 2021-02-10T04:42:23.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:42:23.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: -10T04:42:22.465779+0000 osd.4 (osd.4 2021-02-10T04:42:23.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: ) 1611 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:23.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:42:23.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: -02-10T04:42:22.465787+0000 2021-02-10T04:42:23.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: osd.4 (osd.4) 1612 : cluster [WRN] 2021-02-10T04:42:23.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:23.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: cluster 2021-02-10T04:42:22 2021-02-10T04:42:23.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: .465792+0000 osd.4 (osd.4) 1613 : 2021-02-10T04:42:23.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:23.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:42:23.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: 04:42:22.465796+0000 osd.4 (osd 2021-02-10T04:42:23.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: .4) 1614 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:23.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:42:23.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: 02-10T04:42:22.766794+0000 2021-02-10T04:42:23.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: mon.a (mon.0) 1897 : cluster [DBG] 2021-02-10T04:42:23.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: osdmap e923: 8 total, 8 up, 7 in 2021-02-10T04:42:23.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: cluster 2021-02-10T04:42 2021-02-10T04:42:23.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: :23.300639+0000 mon.a (mon.0) 2021-02-10T04:42:23.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: 1898 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 772 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:42:23.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:42:23.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: -10T04:42:23.313647+0000 mon.a 2021-02-10T04:42:23.951 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: (mon.0) 1899 : cluster [DBG] osdmap e924: 8 total, 8 up, 7 in 2021-02-10T04:42:23.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: 2021-02-10T04:42:23.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: cluster 2021-02-10T04:42:23.952 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: 2021-02-10T04:42:23.510102+0000 osd.6 2021-02-10T04:42:23.953 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: (osd.6) 84 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:42:23.953 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: 2021-02-10T04:42:23.953 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: cluster 2021-02-10T04:42:23 2021-02-10T04:42:23.953 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: .510794+0000 osd.6 (osd.6) 85 : 2021-02-10T04:42:23.953 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:42:23.953 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: audit 2021-02-10T04:42:23.758353+0000 mon.c 2021-02-10T04:42:23.954 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[29319]: (mon.1) 69 : audit [DBG] from='client.? 172.21.15.85:0/2067440624' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:42:23.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[35961]: cluster 2021-02-10T04:42:22.233121 2021-02-10T04:42:23.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 859 : cluster [DBG] pgmap v1534: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 759 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 18 MiB/s rd, 3.1 MiB/s wr, 28 op/s 2021-02-10T04:42:23.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[35961]: cluster 2021-02-10T04:42:22.465779+0000 osd.4 (osd.4) 1611 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:23.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[35961]: cluster 2021-02-10T04:42:22.465787+0000 osd.4 (osd.4) 1612 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:23.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[35961]: cluster 2021-02-10T04:42:22.465792+0000 osd.4 (osd.4) 1613 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:23.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[35961]: cluster 2021-02-10T04:42:22.465796+0000 osd.4 (osd.4) 1614 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:23.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[35961]: cluster 2021-02-10T04:42:22.766794+0000 mon.a (mon.0) 1897 : cluster [DBG] osdmap e923: 8 total, 8 up, 7 in 2021-02-10T04:42:23.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[35961]: cluster 2021-02-10T04:42:23.300639+0000 mon.a (mon.0) 1898 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 772 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:42:23.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[35961]: cluster 2021-02-10T04:42:23.313647+0000 mon.a (mon.0) 1899 : cluster [DBG] osdmap e924: 8 total, 8 up, 7 in 2021-02-10T04:42:23.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[35961]: cluster 2021-02-10T04:42:23.510102+0000 osd.6 (osd.6) 84 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:42:23.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[35961]: cluster 2021-02-10T04:42:23.510794+0000 osd.6 (osd.6) 85 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:42:23.956 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:23 smithi085 conmon[35961]: audit 2021-02-10T04:42:23.758353+0000 mon.c (mon.1) 69 : audit [DBG] from='client.? 172.21.15.85:0/2067440624' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:42:23.966 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:42:23.994 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:42:24.020 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:42:24.021 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:42:24.022 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:42:24.022 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:42:24.022 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:42:24.022 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:42:24.022 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:42:24.022 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:42:24.022 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:42:24.316 INFO:tasks.rados.rados.0.smithi085.stdout:3687: done (4 left) 2021-02-10T04:42:24.317 INFO:tasks.rados.rados.0.smithi085.stdout:3688: snap_remove snap 416 2021-02-10T04:42:24.327 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is set 2021-02-10T04:42:25.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:24 smithi164 conmon[29288]: cluster 2021-02-10T04:42:23.434420+0000 osd.4 (osd.4) 1615 : cluster 2021-02-10T04:42:25.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:24 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:25.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:24 smithi164 conmon[29288]: cluster 2021-02-10T04:42:23.434429+0000 osd.4 (osd.4) 1616 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:25.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:24 smithi164 conmon[29288]: cluster 2021-02-10T04:42:23.434435+0000 osd.4 2021-02-10T04:42:25.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:24 smithi164 conmon[29288]: (osd.4) 1617 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:25.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:24 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:42:25.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:24 smithi164 conmon[29288]: -10T04:42:23.434440+0000 osd.4 2021-02-10T04:42:25.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:24 smithi164 conmon[29288]: (osd.4) 1618 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:25.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:24 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:42:25.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:24 smithi164 conmon[29288]: 10T04:42:24.058597+0000 mon.a 2021-02-10T04:42:25.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:24 smithi164 conmon[29288]: (mon.0) 1900 : audit [INF] from='client.? 172.21.15.85:0/2202793920' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:42:25.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:24 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:42:25.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:24 smithi164 conmon[29288]: 02-10T04:42: 2021-02-10T04:42:25.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:24 smithi164 conmon[29288]: 24.310791+0000 mon.a (mon. 2021-02-10T04:42:25.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:24 smithi164 conmon[29288]: 0) 1901 : cluster [WRN] 2021-02-10T04:42:25.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:24 smithi164 conmon[29288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:42:25.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:24 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:42:25.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:24 smithi164 conmon[29288]: -10T04:42:24.316974 2021-02-10T04:42:25.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:24 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1902 : audit [INF] from='client.? 172.21.15.85:0/2202793920' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:42:25.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:24 smithi164 conmon[29288]: 2021-02-10T04:42:25.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:24 smithi164 conmon[29288]: cluster 2021-02-10T04:42:25.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:24 smithi164 conmon[29288]: 2021-02-10T04:42:24.317034+0000 mon.a (mon. 2021-02-10T04:42:25.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:24 smithi164 conmon[29288]: 0) 1903 : cluster [DBG] osdmap e925: 8 total, 8 up, 7 in 2021-02-10T04:42:25.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[35961]: cluster 2021-02-10T04:42:23. 2021-02-10T04:42:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[35961]: 434420+0000 osd.4 (osd.4) 1615 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[35961]: cluster 2021-02-10T04:42:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[35961]: 2021-02-10T04:42:23.434429+0000 osd.4 (osd.4 2021-02-10T04:42:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[35961]: ) 1616 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[35961]: cluster 2021-02-10T04:42:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[35961]: 2021-02-10T04:42:23 2021-02-10T04:42:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[35961]: .434435+0000 osd.4 (osd.4) 1617 : 2021-02-10T04:42:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:42:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[35961]: T04:42:23.434440+0000 osd.4 ( 2021-02-10T04:42:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[35961]: osd.4) 1618 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[35961]: 2021-02-10T04:42:25.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[35961]: audit 2021 2021-02-10T04:42:25.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[35961]: -02-10T04:42:24.058597 2021-02-10T04:42:25.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1900 : audit 2021-02-10T04:42:25.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[35961]: [INF] from='client.? 172.21.15.85:0/2202793920' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:42:25.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:42:25.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[35961]: :42:24.310791 2021-02-10T04:42:25.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[35961]: +0000 mon.a ( 2021-02-10T04:42:25.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[35961]: mon.0) 1901 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:42:25.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[35961]: 2021-02-10T04:42:25.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:42:25.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[35961]: :42:24.316974+0000 mon.a (mon.0 2021-02-10T04:42:25.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[35961]: ) 1902 : audit [INF] from='client.? 172.21.15.85:0/2202793920' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:42:25.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:42:25.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[35961]: -02-10T04:42:24.317034+0000 2021-02-10T04:42:25.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[35961]: mon.a (mon.0) 1903 : cluster [DBG] osdmap e925: 8 total, 8 up, 7 in 2021-02-10T04:42:25.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[35961]: 2021-02-10T04:42:25.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[29319]: cluster 2021-02-10T04:42:23.434420+0000 osd.4 (osd 2021-02-10T04:42:25.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[29319]: .4) 1615 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:42:25.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[29319]: cluster 2021-02-10T04:42:25.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[29319]: 2021-02-10T04:42:23.434429+0000 osd.4 (osd 2021-02-10T04:42:25.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[29319]: .4) 1616 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:42:25.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[29319]: cluster 2021-02-10T04:42:25.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:42:25.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[29319]: 10T04:42:23.434435+0000 osd.4 (osd.4) 1617 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:42:25.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[29319]: cluster 2021-02-10T04:42:25.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[29319]: 2021-02-10T04:42:25.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[29319]: 2021 2021-02-10T04:42:25.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[29319]: -02-10T04:42: 2021-02-10T04:42:25.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[29319]: 23.434440+0000 osd.4 (osd.4) 1618 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:42:25.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[29319]: audit 2021-02-10T04:42:25.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:42:25.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[29319]: T04:42:24.058597+0000 mon.a (mon.0 2021-02-10T04:42:25.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[29319]: ) 1900 : audit [INF] from='client.? 172.21.15.85:0/2202793920' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:42:25.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:42:25.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[29319]: 02-10T04:42:24.310791+0000 2021-02-10T04:42:25.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[29319]: mon.a (mon.0) 1901 : cluster [WRN] 2021-02-10T04:42:25.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[29319]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:42:25.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:42:25.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[29319]: 02-10T04:42:24.316974+0000 2021-02-10T04:42:25.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[29319]: mon.a (mon.0) 1902 : audit [INF] from='client.? 172.21.15.85:0/2202793920' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:42:25.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[29319]: 2021-02-10T04:42:25.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:42:25.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[29319]: 02-10T04:42:24.317034+0000 2021-02-10T04:42:25.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[29319]: mon.a (mon.0) 1903 : cluster [DBG] 2021-02-10T04:42:25.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:24 smithi085 conmon[29319]: osdmap e925: 8 total, 8 up, 7 in 2021-02-10T04:42:25.323 INFO:tasks.rados.rados.0.smithi085.stdout:3688: done (4 left) 2021-02-10T04:42:25.323 INFO:tasks.rados.rados.0.smithi085.stdout:3689: read oid 34 snap -1 2021-02-10T04:42:25.323 INFO:tasks.rados.rados.0.smithi085.stdout:3689: expect (ObjNum 1145 snap 385 seq_num 1145) 2021-02-10T04:42:25.323 INFO:tasks.rados.rados.0.smithi085.stdout:3690: read oid 5 snap -1 2021-02-10T04:42:25.323 INFO:tasks.rados.rados.0.smithi085.stdout:3690: expect (ObjNum 1244 snap 419 seq_num 1244) 2021-02-10T04:42:25.324 INFO:tasks.rados.rados.0.smithi085.stdout:3691: copy_from oid 14 from oid 32 current snap is 427 2021-02-10T04:42:25.324 INFO:tasks.rados.rados.0.smithi085.stdout:3692: read oid 13 snap 386 2021-02-10T04:42:25.324 INFO:tasks.rados.rados.0.smithi085.stdout:3692: expect deleted 2021-02-10T04:42:25.324 INFO:tasks.rados.rados.0.smithi085.stdout:3693: write (excl) oid 10 current snap is 427 2021-02-10T04:42:25.324 INFO:tasks.rados.rados.0.smithi085.stdout:3693: seq_num 1259 ranges {760013=736736,2190359=401103} 2021-02-10T04:42:25.326 INFO:tasks.rados.rados.0.smithi085.stdout:3693: writing smithi08575397-10 from 760013 to 1496749 tid 1 2021-02-10T04:42:25.334 INFO:tasks.rados.rados.0.smithi085.stdout:3693: writing smithi08575397-10 from 2190359 to 2591462 tid 2 2021-02-10T04:42:25.339 INFO:tasks.rados.rados.0.smithi085.stdout:3689: expect (ObjNum 1145 snap 385 seq_num 1145) 2021-02-10T04:42:25.612 INFO:tasks.rados.rados.0.smithi085.stdout:3689: done (8 left) 2021-02-10T04:42:25.612 INFO:tasks.rados.rados.0.smithi085.stdout:3694: delete oid 28 current snap is 427 2021-02-10T04:42:25.613 INFO:tasks.rados.rados.0.smithi085.stdout:3693: finishing write tid 1 to smithi08575397-10 2021-02-10T04:42:25.613 INFO:tasks.rados.rados.0.smithi085.stdout:3690: expect (ObjNum 1244 snap 419 seq_num 1244) 2021-02-10T04:42:25.736 INFO:tasks.rados.rados.0.smithi085.stdout:3690: done (8 left) 2021-02-10T04:42:25.736 INFO:tasks.rados.rados.0.smithi085.stdout:3694: done (7 left) 2021-02-10T04:42:25.736 INFO:tasks.rados.rados.0.smithi085.stdout:3695: write oid 48 current snap is 427 2021-02-10T04:42:25.736 INFO:tasks.rados.rados.0.smithi085.stdout:3693: finishing write tid 2 to smithi08575397-10 2021-02-10T04:42:25.736 INFO:tasks.rados.rados.0.smithi085.stdout:3695: seq_num 1261 ranges {793627=487234,1790970=404772,2829335=364292} 2021-02-10T04:42:25.743 INFO:tasks.rados.rados.0.smithi085.stdout:3695: writing smithi08575397-48 from 793627 to 1280861 tid 1 2021-02-10T04:42:25.748 INFO:tasks.rados.rados.0.smithi085.stdout:3695: writing smithi08575397-48 from 1790970 to 2195742 tid 2 2021-02-10T04:42:25.755 INFO:tasks.rados.rados.0.smithi085.stdout:3695: writing smithi08575397-48 from 2829335 to 3193627 tid 3 2021-02-10T04:42:25.755 INFO:tasks.rados.rados.0.smithi085.stdout:3692: done (7 left) 2021-02-10T04:42:25.755 INFO:tasks.rados.rados.0.smithi085.stdout:3696: read oid 20 snap 424 2021-02-10T04:42:25.755 INFO:tasks.rados.rados.0.smithi085.stdout:3696: expect deleted 2021-02-10T04:42:25.755 INFO:tasks.rados.rados.0.smithi085.stdout:3697: delete oid 45 current snap is 427 2021-02-10T04:42:25.755 INFO:tasks.rados.rados.0.smithi085.stdout:3693: finishing write tid 3 to smithi08575397-10 2021-02-10T04:42:25.755 INFO:tasks.rados.rados.0.smithi085.stdout:3693: finishing write tid 4 to smithi08575397-10 2021-02-10T04:42:25.755 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 10 v 1596 (ObjNum 1259 snap 427 seq_num 1259) dirty exists 2021-02-10T04:42:25.755 INFO:tasks.rados.rados.0.smithi085.stdout:3693: left oid 10 (ObjNum 1259 snap 427 seq_num 1259) 2021-02-10T04:42:25.755 INFO:tasks.rados.rados.0.smithi085.stdout:3691: finishing copy_from to smithi08575397-14 2021-02-10T04:42:25.756 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 1488 (ObjNum 1227 snap 415 seq_num 1227) dirty exists 2021-02-10T04:42:25.756 INFO:tasks.rados.rados.0.smithi085.stdout:3691: finishing copy_from racing read to smithi08575397-14 2021-02-10T04:42:25.756 INFO:tasks.rados.rados.0.smithi085.stdout:3695: finishing write tid 1 to smithi08575397-48 2021-02-10T04:42:25.756 INFO:tasks.rados.rados.0.smithi085.stdout:3695: finishing write tid 2 to smithi08575397-48 2021-02-10T04:42:25.760 INFO:tasks.rados.rados.0.smithi085.stdout:3695: finishing write tid 3 to smithi08575397-48 2021-02-10T04:42:25.760 INFO:tasks.rados.rados.0.smithi085.stdout:3695: finishing write tid 4 to smithi08575397-48 2021-02-10T04:42:25.761 INFO:tasks.rados.rados.0.smithi085.stdout:3695: finishing write tid 5 to smithi08575397-48 2021-02-10T04:42:25.761 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 1877 (ObjNum 1261 snap 427 seq_num 1261) dirty exists 2021-02-10T04:42:25.761 INFO:tasks.rados.rados.0.smithi085.stdout:3695: left oid 48 (ObjNum 1261 snap 427 seq_num 1261) 2021-02-10T04:42:25.762 INFO:tasks.rados.rados.0.smithi085.stdout:3691: done (8 left) 2021-02-10T04:42:25.762 INFO:tasks.rados.rados.0.smithi085.stdout:3693: done (7 left) 2021-02-10T04:42:25.762 INFO:tasks.rados.rados.0.smithi085.stdout:3695: done (6 left) 2021-02-10T04:42:25.762 INFO:tasks.rados.rados.0.smithi085.stdout:3696: done (5 left) 2021-02-10T04:42:25.762 INFO:tasks.rados.rados.0.smithi085.stdout:3697: done (4 left) 2021-02-10T04:42:25.763 INFO:tasks.rados.rados.0.smithi085.stdout:3698: delete oid 26 current snap is 427 2021-02-10T04:42:25.764 INFO:tasks.rados.rados.0.smithi085.stdout:3698: done (4 left) 2021-02-10T04:42:25.764 INFO:tasks.rados.rados.0.smithi085.stdout:3699: read oid 12 snap -1 2021-02-10T04:42:25.764 INFO:tasks.rados.rados.0.smithi085.stdout:3699: expect deleted 2021-02-10T04:42:25.764 INFO:tasks.rados.rados.0.smithi085.stdout:3700: write oid 23 current snap is 427 2021-02-10T04:42:25.765 INFO:tasks.rados.rados.0.smithi085.stdout:3700: seq_num 1264 ranges {444048=429334,1282769=552099,2627957=616091} 2021-02-10T04:42:25.769 INFO:tasks.rados.rados.0.smithi085.stdout:3700: writing smithi08575397-23 from 444048 to 873382 tid 1 2021-02-10T04:42:25.775 INFO:tasks.rados.rados.0.smithi085.stdout:3700: writing smithi08575397-23 from 1282769 to 1834868 tid 2 2021-02-10T04:42:25.784 INFO:tasks.rados.rados.0.smithi085.stdout:3700: writing smithi08575397-23 from 2627957 to 3244048 tid 3 2021-02-10T04:42:25.784 INFO:tasks.rados.rados.0.smithi085.stdout:3700: finishing write tid 1 to smithi08575397-23 2021-02-10T04:42:25.784 INFO:tasks.rados.rados.0.smithi085.stdout:3700: finishing write tid 2 to smithi08575397-23 2021-02-10T04:42:25.784 INFO:tasks.rados.rados.0.smithi085.stdout:3699: done (5 left) 2021-02-10T04:42:25.784 INFO:tasks.rados.rados.0.smithi085.stdout:3701: delete oid 8 current snap is 427 2021-02-10T04:42:25.788 INFO:tasks.rados.rados.0.smithi085.stdout:3700: finishing write tid 3 to smithi08575397-23 2021-02-10T04:42:25.788 INFO:tasks.rados.rados.0.smithi085.stdout:3700: finishing write tid 4 to smithi08575397-23 2021-02-10T04:42:25.788 INFO:tasks.rados.rados.0.smithi085.stdout:3701: done (5 left) 2021-02-10T04:42:25.788 INFO:tasks.rados.rados.0.smithi085.stdout:3702: copy_from oid 45 from oid 36 current snap is 427 2021-02-10T04:42:25.788 INFO:tasks.rados.rados.0.smithi085.stdout:3703: snap_create 2021-02-10T04:42:25.788 INFO:tasks.rados.rados.0.smithi085.stdout:3700: finishing write tid 5 to smithi08575397-23 2021-02-10T04:42:25.788 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 1365 (ObjNum 1264 snap 427 seq_num 1264) dirty exists 2021-02-10T04:42:25.788 INFO:tasks.rados.rados.0.smithi085.stdout:3700: left oid 23 (ObjNum 1264 snap 427 seq_num 1264) 2021-02-10T04:42:25.789 INFO:tasks.rados.rados.0.smithi085.stdout:3702: finishing copy_from to smithi08575397-45 2021-02-10T04:42:25.789 INFO:tasks.rados.rados.0.smithi085.stdout:3702: got expected ENOENT (src dne) 2021-02-10T04:42:25.789 INFO:tasks.rados.rados.0.smithi085.stdout:3702: finishing copy_from racing read to smithi08575397-45 2021-02-10T04:42:26.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:25 smithi164 conmon[29288]: cluster 2021-02-10T04:42:24.233792 2021-02-10T04:42:26.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:25 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 860 : cluster [DBG] pgmap v1537: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 762 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 1.0 MiB/s wr, 13 op/s 2021-02-10T04:42:26.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:25 smithi164 conmon[29288]: cluster 2021-02-10T04:42:26.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:25 smithi164 conmon[29288]: 2021-02-10T04:42:25.316867+0000 mon.a (mon.0) 1904 : cluster [DBG] 2021-02-10T04:42:26.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:25 smithi164 conmon[29288]: osdmap e926: 8 total, 8 up, 7 in 2021-02-10T04:42:26.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:25 smithi085 conmon[29319]: cluster 2021-02-10T04:42:24.233792 2021-02-10T04:42:26.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:25 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 860 : cluster [DBG] pgmap v1537: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 762 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 1.0 MiB/s wr, 13 op/s 2021-02-10T04:42:26.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:25 smithi085 conmon[29319]: cluster 2021-02-10T04:42:25.316867+0000 mon.a (mon.0) 1904 : cluster [DBG] osdmap e926: 8 total, 8 up, 7 in 2021-02-10T04:42:26.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:25 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:42:26.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:25 smithi085 conmon[35961]: 04:42:24.233792+0000 mgr.y (mgr.14140) 860 : cluster [DBG] pgmap v1537: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 762 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 1.0 MiB/s wr, 13 op/s 2021-02-10T04:42:26.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:25 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:42:26.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:25 smithi085 conmon[35961]: 02-10T04:42:25.316867+0000 mon.a ( 2021-02-10T04:42:26.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:25 smithi085 conmon[35961]: mon.0) 1904 : cluster [DBG] osdmap e926: 8 total, 8 up, 7 in 2021-02-10T04:42:26.791 INFO:tasks.rados.rados.0.smithi085.stdout:3700: done (6 left) 2021-02-10T04:42:26.791 INFO:tasks.rados.rados.0.smithi085.stdout:3702: done (5 left) 2021-02-10T04:42:26.791 INFO:tasks.rados.rados.0.smithi085.stdout:3703: done (4 left) 2021-02-10T04:42:26.791 INFO:tasks.rados.rados.0.smithi085.stdout:3704: delete oid 1 current snap is 428 2021-02-10T04:42:26.796 INFO:tasks.rados.rados.0.smithi085.stdout:3704: done (4 left) 2021-02-10T04:42:26.796 INFO:tasks.rados.rados.0.smithi085.stdout:3705: snap_remove snap 372 2021-02-10T04:42:26.933 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:42:26.941 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:42:26 smithi085 conmon[45443]: debug 2021-02-10T04:42:26.941 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:42:26 smithi085 conmon[45443]: 2021-02-10T04:42:26.451+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:26.941 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:42:26 smithi085 conmon[45443]: 2021-02-10T04:42:27.804 INFO:tasks.rados.rados.0.smithi085.stdout:3705: done (4 left) 2021-02-10T04:42:27.805 INFO:tasks.rados.rados.0.smithi085.stdout:3706: snap_remove snap 422 2021-02-10T04:42:28.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:27 smithi164 conmon[29288]: cluster 2021-02-10T04:42:26.234334+0000 mgr.y (mgr.14140) 861 : 2021-02-10T04:42:28.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:27 smithi164 conmon[29288]: cluster [DBG] pgmap v1540: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 761 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 1.7 MiB/s wr, 13 op/s 2021-02-10T04:42:28.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:27 smithi164 conmon[29288]: cluster 2021-02-10T04:42:26.790831+0000 mon.a (mon.0) 1905 : cluster [DBG] osdmap e927: 8 total, 8 up, 7 in 2021-02-10T04:42:28.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:27 smithi085 conmon[29319]: cluster 2021-02-10T04:42:28.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:27 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:42:28.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:27 smithi085 conmon[29319]: 04:42:26.234334+0000 mgr.y (mgr.14140) 861 : cluster [DBG] pgmap v1540: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 761 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 1.7 MiB/s wr, 13 op/s 2021-02-10T04:42:28.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:27 smithi085 conmon[29319]: cluster 2021-02-10T04:42:26 2021-02-10T04:42:28.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:27 smithi085 conmon[29319]: .790831+0000 mon.a (mon.0) 1905 : cluster [DBG] osdmap e927: 8 total, 8 up, 7 in 2021-02-10T04:42:28.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:27 smithi085 conmon[35961]: cluster 2021-02-10T04:42:28.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:27 smithi085 conmon[35961]: 2021-02-10T04:42:28.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:27 smithi085 conmon[35961]: 2021 2021-02-10T04:42:28.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:27 smithi085 conmon[35961]: - 2021-02-10T04:42:28.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:27 smithi085 conmon[35961]: 02-10T04:42 2021-02-10T04:42:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:27 smithi085 conmon[35961]: :26.234334+0000 mgr.y (mgr.14140) 861 : cluster [DBG] pgmap v1540: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 761 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 1.7 MiB/s wr, 13 op/s 2021-02-10T04:42:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:27 smithi085 conmon[35961]: cluster 2021-02-10T04:42:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:27 smithi085 conmon[35961]: 2021-02 2021-02-10T04:42:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:27 smithi085 conmon[35961]: -10T04:42:26.790831+0000 mon.a (mon.0 2021-02-10T04:42:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:27 smithi085 conmon[35961]: ) 1905 : cluster [DBG] osdmap e927: 8 total, 8 up, 7 in 2021-02-10T04:42:28.316 INFO:tasks.rados.rados.0.smithi085.stdout:3706: done (4 left) 2021-02-10T04:42:28.317 INFO:tasks.rados.rados.0.smithi085.stdout:3707: delete oid 36 current snap is 428 2021-02-10T04:42:28.329 INFO:tasks.rados.rados.0.smithi085.stdout:3707: done (4 left) 2021-02-10T04:42:28.330 INFO:tasks.rados.rados.0.smithi085.stdout:3708: read oid 9 snap -1 2021-02-10T04:42:28.330 INFO:tasks.rados.rados.0.smithi085.stdout:3708: expect (ObjNum 1134 snap 381 seq_num 1134) 2021-02-10T04:42:28.330 INFO:tasks.rados.rados.0.smithi085.stdout:3709: copy_from oid 18 from oid 6 current snap is 428 2021-02-10T04:42:28.330 INFO:tasks.rados.rados.0.smithi085.stdout:3710: copy_from oid 1 from oid 49 current snap is 428 2021-02-10T04:42:28.330 INFO:tasks.rados.rados.0.smithi085.stdout:3711: write oid 25 current snap is 428 2021-02-10T04:42:28.330 INFO:tasks.rados.rados.0.smithi085.stdout:3711: seq_num 1268 ranges {511276=730330,1800344=559437,3021296=289980} 2021-02-10T04:42:28.337 INFO:tasks.rados.rados.0.smithi085.stdout:3711: writing smithi08575397-25 from 511276 to 1241606 tid 1 2021-02-10T04:42:28.344 INFO:tasks.rados.rados.0.smithi085.stdout:3711: writing smithi08575397-25 from 1800344 to 2359781 tid 2 2021-02-10T04:42:28.351 INFO:tasks.rados.rados.0.smithi085.stdout:3711: writing smithi08575397-25 from 3021296 to 3311276 tid 3 2021-02-10T04:42:28.352 INFO:tasks.rados.rados.0.smithi085.stdout:3712: copy_from oid 44 from oid 3 current snap is 428 2021-02-10T04:42:28.352 INFO:tasks.rados.rados.0.smithi085.stdout:3708: expect (ObjNum 1134 snap 381 seq_num 1134) 2021-02-10T04:42:28.533 INFO:tasks.rados.rados.0.smithi085.stdout:3708: done (8 left) 2021-02-10T04:42:28.533 INFO:tasks.rados.rados.0.smithi085.stdout:3713: copy_from oid 40 from oid 32 current snap is 428 2021-02-10T04:42:28.533 INFO:tasks.rados.rados.0.smithi085.stdout:3711: finishing write tid 1 to smithi08575397-25 2021-02-10T04:42:28.533 INFO:tasks.rados.rados.0.smithi085.stdout:3711: finishing write tid 2 to smithi08575397-25 2021-02-10T04:42:28.533 INFO:tasks.rados.rados.0.smithi085.stdout:3711: finishing write tid 3 to smithi08575397-25 2021-02-10T04:42:28.533 INFO:tasks.rados.rados.0.smithi085.stdout:3711: finishing write tid 4 to smithi08575397-25 2021-02-10T04:42:28.533 INFO:tasks.rados.rados.0.smithi085.stdout:3711: finishing write tid 5 to smithi08575397-25 2021-02-10T04:42:28.533 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 25 v 1488 (ObjNum 1268 snap 428 seq_num 1268) dirty exists 2021-02-10T04:42:28.534 INFO:tasks.rados.rados.0.smithi085.stdout:3711: left oid 25 (ObjNum 1268 snap 428 seq_num 1268) 2021-02-10T04:42:28.534 INFO:tasks.rados.rados.0.smithi085.stdout:3709: finishing copy_from to smithi08575397-18 2021-02-10T04:42:28.534 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 18 v 1601 (ObjNum 1175 snap 401 seq_num 1175) dirty exists 2021-02-10T04:42:28.534 INFO:tasks.rados.rados.0.smithi085.stdout:3710: finishing copy_from to smithi08575397-1 2021-02-10T04:42:28.534 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 1590 (ObjNum 1097 snap 368 seq_num 1097) dirty exists 2021-02-10T04:42:28.534 INFO:tasks.rados.rados.0.smithi085.stdout:3709: finishing copy_from racing read to smithi08575397-18 2021-02-10T04:42:28.534 INFO:tasks.rados.rados.0.smithi085.stdout:3710: finishing copy_from racing read to smithi08575397-1 2021-02-10T04:42:28.534 INFO:tasks.rados.rados.0.smithi085.stdout:3712: finishing copy_from to smithi08575397-44 2021-02-10T04:42:28.534 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 1882 (ObjNum 1247 snap 420 seq_num 1247) dirty exists 2021-02-10T04:42:28.534 INFO:tasks.rados.rados.0.smithi085.stdout:3712: finishing copy_from racing read to smithi08575397-44 2021-02-10T04:42:28.534 INFO:tasks.rados.rados.0.smithi085.stdout:3709: done (8 left) 2021-02-10T04:42:28.534 INFO:tasks.rados.rados.0.smithi085.stdout:3710: done (7 left) 2021-02-10T04:42:28.534 INFO:tasks.rados.rados.0.smithi085.stdout:3711: done (6 left) 2021-02-10T04:42:28.535 INFO:tasks.rados.rados.0.smithi085.stdout:3712: done (5 left) 2021-02-10T04:42:28.535 INFO:tasks.rados.rados.0.smithi085.stdout:3714: snap_remove snap 417 2021-02-10T04:42:28.800 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:42:28 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:42:28.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:28 smithi164 conmon[29288]: cluster 2021-02-10T04:42:28.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:28 smithi164 conmon[29288]: 2021-02-10T04:42:28.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:28 smithi164 conmon[29288]: 2021 2021-02-10T04:42:28.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:28 smithi164 conmon[29288]: - 2021-02-10T04:42:28.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:28 smithi164 conmon[29288]: 02 2021-02-10T04:42:28.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:28 smithi164 conmon[29288]: - 2021-02-10T04:42:28.800 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:28 smithi164 conmon[29288]: 10 2021-02-10T04:42:28.801 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:28 smithi164 conmon[29288]: T 2021-02-10T04:42:28.801 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:28 smithi164 conmon[29288]: 04:42:27.801530+0000 mon.a (mon.0) 1906 : 2021-02-10T04:42:28.801 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:28 smithi164 conmon[29288]: cluster [DBG] osdmap e928: 8 total, 8 up, 7 in 2021-02-10T04:42:28.801 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:28 smithi164 conmon[29288]: cluster 2021-02-10T04:42:28.801 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:28 smithi164 conmon[29288]: 2021-02-10T04:42:28.315417+0000 mon.a (mon.0) 2021-02-10T04:42:28.801 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:28 smithi164 conmon[29288]: 1907 : cluster [DBG] osdmap e929: 8 total, 8 up, 7 in 2021-02-10T04:42:29.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:28 smithi085 conmon[35961]: cluster 2021-02-10T04:42:27.801530+0000 2021-02-10T04:42:30.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:28 smithi085 conmon[35961]: mon.a (mon.0) 1906 : cluster [DBG] osdmap e928: 8 total, 8 up, 7 in 2021-02-10T04:42:30.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:28 smithi085 conmon[35961]: cluster 2021-02-10T04:42: 2021-02-10T04:42:30.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:28 smithi085 conmon[35961]: 28.315417+0000 mon.a (mon.0) 1907 : cluster [DBG] osdmap e929: 8 total, 8 up, 7 in 2021-02-10T04:42:30.119 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:28 smithi085 conmon[29319]: cluster 2021-02-10T04:42:27.801530+0000 mon.a ( 2021-02-10T04:42:30.119 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:28 smithi085 conmon[29319]: mon.0) 1906 : cluster [DBG] osdmap e928: 8 total, 8 up, 7 in 2021-02-10T04:42:30.119 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:28 smithi085 conmon[29319]: cluster 2021-02-10T04:42:28.315417+0000 mon.a (mon.0) 1907 : cluster [DBG] osdmap e929: 8 total, 8 up, 7 in 2021-02-10T04:42:30.124 INFO:tasks.rados.rados.0.smithi085.stdout:3713: finishing copy_from to smithi08575397-40 2021-02-10T04:42:30.124 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 1488 (ObjNum 1227 snap 415 seq_num 1227) dirty exists 2021-02-10T04:42:30.124 INFO:tasks.rados.rados.0.smithi085.stdout:3713: finishing copy_from racing read to smithi08575397-40 2021-02-10T04:42:30.124 INFO:tasks.rados.rados.0.smithi085.stdout:3713: done (5 left) 2021-02-10T04:42:30.125 INFO:tasks.rados.rados.0.smithi085.stdout:3714: done (4 left) 2021-02-10T04:42:30.125 INFO:tasks.rados.rados.0.smithi085.stdout:3715: rollback oid 6 current snap is 428 2021-02-10T04:42:30.125 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 6 to 410 2021-02-10T04:42:30.125 INFO:tasks.rados.rados.0.smithi085.stdout:3716: write oid 21 current snap is 428 2021-02-10T04:42:30.125 INFO:tasks.rados.rados.0.smithi085.stdout:3716: seq_num 1269 ranges {528083=405579,1342473=419448,2248719=733949} 2021-02-10T04:42:30.125 INFO:tasks.rados.rados.0.smithi085.stdout:3716: writing smithi08575397-21 from 528083 to 933662 tid 1 2021-02-10T04:42:30.125 INFO:tasks.rados.rados.0.smithi085.stdout:3716: writing smithi08575397-21 from 1342473 to 1761921 tid 2 2021-02-10T04:42:30.125 INFO:tasks.rados.rados.0.smithi085.stdout:3716: writing smithi08575397-21 from 2248719 to 2982668 tid 3 2021-02-10T04:42:30.125 INFO:tasks.rados.rados.0.smithi085.stdout:3717: write oid 35 current snap is 428 2021-02-10T04:42:30.125 INFO:tasks.rados.rados.0.smithi085.stdout:3715: finishing rollback tid 0 to smithi08575397-6 2021-02-10T04:42:30.125 INFO:tasks.rados.rados.0.smithi085.stdout:3716: finishing write tid 1 to smithi08575397-21 2021-02-10T04:42:30.125 INFO:tasks.rados.rados.0.smithi085.stdout:3715: finishing rollback tid 1 to smithi08575397-6 2021-02-10T04:42:30.125 INFO:tasks.rados.rados.0.smithi085.stdout:3715: finishing rollback tid 2 to smithi08575397-6 2021-02-10T04:42:30.126 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 1604 (ObjNum 1208 snap 410 seq_num 1208) dirty exists 2021-02-10T04:42:30.126 INFO:tasks.rados.rados.0.smithi085.stdout:3716: finishing write tid 2 to smithi08575397-21 2021-02-10T04:42:30.126 INFO:tasks.rados.rados.0.smithi085.stdout:3717: seq_num 1270 ranges {544890=597181,1917308=563106,3076142=268748} 2021-02-10T04:42:30.126 INFO:tasks.rados.rados.0.smithi085.stdout:3717: writing smithi08575397-35 from 544890 to 1142071 tid 1 2021-02-10T04:42:30.126 INFO:tasks.rados.rados.0.smithi085.stdout:3717: writing smithi08575397-35 from 1917308 to 2480414 tid 2 2021-02-10T04:42:30.126 INFO:tasks.rados.rados.0.smithi085.stdout:3717: writing smithi08575397-35 from 3076142 to 3344890 tid 3 2021-02-10T04:42:30.126 INFO:tasks.rados.rados.0.smithi085.stdout:3715: done (6 left) 2021-02-10T04:42:30.126 INFO:tasks.rados.rados.0.smithi085.stdout:3718: rollback oid 14 current snap is 428 2021-02-10T04:42:30.126 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 14 to 394 2021-02-10T04:42:30.126 INFO:tasks.rados.rados.0.smithi085.stdout:3716: finishing write tid 3 to smithi08575397-21 2021-02-10T04:42:30.126 INFO:tasks.rados.rados.0.smithi085.stdout:3716: finishing write tid 4 to smithi08575397-21 2021-02-10T04:42:30.126 INFO:tasks.rados.rados.0.smithi085.stdout:3716: finishing write tid 5 to smithi08575397-21 2021-02-10T04:42:30.126 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 1905 (ObjNum 1269 snap 428 seq_num 1269) dirty exists 2021-02-10T04:42:30.127 INFO:tasks.rados.rados.0.smithi085.stdout:3716: left oid 21 (ObjNum 1269 snap 428 seq_num 1269) 2021-02-10T04:42:30.127 INFO:tasks.rados.rados.0.smithi085.stdout:3717: finishing write tid 1 to smithi08575397-35 2021-02-10T04:42:30.127 INFO:tasks.rados.rados.0.smithi085.stdout:3716: done (6 left) 2021-02-10T04:42:30.127 INFO:tasks.rados.rados.0.smithi085.stdout:3719: rollback oid 30 current snap is 428 2021-02-10T04:42:30.127 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 30 to 396 2021-02-10T04:42:30.127 INFO:tasks.rados.rados.0.smithi085.stdout:3717: finishing write tid 2 to smithi08575397-35 2021-02-10T04:42:30.127 INFO:tasks.rados.rados.0.smithi085.stdout:3720: delete oid 32 current snap is 428 2021-02-10T04:42:30.127 INFO:tasks.rados.rados.0.smithi085.stdout:3719: finishing rollback tid 1 to smithi08575397-30 2021-02-10T04:42:30.127 INFO:tasks.rados.rados.0.smithi085.stdout:3717: finishing write tid 3 to smithi08575397-35 2021-02-10T04:42:30.127 INFO:tasks.rados.rados.0.smithi085.stdout:3717: finishing write tid 4 to smithi08575397-35 2021-02-10T04:42:30.127 INFO:tasks.rados.rados.0.smithi085.stdout:3717: finishing write tid 5 to smithi08575397-35 2021-02-10T04:42:30.127 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 35 v 1600 (ObjNum 1270 snap 428 seq_num 1270) dirty exists 2021-02-10T04:42:30.127 INFO:tasks.rados.rados.0.smithi085.stdout:3717: left oid 35 (ObjNum 1270 snap 428 seq_num 1270) 2021-02-10T04:42:30.128 INFO:tasks.rados.rados.0.smithi085.stdout:3719: finishing rollback tid 2 to smithi08575397-30 2021-02-10T04:42:30.128 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 1606 (ObjNum 1025 snap 335 seq_num 1025) dirty exists 2021-02-10T04:42:30.128 INFO:tasks.rados.rados.0.smithi085.stdout:3717: done (7 left) 2021-02-10T04:42:30.128 INFO:tasks.rados.rados.0.smithi085.stdout:3719: done (6 left) 2021-02-10T04:42:30.128 INFO:tasks.rados.rados.0.smithi085.stdout:3720: done (5 left) 2021-02-10T04:42:30.128 INFO:tasks.rados.rados.0.smithi085.stdout:3721: write oid 45 current snap is 428 2021-02-10T04:42:30.128 INFO:tasks.rados.rados.0.smithi085.stdout:3721: seq_num 1272 ranges {578504=747679,2034272=566775,3130988=247516} 2021-02-10T04:42:30.128 INFO:tasks.rados.rados.0.smithi085.stdout:3721: writing smithi08575397-45 from 578504 to 1326183 tid 1 2021-02-10T04:42:30.128 INFO:tasks.rados.rados.0.smithi085.stdout:3721: writing smithi08575397-45 from 2034272 to 2601047 tid 2 2021-02-10T04:42:30.128 INFO:tasks.rados.rados.0.smithi085.stdout:3721: writing smithi08575397-45 from 3130988 to 3378504 tid 3 2021-02-10T04:42:30.128 INFO:tasks.rados.rados.0.smithi085.stdout:3722: rollback oid 35 current snap is 428 2021-02-10T04:42:30.128 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 35 to 393 2021-02-10T04:42:30.128 INFO:tasks.rados.rados.0.smithi085.stdout:3718: finishing rollback tid 0 to smithi08575397-14 2021-02-10T04:42:30.129 INFO:tasks.rados.rados.0.smithi085.stdout:3718: finishing rollback tid 1 to smithi08575397-14 2021-02-10T04:42:30.129 INFO:tasks.rados.rados.0.smithi085.stdout:3718: finishing rollback tid 2 to smithi08575397-14 2021-02-10T04:42:30.129 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 1880 (ObjNum 1049 snap 344 seq_num 1049) dirty exists 2021-02-10T04:42:30.129 INFO:tasks.rados.rados.0.smithi085.stdout:3721: finishing write tid 1 to smithi08575397-45 2021-02-10T04:42:30.129 INFO:tasks.rados.rados.0.smithi085.stdout:3721: finishing write tid 2 to smithi08575397-45 2021-02-10T04:42:30.129 INFO:tasks.rados.rados.0.smithi085.stdout:3718: done (6 left) 2021-02-10T04:42:30.129 INFO:tasks.rados.rados.0.smithi085.stdout:3723: read oid 6 snap -1 2021-02-10T04:42:30.129 INFO:tasks.rados.rados.0.smithi085.stdout:3723: expect (ObjNum 1208 snap 410 seq_num 1208) 2021-02-10T04:42:30.129 INFO:tasks.rados.rados.0.smithi085.stdout:3724: read oid 21 snap -1 2021-02-10T04:42:30.129 INFO:tasks.rados.rados.0.smithi085.stdout:3724: expect (ObjNum 1269 snap 428 seq_num 1269) 2021-02-10T04:42:30.129 INFO:tasks.rados.rados.0.smithi085.stdout:3725: read oid 13 snap -1 2021-02-10T04:42:30.129 INFO:tasks.rados.rados.0.smithi085.stdout:3725: expect (ObjNum 1241 snap 418 seq_num 1241) 2021-02-10T04:42:30.129 INFO:tasks.rados.rados.0.smithi085.stdout:3726: write oid 38 current snap is 428 2021-02-10T04:42:30.129 INFO:tasks.rados.rados.0.smithi085.stdout:3726: seq_num 1273 ranges {595311=422928,1576401=710433,3042058=353253} 2021-02-10T04:42:30.130 INFO:tasks.rados.rados.0.smithi085.stdout:3726: writing smithi08575397-38 from 595311 to 1018239 tid 1 2021-02-10T04:42:30.130 INFO:tasks.rados.rados.0.smithi085.stdout:3726: writing smithi08575397-38 from 1576401 to 2286834 tid 2 2021-02-10T04:42:30.130 INFO:tasks.rados.rados.0.smithi085.stdout:3726: writing smithi08575397-38 from 3042058 to 3395311 tid 3 2021-02-10T04:42:30.130 INFO:tasks.rados.rados.0.smithi085.stdout:3727: read oid 44 snap -1 2021-02-10T04:42:30.130 INFO:tasks.rados.rados.0.smithi085.stdout:3727: expect (ObjNum 1247 snap 420 seq_num 1247) 2021-02-10T04:42:30.130 INFO:tasks.rados.rados.0.smithi085.stdout:3728: copy_from oid 23 from oid 22 current snap is 428 2021-02-10T04:42:30.130 INFO:tasks.rados.rados.0.smithi085.stdout:3722: finishing rollback tid 0 to smithi08575397-35 2021-02-10T04:42:30.130 INFO:tasks.rados.rados.0.smithi085.stdout:3722: finishing rollback tid 1 to smithi08575397-35 2021-02-10T04:42:30.130 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 35 v 1602 (ObjNum 1406329720 snap 32576 seq_num 1) dirty dne 2021-02-10T04:42:30.130 INFO:tasks.rados.rados.0.smithi085.stdout:3723: expect (ObjNum 1208 snap 410 seq_num 1208) 2021-02-10T04:42:30.130 INFO:tasks.rados.rados.0.smithi085.stdout:3722: done (11 left) 2021-02-10T04:42:30.130 INFO:tasks.rados.rados.0.smithi085.stdout:3723: done (10 left) 2021-02-10T04:42:30.130 INFO:tasks.rados.rados.0.smithi085.stdout:3729: copy_from oid 49 from oid 20 current snap is 428 2021-02-10T04:42:30.131 INFO:tasks.rados.rados.0.smithi085.stdout:3726: finishing write tid 1 to smithi08575397-38 2021-02-10T04:42:30.131 INFO:tasks.rados.rados.0.smithi085.stdout:3724: expect (ObjNum 1269 snap 428 seq_num 1269) 2021-02-10T04:42:30.131 INFO:tasks.rados.rados.0.smithi085.stdout:3724: done (10 left) 2021-02-10T04:42:30.131 INFO:tasks.rados.rados.0.smithi085.stdout:3730: snap_create 2021-02-10T04:42:30.131 INFO:tasks.rados.rados.0.smithi085.stdout:3721: finishing write tid 3 to smithi08575397-45 2021-02-10T04:42:30.131 INFO:tasks.rados.rados.0.smithi085.stdout:3721: finishing write tid 4 to smithi08575397-45 2021-02-10T04:42:30.131 INFO:tasks.rados.rados.0.smithi085.stdout:3721: finishing write tid 5 to smithi08575397-45 2021-02-10T04:42:30.131 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 45 v 1500 (ObjNum 1272 snap 428 seq_num 1272) dirty exists 2021-02-10T04:42:30.131 INFO:tasks.rados.rados.0.smithi085.stdout:3721: left oid 45 (ObjNum 1272 snap 428 seq_num 1272) 2021-02-10T04:42:30.131 INFO:tasks.rados.rados.0.smithi085.stdout:3726: finishing write tid 2 to smithi08575397-38 2021-02-10T04:42:30.131 INFO:tasks.rados.rados.0.smithi085.stdout:3726: finishing write tid 3 to smithi08575397-38 2021-02-10T04:42:30.131 INFO:tasks.rados.rados.0.smithi085.stdout:3726: finishing write tid 4 to smithi08575397-38 2021-02-10T04:42:30.131 INFO:tasks.rados.rados.0.smithi085.stdout:3726: finishing write tid 5 to smithi08575397-38 2021-02-10T04:42:30.132 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 1610 (ObjNum 1273 snap 428 seq_num 1273) dirty exists 2021-02-10T04:42:30.132 INFO:tasks.rados.rados.0.smithi085.stdout:3726: left oid 38 (ObjNum 1273 snap 428 seq_num 1273) 2021-02-10T04:42:30.132 INFO:tasks.rados.rados.0.smithi085.stdout:3727: expect (ObjNum 1247 snap 420 seq_num 1247) 2021-02-10T04:42:30.132 INFO:tasks.rados.rados.0.smithi085.stdout:3728: finishing copy_from to smithi08575397-23 2021-02-10T04:42:30.132 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 1489 (ObjNum 1224 snap 414 seq_num 1224) dirty exists 2021-02-10T04:42:30.132 INFO:tasks.rados.rados.0.smithi085.stdout:3728: finishing copy_from racing read to smithi08575397-23 2021-02-10T04:42:30.132 INFO:tasks.rados.rados.0.smithi085.stdout:3725: expect (ObjNum 1241 snap 418 seq_num 1241) 2021-02-10T04:42:30.132 INFO:tasks.rados.rados.0.smithi085.stdout:3729: finishing copy_from to smithi08575397-49 2021-02-10T04:42:30.132 INFO:tasks.rados.rados.0.smithi085.stdout:3729: got expected ENOENT (src dne) 2021-02-10T04:42:30.132 INFO:tasks.rados.rados.0.smithi085.stdout:3729: finishing copy_from racing read to smithi08575397-49 2021-02-10T04:42:30.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:29 smithi164 conmon[29288]: cluster 2021-02-10T04:42:28.235013+0000 2021-02-10T04:42:30.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:29 smithi164 conmon[29288]: mgr.y (mgr.14140) 862 : cluster [DBG] pgmap v1543: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 764 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 5.1 MiB/s rd, 1.6 MiB/s wr, 6 op/s 2021-02-10T04:42:30.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:29 smithi164 conmon[29288]: audit 2021-02-10T04:42:28.518056+0000 mgr.y (mgr.14140) 863 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:42:30.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:29 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:42:30.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:29 smithi164 conmon[29288]: -10T04:42:28.796824+0000 mon.a (mon.0) 1908 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 772 sec, osd.4 has slow ops) 2021-02-10T04:42:30.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:29 smithi164 conmon[29288]: cluster 2021-02-10T04:42:30.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:29 smithi164 conmon[29288]: 2021- 2021-02-10T04:42:30.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:29 smithi164 conmon[29288]: 02-10 2021-02-10T04:42:30.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:29 smithi164 conmon[29288]: T04:42:29.322722+0000 mon.a ( 2021-02-10T04:42:30.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:29 smithi164 conmon[29288]: mon.0) 1909 : cluster [DBG] osdmap e930: 8 total, 8 up, 7 in 2021-02-10T04:42:30.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[29319]: cluster 2021-02-10T04:42:30.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[29319]: 2021-02-10T04:42:30.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[29319]: 2021 2021-02-10T04:42:30.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[29319]: - 2021-02-10T04:42:30.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[29319]: 02 2021-02-10T04:42:30.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[29319]: - 2021-02-10T04:42:30.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[29319]: 10 2021-02-10T04:42:30.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[29319]: T04:42:28. 2021-02-10T04:42:30.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[29319]: 235013+0000 mgr.y (mgr.14140) 862 : cluster [DBG] pgmap v1543: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 764 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 5.1 MiB/s rd, 1.6 MiB/s wr, 6 op/s 2021-02-10T04:42:30.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[29319]: audit 2021-02-10T04:42:28.518056 2021-02-10T04:42:30.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[29319]: +0000 2021-02-10T04:42:30.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[29319]: mgr.y (mgr.14140) 863 : 2021-02-10T04:42:30.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[29319]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:42:30.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:42:30.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[29319]: 02-10T04:42:28.796824+0000 2021-02-10T04:42:30.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[29319]: mon.a (mon.0) 1908 : cluster [INF] 2021-02-10T04:42:30.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[29319]: Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 772 sec, osd.4 has slow ops) 2021-02-10T04:42:30.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[29319]: cluster 2021-02-10T04:42: 2021-02-10T04:42:30.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[29319]: 29.322722+0000 mon.a (mon.0) 1909 2021-02-10T04:42:30.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[29319]: : cluster [DBG] osdmap e930: 8 total, 8 up, 7 in 2021-02-10T04:42:30.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[35961]: cluster 2021-02-10T04:42:30.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[35961]: 2021-02-10T04:42:30.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[35961]: 2021 2021-02-10T04:42:30.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[35961]: - 2021-02-10T04:42:30.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[35961]: 02-10T04:42:28.235013+0000 2021-02-10T04:42:30.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[35961]: mgr.y (mgr.14140) 862 : cluster [DBG] pgmap v1543: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 764 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 5.1 MiB/s rd, 1.6 MiB/s wr, 6 op/s 2021-02-10T04:42:30.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[35961]: audit 2021-02-10T04:42:28.518056+0000 mgr.y (mgr.14140) 863 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:42:30.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[35961]: cluster 2021-02-10T04:42:28.796824+0000 2021-02-10T04:42:30.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[35961]: mon.a (mon.0) 1908 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 772 sec, osd.4 has slow ops) 2021-02-10T04:42:30.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:42:30.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[35961]: 04:42:29.322722+0000 mon.a (mon.0) 2021-02-10T04:42:30.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:29 smithi085 conmon[35961]: 1909 : cluster [DBG] osdmap e930: 8 total, 8 up, 7 in 2021-02-10T04:42:30.325 INFO:tasks.rados.rados.0.smithi085.stdout:3721: done (10 left) 2021-02-10T04:42:30.325 INFO:tasks.rados.rados.0.smithi085.stdout:3725: done (9 left) 2021-02-10T04:42:30.325 INFO:tasks.rados.rados.0.smithi085.stdout:3726: done (8 left) 2021-02-10T04:42:30.326 INFO:tasks.rados.rados.0.smithi085.stdout:3727: done (7 left) 2021-02-10T04:42:30.326 INFO:tasks.rados.rados.0.smithi085.stdout:3728: done (6 left) 2021-02-10T04:42:30.326 INFO:tasks.rados.rados.0.smithi085.stdout:3729: done (5 left) 2021-02-10T04:42:30.326 INFO:tasks.rados.rados.0.smithi085.stdout:3730: done (4 left) 2021-02-10T04:42:30.326 INFO:tasks.rados.rados.0.smithi085.stdout:3731: write (excl) oid 2 current snap is 429 2021-02-10T04:42:30.327 INFO:tasks.rados.rados.0.smithi085.stdout:3731: seq_num 1274 ranges {612118=498177,1634883=570444,2669481=706662} 2021-02-10T04:42:30.332 INFO:tasks.rados.rados.0.smithi085.stdout:3731: writing smithi08575397-2 from 612118 to 1110295 tid 1 2021-02-10T04:42:30.339 INFO:tasks.rados.rados.0.smithi085.stdout:3731: writing smithi08575397-2 from 1634883 to 2205327 tid 2 2021-02-10T04:42:30.347 INFO:tasks.rados.rados.0.smithi085.stdout:3731: writing smithi08575397-2 from 2669481 to 3376143 tid 3 2021-02-10T04:42:30.347 INFO:tasks.rados.rados.0.smithi085.stdout:3732: snap_create 2021-02-10T04:42:30.347 INFO:tasks.rados.rados.0.smithi085.stdout:3731: finishing write tid 1 to smithi08575397-2 2021-02-10T04:42:30.347 INFO:tasks.rados.rados.0.smithi085.stdout:3731: finishing write tid 2 to smithi08575397-2 2021-02-10T04:42:30.351 INFO:tasks.rados.rados.0.smithi085.stdout:3731: finishing write tid 3 to smithi08575397-2 2021-02-10T04:42:30.352 INFO:tasks.rados.rados.0.smithi085.stdout:3731: finishing write tid 4 to smithi08575397-2 2021-02-10T04:42:30.352 INFO:tasks.rados.rados.0.smithi085.stdout:3731: finishing write tid 5 to smithi08575397-2 2021-02-10T04:42:30.352 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 1606 (ObjNum 1274 snap 429 seq_num 1274) dirty exists 2021-02-10T04:42:30.352 INFO:tasks.rados.rados.0.smithi085.stdout:3731: left oid 2 (ObjNum 1274 snap 429 seq_num 1274) 2021-02-10T04:42:31.331 INFO:tasks.rados.rados.0.smithi085.stdout:3731: done (5 left) 2021-02-10T04:42:31.332 INFO:tasks.rados.rados.0.smithi085.stdout:3732: done (4 left) 2021-02-10T04:42:31.332 INFO:tasks.rados.rados.0.smithi085.stdout:3733: write (excl) oid 1 current snap is 430 2021-02-10T04:42:31.332 INFO:tasks.rados.rados.0.smithi085.stdout:3733: seq_num 1275 ranges {628925=573426,1693365=714102,3096904=332021} 2021-02-10T04:42:31.338 INFO:tasks.rados.rados.0.smithi085.stdout:3733: writing smithi08575397-1 from 628925 to 1202351 tid 1 2021-02-10T04:42:31.346 INFO:tasks.rados.rados.0.smithi085.stdout:3733: writing smithi08575397-1 from 1693365 to 2407467 tid 2 2021-02-10T04:42:31.353 INFO:tasks.rados.rados.0.smithi085.stdout:3733: writing smithi08575397-1 from 3096904 to 3428925 tid 3 2021-02-10T04:42:31.353 INFO:tasks.rados.rados.0.smithi085.stdout:3734: read oid 28 snap -1 2021-02-10T04:42:31.353 INFO:tasks.rados.rados.0.smithi085.stdout:3734: expect deleted 2021-02-10T04:42:31.353 INFO:tasks.rados.rados.0.smithi085.stdout:3735: write (excl) oid 45 current snap is 430 2021-02-10T04:42:31.353 INFO:tasks.rados.rados.0.smithi085.stdout:3735: seq_num 1276 ranges {645732=648675,1751847=574113,3124327=321405} 2021-02-10T04:42:31.361 INFO:tasks.rados.rados.0.smithi085.stdout:3735: writing smithi08575397-45 from 645732 to 1294407 tid 1 2021-02-10T04:42:31.367 INFO:tasks.rados.rados.0.smithi085.stdout:3735: writing smithi08575397-45 from 1751847 to 2325960 tid 2 2021-02-10T04:42:31.375 INFO:tasks.rados.rados.0.smithi085.stdout:3735: writing smithi08575397-45 from 3124327 to 3445732 tid 3 2021-02-10T04:42:31.375 INFO:tasks.rados.rados.0.smithi085.stdout:3736: write (excl) oid 43 current snap is 430 2021-02-10T04:42:31.375 INFO:tasks.rados.rados.0.smithi085.stdout:3733: finishing write tid 1 to smithi08575397-1 2021-02-10T04:42:31.375 INFO:tasks.rados.rados.0.smithi085.stdout:3733: finishing write tid 2 to smithi08575397-1 2021-02-10T04:42:31.375 INFO:tasks.rados.rados.0.smithi085.stdout:3733: finishing write tid 3 to smithi08575397-1 2021-02-10T04:42:31.375 INFO:tasks.rados.rados.0.smithi085.stdout:3733: finishing write tid 4 to smithi08575397-1 2021-02-10T04:42:31.375 INFO:tasks.rados.rados.0.smithi085.stdout:3733: finishing write tid 5 to smithi08575397-1 2021-02-10T04:42:31.375 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 1612 (ObjNum 1275 snap 430 seq_num 1275) dirty exists 2021-02-10T04:42:31.375 INFO:tasks.rados.rados.0.smithi085.stdout:3733: left oid 1 (ObjNum 1275 snap 430 seq_num 1275) 2021-02-10T04:42:31.375 INFO:tasks.rados.rados.0.smithi085.stdout:3735: finishing write tid 1 to smithi08575397-45 2021-02-10T04:42:31.375 INFO:tasks.rados.rados.0.smithi085.stdout:3735: finishing write tid 2 to smithi08575397-45 2021-02-10T04:42:31.375 INFO:tasks.rados.rados.0.smithi085.stdout:3736: seq_num 1277 ranges {662539=723924,2093976=717771,3435397=27142} 2021-02-10T04:42:31.383 INFO:tasks.rados.rados.0.smithi085.stdout:3736: writing smithi08575397-43 from 662539 to 1386463 tid 1 2021-02-10T04:42:31.392 INFO:tasks.rados.rados.0.smithi085.stdout:3736: writing smithi08575397-43 from 2093976 to 2811747 tid 2 2021-02-10T04:42:31.397 INFO:tasks.rados.rados.0.smithi085.stdout:3736: writing smithi08575397-43 from 3435397 to 3462539 tid 3 2021-02-10T04:42:31.397 INFO:tasks.rados.rados.0.smithi085.stdout:3733: done (7 left) 2021-02-10T04:42:31.397 INFO:tasks.rados.rados.0.smithi085.stdout:3734: done (6 left) 2021-02-10T04:42:31.397 INFO:tasks.rados.rados.0.smithi085.stdout:3737: read oid 20 snap -1 2021-02-10T04:42:31.397 INFO:tasks.rados.rados.0.smithi085.stdout:3737: expect deleted 2021-02-10T04:42:31.397 INFO:tasks.rados.rados.0.smithi085.stdout:3738: rollback oid 28 current snap is 430 2021-02-10T04:42:31.397 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 28 to 428 2021-02-10T04:42:31.398 INFO:tasks.rados.rados.0.smithi085.stdout:3735: finishing write tid 3 to smithi08575397-45 2021-02-10T04:42:31.398 INFO:tasks.rados.rados.0.smithi085.stdout:3739: rollback oid 5 current snap is 430 2021-02-10T04:42:31.398 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 5 to 406 2021-02-10T04:42:31.398 INFO:tasks.rados.rados.0.smithi085.stdout:3735: finishing write tid 4 to smithi08575397-45 2021-02-10T04:42:31.398 INFO:tasks.rados.rados.0.smithi085.stdout:3735: finishing write tid 5 to smithi08575397-45 2021-02-10T04:42:31.398 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 45 v 1504 (ObjNum 1276 snap 430 seq_num 1276) dirty exists 2021-02-10T04:42:31.398 INFO:tasks.rados.rados.0.smithi085.stdout:3735: left oid 45 (ObjNum 1276 snap 430 seq_num 1276) 2021-02-10T04:42:31.398 INFO:tasks.rados.rados.0.smithi085.stdout:3736: finishing write tid 1 to smithi08575397-43 2021-02-10T04:42:31.398 INFO:tasks.rados.rados.0.smithi085.stdout:3735: done (8 left) 2021-02-10T04:42:31.398 INFO:tasks.rados.rados.0.smithi085.stdout:3740: copy_from oid 46 from oid 48 current snap is 430 2021-02-10T04:42:31.398 INFO:tasks.rados.rados.0.smithi085.stdout:3741: delete oid 29 current snap is 430 2021-02-10T04:42:31.399 INFO:tasks.rados.rados.0.smithi085.stdout:3738: finishing rollback tid 1 to smithi08575397-28 2021-02-10T04:42:31.400 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 1459 (ObjNum 1215200336 snap 32576 seq_num 1215200336) dirty dne 2021-02-10T04:42:31.400 INFO:tasks.rados.rados.0.smithi085.stdout:3739: finishing rollback tid 0 to smithi08575397-5 2021-02-10T04:42:31.400 INFO:tasks.rados.rados.0.smithi085.stdout:3739: finishing rollback tid 1 to smithi08575397-5 2021-02-10T04:42:31.401 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 1573 (ObjNum 83399232 snap 22030 seq_num 0) dirty dne 2021-02-10T04:42:31.401 INFO:tasks.rados.rados.0.smithi085.stdout:3736: finishing write tid 2 to smithi08575397-43 2021-02-10T04:42:31.401 INFO:tasks.rados.rados.0.smithi085.stdout:3736: finishing write tid 3 to smithi08575397-43 2021-02-10T04:42:31.401 INFO:tasks.rados.rados.0.smithi085.stdout:3737: done (9 left) 2021-02-10T04:42:31.402 INFO:tasks.rados.rados.0.smithi085.stdout:3738: done (8 left) 2021-02-10T04:42:31.402 INFO:tasks.rados.rados.0.smithi085.stdout:3739: done (7 left) 2021-02-10T04:42:31.402 INFO:tasks.rados.rados.0.smithi085.stdout:3741: done (6 left) 2021-02-10T04:42:31.402 INFO:tasks.rados.rados.0.smithi085.stdout:3742: read oid 20 snap 424 2021-02-10T04:42:31.402 INFO:tasks.rados.rados.0.smithi085.stdout:3742: expect deleted 2021-02-10T04:42:31.402 INFO:tasks.rados.rados.0.smithi085.stdout:3743: snap_remove snap 410 2021-02-10T04:42:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:31 smithi164 conmon[29288]: cluster 2021-02-10T04:42:30.235959+0000 2021-02-10T04:42:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:31 smithi164 conmon[29288]: mgr.y (mgr.14140) 864 : cluster [DBG] pgmap v1546: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 787 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 8.8 MiB/s rd, 6.1 MiB/s wr, 23 op/s 2021-02-10T04:42:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:31 smithi164 conmon[29288]: cluster 2021-02-10T04:42:30.324050+0000 mon.a (mon.0) 1910 : cluster [DBG] osdmap e931: 8 total, 8 up, 7 in 2021-02-10T04:42:31.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:31 smithi085 conmon[35961]: cluster 2021-02-10T04:42:30.235959+0000 mgr.y (mgr.14140) 864 : cluster [DBG] pgmap v1546: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 787 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 8.8 MiB/s rd, 6.1 MiB/s wr, 23 op/s 2021-02-10T04:42:31.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:31 smithi085 conmon[35961]: 2021-02-10T04:42:31.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:31 smithi085 conmon[35961]: cluster 2021-02-10T04:42:30.324050+0000 mon.a (mon.0 2021-02-10T04:42:31.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:31 smithi085 conmon[35961]: ) 1910 : cluster [DBG] osdmap e931: 8 total, 8 up, 7 in 2021-02-10T04:42:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:31 smithi085 conmon[29319]: cluster 2021-02-10T04:42:30.235959+0000 mgr.y 2021-02-10T04:42:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:31 smithi085 conmon[29319]: (mgr.14140) 864 : cluster [DBG] pgmap v1546: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 787 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 8.8 MiB/s rd, 6.1 MiB/s wr, 23 op/s 2021-02-10T04:42:31.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:31 smithi085 conmon[29319]: cluster 2021-02-10T04:42:30 2021-02-10T04:42:31.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:31 smithi085 conmon[29319]: .324050+0000 mon.a (mon.0) 1910 : cluster [DBG] osdmap e931: 8 total, 8 up, 7 in 2021-02-10T04:42:32.337 INFO:tasks.rados.rados.0.smithi085.stdout:3743: done (7 left) 2021-02-10T04:42:32.337 INFO:tasks.rados.rados.0.smithi085.stdout:3744: snap_create 2021-02-10T04:42:32.337 INFO:tasks.rados.rados.0.smithi085.stdout:3736: finishing write tid 4 to smithi08575397-43 2021-02-10T04:42:32.337 INFO:tasks.rados.rados.0.smithi085.stdout:3736: finishing write tid 5 to smithi08575397-43 2021-02-10T04:42:32.337 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 1886 (ObjNum 1277 snap 430 seq_num 1277) dirty exists 2021-02-10T04:42:32.337 INFO:tasks.rados.rados.0.smithi085.stdout:3736: left oid 43 (ObjNum 1277 snap 430 seq_num 1277) 2021-02-10T04:42:32.337 INFO:tasks.rados.rados.0.smithi085.stdout:3740: finishing copy_from to smithi08575397-46 2021-02-10T04:42:32.337 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 1877 (ObjNum 1261 snap 427 seq_num 1261) dirty exists 2021-02-10T04:42:32.337 INFO:tasks.rados.rados.0.smithi085.stdout:3740: finishing copy_from racing read to smithi08575397-46 2021-02-10T04:42:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:32 smithi164 conmon[29288]: cluster 2021-02-10T04:42:31.330871+0000 mon.a (mon 2021-02-10T04:42:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:32 smithi164 conmon[29288]: .0) 1911 : cluster [DBG] osdmap e932: 8 total, 8 up, 7 in 2021-02-10T04:42:32.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:32 smithi085 conmon[29319]: cluster 2021-02-10T04:42:32.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:32 smithi085 conmon[29319]: 2021-02-10T04:42:32.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:32 smithi085 conmon[29319]: 2021 2021-02-10T04:42:32.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:32 smithi085 conmon[29319]: -02-10T04:42:31.330871 2021-02-10T04:42:32.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:32 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1911 : cluster [DBG] osdmap e932: 8 total, 8 up, 7 in 2021-02-10T04:42:32.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:32 smithi085 conmon[35961]: cluster 2021-02-10T04:42:31. 2021-02-10T04:42:32.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:32 smithi085 conmon[35961]: 330871+0000 mon.a (mon.0) 1911 : cluster [DBG] osdmap e932: 8 total, 8 up, 7 in 2021-02-10T04:42:32.802 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:42:32.830 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:33.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:32 smithi164 conmon[33677]: debug 2021-02-10T04:42:32.889+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:33.269 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:42:33.375 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:42:33.398 INFO:tasks.rados.rados.0.smithi085.stdout:3736: done (7 left) 2021-02-10T04:42:33.398 INFO:tasks.rados.rados.0.smithi085.stdout:3740: done (6 left) 2021-02-10T04:42:33.398 INFO:tasks.rados.rados.0.smithi085.stdout:3742: done (5 left) 2021-02-10T04:42:33.399 INFO:tasks.rados.rados.0.smithi085.stdout:3744: done (4 left) 2021-02-10T04:42:33.399 INFO:tasks.rados.rados.0.smithi085.stdout:3745: delete oid 44 current snap is 431 2021-02-10T04:42:33.406 INFO:tasks.rados.rados.0.smithi085.stdout:3745: done (4 left) 2021-02-10T04:42:33.407 INFO:tasks.rados.rados.0.smithi085.stdout:3746: snap_create 2021-02-10T04:42:33.414 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:33.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:33 smithi164 conmon[29288]: cluster 2021-02-10T04:42:33.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:33 smithi164 conmon[29288]: 2021-02-10T04:42:31. 2021-02-10T04:42:33.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:33 smithi164 conmon[29288]: 556735+0000 osd.3 (osd.3) 88 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:42:33.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:33 smithi164 conmon[29288]: cluster 2021-02-10T04:42:33.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:33 smithi164 conmon[29288]: 2021-02-10T04:42:31.557711+0000 osd.3 (osd.3) 89 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:42:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:33 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:42:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:33 smithi164 conmon[29288]: 42:32.236522+0000 mgr.y (mgr.14140) 865 : cluster 2021-02-10T04:42:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:33 smithi164 conmon[29288]: [DBG] pgmap v1549: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 787 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 3.8 MiB/s rd, 5.1 MiB/s wr, 17 op/s 2021-02-10T04:42:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:33 smithi164 conmon[29288]: cluster 2021-02-10T04:42:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:33 smithi164 conmon[29288]: 2021-02-10T04:42:32.336006+0000 mon.a (mon.0) 1912 : cluster 2021-02-10T04:42:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:33 smithi164 conmon[29288]: [DBG] osdmap e933: 8 total, 8 up, 7 in 2021-02-10T04:42:33.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:33 smithi085 conmon[29319]: cluster 2021-02-10T04:42:31.556735+0000 osd.3 (osd.3) 88 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:42:33.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:33 smithi085 conmon[29319]: cluster 2021-02-10T04:42:31.557711+0000 osd.3 (osd.3) 89 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:42:33.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:33 smithi085 conmon[29319]: cluster 2021-02-10T04:42:32.236522+0000 mgr.y (mgr.14140) 865 : cluster [DBG] pgmap v1549: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 787 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 3.8 MiB/s rd, 5.1 MiB/s wr, 17 op/s 2021-02-10T04:42:33.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:33 smithi085 conmon[29319]: cluster 2021-02-10T04:42:32.336006+0000 mon.a (mon.0) 1912 : cluster [DBG] osdmap e933: 8 total, 8 up, 7 in 2021-02-10T04:42:33.692 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:42:33 smithi085 conmon[45443]: debug 2021-02-10T04:42:33.610+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:33.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:33 smithi085 conmon[35961]: cluster 2021-02-10T04:42:31. 2021-02-10T04:42:33.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:33 smithi085 conmon[35961]: 556735+0000 osd.3 (osd.3) 88 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:42:33.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:33 smithi085 conmon[35961]: cluster 2021-02-10T04:42:33.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:33 smithi085 conmon[35961]: 2021-02-10T04:42:31.557711+0000 osd.3 (osd.3) 89 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:42:33.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:33 smithi085 conmon[35961]: cluster 2021-02-10T04:42:32.236522+0000 mgr.y ( 2021-02-10T04:42:33.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:33 smithi085 conmon[35961]: mgr.14140) 865 : cluster [DBG] pgmap v1549: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 787 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 3.8 MiB/s rd, 5.1 MiB/s wr, 17 op/s 2021-02-10T04:42:33.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:33 smithi085 conmon[35961]: cluster 2021-02-10T04:42:32.336006+0000 mon.a (mon.0 2021-02-10T04:42:33.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:33 smithi085 conmon[35961]: ) 1912 : cluster [DBG] osdmap e933: 8 total, 8 up, 7 in 2021-02-10T04:42:33.940 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:42:34.409 INFO:tasks.rados.rados.0.smithi085.stdout:3746: done (4 left) 2021-02-10T04:42:34.409 INFO:tasks.rados.rados.0.smithi085.stdout:3747: delete oid 48 current snap is 432 2021-02-10T04:42:34.417 INFO:tasks.rados.rados.0.smithi085.stdout:3747: done (4 left) 2021-02-10T04:42:34.417 INFO:tasks.rados.rados.0.smithi085.stdout:3748: copy_from oid 40 from oid 48 current snap is 432 2021-02-10T04:42:34.417 INFO:tasks.rados.rados.0.smithi085.stdout:3749: copy_from oid 4 from oid 12 current snap is 432 2021-02-10T04:42:34.417 INFO:tasks.rados.rados.0.smithi085.stdout:3750: snap_create 2021-02-10T04:42:34.419 INFO:tasks.rados.rados.0.smithi085.stdout:3748: finishing copy_from to smithi08575397-40 2021-02-10T04:42:34.419 INFO:tasks.rados.rados.0.smithi085.stdout:3748: got expected ENOENT (src dne) 2021-02-10T04:42:34.419 INFO:tasks.rados.rados.0.smithi085.stdout:3749: finishing copy_from to smithi08575397-4 2021-02-10T04:42:34.419 INFO:tasks.rados.rados.0.smithi085.stdout:3749: got expected ENOENT (src dne) 2021-02-10T04:42:34.420 INFO:tasks.rados.rados.0.smithi085.stdout:3748: finishing copy_from racing read to smithi08575397-40 2021-02-10T04:42:34.420 INFO:tasks.rados.rados.0.smithi085.stdout:3749: finishing copy_from racing read to smithi08575397-4 2021-02-10T04:42:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:34 smithi085 conmon[29319]: cluster 2021-02-10T04:42:33.393396+0000 mon.a (mon. 2021-02-10T04:42:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:34 smithi085 conmon[29319]: 0) 1913 : cluster [DBG] osdmap e934: 8 total, 8 up, 7 in 2021-02-10T04:42:34.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:34 smithi085 conmon[35961]: cluster 2021-02-10T04:42:34.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:34 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:42:34.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:34 smithi085 conmon[35961]: 10T04:42:33.393396+0000 mon.a (mon.0) 1913 : cluster [DBG] osdmap e934: 8 total, 8 up, 7 in 2021-02-10T04:42:34.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:34 smithi164 conmon[29288]: cluster 2021-02-10T04:42:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:34 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:42:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:34 smithi164 conmon[29288]: 42:33.393396+0000 mon.a (mon.0) 1913 : cluster [DBG] osdmap e934: 8 total, 8 up, 7 in 2021-02-10T04:42:35.425 INFO:tasks.rados.rados.0.smithi085.stdout:3748: done (6 left) 2021-02-10T04:42:35.426 INFO:tasks.rados.rados.0.smithi085.stdout:3749: done (5 left) 2021-02-10T04:42:35.426 INFO:tasks.rados.rados.0.smithi085.stdout:3750: done (4 left) 2021-02-10T04:42:35.426 INFO:tasks.rados.rados.0.smithi085.stdout:3751: write oid 50 current snap is 433 2021-02-10T04:42:35.426 INFO:tasks.rados.rados.0.smithi085.stdout:3751: seq_num 1281 ranges {729767=741273,1927904=725109,3145089=384678} 2021-02-10T04:42:35.431 INFO:tasks.rados.rados.0.smithi085.stdout:3751: writing smithi08575397-50 from 729767 to 1471040 tid 1 2021-02-10T04:42:35.439 INFO:tasks.rados.rados.0.smithi085.stdout:3751: writing smithi08575397-50 from 1927904 to 2653013 tid 2 2021-02-10T04:42:35.444 INFO:tasks.rados.rados.0.smithi085.stdout:3751: writing smithi08575397-50 from 3145089 to 3529767 tid 3 2021-02-10T04:42:35.444 INFO:tasks.rados.rados.0.smithi085.stdout:3752: snap_remove snap 420 2021-02-10T04:42:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:35 smithi085 conmon[29319]: cluster 2021-02-10T04:42:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:35 smithi085 conmon[29319]: 2021-02-10T04:42:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:35 smithi085 conmon[29319]: 2021-02-10T04:42:34.237138+0000 2021-02-10T04:42:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:35 smithi085 conmon[29319]: mgr.y (mgr.14140) 866 : cluster [DBG] pgmap v1552: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 798 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 8.0 MiB/s rd, 1.5 MiB/s wr, 15 op/s 2021-02-10T04:42:35.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:35 smithi085 conmon[29319]: cluster 2021-02-10T04:42:35.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:35 smithi085 conmon[29319]: 2021-02-10T04:42:34.407771+0000 mon.a (mon.0) 1914 : cluster [DBG] 2021-02-10T04:42:35.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:35 smithi085 conmon[29319]: osdmap e935: 8 total, 8 up, 7 in 2021-02-10T04:42:35.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:35 smithi085 conmon[29319]: cluster 2021-02-10T04:42:35.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:35 smithi085 conmon[29319]: 2021-02-10T04:42:35.010678 2021-02-10T04:42:35.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:35 smithi085 conmon[29319]: +0000 osd.0 (osd.0) 93 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:42:35.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:35 smithi085 conmon[29319]: 2021-02-10T04:42:35.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:35 smithi085 conmon[29319]: cluster 2021-02-10T04:42:35.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:35 smithi085 conmon[29319]: 2021-02-10T04:42:35.011804+0000 osd.0 (osd. 2021-02-10T04:42:35.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:35 smithi085 conmon[29319]: 0) 94 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:42:35.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:35 smithi085 conmon[35961]: cluster 2021-02-10T04:42:34.237138+0000 2021-02-10T04:42:35.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:35 smithi085 conmon[35961]: mgr.y (mgr.14140) 866 : cluster [DBG] pgmap v1552: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 798 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 8.0 MiB/s rd, 1.5 MiB/s wr, 15 op/s 2021-02-10T04:42:35.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:35 smithi085 conmon[35961]: cluster 2021-02-10T04:42:35.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:35 smithi085 conmon[35961]: 2021-02-10T04:42:34.407771+0000 mon.a (mon.0) 1914 : 2021-02-10T04:42:35.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:35 smithi085 conmon[35961]: cluster [DBG] osdmap e935: 8 total, 8 up, 7 in 2021-02-10T04:42:35.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:35 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:42:35.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:35 smithi085 conmon[35961]: 02-10T04:42:35.010678+0000 2021-02-10T04:42:35.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:35 smithi085 conmon[35961]: osd.0 (osd.0) 93 : cluster [DBG] 2021-02-10T04:42:35.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:35 smithi085 conmon[35961]: purged_snaps scrub starts 2021-02-10T04:42:35.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:35 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:42:35.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:35 smithi085 conmon[35961]: T04:42:35.011804+0000 osd.0 ( 2021-02-10T04:42:35.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:35 smithi085 conmon[35961]: osd.0) 94 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:42:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:35 smithi164 conmon[29288]: cluster 2021-02-10T04:42:34.237138+0000 mgr.y (mgr.14140) 866 : cluster [DBG] 2021-02-10T04:42:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:35 smithi164 conmon[29288]: pgmap v1552: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 798 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 8.0 MiB/s rd, 1.5 MiB/s wr, 15 op/s 2021-02-10T04:42:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:35 smithi164 conmon[29288]: cluster 2021-02-10T04:42:34.407771+0000 mon.a (mon.0) 1914 : cluster [DBG] osdmap e935: 8 total, 8 up, 7 in 2021-02-10T04:42:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:35 smithi164 conmon[29288]: cluster 2021-02-10T04:42:35.010678 2021-02-10T04:42:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:35 smithi164 conmon[29288]: +0000 osd.0 (osd.0) 93 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:42:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:35 smithi164 conmon[29288]: cluster 2021-02-10T04:42:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:35 smithi164 conmon[29288]: 2021-02-10T04:42:35. 2021-02-10T04:42:35.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:35 smithi164 conmon[29288]: 011804+0000 osd.0 (osd.0) 94 2021-02-10T04:42:35.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:35 smithi164 conmon[29288]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:42:36.436 INFO:tasks.rados.rados.0.smithi085.stdout:3751: finishing write tid 1 to smithi08575397-50 2021-02-10T04:42:36.436 INFO:tasks.rados.rados.0.smithi085.stdout:3751: finishing write tid 2 to smithi08575397-50 2021-02-10T04:42:36.436 INFO:tasks.rados.rados.0.smithi085.stdout:3751: finishing write tid 3 to smithi08575397-50 2021-02-10T04:42:36.436 INFO:tasks.rados.rados.0.smithi085.stdout:3751: finishing write tid 4 to smithi08575397-50 2021-02-10T04:42:36.436 INFO:tasks.rados.rados.0.smithi085.stdout:3751: finishing write tid 5 to smithi08575397-50 2021-02-10T04:42:36.436 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 1616 (ObjNum 1281 snap 433 seq_num 1281) dirty exists 2021-02-10T04:42:36.436 INFO:tasks.rados.rados.0.smithi085.stdout:3751: left oid 50 (ObjNum 1281 snap 433 seq_num 1281) 2021-02-10T04:42:36.436 INFO:tasks.rados.rados.0.smithi085.stdout:3751: done (5 left) 2021-02-10T04:42:36.436 INFO:tasks.rados.rados.0.smithi085.stdout:3752: done (4 left) 2021-02-10T04:42:36.436 INFO:tasks.rados.rados.0.smithi085.stdout:3753: write oid 22 current snap is 433 2021-02-10T04:42:36.437 INFO:tasks.rados.rados.0.smithi085.stdout:3753: seq_num 1282 ranges {746574=416522,1586386=468767,2772512=629544} 2021-02-10T04:42:36.442 INFO:tasks.rados.rados.0.smithi085.stdout:3753: writing smithi08575397-22 from 746574 to 1163096 tid 1 2021-02-10T04:42:36.447 INFO:tasks.rados.rados.0.smithi085.stdout:3753: writing smithi08575397-22 from 1586386 to 2055153 tid 2 2021-02-10T04:42:36.456 INFO:tasks.rados.rados.0.smithi085.stdout:3753: writing smithi08575397-22 from 2772512 to 3402056 tid 3 2021-02-10T04:42:36.456 INFO:tasks.rados.rados.0.smithi085.stdout:3754: rollback oid 13 current snap is 433 2021-02-10T04:42:36.456 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 13 to 428 2021-02-10T04:42:36.456 INFO:tasks.rados.rados.0.smithi085.stdout:3755: read oid 30 snap -1 2021-02-10T04:42:36.456 INFO:tasks.rados.rados.0.smithi085.stdout:3755: expect (ObjNum 1025 snap 335 seq_num 1025) 2021-02-10T04:42:36.457 INFO:tasks.rados.rados.0.smithi085.stdout:3756: write oid 2 current snap is 433 2021-02-10T04:42:36.457 INFO:tasks.rados.rados.0.smithi085.stdout:3756: seq_num 1283 ranges {763381=491771,2044868=728778,3199935=363446} 2021-02-10T04:42:36.464 INFO:tasks.rados.rados.0.smithi085.stdout:3756: writing smithi08575397-2 from 763381 to 1255152 tid 1 2021-02-10T04:42:36.473 INFO:tasks.rados.rados.0.smithi085.stdout:3756: writing smithi08575397-2 from 2044868 to 2773646 tid 2 2021-02-10T04:42:36.478 INFO:tasks.rados.rados.0.smithi085.stdout:3756: writing smithi08575397-2 from 3199935 to 3563381 tid 3 2021-02-10T04:42:36.479 INFO:tasks.rados.rados.0.smithi085.stdout:3757: rollback oid 5 current snap is 433 2021-02-10T04:42:36.479 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 5 to 425 2021-02-10T04:42:36.479 INFO:tasks.rados.rados.0.smithi085.stdout:3753: finishing write tid 1 to smithi08575397-22 2021-02-10T04:42:36.479 INFO:tasks.rados.rados.0.smithi085.stdout:3758: delete oid 7 current snap is 433 2021-02-10T04:42:36.479 INFO:tasks.rados.rados.0.smithi085.stdout:3753: finishing write tid 2 to smithi08575397-22 2021-02-10T04:42:36.479 INFO:tasks.rados.rados.0.smithi085.stdout:3755: expect (ObjNum 1025 snap 335 seq_num 1025) 2021-02-10T04:42:36.670 INFO:tasks.rados.rados.0.smithi085.stdout:3753: finishing write tid 3 to smithi08575397-22 2021-02-10T04:42:36.670 INFO:tasks.rados.rados.0.smithi085.stdout:3754: finishing rollback tid 0 to smithi08575397-13 2021-02-10T04:42:36.670 INFO:tasks.rados.rados.0.smithi085.stdout:3754: finishing rollback tid 1 to smithi08575397-13 2021-02-10T04:42:36.670 INFO:tasks.rados.rados.0.smithi085.stdout:3754: finishing rollback tid 2 to smithi08575397-13 2021-02-10T04:42:36.670 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 13 v 1576 (ObjNum 1241 snap 418 seq_num 1241) dirty exists 2021-02-10T04:42:36.670 INFO:tasks.rados.rados.0.smithi085.stdout:3753: finishing write tid 4 to smithi08575397-22 2021-02-10T04:42:36.670 INFO:tasks.rados.rados.0.smithi085.stdout:3753: finishing write tid 5 to smithi08575397-22 2021-02-10T04:42:36.671 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 1508 (ObjNum 1282 snap 433 seq_num 1282) dirty exists 2021-02-10T04:42:36.671 INFO:tasks.rados.rados.0.smithi085.stdout:3753: left oid 22 (ObjNum 1282 snap 433 seq_num 1282) 2021-02-10T04:42:36.671 INFO:tasks.rados.rados.0.smithi085.stdout:3756: finishing write tid 1 to smithi08575397-2 2021-02-10T04:42:36.671 INFO:tasks.rados.rados.0.smithi085.stdout:3756: finishing write tid 2 to smithi08575397-2 2021-02-10T04:42:36.671 INFO:tasks.rados.rados.0.smithi085.stdout:3756: finishing write tid 3 to smithi08575397-2 2021-02-10T04:42:36.671 INFO:tasks.rados.rados.0.smithi085.stdout:3753: done (9 left) 2021-02-10T04:42:36.671 INFO:tasks.rados.rados.0.smithi085.stdout:3754: done (8 left) 2021-02-10T04:42:36.671 INFO:tasks.rados.rados.0.smithi085.stdout:3755: done (7 left) 2021-02-10T04:42:36.671 INFO:tasks.rados.rados.0.smithi085.stdout:3758: done (6 left) 2021-02-10T04:42:36.671 INFO:tasks.rados.rados.0.smithi085.stdout:3759: write (excl) oid 26 current snap is 433 2021-02-10T04:42:36.671 INFO:tasks.rados.rados.0.smithi085.stdout:3756: finishing write tid 4 to smithi08575397-2 2021-02-10T04:42:36.671 INFO:tasks.rados.rados.0.smithi085.stdout:3756: finishing write tid 5 to smithi08575397-2 2021-02-10T04:42:36.671 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 2 v 1610 (ObjNum 1283 snap 433 seq_num 1283) dirty exists 2021-02-10T04:42:36.672 INFO:tasks.rados.rados.0.smithi085.stdout:3756: left oid 2 (ObjNum 1283 snap 433 seq_num 1283) 2021-02-10T04:42:36.672 INFO:tasks.rados.rados.0.smithi085.stdout:3757: finishing rollback tid 1 to smithi08575397-5 2021-02-10T04:42:36.672 INFO:tasks.rados.rados.0.smithi085.stdout:3757: finishing rollback tid 2 to smithi08575397-5 2021-02-10T04:42:36.672 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 1578 (ObjNum 1244 snap 419 seq_num 1244) dirty exists 2021-02-10T04:42:36.672 INFO:tasks.rados.rados.0.smithi085.stdout:3759: seq_num 1285 ranges {796995=758622,2161832=732447} 2021-02-10T04:42:36.678 INFO:tasks.rados.rados.0.smithi085.stdout:3759: writing smithi08575397-26 from 796995 to 1555617 tid 1 2021-02-10T04:42:36.686 INFO:tasks.rados.rados.0.smithi085.stdout:3759: writing smithi08575397-26 from 2161832 to 2894279 tid 2 2021-02-10T04:42:36.686 INFO:tasks.rados.rados.0.smithi085.stdout:3756: done (6 left) 2021-02-10T04:42:36.686 INFO:tasks.rados.rados.0.smithi085.stdout:3757: done (5 left) 2021-02-10T04:42:36.686 INFO:tasks.rados.rados.0.smithi085.stdout:3760: read oid 40 snap 354 2021-02-10T04:42:36.686 INFO:tasks.rados.rados.0.smithi085.stdout:3760: expect (ObjNum 1021 snap 335 seq_num 1021) 2021-02-10T04:42:36.686 INFO:tasks.rados.rados.0.smithi085.stdout:3761: copy_from oid 46 from oid 6 current snap is 433 2021-02-10T04:42:36.687 INFO:tasks.rados.rados.0.smithi085.stdout:3762: write oid 10 current snap is 433 2021-02-10T04:42:36.687 INFO:tasks.rados.rados.0.smithi085.stdout:3762: seq_num 1286 ranges {413802=433871,1420314=476105,2482204=790985} 2021-02-10T04:42:36.690 INFO:tasks.rados.rados.0.smithi085.stdout:3762: writing smithi08575397-10 from 413802 to 847673 tid 1 2021-02-10T04:42:36.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:36 smithi085 conmon[29319]: cluster 2021-02-10T04:42:35.418579+0000 mon.a ( 2021-02-10T04:42:36.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:36 smithi085 conmon[29319]: mon.0) 1915 : cluster [DBG] osdmap e936: 8 total, 8 up, 7 in 2021-02-10T04:42:36.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:36 smithi085 conmon[35961]: cluster 2021-02-10T04:42:35.418579+0000 mon.a (mon 2021-02-10T04:42:36.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:36 smithi085 conmon[35961]: .0) 1915 : cluster [DBG] osdmap e936: 8 total, 8 up, 7 in 2021-02-10T04:42:36.697 INFO:tasks.rados.rados.0.smithi085.stdout:3762: writing smithi08575397-10 from 1420314 to 1896419 tid 2 2021-02-10T04:42:36.704 INFO:tasks.rados.rados.0.smithi085.stdout:3762: writing smithi08575397-10 from 2482204 to 3273189 tid 3 2021-02-10T04:42:36.705 INFO:tasks.rados.rados.0.smithi085.stdout:3763: write (excl) oid 16 current snap is 433 2021-02-10T04:42:36.705 INFO:tasks.rados.rados.0.smithi085.stdout:3760: expect (ObjNum 1021 snap 335 seq_num 1021) 2021-02-10T04:42:36.892 INFO:tasks.rados.rados.0.smithi085.stdout:3763: seq_num 1287 ranges {430609=509120,1478796=736116,2909627=602257} 2021-02-10T04:42:36.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:36 smithi164 conmon[29288]: cluster 2021-02-10T04:42:36.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:36 smithi164 conmon[29288]: 2021-02 2021-02-10T04:42:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:36 smithi164 conmon[29288]: -10T04:42:35.418579+0000 mon.a (mon.0) 1915 : cluster [DBG] osdmap e936: 8 total, 8 up, 7 in 2021-02-10T04:42:36.897 INFO:tasks.rados.rados.0.smithi085.stdout:3763: writing smithi08575397-16 from 430609 to 939729 tid 1 2021-02-10T04:42:36.905 INFO:tasks.rados.rados.0.smithi085.stdout:3763: writing smithi08575397-16 from 1478796 to 2214912 tid 2 2021-02-10T04:42:36.914 INFO:tasks.rados.rados.0.smithi085.stdout:3763: writing smithi08575397-16 from 2909627 to 3511884 tid 3 2021-02-10T04:42:36.914 INFO:tasks.rados.rados.0.smithi085.stdout:3760: done (8 left) 2021-02-10T04:42:36.914 INFO:tasks.rados.rados.0.smithi085.stdout:3764: read oid 38 snap -1 2021-02-10T04:42:36.914 INFO:tasks.rados.rados.0.smithi085.stdout:3764: expect (ObjNum 1273 snap 428 seq_num 1273) 2021-02-10T04:42:36.914 INFO:tasks.rados.rados.0.smithi085.stdout:3765: copy_from oid 25 from oid 50 current snap is 433 2021-02-10T04:42:36.914 INFO:tasks.rados.rados.0.smithi085.stdout:3766: rollback oid 29 current snap is 433 2021-02-10T04:42:36.914 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 29 to 419 2021-02-10T04:42:36.914 INFO:tasks.rados.rados.0.smithi085.stdout:3767: write (excl) oid 31 current snap is 433 2021-02-10T04:42:36.915 INFO:tasks.rados.rados.0.smithi085.stdout:3767: seq_num 1288 ranges {447416=584369,1537278=479774,2537050=529882,3524784=122632} 2021-02-10T04:42:36.920 INFO:tasks.rados.rados.0.smithi085.stdout:3767: writing smithi08575397-31 from 447416 to 1031785 tid 1 2021-02-10T04:42:36.926 INFO:tasks.rados.rados.0.smithi085.stdout:3767: writing smithi08575397-31 from 1537278 to 2017052 tid 2 2021-02-10T04:42:36.932 INFO:tasks.rados.rados.0.smithi085.stdout:3767: writing smithi08575397-31 from 2537050 to 3066932 tid 3 2021-02-10T04:42:36.937 INFO:tasks.rados.rados.0.smithi085.stdout:3767: writing smithi08575397-31 from 3524784 to 3647416 tid 4 2021-02-10T04:42:36.937 INFO:tasks.rados.rados.0.smithi085.stdout:3768: write oid 23 current snap is 433 2021-02-10T04:42:36.937 INFO:tasks.rados.rados.0.smithi085.stdout:3759: finishing write tid 1 to smithi08575397-26 2021-02-10T04:42:36.937 INFO:tasks.rados.rados.0.smithi085.stdout:3762: finishing write tid 1 to smithi08575397-10 2021-02-10T04:42:36.937 INFO:tasks.rados.rados.0.smithi085.stdout:3762: finishing write tid 2 to smithi08575397-10 2021-02-10T04:42:36.937 INFO:tasks.rados.rados.0.smithi085.stdout:3762: finishing write tid 3 to smithi08575397-10 2021-02-10T04:42:36.937 INFO:tasks.rados.rados.0.smithi085.stdout:3762: finishing write tid 4 to smithi08575397-10 2021-02-10T04:42:36.937 INFO:tasks.rados.rados.0.smithi085.stdout:3762: finishing write tid 5 to smithi08575397-10 2021-02-10T04:42:36.937 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 10 v 1614 (ObjNum 1286 snap 433 seq_num 1286) dirty exists 2021-02-10T04:42:36.938 INFO:tasks.rados.rados.0.smithi085.stdout:3762: left oid 10 (ObjNum 1286 snap 433 seq_num 1286) 2021-02-10T04:42:36.938 INFO:tasks.rados.rados.0.smithi085.stdout:3759: finishing write tid 2 to smithi08575397-26 2021-02-10T04:42:36.938 INFO:tasks.rados.rados.0.smithi085.stdout:3759: finishing write tid 3 to smithi08575397-26 2021-02-10T04:42:36.938 INFO:tasks.rados.rados.0.smithi085.stdout:3759: finishing write tid 4 to smithi08575397-26 2021-02-10T04:42:36.938 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 1380 (ObjNum 1285 snap 433 seq_num 1285) dirty exists 2021-02-10T04:42:36.938 INFO:tasks.rados.rados.0.smithi085.stdout:3759: left oid 26 (ObjNum 1285 snap 433 seq_num 1285) 2021-02-10T04:42:36.938 INFO:tasks.rados.rados.0.smithi085.stdout:3761: finishing copy_from to smithi08575397-46 2021-02-10T04:42:36.938 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 1604 (ObjNum 1208 snap 410 seq_num 1208) dirty exists 2021-02-10T04:42:36.938 INFO:tasks.rados.rados.0.smithi085.stdout:3761: finishing copy_from racing read to smithi08575397-46 2021-02-10T04:42:36.938 INFO:tasks.rados.rados.0.smithi085.stdout:3763: finishing write tid 1 to smithi08575397-16 2021-02-10T04:42:36.938 INFO:tasks.rados.rados.0.smithi085.stdout:3763: finishing write tid 2 to smithi08575397-16 2021-02-10T04:42:36.938 INFO:tasks.rados.rados.0.smithi085.stdout:3766: finishing rollback tid 1 to smithi08575397-29 2021-02-10T04:42:36.938 INFO:tasks.rados.rados.0.smithi085.stdout:3766: finishing rollback tid 2 to smithi08575397-29 2021-02-10T04:42:36.939 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 29 v 1498 (ObjNum 8 snap 0 seq_num 8) dirty exists 2021-02-10T04:42:36.939 INFO:tasks.rados.rados.0.smithi085.stdout:3763: finishing write tid 3 to smithi08575397-16 2021-02-10T04:42:36.939 INFO:tasks.rados.rados.0.smithi085.stdout:3763: finishing write tid 4 to smithi08575397-16 2021-02-10T04:42:36.939 INFO:tasks.rados.rados.0.smithi085.stdout:3763: finishing write tid 5 to smithi08575397-16 2021-02-10T04:42:36.939 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 1916 (ObjNum 1287 snap 433 seq_num 1287) dirty exists 2021-02-10T04:42:36.939 INFO:tasks.rados.rados.0.smithi085.stdout:3763: left oid 16 (ObjNum 1287 snap 433 seq_num 1287) 2021-02-10T04:42:36.939 INFO:tasks.rados.rados.0.smithi085.stdout:3764: expect (ObjNum 1273 snap 428 seq_num 1273) 2021-02-10T04:42:37.113 INFO:tasks.rados.rados.0.smithi085.stdout:3767: finishing write tid 1 to smithi08575397-31 2021-02-10T04:42:37.113 INFO:tasks.rados.rados.0.smithi085.stdout:3765: finishing copy_from to smithi08575397-25 2021-02-10T04:42:37.114 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 25 v 1616 (ObjNum 1281 snap 433 seq_num 1281) dirty exists 2021-02-10T04:42:37.114 INFO:tasks.rados.rados.0.smithi085.stdout:3767: finishing write tid 2 to smithi08575397-31 2021-02-10T04:42:37.114 INFO:tasks.rados.rados.0.smithi085.stdout:3768: seq_num 1289 ranges {464223=659618,1879407=739785,3248120=416103} 2021-02-10T04:42:37.120 INFO:tasks.rados.rados.0.smithi085.stdout:3768: writing smithi08575397-23 from 464223 to 1123841 tid 1 2021-02-10T04:42:37.130 INFO:tasks.rados.rados.0.smithi085.stdout:3768: writing smithi08575397-23 from 1879407 to 2619192 tid 2 2021-02-10T04:42:37.137 INFO:tasks.rados.rados.0.smithi085.stdout:3768: writing smithi08575397-23 from 3248120 to 3664223 tid 3 2021-02-10T04:42:37.137 INFO:tasks.rados.rados.0.smithi085.stdout:3759: done (12 left) 2021-02-10T04:42:37.137 INFO:tasks.rados.rados.0.smithi085.stdout:3761: done (11 left) 2021-02-10T04:42:37.137 INFO:tasks.rados.rados.0.smithi085.stdout:3762: done (10 left) 2021-02-10T04:42:37.137 INFO:tasks.rados.rados.0.smithi085.stdout:3763: done (9 left) 2021-02-10T04:42:37.137 INFO:tasks.rados.rados.0.smithi085.stdout:3764: done (8 left) 2021-02-10T04:42:37.137 INFO:tasks.rados.rados.0.smithi085.stdout:3766: done (7 left) 2021-02-10T04:42:37.137 INFO:tasks.rados.rados.0.smithi085.stdout:3769: copy_from oid 12 from oid 49 current snap is 433 2021-02-10T04:42:37.138 INFO:tasks.rados.rados.0.smithi085.stdout:3765: finishing copy_from racing read to smithi08575397-25 2021-02-10T04:42:37.138 INFO:tasks.rados.rados.0.smithi085.stdout:3767: finishing write tid 3 to smithi08575397-31 2021-02-10T04:42:37.138 INFO:tasks.rados.rados.0.smithi085.stdout:3767: finishing write tid 4 to smithi08575397-31 2021-02-10T04:42:37.138 INFO:tasks.rados.rados.0.smithi085.stdout:3767: finishing write tid 5 to smithi08575397-31 2021-02-10T04:42:37.138 INFO:tasks.rados.rados.0.smithi085.stdout:3767: finishing write tid 6 to smithi08575397-31 2021-02-10T04:42:37.138 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 1513 (ObjNum 1288 snap 433 seq_num 1288) dirty exists 2021-02-10T04:42:37.138 INFO:tasks.rados.rados.0.smithi085.stdout:3767: left oid 31 (ObjNum 1288 snap 433 seq_num 1288) 2021-02-10T04:42:37.138 INFO:tasks.rados.rados.0.smithi085.stdout:3768: finishing write tid 1 to smithi08575397-23 2021-02-10T04:42:37.138 INFO:tasks.rados.rados.0.smithi085.stdout:3768: finishing write tid 2 to smithi08575397-23 2021-02-10T04:42:37.138 INFO:tasks.rados.rados.0.smithi085.stdout:3765: done (7 left) 2021-02-10T04:42:37.138 INFO:tasks.rados.rados.0.smithi085.stdout:3767: done (6 left) 2021-02-10T04:42:37.138 INFO:tasks.rados.rados.0.smithi085.stdout:3770: snap_create 2021-02-10T04:42:37.141 INFO:tasks.rados.rados.0.smithi085.stdout:3768: finishing write tid 3 to smithi08575397-23 2021-02-10T04:42:37.141 INFO:tasks.rados.rados.0.smithi085.stdout:3768: finishing write tid 4 to smithi08575397-23 2021-02-10T04:42:37.142 INFO:tasks.rados.rados.0.smithi085.stdout:3768: finishing write tid 5 to smithi08575397-23 2021-02-10T04:42:37.142 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 1493 (ObjNum 1289 snap 433 seq_num 1289) dirty exists 2021-02-10T04:42:37.143 INFO:tasks.rados.rados.0.smithi085.stdout:3768: left oid 23 (ObjNum 1289 snap 433 seq_num 1289) 2021-02-10T04:42:37.151 INFO:tasks.rados.rados.0.smithi085.stdout:3769: finishing copy_from to smithi08575397-12 2021-02-10T04:42:37.151 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 1590 (ObjNum 1097 snap 368 seq_num 1097) dirty exists 2021-02-10T04:42:37.151 INFO:tasks.rados.rados.0.smithi085.stdout:3769: finishing copy_from racing read to smithi08575397-12 2021-02-10T04:42:37.442 INFO:tasks.rados.rados.0.smithi085.stdout:3768: done (6 left) 2021-02-10T04:42:37.442 INFO:tasks.rados.rados.0.smithi085.stdout:3769: done (5 left) 2021-02-10T04:42:37.442 INFO:tasks.rados.rados.0.smithi085.stdout:3770: done (4 left) 2021-02-10T04:42:37.442 INFO:tasks.rados.rados.0.smithi085.stdout:3771: write (excl) oid 48 current snap is 434 2021-02-10T04:42:37.442 INFO:tasks.rados.rados.0.smithi085.stdout:3771: seq_num 1290 ranges {481030=734867,1937889=483443,2875543=552426} 2021-02-10T04:42:37.449 INFO:tasks.rados.rados.0.smithi085.stdout:3771: writing smithi08575397-48 from 481030 to 1215897 tid 1 2021-02-10T04:42:37.456 INFO:tasks.rados.rados.0.smithi085.stdout:3771: writing smithi08575397-48 from 1937889 to 2421332 tid 2 2021-02-10T04:42:37.462 INFO:tasks.rados.rados.0.smithi085.stdout:3771: writing smithi08575397-48 from 2875543 to 3427969 tid 3 2021-02-10T04:42:37.462 INFO:tasks.rados.rados.0.smithi085.stdout:3772: rollback oid 23 current snap is 434 2021-02-10T04:42:37.463 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 23 to 432 2021-02-10T04:42:37.463 INFO:tasks.rados.rados.0.smithi085.stdout:3771: finishing write tid 1 to smithi08575397-48 2021-02-10T04:42:37.463 INFO:tasks.rados.rados.0.smithi085.stdout:3773: snap_remove snap 421 2021-02-10T04:42:37.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:37 smithi085 conmon[35961]: cluster 2021-02-10T04:42:36.237668 2021-02-10T04:42:37.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:37 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 867 : cluster [DBG] pgmap v1555: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 804 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 8.0 MiB/s rd, 3.0 MiB/s wr, 19 op/s 2021-02-10T04:42:37.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:37 smithi085 conmon[35961]: cluster 2021-02-10T04:42:36.434457+0000 mon.a (mon.0) 1916 : cluster [DBG] osdmap e937: 8 total, 8 up, 7 in 2021-02-10T04:42:37.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:37 smithi085 conmon[35961]: 2021-02-10T04:42:37.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:37 smithi085 conmon[29319]: cluster 2021-02-10T04:42:36.237668+0000 mgr.y 2021-02-10T04:42:37.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:37 smithi085 conmon[29319]: (mgr.14140) 867 : cluster [DBG] pgmap v1555: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 804 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 8.0 MiB/s rd, 3.0 MiB/s wr, 19 op/s 2021-02-10T04:42:37.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:37 smithi085 conmon[29319]: cluster 2021-02-10T04:42:36.434457+0000 mon.a (mon.0) 1916 : cluster [DBG] osdmap e937: 8 total, 8 up, 7 in 2021-02-10T04:42:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:37 smithi164 conmon[29288]: cluster 2021-02-10T04:42:36.237668 2021-02-10T04:42:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:37 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 867 : cluster [DBG] pgmap v1555: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 804 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 8.0 MiB/s rd, 3.0 MiB/s wr, 19 op/s 2021-02-10T04:42:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:37 smithi164 conmon[29288]: cluster 2021-02-10T04:42:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:37 smithi164 conmon[29288]: 2021-02-10T04:42:36.434457+0000 mon.a (mon.0) 2021-02-10T04:42:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:37 smithi164 conmon[29288]: 1916 : cluster [DBG] osdmap e937: 8 total, 8 up, 7 in 2021-02-10T04:42:38.319 INFO:tasks.rados.rados.0.smithi085.stdout:3771: finishing write tid 2 to smithi08575397-48 2021-02-10T04:42:38.320 INFO:tasks.rados.rados.0.smithi085.stdout:3772: finishing rollback tid 0 to smithi08575397-23 2021-02-10T04:42:38.320 INFO:tasks.rados.rados.0.smithi085.stdout:3772: finishing rollback tid 1 to smithi08575397-23 2021-02-10T04:42:38.320 INFO:tasks.rados.rados.0.smithi085.stdout:3772: finishing rollback tid 2 to smithi08575397-23 2021-02-10T04:42:38.320 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 23 v 1496 (ObjNum 1224 snap 414 seq_num 1224) dirty exists 2021-02-10T04:42:38.320 INFO:tasks.rados.rados.0.smithi085.stdout:3771: finishing write tid 3 to smithi08575397-48 2021-02-10T04:42:38.320 INFO:tasks.rados.rados.0.smithi085.stdout:3771: finishing write tid 4 to smithi08575397-48 2021-02-10T04:42:38.320 INFO:tasks.rados.rados.0.smithi085.stdout:3771: finishing write tid 5 to smithi08575397-48 2021-02-10T04:42:38.320 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 1885 (ObjNum 1290 snap 434 seq_num 1290) dirty exists 2021-02-10T04:42:38.320 INFO:tasks.rados.rados.0.smithi085.stdout:3771: left oid 48 (ObjNum 1290 snap 434 seq_num 1290) 2021-02-10T04:42:38.320 INFO:tasks.rados.rados.0.smithi085.stdout:3771: done (6 left) 2021-02-10T04:42:38.320 INFO:tasks.rados.rados.0.smithi085.stdout:3772: done (5 left) 2021-02-10T04:42:38.320 INFO:tasks.rados.rados.0.smithi085.stdout:3773: done (4 left) 2021-02-10T04:42:38.320 INFO:tasks.rados.rados.0.smithi085.stdout:3774: copy_from oid 12 from oid 49 current snap is 434 2021-02-10T04:42:38.321 INFO:tasks.rados.rados.0.smithi085.stdout:3775: snap_create 2021-02-10T04:42:38.400 INFO:tasks.rados.rados.0.smithi085.stdout:3774: finishing copy_from to smithi08575397-12 2021-02-10T04:42:38.400 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 1590 (ObjNum 1097 snap 368 seq_num 1097) dirty exists 2021-02-10T04:42:38.400 INFO:tasks.rados.rados.0.smithi085.stdout:3774: finishing copy_from racing read to smithi08575397-12 2021-02-10T04:42:38.894 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:42:38 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:42:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:38 smithi164 conmon[29288]: cluster 2021-02-10T04:42:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:38 smithi164 conmon[29288]: 2021-02-10T04:42:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:38 smithi164 conmon[29288]: 2021 2021-02-10T04:42:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:38 smithi164 conmon[29288]: - 2021-02-10T04:42:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:38 smithi164 conmon[29288]: 02- 2021-02-10T04:42:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:38 smithi164 conmon[29288]: 10T04:42:37.440483+0000 mon.a (mon.0) 1917 : cluster [DBG] osdmap e938: 8 total, 8 up, 7 in 2021-02-10T04:42:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:38 smithi164 conmon[29288]: audit 2021-02-10T04:42:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:38 smithi164 conmon[29288]: 2021-02-10T04:42:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:38 smithi164 conmon[29288]: 2021 2021-02-10T04:42:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:38 smithi164 conmon[29288]: - 2021-02-10T04:42:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:38 smithi164 conmon[29288]: 02 2021-02-10T04:42:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:38 smithi164 conmon[29288]: - 2021-02-10T04:42:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:38 smithi164 conmon[29288]: 10T04:42:38.146373+0000 2021-02-10T04:42:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:38 smithi164 conmon[29288]: mon.a (mon.0) 1918 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:42:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:38 smithi164 conmon[29288]: audit 2021-02-10T04:42:38.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:38 smithi164 conmon[29288]: 2021-02-10T04:42:38.146669+0000 mon.a (mon.0) 1919 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:42:38.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:38 smithi164 conmon[29288]: cluster 2021-02-10T04:42:38.318322+0000 mon.a (mon. 2021-02-10T04:42:38.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:38 smithi164 conmon[29288]: 0) 1920 : cluster [DBG] osdmap e939: 8 total, 8 up, 7 in 2021-02-10T04:42:38.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:38 smithi085 conmon[29319]: cluster 2021-02-10T04:42:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:38 smithi085 conmon[29319]: 2021-02-10T04:42: 2021-02-10T04:42:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:38 smithi085 conmon[29319]: 37.440483+0000 mon.a (mon.0) 1917 : cluster [DBG] osdmap e938: 8 total, 8 up, 7 in 2021-02-10T04:42:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:38 smithi085 conmon[29319]: audit 2021-02-10T04:42 2021-02-10T04:42:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:38 smithi085 conmon[29319]: :38.146373+0000 mon.a (mon.0) 1918 : audit 2021-02-10T04:42:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:38 smithi085 conmon[29319]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:42:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:38 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:42:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:38 smithi085 conmon[29319]: T04:42:38.146669+0000 mon.a (mon. 2021-02-10T04:42:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:38 smithi085 conmon[29319]: 0) 1919 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:42:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:38 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:42:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:38 smithi085 conmon[29319]: -10T04:42:38.318322+0000 mon.a (mon 2021-02-10T04:42:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:38 smithi085 conmon[29319]: .0) 1920 : cluster [DBG] osdmap e939: 8 total, 8 up, 7 in 2021-02-10T04:42:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:38 smithi085 conmon[35961]: cluster 2021-02-10T04:42:37. 2021-02-10T04:42:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:38 smithi085 conmon[35961]: 440483+0000 mon.a (mon.0) 1917 : cluster [DBG] osdmap e938: 8 total, 8 up, 7 in 2021-02-10T04:42:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:38 smithi085 conmon[35961]: audit 2021-02-10T04:42:38. 2021-02-10T04:42:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:38 smithi085 conmon[35961]: 146373+0000 mon.a (mon.0) 1918 : audit 2021-02-10T04:42:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:38 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:42:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:38 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:42:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:38 smithi085 conmon[35961]: 42:38.146669+0000 mon.a (mon.0 2021-02-10T04:42:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:38 smithi085 conmon[35961]: ) 1919 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:42:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:38 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:42:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:38 smithi085 conmon[35961]: 02-10T04: 2021-02-10T04:42:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:38 smithi085 conmon[35961]: 42:38.318322 2021-02-10T04:42:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:38 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1920 : cluster [DBG] 2021-02-10T04:42:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:38 smithi085 conmon[35961]: osdmap e939: 8 total, 8 up, 7 in 2021-02-10T04:42:39.319 INFO:tasks.rados.rados.0.smithi085.stdout:3774: done (5 left) 2021-02-10T04:42:39.319 INFO:tasks.rados.rados.0.smithi085.stdout:3775: done (4 left) 2021-02-10T04:42:39.319 INFO:tasks.rados.rados.0.smithi085.stdout:3776: write (excl) oid 11 current snap is 435 2021-02-10T04:42:39.320 INFO:tasks.rados.rados.0.smithi085.stdout:3776: seq_num 1291 ranges {497837=410116,1596371=743454,2902966=763698} 2021-02-10T04:42:39.324 INFO:tasks.rados.rados.0.smithi085.stdout:3776: writing smithi08575397-11 from 497837 to 907953 tid 1 2021-02-10T04:42:39.334 INFO:tasks.rados.rados.0.smithi085.stdout:3776: writing smithi08575397-11 from 1596371 to 2339825 tid 2 2021-02-10T04:42:39.341 INFO:tasks.rados.rados.0.smithi085.stdout:3776: writing smithi08575397-11 from 2902966 to 3666664 tid 3 2021-02-10T04:42:39.341 INFO:tasks.rados.rados.0.smithi085.stdout:3776: finishing write tid 1 to smithi08575397-11 2021-02-10T04:42:39.341 INFO:tasks.rados.rados.0.smithi085.stdout:3776: finishing write tid 2 to smithi08575397-11 2021-02-10T04:42:39.342 INFO:tasks.rados.rados.0.smithi085.stdout:3777: snap_create 2021-02-10T04:42:39.348 INFO:tasks.rados.rados.0.smithi085.stdout:3776: finishing write tid 3 to smithi08575397-11 2021-02-10T04:42:39.348 INFO:tasks.rados.rados.0.smithi085.stdout:3776: finishing write tid 4 to smithi08575397-11 2021-02-10T04:42:39.348 INFO:tasks.rados.rados.0.smithi085.stdout:3776: finishing write tid 5 to smithi08575397-11 2021-02-10T04:42:39.348 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 11 v 1923 (ObjNum 1291 snap 435 seq_num 1291) dirty exists 2021-02-10T04:42:39.349 INFO:tasks.rados.rados.0.smithi085.stdout:3776: left oid 11 (ObjNum 1291 snap 435 seq_num 1291) 2021-02-10T04:42:39.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:39 smithi164 conmon[29288]: cluster 2021-02-10T04:42:38.238314+0000 mgr.y (mgr.14140) 868 : cluster [DBG] 2021-02-10T04:42:39.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:39 smithi164 conmon[29288]: pgmap v1558: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 809 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 767 B/s rd, 2.5 MiB/s wr, 8 op/s 2021-02-10T04:42:39.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:39 smithi164 conmon[29288]: audit 2021-02-10T04:42:38.528065+0000 mgr.y (mgr.14140) 869 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:42:39.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:39 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:42:39.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:39 smithi164 conmon[29288]: 42:39.319513+0000 mon.a (mon.0) 1921 : cluster [DBG] osdmap e940: 8 total, 8 up, 7 in 2021-02-10T04:42:39.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:39 smithi085 conmon[29319]: cluster 2021-02-10T04:42:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:39 smithi085 conmon[29319]: 2021-02-10T04:42:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:39 smithi085 conmon[29319]: 2021 2021-02-10T04:42:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:39 smithi085 conmon[29319]: -02-10T04: 2021-02-10T04:42:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:39 smithi085 conmon[29319]: 42:38.238314+0000 mgr.y (mgr.14140) 868 : cluster [DBG] pgmap v1558: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 809 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 767 B/s rd, 2.5 MiB/s wr, 8 op/s 2021-02-10T04:42:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:39 smithi085 conmon[29319]: audit 2021-02-10T04:42:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:39 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:42:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:39 smithi085 conmon[29319]: 42:38.528065+0000 mgr.y (mgr.14140) 869 : audit [DBG] 2021-02-10T04:42:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:39 smithi085 conmon[29319]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:42:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:39 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:42:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:39 smithi085 conmon[29319]: 02-10T04:42:39.319513+0000 mon.a (mon 2021-02-10T04:42:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:39 smithi085 conmon[29319]: .0) 1921 : cluster [DBG] osdmap e940: 8 total, 8 up, 7 in 2021-02-10T04:42:39.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:39 smithi085 conmon[35961]: cluster 2021-02-10T04:42:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:39 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:42:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:39 smithi085 conmon[35961]: 42:38.238314+0000 mgr.y (mgr.14140) 868 : cluster [DBG] pgmap v1558: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 809 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 767 B/s rd, 2.5 MiB/s wr, 8 op/s 2021-02-10T04:42:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:39 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:42:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:39 smithi085 conmon[35961]: -10T04:42:38.528065+0000 mgr.y (mgr. 2021-02-10T04:42:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:39 smithi085 conmon[35961]: 14140) 869 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:42:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:39 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:42:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:39 smithi085 conmon[35961]: 02-10T04:42:39.319513+0000 mon.a (mon 2021-02-10T04:42:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:39 smithi085 conmon[35961]: .0) 1921 : cluster [DBG] osdmap e940: 8 total, 8 up, 7 in 2021-02-10T04:42:40.084 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:42:40.119 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:40.324 INFO:tasks.rados.rados.0.smithi085.stdout:3776: done (5 left) 2021-02-10T04:42:40.324 INFO:tasks.rados.rados.0.smithi085.stdout:3777: done (4 left) 2021-02-10T04:42:40.324 INFO:tasks.rados.rados.0.smithi085.stdout:3778: copy_from oid 21 from oid 36 current snap is 436 2021-02-10T04:42:40.324 INFO:tasks.rados.rados.0.smithi085.stdout:3779: read oid 43 snap -1 2021-02-10T04:42:40.324 INFO:tasks.rados.rados.0.smithi085.stdout:3779: expect (ObjNum 1277 snap 430 seq_num 1277) 2021-02-10T04:42:40.324 INFO:tasks.rados.rados.0.smithi085.stdout:3780: write (excl) oid 46 current snap is 436 2021-02-10T04:42:40.325 INFO:tasks.rados.rados.0.smithi085.stdout:3780: seq_num 1292 ranges {514644=485365,1654853=487112,2930389=574970} 2021-02-10T04:42:40.326 INFO:tasks.rados.rados.0.smithi085.stdout:3780: writing smithi08575397-46 from 514644 to 1000009 tid 1 2021-02-10T04:42:40.333 INFO:tasks.rados.rados.0.smithi085.stdout:3780: writing smithi08575397-46 from 1654853 to 2141965 tid 2 2021-02-10T04:42:40.342 INFO:tasks.rados.rados.0.smithi085.stdout:3780: writing smithi08575397-46 from 2930389 to 3505359 tid 3 2021-02-10T04:42:40.342 INFO:tasks.rados.rados.0.smithi085.stdout:3779: expect (ObjNum 1277 snap 430 seq_num 1277) 2021-02-10T04:42:40.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:40 smithi164 conmon[33677]: debug 2021-02-10T04:42:40.349+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:40.547 INFO:tasks.rados.rados.0.smithi085.stdout:3780: finishing write tid 1 to smithi08575397-46 2021-02-10T04:42:40.547 INFO:tasks.rados.rados.0.smithi085.stdout:3778: finishing copy_from to smithi08575397-21 2021-02-10T04:42:40.548 INFO:tasks.rados.rados.0.smithi085.stdout:3778: got expected ENOENT (src dne) 2021-02-10T04:42:40.548 INFO:tasks.rados.rados.0.smithi085.stdout:3778: finishing copy_from racing read to smithi08575397-21 2021-02-10T04:42:40.548 INFO:tasks.rados.rados.0.smithi085.stdout:3780: finishing write tid 2 to smithi08575397-46 2021-02-10T04:42:40.548 INFO:tasks.rados.rados.0.smithi085.stdout:3778: done (6 left) 2021-02-10T04:42:40.548 INFO:tasks.rados.rados.0.smithi085.stdout:3779: done (5 left) 2021-02-10T04:42:40.548 INFO:tasks.rados.rados.0.smithi085.stdout:3781: read oid 3 snap -1 2021-02-10T04:42:40.548 INFO:tasks.rados.rados.0.smithi085.stdout:3781: expect (ObjNum 1247 snap 420 seq_num 1247) 2021-02-10T04:42:40.548 INFO:tasks.rados.rados.0.smithi085.stdout:3782: read oid 45 snap -1 2021-02-10T04:42:40.548 INFO:tasks.rados.rados.0.smithi085.stdout:3782: expect (ObjNum 1276 snap 430 seq_num 1276) 2021-02-10T04:42:40.548 INFO:tasks.rados.rados.0.smithi085.stdout:3783: read oid 40 snap 308 2021-02-10T04:42:40.548 INFO:tasks.rados.rados.0.smithi085.stdout:3783: expect (ObjNum 926 snap 307 seq_num 926) 2021-02-10T04:42:40.549 INFO:tasks.rados.rados.0.smithi085.stdout:3784: write (excl) oid 28 current snap is 436 2021-02-10T04:42:40.549 INFO:tasks.rados.rados.0.smithi085.stdout:3784: seq_num 1293 ranges {531451=676967,1713335=747123,2957812=502595} 2021-02-10T04:42:40.555 INFO:tasks.rados.rados.0.smithi085.stdout:3784: writing smithi08575397-28 from 531451 to 1208418 tid 1 2021-02-10T04:42:40.563 INFO:tasks.rados.rados.0.smithi085.stdout:3784: writing smithi08575397-28 from 1713335 to 2460458 tid 2 2021-02-10T04:42:40.569 INFO:tasks.rados.rados.0.smithi085.stdout:3784: writing smithi08575397-28 from 2957812 to 3460407 tid 3 2021-02-10T04:42:40.573 INFO:tasks.rados.rados.0.smithi085.stdout:3780: finishing write tid 3 to smithi08575397-46 2021-02-10T04:42:40.573 INFO:tasks.rados.rados.0.smithi085.stdout:3780: finishing write tid 4 to smithi08575397-46 2021-02-10T04:42:40.573 INFO:tasks.rados.rados.0.smithi085.stdout:3780: finishing write tid 5 to smithi08575397-46 2021-02-10T04:42:40.573 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 1881 (ObjNum 1292 snap 436 seq_num 1292) dirty exists 2021-02-10T04:42:40.574 INFO:tasks.rados.rados.0.smithi085.stdout:3780: left oid 46 (ObjNum 1292 snap 436 seq_num 1292) 2021-02-10T04:42:40.574 INFO:tasks.rados.rados.0.smithi085.stdout:3781: expect (ObjNum 1247 snap 420 seq_num 1247) 2021-02-10T04:42:40.750 INFO:tasks.rados.rados.0.smithi085.stdout:3783: expect (ObjNum 926 snap 307 seq_num 926) 2021-02-10T04:42:40.835 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:42:40.940 INFO:tasks.rados.rados.0.smithi085.stdout:3784: finishing write tid 1 to smithi08575397-28 2021-02-10T04:42:40.941 INFO:tasks.rados.rados.0.smithi085.stdout:3782: expect (ObjNum 1276 snap 430 seq_num 1276) 2021-02-10T04:42:40.942 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:42:40.974 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:41.125 INFO:tasks.rados.rados.0.smithi085.stdout:3780: done (8 left) 2021-02-10T04:42:41.125 INFO:tasks.rados.rados.0.smithi085.stdout:3781: done (7 left) 2021-02-10T04:42:41.125 INFO:tasks.rados.rados.0.smithi085.stdout:3782: done (6 left) 2021-02-10T04:42:41.125 INFO:tasks.rados.rados.0.smithi085.stdout:3783: done (5 left) 2021-02-10T04:42:41.125 INFO:tasks.rados.rados.0.smithi085.stdout:3785: delete oid 18 current snap is 436 2021-02-10T04:42:41.125 INFO:tasks.rados.rados.0.smithi085.stdout:3784: finishing write tid 2 to smithi08575397-28 2021-02-10T04:42:41.125 INFO:tasks.rados.rados.0.smithi085.stdout:3784: finishing write tid 3 to smithi08575397-28 2021-02-10T04:42:41.125 INFO:tasks.rados.rados.0.smithi085.stdout:3784: finishing write tid 4 to smithi08575397-28 2021-02-10T04:42:41.125 INFO:tasks.rados.rados.0.smithi085.stdout:3784: finishing write tid 5 to smithi08575397-28 2021-02-10T04:42:41.125 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 1463 (ObjNum 1293 snap 436 seq_num 1293) dirty exists 2021-02-10T04:42:41.125 INFO:tasks.rados.rados.0.smithi085.stdout:3784: left oid 28 (ObjNum 1293 snap 436 seq_num 1293) 2021-02-10T04:42:41.127 INFO:tasks.rados.rados.0.smithi085.stdout:3784: done (5 left) 2021-02-10T04:42:41.127 INFO:tasks.rados.rados.0.smithi085.stdout:3785: done (4 left) 2021-02-10T04:42:41.127 INFO:tasks.rados.rados.0.smithi085.stdout:3786: write (excl) oid 25 current snap is 436 2021-02-10T04:42:41.127 INFO:tasks.rados.rados.0.smithi085.stdout:3786: seq_num 1295 ranges {565065=427465,1430299=634439,2496305=525139,3472104=292961} 2021-02-10T04:42:41.133 INFO:tasks.rados.rados.0.smithi085.stdout:3786: writing smithi08575397-25 from 565065 to 992530 tid 1 2021-02-10T04:42:41.143 INFO:tasks.rados.rados.0.smithi085.stdout:3786: writing smithi08575397-25 from 1430299 to 2064738 tid 2 2021-02-10T04:42:41.144 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:42:41 smithi164 conmon[40892]: debug 2021-02-10T04:42:41.144 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:42:41 smithi164 conmon[40892]: 2021-02-10T04:42:41.055+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:41.145 INFO:tasks.rados.rados.0.smithi085.stdout:3786: writing smithi08575397-25 from 2496305 to 3021444 tid 3 2021-02-10T04:42:41.149 INFO:tasks.rados.rados.0.smithi085.stdout:3786: writing smithi08575397-25 from 3472104 to 3765065 tid 4 2021-02-10T04:42:41.150 INFO:tasks.rados.rados.0.smithi085.stdout:3786: finishing write tid 1 to smithi08575397-25 2021-02-10T04:42:41.150 INFO:tasks.rados.rados.0.smithi085.stdout:3786: finishing write tid 2 to smithi08575397-25 2021-02-10T04:42:41.150 INFO:tasks.rados.rados.0.smithi085.stdout:3786: finishing write tid 3 to smithi08575397-25 2021-02-10T04:42:41.150 INFO:tasks.rados.rados.0.smithi085.stdout:3787: read oid 36 snap -1 2021-02-10T04:42:41.150 INFO:tasks.rados.rados.0.smithi085.stdout:3787: expect deleted 2021-02-10T04:42:41.150 INFO:tasks.rados.rados.0.smithi085.stdout:3788: read oid 45 snap -1 2021-02-10T04:42:41.150 INFO:tasks.rados.rados.0.smithi085.stdout:3788: expect (ObjNum 1276 snap 430 seq_num 1276) 2021-02-10T04:42:41.150 INFO:tasks.rados.rados.0.smithi085.stdout:3789: write oid 38 current snap is 436 2021-02-10T04:42:41.150 INFO:tasks.rados.rados.0.smithi085.stdout:3789: seq_num 1296 ranges {581872=502714,1488781=494450,2640081=736411} 2021-02-10T04:42:41.156 INFO:tasks.rados.rados.0.smithi085.stdout:3789: writing smithi08575397-38 from 581872 to 1084586 tid 1 2021-02-10T04:42:41.162 INFO:tasks.rados.rados.0.smithi085.stdout:3789: writing smithi08575397-38 from 1488781 to 1983231 tid 2 2021-02-10T04:42:41.171 INFO:tasks.rados.rados.0.smithi085.stdout:3789: writing smithi08575397-38 from 2640081 to 3376492 tid 3 2021-02-10T04:42:41.171 INFO:tasks.rados.rados.0.smithi085.stdout:3790: write (excl) oid 42 current snap is 436 2021-02-10T04:42:41.172 INFO:tasks.rados.rados.0.smithi085.stdout:3790: seq_num 1297 ranges {598679=577963,1947263=638108,3351151=447528} 2021-02-10T04:42:41.179 INFO:tasks.rados.rados.0.smithi085.stdout:3790: writing smithi08575397-42 from 598679 to 1176642 tid 1 2021-02-10T04:42:41.188 INFO:tasks.rados.rados.0.smithi085.stdout:3790: writing smithi08575397-42 from 1947263 to 2585371 tid 2 2021-02-10T04:42:41.195 INFO:tasks.rados.rados.0.smithi085.stdout:3790: writing smithi08575397-42 from 3351151 to 3798679 tid 3 2021-02-10T04:42:41.195 INFO:tasks.rados.rados.0.smithi085.stdout:3791: copy_from oid 22 from oid 3 current snap is 436 2021-02-10T04:42:41.196 INFO:tasks.rados.rados.0.smithi085.stdout:3786: finishing write tid 4 to smithi08575397-25 2021-02-10T04:42:41.196 INFO:tasks.rados.rados.0.smithi085.stdout:3786: finishing write tid 5 to smithi08575397-25 2021-02-10T04:42:41.196 INFO:tasks.rados.rados.0.smithi085.stdout:3786: finishing write tid 6 to smithi08575397-25 2021-02-10T04:42:41.196 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 25 v 1622 (ObjNum 1295 snap 436 seq_num 1295) dirty exists 2021-02-10T04:42:41.196 INFO:tasks.rados.rados.0.smithi085.stdout:3786: left oid 25 (ObjNum 1295 snap 436 seq_num 1295) 2021-02-10T04:42:41.196 INFO:tasks.rados.rados.0.smithi085.stdout:3788: expect (ObjNum 1276 snap 430 seq_num 1276) 2021-02-10T04:42:41.309 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:42:41.392 INFO:tasks.rados.rados.0.smithi085.stdout:3789: finishing write tid 1 to smithi08575397-38 2021-02-10T04:42:41.392 INFO:tasks.rados.rados.0.smithi085.stdout:3789: finishing write tid 2 to smithi08575397-38 2021-02-10T04:42:41.392 INFO:tasks.rados.rados.0.smithi085.stdout:3789: finishing write tid 3 to smithi08575397-38 2021-02-10T04:42:41.392 INFO:tasks.rados.rados.0.smithi085.stdout:3789: finishing write tid 4 to smithi08575397-38 2021-02-10T04:42:41.392 INFO:tasks.rados.rados.0.smithi085.stdout:3789: finishing write tid 5 to smithi08575397-38 2021-02-10T04:42:41.392 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 1885 (ObjNum 1296 snap 436 seq_num 1296) dirty exists 2021-02-10T04:42:41.392 INFO:tasks.rados.rados.0.smithi085.stdout:3789: left oid 38 (ObjNum 1296 snap 436 seq_num 1296) 2021-02-10T04:42:41.392 INFO:tasks.rados.rados.0.smithi085.stdout:3790: finishing write tid 1 to smithi08575397-42 2021-02-10T04:42:41.392 INFO:tasks.rados.rados.0.smithi085.stdout:3790: finishing write tid 2 to smithi08575397-42 2021-02-10T04:42:41.392 INFO:tasks.rados.rados.0.smithi085.stdout:3790: finishing write tid 3 to smithi08575397-42 2021-02-10T04:42:41.392 INFO:tasks.rados.rados.0.smithi085.stdout:3790: finishing write tid 4 to smithi08575397-42 2021-02-10T04:42:41.392 INFO:tasks.rados.rados.0.smithi085.stdout:3790: finishing write tid 5 to smithi08575397-42 2021-02-10T04:42:41.393 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 42 v 1889 (ObjNum 1297 snap 436 seq_num 1297) dirty exists 2021-02-10T04:42:41.393 INFO:tasks.rados.rados.0.smithi085.stdout:3790: left oid 42 (ObjNum 1297 snap 436 seq_num 1297) 2021-02-10T04:42:41.393 INFO:tasks.rados.rados.0.smithi085.stdout:3786: done (9 left) 2021-02-10T04:42:41.393 INFO:tasks.rados.rados.0.smithi085.stdout:3787: done (8 left) 2021-02-10T04:42:41.393 INFO:tasks.rados.rados.0.smithi085.stdout:3788: done (7 left) 2021-02-10T04:42:41.393 INFO:tasks.rados.rados.0.smithi085.stdout:3789: done (6 left) 2021-02-10T04:42:41.393 INFO:tasks.rados.rados.0.smithi085.stdout:3790: done (5 left) 2021-02-10T04:42:41.393 INFO:tasks.rados.rados.0.smithi085.stdout:3792: snap_create 2021-02-10T04:42:41.420 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:42:41.458 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:41.572 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:42:41.576 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:42:41 smithi085 conmon[45443]: debug 2021-02-10T04:42:41.528+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:41.576 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:41 smithi085 conmon[29319]: cluster 2021-02-10T04:42:40. 2021-02-10T04:42:41.576 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:41 smithi085 conmon[29319]: 239117+0000 mgr.y (mgr.14140) 870 : cluster [DBG] pgmap v1561: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 831 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 9.8 MiB/s rd, 5.6 MiB/s wr, 26 op/s 2021-02-10T04:42:41.576 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:41 smithi085 conmon[29319]: cluster 2021-02-10T04:42:40.319934+0000 mon.a (mon.0) 1922 : cluster [DBG] osdmap e941: 8 total, 8 up, 7 in 2021-02-10T04:42:41.576 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:41 smithi085 conmon[35961]: cluster 2021-02-10T04:42: 2021-02-10T04:42:41.576 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:41 smithi085 conmon[35961]: 40.239117+0000 mgr.y (mgr.14140) 870 : cluster [DBG] pgmap v1561: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 831 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 9.8 MiB/s rd, 5.6 MiB/s wr, 26 op/s 2021-02-10T04:42:41.577 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:41 smithi085 conmon[35961]: cluster 2021-02-10T04:42:40.319934+0000 mon.a (mon.0) 1922 : cluster [DBG] osdmap e941: 8 total, 8 up, 7 in 2021-02-10T04:42:41.579 INFO:tasks.rados.rados.0.smithi085.stdout:3791: finishing copy_from to smithi08575397-22 2021-02-10T04:42:41.579 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 1882 (ObjNum 1247 snap 420 seq_num 1247) dirty exists 2021-02-10T04:42:41.580 INFO:tasks.rados.rados.0.smithi085.stdout:3791: finishing copy_from racing read to smithi08575397-22 2021-02-10T04:42:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:41 smithi164 conmon[29288]: cluster 2021-02-10T04:42:40.239117+0000 mgr.y (mgr.14140) 870 : cluster [DBG] pgmap v1561: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 831 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 9.8 MiB/s rd, 5.6 MiB/s wr, 26 op/s 2021-02-10T04:42:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:41 smithi164 conmon[29288]: cluster 2021-02-10T04:42:40.319934+0000 mon.a (mon.0) 1922 : cluster [DBG] osdmap e941: 8 total, 8 up, 7 in 2021-02-10T04:42:41.678 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:42:41.709 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:41.811 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:42:41.812 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:42:41.840 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:42:41 smithi085 conmon[49106]: debug 2021-02-10T04:42:41.840 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:42:41 smithi085 conmon[49106]: 2021-02-10T04:42:41.785+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:41.880 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:42:41.987 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:42:42.024 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:42.104 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:42:42 smithi085 conmon[49106]: debug 2021-02-10T04:42:42.102+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:42.332 INFO:tasks.rados.rados.0.smithi085.stdout:3791: done (5 left) 2021-02-10T04:42:42.332 INFO:tasks.rados.rados.0.smithi085.stdout:3792: done (4 left) 2021-02-10T04:42:42.332 INFO:tasks.rados.rados.0.smithi085.stdout:3793: snap_create 2021-02-10T04:42:42.338 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.7 to 1 (10000) 2021-02-10T04:42:42.343 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:42:42.365 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:42 smithi085 conmon[29319]: audit 2021-02-10T04:42:42.074297+0000 mon.a (mon. 2021-02-10T04:42:42.365 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:42 smithi085 conmon[29319]: 0) 1923 : audit [INF] from='client.? 172.21.15.85:0/2623954744' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T04:42:42.366 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:42 smithi085 conmon[35961]: audit 2021-02-10T04:42:42.074297 2021-02-10T04:42:42.366 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:42 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1923 : audit [INF] from='client.? 172.21.15.85:0/2623954744' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T04:42:42.600 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set nodeep-scrub 2021-02-10T04:42:42.617 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:42:42.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:42 smithi164 conmon[29288]: audit 2021-02-10T04:42:42.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:42 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:42:42.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:42 smithi164 conmon[29288]: 04:42:42.074297+0000 mon.a (mon.0) 1923 : audit [INF] from='client.? 172.21.15.85:0/2623954744' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T04:42:42.915 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:42:42.967 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:42:43.327 INFO:tasks.rados.rados.0.smithi085.stdout:3793: done (4 left) 2021-02-10T04:42:43.327 INFO:tasks.rados.rados.0.smithi085.stdout:3794: copy_from oid 34 from oid 21 current snap is 438 2021-02-10T04:42:43.327 INFO:tasks.rados.rados.0.smithi085.stdout:3795: write oid 36 current snap is 438 2021-02-10T04:42:43.327 INFO:tasks.rados.rados.0.smithi085.stdout:3795: seq_num 1298 ranges {615486=653212,1889392=498119,2978574=475308} 2021-02-10T04:42:43.333 INFO:tasks.rados.rados.0.smithi085.stdout:3795: writing smithi08575397-36 from 615486 to 1268698 tid 1 2021-02-10T04:42:43.340 INFO:tasks.rados.rados.0.smithi085.stdout:3795: writing smithi08575397-36 from 1889392 to 2387511 tid 2 2021-02-10T04:42:43.348 INFO:tasks.rados.rados.0.smithi085.stdout:3795: writing smithi08575397-36 from 2978574 to 3453882 tid 3 2021-02-10T04:42:43.348 INFO:tasks.rados.rados.0.smithi085.stdout:3795: finishing write tid 1 to smithi08575397-36 2021-02-10T04:42:43.348 INFO:tasks.rados.rados.0.smithi085.stdout:3795: finishing write tid 2 to smithi08575397-36 2021-02-10T04:42:43.348 INFO:tasks.rados.rados.0.smithi085.stdout:3796: read oid 44 snap -1 2021-02-10T04:42:43.349 INFO:tasks.rados.rados.0.smithi085.stdout:3796: expect deleted 2021-02-10T04:42:43.349 INFO:tasks.rados.rados.0.smithi085.stdout:3797: rollback oid 4 current snap is 438 2021-02-10T04:42:43.349 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 4 to 413 2021-02-10T04:42:43.349 INFO:tasks.rados.rados.0.smithi085.stdout:3798: copy_from oid 30 from oid 43 current snap is 438 2021-02-10T04:42:43.349 INFO:tasks.rados.rados.0.smithi085.stdout:3799: delete oid 40 current snap is 438 2021-02-10T04:42:43.352 INFO:tasks.rados.rados.0.smithi085.stdout:3794: finishing copy_from to smithi08575397-34 2021-02-10T04:42:43.352 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 34 v 1905 (ObjNum 1269 snap 428 seq_num 1269) dirty exists 2021-02-10T04:42:43.352 INFO:tasks.rados.rados.0.smithi085.stdout:3794: finishing copy_from racing read to smithi08575397-34 2021-02-10T04:42:43.353 INFO:tasks.rados.rados.0.smithi085.stdout:3794: done (9 left) 2021-02-10T04:42:43.353 INFO:tasks.rados.rados.0.smithi085.stdout:3796: done (8 left) 2021-02-10T04:42:43.353 INFO:tasks.rados.rados.0.smithi085.stdout:3799: done (7 left) 2021-02-10T04:42:43.354 INFO:tasks.rados.rados.0.smithi085.stdout:3800: read oid 13 snap -1 2021-02-10T04:42:43.354 INFO:tasks.rados.rados.0.smithi085.stdout:3800: expect (ObjNum 1241 snap 418 seq_num 1241) 2021-02-10T04:42:43.354 INFO:tasks.rados.rados.0.smithi085.stdout:3801: write oid 1 current snap is 438 2021-02-10T04:42:43.354 INFO:tasks.rados.rados.0.smithi085.stdout:3801: seq_num 1300 ranges {649100=403710,1606356=501788,2633420=497852,3636005=213095} 2021-02-10T04:42:43.360 INFO:tasks.rados.rados.0.smithi085.stdout:3801: writing smithi08575397-1 from 649100 to 1052810 tid 1 2021-02-10T04:42:43.366 INFO:tasks.rados.rados.0.smithi085.stdout:3801: writing smithi08575397-1 from 1606356 to 2108144 tid 2 2021-02-10T04:42:43.373 INFO:tasks.rados.rados.0.smithi085.stdout:3801: writing smithi08575397-1 from 2633420 to 3131272 tid 3 2021-02-10T04:42:43.378 INFO:tasks.rados.rados.0.smithi085.stdout:3801: writing smithi08575397-1 from 3636005 to 3849100 tid 4 2021-02-10T04:42:43.378 INFO:tasks.rados.rados.0.smithi085.stdout:3802: read oid 23 snap -1 2021-02-10T04:42:43.378 INFO:tasks.rados.rados.0.smithi085.stdout:3802: expect (ObjNum 1224 snap 414 seq_num 1224) 2021-02-10T04:42:43.378 INFO:tasks.rados.rados.0.smithi085.stdout:3803: rollback oid 49 current snap is 438 2021-02-10T04:42:43.378 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 49 to 381 2021-02-10T04:42:43.378 INFO:tasks.rados.rados.0.smithi085.stdout:3797: finishing rollback tid 0 to smithi08575397-4 2021-02-10T04:42:43.379 INFO:tasks.rados.rados.0.smithi085.stdout:3795: finishing write tid 3 to smithi08575397-36 2021-02-10T04:42:43.379 INFO:tasks.rados.rados.0.smithi085.stdout:3797: finishing rollback tid 1 to smithi08575397-4 2021-02-10T04:42:43.379 INFO:tasks.rados.rados.0.smithi085.stdout:3795: finishing write tid 4 to smithi08575397-36 2021-02-10T04:42:43.379 INFO:tasks.rados.rados.0.smithi085.stdout:3804: rollback oid 10 current snap is 438 2021-02-10T04:42:43.379 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 10 to 413 2021-02-10T04:42:43.379 INFO:tasks.rados.rados.0.smithi085.stdout:3795: finishing write tid 5 to smithi08575397-36 2021-02-10T04:42:43.379 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 1500 (ObjNum 1298 snap 438 seq_num 1298) dirty exists 2021-02-10T04:42:43.379 INFO:tasks.rados.rados.0.smithi085.stdout:3795: left oid 36 (ObjNum 1298 snap 438 seq_num 1298) 2021-02-10T04:42:43.379 INFO:tasks.rados.rados.0.smithi085.stdout:3797: finishing rollback tid 2 to smithi08575397-4 2021-02-10T04:42:43.379 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 4 v 1383 (ObjNum 8 snap 0 seq_num 8) dirty exists 2021-02-10T04:42:43.379 INFO:tasks.rados.rados.0.smithi085.stdout:3800: expect (ObjNum 1241 snap 418 seq_num 1241) 2021-02-10T04:42:43.556 INFO:tasks.rados.rados.0.smithi085.stdout:3801: finishing write tid 1 to smithi08575397-1 2021-02-10T04:42:43.556 INFO:tasks.rados.rados.0.smithi085.stdout:3801: finishing write tid 2 to smithi08575397-1 2021-02-10T04:42:43.557 INFO:tasks.rados.rados.0.smithi085.stdout:3801: finishing write tid 3 to smithi08575397-1 2021-02-10T04:42:43.557 INFO:tasks.rados.rados.0.smithi085.stdout:3795: done (11 left) 2021-02-10T04:42:43.557 INFO:tasks.rados.rados.0.smithi085.stdout:3797: done (10 left) 2021-02-10T04:42:43.557 INFO:tasks.rados.rados.0.smithi085.stdout:3800: done (9 left) 2021-02-10T04:42:43.557 INFO:tasks.rados.rados.0.smithi085.stdout:3805: read oid 9 snap -1 2021-02-10T04:42:43.557 INFO:tasks.rados.rados.0.smithi085.stdout:3805: expect (ObjNum 1134 snap 381 seq_num 1134) 2021-02-10T04:42:43.557 INFO:tasks.rados.rados.0.smithi085.stdout:3806: rollback oid 16 current snap is 438 2021-02-10T04:42:43.557 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 16 to 418 2021-02-10T04:42:43.557 INFO:tasks.rados.rados.0.smithi085.stdout:3803: finishing rollback tid 0 to smithi08575397-49 2021-02-10T04:42:43.557 INFO:tasks.rados.rados.0.smithi085.stdout:3801: finishing write tid 4 to smithi08575397-1 2021-02-10T04:42:43.557 INFO:tasks.rados.rados.0.smithi085.stdout:3801: finishing write tid 5 to smithi08575397-1 2021-02-10T04:42:43.557 INFO:tasks.rados.rados.0.smithi085.stdout:3801: finishing write tid 6 to smithi08575397-1 2021-02-10T04:42:43.557 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 1621 (ObjNum 1300 snap 438 seq_num 1300) dirty exists 2021-02-10T04:42:43.558 INFO:tasks.rados.rados.0.smithi085.stdout:3801: left oid 1 (ObjNum 1300 snap 438 seq_num 1300) 2021-02-10T04:42:43.558 INFO:tasks.rados.rados.0.smithi085.stdout:3801: done (10 left) 2021-02-10T04:42:43.558 INFO:tasks.rados.rados.0.smithi085.stdout:3807: snap_remove snap 433 2021-02-10T04:42:43.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[35961]: cluster 2021-02-10T04:42:43.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[35961]: 2021-02-10T04:42:43.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[35961]: 2021 2021-02-10T04:42:43.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[35961]: -02-10T04 2021-02-10T04:42:43.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[35961]: :42:42.239892+0000 mgr.y (mgr.14140) 871 : cluster [DBG] pgmap v1563: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 831 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 8.2 MiB/s rd, 4.7 MiB/s wr, 21 op/s 2021-02-10T04:42:43.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:42:43.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[35961]: :42:42.330089+0000 mon.a (mon.0) 1924 : audit 2021-02-10T04:42:43.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[35961]: [INF] from='client.? 172.21.15.85:0/2623954744' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-02-10T04:42:43.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[35961]: cluster 2021-02-10T04:42:42.330180 2021-02-10T04:42:43.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1925 : cluster [DBG] osdmap e942: 8 total, 8 up, 7 in 2021-02-10T04:42:43.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:42:43.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[35961]: T04:42:42.874293+0000 osd.7 ( 2021-02-10T04:42:43.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[35961]: osd.7) 66 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:42:43.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:42:43.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[35961]: 02-10T04:42:42.874686+0000 osd.7 2021-02-10T04:42:43.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[35961]: (osd.7) 67 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:42:43.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:42:43.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[35961]: 10T04:42:43.319954+0000 mon.a 2021-02-10T04:42:43.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[35961]: (mon.0) 1926 : cluster [DBG] osdmap e943: 8 total, 8 up, 7 in 2021-02-10T04:42:43.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: cluster 2021-02-10T04:42:43.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: 2021-02-10T04:42:43.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: 2021 2021-02-10T04:42:43.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: - 2021-02-10T04:42:43.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: 02 2021-02-10T04:42:43.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: -10T 2021-02-10T04:42:43.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: 04:42:42.239892+0000 mgr.y (mgr.14140) 871 : cluster [DBG] 2021-02-10T04:42:43.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: pgmap v1563: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 831 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 8.2 MiB/s rd, 4.7 MiB/s wr, 21 op/s 2021-02-10T04:42:43.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: audit 2021-02-10T04:42:43.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: 2021- 2021-02-10T04:42:43.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: 02 2021-02-10T04:42:43.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: - 2021-02-10T04:42:43.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: 10 2021-02-10T04:42:43.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: T 2021-02-10T04:42:43.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: 04 2021-02-10T04:42:43.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: : 2021-02-10T04:42:43.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: 42 2021-02-10T04:42:43.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: : 2021-02-10T04:42:43.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: 42 2021-02-10T04:42:43.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: . 2021-02-10T04:42:43.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: 330089 2021-02-10T04:42:43.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: +0000 2021-02-10T04:42:43.698 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: mon.a (mon.0) 1924 2021-02-10T04:42:43.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: : audit [INF] from='client.? 172.21.15.85:0/2623954744' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-02-10T04:42:43.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:42:43.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: 02-10 2021-02-10T04:42:43.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: T04:42:42.330180+0000 mon.a (mon.0) 1925 : cluster [DBG] 2021-02-10T04:42:43.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: osdmap e942: 8 total, 8 up, 7 in 2021-02-10T04:42:43.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:42:43.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: 02-10T04: 2021-02-10T04:42:43.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: 42:42.874293+0000 osd.7 (osd.7) 66 : cluster 2021-02-10T04:42:43.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: [DBG] purged_snaps scrub starts 2021-02-10T04:42:43.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:42:43.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: 42:42.874686+0000 osd.7 (osd.7) 67 2021-02-10T04:42:43.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:42:43.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:42:43.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: 02 2021-02-10T04:42:43.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: - 2021-02-10T04:42:43.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: 10 2021-02-10T04:42:43.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: T 2021-02-10T04:42:43.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: 04 2021-02-10T04:42:43.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: : 2021-02-10T04:42:43.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: 42 2021-02-10T04:42:43.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: :43.319954 2021-02-10T04:42:43.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1926 2021-02-10T04:42:43.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:43 smithi085 conmon[29319]: : cluster [DBG] osdmap e943: 8 total, 8 up, 7 in 2021-02-10T04:42:43.957 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:43 smithi164 conmon[29288]: cluster 2021-02-10T04:42:43.957 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:43 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:42:43.957 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:43 smithi164 conmon[29288]: :42:42.239892+0000 mgr.y (mgr.14140) 871 : cluster [DBG] pgmap v1563: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 831 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 8.2 MiB/s rd, 4.7 MiB/s wr, 21 op/s 2021-02-10T04:42:43.958 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:43 smithi164 conmon[29288]: audit 2021-02-10T04:42:43.958 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:43 smithi164 conmon[29288]: 2021-02-10T04:42:43.958 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:43 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:42:43.958 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:43 smithi164 conmon[29288]: 42:42.330089+0000 mon.a (mon.0) 1924 : audit [INF] from='client.? 172.21.15.85:0/2623954744' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-02-10T04:42:43.958 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:43 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:42:43.959 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:43 smithi164 conmon[29288]: 10T04:42:42.330180+0000 mon.a (mon.0 2021-02-10T04:42:43.959 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:43 smithi164 conmon[29288]: ) 1925 : cluster [DBG] osdmap e942: 8 total, 8 up, 7 in 2021-02-10T04:42:43.959 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:43 smithi164 conmon[29288]: cluster 2021-02-10T04:42:42.874293+0000 osd.7 (osd.7) 66 : 2021-02-10T04:42:43.959 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:43 smithi164 conmon[29288]: cluster [DBG] purged_snaps scrub starts 2021-02-10T04:42:43.959 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:43 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:42:43.960 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:43 smithi164 conmon[29288]: 02-10T04:42:42.874686+0000 osd.7 2021-02-10T04:42:43.960 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:43 smithi164 conmon[29288]: (osd.7) 67 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:42:43.960 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:43 smithi164 conmon[29288]: cluster 2021-02-10T04:42:43.960 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:43 smithi164 conmon[29288]: 2021-02-10T04:42:43.319954+0000 2021-02-10T04:42:43.960 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:43 smithi164 conmon[29288]: mon.a (mon. 2021-02-10T04:42:43.960 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:43 smithi164 conmon[29288]: 0) 1926 : cluster [DBG] osdmap e943: 8 total, 8 up, 7 in 2021-02-10T04:42:44.341 INFO:tasks.rados.rados.0.smithi085.stdout:3807: done (10 left) 2021-02-10T04:42:44.341 INFO:tasks.rados.rados.0.smithi085.stdout:3808: snap_create 2021-02-10T04:42:44.341 INFO:tasks.rados.rados.0.smithi085.stdout:3802: expect (ObjNum 1224 snap 414 seq_num 1224) 2021-02-10T04:42:44.605 INFO:tasks.rados.rados.0.smithi085.stdout:3803: finishing rollback tid 1 to smithi08575397-49 2021-02-10T04:42:44.605 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 49 v 1616 (ObjNum 0 snap 3 seq_num 7) dirty dne 2021-02-10T04:42:44.606 INFO:tasks.rados.rados.0.smithi085.stdout:3804: finishing rollback tid 0 to smithi08575397-10 2021-02-10T04:42:44.606 INFO:tasks.rados.rados.0.smithi085.stdout:3804: finishing rollback tid 1 to smithi08575397-10 2021-02-10T04:42:44.606 INFO:tasks.rados.rados.0.smithi085.stdout:3804: finishing rollback tid 2 to smithi08575397-10 2021-02-10T04:42:44.606 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 10 v 1619 (ObjNum 1167 snap 401 seq_num 1167) dirty exists 2021-02-10T04:42:44.606 INFO:tasks.rados.rados.0.smithi085.stdout:3798: finishing copy_from to smithi08575397-30 2021-02-10T04:42:44.606 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 1886 (ObjNum 1277 snap 430 seq_num 1277) dirty exists 2021-02-10T04:42:44.606 INFO:tasks.rados.rados.0.smithi085.stdout:3798: finishing copy_from racing read to smithi08575397-30 2021-02-10T04:42:44.606 INFO:tasks.rados.rados.0.smithi085.stdout:3806: finishing rollback tid 0 to smithi08575397-16 2021-02-10T04:42:44.606 INFO:tasks.rados.rados.0.smithi085.stdout:3806: finishing rollback tid 1 to smithi08575397-16 2021-02-10T04:42:44.606 INFO:tasks.rados.rados.0.smithi085.stdout:3806: finishing rollback tid 2 to smithi08575397-16 2021-02-10T04:42:44.607 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 16 v 1935 (ObjNum 1191 snap 407 seq_num 1191) dirty exists 2021-02-10T04:42:44.607 INFO:tasks.rados.rados.0.smithi085.stdout:3805: expect (ObjNum 1134 snap 381 seq_num 1134) 2021-02-10T04:42:45.347 INFO:tasks.rados.rados.0.smithi085.stdout:3798: done (10 left) 2021-02-10T04:42:45.348 INFO:tasks.rados.rados.0.smithi085.stdout:3802: done (9 left) 2021-02-10T04:42:45.348 INFO:tasks.rados.rados.0.smithi085.stdout:3803: done (8 left) 2021-02-10T04:42:45.348 INFO:tasks.rados.rados.0.smithi085.stdout:3804: done (7 left) 2021-02-10T04:42:45.348 INFO:tasks.rados.rados.0.smithi085.stdout:3805: done (6 left) 2021-02-10T04:42:45.348 INFO:tasks.rados.rados.0.smithi085.stdout:3806: done (5 left) 2021-02-10T04:42:45.348 INFO:tasks.rados.rados.0.smithi085.stdout:3808: done (4 left) 2021-02-10T04:42:45.348 INFO:tasks.rados.rados.0.smithi085.stdout:3809: copy_from oid 43 from oid 46 current snap is 439 2021-02-10T04:42:45.348 INFO:tasks.rados.rados.0.smithi085.stdout:3810: write (excl) oid 36 current snap is 439 2021-02-10T04:42:45.348 INFO:tasks.rados.rados.0.smithi085.stdout:3810: seq_num 1301 ranges {665907=595312,1781191=645446,3177196=688711} 2021-02-10T04:42:45.355 INFO:tasks.rados.rados.0.smithi085.stdout:3810: writing smithi08575397-36 from 665907 to 1261219 tid 1 2021-02-10T04:42:45.362 INFO:tasks.rados.rados.0.smithi085.stdout:3810: writing smithi08575397-36 from 1781191 to 2426637 tid 2 2021-02-10T04:42:45.371 INFO:tasks.rados.rados.0.smithi085.stdout:3810: writing smithi08575397-36 from 3177196 to 3865907 tid 3 2021-02-10T04:42:45.372 INFO:tasks.rados.rados.0.smithi085.stdout:3810: finishing write tid 1 to smithi08575397-36 2021-02-10T04:42:45.372 INFO:tasks.rados.rados.0.smithi085.stdout:3810: finishing write tid 2 to smithi08575397-36 2021-02-10T04:42:45.372 INFO:tasks.rados.rados.0.smithi085.stdout:3811: snap_create 2021-02-10T04:42:45.379 INFO:tasks.rados.rados.0.smithi085.stdout:3810: finishing write tid 3 to smithi08575397-36 2021-02-10T04:42:45.379 INFO:tasks.rados.rados.0.smithi085.stdout:3810: finishing write tid 4 to smithi08575397-36 2021-02-10T04:42:45.379 INFO:tasks.rados.rados.0.smithi085.stdout:3810: finishing write tid 5 to smithi08575397-36 2021-02-10T04:42:45.379 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 1504 (ObjNum 1301 snap 439 seq_num 1301) dirty exists 2021-02-10T04:42:45.379 INFO:tasks.rados.rados.0.smithi085.stdout:3810: left oid 36 (ObjNum 1301 snap 439 seq_num 1301) 2021-02-10T04:42:45.385 INFO:tasks.rados.rados.0.smithi085.stdout:3809: finishing copy_from to smithi08575397-43 2021-02-10T04:42:45.385 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 1881 (ObjNum 1292 snap 436 seq_num 1292) dirty exists 2021-02-10T04:42:45.385 INFO:tasks.rados.rados.0.smithi085.stdout:3809: finishing copy_from racing read to smithi08575397-43 2021-02-10T04:42:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:45 smithi164 conmon[29288]: cluster 2021-02-10T04:42:44.240552+0000 mgr.y (mgr 2021-02-10T04:42:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:45 smithi164 conmon[29288]: .14140) 872 : cluster [DBG] pgmap v1566: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 838 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 4.5 MiB/s wr, 22 op/s 2021-02-10T04:42:45.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:45 smithi164 conmon[29288]: cluster 2021-02-10T04:42:44.340989+0000 mon.a (mon.0) 1927 : cluster [DBG] osdmap e944: 8 total, 8 up, 7 in 2021-02-10T04:42:45.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:45 smithi085 conmon[35961]: cluster 2021-02-10T04:42:45.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:45 smithi085 conmon[35961]: 2021-02-10T04:42:45.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:45 smithi085 conmon[35961]: 2021 2021-02-10T04:42:45.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:45 smithi085 conmon[35961]: - 2021-02-10T04:42:45.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:45 smithi085 conmon[35961]: 02 2021-02-10T04:42:45.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:45 smithi085 conmon[35961]: - 2021-02-10T04:42:45.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:45 smithi085 conmon[35961]: 10 2021-02-10T04:42:45.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:45 smithi085 conmon[35961]: T 2021-02-10T04:42:45.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:45 smithi085 conmon[35961]: 04 2021-02-10T04:42:45.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:45 smithi085 conmon[35961]: : 2021-02-10T04:42:45.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:45 smithi085 conmon[35961]: 42 2021-02-10T04:42:45.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:45 smithi085 conmon[35961]: : 2021-02-10T04:42:45.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:45 smithi085 conmon[35961]: 44 2021-02-10T04:42:45.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:45 smithi085 conmon[35961]: . 2021-02-10T04:42:45.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:45 smithi085 conmon[35961]: 240552 2021-02-10T04:42:45.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:45 smithi085 conmon[35961]: +0000 2021-02-10T04:42:45.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:45 smithi085 conmon[35961]: 2021-02-10T04:42:45.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:45 smithi085 conmon[35961]: mgr.y 2021-02-10T04:42:45.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:45 smithi085 conmon[35961]: ( 2021-02-10T04:42:45.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:45 smithi085 conmon[35961]: mgr 2021-02-10T04:42:45.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:45 smithi085 conmon[35961]: . 2021-02-10T04:42:45.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:45 smithi085 conmon[35961]: 14140) 872 : cluster [DBG] 2021-02-10T04:42:45.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:45 smithi085 conmon[35961]: pgmap v1566: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 838 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 4.5 MiB/s wr, 22 op/s 2021-02-10T04:42:45.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:45 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:42:45.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:45 smithi085 conmon[35961]: -02-10T04:42:44.340989+0000 2021-02-10T04:42:45.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:45 smithi085 conmon[35961]: mon.a (mon.0) 1927 : cluster [DBG] 2021-02-10T04:42:45.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:45 smithi085 conmon[35961]: osdmap e944: 8 total, 8 up, 7 in 2021-02-10T04:42:45.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:45 smithi085 conmon[29319]: cluster 2021-02-10T04:42:44.240552+0000 mgr.y ( 2021-02-10T04:42:45.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:45 smithi085 conmon[29319]: mgr.14140) 872 : cluster [DBG] pgmap v1566: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 838 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 4.5 MiB/s wr, 22 op/s 2021-02-10T04:42:45.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:45 smithi085 conmon[29319]: cluster 2021-02-10T04:42:44.340989+0000 mon.a (mon.0) 1927 : cluster [DBG] osdmap e944: 8 total, 8 up, 7 in 2021-02-10T04:42:46.364 INFO:tasks.rados.rados.0.smithi085.stdout:3809: done (6 left) 2021-02-10T04:42:46.365 INFO:tasks.rados.rados.0.smithi085.stdout:3810: done (5 left) 2021-02-10T04:42:46.365 INFO:tasks.rados.rados.0.smithi085.stdout:3811: done (4 left) 2021-02-10T04:42:46.365 INFO:tasks.rados.rados.0.smithi085.stdout:3812: read oid 42 snap 426 2021-02-10T04:42:46.365 INFO:tasks.rados.rados.0.smithi085.stdout:3812: expect deleted 2021-02-10T04:42:46.365 INFO:tasks.rados.rados.0.smithi085.stdout:3813: delete oid 28 current snap is 440 2021-02-10T04:42:46.369 INFO:tasks.rados.rados.0.smithi085.stdout:3812: done (5 left) 2021-02-10T04:42:46.369 INFO:tasks.rados.rados.0.smithi085.stdout:3813: done (4 left) 2021-02-10T04:42:46.369 INFO:tasks.rados.rados.0.smithi085.stdout:3814: delete oid 30 current snap is 440 2021-02-10T04:42:46.373 INFO:tasks.rados.rados.0.smithi085.stdout:3814: done (4 left) 2021-02-10T04:42:46.374 INFO:tasks.rados.rados.0.smithi085.stdout:3815: snap_create 2021-02-10T04:42:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:46 smithi164 conmon[29288]: cluster 2021-02-10T04:42:45.345571+0000 mon.a ( 2021-02-10T04:42:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:46 smithi164 conmon[29288]: mon.0) 1928 : cluster [DBG] osdmap e945: 8 total, 8 up, 7 in 2021-02-10T04:42:46.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:46 smithi085 conmon[29319]: cluster 2021-02-10T04:42:46.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:46 smithi085 conmon[29319]: 2021-02-10T04:42:46.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:46 smithi085 conmon[29319]: 2021 2021-02-10T04:42:46.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:46 smithi085 conmon[29319]: - 2021-02-10T04:42:46.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:46 smithi085 conmon[29319]: 02 2021-02-10T04:42:46.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:46 smithi085 conmon[29319]: - 2021-02-10T04:42:46.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:46 smithi085 conmon[29319]: 10 2021-02-10T04:42:46.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:46 smithi085 conmon[29319]: T 2021-02-10T04:42:46.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:46 smithi085 conmon[29319]: 04 2021-02-10T04:42:46.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:46 smithi085 conmon[29319]: : 2021-02-10T04:42:46.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:46 smithi085 conmon[29319]: 42 2021-02-10T04:42:46.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:46 smithi085 conmon[29319]: : 2021-02-10T04:42:46.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:46 smithi085 conmon[29319]: 45 2021-02-10T04:42:46.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:46 smithi085 conmon[29319]: . 2021-02-10T04:42:46.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:46 smithi085 conmon[29319]: 345571+0000 mon.a (mon.0) 2021-02-10T04:42:46.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:46 smithi085 conmon[29319]: 1928 : cluster [DBG] osdmap e945: 8 total, 8 up, 7 in 2021-02-10T04:42:46.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:46 smithi085 conmon[35961]: cluster 2021-02-10T04:42:45.345571+0000 2021-02-10T04:42:46.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:46 smithi085 conmon[35961]: mon.a (mon.0) 1928 : cluster [DBG] osdmap e945: 8 total, 8 up, 7 in 2021-02-10T04:42:47.374 INFO:tasks.rados.rados.0.smithi085.stdout:3815: done (4 left) 2021-02-10T04:42:47.375 INFO:tasks.rados.rados.0.smithi085.stdout:3816: copy_from oid 43 from oid 37 current snap is 441 2021-02-10T04:42:47.375 INFO:tasks.rados.rados.0.smithi085.stdout:3817: write oid 14 current snap is 441 2021-02-10T04:42:47.375 INFO:tasks.rados.rados.0.smithi085.stdout:3817: seq_num 1304 ranges {716328=421059,1840284=509126,3143112=659293} 2021-02-10T04:42:47.381 INFO:tasks.rados.rados.0.smithi085.stdout:3817: writing smithi08575397-14 from 716328 to 1137387 tid 1 2021-02-10T04:42:47.389 INFO:tasks.rados.rados.0.smithi085.stdout:3817: writing smithi08575397-14 from 1840284 to 2349410 tid 2 2021-02-10T04:42:47.399 INFO:tasks.rados.rados.0.smithi085.stdout:3817: writing smithi08575397-14 from 3143112 to 3802405 tid 3 2021-02-10T04:42:47.399 INFO:tasks.rados.rados.0.smithi085.stdout:3818: snap_create 2021-02-10T04:42:47.399 INFO:tasks.rados.rados.0.smithi085.stdout:3817: finishing write tid 1 to smithi08575397-14 2021-02-10T04:42:47.399 INFO:tasks.rados.rados.0.smithi085.stdout:3817: finishing write tid 2 to smithi08575397-14 2021-02-10T04:42:47.403 INFO:tasks.rados.rados.0.smithi085.stdout:3817: finishing write tid 3 to smithi08575397-14 2021-02-10T04:42:47.404 INFO:tasks.rados.rados.0.smithi085.stdout:3817: finishing write tid 4 to smithi08575397-14 2021-02-10T04:42:47.404 INFO:tasks.rados.rados.0.smithi085.stdout:3817: finishing write tid 5 to smithi08575397-14 2021-02-10T04:42:47.404 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 14 v 1893 (ObjNum 1304 snap 441 seq_num 1304) dirty exists 2021-02-10T04:42:47.404 INFO:tasks.rados.rados.0.smithi085.stdout:3817: left oid 14 (ObjNum 1304 snap 441 seq_num 1304) 2021-02-10T04:42:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:47 smithi164 conmon[29288]: cluster 2021-02-10T04:42:46.241057+0000 mgr.y (mgr.14140) 873 2021-02-10T04:42:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:47 smithi164 conmon[29288]: : cluster [DBG] pgmap v1569: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 854 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 18 MiB/s rd, 5.4 MiB/s wr, 32 op/s 2021-02-10T04:42:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:47 smithi164 conmon[29288]: cluster 2021-02-10T04:42:46.363077+0000 mon.a (mon.0) 1929 : cluster [DBG] osdmap e946: 8 total, 8 up, 7 in 2021-02-10T04:42:47.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[35961]: cluster 2021-02-10T04:42:47.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[35961]: 2021-02-10T04:42:47.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[35961]: 2021 2021-02-10T04:42:47.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[35961]: - 2021-02-10T04:42:47.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[35961]: 02 2021-02-10T04:42:47.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[35961]: - 2021-02-10T04:42:47.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[35961]: 10 2021-02-10T04:42:47.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[35961]: T 2021-02-10T04:42:47.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[35961]: 04 2021-02-10T04:42:47.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[35961]: : 2021-02-10T04:42:47.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[35961]: 42 2021-02-10T04:42:47.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[35961]: : 2021-02-10T04:42:47.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[35961]: 46.241057+0000 2021-02-10T04:42:47.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[35961]: mgr.y (mgr.14140) 873 : cluster [DBG] pgmap v1569: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 854 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 18 MiB/s rd, 5.4 MiB/s wr, 32 op/s 2021-02-10T04:42:47.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:42:47.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[35961]: -10T04:42:46.363077+0000 mon.a (mon. 2021-02-10T04:42:47.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[35961]: 0) 1929 : cluster [DBG] osdmap e946: 8 total, 8 up, 7 in 2021-02-10T04:42:47.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[29319]: cluster 2021-02-10T04:42:47.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[29319]: 2021-02-10T04:42:47.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[29319]: 2021 2021-02-10T04:42:47.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[29319]: - 2021-02-10T04:42:47.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[29319]: 02 2021-02-10T04:42:47.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[29319]: - 2021-02-10T04:42:47.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[29319]: 10 2021-02-10T04:42:47.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[29319]: T 2021-02-10T04:42:47.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[29319]: 04 2021-02-10T04:42:47.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[29319]: : 2021-02-10T04:42:47.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[29319]: 42:46.241057+0000 2021-02-10T04:42:47.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[29319]: mgr.y (mgr.14140) 873 : cluster [DBG] pgmap v1569: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 854 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 18 MiB/s rd, 5.4 MiB/s wr, 32 op/s 2021-02-10T04:42:47.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[29319]: cluster 2021-02-10T04:42:47.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[29319]: 2021-02-10T04:42:47.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[29319]: 2021 2021-02-10T04:42:47.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[29319]: - 2021-02-10T04:42:47.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[29319]: 02-10T04: 2021-02-10T04:42:47.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:47 smithi085 conmon[29319]: 42:46.363077+0000 mon.a (mon.0) 1929 : cluster [DBG] osdmap e946: 8 total, 8 up, 7 in 2021-02-10T04:42:47.704 INFO:tasks.rados.rados.0.smithi085.stdout:3816: finishing copy_from to smithi08575397-43 2021-02-10T04:42:47.704 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 1851 (ObjNum 1129 snap 381 seq_num 1129) dirty exists 2021-02-10T04:42:47.704 INFO:tasks.rados.rados.0.smithi085.stdout:3816: finishing copy_from racing read to smithi08575397-43 2021-02-10T04:42:48.409 INFO:tasks.rados.rados.0.smithi085.stdout:3816: done (6 left) 2021-02-10T04:42:48.409 INFO:tasks.rados.rados.0.smithi085.stdout:3817: done (5 left) 2021-02-10T04:42:48.409 INFO:tasks.rados.rados.0.smithi085.stdout:3818: done (4 left) 2021-02-10T04:42:48.409 INFO:tasks.rados.rados.0.smithi085.stdout:3819: read oid 29 snap -1 2021-02-10T04:42:48.409 INFO:tasks.rados.rados.0.smithi085.stdout:3819: expect (ObjNum 8 snap 0 seq_num 8) 2021-02-10T04:42:48.409 INFO:tasks.rados.rados.0.smithi085.stdout:3820: snap_create 2021-02-10T04:42:48.424 INFO:tasks.rados.rados.0.smithi085.stdout:3819: expect (ObjNum 8 snap 0 seq_num 8) 2021-02-10T04:42:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:48 smithi085 conmon[29319]: cluster 2021-02-10T04:42:47.373595+0000 mon.a (mon.0) 2021-02-10T04:42:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:48 smithi085 conmon[29319]: 1930 : cluster [DBG] osdmap e947: 8 total, 8 up, 7 in 2021-02-10T04:42:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:48 smithi085 conmon[29319]: cluster 2021-02-10T04:42:47.935880+0000 osd.4 (osd.4) 1619 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:42:48.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:48 smithi085 conmon[29319]: cluster 2021-02-10T04:42:47.936351+0000 osd.4 (osd.4) 1620 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:42:48.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:48 smithi085 conmon[35961]: cluster 2021-02-10T04:42:47.373595 2021-02-10T04:42:48.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:48 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1930 : cluster [DBG] osdmap e947: 8 total, 8 up, 7 in 2021-02-10T04:42:48.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:48 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:42:48.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:48 smithi085 conmon[35961]: -02-10T04:42:47. 2021-02-10T04:42:48.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:48 smithi085 conmon[35961]: 935880+0000 osd.4 (osd. 2021-02-10T04:42:48.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:48 smithi085 conmon[35961]: 4) 1619 : cluster [DBG] 2021-02-10T04:42:48.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:48 smithi085 conmon[35961]: purged_snaps scrub starts 2021-02-10T04:42:48.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:48 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:42:48.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:48 smithi085 conmon[35961]: -02-10T04:42: 2021-02-10T04:42:48.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:48 smithi085 conmon[35961]: 47.936351+0000 osd.4 ( 2021-02-10T04:42:48.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:48 smithi085 conmon[35961]: osd.4) 1620 : cluster 2021-02-10T04:42:48.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:48 smithi085 conmon[35961]: [DBG] purged_snaps scrub ok 2021-02-10T04:42:48.894 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:42:48 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:42:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:48 smithi164 conmon[29288]: cluster 2021-02-10T04:42:47.373595+0000 mon.a (mon 2021-02-10T04:42:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:48 smithi164 conmon[29288]: .0) 1930 : cluster [DBG] osdmap e947: 8 total, 8 up, 7 in 2021-02-10T04:42:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:48 smithi164 conmon[29288]: cluster 2021-02-10T04:42:47.935880+0000 osd.4 (osd.4) 1619 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:42:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:48 smithi164 conmon[29288]: cluster 2021-02-10T04:42:47.936351+0000 osd.4 ( 2021-02-10T04:42:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:48 smithi164 conmon[29288]: osd.4) 1620 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:42:48.919 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:42:49.081 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:49.410 INFO:tasks.rados.rados.0.smithi085.stdout:3819: done (5 left) 2021-02-10T04:42:49.410 INFO:tasks.rados.rados.0.smithi085.stdout:3820: done (4 left) 2021-02-10T04:42:49.410 INFO:tasks.rados.rados.0.smithi085.stdout:3821: copy_from oid 12 from oid 10 current snap is 443 2021-02-10T04:42:49.410 INFO:tasks.rados.rados.0.smithi085.stdout:3822: snap_create 2021-02-10T04:42:49.413 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:49 smithi085 conmon[29319]: cluster 2021-02-10T04:42:49.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:49 smithi085 conmon[29319]: 2021-02-10T04:42:49.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:49 smithi085 conmon[29319]: 2021 2021-02-10T04:42:49.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:49 smithi085 conmon[29319]: - 2021-02-10T04:42:49.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:49 smithi085 conmon[29319]: 02 2021-02-10T04:42:49.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:49 smithi085 conmon[29319]: - 2021-02-10T04:42:49.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:49 smithi085 conmon[29319]: 10 2021-02-10T04:42:49.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:49 smithi085 conmon[29319]: T04:42:48. 2021-02-10T04:42:49.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:49 smithi085 conmon[29319]: 241750+0000 mgr.y (mgr.14140) 874 : cluster [DBG] pgmap v1572: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 858 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 14 MiB/s rd, 4.1 MiB/s wr, 23 op/s 2021-02-10T04:42:49.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:49 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:42:49.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:49 smithi085 conmon[29319]: 02-10T04:42:48.407818+0000 mon.a (mon. 2021-02-10T04:42:49.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:49 smithi085 conmon[29319]: 0) 1931 : cluster [DBG] osdmap e948: 8 total, 8 up, 7 in 2021-02-10T04:42:49.414 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:49 smithi085 conmon[29319]: audit 2021 2021-02-10T04:42:49.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:49 smithi085 conmon[29319]: -02-10T04:42:48.539042+0000 2021-02-10T04:42:49.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:49 smithi085 conmon[29319]: mgr.y (mgr.14140) 875 : audit [DBG] 2021-02-10T04:42:49.415 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:49 smithi085 conmon[29319]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:42:49.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:49 smithi085 conmon[35961]: cluster 2021-02-10T04:42:49.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:49 smithi085 conmon[35961]: 2021-02-10T04:42:49.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:49 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:42:49.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:49 smithi085 conmon[35961]: 42:48.241750+0000 mgr.y (mgr.14140) 874 : cluster [DBG] pgmap v1572: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 858 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 14 MiB/s rd, 4.1 MiB/s wr, 23 op/s 2021-02-10T04:42:49.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:49 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:42:49.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:49 smithi085 conmon[35961]: 02-10T04:42:48.407818+0000 mon.a 2021-02-10T04:42:49.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:49 smithi085 conmon[35961]: (mon.0) 1931 : cluster [DBG] osdmap e948: 8 total, 8 up, 7 in 2021-02-10T04:42:49.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:49 smithi085 conmon[35961]: audit 2021-02-10T04:42:49.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:49 smithi085 conmon[35961]: 2021-02-10T04:42:48.539042+0000 2021-02-10T04:42:49.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:49 smithi085 conmon[35961]: mgr.y (mgr.14140) 875 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:42:49.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:49 smithi085 conmon[35961]: 2021-02-10T04:42:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:49 smithi164 conmon[29288]: cluster 2021-02-10T04:42:48.241750+0000 mgr.y (mgr 2021-02-10T04:42:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:49 smithi164 conmon[29288]: .14140) 874 : cluster [DBG] pgmap v1572: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 858 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 14 MiB/s rd, 4.1 MiB/s wr, 23 op/s 2021-02-10T04:42:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:49 smithi164 conmon[29288]: cluster 2021-02-10T04:42:48.407818+0000 mon.a ( 2021-02-10T04:42:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:49 smithi164 conmon[29288]: mon.0) 1931 : cluster [DBG] osdmap e948: 8 total, 8 up, 7 in 2021-02-10T04:42:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:49 smithi164 conmon[29288]: audit 2021-02-10T04:42:48.539042+0000 2021-02-10T04:42:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:49 smithi164 conmon[29288]: mgr.y (mgr.14140) 875 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:42:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:49 smithi164 conmon[29288]: 2021-02-10T04:42:50.063 INFO:tasks.rados.rados.0.smithi085.stdout:3821: finishing copy_from to smithi08575397-12 2021-02-10T04:42:50.064 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 1619 (ObjNum 1167 snap 401 seq_num 1167) dirty exists 2021-02-10T04:42:50.064 INFO:tasks.rados.rados.0.smithi085.stdout:3821: finishing copy_from racing read to smithi08575397-12 2021-02-10T04:42:50.418 INFO:tasks.rados.rados.0.smithi085.stdout:3821: done (5 left) 2021-02-10T04:42:50.418 INFO:tasks.rados.rados.0.smithi085.stdout:3822: done (4 left) 2021-02-10T04:42:50.419 INFO:tasks.rados.rados.0.smithi085.stdout:3823: copy_from oid 43 from oid 50 current snap is 444 2021-02-10T04:42:50.419 INFO:tasks.rados.rados.0.smithi085.stdout:3824: rollback oid 22 current snap is 444 2021-02-10T04:42:50.419 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 22 to 401 2021-02-10T04:42:50.419 INFO:tasks.rados.rados.0.smithi085.stdout:3825: snap_create 2021-02-10T04:42:50.426 INFO:tasks.rados.rados.0.smithi085.stdout:3824: finishing rollback tid 0 to smithi08575397-22 2021-02-10T04:42:50.426 INFO:tasks.rados.rados.0.smithi085.stdout:3824: finishing rollback tid 1 to smithi08575397-22 2021-02-10T04:42:50.426 INFO:tasks.rados.rados.0.smithi085.stdout:3824: finishing rollback tid 2 to smithi08575397-22 2021-02-10T04:42:50.426 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 22 v 1885 (ObjNum 1108 snap 374 seq_num 1108) dirty exists 2021-02-10T04:42:50.663 INFO:tasks.rados.rados.0.smithi085.stdout:3823: finishing copy_from to smithi08575397-43 2021-02-10T04:42:50.663 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 1616 (ObjNum 1281 snap 433 seq_num 1281) dirty exists 2021-02-10T04:42:50.663 INFO:tasks.rados.rados.0.smithi085.stdout:3823: finishing copy_from racing read to smithi08575397-43 2021-02-10T04:42:50.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:50 smithi085 conmon[35961]: cluster 2021-02-10T04:42:49.409652+0000 2021-02-10T04:42:50.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:50 smithi085 conmon[35961]: mon.a (mon.0) 1932 : cluster [DBG] osdmap e949: 8 total, 8 up, 7 in 2021-02-10T04:42:50.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:50 smithi085 conmon[29319]: cluster 2021-02-10T04:42:49.409652+0000 2021-02-10T04:42:50.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:50 smithi085 conmon[29319]: mon.a (mon.0) 1932 : cluster [DBG] osdmap e949: 8 total, 8 up, 7 in 2021-02-10T04:42:50.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:50 smithi164 conmon[29288]: cluster 2021-02-10T04:42:49.409652+0000 mon.a 2021-02-10T04:42:50.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:50 smithi164 conmon[29288]: (mon.0) 1932 : cluster [DBG] osdmap e949: 8 total, 8 up, 7 in 2021-02-10T04:42:51.412 INFO:tasks.rados.rados.0.smithi085.stdout:3823: done (6 left) 2021-02-10T04:42:51.413 INFO:tasks.rados.rados.0.smithi085.stdout:3824: done (5 left) 2021-02-10T04:42:51.413 INFO:tasks.rados.rados.0.smithi085.stdout:3825: done (4 left) 2021-02-10T04:42:51.413 INFO:tasks.rados.rados.0.smithi085.stdout:3826: rollback oid 32 current snap is 445 2021-02-10T04:42:51.413 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 32 to 266 2021-02-10T04:42:51.413 INFO:tasks.rados.rados.0.smithi085.stdout:3827: write (excl) oid 20 current snap is 445 2021-02-10T04:42:51.413 INFO:tasks.rados.rados.0.smithi085.stdout:3827: seq_num 1305 ranges {733135=496308,1898766=652784,3170535=470565} 2021-02-10T04:42:51.419 INFO:tasks.rados.rados.0.smithi085.stdout:3827: writing smithi08575397-20 from 733135 to 1229443 tid 1 2021-02-10T04:42:51.427 INFO:tasks.rados.rados.0.smithi085.stdout:3827: writing smithi08575397-20 from 1898766 to 2551550 tid 2 2021-02-10T04:42:51.434 INFO:tasks.rados.rados.0.smithi085.stdout:3827: writing smithi08575397-20 from 3170535 to 3641100 tid 3 2021-02-10T04:42:51.434 INFO:tasks.rados.rados.0.smithi085.stdout:3828: snap_create 2021-02-10T04:42:51.435 INFO:tasks.rados.rados.0.smithi085.stdout:3826: finishing rollback tid 1 to smithi08575397-32 2021-02-10T04:42:51.435 INFO:tasks.rados.rados.0.smithi085.stdout:3826: finishing rollback tid 2 to smithi08575397-32 2021-02-10T04:42:51.435 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 32 v 1500 (ObjNum 85 snap 11 seq_num 85) dirty exists 2021-02-10T04:42:51.435 INFO:tasks.rados.rados.0.smithi085.stdout:3827: finishing write tid 1 to smithi08575397-20 2021-02-10T04:42:51.438 INFO:tasks.rados.rados.0.smithi085.stdout:3827: finishing write tid 2 to smithi08575397-20 2021-02-10T04:42:51.439 INFO:tasks.rados.rados.0.smithi085.stdout:3827: finishing write tid 3 to smithi08575397-20 2021-02-10T04:42:51.439 INFO:tasks.rados.rados.0.smithi085.stdout:3827: finishing write tid 4 to smithi08575397-20 2021-02-10T04:42:51.439 INFO:tasks.rados.rados.0.smithi085.stdout:3827: finishing write tid 5 to smithi08575397-20 2021-02-10T04:42:51.439 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 20 v 1887 (ObjNum 1305 snap 445 seq_num 1305) dirty exists 2021-02-10T04:42:51.439 INFO:tasks.rados.rados.0.smithi085.stdout:3827: left oid 20 (ObjNum 1305 snap 445 seq_num 1305) 2021-02-10T04:42:51.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:51 smithi085 conmon[35961]: cluster 2021-02-10T04:42:50.242630 2021-02-10T04:42:51.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:51 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 876 : cluster [DBG] pgmap v1575: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim_wait, 155 active+clean, 8 active+clean+snaptrim; 862 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 4.7 MiB/s rd, 3.2 MiB/s wr, 10 op/s 2021-02-10T04:42:51.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:51 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:42:51.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:51 smithi085 conmon[35961]: 10T04:42:50.415751+0000 mon.a (mon.0) 1933 2021-02-10T04:42:51.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:51 smithi085 conmon[35961]: : cluster [DBG] osdmap e950: 8 total, 8 up, 7 in 2021-02-10T04:42:51.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:51 smithi085 conmon[35961]: cluster 2021-02-10T04:42: 2021-02-10T04:42:51.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:51 smithi085 conmon[35961]: 51.411994+0000 mon.a (mon.0) 1934 : cluster [DBG] osdmap e951: 8 total, 8 up, 7 in 2021-02-10T04:42:51.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:51 smithi085 conmon[35961]: 2021-02-10T04:42:51.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:51 smithi085 conmon[29319]: cluster 2021-02-10T04:42:50. 2021-02-10T04:42:51.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:51 smithi085 conmon[29319]: 242630+0000 mgr.y (mgr.14140) 876 : cluster [DBG] pgmap v1575: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim_wait, 155 active+clean, 8 active+clean+snaptrim; 862 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 4.7 MiB/s rd, 3.2 MiB/s wr, 10 op/s 2021-02-10T04:42:51.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:51 smithi085 conmon[29319]: cluster 2021-02-10T04:42:50.415751+0000 2021-02-10T04:42:51.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:51 smithi085 conmon[29319]: mon.a (mon.0) 1933 : cluster [DBG] osdmap e950: 8 total, 8 up, 7 in 2021-02-10T04:42:51.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:51 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:42:51.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:51 smithi085 conmon[29319]: T04:42:51.411994+0000 mon.a (mon.0) 1934 : cluster [DBG] 2021-02-10T04:42:51.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:51 smithi085 conmon[29319]: osdmap e951: 8 total, 8 up, 7 in 2021-02-10T04:42:51.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:51 smithi164 conmon[29288]: cluster 2021-02-10T04:42:51.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:51 smithi164 conmon[29288]: 2021-02-10T04:42:51.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:51 smithi164 conmon[29288]: 2021-02-10T04:42: 2021-02-10T04:42:51.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:51 smithi164 conmon[29288]: 50.242630+0000 mgr.y (mgr.14140) 876 : cluster [DBG] pgmap v1575: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim_wait, 155 active+clean, 8 active+clean+snaptrim; 862 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 4.7 MiB/s rd, 3.2 MiB/s wr, 10 op/s 2021-02-10T04:42:51.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:51 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:42:51.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:51 smithi164 conmon[29288]: :42:50.415751+0000 mon.a (mon.0) 1933 2021-02-10T04:42:51.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:51 smithi164 conmon[29288]: : cluster [DBG] osdmap e950: 8 total, 8 up, 7 in 2021-02-10T04:42:51.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:51 smithi164 conmon[29288]: cluster 2021-02-10T04:42 2021-02-10T04:42:51.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:51 smithi164 conmon[29288]: :51.411994+0000 mon.a (mon.0) 1934 : cluster 2021-02-10T04:42:51.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:51 smithi164 conmon[29288]: [DBG] osdmap e951: 8 total, 8 up, 7 in 2021-02-10T04:42:52.024 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.3 dump_historic_ops 2021-02-10T04:42:52.041 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2021-02-10T04:42:52.041 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:42:52.098 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:42:52.250 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:42:52.291 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:42:51 smithi085 conmon[41540]: debug 2021-02-10T04:42:51.978+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:52.293 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:52.334 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:42:52.400 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:42:52.459 INFO:tasks.rados.rados.0.smithi085.stdout:3826: done (6 left) 2021-02-10T04:42:52.459 INFO:tasks.rados.rados.0.smithi085.stdout:3827: done (5 left) 2021-02-10T04:42:52.459 INFO:tasks.rados.rados.0.smithi085.stdout:3828: done (4 left) 2021-02-10T04:42:52.460 INFO:tasks.rados.rados.0.smithi085.stdout:3829: read oid 16 snap -1 2021-02-10T04:42:52.460 INFO:tasks.rados.rados.0.smithi085.stdout:3829: expect (ObjNum 1191 snap 407 seq_num 1191) 2021-02-10T04:42:52.460 INFO:tasks.rados.rados.0.smithi085.stdout:3830: write (excl) oid 8 current snap is 446 2021-02-10T04:42:52.460 INFO:tasks.rados.rados.0.smithi085.stdout:3830: seq_num 1306 ranges {749942=571557,1840895=796442,3365252=584690} 2021-02-10T04:42:52.467 INFO:tasks.rados.rados.0.smithi085.stdout:3830: writing smithi08575397-8 from 749942 to 1321499 tid 1 2021-02-10T04:42:52.481 INFO:tasks.rados.rados.0.smithi085.stdout:3830: writing smithi08575397-8 from 1840895 to 2637337 tid 2 2021-02-10T04:42:52.490 INFO:tasks.rados.rados.0.smithi085.stdout:3830: writing smithi08575397-8 from 3365252 to 3949942 tid 3 2021-02-10T04:42:52.490 INFO:tasks.rados.rados.0.smithi085.stdout:3829: expect (ObjNum 1191 snap 407 seq_num 1191) 2021-02-10T04:42:52.512 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:42:52.543 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:42:52 smithi085 conmon[52836]: debug 2021-02-10T04:42:52.543 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:42:52 smithi085 conmon[52836]: 2021-02-10T04:42:52.468+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:52.580 INFO:tasks.rados.rados.0.smithi085.stdout:3830: finishing write tid 1 to smithi08575397-8 2021-02-10T04:42:52.580 INFO:tasks.rados.rados.0.smithi085.stdout:3830: finishing write tid 2 to smithi08575397-8 2021-02-10T04:42:52.580 INFO:tasks.rados.rados.0.smithi085.stdout:3830: finishing write tid 3 to smithi08575397-8 2021-02-10T04:42:52.580 INFO:tasks.rados.rados.0.smithi085.stdout:3830: finishing write tid 4 to smithi08575397-8 2021-02-10T04:42:52.580 INFO:tasks.rados.rados.0.smithi085.stdout:3830: finishing write tid 5 to smithi08575397-8 2021-02-10T04:42:52.580 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 8 v 1625 (ObjNum 1306 snap 446 seq_num 1306) dirty exists 2021-02-10T04:42:52.580 INFO:tasks.rados.rados.0.smithi085.stdout:3830: left oid 8 (ObjNum 1306 snap 446 seq_num 1306) 2021-02-10T04:42:52.580 INFO:tasks.rados.rados.0.smithi085.stdout:3829: done (5 left) 2021-02-10T04:42:52.580 INFO:tasks.rados.rados.0.smithi085.stdout:3830: done (4 left) 2021-02-10T04:42:52.580 INFO:tasks.rados.rados.0.smithi085.stdout:3831: rollback oid 29 current snap is 446 2021-02-10T04:42:52.581 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 29 to 442 2021-02-10T04:42:52.581 INFO:tasks.rados.rados.0.smithi085.stdout:3832: delete oid 16 current snap is 446 2021-02-10T04:42:52.583 INFO:tasks.rados.rados.0.smithi085.stdout:3832: done (5 left) 2021-02-10T04:42:52.583 INFO:tasks.rados.rados.0.smithi085.stdout:3833: snap_create 2021-02-10T04:42:52.584 INFO:tasks.rados.rados.0.smithi085.stdout:3831: finishing rollback tid 0 to smithi08575397-29 2021-02-10T04:42:52.585 INFO:tasks.rados.rados.0.smithi085.stdout:3831: finishing rollback tid 1 to smithi08575397-29 2021-02-10T04:42:52.585 INFO:tasks.rados.rados.0.smithi085.stdout:3831: finishing rollback tid 2 to smithi08575397-29 2021-02-10T04:42:52.585 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 29 v 1503 (ObjNum 8 snap 0 seq_num 8) dirty exists 2021-02-10T04:42:52.645 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:42:53.468 INFO:tasks.rados.rados.0.smithi085.stdout:3831: done (5 left) 2021-02-10T04:42:53.468 INFO:tasks.rados.rados.0.smithi085.stdout:3833: done (4 left) 2021-02-10T04:42:53.468 INFO:tasks.rados.rados.0.smithi085.stdout:3834: read oid 48 snap -1 2021-02-10T04:42:53.468 INFO:tasks.rados.rados.0.smithi085.stdout:3834: expect (ObjNum 1290 snap 434 seq_num 1290) 2021-02-10T04:42:53.468 INFO:tasks.rados.rados.0.smithi085.stdout:3835: copy_from oid 1 from oid 46 current snap is 447 2021-02-10T04:42:53.468 INFO:tasks.rados.rados.0.smithi085.stdout:3836: snap_create 2021-02-10T04:42:53.483 INFO:tasks.rados.rados.0.smithi085.stdout:3834: expect (ObjNum 1290 snap 434 seq_num 1290) 2021-02-10T04:42:53.648 INFO:tasks.rados.rados.0.smithi085.stdout:3835: finishing copy_from to smithi08575397-1 2021-02-10T04:42:53.649 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 1881 (ObjNum 1292 snap 436 seq_num 1292) dirty exists 2021-02-10T04:42:53.649 INFO:tasks.rados.rados.0.smithi085.stdout:3835: finishing copy_from racing read to smithi08575397-1 2021-02-10T04:42:53.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:53 smithi085 conmon[35961]: cluster 2021-02-10T04:42:53.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:53 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:42:53.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:53 smithi085 conmon[35961]: 04:42:52.243243+0000 mgr.y (mgr.14140) 877 : cluster [DBG] pgmap v1578: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim_wait, 155 active+clean, 8 active+clean+snaptrim; 862 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-02-10T04:42:53.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:53 smithi085 conmon[35961]: cluster 2021-02-10T04:42:53.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:53 smithi085 conmon[35961]: 2021-02-10T04:42:52. 2021-02-10T04:42:53.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:53 smithi085 conmon[35961]: 453552+0000 mon.a (mon.0) 1935 : cluster [DBG] osdmap e952: 8 total, 8 up, 7 in 2021-02-10T04:42:53.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:53 smithi085 conmon[29319]: cluster 2021-02-10T04:42:52.243243+0000 mgr.y (mgr.14140) 877 : cluster [DBG] 2021-02-10T04:42:53.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:53 smithi085 conmon[29319]: pgmap v1578: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim_wait, 155 active+clean, 8 active+clean+snaptrim; 862 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-02-10T04:42:53.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:53 smithi085 conmon[29319]: cluster 2021-02-10T04:42:52.453552+0000 mon.a (mon.0) 1935 : cluster [DBG] osdmap e952: 8 total, 8 up, 7 in 2021-02-10T04:42:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:53 smithi164 conmon[29288]: cluster 2021-02-10T04:42:52.243243+0000 mgr.y (mgr 2021-02-10T04:42:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:53 smithi164 conmon[29288]: .14140) 877 : cluster [DBG] pgmap v1578: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+snaptrim_wait, 155 active+clean, 8 active+clean+snaptrim; 862 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 2.1 MiB/s wr, 4 op/s 2021-02-10T04:42:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:53 smithi164 conmon[29288]: cluster 2021-02-10T04:42:52.453552+0000 mon.a (mon.0) 1935 : cluster [DBG] osdmap e952: 8 total, 8 up, 7 in 2021-02-10T04:42:53.995 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:54.289 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:42:54 smithi164 conmon[33677]: debug 2021-02-10T04:42:54.287+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:54.472 INFO:tasks.rados.rados.0.smithi085.stdout:3834: done (6 left) 2021-02-10T04:42:54.473 INFO:tasks.rados.rados.0.smithi085.stdout:3835: done (5 left) 2021-02-10T04:42:54.473 INFO:tasks.rados.rados.0.smithi085.stdout:3836: done (4 left) 2021-02-10T04:42:54.473 INFO:tasks.rados.rados.0.smithi085.stdout:3837: rollback oid 31 current snap is 448 2021-02-10T04:42:54.473 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 31 to 435 2021-02-10T04:42:54.473 INFO:tasks.rados.rados.0.smithi085.stdout:3838: read oid 14 snap -1 2021-02-10T04:42:54.473 INFO:tasks.rados.rados.0.smithi085.stdout:3838: expect (ObjNum 1304 snap 441 seq_num 1304) 2021-02-10T04:42:54.474 INFO:tasks.rados.rados.0.smithi085.stdout:3839: read oid 12 snap -1 2021-02-10T04:42:54.474 INFO:tasks.rados.rados.0.smithi085.stdout:3839: expect (ObjNum 1167 snap 401 seq_num 1167) 2021-02-10T04:42:54.474 INFO:tasks.rados.rados.0.smithi085.stdout:3840: write (excl) oid 1 current snap is 448 2021-02-10T04:42:54.474 INFO:tasks.rados.rados.0.smithi085.stdout:3840: seq_num 1308 ranges {783556=438408,1674212=400111,2736451=420734,3882270=101286} 2021-02-10T04:42:54.476 INFO:tasks.rados.rados.0.smithi085.stdout:3840: writing smithi08575397-1 from 783556 to 1221964 tid 1 2021-02-10T04:42:54.481 INFO:tasks.rados.rados.0.smithi085.stdout:3840: writing smithi08575397-1 from 1674212 to 2074323 tid 2 2021-02-10T04:42:54.487 INFO:tasks.rados.rados.0.smithi085.stdout:3840: writing smithi08575397-1 from 2736451 to 3157185 tid 3 2021-02-10T04:42:54.492 INFO:tasks.rados.rados.0.smithi085.stdout:3840: writing smithi08575397-1 from 3882270 to 3983556 tid 4 2021-02-10T04:42:54.492 INFO:tasks.rados.rados.0.smithi085.stdout:3841: read oid 43 snap -1 2021-02-10T04:42:54.492 INFO:tasks.rados.rados.0.smithi085.stdout:3841: expect (ObjNum 1281 snap 433 seq_num 1281) 2021-02-10T04:42:54.492 INFO:tasks.rados.rados.0.smithi085.stdout:3842: read oid 29 snap 418 2021-02-10T04:42:54.493 INFO:tasks.rados.rados.0.smithi085.stdout:3842: expect (ObjNum 8 snap 0 seq_num 8) 2021-02-10T04:42:54.493 INFO:tasks.rados.rados.0.smithi085.stdout:3843: snap_create 2021-02-10T04:42:54.493 INFO:tasks.rados.rados.0.smithi085.stdout:3838: expect (ObjNum 1304 snap 441 seq_num 1304) 2021-02-10T04:42:54.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:54 smithi164 conmon[29288]: cluster 2021-02-10T04:42:53.468042+0000 mon.a (mon.0) 1936 : cluster [DBG] 2021-02-10T04:42:54.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:54 smithi164 conmon[29288]: osdmap e953: 8 total, 8 up, 7 in 2021-02-10T04:42:54.712 INFO:tasks.rados.rados.0.smithi085.stdout:3839: expect (ObjNum 1167 snap 401 seq_num 1167) 2021-02-10T04:42:54.786 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:42:54.909 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:42:54.934 INFO:tasks.rados.rados.0.smithi085.stdout:3841: expect (ObjNum 1281 snap 433 seq_num 1281) 2021-02-10T04:42:54.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:54 smithi085 conmon[29319]: cluster 2021-02-10T04:42:53.468042+0000 mon.a 2021-02-10T04:42:54.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:54 smithi085 conmon[29319]: (mon.0) 1936 : cluster [DBG] osdmap e953: 8 total, 8 up, 7 in 2021-02-10T04:42:54.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:54 smithi085 conmon[35961]: cluster 2021-02-10T04:42:53.468042 2021-02-10T04:42:54.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:54 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1936 : cluster [DBG] osdmap e953: 8 total, 8 up, 7 in 2021-02-10T04:42:54.951 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:55.213 INFO:tasks.rados.rados.0.smithi085.stdout:3842: expect (ObjNum 8 snap 0 seq_num 8) 2021-02-10T04:42:55.313 INFO:tasks.rados.rados.0.smithi085.stdout:3837: finishing rollback tid 0 to smithi08575397-31 2021-02-10T04:42:55.313 INFO:tasks.rados.rados.0.smithi085.stdout:3837: finishing rollback tid 1 to smithi08575397-31 2021-02-10T04:42:55.313 INFO:tasks.rados.rados.0.smithi085.stdout:3837: finishing rollback tid 2 to smithi08575397-31 2021-02-10T04:42:55.313 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 1888 (ObjNum 1288 snap 433 seq_num 1288) dirty exists 2021-02-10T04:42:55.314 INFO:tasks.rados.rados.0.smithi085.stdout:3840: finishing write tid 1 to smithi08575397-1 2021-02-10T04:42:55.314 INFO:tasks.rados.rados.0.smithi085.stdout:3840: finishing write tid 2 to smithi08575397-1 2021-02-10T04:42:55.314 INFO:tasks.rados.rados.0.smithi085.stdout:3840: finishing write tid 3 to smithi08575397-1 2021-02-10T04:42:55.314 INFO:tasks.rados.rados.0.smithi085.stdout:3840: finishing write tid 4 to smithi08575397-1 2021-02-10T04:42:55.314 INFO:tasks.rados.rados.0.smithi085.stdout:3840: finishing write tid 5 to smithi08575397-1 2021-02-10T04:42:55.314 INFO:tasks.rados.rados.0.smithi085.stdout:3840: finishing write tid 6 to smithi08575397-1 2021-02-10T04:42:55.314 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 1 v 1886 (ObjNum 1308 snap 448 seq_num 1308) dirty exists 2021-02-10T04:42:55.314 INFO:tasks.rados.rados.0.smithi085.stdout:3840: left oid 1 (ObjNum 1308 snap 448 seq_num 1308) 2021-02-10T04:42:55.484 INFO:tasks.rados.rados.0.smithi085.stdout:3837: done (10 left) 2021-02-10T04:42:55.484 INFO:tasks.rados.rados.0.smithi085.stdout:3838: done (9 left) 2021-02-10T04:42:55.485 INFO:tasks.rados.rados.0.smithi085.stdout:3839: done (8 left) 2021-02-10T04:42:55.485 INFO:tasks.rados.rados.0.smithi085.stdout:3840: done (7 left) 2021-02-10T04:42:55.485 INFO:tasks.rados.rados.0.smithi085.stdout:3841: done (6 left) 2021-02-10T04:42:55.485 INFO:tasks.rados.rados.0.smithi085.stdout:3842: done (5 left) 2021-02-10T04:42:55.485 INFO:tasks.rados.rados.0.smithi085.stdout:3843: done (4 left) 2021-02-10T04:42:55.485 INFO:tasks.rados.rados.0.smithi085.stdout:3844: write oid 10 current snap is 449 2021-02-10T04:42:55.485 INFO:tasks.rados.rados.0.smithi085.stdout:3844: seq_num 1309 ranges {400363=513657,1332694=660122} 2021-02-10T04:42:55.490 INFO:tasks.rados.rados.0.smithi085.stdout:3844: writing smithi08575397-10 from 400363 to 914020 tid 1 2021-02-10T04:42:55.497 INFO:tasks.rados.rados.0.smithi085.stdout:3844: writing smithi08575397-10 from 1332694 to 1992816 tid 2 2021-02-10T04:42:55.497 INFO:tasks.rados.rados.0.smithi085.stdout:3845: write oid 6 current snap is 449 2021-02-10T04:42:55.497 INFO:tasks.rados.rados.0.smithi085.stdout:3845: seq_num 1310 ranges {417170=588906,1674823=342347} 2021-02-10T04:42:55.503 INFO:tasks.rados.rados.0.smithi085.stdout:3845: writing smithi08575397-6 from 417170 to 1006076 tid 1 2021-02-10T04:42:55.510 INFO:tasks.rados.rados.0.smithi085.stdout:3845: writing smithi08575397-6 from 1674823 to 2017170 tid 2 2021-02-10T04:42:55.510 INFO:tasks.rados.rados.0.smithi085.stdout:3844: finishing write tid 1 to smithi08575397-10 2021-02-10T04:42:55.510 INFO:tasks.rados.rados.0.smithi085.stdout:3844: finishing write tid 2 to smithi08575397-10 2021-02-10T04:42:55.510 INFO:tasks.rados.rados.0.smithi085.stdout:3844: finishing write tid 3 to smithi08575397-10 2021-02-10T04:42:55.510 INFO:tasks.rados.rados.0.smithi085.stdout:3844: finishing write tid 4 to smithi08575397-10 2021-02-10T04:42:55.510 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 10 v 1622 (ObjNum 1309 snap 449 seq_num 1309) dirty exists 2021-02-10T04:42:55.510 INFO:tasks.rados.rados.0.smithi085.stdout:3844: left oid 10 (ObjNum 1309 snap 449 seq_num 1309) 2021-02-10T04:42:55.511 INFO:tasks.rados.rados.0.smithi085.stdout:3845: finishing write tid 1 to smithi08575397-6 2021-02-10T04:42:55.511 INFO:tasks.rados.rados.0.smithi085.stdout:3844: done (5 left) 2021-02-10T04:42:55.511 INFO:tasks.rados.rados.0.smithi085.stdout:3846: read oid 13 snap -1 2021-02-10T04:42:55.511 INFO:tasks.rados.rados.0.smithi085.stdout:3846: expect (ObjNum 1241 snap 418 seq_num 1241) 2021-02-10T04:42:55.511 INFO:tasks.rados.rados.0.smithi085.stdout:3847: write oid 18 current snap is 449 2021-02-10T04:42:55.511 INFO:tasks.rados.rados.0.smithi085.stdout:3847: seq_num 1311 ranges {433977=664155,1733305=300672} 2021-02-10T04:42:55.516 INFO:tasks.rados.rados.0.smithi085.stdout:3847: writing smithi08575397-18 from 433977 to 1098132 tid 1 2021-02-10T04:42:55.522 INFO:tasks.rados.rados.0.smithi085.stdout:3847: writing smithi08575397-18 from 1733305 to 2033977 tid 2 2021-02-10T04:42:55.523 INFO:tasks.rados.rados.0.smithi085.stdout:3848: snap_remove snap 446 2021-02-10T04:42:55.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:55 smithi164 conmon[29288]: cluster 2021-02-10T04:42:55.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:55 smithi164 conmon[29288]: 2021-02-10T04:42:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:55 smithi164 conmon[29288]: 2021-02-10T04:42:53.772184 2021-02-10T04:42:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:55 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 463 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:42:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:55 smithi164 conmon[29288]: cluster 2021-02-10T04:42:53.772793 2021-02-10T04:42:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:55 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 464 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:42:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:55 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:42:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:55 smithi164 conmon[29288]: 02-10T04:42:54.243895+0000 mgr.y (mgr.14140) 2021-02-10T04:42:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:55 smithi164 conmon[29288]: 878 : cluster [DBG] pgmap v1581: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim_wait, 158 active+clean, 6 active+clean+snaptrim; 878 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 4.2 MiB/s rd, 2.2 MiB/s wr, 9 op/s 2021-02-10T04:42:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:55 smithi164 conmon[29288]: cluster 2021-02-10T04:42:54. 2021-02-10T04:42:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:55 smithi164 conmon[29288]: 467131+0000 mon.a (mon.0) 1937 : cluster [DBG] osdmap e954: 8 total, 8 up, 7 in 2021-02-10T04:42:55.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:55 smithi085 conmon[29319]: cluster 2021-02-10T04:42:53.772184+0000 osd.1 (osd.1) 463 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:42:55.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:55 smithi085 conmon[29319]: cluster 2021-02-10T04:42:53.772793+0000 osd.1 (osd.1) 464 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:42:55.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:55 smithi085 conmon[29319]: cluster 2021-02-10T04:42:54.243895+0000 mgr.y (mgr.14140) 878 : cluster [DBG] pgmap v1581: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim_wait, 158 active+clean, 6 active+clean+snaptrim; 878 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 4.2 MiB/s rd, 2.2 MiB/s wr, 9 op/s 2021-02-10T04:42:55.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:55 smithi085 conmon[29319]: cluster 2021-02-10T04:42:54.467131+0000 mon.a (mon.0) 1937 : cluster [DBG] osdmap e954: 8 total, 8 up, 7 in 2021-02-10T04:42:55.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:55 smithi085 conmon[35961]: cluster 2021-02-10T04:42:53.772184+0000 2021-02-10T04:42:55.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:55 smithi085 conmon[35961]: osd.1 (osd.1) 463 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:42:55.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:55 smithi085 conmon[35961]: cluster 2021-02-10T04:42:53.772793+0000 osd.1 (osd.1) 464 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:42:55.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:55 smithi085 conmon[35961]: cluster 2021-02-10T04:42:54.243895+0000 mgr.y (mgr.14140) 878 : cluster [DBG] pgmap v1581: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim_wait, 158 active+clean, 6 active+clean+snaptrim; 878 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 4.2 MiB/s rd, 2.2 MiB/s wr, 9 op/s 2021-02-10T04:42:55.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:55 smithi085 conmon[35961]: cluster 2021-02-10T04:42:54.467131+0000 mon.a (mon.0) 1937 : cluster [DBG] osdmap e954: 8 total, 8 up, 7 in 2021-02-10T04:42:55.942 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:42:55 smithi085 conmon[41540]: debug 2021-02-10T04:42:55.643+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:56.178 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:42:56.493 INFO:tasks.rados.rados.0.smithi085.stdout:3848: done (7 left) 2021-02-10T04:42:56.494 INFO:tasks.rados.rados.0.smithi085.stdout:3849: read oid 35 snap -1 2021-02-10T04:42:56.494 INFO:tasks.rados.rados.0.smithi085.stdout:3849: expect deleted 2021-02-10T04:42:56.494 INFO:tasks.rados.rados.0.smithi085.stdout:3850: read oid 16 snap -1 2021-02-10T04:42:56.494 INFO:tasks.rados.rados.0.smithi085.stdout:3850: expect deleted 2021-02-10T04:42:56.494 INFO:tasks.rados.rados.0.smithi085.stdout:3851: read oid 28 snap -1 2021-02-10T04:42:56.494 INFO:tasks.rados.rados.0.smithi085.stdout:3851: expect deleted 2021-02-10T04:42:56.494 INFO:tasks.rados.rados.0.smithi085.stdout:3852: read oid 32 snap -1 2021-02-10T04:42:56.494 INFO:tasks.rados.rados.0.smithi085.stdout:3852: expect (ObjNum 85 snap 11 seq_num 85) 2021-02-10T04:42:56.494 INFO:tasks.rados.rados.0.smithi085.stdout:3853: write oid 20 current snap is 449 2021-02-10T04:42:56.494 INFO:tasks.rados.rados.0.smithi085.stdout:3853: seq_num 1312 ranges {450784=739404,1791787=258997} 2021-02-10T04:42:56.498 INFO:tasks.rados.rados.0.smithi085.stdout:3853: writing smithi08575397-20 from 450784 to 1190188 tid 1 2021-02-10T04:42:56.504 INFO:tasks.rados.rados.0.smithi085.stdout:3853: writing smithi08575397-20 from 1791787 to 2050784 tid 2 2021-02-10T04:42:56.508 INFO:tasks.rados.rados.0.smithi085.stdout:3845: finishing write tid 2 to smithi08575397-6 2021-02-10T04:42:56.508 INFO:tasks.rados.rados.0.smithi085.stdout:3845: finishing write tid 3 to smithi08575397-6 2021-02-10T04:42:56.508 INFO:tasks.rados.rados.0.smithi085.stdout:3845: finishing write tid 4 to smithi08575397-6 2021-02-10T04:42:56.508 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 1890 (ObjNum 1310 snap 449 seq_num 1310) dirty exists 2021-02-10T04:42:56.508 INFO:tasks.rados.rados.0.smithi085.stdout:3845: left oid 6 (ObjNum 1310 snap 449 seq_num 1310) 2021-02-10T04:42:56.508 INFO:tasks.rados.rados.0.smithi085.stdout:3846: expect (ObjNum 1241 snap 418 seq_num 1241) 2021-02-10T04:42:56.676 INFO:tasks.rados.rados.0.smithi085.stdout:3847: finishing write tid 1 to smithi08575397-18 2021-02-10T04:42:56.676 INFO:tasks.rados.rados.0.smithi085.stdout:3847: finishing write tid 2 to smithi08575397-18 2021-02-10T04:42:56.676 INFO:tasks.rados.rados.0.smithi085.stdout:3847: finishing write tid 3 to smithi08575397-18 2021-02-10T04:42:56.676 INFO:tasks.rados.rados.0.smithi085.stdout:3847: finishing write tid 4 to smithi08575397-18 2021-02-10T04:42:56.676 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 18 v 1625 (ObjNum 1311 snap 449 seq_num 1311) dirty exists 2021-02-10T04:42:56.676 INFO:tasks.rados.rados.0.smithi085.stdout:3847: left oid 18 (ObjNum 1311 snap 449 seq_num 1311) 2021-02-10T04:42:56.677 INFO:tasks.rados.rados.0.smithi085.stdout:3845: done (11 left) 2021-02-10T04:42:56.677 INFO:tasks.rados.rados.0.smithi085.stdout:3846: done (10 left) 2021-02-10T04:42:56.677 INFO:tasks.rados.rados.0.smithi085.stdout:3847: done (9 left) 2021-02-10T04:42:56.677 INFO:tasks.rados.rados.0.smithi085.stdout:3854: delete oid 12 current snap is 449 2021-02-10T04:42:56.677 INFO:tasks.rados.rados.0.smithi085.stdout:3853: finishing write tid 1 to smithi08575397-20 2021-02-10T04:42:56.677 INFO:tasks.rados.rados.0.smithi085.stdout:3853: finishing write tid 2 to smithi08575397-20 2021-02-10T04:42:56.677 INFO:tasks.rados.rados.0.smithi085.stdout:3853: finishing write tid 3 to smithi08575397-20 2021-02-10T04:42:56.677 INFO:tasks.rados.rados.0.smithi085.stdout:3853: finishing write tid 4 to smithi08575397-20 2021-02-10T04:42:56.677 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 20 v 1890 (ObjNum 1312 snap 449 seq_num 1312) dirty exists 2021-02-10T04:42:56.677 INFO:tasks.rados.rados.0.smithi085.stdout:3853: left oid 20 (ObjNum 1312 snap 449 seq_num 1312) 2021-02-10T04:42:56.677 INFO:tasks.rados.rados.0.smithi085.stdout:3852: expect (ObjNum 85 snap 11 seq_num 85) 2021-02-10T04:42:56.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:56 smithi164 conmon[29288]: cluster 2021-02-10T04:42:56.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:56 smithi164 conmon[29288]: 2021-02-10T04:42:56.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:56 smithi164 conmon[29288]: 2021 2021-02-10T04:42:56.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:56 smithi164 conmon[29288]: - 2021-02-10T04:42:56.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:56 smithi164 conmon[29288]: 02 2021-02-10T04:42:56.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:56 smithi164 conmon[29288]: - 2021-02-10T04:42:56.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:56 smithi164 conmon[29288]: 10T04:42:55.483717+0000 2021-02-10T04:42:56.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:56 smithi164 conmon[29288]: mon.a (mon.0) 1938 : cluster [DBG] osdmap e955: 8 total, 8 up, 7 in 2021-02-10T04:42:56.926 INFO:tasks.rados.rados.0.smithi085.stdout:3849: done (9 left) 2021-02-10T04:42:56.926 INFO:tasks.rados.rados.0.smithi085.stdout:3850: done (8 left) 2021-02-10T04:42:56.927 INFO:tasks.rados.rados.0.smithi085.stdout:3851: done (7 left) 2021-02-10T04:42:56.927 INFO:tasks.rados.rados.0.smithi085.stdout:3852: done (6 left) 2021-02-10T04:42:56.927 INFO:tasks.rados.rados.0.smithi085.stdout:3853: done (5 left) 2021-02-10T04:42:56.927 INFO:tasks.rados.rados.0.smithi085.stdout:3854: done (4 left) 2021-02-10T04:42:56.927 INFO:tasks.rados.rados.0.smithi085.stdout:3855: read oid 6 snap -1 2021-02-10T04:42:56.927 INFO:tasks.rados.rados.0.smithi085.stdout:3855: expect (ObjNum 1310 snap 449 seq_num 1310) 2021-02-10T04:42:56.927 INFO:tasks.rados.rados.0.smithi085.stdout:3856: delete oid 1 current snap is 449 2021-02-10T04:42:56.930 INFO:tasks.rados.rados.0.smithi085.stdout:3856: done (5 left) 2021-02-10T04:42:56.930 INFO:tasks.rados.rados.0.smithi085.stdout:3857: delete oid 22 current snap is 449 2021-02-10T04:42:56.931 INFO:tasks.rados.rados.0.smithi085.stdout:3855: expect (ObjNum 1310 snap 449 seq_num 1310) 2021-02-10T04:42:56.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:56 smithi085 conmon[29319]: cluster 2021-02-10T04:42:55.483717+0000 mon.a (mon.0) 1938 : 2021-02-10T04:42:56.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:56 smithi085 conmon[29319]: cluster [DBG] osdmap e955: 8 total, 8 up, 7 in 2021-02-10T04:42:56.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:56 smithi085 conmon[35961]: cluster 2021-02-10T04:42:55.483717+0000 mon.a (mon 2021-02-10T04:42:56.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:56 smithi085 conmon[35961]: .0) 1938 : cluster [DBG] osdmap e955: 8 total, 8 up, 7 in 2021-02-10T04:42:57.050 INFO:tasks.rados.rados.0.smithi085.stdout:3855: done (5 left) 2021-02-10T04:42:57.050 INFO:tasks.rados.rados.0.smithi085.stdout:3857: done (4 left) 2021-02-10T04:42:57.051 INFO:tasks.rados.rados.0.smithi085.stdout:3858: snap_create 2021-02-10T04:42:57.495 INFO:tasks.rados.rados.0.smithi085.stdout:3858: done (4 left) 2021-02-10T04:42:57.495 INFO:tasks.rados.rados.0.smithi085.stdout:3859: snap_create 2021-02-10T04:42:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:57 smithi164 conmon[29288]: cluster 2021-02-10T04:42:56.244397+0000 2021-02-10T04:42:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:57 smithi164 conmon[29288]: mgr.y (mgr.14140) 879 : cluster [DBG] pgmap v1584: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim_wait, 161 active+clean, 4 active+clean+snaptrim; 885 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 8.4 MiB/s rd, 3.4 MiB/s wr, 14 op/s 2021-02-10T04:42:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:57 smithi164 conmon[29288]: cephadm 2021-02-10T04:42:56.311067+0000 mgr.y (mgr.14140) 880 : cephadm [INF] 2021-02-10T04:42:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:57 smithi164 conmon[29288]: Refreshing smithi085 facts 2021-02-10T04:42:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:57 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:42:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:57 smithi164 conmon[29288]: 04:42:56.490151+0000 mon.a (mon.0) 1939 : cluster [DBG] 2021-02-10T04:42:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:57 smithi164 conmon[29288]: osdmap e956: 8 total, 8 up, 7 in 2021-02-10T04:42:57.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:57 smithi085 conmon[35961]: cluster 2021-02-10T04:42:56. 2021-02-10T04:42:57.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:57 smithi085 conmon[35961]: 244397+0000 mgr.y (mgr.14140) 879 : cluster [DBG] pgmap v1584: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim_wait, 161 active+clean, 4 active+clean+snaptrim; 885 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 8.4 MiB/s rd, 3.4 MiB/s wr, 14 op/s 2021-02-10T04:42:57.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:57 smithi085 conmon[35961]: cephadm 2021-02-10T04:42:56.311067+0000 2021-02-10T04:42:57.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:57 smithi085 conmon[35961]: mgr.y (mgr.14140) 880 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:42:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:57 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:42:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:57 smithi085 conmon[35961]: -02-10T04:42:56.490151+0000 mon.a 2021-02-10T04:42:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:57 smithi085 conmon[35961]: (mon.0) 1939 : cluster [DBG] osdmap e956: 8 total, 8 up, 7 in 2021-02-10T04:42:57.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:57 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:42:57.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:57 smithi085 conmon[29319]: 04:42:56.244397+0000 mgr.y (mgr.14140) 879 : cluster [DBG] pgmap v1584: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim_wait, 161 active+clean, 4 active+clean+snaptrim; 885 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 8.4 MiB/s rd, 3.4 MiB/s wr, 14 op/s 2021-02-10T04:42:57.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:57 smithi085 conmon[29319]: cephadm 2021-02-10T04:42:56.311067+0000 2021-02-10T04:42:57.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:57 smithi085 conmon[29319]: mgr.y (mgr.14140) 880 : cephadm [INF] 2021-02-10T04:42:57.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:57 smithi085 conmon[29319]: Refreshing smithi085 facts 2021-02-10T04:42:57.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:57 smithi085 conmon[29319]: cluster 2021-02-10T04:42:57.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:57 smithi085 conmon[29319]: 2021-02-10T04:42:56.490151+0000 mon.a 2021-02-10T04:42:57.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:57 smithi085 conmon[29319]: (mon.0) 1939 : cluster [DBG] osdmap e956: 8 total, 8 up, 7 in 2021-02-10T04:42:58.500 INFO:tasks.rados.rados.0.smithi085.stdout:3859: done (4 left) 2021-02-10T04:42:58.501 INFO:tasks.rados.rados.0.smithi085.stdout:3860: snap_remove snap 424 2021-02-10T04:42:58.698 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:42:58.734 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:58.828 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:58 smithi164 conmon[29288]: cluster 2021-02-10T04:42:57.494071+0000 mon.a 2021-02-10T04:42:58.828 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:58 smithi164 conmon[29288]: (mon.0) 1940 : cluster [DBG] osdmap e957: 8 total, 8 up, 7 in 2021-02-10T04:42:58.828 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:58 smithi164 conmon[29288]: audit 2021-02-10T04:42:58.325866+0000 mon.a (mon.0) 1941 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:42:58.828 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:42:58 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:42:58.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:58 smithi085 conmon[29319]: cluster 2021-02-10T04:42:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:58 smithi085 conmon[29319]: 2021-02-10T04:42:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:58 smithi085 conmon[29319]: 2021 2021-02-10T04:42:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:58 smithi085 conmon[29319]: - 2021-02-10T04:42:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:58 smithi085 conmon[29319]: 02 2021-02-10T04:42:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:58 smithi085 conmon[29319]: - 2021-02-10T04:42:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:58 smithi085 conmon[29319]: 10 2021-02-10T04:42:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:58 smithi085 conmon[29319]: T04:42:57.494071+0000 mon.a 2021-02-10T04:42:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:58 smithi085 conmon[29319]: (mon.0) 1940 : cluster [DBG] osdmap e957: 8 total, 8 up, 7 in 2021-02-10T04:42:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:58 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:42:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:58 smithi085 conmon[29319]: :42:58.325866+0000 mon.a (mon.0) 1941 2021-02-10T04:42:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:58 smithi085 conmon[29319]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:42:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:58 smithi085 conmon[35961]: cluster 2021-02-10T04:42:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:58 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:42:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:58 smithi085 conmon[35961]: 04:42:57.494071+0000 mon.a (mon.0) 1940 : cluster [DBG] osdmap e957: 8 total, 8 up, 7 in 2021-02-10T04:42:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:58 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:42:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:58 smithi085 conmon[35961]: 02-10T04:42:58.325866+0000 mon.a (mon.0 2021-02-10T04:42:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:58 smithi085 conmon[35961]: ) 1941 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:42:59.143 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:42:58 smithi164 conmon[40892]: debug 2021-02-10T04:42:59.144 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:42:58 smithi164 conmon[40892]: 2021-02-10T04:42:58.826+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:59.144 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:42:58 smithi164 conmon[40892]: 2021-02-10T04:42:59.373 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:42:59.479 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:42:59.501 INFO:tasks.rados.rados.0.smithi085.stdout:3860: done (4 left) 2021-02-10T04:42:59.501 INFO:tasks.rados.rados.0.smithi085.stdout:3861: write (excl) oid 43 current snap is 451 2021-02-10T04:42:59.501 INFO:tasks.rados.rados.0.smithi085.stdout:3861: seq_num 1316 ranges {518012=756753,2025715=92297} 2021-02-10T04:42:59.510 INFO:tasks.rados.rados.0.smithi085.stdout:3861: writing smithi08575397-43 from 518012 to 1274765 tid 1 2021-02-10T04:42:59.512 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:42:59.515 INFO:tasks.rados.rados.0.smithi085.stdout:3861: writing smithi08575397-43 from 2025715 to 2118012 tid 2 2021-02-10T04:42:59.515 INFO:tasks.rados.rados.0.smithi085.stdout:3862: write (excl) oid 3 current snap is 451 2021-02-10T04:42:59.515 INFO:tasks.rados.rados.0.smithi085.stdout:3862: seq_num 1317 ranges {534819=432002,1684197=450622} 2021-02-10T04:42:59.521 INFO:tasks.rados.rados.0.smithi085.stdout:3862: writing smithi08575397-3 from 534819 to 966821 tid 1 2021-02-10T04:42:59.528 INFO:tasks.rados.rados.0.smithi085.stdout:3862: writing smithi08575397-3 from 1684197 to 2134819 tid 2 2021-02-10T04:42:59.528 INFO:tasks.rados.rados.0.smithi085.stdout:3863: snap_remove snap 431 2021-02-10T04:42:59.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:42:59 smithi164 conmon[37263]: debug 2021-02-10T04:42:59.597+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:42:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:59 smithi164 conmon[29288]: cluster 2021-02-10T04:42:58.245088+0000 mgr.y (mgr.14140) 881 2021-02-10T04:42:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:59 smithi164 conmon[29288]: : cluster [DBG] pgmap v1587: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim_wait, 161 active+clean, 4 active+clean+snaptrim; 884 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 24 MiB/s rd, 1.2 MiB/s wr, 17 op/s 2021-02-10T04:42:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:59 smithi164 conmon[29288]: cephadm 2021-02-10T04:42:58.326677+0000 mgr.y (mgr.14140) 882 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:42:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:59 smithi164 conmon[29288]: cluster 2021-02-10T04:42: 2021-02-10T04:42:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:59 smithi164 conmon[29288]: 58.500695+0000 mon.a (mon.0) 1942 : cluster [DBG] 2021-02-10T04:42:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:59 smithi164 conmon[29288]: osdmap e958: 8 total, 8 up, 7 in 2021-02-10T04:42:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:59 smithi164 conmon[29288]: audit 2021-02-10T04: 2021-02-10T04:42:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:59 smithi164 conmon[29288]: 42:58.549075+0000 mgr.y (mgr.14140) 883 2021-02-10T04:42:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:59 smithi164 conmon[29288]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:42:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:59 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:42:59.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:59 smithi164 conmon[29288]: -10T04:42:58.873336+0000 mon.a (mon 2021-02-10T04:42:59.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:42:59 smithi164 conmon[29288]: .0) 1943 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:42:59.873 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:42:59.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:59 smithi085 conmon[29319]: cluster 2021-02-10T04:42:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:59 smithi085 conmon[29319]: 2021-02-10T04:42: 2021-02-10T04:42:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:59 smithi085 conmon[29319]: 58.245088+0000 mgr.y (mgr.14140) 881 : cluster [DBG] pgmap v1587: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim_wait, 161 active+clean, 4 active+clean+snaptrim; 884 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 24 MiB/s rd, 1.2 MiB/s wr, 17 op/s 2021-02-10T04:42:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:59 smithi085 conmon[29319]: cephadm 2021-02-10T04:42:58.326677+0000 2021-02-10T04:42:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:59 smithi085 conmon[29319]: mgr.y (mgr.14140) 882 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:42:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:59 smithi085 conmon[29319]: 2021-02-10T04:42:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:59 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:42:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:59 smithi085 conmon[29319]: 02-10T04:42:58.500695+0000 mon.a (mon.0) 2021-02-10T04:42:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:59 smithi085 conmon[29319]: 1942 : cluster [DBG] osdmap e958: 8 total, 8 up, 7 in 2021-02-10T04:42:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:59 smithi085 conmon[29319]: audit 2021-02-10T04:42:58.549075+0000 mgr.y (mgr.14140) 883 2021-02-10T04:42:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:59 smithi085 conmon[29319]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:42:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:59 smithi085 conmon[29319]: audit 2021-02-10T04:42:59.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:59 smithi085 conmon[29319]: 2021-02-10T04:42:58.873336+0000 mon.a (mon.0) 2021-02-10T04:42:59.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:42:59 smithi085 conmon[29319]: 1943 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:42:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:59 smithi085 conmon[35961]: cluster 2021-02-10T04:42:58.245088+0000 2021-02-10T04:42:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:59 smithi085 conmon[35961]: mgr.y (mgr.14140) 881 : cluster [DBG] pgmap v1587: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim_wait, 161 active+clean, 4 active+clean+snaptrim; 884 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 24 MiB/s rd, 1.2 MiB/s wr, 17 op/s 2021-02-10T04:42:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:59 smithi085 conmon[35961]: cephadm 2021-02-10T04:42:58.326677+0000 2021-02-10T04:42:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:59 smithi085 conmon[35961]: mgr.y (mgr.14140) 882 : cephadm [INF] 2021-02-10T04:42:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:59 smithi085 conmon[35961]: Refreshing smithi164 facts 2021-02-10T04:42:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:59 smithi085 conmon[35961]: cluster 2021-02-10T04:42 2021-02-10T04:42:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:59 smithi085 conmon[35961]: :58.500695+0000 mon.a (mon.0) 1942 2021-02-10T04:42:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:59 smithi085 conmon[35961]: : cluster [DBG] osdmap e958: 8 total, 8 up, 7 in 2021-02-10T04:42:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:59 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:42:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:59 smithi085 conmon[35961]: 04:42:58.549075+0000 mgr.y (mgr. 2021-02-10T04:42:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:59 smithi085 conmon[35961]: 14140) 883 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:42:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:59 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:42:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:59 smithi085 conmon[35961]: -10T04:42:58.873336+0000 mon.a 2021-02-10T04:42:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:59 smithi085 conmon[35961]: (mon.0) 1943 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:42:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:42:59 smithi085 conmon[35961]: 2021-02-10T04:42:59.978 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:43:00.015 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:43:00.441 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:43:00 smithi085 conmon[45443]: debug 2021-02-10T04:43:00.103+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:43:00.526 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:43:00.526 INFO:tasks.rados.rados.0.smithi085.stdout:3863: done (6 left) 2021-02-10T04:43:00.526 INFO:tasks.rados.rados.0.smithi085.stdout:3864: write oid 44 current snap is 451 2021-02-10T04:43:00.526 INFO:tasks.rados.rados.0.smithi085.stdout:3861: finishing write tid 1 to smithi08575397-43 2021-02-10T04:43:00.527 INFO:tasks.rados.rados.0.smithi085.stdout:3864: seq_num 1318 ranges {551626=507251,1626326=418456} 2021-02-10T04:43:00.532 INFO:tasks.rados.rados.0.smithi085.stdout:3864: writing smithi08575397-44 from 551626 to 1058877 tid 1 2021-02-10T04:43:00.539 INFO:tasks.rados.rados.0.smithi085.stdout:3864: writing smithi08575397-44 from 1626326 to 2044782 tid 2 2021-02-10T04:43:00.539 INFO:tasks.rados.rados.0.smithi085.stdout:3865: read oid 7 snap -1 2021-02-10T04:43:00.539 INFO:tasks.rados.rados.0.smithi085.stdout:3865: expect deleted 2021-02-10T04:43:00.539 INFO:tasks.rados.rados.0.smithi085.stdout:3866: rollback oid 48 current snap is 451 2021-02-10T04:43:00.539 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 48 to 369 2021-02-10T04:43:00.539 INFO:tasks.rados.rados.0.smithi085.stdout:3861: finishing write tid 2 to smithi08575397-43 2021-02-10T04:43:00.539 INFO:tasks.rados.rados.0.smithi085.stdout:3861: finishing write tid 3 to smithi08575397-43 2021-02-10T04:43:00.539 INFO:tasks.rados.rados.0.smithi085.stdout:3861: finishing write tid 4 to smithi08575397-43 2021-02-10T04:43:00.540 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 1890 (ObjNum 1316 snap 451 seq_num 1316) dirty exists 2021-02-10T04:43:00.540 INFO:tasks.rados.rados.0.smithi085.stdout:3861: left oid 43 (ObjNum 1316 snap 451 seq_num 1316) 2021-02-10T04:43:00.540 INFO:tasks.rados.rados.0.smithi085.stdout:3862: finishing write tid 1 to smithi08575397-3 2021-02-10T04:43:00.540 INFO:tasks.rados.rados.0.smithi085.stdout:3862: finishing write tid 2 to smithi08575397-3 2021-02-10T04:43:00.540 INFO:tasks.rados.rados.0.smithi085.stdout:3862: finishing write tid 3 to smithi08575397-3 2021-02-10T04:43:00.540 INFO:tasks.rados.rados.0.smithi085.stdout:3862: finishing write tid 4 to smithi08575397-3 2021-02-10T04:43:00.540 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 1893 (ObjNum 1317 snap 451 seq_num 1317) dirty exists 2021-02-10T04:43:00.540 INFO:tasks.rados.rados.0.smithi085.stdout:3862: left oid 3 (ObjNum 1317 snap 451 seq_num 1317) 2021-02-10T04:43:00.540 INFO:tasks.rados.rados.0.smithi085.stdout:3861: done (8 left) 2021-02-10T04:43:00.540 INFO:tasks.rados.rados.0.smithi085.stdout:3862: done (7 left) 2021-02-10T04:43:00.540 INFO:tasks.rados.rados.0.smithi085.stdout:3867: read oid 36 snap -1 2021-02-10T04:43:00.540 INFO:tasks.rados.rados.0.smithi085.stdout:3867: expect (ObjNum 1301 snap 439 seq_num 1301) 2021-02-10T04:43:00.540 INFO:tasks.rados.rados.0.smithi085.stdout:3868: read oid 38 snap -1 2021-02-10T04:43:00.540 INFO:tasks.rados.rados.0.smithi085.stdout:3868: expect (ObjNum 1296 snap 436 seq_num 1296) 2021-02-10T04:43:00.541 INFO:tasks.rados.rados.0.smithi085.stdout:3869: write oid 3 current snap is 451 2021-02-10T04:43:00.541 INFO:tasks.rados.rados.0.smithi085.stdout:3869: seq_num 1319 ranges {568433=582500,1684808=483625} 2021-02-10T04:43:00.546 INFO:tasks.rados.rados.0.smithi085.stdout:3869: writing smithi08575397-3 from 568433 to 1150933 tid 1 2021-02-10T04:43:00.552 INFO:tasks.rados.rados.0.smithi085.stdout:3869: writing smithi08575397-3 from 1684808 to 2168433 tid 2 2021-02-10T04:43:00.561 INFO:tasks.rados.rados.0.smithi085.stdout:3864: finishing write tid 1 to smithi08575397-44 2021-02-10T04:43:00.562 INFO:tasks.rados.rados.0.smithi085.stdout:3864: finishing write tid 2 to smithi08575397-44 2021-02-10T04:43:00.562 INFO:tasks.rados.rados.0.smithi085.stdout:3866: finishing rollback tid 0 to smithi08575397-48 2021-02-10T04:43:00.562 INFO:tasks.rados.rados.0.smithi085.stdout:3866: finishing rollback tid 1 to smithi08575397-48 2021-02-10T04:43:00.562 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 1896 (ObjNum 71101512 snap 22030 seq_num 1) dirty dne 2021-02-10T04:43:00.563 INFO:tasks.rados.rados.0.smithi085.stdout:3864: finishing write tid 3 to smithi08575397-44 2021-02-10T04:43:00.563 INFO:tasks.rados.rados.0.smithi085.stdout:3864: finishing write tid 4 to smithi08575397-44 2021-02-10T04:43:00.563 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 1893 (ObjNum 1318 snap 451 seq_num 1318) dirty exists 2021-02-10T04:43:00.563 INFO:tasks.rados.rados.0.smithi085.stdout:3864: left oid 44 (ObjNum 1318 snap 451 seq_num 1318) 2021-02-10T04:43:00.564 INFO:tasks.rados.rados.0.smithi085.stdout:3868: expect (ObjNum 1296 snap 436 seq_num 1296) 2021-02-10T04:43:00.817 INFO:tasks.rados.rados.0.smithi085.stdout:3864: done (9 left) 2021-02-10T04:43:00.817 INFO:tasks.rados.rados.0.smithi085.stdout:3865: done (8 left) 2021-02-10T04:43:00.817 INFO:tasks.rados.rados.0.smithi085.stdout:3866: done (7 left) 2021-02-10T04:43:00.818 INFO:tasks.rados.rados.0.smithi085.stdout:3868: done (6 left) 2021-02-10T04:43:00.818 INFO:tasks.rados.rados.0.smithi085.stdout:3870: read oid 40 snap -1 2021-02-10T04:43:00.818 INFO:tasks.rados.rados.0.smithi085.stdout:3870: expect deleted 2021-02-10T04:43:00.818 INFO:tasks.rados.rados.0.smithi085.stdout:3871: read oid 49 snap -1 2021-02-10T04:43:00.818 INFO:tasks.rados.rados.0.smithi085.stdout:3871: expect deleted 2021-02-10T04:43:00.818 INFO:tasks.rados.rados.0.smithi085.stdout:3867: expect (ObjNum 1301 snap 439 seq_num 1301) 2021-02-10T04:43:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:00 smithi164 conmon[29288]: cluster 2021-02-10T04:43:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:00 smithi164 conmon[29288]: 2021-02-10T04:43:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:00 smithi164 conmon[29288]: 2021 2021-02-10T04:43:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:00 smithi164 conmon[29288]: -02-10T04:42:59.500886 2021-02-10T04:43:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:00 smithi164 conmon[29288]: +0000 2021-02-10T04:43:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:00 smithi164 conmon[29288]: 2021-02-10T04:43:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:00 smithi164 conmon[29288]: mon.a 2021-02-10T04:43:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:00 smithi164 conmon[29288]: (mon.0) 1944 2021-02-10T04:43:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:00 smithi164 conmon[29288]: : cluster [DBG] osdmap e959: 8 total, 8 up, 7 in 2021-02-10T04:43:00.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:00 smithi085 conmon[35961]: cluster 2021-02-10T04:42:59.500886+0000 mon.a 2021-02-10T04:43:00.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:00 smithi085 conmon[35961]: (mon.0) 1944 : cluster [DBG] osdmap e959: 8 total, 8 up, 7 in 2021-02-10T04:43:00.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:00 smithi085 conmon[29319]: cluster 2021-02-10T04:43:00.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:00 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:43:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:00 smithi085 conmon[29319]: :42:59.500886+0000 mon.a (mon.0) 1944 : cluster [DBG] osdmap e959: 8 total, 8 up, 7 in 2021-02-10T04:43:01.018 INFO:tasks.rados.rados.0.smithi085.stdout:3869: finishing write tid 1 to smithi08575397-3 2021-02-10T04:43:01.018 INFO:tasks.rados.rados.0.smithi085.stdout:3869: finishing write tid 2 to smithi08575397-3 2021-02-10T04:43:01.019 INFO:tasks.rados.rados.0.smithi085.stdout:3869: finishing write tid 3 to smithi08575397-3 2021-02-10T04:43:01.019 INFO:tasks.rados.rados.0.smithi085.stdout:3869: finishing write tid 4 to smithi08575397-3 2021-02-10T04:43:01.019 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 1896 (ObjNum 1319 snap 451 seq_num 1319) dirty exists 2021-02-10T04:43:01.019 INFO:tasks.rados.rados.0.smithi085.stdout:3869: left oid 3 (ObjNum 1319 snap 451 seq_num 1319) 2021-02-10T04:43:01.019 INFO:tasks.rados.rados.0.smithi085.stdout:3872: read oid 22 snap -1 2021-02-10T04:43:01.019 INFO:tasks.rados.rados.0.smithi085.stdout:3872: expect deleted 2021-02-10T04:43:01.019 INFO:tasks.rados.rados.0.smithi085.stdout:3867: done (8 left) 2021-02-10T04:43:01.019 INFO:tasks.rados.rados.0.smithi085.stdout:3869: done (7 left) 2021-02-10T04:43:01.019 INFO:tasks.rados.rados.0.smithi085.stdout:3873: read oid 45 snap 434 2021-02-10T04:43:01.019 INFO:tasks.rados.rados.0.smithi085.stdout:3873: expect (ObjNum 1276 snap 430 seq_num 1276) 2021-02-10T04:43:01.019 INFO:tasks.rados.rados.0.smithi085.stdout:3874: rollback oid 35 current snap is 451 2021-02-10T04:43:01.019 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 35 to 429 2021-02-10T04:43:01.020 INFO:tasks.rados.rados.0.smithi085.stdout:3870: done (8 left) 2021-02-10T04:43:01.020 INFO:tasks.rados.rados.0.smithi085.stdout:3871: done (7 left) 2021-02-10T04:43:01.020 INFO:tasks.rados.rados.0.smithi085.stdout:3875: delete oid 49 current snap is 451 2021-02-10T04:43:01.020 INFO:tasks.rados.rados.0.smithi085.stdout:3874: finishing rollback tid 1 to smithi08575397-35 2021-02-10T04:43:01.020 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 35 v 1602 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-02-10T04:43:01.025 INFO:tasks.rados.rados.0.smithi085.stdout:3874: done (7 left) 2021-02-10T04:43:01.025 INFO:tasks.rados.rados.0.smithi085.stdout:3875: done (6 left) 2021-02-10T04:43:01.025 INFO:tasks.rados.rados.0.smithi085.stdout:3876: snap_remove snap 428 2021-02-10T04:43:01.529 INFO:tasks.rados.rados.0.smithi085.stdout:3876: done (6 left) 2021-02-10T04:43:01.529 INFO:tasks.rados.rados.0.smithi085.stdout:3877: read oid 18 snap 407 2021-02-10T04:43:01.529 INFO:tasks.rados.rados.0.smithi085.stdout:3877: expect (ObjNum 1161 snap 398 seq_num 1161) 2021-02-10T04:43:01.529 INFO:tasks.rados.rados.0.smithi085.stdout:3878: snap_create 2021-02-10T04:43:01.529 INFO:tasks.rados.rados.0.smithi085.stdout:3873: expect (ObjNum 1276 snap 430 seq_num 1276) 2021-02-10T04:43:01.718 INFO:tasks.rados.rados.0.smithi085.stdout:3877: expect (ObjNum 1161 snap 398 seq_num 1161) 2021-02-10T04:43:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:01 smithi164 conmon[29288]: cluster 2021-02-10T04:43:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:01 smithi164 conmon[29288]: 2021-02-10T04:43:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:01 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:43:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:01 smithi164 conmon[29288]: :43:00.245807+0000 mgr.y (mgr.14140) 884 : cluster [DBG] pgmap v1590: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim_wait, 157 active+clean, 7 active+clean+snaptrim; 892 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 22 MiB/s rd, 1.7 MiB/s wr, 27 op/s 2021-02-10T04:43:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:01 smithi164 conmon[29288]: cluster 2021-02-10T04:43:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:01 smithi164 conmon[29288]: 2021-02-10T04:43:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:01 smithi164 conmon[29288]: 2021 2021-02-10T04:43:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:01 smithi164 conmon[29288]: - 2021-02-10T04:43:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:01 smithi164 conmon[29288]: 02 2021-02-10T04:43:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:01 smithi164 conmon[29288]: - 2021-02-10T04:43:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:01 smithi164 conmon[29288]: 10 2021-02-10T04:43:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:01 smithi164 conmon[29288]: T 2021-02-10T04:43:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:01 smithi164 conmon[29288]: 04 2021-02-10T04:43:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:01 smithi164 conmon[29288]: : 2021-02-10T04:43:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:01 smithi164 conmon[29288]: 43 2021-02-10T04:43:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:01 smithi164 conmon[29288]: : 2021-02-10T04:43:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:01 smithi164 conmon[29288]: 00 2021-02-10T04:43:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:01 smithi164 conmon[29288]: .524771+0000 mon.a (mon 2021-02-10T04:43:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:01 smithi164 conmon[29288]: .0) 1945 : cluster [DBG] osdmap e960: 8 total, 8 up, 7 in 2021-02-10T04:43:01.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:01 smithi085 conmon[29319]: cluster 2021-02-10T04:43:01.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:01 smithi085 conmon[29319]: 2021-02-10T04:43:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:01 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:43:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:01 smithi085 conmon[29319]: T04:43:00.245807+0000 mgr.y (mgr.14140) 884 : cluster [DBG] pgmap v1590: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim_wait, 157 active+clean, 7 active+clean+snaptrim; 892 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 22 MiB/s rd, 1.7 MiB/s wr, 27 op/s 2021-02-10T04:43:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:01 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:43:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:01 smithi085 conmon[29319]: 10T04:43:00.524771+0000 mon.a (mon.0) 1945 : 2021-02-10T04:43:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:01 smithi085 conmon[29319]: cluster [DBG] osdmap e960: 8 total, 8 up, 7 in 2021-02-10T04:43:01.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:01 smithi085 conmon[35961]: cluster 2021-02-10T04:43:01.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:01 smithi085 conmon[35961]: 2021-02-10T04:43:01.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:01 smithi085 conmon[35961]: 2021 2021-02-10T04:43:01.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:01 smithi085 conmon[35961]: - 2021-02-10T04:43:01.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:01 smithi085 conmon[35961]: 02 2021-02-10T04:43:01.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:01 smithi085 conmon[35961]: -10T04:43: 2021-02-10T04:43:01.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:01 smithi085 conmon[35961]: 00.245807+0000 mgr.y (mgr.14140) 884 : cluster [DBG] pgmap v1590: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim_wait, 157 active+clean, 7 active+clean+snaptrim; 892 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 22 MiB/s rd, 1.7 MiB/s wr, 27 op/s 2021-02-10T04:43:01.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:01 smithi085 conmon[35961]: cluster 2021-02-10T04:43:01.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:01 smithi085 conmon[35961]: 2021-02 2021-02-10T04:43:01.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:01 smithi085 conmon[35961]: -10T04:43:00.524771+0000 mon.a (mon.0 2021-02-10T04:43:01.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:01 smithi085 conmon[35961]: ) 1945 : cluster [DBG] osdmap e960: 8 total, 8 up, 7 in 2021-02-10T04:43:02.536 INFO:tasks.rados.rados.0.smithi085.stdout:3872: done (7 left) 2021-02-10T04:43:02.536 INFO:tasks.rados.rados.0.smithi085.stdout:3873: done (6 left) 2021-02-10T04:43:02.536 INFO:tasks.rados.rados.0.smithi085.stdout:3877: done (5 left) 2021-02-10T04:43:02.537 INFO:tasks.rados.rados.0.smithi085.stdout:3878: done (4 left) 2021-02-10T04:43:02.537 INFO:tasks.rados.rados.0.smithi085.stdout:3879: delete oid 12 current snap is 452 2021-02-10T04:43:02.540 INFO:tasks.rados.rados.0.smithi085.stdout:3879: done (4 left) 2021-02-10T04:43:02.540 INFO:tasks.rados.rados.0.smithi085.stdout:3880: copy_from oid 20 from oid 31 current snap is 452 2021-02-10T04:43:02.540 INFO:tasks.rados.rados.0.smithi085.stdout:3881: read oid 46 snap -1 2021-02-10T04:43:02.540 INFO:tasks.rados.rados.0.smithi085.stdout:3881: expect (ObjNum 1292 snap 436 seq_num 1292) 2021-02-10T04:43:02.540 INFO:tasks.rados.rados.0.smithi085.stdout:3882: read oid 8 snap -1 2021-02-10T04:43:02.540 INFO:tasks.rados.rados.0.smithi085.stdout:3882: expect (ObjNum 1306 snap 446 seq_num 1306) 2021-02-10T04:43:02.540 INFO:tasks.rados.rados.0.smithi085.stdout:3883: delete oid 48 current snap is 452 2021-02-10T04:43:02.543 INFO:tasks.rados.rados.0.smithi085.stdout:3883: done (7 left) 2021-02-10T04:43:02.543 INFO:tasks.rados.rados.0.smithi085.stdout:3884: write oid 15 current snap is 452 2021-02-10T04:43:02.543 INFO:tasks.rados.rados.0.smithi085.stdout:3884: seq_num 1323 ranges {635661=599849,1918736=316925} 2021-02-10T04:43:02.552 INFO:tasks.rados.rados.0.smithi085.stdout:3884: writing smithi08575397-15 from 635661 to 1235510 tid 1 2021-02-10T04:43:02.559 INFO:tasks.rados.rados.0.smithi085.stdout:3884: writing smithi08575397-15 from 1918736 to 2235661 tid 2 2021-02-10T04:43:02.559 INFO:tasks.rados.rados.0.smithi085.stdout:3882: expect (ObjNum 1306 snap 446 seq_num 1306) 2021-02-10T04:43:02.733 INFO:tasks.rados.rados.0.smithi085.stdout:3881: expect (ObjNum 1292 snap 436 seq_num 1292) 2021-02-10T04:43:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:02 smithi164 conmon[29288]: cluster 2021-02-10T04:43:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:02 smithi164 conmon[29288]: 2021-02-10T04:43:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:02 smithi164 conmon[29288]: 2021-02-10T04:43: 2021-02-10T04:43:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:02 smithi164 conmon[29288]: 01.528260+0000 mon.a (mon.0) 1946 : cluster [DBG] osdmap e961: 8 total, 8 up, 7 in 2021-02-10T04:43:02.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:02 smithi085 conmon[29319]: cluster 2021-02-10T04:43:02.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:02 smithi085 conmon[29319]: 2021-02-10T04:43:02.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:02 smithi085 conmon[29319]: 2021-02-10T04:43:01. 2021-02-10T04:43:02.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:02 smithi085 conmon[29319]: 528260+0000 mon.a (mon.0) 1946 : cluster [DBG] osdmap e961: 8 total, 8 up, 7 in 2021-02-10T04:43:02.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:02 smithi085 conmon[35961]: cluster 2021-02-10T04:43:01.528260 2021-02-10T04:43:02.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:02 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1946 : cluster [DBG] osdmap e961: 8 total, 8 up, 7 in 2021-02-10T04:43:02.978 INFO:tasks.rados.rados.0.smithi085.stdout:3884: finishing write tid 1 to smithi08575397-15 2021-02-10T04:43:02.978 INFO:tasks.rados.rados.0.smithi085.stdout:3884: finishing write tid 2 to smithi08575397-15 2021-02-10T04:43:02.978 INFO:tasks.rados.rados.0.smithi085.stdout:3884: finishing write tid 3 to smithi08575397-15 2021-02-10T04:43:02.978 INFO:tasks.rados.rados.0.smithi085.stdout:3884: finishing write tid 4 to smithi08575397-15 2021-02-10T04:43:02.978 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 15 v 1506 (ObjNum 1323 snap 452 seq_num 1323) dirty exists 2021-02-10T04:43:02.979 INFO:tasks.rados.rados.0.smithi085.stdout:3884: left oid 15 (ObjNum 1323 snap 452 seq_num 1323) 2021-02-10T04:43:02.979 INFO:tasks.rados.rados.0.smithi085.stdout:3880: finishing copy_from to smithi08575397-20 2021-02-10T04:43:02.979 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 20 v 1888 (ObjNum 1288 snap 433 seq_num 1288) dirty exists 2021-02-10T04:43:02.979 INFO:tasks.rados.rados.0.smithi085.stdout:3880: finishing copy_from racing read to smithi08575397-20 2021-02-10T04:43:02.979 INFO:tasks.rados.rados.0.smithi085.stdout:3880: done (7 left) 2021-02-10T04:43:02.979 INFO:tasks.rados.rados.0.smithi085.stdout:3881: done (6 left) 2021-02-10T04:43:02.979 INFO:tasks.rados.rados.0.smithi085.stdout:3882: done (5 left) 2021-02-10T04:43:02.979 INFO:tasks.rados.rados.0.smithi085.stdout:3884: done (4 left) 2021-02-10T04:43:02.979 INFO:tasks.rados.rados.0.smithi085.stdout:3885: read oid 3 snap -1 2021-02-10T04:43:02.979 INFO:tasks.rados.rados.0.smithi085.stdout:3885: expect (ObjNum 1319 snap 451 seq_num 1319) 2021-02-10T04:43:02.980 INFO:tasks.rados.rados.0.smithi085.stdout:3886: read oid 21 snap -1 2021-02-10T04:43:02.980 INFO:tasks.rados.rados.0.smithi085.stdout:3886: expect (ObjNum 1269 snap 428 seq_num 1269) 2021-02-10T04:43:02.980 INFO:tasks.rados.rados.0.smithi085.stdout:3887: snap_remove snap 448 2021-02-10T04:43:03.326 INFO:tasks.rados.rados.0.smithi085.stdout:3887: done (6 left) 2021-02-10T04:43:03.327 INFO:tasks.rados.rados.0.smithi085.stdout:3888: write (excl) oid 43 current snap is 452 2021-02-10T04:43:03.327 INFO:tasks.rados.rados.0.smithi085.stdout:3888: seq_num 1324 ranges {652468=675098,1977218=275250} 2021-02-10T04:43:03.334 INFO:tasks.rados.rados.0.smithi085.stdout:3888: writing smithi08575397-43 from 652468 to 1327566 tid 1 2021-02-10T04:43:03.340 INFO:tasks.rados.rados.0.smithi085.stdout:3888: writing smithi08575397-43 from 1977218 to 2252468 tid 2 2021-02-10T04:43:03.340 INFO:tasks.rados.rados.0.smithi085.stdout:3889: write (excl) oid 36 current snap is 452 2021-02-10T04:43:03.340 INFO:tasks.rados.rados.0.smithi085.stdout:3889: seq_num 1325 ranges {669275=750347,2035700=233575} 2021-02-10T04:43:03.349 INFO:tasks.rados.rados.0.smithi085.stdout:3889: writing smithi08575397-36 from 669275 to 1419622 tid 1 2021-02-10T04:43:03.355 INFO:tasks.rados.rados.0.smithi085.stdout:3889: writing smithi08575397-36 from 2035700 to 2269275 tid 2 2021-02-10T04:43:03.355 INFO:tasks.rados.rados.0.smithi085.stdout:3890: snap_remove snap 441 2021-02-10T04:43:03.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:03 smithi085 conmon[29319]: cluster 2021-02-10T04:43:03.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:03 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:43:03.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:03 smithi085 conmon[29319]: 04:43:02.246535+0000 mgr.y (mgr.14140) 885 : cluster [DBG] pgmap v1593: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim_wait, 157 active+clean, 7 active+clean+snaptrim; 892 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 14 op/s 2021-02-10T04:43:03.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:03 smithi085 conmon[29319]: cluster 2021-02-10T04:43:03.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:03 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:43:03.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:03 smithi085 conmon[29319]: 04:43:02.535128+0000 mon.a (mon. 2021-02-10T04:43:03.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:03 smithi085 conmon[29319]: 0) 1947 : cluster [DBG] osdmap e962: 8 total, 8 up, 7 in 2021-02-10T04:43:03.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:03 smithi085 conmon[29319]: cluster 2021-02-10T04:43:03.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:03 smithi085 conmon[29319]: 2021-02-10T04:43: 2021-02-10T04:43:03.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:03 smithi085 conmon[29319]: 03.325439+0000 mon.a (mon.0) 2021-02-10T04:43:03.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:03 smithi085 conmon[29319]: 1948 : cluster [DBG] osdmap e963: 8 total, 8 up, 7 in 2021-02-10T04:43:03.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:03 smithi085 conmon[35961]: cluster 2021-02-10T04:43:02.246535 2021-02-10T04:43:03.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:03 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 885 : cluster [DBG] pgmap v1593: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim_wait, 157 active+clean, 7 active+clean+snaptrim; 892 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 14 op/s 2021-02-10T04:43:03.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:03 smithi085 conmon[35961]: cluster 2021-02-10T04:43:02. 2021-02-10T04:43:03.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:03 smithi085 conmon[35961]: 535128+0000 mon.a (mon.0) 1947 : cluster [DBG] osdmap e962: 8 total, 8 up, 7 in 2021-02-10T04:43:03.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:03 smithi085 conmon[35961]: 2021-02-10T04:43:03.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:03 smithi085 conmon[35961]: cluster 2021-02-10T04:43 2021-02-10T04:43:03.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:03 smithi085 conmon[35961]: :03.325439+0000 mon.a (mon.0) 1948 : cluster [DBG] 2021-02-10T04:43:03.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:03 smithi085 conmon[35961]: osdmap e963: 8 total, 8 up, 7 in 2021-02-10T04:43:03.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:03 smithi164 conmon[29288]: cluster 2021-02-10T04:43:02.246535+0000 mgr.y (mgr.14140) 2021-02-10T04:43:03.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:03 smithi164 conmon[29288]: 885 : cluster [DBG] pgmap v1593: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim_wait, 157 active+clean, 7 active+clean+snaptrim; 892 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 14 op/s 2021-02-10T04:43:03.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:03 smithi164 conmon[29288]: cluster 2021-02-10T04:43:02.535128+0000 mon.a (mon.0) 1947 : cluster [DBG] osdmap e962: 8 total, 8 up, 7 in 2021-02-10T04:43:03.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:03 smithi164 conmon[29288]: cluster 2021-02-10T04:43:03.325439+0000 mon.a (mon 2021-02-10T04:43:03.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:03 smithi164 conmon[29288]: .0) 1948 : cluster [DBG] osdmap e963: 8 total, 8 up, 7 in 2021-02-10T04:43:04.333 INFO:tasks.rados.rados.0.smithi085.stdout:3890: done (8 left) 2021-02-10T04:43:04.333 INFO:tasks.rados.rados.0.smithi085.stdout:3891: rollback oid 18 current snap is 452 2021-02-10T04:43:04.333 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 18 to 403 2021-02-10T04:43:04.333 INFO:tasks.rados.rados.0.smithi085.stdout:3892: copy_from oid 9 from oid 4 current snap is 452 2021-02-10T04:43:04.334 INFO:tasks.rados.rados.0.smithi085.stdout:3893: delete oid 2 current snap is 452 2021-02-10T04:43:04.334 INFO:tasks.rados.rados.0.smithi085.stdout:3885: expect (ObjNum 1319 snap 451 seq_num 1319) 2021-02-10T04:43:04.485 INFO:tasks.rados.rados.0.smithi085.stdout:3886: expect (ObjNum 1269 snap 428 seq_num 1269) 2021-02-10T04:43:04.723 INFO:tasks.rados.rados.0.smithi085.stdout:3885: done (10 left) 2021-02-10T04:43:04.723 INFO:tasks.rados.rados.0.smithi085.stdout:3886: done (9 left) 2021-02-10T04:43:04.723 INFO:tasks.rados.rados.0.smithi085.stdout:3893: done (8 left) 2021-02-10T04:43:04.723 INFO:tasks.rados.rados.0.smithi085.stdout:3894: rollback oid 28 current snap is 452 2021-02-10T04:43:04.724 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 28 to 369 2021-02-10T04:43:04.724 INFO:tasks.rados.rados.0.smithi085.stdout:3888: finishing write tid 1 to smithi08575397-43 2021-02-10T04:43:04.724 INFO:tasks.rados.rados.0.smithi085.stdout:3888: finishing write tid 2 to smithi08575397-43 2021-02-10T04:43:04.724 INFO:tasks.rados.rados.0.smithi085.stdout:3888: finishing write tid 3 to smithi08575397-43 2021-02-10T04:43:04.724 INFO:tasks.rados.rados.0.smithi085.stdout:3888: finishing write tid 4 to smithi08575397-43 2021-02-10T04:43:04.724 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 1899 (ObjNum 1324 snap 452 seq_num 1324) dirty exists 2021-02-10T04:43:04.724 INFO:tasks.rados.rados.0.smithi085.stdout:3888: left oid 43 (ObjNum 1324 snap 452 seq_num 1324) 2021-02-10T04:43:04.724 INFO:tasks.rados.rados.0.smithi085.stdout:3889: finishing write tid 1 to smithi08575397-36 2021-02-10T04:43:04.724 INFO:tasks.rados.rados.0.smithi085.stdout:3889: finishing write tid 2 to smithi08575397-36 2021-02-10T04:43:04.724 INFO:tasks.rados.rados.0.smithi085.stdout:3889: finishing write tid 3 to smithi08575397-36 2021-02-10T04:43:04.724 INFO:tasks.rados.rados.0.smithi085.stdout:3889: finishing write tid 4 to smithi08575397-36 2021-02-10T04:43:04.724 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 36 v 1507 (ObjNum 1325 snap 452 seq_num 1325) dirty exists 2021-02-10T04:43:04.724 INFO:tasks.rados.rados.0.smithi085.stdout:3889: left oid 36 (ObjNum 1325 snap 452 seq_num 1325) 2021-02-10T04:43:04.724 INFO:tasks.rados.rados.0.smithi085.stdout:3891: finishing rollback tid 0 to smithi08575397-18 2021-02-10T04:43:04.725 INFO:tasks.rados.rados.0.smithi085.stdout:3891: finishing rollback tid 1 to smithi08575397-18 2021-02-10T04:43:04.725 INFO:tasks.rados.rados.0.smithi085.stdout:3888: done (8 left) 2021-02-10T04:43:04.725 INFO:tasks.rados.rados.0.smithi085.stdout:3889: done (7 left) 2021-02-10T04:43:04.725 INFO:tasks.rados.rados.0.smithi085.stdout:3895: rollback oid 48 current snap is 452 2021-02-10T04:43:04.725 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 48 to 396 2021-02-10T04:43:04.725 INFO:tasks.rados.rados.0.smithi085.stdout:3896: write (excl) oid 32 current snap is 452 2021-02-10T04:43:04.725 INFO:tasks.rados.rados.0.smithi085.stdout:3891: finishing rollback tid 2 to smithi08575397-18 2021-02-10T04:43:04.725 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 18 v 1628 (ObjNum 1161 snap 398 seq_num 1161) dirty exists 2021-02-10T04:43:04.725 INFO:tasks.rados.rados.0.smithi085.stdout:3892: finishing copy_from to smithi08575397-9 2021-02-10T04:43:04.725 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 9 v 1383 (ObjNum 8 snap 0 seq_num 8) dirty exists 2021-02-10T04:43:04.725 INFO:tasks.rados.rados.0.smithi085.stdout:3892: finishing copy_from racing read to smithi08575397-9 2021-02-10T04:43:04.725 INFO:tasks.rados.rados.0.smithi085.stdout:3896: seq_num 1327 ranges {702889=500845,1636311=576790} 2021-02-10T04:43:04.729 INFO:tasks.rados.rados.0.smithi085.stdout:3896: writing smithi08575397-32 from 702889 to 1203734 tid 1 2021-02-10T04:43:04.736 INFO:tasks.rados.rados.0.smithi085.stdout:3896: writing smithi08575397-32 from 1636311 to 2213101 tid 2 2021-02-10T04:43:04.736 INFO:tasks.rados.rados.0.smithi085.stdout:3891: done (8 left) 2021-02-10T04:43:04.736 INFO:tasks.rados.rados.0.smithi085.stdout:3892: done (7 left) 2021-02-10T04:43:04.736 INFO:tasks.rados.rados.0.smithi085.stdout:3897: snap_remove snap 386 2021-02-10T04:43:05.176 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:43:05.208 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:43:05.547 INFO:tasks.rados.rados.0.smithi085.stdout:3897: done (7 left) 2021-02-10T04:43:05.547 INFO:tasks.rados.rados.0.smithi085.stdout:3898: copy_from oid 20 from oid 42 current snap is 452 2021-02-10T04:43:05.547 INFO:tasks.rados.rados.0.smithi085.stdout:3899: read oid 21 snap -1 2021-02-10T04:43:05.547 INFO:tasks.rados.rados.0.smithi085.stdout:3899: expect (ObjNum 1269 snap 428 seq_num 1269) 2021-02-10T04:43:05.547 INFO:tasks.rados.rados.0.smithi085.stdout:3900: write oid 12 current snap is 452 2021-02-10T04:43:05.547 INFO:tasks.rados.rados.0.smithi085.stdout:3900: seq_num 1328 ranges {719696=576094,2094793=224903} 2021-02-10T04:43:05.550 INFO:tasks.rados.rados.0.smithi085.stdout:3900: writing smithi08575397-12 from 719696 to 1295790 tid 1 2021-02-10T04:43:05.557 INFO:tasks.rados.rados.0.smithi085.stdout:3900: writing smithi08575397-12 from 2094793 to 2319696 tid 2 2021-02-10T04:43:05.559 INFO:tasks.rados.rados.0.smithi085.stdout:3894: finishing rollback tid 1 to smithi08575397-28 2021-02-10T04:43:05.559 INFO:tasks.rados.rados.0.smithi085.stdout:3894: finishing rollback tid 2 to smithi08575397-28 2021-02-10T04:43:05.559 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 1466 (ObjNum 1052 snap 345 seq_num 1052) dirty exists 2021-02-10T04:43:05.559 INFO:tasks.rados.rados.0.smithi085.stdout:3895: finishing rollback tid 1 to smithi08575397-48 2021-02-10T04:43:05.560 INFO:tasks.rados.rados.0.smithi085.stdout:3895: finishing rollback tid 2 to smithi08575397-48 2021-02-10T04:43:05.560 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 1898 (ObjNum 1142 snap 385 seq_num 1142) dirty exists 2021-02-10T04:43:05.560 INFO:tasks.rados.rados.0.smithi085.stdout:3896: finishing write tid 1 to smithi08575397-32 2021-02-10T04:43:05.560 INFO:tasks.rados.rados.0.smithi085.stdout:3896: finishing write tid 2 to smithi08575397-32 2021-02-10T04:43:05.560 INFO:tasks.rados.rados.0.smithi085.stdout:3896: finishing write tid 3 to smithi08575397-32 2021-02-10T04:43:05.560 INFO:tasks.rados.rados.0.smithi085.stdout:3896: finishing write tid 4 to smithi08575397-32 2021-02-10T04:43:05.561 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 32 v 1509 (ObjNum 1327 snap 452 seq_num 1327) dirty exists 2021-02-10T04:43:05.561 INFO:tasks.rados.rados.0.smithi085.stdout:3896: left oid 32 (ObjNum 1327 snap 452 seq_num 1327) 2021-02-10T04:43:05.561 INFO:tasks.rados.rados.0.smithi085.stdout:3899: expect (ObjNum 1269 snap 428 seq_num 1269) 2021-02-10T04:43:05.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:05 smithi164 conmon[29288]: cluster 2021-02-10T04:43:04.247222+0000 mgr.y (mgr.14140 2021-02-10T04:43:05.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:05 smithi164 conmon[29288]: ) 886 : cluster [DBG] pgmap v1596: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 893 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 10 MiB/s rd, 224 KiB/s wr, 9 op/s 2021-02-10T04:43:05.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:05 smithi164 conmon[29288]: cluster 2021-02-10T04:43:04.333164+0000 mon.a (mon.0) 1949 : cluster [DBG] osdmap e964: 8 total, 8 up, 7 in 2021-02-10T04:43:05.692 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[45443]: debug 2021-02-10T04:43:05.505+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:43:05.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[29319]: cluster 2021-02-10T04:43:04.247222+0000 mgr.y (mgr.14140) 886 : cluster [DBG] pgmap v1596: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 893 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 10 MiB/s rd, 224 KiB/s wr, 9 op/s 2021-02-10T04:43:05.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:43:05.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[29319]: :43:04.333164+0000 mon.a (mon.0) 1949 : cluster [DBG] osdmap e964: 8 total, 8 up, 7 in 2021-02-10T04:43:05.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: cluster 2021-02-10T04:43:05.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: 2021-02-10T04:43:05.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: 2021 2021-02-10T04:43:05.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: - 2021-02-10T04:43:05.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: 02 2021-02-10T04:43:05.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: - 2021-02-10T04:43:05.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: 10 2021-02-10T04:43:05.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: T 2021-02-10T04:43:05.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: 04 2021-02-10T04:43:05.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: : 2021-02-10T04:43:05.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: 43 2021-02-10T04:43:05.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: : 2021-02-10T04:43:05.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: 04 2021-02-10T04:43:05.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: . 2021-02-10T04:43:05.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: 247222 2021-02-10T04:43:05.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: +0000 2021-02-10T04:43:05.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: 2021-02-10T04:43:05.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: mgr.y 2021-02-10T04:43:05.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: ( 2021-02-10T04:43:05.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: mgr 2021-02-10T04:43:05.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: . 2021-02-10T04:43:05.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: 14140 2021-02-10T04:43:05.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: ) 2021-02-10T04:43:05.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: 886 2021-02-10T04:43:05.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: : 2021-02-10T04:43:05.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: cluster 2021-02-10T04:43:05.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: 2021-02-10T04:43:05.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: [DBG] 2021-02-10T04:43:05.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: 2021-02-10T04:43:05.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: pgmap v1596: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 893 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 10 MiB/s rd, 224 KiB/s wr, 9 op/s 2021-02-10T04:43:05.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:43:05.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: -10T04:43:04.333164+0000 mon.a (mon.0) 1949 : cluster 2021-02-10T04:43:05.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:05 smithi085 conmon[35961]: [DBG] osdmap e964: 8 total, 8 up, 7 in 2021-02-10T04:43:05.705 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:43:05.813 INFO:tasks.rados.rados.0.smithi085.stdout:3900: finishing write tid 1 to smithi08575397-12 2021-02-10T04:43:05.813 INFO:tasks.rados.rados.0.smithi085.stdout:3900: finishing write tid 2 to smithi08575397-12 2021-02-10T04:43:05.813 INFO:tasks.rados.rados.0.smithi085.stdout:3900: finishing write tid 3 to smithi08575397-12 2021-02-10T04:43:05.813 INFO:tasks.rados.rados.0.smithi085.stdout:3900: finishing write tid 4 to smithi08575397-12 2021-02-10T04:43:05.813 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 1901 (ObjNum 1328 snap 452 seq_num 1328) dirty exists 2021-02-10T04:43:05.813 INFO:tasks.rados.rados.0.smithi085.stdout:3900: left oid 12 (ObjNum 1328 snap 452 seq_num 1328) 2021-02-10T04:43:05.813 INFO:tasks.rados.rados.0.smithi085.stdout:3894: done (9 left) 2021-02-10T04:43:05.813 INFO:tasks.rados.rados.0.smithi085.stdout:3895: done (8 left) 2021-02-10T04:43:05.813 INFO:tasks.rados.rados.0.smithi085.stdout:3896: done (7 left) 2021-02-10T04:43:05.813 INFO:tasks.rados.rados.0.smithi085.stdout:3899: done (6 left) 2021-02-10T04:43:05.814 INFO:tasks.rados.rados.0.smithi085.stdout:3900: done (5 left) 2021-02-10T04:43:05.814 INFO:tasks.rados.rados.0.smithi085.stdout:3901: write (excl) oid 50 current snap is 452 2021-02-10T04:43:05.814 INFO:tasks.rados.rados.0.smithi085.stdout:3898: finishing copy_from to smithi08575397-20 2021-02-10T04:43:05.814 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 20 v 1889 (ObjNum 1297 snap 436 seq_num 1297) dirty exists 2021-02-10T04:43:05.814 INFO:tasks.rados.rados.0.smithi085.stdout:3901: seq_num 1329 ranges {736503=651343,2153275=183228} 2021-02-10T04:43:05.821 INFO:tasks.rados.rados.0.smithi085.stdout:3901: writing smithi08575397-50 from 736503 to 1387846 tid 1 2021-02-10T04:43:05.827 INFO:tasks.rados.rados.0.smithi085.stdout:3901: writing smithi08575397-50 from 2153275 to 2336503 tid 2 2021-02-10T04:43:05.827 INFO:tasks.rados.rados.0.smithi085.stdout:3902: read oid 49 snap -1 2021-02-10T04:43:05.827 INFO:tasks.rados.rados.0.smithi085.stdout:3902: expect deleted 2021-02-10T04:43:05.827 INFO:tasks.rados.rados.0.smithi085.stdout:3903: read oid 6 snap -1 2021-02-10T04:43:05.827 INFO:tasks.rados.rados.0.smithi085.stdout:3903: expect (ObjNum 1310 snap 449 seq_num 1310) 2021-02-10T04:43:05.827 INFO:tasks.rados.rados.0.smithi085.stdout:3904: write oid 30 current snap is 452 2021-02-10T04:43:05.828 INFO:tasks.rados.rados.0.smithi085.stdout:3904: seq_num 1330 ranges {753310=726592,2095404=257906} 2021-02-10T04:43:05.829 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:43:05.837 INFO:tasks.rados.rados.0.smithi085.stdout:3904: writing smithi08575397-30 from 753310 to 1479902 tid 1 2021-02-10T04:43:05.842 INFO:tasks.rados.rados.0.smithi085.stdout:3904: writing smithi08575397-30 from 2095404 to 2353310 tid 2 2021-02-10T04:43:05.843 INFO:tasks.rados.rados.0.smithi085.stdout:3905: copy_from oid 28 from oid 3 current snap is 452 2021-02-10T04:43:05.843 INFO:tasks.rados.rados.0.smithi085.stdout:3898: finishing copy_from racing read to smithi08575397-20 2021-02-10T04:43:05.843 INFO:tasks.rados.rados.0.smithi085.stdout:3901: finishing write tid 1 to smithi08575397-50 2021-02-10T04:43:05.843 INFO:tasks.rados.rados.0.smithi085.stdout:3903: expect (ObjNum 1310 snap 449 seq_num 1310) 2021-02-10T04:43:05.869 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:43:05.976 INFO:tasks.rados.rados.0.smithi085.stdout:3901: finishing write tid 2 to smithi08575397-50 2021-02-10T04:43:05.976 INFO:tasks.rados.rados.0.smithi085.stdout:3901: finishing write tid 3 to smithi08575397-50 2021-02-10T04:43:05.976 INFO:tasks.rados.rados.0.smithi085.stdout:3901: finishing write tid 4 to smithi08575397-50 2021-02-10T04:43:05.976 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 50 v 1890 (ObjNum 1329 snap 452 seq_num 1329) dirty exists 2021-02-10T04:43:05.976 INFO:tasks.rados.rados.0.smithi085.stdout:3901: left oid 50 (ObjNum 1329 snap 452 seq_num 1329) 2021-02-10T04:43:05.977 INFO:tasks.rados.rados.0.smithi085.stdout:3904: finishing write tid 1 to smithi08575397-30 2021-02-10T04:43:05.977 INFO:tasks.rados.rados.0.smithi085.stdout:3904: finishing write tid 2 to smithi08575397-30 2021-02-10T04:43:05.977 INFO:tasks.rados.rados.0.smithi085.stdout:3904: finishing write tid 3 to smithi08575397-30 2021-02-10T04:43:05.977 INFO:tasks.rados.rados.0.smithi085.stdout:3904: finishing write tid 4 to smithi08575397-30 2021-02-10T04:43:05.977 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 1893 (ObjNum 1330 snap 452 seq_num 1330) dirty exists 2021-02-10T04:43:05.977 INFO:tasks.rados.rados.0.smithi085.stdout:3904: left oid 30 (ObjNum 1330 snap 452 seq_num 1330) 2021-02-10T04:43:05.978 INFO:tasks.rados.rados.0.smithi085.stdout:3898: done (9 left) 2021-02-10T04:43:05.978 INFO:tasks.rados.rados.0.smithi085.stdout:3901: done (8 left) 2021-02-10T04:43:05.978 INFO:tasks.rados.rados.0.smithi085.stdout:3902: done (7 left) 2021-02-10T04:43:05.978 INFO:tasks.rados.rados.0.smithi085.stdout:3903: done (6 left) 2021-02-10T04:43:05.978 INFO:tasks.rados.rados.0.smithi085.stdout:3904: done (5 left) 2021-02-10T04:43:05.979 INFO:tasks.rados.rados.0.smithi085.stdout:3906: snap_remove snap 443 2021-02-10T04:43:06.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[49106]: debug 2021-02-10T04:43:06.197+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:43:06.535 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:43:06.549 INFO:tasks.rados.rados.0.smithi085.stdout:3906: done (5 left) 2021-02-10T04:43:06.549 INFO:tasks.rados.rados.0.smithi085.stdout:3907: snap_remove snap 347 2021-02-10T04:43:06.645 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:43:06.670 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:43:06.894 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:43:06 smithi164 conmon[37263]: debug 2021-02-10T04:43:06.730+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:43:06.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:06 smithi164 conmon[29288]: cluster 2021-02-10T04:43:05.540892+0000 mon.a ( 2021-02-10T04:43:06.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:06 smithi164 conmon[29288]: mon.0) 1950 : cluster [DBG] osdmap e965: 8 total, 8 up, 7 in 2021-02-10T04:43:06.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: cluster 2021-02-10T04:43:06.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: 2021-02-10T04:43:06.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: 2021 2021-02-10T04:43:06.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: - 2021-02-10T04:43:06.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: 02 2021-02-10T04:43:06.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: - 2021-02-10T04:43:06.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: 10 2021-02-10T04:43:06.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: T 2021-02-10T04:43:06.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: 04 2021-02-10T04:43:06.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: : 2021-02-10T04:43:06.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: 43 2021-02-10T04:43:06.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: : 2021-02-10T04:43:06.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: 05 2021-02-10T04:43:06.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: . 2021-02-10T04:43:06.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: 540892 2021-02-10T04:43:06.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: +0000 2021-02-10T04:43:06.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: 2021-02-10T04:43:06.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: mon.a 2021-02-10T04:43:06.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: ( 2021-02-10T04:43:06.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: mon 2021-02-10T04:43:06.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: . 2021-02-10T04:43:06.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: 0 2021-02-10T04:43:06.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: ) 2021-02-10T04:43:06.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: 1950 2021-02-10T04:43:06.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: : 2021-02-10T04:43:06.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: cluster 2021-02-10T04:43:06.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: 2021-02-10T04:43:06.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: [DBG] 2021-02-10T04:43:06.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: 2021-02-10T04:43:06.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: osdmap e965: 8 total, 8 up, 7 in 2021-02-10T04:43:06.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[29319]: 2021-02-10T04:43:06.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[35961]: cluster 2021-02-10T04:43:05.540892+0000 mon.a (mon.0) 2021-02-10T04:43:06.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:06 smithi085 conmon[35961]: 1950 : cluster [DBG] osdmap e965: 8 total, 8 up, 7 in 2021-02-10T04:43:07.016 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:43:07.017 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:43:07.017 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:43:07.033 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:43:07.052 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:43:07.091 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:43:07.129 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:43:07.138 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:43:07.164 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:43:07.172 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:43:07.218 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:43:07.267 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:43:07.306 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:43:07.309 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:43:07.309 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:43:07.309 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:43:07.309 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:43:07.309 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:43:07.309 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:43:07.309 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:43:07.309 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:43:07.318 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:43:07.321 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:43:07.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:43:07 smithi164 conmon[44544]: debug 2021-02-10T04:43:07.231+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:43:07.456 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:43:07.556 INFO:tasks.rados.rados.0.smithi085.stdout:3907: done (5 left) 2021-02-10T04:43:07.556 INFO:tasks.rados.rados.0.smithi085.stdout:3908: delete oid 44 current snap is 452 2021-02-10T04:43:07.556 INFO:tasks.rados.rados.0.smithi085.stdout:3905: finishing copy_from to smithi08575397-28 2021-02-10T04:43:07.556 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 1896 (ObjNum 1319 snap 451 seq_num 1319) dirty exists 2021-02-10T04:43:07.557 INFO:tasks.rados.rados.0.smithi085.stdout:3905: finishing copy_from racing read to smithi08575397-28 2021-02-10T04:43:07.561 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:43:07.562 INFO:tasks.rados.rados.0.smithi085.stdout:3905: done (5 left) 2021-02-10T04:43:07.562 INFO:tasks.rados.rados.0.smithi085.stdout:3908: done (4 left) 2021-02-10T04:43:07.562 INFO:tasks.rados.rados.0.smithi085.stdout:3909: read oid 15 snap -1 2021-02-10T04:43:07.562 INFO:tasks.rados.rados.0.smithi085.stdout:3909: expect (ObjNum 1323 snap 452 seq_num 1323) 2021-02-10T04:43:07.563 INFO:tasks.rados.rados.0.smithi085.stdout:3910: snap_remove snap 426 2021-02-10T04:43:07.563 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is set 2021-02-10T04:43:07.594 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:43:07.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:07 smithi085 conmon[29319]: cluster 2021-02-10T04:43:06.247734+0000 mgr.y (mgr. 2021-02-10T04:43:07.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:07 smithi085 conmon[29319]: 14140) 887 : cluster [DBG] pgmap v1599: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 907 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 27 MiB/s rd, 2.0 MiB/s wr, 27 op/s 2021-02-10T04:43:07.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:07 smithi085 conmon[29319]: cluster 2021-02-10T04:43:06.545459+0000 mon.a (mon.0) 1951 : cluster [DBG] osdmap e966: 8 total, 8 up, 7 in 2021-02-10T04:43:07.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:07 smithi085 conmon[29319]: cluster 2021-02-10T04:43:06.774339+0000 osd.5 (osd.5) 86 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:43:07.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:07 smithi085 conmon[29319]: cluster 2021-02-10T04:43:06.775221+0000 osd.5 (osd.5) 87 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:43:07.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:07 smithi085 conmon[29319]: audit 2021-02-10T04:43:06.981142+0000 mon.b (mon.2) 42 : audit [DBG] from='client.? 172.21.15.85:0/2914709042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:43:07.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:07 smithi085 conmon[29319]: audit 2021-02-10T04:43:07.330488+0000 mon.b (mon.2) 43 : audit [INF] from='client.? 172.21.15.85:0/998379535' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:43:07.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:07 smithi085 conmon[29319]: audit 2021-02-10T04:43:07.330944+0000 mon.a (mon.0) 1952 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:43:07.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:07 smithi085 conmon[35961]: cluster 2021-02-10T04:43:06.247734+0000 mgr.y 2021-02-10T04:43:07.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:07 smithi085 conmon[35961]: (mgr.14140) 887 : cluster [DBG] pgmap v1599: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 907 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 27 MiB/s rd, 2.0 MiB/s wr, 27 op/s 2021-02-10T04:43:07.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:07 smithi085 conmon[35961]: cluster 2021-02-10T04:43:06.545459+0000 mon.a (mon.0) 1951 : cluster [DBG] osdmap e966: 8 total, 8 up, 7 in 2021-02-10T04:43:07.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:07 smithi085 conmon[35961]: cluster 2021-02-10T04:43:06.774339+0000 osd.5 (osd.5) 86 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:43:07.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:07 smithi085 conmon[35961]: cluster 2021-02-10T04:43:06.775221+0000 osd.5 (osd.5) 87 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:43:07.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:07 smithi085 conmon[35961]: audit 2021-02-10T04:43:06.981142+0000 mon.b (mon.2) 42 : audit [DBG] from='client.? 172.21.15.85:0/2914709042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:43:07.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:07 smithi085 conmon[35961]: audit 2021-02-10T04:43:07.330488+0000 mon.b (mon.2) 43 : audit [INF] from='client.? 172.21.15.85:0/998379535' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:43:07.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:07 smithi085 conmon[35961]: audit 2021-02-10T04:43:07.330944+0000 mon.a (mon.0) 1952 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:43:07.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:07 smithi164 conmon[33677]: debug 2021-02-10T04:43:07.597+0000 7fbec19a1700 -1 osd.4 967 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:07.894 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:43:07 smithi164 conmon[40892]: debug 2021-02-10T04:43:07.678+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:43:07.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:07 smithi164 conmon[29288]: cluster 2021-02-10T04:43:06.247734+0000 2021-02-10T04:43:07.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:07 smithi164 conmon[29288]: mgr.y (mgr.14140) 887 : cluster [DBG] pgmap v1599: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 907 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 27 MiB/s rd, 2.0 MiB/s wr, 27 op/s 2021-02-10T04:43:07.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:07 smithi164 conmon[29288]: cluster 2021-02-10T04:43 2021-02-10T04:43:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:07 smithi164 conmon[29288]: :06.545459+0000 mon.a (mon.0) 1951 : cluster [DBG] osdmap e966: 8 total, 8 up, 7 in 2021-02-10T04:43:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:07 smithi164 conmon[29288]: cluster 2021-02-10T04:43:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:07 smithi164 conmon[29288]: 2021-02-10 2021-02-10T04:43:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:07 smithi164 conmon[29288]: T04:43:06.774339+0000 osd.5 (osd. 2021-02-10T04:43:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:07 smithi164 conmon[29288]: 5) 86 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:43:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:07 smithi164 conmon[29288]: cluster 2021-02-10T04:43:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:07 smithi164 conmon[29288]: 2021-02-10T04:43:06.775221+0000 2021-02-10T04:43:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:07 smithi164 conmon[29288]: osd.5 (osd.5) 87 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:43:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:07 smithi164 conmon[29288]: 2021-02-10T04:43:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:07 smithi164 conmon[29288]: audit 2021-02-10T04: 2021-02-10T04:43:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:07 smithi164 conmon[29288]: 43:06.981142+0000 mon.b (mon.2) 2021-02-10T04:43:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:07 smithi164 conmon[29288]: 42 : audit [DBG] from='client.? 172.21.15.85:0/2914709042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:43:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:07 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:43:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:07 smithi164 conmon[29288]: -10T04:43:07.330488+0000 mon.b (mon 2021-02-10T04:43:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:07 smithi164 conmon[29288]: .2) 43 : audit [INF] from='client.? 172.21.15.85:0/998379535' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:43:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:07 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:43:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:07 smithi164 conmon[29288]: 02-10T04:43:07.330944+0000 mon.a ( 2021-02-10T04:43:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:07 smithi164 conmon[29288]: mon.0) 1952 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:43:08.030 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:43:08.137 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:43:08.173 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:43:14.151 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:43:08 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:43:14.151 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:08 smithi164 conmon[29288]: audit 2021-02-10T04:43:07.291438+0000 mgr.y 2021-02-10T04:43:14.151 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:08 smithi164 conmon[29288]: (mgr.14140) 888 : audit [DBG] from='client.35162 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:43:14.151 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:08 smithi164 conmon[29288]: cluster 2021-02-10T04:43:07.543027+0000 mon.a (mon.0) 1953 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:43:14.151 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:08 smithi164 conmon[29288]: audit 2021-02-10T04:43:07.555761+0000 mon.a (mon 2021-02-10T04:43:14.151 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:08 smithi164 conmon[29288]: .0) 1954 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:43:14.151 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:08 smithi164 conmon[29288]: cluster 2021-02-10T04:43 2021-02-10T04:43:14.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:08 smithi164 conmon[29288]: :07.555832+0000 mon.a (mon.0) 1955 : cluster 2021-02-10T04:43:14.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:08 smithi164 conmon[29288]: [DBG] osdmap e967: 8 total, 8 up, 7 in 2021-02-10T04:43:14.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:08 smithi164 conmon[29288]: cluster 2021-02-10T04:43:14.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:08 smithi164 conmon[29288]: 2021-02-10T04:43:07.598605+0000 osd.4 (osd.4) 1621 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:14.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:08 smithi164 conmon[29288]: cluster 2021-02-10T04:43:07.598612+0000 osd.4 (osd. 2021-02-10T04:43:14.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:08 smithi164 conmon[29288]: 4) 1622 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:14.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:08 smithi164 conmon[29288]: cluster 2021-02-10T04:43:07.598616+0000 2021-02-10T04:43:14.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:08 smithi164 conmon[29288]: osd.4 (osd.4) 1623 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:14.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:08 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:43:14.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:08 smithi164 conmon[29288]: 02-10T04:43:07.598620+0000 osd.4 (osd 2021-02-10T04:43:14.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:08 smithi164 conmon[29288]: .4) 1624 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:14.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:09 smithi164 conmon[29288]: cluster 2021-02-10T04:43:08.248439+0000 mgr.y (mgr.14140) 889 : cluster 2021-02-10T04:43:14.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:09 smithi164 conmon[29288]: [DBG] pgmap v1602: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 910 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 19 MiB/s rd, 2.8 MiB/s wr, 23 op/s 2021-02-10T04:43:14.153 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:09 smithi164 conmon[29288]: cluster 2021-02-10T04:43:08.559790+0000 mon.a (mon.0) 1956 : cluster [DBG] osdmap e968: 8 total, 8 up, 7 in 2021-02-10T04:43:14.153 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:09 smithi164 conmon[29288]: audit 2021-02-10T04:43:08.560698+0000 mgr.y (mgr.14140) 890 : 2021-02-10T04:43:14.153 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:09 smithi164 conmon[29288]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:43:14.153 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:09 smithi164 conmon[29288]: cluster 2021-02-10T04:43:08.595413+0000 osd.4 (osd.4) 1625 : 2021-02-10T04:43:14.153 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:09 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:14.153 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:09 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:43:14.153 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:09 smithi164 conmon[29288]: 04:43:08.595424+0000 osd.4 (osd.4) 1626 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:14.153 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:09 smithi164 conmon[29288]: cluster 2021-02-10T04:43: 2021-02-10T04:43:14.153 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:09 smithi164 conmon[29288]: 08.595431+0000 osd.4 (osd.4) 1627 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:14.153 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:09 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:43:14.153 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:09 smithi164 conmon[29288]: 43:08.595439+0000 osd.4 (osd.4) 1628 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:14.153 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:09 smithi164 conmon[29288]: 2021-02-10T04:43:14.153 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: cluster 2021-02-10T04:43:09.569245+0000 mon.a (mon. 2021-02-10T04:43:14.154 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: 0) 1957 : cluster [DBG] osdmap e969: 8 total, 8 up, 7 in 2021-02-10T04:43:14.154 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: cluster 2021-02-10T04:43:09.619182+0000 osd.4 (osd.4) 1629 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:14.154 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: cluster 2021-02-10T04:43:14.154 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: 2021-02-10T04:43:14.154 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: 2021 2021-02-10T04:43:14.154 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: - 2021-02-10T04:43:14.154 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: 02 2021-02-10T04:43:14.154 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: - 2021-02-10T04:43:14.154 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: 10 2021-02-10T04:43:14.154 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: T 2021-02-10T04:43:14.154 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: 04 2021-02-10T04:43:14.154 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: : 2021-02-10T04:43:14.154 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: 43 2021-02-10T04:43:14.155 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: : 2021-02-10T04:43:14.155 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: 09 2021-02-10T04:43:14.155 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: . 2021-02-10T04:43:14.155 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: 619188 2021-02-10T04:43:14.155 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: +0000 2021-02-10T04:43:14.155 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: 2021-02-10T04:43:14.155 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: osd.4 2021-02-10T04:43:14.155 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: ( 2021-02-10T04:43:14.155 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: osd 2021-02-10T04:43:14.155 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: . 2021-02-10T04:43:14.155 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: 4 2021-02-10T04:43:14.155 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: ) 2021-02-10T04:43:14.155 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: 1630 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:14.156 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: cluster 2021-02-10T04:43: 2021-02-10T04:43:14.156 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: 09.619192+0000 osd.4 (osd.4) 1631 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:14.156 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: cluster 2021-02-10T04:43:14.156 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: 2021 2021-02-10T04:43:14.156 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: - 2021-02-10T04:43:14.156 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: 02 2021-02-10T04:43:14.156 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: - 2021-02-10T04:43:14.156 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: 10 2021-02-10T04:43:14.156 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: T 2021-02-10T04:43:14.156 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: 04 2021-02-10T04:43:14.156 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: : 2021-02-10T04:43:14.156 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: 43 2021-02-10T04:43:14.156 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: : 2021-02-10T04:43:14.157 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: 09 2021-02-10T04:43:14.157 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: . 2021-02-10T04:43:14.157 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: 619195 2021-02-10T04:43:14.157 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: +0000 2021-02-10T04:43:14.157 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: 2021-02-10T04:43:14.157 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: osd.4 2021-02-10T04:43:14.157 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: ( 2021-02-10T04:43:14.157 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: osd 2021-02-10T04:43:14.157 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: . 2021-02-10T04:43:14.157 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: 4 2021-02-10T04:43:14.157 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: ) 2021-02-10T04:43:14.157 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: 1632 2021-02-10T04:43:14.157 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: : 2021-02-10T04:43:14.158 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: cluster 2021-02-10T04:43:14.158 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: 2021-02-10T04:43:14.158 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: [WRN] 2021-02-10T04:43:14.158 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: 2021-02-10T04:43:14.158 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:14.158 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[29288]: 2021-02-10T04:43:14.158 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:11 smithi164 conmon[29288]: cluster 2021-02-10T04:43:10.249166+0000 mgr.y (mgr.14140) 891 2021-02-10T04:43:14.158 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:11 smithi164 conmon[29288]: : cluster [DBG] pgmap v1605: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 910 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 8.7 MiB/s rd, 3.9 MiB/s wr, 19 op/s 2021-02-10T04:43:14.158 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:11 smithi164 conmon[29288]: cluster 2021-02-10T04:43:10.572945+0000 mon.a (mon.0) 1958 : cluster [DBG] osdmap e970: 8 total, 8 up, 7 in 2021-02-10T04:43:14.158 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:11 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:43:14.158 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:11 smithi164 conmon[29288]: 04:43:10.613729+0000 osd.4 (osd.4) 1633 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:14.158 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:11 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:43:14.159 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:11 smithi164 conmon[29288]: 10T04:43:10.613739+0000 osd.4 (osd.4) 1634 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:14.159 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:11 smithi164 conmon[29288]: cluster 2021-02-10T04:43:10.613745+0000 osd.4 (osd.4) 1635 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:14.159 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:11 smithi164 conmon[29288]: cluster 2021-02-10T04:43:10.613751+0000 osd.4 ( 2021-02-10T04:43:14.159 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:11 smithi164 conmon[29288]: osd.4) 1636 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:14.159 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:12 smithi164 conmon[29288]: cluster 2021-02-10T04:43:11.581069+0000 mon.a (mon.0) 2021-02-10T04:43:14.159 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:12 smithi164 conmon[29288]: 1959 : cluster [DBG] osdmap e971: 8 total, 8 up, 7 in 2021-02-10T04:43:14.159 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:12 smithi164 conmon[29288]: cluster 2021-02-10T04:43:11.642046+0000 osd.4 (osd.4) 1637 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:14.159 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:12 smithi164 conmon[29288]: cluster 2021-02-10T04:43:11.642053+0000 osd.4 ( 2021-02-10T04:43:14.159 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:12 smithi164 conmon[29288]: osd.4) 1638 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:14.159 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:12 smithi164 conmon[29288]: cluster 2021-02-10T04:43:11.642060+0000 osd.4 (osd 2021-02-10T04:43:14.159 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:12 smithi164 conmon[29288]: .4) 1639 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:14.159 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:12 smithi164 conmon[29288]: cluster 2021-02-10T04:43:11.642067+0000 osd.4 (osd.4 2021-02-10T04:43:14.159 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:12 smithi164 conmon[29288]: ) 1640 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:14.160 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:12 smithi164 conmon[29288]: cluster 2021-02-10T04:43:12.176433+0000 osd.2 (osd. 2021-02-10T04:43:14.160 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:12 smithi164 conmon[29288]: 2) 1943 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:43:14.160 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:12 smithi164 conmon[29288]: cluster 2021-02-10T04:43:12. 2021-02-10T04:43:14.160 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:12 smithi164 conmon[29288]: 177207+0000 osd.2 (osd.2) 1944 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:43:14.160 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:13 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:43:14.160 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:13 smithi164 conmon[29288]: 04:43:12.249811+0000 mgr.y (mgr.14140) 892 : cluster [DBG] pgmap v1608: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 910 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 6.2 MiB/s rd, 2.9 MiB/s wr, 13 op/s 2021-02-10T04:43:14.160 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:13 smithi164 conmon[29288]: cluster 2021-02-10T04:43:12 2021-02-10T04:43:14.160 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:13 smithi164 conmon[29288]: .578912+0000 mon.a (mon.0) 1960 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 822 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:43:14.160 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:13 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:43:14.160 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:13 smithi164 conmon[29288]: 02-10T04:43:12.583496+0000 mon.a (mon 2021-02-10T04:43:14.160 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:13 smithi164 conmon[29288]: .0) 1961 : cluster [DBG] osdmap e972: 8 total, 8 up, 7 in 2021-02-10T04:43:14.160 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:13 smithi164 conmon[29288]: cluster 2021-02-10T04:43: 2021-02-10T04:43:14.160 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:13 smithi164 conmon[29288]: 12.687493+0000 osd.4 (osd.4) 1641 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:14.161 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:13 smithi164 conmon[29288]: cluster 2021-02-10T04:43:14.161 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:13 smithi164 conmon[29288]: 2021-02-10T04:43:12.687505+0000 osd.4 (osd.4) 1642 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:14.161 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:13 smithi164 conmon[29288]: cluster 2021-02-10T04:43:14.161 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:13 smithi164 conmon[29288]: 2021-02-10T04:43:12.687512+0000 2021-02-10T04:43:14.161 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:13 smithi164 conmon[29288]: osd.4 (osd.4) 1643 : cluster 2021-02-10T04:43:14.161 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:13 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:14.161 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:13 smithi164 conmon[29288]: cluster 2021-02-10T04:43:14.161 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:13 smithi164 conmon[29288]: 2021-02-10T04:43:12.687518+0000 osd.4 (osd 2021-02-10T04:43:14.161 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:13 smithi164 conmon[29288]: .4) 1644 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:14.161 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:08 smithi164 conmon[33677]: debug 2021-02-10T04:43:14.161 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:08 smithi164 conmon[33677]: 2021-02-10T04:43:08.594+0000 7fbec19a1700 -1 osd.4 968 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:14.161 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:09 smithi164 conmon[33677]: debug 2021-02-10T04:43:09.618+0000 7fbec19a1700 -1 osd.4 968 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:14.161 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:10 smithi164 conmon[33677]: debug 2021-02-10T04:43:10.613+0000 7fbec19a1700 -1 osd.4 970 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:14.162 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:11 smithi164 conmon[33677]: debug 2021-02-10T04:43:11.641+0000 7fbec19a1700 -1 osd.4 971 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:14.162 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:12 smithi164 conmon[33677]: debug 2021-02-10T04:43:12.686+0000 7fbec19a1700 -1 osd.4 972 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:14.162 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:13 smithi164 conmon[33677]: debug 2021-02-10T04:43:13.669+0000 7fbec19a1700 -1 osd.4 973 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:14.162 INFO:tasks.rados.rados.0.smithi085.stdout:3910: done (5 left) 2021-02-10T04:43:14.162 INFO:tasks.rados.rados.0.smithi085.stdout:3911: write (excl) oid 30 current snap is 452 2021-02-10T04:43:14.162 INFO:tasks.rados.rados.0.smithi085.stdout:3911: seq_num 1332 ranges {786924=593443,1928721=458203} 2021-02-10T04:43:14.162 INFO:tasks.rados.rados.0.smithi085.stdout:3911: writing smithi08575397-30 from 786924 to 1380367 tid 1 2021-02-10T04:43:14.162 INFO:tasks.rados.rados.0.smithi085.stdout:3911: writing smithi08575397-30 from 1928721 to 2386924 tid 2 2021-02-10T04:43:14.162 INFO:tasks.rados.rados.0.smithi085.stdout:3912: copy_from oid 40 from oid 28 current snap is 452 2021-02-10T04:43:14.162 INFO:tasks.rados.rados.0.smithi085.stdout:3913: read oid 11 snap -1 2021-02-10T04:43:14.162 INFO:tasks.rados.rados.0.smithi085.stdout:3913: expect (ObjNum 1291 snap 435 seq_num 1291) 2021-02-10T04:43:14.162 INFO:tasks.rados.rados.0.smithi085.stdout:3914: snap_create 2021-02-10T04:43:14.163 INFO:tasks.rados.rados.0.smithi085.stdout:3909: expect (ObjNum 1323 snap 452 seq_num 1323) 2021-02-10T04:43:14.163 INFO:tasks.rados.rados.0.smithi085.stdout:3911: finishing write tid 1 to smithi08575397-30 2021-02-10T04:43:14.163 INFO:tasks.rados.rados.0.smithi085.stdout:3911: finishing write tid 2 to smithi08575397-30 2021-02-10T04:43:14.163 INFO:tasks.rados.rados.0.smithi085.stdout:3911: finishing write tid 3 to smithi08575397-30 2021-02-10T04:43:14.163 INFO:tasks.rados.rados.0.smithi085.stdout:3911: finishing write tid 4 to smithi08575397-30 2021-02-10T04:43:14.163 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 1896 (ObjNum 1332 snap 452 seq_num 1332) dirty exists 2021-02-10T04:43:14.163 INFO:tasks.rados.rados.0.smithi085.stdout:3911: left oid 30 (ObjNum 1332 snap 452 seq_num 1332) 2021-02-10T04:43:14.163 INFO:tasks.rados.rados.0.smithi085.stdout:3913: expect (ObjNum 1291 snap 435 seq_num 1291) 2021-02-10T04:43:14.163 INFO:tasks.rados.rados.0.smithi085.stdout:3912: finishing copy_from to smithi08575397-40 2021-02-10T04:43:14.163 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 1896 (ObjNum 1319 snap 451 seq_num 1319) dirty exists 2021-02-10T04:43:14.163 INFO:tasks.rados.rados.0.smithi085.stdout:3912: finishing copy_from racing read to smithi08575397-40 2021-02-10T04:43:14.163 INFO:tasks.rados.rados.0.smithi085.stdout:3909: done (8 left) 2021-02-10T04:43:14.163 INFO:tasks.rados.rados.0.smithi085.stdout:3911: done (7 left) 2021-02-10T04:43:14.164 INFO:tasks.rados.rados.0.smithi085.stdout:3912: done (6 left) 2021-02-10T04:43:14.164 INFO:tasks.rados.rados.0.smithi085.stdout:3913: done (5 left) 2021-02-10T04:43:14.164 INFO:tasks.rados.rados.0.smithi085.stdout:3914: done (4 left) 2021-02-10T04:43:14.164 INFO:tasks.rados.rados.0.smithi085.stdout:3915: write oid 49 current snap is 453 2021-02-10T04:43:14.164 INFO:tasks.rados.rados.0.smithi085.stdout:3915: seq_num 1333 ranges {403731=668692,1587203=587797} 2021-02-10T04:43:14.164 INFO:tasks.rados.rados.0.smithi085.stdout:3915: writing smithi08575397-49 from 403731 to 1072423 tid 1 2021-02-10T04:43:14.164 INFO:tasks.rados.rados.0.smithi085.stdout:3915: writing smithi08575397-49 from 1587203 to 2175000 tid 2 2021-02-10T04:43:14.164 INFO:tasks.rados.rados.0.smithi085.stdout:3916: rollback oid 34 current snap is 453 2021-02-10T04:43:14.164 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 34 to 432 2021-02-10T04:43:14.164 INFO:tasks.rados.rados.0.smithi085.stdout:3915: finishing write tid 1 to smithi08575397-49 2021-02-10T04:43:14.164 INFO:tasks.rados.rados.0.smithi085.stdout:3917: read oid 44 snap -1 2021-02-10T04:43:14.164 INFO:tasks.rados.rados.0.smithi085.stdout:3917: expect deleted 2021-02-10T04:43:14.164 INFO:tasks.rados.rados.0.smithi085.stdout:3918: snap_remove snap 434 2021-02-10T04:43:14.165 INFO:tasks.rados.rados.0.smithi085.stdout:3918: done (7 left) 2021-02-10T04:43:14.165 INFO:tasks.rados.rados.0.smithi085.stdout:3919: write (excl) oid 3 current snap is 453 2021-02-10T04:43:14.165 INFO:tasks.rados.rados.0.smithi085.stdout:3919: seq_num 1334 ranges {420538=743941,1929332=491206} 2021-02-10T04:43:14.165 INFO:tasks.rados.rados.0.smithi085.stdout:3919: writing smithi08575397-3 from 420538 to 1164479 tid 1 2021-02-10T04:43:14.165 INFO:tasks.rados.rados.0.smithi085.stdout:3919: writing smithi08575397-3 from 1929332 to 2420538 tid 2 2021-02-10T04:43:14.165 INFO:tasks.rados.rados.0.smithi085.stdout:3920: snap_remove snap 381 2021-02-10T04:43:14.165 INFO:tasks.rados.rados.0.smithi085.stdout:3915: finishing write tid 2 to smithi08575397-49 2021-02-10T04:43:14.165 INFO:tasks.rados.rados.0.smithi085.stdout:3915: finishing write tid 3 to smithi08575397-49 2021-02-10T04:43:14.165 INFO:tasks.rados.rados.0.smithi085.stdout:3915: finishing write tid 4 to smithi08575397-49 2021-02-10T04:43:14.165 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 49 v 1626 (ObjNum 1333 snap 453 seq_num 1333) dirty exists 2021-02-10T04:43:14.165 INFO:tasks.rados.rados.0.smithi085.stdout:3915: left oid 49 (ObjNum 1333 snap 453 seq_num 1333) 2021-02-10T04:43:14.165 INFO:tasks.rados.rados.0.smithi085.stdout:3916: finishing rollback tid 0 to smithi08575397-34 2021-02-10T04:43:14.165 INFO:tasks.rados.rados.0.smithi085.stdout:3916: finishing rollback tid 1 to smithi08575397-34 2021-02-10T04:43:14.166 INFO:tasks.rados.rados.0.smithi085.stdout:3916: finishing rollback tid 2 to smithi08575397-34 2021-02-10T04:43:14.166 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 34 v 1908 (ObjNum 1145 snap 385 seq_num 1145) dirty exists 2021-02-10T04:43:14.166 INFO:tasks.rados.rados.0.smithi085.stdout:3919: finishing write tid 1 to smithi08575397-3 2021-02-10T04:43:14.166 INFO:tasks.rados.rados.0.smithi085.stdout:3919: finishing write tid 2 to smithi08575397-3 2021-02-10T04:43:14.166 INFO:tasks.rados.rados.0.smithi085.stdout:3919: finishing write tid 3 to smithi08575397-3 2021-02-10T04:43:14.166 INFO:tasks.rados.rados.0.smithi085.stdout:3919: finishing write tid 4 to smithi08575397-3 2021-02-10T04:43:14.166 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 1902 (ObjNum 1334 snap 453 seq_num 1334) dirty exists 2021-02-10T04:43:14.166 INFO:tasks.rados.rados.0.smithi085.stdout:3919: left oid 3 (ObjNum 1334 snap 453 seq_num 1334) 2021-02-10T04:43:14.166 INFO:tasks.rados.rados.0.smithi085.stdout:3915: done (8 left) 2021-02-10T04:43:14.166 INFO:tasks.rados.rados.0.smithi085.stdout:3916: done (7 left) 2021-02-10T04:43:14.166 INFO:tasks.rados.rados.0.smithi085.stdout:3917: done (6 left) 2021-02-10T04:43:14.166 INFO:tasks.rados.rados.0.smithi085.stdout:3919: done (5 left) 2021-02-10T04:43:14.166 INFO:tasks.rados.rados.0.smithi085.stdout:3920: done (4 left) 2021-02-10T04:43:14.167 INFO:tasks.rados.rados.0.smithi085.stdout:3921: read oid 6 snap 437 2021-02-10T04:43:14.167 INFO:tasks.rados.rados.0.smithi085.stdout:3921: expect (ObjNum 1208 snap 410 seq_num 1208) 2021-02-10T04:43:14.167 INFO:tasks.rados.rados.0.smithi085.stdout:3922: snap_remove snap 370 2021-02-10T04:43:14.167 INFO:tasks.rados.rados.0.smithi085.stdout:3922: done (5 left) 2021-02-10T04:43:14.167 INFO:tasks.rados.rados.0.smithi085.stdout:3921: expect (ObjNum 1208 snap 410 seq_num 1208) 2021-02-10T04:43:14.167 INFO:tasks.rados.rados.0.smithi085.stdout:3923: read oid 5 snap -1 2021-02-10T04:43:14.167 INFO:tasks.rados.rados.0.smithi085.stdout:3923: expect (ObjNum 1244 snap 419 seq_num 1244) 2021-02-10T04:43:14.167 INFO:tasks.rados.rados.0.smithi085.stdout:3921: done (5 left) 2021-02-10T04:43:14.167 INFO:tasks.rados.rados.0.smithi085.stdout:3924: rollback oid 44 current snap is 453 2021-02-10T04:43:14.167 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 44 to 393 2021-02-10T04:43:14.167 INFO:tasks.rados.rados.0.smithi085.stdout:3925: delete oid 38 current snap is 453 2021-02-10T04:43:14.167 INFO:tasks.rados.rados.0.smithi085.stdout:3925: done (6 left) 2021-02-10T04:43:14.167 INFO:tasks.rados.rados.0.smithi085.stdout:3926: read oid 2 snap -1 2021-02-10T04:43:14.168 INFO:tasks.rados.rados.0.smithi085.stdout:3926: expect deleted 2021-02-10T04:43:14.168 INFO:tasks.rados.rados.0.smithi085.stdout:3927: snap_remove snap 438 2021-02-10T04:43:14.168 INFO:tasks.rados.rados.0.smithi085.stdout:3927: done (7 left) 2021-02-10T04:43:14.168 INFO:tasks.rados.rados.0.smithi085.stdout:3928: copy_from oid 3 from oid 34 current snap is 453 2021-02-10T04:43:14.168 INFO:tasks.rados.rados.0.smithi085.stdout:3929: snap_create 2021-02-10T04:43:14.168 INFO:tasks.rados.rados.0.smithi085.stdout:3924: finishing rollback tid 1 to smithi08575397-44 2021-02-10T04:43:14.168 INFO:tasks.rados.rados.0.smithi085.stdout:3924: finishing rollback tid 2 to smithi08575397-44 2021-02-10T04:43:14.168 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 1896 (ObjNum 1052 snap 345 seq_num 1052) dirty exists 2021-02-10T04:43:14.168 INFO:tasks.rados.rados.0.smithi085.stdout:3923: expect (ObjNum 1244 snap 419 seq_num 1244) 2021-02-10T04:43:14.168 INFO:tasks.rados.rados.0.smithi085.stdout:3928: finishing copy_from to smithi08575397-3 2021-02-10T04:43:14.168 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 3 v 1908 (ObjNum 1145 snap 385 seq_num 1145) dirty exists 2021-02-10T04:43:14.168 INFO:tasks.rados.rados.0.smithi085.stdout:3928: finishing copy_from racing read to smithi08575397-3 2021-02-10T04:43:14.169 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: audit 2021-02-10T04:43:07.291438+0000 mgr.y ( 2021-02-10T04:43:14.169 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: mgr.14140) 888 : audit [DBG] from='client.35162 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:43:14.169 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: cluster 2021-02-10T04:43:07.543027+0000 mon.a (mon.0) 1953 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:43:14.169 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: audit 2021-02-10T04:43:07.555761+0000 mon.a (mon.0) 1954 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:43:14.169 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: cluster 2021-02-10T04:43: 2021-02-10T04:43:14.169 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: 07.555832+0000 mon.a (mon.0) 1955 : 2021-02-10T04:43:14.169 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: cluster [DBG] osdmap e967: 8 total, 8 up, 7 in 2021-02-10T04:43:14.169 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: cluster 2021-02-10T04:43:14.169 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: 2021-02-10T04:43:14.170 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: 2021-02-10T04:43:07.598605 2021-02-10T04:43:14.170 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1621 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:14.170 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:43:14.170 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: 43:07.598612+0000 osd.4 (osd. 2021-02-10T04:43:14.170 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: 4) 1622 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:14.170 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: cluster 2021-02-10T04:43:14.170 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: 2021-02-10T04:43:14.170 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: 2021 2021-02-10T04:43:14.170 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: - 2021-02-10T04:43:14.170 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: 02 2021-02-10T04:43:14.170 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: - 2021-02-10T04:43:14.170 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: 10 2021-02-10T04:43:14.170 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: T 2021-02-10T04:43:14.171 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: 04 2021-02-10T04:43:14.171 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: : 2021-02-10T04:43:14.171 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: 43 2021-02-10T04:43:14.171 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: : 2021-02-10T04:43:14.171 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: 07 2021-02-10T04:43:14.171 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: . 2021-02-10T04:43:14.171 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: 598616 2021-02-10T04:43:14.171 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: +0000 2021-02-10T04:43:14.171 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: 2021-02-10T04:43:14.171 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: osd.4 2021-02-10T04:43:14.171 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: ( 2021-02-10T04:43:14.171 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: osd 2021-02-10T04:43:14.171 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: . 2021-02-10T04:43:14.172 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: 4) 1623 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:14.172 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: cluster 2021-02-10T04:43:14.172 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: 2021-02-10T04:43:07.598620 2021-02-10T04:43:14.172 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1624 2021-02-10T04:43:14.172 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:14.172 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[29319]: cluster 2021-02-10T04:43:08.248439+0000 mgr.y (mgr.14140) 889 : cluster 2021-02-10T04:43:14.172 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[29319]: [DBG] pgmap v1602: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 910 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 19 MiB/s rd, 2.8 MiB/s wr, 23 op/s 2021-02-10T04:43:14.172 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[29319]: cluster 2021-02-10T04:43:08.559790+0000 mon.a (mon.0) 1956 : cluster [DBG] osdmap e968: 8 total, 8 up, 7 in 2021-02-10T04:43:14.172 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:43:14.172 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[29319]: 10T04:43:08.560698+0000 mgr.y (mgr.14140) 890 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:43:14.172 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[29319]: cluster 2021-02-10T04:43 2021-02-10T04:43:14.172 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[29319]: :08.595413+0000 osd.4 (osd.4) 1625 : 2021-02-10T04:43:14.173 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:14.173 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[29319]: cluster 2021-02-10T04:43:14.173 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[29319]: 2021 2021-02-10T04:43:14.173 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[29319]: - 2021-02-10T04:43:14.173 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[29319]: 02-10T04:43: 2021-02-10T04:43:14.173 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[29319]: 08.595424+0000 osd.4 (osd.4) 1626 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:14.173 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:43:14.173 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[29319]: 10T04:43:08.595431+0000 osd.4 (osd.4) 1627 : cluster 2021-02-10T04:43:14.173 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:14.173 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[29319]: cluster 2021-02-10T04:43: 2021-02-10T04:43:14.173 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[29319]: 08.595439+0000 osd.4 (osd.4) 1628 2021-02-10T04:43:14.173 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:14.173 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[29319]: cluster 2021-02-10T04:43:09.569245+0000 mon.a ( 2021-02-10T04:43:14.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[29319]: mon.0) 1957 : cluster [DBG] osdmap e969: 8 total, 8 up, 7 in 2021-02-10T04:43:14.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[29319]: cluster 2021-02-10T04:43:09.619182+0000 osd.4 (osd.4) 1629 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:14.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[29319]: 2021-02-10T04:43:14.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[29319]: cluster 2021-02-10T04:43:09.619188+0000 2021-02-10T04:43:14.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[29319]: osd.4 (osd.4) 1630 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:14.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:43:14.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[29319]: 04:43:09.619192+0000 osd.4 (osd.4) 2021-02-10T04:43:14.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[29319]: 1631 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:14.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:43:14.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[29319]: :43:09.619195+0000 osd.4 (osd.4) 2021-02-10T04:43:14.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[29319]: 1632 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:14.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[29319]: cluster 2021-02-10T04:43:14.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[29319]: 2021-02-10T04:43:14.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[29319]: 2021 2021-02-10T04:43:14.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[29319]: -02-10T04:43:10 2021-02-10T04:43:14.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[29319]: .249166+0000 mgr.y (mgr.14140) 891 : cluster [DBG] pgmap v1605: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 910 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 8.7 MiB/s rd, 3.9 MiB/s wr, 19 op/s 2021-02-10T04:43:14.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:43:14.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[29319]: -02-10T04:43:10.572945+0000 2021-02-10T04:43:14.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[29319]: mon.a (mon.0) 1958 : cluster [DBG] osdmap e970: 8 total, 8 up, 7 in 2021-02-10T04:43:14.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[29319]: 2021-02-10T04:43:14.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:43:14.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[29319]: 43:10.613729+0000 osd.4 (osd.4 2021-02-10T04:43:14.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[29319]: ) 1633 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:14.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[29319]: cluster 2021-02-10T04:43:14.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[29319]: 2021-02-10T04:43: 2021-02-10T04:43:14.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[29319]: 10.613739+0000 osd.4 (osd.4 2021-02-10T04:43:14.176 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[29319]: ) 1634 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:14.176 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:43:14.176 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[29319]: -02-10T04:43:10.613745 2021-02-10T04:43:14.176 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1635 : cluster 2021-02-10T04:43:14.176 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:14.176 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:43:14.176 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[29319]: 04:43:10.613751+0000 osd.4 (osd 2021-02-10T04:43:14.176 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[29319]: .4) 1636 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:14.176 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: cluster 2021-02-10T04:43:14.176 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: 2021-02-10T04:43:14.176 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: 2021 2021-02-10T04:43:14.176 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: - 2021-02-10T04:43:14.177 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: 02 2021-02-10T04:43:14.177 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: - 2021-02-10T04:43:14.177 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: 10 2021-02-10T04:43:14.177 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: T 2021-02-10T04:43:14.177 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: 04 2021-02-10T04:43:14.177 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: : 2021-02-10T04:43:14.177 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: 43 2021-02-10T04:43:14.177 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: : 2021-02-10T04:43:14.177 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: 11 2021-02-10T04:43:14.177 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: . 2021-02-10T04:43:14.177 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: 581069 2021-02-10T04:43:14.177 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: +0000 2021-02-10T04:43:14.177 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: 2021-02-10T04:43:14.178 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: mon.a 2021-02-10T04:43:14.178 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: ( 2021-02-10T04:43:14.178 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: mon 2021-02-10T04:43:14.178 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: . 2021-02-10T04:43:14.178 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: 0 2021-02-10T04:43:14.178 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: ) 2021-02-10T04:43:14.178 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: 1959 2021-02-10T04:43:14.178 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: : 2021-02-10T04:43:14.178 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: cluster 2021-02-10T04:43:14.178 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: 2021-02-10T04:43:14.178 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: [DBG] 2021-02-10T04:43:14.178 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: osdmap e971: 8 total, 8 up, 7 in 2021-02-10T04:43:14.178 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: cluster 2021-02-10T04:43:14.179 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: 2021-02-10T04:43:11.642046+0000 2021-02-10T04:43:14.179 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: osd.4 (osd.4) 1637 : cluster 2021-02-10T04:43:14.179 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:14.179 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: cluster 2021-02-10T04:43 2021-02-10T04:43:14.179 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: :11.642053+0000 osd.4 (osd.4) 2021-02-10T04:43:14.179 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: 1638 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:14.179 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:43:14.179 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: -10T04:43:11.642060+0000 2021-02-10T04:43:14.179 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: osd.4 (osd.4) 1639 : cluster [WRN] 2021-02-10T04:43:14.179 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:14.179 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: cluster 2021-02-10T04:43:11 2021-02-10T04:43:14.179 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: .642067+0000 osd.4 (osd.4) 1640 : 2021-02-10T04:43:14.179 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:14.180 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:43:14.180 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: 43:12.176433+0000 osd.2 (osd.2 2021-02-10T04:43:14.180 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: ) 1943 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:43:14.180 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:43:14.180 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: -10T04:43:12.177207+0000 osd.2 (osd 2021-02-10T04:43:14.180 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[29319]: .2) 1944 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:43:14.180 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[29319]: cluster 2021-02-10T04:43:12.249811+0000 2021-02-10T04:43:14.180 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[29319]: mgr.y (mgr.14140) 892 : cluster [DBG] pgmap v1608: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 910 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 6.2 MiB/s rd, 2.9 MiB/s wr, 13 op/s 2021-02-10T04:43:14.180 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[29319]: 2021-02-10T04:43:14.180 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[29319]: cluster 2021-02-10T04:43:14.180 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:43:14.180 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[29319]: 04:43:12.578912+0000 mon.a (mon.0) 1960 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 822 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:43:14.180 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[29319]: 2021-02-10T04:43:14.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[29319]: cluster 2021-02-10T04:43:14.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:43:14.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[29319]: 04:43:12.583496+0000 mon.a (mon.0) 2021-02-10T04:43:14.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[29319]: 1961 : cluster [DBG] osdmap e972: 8 total, 8 up, 7 in 2021-02-10T04:43:14.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:43:14.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[29319]: 10T04:43:12.687493+0000 osd.4 (osd.4) 1641 : 2021-02-10T04:43:14.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:14.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[29319]: cluster 2021-02-10T04:43:12. 2021-02-10T04:43:14.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[29319]: 687505+0000 osd.4 (osd.4) 1642 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:14.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[29319]: 2021-02-10T04:43:14.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:43:14.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[29319]: -02-10T04:43:12.687512+0000 osd.4 (osd.4) 1643 : cluster [WRN] 2021-02-10T04:43:14.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:14.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:43:14.182 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[29319]: 02-10T04:43:12.687518+0000 osd.4 (osd.4) 1644 : cluster [WRN] 2021-02-10T04:43:14.182 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:14.182 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: audit 2021-02-10T04:43:14.182 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: 2021-02-10T04:43:14.182 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: 2021 2021-02-10T04:43:14.182 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: - 2021-02-10T04:43:14.182 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: 02 2021-02-10T04:43:14.182 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: - 2021-02-10T04:43:14.182 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: 10 2021-02-10T04:43:14.182 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: T 2021-02-10T04:43:14.182 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: 04 2021-02-10T04:43:14.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: : 2021-02-10T04:43:14.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: 43 2021-02-10T04:43:14.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: :07.291438+0000 2021-02-10T04:43:14.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: mgr.y (mgr.14140) 888 : audit [DBG] from='client.35162 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:43:14.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:43:14.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: 02-10T04:43:07.543027+0000 mon.a (mon 2021-02-10T04:43:14.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: .0) 1953 2021-02-10T04:43:14.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:43:14.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: audit 2021-02-10T04:43:14.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: 2021-02 2021-02-10T04:43:14.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: -10T04:43:07.555761+0000 mon.a (mon.0) 2021-02-10T04:43:14.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: 1954 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:43:14.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: cluster 2021-02-10T04:43:14.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: 2021-02-10T04:43:07. 2021-02-10T04:43:14.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: 555832+0000 mon.a (mon.0) 1955 : cluster [DBG] 2021-02-10T04:43:14.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: osdmap e967: 8 total, 8 up, 7 in 2021-02-10T04:43:14.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:43:14.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: 43:07.598605+0000 osd.4 (osd 2021-02-10T04:43:14.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: .4) 1621 : cluster 2021-02-10T04:43:14.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:14.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:43:14.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: 43:07.598612+0000 osd.4 (osd.4 2021-02-10T04:43:14.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: ) 1622 2021-02-10T04:43:14.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:14.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:43:14.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: -02-10T04:43:07.598616+0000 2021-02-10T04:43:14.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: osd.4 (osd.4) 1623 : cluster [WRN] 2021-02-10T04:43:14.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:14.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:43:14.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: 04:43:07.598620+0000 osd.4 (osd 2021-02-10T04:43:14.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:08 smithi085 conmon[35961]: .4) 1624 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:14.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: cluster 2021-02-10T04:43:14.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: 2021-02-10T04:43:14.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: 2021 2021-02-10T04:43:14.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: - 2021-02-10T04:43:14.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: 02 2021-02-10T04:43:14.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: - 2021-02-10T04:43:14.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: 10 2021-02-10T04:43:14.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: T 2021-02-10T04:43:14.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: 04 2021-02-10T04:43:14.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: : 2021-02-10T04:43:14.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: 43 2021-02-10T04:43:14.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: : 2021-02-10T04:43:14.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: 08 2021-02-10T04:43:14.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: . 2021-02-10T04:43:14.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: 248439 2021-02-10T04:43:14.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: +0000 2021-02-10T04:43:14.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: 2021-02-10T04:43:14.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: mgr.y 2021-02-10T04:43:14.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: ( 2021-02-10T04:43:14.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: mgr 2021-02-10T04:43:14.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: . 2021-02-10T04:43:14.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: 14140 2021-02-10T04:43:14.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: ) 2021-02-10T04:43:14.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: 889 2021-02-10T04:43:14.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: : 2021-02-10T04:43:14.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: cluster 2021-02-10T04:43:14.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: 2021-02-10T04:43:14.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: [DBG] 2021-02-10T04:43:14.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: 2021-02-10T04:43:14.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: pgmap v1602: 168 pgs: 1 active+clean+scrubbing, 5 active+clean+snaptrim_wait, 154 active+clean, 8 active+clean+snaptrim; 910 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 19 MiB/s rd, 2.8 MiB/s wr, 23 op/s 2021-02-10T04:43:14.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: 2021-02-10T04:43:14.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:43:14.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: 43:08.559790+0000 mon.a (mon.0) 1956 : cluster [DBG] osdmap e968: 8 total, 8 up, 7 in 2021-02-10T04:43:14.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:43:14.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: 43:08.560698+0000 mgr.y (mgr.14140) 890 : audit 2021-02-10T04:43:14.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:43:14.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:43:14.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: 43:08.595413+0000 osd.4 (osd.4) 1625 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:14.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: cluster 2021-02-10T04:43:14.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:43:14.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: 04:43:08.595424+0000 osd.4 (osd.4) 1626 : cluster 2021-02-10T04:43:14.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:14.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: cluster 2021-02-10T04:43:14.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:43:14.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: 10T04:43:08.595431+0000 osd.4 (osd.4) 1627 : 2021-02-10T04:43:14.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:14.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:43:14.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: 02-10T04:43: 2021-02-10T04:43:14.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: 08.595439+0000 osd.4 (osd.4 2021-02-10T04:43:14.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[35961]: ) 1628 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:14.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[35961]: cluster 2021-02-10T04:43:14.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[35961]: 2021-02-10T04:43:14.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[35961]: 2021 2021-02-10T04:43:14.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[35961]: - 2021-02-10T04:43:14.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[35961]: 02-10T04:43 2021-02-10T04:43:14.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[35961]: :09.569245+0000 mon.a (mon.0) 1957 : cluster [DBG] osdmap e969: 8 total, 8 up, 7 in 2021-02-10T04:43:14.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[35961]: cluster 2021-02-10T04:43:14.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[35961]: 2021-02-10T04:43:14.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[35961]: 2021 2021-02-10T04:43:14.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[35961]: - 2021-02-10T04:43:14.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[35961]: 02-10T04: 2021-02-10T04:43:14.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[35961]: 43:09.619182+0000 osd.4 (osd.4) 1629 : cluster 2021-02-10T04:43:14.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:14.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[35961]: cluster 2021-02-10T04:43:14.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[35961]: 2021-02-10T04:43:09.619188+0000 osd.4 (osd.4) 1630 : cluster 2021-02-10T04:43:14.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:14.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[35961]: cluster 2021-02-10T04:43:09.619192+0000 osd.4 2021-02-10T04:43:14.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[35961]: (osd.4) 1631 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:14.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:43:14.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[35961]: 02-10T04:43:09.619195+0000 2021-02-10T04:43:14.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[35961]: osd.4 (osd.4) 1632 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:14.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:10 smithi085 conmon[35961]: 2021-02-10T04:43:14.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[35961]: cluster 2021-02-10T04:43:10.249166+0000 mgr.y (mgr.14140 2021-02-10T04:43:14.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[35961]: ) 891 : cluster [DBG] pgmap v1605: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 910 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 8.7 MiB/s rd, 3.9 MiB/s wr, 19 op/s 2021-02-10T04:43:14.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[35961]: cluster 2021-02-10T04:43:10.572945+0000 mon.a (mon.0) 1958 : cluster [DBG] 2021-02-10T04:43:14.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[35961]: osdmap e970: 8 total, 8 up, 7 in 2021-02-10T04:43:14.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:43:14.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[35961]: T04:43:10.613729+0000 osd.4 (osd.4) 1633 2021-02-10T04:43:14.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:14.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[35961]: cluster 2021-02-10T04:43:14.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[35961]: 2021-02 2021-02-10T04:43:14.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[35961]: -10T04:43:10 2021-02-10T04:43:14.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[35961]: .613739+0000 osd.4 (osd.4) 1634 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:14.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[35961]: cluster 2021-02-10T04:43:14.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[35961]: 2021-02-10T04:43: 2021-02-10T04:43:14.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[35961]: 10.613745+0000 osd.4 (osd.4) 1635 2021-02-10T04:43:14.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:14.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[35961]: cluster 2021-02-10T04:43:14.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[35961]: 2021-02-10T04:43:10.613751+0000 2021-02-10T04:43:14.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:11 smithi085 conmon[35961]: osd.4 (osd.4) 1636 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:14.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: cluster 2021-02-10T04:43:14.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: 2021-02-10T04:43:11.581069+0000 mon.a (mon.0) 1959 : cluster [DBG] osdmap e971: 8 total, 8 up, 7 in 2021-02-10T04:43:14.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: cluster 2021-02-10T04:43:14.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: 2021 2021-02-10T04:43:14.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: - 2021-02-10T04:43:14.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: 02 2021-02-10T04:43:14.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: - 2021-02-10T04:43:14.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: 10 2021-02-10T04:43:14.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: T04:43:11.642046+0000 2021-02-10T04:43:14.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: osd.4 (osd.4) 1637 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:14.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:43:14.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: 02-10T04:43:11.642053+0000 osd.4 ( 2021-02-10T04:43:14.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: osd.4) 1638 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:14.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: cluster 2021-02-10T04:43:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: 2021-02-10T04:43:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: 2021-02-10T04:43:11.642060+0000 osd.4 (osd. 2021-02-10T04:43:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: 4) 1639 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: cluster 2021-02-10T04:43:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: 2021-02 2021-02-10T04:43:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: -10T04:43:11.642067+0000 osd.4 (osd. 2021-02-10T04:43:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: 4) 1640 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: 2021-02-10T04:43:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: cluster 2021-02-10T04:43:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:43:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: 10T04:43:12.176433 2021-02-10T04:43:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 2021-02-10T04:43:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: 1943 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:43:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: cluster 2021-02-10T04:43:14.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:43:14.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: 10T04:43:12.177207 2021-02-10T04:43:14.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: +0000 osd.2 (osd.2 2021-02-10T04:43:14.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: ) 1944 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:43:14.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:12 smithi085 conmon[35961]: 2021-02-10T04:43:14.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[35961]: cluster 2021-02-10T04:43:14.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[35961]: 2021-02-10T04:43:14.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[35961]: 2021 2021-02-10T04:43:14.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[35961]: - 2021-02-10T04:43:14.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[35961]: 02 2021-02-10T04:43:14.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[35961]: - 2021-02-10T04:43:14.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[35961]: 10 2021-02-10T04:43:14.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[35961]: T 2021-02-10T04:43:14.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[35961]: 04:43:12.249811+0000 mgr.y (mgr 2021-02-10T04:43:14.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[35961]: .14140) 892 : cluster [DBG] pgmap v1608: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 910 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 6.2 MiB/s rd, 2.9 MiB/s wr, 13 op/s 2021-02-10T04:43:14.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[35961]: cluster 2021-02-10T04:43:14.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:43:14.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[35961]: :43:12.578912+0000 mon.a (mon.0) 1960 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 822 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:43:14.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:43:14.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[35961]: 10T04:43:12.583496+0000 mon.a (mon.0) 2021-02-10T04:43:14.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[35961]: 1961 : cluster [DBG] osdmap e972: 8 total, 8 up, 7 in 2021-02-10T04:43:14.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:43:14.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[35961]: -10T04:43:12.687493+0000 osd.4 (osd 2021-02-10T04:43:14.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[35961]: .4) 1641 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:14.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:43:14.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[35961]: -02-10T04:43:12.687505+0000 2021-02-10T04:43:14.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[35961]: osd.4 (osd.4) 1642 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:14.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:43:14.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[35961]: -02-10T04:43:12.687512+0000 2021-02-10T04:43:14.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[35961]: osd.4 (osd.4) 1643 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:14.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[35961]: 2021-02-10T04:43:14.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[35961]: cluster 2021-02-10T04:43:14.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[35961]: 2021-02-10T04:43:12.687518+0000 osd.4 (osd 2021-02-10T04:43:14.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:13 smithi085 conmon[35961]: .4) 1644 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:14.196 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:43:09 smithi085 conmon[41540]: debug 2021-02-10T04:43:09.663+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:43:14.196 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:43:14.597 INFO:tasks.rados.rados.0.smithi085.stdout:3923: done (8 left) 2021-02-10T04:43:14.597 INFO:tasks.rados.rados.0.smithi085.stdout:3924: done (7 left) 2021-02-10T04:43:14.597 INFO:tasks.rados.rados.0.smithi085.stdout:3926: done (6 left) 2021-02-10T04:43:14.597 INFO:tasks.rados.rados.0.smithi085.stdout:3928: done (5 left) 2021-02-10T04:43:14.597 INFO:tasks.rados.rados.0.smithi085.stdout:3929: done (4 left) 2021-02-10T04:43:14.597 INFO:tasks.rados.rados.0.smithi085.stdout:3930: read oid 23 snap -1 2021-02-10T04:43:14.597 INFO:tasks.rados.rados.0.smithi085.stdout:3930: expect (ObjNum 1224 snap 414 seq_num 1224) 2021-02-10T04:43:14.597 INFO:tasks.rados.rados.0.smithi085.stdout:3931: snap_remove snap 451 2021-02-10T04:43:14.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:14 smithi164 conmon[33677]: debug 2021-02-10T04:43:14.713+0000 7fbec19a1700 -1 osd.4 973 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:14 smithi164 conmon[29288]: cluster 2021-02-10T04:43:13.592195+0000 mon.a ( 2021-02-10T04:43:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:14 smithi164 conmon[29288]: mon.0) 1962 : cluster [DBG] osdmap e973: 8 total, 8 up, 7 in 2021-02-10T04:43:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:14 smithi164 conmon[29288]: cluster 2021-02-10T04:43:13.670287+0000 osd.4 (osd.4) 1645 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:14 smithi164 conmon[29288]: cluster 2021-02-10T04:43:13.670312+0000 osd.4 (osd 2021-02-10T04:43:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:14 smithi164 conmon[29288]: .4) 1646 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:14 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:43:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:14 smithi164 conmon[29288]: -10T04:43:13.670321+0000 osd.4 (osd 2021-02-10T04:43:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:14 smithi164 conmon[29288]: .4) 1647 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:14 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:43:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:14 smithi164 conmon[29288]: 02-10T04:43:13.670328+0000 osd.4 2021-02-10T04:43:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:14 smithi164 conmon[29288]: (osd.4) 1648 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:14 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:43:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:14 smithi164 conmon[29288]: -02-10T04:43:14.596027+0000 mon.a (mon.0 2021-02-10T04:43:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:14 smithi164 conmon[29288]: ) 1963 : cluster [DBG] osdmap e974: 8 total, 8 up, 7 in 2021-02-10T04:43:14.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[29319]: cluster 2021-02-10T04:43:13.592195+0000 mon.a ( 2021-02-10T04:43:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[29319]: mon.0) 1962 : cluster [DBG] osdmap e973: 8 total, 8 up, 7 in 2021-02-10T04:43:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[29319]: cluster 2021-02-10T04:43:13.670287+0000 osd.4 (osd.4) 1645 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[29319]: cluster 2021-02-10T04:43 2021-02-10T04:43:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[29319]: :13.670312+0000 osd.4 (osd.4) 1646 : 2021-02-10T04:43:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:43:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[29319]: :43:13.670321+0000 osd.4 (osd. 2021-02-10T04:43:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[29319]: 4) 1647 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:43:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[29319]: -10T04:43:13.670328 2021-02-10T04:43:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1648 : 2021-02-10T04:43:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:14.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[29319]: cluster 2021-02-10T04:43:14.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[29319]: 2021-02-10T04:43:14.596027+0000 mon.a (mon.0) 1963 : cluster [DBG] osdmap e974: 8 total, 8 up, 7 in 2021-02-10T04:43:14.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[35961]: cluster 2021-02-10T04:43:14.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[35961]: 2021-02-10T04:43 2021-02-10T04:43:14.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[35961]: :13.592195+0000 mon.a (mon.0) 1962 : cluster [DBG] osdmap e973: 8 total, 8 up, 7 in 2021-02-10T04:43:14.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:43:14.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[35961]: :43:13.670287+0000 osd.4 (osd.4) 2021-02-10T04:43:14.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[35961]: 1645 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:14.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:43:14.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[35961]: -02-10T04:43 2021-02-10T04:43:14.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[35961]: :13.670312+0000 osd.4 (osd.4) 2021-02-10T04:43:14.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[35961]: 1646 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:14.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:43:14.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[35961]: 02-10T04:43:13.670321+0000 2021-02-10T04:43:14.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[35961]: osd.4 (osd.4) 1647 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:14.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:43:14.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[35961]: 02-10T04:43:13.670328+0000 osd.4 2021-02-10T04:43:14.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[35961]: (osd.4) 1648 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:14.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:43:14.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[35961]: 10T04:43:14.596027+0000 mon.a (mon 2021-02-10T04:43:14.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:14 smithi085 conmon[35961]: .0) 1963 : cluster [DBG] osdmap e974: 8 total, 8 up, 7 in 2021-02-10T04:43:15.596 INFO:tasks.rados.rados.0.smithi085.stdout:3931: done (5 left) 2021-02-10T04:43:15.596 INFO:tasks.rados.rados.0.smithi085.stdout:3932: copy_from oid 35 from oid 11 current snap is 454 2021-02-10T04:43:15.596 INFO:tasks.rados.rados.0.smithi085.stdout:3930: expect (ObjNum 1224 snap 414 seq_num 1224) 2021-02-10T04:43:15.786 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:43:15.822 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:43:15.866 INFO:tasks.rados.rados.0.smithi085.stdout:3930: done (5 left) 2021-02-10T04:43:15.866 INFO:tasks.rados.rados.0.smithi085.stdout:3933: delete oid 46 current snap is 454 2021-02-10T04:43:15.866 INFO:tasks.rados.rados.0.smithi085.stdout:3932: finishing copy_from to smithi08575397-35 2021-02-10T04:43:15.866 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 35 v 1923 (ObjNum 1291 snap 435 seq_num 1291) dirty exists 2021-02-10T04:43:15.866 INFO:tasks.rados.rados.0.smithi085.stdout:3932: finishing copy_from racing read to smithi08575397-35 2021-02-10T04:43:15.869 INFO:tasks.rados.rados.0.smithi085.stdout:3932: done (5 left) 2021-02-10T04:43:15.869 INFO:tasks.rados.rados.0.smithi085.stdout:3933: done (4 left) 2021-02-10T04:43:15.869 INFO:tasks.rados.rados.0.smithi085.stdout:3934: copy_from oid 6 from oid 37 current snap is 454 2021-02-10T04:43:15.869 INFO:tasks.rados.rados.0.smithi085.stdout:3935: snap_create 2021-02-10T04:43:15.893 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:15 smithi164 conmon[33677]: debug 2021-02-10T04:43:15.695+0000 7fbec19a1700 -1 osd.4 975 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:15.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:15 smithi164 conmon[29288]: cluster 2021-02-10T04:43:15.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:15 smithi164 conmon[29288]: 2021-02-10T04:43:14 2021-02-10T04:43:15.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:15 smithi164 conmon[29288]: .250478+0000 mgr.y (mgr.14140) 893 : cluster [DBG] pgmap v1611: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 909 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 MiB/s rd, 0 B/s wr, 3 op/s 2021-02-10T04:43:15.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:15 smithi164 conmon[29288]: cluster 2021-02-10T04:43:15.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:15 smithi164 conmon[29288]: 2021-02-10T04:43:14.714016+0000 osd.4 (osd.4) 1649 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:15.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:15 smithi164 conmon[29288]: 2021-02-10T04:43:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:15 smithi164 conmon[29288]: cluster 2021-02-10T04:43:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:15 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:43:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:15 smithi164 conmon[29288]: :43:14.714026+0000 osd.4 (osd.4) 1650 : cluster 2021-02-10T04:43:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:15 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:15 smithi164 conmon[29288]: cluster 2021-02-10T04:43:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:15 smithi164 conmon[29288]: 2021-02- 2021-02-10T04:43:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:15 smithi164 conmon[29288]: 10T04:43:14.714045+0000 osd.4 (osd.4) 1651 2021-02-10T04:43:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:15 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:15 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:43:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:15 smithi164 conmon[29288]: -10T04:43:14.714052+0000 osd.4 (osd.4) 2021-02-10T04:43:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:15 smithi164 conmon[29288]: 1652 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:15 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:43:15.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:15 smithi164 conmon[29288]: -10T04:43:15.595217+0000 mon.a (mon.0) 1964 : cluster [DBG] osdmap e975: 8 total, 8 up, 7 in 2021-02-10T04:43:15.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[29319]: cluster 2021-02-10T04:43:15.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[29319]: 2021-02 2021-02-10T04:43:15.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[29319]: -10T04:43:14.250478+0000 mgr.y 2021-02-10T04:43:15.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[29319]: (mgr.14140) 893 : cluster [DBG] pgmap v1611: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 909 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 MiB/s rd, 0 B/s wr, 3 op/s 2021-02-10T04:43:15.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[29319]: cluster 2021-02-10T04:43:15.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[29319]: 2021-02 2021-02-10T04:43:15.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[29319]: -10T04:43:14.714016+0000 osd.4 (osd.4 2021-02-10T04:43:15.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[29319]: ) 1649 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:15.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:43:15.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[29319]: 10T04:43:14.714026+0000 osd.4 (osd. 2021-02-10T04:43:15.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[29319]: 4) 1650 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:15.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:43:15.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[29319]: 02-10T04:43:14.714045+0000 2021-02-10T04:43:15.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[29319]: osd.4 (osd.4) 1651 : cluster [WRN] 2021-02-10T04:43:15.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:15.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[29319]: cluster 2021-02-10T04:43:14 2021-02-10T04:43:15.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[29319]: .714052+0000 osd.4 (osd.4) 1652 : 2021-02-10T04:43:15.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:15.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:43:15.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[29319]: :43:15.595217+0000 mon.a (mon. 2021-02-10T04:43:15.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[29319]: 0) 1964 : cluster [DBG] osdmap e975: 8 total, 8 up, 7 in 2021-02-10T04:43:15.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: cluster 2021-02-10T04:43:15.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: 2021-02-10T04:43:15.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: 2021 2021-02-10T04:43:15.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: - 2021-02-10T04:43:15.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: 02 2021-02-10T04:43:15.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: - 2021-02-10T04:43:15.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: 10 2021-02-10T04:43:15.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: T 2021-02-10T04:43:15.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: 04 2021-02-10T04:43:15.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: : 2021-02-10T04:43:15.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: 43 2021-02-10T04:43:15.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: : 2021-02-10T04:43:15.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: 14 2021-02-10T04:43:15.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: . 2021-02-10T04:43:15.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: 250478 2021-02-10T04:43:15.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: +0000 2021-02-10T04:43:15.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: 2021-02-10T04:43:15.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: mgr.y 2021-02-10T04:43:15.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: ( 2021-02-10T04:43:15.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: mgr 2021-02-10T04:43:15.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: . 2021-02-10T04:43:15.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: 14140 2021-02-10T04:43:15.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: ) 2021-02-10T04:43:15.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: 893 2021-02-10T04:43:15.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: : 2021-02-10T04:43:15.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: cluster 2021-02-10T04:43:15.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: 2021-02-10T04:43:15.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: [DBG] pgmap v1611: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 909 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 MiB/s rd, 0 B/s wr, 3 op/s 2021-02-10T04:43:15.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: cluster 2021-02-10T04:43:14.714016 2021-02-10T04:43:15.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1649 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:15.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:43:15.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: 10T04:43:14.714026+0000 osd.4 (osd. 2021-02-10T04:43:15.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: 4) 1650 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:15.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:43:15.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: 04:43:14.714045+0000 osd.4 (osd.4) 2021-02-10T04:43:15.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: 1651 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:15.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:43:15.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: :43:14.714052+0000 osd.4 (osd.4) 2021-02-10T04:43:15.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: 1652 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:15.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: cluster 2021-02-10T04:43 2021-02-10T04:43:15.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: :15.595217+0000 mon.a (mon.0) 1964 : cluster 2021-02-10T04:43:15.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:15 smithi085 conmon[35961]: [DBG] osdmap e975: 8 total, 8 up, 7 in 2021-02-10T04:43:16.503 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:43:16.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:16 smithi085 conmon[29319]: cluster 2021-02-10T04:43:15.695871+0000 osd.4 ( 2021-02-10T04:43:16.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:16 smithi085 conmon[29319]: osd.4) 1653 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:16.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:16 smithi085 conmon[29319]: cluster 2021-02-10T04:43:15.695878+0000 osd.4 (osd.4) 1654 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:16.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:16 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:43:16.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:16 smithi085 conmon[29319]: 04:43:15.695883+0000 osd.4 (osd.4) 1655 : cluster [WRN] 2021-02-10T04:43:16.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:16 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:16.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:16 smithi085 conmon[29319]: cluster 2021-02-10T04:43:16.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:16 smithi085 conmon[29319]: 2021-02-10T04:43:15.695887+0000 osd.4 (osd.4) 1656 2021-02-10T04:43:16.620 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:16 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:16.620 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:43:16 smithi085 conmon[52836]: debug 2021-02-10T04:43:16.211+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:43:16.621 INFO:tasks.rados.rados.0.smithi085.stdout:3935: done (5 left) 2021-02-10T04:43:16.621 INFO:tasks.rados.rados.0.smithi085.stdout:3936: delete oid 36 current snap is 455 2021-02-10T04:43:16.629 INFO:tasks.rados.rados.0.smithi085.stdout:3936: done (5 left) 2021-02-10T04:43:16.629 INFO:tasks.rados.rados.0.smithi085.stdout:3937: snap_remove snap 308 2021-02-10T04:43:16.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:16 smithi164 conmon[33677]: debug 2021-02-10T04:43:16.730+0000 7fbec19a1700 -1 osd.4 976 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:16.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:16 smithi164 conmon[29288]: cluster 2021-02-10T04:43:15.695871+0000 osd.4 ( 2021-02-10T04:43:16.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:16 smithi164 conmon[29288]: osd.4) 1653 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:16.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:16 smithi164 conmon[29288]: cluster 2021-02-10T04:43:15.695878+0000 osd.4 (osd.4) 1654 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:16 smithi164 conmon[29288]: cluster 2021-02-10T04:43:15.695883+0000 osd.4 (osd.4) 1655 : cluster 2021-02-10T04:43:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:16 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:16 smithi164 conmon[29288]: cluster 2021-02-10T04:43:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:16 smithi164 conmon[29288]: 2021-02-10T04:43:15.695887+0000 osd.4 (osd.4) 1656 : cluster 2021-02-10T04:43:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:16 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:16.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:16 smithi085 conmon[35961]: cluster 2021-02-10T04:43:16.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:16 smithi085 conmon[35961]: 2021-02-10T04:43:16.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:16 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:43:16.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:16 smithi085 conmon[35961]: :43:15.695871+0000 osd.4 (osd.4) 1653 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:16.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:16 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:43:16.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:16 smithi085 conmon[35961]: 04:43:15.695878+0000 osd.4 (osd. 2021-02-10T04:43:16.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:16 smithi085 conmon[35961]: 4) 1654 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:16.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:16 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:43:16.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:16 smithi085 conmon[35961]: 02-10T04:43:15.695883+0000 2021-02-10T04:43:16.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:16 smithi085 conmon[35961]: osd.4 (osd.4) 1655 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:16.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:16 smithi085 conmon[35961]: 2021-02-10T04:43:16.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:16 smithi085 conmon[35961]: cluster 2021-02-10T04:43:16.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:16 smithi085 conmon[35961]: 2021-02-10T04:43:15.695887+0000 osd.4 ( 2021-02-10T04:43:16.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:16 smithi085 conmon[35961]: osd.4) 1656 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:17.623 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:17 smithi085 conmon[29319]: cluster 2021-02-10T04:43:16.251020+0000 mgr.y 2021-02-10T04:43:17.623 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:17 smithi085 conmon[29319]: (mgr.14140) 894 : cluster [DBG] pgmap v1614: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 910 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 7.3 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2021-02-10T04:43:17.623 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:17 smithi085 conmon[29319]: cluster 2021-02-10T04:43:16.621316+0000 mon.a (mon.0) 1965 : cluster 2021-02-10T04:43:17.623 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:17 smithi085 conmon[29319]: [DBG] osdmap e976: 8 total, 8 up, 7 in 2021-02-10T04:43:17.623 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:17 smithi085 conmon[29319]: cluster 2021-02-10T04:43:17.623 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:17 smithi085 conmon[29319]: 2021-02-10T04:43:16.731187+0000 osd.4 (osd.4) 1657 2021-02-10T04:43:17.623 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:17 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:17.623 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:17 smithi085 conmon[29319]: cluster 2021-02-10T04:43 2021-02-10T04:43:17.624 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:17 smithi085 conmon[29319]: :16.731197+0000 osd.4 (osd.4) 1658 : 2021-02-10T04:43:17.624 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:17 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:17.624 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:17 smithi085 conmon[29319]: cluster 2021-02-10T04:43:16 2021-02-10T04:43:17.624 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:17 smithi085 conmon[29319]: .731203+0000 osd.4 (osd.4) 1659 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:17.624 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:17 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:43:17.624 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:17 smithi085 conmon[29319]: -10T04:43:16.731208+0000 osd.4 2021-02-10T04:43:17.624 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:17 smithi085 conmon[29319]: (osd.4) 1660 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:17.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:17 smithi085 conmon[35961]: cluster 2021-02-10T04:43:16.251020+0000 2021-02-10T04:43:17.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:17 smithi085 conmon[35961]: mgr.y (mgr.14140) 894 : cluster [DBG] pgmap v1614: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 910 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 7.3 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2021-02-10T04:43:17.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:17 smithi085 conmon[35961]: cluster 2021-02-10T04:43:16.621316 2021-02-10T04:43:17.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:17 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1965 : cluster [DBG] osdmap e976: 8 total, 8 up, 7 in 2021-02-10T04:43:17.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:17 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:43:17.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:17 smithi085 conmon[35961]: 04:43:16.731187+0000 osd.4 (osd. 2021-02-10T04:43:17.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:17 smithi085 conmon[35961]: 4) 1657 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:17.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:17 smithi085 conmon[35961]: cluster 2021-02-10T04:43 2021-02-10T04:43:17.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:17 smithi085 conmon[35961]: :16.731197+0000 osd.4 (osd.4) 1658 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:17.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:17 smithi085 conmon[35961]: cluster 2021-02-10T04:43:17.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:17 smithi085 conmon[35961]: 2021-02-10T04:43:16.731203+0000 osd.4 (osd.4) 1659 : 2021-02-10T04:43:17.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:17 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:17.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:17 smithi085 conmon[35961]: cluster 2021-02-10T04:43:16 2021-02-10T04:43:17.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:17 smithi085 conmon[35961]: .731208+0000 osd.4 (osd.4) 1660 : cluster 2021-02-10T04:43:17.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:17 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:17.628 INFO:tasks.rados.rados.0.smithi085.stdout:3937: done (5 left) 2021-02-10T04:43:17.628 INFO:tasks.rados.rados.0.smithi085.stdout:3938: read oid 1 snap 413 2021-02-10T04:43:17.628 INFO:tasks.rados.rados.0.smithi085.stdout:3938: expect (ObjNum 1201 snap 409 seq_num 1201) 2021-02-10T04:43:17.628 INFO:tasks.rados.rados.0.smithi085.stdout:3939: copy_from oid 9 from oid 45 current snap is 455 2021-02-10T04:43:17.628 INFO:tasks.rados.rados.0.smithi085.stdout:3934: finishing copy_from to smithi08575397-6 2021-02-10T04:43:17.628 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 6 v 1851 (ObjNum 1129 snap 381 seq_num 1129) dirty exists 2021-02-10T04:43:17.628 INFO:tasks.rados.rados.0.smithi085.stdout:3934: finishing copy_from racing read to smithi08575397-6 2021-02-10T04:43:17.628 INFO:tasks.rados.rados.0.smithi085.stdout:3934: done (6 left) 2021-02-10T04:43:17.628 INFO:tasks.rados.rados.0.smithi085.stdout:3940: rollback oid 31 current snap is 455 2021-02-10T04:43:17.628 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 31 to 399 2021-02-10T04:43:17.628 INFO:tasks.rados.rados.0.smithi085.stdout:3941: snap_create 2021-02-10T04:43:17.641 INFO:tasks.rados.rados.0.smithi085.stdout:3938: expect (ObjNum 1201 snap 409 seq_num 1201) 2021-02-10T04:43:17.731 INFO:tasks.rados.rados.0.smithi085.stdout:3940: finishing rollback tid 0 to smithi08575397-31 2021-02-10T04:43:17.731 INFO:tasks.rados.rados.0.smithi085.stdout:3940: finishing rollback tid 1 to smithi08575397-31 2021-02-10T04:43:17.731 INFO:tasks.rados.rados.0.smithi085.stdout:3940: finishing rollback tid 2 to smithi08575397-31 2021-02-10T04:43:17.731 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 1892 (ObjNum 1097 snap 368 seq_num 1097) dirty exists 2021-02-10T04:43:17.731 INFO:tasks.rados.rados.0.smithi085.stdout:3939: finishing copy_from to smithi08575397-9 2021-02-10T04:43:17.731 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 9 v 1504 (ObjNum 1276 snap 430 seq_num 1276) dirty exists 2021-02-10T04:43:17.732 INFO:tasks.rados.rados.0.smithi085.stdout:3939: finishing copy_from racing read to smithi08575397-9 2021-02-10T04:43:17.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:17 smithi164 conmon[33677]: debug 2021-02-10T04:43:17.757+0000 7fbec19a1700 -1 osd.4 977 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:17.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:17 smithi164 conmon[29288]: cluster 2021-02-10T04:43:16.251020 2021-02-10T04:43:17.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:17 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 894 : cluster [DBG] pgmap v1614: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 910 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 7.3 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2021-02-10T04:43:17.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:17 smithi164 conmon[29288]: cluster 2021-02-10T04:43:16.621316+0000 mon.a (mon.0) 1965 : cluster [DBG] osdmap e976: 8 total, 8 up, 7 in 2021-02-10T04:43:17.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:17 smithi164 conmon[29288]: 2021-02-10T04:43:17.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:17 smithi164 conmon[29288]: cluster 2021-02-10T04:43: 2021-02-10T04:43:17.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:17 smithi164 conmon[29288]: 16.731187+0000 osd.4 (osd.4) 1657 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:17.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:17 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:43:17.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:17 smithi164 conmon[29288]: -02-10T04:43:16. 2021-02-10T04:43:17.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:17 smithi164 conmon[29288]: 731197+0000 osd.4 (osd.4) 1658 : cluster [WRN] 2021-02-10T04:43:17.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:17 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:17.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:17 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:43:17.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:17 smithi164 conmon[29288]: 04:43:16.731203+0000 osd.4 (osd.4 2021-02-10T04:43:17.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:17 smithi164 conmon[29288]: ) 1659 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:17.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:17 smithi164 conmon[29288]: cluster 2021-02-10T04:43:16 2021-02-10T04:43:17.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:17 smithi164 conmon[29288]: .731208+0000 osd.4 (osd.4) 1660 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:18.333 INFO:tasks.rados.rados.0.smithi085.stdout:3938: done (7 left) 2021-02-10T04:43:18.333 INFO:tasks.rados.rados.0.smithi085.stdout:3939: done (6 left) 2021-02-10T04:43:18.333 INFO:tasks.rados.rados.0.smithi085.stdout:3940: done (5 left) 2021-02-10T04:43:18.333 INFO:tasks.rados.rados.0.smithi085.stdout:3941: done (4 left) 2021-02-10T04:43:18.333 INFO:tasks.rados.rados.0.smithi085.stdout:3942: delete oid 48 current snap is 456 2021-02-10T04:43:18.337 INFO:tasks.rados.rados.0.smithi085.stdout:3942: done (4 left) 2021-02-10T04:43:18.337 INFO:tasks.rados.rados.0.smithi085.stdout:3943: delete oid 23 current snap is 456 2021-02-10T04:43:18.340 INFO:tasks.rados.rados.0.smithi085.stdout:3943: done (4 left) 2021-02-10T04:43:18.341 INFO:tasks.rados.rados.0.smithi085.stdout:3944: write (excl) oid 28 current snap is 456 2021-02-10T04:43:18.341 INFO:tasks.rados.rados.0.smithi085.stdout:3944: seq_num 1340 ranges {521380=511788,1480224=742462} 2021-02-10T04:43:18.347 INFO:tasks.rados.rados.0.smithi085.stdout:3944: writing smithi08575397-28 from 521380 to 1033168 tid 1 2021-02-10T04:43:18.356 INFO:tasks.rados.rados.0.smithi085.stdout:3944: writing smithi08575397-28 from 1480224 to 2222686 tid 2 2021-02-10T04:43:18.356 INFO:tasks.rados.rados.0.smithi085.stdout:3945: snap_create 2021-02-10T04:43:18.356 INFO:tasks.rados.rados.0.smithi085.stdout:3944: finishing write tid 1 to smithi08575397-28 2021-02-10T04:43:18.361 INFO:tasks.rados.rados.0.smithi085.stdout:3944: finishing write tid 2 to smithi08575397-28 2021-02-10T04:43:18.361 INFO:tasks.rados.rados.0.smithi085.stdout:3944: finishing write tid 3 to smithi08575397-28 2021-02-10T04:43:18.361 INFO:tasks.rados.rados.0.smithi085.stdout:3944: finishing write tid 4 to smithi08575397-28 2021-02-10T04:43:18.361 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 28 v 1899 (ObjNum 1340 snap 456 seq_num 1340) dirty exists 2021-02-10T04:43:18.361 INFO:tasks.rados.rados.0.smithi085.stdout:3944: left oid 28 (ObjNum 1340 snap 456 seq_num 1340) 2021-02-10T04:43:18.894 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:43:18 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:43:18.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:18 smithi164 conmon[33677]: debug 2021-02-10T04:43:18.748+0000 7fbec19a1700 -1 osd.4 978 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:18.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:18 smithi164 conmon[29288]: cluster 2021-02-10T04:43:17.626787+0000 2021-02-10T04:43:18.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:18 smithi164 conmon[29288]: mon.a (mon.0) 1966 : cluster [DBG] osdmap e977: 8 total, 8 up, 7 in 2021-02-10T04:43:18.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:18 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:43:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:18 smithi164 conmon[29288]: :43:17.758320+0000 osd.4 (osd.4) 1661 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:18 smithi164 conmon[29288]: cluster 2021-02-10T04:43 2021-02-10T04:43:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:18 smithi164 conmon[29288]: :17.758347+0000 osd.4 (osd.4) 1662 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:18 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:43:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:18 smithi164 conmon[29288]: -10T04:43:17.758354+0000 osd.4 2021-02-10T04:43:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:18 smithi164 conmon[29288]: (osd.4) 1663 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:18 smithi164 conmon[29288]: cluster 2021-02-10T04:43:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:18 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:43:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:18 smithi164 conmon[29288]: :43:17.758360+0000 osd.4 (osd.4) 1664 : cluster [WRN] 2021-02-10T04:43:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:18 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:18 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:43:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:18 smithi164 conmon[29288]: -10T04:43:18. 2021-02-10T04:43:18.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:18 smithi164 conmon[29288]: 328931+0000 mon.a (mon.0) 1967 : cluster 2021-02-10T04:43:18.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:18 smithi164 conmon[29288]: [DBG] osdmap e978: 8 total, 8 up, 7 in 2021-02-10T04:43:18.909 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[29319]: cluster 2021-02-10T04:43:18.910 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[29319]: 2021-02-10T04:43:18.910 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[29319]: 2021 2021-02-10T04:43:18.910 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[29319]: -02-10T04:43:17. 2021-02-10T04:43:18.910 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[29319]: 626787+0000 mon.a (mon.0) 1966 : cluster [DBG] osdmap e977: 8 total, 8 up, 7 in 2021-02-10T04:43:18.910 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[29319]: cluster 2021-02-10T04:43:18.910 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[29319]: 2021-02-10T04:43:17.758320 2021-02-10T04:43:18.910 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1661 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:18.910 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[29319]: 2021-02-10T04:43:18.910 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[29319]: cluster 2021-02-10T04:43: 2021-02-10T04:43:18.910 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[29319]: 17.758347+0000 osd.4 (osd.4) 1662 : cluster 2021-02-10T04:43:18.910 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:18.910 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[29319]: cluster 2021-02-10T04:43 2021-02-10T04:43:18.911 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[29319]: :17.758354+0000 osd.4 (osd.4) 2021-02-10T04:43:18.911 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[29319]: 1663 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:18.911 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[29319]: cluster 2021-02-10T04:43:18.911 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:43:18.911 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[29319]: T04:43:17.758360+0000 osd.4 (osd.4) 1664 2021-02-10T04:43:18.911 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:18.911 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:43:18.911 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[29319]: 02-10T04:43:18.328931+0000 mon.a ( 2021-02-10T04:43:18.911 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[29319]: mon.0) 1967 : cluster [DBG] 2021-02-10T04:43:18.911 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[29319]: osdmap e978: 8 total, 8 up, 7 in 2021-02-10T04:43:18.911 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[35961]: cluster 2021-02-10T04:43:17.626787+0000 2021-02-10T04:43:18.912 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[35961]: mon.a (mon.0) 1966 : cluster [DBG] osdmap e977: 8 total, 8 up, 7 in 2021-02-10T04:43:18.912 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:43:18.912 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[35961]: 10T04:43:17.758320+0000 osd.4 (osd.4) 1661 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:18.912 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:43:18.912 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[35961]: T04:43:17.758347+0000 osd.4 (osd.4) 1662 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:18.912 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[35961]: cluster 2021-02-10T04:43 2021-02-10T04:43:18.912 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[35961]: :17.758354+0000 osd.4 (osd.4) 1663 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:18.912 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[35961]: cluster 2021-02-10T04:43:17 2021-02-10T04:43:18.912 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[35961]: .758360+0000 osd.4 (osd.4) 1664 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:18.912 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[35961]: cluster 2021-02-10T04:43:18.328931+0000 mon.a (mon.0) 1967 : 2021-02-10T04:43:18.912 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:18 smithi085 conmon[35961]: cluster [DBG] osdmap e978: 8 total, 8 up, 7 in 2021-02-10T04:43:19.341 INFO:tasks.rados.rados.0.smithi085.stdout:3944: done (5 left) 2021-02-10T04:43:19.342 INFO:tasks.rados.rados.0.smithi085.stdout:3945: done (4 left) 2021-02-10T04:43:19.342 INFO:tasks.rados.rados.0.smithi085.stdout:3946: snap_remove snap 456 2021-02-10T04:43:19.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:19 smithi164 conmon[33677]: debug 2021-02-10T04:43:19.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:19 smithi164 conmon[33677]: 2021-02-10T04:43:19.786+0000 7fbec19a1700 -1 osd.4 979 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:19.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:19 smithi164 conmon[29288]: cluster 2021-02-10T04:43:19.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:19 smithi164 conmon[29288]: 2021-02-10T04:43:19.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:19 smithi164 conmon[29288]: 2021 2021-02-10T04:43:19.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:19 smithi164 conmon[29288]: - 2021-02-10T04:43:19.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:19 smithi164 conmon[29288]: 02-10T04:43 2021-02-10T04:43:19.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:19 smithi164 conmon[29288]: :18.251771+0000 mgr.y (mgr.14140) 895 : cluster [DBG] pgmap v1617: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 910 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 5.3 MiB/s rd, 1.9 MiB/s wr, 7 op/s 2021-02-10T04:43:19.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:19 smithi164 conmon[29288]: audit 2021-02-10T04:43: 2021-02-10T04:43:19.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:19 smithi164 conmon[29288]: 18.571078+0000 mgr.y (mgr.14140) 896 : audit [DBG] 2021-02-10T04:43:19.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:19 smithi164 conmon[29288]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:43:19.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:19 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:43:19.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:19 smithi164 conmon[29288]: 04:43:18.623841+0000 mon.a (mon.0 2021-02-10T04:43:19.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:19 smithi164 conmon[29288]: ) 1968 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 827 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:43:19.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:19 smithi164 conmon[29288]: cluster 2021-02-10T04:43: 2021-02-10T04:43:19.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:19 smithi164 conmon[29288]: 18.749494+0000 osd.4 (osd.4) 1665 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:19.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:19 smithi164 conmon[29288]: cluster 2021-02-10T04:43: 2021-02-10T04:43:19.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:19 smithi164 conmon[29288]: 18.749506+0000 osd.4 (osd.4) 1666 : cluster [WRN] 2021-02-10T04:43:19.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:19 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:19.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:19 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:43:19.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:19 smithi164 conmon[29288]: 43:18.749511+0000 osd.4 (osd.4) 1667 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:19.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:19 smithi164 conmon[29288]: cluster 2021-02-10T04:43: 2021-02-10T04:43:19.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:19 smithi164 conmon[29288]: 18.749516+0000 osd.4 (osd.4) 1668 : cluster [WRN] 2021-02-10T04:43:19.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:19 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:19.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:19 smithi164 conmon[29288]: cluster 2021-02-10T04:43:19 2021-02-10T04:43:19.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:19 smithi164 conmon[29288]: .341657+0000 mon.a (mon.0) 1969 : cluster 2021-02-10T04:43:19.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:19 smithi164 conmon[29288]: [DBG] osdmap e979: 8 total, 8 up, 7 in 2021-02-10T04:43:19.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[29319]: cluster 2021-02-10T04:43:19.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:43:19.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[29319]: :43:18.251771+0000 mgr.y (mgr.14140) 895 : cluster [DBG] pgmap v1617: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 910 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 5.3 MiB/s rd, 1.9 MiB/s wr, 7 op/s 2021-02-10T04:43:19.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[29319]: audit 2021-02-10T04:43:19.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[29319]: 2021-02-10T04:43:19.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[29319]: 2021-02-10T04:43: 2021-02-10T04:43:19.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[29319]: 18.571078+0000 mgr.y (mgr.14140) 896 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:43:19.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:43:19.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[29319]: 43:18.623841+0000 mon.a (mon.0) 1968 : 2021-02-10T04:43:19.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[29319]: cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 827 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:43:19.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[29319]: cluster 2021-02-10T04:43 2021-02-10T04:43:19.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[29319]: :18.749494+0000 osd.4 (osd.4) 1665 : 2021-02-10T04:43:19.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:19.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[29319]: cluster 2021-02-10T04:43: 2021-02-10T04:43:19.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[29319]: 18.749506+0000 osd.4 (osd.4) 1666 : cluster 2021-02-10T04:43:19.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:19.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[29319]: cluster 2021-02-10T04:43:19.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[29319]: 2021-02-10T04:43:18.749511+0000 osd.4 (osd 2021-02-10T04:43:19.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[29319]: .4) 1667 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:19.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:43:19.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[29319]: T04:43:18.749516+0000 osd.4 (osd 2021-02-10T04:43:19.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[29319]: .4) 1668 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:19.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:43:19.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[29319]: 10T04:43:19.341657+0000 mon.a ( 2021-02-10T04:43:19.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[29319]: mon.0) 1969 : cluster [DBG] osdmap e979: 8 total, 8 up, 7 in 2021-02-10T04:43:19.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[35961]: cluster 2021-02-10T04:43:18.251771+0000 2021-02-10T04:43:19.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[35961]: mgr.y (mgr.14140) 895 : cluster [DBG] pgmap v1617: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 910 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 5.3 MiB/s rd, 1.9 MiB/s wr, 7 op/s 2021-02-10T04:43:19.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[35961]: audit 2021-02-10T04:43:18.571078+0000 2021-02-10T04:43:19.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[35961]: mgr.y (mgr.14140) 896 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:43:19.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:43:19.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[35961]: -10T04:43:18. 2021-02-10T04:43:19.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[35961]: 623841+0000 mon.a (mon.0) 2021-02-10T04:43:19.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[35961]: 1968 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 827 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:43:19.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[35961]: cluster 2021-02-10T04:43:19.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:43:19.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[35961]: 43:18.749494+0000 osd.4 (osd.4) 1665 2021-02-10T04:43:19.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:19.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[35961]: cluster 2021-02-10T04:43:19.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[35961]: 2021-02-10T04:43:18.749506+0000 osd.4 (osd.4) 2021-02-10T04:43:19.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[35961]: 1666 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:19.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:43:19.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[35961]: -02-10T04:43:18.749511+0000 osd.4 (osd.4) 1667 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:19.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[35961]: cluster 2021-02-10T04:43:18.749516+0000 2021-02-10T04:43:19.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[35961]: osd.4 (osd.4) 1668 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:19.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:43:19.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[35961]: -10T04:43:19.341657+0000 mon.a (mon.0) 2021-02-10T04:43:19.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:19 smithi085 conmon[35961]: 1969 : cluster [DBG] osdmap e979: 8 total, 8 up, 7 in 2021-02-10T04:43:20.345 INFO:tasks.rados.rados.0.smithi085.stdout:3946: done (4 left) 2021-02-10T04:43:20.345 INFO:tasks.rados.rados.0.smithi085.stdout:3947: read oid 40 snap -1 2021-02-10T04:43:20.345 INFO:tasks.rados.rados.0.smithi085.stdout:3947: expect (ObjNum 1319 snap 451 seq_num 1319) 2021-02-10T04:43:20.345 INFO:tasks.rados.rados.0.smithi085.stdout:3948: write oid 48 current snap is 457 2021-02-10T04:43:20.345 INFO:tasks.rados.rados.0.smithi085.stdout:3948: seq_num 1341 ranges {538187=587037,1538706=602473} 2021-02-10T04:43:20.352 INFO:tasks.rados.rados.0.smithi085.stdout:3948: writing smithi08575397-48 from 538187 to 1125224 tid 1 2021-02-10T04:43:20.359 INFO:tasks.rados.rados.0.smithi085.stdout:3948: writing smithi08575397-48 from 1538706 to 2141179 tid 2 2021-02-10T04:43:20.359 INFO:tasks.rados.rados.0.smithi085.stdout:3949: copy_from oid 35 from oid 11 current snap is 457 2021-02-10T04:43:20.359 INFO:tasks.rados.rados.0.smithi085.stdout:3947: expect (ObjNum 1319 snap 451 seq_num 1319) 2021-02-10T04:43:20.509 INFO:tasks.rados.rados.0.smithi085.stdout:3948: finishing write tid 1 to smithi08575397-48 2021-02-10T04:43:20.509 INFO:tasks.rados.rados.0.smithi085.stdout:3948: finishing write tid 2 to smithi08575397-48 2021-02-10T04:43:20.509 INFO:tasks.rados.rados.0.smithi085.stdout:3948: finishing write tid 3 to smithi08575397-48 2021-02-10T04:43:20.509 INFO:tasks.rados.rados.0.smithi085.stdout:3947: done (6 left) 2021-02-10T04:43:20.509 INFO:tasks.rados.rados.0.smithi085.stdout:3950: read oid 34 snap -1 2021-02-10T04:43:20.510 INFO:tasks.rados.rados.0.smithi085.stdout:3950: expect (ObjNum 1145 snap 385 seq_num 1145) 2021-02-10T04:43:20.510 INFO:tasks.rados.rados.0.smithi085.stdout:3951: read oid 37 snap -1 2021-02-10T04:43:20.510 INFO:tasks.rados.rados.0.smithi085.stdout:3951: expect (ObjNum 1129 snap 381 seq_num 1129) 2021-02-10T04:43:20.510 INFO:tasks.rados.rados.0.smithi085.stdout:3952: write oid 46 current snap is 457 2021-02-10T04:43:20.510 INFO:tasks.rados.rados.0.smithi085.stdout:3952: seq_num 1342 ranges {554994=662286,1997188=557806} 2021-02-10T04:43:20.516 INFO:tasks.rados.rados.0.smithi085.stdout:3952: writing smithi08575397-46 from 554994 to 1217280 tid 1 2021-02-10T04:43:20.525 INFO:tasks.rados.rados.0.smithi085.stdout:3952: writing smithi08575397-46 from 1997188 to 2554994 tid 2 2021-02-10T04:43:20.525 INFO:tasks.rados.rados.0.smithi085.stdout:3953: snap_create 2021-02-10T04:43:20.525 INFO:tasks.rados.rados.0.smithi085.stdout:3948: finishing write tid 4 to smithi08575397-48 2021-02-10T04:43:20.525 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 48 v 1905 (ObjNum 1341 snap 457 seq_num 1341) dirty exists 2021-02-10T04:43:20.525 INFO:tasks.rados.rados.0.smithi085.stdout:3948: left oid 48 (ObjNum 1341 snap 457 seq_num 1341) 2021-02-10T04:43:20.525 INFO:tasks.rados.rados.0.smithi085.stdout:3949: finishing copy_from to smithi08575397-35 2021-02-10T04:43:20.525 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 35 v 1923 (ObjNum 1291 snap 435 seq_num 1291) dirty exists 2021-02-10T04:43:20.526 INFO:tasks.rados.rados.0.smithi085.stdout:3949: finishing copy_from racing read to smithi08575397-35 2021-02-10T04:43:20.526 INFO:tasks.rados.rados.0.smithi085.stdout:3952: finishing write tid 1 to smithi08575397-46 2021-02-10T04:43:20.526 INFO:tasks.rados.rados.0.smithi085.stdout:3951: expect (ObjNum 1129 snap 381 seq_num 1129) 2021-02-10T04:43:20.652 INFO:tasks.rados.rados.0.smithi085.stdout:3950: expect (ObjNum 1145 snap 385 seq_num 1145) 2021-02-10T04:43:20.814 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:43:20.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:20 smithi164 conmon[33677]: debug 2021-02-10T04:43:20.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:20 smithi164 conmon[33677]: 2021-02-10T04:43:20.788+0000 7fbec19a1700 -1 osd.4 980 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:20.895 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:20 smithi164 conmon[33677]: 2021-02-10T04:43:20.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:20 smithi164 conmon[29288]: cluster 2021-02-10T04:43:19.786932+0000 osd.4 (osd.4) 1669 2021-02-10T04:43:20.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:20 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:20.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:20 smithi164 conmon[29288]: cluster 2021-02-10T04:43:19.786940+0000 osd.4 (osd.4) 1670 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:20.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:20 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:43:20.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:20 smithi164 conmon[29288]: 43:19.786946+0000 osd.4 (osd.4) 1671 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:20.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:20 smithi164 conmon[29288]: cluster 2021-02-10T04:43:20.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:20 smithi164 conmon[29288]: 2021-02-10T04:43:19.786952 2021-02-10T04:43:20.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:20 smithi164 conmon[29288]: +0000 osd.4 (osd 2021-02-10T04:43:20.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:20 smithi164 conmon[29288]: .4) 2021-02-10T04:43:20.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:20 smithi164 conmon[29288]: 1672 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:20.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:20 smithi164 conmon[29288]: cluster 2021-02-10T04:43:20.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:20 smithi164 conmon[29288]: 2021-02-10T04:43:20. 2021-02-10T04:43:20.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:20 smithi164 conmon[29288]: 340896+0000 mon.a (mon.0) 1970 : cluster [DBG] osdmap e980: 8 total, 8 up, 7 in 2021-02-10T04:43:20.926 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:20 smithi085 conmon[29319]: cluster 2021-02-10T04:43:20.926 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:20 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:43:20.926 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:20 smithi085 conmon[29319]: 43:19.786932+0000 osd.4 (osd.4) 1669 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:20.927 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:20 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:43:20.927 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:20 smithi085 conmon[29319]: -02-10T04:43:19.786940+0000 osd.4 (osd.4) 1670 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:20.927 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:20 smithi085 conmon[29319]: cluster 2021-02-10T04:43:20.927 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:20 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:43:20.927 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:20 smithi085 conmon[29319]: 43:19.786946+0000 osd.4 (osd.4) 1671 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:20.928 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:20 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:43:20.929 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:20 smithi085 conmon[29319]: 43:19.786952+0000 osd.4 (osd.4) 1672 : cluster 2021-02-10T04:43:20.929 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:20 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:20.929 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:20 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:43:20.929 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:20 smithi085 conmon[29319]: 43:20.340896+0000 mon.a (mon.0) 1970 2021-02-10T04:43:20.929 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:20 smithi085 conmon[29319]: : cluster [DBG] osdmap e980: 8 total, 8 up, 7 in 2021-02-10T04:43:20.929 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:20 smithi085 conmon[35961]: cluster 2021-02-10T04:43:19.786932 2021-02-10T04:43:20.929 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:20 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1669 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:20.929 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:20 smithi085 conmon[35961]: cluster 2021-02-10T04:43:19.786940+0000 osd.4 2021-02-10T04:43:20.929 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:20 smithi085 conmon[35961]: (osd.4) 1670 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:20.930 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:20 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:43:20.930 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:20 smithi085 conmon[35961]: 02-10T04:43:19.786946+0000 osd.4 ( 2021-02-10T04:43:20.930 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:20 smithi085 conmon[35961]: osd.4) 1671 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:20.930 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:20 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:43:20.930 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:20 smithi085 conmon[35961]: -02-10T04:43:19.786952+0000 2021-02-10T04:43:20.930 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:20 smithi085 conmon[35961]: osd.4 (osd.4) 1672 : cluster [WRN] 2021-02-10T04:43:20.930 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:20 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:20.930 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:20 smithi085 conmon[35961]: cluster 2021-02-10T04:43:20.340896 2021-02-10T04:43:20.930 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:20 smithi085 conmon[35961]: +0000 mon.a (mon.0) 1970 : cluster 2021-02-10T04:43:20.930 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:20 smithi085 conmon[35961]: [DBG] osdmap e980: 8 total, 8 up, 7 in 2021-02-10T04:43:20.931 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:43:20.931 INFO:tasks.rados.rados.0.smithi085.stdout:3952: finishing write tid 2 to smithi08575397-46 2021-02-10T04:43:20.931 INFO:tasks.rados.rados.0.smithi085.stdout:3952: finishing write tid 3 to smithi08575397-46 2021-02-10T04:43:20.931 INFO:tasks.rados.rados.0.smithi085.stdout:3952: finishing write tid 4 to smithi08575397-46 2021-02-10T04:43:20.931 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 46 v 1901 (ObjNum 1342 snap 457 seq_num 1342) dirty exists 2021-02-10T04:43:20.931 INFO:tasks.rados.rados.0.smithi085.stdout:3952: left oid 46 (ObjNum 1342 snap 457 seq_num 1342) 2021-02-10T04:43:21.075 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:43:21.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:43:21 smithi085 conmon[52836]: debug 2021-02-10T04:43:21.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:43:21 smithi085 conmon[52836]: 2021-02-10T04:43:21.036+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:43:21.347 INFO:tasks.rados.rados.0.smithi085.stdout:3948: done (9 left) 2021-02-10T04:43:21.348 INFO:tasks.rados.rados.0.smithi085.stdout:3949: done (8 left) 2021-02-10T04:43:21.348 INFO:tasks.rados.rados.0.smithi085.stdout:3950: done (7 left) 2021-02-10T04:43:21.348 INFO:tasks.rados.rados.0.smithi085.stdout:3951: done (6 left) 2021-02-10T04:43:21.348 INFO:tasks.rados.rados.0.smithi085.stdout:3952: done (5 left) 2021-02-10T04:43:21.348 INFO:tasks.rados.rados.0.smithi085.stdout:3953: done (4 left) 2021-02-10T04:43:21.348 INFO:tasks.rados.rados.0.smithi085.stdout:3954: snap_remove snap 406 2021-02-10T04:43:21.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:21 smithi085 conmon[35961]: cluster 2021-02-10T04:43:20.252584+0000 mgr.y 2021-02-10T04:43:21.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:21 smithi085 conmon[35961]: (mgr.14140) 897 : cluster [DBG] pgmap v1620: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 902 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 8 op/s 2021-02-10T04:43:21.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:21 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:43:21.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:21 smithi085 conmon[35961]: :43:20.788873+0000 osd.4 (osd.4) 1673 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:21.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:21 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:43:21.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:21 smithi085 conmon[35961]: 10T04:43:20.788886+0000 osd.4 ( 2021-02-10T04:43:21.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:21 smithi085 conmon[35961]: osd.4) 1674 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:21.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:21 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:43:21.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:21 smithi085 conmon[35961]: 02-10T04:43:20.788893+0000 osd.4 2021-02-10T04:43:21.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:21 smithi085 conmon[35961]: (osd.4) 1675 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:21.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:21 smithi085 conmon[35961]: 2021-02-10T04:43:21.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:21 smithi085 conmon[35961]: cluster 2021-02-10T04:43:21.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:21 smithi085 conmon[35961]: 2021-02-10T04:43:20.788899+0000 osd.4 (osd 2021-02-10T04:43:21.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:21 smithi085 conmon[35961]: .4) 1676 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:21.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:21 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:43:21.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:21 smithi085 conmon[35961]: -10T04:43:21.343282+0000 mon.a ( 2021-02-10T04:43:21.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:21 smithi085 conmon[35961]: mon.0) 1971 : cluster [DBG] osdmap e981: 8 total, 8 up, 7 in 2021-02-10T04:43:21.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:21 smithi085 conmon[29319]: cluster 2021-02-10T04:43:20.252584+0000 mgr.y (mgr.14140) 897 : cluster [DBG] pgmap v1620: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 902 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 8 op/s 2021-02-10T04:43:21.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:21 smithi085 conmon[29319]: 2021-02-10T04:43:21.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:21 smithi085 conmon[29319]: cluster 2021-02-10T04:43:20.788873+0000 osd.4 (osd.4) 1673 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:21.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:21 smithi085 conmon[29319]: cluster 2021-02-10T04:43:20.788886+0000 osd.4 (osd.4) 1674 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:21.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:21 smithi085 conmon[29319]: cluster 2021-02-10T04:43:20.788893+0000 osd.4 (osd.4) 1675 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:21.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:21 smithi085 conmon[29319]: cluster 2021-02-10T04:43:20.788899+0000 osd.4 (osd.4) 1676 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:21.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:21 smithi085 conmon[29319]: cluster 2021-02-10T04:43:21.343282+0000 mon.a (mon.0) 1971 : cluster [DBG] osdmap e981: 8 total, 8 up, 7 in 2021-02-10T04:43:21.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:21 smithi164 conmon[33677]: debug 2021-02-10T04:43:21.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:21 smithi164 conmon[33677]: 2021-02-10T04:43:21.818+0000 7fbec19a1700 -1 osd.4 981 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:21.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:21 smithi164 conmon[33677]: 2021-02-10T04:43:21.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:21 smithi164 conmon[29288]: cluster 2021-02-10T04:43:20.252584+0000 mgr.y (mgr. 2021-02-10T04:43:21.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:21 smithi164 conmon[29288]: 14140) 897 : cluster [DBG] pgmap v1620: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 902 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 8 op/s 2021-02-10T04:43:21.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:21 smithi164 conmon[29288]: cluster 2021-02-10T04:43:20.788873+0000 osd.4 (osd.4) 1673 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:21.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:21 smithi164 conmon[29288]: cluster 2021-02-10T04:43:20.788886+0000 osd.4 (osd.4) 2021-02-10T04:43:21.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:21 smithi164 conmon[29288]: 1674 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:21.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:21 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:43:21.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:21 smithi164 conmon[29288]: 10T04:43:20.788893+0000 osd.4 (osd 2021-02-10T04:43:21.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:21 smithi164 conmon[29288]: .4) 1675 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:21.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:21 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:43:21.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:21 smithi164 conmon[29288]: -02-10T04:43:20.788899+0000 2021-02-10T04:43:21.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:21 smithi164 conmon[29288]: osd.4 (osd.4) 1676 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:21.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:21 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:43:21.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:21 smithi164 conmon[29288]: -10T04:43:21.343282+0000 mon.a (mon. 2021-02-10T04:43:21.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:21 smithi164 conmon[29288]: 0) 1971 : cluster [DBG] osdmap e981: 8 total, 8 up, 7 in 2021-02-10T04:43:22.349 INFO:tasks.rados.rados.0.smithi085.stdout:3954: done (4 left) 2021-02-10T04:43:22.350 INFO:tasks.rados.rados.0.smithi085.stdout:3955: write oid 44 current snap is 458 2021-02-10T04:43:22.350 INFO:tasks.rados.rados.0.smithi085.stdout:3955: seq_num 1343 ranges {571801=737535,1939317=489789} 2021-02-10T04:43:22.354 INFO:tasks.rados.rados.0.smithi085.stdout:3955: writing smithi08575397-44 from 571801 to 1309336 tid 1 2021-02-10T04:43:22.361 INFO:tasks.rados.rados.0.smithi085.stdout:3955: writing smithi08575397-44 from 1939317 to 2429106 tid 2 2021-02-10T04:43:22.361 INFO:tasks.rados.rados.0.smithi085.stdout:3956: snap_remove snap 354 2021-02-10T04:43:22.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[29319]: cluster 2021-02-10T04:43:22.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:43:22.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[29319]: T04:43:21.818991+0000 osd.4 (osd.4) 1677 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:22.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:43:22.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[29319]: 02-10T04:43:21.818997+0000 osd.4 (osd. 2021-02-10T04:43:22.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[29319]: 4) 1678 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:22.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:43:22.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[29319]: -10T04:43:21.819002+0000 osd.4 2021-02-10T04:43:22.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[29319]: (osd.4) 1679 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:22.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[29319]: 2021-02-10T04:43:22.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[29319]: cluster 2021-02-10T04:43:22.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[29319]: 2021-02-10T04:43:21.819009+0000 osd.4 (osd 2021-02-10T04:43:22.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[29319]: .4) 1680 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:22.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:43:22.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[29319]: -10T04:43:22.346303+0000 mon.a 2021-02-10T04:43:22.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[29319]: (mon.0) 1972 : cluster [DBG] osdmap e982: 8 total, 8 up, 7 in 2021-02-10T04:43:22.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[29319]: 2021-02-10T04:43:22.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[35961]: cluster 2021-02-10T04:43:21.818991+0000 osd.4 2021-02-10T04:43:22.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[35961]: (osd.4) 1677 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:22.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:43:22.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[35961]: -02-10T04:43:21.818997+0000 osd.4 2021-02-10T04:43:22.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[35961]: (osd.4) 1678 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:22.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:43:22.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[35961]: 04:43:21.819002+0000 osd.4 (osd.4 2021-02-10T04:43:22.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[35961]: ) 1679 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:22.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:43:22.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[35961]: 43:21.819009+0000 osd.4 (osd.4) 1680 2021-02-10T04:43:22.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:22.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[35961]: cluster 2021-02-10T04:43 2021-02-10T04:43:22.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[35961]: :22.346303+0000 mon.a (mon.0) 1972 : cluster 2021-02-10T04:43:22.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:22 smithi085 conmon[35961]: [DBG] osdmap e982: 8 total, 8 up, 7 in 2021-02-10T04:43:23.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:22 smithi164 conmon[33677]: debug 2021-02-10T04:43:22.837+0000 7fbec19a1700 -1 osd.4 982 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:23.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:22 smithi164 conmon[29288]: cluster 2021-02-10T04:43:21.818991+0000 osd.4 ( 2021-02-10T04:43:23.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:22 smithi164 conmon[29288]: osd.4) 1677 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:23.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:22 smithi164 conmon[29288]: cluster 2021-02-10T04:43:21.818997+0000 osd.4 (osd.4) 1678 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:23.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:22 smithi164 conmon[29288]: cluster 2021-02-10T04:43:21.819002 2021-02-10T04:43:23.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:22 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1679 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:23.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:22 smithi164 conmon[29288]: 2021-02-10T04:43:23.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:22 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:43:23.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:22 smithi164 conmon[29288]: 43:21.819009+0000 osd.4 (osd.4) 2021-02-10T04:43:23.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:22 smithi164 conmon[29288]: 1680 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:23.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:22 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:43:23.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:22 smithi164 conmon[29288]: 43:22.346303+0000 mon.a (mon.0) 2021-02-10T04:43:23.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:22 smithi164 conmon[29288]: 1972 : cluster [DBG] osdmap e982: 8 total, 8 up, 7 in 2021-02-10T04:43:23.320 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.4 dump_ops_in_flight 2021-02-10T04:43:23.332 INFO:tasks.rados.rados.0.smithi085.stdout:3955: finishing write tid 1 to smithi08575397-44 2021-02-10T04:43:23.332 INFO:tasks.rados.rados.0.smithi085.stdout:3955: finishing write tid 2 to smithi08575397-44 2021-02-10T04:43:23.332 INFO:tasks.rados.rados.0.smithi085.stdout:3955: finishing write tid 3 to smithi08575397-44 2021-02-10T04:43:23.332 INFO:tasks.rados.rados.0.smithi085.stdout:3955: finishing write tid 4 to smithi08575397-44 2021-02-10T04:43:23.332 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 1899 (ObjNum 1343 snap 458 seq_num 1343) dirty exists 2021-02-10T04:43:23.332 INFO:tasks.rados.rados.0.smithi085.stdout:3955: left oid 44 (ObjNum 1343 snap 458 seq_num 1343) 2021-02-10T04:43:23.332 INFO:tasks.rados.rados.0.smithi085.stdout:3955: done (5 left) 2021-02-10T04:43:23.333 INFO:tasks.rados.rados.0.smithi085.stdout:3956: done (4 left) 2021-02-10T04:43:23.333 INFO:tasks.rados.rados.0.smithi085.stdout:3957: snap_remove snap 382 2021-02-10T04:43:23.604 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":1600,"stamp":"2021-02-10T04:43:06.542809+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":951177134,"num_objects":746,"num_object_clones":339,"num_object_copies":2238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":746,"num_whiteouts":11,"num_read":9497,"num_read_kb":7430505,"num_write":6371,"num_write_kb":2021564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":431,"num_bytes_recovered":108949605,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15743,"ondisk_log_size":15743,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2185556,"kb_used_data":1869468,"kb_used_omap":0,"kb_used_meta":316032,"kb_avail":654030508,"statfs":{"total":671965249536,"available":669727240192,"internally_reserved":0,"allocated":1914335232,"data_stored":2634409908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":323616768},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4],"upper_bound":2097152},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16061703,"num_objects":7,"num_object_clones":7,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":82,"num_read_kb":110278,"num_write":27,"num_write_kb":8296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.010154"},"pg_stats":[{"pgid":"5.9","version":"131'3","reported_seq":"1026","reported_epoch":"962","state":"active+clean","last_fresh":"2021-02-10T04:43:02.894255+0000","last_change":"2021-02-10T04:33:57.871495+0000","last_active":"2021-02-10T04:43:02.894255+0000","last_peered":"2021-02-10T04:43:02.894255+0000","last_clean":"2021-02-10T04:43:02.894255+0000","last_became_active":"2021-02-10T04:29:07.145525+0000","last_became_peered":"2021-02-10T04:29:07.145525+0000","last_unstale":"2021-02-10T04:43:02.894255+0000","last_undegraded":"2021-02-10T04:43:02.894255+0000","last_fullsized":"2021-02-10T04:43:02.894255+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'3","last_scrub_stamp":"2021-02-10T04:33:57.871402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:33:57.871402+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"538'42","reported_seq":"1009","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.533127+0000","last_change":"2021-02-10T04:33:55.585872+0000","last_active":"2021-02-10T04:43:01.533127+0000","last_peered":"2021-02-10T04:43:01.533127+0000","last_clean":"2021-02-10T04:43:01.533127+0000","last_became_active":"2021-02-10T04:25:49.002058+0000","last_became_peered":"2021-02-10T04:25:49.002058+0000","last_unstale":"2021-02-10T04:43:01.533127+0000","last_undegraded":"2021-02-10T04:43:01.533127+0000","last_fullsized":"2021-02-10T04:43:01.533127+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2021-02-10T04:33:55.585807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:55.585807+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":77,"num_read_kb":60,"num_write":43,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"364","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.333579+0000","last_change":"2021-02-10T04:37:37.973234+0000","last_active":"2021-02-10T04:43:03.333579+0000","last_peered":"2021-02-10T04:43:03.333579+0000","last_clean":"2021-02-10T04:43:03.333579+0000","last_became_active":"2021-02-10T04:37:37.246903+0000","last_became_peered":"2021-02-10T04:37:37.246903+0000","last_unstale":"2021-02-10T04:43:03.333579+0000","last_undegraded":"2021-02-10T04:43:03.333579+0000","last_fullsized":"2021-02-10T04:43:03.333579+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:29:34.787265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:34.787265+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"921","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.534258+0000","last_change":"2021-02-10T04:29:19.405593+0000","last_active":"2021-02-10T04:43:01.534258+0000","last_peered":"2021-02-10T04:43:01.534258+0000","last_clean":"2021-02-10T04:43:01.534258+0000","last_became_active":"2021-02-10T04:25:53.148090+0000","last_became_peered":"2021-02-10T04:25:53.148090+0000","last_unstale":"2021-02-10T04:43:01.534258+0000","last_undegraded":"2021-02-10T04:43:01.534258+0000","last_fullsized":"2021-02-10T04:43:01.534258+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:19.405525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:29:19.405525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"962'1585","reported_seq":"4270","reported_epoch":"962","state":"active+clean+snaptrim_wait","last_fresh":"2021-02-10T04:43:02.540010+0000","last_change":"2021-02-10T04:43:01.679211+0000","last_active":"2021-02-10T04:43:02.540010+0000","last_peered":"2021-02-10T04:43:02.540010+0000","last_clean":"2021-02-10T04:43:02.540010+0000","last_became_active":"2021-02-10T04:33:05.144488+0000","last_became_peered":"2021-02-10T04:33:05.144488+0000","last_unstale":"2021-02-10T04:43:02.540010+0000","last_undegraded":"2021-02-10T04:43:02.540010+0000","last_fullsized":"2021-02-10T04:43:02.540010+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'301","last_scrub_stamp":"2021-02-10T04:33:56.571925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:56.571925+0000","log_size":1585,"ondisk_log_size":1585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"stat_sum":{"num_bytes":134655125,"num_objects":54,"num_object_clones":49,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":2,"num_read":752,"num_read_kb":802744,"num_write":544,"num_write_kb":209772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24f","length":"1"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"1"},{"start":"2a3","length":"1"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"3"},{"start":"2ad","length":"1"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2ca","length":"2"},{"start":"2d1","length":"7"},{"start":"2e0","length":"3"},{"start":"2e4","length":"1"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"}]},{"pgid":"4.b","version":"538'26","reported_seq":"1013","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.333623+0000","last_change":"2021-02-10T04:38:39.265973+0000","last_active":"2021-02-10T04:43:03.333623+0000","last_peered":"2021-02-10T04:43:03.333623+0000","last_clean":"2021-02-10T04:43:03.333623+0000","last_became_active":"2021-02-10T04:38:39.243413+0000","last_became_peered":"2021-02-10T04:38:39.243413+0000","last_unstale":"2021-02-10T04:43:03.333623+0000","last_undegraded":"2021-02-10T04:43:03.333623+0000","last_fullsized":"2021-02-10T04:43:03.333623+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'26","last_scrub_stamp":"2021-02-10T04:38:32.145511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:32.145511+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":53,"num_read_kb":36,"num_write":27,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":894,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"605","reported_epoch":"960","state":"active+clean","last_fresh":"2021-02-10T04:43:00.522250+0000","last_change":"2021-02-10T04:33:05.151251+0000","last_active":"2021-02-10T04:43:00.522250+0000","last_peered":"2021-02-10T04:43:00.522250+0000","last_clean":"2021-02-10T04:43:00.522250+0000","last_became_active":"2021-02-10T04:33:05.150882+0000","last_became_peered":"2021-02-10T04:33:05.150882+0000","last_unstale":"2021-02-10T04:43:00.522250+0000","last_undegraded":"2021-02-10T04:43:00.522250+0000","last_fullsized":"2021-02-10T04:43:00.522250+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:31.102558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:31.102558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"334","reported_epoch":"960","state":"active+clean","last_fresh":"2021-02-10T04:43:00.521969+0000","last_change":"2021-02-10T04:37:37.244470+0000","last_active":"2021-02-10T04:43:00.521969+0000","last_peered":"2021-02-10T04:43:00.521969+0000","last_clean":"2021-02-10T04:43:00.521969+0000","last_became_active":"2021-02-10T04:37:37.242485+0000","last_became_peered":"2021-02-10T04:37:37.242485+0000","last_unstale":"2021-02-10T04:43:00.521969+0000","last_undegraded":"2021-02-10T04:43:00.521969+0000","last_fullsized":"2021-02-10T04:43:00.521969+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:35.805326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.805326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"934","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.337170+0000","last_change":"2021-02-10T04:37:37.254421+0000","last_active":"2021-02-10T04:43:03.337170+0000","last_peered":"2021-02-10T04:43:03.337170+0000","last_clean":"2021-02-10T04:43:03.337170+0000","last_became_active":"2021-02-10T04:37:37.254038+0000","last_became_peered":"2021-02-10T04:37:37.254038+0000","last_unstale":"2021-02-10T04:43:03.337170+0000","last_undegraded":"2021-02-10T04:43:03.337170+0000","last_fullsized":"2021-02-10T04:43:03.337170+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:48.700418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:48.700418+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"259'74","reported_seq":"1770","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.526966+0000","last_change":"2021-02-10T04:43:01.527330+0000","last_active":"2021-02-10T04:43:01.526966+0000","last_peered":"2021-02-10T04:43:01.526966+0000","last_clean":"2021-02-10T04:43:01.526966+0000","last_became_active":"2021-02-10T04:37:37.246533+0000","last_became_peered":"2021-02-10T04:37:37.246533+0000","last_unstale":"2021-02-10T04:43:01.526966+0000","last_undegraded":"2021-02-10T04:43:01.526966+0000","last_fullsized":"2021-02-10T04:43:01.526966+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'62","last_scrub_stamp":"2021-02-10T04:30:10.905905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:10.905905+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2437048,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":48,"num_read_kb":56455,"num_write":38,"num_write_kb":14043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1134570,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24f","length":"1"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"1"},{"start":"2a3","length":"1"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"3"},{"start":"2ad","length":"1"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2ca","length":"2"},{"start":"2d1","length":"7"},{"start":"2e0","length":"3"},{"start":"2e4","length":"1"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"339","length":"1"},{"start":"33d","length":"1"},{"start":"340","length":"1"},{"start":"34f","length":"1"}]},{"pgid":"4.a","version":"538'45","reported_seq":"1006","reported_epoch":"962","state":"active+clean","last_fresh":"2021-02-10T04:43:02.533867+0000","last_change":"2021-02-10T04:29:26.671051+0000","last_active":"2021-02-10T04:43:02.533867+0000","last_peered":"2021-02-10T04:43:02.533867+0000","last_clean":"2021-02-10T04:43:02.533867+0000","last_became_active":"2021-02-10T04:25:49.015913+0000","last_became_peered":"2021-02-10T04:25:49.015913+0000","last_unstale":"2021-02-10T04:43:02.533867+0000","last_undegraded":"2021-02-10T04:43:02.533867+0000","last_fullsized":"2021-02-10T04:43:02.533867+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'29","last_scrub_stamp":"2021-02-10T04:29:26.670950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.670950+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":80,"num_read_kb":60,"num_write":46,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"642'3","reported_seq":"431","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.534595+0000","last_change":"2021-02-10T04:38:24.163263+0000","last_active":"2021-02-10T04:43:01.534595+0000","last_peered":"2021-02-10T04:43:01.534595+0000","last_clean":"2021-02-10T04:43:01.534595+0000","last_became_active":"2021-02-10T04:37:37.234230+0000","last_became_peered":"2021-02-10T04:37:37.234230+0000","last_unstale":"2021-02-10T04:43:01.534595+0000","last_undegraded":"2021-02-10T04:43:01.534595+0000","last_fullsized":"2021-02-10T04:43:01.534595+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'3","last_scrub_stamp":"2021-02-10T04:38:24.163200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:24.163200+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"342","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.333417+0000","last_change":"2021-02-10T04:38:31.121100+0000","last_active":"2021-02-10T04:43:03.333417+0000","last_peered":"2021-02-10T04:43:03.333417+0000","last_clean":"2021-02-10T04:43:03.333417+0000","last_became_active":"2021-02-10T04:37:37.230818+0000","last_became_peered":"2021-02-10T04:37:37.230818+0000","last_unstale":"2021-02-10T04:43:03.333417+0000","last_undegraded":"2021-02-10T04:43:03.333417+0000","last_fullsized":"2021-02-10T04:43:03.333417+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:31.121038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:38:31.121038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"340","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.332250+0000","last_change":"2021-02-10T04:37:37.245491+0000","last_active":"2021-02-10T04:43:03.332250+0000","last_peered":"2021-02-10T04:43:03.332250+0000","last_clean":"2021-02-10T04:43:03.332250+0000","last_became_active":"2021-02-10T04:37:37.245113+0000","last_became_peered":"2021-02-10T04:37:37.245113+0000","last_unstale":"2021-02-10T04:43:03.332250+0000","last_undegraded":"2021-02-10T04:43:03.332250+0000","last_fullsized":"2021-02-10T04:43:03.332250+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.731458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.731458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"963'681","reported_seq":"3464","reported_epoch":"963","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:43:03.773412+0000","last_change":"2021-02-10T04:43:03.771376+0000","last_active":"2021-02-10T04:43:03.773412+0000","last_peered":"2021-02-10T04:43:03.773412+0000","last_clean":"2021-02-10T04:43:03.773412+0000","last_became_active":"2021-02-10T04:28:25.150246+0000","last_became_peered":"2021-02-10T04:28:25.150246+0000","last_unstale":"2021-02-10T04:43:03.773412+0000","last_undegraded":"2021-02-10T04:43:03.773412+0000","last_fullsized":"2021-02-10T04:43:03.773412+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'63","last_scrub_stamp":"2021-02-10T04:29:39.700115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:29:39.700115+0000","log_size":681,"ondisk_log_size":681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"stat_sum":{"num_bytes":50559404,"num_objects":20,"num_object_clones":18,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":256,"num_read_kb":219086,"num_write":190,"num_write_kb":86478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24f","length":"1"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"1"},{"start":"2a3","length":"1"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"3"},{"start":"2ad","length":"1"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2ca","length":"2"},{"start":"2d1","length":"7"},{"start":"2e0","length":"3"},{"start":"2e4","length":"1"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"}]},{"pgid":"4.d","version":"538'47","reported_seq":"1028","reported_epoch":"960","state":"active+clean","last_fresh":"2021-02-10T04:43:00.522693+0000","last_change":"2021-02-10T04:37:37.821860+0000","last_active":"2021-02-10T04:43:00.522693+0000","last_peered":"2021-02-10T04:43:00.522693+0000","last_clean":"2021-02-10T04:43:00.522693+0000","last_became_active":"2021-02-10T04:37:37.237215+0000","last_became_peered":"2021-02-10T04:37:37.237215+0000","last_unstale":"2021-02-10T04:43:00.522693+0000","last_undegraded":"2021-02-10T04:43:00.522693+0000","last_fullsized":"2021-02-10T04:43:00.522693+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'33","last_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:28:49.514984+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":62,"num_read_kb":44,"num_write":36,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"842","reported_epoch":"960","state":"active+clean","last_fresh":"2021-02-10T04:43:00.522354+0000","last_change":"2021-02-10T04:29:07.146266+0000","last_active":"2021-02-10T04:43:00.522354+0000","last_peered":"2021-02-10T04:43:00.522354+0000","last_clean":"2021-02-10T04:43:00.522354+0000","last_became_active":"2021-02-10T04:29:07.145616+0000","last_became_peered":"2021-02-10T04:29:07.145616+0000","last_unstale":"2021-02-10T04:43:00.522354+0000","last_undegraded":"2021-02-10T04:43:00.522354+0000","last_fullsized":"2021-02-10T04:43:00.522354+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:44.410817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:44.410817+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"605","reported_epoch":"960","state":"active+clean","last_fresh":"2021-02-10T04:43:00.521956+0000","last_change":"2021-02-10T04:33:05.147166+0000","last_active":"2021-02-10T04:43:00.521956+0000","last_peered":"2021-02-10T04:43:00.521956+0000","last_clean":"2021-02-10T04:43:00.521956+0000","last_became_active":"2021-02-10T04:33:05.146783+0000","last_became_peered":"2021-02-10T04:33:05.146783+0000","last_unstale":"2021-02-10T04:43:00.521956+0000","last_undegraded":"2021-02-10T04:43:00.521956+0000","last_fullsized":"2021-02-10T04:43:00.521956+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:35.958255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.958255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"342","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.334117+0000","last_change":"2021-02-10T04:38:32.875574+0000","last_active":"2021-02-10T04:43:03.334117+0000","last_peered":"2021-02-10T04:43:03.334117+0000","last_clean":"2021-02-10T04:43:03.334117+0000","last_became_active":"2021-02-10T04:37:37.251479+0000","last_became_peered":"2021-02-10T04:37:37.251479+0000","last_unstale":"2021-02-10T04:43:03.334117+0000","last_undegraded":"2021-02-10T04:43:03.334117+0000","last_fullsized":"2021-02-10T04:43:03.334117+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:32.875498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:38:32.875498+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"962'575","reported_seq":"3988","reported_epoch":"962","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:43:02.565434+0000","last_change":"2021-02-10T04:43:01.679194+0000","last_active":"2021-02-10T04:43:02.565434+0000","last_peered":"2021-02-10T04:43:02.565434+0000","last_clean":"2021-02-10T04:43:02.565434+0000","last_became_active":"2021-02-10T04:37:40.267794+0000","last_became_peered":"2021-02-10T04:37:40.267794+0000","last_unstale":"2021-02-10T04:43:02.565434+0000","last_undegraded":"2021-02-10T04:43:02.565434+0000","last_fullsized":"2021-02-10T04:43:02.565434+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"688'187","last_scrub_stamp":"2021-02-10T04:38:27.225488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:38:27.225488+0000","log_size":575,"ondisk_log_size":575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":49687630,"num_objects":23,"num_object_clones":20,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":318,"num_read_kb":341491,"num_write":216,"num_write_kb":63707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":11919937,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24f","length":"1"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"1"},{"start":"2a3","length":"1"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"3"},{"start":"2ad","length":"1"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2ca","length":"2"},{"start":"2d1","length":"7"},{"start":"2e0","length":"3"},{"start":"2e4","length":"1"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"340","length":"1"},{"start":"34f","length":"1"}]},{"pgid":"1.8","version":"0'0","reported_seq":"873","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.533187+0000","last_change":"2021-02-10T04:37:37.246395+0000","last_active":"2021-02-10T04:43:01.533187+0000","last_peered":"2021-02-10T04:43:01.533187+0000","last_clean":"2021-02-10T04:43:01.533187+0000","last_became_active":"2021-02-10T04:37:37.246004+0000","last_became_peered":"2021-02-10T04:37:37.246004+0000","last_unstale":"2021-02-10T04:43:01.533187+0000","last_undegraded":"2021-02-10T04:43:01.533187+0000","last_fullsized":"2021-02-10T04:43:01.533187+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:37.336092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:37.336092+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"949","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.329615+0000","last_change":"2021-02-10T04:37:37.259938+0000","last_active":"2021-02-10T04:43:03.329615+0000","last_peered":"2021-02-10T04:43:03.329615+0000","last_clean":"2021-02-10T04:43:03.329615+0000","last_became_active":"2021-02-10T04:37:37.259532+0000","last_became_peered":"2021-02-10T04:37:37.259532+0000","last_unstale":"2021-02-10T04:43:03.329615+0000","last_undegraded":"2021-02-10T04:43:03.329615+0000","last_fullsized":"2021-02-10T04:43:03.329615+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:51.858386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:33:51.858386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"538'23","reported_seq":"960","reported_epoch":"960","state":"active+clean","last_fresh":"2021-02-10T04:43:00.522033+0000","last_change":"2021-02-10T04:28:41.427119+0000","last_active":"2021-02-10T04:43:00.522033+0000","last_peered":"2021-02-10T04:43:00.522033+0000","last_clean":"2021-02-10T04:43:00.522033+0000","last_became_active":"2021-02-10T04:25:49.001725+0000","last_became_peered":"2021-02-10T04:25:49.001725+0000","last_unstale":"2021-02-10T04:43:00.522033+0000","last_undegraded":"2021-02-10T04:43:00.522033+0000","last_fullsized":"2021-02-10T04:43:00.522033+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'13","last_scrub_stamp":"2021-02-10T04:28:41.427050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:41.427050+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":28,"num_write":23,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"612","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.331290+0000","last_change":"2021-02-10T04:33:05.168734+0000","last_active":"2021-02-10T04:43:03.331290+0000","last_peered":"2021-02-10T04:43:03.331290+0000","last_clean":"2021-02-10T04:43:03.331290+0000","last_became_active":"2021-02-10T04:33:05.168314+0000","last_became_peered":"2021-02-10T04:33:05.168314+0000","last_unstale":"2021-02-10T04:43:03.331290+0000","last_undegraded":"2021-02-10T04:43:03.331290+0000","last_fullsized":"2021-02-10T04:43:03.331290+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:18.107867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:18.107867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"639'3","reported_seq":"419","reported_epoch":"962","state":"active+clean","last_fresh":"2021-02-10T04:43:02.533803+0000","last_change":"2021-02-10T04:38:31.211769+0000","last_active":"2021-02-10T04:43:02.533803+0000","last_peered":"2021-02-10T04:43:02.533803+0000","last_clean":"2021-02-10T04:43:02.533803+0000","last_became_active":"2021-02-10T04:37:37.233738+0000","last_became_peered":"2021-02-10T04:37:37.233738+0000","last_unstale":"2021-02-10T04:43:02.533803+0000","last_undegraded":"2021-02-10T04:43:02.533803+0000","last_fullsized":"2021-02-10T04:43:02.533803+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"639'3","last_scrub_stamp":"2021-02-10T04:38:31.211662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:38:31.211662+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"929","reported_epoch":"960","state":"active+clean","last_fresh":"2021-02-10T04:43:00.522662+0000","last_change":"2021-02-10T04:37:37.236009+0000","last_active":"2021-02-10T04:43:00.522662+0000","last_peered":"2021-02-10T04:43:00.522662+0000","last_clean":"2021-02-10T04:43:00.522662+0000","last_became_active":"2021-02-10T04:37:37.233109+0000","last_became_peered":"2021-02-10T04:37:37.233109+0000","last_unstale":"2021-02-10T04:43:00.522662+0000","last_undegraded":"2021-02-10T04:43:00.522662+0000","last_fullsized":"2021-02-10T04:43:00.522662+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:38.873102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.873102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"964'1031","reported_seq":"5981","reported_epoch":"964","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:43:04.537892+0000","last_change":"2021-02-10T04:43:04.532918+0000","last_active":"2021-02-10T04:43:04.537892+0000","last_peered":"2021-02-10T04:43:04.537892+0000","last_clean":"2021-02-10T04:43:04.537892+0000","last_became_active":"2021-02-10T04:28:25.149226+0000","last_became_peered":"2021-02-10T04:28:25.149226+0000","last_unstale":"2021-02-10T04:43:04.537892+0000","last_undegraded":"2021-02-10T04:43:04.537892+0000","last_fullsized":"2021-02-10T04:43:04.537892+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'236","last_scrub_stamp":"2021-02-10T04:33:50.351534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:50.351534+0000","log_size":1031,"ondisk_log_size":1031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":17,"stat_sum":{"num_bytes":85923262,"num_objects":39,"num_object_clones":35,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":3,"num_read":553,"num_read_kb":622114,"num_write":361,"num_write_kb":124233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24f","length":"1"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"1"},{"start":"2a3","length":"1"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"3"},{"start":"2ad","length":"1"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2ca","length":"2"},{"start":"2d1","length":"7"},{"start":"2e0","length":"3"},{"start":"2e4","length":"1"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"}]},{"pgid":"1.a","version":"0'0","reported_seq":"873","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.534023+0000","last_change":"2021-02-10T04:37:37.232619+0000","last_active":"2021-02-10T04:43:01.534023+0000","last_peered":"2021-02-10T04:43:01.534023+0000","last_clean":"2021-02-10T04:43:01.534023+0000","last_became_active":"2021-02-10T04:37:37.229648+0000","last_became_peered":"2021-02-10T04:37:37.229648+0000","last_unstale":"2021-02-10T04:43:01.534023+0000","last_undegraded":"2021-02-10T04:43:01.534023+0000","last_fullsized":"2021-02-10T04:43:01.534023+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.384170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.384170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"538'29","reported_seq":"668","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.333871+0000","last_change":"2021-02-10T04:33:05.154998+0000","last_active":"2021-02-10T04:43:03.333871+0000","last_peered":"2021-02-10T04:43:03.333871+0000","last_clean":"2021-02-10T04:43:03.333871+0000","last_became_active":"2021-02-10T04:33:05.149026+0000","last_became_peered":"2021-02-10T04:33:05.149026+0000","last_unstale":"2021-02-10T04:43:03.333871+0000","last_undegraded":"2021-02-10T04:43:03.333871+0000","last_fullsized":"2021-02-10T04:43:03.333871+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'17","last_scrub_stamp":"2021-02-10T04:29:37.933206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:37.933206+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":49,"num_read_kb":36,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"672","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.330179+0000","last_change":"2021-02-10T04:33:05.169654+0000","last_active":"2021-02-10T04:43:03.330179+0000","last_peered":"2021-02-10T04:43:03.330179+0000","last_clean":"2021-02-10T04:43:03.330179+0000","last_became_active":"2021-02-10T04:33:05.169289+0000","last_became_peered":"2021-02-10T04:33:05.169289+0000","last_unstale":"2021-02-10T04:43:03.330179+0000","last_undegraded":"2021-02-10T04:43:03.330179+0000","last_fullsized":"2021-02-10T04:43:03.330179+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_clean_scrub_stamp":"2021-02-10T04:28:39.128781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"78'3","reported_seq":"1131","reported_epoch":"960","state":"active+clean","last_fresh":"2021-02-10T04:43:00.522641+0000","last_change":"2021-02-10T04:28:43.509446+0000","last_active":"2021-02-10T04:43:00.522641+0000","last_peered":"2021-02-10T04:43:00.522641+0000","last_clean":"2021-02-10T04:43:00.522641+0000","last_became_active":"2021-02-10T04:25:51.025218+0000","last_became_peered":"2021-02-10T04:25:51.025218+0000","last_unstale":"2021-02-10T04:43:00.522641+0000","last_undegraded":"2021-02-10T04:43:00.522641+0000","last_fullsized":"2021-02-10T04:43:00.522641+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:28:43.509353+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:28:43.509353+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"921","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.535023+0000","last_change":"2021-02-10T04:28:45.481538+0000","last_active":"2021-02-10T04:43:01.535023+0000","last_peered":"2021-02-10T04:43:01.535023+0000","last_clean":"2021-02-10T04:43:01.535023+0000","last_became_active":"2021-02-10T04:25:53.042697+0000","last_became_peered":"2021-02-10T04:25:53.042697+0000","last_unstale":"2021-02-10T04:43:01.535023+0000","last_undegraded":"2021-02-10T04:43:01.535023+0000","last_fullsized":"2021-02-10T04:43:01.535023+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:45.481472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:28:45.481472+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"961'1032","reported_seq":"4981","reported_epoch":"963","state":"active+clean+snaptrim_wait","last_fresh":"2021-02-10T04:43:03.741287+0000","last_change":"2021-02-10T04:43:03.741340+0000","last_active":"2021-02-10T04:43:03.741287+0000","last_peered":"2021-02-10T04:43:03.741287+0000","last_clean":"2021-02-10T04:43:03.741287+0000","last_became_active":"2021-02-10T04:28:25.148500+0000","last_became_peered":"2021-02-10T04:28:25.148500+0000","last_unstale":"2021-02-10T04:43:03.741287+0000","last_undegraded":"2021-02-10T04:43:03.741287+0000","last_fullsized":"2021-02-10T04:43:03.741287+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'110","last_scrub_stamp":"2021-02-10T04:30:18.463782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:18.463782+0000","log_size":1032,"ondisk_log_size":1032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":49464488,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":490,"num_read_kb":552336,"num_write":348,"num_write_kb":127967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24f","length":"1"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"1"},{"start":"2a3","length":"1"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"3"},{"start":"2ad","length":"1"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2ca","length":"2"},{"start":"2d1","length":"7"},{"start":"2e0","length":"3"},{"start":"2e4","length":"1"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"340","length":"1"},{"start":"34f","length":"1"}]},{"pgid":"4.e","version":"538'39","reported_seq":"988","reported_epoch":"960","state":"active+clean","last_fresh":"2021-02-10T04:43:00.522500+0000","last_change":"2021-02-10T04:28:11.693702+0000","last_active":"2021-02-10T04:43:00.522500+0000","last_peered":"2021-02-10T04:43:00.522500+0000","last_clean":"2021-02-10T04:43:00.522500+0000","last_became_active":"2021-02-10T04:25:49.011093+0000","last_became_peered":"2021-02-10T04:25:49.011093+0000","last_unstale":"2021-02-10T04:43:00.522500+0000","last_undegraded":"2021-02-10T04:43:00.522500+0000","last_fullsized":"2021-02-10T04:43:00.522500+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'31","last_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:28:11.693616+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":56,"num_read_kb":40,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"908","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.326991+0000","last_change":"2021-02-10T04:29:31.683314+0000","last_active":"2021-02-10T04:43:03.326991+0000","last_peered":"2021-02-10T04:43:03.326991+0000","last_clean":"2021-02-10T04:43:03.326991+0000","last_became_active":"2021-02-10T04:29:07.160593+0000","last_became_peered":"2021-02-10T04:29:07.160593+0000","last_unstale":"2021-02-10T04:43:03.326991+0000","last_undegraded":"2021-02-10T04:43:03.326991+0000","last_fullsized":"2021-02-10T04:43:03.326991+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:31.683213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.683213+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"928","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.535580+0000","last_change":"2021-02-10T04:30:10.487631+0000","last_active":"2021-02-10T04:43:01.535580+0000","last_peered":"2021-02-10T04:43:01.535580+0000","last_clean":"2021-02-10T04:43:01.535580+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:43:01.535580+0000","last_undegraded":"2021-02-10T04:43:01.535580+0000","last_fullsized":"2021-02-10T04:43:01.535580+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:10.487528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:30:10.487528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"922","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.332678+0000","last_change":"2021-02-10T04:33:55.849675+0000","last_active":"2021-02-10T04:43:03.332678+0000","last_peered":"2021-02-10T04:43:03.332678+0000","last_clean":"2021-02-10T04:43:03.332678+0000","last_became_active":"2021-02-10T04:25:53.148054+0000","last_became_peered":"2021-02-10T04:25:53.148054+0000","last_unstale":"2021-02-10T04:43:03.332678+0000","last_undegraded":"2021-02-10T04:43:03.332678+0000","last_fullsized":"2021-02-10T04:43:03.332678+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:55.849582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:33:55.849582+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"963'1115","reported_seq":"3965","reported_epoch":"963","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:43:03.745329+0000","last_change":"2021-02-10T04:43:03.741401+0000","last_active":"2021-02-10T04:43:03.745329+0000","last_peered":"2021-02-10T04:43:03.745329+0000","last_clean":"2021-02-10T04:43:03.745329+0000","last_became_active":"2021-02-10T04:28:25.152030+0000","last_became_peered":"2021-02-10T04:28:25.152030+0000","last_unstale":"2021-02-10T04:43:03.745329+0000","last_undegraded":"2021-02-10T04:43:03.745329+0000","last_fullsized":"2021-02-10T04:43:03.745329+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'283","last_scrub_stamp":"2021-02-10T04:33:53.834627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:53.834627+0000","log_size":1115,"ondisk_log_size":1115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":77167139,"num_objects":31,"num_object_clones":27,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":1,"num_read":562,"num_read_kb":589609,"num_write":396,"num_write_kb":177445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24f","length":"1"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"1"},{"start":"2a3","length":"1"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"3"},{"start":"2ad","length":"1"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2ca","length":"2"},{"start":"2d1","length":"7"},{"start":"2e0","length":"3"},{"start":"2e4","length":"1"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"340","length":"1"},{"start":"34f","length":"1"}]},{"pgid":"5.0","version":"158'3","reported_seq":"1134","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.332164+0000","last_change":"2021-02-10T04:29:34.388978+0000","last_active":"2021-02-10T04:43:03.332164+0000","last_peered":"2021-02-10T04:43:03.332164+0000","last_clean":"2021-02-10T04:43:03.332164+0000","last_became_active":"2021-02-10T04:25:51.031906+0000","last_became_peered":"2021-02-10T04:25:51.031906+0000","last_unstale":"2021-02-10T04:43:03.332164+0000","last_undegraded":"2021-02-10T04:43:03.332164+0000","last_fullsized":"2021-02-10T04:43:03.332164+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'3","last_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_deep_scrub":"158'3","last_deep_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_clean_scrub_stamp":"2021-02-10T04:29:34.388912+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"538'40","reported_seq":"993","reported_epoch":"960","state":"active+clean","last_fresh":"2021-02-10T04:43:00.521560+0000","last_change":"2021-02-10T04:29:36.813190+0000","last_active":"2021-02-10T04:43:00.521560+0000","last_peered":"2021-02-10T04:43:00.521560+0000","last_clean":"2021-02-10T04:43:00.521560+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:43:00.521560+0000","last_undegraded":"2021-02-10T04:43:00.521560+0000","last_fullsized":"2021-02-10T04:43:00.521560+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:29:36.813083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.813083+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":72,"num_read_kb":52,"num_write":40,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"974","reported_epoch":"962","state":"active+clean","last_fresh":"2021-02-10T04:43:02.535136+0000","last_change":"2021-02-10T04:37:37.245911+0000","last_active":"2021-02-10T04:43:02.535136+0000","last_peered":"2021-02-10T04:43:02.535136+0000","last_clean":"2021-02-10T04:43:02.535136+0000","last_became_active":"2021-02-10T04:37:37.245391+0000","last_became_peered":"2021-02-10T04:37:37.245391+0000","last_unstale":"2021-02-10T04:43:02.535136+0000","last_undegraded":"2021-02-10T04:43:02.535136+0000","last_fullsized":"2021-02-10T04:43:02.535136+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.632085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.632085+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"858","reported_epoch":"962","state":"active+clean","last_fresh":"2021-02-10T04:43:02.534515+0000","last_change":"2021-02-10T04:37:37.235415+0000","last_active":"2021-02-10T04:43:02.534515+0000","last_peered":"2021-02-10T04:43:02.534515+0000","last_clean":"2021-02-10T04:43:02.534515+0000","last_became_active":"2021-02-10T04:37:37.235082+0000","last_became_peered":"2021-02-10T04:37:37.235082+0000","last_unstale":"2021-02-10T04:43:02.534515+0000","last_undegraded":"2021-02-10T04:43:02.534515+0000","last_fullsized":"2021-02-10T04:43:02.534515+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.658374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.658374+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"963'965","reported_seq":"4325","reported_epoch":"963","state":"active+clean+snaptrim_wait","last_fresh":"2021-02-10T04:43:03.524135+0000","last_change":"2021-02-10T04:43:03.524135+0000","last_active":"2021-02-10T04:43:03.524135+0000","last_peered":"2021-02-10T04:43:03.524135+0000","last_clean":"2021-02-10T04:43:03.524135+0000","last_became_active":"2021-02-10T04:37:45.352779+0000","last_became_peered":"2021-02-10T04:37:45.352779+0000","last_unstale":"2021-02-10T04:43:03.524135+0000","last_undegraded":"2021-02-10T04:43:03.524135+0000","last_fullsized":"2021-02-10T04:43:03.524135+0000","mapping_epoch":646,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":647,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":965,"ondisk_log_size":965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":57686931,"num_objects":22,"num_object_clones":20,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":466,"num_read_kb":483730,"num_write":307,"num_write_kb":100208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":16580116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24f","length":"1"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"1"},{"start":"2a3","length":"1"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"3"},{"start":"2ad","length":"1"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2ca","length":"2"},{"start":"2d1","length":"7"},{"start":"2e0","length":"3"},{"start":"2e4","length":"1"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"33d","length":"1"},{"start":"340","length":"1"},{"start":"34f","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":"866","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.332756+0000","last_change":"2021-02-10T04:30:12.452072+0000","last_active":"2021-02-10T04:43:03.332756+0000","last_peered":"2021-02-10T04:43:03.332756+0000","last_clean":"2021-02-10T04:43:03.332756+0000","last_became_active":"2021-02-10T04:28:49.048525+0000","last_became_peered":"2021-02-10T04:28:49.048525+0000","last_unstale":"2021-02-10T04:43:03.332756+0000","last_undegraded":"2021-02-10T04:43:03.332756+0000","last_fullsized":"2021-02-10T04:43:03.332756+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:12.452000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:30:12.452000+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"922","reported_epoch":"960","state":"active+clean","last_fresh":"2021-02-10T04:43:00.521868+0000","last_change":"2021-02-10T04:29:40.859352+0000","last_active":"2021-02-10T04:43:00.521868+0000","last_peered":"2021-02-10T04:43:00.521868+0000","last_clean":"2021-02-10T04:43:00.521868+0000","last_became_active":"2021-02-10T04:25:51.018706+0000","last_became_peered":"2021-02-10T04:25:51.018706+0000","last_unstale":"2021-02-10T04:43:00.521868+0000","last_undegraded":"2021-02-10T04:43:00.521868+0000","last_fullsized":"2021-02-10T04:43:00.521868+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:40.859274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:40.859274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"538'60","reported_seq":"1028","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.332070+0000","last_change":"2021-02-10T04:29:28.398522+0000","last_active":"2021-02-10T04:43:03.332070+0000","last_peered":"2021-02-10T04:43:03.332070+0000","last_clean":"2021-02-10T04:43:03.332070+0000","last_became_active":"2021-02-10T04:25:48.997481+0000","last_became_peered":"2021-02-10T04:25:48.997481+0000","last_unstale":"2021-02-10T04:43:03.332070+0000","last_undegraded":"2021-02-10T04:43:03.332070+0000","last_fullsized":"2021-02-10T04:43:03.332070+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'48","last_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_deep_scrub":"152'48","last_deep_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.398429+0000","log_size":60,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":88,"num_read_kb":66,"num_write":48,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"620","reported_epoch":"962","state":"active+clean","last_fresh":"2021-02-10T04:43:02.534525+0000","last_change":"2021-02-10T04:33:05.142730+0000","last_active":"2021-02-10T04:43:02.534525+0000","last_peered":"2021-02-10T04:43:02.534525+0000","last_clean":"2021-02-10T04:43:02.534525+0000","last_became_active":"2021-02-10T04:33:05.141066+0000","last_became_peered":"2021-02-10T04:33:05.141066+0000","last_unstale":"2021-02-10T04:43:02.534525+0000","last_undegraded":"2021-02-10T04:43:02.534525+0000","last_fullsized":"2021-02-10T04:43:02.534525+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:28:35.010709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:35.010709+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":423,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"929","reported_epoch":"960","state":"active+clean","last_fresh":"2021-02-10T04:43:00.522049+0000","last_change":"2021-02-10T04:37:37.245330+0000","last_active":"2021-02-10T04:43:00.522049+0000","last_peered":"2021-02-10T04:43:00.522049+0000","last_clean":"2021-02-10T04:43:00.522049+0000","last_became_active":"2021-02-10T04:37:37.244859+0000","last_became_peered":"2021-02-10T04:37:37.244859+0000","last_unstale":"2021-02-10T04:43:00.522049+0000","last_undegraded":"2021-02-10T04:43:00.522049+0000","last_fullsized":"2021-02-10T04:43:00.522049+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:05.641577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:05.641577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"963'1570","reported_seq":"4686","reported_epoch":"963","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:43:03.529910+0000","last_change":"2021-02-10T04:43:03.524179+0000","last_active":"2021-02-10T04:43:03.529910+0000","last_peered":"2021-02-10T04:43:03.529910+0000","last_clean":"2021-02-10T04:43:03.529910+0000","last_became_active":"2021-02-10T04:37:45.352022+0000","last_became_peered":"2021-02-10T04:37:45.352022+0000","last_unstale":"2021-02-10T04:43:03.529910+0000","last_undegraded":"2021-02-10T04:43:03.529910+0000","last_fullsized":"2021-02-10T04:43:03.529910+0000","mapping_epoch":646,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":647,"parent":"0.0","parent_split_bits":0,"last_scrub":"687'1095","last_scrub_stamp":"2021-02-10T04:38:27.895743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:38:27.895743+0000","log_size":1570,"ondisk_log_size":1570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":78797795,"num_objects":31,"num_object_clones":27,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":1,"num_read":816,"num_read_kb":850458,"num_write":615,"num_write_kb":227546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":35873876,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24f","length":"1"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"1"},{"start":"2a3","length":"1"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"3"},{"start":"2ad","length":"1"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2ca","length":"2"},{"start":"2d1","length":"7"},{"start":"2e0","length":"3"},{"start":"2e4","length":"1"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"340","length":"1"},{"start":"34f","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":"872","reported_epoch":"960","state":"active+clean","last_fresh":"2021-02-10T04:43:00.521916+0000","last_change":"2021-02-10T04:37:37.233535+0000","last_active":"2021-02-10T04:43:00.521916+0000","last_peered":"2021-02-10T04:43:00.521916+0000","last_clean":"2021-02-10T04:43:00.521916+0000","last_became_active":"2021-02-10T04:37:37.233158+0000","last_became_peered":"2021-02-10T04:37:37.233158+0000","last_unstale":"2021-02-10T04:43:00.521916+0000","last_undegraded":"2021-02-10T04:43:00.521916+0000","last_fullsized":"2021-02-10T04:43:00.521916+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:33.839335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:33.839335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"995","reported_epoch":"962","state":"active+clean","last_fresh":"2021-02-10T04:43:02.534080+0000","last_change":"2021-02-10T04:33:52.827328+0000","last_active":"2021-02-10T04:43:02.534080+0000","last_peered":"2021-02-10T04:43:02.534080+0000","last_clean":"2021-02-10T04:43:02.534080+0000","last_became_active":"2021-02-10T04:25:51.025636+0000","last_became_peered":"2021-02-10T04:25:51.025636+0000","last_unstale":"2021-02-10T04:43:02.534080+0000","last_undegraded":"2021-02-10T04:43:02.534080+0000","last_fullsized":"2021-02-10T04:43:02.534080+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:52.827257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:33:52.827257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"538'37","reported_seq":"1002","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.333313+0000","last_change":"2021-02-10T04:38:36.138126+0000","last_active":"2021-02-10T04:43:03.333313+0000","last_peered":"2021-02-10T04:43:03.333313+0000","last_clean":"2021-02-10T04:43:03.333313+0000","last_became_active":"2021-02-10T04:25:49.002937+0000","last_became_peered":"2021-02-10T04:25:49.002937+0000","last_unstale":"2021-02-10T04:43:03.333313+0000","last_undegraded":"2021-02-10T04:43:03.333313+0000","last_fullsized":"2021-02-10T04:43:03.333313+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'37","last_scrub_stamp":"2021-02-10T04:38:36.138022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:36.138022+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":72,"num_read_kb":53,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"861","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.338175+0000","last_change":"2021-02-10T04:29:35.711635+0000","last_active":"2021-02-10T04:43:03.338175+0000","last_peered":"2021-02-10T04:43:03.338175+0000","last_clean":"2021-02-10T04:43:03.338175+0000","last_became_active":"2021-02-10T04:29:07.161341+0000","last_became_peered":"2021-02-10T04:29:07.161341+0000","last_unstale":"2021-02-10T04:43:03.338175+0000","last_undegraded":"2021-02-10T04:43:03.338175+0000","last_fullsized":"2021-02-10T04:43:03.338175+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_deep_scrub":"53'2","last_deep_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.711557+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"621","reported_epoch":"962","state":"active+clean","last_fresh":"2021-02-10T04:43:02.533058+0000","last_change":"2021-02-10T04:38:40.414663+0000","last_active":"2021-02-10T04:43:02.533058+0000","last_peered":"2021-02-10T04:43:02.533058+0000","last_clean":"2021-02-10T04:43:02.533058+0000","last_became_active":"2021-02-10T04:37:37.234868+0000","last_became_peered":"2021-02-10T04:37:37.234868+0000","last_unstale":"2021-02-10T04:43:02.533058+0000","last_undegraded":"2021-02-10T04:43:02.533058+0000","last_fullsized":"2021-02-10T04:43:02.533058+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:40.414601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:38:40.414601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"960'1019","reported_seq":"4322","reported_epoch":"961","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:43:01.533580+0000","last_change":"2021-02-10T04:42:59.504503+0000","last_active":"2021-02-10T04:43:01.533580+0000","last_peered":"2021-02-10T04:43:01.533580+0000","last_clean":"2021-02-10T04:43:01.533580+0000","last_became_active":"2021-02-10T04:28:25.146766+0000","last_became_peered":"2021-02-10T04:28:25.146766+0000","last_unstale":"2021-02-10T04:43:01.533580+0000","last_undegraded":"2021-02-10T04:43:01.533580+0000","last_fullsized":"2021-02-10T04:43:01.533580+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'357","last_scrub_stamp":"2021-02-10T04:33:50.582002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:50.582002+0000","log_size":1019,"ondisk_log_size":1019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":48002209,"num_objects":19,"num_object_clones":16,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":464,"num_read_kb":429782,"num_write":364,"num_write_kb":150848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24f","length":"1"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"1"},{"start":"2a3","length":"1"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"3"},{"start":"2ad","length":"1"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2ca","length":"2"},{"start":"2d1","length":"7"},{"start":"2e0","length":"3"},{"start":"2e4","length":"1"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"340","length":"1"},{"start":"34f","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":"946","reported_epoch":"962","state":"active+clean","last_fresh":"2021-02-10T04:43:02.534794+0000","last_change":"2021-02-10T04:29:38.793837+0000","last_active":"2021-02-10T04:43:02.534794+0000","last_peered":"2021-02-10T04:43:02.534794+0000","last_clean":"2021-02-10T04:43:02.534794+0000","last_became_active":"2021-02-10T04:28:49.054638+0000","last_became_peered":"2021-02-10T04:28:49.054638+0000","last_unstale":"2021-02-10T04:43:02.534794+0000","last_undegraded":"2021-02-10T04:43:02.534794+0000","last_fullsized":"2021-02-10T04:43:02.534794+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:38.793760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.793760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"923","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.535652+0000","last_change":"2021-02-10T04:29:40.451948+0000","last_active":"2021-02-10T04:43:01.535652+0000","last_peered":"2021-02-10T04:43:01.535652+0000","last_clean":"2021-02-10T04:43:01.535652+0000","last_became_active":"2021-02-10T04:25:51.032201+0000","last_became_peered":"2021-02-10T04:25:51.032201+0000","last_unstale":"2021-02-10T04:43:01.535652+0000","last_undegraded":"2021-02-10T04:43:01.535652+0000","last_fullsized":"2021-02-10T04:43:01.535652+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:40.451832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:29:40.451832+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"538'28","reported_seq":"1007","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.338380+0000","last_change":"2021-02-10T04:37:38.566112+0000","last_active":"2021-02-10T04:43:03.338380+0000","last_peered":"2021-02-10T04:43:03.338380+0000","last_clean":"2021-02-10T04:43:03.338380+0000","last_became_active":"2021-02-10T04:37:37.247141+0000","last_became_peered":"2021-02-10T04:37:37.247141+0000","last_unstale":"2021-02-10T04:43:03.338380+0000","last_undegraded":"2021-02-10T04:43:03.338380+0000","last_fullsized":"2021-02-10T04:43:03.338380+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'16","last_scrub_stamp":"2021-02-10T04:33:55.288033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:55.288033+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":34,"num_write":28,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"664","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.328235+0000","last_change":"2021-02-10T04:33:05.166788+0000","last_active":"2021-02-10T04:43:03.328235+0000","last_peered":"2021-02-10T04:43:03.328235+0000","last_clean":"2021-02-10T04:43:03.328235+0000","last_became_active":"2021-02-10T04:33:05.166368+0000","last_became_peered":"2021-02-10T04:33:05.166368+0000","last_unstale":"2021-02-10T04:43:03.328235+0000","last_undegraded":"2021-02-10T04:43:03.328235+0000","last_fullsized":"2021-02-10T04:43:03.328235+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:14.204533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:30:14.204533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"927","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.336376+0000","last_change":"2021-02-10T04:29:21.620573+0000","last_active":"2021-02-10T04:43:03.336376+0000","last_peered":"2021-02-10T04:43:03.336376+0000","last_clean":"2021-02-10T04:43:03.336376+0000","last_became_active":"2021-02-10T04:25:49.001782+0000","last_became_peered":"2021-02-10T04:25:49.001782+0000","last_unstale":"2021-02-10T04:43:03.336376+0000","last_undegraded":"2021-02-10T04:43:03.336376+0000","last_fullsized":"2021-02-10T04:43:03.336376+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:21.620482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:21.620482+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"606","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.534505+0000","last_change":"2021-02-10T04:33:05.153205+0000","last_active":"2021-02-10T04:43:01.534505+0000","last_peered":"2021-02-10T04:43:01.534505+0000","last_clean":"2021-02-10T04:43:01.534505+0000","last_became_active":"2021-02-10T04:33:05.152845+0000","last_became_peered":"2021-02-10T04:33:05.152845+0000","last_unstale":"2021-02-10T04:43:01.534505+0000","last_undegraded":"2021-02-10T04:43:01.534505+0000","last_fullsized":"2021-02-10T04:43:01.534505+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.098681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.098681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.5","version":"964'414","reported_seq":"1765","reported_epoch":"964","state":"active+clean+snaptrim_wait","last_fresh":"2021-02-10T04:43:04.532119+0000","last_change":"2021-02-10T04:43:04.532778+0000","last_active":"2021-02-10T04:43:04.532119+0000","last_peered":"2021-02-10T04:43:04.532119+0000","last_clean":"2021-02-10T04:43:04.532119+0000","last_became_active":"2021-02-10T04:37:37.247692+0000","last_became_peered":"2021-02-10T04:37:37.247692+0000","last_unstale":"2021-02-10T04:43:04.532119+0000","last_undegraded":"2021-02-10T04:43:04.532119+0000","last_fullsized":"2021-02-10T04:43:04.532119+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"683'135","last_scrub_stamp":"2021-02-10T04:38:23.183462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:38:23.183462+0000","log_size":414,"ondisk_log_size":414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":44539458,"num_objects":19,"num_object_clones":17,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":157,"num_read_kb":136870,"num_write":141,"num_write_kb":47366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":807386,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24f","length":"1"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"1"},{"start":"2a3","length":"1"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"3"},{"start":"2ad","length":"1"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2ca","length":"2"},{"start":"2d1","length":"7"},{"start":"2e0","length":"3"},{"start":"2e4","length":"1"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"339","length":"1"},{"start":"33d","length":"1"},{"start":"340","length":"1"},{"start":"34a","length":"1"},{"start":"34f","length":"1"}]},{"pgid":"1.3","version":"0'0","reported_seq":"686","reported_epoch":"962","state":"active+clean","last_fresh":"2021-02-10T04:43:02.534157+0000","last_change":"2021-02-10T04:33:05.141311+0000","last_active":"2021-02-10T04:43:02.534157+0000","last_peered":"2021-02-10T04:43:02.534157+0000","last_clean":"2021-02-10T04:43:02.534157+0000","last_became_active":"2021-02-10T04:33:05.140966+0000","last_became_peered":"2021-02-10T04:33:05.140966+0000","last_unstale":"2021-02-10T04:43:02.534157+0000","last_undegraded":"2021-02-10T04:43:02.534157+0000","last_fullsized":"2021-02-10T04:43:02.534157+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.928906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.928906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"338","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.534779+0000","last_change":"2021-02-10T04:37:37.235036+0000","last_active":"2021-02-10T04:43:01.534779+0000","last_peered":"2021-02-10T04:43:01.534779+0000","last_clean":"2021-02-10T04:43:01.534779+0000","last_became_active":"2021-02-10T04:37:37.234644+0000","last_became_peered":"2021-02-10T04:37:37.234644+0000","last_unstale":"2021-02-10T04:43:01.534779+0000","last_undegraded":"2021-02-10T04:43:01.534779+0000","last_fullsized":"2021-02-10T04:43:01.534779+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.805432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.805432+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"538'27","reported_seq":"973","reported_epoch":"962","state":"active+clean","last_fresh":"2021-02-10T04:43:02.535667+0000","last_change":"2021-02-10T04:33:56.837896+0000","last_active":"2021-02-10T04:43:02.535667+0000","last_peered":"2021-02-10T04:43:02.535667+0000","last_clean":"2021-02-10T04:43:02.535667+0000","last_became_active":"2021-02-10T04:25:49.002471+0000","last_became_peered":"2021-02-10T04:25:49.002471+0000","last_unstale":"2021-02-10T04:43:02.535667+0000","last_undegraded":"2021-02-10T04:43:02.535667+0000","last_fullsized":"2021-02-10T04:43:02.535667+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:33:56.837816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:56.837816+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"930","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.327073+0000","last_change":"2021-02-10T04:29:36.663903+0000","last_active":"2021-02-10T04:43:03.327073+0000","last_peered":"2021-02-10T04:43:03.327073+0000","last_clean":"2021-02-10T04:43:03.327073+0000","last_became_active":"2021-02-10T04:29:07.158075+0000","last_became_peered":"2021-02-10T04:29:07.158075+0000","last_unstale":"2021-02-10T04:43:03.327073+0000","last_undegraded":"2021-02-10T04:43:03.327073+0000","last_fullsized":"2021-02-10T04:43:03.327073+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.663840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.663840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"1063","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.534254+0000","last_change":"2021-02-10T04:42:23.062451+0000","last_active":"2021-02-10T04:43:01.534254+0000","last_peered":"2021-02-10T04:43:01.534254+0000","last_clean":"2021-02-10T04:43:01.534254+0000","last_became_active":"2021-02-10T04:37:37.245931+0000","last_became_peered":"2021-02-10T04:37:37.245931+0000","last_unstale":"2021-02-10T04:43:01.534254+0000","last_undegraded":"2021-02-10T04:43:01.534254+0000","last_fullsized":"2021-02-10T04:43:01.534254+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:34.482906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:34.482906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"926","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.331362+0000","last_change":"2021-02-10T04:29:35.382459+0000","last_active":"2021-02-10T04:43:03.331362+0000","last_peered":"2021-02-10T04:43:03.331362+0000","last_clean":"2021-02-10T04:43:03.331362+0000","last_became_active":"2021-02-10T04:25:49.010614+0000","last_became_peered":"2021-02-10T04:25:49.010614+0000","last_unstale":"2021-02-10T04:43:03.331362+0000","last_undegraded":"2021-02-10T04:43:03.331362+0000","last_fullsized":"2021-02-10T04:43:03.331362+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:35.382341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.382341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"921","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.535693+0000","last_change":"2021-02-10T04:28:38.503398+0000","last_active":"2021-02-10T04:43:01.535693+0000","last_peered":"2021-02-10T04:43:01.535693+0000","last_clean":"2021-02-10T04:43:01.535693+0000","last_became_active":"2021-02-10T04:25:53.037506+0000","last_became_peered":"2021-02-10T04:25:53.037506+0000","last_unstale":"2021-02-10T04:43:01.535693+0000","last_undegraded":"2021-02-10T04:43:01.535693+0000","last_fullsized":"2021-02-10T04:43:01.535693+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:38.503309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:38.503309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"961'452","reported_seq":"3331","reported_epoch":"964","state":"active+clean+snaptrim_wait","last_fresh":"2021-02-10T04:43:04.530664+0000","last_change":"2021-02-10T04:43:04.530731+0000","last_active":"2021-02-10T04:43:04.530664+0000","last_peered":"2021-02-10T04:43:04.530664+0000","last_clean":"2021-02-10T04:43:04.530664+0000","last_became_active":"2021-02-10T04:28:25.150109+0000","last_became_peered":"2021-02-10T04:28:25.150109+0000","last_unstale":"2021-02-10T04:43:04.530664+0000","last_undegraded":"2021-02-10T04:43:04.530664+0000","last_fullsized":"2021-02-10T04:43:04.530664+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'36","last_scrub_stamp":"2021-02-10T04:30:13.863430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:13.863430+0000","log_size":452,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":30088123,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":279,"num_read_kb":297136,"num_write":165,"num_write_kb":55048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24f","length":"1"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"1"},{"start":"2a3","length":"1"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"3"},{"start":"2ad","length":"1"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2ca","length":"2"},{"start":"2d1","length":"7"},{"start":"2e0","length":"3"},{"start":"2e4","length":"1"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"339","length":"1"},{"start":"33d","length":"1"},{"start":"340","length":"1"},{"start":"34a","length":"1"},{"start":"34f","length":"1"}]},{"pgid":"5.6","version":"0'0","reported_seq":"272","reported_epoch":"962","state":"active+clean","last_fresh":"2021-02-10T04:43:02.534400+0000","last_change":"2021-02-10T04:38:39.252779+0000","last_active":"2021-02-10T04:43:02.534400+0000","last_peered":"2021-02-10T04:43:02.534400+0000","last_clean":"2021-02-10T04:43:02.534400+0000","last_became_active":"2021-02-10T04:38:39.252266+0000","last_became_peered":"2021-02-10T04:38:39.252266+0000","last_unstale":"2021-02-10T04:43:02.534400+0000","last_undegraded":"2021-02-10T04:43:02.534400+0000","last_fullsized":"2021-02-10T04:43:02.534400+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.779767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.779767+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"538'33","reported_seq":"694","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.535540+0000","last_change":"2021-02-10T04:33:05.147122+0000","last_active":"2021-02-10T04:43:01.535540+0000","last_peered":"2021-02-10T04:43:01.535540+0000","last_clean":"2021-02-10T04:43:01.535540+0000","last_became_active":"2021-02-10T04:33:05.146695+0000","last_became_peered":"2021-02-10T04:33:05.146695+0000","last_unstale":"2021-02-10T04:43:01.535540+0000","last_undegraded":"2021-02-10T04:43:01.535540+0000","last_fullsized":"2021-02-10T04:43:01.535540+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'29","last_scrub_stamp":"2021-02-10T04:29:25.974280+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.974280+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":38,"num_write":28,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"396","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.330106+0000","last_change":"2021-02-10T04:37:37.258992+0000","last_active":"2021-02-10T04:43:03.330106+0000","last_peered":"2021-02-10T04:43:03.330106+0000","last_clean":"2021-02-10T04:43:03.330106+0000","last_became_active":"2021-02-10T04:37:37.257600+0000","last_became_peered":"2021-02-10T04:37:37.257600+0000","last_unstale":"2021-02-10T04:43:03.330106+0000","last_undegraded":"2021-02-10T04:43:03.330106+0000","last_fullsized":"2021-02-10T04:43:03.330106+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:39.690436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:39.690436+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"694","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.326258+0000","last_change":"2021-02-10T04:38:33.243623+0000","last_active":"2021-02-10T04:43:03.326258+0000","last_peered":"2021-02-10T04:43:03.326258+0000","last_clean":"2021-02-10T04:43:03.326258+0000","last_became_active":"2021-02-10T04:37:37.249591+0000","last_became_peered":"2021-02-10T04:37:37.249591+0000","last_unstale":"2021-02-10T04:43:03.326258+0000","last_undegraded":"2021-02-10T04:43:03.326258+0000","last_fullsized":"2021-02-10T04:43:03.326258+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:38:33.243519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:33.243519+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":333,"num_read_kb":328,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"849","reported_epoch":"962","state":"active+clean","last_fresh":"2021-02-10T04:43:02.535843+0000","last_change":"2021-02-10T04:29:33.747527+0000","last_active":"2021-02-10T04:43:02.535843+0000","last_peered":"2021-02-10T04:43:02.535843+0000","last_clean":"2021-02-10T04:43:02.535843+0000","last_became_active":"2021-02-10T04:29:07.158869+0000","last_became_peered":"2021-02-10T04:29:07.158869+0000","last_unstale":"2021-02-10T04:43:02.535843+0000","last_undegraded":"2021-02-10T04:43:02.535843+0000","last_fullsized":"2021-02-10T04:43:02.535843+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:33.747448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:33.747448+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.4","version":"964'601","reported_seq":"4019","reported_epoch":"964","state":"active+clean","last_fresh":"2021-02-10T04:43:04.725676+0000","last_change":"2021-02-10T04:43:04.531673+0000","last_active":"2021-02-10T04:43:04.725676+0000","last_peered":"2021-02-10T04:43:04.725676+0000","last_clean":"2021-02-10T04:43:04.725676+0000","last_became_active":"2021-02-10T04:28:25.151037+0000","last_became_peered":"2021-02-10T04:28:25.151037+0000","last_unstale":"2021-02-10T04:43:04.725676+0000","last_undegraded":"2021-02-10T04:43:04.725676+0000","last_fullsized":"2021-02-10T04:43:04.725676+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'239","last_scrub_stamp":"2021-02-10T04:33:46.270478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:46.270478+0000","log_size":601,"ondisk_log_size":601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32909415,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":285,"num_read_kb":256964,"num_write":253,"num_write_kb":103543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24f","length":"1"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"1"},{"start":"2a3","length":"1"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"3"},{"start":"2ad","length":"1"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2ca","length":"2"},{"start":"2d1","length":"7"},{"start":"2e0","length":"3"},{"start":"2e4","length":"1"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"339","length":"1"},{"start":"33d","length":"1"},{"start":"340","length":"1"},{"start":"34a","length":"1"},{"start":"34f","length":"1"},{"start":"351","length":"1"}]},{"pgid":"1.2","version":"0'0","reported_seq":"940","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.332813+0000","last_change":"2021-02-10T04:29:29.683867+0000","last_active":"2021-02-10T04:43:03.332813+0000","last_peered":"2021-02-10T04:43:03.332813+0000","last_clean":"2021-02-10T04:43:03.332813+0000","last_became_active":"2021-02-10T04:29:07.159607+0000","last_became_peered":"2021-02-10T04:29:07.159607+0000","last_unstale":"2021-02-10T04:43:03.332813+0000","last_undegraded":"2021-02-10T04:43:03.332813+0000","last_fullsized":"2021-02-10T04:43:03.332813+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.683805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.683805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"1015","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.338244+0000","last_change":"2021-02-10T04:42:19.154481+0000","last_active":"2021-02-10T04:43:03.338244+0000","last_peered":"2021-02-10T04:43:03.338244+0000","last_clean":"2021-02-10T04:43:03.338244+0000","last_became_active":"2021-02-10T04:25:51.027624+0000","last_became_peered":"2021-02-10T04:25:51.027624+0000","last_unstale":"2021-02-10T04:43:03.338244+0000","last_undegraded":"2021-02-10T04:43:03.338244+0000","last_fullsized":"2021-02-10T04:43:03.338244+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:43.740853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:28:43.740853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"538'43","reported_seq":"646","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.333176+0000","last_change":"2021-02-10T04:37:38.636607+0000","last_active":"2021-02-10T04:43:03.333176+0000","last_peered":"2021-02-10T04:43:03.333176+0000","last_clean":"2021-02-10T04:43:03.333176+0000","last_became_active":"2021-02-10T04:37:37.244344+0000","last_became_peered":"2021-02-10T04:37:37.244344+0000","last_unstale":"2021-02-10T04:43:03.333176+0000","last_undegraded":"2021-02-10T04:43:03.333176+0000","last_fullsized":"2021-02-10T04:43:03.333176+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'31","last_scrub_stamp":"2021-02-10T04:33:49.423083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:49.423083+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":81,"num_read_kb":57,"num_write":46,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"946","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.534258+0000","last_change":"2021-02-10T04:38:34.289602+0000","last_active":"2021-02-10T04:43:01.534258+0000","last_peered":"2021-02-10T04:43:01.534258+0000","last_clean":"2021-02-10T04:43:01.534258+0000","last_became_active":"2021-02-10T04:28:21.125541+0000","last_became_peered":"2021-02-10T04:28:21.125541+0000","last_unstale":"2021-02-10T04:43:01.534258+0000","last_undegraded":"2021-02-10T04:43:01.534258+0000","last_fullsized":"2021-02-10T04:43:01.534258+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:34.289495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:34.289495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"922","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.333754+0000","last_change":"2021-02-10T04:29:36.396044+0000","last_active":"2021-02-10T04:43:03.333754+0000","last_peered":"2021-02-10T04:43:03.333754+0000","last_clean":"2021-02-10T04:43:03.333754+0000","last_became_active":"2021-02-10T04:25:53.049487+0000","last_became_peered":"2021-02-10T04:25:53.049487+0000","last_unstale":"2021-02-10T04:43:03.333754+0000","last_undegraded":"2021-02-10T04:43:03.333754+0000","last_fullsized":"2021-02-10T04:43:03.333754+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.395943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.395943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"961'1967","reported_seq":"8450","reported_epoch":"962","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:43:02.982666+0000","last_change":"2021-02-10T04:43:00.510604+0000","last_active":"2021-02-10T04:43:02.982666+0000","last_peered":"2021-02-10T04:43:02.982666+0000","last_clean":"2021-02-10T04:43:02.982666+0000","last_became_active":"2021-02-10T04:28:25.147959+0000","last_became_peered":"2021-02-10T04:28:25.147959+0000","last_unstale":"2021-02-10T04:43:02.982666+0000","last_undegraded":"2021-02-10T04:43:02.982666+0000","last_fullsized":"2021-02-10T04:43:02.982666+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'223","last_scrub_stamp":"2021-02-10T04:30:14.478639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:14.478639+0000","log_size":1967,"ondisk_log_size":1967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"stat_sum":{"num_bytes":99119753,"num_objects":43,"num_object_clones":36,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":2,"num_read":1044,"num_read_kb":1011824,"num_write":789,"num_write_kb":301587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24f","length":"1"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"1"},{"start":"2a3","length":"1"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"3"},{"start":"2ad","length":"1"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2ca","length":"2"},{"start":"2d1","length":"7"},{"start":"2e0","length":"3"},{"start":"2e4","length":"1"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"340","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":"873","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.533783+0000","last_change":"2021-02-10T04:37:37.219806+0000","last_active":"2021-02-10T04:43:01.533783+0000","last_peered":"2021-02-10T04:43:01.533783+0000","last_clean":"2021-02-10T04:43:01.533783+0000","last_became_active":"2021-02-10T04:37:37.219422+0000","last_became_peered":"2021-02-10T04:37:37.219422+0000","last_unstale":"2021-02-10T04:43:01.533783+0000","last_undegraded":"2021-02-10T04:43:01.533783+0000","last_fullsized":"2021-02-10T04:43:01.533783+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:39.434210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:39.434210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"58'2","reported_seq":"1139","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.333726+0000","last_change":"2021-02-10T04:38:27.092813+0000","last_active":"2021-02-10T04:43:03.333726+0000","last_peered":"2021-02-10T04:43:03.333726+0000","last_clean":"2021-02-10T04:43:03.333726+0000","last_became_active":"2021-02-10T04:25:51.026251+0000","last_became_peered":"2021-02-10T04:25:51.026251+0000","last_unstale":"2021-02-10T04:43:03.333726+0000","last_undegraded":"2021-02-10T04:43:03.333726+0000","last_fullsized":"2021-02-10T04:43:03.333726+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:38:27.092697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:38:27.092697+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"538'44","reported_seq":"696","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.534472+0000","last_change":"2021-02-10T04:33:05.142280+0000","last_active":"2021-02-10T04:43:01.534472+0000","last_peered":"2021-02-10T04:43:01.534472+0000","last_clean":"2021-02-10T04:43:01.534472+0000","last_became_active":"2021-02-10T04:33:05.141901+0000","last_became_peered":"2021-02-10T04:33:05.141901+0000","last_unstale":"2021-02-10T04:43:01.534472+0000","last_undegraded":"2021-02-10T04:43:01.534472+0000","last_fullsized":"2021-02-10T04:43:01.534472+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'34","last_scrub_stamp":"2021-02-10T04:28:49.959162+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:28:49.959162+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":60,"num_read_kb":42,"num_write":32,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"690","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.333962+0000","last_change":"2021-02-10T04:37:37.247740+0000","last_active":"2021-02-10T04:43:03.333962+0000","last_peered":"2021-02-10T04:43:03.333962+0000","last_clean":"2021-02-10T04:43:03.333962+0000","last_became_active":"2021-02-10T04:37:37.247405+0000","last_became_peered":"2021-02-10T04:37:37.247405+0000","last_unstale":"2021-02-10T04:43:03.333962+0000","last_undegraded":"2021-02-10T04:43:03.333962+0000","last_fullsized":"2021-02-10T04:43:03.333962+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:34.053323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:34.053323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"937","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.328071+0000","last_change":"2021-02-10T04:38:26.124897+0000","last_active":"2021-02-10T04:43:03.328071+0000","last_peered":"2021-02-10T04:43:03.328071+0000","last_clean":"2021-02-10T04:43:03.328071+0000","last_became_active":"2021-02-10T04:37:37.254482+0000","last_became_peered":"2021-02-10T04:37:37.254482+0000","last_unstale":"2021-02-10T04:43:03.328071+0000","last_undegraded":"2021-02-10T04:43:03.328071+0000","last_fullsized":"2021-02-10T04:43:03.328071+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:26.124805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:38:26.124805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"293'77","reported_seq":"2145","reported_epoch":"960","state":"active+clean+scrubbing","last_fresh":"2021-02-10T04:43:00.521456+0000","last_change":"2021-02-10T04:38:41.130429+0000","last_active":"2021-02-10T04:43:00.521456+0000","last_peered":"2021-02-10T04:43:00.521456+0000","last_clean":"2021-02-10T04:43:00.521456+0000","last_became_active":"2021-02-10T04:28:25.154434+0000","last_became_peered":"2021-02-10T04:28:25.154434+0000","last_unstale":"2021-02-10T04:43:00.521456+0000","last_undegraded":"2021-02-10T04:43:00.521456+0000","last_fullsized":"2021-02-10T04:43:00.521456+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":120,"stat_sum":{"num_bytes":7546343,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":74349,"num_write":37,"num_write_kb":13263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"946","reported_epoch":"962","state":"active+clean","last_fresh":"2021-02-10T04:43:02.534890+0000","last_change":"2021-02-10T04:29:39.829039+0000","last_active":"2021-02-10T04:43:02.534890+0000","last_peered":"2021-02-10T04:43:02.534890+0000","last_clean":"2021-02-10T04:43:02.534890+0000","last_became_active":"2021-02-10T04:28:49.053127+0000","last_became_peered":"2021-02-10T04:28:49.053127+0000","last_unstale":"2021-02-10T04:43:02.534890+0000","last_undegraded":"2021-02-10T04:43:02.534890+0000","last_fullsized":"2021-02-10T04:43:02.534890+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:39.828975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:39.828975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"346","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.328398+0000","last_change":"2021-02-10T04:38:39.247167+0000","last_active":"2021-02-10T04:43:03.328398+0000","last_peered":"2021-02-10T04:43:03.328398+0000","last_clean":"2021-02-10T04:43:03.328398+0000","last_became_active":"2021-02-10T04:38:39.246766+0000","last_became_peered":"2021-02-10T04:38:39.246766+0000","last_unstale":"2021-02-10T04:43:03.328398+0000","last_undegraded":"2021-02-10T04:43:03.328398+0000","last_fullsized":"2021-02-10T04:43:03.328398+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:20.757194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:20.757194+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"538'33","reported_seq":"1033","reported_epoch":"960","state":"active+clean","last_fresh":"2021-02-10T04:43:00.521644+0000","last_change":"2021-02-10T04:39:02.586279+0000","last_active":"2021-02-10T04:43:00.521644+0000","last_peered":"2021-02-10T04:43:00.521644+0000","last_clean":"2021-02-10T04:43:00.521644+0000","last_became_active":"2021-02-10T04:39:02.560739+0000","last_became_peered":"2021-02-10T04:39:02.560739+0000","last_unstale":"2021-02-10T04:43:00.521644+0000","last_undegraded":"2021-02-10T04:43:00.521644+0000","last_fullsized":"2021-02-10T04:43:00.521644+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'27","last_scrub_stamp":"2021-02-10T04:28:47.476669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:47.476669+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":67,"num_read_kb":50,"num_write":35,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"1055","reported_epoch":"960","state":"active+clean","last_fresh":"2021-02-10T04:43:00.522948+0000","last_change":"2021-02-10T04:37:37.351533+0000","last_active":"2021-02-10T04:43:00.522948+0000","last_peered":"2021-02-10T04:43:00.522948+0000","last_clean":"2021-02-10T04:43:00.522948+0000","last_became_active":"2021-02-10T04:37:37.238701+0000","last_became_peered":"2021-02-10T04:37:37.238701+0000","last_unstale":"2021-02-10T04:43:00.522948+0000","last_undegraded":"2021-02-10T04:43:00.522948+0000","last_fullsized":"2021-02-10T04:43:00.522948+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:33:49.760989+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:33:49.760989+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"922","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.332935+0000","last_change":"2021-02-10T04:29:37.364781+0000","last_active":"2021-02-10T04:43:03.332935+0000","last_peered":"2021-02-10T04:43:03.332935+0000","last_clean":"2021-02-10T04:43:03.332935+0000","last_became_active":"2021-02-10T04:25:53.047736+0000","last_became_peered":"2021-02-10T04:25:53.047736+0000","last_unstale":"2021-02-10T04:43:03.332935+0000","last_undegraded":"2021-02-10T04:43:03.332935+0000","last_fullsized":"2021-02-10T04:43:03.332935+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:37.364697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:37.364697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"963'1465","reported_seq":"5468","reported_epoch":"963","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:43:03.528625+0000","last_change":"2021-02-10T04:43:01.518832+0000","last_active":"2021-02-10T04:43:03.528625+0000","last_peered":"2021-02-10T04:43:03.528625+0000","last_clean":"2021-02-10T04:43:03.528625+0000","last_became_active":"2021-02-10T04:37:48.215401+0000","last_became_peered":"2021-02-10T04:37:48.215401+0000","last_unstale":"2021-02-10T04:43:03.528625+0000","last_undegraded":"2021-02-10T04:43:03.528625+0000","last_fullsized":"2021-02-10T04:43:03.528625+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":1465,"ondisk_log_size":1465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":102567153,"num_objects":37,"num_object_clones":34,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":1,"num_read":713,"num_read_kb":703868,"num_write":565,"num_write_kb":218335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":42623785,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24f","length":"1"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"1"},{"start":"2a3","length":"1"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"3"},{"start":"2ad","length":"1"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2ca","length":"2"},{"start":"2d1","length":"7"},{"start":"2e0","length":"3"},{"start":"2e4","length":"1"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"340","length":"1"},{"start":"34f","length":"1"}]},{"pgid":"4.16","version":"538'34","reported_seq":"996","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.337648+0000","last_change":"2021-02-10T04:29:25.645462+0000","last_active":"2021-02-10T04:43:03.337648+0000","last_peered":"2021-02-10T04:43:03.337648+0000","last_clean":"2021-02-10T04:43:03.337648+0000","last_became_active":"2021-02-10T04:25:49.008942+0000","last_became_peered":"2021-02-10T04:25:49.008942+0000","last_unstale":"2021-02-10T04:43:03.337648+0000","last_undegraded":"2021-02-10T04:43:03.337648+0000","last_fullsized":"2021-02-10T04:43:03.337648+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'28","last_scrub_stamp":"2021-02-10T04:29:25.645396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.645396+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":71,"num_read_kb":49,"num_write":35,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"355","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.329692+0000","last_change":"2021-02-10T04:38:38.163311+0000","last_active":"2021-02-10T04:43:03.329692+0000","last_peered":"2021-02-10T04:43:03.329692+0000","last_clean":"2021-02-10T04:43:03.329692+0000","last_became_active":"2021-02-10T04:37:37.234649+0000","last_became_peered":"2021-02-10T04:37:37.234649+0000","last_unstale":"2021-02-10T04:43:03.329692+0000","last_undegraded":"2021-02-10T04:43:03.329692+0000","last_fullsized":"2021-02-10T04:43:03.329692+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:38:38.163225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:38.163225+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":247,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"924","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.334178+0000","last_change":"2021-02-10T04:28:33.321856+0000","last_active":"2021-02-10T04:43:03.334178+0000","last_peered":"2021-02-10T04:43:03.334178+0000","last_clean":"2021-02-10T04:43:03.334178+0000","last_became_active":"2021-02-10T04:25:51.030651+0000","last_became_peered":"2021-02-10T04:25:51.030651+0000","last_unstale":"2021-02-10T04:43:03.334178+0000","last_undegraded":"2021-02-10T04:43:03.334178+0000","last_fullsized":"2021-02-10T04:43:03.334178+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:33.321784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:33.321784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"334","reported_epoch":"960","state":"active+clean","last_fresh":"2021-02-10T04:43:00.522086+0000","last_change":"2021-02-10T04:37:37.238450+0000","last_active":"2021-02-10T04:43:00.522086+0000","last_peered":"2021-02-10T04:43:00.522086+0000","last_clean":"2021-02-10T04:43:00.522086+0000","last_became_active":"2021-02-10T04:37:37.238049+0000","last_became_peered":"2021-02-10T04:37:37.238049+0000","last_unstale":"2021-02-10T04:43:00.522086+0000","last_undegraded":"2021-02-10T04:43:00.522086+0000","last_fullsized":"2021-02-10T04:43:00.522086+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.776340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.776340+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"538'19","reported_seq":"1008","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.334054+0000","last_change":"2021-02-10T04:38:26.845289+0000","last_active":"2021-02-10T04:43:03.334054+0000","last_peered":"2021-02-10T04:43:03.334054+0000","last_clean":"2021-02-10T04:43:03.334054+0000","last_became_active":"2021-02-10T04:37:37.247396+0000","last_became_peered":"2021-02-10T04:37:37.247396+0000","last_unstale":"2021-02-10T04:43:03.334054+0000","last_undegraded":"2021-02-10T04:43:03.334054+0000","last_fullsized":"2021-02-10T04:43:03.334054+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'19","last_scrub_stamp":"2021-02-10T04:38:26.845195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:26.845195+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":19,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"940","reported_epoch":"962","state":"active+clean","last_fresh":"2021-02-10T04:43:02.533263+0000","last_change":"2021-02-10T04:38:36.295733+0000","last_active":"2021-02-10T04:43:02.533263+0000","last_peered":"2021-02-10T04:43:02.533263+0000","last_clean":"2021-02-10T04:43:02.533263+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:43:02.533263+0000","last_undegraded":"2021-02-10T04:43:02.533263+0000","last_fullsized":"2021-02-10T04:43:02.533263+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:38:36.295599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:36.295599+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"923","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.535286+0000","last_change":"2021-02-10T04:29:18.429730+0000","last_active":"2021-02-10T04:43:01.535286+0000","last_peered":"2021-02-10T04:43:01.535286+0000","last_clean":"2021-02-10T04:43:01.535286+0000","last_became_active":"2021-02-10T04:25:51.031864+0000","last_became_peered":"2021-02-10T04:25:51.031864+0000","last_unstale":"2021-02-10T04:43:01.535286+0000","last_undegraded":"2021-02-10T04:43:01.535286+0000","last_fullsized":"2021-02-10T04:43:01.535286+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:18.429649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:18.429649+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"849","reported_epoch":"962","state":"active+clean","last_fresh":"2021-02-10T04:43:02.536056+0000","last_change":"2021-02-10T04:29:36.866949+0000","last_active":"2021-02-10T04:43:02.536056+0000","last_peered":"2021-02-10T04:43:02.536056+0000","last_clean":"2021-02-10T04:43:02.536056+0000","last_became_active":"2021-02-10T04:29:07.147249+0000","last_became_peered":"2021-02-10T04:29:07.147249+0000","last_unstale":"2021-02-10T04:43:02.536056+0000","last_undegraded":"2021-02-10T04:43:02.536056+0000","last_fullsized":"2021-02-10T04:43:02.536056+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.866833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.866833+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"538'36","reported_seq":"1132","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.332894+0000","last_change":"2021-02-10T04:37:38.541093+0000","last_active":"2021-02-10T04:43:03.332894+0000","last_peered":"2021-02-10T04:43:03.332894+0000","last_clean":"2021-02-10T04:43:03.332894+0000","last_became_active":"2021-02-10T04:37:37.244921+0000","last_became_peered":"2021-02-10T04:37:37.244921+0000","last_unstale":"2021-02-10T04:43:03.332894+0000","last_undegraded":"2021-02-10T04:43:03.332894+0000","last_fullsized":"2021-02-10T04:43:03.332894+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:33:46.785905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:46.785905+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":36,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"84'7","reported_seq":"952","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.534169+0000","last_change":"2021-02-10T04:30:12.538670+0000","last_active":"2021-02-10T04:43:01.534169+0000","last_peered":"2021-02-10T04:43:01.534169+0000","last_clean":"2021-02-10T04:43:01.534169+0000","last_became_active":"2021-02-10T04:25:36.334366+0000","last_became_peered":"2021-02-10T04:25:36.334366+0000","last_unstale":"2021-02-10T04:43:01.534169+0000","last_undegraded":"2021-02-10T04:43:01.534169+0000","last_fullsized":"2021-02-10T04:43:01.534169+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'7","last_scrub_stamp":"2021-02-10T04:30:12.538549+0000","last_deep_scrub":"84'7","last_deep_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_clean_scrub_stamp":"2021-02-10T04:30:12.538549+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"58'2","reported_seq":"1132","reported_epoch":"962","state":"active+clean","last_fresh":"2021-02-10T04:43:02.533711+0000","last_change":"2021-02-10T04:28:47.496534+0000","last_active":"2021-02-10T04:43:02.533711+0000","last_peered":"2021-02-10T04:43:02.533711+0000","last_clean":"2021-02-10T04:43:02.533711+0000","last_became_active":"2021-02-10T04:25:51.032328+0000","last_became_peered":"2021-02-10T04:25:51.032328+0000","last_unstale":"2021-02-10T04:43:02.533711+0000","last_undegraded":"2021-02-10T04:43:02.533711+0000","last_fullsized":"2021-02-10T04:43:02.533711+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:28:47.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:47.496468+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"928","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.337345+0000","last_change":"2021-02-10T04:38:39.191979+0000","last_active":"2021-02-10T04:43:03.337345+0000","last_peered":"2021-02-10T04:43:03.337345+0000","last_clean":"2021-02-10T04:43:03.337345+0000","last_became_active":"2021-02-10T04:25:53.045177+0000","last_became_peered":"2021-02-10T04:25:53.045177+0000","last_unstale":"2021-02-10T04:43:03.337345+0000","last_undegraded":"2021-02-10T04:43:03.337345+0000","last_fullsized":"2021-02-10T04:43:03.337345+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:39.191910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_clean_scrub_stamp":"2021-02-10T04:38:39.191910+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"538'25","reported_seq":"973","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.534839+0000","last_change":"2021-02-10T04:28:50.493712+0000","last_active":"2021-02-10T04:43:01.534839+0000","last_peered":"2021-02-10T04:43:01.534839+0000","last_clean":"2021-02-10T04:43:01.534839+0000","last_became_active":"2021-02-10T04:25:49.010890+0000","last_became_peered":"2021-02-10T04:25:49.010890+0000","last_unstale":"2021-02-10T04:43:01.534839+0000","last_undegraded":"2021-02-10T04:43:01.534839+0000","last_fullsized":"2021-02-10T04:43:01.534839+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2021-02-10T04:28:50.493641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:50.493641+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":35,"num_write":26,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"939","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.333500+0000","last_change":"2021-02-10T04:29:19.564845+0000","last_active":"2021-02-10T04:43:03.333500+0000","last_peered":"2021-02-10T04:43:03.333500+0000","last_clean":"2021-02-10T04:43:03.333500+0000","last_became_active":"2021-02-10T04:25:34.321096+0000","last_became_peered":"2021-02-10T04:25:34.321096+0000","last_unstale":"2021-02-10T04:43:03.333500+0000","last_undegraded":"2021-02-10T04:43:03.333500+0000","last_fullsized":"2021-02-10T04:43:03.333500+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:19.564725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:29:19.564725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"639'3","reported_seq":"1146","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.333085+0000","last_change":"2021-02-10T04:37:37.922249+0000","last_active":"2021-02-10T04:43:03.333085+0000","last_peered":"2021-02-10T04:43:03.333085+0000","last_clean":"2021-02-10T04:43:03.333085+0000","last_became_active":"2021-02-10T04:37:37.245428+0000","last_became_peered":"2021-02-10T04:37:37.245428+0000","last_unstale":"2021-02-10T04:43:03.333085+0000","last_undegraded":"2021-02-10T04:43:03.333085+0000","last_fullsized":"2021-02-10T04:43:03.333085+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:29:38.343834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.343834+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"929","reported_epoch":"960","state":"active+clean","last_fresh":"2021-02-10T04:43:00.523086+0000","last_change":"2021-02-10T04:37:37.238692+0000","last_active":"2021-02-10T04:43:00.523086+0000","last_peered":"2021-02-10T04:43:00.523086+0000","last_clean":"2021-02-10T04:43:00.523086+0000","last_became_active":"2021-02-10T04:37:37.238360+0000","last_became_peered":"2021-02-10T04:37:37.238360+0000","last_unstale":"2021-02-10T04:43:00.523086+0000","last_undegraded":"2021-02-10T04:43:00.523086+0000","last_fullsized":"2021-02-10T04:43:00.523086+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.934078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.934078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"538'21","reported_seq":"688","reported_epoch":"962","state":"active+clean","last_fresh":"2021-02-10T04:43:02.534179+0000","last_change":"2021-02-10T04:37:38.590027+0000","last_active":"2021-02-10T04:43:02.534179+0000","last_peered":"2021-02-10T04:43:02.534179+0000","last_clean":"2021-02-10T04:43:02.534179+0000","last_became_active":"2021-02-10T04:37:37.249007+0000","last_became_peered":"2021-02-10T04:37:37.249007+0000","last_unstale":"2021-02-10T04:43:02.534179+0000","last_undegraded":"2021-02-10T04:43:02.534179+0000","last_fullsized":"2021-02-10T04:43:02.534179+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:30:11.170034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.170034+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"938","reported_epoch":"962","state":"active+clean","last_fresh":"2021-02-10T04:43:02.534929+0000","last_change":"2021-02-10T04:38:26.127398+0000","last_active":"2021-02-10T04:43:02.534929+0000","last_peered":"2021-02-10T04:43:02.534929+0000","last_clean":"2021-02-10T04:43:02.534929+0000","last_became_active":"2021-02-10T04:25:34.330070+0000","last_became_peered":"2021-02-10T04:25:34.330070+0000","last_unstale":"2021-02-10T04:43:02.534929+0000","last_undegraded":"2021-02-10T04:43:02.534929+0000","last_fullsized":"2021-02-10T04:43:02.534929+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:26.127288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:26.127288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"924","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.332780+0000","last_change":"2021-02-10T04:28:36.311808+0000","last_active":"2021-02-10T04:43:03.332780+0000","last_peered":"2021-02-10T04:43:03.332780+0000","last_clean":"2021-02-10T04:43:03.332780+0000","last_became_active":"2021-02-10T04:25:51.029155+0000","last_became_peered":"2021-02-10T04:25:51.029155+0000","last_unstale":"2021-02-10T04:43:03.332780+0000","last_undegraded":"2021-02-10T04:43:03.332780+0000","last_fullsized":"2021-02-10T04:43:03.332780+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:36.311725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:28:36.311725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"928","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.326787+0000","last_change":"2021-02-10T04:33:49.338303+0000","last_active":"2021-02-10T04:43:03.326787+0000","last_peered":"2021-02-10T04:43:03.326787+0000","last_clean":"2021-02-10T04:43:03.326787+0000","last_became_active":"2021-02-10T04:25:53.039322+0000","last_became_peered":"2021-02-10T04:25:53.039322+0000","last_unstale":"2021-02-10T04:43:03.326787+0000","last_undegraded":"2021-02-10T04:43:03.326787+0000","last_fullsized":"2021-02-10T04:43:03.326787+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:49.338203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:33:49.338203+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"538'23","reported_seq":"963","reported_epoch":"960","state":"active+clean","last_fresh":"2021-02-10T04:43:00.522366+0000","last_change":"2021-02-10T04:38:32.221752+0000","last_active":"2021-02-10T04:43:00.522366+0000","last_peered":"2021-02-10T04:43:00.522366+0000","last_clean":"2021-02-10T04:43:00.522366+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:43:00.522366+0000","last_undegraded":"2021-02-10T04:43:00.522366+0000","last_fullsized":"2021-02-10T04:43:00.522366+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'23","last_scrub_stamp":"2021-02-10T04:38:32.221656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:32.221656+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"622","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.534883+0000","last_change":"2021-02-10T04:38:36.309455+0000","last_active":"2021-02-10T04:43:01.534883+0000","last_peered":"2021-02-10T04:43:01.534883+0000","last_clean":"2021-02-10T04:43:01.534883+0000","last_became_active":"2021-02-10T04:33:05.147956+0000","last_became_peered":"2021-02-10T04:33:05.147956+0000","last_unstale":"2021-02-10T04:43:01.534883+0000","last_undegraded":"2021-02-10T04:43:01.534883+0000","last_fullsized":"2021-02-10T04:43:01.534883+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:38:36.309362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:36.309362+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"606","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.534416+0000","last_change":"2021-02-10T04:33:05.150906+0000","last_active":"2021-02-10T04:43:01.534416+0000","last_peered":"2021-02-10T04:43:01.534416+0000","last_clean":"2021-02-10T04:43:01.534416+0000","last_became_active":"2021-02-10T04:33:05.150529+0000","last_became_peered":"2021-02-10T04:33:05.150529+0000","last_unstale":"2021-02-10T04:43:01.534416+0000","last_undegraded":"2021-02-10T04:43:01.534416+0000","last_fullsized":"2021-02-10T04:43:01.534416+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:33.026988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:29:33.026988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"922","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.329916+0000","last_change":"2021-02-10T04:29:24.421782+0000","last_active":"2021-02-10T04:43:03.329916+0000","last_peered":"2021-02-10T04:43:03.329916+0000","last_clean":"2021-02-10T04:43:03.329916+0000","last_became_active":"2021-02-10T04:25:53.043277+0000","last_became_peered":"2021-02-10T04:25:53.043277+0000","last_unstale":"2021-02-10T04:43:03.329916+0000","last_undegraded":"2021-02-10T04:43:03.329916+0000","last_fullsized":"2021-02-10T04:43:03.329916+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.421710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.421710+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"538'16","reported_seq":"1085","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.331847+0000","last_change":"2021-02-10T04:38:22.919477+0000","last_active":"2021-02-10T04:43:03.331847+0000","last_peered":"2021-02-10T04:43:03.331847+0000","last_clean":"2021-02-10T04:43:03.331847+0000","last_became_active":"2021-02-10T04:26:39.005616+0000","last_became_peered":"2021-02-10T04:26:39.005616+0000","last_unstale":"2021-02-10T04:43:03.331847+0000","last_undegraded":"2021-02-10T04:43:03.331847+0000","last_fullsized":"2021-02-10T04:43:03.331847+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'16","last_scrub_stamp":"2021-02-10T04:38:22.919377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:22.919377+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":36,"num_read_kb":18,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"941","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.535400+0000","last_change":"2021-02-10T04:37:37.230226+0000","last_active":"2021-02-10T04:43:01.535400+0000","last_peered":"2021-02-10T04:43:01.535400+0000","last_clean":"2021-02-10T04:43:01.535400+0000","last_became_active":"2021-02-10T04:37:37.229852+0000","last_became_peered":"2021-02-10T04:37:37.229852+0000","last_unstale":"2021-02-10T04:43:01.535400+0000","last_undegraded":"2021-02-10T04:43:01.535400+0000","last_fullsized":"2021-02-10T04:43:01.535400+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:49.522124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:49.522124+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"924","reported_epoch":"962","state":"active+clean","last_fresh":"2021-02-10T04:43:02.533180+0000","last_change":"2021-02-10T04:29:23.645783+0000","last_active":"2021-02-10T04:43:02.533180+0000","last_peered":"2021-02-10T04:43:02.533180+0000","last_clean":"2021-02-10T04:43:02.533180+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:43:02.533180+0000","last_undegraded":"2021-02-10T04:43:02.533180+0000","last_fullsized":"2021-02-10T04:43:02.533180+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.645694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.645694+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"337","reported_epoch":"960","state":"active+clean","last_fresh":"2021-02-10T04:43:00.521549+0000","last_change":"2021-02-10T04:37:37.244700+0000","last_active":"2021-02-10T04:43:00.521549+0000","last_peered":"2021-02-10T04:43:00.521549+0000","last_clean":"2021-02-10T04:43:00.521549+0000","last_became_active":"2021-02-10T04:37:37.244082+0000","last_became_peered":"2021-02-10T04:37:37.244082+0000","last_unstale":"2021-02-10T04:43:00.521549+0000","last_undegraded":"2021-02-10T04:43:00.521549+0000","last_fullsized":"2021-02-10T04:43:00.521549+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:31.824925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.824925+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.11","version":"538'29","reported_seq":"979","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.330317+0000","last_change":"2021-02-10T04:28:31.382776+0000","last_active":"2021-02-10T04:43:03.330317+0000","last_peered":"2021-02-10T04:43:03.330317+0000","last_clean":"2021-02-10T04:43:03.330317+0000","last_became_active":"2021-02-10T04:25:49.007049+0000","last_became_peered":"2021-02-10T04:25:49.007049+0000","last_unstale":"2021-02-10T04:43:03.330317+0000","last_undegraded":"2021-02-10T04:43:03.330317+0000","last_fullsized":"2021-02-10T04:43:03.330317+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2021-02-10T04:28:31.382671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:31.382671+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":29,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"1078","reported_epoch":"962","state":"active+clean","last_fresh":"2021-02-10T04:43:02.535367+0000","last_change":"2021-02-10T04:38:31.166652+0000","last_active":"2021-02-10T04:43:02.535367+0000","last_peered":"2021-02-10T04:43:02.535367+0000","last_clean":"2021-02-10T04:43:02.535367+0000","last_became_active":"2021-02-10T04:37:37.228840+0000","last_became_peered":"2021-02-10T04:37:37.228840+0000","last_unstale":"2021-02-10T04:43:02.535367+0000","last_undegraded":"2021-02-10T04:43:02.535367+0000","last_fullsized":"2021-02-10T04:43:02.535367+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:38:31.166586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:31.166586+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"273","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.331987+0000","last_change":"2021-02-10T04:38:39.246255+0000","last_active":"2021-02-10T04:43:03.331987+0000","last_peered":"2021-02-10T04:43:03.331987+0000","last_clean":"2021-02-10T04:43:03.331987+0000","last_became_active":"2021-02-10T04:38:39.246046+0000","last_became_peered":"2021-02-10T04:38:39.246046+0000","last_unstale":"2021-02-10T04:43:03.331987+0000","last_undegraded":"2021-02-10T04:43:03.331987+0000","last_fullsized":"2021-02-10T04:43:03.331987+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:47.473024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:47.473024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"922","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.333034+0000","last_change":"2021-02-10T04:33:50.809533+0000","last_active":"2021-02-10T04:43:03.333034+0000","last_peered":"2021-02-10T04:43:03.333034+0000","last_clean":"2021-02-10T04:43:03.333034+0000","last_became_active":"2021-02-10T04:25:53.150845+0000","last_became_peered":"2021-02-10T04:25:53.150845+0000","last_unstale":"2021-02-10T04:43:03.333034+0000","last_undegraded":"2021-02-10T04:43:03.333034+0000","last_fullsized":"2021-02-10T04:43:03.333034+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:50.809468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_clean_scrub_stamp":"2021-02-10T04:33:50.809468+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"538'32","reported_seq":"989","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.337840+0000","last_change":"2021-02-10T04:29:38.663351+0000","last_active":"2021-02-10T04:43:03.337840+0000","last_peered":"2021-02-10T04:43:03.337840+0000","last_clean":"2021-02-10T04:43:03.337840+0000","last_became_active":"2021-02-10T04:25:49.009177+0000","last_became_peered":"2021-02-10T04:25:49.009177+0000","last_unstale":"2021-02-10T04:43:03.337840+0000","last_undegraded":"2021-02-10T04:43:03.337840+0000","last_fullsized":"2021-02-10T04:43:03.337840+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2021-02-10T04:29:38.663272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.663272+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":60,"num_read_kb":42,"num_write":32,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"941","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.533262+0000","last_change":"2021-02-10T04:28:37.515037+0000","last_active":"2021-02-10T04:43:01.533262+0000","last_peered":"2021-02-10T04:43:01.533262+0000","last_clean":"2021-02-10T04:43:01.533262+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:43:01.533262+0000","last_undegraded":"2021-02-10T04:43:01.533262+0000","last_fullsized":"2021-02-10T04:43:01.533262+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:37.514943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:37.514943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"58'2","reported_seq":"1133","reported_epoch":"962","state":"active+clean","last_fresh":"2021-02-10T04:43:02.894122+0000","last_change":"2021-02-10T04:29:31.726022+0000","last_active":"2021-02-10T04:43:02.894122+0000","last_peered":"2021-02-10T04:43:02.894122+0000","last_clean":"2021-02-10T04:43:02.894122+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:43:02.894122+0000","last_undegraded":"2021-02-10T04:43:02.894122+0000","last_fullsized":"2021-02-10T04:43:02.894122+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:29:31.725932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.725932+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"866","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.534467+0000","last_change":"2021-02-10T04:38:39.252110+0000","last_active":"2021-02-10T04:43:01.534467+0000","last_peered":"2021-02-10T04:43:01.534467+0000","last_clean":"2021-02-10T04:43:01.534467+0000","last_became_active":"2021-02-10T04:38:39.251336+0000","last_became_peered":"2021-02-10T04:38:39.251336+0000","last_unstale":"2021-02-10T04:43:01.534467+0000","last_undegraded":"2021-02-10T04:43:01.534467+0000","last_fullsized":"2021-02-10T04:43:01.534467+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:26.437591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.437591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"538'43","reported_seq":"1001","reported_epoch":"962","state":"active+clean","last_fresh":"2021-02-10T04:43:02.535453+0000","last_change":"2021-02-10T04:30:13.526264+0000","last_active":"2021-02-10T04:43:02.535453+0000","last_peered":"2021-02-10T04:43:02.535453+0000","last_clean":"2021-02-10T04:43:02.535453+0000","last_became_active":"2021-02-10T04:25:49.015760+0000","last_became_peered":"2021-02-10T04:25:49.015760+0000","last_unstale":"2021-02-10T04:43:02.535453+0000","last_undegraded":"2021-02-10T04:43:02.535453+0000","last_fullsized":"2021-02-10T04:43:02.535453+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'35","last_scrub_stamp":"2021-02-10T04:30:13.526170+0000","last_deep_scrub":"76'35","last_deep_scrub_stamp":"2021-02-10T04:26:58.198641+0000","last_clean_scrub_stamp":"2021-02-10T04:30:13.526170+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":62,"num_read_kb":42,"num_write":32,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"933","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.330034+0000","last_change":"2021-02-10T04:30:11.485059+0000","last_active":"2021-02-10T04:43:03.330034+0000","last_peered":"2021-02-10T04:43:03.330034+0000","last_clean":"2021-02-10T04:43:03.330034+0000","last_became_active":"2021-02-10T04:25:34.324981+0000","last_became_peered":"2021-02-10T04:25:34.324981+0000","last_unstale":"2021-02-10T04:43:03.330034+0000","last_undegraded":"2021-02-10T04:43:03.330034+0000","last_fullsized":"2021-02-10T04:43:03.330034+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:11.485004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.485004+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"934","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.337902+0000","last_change":"2021-02-10T04:37:37.248748+0000","last_active":"2021-02-10T04:43:03.337902+0000","last_peered":"2021-02-10T04:43:03.337902+0000","last_clean":"2021-02-10T04:43:03.337902+0000","last_became_active":"2021-02-10T04:37:37.246998+0000","last_became_peered":"2021-02-10T04:37:37.246998+0000","last_unstale":"2021-02-10T04:43:03.337902+0000","last_undegraded":"2021-02-10T04:43:03.337902+0000","last_fullsized":"2021-02-10T04:43:03.337902+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.589159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.589159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"606","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.535065+0000","last_change":"2021-02-10T04:33:05.142344+0000","last_active":"2021-02-10T04:43:01.535065+0000","last_peered":"2021-02-10T04:43:01.535065+0000","last_clean":"2021-02-10T04:43:01.535065+0000","last_became_active":"2021-02-10T04:33:05.141911+0000","last_became_peered":"2021-02-10T04:33:05.141911+0000","last_unstale":"2021-02-10T04:43:01.535065+0000","last_undegraded":"2021-02-10T04:43:01.535065+0000","last_fullsized":"2021-02-10T04:43:01.535065+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:25.075937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.075937+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"538'27","reported_seq":"894","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.332983+0000","last_change":"2021-02-10T04:37:39.100758+0000","last_active":"2021-02-10T04:43:03.332983+0000","last_peered":"2021-02-10T04:43:03.332983+0000","last_clean":"2021-02-10T04:43:03.332983+0000","last_became_active":"2021-02-10T04:37:37.257293+0000","last_became_peered":"2021-02-10T04:37:37.257293+0000","last_unstale":"2021-02-10T04:43:03.332983+0000","last_undegraded":"2021-02-10T04:43:03.332983+0000","last_fullsized":"2021-02-10T04:43:03.332983+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:28:46.680176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:46.680176+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"941","reported_epoch":"962","state":"active+clean","last_fresh":"2021-02-10T04:43:02.533995+0000","last_change":"2021-02-10T04:38:32.193334+0000","last_active":"2021-02-10T04:43:02.533995+0000","last_peered":"2021-02-10T04:43:02.533995+0000","last_clean":"2021-02-10T04:43:02.533995+0000","last_became_active":"2021-02-10T04:25:36.336442+0000","last_became_peered":"2021-02-10T04:25:36.336442+0000","last_unstale":"2021-02-10T04:43:02.533995+0000","last_undegraded":"2021-02-10T04:43:02.533995+0000","last_fullsized":"2021-02-10T04:43:02.533995+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:38:32.193240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:32.193240+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"605","reported_epoch":"960","state":"active+clean","last_fresh":"2021-02-10T04:43:00.522167+0000","last_change":"2021-02-10T04:33:05.147757+0000","last_active":"2021-02-10T04:43:00.522167+0000","last_peered":"2021-02-10T04:43:00.522167+0000","last_clean":"2021-02-10T04:43:00.522167+0000","last_became_active":"2021-02-10T04:33:05.146838+0000","last_became_peered":"2021-02-10T04:33:05.146838+0000","last_unstale":"2021-02-10T04:43:00.522167+0000","last_undegraded":"2021-02-10T04:43:00.522167+0000","last_fullsized":"2021-02-10T04:43:00.522167+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:22.078247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:22.078247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"920","reported_epoch":"960","state":"active+clean","last_fresh":"2021-02-10T04:43:00.522549+0000","last_change":"2021-02-10T04:29:31.903091+0000","last_active":"2021-02-10T04:43:00.522549+0000","last_peered":"2021-02-10T04:43:00.522549+0000","last_clean":"2021-02-10T04:43:00.522549+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:43:00.522549+0000","last_undegraded":"2021-02-10T04:43:00.522549+0000","last_fullsized":"2021-02-10T04:43:00.522549+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:31.902996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.902996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"538'28","reported_seq":"981","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.333481+0000","last_change":"2021-02-10T04:38:25.860322+0000","last_active":"2021-02-10T04:43:03.333481+0000","last_peered":"2021-02-10T04:43:03.333481+0000","last_clean":"2021-02-10T04:43:03.333481+0000","last_became_active":"2021-02-10T04:25:49.008997+0000","last_became_peered":"2021-02-10T04:25:49.008997+0000","last_unstale":"2021-02-10T04:43:03.333481+0000","last_undegraded":"2021-02-10T04:43:03.333481+0000","last_fullsized":"2021-02-10T04:43:03.333481+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'28","last_scrub_stamp":"2021-02-10T04:38:25.860247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:25.860247+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":40,"num_write":30,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"945","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.332630+0000","last_change":"2021-02-10T04:28:30.415204+0000","last_active":"2021-02-10T04:43:03.332630+0000","last_peered":"2021-02-10T04:43:03.332630+0000","last_clean":"2021-02-10T04:43:03.332630+0000","last_became_active":"2021-02-10T04:25:36.336893+0000","last_became_peered":"2021-02-10T04:25:36.336893+0000","last_unstale":"2021-02-10T04:43:03.332630+0000","last_undegraded":"2021-02-10T04:43:03.332630+0000","last_fullsized":"2021-02-10T04:43:03.332630+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:30.415134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:26:46.425288+0000","last_clean_scrub_stamp":"2021-02-10T04:28:30.415134+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"931","reported_epoch":"960","state":"active+clean","last_fresh":"2021-02-10T04:43:00.522146+0000","last_change":"2021-02-10T04:37:37.243679+0000","last_active":"2021-02-10T04:43:00.522146+0000","last_peered":"2021-02-10T04:43:00.522146+0000","last_clean":"2021-02-10T04:43:00.522146+0000","last_became_active":"2021-02-10T04:37:37.243275+0000","last_became_peered":"2021-02-10T04:37:37.243275+0000","last_unstale":"2021-02-10T04:43:00.522146+0000","last_undegraded":"2021-02-10T04:43:00.522146+0000","last_fullsized":"2021-02-10T04:43:00.522146+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:52.643099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:33:52.643099+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"922","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.333987+0000","last_change":"2021-02-10T04:29:26.355726+0000","last_active":"2021-02-10T04:43:03.333987+0000","last_peered":"2021-02-10T04:43:03.333987+0000","last_clean":"2021-02-10T04:43:03.333987+0000","last_became_active":"2021-02-10T04:25:53.147831+0000","last_became_peered":"2021-02-10T04:25:53.147831+0000","last_unstale":"2021-02-10T04:43:03.333987+0000","last_undegraded":"2021-02-10T04:43:03.333987+0000","last_fullsized":"2021-02-10T04:43:03.333987+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:26.355652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.355652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"538'31","reported_seq":"987","reported_epoch":"960","state":"active+clean","last_fresh":"2021-02-10T04:43:00.521749+0000","last_change":"2021-02-10T04:28:32.565457+0000","last_active":"2021-02-10T04:43:00.521749+0000","last_peered":"2021-02-10T04:43:00.521749+0000","last_clean":"2021-02-10T04:43:00.521749+0000","last_became_active":"2021-02-10T04:25:49.000911+0000","last_became_peered":"2021-02-10T04:25:49.000911+0000","last_unstale":"2021-02-10T04:43:00.521749+0000","last_undegraded":"2021-02-10T04:43:00.521749+0000","last_fullsized":"2021-02-10T04:43:00.521749+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2021-02-10T04:28:32.565374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:32.565374+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":62,"num_read_kb":46,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"857","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.534550+0000","last_change":"2021-02-10T04:37:37.242121+0000","last_active":"2021-02-10T04:43:01.534550+0000","last_peered":"2021-02-10T04:43:01.534550+0000","last_clean":"2021-02-10T04:43:01.534550+0000","last_became_active":"2021-02-10T04:37:37.241751+0000","last_became_peered":"2021-02-10T04:37:37.241751+0000","last_unstale":"2021-02-10T04:43:01.534550+0000","last_undegraded":"2021-02-10T04:43:01.534550+0000","last_fullsized":"2021-02-10T04:43:01.534550+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:48.646802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:33:48.646802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"932","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.534171+0000","last_change":"2021-02-10T04:38:39.253459+0000","last_active":"2021-02-10T04:43:01.534171+0000","last_peered":"2021-02-10T04:43:01.534171+0000","last_clean":"2021-02-10T04:43:01.534171+0000","last_became_active":"2021-02-10T04:38:39.253122+0000","last_became_peered":"2021-02-10T04:38:39.253122+0000","last_unstale":"2021-02-10T04:43:01.534171+0000","last_undegraded":"2021-02-10T04:43:01.534171+0000","last_fullsized":"2021-02-10T04:43:01.534171+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:46.464108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:46.464108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"928","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.328319+0000","last_change":"2021-02-10T04:38:37.121756+0000","last_active":"2021-02-10T04:43:03.328319+0000","last_peered":"2021-02-10T04:43:03.328319+0000","last_clean":"2021-02-10T04:43:03.328319+0000","last_became_active":"2021-02-10T04:25:53.039844+0000","last_became_peered":"2021-02-10T04:25:53.039844+0000","last_unstale":"2021-02-10T04:43:03.328319+0000","last_undegraded":"2021-02-10T04:43:03.328319+0000","last_fullsized":"2021-02-10T04:43:03.328319+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:37.121654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:38:37.121654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"538'14","reported_seq":"954","reported_epoch":"960","state":"active+clean","last_fresh":"2021-02-10T04:43:00.521813+0000","last_change":"2021-02-10T04:30:13.089576+0000","last_active":"2021-02-10T04:43:00.521813+0000","last_peered":"2021-02-10T04:43:00.521813+0000","last_clean":"2021-02-10T04:43:00.521813+0000","last_became_active":"2021-02-10T04:25:49.013196+0000","last_became_peered":"2021-02-10T04:25:49.013196+0000","last_unstale":"2021-02-10T04:43:00.521813+0000","last_undegraded":"2021-02-10T04:43:00.521813+0000","last_fullsized":"2021-02-10T04:43:00.521813+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2021-02-10T04:30:13.089494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:30:13.089494+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":15,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"849","reported_epoch":"962","state":"active+clean","last_fresh":"2021-02-10T04:43:02.535759+0000","last_change":"2021-02-10T04:30:11.537208+0000","last_active":"2021-02-10T04:43:02.535759+0000","last_peered":"2021-02-10T04:43:02.535759+0000","last_clean":"2021-02-10T04:43:02.535759+0000","last_became_active":"2021-02-10T04:29:07.157944+0000","last_became_peered":"2021-02-10T04:29:07.157944+0000","last_unstale":"2021-02-10T04:43:02.535759+0000","last_undegraded":"2021-02-10T04:43:02.535759+0000","last_fullsized":"2021-02-10T04:43:02.535759+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:11.537136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.537136+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"842","reported_epoch":"960","state":"active+clean","last_fresh":"2021-02-10T04:43:00.521833+0000","last_change":"2021-02-10T04:29:07.146984+0000","last_active":"2021-02-10T04:43:00.521833+0000","last_peered":"2021-02-10T04:43:00.521833+0000","last_clean":"2021-02-10T04:43:00.521833+0000","last_became_active":"2021-02-10T04:29:07.146525+0000","last_became_peered":"2021-02-10T04:29:07.146525+0000","last_unstale":"2021-02-10T04:43:00.521833+0000","last_undegraded":"2021-02-10T04:43:00.521833+0000","last_fullsized":"2021-02-10T04:43:00.521833+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:46.493066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:46.493066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"921","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.533961+0000","last_change":"2021-02-10T04:29:22.475613+0000","last_active":"2021-02-10T04:43:01.533961+0000","last_peered":"2021-02-10T04:43:01.533961+0000","last_clean":"2021-02-10T04:43:01.533961+0000","last_became_active":"2021-02-10T04:25:53.042601+0000","last_became_peered":"2021-02-10T04:25:53.042601+0000","last_unstale":"2021-02-10T04:43:01.533961+0000","last_undegraded":"2021-02-10T04:43:01.533961+0000","last_fullsized":"2021-02-10T04:43:01.533961+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:22.475504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:22.475504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"538'38","reported_seq":"987","reported_epoch":"960","state":"active+clean","last_fresh":"2021-02-10T04:43:00.521251+0000","last_change":"2021-02-10T04:28:35.520489+0000","last_active":"2021-02-10T04:43:00.521251+0000","last_peered":"2021-02-10T04:43:00.521251+0000","last_clean":"2021-02-10T04:43:00.521251+0000","last_became_active":"2021-02-10T04:25:49.010481+0000","last_became_peered":"2021-02-10T04:25:49.010481+0000","last_unstale":"2021-02-10T04:43:00.521251+0000","last_undegraded":"2021-02-10T04:43:00.521251+0000","last_fullsized":"2021-02-10T04:43:00.521251+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'32","last_scrub_stamp":"2021-02-10T04:28:35.520410+0000","last_deep_scrub":"78'32","last_deep_scrub_stamp":"2021-02-10T04:27:09.170428+0000","last_clean_scrub_stamp":"2021-02-10T04:28:35.520410+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":56,"num_read_kb":37,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"933","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.333733+0000","last_change":"2021-02-10T04:29:29.355356+0000","last_active":"2021-02-10T04:43:03.333733+0000","last_peered":"2021-02-10T04:43:03.333733+0000","last_clean":"2021-02-10T04:43:03.333733+0000","last_became_active":"2021-02-10T04:25:34.327162+0000","last_became_peered":"2021-02-10T04:25:34.327162+0000","last_unstale":"2021-02-10T04:43:03.333733+0000","last_undegraded":"2021-02-10T04:43:03.333733+0000","last_fullsized":"2021-02-10T04:43:03.333733+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.355257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.355257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"606","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.533743+0000","last_change":"2021-02-10T04:33:05.169577+0000","last_active":"2021-02-10T04:43:01.533743+0000","last_peered":"2021-02-10T04:43:01.533743+0000","last_clean":"2021-02-10T04:43:01.533743+0000","last_became_active":"2021-02-10T04:33:05.169159+0000","last_became_peered":"2021-02-10T04:33:05.169159+0000","last_unstale":"2021-02-10T04:43:01.533743+0000","last_undegraded":"2021-02-10T04:43:01.533743+0000","last_fullsized":"2021-02-10T04:43:01.533743+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:40.098239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:40.098239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"1006","reported_epoch":"960","state":"active+clean","last_fresh":"2021-02-10T04:43:00.521661+0000","last_change":"2021-02-10T04:38:23.319098+0000","last_active":"2021-02-10T04:43:00.521661+0000","last_peered":"2021-02-10T04:43:00.521661+0000","last_clean":"2021-02-10T04:43:00.521661+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:43:00.521661+0000","last_undegraded":"2021-02-10T04:43:00.521661+0000","last_fullsized":"2021-02-10T04:43:00.521661+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:23.319023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:38:23.319023+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"538'56","reported_seq":"1082","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.330413+0000","last_change":"2021-02-10T04:38:21.961539+0000","last_active":"2021-02-10T04:43:03.330413+0000","last_peered":"2021-02-10T04:43:03.330413+0000","last_clean":"2021-02-10T04:43:03.330413+0000","last_became_active":"2021-02-10T04:37:37.266384+0000","last_became_peered":"2021-02-10T04:37:37.266384+0000","last_unstale":"2021-02-10T04:43:03.330413+0000","last_undegraded":"2021-02-10T04:43:03.330413+0000","last_fullsized":"2021-02-10T04:43:03.330413+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":641,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'56","last_scrub_stamp":"2021-02-10T04:38:21.961470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:21.961470+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":107,"num_read_kb":84,"num_write":57,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":890,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"943","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.338000+0000","last_change":"2021-02-10T04:29:28.719237+0000","last_active":"2021-02-10T04:43:03.338000+0000","last_peered":"2021-02-10T04:43:03.338000+0000","last_clean":"2021-02-10T04:43:03.338000+0000","last_became_active":"2021-02-10T04:25:36.336112+0000","last_became_peered":"2021-02-10T04:25:36.336112+0000","last_unstale":"2021-02-10T04:43:03.338000+0000","last_undegraded":"2021-02-10T04:43:03.338000+0000","last_fullsized":"2021-02-10T04:43:03.338000+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.719147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.719147+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"273","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.331560+0000","last_change":"2021-02-10T04:38:39.244331+0000","last_active":"2021-02-10T04:43:03.331560+0000","last_peered":"2021-02-10T04:43:03.331560+0000","last_clean":"2021-02-10T04:43:03.331560+0000","last_became_active":"2021-02-10T04:38:39.243965+0000","last_became_peered":"2021-02-10T04:38:39.243965+0000","last_unstale":"2021-02-10T04:43:03.331560+0000","last_undegraded":"2021-02-10T04:43:03.331560+0000","last_fullsized":"2021-02-10T04:43:03.331560+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:33.199521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:38:33.199521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"922","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.332537+0000","last_change":"2021-02-10T04:29:25.382883+0000","last_active":"2021-02-10T04:43:03.332537+0000","last_peered":"2021-02-10T04:43:03.332537+0000","last_clean":"2021-02-10T04:43:03.332537+0000","last_became_active":"2021-02-10T04:25:53.150917+0000","last_became_peered":"2021-02-10T04:25:53.150917+0000","last_unstale":"2021-02-10T04:43:03.332537+0000","last_undegraded":"2021-02-10T04:43:03.332537+0000","last_fullsized":"2021-02-10T04:43:03.332537+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:25.382793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.382793+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.20","version":"0'0","reported_seq":"911","reported_epoch":"961","state":"active+clean","last_fresh":"2021-02-10T04:43:01.533460+0000","last_change":"2021-02-10T04:30:11.492843+0000","last_active":"2021-02-10T04:43:01.533460+0000","last_peered":"2021-02-10T04:43:01.533460+0000","last_clean":"2021-02-10T04:43:01.533460+0000","last_became_active":"2021-02-10T04:29:07.148937+0000","last_became_peered":"2021-02-10T04:29:07.148937+0000","last_unstale":"2021-02-10T04:43:01.533460+0000","last_undegraded":"2021-02-10T04:43:01.533460+0000","last_fullsized":"2021-02-10T04:43:01.533460+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:11.492788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:41.375276+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.492788+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":"426","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.328419+0000","last_change":"2021-02-10T04:37:37.247286+0000","last_active":"2021-02-10T04:43:03.328419+0000","last_peered":"2021-02-10T04:43:03.328419+0000","last_clean":"2021-02-10T04:43:03.328419+0000","last_became_active":"2021-02-10T04:37:37.246865+0000","last_became_peered":"2021-02-10T04:37:37.246865+0000","last_unstale":"2021-02-10T04:43:03.328419+0000","last_undegraded":"2021-02-10T04:43:03.328419+0000","last_fullsized":"2021-02-10T04:43:03.328419+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:21.708701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:21.708701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":"947","reported_epoch":"962","state":"active+clean","last_fresh":"2021-02-10T04:43:02.533669+0000","last_change":"2021-02-10T04:33:58.892594+0000","last_active":"2021-02-10T04:43:02.533669+0000","last_peered":"2021-02-10T04:43:02.533669+0000","last_clean":"2021-02-10T04:43:02.533669+0000","last_became_active":"2021-02-10T04:28:23.129832+0000","last_became_peered":"2021-02-10T04:28:23.129832+0000","last_unstale":"2021-02-10T04:43:02.533669+0000","last_undegraded":"2021-02-10T04:43:02.533669+0000","last_fullsized":"2021-02-10T04:43:02.533669+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:58.892504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:33:58.892504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":"887","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.332575+0000","last_change":"2021-02-10T04:28:48.409915+0000","last_active":"2021-02-10T04:43:03.332575+0000","last_peered":"2021-02-10T04:43:03.332575+0000","last_clean":"2021-02-10T04:43:03.332575+0000","last_became_active":"2021-02-10T04:28:23.129523+0000","last_became_peered":"2021-02-10T04:28:23.129523+0000","last_unstale":"2021-02-10T04:43:03.332575+0000","last_undegraded":"2021-02-10T04:43:03.332575+0000","last_fullsized":"2021-02-10T04:43:03.332575+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:48.409826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:48.409826+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":"891","reported_epoch":"962","state":"active+clean","last_fresh":"2021-02-10T04:43:02.535015+0000","last_change":"2021-02-10T04:34:01.895599+0000","last_active":"2021-02-10T04:43:02.535015+0000","last_peered":"2021-02-10T04:43:02.535015+0000","last_clean":"2021-02-10T04:43:02.535015+0000","last_became_active":"2021-02-10T04:28:23.126368+0000","last_became_peered":"2021-02-10T04:28:23.126368+0000","last_unstale":"2021-02-10T04:43:02.535015+0000","last_undegraded":"2021-02-10T04:43:02.535015+0000","last_fullsized":"2021-02-10T04:43:02.535015+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:34:01.895514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:34:01.895514+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.25","version":"53'2","reported_seq":"902","reported_epoch":"960","state":"active+clean","last_fresh":"2021-02-10T04:43:00.522000+0000","last_change":"2021-02-10T04:28:40.422198+0000","last_active":"2021-02-10T04:43:00.522000+0000","last_peered":"2021-02-10T04:43:00.522000+0000","last_clean":"2021-02-10T04:43:00.522000+0000","last_became_active":"2021-02-10T04:28:23.467770+0000","last_became_peered":"2021-02-10T04:28:23.467770+0000","last_unstale":"2021-02-10T04:43:00.522000+0000","last_undegraded":"2021-02-10T04:43:00.522000+0000","last_fullsized":"2021-02-10T04:43:00.522000+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:28:40.422115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:40.422115+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.26","version":"53'1","reported_seq":"347","reported_epoch":"960","state":"active+clean","last_fresh":"2021-02-10T04:43:00.522479+0000","last_change":"2021-02-10T04:37:37.233712+0000","last_active":"2021-02-10T04:43:00.522479+0000","last_peered":"2021-02-10T04:43:00.522479+0000","last_clean":"2021-02-10T04:43:00.522479+0000","last_became_active":"2021-02-10T04:37:37.231081+0000","last_became_peered":"2021-02-10T04:37:37.231081+0000","last_unstale":"2021-02-10T04:43:00.522479+0000","last_undegraded":"2021-02-10T04:43:00.522479+0000","last_fullsized":"2021-02-10T04:43:00.522479+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:29:22.801929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:22.801929+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":423,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":"918","reported_epoch":"960","state":"active+clean","last_fresh":"2021-02-10T04:43:00.522822+0000","last_change":"2021-02-10T04:38:37.219069+0000","last_active":"2021-02-10T04:43:00.522822+0000","last_peered":"2021-02-10T04:43:00.522822+0000","last_clean":"2021-02-10T04:43:00.522822+0000","last_became_active":"2021-02-10T04:29:07.145907+0000","last_became_peered":"2021-02-10T04:29:07.145907+0000","last_unstale":"2021-02-10T04:43:00.522822+0000","last_undegraded":"2021-02-10T04:43:00.522822+0000","last_fullsized":"2021-02-10T04:43:00.522822+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:37.218968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:37.218968+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":"887","reported_epoch":"963","state":"active+clean","last_fresh":"2021-02-10T04:43:03.332324+0000","last_change":"2021-02-10T04:29:23.440451+0000","last_active":"2021-02-10T04:43:03.332324+0000","last_peered":"2021-02-10T04:43:03.332324+0000","last_clean":"2021-02-10T04:43:03.332324+0000","last_became_active":"2021-02-10T04:28:23.129469+0000","last_became_peered":"2021-02-10T04:28:23.129469+0000","last_unstale":"2021-02-10T04:43:03.332324+0000","last_undegraded":"2021-02-10T04:43:03.332324+0000","last_fullsized":"2021-02-10T04:43:03.332324+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.440391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.440391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":"891","reported_epoch":"962","state":"active+clean","last_fresh":"2021-02-10T04:43:02.532977+0000","last_change":"2021-02-10T04:38:22.136377+0000","last_active":"2021-02-10T04:43:02.532977+0000","last_peered":"2021-02-10T04:43:02.532977+0000","last_clean":"2021-02-10T04:43:02.532977+0000","last_became_active":"2021-02-10T04:28:23.126795+0000","last_became_peered":"2021-02-10T04:28:23.126795+0000","last_unstale":"2021-02-10T04:43:02.532977+0000","last_undegraded":"2021-02-10T04:43:02.532977+0000","last_fullsized":"2021-02-10T04:43:02.532977+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:22.136290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_clean_scrub_stamp":"2021-02-10T04:38:22.136290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":951151276,"num_objects":389,"num_object_clones":339,"num_object_copies":1167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":389,"num_whiteouts":11,"num_read":7276,"num_read_kb":7428816,"num_write":5329,"num_write_kb":2021389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":291,"num_bytes_recovered":108939670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1937829888,"data_stored":2680896132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14623,"ondisk_log_size":14623,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":1945,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":18,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":6565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":126,"acting":126,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":333,"num_read_kb":328,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":1876,"num_read_kb":1349,"num_write":1020,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":8770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":75722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1068,"ondisk_log_size":1068,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21,"ondisk_log_size":21,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365012,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":204440,"kb_used_data":162448,"kb_used_omap":0,"kb_used_meta":41984,"kb_avail":93540712,"statfs":{"total":95995035648,"available":95785689088,"internally_reserved":0,"allocated":166346752,"data_stored":255665138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42991616},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:43:02 2021","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":1.679,"15min":1.482},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.46700000000000003},"max":{"1min":1.8009999999999999,"5min":67.277000000000001,"15min":72.805999999999997},"last":1.48},{"interface":"front","average":{"1min":0.96499999999999997,"5min":2.044,"15min":1.6970000000000001},"min":{"1min":0.56599999999999995,"5min":0.54200000000000004,"15min":0.50800000000000001},"max":{"1min":1.829,"5min":66.933999999999997,"15min":72.721999999999994},"last":1.5009999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:43:02 2021","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":1.054,"15min":1.133},"min":{"1min":0.59799999999999998,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.48,"5min":10.140000000000001,"15min":41.966000000000001},"last":1.3999999999999999},{"interface":"front","average":{"1min":0.85999999999999999,"5min":1.0069999999999999,"15min":1.1319999999999999},"min":{"1min":0.60599999999999998,"5min":0.57299999999999995,"15min":0.56499999999999995},"max":{"1min":1.234,"5min":7.1520000000000001,"15min":41.804000000000002},"last":0.80600000000000005}]},{"osd":2,"last update":"Wed Feb 10 04:43:02 2021","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":1.778,"15min":2.125},"min":{"1min":0.628,"5min":0.628,"15min":0.58299999999999996},"max":{"1min":1.375,"5min":66.534000000000006,"15min":125.714},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.2609999999999999,"5min":2.2269999999999999,"15min":2.3599999999999999},"min":{"1min":0.71099999999999997,"5min":0.66900000000000004,"15min":0.59099999999999997},"max":{"1min":2.218,"5min":66.781999999999996,"15min":125.53700000000001},"last":0.96199999999999997}]},{"osd":3,"last update":"Wed Feb 10 04:43:02 2021","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":2.1749999999999998,"15min":2.944},"min":{"1min":0.85699999999999998,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.728,"5min":68.305000000000007,"15min":192.95500000000001},"last":1.849},{"interface":"front","average":{"1min":1.143,"5min":2.0859999999999999,"15min":2.9209999999999998},"min":{"1min":0.82999999999999996,"5min":0.80400000000000005,"15min":0.72199999999999998},"max":{"1min":1.4890000000000001,"5min":69.331999999999994,"15min":192.601},"last":1.369}]},{"osd":4,"last update":"Wed Feb 10 04:43:02 2021","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.6220000000000001,"15min":1.6919999999999999},"min":{"1min":0.79800000000000004,"5min":0.70699999999999996,"15min":0.64600000000000002},"max":{"1min":2.383,"5min":10.781000000000001,"15min":97.733999999999995},"last":1.085},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.4119999999999999,"15min":1.639},"min":{"1min":0.78300000000000003,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.847,"5min":10.326000000000001,"15min":97.691000000000003},"last":1.536}]},{"osd":5,"last update":"Wed Feb 10 04:43:02 2021","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.7889999999999999,"15min":1.6080000000000001},"min":{"1min":0.92300000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.0350000000000001,"5min":35.283000000000001,"15min":35.283000000000001},"last":2.0569999999999999},{"interface":"front","average":{"1min":1.5169999999999999,"5min":2.2330000000000001,"15min":1.8959999999999999},"min":{"1min":0.997,"5min":0.86099999999999999,"15min":0.753},"max":{"1min":2.4860000000000002,"5min":35.353999999999999,"15min":35.353999999999999},"last":1.5740000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:43:02 2021","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":2.0150000000000001,"15min":1.97},"min":{"1min":1.028,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":7.79,"5min":11.343999999999999,"15min":32.616},"last":1.9490000000000001},{"interface":"front","average":{"1min":1.7370000000000001,"5min":1.8149999999999999,"15min":1.849},"min":{"1min":1.002,"5min":0.95699999999999996,"15min":0.79500000000000004},"max":{"1min":8.0739999999999998,"5min":10.428000000000001,"15min":32.424999999999997},"last":1.6619999999999999}]}]},{"osd":6,"up_from":43,"seq":184683593944,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":353404,"kb_used_data":310836,"kb_used_omap":0,"kb_used_meta":42560,"kb_avail":93391748,"statfs":{"total":95995035648,"available":95633149952,"internally_reserved":0,"allocated":318296064,"data_stored":418634632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43581440},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:42:54 2021","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":1.036,"15min":1.431},"min":{"1min":0.59899999999999998,"5min":0.47099999999999997,"15min":0.38800000000000001},"max":{"1min":1.419,"5min":13.393000000000001,"15min":64.164000000000001},"last":1.0609999999999999},{"interface":"front","average":{"1min":1.139,"5min":1.1439999999999999,"15min":1.8420000000000001},"min":{"1min":0.57799999999999996,"5min":0.5,"15min":0.47399999999999998},"max":{"1min":2.2589999999999999,"5min":8.0289999999999999,"15min":64.277000000000001},"last":1.3740000000000001}]},{"osd":1,"last update":"Wed Feb 10 04:42:54 2021","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.101,"15min":1.127},"min":{"1min":0.57299999999999995,"5min":0.54400000000000004,"15min":0.443},"max":{"1min":1.548,"5min":12.978999999999999,"15min":18.177},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.131,"15min":1.1539999999999999},"min":{"1min":0.67000000000000004,"5min":0.60899999999999999,"15min":0.52700000000000002},"max":{"1min":1.6779999999999999,"5min":3.3239999999999998,"15min":18.898},"last":1.1970000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:42:54 2021","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.268,"15min":1.3400000000000001},"min":{"1min":0.76900000000000002,"5min":0.55600000000000005,"15min":0.54100000000000004},"max":{"1min":2.101,"5min":2.6840000000000002,"15min":24.725000000000001},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.252,"15min":1.3280000000000001},"min":{"1min":0.78900000000000003,"5min":0.59899999999999998,"15min":0.55300000000000005},"max":{"1min":2.0230000000000001,"5min":2.1440000000000001,"15min":24.611999999999998},"last":1.591}]},{"osd":3,"last update":"Wed Feb 10 04:42:54 2021","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":2.7829999999999999,"15min":1.919},"min":{"1min":0.88900000000000001,"5min":0.76400000000000001,"15min":0.63200000000000001},"max":{"1min":2.137,"5min":114.17700000000001,"15min":114.17700000000001},"last":1.663},{"interface":"front","average":{"1min":1.3480000000000001,"5min":2.8820000000000001,"15min":1.944},"min":{"1min":0.87,"5min":0.84499999999999997,"15min":0.71699999999999997},"max":{"1min":2.0430000000000001,"5min":116.929,"15min":116.929},"last":1.706}]},{"osd":4,"last update":"Wed Feb 10 04:42:54 2021","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.502,"15min":1.3839999999999999},"min":{"1min":0.95599999999999996,"5min":0.64800000000000002,"15min":0.60599999999999998},"max":{"1min":2.4460000000000002,"5min":4.0810000000000004,"15min":7.5389999999999997},"last":1.774},{"interface":"front","average":{"1min":0.98699999999999999,"5min":1.135,"15min":1.0569999999999999},"min":{"1min":0.73799999999999999,"5min":0.68500000000000005,"15min":0.57999999999999996},"max":{"1min":1.853,"5min":3.2869999999999999,"15min":3.2869999999999999},"last":1.03}]},{"osd":5,"last update":"Wed Feb 10 04:42:54 2021","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.3959999999999999,"15min":1.3200000000000001},"min":{"1min":0.84199999999999997,"5min":0.68799999999999994,"15min":0.64100000000000001},"max":{"1min":2.1320000000000001,"5min":4.6369999999999996,"15min":11.380000000000001},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.544,"5min":1.5860000000000001,"15min":1.5049999999999999},"min":{"1min":0.97499999999999998,"5min":0.78000000000000003,"15min":0.66600000000000004},"max":{"1min":2.254,"5min":2.782,"15min":11.538},"last":2.0289999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:42:16 2021","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.425,"15min":1.3140000000000001},"min":{"1min":0.90700000000000003,"5min":0.755,"15min":0.755},"max":{"1min":2.089,"5min":4.5279999999999996,"15min":4.5279999999999996},"last":1.2889999999999999},{"interface":"front","average":{"1min":2.085,"5min":1.7669999999999999,"15min":1.593},"min":{"1min":1.0329999999999999,"5min":0.91000000000000003,"15min":0.77500000000000002},"max":{"1min":13.315,"5min":13.315,"15min":13.315},"last":1.823}]}]},{"osd":1,"up_from":13,"seq":55834575081,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":414024,"kb_used_data":360960,"kb_used_omap":0,"kb_used_meta":53056,"kb_avail":93331128,"statfs":{"total":95995035648,"available":95571075072,"internally_reserved":0,"allocated":369623040,"data_stored":481318605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54329344},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:42:23 2021","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.6359999999999999,"15min":1.456},"min":{"1min":0.55700000000000005,"5min":0.50600000000000001,"15min":0.47899999999999998},"max":{"1min":3.234,"5min":55.475999999999999,"15min":55.475999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":1.052,"5min":1.7609999999999999,"15min":1.546},"min":{"1min":0.58599999999999997,"5min":0.53000000000000003,"15min":0.51800000000000002},"max":{"1min":2.2970000000000002,"5min":55.368000000000002,"15min":55.368000000000002},"last":0.82199999999999995}]},{"osd":2,"last update":"Wed Feb 10 04:42:46 2021","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.464,"15min":1.4179999999999999},"min":{"1min":0.60599999999999998,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":2.7000000000000002,"5min":32.110999999999997,"15min":40.268000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.123,"5min":1.4419999999999999,"15min":1.4490000000000001},"min":{"1min":0.70699999999999996,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":2.895,"5min":32.302999999999997,"15min":40.651000000000003},"last":0.59899999999999998}]},{"osd":3,"last update":"Wed Feb 10 04:42:01 2021","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.8280000000000001,"15min":1.6180000000000001},"min":{"1min":0.76300000000000001,"5min":0.57899999999999996,"15min":0.45200000000000001},"max":{"1min":2.3319999999999999,"5min":32.432000000000002,"15min":40.314},"last":1.056},{"interface":"front","average":{"1min":1.321,"5min":1.806,"15min":1.617},"min":{"1min":0.77400000000000002,"5min":0.60399999999999998,"15min":0.47199999999999998},"max":{"1min":2.617,"5min":32.811,"15min":40.973999999999997},"last":1.006}]},{"osd":4,"last update":"Wed Feb 10 04:42:23 2021","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.26,"15min":1.2490000000000001},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":3.3849999999999998,"5min":3.7440000000000002,"15min":6.3099999999999996},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.383,"15min":1.349},"min":{"1min":0.89800000000000002,"5min":0.754,"15min":0.65700000000000003},"max":{"1min":3.0089999999999999,"5min":3.0089999999999999,"15min":6.2869999999999999},"last":1.3340000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:42:35 2021","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.331,"15min":1.508},"min":{"1min":0.747,"5min":0.71399999999999997,"15min":0.65200000000000002},"max":{"1min":4.6289999999999996,"5min":7.8049999999999997,"15min":48.686999999999998},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.53,"15min":1.7},"min":{"1min":0.84599999999999997,"5min":0.749,"15min":0.69699999999999995},"max":{"1min":3.4990000000000001,"5min":3.8809999999999998,"15min":48.905000000000001},"last":0.98799999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:42:46 2021","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":1.6539999999999999},"min":{"1min":0.82199999999999995,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":3.125,"5min":3.125,"15min":48.878999999999998},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.417,"5min":1.383,"15min":1.552},"min":{"1min":0.88800000000000001,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":4.7560000000000002,"5min":8.5169999999999995,"15min":49.011000000000003},"last":1.0860000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:42:01 2021","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.514,"15min":1.506},"min":{"1min":0.93999999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.7130000000000001,"5min":3.9990000000000001,"15min":5.0099999999999998},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.5649999999999999,"15min":1.542},"min":{"1min":1.1180000000000001,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":2.2839999999999998,"5min":3.823,"15min":6.1689999999999996},"last":1.119}]}]},{"osd":0,"up_from":8,"seq":34359738607,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":360884,"kb_used_data":315052,"kb_used_omap":0,"kb_used_meta":45824,"kb_avail":93384268,"statfs":{"total":95995035648,"available":95625490432,"internally_reserved":0,"allocated":322613248,"data_stored":457757961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":46923776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 04:42:14 2021","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":1.181,"15min":1.071},"min":{"1min":0.56000000000000005,"5min":0.50900000000000001,"15min":0.48099999999999998},"max":{"1min":1.454,"5min":28.946999999999999,"15min":28.946999999999999},"last":1.5660000000000001},{"interface":"front","average":{"1min":1.171,"5min":1.0549999999999999,"15min":0.98799999999999999},"min":{"1min":0.64200000000000002,"5min":0.52100000000000002,"15min":0.51200000000000001},"max":{"1min":3.7050000000000001,"5min":3.7050000000000001,"15min":7.5549999999999997},"last":0.59199999999999997}]},{"osd":2,"last update":"Wed Feb 10 04:42:47 2021","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.4330000000000001,"15min":1.3029999999999999},"min":{"1min":0.57399999999999995,"5min":0.51400000000000001,"15min":0.48099999999999998},"max":{"1min":2.786,"5min":25.984999999999999,"15min":25.984999999999999},"last":1.738},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.238,"15min":1.1559999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.51300000000000001},"max":{"1min":2.198,"5min":5.4930000000000003,"15min":16.829000000000001},"last":1.0860000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:42:04 2021","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.73,"15min":1.7869999999999999},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.46999999999999997},"max":{"1min":2.0649999999999999,"5min":29.692,"15min":126.587},"last":1.611},{"interface":"front","average":{"1min":1.6990000000000001,"5min":2.024,"15min":2.0659999999999998},"min":{"1min":0.89400000000000002,"5min":0.79400000000000004,"15min":0.54200000000000004},"max":{"1min":5.8369999999999997,"5min":29.814,"15min":126.88200000000001},"last":1.8540000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:42:14 2021","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.452,"15min":1.296},"min":{"1min":0.85799999999999998,"5min":0.67200000000000004,"15min":0.59399999999999997},"max":{"1min":3.214,"5min":5.1680000000000001,"15min":5.1680000000000001},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.379,"15min":1.325},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.63800000000000001},"max":{"1min":1.7509999999999999,"5min":14.672000000000001,"15min":14.672000000000001},"last":1.022}]},{"osd":5,"last update":"Wed Feb 10 04:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.556,"5min":1.79,"15min":1.5409999999999999},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":25.047000000000001,"5min":25.047000000000001,"15min":25.047000000000001},"last":1.1619999999999999},{"interface":"front","average":{"1min":2.8410000000000002,"5min":2.0670000000000002,"15min":1.7829999999999999},"min":{"1min":1.1000000000000001,"5min":0.93899999999999995,"15min":0.81699999999999995},"max":{"1min":24.972999999999999,"5min":28.827999999999999,"15min":28.827999999999999},"last":2.073}]},{"osd":6,"last update":"Wed Feb 10 04:42:59 2021","interfaces":[{"interface":"back","average":{"1min":2.6349999999999998,"5min":1.849,"15min":1.8580000000000001},"min":{"1min":1.05,"5min":0.89100000000000001,"15min":0.88800000000000001},"max":{"1min":24.837,"5min":24.837,"15min":35.872999999999998},"last":2.2989999999999999},{"interface":"front","average":{"1min":2.8340000000000001,"5min":1.869,"15min":1.984},"min":{"1min":1.1759999999999999,"5min":0.96099999999999997,"15min":0.83099999999999996},"max":{"1min":24.870999999999999,"5min":24.870999999999999,"15min":35.911000000000001},"last":1.97}]},{"osd":7,"last update":"Wed Feb 10 04:42:04 2021","interfaces":[{"interface":"back","average":{"1min":1.6359999999999999,"5min":1.6479999999999999,"15min":1.4770000000000001},"min":{"1min":1.0489999999999999,"5min":0.96899999999999997,"15min":0.73499999999999999},"max":{"1min":3.3330000000000002,"5min":5.6050000000000004,"15min":5.6050000000000004},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.518,"5min":1.7330000000000001,"15min":1.7},"min":{"1min":1.2270000000000001,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":2.0510000000000002,"5min":13.968999999999999,"15min":16.545000000000002},"last":1.3360000000000001}]}]},{"osd":2,"up_from":0,"seq":77309411560,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":25,"seq":107374182626,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":438872,"kb_used_data":386192,"kb_used_omap":0,"kb_used_meta":52672,"kb_avail":93306280,"statfs":{"total":95995035648,"available":95545630720,"internally_reserved":0,"allocated":395460608,"data_stored":572337153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53936128},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:42:05 2021","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.1319999999999999,"15min":1.069},"min":{"1min":0.59699999999999998,"5min":0.502,"15min":0.47599999999999998},"max":{"1min":2.113,"5min":7.899,"15min":17.233000000000001},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.171,"5min":1.1659999999999999,"15min":1.198},"min":{"1min":0.63200000000000001,"5min":0.52800000000000002,"15min":0.52300000000000002},"max":{"1min":2.4390000000000001,"5min":3.7330000000000001,"15min":20.216999999999999},"last":0.61099999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:42:05 2021","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.1579999999999999,"15min":1.0980000000000001},"min":{"1min":0.56899999999999995,"5min":0.56599999999999995,"15min":0.54400000000000004},"max":{"1min":2.073,"5min":8.9190000000000005,"15min":8.9190000000000005},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.171,"5min":1.145,"15min":1.034},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.52100000000000002},"max":{"1min":2.165,"5min":4.6769999999999996,"15min":5.4039999999999999},"last":1.0740000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:42:05 2021","interfaces":[{"interface":"back","average":{"1min":1.554,"5min":1.4359999999999999,"15min":1.2509999999999999},"min":{"1min":0.755,"5min":0.624,"15min":0.56799999999999995},"max":{"1min":3.29,"5min":5.024,"15min":5.024},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.276,"15min":1.1579999999999999},"min":{"1min":0.68200000000000005,"5min":0.60799999999999998,"15min":0.56999999999999995},"max":{"1min":2.5449999999999999,"5min":4.5880000000000001,"15min":4.5880000000000001},"last":0.749}]},{"osd":4,"last update":"Wed Feb 10 04:42:11 2021","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.4430000000000001,"15min":1.5860000000000001},"min":{"1min":0.90800000000000003,"5min":0.80700000000000005,"15min":0.629},"max":{"1min":2.4660000000000002,"5min":4.6909999999999998,"15min":30.751999999999999},"last":1.048},{"interface":"front","average":{"1min":1.6459999999999999,"5min":1.5469999999999999,"15min":1.6439999999999999},"min":{"1min":0.92700000000000005,"5min":0.85999999999999999,"15min":0.65700000000000003},"max":{"1min":2.7909999999999999,"5min":4.4699999999999998,"15min":30.928000000000001},"last":1.514}]},{"osd":5,"last update":"Wed Feb 10 04:42:34 2021","interfaces":[{"interface":"back","average":{"1min":1.611,"5min":1.621,"15min":1.873},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":2.948,"5min":3.5529999999999999,"15min":35.616999999999997},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.8129999999999999,"5min":1.752,"15min":1.946},"min":{"1min":1.0740000000000001,"5min":0.81799999999999995,"15min":0.77200000000000002},"max":{"1min":3.0640000000000001,"5min":4.931,"15min":35.661999999999999},"last":2.0790000000000002}]},{"osd":6,"last update":"Wed Feb 10 04:42:58 2021","interfaces":[{"interface":"back","average":{"1min":1.5800000000000001,"5min":1.6779999999999999,"15min":2.0190000000000001},"min":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.83699999999999997},"max":{"1min":2.7320000000000002,"5min":3.1739999999999999,"15min":35.518000000000001},"last":1.175},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.8500000000000001,"15min":2.2589999999999999},"min":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":0.86599999999999999},"max":{"1min":2.8220000000000001,"5min":5.0839999999999996,"15min":53.965000000000003},"last":1.873}]},{"osd":7,"last update":"Wed Feb 10 04:42:05 2021","interfaces":[{"interface":"back","average":{"1min":1.8260000000000001,"5min":1.8029999999999999,"15min":1.8320000000000001},"min":{"1min":1.212,"5min":1.143,"15min":0.99199999999999999},"max":{"1min":2.964,"5min":3.4380000000000002,"15min":25.242999999999999},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.917,"5min":1.9079999999999999,"15min":1.8879999999999999},"min":{"1min":1.1319999999999999,"5min":1.048,"15min":0.98299999999999998},"max":{"1min":3.1520000000000001,"5min":5.2549999999999999,"15min":25.286999999999999},"last":1.6499999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986399,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":153112,"kb_used_data":117456,"kb_used_omap":0,"kb_used_meta":35648,"kb_avail":93592040,"statfs":{"total":95995035648,"available":95838248960,"internally_reserved":0,"allocated":120274944,"data_stored":149357512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36503552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4],"upper_bound":2097152},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:42:18 2021","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":1.595,"15min":1.3500000000000001},"min":{"1min":0.51600000000000001,"5min":0.505,"15min":0.47099999999999997},"max":{"1min":1.5569999999999999,"5min":43.939999999999998,"15min":62.273000000000003},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.91000000000000003,"5min":1.615,"15min":1.4039999999999999},"min":{"1min":0.621,"5min":0.56599999999999995,"15min":0.499},"max":{"1min":1.3169999999999999,"5min":43.829999999999998,"15min":62.375999999999998},"last":0.76600000000000001}]},{"osd":1,"last update":"Wed Feb 10 04:42:18 2021","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":1.0669999999999999,"15min":0.91800000000000004},"min":{"1min":0.58299999999999996,"5min":0.53100000000000003,"15min":0.52800000000000002},"max":{"1min":1.869,"5min":24.295000000000002,"15min":24.295000000000002},"last":0.65500000000000003},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.4159999999999999,"15min":1.171},"min":{"1min":0.629,"5min":0.61099999999999999,"15min":0.59999999999999998},"max":{"1min":1.8029999999999999,"5min":24.943000000000001,"15min":24.943000000000001},"last":0.96799999999999997}]},{"osd":2,"last update":"Wed Feb 10 04:42:18 2021","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.1339999999999999,"15min":1.3879999999999999},"min":{"1min":0.63800000000000001,"5min":0.61399999999999999,"15min":0.56499999999999995},"max":{"1min":1.698,"5min":7.048,"15min":62.521000000000001},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.4650000000000001},"min":{"1min":0.79600000000000004,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.556,"5min":2.0009999999999999,"15min":62.572000000000003},"last":1.004}]},{"osd":3,"last update":"Wed Feb 10 04:42:18 2021","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.536,"15min":1.575},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.60699999999999998},"max":{"1min":2.2320000000000002,"5min":24.530999999999999,"15min":62.113},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.51,"15min":1.534},"min":{"1min":0.873,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.589,"5min":24.669,"15min":62.146000000000001},"last":0.79300000000000004}]},{"osd":5,"last update":"Wed Feb 10 04:42:35 2021","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.3740000000000001,"15min":1.351},"min":{"1min":0.76000000000000001,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.0139999999999998,"5min":3.3719999999999999,"15min":12.773999999999999},"last":1.478},{"interface":"front","average":{"1min":1.464,"5min":1.3169999999999999,"15min":1.2809999999999999},"min":{"1min":0.78800000000000003,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":2.5030000000000001,"5min":2.6709999999999998,"15min":13.038},"last":0.90900000000000003}]},{"osd":6,"last update":"Wed Feb 10 04:42:46 2021","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.95,"15min":1.762},"min":{"1min":0.82599999999999996,"5min":0.76600000000000001,"15min":0.70299999999999996},"max":{"1min":2.2799999999999998,"5min":42.088999999999999,"15min":42.088999999999999},"last":1.101},{"interface":"front","average":{"1min":1.573,"5min":2.0030000000000001,"15min":1.8660000000000001},"min":{"1min":0.97099999999999997,"5min":0.86799999999999999,"15min":0.83899999999999997},"max":{"1min":2.165,"5min":42.027000000000001,"15min":42.027000000000001},"last":1.7609999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:42:08 2021","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.4610000000000001,"15min":1.419},"min":{"1min":0.94799999999999995,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":2.395,"5min":6.9199999999999999,"15min":6.9199999999999999},"last":1.431},{"interface":"front","average":{"1min":1.609,"5min":1.591,"15min":1.5269999999999999},"min":{"1min":1.1799999999999999,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":3.177,"5min":4.1399999999999997,"15min":4.1399999999999997},"last":1.6359999999999999}]}]},{"osd":5,"up_from":37,"seq":158913790172,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":260820,"kb_used_data":216524,"kb_used_omap":0,"kb_used_meta":44288,"kb_avail":93484332,"statfs":{"total":95995035648,"available":95727955968,"internally_reserved":0,"allocated":221720576,"data_stored":299338907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45350912},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:42:31 2021","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.97699999999999998,"15min":1.383},"min":{"1min":0.51800000000000002,"5min":0.47599999999999998,"15min":0.40500000000000003},"max":{"1min":1.5760000000000001,"5min":4.5720000000000001,"15min":104.04300000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.874,"5min":1.1659999999999999,"15min":1.462},"min":{"1min":0.54900000000000004,"5min":0.495,"15min":0.46400000000000002},"max":{"1min":1.851,"5min":15.539999999999999,"15min":107.776},"last":1.3280000000000001}]},{"osd":1,"last update":"Wed Feb 10 04:42:31 2021","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":1.0349999999999999,"15min":1.323},"min":{"1min":0.54700000000000004,"5min":0.54500000000000004,"15min":0.504},"max":{"1min":1.4590000000000001,"5min":5.29,"15min":103.904},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.216,"15min":1.4770000000000001},"min":{"1min":0.63900000000000001,"5min":0.57599999999999996,"15min":0.50800000000000001},"max":{"1min":1.1899999999999999,"5min":11.875999999999999,"15min":104.315},"last":0.89100000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:42:31 2021","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.6379999999999999,"15min":2.2480000000000002},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":2.5779999999999998,"5min":44.462000000000003,"15min":103.65600000000001},"last":0.82399999999999995},{"interface":"front","average":{"1min":1.246,"5min":1.6659999999999999,"15min":2.2240000000000002},"min":{"1min":0.79100000000000004,"5min":0.65000000000000002,"15min":0.63500000000000001},"max":{"1min":2.637,"5min":44.398000000000003,"15min":104.136},"last":0.91000000000000003}]},{"osd":3,"last update":"Wed Feb 10 04:42:31 2021","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.7370000000000001,"15min":2.2490000000000001},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.624},"max":{"1min":2.7000000000000002,"5min":44.609000000000002,"15min":106.18000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":1.357,"5min":1.9350000000000001,"15min":2.242},"min":{"1min":0.98299999999999998,"5min":0.752,"15min":0.64500000000000002},"max":{"1min":1.696,"5min":44.537999999999997,"15min":106.822},"last":1.0289999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:42:31 2021","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.3540000000000001,"15min":1.21},"min":{"1min":0.68400000000000005,"5min":0.57099999999999995,"15min":0.55400000000000005},"max":{"1min":1.7390000000000001,"5min":6.1639999999999997,"15min":6.1639999999999997},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.444,"5min":1.407,"15min":1.361},"min":{"1min":0.872,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":2.895,"5min":2.9830000000000001,"15min":8.1449999999999996},"last":0.76400000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:42:45 2021","interfaces":[{"interface":"back","average":{"1min":1.7070000000000001,"5min":1.6519999999999999,"15min":1.6419999999999999},"min":{"1min":0.77000000000000002,"5min":0.65800000000000003,"15min":0.63500000000000001},"max":{"1min":3.3780000000000001,"5min":6.0300000000000002,"15min":25.969000000000001},"last":1.216},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.581,"15min":1.498},"min":{"1min":0.80500000000000005,"5min":0.77900000000000003,"15min":0.74399999999999999},"max":{"1min":3.073,"5min":4.7960000000000003,"15min":25.902999999999999},"last":1.2709999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:42:09 2021","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.4830000000000001,"15min":1.3879999999999999},"min":{"1min":0.93100000000000005,"5min":0.752,"15min":0.752},"max":{"1min":1.641,"5min":3.0009999999999999,"15min":5.4770000000000003},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.5,"5min":1.6240000000000001,"15min":1.593},"min":{"1min":0.95299999999999996,"5min":0.92300000000000004,"15min":0.83499999999999996},"max":{"1min":2.0299999999999998,"5min":3.1040000000000001,"15min":8.7840000000000007},"last":0.98399999999999999}]}]}]}} 2021-02-10T04:43:23.613 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:43:23.680 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:43:23.720 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:43:23.810 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:23 smithi164 conmon[29288]: cluster 2021-02-10T04:43:22.253326+0000 mgr.y ( 2021-02-10T04:43:23.810 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:23 smithi164 conmon[29288]: mgr.14140) 898 : cluster [DBG] pgmap v1623: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 902 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 7 op/s 2021-02-10T04:43:23.810 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:23 smithi164 conmon[29288]: cluster 2021-02-10T04:43:22.838855+0000 osd.4 (osd.4) 1681 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:23.811 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:23 smithi164 conmon[29288]: cluster 2021-02-10T04:43:22. 2021-02-10T04:43:23.811 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:23 smithi164 conmon[29288]: 838864+0000 osd.4 (osd.4) 1682 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:23.811 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:23 smithi164 conmon[29288]: 2021-02-10T04:43:23.811 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:23 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:43:23.811 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:23 smithi164 conmon[29288]: 02-10T04:43:22.838870+0000 2021-02-10T04:43:23.811 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:23 smithi164 conmon[29288]: osd.4 (osd.4) 1683 : cluster [WRN] 2021-02-10T04:43:23.811 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:23 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:23.811 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:23 smithi164 conmon[29288]: cluster 2021-02-10T04:43: 2021-02-10T04:43:23.811 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:23 smithi164 conmon[29288]: 22.838876+0000 osd.4 (osd.4) 1684 : 2021-02-10T04:43:23.811 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:23 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:23.811 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:23 smithi164 conmon[29288]: cluster 2021-02-10T04:43:23.811 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:23 smithi164 conmon[29288]: 2021-02-10T04:43:23.331181+0000 2021-02-10T04:43:23.811 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:23 smithi164 conmon[29288]: mon.a (mon.0) 1973 : cluster [DBG] osdmap e983: 8 total, 8 up, 7 in 2021-02-10T04:43:23.812 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:23 smithi164 conmon[29288]: 2021-02-10T04:43:23.812 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:23 smithi164 conmon[33677]: debug 2021-02-10T04:43:23.808+0000 7fbec19a1700 -1 osd.4 983 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:23.849 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:43:23.859 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:23 smithi085 conmon[29319]: cluster 2021-02-10T04:43:22.253326+0000 mgr.y 2021-02-10T04:43:23.859 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:23 smithi085 conmon[29319]: (mgr.14140) 898 : cluster [DBG] pgmap v1623: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 902 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 7 op/s 2021-02-10T04:43:23.859 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:23 smithi085 conmon[29319]: cluster 2021-02-10T04:43:22.838855+0000 osd.4 (osd.4) 1681 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:23.859 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:23 smithi085 conmon[29319]: cluster 2021-02-10T04:43:22.838864+0000 osd.4 (osd.4) 1682 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:23.859 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:23 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:43:23.859 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:23 smithi085 conmon[29319]: 04:43:22.838870+0000 osd.4 (osd.4) 1683 : 2021-02-10T04:43:23.859 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:23 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:23.859 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:23 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:43:23.859 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:23 smithi085 conmon[29319]: T04:43:22.838876+0000 osd.4 (osd.4) 1684 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:23.859 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:23 smithi085 conmon[29319]: 2021-02-10T04:43:23.860 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:23 smithi085 conmon[29319]: cluster 2021-02-10T04:43:23.860 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:23 smithi085 conmon[29319]: 2021-02-10T04:43:23.331181+0000 mon.a (mon.0) 1973 2021-02-10T04:43:23.860 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:23 smithi085 conmon[29319]: : cluster [DBG] osdmap e983: 8 total, 8 up, 7 in 2021-02-10T04:43:23.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:23 smithi085 conmon[35961]: cluster 2021-02-10T04:43:22.253326+0000 mgr.y (mgr. 2021-02-10T04:43:23.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:23 smithi085 conmon[35961]: 14140) 898 : cluster [DBG] pgmap v1623: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 902 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 7 op/s 2021-02-10T04:43:23.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:23 smithi085 conmon[35961]: cluster 2021-02-10T04:43:22.838855+0000 osd.4 2021-02-10T04:43:23.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:23 smithi085 conmon[35961]: (osd.4) 1681 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:23.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:23 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:43:23.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:23 smithi085 conmon[35961]: 10T04:43:22.838864+0000 osd.4 (osd.4) 1682 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:23.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:23 smithi085 conmon[35961]: 2021-02-10T04:43:23.860 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:23 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:43:23.861 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:23 smithi085 conmon[35961]: -02-10T04:43:22.838870+0000 osd.4 (osd.4) 1683 : cluster [WRN] 2021-02-10T04:43:23.861 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:23 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:23.861 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:23 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:43:23.861 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:23 smithi085 conmon[35961]: -02-10T04:43:22.838876+0000 osd.4 (osd.4) 1684 2021-02-10T04:43:23.861 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:23 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:23.861 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:23 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:43:23.861 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:23 smithi085 conmon[35961]: 10T04:43:23.331181+0000 mon.a (mon 2021-02-10T04:43:23.861 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:23 smithi085 conmon[35961]: .0) 1973 : cluster [DBG] osdmap e983: 8 total, 8 up, 7 in 2021-02-10T04:43:23.861 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:43:23 smithi085 conmon[41540]: debug 2021-02-10T04:43:23.857+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:43:23.907 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:43:23.964 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:43:24.015 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:43:24.043 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:43:24.216 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:43:24.322 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:43:24.331 INFO:tasks.rados.rados.0.smithi085.stdout:3957: done (4 left) 2021-02-10T04:43:24.331 INFO:tasks.rados.rados.0.smithi085.stdout:3958: rollback oid 44 current snap is 458 2021-02-10T04:43:24.332 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 44 to 435 2021-02-10T04:43:24.332 INFO:tasks.rados.rados.0.smithi085.stdout:3959: read oid 9 snap 439 2021-02-10T04:43:24.332 INFO:tasks.rados.rados.0.smithi085.stdout:3959: expect (ObjNum 1134 snap 381 seq_num 1134) 2021-02-10T04:43:24.332 INFO:tasks.rados.rados.0.smithi085.stdout:3960: read oid 6 snap 393 2021-02-10T04:43:24.332 INFO:tasks.rados.rados.0.smithi085.stdout:3960: expect (ObjNum 1138 snap 382 seq_num 1138) 2021-02-10T04:43:24.332 INFO:tasks.rados.rados.0.smithi085.stdout:3961: snap_create 2021-02-10T04:43:24.336 INFO:tasks.rados.rados.0.smithi085.stdout:3958: finishing rollback tid 0 to smithi08575397-44 2021-02-10T04:43:24.337 INFO:tasks.rados.rados.0.smithi085.stdout:3958: finishing rollback tid 1 to smithi08575397-44 2021-02-10T04:43:24.337 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 1901 (ObjNum 0 snap 22030 seq_num 83357584) dirty dne 2021-02-10T04:43:24.345 INFO:tasks.rados.rados.0.smithi085.stdout:3960: expect (ObjNum 1138 snap 382 seq_num 1138) 2021-02-10T04:43:24.353 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:43:24.442 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:43:24 smithi085 conmon[45443]: debug 2021-02-10T04:43:24.442 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:43:24 smithi085 conmon[45443]: 2021-02-10T04:43:24.138+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:43:24.474 INFO:tasks.rados.rados.0.smithi085.stdout:3959: expect (ObjNum 1134 snap 381 seq_num 1134) 2021-02-10T04:43:24.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:24 smithi085 conmon[29319]: cluster 2021-02-10T04:43:23.810005+0000 osd.4 (osd.4) 1685 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:24.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:24 smithi085 conmon[29319]: cluster 2021-02-10T04:43:23.810014+0000 osd.4 (osd.4) 1686 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:24.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:24 smithi085 conmon[29319]: cluster 2021-02-10T04:43:23.810021+0000 osd.4 (osd.4) 1687 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:24.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:24 smithi085 conmon[29319]: cluster 2021-02-10T04:43:23.810027+0000 osd.4 (osd.4) 1688 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:24.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:24 smithi085 conmon[29319]: cluster 2021-02-10T04:43:24.331601+0000 mon.a (mon.0) 1974 : cluster [DBG] osdmap e984: 8 total, 8 up, 7 in 2021-02-10T04:43:24.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:24 smithi085 conmon[35961]: cluster 2021-02-10T04:43:24.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:24 smithi085 conmon[35961]: 2021-02-10T04:43:23 2021-02-10T04:43:24.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:24 smithi085 conmon[35961]: .810005+0000 osd.4 (osd.4) 1685 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:24.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:24 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:43:24.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:24 smithi085 conmon[35961]: T04:43:23.810014+0000 osd.4 (osd.4) 2021-02-10T04:43:24.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:24 smithi085 conmon[35961]: 1686 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:24.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:24 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:43:24.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:24 smithi085 conmon[35961]: 02-10T04:43:23.810021+0000 2021-02-10T04:43:24.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:24 smithi085 conmon[35961]: osd.4 (osd.4) 1687 : cluster 2021-02-10T04:43:24.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:24 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:24.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:24 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:43:24.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:24 smithi085 conmon[35961]: -02-10T04:43:23.810027 2021-02-10T04:43:24.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:24 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1688 : cluster 2021-02-10T04:43:24.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:24 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:24.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:24 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:43:24.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:24 smithi085 conmon[35961]: 04:43:24.331601+0000 mon.a (mon. 2021-02-10T04:43:24.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:24 smithi085 conmon[35961]: 0) 1974 : cluster [DBG] osdmap e984: 8 total, 8 up, 7 in 2021-02-10T04:43:25.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:24 smithi164 conmon[33677]: debug 2021-02-10T04:43:24.826+0000 7fbec19a1700 -1 osd.4 984 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:25.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:24 smithi164 conmon[29288]: cluster 2021-02-10T04:43:25.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:24 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:43:25.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:24 smithi164 conmon[29288]: :43:23.810005+0000 osd.4 (osd.4) 1685 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:25.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:24 smithi164 conmon[29288]: cluster 2021-02-10T04:43:25.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:24 smithi164 conmon[29288]: 2021-02-10T04:43:25.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:24 smithi164 conmon[29288]: 2021 2021-02-10T04:43:25.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:24 smithi164 conmon[29288]: - 2021-02-10T04:43:25.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:24 smithi164 conmon[29288]: 02 2021-02-10T04:43:25.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:24 smithi164 conmon[29288]: -10T04:43:23.810014+0000 osd.4 (osd.4) 2021-02-10T04:43:25.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:24 smithi164 conmon[29288]: 1686 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:25.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:24 smithi164 conmon[29288]: cluster 2021-02-10T04:43:25.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:24 smithi164 conmon[29288]: 2021-02-10 2021-02-10T04:43:25.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:24 smithi164 conmon[29288]: T04:43:23.810021+0000 osd.4 (osd.4) 1687 : cluster [WRN] 2021-02-10T04:43:25.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:24 smithi164 conmon[29288]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:25.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:24 smithi164 conmon[29288]: cluster 2021-02-10T04:43:25.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:24 smithi164 conmon[29288]: 2021-02-10T04:43:25.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:24 smithi164 conmon[29288]: 2021 2021-02-10T04:43:25.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:24 smithi164 conmon[29288]: -02-10T04:43:23.810027 2021-02-10T04:43:25.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:24 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1688 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:25.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:24 smithi164 conmon[29288]: cluster 2021-02-10T04:43:24.331601+0000 mon.a 2021-02-10T04:43:25.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:24 smithi164 conmon[29288]: (mon.0) 1974 : cluster [DBG] osdmap e984: 8 total, 8 up, 7 in 2021-02-10T04:43:25.334 INFO:tasks.rados.rados.0.smithi085.stdout:3958: done (7 left) 2021-02-10T04:43:25.334 INFO:tasks.rados.rados.0.smithi085.stdout:3959: done (6 left) 2021-02-10T04:43:25.334 INFO:tasks.rados.rados.0.smithi085.stdout:3960: done (5 left) 2021-02-10T04:43:25.334 INFO:tasks.rados.rados.0.smithi085.stdout:3961: done (4 left) 2021-02-10T04:43:25.334 INFO:tasks.rados.rados.0.smithi085.stdout:3962: delete oid 30 current snap is 459 2021-02-10T04:43:25.340 INFO:tasks.rados.rados.0.smithi085.stdout:3962: done (4 left) 2021-02-10T04:43:25.340 INFO:tasks.rados.rados.0.smithi085.stdout:3963: write oid 44 current snap is 459 2021-02-10T04:43:25.340 INFO:tasks.rados.rados.0.smithi085.stdout:3963: seq_num 1345 ranges {605415=488033,1656281=493458} 2021-02-10T04:43:25.347 INFO:tasks.rados.rados.0.smithi085.stdout:3963: writing smithi08575397-44 from 605415 to 1093448 tid 1 2021-02-10T04:43:25.353 INFO:tasks.rados.rados.0.smithi085.stdout:3963: writing smithi08575397-44 from 1656281 to 2149739 tid 2 2021-02-10T04:43:25.353 INFO:tasks.rados.rados.0.smithi085.stdout:3964: snap_create 2021-02-10T04:43:25.353 INFO:tasks.rados.rados.0.smithi085.stdout:3963: finishing write tid 1 to smithi08575397-44 2021-02-10T04:43:25.357 INFO:tasks.rados.rados.0.smithi085.stdout:3963: finishing write tid 2 to smithi08575397-44 2021-02-10T04:43:25.357 INFO:tasks.rados.rados.0.smithi085.stdout:3963: finishing write tid 3 to smithi08575397-44 2021-02-10T04:43:25.357 INFO:tasks.rados.rados.0.smithi085.stdout:3963: finishing write tid 4 to smithi08575397-44 2021-02-10T04:43:25.358 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 44 v 1904 (ObjNum 1345 snap 459 seq_num 1345) dirty exists 2021-02-10T04:43:25.358 INFO:tasks.rados.rados.0.smithi085.stdout:3963: left oid 44 (ObjNum 1345 snap 459 seq_num 1345) 2021-02-10T04:43:25.386 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:43:25.475 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:43:25 smithi164 conmon[37263]: debug 2021-02-10T04:43:25.196+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:43:25.577 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:43:25.611 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:43:25.699 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:43:25.760 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset noscrub 2021-02-10T04:43:25.837 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:25 smithi164 conmon[29288]: cluster 2021-02-10T04:43:24.201667+0000 osd.6 2021-02-10T04:43:25.837 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:25 smithi164 conmon[29288]: (osd.6) 86 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:43:25.837 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:25 smithi164 conmon[29288]: cluster 2021-02-10T04:43:24.202893+0000 osd.6 (osd.6) 87 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:43:25.837 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:25 smithi164 conmon[29288]: cluster 2021-02-10T04:43:24 2021-02-10T04:43:25.837 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:25 smithi164 conmon[29288]: .254069+0000 mgr.y (mgr.14140 2021-02-10T04:43:25.837 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:25 smithi164 conmon[29288]: ) 899 : cluster [DBG] pgmap v1626: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 880 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 9.6 MiB/s rd, 291 KiB/s wr, 9 op/s 2021-02-10T04:43:25.837 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:25 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:43:25.837 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:25 smithi164 conmon[29288]: 10T04:43:24.828468+0000 osd.4 (osd.4) 1689 : cluster 2021-02-10T04:43:25.837 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:25 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:25.837 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:25 smithi164 conmon[29288]: cluster 2021-02-10T04:43:24.828477+0000 osd.4 (osd.4) 1690 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:25.838 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:25 smithi164 conmon[29288]: cluster 2021-02-10T04:43 2021-02-10T04:43:25.838 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:25 smithi164 conmon[29288]: :24.828482+0000 osd.4 (osd.4) 1691 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:25.838 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:25 smithi164 conmon[29288]: 2021-02-10T04:43:25.838 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:25 smithi164 conmon[29288]: cluster 2021-02-10T04:43:25.838 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:25 smithi164 conmon[29288]: 2021-02-10T04:43:24.828489+0000 osd.4 2021-02-10T04:43:25.838 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:25 smithi164 conmon[29288]: (osd.4) 1692 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:25.838 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:25 smithi164 conmon[29288]: cluster 2021-02-10T04:43:25.838 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:25 smithi164 conmon[29288]: 2021-02-10T04:43:25.333022 2021-02-10T04:43:25.838 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:25 smithi164 conmon[29288]: +0000 mon.a (mon.0) 1975 : cluster [DBG] 2021-02-10T04:43:25.838 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:25 smithi164 conmon[29288]: osdmap e985: 8 total, 8 up, 7 in 2021-02-10T04:43:25.838 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:25 smithi164 conmon[33677]: debug 2021-02-10T04:43:25.832+0000 7fbec19a1700 -1 osd.4 985 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:25.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[29319]: cluster 2021-02-10T04:43:24.201667+0000 osd.6 (osd.6) 86 : cluster 2021-02-10T04:43:25.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[29319]: [DBG] purged_snaps scrub starts 2021-02-10T04:43:25.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[29319]: cluster 2021-02-10T04:43:24.202893+0000 osd.6 (osd.6) 87 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:43:25.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[29319]: cluster 2021-02-10T04:43:24.254069+0000 mgr.y (mgr.14140) 899 : cluster [DBG] pgmap v1626: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 880 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 9.6 MiB/s rd, 291 KiB/s wr, 9 op/s 2021-02-10T04:43:25.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[29319]: cluster 2021-02-10T04:43:24.828468+0000 osd.4 (osd.4) 1689 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:25.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[29319]: cluster 2021-02-10T04:43:24.828477+0000 osd.4 (osd.4) 1690 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:25.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[29319]: cluster 2021-02-10T04:43:24.828482+0000 osd.4 (osd.4) 1691 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:25.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:43:25.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[29319]: -10T04:43:24.828489+0000 osd.4 (osd.4) 1692 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:25.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[29319]: cluster 2021-02-10T04:43:25.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[29319]: 2021-02-10T04:43:25.333022+0000 mon.a 2021-02-10T04:43:25.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[29319]: (mon.0) 1975 : cluster [DBG] 2021-02-10T04:43:25.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[29319]: osdmap e985: 8 total, 8 up, 7 in 2021-02-10T04:43:25.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[35961]: cluster 2021-02-10T04:43:24.201667+0000 osd.6 (osd. 2021-02-10T04:43:25.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[35961]: 6) 86 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:43:25.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[35961]: cluster 2021-02-10T04:43 2021-02-10T04:43:25.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[35961]: :24.202893+0000 osd.6 (osd.6) 87 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:43:25.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[35961]: 2021-02-10T04:43:25.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:43:25.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[35961]: 02-10T04:43:24.254069+0000 mgr.y ( 2021-02-10T04:43:25.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[35961]: mgr.14140) 899 : cluster [DBG] pgmap v1626: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 880 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 9.6 MiB/s rd, 291 KiB/s wr, 9 op/s 2021-02-10T04:43:25.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:43:25.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[35961]: 02-10T04:43:24.828468+0000 osd.4 ( 2021-02-10T04:43:25.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[35961]: osd.4) 1689 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:25.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:43:25.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[35961]: -10T04:43:24.828477+0000 osd.4 ( 2021-02-10T04:43:25.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[35961]: osd.4) 1690 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:25.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:43:25.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[35961]: 10T04:43:24.828482+0000 osd.4 (osd.4) 2021-02-10T04:43:25.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[35961]: 1691 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:25.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:43:25.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[35961]: -10T04:43:24.828489+0000 osd.4 (osd.4) 1692 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:25.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[35961]: cluster 2021-02-10T04:43:25.333022+0000 mon.a ( 2021-02-10T04:43:25.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:25 smithi085 conmon[35961]: mon.0) 1975 : cluster [DBG] osdmap e985: 8 total, 8 up, 7 in 2021-02-10T04:43:25.993 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:43:26.053 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:43:26.143 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:43:26 smithi164 conmon[37263]: debug 2021-02-10T04:43:26.051+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:43:26.301 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:43:26.336 INFO:tasks.rados.rados.0.smithi085.stdout:3963: done (5 left) 2021-02-10T04:43:26.336 INFO:tasks.rados.rados.0.smithi085.stdout:3964: done (4 left) 2021-02-10T04:43:26.336 INFO:tasks.rados.rados.0.smithi085.stdout:3965: snap_remove snap 453 2021-02-10T04:43:26.407 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:43:26.436 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:43:26.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:26 smithi164 conmon[33677]: debug 2021-02-10T04:43:26.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:26 smithi164 conmon[33677]: 2021-02-10T04:43:26.511+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:43:26.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:26 smithi164 conmon[33677]: 2021-02-10T04:43:26.807 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:43:26.911 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:43:26.939 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:26 smithi164 conmon[33677]: debug 2021-02-10T04:43:26.844+0000 7fbec19a1700 -1 osd.4 986 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:26.940 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[29319]: cluster 2021-02-10T04:43:26.940 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:43:26.940 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[29319]: 04:43:25.834286+0000 osd.4 (osd.4) 1693 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:26.940 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:43:26.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[29319]: 02-10T04:43:25.834294+0000 osd.4 (osd.4) 1694 : 2021-02-10T04:43:26.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:26.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[29319]: cluster 2021-02-10T04:43:26.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[29319]: 2021-02-10T04:43:25.834301+0000 2021-02-10T04:43:26.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[29319]: osd.4 (osd.4) 1695 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:26.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[29319]: cluster 2021-02-10T04:43:25 2021-02-10T04:43:26.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[29319]: .834308+0000 osd.4 (osd.4) 1696 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:26.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[29319]: cluster 2021-02-10T04:43:26.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[29319]: 2021-02 2021-02-10T04:43:26.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[29319]: -10T04:43:26.334631+0000 mon.a (mon.0) 1976 : cluster [DBG] 2021-02-10T04:43:26.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[29319]: osdmap e986: 8 total, 8 up, 7 in 2021-02-10T04:43:26.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[35961]: cluster 2021-02-10T04:43:25.834286+0000 osd.4 ( 2021-02-10T04:43:26.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[35961]: osd.4) 1693 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:26.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:43:26.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[35961]: -10T04:43:25.834294+0000 osd.4 (osd 2021-02-10T04:43:26.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[35961]: .4) 1694 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:26.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:43:26.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[35961]: 10T04:43:25.834301+0000 osd.4 (osd.4 2021-02-10T04:43:26.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[35961]: ) 1695 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:26.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:43:26.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[35961]: 04:43:25.834308+0000 2021-02-10T04:43:26.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[35961]: osd.4 (osd.4) 1696 : 2021-02-10T04:43:26.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:26.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[35961]: cluster 2021-02-10T04:43:26.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:43:26.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[35961]: 43:26.334631+0000 mon.a (mon.0) 2021-02-10T04:43:26.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:26 smithi085 conmon[35961]: 1976 : cluster [DBG] osdmap e986: 8 total, 8 up, 7 in 2021-02-10T04:43:26.943 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:26 smithi164 conmon[29288]: cluster 2021-02-10T04:43:25.834286+0000 osd.4 2021-02-10T04:43:26.943 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:26 smithi164 conmon[29288]: (osd.4) 1693 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:26.943 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:26 smithi164 conmon[29288]: cluster 2021-02-10T04:43:25.834294+0000 osd.4 (osd.4) 1694 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:26.943 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:26 smithi164 conmon[29288]: cluster 2021-02-10T04:43:25.834301+0000 osd.4 (osd.4) 1695 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:26.943 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:26 smithi164 conmon[29288]: cluster 2021-02-10T04:43:25.834308+0000 osd.4 (osd.4) 1696 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:26.943 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:26 smithi164 conmon[29288]: cluster 2021-02-10T04:43:26.334631+0000 mon.a (mon.0) 1976 : cluster [DBG] osdmap e986: 8 total, 8 up, 7 in 2021-02-10T04:43:26.944 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:43:27.338 INFO:tasks.rados.rados.0.smithi085.stdout:3965: done (4 left) 2021-02-10T04:43:27.339 INFO:tasks.rados.rados.0.smithi085.stdout:3966: read oid 15 snap -1 2021-02-10T04:43:27.339 INFO:tasks.rados.rados.0.smithi085.stdout:3966: expect (ObjNum 1323 snap 452 seq_num 1323) 2021-02-10T04:43:27.339 INFO:tasks.rados.rados.0.smithi085.stdout:3967: copy_from oid 31 from oid 10 current snap is 460 2021-02-10T04:43:27.339 INFO:tasks.rados.rados.0.smithi085.stdout:3968: write (excl) oid 30 current snap is 460 2021-02-10T04:43:27.339 INFO:tasks.rados.rados.0.smithi085.stdout:3968: seq_num 1346 ranges {622222=679635,1831116=753469} 2021-02-10T04:43:27.346 INFO:tasks.rados.rados.0.smithi085.stdout:3968: writing smithi08575397-30 from 622222 to 1301857 tid 1 2021-02-10T04:43:27.356 INFO:tasks.rados.rados.0.smithi085.stdout:3968: writing smithi08575397-30 from 1831116 to 2584585 tid 2 2021-02-10T04:43:27.356 INFO:tasks.rados.rados.0.smithi085.stdout:3969: snap_create 2021-02-10T04:43:27.356 INFO:tasks.rados.rados.0.smithi085.stdout:3966: expect (ObjNum 1323 snap 452 seq_num 1323) 2021-02-10T04:43:27.484 INFO:tasks.rados.rados.0.smithi085.stdout:3968: finishing write tid 1 to smithi08575397-30 2021-02-10T04:43:27.484 INFO:tasks.rados.rados.0.smithi085.stdout:3968: finishing write tid 2 to smithi08575397-30 2021-02-10T04:43:27.484 INFO:tasks.rados.rados.0.smithi085.stdout:3968: finishing write tid 3 to smithi08575397-30 2021-02-10T04:43:27.484 INFO:tasks.rados.rados.0.smithi085.stdout:3968: finishing write tid 4 to smithi08575397-30 2021-02-10T04:43:27.485 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 30 v 1905 (ObjNum 1346 snap 460 seq_num 1346) dirty exists 2021-02-10T04:43:27.485 INFO:tasks.rados.rados.0.smithi085.stdout:3968: left oid 30 (ObjNum 1346 snap 460 seq_num 1346) 2021-02-10T04:43:27.485 INFO:tasks.rados.rados.0.smithi085.stdout:3967: finishing copy_from to smithi08575397-31 2021-02-10T04:43:27.485 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 31 v 1622 (ObjNum 1309 snap 449 seq_num 1309) dirty exists 2021-02-10T04:43:27.485 INFO:tasks.rados.rados.0.smithi085.stdout:3967: finishing copy_from racing read to smithi08575397-31 2021-02-10T04:43:27.832 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:27 smithi164 conmon[33677]: debug 2021-02-10T04:43:27.828+0000 7fbec19a1700 -1 osd.4 987 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:32.967 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[29319]: cluster 2021-02-10T04:43:26.254671 2021-02-10T04:43:32.967 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 900 : cluster [DBG] pgmap v1629: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 874 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 1.7 MiB/s wr, 14 op/s 2021-02-10T04:43:32.967 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:43:32.967 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[29319]: T04:43:26.845952+0000 osd.4 (osd.4) 1697 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:32.967 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[29319]: cluster 2021-02-10T04:43 2021-02-10T04:43:32.967 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[29319]: :26.845961+0000 2021-02-10T04:43:32.967 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[29319]: osd.4 (osd.4 2021-02-10T04:43:32.967 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[29319]: ) 1698 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:32.967 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:43:32.968 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[29319]: -10T04:43:26.845966+0000 osd.4 ( 2021-02-10T04:43:32.968 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[29319]: osd.4) 1699 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:32.968 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:43:32.968 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[29319]: 10T04:43:26.845985+0000 osd.4 (osd 2021-02-10T04:43:32.968 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[29319]: .4) 1700 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:32.968 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:43:32.968 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[29319]: T04:43:27.336564 2021-02-10T04:43:32.968 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[29319]: +0000 mon.a (mon.0) 1977 : cluster [DBG] osdmap e987: 8 total, 8 up, 7 in 2021-02-10T04:43:32.968 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[29319]: cluster 2021-02-10T04:43:32.968 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[29319]: 2021-02-10T04:43:32.968 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[29319]: 2021 2021-02-10T04:43:32.968 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[29319]: -02-10T04:43: 2021-02-10T04:43:32.968 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[29319]: 27.830679+0000 osd.4 (osd.4) 1701 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:32.968 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:43:32.969 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[29319]: 10T04:43:27.830688+0000 osd.4 (osd. 2021-02-10T04:43:32.969 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[29319]: 4) 1702 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:32.969 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:43:32.969 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[29319]: -10T04:43:27.830695+0000 osd.4 ( 2021-02-10T04:43:32.969 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[29319]: osd.4) 1703 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:32.969 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[29319]: cluster 2021-02-10T04:43:32.969 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[29319]: 2021-02-10T04:43: 2021-02-10T04:43:32.969 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[29319]: 27.830701+0000 osd.4 (osd.4) 1704 : cluster 2021-02-10T04:43:32.969 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:32.969 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[29319]: cluster 2021-02-10T04:43:32.969 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[29319]: 2021-02-10T04:43:28. 2021-02-10T04:43:32.969 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[29319]: 327082+0000 mon.a (mon.0) 1978 2021-02-10T04:43:32.969 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[29319]: : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 832 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:43:32.970 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:43:32.970 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[29319]: -10T04:43:28.342811+0000 mon.a ( 2021-02-10T04:43:32.970 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[29319]: mon.0) 1979 : cluster [DBG] osdmap e988: 8 total, 8 up, 7 in 2021-02-10T04:43:32.970 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:29 smithi085 conmon[29319]: cluster 2021-02-10T04:43:28.255409+0000 mgr.y (mgr.14140) 901 : cluster [DBG] pgmap v1632: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 873 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 9.1 MiB/s rd, 1.5 MiB/s wr, 9 op/s 2021-02-10T04:43:32.970 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:29 smithi085 conmon[29319]: audit 2021-02-10T04:43:28.581060+0000 mgr.y (mgr.14140) 902 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:43:32.970 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:29 smithi085 conmon[29319]: cluster 2021-02-10T04:43:28.791036+0000 osd.4 (osd.4) 1705 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:32.970 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:29 smithi085 conmon[29319]: cluster 2021-02-10T04:43:28.791046+0000 osd.4 (osd.4) 1706 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:32.970 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:29 smithi085 conmon[29319]: cluster 2021-02-10T04:43:28.791052+0000 osd.4 (osd.4) 1707 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:32.970 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:29 smithi085 conmon[29319]: cluster 2021-02-10T04:43:28.791059+0000 osd.4 (osd.4) 1708 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:32.970 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:29 smithi085 conmon[29319]: cluster 2021-02-10T04:43:29.344920+0000 mon.a (mon.0) 1980 : cluster [DBG] osdmap e989: 8 total, 8 up, 7 in 2021-02-10T04:43:32.970 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[29319]: cluster 2021-02-10T04:43:32.970 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[29319]: 2021-02-10T04:43:32.970 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[29319]: 2021 2021-02-10T04:43:32.970 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[29319]: - 2021-02-10T04:43:32.971 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[29319]: 02 2021-02-10T04:43:32.971 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[29319]: -10T04:43:29.807095+0000 2021-02-10T04:43:32.971 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[29319]: osd.4 (osd.4) 1709 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:32.971 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:43:32.971 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[29319]: 02-10T04:43:29.807100+0000 osd.4 ( 2021-02-10T04:43:32.971 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[29319]: osd.4) 1710 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:32.971 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[29319]: cluster 2021-02-10T04:43:32.971 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:43:32.971 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[29319]: :43:29.807104+0000 osd.4 (osd.4) 1711 : cluster 2021-02-10T04:43:32.971 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:32.971 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:43:32.971 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[29319]: -10T04:43:29.807107+0000 osd.4 (osd. 2021-02-10T04:43:32.971 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[29319]: 4) 1712 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:32.972 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:43:32.972 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[29319]: -10T04:43:30.346492+0000 mon.a 2021-02-10T04:43:32.972 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[29319]: (mon.0) 1981 : cluster [DBG] osdmap e990: 8 total, 8 up, 7 in 2021-02-10T04:43:32.972 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[29319]: cluster 2021-02-10T04:43:30.256347+0000 2021-02-10T04:43:32.972 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[29319]: mgr.y (mgr.14140) 903 : cluster [DBG] pgmap v1635: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 872 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 8.2 MiB/s rd, 2.5 MiB/s wr, 17 op/s 2021-02-10T04:43:32.972 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:43:32.972 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[29319]: -10T04:43:30.837489+0000 osd.4 (osd.4) 1713 2021-02-10T04:43:32.972 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:32.972 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:43:32.972 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[29319]: -02-10T04:43:30.837498+0000 osd.4 (osd. 2021-02-10T04:43:32.972 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[29319]: 4) 1714 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:32.972 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:43:32.972 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[29319]: -02-10T04:43:30. 2021-02-10T04:43:32.972 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[29319]: 837503+0000 osd.4 (osd.4) 2021-02-10T04:43:32.973 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[29319]: 1715 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:32.973 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[29319]: cluster 2021-02-10T04:43:32.973 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[29319]: 2021-02-10T04:43:30 2021-02-10T04:43:32.973 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[29319]: .837508+0000 osd.4 (osd.4) 1716 : cluster 2021-02-10T04:43:32.973 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:32.973 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:43:32.973 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[29319]: 43:31.347777+0000 mon.a (mon.0) 1982 : cluster 2021-02-10T04:43:32.973 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[29319]: [DBG] osdmap e991: 8 total, 8 up, 7 in 2021-02-10T04:43:32.973 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[29319]: cluster 2021-02-10T04:43:31.720724+0000 osd.3 (osd 2021-02-10T04:43:32.973 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[29319]: .3) 90 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:43:32.973 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[29319]: cluster 2021-02-10T04:43:31.722589+0000 2021-02-10T04:43:32.973 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[29319]: osd.3 (osd.3) 91 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:43:32.973 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:43:32.973 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[29319]: 43:31.819784+0000 osd.4 (osd.4) 2021-02-10T04:43:32.974 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[29319]: 1717 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:32.974 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:43:32.974 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[29319]: 02-10T04:43:31.819792+0000 osd.4 (osd 2021-02-10T04:43:32.974 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[29319]: .4) 1718 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:32.974 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:43:32.974 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[29319]: T04:43:31.819799+0000 osd.4 (osd. 2021-02-10T04:43:32.974 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[29319]: 4) 1719 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:32.974 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:43:32.974 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[29319]: T04:43:31.819806+0000 osd.4 (osd.4 2021-02-10T04:43:32.974 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[29319]: ) 1720 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:32.974 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:43:32.974 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[29319]: 43:32.349140+0000 mon.a (mon.0) 1983 : cluster [DBG] osdmap e992: 8 total, 8 up, 7 in 2021-02-10T04:43:32.974 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[29319]: 2021-02-10T04:43:32.975 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[35961]: cluster 2021-02-10T04:43:32.975 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:43:32.975 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[35961]: T04:43:26.254671+0000 mgr.y (mgr.14140) 900 : cluster [DBG] pgmap v1629: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 874 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 1.7 MiB/s wr, 14 op/s 2021-02-10T04:43:32.975 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:43:32.975 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[35961]: 10T04:43:26.845952+0000 osd.4 (osd 2021-02-10T04:43:32.975 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[35961]: .4) 1697 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:32.975 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:43:32.975 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[35961]: -10T04:43:26.845961+0000 osd.4 (osd.4) 1698 : cluster 2021-02-10T04:43:32.975 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:32.975 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[35961]: cluster 2021-02-10T04:43: 2021-02-10T04:43:32.975 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[35961]: 26.845966+0000 osd.4 (osd.4) 1699 : 2021-02-10T04:43:32.975 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:32.976 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:43:32.976 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[35961]: 43:26.845985+0000 osd.4 (osd.4 2021-02-10T04:43:32.976 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[35961]: ) 1700 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:32.976 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:43:32.976 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[35961]: -02-10T04:43:27.336564+0000 mon.a ( 2021-02-10T04:43:32.976 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:27 smithi085 conmon[35961]: mon.0) 1977 : cluster [DBG] osdmap e987: 8 total, 8 up, 7 in 2021-02-10T04:43:32.976 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[35961]: cluster 2021-02-10T04:43:27.830679 2021-02-10T04:43:32.976 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1701 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:32.976 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[35961]: cluster 2021-02-10T04:43:32.976 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[35961]: 2021-02-10T04:43:27.830688+0000 osd.4 (osd.4) 1702 : cluster 2021-02-10T04:43:32.976 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:32.976 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[35961]: cluster 2021-02-10T04:43:27.830695+0000 osd.4 ( 2021-02-10T04:43:32.976 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[35961]: osd.4) 1703 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:32.976 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:43:32.977 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[35961]: T04:43:27.830701+0000 osd.4 ( 2021-02-10T04:43:32.977 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[35961]: osd.4) 1704 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:32.977 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:43:32.977 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[35961]: 02-10T04:43:28.327082+0000 mon.a 2021-02-10T04:43:32.977 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[35961]: (mon.0) 1978 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 832 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:43:32.977 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:43:32.977 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[35961]: -02-10T04:43:28.342811+0000 2021-02-10T04:43:32.977 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[35961]: mon.a (mon.0) 1979 : cluster [DBG] osdmap e988: 8 total, 8 up, 7 in 2021-02-10T04:43:32.977 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:29 smithi085 conmon[35961]: cluster 2021-02-10T04:43:28.255409 2021-02-10T04:43:32.977 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:29 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 901 : cluster [DBG] pgmap v1632: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 873 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 9.1 MiB/s rd, 1.5 MiB/s wr, 9 op/s 2021-02-10T04:43:32.977 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:29 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:43:32.977 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:29 smithi085 conmon[35961]: 04:43:28.581060+0000 mgr.y (mgr.14140) 902 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:43:32.977 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:29 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:43:32.978 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:29 smithi085 conmon[35961]: 02-10T04:43:28.791036+0000 osd.4 2021-02-10T04:43:32.978 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:29 smithi085 conmon[35961]: (osd.4) 1705 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:32.978 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:29 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:43:32.978 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:29 smithi085 conmon[35961]: 02-10T04:43:28.791046+0000 2021-02-10T04:43:32.978 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:29 smithi085 conmon[35961]: osd.4 (osd.4) 1706 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:32.978 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:29 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:43:32.978 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:29 smithi085 conmon[35961]: -02-10T04:43:28.791052+0000 2021-02-10T04:43:32.978 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:29 smithi085 conmon[35961]: osd.4 (osd.4) 1707 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:32.978 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:29 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:43:32.978 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:29 smithi085 conmon[35961]: 02-10T04:43:28.791059+0000 osd.4 ( 2021-02-10T04:43:32.978 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:29 smithi085 conmon[35961]: osd.4) 1708 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:32.978 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:29 smithi085 conmon[35961]: cluster 2021-02-10T04:43:32.978 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:29 smithi085 conmon[35961]: 2021-02-10T04:43:29.344920+0000 2021-02-10T04:43:32.978 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:29 smithi085 conmon[35961]: mon.a (mon.0) 1980 : cluster [DBG] 2021-02-10T04:43:32.979 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:29 smithi085 conmon[35961]: osdmap e989: 8 total, 8 up, 7 in 2021-02-10T04:43:32.979 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[35961]: cluster 2021-02-10T04:43:29. 2021-02-10T04:43:32.979 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[35961]: 807095+0000 osd.4 (osd.4) 1709 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:32.979 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:43:32.979 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[35961]: 02-10T04:43:29.807100+0000 osd.4 (osd.4) 1710 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:32.979 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[35961]: 2021-02-10T04:43:32.979 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[35961]: cluster 2021-02-10T04:43:32.979 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[35961]: 2021-02-10T04:43:29.807104+0000 osd.4 (osd. 2021-02-10T04:43:32.979 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[35961]: 4) 1711 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:32.979 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:43:32.979 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[35961]: 04:43:29.807107+0000 osd.4 (osd.4) 2021-02-10T04:43:32.979 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[35961]: 1712 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:32.979 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:43:32.979 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[35961]: 04:43:30.346492+0000 mon.a (mon.0 2021-02-10T04:43:32.980 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:30 smithi085 conmon[35961]: ) 1981 : cluster [DBG] osdmap e990: 8 total, 8 up, 7 in 2021-02-10T04:43:32.980 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[35961]: cluster 2021-02-10T04:43:30.256347+0000 2021-02-10T04:43:32.980 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[35961]: mgr.y (mgr.14140) 903 : cluster [DBG] pgmap v1635: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 872 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 8.2 MiB/s rd, 2.5 MiB/s wr, 17 op/s 2021-02-10T04:43:32.980 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[35961]: cluster 2021-02-10T04:43:32.980 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[35961]: 2021-02-10T04:43:30.837489+0000 osd.4 (osd.4) 1713 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:32.980 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[35961]: 2021-02-10T04:43:32.980 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[35961]: cluster 2021-02-10T04:43:32.980 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:43:32.980 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[35961]: 43:30.837498+0000 osd.4 (osd.4) 1714 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:32.980 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:43:32.980 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[35961]: -10T04:43:30.837503+0000 osd.4 (osd. 2021-02-10T04:43:32.980 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[35961]: 4) 1715 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:32.980 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[35961]: cluster 2021-02-10T04:43:32.981 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[35961]: 2021-02-10T04:43:30.837508+0000 osd.4 (osd.4 2021-02-10T04:43:32.981 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[35961]: ) 1716 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:32.981 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:43:32.981 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[35961]: 02-10T04:43:31.347777+0000 2021-02-10T04:43:32.981 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[35961]: mon.a (mon.0) 1982 : cluster [DBG] osdmap e991: 8 total, 8 up, 7 in 2021-02-10T04:43:32.981 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:31 smithi085 conmon[35961]: 2021-02-10T04:43:32.981 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[35961]: cluster 2021-02-10T04:43:31.720724 2021-02-10T04:43:32.981 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[35961]: +0000 osd.3 (osd.3) 90 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:43:32.981 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[35961]: cluster 2021-02-10T04:43:31 2021-02-10T04:43:32.981 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[35961]: .722589+0000 osd.3 (osd.3) 91 : cluster 2021-02-10T04:43:32.981 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[35961]: [DBG] purged_snaps scrub ok 2021-02-10T04:43:32.981 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:43:32.981 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[35961]: 43:31.819784+0000 osd.4 (osd.4 2021-02-10T04:43:32.981 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[35961]: ) 1717 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:32.982 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:43:32.982 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[35961]: 04:43:31.819792+0000 osd.4 (osd. 2021-02-10T04:43:32.982 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[35961]: 4) 1718 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:32.982 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:43:32.982 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[35961]: -10T04:43:31.819799+0000 osd.4 2021-02-10T04:43:32.982 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[35961]: (osd.4) 1719 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:32.982 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[35961]: 2021-02-10T04:43:32.982 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[35961]: cluster 2021-02-10T04:43:32.982 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[35961]: 2021-02-10T04:43:31.819806+0000 osd.4 (osd. 2021-02-10T04:43:32.982 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[35961]: 4) 1720 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:32.982 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:43:32.982 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[35961]: 10T04:43:32.349140+0000 mon.a ( 2021-02-10T04:43:32.982 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:32 smithi085 conmon[35961]: mon.0) 1983 : cluster [DBG] osdmap e992: 8 total, 8 up, 7 in 2021-02-10T04:43:32.983 INFO:tasks.rados.rados.0.smithi085.stdout:3966: done (7 left) 2021-02-10T04:43:32.983 INFO:tasks.rados.rados.0.smithi085.stdout:3967: done (6 left) 2021-02-10T04:43:32.983 INFO:tasks.rados.rados.0.smithi085.stdout:3968: done (5 left) 2021-02-10T04:43:32.983 INFO:tasks.rados.rados.0.smithi085.stdout:3969: done (4 left) 2021-02-10T04:43:32.983 INFO:tasks.rados.rados.0.smithi085.stdout:3970: write (excl) oid 4 current snap is 461 2021-02-10T04:43:32.983 INFO:tasks.rados.rados.0.smithi085.stdout:3970: seq_num 1347 ranges {639029=754884,2173245=465784} 2021-02-10T04:43:32.983 INFO:tasks.rados.rados.0.smithi085.stdout:3970: writing smithi08575397-4 from 639029 to 1393913 tid 1 2021-02-10T04:43:32.983 INFO:tasks.rados.rados.0.smithi085.stdout:3970: writing smithi08575397-4 from 2173245 to 2639029 tid 2 2021-02-10T04:43:32.983 INFO:tasks.rados.rados.0.smithi085.stdout:3971: write (excl) oid 5 current snap is 461 2021-02-10T04:43:32.983 INFO:tasks.rados.rados.0.smithi085.stdout:3971: seq_num 1348 ranges {655836=430133,1831727=757138} 2021-02-10T04:43:32.983 INFO:tasks.rados.rados.0.smithi085.stdout:3971: writing smithi08575397-5 from 655836 to 1085969 tid 1 2021-02-10T04:43:32.983 INFO:tasks.rados.rados.0.smithi085.stdout:3971: writing smithi08575397-5 from 1831727 to 2588865 tid 2 2021-02-10T04:43:32.984 INFO:tasks.rados.rados.0.smithi085.stdout:3972: rollback oid 43 current snap is 461 2021-02-10T04:43:32.984 INFO:tasks.rados.rados.0.smithi085.stdout:rollback oid 43 to 442 2021-02-10T04:43:32.984 INFO:tasks.rados.rados.0.smithi085.stdout:3973: copy_from oid 38 from oid 30 current snap is 461 2021-02-10T04:43:32.984 INFO:tasks.rados.rados.0.smithi085.stdout:3974: delete oid 20 current snap is 461 2021-02-10T04:43:32.984 INFO:tasks.rados.rados.0.smithi085.stdout:3971: finishing write tid 1 to smithi08575397-5 2021-02-10T04:43:32.984 INFO:tasks.rados.rados.0.smithi085.stdout:3970: finishing write tid 1 to smithi08575397-4 2021-02-10T04:43:32.984 INFO:tasks.rados.rados.0.smithi085.stdout:3974: done (8 left) 2021-02-10T04:43:32.984 INFO:tasks.rados.rados.0.smithi085.stdout:3975: delete oid 15 current snap is 461 2021-02-10T04:43:32.984 INFO:tasks.rados.rados.0.smithi085.stdout:3972: finishing rollback tid 0 to smithi08575397-43 2021-02-10T04:43:32.984 INFO:tasks.rados.rados.0.smithi085.stdout:3972: finishing rollback tid 1 to smithi08575397-43 2021-02-10T04:43:32.984 INFO:tasks.rados.rados.0.smithi085.stdout:3972: finishing rollback tid 2 to smithi08575397-43 2021-02-10T04:43:32.984 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 43 v 1911 (ObjNum 1129 snap 381 seq_num 1129) dirty exists 2021-02-10T04:43:32.984 INFO:tasks.rados.rados.0.smithi085.stdout:3972: done (8 left) 2021-02-10T04:43:32.984 INFO:tasks.rados.rados.0.smithi085.stdout:3975: done (7 left) 2021-02-10T04:43:32.985 INFO:tasks.rados.rados.0.smithi085.stdout:3976: read oid 37 snap -1 2021-02-10T04:43:32.985 INFO:tasks.rados.rados.0.smithi085.stdout:3976: expect (ObjNum 1129 snap 381 seq_num 1129) 2021-02-10T04:43:32.985 INFO:tasks.rados.rados.0.smithi085.stdout:3977: write (excl) oid 40 current snap is 461 2021-02-10T04:43:32.985 INFO:tasks.rados.rados.0.smithi085.stdout:3977: seq_num 1351 ranges {706257=655880,1890820=504465} 2021-02-10T04:43:32.985 INFO:tasks.rados.rados.0.smithi085.stdout:3977: writing smithi08575397-40 from 706257 to 1362137 tid 1 2021-02-10T04:43:32.985 INFO:tasks.rados.rados.0.smithi085.stdout:3977: writing smithi08575397-40 from 1890820 to 2395285 tid 2 2021-02-10T04:43:32.985 INFO:tasks.rados.rados.0.smithi085.stdout:3978: delete oid 6 current snap is 461 2021-02-10T04:43:32.985 INFO:tasks.rados.rados.0.smithi085.stdout:3971: finishing write tid 2 to smithi08575397-5 2021-02-10T04:43:32.985 INFO:tasks.rados.rados.0.smithi085.stdout:3970: finishing write tid 2 to smithi08575397-4 2021-02-10T04:43:32.985 INFO:tasks.rados.rados.0.smithi085.stdout:3970: finishing write tid 3 to smithi08575397-4 2021-02-10T04:43:32.985 INFO:tasks.rados.rados.0.smithi085.stdout:3970: finishing write tid 4 to smithi08575397-4 2021-02-10T04:43:32.985 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 4 v 1386 (ObjNum 1347 snap 461 seq_num 1347) dirty exists 2021-02-10T04:43:32.985 INFO:tasks.rados.rados.0.smithi085.stdout:3970: left oid 4 (ObjNum 1347 snap 461 seq_num 1347) 2021-02-10T04:43:32.985 INFO:tasks.rados.rados.0.smithi085.stdout:3976: expect (ObjNum 1129 snap 381 seq_num 1129) 2021-02-10T04:43:32.986 INFO:tasks.rados.rados.0.smithi085.stdout:3971: finishing write tid 3 to smithi08575397-5 2021-02-10T04:43:32.986 INFO:tasks.rados.rados.0.smithi085.stdout:3971: finishing write tid 4 to smithi08575397-5 2021-02-10T04:43:32.986 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 5 v 1911 (ObjNum 1348 snap 461 seq_num 1348) dirty exists 2021-02-10T04:43:32.986 INFO:tasks.rados.rados.0.smithi085.stdout:3971: left oid 5 (ObjNum 1348 snap 461 seq_num 1348) 2021-02-10T04:43:32.986 INFO:tasks.rados.rados.0.smithi085.stdout:3977: finishing write tid 1 to smithi08575397-40 2021-02-10T04:43:32.986 INFO:tasks.rados.rados.0.smithi085.stdout:3977: finishing write tid 2 to smithi08575397-40 2021-02-10T04:43:32.986 INFO:tasks.rados.rados.0.smithi085.stdout:3970: done (9 left) 2021-02-10T04:43:32.986 INFO:tasks.rados.rados.0.smithi085.stdout:3971: done (8 left) 2021-02-10T04:43:32.986 INFO:tasks.rados.rados.0.smithi085.stdout:3976: done (7 left) 2021-02-10T04:43:32.986 INFO:tasks.rados.rados.0.smithi085.stdout:3978: done (6 left) 2021-02-10T04:43:32.986 INFO:tasks.rados.rados.0.smithi085.stdout:3979: snap_create 2021-02-10T04:43:32.986 INFO:tasks.rados.rados.0.smithi085.stdout:3977: finishing write tid 3 to smithi08575397-40 2021-02-10T04:43:32.986 INFO:tasks.rados.rados.0.smithi085.stdout:3977: finishing write tid 4 to smithi08575397-40 2021-02-10T04:43:32.987 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 40 v 2004 (ObjNum 1351 snap 461 seq_num 1351) dirty exists 2021-02-10T04:43:32.987 INFO:tasks.rados.rados.0.smithi085.stdout:3977: left oid 40 (ObjNum 1351 snap 461 seq_num 1351) 2021-02-10T04:43:32.987 INFO:tasks.rados.rados.0.smithi085.stdout:3973: finishing copy_from to smithi08575397-38 2021-02-10T04:43:32.987 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 38 v 1905 (ObjNum 1346 snap 460 seq_num 1346) dirty exists 2021-02-10T04:43:32.987 INFO:tasks.rados.rados.0.smithi085.stdout:3973: finishing copy_from racing read to smithi08575397-38 2021-02-10T04:43:32.987 INFO:tasks.rados.rados.0.smithi085.stdout:3973: done (6 left) 2021-02-10T04:43:32.987 INFO:tasks.rados.rados.0.smithi085.stdout:3977: done (5 left) 2021-02-10T04:43:32.987 INFO:tasks.rados.rados.0.smithi085.stdout:3979: done (4 left) 2021-02-10T04:43:32.987 INFO:tasks.rados.rados.0.smithi085.stdout:3980: copy_from oid 13 from oid 16 current snap is 462 2021-02-10T04:43:32.987 INFO:tasks.rados.rados.0.smithi085.stdout:3981: write (excl) oid 49 current snap is 462 2021-02-10T04:43:32.987 INFO:tasks.rados.rados.0.smithi085.stdout:3981: seq_num 1353 ranges {739871=406378,1607784=508134} 2021-02-10T04:43:32.987 INFO:tasks.rados.rados.0.smithi085.stdout:3981: writing smithi08575397-49 from 739871 to 1146249 tid 1 2021-02-10T04:43:32.987 INFO:tasks.rados.rados.0.smithi085.stdout:3981: writing smithi08575397-49 from 1607784 to 2115918 tid 2 2021-02-10T04:43:32.987 INFO:tasks.rados.rados.0.smithi085.stdout:3982: snap_create 2021-02-10T04:43:32.988 INFO:tasks.rados.rados.0.smithi085.stdout:3980: finishing copy_from to smithi08575397-13 2021-02-10T04:43:32.988 INFO:tasks.rados.rados.0.smithi085.stdout:3980: got expected ENOENT (src dne) 2021-02-10T04:43:32.988 INFO:tasks.rados.rados.0.smithi085.stdout:3980: finishing copy_from racing read to smithi08575397-13 2021-02-10T04:43:32.988 INFO:tasks.rados.rados.0.smithi085.stdout:3981: finishing write tid 1 to smithi08575397-49 2021-02-10T04:43:32.988 INFO:tasks.rados.rados.0.smithi085.stdout:3981: finishing write tid 2 to smithi08575397-49 2021-02-10T04:43:32.988 INFO:tasks.rados.rados.0.smithi085.stdout:3981: finishing write tid 3 to smithi08575397-49 2021-02-10T04:43:32.988 INFO:tasks.rados.rados.0.smithi085.stdout:3981: finishing write tid 4 to smithi08575397-49 2021-02-10T04:43:32.988 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 49 v 1926 (ObjNum 1353 snap 462 seq_num 1353) dirty exists 2021-02-10T04:43:32.988 INFO:tasks.rados.rados.0.smithi085.stdout:3981: left oid 49 (ObjNum 1353 snap 462 seq_num 1353) 2021-02-10T04:43:32.988 INFO:tasks.rados.rados.0.smithi085.stdout:3980: done (6 left) 2021-02-10T04:43:32.988 INFO:tasks.rados.rados.0.smithi085.stdout:3981: done (5 left) 2021-02-10T04:43:32.988 INFO:tasks.rados.rados.0.smithi085.stdout:3982: done (4 left) 2021-02-10T04:43:32.988 INFO:tasks.rados.rados.0.smithi085.stdout:3983: copy_from oid 9 from oid 13 current snap is 463 2021-02-10T04:43:32.988 INFO:tasks.rados.rados.0.smithi085.stdout:3984: copy_from oid 26 from oid 11 current snap is 463 2021-02-10T04:43:32.989 INFO:tasks.rados.rados.0.smithi085.stdout:3985: write (excl) oid 21 current snap is 463 2021-02-10T04:43:32.989 INFO:tasks.rados.rados.0.smithi085.stdout:3985: seq_num 1354 ranges {756678=597980,1782619=768145} 2021-02-10T04:43:32.989 INFO:tasks.rados.rados.0.smithi085.stdout:3985: writing smithi08575397-21 from 756678 to 1354658 tid 1 2021-02-10T04:43:32.989 INFO:tasks.rados.rados.0.smithi085.stdout:3985: writing smithi08575397-21 from 1782619 to 2550764 tid 2 2021-02-10T04:43:32.989 INFO:tasks.rados.rados.0.smithi085.stdout:3986: read oid 42 snap -1 2021-02-10T04:43:32.989 INFO:tasks.rados.rados.0.smithi085.stdout:3986: expect (ObjNum 1297 snap 436 seq_num 1297) 2021-02-10T04:43:32.989 INFO:tasks.rados.rados.0.smithi085.stdout:3987: write oid 12 current snap is 463 2021-02-10T04:43:32.989 INFO:tasks.rados.rados.0.smithi085.stdout:3983: finishing copy_from to smithi08575397-9 2021-02-10T04:43:32.989 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 9 v 1576 (ObjNum 1241 snap 418 seq_num 1241) dirty exists 2021-02-10T04:43:32.989 INFO:tasks.rados.rados.0.smithi085.stdout:3985: finishing write tid 1 to smithi08575397-21 2021-02-10T04:43:32.989 INFO:tasks.rados.rados.0.smithi085.stdout:3983: finishing copy_from racing read to smithi08575397-9 2021-02-10T04:43:32.989 INFO:tasks.rados.rados.0.smithi085.stdout:3987: seq_num 1355 ranges {773485=673229,2124748=511803} 2021-02-10T04:43:32.989 INFO:tasks.rados.rados.0.smithi085.stdout:3987: writing smithi08575397-12 from 773485 to 1446714 tid 1 2021-02-10T04:43:32.989 INFO:tasks.rados.rados.0.smithi085.stdout:3987: writing smithi08575397-12 from 2124748 to 2636551 tid 2 2021-02-10T04:43:32.990 INFO:tasks.rados.rados.0.smithi085.stdout:3985: finishing write tid 2 to smithi08575397-21 2021-02-10T04:43:32.990 INFO:tasks.rados.rados.0.smithi085.stdout:3985: finishing write tid 3 to smithi08575397-21 2021-02-10T04:43:32.990 INFO:tasks.rados.rados.0.smithi085.stdout:3985: finishing write tid 4 to smithi08575397-21 2021-02-10T04:43:32.990 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 21 v 2012 (ObjNum 1354 snap 463 seq_num 1354) dirty exists 2021-02-10T04:43:32.990 INFO:tasks.rados.rados.0.smithi085.stdout:3985: left oid 21 (ObjNum 1354 snap 463 seq_num 1354) 2021-02-10T04:43:32.990 INFO:tasks.rados.rados.0.smithi085.stdout:3986: expect (ObjNum 1297 snap 436 seq_num 1297) 2021-02-10T04:43:32.990 INFO:tasks.rados.rados.0.smithi085.stdout:3987: finishing write tid 1 to smithi08575397-12 2021-02-10T04:43:32.990 INFO:tasks.rados.rados.0.smithi085.stdout:3983: done (8 left) 2021-02-10T04:43:32.990 INFO:tasks.rados.rados.0.smithi085.stdout:3985: done (7 left) 2021-02-10T04:43:32.990 INFO:tasks.rados.rados.0.smithi085.stdout:3986: done (6 left) 2021-02-10T04:43:32.990 INFO:tasks.rados.rados.0.smithi085.stdout:3988: snap_remove snap 447 2021-02-10T04:43:32.990 INFO:tasks.rados.rados.0.smithi085.stdout:3988: done (6 left) 2021-02-10T04:43:32.990 INFO:tasks.rados.rados.0.smithi085.stdout:3989: read oid 16 snap -1 2021-02-10T04:43:32.991 INFO:tasks.rados.rados.0.smithi085.stdout:3989: expect deleted 2021-02-10T04:43:32.991 INFO:tasks.rados.rados.0.smithi085.stdout:3990: write (excl) oid 25 current snap is 463 2021-02-10T04:43:32.991 INFO:tasks.rados.rados.0.smithi085.stdout:3990: seq_num 1356 ranges {790292=748478,2183230=607062} 2021-02-10T04:43:32.991 INFO:tasks.rados.rados.0.smithi085.stdout:3990: writing smithi08575397-25 from 790292 to 1538770 tid 1 2021-02-10T04:43:32.991 INFO:tasks.rados.rados.0.smithi085.stdout:3990: writing smithi08575397-25 from 2183230 to 2790292 tid 2 2021-02-10T04:43:32.991 INFO:tasks.rados.rados.0.smithi085.stdout:3991: snap_remove snap 401 2021-02-10T04:43:32.991 INFO:tasks.rados.rados.0.smithi085.stdout:3991: done (8 left) 2021-02-10T04:43:32.991 INFO:tasks.rados.rados.0.smithi085.stdout:3992: read oid 4 snap 396 2021-02-10T04:43:32.991 INFO:tasks.rados.rados.0.smithi085.stdout:3992: expect deleted 2021-02-10T04:43:32.991 INFO:tasks.rados.rados.0.smithi085.stdout:3993: snap_create 2021-02-10T04:43:32.991 INFO:tasks.rados.rados.0.smithi085.stdout:3987: finishing write tid 2 to smithi08575397-12 2021-02-10T04:43:32.991 INFO:tasks.rados.rados.0.smithi085.stdout:3987: finishing write tid 3 to smithi08575397-12 2021-02-10T04:43:32.991 INFO:tasks.rados.rados.0.smithi085.stdout:3987: finishing write tid 4 to smithi08575397-12 2021-02-10T04:43:32.992 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 12 v 1908 (ObjNum 1355 snap 463 seq_num 1355) dirty exists 2021-02-10T04:43:32.992 INFO:tasks.rados.rados.0.smithi085.stdout:3987: left oid 12 (ObjNum 1355 snap 463 seq_num 1355) 2021-02-10T04:43:32.992 INFO:tasks.rados.rados.0.smithi085.stdout:3984: finishing copy_from to smithi08575397-26 2021-02-10T04:43:32.992 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 26 v 1923 (ObjNum 1291 snap 435 seq_num 1291) dirty exists 2021-02-10T04:43:32.992 INFO:tasks.rados.rados.0.smithi085.stdout:3984: finishing copy_from racing read to smithi08575397-26 2021-02-10T04:43:32.992 INFO:tasks.rados.rados.0.smithi085.stdout:3990: finishing write tid 1 to smithi08575397-25 2021-02-10T04:43:32.992 INFO:tasks.rados.rados.0.smithi085.stdout:3990: finishing write tid 2 to smithi08575397-25 2021-02-10T04:43:32.992 INFO:tasks.rados.rados.0.smithi085.stdout:3990: finishing write tid 3 to smithi08575397-25 2021-02-10T04:43:32.992 INFO:tasks.rados.rados.0.smithi085.stdout:3990: finishing write tid 4 to smithi08575397-25 2021-02-10T04:43:32.992 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 25 v 1631 (ObjNum 1356 snap 463 seq_num 1356) dirty exists 2021-02-10T04:43:32.992 INFO:tasks.rados.rados.0.smithi085.stdout:3990: left oid 25 (ObjNum 1356 snap 463 seq_num 1356) 2021-02-10T04:43:32.992 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:43:28 smithi085 conmon[41540]: debug 2021-02-10T04:43:28.906+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:43:32.993 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:43:32.993 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:43:32.996 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:43:32.996 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:27 smithi164 conmon[29288]: cluster 2021-02-10T04:43:26.254671+0000 mgr.y ( 2021-02-10T04:43:32.996 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:27 smithi164 conmon[29288]: mgr.14140) 900 : cluster [DBG] pgmap v1629: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 874 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 1.7 MiB/s wr, 14 op/s 2021-02-10T04:43:32.996 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:27 smithi164 conmon[29288]: cluster 2021-02-10T04:43:26.845952+0000 osd.4 (osd.4) 1697 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:32.996 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:27 smithi164 conmon[29288]: cluster 2021-02-10T04:43:26.845961+0000 osd.4 (osd.4) 1698 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:32.997 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:27 smithi164 conmon[29288]: cluster 2021-02-10T04:43:26.845966+0000 osd.4 (osd.4) 1699 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:32.997 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:27 smithi164 conmon[29288]: cluster 2021-02-10T04:43:26.845985+0000 osd.4 (osd.4 2021-02-10T04:43:32.997 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:27 smithi164 conmon[29288]: ) 1700 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:32.997 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:27 smithi164 conmon[29288]: cluster 2021-02-10T04:43:27.336564+0000 mon.a (mon.0) 1977 : 2021-02-10T04:43:32.997 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:27 smithi164 conmon[29288]: cluster [DBG] osdmap e987: 8 total, 8 up, 7 in 2021-02-10T04:43:32.997 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:28 smithi164 conmon[29288]: cluster 2021-02-10T04:43:27.830679+0000 osd.4 (osd.4) 1701 : cluster 2021-02-10T04:43:32.997 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:28 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:32.997 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:28 smithi164 conmon[29288]: cluster 2021-02-10T04:43:27.830688+0000 osd.4 (osd.4) 1702 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:32.997 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:28 smithi164 conmon[29288]: cluster 2021-02-10T04:43:27.830695+0000 2021-02-10T04:43:32.997 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:28 smithi164 conmon[29288]: osd.4 (osd.4) 1703 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:32.997 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:28 smithi164 conmon[29288]: cluster 2021-02-10T04:43:27.830701+0000 2021-02-10T04:43:32.997 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:28 smithi164 conmon[29288]: osd.4 (osd.4) 1704 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:32.997 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:28 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:43:32.998 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:28 smithi164 conmon[29288]: 04:43:28.327082+0000 mon.a (mon.0) 1978 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 832 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:43:32.998 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:28 smithi164 conmon[29288]: cluster 2021-02-10T04:43:32.998 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:28 smithi164 conmon[29288]: 2021-02-10T04:43:28.342811+0000 mon.a (mon.0) 1979 : cluster [DBG] 2021-02-10T04:43:32.998 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:28 smithi164 conmon[29288]: osdmap e988: 8 total, 8 up, 7 in 2021-02-10T04:43:32.998 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:29 smithi164 conmon[29288]: cluster 2021-02-10T04:43:28.255409+0000 mgr.y (mgr. 2021-02-10T04:43:32.998 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:29 smithi164 conmon[29288]: 14140) 901 : cluster [DBG] pgmap v1632: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 873 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 9.1 MiB/s rd, 1.5 MiB/s wr, 9 op/s 2021-02-10T04:43:32.998 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:29 smithi164 conmon[29288]: audit 2021-02-10T04:43:28.581060+0000 mgr.y (mgr.14140) 902 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:43:32.998 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:29 smithi164 conmon[29288]: cluster 2021-02-10T04:43:28.791036+0000 osd.4 (osd.4) 1705 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:32.998 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:29 smithi164 conmon[29288]: cluster 2021-02-10T04:43:28 2021-02-10T04:43:32.998 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:29 smithi164 conmon[29288]: .791046+0000 osd.4 (osd.4) 1706 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:32.998 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:29 smithi164 conmon[29288]: cluster 2021-02-10T04:43:28.791052+0000 osd.4 (osd.4) 1707 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:32.998 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:29 smithi164 conmon[29288]: cluster 2021-02-10T04:43:28.791059+0000 osd.4 (osd.4) 1708 : cluster 2021-02-10T04:43:32.999 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:29 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:32.999 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:29 smithi164 conmon[29288]: cluster 2021-02-10T04:43:32.999 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:29 smithi164 conmon[29288]: 2021-02-10T04:43:29.344920+0000 mon.a 2021-02-10T04:43:32.999 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:29 smithi164 conmon[29288]: (mon.0) 1980 : cluster [DBG] 2021-02-10T04:43:32.999 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:29 smithi164 conmon[29288]: osdmap e989: 8 total, 8 up, 7 in 2021-02-10T04:43:32.999 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:30 smithi164 conmon[29288]: cluster 2021-02-10T04:43:32.999 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:30 smithi164 conmon[29288]: 2021-02-10T04:43:32.999 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:30 smithi164 conmon[29288]: 2021 2021-02-10T04:43:32.999 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:30 smithi164 conmon[29288]: - 2021-02-10T04:43:32.999 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:30 smithi164 conmon[29288]: 02 2021-02-10T04:43:32.999 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:30 smithi164 conmon[29288]: - 2021-02-10T04:43:32.999 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:30 smithi164 conmon[29288]: 10 2021-02-10T04:43:32.999 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:30 smithi164 conmon[29288]: T 2021-02-10T04:43:32.999 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:30 smithi164 conmon[29288]: 04:43:29.807095 2021-02-10T04:43:33.000 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:30 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1709 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:33.000 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:30 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:43:33.000 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:30 smithi164 conmon[29288]: 02-10T04:43:29.807100+0000 osd.4 (osd. 2021-02-10T04:43:33.000 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:30 smithi164 conmon[29288]: 4) 1710 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:33.000 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:30 smithi164 conmon[29288]: cluster 2021-02-10T04:43:33.000 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:30 smithi164 conmon[29288]: 2021-02-10T04:43: 2021-02-10T04:43:33.000 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:30 smithi164 conmon[29288]: 29.807104+0000 osd.4 (osd.4) 1711 : 2021-02-10T04:43:33.000 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:30 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:33.000 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:30 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:43:33.000 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:30 smithi164 conmon[29288]: -02-10T04:43:29.807107+0000 2021-02-10T04:43:33.000 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:30 smithi164 conmon[29288]: osd.4 (osd.4) 1712 : cluster [WRN] 2021-02-10T04:43:33.000 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:30 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:33.000 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:30 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:43:33.000 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:30 smithi164 conmon[29288]: -10T04:43:30.346492+0000 mon.a (mon.0 2021-02-10T04:43:33.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:30 smithi164 conmon[29288]: ) 1981 : cluster [DBG] osdmap e990: 8 total, 8 up, 7 in 2021-02-10T04:43:33.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:31 smithi164 conmon[29288]: cluster 2021-02-10T04:43:30.256347 2021-02-10T04:43:33.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:31 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 903 : cluster [DBG] pgmap v1635: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 872 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 8.2 MiB/s rd, 2.5 MiB/s wr, 17 op/s 2021-02-10T04:43:33.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:31 smithi164 conmon[29288]: cluster 2021-02-10T04:43 2021-02-10T04:43:33.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:31 smithi164 conmon[29288]: :30.837489+0000 osd.4 (osd.4) 1713 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:33.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:31 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:43:33.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:31 smithi164 conmon[29288]: -02-10T04:43:30.837498+0000 osd.4 (osd.4) 2021-02-10T04:43:33.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:31 smithi164 conmon[29288]: 1714 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:33.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:31 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:43:33.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:31 smithi164 conmon[29288]: 04:43:30.837503+0000 osd.4 (osd 2021-02-10T04:43:33.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:31 smithi164 conmon[29288]: .4) 1715 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:33.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:31 smithi164 conmon[29288]: cluster 2021-02-10T04:43:33.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:31 smithi164 conmon[29288]: 2021-02-10T04:43:30.837508 2021-02-10T04:43:33.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:31 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1716 : cluster [WRN] 2021-02-10T04:43:33.002 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:31 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:33.002 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:31 smithi164 conmon[29288]: cluster 2021-02-10T04:43 2021-02-10T04:43:33.002 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:31 smithi164 conmon[29288]: :31.347777+0000 mon.a (mon.0) 1982 : 2021-02-10T04:43:33.002 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:31 smithi164 conmon[29288]: cluster [DBG] osdmap e991: 8 total, 8 up, 7 in 2021-02-10T04:43:33.002 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:43:28 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:43:33.002 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:28 smithi164 conmon[33677]: debug 2021-02-10T04:43:33.002 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:28 smithi164 conmon[33677]: 2021-02-10T04:43:28.789+0000 7fbec19a1700 -1 osd.4 988 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:33.002 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:28 smithi164 conmon[33677]: 2021-02-10T04:43:33.002 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:29 smithi164 conmon[33677]: debug 2021-02-10T04:43:33.002 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:29 smithi164 conmon[33677]: 2021-02-10T04:43:29.805+0000 7fbec19a1700 -1 osd.4 989 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:33.002 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:29 smithi164 conmon[33677]: 2021-02-10T04:43:33.003 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:30 smithi164 conmon[33677]: debug 2021-02-10T04:43:30.835+0000 7fbec19a1700 -1 osd.4 990 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:33.003 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:31 smithi164 conmon[33677]: debug 2021-02-10T04:43:31.818+0000 7fbec19a1700 -1 osd.4 991 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:33.094 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:43:33.135 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:43:33.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:32 smithi164 conmon[33677]: debug 2021-02-10T04:43:32.822+0000 7fbec19a1700 -1 osd.4 992 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:33.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:32 smithi164 conmon[29288]: cluster 2021-02-10T04:43:31.720724+0000 osd.3 2021-02-10T04:43:33.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:32 smithi164 conmon[29288]: (osd.3) 90 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:43:33.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:32 smithi164 conmon[29288]: cluster 2021-02-10T04:43:31.722589+0000 osd.3 ( 2021-02-10T04:43:33.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:32 smithi164 conmon[29288]: osd.3) 91 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:43:33.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:32 smithi164 conmon[29288]: cluster 2021-02-10T04:43:33.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:32 smithi164 conmon[29288]: 2021-02-10T04:43:31.819784+0000 osd.4 ( 2021-02-10T04:43:33.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:32 smithi164 conmon[29288]: osd.4) 1717 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:33.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:32 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:43:33.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:32 smithi164 conmon[29288]: T04:43:31.819792+0000 osd.4 (osd.4 2021-02-10T04:43:33.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:32 smithi164 conmon[29288]: ) 1718 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:33.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:32 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:43:33.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:32 smithi164 conmon[29288]: T04:43:31.819799+0000 osd.4 (osd.4) 2021-02-10T04:43:33.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:32 smithi164 conmon[29288]: 1719 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:33.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:32 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:43:33.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:32 smithi164 conmon[29288]: :43:31.819806+0000 osd.4 (osd.4) 1720 : 2021-02-10T04:43:33.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:32 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:33.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:32 smithi164 conmon[29288]: cluster 2021-02-10T04:43 2021-02-10T04:43:33.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:32 smithi164 conmon[29288]: :32.349140+0000 mon.a (mon.0) 1983 : cluster 2021-02-10T04:43:33.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:32 smithi164 conmon[29288]: [DBG] osdmap e992: 8 total, 8 up, 7 in 2021-02-10T04:43:33.357 INFO:tasks.rados.rados.0.smithi085.stdout:3984: done (9 left) 2021-02-10T04:43:33.357 INFO:tasks.rados.rados.0.smithi085.stdout:3987: done (8 left) 2021-02-10T04:43:33.357 INFO:tasks.rados.rados.0.smithi085.stdout:3989: done (7 left) 2021-02-10T04:43:33.358 INFO:tasks.rados.rados.0.smithi085.stdout:3990: done (6 left) 2021-02-10T04:43:33.358 INFO:tasks.rados.rados.0.smithi085.stdout:3992: done (5 left) 2021-02-10T04:43:33.358 INFO:tasks.rados.rados.0.smithi085.stdout:3993: done (4 left) 2021-02-10T04:43:33.358 INFO:tasks.rados.rados.0.smithi085.stdout:3994: read oid 1 snap -1 2021-02-10T04:43:33.358 INFO:tasks.rados.rados.0.smithi085.stdout:3994: expect deleted 2021-02-10T04:43:33.358 INFO:tasks.rados.rados.0.smithi085.stdout:3995: copy_from oid 4 from oid 11 current snap is 464 2021-02-10T04:43:33.358 INFO:tasks.rados.rados.0.smithi085.stdout:3996: read oid 7 snap 445 2021-02-10T04:43:33.358 INFO:tasks.rados.rados.0.smithi085.stdout:3996: expect deleted 2021-02-10T04:43:33.358 INFO:tasks.rados.rados.0.smithi085.stdout:3997: delete oid 29 current snap is 464 2021-02-10T04:43:33.404 INFO:tasks.rados.rados.0.smithi085.stdout:3994: done (7 left) 2021-02-10T04:43:33.404 INFO:tasks.rados.rados.0.smithi085.stdout:3997: done (6 left) 2021-02-10T04:43:33.404 INFO:tasks.rados.rados.0.smithi085.stdout:3998: copy_from oid 32 from oid 21 current snap is 464 2021-02-10T04:43:33.405 INFO:tasks.rados.rados.0.smithi085.stdout:3999: snap_remove snap 439 2021-02-10T04:43:33.688 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[52836]: debug 2021-02-10T04:43:33.689 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[52836]: 2021-02-10T04:43:33.353+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:43:33.749 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:43:33.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: cluster 2021-02-10T04:43:33.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: 2021-02-10T04:43:33.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: 2021 2021-02-10T04:43:33.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: - 2021-02-10T04:43:33.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: 02 2021-02-10T04:43:33.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: -10T04:43:32.257117+0000 2021-02-10T04:43:33.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: mgr.y (mgr.14140) 904 : cluster [DBG] pgmap v1638: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 872 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 12 op/s 2021-02-10T04:43:33.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: 2021-02-10T04:43:33.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: cluster 2021-02-10T04:43:32.824662+0000 osd.4 (osd. 2021-02-10T04:43:33.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: 4) 1721 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:33.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: cluster 2021-02-10T04:43:33.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: 2021- 2021-02-10T04:43:33.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: 02-10T04:43:32.824676+0000 2021-02-10T04:43:33.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: osd.4 2021-02-10T04:43:33.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: (osd.4) 1722 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:33.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:43:33.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: -10T04:43:32.824679 2021-02-10T04:43:33.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2021-02-10T04:43:33.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: 1723 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:33.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:43:33.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: 02-10T04:43:32.824683+0000 osd.4 (osd. 2021-02-10T04:43:33.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: 4) 1724 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:33.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: audit 2021-02-10T04:43 2021-02-10T04:43:33.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: :32.902363+0000 mgr.y (mgr.14140) 905 : 2021-02-10T04:43:33.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: audit [DBG] from='client.25267 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:43:33.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:43:33.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: 43:33.329173+0000 mon.a 2021-02-10T04:43:33.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: (mon.0) 1984 : cluster 2021-02-10T04:43:33.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: [WRN] Health check update: 4 slow ops, oldest one blocked for 842 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:43:33.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: cluster 2021-02-10T04:43:33.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:43:33.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: 43:33.356964+0000 mon.a 2021-02-10T04:43:33.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: (mon.0) 1985 : cluster 2021-02-10T04:43:33.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[29319]: [DBG] osdmap e993: 8 total, 8 up, 7 in 2021-02-10T04:43:33.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[35961]: cluster 2021-02-10T04:43:32.257117+0000 mgr.y (mgr.14140) 904 : cluster [DBG] pgmap v1638: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 872 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 12 op/s 2021-02-10T04:43:33.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[35961]: 2021-02-10T04:43:33.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[35961]: cluster 2021-02-10T04:43:32.824662+0000 osd.4 (osd.4) 1721 : cluster [WRN] 2021-02-10T04:43:33.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:33.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:43:33.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[35961]: :43:32.824676+0000 osd.4 (osd.4) 1722 : cluster 2021-02-10T04:43:33.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:33.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:43:33.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[35961]: T04:43:32.824679 2021-02-10T04:43:33.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1723 : cluster 2021-02-10T04:43:33.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:33.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[35961]: cluster 2021-02-10T04:43:33.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[35961]: 2021-02-10T04:43:32.824683+0000 osd.4 (osd.4) 2021-02-10T04:43:33.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[35961]: 1724 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:33.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[35961]: audit 2021-02-10T04:43:32.902363 2021-02-10T04:43:33.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 905 : 2021-02-10T04:43:33.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[35961]: audit [DBG] 2021-02-10T04:43:33.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[35961]: from='client.25267 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:43:33.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[35961]: cluster 2021-02-10T04:43: 2021-02-10T04:43:33.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[35961]: 33.329173+0000 mon.a (mon.0) 1984 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 842 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:43:33.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[35961]: 2021-02-10T04:43:33.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:43:33.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[35961]: 02-10T04:43:33.356964+0000 mon.a (mon.0 2021-02-10T04:43:33.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:33 smithi085 conmon[35961]: ) 1985 : cluster [DBG] osdmap e993: 8 total, 8 up, 7 in 2021-02-10T04:43:34.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:33 smithi164 conmon[33677]: debug 2021-02-10T04:43:34.145 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:33 smithi164 conmon[33677]: 2021-02-10T04:43:33.798+0000 7fbec19a1700 -1 osd.4 993 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:34.145 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:33 smithi164 conmon[33677]: 2021-02-10T04:43:34.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:33 smithi164 conmon[29288]: cluster 2021-02-10T04:43:32.257117 2021-02-10T04:43:34.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:33 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 904 : cluster [DBG] pgmap v1638: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 872 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 12 op/s 2021-02-10T04:43:34.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:33 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:43:34.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:33 smithi164 conmon[29288]: 04:43:32.824662+0000 osd.4 (osd.4) 1721 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:34.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:33 smithi164 conmon[29288]: cluster 2021-02-10T04:43:34.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:33 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:43:34.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:33 smithi164 conmon[29288]: :43:32.824676+0000 osd.4 (osd.4) 1722 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:34.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:33 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:43:34.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:33 smithi164 conmon[29288]: -02-10T04:43: 2021-02-10T04:43:34.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:33 smithi164 conmon[29288]: 32.824679+0000 osd.4 (osd.4) 1723 : 2021-02-10T04:43:34.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:33 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:34.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:33 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:43:34.148 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:33 smithi164 conmon[29288]: -02-10T04:43:32.824683+0000 2021-02-10T04:43:34.148 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:33 smithi164 conmon[29288]: osd.4 (osd.4) 1724 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:34.148 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:33 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:43:34.148 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:33 smithi164 conmon[29288]: 10T04:43:32.902363+0000 mgr.y (mgr.14140 2021-02-10T04:43:34.148 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:33 smithi164 conmon[29288]: ) 905 : audit [DBG] from='client.25267 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:43:34.149 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:33 smithi164 conmon[29288]: cluster 2021-02-10T04:43:34.149 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:33 smithi164 conmon[29288]: 2021-02-10T04:43:33.329173+0000 2021-02-10T04:43:34.149 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:33 smithi164 conmon[29288]: mon.a (mon.0) 1984 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 842 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:43:34.149 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:33 smithi164 conmon[29288]: cluster 2021-02-10T04:43:34.149 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:33 smithi164 conmon[29288]: 2021-02-10T04:43:33.356964+0000 2021-02-10T04:43:34.149 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:33 smithi164 conmon[29288]: mon.a (mon.0) 1985 : cluster [DBG] osdmap e993: 8 total, 8 up, 7 in 2021-02-10T04:43:34.150 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:33 smithi164 conmon[29288]: 2021-02-10T04:43:34.355 INFO:tasks.rados.rados.0.smithi085.stdout:3999: done (7 left) 2021-02-10T04:43:34.355 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 7 2021-02-10T04:43:34.356 INFO:tasks.rados.rados.0.smithi085.stdout:3998: finishing copy_from to smithi08575397-32 2021-02-10T04:43:34.356 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 32 v 2012 (ObjNum 1354 snap 463 seq_num 1354) dirty exists 2021-02-10T04:43:34.356 INFO:tasks.rados.rados.0.smithi085.stdout:3998: finishing copy_from racing read to smithi08575397-32 2021-02-10T04:43:34.356 INFO:tasks.rados.rados.0.smithi085.stdout:3995: finishing copy_from to smithi08575397-4 2021-02-10T04:43:34.356 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 4 v 1923 (ObjNum 1291 snap 435 seq_num 1291) dirty exists 2021-02-10T04:43:34.356 INFO:tasks.rados.rados.0.smithi085.stdout:3995: finishing copy_from racing read to smithi08575397-4 2021-02-10T04:43:34.356 INFO:tasks.rados.rados.0.smithi085.stdout:3995: done (6 left) 2021-02-10T04:43:34.356 INFO:tasks.rados.rados.0.smithi085.stdout:3996: done (5 left) 2021-02-10T04:43:34.356 INFO:tasks.rados.rados.0.smithi085.stdout:3998: done (4 left) 2021-02-10T04:43:34.356 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 4 2021-02-10T04:43:35.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:34 smithi164 conmon[33677]: debug 2021-02-10T04:43:34.768+0000 7fbec19a1700 -1 osd.4 994 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:35.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:34 smithi164 conmon[29288]: cluster 2021-02-10T04:43:33.800650+0000 osd.4 (osd 2021-02-10T04:43:35.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:34 smithi164 conmon[29288]: .4) 1725 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:35.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:34 smithi164 conmon[29288]: cluster 2021-02-10T04:43:33.800658+0000 osd.4 (osd.4) 1726 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:35.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:34 smithi164 conmon[29288]: cluster 2021-02-10T04:43:35.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:34 smithi164 conmon[29288]: 2021-02-10T04:43:33.800664+0000 osd.4 (osd.4) 1727 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:35.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:34 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:43:35.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:34 smithi164 conmon[29288]: T04:43:33.800670+0000 osd.4 (osd.4 2021-02-10T04:43:35.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:34 smithi164 conmon[29288]: ) 1728 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:35.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:34 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:43:35.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:34 smithi164 conmon[29288]: :43:34.353971+0000 mon.a (mon. 2021-02-10T04:43:35.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:34 smithi164 conmon[29288]: 0) 1986 : cluster [DBG] osdmap e994: 8 total, 8 up, 7 in 2021-02-10T04:43:35.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:34 smithi085 conmon[29319]: cluster 2021-02-10T04:43:33.800650+0000 osd.4 (osd.4) 1725 : 2021-02-10T04:43:35.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:34 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:35.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:34 smithi085 conmon[29319]: cluster 2021-02-10T04:43:33.800658+0000 osd.4 (osd.4) 1726 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:35.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:34 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:43:35.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:34 smithi085 conmon[29319]: -02-10T04:43:33.800664+0000 osd.4 (osd.4) 1727 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:35.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:34 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:43:35.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:34 smithi085 conmon[29319]: 02-10T04:43:33.800670+0000 osd.4 (osd.4) 1728 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:35.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:34 smithi085 conmon[29319]: 2021-02-10T04:43:35.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:34 smithi085 conmon[29319]: cluster 2021-02-10T04:43:35.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:34 smithi085 conmon[29319]: 2021-02-10T04:43:34.353971+0000 mon.a ( 2021-02-10T04:43:35.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:34 smithi085 conmon[29319]: mon.0) 1986 : cluster [DBG] osdmap e994: 8 total, 8 up, 7 in 2021-02-10T04:43:35.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:34 smithi085 conmon[35961]: cluster 2021-02-10T04:43:33.800650+0000 osd.4 (osd.4 2021-02-10T04:43:35.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:34 smithi085 conmon[35961]: ) 1725 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:35.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:34 smithi085 conmon[35961]: cluster 2021-02-10T04:43:33.800658+0000 osd.4 (osd.4) 1726 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:35.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:34 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:43:35.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:34 smithi085 conmon[35961]: -02-10T04:43:33.800664+0000 osd.4 (osd.4) 1727 : cluster [WRN] 2021-02-10T04:43:35.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:34 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:35.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:34 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:43:35.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:34 smithi085 conmon[35961]: -02-10T04:43:33.800670 2021-02-10T04:43:35.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:34 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1728 : cluster 2021-02-10T04:43:35.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:34 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:35.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:34 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:43:35.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:34 smithi085 conmon[35961]: 02-10T04:43:34.353971+0000 2021-02-10T04:43:35.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:34 smithi085 conmon[35961]: mon.a (mon.0) 1986 : cluster [DBG] osdmap e994: 8 total, 8 up, 7 in 2021-02-10T04:43:35.461 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:43:35.567 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:43:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: cluster 2021-02-10T04:43:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: 2021-02-10T04:43:35.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: 2021 2021-02-10T04:43:35.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: - 2021-02-10T04:43:35.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: 02 2021-02-10T04:43:35.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: - 2021-02-10T04:43:35.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: 10 2021-02-10T04:43:35.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: T 2021-02-10T04:43:35.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: 04 2021-02-10T04:43:35.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: : 2021-02-10T04:43:35.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: 43 2021-02-10T04:43:35.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: : 2021-02-10T04:43:35.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: 34 2021-02-10T04:43:35.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: . 2021-02-10T04:43:35.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: 257901 2021-02-10T04:43:35.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: +0000 mgr.y (mgr. 2021-02-10T04:43:35.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: 14140) 906 : cluster [DBG] pgmap v1641: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 875 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 8 op/s 2021-02-10T04:43:35.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:43:35.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: -02-10T04:43:34.770334+0000 2021-02-10T04:43:35.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: osd.4 (osd 2021-02-10T04:43:35.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: .4) 1729 : cluster 2021-02-10T04:43:35.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:35.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:43:35.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: 02-10T04:43:34.770339+0000 2021-02-10T04:43:35.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: osd.4 (osd.4) 1730 : cluster [WRN] 2021-02-10T04:43:35.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:35.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:43:35.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: T04:43:34.770343+0000 osd.4 (osd 2021-02-10T04:43:35.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: .4) 1731 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:35.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: cluster 2021-02-10T04:43:35.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: 2021-02-10T04:43:34.770346+0000 2021-02-10T04:43:35.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: osd.4 (osd.4) 1732 : cluster [WRN] 2021-02-10T04:43:35.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:35.897 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[33677]: debug 2021-02-10T04:43:35.897 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[33677]: 2021-02-10T04:43:35.676+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:43:35.897 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:35 smithi164 conmon[33677]: debug 2021-02-10T04:43:35.752+0000 7fbec19a1700 -1 osd.4 995 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:36.057 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:43:36.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:35 smithi085 conmon[29319]: cluster 2021-02-10T04:43:34. 2021-02-10T04:43:36.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:35 smithi085 conmon[29319]: 257901+0000 mgr.y (mgr.14140) 906 : cluster [DBG] pgmap v1641: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 875 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 8 op/s 2021-02-10T04:43:36.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:35 smithi085 conmon[29319]: cluster 2021-02-10T04:43:34.770334+0000 osd.4 (osd.4) 1729 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:36.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:35 smithi085 conmon[29319]: 2021-02-10T04:43:36.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:35 smithi085 conmon[29319]: cluster 2021-02-10T04:43:34.770339+0000 osd.4 (osd.4) 1730 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:36.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:35 smithi085 conmon[29319]: cluster 2021-02-10T04:43:36.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:35 smithi085 conmon[29319]: 2021-02-10T04:43:34.770343+0000 osd.4 (osd.4 2021-02-10T04:43:36.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:35 smithi085 conmon[29319]: ) 1731 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:36.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:35 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:43:36.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:35 smithi085 conmon[29319]: -02-10T04:43:34.770346 2021-02-10T04:43:36.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:35 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1732 : cluster 2021-02-10T04:43:36.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:35 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:36.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:35 smithi085 conmon[35961]: cluster 2021-02-10T04:43: 2021-02-10T04:43:36.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:35 smithi085 conmon[35961]: 34.257901+0000 mgr.y (mgr.14140) 906 : cluster [DBG] pgmap v1641: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 875 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 8 op/s 2021-02-10T04:43:36.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:35 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:43:36.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:35 smithi085 conmon[35961]: 43:34.770334+0000 osd.4 (osd.4) 1729 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:36.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:35 smithi085 conmon[35961]: cluster 2021-02-10T04:43:34.770339+0000 2021-02-10T04:43:36.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:35 smithi085 conmon[35961]: osd.4 (osd.4) 1730 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:36.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:35 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:43:36.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:35 smithi085 conmon[35961]: 10T04:43:34.770343+0000 osd.4 (osd. 2021-02-10T04:43:36.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:35 smithi085 conmon[35961]: 4) 1731 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:36.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:35 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:43:36.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:35 smithi085 conmon[35961]: 04:43:34.770346+0000 osd.4 (osd.4) 2021-02-10T04:43:36.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:35 smithi085 conmon[35961]: 1732 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:37.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:36 smithi164 conmon[33677]: debug 2021-02-10T04:43:36.745+0000 7fbec19a1700 -1 osd.4 995 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:37.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:36 smithi164 conmon[29288]: cluster 2021-02-10T04:43:35.410845+0000 osd.0 (osd.0) 2021-02-10T04:43:37.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:36 smithi164 conmon[29288]: 95 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:43:37.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:36 smithi164 conmon[29288]: cluster 2021-02-10T04:43:35.411893+0000 osd.0 (osd.0) 96 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:43:37.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:36 smithi164 conmon[29288]: cluster 2021-02-10T04:43:35.699693+0000 mon.a (mon 2021-02-10T04:43:37.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:36 smithi164 conmon[29288]: .0) 1987 : cluster [DBG] osdmap e995: 8 total, 8 up, 7 in 2021-02-10T04:43:37.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:36 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:43:37.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:36 smithi164 conmon[29288]: 10T04:43:35.754211+0000 osd.4 (osd.4) 1733 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:37.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:36 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:43:37.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:36 smithi164 conmon[29288]: -02-10T04:43:35.754220+0000 osd.4 (osd.4) 1734 : cluster 2021-02-10T04:43:37.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:36 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:37.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:36 smithi164 conmon[29288]: cluster 2021-02-10T04:43: 2021-02-10T04:43:37.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:36 smithi164 conmon[29288]: 35.754226+0000 osd.4 (osd.4) 1735 : cluster 2021-02-10T04:43:37.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:36 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:37.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:36 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:43:37.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:36 smithi164 conmon[29288]: 04:43:35.754232+0000 osd.4 (osd.4) 1736 : cluster [WRN] 2021-02-10T04:43:37.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:36 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:37.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[35961]: cluster 2021-02-10T04:43:35.410845 2021-02-10T04:43:37.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[35961]: +0000 osd.0 (osd.0) 95 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:43:37.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[35961]: cluster 2021-02-10T04:43:37.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:43:37.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[35961]: 43:35.411893+0000 osd.0 (osd.0) 2021-02-10T04:43:37.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[35961]: 96 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:43:37.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[35961]: cluster 2021-02-10T04:43:37.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[35961]: 2021-02-10T04:43: 2021-02-10T04:43:37.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[35961]: 35.699693+0000 mon.a (mon.0) 1987 : cluster 2021-02-10T04:43:37.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[35961]: [DBG] osdmap e995: 8 total, 8 up, 7 in 2021-02-10T04:43:37.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[35961]: cluster 2021-02-10T04:43: 2021-02-10T04:43:37.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[35961]: 35.754211+0000 osd.4 (osd.4) 1733 : cluster 2021-02-10T04:43:37.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:37.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:43:37.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[35961]: 10T04:43:35.754220+0000 osd.4 (osd. 2021-02-10T04:43:37.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[35961]: 4) 1734 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:37.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:43:37.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[35961]: -02-10T04:43:35.754226+0000 2021-02-10T04:43:37.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[35961]: osd.4 (osd.4) 1735 : cluster [WRN] 2021-02-10T04:43:37.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[35961]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:37.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[35961]: cluster 2021-02-10T04:43:35. 2021-02-10T04:43:37.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[35961]: 754232+0000 osd.4 (osd.4) 1736 : cluster [WRN] 2021-02-10T04:43:37.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:37.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[29319]: cluster 2021-02-10T04:43:37.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[29319]: 2021-02-10T04:43:37.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[29319]: 2021 2021-02-10T04:43:37.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[29319]: - 2021-02-10T04:43:37.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[29319]: 02 2021-02-10T04:43:37.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[29319]: - 2021-02-10T04:43:37.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[29319]: 10 2021-02-10T04:43:37.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[29319]: T04:43 2021-02-10T04:43:37.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[29319]: :35.410845+0000 osd.0 (osd.0) 95 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:43:37.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[29319]: cluster 2021-02-10T04:43:37.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[29319]: 2021-02-10T04:43:37.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[29319]: 2021-02-10T04:43:35. 2021-02-10T04:43:37.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[29319]: 411893+0000 osd.0 (osd.0) 96 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:43:37.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[29319]: cluster 2021-02-10T04:43:35.699693+0000 2021-02-10T04:43:37.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[29319]: mon.a (mon.0) 1987 : cluster [DBG] osdmap e995: 8 total, 8 up, 7 in 2021-02-10T04:43:37.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:43:37.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[29319]: -02-10T04:43:35.754211+0000 osd.4 (osd.4) 2021-02-10T04:43:37.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[29319]: 1733 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:37.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[29319]: cluster 2021-02-10T04:43: 2021-02-10T04:43:37.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[29319]: 35.754220+0000 osd.4 (osd.4) 1734 : cluster 2021-02-10T04:43:37.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:37.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[29319]: cluster 2021-02-10T04:43 2021-02-10T04:43:37.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[29319]: :35.754226+0000 osd.4 (osd.4) 1735 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:37.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:43:37.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[29319]: 02-10T04:43:35.754232+0000 osd.4 (osd 2021-02-10T04:43:37.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:36 smithi085 conmon[29319]: .4) 1736 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:38.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:37 smithi164 conmon[33677]: debug 2021-02-10T04:43:37.762+0000 7fbec19a1700 -1 osd.4 995 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:38.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:37 smithi164 conmon[29288]: cluster 2021-02-10T04:43:36.258456+0000 mgr.y (mgr.14140 2021-02-10T04:43:38.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:37 smithi164 conmon[29288]: ) 907 : cluster [DBG] pgmap v1644: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 882 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 MiB/s rd, 3.7 MiB/s wr, 10 op/s 2021-02-10T04:43:38.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:37 smithi164 conmon[29288]: cluster 2021-02-10T04:43:36.746823+0000 osd.4 ( 2021-02-10T04:43:38.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:37 smithi164 conmon[29288]: osd.4) 1737 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:38.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:37 smithi164 conmon[29288]: cluster 2021-02-10T04:43:38.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:37 smithi164 conmon[29288]: 2021-02-10T04:43:36.746834+0000 osd.4 2021-02-10T04:43:38.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:37 smithi164 conmon[29288]: (osd.4) 1738 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:38.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:37 smithi164 conmon[29288]: cluster 2021-02-10T04:43:38.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:37 smithi164 conmon[29288]: 2021-02-10T04:43:36.746842+0000 osd.4 (osd.4) 2021-02-10T04:43:38.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:37 smithi164 conmon[29288]: 1739 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:38.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:37 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:43:38.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:37 smithi164 conmon[29288]: 02-10T04:43:36.746849+0000 osd.4 ( 2021-02-10T04:43:38.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:37 smithi164 conmon[29288]: osd.4) 1740 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:38.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:37 smithi085 conmon[35961]: cluster 2021-02-10T04:43:36.258456+0000 mgr.y (mgr. 2021-02-10T04:43:38.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:37 smithi085 conmon[35961]: 14140) 907 : cluster [DBG] pgmap v1644: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 882 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 MiB/s rd, 3.7 MiB/s wr, 10 op/s 2021-02-10T04:43:38.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:37 smithi085 conmon[35961]: cluster 2021-02-10T04:43:38.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:37 smithi085 conmon[35961]: 2021-02-10T04:43:36.746823+0000 osd.4 (osd.4) 1737 : cluster 2021-02-10T04:43:38.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:37 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:38.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:37 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:43:38.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:37 smithi085 conmon[35961]: -10T04:43:36.746834+0000 osd.4 (osd.4) 1738 : cluster [WRN] 2021-02-10T04:43:38.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:37 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:38.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:37 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:43:38.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:37 smithi085 conmon[35961]: 02-10T04:43:36.746842+0000 osd.4 2021-02-10T04:43:38.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:37 smithi085 conmon[35961]: (osd.4) 1739 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:38.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:37 smithi085 conmon[35961]: cluster 2021-02-10T04:43:38.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:37 smithi085 conmon[35961]: 2021-02-10T04:43:38.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:37 smithi085 conmon[35961]: 2021-02-10T04:43: 2021-02-10T04:43:38.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:37 smithi085 conmon[35961]: 36.746849+0000 osd.4 (osd.4) 1740 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:38.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:37 smithi085 conmon[29319]: cluster 2021-02-10T04:43:36.258456+0000 mgr.y (mgr.14140 2021-02-10T04:43:38.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:37 smithi085 conmon[29319]: ) 907 : cluster [DBG] pgmap v1644: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 882 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 MiB/s rd, 3.7 MiB/s wr, 10 op/s 2021-02-10T04:43:38.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:37 smithi085 conmon[29319]: cluster 2021-02-10T04:43:36.746823+0000 osd.4 (osd.4) 1737 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:38.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:37 smithi085 conmon[29319]: cluster 2021-02-10T04:43:36.746834+0000 osd.4 (osd.4) 1738 : cluster 2021-02-10T04:43:38.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:37 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:38.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:37 smithi085 conmon[29319]: cluster 2021-02-10T04:43:36 2021-02-10T04:43:38.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:37 smithi085 conmon[29319]: .746842+0000 osd.4 (osd.4) 1739 : cluster [WRN] 2021-02-10T04:43:38.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:37 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:38.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:37 smithi085 conmon[29319]: cluster 2021-02-10T04:43:36. 2021-02-10T04:43:38.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:37 smithi085 conmon[29319]: 746849+0000 osd.4 (osd.4) 1740 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:38.894 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:43:38 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:43:38.895 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:38 smithi164 conmon[33677]: debug 2021-02-10T04:43:38.767+0000 7fbec19a1700 -1 osd.4 996 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:38 smithi164 conmon[29288]: cluster 2021-02-10T04:43:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:38 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:43:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:38 smithi164 conmon[29288]: 43:37.763813+0000 osd.4 (osd.4) 1741 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:38 smithi164 conmon[29288]: cluster 2021-02-10T04:43:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:38 smithi164 conmon[29288]: 2021-02-10 2021-02-10T04:43:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:38 smithi164 conmon[29288]: T04:43:37.763823+0000 osd.4 ( 2021-02-10T04:43:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:38 smithi164 conmon[29288]: osd.4) 1742 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:38 smithi164 conmon[29288]: 2021-02-10T04:43:38.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:38 smithi164 conmon[29288]: cluster 2021-02-10T04:43:38.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:38 smithi164 conmon[29288]: 2021-02-10T04:43:37 2021-02-10T04:43:38.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:38 smithi164 conmon[29288]: .763830+0000 osd.4 (osd.4) 1743 : cluster 2021-02-10T04:43:38.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:38 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:38.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:38 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:43:38.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:38 smithi164 conmon[29288]: -02-10T04:43:37.763835+0000 osd.4 (osd. 2021-02-10T04:43:38.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:38 smithi164 conmon[29288]: 4) 1744 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:38.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:38 smithi164 conmon[29288]: audit 2021-02-10T04:43:38 2021-02-10T04:43:38.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:38 smithi164 conmon[29288]: .149911+0000 mon.a (mon.0) 1988 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:43:38.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:38 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:43:38.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:38 smithi164 conmon[29288]: -10T04:43:38.153740+0000 mon.a 2021-02-10T04:43:38.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:38 smithi164 conmon[29288]: (mon.0) 1989 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:43:38.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:38 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:43:38.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:38 smithi164 conmon[29288]: 10T04:43:38.335558+0000 mon.a ( 2021-02-10T04:43:38.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:38 smithi164 conmon[29288]: mon.0) 1990 : cluster [DBG] osdmap e996: 8 total, 8 up, 7 in 2021-02-10T04:43:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[29319]: cluster 2021-02-10T04:43:37.763813+0000 osd.4 (osd.4) 1741 : cluster 2021-02-10T04:43:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[29319]: cluster 2021-02-10T04:43:37.763823+0000 osd.4 (osd.4) 1742 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:43:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[29319]: 02-10T04:43:37.763830+0000 osd.4 (osd.4) 1743 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:43:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[29319]: 04:43:37.763835+0000 osd.4 (osd.4 2021-02-10T04:43:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[29319]: ) 1744 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:39.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:43:39.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[29319]: 02-10T04:43:38.149911+0000 mon.a 2021-02-10T04:43:39.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[29319]: (mon.0) 1988 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:43:39.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[29319]: audit 2021 2021-02-10T04:43:39.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[29319]: -02-10T04:43:38.153740+0000 2021-02-10T04:43:39.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[29319]: mon.a (mon.0) 1989 : audit [INF] 2021-02-10T04:43:39.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:43:39.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[29319]: cluster 2021-02-10T04:43 2021-02-10T04:43:39.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[29319]: :38.335558+0000 mon.a (mon.0) 2021-02-10T04:43:39.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[29319]: 1990 : cluster [DBG] osdmap e996: 8 total, 8 up, 7 in 2021-02-10T04:43:39.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: cluster 2021-02-10T04:43:39.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: 2021-02-10T04:43:39.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: 2021 2021-02-10T04:43:39.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: - 2021-02-10T04:43:39.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: 02 2021-02-10T04:43:39.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: - 2021-02-10T04:43:39.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: 10 2021-02-10T04:43:39.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: T 2021-02-10T04:43:39.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: 04 2021-02-10T04:43:39.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: : 2021-02-10T04:43:39.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: 43 2021-02-10T04:43:39.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: : 2021-02-10T04:43:39.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: 37 2021-02-10T04:43:39.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: . 2021-02-10T04:43:39.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: 763813 2021-02-10T04:43:39.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: +0000 2021-02-10T04:43:39.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: 2021-02-10T04:43:39.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: osd.4 2021-02-10T04:43:39.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: ( 2021-02-10T04:43:39.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: osd 2021-02-10T04:43:39.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: . 2021-02-10T04:43:39.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: 4 2021-02-10T04:43:39.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: ) 2021-02-10T04:43:39.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: 1741 2021-02-10T04:43:39.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: : 2021-02-10T04:43:39.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: cluster 2021-02-10T04:43:39.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: 2021-02-10T04:43:39.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: [WRN] 2021-02-10T04:43:39.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: 2021-02-10T04:43:39.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:39.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: 2021-02-10T04:43:39.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: cluster 2021-02-10T04:43:39.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: 2021-02-10T04:43:39.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: 2021 2021-02-10T04:43:39.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: - 2021-02-10T04:43:39.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: 02-10T04:43: 2021-02-10T04:43:39.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: 37.763823+0000 osd.4 (osd.4) 1742 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:39.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: cluster 2021-02-10T04:43 2021-02-10T04:43:39.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: :37.763830+0000 osd.4 (osd.4) 2021-02-10T04:43:39.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: 1743 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:39.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:43:39.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: 10T04:43:37.763835+0000 2021-02-10T04:43:39.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: osd.4 (osd.4) 1744 : cluster [WRN] 2021-02-10T04:43:39.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:39.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: audit 2021-02-10T04:43 2021-02-10T04:43:39.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: :38.149911+0000 mon.a (mon.0) 2021-02-10T04:43:39.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: 1988 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:43:39.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: audit 2021-02-10T04:43:39.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: 2021-02-10T04:43:38.153740+0000 mon.a (mon.0) 1989 : audit 2021-02-10T04:43:39.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:43:39.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: cluster 2021-02-10T04:43 2021-02-10T04:43:39.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:38 smithi085 conmon[35961]: :38.335558+0000 mon.a (mon.0) 1990 : cluster [DBG] osdmap e996: 8 total, 8 up, 7 in 2021-02-10T04:43:40.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:39 smithi164 conmon[33677]: debug 2021-02-10T04:43:39.796+0000 7fbec19a1700 -1 osd.4 996 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:39 smithi164 conmon[29288]: cluster 2021-02-10T04:43:38.259249+0000 mgr.y (mgr. 2021-02-10T04:43:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:39 smithi164 conmon[29288]: 14140) 908 : cluster [DBG] pgmap v1645: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 827 KiB/s rd, 2.9 MiB/s wr, 8 op/s 2021-02-10T04:43:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:39 smithi164 conmon[29288]: audit 2021-02-10T04:43:38.591047+0000 mgr.y (mgr.14140) 909 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:43:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:39 smithi164 conmon[29288]: cluster 2021-02-10T04:43:38.713019+0000 mon.a (mon 2021-02-10T04:43:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:39 smithi164 conmon[29288]: .0) 1991 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 847 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:43:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:39 smithi164 conmon[29288]: cluster 2021-02-10T04:43:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:39 smithi164 conmon[29288]: 2021-02-10T04:43:38.768968+0000 2021-02-10T04:43:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:39 smithi164 conmon[29288]: osd.4 (osd.4) 1745 : cluster [WRN] 2021-02-10T04:43:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:39 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:39 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:43:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:39 smithi164 conmon[29288]: -02-10T04:43:38.768978+0000 2021-02-10T04:43:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:39 smithi164 conmon[29288]: osd.4 (osd.4) 1746 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:39 smithi164 conmon[29288]: 2021-02-10T04:43:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:39 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:43:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:39 smithi164 conmon[29288]: -10T04:43:38.768984+0000 osd.4 2021-02-10T04:43:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:39 smithi164 conmon[29288]: (osd.4) 1747 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:40.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:39 smithi164 conmon[29288]: cluster 2021-02-10T04:43:40.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:39 smithi164 conmon[29288]: 2021-02-10T04:43:38 2021-02-10T04:43:40.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:39 smithi164 conmon[29288]: .768991+0000 osd.4 (osd.4) 1748 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:40.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[35961]: cluster 2021-02-10T04:43:38.259249+0000 2021-02-10T04:43:40.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[35961]: mgr.y (mgr.14140) 908 : cluster [DBG] pgmap v1645: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 827 KiB/s rd, 2.9 MiB/s wr, 8 op/s 2021-02-10T04:43:40.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:43:40.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[35961]: 43:38.591047+0000 mgr.y (mgr.14140) 909 : audit [DBG] 2021-02-10T04:43:40.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[35961]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:43:40.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:43:40.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[35961]: 02-10T04:43:38 2021-02-10T04:43:40.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[35961]: .713019+0000 mon.a (mon. 2021-02-10T04:43:40.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[35961]: 0) 1991 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 847 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:43:40.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:43:40.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[35961]: 02-10T04:43:38.768968+0000 osd.4 (osd.4) 1745 : 2021-02-10T04:43:40.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:40.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:43:40.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[35961]: 02-10T04:43:38.768978+0000 osd.4 (osd.4) 1746 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:40.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[35961]: cluster 2021-02-10T04:43:38.768984+0000 osd.4 2021-02-10T04:43:40.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[35961]: (osd.4) 1747 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:40.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:43:40.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[35961]: 04:43:38.768991+0000 2021-02-10T04:43:40.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[35961]: osd.4 (osd.4) 1748 : 2021-02-10T04:43:40.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:40.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[29319]: cluster 2021-02-10T04:43:38.259249+0000 mgr.y 2021-02-10T04:43:40.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[29319]: (mgr.14140) 908 : cluster [DBG] pgmap v1645: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 827 KiB/s rd, 2.9 MiB/s wr, 8 op/s 2021-02-10T04:43:40.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[29319]: audit 2021-02-10T04:43:38.591047+0000 mgr.y (mgr.14140) 909 : audit [DBG] 2021-02-10T04:43:40.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[29319]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:43:40.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[29319]: cluster 2021-02-10T04:43:38. 2021-02-10T04:43:40.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[29319]: 713019+0000 mon.a (mon.0) 1991 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 847 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:43:40.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:43:40.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[29319]: -10T04:43:38.768968+0000 osd.4 ( 2021-02-10T04:43:40.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[29319]: osd.4) 1745 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:40.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:43:40.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[29319]: 10T04:43:38.768978+0000 osd.4 (osd. 2021-02-10T04:43:40.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[29319]: 4) 1746 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:40.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:43:40.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[29319]: -02-10T04:43 2021-02-10T04:43:40.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[29319]: :38.768984+0000 osd.4 (osd 2021-02-10T04:43:40.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[29319]: .4) 1747 : cluster [WRN] 2021-02-10T04:43:40.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[29319]: slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:40.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:43:40.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[29319]: -02-10T04:43 2021-02-10T04:43:40.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[29319]: :38.768991+0000 osd.4 ( 2021-02-10T04:43:40.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[29319]: osd.4) 1748 : cluster [WRN] 2021-02-10T04:43:40.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:39 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:41.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:40 smithi164 conmon[33677]: debug 2021-02-10T04:43:41.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:40 smithi164 conmon[33677]: 2021-02-10T04:43:40.748+0000 7fbec19a1700 -1 osd.4 996 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:41.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:40 smithi164 conmon[29288]: cluster 2021-02-10T04:43:41.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:40 smithi164 conmon[29288]: 2021-02-10T04:43:41.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:40 smithi164 conmon[29288]: 2021 2021-02-10T04:43:41.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:40 smithi164 conmon[29288]: - 2021-02-10T04:43:41.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:40 smithi164 conmon[29288]: 02 2021-02-10T04:43:41.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:40 smithi164 conmon[29288]: -10T04:43:39.796858 2021-02-10T04:43:41.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:40 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1749 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:41.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:40 smithi164 conmon[29288]: cluster 2021-02-10T04:43:41.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:40 smithi164 conmon[29288]: 2021-02-10T04:43:39. 2021-02-10T04:43:41.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:40 smithi164 conmon[29288]: 796866+0000 osd.4 (osd.4) 1750 2021-02-10T04:43:41.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:40 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:41.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:40 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:43:41.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:40 smithi164 conmon[29288]: 02-10T04:43:39.796869+0000 osd.4 2021-02-10T04:43:41.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:40 smithi164 conmon[29288]: (osd.4) 1751 : cluster 2021-02-10T04:43:41.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:40 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:41.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:40 smithi164 conmon[29288]: cluster 2021-02-10T04:43:41.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:40 smithi164 conmon[29288]: 2021-02-10T04:43:39.796873 2021-02-10T04:43:41.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:40 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1752 : cluster 2021-02-10T04:43:41.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:40 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:41.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:40 smithi085 conmon[29319]: cluster 2021-02-10T04:43:41.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:40 smithi085 conmon[29319]: 2021-02-10T04:43:41.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:40 smithi085 conmon[29319]: 2021 2021-02-10T04:43:41.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:40 smithi085 conmon[29319]: - 2021-02-10T04:43:41.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:40 smithi085 conmon[29319]: 02-10T04:43: 2021-02-10T04:43:41.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:40 smithi085 conmon[29319]: 39.796858+0000 osd.4 (osd.4) 1749 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:41.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:40 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:43:41.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:40 smithi085 conmon[29319]: -10T04:43:39.796866+0000 osd.4 (osd.4) 1750 : cluster [WRN] 2021-02-10T04:43:41.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:40 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:41.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:40 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:43:41.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:40 smithi085 conmon[29319]: 04: 2021-02-10T04:43:41.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:40 smithi085 conmon[29319]: 43:39.796869+0000 osd.4 (osd.4) 1751 : cluster 2021-02-10T04:43:41.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:40 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:41.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:40 smithi085 conmon[29319]: cluster 2021-02-10T04:43:41.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:40 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:43:41.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:40 smithi085 conmon[29319]: 43:39.796873+0000 osd.4 (osd.4) 1752 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:41.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:40 smithi085 conmon[35961]: cluster 2021-02-10T04:43:41.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:40 smithi085 conmon[35961]: 2021-02-10T04:43 2021-02-10T04:43:41.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:40 smithi085 conmon[35961]: :39.796858+0000 osd.4 (osd.4) 1749 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:41.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:40 smithi085 conmon[35961]: cluster 2021-02-10T04:43:41.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:40 smithi085 conmon[35961]: 2021-02-10T04:43:39 2021-02-10T04:43:41.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:40 smithi085 conmon[35961]: .796866+0000 osd.4 (osd.4) 1750 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:41.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:40 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:43:41.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:40 smithi085 conmon[35961]: 02-10T04:43:39.796869+0000 osd.4 (osd.4) 1751 : 2021-02-10T04:43:41.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:40 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:41.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:40 smithi085 conmon[35961]: cluster 2021-02-10T04:43:41.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:40 smithi085 conmon[35961]: 2021-02-10T04:43:39.796873+0000 osd.4 (osd.4 2021-02-10T04:43:41.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:40 smithi085 conmon[35961]: ) 1752 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:42.014 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:41 smithi085 conmon[29319]: cluster 2021-02-10T04:43:40.260145+0000 mgr.y (mgr 2021-02-10T04:43:42.014 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:41 smithi085 conmon[29319]: .14140) 910 : cluster [DBG] pgmap v1647: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 879 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 1.8 MiB/s wr, 3 op/s 2021-02-10T04:43:42.014 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:41 smithi085 conmon[29319]: cluster 2021-02-10T04:43:42.014 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:41 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:43:42.014 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:41 smithi085 conmon[29319]: :43:40.749264+0000 osd.4 (osd.4) 1753 : cluster [WRN] 2021-02-10T04:43:42.014 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:41 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:42.014 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:41 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:43:42.015 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:41 smithi085 conmon[29319]: -10T04:43:40.749273 2021-02-10T04:43:42.015 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:41 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1754 : cluster [WRN] 2021-02-10T04:43:42.015 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:41 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:42.015 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:41 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:43:42.015 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:41 smithi085 conmon[29319]: 04:43:40.749278+0000 osd.4 (osd.4 2021-02-10T04:43:42.015 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:41 smithi085 conmon[29319]: ) 1755 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:42.015 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:41 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:43:42.015 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:41 smithi085 conmon[29319]: 10T04:43:40.749285+0000 osd.4 ( 2021-02-10T04:43:42.015 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:41 smithi085 conmon[29319]: osd.4) 1756 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:42.015 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:41 smithi085 conmon[35961]: cluster 2021-02-10T04:43:40.260145+0000 mgr.y ( 2021-02-10T04:43:42.015 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:41 smithi085 conmon[35961]: mgr.14140) 910 : cluster [DBG] pgmap v1647: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 879 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 1.8 MiB/s wr, 3 op/s 2021-02-10T04:43:42.015 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:41 smithi085 conmon[35961]: cluster 2021-02-10T04:43:40 2021-02-10T04:43:42.016 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:41 smithi085 conmon[35961]: .749264+0000 osd.4 (osd.4) 1753 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:42.016 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:41 smithi085 conmon[35961]: cluster 2021-02-10T04:43:40.749273+0000 2021-02-10T04:43:42.016 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:41 smithi085 conmon[35961]: osd.4 (osd.4) 1754 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:42.016 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:41 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:43:42.016 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:41 smithi085 conmon[35961]: T04:43:40.749278+0000 osd.4 (osd.4 2021-02-10T04:43:42.016 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:41 smithi085 conmon[35961]: ) 1755 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:42.016 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:41 smithi085 conmon[35961]: cluster 2021-02-10T04:43:42.016 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:41 smithi085 conmon[35961]: 2021-02-10T04:43:40. 2021-02-10T04:43:42.016 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:41 smithi085 conmon[35961]: 749285+0000 osd.4 (osd.4) 1756 : cluster 2021-02-10T04:43:42.016 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:41 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:42.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:41 smithi164 conmon[33677]: debug 2021-02-10T04:43:41.772+0000 7fbec19a1700 -1 osd.4 996 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:41 smithi164 conmon[29288]: cluster 2021-02-10T04:43:40.260145+0000 mgr.y 2021-02-10T04:43:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:41 smithi164 conmon[29288]: (mgr.14140) 910 : cluster [DBG] pgmap v1647: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 879 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 1.8 MiB/s wr, 3 op/s 2021-02-10T04:43:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:41 smithi164 conmon[29288]: cluster 2021-02-10T04:43:40.749264+0000 osd.4 (osd.4) 1753 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:41 smithi164 conmon[29288]: cluster 2021-02-10T04:43:40.749273+0000 osd.4 ( 2021-02-10T04:43:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:41 smithi164 conmon[29288]: osd.4) 1754 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:41 smithi164 conmon[29288]: cluster 2021-02-10T04:43:40.749278+0000 osd.4 2021-02-10T04:43:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:41 smithi164 conmon[29288]: (osd.4) 1755 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:41 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:43:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:41 smithi164 conmon[29288]: 10T04:43:40.749285+0000 osd.4 (osd.4) 1756 : cluster 2021-02-10T04:43:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:41 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:42.199 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":1639,"stamp":"2021-02-10T04:43:32.346756+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":913905436,"num_objects":730,"num_object_clones":323,"num_object_copies":2190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":730,"num_whiteouts":9,"num_read":9688,"num_read_kb":7590402,"num_write":6465,"num_write_kb":2070213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":431,"num_bytes_recovered":108949605,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16189,"ondisk_log_size":16189,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2176512,"kb_used_data":1837424,"kb_used_omap":0,"kb_used_meta":338624,"kb_avail":654039552,"statfs":{"total":671965249536,"available":669736501248,"internally_reserved":0,"allocated":1881522176,"data_stored":2542529491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":346750976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,4,0,0,0,0,0,0,0,0,0,0,0,0,0,4],"upper_bound":2097152},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1525232,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":8406,"num_write":27,"num_write_kb":10095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.007545"},"pg_stats":[{"pgid":"5.9","version":"131'3","reported_seq":"1056","reported_epoch":"987","state":"active+clean","last_fresh":"2021-02-10T04:43:27.898237+0000","last_change":"2021-02-10T04:33:57.871495+0000","last_active":"2021-02-10T04:43:27.898237+0000","last_peered":"2021-02-10T04:43:27.898237+0000","last_clean":"2021-02-10T04:43:27.898237+0000","last_became_active":"2021-02-10T04:29:07.145525+0000","last_became_peered":"2021-02-10T04:29:07.145525+0000","last_unstale":"2021-02-10T04:43:27.898237+0000","last_undegraded":"2021-02-10T04:43:27.898237+0000","last_fullsized":"2021-02-10T04:43:27.898237+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'3","last_scrub_stamp":"2021-02-10T04:33:57.871402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:33:57.871402+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"538'42","reported_seq":"1034","reported_epoch":"986","state":"active+clean","last_fresh":"2021-02-10T04:43:26.333263+0000","last_change":"2021-02-10T04:33:55.585872+0000","last_active":"2021-02-10T04:43:26.333263+0000","last_peered":"2021-02-10T04:43:26.333263+0000","last_clean":"2021-02-10T04:43:26.333263+0000","last_became_active":"2021-02-10T04:25:49.002058+0000","last_became_peered":"2021-02-10T04:25:49.002058+0000","last_unstale":"2021-02-10T04:43:26.333263+0000","last_undegraded":"2021-02-10T04:43:26.333263+0000","last_fullsized":"2021-02-10T04:43:26.333263+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2021-02-10T04:33:55.585807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:55.585807+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":77,"num_read_kb":60,"num_write":43,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"389","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.347682+0000","last_change":"2021-02-10T04:37:37.973234+0000","last_active":"2021-02-10T04:43:28.347682+0000","last_peered":"2021-02-10T04:43:28.347682+0000","last_clean":"2021-02-10T04:43:28.347682+0000","last_became_active":"2021-02-10T04:37:37.246903+0000","last_became_peered":"2021-02-10T04:37:37.246903+0000","last_unstale":"2021-02-10T04:43:28.347682+0000","last_undegraded":"2021-02-10T04:43:28.347682+0000","last_fullsized":"2021-02-10T04:43:28.347682+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:29:34.787265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:34.787265+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"946","reported_epoch":"986","state":"active+clean","last_fresh":"2021-02-10T04:43:26.335424+0000","last_change":"2021-02-10T04:29:19.405593+0000","last_active":"2021-02-10T04:43:26.335424+0000","last_peered":"2021-02-10T04:43:26.335424+0000","last_clean":"2021-02-10T04:43:26.335424+0000","last_became_active":"2021-02-10T04:25:53.148090+0000","last_became_peered":"2021-02-10T04:25:53.148090+0000","last_unstale":"2021-02-10T04:43:26.335424+0000","last_undegraded":"2021-02-10T04:43:26.335424+0000","last_fullsized":"2021-02-10T04:43:26.335424+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:19.405525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:29:19.405525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"985'1635","reported_seq":"4374","reported_epoch":"987","state":"active+clean+snaptrim_wait","last_fresh":"2021-02-10T04:43:27.731800+0000","last_change":"2021-02-10T04:43:27.731882+0000","last_active":"2021-02-10T04:43:27.731800+0000","last_peered":"2021-02-10T04:43:27.731800+0000","last_clean":"2021-02-10T04:43:27.731800+0000","last_became_active":"2021-02-10T04:33:05.144488+0000","last_became_peered":"2021-02-10T04:33:05.144488+0000","last_unstale":"2021-02-10T04:43:27.731800+0000","last_undegraded":"2021-02-10T04:43:27.731800+0000","last_fullsized":"2021-02-10T04:43:27.731800+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'301","last_scrub_stamp":"2021-02-10T04:33:56.571925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:56.571925+0000","log_size":1635,"ondisk_log_size":1635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":21,"stat_sum":{"num_bytes":133045055,"num_objects":51,"num_object_clones":46,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":764,"num_read_kb":814702,"num_write":556,"num_write_kb":217168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24e","length":"2"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"3"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"5"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2c9","length":"3"},{"start":"2d1","length":"7"},{"start":"2e0","length":"3"},{"start":"2e4","length":"1"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"}]},{"pgid":"4.b","version":"538'26","reported_seq":"1038","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.414340+0000","last_change":"2021-02-10T04:38:39.265973+0000","last_active":"2021-02-10T04:43:28.414340+0000","last_peered":"2021-02-10T04:43:28.414340+0000","last_clean":"2021-02-10T04:43:28.414340+0000","last_became_active":"2021-02-10T04:38:39.243413+0000","last_became_peered":"2021-02-10T04:38:39.243413+0000","last_unstale":"2021-02-10T04:43:28.414340+0000","last_undegraded":"2021-02-10T04:43:28.414340+0000","last_fullsized":"2021-02-10T04:43:28.414340+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'26","last_scrub_stamp":"2021-02-10T04:38:32.145511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:32.145511+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":53,"num_read_kb":36,"num_write":27,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":894,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"634","reported_epoch":"990","state":"active+clean","last_fresh":"2021-02-10T04:43:30.346600+0000","last_change":"2021-02-10T04:33:05.151251+0000","last_active":"2021-02-10T04:43:30.346600+0000","last_peered":"2021-02-10T04:43:30.346600+0000","last_clean":"2021-02-10T04:43:30.346600+0000","last_became_active":"2021-02-10T04:33:05.150882+0000","last_became_peered":"2021-02-10T04:33:05.150882+0000","last_unstale":"2021-02-10T04:43:30.346600+0000","last_undegraded":"2021-02-10T04:43:30.346600+0000","last_fullsized":"2021-02-10T04:43:30.346600+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:31.102558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:31.102558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"363","reported_epoch":"990","state":"active+clean","last_fresh":"2021-02-10T04:43:30.345923+0000","last_change":"2021-02-10T04:37:37.244470+0000","last_active":"2021-02-10T04:43:30.345923+0000","last_peered":"2021-02-10T04:43:30.345923+0000","last_clean":"2021-02-10T04:43:30.345923+0000","last_became_active":"2021-02-10T04:37:37.242485+0000","last_became_peered":"2021-02-10T04:37:37.242485+0000","last_unstale":"2021-02-10T04:43:30.345923+0000","last_undegraded":"2021-02-10T04:43:30.345923+0000","last_fullsized":"2021-02-10T04:43:30.345923+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:35.805326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.805326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"959","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.414276+0000","last_change":"2021-02-10T04:37:37.254421+0000","last_active":"2021-02-10T04:43:28.414276+0000","last_peered":"2021-02-10T04:43:28.414276+0000","last_clean":"2021-02-10T04:43:28.414276+0000","last_became_active":"2021-02-10T04:37:37.254038+0000","last_became_peered":"2021-02-10T04:37:37.254038+0000","last_unstale":"2021-02-10T04:43:28.414276+0000","last_undegraded":"2021-02-10T04:43:28.414276+0000","last_fullsized":"2021-02-10T04:43:28.414276+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:48.700418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:48.700418+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"259'74","reported_seq":"1871","reported_epoch":"991","state":"active+clean","last_fresh":"2021-02-10T04:43:31.349363+0000","last_change":"2021-02-10T04:43:31.349748+0000","last_active":"2021-02-10T04:43:31.349363+0000","last_peered":"2021-02-10T04:43:31.349363+0000","last_clean":"2021-02-10T04:43:31.349363+0000","last_became_active":"2021-02-10T04:37:37.246533+0000","last_became_peered":"2021-02-10T04:37:37.246533+0000","last_unstale":"2021-02-10T04:43:31.349363+0000","last_undegraded":"2021-02-10T04:43:31.349363+0000","last_fullsized":"2021-02-10T04:43:31.349363+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'62","last_scrub_stamp":"2021-02-10T04:30:10.905905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:10.905905+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2437048,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":48,"num_read_kb":56455,"num_write":38,"num_write_kb":14043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1134570,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24e","length":"2"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"3"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"5"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2c9","length":"3"},{"start":"2d1","length":"7"},{"start":"2dd","length":"1"},{"start":"2df","length":"6"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"307","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"333","length":"1"},{"start":"339","length":"1"},{"start":"33d","length":"1"},{"start":"340","length":"1"},{"start":"342","length":"1"},{"start":"347","length":"1"},{"start":"34a","length":"1"},{"start":"34c","length":"1"},{"start":"34f","length":"3"},{"start":"358","length":"1"},{"start":"364","length":"1"},{"start":"369","length":"1"}]},{"pgid":"4.a","version":"538'45","reported_seq":"1031","reported_epoch":"987","state":"active+clean","last_fresh":"2021-02-10T04:43:27.341195+0000","last_change":"2021-02-10T04:29:26.671051+0000","last_active":"2021-02-10T04:43:27.341195+0000","last_peered":"2021-02-10T04:43:27.341195+0000","last_clean":"2021-02-10T04:43:27.341195+0000","last_became_active":"2021-02-10T04:25:49.015913+0000","last_became_peered":"2021-02-10T04:25:49.015913+0000","last_unstale":"2021-02-10T04:43:27.341195+0000","last_undegraded":"2021-02-10T04:43:27.341195+0000","last_fullsized":"2021-02-10T04:43:27.341195+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'29","last_scrub_stamp":"2021-02-10T04:29:26.670950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.670950+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":80,"num_read_kb":60,"num_write":46,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"642'3","reported_seq":"461","reported_epoch":"986","state":"active+clean","last_fresh":"2021-02-10T04:43:26.337113+0000","last_change":"2021-02-10T04:38:24.163263+0000","last_active":"2021-02-10T04:43:26.337113+0000","last_peered":"2021-02-10T04:43:26.337113+0000","last_clean":"2021-02-10T04:43:26.337113+0000","last_became_active":"2021-02-10T04:37:37.234230+0000","last_became_peered":"2021-02-10T04:37:37.234230+0000","last_unstale":"2021-02-10T04:43:26.337113+0000","last_undegraded":"2021-02-10T04:43:26.337113+0000","last_fullsized":"2021-02-10T04:43:26.337113+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'3","last_scrub_stamp":"2021-02-10T04:38:24.163200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:24.163200+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"367","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.414174+0000","last_change":"2021-02-10T04:38:31.121100+0000","last_active":"2021-02-10T04:43:28.414174+0000","last_peered":"2021-02-10T04:43:28.414174+0000","last_clean":"2021-02-10T04:43:28.414174+0000","last_became_active":"2021-02-10T04:37:37.230818+0000","last_became_peered":"2021-02-10T04:37:37.230818+0000","last_unstale":"2021-02-10T04:43:28.414174+0000","last_undegraded":"2021-02-10T04:43:28.414174+0000","last_fullsized":"2021-02-10T04:43:28.414174+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:31.121038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:38:31.121038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"365","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.344745+0000","last_change":"2021-02-10T04:37:37.245491+0000","last_active":"2021-02-10T04:43:28.344745+0000","last_peered":"2021-02-10T04:43:28.344745+0000","last_clean":"2021-02-10T04:43:28.344745+0000","last_became_active":"2021-02-10T04:37:37.245113+0000","last_became_peered":"2021-02-10T04:37:37.245113+0000","last_unstale":"2021-02-10T04:43:28.344745+0000","last_undegraded":"2021-02-10T04:43:28.344745+0000","last_fullsized":"2021-02-10T04:43:28.344745+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.731458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.731458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"989'721","reported_seq":"3547","reported_epoch":"989","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:43:29.850939+0000","last_change":"2021-02-10T04:43:29.827162+0000","last_active":"2021-02-10T04:43:29.850939+0000","last_peered":"2021-02-10T04:43:29.850939+0000","last_clean":"2021-02-10T04:43:29.850939+0000","last_became_active":"2021-02-10T04:28:25.150246+0000","last_became_peered":"2021-02-10T04:28:25.150246+0000","last_unstale":"2021-02-10T04:43:29.850939+0000","last_undegraded":"2021-02-10T04:43:29.850939+0000","last_fullsized":"2021-02-10T04:43:29.850939+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'63","last_scrub_stamp":"2021-02-10T04:29:39.700115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:29:39.700115+0000","log_size":721,"ondisk_log_size":721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"stat_sum":{"num_bytes":50271403,"num_objects":20,"num_object_clones":18,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":259,"num_read_kb":219088,"num_write":204,"num_write_kb":92347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24e","length":"2"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"3"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"5"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2c9","length":"3"},{"start":"2d1","length":"7"},{"start":"2dd","length":"1"},{"start":"2e0","length":"3"},{"start":"2e4","length":"1"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"}]},{"pgid":"4.d","version":"538'47","reported_seq":"1057","reported_epoch":"990","state":"active+clean","last_fresh":"2021-02-10T04:43:30.346686+0000","last_change":"2021-02-10T04:37:37.821860+0000","last_active":"2021-02-10T04:43:30.346686+0000","last_peered":"2021-02-10T04:43:30.346686+0000","last_clean":"2021-02-10T04:43:30.346686+0000","last_became_active":"2021-02-10T04:37:37.237215+0000","last_became_peered":"2021-02-10T04:37:37.237215+0000","last_unstale":"2021-02-10T04:43:30.346686+0000","last_undegraded":"2021-02-10T04:43:30.346686+0000","last_fullsized":"2021-02-10T04:43:30.346686+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'33","last_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:28:49.514984+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":62,"num_read_kb":44,"num_write":36,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"871","reported_epoch":"990","state":"active+clean","last_fresh":"2021-02-10T04:43:30.346040+0000","last_change":"2021-02-10T04:29:07.146266+0000","last_active":"2021-02-10T04:43:30.346040+0000","last_peered":"2021-02-10T04:43:30.346040+0000","last_clean":"2021-02-10T04:43:30.346040+0000","last_became_active":"2021-02-10T04:29:07.145616+0000","last_became_peered":"2021-02-10T04:29:07.145616+0000","last_unstale":"2021-02-10T04:43:30.346040+0000","last_undegraded":"2021-02-10T04:43:30.346040+0000","last_fullsized":"2021-02-10T04:43:30.346040+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:44.410817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:44.410817+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"634","reported_epoch":"990","state":"active+clean","last_fresh":"2021-02-10T04:43:30.346139+0000","last_change":"2021-02-10T04:33:05.147166+0000","last_active":"2021-02-10T04:43:30.346139+0000","last_peered":"2021-02-10T04:43:30.346139+0000","last_clean":"2021-02-10T04:43:30.346139+0000","last_became_active":"2021-02-10T04:33:05.146783+0000","last_became_peered":"2021-02-10T04:33:05.146783+0000","last_unstale":"2021-02-10T04:43:30.346139+0000","last_undegraded":"2021-02-10T04:43:30.346139+0000","last_fullsized":"2021-02-10T04:43:30.346139+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:35.958255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.958255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"367","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.348678+0000","last_change":"2021-02-10T04:38:32.875574+0000","last_active":"2021-02-10T04:43:28.348678+0000","last_peered":"2021-02-10T04:43:28.348678+0000","last_clean":"2021-02-10T04:43:28.348678+0000","last_became_active":"2021-02-10T04:37:37.251479+0000","last_became_peered":"2021-02-10T04:37:37.251479+0000","last_unstale":"2021-02-10T04:43:28.348678+0000","last_undegraded":"2021-02-10T04:43:28.348678+0000","last_fullsized":"2021-02-10T04:43:28.348678+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:32.875498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:38:32.875498+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"987'605","reported_seq":"4068","reported_epoch":"987","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:43:27.622655+0000","last_change":"2021-02-10T04:43:25.619705+0000","last_active":"2021-02-10T04:43:27.622655+0000","last_peered":"2021-02-10T04:43:27.622655+0000","last_clean":"2021-02-10T04:43:27.622655+0000","last_became_active":"2021-02-10T04:37:40.267794+0000","last_became_peered":"2021-02-10T04:37:40.267794+0000","last_unstale":"2021-02-10T04:43:27.622655+0000","last_undegraded":"2021-02-10T04:43:27.622655+0000","last_fullsized":"2021-02-10T04:43:27.622655+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"688'187","last_scrub_stamp":"2021-02-10T04:38:27.225488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:38:27.225488+0000","log_size":605,"ondisk_log_size":605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"stat_sum":{"num_bytes":46644583,"num_objects":22,"num_object_clones":19,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":339,"num_read_kb":364544,"num_write":220,"num_write_kb":64761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":11919937,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24e","length":"2"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"3"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"3"},{"start":"2ad","length":"1"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2c9","length":"3"},{"start":"2d1","length":"7"},{"start":"2e0","length":"5"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"339","length":"1"},{"start":"33d","length":"1"},{"start":"340","length":"1"},{"start":"34f","length":"1"}]},{"pgid":"1.8","version":"0'0","reported_seq":"898","reported_epoch":"986","state":"active+clean","last_fresh":"2021-02-10T04:43:26.333266+0000","last_change":"2021-02-10T04:37:37.246395+0000","last_active":"2021-02-10T04:43:26.333266+0000","last_peered":"2021-02-10T04:43:26.333266+0000","last_clean":"2021-02-10T04:43:26.333266+0000","last_became_active":"2021-02-10T04:37:37.246004+0000","last_became_peered":"2021-02-10T04:37:37.246004+0000","last_unstale":"2021-02-10T04:43:26.333266+0000","last_undegraded":"2021-02-10T04:43:26.333266+0000","last_fullsized":"2021-02-10T04:43:26.333266+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:37.336092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:37.336092+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"974","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.344816+0000","last_change":"2021-02-10T04:37:37.259938+0000","last_active":"2021-02-10T04:43:28.344816+0000","last_peered":"2021-02-10T04:43:28.344816+0000","last_clean":"2021-02-10T04:43:28.344816+0000","last_became_active":"2021-02-10T04:37:37.259532+0000","last_became_peered":"2021-02-10T04:37:37.259532+0000","last_unstale":"2021-02-10T04:43:28.344816+0000","last_undegraded":"2021-02-10T04:43:28.344816+0000","last_fullsized":"2021-02-10T04:43:28.344816+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:51.858386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:33:51.858386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"538'23","reported_seq":"989","reported_epoch":"990","state":"active+clean","last_fresh":"2021-02-10T04:43:30.346208+0000","last_change":"2021-02-10T04:28:41.427119+0000","last_active":"2021-02-10T04:43:30.346208+0000","last_peered":"2021-02-10T04:43:30.346208+0000","last_clean":"2021-02-10T04:43:30.346208+0000","last_became_active":"2021-02-10T04:25:49.001725+0000","last_became_peered":"2021-02-10T04:25:49.001725+0000","last_unstale":"2021-02-10T04:43:30.346208+0000","last_undegraded":"2021-02-10T04:43:30.346208+0000","last_fullsized":"2021-02-10T04:43:30.346208+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'13","last_scrub_stamp":"2021-02-10T04:28:41.427050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:41.427050+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":28,"num_write":23,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"637","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.346008+0000","last_change":"2021-02-10T04:33:05.168734+0000","last_active":"2021-02-10T04:43:28.346008+0000","last_peered":"2021-02-10T04:43:28.346008+0000","last_clean":"2021-02-10T04:43:28.346008+0000","last_became_active":"2021-02-10T04:33:05.168314+0000","last_became_peered":"2021-02-10T04:33:05.168314+0000","last_unstale":"2021-02-10T04:43:28.346008+0000","last_undegraded":"2021-02-10T04:43:28.346008+0000","last_fullsized":"2021-02-10T04:43:28.346008+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:18.107867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:18.107867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"639'3","reported_seq":"449","reported_epoch":"987","state":"active+clean","last_fresh":"2021-02-10T04:43:27.341359+0000","last_change":"2021-02-10T04:38:31.211769+0000","last_active":"2021-02-10T04:43:27.341359+0000","last_peered":"2021-02-10T04:43:27.341359+0000","last_clean":"2021-02-10T04:43:27.341359+0000","last_became_active":"2021-02-10T04:37:37.233738+0000","last_became_peered":"2021-02-10T04:37:37.233738+0000","last_unstale":"2021-02-10T04:43:27.341359+0000","last_undegraded":"2021-02-10T04:43:27.341359+0000","last_fullsized":"2021-02-10T04:43:27.341359+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"639'3","last_scrub_stamp":"2021-02-10T04:38:31.211662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:38:31.211662+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"958","reported_epoch":"990","state":"active+clean","last_fresh":"2021-02-10T04:43:30.347040+0000","last_change":"2021-02-10T04:37:37.236009+0000","last_active":"2021-02-10T04:43:30.347040+0000","last_peered":"2021-02-10T04:43:30.347040+0000","last_clean":"2021-02-10T04:43:30.347040+0000","last_became_active":"2021-02-10T04:37:37.233109+0000","last_became_peered":"2021-02-10T04:37:37.233109+0000","last_unstale":"2021-02-10T04:43:30.347040+0000","last_undegraded":"2021-02-10T04:43:30.347040+0000","last_fullsized":"2021-02-10T04:43:30.347040+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:38.873102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.873102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"989'1063","reported_seq":"6054","reported_epoch":"989","state":"active+clean+snaptrim_wait","last_fresh":"2021-02-10T04:43:29.824941+0000","last_change":"2021-02-10T04:43:29.825020+0000","last_active":"2021-02-10T04:43:29.824941+0000","last_peered":"2021-02-10T04:43:29.824941+0000","last_clean":"2021-02-10T04:43:29.824941+0000","last_became_active":"2021-02-10T04:28:25.149226+0000","last_became_peered":"2021-02-10T04:28:25.149226+0000","last_unstale":"2021-02-10T04:43:29.824941+0000","last_undegraded":"2021-02-10T04:43:29.824941+0000","last_fullsized":"2021-02-10T04:43:29.824941+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'236","last_scrub_stamp":"2021-02-10T04:33:50.351534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:50.351534+0000","log_size":1063,"ondisk_log_size":1063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":28,"stat_sum":{"num_bytes":81878110,"num_objects":35,"num_object_clones":31,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":1,"num_read":558,"num_read_kb":622116,"num_write":371,"num_write_kb":133580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24e","length":"2"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"3"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"5"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2ca","length":"2"},{"start":"2d1","length":"7"},{"start":"2e0","length":"3"},{"start":"2e4","length":"1"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"}]},{"pgid":"1.a","version":"0'0","reported_seq":"898","reported_epoch":"986","state":"active+clean","last_fresh":"2021-02-10T04:43:26.335289+0000","last_change":"2021-02-10T04:37:37.232619+0000","last_active":"2021-02-10T04:43:26.335289+0000","last_peered":"2021-02-10T04:43:26.335289+0000","last_clean":"2021-02-10T04:43:26.335289+0000","last_became_active":"2021-02-10T04:37:37.229648+0000","last_became_peered":"2021-02-10T04:37:37.229648+0000","last_unstale":"2021-02-10T04:43:26.335289+0000","last_undegraded":"2021-02-10T04:43:26.335289+0000","last_fullsized":"2021-02-10T04:43:26.335289+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.384170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.384170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"538'29","reported_seq":"693","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.348323+0000","last_change":"2021-02-10T04:33:05.154998+0000","last_active":"2021-02-10T04:43:28.348323+0000","last_peered":"2021-02-10T04:43:28.348323+0000","last_clean":"2021-02-10T04:43:28.348323+0000","last_became_active":"2021-02-10T04:33:05.149026+0000","last_became_peered":"2021-02-10T04:33:05.149026+0000","last_unstale":"2021-02-10T04:43:28.348323+0000","last_undegraded":"2021-02-10T04:43:28.348323+0000","last_fullsized":"2021-02-10T04:43:28.348323+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'17","last_scrub_stamp":"2021-02-10T04:29:37.933206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:37.933206+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":49,"num_read_kb":36,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"697","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.345358+0000","last_change":"2021-02-10T04:33:05.169654+0000","last_active":"2021-02-10T04:43:28.345358+0000","last_peered":"2021-02-10T04:43:28.345358+0000","last_clean":"2021-02-10T04:43:28.345358+0000","last_became_active":"2021-02-10T04:33:05.169289+0000","last_became_peered":"2021-02-10T04:33:05.169289+0000","last_unstale":"2021-02-10T04:43:28.345358+0000","last_undegraded":"2021-02-10T04:43:28.345358+0000","last_fullsized":"2021-02-10T04:43:28.345358+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_clean_scrub_stamp":"2021-02-10T04:28:39.128781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"78'3","reported_seq":"1166","reported_epoch":"990","state":"active+clean","last_fresh":"2021-02-10T04:43:30.346961+0000","last_change":"2021-02-10T04:28:43.509446+0000","last_active":"2021-02-10T04:43:30.346961+0000","last_peered":"2021-02-10T04:43:30.346961+0000","last_clean":"2021-02-10T04:43:30.346961+0000","last_became_active":"2021-02-10T04:25:51.025218+0000","last_became_peered":"2021-02-10T04:25:51.025218+0000","last_unstale":"2021-02-10T04:43:30.346961+0000","last_undegraded":"2021-02-10T04:43:30.346961+0000","last_fullsized":"2021-02-10T04:43:30.346961+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:28:43.509353+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:28:43.509353+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"946","reported_epoch":"986","state":"active+clean","last_fresh":"2021-02-10T04:43:26.336156+0000","last_change":"2021-02-10T04:28:45.481538+0000","last_active":"2021-02-10T04:43:26.336156+0000","last_peered":"2021-02-10T04:43:26.336156+0000","last_clean":"2021-02-10T04:43:26.336156+0000","last_became_active":"2021-02-10T04:25:53.042697+0000","last_became_peered":"2021-02-10T04:25:53.042697+0000","last_unstale":"2021-02-10T04:43:26.336156+0000","last_undegraded":"2021-02-10T04:43:26.336156+0000","last_fullsized":"2021-02-10T04:43:26.336156+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:45.481472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:28:45.481472+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"987'1066","reported_seq":"5061","reported_epoch":"987","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:43:27.801195+0000","last_change":"2021-02-10T04:43:27.798025+0000","last_active":"2021-02-10T04:43:27.801195+0000","last_peered":"2021-02-10T04:43:27.801195+0000","last_clean":"2021-02-10T04:43:27.801195+0000","last_became_active":"2021-02-10T04:28:25.148500+0000","last_became_peered":"2021-02-10T04:28:25.148500+0000","last_unstale":"2021-02-10T04:43:27.801195+0000","last_undegraded":"2021-02-10T04:43:27.801195+0000","last_fullsized":"2021-02-10T04:43:27.801195+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'110","last_scrub_stamp":"2021-02-10T04:30:18.463782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:18.463782+0000","log_size":1066,"ondisk_log_size":1066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"stat_sum":{"num_bytes":48821292,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":491,"num_read_kb":552336,"num_write":351,"num_write_kb":129921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24e","length":"2"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"3"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"5"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2c9","length":"3"},{"start":"2d1","length":"7"},{"start":"2dd","length":"1"},{"start":"2e0","length":"3"},{"start":"2e4","length":"1"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"339","length":"1"},{"start":"340","length":"1"},{"start":"34f","length":"1"}]},{"pgid":"4.e","version":"538'39","reported_seq":"1017","reported_epoch":"990","state":"active+clean","last_fresh":"2021-02-10T04:43:30.346744+0000","last_change":"2021-02-10T04:28:11.693702+0000","last_active":"2021-02-10T04:43:30.346744+0000","last_peered":"2021-02-10T04:43:30.346744+0000","last_clean":"2021-02-10T04:43:30.346744+0000","last_became_active":"2021-02-10T04:25:49.011093+0000","last_became_peered":"2021-02-10T04:25:49.011093+0000","last_unstale":"2021-02-10T04:43:30.346744+0000","last_undegraded":"2021-02-10T04:43:30.346744+0000","last_fullsized":"2021-02-10T04:43:30.346744+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'31","last_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:28:11.693616+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":56,"num_read_kb":40,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"933","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.413779+0000","last_change":"2021-02-10T04:29:31.683314+0000","last_active":"2021-02-10T04:43:28.413779+0000","last_peered":"2021-02-10T04:43:28.413779+0000","last_clean":"2021-02-10T04:43:28.413779+0000","last_became_active":"2021-02-10T04:29:07.160593+0000","last_became_peered":"2021-02-10T04:29:07.160593+0000","last_unstale":"2021-02-10T04:43:28.413779+0000","last_undegraded":"2021-02-10T04:43:28.413779+0000","last_fullsized":"2021-02-10T04:43:28.413779+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:31.683213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.683213+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"953","reported_epoch":"986","state":"active+clean","last_fresh":"2021-02-10T04:43:26.336312+0000","last_change":"2021-02-10T04:30:10.487631+0000","last_active":"2021-02-10T04:43:26.336312+0000","last_peered":"2021-02-10T04:43:26.336312+0000","last_clean":"2021-02-10T04:43:26.336312+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:43:26.336312+0000","last_undegraded":"2021-02-10T04:43:26.336312+0000","last_fullsized":"2021-02-10T04:43:26.336312+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:10.487528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:30:10.487528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"947","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.347003+0000","last_change":"2021-02-10T04:33:55.849675+0000","last_active":"2021-02-10T04:43:28.347003+0000","last_peered":"2021-02-10T04:43:28.347003+0000","last_clean":"2021-02-10T04:43:28.347003+0000","last_became_active":"2021-02-10T04:25:53.148054+0000","last_became_peered":"2021-02-10T04:25:53.148054+0000","last_unstale":"2021-02-10T04:43:28.347003+0000","last_undegraded":"2021-02-10T04:43:28.347003+0000","last_fullsized":"2021-02-10T04:43:28.347003+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:55.849582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:33:55.849582+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"986'1143","reported_seq":"4043","reported_epoch":"988","state":"active+clean+snaptrim_wait","last_fresh":"2021-02-10T04:43:28.575298+0000","last_change":"2021-02-10T04:43:28.575398+0000","last_active":"2021-02-10T04:43:28.575298+0000","last_peered":"2021-02-10T04:43:28.575298+0000","last_clean":"2021-02-10T04:43:28.575298+0000","last_became_active":"2021-02-10T04:28:25.152030+0000","last_became_peered":"2021-02-10T04:28:25.152030+0000","last_unstale":"2021-02-10T04:43:28.575298+0000","last_undegraded":"2021-02-10T04:43:28.575298+0000","last_fullsized":"2021-02-10T04:43:28.575298+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'283","last_scrub_stamp":"2021-02-10T04:33:53.834627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:53.834627+0000","log_size":1143,"ondisk_log_size":1143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"stat_sum":{"num_bytes":64446499,"num_objects":27,"num_object_clones":23,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":1,"num_read":573,"num_read_kb":599059,"num_write":400,"num_write_kb":178261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24e","length":"2"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"3"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"5"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2c9","length":"3"},{"start":"2d1","length":"7"},{"start":"2dd","length":"1"},{"start":"2e0","length":"5"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"339","length":"1"},{"start":"340","length":"1"},{"start":"34f","length":"1"}]},{"pgid":"5.0","version":"158'3","reported_seq":"1164","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.344645+0000","last_change":"2021-02-10T04:29:34.388978+0000","last_active":"2021-02-10T04:43:28.344645+0000","last_peered":"2021-02-10T04:43:28.344645+0000","last_clean":"2021-02-10T04:43:28.344645+0000","last_became_active":"2021-02-10T04:25:51.031906+0000","last_became_peered":"2021-02-10T04:25:51.031906+0000","last_unstale":"2021-02-10T04:43:28.344645+0000","last_undegraded":"2021-02-10T04:43:28.344645+0000","last_fullsized":"2021-02-10T04:43:28.344645+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'3","last_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_deep_scrub":"158'3","last_deep_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_clean_scrub_stamp":"2021-02-10T04:29:34.388912+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"538'40","reported_seq":"1022","reported_epoch":"990","state":"active+clean","last_fresh":"2021-02-10T04:43:30.345524+0000","last_change":"2021-02-10T04:29:36.813190+0000","last_active":"2021-02-10T04:43:30.345524+0000","last_peered":"2021-02-10T04:43:30.345524+0000","last_clean":"2021-02-10T04:43:30.345524+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:43:30.345524+0000","last_undegraded":"2021-02-10T04:43:30.345524+0000","last_fullsized":"2021-02-10T04:43:30.345524+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:29:36.813083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.813083+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":72,"num_read_kb":52,"num_write":40,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"999","reported_epoch":"987","state":"active+clean","last_fresh":"2021-02-10T04:43:27.341842+0000","last_change":"2021-02-10T04:37:37.245911+0000","last_active":"2021-02-10T04:43:27.341842+0000","last_peered":"2021-02-10T04:43:27.341842+0000","last_clean":"2021-02-10T04:43:27.341842+0000","last_became_active":"2021-02-10T04:37:37.245391+0000","last_became_peered":"2021-02-10T04:37:37.245391+0000","last_unstale":"2021-02-10T04:43:27.341842+0000","last_undegraded":"2021-02-10T04:43:27.341842+0000","last_fullsized":"2021-02-10T04:43:27.341842+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.632085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.632085+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"883","reported_epoch":"987","state":"active+clean","last_fresh":"2021-02-10T04:43:27.340813+0000","last_change":"2021-02-10T04:37:37.235415+0000","last_active":"2021-02-10T04:43:27.340813+0000","last_peered":"2021-02-10T04:43:27.340813+0000","last_clean":"2021-02-10T04:43:27.340813+0000","last_became_active":"2021-02-10T04:37:37.235082+0000","last_became_peered":"2021-02-10T04:37:37.235082+0000","last_unstale":"2021-02-10T04:43:27.340813+0000","last_undegraded":"2021-02-10T04:43:27.340813+0000","last_fullsized":"2021-02-10T04:43:27.340813+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.658374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.658374+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"989'997","reported_seq":"4411","reported_epoch":"989","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:43:29.615857+0000","last_change":"2021-02-10T04:43:29.615857+0000","last_active":"2021-02-10T04:43:29.615857+0000","last_peered":"2021-02-10T04:43:29.615857+0000","last_clean":"2021-02-10T04:43:29.615857+0000","last_became_active":"2021-02-10T04:37:45.352779+0000","last_became_peered":"2021-02-10T04:37:45.352779+0000","last_unstale":"2021-02-10T04:43:29.615857+0000","last_undegraded":"2021-02-10T04:43:29.615857+0000","last_fullsized":"2021-02-10T04:43:29.615857+0000","mapping_epoch":646,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":647,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":997,"ondisk_log_size":997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"stat_sum":{"num_bytes":51299875,"num_objects":22,"num_object_clones":20,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":2,"num_read":478,"num_read_kb":498806,"num_write":309,"num_write_kb":100208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":16580116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24e","length":"2"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"3"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"5"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2c9","length":"3"},{"start":"2d1","length":"7"},{"start":"2dd","length":"1"},{"start":"2df","length":"6"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"339","length":"1"},{"start":"33d","length":"1"},{"start":"340","length":"1"},{"start":"34f","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":"891","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.347048+0000","last_change":"2021-02-10T04:30:12.452072+0000","last_active":"2021-02-10T04:43:28.347048+0000","last_peered":"2021-02-10T04:43:28.347048+0000","last_clean":"2021-02-10T04:43:28.347048+0000","last_became_active":"2021-02-10T04:28:49.048525+0000","last_became_peered":"2021-02-10T04:28:49.048525+0000","last_unstale":"2021-02-10T04:43:28.347048+0000","last_undegraded":"2021-02-10T04:43:28.347048+0000","last_fullsized":"2021-02-10T04:43:28.347048+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:12.452000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:30:12.452000+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"951","reported_epoch":"990","state":"active+clean","last_fresh":"2021-02-10T04:43:30.345684+0000","last_change":"2021-02-10T04:29:40.859352+0000","last_active":"2021-02-10T04:43:30.345684+0000","last_peered":"2021-02-10T04:43:30.345684+0000","last_clean":"2021-02-10T04:43:30.345684+0000","last_became_active":"2021-02-10T04:25:51.018706+0000","last_became_peered":"2021-02-10T04:25:51.018706+0000","last_unstale":"2021-02-10T04:43:30.345684+0000","last_undegraded":"2021-02-10T04:43:30.345684+0000","last_fullsized":"2021-02-10T04:43:30.345684+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:40.859274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:40.859274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"538'60","reported_seq":"1053","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.344506+0000","last_change":"2021-02-10T04:29:28.398522+0000","last_active":"2021-02-10T04:43:28.344506+0000","last_peered":"2021-02-10T04:43:28.344506+0000","last_clean":"2021-02-10T04:43:28.344506+0000","last_became_active":"2021-02-10T04:25:48.997481+0000","last_became_peered":"2021-02-10T04:25:48.997481+0000","last_unstale":"2021-02-10T04:43:28.344506+0000","last_undegraded":"2021-02-10T04:43:28.344506+0000","last_fullsized":"2021-02-10T04:43:28.344506+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'48","last_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_deep_scrub":"152'48","last_deep_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.398429+0000","log_size":60,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":88,"num_read_kb":66,"num_write":48,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"645","reported_epoch":"987","state":"active+clean","last_fresh":"2021-02-10T04:43:27.341561+0000","last_change":"2021-02-10T04:33:05.142730+0000","last_active":"2021-02-10T04:43:27.341561+0000","last_peered":"2021-02-10T04:43:27.341561+0000","last_clean":"2021-02-10T04:43:27.341561+0000","last_became_active":"2021-02-10T04:33:05.141066+0000","last_became_peered":"2021-02-10T04:33:05.141066+0000","last_unstale":"2021-02-10T04:43:27.341561+0000","last_undegraded":"2021-02-10T04:43:27.341561+0000","last_fullsized":"2021-02-10T04:43:27.341561+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:28:35.010709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:35.010709+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":423,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"958","reported_epoch":"990","state":"active+clean","last_fresh":"2021-02-10T04:43:30.345944+0000","last_change":"2021-02-10T04:37:37.245330+0000","last_active":"2021-02-10T04:43:30.345944+0000","last_peered":"2021-02-10T04:43:30.345944+0000","last_clean":"2021-02-10T04:43:30.345944+0000","last_became_active":"2021-02-10T04:37:37.244859+0000","last_became_peered":"2021-02-10T04:37:37.244859+0000","last_unstale":"2021-02-10T04:43:30.345944+0000","last_undegraded":"2021-02-10T04:43:30.345944+0000","last_fullsized":"2021-02-10T04:43:30.345944+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:05.641577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:05.641577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"989'1619","reported_seq":"4792","reported_epoch":"989","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:43:29.584959+0000","last_change":"2021-02-10T04:43:29.581857+0000","last_active":"2021-02-10T04:43:29.584959+0000","last_peered":"2021-02-10T04:43:29.584959+0000","last_clean":"2021-02-10T04:43:29.584959+0000","last_became_active":"2021-02-10T04:37:45.352022+0000","last_became_peered":"2021-02-10T04:37:45.352022+0000","last_unstale":"2021-02-10T04:43:29.584959+0000","last_undegraded":"2021-02-10T04:43:29.584959+0000","last_fullsized":"2021-02-10T04:43:29.584959+0000","mapping_epoch":646,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":647,"parent":"0.0","parent_split_bits":0,"last_scrub":"687'1095","last_scrub_stamp":"2021-02-10T04:38:27.895743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:38:27.895743+0000","log_size":1619,"ondisk_log_size":1619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"stat_sum":{"num_bytes":89048672,"num_objects":35,"num_object_clones":31,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":1,"num_read":848,"num_read_kb":875347,"num_write":637,"num_write_kb":237595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":35873876,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24e","length":"2"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"3"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"5"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2c9","length":"3"},{"start":"2d1","length":"7"},{"start":"2dd","length":"1"},{"start":"2e0","length":"3"},{"start":"2e4","length":"1"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"339","length":"1"},{"start":"340","length":"1"},{"start":"34f","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":"901","reported_epoch":"990","state":"active+clean","last_fresh":"2021-02-10T04:43:30.345989+0000","last_change":"2021-02-10T04:37:37.233535+0000","last_active":"2021-02-10T04:43:30.345989+0000","last_peered":"2021-02-10T04:43:30.345989+0000","last_clean":"2021-02-10T04:43:30.345989+0000","last_became_active":"2021-02-10T04:37:37.233158+0000","last_became_peered":"2021-02-10T04:37:37.233158+0000","last_unstale":"2021-02-10T04:43:30.345989+0000","last_undegraded":"2021-02-10T04:43:30.345989+0000","last_fullsized":"2021-02-10T04:43:30.345989+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:33.839335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:33.839335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"1020","reported_epoch":"987","state":"active+clean","last_fresh":"2021-02-10T04:43:27.341355+0000","last_change":"2021-02-10T04:33:52.827328+0000","last_active":"2021-02-10T04:43:27.341355+0000","last_peered":"2021-02-10T04:43:27.341355+0000","last_clean":"2021-02-10T04:43:27.341355+0000","last_became_active":"2021-02-10T04:25:51.025636+0000","last_became_peered":"2021-02-10T04:25:51.025636+0000","last_unstale":"2021-02-10T04:43:27.341355+0000","last_undegraded":"2021-02-10T04:43:27.341355+0000","last_fullsized":"2021-02-10T04:43:27.341355+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:52.827257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:33:52.827257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"538'37","reported_seq":"1027","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.413989+0000","last_change":"2021-02-10T04:38:36.138126+0000","last_active":"2021-02-10T04:43:28.413989+0000","last_peered":"2021-02-10T04:43:28.413989+0000","last_clean":"2021-02-10T04:43:28.413989+0000","last_became_active":"2021-02-10T04:25:49.002937+0000","last_became_peered":"2021-02-10T04:25:49.002937+0000","last_unstale":"2021-02-10T04:43:28.413989+0000","last_undegraded":"2021-02-10T04:43:28.413989+0000","last_fullsized":"2021-02-10T04:43:28.413989+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'37","last_scrub_stamp":"2021-02-10T04:38:36.138022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:36.138022+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":72,"num_read_kb":53,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"886","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.415431+0000","last_change":"2021-02-10T04:29:35.711635+0000","last_active":"2021-02-10T04:43:28.415431+0000","last_peered":"2021-02-10T04:43:28.415431+0000","last_clean":"2021-02-10T04:43:28.415431+0000","last_became_active":"2021-02-10T04:29:07.161341+0000","last_became_peered":"2021-02-10T04:29:07.161341+0000","last_unstale":"2021-02-10T04:43:28.415431+0000","last_undegraded":"2021-02-10T04:43:28.415431+0000","last_fullsized":"2021-02-10T04:43:28.415431+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_deep_scrub":"53'2","last_deep_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.711557+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"646","reported_epoch":"987","state":"active+clean","last_fresh":"2021-02-10T04:43:27.340353+0000","last_change":"2021-02-10T04:38:40.414663+0000","last_active":"2021-02-10T04:43:27.340353+0000","last_peered":"2021-02-10T04:43:27.340353+0000","last_clean":"2021-02-10T04:43:27.340353+0000","last_became_active":"2021-02-10T04:37:37.234868+0000","last_became_peered":"2021-02-10T04:37:37.234868+0000","last_unstale":"2021-02-10T04:43:27.340353+0000","last_undegraded":"2021-02-10T04:43:27.340353+0000","last_fullsized":"2021-02-10T04:43:27.340353+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:40.414601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:38:40.414601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"987'1049","reported_seq":"4406","reported_epoch":"987","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:43:27.735001+0000","last_change":"2021-02-10T04:43:27.731931+0000","last_active":"2021-02-10T04:43:27.735001+0000","last_peered":"2021-02-10T04:43:27.735001+0000","last_clean":"2021-02-10T04:43:27.735001+0000","last_became_active":"2021-02-10T04:28:25.146766+0000","last_became_peered":"2021-02-10T04:28:25.146766+0000","last_unstale":"2021-02-10T04:43:27.735001+0000","last_undegraded":"2021-02-10T04:43:27.735001+0000","last_fullsized":"2021-02-10T04:43:27.735001+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'357","last_scrub_stamp":"2021-02-10T04:33:50.582002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:50.582002+0000","log_size":1049,"ondisk_log_size":1049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"stat_sum":{"num_bytes":45385465,"num_objects":17,"num_object_clones":14,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":481,"num_read_kb":453668,"num_write":366,"num_write_kb":150848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24e","length":"2"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"3"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"5"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2c9","length":"3"},{"start":"2d1","length":"7"},{"start":"2dd","length":"1"},{"start":"2e0","length":"3"},{"start":"2e4","length":"1"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"339","length":"1"},{"start":"340","length":"1"},{"start":"34f","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":"971","reported_epoch":"987","state":"active+clean","last_fresh":"2021-02-10T04:43:27.341704+0000","last_change":"2021-02-10T04:29:38.793837+0000","last_active":"2021-02-10T04:43:27.341704+0000","last_peered":"2021-02-10T04:43:27.341704+0000","last_clean":"2021-02-10T04:43:27.341704+0000","last_became_active":"2021-02-10T04:28:49.054638+0000","last_became_peered":"2021-02-10T04:28:49.054638+0000","last_unstale":"2021-02-10T04:43:27.341704+0000","last_undegraded":"2021-02-10T04:43:27.341704+0000","last_fullsized":"2021-02-10T04:43:27.341704+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:38.793760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.793760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"948","reported_epoch":"986","state":"active+clean","last_fresh":"2021-02-10T04:43:26.336464+0000","last_change":"2021-02-10T04:29:40.451948+0000","last_active":"2021-02-10T04:43:26.336464+0000","last_peered":"2021-02-10T04:43:26.336464+0000","last_clean":"2021-02-10T04:43:26.336464+0000","last_became_active":"2021-02-10T04:25:51.032201+0000","last_became_peered":"2021-02-10T04:25:51.032201+0000","last_unstale":"2021-02-10T04:43:26.336464+0000","last_undegraded":"2021-02-10T04:43:26.336464+0000","last_fullsized":"2021-02-10T04:43:26.336464+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:40.451832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:29:40.451832+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"538'28","reported_seq":"1032","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.414973+0000","last_change":"2021-02-10T04:37:38.566112+0000","last_active":"2021-02-10T04:43:28.414973+0000","last_peered":"2021-02-10T04:43:28.414973+0000","last_clean":"2021-02-10T04:43:28.414973+0000","last_became_active":"2021-02-10T04:37:37.247141+0000","last_became_peered":"2021-02-10T04:37:37.247141+0000","last_unstale":"2021-02-10T04:43:28.414973+0000","last_undegraded":"2021-02-10T04:43:28.414973+0000","last_fullsized":"2021-02-10T04:43:28.414973+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'16","last_scrub_stamp":"2021-02-10T04:33:55.288033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:55.288033+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":34,"num_write":28,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"689","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.414079+0000","last_change":"2021-02-10T04:33:05.166788+0000","last_active":"2021-02-10T04:43:28.414079+0000","last_peered":"2021-02-10T04:43:28.414079+0000","last_clean":"2021-02-10T04:43:28.414079+0000","last_became_active":"2021-02-10T04:33:05.166368+0000","last_became_peered":"2021-02-10T04:33:05.166368+0000","last_unstale":"2021-02-10T04:43:28.414079+0000","last_undegraded":"2021-02-10T04:43:28.414079+0000","last_fullsized":"2021-02-10T04:43:28.414079+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:14.204533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:30:14.204533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"952","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.414073+0000","last_change":"2021-02-10T04:29:21.620573+0000","last_active":"2021-02-10T04:43:28.414073+0000","last_peered":"2021-02-10T04:43:28.414073+0000","last_clean":"2021-02-10T04:43:28.414073+0000","last_became_active":"2021-02-10T04:25:49.001782+0000","last_became_peered":"2021-02-10T04:25:49.001782+0000","last_unstale":"2021-02-10T04:43:28.414073+0000","last_undegraded":"2021-02-10T04:43:28.414073+0000","last_fullsized":"2021-02-10T04:43:28.414073+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:21.620482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:21.620482+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"631","reported_epoch":"986","state":"active+clean","last_fresh":"2021-02-10T04:43:26.337025+0000","last_change":"2021-02-10T04:33:05.153205+0000","last_active":"2021-02-10T04:43:26.337025+0000","last_peered":"2021-02-10T04:43:26.337025+0000","last_clean":"2021-02-10T04:43:26.337025+0000","last_became_active":"2021-02-10T04:33:05.152845+0000","last_became_peered":"2021-02-10T04:33:05.152845+0000","last_unstale":"2021-02-10T04:43:26.337025+0000","last_undegraded":"2021-02-10T04:43:26.337025+0000","last_fullsized":"2021-02-10T04:43:26.337025+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.098681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.098681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.5","version":"984'428","reported_seq":"1825","reported_epoch":"989","state":"active+clean+snaptrim_wait","last_fresh":"2021-02-10T04:43:29.825542+0000","last_change":"2021-02-10T04:43:29.825630+0000","last_active":"2021-02-10T04:43:29.825542+0000","last_peered":"2021-02-10T04:43:29.825542+0000","last_clean":"2021-02-10T04:43:29.825542+0000","last_became_active":"2021-02-10T04:37:37.247692+0000","last_became_peered":"2021-02-10T04:37:37.247692+0000","last_unstale":"2021-02-10T04:43:29.825542+0000","last_undegraded":"2021-02-10T04:43:29.825542+0000","last_fullsized":"2021-02-10T04:43:29.825542+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"683'135","last_scrub_stamp":"2021-02-10T04:38:23.183462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:38:23.183462+0000","log_size":428,"ondisk_log_size":428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"stat_sum":{"num_bytes":43156164,"num_objects":18,"num_object_clones":16,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":157,"num_read_kb":136870,"num_write":141,"num_write_kb":47366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":807386,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24e","length":"2"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"3"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"5"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2c9","length":"3"},{"start":"2d1","length":"7"},{"start":"2dd","length":"1"},{"start":"2df","length":"4"},{"start":"2e4","length":"1"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"339","length":"1"},{"start":"33d","length":"1"},{"start":"340","length":"1"},{"start":"34a","length":"1"},{"start":"34f","length":"1"}]},{"pgid":"1.3","version":"0'0","reported_seq":"711","reported_epoch":"987","state":"active+clean","last_fresh":"2021-02-10T04:43:27.340070+0000","last_change":"2021-02-10T04:33:05.141311+0000","last_active":"2021-02-10T04:43:27.340070+0000","last_peered":"2021-02-10T04:43:27.340070+0000","last_clean":"2021-02-10T04:43:27.340070+0000","last_became_active":"2021-02-10T04:33:05.140966+0000","last_became_peered":"2021-02-10T04:33:05.140966+0000","last_unstale":"2021-02-10T04:43:27.340070+0000","last_undegraded":"2021-02-10T04:43:27.340070+0000","last_fullsized":"2021-02-10T04:43:27.340070+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.928906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.928906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"363","reported_epoch":"986","state":"active+clean","last_fresh":"2021-02-10T04:43:26.337198+0000","last_change":"2021-02-10T04:37:37.235036+0000","last_active":"2021-02-10T04:43:26.337198+0000","last_peered":"2021-02-10T04:43:26.337198+0000","last_clean":"2021-02-10T04:43:26.337198+0000","last_became_active":"2021-02-10T04:37:37.234644+0000","last_became_peered":"2021-02-10T04:37:37.234644+0000","last_unstale":"2021-02-10T04:43:26.337198+0000","last_undegraded":"2021-02-10T04:43:26.337198+0000","last_fullsized":"2021-02-10T04:43:26.337198+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.805432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.805432+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"538'27","reported_seq":"998","reported_epoch":"987","state":"active+clean","last_fresh":"2021-02-10T04:43:27.340982+0000","last_change":"2021-02-10T04:33:56.837896+0000","last_active":"2021-02-10T04:43:27.340982+0000","last_peered":"2021-02-10T04:43:27.340982+0000","last_clean":"2021-02-10T04:43:27.340982+0000","last_became_active":"2021-02-10T04:25:49.002471+0000","last_became_peered":"2021-02-10T04:25:49.002471+0000","last_unstale":"2021-02-10T04:43:27.340982+0000","last_undegraded":"2021-02-10T04:43:27.340982+0000","last_fullsized":"2021-02-10T04:43:27.340982+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:33:56.837816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:56.837816+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"955","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.413853+0000","last_change":"2021-02-10T04:29:36.663903+0000","last_active":"2021-02-10T04:43:28.413853+0000","last_peered":"2021-02-10T04:43:28.413853+0000","last_clean":"2021-02-10T04:43:28.413853+0000","last_became_active":"2021-02-10T04:29:07.158075+0000","last_became_peered":"2021-02-10T04:29:07.158075+0000","last_unstale":"2021-02-10T04:43:28.413853+0000","last_undegraded":"2021-02-10T04:43:28.413853+0000","last_fullsized":"2021-02-10T04:43:28.413853+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.663840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.663840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"1088","reported_epoch":"986","state":"active+clean","last_fresh":"2021-02-10T04:43:26.335777+0000","last_change":"2021-02-10T04:42:23.062451+0000","last_active":"2021-02-10T04:43:26.335777+0000","last_peered":"2021-02-10T04:43:26.335777+0000","last_clean":"2021-02-10T04:43:26.335777+0000","last_became_active":"2021-02-10T04:37:37.245931+0000","last_became_peered":"2021-02-10T04:37:37.245931+0000","last_unstale":"2021-02-10T04:43:26.335777+0000","last_undegraded":"2021-02-10T04:43:26.335777+0000","last_fullsized":"2021-02-10T04:43:26.335777+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:34.482906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:34.482906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"951","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.346058+0000","last_change":"2021-02-10T04:29:35.382459+0000","last_active":"2021-02-10T04:43:28.346058+0000","last_peered":"2021-02-10T04:43:28.346058+0000","last_clean":"2021-02-10T04:43:28.346058+0000","last_became_active":"2021-02-10T04:25:49.010614+0000","last_became_peered":"2021-02-10T04:25:49.010614+0000","last_unstale":"2021-02-10T04:43:28.346058+0000","last_undegraded":"2021-02-10T04:43:28.346058+0000","last_fullsized":"2021-02-10T04:43:28.346058+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:35.382341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.382341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"946","reported_epoch":"986","state":"active+clean","last_fresh":"2021-02-10T04:43:26.336546+0000","last_change":"2021-02-10T04:28:38.503398+0000","last_active":"2021-02-10T04:43:26.336546+0000","last_peered":"2021-02-10T04:43:26.336546+0000","last_clean":"2021-02-10T04:43:26.336546+0000","last_became_active":"2021-02-10T04:25:53.037506+0000","last_became_peered":"2021-02-10T04:25:53.037506+0000","last_unstale":"2021-02-10T04:43:26.336546+0000","last_undegraded":"2021-02-10T04:43:26.336546+0000","last_fullsized":"2021-02-10T04:43:26.336546+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:38.503309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:38.503309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"988'472","reported_seq":"3394","reported_epoch":"988","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:43:28.591617+0000","last_change":"2021-02-10T04:43:28.589603+0000","last_active":"2021-02-10T04:43:28.591617+0000","last_peered":"2021-02-10T04:43:28.591617+0000","last_clean":"2021-02-10T04:43:28.591617+0000","last_became_active":"2021-02-10T04:28:25.150109+0000","last_became_peered":"2021-02-10T04:28:25.150109+0000","last_unstale":"2021-02-10T04:43:28.591617+0000","last_undegraded":"2021-02-10T04:43:28.591617+0000","last_fullsized":"2021-02-10T04:43:28.591617+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'36","last_scrub_stamp":"2021-02-10T04:30:13.863430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:13.863430+0000","log_size":472,"ondisk_log_size":472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"stat_sum":{"num_bytes":31347580,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":281,"num_read_kb":297137,"num_write":169,"num_write_kb":56241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24e","length":"2"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"3"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"5"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2c9","length":"3"},{"start":"2d1","length":"7"},{"start":"2e0","length":"5"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"339","length":"1"},{"start":"33d","length":"1"},{"start":"340","length":"1"},{"start":"34a","length":"1"},{"start":"34f","length":"1"}]},{"pgid":"5.6","version":"0'0","reported_seq":"297","reported_epoch":"987","state":"active+clean","last_fresh":"2021-02-10T04:43:27.341487+0000","last_change":"2021-02-10T04:38:39.252779+0000","last_active":"2021-02-10T04:43:27.341487+0000","last_peered":"2021-02-10T04:43:27.341487+0000","last_clean":"2021-02-10T04:43:27.341487+0000","last_became_active":"2021-02-10T04:38:39.252266+0000","last_became_peered":"2021-02-10T04:38:39.252266+0000","last_unstale":"2021-02-10T04:43:27.341487+0000","last_undegraded":"2021-02-10T04:43:27.341487+0000","last_fullsized":"2021-02-10T04:43:27.341487+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.779767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.779767+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"538'33","reported_seq":"719","reported_epoch":"986","state":"active+clean","last_fresh":"2021-02-10T04:43:26.336232+0000","last_change":"2021-02-10T04:33:05.147122+0000","last_active":"2021-02-10T04:43:26.336232+0000","last_peered":"2021-02-10T04:43:26.336232+0000","last_clean":"2021-02-10T04:43:26.336232+0000","last_became_active":"2021-02-10T04:33:05.146695+0000","last_became_peered":"2021-02-10T04:33:05.146695+0000","last_unstale":"2021-02-10T04:43:26.336232+0000","last_undegraded":"2021-02-10T04:43:26.336232+0000","last_fullsized":"2021-02-10T04:43:26.336232+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'29","last_scrub_stamp":"2021-02-10T04:29:25.974280+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.974280+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":38,"num_write":28,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"421","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.345279+0000","last_change":"2021-02-10T04:37:37.258992+0000","last_active":"2021-02-10T04:43:28.345279+0000","last_peered":"2021-02-10T04:43:28.345279+0000","last_clean":"2021-02-10T04:43:28.345279+0000","last_became_active":"2021-02-10T04:37:37.257600+0000","last_became_peered":"2021-02-10T04:37:37.257600+0000","last_unstale":"2021-02-10T04:43:28.345279+0000","last_undegraded":"2021-02-10T04:43:28.345279+0000","last_fullsized":"2021-02-10T04:43:28.345279+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:39.690436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:39.690436+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"744","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.344413+0000","last_change":"2021-02-10T04:38:33.243623+0000","last_active":"2021-02-10T04:43:28.344413+0000","last_peered":"2021-02-10T04:43:28.344413+0000","last_clean":"2021-02-10T04:43:28.344413+0000","last_became_active":"2021-02-10T04:37:37.249591+0000","last_became_peered":"2021-02-10T04:37:37.249591+0000","last_unstale":"2021-02-10T04:43:28.344413+0000","last_undegraded":"2021-02-10T04:43:28.344413+0000","last_fullsized":"2021-02-10T04:43:28.344413+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:38:33.243519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:33.243519+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":358,"num_read_kb":353,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"874","reported_epoch":"987","state":"active+clean","last_fresh":"2021-02-10T04:43:27.341088+0000","last_change":"2021-02-10T04:29:33.747527+0000","last_active":"2021-02-10T04:43:27.341088+0000","last_peered":"2021-02-10T04:43:27.341088+0000","last_clean":"2021-02-10T04:43:27.341088+0000","last_became_active":"2021-02-10T04:29:07.158869+0000","last_became_peered":"2021-02-10T04:29:07.158869+0000","last_unstale":"2021-02-10T04:43:27.341088+0000","last_undegraded":"2021-02-10T04:43:27.341088+0000","last_fullsized":"2021-02-10T04:43:27.341088+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:33.747448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:33.747448+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.4","version":"981'612","reported_seq":"4087","reported_epoch":"989","state":"active+clean+snaptrim_wait","last_fresh":"2021-02-10T04:43:29.827048+0000","last_change":"2021-02-10T04:43:29.827322+0000","last_active":"2021-02-10T04:43:29.827048+0000","last_peered":"2021-02-10T04:43:29.827048+0000","last_clean":"2021-02-10T04:43:29.827048+0000","last_became_active":"2021-02-10T04:28:25.151037+0000","last_became_peered":"2021-02-10T04:28:25.151037+0000","last_unstale":"2021-02-10T04:43:29.827048+0000","last_undegraded":"2021-02-10T04:43:29.827048+0000","last_fullsized":"2021-02-10T04:43:29.827048+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'239","last_scrub_stamp":"2021-02-10T04:33:46.270478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:46.270478+0000","log_size":612,"ondisk_log_size":612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"stat_sum":{"num_bytes":29957984,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":288,"num_read_kb":256965,"num_write":258,"num_write_kb":106887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24e","length":"2"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"3"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"5"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2c9","length":"3"},{"start":"2d1","length":"7"},{"start":"2dd","length":"1"},{"start":"2df","length":"4"},{"start":"2e4","length":"1"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"339","length":"1"},{"start":"33d","length":"1"},{"start":"340","length":"1"},{"start":"34a","length":"1"},{"start":"34f","length":"1"},{"start":"351","length":"1"}]},{"pgid":"1.2","version":"0'0","reported_seq":"965","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.413862+0000","last_change":"2021-02-10T04:29:29.683867+0000","last_active":"2021-02-10T04:43:28.413862+0000","last_peered":"2021-02-10T04:43:28.413862+0000","last_clean":"2021-02-10T04:43:28.413862+0000","last_became_active":"2021-02-10T04:29:07.159607+0000","last_became_peered":"2021-02-10T04:29:07.159607+0000","last_unstale":"2021-02-10T04:43:28.413862+0000","last_undegraded":"2021-02-10T04:43:28.413862+0000","last_fullsized":"2021-02-10T04:43:28.413862+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.683805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.683805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"1040","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.415512+0000","last_change":"2021-02-10T04:42:19.154481+0000","last_active":"2021-02-10T04:43:28.415512+0000","last_peered":"2021-02-10T04:43:28.415512+0000","last_clean":"2021-02-10T04:43:28.415512+0000","last_became_active":"2021-02-10T04:25:51.027624+0000","last_became_peered":"2021-02-10T04:25:51.027624+0000","last_unstale":"2021-02-10T04:43:28.415512+0000","last_undegraded":"2021-02-10T04:43:28.415512+0000","last_fullsized":"2021-02-10T04:43:28.415512+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:43.740853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:28:43.740853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"538'43","reported_seq":"671","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.347272+0000","last_change":"2021-02-10T04:37:38.636607+0000","last_active":"2021-02-10T04:43:28.347272+0000","last_peered":"2021-02-10T04:43:28.347272+0000","last_clean":"2021-02-10T04:43:28.347272+0000","last_became_active":"2021-02-10T04:37:37.244344+0000","last_became_peered":"2021-02-10T04:37:37.244344+0000","last_unstale":"2021-02-10T04:43:28.347272+0000","last_undegraded":"2021-02-10T04:43:28.347272+0000","last_fullsized":"2021-02-10T04:43:28.347272+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'31","last_scrub_stamp":"2021-02-10T04:33:49.423083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:49.423083+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":81,"num_read_kb":57,"num_write":46,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"971","reported_epoch":"986","state":"active+clean","last_fresh":"2021-02-10T04:43:26.335572+0000","last_change":"2021-02-10T04:38:34.289602+0000","last_active":"2021-02-10T04:43:26.335572+0000","last_peered":"2021-02-10T04:43:26.335572+0000","last_clean":"2021-02-10T04:43:26.335572+0000","last_became_active":"2021-02-10T04:28:21.125541+0000","last_became_peered":"2021-02-10T04:28:21.125541+0000","last_unstale":"2021-02-10T04:43:26.335572+0000","last_undegraded":"2021-02-10T04:43:26.335572+0000","last_fullsized":"2021-02-10T04:43:26.335572+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:34.289495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:34.289495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"947","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.347916+0000","last_change":"2021-02-10T04:29:36.396044+0000","last_active":"2021-02-10T04:43:28.347916+0000","last_peered":"2021-02-10T04:43:28.347916+0000","last_clean":"2021-02-10T04:43:28.347916+0000","last_became_active":"2021-02-10T04:25:53.049487+0000","last_became_peered":"2021-02-10T04:25:53.049487+0000","last_unstale":"2021-02-10T04:43:28.347916+0000","last_undegraded":"2021-02-10T04:43:28.347916+0000","last_fullsized":"2021-02-10T04:43:28.347916+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.395943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.395943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"988'2008","reported_seq":"8765","reported_epoch":"988","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:43:28.580113+0000","last_change":"2021-02-10T04:43:28.575475+0000","last_active":"2021-02-10T04:43:28.580113+0000","last_peered":"2021-02-10T04:43:28.580113+0000","last_clean":"2021-02-10T04:43:28.580113+0000","last_became_active":"2021-02-10T04:28:25.147959+0000","last_became_peered":"2021-02-10T04:28:25.147959+0000","last_unstale":"2021-02-10T04:43:28.580113+0000","last_undegraded":"2021-02-10T04:43:28.580113+0000","last_fullsized":"2021-02-10T04:43:28.580113+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'223","last_scrub_stamp":"2021-02-10T04:30:14.478639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:14.478639+0000","log_size":2008,"ondisk_log_size":2008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":15,"stat_sum":{"num_bytes":93990639,"num_objects":40,"num_object_clones":33,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":1,"num_read":1088,"num_read_kb":1063377,"num_write":794,"num_write_kb":304839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24e","length":"2"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"3"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"5"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2c9","length":"3"},{"start":"2d1","length":"7"},{"start":"2e0","length":"5"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"340","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":"898","reported_epoch":"986","state":"active+clean","last_fresh":"2021-02-10T04:43:26.334965+0000","last_change":"2021-02-10T04:37:37.219806+0000","last_active":"2021-02-10T04:43:26.334965+0000","last_peered":"2021-02-10T04:43:26.334965+0000","last_clean":"2021-02-10T04:43:26.334965+0000","last_became_active":"2021-02-10T04:37:37.219422+0000","last_became_peered":"2021-02-10T04:37:37.219422+0000","last_unstale":"2021-02-10T04:43:26.334965+0000","last_undegraded":"2021-02-10T04:43:26.334965+0000","last_fullsized":"2021-02-10T04:43:26.334965+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:39.434210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:39.434210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"58'2","reported_seq":"1169","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.414466+0000","last_change":"2021-02-10T04:38:27.092813+0000","last_active":"2021-02-10T04:43:28.414466+0000","last_peered":"2021-02-10T04:43:28.414466+0000","last_clean":"2021-02-10T04:43:28.414466+0000","last_became_active":"2021-02-10T04:25:51.026251+0000","last_became_peered":"2021-02-10T04:25:51.026251+0000","last_unstale":"2021-02-10T04:43:28.414466+0000","last_undegraded":"2021-02-10T04:43:28.414466+0000","last_fullsized":"2021-02-10T04:43:28.414466+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:38:27.092697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:38:27.092697+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"538'44","reported_seq":"721","reported_epoch":"986","state":"active+clean","last_fresh":"2021-02-10T04:43:26.335827+0000","last_change":"2021-02-10T04:33:05.142280+0000","last_active":"2021-02-10T04:43:26.335827+0000","last_peered":"2021-02-10T04:43:26.335827+0000","last_clean":"2021-02-10T04:43:26.335827+0000","last_became_active":"2021-02-10T04:33:05.141901+0000","last_became_peered":"2021-02-10T04:33:05.141901+0000","last_unstale":"2021-02-10T04:43:26.335827+0000","last_undegraded":"2021-02-10T04:43:26.335827+0000","last_fullsized":"2021-02-10T04:43:26.335827+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'34","last_scrub_stamp":"2021-02-10T04:28:49.959162+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:28:49.959162+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":60,"num_read_kb":42,"num_write":32,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"715","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.414612+0000","last_change":"2021-02-10T04:37:37.247740+0000","last_active":"2021-02-10T04:43:28.414612+0000","last_peered":"2021-02-10T04:43:28.414612+0000","last_clean":"2021-02-10T04:43:28.414612+0000","last_became_active":"2021-02-10T04:37:37.247405+0000","last_became_peered":"2021-02-10T04:37:37.247405+0000","last_unstale":"2021-02-10T04:43:28.414612+0000","last_undegraded":"2021-02-10T04:43:28.414612+0000","last_fullsized":"2021-02-10T04:43:28.414612+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:34.053323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:34.053323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"962","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.413686+0000","last_change":"2021-02-10T04:38:26.124897+0000","last_active":"2021-02-10T04:43:28.413686+0000","last_peered":"2021-02-10T04:43:28.413686+0000","last_clean":"2021-02-10T04:43:28.413686+0000","last_became_active":"2021-02-10T04:37:37.254482+0000","last_became_peered":"2021-02-10T04:37:37.254482+0000","last_unstale":"2021-02-10T04:43:28.413686+0000","last_undegraded":"2021-02-10T04:43:28.413686+0000","last_fullsized":"2021-02-10T04:43:28.413686+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:26.124805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:38:26.124805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"293'77","reported_seq":"2174","reported_epoch":"990","state":"active+clean+scrubbing","last_fresh":"2021-02-10T04:43:30.345370+0000","last_change":"2021-02-10T04:38:41.130429+0000","last_active":"2021-02-10T04:43:30.345370+0000","last_peered":"2021-02-10T04:43:30.345370+0000","last_clean":"2021-02-10T04:43:30.345370+0000","last_became_active":"2021-02-10T04:28:25.154434+0000","last_became_peered":"2021-02-10T04:28:25.154434+0000","last_unstale":"2021-02-10T04:43:30.345370+0000","last_undegraded":"2021-02-10T04:43:30.345370+0000","last_fullsized":"2021-02-10T04:43:30.345370+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":138,"stat_sum":{"num_bytes":7546343,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":74349,"num_write":37,"num_write_kb":13263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"971","reported_epoch":"987","state":"active+clean","last_fresh":"2021-02-10T04:43:27.340447+0000","last_change":"2021-02-10T04:29:39.829039+0000","last_active":"2021-02-10T04:43:27.340447+0000","last_peered":"2021-02-10T04:43:27.340447+0000","last_clean":"2021-02-10T04:43:27.340447+0000","last_became_active":"2021-02-10T04:28:49.053127+0000","last_became_peered":"2021-02-10T04:28:49.053127+0000","last_unstale":"2021-02-10T04:43:27.340447+0000","last_undegraded":"2021-02-10T04:43:27.340447+0000","last_fullsized":"2021-02-10T04:43:27.340447+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:39.828975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:39.828975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"371","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.414245+0000","last_change":"2021-02-10T04:38:39.247167+0000","last_active":"2021-02-10T04:43:28.414245+0000","last_peered":"2021-02-10T04:43:28.414245+0000","last_clean":"2021-02-10T04:43:28.414245+0000","last_became_active":"2021-02-10T04:38:39.246766+0000","last_became_peered":"2021-02-10T04:38:39.246766+0000","last_unstale":"2021-02-10T04:43:28.414245+0000","last_undegraded":"2021-02-10T04:43:28.414245+0000","last_fullsized":"2021-02-10T04:43:28.414245+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:20.757194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:20.757194+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"538'33","reported_seq":"1062","reported_epoch":"990","state":"active+clean","last_fresh":"2021-02-10T04:43:30.345527+0000","last_change":"2021-02-10T04:39:02.586279+0000","last_active":"2021-02-10T04:43:30.345527+0000","last_peered":"2021-02-10T04:43:30.345527+0000","last_clean":"2021-02-10T04:43:30.345527+0000","last_became_active":"2021-02-10T04:39:02.560739+0000","last_became_peered":"2021-02-10T04:39:02.560739+0000","last_unstale":"2021-02-10T04:43:30.345527+0000","last_undegraded":"2021-02-10T04:43:30.345527+0000","last_fullsized":"2021-02-10T04:43:30.345527+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'27","last_scrub_stamp":"2021-02-10T04:28:47.476669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:47.476669+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":67,"num_read_kb":50,"num_write":35,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"1084","reported_epoch":"990","state":"active+clean","last_fresh":"2021-02-10T04:43:30.347033+0000","last_change":"2021-02-10T04:37:37.351533+0000","last_active":"2021-02-10T04:43:30.347033+0000","last_peered":"2021-02-10T04:43:30.347033+0000","last_clean":"2021-02-10T04:43:30.347033+0000","last_became_active":"2021-02-10T04:37:37.238701+0000","last_became_peered":"2021-02-10T04:37:37.238701+0000","last_unstale":"2021-02-10T04:43:30.347033+0000","last_undegraded":"2021-02-10T04:43:30.347033+0000","last_fullsized":"2021-02-10T04:43:30.347033+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:33:49.760989+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:33:49.760989+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"947","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.346663+0000","last_change":"2021-02-10T04:29:37.364781+0000","last_active":"2021-02-10T04:43:28.346663+0000","last_peered":"2021-02-10T04:43:28.346663+0000","last_clean":"2021-02-10T04:43:28.346663+0000","last_became_active":"2021-02-10T04:25:53.047736+0000","last_became_peered":"2021-02-10T04:25:53.047736+0000","last_unstale":"2021-02-10T04:43:28.346663+0000","last_undegraded":"2021-02-10T04:43:28.346663+0000","last_fullsized":"2021-02-10T04:43:28.346663+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:37.364697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:37.364697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"988'1500","reported_seq":"5552","reported_epoch":"989","state":"active+clean+snaptrim_wait","last_fresh":"2021-02-10T04:43:29.615727+0000","last_change":"2021-02-10T04:43:29.615825+0000","last_active":"2021-02-10T04:43:29.615727+0000","last_peered":"2021-02-10T04:43:29.615727+0000","last_clean":"2021-02-10T04:43:29.615727+0000","last_became_active":"2021-02-10T04:37:48.215401+0000","last_became_peered":"2021-02-10T04:37:48.215401+0000","last_unstale":"2021-02-10T04:43:29.615727+0000","last_undegraded":"2021-02-10T04:43:29.615727+0000","last_fullsized":"2021-02-10T04:43:29.615727+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":1500,"ondisk_log_size":1500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"stat_sum":{"num_bytes":94602866,"num_objects":34,"num_object_clones":31,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":1,"num_read":716,"num_read_kb":703869,"num_write":572,"num_write_kb":222710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":42623785,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24e","length":"2"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"3"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"5"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2c9","length":"3"},{"start":"2d1","length":"7"},{"start":"2dd","length":"1"},{"start":"2e0","length":"5"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"33d","length":"1"},{"start":"340","length":"1"},{"start":"34f","length":"1"}]},{"pgid":"4.16","version":"538'34","reported_seq":"1021","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.415160+0000","last_change":"2021-02-10T04:29:25.645462+0000","last_active":"2021-02-10T04:43:28.415160+0000","last_peered":"2021-02-10T04:43:28.415160+0000","last_clean":"2021-02-10T04:43:28.415160+0000","last_became_active":"2021-02-10T04:25:49.008942+0000","last_became_peered":"2021-02-10T04:25:49.008942+0000","last_unstale":"2021-02-10T04:43:28.415160+0000","last_undegraded":"2021-02-10T04:43:28.415160+0000","last_fullsized":"2021-02-10T04:43:28.415160+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'28","last_scrub_stamp":"2021-02-10T04:29:25.645396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.645396+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":71,"num_read_kb":49,"num_write":35,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"380","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.413937+0000","last_change":"2021-02-10T04:38:38.163311+0000","last_active":"2021-02-10T04:43:28.413937+0000","last_peered":"2021-02-10T04:43:28.413937+0000","last_clean":"2021-02-10T04:43:28.413937+0000","last_became_active":"2021-02-10T04:37:37.234649+0000","last_became_peered":"2021-02-10T04:37:37.234649+0000","last_unstale":"2021-02-10T04:43:28.413937+0000","last_undegraded":"2021-02-10T04:43:28.413937+0000","last_fullsized":"2021-02-10T04:43:28.413937+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:38:38.163225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:38.163225+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":247,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"949","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.348747+0000","last_change":"2021-02-10T04:28:33.321856+0000","last_active":"2021-02-10T04:43:28.348747+0000","last_peered":"2021-02-10T04:43:28.348747+0000","last_clean":"2021-02-10T04:43:28.348747+0000","last_became_active":"2021-02-10T04:25:51.030651+0000","last_became_peered":"2021-02-10T04:25:51.030651+0000","last_unstale":"2021-02-10T04:43:28.348747+0000","last_undegraded":"2021-02-10T04:43:28.348747+0000","last_fullsized":"2021-02-10T04:43:28.348747+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:33.321784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:33.321784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"363","reported_epoch":"990","state":"active+clean","last_fresh":"2021-02-10T04:43:30.346279+0000","last_change":"2021-02-10T04:37:37.238450+0000","last_active":"2021-02-10T04:43:30.346279+0000","last_peered":"2021-02-10T04:43:30.346279+0000","last_clean":"2021-02-10T04:43:30.346279+0000","last_became_active":"2021-02-10T04:37:37.238049+0000","last_became_peered":"2021-02-10T04:37:37.238049+0000","last_unstale":"2021-02-10T04:43:30.346279+0000","last_undegraded":"2021-02-10T04:43:30.346279+0000","last_fullsized":"2021-02-10T04:43:30.346279+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.776340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.776340+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"538'19","reported_seq":"1033","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.348583+0000","last_change":"2021-02-10T04:38:26.845289+0000","last_active":"2021-02-10T04:43:28.348583+0000","last_peered":"2021-02-10T04:43:28.348583+0000","last_clean":"2021-02-10T04:43:28.348583+0000","last_became_active":"2021-02-10T04:37:37.247396+0000","last_became_peered":"2021-02-10T04:37:37.247396+0000","last_unstale":"2021-02-10T04:43:28.348583+0000","last_undegraded":"2021-02-10T04:43:28.348583+0000","last_fullsized":"2021-02-10T04:43:28.348583+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'19","last_scrub_stamp":"2021-02-10T04:38:26.845195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:26.845195+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":19,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"965","reported_epoch":"987","state":"active+clean","last_fresh":"2021-02-10T04:43:27.340571+0000","last_change":"2021-02-10T04:38:36.295733+0000","last_active":"2021-02-10T04:43:27.340571+0000","last_peered":"2021-02-10T04:43:27.340571+0000","last_clean":"2021-02-10T04:43:27.340571+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:43:27.340571+0000","last_undegraded":"2021-02-10T04:43:27.340571+0000","last_fullsized":"2021-02-10T04:43:27.340571+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:38:36.295599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:36.295599+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"948","reported_epoch":"986","state":"active+clean","last_fresh":"2021-02-10T04:43:26.336254+0000","last_change":"2021-02-10T04:29:18.429730+0000","last_active":"2021-02-10T04:43:26.336254+0000","last_peered":"2021-02-10T04:43:26.336254+0000","last_clean":"2021-02-10T04:43:26.336254+0000","last_became_active":"2021-02-10T04:25:51.031864+0000","last_became_peered":"2021-02-10T04:25:51.031864+0000","last_unstale":"2021-02-10T04:43:26.336254+0000","last_undegraded":"2021-02-10T04:43:26.336254+0000","last_fullsized":"2021-02-10T04:43:26.336254+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:18.429649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:18.429649+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"874","reported_epoch":"987","state":"active+clean","last_fresh":"2021-02-10T04:43:27.341325+0000","last_change":"2021-02-10T04:29:36.866949+0000","last_active":"2021-02-10T04:43:27.341325+0000","last_peered":"2021-02-10T04:43:27.341325+0000","last_clean":"2021-02-10T04:43:27.341325+0000","last_became_active":"2021-02-10T04:29:07.147249+0000","last_became_peered":"2021-02-10T04:29:07.147249+0000","last_unstale":"2021-02-10T04:43:27.341325+0000","last_undegraded":"2021-02-10T04:43:27.341325+0000","last_fullsized":"2021-02-10T04:43:27.341325+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.866833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.866833+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"538'36","reported_seq":"1157","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.347128+0000","last_change":"2021-02-10T04:37:38.541093+0000","last_active":"2021-02-10T04:43:28.347128+0000","last_peered":"2021-02-10T04:43:28.347128+0000","last_clean":"2021-02-10T04:43:28.347128+0000","last_became_active":"2021-02-10T04:37:37.244921+0000","last_became_peered":"2021-02-10T04:37:37.244921+0000","last_unstale":"2021-02-10T04:43:28.347128+0000","last_undegraded":"2021-02-10T04:43:28.347128+0000","last_fullsized":"2021-02-10T04:43:28.347128+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:33:46.785905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:46.785905+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":36,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"84'7","reported_seq":"977","reported_epoch":"986","state":"active+clean","last_fresh":"2021-02-10T04:43:26.335501+0000","last_change":"2021-02-10T04:30:12.538670+0000","last_active":"2021-02-10T04:43:26.335501+0000","last_peered":"2021-02-10T04:43:26.335501+0000","last_clean":"2021-02-10T04:43:26.335501+0000","last_became_active":"2021-02-10T04:25:36.334366+0000","last_became_peered":"2021-02-10T04:25:36.334366+0000","last_unstale":"2021-02-10T04:43:26.335501+0000","last_undegraded":"2021-02-10T04:43:26.335501+0000","last_fullsized":"2021-02-10T04:43:26.335501+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'7","last_scrub_stamp":"2021-02-10T04:30:12.538549+0000","last_deep_scrub":"84'7","last_deep_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_clean_scrub_stamp":"2021-02-10T04:30:12.538549+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"58'2","reported_seq":"1162","reported_epoch":"987","state":"active+clean","last_fresh":"2021-02-10T04:43:27.341315+0000","last_change":"2021-02-10T04:28:47.496534+0000","last_active":"2021-02-10T04:43:27.341315+0000","last_peered":"2021-02-10T04:43:27.341315+0000","last_clean":"2021-02-10T04:43:27.341315+0000","last_became_active":"2021-02-10T04:25:51.032328+0000","last_became_peered":"2021-02-10T04:25:51.032328+0000","last_unstale":"2021-02-10T04:43:27.341315+0000","last_undegraded":"2021-02-10T04:43:27.341315+0000","last_fullsized":"2021-02-10T04:43:27.341315+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:28:47.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:47.496468+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"953","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.414913+0000","last_change":"2021-02-10T04:38:39.191979+0000","last_active":"2021-02-10T04:43:28.414913+0000","last_peered":"2021-02-10T04:43:28.414913+0000","last_clean":"2021-02-10T04:43:28.414913+0000","last_became_active":"2021-02-10T04:25:53.045177+0000","last_became_peered":"2021-02-10T04:25:53.045177+0000","last_unstale":"2021-02-10T04:43:28.414913+0000","last_undegraded":"2021-02-10T04:43:28.414913+0000","last_fullsized":"2021-02-10T04:43:28.414913+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:39.191910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_clean_scrub_stamp":"2021-02-10T04:38:39.191910+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"538'25","reported_seq":"998","reported_epoch":"986","state":"active+clean","last_fresh":"2021-02-10T04:43:26.335805+0000","last_change":"2021-02-10T04:28:50.493712+0000","last_active":"2021-02-10T04:43:26.335805+0000","last_peered":"2021-02-10T04:43:26.335805+0000","last_clean":"2021-02-10T04:43:26.335805+0000","last_became_active":"2021-02-10T04:25:49.010890+0000","last_became_peered":"2021-02-10T04:25:49.010890+0000","last_unstale":"2021-02-10T04:43:26.335805+0000","last_undegraded":"2021-02-10T04:43:26.335805+0000","last_fullsized":"2021-02-10T04:43:26.335805+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2021-02-10T04:28:50.493641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:50.493641+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":35,"num_write":26,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"964","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.414257+0000","last_change":"2021-02-10T04:29:19.564845+0000","last_active":"2021-02-10T04:43:28.414257+0000","last_peered":"2021-02-10T04:43:28.414257+0000","last_clean":"2021-02-10T04:43:28.414257+0000","last_became_active":"2021-02-10T04:25:34.321096+0000","last_became_peered":"2021-02-10T04:25:34.321096+0000","last_unstale":"2021-02-10T04:43:28.414257+0000","last_undegraded":"2021-02-10T04:43:28.414257+0000","last_fullsized":"2021-02-10T04:43:28.414257+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:19.564725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:29:19.564725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"639'3","reported_seq":"1176","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.347224+0000","last_change":"2021-02-10T04:37:37.922249+0000","last_active":"2021-02-10T04:43:28.347224+0000","last_peered":"2021-02-10T04:43:28.347224+0000","last_clean":"2021-02-10T04:43:28.347224+0000","last_became_active":"2021-02-10T04:37:37.245428+0000","last_became_peered":"2021-02-10T04:37:37.245428+0000","last_unstale":"2021-02-10T04:43:28.347224+0000","last_undegraded":"2021-02-10T04:43:28.347224+0000","last_fullsized":"2021-02-10T04:43:28.347224+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:29:38.343834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.343834+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"958","reported_epoch":"990","state":"active+clean","last_fresh":"2021-02-10T04:43:30.347277+0000","last_change":"2021-02-10T04:37:37.238692+0000","last_active":"2021-02-10T04:43:30.347277+0000","last_peered":"2021-02-10T04:43:30.347277+0000","last_clean":"2021-02-10T04:43:30.347277+0000","last_became_active":"2021-02-10T04:37:37.238360+0000","last_became_peered":"2021-02-10T04:37:37.238360+0000","last_unstale":"2021-02-10T04:43:30.347277+0000","last_undegraded":"2021-02-10T04:43:30.347277+0000","last_fullsized":"2021-02-10T04:43:30.347277+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.934078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.934078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"538'21","reported_seq":"713","reported_epoch":"987","state":"active+clean","last_fresh":"2021-02-10T04:43:27.341501+0000","last_change":"2021-02-10T04:37:38.590027+0000","last_active":"2021-02-10T04:43:27.341501+0000","last_peered":"2021-02-10T04:43:27.341501+0000","last_clean":"2021-02-10T04:43:27.341501+0000","last_became_active":"2021-02-10T04:37:37.249007+0000","last_became_peered":"2021-02-10T04:37:37.249007+0000","last_unstale":"2021-02-10T04:43:27.341501+0000","last_undegraded":"2021-02-10T04:43:27.341501+0000","last_fullsized":"2021-02-10T04:43:27.341501+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:30:11.170034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.170034+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"963","reported_epoch":"987","state":"active+clean","last_fresh":"2021-02-10T04:43:27.340587+0000","last_change":"2021-02-10T04:38:26.127398+0000","last_active":"2021-02-10T04:43:27.340587+0000","last_peered":"2021-02-10T04:43:27.340587+0000","last_clean":"2021-02-10T04:43:27.340587+0000","last_became_active":"2021-02-10T04:25:34.330070+0000","last_became_peered":"2021-02-10T04:25:34.330070+0000","last_unstale":"2021-02-10T04:43:27.340587+0000","last_undegraded":"2021-02-10T04:43:27.340587+0000","last_fullsized":"2021-02-10T04:43:27.340587+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:26.127288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:26.127288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"949","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.346495+0000","last_change":"2021-02-10T04:28:36.311808+0000","last_active":"2021-02-10T04:43:28.346495+0000","last_peered":"2021-02-10T04:43:28.346495+0000","last_clean":"2021-02-10T04:43:28.346495+0000","last_became_active":"2021-02-10T04:25:51.029155+0000","last_became_peered":"2021-02-10T04:25:51.029155+0000","last_unstale":"2021-02-10T04:43:28.346495+0000","last_undegraded":"2021-02-10T04:43:28.346495+0000","last_fullsized":"2021-02-10T04:43:28.346495+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:36.311725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:28:36.311725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"953","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.413687+0000","last_change":"2021-02-10T04:33:49.338303+0000","last_active":"2021-02-10T04:43:28.413687+0000","last_peered":"2021-02-10T04:43:28.413687+0000","last_clean":"2021-02-10T04:43:28.413687+0000","last_became_active":"2021-02-10T04:25:53.039322+0000","last_became_peered":"2021-02-10T04:25:53.039322+0000","last_unstale":"2021-02-10T04:43:28.413687+0000","last_undegraded":"2021-02-10T04:43:28.413687+0000","last_fullsized":"2021-02-10T04:43:28.413687+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:49.338203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:33:49.338203+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"538'23","reported_seq":"992","reported_epoch":"990","state":"active+clean","last_fresh":"2021-02-10T04:43:30.346095+0000","last_change":"2021-02-10T04:38:32.221752+0000","last_active":"2021-02-10T04:43:30.346095+0000","last_peered":"2021-02-10T04:43:30.346095+0000","last_clean":"2021-02-10T04:43:30.346095+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:43:30.346095+0000","last_undegraded":"2021-02-10T04:43:30.346095+0000","last_fullsized":"2021-02-10T04:43:30.346095+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'23","last_scrub_stamp":"2021-02-10T04:38:32.221656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:32.221656+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"647","reported_epoch":"986","state":"active+clean","last_fresh":"2021-02-10T04:43:26.335897+0000","last_change":"2021-02-10T04:38:36.309455+0000","last_active":"2021-02-10T04:43:26.335897+0000","last_peered":"2021-02-10T04:43:26.335897+0000","last_clean":"2021-02-10T04:43:26.335897+0000","last_became_active":"2021-02-10T04:33:05.147956+0000","last_became_peered":"2021-02-10T04:33:05.147956+0000","last_unstale":"2021-02-10T04:43:26.335897+0000","last_undegraded":"2021-02-10T04:43:26.335897+0000","last_fullsized":"2021-02-10T04:43:26.335897+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:38:36.309362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:36.309362+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"631","reported_epoch":"986","state":"active+clean","last_fresh":"2021-02-10T04:43:26.335749+0000","last_change":"2021-02-10T04:33:05.150906+0000","last_active":"2021-02-10T04:43:26.335749+0000","last_peered":"2021-02-10T04:43:26.335749+0000","last_clean":"2021-02-10T04:43:26.335749+0000","last_became_active":"2021-02-10T04:33:05.150529+0000","last_became_peered":"2021-02-10T04:33:05.150529+0000","last_unstale":"2021-02-10T04:43:26.335749+0000","last_undegraded":"2021-02-10T04:43:26.335749+0000","last_fullsized":"2021-02-10T04:43:26.335749+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:33.026988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:29:33.026988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"947","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.345082+0000","last_change":"2021-02-10T04:29:24.421782+0000","last_active":"2021-02-10T04:43:28.345082+0000","last_peered":"2021-02-10T04:43:28.345082+0000","last_clean":"2021-02-10T04:43:28.345082+0000","last_became_active":"2021-02-10T04:25:53.043277+0000","last_became_peered":"2021-02-10T04:25:53.043277+0000","last_unstale":"2021-02-10T04:43:28.345082+0000","last_undegraded":"2021-02-10T04:43:28.345082+0000","last_fullsized":"2021-02-10T04:43:28.345082+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.421710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.421710+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"538'16","reported_seq":"1110","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.343547+0000","last_change":"2021-02-10T04:38:22.919477+0000","last_active":"2021-02-10T04:43:28.343547+0000","last_peered":"2021-02-10T04:43:28.343547+0000","last_clean":"2021-02-10T04:43:28.343547+0000","last_became_active":"2021-02-10T04:26:39.005616+0000","last_became_peered":"2021-02-10T04:26:39.005616+0000","last_unstale":"2021-02-10T04:43:28.343547+0000","last_undegraded":"2021-02-10T04:43:28.343547+0000","last_fullsized":"2021-02-10T04:43:28.343547+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'16","last_scrub_stamp":"2021-02-10T04:38:22.919377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:22.919377+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":36,"num_read_kb":18,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"966","reported_epoch":"986","state":"active+clean","last_fresh":"2021-02-10T04:43:26.336448+0000","last_change":"2021-02-10T04:37:37.230226+0000","last_active":"2021-02-10T04:43:26.336448+0000","last_peered":"2021-02-10T04:43:26.336448+0000","last_clean":"2021-02-10T04:43:26.336448+0000","last_became_active":"2021-02-10T04:37:37.229852+0000","last_became_peered":"2021-02-10T04:37:37.229852+0000","last_unstale":"2021-02-10T04:43:26.336448+0000","last_undegraded":"2021-02-10T04:43:26.336448+0000","last_fullsized":"2021-02-10T04:43:26.336448+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:49.522124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:49.522124+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"949","reported_epoch":"987","state":"active+clean","last_fresh":"2021-02-10T04:43:27.340484+0000","last_change":"2021-02-10T04:29:23.645783+0000","last_active":"2021-02-10T04:43:27.340484+0000","last_peered":"2021-02-10T04:43:27.340484+0000","last_clean":"2021-02-10T04:43:27.340484+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:43:27.340484+0000","last_undegraded":"2021-02-10T04:43:27.340484+0000","last_fullsized":"2021-02-10T04:43:27.340484+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.645694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.645694+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"366","reported_epoch":"990","state":"active+clean","last_fresh":"2021-02-10T04:43:30.345443+0000","last_change":"2021-02-10T04:37:37.244700+0000","last_active":"2021-02-10T04:43:30.345443+0000","last_peered":"2021-02-10T04:43:30.345443+0000","last_clean":"2021-02-10T04:43:30.345443+0000","last_became_active":"2021-02-10T04:37:37.244082+0000","last_became_peered":"2021-02-10T04:37:37.244082+0000","last_unstale":"2021-02-10T04:43:30.345443+0000","last_undegraded":"2021-02-10T04:43:30.345443+0000","last_fullsized":"2021-02-10T04:43:30.345443+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:31.824925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.824925+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.11","version":"538'29","reported_seq":"1004","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.345484+0000","last_change":"2021-02-10T04:28:31.382776+0000","last_active":"2021-02-10T04:43:28.345484+0000","last_peered":"2021-02-10T04:43:28.345484+0000","last_clean":"2021-02-10T04:43:28.345484+0000","last_became_active":"2021-02-10T04:25:49.007049+0000","last_became_peered":"2021-02-10T04:25:49.007049+0000","last_unstale":"2021-02-10T04:43:28.345484+0000","last_undegraded":"2021-02-10T04:43:28.345484+0000","last_fullsized":"2021-02-10T04:43:28.345484+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2021-02-10T04:28:31.382671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:31.382671+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":29,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"1103","reported_epoch":"987","state":"active+clean","last_fresh":"2021-02-10T04:43:27.342228+0000","last_change":"2021-02-10T04:38:31.166652+0000","last_active":"2021-02-10T04:43:27.342228+0000","last_peered":"2021-02-10T04:43:27.342228+0000","last_clean":"2021-02-10T04:43:27.342228+0000","last_became_active":"2021-02-10T04:37:37.228840+0000","last_became_peered":"2021-02-10T04:37:37.228840+0000","last_unstale":"2021-02-10T04:43:27.342228+0000","last_undegraded":"2021-02-10T04:43:27.342228+0000","last_fullsized":"2021-02-10T04:43:27.342228+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:38:31.166586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:31.166586+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"298","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.343726+0000","last_change":"2021-02-10T04:38:39.246255+0000","last_active":"2021-02-10T04:43:28.343726+0000","last_peered":"2021-02-10T04:43:28.343726+0000","last_clean":"2021-02-10T04:43:28.343726+0000","last_became_active":"2021-02-10T04:38:39.246046+0000","last_became_peered":"2021-02-10T04:38:39.246046+0000","last_unstale":"2021-02-10T04:43:28.343726+0000","last_undegraded":"2021-02-10T04:43:28.343726+0000","last_fullsized":"2021-02-10T04:43:28.343726+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:47.473024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:47.473024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"947","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.346858+0000","last_change":"2021-02-10T04:33:50.809533+0000","last_active":"2021-02-10T04:43:28.346858+0000","last_peered":"2021-02-10T04:43:28.346858+0000","last_clean":"2021-02-10T04:43:28.346858+0000","last_became_active":"2021-02-10T04:25:53.150845+0000","last_became_peered":"2021-02-10T04:25:53.150845+0000","last_unstale":"2021-02-10T04:43:28.346858+0000","last_undegraded":"2021-02-10T04:43:28.346858+0000","last_fullsized":"2021-02-10T04:43:28.346858+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:50.809468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_clean_scrub_stamp":"2021-02-10T04:33:50.809468+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"538'32","reported_seq":"1014","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.415353+0000","last_change":"2021-02-10T04:29:38.663351+0000","last_active":"2021-02-10T04:43:28.415353+0000","last_peered":"2021-02-10T04:43:28.415353+0000","last_clean":"2021-02-10T04:43:28.415353+0000","last_became_active":"2021-02-10T04:25:49.009177+0000","last_became_peered":"2021-02-10T04:25:49.009177+0000","last_unstale":"2021-02-10T04:43:28.415353+0000","last_undegraded":"2021-02-10T04:43:28.415353+0000","last_fullsized":"2021-02-10T04:43:28.415353+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2021-02-10T04:29:38.663272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.663272+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":60,"num_read_kb":42,"num_write":32,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"966","reported_epoch":"986","state":"active+clean","last_fresh":"2021-02-10T04:43:26.333631+0000","last_change":"2021-02-10T04:28:37.515037+0000","last_active":"2021-02-10T04:43:26.333631+0000","last_peered":"2021-02-10T04:43:26.333631+0000","last_clean":"2021-02-10T04:43:26.333631+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:43:26.333631+0000","last_undegraded":"2021-02-10T04:43:26.333631+0000","last_fullsized":"2021-02-10T04:43:26.333631+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:37.514943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:37.514943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"58'2","reported_seq":"1163","reported_epoch":"987","state":"active+clean","last_fresh":"2021-02-10T04:43:27.898223+0000","last_change":"2021-02-10T04:29:31.726022+0000","last_active":"2021-02-10T04:43:27.898223+0000","last_peered":"2021-02-10T04:43:27.898223+0000","last_clean":"2021-02-10T04:43:27.898223+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:43:27.898223+0000","last_undegraded":"2021-02-10T04:43:27.898223+0000","last_fullsized":"2021-02-10T04:43:27.898223+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:29:31.725932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.725932+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"891","reported_epoch":"986","state":"active+clean","last_fresh":"2021-02-10T04:43:26.336944+0000","last_change":"2021-02-10T04:38:39.252110+0000","last_active":"2021-02-10T04:43:26.336944+0000","last_peered":"2021-02-10T04:43:26.336944+0000","last_clean":"2021-02-10T04:43:26.336944+0000","last_became_active":"2021-02-10T04:38:39.251336+0000","last_became_peered":"2021-02-10T04:38:39.251336+0000","last_unstale":"2021-02-10T04:43:26.336944+0000","last_undegraded":"2021-02-10T04:43:26.336944+0000","last_fullsized":"2021-02-10T04:43:26.336944+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:26.437591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.437591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"538'43","reported_seq":"1026","reported_epoch":"987","state":"active+clean","last_fresh":"2021-02-10T04:43:27.342063+0000","last_change":"2021-02-10T04:30:13.526264+0000","last_active":"2021-02-10T04:43:27.342063+0000","last_peered":"2021-02-10T04:43:27.342063+0000","last_clean":"2021-02-10T04:43:27.342063+0000","last_became_active":"2021-02-10T04:25:49.015760+0000","last_became_peered":"2021-02-10T04:25:49.015760+0000","last_unstale":"2021-02-10T04:43:27.342063+0000","last_undegraded":"2021-02-10T04:43:27.342063+0000","last_fullsized":"2021-02-10T04:43:27.342063+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'35","last_scrub_stamp":"2021-02-10T04:30:13.526170+0000","last_deep_scrub":"76'35","last_deep_scrub_stamp":"2021-02-10T04:26:58.198641+0000","last_clean_scrub_stamp":"2021-02-10T04:30:13.526170+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":62,"num_read_kb":42,"num_write":32,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"958","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.345205+0000","last_change":"2021-02-10T04:30:11.485059+0000","last_active":"2021-02-10T04:43:28.345205+0000","last_peered":"2021-02-10T04:43:28.345205+0000","last_clean":"2021-02-10T04:43:28.345205+0000","last_became_active":"2021-02-10T04:25:34.324981+0000","last_became_peered":"2021-02-10T04:25:34.324981+0000","last_unstale":"2021-02-10T04:43:28.345205+0000","last_undegraded":"2021-02-10T04:43:28.345205+0000","last_fullsized":"2021-02-10T04:43:28.345205+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:11.485004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.485004+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"959","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.415404+0000","last_change":"2021-02-10T04:37:37.248748+0000","last_active":"2021-02-10T04:43:28.415404+0000","last_peered":"2021-02-10T04:43:28.415404+0000","last_clean":"2021-02-10T04:43:28.415404+0000","last_became_active":"2021-02-10T04:37:37.246998+0000","last_became_peered":"2021-02-10T04:37:37.246998+0000","last_unstale":"2021-02-10T04:43:28.415404+0000","last_undegraded":"2021-02-10T04:43:28.415404+0000","last_fullsized":"2021-02-10T04:43:28.415404+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.589159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.589159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"631","reported_epoch":"986","state":"active+clean","last_fresh":"2021-02-10T04:43:26.336237+0000","last_change":"2021-02-10T04:33:05.142344+0000","last_active":"2021-02-10T04:43:26.336237+0000","last_peered":"2021-02-10T04:43:26.336237+0000","last_clean":"2021-02-10T04:43:26.336237+0000","last_became_active":"2021-02-10T04:33:05.141911+0000","last_became_peered":"2021-02-10T04:33:05.141911+0000","last_unstale":"2021-02-10T04:43:26.336237+0000","last_undegraded":"2021-02-10T04:43:26.336237+0000","last_fullsized":"2021-02-10T04:43:26.336237+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:25.075937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.075937+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"538'27","reported_seq":"919","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.347178+0000","last_change":"2021-02-10T04:37:39.100758+0000","last_active":"2021-02-10T04:43:28.347178+0000","last_peered":"2021-02-10T04:43:28.347178+0000","last_clean":"2021-02-10T04:43:28.347178+0000","last_became_active":"2021-02-10T04:37:37.257293+0000","last_became_peered":"2021-02-10T04:37:37.257293+0000","last_unstale":"2021-02-10T04:43:28.347178+0000","last_undegraded":"2021-02-10T04:43:28.347178+0000","last_fullsized":"2021-02-10T04:43:28.347178+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:28:46.680176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:46.680176+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"966","reported_epoch":"987","state":"active+clean","last_fresh":"2021-02-10T04:43:27.341276+0000","last_change":"2021-02-10T04:38:32.193334+0000","last_active":"2021-02-10T04:43:27.341276+0000","last_peered":"2021-02-10T04:43:27.341276+0000","last_clean":"2021-02-10T04:43:27.341276+0000","last_became_active":"2021-02-10T04:25:36.336442+0000","last_became_peered":"2021-02-10T04:25:36.336442+0000","last_unstale":"2021-02-10T04:43:27.341276+0000","last_undegraded":"2021-02-10T04:43:27.341276+0000","last_fullsized":"2021-02-10T04:43:27.341276+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:38:32.193240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:32.193240+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"634","reported_epoch":"990","state":"active+clean","last_fresh":"2021-02-10T04:43:30.346519+0000","last_change":"2021-02-10T04:33:05.147757+0000","last_active":"2021-02-10T04:43:30.346519+0000","last_peered":"2021-02-10T04:43:30.346519+0000","last_clean":"2021-02-10T04:43:30.346519+0000","last_became_active":"2021-02-10T04:33:05.146838+0000","last_became_peered":"2021-02-10T04:33:05.146838+0000","last_unstale":"2021-02-10T04:43:30.346519+0000","last_undegraded":"2021-02-10T04:43:30.346519+0000","last_fullsized":"2021-02-10T04:43:30.346519+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:22.078247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:22.078247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"949","reported_epoch":"990","state":"active+clean","last_fresh":"2021-02-10T04:43:30.346826+0000","last_change":"2021-02-10T04:29:31.903091+0000","last_active":"2021-02-10T04:43:30.346826+0000","last_peered":"2021-02-10T04:43:30.346826+0000","last_clean":"2021-02-10T04:43:30.346826+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:43:30.346826+0000","last_undegraded":"2021-02-10T04:43:30.346826+0000","last_fullsized":"2021-02-10T04:43:30.346826+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:31.902996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.902996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"538'28","reported_seq":"1006","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.347476+0000","last_change":"2021-02-10T04:38:25.860322+0000","last_active":"2021-02-10T04:43:28.347476+0000","last_peered":"2021-02-10T04:43:28.347476+0000","last_clean":"2021-02-10T04:43:28.347476+0000","last_became_active":"2021-02-10T04:25:49.008997+0000","last_became_peered":"2021-02-10T04:25:49.008997+0000","last_unstale":"2021-02-10T04:43:28.347476+0000","last_undegraded":"2021-02-10T04:43:28.347476+0000","last_fullsized":"2021-02-10T04:43:28.347476+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'28","last_scrub_stamp":"2021-02-10T04:38:25.860247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:25.860247+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":40,"num_write":30,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"970","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.346333+0000","last_change":"2021-02-10T04:28:30.415204+0000","last_active":"2021-02-10T04:43:28.346333+0000","last_peered":"2021-02-10T04:43:28.346333+0000","last_clean":"2021-02-10T04:43:28.346333+0000","last_became_active":"2021-02-10T04:25:36.336893+0000","last_became_peered":"2021-02-10T04:25:36.336893+0000","last_unstale":"2021-02-10T04:43:28.346333+0000","last_undegraded":"2021-02-10T04:43:28.346333+0000","last_fullsized":"2021-02-10T04:43:28.346333+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:30.415134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:26:46.425288+0000","last_clean_scrub_stamp":"2021-02-10T04:28:30.415134+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"960","reported_epoch":"990","state":"active+clean","last_fresh":"2021-02-10T04:43:30.346344+0000","last_change":"2021-02-10T04:37:37.243679+0000","last_active":"2021-02-10T04:43:30.346344+0000","last_peered":"2021-02-10T04:43:30.346344+0000","last_clean":"2021-02-10T04:43:30.346344+0000","last_became_active":"2021-02-10T04:37:37.243275+0000","last_became_peered":"2021-02-10T04:37:37.243275+0000","last_unstale":"2021-02-10T04:43:30.346344+0000","last_undegraded":"2021-02-10T04:43:30.346344+0000","last_fullsized":"2021-02-10T04:43:30.346344+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:52.643099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:33:52.643099+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"947","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.348484+0000","last_change":"2021-02-10T04:29:26.355726+0000","last_active":"2021-02-10T04:43:28.348484+0000","last_peered":"2021-02-10T04:43:28.348484+0000","last_clean":"2021-02-10T04:43:28.348484+0000","last_became_active":"2021-02-10T04:25:53.147831+0000","last_became_peered":"2021-02-10T04:25:53.147831+0000","last_unstale":"2021-02-10T04:43:28.348484+0000","last_undegraded":"2021-02-10T04:43:28.348484+0000","last_fullsized":"2021-02-10T04:43:28.348484+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:26.355652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.355652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"538'31","reported_seq":"1016","reported_epoch":"990","state":"active+clean","last_fresh":"2021-02-10T04:43:30.345655+0000","last_change":"2021-02-10T04:28:32.565457+0000","last_active":"2021-02-10T04:43:30.345655+0000","last_peered":"2021-02-10T04:43:30.345655+0000","last_clean":"2021-02-10T04:43:30.345655+0000","last_became_active":"2021-02-10T04:25:49.000911+0000","last_became_peered":"2021-02-10T04:25:49.000911+0000","last_unstale":"2021-02-10T04:43:30.345655+0000","last_undegraded":"2021-02-10T04:43:30.345655+0000","last_fullsized":"2021-02-10T04:43:30.345655+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2021-02-10T04:28:32.565374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:32.565374+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":62,"num_read_kb":46,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"882","reported_epoch":"986","state":"active+clean","last_fresh":"2021-02-10T04:43:26.337029+0000","last_change":"2021-02-10T04:37:37.242121+0000","last_active":"2021-02-10T04:43:26.337029+0000","last_peered":"2021-02-10T04:43:26.337029+0000","last_clean":"2021-02-10T04:43:26.337029+0000","last_became_active":"2021-02-10T04:37:37.241751+0000","last_became_peered":"2021-02-10T04:37:37.241751+0000","last_unstale":"2021-02-10T04:43:26.337029+0000","last_undegraded":"2021-02-10T04:43:26.337029+0000","last_fullsized":"2021-02-10T04:43:26.337029+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:48.646802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:33:48.646802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"957","reported_epoch":"986","state":"active+clean","last_fresh":"2021-02-10T04:43:26.335691+0000","last_change":"2021-02-10T04:38:39.253459+0000","last_active":"2021-02-10T04:43:26.335691+0000","last_peered":"2021-02-10T04:43:26.335691+0000","last_clean":"2021-02-10T04:43:26.335691+0000","last_became_active":"2021-02-10T04:38:39.253122+0000","last_became_peered":"2021-02-10T04:38:39.253122+0000","last_unstale":"2021-02-10T04:43:26.335691+0000","last_undegraded":"2021-02-10T04:43:26.335691+0000","last_fullsized":"2021-02-10T04:43:26.335691+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:46.464108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:46.464108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"953","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.414161+0000","last_change":"2021-02-10T04:38:37.121756+0000","last_active":"2021-02-10T04:43:28.414161+0000","last_peered":"2021-02-10T04:43:28.414161+0000","last_clean":"2021-02-10T04:43:28.414161+0000","last_became_active":"2021-02-10T04:25:53.039844+0000","last_became_peered":"2021-02-10T04:25:53.039844+0000","last_unstale":"2021-02-10T04:43:28.414161+0000","last_undegraded":"2021-02-10T04:43:28.414161+0000","last_fullsized":"2021-02-10T04:43:28.414161+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:37.121654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:38:37.121654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"538'14","reported_seq":"983","reported_epoch":"990","state":"active+clean","last_fresh":"2021-02-10T04:43:30.345952+0000","last_change":"2021-02-10T04:30:13.089576+0000","last_active":"2021-02-10T04:43:30.345952+0000","last_peered":"2021-02-10T04:43:30.345952+0000","last_clean":"2021-02-10T04:43:30.345952+0000","last_became_active":"2021-02-10T04:25:49.013196+0000","last_became_peered":"2021-02-10T04:25:49.013196+0000","last_unstale":"2021-02-10T04:43:30.345952+0000","last_undegraded":"2021-02-10T04:43:30.345952+0000","last_fullsized":"2021-02-10T04:43:30.345952+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2021-02-10T04:30:13.089494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:30:13.089494+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":15,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"874","reported_epoch":"987","state":"active+clean","last_fresh":"2021-02-10T04:43:27.341045+0000","last_change":"2021-02-10T04:30:11.537208+0000","last_active":"2021-02-10T04:43:27.341045+0000","last_peered":"2021-02-10T04:43:27.341045+0000","last_clean":"2021-02-10T04:43:27.341045+0000","last_became_active":"2021-02-10T04:29:07.157944+0000","last_became_peered":"2021-02-10T04:29:07.157944+0000","last_unstale":"2021-02-10T04:43:27.341045+0000","last_undegraded":"2021-02-10T04:43:27.341045+0000","last_fullsized":"2021-02-10T04:43:27.341045+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:11.537136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.537136+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"871","reported_epoch":"990","state":"active+clean","last_fresh":"2021-02-10T04:43:30.345763+0000","last_change":"2021-02-10T04:29:07.146984+0000","last_active":"2021-02-10T04:43:30.345763+0000","last_peered":"2021-02-10T04:43:30.345763+0000","last_clean":"2021-02-10T04:43:30.345763+0000","last_became_active":"2021-02-10T04:29:07.146525+0000","last_became_peered":"2021-02-10T04:29:07.146525+0000","last_unstale":"2021-02-10T04:43:30.345763+0000","last_undegraded":"2021-02-10T04:43:30.345763+0000","last_fullsized":"2021-02-10T04:43:30.345763+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:46.493066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:46.493066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"946","reported_epoch":"986","state":"active+clean","last_fresh":"2021-02-10T04:43:26.335229+0000","last_change":"2021-02-10T04:29:22.475613+0000","last_active":"2021-02-10T04:43:26.335229+0000","last_peered":"2021-02-10T04:43:26.335229+0000","last_clean":"2021-02-10T04:43:26.335229+0000","last_became_active":"2021-02-10T04:25:53.042601+0000","last_became_peered":"2021-02-10T04:25:53.042601+0000","last_unstale":"2021-02-10T04:43:26.335229+0000","last_undegraded":"2021-02-10T04:43:26.335229+0000","last_fullsized":"2021-02-10T04:43:26.335229+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:22.475504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:22.475504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"538'38","reported_seq":"1016","reported_epoch":"990","state":"active+clean","last_fresh":"2021-02-10T04:43:30.345298+0000","last_change":"2021-02-10T04:28:35.520489+0000","last_active":"2021-02-10T04:43:30.345298+0000","last_peered":"2021-02-10T04:43:30.345298+0000","last_clean":"2021-02-10T04:43:30.345298+0000","last_became_active":"2021-02-10T04:25:49.010481+0000","last_became_peered":"2021-02-10T04:25:49.010481+0000","last_unstale":"2021-02-10T04:43:30.345298+0000","last_undegraded":"2021-02-10T04:43:30.345298+0000","last_fullsized":"2021-02-10T04:43:30.345298+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'32","last_scrub_stamp":"2021-02-10T04:28:35.520410+0000","last_deep_scrub":"78'32","last_deep_scrub_stamp":"2021-02-10T04:27:09.170428+0000","last_clean_scrub_stamp":"2021-02-10T04:28:35.520410+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":56,"num_read_kb":37,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"958","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.347843+0000","last_change":"2021-02-10T04:29:29.355356+0000","last_active":"2021-02-10T04:43:28.347843+0000","last_peered":"2021-02-10T04:43:28.347843+0000","last_clean":"2021-02-10T04:43:28.347843+0000","last_became_active":"2021-02-10T04:25:34.327162+0000","last_became_peered":"2021-02-10T04:25:34.327162+0000","last_unstale":"2021-02-10T04:43:28.347843+0000","last_undegraded":"2021-02-10T04:43:28.347843+0000","last_fullsized":"2021-02-10T04:43:28.347843+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.355257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.355257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"631","reported_epoch":"986","state":"active+clean","last_fresh":"2021-02-10T04:43:26.334884+0000","last_change":"2021-02-10T04:33:05.169577+0000","last_active":"2021-02-10T04:43:26.334884+0000","last_peered":"2021-02-10T04:43:26.334884+0000","last_clean":"2021-02-10T04:43:26.334884+0000","last_became_active":"2021-02-10T04:33:05.169159+0000","last_became_peered":"2021-02-10T04:33:05.169159+0000","last_unstale":"2021-02-10T04:43:26.334884+0000","last_undegraded":"2021-02-10T04:43:26.334884+0000","last_fullsized":"2021-02-10T04:43:26.334884+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:40.098239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:40.098239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"1035","reported_epoch":"990","state":"active+clean","last_fresh":"2021-02-10T04:43:30.345659+0000","last_change":"2021-02-10T04:38:23.319098+0000","last_active":"2021-02-10T04:43:30.345659+0000","last_peered":"2021-02-10T04:43:30.345659+0000","last_clean":"2021-02-10T04:43:30.345659+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:43:30.345659+0000","last_undegraded":"2021-02-10T04:43:30.345659+0000","last_fullsized":"2021-02-10T04:43:30.345659+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:23.319023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:38:23.319023+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"538'56","reported_seq":"1107","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.345537+0000","last_change":"2021-02-10T04:38:21.961539+0000","last_active":"2021-02-10T04:43:28.345537+0000","last_peered":"2021-02-10T04:43:28.345537+0000","last_clean":"2021-02-10T04:43:28.345537+0000","last_became_active":"2021-02-10T04:37:37.266384+0000","last_became_peered":"2021-02-10T04:37:37.266384+0000","last_unstale":"2021-02-10T04:43:28.345537+0000","last_undegraded":"2021-02-10T04:43:28.345537+0000","last_fullsized":"2021-02-10T04:43:28.345537+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":641,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'56","last_scrub_stamp":"2021-02-10T04:38:21.961470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:21.961470+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":107,"num_read_kb":84,"num_write":57,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":890,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"968","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.415624+0000","last_change":"2021-02-10T04:29:28.719237+0000","last_active":"2021-02-10T04:43:28.415624+0000","last_peered":"2021-02-10T04:43:28.415624+0000","last_clean":"2021-02-10T04:43:28.415624+0000","last_became_active":"2021-02-10T04:25:36.336112+0000","last_became_peered":"2021-02-10T04:25:36.336112+0000","last_unstale":"2021-02-10T04:43:28.415624+0000","last_undegraded":"2021-02-10T04:43:28.415624+0000","last_fullsized":"2021-02-10T04:43:28.415624+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.719147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.719147+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"298","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.343195+0000","last_change":"2021-02-10T04:38:39.244331+0000","last_active":"2021-02-10T04:43:28.343195+0000","last_peered":"2021-02-10T04:43:28.343195+0000","last_clean":"2021-02-10T04:43:28.343195+0000","last_became_active":"2021-02-10T04:38:39.243965+0000","last_became_peered":"2021-02-10T04:38:39.243965+0000","last_unstale":"2021-02-10T04:43:28.343195+0000","last_undegraded":"2021-02-10T04:43:28.343195+0000","last_fullsized":"2021-02-10T04:43:28.343195+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:33.199521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:38:33.199521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"947","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.346263+0000","last_change":"2021-02-10T04:29:25.382883+0000","last_active":"2021-02-10T04:43:28.346263+0000","last_peered":"2021-02-10T04:43:28.346263+0000","last_clean":"2021-02-10T04:43:28.346263+0000","last_became_active":"2021-02-10T04:25:53.150917+0000","last_became_peered":"2021-02-10T04:25:53.150917+0000","last_unstale":"2021-02-10T04:43:28.346263+0000","last_undegraded":"2021-02-10T04:43:28.346263+0000","last_fullsized":"2021-02-10T04:43:28.346263+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:25.382793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.382793+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.20","version":"0'0","reported_seq":"936","reported_epoch":"986","state":"active+clean","last_fresh":"2021-02-10T04:43:26.333890+0000","last_change":"2021-02-10T04:30:11.492843+0000","last_active":"2021-02-10T04:43:26.333890+0000","last_peered":"2021-02-10T04:43:26.333890+0000","last_clean":"2021-02-10T04:43:26.333890+0000","last_became_active":"2021-02-10T04:29:07.148937+0000","last_became_peered":"2021-02-10T04:29:07.148937+0000","last_unstale":"2021-02-10T04:43:26.333890+0000","last_undegraded":"2021-02-10T04:43:26.333890+0000","last_fullsized":"2021-02-10T04:43:26.333890+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:11.492788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:41.375276+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.492788+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":"451","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.413904+0000","last_change":"2021-02-10T04:37:37.247286+0000","last_active":"2021-02-10T04:43:28.413904+0000","last_peered":"2021-02-10T04:43:28.413904+0000","last_clean":"2021-02-10T04:43:28.413904+0000","last_became_active":"2021-02-10T04:37:37.246865+0000","last_became_peered":"2021-02-10T04:37:37.246865+0000","last_unstale":"2021-02-10T04:43:28.413904+0000","last_undegraded":"2021-02-10T04:43:28.413904+0000","last_fullsized":"2021-02-10T04:43:28.413904+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:21.708701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:21.708701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":"972","reported_epoch":"987","state":"active+clean","last_fresh":"2021-02-10T04:43:27.341010+0000","last_change":"2021-02-10T04:33:58.892594+0000","last_active":"2021-02-10T04:43:27.341010+0000","last_peered":"2021-02-10T04:43:27.341010+0000","last_clean":"2021-02-10T04:43:27.341010+0000","last_became_active":"2021-02-10T04:28:23.129832+0000","last_became_peered":"2021-02-10T04:28:23.129832+0000","last_unstale":"2021-02-10T04:43:27.341010+0000","last_undegraded":"2021-02-10T04:43:27.341010+0000","last_fullsized":"2021-02-10T04:43:27.341010+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:58.892504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:33:58.892504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":"912","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.346217+0000","last_change":"2021-02-10T04:28:48.409915+0000","last_active":"2021-02-10T04:43:28.346217+0000","last_peered":"2021-02-10T04:43:28.346217+0000","last_clean":"2021-02-10T04:43:28.346217+0000","last_became_active":"2021-02-10T04:28:23.129523+0000","last_became_peered":"2021-02-10T04:28:23.129523+0000","last_unstale":"2021-02-10T04:43:28.346217+0000","last_undegraded":"2021-02-10T04:43:28.346217+0000","last_fullsized":"2021-02-10T04:43:28.346217+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:48.409826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:48.409826+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":"916","reported_epoch":"987","state":"active+clean","last_fresh":"2021-02-10T04:43:27.340713+0000","last_change":"2021-02-10T04:34:01.895599+0000","last_active":"2021-02-10T04:43:27.340713+0000","last_peered":"2021-02-10T04:43:27.340713+0000","last_clean":"2021-02-10T04:43:27.340713+0000","last_became_active":"2021-02-10T04:28:23.126368+0000","last_became_peered":"2021-02-10T04:28:23.126368+0000","last_unstale":"2021-02-10T04:43:27.340713+0000","last_undegraded":"2021-02-10T04:43:27.340713+0000","last_fullsized":"2021-02-10T04:43:27.340713+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:34:01.895514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:34:01.895514+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.25","version":"53'2","reported_seq":"931","reported_epoch":"990","state":"active+clean","last_fresh":"2021-02-10T04:43:30.346039+0000","last_change":"2021-02-10T04:28:40.422198+0000","last_active":"2021-02-10T04:43:30.346039+0000","last_peered":"2021-02-10T04:43:30.346039+0000","last_clean":"2021-02-10T04:43:30.346039+0000","last_became_active":"2021-02-10T04:28:23.467770+0000","last_became_peered":"2021-02-10T04:28:23.467770+0000","last_unstale":"2021-02-10T04:43:30.346039+0000","last_undegraded":"2021-02-10T04:43:30.346039+0000","last_fullsized":"2021-02-10T04:43:30.346039+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:28:40.422115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:40.422115+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.26","version":"53'1","reported_seq":"376","reported_epoch":"990","state":"active+clean","last_fresh":"2021-02-10T04:43:30.346654+0000","last_change":"2021-02-10T04:37:37.233712+0000","last_active":"2021-02-10T04:43:30.346654+0000","last_peered":"2021-02-10T04:43:30.346654+0000","last_clean":"2021-02-10T04:43:30.346654+0000","last_became_active":"2021-02-10T04:37:37.231081+0000","last_became_peered":"2021-02-10T04:37:37.231081+0000","last_unstale":"2021-02-10T04:43:30.346654+0000","last_undegraded":"2021-02-10T04:43:30.346654+0000","last_fullsized":"2021-02-10T04:43:30.346654+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:29:22.801929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:22.801929+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":423,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":"947","reported_epoch":"990","state":"active+clean","last_fresh":"2021-02-10T04:43:30.346853+0000","last_change":"2021-02-10T04:38:37.219069+0000","last_active":"2021-02-10T04:43:30.346853+0000","last_peered":"2021-02-10T04:43:30.346853+0000","last_clean":"2021-02-10T04:43:30.346853+0000","last_became_active":"2021-02-10T04:29:07.145907+0000","last_became_peered":"2021-02-10T04:29:07.145907+0000","last_unstale":"2021-02-10T04:43:30.346853+0000","last_undegraded":"2021-02-10T04:43:30.346853+0000","last_fullsized":"2021-02-10T04:43:30.346853+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:37.218968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:37.218968+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":"912","reported_epoch":"988","state":"active+clean","last_fresh":"2021-02-10T04:43:28.344827+0000","last_change":"2021-02-10T04:29:23.440451+0000","last_active":"2021-02-10T04:43:28.344827+0000","last_peered":"2021-02-10T04:43:28.344827+0000","last_clean":"2021-02-10T04:43:28.344827+0000","last_became_active":"2021-02-10T04:28:23.129469+0000","last_became_peered":"2021-02-10T04:28:23.129469+0000","last_unstale":"2021-02-10T04:43:28.344827+0000","last_undegraded":"2021-02-10T04:43:28.344827+0000","last_fullsized":"2021-02-10T04:43:28.344827+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.440391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.440391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":"916","reported_epoch":"987","state":"active+clean","last_fresh":"2021-02-10T04:43:27.340267+0000","last_change":"2021-02-10T04:38:22.136377+0000","last_active":"2021-02-10T04:43:27.340267+0000","last_peered":"2021-02-10T04:43:27.340267+0000","last_clean":"2021-02-10T04:43:27.340267+0000","last_became_active":"2021-02-10T04:28:23.126795+0000","last_became_peered":"2021-02-10T04:28:23.126795+0000","last_unstale":"2021-02-10T04:43:27.340267+0000","last_undegraded":"2021-02-10T04:43:27.340267+0000","last_fullsized":"2021-02-10T04:43:27.340267+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:22.136290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_clean_scrub_stamp":"2021-02-10T04:38:22.136290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":913879578,"num_objects":373,"num_object_clones":323,"num_object_copies":1119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":373,"num_whiteouts":9,"num_read":7442,"num_read_kb":7588688,"num_write":5423,"num_write_kb":2070038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":291,"num_bytes_recovered":108939670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1903296512,"data_stored":2585452841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15069,"ondisk_log_size":15069,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":1945,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":18,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":6565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":126,"acting":126,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":358,"num_read_kb":353,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":1876,"num_read_kb":1349,"num_write":1020,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":8770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":75722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1068,"ondisk_log_size":1068,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21,"ondisk_log_size":21,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365017,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":205008,"kb_used_data":162312,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":93540144,"statfs":{"total":95995035648,"available":95785107456,"internally_reserved":0,"allocated":166207488,"data_stored":246084531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:43:02 2021","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":1.679,"15min":1.482},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.46700000000000003},"max":{"1min":1.8009999999999999,"5min":67.277000000000001,"15min":72.805999999999997},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.96499999999999997,"5min":2.044,"15min":1.6970000000000001},"min":{"1min":0.56599999999999995,"5min":0.54200000000000004,"15min":0.50800000000000001},"max":{"1min":1.829,"5min":66.933999999999997,"15min":72.721999999999994},"last":0.89200000000000002}]},{"osd":1,"last update":"Wed Feb 10 04:43:02 2021","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":1.054,"15min":1.133},"min":{"1min":0.59799999999999998,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.48,"5min":10.140000000000001,"15min":41.966000000000001},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.85999999999999999,"5min":1.0069999999999999,"15min":1.1319999999999999},"min":{"1min":0.60599999999999998,"5min":0.57299999999999995,"15min":0.56499999999999995},"max":{"1min":1.234,"5min":7.1520000000000001,"15min":41.804000000000002},"last":0.93300000000000005}]},{"osd":2,"last update":"Wed Feb 10 04:43:02 2021","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":1.778,"15min":2.125},"min":{"1min":0.628,"5min":0.628,"15min":0.58299999999999996},"max":{"1min":1.375,"5min":66.534000000000006,"15min":125.714},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.2609999999999999,"5min":2.2269999999999999,"15min":2.3599999999999999},"min":{"1min":0.71099999999999997,"5min":0.66900000000000004,"15min":0.59099999999999997},"max":{"1min":2.218,"5min":66.781999999999996,"15min":125.53700000000001},"last":0.78100000000000003}]},{"osd":3,"last update":"Wed Feb 10 04:43:02 2021","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":2.1749999999999998,"15min":2.944},"min":{"1min":0.85699999999999998,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.728,"5min":68.305000000000007,"15min":192.95500000000001},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.143,"5min":2.0859999999999999,"15min":2.9209999999999998},"min":{"1min":0.82999999999999996,"5min":0.80400000000000005,"15min":0.72199999999999998},"max":{"1min":1.4890000000000001,"5min":69.331999999999994,"15min":192.601},"last":1.6399999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:43:02 2021","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.6220000000000001,"15min":1.6919999999999999},"min":{"1min":0.79800000000000004,"5min":0.70699999999999996,"15min":0.64600000000000002},"max":{"1min":2.383,"5min":10.781000000000001,"15min":97.733999999999995},"last":1.891},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.4119999999999999,"15min":1.639},"min":{"1min":0.78300000000000003,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.847,"5min":10.326000000000001,"15min":97.691000000000003},"last":1.127}]},{"osd":5,"last update":"Wed Feb 10 04:43:02 2021","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.7889999999999999,"15min":1.6080000000000001},"min":{"1min":0.92300000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.0350000000000001,"5min":35.283000000000001,"15min":35.283000000000001},"last":1.7629999999999999},{"interface":"front","average":{"1min":1.5169999999999999,"5min":2.2330000000000001,"15min":1.8959999999999999},"min":{"1min":0.997,"5min":0.86099999999999999,"15min":0.753},"max":{"1min":2.4860000000000002,"5min":35.353999999999999,"15min":35.353999999999999},"last":2.0859999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:43:02 2021","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":2.0150000000000001,"15min":1.97},"min":{"1min":1.028,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":7.79,"5min":11.343999999999999,"15min":32.616},"last":1.9990000000000001},{"interface":"front","average":{"1min":1.7370000000000001,"5min":1.8149999999999999,"15min":1.849},"min":{"1min":1.002,"5min":0.95699999999999996,"15min":0.79500000000000004},"max":{"1min":8.0739999999999998,"5min":10.428000000000001,"15min":32.424999999999997},"last":1.863}]}]},{"osd":6,"up_from":43,"seq":184683593949,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":351452,"kb_used_data":307784,"kb_used_omap":0,"kb_used_meta":43456,"kb_avail":93393700,"statfs":{"total":95995035648,"available":95635148800,"internally_reserved":0,"allocated":315170816,"data_stored":410334960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":44498944},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:42:54 2021","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":1.036,"15min":1.431},"min":{"1min":0.59899999999999998,"5min":0.47099999999999997,"15min":0.38800000000000001},"max":{"1min":1.419,"5min":13.393000000000001,"15min":64.164000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":1.139,"5min":1.1439999999999999,"15min":1.8420000000000001},"min":{"1min":0.57799999999999996,"5min":0.5,"15min":0.47399999999999998},"max":{"1min":2.2589999999999999,"5min":8.0289999999999999,"15min":64.277000000000001},"last":0.56999999999999995}]},{"osd":1,"last update":"Wed Feb 10 04:42:54 2021","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.101,"15min":1.127},"min":{"1min":0.57299999999999995,"5min":0.54400000000000004,"15min":0.443},"max":{"1min":1.548,"5min":12.978999999999999,"15min":18.177},"last":0.59799999999999998},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.131,"15min":1.1539999999999999},"min":{"1min":0.67000000000000004,"5min":0.60899999999999999,"15min":0.52700000000000002},"max":{"1min":1.6779999999999999,"5min":3.3239999999999998,"15min":18.898},"last":0.90500000000000003}]},{"osd":2,"last update":"Wed Feb 10 04:42:54 2021","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.268,"15min":1.3400000000000001},"min":{"1min":0.76900000000000002,"5min":0.55600000000000005,"15min":0.54100000000000004},"max":{"1min":2.101,"5min":2.6840000000000002,"15min":24.725000000000001},"last":1.032},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.252,"15min":1.3280000000000001},"min":{"1min":0.78900000000000003,"5min":0.59899999999999998,"15min":0.55300000000000005},"max":{"1min":2.0230000000000001,"5min":2.1440000000000001,"15min":24.611999999999998},"last":0.96199999999999997}]},{"osd":3,"last update":"Wed Feb 10 04:42:54 2021","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":2.7829999999999999,"15min":1.919},"min":{"1min":0.88900000000000001,"5min":0.76400000000000001,"15min":0.63200000000000001},"max":{"1min":2.137,"5min":114.17700000000001,"15min":114.17700000000001},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.3480000000000001,"5min":2.8820000000000001,"15min":1.944},"min":{"1min":0.87,"5min":0.84499999999999997,"15min":0.71699999999999997},"max":{"1min":2.0430000000000001,"5min":116.929,"15min":116.929},"last":1.091}]},{"osd":4,"last update":"Wed Feb 10 04:42:54 2021","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.502,"15min":1.3839999999999999},"min":{"1min":0.95599999999999996,"5min":0.64800000000000002,"15min":0.60599999999999998},"max":{"1min":2.4460000000000002,"5min":4.0810000000000004,"15min":7.5389999999999997},"last":1.349},{"interface":"front","average":{"1min":0.98699999999999999,"5min":1.135,"15min":1.0569999999999999},"min":{"1min":0.73799999999999999,"5min":0.68500000000000005,"15min":0.57999999999999996},"max":{"1min":1.853,"5min":3.2869999999999999,"15min":3.2869999999999999},"last":1.0620000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:42:54 2021","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.3959999999999999,"15min":1.3200000000000001},"min":{"1min":0.84199999999999997,"5min":0.68799999999999994,"15min":0.64100000000000001},"max":{"1min":2.1320000000000001,"5min":4.6369999999999996,"15min":11.380000000000001},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.544,"5min":1.5860000000000001,"15min":1.5049999999999999},"min":{"1min":0.97499999999999998,"5min":0.78000000000000003,"15min":0.66600000000000004},"max":{"1min":2.254,"5min":2.782,"15min":11.538},"last":1.3200000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:43:18 2021","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.353,"15min":1.3180000000000001},"min":{"1min":0.83899999999999997,"5min":0.755,"15min":0.755},"max":{"1min":1.845,"5min":4.5279999999999996,"15min":4.5279999999999996},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.768,"5min":1.752,"15min":1.611},"min":{"1min":1.258,"5min":0.91000000000000003,"15min":0.77500000000000002},"max":{"1min":2.7240000000000002,"5min":13.315,"15min":13.315},"last":1.198}]}]},{"osd":1,"up_from":13,"seq":55834575086,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":407400,"kb_used_data":349216,"kb_used_omap":0,"kb_used_meta":58176,"kb_avail":93337752,"statfs":{"total":95995035648,"available":95577858048,"internally_reserved":0,"allocated":357597184,"data_stored":459461926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":59572224},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:43:24 2021","interfaces":[{"interface":"back","average":{"1min":2.399,"5min":1.881,"15min":1.5669999999999999},"min":{"1min":0.55900000000000005,"5min":0.50600000000000001,"15min":0.47899999999999998},"max":{"1min":31.198,"5min":55.475999999999999,"15min":55.475999999999999},"last":1.2370000000000001},{"interface":"front","average":{"1min":2.423,"5min":1.958,"15min":1.651},"min":{"1min":0.53300000000000003,"5min":0.53000000000000003,"15min":0.51800000000000002},"max":{"1min":31.558,"5min":55.368000000000002,"15min":55.368000000000002},"last":2.3140000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:42:46 2021","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.464,"15min":1.4179999999999999},"min":{"1min":0.60599999999999998,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":2.7000000000000002,"5min":32.110999999999997,"15min":40.268000000000001},"last":2.0659999999999998},{"interface":"front","average":{"1min":1.123,"5min":1.4419999999999999,"15min":1.4490000000000001},"min":{"1min":0.70699999999999996,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":2.895,"5min":32.302999999999997,"15min":40.651000000000003},"last":1.109}]},{"osd":3,"last update":"Wed Feb 10 04:43:03 2021","interfaces":[{"interface":"back","average":{"1min":2.6850000000000001,"5min":2.0289999999999999,"15min":1.734},"min":{"1min":0.69499999999999995,"5min":0.57899999999999996,"15min":0.45200000000000001},"max":{"1min":30.274999999999999,"5min":32.432000000000002,"15min":40.314},"last":1.351},{"interface":"front","average":{"1min":2.6819999999999999,"5min":2.0209999999999999,"15min":1.7270000000000001},"min":{"1min":0.65500000000000003,"5min":0.60399999999999998,"15min":0.47199999999999998},"max":{"1min":31.443000000000001,"5min":32.811,"15min":40.973999999999997},"last":2.4359999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:43:24 2021","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.256,"15min":1.2549999999999999},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":3.258,"5min":3.7440000000000002,"15min":6.3099999999999996},"last":1.4750000000000001},{"interface":"front","average":{"1min":2.6560000000000001,"5min":1.647,"15min":1.444},"min":{"1min":0.84799999999999998,"5min":0.754,"15min":0.70999999999999996},"max":{"1min":29.547999999999998,"5min":29.547999999999998,"15min":29.547999999999998},"last":1.792}]},{"osd":5,"last update":"Wed Feb 10 04:42:35 2021","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.331,"15min":1.508},"min":{"1min":0.747,"5min":0.71399999999999997,"15min":0.65200000000000002},"max":{"1min":4.6289999999999996,"5min":7.8049999999999997,"15min":48.686999999999998},"last":2.5699999999999998},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.53,"15min":1.7},"min":{"1min":0.84599999999999997,"5min":0.749,"15min":0.69699999999999995},"max":{"1min":3.4990000000000001,"5min":3.8809999999999998,"15min":48.905000000000001},"last":1.585}]},{"osd":6,"last update":"Wed Feb 10 04:42:46 2021","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":1.6539999999999999},"min":{"1min":0.82199999999999995,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":3.125,"5min":3.125,"15min":48.878999999999998},"last":1.931},{"interface":"front","average":{"1min":1.417,"5min":1.383,"15min":1.552},"min":{"1min":0.88800000000000001,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":4.7560000000000002,"5min":8.5169999999999995,"15min":49.011000000000003},"last":2.6800000000000002}]},{"osd":7,"last update":"Wed Feb 10 04:43:03 2021","interfaces":[{"interface":"back","average":{"1min":2.7669999999999999,"5min":1.772,"15min":1.601},"min":{"1min":1.002,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":28.573,"5min":28.573,"15min":28.573},"last":1.6910000000000001},{"interface":"front","average":{"1min":2.9510000000000001,"5min":1.847,"15min":1.6459999999999999},"min":{"1min":1.119,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":29.408999999999999,"5min":29.408999999999999,"15min":29.408999999999999},"last":2.1899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738612,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":347204,"kb_used_data":296176,"kb_used_omap":0,"kb_used_meta":50816,"kb_avail":93397948,"statfs":{"total":95995035648,"available":95639498752,"internally_reserved":0,"allocated":303284224,"data_stored":425923076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52035584},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 04:43:16 2021","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":0.95999999999999996,"15min":1.1000000000000001},"min":{"1min":0.54600000000000004,"5min":0.50900000000000001,"15min":0.48099999999999998},"max":{"1min":2.4159999999999999,"5min":2.4340000000000002,"15min":28.946999999999999},"last":1.5960000000000001},{"interface":"front","average":{"1min":1.181,"5min":1.091,"15min":1.014},"min":{"1min":0.52700000000000002,"5min":0.52100000000000002,"15min":0.51200000000000001},"max":{"1min":3.516,"5min":3.7050000000000001,"15min":7.5549999999999997},"last":1.262}]},{"osd":2,"last update":"Wed Feb 10 04:42:47 2021","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.4330000000000001,"15min":1.3029999999999999},"min":{"1min":0.57399999999999995,"5min":0.51400000000000001,"15min":0.48099999999999998},"max":{"1min":2.786,"5min":25.984999999999999,"15min":25.984999999999999},"last":1.0740000000000001},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.238,"15min":1.1559999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.51300000000000001},"max":{"1min":2.198,"5min":5.4930000000000003,"15min":16.829000000000001},"last":0.81999999999999995}]},{"osd":3,"last update":"Wed Feb 10 04:43:05 2021","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.484,"15min":1.8260000000000001},"min":{"1min":0.61199999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":4.5830000000000002,"5min":29.692,"15min":126.587},"last":1.7},{"interface":"front","average":{"1min":1.639,"5min":1.6950000000000001,"15min":2.1150000000000002},"min":{"1min":0.83399999999999996,"5min":0.79400000000000004,"15min":0.54200000000000004},"max":{"1min":3.6459999999999999,"5min":25.315000000000001,"15min":126.88200000000001},"last":1.2110000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:43:16 2021","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.456,"15min":1.327},"min":{"1min":0.85299999999999998,"5min":0.67200000000000004,"15min":0.59399999999999997},"max":{"1min":2.9430000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.276,"5min":1.2649999999999999,"15min":1.337},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.63800000000000001},"max":{"1min":2.8690000000000002,"5min":4.5810000000000004,"15min":14.672000000000001},"last":1.7909999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:42:35 2021","interfaces":[{"interface":"back","average":{"1min":2.556,"5min":1.79,"15min":1.5409999999999999},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":25.047000000000001,"5min":25.047000000000001,"15min":25.047000000000001},"last":2.6720000000000002},{"interface":"front","average":{"1min":2.8410000000000002,"5min":2.0670000000000002,"15min":1.7829999999999999},"min":{"1min":1.1000000000000001,"5min":0.93899999999999995,"15min":0.81699999999999995},"max":{"1min":24.972999999999999,"5min":28.827999999999999,"15min":28.827999999999999},"last":1.96}]},{"osd":6,"last update":"Wed Feb 10 04:42:59 2021","interfaces":[{"interface":"back","average":{"1min":2.6349999999999998,"5min":1.849,"15min":1.8580000000000001},"min":{"1min":1.05,"5min":0.89100000000000001,"15min":0.88800000000000001},"max":{"1min":24.837,"5min":24.837,"15min":35.872999999999998},"last":2.52},{"interface":"front","average":{"1min":2.8340000000000001,"5min":1.869,"15min":1.984},"min":{"1min":1.1759999999999999,"5min":0.96099999999999997,"15min":0.83099999999999996},"max":{"1min":24.870999999999999,"5min":24.870999999999999,"15min":35.911000000000001},"last":2.0750000000000002}]},{"osd":7,"last update":"Wed Feb 10 04:43:05 2021","interfaces":[{"interface":"back","average":{"1min":1.732,"5min":1.6279999999999999,"15min":1.518},"min":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.73499999999999999},"max":{"1min":3.121,"5min":3.472,"15min":5.6050000000000004},"last":1.5309999999999999},{"interface":"front","average":{"1min":1.6899999999999999,"5min":1.609,"15min":1.728},"min":{"1min":0.97299999999999998,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":3.4300000000000002,"5min":5.1299999999999999,"15min":16.545000000000002},"last":1.869}]}]},{"osd":2,"up_from":0,"seq":77309411565,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":25,"seq":107374182631,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":443160,"kb_used_data":385168,"kb_used_omap":0,"kb_used_meta":57984,"kb_avail":93301992,"statfs":{"total":95995035648,"available":95541239808,"internally_reserved":0,"allocated":394412032,"data_stored":557365788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":59375616},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:43:07 2021","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":1.0620000000000001,"15min":1.0760000000000001},"min":{"1min":0.627,"5min":0.53300000000000003,"15min":0.47599999999999998},"max":{"1min":0.97399999999999998,"5min":7.899,"15min":17.233000000000001},"last":1.466},{"interface":"front","average":{"1min":1.018,"5min":1.129,"15min":1.202},"min":{"1min":0.61099999999999999,"5min":0.53600000000000003,"15min":0.52300000000000002},"max":{"1min":2.6640000000000001,"5min":3.7330000000000001,"15min":20.216999999999999},"last":1.5169999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:43:07 2021","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.075,"15min":1.1060000000000001},"min":{"1min":0.745,"5min":0.56599999999999995,"15min":0.54400000000000004},"max":{"1min":2.4460000000000002,"5min":3.9550000000000001,"15min":8.9190000000000005},"last":0.79000000000000004},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.1180000000000001,"15min":1.04},"min":{"1min":0.64400000000000002,"5min":0.55600000000000005,"15min":0.52100000000000002},"max":{"1min":1.9410000000000001,"5min":2.5390000000000001,"15min":4.6769999999999996},"last":0.82099999999999995}]},{"osd":2,"last update":"Wed Feb 10 04:43:07 2021","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.4139999999999999,"15min":1.2789999999999999},"min":{"1min":0.68999999999999995,"5min":0.624,"15min":0.56799999999999995},"max":{"1min":2.4289999999999998,"5min":4.1900000000000004,"15min":5.024},"last":1.196},{"interface":"front","average":{"1min":1.21,"5min":1.2390000000000001,"15min":1.179},"min":{"1min":0.67100000000000004,"5min":0.65200000000000002,"15min":0.56999999999999995},"max":{"1min":2.1709999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"last":1.0800000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:43:15 2021","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.4039999999999999,"15min":1.5900000000000001},"min":{"1min":0.85499999999999998,"5min":0.80700000000000005,"15min":0.629},"max":{"1min":2.5390000000000001,"5min":4.6909999999999998,"15min":30.751999999999999},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.536,"15min":1.663},"min":{"1min":0.93799999999999994,"5min":0.85999999999999999,"15min":0.65700000000000003},"max":{"1min":2.6970000000000001,"5min":4.4699999999999998,"15min":30.928000000000001},"last":1.0249999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:42:34 2021","interfaces":[{"interface":"back","average":{"1min":1.611,"5min":1.621,"15min":1.873},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":2.948,"5min":3.5529999999999999,"15min":35.616999999999997},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.8129999999999999,"5min":1.752,"15min":1.946},"min":{"1min":1.0740000000000001,"5min":0.81799999999999995,"15min":0.77200000000000002},"max":{"1min":3.0640000000000001,"5min":4.931,"15min":35.661999999999999},"last":1.1259999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:42:58 2021","interfaces":[{"interface":"back","average":{"1min":1.5800000000000001,"5min":1.6779999999999999,"15min":2.0190000000000001},"min":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.83699999999999997},"max":{"1min":2.7320000000000002,"5min":3.1739999999999999,"15min":35.518000000000001},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.8500000000000001,"15min":2.2589999999999999},"min":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":0.86599999999999999},"max":{"1min":2.8220000000000001,"5min":5.0839999999999996,"15min":53.965000000000003},"last":1.2709999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:43:07 2021","interfaces":[{"interface":"back","average":{"1min":1.7330000000000001,"5min":1.7689999999999999,"15min":1.845},"min":{"1min":1.256,"5min":1.1830000000000001,"15min":0.99199999999999999},"max":{"1min":2.7930000000000001,"5min":3.4380000000000002,"15min":25.242999999999999},"last":1.347},{"interface":"front","average":{"1min":2.0539999999999998,"5min":1.903,"15min":1.919},"min":{"1min":1.3,"5min":1.048,"15min":0.98299999999999998},"max":{"1min":5.0270000000000001,"5min":5.1779999999999999,"15min":25.286999999999999},"last":1.5549999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986405,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":165080,"kb_used_data":124560,"kb_used_omap":0,"kb_used_meta":40512,"kb_avail":93580072,"statfs":{"total":95995035648,"available":95825993728,"internally_reserved":0,"allocated":127549440,"data_stored":157317379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41484288},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4],"upper_bound":2097152},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:43:18 2021","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.6339999999999999,"15min":1.3640000000000001},"min":{"1min":0.503,"5min":0.503,"15min":0.48699999999999999},"max":{"1min":5.3419999999999996,"5min":43.939999999999998,"15min":62.273000000000003},"last":1.4339999999999999},{"interface":"front","average":{"1min":0.98499999999999999,"5min":1.645,"15min":1.4099999999999999},"min":{"1min":0.60099999999999998,"5min":0.56599999999999995,"15min":0.499},"max":{"1min":1.7010000000000001,"5min":43.829999999999998,"15min":62.375999999999998},"last":0.60199999999999998}]},{"osd":1,"last update":"Wed Feb 10 04:43:18 2021","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":1.0700000000000001,"15min":0.91700000000000004},"min":{"1min":0.61699999999999999,"5min":0.54300000000000004,"15min":0.52800000000000002},"max":{"1min":1.3580000000000001,"5min":24.295000000000002,"15min":24.295000000000002},"last":0.627},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.5089999999999999,"15min":1.2030000000000001},"min":{"1min":0.63600000000000001,"5min":0.61099999999999999,"15min":0.59999999999999998},"max":{"1min":6.5060000000000002,"5min":24.943000000000001,"15min":24.943000000000001},"last":1.5649999999999999}]},{"osd":2,"last update":"Wed Feb 10 04:43:18 2021","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.1619999999999999,"15min":1.3939999999999999},"min":{"1min":0.66000000000000003,"5min":0.61399999999999999,"15min":0.56499999999999995},"max":{"1min":1.853,"5min":7.048,"15min":62.521000000000001},"last":1.6919999999999999},{"interface":"front","average":{"1min":1.101,"5min":1.1719999999999999,"15min":1.4610000000000001},"min":{"1min":0.68200000000000005,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":2.1280000000000001,"5min":2.1280000000000001,"15min":62.572000000000003},"last":0.91300000000000003}]},{"osd":3,"last update":"Wed Feb 10 04:43:18 2021","interfaces":[{"interface":"back","average":{"1min":1.639,"5min":1.5940000000000001,"15min":1.6040000000000001},"min":{"1min":0.627,"5min":0.627,"15min":0.60699999999999998},"max":{"1min":8.2119999999999997,"5min":24.530999999999999,"15min":62.113},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.5640000000000001,"15min":1.5529999999999999},"min":{"1min":0.79300000000000004,"5min":0.72399999999999998,"15min":0.70299999999999996},"max":{"1min":8.2469999999999999,"5min":24.669,"15min":62.146000000000001},"last":1.1379999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:42:35 2021","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.3740000000000001,"15min":1.351},"min":{"1min":0.76000000000000001,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.0139999999999998,"5min":3.3719999999999999,"15min":12.773999999999999},"last":2.4769999999999999},{"interface":"front","average":{"1min":1.464,"5min":1.3169999999999999,"15min":1.2809999999999999},"min":{"1min":0.78800000000000003,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":2.5030000000000001,"5min":2.6709999999999998,"15min":13.038},"last":1.403}]},{"osd":6,"last update":"Wed Feb 10 04:42:46 2021","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.95,"15min":1.762},"min":{"1min":0.82599999999999996,"5min":0.76600000000000001,"15min":0.70299999999999996},"max":{"1min":2.2799999999999998,"5min":42.088999999999999,"15min":42.088999999999999},"last":1.798},{"interface":"front","average":{"1min":1.573,"5min":2.0030000000000001,"15min":1.8660000000000001},"min":{"1min":0.97099999999999997,"5min":0.86799999999999999,"15min":0.83899999999999997},"max":{"1min":2.165,"5min":42.027000000000001,"15min":42.027000000000001},"last":2.6909999999999998}]},{"osd":7,"last update":"Wed Feb 10 04:43:13 2021","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.4710000000000001,"15min":1.431},"min":{"1min":0.874,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":2.2080000000000002,"5min":6.9199999999999999,"15min":6.9199999999999999},"last":1.903},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.5900000000000001,"15min":1.5389999999999999},"min":{"1min":0.93799999999999994,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.9239999999999999,"5min":4.1399999999999997,"15min":4.1399999999999997},"last":2.593}]}]},{"osd":5,"up_from":37,"seq":158913790177,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":257208,"kb_used_data":212208,"kb_used_omap":0,"kb_used_meta":44992,"kb_avail":93487944,"statfs":{"total":95995035648,"available":95731654656,"internally_reserved":0,"allocated":217300992,"data_stored":286041831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":46071808},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:42:31 2021","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.97699999999999998,"15min":1.383},"min":{"1min":0.51800000000000002,"5min":0.47599999999999998,"15min":0.40500000000000003},"max":{"1min":1.5760000000000001,"5min":4.5720000000000001,"15min":104.04300000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.874,"5min":1.1659999999999999,"15min":1.462},"min":{"1min":0.54900000000000004,"5min":0.495,"15min":0.46400000000000002},"max":{"1min":1.851,"5min":15.539999999999999,"15min":107.776},"last":0.93999999999999995}]},{"osd":1,"last update":"Wed Feb 10 04:42:31 2021","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":1.0349999999999999,"15min":1.323},"min":{"1min":0.54700000000000004,"5min":0.54500000000000004,"15min":0.504},"max":{"1min":1.4590000000000001,"5min":5.29,"15min":103.904},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.216,"15min":1.4770000000000001},"min":{"1min":0.63900000000000001,"5min":0.57599999999999996,"15min":0.50800000000000001},"max":{"1min":1.1899999999999999,"5min":11.875999999999999,"15min":104.315},"last":1.0940000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:42:31 2021","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.6379999999999999,"15min":2.2480000000000002},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":2.5779999999999998,"5min":44.462000000000003,"15min":103.65600000000001},"last":1.619},{"interface":"front","average":{"1min":1.246,"5min":1.6659999999999999,"15min":2.2240000000000002},"min":{"1min":0.79100000000000004,"5min":0.65000000000000002,"15min":0.63500000000000001},"max":{"1min":2.637,"5min":44.398000000000003,"15min":104.136},"last":1.0680000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:42:31 2021","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.7370000000000001,"15min":2.2490000000000001},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.624},"max":{"1min":2.7000000000000002,"5min":44.609000000000002,"15min":106.18000000000001},"last":1.7889999999999999},{"interface":"front","average":{"1min":1.357,"5min":1.9350000000000001,"15min":2.242},"min":{"1min":0.98299999999999998,"5min":0.752,"15min":0.64500000000000002},"max":{"1min":1.696,"5min":44.537999999999997,"15min":106.822},"last":1.1830000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:42:31 2021","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.3540000000000001,"15min":1.21},"min":{"1min":0.68400000000000005,"5min":0.57099999999999995,"15min":0.55400000000000005},"max":{"1min":1.7390000000000001,"5min":6.1639999999999997,"15min":6.1639999999999997},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.444,"5min":1.407,"15min":1.361},"min":{"1min":0.872,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":2.895,"5min":2.9830000000000001,"15min":8.1449999999999996},"last":1.9039999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:42:45 2021","interfaces":[{"interface":"back","average":{"1min":1.7070000000000001,"5min":1.6519999999999999,"15min":1.6419999999999999},"min":{"1min":0.77000000000000002,"5min":0.65800000000000003,"15min":0.63500000000000001},"max":{"1min":3.3780000000000001,"5min":6.0300000000000002,"15min":25.969000000000001},"last":2.6549999999999998},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.581,"15min":1.498},"min":{"1min":0.80500000000000005,"5min":0.77900000000000003,"15min":0.74399999999999999},"max":{"1min":3.073,"5min":4.7960000000000003,"15min":25.902999999999999},"last":1.7390000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:43:10 2021","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.4810000000000001,"15min":1.397},"min":{"1min":1.014,"5min":0.752,"15min":0.752},"max":{"1min":2.0910000000000002,"5min":3.0009999999999999,"15min":5.4770000000000003},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.669,"5min":1.6399999999999999,"15min":1.6020000000000001},"min":{"1min":0.98399999999999999,"5min":0.92300000000000004,"15min":0.83499999999999996},"max":{"1min":2.992,"5min":3.1040000000000001,"15min":8.7840000000000007},"last":2.0089999999999999}]}]}]}} 2021-02-10T04:43:42.210 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2021-02-10T04:43:42.210 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:43:42.211 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2021-02-10T04:43:42.211 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:43:42.278 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:43:42.323 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:43:42.558 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:43:42.559 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:43:42.586 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:43:42 smithi085 conmon[52836]: debug 2021-02-10T04:43:42.472+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:43:42.687 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:43:42.728 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:43:42.914 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:42 smithi085 conmon[29319]: cluster 2021-02-10T04:43:41.773328+0000 osd.4 ( 2021-02-10T04:43:42.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:42 smithi085 conmon[29319]: osd.4) 1757 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:42.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:42 smithi085 conmon[29319]: cluster 2021-02-10T04:43:41.773339+0000 osd.4 (osd.4) 1758 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:42.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:42 smithi085 conmon[29319]: cluster 2021-02-10T04:43:41.773345+0000 osd.4 (osd.4) 1759 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:42.915 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:42 smithi085 conmon[29319]: cluster 2021-02-10T04:43:41.773351+0000 osd.4 (osd.4) 1760 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:42.915 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:42 smithi085 conmon[35961]: cluster 2021-02-10T04:43:41.773328+0000 2021-02-10T04:43:42.915 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:42 smithi085 conmon[35961]: osd.4 (osd.4) 1757 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:42.915 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:42 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:43:42.915 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:42 smithi085 conmon[35961]: T04:43:41.773339+0000 osd.4 (osd.4) 1758 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:42.915 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:42 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:43:42.915 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:42 smithi085 conmon[35961]: 02-10T04:43:41.773345+0000 2021-02-10T04:43:42.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:42 smithi085 conmon[35961]: osd.4 (osd.4) 1759 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:42.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:42 smithi085 conmon[35961]: 2021-02-10T04:43:42.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:42 smithi085 conmon[35961]: cluster 2021-02-10T04:43:42.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:42 smithi085 conmon[35961]: 2021-02-10T04:43:41.773351+0000 osd.4 (osd 2021-02-10T04:43:42.916 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:42 smithi085 conmon[35961]: .4) 1760 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:43.063 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:43:43.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:42 smithi164 conmon[33677]: debug 2021-02-10T04:43:42.740+0000 7fbec19a1700 -1 osd.4 996 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:43:43.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:42 smithi164 conmon[29288]: cluster 2021-02-10T04:43:43.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:42 smithi164 conmon[29288]: 2021-02 2021-02-10T04:43:43.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:42 smithi164 conmon[29288]: -10T04:43:41.773328+0000 osd.4 (osd.4) 1757 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:43.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:42 smithi164 conmon[29288]: cluster 2021-02-10T04:43:41.773339+0000 osd.4 ( 2021-02-10T04:43:43.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:42 smithi164 conmon[29288]: osd.4) 1758 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:43.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:42 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:43:43.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:42 smithi164 conmon[29288]: 02-10T04:43:41.773345+0000 osd.4 (osd 2021-02-10T04:43:43.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:42 smithi164 conmon[29288]: .4) 1759 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:43.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:42 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:43:43.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:42 smithi164 conmon[29288]: -02-10T04:43:41.773351+0000 2021-02-10T04:43:43.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:42 smithi164 conmon[29288]: osd.4 (osd.4) 1760 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:43.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:42 smithi164 conmon[29288]: 2021-02-10T04:43:43.168 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:43:43.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[52836]: debug 2021-02-10T04:43:43.027+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:43:43.201 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:43:43.535 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:43:43.559 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:43:43.575 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:43:43.580 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:43:43 smithi164 conmon[44544]: debug 2021-02-10T04:43:43.344+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:43:43.595 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:43:43.627 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:43:43.660 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:43:43.681 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:43:43.692 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:43:43.718 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:43:43.721 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:43:43.750 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:43:43.752 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:43:43.752 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:43:43.752 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:43:43.752 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:43:43.752 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:43:43.752 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:43:43.752 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:43:43.752 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:43:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:43 smithi164 conmon[29288]: cluster 2021-02-10T04:43:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:43 smithi164 conmon[29288]: 2021-02-10T04:43 2021-02-10T04:43:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:43 smithi164 conmon[29288]: :42.260948+0000 mgr.y (mgr.14140) 911 : cluster [DBG] pgmap v1648: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 879 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 776 B/s rd, 1.4 MiB/s wr, 2 op/s 2021-02-10T04:43:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:43 smithi164 conmon[29288]: cluster 2021-02-10T04:43:42.741398 2021-02-10T04:43:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:43 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1761 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:43 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:43:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:43 smithi164 conmon[29288]: 04:43:42.741406+0000 osd.4 (osd.4) 2021-02-10T04:43:43.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:43 smithi164 conmon[29288]: 1762 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:43.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:43 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:43:43.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:43 smithi164 conmon[29288]: T04:43:42.741413+0000 osd.4 2021-02-10T04:43:43.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:43 smithi164 conmon[29288]: (osd.4) 1763 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:43.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:43 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:43:43.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:43 smithi164 conmon[29288]: -02-10T04:43:42.741419+0000 2021-02-10T04:43:43.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:43 smithi164 conmon[29288]: osd.4 (osd.4) 1764 : cluster [WRN] 2021-02-10T04:43:43.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:43 smithi164 conmon[29288]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:43.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:43 smithi164 conmon[29288]: cluster 2021-02-10T04:43:43.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:43 smithi164 conmon[29288]: 2021-02-10T04:43:42.910523+0000 osd.7 (osd.7) 68 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:43:43.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:43 smithi164 conmon[29288]: 2021-02-10T04:43:43.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:43 smithi164 conmon[29288]: cluster 2021-02-10T04:43:42 2021-02-10T04:43:43.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:43 smithi164 conmon[29288]: .911286+0000 osd.7 (osd.7) 69 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:43:43.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:43 smithi164 conmon[29288]: cluster 2021-02-10T04:43:43.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:43 smithi164 conmon[29288]: 2021-02-10T04:43:43.337966+0000 mon.a (mon.0) 1992 : cluster [DBG] 2021-02-10T04:43:43.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:43 smithi164 conmon[29288]: osdmap e997: 8 total, 8 up, 7 in 2021-02-10T04:43:43.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:43 smithi164 conmon[29288]: audit 2021-02-10T04:43: 2021-02-10T04:43:43.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:43 smithi164 conmon[29288]: 43.500683+0000 mon.c (mon.1) 70 : audit [DBG] from='client.? 172.21.15.85:0/260641147' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:43:44.017 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:43:44.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[49106]: debug 2021-02-10T04:43:43.787+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:43:44.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[29319]: cluster 2021-02-10T04:43:44.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[29319]: 2021-02-10T04:43:44.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[29319]: 2021-02-10T04:43:42. 2021-02-10T04:43:44.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[29319]: 260948+0000 mgr.y (mgr.14140) 911 : cluster [DBG] pgmap v1648: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 879 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 776 B/s rd, 1.4 MiB/s wr, 2 op/s 2021-02-10T04:43:44.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:43:44.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[29319]: -02-10T04:43:42.741398+0000 2021-02-10T04:43:44.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[29319]: osd.4 (osd.4) 1761 : cluster [WRN] 2021-02-10T04:43:44.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:44.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[29319]: cluster 2021-02-10T04:43: 2021-02-10T04:43:44.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[29319]: 42.741406+0000 osd.4 (osd.4) 1762 : 2021-02-10T04:43:44.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:44.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:43:44.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[29319]: :43:42.741413+0000 osd.4 (osd.4 2021-02-10T04:43:44.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[29319]: ) 1763 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:44.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:43:44.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[29319]: 43:42.741419+0000 osd.4 (osd.4) 1764 2021-02-10T04:43:44.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:44.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[29319]: cluster 2021-02-10T04:43: 2021-02-10T04:43:44.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[29319]: 42.910523+0000 osd.7 (osd.7 2021-02-10T04:43:44.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[29319]: ) 68 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:43:44.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:43:44.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[29319]: -10T04:43:42.911286+0000 osd.7 ( 2021-02-10T04:43:44.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[29319]: osd.7) 69 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:43:44.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[29319]: cluster 2021-02-10T04:43:44.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[29319]: 2021-02-10T04:43:43.337966+0000 2021-02-10T04:43:44.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[29319]: mon.a (mon.0) 1992 : cluster [DBG] osdmap e997: 8 total, 8 up, 7 in 2021-02-10T04:43:44.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[29319]: audit 2021-02-10T04:43:43 2021-02-10T04:43:44.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[29319]: .500683+0000 mon.c (mon.1) 70 : audit [DBG] from='client.? 172.21.15.85:0/260641147' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:43:44.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[29319]: 2021-02-10T04:43:44.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: cluster 2021-02-10T04:43:44.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: 2021-02-10T04:43:44.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: 2021 2021-02-10T04:43:44.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: - 2021-02-10T04:43:44.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: 02 2021-02-10T04:43:44.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: - 2021-02-10T04:43:44.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: 10 2021-02-10T04:43:44.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: T 2021-02-10T04:43:44.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: 04:43:42.260948+0000 2021-02-10T04:43:44.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: mgr.y (mgr.14140) 911 : cluster [DBG] pgmap v1648: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 879 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 776 B/s rd, 1.4 MiB/s wr, 2 op/s 2021-02-10T04:43:44.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: cluster 2021-02-10T04:43:44.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:43:44.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: T04:43:42.741398+0000 osd.4 (osd.4 2021-02-10T04:43:44.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: ) 1761 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:43:44.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: cluster 2021-02-10T04:43:44.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:43:44.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: 10T04:43:42.741406+0000 2021-02-10T04:43:44.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: osd.4 (osd.4) 1762 : 2021-02-10T04:43:44.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:43:44.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:43:44.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: 02-10T04:43:42. 2021-02-10T04:43:44.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: 741413+0000 osd.4 (osd.4) 2021-02-10T04:43:44.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: 1763 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:43:44.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: cluster 2021-02-10T04:43:44.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: 2021-02-10T04:43:42. 2021-02-10T04:43:44.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: 741419+0000 osd.4 (osd.4) 1764 : cluster 2021-02-10T04:43:44.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:43:44.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:43:44.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: :43:42.910523+0000 osd.7 (osd.7 2021-02-10T04:43:44.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: ) 68 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:43:44.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:43:44.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: -10T04:43:42.911286+0000 osd.7 ( 2021-02-10T04:43:44.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: osd.7) 69 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:43:44.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:43:44.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: 10T04:43:43.337966+0000 mon.a (mon 2021-02-10T04:43:44.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: .0) 1992 : cluster [DBG] osdmap e997: 8 total, 8 up, 7 in 2021-02-10T04:43:44.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:43:44.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: -10T04:43:43.500683+0000 mon.c (mon 2021-02-10T04:43:44.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:43 smithi085 conmon[35961]: .1) 70 : audit [DBG] from='client.? 172.21.15.85:0/260641147' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:43:44.757 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is unset 2021-02-10T04:43:45.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:44 smithi164 conmon[29288]: audit 2021-02-10T04:43:43.799967+0000 2021-02-10T04:43:45.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:44 smithi164 conmon[29288]: mon.c (mon.1) 71 : audit [INF] from='client.? 172.21.15.85:0/2123608495' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:43:45.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:44 smithi164 conmon[29288]: audit 2021-02-10T04:43:45.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:44 smithi164 conmon[29288]: 2021-02-10T04:43:43.800354+0000 2021-02-10T04:43:45.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:44 smithi164 conmon[29288]: mon.a (mon.0) 1993 : audit 2021-02-10T04:43:45.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:44 smithi164 conmon[29288]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:43:45.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:44 smithi085 conmon[35961]: audit 2021-02-10T04:43:43. 2021-02-10T04:43:45.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:44 smithi085 conmon[35961]: 799967+0000 mon.c (mon.1) 71 : audit [INF] from='client.? 172.21.15.85:0/2123608495' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:43:45.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:44 smithi085 conmon[35961]: audit 2021-02-10T04:43:43 2021-02-10T04:43:45.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:44 smithi085 conmon[35961]: .800354+0000 mon.a (mon.0) 1993 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:43:45.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:44 smithi085 conmon[29319]: audit 2021-02-10T04:43:45.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:44 smithi085 conmon[29319]: 2021-02-10T04:43:45.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:44 smithi085 conmon[29319]: 2021 2021-02-10T04:43:45.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:44 smithi085 conmon[29319]: -02-10T04:43: 2021-02-10T04:43:45.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:44 smithi085 conmon[29319]: 43.799967+0000 mon.c (mon.1) 71 : audit [INF] from='client.? 172.21.15.85:0/2123608495' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:43:45.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:44 smithi085 conmon[29319]: audit 2021 2021-02-10T04:43:45.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:44 smithi085 conmon[29319]: -02-10T04:43:43.800354+0000 mon.a 2021-02-10T04:43:45.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:44 smithi085 conmon[29319]: (mon.0) 1993 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:43:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:45 smithi164 conmon[29288]: cluster 2021-02-10T04:43:44.261685+0000 mgr.y ( 2021-02-10T04:43:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:45 smithi164 conmon[29288]: mgr.14140) 912 : cluster [DBG] pgmap v1650: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 879 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 336 KiB/s wr, 1 op/s 2021-02-10T04:43:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:45 smithi164 conmon[29288]: cluster 2021-02-10T04:43:44.739923+0000 mon.a (mon.0) 1994 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:43:46.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:45 smithi164 conmon[29288]: audit 2021-02-10T04:43:44.750158+0000 mon.a (mon.0) 1995 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:43:46.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:45 smithi164 conmon[29288]: cluster 2021-02-10T04:43:44.750230+0000 mon.a (mon.0) 1996 : cluster [DBG] osdmap e998: 8 total, 8 up, 7 in 2021-02-10T04:43:46.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:45 smithi085 conmon[29319]: cluster 2021-02-10T04:43:46.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:45 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:43:46.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:45 smithi085 conmon[29319]: :43:44.261685+0000 mgr.y (mgr.14140) 912 : cluster [DBG] pgmap v1650: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 879 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 336 KiB/s wr, 1 op/s 2021-02-10T04:43:46.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:45 smithi085 conmon[29319]: cluster 2021-02-10T04:43:44.739923+0000 2021-02-10T04:43:46.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:45 smithi085 conmon[29319]: mon.a (mon.0) 1994 : cluster [WRN] 2021-02-10T04:43:46.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:45 smithi085 conmon[29319]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:43:46.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:45 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:43:46.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:45 smithi085 conmon[29319]: 10T04:43:44.750158+0000 mon.a (mon.0) 1995 : audit [INF] 2021-02-10T04:43:46.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:45 smithi085 conmon[29319]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:43:46.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:45 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:43:46.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:45 smithi085 conmon[29319]: 43:44.750230+0000 mon.a (mon.0) 1996 : cluster [DBG] osdmap e998: 8 total, 8 up, 7 in 2021-02-10T04:43:46.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:45 smithi085 conmon[35961]: cluster 2021-02-10T04:43:44.261685+0000 mgr.y (mgr. 2021-02-10T04:43:46.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:45 smithi085 conmon[35961]: 14140) 912 : cluster [DBG] pgmap v1650: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 879 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 336 KiB/s wr, 1 op/s 2021-02-10T04:43:46.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:45 smithi085 conmon[35961]: cluster 2021-02-10T04:43:44.739923+0000 mon.a (mon 2021-02-10T04:43:46.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:45 smithi085 conmon[35961]: .0) 1994 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:43:46.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:45 smithi085 conmon[35961]: audit 2021 2021-02-10T04:43:46.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:45 smithi085 conmon[35961]: -02-10T04:43:44.750158+0000 mon.a (mon.0) 1995 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:43:46.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:45 smithi085 conmon[35961]: cluster 2021-02-10T04:43:44.750230+0000 mon.a ( 2021-02-10T04:43:46.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:45 smithi085 conmon[35961]: mon.0) 1996 : cluster [DBG] osdmap e998: 8 total, 8 up, 7 in 2021-02-10T04:43:46.233 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:43:46.233 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:43:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:47 smithi164 conmon[29288]: cluster 2021-02-10T04:43:46.262505+0000 mgr.y (mgr.14140) 913 : cluster [DBG] pgmap v1652: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 879 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:43:48.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:47 smithi085 conmon[29319]: cluster 2021-02-10T04:43:46.262505+0000 mgr.y (mgr.14140 2021-02-10T04:43:48.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:47 smithi085 conmon[29319]: ) 913 : cluster [DBG] pgmap v1652: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 879 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:43:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:47 smithi085 conmon[35961]: cluster 2021-02-10T04:43:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:47 smithi085 conmon[35961]: 2021-02-10T04:43:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:47 smithi085 conmon[35961]: 2021 2021-02-10T04:43:48.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:47 smithi085 conmon[35961]: -02-10T04:43:46 2021-02-10T04:43:48.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:47 smithi085 conmon[35961]: .262505+0000 mgr.y (mgr.14140) 913 : cluster [DBG] pgmap v1652: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 879 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:43:48.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:47 smithi085 conmon[35961]: 2021-02-10T04:43:48.894 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:43:48 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:43:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:48 smithi164 conmon[29288]: cluster 2021-02-10T04:43:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:48 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:43:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:48 smithi164 conmon[29288]: :43:48.333505+0000 mon.a (mon.0) 1997 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 852 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:43:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:48 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:43:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:48 smithi164 conmon[29288]: 04:43:48.341222+0000 mon.a (mon.0) 2021-02-10T04:43:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:48 smithi164 conmon[29288]: 1998 : cluster [DBG] osdmap e999: 8 total, 8 up, 7 in 2021-02-10T04:43:49.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:48 smithi085 conmon[29319]: cluster 2021-02-10T04:43:48.333505+0000 mon.a (mon 2021-02-10T04:43:49.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:48 smithi085 conmon[29319]: .0) 1997 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 852 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:43:49.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:48 smithi085 conmon[29319]: cluster 2021-02-10T04:43:48.341222+0000 mon.a (mon.0) 1998 : cluster [DBG] osdmap e999: 8 total, 8 up, 7 in 2021-02-10T04:43:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:48 smithi085 conmon[35961]: cluster 2021-02-10T04:43 2021-02-10T04:43:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:48 smithi085 conmon[35961]: :48.333505+0000 mon.a (mon.0) 1997 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 852 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:43:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:48 smithi085 conmon[35961]: cluster 2021-02-10T04:43:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:48 smithi085 conmon[35961]: 2021-02-10T04:43:48 2021-02-10T04:43:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:48 smithi085 conmon[35961]: .341222+0000 mon.a (mon.0) 1998 : cluster 2021-02-10T04:43:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:48 smithi085 conmon[35961]: [DBG] osdmap e999: 8 total, 8 up, 7 in 2021-02-10T04:43:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:49 smithi164 conmon[29288]: cluster 2021-02-10T04:43:48.263304+0000 mgr.y (mgr.14140 2021-02-10T04:43:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:49 smithi164 conmon[29288]: ) 914 : cluster [DBG] pgmap v1653: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 879 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:43:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:49 smithi164 conmon[29288]: audit 2021-02-10T04:43:48.601067+0000 mgr.y (mgr.14140) 915 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:43:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:49 smithi164 conmon[29288]: cluster 2021-02-10T04:43:48.747696+0000 mon.a (mon.0) 1999 2021-02-10T04:43:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:49 smithi164 conmon[29288]: : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 852 sec, osd.4 has slow ops) 2021-02-10T04:43:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:49 smithi164 conmon[29288]: cluster 2021-02-10T04:43:48.810625+0000 2021-02-10T04:43:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:49 smithi164 conmon[29288]: osd.4 (osd.4) 1765 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:43:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:49 smithi164 conmon[29288]: cluster 2021-02-10T04:43 2021-02-10T04:43:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:49 smithi164 conmon[29288]: :48.811385+0000 osd.4 (osd.4) 1766 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:43:50.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[29319]: cluster 2021-02-10T04:43:48.263304+0000 mgr.y (mgr 2021-02-10T04:43:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[29319]: .14140) 914 : cluster [DBG] pgmap v1653: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 879 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:43:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[29319]: audit 2021-02-10T04:43:48.601067+0000 mgr.y (mgr.14140) 915 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:43:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:43:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[29319]: 02-10T04:43:48.747696+0000 mon.a (mon.0) 1999 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 852 sec, osd.4 has slow ops) 2021-02-10T04:43:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:43:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[29319]: 10T04: 2021-02-10T04:43:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[29319]: 43:48.810625+0000 osd.4 (osd.4) 1765 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:43:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[29319]: 2021-02-10T04:43:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:43:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[29319]: 02-10T04:43:48.811385+0000 osd.4 (osd.4) 1766 : cluster [DBG] 2021-02-10T04:43:50.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[29319]: purged_snaps scrub ok 2021-02-10T04:43:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[35961]: cluster 2021-02-10T04:43:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[35961]: 2021-02-10T04:43:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[35961]: 2021 2021-02-10T04:43:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[35961]: - 2021-02-10T04:43:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[35961]: 02-10T04:43:48. 2021-02-10T04:43:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[35961]: 263304+0000 mgr.y (mgr.14140) 914 : cluster [DBG] pgmap v1653: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 879 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:43:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:43:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[35961]: 02-10T04:43:48. 2021-02-10T04:43:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[35961]: 601067+0000 mgr.y (mgr.14140 2021-02-10T04:43:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[35961]: ) 915 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:43:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:43:50.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[35961]: -02-10T04:43:48 2021-02-10T04:43:50.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[35961]: .747696+0000 mon.a (mon. 2021-02-10T04:43:50.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[35961]: 0) 1999 : cluster [INF] 2021-02-10T04:43:50.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[35961]: Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 852 sec, osd.4 has slow ops) 2021-02-10T04:43:50.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[35961]: cluster 2021-02-10T04:43:48 2021-02-10T04:43:50.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[35961]: .810625+0000 osd.4 (osd.4) 1765 : cluster 2021-02-10T04:43:50.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[35961]: [DBG] purged_snaps scrub starts 2021-02-10T04:43:50.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:43:50.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[35961]: -10T04:43:48.811385+0000 2021-02-10T04:43:50.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[35961]: osd.4 (osd.4) 1766 : cluster 2021-02-10T04:43:50.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:49 smithi085 conmon[35961]: [DBG] purged_snaps scrub ok 2021-02-10T04:43:51.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:50 smithi164 conmon[29288]: cluster 2021-02-10T04:43:49.757218+0000 2021-02-10T04:43:51.716 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:50 smithi164 conmon[29288]: mon.a (mon.0) 2000 : cluster [DBG] osdmap e1000: 8 total, 8 up, 7 in 2021-02-10T04:43:51.717 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:50 smithi085 conmon[29319]: cluster 2021-02-10T04:43:49.757218+0000 2021-02-10T04:43:51.717 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:50 smithi085 conmon[29319]: mon.a (mon.0) 2000 : cluster [DBG] osdmap e1000: 8 total, 8 up, 7 in 2021-02-10T04:43:51.717 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:50 smithi085 conmon[35961]: cluster 2021-02-10T04:43:49. 2021-02-10T04:43:51.717 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:50 smithi085 conmon[35961]: 757218+0000 mon.a (mon.0) 2000 : cluster [DBG] osdmap e1000: 8 total, 8 up, 7 in 2021-02-10T04:43:52.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:51 smithi164 conmon[29288]: cluster 2021-02-10T04:43:52.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:51 smithi164 conmon[29288]: 2021-02-10T04:43:52.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:51 smithi164 conmon[29288]: 2021-02-10T04:43:50 2021-02-10T04:43:52.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:51 smithi164 conmon[29288]: .264147+0000 mgr.y (mgr.14140) 916 : cluster [DBG] pgmap v1656: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 879 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:43:52.165 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:51 smithi085 conmon[35961]: cluster 2021-02-10T04:43:50.264147 2021-02-10T04:43:52.166 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:51 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 916 : cluster [DBG] pgmap v1656: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 879 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:43:52.166 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:51 smithi085 conmon[29319]: cluster 2021-02-10T04:43:50.264147+0000 mgr.y (mgr.14140 2021-02-10T04:43:52.166 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:51 smithi085 conmon[29319]: ) 916 : cluster [DBG] pgmap v1656: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 879 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:43:52.268 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:43:52.307 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:43:52.558 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:43:52.562 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:43:52 smithi164 conmon[33677]: debug 2021-02-10T04:43:52.404+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:43:52.664 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:43:52.698 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:43:52.806 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.4 dump_blocked_ops 2021-02-10T04:43:53.125 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:43:53.158 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:43:53.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:43:52 smithi085 conmon[52836]: debug 2021-02-10T04:43:52.879+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:43:54.072 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:53 smithi164 conmon[29288]: cluster 2021-02-10T04:43:52.264854+0000 mgr.y (mgr.14140) 917 : cluster [DBG] pgmap v1657: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 879 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 681 B/s rd, 0 op/s 2021-02-10T04:43:54.072 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:53 smithi164 conmon[29288]: cluster 2021-02-10T04:43:53.340739+0000 mon.a (mon.0) 2001 : cluster [DBG] osdmap e1001: 8 total, 8 up, 7 in 2021-02-10T04:43:54.072 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:43:53 smithi164 conmon[37263]: debug 2021-02-10T04:43:53.724+0000 7f1c671e2700 -1 osd.5 pg_epoch: 1001 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55665699f1e0) 2021-02-10T04:43:54.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:53 smithi085 conmon[29319]: cluster 2021-02-10T04:43:52.264854 2021-02-10T04:43:54.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:53 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 917 : cluster [DBG] pgmap v1657: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 879 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 681 B/s rd, 0 op/s 2021-02-10T04:43:54.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:53 smithi085 conmon[29319]: cluster 2021-02-10T04:43:53.340739+0000 mon.a (mon.0) 2001 : cluster [DBG] osdmap e1001: 8 total, 8 up, 7 in 2021-02-10T04:43:54.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:53 smithi085 conmon[35961]: cluster 2021-02-10T04:43:54.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:53 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:43:54.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:53 smithi085 conmon[35961]: 10T04:43:52.264854+0000 mgr.y (mgr.14140) 917 : cluster [DBG] pgmap v1657: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 879 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 681 B/s rd, 0 op/s 2021-02-10T04:43:54.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:53 smithi085 conmon[35961]: 2021-02-10T04:43:54.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:53 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:43:54.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:53 smithi085 conmon[35961]: 43:53.340739+0000 mon.a (mon.0) 2021-02-10T04:43:54.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:53 smithi085 conmon[35961]: 2001 : cluster [DBG] osdmap e1001: 8 total, 8 up, 7 in 2021-02-10T04:43:54.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:53 smithi085 conmon[35961]: 2021-02-10T04:43:56.083 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:55 smithi085 conmon[29319]: cluster 2021-02-10T04:43:54.265432+0000 mgr.y ( 2021-02-10T04:43:56.083 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:55 smithi085 conmon[29319]: mgr.14140) 918 : cluster [DBG] pgmap v1659: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 871 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2021-02-10T04:43:56.083 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:55 smithi085 conmon[29319]: cluster 2021-02-10T04:43:54.560429+0000 osd.1 (osd.1) 465 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:43:56.083 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:55 smithi085 conmon[29319]: cluster 2021-02-10T04:43:54.561795+0000 osd.1 (osd.1) 2021-02-10T04:43:56.083 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:55 smithi085 conmon[29319]: 466 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:43:56.084 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:55 smithi085 conmon[29319]: cluster 2021-02-10T04:43:54.775320 2021-02-10T04:43:56.084 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:55 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2002 : cluster [DBG] osdmap e1002: 8 total, 8 up, 7 in 2021-02-10T04:43:56.084 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:43:56 smithi085 conmon[41540]: debug 2021-02-10T04:43:56.084 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:55 smithi085 conmon[35961]: cluster 2021-02-10T04:43:54.265432+0000 2021-02-10T04:43:56.084 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:55 smithi085 conmon[35961]: mgr.y (mgr.14140) 918 : cluster [DBG] pgmap v1659: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 871 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2021-02-10T04:43:56.084 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:55 smithi085 conmon[35961]: cluster 2021-02-10T04:43:54.560429 2021-02-10T04:43:56.084 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:55 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 465 : 2021-02-10T04:43:56.084 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:55 smithi085 conmon[35961]: cluster [DBG] purged_snaps scrub starts 2021-02-10T04:43:56.085 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:55 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:43:56.085 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:55 smithi085 conmon[35961]: 02-10T04:43:54.561795+0000 2021-02-10T04:43:56.085 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:55 smithi085 conmon[35961]: osd.1 (osd.1) 466 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:43:56.085 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:55 smithi085 conmon[35961]: 2021-02-10T04:43:56.085 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:55 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:43:56.085 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:55 smithi085 conmon[35961]: 43:54.775320+0000 mon.a (mon.0 2021-02-10T04:43:56.085 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:55 smithi085 conmon[35961]: ) 2002 : cluster [DBG] osdmap e1002: 8 total, 8 up, 7 in 2021-02-10T04:43:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:55 smithi164 conmon[29288]: cluster 2021-02-10T04:43:54.265432+0000 mgr.y (mgr.14140 2021-02-10T04:43:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:55 smithi164 conmon[29288]: ) 918 : cluster [DBG] pgmap v1659: 168 pgs: 1 active+clean+scrubbing, 6 active+clean+snaptrim_wait, 153 active+clean, 8 active+clean+snaptrim; 871 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2021-02-10T04:43:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:55 smithi164 conmon[29288]: cluster 2021-02-10T04:43:54.560429+0000 osd.1 (osd.1) 465 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:43:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:55 smithi164 conmon[29288]: 2021-02-10T04:43:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:55 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:43:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:55 smithi164 conmon[29288]: 04:43:54.561795+0000 osd.1 (osd.1) 466 : 2021-02-10T04:43:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:55 smithi164 conmon[29288]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:43:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:55 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:43:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:55 smithi164 conmon[29288]: -10T04:43:54.775320+0000 mon.a 2021-02-10T04:43:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:55 smithi164 conmon[29288]: (mon.0) 2002 : cluster [DBG] osdmap e1002: 8 total, 8 up, 7 in 2021-02-10T04:43:56.145 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:43:55 smithi164 conmon[37263]: debug 2021-02-10T04:43:56.145 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:43:55 smithi164 conmon[37263]: 2021-02-10T04:43:55.667+0000 7f1c671e2700 -1 osd.5 pg_epoch: 1002 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x556652d7bce0) 2021-02-10T04:43:56.145 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:43:55 smithi164 conmon[37263]: 2021-02-10T04:43:56.441 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:43:56 smithi085 conmon[41540]: 2021-02-10T04:43:56.081+0000 7fb947b5b700 -1 osd.0 pg_epoch: 1002 pg[5.5( empty local-lis/les=57/58 n=0 ec=57/57 lis/c=57/57 les/c/f=58/58/0 sis=57) [0,1,4] r=0 lpr=57 crt=0'0 mlcod 0'0 active+clean] scrubber pg(5.5) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x555d327ef760) 2021-02-10T04:43:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:57 smithi164 conmon[29288]: cluster 2021-02-10T04:43:56.266129+0000 2021-02-10T04:43:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:57 smithi164 conmon[29288]: mgr.y (mgr.14140) 919 : cluster [DBG] pgmap v1661: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim_wait, 156 active+clean, 8 active+clean+snaptrim; 867 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 786 B/s rd, 0 op/s 2021-02-10T04:43:58.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:57 smithi085 conmon[35961]: cluster 2021-02-10T04:43:56.266129+0000 mgr.y 2021-02-10T04:43:58.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:57 smithi085 conmon[35961]: (mgr.14140) 919 : cluster [DBG] pgmap v1661: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim_wait, 156 active+clean, 8 active+clean+snaptrim; 867 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 786 B/s rd, 0 op/s 2021-02-10T04:43:58.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:57 smithi085 conmon[29319]: cluster 2021-02-10T04:43:56.266129+0000 mgr.y (mgr 2021-02-10T04:43:58.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:57 smithi085 conmon[29319]: .14140) 919 : cluster [DBG] pgmap v1661: 168 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim_wait, 156 active+clean, 8 active+clean+snaptrim; 867 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 786 B/s rd, 0 op/s 2021-02-10T04:43:58.894 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:43:58 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:43:59.333 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:43:59.496 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:43:59.585 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[29319]: cluster 2021-02-10T04:43:58.266874+0000 2021-02-10T04:43:59.585 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[29319]: mgr.y (mgr.14140) 920 : cluster [DBG] pgmap v1662: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim_wait, 158 active+clean, 7 active+clean+snaptrim; 865 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:43:59.585 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[29319]: cluster 2021-02-10T04:43:58.342150+0000 mon.a (mon. 2021-02-10T04:43:59.585 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[29319]: 0) 2003 : cluster [DBG] osdmap e1003: 8 total, 8 up, 7 in 2021-02-10T04:43:59.585 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[29319]: audit 2021-02-10T04:43:59.586 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[29319]: 2021-02-10T04:43:58.611048+0000 mgr.y (mgr.14140) 921 : audit 2021-02-10T04:43:59.586 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[29319]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:43:59.586 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[29319]: cephadm 2021-02-10T04:43:58 2021-02-10T04:43:59.586 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[29319]: .880312+0000 mgr.y (mgr.14140) 922 : 2021-02-10T04:43:59.586 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[29319]: cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:43:59.586 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[29319]: cephadm 2021-02-10T04:43: 2021-02-10T04:43:59.586 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[29319]: 58.880826+0000 mgr.y (mgr.14140) 923 2021-02-10T04:43:59.586 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[29319]: : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:43:59.586 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[35961]: cluster 2021-02-10T04:43: 2021-02-10T04:43:59.586 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[35961]: 58.266874+0000 mgr.y (mgr.14140) 920 : cluster [DBG] 2021-02-10T04:43:59.586 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[35961]: pgmap v1662: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim_wait, 158 active+clean, 7 active+clean+snaptrim; 865 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:43:59.586 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:43:59.587 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[35961]: 02-10T04:43:58.342150+0000 mon.a (mon 2021-02-10T04:43:59.587 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[35961]: .0) 2003 : cluster [DBG] osdmap e1003: 8 total, 8 up, 7 in 2021-02-10T04:43:59.587 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:43:59.587 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[35961]: 02-10T04:43:58.611048+0000 mgr.y ( 2021-02-10T04:43:59.587 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[35961]: mgr.14140) 921 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:43:59.587 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[35961]: cephadm 2021- 2021-02-10T04:43:59.587 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[35961]: 02-10T04:43:58.880312 2021-02-10T04:43:59.587 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 922 2021-02-10T04:43:59.587 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[35961]: : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:43:59.587 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[35961]: cephadm 2021- 2021-02-10T04:43:59.587 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[35961]: 02-10T04:43:58 2021-02-10T04:43:59.587 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[35961]: .880826+0000 mgr.y (mgr. 2021-02-10T04:43:59.587 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[35961]: 14140) 923 : cephadm [INF] 2021-02-10T04:43:59.588 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:43:59 smithi085 conmon[35961]: Refreshing smithi164 facts 2021-02-10T04:43:59.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:59 smithi164 conmon[29288]: cluster 2021-02-10T04:43:59.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:59 smithi164 conmon[29288]: 2021-02-10T04:43:59.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:59 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:43:59.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:59 smithi164 conmon[29288]: 43:58.266874+0000 mgr.y (mgr.14140) 920 : cluster [DBG] pgmap v1662: 168 pgs: 1 active+clean+scrubbing, 2 active+clean+snaptrim_wait, 158 active+clean, 7 active+clean+snaptrim; 865 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:43:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:59 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:43:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:59 smithi164 conmon[29288]: -10T04:43:58.342150+0000 mon.a (mon.0 2021-02-10T04:43:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:59 smithi164 conmon[29288]: ) 2003 : cluster [DBG] osdmap e1003: 8 total, 8 up, 7 in 2021-02-10T04:43:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:59 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:43:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:59 smithi164 conmon[29288]: 02-10T04:43:58.611048+0000 mgr.y 2021-02-10T04:43:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:59 smithi164 conmon[29288]: (mgr.14140) 921 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:43:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:59 smithi164 conmon[29288]: cephadm 2021 2021-02-10T04:43:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:59 smithi164 conmon[29288]: -02-10T04:43:58.880312+0000 2021-02-10T04:43:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:59 smithi164 conmon[29288]: mgr.y (mgr.14140) 922 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:43:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:59 smithi164 conmon[29288]: 2021-02-10T04:43:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:59 smithi164 conmon[29288]: cephadm 2021-02-10T04:43:59.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:59 smithi164 conmon[29288]: 2021-02-10T04:43:58.880826+0000 mgr.y ( 2021-02-10T04:43:59.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:43:59 smithi164 conmon[29288]: mgr.14140) 923 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:44:00.314 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:44:00.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:43:59 smithi164 conmon[37263]: debug 2021-02-10T04:43:59.918+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:44:00.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:00 smithi085 conmon[35961]: audit 2021-02-10T04:43:59.432018+0000 mon.a (mon 2021-02-10T04:44:00.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:00 smithi085 conmon[35961]: .0) 2004 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:44:00.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:00 smithi085 conmon[29319]: audit 2021-02-10T04:43:59.432018+0000 2021-02-10T04:44:00.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:00 smithi085 conmon[29319]: mon.a (mon.0) 2004 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:44:00.828 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:44:00.837 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:44:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:00 smithi164 conmon[29288]: audit 2021-02-10T04:43:59.432018+0000 mon.a ( 2021-02-10T04:44:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:00 smithi164 conmon[29288]: mon.0) 2004 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:44:01.647 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:44:01.689 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset nodeep-scrub 2021-02-10T04:44:01.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:01 smithi085 conmon[29319]: cluster 2021-02-10T04:44:00.267776+0000 mgr.y 2021-02-10T04:44:01.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:01 smithi085 conmon[29319]: (mgr.14140) 924 : cluster [DBG] pgmap v1664: 168 pgs: 1 active+clean+scrubbing, 164 active+clean, 3 active+clean+snaptrim; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-02-10T04:44:01.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:01 smithi085 conmon[29319]: audit 2021-02-10T04:44:01.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:01 smithi085 conmon[29319]: 2021-02-10T04:44:00.799714+0000 mgr.y (mgr.14140) 925 : audit [DBG] from='client.25285 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:44:01.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:01 smithi085 conmon[35961]: cluster 2021-02-10T04:44:01.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:01 smithi085 conmon[35961]: 2021-02-10T04:44:00.267776+0000 mgr.y (mgr.14140) 924 : cluster [DBG] pgmap v1664: 168 pgs: 1 active+clean+scrubbing, 164 active+clean, 3 active+clean+snaptrim; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-02-10T04:44:01.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:01 smithi085 conmon[35961]: audit 2021-02-10T04:44:00.799714+0000 mgr.y 2021-02-10T04:44:01.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:01 smithi085 conmon[35961]: (mgr.14140) 925 : audit [DBG] from='client.25285 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:44:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:01 smithi164 conmon[29288]: cluster 2021-02-10T04:44:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:01 smithi164 conmon[29288]: 2021- 2021-02-10T04:44:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:01 smithi164 conmon[29288]: 02-10T04:44:00.267776+0000 2021-02-10T04:44:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:01 smithi164 conmon[29288]: mgr.y (mgr.14140) 924 : cluster [DBG] pgmap v1664: 168 pgs: 1 active+clean+scrubbing, 164 active+clean, 3 active+clean+snaptrim; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-02-10T04:44:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:01 smithi164 conmon[29288]: 2021-02-10T04:44:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:01 smithi164 conmon[29288]: audit 2021-02-10T04:44:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:01 smithi164 conmon[29288]: 2021-02-10T04:44:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:01 smithi164 conmon[29288]: 2021 2021-02-10T04:44:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:01 smithi164 conmon[29288]: - 2021-02-10T04:44:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:01 smithi164 conmon[29288]: 02 2021-02-10T04:44:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:01 smithi164 conmon[29288]: - 2021-02-10T04:44:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:01 smithi164 conmon[29288]: 10 2021-02-10T04:44:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:01 smithi164 conmon[29288]: T 2021-02-10T04:44:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:01 smithi164 conmon[29288]: 04:44:00.799714+0000 2021-02-10T04:44:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:01 smithi164 conmon[29288]: mgr.y (mgr.14140) 925 : audit [DBG] from='client.25285 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:44:01.969 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:44:02.029 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:44:03.441 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:44:03 smithi085 conmon[41540]: debug 2021-02-10T04:44:03.038+0000 7fb94bb63700 -1 osd.0 pg_epoch: 1003 pg[5.5( empty local-lis/les=57/58 n=0 ec=57/57 lis/c=57/57 les/c/f=58/58/0 sis=57) [0,1,4] r=0 lpr=57 crt=0'0 mlcod 0'0 active+clean] scrubber pg(5.5) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x555d32c72840) 2021-02-10T04:44:03.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:03 smithi164 conmon[29288]: cluster 2021-02-10T04:44:02.268535+0000 mgr.y 2021-02-10T04:44:03.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:03 smithi164 conmon[29288]: (mgr.14140) 926 : cluster [DBG] pgmap v1665: 168 pgs: 1 active+clean+scrubbing, 164 active+clean, 3 active+clean+snaptrim; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:44:03.894 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:44:03 smithi164 conmon[37263]: debug 2021-02-10T04:44:03.676+0000 7f1c671e2700 -1 osd.5 pg_epoch: 1003 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x5566508fc580) 2021-02-10T04:44:03.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:03 smithi085 conmon[35961]: cluster 2021-02-10T04:44 2021-02-10T04:44:03.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:03 smithi085 conmon[35961]: :02.268535+0000 mgr.y (mgr.14140) 926 : cluster [DBG] pgmap v1665: 168 pgs: 1 active+clean+scrubbing, 164 active+clean, 3 active+clean+snaptrim; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:44:03.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:03 smithi085 conmon[29319]: cluster 2021-02-10T04:44:02.268535+0000 mgr.y 2021-02-10T04:44:03.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:03 smithi085 conmon[29319]: (mgr.14140) 926 : cluster [DBG] pgmap v1665: 168 pgs: 1 active+clean+scrubbing, 164 active+clean, 3 active+clean+snaptrim; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:44:05.143 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:44:04 smithi164 conmon[37263]: debug 2021-02-10T04:44:05.144 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:44:04 smithi164 conmon[37263]: 2021-02-10T04:44:04.721+0000 7f1c6b1ea700 -1 osd.5 pg_epoch: 1003 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x5566530798c0) 2021-02-10T04:44:05.144 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:44:04 smithi164 conmon[37263]: 2021-02-10T04:44:05.702 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:05 smithi164 conmon[29288]: cluster 2021-02-10T04:44:04.269443+0000 mgr.y ( 2021-02-10T04:44:05.703 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:05 smithi164 conmon[29288]: mgr.14140) 927 : cluster [DBG] pgmap v1666: 168 pgs: 1 active+clean+scrubbing, 164 active+clean, 3 active+clean+snaptrim; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:44:05.760 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:05 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:44:05.761 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:05 smithi085 conmon[29319]: :44:04.269443+0000 mgr.y (mgr.14140) 927 : cluster [DBG] pgmap v1666: 168 pgs: 1 active+clean+scrubbing, 164 active+clean, 3 active+clean+snaptrim; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:44:05.761 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:05 smithi085 conmon[35961]: cluster 2021-02-10T04:44:04.269443 2021-02-10T04:44:05.761 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:05 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 927 : cluster [DBG] pgmap v1666: 168 pgs: 1 active+clean+scrubbing, 164 active+clean, 3 active+clean+snaptrim; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:44:05.841 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":1664,"stamp":"2021-02-10T04:44:00.267035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":900799606,"num_objects":727,"num_object_clones":320,"num_object_copies":2181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":727,"num_whiteouts":11,"num_read":9739,"num_read_kb":7595315,"num_write":6487,"num_write_kb":2087899,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":431,"num_bytes_recovered":108949605,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16525,"ondisk_log_size":16525,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2134088,"kb_used_data":1807184,"kb_used_omap":0,"kb_used_meta":326848,"kb_avail":654081976,"statfs":{"total":671965249536,"available":669779943424,"internally_reserved":0,"allocated":1850556416,"data_stored":2507361858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":334692352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4],"upper_bound":2097152},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14008847,"num_objects":-6,"num_object_clones":-6,"num_object_copies":-18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.929053"},"pg_stats":[{"pgid":"5.9","version":"131'3","reported_seq":"1077","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:57.904437+0000","last_change":"2021-02-10T04:33:57.871495+0000","last_active":"2021-02-10T04:43:57.904437+0000","last_peered":"2021-02-10T04:43:57.904437+0000","last_clean":"2021-02-10T04:43:57.904437+0000","last_became_active":"2021-02-10T04:29:07.145525+0000","last_became_peered":"2021-02-10T04:29:07.145525+0000","last_unstale":"2021-02-10T04:43:57.904437+0000","last_undegraded":"2021-02-10T04:43:57.904437+0000","last_fullsized":"2021-02-10T04:43:57.904437+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'3","last_scrub_stamp":"2021-02-10T04:33:57.871402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:33:57.871402+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"538'42","reported_seq":"1049","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.343003+0000","last_change":"2021-02-10T04:33:55.585872+0000","last_active":"2021-02-10T04:43:53.343003+0000","last_peered":"2021-02-10T04:43:53.343003+0000","last_clean":"2021-02-10T04:43:53.343003+0000","last_became_active":"2021-02-10T04:25:49.002058+0000","last_became_peered":"2021-02-10T04:25:49.002058+0000","last_unstale":"2021-02-10T04:43:53.343003+0000","last_undegraded":"2021-02-10T04:43:53.343003+0000","last_fullsized":"2021-02-10T04:43:53.343003+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2021-02-10T04:33:55.585807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:55.585807+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":77,"num_read_kb":60,"num_write":43,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"403","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.776917+0000","last_change":"2021-02-10T04:37:37.973234+0000","last_active":"2021-02-10T04:43:54.776917+0000","last_peered":"2021-02-10T04:43:54.776917+0000","last_clean":"2021-02-10T04:43:54.776917+0000","last_became_active":"2021-02-10T04:37:37.246903+0000","last_became_peered":"2021-02-10T04:37:37.246903+0000","last_unstale":"2021-02-10T04:43:54.776917+0000","last_undegraded":"2021-02-10T04:43:54.776917+0000","last_fullsized":"2021-02-10T04:43:54.776917+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:29:34.787265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:34.787265+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"961","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.343667+0000","last_change":"2021-02-10T04:29:19.405593+0000","last_active":"2021-02-10T04:43:53.343667+0000","last_peered":"2021-02-10T04:43:53.343667+0000","last_clean":"2021-02-10T04:43:53.343667+0000","last_became_active":"2021-02-10T04:25:53.148090+0000","last_became_peered":"2021-02-10T04:25:53.148090+0000","last_unstale":"2021-02-10T04:43:53.343667+0000","last_undegraded":"2021-02-10T04:43:53.343667+0000","last_fullsized":"2021-02-10T04:43:53.343667+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:19.405525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:29:19.405525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"1002'1683","reported_seq":"4485","reported_epoch":"1002","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:43:57.795265+0000","last_change":"2021-02-10T04:43:57.792423+0000","last_active":"2021-02-10T04:43:57.795265+0000","last_peered":"2021-02-10T04:43:57.795265+0000","last_clean":"2021-02-10T04:43:57.795265+0000","last_became_active":"2021-02-10T04:33:05.144488+0000","last_became_peered":"2021-02-10T04:33:05.144488+0000","last_unstale":"2021-02-10T04:43:57.795265+0000","last_undegraded":"2021-02-10T04:43:57.795265+0000","last_fullsized":"2021-02-10T04:43:57.795265+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'301","last_scrub_stamp":"2021-02-10T04:33:56.571925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:56.571925+0000","log_size":1683,"ondisk_log_size":1683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"stat_sum":{"num_bytes":126521476,"num_objects":49,"num_object_clones":44,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":773,"num_read_kb":819582,"num_write":561,"num_write_kb":221117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24e","length":"2"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"3"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"5"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2c9","length":"3"},{"start":"2d1","length":"7"},{"start":"2dd","length":"1"},{"start":"2df","length":"6"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2fc","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"307","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"333","length":"1"},{"start":"339","length":"1"},{"start":"33d","length":"1"},{"start":"340","length":"1"}]},{"pgid":"4.b","version":"538'26","reported_seq":"1052","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.775199+0000","last_change":"2021-02-10T04:38:39.265973+0000","last_active":"2021-02-10T04:43:54.775199+0000","last_peered":"2021-02-10T04:43:54.775199+0000","last_clean":"2021-02-10T04:43:54.775199+0000","last_became_active":"2021-02-10T04:38:39.243413+0000","last_became_peered":"2021-02-10T04:38:39.243413+0000","last_unstale":"2021-02-10T04:43:54.775199+0000","last_undegraded":"2021-02-10T04:43:54.775199+0000","last_fullsized":"2021-02-10T04:43:54.775199+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'26","last_scrub_stamp":"2021-02-10T04:38:32.145511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:32.145511+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":53,"num_read_kb":36,"num_write":27,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":894,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"645","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.344707+0000","last_change":"2021-02-10T04:33:05.151251+0000","last_active":"2021-02-10T04:43:53.344707+0000","last_peered":"2021-02-10T04:43:53.344707+0000","last_clean":"2021-02-10T04:43:53.344707+0000","last_became_active":"2021-02-10T04:33:05.150882+0000","last_became_peered":"2021-02-10T04:33:05.150882+0000","last_unstale":"2021-02-10T04:43:53.344707+0000","last_undegraded":"2021-02-10T04:43:53.344707+0000","last_fullsized":"2021-02-10T04:43:53.344707+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:31.102558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:31.102558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"374","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.344072+0000","last_change":"2021-02-10T04:37:37.244470+0000","last_active":"2021-02-10T04:43:53.344072+0000","last_peered":"2021-02-10T04:43:53.344072+0000","last_clean":"2021-02-10T04:43:53.344072+0000","last_became_active":"2021-02-10T04:37:37.242485+0000","last_became_peered":"2021-02-10T04:37:37.242485+0000","last_unstale":"2021-02-10T04:43:53.344072+0000","last_undegraded":"2021-02-10T04:43:53.344072+0000","last_fullsized":"2021-02-10T04:43:53.344072+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:35.805326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.805326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"973","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.774409+0000","last_change":"2021-02-10T04:37:37.254421+0000","last_active":"2021-02-10T04:43:54.774409+0000","last_peered":"2021-02-10T04:43:54.774409+0000","last_clean":"2021-02-10T04:43:54.774409+0000","last_became_active":"2021-02-10T04:37:37.254038+0000","last_became_peered":"2021-02-10T04:37:37.254038+0000","last_unstale":"2021-02-10T04:43:54.774409+0000","last_undegraded":"2021-02-10T04:43:54.774409+0000","last_fullsized":"2021-02-10T04:43:54.774409+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:48.700418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:48.700418+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"259'74","reported_seq":"1890","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.343019+0000","last_change":"2021-02-10T04:43:34.353765+0000","last_active":"2021-02-10T04:43:53.343019+0000","last_peered":"2021-02-10T04:43:53.343019+0000","last_clean":"2021-02-10T04:43:53.343019+0000","last_became_active":"2021-02-10T04:37:37.246533+0000","last_became_peered":"2021-02-10T04:37:37.246533+0000","last_unstale":"2021-02-10T04:43:53.343019+0000","last_undegraded":"2021-02-10T04:43:53.343019+0000","last_fullsized":"2021-02-10T04:43:53.343019+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'62","last_scrub_stamp":"2021-02-10T04:30:10.905905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:10.905905+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2437048,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":48,"num_read_kb":56455,"num_write":38,"num_write_kb":14043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1134570,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24e","length":"2"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"3"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"5"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2c9","length":"3"},{"start":"2d1","length":"7"},{"start":"2dd","length":"1"},{"start":"2df","length":"6"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2fc","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"307","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"333","length":"1"},{"start":"339","length":"1"},{"start":"33d","length":"1"},{"start":"340","length":"1"},{"start":"342","length":"1"},{"start":"347","length":"2"},{"start":"34a","length":"1"},{"start":"34c","length":"1"},{"start":"34f","length":"3"},{"start":"358","length":"1"},{"start":"364","length":"1"},{"start":"369","length":"1"}]},{"pgid":"4.a","version":"538'45","reported_seq":"1046","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.774610+0000","last_change":"2021-02-10T04:29:26.671051+0000","last_active":"2021-02-10T04:43:54.774610+0000","last_peered":"2021-02-10T04:43:54.774610+0000","last_clean":"2021-02-10T04:43:54.774610+0000","last_became_active":"2021-02-10T04:25:49.015913+0000","last_became_peered":"2021-02-10T04:25:49.015913+0000","last_unstale":"2021-02-10T04:43:54.774610+0000","last_undegraded":"2021-02-10T04:43:54.774610+0000","last_fullsized":"2021-02-10T04:43:54.774610+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'29","last_scrub_stamp":"2021-02-10T04:29:26.670950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.670950+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":80,"num_read_kb":60,"num_write":46,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"642'3","reported_seq":"482","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.776902+0000","last_change":"2021-02-10T04:38:24.163263+0000","last_active":"2021-02-10T04:43:54.776902+0000","last_peered":"2021-02-10T04:43:54.776902+0000","last_clean":"2021-02-10T04:43:54.776902+0000","last_became_active":"2021-02-10T04:37:37.234230+0000","last_became_peered":"2021-02-10T04:37:37.234230+0000","last_unstale":"2021-02-10T04:43:54.776902+0000","last_undegraded":"2021-02-10T04:43:54.776902+0000","last_fullsized":"2021-02-10T04:43:54.776902+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'3","last_scrub_stamp":"2021-02-10T04:38:24.163200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:24.163200+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"381","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.775039+0000","last_change":"2021-02-10T04:38:31.121100+0000","last_active":"2021-02-10T04:43:54.775039+0000","last_peered":"2021-02-10T04:43:54.775039+0000","last_clean":"2021-02-10T04:43:54.775039+0000","last_became_active":"2021-02-10T04:37:37.230818+0000","last_became_peered":"2021-02-10T04:37:37.230818+0000","last_unstale":"2021-02-10T04:43:54.775039+0000","last_undegraded":"2021-02-10T04:43:54.775039+0000","last_fullsized":"2021-02-10T04:43:54.775039+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:31.121038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:38:31.121038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"379","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.777169+0000","last_change":"2021-02-10T04:37:37.245491+0000","last_active":"2021-02-10T04:43:54.777169+0000","last_peered":"2021-02-10T04:43:54.777169+0000","last_clean":"2021-02-10T04:43:54.777169+0000","last_became_active":"2021-02-10T04:37:37.245113+0000","last_became_peered":"2021-02-10T04:37:37.245113+0000","last_unstale":"2021-02-10T04:43:54.777169+0000","last_undegraded":"2021-02-10T04:43:54.777169+0000","last_fullsized":"2021-02-10T04:43:54.777169+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.731458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.731458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"1003'739","reported_seq":"3645","reported_epoch":"1003","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:43:58.648858+0000","last_change":"2021-02-10T04:43:58.648858+0000","last_active":"2021-02-10T04:43:58.648858+0000","last_peered":"2021-02-10T04:43:58.648858+0000","last_clean":"2021-02-10T04:43:58.648858+0000","last_became_active":"2021-02-10T04:28:25.150246+0000","last_became_peered":"2021-02-10T04:28:25.150246+0000","last_unstale":"2021-02-10T04:43:58.648858+0000","last_undegraded":"2021-02-10T04:43:58.648858+0000","last_fullsized":"2021-02-10T04:43:58.648858+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'63","last_scrub_stamp":"2021-02-10T04:29:39.700115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:29:39.700115+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":48119777,"num_objects":19,"num_object_clones":17,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":259,"num_read_kb":219088,"num_write":204,"num_write_kb":92347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24e","length":"2"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"3"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"5"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2c9","length":"3"},{"start":"2d1","length":"7"},{"start":"2dd","length":"1"},{"start":"2df","length":"6"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2fc","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"307","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"333","length":"1"},{"start":"339","length":"1"},{"start":"33d","length":"1"},{"start":"340","length":"1"},{"start":"342","length":"1"},{"start":"347","length":"2"},{"start":"34a","length":"1"},{"start":"34c","length":"1"},{"start":"34f","length":"3"},{"start":"358","length":"1"}]},{"pgid":"4.d","version":"538'47","reported_seq":"1068","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.344859+0000","last_change":"2021-02-10T04:37:37.821860+0000","last_active":"2021-02-10T04:43:53.344859+0000","last_peered":"2021-02-10T04:43:53.344859+0000","last_clean":"2021-02-10T04:43:53.344859+0000","last_became_active":"2021-02-10T04:37:37.237215+0000","last_became_peered":"2021-02-10T04:37:37.237215+0000","last_unstale":"2021-02-10T04:43:53.344859+0000","last_undegraded":"2021-02-10T04:43:53.344859+0000","last_fullsized":"2021-02-10T04:43:53.344859+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'33","last_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:28:49.514984+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":62,"num_read_kb":44,"num_write":36,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"882","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.343152+0000","last_change":"2021-02-10T04:29:07.146266+0000","last_active":"2021-02-10T04:43:53.343152+0000","last_peered":"2021-02-10T04:43:53.343152+0000","last_clean":"2021-02-10T04:43:53.343152+0000","last_became_active":"2021-02-10T04:29:07.145616+0000","last_became_peered":"2021-02-10T04:29:07.145616+0000","last_unstale":"2021-02-10T04:43:53.343152+0000","last_undegraded":"2021-02-10T04:43:53.343152+0000","last_fullsized":"2021-02-10T04:43:53.343152+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:44.410817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:44.410817+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"645","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.343040+0000","last_change":"2021-02-10T04:33:05.147166+0000","last_active":"2021-02-10T04:43:53.343040+0000","last_peered":"2021-02-10T04:43:53.343040+0000","last_clean":"2021-02-10T04:43:53.343040+0000","last_became_active":"2021-02-10T04:33:05.146783+0000","last_became_peered":"2021-02-10T04:33:05.146783+0000","last_unstale":"2021-02-10T04:43:53.343040+0000","last_undegraded":"2021-02-10T04:43:53.343040+0000","last_fullsized":"2021-02-10T04:43:53.343040+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:35.958255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.958255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"381","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.778225+0000","last_change":"2021-02-10T04:38:32.875574+0000","last_active":"2021-02-10T04:43:54.778225+0000","last_peered":"2021-02-10T04:43:54.778225+0000","last_clean":"2021-02-10T04:43:54.778225+0000","last_became_active":"2021-02-10T04:37:37.251479+0000","last_became_peered":"2021-02-10T04:37:37.251479+0000","last_unstale":"2021-02-10T04:43:54.778225+0000","last_undegraded":"2021-02-10T04:43:54.778225+0000","last_fullsized":"2021-02-10T04:43:54.778225+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:32.875498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:38:32.875498+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"999'629","reported_seq":"4165","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.673766+0000","last_change":"2021-02-10T04:43:53.673766+0000","last_active":"2021-02-10T04:43:53.673766+0000","last_peered":"2021-02-10T04:43:53.673766+0000","last_clean":"2021-02-10T04:43:53.673766+0000","last_became_active":"2021-02-10T04:37:40.267794+0000","last_became_peered":"2021-02-10T04:37:40.267794+0000","last_unstale":"2021-02-10T04:43:53.673766+0000","last_undegraded":"2021-02-10T04:43:53.673766+0000","last_fullsized":"2021-02-10T04:43:53.673766+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"688'187","last_scrub_stamp":"2021-02-10T04:38:27.225488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:38:27.225488+0000","log_size":629,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49401261,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":2,"num_read":342,"num_read_kb":364544,"num_write":223,"num_write_kb":67454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":11919937,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24e","length":"2"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"3"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"5"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2c9","length":"3"},{"start":"2d1","length":"7"},{"start":"2dd","length":"1"},{"start":"2df","length":"6"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2fc","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"307","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"333","length":"1"},{"start":"339","length":"1"},{"start":"33d","length":"1"},{"start":"340","length":"1"},{"start":"342","length":"1"},{"start":"347","length":"2"},{"start":"34a","length":"1"},{"start":"34c","length":"1"},{"start":"34f","length":"3"},{"start":"358","length":"1"},{"start":"364","length":"1"},{"start":"369","length":"1"}]},{"pgid":"1.8","version":"0'0","reported_seq":"913","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.343195+0000","last_change":"2021-02-10T04:37:37.246395+0000","last_active":"2021-02-10T04:43:53.343195+0000","last_peered":"2021-02-10T04:43:53.343195+0000","last_clean":"2021-02-10T04:43:53.343195+0000","last_became_active":"2021-02-10T04:37:37.246004+0000","last_became_peered":"2021-02-10T04:37:37.246004+0000","last_unstale":"2021-02-10T04:43:53.343195+0000","last_undegraded":"2021-02-10T04:43:53.343195+0000","last_fullsized":"2021-02-10T04:43:53.343195+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:37.336092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:37.336092+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"988","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.776148+0000","last_change":"2021-02-10T04:37:37.259938+0000","last_active":"2021-02-10T04:43:54.776148+0000","last_peered":"2021-02-10T04:43:54.776148+0000","last_clean":"2021-02-10T04:43:54.776148+0000","last_became_active":"2021-02-10T04:37:37.259532+0000","last_became_peered":"2021-02-10T04:37:37.259532+0000","last_unstale":"2021-02-10T04:43:54.776148+0000","last_undegraded":"2021-02-10T04:43:54.776148+0000","last_fullsized":"2021-02-10T04:43:54.776148+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:51.858386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:33:51.858386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"538'23","reported_seq":"1000","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.345010+0000","last_change":"2021-02-10T04:28:41.427119+0000","last_active":"2021-02-10T04:43:53.345010+0000","last_peered":"2021-02-10T04:43:53.345010+0000","last_clean":"2021-02-10T04:43:53.345010+0000","last_became_active":"2021-02-10T04:25:49.001725+0000","last_became_peered":"2021-02-10T04:25:49.001725+0000","last_unstale":"2021-02-10T04:43:53.345010+0000","last_undegraded":"2021-02-10T04:43:53.345010+0000","last_fullsized":"2021-02-10T04:43:53.345010+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'13","last_scrub_stamp":"2021-02-10T04:28:41.427050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:41.427050+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":28,"num_write":23,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"651","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.778443+0000","last_change":"2021-02-10T04:33:05.168734+0000","last_active":"2021-02-10T04:43:54.778443+0000","last_peered":"2021-02-10T04:43:54.778443+0000","last_clean":"2021-02-10T04:43:54.778443+0000","last_became_active":"2021-02-10T04:33:05.168314+0000","last_became_peered":"2021-02-10T04:33:05.168314+0000","last_unstale":"2021-02-10T04:43:54.778443+0000","last_undegraded":"2021-02-10T04:43:54.778443+0000","last_fullsized":"2021-02-10T04:43:54.778443+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:18.107867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:18.107867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"639'3","reported_seq":"470","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.776529+0000","last_change":"2021-02-10T04:38:31.211769+0000","last_active":"2021-02-10T04:43:54.776529+0000","last_peered":"2021-02-10T04:43:54.776529+0000","last_clean":"2021-02-10T04:43:54.776529+0000","last_became_active":"2021-02-10T04:37:37.233738+0000","last_became_peered":"2021-02-10T04:37:37.233738+0000","last_unstale":"2021-02-10T04:43:54.776529+0000","last_undegraded":"2021-02-10T04:43:54.776529+0000","last_fullsized":"2021-02-10T04:43:54.776529+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"639'3","last_scrub_stamp":"2021-02-10T04:38:31.211662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:38:31.211662+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"969","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.344943+0000","last_change":"2021-02-10T04:37:37.236009+0000","last_active":"2021-02-10T04:43:53.344943+0000","last_peered":"2021-02-10T04:43:53.344943+0000","last_clean":"2021-02-10T04:43:53.344943+0000","last_became_active":"2021-02-10T04:37:37.233109+0000","last_became_peered":"2021-02-10T04:37:37.233109+0000","last_unstale":"2021-02-10T04:43:53.344943+0000","last_undegraded":"2021-02-10T04:43:53.344943+0000","last_fullsized":"2021-02-10T04:43:53.344943+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:38.873102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.873102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"1003'1085","reported_seq":"6157","reported_epoch":"1003","state":"active+clean+snaptrim","last_fresh":"2021-02-10T04:43:59.979506+0000","last_change":"2021-02-10T04:43:59.979506+0000","last_active":"2021-02-10T04:43:59.979506+0000","last_peered":"2021-02-10T04:43:59.979506+0000","last_clean":"2021-02-10T04:43:59.979506+0000","last_became_active":"2021-02-10T04:28:25.149226+0000","last_became_peered":"2021-02-10T04:28:25.149226+0000","last_unstale":"2021-02-10T04:43:59.979506+0000","last_undegraded":"2021-02-10T04:43:59.979506+0000","last_fullsized":"2021-02-10T04:43:59.979506+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'236","last_scrub_stamp":"2021-02-10T04:33:50.351534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:50.351534+0000","log_size":1085,"ondisk_log_size":1085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"stat_sum":{"num_bytes":78869393,"num_objects":34,"num_object_clones":30,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":1,"num_read":558,"num_read_kb":622116,"num_write":371,"num_write_kb":133580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24e","length":"2"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"3"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"5"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2c9","length":"3"},{"start":"2d1","length":"7"},{"start":"2dd","length":"1"},{"start":"2df","length":"6"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2fc","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"307","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"333","length":"1"}]},{"pgid":"1.a","version":"0'0","reported_seq":"913","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.343598+0000","last_change":"2021-02-10T04:37:37.232619+0000","last_active":"2021-02-10T04:43:53.343598+0000","last_peered":"2021-02-10T04:43:53.343598+0000","last_clean":"2021-02-10T04:43:53.343598+0000","last_became_active":"2021-02-10T04:37:37.229648+0000","last_became_peered":"2021-02-10T04:37:37.229648+0000","last_unstale":"2021-02-10T04:43:53.343598+0000","last_undegraded":"2021-02-10T04:43:53.343598+0000","last_fullsized":"2021-02-10T04:43:53.343598+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.384170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.384170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"538'29","reported_seq":"707","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.777522+0000","last_change":"2021-02-10T04:33:05.154998+0000","last_active":"2021-02-10T04:43:54.777522+0000","last_peered":"2021-02-10T04:43:54.777522+0000","last_clean":"2021-02-10T04:43:54.777522+0000","last_became_active":"2021-02-10T04:33:05.149026+0000","last_became_peered":"2021-02-10T04:33:05.149026+0000","last_unstale":"2021-02-10T04:43:54.777522+0000","last_undegraded":"2021-02-10T04:43:54.777522+0000","last_fullsized":"2021-02-10T04:43:54.777522+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'17","last_scrub_stamp":"2021-02-10T04:29:37.933206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:37.933206+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":49,"num_read_kb":36,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"711","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.776658+0000","last_change":"2021-02-10T04:33:05.169654+0000","last_active":"2021-02-10T04:43:54.776658+0000","last_peered":"2021-02-10T04:43:54.776658+0000","last_clean":"2021-02-10T04:43:54.776658+0000","last_became_active":"2021-02-10T04:33:05.169289+0000","last_became_peered":"2021-02-10T04:33:05.169289+0000","last_unstale":"2021-02-10T04:43:54.776658+0000","last_undegraded":"2021-02-10T04:43:54.776658+0000","last_fullsized":"2021-02-10T04:43:54.776658+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_clean_scrub_stamp":"2021-02-10T04:28:39.128781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"78'3","reported_seq":"1182","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.344858+0000","last_change":"2021-02-10T04:28:43.509446+0000","last_active":"2021-02-10T04:43:53.344858+0000","last_peered":"2021-02-10T04:43:53.344858+0000","last_clean":"2021-02-10T04:43:53.344858+0000","last_became_active":"2021-02-10T04:25:51.025218+0000","last_became_peered":"2021-02-10T04:25:51.025218+0000","last_unstale":"2021-02-10T04:43:53.344858+0000","last_undegraded":"2021-02-10T04:43:53.344858+0000","last_fullsized":"2021-02-10T04:43:53.344858+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:28:43.509353+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:28:43.509353+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"961","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.346381+0000","last_change":"2021-02-10T04:28:45.481538+0000","last_active":"2021-02-10T04:43:53.346381+0000","last_peered":"2021-02-10T04:43:53.346381+0000","last_clean":"2021-02-10T04:43:53.346381+0000","last_became_active":"2021-02-10T04:25:53.042697+0000","last_became_peered":"2021-02-10T04:25:53.042697+0000","last_unstale":"2021-02-10T04:43:53.346381+0000","last_undegraded":"2021-02-10T04:43:53.346381+0000","last_fullsized":"2021-02-10T04:43:53.346381+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:45.481472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:28:45.481472+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"1002'1090","reported_seq":"5159","reported_epoch":"1003","state":"active+clean","last_fresh":"2021-02-10T04:43:58.639085+0000","last_change":"2021-02-10T04:43:58.639400+0000","last_active":"2021-02-10T04:43:58.639085+0000","last_peered":"2021-02-10T04:43:58.639085+0000","last_clean":"2021-02-10T04:43:58.639085+0000","last_became_active":"2021-02-10T04:28:25.148500+0000","last_became_peered":"2021-02-10T04:28:25.148500+0000","last_unstale":"2021-02-10T04:43:58.639085+0000","last_undegraded":"2021-02-10T04:43:58.639085+0000","last_fullsized":"2021-02-10T04:43:58.639085+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'110","last_scrub_stamp":"2021-02-10T04:30:18.463782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:18.463782+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":48821292,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":491,"num_read_kb":552336,"num_write":351,"num_write_kb":129921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24e","length":"2"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"3"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"5"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2c9","length":"3"},{"start":"2d1","length":"7"},{"start":"2dd","length":"1"},{"start":"2df","length":"6"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2fc","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"307","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"333","length":"1"},{"start":"339","length":"1"},{"start":"33d","length":"1"},{"start":"340","length":"1"},{"start":"342","length":"1"},{"start":"347","length":"2"},{"start":"34a","length":"1"},{"start":"34c","length":"1"},{"start":"34f","length":"3"},{"start":"358","length":"1"},{"start":"364","length":"1"},{"start":"369","length":"1"}]},{"pgid":"4.e","version":"538'39","reported_seq":"1028","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.344622+0000","last_change":"2021-02-10T04:28:11.693702+0000","last_active":"2021-02-10T04:43:53.344622+0000","last_peered":"2021-02-10T04:43:53.344622+0000","last_clean":"2021-02-10T04:43:53.344622+0000","last_became_active":"2021-02-10T04:25:49.011093+0000","last_became_peered":"2021-02-10T04:25:49.011093+0000","last_unstale":"2021-02-10T04:43:53.344622+0000","last_undegraded":"2021-02-10T04:43:53.344622+0000","last_fullsized":"2021-02-10T04:43:53.344622+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'31","last_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:28:11.693616+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":56,"num_read_kb":40,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"947","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.773876+0000","last_change":"2021-02-10T04:29:31.683314+0000","last_active":"2021-02-10T04:43:54.773876+0000","last_peered":"2021-02-10T04:43:54.773876+0000","last_clean":"2021-02-10T04:43:54.773876+0000","last_became_active":"2021-02-10T04:29:07.160593+0000","last_became_peered":"2021-02-10T04:29:07.160593+0000","last_unstale":"2021-02-10T04:43:54.773876+0000","last_undegraded":"2021-02-10T04:43:54.773876+0000","last_fullsized":"2021-02-10T04:43:54.773876+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:31.683213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.683213+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"968","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.347337+0000","last_change":"2021-02-10T04:30:10.487631+0000","last_active":"2021-02-10T04:43:53.347337+0000","last_peered":"2021-02-10T04:43:53.347337+0000","last_clean":"2021-02-10T04:43:53.347337+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:43:53.347337+0000","last_undegraded":"2021-02-10T04:43:53.347337+0000","last_fullsized":"2021-02-10T04:43:53.347337+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:10.487528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:30:10.487528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"961","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.776480+0000","last_change":"2021-02-10T04:33:55.849675+0000","last_active":"2021-02-10T04:43:54.776480+0000","last_peered":"2021-02-10T04:43:54.776480+0000","last_clean":"2021-02-10T04:43:54.776480+0000","last_became_active":"2021-02-10T04:25:53.148054+0000","last_became_peered":"2021-02-10T04:25:53.148054+0000","last_unstale":"2021-02-10T04:43:54.776480+0000","last_undegraded":"2021-02-10T04:43:54.776480+0000","last_fullsized":"2021-02-10T04:43:54.776480+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:55.849582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:33:55.849582+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"1001'1165","reported_seq":"4137","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:57.854903+0000","last_change":"2021-02-10T04:43:57.854903+0000","last_active":"2021-02-10T04:43:57.854903+0000","last_peered":"2021-02-10T04:43:57.854903+0000","last_clean":"2021-02-10T04:43:57.854903+0000","last_became_active":"2021-02-10T04:28:25.152030+0000","last_became_peered":"2021-02-10T04:28:25.152030+0000","last_unstale":"2021-02-10T04:43:57.854903+0000","last_undegraded":"2021-02-10T04:43:57.854903+0000","last_fullsized":"2021-02-10T04:43:57.854903+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'283","last_scrub_stamp":"2021-02-10T04:33:53.834627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:53.834627+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":59203690,"num_objects":25,"num_object_clones":21,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":573,"num_read_kb":599059,"num_write":400,"num_write_kb":178261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24e","length":"2"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"3"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"5"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2c9","length":"3"},{"start":"2d1","length":"7"},{"start":"2dd","length":"1"},{"start":"2df","length":"6"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2fc","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"307","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"333","length":"1"},{"start":"339","length":"1"},{"start":"33d","length":"1"},{"start":"340","length":"1"},{"start":"342","length":"1"},{"start":"347","length":"2"},{"start":"34a","length":"1"},{"start":"34c","length":"1"},{"start":"34f","length":"3"},{"start":"358","length":"1"},{"start":"364","length":"1"},{"start":"369","length":"1"}]},{"pgid":"5.0","version":"158'3","reported_seq":"1184","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:57.904279+0000","last_change":"2021-02-10T04:29:34.388978+0000","last_active":"2021-02-10T04:43:57.904279+0000","last_peered":"2021-02-10T04:43:57.904279+0000","last_clean":"2021-02-10T04:43:57.904279+0000","last_became_active":"2021-02-10T04:25:51.031906+0000","last_became_peered":"2021-02-10T04:25:51.031906+0000","last_unstale":"2021-02-10T04:43:57.904279+0000","last_undegraded":"2021-02-10T04:43:57.904279+0000","last_fullsized":"2021-02-10T04:43:57.904279+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'3","last_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_deep_scrub":"158'3","last_deep_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_clean_scrub_stamp":"2021-02-10T04:29:34.388912+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"538'40","reported_seq":"1033","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.343231+0000","last_change":"2021-02-10T04:29:36.813190+0000","last_active":"2021-02-10T04:43:53.343231+0000","last_peered":"2021-02-10T04:43:53.343231+0000","last_clean":"2021-02-10T04:43:53.343231+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:43:53.343231+0000","last_undegraded":"2021-02-10T04:43:53.343231+0000","last_fullsized":"2021-02-10T04:43:53.343231+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:29:36.813083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.813083+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":72,"num_read_kb":52,"num_write":40,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"1014","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.774454+0000","last_change":"2021-02-10T04:37:37.245911+0000","last_active":"2021-02-10T04:43:54.774454+0000","last_peered":"2021-02-10T04:43:54.774454+0000","last_clean":"2021-02-10T04:43:54.774454+0000","last_became_active":"2021-02-10T04:37:37.245391+0000","last_became_peered":"2021-02-10T04:37:37.245391+0000","last_unstale":"2021-02-10T04:43:54.774454+0000","last_undegraded":"2021-02-10T04:43:54.774454+0000","last_fullsized":"2021-02-10T04:43:54.774454+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.632085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.632085+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"898","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.774527+0000","last_change":"2021-02-10T04:37:37.235415+0000","last_active":"2021-02-10T04:43:54.774527+0000","last_peered":"2021-02-10T04:43:54.774527+0000","last_clean":"2021-02-10T04:43:54.774527+0000","last_became_active":"2021-02-10T04:37:37.235082+0000","last_became_peered":"2021-02-10T04:37:37.235082+0000","last_unstale":"2021-02-10T04:43:54.774527+0000","last_undegraded":"2021-02-10T04:43:54.774527+0000","last_fullsized":"2021-02-10T04:43:54.774527+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.658374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.658374+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"998'1009","reported_seq":"4485","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.776268+0000","last_change":"2021-02-10T04:43:51.660895+0000","last_active":"2021-02-10T04:43:54.776268+0000","last_peered":"2021-02-10T04:43:54.776268+0000","last_clean":"2021-02-10T04:43:54.776268+0000","last_became_active":"2021-02-10T04:37:45.352779+0000","last_became_peered":"2021-02-10T04:37:45.352779+0000","last_unstale":"2021-02-10T04:43:54.776268+0000","last_undegraded":"2021-02-10T04:43:54.776268+0000","last_fullsized":"2021-02-10T04:43:54.776268+0000","mapping_epoch":646,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":647,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49420827,"num_objects":21,"num_object_clones":19,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":2,"num_read":478,"num_read_kb":498806,"num_write":309,"num_write_kb":100208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":16580116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24e","length":"2"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"3"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"5"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2c9","length":"3"},{"start":"2d1","length":"7"},{"start":"2dd","length":"1"},{"start":"2df","length":"6"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2fc","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"307","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"333","length":"1"},{"start":"339","length":"1"},{"start":"33d","length":"1"},{"start":"340","length":"1"},{"start":"342","length":"1"},{"start":"347","length":"2"},{"start":"34a","length":"1"},{"start":"34c","length":"1"},{"start":"34f","length":"3"},{"start":"358","length":"1"},{"start":"364","length":"1"},{"start":"369","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":"905","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.776565+0000","last_change":"2021-02-10T04:30:12.452072+0000","last_active":"2021-02-10T04:43:54.776565+0000","last_peered":"2021-02-10T04:43:54.776565+0000","last_clean":"2021-02-10T04:43:54.776565+0000","last_became_active":"2021-02-10T04:28:49.048525+0000","last_became_peered":"2021-02-10T04:28:49.048525+0000","last_unstale":"2021-02-10T04:43:54.776565+0000","last_undegraded":"2021-02-10T04:43:54.776565+0000","last_fullsized":"2021-02-10T04:43:54.776565+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:12.452000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:30:12.452000+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"962","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.343464+0000","last_change":"2021-02-10T04:29:40.859352+0000","last_active":"2021-02-10T04:43:53.343464+0000","last_peered":"2021-02-10T04:43:53.343464+0000","last_clean":"2021-02-10T04:43:53.343464+0000","last_became_active":"2021-02-10T04:25:51.018706+0000","last_became_peered":"2021-02-10T04:25:51.018706+0000","last_unstale":"2021-02-10T04:43:53.343464+0000","last_undegraded":"2021-02-10T04:43:53.343464+0000","last_fullsized":"2021-02-10T04:43:53.343464+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:40.859274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:40.859274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"538'60","reported_seq":"1067","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.777084+0000","last_change":"2021-02-10T04:29:28.398522+0000","last_active":"2021-02-10T04:43:54.777084+0000","last_peered":"2021-02-10T04:43:54.777084+0000","last_clean":"2021-02-10T04:43:54.777084+0000","last_became_active":"2021-02-10T04:25:48.997481+0000","last_became_peered":"2021-02-10T04:25:48.997481+0000","last_unstale":"2021-02-10T04:43:54.777084+0000","last_undegraded":"2021-02-10T04:43:54.777084+0000","last_fullsized":"2021-02-10T04:43:54.777084+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'48","last_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_deep_scrub":"152'48","last_deep_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.398429+0000","log_size":60,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":88,"num_read_kb":66,"num_write":48,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"660","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.774956+0000","last_change":"2021-02-10T04:33:05.142730+0000","last_active":"2021-02-10T04:43:54.774956+0000","last_peered":"2021-02-10T04:43:54.774956+0000","last_clean":"2021-02-10T04:43:54.774956+0000","last_became_active":"2021-02-10T04:33:05.141066+0000","last_became_peered":"2021-02-10T04:33:05.141066+0000","last_unstale":"2021-02-10T04:43:54.774956+0000","last_undegraded":"2021-02-10T04:43:54.774956+0000","last_fullsized":"2021-02-10T04:43:54.774956+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:28:35.010709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:35.010709+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":423,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"969","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.344153+0000","last_change":"2021-02-10T04:37:37.245330+0000","last_active":"2021-02-10T04:43:53.344153+0000","last_peered":"2021-02-10T04:43:53.344153+0000","last_clean":"2021-02-10T04:43:53.344153+0000","last_became_active":"2021-02-10T04:37:37.244859+0000","last_became_peered":"2021-02-10T04:37:37.244859+0000","last_unstale":"2021-02-10T04:43:53.344153+0000","last_undegraded":"2021-02-10T04:43:53.344153+0000","last_fullsized":"2021-02-10T04:43:53.344153+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:05.641577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:05.641577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"1002'1657","reported_seq":"4899","reported_epoch":"1003","state":"active+clean","last_fresh":"2021-02-10T04:43:59.686583+0000","last_change":"2021-02-10T04:43:59.686971+0000","last_active":"2021-02-10T04:43:59.686583+0000","last_peered":"2021-02-10T04:43:59.686583+0000","last_clean":"2021-02-10T04:43:59.686583+0000","last_became_active":"2021-02-10T04:37:45.352022+0000","last_became_peered":"2021-02-10T04:37:45.352022+0000","last_unstale":"2021-02-10T04:43:59.686583+0000","last_undegraded":"2021-02-10T04:43:59.686583+0000","last_fullsized":"2021-02-10T04:43:59.686583+0000","mapping_epoch":646,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":647,"parent":"0.0","parent_split_bits":0,"last_scrub":"687'1095","last_scrub_stamp":"2021-02-10T04:38:27.895743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:38:27.895743+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":85586133,"num_objects":34,"num_object_clones":30,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":1,"num_read":848,"num_read_kb":875347,"num_write":637,"num_write_kb":237595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":35873876,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24e","length":"2"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"3"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"5"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2c9","length":"3"},{"start":"2d1","length":"7"},{"start":"2dd","length":"1"},{"start":"2df","length":"6"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2fc","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"307","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"333","length":"1"},{"start":"339","length":"1"},{"start":"33d","length":"1"},{"start":"340","length":"1"},{"start":"342","length":"1"},{"start":"347","length":"2"},{"start":"34a","length":"1"},{"start":"34c","length":"1"},{"start":"34f","length":"3"},{"start":"358","length":"1"},{"start":"364","length":"1"},{"start":"369","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":"912","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.345087+0000","last_change":"2021-02-10T04:37:37.233535+0000","last_active":"2021-02-10T04:43:53.345087+0000","last_peered":"2021-02-10T04:43:53.345087+0000","last_clean":"2021-02-10T04:43:53.345087+0000","last_became_active":"2021-02-10T04:37:37.233158+0000","last_became_peered":"2021-02-10T04:37:37.233158+0000","last_unstale":"2021-02-10T04:43:53.345087+0000","last_undegraded":"2021-02-10T04:43:53.345087+0000","last_fullsized":"2021-02-10T04:43:53.345087+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:33.839335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:33.839335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"1035","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.774765+0000","last_change":"2021-02-10T04:33:52.827328+0000","last_active":"2021-02-10T04:43:54.774765+0000","last_peered":"2021-02-10T04:43:54.774765+0000","last_clean":"2021-02-10T04:43:54.774765+0000","last_became_active":"2021-02-10T04:25:51.025636+0000","last_became_peered":"2021-02-10T04:25:51.025636+0000","last_unstale":"2021-02-10T04:43:54.774765+0000","last_undegraded":"2021-02-10T04:43:54.774765+0000","last_fullsized":"2021-02-10T04:43:54.774765+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:52.827257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:33:52.827257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"538'37","reported_seq":"1041","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.774962+0000","last_change":"2021-02-10T04:38:36.138126+0000","last_active":"2021-02-10T04:43:54.774962+0000","last_peered":"2021-02-10T04:43:54.774962+0000","last_clean":"2021-02-10T04:43:54.774962+0000","last_became_active":"2021-02-10T04:25:49.002937+0000","last_became_peered":"2021-02-10T04:25:49.002937+0000","last_unstale":"2021-02-10T04:43:54.774962+0000","last_undegraded":"2021-02-10T04:43:54.774962+0000","last_fullsized":"2021-02-10T04:43:54.774962+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'37","last_scrub_stamp":"2021-02-10T04:38:36.138022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:36.138022+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":72,"num_read_kb":53,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"900","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.776268+0000","last_change":"2021-02-10T04:29:35.711635+0000","last_active":"2021-02-10T04:43:54.776268+0000","last_peered":"2021-02-10T04:43:54.776268+0000","last_clean":"2021-02-10T04:43:54.776268+0000","last_became_active":"2021-02-10T04:29:07.161341+0000","last_became_peered":"2021-02-10T04:29:07.161341+0000","last_unstale":"2021-02-10T04:43:54.776268+0000","last_undegraded":"2021-02-10T04:43:54.776268+0000","last_fullsized":"2021-02-10T04:43:54.776268+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_deep_scrub":"53'2","last_deep_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.711557+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"661","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.774008+0000","last_change":"2021-02-10T04:38:40.414663+0000","last_active":"2021-02-10T04:43:54.774008+0000","last_peered":"2021-02-10T04:43:54.774008+0000","last_clean":"2021-02-10T04:43:54.774008+0000","last_became_active":"2021-02-10T04:37:37.234868+0000","last_became_peered":"2021-02-10T04:37:37.234868+0000","last_unstale":"2021-02-10T04:43:54.774008+0000","last_undegraded":"2021-02-10T04:43:54.774008+0000","last_fullsized":"2021-02-10T04:43:54.774008+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:40.414601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:38:40.414601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"1000'1071","reported_seq":"4505","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.677608+0000","last_change":"2021-02-10T04:43:53.677608+0000","last_active":"2021-02-10T04:43:53.677608+0000","last_peered":"2021-02-10T04:43:53.677608+0000","last_clean":"2021-02-10T04:43:53.677608+0000","last_became_active":"2021-02-10T04:28:25.146766+0000","last_became_peered":"2021-02-10T04:28:25.146766+0000","last_unstale":"2021-02-10T04:43:53.677608+0000","last_undegraded":"2021-02-10T04:43:53.677608+0000","last_fullsized":"2021-02-10T04:43:53.677608+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'357","last_scrub_stamp":"2021-02-10T04:33:50.582002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:50.582002+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46572736,"num_objects":18,"num_object_clones":15,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":484,"num_read_kb":453669,"num_write":370,"num_write_kb":152009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24e","length":"2"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"3"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"5"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2c9","length":"3"},{"start":"2d1","length":"7"},{"start":"2dd","length":"1"},{"start":"2df","length":"6"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2fc","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"307","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"333","length":"1"},{"start":"339","length":"1"},{"start":"33d","length":"1"},{"start":"340","length":"1"},{"start":"342","length":"1"},{"start":"347","length":"2"},{"start":"34a","length":"1"},{"start":"34c","length":"1"},{"start":"34f","length":"3"},{"start":"358","length":"1"},{"start":"364","length":"1"},{"start":"369","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":"986","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.775099+0000","last_change":"2021-02-10T04:29:38.793837+0000","last_active":"2021-02-10T04:43:54.775099+0000","last_peered":"2021-02-10T04:43:54.775099+0000","last_clean":"2021-02-10T04:43:54.775099+0000","last_became_active":"2021-02-10T04:28:49.054638+0000","last_became_peered":"2021-02-10T04:28:49.054638+0000","last_unstale":"2021-02-10T04:43:54.775099+0000","last_undegraded":"2021-02-10T04:43:54.775099+0000","last_fullsized":"2021-02-10T04:43:54.775099+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:38.793760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.793760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"963","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.347466+0000","last_change":"2021-02-10T04:29:40.451948+0000","last_active":"2021-02-10T04:43:53.347466+0000","last_peered":"2021-02-10T04:43:53.347466+0000","last_clean":"2021-02-10T04:43:53.347466+0000","last_became_active":"2021-02-10T04:25:51.032201+0000","last_became_peered":"2021-02-10T04:25:51.032201+0000","last_unstale":"2021-02-10T04:43:53.347466+0000","last_undegraded":"2021-02-10T04:43:53.347466+0000","last_fullsized":"2021-02-10T04:43:53.347466+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:40.451832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:29:40.451832+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"538'28","reported_seq":"1046","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.776641+0000","last_change":"2021-02-10T04:37:38.566112+0000","last_active":"2021-02-10T04:43:54.776641+0000","last_peered":"2021-02-10T04:43:54.776641+0000","last_clean":"2021-02-10T04:43:54.776641+0000","last_became_active":"2021-02-10T04:37:37.247141+0000","last_became_peered":"2021-02-10T04:37:37.247141+0000","last_unstale":"2021-02-10T04:43:54.776641+0000","last_undegraded":"2021-02-10T04:43:54.776641+0000","last_fullsized":"2021-02-10T04:43:54.776641+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'16","last_scrub_stamp":"2021-02-10T04:33:55.288033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:55.288033+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":34,"num_write":28,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"703","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.774165+0000","last_change":"2021-02-10T04:33:05.166788+0000","last_active":"2021-02-10T04:43:54.774165+0000","last_peered":"2021-02-10T04:43:54.774165+0000","last_clean":"2021-02-10T04:43:54.774165+0000","last_became_active":"2021-02-10T04:33:05.166368+0000","last_became_peered":"2021-02-10T04:33:05.166368+0000","last_unstale":"2021-02-10T04:43:54.774165+0000","last_undegraded":"2021-02-10T04:43:54.774165+0000","last_fullsized":"2021-02-10T04:43:54.774165+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:14.204533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:30:14.204533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"966","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.774282+0000","last_change":"2021-02-10T04:29:21.620573+0000","last_active":"2021-02-10T04:43:54.774282+0000","last_peered":"2021-02-10T04:43:54.774282+0000","last_clean":"2021-02-10T04:43:54.774282+0000","last_became_active":"2021-02-10T04:25:49.001782+0000","last_became_peered":"2021-02-10T04:25:49.001782+0000","last_unstale":"2021-02-10T04:43:54.774282+0000","last_undegraded":"2021-02-10T04:43:54.774282+0000","last_fullsized":"2021-02-10T04:43:54.774282+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:21.620482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:21.620482+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"646","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.346593+0000","last_change":"2021-02-10T04:33:05.153205+0000","last_active":"2021-02-10T04:43:53.346593+0000","last_peered":"2021-02-10T04:43:53.346593+0000","last_clean":"2021-02-10T04:43:53.346593+0000","last_became_active":"2021-02-10T04:33:05.152845+0000","last_became_peered":"2021-02-10T04:33:05.152845+0000","last_unstale":"2021-02-10T04:43:53.346593+0000","last_undegraded":"2021-02-10T04:43:53.346593+0000","last_fullsized":"2021-02-10T04:43:53.346593+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.098681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.098681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.5","version":"1000'446","reported_seq":"1906","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.775916+0000","last_change":"2021-02-10T04:43:54.640694+0000","last_active":"2021-02-10T04:43:54.775916+0000","last_peered":"2021-02-10T04:43:54.775916+0000","last_clean":"2021-02-10T04:43:54.775916+0000","last_became_active":"2021-02-10T04:37:37.247692+0000","last_became_peered":"2021-02-10T04:37:37.247692+0000","last_unstale":"2021-02-10T04:43:54.775916+0000","last_undegraded":"2021-02-10T04:43:54.775916+0000","last_fullsized":"2021-02-10T04:43:54.775916+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"683'135","last_scrub_stamp":"2021-02-10T04:38:23.183462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:38:23.183462+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44511704,"num_objects":19,"num_object_clones":17,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":159,"num_read_kb":136871,"num_write":145,"num_write_kb":48690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":807386,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24e","length":"2"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"3"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"5"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2c9","length":"3"},{"start":"2d1","length":"7"},{"start":"2dd","length":"1"},{"start":"2df","length":"6"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2fc","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"307","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"333","length":"1"},{"start":"339","length":"1"},{"start":"33d","length":"1"},{"start":"340","length":"1"},{"start":"342","length":"1"},{"start":"347","length":"2"},{"start":"34a","length":"1"},{"start":"34c","length":"1"},{"start":"34f","length":"3"},{"start":"358","length":"1"},{"start":"364","length":"1"},{"start":"369","length":"1"}]},{"pgid":"1.3","version":"0'0","reported_seq":"726","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.774193+0000","last_change":"2021-02-10T04:33:05.141311+0000","last_active":"2021-02-10T04:43:54.774193+0000","last_peered":"2021-02-10T04:43:54.774193+0000","last_clean":"2021-02-10T04:43:54.774193+0000","last_became_active":"2021-02-10T04:33:05.140966+0000","last_became_peered":"2021-02-10T04:33:05.140966+0000","last_unstale":"2021-02-10T04:43:54.774193+0000","last_undegraded":"2021-02-10T04:43:54.774193+0000","last_fullsized":"2021-02-10T04:43:54.774193+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.928906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.928906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"378","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.346781+0000","last_change":"2021-02-10T04:37:37.235036+0000","last_active":"2021-02-10T04:43:53.346781+0000","last_peered":"2021-02-10T04:43:53.346781+0000","last_clean":"2021-02-10T04:43:53.346781+0000","last_became_active":"2021-02-10T04:37:37.234644+0000","last_became_peered":"2021-02-10T04:37:37.234644+0000","last_unstale":"2021-02-10T04:43:53.346781+0000","last_undegraded":"2021-02-10T04:43:53.346781+0000","last_fullsized":"2021-02-10T04:43:53.346781+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.805432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.805432+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"538'27","reported_seq":"1013","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.774743+0000","last_change":"2021-02-10T04:33:56.837896+0000","last_active":"2021-02-10T04:43:54.774743+0000","last_peered":"2021-02-10T04:43:54.774743+0000","last_clean":"2021-02-10T04:43:54.774743+0000","last_became_active":"2021-02-10T04:25:49.002471+0000","last_became_peered":"2021-02-10T04:25:49.002471+0000","last_unstale":"2021-02-10T04:43:54.774743+0000","last_undegraded":"2021-02-10T04:43:54.774743+0000","last_fullsized":"2021-02-10T04:43:54.774743+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:33:56.837816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:56.837816+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"969","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.773952+0000","last_change":"2021-02-10T04:29:36.663903+0000","last_active":"2021-02-10T04:43:54.773952+0000","last_peered":"2021-02-10T04:43:54.773952+0000","last_clean":"2021-02-10T04:43:54.773952+0000","last_became_active":"2021-02-10T04:29:07.158075+0000","last_became_peered":"2021-02-10T04:29:07.158075+0000","last_unstale":"2021-02-10T04:43:54.773952+0000","last_undegraded":"2021-02-10T04:43:54.773952+0000","last_fullsized":"2021-02-10T04:43:54.773952+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.663840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.663840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"1109","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:55.667568+0000","last_change":"2021-02-10T04:43:55.668271+0000","last_active":"2021-02-10T04:43:55.667568+0000","last_peered":"2021-02-10T04:43:55.667568+0000","last_clean":"2021-02-10T04:43:55.667568+0000","last_became_active":"2021-02-10T04:37:37.245931+0000","last_became_peered":"2021-02-10T04:37:37.245931+0000","last_unstale":"2021-02-10T04:43:55.667568+0000","last_undegraded":"2021-02-10T04:43:55.667568+0000","last_fullsized":"2021-02-10T04:43:55.667568+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:34.482906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:34.482906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"965","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.777778+0000","last_change":"2021-02-10T04:29:35.382459+0000","last_active":"2021-02-10T04:43:54.777778+0000","last_peered":"2021-02-10T04:43:54.777778+0000","last_clean":"2021-02-10T04:43:54.777778+0000","last_became_active":"2021-02-10T04:25:49.010614+0000","last_became_peered":"2021-02-10T04:25:49.010614+0000","last_unstale":"2021-02-10T04:43:54.777778+0000","last_undegraded":"2021-02-10T04:43:54.777778+0000","last_fullsized":"2021-02-10T04:43:54.777778+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:35.382341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.382341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"961","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.348110+0000","last_change":"2021-02-10T04:28:38.503398+0000","last_active":"2021-02-10T04:43:53.348110+0000","last_peered":"2021-02-10T04:43:53.348110+0000","last_clean":"2021-02-10T04:43:53.348110+0000","last_became_active":"2021-02-10T04:25:53.037506+0000","last_became_peered":"2021-02-10T04:25:53.037506+0000","last_unstale":"2021-02-10T04:43:53.348110+0000","last_undegraded":"2021-02-10T04:43:53.348110+0000","last_fullsized":"2021-02-10T04:43:53.348110+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:38.503309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:28:38.503309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"1000'490","reported_seq":"3478","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:55.971132+0000","last_change":"2021-02-10T04:43:55.971132+0000","last_active":"2021-02-10T04:43:55.971132+0000","last_peered":"2021-02-10T04:43:55.971132+0000","last_clean":"2021-02-10T04:43:55.971132+0000","last_became_active":"2021-02-10T04:28:25.150109+0000","last_became_peered":"2021-02-10T04:28:25.150109+0000","last_unstale":"2021-02-10T04:43:55.971132+0000","last_undegraded":"2021-02-10T04:43:55.971132+0000","last_fullsized":"2021-02-10T04:43:55.971132+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'36","last_scrub_stamp":"2021-02-10T04:30:13.863430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:13.863430+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38743254,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":283,"num_read_kb":297137,"num_write":171,"num_write_kb":63465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24e","length":"2"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"3"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"5"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2c9","length":"3"},{"start":"2d1","length":"7"},{"start":"2dd","length":"1"},{"start":"2df","length":"6"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2fc","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"307","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"333","length":"1"},{"start":"339","length":"1"},{"start":"33d","length":"1"},{"start":"340","length":"1"},{"start":"342","length":"1"},{"start":"347","length":"2"},{"start":"34a","length":"1"},{"start":"34c","length":"1"},{"start":"34f","length":"3"},{"start":"358","length":"1"},{"start":"364","length":"1"},{"start":"369","length":"1"}]},{"pgid":"5.6","version":"0'0","reported_seq":"312","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.774542+0000","last_change":"2021-02-10T04:38:39.252779+0000","last_active":"2021-02-10T04:43:54.774542+0000","last_peered":"2021-02-10T04:43:54.774542+0000","last_clean":"2021-02-10T04:43:54.774542+0000","last_became_active":"2021-02-10T04:38:39.252266+0000","last_became_peered":"2021-02-10T04:38:39.252266+0000","last_unstale":"2021-02-10T04:43:54.774542+0000","last_undegraded":"2021-02-10T04:43:54.774542+0000","last_fullsized":"2021-02-10T04:43:54.774542+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.779767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.779767+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"538'33","reported_seq":"734","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.347260+0000","last_change":"2021-02-10T04:33:05.147122+0000","last_active":"2021-02-10T04:43:53.347260+0000","last_peered":"2021-02-10T04:43:53.347260+0000","last_clean":"2021-02-10T04:43:53.347260+0000","last_became_active":"2021-02-10T04:33:05.146695+0000","last_became_peered":"2021-02-10T04:33:05.146695+0000","last_unstale":"2021-02-10T04:43:53.347260+0000","last_undegraded":"2021-02-10T04:43:53.347260+0000","last_fullsized":"2021-02-10T04:43:53.347260+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'29","last_scrub_stamp":"2021-02-10T04:29:25.974280+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.974280+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":38,"num_write":28,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"435","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.776604+0000","last_change":"2021-02-10T04:37:37.258992+0000","last_active":"2021-02-10T04:43:54.776604+0000","last_peered":"2021-02-10T04:43:54.776604+0000","last_clean":"2021-02-10T04:43:54.776604+0000","last_became_active":"2021-02-10T04:37:37.257600+0000","last_became_peered":"2021-02-10T04:37:37.257600+0000","last_unstale":"2021-02-10T04:43:54.776604+0000","last_undegraded":"2021-02-10T04:43:54.776604+0000","last_fullsized":"2021-02-10T04:43:54.776604+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:39.690436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:39.690436+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"789","reported_epoch":"1003","state":"active+clean","last_fresh":"2021-02-10T04:43:58.342071+0000","last_change":"2021-02-10T04:38:33.243623+0000","last_active":"2021-02-10T04:43:58.342071+0000","last_peered":"2021-02-10T04:43:58.342071+0000","last_clean":"2021-02-10T04:43:58.342071+0000","last_became_active":"2021-02-10T04:37:37.249591+0000","last_became_peered":"2021-02-10T04:37:37.249591+0000","last_unstale":"2021-02-10T04:43:58.342071+0000","last_undegraded":"2021-02-10T04:43:58.342071+0000","last_fullsized":"2021-02-10T04:43:58.342071+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:38:33.243519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:33.243519+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":388,"num_read_kb":383,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"889","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.774880+0000","last_change":"2021-02-10T04:29:33.747527+0000","last_active":"2021-02-10T04:43:54.774880+0000","last_peered":"2021-02-10T04:43:54.774880+0000","last_clean":"2021-02-10T04:43:54.774880+0000","last_became_active":"2021-02-10T04:29:07.158869+0000","last_became_peered":"2021-02-10T04:29:07.158869+0000","last_unstale":"2021-02-10T04:43:54.774880+0000","last_undegraded":"2021-02-10T04:43:54.774880+0000","last_fullsized":"2021-02-10T04:43:54.774880+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:33.747448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:33.747448+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.4","version":"1000'620","reported_seq":"4156","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.775628+0000","last_change":"2021-02-10T04:43:53.965548+0000","last_active":"2021-02-10T04:43:54.775628+0000","last_peered":"2021-02-10T04:43:54.775628+0000","last_clean":"2021-02-10T04:43:54.775628+0000","last_became_active":"2021-02-10T04:28:25.151037+0000","last_became_peered":"2021-02-10T04:28:25.151037+0000","last_unstale":"2021-02-10T04:43:54.775628+0000","last_undegraded":"2021-02-10T04:43:54.775628+0000","last_fullsized":"2021-02-10T04:43:54.775628+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'239","last_scrub_stamp":"2021-02-10T04:33:46.270478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:46.270478+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28530390,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":288,"num_read_kb":256965,"num_write":258,"num_write_kb":106887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24e","length":"2"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"3"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"5"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2c9","length":"3"},{"start":"2d1","length":"7"},{"start":"2dd","length":"1"},{"start":"2df","length":"6"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2fc","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"307","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"333","length":"1"},{"start":"339","length":"1"},{"start":"33d","length":"1"},{"start":"340","length":"1"},{"start":"342","length":"1"},{"start":"347","length":"2"},{"start":"34a","length":"1"},{"start":"34c","length":"1"},{"start":"34f","length":"3"},{"start":"358","length":"1"},{"start":"364","length":"1"},{"start":"369","length":"1"}]},{"pgid":"1.2","version":"0'0","reported_seq":"979","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.774578+0000","last_change":"2021-02-10T04:29:29.683867+0000","last_active":"2021-02-10T04:43:54.774578+0000","last_peered":"2021-02-10T04:43:54.774578+0000","last_clean":"2021-02-10T04:43:54.774578+0000","last_became_active":"2021-02-10T04:29:07.159607+0000","last_became_peered":"2021-02-10T04:29:07.159607+0000","last_unstale":"2021-02-10T04:43:54.774578+0000","last_undegraded":"2021-02-10T04:43:54.774578+0000","last_fullsized":"2021-02-10T04:43:54.774578+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.683805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.683805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"1060","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:56.082553+0000","last_change":"2021-02-10T04:43:56.082553+0000","last_active":"2021-02-10T04:43:56.082553+0000","last_peered":"2021-02-10T04:43:56.082553+0000","last_clean":"2021-02-10T04:43:56.082553+0000","last_became_active":"2021-02-10T04:25:51.027624+0000","last_became_peered":"2021-02-10T04:25:51.027624+0000","last_unstale":"2021-02-10T04:43:56.082553+0000","last_undegraded":"2021-02-10T04:43:56.082553+0000","last_fullsized":"2021-02-10T04:43:56.082553+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:43.740853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:28:43.740853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"538'43","reported_seq":"685","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.777201+0000","last_change":"2021-02-10T04:37:38.636607+0000","last_active":"2021-02-10T04:43:54.777201+0000","last_peered":"2021-02-10T04:43:54.777201+0000","last_clean":"2021-02-10T04:43:54.777201+0000","last_became_active":"2021-02-10T04:37:37.244344+0000","last_became_peered":"2021-02-10T04:37:37.244344+0000","last_unstale":"2021-02-10T04:43:54.777201+0000","last_undegraded":"2021-02-10T04:43:54.777201+0000","last_fullsized":"2021-02-10T04:43:54.777201+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'31","last_scrub_stamp":"2021-02-10T04:33:49.423083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:49.423083+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":81,"num_read_kb":57,"num_write":46,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"986","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.343792+0000","last_change":"2021-02-10T04:38:34.289602+0000","last_active":"2021-02-10T04:43:53.343792+0000","last_peered":"2021-02-10T04:43:53.343792+0000","last_clean":"2021-02-10T04:43:53.343792+0000","last_became_active":"2021-02-10T04:28:21.125541+0000","last_became_peered":"2021-02-10T04:28:21.125541+0000","last_unstale":"2021-02-10T04:43:53.343792+0000","last_undegraded":"2021-02-10T04:43:53.343792+0000","last_fullsized":"2021-02-10T04:43:53.343792+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:34.289495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:34.289495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"961","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.777268+0000","last_change":"2021-02-10T04:29:36.396044+0000","last_active":"2021-02-10T04:43:54.777268+0000","last_peered":"2021-02-10T04:43:54.777268+0000","last_clean":"2021-02-10T04:43:54.777268+0000","last_became_active":"2021-02-10T04:25:53.049487+0000","last_became_peered":"2021-02-10T04:25:53.049487+0000","last_unstale":"2021-02-10T04:43:54.777268+0000","last_undegraded":"2021-02-10T04:43:54.777268+0000","last_fullsized":"2021-02-10T04:43:54.777268+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.395943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.395943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"1002'2042","reported_seq":"9186","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:57.858703+0000","last_change":"2021-02-10T04:43:57.858703+0000","last_active":"2021-02-10T04:43:57.858703+0000","last_peered":"2021-02-10T04:43:57.858703+0000","last_clean":"2021-02-10T04:43:57.858703+0000","last_became_active":"2021-02-10T04:28:25.147959+0000","last_became_peered":"2021-02-10T04:28:25.147959+0000","last_unstale":"2021-02-10T04:43:57.858703+0000","last_undegraded":"2021-02-10T04:43:57.858703+0000","last_fullsized":"2021-02-10T04:43:57.858703+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'223","last_scrub_stamp":"2021-02-10T04:30:14.478639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:14.478639+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92600152,"num_objects":40,"num_object_clones":33,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":1,"num_read":1090,"num_read_kb":1063378,"num_write":798,"num_write_kb":306174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24e","length":"2"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"3"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"5"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2c9","length":"3"},{"start":"2d1","length":"7"},{"start":"2dd","length":"1"},{"start":"2df","length":"6"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2fc","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"307","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"333","length":"1"},{"start":"339","length":"1"},{"start":"33d","length":"1"},{"start":"340","length":"1"},{"start":"342","length":"1"},{"start":"347","length":"2"},{"start":"34a","length":"1"},{"start":"34c","length":"1"},{"start":"34f","length":"3"},{"start":"358","length":"1"},{"start":"364","length":"1"},{"start":"369","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":"913","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.343363+0000","last_change":"2021-02-10T04:37:37.219806+0000","last_active":"2021-02-10T04:43:53.343363+0000","last_peered":"2021-02-10T04:43:53.343363+0000","last_clean":"2021-02-10T04:43:53.343363+0000","last_became_active":"2021-02-10T04:37:37.219422+0000","last_became_peered":"2021-02-10T04:37:37.219422+0000","last_unstale":"2021-02-10T04:43:53.343363+0000","last_undegraded":"2021-02-10T04:43:53.343363+0000","last_fullsized":"2021-02-10T04:43:53.343363+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:39.434210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:39.434210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"58'2","reported_seq":"1189","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:57.904682+0000","last_change":"2021-02-10T04:38:27.092813+0000","last_active":"2021-02-10T04:43:57.904682+0000","last_peered":"2021-02-10T04:43:57.904682+0000","last_clean":"2021-02-10T04:43:57.904682+0000","last_became_active":"2021-02-10T04:25:51.026251+0000","last_became_peered":"2021-02-10T04:25:51.026251+0000","last_unstale":"2021-02-10T04:43:57.904682+0000","last_undegraded":"2021-02-10T04:43:57.904682+0000","last_fullsized":"2021-02-10T04:43:57.904682+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:38:27.092697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:38:27.092697+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"538'44","reported_seq":"736","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.343979+0000","last_change":"2021-02-10T04:33:05.142280+0000","last_active":"2021-02-10T04:43:53.343979+0000","last_peered":"2021-02-10T04:43:53.343979+0000","last_clean":"2021-02-10T04:43:53.343979+0000","last_became_active":"2021-02-10T04:33:05.141901+0000","last_became_peered":"2021-02-10T04:33:05.141901+0000","last_unstale":"2021-02-10T04:43:53.343979+0000","last_undegraded":"2021-02-10T04:43:53.343979+0000","last_fullsized":"2021-02-10T04:43:53.343979+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'34","last_scrub_stamp":"2021-02-10T04:28:49.959162+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:28:49.959162+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":60,"num_read_kb":42,"num_write":32,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"729","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.775440+0000","last_change":"2021-02-10T04:37:37.247740+0000","last_active":"2021-02-10T04:43:54.775440+0000","last_peered":"2021-02-10T04:43:54.775440+0000","last_clean":"2021-02-10T04:43:54.775440+0000","last_became_active":"2021-02-10T04:37:37.247405+0000","last_became_peered":"2021-02-10T04:37:37.247405+0000","last_unstale":"2021-02-10T04:43:54.775440+0000","last_undegraded":"2021-02-10T04:43:54.775440+0000","last_fullsized":"2021-02-10T04:43:54.775440+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:34.053323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:34.053323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"976","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.773799+0000","last_change":"2021-02-10T04:38:26.124897+0000","last_active":"2021-02-10T04:43:54.773799+0000","last_peered":"2021-02-10T04:43:54.773799+0000","last_clean":"2021-02-10T04:43:54.773799+0000","last_became_active":"2021-02-10T04:37:37.254482+0000","last_became_peered":"2021-02-10T04:37:37.254482+0000","last_unstale":"2021-02-10T04:43:54.773799+0000","last_undegraded":"2021-02-10T04:43:54.773799+0000","last_fullsized":"2021-02-10T04:43:54.773799+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:26.124805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:38:26.124805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"293'77","reported_seq":"2185","reported_epoch":"1001","state":"active+clean+scrubbing","last_fresh":"2021-02-10T04:43:53.343042+0000","last_change":"2021-02-10T04:38:41.130429+0000","last_active":"2021-02-10T04:43:53.343042+0000","last_peered":"2021-02-10T04:43:53.343042+0000","last_clean":"2021-02-10T04:43:53.343042+0000","last_became_active":"2021-02-10T04:28:25.154434+0000","last_became_peered":"2021-02-10T04:28:25.154434+0000","last_unstale":"2021-02-10T04:43:53.343042+0000","last_undegraded":"2021-02-10T04:43:53.343042+0000","last_fullsized":"2021-02-10T04:43:53.343042+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":140,"stat_sum":{"num_bytes":7546343,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":74349,"num_write":37,"num_write_kb":13263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"986","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.774215+0000","last_change":"2021-02-10T04:29:39.829039+0000","last_active":"2021-02-10T04:43:54.774215+0000","last_peered":"2021-02-10T04:43:54.774215+0000","last_clean":"2021-02-10T04:43:54.774215+0000","last_became_active":"2021-02-10T04:28:49.053127+0000","last_became_peered":"2021-02-10T04:28:49.053127+0000","last_unstale":"2021-02-10T04:43:54.774215+0000","last_undegraded":"2021-02-10T04:43:54.774215+0000","last_fullsized":"2021-02-10T04:43:54.774215+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:39.828975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:39.828975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"385","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.774314+0000","last_change":"2021-02-10T04:38:39.247167+0000","last_active":"2021-02-10T04:43:54.774314+0000","last_peered":"2021-02-10T04:43:54.774314+0000","last_clean":"2021-02-10T04:43:54.774314+0000","last_became_active":"2021-02-10T04:38:39.246766+0000","last_became_peered":"2021-02-10T04:38:39.246766+0000","last_unstale":"2021-02-10T04:43:54.774314+0000","last_undegraded":"2021-02-10T04:43:54.774314+0000","last_fullsized":"2021-02-10T04:43:54.774314+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:20.757194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:20.757194+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"538'33","reported_seq":"1073","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.343315+0000","last_change":"2021-02-10T04:39:02.586279+0000","last_active":"2021-02-10T04:43:53.343315+0000","last_peered":"2021-02-10T04:43:53.343315+0000","last_clean":"2021-02-10T04:43:53.343315+0000","last_became_active":"2021-02-10T04:39:02.560739+0000","last_became_peered":"2021-02-10T04:39:02.560739+0000","last_unstale":"2021-02-10T04:43:53.343315+0000","last_undegraded":"2021-02-10T04:43:53.343315+0000","last_fullsized":"2021-02-10T04:43:53.343315+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'27","last_scrub_stamp":"2021-02-10T04:28:47.476669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:47.476669+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":67,"num_read_kb":50,"num_write":35,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"1095","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.345471+0000","last_change":"2021-02-10T04:37:37.351533+0000","last_active":"2021-02-10T04:43:53.345471+0000","last_peered":"2021-02-10T04:43:53.345471+0000","last_clean":"2021-02-10T04:43:53.345471+0000","last_became_active":"2021-02-10T04:37:37.238701+0000","last_became_peered":"2021-02-10T04:37:37.238701+0000","last_unstale":"2021-02-10T04:43:53.345471+0000","last_undegraded":"2021-02-10T04:43:53.345471+0000","last_fullsized":"2021-02-10T04:43:53.345471+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:33:49.760989+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:33:49.760989+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"961","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.777609+0000","last_change":"2021-02-10T04:29:37.364781+0000","last_active":"2021-02-10T04:43:54.777609+0000","last_peered":"2021-02-10T04:43:54.777609+0000","last_clean":"2021-02-10T04:43:54.777609+0000","last_became_active":"2021-02-10T04:25:53.047736+0000","last_became_peered":"2021-02-10T04:25:53.047736+0000","last_unstale":"2021-02-10T04:43:54.777609+0000","last_undegraded":"2021-02-10T04:43:54.777609+0000","last_fullsized":"2021-02-10T04:43:54.777609+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:37.364697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:37.364697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"1001'1528","reported_seq":"5649","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:55.673378+0000","last_change":"2021-02-10T04:43:55.673378+0000","last_active":"2021-02-10T04:43:55.673378+0000","last_peered":"2021-02-10T04:43:55.673378+0000","last_clean":"2021-02-10T04:43:55.673378+0000","last_became_active":"2021-02-10T04:37:48.215401+0000","last_became_peered":"2021-02-10T04:37:48.215401+0000","last_unstale":"2021-02-10T04:43:55.673378+0000","last_undegraded":"2021-02-10T04:43:55.673378+0000","last_fullsized":"2021-02-10T04:43:55.673378+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":93888272,"num_objects":33,"num_object_clones":30,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":1,"num_read":716,"num_read_kb":703869,"num_write":572,"num_write_kb":222710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":42623785,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b6","length":"1"},{"start":"1c1","length":"1"},{"start":"20a","length":"1"},{"start":"210","length":"3"},{"start":"214","length":"1"},{"start":"21c","length":"1"},{"start":"221","length":"1"},{"start":"224","length":"1"},{"start":"227","length":"1"},{"start":"22a","length":"1"},{"start":"22e","length":"2"},{"start":"231","length":"1"},{"start":"234","length":"1"},{"start":"237","length":"1"},{"start":"23d","length":"1"},{"start":"23f","length":"1"},{"start":"241","length":"2"},{"start":"245","length":"1"},{"start":"248","length":"2"},{"start":"24e","length":"2"},{"start":"251","length":"1"},{"start":"254","length":"2"},{"start":"257","length":"1"},{"start":"25b","length":"1"},{"start":"25d","length":"4"},{"start":"263","length":"1"},{"start":"265","length":"4"},{"start":"26c","length":"2"},{"start":"272","length":"1"},{"start":"274","length":"4"},{"start":"279","length":"2"},{"start":"283","length":"1"},{"start":"287","length":"3"},{"start":"28b","length":"2"},{"start":"28f","length":"1"},{"start":"292","length":"1"},{"start":"294","length":"2"},{"start":"298","length":"1"},{"start":"29d","length":"1"},{"start":"2a1","length":"3"},{"start":"2a5","length":"1"},{"start":"2a7","length":"1"},{"start":"2a9","length":"5"},{"start":"2b1","length":"4"},{"start":"2b6","length":"1"},{"start":"2b8","length":"2"},{"start":"2bd","length":"3"},{"start":"2c1","length":"1"},{"start":"2c7","length":"1"},{"start":"2c9","length":"3"},{"start":"2d1","length":"7"},{"start":"2dd","length":"1"},{"start":"2df","length":"6"},{"start":"2e7","length":"2"},{"start":"2ec","length":"1"},{"start":"2ef","length":"1"},{"start":"2f1","length":"1"},{"start":"2f3","length":"1"},{"start":"2f7","length":"1"},{"start":"2fc","length":"1"},{"start":"2ff","length":"1"},{"start":"302","length":"1"},{"start":"304","length":"1"},{"start":"307","length":"1"},{"start":"309","length":"1"},{"start":"30d","length":"1"},{"start":"311","length":"1"},{"start":"314","length":"1"},{"start":"319","length":"1"},{"start":"31d","length":"1"},{"start":"320","length":"1"},{"start":"325","length":"1"},{"start":"327","length":"1"},{"start":"32b","length":"2"},{"start":"32e","length":"1"},{"start":"331","length":"1"},{"start":"333","length":"1"},{"start":"339","length":"1"},{"start":"33d","length":"1"},{"start":"340","length":"1"},{"start":"342","length":"1"},{"start":"347","length":"2"},{"start":"34a","length":"1"},{"start":"34c","length":"1"},{"start":"34f","length":"3"},{"start":"358","length":"1"},{"start":"364","length":"1"},{"start":"369","length":"1"}]},{"pgid":"4.16","version":"538'34","reported_seq":"1035","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.776860+0000","last_change":"2021-02-10T04:29:25.645462+0000","last_active":"2021-02-10T04:43:54.776860+0000","last_peered":"2021-02-10T04:43:54.776860+0000","last_clean":"2021-02-10T04:43:54.776860+0000","last_became_active":"2021-02-10T04:25:49.008942+0000","last_became_peered":"2021-02-10T04:25:49.008942+0000","last_unstale":"2021-02-10T04:43:54.776860+0000","last_undegraded":"2021-02-10T04:43:54.776860+0000","last_fullsized":"2021-02-10T04:43:54.776860+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'28","last_scrub_stamp":"2021-02-10T04:29:25.645396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.645396+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":71,"num_read_kb":49,"num_write":35,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"394","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.774031+0000","last_change":"2021-02-10T04:38:38.163311+0000","last_active":"2021-02-10T04:43:54.774031+0000","last_peered":"2021-02-10T04:43:54.774031+0000","last_clean":"2021-02-10T04:43:54.774031+0000","last_became_active":"2021-02-10T04:37:37.234649+0000","last_became_peered":"2021-02-10T04:37:37.234649+0000","last_unstale":"2021-02-10T04:43:54.774031+0000","last_undegraded":"2021-02-10T04:43:54.774031+0000","last_fullsized":"2021-02-10T04:43:54.774031+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:38:38.163225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:38.163225+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":247,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"963","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.778283+0000","last_change":"2021-02-10T04:28:33.321856+0000","last_active":"2021-02-10T04:43:54.778283+0000","last_peered":"2021-02-10T04:43:54.778283+0000","last_clean":"2021-02-10T04:43:54.778283+0000","last_became_active":"2021-02-10T04:25:51.030651+0000","last_became_peered":"2021-02-10T04:25:51.030651+0000","last_unstale":"2021-02-10T04:43:54.778283+0000","last_undegraded":"2021-02-10T04:43:54.778283+0000","last_fullsized":"2021-02-10T04:43:54.778283+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:33.321784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:33.321784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"374","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.345093+0000","last_change":"2021-02-10T04:37:37.238450+0000","last_active":"2021-02-10T04:43:53.345093+0000","last_peered":"2021-02-10T04:43:53.345093+0000","last_clean":"2021-02-10T04:43:53.345093+0000","last_became_active":"2021-02-10T04:37:37.238049+0000","last_became_peered":"2021-02-10T04:37:37.238049+0000","last_unstale":"2021-02-10T04:43:53.345093+0000","last_undegraded":"2021-02-10T04:43:53.345093+0000","last_fullsized":"2021-02-10T04:43:53.345093+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.776340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.776340+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"538'19","reported_seq":"1047","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.778128+0000","last_change":"2021-02-10T04:38:26.845289+0000","last_active":"2021-02-10T04:43:54.778128+0000","last_peered":"2021-02-10T04:43:54.778128+0000","last_clean":"2021-02-10T04:43:54.778128+0000","last_became_active":"2021-02-10T04:37:37.247396+0000","last_became_peered":"2021-02-10T04:37:37.247396+0000","last_unstale":"2021-02-10T04:43:54.778128+0000","last_undegraded":"2021-02-10T04:43:54.778128+0000","last_fullsized":"2021-02-10T04:43:54.778128+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'19","last_scrub_stamp":"2021-02-10T04:38:26.845195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:26.845195+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":19,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"980","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.774102+0000","last_change":"2021-02-10T04:38:36.295733+0000","last_active":"2021-02-10T04:43:54.774102+0000","last_peered":"2021-02-10T04:43:54.774102+0000","last_clean":"2021-02-10T04:43:54.774102+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:43:54.774102+0000","last_undegraded":"2021-02-10T04:43:54.774102+0000","last_fullsized":"2021-02-10T04:43:54.774102+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:38:36.295599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:36.295599+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"963","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.347767+0000","last_change":"2021-02-10T04:29:18.429730+0000","last_active":"2021-02-10T04:43:53.347767+0000","last_peered":"2021-02-10T04:43:53.347767+0000","last_clean":"2021-02-10T04:43:53.347767+0000","last_became_active":"2021-02-10T04:25:51.031864+0000","last_became_peered":"2021-02-10T04:25:51.031864+0000","last_unstale":"2021-02-10T04:43:53.347767+0000","last_undegraded":"2021-02-10T04:43:53.347767+0000","last_fullsized":"2021-02-10T04:43:53.347767+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:18.429649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:18.429649+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"889","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.775014+0000","last_change":"2021-02-10T04:29:36.866949+0000","last_active":"2021-02-10T04:43:54.775014+0000","last_peered":"2021-02-10T04:43:54.775014+0000","last_clean":"2021-02-10T04:43:54.775014+0000","last_became_active":"2021-02-10T04:29:07.147249+0000","last_became_peered":"2021-02-10T04:29:07.147249+0000","last_unstale":"2021-02-10T04:43:54.775014+0000","last_undegraded":"2021-02-10T04:43:54.775014+0000","last_fullsized":"2021-02-10T04:43:54.775014+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.866833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.866833+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"538'36","reported_seq":"1171","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.776746+0000","last_change":"2021-02-10T04:37:38.541093+0000","last_active":"2021-02-10T04:43:54.776746+0000","last_peered":"2021-02-10T04:43:54.776746+0000","last_clean":"2021-02-10T04:43:54.776746+0000","last_became_active":"2021-02-10T04:37:37.244921+0000","last_became_peered":"2021-02-10T04:37:37.244921+0000","last_unstale":"2021-02-10T04:43:54.776746+0000","last_undegraded":"2021-02-10T04:43:54.776746+0000","last_fullsized":"2021-02-10T04:43:54.776746+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:33:46.785905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:46.785905+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":36,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"84'7","reported_seq":"992","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.343711+0000","last_change":"2021-02-10T04:30:12.538670+0000","last_active":"2021-02-10T04:43:53.343711+0000","last_peered":"2021-02-10T04:43:53.343711+0000","last_clean":"2021-02-10T04:43:53.343711+0000","last_became_active":"2021-02-10T04:25:36.334366+0000","last_became_peered":"2021-02-10T04:25:36.334366+0000","last_unstale":"2021-02-10T04:43:53.343711+0000","last_undegraded":"2021-02-10T04:43:53.343711+0000","last_fullsized":"2021-02-10T04:43:53.343711+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'7","last_scrub_stamp":"2021-02-10T04:30:12.538549+0000","last_deep_scrub":"84'7","last_deep_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_clean_scrub_stamp":"2021-02-10T04:30:12.538549+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"58'2","reported_seq":"1183","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.776470+0000","last_change":"2021-02-10T04:28:47.496534+0000","last_active":"2021-02-10T04:43:54.776470+0000","last_peered":"2021-02-10T04:43:54.776470+0000","last_clean":"2021-02-10T04:43:54.776470+0000","last_became_active":"2021-02-10T04:25:51.032328+0000","last_became_peered":"2021-02-10T04:25:51.032328+0000","last_unstale":"2021-02-10T04:43:54.776470+0000","last_undegraded":"2021-02-10T04:43:54.776470+0000","last_fullsized":"2021-02-10T04:43:54.776470+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:28:47.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:47.496468+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"967","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.776554+0000","last_change":"2021-02-10T04:38:39.191979+0000","last_active":"2021-02-10T04:43:54.776554+0000","last_peered":"2021-02-10T04:43:54.776554+0000","last_clean":"2021-02-10T04:43:54.776554+0000","last_became_active":"2021-02-10T04:25:53.045177+0000","last_became_peered":"2021-02-10T04:25:53.045177+0000","last_unstale":"2021-02-10T04:43:54.776554+0000","last_undegraded":"2021-02-10T04:43:54.776554+0000","last_fullsized":"2021-02-10T04:43:54.776554+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:39.191910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_clean_scrub_stamp":"2021-02-10T04:38:39.191910+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"538'25","reported_seq":"1013","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.346003+0000","last_change":"2021-02-10T04:28:50.493712+0000","last_active":"2021-02-10T04:43:53.346003+0000","last_peered":"2021-02-10T04:43:53.346003+0000","last_clean":"2021-02-10T04:43:53.346003+0000","last_became_active":"2021-02-10T04:25:49.010890+0000","last_became_peered":"2021-02-10T04:25:49.010890+0000","last_unstale":"2021-02-10T04:43:53.346003+0000","last_undegraded":"2021-02-10T04:43:53.346003+0000","last_fullsized":"2021-02-10T04:43:53.346003+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2021-02-10T04:28:50.493641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:50.493641+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":35,"num_write":26,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"978","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.775115+0000","last_change":"2021-02-10T04:29:19.564845+0000","last_active":"2021-02-10T04:43:54.775115+0000","last_peered":"2021-02-10T04:43:54.775115+0000","last_clean":"2021-02-10T04:43:54.775115+0000","last_became_active":"2021-02-10T04:25:34.321096+0000","last_became_peered":"2021-02-10T04:25:34.321096+0000","last_unstale":"2021-02-10T04:43:54.775115+0000","last_undegraded":"2021-02-10T04:43:54.775115+0000","last_fullsized":"2021-02-10T04:43:54.775115+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:19.564725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:29:19.564725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"639'3","reported_seq":"1196","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:57.904269+0000","last_change":"2021-02-10T04:37:37.922249+0000","last_active":"2021-02-10T04:43:57.904269+0000","last_peered":"2021-02-10T04:43:57.904269+0000","last_clean":"2021-02-10T04:43:57.904269+0000","last_became_active":"2021-02-10T04:37:37.245428+0000","last_became_peered":"2021-02-10T04:37:37.245428+0000","last_unstale":"2021-02-10T04:43:57.904269+0000","last_undegraded":"2021-02-10T04:43:57.904269+0000","last_fullsized":"2021-02-10T04:43:57.904269+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:29:38.343834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.343834+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"969","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.345819+0000","last_change":"2021-02-10T04:37:37.238692+0000","last_active":"2021-02-10T04:43:53.345819+0000","last_peered":"2021-02-10T04:43:53.345819+0000","last_clean":"2021-02-10T04:43:53.345819+0000","last_became_active":"2021-02-10T04:37:37.238360+0000","last_became_peered":"2021-02-10T04:37:37.238360+0000","last_unstale":"2021-02-10T04:43:53.345819+0000","last_undegraded":"2021-02-10T04:43:53.345819+0000","last_fullsized":"2021-02-10T04:43:53.345819+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.934078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.934078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"538'21","reported_seq":"728","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.774861+0000","last_change":"2021-02-10T04:37:38.590027+0000","last_active":"2021-02-10T04:43:54.774861+0000","last_peered":"2021-02-10T04:43:54.774861+0000","last_clean":"2021-02-10T04:43:54.774861+0000","last_became_active":"2021-02-10T04:37:37.249007+0000","last_became_peered":"2021-02-10T04:37:37.249007+0000","last_unstale":"2021-02-10T04:43:54.774861+0000","last_undegraded":"2021-02-10T04:43:54.774861+0000","last_fullsized":"2021-02-10T04:43:54.774861+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:30:11.170034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.170034+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"978","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.774475+0000","last_change":"2021-02-10T04:38:26.127398+0000","last_active":"2021-02-10T04:43:54.774475+0000","last_peered":"2021-02-10T04:43:54.774475+0000","last_clean":"2021-02-10T04:43:54.774475+0000","last_became_active":"2021-02-10T04:25:34.330070+0000","last_became_peered":"2021-02-10T04:25:34.330070+0000","last_unstale":"2021-02-10T04:43:54.774475+0000","last_undegraded":"2021-02-10T04:43:54.774475+0000","last_fullsized":"2021-02-10T04:43:54.774475+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:26.127288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:26.127288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"963","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.777433+0000","last_change":"2021-02-10T04:28:36.311808+0000","last_active":"2021-02-10T04:43:54.777433+0000","last_peered":"2021-02-10T04:43:54.777433+0000","last_clean":"2021-02-10T04:43:54.777433+0000","last_became_active":"2021-02-10T04:25:51.029155+0000","last_became_peered":"2021-02-10T04:25:51.029155+0000","last_unstale":"2021-02-10T04:43:54.777433+0000","last_undegraded":"2021-02-10T04:43:54.777433+0000","last_fullsized":"2021-02-10T04:43:54.777433+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:36.311725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:28:36.311725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"967","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.773714+0000","last_change":"2021-02-10T04:33:49.338303+0000","last_active":"2021-02-10T04:43:54.773714+0000","last_peered":"2021-02-10T04:43:54.773714+0000","last_clean":"2021-02-10T04:43:54.773714+0000","last_became_active":"2021-02-10T04:25:53.039322+0000","last_became_peered":"2021-02-10T04:25:53.039322+0000","last_unstale":"2021-02-10T04:43:54.773714+0000","last_undegraded":"2021-02-10T04:43:54.773714+0000","last_fullsized":"2021-02-10T04:43:54.773714+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:49.338203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:33:49.338203+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"538'23","reported_seq":"1003","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.343231+0000","last_change":"2021-02-10T04:38:32.221752+0000","last_active":"2021-02-10T04:43:53.343231+0000","last_peered":"2021-02-10T04:43:53.343231+0000","last_clean":"2021-02-10T04:43:53.343231+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:43:53.343231+0000","last_undegraded":"2021-02-10T04:43:53.343231+0000","last_fullsized":"2021-02-10T04:43:53.343231+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'23","last_scrub_stamp":"2021-02-10T04:38:32.221656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:32.221656+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"662","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.346091+0000","last_change":"2021-02-10T04:38:36.309455+0000","last_active":"2021-02-10T04:43:53.346091+0000","last_peered":"2021-02-10T04:43:53.346091+0000","last_clean":"2021-02-10T04:43:53.346091+0000","last_became_active":"2021-02-10T04:33:05.147956+0000","last_became_peered":"2021-02-10T04:33:05.147956+0000","last_unstale":"2021-02-10T04:43:53.346091+0000","last_undegraded":"2021-02-10T04:43:53.346091+0000","last_fullsized":"2021-02-10T04:43:53.346091+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:38:36.309362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:36.309362+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"646","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.343934+0000","last_change":"2021-02-10T04:33:05.150906+0000","last_active":"2021-02-10T04:43:53.343934+0000","last_peered":"2021-02-10T04:43:53.343934+0000","last_clean":"2021-02-10T04:43:53.343934+0000","last_became_active":"2021-02-10T04:33:05.150529+0000","last_became_peered":"2021-02-10T04:33:05.150529+0000","last_unstale":"2021-02-10T04:43:53.343934+0000","last_undegraded":"2021-02-10T04:43:53.343934+0000","last_fullsized":"2021-02-10T04:43:53.343934+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:33.026988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:29:33.026988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"961","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.776507+0000","last_change":"2021-02-10T04:29:24.421782+0000","last_active":"2021-02-10T04:43:54.776507+0000","last_peered":"2021-02-10T04:43:54.776507+0000","last_clean":"2021-02-10T04:43:54.776507+0000","last_became_active":"2021-02-10T04:25:53.043277+0000","last_became_peered":"2021-02-10T04:25:53.043277+0000","last_unstale":"2021-02-10T04:43:54.776507+0000","last_undegraded":"2021-02-10T04:43:54.776507+0000","last_fullsized":"2021-02-10T04:43:54.776507+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.421710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.421710+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"538'16","reported_seq":"1124","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.776991+0000","last_change":"2021-02-10T04:38:22.919477+0000","last_active":"2021-02-10T04:43:54.776991+0000","last_peered":"2021-02-10T04:43:54.776991+0000","last_clean":"2021-02-10T04:43:54.776991+0000","last_became_active":"2021-02-10T04:26:39.005616+0000","last_became_peered":"2021-02-10T04:26:39.005616+0000","last_unstale":"2021-02-10T04:43:54.776991+0000","last_undegraded":"2021-02-10T04:43:54.776991+0000","last_fullsized":"2021-02-10T04:43:54.776991+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'16","last_scrub_stamp":"2021-02-10T04:38:22.919377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:22.919377+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":36,"num_read_kb":18,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"981","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.347976+0000","last_change":"2021-02-10T04:37:37.230226+0000","last_active":"2021-02-10T04:43:53.347976+0000","last_peered":"2021-02-10T04:43:53.347976+0000","last_clean":"2021-02-10T04:43:53.347976+0000","last_became_active":"2021-02-10T04:37:37.229852+0000","last_became_peered":"2021-02-10T04:37:37.229852+0000","last_unstale":"2021-02-10T04:43:53.347976+0000","last_undegraded":"2021-02-10T04:43:53.347976+0000","last_fullsized":"2021-02-10T04:43:53.347976+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:49.522124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:49.522124+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"964","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.774082+0000","last_change":"2021-02-10T04:29:23.645783+0000","last_active":"2021-02-10T04:43:54.774082+0000","last_peered":"2021-02-10T04:43:54.774082+0000","last_clean":"2021-02-10T04:43:54.774082+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:43:54.774082+0000","last_undegraded":"2021-02-10T04:43:54.774082+0000","last_fullsized":"2021-02-10T04:43:54.774082+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.645694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.645694+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"377","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.343582+0000","last_change":"2021-02-10T04:37:37.244700+0000","last_active":"2021-02-10T04:43:53.343582+0000","last_peered":"2021-02-10T04:43:53.343582+0000","last_clean":"2021-02-10T04:43:53.343582+0000","last_became_active":"2021-02-10T04:37:37.244082+0000","last_became_peered":"2021-02-10T04:37:37.244082+0000","last_unstale":"2021-02-10T04:43:53.343582+0000","last_undegraded":"2021-02-10T04:43:53.343582+0000","last_fullsized":"2021-02-10T04:43:53.343582+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:31.824925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.824925+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.11","version":"538'29","reported_seq":"1018","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.776749+0000","last_change":"2021-02-10T04:28:31.382776+0000","last_active":"2021-02-10T04:43:54.776749+0000","last_peered":"2021-02-10T04:43:54.776749+0000","last_clean":"2021-02-10T04:43:54.776749+0000","last_became_active":"2021-02-10T04:25:49.007049+0000","last_became_peered":"2021-02-10T04:25:49.007049+0000","last_unstale":"2021-02-10T04:43:54.776749+0000","last_undegraded":"2021-02-10T04:43:54.776749+0000","last_fullsized":"2021-02-10T04:43:54.776749+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2021-02-10T04:28:31.382671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:31.382671+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":29,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"1118","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.775230+0000","last_change":"2021-02-10T04:38:31.166652+0000","last_active":"2021-02-10T04:43:54.775230+0000","last_peered":"2021-02-10T04:43:54.775230+0000","last_clean":"2021-02-10T04:43:54.775230+0000","last_became_active":"2021-02-10T04:37:37.228840+0000","last_became_peered":"2021-02-10T04:37:37.228840+0000","last_unstale":"2021-02-10T04:43:54.775230+0000","last_undegraded":"2021-02-10T04:43:54.775230+0000","last_fullsized":"2021-02-10T04:43:54.775230+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:38:31.166586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:31.166586+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"312","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.777035+0000","last_change":"2021-02-10T04:38:39.246255+0000","last_active":"2021-02-10T04:43:54.777035+0000","last_peered":"2021-02-10T04:43:54.777035+0000","last_clean":"2021-02-10T04:43:54.777035+0000","last_became_active":"2021-02-10T04:38:39.246046+0000","last_became_peered":"2021-02-10T04:38:39.246046+0000","last_unstale":"2021-02-10T04:43:54.777035+0000","last_undegraded":"2021-02-10T04:43:54.777035+0000","last_fullsized":"2021-02-10T04:43:54.777035+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:47.473024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:47.473024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"961","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.777862+0000","last_change":"2021-02-10T04:33:50.809533+0000","last_active":"2021-02-10T04:43:54.777862+0000","last_peered":"2021-02-10T04:43:54.777862+0000","last_clean":"2021-02-10T04:43:54.777862+0000","last_became_active":"2021-02-10T04:25:53.150845+0000","last_became_peered":"2021-02-10T04:25:53.150845+0000","last_unstale":"2021-02-10T04:43:54.777862+0000","last_undegraded":"2021-02-10T04:43:54.777862+0000","last_fullsized":"2021-02-10T04:43:54.777862+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:50.809468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_clean_scrub_stamp":"2021-02-10T04:33:50.809468+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"538'32","reported_seq":"1028","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.777088+0000","last_change":"2021-02-10T04:29:38.663351+0000","last_active":"2021-02-10T04:43:54.777088+0000","last_peered":"2021-02-10T04:43:54.777088+0000","last_clean":"2021-02-10T04:43:54.777088+0000","last_became_active":"2021-02-10T04:25:49.009177+0000","last_became_peered":"2021-02-10T04:25:49.009177+0000","last_unstale":"2021-02-10T04:43:54.777088+0000","last_undegraded":"2021-02-10T04:43:54.777088+0000","last_fullsized":"2021-02-10T04:43:54.777088+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2021-02-10T04:29:38.663272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.663272+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":60,"num_read_kb":42,"num_write":32,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"981","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.343283+0000","last_change":"2021-02-10T04:28:37.515037+0000","last_active":"2021-02-10T04:43:53.343283+0000","last_peered":"2021-02-10T04:43:53.343283+0000","last_clean":"2021-02-10T04:43:53.343283+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:43:53.343283+0000","last_undegraded":"2021-02-10T04:43:53.343283+0000","last_fullsized":"2021-02-10T04:43:53.343283+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:37.514943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:37.514943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"58'2","reported_seq":"1184","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:57.904196+0000","last_change":"2021-02-10T04:29:31.726022+0000","last_active":"2021-02-10T04:43:57.904196+0000","last_peered":"2021-02-10T04:43:57.904196+0000","last_clean":"2021-02-10T04:43:57.904196+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:43:57.904196+0000","last_undegraded":"2021-02-10T04:43:57.904196+0000","last_fullsized":"2021-02-10T04:43:57.904196+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:29:31.725932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.725932+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"906","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.346559+0000","last_change":"2021-02-10T04:38:39.252110+0000","last_active":"2021-02-10T04:43:53.346559+0000","last_peered":"2021-02-10T04:43:53.346559+0000","last_clean":"2021-02-10T04:43:53.346559+0000","last_became_active":"2021-02-10T04:38:39.251336+0000","last_became_peered":"2021-02-10T04:38:39.251336+0000","last_unstale":"2021-02-10T04:43:53.346559+0000","last_undegraded":"2021-02-10T04:43:53.346559+0000","last_fullsized":"2021-02-10T04:43:53.346559+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:26.437591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.437591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"538'43","reported_seq":"1041","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.775323+0000","last_change":"2021-02-10T04:30:13.526264+0000","last_active":"2021-02-10T04:43:54.775323+0000","last_peered":"2021-02-10T04:43:54.775323+0000","last_clean":"2021-02-10T04:43:54.775323+0000","last_became_active":"2021-02-10T04:25:49.015760+0000","last_became_peered":"2021-02-10T04:25:49.015760+0000","last_unstale":"2021-02-10T04:43:54.775323+0000","last_undegraded":"2021-02-10T04:43:54.775323+0000","last_fullsized":"2021-02-10T04:43:54.775323+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'35","last_scrub_stamp":"2021-02-10T04:30:13.526170+0000","last_deep_scrub":"76'35","last_deep_scrub_stamp":"2021-02-10T04:26:58.198641+0000","last_clean_scrub_stamp":"2021-02-10T04:30:13.526170+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":62,"num_read_kb":42,"num_write":32,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"972","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.776559+0000","last_change":"2021-02-10T04:30:11.485059+0000","last_active":"2021-02-10T04:43:54.776559+0000","last_peered":"2021-02-10T04:43:54.776559+0000","last_clean":"2021-02-10T04:43:54.776559+0000","last_became_active":"2021-02-10T04:25:34.324981+0000","last_became_peered":"2021-02-10T04:25:34.324981+0000","last_unstale":"2021-02-10T04:43:54.776559+0000","last_undegraded":"2021-02-10T04:43:54.776559+0000","last_fullsized":"2021-02-10T04:43:54.776559+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:11.485004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.485004+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"973","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.777170+0000","last_change":"2021-02-10T04:37:37.248748+0000","last_active":"2021-02-10T04:43:54.777170+0000","last_peered":"2021-02-10T04:43:54.777170+0000","last_clean":"2021-02-10T04:43:54.777170+0000","last_became_active":"2021-02-10T04:37:37.246998+0000","last_became_peered":"2021-02-10T04:37:37.246998+0000","last_unstale":"2021-02-10T04:43:54.777170+0000","last_undegraded":"2021-02-10T04:43:54.777170+0000","last_fullsized":"2021-02-10T04:43:54.777170+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.589159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.589159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"646","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.346457+0000","last_change":"2021-02-10T04:33:05.142344+0000","last_active":"2021-02-10T04:43:53.346457+0000","last_peered":"2021-02-10T04:43:53.346457+0000","last_clean":"2021-02-10T04:43:53.346457+0000","last_became_active":"2021-02-10T04:33:05.141911+0000","last_became_peered":"2021-02-10T04:33:05.141911+0000","last_unstale":"2021-02-10T04:43:53.346457+0000","last_undegraded":"2021-02-10T04:43:53.346457+0000","last_fullsized":"2021-02-10T04:43:53.346457+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:25.075937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.075937+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"538'27","reported_seq":"933","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.776840+0000","last_change":"2021-02-10T04:37:39.100758+0000","last_active":"2021-02-10T04:43:54.776840+0000","last_peered":"2021-02-10T04:43:54.776840+0000","last_clean":"2021-02-10T04:43:54.776840+0000","last_became_active":"2021-02-10T04:37:37.257293+0000","last_became_peered":"2021-02-10T04:37:37.257293+0000","last_unstale":"2021-02-10T04:43:54.776840+0000","last_undegraded":"2021-02-10T04:43:54.776840+0000","last_fullsized":"2021-02-10T04:43:54.776840+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:28:46.680176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:46.680176+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"981","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.774645+0000","last_change":"2021-02-10T04:38:32.193334+0000","last_active":"2021-02-10T04:43:54.774645+0000","last_peered":"2021-02-10T04:43:54.774645+0000","last_clean":"2021-02-10T04:43:54.774645+0000","last_became_active":"2021-02-10T04:25:36.336442+0000","last_became_peered":"2021-02-10T04:25:36.336442+0000","last_unstale":"2021-02-10T04:43:54.774645+0000","last_undegraded":"2021-02-10T04:43:54.774645+0000","last_fullsized":"2021-02-10T04:43:54.774645+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:38:32.193240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:32.193240+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"645","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.344565+0000","last_change":"2021-02-10T04:33:05.147757+0000","last_active":"2021-02-10T04:43:53.344565+0000","last_peered":"2021-02-10T04:43:53.344565+0000","last_clean":"2021-02-10T04:43:53.344565+0000","last_became_active":"2021-02-10T04:33:05.146838+0000","last_became_peered":"2021-02-10T04:33:05.146838+0000","last_unstale":"2021-02-10T04:43:53.344565+0000","last_undegraded":"2021-02-10T04:43:53.344565+0000","last_fullsized":"2021-02-10T04:43:53.344565+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:22.078247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:22.078247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"960","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.344706+0000","last_change":"2021-02-10T04:29:31.903091+0000","last_active":"2021-02-10T04:43:53.344706+0000","last_peered":"2021-02-10T04:43:53.344706+0000","last_clean":"2021-02-10T04:43:53.344706+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:43:53.344706+0000","last_undegraded":"2021-02-10T04:43:53.344706+0000","last_fullsized":"2021-02-10T04:43:53.344706+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:31.902996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.902996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"538'28","reported_seq":"1020","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.776366+0000","last_change":"2021-02-10T04:38:25.860322+0000","last_active":"2021-02-10T04:43:54.776366+0000","last_peered":"2021-02-10T04:43:54.776366+0000","last_clean":"2021-02-10T04:43:54.776366+0000","last_became_active":"2021-02-10T04:25:49.008997+0000","last_became_peered":"2021-02-10T04:25:49.008997+0000","last_unstale":"2021-02-10T04:43:54.776366+0000","last_undegraded":"2021-02-10T04:43:54.776366+0000","last_fullsized":"2021-02-10T04:43:54.776366+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'28","last_scrub_stamp":"2021-02-10T04:38:25.860247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:25.860247+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":40,"num_write":30,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"984","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.777776+0000","last_change":"2021-02-10T04:28:30.415204+0000","last_active":"2021-02-10T04:43:54.777776+0000","last_peered":"2021-02-10T04:43:54.777776+0000","last_clean":"2021-02-10T04:43:54.777776+0000","last_became_active":"2021-02-10T04:25:36.336893+0000","last_became_peered":"2021-02-10T04:25:36.336893+0000","last_unstale":"2021-02-10T04:43:54.777776+0000","last_undegraded":"2021-02-10T04:43:54.777776+0000","last_fullsized":"2021-02-10T04:43:54.777776+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:30.415134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:26:46.425288+0000","last_clean_scrub_stamp":"2021-02-10T04:28:30.415134+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"971","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.345167+0000","last_change":"2021-02-10T04:37:37.243679+0000","last_active":"2021-02-10T04:43:53.345167+0000","last_peered":"2021-02-10T04:43:53.345167+0000","last_clean":"2021-02-10T04:43:53.345167+0000","last_became_active":"2021-02-10T04:37:37.243275+0000","last_became_peered":"2021-02-10T04:37:37.243275+0000","last_unstale":"2021-02-10T04:43:53.345167+0000","last_undegraded":"2021-02-10T04:43:53.345167+0000","last_fullsized":"2021-02-10T04:43:53.345167+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:52.643099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:33:52.643099+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"961","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.778400+0000","last_change":"2021-02-10T04:29:26.355726+0000","last_active":"2021-02-10T04:43:54.778400+0000","last_peered":"2021-02-10T04:43:54.778400+0000","last_clean":"2021-02-10T04:43:54.778400+0000","last_became_active":"2021-02-10T04:25:53.147831+0000","last_became_peered":"2021-02-10T04:25:53.147831+0000","last_unstale":"2021-02-10T04:43:54.778400+0000","last_undegraded":"2021-02-10T04:43:54.778400+0000","last_fullsized":"2021-02-10T04:43:54.778400+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:26.355652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.355652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"538'31","reported_seq":"1027","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.343758+0000","last_change":"2021-02-10T04:28:32.565457+0000","last_active":"2021-02-10T04:43:53.343758+0000","last_peered":"2021-02-10T04:43:53.343758+0000","last_clean":"2021-02-10T04:43:53.343758+0000","last_became_active":"2021-02-10T04:25:49.000911+0000","last_became_peered":"2021-02-10T04:25:49.000911+0000","last_unstale":"2021-02-10T04:43:53.343758+0000","last_undegraded":"2021-02-10T04:43:53.343758+0000","last_fullsized":"2021-02-10T04:43:53.343758+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2021-02-10T04:28:32.565374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:28:32.565374+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":62,"num_read_kb":46,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"897","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.346644+0000","last_change":"2021-02-10T04:37:37.242121+0000","last_active":"2021-02-10T04:43:53.346644+0000","last_peered":"2021-02-10T04:43:53.346644+0000","last_clean":"2021-02-10T04:43:53.346644+0000","last_became_active":"2021-02-10T04:37:37.241751+0000","last_became_peered":"2021-02-10T04:37:37.241751+0000","last_unstale":"2021-02-10T04:43:53.346644+0000","last_undegraded":"2021-02-10T04:43:53.346644+0000","last_fullsized":"2021-02-10T04:43:53.346644+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:48.646802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:33:48.646802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"972","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.346269+0000","last_change":"2021-02-10T04:38:39.253459+0000","last_active":"2021-02-10T04:43:53.346269+0000","last_peered":"2021-02-10T04:43:53.346269+0000","last_clean":"2021-02-10T04:43:53.346269+0000","last_became_active":"2021-02-10T04:38:39.253122+0000","last_became_peered":"2021-02-10T04:38:39.253122+0000","last_unstale":"2021-02-10T04:43:53.346269+0000","last_undegraded":"2021-02-10T04:43:53.346269+0000","last_fullsized":"2021-02-10T04:43:53.346269+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:46.464108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:46.464108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"967","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.774241+0000","last_change":"2021-02-10T04:38:37.121756+0000","last_active":"2021-02-10T04:43:54.774241+0000","last_peered":"2021-02-10T04:43:54.774241+0000","last_clean":"2021-02-10T04:43:54.774241+0000","last_became_active":"2021-02-10T04:25:53.039844+0000","last_became_peered":"2021-02-10T04:25:53.039844+0000","last_unstale":"2021-02-10T04:43:54.774241+0000","last_undegraded":"2021-02-10T04:43:54.774241+0000","last_fullsized":"2021-02-10T04:43:54.774241+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:37.121654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:38:37.121654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"538'14","reported_seq":"994","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.343076+0000","last_change":"2021-02-10T04:30:13.089576+0000","last_active":"2021-02-10T04:43:53.343076+0000","last_peered":"2021-02-10T04:43:53.343076+0000","last_clean":"2021-02-10T04:43:53.343076+0000","last_became_active":"2021-02-10T04:25:49.013196+0000","last_became_peered":"2021-02-10T04:25:49.013196+0000","last_unstale":"2021-02-10T04:43:53.343076+0000","last_undegraded":"2021-02-10T04:43:53.343076+0000","last_fullsized":"2021-02-10T04:43:53.343076+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2021-02-10T04:30:13.089494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:30:13.089494+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":15,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"889","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.774820+0000","last_change":"2021-02-10T04:30:11.537208+0000","last_active":"2021-02-10T04:43:54.774820+0000","last_peered":"2021-02-10T04:43:54.774820+0000","last_clean":"2021-02-10T04:43:54.774820+0000","last_became_active":"2021-02-10T04:29:07.157944+0000","last_became_peered":"2021-02-10T04:29:07.157944+0000","last_unstale":"2021-02-10T04:43:54.774820+0000","last_undegraded":"2021-02-10T04:43:54.774820+0000","last_fullsized":"2021-02-10T04:43:54.774820+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:11.537136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.537136+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"882","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.343888+0000","last_change":"2021-02-10T04:29:07.146984+0000","last_active":"2021-02-10T04:43:53.343888+0000","last_peered":"2021-02-10T04:43:53.343888+0000","last_clean":"2021-02-10T04:43:53.343888+0000","last_became_active":"2021-02-10T04:29:07.146525+0000","last_became_peered":"2021-02-10T04:29:07.146525+0000","last_unstale":"2021-02-10T04:43:53.343888+0000","last_undegraded":"2021-02-10T04:43:53.343888+0000","last_fullsized":"2021-02-10T04:43:53.343888+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:46.493066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:46.493066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"961","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.343478+0000","last_change":"2021-02-10T04:29:22.475613+0000","last_active":"2021-02-10T04:43:53.343478+0000","last_peered":"2021-02-10T04:43:53.343478+0000","last_clean":"2021-02-10T04:43:53.343478+0000","last_became_active":"2021-02-10T04:25:53.042601+0000","last_became_peered":"2021-02-10T04:25:53.042601+0000","last_unstale":"2021-02-10T04:43:53.343478+0000","last_undegraded":"2021-02-10T04:43:53.343478+0000","last_fullsized":"2021-02-10T04:43:53.343478+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:22.475504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:22.475504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"538'38","reported_seq":"1027","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.342797+0000","last_change":"2021-02-10T04:28:35.520489+0000","last_active":"2021-02-10T04:43:53.342797+0000","last_peered":"2021-02-10T04:43:53.342797+0000","last_clean":"2021-02-10T04:43:53.342797+0000","last_became_active":"2021-02-10T04:25:49.010481+0000","last_became_peered":"2021-02-10T04:25:49.010481+0000","last_unstale":"2021-02-10T04:43:53.342797+0000","last_undegraded":"2021-02-10T04:43:53.342797+0000","last_fullsized":"2021-02-10T04:43:53.342797+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'32","last_scrub_stamp":"2021-02-10T04:28:35.520410+0000","last_deep_scrub":"78'32","last_deep_scrub_stamp":"2021-02-10T04:27:09.170428+0000","last_clean_scrub_stamp":"2021-02-10T04:28:35.520410+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":56,"num_read_kb":37,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"972","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.777190+0000","last_change":"2021-02-10T04:29:29.355356+0000","last_active":"2021-02-10T04:43:54.777190+0000","last_peered":"2021-02-10T04:43:54.777190+0000","last_clean":"2021-02-10T04:43:54.777190+0000","last_became_active":"2021-02-10T04:25:34.327162+0000","last_became_peered":"2021-02-10T04:25:34.327162+0000","last_unstale":"2021-02-10T04:43:54.777190+0000","last_undegraded":"2021-02-10T04:43:54.777190+0000","last_fullsized":"2021-02-10T04:43:54.777190+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.355257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.355257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"646","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.343293+0000","last_change":"2021-02-10T04:33:05.169577+0000","last_active":"2021-02-10T04:43:53.343293+0000","last_peered":"2021-02-10T04:43:53.343293+0000","last_clean":"2021-02-10T04:43:53.343293+0000","last_became_active":"2021-02-10T04:33:05.169159+0000","last_became_peered":"2021-02-10T04:33:05.169159+0000","last_unstale":"2021-02-10T04:43:53.343293+0000","last_undegraded":"2021-02-10T04:43:53.343293+0000","last_fullsized":"2021-02-10T04:43:53.343293+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:40.098239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:40.098239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"1046","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.343684+0000","last_change":"2021-02-10T04:38:23.319098+0000","last_active":"2021-02-10T04:43:53.343684+0000","last_peered":"2021-02-10T04:43:53.343684+0000","last_clean":"2021-02-10T04:43:53.343684+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:43:53.343684+0000","last_undegraded":"2021-02-10T04:43:53.343684+0000","last_fullsized":"2021-02-10T04:43:53.343684+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:23.319023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:38:23.319023+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"538'56","reported_seq":"1121","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.776770+0000","last_change":"2021-02-10T04:38:21.961539+0000","last_active":"2021-02-10T04:43:54.776770+0000","last_peered":"2021-02-10T04:43:54.776770+0000","last_clean":"2021-02-10T04:43:54.776770+0000","last_became_active":"2021-02-10T04:37:37.266384+0000","last_became_peered":"2021-02-10T04:37:37.266384+0000","last_unstale":"2021-02-10T04:43:54.776770+0000","last_undegraded":"2021-02-10T04:43:54.776770+0000","last_fullsized":"2021-02-10T04:43:54.776770+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":641,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'56","last_scrub_stamp":"2021-02-10T04:38:21.961470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:21.961470+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":107,"num_read_kb":84,"num_write":57,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":890,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"982","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.774540+0000","last_change":"2021-02-10T04:29:28.719237+0000","last_active":"2021-02-10T04:43:54.774540+0000","last_peered":"2021-02-10T04:43:54.774540+0000","last_clean":"2021-02-10T04:43:54.774540+0000","last_became_active":"2021-02-10T04:25:36.336112+0000","last_became_peered":"2021-02-10T04:25:36.336112+0000","last_unstale":"2021-02-10T04:43:54.774540+0000","last_undegraded":"2021-02-10T04:43:54.774540+0000","last_fullsized":"2021-02-10T04:43:54.774540+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.719147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.719147+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"312","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.776286+0000","last_change":"2021-02-10T04:38:39.244331+0000","last_active":"2021-02-10T04:43:54.776286+0000","last_peered":"2021-02-10T04:43:54.776286+0000","last_clean":"2021-02-10T04:43:54.776286+0000","last_became_active":"2021-02-10T04:38:39.243965+0000","last_became_peered":"2021-02-10T04:38:39.243965+0000","last_unstale":"2021-02-10T04:43:54.776286+0000","last_undegraded":"2021-02-10T04:43:54.776286+0000","last_fullsized":"2021-02-10T04:43:54.776286+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:33.199521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:38:33.199521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"961","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.776405+0000","last_change":"2021-02-10T04:29:25.382883+0000","last_active":"2021-02-10T04:43:54.776405+0000","last_peered":"2021-02-10T04:43:54.776405+0000","last_clean":"2021-02-10T04:43:54.776405+0000","last_became_active":"2021-02-10T04:25:53.150917+0000","last_became_peered":"2021-02-10T04:25:53.150917+0000","last_unstale":"2021-02-10T04:43:54.776405+0000","last_undegraded":"2021-02-10T04:43:54.776405+0000","last_fullsized":"2021-02-10T04:43:54.776405+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:25.382793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.382793+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.20","version":"0'0","reported_seq":"951","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.343864+0000","last_change":"2021-02-10T04:30:11.492843+0000","last_active":"2021-02-10T04:43:53.343864+0000","last_peered":"2021-02-10T04:43:53.343864+0000","last_clean":"2021-02-10T04:43:53.343864+0000","last_became_active":"2021-02-10T04:29:07.148937+0000","last_became_peered":"2021-02-10T04:29:07.148937+0000","last_unstale":"2021-02-10T04:43:53.343864+0000","last_undegraded":"2021-02-10T04:43:53.343864+0000","last_fullsized":"2021-02-10T04:43:53.343864+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:11.492788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:41.375276+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.492788+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":"465","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.774118+0000","last_change":"2021-02-10T04:37:37.247286+0000","last_active":"2021-02-10T04:43:54.774118+0000","last_peered":"2021-02-10T04:43:54.774118+0000","last_clean":"2021-02-10T04:43:54.774118+0000","last_became_active":"2021-02-10T04:37:37.246865+0000","last_became_peered":"2021-02-10T04:37:37.246865+0000","last_unstale":"2021-02-10T04:43:54.774118+0000","last_undegraded":"2021-02-10T04:43:54.774118+0000","last_fullsized":"2021-02-10T04:43:54.774118+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:21.708701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:21.708701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":"987","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.774067+0000","last_change":"2021-02-10T04:33:58.892594+0000","last_active":"2021-02-10T04:43:54.774067+0000","last_peered":"2021-02-10T04:43:54.774067+0000","last_clean":"2021-02-10T04:43:54.774067+0000","last_became_active":"2021-02-10T04:28:23.129832+0000","last_became_peered":"2021-02-10T04:28:23.129832+0000","last_unstale":"2021-02-10T04:43:54.774067+0000","last_undegraded":"2021-02-10T04:43:54.774067+0000","last_fullsized":"2021-02-10T04:43:54.774067+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:58.892504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:33:58.892504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":"926","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.777222+0000","last_change":"2021-02-10T04:28:48.409915+0000","last_active":"2021-02-10T04:43:54.777222+0000","last_peered":"2021-02-10T04:43:54.777222+0000","last_clean":"2021-02-10T04:43:54.777222+0000","last_became_active":"2021-02-10T04:28:23.129523+0000","last_became_peered":"2021-02-10T04:28:23.129523+0000","last_unstale":"2021-02-10T04:43:54.777222+0000","last_undegraded":"2021-02-10T04:43:54.777222+0000","last_fullsized":"2021-02-10T04:43:54.777222+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:48.409826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:48.409826+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":"931","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.774616+0000","last_change":"2021-02-10T04:34:01.895599+0000","last_active":"2021-02-10T04:43:54.774616+0000","last_peered":"2021-02-10T04:43:54.774616+0000","last_clean":"2021-02-10T04:43:54.774616+0000","last_became_active":"2021-02-10T04:28:23.126368+0000","last_became_peered":"2021-02-10T04:28:23.126368+0000","last_unstale":"2021-02-10T04:43:54.774616+0000","last_undegraded":"2021-02-10T04:43:54.774616+0000","last_fullsized":"2021-02-10T04:43:54.774616+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:34:01.895514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:34:01.895514+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.25","version":"53'2","reported_seq":"942","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.344313+0000","last_change":"2021-02-10T04:28:40.422198+0000","last_active":"2021-02-10T04:43:53.344313+0000","last_peered":"2021-02-10T04:43:53.344313+0000","last_clean":"2021-02-10T04:43:53.344313+0000","last_became_active":"2021-02-10T04:28:23.467770+0000","last_became_peered":"2021-02-10T04:28:23.467770+0000","last_unstale":"2021-02-10T04:43:53.344313+0000","last_undegraded":"2021-02-10T04:43:53.344313+0000","last_fullsized":"2021-02-10T04:43:53.344313+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:28:40.422115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:40.422115+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.26","version":"53'1","reported_seq":"387","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.344532+0000","last_change":"2021-02-10T04:37:37.233712+0000","last_active":"2021-02-10T04:43:53.344532+0000","last_peered":"2021-02-10T04:43:53.344532+0000","last_clean":"2021-02-10T04:43:53.344532+0000","last_became_active":"2021-02-10T04:37:37.231081+0000","last_became_peered":"2021-02-10T04:37:37.231081+0000","last_unstale":"2021-02-10T04:43:53.344532+0000","last_undegraded":"2021-02-10T04:43:53.344532+0000","last_fullsized":"2021-02-10T04:43:53.344532+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:29:22.801929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:22.801929+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":423,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":"958","reported_epoch":"1001","state":"active+clean","last_fresh":"2021-02-10T04:43:53.345259+0000","last_change":"2021-02-10T04:38:37.219069+0000","last_active":"2021-02-10T04:43:53.345259+0000","last_peered":"2021-02-10T04:43:53.345259+0000","last_clean":"2021-02-10T04:43:53.345259+0000","last_became_active":"2021-02-10T04:29:07.145907+0000","last_became_peered":"2021-02-10T04:29:07.145907+0000","last_unstale":"2021-02-10T04:43:53.345259+0000","last_undegraded":"2021-02-10T04:43:53.345259+0000","last_fullsized":"2021-02-10T04:43:53.345259+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:37.218968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:37.218968+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":"926","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.778129+0000","last_change":"2021-02-10T04:29:23.440451+0000","last_active":"2021-02-10T04:43:54.778129+0000","last_peered":"2021-02-10T04:43:54.778129+0000","last_clean":"2021-02-10T04:43:54.778129+0000","last_became_active":"2021-02-10T04:28:23.129469+0000","last_became_peered":"2021-02-10T04:28:23.129469+0000","last_unstale":"2021-02-10T04:43:54.778129+0000","last_undegraded":"2021-02-10T04:43:54.778129+0000","last_fullsized":"2021-02-10T04:43:54.778129+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.440391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.440391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":"931","reported_epoch":"1002","state":"active+clean","last_fresh":"2021-02-10T04:43:54.773949+0000","last_change":"2021-02-10T04:38:22.136377+0000","last_active":"2021-02-10T04:43:54.773949+0000","last_peered":"2021-02-10T04:43:54.773949+0000","last_clean":"2021-02-10T04:43:54.773949+0000","last_became_active":"2021-02-10T04:28:23.126795+0000","last_became_peered":"2021-02-10T04:28:23.126795+0000","last_unstale":"2021-02-10T04:43:54.773949+0000","last_undegraded":"2021-02-10T04:43:54.773949+0000","last_fullsized":"2021-02-10T04:43:54.773949+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:22.136290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_clean_scrub_stamp":"2021-02-10T04:38:22.136290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":900773748,"num_objects":370,"num_object_clones":320,"num_object_copies":1110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":370,"num_whiteouts":11,"num_read":7463,"num_read_kb":7593571,"num_write":5445,"num_write_kb":2087724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":291,"num_bytes_recovered":108939670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1872670720,"data_stored":2551278067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15405,"ondisk_log_size":15405,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":1945,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":18,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":6565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":126,"acting":126,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":388,"num_read_kb":383,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":1876,"num_read_kb":1349,"num_write":1020,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":8770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":75722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1068,"ondisk_log_size":1068,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21,"ondisk_log_size":21,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365022,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":211372,"kb_used_data":168228,"kb_used_omap":0,"kb_used_meta":43136,"kb_avail":93533780,"statfs":{"total":95995035648,"available":95778590720,"internally_reserved":0,"allocated":172265472,"data_stored":253269011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":44171264},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:43:02 2021","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":1.679,"15min":1.482},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.46700000000000003},"max":{"1min":1.8009999999999999,"5min":67.277000000000001,"15min":72.805999999999997},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.96499999999999997,"5min":2.044,"15min":1.6970000000000001},"min":{"1min":0.56599999999999995,"5min":0.54200000000000004,"15min":0.50800000000000001},"max":{"1min":1.829,"5min":66.933999999999997,"15min":72.721999999999994},"last":1.0349999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:43:02 2021","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":1.054,"15min":1.133},"min":{"1min":0.59799999999999998,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.48,"5min":10.140000000000001,"15min":41.966000000000001},"last":1.9830000000000001},{"interface":"front","average":{"1min":0.85999999999999999,"5min":1.0069999999999999,"15min":1.1319999999999999},"min":{"1min":0.60599999999999998,"5min":0.57299999999999995,"15min":0.56499999999999995},"max":{"1min":1.234,"5min":7.1520000000000001,"15min":41.804000000000002},"last":1.1120000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:43:02 2021","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":1.778,"15min":2.125},"min":{"1min":0.628,"5min":0.628,"15min":0.58299999999999996},"max":{"1min":1.375,"5min":66.534000000000006,"15min":125.714},"last":1.228},{"interface":"front","average":{"1min":1.2609999999999999,"5min":2.2269999999999999,"15min":2.3599999999999999},"min":{"1min":0.71099999999999997,"5min":0.66900000000000004,"15min":0.59099999999999997},"max":{"1min":2.218,"5min":66.781999999999996,"15min":125.53700000000001},"last":1.1559999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:43:02 2021","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":2.1749999999999998,"15min":2.944},"min":{"1min":0.85699999999999998,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.728,"5min":68.305000000000007,"15min":192.95500000000001},"last":2.0640000000000001},{"interface":"front","average":{"1min":1.143,"5min":2.0859999999999999,"15min":2.9209999999999998},"min":{"1min":0.82999999999999996,"5min":0.80400000000000005,"15min":0.72199999999999998},"max":{"1min":1.4890000000000001,"5min":69.331999999999994,"15min":192.601},"last":1.349}]},{"osd":4,"last update":"Wed Feb 10 04:43:02 2021","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.6220000000000001,"15min":1.6919999999999999},"min":{"1min":0.79800000000000004,"5min":0.70699999999999996,"15min":0.64600000000000002},"max":{"1min":2.383,"5min":10.781000000000001,"15min":97.733999999999995},"last":3.1920000000000002},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.4119999999999999,"15min":1.639},"min":{"1min":0.78300000000000003,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.847,"5min":10.326000000000001,"15min":97.691000000000003},"last":2.552}]},{"osd":5,"last update":"Wed Feb 10 04:43:02 2021","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.7889999999999999,"15min":1.6080000000000001},"min":{"1min":0.92300000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.0350000000000001,"5min":35.283000000000001,"15min":35.283000000000001},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.5169999999999999,"5min":2.2330000000000001,"15min":1.8959999999999999},"min":{"1min":0.997,"5min":0.86099999999999999,"15min":0.753},"max":{"1min":2.4860000000000002,"5min":35.353999999999999,"15min":35.353999999999999},"last":2.3570000000000002}]},{"osd":6,"last update":"Wed Feb 10 04:43:02 2021","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":2.0150000000000001,"15min":1.97},"min":{"1min":1.028,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":7.79,"5min":11.343999999999999,"15min":32.616},"last":2.2519999999999998},{"interface":"front","average":{"1min":1.7370000000000001,"5min":1.8149999999999999,"15min":1.849},"min":{"1min":1.002,"5min":0.95699999999999996,"15min":0.79500000000000004},"max":{"1min":8.0739999999999998,"5min":10.428000000000001,"15min":32.424999999999997},"last":2.4590000000000001}]}]},{"osd":6,"up_from":43,"seq":184683593955,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":336144,"kb_used_data":292040,"kb_used_omap":0,"kb_used_meta":44096,"kb_avail":93409008,"statfs":{"total":95995035648,"available":95650824192,"internally_reserved":0,"allocated":299048960,"data_stored":391384824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45154304},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:43:57 2021","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.0980000000000001,"15min":1.444},"min":{"1min":0.59099999999999997,"5min":0.47099999999999997,"15min":0.38800000000000001},"max":{"1min":4.0759999999999996,"5min":13.393000000000001,"15min":64.164000000000001},"last":1.145},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.1299999999999999,"15min":1.8500000000000001},"min":{"1min":0.56999999999999995,"5min":0.5,"15min":0.47399999999999998},"max":{"1min":1.8149999999999999,"5min":8.0289999999999999,"15min":64.277000000000001},"last":1.288}]},{"osd":1,"last update":"Wed Feb 10 04:43:57 2021","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.145,"15min":1.1339999999999999},"min":{"1min":0.58699999999999997,"5min":0.54400000000000004,"15min":0.443},"max":{"1min":2.1070000000000002,"5min":12.978999999999999,"15min":18.177},"last":1.5009999999999999},{"interface":"front","average":{"1min":1.115,"5min":1.093,"15min":1.1559999999999999},"min":{"1min":0.63600000000000001,"5min":0.60899999999999999,"15min":0.57799999999999996},"max":{"1min":3.4199999999999999,"5min":3.4199999999999999,"15min":18.898},"last":1.0469999999999999}]},{"osd":2,"last update":"Wed Feb 10 04:43:57 2021","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.2370000000000001,"15min":1.3420000000000001},"min":{"1min":0.58699999999999997,"5min":0.55600000000000005,"15min":0.54100000000000004},"max":{"1min":1.762,"5min":2.6840000000000002,"15min":24.725000000000001},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.294,"15min":1.3420000000000001},"min":{"1min":0.88500000000000001,"5min":0.59899999999999998,"15min":0.55300000000000005},"max":{"1min":2.2330000000000001,"5min":2.2330000000000001,"15min":24.611999999999998},"last":1.6240000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:43:57 2021","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.413,"15min":1.925},"min":{"1min":0.84699999999999998,"5min":0.76400000000000001,"15min":0.63200000000000001},"max":{"1min":2.048,"5min":4.1779999999999999,"15min":114.17700000000001},"last":1.5369999999999999},{"interface":"front","average":{"1min":1.544,"5min":1.5620000000000001,"15min":1.9650000000000001},"min":{"1min":0.98699999999999999,"5min":0.87,"15min":0.71699999999999997},"max":{"1min":2.4950000000000001,"5min":13.106,"15min":116.929},"last":1.7350000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:43:57 2021","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.4590000000000001,"15min":1.3979999999999999},"min":{"1min":0.83399999999999996,"5min":0.81699999999999995,"15min":0.60599999999999998},"max":{"1min":3.141,"5min":4.0810000000000004,"15min":7.5389999999999997},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.105,"15min":1.0600000000000001},"min":{"1min":0.68600000000000005,"5min":0.68500000000000005,"15min":0.57999999999999996},"max":{"1min":2.2400000000000002,"5min":2.2400000000000002,"15min":3.2869999999999999},"last":1.573}]},{"osd":5,"last update":"Wed Feb 10 04:43:57 2021","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.329,"15min":1.331},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":2.153,"5min":2.1970000000000001,"15min":11.380000000000001},"last":1.681},{"interface":"front","average":{"1min":1.8160000000000001,"5min":1.6299999999999999,"15min":1.534},"min":{"1min":1.286,"5min":0.78000000000000003,"15min":0.66600000000000004},"max":{"1min":2.996,"5min":2.996,"15min":11.538},"last":2.0590000000000002}]},{"osd":7,"last update":"Wed Feb 10 04:43:18 2021","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.353,"15min":1.3180000000000001},"min":{"1min":0.83899999999999997,"5min":0.755,"15min":0.755},"max":{"1min":1.845,"5min":4.5279999999999996,"15min":4.5279999999999996},"last":1.7809999999999999},{"interface":"front","average":{"1min":1.768,"5min":1.752,"15min":1.611},"min":{"1min":1.258,"5min":0.91000000000000003,"15min":0.77500000000000002},"max":{"1min":2.7240000000000002,"5min":13.315,"15min":13.315},"last":2.327}]}]},{"osd":1,"up_from":13,"seq":55834575092,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":376300,"kb_used_data":333604,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":93368852,"statfs":{"total":95995035648,"available":95609704448,"internally_reserved":0,"allocated":341610496,"data_stored":442341063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:43:24 2021","interfaces":[{"interface":"back","average":{"1min":2.399,"5min":1.881,"15min":1.5669999999999999},"min":{"1min":0.55900000000000005,"5min":0.50600000000000001,"15min":0.47899999999999998},"max":{"1min":31.198,"5min":55.475999999999999,"15min":55.475999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":2.423,"5min":1.958,"15min":1.651},"min":{"1min":0.53300000000000003,"5min":0.53000000000000003,"15min":0.51800000000000002},"max":{"1min":31.558,"5min":55.368000000000002,"15min":55.368000000000002},"last":0.85499999999999998}]},{"osd":2,"last update":"Wed Feb 10 04:43:47 2021","interfaces":[{"interface":"back","average":{"1min":2.9729999999999999,"5min":1.829,"15min":1.5629999999999999},"min":{"1min":0.57499999999999996,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":30.585000000000001,"5min":32.110999999999997,"15min":40.268000000000001},"last":1.0089999999999999},{"interface":"front","average":{"1min":2.9830000000000001,"5min":1.8280000000000001,"15min":1.5940000000000001},"min":{"1min":0.59899999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":30.167999999999999,"5min":32.302999999999997,"15min":40.651000000000003},"last":0.79100000000000004}]},{"osd":3,"last update":"Wed Feb 10 04:43:03 2021","interfaces":[{"interface":"back","average":{"1min":2.6850000000000001,"5min":2.0289999999999999,"15min":1.734},"min":{"1min":0.69499999999999995,"5min":0.57899999999999996,"15min":0.45200000000000001},"max":{"1min":30.274999999999999,"5min":32.432000000000002,"15min":40.314},"last":1.3859999999999999},{"interface":"front","average":{"1min":2.6819999999999999,"5min":2.0209999999999999,"15min":1.7270000000000001},"min":{"1min":0.65500000000000003,"5min":0.60399999999999998,"15min":0.47199999999999998},"max":{"1min":31.443000000000001,"5min":32.811,"15min":40.973999999999997},"last":1.099}]},{"osd":4,"last update":"Wed Feb 10 04:43:24 2021","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.256,"15min":1.2549999999999999},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":3.258,"5min":3.7440000000000002,"15min":6.3099999999999996},"last":0.70899999999999996},{"interface":"front","average":{"1min":2.6560000000000001,"5min":1.647,"15min":1.444},"min":{"1min":0.84799999999999998,"5min":0.754,"15min":0.70999999999999996},"max":{"1min":29.547999999999998,"5min":29.547999999999998,"15min":29.547999999999998},"last":1.2270000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:43:38 2021","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.2709999999999999,"15min":1.5129999999999999},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":2.5699999999999998,"5min":7.8049999999999997,"15min":48.686999999999998},"last":1.7749999999999999},{"interface":"front","average":{"1min":2.8039999999999998,"5min":1.7789999999999999,"15min":1.804},"min":{"1min":0.98799999999999999,"5min":0.749,"15min":0.69699999999999995},"max":{"1min":28.678000000000001,"5min":28.678000000000001,"15min":48.905000000000001},"last":1.056}]},{"osd":6,"last update":"Wed Feb 10 04:43:47 2021","interfaces":[{"interface":"back","average":{"1min":3.1949999999999998,"5min":1.8740000000000001,"15min":1.788},"min":{"1min":0.97899999999999998,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":29.686,"5min":29.686,"15min":48.878999999999998},"last":1.6080000000000001},{"interface":"front","average":{"1min":3.085,"5min":1.649,"15min":1.6850000000000001},"min":{"1min":0.97699999999999998,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":28.475000000000001,"5min":28.475000000000001,"15min":49.011000000000003},"last":1.643}]},{"osd":7,"last update":"Wed Feb 10 04:43:03 2021","interfaces":[{"interface":"back","average":{"1min":2.7669999999999999,"5min":1.772,"15min":1.601},"min":{"1min":1.002,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":28.573,"5min":28.573,"15min":28.573},"last":0.96299999999999997},{"interface":"front","average":{"1min":2.9510000000000001,"5min":1.847,"15min":1.6459999999999999},"min":{"1min":1.119,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":29.408999999999999,"5min":29.408999999999999,"15min":29.408999999999999},"last":1.4790000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738618,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":349488,"kb_used_data":297832,"kb_used_omap":0,"kb_used_meta":51648,"kb_avail":93395664,"statfs":{"total":95995035648,"available":95637159936,"internally_reserved":0,"allocated":304979968,"data_stored":427010362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52887552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 04:43:16 2021","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":0.95999999999999996,"15min":1.1000000000000001},"min":{"1min":0.54600000000000004,"5min":0.50900000000000001,"15min":0.48099999999999998},"max":{"1min":2.4159999999999999,"5min":2.4340000000000002,"15min":28.946999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":1.181,"5min":1.091,"15min":1.014},"min":{"1min":0.52700000000000002,"5min":0.52100000000000002,"15min":0.51200000000000001},"max":{"1min":3.516,"5min":3.7050000000000001,"15min":7.5549999999999997},"last":0.78800000000000003}]},{"osd":2,"last update":"Wed Feb 10 04:43:51 2021","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.22,"15min":1.3460000000000001},"min":{"1min":0.63300000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":3.903,"5min":3.903,"15min":25.984999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.22,"15min":1.1899999999999999},"min":{"1min":0.63100000000000001,"5min":0.56999999999999995,"15min":0.51300000000000001},"max":{"1min":3.2429999999999999,"5min":3.4580000000000002,"15min":16.829000000000001},"last":0.68700000000000006}]},{"osd":3,"last update":"Wed Feb 10 04:43:05 2021","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.484,"15min":1.8260000000000001},"min":{"1min":0.61199999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":4.5830000000000002,"5min":29.692,"15min":126.587},"last":0.92500000000000004},{"interface":"front","average":{"1min":1.639,"5min":1.6950000000000001,"15min":2.1150000000000002},"min":{"1min":0.83399999999999996,"5min":0.79400000000000004,"15min":0.54200000000000004},"max":{"1min":3.6459999999999999,"5min":25.315000000000001,"15min":126.88200000000001},"last":1.1899999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:43:16 2021","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.456,"15min":1.327},"min":{"1min":0.85299999999999998,"5min":0.67200000000000004,"15min":0.59399999999999997},"max":{"1min":2.9430000000000001,"5min":5.1680000000000001,"15min":5.1680000000000001},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.276,"5min":1.2649999999999999,"15min":1.337},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.63800000000000001},"max":{"1min":2.8690000000000002,"5min":4.5810000000000004,"15min":14.672000000000001},"last":0.84199999999999997}]},{"osd":5,"last update":"Wed Feb 10 04:43:36 2021","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.6930000000000001,"15min":1.5629999999999999},"min":{"1min":0.80900000000000005,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":3.3039999999999998,"5min":25.047000000000001,"15min":25.047000000000001},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.758,"5min":1.837,"15min":1.819},"min":{"1min":1.0580000000000001,"5min":0.97799999999999998,"15min":0.88800000000000001},"max":{"1min":4.1200000000000001,"5min":24.972999999999999,"15min":28.827999999999999},"last":1.383}]},{"osd":6,"last update":"Wed Feb 10 04:42:59 2021","interfaces":[{"interface":"back","average":{"1min":2.6349999999999998,"5min":1.849,"15min":1.8580000000000001},"min":{"1min":1.05,"5min":0.89100000000000001,"15min":0.88800000000000001},"max":{"1min":24.837,"5min":24.837,"15min":35.872999999999998},"last":1.22},{"interface":"front","average":{"1min":2.8340000000000001,"5min":1.869,"15min":1.984},"min":{"1min":1.1759999999999999,"5min":0.96099999999999997,"15min":0.83099999999999996},"max":{"1min":24.870999999999999,"5min":24.870999999999999,"15min":35.911000000000001},"last":1.2749999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:43:05 2021","interfaces":[{"interface":"back","average":{"1min":1.732,"5min":1.6279999999999999,"15min":1.518},"min":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.73499999999999999},"max":{"1min":3.121,"5min":3.472,"15min":5.6050000000000004},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.6899999999999999,"5min":1.609,"15min":1.728},"min":{"1min":0.97299999999999998,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":3.4300000000000002,"5min":5.1299999999999999,"15min":16.545000000000002},"last":1.075}]}]},{"osd":2,"up_from":0,"seq":77309411570,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":25,"seq":107374182637,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":445916,"kb_used_data":387028,"kb_used_omap":0,"kb_used_meta":58880,"kb_avail":93299236,"statfs":{"total":95995035648,"available":95538417664,"internally_reserved":0,"allocated":396316672,"data_stored":558424472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":60293120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:43:07 2021","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":1.0620000000000001,"15min":1.0760000000000001},"min":{"1min":0.627,"5min":0.53300000000000003,"15min":0.47599999999999998},"max":{"1min":0.97399999999999998,"5min":7.899,"15min":17.233000000000001},"last":1.256},{"interface":"front","average":{"1min":1.018,"5min":1.129,"15min":1.202},"min":{"1min":0.61099999999999999,"5min":0.53600000000000003,"15min":0.52300000000000002},"max":{"1min":2.6640000000000001,"5min":3.7330000000000001,"15min":20.216999999999999},"last":1.214}]},{"osd":1,"last update":"Wed Feb 10 04:43:07 2021","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.075,"15min":1.1060000000000001},"min":{"1min":0.745,"5min":0.56599999999999995,"15min":0.54400000000000004},"max":{"1min":2.4460000000000002,"5min":3.9550000000000001,"15min":8.9190000000000005},"last":1.296},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.1180000000000001,"15min":1.04},"min":{"1min":0.64400000000000002,"5min":0.55600000000000005,"15min":0.52100000000000002},"max":{"1min":1.9410000000000001,"5min":2.5390000000000001,"15min":4.6769999999999996},"last":0.84699999999999998}]},{"osd":2,"last update":"Wed Feb 10 04:43:07 2021","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.4139999999999999,"15min":1.2789999999999999},"min":{"1min":0.68999999999999995,"5min":0.624,"15min":0.56799999999999995},"max":{"1min":2.4289999999999998,"5min":4.1900000000000004,"15min":5.024},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.21,"5min":1.2390000000000001,"15min":1.179},"min":{"1min":0.67100000000000004,"5min":0.65200000000000002,"15min":0.56999999999999995},"max":{"1min":2.1709999999999998,"5min":4.5880000000000001,"15min":4.5880000000000001},"last":1.1279999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:43:15 2021","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.4039999999999999,"15min":1.5900000000000001},"min":{"1min":0.85499999999999998,"5min":0.80700000000000005,"15min":0.629},"max":{"1min":2.5390000000000001,"5min":4.6909999999999998,"15min":30.751999999999999},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.536,"15min":1.663},"min":{"1min":0.93799999999999994,"5min":0.85999999999999999,"15min":0.65700000000000003},"max":{"1min":2.6970000000000001,"5min":4.4699999999999998,"15min":30.928000000000001},"last":1.6890000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:43:36 2021","interfaces":[{"interface":"back","average":{"1min":2.653,"5min":1.843,"15min":1.9730000000000001},"min":{"1min":1.1120000000000001,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":18.356999999999999,"5min":18.356999999999999,"15min":35.616999999999997},"last":1.8939999999999999},{"interface":"front","average":{"1min":2.665,"5min":1.958,"15min":2.04},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.77200000000000002},"max":{"1min":18.100000000000001,"5min":18.100000000000001,"15min":35.661999999999999},"last":2.3580000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:43:58 2021","interfaces":[{"interface":"back","average":{"1min":2.5350000000000001,"5min":1.8620000000000001,"15min":2.1059999999999999},"min":{"1min":0.94799999999999995,"5min":0.88500000000000001,"15min":0.88500000000000001},"max":{"1min":18.614999999999998,"5min":18.614999999999998,"15min":35.518000000000001},"last":1.591},{"interface":"front","average":{"1min":2.5409999999999999,"5min":1.9950000000000001,"15min":2.3439999999999999},"min":{"1min":1.0509999999999999,"5min":1.0229999999999999,"15min":0.86599999999999999},"max":{"1min":18.036999999999999,"5min":18.036999999999999,"15min":53.965000000000003},"last":2.2530000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:43:07 2021","interfaces":[{"interface":"back","average":{"1min":1.7330000000000001,"5min":1.7689999999999999,"15min":1.845},"min":{"1min":1.256,"5min":1.1830000000000001,"15min":0.99199999999999999},"max":{"1min":2.7930000000000001,"5min":3.4380000000000002,"15min":25.242999999999999},"last":1.7769999999999999},{"interface":"front","average":{"1min":2.0539999999999998,"5min":1.903,"15min":1.919},"min":{"1min":1.3,"5min":1.048,"15min":0.98299999999999998},"max":{"1min":5.0270000000000001,"5min":5.1779999999999999,"15min":25.286999999999999},"last":2.1150000000000002}]}]},{"osd":4,"up_from":31,"seq":133143986410,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":160160,"kb_used_data":119320,"kb_used_omap":0,"kb_used_meta":40832,"kb_avail":93584992,"statfs":{"total":95995035648,"available":95831031808,"internally_reserved":0,"allocated":122183680,"data_stored":151507764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41811968},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4],"upper_bound":2097152},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:43:18 2021","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.6339999999999999,"15min":1.3640000000000001},"min":{"1min":0.503,"5min":0.503,"15min":0.48699999999999999},"max":{"1min":5.3419999999999996,"5min":43.939999999999998,"15min":62.273000000000003},"last":0.623},{"interface":"front","average":{"1min":0.98499999999999999,"5min":1.645,"15min":1.4099999999999999},"min":{"1min":0.60099999999999998,"5min":0.56599999999999995,"15min":0.499},"max":{"1min":1.7010000000000001,"5min":43.829999999999998,"15min":62.375999999999998},"last":0.91400000000000003}]},{"osd":1,"last update":"Wed Feb 10 04:43:18 2021","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":1.0700000000000001,"15min":0.91700000000000004},"min":{"1min":0.61699999999999999,"5min":0.54300000000000004,"15min":0.52800000000000002},"max":{"1min":1.3580000000000001,"5min":24.295000000000002,"15min":24.295000000000002},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.5089999999999999,"15min":1.2030000000000001},"min":{"1min":0.63600000000000001,"5min":0.61099999999999999,"15min":0.59999999999999998},"max":{"1min":6.5060000000000002,"5min":24.943000000000001,"15min":24.943000000000001},"last":0.94999999999999996}]},{"osd":2,"last update":"Wed Feb 10 04:43:18 2021","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.1619999999999999,"15min":1.3939999999999999},"min":{"1min":0.66000000000000003,"5min":0.61399999999999999,"15min":0.56499999999999995},"max":{"1min":1.853,"5min":7.048,"15min":62.521000000000001},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.101,"5min":1.1719999999999999,"15min":1.4610000000000001},"min":{"1min":0.68200000000000005,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":2.1280000000000001,"5min":2.1280000000000001,"15min":62.572000000000003},"last":1.0629999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:43:18 2021","interfaces":[{"interface":"back","average":{"1min":1.639,"5min":1.5940000000000001,"15min":1.6040000000000001},"min":{"1min":0.627,"5min":0.627,"15min":0.60699999999999998},"max":{"1min":8.2119999999999997,"5min":24.530999999999999,"15min":62.113},"last":1.319},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.5640000000000001,"15min":1.5529999999999999},"min":{"1min":0.79300000000000004,"5min":0.72399999999999998,"15min":0.70299999999999996},"max":{"1min":8.2469999999999999,"5min":24.669,"15min":62.146000000000001},"last":1.2569999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:43:36 2021","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.383,"15min":1.373},"min":{"1min":0.68200000000000005,"5min":0.63,"15min":0.60999999999999999},"max":{"1min":2.4769999999999999,"5min":3.3719999999999999,"15min":12.773999999999999},"last":0.78300000000000003},{"interface":"front","average":{"1min":1.204,"5min":1.3240000000000001,"15min":1.294},"min":{"1min":0.77200000000000002,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":2.2320000000000002,"5min":2.6709999999999998,"15min":13.038},"last":0.879}]},{"osd":6,"last update":"Wed Feb 10 04:43:49 2021","interfaces":[{"interface":"back","average":{"1min":1.5840000000000001,"5min":1.595,"15min":1.778},"min":{"1min":0.995,"5min":0.76600000000000001,"15min":0.70299999999999996},"max":{"1min":1.9950000000000001,"5min":7.1660000000000004,"15min":42.088999999999999},"last":2.1230000000000002},{"interface":"front","average":{"1min":1.6779999999999999,"5min":1.647,"15min":1.881},"min":{"1min":1.03,"5min":0.86799999999999999,"15min":0.83899999999999997},"max":{"1min":2.6909999999999998,"5min":3.0680000000000001,"15min":42.027000000000001},"last":1.9359999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:43:13 2021","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.4710000000000001,"15min":1.431},"min":{"1min":0.874,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":2.2080000000000002,"5min":6.9199999999999999,"15min":6.9199999999999999},"last":1.014},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.5900000000000001,"15min":1.5389999999999999},"min":{"1min":0.93799999999999994,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.9239999999999999,"5min":4.1399999999999997,"15min":4.1399999999999997},"last":1.2310000000000001}]}]},{"osd":5,"up_from":37,"seq":158913790183,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":254708,"kb_used_data":209132,"kb_used_omap":0,"kb_used_meta":45568,"kb_avail":93490444,"statfs":{"total":95995035648,"available":95734214656,"internally_reserved":0,"allocated":214151168,"data_stored":283424362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":46661632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:43:35 2021","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":0.96699999999999997,"15min":1.3859999999999999},"min":{"1min":0.57999999999999996,"5min":0.47599999999999998,"15min":0.40500000000000003},"max":{"1min":1.6020000000000001,"5min":4.5720000000000001,"15min":104.04300000000001},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.91600000000000004,"5min":1.135,"15min":1.4610000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.46400000000000002},"max":{"1min":1.754,"5min":15.539999999999999,"15min":107.776},"last":0.65200000000000002}]},{"osd":1,"last update":"Wed Feb 10 04:43:35 2021","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":1.004,"15min":1.3120000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.504},"max":{"1min":1.0309999999999999,"5min":5.29,"15min":103.904},"last":1.369},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.2030000000000001,"15min":1.4810000000000001},"min":{"1min":0.72799999999999998,"5min":0.57599999999999996,"15min":0.50800000000000001},"max":{"1min":1.54,"5min":11.875999999999999,"15min":104.315},"last":1.8180000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:43:35 2021","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.637,"15min":2.254},"min":{"1min":0.63600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.8049999999999999,"5min":44.462000000000003,"15min":103.65600000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.611,"15min":2.214},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.63500000000000001},"max":{"1min":1.2609999999999999,"5min":44.398000000000003,"15min":104.136},"last":1.45}]},{"osd":3,"last update":"Wed Feb 10 04:43:35 2021","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.742,"15min":2.2599999999999998},"min":{"1min":0.70499999999999996,"5min":0.66100000000000003,"15min":0.624},"max":{"1min":2.0899999999999999,"5min":44.609000000000002,"15min":106.18000000000001},"last":1.393},{"interface":"front","average":{"1min":1.6539999999999999,"5min":1.9730000000000001,"15min":2.2709999999999999},"min":{"1min":0.76500000000000001,"5min":0.752,"15min":0.64500000000000002},"max":{"1min":9.3949999999999996,"5min":44.537999999999997,"15min":106.822},"last":1.5580000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:43:35 2021","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.3100000000000001,"15min":1.2130000000000001},"min":{"1min":0.71299999999999997,"5min":0.57099999999999995,"15min":0.55400000000000005},"max":{"1min":1.6160000000000001,"5min":6.1639999999999997,"15min":6.1639999999999997},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.391,"15min":1.3600000000000001},"min":{"1min":0.76400000000000001,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":2.1989999999999998,"5min":2.895,"15min":8.1449999999999996},"last":3.0209999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:43:48 2021","interfaces":[{"interface":"back","average":{"1min":19.244,"5min":5.1769999999999996,"15min":2.831},"min":{"1min":1.0880000000000001,"5min":0.65800000000000003,"15min":0.63500000000000001},"max":{"1min":319.51999999999998,"5min":319.51999999999998,"15min":319.51999999999998},"last":1.619},{"interface":"front","average":{"1min":19.132000000000001,"5min":5.0679999999999996,"15min":2.6859999999999999},"min":{"1min":0.91300000000000003,"5min":0.77900000000000003,"15min":0.74399999999999999},"max":{"1min":319.22300000000001,"5min":319.22300000000001,"15min":319.22300000000001},"last":1.6779999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:43:10 2021","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.4810000000000001,"15min":1.397},"min":{"1min":1.014,"5min":0.752,"15min":0.752},"max":{"1min":2.0910000000000002,"5min":3.0009999999999999,"15min":5.4770000000000003},"last":1.5800000000000001},{"interface":"front","average":{"1min":1.669,"5min":1.6399999999999999,"15min":1.6020000000000001},"min":{"1min":0.98399999999999999,"5min":0.92300000000000004,"15min":0.83499999999999996},"max":{"1min":2.992,"5min":3.1040000000000001,"15min":8.7840000000000007},"last":0.996}]}]}]}} 2021-02-10T04:44:05.850 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2021-02-10T04:44:05.850 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pool set unique_pool_0 pg_num 26 2021-02-10T04:44:06.144 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:44:05 smithi164 conmon[37263]: debug 2021-02-10T04:44:06.144 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:44:05 smithi164 conmon[37263]: 2021-02-10T04:44:05.700+0000 7f1c671e2700 -1 osd.5 pg_epoch: 1003 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x5566508fc580) 2021-02-10T04:44:06.144 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:44:05 smithi164 conmon[37263]: 2021-02-10T04:44:06.254 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:44:07.764 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:07 smithi164 conmon[29288]: cluster 2021-02-10T04:44:06.270202+0000 mgr.y (mgr.14140) 928 : 2021-02-10T04:44:07.764 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:07 smithi164 conmon[29288]: cluster [DBG] pgmap v1667: 168 pgs: 1 active+clean+scrubbing, 165 active+clean, 2 active+clean+snaptrim; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:44:07.764 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:07 smithi164 conmon[29288]: cluster 2021-02-10T04:44:06.817223+0000 osd.5 (osd.5) 88 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:44:07.764 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:07 smithi164 conmon[29288]: cluster 2021-02-10T04:44:06.818302+0000 osd.5 (osd.5) 89 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:44:07.764 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:44:07 smithi164 conmon[37263]: debug 2021-02-10T04:44:07.762+0000 7f1c671e2700 -1 osd.5 pg_epoch: 1004 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x5566566b8f20) 2021-02-10T04:44:07.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:07 smithi085 conmon[29319]: cluster 2021-02-10T04:44:06.270202+0000 mgr.y (mgr.14140) 928 : cluster [DBG] pgmap v1667: 168 pgs: 1 active+clean+scrubbing, 165 active+clean, 2 active+clean+snaptrim; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:44:07.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:07 smithi085 conmon[29319]: cluster 2021-02-10T04:44:06.817223+0000 osd.5 (osd.5) 88 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:44:07.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:07 smithi085 conmon[29319]: cluster 2021-02-10T04:44:06.818302+0000 osd.5 (osd.5) 89 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:44:07.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:07 smithi085 conmon[35961]: cluster 2021-02-10T04:44:07.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:07 smithi085 conmon[35961]: 2021-02-10T04:44:07.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:07 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:44:07.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:07 smithi085 conmon[35961]: 44:06.270202+0000 mgr.y (mgr.14140) 928 : cluster [DBG] pgmap v1667: 168 pgs: 1 active+clean+scrubbing, 165 active+clean, 2 active+clean+snaptrim; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:44:07.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:07 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:44:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:07 smithi085 conmon[35961]: :44:06.817223+0000 osd.5 (osd.5) 2021-02-10T04:44:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:07 smithi085 conmon[35961]: 88 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:44:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:07 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:44:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:07 smithi085 conmon[35961]: -02-10T04: 2021-02-10T04:44:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:07 smithi085 conmon[35961]: 44:06.818302+0000 osd.5 (osd.5) 2021-02-10T04:44:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:07 smithi085 conmon[35961]: 89 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:44:08.295 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:44:08.331 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:44:08.474 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:08 smithi164 conmon[29288]: cluster 2021-02-10T04:44:07.454266+0000 mon.a ( 2021-02-10T04:44:08.475 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:08 smithi164 conmon[29288]: mon.0) 2005 : cluster [DBG] osdmap e1004: 8 total, 8 up, 7 in 2021-02-10T04:44:08.514 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:44:08.618 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:44:08.753 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:44:08 smithi164 conmon[40892]: debug 2021-02-10T04:44:08.479+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:44:08.753 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:44:08 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:44:09.000 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:08 smithi085 conmon[29319]: cluster 2021-02-10T04:44:10.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:08 smithi085 conmon[29319]: 2021-02-10T04:44:10.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:08 smithi085 conmon[29319]: 2021 2021-02-10T04:44:10.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:08 smithi085 conmon[29319]: - 2021-02-10T04:44:10.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:08 smithi085 conmon[29319]: 02 2021-02-10T04:44:10.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:08 smithi085 conmon[29319]: - 2021-02-10T04:44:10.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:08 smithi085 conmon[29319]: 10 2021-02-10T04:44:10.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:08 smithi085 conmon[29319]: T 2021-02-10T04:44:10.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:08 smithi085 conmon[29319]: 04 2021-02-10T04:44:10.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:08 smithi085 conmon[29319]: : 2021-02-10T04:44:10.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:08 smithi085 conmon[29319]: 44 2021-02-10T04:44:10.867 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:08 smithi085 conmon[29319]: : 2021-02-10T04:44:10.867 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:08 smithi085 conmon[29319]: 07 2021-02-10T04:44:10.867 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:08 smithi085 conmon[29319]: . 2021-02-10T04:44:10.867 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:08 smithi085 conmon[29319]: 454266 2021-02-10T04:44:10.867 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:08 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2005 : cluster [DBG] 2021-02-10T04:44:10.867 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:08 smithi085 conmon[29319]: osdmap e1004: 8 total, 8 up, 7 in 2021-02-10T04:44:10.867 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:08 smithi085 conmon[35961]: cluster 2021-02-10T04:44:10.867 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:08 smithi085 conmon[35961]: 2021-02-10T04:44:10.867 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:08 smithi085 conmon[35961]: 2021 2021-02-10T04:44:10.867 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:08 smithi085 conmon[35961]: - 2021-02-10T04:44:10.867 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:08 smithi085 conmon[35961]: 02 2021-02-10T04:44:10.868 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:08 smithi085 conmon[35961]: - 2021-02-10T04:44:10.868 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:08 smithi085 conmon[35961]: 10 2021-02-10T04:44:10.868 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:08 smithi085 conmon[35961]: T 2021-02-10T04:44:10.868 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:08 smithi085 conmon[35961]: 04 2021-02-10T04:44:10.868 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:08 smithi085 conmon[35961]: : 2021-02-10T04:44:10.868 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:08 smithi085 conmon[35961]: 44 2021-02-10T04:44:10.868 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:08 smithi085 conmon[35961]: :07.454266+0000 mon.a (mon. 2021-02-10T04:44:10.868 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:08 smithi085 conmon[35961]: 0) 2005 : cluster [DBG] osdmap e1004: 8 total, 8 up, 7 in 2021-02-10T04:44:10.869 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:44:10.870 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:44:10.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:09 smithi164 conmon[29288]: cluster 2021-02-10T04:44:08.270937+0000 2021-02-10T04:44:10.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:09 smithi164 conmon[29288]: mgr.y (mgr.14140) 929 : cluster [DBG] pgmap v1669: 168 pgs: 1 active+clean+scrubbing, 166 active+clean, 1 active+clean+snaptrim; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:44:10.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:09 smithi164 conmon[29288]: audit 2021-02-10T04:44:08.621053+0000 mgr.y (mgr.14140) 930 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:44:10.870 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:44:10.870 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:44:10.871 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:44:09 smithi085 conmon[49106]: debug 2021-02-10T04:44:09.541+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:44:10.871 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:09 smithi085 conmon[35961]: cluster 2021-02-10T04:44:08 2021-02-10T04:44:10.871 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:09 smithi085 conmon[35961]: .270937+0000 mgr.y (mgr.14140) 929 : cluster [DBG] pgmap v1669: 168 pgs: 1 active+clean+scrubbing, 166 active+clean, 1 active+clean+snaptrim; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:44:10.871 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:09 smithi085 conmon[35961]: audit 2021-02-10T04:44:10.871 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:09 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:44:10.871 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:09 smithi085 conmon[35961]: T04:44:08.621053+0000 mgr.y 2021-02-10T04:44:10.871 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:09 smithi085 conmon[35961]: (mgr.14140) 930 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:44:10.871 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:09 smithi085 conmon[29319]: cluster 2021-02-10T04:44:10.871 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:09 smithi085 conmon[29319]: 2021-02-10T04:44 2021-02-10T04:44:10.871 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:09 smithi085 conmon[29319]: :08.270937+0000 mgr.y (mgr.14140) 929 : cluster [DBG] pgmap v1669: 168 pgs: 1 active+clean+scrubbing, 166 active+clean, 1 active+clean+snaptrim; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:44:10.871 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:09 smithi085 conmon[29319]: audit 2021-02-10T04:44:08. 2021-02-10T04:44:10.872 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:09 smithi085 conmon[29319]: 621053+0000 mgr.y (mgr.14140) 930 : audit 2021-02-10T04:44:10.872 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:09 smithi085 conmon[29319]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:44:11.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:11 smithi164 conmon[29288]: cluster 2021-02-10T04:44:10.271931 2021-02-10T04:44:11.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:11 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 931 : cluster [DBG] pgmap v1670: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:44:11.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:11 smithi085 conmon[29319]: cluster 2021-02-10T04:44:11.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:11 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:44:11.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:11 smithi085 conmon[29319]: 44:10.271931+0000 mgr.y (mgr.14140) 931 : cluster [DBG] pgmap v1670: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:44:11.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:11 smithi085 conmon[35961]: cluster 2021-02-10T04:44:10 2021-02-10T04:44:11.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:11 smithi085 conmon[35961]: .271931+0000 mgr.y (mgr.14140) 931 : cluster [DBG] pgmap v1670: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:44:13.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:13 smithi164 conmon[29288]: cluster 2021-02-10T04:44:12.272607 2021-02-10T04:44:13.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:13 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 932 : cluster [DBG] pgmap v1671: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:44:14.023 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:13 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:44:14.023 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:13 smithi085 conmon[29319]: 10T04:44:12.272607+0000 mgr.y (mgr.14140) 932 : cluster [DBG] pgmap v1671: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:44:14.023 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:13 smithi085 conmon[35961]: cluster 2021-02-10T04:44:12.272607 2021-02-10T04:44:14.023 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:13 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 932 : cluster [DBG] pgmap v1671: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:44:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:14 smithi164 conmon[29288]: cluster 2021-02-10T04:44:13.063067+0000 osd.2 (osd.2) 1945 2021-02-10T04:44:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:14 smithi164 conmon[29288]: : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:44:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:14 smithi164 conmon[29288]: cluster 2021-02-10T04:44:13.063724+0000 osd.2 (osd.2) 1946 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:44:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:14 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:44:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:14 smithi164 conmon[29288]: 04:44:13.490787+0000 mon.a (mon.0) 2006 : cluster [DBG] 2021-02-10T04:44:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:14 smithi164 conmon[29288]: osdmap e1005: 8 total, 8 up, 7 in 2021-02-10T04:44:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:14 smithi085 conmon[29319]: cluster 2021-02-10T04:44:13.063067+0000 osd.2 (osd 2021-02-10T04:44:14.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:14 smithi085 conmon[29319]: .2) 1945 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:44:14.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:14 smithi085 conmon[29319]: cluster 2021-02-10T04:44:13.063724+0000 osd.2 (osd.2) 1946 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:44:14.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:14 smithi085 conmon[29319]: cluster 2021-02-10T04:44:14.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:14 smithi085 conmon[29319]: 2021-02-10T04:44:13.490787+0000 mon.a (mon.0) 2006 : cluster [DBG] osdmap e1005: 8 total, 8 up, 7 in 2021-02-10T04:44:14.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:14 smithi085 conmon[35961]: cluster 2021-02-10T04:44:13.063067+0000 2021-02-10T04:44:14.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:14 smithi085 conmon[35961]: osd.2 (osd.2) 1945 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:44:14.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:14 smithi085 conmon[35961]: cluster 2021-02-10T04:44:13.063724+0000 2021-02-10T04:44:14.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:14 smithi085 conmon[35961]: osd.2 (osd.2) 1946 : 2021-02-10T04:44:14.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:14 smithi085 conmon[35961]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:44:14.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:14 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:44:14.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:14 smithi085 conmon[35961]: 10T04:44:13.490787+0000 mon.a ( 2021-02-10T04:44:14.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:14 smithi085 conmon[35961]: mon.0) 2006 : cluster [DBG] osdmap e1005: 8 total, 8 up, 7 in 2021-02-10T04:44:15.411 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:44:15.677 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:15 smithi085 conmon[29319]: cluster 2021-02-10T04:44:14.273359+0000 2021-02-10T04:44:15.677 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:15 smithi085 conmon[29319]: mgr.y (mgr.14140) 933 : cluster [DBG] pgmap v1673: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:44:15.677 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:15 smithi085 conmon[35961]: cluster 2021-02-10T04:44:15.677 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:15 smithi085 conmon[35961]: 2021-02-10T04:44: 2021-02-10T04:44:15.677 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:15 smithi085 conmon[35961]: 14.273359+0000 mgr.y (mgr.14140) 933 : cluster [DBG] pgmap v1673: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:44:15.679 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:44:15.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:15 smithi164 conmon[29288]: cluster 2021-02-10T04:44:15.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:15 smithi164 conmon[29288]: 2021-02-10T04:44:14.273359+0000 mgr.y (mgr.14140) 933 2021-02-10T04:44:15.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:15 smithi164 conmon[29288]: : cluster [DBG] pgmap v1673: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:44:15.942 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:44:15 smithi085 conmon[41540]: debug 2021-02-10T04:44:15.795+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:44:16.025 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:44:17.143 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:44:16 smithi164 conmon[37263]: debug 2021-02-10T04:44:16.778+0000 7f1c671e2700 -1 osd.5 pg_epoch: 1005 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x5566562949a0) 2021-02-10T04:44:17.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:17 smithi164 conmon[29288]: cluster 2021-02-10T04:44:17.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:17 smithi164 conmon[29288]: 2021-02-10T04:44:17.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:17 smithi164 conmon[29288]: 2021 2021-02-10T04:44:17.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:17 smithi164 conmon[29288]: -02-10T04:44:16.274277 2021-02-10T04:44:17.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:17 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 934 : cluster [DBG] pgmap v1674: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:44:17.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:17 smithi085 conmon[35961]: cluster 2021-02-10T04:44:16.274277 2021-02-10T04:44:17.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:17 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 934 : cluster [DBG] pgmap v1674: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:44:17.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:17 smithi085 conmon[29319]: cluster 2021-02-10T04:44:16.274277+0000 mgr.y (mgr.14140 2021-02-10T04:44:17.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:17 smithi085 conmon[29319]: ) 934 : cluster [DBG] pgmap v1674: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:44:18.251 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:44:18.286 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:44:18.893 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:44:18 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:44:18.981 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:44:19.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:44:18 smithi085 conmon[52836]: debug 2021-02-10T04:44:18.725+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:44:19.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:19 smithi164 conmon[29288]: cluster 2021-02-10T04:44:18.275179+0000 2021-02-10T04:44:19.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:19 smithi164 conmon[29288]: mgr.y (mgr.14140) 935 : cluster [DBG] pgmap v1675: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:44:19.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:19 smithi164 conmon[29288]: audit 2021-02-10T04:44:18.631035+0000 mgr.y (mgr.14140 2021-02-10T04:44:19.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:19 smithi164 conmon[29288]: ) 936 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:44:19.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:19 smithi085 conmon[35961]: cluster 2021-02-10T04:44:18.275179 2021-02-10T04:44:19.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:19 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 935 : cluster [DBG] pgmap v1675: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:44:19.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:19 smithi085 conmon[35961]: audit 2021-02-10T04:44:18. 2021-02-10T04:44:19.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:19 smithi085 conmon[35961]: 631035+0000 mgr.y (mgr.14140) 936 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:44:19.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:19 smithi085 conmon[29319]: cluster 2021-02-10T04:44:19.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:19 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:44:19.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:19 smithi085 conmon[29319]: 10T04:44:18.275179+0000 mgr.y (mgr.14140) 935 : cluster [DBG] pgmap v1675: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:44:19.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:19 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:44:19.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:19 smithi085 conmon[29319]: 02-10T04:44:18.631035+0000 mgr.y (mgr. 2021-02-10T04:44:19.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:19 smithi085 conmon[29319]: 14140) 936 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:44:20.722 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:44:20.753 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:44:21.144 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:44:20 smithi164 conmon[44544]: debug 2021-02-10T04:44:20.835+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:44:21.553 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:44:21.619 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:21 smithi085 conmon[29319]: cluster 2021-02-10T04:44:20.276154+0000 mgr.y (mgr.14140) 937 : cluster [DBG] pgmap v1676: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:44:21.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:21 smithi085 conmon[35961]: cluster 2021-02-10T04:44:21.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:21 smithi085 conmon[35961]: 2021-02-10T04:44:21.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:21 smithi085 conmon[35961]: 2021 2021-02-10T04:44:21.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:21 smithi085 conmon[35961]: - 2021-02-10T04:44:21.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:21 smithi085 conmon[35961]: 02-10T04:44:20.276154+0000 mgr.y (mgr.14140) 937 : cluster [DBG] pgmap v1676: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:44:21.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:21 smithi164 conmon[29288]: cluster 2021-02-10T04:44:21.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:21 smithi164 conmon[29288]: 2021-02-10T04:44:21.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:21 smithi164 conmon[29288]: 2021 2021-02-10T04:44:21.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:21 smithi164 conmon[29288]: - 2021-02-10T04:44:21.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:21 smithi164 conmon[29288]: 02 2021-02-10T04:44:21.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:21 smithi164 conmon[29288]: -10T04:44:20.276154 2021-02-10T04:44:21.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:21 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 937 : cluster [DBG] pgmap v1676: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:44:22.066 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:44:22.102 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:44:22.136 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:44:22.168 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:44:22.200 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:44:22.235 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:44:22.266 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:44:22.292 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:44:22.293 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:44:22.293 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:44:22.294 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:44:22.294 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:44:22.294 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:44:22.294 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:44:22.294 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:44:22.294 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:44:22.521 INFO:teuthology.orchestra.run.smithi085.stderr:set pool 7 pg_num to 26 2021-02-10T04:44:22.527 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is unset 2021-02-10T04:44:22.651 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:44:22.680 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:44:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[29319]: audit 2021-02-10T04:44:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[29319]: 2021-02-10T04:44:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[29319]: 2021 2021-02-10T04:44:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[29319]: - 2021-02-10T04:44:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[29319]: 02 2021-02-10T04:44:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[29319]: - 2021-02-10T04:44:22.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[29319]: 10T04:44:22.023824+0000 2021-02-10T04:44:22.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[29319]: mon.c (mon.1) 72 : audit [DBG] from='client.? 172.21.15.85:0/756344035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:44:22.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:44:22.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[29319]: :44:22.316088+0000 mon.c (mon.1 2021-02-10T04:44:22.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[29319]: ) 73 : audit [INF] from='client.? 172.21.15.85:0/600478021' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2021-02-10T04:44:22.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:44:22.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[29319]: -10T04:44:22.316714+0000 mon.a ( 2021-02-10T04:44:22.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[29319]: mon.0) 2007 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2021-02-10T04:44:22.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[29319]: audit 2021-02-10T04:44:22 2021-02-10T04:44:22.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[29319]: .345030+0000 mon.a (mon.0) 2008 : audit 2021-02-10T04:44:22.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[29319]: [INF] from='client.? 172.21.15.85:0/3676812345' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:44:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[35961]: audit 2021-02-10T04:44:22.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[35961]: 2021-02-10T04:44:22.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[35961]: 2021 2021-02-10T04:44:22.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[35961]: - 2021-02-10T04:44:22.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[35961]: 02 2021-02-10T04:44:22.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[35961]: - 2021-02-10T04:44:22.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[35961]: 10 2021-02-10T04:44:22.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[35961]: T 2021-02-10T04:44:22.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[35961]: 04 2021-02-10T04:44:22.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[35961]: :44:22.023824+0000 mon.c (mon.1) 72 : audit [DBG] 2021-02-10T04:44:22.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/756344035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:44:22.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:44:22.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[35961]: :44:22.316088+0000 mon.c (mon.1) 73 : audit [INF] from='client.? 172.21.15.85:0/600478021' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2021-02-10T04:44:22.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[35961]: audit 2021-02-10T04:44:22.316714+0000 mon.a (mon.0) 2007 : audit [INF] 2021-02-10T04:44:22.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[35961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2021-02-10T04:44:22.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:44:22.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[35961]: 02-10T04:44:22.345030+0000 mon.a (mon.0) 2008 : audit 2021-02-10T04:44:22.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:22 smithi085 conmon[35961]: [INF] from='client.? 172.21.15.85:0/3676812345' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:44:22.816 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:22 smithi164 conmon[29288]: audit 2021-02-10T04:44:22.023824+0000 mon.c (mon.1) 2021-02-10T04:44:22.816 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:22 smithi164 conmon[29288]: 72 : audit [DBG] from='client.? 172.21.15.85:0/756344035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:44:22.816 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:22 smithi164 conmon[29288]: audit 2021-02-10T04:44:22.316088+0000 mon.c (mon.1) 73 : audit [INF] from='client.? 172.21.15.85:0/600478021' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2021-02-10T04:44:22.816 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:22 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:44:22.816 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:22 smithi164 conmon[29288]: 10T04:44:22.316714+0000 mon.a (mon.0) 2007 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2021-02-10T04:44:22.816 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:22 smithi164 conmon[29288]: audit 2021-02-10T04:44:22. 2021-02-10T04:44:22.816 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:22 smithi164 conmon[29288]: 345030+0000 mon.a (mon.0) 2008 : audit [INF] 2021-02-10T04:44:22.816 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:22 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/3676812345' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:44:22.817 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:44:22 smithi164 conmon[40892]: debug 2021-02-10T04:44:22.813+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:44:22.852 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:44:22.919 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.4 dump_historic_ops 2021-02-10T04:44:22.957 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:44:23.070 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:44:23.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:44:22 smithi164 conmon[33677]: debug 2021-02-10T04:44:22.975+0000 7fbec19a1700 -1 osd.4 1006 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:44:23.252 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:44:23.804 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:44:23 smithi164 conmon[37263]: debug 2021-02-10T04:44:23.732+0000 7f1c6b1ea700 -1 osd.5 pg_epoch: 1006 pg[2.3( empty local-lis/les=638/639 n=0 ec=48/48 lis/c=638/638 les/c/f=639/639/0 sis=638) [5,7,3] r=0 lpr=638 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x5566582ea000) 2021-02-10T04:44:23.813 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:23 smithi164 conmon[29288]: cluster 2021-02-10T04:44:22.276970+0000 mgr.y ( 2021-02-10T04:44:23.813 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:23 smithi164 conmon[29288]: mgr.14140) 938 : cluster [DBG] pgmap v1677: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:44:23.813 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:23 smithi164 conmon[29288]: cluster 2021-02-10T04:44:22.504935+0000 mon.a (mon.0) 2009 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:44:23.813 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:23 smithi164 conmon[29288]: cluster 2021-02-10T04:44:22.504967+0000 mon.a ( 2021-02-10T04:44:23.813 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:23 smithi164 conmon[29288]: mon.0) 2010 : cluster [INF] Cluster is now healthy 2021-02-10T04:44:23.814 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:23 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:44:23.814 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:23 smithi164 conmon[29288]: T04:44:22.514828+0000 mon.a (mon.0 2021-02-10T04:44:23.814 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:23 smithi164 conmon[29288]: ) 2011 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2021-02-10T04:44:23.814 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:23 smithi164 conmon[29288]: audit 2021-02-10T04:44:22 2021-02-10T04:44:23.814 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:23 smithi164 conmon[29288]: .514910+0000 mon.a (mon.0) 2012 : audit [INF] 2021-02-10T04:44:23.815 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:23 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/3676812345' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:44:23.815 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:23 smithi164 conmon[29288]: cluster 2021-02-10T04:44:22.514942+0000 mon.a (mon.0) 2013 : cluster [DBG] 2021-02-10T04:44:23.815 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:23 smithi164 conmon[29288]: osdmap e1006: 8 total, 8 up, 7 in 2021-02-10T04:44:23.815 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:23 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:44:23.815 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:23 smithi164 conmon[29288]: :44:22.976868+0000 osd.4 (osd.4) 2021-02-10T04:44:23.816 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:23 smithi164 conmon[29288]: 1767 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:44:23.816 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:23 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:44:23.816 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:23 smithi164 conmon[29288]: 02-10T04:44:22.976894+0000 2021-02-10T04:44:23.816 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:23 smithi164 conmon[29288]: osd.4 (osd.4) 1768 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:44:23.816 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:23 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:44:23.816 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:23 smithi164 conmon[29288]: 02-10T04:44:22.976902+0000 2021-02-10T04:44:23.817 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:23 smithi164 conmon[29288]: osd.4 (osd.4) 1769 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:44:23.817 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:23 smithi164 conmon[29288]: 2021-02-10T04:44:23.817 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:23 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:44:23.817 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:23 smithi164 conmon[29288]: -10T04:44:22.976908+0000 osd.4 (osd.4) 2021-02-10T04:44:23.817 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:23 smithi164 conmon[29288]: 1770 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:44:23.836 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:44:23.941 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[45443]: debug 2021-02-10T04:44:23.657+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:44:24.097 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: cluster 2021-02-10T04:44:24.097 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: 2021-02-10T04:44:24.097 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: 2021-02-10T04:44:22.276970+0000 mgr.y (mgr.14140) 938 : cluster [DBG] pgmap v1677: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:44:24.097 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:44:24.097 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: -10T04:44:22.504935+0000 mon.a (mon.0) 2021-02-10T04:44:24.098 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: 2009 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:44:24.098 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:44:24.098 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: -10T04:44:22.504967+0000 mon.a (mon 2021-02-10T04:44:24.098 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: .0) 2010 : cluster [INF] Cluster is now healthy 2021-02-10T04:44:24.098 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:44:24.098 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: 02-10T04:44:22.514828+0000 mon.a ( 2021-02-10T04:44:24.098 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: mon.0) 2011 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2021-02-10T04:44:24.098 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:44:24.098 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: -10T04:44:22.514910+0000 mon.a (mon.0) 2021-02-10T04:44:24.098 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: 2012 : audit [INF] from='client.? 172.21.15.85:0/3676812345' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:44:24.098 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:44:24.098 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: 44:22.514942+0000 mon.a (mon. 2021-02-10T04:44:24.098 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: 0) 2013 : cluster [DBG] 2021-02-10T04:44:24.099 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: osdmap e1006: 8 total, 8 up, 7 in 2021-02-10T04:44:24.099 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: cluster 2021-02-10T04:44 2021-02-10T04:44:24.099 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: :22.976868+0000 osd.4 (osd.4) 1767 2021-02-10T04:44:24.099 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:44:24.099 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:44:24.099 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: -02-10T04:44:22.976894+0000 2021-02-10T04:44:24.099 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: osd.4 (osd.4) 1768 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:44:24.099 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:44:24.099 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: -10T04:44:22.976902+0000 osd.4 (osd.4 2021-02-10T04:44:24.099 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: ) 1769 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:44:24.099 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:44:24.099 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: 44:22.976908+0000 osd.4 (osd.4) 1770 : 2021-02-10T04:44:24.100 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:44:24.100 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[35961]: cluster 2021-02-10T04:44:22.276970+0000 mgr.y (mgr.14140) 938 : cluster [DBG] pgmap v1677: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:44:24.100 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[35961]: cluster 2021-02-10T04:44:22.504935+0000 mon.a (mon.0) 2009 : 2021-02-10T04:44:24.100 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[35961]: cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:44:24.100 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[35961]: cluster 2021-02-10T04:44:24.100 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[35961]: 2021-02-10T04:44:22.504967+0000 mon.a (mon.0) 2010 2021-02-10T04:44:24.101 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[35961]: : cluster [INF] Cluster is now healthy 2021-02-10T04:44:24.101 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[35961]: audit 2021-02-10T04:44 2021-02-10T04:44:24.101 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[35961]: :22.514828+0000 mon.a (mon.0) 2011 : 2021-02-10T04:44:24.101 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[35961]: audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2021-02-10T04:44:24.101 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[35961]: audit 2021-02-10T04:44 2021-02-10T04:44:24.101 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[35961]: :22.514910+0000 mon.a (mon.0) 2012 2021-02-10T04:44:24.101 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[35961]: : audit [INF] from='client.? 172.21.15.85:0/3676812345' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:44:24.101 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:44:24.101 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[35961]: -10T04:44:22.514942+0000 mon.a 2021-02-10T04:44:24.101 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[35961]: (mon.0) 2013 : cluster [DBG] osdmap e1006: 8 total, 8 up, 7 in 2021-02-10T04:44:24.101 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:44:24.101 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[35961]: -10T04:44:22.976868+0000 osd.4 2021-02-10T04:44:24.101 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[35961]: (osd.4) 1767 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:44:24.101 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[35961]: 2021-02-10T04:44:24.102 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:44:24.102 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[35961]: 02-10T04 2021-02-10T04:44:24.102 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[35961]: :44:22.976894+0000 2021-02-10T04:44:24.102 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[35961]: osd.4 (osd.4) 1768 : cluster [WRN] 2021-02-10T04:44:24.102 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:44:24.102 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:44:24.102 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[35961]: 04:44:22.976902+0000 osd.4 (osd 2021-02-10T04:44:24.102 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[35961]: .4) 1769 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:44:24.102 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[35961]: cluster 2021-02-10T04:44:24.102 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[35961]: 2021-02-10T04:44:22.976908+0000 osd.4 (osd. 2021-02-10T04:44:24.102 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:23 smithi085 conmon[35961]: 4) 1770 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:44:24.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:44:23 smithi164 conmon[33677]: debug 2021-02-10T04:44:23.944+0000 7fbec19a1700 -1 osd.4 1006 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) 2021-02-10T04:44:24.520 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:24 smithi164 conmon[29288]: cluster 2021-02-10T04:44:23.946620+0000 osd.4 (osd.4) 1771 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:44:24.521 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:24 smithi164 conmon[29288]: cluster 2021-02-10T04:44:23.946631+0000 osd.4 (osd.4) 1772 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:44:24.521 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:24 smithi164 conmon[29288]: cluster 2021-02-10T04:44:23.946638+0000 osd.4 (osd.4) 1773 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:44:24.521 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:24 smithi164 conmon[29288]: cluster 2021-02-10T04:44:23.946645+0000 osd.4 (osd.4) 1774 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:44:24.521 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:24 smithi164 conmon[29288]: audit 2021-02-10T04:44:24.278701+0000 mon.a (mon.0) 2014 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2021-02-10T04:44:24.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: cluster 2021-02-10T04:44:24.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: 2021-02-10T04:44:24.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: 2021 2021-02-10T04:44:24.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: - 2021-02-10T04:44:24.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: 02 2021-02-10T04:44:24.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: - 2021-02-10T04:44:24.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: 10 2021-02-10T04:44:24.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: T 2021-02-10T04:44:24.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: 04 2021-02-10T04:44:24.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: : 2021-02-10T04:44:24.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: 44 2021-02-10T04:44:24.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: : 2021-02-10T04:44:24.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: 23 2021-02-10T04:44:24.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: . 2021-02-10T04:44:24.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: 946620 2021-02-10T04:44:24.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: +0000 2021-02-10T04:44:24.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: 2021-02-10T04:44:24.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: osd.4 2021-02-10T04:44:24.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: ( 2021-02-10T04:44:24.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: osd 2021-02-10T04:44:24.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: . 2021-02-10T04:44:24.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: 4 2021-02-10T04:44:24.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: ) 1771 : cluster [WRN] 2021-02-10T04:44:24.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:44:24.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:44:24.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: -10T04:44:23.946631+0000 osd.4 (osd 2021-02-10T04:44:24.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: .4) 1772 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:44:24.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:44:24.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: 02-10T04:44:23.946638+0000 2021-02-10T04:44:24.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: osd.4 (osd.4) 1773 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:44:24.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: cluster 2021-02-10T04:44:24.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: 2021-02-10T04:44:23.946645+0000 2021-02-10T04:44:24.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: osd.4 (osd.4) 1774 : cluster [WRN] 2021-02-10T04:44:24.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:44:24.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: audit 2021-02-10T04:44: 2021-02-10T04:44:24.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: 24.278701+0000 mon.a (mon.0) 2014 2021-02-10T04:44:24.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[29319]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2021-02-10T04:44:24.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: cluster 2021-02-10T04:44:24.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:44:24.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: 04:44:23.946620+0000 osd.4 (osd.4) 1771 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] ondisk+write+known_if_redirected e156) initiated 2021-02-10T04:29:32.164063+0000 currently started 2021-02-10T04:44:24.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: cluster 2021-02-10T04:44:24.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: 2021-02-10T04:44:24.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: 2021 2021-02-10T04:44:24.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: - 2021-02-10T04:44:24.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: 02 2021-02-10T04:44:24.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: - 2021-02-10T04:44:24.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: 10 2021-02-10T04:44:24.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: T 2021-02-10T04:44:24.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: 04 2021-02-10T04:44:24.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: : 2021-02-10T04:44:24.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: 44 2021-02-10T04:44:24.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: : 2021-02-10T04:44:24.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: 23 2021-02-10T04:44:24.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: . 2021-02-10T04:44:24.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: 946631 2021-02-10T04:44:24.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: +0000 2021-02-10T04:44:24.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: 2021-02-10T04:44:24.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: osd.4 2021-02-10T04:44:24.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: ( 2021-02-10T04:44:24.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: osd 2021-02-10T04:44:24.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: . 2021-02-10T04:44:24.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: 4 2021-02-10T04:44:24.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: ) 2021-02-10T04:44:24.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: 1772 2021-02-10T04:44:24.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: : 2021-02-10T04:44:24.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: cluster 2021-02-10T04:44:24.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: 2021-02-10T04:44:24.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: [WRN] 2021-02-10T04:44:24.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: 2021-02-10T04:44:24.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e156) initiated 2021-02-10T04:29:32.164227+0000 currently delayed 2021-02-10T04:44:24.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: 2021-02-10T04:44:24.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: cluster 2021-02-10T04:44:24.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: 2021-02-10T04:44:24.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: 2021 2021-02-10T04:44:24.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: - 2021-02-10T04:44:24.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: 02 2021-02-10T04:44:24.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: - 2021-02-10T04:44:24.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: 10 2021-02-10T04:44:24.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: T04:44:23.946638+0000 2021-02-10T04:44:24.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: osd.4 (osd.4) 1773 : cluster [WRN] slow request osd_op(client.14817.0:865 7.1 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29 in=55b] snapc 37=[36,32,2e] ondisk+write+known_if_redirected e153) initiated 2021-02-10T04:29:29.061753+0000 currently started 2021-02-10T04:44:24.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: cluster 2021-02-10T04:44:23. 2021-02-10T04:44:24.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: 946645+0000 osd.4 (osd.4) 1774 : cluster [WRN] slow request osd_op(client.14817.0:866 7.1 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e153) initiated 2021-02-10T04:29:29.061891+0000 currently delayed 2021-02-10T04:44:24.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: audit 2021-02-10T04:44:24.278701+0000 mon.a (mon.0 2021-02-10T04:44:24.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:24 smithi085 conmon[35961]: ) 2014 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2021-02-10T04:44:25.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:25 smithi164 conmon[29288]: cluster 2021-02-10T04:44:24.229798+0000 osd.6 ( 2021-02-10T04:44:29.578 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:25 smithi164 conmon[29288]: osd.6) 88 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:44:29.578 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:25 smithi164 conmon[29288]: cluster 2021-02-10T04:44:24.230794+0000 osd.6 (osd.6) 89 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:44:29.578 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:25 smithi164 conmon[29288]: cluster 2021-02-10T04:44:24.277724+0000 mgr.y (mgr.14140) 939 : cluster [DBG] pgmap v1679: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:44:29.578 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:25 smithi164 conmon[29288]: audit 2021-02-10T04:44:24.525070+0000 mon.a (mon.0) 2015 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2021-02-10T04:44:29.578 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:25 smithi164 conmon[29288]: cluster 2021-02-10T04:44:24.525140+0000 mon.a (mon.0) 2016 : cluster [DBG] osdmap e1007: 8 total, 8 up, 7 in 2021-02-10T04:44:29.578 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:25 smithi164 conmon[29288]: cluster 2021-02-10T04:44:24.905178+0000 osd.4 (osd.4) 1775 : cluster [DBG] 4.e scrub ok 2021-02-10T04:44:29.581 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: cluster 2021-02-10T04:44:29.581 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 2021-02-10T04:44:29.581 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 2021 2021-02-10T04:44:29.581 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: - 2021-02-10T04:44:29.581 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 02 2021-02-10T04:44:29.581 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: - 2021-02-10T04:44:29.581 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 10 2021-02-10T04:44:29.581 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: T 2021-02-10T04:44:29.581 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 04 2021-02-10T04:44:29.581 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: : 2021-02-10T04:44:29.581 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 44 2021-02-10T04:44:29.582 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: : 2021-02-10T04:44:29.582 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 24 2021-02-10T04:44:29.582 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: . 2021-02-10T04:44:29.582 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 229798 2021-02-10T04:44:29.582 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: +0000 2021-02-10T04:44:29.582 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 2021-02-10T04:44:29.582 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: osd.6 2021-02-10T04:44:29.582 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: ( 2021-02-10T04:44:29.582 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: osd 2021-02-10T04:44:29.582 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: . 2021-02-10T04:44:29.582 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 6 2021-02-10T04:44:29.582 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: ) 2021-02-10T04:44:29.583 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 88 2021-02-10T04:44:29.583 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: : 2021-02-10T04:44:29.583 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: cluster 2021-02-10T04:44:29.583 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 2021-02-10T04:44:29.583 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: [DBG] 2021-02-10T04:44:29.583 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 2021-02-10T04:44:29.583 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: purged_snaps scrub starts 2021-02-10T04:44:29.583 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 2021-02-10T04:44:29.583 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: cluster 2021-02-10T04:44:29.583 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 2021-02-10T04:44:29.583 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 2021 2021-02-10T04:44:29.583 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: - 2021-02-10T04:44:29.583 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 02 2021-02-10T04:44:29.584 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: - 2021-02-10T04:44:29.584 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 10 2021-02-10T04:44:29.584 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: T 2021-02-10T04:44:29.584 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 04 2021-02-10T04:44:29.584 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: : 2021-02-10T04:44:29.584 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 44 2021-02-10T04:44:29.584 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: : 2021-02-10T04:44:29.584 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 24 2021-02-10T04:44:29.584 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: . 2021-02-10T04:44:29.584 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 230794 2021-02-10T04:44:29.584 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: +0000 2021-02-10T04:44:29.584 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 2021-02-10T04:44:29.584 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: osd.6 2021-02-10T04:44:29.584 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: ( 2021-02-10T04:44:29.585 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: osd 2021-02-10T04:44:29.585 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: . 2021-02-10T04:44:29.585 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 6 2021-02-10T04:44:29.585 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: ) 2021-02-10T04:44:29.585 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 89 2021-02-10T04:44:29.585 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: : 2021-02-10T04:44:29.586 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: cluster 2021-02-10T04:44:29.586 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 2021-02-10T04:44:29.586 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: [DBG] 2021-02-10T04:44:29.586 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 2021-02-10T04:44:29.586 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: purged_snaps scrub ok 2021-02-10T04:44:29.586 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 2021-02-10T04:44:29.586 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: cluster 2021-02-10T04:44:29.586 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 2021-02-10T04:44:29.586 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 2021 2021-02-10T04:44:29.586 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: - 2021-02-10T04:44:29.587 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 02 2021-02-10T04:44:29.587 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: - 2021-02-10T04:44:29.587 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 10 2021-02-10T04:44:29.587 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: T 2021-02-10T04:44:29.587 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 04 2021-02-10T04:44:29.587 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: : 2021-02-10T04:44:29.587 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 44 2021-02-10T04:44:29.587 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: : 2021-02-10T04:44:29.587 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 24.277724+0000 mgr.y (mgr.14140) 2021-02-10T04:44:29.587 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 939 : cluster [DBG] pgmap v1679: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:44:29.588 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: audit 2021-02-10T 2021-02-10T04:44:29.588 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: 04:44:24.525070+0000 mon.a (mon.0) 2015 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2021-02-10T04:44:29.588 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:44:29.588 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: -10T04:44:24.525140+0000 mon.a ( 2021-02-10T04:44:29.588 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: mon.0) 2016 : cluster [DBG] osdmap e1007: 8 total, 8 up, 7 in 2021-02-10T04:44:29.588 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:44:29.588 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: -02-10T04:44:24.905178+0000 2021-02-10T04:44:29.588 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[29319]: osd.4 (osd.4) 1775 : cluster [DBG] 4.e scrub ok 2021-02-10T04:44:29.588 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:26 smithi085 conmon[29319]: cluster 2021-02-10T04:44:29.588 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:26 smithi085 conmon[29319]: 2021-02-10T04:44:29.588 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:26 smithi085 conmon[29319]: 2021 2021-02-10T04:44:29.588 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:26 smithi085 conmon[29319]: -02-10T04: 2021-02-10T04:44:29.588 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:26 smithi085 conmon[29319]: 44:24.718973+0000 osd.6 (osd.6) 90 : cluster [DBG] 2.7 scrub ok 2021-02-10T04:44:29.589 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:26 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:44:29.589 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:26 smithi085 conmon[29319]: -10T04:44:25.522761+0000 mon.a (mon.0) 2021-02-10T04:44:29.589 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:26 smithi085 conmon[29319]: 2017 : cluster [DBG] osdmap e1008: 8 total, 8 up, 7 in 2021-02-10T04:44:29.589 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:26 smithi085 conmon[29319]: cluster 2021-02-10T04:44:29.589 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:26 smithi085 conmon[29319]: 2021-02-10T04:44:26.031767 2021-02-10T04:44:29.589 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:26 smithi085 conmon[29319]: +0000 osd.0 (osd.0) 97 : cluster 2021-02-10T04:44:29.589 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:26 smithi085 conmon[29319]: [DBG] 5.5 scrub ok 2021-02-10T04:44:29.589 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:26 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:44:29.589 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:26 smithi085 conmon[29319]: 02-10T04:44:26.279544+0000 mon.a ( 2021-02-10T04:44:29.589 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:26 smithi085 conmon[29319]: mon.0) 2018 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2021-02-10T04:44:29.589 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[29319]: cluster 2021-02-10T04:44:29.589 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[29319]: 2021-02-10T04:44:29.590 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[29319]: 2021 2021-02-10T04:44:29.590 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[29319]: - 2021-02-10T04:44:29.590 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[29319]: 02 2021-02-10T04:44:29.590 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[29319]: - 2021-02-10T04:44:29.590 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[29319]: 10T04:44:26.278364+0000 mgr.y (mgr 2021-02-10T04:44:29.590 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[29319]: .14140) 940 : cluster [DBG] pgmap v1682: 178 pgs: 6 peering, 10 unknown, 1 active+clean+scrubbing, 161 active+clean; 718 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:44:29.590 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[29319]: audit 2021-02-10T04:44:26 2021-02-10T04:44:29.590 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[29319]: .531477+0000 mon.a (mon.0) 2019 : 2021-02-10T04:44:29.590 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[29319]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2021-02-10T04:44:29.590 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:44:29.590 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[29319]: -02-10T04:44:26.531718 2021-02-10T04:44:29.590 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2020 : cluster [DBG] 2021-02-10T04:44:29.590 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[29319]: osdmap e1009: 8 total, 8 up, 7 in 2021-02-10T04:44:29.591 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:44:29.591 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[29319]: -02-10T04:44: 2021-02-10T04:44:29.591 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[29319]: 26.684865+0000 osd.5 (osd 2021-02-10T04:44:29.591 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[29319]: .5) 90 : cluster 2021-02-10T04:44:29.591 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[29319]: [DBG] 2.3 scrub ok 2021-02-10T04:44:29.591 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:44:29.591 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[29319]: :44:27.530725+0000 mon.a (mon.0) 2021 : cluster [DBG] osdmap e1010: 8 total, 8 up, 7 in 2021-02-10T04:44:29.591 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[29319]: cluster 2021-02-10T04:44:29.591 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[29319]: 2021-02 2021-02-10T04:44:29.591 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[29319]: -10T04:44:26.665934+0000 osd.3 (osd.3) 92 : cluster [DBG] 7.a scrub ok 2021-02-10T04:44:29.591 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[29319]: 2021-02-10T04:44:29.591 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:44:29.591 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[29319]: 02-10T04:44:27.926658+0000 osd.4 ( 2021-02-10T04:44:29.592 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[29319]: osd.4) 1776 : cluster [DBG] 6.2 scrub ok 2021-02-10T04:44:29.592 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:44:29.592 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[29319]: 02-10T04:44:28.532387+0000 mon.a ( 2021-02-10T04:44:29.592 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[29319]: mon.0) 2022 : cluster [DBG] osdmap e1011: 8 total, 8 up, 7 in 2021-02-10T04:44:29.592 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[35961]: cluster 2021-02-10T04:44:24.229798+0000 2021-02-10T04:44:29.592 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[35961]: osd.6 (osd.6) 88 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:44:29.592 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:44:29.592 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[35961]: 44:24.230794+0000 osd.6 (osd.6) 89 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:44:29.592 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[35961]: cluster 2021-02-10T04:44:29.592 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[35961]: 2021-02-10T04:44:24.277724+0000 2021-02-10T04:44:29.592 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[35961]: mgr.y (mgr.14140) 939 : cluster [DBG] 2021-02-10T04:44:29.592 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[35961]: pgmap v1679: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 859 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:44:29.593 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:44:29.593 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[35961]: T04:44:24.525070+0000 mon.a (mon 2021-02-10T04:44:29.593 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[35961]: .0) 2015 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2021-02-10T04:44:29.593 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[35961]: cluster 2021-02-10T04:44:29.593 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:44:29.593 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[35961]: T04:44:24.525140+0000 mon.a (mon 2021-02-10T04:44:29.593 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[35961]: .0) 2016 : cluster [DBG] osdmap e1007: 8 total, 8 up, 7 in 2021-02-10T04:44:29.593 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[35961]: cluster 2021-02-10T04:44:29.593 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[35961]: 2021-02-10T04:44:24.905178 2021-02-10T04:44:29.593 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1775 : cluster 2021-02-10T04:44:29.593 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:25 smithi085 conmon[35961]: [DBG] 4.e scrub ok 2021-02-10T04:44:29.593 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:26 smithi085 conmon[35961]: cluster 2021-02-10T04:44:29.593 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:26 smithi085 conmon[35961]: 2021-02-10T04:44:29.594 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:26 smithi085 conmon[35961]: 2021 2021-02-10T04:44:29.594 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:26 smithi085 conmon[35961]: -02-10T 2021-02-10T04:44:29.594 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:26 smithi085 conmon[35961]: 04:44:24.718973+0000 osd.6 (osd.6) 90 : cluster [DBG] 2.7 scrub ok 2021-02-10T04:44:29.594 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:26 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:44:29.594 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:26 smithi085 conmon[35961]: -10T04:44:25.522761+0000 mon.a (mon.0) 2017 : cluster [DBG] osdmap e1008: 8 total, 8 up, 7 in 2021-02-10T04:44:29.594 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:26 smithi085 conmon[35961]: cluster 2021-02-10T04:44:26.031767+0000 osd.0 (osd.0) 97 2021-02-10T04:44:29.594 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:26 smithi085 conmon[35961]: : cluster [DBG] 5.5 scrub ok 2021-02-10T04:44:29.594 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:26 smithi085 conmon[35961]: audit 2021-02-10T04:44:26.279544+0000 mon.a (mon.0) 2021-02-10T04:44:29.594 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:26 smithi085 conmon[35961]: 2018 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2021-02-10T04:44:29.594 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[35961]: cluster 2021-02-10T04:44: 2021-02-10T04:44:29.594 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[35961]: 26.278364+0000 mgr.y (mgr.14140) 940 : cluster [DBG] pgmap v1682: 178 pgs: 6 peering, 10 unknown, 1 active+clean+scrubbing, 161 active+clean; 718 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:44:29.594 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[35961]: audit 2021 2021-02-10T04:44:29.594 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[35961]: -02-10T04:44:26.531477+0000 mon.a (mon.0) 2021-02-10T04:44:29.595 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[35961]: 2019 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2021-02-10T04:44:29.595 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:44:29.595 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[35961]: T04:44:26.531718+0000 mon.a (mon.0) 2020 : cluster 2021-02-10T04:44:29.595 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[35961]: [DBG] osdmap e1009: 8 total, 8 up, 7 in 2021-02-10T04:44:29.595 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[35961]: cluster 2021-02-10T04:44:26.684865+0000 2021-02-10T04:44:29.595 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[35961]: osd.5 (osd.5) 90 : cluster [DBG] 2.3 scrub ok 2021-02-10T04:44:29.595 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:44:29.595 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[35961]: 02-10T04:44:27.530725+0000 mon.a (mon. 2021-02-10T04:44:29.595 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:27 smithi085 conmon[35961]: 0) 2021 : cluster [DBG] osdmap e1010: 8 total, 8 up, 7 in 2021-02-10T04:44:29.595 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[35961]: cluster 2021-02-10T04:44:29.595 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:44:29.595 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[35961]: 44:26.665934+0000 osd.3 (osd.3) 92 : cluster [DBG] 7.a scrub ok 2021-02-10T04:44:29.595 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[35961]: cluster 2021-02-10T04:44:29.596 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[35961]: 2021-02-10T04:44:29.596 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[35961]: 2021 2021-02-10T04:44:29.596 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[35961]: - 2021-02-10T04:44:29.596 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[35961]: 02-10T04:44:27 2021-02-10T04:44:29.596 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[35961]: .926658+0000 osd.4 (osd.4) 1776 : cluster [DBG] 6.2 scrub ok 2021-02-10T04:44:29.596 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[35961]: cluster 2021-02-10T04:44:29.596 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[35961]: 2021-02-10T04:44:29.596 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[35961]: 2021 2021-02-10T04:44:29.596 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[35961]: - 2021-02-10T04:44:29.596 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[35961]: 02 2021-02-10T04:44:29.596 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[35961]: - 2021-02-10T04:44:29.596 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[35961]: 10 2021-02-10T04:44:29.596 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[35961]: T 2021-02-10T04:44:29.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[35961]: 04 2021-02-10T04:44:29.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[35961]: : 2021-02-10T04:44:29.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[35961]: 44 2021-02-10T04:44:29.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[35961]: : 2021-02-10T04:44:29.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[35961]: 28 2021-02-10T04:44:29.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[35961]: .532387+0000 mon.a (mon.0 2021-02-10T04:44:29.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:28 smithi085 conmon[35961]: ) 2022 : cluster [DBG] osdmap e1011: 8 total, 8 up, 7 in 2021-02-10T04:44:29.597 INFO:tasks.rados.rados.0.smithi085.stdout:310: finishing copy_from to smithi08575397-17 2021-02-10T04:44:29.597 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 17 v 29 (ObjNum 85 snap 11 seq_num 85) dirty exists 2021-02-10T04:44:29.597 INFO:tasks.rados.rados.0.smithi085.stdout:310: finishing copy_from racing read to smithi08575397-17 2021-02-10T04:44:29.597 INFO:tasks.rados.rados.0.smithi085.stdout:310: done (3 left) 2021-02-10T04:44:29.597 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 3 2021-02-10T04:44:29.598 INFO:tasks.rados.rados.0.smithi085.stdout:1064: finishing copy_from to smithi08575397-27 2021-02-10T04:44:29.598 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 27 v 168 (ObjNum 388 snap 100 seq_num 388) dirty exists 2021-02-10T04:44:29.598 INFO:tasks.rados.rados.0.smithi085.stdout:1064: finishing copy_from racing read to smithi08575397-27 2021-02-10T04:44:29.598 INFO:tasks.rados.rados.0.smithi085.stdout:1064: done (2 left) 2021-02-10T04:44:29.598 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 2 2021-02-10T04:44:29.598 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:26 smithi164 conmon[29288]: cluster 2021-02-10T04:44:29.598 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:26 smithi164 conmon[29288]: 2021-02-10T04:44:29.598 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:26 smithi164 conmon[29288]: 2021-02-10T04:44 2021-02-10T04:44:29.598 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:26 smithi164 conmon[29288]: :24.718973+0000 osd.6 (osd.6) 90 : cluster [DBG] 2.7 scrub ok 2021-02-10T04:44:29.598 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:26 smithi164 conmon[29288]: cluster 2021-02-10T04:44: 2021-02-10T04:44:29.598 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:26 smithi164 conmon[29288]: 25.522761+0000 mon.a (mon.0) 2017 : cluster [DBG] osdmap e1008: 8 total, 8 up, 7 in 2021-02-10T04:44:29.598 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:26 smithi164 conmon[29288]: cluster 2021-02-10T04:44:26.031767+0000 osd.0 (osd.0) 97 : cluster [DBG] 5.5 scrub ok 2021-02-10T04:44:29.599 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:26 smithi164 conmon[29288]: audit 2021-02-10T04:44:26.279544+0000 mon.a (mon.0) 2018 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2021-02-10T04:44:29.599 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:27 smithi164 conmon[29288]: cluster 2021-02-10T04:44:26.278364+0000 mgr.y (mgr.14140) 940 : cluster [DBG] pgmap v1682: 178 pgs: 6 peering, 10 unknown, 1 active+clean+scrubbing, 161 active+clean; 718 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:44:29.599 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:27 smithi164 conmon[29288]: audit 2021-02-10T04:44:26.531477+0000 mon.a (mon.0) 2019 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2021-02-10T04:44:29.599 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:27 smithi164 conmon[29288]: cluster 2021-02-10T04:44:26.531718+0000 mon.a (mon.0) 2020 : cluster [DBG] osdmap e1009: 8 total, 8 up, 7 in 2021-02-10T04:44:29.599 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:27 smithi164 conmon[29288]: cluster 2021-02-10T04:44:29.599 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:27 smithi164 conmon[29288]: 2021-02-10T04:44:26.684865+0000 osd.5 (osd.5) 90 : cluster [DBG] 2.3 scrub ok 2021-02-10T04:44:29.599 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:27 smithi164 conmon[29288]: cluster 2021-02-10T04:44:27.530725+0000 mon.a (mon.0) 2021 : cluster [DBG] osdmap e1010: 8 total, 8 up, 7 in 2021-02-10T04:44:29.599 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:27 smithi164 conmon[29288]: 2021-02-10T04:44:29.599 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:28 smithi164 conmon[29288]: cluster 2021-02-10T04:44:29.599 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:28 smithi164 conmon[29288]: 2021-02-10T04:44 2021-02-10T04:44:29.599 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:28 smithi164 conmon[29288]: :26.665934+0000 osd.3 (osd.3) 92 : cluster [DBG] 7.a scrub ok 2021-02-10T04:44:29.599 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:28 smithi164 conmon[29288]: cluster 2021-02-10T04:44:29.600 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:28 smithi164 conmon[29288]: 2021-02-10T04:44:27.926658+0000 osd.4 (osd.4) 1776 : cluster 2021-02-10T04:44:29.600 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:28 smithi164 conmon[29288]: [DBG] 6.2 scrub ok 2021-02-10T04:44:29.600 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:28 smithi164 conmon[29288]: cluster 2021-02-10T04:44:29.600 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:28 smithi164 conmon[29288]: 2021-02-10T04:44:28.532387+0000 mon.a ( 2021-02-10T04:44:29.600 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:28 smithi164 conmon[29288]: mon.0) 2022 : cluster [DBG] osdmap e1011: 8 total, 8 up, 7 in 2021-02-10T04:44:29.600 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:44:28 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:44:29.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:29 smithi164 conmon[29288]: cluster 2021-02-10T04:44:28.279209+0000 mgr.y ( 2021-02-10T04:44:29.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:29 smithi164 conmon[29288]: mgr.14140) 941 : cluster [DBG] pgmap v1685: 178 pgs: 1 remapped+peering, 7 peering, 8 unknown, 162 active+clean; 715 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:44:29.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:29 smithi164 conmon[29288]: audit 2021-02-10T04:44:28.641095+0000 mgr.y (mgr.14140) 942 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:44:29.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[35961]: cluster 2021-02-10T04:44:28. 2021-02-10T04:44:29.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[35961]: 279209+0000 mgr.y (mgr.14140) 941 : cluster [DBG] pgmap v1685: 178 pgs: 1 remapped+peering, 7 peering, 8 unknown, 162 active+clean; 715 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:44:29.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[35961]: audit 2021-02-10T04:44:29.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[35961]: 2021-02-10T04:44:28.641095+0000 mgr.y (mgr.14140) 942 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:44:29.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: cluster 2021-02-10T04:44:29.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: 2021-02-10T04:44:29.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: 2021 2021-02-10T04:44:29.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: - 2021-02-10T04:44:29.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: 02 2021-02-10T04:44:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: - 2021-02-10T04:44:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: 10 2021-02-10T04:44:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: T 2021-02-10T04:44:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: 04 2021-02-10T04:44:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: : 2021-02-10T04:44:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: 44 2021-02-10T04:44:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: : 2021-02-10T04:44:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: 28 2021-02-10T04:44:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: . 2021-02-10T04:44:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: 279209 2021-02-10T04:44:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: +0000 2021-02-10T04:44:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: 2021-02-10T04:44:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: mgr.y 2021-02-10T04:44:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: ( 2021-02-10T04:44:29.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: mgr 2021-02-10T04:44:29.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: . 2021-02-10T04:44:29.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: 14140 2021-02-10T04:44:29.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: ) 2021-02-10T04:44:29.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: 941 2021-02-10T04:44:29.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: : 2021-02-10T04:44:29.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: cluster 2021-02-10T04:44:29.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: 2021-02-10T04:44:29.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: [DBG] 2021-02-10T04:44:29.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: 2021-02-10T04:44:29.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: pgmap v1685: 178 pgs: 1 remapped+peering, 7 peering, 8 unknown, 162 active+clean; 715 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:44:29.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: 2021-02-10T04:44:29.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: audit 2021-02-10T04:44:29.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: 2021-02-10T04:44:29.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: 2021 2021-02-10T04:44:29.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: - 2021-02-10T04:44:29.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: 02 2021-02-10T04:44:29.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: - 2021-02-10T04:44:29.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: 10 2021-02-10T04:44:29.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: T 2021-02-10T04:44:29.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: 04 2021-02-10T04:44:29.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: : 2021-02-10T04:44:29.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: 44 2021-02-10T04:44:29.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: : 2021-02-10T04:44:29.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: 28 2021-02-10T04:44:29.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: . 2021-02-10T04:44:29.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: 641095 2021-02-10T04:44:29.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: +0000 2021-02-10T04:44:29.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: 2021-02-10T04:44:29.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: mgr.y 2021-02-10T04:44:29.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: ( 2021-02-10T04:44:29.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: mgr 2021-02-10T04:44:29.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: . 2021-02-10T04:44:29.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: 14140 2021-02-10T04:44:29.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: ) 2021-02-10T04:44:29.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: 942 2021-02-10T04:44:29.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: : 2021-02-10T04:44:29.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: audit 2021-02-10T04:44:29.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: 2021-02-10T04:44:29.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: [DBG] 2021-02-10T04:44:29.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: 2021-02-10T04:44:29.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:44:29.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:29 smithi085 conmon[29319]: 2021-02-10T04:44:30.849 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:30 smithi164 conmon[29288]: cluster 2021-02-10T04:44:29.554251+0000 mon.a (mon.0 2021-02-10T04:44:30.849 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:30 smithi164 conmon[29288]: ) 2023 : cluster [DBG] osdmap e1012: 8 total, 8 up, 7 in 2021-02-10T04:44:30.850 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:44:30 smithi164 conmon[40892]: debug 2021-02-10T04:44:30.652+0000 7f3037cca700 -1 osd.6 pg_epoch: 1013 pg[4.a( v 538'45 (0'0,538'45] local-lis/les=55/56 n=14 ec=55/55 lis/c=55/55 les/c/f=56/56/0 sis=55) [6,1,7] r=0 lpr=55 crt=538'45 lcod 538'44 mlcod 538'44 active+clean] scrubber pg(4.a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x557a6f970420) 2021-02-10T04:44:30.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:30 smithi085 conmon[35961]: cluster 2021-02-10T04:44:29.554251+0000 mon.a (mon. 2021-02-10T04:44:30.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:30 smithi085 conmon[35961]: 0) 2023 : cluster [DBG] osdmap e1012: 8 total, 8 up, 7 in 2021-02-10T04:44:30.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:30 smithi085 conmon[29319]: cluster 2021-02-10T04:44:29.554251+0000 mon.a (mon.0 2021-02-10T04:44:30.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:30 smithi085 conmon[29319]: ) 2023 : cluster [DBG] osdmap e1012: 8 total, 8 up, 7 in 2021-02-10T04:44:31.391 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:44:31.414 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:44:31.414 INFO:teuthology.orchestra.run.smithi164.stdout: "size": 20, 2021-02-10T04:44:31.414 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 600, 2021-02-10T04:44:31.414 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [ 2021-02-10T04:44:31.414 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.414 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:6078 7.9 e615/93 7:93db45b2:::smithi08575397-44:head v 615'373, mlcod=613'369)", 2021-02-10T04:44:31.414 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:37:13.217942+0000", 2021-02-10T04:44:31.414 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 438.19155362399999, 2021-02-10T04:44:31.415 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.035029181, 2021-02-10T04:44:31.415 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:44:31.415 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:44:31.415 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:44:31.415 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.415 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:44:31.415 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:13.217942+0000", 2021-02-10T04:44:31.415 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:44:31.415 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.415 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.415 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:44:31.415 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:13.217942+0000", 2021-02-10T04:44:31.415 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0057270999999999997 2021-02-10T04:44:31.416 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.416 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.416 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:44:31.416 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:13.223669+0000", 2021-02-10T04:44:31.416 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0139999999999999e-05 2021-02-10T04:44:31.416 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.416 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.416 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:44:31.416 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:13.223679+0000", 2021-02-10T04:44:31.416 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0360000000000001e-05 2021-02-10T04:44:31.416 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.416 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.416 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:44:31.417 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:13.223689+0000", 2021-02-10T04:44:31.417 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.3679999999999996e-06 2021-02-10T04:44:31.417 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.417 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.417 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:44:31.417 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:13.223697+0000", 2021-02-10T04:44:31.417 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.5432000000000001e-05 2021-02-10T04:44:31.417 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.417 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.417 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:44:31.417 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:13.223752+0000", 2021-02-10T04:44:31.417 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8068999999999998e-05 2021-02-10T04:44:31.417 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.418 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.418 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:44:31.418 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:13.223780+0000", 2021-02-10T04:44:31.418 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.029116389999999999 2021-02-10T04:44:31.418 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.418 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.418 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:44:31.418 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:13.252897+0000", 2021-02-10T04:44:31.418 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.4321999999999998e-05 2021-02-10T04:44:31.418 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.418 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.419 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:44:31.419 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:13.252971+0000", 2021-02-10T04:44:31.419 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.035029181 2021-02-10T04:44:31.419 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.419 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:44:31.419 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.419 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.419 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.419 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:6432 7.3 e650/646 7:c765b4cc:::smithi08575397-30:head v 650'1042, mlcod=649'1039)", 2021-02-10T04:44:31.419 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:37:48.240225+0000", 2021-02-10T04:44:31.419 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 403.169270703, 2021-02-10T04:44:31.420 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.025241980000000001, 2021-02-10T04:44:31.420 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:44:31.420 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:44:31.420 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:44:31.420 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.420 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:44:31.420 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:48.240225+0000", 2021-02-10T04:44:31.420 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:44:31.420 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.420 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.420 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:44:31.421 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:48.240225+0000", 2021-02-10T04:44:31.421 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0011735770000000001 2021-02-10T04:44:31.421 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.421 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.421 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:44:31.421 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:48.241398+0000", 2021-02-10T04:44:31.421 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7430000000000002e-06 2021-02-10T04:44:31.421 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.421 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.421 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:44:31.421 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:48.241406+0000", 2021-02-10T04:44:31.421 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.9700000000000005e-06 2021-02-10T04:44:31.422 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.422 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.422 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:44:31.422 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:48.241415+0000", 2021-02-10T04:44:31.422 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.815e-06 2021-02-10T04:44:31.422 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.422 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.422 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:44:31.422 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:48.241422+0000", 2021-02-10T04:44:31.422 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0995999999999997e-05 2021-02-10T04:44:31.422 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.423 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.423 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:44:31.423 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:48.241483+0000", 2021-02-10T04:44:31.423 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.9363999999999999e-05 2021-02-10T04:44:31.423 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.423 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.423 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:44:31.423 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:48.241512+0000", 2021-02-10T04:44:31.423 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.023903297 2021-02-10T04:44:31.423 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.423 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.424 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:44:31.424 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:48.265416+0000", 2021-02-10T04:44:31.424 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.1218000000000003e-05 2021-02-10T04:44:31.424 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.424 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.424 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:44:31.424 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:48.265467+0000", 2021-02-10T04:44:31.424 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.025241980000000001 2021-02-10T04:44:31.424 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.424 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:44:31.424 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.424 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.425 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.425 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:6433 7.3 e650/646 7:c765b4cc:::smithi08575397-30:head v 650'1043, mlcod=649'1039)", 2021-02-10T04:44:31.425 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:37:48.243037+0000", 2021-02-10T04:44:31.425 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 403.16645886200001, 2021-02-10T04:44:31.425 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.022472608000000002, 2021-02-10T04:44:31.425 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:44:31.425 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:44:31.425 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:44:31.425 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.425 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:44:31.425 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:48.243037+0000", 2021-02-10T04:44:31.425 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:44:31.425 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.425 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.426 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:44:31.426 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:48.243037+0000", 2021-02-10T04:44:31.426 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.3674999999999998e-05 2021-02-10T04:44:31.426 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.426 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.426 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:44:31.426 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:48.243080+0000", 2021-02-10T04:44:31.426 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7330000000000003e-06 2021-02-10T04:44:31.426 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.426 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.426 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:44:31.426 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:48.243088+0000", 2021-02-10T04:44:31.426 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1447000000000001e-05 2021-02-10T04:44:31.427 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.427 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.427 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:44:31.427 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:48.243099+0000", 2021-02-10T04:44:31.427 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.2289999999999992e-06 2021-02-10T04:44:31.427 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.427 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.427 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:44:31.427 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:48.243109+0000", 2021-02-10T04:44:31.427 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.8050000000000003e-05 2021-02-10T04:44:31.427 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.427 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.427 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:44:31.428 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:48.243147+0000", 2021-02-10T04:44:31.428 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3259e-05 2021-02-10T04:44:31.428 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.428 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.428 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:44:31.428 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:48.243160+0000", 2021-02-10T04:44:31.428 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.022312302999999999 2021-02-10T04:44:31.428 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.428 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.428 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:44:31.428 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:48.265472+0000", 2021-02-10T04:44:31.428 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.6912000000000002e-05 2021-02-10T04:44:31.428 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.429 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.429 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:44:31.429 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:48.265509+0000", 2021-02-10T04:44:31.429 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.022472608000000002 2021-02-10T04:44:31.429 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.429 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:44:31.429 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.429 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.429 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.429 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:8071 7.4 e813/93 7:2dcefd69:::smithi08575397-28:head v 813'512, mlcod=806'508)", 2021-02-10T04:44:31.429 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:40:32.451156+0000", 2021-02-10T04:44:31.429 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 238.95833984000001, 2021-02-10T04:44:31.429 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010155643000000001, 2021-02-10T04:44:31.430 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:44:31.430 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:44:31.430 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:44:31.430 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.430 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:44:31.430 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:32.451156+0000", 2021-02-10T04:44:31.430 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:44:31.430 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.430 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.430 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:44:31.430 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:32.451156+0000", 2021-02-10T04:44:31.431 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0036666120000000001 2021-02-10T04:44:31.431 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.431 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.431 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:44:31.431 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:32.454822+0000", 2021-02-10T04:44:31.431 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6089999999999997e-06 2021-02-10T04:44:31.431 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.431 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.431 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:44:31.431 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:32.454829+0000", 2021-02-10T04:44:31.431 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.9270000000000005e-06 2021-02-10T04:44:31.431 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.431 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.432 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:44:31.432 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:32.454837+0000", 2021-02-10T04:44:31.432 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.6299999999999998e-06 2021-02-10T04:44:31.432 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.432 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.432 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:44:31.432 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:32.454844+0000", 2021-02-10T04:44:31.432 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.1373999999999999e-05 2021-02-10T04:44:31.432 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.432 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.432 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:44:31.432 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:32.454906+0000", 2021-02-10T04:44:31.432 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4502999999999999e-05 2021-02-10T04:44:31.433 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.433 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.433 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:44:31.433 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:32.454930+0000", 2021-02-10T04:44:31.433 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0063078759999999996 2021-02-10T04:44:31.433 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.433 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.433 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:44:31.433 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:32.461238+0000", 2021-02-10T04:44:31.433 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.3112000000000001e-05 2021-02-10T04:44:31.433 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.433 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.433 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:44:31.434 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:32.461311+0000", 2021-02-10T04:44:31.434 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010155643000000001 2021-02-10T04:44:31.434 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.434 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:44:31.434 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.434 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.434 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.434 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:8273 7.4 e831/93 7:2dcefd69:::smithi08575397-28:head v 831'522, mlcod=830'519)", 2021-02-10T04:44:31.434 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:40:50.521363+0000", 2021-02-10T04:44:31.434 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 220.888132531, 2021-02-10T04:44:31.434 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010385768, 2021-02-10T04:44:31.434 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:44:31.434 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:44:31.435 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:44:31.435 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.435 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:44:31.435 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:50.521363+0000", 2021-02-10T04:44:31.435 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:44:31.435 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.435 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.435 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:44:31.435 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:50.521363+0000", 2021-02-10T04:44:31.435 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0069294070000000003 2021-02-10T04:44:31.435 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.435 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.435 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:44:31.436 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:50.528292+0000", 2021-02-10T04:44:31.436 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0530000000000001e-05 2021-02-10T04:44:31.436 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.436 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.436 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:44:31.436 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:50.528303+0000", 2021-02-10T04:44:31.436 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4608e-05 2021-02-10T04:44:31.436 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.436 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.436 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:44:31.437 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:50.528318+0000", 2021-02-10T04:44:31.437 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.8810000000000002e-06 2021-02-10T04:44:31.437 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.437 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.437 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:44:31.437 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:50.528326+0000", 2021-02-10T04:44:31.437 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.9308999999999997e-05 2021-02-10T04:44:31.437 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.437 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.437 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:44:31.437 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:50.528386+0000", 2021-02-10T04:44:31.437 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.7325e-05 2021-02-10T04:44:31.437 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.437 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.438 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:44:31.438 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:50.528413+0000", 2021-02-10T04:44:31.438 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0032678939999999999 2021-02-10T04:44:31.438 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.438 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.438 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:44:31.438 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:50.531681+0000", 2021-02-10T04:44:31.438 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.7813999999999998e-05 2021-02-10T04:44:31.438 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.438 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.438 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:44:31.438 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:50.531749+0000", 2021-02-10T04:44:31.439 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010385768 2021-02-10T04:44:31.439 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.439 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:44:31.439 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.439 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.439 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.439 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:8944 7.3 e892/646 7:cf4c8c3b:::smithi08575397-6:head v 892'1451, mlcod=890'1447)", 2021-02-10T04:44:31.439 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:41:51.553544+0000", 2021-02-10T04:44:31.439 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 159.855951565, 2021-02-10T04:44:31.439 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012481709000000001, 2021-02-10T04:44:31.439 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:44:31.439 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:44:31.439 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:44:31.440 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.440 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:44:31.440 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:41:51.553544+0000", 2021-02-10T04:44:31.440 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:44:31.440 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.440 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.440 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:44:31.440 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:41:51.553544+0000", 2021-02-10T04:44:31.440 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0054530079999999996 2021-02-10T04:44:31.440 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.440 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.440 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:44:31.440 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:41:51.558997+0000", 2021-02-10T04:44:31.441 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.8299999999999996e-06 2021-02-10T04:44:31.441 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.441 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.441 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:44:31.441 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:41:51.559005+0000", 2021-02-10T04:44:31.441 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2683999999999999e-05 2021-02-10T04:44:31.441 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.441 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.441 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:44:31.441 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:41:51.559017+0000", 2021-02-10T04:44:31.441 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.3310000000000004e-06 2021-02-10T04:44:31.441 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.441 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.442 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:44:31.442 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:41:51.559026+0000", 2021-02-10T04:44:31.442 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9830999999999998e-05 2021-02-10T04:44:31.442 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.442 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.442 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:44:31.442 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:41:51.559076+0000", 2021-02-10T04:44:31.442 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4417e-05 2021-02-10T04:44:31.442 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.442 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.442 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:44:31.442 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:41:51.559100+0000", 2021-02-10T04:44:31.443 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.006869539 2021-02-10T04:44:31.443 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.443 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.443 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:44:31.443 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:41:51.565970+0000", 2021-02-10T04:44:31.443 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6069000000000001e-05 2021-02-10T04:44:31.443 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.443 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.443 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:44:31.443 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:41:51.566026+0000", 2021-02-10T04:44:31.443 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012481709000000001 2021-02-10T04:44:31.443 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.443 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:44:31.444 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.444 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.444 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.444 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:9913 7.3 e976/646 7:cf4c8c3b:::smithi08575397-6:head v 976'1596, mlcod=974'1592)", 2021-02-10T04:44:31.444 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:43:16.628245+0000", 2021-02-10T04:44:31.444 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 74.781250439000004, 2021-02-10T04:44:31.444 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0094159880000000001, 2021-02-10T04:44:31.444 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:44:31.444 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:44:31.444 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:44:31.444 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.444 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:44:31.444 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:16.628245+0000", 2021-02-10T04:44:31.445 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:44:31.445 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.445 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.445 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:44:31.445 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:16.628245+0000", 2021-02-10T04:44:31.445 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0055196899999999998 2021-02-10T04:44:31.445 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.445 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.445 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:44:31.445 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:16.633765+0000", 2021-02-10T04:44:31.445 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.2640000000000002e-06 2021-02-10T04:44:31.445 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.446 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.446 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:44:31.446 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:16.633772+0000", 2021-02-10T04:44:31.446 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7260000000000001e-05 2021-02-10T04:44:31.446 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.446 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.446 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:44:31.446 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:16.633789+0000", 2021-02-10T04:44:31.446 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.2090000000000002e-06 2021-02-10T04:44:31.446 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.446 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.446 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:44:31.447 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:16.633795+0000", 2021-02-10T04:44:31.447 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.3785000000000002e-05 2021-02-10T04:44:31.447 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.447 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.447 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:44:31.447 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:16.633839+0000", 2021-02-10T04:44:31.447 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4403999999999999e-05 2021-02-10T04:44:31.447 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.447 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.447 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:44:31.447 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:16.633864+0000", 2021-02-10T04:44:31.447 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0037348640000000001 2021-02-10T04:44:31.448 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.448 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.448 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:44:31.448 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:16.637599+0000", 2021-02-10T04:44:31.448 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.2512000000000001e-05 2021-02-10T04:44:31.448 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.448 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.448 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:44:31.448 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:16.637661+0000", 2021-02-10T04:44:31.448 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0094159880000000001 2021-02-10T04:44:31.448 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.448 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:44:31.448 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.449 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.449 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.449 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:865 7.11 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29] snapc 37=[36,32,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007)", 2021-02-10T04:44:31.449 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:24.520229+0000", 2021-02-10T04:44:31.449 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 6.8892669059999996, 2021-02-10T04:44:31.449 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0090932100000001, 2021-02-10T04:44:31.449 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:44:31.449 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:44:31.449 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:44:31.449 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:44:31.449 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:44:31.449 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 865 2021-02-10T04:44:31.450 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.450 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:44:31.450 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.450 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:44:31.450 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520229+0000", 2021-02-10T04:44:31.450 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:44:31.450 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.450 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.450 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:44:31.450 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520229+0000", 2021-02-10T04:44:31.450 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5864e-05 2021-02-10T04:44:31.450 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.450 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.450 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:44:31.451 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520274+0000", 2021-02-10T04:44:31.451 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.2700000000000004e-06 2021-02-10T04:44:31.451 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.451 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.451 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:44:31.451 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520283+0000", 2021-02-10T04:44:31.451 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4335000000000001e-05 2021-02-10T04:44:31.451 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.451 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.451 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:44:31.451 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520297+0000", 2021-02-10T04:44:31.451 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1950000000000003e-06 2021-02-10T04:44:31.451 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.452 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.452 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:44:31.452 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520305+0000", 2021-02-10T04:44:31.452 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.038909046000000003 2021-02-10T04:44:31.452 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.452 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.452 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:44:31.452 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.559214+0000", 2021-02-10T04:44:31.452 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1300000000000001e-06 2021-02-10T04:44:31.452 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.452 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.452 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for peered", 2021-02-10T04:44:31.452 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.559222+0000", 2021-02-10T04:44:31.453 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0032650030000001 2021-02-10T04:44:31.453 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.453 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.453 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:44:31.453 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:25.562487+0000", 2021-02-10T04:44:31.453 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00013934400000000001 2021-02-10T04:44:31.453 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.453 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.453 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:44:31.453 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:25.562627+0000", 2021-02-10T04:44:31.453 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.96669502299999999 2021-02-10T04:44:31.453 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.454 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.454 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:44:31.454 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.529322+0000", 2021-02-10T04:44:31.454 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0090932100000001 2021-02-10T04:44:31.454 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.454 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:44:31.454 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.454 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.454 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.454 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:866 7.11 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007)", 2021-02-10T04:44:31.454 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:24.520353+0000", 2021-02-10T04:44:31.454 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 6.8891427619999996, 2021-02-10T04:44:31.454 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0096650170000001, 2021-02-10T04:44:31.455 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:44:31.455 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T04:44:31.455 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:44:31.455 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:44:31.455 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:44:31.455 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 866 2021-02-10T04:44:31.455 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.455 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:44:31.455 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.455 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:44:31.455 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520353+0000", 2021-02-10T04:44:31.455 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:44:31.455 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.455 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.456 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:44:31.456 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520353+0000", 2021-02-10T04:44:31.456 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.6482000000000001e-05 2021-02-10T04:44:31.456 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.456 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.456 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:44:31.456 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520389+0000", 2021-02-10T04:44:31.456 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.7499999999999997e-06 2021-02-10T04:44:31.456 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.456 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.456 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:44:31.456 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520396+0000", 2021-02-10T04:44:31.456 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2269999999999999e-05 2021-02-10T04:44:31.457 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.457 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.457 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:44:31.457 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520408+0000", 2021-02-10T04:44:31.457 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.5300000000000004e-06 2021-02-10T04:44:31.457 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.457 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.457 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:44:31.457 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520414+0000", 2021-02-10T04:44:31.457 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.038829508999999998 2021-02-10T04:44:31.457 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.457 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.458 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:44:31.458 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.559243+0000", 2021-02-10T04:44:31.458 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.3689999999999998e-06 2021-02-10T04:44:31.458 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.458 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.458 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for peered", 2021-02-10T04:44:31.458 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.559249+0000", 2021-02-10T04:44:31.458 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.003527509 2021-02-10T04:44:31.459 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.459 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.459 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:44:31.459 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:25.562776+0000", 2021-02-10T04:44:31.459 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5167999999999999e-05 2021-02-10T04:44:31.459 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.459 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.459 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:44:31.459 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:25.562821+0000", 2021-02-10T04:44:31.459 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.967184818 2021-02-10T04:44:31.459 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.459 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.460 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:44:31.460 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.530006+0000", 2021-02-10T04:44:31.460 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1612e-05 2021-02-10T04:44:31.460 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.460 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.460 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:44:31.460 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.530018+0000", 2021-02-10T04:44:31.460 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0096650170000001 2021-02-10T04:44:31.460 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.460 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:44:31.460 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.460 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.460 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.461 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:2821 7.11 7:8eb2e6b5:::smithi08575397-27:head [copy-from ver 168] snapc cc=[cc,cb,c9] RETRY=1 ondisk+retry+write+known_if_redirected e1007)", 2021-02-10T04:44:31.461 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:24.520839+0000", 2021-02-10T04:44:31.461 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 6.8886567010000004, 2021-02-10T04:44:31.461 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0085211759999999, 2021-02-10T04:44:31.461 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:44:31.461 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:44:31.461 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:44:31.461 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:44:31.461 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:44:31.461 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 2821 2021-02-10T04:44:31.461 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.461 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:44:31.462 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.462 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:44:31.462 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520839+0000", 2021-02-10T04:44:31.462 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:44:31.462 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.462 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.462 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:44:31.462 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520839+0000", 2021-02-10T04:44:31.462 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2160999999999998e-05 2021-02-10T04:44:31.462 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.462 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.462 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:44:31.462 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520881+0000", 2021-02-10T04:44:31.463 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.9889999999999997e-06 2021-02-10T04:44:31.463 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.463 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.463 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:44:31.463 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520887+0000", 2021-02-10T04:44:31.463 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2689e-05 2021-02-10T04:44:31.463 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.463 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.463 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:44:31.463 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520900+0000", 2021-02-10T04:44:31.463 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.0610000000000003e-06 2021-02-10T04:44:31.463 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.463 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.464 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:44:31.464 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520905+0000", 2021-02-10T04:44:31.464 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.038414679 2021-02-10T04:44:31.464 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.464 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.464 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:44:31.464 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.559319+0000", 2021-02-10T04:44:31.464 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.2280000000000003e-06 2021-02-10T04:44:31.464 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.464 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.465 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for peered", 2021-02-10T04:44:31.465 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.559326+0000", 2021-02-10T04:44:31.465 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0035212650000001 2021-02-10T04:44:31.465 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.465 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.465 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:44:31.465 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:25.562847+0000", 2021-02-10T04:44:31.465 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000111644 2021-02-10T04:44:31.465 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.465 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.465 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:44:31.465 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:25.562959+0000", 2021-02-10T04:44:31.466 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.96640146000000005 2021-02-10T04:44:31.466 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.466 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.466 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:44:31.466 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.529360+0000", 2021-02-10T04:44:31.466 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0085211759999999 2021-02-10T04:44:31.466 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.466 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:44:31.466 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.466 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.466 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.466 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:2822 7.11 7:8eb2e6b5:::smithi08575397-27:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007)", 2021-02-10T04:44:31.467 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:24.520938+0000", 2021-02-10T04:44:31.467 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 6.888557896, 2021-02-10T04:44:31.467 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0089547369999998, 2021-02-10T04:44:31.467 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:44:31.467 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T04:44:31.467 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:44:31.467 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:44:31.467 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:44:31.467 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 2822 2021-02-10T04:44:31.467 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.467 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:44:31.467 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.467 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:44:31.468 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520938+0000", 2021-02-10T04:44:31.468 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:44:31.468 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.468 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.468 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:44:31.468 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520938+0000", 2021-02-10T04:44:31.468 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1031999999999997e-05 2021-02-10T04:44:31.468 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.468 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.468 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:44:31.468 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520969+0000", 2021-02-10T04:44:31.468 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6210000000000001e-06 2021-02-10T04:44:31.468 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.469 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.469 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:44:31.469 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520974+0000", 2021-02-10T04:44:31.469 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4473e-05 2021-02-10T04:44:31.469 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.469 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.469 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:44:31.469 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520989+0000", 2021-02-10T04:44:31.469 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.921e-06 2021-02-10T04:44:31.469 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.469 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.469 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:44:31.469 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520994+0000", 2021-02-10T04:44:31.469 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.038350782 2021-02-10T04:44:31.470 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.470 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.470 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:44:31.470 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.559344+0000", 2021-02-10T04:44:31.470 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0140000000000004e-06 2021-02-10T04:44:31.470 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.470 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.470 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for peered", 2021-02-10T04:44:31.470 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.559350+0000", 2021-02-10T04:44:31.470 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0037162040000001 2021-02-10T04:44:31.470 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.471 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.471 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:44:31.471 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:25.563067+0000", 2021-02-10T04:44:31.471 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.1128999999999999e-05 2021-02-10T04:44:31.471 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.471 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.471 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:44:31.471 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:25.563108+0000", 2021-02-10T04:44:31.471 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.966768399 2021-02-10T04:44:31.471 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.471 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.471 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:44:31.471 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.529876+0000", 2021-02-10T04:44:31.472 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6161999999999999e-05 2021-02-10T04:44:31.472 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.472 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.472 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:44:31.472 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.529892+0000", 2021-02-10T04:44:31.472 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0089547369999998 2021-02-10T04:44:31.472 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.472 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:44:31.472 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.472 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.472 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.472 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.16 1011/1010 [PushOp(7:6cacee86:::smithi08575397-26:2f7, version: 988'471, data_included: [1559253~178321,2089081~578711], data_size: 757032, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:6cacee86:::smithi08575397-26:2f7@988'471, size: 2756612, copy_subset: [0~2756612], clone_subset: {7:6cacee86:::smithi08575397-26:2b9=[]}, snapset: 374=[]:{2b9=[2ae,27c],2f7=[2f5,2f0,2ee,2ed,2eb,2ea,2d9],30d=[308,300],333=[332,32f,32a,328,31a],374=[374,373,372,370,36f,36b,368,366,35f,354,353,34e,34d,34b,349,345,344,343]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2756612, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1559253, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:44:31.472 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:28.656373+0000", 2021-02-10T04:44:31.473 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 2.753122066, 2021-02-10T04:44:31.473 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0099672589999999991, 2021-02-10T04:44:31.473 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:44:31.473 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:44:31.473 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:44:31.473 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.473 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:44:31.473 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.656373+0000", 2021-02-10T04:44:31.473 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:44:31.473 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.473 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.473 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:44:31.473 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.656373+0000", 2021-02-10T04:44:31.474 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0013001670000000001 2021-02-10T04:44:31.474 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.474 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.474 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:44:31.474 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.657674+0000", 2021-02-10T04:44:31.474 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0499000000000001e-05 2021-02-10T04:44:31.474 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.474 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.474 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:44:31.474 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.657694+0000", 2021-02-10T04:44:31.474 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.3170999999999998e-05 2021-02-10T04:44:31.474 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.475 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.475 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:44:31.475 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.657727+0000", 2021-02-10T04:44:31.475 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8783999999999998e-05 2021-02-10T04:44:31.475 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.475 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.475 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:44:31.475 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.657746+0000", 2021-02-10T04:44:31.475 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0073734930000000001 2021-02-10T04:44:31.475 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.475 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.475 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:44:31.475 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.665120+0000", 2021-02-10T04:44:31.476 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0694e-05 2021-02-10T04:44:31.476 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.476 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.476 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:44:31.476 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.665130+0000", 2021-02-10T04:44:31.476 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001210451 2021-02-10T04:44:31.476 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.476 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.476 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:44:31.476 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.666341+0000", 2021-02-10T04:44:31.476 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0099672589999999991 2021-02-10T04:44:31.476 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.476 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:44:31.477 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.477 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.477 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.477 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.16 1011/1010 [PushOp(7:6cacee86:::smithi08575397-26:30d, version: 995'479, data_included: [480997~434941,1407473~613635], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:6cacee86:::smithi08575397-26:30d@995'479, size: 3680997, copy_subset: [480997~434941,1407473~705079,2557501~1123496], clone_subset: {7:6cacee86:::smithi08575397-26:2f7=[0~480997,915938~491535,2112552~444949]}, snapset: 374=[]:{2b9=[2ae,27c],2f7=[2f5,2f0,2ee,2ed,2eb,2ea,2d9],30d=[308,300],333=[332,32f,32a,328,31a],374=[374,373,372,370,36f,36b,368,366,35f,354,353,34e,34d,34b,349,345,344,343]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2021108, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2021-02-10T04:44:31.477 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:28.668911+0000", 2021-02-10T04:44:31.477 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 2.7405840490000002, 2021-02-10T04:44:31.477 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0088416129999999999, 2021-02-10T04:44:31.477 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:44:31.477 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:44:31.477 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:44:31.477 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.477 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:44:31.477 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.668911+0000", 2021-02-10T04:44:31.478 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:44:31.478 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.478 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.478 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:44:31.478 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.668911+0000", 2021-02-10T04:44:31.478 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00097853400000000009 2021-02-10T04:44:31.478 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.478 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.478 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:44:31.478 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.669890+0000", 2021-02-10T04:44:31.478 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2739e-05 2021-02-10T04:44:31.478 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.478 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.479 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:44:31.479 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.669913+0000", 2021-02-10T04:44:31.479 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.9453999999999999e-05 2021-02-10T04:44:31.479 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.479 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.479 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:44:31.479 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.669952+0000", 2021-02-10T04:44:31.479 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2317e-05 2021-02-10T04:44:31.479 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.479 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.479 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:44:31.479 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.669974+0000", 2021-02-10T04:44:31.479 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0070590729999999999 2021-02-10T04:44:31.479 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.480 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.480 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:44:31.480 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.677034+0000", 2021-02-10T04:44:31.480 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0604999999999999e-05 2021-02-10T04:44:31.480 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.480 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.480 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:44:31.480 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.677044+0000", 2021-02-10T04:44:31.480 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00070889100000000001 2021-02-10T04:44:31.480 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.480 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.480 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:44:31.480 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.677753+0000", 2021-02-10T04:44:31.481 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0088416129999999999 2021-02-10T04:44:31.481 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.481 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:44:31.481 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.481 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.481 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.481 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.16 1011/1010 [PushOp(7:6cacee86:::smithi08575397-26:30d, version: 995'479, data_included: [2021108~91444,2557501~785880], data_size: 877324, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:6cacee86:::smithi08575397-26:30d@995'479, size: 3680997, copy_subset: [480997~434941,1407473~705079,2557501~1123496], clone_subset: {7:6cacee86:::smithi08575397-26:2f7=[0~480997,915938~491535,2112552~444949]}, snapset: 374=[]:{2b9=[2ae,27c],2f7=[2f5,2f0,2ee,2ed,2eb,2ea,2d9],30d=[308,300],333=[332,32f,32a,328,31a],374=[374,373,372,370,36f,36b,368,366,35f,354,353,34e,34d,34b,349,345,344,343]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3680997, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2021108, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:44:31.481 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:28.680110+0000", 2021-02-10T04:44:31.481 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 2.7293856430000001, 2021-02-10T04:44:31.481 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0099816599999999998, 2021-02-10T04:44:31.481 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:44:31.481 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:44:31.481 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:44:31.482 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.482 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:44:31.482 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.680110+0000", 2021-02-10T04:44:31.482 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:44:31.482 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.482 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.482 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:44:31.482 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.680110+0000", 2021-02-10T04:44:31.482 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0011890309999999999 2021-02-10T04:44:31.482 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.482 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.482 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:44:31.482 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.681299+0000", 2021-02-10T04:44:31.483 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4763e-05 2021-02-10T04:44:31.483 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.483 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.483 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:44:31.483 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.681324+0000", 2021-02-10T04:44:31.483 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.0151999999999998e-05 2021-02-10T04:44:31.483 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.483 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.483 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:44:31.483 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.681364+0000", 2021-02-10T04:44:31.483 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0234e-05 2021-02-10T04:44:31.483 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.483 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.484 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:44:31.484 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.681384+0000", 2021-02-10T04:44:31.484 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0075636619999999996 2021-02-10T04:44:31.484 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.484 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.484 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:44:31.484 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.688948+0000", 2021-02-10T04:44:31.484 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0399e-05 2021-02-10T04:44:31.484 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.484 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.484 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:44:31.484 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.688958+0000", 2021-02-10T04:44:31.484 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0011334190000000001 2021-02-10T04:44:31.484 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.485 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.485 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:44:31.485 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.690092+0000", 2021-02-10T04:44:31.485 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0099816599999999998 2021-02-10T04:44:31.485 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.485 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:44:31.485 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.485 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.485 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.485 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.16 1011/1010 [PushOp(7:6cacee86:::smithi08575397-26:333, version: 996'485, data_included: [686049~409317,1847413~438636], data_size: 847953, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:6cacee86:::smithi08575397-26:333@996'485, size: 2286049, copy_subset: [686049~409317,1847413~438636], clone_subset: {7:6cacee86:::smithi08575397-26:30d=[0~686049,1095366~752047]}, snapset: 374=[]:{2b9=[2ae,27c],2f7=[2f5,2f0,2ee,2ed,2eb,2ea,2d9],30d=[308,300],333=[332,32f,32a,328,31a],374=[374,373,372,370,36f,36b,368,366,35f,354,353,34e,34d,34b,349,345,344,343]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2286049, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2021-02-10T04:44:31.485 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:28.691816+0000", 2021-02-10T04:44:31.485 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 2.717679908, 2021-02-10T04:44:31.485 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010082539, 2021-02-10T04:44:31.486 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:44:31.486 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:44:31.486 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:44:31.486 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.486 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:44:31.486 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.691816+0000", 2021-02-10T04:44:31.486 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:44:31.486 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.486 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.486 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:44:31.486 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.691816+0000", 2021-02-10T04:44:31.486 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0011081540000000001 2021-02-10T04:44:31.487 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.487 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.487 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:44:31.487 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.692924+0000", 2021-02-10T04:44:31.487 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8170000000000001e-05 2021-02-10T04:44:31.487 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.487 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.487 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:44:31.487 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.692942+0000", 2021-02-10T04:44:31.487 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1245000000000003e-05 2021-02-10T04:44:31.487 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.487 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.487 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:44:31.487 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.692973+0000", 2021-02-10T04:44:31.488 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8142999999999999e-05 2021-02-10T04:44:31.488 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.488 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.488 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:44:31.488 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.692991+0000", 2021-02-10T04:44:31.488 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0078590580000000004 2021-02-10T04:44:31.488 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.488 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.488 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:44:31.488 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.700850+0000", 2021-02-10T04:44:31.488 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1731e-05 2021-02-10T04:44:31.488 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.489 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.489 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:44:31.489 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.700862+0000", 2021-02-10T04:44:31.489 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001036038 2021-02-10T04:44:31.489 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.489 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.489 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:44:31.489 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.701898+0000", 2021-02-10T04:44:31.489 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010082539 2021-02-10T04:44:31.489 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.489 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:44:31.489 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.489 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.489 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.490 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.16 1011/1010 [PushOp(7:6cacee86:::smithi08575397-26:374, version: 1000'489, data_included: [796995~758622,2161832~289954], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:6cacee86:::smithi08575397-26:374@1000'489, size: 3596995, copy_subset: [0~3596995], clone_subset: {7:6cacee86:::smithi08575397-26:333=[]}, snapset: 374=[]:{2b9=[2ae,27c],2f7=[2f5,2f0,2ee,2ed,2eb,2ea,2d9],30d=[308,300],333=[332,32f,32a,328,31a],374=[374,373,372,370,36f,36b,368,366,35f,354,353,34e,34d,34b,349,345,344,343]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2451786, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2021-02-10T04:44:31.490 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:28.704687+0000", 2021-02-10T04:44:31.490 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 2.7048080909999999, 2021-02-10T04:44:31.490 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0089517510000000008, 2021-02-10T04:44:31.490 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:44:31.490 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:44:31.490 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:44:31.490 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.490 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:44:31.490 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.704687+0000", 2021-02-10T04:44:31.490 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:44:31.490 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.491 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.491 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:44:31.491 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.704687+0000", 2021-02-10T04:44:31.491 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00099512100000000007 2021-02-10T04:44:31.491 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.491 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.491 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:44:31.491 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.705683+0000", 2021-02-10T04:44:31.491 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.946e-05 2021-02-10T04:44:31.491 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.491 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.491 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:44:31.491 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.705702+0000", 2021-02-10T04:44:31.492 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.1569000000000001e-05 2021-02-10T04:44:31.492 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.492 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.492 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:44:31.492 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.705744+0000", 2021-02-10T04:44:31.492 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5209e-05 2021-02-10T04:44:31.492 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.492 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.492 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:44:31.492 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.705759+0000", 2021-02-10T04:44:31.492 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0070010109999999997 2021-02-10T04:44:31.492 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.492 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.492 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:44:31.493 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.712760+0000", 2021-02-10T04:44:31.493 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.3649999999999998e-06 2021-02-10T04:44:31.493 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.493 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.493 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:44:31.493 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.712767+0000", 2021-02-10T04:44:31.493 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00087201599999999998 2021-02-10T04:44:31.493 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.493 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.493 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:44:31.493 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.713639+0000", 2021-02-10T04:44:31.493 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0089517510000000008 2021-02-10T04:44:31.493 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.494 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:44:31.494 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.494 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.494 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.494 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.16 1011/1010 [PushOp(7:6cacee86:::smithi08575397-26:head, version: 1000'490, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7:6cacee86:::smithi08575397-26:head@1000'490, size: 3697837, copy_subset: [0~3697837], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2021-02-10T04:44:31.494 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:28.721110+0000", 2021-02-10T04:44:31.494 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 2.688385443, 2021-02-10T04:44:31.494 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0087620460000000008, 2021-02-10T04:44:31.494 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:44:31.494 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:44:31.494 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:44:31.494 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.494 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:44:31.495 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.721110+0000", 2021-02-10T04:44:31.495 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:44:31.495 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.495 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.495 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:44:31.495 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.721110+0000", 2021-02-10T04:44:31.495 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001579193 2021-02-10T04:44:31.495 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.495 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.495 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:44:31.495 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.722689+0000", 2021-02-10T04:44:31.495 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8491e-05 2021-02-10T04:44:31.495 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.496 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.496 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:44:31.496 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.722708+0000", 2021-02-10T04:44:31.496 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.451e-05 2021-02-10T04:44:31.496 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.496 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.496 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:44:31.496 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.722742+0000", 2021-02-10T04:44:31.496 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2714999999999999e-05 2021-02-10T04:44:31.496 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.496 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.496 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:44:31.496 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.722755+0000", 2021-02-10T04:44:31.496 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0063422640000000002 2021-02-10T04:44:31.497 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.497 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.497 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:44:31.497 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.729097+0000", 2021-02-10T04:44:31.497 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1415e-05 2021-02-10T04:44:31.497 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.497 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.497 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:44:31.497 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.729109+0000", 2021-02-10T04:44:31.497 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00076345799999999995 2021-02-10T04:44:31.497 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.497 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.497 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:44:31.498 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.729872+0000", 2021-02-10T04:44:31.498 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0087620460000000008 2021-02-10T04:44:31.498 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.498 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:44:31.498 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.498 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.498 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.498 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.16 1011/1010 [PushOp(7:6cacee86:::smithi08575397-26:head, version: 1000'490, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:6cacee86:::smithi08575397-26:head@1000'490, size: 3697837, copy_subset: [0~3697837], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:44:31.498 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:28.731732+0000", 2021-02-10T04:44:31.498 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 2.6777635310000001, 2021-02-10T04:44:31.498 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0099831009999999994, 2021-02-10T04:44:31.498 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:44:31.499 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:44:31.499 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:44:31.499 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.499 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:44:31.499 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.731732+0000", 2021-02-10T04:44:31.499 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:44:31.499 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.499 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.499 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:44:31.499 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.731732+0000", 2021-02-10T04:44:31.499 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0010008879999999999 2021-02-10T04:44:31.499 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.499 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.499 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:44:31.500 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.732733+0000", 2021-02-10T04:44:31.500 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6137000000000002e-05 2021-02-10T04:44:31.500 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.500 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.500 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:44:31.500 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.732749+0000", 2021-02-10T04:44:31.500 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.5225e-05 2021-02-10T04:44:31.500 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.500 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.500 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:44:31.500 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.732774+0000", 2021-02-10T04:44:31.500 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2556e-05 2021-02-10T04:44:31.500 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.501 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.501 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:44:31.501 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.732787+0000", 2021-02-10T04:44:31.501 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0082074980000000006 2021-02-10T04:44:31.501 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.501 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.501 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:44:31.501 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.740994+0000", 2021-02-10T04:44:31.501 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4891e-05 2021-02-10T04:44:31.501 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.501 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.501 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:44:31.501 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.741009+0000", 2021-02-10T04:44:31.502 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00070590599999999996 2021-02-10T04:44:31.502 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.502 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.502 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:44:31.502 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.741715+0000", 2021-02-10T04:44:31.502 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0099831009999999994 2021-02-10T04:44:31.502 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.502 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:44:31.502 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.502 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.502 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.502 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.16 1011/1010 [PushOp(7:6cacee86:::smithi08575397-26:head, version: 1000'490, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:6cacee86:::smithi08575397-26:head@1000'490, size: 3697837, copy_subset: [0~3697837], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:44:31.503 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:28.743396+0000", 2021-02-10T04:44:31.503 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 2.666099854, 2021-02-10T04:44:31.503 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010392541999999999, 2021-02-10T04:44:31.503 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:44:31.503 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:44:31.503 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:44:31.503 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.503 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:44:31.503 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.743396+0000", 2021-02-10T04:44:31.503 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:44:31.503 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.503 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.503 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:44:31.503 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.743396+0000", 2021-02-10T04:44:31.504 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00097603700000000002 2021-02-10T04:44:31.504 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.504 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.504 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:44:31.504 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.744372+0000", 2021-02-10T04:44:31.504 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4667999999999999e-05 2021-02-10T04:44:31.504 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.504 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.504 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:44:31.504 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.744386+0000", 2021-02-10T04:44:31.504 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.6957999999999998e-05 2021-02-10T04:44:31.504 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.504 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.505 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:44:31.505 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.744413+0000", 2021-02-10T04:44:31.505 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2369e-05 2021-02-10T04:44:31.505 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.505 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.505 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:44:31.505 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.744426+0000", 2021-02-10T04:44:31.505 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00859854 2021-02-10T04:44:31.505 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.505 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.505 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:44:31.505 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.753024+0000", 2021-02-10T04:44:31.505 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7890000000000006e-06 2021-02-10T04:44:31.506 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.506 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.506 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:44:31.506 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.753034+0000", 2021-02-10T04:44:31.506 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00075418100000000004 2021-02-10T04:44:31.506 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.506 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.506 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:44:31.506 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.753788+0000", 2021-02-10T04:44:31.506 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010392541999999999 2021-02-10T04:44:31.506 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.506 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:44:31.506 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.507 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.507 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.507 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.16 1011/1010 [PushOp(7:6cacee86:::smithi08575397-26:head, version: 1000'490, data_included: [3145728~552109], data_size: 552109, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:6cacee86:::smithi08575397-26:head@1000'490, size: 3697837, copy_subset: [0~3697837], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3697837, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:44:31.507 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:28.755386+0000", 2021-02-10T04:44:31.507 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 2.6541090660000002, 2021-02-10T04:44:31.507 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0099515290000000006, 2021-02-10T04:44:31.507 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:44:31.507 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:44:31.507 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:44:31.507 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.507 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:44:31.507 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.755386+0000", 2021-02-10T04:44:31.507 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:44:31.508 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.508 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.508 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:44:31.508 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.755386+0000", 2021-02-10T04:44:31.508 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00085900099999999997 2021-02-10T04:44:31.508 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.508 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.508 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:44:31.508 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.756245+0000", 2021-02-10T04:44:31.508 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5361000000000001e-05 2021-02-10T04:44:31.508 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.508 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.508 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:44:31.509 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.756261+0000", 2021-02-10T04:44:31.509 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.7002999999999998e-05 2021-02-10T04:44:31.509 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.509 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.509 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:44:31.509 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.756298+0000", 2021-02-10T04:44:31.509 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1515000000000001e-05 2021-02-10T04:44:31.509 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.509 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.509 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:44:31.509 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.756309+0000", 2021-02-10T04:44:31.509 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0084687660000000008 2021-02-10T04:44:31.509 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.509 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.510 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:44:31.510 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.764778+0000", 2021-02-10T04:44:31.510 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1851000000000001e-05 2021-02-10T04:44:31.510 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.510 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.510 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:44:31.510 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.764790+0000", 2021-02-10T04:44:31.510 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00054803199999999995 2021-02-10T04:44:31.510 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:44:31.510 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:44:31.510 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:44:31.510 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.765338+0000", 2021-02-10T04:44:31.511 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0099515290000000006 2021-02-10T04:44:31.511 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.511 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:44:31.511 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.511 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:44:31.511 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:44:31.511 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:44:31.512 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:44:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:31 smithi164 conmon[29288]: cluster 2021-02-10T04:44:30.280168+0000 mgr.y (mgr 2021-02-10T04:44:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:31 smithi164 conmon[29288]: .14140) 943 : cluster [DBG] pgmap v1688: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 3 peering, 3 unknown, 170 active+clean; 758 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 68 MiB/s rd, 18 MiB/s wr, 117 op/s; 60/2046 objects misplaced (2.933%); 6.9 MiB/s, 25 objects/s recovering 2021-02-10T04:44:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:31 smithi164 conmon[29288]: cluster 2021-02-10T04:44:30.558564+0000 mon.a (mon.0) 2024 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2021-02-10T04:44:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:31 smithi164 conmon[29288]: cluster 2021-02-10T04:44:30.569965+0000 mon.a (mon.0) 2025 : cluster [DBG] osdmap e1013: 8 total, 8 up, 7 in 2021-02-10T04:44:31.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:31 smithi085 conmon[35961]: cluster 2021-02-10T04:44:30 2021-02-10T04:44:31.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:31 smithi085 conmon[35961]: .280168+0000 mgr.y (mgr.14140) 943 : cluster [DBG] pgmap v1688: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 3 peering, 3 unknown, 170 active+clean; 758 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 68 MiB/s rd, 18 MiB/s wr, 117 op/s; 60/2046 objects misplaced (2.933%); 6.9 MiB/s, 25 objects/s recovering 2021-02-10T04:44:31.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:31 smithi085 conmon[35961]: cluster 2021-02-10T04:44:30.558564+0000 mon.a (mon.0) 2024 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2021-02-10T04:44:31.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:31 smithi085 conmon[35961]: cluster 2021-02-10T04:44:30.569965+0000 mon.a (mon.0) 2025 : cluster [DBG] osdmap e1013: 8 total, 8 up, 7 in 2021-02-10T04:44:31.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:31 smithi085 conmon[29319]: cluster 2021-02-10T04:44:30.280168+0000 mgr.y (mgr. 2021-02-10T04:44:31.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:31 smithi085 conmon[29319]: 14140) 943 : cluster [DBG] pgmap v1688: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 3 peering, 3 unknown, 170 active+clean; 758 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 68 MiB/s rd, 18 MiB/s wr, 117 op/s; 60/2046 objects misplaced (2.933%); 6.9 MiB/s, 25 objects/s recovering 2021-02-10T04:44:31.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:31 smithi085 conmon[29319]: cluster 2021-02-10T04:44:30.558564+0000 mon.a (mon.0) 2024 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2021-02-10T04:44:31.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:31 smithi085 conmon[29319]: cluster 2021-02-10T04:44:30. 2021-02-10T04:44:31.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:31 smithi085 conmon[29319]: 569965+0000 mon.a (mon.0) 2025 : cluster [DBG] 2021-02-10T04:44:31.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:31 smithi085 conmon[29319]: osdmap e1013: 8 total, 8 up, 7 in 2021-02-10T04:44:32.358 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:44:32.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:44:32 smithi164 conmon[40892]: debug 2021-02-10T04:44:32.068+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:44:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:33 smithi164 conmon[29288]: cluster 2021-02-10T04:44:31.738147+0000 osd.3 (osd.3) 93 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:44:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:33 smithi164 conmon[29288]: cluster 2021-02-10T04:44:31.739991+0000 osd.3 (osd.3) 94 : cluster [DBG] purged_snaps scrub found 4 strays 2021-02-10T04:44:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:33 smithi164 conmon[29288]: cluster 2021-02-10T04:44:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:33 smithi164 conmon[29288]: 2021-02-10T04:44:32.280913+0000 mgr.y (mgr.14140) 944 : cluster [DBG] pgmap v1690: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 2 peering, 3 unknown, 171 active+clean; 776 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 101 MiB/s rd, 25 MiB/s wr, 167 op/s; 60/2067 objects misplaced (2.903%); 8.1 MiB/s, 28 objects/s recovering 2021-02-10T04:44:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:33 smithi164 conmon[29288]: 2021-02-10T04:44:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:33 smithi164 conmon[29288]: cluster 2021-02-10T04:44:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:33 smithi164 conmon[29288]: 2021-02-10T04:44:32.576977+0000 mon.a ( 2021-02-10T04:44:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:33 smithi164 conmon[29288]: mon.0) 2026 : cluster [DBG] osdmap e1014: 8 total, 8 up, 7 in 2021-02-10T04:44:33.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:33 smithi085 conmon[35961]: cluster 2021-02-10T04:44:31.738147+0000 osd.3 ( 2021-02-10T04:44:33.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:33 smithi085 conmon[35961]: osd.3) 93 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:44:33.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:33 smithi085 conmon[35961]: cluster 2021-02-10T04:44:31.739991+0000 osd.3 (osd 2021-02-10T04:44:33.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:33 smithi085 conmon[35961]: .3) 94 : cluster [DBG] purged_snaps scrub found 4 strays 2021-02-10T04:44:33.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:33 smithi085 conmon[35961]: cluster 2021-02-10T04:44:33.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:33 smithi085 conmon[35961]: 2021-02-10T04:44:32.280913+0000 mgr.y 2021-02-10T04:44:33.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:33 smithi085 conmon[35961]: (mgr.14140) 944 : cluster [DBG] pgmap v1690: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 2 peering, 3 unknown, 171 active+clean; 776 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 101 MiB/s rd, 25 MiB/s wr, 167 op/s; 60/2067 objects misplaced (2.903%); 8.1 MiB/s, 28 objects/s recovering 2021-02-10T04:44:33.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:33 smithi085 conmon[35961]: cluster 2021-02-10T04:44:33.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:33 smithi085 conmon[35961]: 2021-02 2021-02-10T04:44:33.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:33 smithi085 conmon[35961]: -10T04:44:32. 2021-02-10T04:44:33.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:33 smithi085 conmon[35961]: 576977+0000 mon.a (mon.0) 2026 2021-02-10T04:44:33.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:33 smithi085 conmon[35961]: : cluster [DBG] osdmap e1014: 8 total, 8 up, 7 in 2021-02-10T04:44:33.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:33 smithi085 conmon[29319]: cluster 2021-02-10T04:44:31.738147+0000 osd.3 2021-02-10T04:44:33.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:33 smithi085 conmon[29319]: (osd.3) 93 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:44:33.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:33 smithi085 conmon[29319]: cluster 2021-02-10T04:44:31.739991+0000 osd.3 (osd.3) 94 : 2021-02-10T04:44:33.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:33 smithi085 conmon[29319]: cluster [DBG] purged_snaps scrub found 4 strays 2021-02-10T04:44:33.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:33 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:44:33.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:33 smithi085 conmon[29319]: -10T04:44:32.280913+0000 mgr.y (mgr.14140) 944 : cluster [DBG] 2021-02-10T04:44:33.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:33 smithi085 conmon[29319]: pgmap v1690: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 2 peering, 3 unknown, 171 active+clean; 776 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 101 MiB/s rd, 25 MiB/s wr, 167 op/s; 60/2067 objects misplaced (2.903%); 8.1 MiB/s, 28 objects/s recovering 2021-02-10T04:44:33.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:33 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:44:33.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:33 smithi085 conmon[29319]: 04:44:32.576977+0000 mon.a (mon.0) 2026 : cluster [DBG] osdmap e1014: 8 total, 8 up, 7 in 2021-02-10T04:44:34.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:34 smithi164 conmon[29288]: cluster 2021-02-10T04:44:33.970529+0000 osd.0 (osd. 2021-02-10T04:44:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:34 smithi164 conmon[29288]: 0) 98 : cluster [DBG] 6.9 scrub ok 2021-02-10T04:44:34.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:34 smithi085 conmon[29319]: cluster 2021-02-10T04:44:33.970529+0000 osd.0 (osd.0) 98 : cluster 2021-02-10T04:44:34.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:34 smithi085 conmon[29319]: [DBG] 6.9 scrub ok 2021-02-10T04:44:34.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:34 smithi085 conmon[35961]: cluster 2021-02-10T04:44:33.970529+0000 osd.0 2021-02-10T04:44:34.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:34 smithi085 conmon[35961]: (osd.0) 98 : cluster [DBG] 6.9 scrub ok 2021-02-10T04:44:35.875 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:35 smithi085 conmon[29319]: cluster 2021-02-10T04:44:34.282064+0000 mgr.y (mgr.14140 2021-02-10T04:44:35.875 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:35 smithi085 conmon[29319]: ) 945 : cluster [DBG] pgmap v1692: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 175 active+clean; 865 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 200 MiB/s rd, 63 MiB/s wr, 376 op/s; 60/2187 objects misplaced (2.743%); 19 MiB/s, 48 objects/s recovering 2021-02-10T04:44:35.875 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:35 smithi085 conmon[29319]: cluster 2021-02-10T04:44:34.585643+0000 mon.a (mon.0) 2027 : cluster [DBG] osdmap e1015: 8 total, 8 up, 7 in 2021-02-10T04:44:35.875 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:35 smithi085 conmon[29319]: cluster 2021-02-10T04:44:34.667841+0000 osd.5 (osd 2021-02-10T04:44:35.875 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:35 smithi085 conmon[29319]: .5) 91 : cluster [DBG] 6.7 scrub ok 2021-02-10T04:44:35.875 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:35 smithi085 conmon[35961]: cluster 2021-02-10T04:44:34.282064+0000 mgr.y 2021-02-10T04:44:35.876 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:35 smithi085 conmon[35961]: (mgr.14140) 945 : cluster [DBG] pgmap v1692: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 175 active+clean; 865 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 200 MiB/s rd, 63 MiB/s wr, 376 op/s; 60/2187 objects misplaced (2.743%); 19 MiB/s, 48 objects/s recovering 2021-02-10T04:44:35.876 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:35 smithi085 conmon[35961]: cluster 2021-02-10T04:44:34.585643+0000 mon.a ( 2021-02-10T04:44:35.876 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:35 smithi085 conmon[35961]: mon.0) 2027 : cluster [DBG] osdmap e1015: 8 total, 8 up, 7 in 2021-02-10T04:44:35.876 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:35 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:44:35.876 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:35 smithi085 conmon[35961]: 44:34.667841+0000 osd.5 (osd.5) 2021-02-10T04:44:35.876 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:35 smithi085 conmon[35961]: 91 : cluster [DBG] 6.7 scrub ok 2021-02-10T04:44:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:35 smithi164 conmon[29288]: cluster 2021-02-10T04:44:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:35 smithi164 conmon[29288]: 2021-02-10T04:44:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:35 smithi164 conmon[29288]: 2021 2021-02-10T04:44:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:35 smithi164 conmon[29288]: - 2021-02-10T04:44:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:35 smithi164 conmon[29288]: 02-10T04:44:34.282064 2021-02-10T04:44:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:35 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 945 : cluster [DBG] pgmap v1692: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 175 active+clean; 865 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 200 MiB/s rd, 63 MiB/s wr, 376 op/s; 60/2187 objects misplaced (2.743%); 19 MiB/s, 48 objects/s recovering 2021-02-10T04:44:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:35 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:44:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:35 smithi164 conmon[29288]: :44:34.585643+0000 mon.a (mon.0) 2027 : cluster 2021-02-10T04:44:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:35 smithi164 conmon[29288]: [DBG] osdmap e1015: 8 total, 8 up, 7 in 2021-02-10T04:44:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:35 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:44:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:35 smithi164 conmon[29288]: 44:34.667841+0000 osd.5 (osd.5) 91 2021-02-10T04:44:35.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:35 smithi164 conmon[29288]: : cluster [DBG] 6.7 scrub ok 2021-02-10T04:44:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:36 smithi164 conmon[29288]: cluster 2021-02-10T04:44:34.706677+0000 osd.6 (osd.6) 91 : cluster [DBG] 4.a scrub ok 2021-02-10T04:44:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:36 smithi164 conmon[29288]: 2021-02-10T04:44:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:36 smithi164 conmon[29288]: cluster 2021-02-10T04:44:35.590176+0000 mon.a (mon.0) 2028 : cluster [DBG] osdmap e1016: 8 total, 8 up, 7 in 2021-02-10T04:44:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:36 smithi164 conmon[29288]: cluster 2021-02-10T04:44:35.672894+0000 osd.0 (osd.0) 99 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:44:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:36 smithi164 conmon[29288]: cluster 2021-02-10T04:44:35.673928+0000 osd.0 2021-02-10T04:44:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:36 smithi164 conmon[29288]: (osd.0) 100 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:44:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:36 smithi164 conmon[29288]: cluster 2021-02-10T04:44:35.689196+0000 osd.6 (osd.6) 92 : cluster 2021-02-10T04:44:36.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:36 smithi164 conmon[29288]: [DBG] 1.6 scrub ok 2021-02-10T04:44:36.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:36 smithi085 conmon[35961]: cluster 2021-02-10T04:44:34 2021-02-10T04:44:36.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:36 smithi085 conmon[35961]: .706677+0000 osd.6 (osd.6) 91 : cluster [DBG] 4.a scrub ok 2021-02-10T04:44:36.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:36 smithi085 conmon[35961]: cluster 2021-02-10T04:44:35 2021-02-10T04:44:36.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:36 smithi085 conmon[35961]: .590176+0000 mon.a (mon.0) 2028 : cluster [DBG] 2021-02-10T04:44:36.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:36 smithi085 conmon[35961]: osdmap e1016: 8 total, 8 up, 7 in 2021-02-10T04:44:36.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:36 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:44:36.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:36 smithi085 conmon[35961]: -10T04:44:35.672894+0000 osd.0 2021-02-10T04:44:36.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:36 smithi085 conmon[35961]: (osd.0) 99 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:44:36.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:36 smithi085 conmon[35961]: cluster 2021-02-10T04:44:36.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:36 smithi085 conmon[35961]: 2021-02-10T04:44:35.673928+0000 2021-02-10T04:44:36.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:36 smithi085 conmon[35961]: osd.0 (osd.0) 100 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:44:36.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:36 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:44:36.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:36 smithi085 conmon[35961]: -02-10T04:44:35.689196+0000 osd.6 ( 2021-02-10T04:44:36.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:36 smithi085 conmon[35961]: osd.6) 92 : cluster [DBG] 1.6 scrub ok 2021-02-10T04:44:36.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:36 smithi085 conmon[29319]: cluster 2021-02-10T04:44:34.706677+0000 osd.6 ( 2021-02-10T04:44:36.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:36 smithi085 conmon[29319]: osd.6) 91 : cluster [DBG] 4.a scrub ok 2021-02-10T04:44:36.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:36 smithi085 conmon[29319]: cluster 2021-02-10T04:44:35.590176+0000 2021-02-10T04:44:36.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:36 smithi085 conmon[29319]: mon.a (mon.0) 2028 : cluster [DBG] osdmap e1016: 8 total, 8 up, 7 in 2021-02-10T04:44:36.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:36 smithi085 conmon[29319]: cluster 2021-02-10T04:44:35. 2021-02-10T04:44:36.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:36 smithi085 conmon[29319]: 672894+0000 osd.0 (osd.0) 99 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:44:36.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:36 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:44:36.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:36 smithi085 conmon[29319]: T04:44:35.673928+0000 osd.0 (osd.0) 100 : 2021-02-10T04:44:36.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:36 smithi085 conmon[29319]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:44:36.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:36 smithi085 conmon[29319]: cluster 2021-02-10T04:44:36.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:36 smithi085 conmon[29319]: 2021-02-10T04:44:35.689196+0000 osd.6 (osd.6) 92 : cluster [DBG] 2021-02-10T04:44:36.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:36 smithi085 conmon[29319]: 1.6 scrub ok 2021-02-10T04:44:37.886 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:44:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:37 smithi164 conmon[29288]: cluster 2021-02-10T04:44:36.282803+0000 mgr.y (mgr.14140) 946 : cluster [DBG] pgmap v1695: 178 pgs: 1 peering, 177 active+clean; 865 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 147 MiB/s rd, 44 MiB/s wr, 273 op/s; 15 MiB/s, 29 objects/s recovering 2021-02-10T04:44:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:37 smithi164 conmon[29288]: 2021-02-10T04:44:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:37 smithi164 conmon[29288]: cluster 2021-02-10T04:44:36.594537+0000 mon.a (mon.0) 2029 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2021-02-10T04:44:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:37 smithi164 conmon[29288]: cluster 2021-02-10T04:44:36.594589+0000 mon.a (mon.0) 2030 : cluster 2021-02-10T04:44:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:37 smithi164 conmon[29288]: [INF] Cluster is now healthy 2021-02-10T04:44:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:37 smithi164 conmon[29288]: cluster 2021-02-10T04:44:36.606480+0000 mon.a ( 2021-02-10T04:44:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:37 smithi164 conmon[29288]: mon.0) 2031 : cluster [DBG] osdmap e1017: 8 total, 8 up, 7 in 2021-02-10T04:44:37.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:37 smithi164 conmon[29288]: cluster 2021-02-10T04:44:36.641960+0000 2021-02-10T04:44:37.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:37 smithi164 conmon[29288]: osd.3 (osd.3) 95 : cluster [DBG] 2.1b scrub ok 2021-02-10T04:44:37.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:37 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:44:37.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:37 smithi164 conmon[29288]: 04:44:36.852510+0000 osd.0 (osd.0) 101 2021-02-10T04:44:37.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:37 smithi164 conmon[29288]: : cluster [DBG] 3.1 scrub ok 2021-02-10T04:44:37.896 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set noscrub 2021-02-10T04:44:37.918 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[35961]: cluster 2021-02-10T04:44:36.282803+0000 mgr.y (mgr 2021-02-10T04:44:37.918 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[35961]: .14140) 946 : cluster [DBG] pgmap v1695: 178 pgs: 1 peering, 177 active+clean; 865 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 147 MiB/s rd, 44 MiB/s wr, 273 op/s; 15 MiB/s, 29 objects/s recovering 2021-02-10T04:44:37.918 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[35961]: cluster 2021-02-10T04:44 2021-02-10T04:44:37.918 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[35961]: :36.594537+0000 mon.a (mon.0) 2029 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2021-02-10T04:44:37.919 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:44:37.919 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[35961]: T04:44:36.594589+0000 mon.a (mon 2021-02-10T04:44:37.919 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[35961]: .0) 2030 : cluster [INF] Cluster is now healthy 2021-02-10T04:44:37.919 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:44:37.919 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[35961]: -02-10T04:44:36.606480+0000 2021-02-10T04:44:37.919 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[35961]: mon.a (mon.0) 2031 : cluster [DBG] 2021-02-10T04:44:37.919 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[35961]: osdmap e1017: 8 total, 8 up, 7 in 2021-02-10T04:44:37.919 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:44:37.919 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[35961]: 02-10T04:44:36.641960+0000 osd.3 (osd.3) 95 : 2021-02-10T04:44:37.919 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[35961]: cluster [DBG] 2.1b scrub ok 2021-02-10T04:44:37.919 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[35961]: cluster 2021-02-10T04:44:37.919 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[35961]: 2021-02-10T04:44:36.852510+0000 osd.0 (osd.0) 101 : cluster [DBG] 2021-02-10T04:44:37.919 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[35961]: 3.1 scrub ok 2021-02-10T04:44:37.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[29319]: cluster 2021-02-10T04:44:36.282803+0000 mgr.y ( 2021-02-10T04:44:37.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[29319]: mgr.14140) 946 : cluster [DBG] pgmap v1695: 178 pgs: 1 peering, 177 active+clean; 865 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 147 MiB/s rd, 44 MiB/s wr, 273 op/s; 15 MiB/s, 29 objects/s recovering 2021-02-10T04:44:37.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[29319]: cluster 2021-02-10T04:44:36. 2021-02-10T04:44:37.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[29319]: 594537+0000 mon.a (mon.0) 2029 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2021-02-10T04:44:37.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:44:37.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[29319]: 04:44:36.594589+0000 mon.a (mon.0) 2021-02-10T04:44:37.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[29319]: 2030 : cluster [INF] Cluster is now healthy 2021-02-10T04:44:37.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:44:37.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[29319]: -02-10T04:44:36.606480+0000 mon.a (mon. 2021-02-10T04:44:37.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[29319]: 0) 2031 : cluster [DBG] osdmap e1017: 8 total, 8 up, 7 in 2021-02-10T04:44:37.920 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:44:37.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[29319]: -02-10T04:44:36.641960+0000 2021-02-10T04:44:37.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[29319]: osd.3 (osd.3) 95 : cluster [DBG] 2.1b scrub ok 2021-02-10T04:44:37.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[29319]: cluster 2021-02-10T04:44:37.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[29319]: 2021-02-10T04:44:37.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[29319]: 2021 2021-02-10T04:44:37.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[29319]: - 2021-02-10T04:44:37.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[29319]: 02 2021-02-10T04:44:37.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[29319]: - 2021-02-10T04:44:37.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[29319]: 10 2021-02-10T04:44:37.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[29319]: T 2021-02-10T04:44:37.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[29319]: 04 2021-02-10T04:44:37.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[29319]: : 2021-02-10T04:44:37.921 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[29319]: 44 2021-02-10T04:44:37.922 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[29319]: : 2021-02-10T04:44:37.922 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[29319]: 36 2021-02-10T04:44:37.922 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[29319]: . 2021-02-10T04:44:37.922 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[29319]: 852510 2021-02-10T04:44:37.922 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[29319]: +0000 osd.0 ( 2021-02-10T04:44:37.922 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:37 smithi085 conmon[29319]: osd.0) 101 : cluster [DBG] 3.1 scrub ok 2021-02-10T04:44:38.219 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:44:38.220 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:44:38.894 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:44:38 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:44:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:38 smithi164 conmon[29288]: cluster 2021-02-10T04:44:36.703966+0000 osd.6 (osd.6) 93 2021-02-10T04:44:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:38 smithi164 conmon[29288]: : cluster [DBG] 6.15 scrub ok 2021-02-10T04:44:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:38 smithi164 conmon[29288]: cluster 2021-02-10T04:44:37.827095+0000 osd.0 (osd.0) 102 : cluster [DBG] 5.1e scrub ok 2021-02-10T04:44:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:38 smithi164 conmon[29288]: audit 2021-02-10T04:44:38.162061+0000 mon.a (mon.0) 2032 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:44:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:38 smithi164 conmon[29288]: audit 2021-02-10T04:44:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:38 smithi164 conmon[29288]: 2021-02-10T04:44:38 2021-02-10T04:44:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:38 smithi164 conmon[29288]: .162320+0000 mon.a (mon.0) 2033 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:44:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:38 smithi164 conmon[29288]: cluster 2021-02-10T04:44:38.354185+0000 mon.a (mon.0) 2034 : cluster [DBG] osdmap e1018: 8 total, 8 up, 7 in 2021-02-10T04:44:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:38 smithi164 conmon[29288]: 2021-02-10T04:44:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:38 smithi085 conmon[29319]: cluster 2021-02-10T04:44:36.703966+0000 osd.6 2021-02-10T04:44:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:38 smithi085 conmon[29319]: (osd.6) 93 : cluster [DBG] 6.15 scrub ok 2021-02-10T04:44:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:38 smithi085 conmon[29319]: cluster 2021-02-10T04:44:37.827095+0000 osd.0 (osd.0) 102 : cluster [DBG] 5.1e scrub ok 2021-02-10T04:44:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:38 smithi085 conmon[29319]: audit 2021-02-10T04:44:38.162061+0000 mon.a (mon.0) 2032 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:44:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:38 smithi085 conmon[29319]: audit 2021-02-10T04:44:38.162320+0000 mon.a (mon.0) 2033 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:44:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:38 smithi085 conmon[29319]: cluster 2021-02-10T04:44:38.354185+0000 mon.a (mon.0) 2034 : cluster [DBG] osdmap e1018: 8 total, 8 up, 7 in 2021-02-10T04:44:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:38 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:44:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:38 smithi085 conmon[35961]: :44:36.703966+0000 osd.6 (osd.6) 93 : cluster [DBG] 6.15 scrub ok 2021-02-10T04:44:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:38 smithi085 conmon[35961]: cluster 2021-02-10T04:44:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:38 smithi085 conmon[35961]: 2021-02-10T04:44:37.827095+0000 osd.0 (osd.0) 2021-02-10T04:44:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:38 smithi085 conmon[35961]: 102 : cluster [DBG] 5.1e scrub ok 2021-02-10T04:44:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:38 smithi085 conmon[35961]: audit 2021 2021-02-10T04:44:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:38 smithi085 conmon[35961]: -02-10T04:44:38. 2021-02-10T04:44:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:38 smithi085 conmon[35961]: 162061+0000 mon.a (mon.0) 2032 : audit 2021-02-10T04:44:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:38 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:44:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:38 smithi085 conmon[35961]: audit 2021-02-10T04:44:38.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:38 smithi085 conmon[35961]: 2021-02-10T04:44:38. 2021-02-10T04:44:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:38 smithi085 conmon[35961]: 162320+0000 mon.a (mon.0) 2033 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:44:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:38 smithi085 conmon[35961]: cluster 2021-02-10T04:44:38.354185+0000 2021-02-10T04:44:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:38 smithi085 conmon[35961]: mon.a (mon.0) 2034 : cluster [DBG] osdmap e1018: 8 total, 8 up, 7 in 2021-02-10T04:44:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:38 smithi085 conmon[35961]: 2021-02-10T04:44:39.494 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:44:39.534 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:44:39.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:39 smithi085 conmon[35961]: cluster 2021-02-10T04:44:38. 2021-02-10T04:44:40.995 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:39 smithi085 conmon[35961]: 283815+0000 mgr.y (mgr.14140) 947 : cluster [DBG] pgmap v1697: 178 pgs: 1 peering, 177 active+clean; 865 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 121 MiB/s rd, 38 MiB/s wr, 233 op/s; 13 MiB/s, 24 objects/s recovering 2021-02-10T04:44:40.996 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:39 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:44:40.996 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:39 smithi085 conmon[35961]: -02-10T04:44:38.636550+0000 osd.5 (osd.5) 92 : cluster [DBG] 2021-02-10T04:44:40.996 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:39 smithi085 conmon[35961]: 2.18 scrub ok 2021-02-10T04:44:40.996 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:39 smithi085 conmon[35961]: audit 2021-02-10T04:44:40.996 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:39 smithi085 conmon[35961]: 2021-02-10T04:44:38.652014+0000 mgr.y 2021-02-10T04:44:40.996 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:39 smithi085 conmon[35961]: (mgr.14140) 948 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:44:40.997 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:39 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:44:40.997 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:39 smithi085 conmon[35961]: 10T04:44:38.877237+0000 osd.4 ( 2021-02-10T04:44:40.997 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:39 smithi085 conmon[35961]: osd.4) 1777 : cluster [DBG] 4.1a scrub ok 2021-02-10T04:44:40.997 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-02-10T04:44:40.997 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:44:40.998 INFO:tasks.thrashosds.thrasher:Growing pool zone.rgw.meta 2021-02-10T04:44:40.998 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:44:41.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:39 smithi164 conmon[29288]: cluster 2021-02-10T04:44:38 2021-02-10T04:44:41.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:39 smithi164 conmon[29288]: .283815+0000 mgr.y (mgr.14140) 947 : cluster [DBG] pgmap v1697: 178 pgs: 1 peering, 177 active+clean; 865 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 121 MiB/s rd, 38 MiB/s wr, 233 op/s; 13 MiB/s, 24 objects/s recovering 2021-02-10T04:44:41.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:39 smithi164 conmon[29288]: cluster 2021-02-10T04:44:38.636550+0000 osd.5 2021-02-10T04:44:41.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:39 smithi164 conmon[29288]: (osd.5) 92 : cluster [DBG] 2.18 scrub ok 2021-02-10T04:44:41.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:39 smithi164 conmon[29288]: audit 2021-02-10T04:44:38.652014 2021-02-10T04:44:41.002 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:39 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 948 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:44:41.002 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:39 smithi164 conmon[29288]: 2021-02-10T04:44:41.002 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:39 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:44:41.002 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:39 smithi164 conmon[29288]: 44:38.877237+0000 osd.4 (osd.4) 1777 : cluster 2021-02-10T04:44:41.002 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:39 smithi164 conmon[29288]: [DBG] 4.1a scrub ok 2021-02-10T04:44:41.002 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:40 smithi164 conmon[29288]: cluster 2021-02-10T04:44:41.003 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:40 smithi164 conmon[29288]: 2021-02-10T04:44:41.003 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:40 smithi164 conmon[29288]: 2021-02-10T04:44:38 2021-02-10T04:44:41.003 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:40 smithi164 conmon[29288]: .740152+0000 osd.6 (osd.6) 94 : cluster [DBG] 1.7 scrub ok 2021-02-10T04:44:41.003 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:40 smithi164 conmon[29288]: cluster 2021-02-10T04:44:41.003 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:40 smithi164 conmon[29288]: 2021-02-10T04:44:41.003 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:40 smithi164 conmon[29288]: 2021 2021-02-10T04:44:41.004 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:40 smithi164 conmon[29288]: - 2021-02-10T04:44:41.004 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:40 smithi164 conmon[29288]: 02 2021-02-10T04:44:41.004 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:40 smithi164 conmon[29288]: - 2021-02-10T04:44:41.004 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:40 smithi164 conmon[29288]: 10 2021-02-10T04:44:41.004 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:40 smithi164 conmon[29288]: T 2021-02-10T04:44:41.004 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:40 smithi164 conmon[29288]: 04 2021-02-10T04:44:41.004 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:40 smithi164 conmon[29288]: : 2021-02-10T04:44:41.005 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:40 smithi164 conmon[29288]: 44 2021-02-10T04:44:41.005 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:40 smithi164 conmon[29288]: : 2021-02-10T04:44:41.005 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:40 smithi164 conmon[29288]: 39 2021-02-10T04:44:41.005 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:40 smithi164 conmon[29288]: . 2021-02-10T04:44:41.005 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:40 smithi164 conmon[29288]: 599254 2021-02-10T04:44:41.005 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:40 smithi164 conmon[29288]: +0000 osd.3 (osd.3) 96 : 2021-02-10T04:44:41.006 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:40 smithi164 conmon[29288]: cluster [DBG] 5.17 scrub ok 2021-02-10T04:44:41.006 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:40 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:44:41.006 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:40 smithi164 conmon[29288]: -10T04:44:39.845404+0000 osd.0 (osd.0) 103 : cluster 2021-02-10T04:44:41.007 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:40 smithi164 conmon[29288]: [DBG] 2.1f scrub ok 2021-02-10T04:44:41.007 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:40 smithi164 conmon[29288]: cluster 2021-02-10T04:44:41.007 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:40 smithi164 conmon[29288]: 2021-02-10T04:44:39.917231+0000 osd.4 (osd 2021-02-10T04:44:41.007 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:40 smithi164 conmon[29288]: .4) 1778 : cluster [DBG] 2.25 scrub ok 2021-02-10T04:44:41.007 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:39 smithi085 conmon[29319]: cluster 2021-02-10T04:44:38.283815 2021-02-10T04:44:41.007 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:39 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 947 : cluster [DBG] pgmap v1697: 178 pgs: 1 peering, 177 active+clean; 865 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 121 MiB/s rd, 38 MiB/s wr, 233 op/s; 13 MiB/s, 24 objects/s recovering 2021-02-10T04:44:41.008 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:39 smithi085 conmon[29319]: cluster 2021-02-10T04:44:38.636550+0000 osd.5 (osd.5) 92 : cluster [DBG] 2.18 scrub ok 2021-02-10T04:44:41.008 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:39 smithi085 conmon[29319]: audit 2021-02-10T04:44:38.652014+0000 mgr.y (mgr.14140) 948 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:44:41.008 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:39 smithi085 conmon[29319]: cluster 2021-02-10T04:44:38.877237+0000 osd.4 (osd.4) 1777 : cluster [DBG] 4.1a scrub ok 2021-02-10T04:44:41.009 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:40 smithi085 conmon[29319]: cluster 2021-02-10T04:44:38.740152 2021-02-10T04:44:41.009 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:40 smithi085 conmon[29319]: +0000 osd.6 (osd.6) 94 : cluster [DBG] 1.7 scrub ok 2021-02-10T04:44:41.009 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:40 smithi085 conmon[29319]: cluster 2021-02-10T04:44:39.599254+0000 osd.3 (osd.3) 96 : cluster [DBG] 5.17 scrub ok 2021-02-10T04:44:41.009 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:40 smithi085 conmon[29319]: cluster 2021-02-10T04:44:41.010 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:40 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:44:41.010 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:40 smithi085 conmon[29319]: 44:39.845404+0000 osd.0 (osd.0) 103 : cluster [DBG] 2.1f scrub ok 2021-02-10T04:44:41.010 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:40 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:44:41.010 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:40 smithi085 conmon[29319]: -02-10T04:44:39.917231+0000 osd.4 ( 2021-02-10T04:44:41.010 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:40 smithi085 conmon[29319]: osd.4) 1778 : cluster [DBG] 2021-02-10T04:44:41.010 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:40 smithi085 conmon[29319]: 2.25 scrub ok 2021-02-10T04:44:41.011 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:44:41.011 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:40 smithi085 conmon[35961]: cluster 2021-02-10T04:44:38.740152+0000 2021-02-10T04:44:41.011 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:40 smithi085 conmon[35961]: osd.6 (osd.6) 94 : cluster [DBG] 1.7 scrub ok 2021-02-10T04:44:41.011 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:40 smithi085 conmon[35961]: cluster 2021-02-10T04:44:39.599254+0000 osd.3 (osd 2021-02-10T04:44:41.011 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:40 smithi085 conmon[35961]: .3) 96 : cluster [DBG] 5.17 scrub ok 2021-02-10T04:44:41.011 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:40 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:44:41.011 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:40 smithi085 conmon[35961]: -02-10T04:44:39.845404+0000 osd.0 (osd.0 2021-02-10T04:44:41.011 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:40 smithi085 conmon[35961]: ) 103 : cluster [DBG] 2.1f scrub ok 2021-02-10T04:44:41.011 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:40 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:44:41.011 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:40 smithi085 conmon[35961]: 10T04:44:39.917231+0000 osd.4 2021-02-10T04:44:41.012 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:40 smithi085 conmon[35961]: (osd.4) 1778 : cluster [DBG] 2.25 scrub ok 2021-02-10T04:44:41.012 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:40 smithi085 conmon[35961]: 2021-02-10T04:44:41.012 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:44:40 smithi085 conmon[45443]: debug 2021-02-10T04:44:40.681+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:44:41.285 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.5 dump_ops_in_flight 2021-02-10T04:44:47.279 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:44:47.280 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:41 smithi085 conmon[29319]: cluster 2021-02-10T04:44:40.284919+0000 2021-02-10T04:44:47.280 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:41 smithi085 conmon[29319]: mgr.y (mgr.14140) 949 : cluster [DBG] pgmap v1699: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 898 B/s rd, 0 op/s 2021-02-10T04:44:47.280 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:42 smithi085 conmon[29319]: cluster 2021-02-10T04:44:41.866126 2021-02-10T04:44:47.280 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:42 smithi085 conmon[29319]: +0000 osd.0 (osd.0) 104 : cluster [DBG] 2.13 scrub ok 2021-02-10T04:44:47.280 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:42 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:44:47.280 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:42 smithi085 conmon[29319]: :44:41.973891+0000 osd.4 (osd.4) 1779 : cluster [DBG] 4.9 scrub ok 2021-02-10T04:44:47.280 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:43 smithi085 conmon[29319]: cluster 2021-02-10T04:44:41.760049+0000 osd.6 2021-02-10T04:44:47.280 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:43 smithi085 conmon[29319]: (osd.6) 95 : cluster [DBG] 5.6 scrub ok 2021-02-10T04:44:47.280 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:43 smithi085 conmon[29319]: cluster 2021-02-10T04:44:42.285699+0000 mgr.y (mgr.14140) 950 : cluster [DBG] pgmap v1700: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 764 B/s rd, 0 op/s 2021-02-10T04:44:47.281 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:43 smithi085 conmon[29319]: cluster 2021-02-10T04:44:43.015711+0000 osd.4 (osd.4) 1780 : cluster [DBG] 2.d scrub ok 2021-02-10T04:44:47.281 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:43 smithi085 conmon[29319]: cluster 2021-02-10T04:44:43.358640+0000 mon.a 2021-02-10T04:44:47.281 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:43 smithi085 conmon[29319]: (mon.0) 2035 : cluster [DBG] osdmap e1019: 8 total, 8 up, 7 in 2021-02-10T04:44:47.281 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: cluster 2021-02-10T04:44:47.281 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: 2021-02-10T04:44:47.281 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: 2021 2021-02-10T04:44:47.281 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: - 2021-02-10T04:44:47.281 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: 02 2021-02-10T04:44:47.281 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: - 2021-02-10T04:44:47.281 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: 10 2021-02-10T04:44:47.281 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: T 2021-02-10T04:44:47.281 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: 04 2021-02-10T04:44:47.281 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: : 2021-02-10T04:44:47.282 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: 44 2021-02-10T04:44:47.282 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: : 2021-02-10T04:44:47.282 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: 43 2021-02-10T04:44:47.282 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: . 2021-02-10T04:44:47.282 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: 810419 2021-02-10T04:44:47.282 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: +0000 2021-02-10T04:44:47.282 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: 2021-02-10T04:44:47.282 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: osd.7 2021-02-10T04:44:47.282 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: ( 2021-02-10T04:44:47.282 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: osd 2021-02-10T04:44:47.282 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: . 2021-02-10T04:44:47.282 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: 7 2021-02-10T04:44:47.282 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: ) 2021-02-10T04:44:47.283 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: 70 2021-02-10T04:44:47.283 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: : 2021-02-10T04:44:47.283 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: cluster 2021-02-10T04:44:47.283 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: 2021-02-10T04:44:47.283 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: [DBG] 2021-02-10T04:44:47.283 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: purged_snaps scrub starts 2021-02-10T04:44:47.283 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: cluster 2021-02-10T04:44:43.812858 2021-02-10T04:44:47.283 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[29319]: +0000 osd.7 (osd.7) 71 : cluster [DBG] purged_snaps scrub found 4 strays 2021-02-10T04:44:47.283 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:45 smithi085 conmon[29319]: cluster 2021-02-10T04:44:47.283 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:45 smithi085 conmon[29319]: 2021-02-10T04:44:47.283 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:45 smithi085 conmon[29319]: 2021-02-10T04:44:44.286338+0000 2021-02-10T04:44:47.283 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:45 smithi085 conmon[29319]: mgr.y (mgr.14140) 951 : cluster [DBG] pgmap v1702: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:44:47.284 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:45 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:44:47.284 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:45 smithi085 conmon[29319]: 04:44:44.573813+0000 2021-02-10T04:44:47.284 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:45 smithi085 conmon[29319]: osd.3 (osd.3) 97 : cluster [DBG] 2021-02-10T04:44:47.284 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:45 smithi085 conmon[29319]: 4.11 scrub ok 2021-02-10T04:44:47.284 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:45 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:44:47.284 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:45 smithi085 conmon[29319]: T04:44:44.580167+0000 osd.5 (osd. 2021-02-10T04:44:47.284 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:45 smithi085 conmon[29319]: 5) 93 : cluster [DBG] 4.15 scrub ok 2021-02-10T04:44:47.284 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:45 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:44:47.284 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:45 smithi085 conmon[29319]: -10T04:44:44.633930+0000 mon.a (mon.0) 2036 : cluster [DBG] osdmap e1020: 8 total, 8 up, 7 in 2021-02-10T04:44:47.284 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:46 smithi085 conmon[29319]: cluster 2021-02-10T04:44:47.284 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:46 smithi085 conmon[29319]: 2021-02-10T04:44:47.284 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:46 smithi085 conmon[29319]: 2021 2021-02-10T04:44:47.284 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:46 smithi085 conmon[29319]: - 2021-02-10T04:44:47.284 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:46 smithi085 conmon[29319]: 02 2021-02-10T04:44:47.285 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:46 smithi085 conmon[29319]: - 2021-02-10T04:44:47.285 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:46 smithi085 conmon[29319]: 10 2021-02-10T04:44:47.285 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:46 smithi085 conmon[29319]: T 2021-02-10T04:44:47.285 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:46 smithi085 conmon[29319]: 04:44:45.524970 2021-02-10T04:44:47.285 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:46 smithi085 conmon[29319]: +0000 osd.3 (osd.3) 98 : cluster [DBG] 5.13 scrub ok 2021-02-10T04:44:47.285 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:46 smithi085 conmon[29319]: cluster 2021-02-10T04:44:47.285 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:46 smithi085 conmon[29319]: 2021-02-10T04:44 2021-02-10T04:44:47.285 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:46 smithi085 conmon[29319]: :45.576798+0000 osd.5 (osd.5) 2021-02-10T04:44:47.285 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:46 smithi085 conmon[29319]: 94 : cluster [DBG] 5.19 scrub ok 2021-02-10T04:44:47.285 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:46 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:44:47.285 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:46 smithi085 conmon[29319]: -10T04:44:45.683896+0000 osd.6 2021-02-10T04:44:47.285 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:46 smithi085 conmon[29319]: (osd.6) 96 : cluster [DBG] 6.5 deep-scrub ok 2021-02-10T04:44:47.285 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:46 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:44:47.286 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:46 smithi085 conmon[29319]: 02-10T04:44:45.876788+0000 osd.0 (osd. 2021-02-10T04:44:47.286 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:46 smithi085 conmon[29319]: 0) 105 : cluster [DBG] 1.2 scrub ok 2021-02-10T04:44:47.286 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:41 smithi085 conmon[35961]: cluster 2021-02-10T04:44:47.286 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:41 smithi085 conmon[35961]: 2021-02-10T04:44:47.286 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:41 smithi085 conmon[35961]: 2021 2021-02-10T04:44:47.286 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:41 smithi085 conmon[35961]: - 2021-02-10T04:44:47.286 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:41 smithi085 conmon[35961]: 02 2021-02-10T04:44:47.286 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:41 smithi085 conmon[35961]: - 2021-02-10T04:44:47.286 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:41 smithi085 conmon[35961]: 10 2021-02-10T04:44:47.286 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:41 smithi085 conmon[35961]: T 2021-02-10T04:44:47.286 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:41 smithi085 conmon[35961]: 04 2021-02-10T04:44:47.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:41 smithi085 conmon[35961]: : 2021-02-10T04:44:47.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:41 smithi085 conmon[35961]: 44 2021-02-10T04:44:47.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:41 smithi085 conmon[35961]: : 2021-02-10T04:44:47.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:41 smithi085 conmon[35961]: 40.284919+0000 mgr.y (mgr. 2021-02-10T04:44:47.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:41 smithi085 conmon[35961]: 14140) 949 : cluster [DBG] pgmap v1699: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 898 B/s rd, 0 op/s 2021-02-10T04:44:47.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:42 smithi085 conmon[35961]: cluster 2021-02-10T04:44:41.866126 2021-02-10T04:44:47.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:42 smithi085 conmon[35961]: +0000 osd.0 (osd.0) 104 : cluster [DBG] 2.13 scrub ok 2021-02-10T04:44:47.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:42 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:44:47.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:42 smithi085 conmon[35961]: 10T04:44:41.973891+0000 osd.4 (osd.4) 1779 : cluster [DBG] 4.9 scrub ok 2021-02-10T04:44:47.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:43 smithi085 conmon[35961]: cluster 2021-02-10T04:44:47.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:43 smithi085 conmon[35961]: 2021-02-10T04:44:41 2021-02-10T04:44:47.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:43 smithi085 conmon[35961]: .760049+0000 osd.6 (osd.6) 95 : cluster [DBG] 5.6 scrub ok 2021-02-10T04:44:47.287 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:43 smithi085 conmon[35961]: cluster 2021-02-10T04:44:42 2021-02-10T04:44:47.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:43 smithi085 conmon[35961]: .285699+0000 mgr.y (mgr.14140) 950 : cluster [DBG] pgmap v1700: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 764 B/s rd, 0 op/s 2021-02-10T04:44:47.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:43 smithi085 conmon[35961]: 2021-02-10T04:44:47.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:43 smithi085 conmon[35961]: cluster 2021-02-10T04:44:47.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:43 smithi085 conmon[35961]: 2021-02-10T04:44:43.015711+0000 osd.4 (osd. 2021-02-10T04:44:47.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:43 smithi085 conmon[35961]: 4) 1780 : cluster [DBG] 2.d scrub ok 2021-02-10T04:44:47.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:43 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:44:47.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:43 smithi085 conmon[35961]: 04:44:43.358640+0000 mon.a (mon.0) 2021-02-10T04:44:47.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:43 smithi085 conmon[35961]: 2035 : cluster [DBG] osdmap e1019: 8 total, 8 up, 7 in 2021-02-10T04:44:47.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[35961]: cluster 2021-02-10T04:44:47.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[35961]: 2021-02-10T04:44:47.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[35961]: 2021 2021-02-10T04:44:47.288 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[35961]: - 2021-02-10T04:44:47.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[35961]: 02 2021-02-10T04:44:47.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[35961]: - 2021-02-10T04:44:47.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[35961]: 10 2021-02-10T04:44:47.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[35961]: T 2021-02-10T04:44:47.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[35961]: 04 2021-02-10T04:44:47.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[35961]: : 2021-02-10T04:44:47.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[35961]: 44 2021-02-10T04:44:47.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[35961]: : 2021-02-10T04:44:47.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[35961]: 43.810419+0000 osd.7 2021-02-10T04:44:47.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[35961]: (osd.7) 70 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:44:47.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:44:47.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[35961]: 04:44:43.812858+0000 osd.7 (osd.7) 71 : 2021-02-10T04:44:47.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:44 smithi085 conmon[35961]: cluster [DBG] purged_snaps scrub found 4 strays 2021-02-10T04:44:47.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:45 smithi085 conmon[35961]: cluster 2021-02-10T04:44:47.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:45 smithi085 conmon[35961]: 2021-02-10T04:44:47.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:45 smithi085 conmon[35961]: 2021-02-10T04:44:44.286338 2021-02-10T04:44:47.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:45 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 951 : cluster [DBG] pgmap v1702: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:44:47.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:45 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:44:47.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:45 smithi085 conmon[35961]: :44:44.573813+0000 osd.3 (osd.3) 97 : cluster [DBG] 2021-02-10T04:44:47.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:45 smithi085 conmon[35961]: 4.11 scrub ok 2021-02-10T04:44:47.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:45 smithi085 conmon[35961]: cluster 2021-02-10T04:44:47.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:45 smithi085 conmon[35961]: 2021-02-10T04:44:44.580167+0000 osd.5 (osd.5) 93 : cluster [DBG] 2021-02-10T04:44:47.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:45 smithi085 conmon[35961]: 4.15 scrub ok 2021-02-10T04:44:47.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:45 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:44:47.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:45 smithi085 conmon[35961]: T04:44:44.633930+0000 mon.a ( 2021-02-10T04:44:47.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:45 smithi085 conmon[35961]: mon.0) 2036 : cluster [DBG] osdmap e1020: 8 total, 8 up, 7 in 2021-02-10T04:44:47.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:46 smithi085 conmon[35961]: cluster 2021-02-10T04:44:45.524970 2021-02-10T04:44:47.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:46 smithi085 conmon[35961]: +0000 osd.3 (osd.3) 98 : cluster [DBG] 5.13 scrub ok 2021-02-10T04:44:47.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:46 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:44:47.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:46 smithi085 conmon[35961]: 04:44:45.576798+0000 osd.5 (osd.5) 94 : cluster [DBG] 5.19 scrub ok 2021-02-10T04:44:47.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:46 smithi085 conmon[35961]: cluster 2021-02-10T04:44: 2021-02-10T04:44:47.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:46 smithi085 conmon[35961]: 45.683896+0000 osd.6 (osd.6) 96 : 2021-02-10T04:44:47.292 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:46 smithi085 conmon[35961]: cluster [DBG] 6.5 deep-scrub ok 2021-02-10T04:44:47.292 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:46 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:44:47.292 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:46 smithi085 conmon[35961]: 44:45.876788+0000 osd.0 (osd.0) 2021-02-10T04:44:47.292 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:46 smithi085 conmon[35961]: 105 : cluster [DBG] 1.2 scrub ok 2021-02-10T04:44:47.292 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:41 smithi164 conmon[29288]: cluster 2021-02-10T04:44:40.284919 2021-02-10T04:44:47.292 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:41 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 949 : cluster [DBG] pgmap v1699: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 898 B/s rd, 0 op/s 2021-02-10T04:44:47.292 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:42 smithi164 conmon[29288]: cluster 2021-02-10T04:44:41.866126+0000 2021-02-10T04:44:47.292 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:42 smithi164 conmon[29288]: osd.0 (osd.0) 104 : cluster [DBG] 2.13 scrub ok 2021-02-10T04:44:47.292 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:42 smithi164 conmon[29288]: cluster 2021-02-10T04:44 2021-02-10T04:44:47.292 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:42 smithi164 conmon[29288]: :41.973891+0000 osd.4 (osd.4) 1779 : cluster [DBG] 4.9 scrub ok 2021-02-10T04:44:47.293 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:43 smithi164 conmon[29288]: cluster 2021-02-10T04:44:41.760049+0000 2021-02-10T04:44:47.293 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:43 smithi164 conmon[29288]: osd.6 (osd.6) 95 : cluster [DBG] 5.6 scrub ok 2021-02-10T04:44:47.293 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:43 smithi164 conmon[29288]: cluster 2021-02-10T04:44:42.285699+0000 mgr.y (mgr.14140) 950 : cluster [DBG] 2021-02-10T04:44:47.293 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:43 smithi164 conmon[29288]: pgmap v1700: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 764 B/s rd, 0 op/s 2021-02-10T04:44:47.293 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:43 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:44:47.293 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:43 smithi164 conmon[29288]: T04:44:43.015711+0000 osd.4 (osd.4 2021-02-10T04:44:47.293 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:43 smithi164 conmon[29288]: ) 1780 : cluster [DBG] 2021-02-10T04:44:47.293 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:43 smithi164 conmon[29288]: 2.d scrub ok 2021-02-10T04:44:47.293 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:43 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:44:47.293 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:43 smithi164 conmon[29288]: 02-10T04:44:43.358640+0000 mon.a (mon.0) 2035 2021-02-10T04:44:47.293 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:43 smithi164 conmon[29288]: : cluster [DBG] osdmap e1019: 8 total, 8 up, 7 in 2021-02-10T04:44:47.293 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:44 smithi164 conmon[29288]: cluster 2021-02-10T04:44:47.294 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:44 smithi164 conmon[29288]: 2021-02-10T04:44:47.294 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:44 smithi164 conmon[29288]: 2021 2021-02-10T04:44:47.294 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:44 smithi164 conmon[29288]: -02-10T04 2021-02-10T04:44:47.294 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:44 smithi164 conmon[29288]: :44:43.810419+0000 osd.7 (osd.7) 70 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:44:47.294 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:44 smithi164 conmon[29288]: cluster 2021-02-10T04:44:47.294 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:44 smithi164 conmon[29288]: 2021-02-10T04:44:43.812858+0000 2021-02-10T04:44:47.294 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:44 smithi164 conmon[29288]: osd.7 (osd.7) 71 : cluster [DBG] purged_snaps scrub found 4 strays 2021-02-10T04:44:47.294 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:45 smithi164 conmon[29288]: cluster 2021-02-10T04:44:44.286338+0000 mgr.y 2021-02-10T04:44:47.294 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:45 smithi164 conmon[29288]: (mgr.14140) 951 : cluster [DBG] pgmap v1702: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:44:47.294 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:45 smithi164 conmon[29288]: cluster 2021-02-10T04:44:44.573813+0000 osd.3 (osd.3) 97 : cluster [DBG] 4.11 scrub ok 2021-02-10T04:44:47.294 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:45 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:44:47.294 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:45 smithi164 conmon[29288]: 44:44.580167+0000 osd.5 (osd.5) 93 2021-02-10T04:44:47.295 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:45 smithi164 conmon[29288]: : cluster [DBG] 4.15 scrub ok 2021-02-10T04:44:47.295 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:45 smithi164 conmon[29288]: cluster 2021-02-10T04:44:47.295 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:45 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:44:47.295 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:45 smithi164 conmon[29288]: :44:44.633930+0000 mon.a (mon. 2021-02-10T04:44:47.295 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:45 smithi164 conmon[29288]: 0) 2036 : cluster [DBG] osdmap e1020: 8 total, 8 up, 7 in 2021-02-10T04:44:47.295 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:46 smithi164 conmon[29288]: cluster 2021-02-10T04:44:45.524970+0000 osd.3 ( 2021-02-10T04:44:47.295 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:46 smithi164 conmon[29288]: osd.3) 98 : cluster [DBG] 5.13 scrub ok 2021-02-10T04:44:47.295 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:46 smithi164 conmon[29288]: cluster 2021-02-10T04:44:45.576798+0000 osd.5 (osd.5) 94 : cluster [DBG] 5.19 scrub ok 2021-02-10T04:44:47.295 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:46 smithi164 conmon[29288]: cluster 2021-02-10T04:44:45.683896 2021-02-10T04:44:47.295 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:46 smithi164 conmon[29288]: +0000 osd.6 (osd.6) 96 : cluster [DBG] 6.5 deep-scrub ok 2021-02-10T04:44:47.295 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:46 smithi164 conmon[29288]: cluster 2021-02-10T04:44:45.876788+0000 osd.0 2021-02-10T04:44:47.295 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:46 smithi164 conmon[29288]: (osd.0) 105 : cluster [DBG] 1.2 scrub ok 2021-02-10T04:44:47.634 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:44:47.680 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:47 smithi164 conmon[29288]: cluster 2021-02-10T04:44:46.287067+0000 mgr.y ( 2021-02-10T04:44:47.680 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:47 smithi164 conmon[29288]: mgr.14140) 952 : cluster [DBG] pgmap v1704: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:44:47.680 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:47 smithi164 conmon[29288]: cluster 2021-02-10T04:44:46.552375+0000 osd.3 (osd.3) 99 : cluster [DBG] 4.1c scrub ok 2021-02-10T04:44:47.684 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:44:47.707 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:44:47.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:47 smithi085 conmon[35961]: cluster 2021-02-10T04:44:47.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:47 smithi085 conmon[35961]: 2021- 2021-02-10T04:44:47.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:47 smithi085 conmon[35961]: 02-10T04:44:46.287067+0000 mgr.y (mgr.14140) 952 : cluster [DBG] pgmap v1704: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:44:47.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:47 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:44:47.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:47 smithi085 conmon[35961]: 10T04:44:46.552375+0000 osd.3 (osd.3 2021-02-10T04:44:47.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:47 smithi085 conmon[35961]: ) 99 : cluster [DBG] 4.1c scrub ok 2021-02-10T04:44:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:47 smithi085 conmon[29319]: cluster 2021-02-10T04:44:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:47 smithi085 conmon[29319]: 2021-02-10T04:44:46 2021-02-10T04:44:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:47 smithi085 conmon[29319]: .287067+0000 mgr.y (mgr.14140) 952 : cluster [DBG] pgmap v1704: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:44:47.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:47 smithi085 conmon[29319]: cluster 2021-02-10T04:44:47.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:47 smithi085 conmon[29319]: 2021-02 2021-02-10T04:44:47.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:47 smithi085 conmon[29319]: -10T04:44:46.552375+0000 osd.3 (osd.3) 99 : cluster [DBG] 2021-02-10T04:44:47.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:47 smithi085 conmon[29319]: 4.1c scrub ok 2021-02-10T04:44:48.143 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:44:47 smithi164 conmon[37263]: debug 2021-02-10T04:44:48.144 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:44:47 smithi164 conmon[37263]: 2021-02-10T04:44:47.783+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:44:48.345 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:44:48.451 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:44:48.627 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:44:49.145 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:44:48 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:44:49.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:49 smithi164 conmon[29288]: cluster 2021-02-10T04:44:47.730565+0000 osd.6 ( 2021-02-10T04:44:49.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:49 smithi164 conmon[29288]: osd.6) 97 : cluster [DBG] 5.11 scrub ok 2021-02-10T04:44:49.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:49 smithi164 conmon[29288]: cluster 2021-02-10T04:44:48.287716+0000 mgr.y (mgr.14140) 953 : cluster [DBG] 2021-02-10T04:44:49.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:49 smithi164 conmon[29288]: pgmap v1705: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:44:49.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:49 smithi164 conmon[29288]: cluster 2021-02-10T04:44:49.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:49 smithi164 conmon[29288]: 2021-02-10T04:44:48. 2021-02-10T04:44:49.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:49 smithi164 conmon[29288]: 360083+0000 mon.a (mon.0 2021-02-10T04:44:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:49 smithi164 conmon[29288]: ) 2037 : cluster [DBG] osdmap e1021: 8 total, 8 up, 7 in 2021-02-10T04:44:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:49 smithi164 conmon[29288]: 2021-02-10T04:44:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:49 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:44:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:49 smithi164 conmon[29288]: 10T04:44:48.662105+0000 mgr.y ( 2021-02-10T04:44:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:49 smithi164 conmon[29288]: mgr.14140) 954 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:44:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:49 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:44:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:49 smithi164 conmon[29288]: :44:48.841765+0000 osd.4 ( 2021-02-10T04:44:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:49 smithi164 conmon[29288]: osd.4) 1781 : cluster 2021-02-10T04:44:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:49 smithi164 conmon[29288]: [DBG] purged_snaps scrub starts 2021-02-10T04:44:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:49 smithi164 conmon[29288]: 2021-02-10T04:44:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:49 smithi164 conmon[29288]: cluster 2021-02-10T04:44:48.843164+0000 osd.4 ( 2021-02-10T04:44:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:49 smithi164 conmon[29288]: osd.4) 1782 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:44:49.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[29319]: cluster 2021-02-10T04:44:47.730565+0000 osd.6 (osd.6) 97 : cluster [DBG] 5.11 scrub ok 2021-02-10T04:44:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[29319]: cluster 2021-02-10T04:44:48.287716+0000 mgr.y (mgr.14140) 953 : cluster [DBG] pgmap v1705: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:44:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[29319]: cluster 2021-02-10T04:44:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[29319]: 2021-02-10T04:44:48.360083+0000 mon.a (mon.0) 2037 : cluster [DBG] osdmap e1021: 8 total, 8 up, 7 in 2021-02-10T04:44:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[29319]: 2021-02-10T04:44:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[29319]: audit 2021-02-10T04:44:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[29319]: 2021-02-10T04:44:48.662105+0000 mgr.y (mgr. 2021-02-10T04:44:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[29319]: 14140) 954 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:44:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:44:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[29319]: -10T04:44:48. 2021-02-10T04:44:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[29319]: 841765+0000 osd.4 (osd.4) 1781 : cluster 2021-02-10T04:44:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[29319]: [DBG] purged_snaps scrub starts 2021-02-10T04:44:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:44:49.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[29319]: -10T04:44:48.843164 2021-02-10T04:44:49.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1782 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:44:49.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[35961]: cluster 2021-02-10T04:44:49.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[35961]: 2021-02-10T04:44:47.730565 2021-02-10T04:44:49.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[35961]: +0000 osd.6 (osd.6) 97 : cluster [DBG] 5.11 scrub ok 2021-02-10T04:44:49.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[35961]: cluster 2021-02-10T04:44:48.287716+0000 2021-02-10T04:44:49.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[35961]: mgr.y (mgr.14140) 953 : cluster 2021-02-10T04:44:49.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[35961]: [DBG] pgmap v1705: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:44:49.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:44:49.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[35961]: 10T04:44:48.360083+0000 mon.a (mon 2021-02-10T04:44:49.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[35961]: .0) 2037 : cluster [DBG] osdmap e1021: 8 total, 8 up, 7 in 2021-02-10T04:44:49.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[35961]: 2021-02-10T04:44:49.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[35961]: audit 2021 2021-02-10T04:44:49.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[35961]: -02-10T04:44: 2021-02-10T04:44:49.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[35961]: 48.662105+0000 mgr.y (mgr. 2021-02-10T04:44:49.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[35961]: 14140) 954 : audit [DBG] 2021-02-10T04:44:49.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[35961]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:44:49.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:44:49.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[35961]: -10T04:44:48.841765+0000 2021-02-10T04:44:49.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[35961]: osd.4 (osd.4) 1781 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:44:49.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[35961]: 2021-02-10T04:44:49.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[35961]: cluster 2021-02-10T04:44:49.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[35961]: 2021-02-10T04:44:48.843164+0000 osd.4 ( 2021-02-10T04:44:49.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:49 smithi085 conmon[35961]: osd.4) 1782 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:44:50.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:50 smithi085 conmon[29319]: cluster 2021-02-10T04:44:49.648994+0000 mon.a ( 2021-02-10T04:44:50.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:50 smithi085 conmon[29319]: mon.0) 2038 : cluster [DBG] osdmap e1022: 8 total, 8 up, 7 in 2021-02-10T04:44:50.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:50 smithi085 conmon[35961]: cluster 2021-02-10T04:44:49.648994+0000 mon.a (mon.0) 2038 : cluster [DBG] osdmap e1022: 8 total, 8 up, 7 in 2021-02-10T04:44:51.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:50 smithi164 conmon[29288]: cluster 2021-02-10T04:44:51.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:50 smithi164 conmon[29288]: 2021-02-10T04:44:49 2021-02-10T04:44:51.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:50 smithi164 conmon[29288]: .648994+0000 mon.a (mon.0) 2038 : cluster [DBG] osdmap e1022: 8 total, 8 up, 7 in 2021-02-10T04:44:51.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:51 smithi085 conmon[29319]: cluster 2021-02-10T04:44:50.288342+0000 mgr.y 2021-02-10T04:44:51.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:51 smithi085 conmon[29319]: (mgr.14140) 955 : cluster [DBG] pgmap v1708: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:44:51.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:51 smithi085 conmon[29319]: cluster 2021-02-10T04:44:50.573045+0000 osd.5 (osd.5) 95 : cluster [DBG] 2.16 scrub ok 2021-02-10T04:44:51.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:51 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:44:51.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:51 smithi085 conmon[29319]: -02-10T04:44:50.832081+0000 osd.0 ( 2021-02-10T04:44:51.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:51 smithi085 conmon[29319]: osd.0) 106 : cluster [DBG] 4.16 scrub ok 2021-02-10T04:44:51.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:51 smithi085 conmon[35961]: cluster 2021-02-10T04:44:50.288342 2021-02-10T04:44:51.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:51 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 955 : cluster [DBG] pgmap v1708: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:44:51.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:51 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:44:51.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:51 smithi085 conmon[35961]: T04:44:50.573045+0000 osd.5 (osd. 2021-02-10T04:44:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:51 smithi085 conmon[35961]: 5) 95 : cluster [DBG] 2.16 scrub ok 2021-02-10T04:44:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:51 smithi085 conmon[35961]: cluster 2021-02-10T04:44:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:51 smithi085 conmon[35961]: 2021-02-10T04:44:50.832081+0000 2021-02-10T04:44:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:51 smithi085 conmon[35961]: osd.0 (osd.0) 106 : cluster [DBG] 2021-02-10T04:44:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:51 smithi085 conmon[35961]: 4.16 scrub ok 2021-02-10T04:44:52.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:51 smithi164 conmon[29288]: cluster 2021-02-10T04:44:50. 2021-02-10T04:44:52.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:51 smithi164 conmon[29288]: 288342+0000 mgr.y (mgr.14140) 955 : cluster [DBG] pgmap v1708: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:44:52.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:51 smithi164 conmon[29288]: cluster 2021-02-10T04:44:50 2021-02-10T04:44:52.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:51 smithi164 conmon[29288]: .573045+0000 osd.5 (osd.5) 95 : cluster [DBG] 2.16 scrub ok 2021-02-10T04:44:52.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:51 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:44:52.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:51 smithi164 conmon[29288]: 10T04:44:50.832081+0000 osd.0 (osd.0 2021-02-10T04:44:52.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:51 smithi164 conmon[29288]: ) 106 : cluster [DBG] 4.16 scrub ok 2021-02-10T04:44:52.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:52 smithi085 conmon[29319]: cluster 2021-02-10T04:44:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:52 smithi085 conmon[29319]: 2021-02-10T04:44:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:52 smithi085 conmon[29319]: 2021 2021-02-10T04:44:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:52 smithi085 conmon[29319]: -02-10T04:44: 2021-02-10T04:44:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:52 smithi085 conmon[29319]: 50.716545+0000 osd.6 (osd.6) 98 : cluster [DBG] 2021-02-10T04:44:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:52 smithi085 conmon[29319]: 5.1f scrub ok 2021-02-10T04:44:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:52 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:44:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:52 smithi085 conmon[29319]: 10T04:44:51.575456+0000 osd.5 (osd.5) 96 : cluster [DBG] 5.1b scrub ok 2021-02-10T04:44:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:52 smithi085 conmon[29319]: cluster 2021-02-10T04:44:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:52 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:44:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:52 smithi085 conmon[29319]: :44:51.620099+0000 osd.3 (osd.3) 100 : cluster [DBG] 2021-02-10T04:44:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:52 smithi085 conmon[29319]: 2.23 scrub ok 2021-02-10T04:44:52.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:52 smithi085 conmon[35961]: cluster 2021-02-10T04:44:50.716545+0000 2021-02-10T04:44:52.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:52 smithi085 conmon[35961]: osd.6 (osd.6) 98 : cluster [DBG] 5.1f scrub ok 2021-02-10T04:44:52.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:52 smithi085 conmon[35961]: cluster 2021-02-10T04:44:51.575456+0000 2021-02-10T04:44:52.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:52 smithi085 conmon[35961]: osd.5 (osd.5) 96 : cluster [DBG] 5.1b scrub ok 2021-02-10T04:44:52.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:52 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:44:52.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:52 smithi085 conmon[35961]: 04:44:51.620099+0000 osd.3 (osd.3) 100 : cluster [DBG] 2.23 scrub ok 2021-02-10T04:44:53.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:52 smithi164 conmon[29288]: cluster 2021-02-10T04:44:50.716545+0000 osd.6 (osd.6) 98 2021-02-10T04:44:53.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:52 smithi164 conmon[29288]: : cluster [DBG] 5.1f scrub ok 2021-02-10T04:44:53.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:52 smithi164 conmon[29288]: cluster 2021-02-10T04:44:51.575456+0000 osd.5 (osd.5) 96 : cluster [DBG] 5.1b scrub ok 2021-02-10T04:44:53.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:52 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:44:53.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:52 smithi164 conmon[29288]: -02-10T04:44:51.620099+0000 osd.3 (osd.3) 100 : cluster 2021-02-10T04:44:53.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:52 smithi164 conmon[29288]: [DBG] 2.23 scrub ok 2021-02-10T04:44:53.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:53 smithi085 conmon[29319]: cluster 2021-02-10T04:44:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:53 smithi085 conmon[29319]: 2021-02-10T04:44:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:53 smithi085 conmon[29319]: 2021 2021-02-10T04:44:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:53 smithi085 conmon[29319]: -02-10T04:44:52.288870 2021-02-10T04:44:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:53 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 956 : cluster [DBG] pgmap v1709: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 668 B/s rd, 0 op/s 2021-02-10T04:44:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:53 smithi085 conmon[29319]: cluster 2021-02-10T04:44:53.361846 2021-02-10T04:44:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:53 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2039 : cluster [DBG] osdmap e1023: 8 total, 8 up, 7 in 2021-02-10T04:44:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:53 smithi085 conmon[29319]: 2021-02-10T04:44:53.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:53 smithi085 conmon[35961]: cluster 2021-02-10T04:44:52.288870+0000 2021-02-10T04:44:53.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:53 smithi085 conmon[35961]: mgr.y (mgr.14140) 956 : cluster [DBG] pgmap v1709: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 668 B/s rd, 0 op/s 2021-02-10T04:44:53.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:53 smithi085 conmon[35961]: cluster 2021-02-10T04:44:53.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:53 smithi085 conmon[35961]: 2021-02-10T04:44:53.361846+0000 mon.a (mon.0 2021-02-10T04:44:53.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:53 smithi085 conmon[35961]: ) 2039 : cluster [DBG] osdmap e1023: 8 total, 8 up, 7 in 2021-02-10T04:44:54.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:53 smithi164 conmon[29288]: cluster 2021-02-10T04:44:52.288870+0000 2021-02-10T04:44:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:53 smithi164 conmon[29288]: mgr.y (mgr.14140) 956 : cluster [DBG] pgmap v1709: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 668 B/s rd, 0 op/s 2021-02-10T04:44:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:53 smithi164 conmon[29288]: cluster 2021-02-10T04:44:53.361846+0000 mon.a (mon.0) 2039 : cluster 2021-02-10T04:44:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:53 smithi164 conmon[29288]: [DBG] osdmap e1023: 8 total, 8 up, 7 in 2021-02-10T04:44:54.326 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:44:54 smithi085 conmon[45443]: debug 2021-02-10T04:44:54.204+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:44:54.354 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:44:54.478 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:44:54.509 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:44:54.672 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:54 smithi085 conmon[29319]: cluster 2021-02-10T04:44:52.779748+0000 osd.6 (osd 2021-02-10T04:44:54.672 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:54 smithi085 conmon[29319]: .6) 99 : cluster [DBG] 6.3 scrub ok 2021-02-10T04:44:54.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:44:54 smithi164 conmon[33677]: debug 2021-02-10T04:44:54.608+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:44:54.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:54 smithi164 conmon[29288]: cluster 2021-02-10T04:44:54.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:54 smithi164 conmon[29288]: 2021-02-10T04:44:54.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:54 smithi164 conmon[29288]: 2021-02-10T04:44: 2021-02-10T04:44:54.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:54 smithi164 conmon[29288]: 52.779748+0000 osd.6 (osd.6) 99 : cluster [DBG] 6.3 scrub ok 2021-02-10T04:44:54.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:54 smithi085 conmon[35961]: cluster 2021-02-10T04:44:52. 2021-02-10T04:44:54.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:54 smithi085 conmon[35961]: 779748+0000 osd.6 (osd.6) 99 : cluster [DBG] 6.3 scrub ok 2021-02-10T04:44:55.154 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:44:55.394 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:44:55 smithi164 conmon[33677]: debug 2021-02-10T04:44:55.082+0000 7fbec19a1700 -1 osd.4 1023 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:44:55.688 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[29319]: cluster 2021-02-10T04:44:55.688 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[29319]: 2021-02-10T04:44:55.689 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[29319]: 2021 2021-02-10T04:44:55.689 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[29319]: - 2021-02-10T04:44:55.689 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[29319]: 02 2021-02-10T04:44:55.689 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[29319]: - 2021-02-10T04:44:55.689 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[29319]: 10 2021-02-10T04:44:55.689 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[29319]: T 2021-02-10T04:44:55.689 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[29319]: 04 2021-02-10T04:44:55.689 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[29319]: :44:54.289636+0000 mgr.y (mgr. 2021-02-10T04:44:55.689 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[29319]: 14140) 957 : cluster [DBG] pgmap v1711: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2021-02-10T04:44:55.689 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[29319]: cluster 2021-02-10T04:44:54.669594+0000 osd.3 (osd.3) 101 : cluster 2021-02-10T04:44:55.689 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[29319]: [DBG] 5.10 scrub ok 2021-02-10T04:44:55.689 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[29319]: cluster 2021-02-10T04:44:55.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[29319]: 2021-02-10T04:44:55.082847+0000 osd.4 (osd.4) 1783 2021-02-10T04:44:55.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:44:55.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[29319]: cluster 2021-02-10T04:44:55.082879+0000 osd.4 (osd.4) 2021-02-10T04:44:55.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[29319]: 1784 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:44:55.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:44:55.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[29319]: 44:55.082896+0000 osd.4 (osd.4) 1785 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:44:55.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:44:55.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[29319]: 10T04:44:55.082932+0000 osd.4 (osd. 2021-02-10T04:44:55.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[29319]: 4) 1786 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:44:55.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:44:55.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[29319]: :44:55.375903+0000 osd.1 (osd.1 2021-02-10T04:44:55.690 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[29319]: ) 467 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:44:55.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:44:55.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[29319]: :44:55.379741+0000 osd.1 (osd.1) 468 2021-02-10T04:44:55.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[29319]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:44:55.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[35961]: cluster 2021-02-10T04:44:54.289636+0000 mgr.y (mgr. 2021-02-10T04:44:55.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[35961]: 14140) 957 : cluster [DBG] pgmap v1711: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2021-02-10T04:44:55.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[35961]: cluster 2021-02-10T04:44:54 2021-02-10T04:44:55.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[35961]: .669594+0000 osd.3 (osd.3) 101 : cluster [DBG] 5.10 scrub ok 2021-02-10T04:44:55.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:44:55.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[35961]: T 2021-02-10T04:44:55.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[35961]: 04:44:55.082847+0000 osd.4 ( 2021-02-10T04:44:55.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[35961]: osd.4) 1783 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:44:55.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:44:55.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[35961]: 10T04:44:55.082879+0000 osd.4 (osd.4) 1784 : 2021-02-10T04:44:55.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:44:55.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:44:55.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[35961]: 10T04:44:55.082896+0000 osd.4 (osd.4) 1785 : cluster 2021-02-10T04:44:55.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:44:55.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:44:55.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[35961]: :44:55.082932+0000 osd.4 (osd. 2021-02-10T04:44:55.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[35961]: 4) 1786 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:44:55.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:44:55.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[35961]: 10T04:44:55.375903+0000 osd.1 2021-02-10T04:44:55.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[35961]: (osd.1) 467 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:44:55.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[35961]: 2021-02-10T04:44:55.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:44:55.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[35961]: 02-10T04:44:55.379741+0000 osd.1 (osd. 2021-02-10T04:44:55.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:55 smithi085 conmon[35961]: 1) 468 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:44:55.921 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:44:55.960 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:44:56.114 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:55 smithi164 conmon[29288]: cluster 2021-02-10T04:44:56.114 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:55 smithi164 conmon[29288]: 2021-02- 2021-02-10T04:44:56.114 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:55 smithi164 conmon[29288]: 10T04:44:54.289636+0000 mgr.y (mgr.14140) 957 : cluster [DBG] pgmap v1711: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2021-02-10T04:44:56.115 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:55 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:44:56.115 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:55 smithi164 conmon[29288]: 02-10T04:44:54.669594+0000 osd.3 ( 2021-02-10T04:44:56.115 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:55 smithi164 conmon[29288]: osd.3) 101 : cluster [DBG] 5.10 scrub ok 2021-02-10T04:44:56.115 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:55 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:44:56.115 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:55 smithi164 conmon[29288]: -10T04:44:55.082847+0000 2021-02-10T04:44:56.115 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:55 smithi164 conmon[29288]: osd.4 (osd.4) 1783 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:44:56.115 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:55 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:44:56.115 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:55 smithi164 conmon[29288]: -10T04:44:55.082879+0000 osd.4 ( 2021-02-10T04:44:56.115 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:55 smithi164 conmon[29288]: osd.4) 1784 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:44:56.115 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:55 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:44:56.115 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:55 smithi164 conmon[29288]: 02-10T04:44:55.082896+0000 osd.4 2021-02-10T04:44:56.115 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:55 smithi164 conmon[29288]: (osd.4) 1785 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:44:56.115 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:55 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:44:56.115 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:55 smithi164 conmon[29288]: T04:44:55.082932+0000 osd.4 (osd.4) 1786 : 2021-02-10T04:44:56.116 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:55 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:44:56.116 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:55 smithi164 conmon[29288]: cluster 2021-02-10T04:44:56.116 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:55 smithi164 conmon[29288]: 2021-02-10T04:44:55.375903+0000 osd.1 (osd.1) 2021-02-10T04:44:56.116 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:55 smithi164 conmon[29288]: 467 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:44:56.116 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:55 smithi164 conmon[29288]: cluster 2021-02-10T04:44:56.116 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:55 smithi164 conmon[29288]: 2021-02-10T04:44:55.379741+0000 osd.1 (osd.1) 2021-02-10T04:44:56.116 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:55 smithi164 conmon[29288]: 468 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:44:56.240 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:44:56.366 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:44:56 smithi164 conmon[33677]: debug 2021-02-10T04:44:56.113+0000 7fbec19a1700 -1 osd.4 1024 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:44:56.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[49106]: debug 2021-02-10T04:44:56.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[49106]: 2021-02-10T04:44:56.044+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:44:56.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[49106]: 2021-02-10T04:44:56.940 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:44:58.590 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2021-02-10T04:44:58.590 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 0 2021-02-10T04:44:58.590 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:44:58.592 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[29319]: cluster 2021-02-10T04:44:55.676559+0000 mon.a (mon. 2021-02-10T04:44:58.592 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[29319]: 0) 2040 : cluster [DBG] osdmap e1024: 8 total, 8 up, 7 in 2021-02-10T04:44:58.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[29319]: cluster 2021-02-10T04:44:55.693379+0000 osd.5 (osd.5) 97 : cluster [DBG] 6.d scrub ok 2021-02-10T04:44:58.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[29319]: cluster 2021-02-10T04:44:55.738442+0000 2021-02-10T04:44:58.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[29319]: osd.0 (osd.0) 107 : cluster [DBG] 1.0 scrub ok 2021-02-10T04:44:58.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:44:58.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[29319]: 02-10T04:44:56.113922+0000 osd.4 (osd. 2021-02-10T04:44:58.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[29319]: 4) 1787 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:44:58.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[29319]: cluster 2021-02-10T04:44:56.113933+0000 2021-02-10T04:44:58.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[29319]: osd.4 (osd.4) 1788 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:44:58.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[29319]: 2021-02-10T04:44:58.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:44:58.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[29319]: -10T04:44:56.113940 2021-02-10T04:44:58.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1789 : cluster [WRN] 2021-02-10T04:44:58.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:44:58.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[29319]: cluster 2021-02-10T04:44 2021-02-10T04:44:58.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[29319]: :56.113947+0000 osd.4 (osd.4) 1790 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:44:58.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[29319]: cluster 2021-02-10T04:44:58.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[29319]: 2021-02-10T04:44:56.117131+0000 osd.4 (osd.4) 2021-02-10T04:44:58.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[29319]: 1791 : cluster [DBG] 4.c scrub ok 2021-02-10T04:44:58.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: cluster 2021-02-10T04:44:55.705286+0000 osd.6 (osd.6 2021-02-10T04:44:58.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: ) 100 : cluster [DBG] 4.12 scrub ok 2021-02-10T04:44:58.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: cluster 2021-02-10T04:44:56.290061+0000 mgr.y (mgr.14140) 958 : cluster [DBG] pgmap v1713: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 770 B/s rd, 0 op/s 2021-02-10T04:44:58.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: cluster 2021-02-10T04:44:56.759226+0000 osd.0 (osd.0 2021-02-10T04:44:58.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: ) 108 : cluster [DBG] 2.8 scrub ok 2021-02-10T04:44:58.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:44:58.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: 02-10T04: 2021-02-10T04:44:58.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: 44:57.121162+0000 osd.4 (osd.4) 1792 2021-02-10T04:44:58.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:44:58.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:44:58.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: 02-10T04:44:57.121176+0000 osd.4 2021-02-10T04:44:58.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: (osd.4) 1793 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:44:58.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: 2021-02-10T04:44:58.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:44:58.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: 02-10T04:44:57.121181+0000 2021-02-10T04:44:58.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: osd.4 (osd.4) 1794 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:44:58.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: 2021-02-10T04:44:58.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: cluster 2021-02-10T04:44: 2021-02-10T04:44:58.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: 57.121184+0000 osd.4 (osd.4) 1795 : 2021-02-10T04:44:58.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:44:58.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:44:58.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: T04: 2021-02-10T04:44:58.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: 44:57.308218+0000 mon.a (mon.0) 2021-02-10T04:44:58.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: 2041 : audit [DBG] from='client.? 172.21.15.85:0/4294924178' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:44:58.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:44:58.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: T04:44:57.636092+0000 mon.b (mon 2021-02-10T04:44:58.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: .2) 44 : audit [INF] from='client.? 172.21.15.85:0/3875478184' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:44:58.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: 2021-02-10T04:44:58.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: audit 2021-02-10T04: 2021-02-10T04:44:58.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: 44:57.636901+0000 mon.a (mon.0) 2021-02-10T04:44:58.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[29319]: 2042 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:44:58.596 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[35961]: cluster 2021-02-10T04:44:55 2021-02-10T04:44:58.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[35961]: .676559+0000 mon.a (mon.0) 2040 : cluster [DBG] osdmap e1024: 8 total, 8 up, 7 in 2021-02-10T04:44:58.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:44:58.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[35961]: 04:44:55.693379+0000 osd.5 (osd.5) 97 : cluster [DBG] 2021-02-10T04:44:58.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[35961]: 6.d scrub ok 2021-02-10T04:44:58.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:44:58.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[35961]: -02-10T04:44:55.738442+0000 osd.0 (osd.0) 107 : cluster [DBG] 1.0 scrub ok 2021-02-10T04:44:58.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[35961]: cluster 2021-02-10T04:44: 2021-02-10T04:44:58.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[35961]: 56.113922+0000 osd.4 (osd.4) 1787 : cluster 2021-02-10T04:44:58.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:44:58.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[35961]: cluster 2021-02-10T04:44:56.113933+0000 2021-02-10T04:44:58.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[35961]: osd.4 (osd.4) 1788 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:44:58.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:44:58.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[35961]: 04:44:56.113940+0000 osd.4 (osd.4) 1789 : cluster 2021-02-10T04:44:58.597 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:44:58.598 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[35961]: cluster 2021-02-10T04:44:56.113947+0000 2021-02-10T04:44:58.598 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[35961]: osd.4 (osd.4) 1790 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:44:58.598 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:44:58.598 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[35961]: :44:56.117131+0000 osd.4 (osd.4) 1791 : cluster [DBG] 2021-02-10T04:44:58.598 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:56 smithi085 conmon[35961]: 4.c scrub ok 2021-02-10T04:44:58.598 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[35961]: cluster 2021-02-10T04:44:55.705286+0000 osd.6 (osd.6) 100 2021-02-10T04:44:58.598 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[35961]: : cluster [DBG] 4.12 scrub ok 2021-02-10T04:44:58.598 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:44:58.598 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[35961]: 10T04:44:56.290061+0000 mgr.y (mgr.14140) 958 : cluster 2021-02-10T04:44:58.598 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[35961]: [DBG] pgmap v1713: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 770 B/s rd, 0 op/s 2021-02-10T04:44:58.598 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[35961]: cluster 2021-02-10T04:44:56.759226+0000 osd.0 (osd.0) 108 : cluster [DBG] 2.8 scrub ok 2021-02-10T04:44:58.598 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:44:58.599 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[35961]: 02-10T04:44:57.121162+0000 osd.4 (osd.4 2021-02-10T04:44:58.599 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[35961]: ) 1792 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:44:58.599 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:44:58.599 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[35961]: -10T04:44:57.121176+0000 osd.4 2021-02-10T04:44:58.600 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[35961]: (osd.4) 1793 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:44:58.600 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:44:58.600 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[35961]: 02-10T04:44:57.121181+0000 2021-02-10T04:44:58.600 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[35961]: osd.4 (osd.4) 1794 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:44:58.600 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[35961]: cluster 2021-02-10T04:44 2021-02-10T04:44:58.600 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[35961]: :57.121184+0000 osd.4 (osd.4) 1795 : 2021-02-10T04:44:58.600 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:44:58.600 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:44:58.600 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[35961]: T04:44:57.308218+0000 mon.a (mon 2021-02-10T04:44:58.600 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[35961]: .0) 2041 : audit [DBG] from='client.? 172.21.15.85:0/4294924178' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:44:58.601 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[35961]: audit 2021 2021-02-10T04:44:58.601 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[35961]: -02-10T04:44:57.636092+0000 mon.b (mon 2021-02-10T04:44:58.601 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[35961]: .2) 44 : audit [INF] from='client.? 172.21.15.85:0/3875478184' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:44:58.601 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[35961]: audit 2021-02-10T04:44:57.636901+0000 2021-02-10T04:44:58.601 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:57 smithi085 conmon[35961]: mon.a (mon.0) 2042 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:44:58.601 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:44:58.601 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:44:58.601 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:44:58.601 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:44:58.602 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:44:58.602 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:44:58.602 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:44:58.602 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:44:58.602 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:44:58.602 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:44:58.602 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:44:58.602 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:44:58.602 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:44:58.602 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:44:58.603 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:44:58.603 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:44:58.603 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is set 2021-02-10T04:44:58.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:56 smithi164 conmon[29288]: cluster 2021-02-10T04:44:55.676559+0000 mon.a (mon 2021-02-10T04:44:58.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:56 smithi164 conmon[29288]: .0) 2040 : cluster [DBG] osdmap e1024: 8 total, 8 up, 7 in 2021-02-10T04:44:58.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:56 smithi164 conmon[29288]: cluster 2021-02-10T04:44:55.693379+0000 osd.5 (osd.5) 97 : cluster [DBG] 6.d scrub ok 2021-02-10T04:44:58.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:56 smithi164 conmon[29288]: cluster 2021-02-10T04:44:55.738442+0000 osd.0 (osd.0) 107 : cluster [DBG] 1.0 scrub ok 2021-02-10T04:44:58.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:56 smithi164 conmon[29288]: cluster 2021-02-10T04:44:56.113922+0000 osd.4 (osd.4) 1787 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:44:58.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:56 smithi164 conmon[29288]: cluster 2021-02-10T04:44:56.113933+0000 osd.4 (osd.4) 1788 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:44:58.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:56 smithi164 conmon[29288]: cluster 2021-02-10T04:44:56.113940+0000 osd.4 (osd.4) 1789 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:44:58.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:56 smithi164 conmon[29288]: cluster 2021-02-10T04:44:56.113947+0000 osd.4 (osd.4) 1790 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:44:58.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:56 smithi164 conmon[29288]: cluster 2021-02-10T04:44:56.117131+0000 osd.4 (osd.4) 1791 : cluster [DBG] 4.c scrub ok 2021-02-10T04:44:58.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:57 smithi164 conmon[29288]: cluster 2021-02-10T04:44:55.705286+0000 osd.6 (osd. 2021-02-10T04:44:58.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:57 smithi164 conmon[29288]: 6) 100 : cluster [DBG] 4.12 scrub ok 2021-02-10T04:44:58.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:57 smithi164 conmon[29288]: cluster 2021-02-10T04:44:56.290061+0000 mgr.y (mgr.14140) 958 : cluster [DBG] pgmap v1713: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 770 B/s rd, 0 op/s 2021-02-10T04:44:58.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:57 smithi164 conmon[29288]: cluster 2021-02-10T04:44:56.759226+0000 osd.0 (osd.0 2021-02-10T04:44:58.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:57 smithi164 conmon[29288]: ) 108 : cluster [DBG] 2.8 scrub ok 2021-02-10T04:44:58.605 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:57 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:44:58.605 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:57 smithi164 conmon[29288]: 02-10T04:44:57.121162+0000 osd.4 2021-02-10T04:44:58.605 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:57 smithi164 conmon[29288]: (osd.4) 1792 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:44:58.605 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:57 smithi164 conmon[29288]: cluster 2021-02-10T04:44:58.605 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:57 smithi164 conmon[29288]: 2021-02-10T04:44:57.121176+0000 2021-02-10T04:44:58.605 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:57 smithi164 conmon[29288]: osd.4 (osd.4) 1793 : cluster [WRN] 2021-02-10T04:44:58.605 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:57 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:44:58.605 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:57 smithi164 conmon[29288]: cluster 2021-02-10T04:44:57.121181+0000 osd.4 (osd.4) 1794 : cluster 2021-02-10T04:44:58.605 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:57 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:44:58.605 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:57 smithi164 conmon[29288]: cluster 2021-02-10T04:44:57.121184+0000 osd.4 (osd.4) 1795 : cluster 2021-02-10T04:44:58.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:57 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:44:58.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:57 smithi164 conmon[29288]: audit 2021-02-10T04:44: 2021-02-10T04:44:58.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:57 smithi164 conmon[29288]: 57.308218+0000 mon.a (mon.0) 2041 2021-02-10T04:44:58.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:57 smithi164 conmon[29288]: : audit [DBG] from='client.? 172.21.15.85:0/4294924178' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:44:58.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:57 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:44:58.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:57 smithi164 conmon[29288]: 02-10T04:44:57.636092+0000 mon.b 2021-02-10T04:44:58.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:57 smithi164 conmon[29288]: (mon.2) 44 : audit [INF] from='client.? 172.21.15.85:0/3875478184' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:44:58.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:57 smithi164 conmon[29288]: 2021-02-10T04:44:58.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:57 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:44:58.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:57 smithi164 conmon[29288]: 10T04:44:57.636901+0000 mon.a (mon.0) 2042 : audit [INF] 2021-02-10T04:44:58.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:57 smithi164 conmon[29288]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:44:58.606 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:44:57 smithi164 conmon[33677]: debug 2021-02-10T04:44:58.607 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:44:57 smithi164 conmon[33677]: 2021-02-10T04:44:57.120+0000 7fbec19a1700 -1 osd.4 1024 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:44:59.144 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:44:58 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:44:59.348 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:58 smithi164 conmon[29288]: cluster 2021-02-10T04:44:56.738762+0000 osd.6 ( 2021-02-10T04:44:59.348 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:58 smithi164 conmon[29288]: osd.6) 101 : cluster [DBG] 4.1f scrub ok 2021-02-10T04:44:59.348 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:58 smithi164 conmon[29288]: cluster 2021-02-10T04:44:57.688919+0000 mon.a (mon.0) 2043 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:44:59.348 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:58 smithi164 conmon[29288]: audit 2021-02-10T04:44:57.699298+0000 2021-02-10T04:44:59.348 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:58 smithi164 conmon[29288]: mon.a (mon.0) 2044 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:44:59.348 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:58 smithi164 conmon[29288]: 2021-02-10T04:44:59.348 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:58 smithi164 conmon[29288]: cluster 2021-02-10T04:44:59.348 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:58 smithi164 conmon[29288]: 2021-02-10T04:44:57.699359+0000 mon.a (mon. 2021-02-10T04:44:59.349 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:58 smithi164 conmon[29288]: 0) 2045 : cluster [DBG] osdmap e1025: 8 total, 8 up, 7 in 2021-02-10T04:44:59.349 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:58 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:44:59.349 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:58 smithi164 conmon[29288]: T04:44:58.363032+0000 mon.a (mon. 2021-02-10T04:44:59.349 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:58 smithi164 conmon[29288]: 0) 2046 : cluster [DBG] osdmap e1026: 8 total, 8 up, 7 in 2021-02-10T04:44:59.349 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:58 smithi085 conmon[29319]: cluster 2021-02-10T04:44:56.738762+0000 osd.6 (osd.6) 101 : cluster [DBG] 4.1f scrub ok 2021-02-10T04:44:59.350 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:58 smithi085 conmon[29319]: cluster 2021-02-10T04:44:57.688919+0000 mon.a (mon.0) 2043 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:44:59.350 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:58 smithi085 conmon[29319]: audit 2021-02-10T04:44:57.699298+0000 mon.a (mon.0) 2044 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:44:59.350 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:58 smithi085 conmon[29319]: cluster 2021-02-10T04:44:57.699359+0000 mon.a (mon.0) 2045 : cluster [DBG] osdmap e1025: 8 total, 8 up, 7 in 2021-02-10T04:44:59.350 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:58 smithi085 conmon[29319]: cluster 2021-02-10T04:44:58.363032+0000 mon.a (mon.0) 2046 : cluster [DBG] osdmap e1026: 8 total, 8 up, 7 in 2021-02-10T04:44:59.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:58 smithi085 conmon[35961]: cluster 2021-02-10T04:44:56.738762 2021-02-10T04:44:59.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:58 smithi085 conmon[35961]: +0000 osd.6 (osd.6) 101 : cluster [DBG] 4.1f scrub ok 2021-02-10T04:44:59.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:58 smithi085 conmon[35961]: cluster 2021-02-10T04:44:57.688919+0000 mon.a (mon. 2021-02-10T04:44:59.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:58 smithi085 conmon[35961]: 0) 2043 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:44:59.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:58 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:44:59.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:58 smithi085 conmon[35961]: -10T04:44:57.699298+0000 mon.a (mon. 2021-02-10T04:44:59.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:58 smithi085 conmon[35961]: 0) 2044 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:44:59.351 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:58 smithi085 conmon[35961]: 2021-02-10T04:44:59.351 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:58 smithi085 conmon[35961]: cluster 2021-02-10T04:44:59.351 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:58 smithi085 conmon[35961]: 2021-02-10T04:44:57 2021-02-10T04:44:59.351 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:58 smithi085 conmon[35961]: .699359+0000 mon.a (mon.0) 2021-02-10T04:44:59.351 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:58 smithi085 conmon[35961]: 2045 : cluster [DBG] osdmap e1025: 8 total, 8 up, 7 in 2021-02-10T04:44:59.351 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:58 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:44:59.351 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:58 smithi085 conmon[35961]: 02-10T04:44:58.363032+0000 mon.a (mon 2021-02-10T04:44:59.351 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:58 smithi085 conmon[35961]: .0) 2046 : cluster [DBG] osdmap e1026: 8 total, 8 up, 7 in 2021-02-10T04:45:00.121 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:59 smithi085 conmon[29319]: cluster 2021-02-10T04:44:58 2021-02-10T04:45:00.121 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:59 smithi085 conmon[29319]: .290606+0000 mgr.y (mgr.14140) 959 : cluster [DBG] pgmap v1715: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:45:00.121 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:59 smithi085 conmon[29319]: audit 2021-02-10T04:44:58.673062+0000 mgr.y (mgr.14140) 960 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:45:00.121 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:59 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:45:00.121 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:44:59 smithi085 conmon[29319]: 02-10T04:44:58.695779+0000 mon.a (mon.0) 2047 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 32 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:45:00.121 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:59 smithi085 conmon[35961]: cluster 2021-02-10T04:44:58.290606+0000 mgr.y (mgr.14140 2021-02-10T04:45:00.121 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:59 smithi085 conmon[35961]: ) 959 : cluster [DBG] pgmap v1715: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:45:00.121 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:59 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:45:00.121 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:59 smithi085 conmon[35961]: :44:58.673062+0000 mgr.y (mgr.14140) 960 2021-02-10T04:45:00.122 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:59 smithi085 conmon[35961]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:45:00.122 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:59 smithi085 conmon[35961]: cluster 2021-02-10T04:44:58. 2021-02-10T04:45:00.122 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:59 smithi085 conmon[35961]: 695779+0000 mon.a (mon.0) 2047 : cluster [WRN] 2021-02-10T04:45:00.122 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:44:59 smithi085 conmon[35961]: Health check failed: 4 slow ops, oldest one blocked for 32 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:45:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:59 smithi164 conmon[29288]: cluster 2021-02-10T04:44:58.290606+0000 mgr.y ( 2021-02-10T04:45:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:59 smithi164 conmon[29288]: mgr.14140) 959 : cluster [DBG] pgmap v1715: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:45:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:59 smithi164 conmon[29288]: audit 2021-02-10T04:44:58.673062+0000 mgr.y (mgr.14140) 960 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:45:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:59 smithi164 conmon[29288]: 2021-02-10T04:45:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:59 smithi164 conmon[29288]: cluster 2021-02-10T04:44 2021-02-10T04:45:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:59 smithi164 conmon[29288]: :58.695779+0000 mon.a (mon.0) 2047 : cluster [WRN] 2021-02-10T04:45:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:44:59 smithi164 conmon[29288]: Health check failed: 4 slow ops, oldest one blocked for 32 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:45:00.223 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:45:00.260 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:45:01.299 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:01 smithi085 conmon[29319]: cephadm 2021-02-10T04:44:59.438850+0000 2021-02-10T04:45:01.299 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:01 smithi085 conmon[29319]: mgr.y (mgr.14140) 961 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:45:01.299 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:01 smithi085 conmon[29319]: cephadm 2021-02-10T04:44:59.439195+0000 mgr.y (mgr.14140) 962 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:45:01.299 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:01 smithi085 conmon[29319]: audit 2021-02-10T04:45:00.039130+0000 mon.a (mon.0) 2048 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:45:01.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:01 smithi085 conmon[35961]: cephadm 2021-02-10T04:44:59.438850+0000 mgr.y (mgr 2021-02-10T04:45:01.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:01 smithi085 conmon[35961]: .14140) 961 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:45:01.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:01 smithi085 conmon[35961]: cephadm 2021-02- 2021-02-10T04:45:01.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:01 smithi085 conmon[35961]: 10T04:44:59.439195+0000 mgr.y (mgr. 2021-02-10T04:45:01.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:01 smithi085 conmon[35961]: 14140) 962 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:45:01.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:01 smithi085 conmon[35961]: audit 2021 2021-02-10T04:45:01.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:01 smithi085 conmon[35961]: -02-10T04:45:00.039130+0000 2021-02-10T04:45:01.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:01 smithi085 conmon[35961]: mon.a (mon.0) 2048 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:45:01.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:01 smithi085 conmon[35961]: 2021-02-10T04:45:01.300 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:45:01 smithi085 conmon[45443]: debug 2021-02-10T04:45:01.296+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:45:01.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:01 smithi164 conmon[29288]: cephadm 2021-02-10T04:45:01.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:01 smithi164 conmon[29288]: 2021-02- 2021-02-10T04:45:01.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:01 smithi164 conmon[29288]: 10T04:44:59.438850+0000 mgr.y (mgr.14140) 961 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:45:01.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:01 smithi164 conmon[29288]: cephadm 2021- 2021-02-10T04:45:01.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:01 smithi164 conmon[29288]: 02-10T04:44:59.439195+0000 mgr.y ( 2021-02-10T04:45:01.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:01 smithi164 conmon[29288]: mgr.14140) 962 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:45:01.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:01 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:45:01.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:01 smithi164 conmon[29288]: 02-10T04:45:00.039130+0000 mon.a (mon 2021-02-10T04:45:01.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:01 smithi164 conmon[29288]: .0) 2048 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:45:01.571 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:45:02.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:02 smithi164 conmon[29288]: cluster 2021-02-10T04:45:00.291373+0000 2021-02-10T04:45:02.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:02 smithi164 conmon[29288]: mgr.y (mgr.14140) 963 : cluster [DBG] pgmap v1717: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:45:02.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:02 smithi085 conmon[29319]: cluster 2021-02-10T04:45:00.291373+0000 mgr.y (mgr.14140) 963 : cluster [DBG] pgmap v1717: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:45:02.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:02 smithi085 conmon[35961]: cluster 2021-02-10T04:45:00.291373+0000 2021-02-10T04:45:02.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:02 smithi085 conmon[35961]: mgr.y (mgr.14140) 963 : cluster [DBG] pgmap v1717: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:45:03.089 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.5 dump_blocked_ops 2021-02-10T04:45:03.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:03 smithi164 conmon[29288]: cluster 2021-02-10T04:45:03.044546+0000 2021-02-10T04:45:03.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:03 smithi164 conmon[29288]: mon.a (mon.0) 2049 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 32 sec, osd.4 has slow ops) 2021-02-10T04:45:03.408 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:45:03.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:03 smithi085 conmon[29319]: cluster 2021-02-10T04:45:03.044546+0000 mon.a ( 2021-02-10T04:45:03.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:03 smithi085 conmon[29319]: mon.0) 2049 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 32 sec, osd.4 has slow ops) 2021-02-10T04:45:03.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:03 smithi085 conmon[35961]: cluster 2021-02-10T04:45:03.044546+0000 2021-02-10T04:45:03.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:03 smithi085 conmon[35961]: mon.a (mon.0) 2049 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 32 sec, osd.4 has slow ops) 2021-02-10T04:45:04.100 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:04 smithi164 conmon[29288]: cluster 2021-02-10T04:45:04.100 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:04 smithi164 conmon[29288]: 2021-02 2021-02-10T04:45:04.100 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:04 smithi164 conmon[29288]: -10T04:45:02.291824+0000 mgr.y (mgr.14140) 964 : cluster [DBG] pgmap v1718: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 773 B/s rd, 0 op/s 2021-02-10T04:45:04.100 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:04 smithi164 conmon[29288]: 2021-02-10T04:45:04.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:04 smithi085 conmon[35961]: cluster 2021-02-10T04:45:02 2021-02-10T04:45:04.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:04 smithi085 conmon[35961]: .291824+0000 mgr.y (mgr.14140) 964 : cluster [DBG] pgmap v1718: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 773 B/s rd, 0 op/s 2021-02-10T04:45:04.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:04 smithi085 conmon[29319]: cluster 2021-02-10T04:45:02.291824+0000 2021-02-10T04:45:04.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:04 smithi085 conmon[29319]: mgr.y (mgr.14140) 964 : cluster [DBG] pgmap v1718: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 773 B/s rd, 0 op/s 2021-02-10T04:45:05.334 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:45:05.366 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:45:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:06 smithi164 conmon[29288]: cluster 2021-02-10T04:45:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:06 smithi164 conmon[29288]: 2021-02-10T04:45:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:06 smithi164 conmon[29288]: 2021-02-10T04:45:04 2021-02-10T04:45:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:06 smithi164 conmon[29288]: .292368+0000 mgr.y (mgr.14140) 965 : cluster [DBG] pgmap v1719: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:45:06.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:06 smithi085 conmon[35961]: cluster 2021-02-10T04:45:04.292368+0000 mgr.y (mgr.14140 2021-02-10T04:45:06.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:06 smithi085 conmon[35961]: ) 965 : cluster [DBG] pgmap v1719: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:45:06.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:06 smithi085 conmon[29319]: cluster 2021-02-10T04:45:04.292368+0000 mgr.y ( 2021-02-10T04:45:06.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:06 smithi085 conmon[29319]: mgr.14140) 965 : cluster [DBG] pgmap v1719: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:45:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:08 smithi164 conmon[29288]: cluster 2021-02-10T04:45:06.293106+0000 mgr.y ( 2021-02-10T04:45:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:08 smithi164 conmon[29288]: mgr.14140) 966 : cluster [DBG] pgmap v1720: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:45:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:08 smithi164 conmon[29288]: cluster 2021-02-10T04:45:07.258243+0000 osd.5 (osd.5) 98 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:45:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:08 smithi164 conmon[29288]: cluster 2021-02-10T04:45:07.261477+0000 osd.5 (osd.5) 2021-02-10T04:45:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:08 smithi164 conmon[29288]: 99 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:45:08.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:08 smithi085 conmon[29319]: cluster 2021-02-10T04:45:06.293106+0000 mgr.y (mgr.14140) 966 : cluster [DBG] pgmap v1720: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:45:08.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:08 smithi085 conmon[29319]: cluster 2021-02-10T04:45:07.258243+0000 osd.5 (osd.5) 98 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:45:08.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:08 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:45:08.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:08 smithi085 conmon[29319]: :45:07.261477+0000 osd.5 (osd.5) 99 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:45:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:08 smithi085 conmon[35961]: cluster 2021-02-10T04:45:06.293106+0000 2021-02-10T04:45:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:08 smithi085 conmon[35961]: mgr.y (mgr.14140) 966 : cluster [DBG] pgmap v1720: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:45:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:08 smithi085 conmon[35961]: cluster 2021-02-10T04:45:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:08 smithi085 conmon[35961]: 2021-02-10T04:45:07.258243+0000 osd.5 (osd.5) 2021-02-10T04:45:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:08 smithi085 conmon[35961]: 98 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:45:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:08 smithi085 conmon[35961]: cluster 2021-02-10T04:45:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:08 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:45:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:08 smithi085 conmon[35961]: 45:07.261477+0000 osd.5 ( 2021-02-10T04:45:08.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:08 smithi085 conmon[35961]: osd.5) 99 : cluster [DBG] 2021-02-10T04:45:08.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:08 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:45:08.566 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:45:08.672 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:45:08.702 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:45:08.869 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:45:08 smithi085 conmon[52836]: debug 2021-02-10T04:45:08.532+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:45:09.116 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:45:08 smithi164 conmon[37263]: debug 2021-02-10T04:45:08.787+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:45:09.117 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:45:08 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:45:09.118 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:09 smithi164 conmon[29288]: cluster 2021-02-10T04:45:08.106436+0000 2021-02-10T04:45:09.118 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:09 smithi164 conmon[29288]: mon.a (mon.0) 2050 : cluster [DBG] osdmap e1027: 8 total, 8 up, 7 in 2021-02-10T04:45:09.118 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:09 smithi164 conmon[29288]: cluster 2021-02-10T04:45:08.367070+0000 mon.a (mon.0) 2051 : cluster 2021-02-10T04:45:09.118 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:09 smithi164 conmon[29288]: [DBG] osdmap e1028: 8 total, 8 up, 7 in 2021-02-10T04:45:09.198 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:09 smithi085 conmon[29319]: cluster 2021-02-10T04:45:09.198 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:09 smithi085 conmon[29319]: 2021-02-10T04:45:09.198 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:09 smithi085 conmon[29319]: 2021 2021-02-10T04:45:09.198 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:09 smithi085 conmon[29319]: - 2021-02-10T04:45:09.198 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:09 smithi085 conmon[29319]: 02-10T04:45 2021-02-10T04:45:09.199 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:09 smithi085 conmon[29319]: :08.106436+0000 mon.a (mon.0) 2050 : cluster [DBG] osdmap e1027: 8 total, 8 up, 7 in 2021-02-10T04:45:09.199 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:09 smithi085 conmon[29319]: cluster 2021-02-10T04:45:09.199 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:09 smithi085 conmon[29319]: 2021-02 2021-02-10T04:45:09.199 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:09 smithi085 conmon[29319]: -10T04:45:08.367070+0000 mon.a (mon.0 2021-02-10T04:45:09.199 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:09 smithi085 conmon[29319]: ) 2051 : cluster [DBG] 2021-02-10T04:45:09.199 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:09 smithi085 conmon[29319]: osdmap e1028: 8 total, 8 up, 7 in 2021-02-10T04:45:09.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:09 smithi085 conmon[35961]: cluster 2021-02-10T04:45:08.106436 2021-02-10T04:45:09.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:09 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2050 : cluster [DBG] osdmap e1027: 8 total, 8 up, 7 in 2021-02-10T04:45:09.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:09 smithi085 conmon[35961]: cluster 2021-02-10T04:45:08.367070+0000 mon.a (mon.0) 2051 : cluster [DBG] osdmap e1028: 8 total, 8 up, 7 in 2021-02-10T04:45:09.269 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:45:09.800 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:45:09.805 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:45:10.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:10 smithi164 conmon[29288]: cluster 2021-02-10T04:45:08.293573+0000 mgr.y (mgr.14140) 967 : cluster [DBG] pgmap v1722: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:45:10.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:10 smithi164 conmon[29288]: audit 2021-02-10T04:45:08.683067+0000 mgr.y (mgr.14140) 968 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:45:10.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:10 smithi085 conmon[35961]: cluster 2021-02-10T04:45:08.293573+0000 mgr.y (mgr 2021-02-10T04:45:10.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:10 smithi085 conmon[35961]: .14140) 967 : cluster [DBG] pgmap v1722: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:45:10.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:10 smithi085 conmon[35961]: audit 2021-02-10T04:45:08.683067+0000 mgr.y (mgr.14140) 968 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:45:10.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:10 smithi085 conmon[29319]: cluster 2021-02-10T04:45:10.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:10 smithi085 conmon[29319]: 2021-02-10T04:45:08.293573+0000 mgr.y (mgr.14140) 967 : cluster [DBG] pgmap v1722: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:45:10.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:10 smithi085 conmon[29319]: audit 2021-02-10T04:45:08.683067+0000 mgr.y (mgr.14140) 968 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:45:10.574 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:45:10.649 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set nodeep-scrub 2021-02-10T04:45:10.895 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:45:10.985 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:45:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:11 smithi164 conmon[29288]: audit 2021-02-10T04:45:09.770775+0000 mgr.y (mgr.14140) 969 : audit [DBG] 2021-02-10T04:45:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:11 smithi164 conmon[29288]: from='client.25318 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:45:11.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:11 smithi085 conmon[29319]: audit 2021-02-10T04:45:09.770775 2021-02-10T04:45:11.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:11 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 969 : audit [DBG] from='client.25318 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:45:11.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:11 smithi085 conmon[35961]: audit 2021-02-10T04:45:11.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:11 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:45:11.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:11 smithi085 conmon[35961]: 45:09.770775+0000 mgr.y (mgr.14140) 969 : audit [DBG] from='client.25318 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:45:11.549 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:45:11.579 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:45:12.369 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:45:12.374 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:12 smithi164 conmon[33677]: debug 2021-02-10T04:45:12.374 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:12 smithi164 conmon[33677]: 2021-02-10T04:45:12.109+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:45:12.374 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:12 smithi164 conmon[29288]: cluster 2021-02-10T04:45:10.294252+0000 2021-02-10T04:45:12.374 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:12 smithi164 conmon[29288]: mgr.y (mgr.14140) 970 : cluster [DBG] pgmap v1724: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:45:12.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:12 smithi085 conmon[35961]: cluster 2021-02-10T04:45:10.294252 2021-02-10T04:45:12.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:12 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 970 : cluster [DBG] pgmap v1724: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:45:12.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:12 smithi085 conmon[29319]: cluster 2021-02-10T04:45:10.294252+0000 mgr.y (mgr. 2021-02-10T04:45:12.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:12 smithi085 conmon[29319]: 14140) 970 : cluster [DBG] pgmap v1724: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:45:12.476 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:45:12.511 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:45:13.255 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:45:13.261 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:13 smithi164 conmon[33677]: debug 2021-02-10T04:45:13.261 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:13 smithi164 conmon[33677]: 2021-02-10T04:45:13.008+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:45:13.261 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:13 smithi164 conmon[33677]: 2021-02-10T04:45:13.363 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:45:13.389 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:45:14.022 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:45:14.063 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:45:13 smithi164 conmon[44544]: debug 2021-02-10T04:45:13.866+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:45:14.127 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:45:17.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:14 smithi164 conmon[29288]: cluster 2021-02-10T04:45:17.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:14 smithi164 conmon[29288]: 2021-02-10T04:45:17.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:14 smithi164 conmon[29288]: 2021 2021-02-10T04:45:17.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:14 smithi164 conmon[29288]: - 2021-02-10T04:45:17.588 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:14 smithi164 conmon[29288]: 02 2021-02-10T04:45:17.589 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:14 smithi164 conmon[29288]: - 2021-02-10T04:45:17.589 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:14 smithi164 conmon[29288]: 10 2021-02-10T04:45:17.589 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:14 smithi164 conmon[29288]: T 2021-02-10T04:45:17.589 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:14 smithi164 conmon[29288]: 04 2021-02-10T04:45:17.589 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:14 smithi164 conmon[29288]: : 2021-02-10T04:45:17.589 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:14 smithi164 conmon[29288]: 45 2021-02-10T04:45:17.589 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:14 smithi164 conmon[29288]: : 2021-02-10T04:45:17.589 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:14 smithi164 conmon[29288]: 12 2021-02-10T04:45:17.589 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:14 smithi164 conmon[29288]: .294830+0000 mgr.y (mgr.14140) 2021-02-10T04:45:17.589 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:14 smithi164 conmon[29288]: 971 : cluster [DBG] pgmap v1725: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:45:17.589 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:15 smithi164 conmon[29288]: cluster 2021-02-10T04:45:13.989400+0000 osd.2 (osd.2) 1947 : cluster [DBG] 2021-02-10T04:45:17.589 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:15 smithi164 conmon[29288]: purged_snaps scrub starts 2021-02-10T04:45:17.590 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:15 smithi164 conmon[29288]: cluster 2021-02-10T04:45:13.990116+0000 osd.2 (osd.2) 1948 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:45:17.590 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:15 smithi164 conmon[29288]: cluster 2021-02-10T04:45:14.295438+0000 mgr.y (mgr.14140) 972 : cluster [DBG] pgmap v1726: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:45:17.590 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:15 smithi164 conmon[29288]: cluster 2021-02-10T04:45:14.380007+0000 mon.a (mon.0) 2052 : cluster [DBG] osdmap e1029: 8 total, 8 up, 7 in 2021-02-10T04:45:17.590 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:45:17.590 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:45:17.591 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":1723,"stamp":"2021-02-10T04:45:08.364930+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":906749317,"num_objects":729,"num_object_clones":322,"num_object_copies":2187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":729,"num_whiteouts":11,"num_read":9800,"num_read_kb":7588531,"num_write":6489,"num_write_kb":2093711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":619,"num_bytes_recovered":202285507,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24768,"ondisk_log_size":24768,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":534,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2202076,"kb_used_data":1856548,"kb_used_omap":0,"kb_used_meta":345472,"kb_avail":654013988,"statfs":{"total":671965249536,"available":669710323712,"internally_reserved":0,"allocated":1901105152,"data_stored":2525121674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":353763328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4],"upper_bound":131072},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.074149"},"pg_stats":[{"pgid":"5.9","version":"131'3","reported_seq":"1113","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.363527+0000","last_change":"2021-02-10T04:33:57.871495+0000","last_active":"2021-02-10T04:44:58.363527+0000","last_peered":"2021-02-10T04:44:58.363527+0000","last_clean":"2021-02-10T04:44:58.363527+0000","last_became_active":"2021-02-10T04:29:07.145525+0000","last_became_peered":"2021-02-10T04:29:07.145525+0000","last_unstale":"2021-02-10T04:44:58.363527+0000","last_undegraded":"2021-02-10T04:44:58.363527+0000","last_fullsized":"2021-02-10T04:44:58.363527+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'3","last_scrub_stamp":"2021-02-10T04:33:57.871402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:33:57.871402+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"538'42","reported_seq":"1074","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.362775+0000","last_change":"2021-02-10T04:33:55.585872+0000","last_active":"2021-02-10T04:44:58.362775+0000","last_peered":"2021-02-10T04:44:58.362775+0000","last_clean":"2021-02-10T04:44:58.362775+0000","last_became_active":"2021-02-10T04:25:49.002058+0000","last_became_peered":"2021-02-10T04:25:49.002058+0000","last_unstale":"2021-02-10T04:44:58.362775+0000","last_undegraded":"2021-02-10T04:44:58.362775+0000","last_fullsized":"2021-02-10T04:44:58.362775+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2021-02-10T04:33:55.585807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:55.585807+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":77,"num_read_kb":60,"num_write":43,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"426","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.701115+0000","last_change":"2021-02-10T04:37:37.973234+0000","last_active":"2021-02-10T04:44:57.701115+0000","last_peered":"2021-02-10T04:44:57.701115+0000","last_clean":"2021-02-10T04:44:57.701115+0000","last_became_active":"2021-02-10T04:37:37.246903+0000","last_became_peered":"2021-02-10T04:37:37.246903+0000","last_unstale":"2021-02-10T04:44:57.701115+0000","last_undegraded":"2021-02-10T04:44:57.701115+0000","last_fullsized":"2021-02-10T04:44:57.701115+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:29:34.787265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:34.787265+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"986","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.363272+0000","last_change":"2021-02-10T04:29:19.405593+0000","last_active":"2021-02-10T04:44:58.363272+0000","last_peered":"2021-02-10T04:44:58.363272+0000","last_clean":"2021-02-10T04:44:58.363272+0000","last_became_active":"2021-02-10T04:25:53.148090+0000","last_became_peered":"2021-02-10T04:25:53.148090+0000","last_unstale":"2021-02-10T04:44:58.363272+0000","last_undegraded":"2021-02-10T04:44:58.363272+0000","last_fullsized":"2021-02-10T04:44:58.363272+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:19.405525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:29:19.405525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"1003'1689","reported_seq":"4559","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.363907+0000","last_change":"2021-02-10T04:44:03.811251+0000","last_active":"2021-02-10T04:44:58.363907+0000","last_peered":"2021-02-10T04:44:58.363907+0000","last_clean":"2021-02-10T04:44:58.363907+0000","last_became_active":"2021-02-10T04:33:05.144488+0000","last_became_peered":"2021-02-10T04:33:05.144488+0000","last_unstale":"2021-02-10T04:44:58.363907+0000","last_undegraded":"2021-02-10T04:44:58.363907+0000","last_fullsized":"2021-02-10T04:44:58.363907+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'301","last_scrub_stamp":"2021-02-10T04:33:56.571925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:56.571925+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":126521476,"num_objects":49,"num_object_clones":44,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":773,"num_read_kb":819582,"num_write":561,"num_write_kb":221117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"538'26","reported_seq":"1075","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.705023+0000","last_change":"2021-02-10T04:38:39.265973+0000","last_active":"2021-02-10T04:44:57.705023+0000","last_peered":"2021-02-10T04:44:57.705023+0000","last_clean":"2021-02-10T04:44:57.705023+0000","last_became_active":"2021-02-10T04:38:39.243413+0000","last_became_peered":"2021-02-10T04:38:39.243413+0000","last_unstale":"2021-02-10T04:44:57.705023+0000","last_undegraded":"2021-02-10T04:44:57.705023+0000","last_fullsized":"2021-02-10T04:44:57.705023+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'26","last_scrub_stamp":"2021-02-10T04:38:32.145511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:32.145511+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":53,"num_read_kb":36,"num_write":27,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":894,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"674","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.007082+0000","last_change":"2021-02-10T04:44:43.015787+0000","last_active":"2021-02-10T04:44:58.007082+0000","last_peered":"2021-02-10T04:44:58.007082+0000","last_clean":"2021-02-10T04:44:58.007082+0000","last_became_active":"2021-02-10T04:33:05.150882+0000","last_became_peered":"2021-02-10T04:33:05.150882+0000","last_unstale":"2021-02-10T04:44:58.007082+0000","last_undegraded":"2021-02-10T04:44:58.007082+0000","last_fullsized":"2021-02-10T04:44:58.007082+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:43.015723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:44:43.015723+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"398","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.005597+0000","last_change":"2021-02-10T04:37:37.244470+0000","last_active":"2021-02-10T04:44:58.005597+0000","last_peered":"2021-02-10T04:44:58.005597+0000","last_clean":"2021-02-10T04:44:58.005597+0000","last_became_active":"2021-02-10T04:37:37.242485+0000","last_became_peered":"2021-02-10T04:37:37.242485+0000","last_unstale":"2021-02-10T04:44:58.005597+0000","last_undegraded":"2021-02-10T04:44:58.005597+0000","last_fullsized":"2021-02-10T04:44:58.005597+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:35.805326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.805326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"1001","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.703872+0000","last_change":"2021-02-10T04:44:33.970634+0000","last_active":"2021-02-10T04:44:57.703872+0000","last_peered":"2021-02-10T04:44:57.703872+0000","last_clean":"2021-02-10T04:44:57.703872+0000","last_became_active":"2021-02-10T04:37:37.254038+0000","last_became_peered":"2021-02-10T04:37:37.254038+0000","last_unstale":"2021-02-10T04:44:57.703872+0000","last_undegraded":"2021-02-10T04:44:57.703872+0000","last_fullsized":"2021-02-10T04:44:57.703872+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:33.970536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:44:33.970536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"259'74","reported_seq":"1926","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.004838+0000","last_change":"2021-02-10T04:44:25.545822+0000","last_active":"2021-02-10T04:44:58.004838+0000","last_peered":"2021-02-10T04:44:58.004838+0000","last_clean":"2021-02-10T04:44:58.004838+0000","last_became_active":"2021-02-10T04:44:25.544898+0000","last_became_peered":"2021-02-10T04:44:25.544898+0000","last_unstale":"2021-02-10T04:44:58.004838+0000","last_undegraded":"2021-02-10T04:44:58.004838+0000","last_fullsized":"2021-02-10T04:44:58.004838+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'62","last_scrub_stamp":"2021-02-10T04:30:10.905905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:10.905905+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1218524,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":28227,"num_write":19,"num_write_kb":7021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":567285,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"538'45","reported_seq":"1080","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.699336+0000","last_change":"2021-02-10T04:44:34.706772+0000","last_active":"2021-02-10T04:44:57.699336+0000","last_peered":"2021-02-10T04:44:57.699336+0000","last_clean":"2021-02-10T04:44:57.699336+0000","last_became_active":"2021-02-10T04:25:49.015913+0000","last_became_peered":"2021-02-10T04:25:49.015913+0000","last_unstale":"2021-02-10T04:44:57.699336+0000","last_undegraded":"2021-02-10T04:44:57.699336+0000","last_fullsized":"2021-02-10T04:44:57.699336+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'45","last_scrub_stamp":"2021-02-10T04:44:34.706681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:44:34.706681+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":80,"num_read_kb":60,"num_write":46,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"642'3","reported_seq":"521","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:45:05.357365+0000","last_change":"2021-02-10T04:38:24.163263+0000","last_active":"2021-02-10T04:45:05.357365+0000","last_peered":"2021-02-10T04:45:05.357365+0000","last_clean":"2021-02-10T04:45:05.357365+0000","last_became_active":"2021-02-10T04:37:37.234230+0000","last_became_peered":"2021-02-10T04:37:37.234230+0000","last_unstale":"2021-02-10T04:45:05.357365+0000","last_undegraded":"2021-02-10T04:45:05.357365+0000","last_fullsized":"2021-02-10T04:45:05.357365+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'3","last_scrub_stamp":"2021-02-10T04:38:24.163200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:24.163200+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"404","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.704200+0000","last_change":"2021-02-10T04:38:31.121100+0000","last_active":"2021-02-10T04:44:57.704200+0000","last_peered":"2021-02-10T04:44:57.704200+0000","last_clean":"2021-02-10T04:44:57.704200+0000","last_became_active":"2021-02-10T04:37:37.230818+0000","last_became_peered":"2021-02-10T04:37:37.230818+0000","last_unstale":"2021-02-10T04:44:57.704200+0000","last_undegraded":"2021-02-10T04:44:57.704200+0000","last_fullsized":"2021-02-10T04:44:57.704200+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:31.121038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:38:31.121038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"402","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.699278+0000","last_change":"2021-02-10T04:37:37.245491+0000","last_active":"2021-02-10T04:44:57.699278+0000","last_peered":"2021-02-10T04:44:57.699278+0000","last_clean":"2021-02-10T04:44:57.699278+0000","last_became_active":"2021-02-10T04:37:37.245113+0000","last_became_peered":"2021-02-10T04:37:37.245113+0000","last_unstale":"2021-02-10T04:44:57.699278+0000","last_undegraded":"2021-02-10T04:44:57.699278+0000","last_fullsized":"2021-02-10T04:44:57.699278+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.731458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.731458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"1003'739","reported_seq":"3686","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.703402+0000","last_change":"2021-02-10T04:44:25.546236+0000","last_active":"2021-02-10T04:44:57.703402+0000","last_peered":"2021-02-10T04:44:57.703402+0000","last_clean":"2021-02-10T04:44:57.703402+0000","last_became_active":"2021-02-10T04:44:25.545483+0000","last_became_peered":"2021-02-10T04:44:25.545483+0000","last_unstale":"2021-02-10T04:44:57.703402+0000","last_undegraded":"2021-02-10T04:44:57.703402+0000","last_fullsized":"2021-02-10T04:44:57.703402+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'63","last_scrub_stamp":"2021-02-10T04:29:39.700115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:29:39.700115+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24059888,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":129,"num_read_kb":109544,"num_write":102,"num_write_kb":46173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"538'47","reported_seq":"1092","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.007730+0000","last_change":"2021-02-10T04:37:37.821860+0000","last_active":"2021-02-10T04:44:58.007730+0000","last_peered":"2021-02-10T04:44:58.007730+0000","last_clean":"2021-02-10T04:44:58.007730+0000","last_became_active":"2021-02-10T04:37:37.237215+0000","last_became_peered":"2021-02-10T04:37:37.237215+0000","last_unstale":"2021-02-10T04:44:58.007730+0000","last_undegraded":"2021-02-10T04:44:58.007730+0000","last_fullsized":"2021-02-10T04:44:58.007730+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'33","last_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:28:49.514984+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":62,"num_read_kb":44,"num_write":36,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"906","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.005932+0000","last_change":"2021-02-10T04:29:07.146266+0000","last_active":"2021-02-10T04:44:58.005932+0000","last_peered":"2021-02-10T04:44:58.005932+0000","last_clean":"2021-02-10T04:44:58.005932+0000","last_became_active":"2021-02-10T04:29:07.145616+0000","last_became_peered":"2021-02-10T04:29:07.145616+0000","last_unstale":"2021-02-10T04:44:58.005932+0000","last_undegraded":"2021-02-10T04:44:58.005932+0000","last_fullsized":"2021-02-10T04:44:58.005932+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:44.410817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:44.410817+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"669","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.006321+0000","last_change":"2021-02-10T04:33:05.147166+0000","last_active":"2021-02-10T04:44:58.006321+0000","last_peered":"2021-02-10T04:44:58.006321+0000","last_clean":"2021-02-10T04:44:58.006321+0000","last_became_active":"2021-02-10T04:33:05.146783+0000","last_became_peered":"2021-02-10T04:33:05.146783+0000","last_unstale":"2021-02-10T04:44:58.006321+0000","last_undegraded":"2021-02-10T04:44:58.006321+0000","last_fullsized":"2021-02-10T04:44:58.006321+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:35.958255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.958255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"404","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.701950+0000","last_change":"2021-02-10T04:38:32.875574+0000","last_active":"2021-02-10T04:44:57.701950+0000","last_peered":"2021-02-10T04:44:57.701950+0000","last_clean":"2021-02-10T04:44:57.701950+0000","last_became_active":"2021-02-10T04:37:37.251479+0000","last_became_peered":"2021-02-10T04:37:37.251479+0000","last_unstale":"2021-02-10T04:44:57.701950+0000","last_undegraded":"2021-02-10T04:44:57.701950+0000","last_fullsized":"2021-02-10T04:44:57.701950+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:32.875498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:38:32.875498+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"999'629","reported_seq":"4192","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.364687+0000","last_change":"2021-02-10T04:43:53.673766+0000","last_active":"2021-02-10T04:44:58.364687+0000","last_peered":"2021-02-10T04:44:58.364687+0000","last_clean":"2021-02-10T04:44:58.364687+0000","last_became_active":"2021-02-10T04:37:40.267794+0000","last_became_peered":"2021-02-10T04:37:40.267794+0000","last_unstale":"2021-02-10T04:44:58.364687+0000","last_undegraded":"2021-02-10T04:44:58.364687+0000","last_fullsized":"2021-02-10T04:44:58.364687+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"688'187","last_scrub_stamp":"2021-02-10T04:38:27.225488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:38:27.225488+0000","log_size":629,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49401261,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":2,"num_read":342,"num_read_kb":364544,"num_write":223,"num_write_kb":67454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":11919937,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"938","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.362856+0000","last_change":"2021-02-10T04:37:37.246395+0000","last_active":"2021-02-10T04:44:58.362856+0000","last_peered":"2021-02-10T04:44:58.362856+0000","last_clean":"2021-02-10T04:44:58.362856+0000","last_became_active":"2021-02-10T04:37:37.246004+0000","last_became_peered":"2021-02-10T04:37:37.246004+0000","last_unstale":"2021-02-10T04:44:58.362856+0000","last_undegraded":"2021-02-10T04:44:58.362856+0000","last_fullsized":"2021-02-10T04:44:58.362856+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:37.336092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:37.336092+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"1011","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.700131+0000","last_change":"2021-02-10T04:37:37.259938+0000","last_active":"2021-02-10T04:44:57.700131+0000","last_peered":"2021-02-10T04:44:57.700131+0000","last_clean":"2021-02-10T04:44:57.700131+0000","last_became_active":"2021-02-10T04:37:37.259532+0000","last_became_peered":"2021-02-10T04:37:37.259532+0000","last_unstale":"2021-02-10T04:44:57.700131+0000","last_undegraded":"2021-02-10T04:44:57.700131+0000","last_fullsized":"2021-02-10T04:44:57.700131+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:51.858386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:33:51.858386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"538'23","reported_seq":"1029","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.006384+0000","last_change":"2021-02-10T04:44:56.117260+0000","last_active":"2021-02-10T04:44:58.006384+0000","last_peered":"2021-02-10T04:44:58.006384+0000","last_clean":"2021-02-10T04:44:58.006384+0000","last_became_active":"2021-02-10T04:25:49.001725+0000","last_became_peered":"2021-02-10T04:25:49.001725+0000","last_unstale":"2021-02-10T04:44:58.006384+0000","last_undegraded":"2021-02-10T04:44:58.006384+0000","last_fullsized":"2021-02-10T04:44:58.006384+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'23","last_scrub_stamp":"2021-02-10T04:44:56.117138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:44:56.117138+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":28,"num_write":23,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"674","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.699739+0000","last_change":"2021-02-10T04:33:05.168734+0000","last_active":"2021-02-10T04:44:57.699739+0000","last_peered":"2021-02-10T04:44:57.699739+0000","last_clean":"2021-02-10T04:44:57.699739+0000","last_became_active":"2021-02-10T04:33:05.168314+0000","last_became_peered":"2021-02-10T04:33:05.168314+0000","last_unstale":"2021-02-10T04:44:57.699739+0000","last_undegraded":"2021-02-10T04:44:57.699739+0000","last_fullsized":"2021-02-10T04:44:57.699739+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:18.107867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:18.107867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"639'3","reported_seq":"509","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:45:07.917008+0000","last_change":"2021-02-10T04:38:31.211769+0000","last_active":"2021-02-10T04:45:07.917008+0000","last_peered":"2021-02-10T04:45:07.917008+0000","last_clean":"2021-02-10T04:45:07.917008+0000","last_became_active":"2021-02-10T04:37:37.233738+0000","last_became_peered":"2021-02-10T04:37:37.233738+0000","last_unstale":"2021-02-10T04:45:07.917008+0000","last_undegraded":"2021-02-10T04:45:07.917008+0000","last_fullsized":"2021-02-10T04:45:07.917008+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"639'3","last_scrub_stamp":"2021-02-10T04:38:31.211662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:38:31.211662+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"993","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.011133+0000","last_change":"2021-02-10T04:37:37.236009+0000","last_active":"2021-02-10T04:44:58.011133+0000","last_peered":"2021-02-10T04:44:58.011133+0000","last_clean":"2021-02-10T04:44:58.011133+0000","last_became_active":"2021-02-10T04:37:37.233109+0000","last_became_peered":"2021-02-10T04:37:37.233109+0000","last_unstale":"2021-02-10T04:44:58.011133+0000","last_undegraded":"2021-02-10T04:44:58.011133+0000","last_fullsized":"2021-02-10T04:44:58.011133+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:38.873102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.873102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"1004'1093","reported_seq":"6244","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.704902+0000","last_change":"2021-02-10T04:44:10.007828+0000","last_active":"2021-02-10T04:44:57.704902+0000","last_peered":"2021-02-10T04:44:57.704902+0000","last_clean":"2021-02-10T04:44:57.704902+0000","last_became_active":"2021-02-10T04:28:25.149226+0000","last_became_peered":"2021-02-10T04:28:25.149226+0000","last_unstale":"2021-02-10T04:44:57.704902+0000","last_undegraded":"2021-02-10T04:44:57.704902+0000","last_fullsized":"2021-02-10T04:44:57.704902+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'236","last_scrub_stamp":"2021-02-10T04:33:50.351534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:50.351534+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":78869393,"num_objects":34,"num_object_clones":30,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":1,"num_read":558,"num_read_kb":622116,"num_write":371,"num_write_kb":133580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"938","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.363149+0000","last_change":"2021-02-10T04:37:37.232619+0000","last_active":"2021-02-10T04:44:58.363149+0000","last_peered":"2021-02-10T04:44:58.363149+0000","last_clean":"2021-02-10T04:44:58.363149+0000","last_became_active":"2021-02-10T04:37:37.229648+0000","last_became_peered":"2021-02-10T04:37:37.229648+0000","last_unstale":"2021-02-10T04:44:58.363149+0000","last_undegraded":"2021-02-10T04:44:58.363149+0000","last_fullsized":"2021-02-10T04:44:58.363149+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.384170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.384170+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"538'29","reported_seq":"730","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.701741+0000","last_change":"2021-02-10T04:33:05.154998+0000","last_active":"2021-02-10T04:44:57.701741+0000","last_peered":"2021-02-10T04:44:57.701741+0000","last_clean":"2021-02-10T04:44:57.701741+0000","last_became_active":"2021-02-10T04:33:05.149026+0000","last_became_peered":"2021-02-10T04:33:05.149026+0000","last_unstale":"2021-02-10T04:44:57.701741+0000","last_undegraded":"2021-02-10T04:44:57.701741+0000","last_fullsized":"2021-02-10T04:44:57.701741+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'17","last_scrub_stamp":"2021-02-10T04:29:37.933206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:37.933206+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":49,"num_read_kb":36,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"734","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.700556+0000","last_change":"2021-02-10T04:33:05.169654+0000","last_active":"2021-02-10T04:44:57.700556+0000","last_peered":"2021-02-10T04:44:57.700556+0000","last_clean":"2021-02-10T04:44:57.700556+0000","last_became_active":"2021-02-10T04:33:05.169289+0000","last_became_peered":"2021-02-10T04:33:05.169289+0000","last_unstale":"2021-02-10T04:44:57.700556+0000","last_undegraded":"2021-02-10T04:44:57.700556+0000","last_fullsized":"2021-02-10T04:44:57.700556+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_clean_scrub_stamp":"2021-02-10T04:28:39.128781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"78'3","reported_seq":"1220","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.817030+0000","last_change":"2021-02-10T04:28:43.509446+0000","last_active":"2021-02-10T04:44:58.817030+0000","last_peered":"2021-02-10T04:44:58.817030+0000","last_clean":"2021-02-10T04:44:58.817030+0000","last_became_active":"2021-02-10T04:25:51.025218+0000","last_became_peered":"2021-02-10T04:25:51.025218+0000","last_unstale":"2021-02-10T04:44:58.817030+0000","last_undegraded":"2021-02-10T04:44:58.817030+0000","last_fullsized":"2021-02-10T04:44:58.817030+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:28:43.509353+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:28:43.509353+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"991","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.366206+0000","last_change":"2021-02-10T04:44:55.693471+0000","last_active":"2021-02-10T04:44:58.366206+0000","last_peered":"2021-02-10T04:44:58.366206+0000","last_clean":"2021-02-10T04:44:58.366206+0000","last_became_active":"2021-02-10T04:25:53.042697+0000","last_became_peered":"2021-02-10T04:25:53.042697+0000","last_unstale":"2021-02-10T04:44:58.366206+0000","last_undegraded":"2021-02-10T04:44:58.366206+0000","last_fullsized":"2021-02-10T04:44:58.366206+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:55.693386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:44:55.693386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"1002'1090","reported_seq":"5184","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.700007+0000","last_change":"2021-02-10T04:43:58.639400+0000","last_active":"2021-02-10T04:44:57.700007+0000","last_peered":"2021-02-10T04:44:57.700007+0000","last_clean":"2021-02-10T04:44:57.700007+0000","last_became_active":"2021-02-10T04:28:25.148500+0000","last_became_peered":"2021-02-10T04:28:25.148500+0000","last_unstale":"2021-02-10T04:44:57.700007+0000","last_undegraded":"2021-02-10T04:44:57.700007+0000","last_fullsized":"2021-02-10T04:44:57.700007+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'110","last_scrub_stamp":"2021-02-10T04:30:18.463782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:18.463782+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":48821292,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":491,"num_read_kb":552336,"num_write":351,"num_write_kb":129921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"538'39","reported_seq":"1057","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.010773+0000","last_change":"2021-02-10T04:44:24.905303+0000","last_active":"2021-02-10T04:44:58.010773+0000","last_peered":"2021-02-10T04:44:58.010773+0000","last_clean":"2021-02-10T04:44:58.010773+0000","last_became_active":"2021-02-10T04:25:49.011093+0000","last_became_peered":"2021-02-10T04:25:49.011093+0000","last_unstale":"2021-02-10T04:44:58.010773+0000","last_undegraded":"2021-02-10T04:44:58.010773+0000","last_fullsized":"2021-02-10T04:44:58.010773+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'39","last_scrub_stamp":"2021-02-10T04:44:24.905188+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:44:24.905188+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":56,"num_read_kb":40,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"975","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.703107+0000","last_change":"2021-02-10T04:44:56.759342+0000","last_active":"2021-02-10T04:44:57.703107+0000","last_peered":"2021-02-10T04:44:57.703107+0000","last_clean":"2021-02-10T04:44:57.703107+0000","last_became_active":"2021-02-10T04:29:07.160593+0000","last_became_peered":"2021-02-10T04:29:07.160593+0000","last_unstale":"2021-02-10T04:44:57.703107+0000","last_undegraded":"2021-02-10T04:44:57.703107+0000","last_fullsized":"2021-02-10T04:44:57.703107+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:56.759233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:44:56.759233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"993","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.365372+0000","last_change":"2021-02-10T04:30:10.487631+0000","last_active":"2021-02-10T04:44:58.365372+0000","last_peered":"2021-02-10T04:44:58.365372+0000","last_clean":"2021-02-10T04:44:58.365372+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:44:58.365372+0000","last_undegraded":"2021-02-10T04:44:58.365372+0000","last_fullsized":"2021-02-10T04:44:58.365372+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:10.487528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:30:10.487528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"984","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.700623+0000","last_change":"2021-02-10T04:33:55.849675+0000","last_active":"2021-02-10T04:44:57.700623+0000","last_peered":"2021-02-10T04:44:57.700623+0000","last_clean":"2021-02-10T04:44:57.700623+0000","last_became_active":"2021-02-10T04:25:53.148054+0000","last_became_peered":"2021-02-10T04:25:53.148054+0000","last_unstale":"2021-02-10T04:44:57.700623+0000","last_undegraded":"2021-02-10T04:44:57.700623+0000","last_fullsized":"2021-02-10T04:44:57.700623+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:55.849582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:33:55.849582+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"1001'1165","reported_seq":"4163","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.699313+0000","last_change":"2021-02-10T04:43:57.854903+0000","last_active":"2021-02-10T04:44:57.699313+0000","last_peered":"2021-02-10T04:44:57.699313+0000","last_clean":"2021-02-10T04:44:57.699313+0000","last_became_active":"2021-02-10T04:28:25.152030+0000","last_became_peered":"2021-02-10T04:28:25.152030+0000","last_unstale":"2021-02-10T04:44:57.699313+0000","last_undegraded":"2021-02-10T04:44:57.699313+0000","last_fullsized":"2021-02-10T04:44:57.699313+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'283","last_scrub_stamp":"2021-02-10T04:33:53.834627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:53.834627+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":59203690,"num_objects":25,"num_object_clones":21,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":573,"num_read_kb":599059,"num_write":400,"num_write_kb":178261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"158'3","reported_seq":"1220","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.366710+0000","last_change":"2021-02-10T04:29:34.388978+0000","last_active":"2021-02-10T04:44:58.366710+0000","last_peered":"2021-02-10T04:44:58.366710+0000","last_clean":"2021-02-10T04:44:58.366710+0000","last_became_active":"2021-02-10T04:25:51.031906+0000","last_became_peered":"2021-02-10T04:25:51.031906+0000","last_unstale":"2021-02-10T04:44:58.366710+0000","last_undegraded":"2021-02-10T04:44:58.366710+0000","last_fullsized":"2021-02-10T04:44:58.366710+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'3","last_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_deep_scrub":"158'3","last_deep_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_clean_scrub_stamp":"2021-02-10T04:29:34.388912+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"538'40","reported_seq":"1057","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.004915+0000","last_change":"2021-02-10T04:29:36.813190+0000","last_active":"2021-02-10T04:44:58.004915+0000","last_peered":"2021-02-10T04:44:58.004915+0000","last_clean":"2021-02-10T04:44:58.004915+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:44:58.004915+0000","last_undegraded":"2021-02-10T04:44:58.004915+0000","last_fullsized":"2021-02-10T04:44:58.004915+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:29:36.813083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.813083+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":72,"num_read_kb":52,"num_write":40,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"1042","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.699314+0000","last_change":"2021-02-10T04:44:24.719062+0000","last_active":"2021-02-10T04:44:57.699314+0000","last_peered":"2021-02-10T04:44:57.699314+0000","last_clean":"2021-02-10T04:44:57.699314+0000","last_became_active":"2021-02-10T04:37:37.245391+0000","last_became_peered":"2021-02-10T04:37:37.245391+0000","last_unstale":"2021-02-10T04:44:57.699314+0000","last_undegraded":"2021-02-10T04:44:57.699314+0000","last_fullsized":"2021-02-10T04:44:57.699314+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:24.718981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:44:24.718981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"926","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.700640+0000","last_change":"2021-02-10T04:44:52.779825+0000","last_active":"2021-02-10T04:44:57.700640+0000","last_peered":"2021-02-10T04:44:57.700640+0000","last_clean":"2021-02-10T04:44:57.700640+0000","last_became_active":"2021-02-10T04:37:37.235082+0000","last_became_peered":"2021-02-10T04:37:37.235082+0000","last_unstale":"2021-02-10T04:44:57.700640+0000","last_undegraded":"2021-02-10T04:44:57.700640+0000","last_fullsized":"2021-02-10T04:44:57.700640+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:52.779754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:44:52.779754+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"998'1009","reported_seq":"4519","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.699331+0000","last_change":"2021-02-10T04:44:25.543675+0000","last_active":"2021-02-10T04:44:57.699331+0000","last_peered":"2021-02-10T04:44:57.699331+0000","last_clean":"2021-02-10T04:44:57.699331+0000","last_became_active":"2021-02-10T04:44:25.543138+0000","last_became_peered":"2021-02-10T04:44:25.543138+0000","last_unstale":"2021-02-10T04:44:57.699331+0000","last_undegraded":"2021-02-10T04:44:57.699331+0000","last_fullsized":"2021-02-10T04:44:57.699331+0000","mapping_epoch":646,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24710413,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":239,"num_read_kb":249403,"num_write":154,"num_write_kb":50104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":8290058,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"928","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.700699+0000","last_change":"2021-02-10T04:30:12.452072+0000","last_active":"2021-02-10T04:44:57.700699+0000","last_peered":"2021-02-10T04:44:57.700699+0000","last_clean":"2021-02-10T04:44:57.700699+0000","last_became_active":"2021-02-10T04:28:49.048525+0000","last_became_peered":"2021-02-10T04:28:49.048525+0000","last_unstale":"2021-02-10T04:44:57.700699+0000","last_undegraded":"2021-02-10T04:44:57.700699+0000","last_fullsized":"2021-02-10T04:44:57.700699+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:12.452000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:30:12.452000+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"986","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.005063+0000","last_change":"2021-02-10T04:29:40.859352+0000","last_active":"2021-02-10T04:44:58.005063+0000","last_peered":"2021-02-10T04:44:58.005063+0000","last_clean":"2021-02-10T04:44:58.005063+0000","last_became_active":"2021-02-10T04:25:51.018706+0000","last_became_peered":"2021-02-10T04:25:51.018706+0000","last_unstale":"2021-02-10T04:44:58.005063+0000","last_undegraded":"2021-02-10T04:44:58.005063+0000","last_fullsized":"2021-02-10T04:44:58.005063+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:40.859274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:40.859274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"538'60","reported_seq":"1090","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.699165+0000","last_change":"2021-02-10T04:29:28.398522+0000","last_active":"2021-02-10T04:44:57.699165+0000","last_peered":"2021-02-10T04:44:57.699165+0000","last_clean":"2021-02-10T04:44:57.699165+0000","last_became_active":"2021-02-10T04:25:48.997481+0000","last_became_peered":"2021-02-10T04:25:48.997481+0000","last_unstale":"2021-02-10T04:44:57.699165+0000","last_undegraded":"2021-02-10T04:44:57.699165+0000","last_fullsized":"2021-02-10T04:44:57.699165+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'48","last_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_deep_scrub":"152'48","last_deep_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.398429+0000","log_size":60,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":88,"num_read_kb":66,"num_write":48,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"683","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.699164+0000","last_change":"2021-02-10T04:33:05.142730+0000","last_active":"2021-02-10T04:44:57.699164+0000","last_peered":"2021-02-10T04:44:57.699164+0000","last_clean":"2021-02-10T04:44:57.699164+0000","last_became_active":"2021-02-10T04:33:05.141066+0000","last_became_peered":"2021-02-10T04:33:05.141066+0000","last_unstale":"2021-02-10T04:44:57.699164+0000","last_undegraded":"2021-02-10T04:44:57.699164+0000","last_fullsized":"2021-02-10T04:44:57.699164+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:28:35.010709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:35.010709+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":423,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"998","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.005693+0000","last_change":"2021-02-10T04:44:27.926730+0000","last_active":"2021-02-10T04:44:58.005693+0000","last_peered":"2021-02-10T04:44:58.005693+0000","last_clean":"2021-02-10T04:44:58.005693+0000","last_became_active":"2021-02-10T04:37:37.244859+0000","last_became_peered":"2021-02-10T04:37:37.244859+0000","last_unstale":"2021-02-10T04:44:58.005693+0000","last_undegraded":"2021-02-10T04:44:58.005693+0000","last_fullsized":"2021-02-10T04:44:58.005693+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:27.926662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:44:27.926662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"1002'1657","reported_seq":"4934","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.700731+0000","last_change":"2021-02-10T04:44:25.550700+0000","last_active":"2021-02-10T04:44:57.700731+0000","last_peered":"2021-02-10T04:44:57.700731+0000","last_clean":"2021-02-10T04:44:57.700731+0000","last_became_active":"2021-02-10T04:44:25.549094+0000","last_became_peered":"2021-02-10T04:44:25.549094+0000","last_unstale":"2021-02-10T04:44:57.700731+0000","last_undegraded":"2021-02-10T04:44:57.700731+0000","last_fullsized":"2021-02-10T04:44:57.700731+0000","mapping_epoch":646,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"687'1095","last_scrub_stamp":"2021-02-10T04:38:27.895743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:38:27.895743+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":42793066,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":424,"num_read_kb":437673,"num_write":318,"num_write_kb":118797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":17936938,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"936","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.006605+0000","last_change":"2021-02-10T04:37:37.233535+0000","last_active":"2021-02-10T04:44:58.006605+0000","last_peered":"2021-02-10T04:44:58.006605+0000","last_clean":"2021-02-10T04:44:58.006605+0000","last_became_active":"2021-02-10T04:37:37.233158+0000","last_became_peered":"2021-02-10T04:37:37.233158+0000","last_unstale":"2021-02-10T04:44:58.006605+0000","last_undegraded":"2021-02-10T04:44:58.006605+0000","last_fullsized":"2021-02-10T04:44:58.006605+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:33.839335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:33.839335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"1058","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.699881+0000","last_change":"2021-02-10T04:33:52.827328+0000","last_active":"2021-02-10T04:44:57.699881+0000","last_peered":"2021-02-10T04:44:57.699881+0000","last_clean":"2021-02-10T04:44:57.699881+0000","last_became_active":"2021-02-10T04:25:51.025636+0000","last_became_peered":"2021-02-10T04:25:51.025636+0000","last_unstale":"2021-02-10T04:44:57.699881+0000","last_undegraded":"2021-02-10T04:44:57.699881+0000","last_fullsized":"2021-02-10T04:44:57.699881+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:52.827257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:33:52.827257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"538'37","reported_seq":"1064","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.704071+0000","last_change":"2021-02-10T04:38:36.138126+0000","last_active":"2021-02-10T04:44:57.704071+0000","last_peered":"2021-02-10T04:44:57.704071+0000","last_clean":"2021-02-10T04:44:57.704071+0000","last_became_active":"2021-02-10T04:25:49.002937+0000","last_became_peered":"2021-02-10T04:25:49.002937+0000","last_unstale":"2021-02-10T04:44:57.704071+0000","last_undegraded":"2021-02-10T04:44:57.704071+0000","last_fullsized":"2021-02-10T04:44:57.704071+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'37","last_scrub_stamp":"2021-02-10T04:38:36.138022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:36.138022+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":72,"num_read_kb":53,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"923","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.704508+0000","last_change":"2021-02-10T04:29:35.711635+0000","last_active":"2021-02-10T04:44:57.704508+0000","last_peered":"2021-02-10T04:44:57.704508+0000","last_clean":"2021-02-10T04:44:57.704508+0000","last_became_active":"2021-02-10T04:29:07.161341+0000","last_became_peered":"2021-02-10T04:29:07.161341+0000","last_unstale":"2021-02-10T04:44:57.704508+0000","last_undegraded":"2021-02-10T04:44:57.704508+0000","last_fullsized":"2021-02-10T04:44:57.704508+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_deep_scrub":"53'2","last_deep_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.711557+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"684","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.701168+0000","last_change":"2021-02-10T04:38:40.414663+0000","last_active":"2021-02-10T04:44:57.701168+0000","last_peered":"2021-02-10T04:44:57.701168+0000","last_clean":"2021-02-10T04:44:57.701168+0000","last_became_active":"2021-02-10T04:37:37.234868+0000","last_became_peered":"2021-02-10T04:37:37.234868+0000","last_unstale":"2021-02-10T04:44:57.701168+0000","last_undegraded":"2021-02-10T04:44:57.701168+0000","last_fullsized":"2021-02-10T04:44:57.701168+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:40.414601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:38:40.414601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"1000'1071","reported_seq":"4542","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.363256+0000","last_change":"2021-02-10T04:44:25.539713+0000","last_active":"2021-02-10T04:44:58.363256+0000","last_peered":"2021-02-10T04:44:58.363256+0000","last_clean":"2021-02-10T04:44:58.363256+0000","last_became_active":"2021-02-10T04:44:25.536840+0000","last_became_peered":"2021-02-10T04:44:25.536840+0000","last_unstale":"2021-02-10T04:44:58.363256+0000","last_undegraded":"2021-02-10T04:44:58.363256+0000","last_fullsized":"2021-02-10T04:44:58.363256+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'357","last_scrub_stamp":"2021-02-10T04:33:50.582002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:50.582002+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23286368,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":242,"num_read_kb":226834,"num_write":185,"num_write_kb":76004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"1014","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.699415+0000","last_change":"2021-02-10T04:44:35.689296+0000","last_active":"2021-02-10T04:44:57.699415+0000","last_peered":"2021-02-10T04:44:57.699415+0000","last_clean":"2021-02-10T04:44:57.699415+0000","last_became_active":"2021-02-10T04:28:49.054638+0000","last_became_peered":"2021-02-10T04:28:49.054638+0000","last_unstale":"2021-02-10T04:44:57.699415+0000","last_undegraded":"2021-02-10T04:44:57.699415+0000","last_fullsized":"2021-02-10T04:44:57.699415+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:35.689202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:44:35.689202+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"988","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.365497+0000","last_change":"2021-02-10T04:29:40.451948+0000","last_active":"2021-02-10T04:44:58.365497+0000","last_peered":"2021-02-10T04:44:58.365497+0000","last_clean":"2021-02-10T04:44:58.365497+0000","last_became_active":"2021-02-10T04:25:51.032201+0000","last_became_peered":"2021-02-10T04:25:51.032201+0000","last_unstale":"2021-02-10T04:44:58.365497+0000","last_undegraded":"2021-02-10T04:44:58.365497+0000","last_fullsized":"2021-02-10T04:44:58.365497+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:40.451832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:29:40.451832+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"538'28","reported_seq":"1069","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.704547+0000","last_change":"2021-02-10T04:37:38.566112+0000","last_active":"2021-02-10T04:44:57.704547+0000","last_peered":"2021-02-10T04:44:57.704547+0000","last_clean":"2021-02-10T04:44:57.704547+0000","last_became_active":"2021-02-10T04:37:37.247141+0000","last_became_peered":"2021-02-10T04:37:37.247141+0000","last_unstale":"2021-02-10T04:44:57.704547+0000","last_undegraded":"2021-02-10T04:44:57.704547+0000","last_fullsized":"2021-02-10T04:44:57.704547+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'16","last_scrub_stamp":"2021-02-10T04:33:55.288033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:55.288033+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":34,"num_write":28,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"726","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.703387+0000","last_change":"2021-02-10T04:33:05.166788+0000","last_active":"2021-02-10T04:44:57.703387+0000","last_peered":"2021-02-10T04:44:57.703387+0000","last_clean":"2021-02-10T04:44:57.703387+0000","last_became_active":"2021-02-10T04:33:05.166368+0000","last_became_peered":"2021-02-10T04:33:05.166368+0000","last_unstale":"2021-02-10T04:44:57.703387+0000","last_undegraded":"2021-02-10T04:44:57.703387+0000","last_fullsized":"2021-02-10T04:44:57.703387+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:14.204533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:30:14.204533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"994","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.703661+0000","last_change":"2021-02-10T04:44:36.852583+0000","last_active":"2021-02-10T04:44:57.703661+0000","last_peered":"2021-02-10T04:44:57.703661+0000","last_clean":"2021-02-10T04:44:57.703661+0000","last_became_active":"2021-02-10T04:25:49.001782+0000","last_became_peered":"2021-02-10T04:25:49.001782+0000","last_unstale":"2021-02-10T04:44:57.703661+0000","last_undegraded":"2021-02-10T04:44:57.703661+0000","last_fullsized":"2021-02-10T04:44:57.703661+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:36.852513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:44:36.852513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"671","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.364555+0000","last_change":"2021-02-10T04:33:05.153205+0000","last_active":"2021-02-10T04:44:58.364555+0000","last_peered":"2021-02-10T04:44:58.364555+0000","last_clean":"2021-02-10T04:44:58.364555+0000","last_became_active":"2021-02-10T04:33:05.152845+0000","last_became_peered":"2021-02-10T04:33:05.152845+0000","last_unstale":"2021-02-10T04:44:58.364555+0000","last_undegraded":"2021-02-10T04:44:58.364555+0000","last_fullsized":"2021-02-10T04:44:58.364555+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.098681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.098681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.5","version":"1000'446","reported_seq":"1941","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.704146+0000","last_change":"2021-02-10T04:44:25.550801+0000","last_active":"2021-02-10T04:44:57.704146+0000","last_peered":"2021-02-10T04:44:57.704146+0000","last_clean":"2021-02-10T04:44:57.704146+0000","last_became_active":"2021-02-10T04:44:25.549930+0000","last_became_peered":"2021-02-10T04:44:25.549930+0000","last_unstale":"2021-02-10T04:44:57.704146+0000","last_undegraded":"2021-02-10T04:44:57.704146+0000","last_fullsized":"2021-02-10T04:44:57.704146+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"683'135","last_scrub_stamp":"2021-02-10T04:38:23.183462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:38:23.183462+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22255852,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68435,"num_write":72,"num_write_kb":24345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":403693,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"749","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.700102+0000","last_change":"2021-02-10T04:33:05.141311+0000","last_active":"2021-02-10T04:44:57.700102+0000","last_peered":"2021-02-10T04:44:57.700102+0000","last_clean":"2021-02-10T04:44:57.700102+0000","last_became_active":"2021-02-10T04:33:05.140966+0000","last_became_peered":"2021-02-10T04:33:05.140966+0000","last_unstale":"2021-02-10T04:44:57.700102+0000","last_undegraded":"2021-02-10T04:44:57.700102+0000","last_fullsized":"2021-02-10T04:44:57.700102+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.928906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.928906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"403","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.364920+0000","last_change":"2021-02-10T04:37:37.235036+0000","last_active":"2021-02-10T04:44:58.364920+0000","last_peered":"2021-02-10T04:44:58.364920+0000","last_clean":"2021-02-10T04:44:58.364920+0000","last_became_active":"2021-02-10T04:37:37.234644+0000","last_became_peered":"2021-02-10T04:37:37.234644+0000","last_unstale":"2021-02-10T04:44:58.364920+0000","last_undegraded":"2021-02-10T04:44:58.364920+0000","last_fullsized":"2021-02-10T04:44:58.364920+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.805432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.805432+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"538'27","reported_seq":"1036","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.699038+0000","last_change":"2021-02-10T04:33:56.837896+0000","last_active":"2021-02-10T04:44:57.699038+0000","last_peered":"2021-02-10T04:44:57.699038+0000","last_clean":"2021-02-10T04:44:57.699038+0000","last_became_active":"2021-02-10T04:25:49.002471+0000","last_became_peered":"2021-02-10T04:25:49.002471+0000","last_unstale":"2021-02-10T04:44:57.699038+0000","last_undegraded":"2021-02-10T04:44:57.699038+0000","last_fullsized":"2021-02-10T04:44:57.699038+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2021-02-10T04:33:56.837816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:56.837816+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"997","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.703151+0000","last_change":"2021-02-10T04:44:55.738512+0000","last_active":"2021-02-10T04:44:57.703151+0000","last_peered":"2021-02-10T04:44:57.703151+0000","last_clean":"2021-02-10T04:44:57.703151+0000","last_became_active":"2021-02-10T04:29:07.158075+0000","last_became_peered":"2021-02-10T04:29:07.158075+0000","last_unstale":"2021-02-10T04:44:57.703151+0000","last_undegraded":"2021-02-10T04:44:57.703151+0000","last_fullsized":"2021-02-10T04:44:57.703151+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:55.738445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:44:55.738445+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"1163","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.364021+0000","last_change":"2021-02-10T04:44:26.684937+0000","last_active":"2021-02-10T04:44:58.364021+0000","last_peered":"2021-02-10T04:44:58.364021+0000","last_clean":"2021-02-10T04:44:58.364021+0000","last_became_active":"2021-02-10T04:37:37.245931+0000","last_became_peered":"2021-02-10T04:37:37.245931+0000","last_unstale":"2021-02-10T04:44:58.364021+0000","last_undegraded":"2021-02-10T04:44:58.364021+0000","last_fullsized":"2021-02-10T04:44:58.364021+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:26.684870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:44:26.684870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"988","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.699811+0000","last_change":"2021-02-10T04:29:35.382459+0000","last_active":"2021-02-10T04:44:57.699811+0000","last_peered":"2021-02-10T04:44:57.699811+0000","last_clean":"2021-02-10T04:44:57.699811+0000","last_became_active":"2021-02-10T04:25:49.010614+0000","last_became_peered":"2021-02-10T04:25:49.010614+0000","last_unstale":"2021-02-10T04:44:57.699811+0000","last_undegraded":"2021-02-10T04:44:57.699811+0000","last_fullsized":"2021-02-10T04:44:57.699811+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:35.382341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.382341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"991","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.365546+0000","last_change":"2021-02-10T04:44:34.667926+0000","last_active":"2021-02-10T04:44:58.365546+0000","last_peered":"2021-02-10T04:44:58.365546+0000","last_clean":"2021-02-10T04:44:58.365546+0000","last_became_active":"2021-02-10T04:25:53.037506+0000","last_became_peered":"2021-02-10T04:25:53.037506+0000","last_unstale":"2021-02-10T04:44:58.365546+0000","last_undegraded":"2021-02-10T04:44:58.365546+0000","last_fullsized":"2021-02-10T04:44:58.365546+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:34.667847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:44:34.667847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"1000'490","reported_seq":"3513","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.704647+0000","last_change":"2021-02-10T04:44:25.551027+0000","last_active":"2021-02-10T04:44:57.704647+0000","last_peered":"2021-02-10T04:44:57.704647+0000","last_clean":"2021-02-10T04:44:57.704647+0000","last_became_active":"2021-02-10T04:44:25.550384+0000","last_became_peered":"2021-02-10T04:44:25.550384+0000","last_unstale":"2021-02-10T04:44:57.704647+0000","last_undegraded":"2021-02-10T04:44:57.704647+0000","last_fullsized":"2021-02-10T04:44:57.704647+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'36","last_scrub_stamp":"2021-02-10T04:30:13.863430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:13.863430+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19371627,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":141,"num_read_kb":148568,"num_write":85,"num_write_kb":31732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"340","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.699027+0000","last_change":"2021-02-10T04:44:41.760153+0000","last_active":"2021-02-10T04:44:57.699027+0000","last_peered":"2021-02-10T04:44:57.699027+0000","last_clean":"2021-02-10T04:44:57.699027+0000","last_became_active":"2021-02-10T04:38:39.252266+0000","last_became_peered":"2021-02-10T04:38:39.252266+0000","last_unstale":"2021-02-10T04:44:57.699027+0000","last_undegraded":"2021-02-10T04:44:57.699027+0000","last_fullsized":"2021-02-10T04:44:57.699027+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:41.760055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:44:41.760055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"538'33","reported_seq":"759","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.365320+0000","last_change":"2021-02-10T04:33:05.147122+0000","last_active":"2021-02-10T04:44:58.365320+0000","last_peered":"2021-02-10T04:44:58.365320+0000","last_clean":"2021-02-10T04:44:58.365320+0000","last_became_active":"2021-02-10T04:33:05.146695+0000","last_became_peered":"2021-02-10T04:33:05.146695+0000","last_unstale":"2021-02-10T04:44:58.365320+0000","last_undegraded":"2021-02-10T04:44:58.365320+0000","last_fullsized":"2021-02-10T04:44:58.365320+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'29","last_scrub_stamp":"2021-02-10T04:29:25.974280+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.974280+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":38,"num_write":28,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"458","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.701278+0000","last_change":"2021-02-10T04:37:37.258992+0000","last_active":"2021-02-10T04:44:57.701278+0000","last_peered":"2021-02-10T04:44:57.701278+0000","last_clean":"2021-02-10T04:44:57.701278+0000","last_became_active":"2021-02-10T04:37:37.257600+0000","last_became_peered":"2021-02-10T04:37:37.257600+0000","last_unstale":"2021-02-10T04:44:57.701278+0000","last_undegraded":"2021-02-10T04:44:57.701278+0000","last_fullsized":"2021-02-10T04:44:57.701278+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:39.690436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:39.690436+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"877","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:45:02.833257+0000","last_change":"2021-02-10T04:38:33.243623+0000","last_active":"2021-02-10T04:45:02.833257+0000","last_peered":"2021-02-10T04:45:02.833257+0000","last_clean":"2021-02-10T04:45:02.833257+0000","last_became_active":"2021-02-10T04:37:37.249591+0000","last_became_peered":"2021-02-10T04:37:37.249591+0000","last_unstale":"2021-02-10T04:45:02.833257+0000","last_undegraded":"2021-02-10T04:45:02.833257+0000","last_fullsized":"2021-02-10T04:45:02.833257+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:38:33.243519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:33.243519+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":453,"num_read_kb":448,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"917","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.699222+0000","last_change":"2021-02-10T04:44:45.683998+0000","last_active":"2021-02-10T04:44:57.699222+0000","last_peered":"2021-02-10T04:44:57.699222+0000","last_clean":"2021-02-10T04:44:57.699222+0000","last_became_active":"2021-02-10T04:29:07.158869+0000","last_became_peered":"2021-02-10T04:29:07.158869+0000","last_unstale":"2021-02-10T04:44:57.699222+0000","last_undegraded":"2021-02-10T04:44:57.699222+0000","last_fullsized":"2021-02-10T04:44:57.699222+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:44:45.683902+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.4","version":"1000'620","reported_seq":"4190","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.704738+0000","last_change":"2021-02-10T04:44:25.538085+0000","last_active":"2021-02-10T04:44:57.704738+0000","last_peered":"2021-02-10T04:44:57.704738+0000","last_clean":"2021-02-10T04:44:57.704738+0000","last_became_active":"2021-02-10T04:44:25.537632+0000","last_became_peered":"2021-02-10T04:44:25.537632+0000","last_unstale":"2021-02-10T04:44:57.704738+0000","last_undegraded":"2021-02-10T04:44:57.704738+0000","last_fullsized":"2021-02-10T04:44:57.704738+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'239","last_scrub_stamp":"2021-02-10T04:33:46.270478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:46.270478+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14265195,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":144,"num_read_kb":128482,"num_write":129,"num_write_kb":53443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"1007","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.703944+0000","last_change":"2021-02-10T04:44:45.876889+0000","last_active":"2021-02-10T04:44:57.703944+0000","last_peered":"2021-02-10T04:44:57.703944+0000","last_clean":"2021-02-10T04:44:57.703944+0000","last_became_active":"2021-02-10T04:29:07.159607+0000","last_became_peered":"2021-02-10T04:29:07.159607+0000","last_unstale":"2021-02-10T04:44:57.703944+0000","last_undegraded":"2021-02-10T04:44:57.703944+0000","last_fullsized":"2021-02-10T04:44:57.703944+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:45.876793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:44:45.876793+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"1108","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.704597+0000","last_change":"2021-02-10T04:44:26.031868+0000","last_active":"2021-02-10T04:44:57.704597+0000","last_peered":"2021-02-10T04:44:57.704597+0000","last_clean":"2021-02-10T04:44:57.704597+0000","last_became_active":"2021-02-10T04:25:51.027624+0000","last_became_peered":"2021-02-10T04:25:51.027624+0000","last_unstale":"2021-02-10T04:44:57.704597+0000","last_undegraded":"2021-02-10T04:44:57.704597+0000","last_fullsized":"2021-02-10T04:44:57.704597+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:26.031771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:44:26.031771+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"538'43","reported_seq":"708","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.701112+0000","last_change":"2021-02-10T04:37:38.636607+0000","last_active":"2021-02-10T04:44:57.701112+0000","last_peered":"2021-02-10T04:44:57.701112+0000","last_clean":"2021-02-10T04:44:57.701112+0000","last_became_active":"2021-02-10T04:37:37.244344+0000","last_became_peered":"2021-02-10T04:37:37.244344+0000","last_unstale":"2021-02-10T04:44:57.701112+0000","last_undegraded":"2021-02-10T04:44:57.701112+0000","last_fullsized":"2021-02-10T04:44:57.701112+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'31","last_scrub_stamp":"2021-02-10T04:33:49.423083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:49.423083+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":81,"num_read_kb":57,"num_write":46,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"1011","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.363426+0000","last_change":"2021-02-10T04:38:34.289602+0000","last_active":"2021-02-10T04:44:58.363426+0000","last_peered":"2021-02-10T04:44:58.363426+0000","last_clean":"2021-02-10T04:44:58.363426+0000","last_became_active":"2021-02-10T04:28:21.125541+0000","last_became_peered":"2021-02-10T04:28:21.125541+0000","last_unstale":"2021-02-10T04:44:58.363426+0000","last_undegraded":"2021-02-10T04:44:58.363426+0000","last_fullsized":"2021-02-10T04:44:58.363426+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:34.289495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:34.289495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"984","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.701366+0000","last_change":"2021-02-10T04:29:36.396044+0000","last_active":"2021-02-10T04:44:57.701366+0000","last_peered":"2021-02-10T04:44:57.701366+0000","last_clean":"2021-02-10T04:44:57.701366+0000","last_became_active":"2021-02-10T04:25:53.049487+0000","last_became_peered":"2021-02-10T04:25:53.049487+0000","last_unstale":"2021-02-10T04:44:57.701366+0000","last_undegraded":"2021-02-10T04:44:57.701366+0000","last_fullsized":"2021-02-10T04:44:57.701366+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.395943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.395943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"1002'2042","reported_seq":"9221","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.701247+0000","last_change":"2021-02-10T04:44:25.552653+0000","last_active":"2021-02-10T04:44:57.701247+0000","last_peered":"2021-02-10T04:44:57.701247+0000","last_clean":"2021-02-10T04:44:57.701247+0000","last_became_active":"2021-02-10T04:44:25.549177+0000","last_became_peered":"2021-02-10T04:44:25.549177+0000","last_unstale":"2021-02-10T04:44:57.701247+0000","last_undegraded":"2021-02-10T04:44:57.701247+0000","last_fullsized":"2021-02-10T04:44:57.701247+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'223","last_scrub_stamp":"2021-02-10T04:30:14.478639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:14.478639+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46300076,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":545,"num_read_kb":531689,"num_write":399,"num_write_kb":153087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"938","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.363349+0000","last_change":"2021-02-10T04:37:37.219806+0000","last_active":"2021-02-10T04:44:58.363349+0000","last_peered":"2021-02-10T04:44:58.363349+0000","last_clean":"2021-02-10T04:44:58.363349+0000","last_became_active":"2021-02-10T04:37:37.219422+0000","last_became_peered":"2021-02-10T04:37:37.219422+0000","last_unstale":"2021-02-10T04:44:58.363349+0000","last_undegraded":"2021-02-10T04:44:58.363349+0000","last_fullsized":"2021-02-10T04:44:58.363349+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:39.434210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:29:39.434210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"58'2","reported_seq":"1225","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.363498+0000","last_change":"2021-02-10T04:38:27.092813+0000","last_active":"2021-02-10T04:44:58.363498+0000","last_peered":"2021-02-10T04:44:58.363498+0000","last_clean":"2021-02-10T04:44:58.363498+0000","last_became_active":"2021-02-10T04:25:51.026251+0000","last_became_peered":"2021-02-10T04:25:51.026251+0000","last_unstale":"2021-02-10T04:44:58.363498+0000","last_undegraded":"2021-02-10T04:44:58.363498+0000","last_fullsized":"2021-02-10T04:44:58.363498+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:38:27.092697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:38:27.092697+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"538'44","reported_seq":"761","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.363688+0000","last_change":"2021-02-10T04:33:05.142280+0000","last_active":"2021-02-10T04:44:58.363688+0000","last_peered":"2021-02-10T04:44:58.363688+0000","last_clean":"2021-02-10T04:44:58.363688+0000","last_became_active":"2021-02-10T04:33:05.141901+0000","last_became_peered":"2021-02-10T04:33:05.141901+0000","last_unstale":"2021-02-10T04:44:58.363688+0000","last_undegraded":"2021-02-10T04:44:58.363688+0000","last_fullsized":"2021-02-10T04:44:58.363688+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'34","last_scrub_stamp":"2021-02-10T04:28:49.959162+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:28:49.959162+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":60,"num_read_kb":42,"num_write":32,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"752","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.704592+0000","last_change":"2021-02-10T04:37:37.247740+0000","last_active":"2021-02-10T04:44:57.704592+0000","last_peered":"2021-02-10T04:44:57.704592+0000","last_clean":"2021-02-10T04:44:57.704592+0000","last_became_active":"2021-02-10T04:37:37.247405+0000","last_became_peered":"2021-02-10T04:37:37.247405+0000","last_unstale":"2021-02-10T04:44:57.704592+0000","last_undegraded":"2021-02-10T04:44:57.704592+0000","last_fullsized":"2021-02-10T04:44:57.704592+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:34.053323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:34.053323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"999","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.703068+0000","last_change":"2021-02-10T04:38:26.124897+0000","last_active":"2021-02-10T04:44:57.703068+0000","last_peered":"2021-02-10T04:44:57.703068+0000","last_clean":"2021-02-10T04:44:57.703068+0000","last_became_active":"2021-02-10T04:37:37.254482+0000","last_became_peered":"2021-02-10T04:37:37.254482+0000","last_unstale":"2021-02-10T04:44:57.703068+0000","last_undegraded":"2021-02-10T04:44:57.703068+0000","last_fullsized":"2021-02-10T04:44:57.703068+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:26.124805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:38:26.124805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"293'77","reported_seq":"2781","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.004682+0000","last_change":"2021-02-10T04:44:25.642817+0000","last_active":"2021-02-10T04:44:58.004682+0000","last_peered":"2021-02-10T04:44:58.004682+0000","last_clean":"2021-02-10T04:44:58.004682+0000","last_became_active":"2021-02-10T04:44:25.542375+0000","last_became_peered":"2021-02-10T04:44:25.542375+0000","last_unstale":"2021-02-10T04:44:58.004682+0000","last_undegraded":"2021-02-10T04:44:58.004682+0000","last_fullsized":"2021-02-10T04:44:58.004682+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3773171,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":36,"num_read_kb":37174,"num_write":18,"num_write_kb":6631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"1014","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.699077+0000","last_change":"2021-02-10T04:44:38.740220+0000","last_active":"2021-02-10T04:44:57.699077+0000","last_peered":"2021-02-10T04:44:57.699077+0000","last_clean":"2021-02-10T04:44:57.699077+0000","last_became_active":"2021-02-10T04:28:49.053127+0000","last_became_peered":"2021-02-10T04:28:49.053127+0000","last_unstale":"2021-02-10T04:44:57.699077+0000","last_undegraded":"2021-02-10T04:44:57.699077+0000","last_fullsized":"2021-02-10T04:44:57.699077+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:38.740156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:44:38.740156+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"408","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.703547+0000","last_change":"2021-02-10T04:38:39.247167+0000","last_active":"2021-02-10T04:44:57.703547+0000","last_peered":"2021-02-10T04:44:57.703547+0000","last_clean":"2021-02-10T04:44:57.703547+0000","last_became_active":"2021-02-10T04:38:39.246766+0000","last_became_peered":"2021-02-10T04:38:39.246766+0000","last_unstale":"2021-02-10T04:44:57.703547+0000","last_undegraded":"2021-02-10T04:44:57.703547+0000","last_fullsized":"2021-02-10T04:44:57.703547+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:20.757194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:20.757194+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"538'33","reported_seq":"1102","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.004930+0000","last_change":"2021-02-10T04:44:41.973978+0000","last_active":"2021-02-10T04:44:58.004930+0000","last_peered":"2021-02-10T04:44:58.004930+0000","last_clean":"2021-02-10T04:44:58.004930+0000","last_became_active":"2021-02-10T04:39:02.560739+0000","last_became_peered":"2021-02-10T04:39:02.560739+0000","last_unstale":"2021-02-10T04:44:58.004930+0000","last_undegraded":"2021-02-10T04:44:58.004930+0000","last_fullsized":"2021-02-10T04:44:58.004930+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'33","last_scrub_stamp":"2021-02-10T04:44:41.973895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:44:41.973895+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":67,"num_read_kb":50,"num_write":35,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"1119","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.011571+0000","last_change":"2021-02-10T04:37:37.351533+0000","last_active":"2021-02-10T04:44:58.011571+0000","last_peered":"2021-02-10T04:44:58.011571+0000","last_clean":"2021-02-10T04:44:58.011571+0000","last_became_active":"2021-02-10T04:37:37.238701+0000","last_became_peered":"2021-02-10T04:37:37.238701+0000","last_unstale":"2021-02-10T04:44:58.011571+0000","last_undegraded":"2021-02-10T04:44:58.011571+0000","last_fullsized":"2021-02-10T04:44:58.011571+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:33:49.760989+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:33:49.760989+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"984","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.700645+0000","last_change":"2021-02-10T04:29:37.364781+0000","last_active":"2021-02-10T04:44:57.700645+0000","last_peered":"2021-02-10T04:44:57.700645+0000","last_clean":"2021-02-10T04:44:57.700645+0000","last_became_active":"2021-02-10T04:25:53.047736+0000","last_became_peered":"2021-02-10T04:25:53.047736+0000","last_unstale":"2021-02-10T04:44:57.700645+0000","last_undegraded":"2021-02-10T04:44:57.700645+0000","last_fullsized":"2021-02-10T04:44:57.700645+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:37.364697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:37.364697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"1001'1528","reported_seq":"5679","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.699537+0000","last_change":"2021-02-10T04:44:26.666123+0000","last_active":"2021-02-10T04:44:57.699537+0000","last_peered":"2021-02-10T04:44:57.699537+0000","last_clean":"2021-02-10T04:44:57.699537+0000","last_became_active":"2021-02-10T04:37:48.215401+0000","last_became_peered":"2021-02-10T04:37:48.215401+0000","last_unstale":"2021-02-10T04:44:57.699537+0000","last_undegraded":"2021-02-10T04:44:57.699537+0000","last_fullsized":"2021-02-10T04:44:57.699537+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:44:26.665941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:44:26.665941+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":93888272,"num_objects":33,"num_object_clones":30,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":1,"num_read":716,"num_read_kb":703869,"num_write":572,"num_write_kb":222710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":42623785,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.15","version":"1000'446","reported_seq":"1951","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.365811+0000","last_change":"2021-02-10T04:44:30.654168+0000","last_active":"2021-02-10T04:44:58.365811+0000","last_peered":"2021-02-10T04:44:58.365811+0000","last_clean":"2021-02-10T04:44:58.365811+0000","last_became_active":"2021-02-10T04:44:30.653354+0000","last_became_peered":"2021-02-10T04:44:30.653354+0000","last_unstale":"2021-02-10T04:44:58.365811+0000","last_undegraded":"2021-02-10T04:44:58.365811+0000","last_fullsized":"2021-02-10T04:44:58.365811+0000","mapping_epoch":1012,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1013,"parent":"0.0","parent_split_bits":5,"last_scrub":"683'135","last_scrub_stamp":"2021-02-10T04:38:23.183462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:38:23.183462+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22255852,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":80,"num_read_kb":68436,"num_write":73,"num_write_kb":24345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":19387101,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"538'34","reported_seq":"1063","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.704795+0000","last_change":"2021-02-10T04:44:50.832167+0000","last_active":"2021-02-10T04:44:57.704795+0000","last_peered":"2021-02-10T04:44:57.704795+0000","last_clean":"2021-02-10T04:44:57.704795+0000","last_became_active":"2021-02-10T04:25:49.008942+0000","last_became_peered":"2021-02-10T04:25:49.008942+0000","last_unstale":"2021-02-10T04:44:57.704795+0000","last_undegraded":"2021-02-10T04:44:57.704795+0000","last_fullsized":"2021-02-10T04:44:57.704795+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'34","last_scrub_stamp":"2021-02-10T04:44:50.832088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:44:50.832088+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":71,"num_read_kb":49,"num_write":35,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"417","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.703201+0000","last_change":"2021-02-10T04:38:38.163311+0000","last_active":"2021-02-10T04:44:57.703201+0000","last_peered":"2021-02-10T04:44:57.703201+0000","last_clean":"2021-02-10T04:44:57.703201+0000","last_became_active":"2021-02-10T04:37:37.234649+0000","last_became_peered":"2021-02-10T04:37:37.234649+0000","last_unstale":"2021-02-10T04:44:57.703201+0000","last_undegraded":"2021-02-10T04:44:57.703201+0000","last_fullsized":"2021-02-10T04:44:57.703201+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:38:38.163225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:38.163225+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":247,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"991","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.701961+0000","last_change":"2021-02-10T04:44:39.599352+0000","last_active":"2021-02-10T04:44:57.701961+0000","last_peered":"2021-02-10T04:44:57.701961+0000","last_clean":"2021-02-10T04:44:57.701961+0000","last_became_active":"2021-02-10T04:25:51.030651+0000","last_became_peered":"2021-02-10T04:25:51.030651+0000","last_unstale":"2021-02-10T04:44:57.701961+0000","last_undegraded":"2021-02-10T04:44:57.701961+0000","last_fullsized":"2021-02-10T04:44:57.701961+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:39.599260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:44:39.599260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"398","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.006441+0000","last_change":"2021-02-10T04:37:37.238450+0000","last_active":"2021-02-10T04:44:58.006441+0000","last_peered":"2021-02-10T04:44:58.006441+0000","last_clean":"2021-02-10T04:44:58.006441+0000","last_became_active":"2021-02-10T04:37:37.238049+0000","last_became_peered":"2021-02-10T04:37:37.238049+0000","last_unstale":"2021-02-10T04:44:58.006441+0000","last_undegraded":"2021-02-10T04:44:58.006441+0000","last_fullsized":"2021-02-10T04:44:58.006441+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:28.776340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:28.776340+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"1000'620","reported_seq":"4199","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.700468+0000","last_change":"2021-02-10T04:44:30.704509+0000","last_active":"2021-02-10T04:44:57.700468+0000","last_peered":"2021-02-10T04:44:57.700468+0000","last_clean":"2021-02-10T04:44:57.700468+0000","last_became_active":"2021-02-10T04:44:30.702669+0000","last_became_peered":"2021-02-10T04:44:30.702669+0000","last_unstale":"2021-02-10T04:44:57.700468+0000","last_undegraded":"2021-02-10T04:44:57.700468+0000","last_fullsized":"2021-02-10T04:44:57.700468+0000","mapping_epoch":1012,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1013,"parent":"0.0","parent_split_bits":5,"last_scrub":"400'239","last_scrub_stamp":"2021-02-10T04:33:46.270478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:46.270478+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14265195,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":144,"num_read_kb":128483,"num_write":129,"num_write_kb":53444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"538'19","reported_seq":"1070","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.701909+0000","last_change":"2021-02-10T04:38:26.845289+0000","last_active":"2021-02-10T04:44:57.701909+0000","last_peered":"2021-02-10T04:44:57.701909+0000","last_clean":"2021-02-10T04:44:57.701909+0000","last_became_active":"2021-02-10T04:37:37.247396+0000","last_became_peered":"2021-02-10T04:37:37.247396+0000","last_unstale":"2021-02-10T04:44:57.701909+0000","last_undegraded":"2021-02-10T04:44:57.701909+0000","last_fullsized":"2021-02-10T04:44:57.701909+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'19","last_scrub_stamp":"2021-02-10T04:38:26.845195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:26.845195+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":19,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"1003","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.701416+0000","last_change":"2021-02-10T04:38:36.295733+0000","last_active":"2021-02-10T04:44:57.701416+0000","last_peered":"2021-02-10T04:44:57.701416+0000","last_clean":"2021-02-10T04:44:57.701416+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:44:57.701416+0000","last_undegraded":"2021-02-10T04:44:57.701416+0000","last_fullsized":"2021-02-10T04:44:57.701416+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:38:36.295599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:36.295599+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"988","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.364475+0000","last_change":"2021-02-10T04:29:18.429730+0000","last_active":"2021-02-10T04:44:58.364475+0000","last_peered":"2021-02-10T04:44:58.364475+0000","last_clean":"2021-02-10T04:44:58.364475+0000","last_became_active":"2021-02-10T04:25:51.031864+0000","last_became_peered":"2021-02-10T04:25:51.031864+0000","last_unstale":"2021-02-10T04:44:58.364475+0000","last_undegraded":"2021-02-10T04:44:58.364475+0000","last_fullsized":"2021-02-10T04:44:58.364475+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:18.429649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:18.429649+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"917","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.699444+0000","last_change":"2021-02-10T04:44:36.704043+0000","last_active":"2021-02-10T04:44:57.699444+0000","last_peered":"2021-02-10T04:44:57.699444+0000","last_clean":"2021-02-10T04:44:57.699444+0000","last_became_active":"2021-02-10T04:29:07.147249+0000","last_became_peered":"2021-02-10T04:29:07.147249+0000","last_unstale":"2021-02-10T04:44:57.699444+0000","last_undegraded":"2021-02-10T04:44:57.699444+0000","last_fullsized":"2021-02-10T04:44:57.699444+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:36.703969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:44:36.703969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.17","version":"1002'2042","reported_seq":"9181","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.364818+0000","last_change":"2021-02-10T04:44:30.658263+0000","last_active":"2021-02-10T04:44:58.364818+0000","last_peered":"2021-02-10T04:44:58.364818+0000","last_clean":"2021-02-10T04:44:58.364818+0000","last_became_active":"2021-02-10T04:44:30.657433+0000","last_became_peered":"2021-02-10T04:44:30.657433+0000","last_unstale":"2021-02-10T04:44:58.364818+0000","last_undegraded":"2021-02-10T04:44:58.364818+0000","last_fullsized":"2021-02-10T04:44:58.364818+0000","mapping_epoch":1012,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1013,"parent":"0.0","parent_split_bits":5,"last_scrub":"198'223","last_scrub_stamp":"2021-02-10T04:30:14.478639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:14.478639+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46300076,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":545,"num_read_kb":531689,"num_write":399,"num_write_kb":153087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":5403183,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"538'36","reported_seq":"1194","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.700838+0000","last_change":"2021-02-10T04:37:38.541093+0000","last_active":"2021-02-10T04:44:57.700838+0000","last_peered":"2021-02-10T04:44:57.700838+0000","last_clean":"2021-02-10T04:44:57.700838+0000","last_became_active":"2021-02-10T04:37:37.244921+0000","last_became_peered":"2021-02-10T04:37:37.244921+0000","last_unstale":"2021-02-10T04:44:57.700838+0000","last_undegraded":"2021-02-10T04:44:57.700838+0000","last_fullsized":"2021-02-10T04:44:57.700838+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:33:46.785905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:46.785905+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":36,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"84'7","reported_seq":"1017","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.363353+0000","last_change":"2021-02-10T04:30:12.538670+0000","last_active":"2021-02-10T04:44:58.363353+0000","last_peered":"2021-02-10T04:44:58.363353+0000","last_clean":"2021-02-10T04:44:58.363353+0000","last_became_active":"2021-02-10T04:25:36.334366+0000","last_became_peered":"2021-02-10T04:25:36.334366+0000","last_unstale":"2021-02-10T04:44:58.363353+0000","last_undegraded":"2021-02-10T04:44:58.363353+0000","last_fullsized":"2021-02-10T04:44:58.363353+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'7","last_scrub_stamp":"2021-02-10T04:30:12.538549+0000","last_deep_scrub":"84'7","last_deep_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_clean_scrub_stamp":"2021-02-10T04:30:12.538549+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"58'2","reported_seq":"1222","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:45:07.916337+0000","last_change":"2021-02-10T04:28:47.496534+0000","last_active":"2021-02-10T04:45:07.916337+0000","last_peered":"2021-02-10T04:45:07.916337+0000","last_clean":"2021-02-10T04:45:07.916337+0000","last_became_active":"2021-02-10T04:25:51.032328+0000","last_became_peered":"2021-02-10T04:25:51.032328+0000","last_unstale":"2021-02-10T04:45:07.916337+0000","last_undegraded":"2021-02-10T04:45:07.916337+0000","last_fullsized":"2021-02-10T04:45:07.916337+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:28:47.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:47.496468+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"990","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.704465+0000","last_change":"2021-02-10T04:38:39.191979+0000","last_active":"2021-02-10T04:44:57.704465+0000","last_peered":"2021-02-10T04:44:57.704465+0000","last_clean":"2021-02-10T04:44:57.704465+0000","last_became_active":"2021-02-10T04:25:53.045177+0000","last_became_peered":"2021-02-10T04:25:53.045177+0000","last_unstale":"2021-02-10T04:44:57.704465+0000","last_undegraded":"2021-02-10T04:44:57.704465+0000","last_fullsized":"2021-02-10T04:44:57.704465+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:39.191910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_clean_scrub_stamp":"2021-02-10T04:38:39.191910+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"1000'490","reported_seq":"3522","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.011219+0000","last_change":"2021-02-10T04:44:30.703533+0000","last_active":"2021-02-10T04:44:58.011219+0000","last_peered":"2021-02-10T04:44:58.011219+0000","last_clean":"2021-02-10T04:44:58.011219+0000","last_became_active":"2021-02-10T04:44:30.702606+0000","last_became_peered":"2021-02-10T04:44:30.702606+0000","last_unstale":"2021-02-10T04:44:58.011219+0000","last_undegraded":"2021-02-10T04:44:58.011219+0000","last_fullsized":"2021-02-10T04:44:58.011219+0000","mapping_epoch":1012,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1013,"parent":"0.0","parent_split_bits":5,"last_scrub":"197'36","last_scrub_stamp":"2021-02-10T04:30:13.863430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:13.863430+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19371627,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":142,"num_read_kb":148569,"num_write":86,"num_write_kb":31733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":11113379,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"538'25","reported_seq":"1043","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.365993+0000","last_change":"2021-02-10T04:44:44.580263+0000","last_active":"2021-02-10T04:44:58.365993+0000","last_peered":"2021-02-10T04:44:58.365993+0000","last_clean":"2021-02-10T04:44:58.365993+0000","last_became_active":"2021-02-10T04:25:49.010890+0000","last_became_peered":"2021-02-10T04:25:49.010890+0000","last_unstale":"2021-02-10T04:44:58.365993+0000","last_undegraded":"2021-02-10T04:44:58.365993+0000","last_fullsized":"2021-02-10T04:44:58.365993+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'25","last_scrub_stamp":"2021-02-10T04:44:44.580182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:44:44.580182+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":35,"num_write":26,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"1006","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.704272+0000","last_change":"2021-02-10T04:44:41.866217+0000","last_active":"2021-02-10T04:44:57.704272+0000","last_peered":"2021-02-10T04:44:57.704272+0000","last_clean":"2021-02-10T04:44:57.704272+0000","last_became_active":"2021-02-10T04:25:34.321096+0000","last_became_peered":"2021-02-10T04:25:34.321096+0000","last_unstale":"2021-02-10T04:44:57.704272+0000","last_undegraded":"2021-02-10T04:44:57.704272+0000","last_fullsized":"2021-02-10T04:44:57.704272+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:41.866132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:44:41.866132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"639'3","reported_seq":"1232","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.368444+0000","last_change":"2021-02-10T04:37:37.922249+0000","last_active":"2021-02-10T04:44:58.368444+0000","last_peered":"2021-02-10T04:44:58.368444+0000","last_clean":"2021-02-10T04:44:58.368444+0000","last_became_active":"2021-02-10T04:37:37.245428+0000","last_became_peered":"2021-02-10T04:37:37.245428+0000","last_unstale":"2021-02-10T04:44:58.368444+0000","last_undegraded":"2021-02-10T04:44:58.368444+0000","last_fullsized":"2021-02-10T04:44:58.368444+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:29:38.343834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.343834+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"993","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.011874+0000","last_change":"2021-02-10T04:37:37.238692+0000","last_active":"2021-02-10T04:44:58.011874+0000","last_peered":"2021-02-10T04:44:58.011874+0000","last_clean":"2021-02-10T04:44:58.011874+0000","last_became_active":"2021-02-10T04:37:37.238360+0000","last_became_peered":"2021-02-10T04:37:37.238360+0000","last_unstale":"2021-02-10T04:44:58.011874+0000","last_undegraded":"2021-02-10T04:44:58.011874+0000","last_fullsized":"2021-02-10T04:44:58.011874+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.934078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.934078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"1011'81","reported_seq":"747","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.701420+0000","last_change":"2021-02-10T04:44:27.690727+0000","last_active":"2021-02-10T04:44:57.701420+0000","last_peered":"2021-02-10T04:44:57.701420+0000","last_clean":"2021-02-10T04:44:57.701420+0000","last_became_active":"2021-02-10T04:44:27.540200+0000","last_became_peered":"2021-02-10T04:44:27.540200+0000","last_unstale":"2021-02-10T04:44:57.701420+0000","last_undegraded":"2021-02-10T04:44:57.701420+0000","last_fullsized":"2021-02-10T04:44:57.701420+0000","mapping_epoch":1009,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1010,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":81,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9722883,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":30326,"num_write":21,"num_write_kb":12444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3210022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"538'21","reported_seq":"756","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.700888+0000","last_change":"2021-02-10T04:44:55.705392+0000","last_active":"2021-02-10T04:44:57.700888+0000","last_peered":"2021-02-10T04:44:57.700888+0000","last_clean":"2021-02-10T04:44:57.700888+0000","last_became_active":"2021-02-10T04:37:37.249007+0000","last_became_peered":"2021-02-10T04:37:37.249007+0000","last_unstale":"2021-02-10T04:44:57.700888+0000","last_undegraded":"2021-02-10T04:44:57.700888+0000","last_fullsized":"2021-02-10T04:44:57.700888+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'21","last_scrub_stamp":"2021-02-10T04:44:55.705293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:44:55.705293+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"1001","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.701609+0000","last_change":"2021-02-10T04:38:26.127398+0000","last_active":"2021-02-10T04:44:57.701609+0000","last_peered":"2021-02-10T04:44:57.701609+0000","last_clean":"2021-02-10T04:44:57.701609+0000","last_became_active":"2021-02-10T04:25:34.330070+0000","last_became_peered":"2021-02-10T04:25:34.330070+0000","last_unstale":"2021-02-10T04:44:57.701609+0000","last_undegraded":"2021-02-10T04:44:57.701609+0000","last_fullsized":"2021-02-10T04:44:57.701609+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:26.127288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:26.127288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"991","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.700875+0000","last_change":"2021-02-10T04:44:45.525061+0000","last_active":"2021-02-10T04:44:57.700875+0000","last_peered":"2021-02-10T04:44:57.700875+0000","last_clean":"2021-02-10T04:44:57.700875+0000","last_became_active":"2021-02-10T04:25:51.029155+0000","last_became_peered":"2021-02-10T04:25:51.029155+0000","last_unstale":"2021-02-10T04:44:57.700875+0000","last_undegraded":"2021-02-10T04:44:57.700875+0000","last_fullsized":"2021-02-10T04:44:57.700875+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:45.524973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:44:45.524973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"990","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.703027+0000","last_change":"2021-02-10T04:33:49.338303+0000","last_active":"2021-02-10T04:44:57.703027+0000","last_peered":"2021-02-10T04:44:57.703027+0000","last_clean":"2021-02-10T04:44:57.703027+0000","last_became_active":"2021-02-10T04:25:53.039322+0000","last_became_peered":"2021-02-10T04:25:53.039322+0000","last_unstale":"2021-02-10T04:44:57.703027+0000","last_undegraded":"2021-02-10T04:44:57.703027+0000","last_fullsized":"2021-02-10T04:44:57.703027+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:49.338203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:33:49.338203+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"1000'1071","reported_seq":"4509","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.699008+0000","last_change":"2021-02-10T04:44:30.662526+0000","last_active":"2021-02-10T04:44:57.699008+0000","last_peered":"2021-02-10T04:44:57.699008+0000","last_clean":"2021-02-10T04:44:57.699008+0000","last_became_active":"2021-02-10T04:44:30.661975+0000","last_became_peered":"2021-02-10T04:44:30.661975+0000","last_unstale":"2021-02-10T04:44:57.699008+0000","last_undegraded":"2021-02-10T04:44:57.699008+0000","last_fullsized":"2021-02-10T04:44:57.699008+0000","mapping_epoch":1012,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1013,"parent":"0.0","parent_split_bits":5,"last_scrub":"412'357","last_scrub_stamp":"2021-02-10T04:33:50.582002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:50.582002+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23286368,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":242,"num_read_kb":226835,"num_write":185,"num_write_kb":76005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"538'23","reported_seq":"1027","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.006038+0000","last_change":"2021-02-10T04:38:32.221752+0000","last_active":"2021-02-10T04:44:58.006038+0000","last_peered":"2021-02-10T04:44:58.006038+0000","last_clean":"2021-02-10T04:44:58.006038+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:44:58.006038+0000","last_undegraded":"2021-02-10T04:44:58.006038+0000","last_fullsized":"2021-02-10T04:44:58.006038+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'23","last_scrub_stamp":"2021-02-10T04:38:32.221656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:32.221656+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"687","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.366038+0000","last_change":"2021-02-10T04:38:36.309455+0000","last_active":"2021-02-10T04:44:58.366038+0000","last_peered":"2021-02-10T04:44:58.366038+0000","last_clean":"2021-02-10T04:44:58.366038+0000","last_became_active":"2021-02-10T04:33:05.147956+0000","last_became_peered":"2021-02-10T04:33:05.147956+0000","last_unstale":"2021-02-10T04:44:58.366038+0000","last_undegraded":"2021-02-10T04:44:58.366038+0000","last_fullsized":"2021-02-10T04:44:58.366038+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:38:36.309362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:36.309362+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"671","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.363586+0000","last_change":"2021-02-10T04:33:05.150906+0000","last_active":"2021-02-10T04:44:58.363586+0000","last_peered":"2021-02-10T04:44:58.363586+0000","last_clean":"2021-02-10T04:44:58.363586+0000","last_became_active":"2021-02-10T04:33:05.150529+0000","last_became_peered":"2021-02-10T04:33:05.150529+0000","last_unstale":"2021-02-10T04:44:58.363586+0000","last_undegraded":"2021-02-10T04:44:58.363586+0000","last_fullsized":"2021-02-10T04:44:58.363586+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:33.026988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:29:33.026988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"984","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.700289+0000","last_change":"2021-02-10T04:29:24.421782+0000","last_active":"2021-02-10T04:44:57.700289+0000","last_peered":"2021-02-10T04:44:57.700289+0000","last_clean":"2021-02-10T04:44:57.700289+0000","last_became_active":"2021-02-10T04:25:53.043277+0000","last_became_peered":"2021-02-10T04:25:53.043277+0000","last_unstale":"2021-02-10T04:44:57.700289+0000","last_undegraded":"2021-02-10T04:44:57.700289+0000","last_fullsized":"2021-02-10T04:44:57.700289+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.421710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.421710+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"1002'1657","reported_seq":"4939","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.703884+0000","last_change":"2021-02-10T04:44:35.602106+0000","last_active":"2021-02-10T04:44:57.703884+0000","last_peered":"2021-02-10T04:44:57.703884+0000","last_clean":"2021-02-10T04:44:57.703884+0000","last_became_active":"2021-02-10T04:44:35.600743+0000","last_became_peered":"2021-02-10T04:44:35.600743+0000","last_unstale":"2021-02-10T04:44:57.703884+0000","last_undegraded":"2021-02-10T04:44:57.703884+0000","last_fullsized":"2021-02-10T04:44:57.703884+0000","mapping_epoch":1015,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1016,"parent":"0.0","parent_split_bits":5,"last_scrub":"687'1095","last_scrub_stamp":"2021-02-10T04:38:27.895743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:38:27.895743+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":42793067,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":424,"num_read_kb":437674,"num_write":319,"num_write_kb":118798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":17936938,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"538'16","reported_seq":"1147","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.698955+0000","last_change":"2021-02-10T04:38:22.919477+0000","last_active":"2021-02-10T04:44:57.698955+0000","last_peered":"2021-02-10T04:44:57.698955+0000","last_clean":"2021-02-10T04:44:57.698955+0000","last_became_active":"2021-02-10T04:26:39.005616+0000","last_became_peered":"2021-02-10T04:26:39.005616+0000","last_unstale":"2021-02-10T04:44:57.698955+0000","last_undegraded":"2021-02-10T04:44:57.698955+0000","last_fullsized":"2021-02-10T04:44:57.698955+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'16","last_scrub_stamp":"2021-02-10T04:38:22.919377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:22.919377+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":36,"num_read_kb":18,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"1011","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.364606+0000","last_change":"2021-02-10T04:44:50.573117+0000","last_active":"2021-02-10T04:44:58.364606+0000","last_peered":"2021-02-10T04:44:58.364606+0000","last_clean":"2021-02-10T04:44:58.364606+0000","last_became_active":"2021-02-10T04:37:37.229852+0000","last_became_peered":"2021-02-10T04:37:37.229852+0000","last_unstale":"2021-02-10T04:44:58.364606+0000","last_undegraded":"2021-02-10T04:44:58.364606+0000","last_fullsized":"2021-02-10T04:44:58.364606+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:50.573048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:44:50.573048+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"992","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.701332+0000","last_change":"2021-02-10T04:44:47.730663+0000","last_active":"2021-02-10T04:44:57.701332+0000","last_peered":"2021-02-10T04:44:57.701332+0000","last_clean":"2021-02-10T04:44:57.701332+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:44:57.701332+0000","last_undegraded":"2021-02-10T04:44:57.701332+0000","last_fullsized":"2021-02-10T04:44:57.701332+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:47.730571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:44:47.730571+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"401","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.004584+0000","last_change":"2021-02-10T04:37:37.244700+0000","last_active":"2021-02-10T04:44:58.004584+0000","last_peered":"2021-02-10T04:44:58.004584+0000","last_clean":"2021-02-10T04:44:58.004584+0000","last_became_active":"2021-02-10T04:37:37.244082+0000","last_became_peered":"2021-02-10T04:37:37.244082+0000","last_unstale":"2021-02-10T04:44:58.004584+0000","last_undegraded":"2021-02-10T04:44:58.004584+0000","last_fullsized":"2021-02-10T04:44:58.004584+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:31.824925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.824925+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.12","version":"998'1009","reported_seq":"4503","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.698927+0000","last_change":"2021-02-10T04:44:30.703845+0000","last_active":"2021-02-10T04:44:57.698927+0000","last_peered":"2021-02-10T04:44:57.698927+0000","last_clean":"2021-02-10T04:44:57.698927+0000","last_became_active":"2021-02-10T04:44:30.702334+0000","last_became_peered":"2021-02-10T04:44:30.702334+0000","last_unstale":"2021-02-10T04:44:57.698927+0000","last_undegraded":"2021-02-10T04:44:57.698927+0000","last_fullsized":"2021-02-10T04:44:57.698927+0000","mapping_epoch":1012,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1013,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24710414,"num_objects":11,"num_object_clones":10,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":239,"num_read_kb":249403,"num_write":155,"num_write_kb":50104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":45053856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"538'29","reported_seq":"1046","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.700750+0000","last_change":"2021-02-10T04:44:44.573935+0000","last_active":"2021-02-10T04:44:57.700750+0000","last_peered":"2021-02-10T04:44:57.700750+0000","last_clean":"2021-02-10T04:44:57.700750+0000","last_became_active":"2021-02-10T04:25:49.007049+0000","last_became_peered":"2021-02-10T04:25:49.007049+0000","last_unstale":"2021-02-10T04:44:57.700750+0000","last_undegraded":"2021-02-10T04:44:57.700750+0000","last_fullsized":"2021-02-10T04:44:57.700750+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'29","last_scrub_stamp":"2021-02-10T04:44:44.573821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:44:44.573821+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":29,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"1141","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.699637+0000","last_change":"2021-02-10T04:38:31.166652+0000","last_active":"2021-02-10T04:44:57.699637+0000","last_peered":"2021-02-10T04:44:57.699637+0000","last_clean":"2021-02-10T04:44:57.699637+0000","last_became_active":"2021-02-10T04:37:37.228840+0000","last_became_peered":"2021-02-10T04:37:37.228840+0000","last_unstale":"2021-02-10T04:44:57.699637+0000","last_undegraded":"2021-02-10T04:44:57.699637+0000","last_fullsized":"2021-02-10T04:44:57.699637+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:38:31.166586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:31.166586+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"340","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.699098+0000","last_change":"2021-02-10T04:44:54.669664+0000","last_active":"2021-02-10T04:44:57.699098+0000","last_peered":"2021-02-10T04:44:57.699098+0000","last_clean":"2021-02-10T04:44:57.699098+0000","last_became_active":"2021-02-10T04:38:39.246046+0000","last_became_peered":"2021-02-10T04:38:39.246046+0000","last_unstale":"2021-02-10T04:44:57.699098+0000","last_undegraded":"2021-02-10T04:44:57.699098+0000","last_fullsized":"2021-02-10T04:44:57.699098+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:54.669597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:44:54.669597+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"984","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.700811+0000","last_change":"2021-02-10T04:33:50.809533+0000","last_active":"2021-02-10T04:44:57.700811+0000","last_peered":"2021-02-10T04:44:57.700811+0000","last_clean":"2021-02-10T04:44:57.700811+0000","last_became_active":"2021-02-10T04:25:53.150845+0000","last_became_peered":"2021-02-10T04:25:53.150845+0000","last_unstale":"2021-02-10T04:44:57.700811+0000","last_undegraded":"2021-02-10T04:44:57.700811+0000","last_fullsized":"2021-02-10T04:44:57.700811+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:50.809468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_clean_scrub_stamp":"2021-02-10T04:33:50.809468+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"538'32","reported_seq":"1051","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.705521+0000","last_change":"2021-02-10T04:29:38.663351+0000","last_active":"2021-02-10T04:44:57.705521+0000","last_peered":"2021-02-10T04:44:57.705521+0000","last_clean":"2021-02-10T04:44:57.705521+0000","last_became_active":"2021-02-10T04:25:49.009177+0000","last_became_peered":"2021-02-10T04:25:49.009177+0000","last_unstale":"2021-02-10T04:44:57.705521+0000","last_undegraded":"2021-02-10T04:44:57.705521+0000","last_fullsized":"2021-02-10T04:44:57.705521+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2021-02-10T04:29:38.663272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.663272+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":60,"num_read_kb":42,"num_write":32,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"1011","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.362931+0000","last_change":"2021-02-10T04:44:38.636641+0000","last_active":"2021-02-10T04:44:58.362931+0000","last_peered":"2021-02-10T04:44:58.362931+0000","last_clean":"2021-02-10T04:44:58.362931+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:44:58.362931+0000","last_undegraded":"2021-02-10T04:44:58.362931+0000","last_fullsized":"2021-02-10T04:44:58.362931+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:38.636558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:44:38.636558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"58'2","reported_seq":"1225","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.365112+0000","last_change":"2021-02-10T04:44:50.716644+0000","last_active":"2021-02-10T04:44:58.365112+0000","last_peered":"2021-02-10T04:44:58.365112+0000","last_clean":"2021-02-10T04:44:58.365112+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:44:58.365112+0000","last_undegraded":"2021-02-10T04:44:58.365112+0000","last_fullsized":"2021-02-10T04:44:58.365112+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:44:50.716551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:44:50.716551+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"931","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.364377+0000","last_change":"2021-02-10T04:38:39.252110+0000","last_active":"2021-02-10T04:44:58.364377+0000","last_peered":"2021-02-10T04:44:58.364377+0000","last_clean":"2021-02-10T04:44:58.364377+0000","last_became_active":"2021-02-10T04:38:39.251336+0000","last_became_peered":"2021-02-10T04:38:39.251336+0000","last_unstale":"2021-02-10T04:44:58.364377+0000","last_undegraded":"2021-02-10T04:44:58.364377+0000","last_fullsized":"2021-02-10T04:44:58.364377+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:26.437591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.437591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"538'43","reported_seq":"1069","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.699729+0000","last_change":"2021-02-10T04:44:56.738861+0000","last_active":"2021-02-10T04:44:57.699729+0000","last_peered":"2021-02-10T04:44:57.699729+0000","last_clean":"2021-02-10T04:44:57.699729+0000","last_became_active":"2021-02-10T04:25:49.015760+0000","last_became_peered":"2021-02-10T04:25:49.015760+0000","last_unstale":"2021-02-10T04:44:57.699729+0000","last_undegraded":"2021-02-10T04:44:57.699729+0000","last_fullsized":"2021-02-10T04:44:57.699729+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'43","last_scrub_stamp":"2021-02-10T04:44:56.738770+0000","last_deep_scrub":"76'35","last_deep_scrub_stamp":"2021-02-10T04:26:58.198641+0000","last_clean_scrub_stamp":"2021-02-10T04:44:56.738770+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":62,"num_read_kb":42,"num_write":32,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"995","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.700425+0000","last_change":"2021-02-10T04:30:11.485059+0000","last_active":"2021-02-10T04:44:57.700425+0000","last_peered":"2021-02-10T04:44:57.700425+0000","last_clean":"2021-02-10T04:44:57.700425+0000","last_became_active":"2021-02-10T04:25:34.324981+0000","last_became_peered":"2021-02-10T04:25:34.324981+0000","last_unstale":"2021-02-10T04:44:57.700425+0000","last_undegraded":"2021-02-10T04:44:57.700425+0000","last_fullsized":"2021-02-10T04:44:57.700425+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:11.485004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.485004+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"1001","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.705657+0000","last_change":"2021-02-10T04:44:37.827187+0000","last_active":"2021-02-10T04:44:57.705657+0000","last_peered":"2021-02-10T04:44:57.705657+0000","last_clean":"2021-02-10T04:44:57.705657+0000","last_became_active":"2021-02-10T04:37:37.246998+0000","last_became_peered":"2021-02-10T04:37:37.246998+0000","last_unstale":"2021-02-10T04:44:57.705657+0000","last_undegraded":"2021-02-10T04:44:57.705657+0000","last_fullsized":"2021-02-10T04:44:57.705657+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:37.827101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:44:37.827101+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"671","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.366271+0000","last_change":"2021-02-10T04:33:05.142344+0000","last_active":"2021-02-10T04:44:58.366271+0000","last_peered":"2021-02-10T04:44:58.366271+0000","last_clean":"2021-02-10T04:44:58.366271+0000","last_became_active":"2021-02-10T04:33:05.141911+0000","last_became_peered":"2021-02-10T04:33:05.141911+0000","last_unstale":"2021-02-10T04:44:58.366271+0000","last_undegraded":"2021-02-10T04:44:58.366271+0000","last_fullsized":"2021-02-10T04:44:58.366271+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:25.075937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.075937+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"538'27","reported_seq":"961","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.700930+0000","last_change":"2021-02-10T04:44:46.552488+0000","last_active":"2021-02-10T04:44:57.700930+0000","last_peered":"2021-02-10T04:44:57.700930+0000","last_clean":"2021-02-10T04:44:57.700930+0000","last_became_active":"2021-02-10T04:37:37.257293+0000","last_became_peered":"2021-02-10T04:37:37.257293+0000","last_unstale":"2021-02-10T04:44:57.700930+0000","last_undegraded":"2021-02-10T04:44:57.700930+0000","last_fullsized":"2021-02-10T04:44:57.700930+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'27","last_scrub_stamp":"2021-02-10T04:44:46.552383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:44:46.552383+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"1004","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.699420+0000","last_change":"2021-02-10T04:38:32.193334+0000","last_active":"2021-02-10T04:44:57.699420+0000","last_peered":"2021-02-10T04:44:57.699420+0000","last_clean":"2021-02-10T04:44:57.699420+0000","last_became_active":"2021-02-10T04:25:36.336442+0000","last_became_peered":"2021-02-10T04:25:36.336442+0000","last_unstale":"2021-02-10T04:44:57.699420+0000","last_undegraded":"2021-02-10T04:44:57.699420+0000","last_fullsized":"2021-02-10T04:44:57.699420+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:38:32.193240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:32.193240+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"669","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.006948+0000","last_change":"2021-02-10T04:33:05.147757+0000","last_active":"2021-02-10T04:44:58.006948+0000","last_peered":"2021-02-10T04:44:58.006948+0000","last_clean":"2021-02-10T04:44:58.006948+0000","last_became_active":"2021-02-10T04:33:05.146838+0000","last_became_peered":"2021-02-10T04:33:05.146838+0000","last_unstale":"2021-02-10T04:44:58.006948+0000","last_undegraded":"2021-02-10T04:44:58.006948+0000","last_fullsized":"2021-02-10T04:44:58.006948+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:22.078247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:22.078247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"984","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.010837+0000","last_change":"2021-02-10T04:29:31.903091+0000","last_active":"2021-02-10T04:44:58.010837+0000","last_peered":"2021-02-10T04:44:58.010837+0000","last_clean":"2021-02-10T04:44:58.010837+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:44:58.010837+0000","last_undegraded":"2021-02-10T04:44:58.010837+0000","last_fullsized":"2021-02-10T04:44:58.010837+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:31.902996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:31.902996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"538'28","reported_seq":"1043","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.700516+0000","last_change":"2021-02-10T04:38:25.860322+0000","last_active":"2021-02-10T04:44:57.700516+0000","last_peered":"2021-02-10T04:44:57.700516+0000","last_clean":"2021-02-10T04:44:57.700516+0000","last_became_active":"2021-02-10T04:25:49.008997+0000","last_became_peered":"2021-02-10T04:25:49.008997+0000","last_unstale":"2021-02-10T04:44:57.700516+0000","last_undegraded":"2021-02-10T04:44:57.700516+0000","last_fullsized":"2021-02-10T04:44:57.700516+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'28","last_scrub_stamp":"2021-02-10T04:38:25.860247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:25.860247+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":40,"num_write":30,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"1012","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.700441+0000","last_change":"2021-02-10T04:44:36.642074+0000","last_active":"2021-02-10T04:44:57.700441+0000","last_peered":"2021-02-10T04:44:57.700441+0000","last_clean":"2021-02-10T04:44:57.700441+0000","last_became_active":"2021-02-10T04:25:36.336893+0000","last_became_peered":"2021-02-10T04:25:36.336893+0000","last_unstale":"2021-02-10T04:44:57.700441+0000","last_undegraded":"2021-02-10T04:44:57.700441+0000","last_fullsized":"2021-02-10T04:44:57.700441+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:36.641977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:26:46.425288+0000","last_clean_scrub_stamp":"2021-02-10T04:44:36.641977+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"995","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.006510+0000","last_change":"2021-02-10T04:37:37.243679+0000","last_active":"2021-02-10T04:44:58.006510+0000","last_peered":"2021-02-10T04:44:58.006510+0000","last_clean":"2021-02-10T04:44:58.006510+0000","last_became_active":"2021-02-10T04:37:37.243275+0000","last_became_peered":"2021-02-10T04:37:37.243275+0000","last_unstale":"2021-02-10T04:44:58.006510+0000","last_undegraded":"2021-02-10T04:44:58.006510+0000","last_fullsized":"2021-02-10T04:44:58.006510+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:52.643099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:33:52.643099+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"984","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.701869+0000","last_change":"2021-02-10T04:29:26.355726+0000","last_active":"2021-02-10T04:44:57.701869+0000","last_peered":"2021-02-10T04:44:57.701869+0000","last_clean":"2021-02-10T04:44:57.701869+0000","last_became_active":"2021-02-10T04:25:53.147831+0000","last_became_peered":"2021-02-10T04:25:53.147831+0000","last_unstale":"2021-02-10T04:44:57.701869+0000","last_undegraded":"2021-02-10T04:44:57.701869+0000","last_fullsized":"2021-02-10T04:44:57.701869+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:26.355652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:26.355652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"1003'739","reported_seq":"3695","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.697853+0000","last_change":"2021-02-10T04:44:30.581309+0000","last_active":"2021-02-10T04:44:57.697853+0000","last_peered":"2021-02-10T04:44:57.697853+0000","last_clean":"2021-02-10T04:44:57.697853+0000","last_became_active":"2021-02-10T04:44:30.579847+0000","last_became_peered":"2021-02-10T04:44:30.579847+0000","last_unstale":"2021-02-10T04:44:57.697853+0000","last_undegraded":"2021-02-10T04:44:57.697853+0000","last_fullsized":"2021-02-10T04:44:57.697853+0000","mapping_epoch":1012,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1013,"parent":"0.0","parent_split_bits":5,"last_scrub":"154'63","last_scrub_stamp":"2021-02-10T04:29:39.700115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:29:39.700115+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24059889,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":130,"num_read_kb":109544,"num_write":102,"num_write_kb":46174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":18429397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"538'31","reported_seq":"1058","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.005283+0000","last_change":"2021-02-10T04:44:38.877308+0000","last_active":"2021-02-10T04:44:58.005283+0000","last_peered":"2021-02-10T04:44:58.005283+0000","last_clean":"2021-02-10T04:44:58.005283+0000","last_became_active":"2021-02-10T04:25:49.000911+0000","last_became_peered":"2021-02-10T04:25:49.000911+0000","last_unstale":"2021-02-10T04:44:58.005283+0000","last_undegraded":"2021-02-10T04:44:58.005283+0000","last_fullsized":"2021-02-10T04:44:58.005283+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'31","last_scrub_stamp":"2021-02-10T04:44:38.877241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:44:38.877241+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":62,"num_read_kb":46,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"922","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.364637+0000","last_change":"2021-02-10T04:37:37.242121+0000","last_active":"2021-02-10T04:44:58.364637+0000","last_peered":"2021-02-10T04:44:58.364637+0000","last_clean":"2021-02-10T04:44:58.364637+0000","last_became_active":"2021-02-10T04:37:37.241751+0000","last_became_peered":"2021-02-10T04:37:37.241751+0000","last_unstale":"2021-02-10T04:44:58.364637+0000","last_undegraded":"2021-02-10T04:44:58.364637+0000","last_fullsized":"2021-02-10T04:44:58.364637+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:48.646802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:33:48.646802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"1002","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.363981+0000","last_change":"2021-02-10T04:44:51.575527+0000","last_active":"2021-02-10T04:44:58.363981+0000","last_peered":"2021-02-10T04:44:58.363981+0000","last_clean":"2021-02-10T04:44:58.363981+0000","last_became_active":"2021-02-10T04:38:39.253122+0000","last_became_peered":"2021-02-10T04:38:39.253122+0000","last_unstale":"2021-02-10T04:44:58.363981+0000","last_undegraded":"2021-02-10T04:44:58.363981+0000","last_fullsized":"2021-02-10T04:44:58.363981+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:51.575462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:44:51.575462+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"990","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.703467+0000","last_change":"2021-02-10T04:38:37.121756+0000","last_active":"2021-02-10T04:44:57.703467+0000","last_peered":"2021-02-10T04:44:57.703467+0000","last_clean":"2021-02-10T04:44:57.703467+0000","last_became_active":"2021-02-10T04:25:53.039844+0000","last_became_peered":"2021-02-10T04:25:53.039844+0000","last_unstale":"2021-02-10T04:44:57.703467+0000","last_undegraded":"2021-02-10T04:44:57.703467+0000","last_fullsized":"2021-02-10T04:44:57.703467+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:37.121654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:38:37.121654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"259'74","reported_seq":"28","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.702927+0000","last_change":"2021-02-10T04:44:27.547465+0000","last_active":"2021-02-10T04:44:57.702927+0000","last_peered":"2021-02-10T04:44:57.702927+0000","last_clean":"2021-02-10T04:44:57.702927+0000","last_became_active":"2021-02-10T04:44:27.546590+0000","last_became_peered":"2021-02-10T04:44:27.546590+0000","last_unstale":"2021-02-10T04:44:57.702927+0000","last_undegraded":"2021-02-10T04:44:57.702927+0000","last_fullsized":"2021-02-10T04:44:57.702927+0000","mapping_epoch":1009,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1010,"parent":"0.0","parent_split_bits":5,"last_scrub":"191'62","last_scrub_stamp":"2021-02-10T04:30:10.905905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:10.905905+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1218524,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":28228,"num_write":19,"num_write_kb":7022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"538'14","reported_seq":"1018","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.005838+0000","last_change":"2021-02-10T04:30:13.089576+0000","last_active":"2021-02-10T04:44:58.005838+0000","last_peered":"2021-02-10T04:44:58.005838+0000","last_clean":"2021-02-10T04:44:58.005838+0000","last_became_active":"2021-02-10T04:25:49.013196+0000","last_became_peered":"2021-02-10T04:25:49.013196+0000","last_unstale":"2021-02-10T04:44:58.005838+0000","last_undegraded":"2021-02-10T04:44:58.005838+0000","last_fullsized":"2021-02-10T04:44:58.005838+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2021-02-10T04:30:13.089494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:30:13.089494+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":15,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"912","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.699135+0000","last_change":"2021-02-10T04:30:11.537208+0000","last_active":"2021-02-10T04:44:57.699135+0000","last_peered":"2021-02-10T04:44:57.699135+0000","last_clean":"2021-02-10T04:44:57.699135+0000","last_became_active":"2021-02-10T04:29:07.157944+0000","last_became_peered":"2021-02-10T04:29:07.157944+0000","last_unstale":"2021-02-10T04:44:57.699135+0000","last_undegraded":"2021-02-10T04:44:57.699135+0000","last_fullsized":"2021-02-10T04:44:57.699135+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:11.537136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.537136+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"906","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.005390+0000","last_change":"2021-02-10T04:29:07.146984+0000","last_active":"2021-02-10T04:44:58.005390+0000","last_peered":"2021-02-10T04:44:58.005390+0000","last_clean":"2021-02-10T04:44:58.005390+0000","last_became_active":"2021-02-10T04:29:07.146525+0000","last_became_peered":"2021-02-10T04:29:07.146525+0000","last_unstale":"2021-02-10T04:44:58.005390+0000","last_undegraded":"2021-02-10T04:44:58.005390+0000","last_fullsized":"2021-02-10T04:44:58.005390+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:46.493066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:28:46.493066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"986","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.363489+0000","last_change":"2021-02-10T04:29:22.475613+0000","last_active":"2021-02-10T04:44:58.363489+0000","last_peered":"2021-02-10T04:44:58.363489+0000","last_clean":"2021-02-10T04:44:58.363489+0000","last_became_active":"2021-02-10T04:25:53.042601+0000","last_became_peered":"2021-02-10T04:25:53.042601+0000","last_unstale":"2021-02-10T04:44:58.363489+0000","last_undegraded":"2021-02-10T04:44:58.363489+0000","last_fullsized":"2021-02-10T04:44:58.363489+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:22.475504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:22.475504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"538'38","reported_seq":"1051","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.004577+0000","last_change":"2021-02-10T04:28:35.520489+0000","last_active":"2021-02-10T04:44:58.004577+0000","last_peered":"2021-02-10T04:44:58.004577+0000","last_clean":"2021-02-10T04:44:58.004577+0000","last_became_active":"2021-02-10T04:25:49.010481+0000","last_became_peered":"2021-02-10T04:25:49.010481+0000","last_unstale":"2021-02-10T04:44:58.004577+0000","last_undegraded":"2021-02-10T04:44:58.004577+0000","last_fullsized":"2021-02-10T04:44:58.004577+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'32","last_scrub_stamp":"2021-02-10T04:28:35.520410+0000","last_deep_scrub":"78'32","last_deep_scrub_stamp":"2021-02-10T04:27:09.170428+0000","last_clean_scrub_stamp":"2021-02-10T04:28:35.520410+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":56,"num_read_kb":37,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"995","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.701319+0000","last_change":"2021-02-10T04:29:29.355356+0000","last_active":"2021-02-10T04:44:57.701319+0000","last_peered":"2021-02-10T04:44:57.701319+0000","last_clean":"2021-02-10T04:44:57.701319+0000","last_became_active":"2021-02-10T04:25:34.327162+0000","last_became_peered":"2021-02-10T04:25:34.327162+0000","last_unstale":"2021-02-10T04:44:57.701319+0000","last_undegraded":"2021-02-10T04:44:57.701319+0000","last_fullsized":"2021-02-10T04:44:57.701319+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:29.355257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:29:29.355257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"676","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.363272+0000","last_change":"2021-02-10T04:44:45.577005+0000","last_active":"2021-02-10T04:44:58.363272+0000","last_peered":"2021-02-10T04:44:58.363272+0000","last_clean":"2021-02-10T04:44:58.363272+0000","last_became_active":"2021-02-10T04:33:05.169159+0000","last_became_peered":"2021-02-10T04:33:05.169159+0000","last_unstale":"2021-02-10T04:44:58.363272+0000","last_undegraded":"2021-02-10T04:44:58.363272+0000","last_fullsized":"2021-02-10T04:44:58.363272+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:45.576804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:44:45.576804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"1070","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.005241+0000","last_change":"2021-02-10T04:38:23.319098+0000","last_active":"2021-02-10T04:44:58.005241+0000","last_peered":"2021-02-10T04:44:58.005241+0000","last_clean":"2021-02-10T04:44:58.005241+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:44:58.005241+0000","last_undegraded":"2021-02-10T04:44:58.005241+0000","last_fullsized":"2021-02-10T04:44:58.005241+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:23.319023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:38:23.319023+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"538'56","reported_seq":"1144","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.700838+0000","last_change":"2021-02-10T04:38:21.961539+0000","last_active":"2021-02-10T04:44:57.700838+0000","last_peered":"2021-02-10T04:44:57.700838+0000","last_clean":"2021-02-10T04:44:57.700838+0000","last_became_active":"2021-02-10T04:37:37.266384+0000","last_became_peered":"2021-02-10T04:37:37.266384+0000","last_unstale":"2021-02-10T04:44:57.700838+0000","last_undegraded":"2021-02-10T04:44:57.700838+0000","last_fullsized":"2021-02-10T04:44:57.700838+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":641,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'56","last_scrub_stamp":"2021-02-10T04:38:21.961470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:21.961470+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":107,"num_read_kb":84,"num_write":57,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":890,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"1010","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.704757+0000","last_change":"2021-02-10T04:44:39.845478+0000","last_active":"2021-02-10T04:44:57.704757+0000","last_peered":"2021-02-10T04:44:57.704757+0000","last_clean":"2021-02-10T04:44:57.704757+0000","last_became_active":"2021-02-10T04:25:36.336112+0000","last_became_peered":"2021-02-10T04:25:36.336112+0000","last_unstale":"2021-02-10T04:44:57.704757+0000","last_undegraded":"2021-02-10T04:44:57.704757+0000","last_fullsized":"2021-02-10T04:44:57.704757+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:39.845409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:44:39.845409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"335","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.698853+0000","last_change":"2021-02-10T04:38:39.244331+0000","last_active":"2021-02-10T04:44:57.698853+0000","last_peered":"2021-02-10T04:44:57.698853+0000","last_clean":"2021-02-10T04:44:57.698853+0000","last_became_active":"2021-02-10T04:38:39.243965+0000","last_became_peered":"2021-02-10T04:38:39.243965+0000","last_unstale":"2021-02-10T04:44:57.698853+0000","last_undegraded":"2021-02-10T04:44:57.698853+0000","last_fullsized":"2021-02-10T04:44:57.698853+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:33.199521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:38:33.199521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"984","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.700323+0000","last_change":"2021-02-10T04:29:25.382883+0000","last_active":"2021-02-10T04:44:57.700323+0000","last_peered":"2021-02-10T04:44:57.700323+0000","last_clean":"2021-02-10T04:44:57.700323+0000","last_became_active":"2021-02-10T04:25:53.150917+0000","last_became_peered":"2021-02-10T04:25:53.150917+0000","last_unstale":"2021-02-10T04:44:57.700323+0000","last_undegraded":"2021-02-10T04:44:57.700323+0000","last_fullsized":"2021-02-10T04:44:57.700323+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:25.382793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:25.382793+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.20","version":"0'0","reported_seq":"976","reported_epoch":"1026","state":"active+clean","last_fresh":"2021-02-10T04:44:58.363167+0000","last_change":"2021-02-10T04:30:11.492843+0000","last_active":"2021-02-10T04:44:58.363167+0000","last_peered":"2021-02-10T04:44:58.363167+0000","last_clean":"2021-02-10T04:44:58.363167+0000","last_became_active":"2021-02-10T04:29:07.148937+0000","last_became_peered":"2021-02-10T04:29:07.148937+0000","last_unstale":"2021-02-10T04:44:58.363167+0000","last_undegraded":"2021-02-10T04:44:58.363167+0000","last_fullsized":"2021-02-10T04:44:58.363167+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:11.492788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:41.375276+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.492788+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":"488","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.703479+0000","last_change":"2021-02-10T04:37:37.247286+0000","last_active":"2021-02-10T04:44:57.703479+0000","last_peered":"2021-02-10T04:44:57.703479+0000","last_clean":"2021-02-10T04:44:57.703479+0000","last_became_active":"2021-02-10T04:37:37.246865+0000","last_became_peered":"2021-02-10T04:37:37.246865+0000","last_unstale":"2021-02-10T04:44:57.703479+0000","last_undegraded":"2021-02-10T04:44:57.703479+0000","last_fullsized":"2021-02-10T04:44:57.703479+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:21.708701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:21.708701+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":"1010","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.699136+0000","last_change":"2021-02-10T04:33:58.892594+0000","last_active":"2021-02-10T04:44:57.699136+0000","last_peered":"2021-02-10T04:44:57.699136+0000","last_clean":"2021-02-10T04:44:57.699136+0000","last_became_active":"2021-02-10T04:28:23.129832+0000","last_became_peered":"2021-02-10T04:28:23.129832+0000","last_unstale":"2021-02-10T04:44:57.699136+0000","last_undegraded":"2021-02-10T04:44:57.699136+0000","last_fullsized":"2021-02-10T04:44:57.699136+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:58.892504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:33:58.892504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":"954","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.700058+0000","last_change":"2021-02-10T04:44:51.620170+0000","last_active":"2021-02-10T04:44:57.700058+0000","last_peered":"2021-02-10T04:44:57.700058+0000","last_clean":"2021-02-10T04:44:57.700058+0000","last_became_active":"2021-02-10T04:28:23.129523+0000","last_became_peered":"2021-02-10T04:28:23.129523+0000","last_unstale":"2021-02-10T04:44:57.700058+0000","last_undegraded":"2021-02-10T04:44:57.700058+0000","last_fullsized":"2021-02-10T04:44:57.700058+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:51.620102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:44:51.620102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":"954","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.699906+0000","last_change":"2021-02-10T04:34:01.895599+0000","last_active":"2021-02-10T04:44:57.699906+0000","last_peered":"2021-02-10T04:44:57.699906+0000","last_clean":"2021-02-10T04:44:57.699906+0000","last_became_active":"2021-02-10T04:28:23.126368+0000","last_became_peered":"2021-02-10T04:28:23.126368+0000","last_unstale":"2021-02-10T04:44:57.699906+0000","last_undegraded":"2021-02-10T04:44:57.699906+0000","last_fullsized":"2021-02-10T04:44:57.699906+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:34:01.895514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:34:01.895514+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.25","version":"53'2","reported_seq":"971","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.006697+0000","last_change":"2021-02-10T04:44:39.917303+0000","last_active":"2021-02-10T04:44:58.006697+0000","last_peered":"2021-02-10T04:44:58.006697+0000","last_clean":"2021-02-10T04:44:58.006697+0000","last_became_active":"2021-02-10T04:28:23.467770+0000","last_became_peered":"2021-02-10T04:28:23.467770+0000","last_unstale":"2021-02-10T04:44:58.006697+0000","last_undegraded":"2021-02-10T04:44:58.006697+0000","last_fullsized":"2021-02-10T04:44:58.006697+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:44:39.917235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:44:39.917235+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.26","version":"53'1","reported_seq":"411","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.010676+0000","last_change":"2021-02-10T04:37:37.233712+0000","last_active":"2021-02-10T04:44:58.010676+0000","last_peered":"2021-02-10T04:44:58.010676+0000","last_clean":"2021-02-10T04:44:58.010676+0000","last_became_active":"2021-02-10T04:37:37.231081+0000","last_became_peered":"2021-02-10T04:37:37.231081+0000","last_unstale":"2021-02-10T04:44:58.010676+0000","last_undegraded":"2021-02-10T04:44:58.010676+0000","last_fullsized":"2021-02-10T04:44:58.010676+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:29:22.801929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:22.801929+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":423,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":"982","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:58.011422+0000","last_change":"2021-02-10T04:38:37.219069+0000","last_active":"2021-02-10T04:44:58.011422+0000","last_peered":"2021-02-10T04:44:58.011422+0000","last_clean":"2021-02-10T04:44:58.011422+0000","last_became_active":"2021-02-10T04:29:07.145907+0000","last_became_peered":"2021-02-10T04:29:07.145907+0000","last_unstale":"2021-02-10T04:44:58.011422+0000","last_undegraded":"2021-02-10T04:44:58.011422+0000","last_fullsized":"2021-02-10T04:44:58.011422+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:37.218968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:37.218968+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":"949","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.699346+0000","last_change":"2021-02-10T04:29:23.440451+0000","last_active":"2021-02-10T04:44:57.699346+0000","last_peered":"2021-02-10T04:44:57.699346+0000","last_clean":"2021-02-10T04:44:57.699346+0000","last_became_active":"2021-02-10T04:28:23.129469+0000","last_became_peered":"2021-02-10T04:28:23.129469+0000","last_unstale":"2021-02-10T04:44:57.699346+0000","last_undegraded":"2021-02-10T04:44:57.699346+0000","last_fullsized":"2021-02-10T04:44:57.699346+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.440391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.440391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":"954","reported_epoch":"1025","state":"active+clean","last_fresh":"2021-02-10T04:44:57.701089+0000","last_change":"2021-02-10T04:38:22.136377+0000","last_active":"2021-02-10T04:44:57.701089+0000","last_peered":"2021-02-10T04:44:57.701089+0000","last_clean":"2021-02-10T04:44:57.701089+0000","last_became_active":"2021-02-10T04:28:23.126795+0000","last_became_peered":"2021-02-10T04:28:23.126795+0000","last_unstale":"2021-02-10T04:44:57.701089+0000","last_undegraded":"2021-02-10T04:44:57.701089+0000","last_fullsized":"2021-02-10T04:44:57.701089+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:22.136290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_clean_scrub_stamp":"2021-02-10T04:38:22.136290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":906723459,"num_objects":372,"num_object_clones":322,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":372,"num_whiteouts":11,"num_read":7459,"num_read_kb":7586722,"num_write":5447,"num_write_kb":2093536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":479,"num_bytes_recovered":202275572,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1923244032,"data_stored":2569127200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23648,"ondisk_log_size":23648,"up":78,"acting":78,"num_store_stats":8},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":1945,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":18,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":6565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":126,"acting":126,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":453,"num_read_kb":448,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":1876,"num_read_kb":1349,"num_write":1020,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":8770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":75722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1068,"ondisk_log_size":1068,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21,"ondisk_log_size":21,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365036,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":288736,"kb_used_data":239192,"kb_used_omap":0,"kb_used_meta":49536,"kb_avail":93456416,"statfs":{"total":95995035648,"available":95699369984,"internally_reserved":0,"allocated":244932608,"data_stored":339064642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":50724864},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:44:05 2021","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":1.6719999999999999,"15min":1.4690000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.46700000000000003},"max":{"1min":1.48,"5min":67.277000000000001,"15min":72.805999999999997},"last":0.65500000000000003},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.96,"15min":1.7050000000000001},"min":{"1min":0.59699999999999998,"5min":0.54200000000000004,"15min":0.50800000000000001},"max":{"1min":1.9059999999999999,"5min":66.933999999999997,"15min":72.721999999999994},"last":0.68200000000000005}]},{"osd":1,"last update":"Wed Feb 10 04:44:05 2021","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.121,"15min":1.1499999999999999},"min":{"1min":0.58099999999999996,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":2.5030000000000001,"5min":10.140000000000001,"15min":41.966000000000001},"last":1.4730000000000001},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.99399999999999999,"15min":1.127},"min":{"1min":0.64900000000000002,"5min":0.58899999999999997,"15min":0.56499999999999995},"max":{"1min":1.548,"5min":7.1520000000000001,"15min":41.804000000000002},"last":1.641}]},{"osd":2,"last update":"Wed Feb 10 04:44:05 2021","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.784,"15min":2.1259999999999999},"min":{"1min":0.79000000000000004,"5min":0.628,"15min":0.58299999999999996},"max":{"1min":1.6060000000000001,"5min":66.534000000000006,"15min":125.714},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.1910000000000001,"5min":2.1190000000000002,"15min":2.3639999999999999},"min":{"1min":0.68700000000000006,"5min":0.66900000000000004,"15min":0.59099999999999997},"max":{"1min":1.9330000000000001,"5min":66.781999999999996,"15min":125.53700000000001},"last":1.2030000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:44:05 2021","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":2.2210000000000001,"15min":2.3330000000000002},"min":{"1min":0.75700000000000001,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":3.4790000000000001,"5min":68.305000000000007,"15min":125.625},"last":1.6839999999999999},{"interface":"front","average":{"1min":1.226,"5min":2.0609999999999999,"15min":2.2789999999999999},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.72199999999999998},"max":{"1min":1.7270000000000001,"5min":69.331999999999994,"15min":125.598},"last":1.7509999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:44:05 2021","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.534,"15min":1.712},"min":{"1min":0.82799999999999996,"5min":0.70699999999999996,"15min":0.64600000000000002},"max":{"1min":3.1920000000000002,"5min":10.781000000000001,"15min":97.733999999999995},"last":1.361},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.4530000000000001,"15min":1.6599999999999999},"min":{"1min":0.73399999999999999,"5min":0.68100000000000005,"15min":0.65100000000000002},"max":{"1min":3.258,"5min":10.326000000000001,"15min":97.691000000000003},"last":1.988}]},{"osd":5,"last update":"Wed Feb 10 04:44:05 2021","interfaces":[{"interface":"back","average":{"1min":1.581,"5min":1.8420000000000001,"15min":1.6220000000000001},"min":{"1min":0.88400000000000001,"5min":0.76000000000000001,"15min":0.66700000000000004},"max":{"1min":3.387,"5min":35.283000000000001,"15min":35.283000000000001},"last":1.8799999999999999},{"interface":"front","average":{"1min":1.758,"5min":2.181,"15min":1.9199999999999999},"min":{"1min":1.1579999999999999,"5min":0.86099999999999999,"15min":0.753},"max":{"1min":2.7490000000000001,"5min":35.353999999999999,"15min":35.353999999999999},"last":1.8340000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:44:05 2021","interfaces":[{"interface":"back","average":{"1min":1.865,"5min":1.9730000000000001,"15min":1.996},"min":{"1min":1.123,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":5.3620000000000001,"5min":11.343999999999999,"15min":32.616},"last":1.7230000000000001},{"interface":"front","average":{"1min":1.823,"5min":1.8680000000000001,"15min":1.881},"min":{"1min":1.0489999999999999,"5min":0.99299999999999999,"15min":0.79500000000000004},"max":{"1min":3.5939999999999999,"5min":10.428000000000001,"15min":32.424999999999997},"last":2.0950000000000002}]}]},{"osd":6,"up_from":43,"seq":184683593968,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":361520,"kb_used_data":310312,"kb_used_omap":0,"kb_used_meta":51200,"kb_avail":93383632,"statfs":{"total":95995035648,"available":95624839168,"internally_reserved":0,"allocated":317759488,"data_stored":405500616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52428800},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:44:59 2021","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":1.091,"15min":1.387},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.38800000000000001},"max":{"1min":1.627,"5min":13.393000000000001,"15min":64.164000000000001},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.1319999999999999,"15min":1.762},"min":{"1min":0.57499999999999996,"5min":0.50900000000000001,"15min":0.47399999999999998},"max":{"1min":2.0590000000000002,"5min":8.0289999999999999,"15min":64.277000000000001},"last":0.84899999999999998}]},{"osd":1,"last update":"Wed Feb 10 04:44:59 2021","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.1539999999999999,"15min":1.0660000000000001},"min":{"1min":0.67300000000000004,"5min":0.54400000000000004,"15min":0.443},"max":{"1min":1.8460000000000001,"5min":12.978999999999999,"15min":12.978999999999999},"last":1.052},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.121,"15min":1.093},"min":{"1min":0.65100000000000002,"5min":0.60899999999999999,"15min":0.57799999999999996},"max":{"1min":1.867,"5min":3.4199999999999999,"15min":6.0960000000000001},"last":1.3100000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:44:59 2021","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.254,"15min":1.349},"min":{"1min":0.71399999999999997,"5min":0.55600000000000005,"15min":0.54100000000000004},"max":{"1min":1.9910000000000001,"5min":2.6840000000000002,"15min":24.725000000000001},"last":1.5289999999999999},{"interface":"front","average":{"1min":1.23,"5min":1.2689999999999999,"15min":1.3440000000000001},"min":{"1min":0.61799999999999999,"5min":0.59899999999999998,"15min":0.55300000000000005},"max":{"1min":1.962,"5min":2.2330000000000001,"15min":24.611999999999998},"last":1.2889999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:44:59 2021","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.429,"15min":1.883},"min":{"1min":0.79800000000000004,"5min":0.76400000000000001,"15min":0.63200000000000001},"max":{"1min":2.2410000000000001,"5min":4.1779999999999999,"15min":114.17700000000001},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.425,"5min":1.5489999999999999,"15min":1.913},"min":{"1min":0.745,"5min":0.745,"15min":0.71699999999999997},"max":{"1min":2.113,"5min":13.106,"15min":116.929},"last":1.4530000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:44:59 2021","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.464,"15min":1.4059999999999999},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.60599999999999998},"max":{"1min":2.02,"5min":4.0810000000000004,"15min":7.5389999999999997},"last":1.698},{"interface":"front","average":{"1min":1.383,"5min":1.153,"15min":1.0800000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.57999999999999996},"max":{"1min":5.0640000000000001,"5min":5.0640000000000001,"15min":5.0640000000000001},"last":2.379}]},{"osd":5,"last update":"Wed Feb 10 04:44:59 2021","interfaces":[{"interface":"back","average":{"1min":6.0289999999999999,"5min":2.254,"15min":1.637},"min":{"1min":0.90300000000000002,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":81.540000000000006,"5min":81.540000000000006,"15min":81.540000000000006},"last":2.262},{"interface":"front","average":{"1min":6.1120000000000001,"5min":2.512,"15min":1.841},"min":{"1min":0.94099999999999995,"5min":0.78000000000000003,"15min":0.66600000000000004},"max":{"1min":81.492999999999995,"5min":81.492999999999995,"15min":81.492999999999995},"last":1.556}]},{"osd":7,"last update":"Wed Feb 10 04:44:18 2021","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.347,"15min":1.325},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.755},"max":{"1min":2.052,"5min":3.117,"15min":4.5279999999999996},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.7210000000000001,"5min":1.7609999999999999,"15min":1.6240000000000001},"min":{"1min":1.198,"5min":0.91000000000000003,"15min":0.77500000000000002},"max":{"1min":2.327,"5min":13.315,"15min":13.315},"last":1.7190000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575106,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":324868,"kb_used_data":276028,"kb_used_omap":0,"kb_used_meta":48832,"kb_avail":93420284,"statfs":{"total":95995035648,"available":95662370816,"internally_reserved":0,"allocated":282652672,"data_stored":363830089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":50003968},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:44:27 2021","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.847,"15min":1.581},"min":{"1min":0.53000000000000003,"5min":0.50600000000000001,"15min":0.502},"max":{"1min":1.823,"5min":55.475999999999999,"15min":55.475999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.95699999999999996,"5min":1.923,"15min":1.6639999999999999},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":2.9990000000000001,"5min":55.368000000000002,"15min":55.368000000000002},"last":1.294}]},{"osd":2,"last update":"Wed Feb 10 04:44:50 2021","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.8089999999999999,"15min":1.5649999999999999},"min":{"1min":0.59199999999999997,"5min":0.57499999999999996,"15min":0.49399999999999999},"max":{"1min":1.508,"5min":32.110999999999997,"15min":40.268000000000001},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.839,"15min":1.6040000000000001},"min":{"1min":0.64300000000000002,"5min":0.59899999999999998,"15min":0.46800000000000003},"max":{"1min":1.827,"5min":32.302999999999997,"15min":40.651000000000003},"last":0.65200000000000002}]},{"osd":3,"last update":"Wed Feb 10 04:44:04 2021","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":2.0209999999999999,"15min":1.76},"min":{"1min":0.76100000000000001,"5min":0.57899999999999996,"15min":0.45200000000000001},"max":{"1min":2.4969999999999999,"5min":32.432000000000002,"15min":40.314},"last":1.0489999999999999},{"interface":"front","average":{"1min":1.234,"5min":1.9910000000000001,"15min":1.746},"min":{"1min":0.64600000000000002,"5min":0.60399999999999998,"15min":0.47199999999999998},"max":{"1min":2.702,"5min":32.811,"15min":40.973999999999997},"last":1.1399999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:44:27 2021","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.2310000000000001,"15min":1.26},"min":{"1min":0.70899999999999996,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":2.1709999999999998,"5min":3.7440000000000002,"15min":6.3099999999999996},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.6160000000000001,"15min":1.4419999999999999},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.70999999999999996},"max":{"1min":2.3730000000000002,"5min":29.547999999999998,"15min":29.547999999999998},"last":1.004}]},{"osd":5,"last update":"Wed Feb 10 04:44:41 2021","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.206,"15min":1.526},"min":{"1min":0.77600000000000002,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":2.581,"5min":4.6289999999999996,"15min":48.686999999999998},"last":1.069},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.7869999999999999,"15min":1.8149999999999999},"min":{"1min":1.056,"5min":0.749,"15min":0.69699999999999995},"max":{"1min":2.6269999999999998,"5min":28.678000000000001,"15min":48.905000000000001},"last":1.202}]},{"osd":6,"last update":"Wed Feb 10 04:44:50 2021","interfaces":[{"interface":"back","average":{"1min":1.516,"5min":1.8660000000000001,"15min":1.8009999999999999},"min":{"1min":1.0509999999999999,"5min":0.82199999999999995,"15min":0.68400000000000005},"max":{"1min":5.0860000000000003,"5min":29.686,"15min":48.878999999999998},"last":1.5660000000000001},{"interface":"front","average":{"1min":1.236,"5min":1.631,"15min":1.694},"min":{"1min":0.83699999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.3279999999999998,"5min":28.475000000000001,"15min":49.011000000000003},"last":1.1890000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:44:04 2021","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":1.75,"15min":1.6120000000000001},"min":{"1min":0.96299999999999997,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":2.2719999999999998,"5min":28.573,"15min":28.573},"last":1.274},{"interface":"front","average":{"1min":1.595,"5min":1.875,"15min":1.6639999999999999},"min":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"max":{"1min":3.2010000000000001,"5min":29.408999999999999,"15min":29.408999999999999},"last":1.4039999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738631,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":389752,"kb_used_data":334960,"kb_used_omap":0,"kb_used_meta":54784,"kb_avail":93355400,"statfs":{"total":95995035648,"available":95595929600,"internally_reserved":0,"allocated":342999040,"data_stored":468155744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":56098816},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 04:44:20 2021","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.98199999999999998,"15min":1.099},"min":{"1min":0.59199999999999997,"5min":0.50900000000000001,"15min":0.48199999999999998},"max":{"1min":1.5960000000000001,"5min":2.4340000000000002,"15min":28.946999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.94499999999999995,"5min":1.091,"15min":0.999},"min":{"1min":0.66600000000000004,"5min":0.52700000000000002,"15min":0.51200000000000001},"max":{"1min":1.423,"5min":3.7050000000000001,"15min":3.7050000000000001},"last":1.0629999999999999}]},{"osd":2,"last update":"Wed Feb 10 04:44:53 2021","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.204,"15min":1.3420000000000001},"min":{"1min":0.68200000000000005,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.5960000000000001,"5min":3.903,"15min":25.984999999999999},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.044,"5min":1.1719999999999999,"15min":1.2010000000000001},"min":{"1min":0.59499999999999997,"5min":0.56999999999999995,"15min":0.52800000000000002},"max":{"1min":2.0550000000000002,"5min":3.4580000000000002,"15min":16.829000000000001},"last":1.2070000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:44:08 2021","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":1.2170000000000001,"15min":1.8560000000000001},"min":{"1min":0.60499999999999998,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.925,"5min":4.7069999999999999,"15min":126.587},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.413,"5min":1.5029999999999999,"15min":2.1459999999999999},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.54200000000000004},"max":{"1min":2.5430000000000001,"5min":5.8369999999999997,"15min":126.88200000000001},"last":0.997}]},{"osd":4,"last update":"Wed Feb 10 04:44:20 2021","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.4319999999999999,"15min":1.327},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.59399999999999997},"max":{"1min":1.837,"5min":5.1680000000000001,"15min":5.1680000000000001},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.111,"5min":1.2569999999999999,"15min":1.331},"min":{"1min":0.79300000000000004,"5min":0.72299999999999998,"15min":0.63800000000000001},"max":{"1min":1.7909999999999999,"5min":4.5810000000000004,"15min":14.672000000000001},"last":0.85399999999999998}]},{"osd":5,"last update":"Wed Feb 10 04:44:37 2021","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.7030000000000001,"15min":1.569},"min":{"1min":0.76700000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":2.169,"5min":25.047000000000001,"15min":25.047000000000001},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.544,"5min":1.861,"15min":1.8320000000000001},"min":{"1min":1.1000000000000001,"5min":0.97799999999999998,"15min":0.88800000000000001},"max":{"1min":2.2679999999999998,"5min":24.972999999999999,"15min":28.827999999999999},"last":1.296}]},{"osd":6,"last update":"Wed Feb 10 04:44:03 2021","interfaces":[{"interface":"back","average":{"1min":1.653,"5min":1.821,"15min":1.883},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":3.7410000000000001,"5min":24.837,"15min":35.872999999999998},"last":1.367},{"interface":"front","average":{"1min":1.8169999999999999,"5min":1.929,"15min":2.016},"min":{"1min":1.2050000000000001,"5min":0.96899999999999997,"15min":0.83099999999999996},"max":{"1min":4.2380000000000004,"5min":24.870999999999999,"15min":35.911000000000001},"last":1.1799999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:44:08 2021","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":1.6379999999999999,"15min":1.5369999999999999},"min":{"1min":1.0780000000000001,"5min":0.92800000000000005,"15min":0.73499999999999999},"max":{"1min":2.4529999999999998,"5min":3.472,"15min":5.6050000000000004},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.579,"15min":1.73},"min":{"1min":1.044,"5min":0.97299999999999998,"15min":0.94199999999999995},"max":{"1min":2.0609999999999999,"5min":5.1299999999999999,"15min":16.545000000000002},"last":1.248}]}]},{"osd":2,"up_from":0,"seq":77309411584,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":25,"seq":107374182650,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":395952,"kb_used_data":333480,"kb_used_omap":0,"kb_used_meta":62464,"kb_avail":93349200,"statfs":{"total":95995035648,"available":95589580800,"internally_reserved":0,"allocated":341483520,"data_stored":473575185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":63963136},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:44:09 2021","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.03,"15min":1.0920000000000001},"min":{"1min":0.59099999999999997,"5min":0.56000000000000005,"15min":0.47599999999999998},"max":{"1min":1.466,"5min":7.899,"15min":17.233000000000001},"last":1.294},{"interface":"front","average":{"1min":1.095,"5min":1.1240000000000001,"15min":1.228},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.52300000000000002},"max":{"1min":2.0430000000000001,"5min":3.7330000000000001,"15min":20.216999999999999},"last":1.0980000000000001}]},{"osd":1,"last update":"Wed Feb 10 04:44:09 2021","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.105,"15min":1.131},"min":{"1min":0.64500000000000002,"5min":0.56899999999999995,"15min":0.55400000000000005},"max":{"1min":2.7170000000000001,"5min":3.9550000000000001,"15min":8.9190000000000005},"last":1.2150000000000001},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.1040000000000001,"15min":1.0600000000000001},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.52100000000000002},"max":{"1min":2.3740000000000001,"5min":2.3740000000000001,"15min":4.6769999999999996},"last":0.59899999999999998}]},{"osd":2,"last update":"Wed Feb 10 04:44:09 2021","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.3640000000000001,"15min":1.3049999999999999},"min":{"1min":0.68100000000000005,"5min":0.624,"15min":0.56799999999999995},"max":{"1min":1.9099999999999999,"5min":3.29,"15min":5.024},"last":0.75},{"interface":"front","average":{"1min":1.052,"5min":1.1870000000000001,"15min":1.1890000000000001},"min":{"1min":0.79200000000000004,"5min":0.67100000000000004,"15min":0.56999999999999995},"max":{"1min":1.837,"5min":3.2559999999999998,"15min":4.5880000000000001},"last":1.4059999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:44:18 2021","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.3919999999999999,"15min":1.5620000000000001},"min":{"1min":0.85099999999999998,"5min":0.82299999999999995,"15min":0.629},"max":{"1min":1.8,"5min":4.6909999999999998,"15min":30.751999999999999},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.4970000000000001,"15min":1.6539999999999999},"min":{"1min":0.749,"5min":0.749,"15min":0.65700000000000003},"max":{"1min":1.7909999999999999,"5min":3.226,"15min":30.928000000000001},"last":0.81699999999999995}]},{"osd":5,"last update":"Wed Feb 10 04:44:37 2021","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.8129999999999999,"15min":1.9490000000000001},"min":{"1min":0.873,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":2.4449999999999998,"5min":18.356999999999999,"15min":35.616999999999997},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.671,"5min":1.9410000000000001,"15min":2.0600000000000001},"min":{"1min":0.88800000000000001,"5min":0.79600000000000004,"15min":0.77200000000000002},"max":{"1min":2.5329999999999999,"5min":18.100000000000001,"15min":35.661999999999999},"last":0.91800000000000004}]},{"osd":6,"last update":"Wed Feb 10 04:45:02 2021","interfaces":[{"interface":"back","average":{"1min":1.9079999999999999,"5min":1.893,"15min":2.0790000000000002},"min":{"1min":1.0680000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"max":{"1min":3.2869999999999999,"5min":18.614999999999998,"15min":35.518000000000001},"last":1.5009999999999999},{"interface":"front","average":{"1min":1.835,"5min":1.9810000000000001,"15min":2.3540000000000001},"min":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.92400000000000004},"max":{"1min":3.1480000000000001,"5min":18.036999999999999,"15min":53.965000000000003},"last":1.119}]},{"osd":7,"last update":"Wed Feb 10 04:44:09 2021","interfaces":[{"interface":"back","average":{"1min":1.661,"5min":1.7509999999999999,"15min":1.821},"min":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99199999999999999},"max":{"1min":2.2890000000000001,"5min":3.4380000000000002,"15min":25.242999999999999},"last":1.5580000000000001},{"interface":"front","average":{"1min":1.6759999999999999,"5min":1.879,"15min":1.9279999999999999},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":2.327,"5min":5.0270000000000001,"15min":25.286999999999999},"last":1.03}]}]},{"osd":4,"up_from":31,"seq":133143986424,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":96296,"kb_used_data":53216,"kb_used_omap":0,"kb_used_meta":43072,"kb_avail":93648856,"statfs":{"total":95995035648,"available":95896428544,"internally_reserved":0,"allocated":54493184,"data_stored":70063596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":44105728},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4],"upper_bound":131072},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:44:21 2021","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.218,"15min":1.3919999999999999},"min":{"1min":0.58199999999999996,"5min":0.503,"15min":0.48699999999999999},"max":{"1min":6.0880000000000001,"5min":25.285,"15min":62.273000000000003},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.093,"5min":1.2569999999999999,"15min":1.4259999999999999},"min":{"1min":0.60199999999999998,"5min":0.57699999999999996,"15min":0.499},"max":{"1min":6.1479999999999997,"5min":26.123000000000001,"15min":62.375999999999998},"last":0.89400000000000002}]},{"osd":1,"last update":"Wed Feb 10 04:44:21 2021","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.111,"15min":0.92600000000000005},"min":{"1min":0.626,"5min":0.58299999999999996,"15min":0.52800000000000002},"max":{"1min":2.085,"5min":24.295000000000002,"15min":24.295000000000002},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.3600000000000001,"15min":1.1990000000000001},"min":{"1min":0.73299999999999998,"5min":0.61099999999999999,"15min":0.59999999999999998},"max":{"1min":1.5649999999999999,"5min":24.943000000000001,"15min":24.943000000000001},"last":1.131}]},{"osd":2,"last update":"Wed Feb 10 04:44:21 2021","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.1850000000000001,"15min":1.4179999999999999},"min":{"1min":0.64400000000000002,"5min":0.61399999999999999,"15min":0.56499999999999995},"max":{"1min":5.7489999999999997,"5min":5.7489999999999997,"15min":62.521000000000001},"last":1.4750000000000001},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.1919999999999999,"15min":1.462},"min":{"1min":0.80100000000000005,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.974,"5min":2.1280000000000001,"15min":62.572000000000003},"last":1.4159999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:44:21 2021","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.617,"15min":1.613},"min":{"1min":1.0329999999999999,"5min":0.627,"15min":0.60699999999999998},"max":{"1min":2.1829999999999998,"5min":24.530999999999999,"15min":62.113},"last":3.7890000000000001},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.5589999999999999,"15min":1.5640000000000001},"min":{"1min":0.76300000000000001,"5min":0.73099999999999998,"15min":0.70299999999999996},"max":{"1min":1.9119999999999999,"5min":24.669,"15min":62.146000000000001},"last":3.762}]},{"osd":5,"last update":"Wed Feb 10 04:44:37 2021","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.4259999999999999,"15min":1.3899999999999999},"min":{"1min":0.78300000000000003,"5min":0.65300000000000002,"15min":0.60999999999999999},"max":{"1min":4.4660000000000002,"5min":4.4660000000000002,"15min":12.773999999999999},"last":2.0619999999999998},{"interface":"front","average":{"1min":1.421,"5min":1.341,"15min":1.3069999999999999},"min":{"1min":0.70299999999999996,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":2.8050000000000002,"5min":2.8050000000000002,"15min":13.038},"last":1.877}]},{"osd":6,"last update":"Wed Feb 10 04:44:50 2021","interfaces":[{"interface":"back","average":{"1min":1.6439999999999999,"5min":1.5800000000000001,"15min":1.796},"min":{"1min":0.91000000000000003,"5min":0.76600000000000001,"15min":0.70299999999999996},"max":{"1min":2.5369999999999999,"5min":2.6629999999999998,"15min":42.088999999999999},"last":1.3720000000000001},{"interface":"front","average":{"1min":1.728,"5min":1.649,"15min":1.8919999999999999},"min":{"1min":1.0600000000000001,"5min":0.86799999999999999,"15min":0.83899999999999997},"max":{"1min":2.3300000000000001,"5min":2.6989999999999998,"15min":42.027000000000001},"last":1.927}]},{"osd":7,"last update":"Wed Feb 10 04:44:13 2021","interfaces":[{"interface":"back","average":{"1min":1.8149999999999999,"5min":1.52,"15min":1.4530000000000001},"min":{"1min":1.014,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":5.8920000000000003,"5min":5.8920000000000003,"15min":6.9199999999999999},"last":1.6739999999999999},{"interface":"front","average":{"1min":1.6850000000000001,"5min":1.631,"15min":1.5449999999999999},"min":{"1min":1.2310000000000001,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.593,"5min":4.1399999999999997,"15min":4.1399999999999997},"last":1.7789999999999999}]}]},{"osd":5,"up_from":37,"seq":158913790197,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":344952,"kb_used_data":309360,"kb_used_omap":0,"kb_used_meta":35584,"kb_avail":93400200,"statfs":{"total":95995035648,"available":95641804800,"internally_reserved":0,"allocated":316784640,"data_stored":404931802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36438016},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:44:39 2021","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":0.98799999999999999,"15min":1.397},"min":{"1min":0.59899999999999998,"5min":0.51800000000000002,"15min":0.40500000000000003},"max":{"1min":1.75,"5min":4.5720000000000001,"15min":104.04300000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.92600000000000005,"5min":1.1120000000000001,"15min":1.456},"min":{"1min":0.501,"5min":0.501,"15min":0.46400000000000002},"max":{"1min":1.55,"5min":15.539999999999999,"15min":107.776},"last":0.93700000000000006}]},{"osd":1,"last update":"Wed Feb 10 04:44:39 2021","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":1.0289999999999999,"15min":1.3169999999999999},"min":{"1min":0.60399999999999998,"5min":0.51900000000000002,"15min":0.504},"max":{"1min":1.51,"5min":5.29,"15min":103.904},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.153,"5min":1.214,"15min":1.4830000000000001},"min":{"1min":0.71099999999999997,"5min":0.57599999999999996,"15min":0.50800000000000001},"max":{"1min":1.8180000000000001,"5min":11.875999999999999,"15min":104.315},"last":0.59099999999999997}]},{"osd":2,"last update":"Wed Feb 10 04:44:39 2021","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.623,"15min":1.9850000000000001},"min":{"1min":0.622,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.8759999999999999,"5min":44.462000000000003,"15min":103.65600000000001},"last":1.379},{"interface":"front","average":{"1min":1.202,"5min":1.6060000000000001,"15min":1.9690000000000001},"min":{"1min":0.73399999999999999,"5min":0.64000000000000001,"15min":0.63500000000000001},"max":{"1min":1.7130000000000001,"5min":44.398000000000003,"15min":104.136},"last":1.2130000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:44:39 2021","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.7509999999999999,"15min":2.0190000000000001},"min":{"1min":0.73299999999999998,"5min":0.66100000000000003,"15min":0.624},"max":{"1min":2.5299999999999998,"5min":44.609000000000002,"15min":106.18000000000001},"last":45.643000000000001},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.9810000000000001,"15min":2.0419999999999998},"min":{"1min":0.754,"5min":0.754,"15min":0.64500000000000002},"max":{"1min":1.863,"5min":44.537999999999997,"15min":106.822},"last":45.517000000000003}]},{"osd":4,"last update":"Wed Feb 10 04:44:39 2021","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.264,"15min":1.222},"min":{"1min":0.86799999999999999,"5min":0.57099999999999995,"15min":0.55400000000000005},"max":{"1min":1.6839999999999999,"5min":6.1639999999999997,"15min":6.1639999999999997},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.585,"5min":1.403,"15min":1.375},"min":{"1min":1.004,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":3.0209999999999999,"5min":3.0209999999999999,"15min":8.1449999999999996},"last":1.4450000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:44:50 2021","interfaces":[{"interface":"back","average":{"1min":1.702,"5min":5.1699999999999999,"15min":2.8490000000000002},"min":{"1min":0.83199999999999996,"5min":0.65800000000000003,"15min":0.63500000000000001},"max":{"1min":2.7120000000000002,"5min":319.51999999999998,"15min":319.51999999999998},"last":1.7689999999999999},{"interface":"front","average":{"1min":1.639,"5min":5.0789999999999997,"15min":2.7090000000000001},"min":{"1min":0.96599999999999997,"5min":0.77900000000000003,"15min":0.74399999999999999},"max":{"1min":2.8210000000000002,"5min":319.22300000000001,"15min":319.22300000000001},"last":1.27}]},{"osd":7,"last update":"Wed Feb 10 04:44:13 2021","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.4370000000000001,"15min":1.403},"min":{"1min":0.89500000000000002,"5min":0.752,"15min":0.752},"max":{"1min":1.8700000000000001,"5min":3.0009999999999999,"15min":5.4770000000000003},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.585,"15min":1.597},"min":{"1min":0.996,"5min":0.92300000000000004,"15min":0.83499999999999996},"max":{"1min":2.073,"5min":2.992,"15min":8.7840000000000007},"last":1.9379999999999999}]}]}]}} 2021-02-10T04:45:17.592 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:14 smithi085 conmon[29319]: cluster 2021-02-10T04:45:12.294830+0000 mgr.y (mgr 2021-02-10T04:45:17.592 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:14 smithi085 conmon[29319]: .14140) 971 : cluster [DBG] pgmap v1725: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:45:17.592 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:15 smithi085 conmon[29319]: cluster 2021-02-10T04:45:13.989400+0000 osd.2 (osd.2) 1947 : cluster [DBG] 2021-02-10T04:45:17.592 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:15 smithi085 conmon[29319]: purged_snaps scrub starts 2021-02-10T04:45:17.592 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:15 smithi085 conmon[29319]: cluster 2021-02-10T04:45:13.990116+0000 osd.2 (osd.2) 1948 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:45:17.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:15 smithi085 conmon[29319]: cluster 2021-02-10T04:45:14.295438+0000 mgr.y (mgr.14140) 972 : cluster [DBG] pgmap v1726: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:45:17.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:15 smithi085 conmon[29319]: cluster 2021-02-10T04:45:14.380007+0000 mon.a (mon.0) 2052 : cluster [DBG] osdmap e1029: 8 total, 8 up, 7 in 2021-02-10T04:45:17.593 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:14 smithi085 conmon[35961]: cluster 2021-02-10T04:45:12.294830+0000 2021-02-10T04:45:17.593 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:14 smithi085 conmon[35961]: mgr.y (mgr.14140) 971 : cluster [DBG] pgmap v1725: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:45:17.593 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:15 smithi085 conmon[35961]: cluster 2021-02-10T04:45:17.593 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:15 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:45:17.593 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:15 smithi085 conmon[35961]: 04:45:13.989400+0000 osd.2 (osd.2) 1947 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:45:17.593 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:15 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:45:17.593 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:15 smithi085 conmon[35961]: 02-10T04:45:13.990116+0000 osd.2 (osd 2021-02-10T04:45:17.593 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:15 smithi085 conmon[35961]: .2) 1948 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:45:17.593 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:15 smithi085 conmon[35961]: cluster 2021-02-10T04:45:17.594 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:15 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:45:17.594 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:15 smithi085 conmon[35961]: 45:14.295438+0000 mgr.y (mgr.14140) 972 : cluster [DBG] 2021-02-10T04:45:17.594 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:15 smithi085 conmon[35961]: pgmap v1726: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:45:17.594 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:15 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:45:17.594 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:15 smithi085 conmon[35961]: 04:45:14.380007+0000 mon.a (mon.0) 2052 : cluster [DBG] osdmap e1029: 8 total, 8 up, 7 in 2021-02-10T04:45:17.602 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2021-02-10T04:45:17.602 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pool set zone.rgw.meta pg_num 42 2021-02-10T04:45:17.638 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:17 smithi164 conmon[29288]: cluster 2021-02-10T04:45:16.296120+0000 mgr.y 2021-02-10T04:45:17.638 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:17 smithi164 conmon[29288]: (mgr.14140) 973 : cluster [DBG] pgmap v1728: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:45:17.643 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:45:17.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:17 smithi085 conmon[29319]: cluster 2021-02-10T04:45:17.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:17 smithi085 conmon[29319]: 2021-02-10T04:45:17.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:17 smithi085 conmon[29319]: 2021 2021-02-10T04:45:17.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:17 smithi085 conmon[29319]: - 2021-02-10T04:45:17.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:17 smithi085 conmon[29319]: 02 2021-02-10T04:45:17.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:17 smithi085 conmon[29319]: - 2021-02-10T04:45:17.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:17 smithi085 conmon[29319]: 10 2021-02-10T04:45:17.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:17 smithi085 conmon[29319]: T 2021-02-10T04:45:17.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:17 smithi085 conmon[29319]: 04 2021-02-10T04:45:17.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:17 smithi085 conmon[29319]: :45:16.296120 2021-02-10T04:45:17.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:17 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 973 : cluster [DBG] pgmap v1728: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:45:17.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:17 smithi085 conmon[29319]: 2021-02-10T04:45:17.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:17 smithi085 conmon[35961]: cluster 2021-02-10T04:45: 2021-02-10T04:45:17.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:17 smithi085 conmon[35961]: 16.296120+0000 mgr.y (mgr.14140) 973 : cluster [DBG] pgmap v1728: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:45:17.894 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:45:17 smithi164 conmon[40892]: debug 2021-02-10T04:45:17.894 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:45:17 smithi164 conmon[40892]: 2021-02-10T04:45:17.724+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:45:18.260 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:45:19.144 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:45:18 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:45:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:19 smithi164 conmon[29288]: cluster 2021-02-10T04:45:18.296790 2021-02-10T04:45:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:19 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 974 : cluster [DBG] pgmap v1729: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:45:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:19 smithi164 conmon[29288]: cluster 2021-02-10T04:45:18.368866+0000 2021-02-10T04:45:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:19 smithi164 conmon[29288]: mon.a (mon.0) 2053 : cluster [DBG] osdmap e1030: 8 total, 8 up, 7 in 2021-02-10T04:45:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:19 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:45:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:19 smithi164 conmon[29288]: 02-10T04:45:18.693044+0000 2021-02-10T04:45:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:19 smithi164 conmon[29288]: mgr.y (mgr.14140) 975 : audit [DBG] 2021-02-10T04:45:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:19 smithi164 conmon[29288]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:45:19.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:19 smithi085 conmon[35961]: cluster 2021-02-10T04:45 2021-02-10T04:45:19.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:19 smithi085 conmon[35961]: :18.296790+0000 mgr.y (mgr.14140) 974 : cluster [DBG] pgmap v1729: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:45:19.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:19 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:45:19.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:19 smithi085 conmon[35961]: 02-10T04:45:18.368866+0000 mon.a (mon.0) 2053 : cluster [DBG] osdmap e1030: 8 total, 8 up, 7 in 2021-02-10T04:45:19.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:19 smithi085 conmon[35961]: audit 2021-02-10T04:45:18. 2021-02-10T04:45:19.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:19 smithi085 conmon[35961]: 693044+0000 mgr.y (mgr.14140) 975 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:45:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:19 smithi085 conmon[29319]: cluster 2021-02-10T04:45:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:19 smithi085 conmon[29319]: 2021-02-10T04:45:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:19 smithi085 conmon[29319]: 2021-02-10T04:45:18.296790 2021-02-10T04:45:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:19 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 974 : cluster [DBG] pgmap v1729: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:45:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:19 smithi085 conmon[29319]: cluster 2021-02-10T04:45: 2021-02-10T04:45:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:19 smithi085 conmon[29319]: 18.368866+0000 mon.a (mon.0) 2053 : cluster [DBG] osdmap e1030: 8 total, 8 up, 7 in 2021-02-10T04:45:19.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:19 smithi085 conmon[29319]: 2021-02-10T04:45:19.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:19 smithi085 conmon[29319]: audit 2021-02-10T04:45:19.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:19 smithi085 conmon[29319]: 2021-02-10T04:45:18.693044+0000 mgr.y (mgr. 2021-02-10T04:45:19.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:19 smithi085 conmon[29319]: 14140) 975 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:45:20.792 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:45:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:21 smithi164 conmon[29288]: cluster 2021-02-10T04:45:20.297496+0000 mgr.y (mgr. 2021-02-10T04:45:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:21 smithi164 conmon[29288]: 14140) 976 : cluster [DBG] pgmap v1731: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:45:21.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:21 smithi085 conmon[29319]: cluster 2021-02-10T04:45:20.297496+0000 2021-02-10T04:45:21.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:21 smithi085 conmon[29319]: mgr.y (mgr.14140) 976 : cluster [DBG] pgmap v1731: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:45:21.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:21 smithi085 conmon[35961]: cluster 2021-02-10T04:45:20.297496+0000 2021-02-10T04:45:21.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:21 smithi085 conmon[35961]: mgr.y (mgr.14140) 976 : cluster [DBG] pgmap v1731: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:45:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:23 smithi164 conmon[29288]: cluster 2021-02-10T04:45:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:23 smithi164 conmon[29288]: 2021-02-10T04:45:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:23 smithi164 conmon[29288]: 2021-02-10T04:45:22. 2021-02-10T04:45:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:23 smithi164 conmon[29288]: 297981+0000 mgr.y (mgr.14140) 977 : cluster [DBG] pgmap v1732: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:45:23.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:23 smithi085 conmon[35961]: cluster 2021-02-10T04:45:22.297981 2021-02-10T04:45:23.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:23 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 977 : cluster [DBG] pgmap v1732: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:45:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:23 smithi085 conmon[29319]: cluster 2021-02-10T04:45:22.297981+0000 mgr.y (mgr.14140) 977 : cluster [DBG] pgmap v1732: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:45:24.890 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:45:24.924 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:45:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:25 smithi164 conmon[29288]: cluster 2021-02-10T04:45:24.298559+0000 mgr.y (mgr.14140) 978 : cluster [DBG] pgmap v1733: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:45:25.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:25 smithi085 conmon[29319]: cluster 2021-02-10T04:45:25.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:25 smithi085 conmon[29319]: 2021-02-10T04:45:25.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:25 smithi085 conmon[29319]: 2021 2021-02-10T04:45:25.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:25 smithi085 conmon[29319]: - 2021-02-10T04:45:25.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:25 smithi085 conmon[29319]: 02 2021-02-10T04:45:25.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:25 smithi085 conmon[29319]: - 2021-02-10T04:45:25.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:25 smithi085 conmon[29319]: 10 2021-02-10T04:45:25.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:25 smithi085 conmon[29319]: T 2021-02-10T04:45:25.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:25 smithi085 conmon[29319]: 04:45:24.298559 2021-02-10T04:45:25.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:25 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 978 : cluster [DBG] pgmap v1733: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:45:25.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:25 smithi085 conmon[35961]: cluster 2021-02-10T04:45: 2021-02-10T04:45:25.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:25 smithi085 conmon[35961]: 24.298559+0000 mgr.y (mgr.14140) 978 : cluster [DBG] pgmap v1733: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:45:26.110 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.5 dump_historic_ops 2021-02-10T04:45:26.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:45:26 smithi164 conmon[44544]: debug 2021-02-10T04:45:26.110+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:45:26.423 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:45:26.443 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:45:26.529 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:45:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:26 smithi164 conmon[29288]: cluster 2021-02-10T04:45:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:26 smithi164 conmon[29288]: 2021-02-10T04:45:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:26 smithi164 conmon[29288]: 2021 2021-02-10T04:45:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:26 smithi164 conmon[29288]: - 2021-02-10T04:45:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:26 smithi164 conmon[29288]: 02-10T04:45:25.136812+0000 2021-02-10T04:45:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:26 smithi164 conmon[29288]: osd.6 (osd.6) 102 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:45:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:26 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:45:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:26 smithi164 conmon[29288]: 02-10T04:45:25.140946+0000 osd.6 (osd.6) 103 : cluster [DBG] 2021-02-10T04:45:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:26 smithi164 conmon[29288]: purged_snaps scrub found 4 strays 2021-02-10T04:45:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:26 smithi164 conmon[29288]: cluster 2021-02-10T04:45:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:26 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:45:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:26 smithi164 conmon[29288]: :45:25.390210+0000 mon.a (mon.0) 2054 : cluster 2021-02-10T04:45:26.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:26 smithi164 conmon[29288]: [DBG] osdmap e1031: 8 total, 8 up, 7 in 2021-02-10T04:45:26.682 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:45:26.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:26 smithi085 conmon[29319]: cluster 2021-02-10T04:45:25.136812+0000 2021-02-10T04:45:26.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:26 smithi085 conmon[29319]: osd.6 (osd.6) 102 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:45:26.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:26 smithi085 conmon[29319]: cluster 2021-02-10T04:45:25.140946+0000 osd.6 (osd.6) 103 : cluster [DBG] purged_snaps scrub found 4 strays 2021-02-10T04:45:26.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:26 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:45:26.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:26 smithi085 conmon[29319]: 45:25.390210+0000 mon.a (mon.0) 2054 : cluster [DBG] osdmap e1031: 8 total, 8 up, 7 in 2021-02-10T04:45:26.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:26 smithi085 conmon[35961]: cluster 2021-02-10T04:45:25.136812+0000 osd.6 (osd.6) 102 : cluster [DBG] 2021-02-10T04:45:26.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:26 smithi085 conmon[35961]: purged_snaps scrub starts 2021-02-10T04:45:26.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:26 smithi085 conmon[35961]: cluster 2021-02-10T04:45:25.140946+0000 osd.6 (osd.6) 103 : cluster [DBG] purged_snaps scrub found 4 strays 2021-02-10T04:45:26.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:26 smithi085 conmon[35961]: cluster 2021-02-10T04:45:25.390210+0000 mon.a (mon.0) 2054 : cluster [DBG] osdmap e1031: 8 total, 8 up, 7 in 2021-02-10T04:45:27.680 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:27 smithi085 conmon[29319]: cluster 2021-02-10T04:45:26.298952+0000 mgr.y (mgr 2021-02-10T04:45:27.680 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:27 smithi085 conmon[29319]: .14140) 979 : cluster [DBG] pgmap v1735: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:45:27.681 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:27 smithi085 conmon[35961]: cluster 2021-02-10T04:45:27.681 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:27 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:45:27.681 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:27 smithi085 conmon[35961]: 10T04:45:26.298952+0000 mgr.y (mgr.14140) 979 : cluster [DBG] pgmap v1735: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:45:27.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:27 smithi164 conmon[29288]: cluster 2021-02-10T04:45:26.298952+0000 mgr.y ( 2021-02-10T04:45:27.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:27 smithi164 conmon[29288]: mgr.14140) 979 : cluster [DBG] pgmap v1735: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:45:27.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:45:27 smithi085 conmon[49106]: debug 2021-02-10T04:45:27.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:45:27 smithi085 conmon[49106]: 2021-02-10T04:45:27.679+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:45:28.082 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:45:28.218 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:45:28.546 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:45:28.675 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:45:28.781 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:45:28.806 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:45:28 smithi164 conmon[40892]: debug 2021-02-10T04:45:28.637+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:45:28.806 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:45:28 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:45:28.810 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:45:29.027 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:45:29.133 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:45:29.143 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:28 smithi164 conmon[33677]: debug 2021-02-10T04:45:28.944+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:45:29.166 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:45:29.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:29 smithi085 conmon[29319]: cluster 2021-02-10T04:45:28.299516+0000 mgr.y (mgr.14140) 980 : cluster [DBG] pgmap v1736: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:45:29.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:29 smithi085 conmon[29319]: cluster 2021-02-10T04:45:28.371414+0000 mon.a (mon.0) 2055 : cluster [DBG] osdmap e1032: 8 total, 8 up, 7 in 2021-02-10T04:45:29.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:29 smithi085 conmon[29319]: audit 2021-02-10T04:45:28.703038+0000 mgr.y (mgr.14140) 981 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:45:29.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:29 smithi085 conmon[35961]: cluster 2021-02-10T04:45:28.299516+0000 mgr.y (mgr.14140) 980 : cluster [DBG] pgmap v1736: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:45:29.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:29 smithi085 conmon[35961]: cluster 2021-02-10T04:45:28.371414+0000 mon.a (mon.0) 2055 : cluster [DBG] osdmap e1032: 8 total, 8 up, 7 in 2021-02-10T04:45:29.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:29 smithi085 conmon[35961]: audit 2021-02-10T04:45:28.703038+0000 mgr.y (mgr.14140) 981 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:45:29.508 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:45:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:29 smithi164 conmon[29288]: cluster 2021-02-10T04:45:28.299516+0000 mgr.y (mgr. 2021-02-10T04:45:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:29 smithi164 conmon[29288]: 14140) 980 : cluster [DBG] pgmap v1736: 178 pgs: 178 active+clean; 865 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:45:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:29 smithi164 conmon[29288]: cluster 2021-02-10T04:45:28.371414+0000 mon.a (mon.0) 2055 : cluster [DBG] osdmap e1032: 8 total, 8 up, 7 in 2021-02-10T04:45:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:29 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:45:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:29 smithi164 conmon[29288]: 10T04:45:28.703038+0000 mgr.y (mgr.14140) 981 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:45:29.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:45:29 smithi164 conmon[37263]: debug 2021-02-10T04:45:29.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:45:29 smithi164 conmon[37263]: 2021-02-10T04:45:29.248+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:45:31.258 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:45:34.591 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:31 smithi164 conmon[29288]: cluster 2021-02-10T04:45:30.300330+0000 2021-02-10T04:45:34.591 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:31 smithi164 conmon[29288]: mgr.y (mgr.14140) 982 : cluster [DBG] pgmap v1738: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 863 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:45:34.591 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:31 smithi164 conmon[29288]: audit 2021-02-10T04:45:31.221830+0000 mon.b (mon.2) 45 : 2021-02-10T04:45:34.591 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:31 smithi164 conmon[29288]: audit [DBG] from='client.? 172.21.15.85:0/2986859701' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:45:34.591 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:32 smithi164 conmon[29288]: audit 2021-02-10T04:45:31.526168+0000 mon.b (mon.2) 46 : audit [INF] from='client.? 172.21.15.85:0/3268958103' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:45:34.591 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:32 smithi164 conmon[29288]: 2021-02-10T04:45:34.591 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:32 smithi164 conmon[29288]: audit 2021-02-10T04:45:31.526634+0000 mon.a (mon.0) 2056 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:45:34.592 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:33 smithi164 conmon[29288]: cluster 2021-02-10T04:45:32.300900+0000 mgr.y (mgr.14140) 983 : cluster [DBG] 2021-02-10T04:45:34.592 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:33 smithi164 conmon[29288]: pgmap v1739: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 863 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:45:34.592 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:33 smithi164 conmon[29288]: cluster 2021-02-10T04:45:32.337722+0000 osd.4 (osd.4) 1796 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:34.592 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:33 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:45:34.592 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:33 smithi164 conmon[29288]: 04:45:32.337732+0000 osd.4 (osd.4) 1797 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:34.592 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:33 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:45:34.592 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:33 smithi164 conmon[29288]: 02-10T04:45:32.337739+0000 osd.4 (osd.4) 1798 : cluster 2021-02-10T04:45:34.592 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:33 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:34.592 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:33 smithi164 conmon[29288]: cluster 2021-02-10T04:45:34.592 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:33 smithi164 conmon[29288]: 2021-02-10T04:45:32.337745+0000 osd.4 (osd 2021-02-10T04:45:34.592 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:33 smithi164 conmon[29288]: .4) 1799 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:34.592 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:33 smithi164 conmon[29288]: cluster 2021-02-10T04:45 2021-02-10T04:45:34.592 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:33 smithi164 conmon[29288]: :32.368666+0000 osd.3 (osd.3) 102 : 2021-02-10T04:45:34.593 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:33 smithi164 conmon[29288]: cluster [DBG] purged_snaps scrub starts 2021-02-10T04:45:34.593 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:33 smithi164 conmon[29288]: cluster 2021-02-10T04:45: 2021-02-10T04:45:34.593 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:33 smithi164 conmon[29288]: 32.373893+0000 mon.a (mon.0) 2057 : cluster 2021-02-10T04:45:34.593 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:33 smithi164 conmon[29288]: [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:45:34.593 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:33 smithi164 conmon[29288]: cluster 2021-02-10T04:45: 2021-02-10T04:45:34.593 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:33 smithi164 conmon[29288]: 32.374152+0000 osd.3 (osd.3) 103 : cluster [DBG] 2021-02-10T04:45:34.593 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:33 smithi164 conmon[29288]: purged_snaps scrub ok 2021-02-10T04:45:34.593 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:33 smithi164 conmon[29288]: audit 2021-02-10T04:45:34.593 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:33 smithi164 conmon[29288]: 2021-02-10T04:45:32.382788+0000 mon.a (mon. 2021-02-10T04:45:34.593 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:33 smithi164 conmon[29288]: 0) 2058 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:45:34.593 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:33 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:45:34.593 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:33 smithi164 conmon[29288]: T04:45:32.382994+0000 mon.a (mon.0) 2021-02-10T04:45:34.593 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:33 smithi164 conmon[29288]: 2059 : cluster [DBG] osdmap e1033: 8 total, 8 up, 7 in 2021-02-10T04:45:34.593 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:33 smithi164 conmon[29288]: cluster 2021-02-10T04:45 2021-02-10T04:45:34.594 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:33 smithi164 conmon[29288]: :33.372565+0000 mon.a (mon.0) 2060 : cluster 2021-02-10T04:45:34.594 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:33 smithi164 conmon[29288]: [DBG] osdmap e1034: 8 total, 8 up, 7 in 2021-02-10T04:45:34.594 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:32 smithi164 conmon[33677]: debug 2021-02-10T04:45:32.336+0000 7fbec19a1700 -1 osd.4 1032 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:45:34.594 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:33 smithi164 conmon[33677]: debug 2021-02-10T04:45:33.355+0000 7fbec19a1700 -1 osd.4 1033 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:45:34.594 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:45:34.594 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:45:34.594 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:45:34.594 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:45:34.594 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:45:34.594 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:45:34.595 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:45:34.595 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:45:34.595 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:45:34.595 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:45:34.595 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:45:34.595 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:45:34.595 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:45:34.595 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:45:34.595 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:45:34.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:31 smithi085 conmon[29319]: cluster 2021-02-10T04:45:30.300330+0000 mgr.y (mgr.14140) 982 : 2021-02-10T04:45:34.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:31 smithi085 conmon[29319]: cluster [DBG] pgmap v1738: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 863 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:45:34.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:31 smithi085 conmon[29319]: audit 2021-02-10T04:45:31.221830+0000 mon.b (mon.2) 45 : audit [DBG] from='client.? 172.21.15.85:0/2986859701' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:45:34.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:32 smithi085 conmon[29319]: audit 2021-02-10T04:45:31.526168+0000 mon.b (mon.2) 46 2021-02-10T04:45:34.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:32 smithi085 conmon[29319]: : audit [INF] from='client.? 172.21.15.85:0/3268958103' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:45:34.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:32 smithi085 conmon[29319]: audit 2021-02-10T04:45:31.526634+0000 mon.a (mon.0) 2056 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:45:34.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: cluster 2021-02-10T04:45:32.300900+0000 mgr.y (mgr 2021-02-10T04:45:34.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: .14140) 983 : cluster [DBG] pgmap v1739: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 863 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:45:34.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: cluster 2021-02-10T04:45:32.337722+0000 osd.4 (osd.4) 1796 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:34.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: cluster 2021-02-10T04:45:32.337732+0000 osd.4 2021-02-10T04:45:34.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: (osd.4) 1797 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:34.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:45:34.597 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: -02-10T04:45:32.337739+0000 osd.4 (osd.4) 1798 : cluster [WRN] 2021-02-10T04:45:34.597 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:34.597 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: cluster 2021-02-10T04:45:34.597 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: 2021-02-10T04:45:32.337745+0000 osd.4 (osd 2021-02-10T04:45:34.597 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: .4) 1799 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:34.597 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:45:34.597 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: -02-10T04:45:32. 2021-02-10T04:45:34.597 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: 368666+0000 osd.3 (osd.3 2021-02-10T04:45:34.597 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: ) 102 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:45:34.597 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: 2021-02-10T04:45:34.597 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: cluster 2021-02-10T04:45:34.597 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: 2021-02-10T04:45:32 2021-02-10T04:45:34.597 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: .373893+0000 mon.a (mon 2021-02-10T04:45:34.598 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: .0) 2057 : cluster [WRN] 2021-02-10T04:45:34.598 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:45:34.598 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:45:34.598 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: :45:32.374152+0000 osd.3 (osd.3) 103 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:45:34.599 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: audit 2021-02-10T04:45: 2021-02-10T04:45:34.599 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: 32.382788+0000 mon.a (mon.0) 2058 : 2021-02-10T04:45:34.599 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:45:34.600 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:45:34.600 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: -02-10T04:45 2021-02-10T04:45:34.600 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: :32.382994+0000 mon.a ( 2021-02-10T04:45:34.600 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: mon.0) 2059 : cluster 2021-02-10T04:45:34.600 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: [DBG] osdmap e1033: 8 total, 8 up, 7 in 2021-02-10T04:45:34.600 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:45:34.600 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: -10T04:45:33. 2021-02-10T04:45:34.600 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: 372565+0000 mon.a (mon.0 2021-02-10T04:45:34.600 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[29319]: ) 2060 : cluster [DBG] osdmap e1034: 8 total, 8 up, 7 in 2021-02-10T04:45:34.600 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:31 smithi085 conmon[35961]: cluster 2021-02-10T04:45:30. 2021-02-10T04:45:34.600 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:31 smithi085 conmon[35961]: 300330+0000 mgr.y (mgr.14140) 982 : cluster [DBG] pgmap v1738: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 863 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:45:34.600 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:31 smithi085 conmon[35961]: audit 2021-02-10T04:45 2021-02-10T04:45:34.601 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:31 smithi085 conmon[35961]: :31.221830+0000 mon.b (mon.2) 45 : audit [DBG] from='client.? 172.21.15.85:0/2986859701' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:45:34.601 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:32 smithi085 conmon[35961]: audit 2021-02-10T04:45:31.526168+0000 mon.b ( 2021-02-10T04:45:34.601 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:32 smithi085 conmon[35961]: mon.2) 46 : audit [INF] from='client.? 172.21.15.85:0/3268958103' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:45:34.601 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:32 smithi085 conmon[35961]: audit 2021-02-10T04:45 2021-02-10T04:45:34.601 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:32 smithi085 conmon[35961]: :31.526634+0000 mon.a (mon.0) 2056 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:45:34.601 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: cluster 2021-02-10T04:45:34.601 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: 2021-02-10T04:45 2021-02-10T04:45:34.601 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: :32.300900+0000 mgr.y (mgr.14140) 983 : cluster [DBG] pgmap v1739: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 863 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:45:34.601 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:45:34.601 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: 45:32.337722+0000 osd.4 (osd.4) 2021-02-10T04:45:34.601 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: 1796 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:34.601 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:45:34.601 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: :45:32.337732+0000 osd.4 (osd. 2021-02-10T04:45:34.601 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: 4) 1797 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:34.602 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:45:34.602 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: 04:45:32.337739+0000 osd.4 (osd. 2021-02-10T04:45:34.602 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: 4) 1798 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:34.602 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:45:34.602 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: 04:45:32.337745+0000 osd.4 (osd.4) 1799 : cluster 2021-02-10T04:45:34.602 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:34.602 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: cluster 2021-02-10T04:45:32 2021-02-10T04:45:34.602 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: .368666+0000 osd.3 (osd.3) 2021-02-10T04:45:34.602 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: 102 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:45:34.602 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:45:34.602 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: 10T04:45:32.373893+0000 mon.a (mon 2021-02-10T04:45:34.602 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: .0) 2057 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:45:34.602 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:45:34.603 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: 02-10T04:45:32.374152+0000 2021-02-10T04:45:34.603 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: osd.3 (osd.3) 103 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:45:34.603 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: 2021-02-10T04:45:34.603 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: audit 2021-02-10T04:45:34.603 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: 2021-02-10T04:45:32.382788+0000 mon.a (mon.0 2021-02-10T04:45:34.603 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: ) 2058 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:45:34.603 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:45:34.603 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: 45:32.382994+0000 mon.a (mon.0) 2021-02-10T04:45:34.603 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: 2059 : cluster [DBG] osdmap e1033: 8 total, 8 up, 7 in 2021-02-10T04:45:34.603 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: cluster 2021-02-10T04:45 2021-02-10T04:45:34.603 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: :33.372565+0000 mon.a (mon.0) 2060 2021-02-10T04:45:34.603 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:33 smithi085 conmon[35961]: : cluster [DBG] osdmap e1034: 8 total, 8 up, 7 in 2021-02-10T04:45:34.603 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is set 2021-02-10T04:45:34.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:34 smithi164 conmon[33677]: debug 2021-02-10T04:45:34.377+0000 7fbec19a1700 -1 osd.4 1034 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:45:34.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:34 smithi164 conmon[29288]: cluster 2021-02-10T04:45:33.356869+0000 osd.4 2021-02-10T04:45:34.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:34 smithi164 conmon[29288]: (osd.4) 1800 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:34.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:34 smithi164 conmon[29288]: cluster 2021-02-10T04:45:33.356879+0000 osd.4 (osd.4) 1801 : cluster [WRN] 2021-02-10T04:45:34.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:34 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:34.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:34 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:45:34.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:34 smithi164 conmon[29288]: 04:45:33.356886+0000 osd.4 (osd.4) 1802 : 2021-02-10T04:45:34.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:34 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:34.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:34 smithi164 conmon[29288]: cluster 2021-02-10T04:45:33.356893+0000 2021-02-10T04:45:34.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:34 smithi164 conmon[29288]: osd.4 (osd.4) 1803 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:34 smithi085 conmon[29319]: cluster 2021-02-10T04:45:33.356869+0000 2021-02-10T04:45:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:34 smithi085 conmon[29319]: osd.4 (osd.4) 1800 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:34 smithi085 conmon[29319]: cluster 2021-02-10T04:45:33.356879+0000 osd.4 (osd.4) 1801 : cluster 2021-02-10T04:45:34.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:34 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:34 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:45:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:34 smithi085 conmon[29319]: -10T04:45:33.356886+0000 osd.4 (osd.4) 2021-02-10T04:45:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:34 smithi085 conmon[29319]: 1802 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:34 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:45:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:34 smithi085 conmon[29319]: 02-10T04:45:33.356893+0000 2021-02-10T04:45:34.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:34 smithi085 conmon[29319]: osd.4 (osd.4) 1803 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:34.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:34 smithi085 conmon[35961]: cluster 2021-02-10T04:45:34.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:34 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:45:34.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:34 smithi085 conmon[35961]: 04:45:33.356869+0000 osd.4 (osd.4) 1800 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:34.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:34 smithi085 conmon[35961]: cluster 2021-02-10T04:45:34.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:34 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:45:34.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:34 smithi085 conmon[35961]: :45:33.356879+0000 osd.4 (osd.4) 1801 : cluster 2021-02-10T04:45:34.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:34 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:34.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:34 smithi085 conmon[35961]: cluster 2021-02-10T04:45:34.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:34 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:45:34.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:34 smithi085 conmon[35961]: 04:45:33.356886+0000 osd.4 (osd.4 2021-02-10T04:45:34.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:34 smithi085 conmon[35961]: ) 1802 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:34.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:34 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:45:34.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:34 smithi085 conmon[35961]: -02-10T04:45:33.356893+0000 osd.4 2021-02-10T04:45:34.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:34 smithi085 conmon[35961]: (osd.4) 1803 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:35.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:35 smithi164 conmon[33677]: debug 2021-02-10T04:45:35.389+0000 7fbec19a1700 -1 osd.4 1034 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:45:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:35 smithi164 conmon[29288]: cluster 2021-02-10T04:45:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:35 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:45:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:35 smithi164 conmon[29288]: 45:34.301619+0000 mgr.y (mgr.14140) 984 : cluster [DBG] pgmap v1742: 178 pgs: 178 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2021-02-10T04:45:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:35 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:45:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:35 smithi164 conmon[29288]: 10T04:45:34.379128+0000 osd.4 (osd.4) 1804 : 2021-02-10T04:45:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:35 smithi164 conmon[29288]: cluster 2021-02-10T04:45:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:35 smithi164 conmon[29288]: 2021-02-10T04:45:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:35 smithi164 conmon[29288]: [WRN] 2021-02-10T04:45:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:35 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:35 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:45:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:35 smithi164 conmon[29288]: -02-10T04:45:34.379135+0000 osd.4 ( 2021-02-10T04:45:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:35 smithi164 conmon[29288]: osd.4 2021-02-10T04:45:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:35 smithi164 conmon[29288]: ) 1805 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:35 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:45:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:35 smithi164 conmon[29288]: -02-10T04:45:34.379138+0000 2021-02-10T04:45:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:35 smithi164 conmon[29288]: osd.4 (osd.4) 1806 : cluster [WRN] 2021-02-10T04:45:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:35 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:35 smithi164 conmon[29288]: cluster 2021-02-10T04:45:34 2021-02-10T04:45:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:35 smithi164 conmon[29288]: .379142+0000 osd.4 (osd.4) 1807 : cluster 2021-02-10T04:45:35.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:35 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:35.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:35 smithi085 conmon[29319]: cluster 2021-02-10T04:45:34.301619+0000 mgr.y (mgr.14140) 984 : cluster [DBG] pgmap v1742: 178 pgs: 178 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2021-02-10T04:45:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:35 smithi085 conmon[29319]: 2021-02-10T04:45:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:35 smithi085 conmon[29319]: cluster 2021-02-10T04:45:34.379128+0000 osd.4 (osd.4) 1804 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:35 smithi085 conmon[29319]: cluster 2021-02-10T04:45:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:35 smithi085 conmon[29319]: 2021-02-10T04:45:34.379135+0000 osd.4 (osd.4) 1805 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:35 smithi085 conmon[29319]: 2021-02-10T04:45:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:35 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:45:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:35 smithi085 conmon[29319]: 45:34.379138+0000 osd.4 (osd. 2021-02-10T04:45:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:35 smithi085 conmon[29319]: 4) 1806 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:35 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:45:35.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:35 smithi085 conmon[29319]: T04:45:34.379142+0000 osd.4 (osd 2021-02-10T04:45:35.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:35 smithi085 conmon[29319]: .4) 1807 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:35.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:35 smithi085 conmon[35961]: cluster 2021-02-10T04:45:34.301619+0000 2021-02-10T04:45:35.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:35 smithi085 conmon[35961]: mgr.y (mgr.14140) 984 : cluster [DBG] pgmap v1742: 178 pgs: 178 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2021-02-10T04:45:35.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:35 smithi085 conmon[35961]: cluster 2021-02-10T04:45:34.379128+0000 osd.4 2021-02-10T04:45:35.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:35 smithi085 conmon[35961]: (osd.4) 1804 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:35.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:35 smithi085 conmon[35961]: cluster 2021-02-10T04:45:35.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:35 smithi085 conmon[35961]: 2021-02-10T04:45:34.379135+0000 osd.4 (osd.4) 1805 : cluster [WRN] 2021-02-10T04:45:35.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:35 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:35.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:35 smithi085 conmon[35961]: cluster 2021-02-10T04:45 2021-02-10T04:45:35.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:35 smithi085 conmon[35961]: :34.379138+0000 osd.4 (osd.4) 1806 2021-02-10T04:45:35.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:35 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:35.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:35 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:45:35.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:35 smithi085 conmon[35961]: 45:34.379142+0000 osd.4 (osd.4) 2021-02-10T04:45:35.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:35 smithi085 conmon[35961]: 1807 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:35.939 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:45:35.973 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:45:36.645 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:36 smithi164 conmon[33677]: debug 2021-02-10T04:45:36.380+0000 7fbec19a1700 -1 osd.4 1034 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:45:36.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:36 smithi164 conmon[29288]: cluster 2021-02-10T04:45:35.390395+0000 osd.4 2021-02-10T04:45:36.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:36 smithi164 conmon[29288]: (osd.4) 1808 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:36.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:36 smithi164 conmon[29288]: cluster 2021-02-10T04:45:35.390401+0000 osd.4 (osd.4) 1809 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:36.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:36 smithi164 conmon[29288]: cluster 2021-02-10T04:45:35.390404+0000 osd.4 ( 2021-02-10T04:45:36.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:36 smithi164 conmon[29288]: osd.4) 1810 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:36.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:36 smithi164 conmon[29288]: cluster 2021-02-10T04:45:36.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:36 smithi164 conmon[29288]: 2021-02-10T04:45:35.390407+0000 2021-02-10T04:45:36.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:36 smithi164 conmon[29288]: osd.4 (osd.4) 1811 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:36.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:36 smithi164 conmon[29288]: cluster 2021-02-10T04:45:36.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:36 smithi164 conmon[29288]: 2021-02-10T04:45:36.219191+0000 2021-02-10T04:45:36.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:36 smithi164 conmon[29288]: osd.0 (osd.0) 109 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:45:36.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:36 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:45:36.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:36 smithi164 conmon[29288]: 02-10T04:45:36.223278+0000 osd.0 (osd.0) 110 2021-02-10T04:45:36.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:36 smithi164 conmon[29288]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:45:36.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[35961]: cluster 2021-02-10T04:45:35.390395+0000 2021-02-10T04:45:36.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[35961]: osd.4 (osd.4) 1808 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:36.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:45:36.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[35961]: T04:45:35.390401+0000 osd.4 (osd.4 2021-02-10T04:45:36.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[35961]: ) 1809 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:36.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:45:36.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[35961]: 02-10T04:45:35.390404+0000 osd.4 2021-02-10T04:45:36.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[35961]: (osd.4) 1810 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:36.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[35961]: cluster 2021-02-10T04:45:36.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[35961]: 2021-02-10T04:45:35.390407 2021-02-10T04:45:36.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1811 : cluster [WRN] 2021-02-10T04:45:36.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:36.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[35961]: cluster 2021-02-10T04:45:36 2021-02-10T04:45:36.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[35961]: .219191+0000 osd.0 (osd.0) 109 : cluster 2021-02-10T04:45:36.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[35961]: [DBG] purged_snaps scrub starts 2021-02-10T04:45:36.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:45:36.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[35961]: 45:36.223278+0000 osd.0 (osd.0) 2021-02-10T04:45:36.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[35961]: 110 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:45:36.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: cluster 2021-02-10T04:45:35.390395+0000 2021-02-10T04:45:36.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: osd.4 (osd.4) 1808 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:36.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: cluster 2021-02-10T04:45:35 2021-02-10T04:45:36.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: .390401+0000 osd.4 (osd.4) 1809 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:36.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: cluster 2021-02-10T04:45:36.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: 2021-02-10T04:45:36.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: 2021 2021-02-10T04:45:36.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: - 2021-02-10T04:45:36.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: 02 2021-02-10T04:45:36.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: - 2021-02-10T04:45:36.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: 10 2021-02-10T04:45:36.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: T 2021-02-10T04:45:36.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: 04 2021-02-10T04:45:36.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: : 2021-02-10T04:45:36.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: 45 2021-02-10T04:45:36.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: : 2021-02-10T04:45:36.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: 35 2021-02-10T04:45:36.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: . 2021-02-10T04:45:36.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: 390404+0000 osd.4 (osd 2021-02-10T04:45:36.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: .4) 1810 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:36.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: cluster 2021-02-10T04:45:36.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: 2021- 2021-02-10T04:45:36.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: 02-10T04:45:35.390407+0000 osd.4 (osd. 2021-02-10T04:45:36.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: 4) 1811 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:36.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: cluster 2021-02-10T04:45:36.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: 2021-02-10T04:45: 2021-02-10T04:45:36.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: 36.219191+0000 osd.0 (osd.0) 109 2021-02-10T04:45:36.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:45:36.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: cluster 2021-02-10T04:45:36.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:45:36.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: 04:45:36.223278+0000 osd.0 (osd 2021-02-10T04:45:36.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:36 smithi085 conmon[29319]: .0) 110 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:45:37.148 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:45:37.254 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:45:37.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:45:36 smithi164 conmon[40892]: debug 2021-02-10T04:45:37.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:45:36 smithi164 conmon[40892]: 2021-02-10T04:45:36.950+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:45:37.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[35961]: cluster 2021-02-10T04:45:37.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[35961]: 2021-02-10T04:45: 2021-02-10T04:45:37.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[35961]: 36.302277+0000 mgr.y (mgr.14140) 985 : cluster [DBG] pgmap v1743: 178 pgs: 178 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:45:37.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:45:37.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[35961]: 10T04:45:36.381191+0000 osd.4 (osd.4 2021-02-10T04:45:37.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[35961]: ) 1812 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:37.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:45:37.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[35961]: :45:36.381200+0000 osd.4 (osd.4) 1813 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:37.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:45:37.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[35961]: :45:36.381207+0000 osd.4 (osd. 2021-02-10T04:45:37.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[35961]: 4) 1814 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:37.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:45:37.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[35961]: 02-10T04:45:36.381213+0000 2021-02-10T04:45:37.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[35961]: osd.4 (osd.4) 1815 : cluster 2021-02-10T04:45:37.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:37.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:45:37.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[35961]: 10T04:45:36.402086+0000 mon.a (mon.0) 2021-02-10T04:45:37.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[35961]: 2061 : cluster [DBG] osdmap e1035: 8 total, 8 up, 7 in 2021-02-10T04:45:37.667 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[29319]: cluster 2021-02-10T04:45:36.302277+0000 mgr.y (mgr.14140) 985 : cluster 2021-02-10T04:45:37.667 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[29319]: [DBG] pgmap v1743: 178 pgs: 178 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:45:37.667 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[29319]: cluster 2021-02-10T04:45:36.381191+0000 osd.4 (osd.4) 1812 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:37.667 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:45:37.667 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[29319]: -10T04:45:36.381200+0000 osd.4 ( 2021-02-10T04:45:37.667 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[29319]: osd.4) 1813 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:37.668 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:45:37.668 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[29319]: T04:45:36.381207+0000 osd.4 (osd.4) 1814 : cluster 2021-02-10T04:45:37.668 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:37.668 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[29319]: cluster 2021-02-10T04:45:36.381213 2021-02-10T04:45:37.668 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1815 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:37.668 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[29319]: cluster 2021-02-10T04:45:36.402086+0000 mon.a 2021-02-10T04:45:37.668 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[29319]: (mon.0) 2061 : cluster [DBG] osdmap e1035: 8 total, 8 up, 7 in 2021-02-10T04:45:37.668 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:45:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:37 smithi164 conmon[29288]: cluster 2021-02-10T04:45:36. 2021-02-10T04:45:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:37 smithi164 conmon[29288]: 302277+0000 mgr.y (mgr.14140) 985 : cluster [DBG] pgmap v1743: 178 pgs: 178 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:45:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:37 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:45:37.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:37 smithi164 conmon[29288]: 10T04:45:36.381191+0000 osd.4 (osd.4) 1812 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:37.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:37 smithi164 conmon[29288]: cluster 2021-02-10T04:45:36.381200+0000 osd.4 (osd.4) 1813 2021-02-10T04:45:37.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:37 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:37.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:37 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:45:37.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:37 smithi164 conmon[29288]: -10T04:45:36.381207+0000 osd.4 (osd.4) 1814 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:37.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:37 smithi164 conmon[29288]: cluster 2021-02-10T04:45:36.381213+0000 2021-02-10T04:45:37.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:37 smithi164 conmon[29288]: osd.4 (osd.4) 1815 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:37.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:37 smithi164 conmon[29288]: cluster 2021-02-10T04:45:36.402086+0000 mon.a (mon.0) 2061 : cluster [DBG] osdmap e1035: 8 total, 8 up, 7 in 2021-02-10T04:45:37.896 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:37 smithi164 conmon[33677]: debug 2021-02-10T04:45:37.399+0000 7fbec19a1700 -1 osd.4 1035 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:45:37.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:45:37 smithi085 conmon[49106]: debug 2021-02-10T04:45:37.814+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:45:38.223 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:45:38.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[33677]: debug 2021-02-10T04:45:38.360+0000 7fbec19a1700 -1 osd.4 1035 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:45:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: cluster 2021-02-10T04:45:37.400063+0000 osd.4 2021-02-10T04:45:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: (osd.4) 1816 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: cluster 2021-02-10T04:45:37.400072+0000 osd.4 (osd. 2021-02-10T04:45:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: 4) 1817 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: cluster 2021-02-10T04:45: 2021-02-10T04:45:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: 37.400078+0000 osd.4 (osd.4) 1818 : cluster [WRN] 2021-02-10T04:45:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: cluster 2021-02-10T04:45: 2021-02-10T04:45:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: 37.400085+0000 osd.4 (osd.4) 1819 : cluster 2021-02-10T04:45:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: audit 2021-02-10T04:45 2021-02-10T04:45:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: :38.167402+0000 mon.a (mon.0) 2062 : audit 2021-02-10T04:45:38.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:45:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:45:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: :45:38.167820+0000 mon.a (mon.0 2021-02-10T04:45:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: ) 2063 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:45:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:45:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: T04:45:38.249815+0000 mon.a (mon 2021-02-10T04:45:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: .0) 2064 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.6", "id": [3, 1]}]: dispatch 2021-02-10T04:45:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:45:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: -10T04:45:38.250135+0000 mon.a (mon.0 2021-02-10T04:45:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: ) 2065 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.10", "id": [7, 5]}]: dispatch 2021-02-10T04:45:38.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:45:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: T04:45:38.250480+0000 mon.a (mon 2021-02-10T04:45:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: .0) 2066 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.12", "id": [7, 5]}]: dispatch 2021-02-10T04:45:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:45:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: 02-10T04:45:38.250784+0000 mon.a 2021-02-10T04:45:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: (mon.0) 2067 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.15", "id": [7, 4]}]: dispatch 2021-02-10T04:45:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: audit 2021 2021-02-10T04:45:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: -02-10T04:45:38.375746+0000 2021-02-10T04:45:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: mon.a (mon.0) 2068 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.6", "id": [3, 1]}]': finished 2021-02-10T04:45:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: audit 2021 2021-02-10T04:45:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: -02-10T04:45:38.375907+0000 2021-02-10T04:45:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: mon.a (mon.0) 2069 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.10", "id": [7, 5]}]': finished 2021-02-10T04:45:38.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:45:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: -10T04:45:38.376111+0000 mon.a (mon.0) 2070 2021-02-10T04:45:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.12", "id": [7, 5]}]': finished 2021-02-10T04:45:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:45:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: 02-10T04:45:38.376289+0000 mon.a 2021-02-10T04:45:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: (mon.0) 2071 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.15", "id": [7, 4]}]': finished 2021-02-10T04:45:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: cluster 2021-02-10T04:45:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: 2021-02-10T04:45:38.376439 2021-02-10T04:45:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2072 : cluster [DBG] 2021-02-10T04:45:38.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[29288]: osdmap e1036: 8 total, 8 up, 7 in 2021-02-10T04:45:38.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: cluster 2021-02-10T04:45:38.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: 2021-02-10T04:45:38.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: 2021 2021-02-10T04:45:38.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: - 2021-02-10T04:45:38.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: 02 2021-02-10T04:45:38.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: - 2021-02-10T04:45:38.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: 10 2021-02-10T04:45:38.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: T 2021-02-10T04:45:38.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: 04 2021-02-10T04:45:38.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: : 2021-02-10T04:45:38.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: 45 2021-02-10T04:45:38.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: : 2021-02-10T04:45:38.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: 37 2021-02-10T04:45:38.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: . 2021-02-10T04:45:38.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: 400063+0000 osd.4 ( 2021-02-10T04:45:38.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: osd.4) 1816 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:38.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: cluster 2021-02-10T04:45:37.400072+0000 osd.4 (osd.4) 1817 2021-02-10T04:45:38.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:38.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:45:38.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: :45:37.400078+0000 osd.4 (osd.4) 1818 : cluster 2021-02-10T04:45:38.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:38.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: cluster 2021-02-10T04:45: 2021-02-10T04:45:38.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: 37.400085+0000 osd.4 (osd.4) 1819 : 2021-02-10T04:45:38.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:38.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:45:38.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: 45:38.167402+0000 mon.a (mon.0 2021-02-10T04:45:38.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: ) 2062 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:45:38.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:45:38.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: T04:45:38.167820+0000 mon.a (mon 2021-02-10T04:45:38.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: .0) 2063 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:45:38.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:45:38.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: 10T04:45:38.249815+0000 mon.a ( 2021-02-10T04:45:38.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: mon.0) 2064 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.6", "id": [3, 1]}]: dispatch 2021-02-10T04:45:38.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: audit 2021-02-10T04:45:38 2021-02-10T04:45:38.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: .250135+0000 mon.a (mon.0) 2065 : 2021-02-10T04:45:38.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.10", "id": [7, 5]}]: dispatch 2021-02-10T04:45:38.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: audit 2021-02-10T04:45 2021-02-10T04:45:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: :38.250480+0000 mon.a (mon.0) 2066 2021-02-10T04:45:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.12", "id": [7, 5]}]: dispatch 2021-02-10T04:45:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:45:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: 04:45:38.250784+0000 mon.a (mon. 2021-02-10T04:45:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: 0) 2067 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.15", "id": [7, 4]}]: dispatch 2021-02-10T04:45:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:45:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: 10T04:45:38.375746+0000 mon.a 2021-02-10T04:45:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: (mon.0) 2068 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.6", "id": [3, 1]}]': finished 2021-02-10T04:45:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: 2021-02-10T04:45:38.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: audit 2021 2021-02-10T04:45:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: -02-10T04:45:38.375907+0000 mon.a (mon.0 2021-02-10T04:45:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: ) 2069 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.10", "id": [7, 5]}]': finished 2021-02-10T04:45:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:45:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: 04:45:38.376111+0000 mon.a (mon. 2021-02-10T04:45:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: 0) 2070 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.12", "id": [7, 5]}]': finished 2021-02-10T04:45:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:45:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: 10T04:45:38.376289+0000 mon.a ( 2021-02-10T04:45:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: mon.0) 2071 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.15", "id": [7, 4]}]': finished 2021-02-10T04:45:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:45:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: -02-10T04:45:38.376439+0000 2021-02-10T04:45:38.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: mon.a (mon.0) 2072 : cluster [DBG] 2021-02-10T04:45:38.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[35961]: osdmap e1036: 8 total, 8 up, 7 in 2021-02-10T04:45:38.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: cluster 2021-02-10T04:45:37.400063+0000 2021-02-10T04:45:38.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: osd.4 (osd.4) 1816 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:38.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: cluster 2021-02-10T04:45:37.400072+0000 osd.4 (osd.4) 1817 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:38.699 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: cluster 2021-02-10T04:45:37.400078+0000 osd.4 (osd 2021-02-10T04:45:38.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: .4) 1818 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:38.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: cluster 2021-02-10T04:45:38.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: 2021 2021-02-10T04:45:38.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: - 2021-02-10T04:45:38.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: 02 2021-02-10T04:45:38.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: - 2021-02-10T04:45:38.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: 10 2021-02-10T04:45:38.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: T 2021-02-10T04:45:38.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: 04 2021-02-10T04:45:38.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: :45:37.400085 2021-02-10T04:45:38.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1819 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:38.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: audit 2021 2021-02-10T04:45:38.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: -02-10T04:45:38.167402+0000 mon.a 2021-02-10T04:45:38.700 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: (mon.0) 2062 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:45:38.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: audit 2021-02-10T04:45:38.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: 2021-02-10T04:45:38.167820+0000 2021-02-10T04:45:38.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: mon.a (mon.0) 2063 : audit [INF] 2021-02-10T04:45:38.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:45:38.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: audit 2021-02-10T04:45:38. 2021-02-10T04:45:38.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: 249815+0000 mon.a (mon.0) 2064 : audit 2021-02-10T04:45:38.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.6", "id": [3, 1]}]: dispatch 2021-02-10T04:45:38.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: audit 2021-02-10T04:45: 2021-02-10T04:45:38.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: 38.250135+0000 mon.a (mon.0) 2065 2021-02-10T04:45:38.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.10", "id": [7, 5]}]: dispatch 2021-02-10T04:45:38.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: audit 2021-02-10T04:45:38.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: 2021-02-10T04:45:38. 2021-02-10T04:45:38.701 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: 250480+0000 mon.a (mon.0) 2066 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.12", "id": [7, 5]}]: dispatch 2021-02-10T04:45:38.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: audit 2021 2021-02-10T04:45:38.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: -02-10T04:45:38.250784+0000 2021-02-10T04:45:38.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: mon.a (mon.0) 2067 : audit [INF] 2021-02-10T04:45:38.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.15", "id": [7, 4]}]: dispatch 2021-02-10T04:45:38.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:45:38.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: 10T04:45:38.375746+0000 mon.a (mon.0) 2068 : 2021-02-10T04:45:38.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.6", "id": [3, 1]}]': finished 2021-02-10T04:45:38.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: audit 2021-02-10T04:45:38 2021-02-10T04:45:38.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: .375907+0000 mon.a (mon.0) 2069 : audit [INF] 2021-02-10T04:45:38.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.10", "id": [7, 5]}]': finished 2021-02-10T04:45:38.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: audit 2021-02-10T04:45:38.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: 2021-02-10T04:45:38.376111+0000 mon.a (mon.0 2021-02-10T04:45:38.702 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: ) 2070 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.12", "id": [7, 5]}]': finished 2021-02-10T04:45:38.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:45:38.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: 02-10T04:45:38.376289+0000 2021-02-10T04:45:38.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: mon.a (mon.0) 2071 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.15", "id": [7, 4]}]': finished 2021-02-10T04:45:38.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: 2021-02-10T04:45:38.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: cluster 2021-02-10T04:45:38.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: 2021-02-10T04:45:38.376439+0000 2021-02-10T04:45:38.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: mon.a (mon.0) 2072 : cluster [DBG] 2021-02-10T04:45:38.703 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:38 smithi085 conmon[29319]: osdmap e1036: 8 total, 8 up, 7 in 2021-02-10T04:45:38.966 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:45:38 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:45:39.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[35961]: cluster 2021-02-10T04:45:39.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[35961]: 2021-02-10T04:45:38 2021-02-10T04:45:39.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[35961]: .302924+0000 mgr.y (mgr.14140) 986 : cluster [DBG] pgmap v1745: 178 pgs: 178 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:45:39.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[35961]: cluster 2021-02-10T04:45:38.360733+0000 osd.4 (osd.4) 1820 : cluster [WRN] 2021-02-10T04:45:39.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:39.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[35961]: cluster 2021-02-10T04:45:38 2021-02-10T04:45:39.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[35961]: .360741+0000 osd.4 (osd.4) 1821 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:39.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[35961]: cluster 2021-02-10T04:45 2021-02-10T04:45:39.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[35961]: :38.360745+0000 osd.4 (osd.4) 1822 : cluster [WRN] 2021-02-10T04:45:39.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[35961]: cluster 2021-02-10T04:45:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[35961]: 2021-02-10T04:45:38.360749+0000 osd.4 (osd.4 2021-02-10T04:45:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[35961]: ) 1823 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:45:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[35961]: 10T04:45:38.397951+0000 mon.a (mon. 2021-02-10T04:45:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[35961]: 0) 2073 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 71 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:45:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[35961]: audit 2021-02-10T04:45:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[35961]: 2021-02-10T04:45:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[35961]: 2021-02-10T04:45: 2021-02-10T04:45:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[35961]: 38.713391+0000 mgr.y (mgr.14140) 987 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:45:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[35961]: cluster 2021-02-10T04:45:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[35961]: 2021-02-10T04:45:39.391175+0000 2021-02-10T04:45:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[35961]: mon.a (mon.0) 2074 : cluster [DBG] osdmap e1037: 8 total, 8 up, 7 in 2021-02-10T04:45:39.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[35961]: 2021-02-10T04:45:39.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:45:39.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[29319]: 45:38.302924+0000 mgr.y (mgr.14140) 986 : cluster [DBG] pgmap v1745: 178 pgs: 178 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:45:39.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[29319]: cluster 2021-02-10T04:45:38.360733+0000 osd.4 (osd 2021-02-10T04:45:39.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[29319]: .4) 1820 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:39.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:45:39.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[29319]: 02-10T04:45:38.360741+0000 osd.4 (osd.4) 1821 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:39.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[29319]: 2021-02-10T04:45:39.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:45:39.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[29319]: 02-10T04:45:38.360745+0000 osd.4 (osd.4) 1822 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:39.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[29319]: cluster 2021-02-10T04:45 2021-02-10T04:45:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[29319]: :38.360749+0000 osd.4 (osd.4) 1823 : 2021-02-10T04:45:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[29319]: cluster 2021-02-10T04:45:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:45:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[29319]: :45:38.397951+0000 mon.a (mon.0) 2073 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 71 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:45:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[29319]: 2021-02-10T04:45:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[29319]: audit 2021-02-10T04: 2021-02-10T04:45:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[29319]: 45:38.713391+0000 mgr.y (mgr.14140) 987 : audit 2021-02-10T04:45:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[29319]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:45:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[29319]: cluster 2021-02-10T04:45: 2021-02-10T04:45:39.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[29319]: 39.391175+0000 mon.a (mon.0) 2074 : cluster 2021-02-10T04:45:39.697 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:39 smithi085 conmon[29319]: [DBG] osdmap e1037: 8 total, 8 up, 7 in 2021-02-10T04:45:39.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:39 smithi164 conmon[29288]: cluster 2021-02-10T04:45:38.302924+0000 mgr.y 2021-02-10T04:45:39.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:39 smithi164 conmon[29288]: (mgr.14140) 986 : cluster [DBG] pgmap v1745: 178 pgs: 178 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:45:39.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:39 smithi164 conmon[29288]: cluster 2021-02-10T04:45:38.360733+0000 osd.4 (osd.4) 1820 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:39.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:39 smithi164 conmon[29288]: cluster 2021-02-10T04:45:38.360741+0000 osd.4 (osd.4) 1821 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:39.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:39 smithi164 conmon[29288]: cluster 2021-02-10T04:45:38.360745+0000 osd.4 (osd.4) 1822 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:39.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:39 smithi164 conmon[29288]: cluster 2021-02-10T04:45:38.360749+0000 osd.4 (osd.4) 1823 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:39.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:39 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:45:39.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:39 smithi164 conmon[29288]: -02-10T04:45:38.397951+0000 mon.a 2021-02-10T04:45:39.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:39 smithi164 conmon[29288]: (mon.0) 2073 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 71 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:45:39.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:39 smithi164 conmon[29288]: audit 2021-02-10T04: 2021-02-10T04:45:39.872 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:39 smithi164 conmon[29288]: 45:38.713391+0000 mgr.y (mgr.14140) 2021-02-10T04:45:39.872 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:39 smithi164 conmon[29288]: 987 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:45:39.872 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:39 smithi164 conmon[29288]: cluster 2021-02-10T04:45:39.872 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:39 smithi164 conmon[29288]: 2021-02-10T04:45:39.391175 2021-02-10T04:45:39.872 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:39 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2074 : cluster [DBG] 2021-02-10T04:45:39.872 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:39 smithi164 conmon[29288]: osdmap e1037: 8 total, 8 up, 7 in 2021-02-10T04:45:39.873 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:39 smithi164 conmon[33677]: debug 2021-02-10T04:45:39.873 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:39 smithi164 conmon[33677]: 2021-02-10T04:45:39.408+0000 7fbec19a1700 -1 osd.4 1037 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:45:40.444 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:45:40.445 INFO:teuthology.orchestra.run.smithi164.stdout: "size": 20, 2021-02-10T04:45:40.445 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 600, 2021-02-10T04:45:40.445 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [ 2021-02-10T04:45:40.445 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.445 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:5977 7.b 7:db4ad6c0:::smithi08575397-12:head [copy-from ver 693] snapc 1f4=[1f4,1f3,1f2,1f0,1ef,1ee,1e9,1e2,1dc,1db,1d3,1d0,1c4,1c1,1b6,1b2,1af,1ac,19a,181,17e,16e,15e] ondisk+write+known_if_redirected e605)", 2021-02-10T04:45:40.445 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:37:03.213966+0000", 2021-02-10T04:45:40.445 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 517.22550530800004, 2021-02-10T04:45:40.445 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.074004782000000005, 2021-02-10T04:45:40.445 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:45:40.445 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:45:40.446 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:45:40.446 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:45:40.446 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:45:40.446 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 5977 2021-02-10T04:45:40.446 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.446 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:45:40.446 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.446 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:45:40.446 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:03.213966+0000", 2021-02-10T04:45:40.446 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:45:40.446 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.446 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.446 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:45:40.447 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:03.213966+0000", 2021-02-10T04:45:40.447 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.4237999999999999e-05 2021-02-10T04:45:40.447 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.447 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.447 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:45:40.447 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:03.214010+0000", 2021-02-10T04:45:40.447 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.3990000000000004e-06 2021-02-10T04:45:40.447 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.447 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.447 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:45:40.447 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:03.214019+0000", 2021-02-10T04:45:40.447 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4249e-05 2021-02-10T04:45:40.447 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.447 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.448 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:45:40.448 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:03.214033+0000", 2021-02-10T04:45:40.448 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.4530000000000002e-06 2021-02-10T04:45:40.448 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.448 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.448 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:45:40.448 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:03.214041+0000", 2021-02-10T04:45:40.448 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.7387999999999999e-05 2021-02-10T04:45:40.448 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.448 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.448 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.448 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:03.214109+0000", 2021-02-10T04:45:40.449 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00014385 2021-02-10T04:45:40.449 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.449 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.449 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:45:40.449 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:03.214253+0000", 2021-02-10T04:45:40.449 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.013495316 2021-02-10T04:45:40.449 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.449 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.449 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 1,6", 2021-02-10T04:45:40.449 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:03.227748+0000", 2021-02-10T04:45:40.450 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.019229670000000001 2021-02-10T04:45:40.450 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.450 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.450 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:45:40.450 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:03.246978+0000", 2021-02-10T04:45:40.450 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001582113 2021-02-10T04:45:40.450 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.450 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.450 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:45:40.450 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:03.248560+0000", 2021-02-10T04:45:40.451 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.039233241000000002 2021-02-10T04:45:40.451 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.451 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.451 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:45:40.451 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:03.287793+0000", 2021-02-10T04:45:40.451 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.4778999999999997e-05 2021-02-10T04:45:40.451 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.451 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.451 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:45:40.451 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:03.287868+0000", 2021-02-10T04:45:40.451 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000103086 2021-02-10T04:45:40.451 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.451 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.452 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:45:40.452 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:03.287971+0000", 2021-02-10T04:45:40.452 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.074004782000000005 2021-02-10T04:45:40.452 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.452 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:45:40.452 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.452 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.452 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.452 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:5978 7.b 7:db4ad6c0:::smithi08575397-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e605)", 2021-02-10T04:45:40.452 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:37:03.214089+0000", 2021-02-10T04:45:40.452 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 517.22538252100003, 2021-02-10T04:45:40.452 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.074149581000000006, 2021-02-10T04:45:40.452 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:45:40.453 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:45:40.453 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:45:40.453 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:45:40.453 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:45:40.453 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 5978 2021-02-10T04:45:40.453 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.453 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:45:40.453 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.453 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:45:40.453 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:03.214089+0000", 2021-02-10T04:45:40.453 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:45:40.453 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.453 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.454 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:45:40.454 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:03.214089+0000", 2021-02-10T04:45:40.454 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.0979999999999998e-05 2021-02-10T04:45:40.454 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.454 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.454 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:45:40.454 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:03.214120+0000", 2021-02-10T04:45:40.454 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.7880000000000003e-06 2021-02-10T04:45:40.454 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.454 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.454 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:45:40.454 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:03.214126+0000", 2021-02-10T04:45:40.455 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0893999999999999e-05 2021-02-10T04:45:40.455 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.455 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.455 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:45:40.455 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:03.214137+0000", 2021-02-10T04:45:40.455 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.809e-06 2021-02-10T04:45:40.455 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.455 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.455 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:45:40.455 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:03.214141+0000", 2021-02-10T04:45:40.455 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00068127499999999996 2021-02-10T04:45:40.455 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.455 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.456 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.456 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:03.214823+0000", 2021-02-10T04:45:40.456 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6918e-05 2021-02-10T04:45:40.456 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.456 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.456 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:45:40.456 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:03.214880+0000", 2021-02-10T04:45:40.456 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.016968496 2021-02-10T04:45:40.456 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.456 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.456 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.456 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:03.231848+0000", 2021-02-10T04:45:40.456 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000106446 2021-02-10T04:45:40.457 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.457 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.457 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2021-02-10T04:45:40.457 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:03.231954+0000", 2021-02-10T04:45:40.457 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.056052795000000002 2021-02-10T04:45:40.457 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.457 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.457 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.457 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:03.288007+0000", 2021-02-10T04:45:40.457 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000107531 2021-02-10T04:45:40.457 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.457 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.458 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:45:40.458 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:03.288115+0000", 2021-02-10T04:45:40.458 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00012364900000000001 2021-02-10T04:45:40.458 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.458 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.458 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:45:40.458 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:37:03.288238+0000", 2021-02-10T04:45:40.458 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.074149581000000006 2021-02-10T04:45:40.458 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.459 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:45:40.459 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.459 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.459 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.459 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:7209 7.e 7:70c651f0:::smithi08575397-15:head [copy-from ver 1340] snapc 26c=[26c,268,267,266,260,25f,25e,25b,257,255,254,251,24f,24e,249,248,245,242,241,23f,23d,234,231,22f,22a,221,21c,211,203,1c1,1b6] ondisk+write+known_if_redirected e725)", 2021-02-10T04:45:40.459 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:39:04.388691+0000", 2021-02-10T04:45:40.459 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 396.05078035399998, 2021-02-10T04:45:40.459 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.064434799000000001, 2021-02-10T04:45:40.459 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:45:40.459 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:45:40.459 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:45:40.459 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:45:40.460 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:45:40.460 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 7209 2021-02-10T04:45:40.460 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.460 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:45:40.460 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.460 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:45:40.460 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:04.388691+0000", 2021-02-10T04:45:40.460 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:45:40.460 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.460 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.460 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:45:40.460 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:04.388691+0000", 2021-02-10T04:45:40.461 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.8244000000000001e-05 2021-02-10T04:45:40.461 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.461 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.461 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:45:40.461 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:04.388739+0000", 2021-02-10T04:45:40.461 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.0650000000000005e-06 2021-02-10T04:45:40.461 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.461 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.461 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:45:40.461 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:04.388748+0000", 2021-02-10T04:45:40.461 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3604000000000001e-05 2021-02-10T04:45:40.461 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.461 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.462 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:45:40.462 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:04.388762+0000", 2021-02-10T04:45:40.462 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.4309999999999994e-06 2021-02-10T04:45:40.462 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.462 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.462 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:45:40.462 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:04.388770+0000", 2021-02-10T04:45:40.462 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.9959e-05 2021-02-10T04:45:40.462 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.462 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.462 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.462 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:04.388830+0000", 2021-02-10T04:45:40.462 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00015729900000000001 2021-02-10T04:45:40.463 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.463 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.463 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:45:40.463 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:04.388988+0000", 2021-02-10T04:45:40.463 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.064138196999999994 2021-02-10T04:45:40.463 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.463 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.463 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:45:40.463 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:04.453126+0000", 2021-02-10T04:45:40.463 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.064434799000000001 2021-02-10T04:45:40.463 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.463 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:45:40.463 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.464 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.464 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.464 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:7210 7.e 7:70c651f0:::smithi08575397-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e725)", 2021-02-10T04:45:40.464 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:39:04.388826+0000", 2021-02-10T04:45:40.464 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 396.05064537200002, 2021-02-10T04:45:40.464 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.064658860999999998, 2021-02-10T04:45:40.464 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:45:40.464 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:45:40.464 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:45:40.464 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:45:40.464 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:45:40.464 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 7210 2021-02-10T04:45:40.464 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.465 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:45:40.465 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.465 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:45:40.465 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:04.388826+0000", 2021-02-10T04:45:40.465 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:45:40.465 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.465 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.465 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:45:40.465 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:04.388826+0000", 2021-02-10T04:45:40.465 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.0422000000000001e-05 2021-02-10T04:45:40.465 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.465 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.465 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:45:40.466 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:04.388856+0000", 2021-02-10T04:45:40.466 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.3740000000000003e-06 2021-02-10T04:45:40.466 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.466 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.466 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:45:40.466 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:04.388863+0000", 2021-02-10T04:45:40.466 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0675e-05 2021-02-10T04:45:40.466 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.466 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.466 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:45:40.466 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:04.388873+0000", 2021-02-10T04:45:40.466 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.8550000000000001e-06 2021-02-10T04:45:40.466 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.467 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.467 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:45:40.467 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:04.388878+0000", 2021-02-10T04:45:40.467 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00067541200000000002 2021-02-10T04:45:40.467 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.467 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.467 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.467 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:04.389554+0000", 2021-02-10T04:45:40.467 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.2668000000000004e-05 2021-02-10T04:45:40.467 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.467 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.467 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:45:40.468 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:04.389616+0000", 2021-02-10T04:45:40.468 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.063600313000000006 2021-02-10T04:45:40.468 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.468 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.468 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.468 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:04.453217+0000", 2021-02-10T04:45:40.468 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000120889 2021-02-10T04:45:40.468 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.468 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.468 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:45:40.468 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:04.453338+0000", 2021-02-10T04:45:40.468 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00014725299999999999 2021-02-10T04:45:40.469 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.469 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.469 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:45:40.469 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:04.453485+0000", 2021-02-10T04:45:40.469 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.064658860999999998 2021-02-10T04:45:40.469 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.469 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:45:40.469 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.469 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.469 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.469 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:7969 7.0 7:03f22018:::smithi08575397-13:head [copy-from ver 1469] snapc 2b4=[2b4,2b3,2b2,2b1,2ae,2ad,2ac,2ab,2aa,2a9,2a7,2a5,2a3,2a2,2a1,29d,298,295,294,28f,28b,289,27c,26d,24e,249,245,231,21c,203] ondisk+write+known_if_redirected e799)", 2021-02-10T04:45:40.469 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:40:18.402923+0000", 2021-02-10T04:45:40.469 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 322.03654788799997, 2021-02-10T04:45:40.470 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.121396403, 2021-02-10T04:45:40.470 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:45:40.470 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:45:40.470 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:45:40.470 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:45:40.470 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:45:40.470 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 7969 2021-02-10T04:45:40.470 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.470 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:45:40.470 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.470 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:45:40.470 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:18.402923+0000", 2021-02-10T04:45:40.470 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:45:40.471 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.471 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.471 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:45:40.471 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:18.402923+0000", 2021-02-10T04:45:40.471 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.8294999999999998e-05 2021-02-10T04:45:40.471 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.471 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.471 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:45:40.471 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:18.402962+0000", 2021-02-10T04:45:40.471 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.1410000000000002e-06 2021-02-10T04:45:40.471 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.471 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.471 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:45:40.472 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:18.402968+0000", 2021-02-10T04:45:40.472 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8008000000000001e-05 2021-02-10T04:45:40.472 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.472 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.472 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:45:40.472 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:18.402986+0000", 2021-02-10T04:45:40.472 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.5300000000000004e-06 2021-02-10T04:45:40.472 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.472 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.472 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:45:40.472 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:18.402991+0000", 2021-02-10T04:45:40.472 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2147999999999997e-05 2021-02-10T04:45:40.472 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.473 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.473 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.473 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:18.403034+0000", 2021-02-10T04:45:40.473 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.5394000000000004e-05 2021-02-10T04:45:40.473 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.473 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.473 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:45:40.473 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:18.403109+0000", 2021-02-10T04:45:40.473 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.10676089599999999 2021-02-10T04:45:40.473 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.473 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.473 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 1,3", 2021-02-10T04:45:40.473 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:18.509870+0000", 2021-02-10T04:45:40.474 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0066581390000000004 2021-02-10T04:45:40.474 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.474 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.474 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:45:40.474 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:18.516528+0000", 2021-02-10T04:45:40.474 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.007278723 2021-02-10T04:45:40.474 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.474 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.474 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:45:40.474 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:18.523807+0000", 2021-02-10T04:45:40.474 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00041343400000000001 2021-02-10T04:45:40.474 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.474 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.475 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:45:40.475 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:18.524220+0000", 2021-02-10T04:45:40.475 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.6961e-05 2021-02-10T04:45:40.475 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.475 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.475 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:45:40.475 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:18.524267+0000", 2021-02-10T04:45:40.475 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.2734e-05 2021-02-10T04:45:40.475 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.475 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.476 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:45:40.476 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:18.524320+0000", 2021-02-10T04:45:40.476 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.121396403 2021-02-10T04:45:40.476 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.476 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:45:40.476 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.476 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.476 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.476 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:7970 7.0 7:03f22018:::smithi08575397-13:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e799)", 2021-02-10T04:45:40.476 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:40:18.403032+0000", 2021-02-10T04:45:40.476 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 322.03643899600002, 2021-02-10T04:45:40.476 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.121514312, 2021-02-10T04:45:40.476 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:45:40.477 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:45:40.477 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:45:40.477 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:45:40.477 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:45:40.477 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 7970 2021-02-10T04:45:40.477 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.477 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:45:40.477 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.477 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:45:40.477 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:18.403032+0000", 2021-02-10T04:45:40.477 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:45:40.477 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.477 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.478 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:45:40.478 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:18.403032+0000", 2021-02-10T04:45:40.478 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.5953e-05 2021-02-10T04:45:40.478 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.478 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.478 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:45:40.478 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:18.403058+0000", 2021-02-10T04:45:40.478 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9810000000000003e-06 2021-02-10T04:45:40.478 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.478 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.478 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:45:40.478 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:18.403063+0000", 2021-02-10T04:45:40.479 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0601999999999999e-05 2021-02-10T04:45:40.479 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.479 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.479 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:45:40.479 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:18.403074+0000", 2021-02-10T04:45:40.479 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9899999999999997e-06 2021-02-10T04:45:40.479 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.479 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.479 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:45:40.479 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:18.403079+0000", 2021-02-10T04:45:40.479 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00034261300000000002 2021-02-10T04:45:40.479 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.479 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.480 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.480 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:18.403422+0000", 2021-02-10T04:45:40.480 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.9505999999999998e-05 2021-02-10T04:45:40.480 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.480 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.480 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:45:40.480 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:18.403451+0000", 2021-02-10T04:45:40.480 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.111036014 2021-02-10T04:45:40.480 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.480 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.480 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.480 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:18.514487+0000", 2021-02-10T04:45:40.481 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.7698e-05 2021-02-10T04:45:40.481 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.481 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.481 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2021-02-10T04:45:40.481 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:18.514575+0000", 2021-02-10T04:45:40.481 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0097946649999999993 2021-02-10T04:45:40.481 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.481 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.481 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.481 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:18.524369+0000", 2021-02-10T04:45:40.481 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.2468999999999999e-05 2021-02-10T04:45:40.481 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.482 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.482 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:45:40.482 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:18.524452+0000", 2021-02-10T04:45:40.482 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.4821000000000006e-05 2021-02-10T04:45:40.482 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.482 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.482 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:45:40.482 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:18.524547+0000", 2021-02-10T04:45:40.482 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.121514312 2021-02-10T04:45:40.482 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.482 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:45:40.482 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.482 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.483 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.483 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:8302 7.b 7:d39fd0bb:::smithi08575397-9:head [write 474261~408518] snapc 2d7=[2d7,2d6,2d5,2d3,2d2,2d1,2cb,2ca,2c9,2c8,2c7,2c1,2b4,2b3,2b1,2ae,2ac,2aa,2a9,2a7,2a5,2a2,2a1,298,295,294,289,27c,26d,24e,231,21c,203] ondisk+write+known_if_redirected e834)", 2021-02-10T04:45:40.483 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:40:53.313446+0000", 2021-02-10T04:45:40.483 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 287.12602503699998, 2021-02-10T04:45:40.483 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.064590823000000006, 2021-02-10T04:45:40.483 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:45:40.483 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:45:40.483 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:45:40.483 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:45:40.483 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:45:40.483 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 8302 2021-02-10T04:45:40.483 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.483 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:45:40.484 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.484 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:45:40.484 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.313446+0000", 2021-02-10T04:45:40.484 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:45:40.484 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.484 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.484 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:45:40.484 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.313446+0000", 2021-02-10T04:45:40.484 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00046669600000000001 2021-02-10T04:45:40.484 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.484 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.484 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:45:40.484 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.313913+0000", 2021-02-10T04:45:40.485 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0580000000000003e-06 2021-02-10T04:45:40.485 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.485 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.485 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:45:40.485 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.313919+0000", 2021-02-10T04:45:40.485 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0363000000000001e-05 2021-02-10T04:45:40.485 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.485 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.485 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:45:40.485 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.313929+0000", 2021-02-10T04:45:40.485 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.4680000000000003e-06 2021-02-10T04:45:40.485 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.485 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.486 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:45:40.486 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.313937+0000", 2021-02-10T04:45:40.486 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.6767999999999997e-05 2021-02-10T04:45:40.486 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.486 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.486 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.486 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.313984+0000", 2021-02-10T04:45:40.486 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00010312800000000001 2021-02-10T04:45:40.486 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.486 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.486 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:45:40.486 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.314087+0000", 2021-02-10T04:45:40.486 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00025869700000000001 2021-02-10T04:45:40.487 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.487 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.487 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 1,6", 2021-02-10T04:45:40.487 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.314346+0000", 2021-02-10T04:45:40.487 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.022387872999999999 2021-02-10T04:45:40.487 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.487 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.487 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:45:40.487 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.336733+0000", 2021-02-10T04:45:40.487 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2909e-05 2021-02-10T04:45:40.487 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.487 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.487 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:45:40.487 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.336756+0000", 2021-02-10T04:45:40.488 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.041157053999999998 2021-02-10T04:45:40.488 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.488 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.488 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:45:40.488 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.377913+0000", 2021-02-10T04:45:40.488 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.5475e-05 2021-02-10T04:45:40.488 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.488 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.488 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:45:40.488 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.377969+0000", 2021-02-10T04:45:40.488 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.8334e-05 2021-02-10T04:45:40.488 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.488 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.489 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:45:40.489 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.378037+0000", 2021-02-10T04:45:40.489 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.064590823000000006 2021-02-10T04:45:40.489 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.489 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:45:40.489 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.489 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.489 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.489 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:8303 7.b 7:d39fd0bb:::smithi08575397-9:head [write 1582102~449729] snapc 2d7=[2d7,2d6,2d5,2d3,2d2,2d1,2cb,2ca,2c9,2c8,2c7,2c1,2b4,2b3,2b1,2ae,2ac,2aa,2a9,2a7,2a5,2a2,2a1,298,295,294,289,27c,26d,24e,231,21c,203] ondisk+write+known_if_redirected e834)", 2021-02-10T04:45:40.489 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:40:53.321478+0000", 2021-02-10T04:45:40.489 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 287.11799334699998, 2021-02-10T04:45:40.489 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.060561006000000001, 2021-02-10T04:45:40.489 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:45:40.490 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:45:40.490 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:45:40.490 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:45:40.490 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:45:40.490 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 8303 2021-02-10T04:45:40.490 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.490 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:45:40.490 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.490 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:45:40.490 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.321478+0000", 2021-02-10T04:45:40.490 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:45:40.490 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.491 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.491 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:45:40.491 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.321478+0000", 2021-02-10T04:45:40.491 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00039360399999999998 2021-02-10T04:45:40.491 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.491 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.491 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:45:40.491 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.321872+0000", 2021-02-10T04:45:40.491 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.3459999999999999e-06 2021-02-10T04:45:40.491 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.491 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.491 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:45:40.491 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.321878+0000", 2021-02-10T04:45:40.492 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.4220000000000006e-06 2021-02-10T04:45:40.492 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.492 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.492 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:45:40.492 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.321887+0000", 2021-02-10T04:45:40.492 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.3180000000000001e-06 2021-02-10T04:45:40.492 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.492 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.492 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:45:40.492 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.321895+0000", 2021-02-10T04:45:40.492 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.2172999999999997e-05 2021-02-10T04:45:40.492 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.492 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.493 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.493 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.321947+0000", 2021-02-10T04:45:40.493 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000125661 2021-02-10T04:45:40.493 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.493 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.493 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:45:40.493 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.322073+0000", 2021-02-10T04:45:40.493 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00023375199999999999 2021-02-10T04:45:40.493 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.493 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.493 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 1,6", 2021-02-10T04:45:40.493 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.322306+0000", 2021-02-10T04:45:40.493 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.014471845000000001 2021-02-10T04:45:40.493 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.494 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.494 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:45:40.494 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.336778+0000", 2021-02-10T04:45:40.494 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00033012699999999999 2021-02-10T04:45:40.494 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.494 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.494 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:45:40.494 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.337108+0000", 2021-02-10T04:45:40.494 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.044771285000000001 2021-02-10T04:45:40.494 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.494 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.494 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:45:40.495 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.381880+0000", 2021-02-10T04:45:40.495 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1068000000000004e-05 2021-02-10T04:45:40.495 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.495 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.495 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:45:40.495 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.381961+0000", 2021-02-10T04:45:40.495 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.8405000000000006e-05 2021-02-10T04:45:40.495 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.495 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.495 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:45:40.495 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.382039+0000", 2021-02-10T04:45:40.495 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.060561006000000001 2021-02-10T04:45:40.495 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.495 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:45:40.496 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.496 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.496 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.496 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:8304 7.b 7:d39fd0bb:::smithi08575397-9:head [write 2739315~134946] snapc 2d7=[2d7,2d6,2d5,2d3,2d2,2d1,2cb,2ca,2c9,2c8,2c7,2c1,2b4,2b3,2b1,2ae,2ac,2aa,2a9,2a7,2a5,2a2,2a1,298,295,294,289,27c,26d,24e,231,21c,203] ondisk+write+known_if_redirected e834)", 2021-02-10T04:45:40.496 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:40:53.326742+0000", 2021-02-10T04:45:40.496 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 287.112729092, 2021-02-10T04:45:40.496 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.055480234000000003, 2021-02-10T04:45:40.496 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:45:40.496 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:45:40.496 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:45:40.496 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:45:40.496 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:45:40.496 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 8304 2021-02-10T04:45:40.497 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.497 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:45:40.497 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.497 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:45:40.497 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.326742+0000", 2021-02-10T04:45:40.497 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:45:40.497 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.497 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.497 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:45:40.497 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.326742+0000", 2021-02-10T04:45:40.497 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00013474099999999999 2021-02-10T04:45:40.497 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.497 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.498 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:45:40.498 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.326877+0000", 2021-02-10T04:45:40.498 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.3509999999999996e-06 2021-02-10T04:45:40.498 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.498 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.498 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:45:40.498 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.326884+0000", 2021-02-10T04:45:40.498 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6415000000000001e-05 2021-02-10T04:45:40.498 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.498 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.498 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:45:40.498 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.326901+0000", 2021-02-10T04:45:40.499 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0730000000000001e-06 2021-02-10T04:45:40.499 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.499 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.499 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:45:40.499 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.326907+0000", 2021-02-10T04:45:40.499 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.1187999999999998e-05 2021-02-10T04:45:40.499 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.499 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.499 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.499 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.326958+0000", 2021-02-10T04:45:40.499 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000120457 2021-02-10T04:45:40.499 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.499 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.499 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:45:40.500 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.327079+0000", 2021-02-10T04:45:40.500 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000156064 2021-02-10T04:45:40.500 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.500 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.500 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 1,6", 2021-02-10T04:45:40.500 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.327235+0000", 2021-02-10T04:45:40.500 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0095474779999999999 2021-02-10T04:45:40.500 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.500 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.500 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:45:40.500 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.336782+0000", 2021-02-10T04:45:40.500 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00038976400000000002 2021-02-10T04:45:40.500 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.501 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.501 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:45:40.501 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.337172+0000", 2021-02-10T04:45:40.501 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.044917601000000001 2021-02-10T04:45:40.501 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.501 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.501 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:45:40.501 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.382089+0000", 2021-02-10T04:45:40.501 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.7540999999999996e-05 2021-02-10T04:45:40.501 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.501 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.501 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:45:40.501 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.382157+0000", 2021-02-10T04:45:40.502 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.5561000000000005e-05 2021-02-10T04:45:40.502 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.502 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.502 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:45:40.502 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.382223+0000", 2021-02-10T04:45:40.502 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.055480234000000003 2021-02-10T04:45:40.502 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.502 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:45:40.502 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.502 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.502 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.502 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:8305 7.b 7:d39fd0bb:::smithi08575397-9:head [setxattr _header (57) in=7b,truncate 2874261] snapc 2d7=[2d7,2d6,2d5,2d3,2d2,2d1,2cb,2ca,2c9,2c8,2c7,2c1,2b4,2b3,2b1,2ae,2ac,2aa,2a9,2a7,2a5,2a2,2a1,298,295,294,289,27c,26d,24e,231,21c,203] ondisk+write+known_if_redirected e834)", 2021-02-10T04:45:40.502 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:40:53.326948+0000", 2021-02-10T04:45:40.503 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 287.11252293199999, 2021-02-10T04:45:40.503 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.055418584999999999, 2021-02-10T04:45:40.503 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:45:40.503 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:45:40.503 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:45:40.503 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:45:40.503 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:45:40.503 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 8305 2021-02-10T04:45:40.503 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.503 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:45:40.503 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.503 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:45:40.503 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.326948+0000", 2021-02-10T04:45:40.504 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:45:40.504 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.504 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.504 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:45:40.504 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.326948+0000", 2021-02-10T04:45:40.504 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2035e-05 2021-02-10T04:45:40.504 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.504 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.504 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:45:40.504 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.326990+0000", 2021-02-10T04:45:40.504 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0800000000000002e-06 2021-02-10T04:45:40.505 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.505 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.505 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:45:40.505 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.326997+0000", 2021-02-10T04:45:40.505 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0919e-05 2021-02-10T04:45:40.505 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.505 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.505 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:45:40.505 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.327007+0000", 2021-02-10T04:45:40.505 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.3560000000000003e-06 2021-02-10T04:45:40.506 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.506 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.506 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:45:40.506 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.327012+0000", 2021-02-10T04:45:40.506 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0012005640000000001 2021-02-10T04:45:40.506 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.506 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.506 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.506 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.328212+0000", 2021-02-10T04:45:40.506 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000125442 2021-02-10T04:45:40.506 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.506 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.507 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:45:40.507 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.328338+0000", 2021-02-10T04:45:40.507 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000208056 2021-02-10T04:45:40.507 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.507 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.507 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 1,6", 2021-02-10T04:45:40.507 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.328546+0000", 2021-02-10T04:45:40.507 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0082398899999999997 2021-02-10T04:45:40.507 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.507 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.507 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:45:40.507 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.336786+0000", 2021-02-10T04:45:40.507 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00048937199999999996 2021-02-10T04:45:40.508 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.508 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.508 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:45:40.508 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.337275+0000", 2021-02-10T04:45:40.508 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.045001724999999999 2021-02-10T04:45:40.508 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.508 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.508 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:45:40.508 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.382277+0000", 2021-02-10T04:45:40.508 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.4227000000000002e-05 2021-02-10T04:45:40.508 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.508 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.508 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:45:40.509 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.382321+0000", 2021-02-10T04:45:40.509 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5918999999999999e-05 2021-02-10T04:45:40.509 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.509 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.509 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:45:40.509 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.382367+0000", 2021-02-10T04:45:40.509 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.055418584999999999 2021-02-10T04:45:40.509 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.509 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:45:40.509 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.509 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.509 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.509 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:8306 7.b 7:d39fd0bb:::smithi08575397-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e834)", 2021-02-10T04:45:40.510 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:40:53.327049+0000", 2021-02-10T04:45:40.510 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 287.112422804, 2021-02-10T04:45:40.510 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.055504255000000002, 2021-02-10T04:45:40.510 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:45:40.510 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:45:40.510 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:45:40.510 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:45:40.510 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:45:40.510 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 8306 2021-02-10T04:45:40.510 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.510 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:45:40.510 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.510 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:45:40.511 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.327049+0000", 2021-02-10T04:45:40.511 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:45:40.511 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.511 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.511 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:45:40.511 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.327049+0000", 2021-02-10T04:45:40.511 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.7494000000000001e-05 2021-02-10T04:45:40.511 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.511 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.511 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:45:40.511 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.327076+0000", 2021-02-10T04:45:40.511 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6149999999999996e-06 2021-02-10T04:45:40.511 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.512 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.512 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:45:40.512 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.327082+0000", 2021-02-10T04:45:40.512 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0869999999999999e-05 2021-02-10T04:45:40.512 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.512 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.512 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:45:40.512 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.327093+0000", 2021-02-10T04:45:40.512 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.8550000000000001e-06 2021-02-10T04:45:40.512 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.512 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.512 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:45:40.512 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.327097+0000", 2021-02-10T04:45:40.512 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.002070119 2021-02-10T04:45:40.513 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.513 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.513 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.513 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.329168+0000", 2021-02-10T04:45:40.513 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.8728999999999996e-05 2021-02-10T04:45:40.513 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.513 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.513 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2021-02-10T04:45:40.513 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.329266+0000", 2021-02-10T04:45:40.513 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.053125838000000002 2021-02-10T04:45:40.513 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.513 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.514 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.514 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.382392+0000", 2021-02-10T04:45:40.514 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.5846000000000005e-05 2021-02-10T04:45:40.514 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.514 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.514 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:45:40.514 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.382468+0000", 2021-02-10T04:45:40.514 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.4889000000000007e-05 2021-02-10T04:45:40.514 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.514 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.514 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:45:40.515 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:53.382553+0000", 2021-02-10T04:45:40.515 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.055504255000000002 2021-02-10T04:45:40.515 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.515 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:45:40.515 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.515 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.515 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.515 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:9146 7.e 7:70c651f0:::smithi08575397-15:head [delete] snapc 328=[328,327,325,31d,31a,311,308,307,300,2fc,2f5,2f0,2ef,2ee,2ed,2eb,2ea,2e3,2e2,2df,2dd,2d9,2cb,2c9,2c8,2ae,2ac,2a2,27c,24e,203] ondisk+write+peerstat_old+known_if_redirected e914)", 2021-02-10T04:45:40.515 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:42:13.317433+0000", 2021-02-10T04:45:40.515 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 207.122038774, 2021-02-10T04:45:40.515 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.074089991999999993, 2021-02-10T04:45:40.515 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:45:40.515 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:45:40.516 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:45:40.516 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:45:40.516 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:45:40.516 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 9146 2021-02-10T04:45:40.516 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.516 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:45:40.516 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.516 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:45:40.516 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.317433+0000", 2021-02-10T04:45:40.516 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:45:40.516 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.516 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.517 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:45:40.517 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.317433+0000", 2021-02-10T04:45:40.517 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.8133999999999998e-05 2021-02-10T04:45:40.517 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.517 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.517 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:45:40.517 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.317471+0000", 2021-02-10T04:45:40.517 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0299999999999999e-06 2021-02-10T04:45:40.517 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.517 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.518 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:45:40.518 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.317477+0000", 2021-02-10T04:45:40.518 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0444e-05 2021-02-10T04:45:40.518 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.518 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.518 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:45:40.518 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.317487+0000", 2021-02-10T04:45:40.518 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.741e-06 2021-02-10T04:45:40.518 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.518 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.518 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:45:40.519 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.317492+0000", 2021-02-10T04:45:40.519 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0055002209999999996 2021-02-10T04:45:40.519 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.519 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.519 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.519 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.322992+0000", 2021-02-10T04:45:40.519 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7151999999999994e-05 2021-02-10T04:45:40.519 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.519 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.519 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:45:40.519 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.323089+0000", 2021-02-10T04:45:40.519 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.068433270000000004 2021-02-10T04:45:40.520 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.520 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.520 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:45:40.520 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.391523+0000", 2021-02-10T04:45:40.520 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.074089991999999993 2021-02-10T04:45:40.520 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.520 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:45:40.520 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.520 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.520 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.520 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:9224 7.b 7:d39fd0bb:::smithi08575397-9:head [copy-from ver 1804] snapc 32b=[32b,32a,328,327,325,31d,31a,311,308,307,300,2fc,2f5,2f0,2ee,2ed,2eb,2ea,2e3,2e2,2df,2dd,2d9,2cb,2c9,2c8,2ae,2ac,2a2,27c,24e,203] ondisk+write+known_if_redirected e917)", 2021-02-10T04:45:40.521 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:42:16.336521+0000", 2021-02-10T04:45:40.521 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 204.102950386, 2021-02-10T04:45:40.521 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.061727572000000001, 2021-02-10T04:45:40.521 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:45:40.521 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:45:40.521 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:45:40.521 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:45:40.521 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:45:40.521 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 9224 2021-02-10T04:45:40.521 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.521 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:45:40.521 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.521 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:45:40.522 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:16.336521+0000", 2021-02-10T04:45:40.522 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:45:40.522 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.522 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.522 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:45:40.522 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:16.336521+0000", 2021-02-10T04:45:40.522 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.3064000000000005e-05 2021-02-10T04:45:40.522 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.522 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.522 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:45:40.522 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:16.336584+0000", 2021-02-10T04:45:40.522 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.4090000000000003e-06 2021-02-10T04:45:40.522 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.523 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.523 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:45:40.523 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:16.336592+0000", 2021-02-10T04:45:40.523 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4022000000000001e-05 2021-02-10T04:45:40.523 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.523 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.523 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:45:40.523 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:16.336606+0000", 2021-02-10T04:45:40.523 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.5629999999999992e-06 2021-02-10T04:45:40.523 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.523 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.523 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:45:40.524 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:16.336615+0000", 2021-02-10T04:45:40.524 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6116999999999998e-05 2021-02-10T04:45:40.524 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.524 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.524 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.524 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:16.336671+0000", 2021-02-10T04:45:40.524 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.8738000000000002e-05 2021-02-10T04:45:40.524 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.524 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.524 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:45:40.524 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:16.336770+0000", 2021-02-10T04:45:40.524 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.054657809000000002 2021-02-10T04:45:40.524 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.525 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.525 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 1,6", 2021-02-10T04:45:40.525 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:16.391428+0000", 2021-02-10T04:45:40.525 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0056630359999999998 2021-02-10T04:45:40.525 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.525 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.525 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:45:40.525 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:16.397091+0000", 2021-02-10T04:45:40.525 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00047696099999999998 2021-02-10T04:45:40.525 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.525 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.525 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:45:40.525 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:16.397568+0000", 2021-02-10T04:45:40.526 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000541742 2021-02-10T04:45:40.526 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.526 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.526 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:45:40.526 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:16.398109+0000", 2021-02-10T04:45:40.526 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.8221999999999998e-05 2021-02-10T04:45:40.526 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.526 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.526 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:45:40.526 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:16.398178+0000", 2021-02-10T04:45:40.526 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.0889000000000005e-05 2021-02-10T04:45:40.527 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.527 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.527 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:45:40.527 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:16.398249+0000", 2021-02-10T04:45:40.527 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.061727572000000001 2021-02-10T04:45:40.527 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.527 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:45:40.527 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.527 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.527 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.527 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:9225 7.b 7:d39fd0bb:::smithi08575397-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e917)", 2021-02-10T04:45:40.527 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:42:16.336663+0000", 2021-02-10T04:45:40.528 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 204.10280842, 2021-02-10T04:45:40.528 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.061778513, 2021-02-10T04:45:40.528 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:45:40.528 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:45:40.528 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:45:40.528 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:45:40.528 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:45:40.528 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 9225 2021-02-10T04:45:40.528 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.528 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:45:40.528 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.528 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:45:40.528 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:16.336663+0000", 2021-02-10T04:45:40.528 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:45:40.529 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.529 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.529 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:45:40.529 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:16.336663+0000", 2021-02-10T04:45:40.529 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.9017e-05 2021-02-10T04:45:40.529 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.529 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.529 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:45:40.529 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:16.336692+0000", 2021-02-10T04:45:40.529 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.93e-06 2021-02-10T04:45:40.529 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.529 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.529 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:45:40.530 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:16.336698+0000", 2021-02-10T04:45:40.530 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1749999999999999e-05 2021-02-10T04:45:40.530 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.530 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.530 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:45:40.530 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:16.336710+0000", 2021-02-10T04:45:40.530 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.1739999999999999e-06 2021-02-10T04:45:40.530 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.530 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.530 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:45:40.530 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:16.336715+0000", 2021-02-10T04:45:40.530 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00068172600000000001 2021-02-10T04:45:40.530 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.531 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.531 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.531 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:16.337397+0000", 2021-02-10T04:45:40.531 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.4029000000000006e-05 2021-02-10T04:45:40.531 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.531 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.531 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:45:40.531 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:16.337471+0000", 2021-02-10T04:45:40.531 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.056769950999999999 2021-02-10T04:45:40.531 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.531 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.531 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.531 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:16.394241+0000", 2021-02-10T04:45:40.532 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.4623999999999996e-05 2021-02-10T04:45:40.532 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.532 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.532 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2021-02-10T04:45:40.532 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:16.394315+0000", 2021-02-10T04:45:40.532 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.003963505 2021-02-10T04:45:40.532 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.532 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.532 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.532 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:16.398279+0000", 2021-02-10T04:45:40.532 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.9268e-05 2021-02-10T04:45:40.532 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.532 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.533 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:45:40.533 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:16.398368+0000", 2021-02-10T04:45:40.533 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.3539000000000002e-05 2021-02-10T04:45:40.533 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.533 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.533 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:45:40.533 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:16.398441+0000", 2021-02-10T04:45:40.533 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.061778513 2021-02-10T04:45:40.533 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.533 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:45:40.533 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.533 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.533 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.534 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:9994 7.0 7:06aed406:::smithi08575397-5:head [setxattr _header (57) in=7b,truncate 2655836] snapc 372=[372,370,36f,36b,368,366,35f,354,353,350,34e,34d,34b,349,348,345,344,343,33e,33c,33a,335,332,32f,32a,328,31a,308,300,2fc,2f5,2f0,2ee,2ed,2eb,2ea,2d9,2c8,2ae,27c,203] ondisk+write+known_if_redirected e988)", 2021-02-10T04:45:40.534 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:43:28.381162+0000", 2021-02-10T04:45:40.534 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 132.058308936, 2021-02-10T04:45:40.534 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.071161233000000004, 2021-02-10T04:45:40.534 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:45:40.534 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:45:40.534 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:45:40.534 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:45:40.534 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:45:40.534 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 9994 2021-02-10T04:45:40.534 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.534 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:45:40.535 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.535 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:45:40.535 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:28.381162+0000", 2021-02-10T04:45:40.535 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:45:40.535 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.535 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.535 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:45:40.535 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:28.381162+0000", 2021-02-10T04:45:40.535 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.3461000000000001e-05 2021-02-10T04:45:40.535 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.535 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.535 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:45:40.536 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:28.381186+0000", 2021-02-10T04:45:40.536 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.3299999999999999e-06 2021-02-10T04:45:40.536 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.536 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.536 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:45:40.536 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:28.381189+0000", 2021-02-10T04:45:40.536 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6590000000000004e-06 2021-02-10T04:45:40.536 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.536 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.536 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:45:40.536 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:28.381195+0000", 2021-02-10T04:45:40.536 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.6759999999999999e-06 2021-02-10T04:45:40.536 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.537 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.537 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:45:40.537 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:28.381198+0000", 2021-02-10T04:45:40.537 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00012388300000000001 2021-02-10T04:45:40.537 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.537 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.537 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.537 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:28.381321+0000", 2021-02-10T04:45:40.537 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00013975599999999999 2021-02-10T04:45:40.537 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.537 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.537 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2021-02-10T04:45:40.537 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:28.381461+0000", 2021-02-10T04:45:40.538 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.044161857999999998 2021-02-10T04:45:40.538 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.538 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.538 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.538 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:28.425623+0000", 2021-02-10T04:45:40.538 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000320969 2021-02-10T04:45:40.538 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.538 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.538 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:45:40.538 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:28.425944+0000", 2021-02-10T04:45:40.538 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00017978199999999999 2021-02-10T04:45:40.538 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.538 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.539 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 1,3", 2021-02-10T04:45:40.539 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:28.426124+0000", 2021-02-10T04:45:40.539 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.002440097 2021-02-10T04:45:40.539 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.539 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.539 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:45:40.539 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:28.428564+0000", 2021-02-10T04:45:40.539 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001935722 2021-02-10T04:45:40.539 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.539 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.539 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:45:40.539 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:28.430500+0000", 2021-02-10T04:45:40.539 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.021607040000000001 2021-02-10T04:45:40.540 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.540 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.540 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:45:40.540 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:28.452107+0000", 2021-02-10T04:45:40.540 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.3131e-05 2021-02-10T04:45:40.540 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.540 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.540 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:45:40.540 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:28.452200+0000", 2021-02-10T04:45:40.540 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000123869 2021-02-10T04:45:40.540 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.540 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.540 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:45:40.541 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:28.452324+0000", 2021-02-10T04:45:40.541 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.071161233000000004 2021-02-10T04:45:40.541 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.541 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:45:40.541 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.541 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.541 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.541 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:9995 7.0 7:06aed406:::smithi08575397-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e988)", 2021-02-10T04:45:40.541 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:43:28.381219+0000", 2021-02-10T04:45:40.541 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 132.058252622, 2021-02-10T04:45:40.541 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.071460140000000005, 2021-02-10T04:45:40.541 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:45:40.541 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:45:40.542 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:45:40.542 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:45:40.542 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:45:40.542 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 9995 2021-02-10T04:45:40.542 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.542 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:45:40.542 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.542 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:45:40.542 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:28.381219+0000", 2021-02-10T04:45:40.542 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:45:40.542 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.542 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.542 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:45:40.543 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:28.381219+0000", 2021-02-10T04:45:40.543 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6149000000000001e-05 2021-02-10T04:45:40.543 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.543 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.543 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:45:40.543 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:28.381235+0000", 2021-02-10T04:45:40.543 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.0120000000000001e-06 2021-02-10T04:45:40.543 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.543 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.543 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:45:40.543 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:28.381238+0000", 2021-02-10T04:45:40.543 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.8220000000000003e-06 2021-02-10T04:45:40.543 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.543 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.544 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:45:40.544 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:28.381244+0000", 2021-02-10T04:45:40.544 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2979999999999999e-06 2021-02-10T04:45:40.544 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.544 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.544 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:45:40.544 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:28.381246+0000", 2021-02-10T04:45:40.544 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000238497 2021-02-10T04:45:40.544 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.544 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.544 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.544 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:28.381485+0000", 2021-02-10T04:45:40.545 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.5738e-05 2021-02-10T04:45:40.545 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.545 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.545 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2021-02-10T04:45:40.545 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:28.381550+0000", 2021-02-10T04:45:40.545 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.045367668 2021-02-10T04:45:40.545 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.545 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.545 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.545 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:28.426918+0000", 2021-02-10T04:45:40.545 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000119191 2021-02-10T04:45:40.545 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.546 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.546 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2021-02-10T04:45:40.546 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:28.427037+0000", 2021-02-10T04:45:40.546 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.025330194 2021-02-10T04:45:40.546 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.546 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.546 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.546 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:28.452367+0000", 2021-02-10T04:45:40.546 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000130846 2021-02-10T04:45:40.546 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.546 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.546 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:45:40.546 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:28.452498+0000", 2021-02-10T04:45:40.547 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00018072499999999999 2021-02-10T04:45:40.547 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.547 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.547 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:45:40.547 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:28.452679+0000", 2021-02-10T04:45:40.547 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.071460140000000005 2021-02-10T04:45:40.547 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.547 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:45:40.547 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.547 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.547 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.547 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:10056 7.e 7:7016ed55:::smithi08575397-32:head [copy-from ver 2012] snapc 377=[377,374,373,372,370,36f,36b,368,366,35f,354,353,34e,34d,34b,349,348,345,344,343,33e,33c,33a,335,332,32f,32a,328,31a,308,300,2f5,2f0,2ee,2ed,2eb,2ea,2d9,2c8,2ae,27c,203] ondisk+write+known_if_redirected e993)", 2021-02-10T04:45:40.548 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:43:33.404508+0000", 2021-02-10T04:45:40.548 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 127.034963162, 2021-02-10T04:45:40.548 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.22154839400000001, 2021-02-10T04:45:40.548 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:45:40.548 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:45:40.548 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:45:40.548 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:45:40.548 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:45:40.548 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 10056 2021-02-10T04:45:40.548 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.548 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:45:40.548 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.548 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:45:40.549 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:33.404508+0000", 2021-02-10T04:45:40.549 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:45:40.549 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.549 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.549 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:45:40.549 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:33.404508+0000", 2021-02-10T04:45:40.549 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.4743999999999997e-05 2021-02-10T04:45:40.549 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.549 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.549 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:45:40.549 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:33.404553+0000", 2021-02-10T04:45:40.549 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.0879999999999997e-06 2021-02-10T04:45:40.549 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.550 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.550 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:45:40.550 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:33.404560+0000", 2021-02-10T04:45:40.550 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2818e-05 2021-02-10T04:45:40.550 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.550 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.550 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:45:40.550 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:33.404573+0000", 2021-02-10T04:45:40.550 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.1109999999999996e-06 2021-02-10T04:45:40.550 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.550 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.550 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:45:40.550 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:33.404579+0000", 2021-02-10T04:45:40.551 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.4860000000000001e-05 2021-02-10T04:45:40.551 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.551 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.551 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.551 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:33.404624+0000", 2021-02-10T04:45:40.551 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.9123999999999995e-05 2021-02-10T04:45:40.551 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.551 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.551 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:45:40.551 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:33.404723+0000", 2021-02-10T04:45:40.551 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.207151327 2021-02-10T04:45:40.551 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.551 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.552 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 0,3", 2021-02-10T04:45:40.552 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:33.611874+0000", 2021-02-10T04:45:40.552 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0052850029999999999 2021-02-10T04:45:40.552 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.552 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.552 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:45:40.552 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:33.617159+0000", 2021-02-10T04:45:40.552 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0081305500000000003 2021-02-10T04:45:40.552 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.552 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.552 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:45:40.552 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:33.625290+0000", 2021-02-10T04:45:40.552 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00055538199999999999 2021-02-10T04:45:40.553 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.553 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.553 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:45:40.553 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:33.625845+0000", 2021-02-10T04:45:40.553 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.2275999999999996e-05 2021-02-10T04:45:40.553 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.553 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.553 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:45:40.553 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:33.625927+0000", 2021-02-10T04:45:40.553 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000129111 2021-02-10T04:45:40.553 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.553 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.553 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:45:40.554 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:33.626057+0000", 2021-02-10T04:45:40.554 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.22154839400000001 2021-02-10T04:45:40.554 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.554 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:45:40.554 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.554 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.554 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.554 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:10057 7.e 7:7016ed55:::smithi08575397-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e993)", 2021-02-10T04:45:40.554 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:43:33.404622+0000", 2021-02-10T04:45:40.554 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 127.03484923000001, 2021-02-10T04:45:40.554 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.22175328599999999, 2021-02-10T04:45:40.554 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:45:40.554 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:45:40.555 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:45:40.555 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:45:40.555 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:45:40.555 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 10057 2021-02-10T04:45:40.555 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.555 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:45:40.555 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.555 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:45:40.555 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:33.404622+0000", 2021-02-10T04:45:40.555 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:45:40.555 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.555 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.555 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:45:40.556 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:33.404622+0000", 2021-02-10T04:45:40.556 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.6305e-05 2021-02-10T04:45:40.556 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.556 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.556 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:45:40.556 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:33.404648+0000", 2021-02-10T04:45:40.556 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.7560000000000001e-06 2021-02-10T04:45:40.556 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.556 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.556 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:45:40.556 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:33.404652+0000", 2021-02-10T04:45:40.556 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.9450000000000004e-06 2021-02-10T04:45:40.556 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.557 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.557 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:45:40.557 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:33.404659+0000", 2021-02-10T04:45:40.557 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.4139999999999998e-06 2021-02-10T04:45:40.557 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.557 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.557 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:45:40.557 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:33.404663+0000", 2021-02-10T04:45:40.557 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00069991900000000002 2021-02-10T04:45:40.557 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.557 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.557 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.557 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:33.405362+0000", 2021-02-10T04:45:40.558 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.7698000000000002e-05 2021-02-10T04:45:40.558 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.558 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.558 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:45:40.558 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:33.405420+0000", 2021-02-10T04:45:40.558 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.211011384 2021-02-10T04:45:40.558 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.558 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.558 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.558 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:33.616432+0000", 2021-02-10T04:45:40.558 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00012764800000000001 2021-02-10T04:45:40.558 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.558 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.559 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2021-02-10T04:45:40.559 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:33.616559+0000", 2021-02-10T04:45:40.559 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0095328819999999995 2021-02-10T04:45:40.559 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.559 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.559 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.559 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:33.626092+0000", 2021-02-10T04:45:40.559 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000113224 2021-02-10T04:45:40.559 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.559 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.559 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:45:40.559 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:33.626205+0000", 2021-02-10T04:45:40.559 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000170111 2021-02-10T04:45:40.560 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.560 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.560 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:45:40.560 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:33.626375+0000", 2021-02-10T04:45:40.560 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.22175328599999999 2021-02-10T04:45:40.560 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.560 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:45:40.560 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.560 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.560 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.560 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.24373.0:2776 5.15 5:a93a5511:::notify.2:head [watch ping cookie 94187755105536] snapc 0=[] ondisk+write+known_if_redirected e1033)", 2021-02-10T04:45:40.560 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:45:37.921913+0000", 2021-02-10T04:45:40.560 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 2.5175582479999998, 2021-02-10T04:45:40.561 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00095448699999999996, 2021-02-10T04:45:40.561 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:45:40.561 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:45:40.561 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:45:40.561 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.24373", 2021-02-10T04:45:40.561 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:45:40.561 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 2776 2021-02-10T04:45:40.561 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.561 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:45:40.561 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.561 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:45:40.561 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:37.921913+0000", 2021-02-10T04:45:40.561 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:45:40.562 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.562 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.562 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:45:40.562 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:37.921913+0000", 2021-02-10T04:45:40.562 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.2039999999999998e-05 2021-02-10T04:45:40.562 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.562 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.562 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:45:40.562 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:37.921945+0000", 2021-02-10T04:45:40.562 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0445000000000002e-05 2021-02-10T04:45:40.562 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.562 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.562 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:45:40.563 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:37.921966+0000", 2021-02-10T04:45:40.563 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4121000000000001e-05 2021-02-10T04:45:40.563 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.563 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.563 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:45:40.563 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:37.921980+0000", 2021-02-10T04:45:40.563 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.9049999999999992e-06 2021-02-10T04:45:40.563 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.563 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.563 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:45:40.563 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:37.921990+0000", 2021-02-10T04:45:40.563 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.2291000000000001e-05 2021-02-10T04:45:40.563 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.564 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.564 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.564 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:37.922082+0000", 2021-02-10T04:45:40.564 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00067829900000000005 2021-02-10T04:45:40.564 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.564 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.564 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:45:40.564 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:37.922760+0000", 2021-02-10T04:45:40.564 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000107386 2021-02-10T04:45:40.564 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.564 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.564 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:45:40.564 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:37.922868+0000", 2021-02-10T04:45:40.565 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00095448699999999996 2021-02-10T04:45:40.565 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.565 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:45:40.565 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.565 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.565 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.565 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.24373.0:2777 5.d 5:b4812045:::notify.4:head [watch ping cookie 94187770078336 gen 1] snapc 0=[] ondisk+write+known_if_redirected e1033)", 2021-02-10T04:45:40.565 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:45:37.922041+0000", 2021-02-10T04:45:40.565 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 2.517429935, 2021-02-10T04:45:40.565 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00087813799999999999, 2021-02-10T04:45:40.565 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:45:40.565 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:45:40.566 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:45:40.566 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.24373", 2021-02-10T04:45:40.566 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:45:40.566 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 2777 2021-02-10T04:45:40.566 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.566 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:45:40.566 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.566 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:45:40.566 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:37.922041+0000", 2021-02-10T04:45:40.566 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:45:40.566 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.566 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.567 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:45:40.567 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:37.922041+0000", 2021-02-10T04:45:40.567 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.6428999999999999e-05 2021-02-10T04:45:40.567 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.567 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.567 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:45:40.567 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:37.922088+0000", 2021-02-10T04:45:40.567 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.0099999999999998e-06 2021-02-10T04:45:40.567 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.567 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.567 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:45:40.567 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:37.922095+0000", 2021-02-10T04:45:40.567 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1029999999999999e-05 2021-02-10T04:45:40.568 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.568 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.568 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:45:40.568 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:37.922106+0000", 2021-02-10T04:45:40.568 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.5480000000000001e-06 2021-02-10T04:45:40.568 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.568 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.568 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:45:40.568 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:37.922111+0000", 2021-02-10T04:45:40.568 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00065205299999999999 2021-02-10T04:45:40.568 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.568 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.568 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:45:40.568 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:37.922764+0000", 2021-02-10T04:45:40.569 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.3233999999999997e-05 2021-02-10T04:45:40.569 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.569 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.569 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:45:40.569 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:37.922837+0000", 2021-02-10T04:45:40.569 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.2834e-05 2021-02-10T04:45:40.569 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:45:40.569 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:45:40.569 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:45:40.569 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:37.922920+0000", 2021-02-10T04:45:40.569 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00087813799999999999 2021-02-10T04:45:40.569 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.570 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:45:40.570 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.570 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:45:40.570 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:45:40.570 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:45:40.672 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:45:40.692 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:40 smithi164 conmon[33677]: debug 2021-02-10T04:45:40.396+0000 7fbec19a1700 -1 osd.4 1038 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:45:40.692 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:40 smithi164 conmon[29288]: cluster 2021-02-10T04:45:39.409483+0000 osd.4 (osd. 2021-02-10T04:45:40.692 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:40 smithi164 conmon[29288]: 4) 1824 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:40.692 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:40 smithi164 conmon[29288]: cluster 2021-02-10T04:45:39.409494+0000 osd.4 (osd.4) 1825 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:40.692 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:40 smithi164 conmon[29288]: cluster 2021-02-10T04:45: 2021-02-10T04:45:40.692 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:40 smithi164 conmon[29288]: 39.409501+0000 osd.4 (osd.4 2021-02-10T04:45:40.693 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:40 smithi164 conmon[29288]: ) 1826 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:40.693 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:40 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:45:40.693 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:40 smithi164 conmon[29288]: T04:45:39.409507+0000 osd.4 (osd.4) 1827 : cluster [WRN] 2021-02-10T04:45:40.693 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:40 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:40.693 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:40 smithi164 conmon[29288]: cluster 2021-02-10T04:45:40.693 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:40 smithi164 conmon[29288]: 2021-02 2021-02-10T04:45:40.693 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:40 smithi164 conmon[29288]: -10T04:45:40.394641+0000 mon.a (mon.0 2021-02-10T04:45:40.693 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:40 smithi164 conmon[29288]: ) 2075 : cluster [DBG] osdmap e1038: 8 total, 8 up, 7 in 2021-02-10T04:45:40.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:40 smithi085 conmon[29319]: cluster 2021-02-10T04:45:39.409483+0000 osd.4 (osd.4) 1824 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:40.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:40 smithi085 conmon[29319]: cluster 2021-02-10T04:45:40.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:40 smithi085 conmon[29319]: 2021 2021-02-10T04:45:40.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:40 smithi085 conmon[29319]: -02-10T04: 2021-02-10T04:45:40.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:40 smithi085 conmon[29319]: 45:39.409494+0000 osd.4 (osd.4) 1825 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:40.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:40 smithi085 conmon[29319]: 2021-02-10T04:45:40.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:40 smithi085 conmon[29319]: cluster 2021-02-10T04:45:40.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:40 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:45:40.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:40 smithi085 conmon[29319]: 04:45:39.409501+0000 osd.4 (osd.4) 1826 : cluster 2021-02-10T04:45:40.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:40 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:40.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:40 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:45:40.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:40 smithi085 conmon[29319]: 10T04:45:39.409507+0000 osd.4 ( 2021-02-10T04:45:40.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:40 smithi085 conmon[29319]: osd.4) 1827 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:40.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:40 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:45:40.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:40 smithi085 conmon[29319]: -10T04:45:40.394641+0000 mon.a (mon 2021-02-10T04:45:40.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:40 smithi085 conmon[29319]: .0) 2075 : cluster [DBG] osdmap e1038: 8 total, 8 up, 7 in 2021-02-10T04:45:40.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:40 smithi085 conmon[35961]: cluster 2021-02-10T04:45:39.409483 2021-02-10T04:45:40.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:40 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1824 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:40.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:40 smithi085 conmon[35961]: cluster 2021-02-10T04:45 2021-02-10T04:45:40.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:40 smithi085 conmon[35961]: :39.409494+0000 osd.4 (osd.4) 1825 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:40.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:40 smithi085 conmon[35961]: cluster 2021-02-10T04:45: 2021-02-10T04:45:40.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:40 smithi085 conmon[35961]: 39.409501+0000 osd.4 (osd.4) 1826 : cluster 2021-02-10T04:45:40.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:40 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:40.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:40 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:45:40.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:40 smithi085 conmon[35961]: -02-10T04:45:39.409507+0000 osd.4 (osd.4) 2021-02-10T04:45:40.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:40 smithi085 conmon[35961]: 1827 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:40.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:40 smithi085 conmon[35961]: cluster 2021-02-10T04:45: 2021-02-10T04:45:40.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:40 smithi085 conmon[35961]: 40.394641+0000 mon.a (mon.0) 2075 : cluster 2021-02-10T04:45:40.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:40 smithi085 conmon[35961]: [DBG] osdmap e1038: 8 total, 8 up, 7 in 2021-02-10T04:45:40.710 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:45:41.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[29319]: cluster 2021-02-10T04:45:40.303581+0000 mgr.y (mgr 2021-02-10T04:45:41.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[29319]: .14140) 988 : cluster [DBG] pgmap v1748: 178 pgs: 2 remapped+peering, 1 peering, 175 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:45:41.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[29319]: cluster 2021-02-10T04:45:40.397396+0000 osd.4 (osd.4) 1828 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:41.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[29319]: cluster 2021-02-10T04:45:40.397408+0000 osd.4 (osd.4) 1829 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:41.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[29319]: cluster 2021-02-10T04:45:40.397417+0000 osd.4 (osd.4) 1830 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:41.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[29319]: cluster 2021-02-10T04:45:40.397425+0000 osd.4 (osd.4) 1831 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:41.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:45:41.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[29319]: 10T04:45:41.412732+0000 mon.a (mon.0) 2076 : cluster [DBG] osdmap e1039: 8 total, 8 up, 7 in 2021-02-10T04:45:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: cluster 2021-02-10T04:45:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021-02-10T04:45:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021 2021-02-10T04:45:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: - 2021-02-10T04:45:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 02 2021-02-10T04:45:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: - 2021-02-10T04:45:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 10 2021-02-10T04:45:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: T 2021-02-10T04:45:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 04 2021-02-10T04:45:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: : 2021-02-10T04:45:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 45 2021-02-10T04:45:41.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: : 2021-02-10T04:45:41.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 40 2021-02-10T04:45:41.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: . 2021-02-10T04:45:41.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 303581 2021-02-10T04:45:41.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: +0000 2021-02-10T04:45:41.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021-02-10T04:45:41.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: mgr.y 2021-02-10T04:45:41.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: ( 2021-02-10T04:45:41.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: mgr 2021-02-10T04:45:41.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: . 2021-02-10T04:45:41.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 14140 2021-02-10T04:45:41.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: ) 2021-02-10T04:45:41.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 988 2021-02-10T04:45:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: : 2021-02-10T04:45:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: cluster 2021-02-10T04:45:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021-02-10T04:45:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: [DBG] 2021-02-10T04:45:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021-02-10T04:45:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: pgmap v1748: 178 pgs: 2 remapped+peering, 1 peering, 175 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:45:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021-02-10T04:45:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: cluster 2021-02-10T04:45:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021-02-10T04:45:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021 2021-02-10T04:45:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: - 2021-02-10T04:45:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 02 2021-02-10T04:45:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: - 2021-02-10T04:45:41.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 10 2021-02-10T04:45:41.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: T 2021-02-10T04:45:41.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 04 2021-02-10T04:45:41.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: : 2021-02-10T04:45:41.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 45 2021-02-10T04:45:41.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: : 2021-02-10T04:45:41.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 40 2021-02-10T04:45:41.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: . 2021-02-10T04:45:41.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 397396 2021-02-10T04:45:41.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: +0000 2021-02-10T04:45:41.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021-02-10T04:45:41.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: osd.4 2021-02-10T04:45:41.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: ( 2021-02-10T04:45:41.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: osd 2021-02-10T04:45:41.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: . 2021-02-10T04:45:41.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 4 2021-02-10T04:45:41.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: ) 2021-02-10T04:45:41.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 1828 2021-02-10T04:45:41.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: : 2021-02-10T04:45:41.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: cluster 2021-02-10T04:45:41.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021-02-10T04:45:41.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: [WRN] 2021-02-10T04:45:41.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021-02-10T04:45:41.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:41.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021-02-10T04:45:41.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: cluster 2021-02-10T04:45:41.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021-02-10T04:45:41.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021 2021-02-10T04:45:41.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: - 2021-02-10T04:45:41.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 02 2021-02-10T04:45:41.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: - 2021-02-10T04:45:41.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 10 2021-02-10T04:45:41.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: T 2021-02-10T04:45:41.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 04 2021-02-10T04:45:41.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: : 2021-02-10T04:45:41.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 45 2021-02-10T04:45:41.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: : 2021-02-10T04:45:41.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 40 2021-02-10T04:45:41.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: . 2021-02-10T04:45:41.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 397408 2021-02-10T04:45:41.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: +0000 2021-02-10T04:45:41.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021-02-10T04:45:41.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: osd.4 2021-02-10T04:45:41.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: ( 2021-02-10T04:45:41.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: osd 2021-02-10T04:45:41.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: . 2021-02-10T04:45:41.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 4 2021-02-10T04:45:41.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: ) 2021-02-10T04:45:41.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 1829 2021-02-10T04:45:41.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: : 2021-02-10T04:45:41.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: cluster 2021-02-10T04:45:41.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021-02-10T04:45:41.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: [WRN] 2021-02-10T04:45:41.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021-02-10T04:45:41.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:41.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021-02-10T04:45:41.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: cluster 2021-02-10T04:45:41.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021-02-10T04:45:41.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021 2021-02-10T04:45:41.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: - 2021-02-10T04:45:41.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 02 2021-02-10T04:45:41.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: - 2021-02-10T04:45:41.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 10 2021-02-10T04:45:41.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: T 2021-02-10T04:45:41.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 04 2021-02-10T04:45:41.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: : 2021-02-10T04:45:41.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 45 2021-02-10T04:45:41.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: : 2021-02-10T04:45:41.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 40 2021-02-10T04:45:41.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: . 2021-02-10T04:45:41.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 397417 2021-02-10T04:45:41.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: +0000 2021-02-10T04:45:41.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021-02-10T04:45:41.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: osd.4 2021-02-10T04:45:41.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: ( 2021-02-10T04:45:41.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: osd 2021-02-10T04:45:41.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: . 2021-02-10T04:45:41.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 4 2021-02-10T04:45:41.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: ) 2021-02-10T04:45:41.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 1830 2021-02-10T04:45:41.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: : 2021-02-10T04:45:41.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: cluster 2021-02-10T04:45:41.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021-02-10T04:45:41.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: [WRN] 2021-02-10T04:45:41.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021-02-10T04:45:41.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:41.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021-02-10T04:45:41.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: cluster 2021-02-10T04:45:41.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021-02-10T04:45:41.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021 2021-02-10T04:45:41.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: - 2021-02-10T04:45:41.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 02 2021-02-10T04:45:41.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: - 2021-02-10T04:45:41.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 10 2021-02-10T04:45:41.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: T 2021-02-10T04:45:41.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 04 2021-02-10T04:45:41.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: : 2021-02-10T04:45:41.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 45 2021-02-10T04:45:41.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: : 2021-02-10T04:45:41.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 40 2021-02-10T04:45:41.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: . 2021-02-10T04:45:41.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 397425 2021-02-10T04:45:41.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: +0000 2021-02-10T04:45:41.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021-02-10T04:45:41.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: osd.4 2021-02-10T04:45:41.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: ( 2021-02-10T04:45:41.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: osd 2021-02-10T04:45:41.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: . 2021-02-10T04:45:41.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 4 2021-02-10T04:45:41.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: ) 2021-02-10T04:45:41.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 1831 2021-02-10T04:45:41.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: : 2021-02-10T04:45:41.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: cluster 2021-02-10T04:45:41.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021-02-10T04:45:41.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: [WRN] 2021-02-10T04:45:41.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021-02-10T04:45:41.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:41.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021-02-10T04:45:41.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: cluster 2021-02-10T04:45:41.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021-02-10T04:45:41.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 2021 2021-02-10T04:45:41.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: - 2021-02-10T04:45:41.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 02 2021-02-10T04:45:41.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: - 2021-02-10T04:45:41.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: 10T04:45:41.412732+0000 2021-02-10T04:45:41.706 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:41 smithi085 conmon[35961]: mon.a (mon.0) 2076 : cluster [DBG] osdmap e1039: 8 total, 8 up, 7 in 2021-02-10T04:45:41.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:41 smithi164 conmon[33677]: debug 2021-02-10T04:45:41.424+0000 7fbec19a1700 -1 osd.4 1038 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:45:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:41 smithi164 conmon[29288]: cluster 2021-02-10T04:45:40.303581+0000 mgr.y ( 2021-02-10T04:45:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:41 smithi164 conmon[29288]: mgr.14140) 988 : cluster [DBG] pgmap v1748: 178 pgs: 2 remapped+peering, 1 peering, 175 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:45:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:41 smithi164 conmon[29288]: cluster 2021-02-10T04:45:40.397396+0000 osd.4 (osd.4) 1828 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:41.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:41 smithi164 conmon[29288]: cluster 2021-02-10T04:45:40.397408+0000 osd.4 2021-02-10T04:45:41.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:41 smithi164 conmon[29288]: (osd.4) 1829 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:41.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:41 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:45:41.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:41 smithi164 conmon[29288]: -02-10T04:45:40.397417+0000 2021-02-10T04:45:41.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:41 smithi164 conmon[29288]: osd.4 (osd.4) 1830 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:41.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:41 smithi164 conmon[29288]: 2021-02-10T04:45:41.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:41 smithi164 conmon[29288]: cluster 2021-02-10T04:45:41.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:41 smithi164 conmon[29288]: 2021-02-10T04:45:40.397425+0000 osd.4 (osd 2021-02-10T04:45:41.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:41 smithi164 conmon[29288]: .4) 1831 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:41.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:41 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:45:41.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:41 smithi164 conmon[29288]: -10T04:45:41.412732+0000 mon.a 2021-02-10T04:45:41.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:41 smithi164 conmon[29288]: (mon.0) 2076 : cluster [DBG] osdmap e1039: 8 total, 8 up, 7 in 2021-02-10T04:45:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[35961]: cluster 2021-02-10T04:45:41.424818 2021-02-10T04:45:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1832 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:45:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[35961]: 10T04:45:41.424827+0000 osd.4 (osd.4) 1833 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[35961]: 2021-02-10T04:45:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:45:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[35961]: -02-10T04:45:41.424831+0000 osd.4 ( 2021-02-10T04:45:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[35961]: osd.4) 1834 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:45:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[35961]: -10T04:45:41.424835+0000 osd.4 (osd.4) 1835 2021-02-10T04:45:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:42.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:45:42.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[35961]: T04:45:42.412713+0000 mon.a (mon.0 2021-02-10T04:45:42.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[35961]: ) 2077 : cluster [DBG] osdmap e1040: 8 total, 8 up, 7 in 2021-02-10T04:45:42.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[29319]: cluster 2021-02-10T04:45:42.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[29319]: 2021-02-10T04:45:41. 2021-02-10T04:45:42.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[29319]: 424818+0000 osd.4 (osd.4) 1832 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:42.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:45:42.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[29319]: -10T04:45:41.424827+0000 osd.4 (osd.4) 1833 : cluster [WRN] 2021-02-10T04:45:42.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:42.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[29319]: cluster 2021-02-10T04:45:42.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[29319]: 2021 2021-02-10T04:45:42.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[29319]: - 2021-02-10T04:45:42.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[29319]: 02-10T04:45 2021-02-10T04:45:42.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[29319]: :41.424831+0000 osd.4 (osd.4) 1834 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:42.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[29319]: cluster 2021-02-10T04:45:42.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[29319]: 2021-02-10T04:45:42.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:45:42.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[29319]: 04:45:41.424835+0000 osd.4 (osd.4) 1835 : cluster 2021-02-10T04:45:42.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:42.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[29319]: cluster 2021-02-10T04:45:42. 2021-02-10T04:45:42.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:42 smithi085 conmon[29319]: 412713+0000 mon.a (mon.0) 2077 : cluster [DBG] osdmap e1040: 8 total, 8 up, 7 in 2021-02-10T04:45:42.893 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:42 smithi164 conmon[33677]: debug 2021-02-10T04:45:42.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:42 smithi164 conmon[33677]: 2021-02-10T04:45:42.413+0000 7fbec19a1700 -1 osd.4 1040 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:45:42.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:42 smithi164 conmon[33677]: 2021-02-10T04:45:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:42 smithi164 conmon[29288]: cluster 2021-02-10T04:45:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:42 smithi164 conmon[29288]: 2021-02-10T04:45:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:42 smithi164 conmon[29288]: 2021 2021-02-10T04:45:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:42 smithi164 conmon[29288]: - 2021-02-10T04:45:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:42 smithi164 conmon[29288]: 02 2021-02-10T04:45:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:42 smithi164 conmon[29288]: -10T04:45: 2021-02-10T04:45:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:42 smithi164 conmon[29288]: 41.424818+0000 osd.4 (osd.4) 1832 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:42 smithi164 conmon[29288]: cluster 2021-02-10T04:45:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:42 smithi164 conmon[29288]: 2021-02 2021-02-10T04:45:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:42 smithi164 conmon[29288]: -10T04:45:41.424827+0000 osd.4 (osd.4) 2021-02-10T04:45:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:42 smithi164 conmon[29288]: 1833 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:42 smithi164 conmon[29288]: cluster 2021-02-10T04:45:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:42 smithi164 conmon[29288]: 2021-02-10T04:45:41.424831+0000 2021-02-10T04:45:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:42 smithi164 conmon[29288]: osd.4 (osd.4) 1834 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:42 smithi164 conmon[29288]: 2021-02-10T04:45:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:42 smithi164 conmon[29288]: cluster 2021-02-10T04:45:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:42 smithi164 conmon[29288]: 2021-02-10T04:45:41.424835+0000 osd.4 ( 2021-02-10T04:45:42.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:42 smithi164 conmon[29288]: osd.4) 1835 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:42.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:42 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:45:42.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:42 smithi164 conmon[29288]: -02-10T04:45:42.412713+0000 2021-02-10T04:45:42.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:42 smithi164 conmon[29288]: mon.a (mon.0) 2077 : cluster [DBG] 2021-02-10T04:45:42.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:42 smithi164 conmon[29288]: osdmap e1040: 8 total, 8 up, 7 in 2021-02-10T04:45:43.611 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:45:43.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:43 smithi085 conmon[29319]: cluster 2021-02-10T04:45:42.305730+0000 mgr.y (mgr.14140) 989 : cluster [DBG] pgmap v1751: 178 pgs: 2 remapped+peering, 1 peering, 175 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:45:43.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:43 smithi085 conmon[29319]: cluster 2021-02-10T04:45:42.414320+0000 osd.4 (osd.4) 1836 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:43.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:43 smithi085 conmon[29319]: cluster 2021-02-10T04:45:42.414329+0000 osd.4 (osd.4) 1837 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:43.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:43 smithi085 conmon[29319]: cluster 2021-02-10T04:45:42.414335+0000 osd.4 (osd.4) 1838 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:43.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:43 smithi085 conmon[29319]: cluster 2021-02-10T04:45:42.414343+0000 osd.4 (osd.4) 1839 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:43.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:43 smithi085 conmon[35961]: cluster 2021-02-10T04:45:42.305730+0000 mgr.y (mgr.14140) 989 : 2021-02-10T04:45:43.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:43 smithi085 conmon[35961]: cluster [DBG] pgmap v1751: 178 pgs: 2 remapped+peering, 1 peering, 175 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:45:43.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:43 smithi085 conmon[35961]: cluster 2021-02-10T04:45:42.414320+0000 osd.4 (osd.4) 1836 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:43.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:43 smithi085 conmon[35961]: cluster 2021-02-10T04:45:42. 2021-02-10T04:45:43.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:43 smithi085 conmon[35961]: 414329+0000 osd.4 (osd.4) 1837 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:43.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:43 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:45:43.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:43 smithi085 conmon[35961]: :45:42.414335+0000 osd.4 (osd.4) 1838 : cluster [WRN] 2021-02-10T04:45:43.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:43 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:43.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:43 smithi085 conmon[35961]: cluster 2021-02-10T04:45:43.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:43 smithi085 conmon[35961]: 2021-02-10T04:45:42.414343+0000 osd.4 (osd.4) 1839 : 2021-02-10T04:45:43.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:43 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:43.693 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:45:43 smithi085 conmon[41540]: debug 2021-02-10T04:45:43.535+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:45:43.717 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:45:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:43 smithi164 conmon[29288]: cluster 2021-02-10T04:45:42.305730+0000 mgr.y (mgr. 2021-02-10T04:45:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:43 smithi164 conmon[29288]: 14140) 989 : cluster [DBG] pgmap v1751: 178 pgs: 2 remapped+peering, 1 peering, 175 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:45:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:43 smithi164 conmon[29288]: cluster 2021-02-10T04:45:42.414320+0000 osd.4 (osd.4) 1836 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:43 smithi164 conmon[29288]: cluster 2021-02-10T04:45:42.414329+0000 osd.4 (osd.4 2021-02-10T04:45:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:43 smithi164 conmon[29288]: ) 1837 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:43 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:45:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:43 smithi164 conmon[29288]: T04:45:42.414335+0000 osd.4 (osd. 2021-02-10T04:45:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:43 smithi164 conmon[29288]: 4) 1838 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:43 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:45:43.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:43 smithi164 conmon[29288]: 04:45:42.414343+0000 osd.4 (osd.4) 2021-02-10T04:45:43.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:43 smithi164 conmon[29288]: 1839 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:43.895 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:43 smithi164 conmon[33677]: debug 2021-02-10T04:45:43.895 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:43 smithi164 conmon[33677]: 2021-02-10T04:45:43.452+0000 7fbec19a1700 -1 osd.4 1040 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:45:43.895 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:43 smithi164 conmon[33677]: 2021-02-10T04:45:44.586 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:45:44.851 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:44 smithi164 conmon[29288]: cluster 2021-02-10T04:45:43.452971+0000 osd.4 (osd.4 2021-02-10T04:45:44.851 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:44 smithi164 conmon[29288]: ) 1840 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:44.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:44 smithi164 conmon[29288]: cluster 2021-02-10T04:45:43.452981+0000 osd.4 (osd.4) 1841 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:44.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:44 smithi164 conmon[29288]: cluster 2021-02-10T04:45:43.452987+0000 2021-02-10T04:45:44.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:44 smithi164 conmon[29288]: osd.4 (osd.4) 1842 : cluster [WRN] 2021-02-10T04:45:44.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:44 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:44.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:44 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:45:44.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:44 smithi164 conmon[29288]: -02-10T04:45:43.452993+0000 osd.4 (osd 2021-02-10T04:45:44.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:44 smithi164 conmon[29288]: .4) 1843 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:44.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:44 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:45:44.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:44 smithi164 conmon[29288]: 10T04:45:43.489909+0000 mon.a (mon. 2021-02-10T04:45:44.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:44 smithi164 conmon[29288]: 0) 2078 : cluster [DBG] osdmap e1041: 8 total, 8 up, 7 in 2021-02-10T04:45:44.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:44 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:45:44.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:44 smithi164 conmon[29288]: 10T04:45:43.877632+0000 osd.7 (osd.7) 2021-02-10T04:45:44.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:44 smithi164 conmon[29288]: 72 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:45:44.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:44 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:45:44.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:44 smithi164 conmon[29288]: 45:43.880385+0000 osd.7 (osd.7) 73 : 2021-02-10T04:45:44.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:44 smithi164 conmon[29288]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:45:44.853 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:44 smithi164 conmon[33677]: debug 2021-02-10T04:45:44.465+0000 7fbec19a1700 -1 osd.4 1041 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:45:44.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:44 smithi085 conmon[29319]: cluster 2021-02-10T04:45:43.452971+0000 osd.4 ( 2021-02-10T04:45:44.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:44 smithi085 conmon[29319]: osd.4) 1840 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:44.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:44 smithi085 conmon[29319]: cluster 2021-02-10T04:45:43.452981+0000 osd.4 (osd.4) 1841 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:44.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:44 smithi085 conmon[29319]: cluster 2021-02-10T04:45:43.452987+0000 osd.4 (osd.4) 1842 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:44.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:44 smithi085 conmon[29319]: cluster 2021-02-10T04:45:43.452993+0000 osd.4 (osd.4) 1843 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:44.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:44 smithi085 conmon[29319]: cluster 2021-02-10T04:45:43.489909+0000 mon.a (mon.0) 2078 : cluster [DBG] osdmap e1041: 8 total, 8 up, 7 in 2021-02-10T04:45:44.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:44 smithi085 conmon[29319]: cluster 2021-02-10T04:45:43.877632+0000 osd.7 (osd.7) 72 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:45:44.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:44 smithi085 conmon[29319]: cluster 2021-02-10T04:45:43.880385+0000 osd.7 (osd.7) 73 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:45:44.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:44 smithi085 conmon[35961]: cluster 2021-02-10T04:45:44.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:44 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:45:44.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:44 smithi085 conmon[35961]: 45:43.452971+0000 osd.4 (osd.4) 1840 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:44.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:44 smithi085 conmon[35961]: cluster 2021-02-10T04:45:44.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:44 smithi085 conmon[35961]: 2021-02-10T04:45:43. 2021-02-10T04:45:44.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:44 smithi085 conmon[35961]: 452981+0000 osd.4 (osd.4) 1841 : cluster 2021-02-10T04:45:44.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:44 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:44.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:44 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:45:44.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:44 smithi085 conmon[35961]: -02-10T04:45:43.452987+0000 osd.4 2021-02-10T04:45:44.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:44 smithi085 conmon[35961]: (osd.4) 1842 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:44.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:44 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:45:44.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:44 smithi085 conmon[35961]: 10T04:45:43.452993+0000 osd.4 (osd.4) 2021-02-10T04:45:44.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:44 smithi085 conmon[35961]: 1843 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:44.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:44 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:45:44.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:44 smithi085 conmon[35961]: 02-10T04:45:43.489909+0000 mon.a 2021-02-10T04:45:44.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:44 smithi085 conmon[35961]: (mon.0) 2078 : cluster [DBG] osdmap e1041: 8 total, 8 up, 7 in 2021-02-10T04:45:44.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:44 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:45:44.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:44 smithi085 conmon[35961]: -10T04:45:43.877632+0000 osd.7 ( 2021-02-10T04:45:44.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:44 smithi085 conmon[35961]: osd.7) 72 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:45:44.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:44 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:45:44.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:44 smithi085 conmon[35961]: -02-10T04:45:43.880385+0000 osd.7 2021-02-10T04:45:44.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:44 smithi085 conmon[35961]: (osd.7) 73 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:45:45.142 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:45:45.144 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:45:44 smithi164 conmon[40892]: debug 2021-02-10T04:45:44.849+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:45:45.519 INFO:teuthology.orchestra.run.smithi085.stderr:set pool 6 pg_num to 42 2021-02-10T04:45:45.549 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset noscrub 2021-02-10T04:45:45.588 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:45:45.896 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:45 smithi164 conmon[33677]: debug 2021-02-10T04:45:45.433+0000 7fbec19a1700 -1 osd.4 1042 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:45:45.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:45 smithi164 conmon[29288]: cluster 2021-02-10T04:45:45.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:45 smithi164 conmon[29288]: 2021-02-10T04:45:45.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:45 smithi164 conmon[29288]: 2021-02-10T04:45 2021-02-10T04:45:45.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:45 smithi164 conmon[29288]: :44.306272+0000 mgr.y (mgr.14140) 990 : cluster [DBG] pgmap v1754: 178 pgs: 1 active+recovering+undersized+remapped, 2 remapped+peering, 175 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/2181 objects misplaced (0.046%); 8.8 MiB/s, 19 objects/s recovering 2021-02-10T04:45:45.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:45 smithi164 conmon[29288]: cluster 2021-02-10T04:45:45.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:45 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:45:45.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:45 smithi164 conmon[29288]: 45:44.466288+0000 osd.4 (osd.4) 2021-02-10T04:45:45.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:45 smithi164 conmon[29288]: 1844 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:45.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:45 smithi164 conmon[29288]: cluster 2021-02-10T04:45:45.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:45 smithi164 conmon[29288]: 2021-02-10T04:45:44.466296 2021-02-10T04:45:45.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:45 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1845 : cluster [WRN] 2021-02-10T04:45:45.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:45 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:45.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:45 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:45:45.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:45 smithi164 conmon[29288]: T04:45:44.466302+0000 osd.4 (osd.4 2021-02-10T04:45:45.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:45 smithi164 conmon[29288]: ) 1846 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:45.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:45 smithi164 conmon[29288]: cluster 2021-02-10T04:45:45.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:45 smithi164 conmon[29288]: 2021-02-10T04:45: 2021-02-10T04:45:45.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:45 smithi164 conmon[29288]: 44.466308+0000 osd.4 (osd.4) 1847 : cluster 2021-02-10T04:45:45.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:45 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:45.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:45 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:45:45.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:45 smithi164 conmon[29288]: 02-10T04:45:44.502680+0000 mon.a ( 2021-02-10T04:45:45.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:45 smithi164 conmon[29288]: mon.0) 2079 : cluster [DBG] osdmap e1042: 8 total, 8 up, 7 in 2021-02-10T04:45:45.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:45 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:45:45.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:45 smithi164 conmon[29288]: 02-10T04:45:44.770781+0000 mon.a (mon 2021-02-10T04:45:45.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:45 smithi164 conmon[29288]: .0) 2080 : audit [INF] from='client.? 172.21.15.85:0/213131588' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pg_num", "val": "42"}]: dispatch 2021-02-10T04:45:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[29319]: cluster 2021-02-10T04:45:44.306272+0000 mgr.y (mgr.14140 2021-02-10T04:45:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[29319]: ) 990 : cluster [DBG] pgmap v1754: 178 pgs: 1 active+recovering+undersized+remapped, 2 remapped+peering, 175 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/2181 objects misplaced (0.046%); 8.8 MiB/s, 19 objects/s recovering 2021-02-10T04:45:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[29319]: cluster 2021-02-10T04:45:44.466288+0000 osd.4 (osd.4) 1844 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[29319]: cluster 2021-02-10T04:45:44.466296+0000 osd.4 (osd.4) 1845 : cluster 2021-02-10T04:45:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:45:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[29319]: :45:44.466302+0000 osd.4 (osd.4 2021-02-10T04:45:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[29319]: ) 1846 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:45:45.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[29319]: -02-10T04:45:44.466308+0000 2021-02-10T04:45:45.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[29319]: osd.4 (osd.4) 1847 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:45.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[29319]: 2021-02-10T04:45:45.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:45:45.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[29319]: 45:44.502680+0000 mon.a (mon.0) 2021-02-10T04:45:45.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[29319]: 2079 : cluster [DBG] osdmap e1042: 8 total, 8 up, 7 in 2021-02-10T04:45:45.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[29319]: audit 2021 2021-02-10T04:45:45.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[29319]: -02-10T04:45:44.770781+0000 2021-02-10T04:45:45.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[29319]: mon.a (mon.0) 2080 : audit [INF] 2021-02-10T04:45:45.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[29319]: from='client.? 172.21.15.85:0/213131588' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pg_num", "val": "42"}]: dispatch 2021-02-10T04:45:45.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[35961]: cluster 2021-02-10T04:45:44.306272+0000 mgr.y 2021-02-10T04:45:45.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[35961]: (mgr.14140) 990 : cluster [DBG] pgmap v1754: 178 pgs: 1 active+recovering+undersized+remapped, 2 remapped+peering, 175 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/2181 objects misplaced (0.046%); 8.8 MiB/s, 19 objects/s recovering 2021-02-10T04:45:45.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[35961]: cluster 2021-02-10T04:45:44.466288+0000 osd.4 (osd.4) 1844 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:45.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[35961]: 2021-02-10T04:45:45.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[35961]: cluster 2021-02-10T04:45:44 2021-02-10T04:45:45.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[35961]: .466296+0000 osd.4 (osd.4) 1845 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:45.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[35961]: cluster 2021-02-10T04:45:45.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[35961]: 2021-02-10T04:45:44.466302+0000 osd.4 (osd.4) 1846 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:45.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[35961]: cluster 2021-02-10T04:45:44.466308+0000 2021-02-10T04:45:45.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[35961]: osd.4 (osd.4) 1847 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:45.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[35961]: cluster 2021-02-10T04:45:44.502680+0000 2021-02-10T04:45:45.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[35961]: mon.a (mon.0) 2079 : cluster [DBG] osdmap e1042: 8 total, 8 up, 7 in 2021-02-10T04:45:45.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[35961]: audit 2021-02-10T04:45:44. 2021-02-10T04:45:45.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:45 smithi085 conmon[35961]: 770781+0000 mon.a (mon.0) 2080 : audit [INF] from='client.? 172.21.15.85:0/213131588' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pg_num", "val": "42"}]: dispatch 2021-02-10T04:45:46.205 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:45:47.852 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:46 smithi164 conmon[33677]: debug 2021-02-10T04:45:47.852 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:46 smithi164 conmon[33677]: 2021-02-10T04:45:46.440+0000 7fbec19a1700 -1 osd.4 1043 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:45:47.852 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:46 smithi164 conmon[33677]: 2021-02-10T04:45:47.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:46 smithi164 conmon[29288]: cluster 2021-02-10T04:45:45.434080+0000 osd.4 ( 2021-02-10T04:45:47.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:46 smithi164 conmon[29288]: osd.4) 1848 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:47.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:46 smithi164 conmon[29288]: cluster 2021-02-10T04:45:45.434094+0000 osd.4 (osd.4) 1849 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:47.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:46 smithi164 conmon[29288]: cluster 2021-02-10T04:45:45.434102+0000 osd.4 ( 2021-02-10T04:45:47.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:46 smithi164 conmon[29288]: osd.4) 1850 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:47.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:46 smithi164 conmon[29288]: cluster 2021-02-10T04:45:47.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:46 smithi164 conmon[29288]: 2021-02-10T04:45:45.434109+0000 2021-02-10T04:45:47.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:46 smithi164 conmon[29288]: osd.4 (osd.4) 1851 : cluster [WRN] 2021-02-10T04:45:47.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:46 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:47.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:46 smithi164 conmon[29288]: audit 2021-02-10T04:45: 2021-02-10T04:45:47.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:46 smithi164 conmon[29288]: 45.511724+0000 mon.a (mon.0) 2081 : audit 2021-02-10T04:45:47.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:46 smithi164 conmon[29288]: [INF] from='client.? 172.21.15.85:0/213131588' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pg_num", "val": "42"}]': finished 2021-02-10T04:45:47.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:46 smithi164 conmon[29288]: cluster 2021-02-10T04:45:45.511790+0000 mon.a (mon.0) 2082 : 2021-02-10T04:45:47.853 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:46 smithi164 conmon[29288]: cluster [DBG] osdmap e1043: 8 total, 8 up, 7 in 2021-02-10T04:45:47.854 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:46 smithi164 conmon[29288]: audit 2021-02-10T04:45:47.854 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:46 smithi164 conmon[29288]: 2021-02-10T04:45:46.307895+0000 mon.a ( 2021-02-10T04:45:47.854 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:46 smithi164 conmon[29288]: mon.0) 2083 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pg_num_actual", "val": "42"}]: dispatch 2021-02-10T04:45:47.854 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:45:47.854 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:46 smithi085 conmon[29319]: cluster 2021-02-10T04:45:45. 2021-02-10T04:45:47.854 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:46 smithi085 conmon[29319]: 434080+0000 osd.4 (osd.4) 1848 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:47.854 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:46 smithi085 conmon[29319]: cluster 2021-02-10T04:45:45.434094+0000 osd.4 (osd.4) 1849 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:47.854 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:46 smithi085 conmon[29319]: cluster 2021-02-10T04:45:45.434102+0000 osd.4 (osd.4) 1850 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:47.854 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:46 smithi085 conmon[29319]: cluster 2021-02-10T04:45:45.434109+0000 osd.4 (osd.4) 1851 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:47.855 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:46 smithi085 conmon[29319]: audit 2021-02-10T04:45:45.511724+0000 mon.a (mon.0) 2081 : audit [INF] from='client.? 172.21.15.85:0/213131588' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pg_num", "val": "42"}]': finished 2021-02-10T04:45:47.855 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:46 smithi085 conmon[29319]: cluster 2021-02-10T04:45:45.511790+0000 mon.a (mon.0) 2082 : cluster [DBG] osdmap e1043: 8 total, 8 up, 7 in 2021-02-10T04:45:47.855 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:46 smithi085 conmon[29319]: audit 2021-02-10T04:45:46.307895+0000 mon.a (mon.0) 2083 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pg_num_actual", "val": "42"}]: dispatch 2021-02-10T04:45:47.855 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:46 smithi085 conmon[35961]: cluster 2021-02-10T04:45:45.434080+0000 2021-02-10T04:45:47.855 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:46 smithi085 conmon[35961]: osd.4 (osd.4) 1848 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:47.855 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:46 smithi085 conmon[35961]: cluster 2021-02-10T04:45:45.434094+0000 osd.4 (osd.4) 1849 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:47.855 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:46 smithi085 conmon[35961]: cluster 2021-02-10T04:45:45.434102+0000 osd.4 (osd.4) 1850 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:47.855 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:46 smithi085 conmon[35961]: cluster 2021-02-10T04:45:45.434109+0000 osd.4 (osd.4) 1851 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:47.855 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:46 smithi085 conmon[35961]: audit 2021-02-10T04:45:45.511724+0000 mon.a (mon.0) 2081 : audit [INF] from='client.? 172.21.15.85:0/213131588' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pg_num", "val": "42"}]': finished 2021-02-10T04:45:47.855 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:46 smithi085 conmon[35961]: cluster 2021-02-10T04:45:45.511790+0000 mon.a (mon.0) 2082 : cluster [DBG] osdmap e1043: 8 total, 8 up, 7 in 2021-02-10T04:45:47.855 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:46 smithi085 conmon[35961]: audit 2021-02-10T04:45:47.855 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:46 smithi085 conmon[35961]: 2021-02-10T04:45:46.307895+0000 mon.a (mon.0) 2021-02-10T04:45:47.856 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:46 smithi085 conmon[35961]: 2083 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pg_num_actual", "val": "42"}]: dispatch 2021-02-10T04:45:47.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:47 smithi164 conmon[33677]: debug 2021-02-10T04:45:47.485+0000 7fbec19a1700 -1 osd.4 1044 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:45:47.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:47 smithi164 conmon[29288]: cluster 2021-02-10T04:45:46.306745+0000 mgr.y ( 2021-02-10T04:45:47.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:47 smithi164 conmon[29288]: mgr.14140) 991 : cluster [DBG] pgmap v1757: 178 pgs: 1 active+recovering+undersized+remapped, 177 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/2181 objects misplaced (0.046%); 13 MiB/s, 22 objects/s recovering 2021-02-10T04:45:47.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:47 smithi164 conmon[29288]: cluster 2021-02-10T04:45:46.441441+0000 osd.4 (osd.4) 1852 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:47.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:47 smithi164 conmon[29288]: cluster 2021-02-10T04:45:46.441450+0000 osd.4 (osd.4) 1853 : cluster [WRN] 2021-02-10T04:45:47.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:47 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:47.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:47 smithi164 conmon[29288]: cluster 2021-02-10T04:45:46.441456 2021-02-10T04:45:47.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:47 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1854 : cluster [WRN] 2021-02-10T04:45:47.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:47 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:47.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:47 smithi164 conmon[29288]: cluster 2021-02-10T04:45:46.441462 2021-02-10T04:45:47.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:47 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1855 : cluster [WRN] 2021-02-10T04:45:47.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:47 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:47.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:47 smithi164 conmon[29288]: audit 2021-02-10T04:45:46 2021-02-10T04:45:47.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:47 smithi164 conmon[29288]: .518256+0000 mon.a (mon.0) 2084 : audit 2021-02-10T04:45:47.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:47 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pg_num_actual", "val": "42"}]': finished 2021-02-10T04:45:47.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:47 smithi164 conmon[29288]: cluster 2021-02-10T04:45: 2021-02-10T04:45:47.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:47 smithi164 conmon[29288]: 46.518320+0000 mon.a (mon.0) 2085 2021-02-10T04:45:47.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:47 smithi164 conmon[29288]: : cluster [DBG] osdmap e1044: 8 total, 8 up, 7 in 2021-02-10T04:45:47.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[29319]: cluster 2021-02-10T04:45:46.306745+0000 mgr.y (mgr 2021-02-10T04:45:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[29319]: .14140) 991 : cluster [DBG] pgmap v1757: 178 pgs: 1 active+recovering+undersized+remapped, 177 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/2181 objects misplaced (0.046%); 13 MiB/s, 22 objects/s recovering 2021-02-10T04:45:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[29319]: cluster 2021-02-10T04:45:46.441441+0000 osd.4 (osd.4) 1852 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[29319]: cluster 2021-02-10T04:45:46.441450+0000 osd.4 (osd. 2021-02-10T04:45:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[29319]: 4) 1853 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:45:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[29319]: -10T04:45:46.441456+0000 osd.4 ( 2021-02-10T04:45:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[29319]: osd.4) 1854 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[29319]: cluster 2021-02-10T04:45:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[29319]: 2021-02-10T04:45:46.441462 2021-02-10T04:45:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1855 : cluster 2021-02-10T04:45:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:45:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[29319]: -10T04:45:46.518256+0000 mon.a 2021-02-10T04:45:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[29319]: (mon.0) 2084 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pg_num_actual", "val": "42"}]': finished 2021-02-10T04:45:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[29319]: 2021-02-10T04:45:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[29319]: cluster 2021-02-10T04:45 2021-02-10T04:45:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[29319]: :46.518320+0000 mon.a (mon.0) 2085 2021-02-10T04:45:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[29319]: : cluster [DBG] osdmap e1044: 8 total, 8 up, 7 in 2021-02-10T04:45:47.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[35961]: cluster 2021-02-10T04:45:46.306745+0000 mgr.y (mgr.14140) 991 : cluster 2021-02-10T04:45:47.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[35961]: [DBG] pgmap v1757: 178 pgs: 1 active+recovering+undersized+remapped, 177 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/2181 objects misplaced (0.046%); 13 MiB/s, 22 objects/s recovering 2021-02-10T04:45:47.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[35961]: cluster 2021-02-10T04:45:47.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:45:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[35961]: T04:45:46.441441+0000 osd.4 (osd.4) 1852 : cluster [WRN] 2021-02-10T04:45:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[35961]: cluster 2021-02-10T04:45:46 2021-02-10T04:45:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[35961]: .441450+0000 osd.4 (osd.4) 1853 : cluster [WRN] 2021-02-10T04:45:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[35961]: cluster 2021-02-10T04:45 2021-02-10T04:45:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[35961]: :46.441456+0000 osd.4 (osd.4) 1854 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:45:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[35961]: T04:45:46.441462+0000 osd.4 (osd.4) 1855 : cluster 2021-02-10T04:45:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:45:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[35961]: 45:46.518256+0000 mon.a (mon.0) 2084 2021-02-10T04:45:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[35961]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pg_num_actual", "val": "42"}]': finished 2021-02-10T04:45:47.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[35961]: cluster 2021-02-10T04:45:46. 2021-02-10T04:45:47.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[35961]: 518320+0000 mon.a (mon.0) 2085 : cluster [DBG] 2021-02-10T04:45:47.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:47 smithi085 conmon[35961]: osdmap e1044: 8 total, 8 up, 7 in 2021-02-10T04:45:48.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:48 smithi164 conmon[33677]: debug 2021-02-10T04:45:48.530+0000 7fbec19a1700 -1 osd.4 1045 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:45:48.894 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:45:48 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:45:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:48 smithi164 conmon[29288]: cluster 2021-02-10T04:45:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:48 smithi164 conmon[29288]: 2021-02-10T04:45:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:48 smithi164 conmon[29288]: 2021-02-10T04:45:47.486654+0000 osd.4 (osd.4 2021-02-10T04:45:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:48 smithi164 conmon[29288]: ) 1856 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:48 smithi164 conmon[29288]: cluster 2021-02-10T04:45:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:48 smithi164 conmon[29288]: 2021-02-10T04:45:47.486677+0000 osd.4 (osd.4) 1857 2021-02-10T04:45:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:48 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:48 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:45:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:48 smithi164 conmon[29288]: :45:47.486687+0000 osd.4 (osd.4) 2021-02-10T04:45:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:48 smithi164 conmon[29288]: 1858 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:48 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:45:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:48 smithi164 conmon[29288]: 04:45:47.486692+0000 osd.4 (osd. 2021-02-10T04:45:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:48 smithi164 conmon[29288]: 4) 1859 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:48 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:45:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:48 smithi164 conmon[29288]: -10T04:45:47.520675+0000 mon.a 2021-02-10T04:45:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:48 smithi164 conmon[29288]: (mon.0) 2086 : cluster [DBG] osdmap e1045: 8 total, 8 up, 7 in 2021-02-10T04:45:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:48 smithi164 conmon[29288]: audit 2021 2021-02-10T04:45:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:48 smithi164 conmon[29288]: -02-10T04:45:48.308577 2021-02-10T04:45:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:48 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2087 : audit [INF] 2021-02-10T04:45:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:48 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pgp_num_actual", "val": "42"}]: dispatch 2021-02-10T04:45:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:48 smithi164 conmon[29288]: cluster 2021-02-10T04:45:48 2021-02-10T04:45:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:48 smithi164 conmon[29288]: .374701+0000 mon.a (mon.0) 2088 : cluster 2021-02-10T04:45:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:48 smithi164 conmon[29288]: [WRN] Health check update: 4 slow ops, oldest one blocked for 76 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:45:48.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[29319]: cluster 2021-02-10T04:45:47.486654+0000 osd.4 2021-02-10T04:45:48.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[29319]: (osd.4) 1856 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:48.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[29319]: cluster 2021-02-10T04:45:47.486677+0000 osd.4 (osd.4) 1857 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:48.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[29319]: 2021-02-10T04:45:48.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[29319]: cluster 2021-02-10T04:45:47. 2021-02-10T04:45:48.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[29319]: 486687+0000 osd.4 (osd.4) 1858 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:48.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[29319]: 2021-02-10T04:45:48.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[29319]: cluster 2021-02-10T04:45:48.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[29319]: 2021-02-10T04:45:47.486692+0000 2021-02-10T04:45:48.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[29319]: osd.4 (osd.4) 1859 : cluster [WRN] 2021-02-10T04:45:48.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:48.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[29319]: cluster 2021-02-10T04:45 2021-02-10T04:45:48.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[29319]: :47.520675+0000 mon.a (mon.0) 2086 2021-02-10T04:45:48.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[29319]: : cluster [DBG] osdmap e1045: 8 total, 8 up, 7 in 2021-02-10T04:45:48.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[29319]: audit 2021 2021-02-10T04:45:48.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[29319]: -02-10T04:45:48.308577 2021-02-10T04:45:48.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2087 : audit 2021-02-10T04:45:48.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[29319]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pgp_num_actual", "val": "42"}]: dispatch 2021-02-10T04:45:48.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[29319]: cluster 2021-02-10T04:45: 2021-02-10T04:45:48.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[29319]: 48.374701+0000 mon.a (mon.0) 2088 2021-02-10T04:45:48.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[29319]: : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 76 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:45:48.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[35961]: cluster 2021-02-10T04:45:47.486654+0000 osd.4 2021-02-10T04:45:48.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[35961]: (osd.4) 1856 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:48.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[35961]: cluster 2021-02-10T04:45:47.486677+0000 2021-02-10T04:45:48.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[35961]: osd.4 (osd.4) 1857 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:48.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[35961]: cluster 2021-02-10T04:45:48.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[35961]: 2021-02-10T04:45:47.486687+0000 osd.4 (osd.4 2021-02-10T04:45:48.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[35961]: ) 1858 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:48.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[35961]: cluster 2021-02-10T04:45: 2021-02-10T04:45:48.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[35961]: 47.486692+0000 osd.4 (osd.4) 1859 : cluster 2021-02-10T04:45:48.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:48.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[35961]: cluster 2021-02-10T04:45:48.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[35961]: 2021-02-10T04:45:47.520675+0000 mon.a (mon. 2021-02-10T04:45:48.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[35961]: 0) 2086 : cluster [DBG] osdmap e1045: 8 total, 8 up, 7 in 2021-02-10T04:45:48.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:45:48.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[35961]: 04:45:48.308577+0000 mon.a (mon.0 2021-02-10T04:45:48.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[35961]: ) 2087 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pgp_num_actual", "val": "42"}]: dispatch 2021-02-10T04:45:48.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[35961]: cluster 2021-02-10T04:45: 2021-02-10T04:45:48.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[35961]: 48.374701+0000 mon.a (mon.0) 2088 : cluster 2021-02-10T04:45:48.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:48 smithi085 conmon[35961]: [WRN] Health check update: 4 slow ops, oldest one blocked for 76 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:45:49.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:49 smithi164 conmon[33677]: debug 2021-02-10T04:45:49.566+0000 7fbec19a1700 -1 osd.4 1047 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:45:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:49 smithi164 conmon[29288]: cluster 2021-02-10T04:45:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:49 smithi164 conmon[29288]: 2021-02-10T04:45:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:49 smithi164 conmon[29288]: 2021 2021-02-10T04:45:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:49 smithi164 conmon[29288]: -02-10T04:45 2021-02-10T04:45:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:49 smithi164 conmon[29288]: :48.307378+0000 mgr.y (mgr.14140) 992 : cluster [DBG] pgmap v1760: 188 pgs: 1 peering, 8 unknown, 1 active+recovering+undersized+remapped, 178 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1/2181 objects misplaced (0.046%); 4.0 MiB/s, 2 objects/s recovering 2021-02-10T04:45:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:49 smithi164 conmon[29288]: cluster 2021-02-10T04:45:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:49 smithi164 conmon[29288]: 2021-02-10T04:45:48.531437+0000 2021-02-10T04:45:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:49 smithi164 conmon[29288]: osd.4 (osd.4) 1860 : cluster [WRN] 2021-02-10T04:45:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:49 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:49 smithi164 conmon[29288]: cluster 2021-02-10T04:45: 2021-02-10T04:45:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:49 smithi164 conmon[29288]: 48.531443+0000 osd.4 (osd.4) 1861 : 2021-02-10T04:45:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:49 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:49 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:45:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:49 smithi164 conmon[29288]: 45:48.531447+0000 osd.4 (osd.4) 1862 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:49 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:45:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:49 smithi164 conmon[29288]: :45:48.531452+0000 osd.4 (osd.4 2021-02-10T04:45:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:49 smithi164 conmon[29288]: ) 1863 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:49 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:45:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:49 smithi164 conmon[29288]: 10T04:45:48.535017+0000 mon.a ( 2021-02-10T04:45:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:49 smithi164 conmon[29288]: mon.0) 2089 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pgp_num_actual", "val": "42"}]': finished 2021-02-10T04:45:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:49 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:45:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:49 smithi164 conmon[29288]: -02-10T04:45:48.535101+0000 2021-02-10T04:45:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:49 smithi164 conmon[29288]: mon.a (mon.0) 2090 : cluster [DBG] osdmap e1046: 8 total, 8 up, 7 in 2021-02-10T04:45:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:49 smithi164 conmon[29288]: 2021-02-10T04:45:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:49 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:45:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:49 smithi164 conmon[29288]: T04:45:48.723073+0000 mgr.y (mgr.14140) 993 : audit 2021-02-10T04:45:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:49 smithi164 conmon[29288]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:45:49.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[29319]: cluster 2021-02-10T04:45:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[29319]: 2021-02-10T04:45 2021-02-10T04:45:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[29319]: :48.307378+0000 mgr.y (mgr.14140) 992 : cluster [DBG] pgmap v1760: 188 pgs: 1 peering, 8 unknown, 1 active+recovering+undersized+remapped, 178 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1/2181 objects misplaced (0.046%); 4.0 MiB/s, 2 objects/s recovering 2021-02-10T04:45:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:45:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[29319]: 10T04:45:48.531437+0000 osd.4 (osd. 2021-02-10T04:45:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[29319]: 4) 1860 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:45:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[29319]: 45:48.531443+0000 osd.4 (osd.4) 1861 2021-02-10T04:45:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:45:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[29319]: 04:45:48.531447+0000 osd.4 (osd.4) 1862 : 2021-02-10T04:45:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:45:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[29319]: -02-10T04:45:48.531452+0000 osd.4 (osd.4) 1863 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[29319]: audit 2021-02-10T04:45: 2021-02-10T04:45:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[29319]: 48.535017+0000 mon.a (mon.0) 2089 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pgp_num_actual", "val": "42"}]': finished 2021-02-10T04:45:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[29319]: cluster 2021-02-10T04:45:48.535101+0000 2021-02-10T04:45:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[29319]: mon.a (mon.0) 2090 : cluster [DBG] osdmap e1046: 8 total, 8 up, 7 in 2021-02-10T04:45:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[29319]: audit 2021-02-10T04:45:48.723073+0000 2021-02-10T04:45:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[29319]: mgr.y (mgr.14140) 993 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:45:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[35961]: cluster 2021-02-10T04:45:48.307378+0000 2021-02-10T04:45:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[35961]: mgr.y (mgr.14140) 992 : cluster [DBG] pgmap v1760: 188 pgs: 1 peering, 8 unknown, 1 active+recovering+undersized+remapped, 178 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1/2181 objects misplaced (0.046%); 4.0 MiB/s, 2 objects/s recovering 2021-02-10T04:45:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:45:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[35961]: T04:45:48.531437+0000 osd.4 (osd.4) 1860 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:45:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[35961]: 45:48.531443+0000 osd.4 (osd.4) 1861 2021-02-10T04:45:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:45:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[35961]: T04:45:48.531447+0000 osd.4 (osd.4) 1862 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[35961]: cluster 2021-02-10T04:45: 2021-02-10T04:45:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[35961]: 48.531452+0000 osd.4 (osd.4) 1863 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[35961]: audit 2021-02-10T04:45:48 2021-02-10T04:45:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[35961]: .535017+0000 mon.a (mon.0) 2089 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pgp_num_actual", "val": "42"}]': finished 2021-02-10T04:45:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[35961]: cluster 2021-02-10T04:45:48.535101 2021-02-10T04:45:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2090 : cluster [DBG] osdmap e1046: 8 total, 8 up, 7 in 2021-02-10T04:45:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[35961]: audit 2021-02-10T04:45: 2021-02-10T04:45:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:49 smithi085 conmon[35961]: 48.723073+0000 mgr.y (mgr.14140) 993 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:45:50.068 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:45:50.102 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:45:50.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:50 smithi164 conmon[33677]: debug 2021-02-10T04:45:50.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:50 smithi164 conmon[33677]: 2021-02-10T04:45:50.582+0000 7fbec19a1700 -1 osd.4 1048 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:45:50.895 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:50 smithi164 conmon[33677]: 2021-02-10T04:45:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:50 smithi164 conmon[29288]: cluster 2021-02-10T04:45:49.542192+0000 mon.a (mon.0 2021-02-10T04:45:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:50 smithi164 conmon[29288]: ) 2091 : cluster [DBG] osdmap e1047: 8 total, 8 up, 7 in 2021-02-10T04:45:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:50 smithi164 conmon[29288]: cluster 2021-02-10T04:45:49.567569+0000 osd.4 (osd.4) 1864 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:50 smithi164 conmon[29288]: cluster 2021-02-10T04:45:49.567575+0000 osd.4 (osd.4 2021-02-10T04:45:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:50 smithi164 conmon[29288]: ) 1865 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:50 smithi164 conmon[29288]: cluster 2021-02-10T04:45: 2021-02-10T04:45:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:50 smithi164 conmon[29288]: 49.567579+0000 osd.4 (osd.4) 1866 : 2021-02-10T04:45:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:50 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:50 smithi164 conmon[29288]: cluster 2021-02-10T04:45: 2021-02-10T04:45:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:50 smithi164 conmon[29288]: 49.567583+0000 osd.4 (osd.4) 1867 2021-02-10T04:45:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:50 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:50 smithi164 conmon[29288]: cluster 2021-02-10T04:45:49. 2021-02-10T04:45:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:50 smithi164 conmon[29288]: 757547+0000 osd.4 (osd.4) 1868 : cluster 2021-02-10T04:45:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:50 smithi164 conmon[29288]: [DBG] purged_snaps scrub starts 2021-02-10T04:45:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:50 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:45:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:50 smithi164 conmon[29288]: T04:45:49.759071+0000 osd.4 (osd.4 2021-02-10T04:45:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:50 smithi164 conmon[29288]: ) 1869 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:45:50.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:50 smithi085 conmon[35961]: cluster 2021-02-10T04:45:49. 2021-02-10T04:45:50.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:50 smithi085 conmon[35961]: 542192+0000 mon.a (mon.0) 2091 : cluster [DBG] osdmap e1047: 8 total, 8 up, 7 in 2021-02-10T04:45:50.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:50 smithi085 conmon[35961]: cluster 2021-02-10T04:45 2021-02-10T04:45:50.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:50 smithi085 conmon[35961]: :49.567569+0000 osd.4 (osd.4) 1864 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:50.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:50 smithi085 conmon[35961]: cluster 2021-02-10T04:45:49.567575+0000 osd.4 (osd. 2021-02-10T04:45:50.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:50 smithi085 conmon[35961]: 4) 1865 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:50.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:50 smithi085 conmon[35961]: cluster 2021-02-10T04:45:49.567579+0000 osd.4 (osd 2021-02-10T04:45:50.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:50 smithi085 conmon[35961]: .4) 1866 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:50.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:50 smithi085 conmon[35961]: cluster 2021-02-10T04:45:49.567583+0000 osd.4 (osd 2021-02-10T04:45:50.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:50 smithi085 conmon[35961]: .4) 1867 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:50.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:50 smithi085 conmon[35961]: cluster 2021-02-10T04:45:49.757547+0000 osd.4 2021-02-10T04:45:50.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:50 smithi085 conmon[35961]: (osd.4) 1868 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:45:50.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:50 smithi085 conmon[35961]: cluster 2021-02-10T04:45:49.759071+0000 2021-02-10T04:45:50.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:50 smithi085 conmon[35961]: osd.4 (osd.4) 1869 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:45:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:50 smithi085 conmon[29319]: cluster 2021-02-10T04:45:49.542192+0000 mon.a (mon. 2021-02-10T04:45:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:50 smithi085 conmon[29319]: 0) 2091 : cluster [DBG] osdmap e1047: 8 total, 8 up, 7 in 2021-02-10T04:45:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:50 smithi085 conmon[29319]: cluster 2021-02-10T04:45:49.567569+0000 osd.4 (osd.4) 1864 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:50 smithi085 conmon[29319]: cluster 2021-02-10T04:45:49.567575+0000 osd.4 (osd.4) 1865 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:50 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:45:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:50 smithi085 conmon[29319]: 02-10T04:45:49.567579+0000 2021-02-10T04:45:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:50 smithi085 conmon[29319]: osd.4 (osd.4) 1866 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:50.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:50 smithi085 conmon[29319]: 2021-02-10T04:45:50.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:50 smithi085 conmon[29319]: cluster 2021-02-10T04:45: 2021-02-10T04:45:50.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:50 smithi085 conmon[29319]: 49.567583+0000 osd.4 (osd.4) 1867 : 2021-02-10T04:45:50.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:50 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:50.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:50 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:45:50.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:50 smithi085 conmon[29319]: 04:45:49.757547+0000 osd.4 (osd.4 2021-02-10T04:45:50.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:50 smithi085 conmon[29319]: ) 1868 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:45:50.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:50 smithi085 conmon[29319]: cluster 2021-02-10T04:45: 2021-02-10T04:45:50.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:50 smithi085 conmon[29319]: 49.759071+0000 osd.4 (osd.4) 1869 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:45:51.416 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.6 dump_ops_in_flight 2021-02-10T04:45:51.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:51 smithi164 conmon[29288]: cluster 2021-02-10T04:45:51.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:51 smithi164 conmon[29288]: 2021-02-10T04:45:51.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:51 smithi164 conmon[29288]: 2021 2021-02-10T04:45:51.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:51 smithi164 conmon[29288]: - 2021-02-10T04:45:51.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:51 smithi164 conmon[29288]: 02 2021-02-10T04:45:51.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:51 smithi164 conmon[29288]: - 2021-02-10T04:45:51.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:51 smithi164 conmon[29288]: 10 2021-02-10T04:45:51.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:51 smithi164 conmon[29288]: T 2021-02-10T04:45:51.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:51 smithi164 conmon[29288]: 04:45:50.308270+0000 2021-02-10T04:45:51.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:51 smithi164 conmon[29288]: mgr.y (mgr.14140) 994 : cluster [DBG] pgmap v1763: 188 pgs: 7 peering, 2 unknown, 179 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2021-02-10T04:45:51.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:51 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:45:51.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:51 smithi164 conmon[29288]: T04:45:50.562282+0000 mon.a (mon.0 2021-02-10T04:45:51.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:51 smithi164 conmon[29288]: ) 2092 : cluster [DBG] osdmap e1048: 8 total, 8 up, 7 in 2021-02-10T04:45:51.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:51 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:45:51.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:51 smithi164 conmon[29288]: :45:50.582944+0000 osd.4 (osd.4) 1870 : cluster 2021-02-10T04:45:51.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:51 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:51.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:51 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:45:51.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:51 smithi164 conmon[29288]: 02-10T04:45:50 2021-02-10T04:45:51.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:51 smithi164 conmon[29288]: .582953+0000 osd.4 (osd. 2021-02-10T04:45:51.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:51 smithi164 conmon[29288]: 4) 1871 : cluster [WRN] 2021-02-10T04:45:51.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:51 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:51.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:51 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:45:51.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:51 smithi164 conmon[29288]: -10T04:45:50.582958+0000 osd.4 ( 2021-02-10T04:45:51.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:51 smithi164 conmon[29288]: osd.4) 1872 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:51.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:51 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:45:51.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:51 smithi164 conmon[29288]: -02-10T04:45:50.582964+0000 2021-02-10T04:45:51.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:51 smithi164 conmon[29288]: osd.4 (osd.4) 1873 : cluster [WRN] 2021-02-10T04:45:51.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:51 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:51.649 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:51 smithi164 conmon[33677]: debug 2021-02-10T04:45:51.601+0000 7fbec19a1700 -1 osd.4 1048 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:45:51.798 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:45:51.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:51 smithi085 conmon[35961]: cluster 2021-02-10T04:45:50.308270+0000 mgr.y ( 2021-02-10T04:45:51.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:51 smithi085 conmon[35961]: mgr.14140) 994 : cluster [DBG] pgmap v1763: 188 pgs: 7 peering, 2 unknown, 179 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2021-02-10T04:45:51.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:51 smithi085 conmon[35961]: cluster 2021-02-10T04:45:50.562282+0000 mon.a (mon.0) 2092 : cluster [DBG] osdmap e1048: 8 total, 8 up, 7 in 2021-02-10T04:45:51.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:51 smithi085 conmon[35961]: cluster 2021-02-10T04:45:50.582944+0000 2021-02-10T04:45:51.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:51 smithi085 conmon[35961]: osd.4 (osd.4) 1870 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:51.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:51 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:45:51.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:51 smithi085 conmon[35961]: -10T04:45:50.582953+0000 osd.4 ( 2021-02-10T04:45:51.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:51 smithi085 conmon[35961]: osd.4) 1871 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:51.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:51 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:45:51.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:51 smithi085 conmon[35961]: -02-10T04:45:50.582958+0000 osd.4 (osd.4) 1872 2021-02-10T04:45:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:51 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:51 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:45:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:51 smithi085 conmon[35961]: 10T04:45:50.582964+0000 osd.4 (osd.4) 1873 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:51.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:51 smithi085 conmon[29319]: cluster 2021-02-10T04:45:50.308270+0000 mgr.y (mgr.14140) 994 : cluster [DBG] pgmap v1763: 188 pgs: 7 peering, 2 unknown, 179 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2021-02-10T04:45:51.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:51 smithi085 conmon[29319]: cluster 2021-02-10T04:45:50.562282+0000 mon.a (mon.0) 2092 : cluster [DBG] osdmap e1048: 8 total, 8 up, 7 in 2021-02-10T04:45:51.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:51 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:45:51.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:51 smithi085 conmon[29319]: 04:45:50.582944+0000 osd.4 (osd. 2021-02-10T04:45:51.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:51 smithi085 conmon[29319]: 4) 1870 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:51.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:51 smithi085 conmon[29319]: cluster 2021-02-10T04:45:51.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:51 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:45:51.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:51 smithi085 conmon[29319]: 45:50.582953+0000 osd.4 (osd 2021-02-10T04:45:51.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:51 smithi085 conmon[29319]: .4) 1871 : 2021-02-10T04:45:51.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:51 smithi085 conmon[29319]: cluster 2021-02-10T04:45:51.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:51 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:51.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:51 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:45:51.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:51 smithi085 conmon[29319]: 10T04:45:50.582958+0000 osd.4 (osd.4 2021-02-10T04:45:51.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:51 smithi085 conmon[29319]: ) 1872 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:51.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:51 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:45:51.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:51 smithi085 conmon[29319]: T04:45:50.582964+0000 osd.4 (osd. 2021-02-10T04:45:51.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:51 smithi085 conmon[29319]: 4) 1873 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:52 smithi164 conmon[29288]: cluster 2021-02-10T04:45:51.602080+0000 osd.4 2021-02-10T04:45:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:52 smithi164 conmon[29288]: (osd.4) 1874 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:52 smithi164 conmon[29288]: cluster 2021-02-10T04:45:51.602091+0000 osd.4 (osd.4) 1875 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:52 smithi164 conmon[29288]: cluster 2021-02-10T04:45:51.602099+0000 osd.4 (osd.4) 1876 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:52 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:45:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:52 smithi164 conmon[29288]: :45:51.602103+0000 osd.4 (osd.4) 1877 : cluster [WRN] 2021-02-10T04:45:52.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:52 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:52.895 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:52 smithi164 conmon[33677]: debug 2021-02-10T04:45:52.580+0000 7fbec19a1700 -1 osd.4 1048 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:45:52.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:52 smithi085 conmon[35961]: cluster 2021-02-10T04:45:52.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:52 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:45:52.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:52 smithi085 conmon[35961]: 10T04:45:51.602080+0000 osd.4 (osd.4) 1874 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:52.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:52 smithi085 conmon[35961]: cluster 2021-02-10T04:45:52.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:52 smithi085 conmon[35961]: 2021-02-10T04:45 2021-02-10T04:45:52.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:52 smithi085 conmon[35961]: :51.602091+0000 osd.4 (osd.4) 1875 : cluster [WRN] 2021-02-10T04:45:52.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:52 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:52.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:52 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:45:52.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:52 smithi085 conmon[35961]: 45:51.602099+0000 osd.4 (osd.4) 1876 2021-02-10T04:45:52.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:52 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:52.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:52 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:45:52.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:52 smithi085 conmon[35961]: T04:45:51.602103+0000 osd.4 (osd. 2021-02-10T04:45:52.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:52 smithi085 conmon[35961]: 4) 1877 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:52 smithi085 conmon[29319]: cluster 2021-02-10T04:45:51.602080 2021-02-10T04:45:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:52 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1874 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:52 smithi085 conmon[29319]: cluster 2021-02-10T04:45:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:52 smithi085 conmon[29319]: 2021-02-10T04:45:51.602091+0000 osd.4 (osd.4) 1875 : 2021-02-10T04:45:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:52 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:52 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:45:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:52 smithi085 conmon[29319]: :45:51.602099+0000 osd.4 (osd.4 2021-02-10T04:45:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:52 smithi085 conmon[29319]: ) 1876 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:52 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:45:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:52 smithi085 conmon[29319]: 10T04:45:51.602103+0000 osd.4 (osd.4) 1877 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:52.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:52 smithi085 conmon[29319]: 2021-02-10T04:45:53.160 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:45:53.334 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:45:53.411 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[52836]: debug 2021-02-10T04:45:53.097+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:45:53.414 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:45:53.568 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:53 smithi164 conmon[29288]: cluster 2021-02-10T04:45:52.308826+0000 mgr.y (mgr.14140) 995 : cluster [DBG] pgmap v1765: 188 pgs: 6 peering, 1 unknown, 181 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2021-02-10T04:45:53.568 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:53 smithi164 conmon[29288]: cluster 2021-02-10T04:45:52.581483+0000 osd.4 (osd.4) 1878 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:53.568 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:53 smithi164 conmon[29288]: cluster 2021-02-10T04:45:52.581495+0000 osd.4 (osd.4) 1879 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:53.568 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:53 smithi164 conmon[29288]: cluster 2021-02-10T04:45:52.581504+0000 osd.4 (osd.4) 1880 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:53.569 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:53 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:45:53.569 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:53 smithi164 conmon[29288]: -10T04:45:52.581510+0000 osd.4 (osd.4) 1881 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:53.569 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:53 smithi164 conmon[29288]: cluster 2021-02-10T04:45:53.376087+0000 mon.a (mon.0) 2093 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 87 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:45:53.576 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:45:53.680 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[45443]: debug 2021-02-10T04:45:53.540+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:45:53.681 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:45:53.682 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[29319]: cluster 2021-02-10T04:45:52.308826+0000 mgr.y ( 2021-02-10T04:45:53.682 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[29319]: mgr.14140) 995 : cluster [DBG] pgmap v1765: 188 pgs: 6 peering, 1 unknown, 181 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2021-02-10T04:45:53.682 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[29319]: cluster 2021-02-10T04:45:52.581483+0000 2021-02-10T04:45:53.682 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[29319]: osd.4 (osd.4) 1878 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:53.682 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:45:53.682 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[29319]: -10T04:45:52.581495+0000 osd.4 (osd.4) 1879 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:53.682 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[29319]: cluster 2021-02-10T04:45:52 2021-02-10T04:45:53.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[29319]: .581504+0000 osd.4 (osd.4 2021-02-10T04:45:53.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[29319]: ) 1880 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:53.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:45:53.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[29319]: 02-10T04:45:52.581510+0000 osd.4 (osd.4) 2021-02-10T04:45:53.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[29319]: 1881 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:53.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:45:53.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[29319]: 45:53.376087+0000 mon.a (mon.0) 2093 : cluster [WRN] 2021-02-10T04:45:53.683 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[29319]: Health check update: 4 slow ops, oldest one blocked for 87 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:45:53.683 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[35961]: cluster 2021-02-10T04:45:52.308826 2021-02-10T04:45:53.683 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 995 : cluster [DBG] pgmap v1765: 188 pgs: 6 peering, 1 unknown, 181 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2021-02-10T04:45:53.683 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[35961]: cluster 2021-02-10T04:45:52.581483 2021-02-10T04:45:53.684 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1878 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:53.684 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:45:53.684 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[35961]: 10T04:45:52.581495+0000 osd.4 (osd.4) 1879 : cluster 2021-02-10T04:45:53.684 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:53.684 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:45:53.684 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[35961]: -02-10T04:45:52.581504+0000 2021-02-10T04:45:53.684 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[35961]: osd.4 (osd.4) 1880 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:53.684 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[35961]: 2021-02-10T04:45:53.684 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:45:53.684 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[35961]: 02-10T04:45:52.581510+0000 osd.4 (osd.4) 2021-02-10T04:45:53.684 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[35961]: 1881 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:53.684 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[35961]: cluster 2021-02-10T04:45: 2021-02-10T04:45:53.684 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[35961]: 53.376087+0000 mon.a (mon.0) 2093 : 2021-02-10T04:45:53.685 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[35961]: cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 87 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:45:53.718 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:45:53.892 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:53 smithi164 conmon[33677]: debug 2021-02-10T04:45:53.893 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:53 smithi164 conmon[33677]: 2021-02-10T04:45:53.605+0000 7fbec19a1700 -1 osd.4 1048 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:45:53.893 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:45:53.941 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:45:53 smithi085 conmon[41540]: debug 2021-02-10T04:45:53.800+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:45:53.998 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:45:54.031 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:45:54.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:54 smithi085 conmon[35961]: cluster 2021-02-10T04:45:53.606435+0000 osd.4 (osd 2021-02-10T04:45:54.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:54 smithi085 conmon[35961]: .4) 1882 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:54.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:54 smithi085 conmon[35961]: cluster 2021-02-10T04:45:53.606444 2021-02-10T04:45:54.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:54 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1883 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:54.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:54 smithi085 conmon[35961]: cluster 2021-02-10T04:45:54.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:54 smithi085 conmon[35961]: 2021-02-10T04:45:53.606450+0000 osd.4 (osd.4) 1884 : 2021-02-10T04:45:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:54 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:54 smithi085 conmon[35961]: cluster 2021-02-10T04:45:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:54 smithi085 conmon[35961]: 2021-02-10T04:45:53.606456 2021-02-10T04:45:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:54 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1885 : cluster [WRN] 2021-02-10T04:45:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:54 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:54.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:54 smithi085 conmon[29319]: cluster 2021-02-10T04:45:53.606435 2021-02-10T04:45:54.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:54 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1882 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:54.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:54 smithi085 conmon[29319]: cluster 2021-02-10T04:45:53. 2021-02-10T04:45:54.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:54 smithi085 conmon[29319]: 606444+0000 osd.4 (osd.4) 1883 : cluster 2021-02-10T04:45:54.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:54 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:54.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:54 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:45:54.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:54 smithi085 conmon[29319]: 02-10T04:45:53 2021-02-10T04:45:54.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:54 smithi085 conmon[29319]: .606450+0000 osd.4 (osd. 2021-02-10T04:45:54.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:54 smithi085 conmon[29319]: 4) 1884 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:54.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:54 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:45:54.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:54 smithi085 conmon[29319]: -02-10T04:45:53 2021-02-10T04:45:54.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:54 smithi085 conmon[29319]: .606456+0000 osd.4 (osd. 2021-02-10T04:45:54.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:54 smithi085 conmon[29319]: 4) 1885 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:54.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:54 smithi085 conmon[29319]: 2021-02-10T04:45:54.851 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:54 smithi164 conmon[29288]: cluster 2021-02-10T04:45:53.606435+0000 osd.4 (osd.4) 2021-02-10T04:45:54.851 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:54 smithi164 conmon[29288]: 1882 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:54.851 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:54 smithi164 conmon[29288]: cluster 2021-02-10T04:45:53.606444+0000 osd.4 (osd.4) 1883 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:54.851 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:54 smithi164 conmon[29288]: cluster 2021-02-10T04:45:53.606450+0000 osd.4 (osd.4) 1884 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:54.852 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:54 smithi164 conmon[29288]: cluster 2021-02-10T04:45:53.606456+0000 osd.4 (osd.4) 1885 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:54.852 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:54 smithi164 conmon[33677]: debug 2021-02-10T04:45:54.609+0000 7fbec19a1700 -1 osd.4 1048 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:45:55.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:45:55 smithi085 conmon[49106]: debug 2021-02-10T04:45:55.171+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:45:55.444 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:45:55.895 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:55 smithi164 conmon[33677]: debug 2021-02-10T04:45:55.657+0000 7fbec19a1700 -1 osd.4 1048 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:45:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:55 smithi164 conmon[29288]: cluster 2021-02-10T04:45:54.309524+0000 2021-02-10T04:45:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:55 smithi164 conmon[29288]: mgr.y (mgr.14140) 996 : cluster [DBG] pgmap v1766: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2021-02-10T04:45:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:55 smithi164 conmon[29288]: cluster 2021-02-10T04:45:54.610550+0000 osd.4 (osd.4) 1886 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:55 smithi164 conmon[29288]: cluster 2021-02-10T04:45:54.610559 2021-02-10T04:45:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:55 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1887 2021-02-10T04:45:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:55 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:55 smithi164 conmon[29288]: cluster 2021-02-10T04:45:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:55 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:45:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:55 smithi164 conmon[29288]: :45:54.610566+0000 osd.4 (osd.4) 1888 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:55 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:45:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:55 smithi164 conmon[29288]: -02-10T04:45:54. 2021-02-10T04:45:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:55 smithi164 conmon[29288]: 610570+0000 osd.4 (osd.4) 1889 2021-02-10T04:45:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:55 smithi164 conmon[29288]: : cluster [WRN] 2021-02-10T04:45:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:55 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:55.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:55 smithi085 conmon[35961]: cluster 2021-02-10T04:45:55.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:55 smithi085 conmon[35961]: 2021-02-10T04:45:55.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:55 smithi085 conmon[35961]: 2021-02-10T04:45:54.309524 2021-02-10T04:45:55.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:55 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 996 : cluster [DBG] pgmap v1766: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2021-02-10T04:45:55.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:55 smithi085 conmon[35961]: cluster 2021-02-10T04:45:55.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:55 smithi085 conmon[35961]: 2021-02-10T04:45:54 2021-02-10T04:45:55.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:55 smithi085 conmon[35961]: .610550+0000 osd.4 (osd.4) 1886 : 2021-02-10T04:45:55.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:55 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:55.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:55 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:45:55.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:55 smithi085 conmon[35961]: T04:45:54.610559+0000 osd.4 ( 2021-02-10T04:45:55.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:55 smithi085 conmon[35961]: osd.4) 1887 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:55.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:55 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:45:55.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:55 smithi085 conmon[35961]: -02-10T04:45:54.610566+0000 osd.4 (osd.4) 1888 2021-02-10T04:45:55.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:55 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:55.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:55 smithi085 conmon[35961]: cluster 2021-02-10T04:45:54.610570+0000 2021-02-10T04:45:55.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:55 smithi085 conmon[35961]: osd.4 (osd.4) 1889 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:55.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:55 smithi085 conmon[29319]: cluster 2021-02-10T04:45:54.309524+0000 2021-02-10T04:45:55.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:55 smithi085 conmon[29319]: mgr.y (mgr.14140) 996 : cluster [DBG] pgmap v1766: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2021-02-10T04:45:55.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:55 smithi085 conmon[29319]: cluster 2021-02-10T04:45:54.610550+0000 osd.4 (osd.4) 1886 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:55.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:55 smithi085 conmon[29319]: cluster 2021-02-10T04:45:54.610559 2021-02-10T04:45:55.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:55 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1887 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:55.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:55 smithi085 conmon[29319]: cluster 2021-02-10T04:45:54.610566+0000 osd.4 (osd.4) 1888 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:55.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:55 smithi085 conmon[29319]: cluster 2021-02-10T04:45:54.610570+0000 osd.4 (osd 2021-02-10T04:45:55.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:55 smithi085 conmon[29319]: .4) 1889 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:56.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:56 smithi164 conmon[33677]: debug 2021-02-10T04:45:56.627+0000 7fbec19a1700 -1 osd.4 1049 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:45:56.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:56 smithi164 conmon[29288]: cluster 2021-02-10T04:45:56.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:56 smithi164 conmon[29288]: 2021-02-10T04:45:56.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:56 smithi164 conmon[29288]: 2021 2021-02-10T04:45:56.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:56 smithi164 conmon[29288]: - 2021-02-10T04:45:56.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:56 smithi164 conmon[29288]: 02 2021-02-10T04:45:56.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:56 smithi164 conmon[29288]: - 2021-02-10T04:45:56.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:56 smithi164 conmon[29288]: 10 2021-02-10T04:45:56.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:56 smithi164 conmon[29288]: T04:45:55.616247+0000 osd.1 2021-02-10T04:45:56.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:56 smithi164 conmon[29288]: (osd.1) 469 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:45:56.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:56 smithi164 conmon[29288]: cluster 2021-02-10T04:45:55.620151+0000 osd.1 (osd.1) 470 : cluster [DBG] 2021-02-10T04:45:56.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:56 smithi164 conmon[29288]: purged_snaps scrub ok 2021-02-10T04:45:56.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:56 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:45:56.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:56 smithi164 conmon[29288]: -10T04:45:55.658013+0000 osd.4 (osd.4) 1890 : cluster 2021-02-10T04:45:56.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:56 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:56.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:56 smithi164 conmon[29288]: cluster 2021-02-10T04:45:55.658023 2021-02-10T04:45:56.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:56 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1891 : cluster [WRN] 2021-02-10T04:45:56.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:56 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:56.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:56 smithi164 conmon[29288]: cluster 2021-02-10T04:45:55.658031 2021-02-10T04:45:56.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:56 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1892 : cluster [WRN] 2021-02-10T04:45:56.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:56 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:56.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:56 smithi164 conmon[29288]: cluster 2021-02-10T04:45:55 2021-02-10T04:45:56.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:56 smithi164 conmon[29288]: .658038+0000 osd.4 (osd.4) 1893 : cluster 2021-02-10T04:45:56.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:56 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[29319]: cluster 2021-02-10T04:45:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:45:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[29319]: T04:45:55.616247+0000 osd.1 (osd.1) 469 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:45:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[29319]: cluster 2021-02-10T04:45:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[29319]: 2021-02-10T04:45:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[29319]: 2021 2021-02-10T04:45:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[29319]: - 2021-02-10T04:45:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[29319]: 02 2021-02-10T04:45:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[29319]: - 2021-02-10T04:45:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[29319]: 10 2021-02-10T04:45:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[29319]: T 2021-02-10T04:45:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[29319]: 04 2021-02-10T04:45:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[29319]: : 2021-02-10T04:45:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[29319]: 45 2021-02-10T04:45:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[29319]: :55.620151+0000 osd.1 (osd 2021-02-10T04:45:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[29319]: .1) 470 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:45:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[29319]: cluster 2021-02-10T04:45:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[29319]: 2021-02-10T04:45:55. 2021-02-10T04:45:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[29319]: 658013+0000 osd.4 (osd.4) 1890 : cluster [WRN] 2021-02-10T04:45:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[29319]: cluster 2021-02-10T04:45: 2021-02-10T04:45:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[29319]: 55.658023+0000 osd.4 (osd.4) 1891 : cluster 2021-02-10T04:45:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:45:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[29319]: 45:55.658031+0000 osd.4 (osd 2021-02-10T04:45:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[29319]: .4) 1892 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[29319]: cluster 2021-02-10T04:45:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[29319]: 2021-02-10T04:45:55.658038+0000 2021-02-10T04:45:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[29319]: osd.4 (osd.4) 1893 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[35961]: cluster 2021-02-10T04:45:55.616247 2021-02-10T04:45:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 469 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:45:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[35961]: cluster 2021-02-10T04:45:55.620151+0000 osd.1 2021-02-10T04:45:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[35961]: (osd.1) 470 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:45:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[35961]: cluster 2021-02-10T04:45: 2021-02-10T04:45:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[35961]: 55.658013+0000 osd.4 (osd.4) 1890 2021-02-10T04:45:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:45:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[35961]: 45:55.658023+0000 osd.4 (osd.4 2021-02-10T04:45:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[35961]: ) 1891 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:45:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[35961]: T04:45:55.658031+0000 osd.4 ( 2021-02-10T04:45:56.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[35961]: osd.4) 1892 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:56.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:45:56.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[35961]: -02-10T04:45:55.658038+0000 2021-02-10T04:45:56.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:56 smithi085 conmon[35961]: osd.4 (osd.4) 1893 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:57.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:57 smithi164 conmon[33677]: debug 2021-02-10T04:45:57.669+0000 7fbec19a1700 -1 osd.4 1049 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:45:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:57 smithi164 conmon[29288]: cluster 2021-02-10T04:45:56.310226+0000 mgr.y ( 2021-02-10T04:45:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:57 smithi164 conmon[29288]: mgr.14140) 997 : cluster [DBG] pgmap v1767: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2021-02-10T04:45:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:57 smithi164 conmon[29288]: cluster 2021-02-10T04:45:56.582224+0000 mon.a (mon.0) 2021-02-10T04:45:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:57 smithi164 conmon[29288]: 2094 : cluster [DBG] osdmap e1049: 8 total, 8 up, 7 in 2021-02-10T04:45:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:57 smithi164 conmon[29288]: cluster 2021-02-10T04:45: 2021-02-10T04:45:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:57 smithi164 conmon[29288]: 56.628589+0000 osd.4 (osd.4) 1894 : cluster 2021-02-10T04:45:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:57 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:57 smithi164 conmon[29288]: cluster 2021-02-10T04:45:56.628600+0000 osd.4 (osd.4) 1895 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:57 smithi164 conmon[29288]: cluster 2021-02-10T04:45:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:57 smithi164 conmon[29288]: 2021-02-10T04:45:56.628607+0000 osd.4 (osd.4) 2021-02-10T04:45:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:57 smithi164 conmon[29288]: 1896 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:57 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:45:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:57 smithi164 conmon[29288]: -02-10T04:45:56. 2021-02-10T04:45:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:57 smithi164 conmon[29288]: 628615+0000 osd.4 (osd.4) 2021-02-10T04:45:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:57 smithi164 conmon[29288]: 1897 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[35961]: cluster 2021-02-10T04:45:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[35961]: 2021-02-10T04:45:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[35961]: 2021 2021-02-10T04:45:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[35961]: -02-10T04: 2021-02-10T04:45:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[35961]: 45:56.310226+0000 mgr.y (mgr.14140) 997 : cluster [DBG] pgmap v1767: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2021-02-10T04:45:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:45:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[35961]: 02-10T04:45:56.582224+0000 mon.a ( 2021-02-10T04:45:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[35961]: mon.0) 2094 : cluster [DBG] osdmap e1049: 8 total, 8 up, 7 in 2021-02-10T04:45:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:45:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[35961]: -02-10T04:45:56.628589+0000 2021-02-10T04:45:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[35961]: osd.4 (osd.4) 2021-02-10T04:45:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[35961]: 1894 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[35961]: cluster 2021-02-10T04:45:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[35961]: 2021-02-10T04:45 2021-02-10T04:45:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[35961]: :56.628600+0000 osd.4 (osd.4) 1895 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:45:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[35961]: T04:45:56.628607+0000 2021-02-10T04:45:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[35961]: osd.4 (osd.4 2021-02-10T04:45:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[35961]: ) 1896 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[35961]: 2021-02-10T04:45:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[35961]: cluster 2021-02-10T04:45:56.628615+0000 osd.4 (osd.4) 2021-02-10T04:45:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[35961]: 1897 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[29319]: cluster 2021-02-10T04:45:56.310226+0000 mgr.y (mgr.14140) 997 : cluster [DBG] pgmap v1767: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2021-02-10T04:45:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[29319]: 2021-02-10T04:45:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[29319]: cluster 2021-02-10T04:45:56.582224+0000 mon.a (mon.0) 2094 : cluster [DBG] osdmap e1049: 8 total, 8 up, 7 in 2021-02-10T04:45:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:45:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[29319]: 45:56.628589+0000 osd.4 (osd.4) 1894 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:45:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[29319]: -02-10T04:45:56.628600+0000 osd.4 (osd.4) 1895 : cluster 2021-02-10T04:45:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:45:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[29319]: -10T04:45:56.628607+0000 osd.4 (osd.4) 1896 : cluster [WRN] 2021-02-10T04:45:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[29319]: cluster 2021-02-10T04:45:57.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[29319]: 2021-02-10T04:45:56. 2021-02-10T04:45:57.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[29319]: 628615+0000 osd.4 (osd.4) 1897 : 2021-02-10T04:45:57.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:57 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:58.337 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-02-10T04:45:58.337 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:45:58.337 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd dump -f json-pretty 2021-02-10T04:45:58.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:58 smithi085 conmon[29319]: cluster 2021-02-10T04:45:57.669713 2021-02-10T04:45:58.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:58 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1898 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:58.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:58 smithi085 conmon[29319]: cluster 2021-02-10T04:45:57.669736+0000 osd.4 (osd.4) 1899 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:58.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:58 smithi085 conmon[29319]: cluster 2021-02-10T04:45:57.669744+0000 osd.4 (osd.4) 1900 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:58.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:58 smithi085 conmon[29319]: cluster 2021-02-10T04:45:57.669750+0000 osd.4 (osd.4) 1901 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:58 smithi085 conmon[35961]: cluster 2021-02-10T04:45:57.669713+0000 2021-02-10T04:45:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:58 smithi085 conmon[35961]: osd.4 (osd.4) 1898 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:58.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:58 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:45:58.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:58 smithi085 conmon[35961]: T04:45:57.669736+0000 osd.4 (osd.4) 1899 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:58.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:58 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:45:58.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:58 smithi085 conmon[35961]: -02-10T04:45:57.669744+0000 osd.4 2021-02-10T04:45:58.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:58 smithi085 conmon[35961]: (osd.4) 1900 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:58.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:58 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:45:58.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:58 smithi085 conmon[35961]: -10T04:45:57.669750+0000 osd.4 2021-02-10T04:45:58.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:58 smithi085 conmon[35961]: (osd.4) 1901 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:58.894 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:45:58 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:45:58.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:58 smithi164 conmon[33677]: debug 2021-02-10T04:45:58.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:58 smithi164 conmon[33677]: 2021-02-10T04:45:58.648+0000 7fbec19a1700 -1 osd.4 1049 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:45:58.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:58 smithi164 conmon[29288]: cluster 2021-02-10T04:45:57.669713+0000 osd.4 2021-02-10T04:45:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:58 smithi164 conmon[29288]: (osd.4) 1898 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:58 smithi164 conmon[29288]: cluster 2021-02-10T04:45:57.669736+0000 osd.4 (osd.4) 1899 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:58 smithi164 conmon[29288]: cluster 2021-02-10T04:45:57.669744 2021-02-10T04:45:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:58 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1900 : cluster 2021-02-10T04:45:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:58 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:58 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:45:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:58 smithi164 conmon[29288]: :45:57.669750+0000 osd.4 (osd.4) 2021-02-10T04:45:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:58 smithi164 conmon[29288]: 1901 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:59.194 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:45:59.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:45:59 smithi164 conmon[33677]: debug 2021-02-10T04:45:59.628+0000 7fbec19a1700 -1 osd.4 1049 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:45:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:59 smithi164 conmon[29288]: cluster 2021-02-10T04:45:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:59 smithi164 conmon[29288]: 2021-02-10T04:45: 2021-02-10T04:45:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:59 smithi164 conmon[29288]: 58.310861+0000 mgr.y (mgr.14140) 998 : cluster [DBG] pgmap v1769: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20 KiB/s rd, 0 B/s wr, 33 op/s 2021-02-10T04:45:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:59 smithi164 conmon[29288]: cluster 2021-02-10T04:45:58.579529+0000 2021-02-10T04:45:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:59 smithi164 conmon[29288]: mon.a (mon.0) 2095 : cluster [WRN] 2021-02-10T04:45:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:59 smithi164 conmon[29288]: Health check update: 4 slow ops, oldest one blocked for 92 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:45:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:59 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:45:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:59 smithi164 conmon[29288]: -02-10T04:45: 2021-02-10T04:45:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:59 smithi164 conmon[29288]: 58.648757+0000 osd.4 (osd. 2021-02-10T04:45:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:59 smithi164 conmon[29288]: 4) 1902 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:59 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:45:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:59 smithi164 conmon[29288]: -10T04:45:58.648767+0000 2021-02-10T04:45:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:59 smithi164 conmon[29288]: osd.4 (osd.4) 1903 : cluster 2021-02-10T04:45:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:59 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:59.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:59 smithi164 conmon[29288]: cluster 2021-02-10T04:45:59.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:59 smithi164 conmon[29288]: 2021-02-10T04:45 2021-02-10T04:45:59.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:59 smithi164 conmon[29288]: :58.648773+0000 osd.4 (osd.4) 1904 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:59.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:59 smithi164 conmon[29288]: cluster 2021-02-10T04:45:58 2021-02-10T04:45:59.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:59 smithi164 conmon[29288]: .648780+0000 osd.4 (osd.4) 1905 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:59.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:59 smithi164 conmon[29288]: audit 2021-02-10T04:45:58.733024 2021-02-10T04:45:59.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:59 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 999 : audit [DBG] 2021-02-10T04:45:59.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:45:59 smithi164 conmon[29288]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:45:59.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:59 smithi085 conmon[29319]: cluster 2021-02-10T04:45:58.310861+0000 mgr.y (mgr.14140) 998 : 2021-02-10T04:45:59.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:59 smithi085 conmon[29319]: cluster [DBG] pgmap v1769: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20 KiB/s rd, 0 B/s wr, 33 op/s 2021-02-10T04:45:59.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:59 smithi085 conmon[29319]: cluster 2021-02-10T04:45:58.579529+0000 mon.a (mon.0) 2095 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 92 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:45:59.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:59 smithi085 conmon[29319]: cluster 2021-02-10T04:45:58.648757+0000 osd.4 (osd.4) 1902 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:59 smithi085 conmon[29319]: cluster 2021-02-10T04:45:58.648767+0000 osd.4 (osd.4) 1903 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:59 smithi085 conmon[29319]: cluster 2021-02-10T04:45:58.648773+0000 osd.4 (osd.4) 1904 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:59 smithi085 conmon[29319]: cluster 2021-02-10T04:45:58.648780+0000 osd.4 (osd.4) 1905 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:45:59 smithi085 conmon[29319]: audit 2021-02-10T04:45:58.733024+0000 mgr.y (mgr.14140) 999 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:45:59.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:59 smithi085 conmon[35961]: cluster 2021-02-10T04:45:58.310861+0000 mgr.y (mgr.14140) 998 : 2021-02-10T04:45:59.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:59 smithi085 conmon[35961]: cluster [DBG] pgmap v1769: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20 KiB/s rd, 0 B/s wr, 33 op/s 2021-02-10T04:45:59.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:59 smithi085 conmon[35961]: cluster 2021-02-10T04:45:58.579529+0000 mon.a (mon.0) 2095 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 92 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:45:59.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:59 smithi085 conmon[35961]: cluster 2021-02-10T04:45:58.648757+0000 osd.4 (osd.4) 1902 : cluster 2021-02-10T04:45:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:59 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:45:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:59 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:45:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:59 smithi085 conmon[35961]: -02-10T04:45:58.648767+0000 osd.4 (osd.4) 1903 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:45:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:59 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:45:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:59 smithi085 conmon[35961]: 04:45:58.648773+0000 osd.4 (osd.4) 1904 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:45:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:59 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:45:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:59 smithi085 conmon[35961]: 45:58.648780+0000 osd.4 (osd.4) 1905 : cluster [WRN] 2021-02-10T04:45:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:59 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:45:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:59 smithi085 conmon[35961]: audit 2021-02-10T04:45:58 2021-02-10T04:45:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:59 smithi085 conmon[35961]: .733024+0000 mgr.y (mgr.14140) 2021-02-10T04:45:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:45:59 smithi085 conmon[35961]: 999 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:46:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:00 smithi164 conmon[29288]: cluster 2021-02-10T04:45:59.629169+0000 osd.4 (osd. 2021-02-10T04:46:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:00 smithi164 conmon[29288]: 4) 1906 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:00 smithi164 conmon[29288]: cluster 2021-02-10T04:45:59.629175+0000 osd.4 (osd.4) 1907 : 2021-02-10T04:46:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:00 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:00 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:46:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:00 smithi164 conmon[29288]: -10T04:45:59.629179+0000 osd.4 (osd.4) 1908 : cluster [WRN] 2021-02-10T04:46:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:00 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:00 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:46:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:00 smithi164 conmon[29288]: 10T04:45:59.629182+0000 osd.4 (osd 2021-02-10T04:46:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:00 smithi164 conmon[29288]: .4) 1909 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:00 smithi164 conmon[29288]: cephadm 2021- 2021-02-10T04:46:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:00 smithi164 conmon[29288]: 02-10T04:46:00.044275+0000 mgr.y 2021-02-10T04:46:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:00 smithi164 conmon[29288]: (mgr.14140) 1000 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:46:00.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:00 smithi164 conmon[29288]: cephadm 2021 2021-02-10T04:46:00.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:00 smithi164 conmon[29288]: -02-10T04:46:00.044828+0000 2021-02-10T04:46:00.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:00 smithi164 conmon[29288]: mgr.y (mgr.14140) 1001 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:46:00.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:00 smithi164 conmon[29288]: 2021-02-10T04:46:00.896 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:00 smithi164 conmon[33677]: debug 2021-02-10T04:46:00.664+0000 7fbec19a1700 -1 osd.4 1049 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:46:00.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[29319]: cluster 2021-02-10T04:45:59.629169+0000 2021-02-10T04:46:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[29319]: osd.4 (osd.4) 1906 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:46:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[29319]: :45:59.629175+0000 osd.4 (osd.4 2021-02-10T04:46:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[29319]: ) 1907 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:46:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[29319]: T04:45:59.629179+0000 osd.4 (osd.4) 2021-02-10T04:46:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[29319]: 1908 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[29319]: cluster 2021-02-10T04:46:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[29319]: 2021-02 2021-02-10T04:46:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[29319]: -10T04:45:59.629182+0000 2021-02-10T04:46:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[29319]: osd.4 (osd.4) 1909 : 2021-02-10T04:46:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[29319]: cephadm 2021 2021-02-10T04:46:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[29319]: -02-10T04:46:00.044275+0000 2021-02-10T04:46:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[29319]: mgr.y (mgr.14140) 1000 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:46:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[29319]: cephadm 2021-02-10T04:46:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[29319]: 2021-02-10T04:46:00.044828+0000 2021-02-10T04:46:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[29319]: mgr.y (mgr.14140) 1001 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:46:00.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: cluster 2021-02-10T04:46:00.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: 2021-02-10T04:46:00.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: 2021 2021-02-10T04:46:00.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: - 2021-02-10T04:46:00.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: 02 2021-02-10T04:46:00.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: - 2021-02-10T04:46:00.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: 10 2021-02-10T04:46:00.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: T 2021-02-10T04:46:00.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: 04 2021-02-10T04:46:00.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: : 2021-02-10T04:46:00.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: 45 2021-02-10T04:46:00.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: : 2021-02-10T04:46:00.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: 59 2021-02-10T04:46:00.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: . 2021-02-10T04:46:00.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: 629169 2021-02-10T04:46:00.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: +0000 2021-02-10T04:46:00.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: 2021-02-10T04:46:00.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: osd.4 2021-02-10T04:46:00.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: ( 2021-02-10T04:46:00.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: osd 2021-02-10T04:46:00.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: . 2021-02-10T04:46:00.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: 4 2021-02-10T04:46:00.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: ) 2021-02-10T04:46:00.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: 1906 2021-02-10T04:46:00.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: : 2021-02-10T04:46:00.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: cluster 2021-02-10T04:46:00.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: 2021-02-10T04:46:00.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: [WRN] 2021-02-10T04:46:00.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: 2021-02-10T04:46:00.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:00.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: 2021-02-10T04:46:00.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: cluster 2021-02-10T04:46:00.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: 2021-02-10T04:46:00.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: 2021 2021-02-10T04:46:00.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: - 2021-02-10T04:46:00.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: 02 2021-02-10T04:46:00.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: -10T04:45:59.629175+0000 osd.4 (osd.4) 1907 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:00.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: cluster 2021-02-10T04:45:59.629179+0000 osd.4 ( 2021-02-10T04:46:00.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: osd.4) 1908 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:00.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:46:00.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: 02-10T04:45:59.629182+0000 osd.4 2021-02-10T04:46:00.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: (osd.4) 1909 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:00.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: 2021-02-10T04:46:00.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: cephadm 2021-02-10T04 2021-02-10T04:46:00.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: :46:00.044275+0000 2021-02-10T04:46:00.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: mgr.y (mgr.14140) 1000 : cephadm 2021-02-10T04:46:00.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: [INF] Refreshing smithi085 facts 2021-02-10T04:46:00.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: cephadm 2021-02-10T04:46:00.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: 2021-02-10T04:46:00. 2021-02-10T04:46:00.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: 044828+0000 mgr.y (mgr.14140) 1001 : cephadm [INF] 2021-02-10T04:46:00.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:00 smithi085 conmon[35961]: Refreshing smithi164 facts 2021-02-10T04:46:01.879 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[29319]: cluster 2021-02-10T04:46:00.311750+0000 mgr.y ( 2021-02-10T04:46:01.879 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[29319]: mgr.14140) 1002 : cluster [DBG] pgmap v1770: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2021-02-10T04:46:01.879 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:46:01.879 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[29319]: T04:46:00.665385+0000 osd.4 (osd.4) 1910 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:01.879 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[29319]: cluster 2021-02-10T04:46 2021-02-10T04:46:01.880 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[29319]: :00.665397+0000 osd.4 (osd.4) 1911 : cluster 2021-02-10T04:46:01.880 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:01.880 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[29319]: cluster 2021-02-10T04:46:01.880 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:46:01.880 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[29319]: 04:46:00.665404+0000 osd.4 2021-02-10T04:46:01.880 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[29319]: (osd.4) 1912 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:01.880 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[29319]: 2021-02-10T04:46:01.880 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[29319]: cluster 2021-02-10T04:46:01.880 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[29319]: 2021-02-10T04:46 2021-02-10T04:46:01.880 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[29319]: :00.665410+0000 osd.4 (osd. 2021-02-10T04:46:01.880 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[29319]: 4) 1913 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:01.880 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[29319]: audit 2021-02-10T04: 2021-02-10T04:46:01.880 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[29319]: 46:01.512394+0000 mon.a (mon.0) 2096 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:46:01.881 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[35961]: cluster 2021-02-10T04:46 2021-02-10T04:46:01.881 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[35961]: :00.311750+0000 mgr.y (mgr.14140) 1002 : cluster [DBG] pgmap v1770: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2021-02-10T04:46:01.881 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:46:01.881 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[35961]: 02-10T04:46:00 2021-02-10T04:46:01.881 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[35961]: .665385+0000 osd.4 (osd. 2021-02-10T04:46:01.881 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[35961]: 4) 1910 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:01.881 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[35961]: 2021-02-10T04:46:01.881 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:46:01.881 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[35961]: -02-10T04:46:00.665397 2021-02-10T04:46:01.881 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2021-02-10T04:46:01.881 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[35961]: 1911 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:01.881 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:46:01.882 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[35961]: 02-10T04:46:00.665404 2021-02-10T04:46:01.882 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[35961]: +0000 osd.4 (osd 2021-02-10T04:46:01.882 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[35961]: .4) 1912 2021-02-10T04:46:01.882 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:01.882 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:46:01.882 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[35961]: 02-10T04:46:00.665410+0000 2021-02-10T04:46:01.882 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[35961]: osd.4 (osd.4) 1913 : cluster 2021-02-10T04:46:01.882 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:01.882 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[35961]: audit 2021-02-10T04:46: 2021-02-10T04:46:01.882 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:01 smithi085 conmon[35961]: 01.512394+0000 mon.a (mon.0) 2096 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:46:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:01 smithi164 conmon[29288]: cluster 2021-02-10T04:46:00.311750+0000 mgr.y 2021-02-10T04:46:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:01 smithi164 conmon[29288]: (mgr.14140) 1002 : cluster [DBG] pgmap v1770: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s 2021-02-10T04:46:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:01 smithi164 conmon[29288]: cluster 2021-02-10T04:46:00.665385+0000 osd.4 (osd.4) 1910 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:01 smithi164 conmon[29288]: cluster 2021-02-10T04:46:00.665397+0000 osd.4 (osd.4) 1911 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:01 smithi164 conmon[29288]: cluster 2021-02-10T04:46:00.665404+0000 osd.4 (osd.4) 2021-02-10T04:46:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:01 smithi164 conmon[29288]: 1912 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:01 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:46:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:01 smithi164 conmon[29288]: T04:46:00.665410+0000 osd.4 (osd.4) 1913 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:01 smithi164 conmon[29288]: audit 2021-02-10T04:46:01.512394+0000 mon.a ( 2021-02-10T04:46:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:01 smithi164 conmon[29288]: mon.0) 2096 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:46:01.895 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:01 smithi164 conmon[33677]: debug 2021-02-10T04:46:01.712+0000 7fbec19a1700 -1 osd.4 1049 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:46:01.984 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:46:02.019 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:46:02.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:46:02 smithi164 conmon[40892]: debug 2021-02-10T04:46:02.110+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:46:05.985 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:02 smithi085 conmon[35961]: cluster 2021-02-10T04:46:01.713249+0000 2021-02-10T04:46:05.985 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:02 smithi085 conmon[35961]: osd.4 (osd.4) 1914 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:05.985 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:02 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:46:05.985 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:02 smithi085 conmon[35961]: 04:46:01.713257+0000 osd.4 (osd.4) 1915 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:05.986 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:02 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:46:05.986 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:02 smithi085 conmon[35961]: 02-10T04:46:01.713263+0000 osd.4 2021-02-10T04:46:05.986 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:02 smithi085 conmon[35961]: (osd.4) 1916 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:05.986 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:02 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:46:05.986 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:02 smithi085 conmon[35961]: 10T04:46:01.713266+0000 osd.4 (osd 2021-02-10T04:46:05.986 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:02 smithi085 conmon[35961]: .4) 1917 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:05.986 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[35961]: cluster 2021-02-10T04:46:02.312276+0000 2021-02-10T04:46:05.986 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[35961]: mgr.y (mgr.14140) 1003 : cluster [DBG] pgmap v1771: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2021-02-10T04:46:05.986 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:46:05.986 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[35961]: 10T04:46:02.738809+0000 osd.4 (osd.4) 1918 : cluster 2021-02-10T04:46:05.986 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:05.986 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:46:05.986 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[35961]: 46:02.738815+0000 osd.4 (osd.4) 2021-02-10T04:46:05.987 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[35961]: 1919 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:05.987 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:46:05.987 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[35961]: T04:46:02. 2021-02-10T04:46:05.987 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[35961]: 738820+0000 osd.4 (osd.4) 1920 : cluster 2021-02-10T04:46:05.987 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:05.987 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:46:05.987 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[35961]: 10T04:46:02.738824+0000 osd.4 (osd.4 2021-02-10T04:46:05.987 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[35961]: ) 1921 2021-02-10T04:46:05.987 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:05.987 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[35961]: cluster 2021-02-10T04:46:05.987 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:46:05.987 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[35961]: 46:03.787327+0000 osd.4 (osd.4) 1922 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:05.987 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[35961]: cluster 2021-02-10T04:46:05.988 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[35961]: 2021-02-10T04:46:03 2021-02-10T04:46:05.988 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[35961]: .787338+0000 osd.4 (osd.4) 1923 : 2021-02-10T04:46:05.988 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:05.988 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:46:05.988 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[35961]: -02-10T04:46:03.787346+0000 2021-02-10T04:46:05.988 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[35961]: osd.4 (osd.4) 1924 : cluster [WRN] 2021-02-10T04:46:05.988 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:05.988 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:46:05.988 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[35961]: 04:46:03.787355+0000 osd.4 (osd 2021-02-10T04:46:05.988 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[35961]: .4) 1925 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:05.988 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:05 smithi085 conmon[35961]: cluster 2021-02-10T04:46:05.988 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:05 smithi085 conmon[35961]: 2021-02-10T04:46:05.988 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:05 smithi085 conmon[35961]: 2021-02-10T04:46:04 2021-02-10T04:46:05.989 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:05 smithi085 conmon[35961]: .312987+0000 mgr.y (mgr.14140) 1004 : cluster [DBG] pgmap v1772: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2021-02-10T04:46:05.989 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:05 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:46:05.989 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:05 smithi085 conmon[35961]: -02-10T04:46:04.747772 2021-02-10T04:46:05.989 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:05 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1926 : cluster [WRN] 2021-02-10T04:46:05.989 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:05 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:05.989 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:05 smithi085 conmon[35961]: cluster 2021-02-10T04:46:04 2021-02-10T04:46:05.989 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:05 smithi085 conmon[35961]: .747783+0000 osd.4 (osd.4) 1927 : 2021-02-10T04:46:05.989 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:05 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:05.989 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:05 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:46:05.989 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:05 smithi085 conmon[35961]: 10T04:46:04.747790+0000 osd.4 (osd 2021-02-10T04:46:05.989 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:05 smithi085 conmon[35961]: .4) 1928 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:05.989 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:05 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:46:05.989 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:05 smithi085 conmon[35961]: -02-10T04:46:04.747797+0000 2021-02-10T04:46:05.990 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:05 smithi085 conmon[35961]: osd.4 (osd.4) 1929 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:05.990 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:02 smithi085 conmon[29319]: cluster 2021-02-10T04:46 2021-02-10T04:46:05.990 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:02 smithi085 conmon[29319]: :01.713249+0000 osd.4 (osd.4) 1914 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:05.990 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:02 smithi085 conmon[29319]: cluster 2021-02-10T04:46:05.990 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:02 smithi085 conmon[29319]: 2021-02-10T04:46:01.713257+0000 osd.4 (osd.4) 1915 : cluster [WRN] 2021-02-10T04:46:05.990 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:02 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:05.990 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:02 smithi085 conmon[29319]: cluster 2021-02-10T04:46:01 2021-02-10T04:46:05.990 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:02 smithi085 conmon[29319]: .713263+0000 osd.4 (osd.4) 1916 : cluster [WRN] 2021-02-10T04:46:05.990 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:02 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:05.990 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:02 smithi085 conmon[29319]: cluster 2021-02-10T04:46:05.990 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:02 smithi085 conmon[29319]: 2021-02-10T04:46:01.713266+0000 osd.4 (osd.4) 2021-02-10T04:46:05.990 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:02 smithi085 conmon[29319]: 1917 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:05.991 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[29319]: cluster 2021-02-10T04:46:05.991 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[29319]: 2021-02 2021-02-10T04:46:05.991 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[29319]: -10T04:46:02.312276+0000 mgr.y (mgr.14140) 1003 : cluster [DBG] pgmap v1771: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2021-02-10T04:46:05.991 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:46:05.991 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[29319]: 02-10T04:46:02. 2021-02-10T04:46:05.991 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[29319]: 738809+0000 osd.4 (osd.4) 2021-02-10T04:46:05.991 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[29319]: 1918 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:05.991 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:46:05.991 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[29319]: 02-10T04:46:02.738815 2021-02-10T04:46:05.991 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1919 : cluster [WRN] 2021-02-10T04:46:05.991 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:05.991 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[29319]: cluster 2021-02-10T04:46:05.991 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[29319]: 2021-02-10T04:46:02 2021-02-10T04:46:05.992 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[29319]: .738820+0000 osd.4 2021-02-10T04:46:05.992 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[29319]: (osd.4) 1920 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:05.992 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:46:05.992 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[29319]: -10T04:46:02.738824+0000 osd.4 (osd. 2021-02-10T04:46:05.992 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:03 smithi085 conmon[29319]: 4) 1921 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:05.992 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[29319]: cluster 2021-02-10T04:46:05.992 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[29319]: 2021-02-10T04:46:05.992 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[29319]: 2021 2021-02-10T04:46:05.992 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[29319]: - 2021-02-10T04:46:05.992 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[29319]: 02 2021-02-10T04:46:05.992 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[29319]: - 2021-02-10T04:46:05.992 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[29319]: 10T04:46:03.787327+0000 osd.4 2021-02-10T04:46:05.992 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[29319]: (osd.4) 1922 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:05.992 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:46:05.993 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[29319]: 04:46:03.787338+0000 osd.4 2021-02-10T04:46:05.993 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[29319]: (osd.4) 1923 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:05.993 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:46:05.993 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[29319]: -10T04:46:03.787346+0000 2021-02-10T04:46:05.993 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[29319]: osd.4 (osd.4) 1924 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:05.993 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:46:05.993 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[29319]: 02-10T04:46:03.787355+0000 osd.4 (osd. 2021-02-10T04:46:05.993 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[29319]: 4) 1925 : cluster 2021-02-10T04:46:05.993 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:04 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:05.993 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:05 smithi085 conmon[29319]: cluster 2021-02-10T04:46:04.312987+0000 mgr.y (mgr.14140) 1004 : cluster [DBG] pgmap v1772: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2021-02-10T04:46:05.993 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:05 smithi085 conmon[29319]: cluster 2021-02-10T04:46:04.747772+0000 osd.4 (osd.4) 1926 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:05.993 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:05 smithi085 conmon[29319]: cluster 2021-02-10T04:46:04.747783+0000 osd.4 (osd.4) 1927 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:05.993 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:05 smithi085 conmon[29319]: cluster 2021-02-10T04:46:04.747790+0000 osd.4 (osd.4) 1928 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:05.994 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:05 smithi085 conmon[29319]: cluster 2021-02-10T04:46:04.747797+0000 osd.4 (osd.4) 1929 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:05.994 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:46:05.994 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:02 smithi164 conmon[33677]: debug 2021-02-10T04:46:02.738+0000 7fbec19a1700 -1 osd.4 1049 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:46:05.994 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:03 smithi164 conmon[33677]: debug 2021-02-10T04:46:03.786+0000 7fbec19a1700 -1 osd.4 1049 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:46:05.994 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:04 smithi164 conmon[33677]: debug 2021-02-10T04:46:05.994 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:04 smithi164 conmon[33677]: 2021-02-10T04:46:04.747+0000 7fbec19a1700 -1 osd.4 1049 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:46:05.994 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:05 smithi164 conmon[33677]: debug 2021-02-10T04:46:05.722+0000 7fbec19a1700 -1 osd.4 1049 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:46:05.994 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:02 smithi164 conmon[29288]: cluster 2021-02-10T04:46:01.713249+0000 2021-02-10T04:46:05.995 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:02 smithi164 conmon[29288]: osd.4 (osd.4) 1914 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:05.995 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:02 smithi164 conmon[29288]: cluster 2021-02-10T04:46:01.713257+0000 osd.4 (osd.4) 1915 : cluster [WRN] 2021-02-10T04:46:05.995 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:02 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:05.995 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:02 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:46:05.995 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:02 smithi164 conmon[29288]: -10T04:46:01.713263+0000 osd.4 (osd.4) 1916 : cluster 2021-02-10T04:46:05.995 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:02 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:05.995 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:02 smithi164 conmon[29288]: cluster 2021-02-10T04:46:01.713266+0000 osd.4 (osd.4) 1917 : cluster [WRN] 2021-02-10T04:46:05.995 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:02 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:05.995 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:03 smithi164 conmon[29288]: cluster 2021-02-10T04:46:02.312276+0000 2021-02-10T04:46:05.995 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:03 smithi164 conmon[29288]: mgr.y (mgr.14140) 1003 : cluster [DBG] pgmap v1771: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2021-02-10T04:46:05.995 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:03 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:46:05.995 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:03 smithi164 conmon[29288]: -10T04:46:02.738809+0000 2021-02-10T04:46:05.995 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:03 smithi164 conmon[29288]: osd.4 (osd.4) 1918 : cluster 2021-02-10T04:46:05.996 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:03 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:05.996 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:03 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:46:05.996 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:03 smithi164 conmon[29288]: 46:02.738815+0000 osd.4 (osd.4) 1919 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:05.996 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:03 smithi164 conmon[29288]: cluster 2021-02-10T04:46:02. 2021-02-10T04:46:05.996 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:03 smithi164 conmon[29288]: 738820+0000 osd.4 (osd.4) 1920 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:05.996 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:03 smithi164 conmon[29288]: cluster 2021-02-10T04:46:02.738824 2021-02-10T04:46:05.996 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:03 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1921 : cluster [WRN] 2021-02-10T04:46:05.996 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:03 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:05.996 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:04 smithi164 conmon[29288]: cluster 2021-02-10T04:46: 2021-02-10T04:46:05.996 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:04 smithi164 conmon[29288]: 03.787327+0000 osd.4 (osd.4) 1922 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:05.996 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:04 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:46:05.996 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:04 smithi164 conmon[29288]: 04:46:03.787338+0000 osd.4 (osd.4) 1923 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:05.996 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:04 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:46:05.997 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:04 smithi164 conmon[29288]: 46:03.787346+0000 osd.4 (osd.4) 1924 : cluster [WRN] 2021-02-10T04:46:05.997 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:04 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:05.997 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:04 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:46:05.997 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:04 smithi164 conmon[29288]: 04:46:03.787355+0000 osd.4 (osd.4) 1925 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:05.997 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:05 smithi164 conmon[29288]: cluster 2021-02-10T04:46:04.312987+0000 mgr.y (mgr.14140) 1004 : cluster 2021-02-10T04:46:05.997 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:05 smithi164 conmon[29288]: [DBG] pgmap v1772: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2021-02-10T04:46:05.997 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:05 smithi164 conmon[29288]: cluster 2021-02-10T04:46:04.747772+0000 osd.4 (osd.4) 1926 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:05.997 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:05 smithi164 conmon[29288]: cluster 2021-02-10T04:46:04.747783+0000 osd.4 (osd.4) 1927 : cluster 2021-02-10T04:46:05.997 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:05 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:05.997 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:05 smithi164 conmon[29288]: cluster 2021-02-10T04:46:04.747790+0000 2021-02-10T04:46:05.997 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:05 smithi164 conmon[29288]: osd.4 (osd.4) 1928 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:05.997 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:05 smithi164 conmon[29288]: cluster 2021-02-10T04:46:05.997 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:05 smithi164 conmon[29288]: 2021-02-10T04:46:04. 2021-02-10T04:46:05.998 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:05 smithi164 conmon[29288]: 747797+0000 osd.4 (osd.4) 1929 : cluster 2021-02-10T04:46:05.998 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:05 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:06.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:06 smithi164 conmon[33677]: debug 2021-02-10T04:46:06.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:06 smithi164 conmon[33677]: 2021-02-10T04:46:06.746+0000 7fbec19a1700 -1 osd.4 1049 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:46:06.895 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:06 smithi164 conmon[33677]: 2021-02-10T04:46:06.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:06 smithi164 conmon[29288]: cluster 2021-02-10T04:46:06.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:06 smithi164 conmon[29288]: 2021-02 2021-02-10T04:46:06.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:06 smithi164 conmon[29288]: -10T04:46:05.723477+0000 osd.4 (osd.4) 1930 : cluster [WRN] 2021-02-10T04:46:06.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:06 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:06.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:06 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:46:06.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:06 smithi164 conmon[29288]: -10T04:46:05.723485+0000 osd.4 ( 2021-02-10T04:46:06.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:06 smithi164 conmon[29288]: osd.4) 1931 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:06.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:06 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:46:06.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:06 smithi164 conmon[29288]: -02-10T04:46:05.723492+0000 2021-02-10T04:46:06.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:06 smithi164 conmon[29288]: osd.4 (osd.4) 1932 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:06.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:06 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:46:06.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:06 smithi164 conmon[29288]: 04:46:05.723498+0000 osd.4 (osd.4 2021-02-10T04:46:06.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:06 smithi164 conmon[29288]: ) 1933 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:06.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:06 smithi085 conmon[29319]: cluster 2021-02-10T04:46:05.723477+0000 osd.4 (osd.4) 1930 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:06.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:06 smithi085 conmon[29319]: cluster 2021-02-10T04:46:05.723485+0000 osd.4 (osd.4) 1931 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:06.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:06 smithi085 conmon[29319]: cluster 2021-02-10T04:46:05.723492+0000 osd.4 (osd.4) 1932 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:06.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:06 smithi085 conmon[29319]: cluster 2021-02-10T04:46:06.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:06 smithi085 conmon[29319]: 2021-02-10T04:46:05.723498+0000 osd.4 (osd.4) 1933 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:06.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:06 smithi085 conmon[35961]: cluster 2021-02-10T04:46:05.723477+0000 2021-02-10T04:46:06.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:06 smithi085 conmon[35961]: osd.4 (osd.4) 1930 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:06.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:06 smithi085 conmon[35961]: cluster 2021-02-10T04:46:05.723485+0000 osd.4 (osd.4) 1931 : cluster [WRN] 2021-02-10T04:46:06.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:06 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:06.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:06 smithi085 conmon[35961]: cluster 2021-02-10T04:46:05.723492+0000 osd.4 (osd.4) 2021-02-10T04:46:06.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:06 smithi085 conmon[35961]: 1932 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:06.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:06 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:46:06.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:06 smithi085 conmon[35961]: T04:46:05.723498+0000 osd.4 (osd 2021-02-10T04:46:06.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:06 smithi085 conmon[35961]: .4) 1933 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:07.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:07 smithi164 conmon[33677]: debug 2021-02-10T04:46:07.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:07 smithi164 conmon[33677]: 2021-02-10T04:46:07.771+0000 7fbec19a1700 -1 osd.4 1050 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:46:07.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:07 smithi164 conmon[33677]: 2021-02-10T04:46:07.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:07 smithi164 conmon[29288]: cluster 2021-02-10T04:46:06.313687+0000 mgr.y 2021-02-10T04:46:07.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:07 smithi164 conmon[29288]: (mgr.14140) 1005 : cluster [DBG] pgmap v1773: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2021-02-10T04:46:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:07 smithi164 conmon[29288]: cluster 2021-02-10T04:46:06.747154+0000 osd.4 (osd.4) 1934 : 2021-02-10T04:46:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:07 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:07 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:46:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:07 smithi164 conmon[29288]: 46:06.747163+0000 osd.4 ( 2021-02-10T04:46:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:07 smithi164 conmon[29288]: osd.4) 1935 : cluster [WRN] 2021-02-10T04:46:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:07 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:07 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:46:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:07 smithi164 conmon[29288]: -02-10T04:46:06.747170+0000 2021-02-10T04:46:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:07 smithi164 conmon[29288]: osd.4 (osd.4) 1936 : cluster [WRN] 2021-02-10T04:46:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:07 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:07 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:46:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:07 smithi164 conmon[29288]: -02-10T04:46 2021-02-10T04:46:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:07 smithi164 conmon[29288]: :06.747175+0000 osd.4 ( 2021-02-10T04:46:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:07 smithi164 conmon[29288]: osd.4) 1937 : cluster [WRN] 2021-02-10T04:46:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:07 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:07.915 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:46:07.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[29319]: cluster 2021-02-10T04:46:06.313687+0000 mgr.y (mgr.14140) 1005 : 2021-02-10T04:46:07.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[29319]: cluster [DBG] pgmap v1773: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2021-02-10T04:46:07.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[29319]: cluster 2021-02-10T04:46:06.747154+0000 osd.4 (osd.4) 1934 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:07.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[29319]: cluster 2021-02-10T04:46:06.747163+0000 osd.4 (osd.4) 1935 : cluster 2021-02-10T04:46:07.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:07.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[29319]: cluster 2021-02-10T04:46 2021-02-10T04:46:07.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[29319]: :06.747170+0000 osd.4 (osd.4) 1936 2021-02-10T04:46:07.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:07.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[29319]: cluster 2021-02-10T04:46 2021-02-10T04:46:07.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[29319]: :06.747175+0000 osd.4 (osd.4) 1937 2021-02-10T04:46:07.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[35961]: cluster 2021-02-10T04:46:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:46:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[35961]: 04:46:06.313687+0000 mgr.y (mgr.14140) 1005 : cluster [DBG] pgmap v1773: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2021-02-10T04:46:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[35961]: 2021-02-10T04:46:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:46:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[35961]: -02-10T04:46:06.747154+0000 osd.4 (osd.4) 1934 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[35961]: 2021-02-10T04:46:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[35961]: cluster 2021-02-10T04:46:07.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:46:07.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[35961]: 10T04:46:06.747163+0000 2021-02-10T04:46:07.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[35961]: osd.4 2021-02-10T04:46:07.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[35961]: (osd.4 2021-02-10T04:46:07.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[35961]: ) 1935 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:07.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[35961]: cluster 2021-02-10T04:46:07.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[35961]: 2021-02-10T04:46: 2021-02-10T04:46:07.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[35961]: 06.747170+0000 osd.4 (osd. 2021-02-10T04:46:07.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[35961]: 4) 1936 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:07.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:46:07.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[35961]: -02-10T04:46:06.747175+0000 osd.4 2021-02-10T04:46:07.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[35961]: (osd.4) 1937 : cluster [WRN] 2021-02-10T04:46:07.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:07 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:07.983 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:46:08.016 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:46:08.042 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:46:08.071 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:46:08.108 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:46:08.148 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:46:08.163 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:46:08.179 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:46:08.215 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:46:08.223 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:46:08.223 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:46:08.223 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:46:08.223 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:46:08.223 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:46:08.223 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:46:08.223 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:46:08.223 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:46:08.642 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[29319]: cluster 2021-02-10T04:46:08.643 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: cluster 2021-02-10T04:46:08.653 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is unset 2021-02-10T04:46:08.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:46:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[29319]: 10T04:46:07.414078+0000 osd.5 (osd.5) 100 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:46:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[29319]: cluster 2021-02-10T04:46:07.417180+0000 osd.5 2021-02-10T04:46:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[29319]: (osd.5) 101 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:46:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:46:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[29319]: 04:46:07.636990+0000 mon.a (mon.0) 2097 : cluster 2021-02-10T04:46:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[29319]: [DBG] osdmap e1050: 8 total, 8 up, 7 in 2021-02-10T04:46:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:46:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[29319]: -02-10T04:46:07.771906+0000 osd.4 (osd.4) 1938 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[29319]: 2021-02-10T04:46:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[29319]: cluster 2021-02-10T04:46:07. 2021-02-10T04:46:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[29319]: 771917+0000 osd.4 (osd.4) 1939 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[29319]: cluster 2021-02-10T04:46:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:46:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[29319]: 46:07.771925+0000 osd.4 (osd.4) 1940 2021-02-10T04:46:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:46:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[29319]: 10T04:46:07.771931+0000 osd.4 (osd.4) 1941 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:46:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[29319]: 02-10T04:46:07.950043+0000 mon.c (mon.1) 74 : audit [DBG] from='client.? 172.21.15.85:0/620635148' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:46:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[29319]: audit 2021-02-10T04:46:08.267239+0000 mon.c ( 2021-02-10T04:46:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[29319]: mon.1) 75 : audit [INF] from='client.? 172.21.15.85:0/2671763909' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:46:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[29319]: audit 2021-02-10T04:46:08. 2021-02-10T04:46:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[29319]: 267860+0000 mon.a (mon.0) 2098 : audit [INF] 2021-02-10T04:46:08.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[29319]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:46:08.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[29319]: cluster 2021-02-10T04:46:08.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[29319]: 2021-02-10T04:46:08.382628+0000 2021-02-10T04:46:08.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[29319]: mon.a (mon.0) 2099 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 97 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:46:08.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: 2021-02-10T04:46:07 2021-02-10T04:46:08.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: .414078+0000 osd.5 (osd.5) 100 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:46:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:46:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: -02-10T04:46:07.417180+0000 2021-02-10T04:46:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: osd.5 (osd.5) 101 : cluster [DBG] 2021-02-10T04:46:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:46:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:46:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: 46:07.636990+0000 mon.a (mon.0 2021-02-10T04:46:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: ) 2097 : cluster [DBG] osdmap e1050: 8 total, 8 up, 7 in 2021-02-10T04:46:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:46:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: -10T04:46:07.771906+0000 osd.4 ( 2021-02-10T04:46:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: osd.4) 1938 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:46:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: -10T04:46:07.771917+0000 osd.4 2021-02-10T04:46:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: (osd.4) 1939 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: 2021-02-10T04:46:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: cluster 2021-02-10T04:46:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: 2021-02-10T04:46:07.771925+0000 osd.4 ( 2021-02-10T04:46:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: osd.4) 1940 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:46:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: 02-10T04:46:07.771931+0000 2021-02-10T04:46:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: osd.4 (osd.4) 1941 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: 2021-02-10T04:46:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: audit 2021-02-10T04:46:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: 2021-02-10T04:46:07.950043+0000 mon.c 2021-02-10T04:46:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: (mon.1) 74 : audit [DBG] from='client.? 172.21.15.85:0/620635148' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:46:08.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: 2021-02-10T04:46:08.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: audit 2021-02-10T04:46:08.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: 2021-02-10T04:46:08.267239+0000 mon.c ( 2021-02-10T04:46:08.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: mon.1) 75 : audit [INF] from='client.? 172.21.15.85:0/2671763909' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:46:08.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: audit 2021 2021-02-10T04:46:08.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: -02-10T04:46:08.267860+0000 2021-02-10T04:46:08.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: mon.a (mon.0) 2098 : audit [INF] 2021-02-10T04:46:08.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:46:08.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: cluster 2021-02-10T04:46:08 2021-02-10T04:46:08.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: .382628+0000 mon.a (mon.0) 2099 : 2021-02-10T04:46:08.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:08 smithi085 conmon[35961]: cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 97 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:46:09.144 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:46:09.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: cluster 2021-02-10T04:46:09.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: 2021-02-10T04:46:07.414078 2021-02-10T04:46:09.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: +0000 osd.5 (osd.5) 100 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:46:09.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:46:09.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: -02-10T04:46:07.417180+0000 osd.5 2021-02-10T04:46:09.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: (osd.5) 101 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:46:09.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: cluster 2021-02-10T04:46:09.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:46:09.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: :46:07.636990+0000 mon.a (mon.0) 2097 : cluster [DBG] osdmap e1050: 8 total, 8 up, 7 in 2021-02-10T04:46:09.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: cluster 2021-02-10T04:46:09.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:46:09.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: 04:46:07.771906+0000 osd.4 (osd. 2021-02-10T04:46:09.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: 4) 1938 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:09.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: 2021-02-10T04:46:09.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: cluster 2021-02-10T04:46:09.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: 2021- 2021-02-10T04:46:09.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: 02-10T04:46:07.771917+0000 2021-02-10T04:46:09.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: osd.4 (osd.4) 1939 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:09.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:46:09.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: -02-10T04:46:07.771925+0000 osd.4 2021-02-10T04:46:09.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: (osd.4) 1940 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:09.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:46:09.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: T04:46:07.771931+0000 osd.4 (osd.4) 1941 2021-02-10T04:46:09.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:09.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: audit 2021-02-10T04:46:07 2021-02-10T04:46:09.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: .950043+0000 mon.c (mon.1) 74 : audit [DBG] 2021-02-10T04:46:09.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/620635148' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:46:09.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: audit 2021-02-10T04:46:08.267239+0000 mon.c (mon 2021-02-10T04:46:09.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: .1) 75 : audit [INF] from='client.? 172.21.15.85:0/2671763909' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:46:09.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: audit 2021-02-10T04:46 2021-02-10T04:46:09.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: :08.267860+0000 mon.a (mon.0) 2098 : audit 2021-02-10T04:46:09.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:46:09.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: cluster 2021-02-10T04:46:08 2021-02-10T04:46:09.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: .382628+0000 mon.a (mon.0) 2099 : cluster 2021-02-10T04:46:09.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:08 smithi164 conmon[29288]: [WRN] Health check update: 4 slow ops, oldest one blocked for 97 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:46:09.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[29319]: cluster 2021-02-10T04:46:09.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:46:09.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[29319]: :46:08.314311+0000 mgr.y (mgr.14140) 1006 : cluster [DBG] pgmap v1775: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 28 KiB/s rd, 0 B/s wr, 45 op/s 2021-02-10T04:46:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:46:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[29319]: 46:08.633981+0000 mon.a (mon.0) 2100 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:46:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:46:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[29319]: -10T04:46:08.647238+0000 mon.a (mon.0) 2101 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:46:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[29319]: 2021-02-10T04:46:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[29319]: cluster 2021-02-10T04:46:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:46:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[29319]: 04:46:08.647307+0000 mon.a (mon. 2021-02-10T04:46:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[29319]: 0) 2102 : cluster [DBG] 2021-02-10T04:46:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[29319]: osdmap e1051: 8 total, 8 up, 7 in 2021-02-10T04:46:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[29319]: audit 2021-02-10T04:46:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:46:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[29319]: T04:46:08.743159+0000 2021-02-10T04:46:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[29319]: mgr.y (mgr. 2021-02-10T04:46:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[29319]: 14140) 1007 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:46:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[29319]: cluster 2021-02-10T04:46:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[29319]: 2021-02-10T04:46: 2021-02-10T04:46:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[29319]: 08.770070+0000 osd.0 (osd.0) 111 : cluster [DBG] 5.8 scrub ok 2021-02-10T04:46:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:46:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[29319]: -10T04:46: 2021-02-10T04:46:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[29319]: 08.846209+0000 osd.4 (osd 2021-02-10T04:46:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[29319]: .4) 1942 : cluster [DBG] 2021-02-10T04:46:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[29319]: 4.18 scrub ok 2021-02-10T04:46:09.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[35961]: cluster 2021-02-10T04:46:09.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:46:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[35961]: 46:08.314311+0000 mgr.y (mgr.14140) 1006 : cluster [DBG] pgmap v1775: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 28 KiB/s rd, 0 B/s wr, 45 op/s 2021-02-10T04:46:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:46:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[35961]: 02-10T04:46:08.633981+0000 2021-02-10T04:46:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[35961]: mon.a (mon.0) 2100 : cluster [WRN] 2021-02-10T04:46:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[35961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:46:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:46:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[35961]: 46:08.647238+0000 mon.a (mon.0) 2101 : audit [INF] 2021-02-10T04:46:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[35961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:46:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[35961]: cluster 2021-02-10T04:46:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[35961]: 2021-02-10T04:46:08.647307+0000 mon.a (mon.0 2021-02-10T04:46:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[35961]: ) 2102 : cluster [DBG] osdmap e1051: 8 total, 8 up, 7 in 2021-02-10T04:46:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:46:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[35961]: 04:46:08.743159+0000 mgr.y (mgr. 2021-02-10T04:46:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[35961]: 14140) 1007 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:46:09.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:46:09.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[35961]: 04:46:08.770070+0000 osd.0 (osd.0 2021-02-10T04:46:09.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[35961]: ) 111 : cluster [DBG] 5.8 scrub ok 2021-02-10T04:46:09.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[35961]: cluster 2021-02-10T04:46:08 2021-02-10T04:46:09.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[35961]: .846209+0000 osd.4 (osd.4) 1942 : cluster [DBG] 2021-02-10T04:46:09.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:09 smithi085 conmon[35961]: 4.18 scrub ok 2021-02-10T04:46:10.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:09 smithi164 conmon[29288]: cluster 2021-02-10T04:46:10.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:09 smithi164 conmon[29288]: 2021-02-10T04:46:10.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:09 smithi164 conmon[29288]: 2021 2021-02-10T04:46:10.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:09 smithi164 conmon[29288]: - 2021-02-10T04:46:10.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:09 smithi164 conmon[29288]: 02 2021-02-10T04:46:10.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:09 smithi164 conmon[29288]: - 2021-02-10T04:46:10.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:09 smithi164 conmon[29288]: 10 2021-02-10T04:46:10.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:09 smithi164 conmon[29288]: T 2021-02-10T04:46:10.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:09 smithi164 conmon[29288]: 04 2021-02-10T04:46:10.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:09 smithi164 conmon[29288]: : 2021-02-10T04:46:10.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:09 smithi164 conmon[29288]: 46 2021-02-10T04:46:10.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:09 smithi164 conmon[29288]: : 2021-02-10T04:46:10.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:09 smithi164 conmon[29288]: 08.314311+0000 mgr.y (mgr.14140) 2021-02-10T04:46:10.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:09 smithi164 conmon[29288]: 1006 : cluster [DBG] pgmap v1775: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 28 KiB/s rd, 0 B/s wr, 45 op/s 2021-02-10T04:46:10.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:09 smithi164 conmon[29288]: cluster 2021-02-10T04:46 2021-02-10T04:46:10.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:09 smithi164 conmon[29288]: :08.633981+0000 mon.a (mon.0) 2100 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:46:10.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:09 smithi164 conmon[29288]: audit 2021-02-10T04:46:08 2021-02-10T04:46:10.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:09 smithi164 conmon[29288]: .647238+0000 mon.a (mon.0) 2101 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:46:10.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:09 smithi164 conmon[29288]: cluster 2021-02-10T04:46:08.647307 2021-02-10T04:46:10.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:09 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2102 : cluster [DBG] osdmap e1051: 8 total, 8 up, 7 in 2021-02-10T04:46:10.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:09 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:46:10.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:09 smithi164 conmon[29288]: 10T04:46:08.743159+0000 mgr.y 2021-02-10T04:46:10.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:09 smithi164 conmon[29288]: (mgr.14140) 1007 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:46:10.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:09 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:46:10.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:09 smithi164 conmon[29288]: -02-10T04:46:08.770070+0000 2021-02-10T04:46:10.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:09 smithi164 conmon[29288]: osd.0 (osd.0) 111 : cluster [DBG] 5.8 scrub ok 2021-02-10T04:46:10.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:09 smithi164 conmon[29288]: 2021-02-10T04:46:10.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:09 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:46:10.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:09 smithi164 conmon[29288]: -02-10T04:46:08.846209+0000 osd.4 (osd. 2021-02-10T04:46:10.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:09 smithi164 conmon[29288]: 4) 1942 : cluster [DBG] 4.18 scrub ok 2021-02-10T04:46:10.390 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:46:10.500 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:46:10.541 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:46:10.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:10 smithi164 conmon[33677]: debug 2021-02-10T04:46:10.644 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:10 smithi164 conmon[33677]: 2021-02-10T04:46:10.195+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:46:10.918 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:10 smithi164 conmon[29288]: cluster 2021-02-10T04:46:09.107001+0000 osd.5 (osd.5) 102 2021-02-10T04:46:10.918 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:10 smithi164 conmon[29288]: : cluster [DBG] 5.15 scrub ok 2021-02-10T04:46:10.918 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:10 smithi164 conmon[29288]: cluster 2021-02-10T04:46:09.627899+0000 osd.3 (osd.3) 104 : cluster [DBG] 6.11 scrub ok 2021-02-10T04:46:10.918 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:10 smithi164 conmon[29288]: cluster 2021-02-10T04:46:09. 2021-02-10T04:46:10.918 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:10 smithi164 conmon[29288]: 751635+0000 osd.0 (osd.0) 112 : cluster [DBG] 2.5 scrub ok 2021-02-10T04:46:10.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:10 smithi085 conmon[29319]: cluster 2021-02-10T04:46:09.107001+0000 2021-02-10T04:46:10.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:10 smithi085 conmon[29319]: osd.5 (osd.5) 102 : cluster [DBG] 5.15 scrub ok 2021-02-10T04:46:10.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:10 smithi085 conmon[29319]: cluster 2021-02-10T04:46:09.627899+0000 osd.3 (osd.3) 104 : cluster [DBG] 6.11 scrub ok 2021-02-10T04:46:10.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:10 smithi085 conmon[29319]: cluster 2021-02-10T04:46:09.751635+0000 osd.0 (osd.0) 112 : cluster [DBG] 2.5 scrub ok 2021-02-10T04:46:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:10 smithi085 conmon[35961]: cluster 2021-02-10T04:46:09.107001 2021-02-10T04:46:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:10 smithi085 conmon[35961]: +0000 osd.5 (osd.5) 102 : cluster [DBG] 5.15 scrub ok 2021-02-10T04:46:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:10 smithi085 conmon[35961]: cluster 2021-02-10T04:46:09.627899+0000 osd.3 (osd.3) 104 : cluster [DBG] 6.11 scrub ok 2021-02-10T04:46:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:10 smithi085 conmon[35961]: cluster 2021-02-10T04:46:09.751635+0000 osd.0 2021-02-10T04:46:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:10 smithi085 conmon[35961]: (osd.0) 112 : cluster [DBG] 2.5 scrub ok 2021-02-10T04:46:11.523 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:46:11.524 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:46:11.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:11 smithi164 conmon[29288]: cluster 2021-02-10T04:46:10.064465+0000 osd.6 (osd.6) 104 : cluster [DBG] 2.1d scrub ok 2021-02-10T04:46:11.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:11 smithi164 conmon[29288]: cluster 2021-02-10T04:46:10.315057+0000 mgr.y (mgr.14140) 1008 : cluster [DBG] pgmap v1777: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:46:11.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:11 smithi085 conmon[29319]: cluster 2021-02-10T04:46:10.064465+0000 2021-02-10T04:46:11.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:11 smithi085 conmon[29319]: osd.6 (osd.6) 104 : cluster [DBG] 2.1d scrub ok 2021-02-10T04:46:11.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:11 smithi085 conmon[29319]: cluster 2021-02-10T04:46:10.315057+0000 mgr.y (mgr.14140) 1008 : cluster 2021-02-10T04:46:11.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:11 smithi085 conmon[29319]: [DBG] pgmap v1777: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:46:11.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:11 smithi085 conmon[35961]: cluster 2021-02-10T04:46 2021-02-10T04:46:11.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:11 smithi085 conmon[35961]: :10.064465+0000 osd.6 (osd.6) 104 : cluster [DBG] 2.1d scrub ok 2021-02-10T04:46:11.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:11 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:46:11.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:11 smithi085 conmon[35961]: 02-10T04:46:10.315057 2021-02-10T04:46:11.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:11 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1008 : 2021-02-10T04:46:11.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:11 smithi085 conmon[35961]: cluster [DBG] pgmap v1777: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:46:12.413 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:46:12.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:46:12 smithi085 conmon[49106]: debug 2021-02-10T04:46:12.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:46:12 smithi085 conmon[49106]: 2021-02-10T04:46:12.088+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:46:12.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:12 smithi085 conmon[35961]: cluster 2021-02-10T04:46:11.118431 2021-02-10T04:46:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:12 smithi085 conmon[35961]: +0000 osd.5 (osd.5) 103 : cluster [DBG] 4.2 scrub ok 2021-02-10T04:46:12.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:12 smithi085 conmon[29319]: cluster 2021-02-10T04:46:12.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:12 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:46:12.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:12 smithi085 conmon[29319]: 04:46:11.118431+0000 osd.5 (osd.5) 103 : cluster [DBG] 4.2 scrub ok 2021-02-10T04:46:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:12 smithi164 conmon[29288]: cluster 2021-02-10T04:46:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:12 smithi164 conmon[29288]: 2021-02-10T04:46:11 2021-02-10T04:46:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:12 smithi164 conmon[29288]: .118431+0000 osd.5 (osd.5) 103 : cluster [DBG] 4.2 scrub ok 2021-02-10T04:46:13.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:13 smithi085 conmon[29319]: cluster 2021-02-10T04:46:12.315561+0000 mgr.y (mgr.14140) 1009 : 2021-02-10T04:46:13.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:13 smithi085 conmon[29319]: cluster [DBG] pgmap v1778: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:46:13.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:13 smithi085 conmon[29319]: cluster 2021-02-10T04:46:12.555663+0000 osd.3 (osd.3) 105 : cluster [DBG] 2.28 scrub ok 2021-02-10T04:46:13.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:13 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:46:13.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:13 smithi085 conmon[29319]: -02-10T04:46:12.654326+0000 mon.a (mon.0) 2103 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 102 sec, osd.4 has slow ops) 2021-02-10T04:46:13.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:13 smithi085 conmon[35961]: cluster 2021-02-10T04:46:12. 2021-02-10T04:46:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:13 smithi085 conmon[35961]: 315561+0000 mgr.y (mgr.14140) 1009 : cluster [DBG] pgmap v1778: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:46:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:13 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:46:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:13 smithi085 conmon[35961]: :46:12.555663+0000 osd.3 (osd.3) 105 : cluster [DBG] 2.28 scrub ok 2021-02-10T04:46:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:13 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:46:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:13 smithi085 conmon[35961]: 10T04:46:12.654326+0000 mon.a (mon.0) 2103 2021-02-10T04:46:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:13 smithi085 conmon[35961]: : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 102 sec, osd.4 has slow ops) 2021-02-10T04:46:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:13 smithi164 conmon[29288]: cluster 2021-02-10T04:46:12.315561+0000 mgr.y ( 2021-02-10T04:46:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:13 smithi164 conmon[29288]: mgr.14140) 1009 : cluster [DBG] pgmap v1778: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:46:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:13 smithi164 conmon[29288]: cluster 2021-02-10T04:46:12.555663+0000 osd.3 (osd.3) 105 2021-02-10T04:46:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:13 smithi164 conmon[29288]: : cluster [DBG] 2.28 scrub ok 2021-02-10T04:46:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:13 smithi164 conmon[29288]: cluster 2021-02-10T04:46:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:13 smithi164 conmon[29288]: 2021-02-10T04:46:12.654326+0000 mon.a (mon.0 2021-02-10T04:46:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:13 smithi164 conmon[29288]: ) 2103 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 102 sec, osd.4 has slow ops) 2021-02-10T04:46:14.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:14 smithi085 conmon[35961]: cluster 2021-02-10T04:46:13.849748+0000 2021-02-10T04:46:14.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:14 smithi085 conmon[35961]: osd.0 (osd.0) 113 : cluster [DBG] 4.1e scrub ok 2021-02-10T04:46:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:14 smithi085 conmon[29319]: cluster 2021-02-10T04:46:13.849748+0000 osd.0 2021-02-10T04:46:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:14 smithi085 conmon[29319]: (osd.0) 113 : cluster [DBG] 4.1e scrub ok 2021-02-10T04:46:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:14 smithi164 conmon[29288]: cluster 2021-02-10T04:46:13.849748+0000 2021-02-10T04:46:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:14 smithi164 conmon[29288]: osd.0 (osd.0) 113 : cluster [DBG] 4.1e scrub ok 2021-02-10T04:46:15.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[29319]: cluster 2021-02-10T04:46:14.316177+0000 mgr.y (mgr.14140 2021-02-10T04:46:15.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[29319]: ) 1010 : cluster [DBG] pgmap v1779: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:46:15.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[29319]: cluster 2021-02-10T04:46:14.549129+0000 osd.3 (osd.3 2021-02-10T04:46:15.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[29319]: ) 106 : cluster [DBG] 6.1b scrub ok 2021-02-10T04:46:15.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:46:15.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[29319]: -10T04:46:14 2021-02-10T04:46:15.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[29319]: .970670+0000 osd.2 (osd. 2021-02-10T04:46:15.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[29319]: 2) 1949 : cluster 2021-02-10T04:46:15.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[29319]: [DBG] purged_snaps scrub starts 2021-02-10T04:46:15.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[29319]: cluster 2021-02-10T04:46:15.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:46:15.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[29319]: 46:14.971468+0000 2021-02-10T04:46:15.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[29319]: osd.2 (osd.2) 1950 2021-02-10T04:46:15.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[29319]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:46:15.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:46:15.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[29319]: T04:46:15.536654+0000 osd.3 (osd.3) 107 : cluster [DBG] 6.b scrub ok 2021-02-10T04:46:15.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[35961]: cluster 2021-02-10T04:46:14 2021-02-10T04:46:15.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[35961]: .316177+0000 mgr.y (mgr.14140) 1010 : cluster [DBG] pgmap v1779: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:46:15.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[35961]: cluster 2021-02-10T04:46:15.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[35961]: 2021-02-10T04:46:14.549129+0000 osd.3 (osd.3) 106 : cluster 2021-02-10T04:46:15.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[35961]: [DBG] 6.1b scrub ok 2021-02-10T04:46:15.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[35961]: cluster 2021-02-10T04:46 2021-02-10T04:46:15.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[35961]: :14.970670+0000 osd.2 (osd.2) 2021-02-10T04:46:15.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[35961]: 1949 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:46:15.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:46:15.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[35961]: -02-10T04:46:14.971468+0000 2021-02-10T04:46:15.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[35961]: osd.2 (osd.2) 1950 : cluster [DBG] 2021-02-10T04:46:15.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:46:15.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[35961]: cluster 2021-02-10T04:46:15 2021-02-10T04:46:15.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[35961]: .536654+0000 osd.3 (osd.3) 107 : 2021-02-10T04:46:15.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:15 smithi085 conmon[35961]: cluster [DBG] 6.b scrub ok 2021-02-10T04:46:16.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:15 smithi164 conmon[29288]: cluster 2021-02-10T04:46:14.316177+0000 mgr.y (mgr.14140) 1010 : 2021-02-10T04:46:16.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:15 smithi164 conmon[29288]: cluster [DBG] pgmap v1779: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:46:16.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:15 smithi164 conmon[29288]: cluster 2021-02-10T04:46:14.549129+0000 osd.3 (osd.3) 106 : cluster [DBG] 6.1b scrub ok 2021-02-10T04:46:16.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:15 smithi164 conmon[29288]: cluster 2021-02-10T04:46:14.970670+0000 osd.2 (osd.2) 1949 : cluster [DBG] 2021-02-10T04:46:16.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:15 smithi164 conmon[29288]: purged_snaps scrub starts 2021-02-10T04:46:16.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:15 smithi164 conmon[29288]: cluster 2021-02-10T04:46:14.971468+0000 osd.2 (osd.2) 1950 2021-02-10T04:46:16.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:15 smithi164 conmon[29288]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:46:16.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:15 smithi164 conmon[29288]: cluster 2021-02-10T04:46:16.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:15 smithi164 conmon[29288]: 2021-02-10T04:46:15.536654+0000 osd.3 ( 2021-02-10T04:46:16.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:15 smithi164 conmon[29288]: osd.3) 107 : cluster [DBG] 6.b scrub ok 2021-02-10T04:46:16.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:16 smithi085 conmon[29319]: cluster 2021-02-10T04:46:16.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:16 smithi085 conmon[29319]: 2021-02 2021-02-10T04:46:16.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:16 smithi085 conmon[29319]: -10T04:46:15.009734+0000 osd.5 (osd.5) 104 : cluster [DBG] 6.19 scrub ok 2021-02-10T04:46:16.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:16 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:46:16.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:16 smithi085 conmon[29319]: 02-10T04:46:15.684698+0000 mon.a (mon. 2021-02-10T04:46:16.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:16 smithi085 conmon[29319]: 0) 2104 : cluster [DBG] osdmap e1052: 8 total, 8 up, 7 in 2021-02-10T04:46:16.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:16 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:46:16.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:16 smithi085 conmon[29319]: -02-10T04:46:15.788083+0000 2021-02-10T04:46:16.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:16 smithi085 conmon[29319]: osd.0 (osd.0) 114 : cluster [DBG] 2.0 scrub ok 2021-02-10T04:46:16.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:16 smithi085 conmon[29319]: 2021-02-10T04:46:16.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:16 smithi085 conmon[29319]: cluster 2021-02-10T04:46 2021-02-10T04:46:16.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:16 smithi085 conmon[29319]: :15.813933+0000 osd.4 (osd.4) 1943 2021-02-10T04:46:16.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:16 smithi085 conmon[29319]: : cluster [DBG] 5.1a scrub ok 2021-02-10T04:46:16.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:16 smithi085 conmon[35961]: cluster 2021-02-10T04:46:15. 2021-02-10T04:46:16.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:16 smithi085 conmon[35961]: 009734+0000 osd.5 (osd.5) 104 : cluster [DBG] 6.19 scrub ok 2021-02-10T04:46:16.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:16 smithi085 conmon[35961]: cluster 2021-02-10T04:46:16.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:16 smithi085 conmon[35961]: 2021-02-10T04:46:15.684698+0000 2021-02-10T04:46:16.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:16 smithi085 conmon[35961]: mon.a (mon.0) 2104 : cluster 2021-02-10T04:46:16.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:16 smithi085 conmon[35961]: [DBG] osdmap e1052: 8 total, 8 up, 7 in 2021-02-10T04:46:16.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:16 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:46:16.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:16 smithi085 conmon[35961]: 04:46:15.788083+0000 osd.0 (osd 2021-02-10T04:46:16.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:16 smithi085 conmon[35961]: .0) 114 : cluster [DBG] 2.0 scrub ok 2021-02-10T04:46:16.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:16 smithi085 conmon[35961]: cluster 2021-02-10T04:46:16.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:16 smithi085 conmon[35961]: 2021-02-10T04:46:15.813933+0000 2021-02-10T04:46:16.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:16 smithi085 conmon[35961]: osd.4 (osd.4) 1943 : cluster [DBG] 2021-02-10T04:46:16.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:16 smithi085 conmon[35961]: 5.1a scrub ok 2021-02-10T04:46:17.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:16 smithi164 conmon[29288]: cluster 2021-02-10T04:46:15.009734+0000 osd.5 (osd. 2021-02-10T04:46:17.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:16 smithi164 conmon[29288]: 5) 104 : cluster [DBG] 6.19 scrub ok 2021-02-10T04:46:17.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:16 smithi164 conmon[29288]: cluster 2021-02-10T04:46:15.684698+0000 mon.a (mon.0) 2104 : cluster [DBG] osdmap e1052: 8 total, 8 up, 7 in 2021-02-10T04:46:17.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:16 smithi164 conmon[29288]: cluster 2021-02-10T04:46 2021-02-10T04:46:17.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:16 smithi164 conmon[29288]: :15.788083+0000 osd.0 (osd.0) 114 : cluster [DBG] 2.0 scrub ok 2021-02-10T04:46:17.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:16 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:46:17.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:16 smithi164 conmon[29288]: -10T04:46:15.813933+0000 2021-02-10T04:46:17.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:16 smithi164 conmon[29288]: osd.4 (osd.4) 1943 : 2021-02-10T04:46:17.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:16 smithi164 conmon[29288]: cluster [DBG] 5.1a scrub ok 2021-02-10T04:46:17.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:17 smithi085 conmon[29319]: cluster 2021-02-10T04:46:16.042345 2021-02-10T04:46:17.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:17 smithi085 conmon[29319]: +0000 osd.5 (osd.5) 105 : cluster [DBG] 6.a scrub ok 2021-02-10T04:46:17.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:17 smithi085 conmon[29319]: cluster 2021-02-10T04:46:16.316579+0000 2021-02-10T04:46:17.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:17 smithi085 conmon[29319]: mgr.y (mgr.14140) 1011 : cluster [DBG] pgmap v1781: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:46:17.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:17 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:46:17.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:17 smithi085 conmon[29319]: -02-10T04:46:17.537061 2021-02-10T04:46:17.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:17 smithi085 conmon[29319]: +0000 osd.3 (osd.3) 108 : cluster [DBG] 2021-02-10T04:46:17.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:17 smithi085 conmon[29319]: 3.2 scrub ok 2021-02-10T04:46:17.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:17 smithi085 conmon[35961]: cluster 2021-02-10T04:46:16.042345+0000 2021-02-10T04:46:17.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:17 smithi085 conmon[35961]: osd.5 (osd.5) 105 : cluster [DBG] 6.a scrub ok 2021-02-10T04:46:17.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:17 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:46:17.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:17 smithi085 conmon[35961]: T04:46:16.316579+0000 mgr.y (mgr.14140) 1011 : cluster [DBG] pgmap v1781: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:46:17.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:17 smithi085 conmon[35961]: 2021-02-10T04:46:17.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:17 smithi085 conmon[35961]: cluster 2021-02-10T04:46:17 2021-02-10T04:46:17.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:17 smithi085 conmon[35961]: .537061+0000 osd.3 (osd.3) 108 : 2021-02-10T04:46:17.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:17 smithi085 conmon[35961]: cluster [DBG] 3.2 scrub ok 2021-02-10T04:46:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:17 smithi164 conmon[29288]: cluster 2021-02-10T04:46:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:17 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:46:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:17 smithi164 conmon[29288]: 04:46:16.042345+0000 osd.5 (osd.5) 105 : cluster [DBG] 6.a scrub ok 2021-02-10T04:46:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:17 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:46:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:17 smithi164 conmon[29288]: :46:16.316579+0000 mgr.y (mgr.14140) 1011 : cluster 2021-02-10T04:46:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:17 smithi164 conmon[29288]: [DBG] pgmap v1781: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:46:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:17 smithi164 conmon[29288]: cluster 2021-02-10T04:46 2021-02-10T04:46:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:17 smithi164 conmon[29288]: :17.537061+0000 osd.3 (osd.3 2021-02-10T04:46:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:17 smithi164 conmon[29288]: ) 108 : cluster [DBG] 3.2 scrub ok 2021-02-10T04:46:19.144 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:46:18 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:46:25.069 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:18 smithi164 conmon[29288]: cluster 2021-02-10T04:46:17.816875+0000 osd.4 2021-02-10T04:46:25.069 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:18 smithi164 conmon[29288]: (osd.4) 1944 : cluster [DBG] 5.e scrub ok 2021-02-10T04:46:25.074 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:18 smithi085 conmon[35961]: cluster 2021-02-10T04:46:17.816875 2021-02-10T04:46:25.074 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:18 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1944 : cluster [DBG] 5.e scrub ok 2021-02-10T04:46:25.074 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:19 smithi085 conmon[35961]: cluster 2021-02-10T04:46:25.074 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:19 smithi085 conmon[35961]: 2021-02-10T04:46:25.074 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:19 smithi085 conmon[35961]: 2021 2021-02-10T04:46:25.074 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:19 smithi085 conmon[35961]: - 2021-02-10T04:46:25.074 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:19 smithi085 conmon[35961]: 02-10T04:46: 2021-02-10T04:46:25.074 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:19 smithi085 conmon[35961]: 18.092529+0000 osd.5 (osd.5) 106 : cluster [DBG] 5.16 scrub ok 2021-02-10T04:46:25.074 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:19 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:46:25.074 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:19 smithi085 conmon[35961]: 02-10T04:46:18.317170+0000 mgr.y (mgr. 2021-02-10T04:46:25.074 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:19 smithi085 conmon[35961]: 14140) 1012 : cluster [DBG] pgmap v1782: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:46:25.074 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:19 smithi085 conmon[35961]: audit 2021 2021-02-10T04:46:25.075 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:19 smithi085 conmon[35961]: -02-10T04:46:18.753024+0000 2021-02-10T04:46:25.075 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:19 smithi085 conmon[35961]: mgr.y (mgr.14140) 1013 : audit [DBG] 2021-02-10T04:46:25.075 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:19 smithi085 conmon[35961]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:46:25.075 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:19 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:46:25.075 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:19 smithi085 conmon[35961]: 46:18.777652+0000 osd.0 (osd.0 2021-02-10T04:46:25.075 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:19 smithi085 conmon[35961]: ) 115 : cluster [DBG] 4.6 scrub ok 2021-02-10T04:46:25.075 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:19 smithi085 conmon[35961]: cluster 2021-02-10T04:46:19.573099+0000 osd.3 (osd 2021-02-10T04:46:25.075 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:19 smithi085 conmon[35961]: .3) 109 : cluster [DBG] 2.a scrub ok 2021-02-10T04:46:25.075 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:20 smithi085 conmon[35961]: cluster 2021-02-10T04:46:19.084076 2021-02-10T04:46:25.075 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:20 smithi085 conmon[35961]: +0000 osd.6 (osd.6) 105 : cluster [DBG] 7.c scrub ok 2021-02-10T04:46:25.075 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:20 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:46:25.075 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:20 smithi085 conmon[35961]: 10T04:46:19.757479+0000 osd.0 (osd.0) 116 : cluster [DBG] 2021-02-10T04:46:25.075 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:20 smithi085 conmon[35961]: 7.f scrub ok 2021-02-10T04:46:25.076 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:21 smithi085 conmon[35961]: cluster 2021-02-10T04:46:20.029740+0000 osd.5 (osd 2021-02-10T04:46:25.076 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:21 smithi085 conmon[35961]: .5) 107 : cluster [DBG] 6.1d scrub ok 2021-02-10T04:46:25.076 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:21 smithi085 conmon[35961]: cluster 2021-02-10T04:46:20.318000+0000 mgr.y (mgr.14140) 2021-02-10T04:46:25.076 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:21 smithi085 conmon[35961]: 1014 : cluster [DBG] pgmap v1783: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:46:25.076 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:21 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:46:25.076 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:21 smithi085 conmon[35961]: 10T04:46:20.788079+0000 osd.4 (osd.4) 1945 : cluster [DBG] 4.d scrub ok 2021-02-10T04:46:25.076 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:22 smithi085 conmon[35961]: cluster 2021-02-10T04:46:25.076 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:22 smithi085 conmon[35961]: 2021-02-10T04:46:25.076 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:22 smithi085 conmon[35961]: 2021 2021-02-10T04:46:25.076 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:22 smithi085 conmon[35961]: - 2021-02-10T04:46:25.076 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:22 smithi085 conmon[35961]: 02 2021-02-10T04:46:25.076 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:22 smithi085 conmon[35961]: -10T04:46:21.050628+0000 2021-02-10T04:46:25.076 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:22 smithi085 conmon[35961]: osd.6 (osd.6) 106 : cluster [DBG] 1.3 scrub ok 2021-02-10T04:46:25.077 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:22 smithi085 conmon[35961]: cluster 2021-02-10T04:46:21 2021-02-10T04:46:25.077 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:22 smithi085 conmon[35961]: .768608+0000 osd.0 (osd.0) 117 : cluster [DBG] 6.10 scrub ok 2021-02-10T04:46:25.077 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:22 smithi085 conmon[35961]: 2021-02-10T04:46:25.077 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:22 smithi085 conmon[35961]: cluster 2021-02-10T04:46: 2021-02-10T04:46:25.077 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:22 smithi085 conmon[35961]: 21.815151+0000 osd.4 (osd.4) 1946 : 2021-02-10T04:46:25.077 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:22 smithi085 conmon[35961]: cluster [DBG] 2.b scrub ok 2021-02-10T04:46:25.077 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:23 smithi085 conmon[35961]: cluster 2021-02-10T04:46:22.069240+0000 osd.6 (osd.6) 107 : cluster [DBG] 2.6 scrub ok 2021-02-10T04:46:25.077 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:23 smithi085 conmon[35961]: cluster 2021-02-10T04:46:22.318451+0000 mgr.y (mgr.14140) 1015 : cluster [DBG] pgmap v1784: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:46:25.077 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:23 smithi085 conmon[35961]: cluster 2021-02-10T04:46:22.735006+0000 osd.0 (osd.0) 118 : cluster [DBG] 2.4 scrub ok 2021-02-10T04:46:25.077 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:18 smithi085 conmon[29319]: cluster 2021-02-10T04:46:17. 2021-02-10T04:46:25.077 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:18 smithi085 conmon[29319]: 816875+0000 osd.4 (osd.4) 1944 : cluster [DBG] 5.e scrub ok 2021-02-10T04:46:25.077 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:19 smithi085 conmon[29319]: cluster 2021-02-10T04:46:18.092529+0000 osd.5 2021-02-10T04:46:25.078 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:19 smithi085 conmon[29319]: (osd.5) 106 : cluster [DBG] 5.16 scrub ok 2021-02-10T04:46:25.078 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:19 smithi085 conmon[29319]: cluster 2021-02-10T04:46:18.317170+0000 mgr.y (mgr.14140) 1012 : cluster [DBG] pgmap v1782: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:46:25.078 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:19 smithi085 conmon[29319]: audit 2021-02-10T04:46:18.753024+0000 mgr.y (mgr.14140) 1013 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:46:25.078 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:19 smithi085 conmon[29319]: 2021-02-10T04:46:25.078 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:19 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:46:25.078 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:19 smithi085 conmon[29319]: -10T04:46:18.777652+0000 osd.0 (osd.0) 115 : cluster [DBG] 4.6 scrub ok 2021-02-10T04:46:25.078 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:19 smithi085 conmon[29319]: cluster 2021-02-10T04:46:19.573099+0000 2021-02-10T04:46:25.078 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:19 smithi085 conmon[29319]: osd.3 (osd.3) 109 : cluster [DBG] 2.a scrub ok 2021-02-10T04:46:25.078 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:20 smithi085 conmon[29319]: cluster 2021-02-10T04:46:19.084076+0000 osd.6 (osd 2021-02-10T04:46:25.078 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:20 smithi085 conmon[29319]: .6) 105 : cluster [DBG] 7.c scrub ok 2021-02-10T04:46:25.078 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:20 smithi085 conmon[29319]: cluster 2021-02-10T04:46:19.757479+0000 osd.0 (osd.0) 116 : cluster [DBG] 7.f scrub ok 2021-02-10T04:46:25.078 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:21 smithi085 conmon[29319]: cluster 2021-02-10T04:46:20.029740+0000 osd.5 ( 2021-02-10T04:46:25.078 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:21 smithi085 conmon[29319]: osd.5) 107 : cluster [DBG] 6.1d scrub ok 2021-02-10T04:46:25.079 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:21 smithi085 conmon[29319]: cluster 2021-02-10T04:46:20.318000+0000 mgr.y (mgr.14140) 1014 2021-02-10T04:46:25.079 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:21 smithi085 conmon[29319]: : cluster [DBG] pgmap v1783: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:46:25.079 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:21 smithi085 conmon[29319]: cluster 2021-02-10T04:46:25.079 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:21 smithi085 conmon[29319]: 2021-02-10T04:46:20.788079+0000 osd.4 (osd.4 2021-02-10T04:46:25.079 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:21 smithi085 conmon[29319]: ) 1945 : cluster [DBG] 4.d scrub ok 2021-02-10T04:46:25.079 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:22 smithi085 conmon[29319]: cluster 2021-02-10T04:46:21.050628+0000 2021-02-10T04:46:25.079 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:22 smithi085 conmon[29319]: osd.6 (osd.6) 106 : cluster [DBG] 1.3 scrub ok 2021-02-10T04:46:25.079 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:22 smithi085 conmon[29319]: cluster 2021-02-10T04:46:21.768608+0000 osd.0 (osd.0) 117 : cluster [DBG] 6.10 scrub ok 2021-02-10T04:46:25.080 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:22 smithi085 conmon[29319]: cluster 2021-02-10T04:46:21.815151+0000 osd.4 (osd.4) 1946 : cluster [DBG] 2.b scrub ok 2021-02-10T04:46:25.080 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:23 smithi085 conmon[29319]: cluster 2021-02-10T04:46:25.080 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:23 smithi085 conmon[29319]: 2021-02-10T04:46: 2021-02-10T04:46:25.080 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:23 smithi085 conmon[29319]: 22.069240+0000 osd.6 (osd.6) 107 : cluster [DBG] 2.6 scrub ok 2021-02-10T04:46:25.080 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:23 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:46:25.080 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:23 smithi085 conmon[29319]: :46:22.318451+0000 mgr.y (mgr.14140) 1015 : cluster [DBG] pgmap v1784: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:46:25.080 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:23 smithi085 conmon[29319]: cluster 2021-02-10T04:46:22.735006+0000 osd.0 ( 2021-02-10T04:46:25.080 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:23 smithi085 conmon[29319]: osd.0) 118 : cluster [DBG] 2.4 scrub ok 2021-02-10T04:46:25.081 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:46:25.081 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:46:25.081 INFO:teuthology.orchestra.run.smithi085.stdout: "epoch": 1052, 2021-02-10T04:46:25.081 INFO:teuthology.orchestra.run.smithi085.stdout: "fsid": "42224fce-6b57-11eb-8feb-001a4aab830c", 2021-02-10T04:46:25.082 INFO:teuthology.orchestra.run.smithi085.stdout: "created": "2021-02-10T04:20:57.933279+0000", 2021-02-10T04:46:25.082 INFO:teuthology.orchestra.run.smithi085.stdout: "modified": "2021-02-10T04:46:15.671873+0000", 2021-02-10T04:46:25.082 INFO:teuthology.orchestra.run.smithi085.stdout: "last_up_change": "2021-02-10T04:25:35.312604+0000", 2021-02-10T04:46:25.082 INFO:teuthology.orchestra.run.smithi085.stdout: "last_in_change": "2021-02-10T04:37:36.200492+0000", 2021-02-10T04:46:25.082 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T04:46:25.082 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_num": 5804032, 2021-02-10T04:46:25.082 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_set": [ 2021-02-10T04:46:25.082 INFO:teuthology.orchestra.run.smithi085.stdout: "nodeep-scrub", 2021-02-10T04:46:25.082 INFO:teuthology.orchestra.run.smithi085.stdout: "pglog_hardlimit", 2021-02-10T04:46:25.082 INFO:teuthology.orchestra.run.smithi085.stdout: "purged_snapdirs", 2021-02-10T04:46:25.082 INFO:teuthology.orchestra.run.smithi085.stdout: "recovery_deletes", 2021-02-10T04:46:25.082 INFO:teuthology.orchestra.run.smithi085.stdout: "sortbitwise" 2021-02-10T04:46:25.082 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:46:25.083 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_version": 18, 2021-02-10T04:46:25.083 INFO:teuthology.orchestra.run.smithi085.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T04:46:25.083 INFO:teuthology.orchestra.run.smithi085.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T04:46:25.083 INFO:teuthology.orchestra.run.smithi085.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T04:46:25.083 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_snapshot": "", 2021-02-10T04:46:25.083 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_max": 7, 2021-02-10T04:46:25.083 INFO:teuthology.orchestra.run.smithi085.stdout: "max_osd": 8, 2021-02-10T04:46:25.083 INFO:teuthology.orchestra.run.smithi085.stdout: "require_min_compat_client": "luminous", 2021-02-10T04:46:25.083 INFO:teuthology.orchestra.run.smithi085.stdout: "min_compat_client": "luminous", 2021-02-10T04:46:25.083 INFO:teuthology.orchestra.run.smithi085.stdout: "require_osd_release": "quincy", 2021-02-10T04:46:25.083 INFO:teuthology.orchestra.run.smithi085.stdout: "pools": [ 2021-02-10T04:46:25.083 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.083 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 1, 2021-02-10T04:46:25.084 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "device_health_metrics", 2021-02-10T04:46:25.084 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:24:05.258028+0000", 2021-02-10T04:46:25.084 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:46:25.084 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:46:25.084 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:46:25.084 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:46:25.084 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:46:25.084 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:46:25.084 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:46:25.084 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:46:25.084 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:46:25.084 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:46:25.084 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:46:25.085 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:46:25.085 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 11, 2021-02-10T04:46:25.085 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 11, 2021-02-10T04:46:25.085 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 11, 2021-02-10T04:46:25.085 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 11, 2021-02-10T04:46:25.085 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 11, 2021-02-10T04:46:25.085 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:46:25.085 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:46:25.085 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:46:25.085 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:46:25.085 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:46:25.085 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:46:25.085 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:46:25.086 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.086 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "112", 2021-02-10T04:46:25.086 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:46:25.086 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:46:25.086 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "110", 2021-02-10T04:46:25.086 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:46:25.086 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:46:25.086 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:46:25.086 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:46:25.086 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:46:25.086 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:46:25.086 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:46:25.087 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:46:25.087 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:46:25.087 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:46:25.087 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:46:25.087 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:46:25.087 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:46:25.087 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:46:25.087 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:46:25.087 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:46:25.087 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:46:25.087 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:46:25.087 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:46:25.087 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:46:25.088 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:46:25.088 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:46:25.088 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:46:25.088 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.088 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:46:25.088 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:46:25.088 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:46:25.088 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:46:25.088 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:46:25.088 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:46:25.088 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:46:25.088 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:46:25.088 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:46:25.089 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:46:25.089 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:46:25.089 INFO:teuthology.orchestra.run.smithi085.stdout: "options": { 2021-02-10T04:46:25.089 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_min": 1 2021-02-10T04:46:25.089 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.089 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:46:25.089 INFO:teuthology.orchestra.run.smithi085.stdout: "mgr_devicehealth": {} 2021-02-10T04:46:25.089 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.089 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.089 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.089 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 2, 2021-02-10T04:46:25.089 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": ".rgw.root", 2021-02-10T04:46:25.089 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:32.688554+0000", 2021-02-10T04:46:25.090 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:46:25.090 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:46:25.090 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:46:25.090 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:46:25.090 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:46:25.090 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:46:25.090 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:46:25.090 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:46:25.090 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:46:25.090 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:46:25.090 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:46:25.090 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:46:25.091 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 42, 2021-02-10T04:46:25.091 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 42, 2021-02-10T04:46:25.091 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 42, 2021-02-10T04:46:25.091 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 42, 2021-02-10T04:46:25.091 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 42, 2021-02-10T04:46:25.091 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:46:25.091 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:46:25.091 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:46:25.091 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:46:25.091 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:46:25.091 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:46:25.091 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:46:25.091 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.092 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "91", 2021-02-10T04:46:25.092 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:46:25.092 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:46:25.092 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "89", 2021-02-10T04:46:25.092 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:46:25.092 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:46:25.092 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:46:25.092 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:46:25.092 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:46:25.092 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:46:25.092 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:46:25.092 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:46:25.092 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:46:25.093 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:46:25.093 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:46:25.093 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:46:25.093 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:46:25.093 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:46:25.093 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:46:25.093 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:46:25.093 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:46:25.093 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:46:25.093 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:46:25.093 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:46:25.093 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:46:25.093 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:46:25.094 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:46:25.094 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.094 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:46:25.094 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:46:25.094 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:46:25.094 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:46:25.094 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:46:25.094 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:46:25.094 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:46:25.094 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:46:25.094 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:46:25.094 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:46:25.094 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:46:25.095 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:46:25.095 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:46:25.095 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T04:46:25.095 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.095 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.095 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.095 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 3, 2021-02-10T04:46:25.095 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "iscsi", 2021-02-10T04:46:25.095 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:47.658613+0000", 2021-02-10T04:46:25.095 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:46:25.095 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:46:25.095 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:46:25.095 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:46:25.096 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:46:25.096 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:46:25.096 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:46:25.096 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:46:25.096 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:46:25.096 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:46:25.096 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:46:25.096 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:46:25.096 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 3, 2021-02-10T04:46:25.096 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 3, 2021-02-10T04:46:25.096 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 3, 2021-02-10T04:46:25.096 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 3, 2021-02-10T04:46:25.096 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 3, 2021-02-10T04:46:25.097 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:46:25.097 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:46:25.097 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:46:25.097 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:46:25.097 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:46:25.097 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:46:25.097 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:46:25.097 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.097 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "58", 2021-02-10T04:46:25.097 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:46:25.097 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:46:25.097 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T04:46:25.097 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:46:25.098 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:46:25.098 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:46:25.098 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:46:25.098 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:46:25.098 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:46:25.098 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:46:25.098 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:46:25.098 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:46:25.098 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:46:25.098 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:46:25.098 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:46:25.098 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:46:25.099 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:46:25.099 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:46:25.099 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:46:25.099 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:46:25.099 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:46:25.099 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:46:25.099 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:46:25.099 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:46:25.099 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:46:25.099 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:46:25.099 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.099 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:46:25.099 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:46:25.100 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:46:25.100 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:46:25.100 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:46:25.100 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:46:25.100 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:46:25.100 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:46:25.100 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:46:25.100 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:46:25.100 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:46:25.100 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:46:25.100 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:46:25.100 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd": {} 2021-02-10T04:46:25.100 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.101 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.101 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.101 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 4, 2021-02-10T04:46:25.101 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "zone.rgw.log", 2021-02-10T04:46:25.101 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:47.968727+0000", 2021-02-10T04:46:25.101 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:46:25.101 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:46:25.101 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:46:25.101 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:46:25.101 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:46:25.101 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:46:25.101 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:46:25.101 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:46:25.101 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:46:25.102 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:46:25.102 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:46:25.102 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:46:25.102 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 32, 2021-02-10T04:46:25.102 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 32, 2021-02-10T04:46:25.102 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 32, 2021-02-10T04:46:25.102 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 32, 2021-02-10T04:46:25.102 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 32, 2021-02-10T04:46:25.102 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:46:25.102 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:46:25.102 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:46:25.102 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:46:25.102 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:46:25.103 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:46:25.103 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:46:25.103 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.103 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "57", 2021-02-10T04:46:25.103 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:46:25.103 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:46:25.103 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T04:46:25.103 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:46:25.103 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:46:25.103 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:46:25.103 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:46:25.103 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:46:25.103 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:46:25.104 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:46:25.104 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:46:25.104 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:46:25.104 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:46:25.104 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:46:25.104 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:46:25.104 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:46:25.104 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:46:25.104 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:46:25.104 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:46:25.104 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:46:25.104 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:46:25.104 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:46:25.105 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:46:25.105 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:46:25.105 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:46:25.105 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:46:25.105 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.105 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:46:25.105 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:46:25.105 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:46:25.105 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:46:25.105 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:46:25.105 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:46:25.105 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:46:25.105 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:46:25.106 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:46:25.106 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:46:25.106 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:46:25.106 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:46:25.106 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:46:25.106 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T04:46:25.106 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.106 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.106 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.106 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 5, 2021-02-10T04:46:25.106 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "zone.rgw.control", 2021-02-10T04:46:25.106 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:49.278260+0000", 2021-02-10T04:46:25.106 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:46:25.107 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:46:25.107 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:46:25.107 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:46:25.107 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:46:25.107 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:46:25.107 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:46:25.107 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:46:25.107 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:46:25.107 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:46:25.107 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:46:25.107 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:46:25.107 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 32, 2021-02-10T04:46:25.107 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 32, 2021-02-10T04:46:25.107 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 32, 2021-02-10T04:46:25.108 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 32, 2021-02-10T04:46:25.108 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 32, 2021-02-10T04:46:25.108 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:46:25.108 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:46:25.108 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:46:25.108 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:46:25.108 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:46:25.108 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:46:25.108 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:46:25.108 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.108 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "59", 2021-02-10T04:46:25.108 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:46:25.108 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:46:25.109 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T04:46:25.109 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:46:25.109 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:46:25.109 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:46:25.109 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:46:25.109 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:46:25.109 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:46:25.109 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:46:25.109 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:46:25.109 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:46:25.109 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:46:25.109 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:46:25.109 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:46:25.110 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:46:25.110 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:46:25.110 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:46:25.110 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:46:25.110 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:46:25.110 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:46:25.110 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:46:25.110 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:46:25.110 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:46:25.110 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:46:25.110 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:46:25.110 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.110 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:46:25.111 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:46:25.111 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:46:25.111 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:46:25.111 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:46:25.111 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:46:25.111 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:46:25.111 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:46:25.111 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:46:25.111 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:46:25.111 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:46:25.111 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:46:25.111 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:46:25.111 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T04:46:25.112 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.112 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.112 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.112 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 6, 2021-02-10T04:46:25.112 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "zone.rgw.meta", 2021-02-10T04:46:25.112 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:51.792514+0000", 2021-02-10T04:46:25.112 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:46:25.112 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:46:25.112 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:46:25.112 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:46:25.112 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:46:25.112 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:46:25.112 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:46:25.112 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:46:25.113 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:46:25.113 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:46:25.113 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:46:25.113 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:46:25.113 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 42, 2021-02-10T04:46:25.113 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 42, 2021-02-10T04:46:25.113 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 42, 2021-02-10T04:46:25.113 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 42, 2021-02-10T04:46:25.113 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 42, 2021-02-10T04:46:25.113 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:46:25.113 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:46:25.113 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:46:25.113 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:46:25.113 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:46:25.114 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:46:25.114 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:46:25.114 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.114 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "1046", 2021-02-10T04:46:25.114 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:46:25.114 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:46:25.114 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "1044", 2021-02-10T04:46:25.114 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:46:25.114 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:46:25.114 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:46:25.114 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:46:25.114 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:46:25.114 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:46:25.115 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:46:25.115 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:46:25.115 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:46:25.115 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:46:25.115 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:46:25.115 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:46:25.115 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:46:25.115 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:46:25.115 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:46:25.115 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:46:25.115 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:46:25.115 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:46:25.115 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:46:25.115 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:46:25.115 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:46:25.116 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:46:25.116 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:46:25.116 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.116 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:46:25.116 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:46:25.116 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:46:25.116 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:46:25.116 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:46:25.116 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:46:25.116 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:46:25.116 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:46:25.116 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:46:25.116 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:46:25.117 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:46:25.117 INFO:teuthology.orchestra.run.smithi085.stdout: "options": { 2021-02-10T04:46:25.117 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_bias": 4, 2021-02-10T04:46:25.117 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_min": 8 2021-02-10T04:46:25.117 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.117 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:46:25.117 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T04:46:25.117 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.117 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.117 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.117 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 7, 2021-02-10T04:46:25.117 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "unique_pool_0", 2021-02-10T04:46:25.117 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:28:23.932391+0000", 2021-02-10T04:46:25.118 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 8193, 2021-02-10T04:46:25.118 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2021-02-10T04:46:25.118 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:46:25.118 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:46:25.118 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:46:25.118 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:46:25.118 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:46:25.118 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:46:25.118 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:46:25.118 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:46:25.118 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:46:25.118 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:46:25.118 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 26, 2021-02-10T04:46:25.119 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 26, 2021-02-10T04:46:25.119 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 26, 2021-02-10T04:46:25.119 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 26, 2021-02-10T04:46:25.119 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 26, 2021-02-10T04:46:25.119 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:46:25.119 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:46:25.119 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:46:25.119 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:46:25.119 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:46:25.119 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:46:25.119 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:46:25.119 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.119 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "1009", 2021-02-10T04:46:25.120 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:46:25.120 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:46:25.120 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "1007", 2021-02-10T04:46:25.120 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:46:25.120 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:46:25.120 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 888, 2021-02-10T04:46:25.120 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 994, 2021-02-10T04:46:25.120 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:46:25.120 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:46:25.120 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:46:25.120 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:46:25.120 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:46:25.120 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:46:25.121 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:46:25.121 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:46:25.121 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:46:25.121 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:46:25.121 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:46:25.121 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:46:25.121 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:46:25.121 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:46:25.121 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:46:25.121 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:46:25.121 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:46:25.121 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:46:25.122 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:46:25.122 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.122 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:46:25.122 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:46:25.122 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:46:25.122 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:46:25.122 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:46:25.122 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:46:25.122 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:46:25.122 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:46:25.122 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:46:25.122 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:46:25.122 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:46:25.123 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:46:25.123 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:46:25.123 INFO:teuthology.orchestra.run.smithi085.stdout: "rados": {} 2021-02-10T04:46:25.123 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.123 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.123 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:46:25.123 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T04:46:25.123 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.123 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 0, 2021-02-10T04:46:25.123 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "45d0bb5f-35fa-4596-964f-75abcee53bee", 2021-02-10T04:46:25.123 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:46:25.123 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:46:25.123 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:46:25.124 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:46:25.124 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:46:25.124 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:46:25.124 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 8, 2021-02-10T04:46:25.124 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1046, 2021-02-10T04:46:25.124 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:46:25.124 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:46:25.124 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:46:25.124 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.124 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.124 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.124 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6801", 2021-02-10T04:46:25.124 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.125 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.125 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.125 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.125 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:46:25.125 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.125 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.125 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.125 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6802", 2021-02-10T04:46:25.125 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.125 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.125 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.125 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.125 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:46:25.126 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.126 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.126 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.126 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6804", 2021-02-10T04:46:25.126 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.126 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.126 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.126 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.126 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:46:25.126 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.126 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.126 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.126 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6803", 2021-02-10T04:46:25.127 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.127 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.127 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.127 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.127 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6801/7", 2021-02-10T04:46:25.127 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6802/7", 2021-02-10T04:46:25.127 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6804/7", 2021-02-10T04:46:25.127 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6803/7", 2021-02-10T04:46:25.127 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:46:25.127 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:46:25.127 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:46:25.127 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.127 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.128 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.128 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 1, 2021-02-10T04:46:25.128 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "3421a5ce-872e-4b52-adf7-6aaa595871a4", 2021-02-10T04:46:25.128 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:46:25.128 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:46:25.128 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:46:25.128 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 0, 2021-02-10T04:46:25.128 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:46:25.128 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:46:25.128 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 13, 2021-02-10T04:46:25.128 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1039, 2021-02-10T04:46:25.128 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:46:25.128 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:46:25.129 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:46:25.129 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.129 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.129 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.129 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6805", 2021-02-10T04:46:25.129 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.129 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.129 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.129 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.129 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:46:25.129 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.129 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.130 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.130 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6806", 2021-02-10T04:46:25.130 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.130 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.130 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.130 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.130 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:46:25.130 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.130 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.130 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.130 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6808", 2021-02-10T04:46:25.130 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.131 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.131 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.131 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.131 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:46:25.131 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.131 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.131 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.131 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6807", 2021-02-10T04:46:25.131 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.131 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.131 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.131 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.131 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6805/7", 2021-02-10T04:46:25.132 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6806/7", 2021-02-10T04:46:25.132 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6808/7", 2021-02-10T04:46:25.132 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6807/7", 2021-02-10T04:46:25.132 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:46:25.132 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:46:25.132 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:46:25.132 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.132 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.132 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.132 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 2, 2021-02-10T04:46:25.132 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "96e1952a-1042-4316-9fc9-86813e88b912", 2021-02-10T04:46:25.133 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:46:25.133 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 0, 2021-02-10T04:46:25.133 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 0, 2021-02-10T04:46:25.133 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:46:25.133 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:46:25.133 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:46:25.133 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 18, 2021-02-10T04:46:25.133 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 367, 2021-02-10T04:46:25.133 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:46:25.133 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:46:25.133 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:46:25.133 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.134 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.134 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.134 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6809", 2021-02-10T04:46:25.134 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.134 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.134 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.134 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.134 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:46:25.134 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.134 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.134 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.135 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6810", 2021-02-10T04:46:25.135 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.135 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.135 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.135 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.135 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:46:25.135 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.135 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.135 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.135 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6812", 2021-02-10T04:46:25.135 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.135 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.135 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.136 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.136 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:46:25.136 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.136 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.136 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.136 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6811", 2021-02-10T04:46:25.136 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.136 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.136 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.136 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.136 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6809/7", 2021-02-10T04:46:25.136 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6810/7", 2021-02-10T04:46:25.136 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6812/7", 2021-02-10T04:46:25.137 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6811/7", 2021-02-10T04:46:25.137 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:46:25.137 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:46:25.137 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:46:25.137 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.137 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.137 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.137 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 3, 2021-02-10T04:46:25.137 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "e019f49f-5dd4-4104-9ced-94c710616642", 2021-02-10T04:46:25.137 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:46:25.137 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:46:25.137 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:46:25.137 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:46:25.138 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:46:25.138 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:46:25.138 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 25, 2021-02-10T04:46:25.138 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1044, 2021-02-10T04:46:25.138 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:46:25.138 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:46:25.138 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:46:25.138 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.138 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.138 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.138 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6813", 2021-02-10T04:46:25.138 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.138 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.139 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.139 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.139 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:46:25.139 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.139 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.139 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.139 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6814", 2021-02-10T04:46:25.139 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.139 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.139 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.139 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.139 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:46:25.139 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.140 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.140 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.140 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6816", 2021-02-10T04:46:25.140 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.140 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.140 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.140 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.140 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:46:25.140 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.140 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.140 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.140 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6815", 2021-02-10T04:46:25.140 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.141 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.141 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.141 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.141 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6813/7", 2021-02-10T04:46:25.141 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6814/7", 2021-02-10T04:46:25.141 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6816/7", 2021-02-10T04:46:25.141 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6815/7", 2021-02-10T04:46:25.141 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:46:25.141 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:46:25.141 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:46:25.141 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.141 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.141 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.141 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 4, 2021-02-10T04:46:25.142 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "91c8a80a-37b6-4c30-a32a-647b2e6d801f", 2021-02-10T04:46:25.142 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:46:25.142 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:46:25.142 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:46:25.142 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:46:25.142 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:46:25.142 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:46:25.142 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 31, 2021-02-10T04:46:25.142 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1046, 2021-02-10T04:46:25.142 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:46:25.142 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:46:25.142 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:46:25.142 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.143 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.143 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.143 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6800", 2021-02-10T04:46:25.143 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.143 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.143 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.143 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.143 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:46:25.143 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.143 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.143 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.143 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6801", 2021-02-10T04:46:25.143 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.144 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.144 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.144 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.144 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:46:25.144 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.144 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.144 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.144 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6803", 2021-02-10T04:46:25.144 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.144 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.144 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.144 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.144 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:46:25.145 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.145 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.145 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.145 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6802", 2021-02-10T04:46:25.145 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.145 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.145 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.145 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.145 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6800/7", 2021-02-10T04:46:25.145 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6801/7", 2021-02-10T04:46:25.145 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6803/7", 2021-02-10T04:46:25.145 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6802/7", 2021-02-10T04:46:25.145 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:46:25.146 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:46:25.146 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:46:25.146 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.146 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.146 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.146 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 5, 2021-02-10T04:46:25.146 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e", 2021-02-10T04:46:25.146 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:46:25.146 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:46:25.146 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:46:25.146 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:46:25.146 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:46:25.147 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:46:25.147 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 37, 2021-02-10T04:46:25.147 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1046, 2021-02-10T04:46:25.147 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:46:25.147 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:46:25.147 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:46:25.147 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.147 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.147 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.147 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6804", 2021-02-10T04:46:25.147 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.147 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.147 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.147 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.148 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:46:25.148 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.148 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.148 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.148 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6805", 2021-02-10T04:46:25.148 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.148 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.148 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.148 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.148 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:46:25.148 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.148 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.148 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.149 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6807", 2021-02-10T04:46:25.149 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.149 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.149 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.149 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.149 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:46:25.149 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.149 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.149 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.149 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6806", 2021-02-10T04:46:25.149 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.149 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.149 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.150 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.150 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6804/7", 2021-02-10T04:46:25.150 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6805/7", 2021-02-10T04:46:25.150 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6807/7", 2021-02-10T04:46:25.150 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6806/7", 2021-02-10T04:46:25.150 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:46:25.150 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:46:25.150 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:46:25.150 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.150 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.150 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.150 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 6, 2021-02-10T04:46:25.150 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "adc40430-3eeb-4003-a824-343021243077", 2021-02-10T04:46:25.151 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:46:25.151 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:46:25.151 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:46:25.151 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:46:25.151 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:46:25.151 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:46:25.151 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 43, 2021-02-10T04:46:25.151 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1046, 2021-02-10T04:46:25.151 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:46:25.151 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:46:25.151 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:46:25.151 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.151 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.152 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.152 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6808", 2021-02-10T04:46:25.152 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.152 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.152 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.152 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.152 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:46:25.152 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.152 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.152 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.152 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6809", 2021-02-10T04:46:25.152 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.152 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.153 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.153 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.153 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:46:25.153 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.153 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.153 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.153 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6811", 2021-02-10T04:46:25.153 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.153 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.153 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.153 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.153 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:46:25.153 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.154 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.154 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.154 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6810", 2021-02-10T04:46:25.154 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.154 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.154 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.154 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.154 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6808/7", 2021-02-10T04:46:25.154 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6809/7", 2021-02-10T04:46:25.154 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6811/7", 2021-02-10T04:46:25.154 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6810/7", 2021-02-10T04:46:25.154 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:46:25.154 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:46:25.155 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:46:25.155 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.155 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.155 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.155 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 7, 2021-02-10T04:46:25.155 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37", 2021-02-10T04:46:25.155 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:46:25.155 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:46:25.155 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:46:25.155 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 0, 2021-02-10T04:46:25.155 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:46:25.155 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:46:25.156 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 50, 2021-02-10T04:46:25.156 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 112, 2021-02-10T04:46:25.156 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:46:25.156 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:46:25.156 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:46:25.156 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.156 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.156 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.156 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6812", 2021-02-10T04:46:25.156 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.156 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.156 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.156 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.157 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:46:25.157 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.157 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.157 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.157 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6813", 2021-02-10T04:46:25.157 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.157 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.157 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.157 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.157 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:46:25.157 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.157 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.157 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.158 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6815", 2021-02-10T04:46:25.158 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.158 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.158 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.158 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.158 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:46:25.158 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:46:25.158 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.158 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:46:25.158 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6814", 2021-02-10T04:46:25.158 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:46:25.158 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.158 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.159 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.159 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6812/7", 2021-02-10T04:46:25.159 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6813/7", 2021-02-10T04:46:25.159 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6815/7", 2021-02-10T04:46:25.159 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6814/7", 2021-02-10T04:46:25.159 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:46:25.159 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:46:25.159 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:46:25.159 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.159 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.159 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:46:25.159 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_xinfo": [ 2021-02-10T04:46:25.159 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.160 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 0, 2021-02-10T04:46:25.160 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:46:25.160 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:46:25.160 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:46:25.160 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:46:25.160 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:46:25.160 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:45:36.223283+0000", 2021-02-10T04:46:25.160 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:46:25.160 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.160 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.160 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 1, 2021-02-10T04:46:25.160 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:46:25.160 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:46:25.161 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:46:25.161 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:46:25.161 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:46:25.161 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:45:55.620154+0000", 2021-02-10T04:46:25.161 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:46:25.161 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.161 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.161 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 2, 2021-02-10T04:46:25.161 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:46:25.161 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:46:25.161 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:46:25.161 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:46:25.161 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 65536, 2021-02-10T04:46:25.162 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:46:14.971471+0000", 2021-02-10T04:46:25.162 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:46:25.162 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.162 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.162 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 3, 2021-02-10T04:46:25.162 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:46:25.162 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:46:25.162 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:46:25.162 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:46:25.162 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:46:25.162 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:45:32.374158+0000", 2021-02-10T04:46:25.162 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:46:25.162 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.163 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.163 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 4, 2021-02-10T04:46:25.163 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:46:25.163 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:46:25.163 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:46:25.163 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:46:25.163 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:46:25.163 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:45:49.759075+0000", 2021-02-10T04:46:25.163 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:46:25.163 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.163 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.163 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 5, 2021-02-10T04:46:25.163 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:46:25.164 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:46:25.164 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:46:25.164 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:46:25.164 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:46:25.164 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:46:07.417184+0000", 2021-02-10T04:46:25.164 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:46:25.164 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.164 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.164 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 6, 2021-02-10T04:46:25.164 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:46:25.164 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:46:25.164 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:46:25.164 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:46:25.165 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:46:25.165 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:45:25.141031+0000", 2021-02-10T04:46:25.165 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:46:25.165 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.165 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.165 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 7, 2021-02-10T04:46:25.165 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:46:25.165 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:46:25.165 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:46:25.165 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:46:25.165 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:46:25.165 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:45:43.880389+0000", 2021-02-10T04:46:25.165 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:46:25.165 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.166 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:46:25.166 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_upmap": [], 2021-02-10T04:46:25.166 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_upmap_items": [ 2021-02-10T04:46:25.166 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.166 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "2.7", 2021-02-10T04:46:25.166 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:46:25.166 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.166 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:46:25.166 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 4 2021-02-10T04:46:25.166 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.166 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.166 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.166 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.167 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "2.21", 2021-02-10T04:46:25.167 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:46:25.167 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.167 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:46:25.167 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 1 2021-02-10T04:46:25.167 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.167 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.167 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.167 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.167 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.9", 2021-02-10T04:46:25.167 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:46:25.167 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.167 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 1, 2021-02-10T04:46:25.167 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 6 2021-02-10T04:46:25.168 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.168 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.168 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 3, 2021-02-10T04:46:25.168 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 7 2021-02-10T04:46:25.168 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.168 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.168 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.168 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.168 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.b", 2021-02-10T04:46:25.168 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:46:25.168 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.168 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 1, 2021-02-10T04:46:25.168 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 5 2021-02-10T04:46:25.168 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.169 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.169 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.169 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.169 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.10", 2021-02-10T04:46:25.169 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:46:25.169 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.169 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 1, 2021-02-10T04:46:25.169 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 5 2021-02-10T04:46:25.169 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.169 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.169 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.169 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.169 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.6", 2021-02-10T04:46:25.170 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:46:25.170 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.170 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 5, 2021-02-10T04:46:25.170 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 6 2021-02-10T04:46:25.170 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.170 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.170 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.170 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.170 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.8", 2021-02-10T04:46:25.170 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:46:25.170 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.170 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 1, 2021-02-10T04:46:25.170 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 3 2021-02-10T04:46:25.171 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.171 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.171 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.171 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.171 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.10", 2021-02-10T04:46:25.171 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:46:25.171 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.171 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:46:25.171 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 3 2021-02-10T04:46:25.171 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.171 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.171 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.171 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.172 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.18", 2021-02-10T04:46:25.172 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:46:25.172 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.172 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 4, 2021-02-10T04:46:25.172 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 3 2021-02-10T04:46:25.172 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.172 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.172 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.172 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.172 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.1b", 2021-02-10T04:46:25.172 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:46:25.172 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.172 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 0, 2021-02-10T04:46:25.173 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 3 2021-02-10T04:46:25.173 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.173 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.173 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.173 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.173 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "6.1c", 2021-02-10T04:46:25.173 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:46:25.173 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.173 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 3, 2021-02-10T04:46:25.173 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 0 2021-02-10T04:46:25.173 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.173 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.173 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.174 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.174 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "7.6", 2021-02-10T04:46:25.174 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:46:25.174 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.174 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 3, 2021-02-10T04:46:25.174 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 1 2021-02-10T04:46:25.174 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.174 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.174 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.174 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.174 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "7.10", 2021-02-10T04:46:25.174 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:46:25.175 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.175 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:46:25.175 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 5 2021-02-10T04:46:25.175 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.175 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.175 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.175 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.175 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "7.12", 2021-02-10T04:46:25.175 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:46:25.175 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.175 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:46:25.175 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 5 2021-02-10T04:46:25.175 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.176 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.176 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.176 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.176 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "7.15", 2021-02-10T04:46:25.176 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:46:25.176 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:46:25.176 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:46:25.176 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 4 2021-02-10T04:46:25.176 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.176 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:46:25.176 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.176 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:46:25.176 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_temp": [], 2021-02-10T04:46:25.177 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_temp": [], 2021-02-10T04:46:25.177 INFO:teuthology.orchestra.run.smithi085.stdout: "blocklist": { 2021-02-10T04:46:25.177 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/801393150": "2021-02-11T04:21:37.851607+0000", 2021-02-10T04:46:25.177 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/3049671696": "2021-02-11T04:21:37.851607+0000", 2021-02-10T04:46:25.177 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/4056505602": "2021-02-11T04:21:20.700478+0000", 2021-02-10T04:46:25.177 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:6800/7": "2021-02-11T04:21:20.700478+0000", 2021-02-10T04:46:25.177 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/1879026062": "2021-02-11T04:21:20.700478+0000" 2021-02-10T04:46:25.177 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.177 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profiles": { 2021-02-10T04:46:25.177 INFO:teuthology.orchestra.run.smithi085.stdout: "default": { 2021-02-10T04:46:25.177 INFO:teuthology.orchestra.run.smithi085.stdout: "crush-failure-domain": "osd", 2021-02-10T04:46:25.177 INFO:teuthology.orchestra.run.smithi085.stdout: "k": "2", 2021-02-10T04:46:25.177 INFO:teuthology.orchestra.run.smithi085.stdout: "m": "1", 2021-02-10T04:46:25.178 INFO:teuthology.orchestra.run.smithi085.stdout: "plugin": "jerasure", 2021-02-10T04:46:25.178 INFO:teuthology.orchestra.run.smithi085.stdout: "ruleset-failure-domain": "osd", 2021-02-10T04:46:25.178 INFO:teuthology.orchestra.run.smithi085.stdout: "technique": "reed_sol_van" 2021-02-10T04:46:25.178 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.178 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:46:25.178 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps_queue": [], 2021-02-10T04:46:25.178 INFO:teuthology.orchestra.run.smithi085.stdout: "new_removed_snaps": [], 2021-02-10T04:46:25.178 INFO:teuthology.orchestra.run.smithi085.stdout: "new_purged_snaps": [], 2021-02-10T04:46:25.178 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_node_flags": {}, 2021-02-10T04:46:25.178 INFO:teuthology.orchestra.run.smithi085.stdout: "device_class_flags": {}, 2021-02-10T04:46:25.178 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_mode": { 2021-02-10T04:46:25.178 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_mode_enabled": false, 2021-02-10T04:46:25.178 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_bucket_count": 0, 2021-02-10T04:46:25.179 INFO:teuthology.orchestra.run.smithi085.stdout: "degraded_stretch_mode": 0, 2021-02-10T04:46:25.179 INFO:teuthology.orchestra.run.smithi085.stdout: "recovering_stretch_mode": 0, 2021-02-10T04:46:25.179 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_mode_bucket": 0 2021-02-10T04:46:25.179 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:46:25.179 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:46:25.179 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:19 smithi164 conmon[29288]: cluster 2021-02-10T04:46:18.092529+0000 osd.5 (osd.5) 106 : cluster [DBG] 2021-02-10T04:46:25.179 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:19 smithi164 conmon[29288]: 5.16 scrub ok 2021-02-10T04:46:25.179 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:19 smithi164 conmon[29288]: cluster 2021-02-10T04:46:18.317170+0000 mgr.y (mgr.14140) 1012 : cluster [DBG] pgmap v1782: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:46:25.179 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:19 smithi164 conmon[29288]: audit 2021-02-10T04:46:18.753024+0000 mgr.y (mgr.14140) 1013 2021-02-10T04:46:25.179 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:19 smithi164 conmon[29288]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:46:25.179 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:19 smithi164 conmon[29288]: cluster 2021-02-10T04:46:18.777652+0000 2021-02-10T04:46:25.179 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:19 smithi164 conmon[29288]: osd.0 (osd.0) 115 : cluster [DBG] 4.6 scrub ok 2021-02-10T04:46:25.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:19 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:46:25.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:19 smithi164 conmon[29288]: -02-10T04:46:19.573099+0000 osd.3 (osd.3) 109 : cluster [DBG] 2.a scrub ok 2021-02-10T04:46:25.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:20 smithi164 conmon[29288]: cluster 2021-02-10T04:46:25.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:20 smithi164 conmon[29288]: 2021-02-10T04:46:19 2021-02-10T04:46:25.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:20 smithi164 conmon[29288]: .084076+0000 osd.6 (osd.6) 105 : cluster [DBG] 7.c scrub ok 2021-02-10T04:46:25.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:20 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:46:25.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:20 smithi164 conmon[29288]: -02-10T04:46:19.757479+0000 osd.0 (osd.0) 116 : cluster [DBG] 2021-02-10T04:46:25.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:20 smithi164 conmon[29288]: 7.f scrub ok 2021-02-10T04:46:25.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:21 smithi164 conmon[29288]: cluster 2021-02-10T04:46:20.029740+0000 2021-02-10T04:46:25.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:21 smithi164 conmon[29288]: osd.5 (osd.5) 107 : cluster [DBG] 6.1d scrub ok 2021-02-10T04:46:25.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:21 smithi164 conmon[29288]: cluster 2021-02-10T04:46:20.318000+0000 mgr.y (mgr.14140) 1014 : cluster [DBG] pgmap v1783: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:46:25.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:21 smithi164 conmon[29288]: cluster 2021-02-10T04:46:20.788079+0000 osd.4 (osd.4) 1945 : cluster [DBG] 4.d scrub ok 2021-02-10T04:46:25.180 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:22 smithi164 conmon[29288]: cluster 2021-02-10T04:46:21. 2021-02-10T04:46:25.181 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:22 smithi164 conmon[29288]: 050628+0000 osd.6 (osd.6) 106 : cluster [DBG] 1.3 scrub ok 2021-02-10T04:46:25.181 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:22 smithi164 conmon[29288]: cluster 2021-02-10T04:46:21.768608+0000 osd.0 (osd 2021-02-10T04:46:25.181 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:22 smithi164 conmon[29288]: .0) 117 : cluster [DBG] 6.10 scrub ok 2021-02-10T04:46:25.181 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:22 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:46:25.181 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:22 smithi164 conmon[29288]: 04:46:21.815151+0000 osd.4 (osd.4) 1946 2021-02-10T04:46:25.181 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:22 smithi164 conmon[29288]: : cluster [DBG] 2.b scrub ok 2021-02-10T04:46:25.181 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:23 smithi164 conmon[29288]: cluster 2021-02-10T04:46:25.181 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:23 smithi164 conmon[29288]: 2021-02-10T04:46:25.181 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:23 smithi164 conmon[29288]: 2021-02-10T04:46:22.069240+0000 osd.6 (osd. 2021-02-10T04:46:25.181 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:23 smithi164 conmon[29288]: 6) 107 : cluster [DBG] 2.6 scrub ok 2021-02-10T04:46:25.181 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:23 smithi164 conmon[29288]: cluster 2021-02-10T04:46:25.181 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:23 smithi164 conmon[29288]: 2021-02-10T04:46:22.318451+0000 mgr.y ( 2021-02-10T04:46:25.181 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:23 smithi164 conmon[29288]: mgr.14140) 1015 : cluster [DBG] pgmap v1784: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:46:25.181 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:23 smithi164 conmon[29288]: cluster 2021-02-10T04:46:25.182 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:23 smithi164 conmon[29288]: 2021-02-10T04:46:22 2021-02-10T04:46:25.182 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:23 smithi164 conmon[29288]: .735006+0000 osd.0 (osd.0) 118 : cluster 2021-02-10T04:46:25.182 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:23 smithi164 conmon[29288]: [DBG] 2.4 scrub ok 2021-02-10T04:46:25.182 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.6 dump_blocked_ops 2021-02-10T04:46:25.183 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:46:25.184 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:24 smithi164 conmon[29288]: cluster 2021-02-10T04:46:22.990479+0000 osd.5 2021-02-10T04:46:25.184 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:24 smithi164 conmon[29288]: (osd.5) 108 : cluster [DBG] 5.12 scrub ok 2021-02-10T04:46:25.184 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:24 smithi164 conmon[29288]: cluster 2021-02-10T04:46:23.691470+0000 osd.0 (osd.0) 119 : cluster [DBG] 2.21 scrub ok 2021-02-10T04:46:25.184 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:24 smithi164 conmon[29288]: cluster 2021-02-10T04:46:23. 2021-02-10T04:46:25.184 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:24 smithi164 conmon[29288]: 812322+0000 osd.4 (osd.4) 1947 : cluster [DBG] 2021-02-10T04:46:25.184 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:24 smithi164 conmon[29288]: 6.17 scrub ok 2021-02-10T04:46:25.184 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:24 smithi164 conmon[29288]: audit 2021 2021-02-10T04:46:25.184 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:24 smithi164 conmon[29288]: -02-10T04:46:23.919484+0000 mon.a (mon. 2021-02-10T04:46:25.184 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:24 smithi164 conmon[29288]: 0) 2105 : audit [DBG] from='client.? 172.21.15.85:0/278075457' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-02-10T04:46:25.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:24 smithi085 conmon[29319]: cluster 2021-02-10T04:46:22.990479+0000 2021-02-10T04:46:25.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:24 smithi085 conmon[29319]: osd.5 (osd.5) 108 : cluster [DBG] 5.12 scrub ok 2021-02-10T04:46:25.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:24 smithi085 conmon[29319]: cluster 2021-02-10T04:46:23.691470+0000 osd.0 (osd.0) 119 : cluster [DBG] 2.21 scrub ok 2021-02-10T04:46:25.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:24 smithi085 conmon[29319]: cluster 2021-02-10T04:46:23.812322+0000 osd.4 (osd.4) 1947 : cluster [DBG] 6.17 scrub ok 2021-02-10T04:46:25.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:24 smithi085 conmon[29319]: audit 2021-02-10T04:46:23.919484+0000 mon.a (mon.0) 2105 : audit [DBG] from='client.? 172.21.15.85:0/278075457' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-02-10T04:46:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:24 smithi085 conmon[35961]: cluster 2021-02-10T04:46:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:24 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:46:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:24 smithi085 conmon[35961]: 46:22.990479+0000 osd.5 (osd.5) 108 : cluster [DBG] 5.12 scrub ok 2021-02-10T04:46:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:24 smithi085 conmon[35961]: cluster 2021-02-10T04:46:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:24 smithi085 conmon[35961]: 2021-02-10T04:46:23.691470+0000 osd.0 ( 2021-02-10T04:46:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:24 smithi085 conmon[35961]: osd.0) 119 : cluster [DBG] 2.21 scrub ok 2021-02-10T04:46:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:24 smithi085 conmon[35961]: cluster 2021-02-10T04:46:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:24 smithi085 conmon[35961]: 2021-02-10T04:46:23.812322+0000 osd.4 (osd.4) 1947 : cluster 2021-02-10T04:46:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:24 smithi085 conmon[35961]: [DBG] 6.17 scrub ok 2021-02-10T04:46:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:24 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:46:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:24 smithi085 conmon[35961]: T04:46:23.919484+0000 mon.a (mon. 2021-02-10T04:46:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:24 smithi085 conmon[35961]: 0) 2105 : audit [DBG] from='client.? 172.21.15.85:0/278075457' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-02-10T04:46:25.220 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:46:25.635 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:46:26.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:25 smithi164 conmon[29288]: cluster 2021-02-10T04:46:24.021548+0000 osd.6 2021-02-10T04:46:26.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:25 smithi164 conmon[29288]: (osd.6) 108 : cluster [DBG] 5.2 scrub ok 2021-02-10T04:46:26.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:25 smithi164 conmon[29288]: cluster 2021-02-10T04:46:24.319293+0000 mgr.y (mgr.14140) 1016 : cluster [DBG] pgmap v1785: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:46:26.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:25 smithi164 conmon[29288]: cluster 2021-02-10T04:46:24.697782+0000 osd.0 (osd.0) 120 : cluster [DBG] 6.18 scrub ok 2021-02-10T04:46:26.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:25 smithi085 conmon[29319]: cluster 2021-02-10T04:46:24.021548+0000 2021-02-10T04:46:26.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:25 smithi085 conmon[29319]: osd.6 (osd.6) 108 : cluster [DBG] 5.2 scrub ok 2021-02-10T04:46:26.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:25 smithi085 conmon[29319]: cluster 2021-02-10T04:46:24.319293+0000 mgr.y (mgr 2021-02-10T04:46:26.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:25 smithi085 conmon[29319]: .14140) 1016 : cluster [DBG] pgmap v1785: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:46:26.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:25 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:46:26.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:25 smithi085 conmon[29319]: :46:24.697782+0000 osd.0 (osd.0) 2021-02-10T04:46:26.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:25 smithi085 conmon[29319]: 120 : cluster [DBG] 6.18 scrub ok 2021-02-10T04:46:26.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:25 smithi085 conmon[35961]: cluster 2021-02-10T04:46:24.021548+0000 osd.6 (osd.6 2021-02-10T04:46:26.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:25 smithi085 conmon[35961]: ) 108 : cluster [DBG] 5.2 scrub ok 2021-02-10T04:46:26.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:25 smithi085 conmon[35961]: cluster 2021-02-10T04:46:24.319293+0000 mgr.y ( 2021-02-10T04:46:26.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:25 smithi085 conmon[35961]: mgr.14140) 1016 : cluster [DBG] pgmap v1785: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:46:26.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:25 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:46:26.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:25 smithi085 conmon[35961]: :46:24.697782+0000 osd.0 (osd.0) 120 2021-02-10T04:46:26.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:25 smithi085 conmon[35961]: : cluster [DBG] 6.18 scrub ok 2021-02-10T04:46:27.081 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset nodeep-scrub 2021-02-10T04:46:27.082 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:46:27.118 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:26 smithi085 conmon[35961]: cluster 2021-02-10T04:46:27.118 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:26 smithi085 conmon[35961]: 2021-02-10T04:46:27.118 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:26 smithi085 conmon[35961]: 2021 2021-02-10T04:46:27.118 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:26 smithi085 conmon[35961]: - 2021-02-10T04:46:27.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:26 smithi085 conmon[35961]: 02 2021-02-10T04:46:27.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:26 smithi085 conmon[35961]: -10T04:46: 2021-02-10T04:46:27.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:26 smithi085 conmon[35961]: 25.711931+0000 osd.0 (osd.0) 121 : cluster [DBG] 5.b scrub ok 2021-02-10T04:46:27.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:26 smithi085 conmon[35961]: cluster 2021-02-10T04:46:27.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:26 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:46:27.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:26 smithi085 conmon[35961]: 04:46:25.888221+0000 osd.4 (osd.4) 1948 : cluster 2021-02-10T04:46:27.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:26 smithi085 conmon[35961]: [DBG] 5.1d scrub ok 2021-02-10T04:46:27.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:26 smithi085 conmon[35961]: cluster 2021-02-10T04:46:27.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:26 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:46:27.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:26 smithi085 conmon[35961]: 04:46:25.979210+0000 osd.5 (osd 2021-02-10T04:46:27.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:26 smithi085 conmon[35961]: .5) 109 : cluster [DBG] 1.1 scrub ok 2021-02-10T04:46:27.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:26 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:46:27.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:26 smithi085 conmon[35961]: -10T04:46:26.576956+0000 osd.3 (osd. 2021-02-10T04:46:27.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:26 smithi085 conmon[35961]: 3) 110 : cluster [DBG] 6.1f scrub ok 2021-02-10T04:46:27.120 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:26 smithi085 conmon[29319]: cluster 2021-02-10T04:46:25.711931+0000 osd.0 (osd.0) 121 : cluster [DBG] 5.b scrub ok 2021-02-10T04:46:27.120 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:26 smithi085 conmon[29319]: cluster 2021-02-10T04:46:25.888221+0000 osd.4 (osd.4) 1948 : cluster [DBG] 5.1d scrub ok 2021-02-10T04:46:27.120 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:26 smithi085 conmon[29319]: cluster 2021-02-10T04:46:25.979210+0000 osd.5 (osd 2021-02-10T04:46:27.120 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:26 smithi085 conmon[29319]: .5) 109 : cluster [DBG] 1.1 scrub ok 2021-02-10T04:46:27.120 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:26 smithi085 conmon[29319]: cluster 2021-02-10T04:46:26.576956+0000 osd.3 (osd.3) 110 : cluster [DBG] 6.1f scrub ok 2021-02-10T04:46:27.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:26 smithi164 conmon[29288]: cluster 2021-02-10T04:46:25.711931+0000 osd.0 (osd 2021-02-10T04:46:27.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:26 smithi164 conmon[29288]: .0) 121 : cluster [DBG] 5.b scrub ok 2021-02-10T04:46:27.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:26 smithi164 conmon[29288]: cluster 2021-02-10T04:46:25.888221+0000 osd.4 (osd.4) 1948 : cluster [DBG] 2021-02-10T04:46:27.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:26 smithi164 conmon[29288]: 5.1d scrub ok 2021-02-10T04:46:27.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:26 smithi164 conmon[29288]: cluster 2021-02-10T04:46:25 2021-02-10T04:46:27.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:26 smithi164 conmon[29288]: .979210+0000 osd.5 (osd.5) 109 : cluster [DBG] 1.1 scrub ok 2021-02-10T04:46:27.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:26 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:46:27.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:26 smithi164 conmon[29288]: 04:46:26.576956+0000 osd.3 (osd.3) 110 2021-02-10T04:46:27.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:26 smithi164 conmon[29288]: : cluster [DBG] 6.1f scrub ok 2021-02-10T04:46:27.676 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:46:27.677 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:46:28.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:27 smithi164 conmon[33677]: debug 2021-02-10T04:46:27.747+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:46:28.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:27 smithi164 conmon[29288]: cluster 2021-02-10T04:46:28.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:27 smithi164 conmon[29288]: 2021-02-10T04:46:28.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:27 smithi164 conmon[29288]: 2021 2021-02-10T04:46:28.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:27 smithi164 conmon[29288]: -02-10T04:46:25.951584+0000 osd.6 (osd.6 2021-02-10T04:46:28.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:27 smithi164 conmon[29288]: ) 109 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:46:28.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:27 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:46:28.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:27 smithi164 conmon[29288]: 10T04:46:25.955551+0000 osd.6 (osd.6 2021-02-10T04:46:28.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:27 smithi164 conmon[29288]: ) 110 : cluster [DBG] 2021-02-10T04:46:28.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:27 smithi164 conmon[29288]: purged_snaps scrub ok 2021-02-10T04:46:28.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:27 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:46:28.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:27 smithi164 conmon[29288]: 02-10T04:46:26.319984+0000 2021-02-10T04:46:28.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:27 smithi164 conmon[29288]: mgr.y 2021-02-10T04:46:28.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:27 smithi164 conmon[29288]: ( 2021-02-10T04:46:28.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:27 smithi164 conmon[29288]: mgr.14140) 1017 : cluster [DBG] 2021-02-10T04:46:28.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:27 smithi164 conmon[29288]: pgmap v1786: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 962 B/s rd, 0 op/s 2021-02-10T04:46:28.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:27 smithi164 conmon[29288]: cluster 2021-02-10T04:46:28.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:27 smithi164 conmon[29288]: 2021-02-10T04:46:26 2021-02-10T04:46:28.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:27 smithi164 conmon[29288]: .743243+0000 mon.a (mon.0) 2106 : cluster 2021-02-10T04:46:28.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:27 smithi164 conmon[29288]: [DBG] osdmap e1053: 8 total, 8 up, 7 in 2021-02-10T04:46:28.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:27 smithi164 conmon[29288]: cluster 2021-02-10T04:46:28.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:27 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:46:28.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:27 smithi164 conmon[29288]: :46:26.934580+0000 osd.5 (osd.5) 110 : cluster [DBG] 5.4 scrub ok 2021-02-10T04:46:28.151 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:46:28.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[29319]: cluster 2021-02-10T04:46:25.951584+0000 osd.6 ( 2021-02-10T04:46:28.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[29319]: osd.6) 109 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:46:28.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[29319]: cluster 2021-02-10T04:46:25.955551+0000 osd.6 (osd.6) 110 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:46:28.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[29319]: cluster 2021-02-10T04:46:26.319984+0000 mgr.y (mgr.14140) 1017 : cluster [DBG] pgmap v1786: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 962 B/s rd, 0 op/s 2021-02-10T04:46:28.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[29319]: cluster 2021-02-10T04:46:26.743243+0000 mon.a (mon.0) 2106 : cluster [DBG] osdmap e1053: 8 total, 8 up, 7 in 2021-02-10T04:46:28.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[29319]: cluster 2021-02-10T04:46:26.934580+0000 osd.5 (osd.5) 110 : cluster [DBG] 5.4 scrub ok 2021-02-10T04:46:28.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[35961]: cluster 2021-02-10T04:46:25.951584+0000 2021-02-10T04:46:28.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[35961]: osd.6 (osd.6) 109 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:46:28.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:46:28.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[35961]: 10T04:46:25.955551+0000 osd.6 (osd.6) 110 : cluster [DBG] 2021-02-10T04:46:28.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:46:28.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:46:28.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[35961]: -10T04:46:26.319984+0000 mgr.y (mgr.14140) 1017 : cluster [DBG] pgmap v1786: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 962 B/s rd, 0 op/s 2021-02-10T04:46:28.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[35961]: 2021-02-10T04:46:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:46:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[35961]: 02-10T04:46:26.743243+0000 mon.a 2021-02-10T04:46:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[35961]: (mon.0) 2106 : cluster [DBG] osdmap e1053: 8 total, 8 up, 7 in 2021-02-10T04:46:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[35961]: cluster 2021-02-10T04:46:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[35961]: 2021-02-10T04:46:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[35961]: 2021 2021-02-10T04:46:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[35961]: - 2021-02-10T04:46:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[35961]: 02 2021-02-10T04:46:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[35961]: - 2021-02-10T04:46:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[35961]: 10 2021-02-10T04:46:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[35961]: T 2021-02-10T04:46:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[35961]: 04 2021-02-10T04:46:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[35961]: : 2021-02-10T04:46:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[35961]: 46 2021-02-10T04:46:28.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[35961]: : 2021-02-10T04:46:28.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[35961]: 26.934580+0000 osd.5 ( 2021-02-10T04:46:28.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:27 smithi085 conmon[35961]: osd.5) 110 : cluster [DBG] 5.4 scrub ok 2021-02-10T04:46:28.702 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:46:28.823 INFO:tasks.thrashosds.thrasher:j is {'epoch': 1052, 'fsid': '42224fce-6b57-11eb-8feb-001a4aab830c', 'created': '2021-02-10T04:20:57.933279+0000', 'modified': '2021-02-10T04:46:15.671873+0000', 'last_up_change': '2021-02-10T04:25:35.312604+0000', 'last_in_change': '2021-02-10T04:37:36.200492+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-02-10T04:24:05.258028+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '110', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2021-02-10T04:25:32.688554+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 42, 'pg_placement_num': 42, 'pg_placement_num_target': 42, 'pg_num_target': 42, 'pg_num_pending': 42, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '89', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2021-02-10T04:25:47.658613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'zone.rgw.log', 'create_time': '2021-02-10T04:25:47.968727+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'zone.rgw.control', 'create_time': '2021-02-10T04:25:49.278260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'zone.rgw.meta', 'create_time': '2021-02-10T04:25:51.792514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 42, 'pg_placement_num': 42, 'pg_placement_num_target': 42, 'pg_num_target': 42, 'pg_num_pending': 42, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1046', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '1044', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 8}, 'application_metadata': {'rgw': {}}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2021-02-10T04:28:23.932391+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1009', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '1007', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 888, 'snap_epoch': 994, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '45d0bb5f-35fa-4596-964f-75abcee53bee', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 1046, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6801', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6802', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6804', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6803', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6801/7', 'cluster_addr': '172.21.15.85:6802/7', 'heartbeat_back_addr': '172.21.15.85:6804/7', 'heartbeat_front_addr': '172.21.15.85:6803/7', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '3421a5ce-872e-4b52-adf7-6aaa595871a4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 1039, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6805', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6806', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6808', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6807', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6805/7', 'cluster_addr': '172.21.15.85:6806/7', 'heartbeat_back_addr': '172.21.15.85:6808/7', 'heartbeat_front_addr': '172.21.15.85:6807/7', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '96e1952a-1042-4316-9fc9-86813e88b912', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 367, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6809', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6810', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6812', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6811', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6809/7', 'cluster_addr': '172.21.15.85:6810/7', 'heartbeat_back_addr': '172.21.15.85:6812/7', 'heartbeat_front_addr': '172.21.15.85:6811/7', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'e019f49f-5dd4-4104-9ced-94c710616642', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 1044, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6813', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6814', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6816', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6815', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6813/7', 'cluster_addr': '172.21.15.85:6814/7', 'heartbeat_back_addr': '172.21.15.85:6816/7', 'heartbeat_front_addr': '172.21.15.85:6815/7', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '91c8a80a-37b6-4c30-a32a-647b2e6d801f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 1046, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6800', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6801', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6803', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6802', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6800/7', 'cluster_addr': '172.21.15.164:6801/7', 'heartbeat_back_addr': '172.21.15.164:6803/7', 'heartbeat_front_addr': '172.21.15.164:6802/7', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 1046, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6804', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6805', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6807', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6806', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6804/7', 'cluster_addr': '172.21.15.164:6805/7', 'heartbeat_back_addr': '172.21.15.164:6807/7', 'heartbeat_front_addr': '172.21.15.164:6806/7', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'adc40430-3eeb-4003-a824-343021243077', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 43, 'up_thru': 1046, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6808', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6809', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6811', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6810', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6808/7', 'cluster_addr': '172.21.15.164:6809/7', 'heartbeat_back_addr': '172.21.15.164:6811/7', 'heartbeat_front_addr': '172.21.15.164:6810/7', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 112, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6812', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6813', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6815', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6814', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6812/7', 'cluster_addr': '172.21.15.164:6813/7', 'heartbeat_back_addr': '172.21.15.164:6815/7', 'heartbeat_front_addr': '172.21.15.164:6814/7', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:45:36.223283+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:45:55.620154+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 65536, 'last_purged_snaps_scrub': '2021-02-10T04:46:14.971471+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:45:32.374158+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:45:49.759075+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:46:07.417184+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:45:25.141031+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:45:43.880389+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.7', 'mappings': [{'from': 7, 'to': 4}]}, {'pgid': '2.21', 'mappings': [{'from': 7, 'to': 1}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 6}, {'from': 3, 'to': 7}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.10', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '5.6', 'mappings': [{'from': 5, 'to': 6}]}, {'pgid': '5.8', 'mappings': [{'from': 1, 'to': 3}]}, {'pgid': '5.10', 'mappings': [{'from': 7, 'to': 3}]}, {'pgid': '5.18', 'mappings': [{'from': 4, 'to': 3}]}, {'pgid': '5.1b', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '6.1c', 'mappings': [{'from': 3, 'to': 0}]}, {'pgid': '7.6', 'mappings': [{'from': 3, 'to': 1}]}, {'pgid': '7.10', 'mappings': [{'from': 7, 'to': 5}]}, {'pgid': '7.12', 'mappings': [{'from': 7, 'to': 5}]}, {'pgid': '7.15', 'mappings': [{'from': 7, 'to': 4}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.85:0/801393150': '2021-02-11T04:21:37.851607+0000', '172.21.15.85:0/3049671696': '2021-02-11T04:21:37.851607+0000', '172.21.15.85:0/4056505602': '2021-02-11T04:21:20.700478+0000', '172.21.15.85:6800/7': '2021-02-11T04:21:20.700478+0000', '172.21.15.85:0/1879026062': '2021-02-11T04:21:20.700478+0000'}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2021-02-10T04:46:28.823 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:46:28.894 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:46:28 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:46:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:28 smithi164 conmon[29288]: cluster 2021-02-10T04:46:27.748216 2021-02-10T04:46:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:28 smithi164 conmon[29288]: +0000 osd.0 (osd.0) 122 : cluster [DBG] 5.3 scrub ok 2021-02-10T04:46:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:28 smithi164 conmon[29288]: cluster 2021-02-10T04:46 2021-02-10T04:46:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:28 smithi164 conmon[29288]: :27.889128+0000 osd.4 (osd.4) 1949 : cluster [DBG] 6.14 scrub ok 2021-02-10T04:46:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:28 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:46:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:28 smithi164 conmon[29288]: 02-10T04:46:28.515507+0000 osd.3 ( 2021-02-10T04:46:28.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:28 smithi164 conmon[29288]: osd.3) 111 : cluster [DBG] 4.f scrub ok 2021-02-10T04:46:29.009 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:28 smithi085 conmon[29319]: cluster 2021-02-10T04:46:27.748216 2021-02-10T04:46:29.009 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:28 smithi085 conmon[29319]: +0000 osd.0 (osd.0) 122 : cluster [DBG] 5.3 scrub ok 2021-02-10T04:46:29.009 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:28 smithi085 conmon[29319]: cluster 2021-02-10T04:46:27.889128+0000 osd.4 (osd.4) 1949 : cluster [DBG] 2021-02-10T04:46:29.009 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:28 smithi085 conmon[29319]: 6.14 scrub ok 2021-02-10T04:46:29.009 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:28 smithi085 conmon[29319]: cluster 2021-02-10T04:46:28.515507 2021-02-10T04:46:29.009 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:28 smithi085 conmon[29319]: +0000 osd.3 (osd.3) 111 : cluster [DBG] 4.f scrub ok 2021-02-10T04:46:29.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:28 smithi085 conmon[35961]: cluster 2021-02-10T04:46:27.748216+0000 osd.0 2021-02-10T04:46:29.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:28 smithi085 conmon[35961]: (osd.0) 122 : cluster [DBG] 5.3 scrub ok 2021-02-10T04:46:29.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:28 smithi085 conmon[35961]: cluster 2021-02-10T04:46:29.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:28 smithi085 conmon[35961]: 2021 2021-02-10T04:46:29.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:28 smithi085 conmon[35961]: - 2021-02-10T04:46:29.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:28 smithi085 conmon[35961]: 02-10T04:46:27.889128 2021-02-10T04:46:29.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:28 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1949 : cluster [DBG] 6.14 scrub ok 2021-02-10T04:46:29.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:28 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:46:29.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:28 smithi085 conmon[35961]: 10T04:46:28.515507+0000 osd.3 (osd.3) 111 : cluster 2021-02-10T04:46:29.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:28 smithi085 conmon[35961]: [DBG] 4.f scrub ok 2021-02-10T04:46:29.155 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:46:29.200 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:46:29.462 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:29 smithi164 conmon[33677]: debug 2021-02-10T04:46:29.328+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:46:29.532 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:46:29.638 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:46:29.671 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:46:29.753 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:29 smithi164 conmon[29288]: cluster 2021-02-10T04:46:28.320702+0000 mgr.y ( 2021-02-10T04:46:29.753 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:29 smithi164 conmon[29288]: mgr.14140) 1018 : cluster [DBG] pgmap v1788: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:46:29.753 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:29 smithi164 conmon[29288]: cluster 2021-02-10T04:46:28.715362+0000 osd.0 (osd.0) 123 : cluster [DBG] 2.10 scrub ok 2021-02-10T04:46:29.753 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:29 smithi164 conmon[29288]: audit 2021-02-10T04:46:28.763063+0000 mgr.y (mgr.14140) 1019 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:46:29.754 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:29 smithi164 conmon[29288]: cluster 2021-02-10T04:46:29.097644+0000 osd.6 (osd.6) 111 : cluster [DBG] 5.9 scrub ok 2021-02-10T04:46:29.790 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:46:29.896 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:46:29.928 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:29 smithi085 conmon[35961]: cluster 2021-02-10T04:46:28.320702+0000 mgr.y (mgr.14140) 1018 : cluster [DBG] pgmap v1788: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:46:29.928 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:29 smithi085 conmon[35961]: cluster 2021-02-10T04:46:28.715362+0000 osd.0 (osd.0) 123 : cluster [DBG] 2.10 scrub ok 2021-02-10T04:46:29.928 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:29 smithi085 conmon[35961]: audit 2021-02-10T04:46:28.763063+0000 mgr.y (mgr.14140) 1019 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:46:29.928 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:29 smithi085 conmon[35961]: cluster 2021-02-10T04:46:29.097644+0000 osd.6 (osd.6) 111 : cluster [DBG] 5.9 scrub ok 2021-02-10T04:46:29.929 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:29 smithi085 conmon[29319]: cluster 2021-02-10T04:46:28.320702+0000 mgr.y (mgr.14140) 1018 : cluster [DBG] pgmap v1788: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:46:29.929 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:29 smithi085 conmon[29319]: cluster 2021-02-10T04:46:28.715362+0000 osd.0 (osd.0) 123 : cluster [DBG] 2.10 scrub ok 2021-02-10T04:46:29.929 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:29 smithi085 conmon[29319]: audit 2021-02-10T04:46:28.763063+0000 mgr.y (mgr.14140) 1019 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:46:29.929 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:29 smithi085 conmon[29319]: cluster 2021-02-10T04:46:29.097644+0000 osd.6 (osd.6) 111 : cluster [DBG] 5.9 scrub ok 2021-02-10T04:46:29.932 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:46:30.084 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:29 smithi164 conmon[33677]: debug 2021-02-10T04:46:30.085 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:29 smithi164 conmon[33677]: 2021-02-10T04:46:29.752+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:46:31.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:30 smithi164 conmon[29288]: cluster 2021-02-10T04:46:29.672137+0000 osd.0 2021-02-10T04:46:31.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:30 smithi164 conmon[29288]: (osd.0) 124 : cluster [DBG] 4.3 scrub ok 2021-02-10T04:46:31.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:30 smithi164 conmon[29288]: cluster 2021-02-10T04:46:30.077961+0000 osd.6 (osd.6) 112 : cluster [DBG] 7.d scrub ok 2021-02-10T04:46:31.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:30 smithi164 conmon[29288]: cluster 2021-02-10T04:46:30.515019+0000 osd.3 (osd.3) 112 : cluster [DBG] 4.0 scrub ok 2021-02-10T04:46:31.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:30 smithi085 conmon[29319]: cluster 2021-02-10T04:46:31.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:30 smithi085 conmon[29319]: 2021 2021-02-10T04:46:31.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:30 smithi085 conmon[29319]: -02-10T04:46:29.672137+0000 osd.0 (osd.0) 2021-02-10T04:46:31.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:30 smithi085 conmon[29319]: 124 : cluster [DBG] 4.3 scrub ok 2021-02-10T04:46:31.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:30 smithi085 conmon[29319]: cluster 2021-02-10T04:46:31.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:30 smithi085 conmon[29319]: 2021-02-10T04:46:31.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:30 smithi085 conmon[29319]: 2021-02-10T04:46:30.077961 2021-02-10T04:46:31.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:30 smithi085 conmon[29319]: +0000 osd.6 (osd.6) 112 : cluster [DBG] 7.d scrub ok 2021-02-10T04:46:31.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:30 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:46:31.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:30 smithi085 conmon[29319]: 02-10T04:46:30.515019+0000 osd.3 (osd.3) 112 : cluster [DBG] 4.0 scrub ok 2021-02-10T04:46:31.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:30 smithi085 conmon[29319]: 2021-02-10T04:46:31.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:30 smithi085 conmon[35961]: cluster 2021-02-10T04:46:31.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:30 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:46:31.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:30 smithi085 conmon[35961]: 46:29.672137+0000 osd.0 (osd.0) 124 : cluster [DBG] 4.3 scrub ok 2021-02-10T04:46:31.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:30 smithi085 conmon[35961]: cluster 2021-02-10T04:46:31.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:30 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:46:31.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:30 smithi085 conmon[35961]: 10T04:46:30.077961+0000 osd.6 (osd.6) 112 : cluster [DBG] 7.d scrub ok 2021-02-10T04:46:31.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:30 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:46:31.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:30 smithi085 conmon[35961]: -02-10T04:46:30.515019+0000 2021-02-10T04:46:31.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:30 smithi085 conmon[35961]: osd.3 (osd.3) 112 : cluster [DBG] 4.0 scrub ok 2021-02-10T04:46:31.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:30 smithi085 conmon[35961]: 2021-02-10T04:46:31.760 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:31 smithi085 conmon[29319]: cluster 2021-02-10T04:46:30.321360+0000 mgr.y (mgr.14140) 1020 : cluster [DBG] pgmap v1789: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:46:31.760 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:31 smithi085 conmon[29319]: 2021-02-10T04:46:31.760 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:31 smithi085 conmon[29319]: cluster 2021-02-10T04:46:30.848041+0000 osd.4 (osd.4) 1950 : cluster [DBG] 1.5 scrub ok 2021-02-10T04:46:32.034 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:31 smithi085 conmon[35961]: cluster 2021-02-10T04:46: 2021-02-10T04:46:32.034 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:31 smithi085 conmon[35961]: 30.321360+0000 mgr.y (mgr.14140) 1020 : cluster [DBG] pgmap v1789: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:46:32.034 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:31 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:46:32.034 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:31 smithi085 conmon[35961]: 46:30.848041+0000 osd.4 (osd.4) 1950 : 2021-02-10T04:46:32.034 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:31 smithi085 conmon[35961]: cluster [DBG] 1.5 scrub ok 2021-02-10T04:46:32.034 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:31 smithi085 conmon[35961]: cluster 2021-02-10T04:46:32.034 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:31 smithi085 conmon[35961]: 2021-02-10T04:46:31.043531+0000 osd.5 (osd.5 2021-02-10T04:46:32.034 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:31 smithi085 conmon[35961]: ) 111 : cluster [DBG] 1.a scrub ok 2021-02-10T04:46:32.034 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:31 smithi085 conmon[29319]: cluster 2021-02-10T04:46:32.035 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:31 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:46:32.035 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:31 smithi085 conmon[29319]: 10T04:46:31.043531+0000 osd.5 (osd.5) 111 2021-02-10T04:46:32.035 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:31 smithi085 conmon[29319]: : cluster [DBG] 1.a scrub ok 2021-02-10T04:46:32.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:31 smithi164 conmon[29288]: cluster 2021-02-10T04:46:30.321360+0000 mgr.y (mgr.14140) 1020 : cluster [DBG] pgmap v1789: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:46:32.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:31 smithi164 conmon[29288]: cluster 2021-02-10T04:46:30.848041+0000 osd.4 (osd.4) 1950 : cluster [DBG] 1.5 scrub ok 2021-02-10T04:46:32.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:31 smithi164 conmon[29288]: cluster 2021-02-10T04:46:31.043531+0000 osd.5 (osd.5) 111 : cluster [DBG] 1.a scrub ok 2021-02-10T04:46:32.441 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:46:32 smithi085 conmon[52836]: debug 2021-02-10T04:46:32.441 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:46:32 smithi085 conmon[52836]: 2021-02-10T04:46:32.032+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:46:32.521 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:46:33.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:32 smithi164 conmon[29288]: cluster 2021-02-10T04:46:31.614693+0000 osd.0 (osd.0) 125 : 2021-02-10T04:46:33.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:32 smithi164 conmon[29288]: cluster [DBG] 4.b scrub ok 2021-02-10T04:46:33.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:32 smithi164 conmon[29288]: cluster 2021-02-10T04:46:32.031094+0000 osd.6 (osd.6) 113 : cluster [DBG] 2.22 scrub ok 2021-02-10T04:46:33.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:32 smithi085 conmon[35961]: cluster 2021-02-10T04:46:33.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:32 smithi085 conmon[35961]: 2021-02-10T04:46:33.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:32 smithi085 conmon[35961]: 2021 2021-02-10T04:46:33.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:32 smithi085 conmon[35961]: - 2021-02-10T04:46:33.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:32 smithi085 conmon[35961]: 02-10T04:46:31. 2021-02-10T04:46:33.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:32 smithi085 conmon[35961]: 614693+0000 osd.0 (osd.0) 125 : cluster [DBG] 4.b scrub ok 2021-02-10T04:46:33.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:32 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:46:33.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:32 smithi085 conmon[35961]: 02-10T04:46:32.031094+0000 osd.6 (osd 2021-02-10T04:46:33.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:32 smithi085 conmon[35961]: .6) 113 : cluster [DBG] 2.22 scrub ok 2021-02-10T04:46:33.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:32 smithi085 conmon[29319]: cluster 2021-02-10T04:46:33.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:32 smithi085 conmon[29319]: 2021-02-10T04:46:31 2021-02-10T04:46:33.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:32 smithi085 conmon[29319]: .614693+0000 osd.0 (osd.0) 125 : cluster [DBG] 4.b scrub ok 2021-02-10T04:46:33.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:32 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:46:33.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:32 smithi085 conmon[29319]: -02-10T04:46: 2021-02-10T04:46:33.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:32 smithi085 conmon[29319]: 32.031094+0000 osd.6 (osd.6 2021-02-10T04:46:33.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:32 smithi085 conmon[29319]: ) 113 : cluster [DBG] 2.22 scrub ok 2021-02-10T04:46:34.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:33 smithi164 conmon[29288]: cluster 2021-02-10T04:46:32.321956+0000 mgr.y (mgr. 2021-02-10T04:46:36.046 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:33 smithi164 conmon[29288]: 14140) 1021 : cluster [DBG] pgmap v1790: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:46:36.046 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:33 smithi164 conmon[29288]: cluster 2021-02-10T04:46:33.323390+0000 osd.3 (osd.3) 113 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:46:36.046 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:33 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:46:36.047 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:33 smithi164 conmon[29288]: -10T04:46:33.328018+0000 osd.3 (osd.3) 114 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:46:36.047 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:33 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:46:36.047 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:33 smithi164 conmon[29288]: 02-10T04:46:33.543553+0000 osd.3 (osd. 2021-02-10T04:46:36.047 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:33 smithi164 conmon[29288]: 3) 115 : cluster [DBG] 2.1e scrub ok 2021-02-10T04:46:36.048 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:33 smithi085 conmon[29319]: cluster 2021-02-10T04:46:32.321956+0000 mgr.y (mgr.14140) 1021 : cluster [DBG] pgmap v1790: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:46:36.048 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:33 smithi085 conmon[29319]: 2021-02-10T04:46:36.048 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:33 smithi085 conmon[29319]: cluster 2021-02-10T04:46:33.323390+0000 osd.3 (osd.3) 113 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:46:36.048 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:33 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:46:36.048 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:33 smithi085 conmon[29319]: :46:33.328018+0000 2021-02-10T04:46:36.048 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:33 smithi085 conmon[29319]: osd.3 (osd.3) 114 : 2021-02-10T04:46:36.048 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:33 smithi085 conmon[29319]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:46:36.048 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:33 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:46:36.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:33 smithi085 conmon[29319]: 02-10T04:46:33. 2021-02-10T04:46:36.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:33 smithi085 conmon[29319]: 543553+0000 osd.3 (osd.3) 2021-02-10T04:46:36.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:33 smithi085 conmon[29319]: 115 : cluster [DBG] 2.1e scrub ok 2021-02-10T04:46:36.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[29319]: cluster 2021-02-10T04:46:36.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[29319]: 2021-02-10T04:46:36.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[29319]: 2021 2021-02-10T04:46:36.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[29319]: -02-10T04:46:33.677432 2021-02-10T04:46:36.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[29319]: +0000 osd.0 (osd.0) 126 : cluster [DBG] 6.16 scrub ok 2021-02-10T04:46:36.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:46:36.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[29319]: :46:33.775546+0000 mon.a (mon.0) 2107 : 2021-02-10T04:46:36.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[29319]: cluster [DBG] osdmap e1054: 8 total, 8 up, 7 in 2021-02-10T04:46:36.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:46:36.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[29319]: :46:33.962998+0000 osd.5 (osd.5) 112 : 2021-02-10T04:46:36.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[29319]: cluster [DBG] 1.8 scrub ok 2021-02-10T04:46:36.050 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:46:36.050 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[29319]: 46:34.039264+0000 osd.6 (osd.6) 114 : 2021-02-10T04:46:36.050 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[29319]: cluster [DBG] 4.5 scrub ok 2021-02-10T04:46:36.050 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:46:36.050 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[29319]: 46:34.577679+0000 osd.3 (osd.3) 2021-02-10T04:46:36.050 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[29319]: 116 : cluster [DBG] 2.e scrub ok 2021-02-10T04:46:36.050 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:33 smithi085 conmon[35961]: cluster 2021-02-10T04:46: 2021-02-10T04:46:36.050 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:33 smithi085 conmon[35961]: 32.321956+0000 mgr.y (mgr.14140) 1021 : cluster [DBG] pgmap v1790: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:46:36.050 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:33 smithi085 conmon[35961]: cluster 2021-02-10T04:46:36.050 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:33 smithi085 conmon[35961]: 2021-02-10T04:46:33.323390+0000 2021-02-10T04:46:36.050 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:33 smithi085 conmon[35961]: osd.3 (osd.3) 113 : cluster [DBG] 2021-02-10T04:46:36.051 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:33 smithi085 conmon[35961]: purged_snaps scrub starts 2021-02-10T04:46:36.051 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:33 smithi085 conmon[35961]: cluster 2021-02-10T04:46:36.051 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:33 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:46:36.051 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:33 smithi085 conmon[35961]: 04:46:33.328018+0000 osd.3 2021-02-10T04:46:36.051 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:33 smithi085 conmon[35961]: (osd.3) 114 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:46:36.051 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:33 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:46:36.051 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:33 smithi085 conmon[35961]: -02-10T04:46:33.543553+0000 osd.3 2021-02-10T04:46:36.051 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:33 smithi085 conmon[35961]: (osd.3) 115 : cluster 2021-02-10T04:46:36.051 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:33 smithi085 conmon[35961]: [DBG] 2.1e scrub ok 2021-02-10T04:46:36.051 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[35961]: cluster 2021-02-10T04:46:33.677432+0000 2021-02-10T04:46:36.051 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[35961]: osd.0 (osd.0) 126 : cluster [DBG] 6.16 scrub ok 2021-02-10T04:46:36.051 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:46:36.051 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[35961]: -10T04:46:33.775546+0000 mon.a (mon.0) 2107 : cluster [DBG] osdmap e1054: 8 total, 8 up, 7 in 2021-02-10T04:46:36.052 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[35961]: 2021-02-10T04:46:36.052 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[35961]: cluster 2021-02-10T04:46: 2021-02-10T04:46:36.052 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[35961]: 33.962998+0000 osd.5 (osd.5) 112 2021-02-10T04:46:36.052 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[35961]: : cluster [DBG] 1.8 scrub ok 2021-02-10T04:46:36.052 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:46:36.052 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[35961]: -02-10T04:46:34.039264+0000 2021-02-10T04:46:36.052 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[35961]: osd.6 (osd.6) 114 : cluster [DBG] 2021-02-10T04:46:36.052 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[35961]: 4.5 scrub ok 2021-02-10T04:46:36.052 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:46:36.052 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[35961]: 04:46:34.577679+0000 osd.3 (osd.3 2021-02-10T04:46:36.052 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:34 smithi085 conmon[35961]: ) 116 : cluster [DBG] 2.e scrub ok 2021-02-10T04:46:36.052 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:34 smithi164 conmon[29288]: cluster 2021-02-10T04:46:36.052 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:34 smithi164 conmon[29288]: 2021-02-10T04:46:33 2021-02-10T04:46:36.053 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:34 smithi164 conmon[29288]: .677432+0000 osd.0 (osd.0) 126 : cluster [DBG] 6.16 scrub ok 2021-02-10T04:46:36.053 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:34 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:46:36.053 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:34 smithi164 conmon[29288]: 46:33.775546+0000 mon.a (mon.0 2021-02-10T04:46:36.054 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:34 smithi164 conmon[29288]: ) 2107 : cluster [DBG] osdmap e1054: 8 total, 8 up, 7 in 2021-02-10T04:46:36.054 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:34 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:46:36.054 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:34 smithi164 conmon[29288]: 02-10T04:46:33.962998+0000 2021-02-10T04:46:36.054 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:34 smithi164 conmon[29288]: osd.5 (osd.5) 112 : cluster [DBG] 1.8 scrub ok 2021-02-10T04:46:36.054 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:34 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:46:36.054 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:34 smithi164 conmon[29288]: 10T04:46:34.039264+0000 osd.6 (osd.6) 114 : cluster [DBG] 4.5 scrub ok 2021-02-10T04:46:36.054 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:34 smithi164 conmon[29288]: 2021-02-10T04:46:36.054 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:34 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:46:36.054 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:34 smithi164 conmon[29288]: -02-10T04:46:34.577679+0000 osd.3 (osd.3) 116 : cluster [DBG] 2021-02-10T04:46:36.054 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:34 smithi164 conmon[29288]: 2.e scrub ok 2021-02-10T04:46:36.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:35 smithi164 conmon[29288]: cluster 2021-02-10T04:46:34.322503+0000 mgr.y (mgr 2021-02-10T04:46:36.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:35 smithi164 conmon[29288]: .14140) 1022 : cluster [DBG] pgmap v1792: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:46:36.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:35 smithi164 conmon[29288]: cluster 2021-02-10T04:46:34.914282+0000 2021-02-10T04:46:36.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:35 smithi164 conmon[29288]: osd.5 (osd.5) 113 : cluster [DBG] 6.1c scrub ok 2021-02-10T04:46:36.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:35 smithi085 conmon[35961]: cluster 2021-02-10T04:46:34. 2021-02-10T04:46:36.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:35 smithi085 conmon[35961]: 322503+0000 mgr.y (mgr.14140) 1022 : cluster [DBG] pgmap v1792: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:46:36.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:35 smithi085 conmon[35961]: cluster 2021-02-10T04:46:36.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:35 smithi085 conmon[35961]: 2021-02-10T04:46:34.914282+0000 osd.5 (osd.5) 113 : cluster 2021-02-10T04:46:36.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:35 smithi085 conmon[35961]: [DBG] 6.1c scrub ok 2021-02-10T04:46:36.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:35 smithi085 conmon[29319]: cluster 2021-02-10T04:46:36.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:35 smithi085 conmon[29319]: 2021-02-10T04:46:34.322503+0000 mgr.y (mgr 2021-02-10T04:46:36.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:35 smithi085 conmon[29319]: .14140) 1022 : cluster [DBG] pgmap v1792: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:46:36.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:35 smithi085 conmon[29319]: cluster 2021-02-10T04:46:34.914282+0000 2021-02-10T04:46:36.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:35 smithi085 conmon[29319]: osd.5 (osd.5) 113 : cluster [DBG] 6.1c scrub ok 2021-02-10T04:46:37.064 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: cluster 2021-02-10T04:46:43.287 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: 2021-02-10T04:46:43.288 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: 2021 2021-02-10T04:46:43.288 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: - 2021-02-10T04:46:43.288 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: 02 2021-02-10T04:46:43.288 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: - 2021-02-10T04:46:43.288 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: 10 2021-02-10T04:46:43.289 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: T 2021-02-10T04:46:43.289 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: 04 2021-02-10T04:46:43.289 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: : 2021-02-10T04:46:43.289 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: 46 2021-02-10T04:46:43.289 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: : 2021-02-10T04:46:43.289 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: 35 2021-02-10T04:46:43.290 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: . 2021-02-10T04:46:43.290 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: 785510 2021-02-10T04:46:43.290 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: +0000 2021-02-10T04:46:43.290 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: 2021-02-10T04:46:43.290 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: osd.4 2021-02-10T04:46:43.291 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: ( 2021-02-10T04:46:43.291 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: osd 2021-02-10T04:46:43.291 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: . 2021-02-10T04:46:43.291 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: 4 2021-02-10T04:46:43.291 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: ) 2021-02-10T04:46:43.291 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: 1951 2021-02-10T04:46:43.291 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: : 2021-02-10T04:46:43.291 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: cluster 2021-02-10T04:46:43.292 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: 2021-02-10T04:46:43.292 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: [DBG] 2021-02-10T04:46:43.292 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: 2021-02-10T04:46:43.292 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: 5.1 scrub ok 2021-02-10T04:46:43.292 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: 2021-02-10T04:46:43.292 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: cluster 2021-02-10T04:46:43.292 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: 2021-02-10T04:46:43.292 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: 2021 2021-02-10T04:46:43.292 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: - 2021-02-10T04:46:43.293 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: 02 2021-02-10T04:46:43.293 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: -10T04:46:36.557033+0000 2021-02-10T04:46:43.293 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:36 smithi164 conmon[29288]: osd.3 (osd.3) 117 : cluster [DBG] 5.0 scrub ok 2021-02-10T04:46:43.296 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:36 smithi085 conmon[35961]: cluster 2021-02-10T04:46: 2021-02-10T04:46:43.296 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:36 smithi085 conmon[35961]: 35.785510+0000 osd.4 (osd.4) 1951 : cluster [DBG] 5.1 scrub ok 2021-02-10T04:46:43.296 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:36 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:46:43.297 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:36 smithi085 conmon[35961]: 04:46:36.557033+0000 osd.3 (osd.3) 2021-02-10T04:46:43.297 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:36 smithi085 conmon[35961]: 117 : cluster [DBG] 5.0 scrub ok 2021-02-10T04:46:43.297 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[35961]: cluster 2021-02-10T04:46:43.297 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[35961]: 2021-02-10T04:46:43.297 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[35961]: 2021-02-10T04:46 2021-02-10T04:46:43.297 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[35961]: :36.323155+0000 mgr.y (mgr.14140) 1023 : cluster [DBG] pgmap v1793: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 961 B/s rd, 0 op/s 2021-02-10T04:46:43.297 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[35961]: cluster 2021-02-10T04:46:36.790480+0000 osd.4 (osd.4) 1952 : cluster 2021-02-10T04:46:43.298 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[35961]: [DBG] 6.1e scrub ok 2021-02-10T04:46:43.298 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:46:43.298 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[35961]: T04:46:36.919201+0000 osd.5 (osd.5) 114 : cluster [DBG] 2021-02-10T04:46:43.298 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[35961]: 4.7 scrub ok 2021-02-10T04:46:43.298 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[35961]: cluster 2021-02-10T04:46:37.022672+0000 2021-02-10T04:46:43.298 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[35961]: osd.0 (osd.0) 127 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:46:43.298 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[35961]: cluster 2021-02-10T04:46:37.026293+0000 2021-02-10T04:46:43.299 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[35961]: osd.0 (osd.0) 128 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:46:43.299 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[35961]: cluster 2021-02-10T04:46:37.153682+0000 osd.6 ( 2021-02-10T04:46:43.299 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[35961]: osd.6) 115 : cluster [DBG] 2.24 scrub ok 2021-02-10T04:46:43.299 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[35961]: cluster 2021-02-10T04:46:37.748254+0000 2021-02-10T04:46:43.299 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[35961]: osd.4 (osd.4) 1953 : cluster [DBG] 6.12 scrub ok 2021-02-10T04:46:43.299 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[35961]: cluster 2021-02-10T04:46:37 2021-02-10T04:46:43.299 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[35961]: .802647+0000 mon.a (mon.0) 2021-02-10T04:46:43.299 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[35961]: 2108 : cluster [DBG] osdmap e1055: 8 total, 8 up, 7 in 2021-02-10T04:46:43.299 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:46:43.299 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[35961]: T04:46:37.897969+0000 osd.5 (osd.5 2021-02-10T04:46:43.299 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[35961]: ) 115 : cluster [DBG] 5.7 scrub ok 2021-02-10T04:46:43.299 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[35961]: audit 2021-02-10T04:46:43.299 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[35961]: 2021-02-10T04:46:38.174462 2021-02-10T04:46:43.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2109 : audit 2021-02-10T04:46:43.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:46:43.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[35961]: audit 2021-02-10T04:46:43.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[35961]: 2021 2021-02-10T04:46:43.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[35961]: -02-10T04:46:38 2021-02-10T04:46:43.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[35961]: .174753+0000 mon.a (mon.0) 2110 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:46:43.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[35961]: cluster 2021-02-10T04:46:43.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[35961]: 2021-02-10T04:46:38.398537+0000 mon.a (mon.0) 2111 : cluster [DBG] osdmap e1056: 8 total, 8 up, 7 in 2021-02-10T04:46:43.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:39 smithi085 conmon[35961]: cluster 2021-02-10T04:46:38.323778+0000 mgr.y ( 2021-02-10T04:46:43.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:39 smithi085 conmon[35961]: mgr.14140) 1024 : cluster [DBG] pgmap v1795: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2021-02-10T04:46:43.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:39 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:46:43.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:39 smithi085 conmon[35961]: T04:46:38.773047+0000 mgr.y (mgr.14140) 1025 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:46:43.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:39 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:46:43.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:39 smithi085 conmon[35961]: 10T04:46:38.774787+0000 osd.0 (osd.0) 129 2021-02-10T04:46:43.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:39 smithi085 conmon[35961]: : cluster [DBG] 7.5 scrub ok 2021-02-10T04:46:43.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:39 smithi085 conmon[35961]: cluster 2021-02-10T04:46 2021-02-10T04:46:43.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:39 smithi085 conmon[35961]: :38.776800+0000 osd.4 (osd.4) 1954 : cluster [DBG] 2021-02-10T04:46:43.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:39 smithi085 conmon[35961]: 4.1 scrub ok 2021-02-10T04:46:43.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:39 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:46:43.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:39 smithi085 conmon[35961]: -02-10T04:46:39.502891+0000 osd.3 (osd.3) 118 : cluster 2021-02-10T04:46:43.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:39 smithi085 conmon[35961]: [DBG] 5.14 scrub ok 2021-02-10T04:46:43.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:41 smithi085 conmon[35961]: cluster 2021-02-10T04:46:40. 2021-02-10T04:46:43.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:41 smithi085 conmon[35961]: 324582+0000 mgr.y (mgr.14140) 1026 : cluster [DBG] pgmap v1797: 188 pgs: 188 active+clean; 858 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:46:43.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:41 smithi085 conmon[35961]: cluster 2021-02-10T04:46: 2021-02-10T04:46:43.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:41 smithi085 conmon[35961]: 40.753574+0000 osd.0 (osd.0) 130 : 2021-02-10T04:46:43.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:41 smithi085 conmon[35961]: cluster [DBG] 7.4 scrub ok 2021-02-10T04:46:43.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:42 smithi085 conmon[35961]: cluster 2021-02-10T04:46:42.217444 2021-02-10T04:46:43.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:42 smithi085 conmon[35961]: +0000 osd.6 (osd.6) 116 : cluster [DBG] 2.14 scrub ok 2021-02-10T04:46:43.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:42 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:46:43.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:42 smithi085 conmon[35961]: T04:46:42.537438+0000 osd.3 (osd.3) 119 : cluster [DBG] 2.19 scrub ok 2021-02-10T04:46:43.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:42 smithi085 conmon[35961]: 2021-02-10T04:46:43.303 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:36 smithi085 conmon[29319]: cluster 2021-02-10T04:46:43.303 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:36 smithi085 conmon[29319]: 2021-02-10T04:46:43.303 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:36 smithi085 conmon[29319]: 2021-02-10T04:46 2021-02-10T04:46:43.303 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:36 smithi085 conmon[29319]: :35.785510+0000 osd.4 (osd.4) 1951 : cluster [DBG] 5.1 scrub ok 2021-02-10T04:46:43.303 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:36 smithi085 conmon[29319]: cluster 2021-02-10T04:46:43.303 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:36 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:46:43.303 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:36 smithi085 conmon[29319]: 04:46:36.557033+0000 osd.3 (osd.3) 117 : cluster 2021-02-10T04:46:43.303 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:36 smithi085 conmon[29319]: [DBG] 5.0 scrub ok 2021-02-10T04:46:43.303 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[29319]: cluster 2021-02-10T04:46:36.323155+0000 2021-02-10T04:46:43.303 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[29319]: mgr.y (mgr.14140) 1023 : cluster [DBG] pgmap v1793: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 961 B/s rd, 0 op/s 2021-02-10T04:46:43.303 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[29319]: cluster 2021-02-10T04:46:36.790480+0000 osd.4 (osd.4) 1952 : cluster [DBG] 2021-02-10T04:46:43.304 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[29319]: 6.1e scrub ok 2021-02-10T04:46:43.304 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:46:43.304 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[29319]: :46:36.919201+0000 osd.5 (osd.5) 2021-02-10T04:46:43.304 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[29319]: 114 : cluster [DBG] 4.7 scrub ok 2021-02-10T04:46:43.304 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:46:43.304 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[29319]: -02-10T04:46:37.022672+0000 osd.0 2021-02-10T04:46:43.304 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[29319]: (osd.0) 127 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:46:43.304 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[29319]: cluster 2021-02-10T04:46:43.304 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[29319]: 2021-02-10T04:46:43.304 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[29319]: 2021 2021-02-10T04:46:43.304 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[29319]: - 2021-02-10T04:46:43.304 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[29319]: 02-10T04:46:37. 2021-02-10T04:46:43.305 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[29319]: 026293+0000 osd.0 (osd.0) 128 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:46:43.305 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:46:43.305 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[29319]: 10T04:46:37.153682+0000 osd.6 (osd. 2021-02-10T04:46:43.305 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:37 smithi085 conmon[29319]: 6) 115 : cluster [DBG] 2.24 scrub ok 2021-02-10T04:46:43.305 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:46:43.305 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[29319]: 04:46:37.748254+0000 osd.4 (osd.4) 1953 : cluster [DBG] 6.12 scrub ok 2021-02-10T04:46:43.305 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[29319]: cluster 2021-02-10T04:46:37.802647+0000 mon.a ( 2021-02-10T04:46:43.305 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[29319]: mon.0) 2108 : cluster [DBG] osdmap e1055: 8 total, 8 up, 7 in 2021-02-10T04:46:43.305 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[29319]: cluster 2021-02-10T04:46:43.306 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[29319]: 2021-02-10T04:46:37.897969+0000 osd.5 2021-02-10T04:46:43.306 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[29319]: (osd.5) 115 : cluster [DBG] 5.7 scrub ok 2021-02-10T04:46:43.306 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:46:43.306 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[29319]: 02-10T04:46:38.174462+0000 mon.a 2021-02-10T04:46:43.306 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[29319]: (mon.0) 2109 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:46:43.306 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:46:43.306 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[29319]: 10T04:46:38.174753+0000 mon.a ( 2021-02-10T04:46:43.306 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[29319]: mon.0) 2110 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:46:43.307 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[29319]: cluster 2021-02-10T04:46:43.307 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[29319]: 2021-02-10T04:46:38.398537 2021-02-10T04:46:43.307 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2111 : cluster [DBG] 2021-02-10T04:46:43.307 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:38 smithi085 conmon[29319]: osdmap e1056: 8 total, 8 up, 7 in 2021-02-10T04:46:43.307 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:39 smithi085 conmon[29319]: cluster 2021-02-10T04:46:38.323778+0000 mgr.y ( 2021-02-10T04:46:43.307 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:39 smithi085 conmon[29319]: mgr.14140) 1024 : cluster [DBG] pgmap v1795: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2021-02-10T04:46:43.307 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:39 smithi085 conmon[29319]: audit 2021-02-10T04:46:38.773047+0000 mgr.y (mgr.14140) 1025 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:46:43.307 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:39 smithi085 conmon[29319]: cluster 2021-02-10T04:46: 2021-02-10T04:46:43.307 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:39 smithi085 conmon[29319]: 38.774787+0000 osd.0 (osd.0) 129 : cluster 2021-02-10T04:46:43.307 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:39 smithi085 conmon[29319]: [DBG] 7.5 scrub ok 2021-02-10T04:46:43.307 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:39 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:46:43.307 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:39 smithi085 conmon[29319]: -10T04 2021-02-10T04:46:43.308 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:39 smithi085 conmon[29319]: :46:38.776800+0000 osd.4 (osd.4) 1954 : cluster 2021-02-10T04:46:43.308 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:39 smithi085 conmon[29319]: [DBG] 4.1 scrub ok 2021-02-10T04:46:43.308 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:39 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:46:43.308 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:39 smithi085 conmon[29319]: 02-10T04:46:39.502891+0000 2021-02-10T04:46:43.308 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:39 smithi085 conmon[29319]: osd.3 (osd.3) 118 : cluster [DBG] 5.14 scrub ok 2021-02-10T04:46:43.308 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:41 smithi085 conmon[29319]: cluster 2021-02-10T04:46:43.308 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:41 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:46:43.308 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:41 smithi085 conmon[29319]: :46:40.324582+0000 mgr.y (mgr.14140) 1026 : cluster [DBG] pgmap v1797: 188 pgs: 188 active+clean; 858 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:46:43.308 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:41 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:46:43.308 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:41 smithi085 conmon[29319]: 04:46:40.753574+0000 osd.0 (osd.0) 2021-02-10T04:46:43.308 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:41 smithi085 conmon[29319]: 130 : cluster [DBG] 7.4 scrub ok 2021-02-10T04:46:43.308 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:42 smithi085 conmon[29319]: cluster 2021-02-10T04:46:43.308 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:42 smithi085 conmon[29319]: 2021-02-10T04:46:43.308 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:42 smithi085 conmon[29319]: 2021 2021-02-10T04:46:43.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:42 smithi085 conmon[29319]: - 2021-02-10T04:46:43.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:42 smithi085 conmon[29319]: 02 2021-02-10T04:46:43.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:42 smithi085 conmon[29319]: - 2021-02-10T04:46:43.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:42 smithi085 conmon[29319]: 10 2021-02-10T04:46:43.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:42 smithi085 conmon[29319]: T 2021-02-10T04:46:43.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:42 smithi085 conmon[29319]: 04:46:42.217444 2021-02-10T04:46:43.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:42 smithi085 conmon[29319]: +0000 osd.6 (osd.6) 116 : cluster [DBG] 2.14 scrub ok 2021-02-10T04:46:43.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:42 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:46:43.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:42 smithi085 conmon[29319]: 02-10T04:46:42.537438+0000 osd.3 (osd. 2021-02-10T04:46:43.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:42 smithi085 conmon[29319]: 3) 119 : cluster [DBG] 2.19 scrub ok 2021-02-10T04:46:43.310 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:46:43.310 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:46:43.310 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:37 smithi164 conmon[29288]: cluster 2021-02-10T04:46:43.310 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:37 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:46:43.310 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:37 smithi164 conmon[29288]: 46:36.323155+0000 mgr.y (mgr.14140) 1023 : cluster [DBG] pgmap v1793: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 961 B/s rd, 0 op/s 2021-02-10T04:46:43.310 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:37 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:46:43.310 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:37 smithi164 conmon[29288]: T04:46:36.790480+0000 osd.4 (osd.4) 1952 : cluster [DBG] 6.1e scrub ok 2021-02-10T04:46:43.310 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:37 smithi164 conmon[29288]: cluster 2021-02-10T04:46:36.919201+0000 osd.5 2021-02-10T04:46:43.310 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:37 smithi164 conmon[29288]: (osd.5) 114 : cluster [DBG] 4.7 scrub ok 2021-02-10T04:46:43.311 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:37 smithi164 conmon[29288]: cluster 2021-02-10T04:46:43.311 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:37 smithi164 conmon[29288]: 2021-02-10T04:46:37.022672+0000 osd.0 (osd.0) 127 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:46:43.311 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:37 smithi164 conmon[29288]: cluster 2021-02-10T04:46:37.026293 2021-02-10T04:46:43.311 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:37 smithi164 conmon[29288]: +0000 osd.0 (osd.0) 128 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:46:43.311 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:37 smithi164 conmon[29288]: 2021-02-10T04:46:43.312 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:37 smithi164 conmon[29288]: cluster 2021-02-10T04:46:43.312 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:37 smithi164 conmon[29288]: 2021-02-10T04:46:37.153682+0000 osd.6 ( 2021-02-10T04:46:43.312 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:37 smithi164 conmon[29288]: osd.6) 115 : cluster [DBG] 2.24 scrub ok 2021-02-10T04:46:43.312 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:38 smithi164 conmon[29288]: cluster 2021-02-10T04:46:37.748254+0000 osd.4 (osd.4) 1953 2021-02-10T04:46:43.312 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:38 smithi164 conmon[29288]: : cluster [DBG] 6.12 scrub ok 2021-02-10T04:46:43.312 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:38 smithi164 conmon[29288]: cluster 2021-02-10T04:46:37.802647+0000 mon.a (mon.0) 2108 : cluster [DBG] osdmap e1055: 8 total, 8 up, 7 in 2021-02-10T04:46:43.313 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:38 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:46:43.313 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:38 smithi164 conmon[29288]: -10T04:46:37.897969+0000 osd.5 (osd.5) 115 : cluster [DBG] 5.7 scrub ok 2021-02-10T04:46:43.313 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:38 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:46:43.313 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:38 smithi164 conmon[29288]: T04:46:38.174462+0000 mon.a (mon.0) 2109 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:46:43.313 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:38 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:46:43.313 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:38 smithi164 conmon[29288]: T04:46:38.174753+0000 mon.a (mon.0) 2110 : 2021-02-10T04:46:43.314 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:38 smithi164 conmon[29288]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:46:43.314 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:38 smithi164 conmon[29288]: cluster 2021-02-10T04:46:43.314 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:38 smithi164 conmon[29288]: 2021-02-10T04:46:38.398537+0000 mon.a (mon.0) 2111 2021-02-10T04:46:43.314 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:38 smithi164 conmon[29288]: : cluster [DBG] osdmap e1056: 8 total, 8 up, 7 in 2021-02-10T04:46:43.314 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:39 smithi164 conmon[29288]: cluster 2021-02-10T04:46:38.323778+0000 mgr.y ( 2021-02-10T04:46:43.315 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:39 smithi164 conmon[29288]: mgr.14140) 1024 : cluster [DBG] pgmap v1795: 188 pgs: 188 active+clean; 860 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2021-02-10T04:46:43.315 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:39 smithi164 conmon[29288]: audit 2021-02-10T04:46:38.773047+0000 mgr.y (mgr.14140) 1025 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:46:43.315 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:39 smithi164 conmon[29288]: cluster 2021-02-10T04:46:38.774787+0000 osd.0 (osd.0) 2021-02-10T04:46:43.315 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:39 smithi164 conmon[29288]: 129 : cluster [DBG] 7.5 scrub ok 2021-02-10T04:46:43.315 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:39 smithi164 conmon[29288]: cluster 2021-02-10T04:46:43.315 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:39 smithi164 conmon[29288]: 2021-02-10T04:46:38.776800 2021-02-10T04:46:43.316 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:39 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1954 : cluster [DBG] 2021-02-10T04:46:43.316 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:39 smithi164 conmon[29288]: 4.1 scrub ok 2021-02-10T04:46:43.316 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:39 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:46:43.316 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:39 smithi164 conmon[29288]: -02-10T04:46:39.502891+0000 2021-02-10T04:46:43.316 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:39 smithi164 conmon[29288]: osd.3 (osd.3) 118 : cluster [DBG] 2021-02-10T04:46:43.317 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:39 smithi164 conmon[29288]: 5.14 scrub ok 2021-02-10T04:46:43.317 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:41 smithi164 conmon[29288]: cluster 2021-02-10T04:46:40.324582+0000 2021-02-10T04:46:43.317 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:41 smithi164 conmon[29288]: mgr.y (mgr.14140) 1026 : cluster [DBG] pgmap v1797: 188 pgs: 188 active+clean; 858 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:46:43.317 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:41 smithi164 conmon[29288]: cluster 2021-02-10T04:46:40.753574+0000 osd.0 (osd.0) 130 : cluster [DBG] 2021-02-10T04:46:43.317 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:41 smithi164 conmon[29288]: 7.4 scrub ok 2021-02-10T04:46:43.317 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:42 smithi164 conmon[29288]: cluster 2021-02-10T04:46:43.318 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:42 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:46:43.318 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:42 smithi164 conmon[29288]: 04:46:42.217444+0000 osd.6 (osd.6) 116 : cluster [DBG] 2.14 scrub ok 2021-02-10T04:46:43.318 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:42 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:46:43.318 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:42 smithi164 conmon[29288]: 04:46:42.537438+0000 osd.3 (osd.3) 119 : cluster [DBG] 2.19 scrub ok 2021-02-10T04:46:43.318 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:42 smithi164 conmon[29288]: 2021-02-10T04:46:43.319 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:46:38 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:46:43.410 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:46:43.444 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:46:44.132 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:46:44.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:43 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:46:44.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:43 smithi164 conmon[29288]: T04:46:42.325149+0000 mgr.y (mgr.14140) 1027 : cluster [DBG] pgmap v1798: 188 pgs: 188 active+clean; 858 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:46:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:43 smithi164 conmon[29288]: cluster 2021-02-10T04:46:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:43 smithi164 conmon[29288]: 2021-02-10T04:46:43.397286+0000 mon.a (mon 2021-02-10T04:46:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:43 smithi164 conmon[29288]: .0) 2112 : cluster [DBG] osdmap e1057: 8 total, 8 up, 7 in 2021-02-10T04:46:44.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:46:43 smithi085 conmon[49106]: debug 2021-02-10T04:46:43.882+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:46:44.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:43 smithi085 conmon[35961]: cluster 2021-02-10T04:46:42.325149+0000 mgr.y (mgr. 2021-02-10T04:46:44.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:43 smithi085 conmon[35961]: 14140) 1027 : cluster [DBG] pgmap v1798: 188 pgs: 188 active+clean; 858 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:46:44.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:43 smithi085 conmon[35961]: cluster 2021-02-10T04:46:43.397286+0000 mon.a (mon 2021-02-10T04:46:44.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:43 smithi085 conmon[35961]: .0) 2112 : cluster [DBG] osdmap e1057: 8 total, 8 up, 7 in 2021-02-10T04:46:44.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:43 smithi085 conmon[29319]: cluster 2021-02-10T04:46:42.325149+0000 mgr.y 2021-02-10T04:46:44.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:43 smithi085 conmon[29319]: (mgr.14140) 1027 : cluster [DBG] pgmap v1798: 188 pgs: 188 active+clean; 858 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:46:44.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:43 smithi085 conmon[29319]: cluster 2021-02-10T04:46:43.397286+0000 mon.a (mon.0) 2112 : cluster [DBG] osdmap e1057: 8 total, 8 up, 7 in 2021-02-10T04:46:45.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:44 smithi164 conmon[29288]: cluster 2021-02-10T04:46:45.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:44 smithi164 conmon[29288]: 2021-02-10T04:46:45.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:44 smithi164 conmon[29288]: 2021 2021-02-10T04:46:45.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:44 smithi164 conmon[29288]: - 2021-02-10T04:46:45.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:44 smithi164 conmon[29288]: 02 2021-02-10T04:46:45.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:44 smithi164 conmon[29288]: - 2021-02-10T04:46:45.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:44 smithi164 conmon[29288]: 10T04:46:43.762736+0000 osd.4 (osd.4) 1955 : cluster [DBG] 5.c scrub ok 2021-02-10T04:46:45.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:44 smithi164 conmon[29288]: 2021-02-10T04:46:45.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:44 smithi164 conmon[29288]: cluster 2021-02-10T04:46:45.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:44 smithi164 conmon[29288]: 2021-02-10T04:46:43.936357+0000 osd.7 (osd.7) 74 2021-02-10T04:46:45.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:44 smithi164 conmon[29288]: : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:46:45.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:44 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:46:45.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:44 smithi164 conmon[29288]: -10T04:46:43.938573 2021-02-10T04:46:45.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:44 smithi164 conmon[29288]: +0000 osd.7 (osd.7) 75 2021-02-10T04:46:45.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:44 smithi164 conmon[29288]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:46:45.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:44 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:46:45.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:44 smithi164 conmon[29288]: 46:44.241676+0000 osd.6 (osd.6) 117 : cluster [DBG] 2.17 scrub ok 2021-02-10T04:46:45.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:44 smithi085 conmon[35961]: cluster 2021-02-10T04:46:43.762736+0000 osd.4 (osd 2021-02-10T04:46:45.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:44 smithi085 conmon[35961]: .4) 1955 : cluster [DBG] 5.c scrub ok 2021-02-10T04:46:45.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:44 smithi085 conmon[35961]: cluster 2021-02-10T04:46:43.936357+0000 osd.7 (osd.7) 74 : cluster [DBG] 2021-02-10T04:46:45.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:44 smithi085 conmon[35961]: purged_snaps scrub starts 2021-02-10T04:46:45.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:44 smithi085 conmon[35961]: cluster 2021-02-10T04:46:43.938573+0000 osd.7 2021-02-10T04:46:45.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:44 smithi085 conmon[35961]: (osd.7) 75 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:46:45.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:44 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:46:45.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:44 smithi085 conmon[35961]: 02-10T04:46:44.241676+0000 osd.6 (osd.6) 117 : cluster [DBG] 2021-02-10T04:46:45.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:44 smithi085 conmon[35961]: 2.17 scrub ok 2021-02-10T04:46:45.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:44 smithi085 conmon[29319]: cluster 2021-02-10T04:46:43.762736+0000 osd.4 2021-02-10T04:46:45.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:44 smithi085 conmon[29319]: (osd.4) 1955 : cluster [DBG] 5.c scrub ok 2021-02-10T04:46:45.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:44 smithi085 conmon[29319]: cluster 2021-02-10T04:46:43.936357+0000 osd.7 (osd.7) 74 : cluster 2021-02-10T04:46:45.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:44 smithi085 conmon[29319]: [DBG] purged_snaps scrub starts 2021-02-10T04:46:45.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:44 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:46:45.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:44 smithi085 conmon[29319]: 02-10T04:46:43.938573+0000 2021-02-10T04:46:45.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:44 smithi085 conmon[29319]: osd.7 (osd.7) 75 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:46:45.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:44 smithi085 conmon[29319]: 2021-02-10T04:46:45.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:44 smithi085 conmon[29319]: cluster 2021-02-10T04:46:45.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:44 smithi085 conmon[29319]: 2021-02-10T04:46:44. 2021-02-10T04:46:45.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:44 smithi085 conmon[29319]: 241676+0000 osd.6 (osd.6) 117 : cluster [DBG] 2.17 scrub ok 2021-02-10T04:46:45.530 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.6 dump_historic_ops 2021-02-10T04:46:45.828 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:46:45.836 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:45 smithi164 conmon[29288]: cluster 2021-02-10T04:46:44.325753+0000 mgr.y ( 2021-02-10T04:46:45.836 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:45 smithi164 conmon[29288]: mgr.14140) 1028 : cluster [DBG] pgmap v1800: 188 pgs: 188 active+clean; 858 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:46:45.836 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:45 smithi164 conmon[29288]: cluster 2021-02-10T04:46:44.836482+0000 mon.a (mon.0) 2113 : cluster [DBG] osdmap e1058: 8 total, 8 up, 7 in 2021-02-10T04:46:45.997 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:46:46.023 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:46:46.171 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:45 smithi085 conmon[35961]: cluster 2021-02-10T04:46:44.325753+0000 mgr.y ( 2021-02-10T04:46:46.171 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:45 smithi085 conmon[35961]: mgr.14140) 1028 : cluster [DBG] pgmap v1800: 188 pgs: 188 active+clean; 858 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:46:46.171 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:45 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:46:46.171 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:45 smithi085 conmon[35961]: :46:44.836482+0000 mon.a (mon.0) 2113 : cluster [DBG] osdmap e1058: 8 total, 8 up, 7 in 2021-02-10T04:46:46.171 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:45 smithi085 conmon[29319]: cluster 2021-02-10T04:46:44.325753+0000 2021-02-10T04:46:46.171 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:45 smithi085 conmon[29319]: mgr.y (mgr.14140) 1028 : cluster [DBG] pgmap v1800: 188 pgs: 188 active+clean; 858 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:46:46.171 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:45 smithi085 conmon[29319]: cluster 2021-02-10T04:46:44.836482+0000 mon.a (mon.0) 2113 : cluster [DBG] osdmap e1058: 8 total, 8 up, 7 in 2021-02-10T04:46:46.644 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:46:46 smithi164 conmon[44544]: debug 2021-02-10T04:46:46.410+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:46:46.663 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:46:46.768 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:46:46.805 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:46:46.842 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:46 smithi085 conmon[29319]: cluster 2021-02-10T04:46:46.592371+0000 osd.3 (osd.3) 120 : cluster [DBG] 1.4 scrub ok 2021-02-10T04:46:46.842 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:46 smithi085 conmon[35961]: cluster 2021-02-10T04:46:46.592371+0000 osd.3 (osd.3) 120 : cluster [DBG] 1.4 scrub ok 2021-02-10T04:46:46.937 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:46:46.942 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:46 smithi164 conmon[29288]: cluster 2021-02-10T04:46:46.592371+0000 osd.3 (osd.3) 120 : cluster [DBG] 1.4 scrub ok 2021-02-10T04:46:46.942 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:46 smithi164 conmon[33677]: debug 2021-02-10T04:46:46.899+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:46:47.044 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:46:47.081 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:46:47.260 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:46:47.365 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:46:47.395 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:46:47.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:46:47 smithi085 conmon[49106]: debug 2021-02-10T04:46:47.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:46:47 smithi085 conmon[49106]: 2021-02-10T04:46:47.169+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:46:47.604 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:46:47.638 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:46:47.644 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:46:47 smithi164 conmon[44544]: debug 2021-02-10T04:46:47.463+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:46:47.663 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:46:47.688 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:46:47.690 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:46:47.723 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:46:47.758 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:46:47.794 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:46:47.828 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:46:47.829 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:46:47.829 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:46:47.829 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:46:47.830 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:46:47.830 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:46:47.830 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:46:47.830 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:46:47.830 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:46:47.868 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:46:47.872 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:46:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:47 smithi164 conmon[29288]: cluster 2021-02-10T04:46:46.326483+0000 2021-02-10T04:46:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:47 smithi164 conmon[29288]: mgr.y (mgr.14140) 1029 : cluster [DBG] pgmap v1802: 188 pgs: 188 active+clean; 844 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:46:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:47 smithi164 conmon[29288]: cluster 2021-02-10T04:46:46.664979+0000 osd.4 (osd.4) 1956 2021-02-10T04:46:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:47 smithi164 conmon[29288]: : cluster [DBG] 5.a scrub ok 2021-02-10T04:46:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:47 smithi164 conmon[29288]: cluster 2021-02-10T04:46:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:47 smithi164 conmon[29288]: 2021-02 2021-02-10T04:46:48.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:47 smithi164 conmon[29288]: -10T04:46:47. 2021-02-10T04:46:48.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:47 smithi164 conmon[29288]: 142969+0000 osd.6 (osd. 2021-02-10T04:46:48.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:47 smithi164 conmon[29288]: 6) 118 : cluster [DBG] 2.1a scrub ok 2021-02-10T04:46:48.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:47 smithi164 conmon[29288]: 2021-02-10T04:46:48.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:47 smithi164 conmon[29288]: audit 2021-02-10T04:46:48.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:47 smithi164 conmon[29288]: 2021-02-10 2021-02-10T04:46:48.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:47 smithi164 conmon[29288]: T04:46:47.555968+0000 mon.c (mon.1) 76 : audit [DBG] from='client.? 172.21.15.85:0/362712082' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:46:48.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:47 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:46:48.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:47 smithi164 conmon[29288]: 46:47.605354+0000 osd.3 (osd.3) 121 : cluster [DBG] 2.9 scrub ok 2021-02-10T04:46:48.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:47 smithi085 conmon[35961]: cluster 2021-02-10T04:46:46.326483+0000 2021-02-10T04:46:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:47 smithi085 conmon[35961]: mgr.y (mgr.14140) 1029 : cluster [DBG] pgmap v1802: 188 pgs: 188 active+clean; 844 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:46:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:47 smithi085 conmon[35961]: cluster 2021-02-10T04:46:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:47 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:46:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:47 smithi085 conmon[35961]: 04:46:46.664979+0000 osd.4 (osd.4) 1956 2021-02-10T04:46:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:47 smithi085 conmon[35961]: : cluster [DBG] 5.a scrub ok 2021-02-10T04:46:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:47 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:46:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:47 smithi085 conmon[35961]: 02-10T04:46:47.142969+0000 osd.6 (osd 2021-02-10T04:46:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:47 smithi085 conmon[35961]: .6) 118 : cluster [DBG] 2.1a scrub ok 2021-02-10T04:46:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:47 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:46:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:47 smithi085 conmon[35961]: :46:47.555968+0000 mon.c (mon.1) 76 2021-02-10T04:46:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:47 smithi085 conmon[35961]: : audit [DBG] from='client.? 172.21.15.85:0/362712082' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:46:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:47 smithi085 conmon[35961]: cluster 2021-02-10T04:46: 2021-02-10T04:46:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:47 smithi085 conmon[35961]: 47.605354+0000 osd.3 (osd.3) 121 : cluster [DBG] 2021-02-10T04:46:48.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:47 smithi085 conmon[35961]: 2.9 scrub ok 2021-02-10T04:46:48.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:47 smithi085 conmon[29319]: cluster 2021-02-10T04:46:46.326483+0000 mgr.y (mgr 2021-02-10T04:46:48.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:47 smithi085 conmon[29319]: .14140) 1029 : cluster [DBG] pgmap v1802: 188 pgs: 188 active+clean; 844 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:46:48.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:47 smithi085 conmon[29319]: cluster 2021-02-10T04:46:46.664979+0000 osd.4 (osd.4) 1956 : cluster [DBG] 5.a scrub ok 2021-02-10T04:46:48.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:47 smithi085 conmon[29319]: cluster 2021-02-10T04:46:47.142969+0000 osd.6 (osd.6) 118 2021-02-10T04:46:48.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:47 smithi085 conmon[29319]: : cluster [DBG] 2.1a scrub ok 2021-02-10T04:46:48.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:47 smithi085 conmon[29319]: audit 2021 2021-02-10T04:46:48.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:47 smithi085 conmon[29319]: -02- 2021-02-10T04:46:48.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:47 smithi085 conmon[29319]: 10T04:46:47.555968+0000 mon.c 2021-02-10T04:46:48.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:47 smithi085 conmon[29319]: (mon.1) 76 : audit [DBG] from='client.? 172.21.15.85:0/362712082' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:46:48.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:47 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:46:48.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:47 smithi085 conmon[29319]: 46:47.605354+0000 osd.3 (osd.3) 121 : cluster [DBG] 2.9 scrub ok 2021-02-10T04:46:48.412 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is unset 2021-02-10T04:46:48.846 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:48 smithi085 conmon[29319]: cluster 2021-02-10T04:46:47.806100+0000 2021-02-10T04:46:48.846 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:48 smithi085 conmon[29319]: osd.0 (osd.0) 131 : cluster [DBG] 7.9 scrub ok 2021-02-10T04:46:48.846 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:48 smithi085 conmon[29319]: audit 2021-02-10T04:46:47.836959+0000 mgr.y (mgr.14140) 1030 : audit [DBG] from='client.25366 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:46:48.846 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:48 smithi085 conmon[29319]: audit 2021-02-10T04:46:47.860671+0000 mon.a (mon.0) 2114 : audit [INF] from='client.? 172.21.15.85:0/1504932964' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:46:48.846 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:48 smithi085 conmon[29319]: cluster 2021-02-10T04:46:48.394994+0000 mon.a (mon.0) 2115 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:46:48.846 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:48 smithi085 conmon[29319]: cluster 2021-02-10T04:46:48.395043+0000 mon.a (mon.0) 2116 : cluster [INF] Cluster is now healthy 2021-02-10T04:46:48.846 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:48 smithi085 conmon[29319]: 2021-02-10T04:46:48.847 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:48 smithi085 conmon[29319]: audit 2021-02-10T04: 2021-02-10T04:46:48.847 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:48 smithi085 conmon[29319]: 46:48.400163+0000 mon.a (mon.0) 2021-02-10T04:46:48.847 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:48 smithi085 conmon[29319]: 2117 : audit [INF] from='client.? 172.21.15.85:0/1504932964' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:46:48.847 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:48 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:46:48.847 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:48 smithi085 conmon[29319]: 02-10T04:46:48.400224+0000 2021-02-10T04:46:48.847 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:48 smithi085 conmon[29319]: mon.a (mon.0) 2118 : cluster [DBG] osdmap e1059: 8 total, 8 up, 7 in 2021-02-10T04:46:48.847 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:48 smithi085 conmon[35961]: cluster 2021-02-10T04:46:47.806100+0000 osd.0 2021-02-10T04:46:48.847 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:48 smithi085 conmon[35961]: (osd.0) 131 : cluster [DBG] 7.9 scrub ok 2021-02-10T04:46:48.847 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:48 smithi085 conmon[35961]: audit 2021-02-10T04:46:47.836959+0000 mgr.y (mgr.14140) 1030 : audit [DBG] from='client.25366 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:46:48.894 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:46:48 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:46:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:48 smithi164 conmon[29288]: cluster 2021-02-10T04:46:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:48 smithi164 conmon[29288]: 2021-02-10T04:46:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:48 smithi164 conmon[29288]: 2021 2021-02-10T04:46:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:48 smithi164 conmon[29288]: - 2021-02-10T04:46:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:48 smithi164 conmon[29288]: 02 2021-02-10T04:46:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:48 smithi164 conmon[29288]: - 2021-02-10T04:46:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:48 smithi164 conmon[29288]: 10 2021-02-10T04:46:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:48 smithi164 conmon[29288]: T04:46:47.806100+0000 2021-02-10T04:46:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:48 smithi164 conmon[29288]: osd.0 (osd.0) 131 : cluster [DBG] 7.9 scrub ok 2021-02-10T04:46:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:48 smithi164 conmon[29288]: audit 2021-02-10T04:46:47.836959+0000 mgr.y (mgr.14140) 1030 : audit 2021-02-10T04:46:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:48 smithi164 conmon[29288]: [DBG] from='client.25366 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:46:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:48 smithi164 conmon[29288]: audit 2021-02-10T04:46:47 2021-02-10T04:46:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:48 smithi164 conmon[29288]: .860671+0000 mon.a (mon.0) 2114 : audit 2021-02-10T04:46:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:48 smithi164 conmon[29288]: [INF] from='client.? 172.21.15.85:0/1504932964' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:46:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:48 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:46:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:48 smithi164 conmon[29288]: T04:46:48.394994+0000 mon.a (mon.0 2021-02-10T04:46:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:48 smithi164 conmon[29288]: ) 2115 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:46:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:48 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:46:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:48 smithi164 conmon[29288]: 10T04:46:48.395043+0000 mon.a (mon. 2021-02-10T04:46:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:48 smithi164 conmon[29288]: 0) 2116 : cluster [INF] Cluster is now healthy 2021-02-10T04:46:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:48 smithi164 conmon[29288]: audit 2021-02-10T04:46:48.400163+0000 mon.a ( 2021-02-10T04:46:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:48 smithi164 conmon[29288]: mon.0) 2117 : audit [INF] from='client.? 172.21.15.85:0/1504932964' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:46:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:48 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:46:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:48 smithi164 conmon[29288]: 02-10T04:46:48.400224+0000 mon.a ( 2021-02-10T04:46:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:48 smithi164 conmon[29288]: mon.0) 2118 : cluster [DBG] osdmap e1059: 8 total, 8 up, 7 in 2021-02-10T04:46:48.896 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:48 smithi164 conmon[33677]: debug 2021-02-10T04:46:48.599+0000 7fbec19a1700 -1 osd.4 1059 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:46:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:48 smithi085 conmon[35961]: audit 2021-02-10T04:46:47.860671+0000 mon.a (mon.0) 2114 2021-02-10T04:46:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:48 smithi085 conmon[35961]: : audit [INF] from='client.? 172.21.15.85:0/1504932964' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:46:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:48 smithi085 conmon[35961]: cluster 2021-02-10T04:46:48.394994+0000 mon.a (mon 2021-02-10T04:46:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:48 smithi085 conmon[35961]: .0) 2115 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:46:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:48 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:46:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:48 smithi085 conmon[35961]: 04:46:48.395043+0000 mon.a (mon.0) 2116 2021-02-10T04:46:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:48 smithi085 conmon[35961]: : cluster [INF] Cluster is now healthy 2021-02-10T04:46:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:48 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:46:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:48 smithi085 conmon[35961]: T04:46:48.400163+0000 mon.a (mon.0) 2117 : audit [INF] 2021-02-10T04:46:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:48 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/1504932964' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:46:49.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:48 smithi085 conmon[35961]: cluster 2021-02-10T04:46:48 2021-02-10T04:46:49.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:48 smithi085 conmon[35961]: .400224+0000 mon.a (mon.0) 2118 : cluster [DBG] osdmap e1059: 8 total, 8 up, 7 in 2021-02-10T04:46:49.851 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:49 smithi164 conmon[33677]: debug 2021-02-10T04:46:49.584+0000 7fbec19a1700 -1 osd.4 1059 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:46:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:49 smithi164 conmon[29288]: cluster 2021-02-10T04:46:48.327112+0000 mgr.y (mgr. 2021-02-10T04:46:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:49 smithi164 conmon[29288]: 14140) 1031 : cluster [DBG] pgmap v1803: 188 pgs: 188 active+clean; 844 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:46:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:49 smithi164 conmon[29288]: cluster 2021-02-10T04:46:48.600683+0000 osd.4 (osd.4) 1957 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:49 smithi164 conmon[29288]: cluster 2021-02-10T04:46:48.600688+0000 osd.4 (osd.4) 1958 : cluster [WRN] 2021-02-10T04:46:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:49 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:49 smithi164 conmon[29288]: cluster 2021-02-10T04:46:48.600694 2021-02-10T04:46:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:49 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 1959 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:49 smithi164 conmon[29288]: 2021-02-10T04:46:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:49 smithi164 conmon[29288]: cluster 2021-02-10T04:46:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:49 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:46:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:49 smithi164 conmon[29288]: :46:48.600697+0000 osd.4 (osd 2021-02-10T04:46:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:49 smithi164 conmon[29288]: .4) 1960 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:49 smithi164 conmon[29288]: cluster 2021-02-10T04:46: 2021-02-10T04:46:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:49 smithi164 conmon[29288]: 48.602813+0000 osd.4 (osd.4) 1961 : cluster [DBG] 6.e scrub ok 2021-02-10T04:46:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:49 smithi164 conmon[29288]: 2021-02-10T04:46:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:49 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:46:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:49 smithi164 conmon[29288]: 04:46:48.783048+0000 mgr.y (mgr. 2021-02-10T04:46:50.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:49 smithi164 conmon[29288]: 14140) 1032 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:46:50.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:49 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:46:50.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:49 smithi164 conmon[29288]: -02-10T04:46:48.841964+0000 2021-02-10T04:46:50.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:49 smithi164 conmon[29288]: osd.5 (osd.5) 116 : cluster [DBG] 5.f scrub ok 2021-02-10T04:46:50.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:49 smithi164 conmon[29288]: 2021-02-10T04:46:50.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[35961]: cluster 2021-02-10T04:46:48.327112 2021-02-10T04:46:50.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1031 : cluster [DBG] pgmap v1803: 188 pgs: 188 active+clean; 844 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:46:50.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[35961]: cluster 2021-02-10T04:46: 2021-02-10T04:46:50.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[35961]: 48.600683+0000 osd.4 (osd.4) 1957 2021-02-10T04:46:50.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:50.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:46:50.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[35961]: :46:48.600688+0000 osd.4 (osd.4) 2021-02-10T04:46:50.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[35961]: 1958 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:50.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:46:50.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[35961]: 10T04:46:48.600694+0000 osd.4 2021-02-10T04:46:50.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[35961]: (osd.4) 1959 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:50.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:46:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[35961]: 02-10T04:46:48.600697+0000 osd.4 2021-02-10T04:46:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[35961]: (osd.4) 1960 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[35961]: cluster 2021-02-10T04:46:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[35961]: 2021-02-10T04:46:48.602813+0000 2021-02-10T04:46:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[35961]: osd.4 (osd.4) 1961 : cluster [DBG] 2021-02-10T04:46:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[35961]: 6.e scrub ok 2021-02-10T04:46:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[35961]: audit 2021-02-10T04:46:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[35961]: 2021-02-10T04:46:48.783048+0000 mgr.y ( 2021-02-10T04:46:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[35961]: mgr.14140) 1032 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:46:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:46:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[35961]: 02-10T04:46:48.841964+0000 osd.5 2021-02-10T04:46:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[35961]: (osd.5) 116 : cluster [DBG] 5.f scrub ok 2021-02-10T04:46:50.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:46:50.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[29319]: :46:48.327112+0000 mgr.y (mgr.14140) 1031 : cluster [DBG] pgmap v1803: 188 pgs: 188 active+clean; 844 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:46:50.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[29319]: cluster 2021-02-10T04:46:50.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[29319]: 2021 2021-02-10T04:46:50.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[29319]: - 2021-02-10T04:46:50.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[29319]: 02 2021-02-10T04:46:50.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[29319]: -10T04:46:48.600683+0000 2021-02-10T04:46:50.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[29319]: osd.4 (osd.4) 1957 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:50.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:46:50.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[29319]: :46:48.600688+0000 osd.4 (osd.4 2021-02-10T04:46:50.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[29319]: ) 1958 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:50.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[29319]: cluster 2021-02-10T04:46:50.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[29319]: 2021-02-10T04:46 2021-02-10T04:46:50.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[29319]: :48.600694+0000 osd.4 (osd 2021-02-10T04:46:50.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[29319]: .4) 1959 : cluster [WRN] 2021-02-10T04:46:50.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:50.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[29319]: cluster 2021-02-10T04:46:48.600697+0000 osd.4 (osd.4) 1960 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:50.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[29319]: cluster 2021-02-10T04:46:50.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[29319]: 2021-02-10T04:46:48. 2021-02-10T04:46:50.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[29319]: 602813+0000 osd.4 (osd.4) 1961 : cluster [DBG] 6.e scrub ok 2021-02-10T04:46:50.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[29319]: 2021-02-10T04:46:50.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[29319]: audit 2021-02-10T04: 2021-02-10T04:46:50.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[29319]: 46:48.783048+0000 mgr.y (mgr.14140) 1032 2021-02-10T04:46:50.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[29319]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:46:50.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[29319]: cluster 2021-02-10T04:46:50.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[29319]: 2021-02-10T04:46:48. 2021-02-10T04:46:50.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:49 smithi085 conmon[29319]: 841964+0000 osd.5 (osd.5) 116 : cluster [DBG] 5.f scrub ok 2021-02-10T04:46:50.868 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[33677]: debug 2021-02-10T04:46:50.597+0000 7fbec19a1700 -1 osd.4 1060 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:46:50.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: cluster 2021-02-10T04:46: 2021-02-10T04:46:50.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: 49.584814+0000 osd.4 (osd.4) 1962 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:50.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: cluster 2021-02-10T04:46: 2021-02-10T04:46:50.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: 49.584825+0000 osd.4 (osd.4) 1963 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:50.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:46:50.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: T04:46:49.584831+0000 osd.4 (osd.4) 2021-02-10T04:46:50.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: 1964 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:50.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:46:50.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: -10T04:46:49.584838+0000 2021-02-10T04:46:50.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: osd.4 (osd.4) 1965 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:50.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: 2021-02-10T04:46:50.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:46:50.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: T04:46:49.745097+0000 osd.0 (osd. 2021-02-10T04:46:50.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: 0) 132 : cluster [DBG] 7.18 deep-scrub ok 2021-02-10T04:46:50.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: cluster 2021-02-10T04:46:50.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: 2021-02-10T04:46:50.872 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: 2021-02-10T04:46:49 2021-02-10T04:46:50.872 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: .800194+0000 osd.4 (osd.4) 1966 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:46:50.872 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: cluster 2021-02-10T04:46:50.872 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: 2021-02-10T04:46:49.801685+0000 osd.4 ( 2021-02-10T04:46:50.872 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: osd.4) 1967 : cluster 2021-02-10T04:46:50.872 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: [DBG] purged_snaps scrub ok 2021-02-10T04:46:50.872 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: cluster 2021-02-10T04:46:50.872 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: 2021-02-10T04:46:49.860222 2021-02-10T04:46:50.872 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2119 2021-02-10T04:46:50.872 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: : cluster [DBG] osdmap e1060: 8 total, 8 up, 7 in 2021-02-10T04:46:50.872 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:46:50.872 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: -10T04:46:49.869496+0000 osd.5 ( 2021-02-10T04:46:50.872 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: osd.5) 117 : cluster [DBG] 2.12 scrub ok 2021-02-10T04:46:50.873 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:46:50.873 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: 02-10T04:46:50.057158+0000 osd.6 2021-02-10T04:46:50.873 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: (osd.6) 119 : cluster [DBG] 2.29 scrub ok 2021-02-10T04:46:50.873 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:50 smithi164 conmon[29288]: 2021-02-10T04:46:51.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[29319]: cluster 2021-02-10T04:46:52.823 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:46:52.823 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[29319]: :46:49.584814+0000 osd.4 (osd.4) 1962 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:52.823 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:46:52.823 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[29319]: 46:49.584825+0000 osd.4 (osd.4) 1963 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:52.824 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[29319]: 2021-02-10T04:46:52.824 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:46:52.824 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[29319]: -02-10T04:46:49.584831+0000 osd.4 (osd.4) 1964 : cluster 2021-02-10T04:46:52.824 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:52.824 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[29319]: cluster 2021-02-10T04:46:52.824 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[29319]: 2021-02-10T04:46:49.584838+0000 osd.4 (osd.4) 2021-02-10T04:46:52.824 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[29319]: 1965 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:52.824 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:46:52.824 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[29319]: -02-10T04:46:49 2021-02-10T04:46:52.824 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[29319]: .745097+0000 osd.0 (osd.0) 132 : cluster [DBG] 2021-02-10T04:46:52.824 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[29319]: 7.18 deep-scrub ok 2021-02-10T04:46:52.825 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[29319]: cluster 2021-02-10T04:46:52.825 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[29319]: 2021-02-10T04:46:49.800194+0000 osd.4 ( 2021-02-10T04:46:52.825 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[29319]: osd.4) 1966 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:46:52.825 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[29319]: cluster 2021-02-10T04:46:49. 2021-02-10T04:46:52.825 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[29319]: 801685+0000 osd.4 (osd.4) 1967 : cluster [DBG] 2021-02-10T04:46:52.825 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[29319]: purged_snaps scrub ok 2021-02-10T04:46:52.825 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[29319]: cluster 2021-02-10T04:46:49 2021-02-10T04:46:52.825 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[29319]: .860222+0000 mon.a (mon.0) 2119 : 2021-02-10T04:46:52.825 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[29319]: cluster [DBG] osdmap e1060: 8 total, 8 up, 7 in 2021-02-10T04:46:52.825 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[29319]: cluster 2021-02-10T04:46:52.825 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[29319]: 2021-02-10T04:46:49.869496+0000 osd.5 (osd. 2021-02-10T04:46:52.825 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[29319]: 5) 117 : cluster [DBG] 2.12 scrub ok 2021-02-10T04:46:52.825 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:46:52.825 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[29319]: T04:46:50.057158+0000 osd.6 (osd.6) 119 : cluster [DBG] 2.29 scrub ok 2021-02-10T04:46:52.826 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: cluster 2021-02-10T04:46:52.826 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:46:52.826 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: 04:46:49.584814+0000 osd.4 (osd.4) 1962 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:52.826 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:46:52.826 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: 02-10T04:46:49.584825+0000 osd.4 (osd. 2021-02-10T04:46:52.826 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: 4) 1963 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:52.826 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: cluster 2021-02-10T04:46:52.826 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: 2021-02 2021-02-10T04:46:52.827 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: -10T04:46:49.584831+0000 osd.4 ( 2021-02-10T04:46:52.827 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: osd.4) 1964 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:52.827 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: cluster 2021-02-10T04:46:52.827 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: 2021-02-10T04:46: 2021-02-10T04:46:52.827 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: 49.584838+0000 osd.4 (osd.4) 1965 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:52.827 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: 2021-02-10T04:46:52.827 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: cluster 2021-02-10T04:46:52.827 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:46:52.827 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: 46:49.745097+0000 osd.0 (osd.0 2021-02-10T04:46:52.827 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: ) 132 : cluster [DBG] 7.18 deep-scrub ok 2021-02-10T04:46:52.827 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:46:52.827 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: 02-10T04:46:49.800194+0000 2021-02-10T04:46:52.827 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: osd.4 (osd.4) 1966 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:46:52.828 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: cluster 2021-02-10T04:46:52.828 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: 2021-02-10T04:46:49.801685+0000 2021-02-10T04:46:52.828 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: osd.4 (osd.4) 1967 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:46:52.828 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:46:52.828 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: 10T04:46:49.860222+0000 mon.a (mon.0 2021-02-10T04:46:52.828 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: ) 2119 : cluster [DBG] osdmap e1060: 8 total, 8 up, 7 in 2021-02-10T04:46:52.828 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:46:52.828 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: 04:46:49.869496+0000 osd.5 (osd.5 2021-02-10T04:46:52.828 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: ) 117 : cluster [DBG] 2.12 scrub ok 2021-02-10T04:46:52.828 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: cluster 2021-02-10T04:46 2021-02-10T04:46:52.828 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: :50.057158+0000 osd.6 (osd.6) 119 : cluster 2021-02-10T04:46:52.828 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:50 smithi085 conmon[35961]: [DBG] 2.29 scrub ok 2021-02-10T04:46:52.830 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[33677]: debug 2021-02-10T04:46:51.595+0000 7fbec19a1700 -1 osd.4 1060 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:46:52.830 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:46:52.830 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: -10T04:46:50.327785+0000 mgr.y (mgr.14140) 1033 : cluster [DBG] pgmap v1806: 188 pgs: 188 active+clean; 845 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:46:52.830 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: cluster 2021-02-10T04:46:50.598535+0000 2021-02-10T04:46:52.830 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: osd.4 (osd.4) 1968 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:52.830 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: cluster 2021-02-10T04:46:52.831 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: 2021-02-10T04:46:52.831 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: 2021 2021-02-10T04:46:52.831 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: - 2021-02-10T04:46:52.831 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: 02-10T04:46 2021-02-10T04:46:52.831 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: :50.598546+0000 osd.4 (osd.4) 1969 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:52.831 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: cluster 2021-02-10T04:46:52.831 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: 2021-02 2021-02-10T04:46:52.831 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: -10T04:46:50.598553+0000 osd.4 (osd.4 2021-02-10T04:46:52.831 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: ) 1970 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:52.831 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:46:52.831 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: -10T04:46:50.598560+0000 osd.4 (osd.4) 1971 : cluster [WRN] 2021-02-10T04:46:52.831 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:52.832 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:46:52.832 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: 04 2021-02-10T04:46:52.832 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: : 2021-02-10T04:46:52.832 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: 46 2021-02-10T04:46:52.832 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: : 2021-02-10T04:46:52.832 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: 50 2021-02-10T04:46:52.832 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: . 2021-02-10T04:46:52.832 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: 859186 2021-02-10T04:46:52.832 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: +0000 2021-02-10T04:46:52.832 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: 2021-02-10T04:46:52.832 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: osd.5 2021-02-10T04:46:52.832 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: ( 2021-02-10T04:46:52.832 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: osd 2021-02-10T04:46:52.833 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: . 2021-02-10T04:46:52.833 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: 5 2021-02-10T04:46:52.833 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: ) 2021-02-10T04:46:52.833 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: 118 2021-02-10T04:46:52.833 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: : 2021-02-10T04:46:52.833 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: cluster 2021-02-10T04:46:52.833 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: 2021-02-10T04:46:52.833 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: [DBG] 2021-02-10T04:46:52.833 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: 2021-02-10T04:46:52.833 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: 2.20 scrub ok 2021-02-10T04:46:52.833 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:51 smithi164 conmon[29288]: 2021-02-10T04:46:52.833 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:51 smithi085 conmon[29319]: cluster 2021-02-10T04:46:50.327785+0000 mgr.y (mgr.14140) 1033 : cluster 2021-02-10T04:46:52.834 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:51 smithi085 conmon[29319]: [DBG] pgmap v1806: 188 pgs: 188 active+clean; 845 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:46:52.834 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:51 smithi085 conmon[29319]: cluster 2021-02-10T04:46:50.598535+0000 osd.4 (osd.4) 1968 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:52.834 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:51 smithi085 conmon[29319]: cluster 2021-02-10T04:46:50.598546+0000 osd.4 (osd.4) 1969 : 2021-02-10T04:46:52.834 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:51 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:52.834 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:51 smithi085 conmon[29319]: cluster 2021-02-10T04:46:50.598553+0000 osd.4 (osd.4) 1970 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:52.834 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:51 smithi085 conmon[29319]: cluster 2021-02-10T04:46:50.598560+0000 osd.4 (osd.4) 1971 : cluster 2021-02-10T04:46:52.834 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:51 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:52.834 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:51 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:46:52.834 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:51 smithi085 conmon[29319]: -10T04:46:50.859186+0000 osd.5 (osd.5) 118 : cluster [DBG] 2.20 scrub ok 2021-02-10T04:46:52.834 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:51 smithi085 conmon[35961]: cluster 2021-02-10T04:46:52.835 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:51 smithi085 conmon[35961]: 2021-02 2021-02-10T04:46:52.835 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:51 smithi085 conmon[35961]: -10T04:46:50.327785+0000 mgr.y (mgr.14140) 1033 : cluster [DBG] 2021-02-10T04:46:52.835 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:51 smithi085 conmon[35961]: pgmap v1806: 188 pgs: 188 active+clean; 845 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:46:52.835 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:51 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:46:52.835 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:51 smithi085 conmon[35961]: -02-10T04:46:50.598535+0000 2021-02-10T04:46:52.835 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:51 smithi085 conmon[35961]: osd.4 (osd.4) 1968 : cluster [WRN] 2021-02-10T04:46:52.835 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:51 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:52.835 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:51 smithi085 conmon[35961]: cluster 2021-02-10T04:46:52.835 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:51 smithi085 conmon[35961]: 2021-02-10T04:46:50.598546+0000 osd.4 (osd.4) 2021-02-10T04:46:52.835 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:51 smithi085 conmon[35961]: 1969 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:52.835 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:51 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:46:52.835 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:51 smithi085 conmon[35961]: :46:50.598553+0000 osd.4 (osd.4 2021-02-10T04:46:52.835 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:51 smithi085 conmon[35961]: ) 1970 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:52.836 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:51 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:46:52.836 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:51 smithi085 conmon[35961]: T04:46:50.598560+0000 osd.4 (osd.4) 1971 2021-02-10T04:46:52.836 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:51 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:52.836 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:51 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:46:52.836 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:51 smithi085 conmon[35961]: :46:50.859186+0000 osd.5 (osd.5 2021-02-10T04:46:52.836 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:51 smithi085 conmon[35961]: ) 118 : cluster [DBG] 2.20 scrub ok 2021-02-10T04:46:52.876 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:52 smithi164 conmon[33677]: debug 2021-02-10T04:46:52.602+0000 7fbec19a1700 -1 osd.4 1060 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:46:52.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:52 smithi164 conmon[29288]: cluster 2021-02-10T04:46:51.596265+0000 osd.4 (osd.4) 1972 : cluster [WRN] 2021-02-10T04:46:52.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:52 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:52.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:52 smithi164 conmon[29288]: cluster 2021-02-10T04:46:51.596284+0000 osd.4 (osd.4) 1973 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:52.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:52 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:46:52.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:52 smithi164 conmon[29288]: -10T04:46:51.596291+0000 osd.4 (osd.4) 1974 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:52.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:52 smithi164 conmon[29288]: cluster 2021-02-10T04:46: 2021-02-10T04:46:52.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:52 smithi164 conmon[29288]: 51.596297+0000 osd.4 (osd.4) 1975 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:52.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:52 smithi164 conmon[29288]: cluster 2021-02-10T04:46:52.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:52 smithi164 conmon[29288]: 2021-02-10T04:46:51.840041 2021-02-10T04:46:52.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:52 smithi164 conmon[29288]: +0000 osd.5 (osd.5) 119 : 2021-02-10T04:46:52.878 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:52 smithi164 conmon[29288]: cluster [DBG] 4.8 scrub ok 2021-02-10T04:46:52.932 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:46:52.968 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:46:53.000 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:46:53.001 INFO:teuthology.orchestra.run.smithi164.stdout: "size": 20, 2021-02-10T04:46:53.001 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 600, 2021-02-10T04:46:53.001 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [ 2021-02-10T04:46:53.001 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.001 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:6628 7.7 7:e498d07f:::smithi08575397-21:head [copy-from ver 633] snapc 238=[238,237,236,234,233,231,22f,22e,22a,229,227,224,223,221,21f,21c,218,215,214,212,211,210,20a,203,1c1,1b6] ondisk+write+known_if_redirected e674)", 2021-02-10T04:46:53.001 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:38:12.279862+0000", 2021-02-10T04:46:53.001 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 520.71564793799996, 2021-02-10T04:46:53.001 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.19974309000000001, 2021-02-10T04:46:53.002 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:46:53.002 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:46:53.002 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:46:53.002 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:46:53.002 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:46:53.002 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 6628 2021-02-10T04:46:53.002 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.003 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:46:53.003 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.003 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:46:53.003 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:38:12.279862+0000", 2021-02-10T04:46:53.003 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:46:53.003 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.003 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.003 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:46:53.004 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:38:12.279862+0000", 2021-02-10T04:46:53.004 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9147000000000002e-05 2021-02-10T04:46:53.004 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.004 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.004 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:46:53.004 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:38:12.279911+0000", 2021-02-10T04:46:53.004 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.1610000000000002e-06 2021-02-10T04:46:53.004 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.005 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.005 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:46:53.005 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:38:12.279916+0000", 2021-02-10T04:46:53.005 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.5289999999999997e-06 2021-02-10T04:46:53.005 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.005 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.006 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:46:53.006 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:38:12.279923+0000", 2021-02-10T04:46:53.006 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.236e-06 2021-02-10T04:46:53.006 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.006 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.006 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:46:53.006 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:38:12.279926+0000", 2021-02-10T04:46:53.006 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9759000000000003e-05 2021-02-10T04:46:53.007 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.007 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.007 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.007 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:38:12.279976+0000", 2021-02-10T04:46:53.007 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000113689 2021-02-10T04:46:53.007 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.007 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.007 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:46:53.007 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:38:12.280090+0000", 2021-02-10T04:46:53.007 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.18951578199999999 2021-02-10T04:46:53.007 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.007 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.007 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 0,3", 2021-02-10T04:46:53.008 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:38:12.469605+0000", 2021-02-10T04:46:53.008 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00387741 2021-02-10T04:46:53.008 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.008 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.008 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:46:53.008 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:38:12.473483+0000", 2021-02-10T04:46:53.008 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0058778620000000002 2021-02-10T04:46:53.008 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.008 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.008 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:46:53.008 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:38:12.479361+0000", 2021-02-10T04:46:53.008 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.0661000000000002e-05 2021-02-10T04:46:53.008 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.009 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.009 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:46:53.009 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:38:12.479411+0000", 2021-02-10T04:46:53.009 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.2238999999999996e-05 2021-02-10T04:46:53.009 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.009 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.009 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:46:53.009 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:38:12.479504+0000", 2021-02-10T04:46:53.009 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000101615 2021-02-10T04:46:53.009 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.009 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.009 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:46:53.009 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:38:12.479605+0000", 2021-02-10T04:46:53.009 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.19974309000000001 2021-02-10T04:46:53.010 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.010 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:46:53.010 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.010 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.010 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.010 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:6629 7.7 7:e498d07f:::smithi08575397-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e674)", 2021-02-10T04:46:53.010 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:38:12.279961+0000", 2021-02-10T04:46:53.010 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 520.715549162, 2021-02-10T04:46:53.010 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.19999650099999999, 2021-02-10T04:46:53.010 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:46:53.010 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:46:53.010 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:46:53.010 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:46:53.011 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:46:53.011 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 6629 2021-02-10T04:46:53.011 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.011 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:46:53.011 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.011 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:46:53.011 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:38:12.279961+0000", 2021-02-10T04:46:53.011 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:46:53.011 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.011 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.011 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:46:53.011 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:38:12.279961+0000", 2021-02-10T04:46:53.011 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.2829e-05 2021-02-10T04:46:53.012 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.012 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.012 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:46:53.012 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:38:12.279994+0000", 2021-02-10T04:46:53.012 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0510000000000002e-06 2021-02-10T04:46:53.012 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.012 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.012 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:46:53.012 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:38:12.280000+0000", 2021-02-10T04:46:53.012 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1026000000000001e-05 2021-02-10T04:46:53.012 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.012 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.012 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:46:53.012 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:38:12.280011+0000", 2021-02-10T04:46:53.013 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.8250000000000004e-06 2021-02-10T04:46:53.013 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.013 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.013 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:46:53.013 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:38:12.280016+0000", 2021-02-10T04:46:53.013 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00024596200000000001 2021-02-10T04:46:53.013 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.013 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.013 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.013 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:38:12.280262+0000", 2021-02-10T04:46:53.013 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.1379999999999998e-05 2021-02-10T04:46:53.013 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.014 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.014 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:46:53.014 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:38:12.280323+0000", 2021-02-10T04:46:53.014 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.19251802700000001 2021-02-10T04:46:53.014 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.014 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.014 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.015 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:38:12.472841+0000", 2021-02-10T04:46:53.015 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000138529 2021-02-10T04:46:53.015 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.015 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.015 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2021-02-10T04:46:53.015 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:38:12.472980+0000", 2021-02-10T04:46:53.015 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0066710500000000004 2021-02-10T04:46:53.015 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.015 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.015 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.016 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:38:12.479651+0000", 2021-02-10T04:46:53.016 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00015248799999999999 2021-02-10T04:46:53.016 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.016 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.016 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:46:53.016 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:38:12.479803+0000", 2021-02-10T04:46:53.016 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000154334 2021-02-10T04:46:53.016 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.016 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.017 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:46:53.017 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:38:12.479957+0000", 2021-02-10T04:46:53.017 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.19999650099999999 2021-02-10T04:46:53.017 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.017 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:46:53.017 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.017 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.017 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.017 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:8015 7.7 7:e498d07f:::smithi08575397-21:head [copy-from ver 1439] snapc 2b9=[2b8,2b4,2b3,2b2,2b1,2ae,2ac,2aa,2a9,2a7,2a5,2a3,2a2,2a1,29d,298,295,294,28f,28b,289,27c,26d,24e,249,245,231,21c,203] ondisk+write+known_if_redirected e806)", 2021-02-10T04:46:53.017 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:40:25.907679+0000", 2021-02-10T04:46:53.017 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 387.08783067600001, 2021-02-10T04:46:53.018 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.042626618999999998, 2021-02-10T04:46:53.018 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:46:53.018 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:46:53.018 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:46:53.018 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:46:53.018 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:46:53.018 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 8015 2021-02-10T04:46:53.018 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.018 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:46:53.018 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.018 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:46:53.018 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:25.907679+0000", 2021-02-10T04:46:53.018 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:46:53.019 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.019 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.019 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:46:53.019 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:25.907679+0000", 2021-02-10T04:46:53.019 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2408e-05 2021-02-10T04:46:53.019 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.019 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.019 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:46:53.019 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:25.907702+0000", 2021-02-10T04:46:53.019 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5940000000000004e-06 2021-02-10T04:46:53.019 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.019 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.019 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:46:53.020 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:25.907706+0000", 2021-02-10T04:46:53.020 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.6500000000000008e-06 2021-02-10T04:46:53.020 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.020 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.020 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:46:53.020 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:25.907716+0000", 2021-02-10T04:46:53.020 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5789000000000001e-05 2021-02-10T04:46:53.020 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.020 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.020 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:46:53.020 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:25.907732+0000", 2021-02-10T04:46:53.020 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00028310999999999999 2021-02-10T04:46:53.020 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.021 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.021 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.021 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:25.908015+0000", 2021-02-10T04:46:53.021 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000119274 2021-02-10T04:46:53.021 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.021 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.021 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:46:53.021 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:25.908134+0000", 2021-02-10T04:46:53.021 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.022240320000000001 2021-02-10T04:46:53.021 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.021 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.021 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 0,3", 2021-02-10T04:46:53.021 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:25.930375+0000", 2021-02-10T04:46:53.021 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0082387250000000006 2021-02-10T04:46:53.022 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.022 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.022 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:46:53.022 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:25.938613+0000", 2021-02-10T04:46:53.022 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0088632559999999999 2021-02-10T04:46:53.022 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.022 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.022 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:46:53.022 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:25.947477+0000", 2021-02-10T04:46:53.022 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.002680694 2021-02-10T04:46:53.022 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.022 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.022 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:46:53.023 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:25.950157+0000", 2021-02-10T04:46:53.023 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.3811000000000003e-05 2021-02-10T04:46:53.023 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.023 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.023 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:46:53.023 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:25.950221+0000", 2021-02-10T04:46:53.023 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.4987999999999993e-05 2021-02-10T04:46:53.023 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.023 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.023 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:46:53.023 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:25.950306+0000", 2021-02-10T04:46:53.023 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.042626618999999998 2021-02-10T04:46:53.023 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.023 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:46:53.024 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.024 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.024 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.024 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:8016 7.7 7:e498d07f:::smithi08575397-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e806)", 2021-02-10T04:46:53.024 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:40:25.907767+0000", 2021-02-10T04:46:53.024 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 387.08774258800003, 2021-02-10T04:46:53.024 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.042747724000000001, 2021-02-10T04:46:53.024 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:46:53.024 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:46:53.024 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:46:53.024 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:46:53.024 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:46:53.024 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 8016 2021-02-10T04:46:53.025 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.025 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:46:53.025 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.025 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:46:53.025 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:25.907767+0000", 2021-02-10T04:46:53.025 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:46:53.025 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.025 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.025 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:46:53.025 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:25.907767+0000", 2021-02-10T04:46:53.025 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8268000000000001e-05 2021-02-10T04:46:53.025 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.025 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.025 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:46:53.026 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:25.907796+0000", 2021-02-10T04:46:53.026 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.8019999999999997e-06 2021-02-10T04:46:53.026 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.026 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.026 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:46:53.026 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:25.907802+0000", 2021-02-10T04:46:53.026 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0906e-05 2021-02-10T04:46:53.026 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.026 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.026 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:46:53.026 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:25.907812+0000", 2021-02-10T04:46:53.026 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.271e-06 2021-02-10T04:46:53.026 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.026 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.027 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:46:53.027 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:25.907818+0000", 2021-02-10T04:46:53.027 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00055674999999999995 2021-02-10T04:46:53.027 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.027 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.027 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.027 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:25.908374+0000", 2021-02-10T04:46:53.027 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000197307 2021-02-10T04:46:53.027 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.027 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.027 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:46:53.027 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:25.908572+0000", 2021-02-10T04:46:53.027 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.029268829999999999 2021-02-10T04:46:53.028 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.028 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.028 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.028 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:25.937841+0000", 2021-02-10T04:46:53.028 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000141488 2021-02-10T04:46:53.028 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.028 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.028 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2021-02-10T04:46:53.028 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:25.937982+0000", 2021-02-10T04:46:53.028 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012350767 2021-02-10T04:46:53.028 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.028 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.028 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.028 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:25.950333+0000", 2021-02-10T04:46:53.029 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.7538000000000001e-05 2021-02-10T04:46:53.029 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.029 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.029 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:46:53.029 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:25.950420+0000", 2021-02-10T04:46:53.029 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.4796999999999994e-05 2021-02-10T04:46:53.029 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.029 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.029 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:46:53.029 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:25.950515+0000", 2021-02-10T04:46:53.029 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.042747724000000001 2021-02-10T04:46:53.029 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.029 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:46:53.030 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.030 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.030 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.030 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:8103 7.d 7:b09ccbba:::smithi08575397-1:head [copy-from ver 1552] snapc 2c7=[2c7,2c1,2be,2bd,2b4,2b3,2b2,2b1,2ae,2ac,2aa,2a9,2a7,2a5,2a3,2a2,2a1,29d,298,295,294,289,27c,26d,24e,231,21c,203] ondisk+write+known_if_redirected e817)", 2021-02-10T04:46:53.030 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:40:36.470206+0000", 2021-02-10T04:46:53.030 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 376.52530391099998, 2021-02-10T04:46:53.030 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.058048001000000002, 2021-02-10T04:46:53.030 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:46:53.030 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:46:53.030 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:46:53.030 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:46:53.030 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:46:53.030 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 8103 2021-02-10T04:46:53.030 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.031 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:46:53.031 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.031 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:46:53.031 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.470206+0000", 2021-02-10T04:46:53.031 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:46:53.031 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.031 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.031 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:46:53.031 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.470206+0000", 2021-02-10T04:46:53.031 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5265000000000001e-05 2021-02-10T04:46:53.031 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.031 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.032 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:46:53.032 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.470251+0000", 2021-02-10T04:46:53.032 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.2740000000000004e-06 2021-02-10T04:46:53.032 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.032 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.032 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:46:53.032 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.470258+0000", 2021-02-10T04:46:53.032 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2533e-05 2021-02-10T04:46:53.032 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.032 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.032 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:46:53.032 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.470270+0000", 2021-02-10T04:46:53.032 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.866e-06 2021-02-10T04:46:53.032 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.033 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.033 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:46:53.033 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.470277+0000", 2021-02-10T04:46:53.033 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000100834 2021-02-10T04:46:53.033 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.033 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.033 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.033 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.470378+0000", 2021-02-10T04:46:53.033 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.5010000000000004e-06 2021-02-10T04:46:53.033 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.033 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.033 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2021-02-10T04:46:53.033 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.470386+0000", 2021-02-10T04:46:53.034 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0010073759999999999 2021-02-10T04:46:53.034 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.034 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.034 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.034 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.471394+0000", 2021-02-10T04:46:53.034 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000104145 2021-02-10T04:46:53.034 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.034 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.034 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:46:53.034 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.471498+0000", 2021-02-10T04:46:53.034 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.047454280000000001 2021-02-10T04:46:53.034 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.035 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.035 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 0,1", 2021-02-10T04:46:53.035 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.518952+0000", 2021-02-10T04:46:53.035 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0037624199999999998 2021-02-10T04:46:53.035 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.035 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.035 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:46:53.035 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.522715+0000", 2021-02-10T04:46:53.035 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0036273550000000001 2021-02-10T04:46:53.035 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.035 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.035 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:46:53.035 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.526342+0000", 2021-02-10T04:46:53.035 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0018082910000000001 2021-02-10T04:46:53.036 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.036 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.036 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:46:53.036 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.528150+0000", 2021-02-10T04:46:53.036 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.6406000000000003e-05 2021-02-10T04:46:53.036 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.036 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.036 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:46:53.036 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.528197+0000", 2021-02-10T04:46:53.036 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.7454999999999998e-05 2021-02-10T04:46:53.036 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.036 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.036 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:46:53.036 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.528254+0000", 2021-02-10T04:46:53.037 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.058048001000000002 2021-02-10T04:46:53.037 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.037 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:46:53.037 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.037 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.037 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.037 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:8104 7.d 7:b09ccbba:::smithi08575397-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e817)", 2021-02-10T04:46:53.037 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:40:36.471768+0000", 2021-02-10T04:46:53.037 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 376.523742029, 2021-02-10T04:46:53.037 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.056740802, 2021-02-10T04:46:53.037 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:46:53.037 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:46:53.037 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:46:53.038 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:46:53.038 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:46:53.038 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 8104 2021-02-10T04:46:53.038 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.038 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:46:53.038 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.038 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:46:53.038 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.471768+0000", 2021-02-10T04:46:53.038 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:46:53.038 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.038 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.038 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:46:53.038 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.471768+0000", 2021-02-10T04:46:53.038 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.7152999999999999e-05 2021-02-10T04:46:53.039 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.039 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.039 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:46:53.039 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.471795+0000", 2021-02-10T04:46:53.039 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0990000000000004e-06 2021-02-10T04:46:53.039 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.039 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.039 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:46:53.039 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.471801+0000", 2021-02-10T04:46:53.039 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1206000000000001e-05 2021-02-10T04:46:53.039 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.039 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.039 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:46:53.040 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.471812+0000", 2021-02-10T04:46:53.040 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.3940000000000004e-06 2021-02-10T04:46:53.040 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.040 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.040 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:46:53.040 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.471818+0000", 2021-02-10T04:46:53.040 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000146513 2021-02-10T04:46:53.040 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.040 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.040 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.040 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.471964+0000", 2021-02-10T04:46:53.040 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.7620999999999997e-05 2021-02-10T04:46:53.040 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.040 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:46:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.472012+0000", 2021-02-10T04:46:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.049990136999999997 2021-02-10T04:46:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.522002+0000", 2021-02-10T04:46:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.2163999999999995e-05 2021-02-10T04:46:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2021-02-10T04:46:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.522084+0000", 2021-02-10T04:46:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.006200807 2021-02-10T04:46:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.528285+0000", 2021-02-10T04:46:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000103692 2021-02-10T04:46:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:46:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.528389+0000", 2021-02-10T04:46:53.043 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000120016 2021-02-10T04:46:53.043 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.043 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.043 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:46:53.043 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.528509+0000", 2021-02-10T04:46:53.044 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.056740802 2021-02-10T04:46:53.044 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.044 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:46:53.044 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.044 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.044 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.044 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:8134 7.d 7:b4869c53:::smithi08575397-8:head [copy-from ver 1554] snapc 2ca=[2ca,2c9,2c8,2c7,2c1,2be,2bd,2b4,2b3,2b2,2b1,2ae,2ac,2aa,2a9,2a7,2a5,2a3,2a2,2a1,29d,298,295,294,289,27c,26d,24e,231,21c,203] ondisk+write+known_if_redirected e820)", 2021-02-10T04:46:53.044 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:40:39.284258+0000", 2021-02-10T04:46:53.045 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 373.71125223199999, 2021-02-10T04:46:53.045 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.045214365999999999, 2021-02-10T04:46:53.045 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:46:53.045 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:46:53.045 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:46:53.045 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:46:53.045 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:46:53.045 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 8134 2021-02-10T04:46:53.046 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.046 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:46:53.046 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.046 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:46:53.046 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.284258+0000", 2021-02-10T04:46:53.046 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:46:53.046 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.046 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.046 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:46:53.046 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.284258+0000", 2021-02-10T04:46:53.046 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9895999999999998e-05 2021-02-10T04:46:53.047 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.047 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.047 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:46:53.047 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.284308+0000", 2021-02-10T04:46:53.047 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0421999999999999e-05 2021-02-10T04:46:53.047 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.047 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.047 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:46:53.047 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.284318+0000", 2021-02-10T04:46:53.047 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2187e-05 2021-02-10T04:46:53.047 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.047 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.047 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:46:53.048 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.284340+0000", 2021-02-10T04:46:53.048 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1131000000000001e-05 2021-02-10T04:46:53.048 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.048 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.048 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:46:53.048 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.284351+0000", 2021-02-10T04:46:53.048 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.5407e-05 2021-02-10T04:46:53.048 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.048 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.048 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.048 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.284407+0000", 2021-02-10T04:46:53.048 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.6250000000000002e-06 2021-02-10T04:46:53.048 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.048 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.049 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2021-02-10T04:46:53.049 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.284417+0000", 2021-02-10T04:46:53.049 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0029206280000000002 2021-02-10T04:46:53.049 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.049 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.049 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.049 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.287337+0000", 2021-02-10T04:46:53.049 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00013877400000000001 2021-02-10T04:46:53.049 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.049 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.049 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:46:53.049 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.287476+0000", 2021-02-10T04:46:53.049 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.032017190000000001 2021-02-10T04:46:53.050 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.050 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.050 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 0,1", 2021-02-10T04:46:53.050 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.319493+0000", 2021-02-10T04:46:53.050 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0027806599999999999 2021-02-10T04:46:53.050 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.050 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.050 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:46:53.050 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.322274+0000", 2021-02-10T04:46:53.050 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0066410779999999999 2021-02-10T04:46:53.050 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.050 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.051 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:46:53.051 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.328915+0000", 2021-02-10T04:46:53.051 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00035480499999999998 2021-02-10T04:46:53.051 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.051 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.051 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:46:53.051 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.329270+0000", 2021-02-10T04:46:53.051 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.8529000000000007e-05 2021-02-10T04:46:53.051 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.051 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.051 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:46:53.051 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.329348+0000", 2021-02-10T04:46:53.052 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000124034 2021-02-10T04:46:53.052 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.052 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.052 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:46:53.052 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.329472+0000", 2021-02-10T04:46:53.052 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.045214365999999999 2021-02-10T04:46:53.052 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.052 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:46:53.052 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.052 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.052 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.052 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:8135 7.d 7:b4869c53:::smithi08575397-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e820)", 2021-02-10T04:46:53.052 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:40:39.284397+0000", 2021-02-10T04:46:53.053 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 373.71111304300001, 2021-02-10T04:46:53.053 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.045370463, 2021-02-10T04:46:53.053 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:46:53.053 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:46:53.053 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:46:53.053 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:46:53.053 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:46:53.053 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 8135 2021-02-10T04:46:53.053 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.053 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:46:53.053 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.053 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:46:53.053 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.284397+0000", 2021-02-10T04:46:53.053 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:46:53.054 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.054 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.054 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:46:53.054 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.284397+0000", 2021-02-10T04:46:53.054 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.9133999999999999e-05 2021-02-10T04:46:53.054 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.054 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.054 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:46:53.054 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.284426+0000", 2021-02-10T04:46:53.054 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.1550000000000004e-06 2021-02-10T04:46:53.054 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.054 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.054 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:46:53.054 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.284432+0000", 2021-02-10T04:46:53.055 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0885e-05 2021-02-10T04:46:53.055 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.055 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.055 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:46:53.055 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.284443+0000", 2021-02-10T04:46:53.055 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.1839999999999998e-06 2021-02-10T04:46:53.055 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.055 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.055 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:46:53.055 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.284448+0000", 2021-02-10T04:46:53.055 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.5123999999999998e-05 2021-02-10T04:46:53.055 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.056 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.056 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.056 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.284484+0000", 2021-02-10T04:46:53.056 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.2860000000000002e-06 2021-02-10T04:46:53.056 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.056 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.056 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting_for_map not empty", 2021-02-10T04:46:53.056 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.284491+0000", 2021-02-10T04:46:53.056 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0031584899999999999 2021-02-10T04:46:53.056 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.056 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.056 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.056 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.287649+0000", 2021-02-10T04:46:53.057 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.2240000000000001e-05 2021-02-10T04:46:53.057 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.057 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.057 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:46:53.057 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.287702+0000", 2021-02-10T04:46:53.057 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.033955863000000003 2021-02-10T04:46:53.057 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.057 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.057 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.057 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.321657+0000", 2021-02-10T04:46:53.057 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000135258 2021-02-10T04:46:53.057 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.057 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.057 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2021-02-10T04:46:53.058 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.321793+0000", 2021-02-10T04:46:53.058 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0077229519999999999 2021-02-10T04:46:53.058 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.058 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.058 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.058 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.329516+0000", 2021-02-10T04:46:53.058 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000105497 2021-02-10T04:46:53.058 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.058 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.058 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:46:53.058 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.329621+0000", 2021-02-10T04:46:53.058 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00014639500000000001 2021-02-10T04:46:53.058 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.059 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.059 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:46:53.059 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:39.329767+0000", 2021-02-10T04:46:53.059 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.045370463 2021-02-10T04:46:53.059 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.059 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:46:53.059 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.059 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.059 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.059 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:9135 7.7 7:e7e69144:::smithi08575397-40:head [copy-from ver 1801] snapc 328=[328,327,325,31d,31a,311,308,307,300,2fc,2f5,2f0,2ef,2ee,2ed,2eb,2ea,2e3,2e2,2df,2dd,2d9,2cb,2c9,2c8,2ae,2ac,2a2,27c,24e,203] ondisk+write+known_if_redirected e914)", 2021-02-10T04:46:53.059 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:42:13.316878+0000", 2021-02-10T04:46:53.059 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 279.67863157199997, 2021-02-10T04:46:53.059 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.087144743999999996, 2021-02-10T04:46:53.059 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:46:53.060 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:46:53.060 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:46:53.060 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:46:53.060 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:46:53.060 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 9135 2021-02-10T04:46:53.060 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.060 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:46:53.060 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.060 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:46:53.060 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.316878+0000", 2021-02-10T04:46:53.060 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:46:53.060 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.061 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.061 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:46:53.061 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.316878+0000", 2021-02-10T04:46:53.061 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.4186e-05 2021-02-10T04:46:53.061 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.061 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.061 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:46:53.061 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.316923+0000", 2021-02-10T04:46:53.061 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6629999999999998e-06 2021-02-10T04:46:53.061 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.061 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.061 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:46:53.062 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.316928+0000", 2021-02-10T04:46:53.062 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1773000000000001e-05 2021-02-10T04:46:53.062 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.062 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.062 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:46:53.062 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.316940+0000", 2021-02-10T04:46:53.062 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.6940000000000003e-06 2021-02-10T04:46:53.062 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.062 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.062 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:46:53.062 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.316945+0000", 2021-02-10T04:46:53.062 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00022126700000000001 2021-02-10T04:46:53.062 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.062 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.063 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.063 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.317166+0000", 2021-02-10T04:46:53.063 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.2390000000000002e-06 2021-02-10T04:46:53.063 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.063 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.063 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting_for_map not empty", 2021-02-10T04:46:53.063 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.317171+0000", 2021-02-10T04:46:53.063 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0077242430000000004 2021-02-10T04:46:53.063 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.063 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.063 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.063 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.324896+0000", 2021-02-10T04:46:53.064 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.5098000000000005e-05 2021-02-10T04:46:53.064 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.064 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.064 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:46:53.064 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.324971+0000", 2021-02-10T04:46:53.064 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.003291314 2021-02-10T04:46:53.064 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.064 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.064 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 0,3", 2021-02-10T04:46:53.064 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.328262+0000", 2021-02-10T04:46:53.064 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010870018 2021-02-10T04:46:53.064 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.064 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.064 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:46:53.065 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.339132+0000", 2021-02-10T04:46:53.065 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0627862 2021-02-10T04:46:53.065 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.065 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.065 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:46:53.065 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.401918+0000", 2021-02-10T04:46:53.065 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0019580740000000002 2021-02-10T04:46:53.065 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.065 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.065 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:46:53.065 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.403876+0000", 2021-02-10T04:46:53.065 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5034000000000003e-05 2021-02-10T04:46:53.065 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.065 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.066 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:46:53.066 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.403921+0000", 2021-02-10T04:46:53.066 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00010194100000000001 2021-02-10T04:46:53.066 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.066 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.066 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:46:53.066 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.404023+0000", 2021-02-10T04:46:53.066 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.087144743999999996 2021-02-10T04:46:53.066 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.066 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:46:53.066 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.066 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.066 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.067 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:9136 7.7 7:e7e69144:::smithi08575397-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e914)", 2021-02-10T04:46:53.067 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:42:13.316976+0000", 2021-02-10T04:46:53.067 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 279.67853377699998, 2021-02-10T04:46:53.067 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.087266036000000005, 2021-02-10T04:46:53.067 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:46:53.067 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:46:53.067 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:46:53.067 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:46:53.067 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:46:53.067 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 9136 2021-02-10T04:46:53.067 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.067 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:46:53.067 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.068 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:46:53.068 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.316976+0000", 2021-02-10T04:46:53.068 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:46:53.068 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.068 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.068 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:46:53.068 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.316976+0000", 2021-02-10T04:46:53.068 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.7557000000000001e-05 2021-02-10T04:46:53.068 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.068 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.068 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:46:53.068 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.317004+0000", 2021-02-10T04:46:53.068 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6030000000000004e-06 2021-02-10T04:46:53.068 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.069 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.069 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:46:53.069 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.317009+0000", 2021-02-10T04:46:53.069 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3175e-05 2021-02-10T04:46:53.069 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.069 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.069 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:46:53.069 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.317023+0000", 2021-02-10T04:46:53.069 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9980000000000003e-06 2021-02-10T04:46:53.069 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.069 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.069 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:46:53.069 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.317028+0000", 2021-02-10T04:46:53.069 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00016252500000000001 2021-02-10T04:46:53.070 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.070 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.070 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.070 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.317190+0000", 2021-02-10T04:46:53.070 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.8539999999999996e-06 2021-02-10T04:46:53.070 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.070 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.070 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting_for_map not empty", 2021-02-10T04:46:53.070 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.317195+0000", 2021-02-10T04:46:53.070 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0078874380000000001 2021-02-10T04:46:53.070 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.070 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.070 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.071 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.325082+0000", 2021-02-10T04:46:53.071 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.4582000000000002e-05 2021-02-10T04:46:53.071 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.071 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.071 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:46:53.071 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.325117+0000", 2021-02-10T04:46:53.071 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0065912979999999998 2021-02-10T04:46:53.071 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.071 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.071 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.071 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.331708+0000", 2021-02-10T04:46:53.071 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.5375000000000001e-05 2021-02-10T04:46:53.071 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.072 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.072 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2021-02-10T04:46:53.072 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.331804+0000", 2021-02-10T04:46:53.072 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.072246911999999996 2021-02-10T04:46:53.072 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.072 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.072 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.072 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.404051+0000", 2021-02-10T04:46:53.072 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.9856000000000002e-05 2021-02-10T04:46:53.072 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.072 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.072 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:46:53.072 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.404140+0000", 2021-02-10T04:46:53.073 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000101863 2021-02-10T04:46:53.073 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.073 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.073 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:46:53.073 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:42:13.404242+0000", 2021-02-10T04:46:53.073 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.087266036000000005 2021-02-10T04:46:53.073 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.073 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:46:53.073 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.073 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.073 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.073 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:865 7.11 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29] snapc 37=[36,32,2e] RETRY=2 ondisk+retry+write+known_if_redirected e1009)", 2021-02-10T04:46:53.073 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:26.531400+0000", 2021-02-10T04:46:53.073 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 146.46411023900001, 2021-02-10T04:46:53.074 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0358940320000001, 2021-02-10T04:46:53.074 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:46:53.074 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:46:53.074 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:46:53.074 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:46:53.074 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:46:53.074 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 865 2021-02-10T04:46:53.074 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.074 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:46:53.074 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.074 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:46:53.074 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.531400+0000", 2021-02-10T04:46:53.074 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:46:53.075 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.075 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.075 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:46:53.075 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.531400+0000", 2021-02-10T04:46:53.075 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2644e-05 2021-02-10T04:46:53.075 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.075 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.075 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:46:53.075 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.531442+0000", 2021-02-10T04:46:53.075 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.8469999999999998e-06 2021-02-10T04:46:53.075 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.075 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.075 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:46:53.075 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.531449+0000", 2021-02-10T04:46:53.076 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2095e-05 2021-02-10T04:46:53.076 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.076 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.076 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:46:53.076 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.531461+0000", 2021-02-10T04:46:53.076 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.4970000000000004e-06 2021-02-10T04:46:53.076 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.076 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.076 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:46:53.076 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.531469+0000", 2021-02-10T04:46:53.076 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.2267999999999994e-05 2021-02-10T04:46:53.076 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.076 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.076 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.077 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.531531+0000", 2021-02-10T04:46:53.077 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.85e-06 2021-02-10T04:46:53.077 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.077 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.077 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2021-02-10T04:46:53.077 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.531540+0000", 2021-02-10T04:46:53.077 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0022664920000000002 2021-02-10T04:46:53.077 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.077 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.077 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.077 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.533807+0000", 2021-02-10T04:46:53.077 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.0450000000000001e-06 2021-02-10T04:46:53.077 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.078 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.078 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for peered", 2021-02-10T04:46:53.078 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.533815+0000", 2021-02-10T04:46:53.078 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0068737320000001 2021-02-10T04:46:53.078 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.078 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.078 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.078 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:27.540688+0000", 2021-02-10T04:46:53.078 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0012929669999999999 2021-02-10T04:46:53.078 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.078 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.078 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for degraded object", 2021-02-10T04:46:53.078 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:27.541981+0000", 2021-02-10T04:46:53.079 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0087132900000000003 2021-02-10T04:46:53.079 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.079 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.079 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.079 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:27.550695+0000", 2021-02-10T04:46:53.079 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.2730000000000003e-05 2021-02-10T04:46:53.079 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.079 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.079 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:46:53.079 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:27.550767+0000", 2021-02-10T04:46:53.079 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0066241829999999996 2021-02-10T04:46:53.079 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.079 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.079 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 3,7", 2021-02-10T04:46:53.080 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:27.557391+0000", 2021-02-10T04:46:53.080 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0082081610000000003 2021-02-10T04:46:53.080 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.080 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.080 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:46:53.080 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:27.565600+0000", 2021-02-10T04:46:53.080 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0014154759999999999 2021-02-10T04:46:53.080 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.080 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.080 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:46:53.080 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:27.567015+0000", 2021-02-10T04:46:53.080 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.3695999999999994e-05 2021-02-10T04:46:53.080 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.080 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.081 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:46:53.081 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:27.567089+0000", 2021-02-10T04:46:53.081 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.1721000000000003e-05 2021-02-10T04:46:53.081 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.081 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.081 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:46:53.081 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:27.567131+0000", 2021-02-10T04:46:53.081 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00016333799999999999 2021-02-10T04:46:53.081 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.081 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.081 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:46:53.081 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:27.567294+0000", 2021-02-10T04:46:53.081 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0358940320000001 2021-02-10T04:46:53.081 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.082 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:46:53.082 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.082 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.082 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.082 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:866 7.11 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1009)", 2021-02-10T04:46:53.082 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:26.531515+0000", 2021-02-10T04:46:53.082 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 146.46399540499999, 2021-02-10T04:46:53.082 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0360594249999999, 2021-02-10T04:46:53.082 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:46:53.082 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:46:53.082 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:46:53.082 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:46:53.082 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:46:53.083 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 866 2021-02-10T04:46:53.083 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.083 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:46:53.083 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.083 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:46:53.083 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.531515+0000", 2021-02-10T04:46:53.083 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:46:53.083 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.083 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.083 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:46:53.083 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.531515+0000", 2021-02-10T04:46:53.083 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.7936000000000002e-05 2021-02-10T04:46:53.083 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.084 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.084 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:46:53.084 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.531543+0000", 2021-02-10T04:46:53.084 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.8900000000000004e-06 2021-02-10T04:46:53.084 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.084 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.084 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:46:53.084 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.531548+0000", 2021-02-10T04:46:53.084 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0927999999999999e-05 2021-02-10T04:46:53.084 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.084 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.084 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:46:53.084 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.531559+0000", 2021-02-10T04:46:53.084 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.5600000000000001e-06 2021-02-10T04:46:53.085 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.085 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.085 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:46:53.085 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.531565+0000", 2021-02-10T04:46:53.085 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00068616100000000002 2021-02-10T04:46:53.085 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.085 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.085 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.085 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.532251+0000", 2021-02-10T04:46:53.085 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.2929999999999997e-06 2021-02-10T04:46:53.085 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.085 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.085 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting_for_map not empty", 2021-02-10T04:46:53.086 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.532257+0000", 2021-02-10T04:46:53.086 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001588445 2021-02-10T04:46:53.086 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.086 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.086 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.086 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.533846+0000", 2021-02-10T04:46:53.086 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7009999999999996e-06 2021-02-10T04:46:53.086 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.086 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.086 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for peered", 2021-02-10T04:46:53.086 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.533854+0000", 2021-02-10T04:46:53.086 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0081694830000001 2021-02-10T04:46:53.086 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.086 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.087 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.087 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:27.542023+0000", 2021-02-10T04:46:53.087 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.0058000000000002e-05 2021-02-10T04:46:53.087 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.087 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.087 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for degraded object", 2021-02-10T04:46:53.087 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:27.542063+0000", 2021-02-10T04:46:53.087 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.008814812 2021-02-10T04:46:53.087 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.087 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.087 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.087 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:27.550878+0000", 2021-02-10T04:46:53.087 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9184999999999997e-05 2021-02-10T04:46:53.088 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.088 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.088 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:46:53.088 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:27.550927+0000", 2021-02-10T04:46:53.088 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.011356125999999999 2021-02-10T04:46:53.088 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.088 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.088 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.088 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:27.562283+0000", 2021-02-10T04:46:53.088 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000116866 2021-02-10T04:46:53.088 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.088 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.088 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2021-02-10T04:46:53.088 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:27.562400+0000", 2021-02-10T04:46:53.089 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0049265660000000003 2021-02-10T04:46:53.089 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.089 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.089 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.089 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:27.567327+0000", 2021-02-10T04:46:53.089 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0798999999999998e-05 2021-02-10T04:46:53.089 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.089 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.089 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:46:53.089 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:27.567387+0000", 2021-02-10T04:46:53.089 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00018661599999999999 2021-02-10T04:46:53.089 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.089 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.089 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:46:53.090 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:27.567574+0000", 2021-02-10T04:46:53.090 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0360594249999999 2021-02-10T04:46:53.090 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.090 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:46:53.090 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.090 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.090 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.090 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:2821 7.11 7:8eb2e6b5:::smithi08575397-27:head [copy-from ver 168] snapc cc=[cc,cb,c9] RETRY=2 ondisk+retry+write+known_if_redirected e1009)", 2021-02-10T04:46:53.090 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:26.531603+0000", 2021-02-10T04:46:53.090 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 146.463906788, 2021-02-10T04:46:53.090 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0854499309999999, 2021-02-10T04:46:53.090 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:46:53.090 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:46:53.091 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:46:53.091 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:46:53.091 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:46:53.091 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 2821 2021-02-10T04:46:53.091 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.091 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:46:53.091 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.091 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:46:53.091 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.531603+0000", 2021-02-10T04:46:53.091 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:46:53.091 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.091 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.092 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:46:53.092 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.531603+0000", 2021-02-10T04:46:53.092 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.4869e-05 2021-02-10T04:46:53.092 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.092 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.092 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:46:53.092 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.531648+0000", 2021-02-10T04:46:53.092 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.4370000000000004e-06 2021-02-10T04:46:53.092 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.092 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.092 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:46:53.092 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.531655+0000", 2021-02-10T04:46:53.092 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6852000000000001e-05 2021-02-10T04:46:53.093 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.093 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.093 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:46:53.093 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.531671+0000", 2021-02-10T04:46:53.093 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.7640000000000002e-06 2021-02-10T04:46:53.093 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.093 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.093 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:46:53.093 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.531677+0000", 2021-02-10T04:46:53.093 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.006804432 2021-02-10T04:46:53.093 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.093 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.093 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.093 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.538482+0000", 2021-02-10T04:46:53.094 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.1719999999999996e-06 2021-02-10T04:46:53.094 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.094 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.094 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for peered", 2021-02-10T04:46:53.094 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.538491+0000", 2021-02-10T04:46:53.094 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0036052740000001 2021-02-10T04:46:53.094 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.094 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.094 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.094 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:27.542096+0000", 2021-02-10T04:46:53.094 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001391598 2021-02-10T04:46:53.094 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.094 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.095 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for degraded object", 2021-02-10T04:46:53.095 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:27.543488+0000", 2021-02-10T04:46:53.095 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0038396789999999999 2021-02-10T04:46:53.095 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.095 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.095 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.095 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:27.547327+0000", 2021-02-10T04:46:53.095 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000118177 2021-02-10T04:46:53.095 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.095 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.095 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:46:53.095 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:27.547446+0000", 2021-02-10T04:46:53.095 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0617325639999999 2021-02-10T04:46:53.096 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.096 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.096 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 3,7", 2021-02-10T04:46:53.096 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.609178+0000", 2021-02-10T04:46:53.096 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0056276779999999997 2021-02-10T04:46:53.096 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.096 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.096 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:46:53.096 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.614806+0000", 2021-02-10T04:46:53.096 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001688293 2021-02-10T04:46:53.096 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.096 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.096 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:46:53.096 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.616494+0000", 2021-02-10T04:46:53.097 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000421882 2021-02-10T04:46:53.097 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.097 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.097 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:46:53.097 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.616916+0000", 2021-02-10T04:46:53.097 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.1493000000000002e-05 2021-02-10T04:46:53.097 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.097 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.097 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:46:53.097 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.616977+0000", 2021-02-10T04:46:53.097 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.5766999999999995e-05 2021-02-10T04:46:53.097 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.097 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.097 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:46:53.098 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.617053+0000", 2021-02-10T04:46:53.098 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0854499309999999 2021-02-10T04:46:53.098 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.098 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:46:53.098 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.098 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.098 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.098 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:2822 7.11 7:8eb2e6b5:::smithi08575397-27:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1009)", 2021-02-10T04:46:53.098 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:26.531723+0000", 2021-02-10T04:46:53.098 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 146.463786683, 2021-02-10T04:46:53.098 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0855016659999999, 2021-02-10T04:46:53.098 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:46:53.098 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:46:53.099 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:46:53.099 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:46:53.099 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:46:53.099 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 2822 2021-02-10T04:46:53.099 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.099 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:46:53.099 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.099 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:46:53.099 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.531723+0000", 2021-02-10T04:46:53.099 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:46:53.099 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.099 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.099 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:46:53.099 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.531723+0000", 2021-02-10T04:46:53.100 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.0499999999999999e-05 2021-02-10T04:46:53.100 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.100 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.100 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:46:53.100 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.531754+0000", 2021-02-10T04:46:53.100 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.2090000000000002e-06 2021-02-10T04:46:53.100 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.100 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.100 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:46:53.100 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.531760+0000", 2021-02-10T04:46:53.100 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1235e-05 2021-02-10T04:46:53.100 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.100 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.101 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:46:53.101 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.531771+0000", 2021-02-10T04:46:53.101 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.2669999999999997e-06 2021-02-10T04:46:53.101 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.101 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.101 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:46:53.101 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.531777+0000", 2021-02-10T04:46:53.101 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0067358299999999999 2021-02-10T04:46:53.101 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.101 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.101 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.101 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.538512+0000", 2021-02-10T04:46:53.101 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0159999999999997e-06 2021-02-10T04:46:53.101 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.102 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.102 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for peered", 2021-02-10T04:46:53.102 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.538518+0000", 2021-02-10T04:46:53.102 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0050432439999999 2021-02-10T04:46:53.102 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.102 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.102 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.102 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:27.543562+0000", 2021-02-10T04:46:53.102 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000104905 2021-02-10T04:46:53.102 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.102 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.102 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for degraded object", 2021-02-10T04:46:53.102 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:27.543667+0000", 2021-02-10T04:46:53.102 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0039921640000000003 2021-02-10T04:46:53.103 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.103 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.103 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.103 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:27.547659+0000", 2021-02-10T04:46:53.103 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.3837000000000001e-05 2021-02-10T04:46:53.103 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.103 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.103 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:46:53.103 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:27.547703+0000", 2021-02-10T04:46:53.103 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0651203 2021-02-10T04:46:53.103 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.103 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.103 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.103 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.612823+0000", 2021-02-10T04:46:53.104 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.6318999999999998e-05 2021-02-10T04:46:53.104 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.104 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.104 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for rw locks", 2021-02-10T04:46:53.104 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.612909+0000", 2021-02-10T04:46:53.104 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0041643799999999996 2021-02-10T04:46:53.104 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.104 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.104 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.104 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.617074+0000", 2021-02-10T04:46:53.104 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.4686999999999999e-05 2021-02-10T04:46:53.104 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.104 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.104 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:46:53.105 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.617138+0000", 2021-02-10T04:46:53.105 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.6773000000000004e-05 2021-02-10T04:46:53.105 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.105 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.105 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:46:53.105 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.617225+0000", 2021-02-10T04:46:53.105 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0855016659999999 2021-02-10T04:46:53.105 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.105 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:46:53.105 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.105 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.105 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.105 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPushReply(7.12 1040/1037 [PushReplyOp(7:4e1089df:::smithi08575397-36:32b)])", 2021-02-10T04:46:53.105 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:45:43.256098+0000", 2021-02-10T04:46:53.106 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 69.739412131999998, 2021-02-10T04:46:53.106 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010011730999999999, 2021-02-10T04:46:53.106 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:46:53.106 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T04:46:53.106 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:46:53.106 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.106 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:46:53.106 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:43.256098+0000", 2021-02-10T04:46:53.106 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:46:53.106 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.106 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.106 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:46:53.106 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:43.256098+0000", 2021-02-10T04:46:53.107 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6180000000000001e-05 2021-02-10T04:46:53.107 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.107 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.107 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:46:53.107 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:43.256114+0000", 2021-02-10T04:46:53.107 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9259999999999999e-06 2021-02-10T04:46:53.107 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.107 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.107 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:46:53.107 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:43.256119+0000", 2021-02-10T04:46:53.107 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.2749999999999998e-06 2021-02-10T04:46:53.107 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.107 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.107 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:46:53.108 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:43.256126+0000", 2021-02-10T04:46:53.108 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5050000000000001e-06 2021-02-10T04:46:53.108 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.108 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.108 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:46:53.108 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:43.256131+0000", 2021-02-10T04:46:53.108 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0097195949999999993 2021-02-10T04:46:53.108 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.108 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.108 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.108 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:43.265850+0000", 2021-02-10T04:46:53.108 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00025924999999999999 2021-02-10T04:46:53.108 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.108 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.109 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:46:53.109 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:43.266110+0000", 2021-02-10T04:46:53.109 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010011730999999999 2021-02-10T04:46:53.109 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.109 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:46:53.109 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.109 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.109 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.109 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPushReply(7.12 1041/1037 [PushReplyOp(7:4e1089df:::smithi08575397-36:358)])", 2021-02-10T04:46:53.109 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:45:44.017464+0000", 2021-02-10T04:46:53.109 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 68.978046074999995, 2021-02-10T04:46:53.109 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010633635000000001, 2021-02-10T04:46:53.109 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:46:53.110 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T04:46:53.110 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:46:53.110 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.110 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:46:53.110 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:44.017464+0000", 2021-02-10T04:46:53.110 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:46:53.110 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.110 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.110 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:46:53.110 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:44.017464+0000", 2021-02-10T04:46:53.110 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.358e-05 2021-02-10T04:46:53.110 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.110 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.110 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:46:53.111 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:44.017498+0000", 2021-02-10T04:46:53.111 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.2939999999999994e-06 2021-02-10T04:46:53.111 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.111 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.111 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:46:53.111 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:44.017507+0000", 2021-02-10T04:46:53.111 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6359999999999999e-05 2021-02-10T04:46:53.111 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.111 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.111 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:46:53.111 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:44.017523+0000", 2021-02-10T04:46:53.111 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.3000000000000002e-06 2021-02-10T04:46:53.111 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.111 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.112 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:46:53.112 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:44.017532+0000", 2021-02-10T04:46:53.112 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010213376999999999 2021-02-10T04:46:53.112 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.112 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.112 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.112 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:44.027745+0000", 2021-02-10T04:46:53.112 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00035272400000000001 2021-02-10T04:46:53.112 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.112 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.112 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:46:53.112 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:44.028098+0000", 2021-02-10T04:46:53.112 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010633635000000001 2021-02-10T04:46:53.112 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.113 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:46:53.113 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.113 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.113 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.113 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.24382.0:1513 4.a 4:50d64734:::obj_delete_at_hint.0000000009:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e1048)", 2021-02-10T04:46:53.113 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:45:56.293095+0000", 2021-02-10T04:46:53.113 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 56.702414574999999, 2021-02-10T04:46:53.113 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0038114820000000002, 2021-02-10T04:46:53.113 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:46:53.113 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:46:53.113 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:46:53.113 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.24382", 2021-02-10T04:46:53.113 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/3915233377", 2021-02-10T04:46:53.114 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 1513 2021-02-10T04:46:53.114 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.114 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:46:53.114 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.114 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:46:53.114 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:56.293095+0000", 2021-02-10T04:46:53.114 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:46:53.114 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.114 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.114 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:46:53.114 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:56.293095+0000", 2021-02-10T04:46:53.114 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.7287999999999999e-05 2021-02-10T04:46:53.114 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.115 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.115 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:46:53.115 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:56.293143+0000", 2021-02-10T04:46:53.115 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.8430000000000004e-06 2021-02-10T04:46:53.115 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.115 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.115 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:46:53.115 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:56.293150+0000", 2021-02-10T04:46:53.115 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2342e-05 2021-02-10T04:46:53.115 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.115 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.115 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:46:53.115 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:56.293162+0000", 2021-02-10T04:46:53.115 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0739999999999998e-06 2021-02-10T04:46:53.116 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.116 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.116 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:46:53.116 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:56.293168+0000", 2021-02-10T04:46:53.116 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.7540000000000002e-05 2021-02-10T04:46:53.116 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.116 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.116 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.116 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:56.293216+0000", 2021-02-10T04:46:53.116 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.5549000000000003e-05 2021-02-10T04:46:53.116 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.116 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.116 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:46:53.116 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:56.293311+0000", 2021-02-10T04:46:53.117 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000177572 2021-02-10T04:46:53.117 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.117 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.117 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 1,7", 2021-02-10T04:46:53.117 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:56.293489+0000", 2021-02-10T04:46:53.117 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00043564100000000001 2021-02-10T04:46:53.117 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.117 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.117 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:46:53.117 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:56.293924+0000", 2021-02-10T04:46:53.117 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00047226199999999999 2021-02-10T04:46:53.117 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.117 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.118 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:46:53.118 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:56.294397+0000", 2021-02-10T04:46:53.118 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00245648 2021-02-10T04:46:53.118 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.118 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.118 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:46:53.118 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:56.296853+0000", 2021-02-10T04:46:53.118 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.3312000000000002e-05 2021-02-10T04:46:53.118 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.118 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.118 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:46:53.118 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:56.296886+0000", 2021-02-10T04:46:53.118 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0579000000000001e-05 2021-02-10T04:46:53.118 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.119 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.119 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:46:53.119 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:56.296907+0000", 2021-02-10T04:46:53.119 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0038114820000000002 2021-02-10T04:46:53.119 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.119 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:46:53.119 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.119 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.119 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.119 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.24382.0:1580 4.1f 4:f93251ce:::obj_delete_at_hint.0000000042:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e1048)", 2021-02-10T04:46:53.119 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:45:56.436682+0000", 2021-02-10T04:46:53.119 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 56.558827733000001, 2021-02-10T04:46:53.120 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0023090379999999998, 2021-02-10T04:46:53.120 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:46:53.120 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:46:53.120 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:46:53.120 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.24382", 2021-02-10T04:46:53.120 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/3915233377", 2021-02-10T04:46:53.120 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 1580 2021-02-10T04:46:53.120 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.120 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:46:53.120 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.120 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:46:53.120 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:56.436682+0000", 2021-02-10T04:46:53.120 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:46:53.121 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.121 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.121 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:46:53.121 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:56.436682+0000", 2021-02-10T04:46:53.121 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.6471000000000001e-05 2021-02-10T04:46:53.121 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.121 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.121 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:46:53.121 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:56.436709+0000", 2021-02-10T04:46:53.121 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5589999999999999e-06 2021-02-10T04:46:53.121 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.122 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.122 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:46:53.122 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:56.436713+0000", 2021-02-10T04:46:53.122 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3879e-05 2021-02-10T04:46:53.122 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.122 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.122 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:46:53.122 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:56.436727+0000", 2021-02-10T04:46:53.122 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.2260000000000002e-06 2021-02-10T04:46:53.122 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.122 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.122 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:46:53.122 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:56.436733+0000", 2021-02-10T04:46:53.123 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.0584999999999999e-05 2021-02-10T04:46:53.123 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.123 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.123 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.123 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:56.436774+0000", 2021-02-10T04:46:53.123 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6959000000000001e-05 2021-02-10T04:46:53.123 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.123 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.123 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:46:53.123 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:56.436831+0000", 2021-02-10T04:46:53.123 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000100117 2021-02-10T04:46:53.123 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.123 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.123 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 1,5", 2021-02-10T04:46:53.124 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:56.436931+0000", 2021-02-10T04:46:53.124 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00040290399999999999 2021-02-10T04:46:53.124 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.124 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.124 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T04:46:53.124 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:56.437334+0000", 2021-02-10T04:46:53.124 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00050659399999999997 2021-02-10T04:46:53.124 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.124 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.124 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:46:53.124 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:56.437841+0000", 2021-02-10T04:46:53.124 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001075103 2021-02-10T04:46:53.124 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.124 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.125 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T04:46:53.125 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:56.438916+0000", 2021-02-10T04:46:53.125 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.4994999999999998e-05 2021-02-10T04:46:53.125 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.125 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.125 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:46:53.125 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:56.438971+0000", 2021-02-10T04:46:53.125 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0645999999999999e-05 2021-02-10T04:46:53.125 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.125 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.125 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:46:53.125 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:56.438991+0000", 2021-02-10T04:46:53.125 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0023090379999999998 2021-02-10T04:46:53.126 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.126 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:46:53.126 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.126 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.126 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.126 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDScrubReserve(5.f REQUEST e1059)", 2021-02-10T04:46:53.126 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:46:48.837838+0000", 2021-02-10T04:46:53.126 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 4.1576719449999997, 2021-02-10T04:46:53.126 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00061843499999999997, 2021-02-10T04:46:53.126 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:46:53.126 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:46:53.126 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:46:53.126 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.127 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:46:53.127 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:46:48.837838+0000", 2021-02-10T04:46:53.127 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:46:53.127 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.127 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.127 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:46:53.127 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:46:48.837838+0000", 2021-02-10T04:46:53.127 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.0241000000000001e-05 2021-02-10T04:46:53.127 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.127 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.127 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:46:53.127 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:46:48.837868+0000", 2021-02-10T04:46:53.127 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6810000000000001e-06 2021-02-10T04:46:53.127 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.127 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.128 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:46:53.128 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:46:48.837875+0000", 2021-02-10T04:46:53.128 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.1770000000000004e-06 2021-02-10T04:46:53.128 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.128 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.128 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:46:53.128 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:46:48.837884+0000", 2021-02-10T04:46:53.128 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.9399999999999999e-06 2021-02-10T04:46:53.128 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.128 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.128 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:46:53.128 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:46:48.837890+0000", 2021-02-10T04:46:53.128 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00052105200000000004 2021-02-10T04:46:53.129 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.129 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.129 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.129 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:46:48.838411+0000", 2021-02-10T04:46:53.129 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2238999999999999e-05 2021-02-10T04:46:53.129 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.129 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.129 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:46:53.129 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:46:48.838423+0000", 2021-02-10T04:46:53.129 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.3105000000000002e-05 2021-02-10T04:46:53.129 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.129 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.129 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:46:53.129 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:46:48.838457+0000", 2021-02-10T04:46:53.130 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00061843499999999997 2021-02-10T04:46:53.130 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.130 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:46:53.130 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.130 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.130 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.130 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDScrubReserve(5.f RELEASE e1059)", 2021-02-10T04:46:53.130 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:46:48.843235+0000", 2021-02-10T04:46:53.130 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 4.1522751360000001, 2021-02-10T04:46:53.130 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0032251409999999999, 2021-02-10T04:46:53.130 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:46:53.130 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:46:53.130 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:46:53.131 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.131 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:46:53.131 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:46:48.843235+0000", 2021-02-10T04:46:53.131 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:46:53.131 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.131 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.131 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:46:53.131 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:46:48.843235+0000", 2021-02-10T04:46:53.131 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.2783000000000001e-05 2021-02-10T04:46:53.131 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.131 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.131 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:46:53.131 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:46:48.843268+0000", 2021-02-10T04:46:53.132 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.3509999999999998e-06 2021-02-10T04:46:53.132 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.132 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.132 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:46:53.132 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:46:48.843274+0000", 2021-02-10T04:46:53.132 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.9490000000000008e-06 2021-02-10T04:46:53.132 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.132 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.132 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:46:53.132 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:46:48.843284+0000", 2021-02-10T04:46:53.132 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.7300000000000002e-06 2021-02-10T04:46:53.132 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.132 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.133 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:46:53.133 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:46:48.843290+0000", 2021-02-10T04:46:53.133 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0031600909999999999 2021-02-10T04:46:53.133 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.133 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.133 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:46:53.133 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:46:48.846450+0000", 2021-02-10T04:46:53.133 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.6109999999999996e-06 2021-02-10T04:46:53.133 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.133 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.133 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:46:53.133 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:46:48.846457+0000", 2021-02-10T04:46:53.133 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.6259999999999999e-06 2021-02-10T04:46:53.133 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:46:53.134 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:46:53.134 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:46:53.134 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:46:48.846460+0000", 2021-02-10T04:46:53.134 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0032251409999999999 2021-02-10T04:46:53.134 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.134 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:46:53.134 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.134 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:46:53.134 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:46:53.134 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:46:53.144 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[44544]: debug 2021-02-10T04:46:53.068+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:46:53.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[29319]: cluster 2021-02-10T04:46:53.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[29319]: 2021-02-10T04:46:53.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[29319]: 2021 2021-02-10T04:46:53.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[29319]: -02-10T04:46:51. 2021-02-10T04:46:53.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[29319]: 596265+0000 osd.4 (osd.4 2021-02-10T04:46:53.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[29319]: ) 1972 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:53.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[29319]: cluster 2021-02-10T04:46:53.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[29319]: 2021-02-10T04:46:53.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[29319]: 2021 2021-02-10T04:46:53.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[29319]: - 2021-02-10T04:46:53.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[29319]: 02 2021-02-10T04:46:53.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[29319]: - 2021-02-10T04:46:53.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[29319]: 10 2021-02-10T04:46:53.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[29319]: T04:46:51.596284+0000 2021-02-10T04:46:53.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[29319]: osd.4 (osd.4) 1973 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:53.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:46:53.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[29319]: T04:46:51.596291+0000 osd.4 (osd.4) 1974 2021-02-10T04:46:53.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:53.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[29319]: cluster 2021-02-10T04:46:53.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[29319]: 2021-02-10T04:46:51.596297+0000 osd.4 (osd.4) 1975 : cluster [WRN] 2021-02-10T04:46:53.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:53.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[29319]: cluster 2021-02-10T04:46:53.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[29319]: 2021-02-10T04:46:51.840041+0000 osd.5 (osd.5 2021-02-10T04:46:53.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[29319]: ) 119 : cluster [DBG] 4.8 scrub ok 2021-02-10T04:46:53.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[35961]: cluster 2021-02-10T04:46:53.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[35961]: 2021-02-10T04:46:53.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[35961]: 2021 2021-02-10T04:46:53.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[35961]: -02-10T04:46:51.596265 2021-02-10T04:46:53.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1972 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:53.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:46:53.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[35961]: 10T04:46:51.596284+0000 2021-02-10T04:46:53.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[35961]: osd.4 (osd.4) 1973 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:53.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[35961]: cluster 2021-02-10T04:46:53.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[35961]: 2021-02-10T04:46:53.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[35961]: 2021 2021-02-10T04:46:53.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[35961]: - 2021-02-10T04:46:53.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[35961]: 02 2021-02-10T04:46:53.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[35961]: - 2021-02-10T04:46:53.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[35961]: 10T04:46:51.596291 2021-02-10T04:46:53.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1974 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:53.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:46:53.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[35961]: 10T04:46:51.596297+0000 osd.4 (osd 2021-02-10T04:46:53.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[35961]: .4) 1975 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:53.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:46:53.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[35961]: 02-10T04:46:51.840041+0000 osd.5 (osd 2021-02-10T04:46:53.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:52 smithi085 conmon[35961]: .5) 119 : cluster [DBG] 4.8 scrub ok 2021-02-10T04:46:53.836 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:46:53.882 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[29288]: cluster 2021-02-10T04:46:52.328324+0000 mgr.y 2021-02-10T04:46:53.882 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[29288]: (mgr.14140) 1034 : cluster [DBG] pgmap v1807: 188 pgs: 188 active+clean; 845 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:46:53.882 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[29288]: cluster 2021-02-10T04:46:52.603614+0000 osd.4 (osd.4) 1976 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:53.882 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[29288]: 2021-02-10T04:46:53.882 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:46:53.883 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[29288]: 46:52.603622+0000 osd.4 (osd.4) 1977 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:53.883 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:46:53.883 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[29288]: 02-10T04:46: 2021-02-10T04:46:53.883 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[29288]: 52.603626+0000 osd.4 (osd.4) 1978 : 2021-02-10T04:46:53.883 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:53.883 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:46:53.883 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[29288]: 02-10T04:46:52.603630+0000 osd.4 (osd.4 2021-02-10T04:46:53.883 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[29288]: ) 1979 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:53.883 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:46:53.883 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[29288]: T04:46:52.869088+0000 mon.a ( 2021-02-10T04:46:53.883 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[29288]: mon.0) 2120 : cluster [WRN] 2021-02-10T04:46:53.883 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[29288]: Health check failed: 4 slow ops, oldest one blocked for 147 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:46:53.883 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:46:53.884 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[29288]: 10T04:46:53.011073+0000 osd.6 (osd 2021-02-10T04:46:53.884 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[29288]: .6) 120 : cluster [DBG] 3.0 scrub ok 2021-02-10T04:46:53.884 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:46:53.884 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[29288]: :46:53.407011+0000 mon.a (mon.0 2021-02-10T04:46:53.884 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[29288]: ) 2121 : cluster [DBG] osdmap e1061: 8 total, 8 up, 7 in 2021-02-10T04:46:53.884 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[29288]: cluster 2021-02-10T04:46:53.884 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[29288]: 2021-02-10T04:46:53.884 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[29288]: 2021-02-10T04:46:53 2021-02-10T04:46:53.884 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[29288]: .528220+0000 osd.3 (osd.3) 122 : cluster [DBG] 1.9 scrub ok 2021-02-10T04:46:53.884 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[33677]: debug 2021-02-10T04:46:53.884 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[33677]: 2021-02-10T04:46:53.651+0000 7fbec19a1700 -1 osd.4 1061 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:46:53.884 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:53 smithi164 conmon[33677]: 2021-02-10T04:46:54.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[29319]: cluster 2021-02-10T04:46:52.328324+0000 mgr.y 2021-02-10T04:46:54.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[29319]: (mgr.14140) 1034 : cluster [DBG] pgmap v1807: 188 pgs: 188 active+clean; 845 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:46:54.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:46:54.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[29319]: :46:52.603614+0000 osd.4 (osd.4) 1976 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:54.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:46:54.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[29319]: 04:46:52.603622+0000 osd.4 (osd.4 2021-02-10T04:46:54.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[29319]: ) 1977 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:54.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:46:54.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[29319]: 10T04:46:52.603626+0000 osd.4 (osd 2021-02-10T04:46:54.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[29319]: .4) 1978 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:54.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:46:54.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[29319]: 02-10T04:46:52.603630+0000 2021-02-10T04:46:54.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[29319]: osd.4 (osd.4) 1979 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:54.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[29319]: cluster 2021-02-10T04:46:54.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[29319]: 2021-02-10T04:46:54.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:46:54.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[29319]: 46:52.869088+0000 mon.a (mon.0) 2021-02-10T04:46:54.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[29319]: 2120 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 147 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:46:54.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[29319]: cluster 2021-02-10T04:46:54.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[29319]: 2021-02-10T04:46:53.011073+0000 osd.6 (osd.6) 120 : cluster [DBG] 3.0 scrub ok 2021-02-10T04:46:54.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[29319]: cluster 2021-02-10T04:46:53.407011+0000 2021-02-10T04:46:54.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[29319]: mon.a (mon.0) 2121 : cluster [DBG] osdmap e1061: 8 total, 8 up, 7 in 2021-02-10T04:46:54.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:46:54.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[29319]: 46:53.528220+0000 osd.3 (osd.3) 122 : cluster [DBG] 2021-02-10T04:46:54.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[29319]: 1.9 scrub ok 2021-02-10T04:46:54.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[35961]: cluster 2021-02-10T04:46:54.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[35961]: 2021-02-10T04:46 2021-02-10T04:46:54.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[35961]: :52.328324+0000 mgr.y (mgr.14140) 1034 : cluster [DBG] pgmap v1807: 188 pgs: 188 active+clean; 845 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:46:54.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:46:54.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[35961]: -10T04:46:52.603614+0000 osd.4 2021-02-10T04:46:54.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[35961]: (osd.4) 1976 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:54.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:46:54.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[35961]: -02-10T04:46:52.603622 2021-02-10T04:46:54.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2021-02-10T04:46:54.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[35961]: 1977 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:54.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:46:54.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[35961]: T04:46:52.603626+0000 osd.4 (osd. 2021-02-10T04:46:54.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[35961]: 4) 1978 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:54.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:46:54.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[35961]: :46:52.603630+0000 osd.4 (osd.4 2021-02-10T04:46:54.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[35961]: ) 1979 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:54.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:46:54.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[35961]: -02-10T04:46:52 2021-02-10T04:46:54.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[35961]: .869088+0000 mon.a (mon.0) 2120 : cluster 2021-02-10T04:46:54.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[35961]: [WRN] Health check failed: 4 slow ops, oldest one blocked for 147 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:46:54.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[35961]: cluster 2021-02-10T04:46:53 2021-02-10T04:46:54.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[35961]: .011073+0000 osd.6 (osd.6) 120 : cluster 2021-02-10T04:46:54.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[35961]: [DBG] 3.0 scrub ok 2021-02-10T04:46:54.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[35961]: cluster 2021-02-10T04:46:54.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[35961]: 2021-02-10T04:46: 2021-02-10T04:46:54.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[35961]: 53.407011+0000 mon.a (mon.0) 2021-02-10T04:46:54.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[35961]: 2121 : cluster [DBG] osdmap e1061: 8 total, 8 up, 7 in 2021-02-10T04:46:54.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:46:54.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[35961]: 10T04:46:53.528220+0000 osd.3 ( 2021-02-10T04:46:54.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:53 smithi085 conmon[35961]: osd.3) 122 : cluster [DBG] 1.9 scrub ok 2021-02-10T04:46:54.887 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:54 smithi164 conmon[33677]: debug 2021-02-10T04:46:54.630+0000 7fbec19a1700 -1 osd.4 1061 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:46:54.887 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:54 smithi164 conmon[29288]: cluster 2021-02-10T04:46:53.652660+0000 osd.4 (osd.4) 1980 : cluster [WRN] 2021-02-10T04:46:54.887 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:54 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:54.887 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:54 smithi164 conmon[29288]: cluster 2021-02-10T04:46:53.652669+0000 osd.4 (osd.4) 1981 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:54.887 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:54 smithi164 conmon[29288]: cluster 2021-02-10T04:46:53.652675+0000 osd.4 (osd.4) 1982 : cluster 2021-02-10T04:46:54.887 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:54 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:54.887 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:54 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:46:54.887 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:54 smithi164 conmon[29288]: :46:53.652681+0000 osd.4 (osd.4) 1983 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:55.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:54 smithi085 conmon[29319]: cluster 2021-02-10T04:46:53.652660+0000 osd.4 (osd.4) 1980 2021-02-10T04:46:55.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:54 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:55.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:54 smithi085 conmon[29319]: cluster 2021-02-10T04:46:53.652669+0000 osd.4 (osd.4) 1981 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:55.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:54 smithi085 conmon[29319]: cluster 2021-02-10T04:46:53.652675+0000 osd.4 (osd.4) 1982 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:55.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:54 smithi085 conmon[29319]: cluster 2021-02-10T04:46:53.652681+0000 2021-02-10T04:46:55.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:54 smithi085 conmon[29319]: osd.4 (osd.4) 1983 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:55.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:54 smithi085 conmon[35961]: cluster 2021-02-10T04:46:55.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:54 smithi085 conmon[35961]: 2021-02-10T04:46:55.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:54 smithi085 conmon[35961]: 2021 2021-02-10T04:46:55.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:54 smithi085 conmon[35961]: - 2021-02-10T04:46:55.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:54 smithi085 conmon[35961]: 02 2021-02-10T04:46:55.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:54 smithi085 conmon[35961]: - 2021-02-10T04:46:55.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:54 smithi085 conmon[35961]: 10 2021-02-10T04:46:55.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:54 smithi085 conmon[35961]: T04:46:53. 2021-02-10T04:46:55.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:54 smithi085 conmon[35961]: 652660+0000 osd.4 (osd.4) 1980 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:55.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:54 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:46:55.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:54 smithi085 conmon[35961]: 02-10T04:46:53.652669+0000 osd.4 ( 2021-02-10T04:46:55.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:54 smithi085 conmon[35961]: osd.4) 2021-02-10T04:46:55.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:54 smithi085 conmon[35961]: 1981 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:55.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:54 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:46:55.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:54 smithi085 conmon[35961]: 10T04:46:53.652675+0000 osd.4 (osd.4) 2021-02-10T04:46:55.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:54 smithi085 conmon[35961]: 1982 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:55.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:54 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:46:55.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:54 smithi085 conmon[35961]: 02-10T04:46:53.652681 2021-02-10T04:46:55.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:54 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 1983 : cluster [WRN] 2021-02-10T04:46:55.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:54 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:55.896 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:55 smithi164 conmon[33677]: debug 2021-02-10T04:46:55.896 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:55 smithi164 conmon[33677]: 2021-02-10T04:46:55.639+0000 7fbec19a1700 -1 osd.4 1061 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:46:55.896 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:55 smithi164 conmon[33677]: 2021-02-10T04:46:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:55 smithi164 conmon[29288]: cluster 2021-02-10T04:46:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:55 smithi164 conmon[29288]: 2021-02 2021-02-10T04:46:55.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:55 smithi164 conmon[29288]: -10T04:46:54.328885+0000 mgr.y (mgr.14140) 1035 : cluster [DBG] pgmap v1809: 188 pgs: 188 active+clean; 845 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:46:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:55 smithi164 conmon[29288]: cluster 2021-02-10T04:46 2021-02-10T04:46:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:55 smithi164 conmon[29288]: :54.631170+0000 osd.4 (osd.4) 1984 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:55 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:46:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:55 smithi164 conmon[29288]: -10T04:46:54.631178+0000 osd.4 (osd.4) 1985 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:55 smithi164 conmon[29288]: cluster 2021-02-10T04:46:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:55 smithi164 conmon[29288]: 2021-02-10T04:46:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:55 smithi164 conmon[29288]: 2021-02-10T04:46:54.631184+0000 osd.4 (osd.4) 1986 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:55 smithi164 conmon[29288]: cluster 2021-02-10T04:46:54.631190+0000 osd.4 (osd.4) 1987 : cluster 2021-02-10T04:46:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:55 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:55 smithi164 conmon[29288]: cluster 2021-02-10T04:46:54.814016+0000 osd.5 (osd.5) 120 : cluster [DBG] 2021-02-10T04:46:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:55 smithi164 conmon[29288]: 7.b scrub ok 2021-02-10T04:46:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:55 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:46:55.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:55 smithi164 conmon[29288]: 04:46:54.881404+0000 osd.0 (osd. 2021-02-10T04:46:55.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:55 smithi164 conmon[29288]: 0) 133 : cluster [DBG] 7.13 scrub ok 2021-02-10T04:46:55.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:55 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:46:55.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:55 smithi164 conmon[29288]: -02-10T04:46:55.046152+0000 2021-02-10T04:46:55.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:55 smithi164 conmon[29288]: osd.6 (osd.6) 121 : cluster [DBG] 2.11 scrub ok 2021-02-10T04:46:55.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:55 smithi164 conmon[29288]: 2021-02-10T04:46:55.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:55 smithi164 conmon[29288]: cluster 2021-02-10T04:46:55.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:55 smithi164 conmon[29288]: 2021-02-10T04:46:55.516753+0000 osd.3 (osd. 2021-02-10T04:46:55.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:55 smithi164 conmon[29288]: 3) 123 : cluster [DBG] 4.4 scrub ok 2021-02-10T04:46:56.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[29319]: cluster 2021-02-10T04:46:54.328885+0000 2021-02-10T04:46:56.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[29319]: mgr.y (mgr.14140) 1035 : cluster [DBG] pgmap v1809: 188 pgs: 188 active+clean; 845 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:46:56.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[29319]: cluster 2021-02-10T04:46:54. 2021-02-10T04:46:56.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[29319]: 631170+0000 osd.4 (osd.4) 1984 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:56.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:46:56.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[29319]: 02-10T04:46:54.631178+0000 2021-02-10T04:46:56.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[29319]: osd.4 (osd.4) 1985 : cluster 2021-02-10T04:46:56.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:56.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:46:56.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[29319]: 02-10T04:46:54.631184+0000 osd.4 ( 2021-02-10T04:46:56.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[29319]: osd.4) 1986 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:56.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:46:56.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[29319]: -10T04:46:54.631190+0000 osd.4 (osd 2021-02-10T04:46:56.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[29319]: .4) 1987 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:56.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[29319]: cluster 2021-02-10T04:46:56.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[29319]: 2021-02-10T04:46:54.814016 2021-02-10T04:46:56.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[29319]: +0000 osd.5 (osd.5) 2021-02-10T04:46:56.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[29319]: 120 : cluster [DBG] 7.b scrub ok 2021-02-10T04:46:56.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[29319]: cluster 2021-02-10T04:46:56.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[29319]: 2021-02-10T04:46:54.881404 2021-02-10T04:46:56.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[29319]: +0000 osd.0 (osd.0) 133 : cluster [DBG] 2021-02-10T04:46:56.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[29319]: 7.13 scrub ok 2021-02-10T04:46:56.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:46:56.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[29319]: 04:46:55.046152+0000 osd.6 (osd. 2021-02-10T04:46:56.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[29319]: 6) 121 : cluster [DBG] 2.11 scrub ok 2021-02-10T04:46:56.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:46:56.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[29319]: 02-10T04:46:55.516753+0000 2021-02-10T04:46:56.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[29319]: osd.3 (osd.3) 123 : cluster [DBG] 4.4 scrub ok 2021-02-10T04:46:56.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[29319]: 2021-02-10T04:46:56.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[35961]: cluster 2021-02-10T04:46:54. 2021-02-10T04:46:56.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[35961]: 328885+0000 mgr.y (mgr.14140) 1035 : cluster [DBG] pgmap v1809: 188 pgs: 188 active+clean; 845 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:46:56.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:46:56.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[35961]: 04:46:54.631170+0000 osd.4 (osd.4) 1984 2021-02-10T04:46:56.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:56.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[35961]: cluster 2021-02-10T04:46: 2021-02-10T04:46:56.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[35961]: 54.631178+0000 osd.4 (osd.4) 1985 : cluster [WRN] 2021-02-10T04:46:56.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:56.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[35961]: cluster 2021-02-10T04:46:56.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[35961]: 2021-02-10T04:46:54.631184+0000 osd.4 (osd.4) 1986 : cluster [WRN] 2021-02-10T04:46:56.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:56.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:46:56.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[35961]: -02-10T04:46:54.631190+0000 osd.4 (osd.4) 1987 : cluster [WRN] 2021-02-10T04:46:56.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:56.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:46:56.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[35961]: 02-10T04:46:54.814016+0000 osd.5 (osd.5) 120 : cluster [DBG] 2021-02-10T04:46:56.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[35961]: 7.b scrub ok 2021-02-10T04:46:56.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:46:56.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[35961]: 10T04:46:54.881404+0000 osd.0 (osd.0) 133 : cluster [DBG] 2021-02-10T04:46:56.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[35961]: 7.13 scrub ok 2021-02-10T04:46:56.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:46:56.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[35961]: -02-10T04:46:55.046152+0000 osd.6 2021-02-10T04:46:56.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[35961]: (osd.6) 121 : cluster [DBG] 2.11 scrub ok 2021-02-10T04:46:56.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[35961]: cluster 2021-02-10T04:46:56.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[35961]: 2021-02-10T04:46 2021-02-10T04:46:56.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:55 smithi085 conmon[35961]: :55.516753+0000 osd.3 (osd.3) 123 : cluster [DBG] 4.4 scrub ok 2021-02-10T04:46:56.900 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:56 smithi164 conmon[33677]: debug 2021-02-10T04:46:56.900 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:56 smithi164 conmon[33677]: 2021-02-10T04:46:56.647+0000 7fbec19a1700 -1 osd.4 1062 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:46:56.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:56 smithi164 conmon[29288]: cluster 2021-02-10T04:46:55.627210+0000 2021-02-10T04:46:56.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:56 smithi164 conmon[29288]: osd.1 (osd.1) 471 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:46:56.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:56 smithi164 conmon[29288]: cluster 2021-02-10T04:46:55.631095+0000 osd.1 (osd.1 2021-02-10T04:46:56.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:56 smithi164 conmon[29288]: ) 472 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:46:56.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:56 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:46:56.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:56 smithi164 conmon[29288]: 02-10T04:46:55.640580+0000 osd.4 (osd.4) 1988 : 2021-02-10T04:46:56.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:56 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:56.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:56 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:46:56.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:56 smithi164 conmon[29288]: 04:46:55.640590+0000 osd.4 (osd 2021-02-10T04:46:56.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:56 smithi164 conmon[29288]: .4) 1989 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:56.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:56 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:46:56.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:56 smithi164 conmon[29288]: 02-10T04:46:55.640597+0000 osd.4 2021-02-10T04:46:56.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:56 smithi164 conmon[29288]: (osd.4) 1990 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:56.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:56 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:46:56.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:56 smithi164 conmon[29288]: -02-10T04:46:55.640603+0000 osd.4 (osd. 2021-02-10T04:46:56.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:56 smithi164 conmon[29288]: 4) 1991 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:56.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:56 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:46:56.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:56 smithi164 conmon[29288]: 02-10T04:46:55.646251+0000 osd.4 ( 2021-02-10T04:46:56.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:56 smithi164 conmon[29288]: osd.4) 1992 : cluster [DBG] 4.1b deep-scrub ok 2021-02-10T04:46:56.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:56 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:46:56.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:56 smithi164 conmon[29288]: -02-10T04:46:55.895275+0000 2021-02-10T04:46:56.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:56 smithi164 conmon[29288]: mon.a (mon.0) 2122 : 2021-02-10T04:46:56.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:56 smithi164 conmon[29288]: cluster [DBG] osdmap e1062: 8 total, 8 up, 7 in 2021-02-10T04:46:56.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:56 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:46:56.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:56 smithi164 conmon[29288]: -02-10T04:46:56.512283+0000 2021-02-10T04:46:56.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:56 smithi164 conmon[29288]: osd.3 (osd.3) 124 : 2021-02-10T04:46:56.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:56 smithi164 conmon[29288]: cluster [DBG] 4.14 scrub ok 2021-02-10T04:46:57.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[29319]: cluster 2021-02-10T04:46:57.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[29319]: 2021-02-10T04:46:55.627210 2021-02-10T04:46:57.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[29319]: +0000 osd.1 (osd.1) 471 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:46:57.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[29319]: cluster 2021-02-10T04:46:57.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[29319]: 2021-02-10T04:46:55.631095+0000 osd.1 (osd.1) 472 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:46:57.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:46:57.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[29319]: 46:55.640580+0000 osd.4 (osd.4) 1988 : cluster 2021-02-10T04:46:57.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:57.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[29319]: cluster 2021-02-10T04:46:57.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[29319]: 2021-02-10T04:46:55.640590+0000 2021-02-10T04:46:57.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[29319]: osd.4 (osd.4) 1989 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:57.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[29319]: cluster 2021-02-10T04:46:57.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[29319]: 2021-02-10T04:46:55.640597 2021-02-10T04:46:57.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 1990 : cluster 2021-02-10T04:46:57.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:57.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:46:57.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[29319]: 04:46:55.640603+0000 osd.4 ( 2021-02-10T04:46:57.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[29319]: osd.4) 1991 : cluster [WRN] 2021-02-10T04:46:57.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:57.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[29319]: cluster 2021-02-10T04:46: 2021-02-10T04:46:57.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[29319]: 55.646251+0000 osd.4 (osd.4) 1992 : cluster [DBG] 2021-02-10T04:46:57.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[29319]: 4.1b deep-scrub ok 2021-02-10T04:46:57.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[29319]: cluster 2021-02-10T04:46:55. 2021-02-10T04:46:57.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[29319]: 895275+0000 mon.a (mon.0) 2122 : cluster [DBG] 2021-02-10T04:46:57.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[29319]: osdmap e1062: 8 total, 8 up, 7 in 2021-02-10T04:46:57.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:46:57.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[29319]: :46:56.512283+0000 osd.3 (osd.3) 2021-02-10T04:46:57.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[29319]: 124 : cluster [DBG] 4.14 scrub ok 2021-02-10T04:46:57.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: cluster 2021-02-10T04:46:57.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: 2021-02-10T04:46:57.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: 2021 2021-02-10T04:46:57.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: - 2021-02-10T04:46:57.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: 02 2021-02-10T04:46:57.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: - 2021-02-10T04:46:57.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: 10T04:46:55 2021-02-10T04:46:57.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: .627210+0000 osd.1 (osd.1) 471 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:46:57.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:46:57.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: 02-10T04:46:55.631095+0000 osd.1 (osd. 2021-02-10T04:46:57.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: 1) 472 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:46:57.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:46:57.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: -10T04:46:55.640580+0000 osd.4 2021-02-10T04:46:57.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: (osd.4) 1988 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:57.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:46:57.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: -02-10T04:46:55.640590+0000 2021-02-10T04:46:57.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: osd.4 (osd.4) 1989 : cluster [WRN] 2021-02-10T04:46:57.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:57.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: cluster 2021-02-10T04:46:57.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: 2021-02-10T04:46:55.640597+0000 osd.4 ( 2021-02-10T04:46:57.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: osd.4) 1990 : cluster [WRN] 2021-02-10T04:46:57.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:57.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:46:57.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: 46:55.640603+0000 osd.4 (osd.4) 2021-02-10T04:46:57.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: 1991 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:57.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:46:57.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: T04:46:55.646251+0000 osd.4 (osd 2021-02-10T04:46:57.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: .4) 1992 : cluster [DBG] 4.1b deep-scrub ok 2021-02-10T04:46:57.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:46:57.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: -02-10T04:46:55.895275+0000 2021-02-10T04:46:57.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: mon.a (mon.0) 2122 : cluster [DBG] 2021-02-10T04:46:57.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: osdmap e1062: 8 total, 8 up, 7 in 2021-02-10T04:46:57.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: cluster 2021-02-10T04:46: 2021-02-10T04:46:57.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: 56.512283+0000 osd.3 (osd.3) 124 2021-02-10T04:46:57.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:56 smithi085 conmon[35961]: : cluster [DBG] 4.14 scrub ok 2021-02-10T04:46:57.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:57 smithi164 conmon[33677]: debug 2021-02-10T04:46:57.623+0000 7fbec19a1700 -1 osd.4 1062 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:46:58.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:46:58.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[29319]: 04:46:56.329287+0000 mgr.y (mgr.14140) 1036 : cluster [DBG] pgmap v1811: 188 pgs: 188 active+clean; 849 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:46:58.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[29319]: cluster 2021-02-10T04:46:56.648180+0000 osd.4 2021-02-10T04:46:58.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[29319]: (osd.4) 1993 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:58.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[29319]: cluster 2021-02-10T04:46:58.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[29319]: 2021-02-10T04:46:56.648189+0000 osd.4 (osd 2021-02-10T04:46:58.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[29319]: .4) 1994 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:58.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[29319]: cluster 2021-02-10T04:46:58.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[29319]: 2021-02-10T04:46:56.648195+0000 2021-02-10T04:46:58.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[29319]: osd.4 (osd.4) 1995 : cluster [WRN] 2021-02-10T04:46:58.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:58.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[29319]: cluster 2021-02-10T04:46:56.648201+0000 osd.4 2021-02-10T04:46:58.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[29319]: (osd.4) 1996 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:58.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:46:58.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[29319]: -02-10T04:46:57.054224+0000 osd.6 2021-02-10T04:46:58.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[29319]: (osd.6) 122 : cluster [DBG] 7.7 scrub ok 2021-02-10T04:46:58.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[35961]: cluster 2021-02-10T04:46:56 2021-02-10T04:46:58.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[35961]: .329287+0000 mgr.y (mgr.14140) 1036 : cluster [DBG] pgmap v1811: 188 pgs: 188 active+clean; 849 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:46:58.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:46:58.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[35961]: 46:56.648180+0000 osd.4 (osd.4) 1993 2021-02-10T04:46:58.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:58.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:46:58.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[35961]: 46:56.648189+0000 osd.4 (osd.4) 1994 : 2021-02-10T04:46:58.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:58.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[35961]: cluster 2021-02-10T04:46:58.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[35961]: 2021-02-10T04:46 2021-02-10T04:46:58.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[35961]: :56.648195+0000 osd.4 (osd 2021-02-10T04:46:58.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[35961]: .4) 1995 : cluster [WRN] 2021-02-10T04:46:58.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:58.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[35961]: cluster 2021-02-10T04:46:58.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[35961]: 2021-02-10T04:46 2021-02-10T04:46:58.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[35961]: :56.648201+0000 osd.4 ( 2021-02-10T04:46:58.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[35961]: osd.4) 1996 : cluster 2021-02-10T04:46:58.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:58.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:46:58.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:57 smithi085 conmon[35961]: T04:46:57.054224+0000 osd.6 (osd.6) 122 : cluster [DBG] 7.7 scrub ok 2021-02-10T04:46:58.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:57 smithi164 conmon[29288]: cluster 2021-02-10T04:46:58.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:57 smithi164 conmon[29288]: 2021-02-10T04:46 2021-02-10T04:46:58.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:57 smithi164 conmon[29288]: :56.329287+0000 mgr.y (mgr.14140) 1036 : cluster [DBG] pgmap v1811: 188 pgs: 188 active+clean; 849 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:46:58.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:57 smithi164 conmon[29288]: cluster 2021-02-10T04:46:58.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:57 smithi164 conmon[29288]: 2021- 2021-02-10T04:46:58.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:57 smithi164 conmon[29288]: 02-10T04:46:56.648180+0000 osd.4 (osd.4) 1993 : 2021-02-10T04:46:58.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:57 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:58.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:57 smithi164 conmon[29288]: cluster 2021-02-10T04:46:58.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:57 smithi164 conmon[29288]: 2021-02-10T04:46:56.648189+0000 2021-02-10T04:46:58.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:57 smithi164 conmon[29288]: osd.4 (osd.4) 1994 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:58.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:57 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:46:58.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:57 smithi164 conmon[29288]: 02-10T04:46:56.648195+0000 osd.4 (osd. 2021-02-10T04:46:58.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:57 smithi164 conmon[29288]: 4) 1995 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:58.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:57 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:46:58.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:57 smithi164 conmon[29288]: T04:46:56.648201+0000 osd.4 (osd 2021-02-10T04:46:58.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:57 smithi164 conmon[29288]: .4) 1996 : cluster 2021-02-10T04:46:58.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:57 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:58.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:57 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:46:58.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:57 smithi164 conmon[29288]: -10T04:46:57. 2021-02-10T04:46:58.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:57 smithi164 conmon[29288]: 054224+0000 osd.6 (osd.6) 2021-02-10T04:46:58.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:57 smithi164 conmon[29288]: 122 : cluster [DBG] 7.7 scrub ok 2021-02-10T04:46:59.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:58 smithi164 conmon[33677]: debug 2021-02-10T04:46:58.663+0000 7fbec19a1700 -1 osd.4 1063 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:46:59.144 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:46:58 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:46:59.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:58 smithi164 conmon[29288]: cluster 2021-02-10T04:46:57.623746+0000 osd.4 (osd.4) 1997 : cluster 2021-02-10T04:46:59.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:58 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:59.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:58 smithi164 conmon[29288]: cluster 2021-02-10T04:46 2021-02-10T04:46:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:58 smithi164 conmon[29288]: :57.623754+0000 osd.4 (osd.4) 1998 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:58 smithi164 conmon[29288]: cluster 2021-02-10T04:46:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:58 smithi164 conmon[29288]: 2021-02-10T04:46:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:58 smithi164 conmon[29288]: 2021-02-10T04:46 2021-02-10T04:46:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:58 smithi164 conmon[29288]: :57.623762+0000 osd.4 (osd.4) 1999 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:58 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:46:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:58 smithi164 conmon[29288]: :46:57.623768+0000 osd.4 (osd.4) 2000 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:58 smithi164 conmon[29288]: cluster 2021-02-10T04:46:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:58 smithi164 conmon[29288]: 2021-02-10T04:46:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:58 smithi164 conmon[29288]: 2021 2021-02-10T04:46:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:58 smithi164 conmon[29288]: - 2021-02-10T04:46:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:58 smithi164 conmon[29288]: 02 2021-02-10T04:46:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:58 smithi164 conmon[29288]: -10T04:46:58.403021 2021-02-10T04:46:59.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:58 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2123 : cluster [DBG] osdmap e1063: 8 total, 8 up, 7 in 2021-02-10T04:46:59.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:58 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:46:59.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:58 smithi164 conmon[29288]: 04:46:58.514305+0000 2021-02-10T04:46:59.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:58 smithi164 conmon[29288]: osd.3 (osd.3) 125 : cluster [DBG] 2021-02-10T04:46:59.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:58 smithi164 conmon[29288]: 6.13 scrub ok 2021-02-10T04:46:59.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: cluster 2021-02-10T04:46:59.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: 2021-02-10T04:46:59.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: 2021 2021-02-10T04:46:59.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: -02-10T04:46: 2021-02-10T04:46:59.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: 57.623746+0000 osd.4 (osd.4) 1997 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:59.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: cluster 2021-02-10T04:46:59.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: 2021-02-10T04:46:59.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: 2021 2021-02-10T04:46:59.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: - 2021-02-10T04:46:59.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: 02 2021-02-10T04:46:59.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: - 2021-02-10T04:46:59.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: 10 2021-02-10T04:46:59.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: T 2021-02-10T04:46:59.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: 04 2021-02-10T04:46:59.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: : 2021-02-10T04:46:59.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: 46 2021-02-10T04:46:59.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: :57.623754+0000 2021-02-10T04:46:59.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: osd.4 (osd.4) 1998 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:59.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: cluster 2021-02-10T04:46:59.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: 2021-02-10T04:46:57.623762+0000 2021-02-10T04:46:59.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: osd.4 (osd.4) 1999 : cluster [WRN] 2021-02-10T04:46:59.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:59.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: cluster 2021-02-10T04:46: 2021-02-10T04:46:59.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: 57.623768+0000 osd.4 (osd.4) 2000 2021-02-10T04:46:59.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:59.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:46:59.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: :46:58.403021+0000 mon.a (mon. 2021-02-10T04:46:59.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: 0) 2123 : cluster [DBG] osdmap e1063: 8 total, 8 up, 7 in 2021-02-10T04:46:59.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:46:59.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: 02-10T04:46:58.514305+0000 osd.3 2021-02-10T04:46:59.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: (osd.3) 2021-02-10T04:46:59.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[29319]: 125 : cluster [DBG] 6.13 scrub ok 2021-02-10T04:46:59.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[35961]: cluster 2021-02-10T04:46:59.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[35961]: 2021-02-10T04:46:59.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[35961]: 2021 2021-02-10T04:46:59.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[35961]: - 2021-02-10T04:46:59.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[35961]: 02 2021-02-10T04:46:59.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[35961]: - 2021-02-10T04:46:59.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[35961]: 10 2021-02-10T04:46:59.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[35961]: T04:46:57.623746+0000 osd.4 (osd.4) 1997 : cluster [WRN] 2021-02-10T04:46:59.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:46:59.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[35961]: cluster 2021-02-10T04:46:59.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[35961]: 2021-02-10T04:46:57.623754+0000 osd.4 (osd.4) 1998 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:46:59.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[35961]: cluster 2021-02-10T04:46: 2021-02-10T04:46:59.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[35961]: 57.623762+0000 osd.4 (osd.4) 1999 : cluster 2021-02-10T04:46:59.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:46:59.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:46:59.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[35961]: 46:57.623768+0000 osd.4 (osd.4) 2000 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:46:59.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[35961]: cluster 2021-02-10T04:46:58.403021+0000 2021-02-10T04:46:59.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[35961]: mon.a (mon.0) 2123 : cluster [DBG] osdmap e1063: 8 total, 8 up, 7 in 2021-02-10T04:46:59.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[35961]: cluster 2021-02-10T04:46: 2021-02-10T04:46:59.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:58 smithi085 conmon[35961]: 58.514305+0000 osd.3 (osd.3) 125 : cluster [DBG] 6.13 scrub ok 2021-02-10T04:47:00.143 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[33677]: debug 2021-02-10T04:46:59.685+0000 7fbec19a1700 -1 osd.4 1063 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[29288]: cluster 2021-02-10T04:46:58.329914+0000 mgr.y 2021-02-10T04:47:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[29288]: (mgr.14140) 1037 : cluster [DBG] pgmap v1812: 188 pgs: 188 active+clean; 849 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[29288]: cluster 2021-02-10T04:47:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[29288]: 2021 2021-02-10T04:47:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[29288]: - 2021-02-10T04:47:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[29288]: 02 2021-02-10T04:47:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[29288]: - 2021-02-10T04:47:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[29288]: 10 2021-02-10T04:47:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[29288]: T04:46:58.664349 2021-02-10T04:47:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2001 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[29288]: cluster 2021-02-10T04:46:58.664357+0000 osd.4 (osd.4) 2002 2021-02-10T04:47:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:47:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[29288]: 46:58.664363+0000 osd.4 (osd.4) 2003 : cluster 2021-02-10T04:47:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:47:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[29288]: 46:58.664366+0000 osd.4 (osd.4) 2004 : 2021-02-10T04:47:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:47:00.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[29288]: -10T04:46:58.666258+0000 osd.4 (osd. 2021-02-10T04:47:00.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[29288]: 4) 2005 : cluster [DBG] 5.1c deep-scrub ok 2021-02-10T04:47:00.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[29288]: audit 2021-02-10T04:47:00.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[29288]: 2021 2021-02-10T04:47:00.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[29288]: - 2021-02-10T04:47:00.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[29288]: 02-10T04:46: 2021-02-10T04:47:00.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[29288]: 58.793040+0000 mgr.y (mgr.14140) 1038 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:47:00.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:47:00.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[29288]: -02-10T04:46:58. 2021-02-10T04:47:00.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[29288]: 897774+0000 mon.a (mon.0 2021-02-10T04:47:00.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:46:59 smithi164 conmon[29288]: ) 2124 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 152 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:47:00.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: cluster 2021-02-10T04:47:00.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: 2021-02-10T04:47:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: 2021 2021-02-10T04:47:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: - 2021-02-10T04:47:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: 02-10T04:46:58. 2021-02-10T04:47:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: 329914+0000 mgr.y (mgr.14140) 1037 : cluster [DBG] pgmap v1812: 188 pgs: 188 active+clean; 849 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:47:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: 02-10T04:46:58.664349+0000 osd.4 ( 2021-02-10T04:47:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: osd.4) 2001 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:47:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: -02-10T04:46:58.664357+0000 osd.4 2021-02-10T04:47:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: (osd.4) 2002 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: 2021-02-10T04:47:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: cluster 2021-02-10T04:47:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:47:00.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: 04:46:58.664363+0000 osd.4 (osd.4) 2003 2021-02-10T04:47:00.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:00.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:47:00.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: -02-10T04:46:58.664366+0000 2021-02-10T04:47:00.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: osd.4 (osd.4) 2004 : cluster [WRN] 2021-02-10T04:47:00.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:00.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: cluster 2021-02-10T04:47:00.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: 2021-02-10T04:47:00.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: 2021 2021-02-10T04:47:00.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: -02-10T04 2021-02-10T04:47:00.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: :46:58.666258+0000 osd.4 (osd.4) 2005 : cluster [DBG] 2021-02-10T04:47:00.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: 5.1c deep-scrub ok 2021-02-10T04:47:00.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: audit 2021-02-10T04:47:00.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: 2021-02-10T04:47:00.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: 2021 2021-02-10T04:47:00.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: -02-10T04:46:58.793040+0000 2021-02-10T04:47:00.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: mgr.y (mgr.14140) 1038 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:47:00.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:47:00.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: 46:58.897774+0000 mon.a (mon.0) 2124 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 152 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:47:00.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[35961]: 2021-02-10T04:47:00.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: cluster 2021-02-10T04:47:00.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: 2021-02-10T04:47:00.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: 2021 2021-02-10T04:47:00.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: - 2021-02-10T04:47:00.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: 02 2021-02-10T04:47:00.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: - 2021-02-10T04:47:00.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: 10 2021-02-10T04:47:00.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: T 2021-02-10T04:47:00.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: 04 2021-02-10T04:47:00.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: : 2021-02-10T04:47:00.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: 46 2021-02-10T04:47:00.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: : 2021-02-10T04:47:00.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: 58 2021-02-10T04:47:00.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: . 2021-02-10T04:47:00.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: 329914 2021-02-10T04:47:00.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: +0000 2021-02-10T04:47:00.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: 2021-02-10T04:47:00.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: mgr.y 2021-02-10T04:47:00.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: ( 2021-02-10T04:47:00.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: mgr 2021-02-10T04:47:00.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: . 2021-02-10T04:47:00.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: 14140 2021-02-10T04:47:00.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: ) 1037 : cluster [DBG] 2021-02-10T04:47:00.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: pgmap v1812: 188 pgs: 188 active+clean; 849 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:00.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: cluster 2021-02-10T04:46:58.664349+0000 osd.4 (osd. 2021-02-10T04:47:00.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: 4) 2001 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:00.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:47:00.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: -10T04:46:58.664357+0000 osd.4 2021-02-10T04:47:00.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: (osd.4) 2002 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:00.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: 2021-02-10T04:47:00.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: cluster 2021-02-10T04:47:00.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: 2021-02-10T04:46:58.664363+0000 osd.4 ( 2021-02-10T04:47:00.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: osd.4) 2003 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:00.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:47:00.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: 02-10T04:46:58.664366+0000 2021-02-10T04:47:00.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: osd.4 (osd.4) 2004 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:00.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: 2021-02-10T04:47:00.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:47:00.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: 46:58.666258+0000 osd.4 (osd.4 2021-02-10T04:47:00.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: ) 2005 : cluster [DBG] 5.1c deep-scrub ok 2021-02-10T04:47:00.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: audit 2021 2021-02-10T04:47:00.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: -02-10T04:46:58.793040+0000 2021-02-10T04:47:00.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: mgr.y (mgr.14140) 1038 : audit [DBG] 2021-02-10T04:47:00.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:47:00.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:47:00.198 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: 46:58.897774+0000 mon.a (mon.0 2021-02-10T04:47:00.198 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:46:59 smithi085 conmon[29319]: ) 2124 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 152 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:47:01.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:00 smithi164 conmon[33677]: debug 2021-02-10T04:47:01.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:00 smithi164 conmon[33677]: 2021-02-10T04:47:00.698+0000 7fbec19a1700 -1 osd.4 1063 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:00 smithi164 conmon[29288]: cluster 2021-02-10T04:46:59.686560+0000 2021-02-10T04:47:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:00 smithi164 conmon[29288]: osd.4 (osd.4) 2006 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:00 smithi164 conmon[29288]: cluster 2021-02-10T04:46:59.686568+0000 osd.4 (osd.4) 2007 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:00 smithi164 conmon[29288]: cluster 2021-02-10T04:47:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:00 smithi164 conmon[29288]: 2021-02-10T04:46:59.686572 2021-02-10T04:47:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:00 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2008 : cluster 2021-02-10T04:47:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:00 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:00 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:47:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:00 smithi164 conmon[29288]: -10T04:46:59.686575+0000 2021-02-10T04:47:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:00 smithi164 conmon[29288]: osd.4 (osd.4) 2009 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:00 smithi164 conmon[29288]: 2021-02-10T04:47:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:00 smithi164 conmon[29288]: cluster 2021-02-10T04:46:59.760258+0000 osd.5 ( 2021-02-10T04:47:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:00 smithi164 conmon[29288]: osd.5) 121 : cluster [DBG] 2.1c scrub ok 2021-02-10T04:47:01.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[29319]: cluster 2021-02-10T04:46:59.686560+0000 2021-02-10T04:47:01.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[29319]: osd.4 (osd.4) 2006 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:01.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[29319]: cluster 2021-02-10T04:46:59.686568+0000 osd.4 2021-02-10T04:47:01.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[29319]: (osd.4) 2007 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:01.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:47:01.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[29319]: 04:46:59.686572+0000 osd.4 (osd.4) 2008 2021-02-10T04:47:01.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:01.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[29319]: cluster 2021-02-10T04:46:59.686575+0000 osd.4 (osd. 2021-02-10T04:47:01.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[29319]: 4) 2009 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:01.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:47:01.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[29319]: -10T04:46:59.760258+0000 osd.5 ( 2021-02-10T04:47:01.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[29319]: osd.5) 121 : cluster [DBG] 2021-02-10T04:47:01.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[29319]: 2.1c scrub ok 2021-02-10T04:47:01.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[35961]: cluster 2021-02-10T04:47:01.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[35961]: 2021-02-10T04:47:01.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[35961]: 2021 2021-02-10T04:47:01.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[35961]: - 2021-02-10T04:47:01.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[35961]: 02 2021-02-10T04:47:01.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[35961]: - 2021-02-10T04:47:01.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[35961]: 10T04:46:59.686560+0000 2021-02-10T04:47:01.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[35961]: osd.4 (osd.4) 2006 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:01.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:47:01.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[35961]: -10T04:46:59.686568+0000 osd.4 ( 2021-02-10T04:47:01.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[35961]: osd.4) 2007 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:01.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:47:01.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[35961]: 10T04:46:59.686572+0000 osd.4 (osd. 2021-02-10T04:47:01.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[35961]: 4) 2008 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:01.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[35961]: cluster 2021-02-10T04:47:01.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[35961]: 2021-02-10T04:46:59.686575+0000 osd.4 (osd.4) 2009 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:01.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[35961]: cluster 2021-02-10T04:47:01.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:00 smithi085 conmon[35961]: 2021-02-10T04:46:59.760258+0000 osd.5 (osd.5) 121 : cluster [DBG] 2.1c scrub ok 2021-02-10T04:47:02.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:01 smithi164 conmon[33677]: debug 2021-02-10T04:47:01.744+0000 7fbec19a1700 -1 osd.4 1063 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:01 smithi164 conmon[29288]: cluster 2021-02-10T04:47:00.330582+0000 mgr.y 2021-02-10T04:47:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:01 smithi164 conmon[29288]: (mgr.14140) 1039 : cluster [DBG] pgmap v1814: 188 pgs: 188 active+clean; 887 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:01 smithi164 conmon[29288]: cluster 2021-02-10T04:47:00.699064+0000 osd.4 (osd.4) 2010 : 2021-02-10T04:47:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:01 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:01 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:47:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:01 smithi164 conmon[29288]: 04:47:00.699073+0000 osd.4 (osd.4) 2011 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:01 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:47:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:01 smithi164 conmon[29288]: 02-10T04:47:00.699079+0000 osd.4 2021-02-10T04:47:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:01 smithi164 conmon[29288]: (osd.4) 2012 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:01 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:47:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:01 smithi164 conmon[29288]: 47:00.699086+0000 osd.4 (osd.4) 2013 : cluster 2021-02-10T04:47:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:01 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:01 smithi164 conmon[29288]: cluster 2021-02-10T04:47: 2021-02-10T04:47:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:01 smithi164 conmon[29288]: 00.950110+0000 osd.6 (osd.6) 123 : cluster [DBG] 2021-02-10T04:47:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:01 smithi164 conmon[29288]: 7.11 scrub ok 2021-02-10T04:47:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:01 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:47:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:01 smithi164 conmon[29288]: 02-10T04:47:01 2021-02-10T04:47:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:01 smithi164 conmon[29288]: .565992+0000 osd.3 (osd. 2021-02-10T04:47:02.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:01 smithi164 conmon[29288]: 3) 126 : cluster [DBG] 6.c scrub ok 2021-02-10T04:47:02.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:01 smithi164 conmon[29288]: 2021-02-10T04:47:02.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: cluster 2021-02-10T04:47:02.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: 2021-02-10T04:47:02.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: 2021 2021-02-10T04:47:02.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: - 2021-02-10T04:47:02.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: 02 2021-02-10T04:47:02.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: - 2021-02-10T04:47:02.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: 10 2021-02-10T04:47:02.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: T 2021-02-10T04:47:02.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: 04 2021-02-10T04:47:02.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: : 2021-02-10T04:47:02.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: 47 2021-02-10T04:47:02.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: : 2021-02-10T04:47:02.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: 00 2021-02-10T04:47:02.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: . 2021-02-10T04:47:02.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: 330582 2021-02-10T04:47:02.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: +0000 2021-02-10T04:47:02.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: 2021-02-10T04:47:02.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: mgr.y 2021-02-10T04:47:02.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: ( 2021-02-10T04:47:02.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: mgr 2021-02-10T04:47:02.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: . 2021-02-10T04:47:02.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: 14140 2021-02-10T04:47:02.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: ) 2021-02-10T04:47:02.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: 1039 2021-02-10T04:47:02.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: : 2021-02-10T04:47:02.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: cluster [DBG] pgmap v1814: 188 pgs: 188 active+clean; 887 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:02.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: cluster 2021-02-10T04:47:02.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: 2021-02-10T04:47:00.699064+0000 2021-02-10T04:47:02.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: osd.4 (osd.4) 2010 : cluster [WRN] 2021-02-10T04:47:02.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:02.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:47:02.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: 04:47:00.699073+0000 osd.4 (osd 2021-02-10T04:47:02.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: .4) 2011 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:02.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:47:02.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: 10T04:47:00.699079+0000 osd.4 (osd 2021-02-10T04:47:02.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: .4) 2012 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:02.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:47:02.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: T04:47:00.699086+0000 osd.4 (osd. 2021-02-10T04:47:02.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: 4) 2013 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:02.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: cluster 2021-02-10T04:47:02.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: 2021-02-10T04:47:02.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: 2021 2021-02-10T04:47:02.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: - 2021-02-10T04:47:02.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: 02 2021-02-10T04:47:02.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: - 2021-02-10T04:47:02.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: 10 2021-02-10T04:47:02.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: T 2021-02-10T04:47:02.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: 04 2021-02-10T04:47:02.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: : 2021-02-10T04:47:02.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: 47 2021-02-10T04:47:02.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: : 2021-02-10T04:47:02.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: 00 2021-02-10T04:47:02.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: . 2021-02-10T04:47:02.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: 950110 2021-02-10T04:47:02.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: +0000 2021-02-10T04:47:02.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: 2021-02-10T04:47:02.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: osd.6 2021-02-10T04:47:02.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: ( 2021-02-10T04:47:02.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: osd 2021-02-10T04:47:02.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: . 2021-02-10T04:47:02.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: 6 2021-02-10T04:47:02.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: ) 2021-02-10T04:47:02.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: 123 2021-02-10T04:47:02.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: : 2021-02-10T04:47:02.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: cluster [DBG] 7.11 scrub ok 2021-02-10T04:47:02.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:47:02.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: 02-10T04:47:01.565992 2021-02-10T04:47:02.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: +0000 osd.3 (osd.3) 126 2021-02-10T04:47:02.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[29319]: : cluster [DBG] 6.c scrub ok 2021-02-10T04:47:02.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: cluster 2021-02-10T04:47:02.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: 2021-02-10T04:47:02.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: 2021 2021-02-10T04:47:02.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: - 2021-02-10T04:47:02.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: 02 2021-02-10T04:47:02.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: -10T04:47:00.330582 2021-02-10T04:47:02.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1039 : cluster [DBG] pgmap v1814: 188 pgs: 188 active+clean; 887 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:02.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:47:02.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: -02-10T04:47:00.699064+0000 osd.4 (osd 2021-02-10T04:47:02.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: .4) 2010 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:02.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: cluster 2021-02-10T04:47:02.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: 2021-02-10T04:47:02.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:47:02.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: 47:00.699073+0000 osd.4 (osd.4) 2011 : cluster 2021-02-10T04:47:02.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:02.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: cluster 2021-02-10T04:47:02.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: 2021-02-10T04:47:02.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: 2021 2021-02-10T04:47:02.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: - 2021-02-10T04:47:02.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: 02 2021-02-10T04:47:02.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: - 2021-02-10T04:47:02.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: 10 2021-02-10T04:47:02.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: T 2021-02-10T04:47:02.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: 04 2021-02-10T04:47:02.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: : 2021-02-10T04:47:02.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: 47 2021-02-10T04:47:02.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: : 2021-02-10T04:47:02.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: 00 2021-02-10T04:47:02.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: .699079+0000 osd.4 (osd. 2021-02-10T04:47:02.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: 4) 2012 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:02.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: cluster 2021-02-10T04:47:02.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: 2021 2021-02-10T04:47:02.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: -02-10T04:47:00.699086+0000 osd.4 (osd.4) 2013 : 2021-02-10T04:47:02.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:02.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:47:02.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: 02-10T04:47:00.950110 2021-02-10T04:47:02.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: +0000 osd.6 (osd 2021-02-10T04:47:02.201 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: .6) 2021-02-10T04:47:02.201 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: 123 : cluster [DBG] 7.11 scrub ok 2021-02-10T04:47:02.201 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: cluster 2021-02-10T04:47:02.201 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: 2021-02-10T04:47 2021-02-10T04:47:02.201 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:01 smithi085 conmon[35961]: :01.565992+0000 osd.3 (osd.3) 126 : cluster [DBG] 6.c scrub ok 2021-02-10T04:47:03.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:02 smithi164 conmon[33677]: debug 2021-02-10T04:47:03.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:02 smithi164 conmon[33677]: 2021-02-10T04:47:02.737+0000 7fbec19a1700 -1 osd.4 1063 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:03.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:02 smithi164 conmon[33677]: 2021-02-10T04:47:03.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:02 smithi164 conmon[29288]: cephadm 2021-02-10T04:47:01.518186+0000 mgr.y 2021-02-10T04:47:03.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:02 smithi164 conmon[29288]: (mgr.14140) 1040 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:47:03.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:02 smithi164 conmon[29288]: cephadm 2021-02-10T04:47:01.518643+0000 mgr.y (mgr.14140) 1041 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:47:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:02 smithi164 conmon[29288]: cluster 2021-02-10T04:47:01.744808+0000 osd.4 (osd.4 2021-02-10T04:47:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:02 smithi164 conmon[29288]: ) 2014 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:02 smithi164 conmon[29288]: cluster 2021-02-10T04:47:01. 2021-02-10T04:47:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:02 smithi164 conmon[29288]: 744817+0000 osd.4 (osd.4) 2015 : cluster [WRN] 2021-02-10T04:47:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:02 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:02 smithi164 conmon[29288]: cluster 2021-02-10T04:47:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:02 smithi164 conmon[29288]: 2021-02-10T04:47:01.744825+0000 osd.4 (osd.4) 2016 2021-02-10T04:47:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:02 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:02 smithi164 conmon[29288]: cluster 2021-02-10T04:47: 2021-02-10T04:47:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:02 smithi164 conmon[29288]: 01.744831+0000 osd.4 (osd.4) 2017 : cluster [WRN] 2021-02-10T04:47:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:02 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:02 smithi164 conmon[29288]: cluster 2021-02-10T04:47:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:02 smithi164 conmon[29288]: 2021-02-10T04:47:01.756899+0000 osd.5 (osd. 2021-02-10T04:47:03.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:02 smithi164 conmon[29288]: 5) 122 : cluster [DBG] 5.d scrub ok 2021-02-10T04:47:03.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:02 smithi164 conmon[29288]: audit 2021-02-10T04: 2021-02-10T04:47:03.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:02 smithi164 conmon[29288]: 47:02.236233+0000 mon.a (mon.0) 2125 : audit 2021-02-10T04:47:03.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:02 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:47:03.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[29319]: cephadm 2021-02- 2021-02-10T04:47:03.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[29319]: 10T04:47:01.518186+0000 mgr.y (mgr.14140) 1040 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:47:03.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[29319]: cephadm 2021 2021-02-10T04:47:03.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[29319]: -02-10T04:47:01.518643+0000 2021-02-10T04:47:03.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[29319]: mgr.y (mgr.14140) 1041 : cephadm [INF] 2021-02-10T04:47:03.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[29319]: Refreshing smithi164 facts 2021-02-10T04:47:03.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[29319]: cluster 2021-02-10T04:47:03.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[29319]: 2021-02-10T04:47: 2021-02-10T04:47:03.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[29319]: 01.744808+0000 osd.4 (osd.4) 2021-02-10T04:47:03.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[29319]: 2014 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:03.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:47:03.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[29319]: 02-10T04:47:01.744817+0000 osd.4 (osd. 2021-02-10T04:47:03.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[29319]: 4) 2015 : cluster [WRN] 2021-02-10T04:47:03.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:03.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:47:03.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[29319]: 02-10T04:47:01.744825+0000 2021-02-10T04:47:03.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[29319]: osd.4 (osd.4) 2016 : cluster [WRN] 2021-02-10T04:47:03.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:03.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:47:03.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[29319]: -02-10T04: 2021-02-10T04:47:03.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[29319]: 47:01.744831+0000 osd.4 ( 2021-02-10T04:47:03.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[29319]: osd.4) 2017 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:03.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:47:03.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[29319]: -10T04: 2021-02-10T04:47:03.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[29319]: 47:01.756899+0000 osd.5 (osd.5 2021-02-10T04:47:03.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[29319]: ) 122 : cluster [DBG] 5.d scrub ok 2021-02-10T04:47:03.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[29319]: audit 2021 2021-02-10T04:47:03.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[29319]: -02-10T04:47:02.236233+0000 2021-02-10T04:47:03.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[29319]: mon.a (mon.0) 2125 : audit [INF] 2021-02-10T04:47:03.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:47:03.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[35961]: cephadm 2021-02-10T04:47:01.518186+0000 2021-02-10T04:47:03.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[35961]: mgr.y (mgr.14140) 1040 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:47:03.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[35961]: cephadm 2021-02-10T04:47:01 2021-02-10T04:47:03.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[35961]: .518643+0000 mgr.y (mgr.14140) 2021-02-10T04:47:03.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[35961]: 1041 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:47:03.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:47:03.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[35961]: -02-10T04:47:01.744808+0000 osd.4 (osd 2021-02-10T04:47:03.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[35961]: .4) 2014 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:03.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[35961]: cluster 2021-02-10T04:47:03.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:47:03.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[35961]: T04:47:01.744817+0000 osd.4 (osd.4) 2021-02-10T04:47:03.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[35961]: 2015 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:03.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[35961]: cluster 2021-02-10T04:47:03.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[35961]: 2021-02-10T04:47:01.744825 2021-02-10T04:47:03.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2016 : cluster [WRN] 2021-02-10T04:47:03.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:03.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:47:03.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[35961]: -02-10T04:47:01.744831+0000 2021-02-10T04:47:03.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[35961]: osd.4 (osd.4) 2017 : cluster [WRN] 2021-02-10T04:47:03.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:03.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:47:03.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[35961]: 02-10T04:47:01.756899+0000 osd.5 2021-02-10T04:47:03.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[35961]: (osd.5) 122 : cluster [DBG] 5.d scrub ok 2021-02-10T04:47:03.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[35961]: 2021-02-10T04:47:03.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[35961]: audit 2021-02-10T04:47: 2021-02-10T04:47:03.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[35961]: 02.236233+0000 mon.a (mon.0) 2125 : 2021-02-10T04:47:03.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:02 smithi085 conmon[35961]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:47:03.786 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":1802,"stamp":"2021-02-10T04:46:46.325949+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":885339803,"num_objects":723,"num_object_clones":316,"num_object_copies":2169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":723,"num_whiteouts":11,"num_read":10105,"num_read_kb":7392142,"num_write":6574,"num_write_kb":2015923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":704,"num_bytes_recovered":255536541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25062,"ondisk_log_size":25062,"up":564,"acting":564,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":564,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2206648,"kb_used_data":1864192,"kb_used_omap":0,"kb_used_meta":342400,"kb_avail":654009416,"statfs":{"total":671965249536,"available":669705641984,"internally_reserved":0,"allocated":1908932608,"data_stored":2514772104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":350617600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4],"upper_bound":524288},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16905628,"num_objects":-5,"num_object_clones":-4,"num_object_copies":-15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5,"num_whiteouts":0,"num_read":-213,"num_read_kb":-196907,"num_write":-201,"num_write_kb":-77788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-5,"num_bytes_recovered":-403693,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.929847"},"pg_stats":[{"pgid":"6.29","version":"0'0","reported_seq":"1050","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.396655+0000","last_change":"2021-02-10T04:45:49.759370+0000","last_active":"2021-02-10T04:46:43.396655+0000","last_peered":"2021-02-10T04:46:43.396655+0000","last_clean":"2021-02-10T04:46:43.396655+0000","last_became_active":"2021-02-10T04:45:49.759065+0000","last_became_peered":"2021-02-10T04:45:49.759065+0000","last_unstale":"2021-02-10T04:46:43.396655+0000","last_undegraded":"2021-02-10T04:46:43.396655+0000","last_fullsized":"2021-02-10T04:46:43.396655+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:33.970536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:44:33.970536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":"442","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.403233+0000","last_change":"2021-02-10T04:45:49.761885+0000","last_active":"2021-02-10T04:46:38.403233+0000","last_peered":"2021-02-10T04:46:38.403233+0000","last_clean":"2021-02-10T04:46:38.403233+0000","last_became_active":"2021-02-10T04:45:49.761512+0000","last_became_peered":"2021-02-10T04:45:49.761512+0000","last_unstale":"2021-02-10T04:46:38.403233+0000","last_undegraded":"2021-02-10T04:46:38.403233+0000","last_fullsized":"2021-02-10T04:46:38.403233+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.731458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.731458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"131'3","reported_seq":"1169","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:42.934537+0000","last_change":"2021-02-10T04:46:29.097763+0000","last_active":"2021-02-10T04:46:42.934537+0000","last_peered":"2021-02-10T04:46:42.934537+0000","last_clean":"2021-02-10T04:46:42.934537+0000","last_became_active":"2021-02-10T04:29:07.145525+0000","last_became_peered":"2021-02-10T04:29:07.145525+0000","last_unstale":"2021-02-10T04:46:42.934537+0000","last_undegraded":"2021-02-10T04:46:42.934537+0000","last_fullsized":"2021-02-10T04:46:42.934537+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'3","last_scrub_stamp":"2021-02-10T04:46:29.097651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:46:29.097651+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"1049'54","reported_seq":"1121","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.800595+0000","last_change":"2021-02-10T04:33:55.585872+0000","last_active":"2021-02-10T04:46:37.800595+0000","last_peered":"2021-02-10T04:46:37.800595+0000","last_clean":"2021-02-10T04:46:37.800595+0000","last_became_active":"2021-02-10T04:25:49.002058+0000","last_became_peered":"2021-02-10T04:25:49.002058+0000","last_unstale":"2021-02-10T04:46:37.800595+0000","last_undegraded":"2021-02-10T04:46:37.800595+0000","last_fullsized":"2021-02-10T04:46:37.800595+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2021-02-10T04:33:55.585807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:55.585807+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":95,"num_read_kb":78,"num_write":55,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"463","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.401311+0000","last_change":"2021-02-10T04:46:34.577765+0000","last_active":"2021-02-10T04:46:43.401311+0000","last_peered":"2021-02-10T04:46:43.401311+0000","last_clean":"2021-02-10T04:46:43.401311+0000","last_became_active":"2021-02-10T04:37:37.246903+0000","last_became_peered":"2021-02-10T04:37:37.246903+0000","last_unstale":"2021-02-10T04:46:43.401311+0000","last_undegraded":"2021-02-10T04:46:43.401311+0000","last_fullsized":"2021-02-10T04:46:43.401311+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:46:34.577684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:34.577684+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"1020","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.804221+0000","last_change":"2021-02-10T04:46:16.042428+0000","last_active":"2021-02-10T04:46:37.804221+0000","last_peered":"2021-02-10T04:46:37.804221+0000","last_clean":"2021-02-10T04:46:37.804221+0000","last_became_active":"2021-02-10T04:25:53.148090+0000","last_became_peered":"2021-02-10T04:25:53.148090+0000","last_unstale":"2021-02-10T04:46:37.804221+0000","last_undegraded":"2021-02-10T04:46:37.804221+0000","last_fullsized":"2021-02-10T04:46:37.804221+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:16.042351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:46:16.042351+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"1003'1689","reported_seq":"4588","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.802209+0000","last_change":"2021-02-10T04:44:03.811251+0000","last_active":"2021-02-10T04:46:37.802209+0000","last_peered":"2021-02-10T04:46:37.802209+0000","last_clean":"2021-02-10T04:46:37.802209+0000","last_became_active":"2021-02-10T04:33:05.144488+0000","last_became_peered":"2021-02-10T04:33:05.144488+0000","last_unstale":"2021-02-10T04:46:37.802209+0000","last_undegraded":"2021-02-10T04:46:37.802209+0000","last_fullsized":"2021-02-10T04:46:37.802209+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'301","last_scrub_stamp":"2021-02-10T04:33:56.571925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:56.571925+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":126521476,"num_objects":49,"num_object_clones":44,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":773,"num_read_kb":819582,"num_write":561,"num_write_kb":221117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"1049'32","reported_seq":"1121","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.398301+0000","last_change":"2021-02-10T04:46:31.614762+0000","last_active":"2021-02-10T04:46:43.398301+0000","last_peered":"2021-02-10T04:46:43.398301+0000","last_clean":"2021-02-10T04:46:43.398301+0000","last_became_active":"2021-02-10T04:38:39.243413+0000","last_became_peered":"2021-02-10T04:38:39.243413+0000","last_unstale":"2021-02-10T04:46:43.398301+0000","last_undegraded":"2021-02-10T04:46:43.398301+0000","last_fullsized":"2021-02-10T04:46:43.398301+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'32","last_scrub_stamp":"2021-02-10T04:46:31.614697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:31.614697+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":62,"num_read_kb":45,"num_write":33,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":894,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"704","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.805634+0000","last_change":"2021-02-10T04:44:43.015787+0000","last_active":"2021-02-10T04:46:37.805634+0000","last_peered":"2021-02-10T04:46:37.805634+0000","last_clean":"2021-02-10T04:46:37.805634+0000","last_became_active":"2021-02-10T04:33:05.150882+0000","last_became_peered":"2021-02-10T04:33:05.150882+0000","last_unstale":"2021-02-10T04:46:37.805634+0000","last_undegraded":"2021-02-10T04:46:37.805634+0000","last_fullsized":"2021-02-10T04:46:37.805634+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:43.015723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:44:43.015723+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"428","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.804002+0000","last_change":"2021-02-10T04:37:37.244470+0000","last_active":"2021-02-10T04:46:37.804002+0000","last_peered":"2021-02-10T04:46:37.804002+0000","last_clean":"2021-02-10T04:46:37.804002+0000","last_became_active":"2021-02-10T04:37:37.242485+0000","last_became_peered":"2021-02-10T04:37:37.242485+0000","last_unstale":"2021-02-10T04:46:37.804002+0000","last_undegraded":"2021-02-10T04:46:37.804002+0000","last_fullsized":"2021-02-10T04:46:37.804002+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:35.805326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.805326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"1042","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.400978+0000","last_change":"2021-02-10T04:45:47.538655+0000","last_active":"2021-02-10T04:46:43.400978+0000","last_peered":"2021-02-10T04:46:43.400978+0000","last_clean":"2021-02-10T04:46:43.400978+0000","last_became_active":"2021-02-10T04:45:47.537914+0000","last_became_peered":"2021-02-10T04:45:47.537914+0000","last_unstale":"2021-02-10T04:46:43.400978+0000","last_undegraded":"2021-02-10T04:46:43.400978+0000","last_fullsized":"2021-02-10T04:46:43.400978+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1045,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:33.970536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:44:33.970536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"259'74","reported_seq":"1956","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.802666+0000","last_change":"2021-02-10T04:44:25.545822+0000","last_active":"2021-02-10T04:46:37.802666+0000","last_peered":"2021-02-10T04:46:37.802666+0000","last_clean":"2021-02-10T04:46:37.802666+0000","last_became_active":"2021-02-10T04:44:25.544898+0000","last_became_peered":"2021-02-10T04:44:25.544898+0000","last_unstale":"2021-02-10T04:46:37.802666+0000","last_undegraded":"2021-02-10T04:46:37.802666+0000","last_fullsized":"2021-02-10T04:46:37.802666+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'62","last_scrub_stamp":"2021-02-10T04:30:10.905905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:10.905905+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1218524,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":28227,"num_write":19,"num_write_kb":7021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":567285,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"1049'61","reported_seq":"1135","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.403734+0000","last_change":"2021-02-10T04:44:34.706772+0000","last_active":"2021-02-10T04:46:38.403734+0000","last_peered":"2021-02-10T04:46:38.403734+0000","last_clean":"2021-02-10T04:46:38.403734+0000","last_became_active":"2021-02-10T04:25:49.015913+0000","last_became_peered":"2021-02-10T04:25:49.015913+0000","last_unstale":"2021-02-10T04:46:38.403734+0000","last_undegraded":"2021-02-10T04:46:38.403734+0000","last_fullsized":"2021-02-10T04:46:38.403734+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'45","last_scrub_stamp":"2021-02-10T04:44:34.706681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:44:34.706681+0000","log_size":61,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":104,"num_read_kb":84,"num_write":62,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"642'3","reported_seq":"569","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.399059+0000","last_change":"2021-02-10T04:38:24.163263+0000","last_active":"2021-02-10T04:46:38.399059+0000","last_peered":"2021-02-10T04:46:38.399059+0000","last_clean":"2021-02-10T04:46:38.399059+0000","last_became_active":"2021-02-10T04:37:37.234230+0000","last_became_peered":"2021-02-10T04:37:37.234230+0000","last_unstale":"2021-02-10T04:46:38.399059+0000","last_undegraded":"2021-02-10T04:46:38.399059+0000","last_fullsized":"2021-02-10T04:46:38.399059+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'3","last_scrub_stamp":"2021-02-10T04:38:24.163200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:24.163200+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"441","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.398129+0000","last_change":"2021-02-10T04:46:25.712022+0000","last_active":"2021-02-10T04:46:43.398129+0000","last_peered":"2021-02-10T04:46:43.398129+0000","last_clean":"2021-02-10T04:46:43.398129+0000","last_became_active":"2021-02-10T04:37:37.230818+0000","last_became_peered":"2021-02-10T04:37:37.230818+0000","last_unstale":"2021-02-10T04:46:43.398129+0000","last_undegraded":"2021-02-10T04:46:43.398129+0000","last_fullsized":"2021-02-10T04:46:43.398129+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:25.711938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:46:25.711938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"443","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.398275+0000","last_change":"2021-02-10T04:45:47.534623+0000","last_active":"2021-02-10T04:46:43.398275+0000","last_peered":"2021-02-10T04:46:43.398275+0000","last_clean":"2021-02-10T04:46:43.398275+0000","last_became_active":"2021-02-10T04:45:47.534309+0000","last_became_peered":"2021-02-10T04:45:47.534309+0000","last_unstale":"2021-02-10T04:46:43.398275+0000","last_undegraded":"2021-02-10T04:46:43.398275+0000","last_fullsized":"2021-02-10T04:46:43.398275+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1045,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.731458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.731458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"1003'739","reported_seq":"3718","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.398504+0000","last_change":"2021-02-10T04:44:25.546236+0000","last_active":"2021-02-10T04:46:43.398504+0000","last_peered":"2021-02-10T04:46:43.398504+0000","last_clean":"2021-02-10T04:46:43.398504+0000","last_became_active":"2021-02-10T04:44:25.545483+0000","last_became_peered":"2021-02-10T04:44:25.545483+0000","last_unstale":"2021-02-10T04:46:43.398504+0000","last_undegraded":"2021-02-10T04:46:43.398504+0000","last_fullsized":"2021-02-10T04:46:43.398504+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'63","last_scrub_stamp":"2021-02-10T04:29:39.700115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:29:39.700115+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24059888,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":129,"num_read_kb":109544,"num_write":102,"num_write_kb":46173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"1049'61","reported_seq":"1148","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.805836+0000","last_change":"2021-02-10T04:46:20.788190+0000","last_active":"2021-02-10T04:46:37.805836+0000","last_peered":"2021-02-10T04:46:37.805836+0000","last_clean":"2021-02-10T04:46:37.805836+0000","last_became_active":"2021-02-10T04:37:37.237215+0000","last_became_peered":"2021-02-10T04:37:37.237215+0000","last_unstale":"2021-02-10T04:46:37.805836+0000","last_undegraded":"2021-02-10T04:46:37.805836+0000","last_fullsized":"2021-02-10T04:46:37.805836+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'61","last_scrub_stamp":"2021-02-10T04:46:20.788088+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:46:20.788088+0000","log_size":61,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":83,"num_read_kb":65,"num_write":50,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"941","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.804367+0000","last_change":"2021-02-10T04:46:21.815240+0000","last_active":"2021-02-10T04:46:37.804367+0000","last_peered":"2021-02-10T04:46:37.804367+0000","last_clean":"2021-02-10T04:46:37.804367+0000","last_became_active":"2021-02-10T04:29:07.145616+0000","last_became_peered":"2021-02-10T04:29:07.145616+0000","last_unstale":"2021-02-10T04:46:37.804367+0000","last_undegraded":"2021-02-10T04:46:37.804367+0000","last_fullsized":"2021-02-10T04:46:37.804367+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:21.815158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:21.815158+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"699","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.804749+0000","last_change":"2021-02-10T04:33:05.147166+0000","last_active":"2021-02-10T04:46:37.804749+0000","last_peered":"2021-02-10T04:46:37.804749+0000","last_clean":"2021-02-10T04:46:37.804749+0000","last_became_active":"2021-02-10T04:33:05.146783+0000","last_became_peered":"2021-02-10T04:33:05.146783+0000","last_unstale":"2021-02-10T04:46:37.804749+0000","last_undegraded":"2021-02-10T04:46:37.804749+0000","last_fullsized":"2021-02-10T04:46:37.804749+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:35.958255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:29:35.958255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"436","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.401469+0000","last_change":"2021-02-10T04:38:32.875574+0000","last_active":"2021-02-10T04:46:43.401469+0000","last_peered":"2021-02-10T04:46:43.401469+0000","last_clean":"2021-02-10T04:46:43.401469+0000","last_became_active":"2021-02-10T04:37:37.251479+0000","last_became_peered":"2021-02-10T04:37:37.251479+0000","last_unstale":"2021-02-10T04:46:43.401469+0000","last_undegraded":"2021-02-10T04:46:43.401469+0000","last_fullsized":"2021-02-10T04:46:43.401469+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:32.875498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:38:32.875498+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"999'629","reported_seq":"4221","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.803571+0000","last_change":"2021-02-10T04:43:53.673766+0000","last_active":"2021-02-10T04:46:37.803571+0000","last_peered":"2021-02-10T04:46:37.803571+0000","last_clean":"2021-02-10T04:46:37.803571+0000","last_became_active":"2021-02-10T04:37:40.267794+0000","last_became_peered":"2021-02-10T04:37:40.267794+0000","last_unstale":"2021-02-10T04:46:37.803571+0000","last_undegraded":"2021-02-10T04:46:37.803571+0000","last_fullsized":"2021-02-10T04:46:37.803571+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"688'187","last_scrub_stamp":"2021-02-10T04:38:27.225488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:38:27.225488+0000","log_size":629,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49401261,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":2,"num_read":342,"num_read_kb":364544,"num_write":223,"num_write_kb":67454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":11919937,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"972","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.800673+0000","last_change":"2021-02-10T04:46:33.963106+0000","last_active":"2021-02-10T04:46:37.800673+0000","last_peered":"2021-02-10T04:46:37.800673+0000","last_clean":"2021-02-10T04:46:37.800673+0000","last_became_active":"2021-02-10T04:37:37.246004+0000","last_became_peered":"2021-02-10T04:37:37.246004+0000","last_unstale":"2021-02-10T04:46:37.800673+0000","last_undegraded":"2021-02-10T04:46:37.800673+0000","last_fullsized":"2021-02-10T04:46:37.800673+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:33.963005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:46:33.963005+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"1043","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.398022+0000","last_change":"2021-02-10T04:37:37.259938+0000","last_active":"2021-02-10T04:46:43.398022+0000","last_peered":"2021-02-10T04:46:43.398022+0000","last_clean":"2021-02-10T04:46:43.398022+0000","last_became_active":"2021-02-10T04:37:37.259532+0000","last_became_peered":"2021-02-10T04:37:37.259532+0000","last_unstale":"2021-02-10T04:46:43.398022+0000","last_undegraded":"2021-02-10T04:46:43.398022+0000","last_fullsized":"2021-02-10T04:46:43.398022+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:51.858386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:33:51.858386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"1049'33","reported_seq":"1074","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.804832+0000","last_change":"2021-02-10T04:44:56.117260+0000","last_active":"2021-02-10T04:46:37.804832+0000","last_peered":"2021-02-10T04:46:37.804832+0000","last_clean":"2021-02-10T04:46:37.804832+0000","last_became_active":"2021-02-10T04:25:49.001725+0000","last_became_peered":"2021-02-10T04:25:49.001725+0000","last_unstale":"2021-02-10T04:46:37.804832+0000","last_undegraded":"2021-02-10T04:46:37.804832+0000","last_fullsized":"2021-02-10T04:46:37.804832+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'23","last_scrub_stamp":"2021-02-10T04:44:56.117138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:44:56.117138+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":43,"num_write":33,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"711","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.398480+0000","last_change":"2021-02-10T04:46:19.573168+0000","last_active":"2021-02-10T04:46:43.398480+0000","last_peered":"2021-02-10T04:46:43.398480+0000","last_clean":"2021-02-10T04:46:43.398480+0000","last_became_active":"2021-02-10T04:33:05.168314+0000","last_became_peered":"2021-02-10T04:33:05.168314+0000","last_unstale":"2021-02-10T04:46:43.398480+0000","last_undegraded":"2021-02-10T04:46:43.398480+0000","last_fullsized":"2021-02-10T04:46:43.398480+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:19.573103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:19.573103+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"639'3","reported_seq":"557","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.399533+0000","last_change":"2021-02-10T04:38:31.211769+0000","last_active":"2021-02-10T04:46:38.399533+0000","last_peered":"2021-02-10T04:46:38.399533+0000","last_clean":"2021-02-10T04:46:38.399533+0000","last_became_active":"2021-02-10T04:37:37.233738+0000","last_became_peered":"2021-02-10T04:37:37.233738+0000","last_unstale":"2021-02-10T04:46:38.399533+0000","last_undegraded":"2021-02-10T04:46:38.399533+0000","last_fullsized":"2021-02-10T04:46:38.399533+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"639'3","last_scrub_stamp":"2021-02-10T04:38:31.211662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:38:31.211662+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"1023","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.805995+0000","last_change":"2021-02-10T04:37:37.236009+0000","last_active":"2021-02-10T04:46:37.805995+0000","last_peered":"2021-02-10T04:46:37.805995+0000","last_clean":"2021-02-10T04:46:37.805995+0000","last_became_active":"2021-02-10T04:37:37.233109+0000","last_became_peered":"2021-02-10T04:37:37.233109+0000","last_unstale":"2021-02-10T04:46:37.805995+0000","last_undegraded":"2021-02-10T04:46:37.805995+0000","last_fullsized":"2021-02-10T04:46:37.805995+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:38.873102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:38.873102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"1004'1093","reported_seq":"6281","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.401276+0000","last_change":"2021-02-10T04:46:19.758247+0000","last_active":"2021-02-10T04:46:43.401276+0000","last_peered":"2021-02-10T04:46:43.401276+0000","last_clean":"2021-02-10T04:46:43.401276+0000","last_became_active":"2021-02-10T04:28:25.149226+0000","last_became_peered":"2021-02-10T04:28:25.149226+0000","last_unstale":"2021-02-10T04:46:43.401276+0000","last_undegraded":"2021-02-10T04:46:43.401276+0000","last_fullsized":"2021-02-10T04:46:43.401276+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:46:19.757486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:19.757486+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":78869393,"num_objects":34,"num_object_clones":30,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":1,"num_read":558,"num_read_kb":622116,"num_write":371,"num_write_kb":133580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"972","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.804082+0000","last_change":"2021-02-10T04:46:31.043636+0000","last_active":"2021-02-10T04:46:37.804082+0000","last_peered":"2021-02-10T04:46:37.804082+0000","last_clean":"2021-02-10T04:46:37.804082+0000","last_became_active":"2021-02-10T04:37:37.229648+0000","last_became_peered":"2021-02-10T04:37:37.229648+0000","last_unstale":"2021-02-10T04:46:37.804082+0000","last_undegraded":"2021-02-10T04:46:37.804082+0000","last_fullsized":"2021-02-10T04:46:37.804082+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:31.043538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:46:31.043538+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"1049'41","reported_seq":"785","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.401337+0000","last_change":"2021-02-10T04:46:28.515609+0000","last_active":"2021-02-10T04:46:43.401337+0000","last_peered":"2021-02-10T04:46:43.401337+0000","last_clean":"2021-02-10T04:46:43.401337+0000","last_became_active":"2021-02-10T04:33:05.149026+0000","last_became_peered":"2021-02-10T04:33:05.149026+0000","last_unstale":"2021-02-10T04:46:43.401337+0000","last_undegraded":"2021-02-10T04:46:43.401337+0000","last_fullsized":"2021-02-10T04:46:43.401337+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'41","last_scrub_stamp":"2021-02-10T04:46:28.515513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:28.515513+0000","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":67,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"766","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.398554+0000","last_change":"2021-02-10T04:33:05.169654+0000","last_active":"2021-02-10T04:46:43.398554+0000","last_peered":"2021-02-10T04:46:43.398554+0000","last_clean":"2021-02-10T04:46:43.398554+0000","last_became_active":"2021-02-10T04:33:05.169289+0000","last_became_peered":"2021-02-10T04:33:05.169289+0000","last_unstale":"2021-02-10T04:46:43.398554+0000","last_undegraded":"2021-02-10T04:46:43.398554+0000","last_fullsized":"2021-02-10T04:46:43.398554+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_clean_scrub_stamp":"2021-02-10T04:28:39.128781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"78'3","reported_seq":"1274","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.933280+0000","last_change":"2021-02-10T04:46:17.816956+0000","last_active":"2021-02-10T04:46:37.933280+0000","last_peered":"2021-02-10T04:46:37.933280+0000","last_clean":"2021-02-10T04:46:37.933280+0000","last_became_active":"2021-02-10T04:25:51.025218+0000","last_became_peered":"2021-02-10T04:25:51.025218+0000","last_unstale":"2021-02-10T04:46:37.933280+0000","last_undegraded":"2021-02-10T04:46:37.933280+0000","last_fullsized":"2021-02-10T04:46:37.933280+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:46:17.816879+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:46:17.816879+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"1020","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.803268+0000","last_change":"2021-02-10T04:44:55.693471+0000","last_active":"2021-02-10T04:46:37.803268+0000","last_peered":"2021-02-10T04:46:37.803268+0000","last_clean":"2021-02-10T04:46:37.803268+0000","last_became_active":"2021-02-10T04:25:53.042697+0000","last_became_peered":"2021-02-10T04:25:53.042697+0000","last_unstale":"2021-02-10T04:46:37.803268+0000","last_undegraded":"2021-02-10T04:46:37.803268+0000","last_fullsized":"2021-02-10T04:46:37.803268+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:55.693386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:44:55.693386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"1002'1090","reported_seq":"5220","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.403427+0000","last_change":"2021-02-10T04:46:19.084188+0000","last_active":"2021-02-10T04:46:38.403427+0000","last_peered":"2021-02-10T04:46:38.403427+0000","last_clean":"2021-02-10T04:46:38.403427+0000","last_became_active":"2021-02-10T04:28:25.148500+0000","last_became_peered":"2021-02-10T04:28:25.148500+0000","last_unstale":"2021-02-10T04:46:38.403427+0000","last_undegraded":"2021-02-10T04:46:38.403427+0000","last_fullsized":"2021-02-10T04:46:38.403427+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:46:19.084082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:19.084082+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":48821292,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":491,"num_read_kb":552336,"num_write":351,"num_write_kb":129921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"1049'47","reported_seq":"1099","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.805661+0000","last_change":"2021-02-10T04:44:24.905303+0000","last_active":"2021-02-10T04:46:37.805661+0000","last_peered":"2021-02-10T04:46:37.805661+0000","last_clean":"2021-02-10T04:46:37.805661+0000","last_became_active":"2021-02-10T04:25:49.011093+0000","last_became_peered":"2021-02-10T04:25:49.011093+0000","last_unstale":"2021-02-10T04:46:37.805661+0000","last_undegraded":"2021-02-10T04:46:37.805661+0000","last_fullsized":"2021-02-10T04:46:37.805661+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'39","last_scrub_stamp":"2021-02-10T04:44:24.905188+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:44:24.905188+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":68,"num_read_kb":52,"num_write":38,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"1007","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.396735+0000","last_change":"2021-02-10T04:44:56.759342+0000","last_active":"2021-02-10T04:46:43.396735+0000","last_peered":"2021-02-10T04:46:43.396735+0000","last_clean":"2021-02-10T04:46:43.396735+0000","last_became_active":"2021-02-10T04:29:07.160593+0000","last_became_peered":"2021-02-10T04:29:07.160593+0000","last_unstale":"2021-02-10T04:46:43.396735+0000","last_undegraded":"2021-02-10T04:46:43.396735+0000","last_fullsized":"2021-02-10T04:46:43.396735+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:56.759233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:44:56.759233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"1022","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.802524+0000","last_change":"2021-02-10T04:30:10.487631+0000","last_active":"2021-02-10T04:46:37.802524+0000","last_peered":"2021-02-10T04:46:37.802524+0000","last_clean":"2021-02-10T04:46:37.802524+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:46:37.802524+0000","last_undegraded":"2021-02-10T04:46:37.802524+0000","last_fullsized":"2021-02-10T04:46:37.802524+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:10.487528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:30:10.487528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"1016","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.400587+0000","last_change":"2021-02-10T04:33:55.849675+0000","last_active":"2021-02-10T04:46:43.400587+0000","last_peered":"2021-02-10T04:46:43.400587+0000","last_clean":"2021-02-10T04:46:43.400587+0000","last_became_active":"2021-02-10T04:25:53.148054+0000","last_became_peered":"2021-02-10T04:25:53.148054+0000","last_unstale":"2021-02-10T04:46:43.400587+0000","last_undegraded":"2021-02-10T04:46:43.400587+0000","last_fullsized":"2021-02-10T04:46:43.400587+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:55.849582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:33:55.849582+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"1001'1165","reported_seq":"4199","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.403915+0000","last_change":"2021-02-10T04:46:30.078162+0000","last_active":"2021-02-10T04:46:38.403915+0000","last_peered":"2021-02-10T04:46:38.403915+0000","last_clean":"2021-02-10T04:46:38.403915+0000","last_became_active":"2021-02-10T04:28:25.152030+0000","last_became_peered":"2021-02-10T04:28:25.152030+0000","last_unstale":"2021-02-10T04:46:38.403915+0000","last_undegraded":"2021-02-10T04:46:38.403915+0000","last_fullsized":"2021-02-10T04:46:38.403915+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:46:30.077967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:30.077967+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":59203690,"num_objects":25,"num_object_clones":21,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":573,"num_read_kb":599059,"num_write":400,"num_write_kb":178261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"158'3","reported_seq":"1277","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.397645+0000","last_change":"2021-02-10T04:46:36.557107+0000","last_active":"2021-02-10T04:46:43.397645+0000","last_peered":"2021-02-10T04:46:43.397645+0000","last_clean":"2021-02-10T04:46:43.397645+0000","last_became_active":"2021-02-10T04:25:51.031906+0000","last_became_peered":"2021-02-10T04:25:51.031906+0000","last_unstale":"2021-02-10T04:46:43.397645+0000","last_undegraded":"2021-02-10T04:46:43.397645+0000","last_fullsized":"2021-02-10T04:46:43.397645+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'3","last_scrub_stamp":"2021-02-10T04:46:36.557037+0000","last_deep_scrub":"158'3","last_deep_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_clean_scrub_stamp":"2021-02-10T04:46:36.557037+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"1049'54","reported_seq":"1113","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.776905+0000","last_change":"2021-02-10T04:46:38.776905+0000","last_active":"2021-02-10T04:46:38.776905+0000","last_peered":"2021-02-10T04:46:38.776905+0000","last_clean":"2021-02-10T04:46:38.776905+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:46:38.776905+0000","last_undegraded":"2021-02-10T04:46:38.776905+0000","last_fullsized":"2021-02-10T04:46:38.776905+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:46:38.776809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:38.776809+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":93,"num_read_kb":73,"num_write":54,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"1073","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.404965+0000","last_change":"2021-02-10T04:44:24.719062+0000","last_active":"2021-02-10T04:46:38.404965+0000","last_peered":"2021-02-10T04:46:38.404965+0000","last_clean":"2021-02-10T04:46:38.404965+0000","last_became_active":"2021-02-10T04:37:37.245391+0000","last_became_peered":"2021-02-10T04:37:37.245391+0000","last_unstale":"2021-02-10T04:46:38.404965+0000","last_undegraded":"2021-02-10T04:46:38.404965+0000","last_fullsized":"2021-02-10T04:46:38.404965+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:24.718981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:44:24.718981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"966","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.404505+0000","last_change":"2021-02-10T04:45:47.534475+0000","last_active":"2021-02-10T04:46:38.404505+0000","last_peered":"2021-02-10T04:46:38.404505+0000","last_clean":"2021-02-10T04:46:38.404505+0000","last_became_active":"2021-02-10T04:45:47.533648+0000","last_became_peered":"2021-02-10T04:45:47.533648+0000","last_unstale":"2021-02-10T04:46:38.404505+0000","last_undegraded":"2021-02-10T04:46:38.404505+0000","last_fullsized":"2021-02-10T04:46:38.404505+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1045,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:52.779754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:44:52.779754+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"998'1009","reported_seq":"4551","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.398046+0000","last_change":"2021-02-10T04:44:25.543675+0000","last_active":"2021-02-10T04:46:43.398046+0000","last_peered":"2021-02-10T04:46:43.398046+0000","last_clean":"2021-02-10T04:46:43.398046+0000","last_became_active":"2021-02-10T04:44:25.543138+0000","last_became_peered":"2021-02-10T04:44:25.543138+0000","last_unstale":"2021-02-10T04:46:43.398046+0000","last_undegraded":"2021-02-10T04:46:43.398046+0000","last_fullsized":"2021-02-10T04:46:43.398046+0000","mapping_epoch":646,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24710413,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":239,"num_read_kb":249403,"num_write":154,"num_write_kb":50104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":8290058,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"960","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.400666+0000","last_change":"2021-02-10T04:30:12.452072+0000","last_active":"2021-02-10T04:46:43.400666+0000","last_peered":"2021-02-10T04:46:43.400666+0000","last_clean":"2021-02-10T04:46:43.400666+0000","last_became_active":"2021-02-10T04:28:49.048525+0000","last_became_peered":"2021-02-10T04:28:49.048525+0000","last_unstale":"2021-02-10T04:46:43.400666+0000","last_undegraded":"2021-02-10T04:46:43.400666+0000","last_fullsized":"2021-02-10T04:46:43.400666+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:12.452000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:30:12.452000+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"1021","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.803397+0000","last_change":"2021-02-10T04:46:35.785621+0000","last_active":"2021-02-10T04:46:37.803397+0000","last_peered":"2021-02-10T04:46:37.803397+0000","last_clean":"2021-02-10T04:46:37.803397+0000","last_became_active":"2021-02-10T04:25:51.018706+0000","last_became_peered":"2021-02-10T04:25:51.018706+0000","last_unstale":"2021-02-10T04:46:37.803397+0000","last_undegraded":"2021-02-10T04:46:37.803397+0000","last_fullsized":"2021-02-10T04:46:37.803397+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:35.785517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:46:35.785517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"1049'72","reported_seq":"1145","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.398421+0000","last_change":"2021-02-10T04:46:30.515128+0000","last_active":"2021-02-10T04:46:43.398421+0000","last_peered":"2021-02-10T04:46:43.398421+0000","last_clean":"2021-02-10T04:46:43.398421+0000","last_became_active":"2021-02-10T04:25:48.997481+0000","last_became_peered":"2021-02-10T04:25:48.997481+0000","last_unstale":"2021-02-10T04:46:43.398421+0000","last_undegraded":"2021-02-10T04:46:43.398421+0000","last_fullsized":"2021-02-10T04:46:43.398421+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'72","last_scrub_stamp":"2021-02-10T04:46:30.515026+0000","last_deep_scrub":"152'48","last_deep_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_clean_scrub_stamp":"2021-02-10T04:46:30.515026+0000","log_size":72,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":106,"num_read_kb":84,"num_write":60,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"719","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.404650+0000","last_change":"2021-02-10T04:46:22.069341+0000","last_active":"2021-02-10T04:46:38.404650+0000","last_peered":"2021-02-10T04:46:38.404650+0000","last_clean":"2021-02-10T04:46:38.404650+0000","last_became_active":"2021-02-10T04:33:05.141066+0000","last_became_peered":"2021-02-10T04:33:05.141066+0000","last_unstale":"2021-02-10T04:46:38.404650+0000","last_undegraded":"2021-02-10T04:46:38.404650+0000","last_fullsized":"2021-02-10T04:46:38.404650+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:46:22.069247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:22.069247+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"1037","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.804046+0000","last_change":"2021-02-10T04:45:47.535163+0000","last_active":"2021-02-10T04:46:37.804046+0000","last_peered":"2021-02-10T04:46:37.804046+0000","last_clean":"2021-02-10T04:46:37.804046+0000","last_became_active":"2021-02-10T04:45:47.534798+0000","last_became_peered":"2021-02-10T04:45:47.534798+0000","last_unstale":"2021-02-10T04:46:37.804046+0000","last_undegraded":"2021-02-10T04:46:37.804046+0000","last_fullsized":"2021-02-10T04:46:37.804046+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1045,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:27.926662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:44:27.926662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"1002'1657","reported_seq":"4966","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.400219+0000","last_change":"2021-02-10T04:44:25.550700+0000","last_active":"2021-02-10T04:46:43.400219+0000","last_peered":"2021-02-10T04:46:43.400219+0000","last_clean":"2021-02-10T04:46:43.400219+0000","last_became_active":"2021-02-10T04:44:25.549094+0000","last_became_peered":"2021-02-10T04:44:25.549094+0000","last_unstale":"2021-02-10T04:46:43.400219+0000","last_undegraded":"2021-02-10T04:46:43.400219+0000","last_fullsized":"2021-02-10T04:46:43.400219+0000","mapping_epoch":646,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"687'1095","last_scrub_stamp":"2021-02-10T04:38:27.895743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:38:27.895743+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":42793066,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":424,"num_read_kb":437673,"num_write":318,"num_write_kb":118797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":17936938,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"971","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.802338+0000","last_change":"2021-02-10T04:46:30.848110+0000","last_active":"2021-02-10T04:46:37.802338+0000","last_peered":"2021-02-10T04:46:37.802338+0000","last_clean":"2021-02-10T04:46:37.802338+0000","last_became_active":"2021-02-10T04:37:37.233158+0000","last_became_peered":"2021-02-10T04:37:37.233158+0000","last_unstale":"2021-02-10T04:46:37.802338+0000","last_undegraded":"2021-02-10T04:46:37.802338+0000","last_fullsized":"2021-02-10T04:46:37.802338+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:30.848045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:46:30.848045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"1094","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.403908+0000","last_change":"2021-02-10T04:46:24.021616+0000","last_active":"2021-02-10T04:46:38.403908+0000","last_peered":"2021-02-10T04:46:38.403908+0000","last_clean":"2021-02-10T04:46:38.403908+0000","last_became_active":"2021-02-10T04:25:51.025636+0000","last_became_peered":"2021-02-10T04:25:51.025636+0000","last_unstale":"2021-02-10T04:46:38.403908+0000","last_undegraded":"2021-02-10T04:46:38.403908+0000","last_fullsized":"2021-02-10T04:46:38.403908+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:24.021552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:46:24.021552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"1049'47","reported_seq":"1116","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.398050+0000","last_change":"2021-02-10T04:46:29.672214+0000","last_active":"2021-02-10T04:46:43.398050+0000","last_peered":"2021-02-10T04:46:43.398050+0000","last_clean":"2021-02-10T04:46:43.398050+0000","last_became_active":"2021-02-10T04:25:49.002937+0000","last_became_peered":"2021-02-10T04:25:49.002937+0000","last_unstale":"2021-02-10T04:46:43.398050+0000","last_undegraded":"2021-02-10T04:46:43.398050+0000","last_fullsized":"2021-02-10T04:46:43.398050+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'47","last_scrub_stamp":"2021-02-10T04:46:29.672141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:29.672141+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":87,"num_read_kb":68,"num_write":50,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"960","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.400655+0000","last_change":"2021-02-10T04:46:09.751739+0000","last_active":"2021-02-10T04:46:43.400655+0000","last_peered":"2021-02-10T04:46:43.400655+0000","last_clean":"2021-02-10T04:46:43.400655+0000","last_became_active":"2021-02-10T04:29:07.161341+0000","last_became_peered":"2021-02-10T04:29:07.161341+0000","last_unstale":"2021-02-10T04:46:43.400655+0000","last_undegraded":"2021-02-10T04:46:43.400655+0000","last_fullsized":"2021-02-10T04:46:43.400655+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:46:09.751642+0000","last_deep_scrub":"53'2","last_deep_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_clean_scrub_stamp":"2021-02-10T04:46:09.751642+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"724","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.402977+0000","last_change":"2021-02-10T04:45:47.528412+0000","last_active":"2021-02-10T04:46:38.402977+0000","last_peered":"2021-02-10T04:46:38.402977+0000","last_clean":"2021-02-10T04:46:38.402977+0000","last_became_active":"2021-02-10T04:45:47.528080+0000","last_became_peered":"2021-02-10T04:45:47.528080+0000","last_unstale":"2021-02-10T04:46:38.402977+0000","last_undegraded":"2021-02-10T04:46:38.402977+0000","last_fullsized":"2021-02-10T04:46:38.402977+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1045,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:40.414601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:38:40.414601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"1000'1071","reported_seq":"4571","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.801016+0000","last_change":"2021-02-10T04:44:25.539713+0000","last_active":"2021-02-10T04:46:37.801016+0000","last_peered":"2021-02-10T04:46:37.801016+0000","last_clean":"2021-02-10T04:46:37.801016+0000","last_became_active":"2021-02-10T04:44:25.536840+0000","last_became_peered":"2021-02-10T04:44:25.536840+0000","last_unstale":"2021-02-10T04:46:37.801016+0000","last_undegraded":"2021-02-10T04:46:37.801016+0000","last_fullsized":"2021-02-10T04:46:37.801016+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'357","last_scrub_stamp":"2021-02-10T04:33:50.582002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:50.582002+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23286368,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":242,"num_read_kb":226834,"num_write":185,"num_write_kb":76004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"1045","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.404946+0000","last_change":"2021-02-10T04:44:35.689296+0000","last_active":"2021-02-10T04:46:38.404946+0000","last_peered":"2021-02-10T04:46:38.404946+0000","last_clean":"2021-02-10T04:46:38.404946+0000","last_became_active":"2021-02-10T04:28:49.054638+0000","last_became_peered":"2021-02-10T04:28:49.054638+0000","last_unstale":"2021-02-10T04:46:38.404946+0000","last_undegraded":"2021-02-10T04:46:38.404946+0000","last_fullsized":"2021-02-10T04:46:38.404946+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:35.689202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:44:35.689202+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"1022","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.898195+0000","last_change":"2021-02-10T04:46:37.898092+0000","last_active":"2021-02-10T04:46:37.898195+0000","last_peered":"2021-02-10T04:46:37.898195+0000","last_clean":"2021-02-10T04:46:37.898195+0000","last_became_active":"2021-02-10T04:25:51.032201+0000","last_became_peered":"2021-02-10T04:25:51.032201+0000","last_unstale":"2021-02-10T04:46:37.898195+0000","last_undegraded":"2021-02-10T04:46:37.898195+0000","last_fullsized":"2021-02-10T04:46:37.898195+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:37.897976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:46:37.897976+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"1049'40","reported_seq":"1124","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.399890+0000","last_change":"2021-02-10T04:46:18.777738+0000","last_active":"2021-02-10T04:46:43.399890+0000","last_peered":"2021-02-10T04:46:43.399890+0000","last_clean":"2021-02-10T04:46:43.399890+0000","last_became_active":"2021-02-10T04:37:37.247141+0000","last_became_peered":"2021-02-10T04:37:37.247141+0000","last_unstale":"2021-02-10T04:46:43.399890+0000","last_undegraded":"2021-02-10T04:46:43.399890+0000","last_fullsized":"2021-02-10T04:46:43.399890+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'40","last_scrub_stamp":"2021-02-10T04:46:18.777661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:18.777661+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":52,"num_write":40,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"763","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.397048+0000","last_change":"2021-02-10T04:46:15.788177+0000","last_active":"2021-02-10T04:46:43.397048+0000","last_peered":"2021-02-10T04:46:43.397048+0000","last_clean":"2021-02-10T04:46:43.397048+0000","last_became_active":"2021-02-10T04:33:05.166368+0000","last_became_peered":"2021-02-10T04:33:05.166368+0000","last_unstale":"2021-02-10T04:46:43.397048+0000","last_undegraded":"2021-02-10T04:46:43.397048+0000","last_fullsized":"2021-02-10T04:46:43.397048+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:15.788089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:15.788089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"1026","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.398722+0000","last_change":"2021-02-10T04:44:36.852583+0000","last_active":"2021-02-10T04:46:43.398722+0000","last_peered":"2021-02-10T04:46:43.398722+0000","last_clean":"2021-02-10T04:46:43.398722+0000","last_became_active":"2021-02-10T04:25:49.001782+0000","last_became_peered":"2021-02-10T04:25:49.001782+0000","last_unstale":"2021-02-10T04:46:43.398722+0000","last_undegraded":"2021-02-10T04:46:43.398722+0000","last_fullsized":"2021-02-10T04:46:43.398722+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:36.852513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:44:36.852513+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"709","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.806654+0000","last_change":"2021-02-10T04:45:47.538321+0000","last_active":"2021-02-10T04:46:37.806654+0000","last_peered":"2021-02-10T04:46:37.806654+0000","last_clean":"2021-02-10T04:46:37.806654+0000","last_became_active":"2021-02-10T04:45:47.537973+0000","last_became_peered":"2021-02-10T04:45:47.537973+0000","last_unstale":"2021-02-10T04:46:37.806654+0000","last_undegraded":"2021-02-10T04:46:37.806654+0000","last_fullsized":"2021-02-10T04:46:37.806654+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1045,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.098681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.098681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.5","version":"1000'446","reported_seq":"1978","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.399169+0000","last_change":"2021-02-10T04:46:38.774890+0000","last_active":"2021-02-10T04:46:43.399169+0000","last_peered":"2021-02-10T04:46:43.399169+0000","last_clean":"2021-02-10T04:46:43.399169+0000","last_became_active":"2021-02-10T04:44:25.549930+0000","last_became_peered":"2021-02-10T04:44:25.549930+0000","last_unstale":"2021-02-10T04:46:43.399169+0000","last_undegraded":"2021-02-10T04:46:43.399169+0000","last_fullsized":"2021-02-10T04:46:43.399169+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:46:38.774795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:38.774795+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19615419,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"785","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.404194+0000","last_change":"2021-02-10T04:46:21.050706+0000","last_active":"2021-02-10T04:46:38.404194+0000","last_peered":"2021-02-10T04:46:38.404194+0000","last_clean":"2021-02-10T04:46:38.404194+0000","last_became_active":"2021-02-10T04:33:05.140966+0000","last_became_peered":"2021-02-10T04:33:05.140966+0000","last_unstale":"2021-02-10T04:46:38.404194+0000","last_undegraded":"2021-02-10T04:46:38.404194+0000","last_fullsized":"2021-02-10T04:46:38.404194+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:21.050634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:46:21.050634+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"437","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.807075+0000","last_change":"2021-02-10T04:46:26.934663+0000","last_active":"2021-02-10T04:46:37.807075+0000","last_peered":"2021-02-10T04:46:37.807075+0000","last_clean":"2021-02-10T04:46:37.807075+0000","last_became_active":"2021-02-10T04:37:37.234644+0000","last_became_peered":"2021-02-10T04:37:37.234644+0000","last_unstale":"2021-02-10T04:46:37.807075+0000","last_undegraded":"2021-02-10T04:46:37.807075+0000","last_fullsized":"2021-02-10T04:46:37.807075+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:26.934585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:46:26.934585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"1049'39","reported_seq":"1090","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.403748+0000","last_change":"2021-02-10T04:46:34.039385+0000","last_active":"2021-02-10T04:46:38.403748+0000","last_peered":"2021-02-10T04:46:38.403748+0000","last_clean":"2021-02-10T04:46:38.403748+0000","last_became_active":"2021-02-10T04:25:49.002471+0000","last_became_peered":"2021-02-10T04:25:49.002471+0000","last_unstale":"2021-02-10T04:46:38.403748+0000","last_undegraded":"2021-02-10T04:46:38.403748+0000","last_fullsized":"2021-02-10T04:46:38.403748+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'39","last_scrub_stamp":"2021-02-10T04:46:34.039273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:34.039273+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":51,"num_write":39,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"1029","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.396808+0000","last_change":"2021-02-10T04:44:55.738512+0000","last_active":"2021-02-10T04:46:43.396808+0000","last_peered":"2021-02-10T04:46:43.396808+0000","last_clean":"2021-02-10T04:46:43.396808+0000","last_became_active":"2021-02-10T04:29:07.158075+0000","last_became_peered":"2021-02-10T04:29:07.158075+0000","last_unstale":"2021-02-10T04:46:43.396808+0000","last_undegraded":"2021-02-10T04:46:43.396808+0000","last_fullsized":"2021-02-10T04:46:43.396808+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:55.738445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:44:55.738445+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"1192","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.802424+0000","last_change":"2021-02-10T04:44:26.684937+0000","last_active":"2021-02-10T04:46:37.802424+0000","last_peered":"2021-02-10T04:46:37.802424+0000","last_clean":"2021-02-10T04:46:37.802424+0000","last_became_active":"2021-02-10T04:37:37.245931+0000","last_became_peered":"2021-02-10T04:37:37.245931+0000","last_unstale":"2021-02-10T04:46:37.802424+0000","last_undegraded":"2021-02-10T04:46:37.802424+0000","last_fullsized":"2021-02-10T04:46:37.802424+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:26.684870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:44:26.684870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"1025","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.398554+0000","last_change":"2021-02-10T04:46:17.537141+0000","last_active":"2021-02-10T04:46:43.398554+0000","last_peered":"2021-02-10T04:46:43.398554+0000","last_clean":"2021-02-10T04:46:43.398554+0000","last_became_active":"2021-02-10T04:25:49.010614+0000","last_became_peered":"2021-02-10T04:25:49.010614+0000","last_unstale":"2021-02-10T04:46:43.398554+0000","last_undegraded":"2021-02-10T04:46:43.398554+0000","last_fullsized":"2021-02-10T04:46:43.398554+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:17.537072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:17.537072+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"1029","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.802960+0000","last_change":"2021-02-10T04:45:47.541656+0000","last_active":"2021-02-10T04:46:37.802960+0000","last_peered":"2021-02-10T04:46:37.802960+0000","last_clean":"2021-02-10T04:46:37.802960+0000","last_became_active":"2021-02-10T04:45:47.541134+0000","last_became_peered":"2021-02-10T04:45:47.541134+0000","last_unstale":"2021-02-10T04:46:37.802960+0000","last_undegraded":"2021-02-10T04:46:37.802960+0000","last_fullsized":"2021-02-10T04:46:37.802960+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1045,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:34.667847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:44:34.667847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"1000'490","reported_seq":"3588","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.400891+0000","last_change":"2021-02-10T04:45:42.423029+0000","last_active":"2021-02-10T04:46:43.400891+0000","last_peered":"2021-02-10T04:46:43.400891+0000","last_clean":"2021-02-10T04:46:43.400891+0000","last_became_active":"2021-02-10T04:45:42.422424+0000","last_became_peered":"2021-02-10T04:45:42.422424+0000","last_unstale":"2021-02-10T04:46:43.400891+0000","last_undegraded":"2021-02-10T04:46:43.400891+0000","last_fullsized":"2021-02-10T04:46:43.400891+0000","mapping_epoch":1039,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1040,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'36","last_scrub_stamp":"2021-02-10T04:30:13.863430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:13.863430+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19371627,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":141,"num_read_kb":148568,"num_write":85,"num_write_kb":31732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":16890929,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"371","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.404056+0000","last_change":"2021-02-10T04:44:41.760153+0000","last_active":"2021-02-10T04:46:38.404056+0000","last_peered":"2021-02-10T04:46:38.404056+0000","last_clean":"2021-02-10T04:46:38.404056+0000","last_became_active":"2021-02-10T04:38:39.252266+0000","last_became_peered":"2021-02-10T04:38:39.252266+0000","last_unstale":"2021-02-10T04:46:38.404056+0000","last_undegraded":"2021-02-10T04:46:38.404056+0000","last_fullsized":"2021-02-10T04:46:38.404056+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:41.760055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:44:41.760055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"1049'37","reported_seq":"799","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.802444+0000","last_change":"2021-02-10T04:46:36.919290+0000","last_active":"2021-02-10T04:46:37.802444+0000","last_peered":"2021-02-10T04:46:37.802444+0000","last_clean":"2021-02-10T04:46:37.802444+0000","last_became_active":"2021-02-10T04:33:05.146695+0000","last_became_peered":"2021-02-10T04:33:05.146695+0000","last_unstale":"2021-02-10T04:46:37.802444+0000","last_undegraded":"2021-02-10T04:46:37.802444+0000","last_fullsized":"2021-02-10T04:46:37.802444+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:46:36.919205+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:46:36.919205+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"490","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.398479+0000","last_change":"2021-02-10T04:37:37.258992+0000","last_active":"2021-02-10T04:46:43.398479+0000","last_peered":"2021-02-10T04:46:43.398479+0000","last_clean":"2021-02-10T04:46:43.398479+0000","last_became_active":"2021-02-10T04:37:37.257600+0000","last_became_peered":"2021-02-10T04:37:37.257600+0000","last_unstale":"2021-02-10T04:46:43.398479+0000","last_undegraded":"2021-02-10T04:46:43.398479+0000","last_fullsized":"2021-02-10T04:46:43.398479+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:39.690436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:28:39.690436+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"1007","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:43.100118+0000","last_change":"2021-02-10T04:38:33.243623+0000","last_active":"2021-02-10T04:46:43.100118+0000","last_peered":"2021-02-10T04:46:43.100118+0000","last_clean":"2021-02-10T04:46:43.100118+0000","last_became_active":"2021-02-10T04:37:37.249591+0000","last_became_peered":"2021-02-10T04:37:37.249591+0000","last_unstale":"2021-02-10T04:46:43.100118+0000","last_undegraded":"2021-02-10T04:46:43.100118+0000","last_fullsized":"2021-02-10T04:46:43.100118+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:38:33.243519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:33.243519+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":552,"num_read_kb":547,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"957","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.403972+0000","last_change":"2021-02-10T04:45:47.536989+0000","last_active":"2021-02-10T04:46:38.403972+0000","last_peered":"2021-02-10T04:46:38.403972+0000","last_clean":"2021-02-10T04:46:38.403972+0000","last_became_active":"2021-02-10T04:45:47.536491+0000","last_became_peered":"2021-02-10T04:45:47.536491+0000","last_unstale":"2021-02-10T04:46:38.403972+0000","last_undegraded":"2021-02-10T04:46:38.403972+0000","last_fullsized":"2021-02-10T04:46:38.403972+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1045,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:44:45.683902+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.4","version":"1000'620","reported_seq":"4227","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.399278+0000","last_change":"2021-02-10T04:46:40.753673+0000","last_active":"2021-02-10T04:46:43.399278+0000","last_peered":"2021-02-10T04:46:43.399278+0000","last_clean":"2021-02-10T04:46:43.399278+0000","last_became_active":"2021-02-10T04:44:25.537632+0000","last_became_peered":"2021-02-10T04:44:25.537632+0000","last_unstale":"2021-02-10T04:46:43.399278+0000","last_undegraded":"2021-02-10T04:46:43.399278+0000","last_fullsized":"2021-02-10T04:46:43.399278+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:46:40.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:40.753581+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"1039","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.397646+0000","last_change":"2021-02-10T04:44:45.876889+0000","last_active":"2021-02-10T04:46:43.397646+0000","last_peered":"2021-02-10T04:46:43.397646+0000","last_clean":"2021-02-10T04:46:43.397646+0000","last_became_active":"2021-02-10T04:29:07.159607+0000","last_became_peered":"2021-02-10T04:29:07.159607+0000","last_unstale":"2021-02-10T04:46:43.397646+0000","last_undegraded":"2021-02-10T04:46:43.397646+0000","last_fullsized":"2021-02-10T04:46:43.397646+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:45.876793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:44:45.876793+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"1140","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.400717+0000","last_change":"2021-02-10T04:44:26.031868+0000","last_active":"2021-02-10T04:46:43.400717+0000","last_peered":"2021-02-10T04:46:43.400717+0000","last_clean":"2021-02-10T04:46:43.400717+0000","last_became_active":"2021-02-10T04:25:51.027624+0000","last_became_peered":"2021-02-10T04:25:51.027624+0000","last_unstale":"2021-02-10T04:46:43.400717+0000","last_undegraded":"2021-02-10T04:46:43.400717+0000","last_fullsized":"2021-02-10T04:46:43.400717+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:26.031771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:44:26.031771+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"1049'55","reported_seq":"758","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.401056+0000","last_change":"2021-02-10T04:37:38.636607+0000","last_active":"2021-02-10T04:46:43.401056+0000","last_peered":"2021-02-10T04:46:43.401056+0000","last_clean":"2021-02-10T04:46:43.401056+0000","last_became_active":"2021-02-10T04:37:37.244344+0000","last_became_peered":"2021-02-10T04:37:37.244344+0000","last_unstale":"2021-02-10T04:46:43.401056+0000","last_undegraded":"2021-02-10T04:46:43.401056+0000","last_fullsized":"2021-02-10T04:46:43.401056+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'31","last_scrub_stamp":"2021-02-10T04:33:49.423083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:49.423083+0000","log_size":55,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":99,"num_read_kb":75,"num_write":58,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"1040","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.804384+0000","last_change":"2021-02-10T04:38:34.289602+0000","last_active":"2021-02-10T04:46:37.804384+0000","last_peered":"2021-02-10T04:46:37.804384+0000","last_clean":"2021-02-10T04:46:37.804384+0000","last_became_active":"2021-02-10T04:28:21.125541+0000","last_became_peered":"2021-02-10T04:28:21.125541+0000","last_unstale":"2021-02-10T04:46:37.804384+0000","last_undegraded":"2021-02-10T04:46:37.804384+0000","last_fullsized":"2021-02-10T04:46:37.804384+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:34.289495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:34.289495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"1025","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.401708+0000","last_change":"2021-02-10T04:45:47.535908+0000","last_active":"2021-02-10T04:46:43.401708+0000","last_peered":"2021-02-10T04:46:43.401708+0000","last_clean":"2021-02-10T04:46:43.401708+0000","last_became_active":"2021-02-10T04:45:47.535555+0000","last_became_peered":"2021-02-10T04:45:47.535555+0000","last_unstale":"2021-02-10T04:46:43.401708+0000","last_undegraded":"2021-02-10T04:46:43.401708+0000","last_fullsized":"2021-02-10T04:46:43.401708+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1045,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.395943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.395943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"1002'2042","reported_seq":"9252","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.404954+0000","last_change":"2021-02-10T04:44:25.552653+0000","last_active":"2021-02-10T04:46:38.404954+0000","last_peered":"2021-02-10T04:46:38.404954+0000","last_clean":"2021-02-10T04:46:38.404954+0000","last_became_active":"2021-02-10T04:44:25.549177+0000","last_became_peered":"2021-02-10T04:44:25.549177+0000","last_unstale":"2021-02-10T04:46:38.404954+0000","last_undegraded":"2021-02-10T04:46:38.404954+0000","last_fullsized":"2021-02-10T04:46:38.404954+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'223","last_scrub_stamp":"2021-02-10T04:30:14.478639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:14.478639+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46300076,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":545,"num_read_kb":531689,"num_write":399,"num_write_kb":153087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"972","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.801391+0000","last_change":"2021-02-10T04:46:25.979309+0000","last_active":"2021-02-10T04:46:37.801391+0000","last_peered":"2021-02-10T04:46:37.801391+0000","last_clean":"2021-02-10T04:46:37.801391+0000","last_became_active":"2021-02-10T04:37:37.219422+0000","last_became_peered":"2021-02-10T04:37:37.219422+0000","last_unstale":"2021-02-10T04:46:37.801391+0000","last_undegraded":"2021-02-10T04:46:37.801391+0000","last_fullsized":"2021-02-10T04:46:37.801391+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:25.979216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:46:25.979216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"58'2","reported_seq":"1282","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.398382+0000","last_change":"2021-02-10T04:46:27.748328+0000","last_active":"2021-02-10T04:46:43.398382+0000","last_peered":"2021-02-10T04:46:43.398382+0000","last_clean":"2021-02-10T04:46:43.398382+0000","last_became_active":"2021-02-10T04:25:51.026251+0000","last_became_peered":"2021-02-10T04:25:51.026251+0000","last_unstale":"2021-02-10T04:46:43.398382+0000","last_undegraded":"2021-02-10T04:46:43.398382+0000","last_fullsized":"2021-02-10T04:46:43.398382+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:46:27.748223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:46:27.748223+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"1049'54","reported_seq":"810","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.804642+0000","last_change":"2021-02-10T04:46:11.118546+0000","last_active":"2021-02-10T04:46:37.804642+0000","last_peered":"2021-02-10T04:46:37.804642+0000","last_clean":"2021-02-10T04:46:37.804642+0000","last_became_active":"2021-02-10T04:33:05.141901+0000","last_became_peered":"2021-02-10T04:33:05.141901+0000","last_unstale":"2021-02-10T04:46:37.804642+0000","last_undegraded":"2021-02-10T04:46:37.804642+0000","last_fullsized":"2021-02-10T04:46:37.804642+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:46:11.118437+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:46:11.118437+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":75,"num_read_kb":57,"num_write":42,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"789","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.399159+0000","last_change":"2021-02-10T04:46:22.735101+0000","last_active":"2021-02-10T04:46:43.399159+0000","last_peered":"2021-02-10T04:46:43.399159+0000","last_clean":"2021-02-10T04:46:43.399159+0000","last_became_active":"2021-02-10T04:37:37.247405+0000","last_became_peered":"2021-02-10T04:37:37.247405+0000","last_unstale":"2021-02-10T04:46:43.399159+0000","last_undegraded":"2021-02-10T04:46:43.399159+0000","last_fullsized":"2021-02-10T04:46:43.399159+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:22.735012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:22.735012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"1040","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.396655+0000","last_change":"2021-02-10T04:45:47.535960+0000","last_active":"2021-02-10T04:46:43.396655+0000","last_peered":"2021-02-10T04:46:43.396655+0000","last_clean":"2021-02-10T04:46:43.396655+0000","last_became_active":"2021-02-10T04:45:47.535481+0000","last_became_peered":"2021-02-10T04:45:47.535481+0000","last_unstale":"2021-02-10T04:46:43.396655+0000","last_undegraded":"2021-02-10T04:46:43.396655+0000","last_fullsized":"2021-02-10T04:46:43.396655+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1045,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:26.124805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:38:26.124805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"293'77","reported_seq":"2811","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.802882+0000","last_change":"2021-02-10T04:44:25.642817+0000","last_active":"2021-02-10T04:46:37.802882+0000","last_peered":"2021-02-10T04:46:37.802882+0000","last_clean":"2021-02-10T04:46:37.802882+0000","last_became_active":"2021-02-10T04:44:25.542375+0000","last_became_peered":"2021-02-10T04:44:25.542375+0000","last_unstale":"2021-02-10T04:46:37.802882+0000","last_undegraded":"2021-02-10T04:46:37.802882+0000","last_fullsized":"2021-02-10T04:46:37.802882+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3773171,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":36,"num_read_kb":37174,"num_write":18,"num_write_kb":6631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"1045","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.404889+0000","last_change":"2021-02-10T04:44:38.740220+0000","last_active":"2021-02-10T04:46:38.404889+0000","last_peered":"2021-02-10T04:46:38.404889+0000","last_clean":"2021-02-10T04:46:38.404889+0000","last_became_active":"2021-02-10T04:28:49.053127+0000","last_became_peered":"2021-02-10T04:28:49.053127+0000","last_unstale":"2021-02-10T04:46:38.404889+0000","last_undegraded":"2021-02-10T04:46:38.404889+0000","last_fullsized":"2021-02-10T04:46:38.404889+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:38.740156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:44:38.740156+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"445","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.397199+0000","last_change":"2021-02-10T04:46:08.770167+0000","last_active":"2021-02-10T04:46:43.397199+0000","last_peered":"2021-02-10T04:46:43.397199+0000","last_clean":"2021-02-10T04:46:43.397199+0000","last_became_active":"2021-02-10T04:38:39.246766+0000","last_became_peered":"2021-02-10T04:38:39.246766+0000","last_unstale":"2021-02-10T04:46:43.397199+0000","last_undegraded":"2021-02-10T04:46:43.397199+0000","last_fullsized":"2021-02-10T04:46:43.397199+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:08.770076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:46:08.770076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"1048'39","reported_seq":"1141","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.803264+0000","last_change":"2021-02-10T04:44:41.973978+0000","last_active":"2021-02-10T04:46:37.803264+0000","last_peered":"2021-02-10T04:46:37.803264+0000","last_clean":"2021-02-10T04:46:37.803264+0000","last_became_active":"2021-02-10T04:39:02.560739+0000","last_became_peered":"2021-02-10T04:39:02.560739+0000","last_unstale":"2021-02-10T04:46:37.803264+0000","last_undegraded":"2021-02-10T04:46:37.803264+0000","last_fullsized":"2021-02-10T04:46:37.803264+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'33","last_scrub_stamp":"2021-02-10T04:44:41.973895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:44:41.973895+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":76,"num_read_kb":59,"num_write":41,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"1149","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.805779+0000","last_change":"2021-02-10T04:37:37.351533+0000","last_active":"2021-02-10T04:46:37.805779+0000","last_peered":"2021-02-10T04:46:37.805779+0000","last_clean":"2021-02-10T04:46:37.805779+0000","last_became_active":"2021-02-10T04:37:37.238701+0000","last_became_peered":"2021-02-10T04:37:37.238701+0000","last_unstale":"2021-02-10T04:46:37.805779+0000","last_undegraded":"2021-02-10T04:46:37.805779+0000","last_fullsized":"2021-02-10T04:46:37.805779+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:33:49.760989+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:33:49.760989+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"1021","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.400135+0000","last_change":"2021-02-10T04:46:15.536753+0000","last_active":"2021-02-10T04:46:43.400135+0000","last_peered":"2021-02-10T04:46:43.400135+0000","last_clean":"2021-02-10T04:46:43.400135+0000","last_became_active":"2021-02-10T04:25:53.047736+0000","last_became_peered":"2021-02-10T04:25:53.047736+0000","last_unstale":"2021-02-10T04:46:43.400135+0000","last_undegraded":"2021-02-10T04:46:43.400135+0000","last_fullsized":"2021-02-10T04:46:43.400135+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:15.536661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:15.536661+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"1001'1528","reported_seq":"5711","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.398150+0000","last_change":"2021-02-10T04:44:26.666123+0000","last_active":"2021-02-10T04:46:43.398150+0000","last_peered":"2021-02-10T04:46:43.398150+0000","last_clean":"2021-02-10T04:46:43.398150+0000","last_became_active":"2021-02-10T04:37:48.215401+0000","last_became_peered":"2021-02-10T04:37:48.215401+0000","last_unstale":"2021-02-10T04:46:43.398150+0000","last_undegraded":"2021-02-10T04:46:43.398150+0000","last_fullsized":"2021-02-10T04:46:43.398150+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:44:26.665941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:44:26.665941+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":93888272,"num_objects":33,"num_object_clones":30,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":1,"num_read":716,"num_read_kb":703869,"num_write":572,"num_write_kb":222710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":42623785,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.15","version":"1000'446","reported_seq":"1922","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.805728+0000","last_change":"2021-02-10T04:45:43.499811+0000","last_active":"2021-02-10T04:46:37.805728+0000","last_peered":"2021-02-10T04:46:37.805728+0000","last_clean":"2021-02-10T04:46:37.805728+0000","last_became_active":"2021-02-10T04:45:43.499003+0000","last_became_peered":"2021-02-10T04:45:43.499003+0000","last_unstale":"2021-02-10T04:46:37.805728+0000","last_undegraded":"2021-02-10T04:46:37.805728+0000","last_fullsized":"2021-02-10T04:46:37.805728+0000","mapping_epoch":1040,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1041,"parent":"0.0","parent_split_bits":5,"last_scrub":"683'135","last_scrub_stamp":"2021-02-10T04:38:23.183462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:38:23.183462+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22255852,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":80,"num_read_kb":68436,"num_write":73,"num_write_kb":24345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":19387101,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"1048'40","reported_seq":"1104","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.400048+0000","last_change":"2021-02-10T04:44:50.832167+0000","last_active":"2021-02-10T04:46:43.400048+0000","last_peered":"2021-02-10T04:46:43.400048+0000","last_clean":"2021-02-10T04:46:43.400048+0000","last_became_active":"2021-02-10T04:25:49.008942+0000","last_became_peered":"2021-02-10T04:25:49.008942+0000","last_unstale":"2021-02-10T04:46:43.400048+0000","last_undegraded":"2021-02-10T04:46:43.400048+0000","last_fullsized":"2021-02-10T04:46:43.400048+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'34","last_scrub_stamp":"2021-02-10T04:44:50.832088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:44:50.832088+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":80,"num_read_kb":58,"num_write":41,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"454","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.396909+0000","last_change":"2021-02-10T04:46:28.715466+0000","last_active":"2021-02-10T04:46:43.396909+0000","last_peered":"2021-02-10T04:46:43.396909+0000","last_clean":"2021-02-10T04:46:43.396909+0000","last_became_active":"2021-02-10T04:37:37.234649+0000","last_became_peered":"2021-02-10T04:37:37.234649+0000","last_unstale":"2021-02-10T04:46:43.396909+0000","last_undegraded":"2021-02-10T04:46:43.396909+0000","last_fullsized":"2021-02-10T04:46:43.396909+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:46:28.715369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:28.715369+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":247,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"1023","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.401498+0000","last_change":"2021-02-10T04:44:39.599352+0000","last_active":"2021-02-10T04:46:43.401498+0000","last_peered":"2021-02-10T04:46:43.401498+0000","last_clean":"2021-02-10T04:46:43.401498+0000","last_became_active":"2021-02-10T04:25:51.030651+0000","last_became_peered":"2021-02-10T04:25:51.030651+0000","last_unstale":"2021-02-10T04:46:43.401498+0000","last_undegraded":"2021-02-10T04:46:43.401498+0000","last_fullsized":"2021-02-10T04:46:43.401498+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:39.599260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:44:39.599260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"433","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.804916+0000","last_change":"2021-02-10T04:46:27.889189+0000","last_active":"2021-02-10T04:46:37.804916+0000","last_peered":"2021-02-10T04:46:37.804916+0000","last_clean":"2021-02-10T04:46:37.804916+0000","last_became_active":"2021-02-10T04:37:37.238049+0000","last_became_peered":"2021-02-10T04:37:37.238049+0000","last_unstale":"2021-02-10T04:46:37.804916+0000","last_undegraded":"2021-02-10T04:46:37.804916+0000","last_fullsized":"2021-02-10T04:46:37.804916+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:27.889132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:27.889132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"1000'620","reported_seq":"4231","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.400445+0000","last_change":"2021-02-10T04:44:30.704509+0000","last_active":"2021-02-10T04:46:43.400445+0000","last_peered":"2021-02-10T04:46:43.400445+0000","last_clean":"2021-02-10T04:46:43.400445+0000","last_became_active":"2021-02-10T04:44:30.702669+0000","last_became_peered":"2021-02-10T04:44:30.702669+0000","last_unstale":"2021-02-10T04:46:43.400445+0000","last_undegraded":"2021-02-10T04:46:43.400445+0000","last_fullsized":"2021-02-10T04:46:43.400445+0000","mapping_epoch":1012,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1013,"parent":"0.0","parent_split_bits":5,"last_scrub":"400'239","last_scrub_stamp":"2021-02-10T04:33:46.270478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:46.270478+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14265195,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":144,"num_read_kb":128483,"num_write":129,"num_write_kb":53444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"1049'25","reported_seq":"1111","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.401427+0000","last_change":"2021-02-10T04:38:26.845289+0000","last_active":"2021-02-10T04:46:43.401427+0000","last_peered":"2021-02-10T04:46:43.401427+0000","last_clean":"2021-02-10T04:46:43.401427+0000","last_became_active":"2021-02-10T04:37:37.247396+0000","last_became_peered":"2021-02-10T04:37:37.247396+0000","last_unstale":"2021-02-10T04:46:43.401427+0000","last_undegraded":"2021-02-10T04:46:43.401427+0000","last_fullsized":"2021-02-10T04:46:43.401427+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'19","last_scrub_stamp":"2021-02-10T04:38:26.845195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:26.845195+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"1034","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.403231+0000","last_change":"2021-02-10T04:38:36.295733+0000","last_active":"2021-02-10T04:46:38.403231+0000","last_peered":"2021-02-10T04:46:38.403231+0000","last_clean":"2021-02-10T04:46:38.403231+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:46:38.403231+0000","last_undegraded":"2021-02-10T04:46:38.403231+0000","last_fullsized":"2021-02-10T04:46:38.403231+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:38:36.295599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:36.295599+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"1022","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.803648+0000","last_change":"2021-02-10T04:46:18.092627+0000","last_active":"2021-02-10T04:46:37.803648+0000","last_peered":"2021-02-10T04:46:37.803648+0000","last_clean":"2021-02-10T04:46:37.803648+0000","last_became_active":"2021-02-10T04:25:51.031864+0000","last_became_peered":"2021-02-10T04:25:51.031864+0000","last_unstale":"2021-02-10T04:46:37.803648+0000","last_undegraded":"2021-02-10T04:46:37.803648+0000","last_fullsized":"2021-02-10T04:46:37.803648+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:18.092535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:46:18.092535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"948","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.404005+0000","last_change":"2021-02-10T04:44:36.704043+0000","last_active":"2021-02-10T04:46:38.404005+0000","last_peered":"2021-02-10T04:46:38.404005+0000","last_clean":"2021-02-10T04:46:38.404005+0000","last_became_active":"2021-02-10T04:29:07.147249+0000","last_became_peered":"2021-02-10T04:29:07.147249+0000","last_unstale":"2021-02-10T04:46:38.404005+0000","last_undegraded":"2021-02-10T04:46:38.404005+0000","last_fullsized":"2021-02-10T04:46:38.404005+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:36.703969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:44:36.703969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.17","version":"1002'2042","reported_seq":"9210","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.802007+0000","last_change":"2021-02-10T04:44:30.658263+0000","last_active":"2021-02-10T04:46:37.802007+0000","last_peered":"2021-02-10T04:46:37.802007+0000","last_clean":"2021-02-10T04:46:37.802007+0000","last_became_active":"2021-02-10T04:44:30.657433+0000","last_became_peered":"2021-02-10T04:44:30.657433+0000","last_unstale":"2021-02-10T04:46:37.802007+0000","last_undegraded":"2021-02-10T04:46:37.802007+0000","last_fullsized":"2021-02-10T04:46:37.802007+0000","mapping_epoch":1012,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1013,"parent":"0.0","parent_split_bits":5,"last_scrub":"198'223","last_scrub_stamp":"2021-02-10T04:30:14.478639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:14.478639+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46300076,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":545,"num_read_kb":531689,"num_write":399,"num_write_kb":153087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":5403183,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"1049'46","reported_seq":"1241","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.400865+0000","last_change":"2021-02-10T04:37:38.541093+0000","last_active":"2021-02-10T04:46:43.400865+0000","last_peered":"2021-02-10T04:46:43.400865+0000","last_clean":"2021-02-10T04:46:43.400865+0000","last_became_active":"2021-02-10T04:37:37.244921+0000","last_became_peered":"2021-02-10T04:37:37.244921+0000","last_unstale":"2021-02-10T04:46:43.400865+0000","last_undegraded":"2021-02-10T04:46:43.400865+0000","last_fullsized":"2021-02-10T04:46:43.400865+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'26","last_scrub_stamp":"2021-02-10T04:33:46.785905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:33:46.785905+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":81,"num_read_kb":66,"num_write":46,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"84'7","reported_seq":"1046","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.804305+0000","last_change":"2021-02-10T04:30:12.538670+0000","last_active":"2021-02-10T04:46:37.804305+0000","last_peered":"2021-02-10T04:46:37.804305+0000","last_clean":"2021-02-10T04:46:37.804305+0000","last_became_active":"2021-02-10T04:25:36.334366+0000","last_became_peered":"2021-02-10T04:25:36.334366+0000","last_unstale":"2021-02-10T04:46:37.804305+0000","last_undegraded":"2021-02-10T04:46:37.804305+0000","last_fullsized":"2021-02-10T04:46:37.804305+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'7","last_scrub_stamp":"2021-02-10T04:30:12.538549+0000","last_deep_scrub":"84'7","last_deep_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_clean_scrub_stamp":"2021-02-10T04:30:12.538549+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"58'2","reported_seq":"1275","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.399489+0000","last_change":"2021-02-10T04:46:09.107097+0000","last_active":"2021-02-10T04:46:38.399489+0000","last_peered":"2021-02-10T04:46:38.399489+0000","last_clean":"2021-02-10T04:46:38.399489+0000","last_became_active":"2021-02-10T04:25:51.032328+0000","last_became_peered":"2021-02-10T04:25:51.032328+0000","last_unstale":"2021-02-10T04:46:38.399489+0000","last_undegraded":"2021-02-10T04:46:38.399489+0000","last_fullsized":"2021-02-10T04:46:38.399489+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:46:09.107007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:46:09.107007+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"1027","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.399803+0000","last_change":"2021-02-10T04:46:33.677552+0000","last_active":"2021-02-10T04:46:43.399803+0000","last_peered":"2021-02-10T04:46:43.399803+0000","last_clean":"2021-02-10T04:46:43.399803+0000","last_became_active":"2021-02-10T04:25:53.045177+0000","last_became_peered":"2021-02-10T04:25:53.045177+0000","last_unstale":"2021-02-10T04:46:43.399803+0000","last_undegraded":"2021-02-10T04:46:43.399803+0000","last_fullsized":"2021-02-10T04:46:43.399803+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:33.677438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_clean_scrub_stamp":"2021-02-10T04:46:33.677438+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"1000'490","reported_seq":"3552","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.806089+0000","last_change":"2021-02-10T04:44:30.703533+0000","last_active":"2021-02-10T04:46:37.806089+0000","last_peered":"2021-02-10T04:46:37.806089+0000","last_clean":"2021-02-10T04:46:37.806089+0000","last_became_active":"2021-02-10T04:44:30.702606+0000","last_became_peered":"2021-02-10T04:44:30.702606+0000","last_unstale":"2021-02-10T04:46:37.806089+0000","last_undegraded":"2021-02-10T04:46:37.806089+0000","last_fullsized":"2021-02-10T04:46:37.806089+0000","mapping_epoch":1012,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1013,"parent":"0.0","parent_split_bits":5,"last_scrub":"197'36","last_scrub_stamp":"2021-02-10T04:30:13.863430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:13.863430+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19371627,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":142,"num_read_kb":148569,"num_write":86,"num_write_kb":31733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":11113379,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"1049'31","reported_seq":"1081","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.802923+0000","last_change":"2021-02-10T04:44:44.580263+0000","last_active":"2021-02-10T04:46:37.802923+0000","last_peered":"2021-02-10T04:46:37.802923+0000","last_clean":"2021-02-10T04:46:37.802923+0000","last_became_active":"2021-02-10T04:25:49.010890+0000","last_became_peered":"2021-02-10T04:25:49.010890+0000","last_unstale":"2021-02-10T04:46:37.802923+0000","last_undegraded":"2021-02-10T04:46:37.802923+0000","last_fullsized":"2021-02-10T04:46:37.802923+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'25","last_scrub_stamp":"2021-02-10T04:44:44.580182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:44:44.580182+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":59,"num_read_kb":44,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"1038","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.398214+0000","last_change":"2021-02-10T04:44:41.866217+0000","last_active":"2021-02-10T04:46:43.398214+0000","last_peered":"2021-02-10T04:46:43.398214+0000","last_clean":"2021-02-10T04:46:43.398214+0000","last_became_active":"2021-02-10T04:25:34.321096+0000","last_became_peered":"2021-02-10T04:25:34.321096+0000","last_unstale":"2021-02-10T04:46:43.398214+0000","last_undegraded":"2021-02-10T04:46:43.398214+0000","last_fullsized":"2021-02-10T04:46:43.398214+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:41.866132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:44:41.866132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"639'3","reported_seq":"1289","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.400988+0000","last_change":"2021-02-10T04:46:39.502981+0000","last_active":"2021-02-10T04:46:43.400988+0000","last_peered":"2021-02-10T04:46:43.400988+0000","last_clean":"2021-02-10T04:46:43.400988+0000","last_became_active":"2021-02-10T04:37:37.245428+0000","last_became_peered":"2021-02-10T04:37:37.245428+0000","last_unstale":"2021-02-10T04:46:43.400988+0000","last_undegraded":"2021-02-10T04:46:43.400988+0000","last_fullsized":"2021-02-10T04:46:43.400988+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"639'3","last_scrub_stamp":"2021-02-10T04:46:39.502896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:46:39.502896+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"1028","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.805996+0000","last_change":"2021-02-10T04:46:23.812423+0000","last_active":"2021-02-10T04:46:37.805996+0000","last_peered":"2021-02-10T04:46:37.805996+0000","last_clean":"2021-02-10T04:46:37.805996+0000","last_became_active":"2021-02-10T04:37:37.238360+0000","last_became_peered":"2021-02-10T04:37:37.238360+0000","last_unstale":"2021-02-10T04:46:37.805996+0000","last_undegraded":"2021-02-10T04:46:37.805996+0000","last_fullsized":"2021-02-10T04:46:37.805996+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:23.812328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:23.812328+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"1032'89","reported_seq":"802","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.403328+0000","last_change":"2021-02-10T04:45:33.153432+0000","last_active":"2021-02-10T04:46:38.403328+0000","last_peered":"2021-02-10T04:46:38.403328+0000","last_clean":"2021-02-10T04:46:38.403328+0000","last_became_active":"2021-02-10T04:44:27.540200+0000","last_became_peered":"2021-02-10T04:44:27.540200+0000","last_unstale":"2021-02-10T04:46:38.403328+0000","last_undegraded":"2021-02-10T04:46:38.403328+0000","last_fullsized":"2021-02-10T04:46:38.403328+0000","mapping_epoch":1009,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1010,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":89,"ondisk_log_size":89,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5218574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":30326,"num_write":21,"num_write_kb":12444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3210022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"1048'27","reported_seq":"796","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.404639+0000","last_change":"2021-02-10T04:44:55.705392+0000","last_active":"2021-02-10T04:46:38.404639+0000","last_peered":"2021-02-10T04:46:38.404639+0000","last_clean":"2021-02-10T04:46:38.404639+0000","last_became_active":"2021-02-10T04:37:37.249007+0000","last_became_peered":"2021-02-10T04:37:37.249007+0000","last_unstale":"2021-02-10T04:46:38.404639+0000","last_undegraded":"2021-02-10T04:46:38.404639+0000","last_fullsized":"2021-02-10T04:46:38.404639+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'21","last_scrub_stamp":"2021-02-10T04:44:55.705293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:44:55.705293+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":34,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"1037","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:42.217606+0000","last_change":"2021-02-10T04:46:42.217536+0000","last_active":"2021-02-10T04:46:42.217606+0000","last_peered":"2021-02-10T04:46:42.217606+0000","last_clean":"2021-02-10T04:46:42.217606+0000","last_became_active":"2021-02-10T04:25:34.330070+0000","last_became_peered":"2021-02-10T04:25:34.330070+0000","last_unstale":"2021-02-10T04:46:42.217606+0000","last_undegraded":"2021-02-10T04:46:42.217606+0000","last_fullsized":"2021-02-10T04:46:42.217606+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:42.217450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:42.217450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"1023","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.399138+0000","last_change":"2021-02-10T04:44:45.525061+0000","last_active":"2021-02-10T04:46:43.399138+0000","last_peered":"2021-02-10T04:46:43.399138+0000","last_clean":"2021-02-10T04:46:43.399138+0000","last_became_active":"2021-02-10T04:25:51.029155+0000","last_became_peered":"2021-02-10T04:25:51.029155+0000","last_unstale":"2021-02-10T04:46:43.399138+0000","last_undegraded":"2021-02-10T04:46:43.399138+0000","last_fullsized":"2021-02-10T04:46:43.399138+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:45.524973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:44:45.524973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"1027","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.396495+0000","last_change":"2021-02-10T04:46:21.768707+0000","last_active":"2021-02-10T04:46:43.396495+0000","last_peered":"2021-02-10T04:46:43.396495+0000","last_clean":"2021-02-10T04:46:43.396495+0000","last_became_active":"2021-02-10T04:25:53.039322+0000","last_became_peered":"2021-02-10T04:25:53.039322+0000","last_unstale":"2021-02-10T04:46:43.396495+0000","last_undegraded":"2021-02-10T04:46:43.396495+0000","last_fullsized":"2021-02-10T04:46:43.396495+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:21.768615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:21.768615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"1000'1071","reported_seq":"4558","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.397282+0000","last_change":"2021-02-10T04:45:42.428929+0000","last_active":"2021-02-10T04:46:43.397282+0000","last_peered":"2021-02-10T04:46:43.397282+0000","last_clean":"2021-02-10T04:46:43.397282+0000","last_became_active":"2021-02-10T04:45:42.428565+0000","last_became_peered":"2021-02-10T04:45:42.428565+0000","last_unstale":"2021-02-10T04:46:43.397282+0000","last_undegraded":"2021-02-10T04:46:43.397282+0000","last_fullsized":"2021-02-10T04:46:43.397282+0000","mapping_epoch":1039,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1040,"parent":"0.0","parent_split_bits":5,"last_scrub":"412'357","last_scrub_stamp":"2021-02-10T04:33:50.582002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:50.582002+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23286368,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":242,"num_read_kb":226835,"num_write":185,"num_write_kb":76005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"1049'31","reported_seq":"1069","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.804450+0000","last_change":"2021-02-10T04:38:32.221752+0000","last_active":"2021-02-10T04:46:37.804450+0000","last_peered":"2021-02-10T04:46:37.804450+0000","last_clean":"2021-02-10T04:46:37.804450+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:46:37.804450+0000","last_undegraded":"2021-02-10T04:46:37.804450+0000","last_fullsized":"2021-02-10T04:46:37.804450+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'23","last_scrub_stamp":"2021-02-10T04:38:32.221656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:32.221656+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":41,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"716","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.803013+0000","last_change":"2021-02-10T04:38:36.309455+0000","last_active":"2021-02-10T04:46:37.803013+0000","last_peered":"2021-02-10T04:46:37.803013+0000","last_clean":"2021-02-10T04:46:37.803013+0000","last_became_active":"2021-02-10T04:33:05.147956+0000","last_became_peered":"2021-02-10T04:33:05.147956+0000","last_unstale":"2021-02-10T04:46:37.803013+0000","last_undegraded":"2021-02-10T04:46:37.803013+0000","last_fullsized":"2021-02-10T04:46:37.803013+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:38:36.309362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:36.309362+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"705","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.804560+0000","last_change":"2021-02-10T04:46:22.990584+0000","last_active":"2021-02-10T04:46:37.804560+0000","last_peered":"2021-02-10T04:46:37.804560+0000","last_clean":"2021-02-10T04:46:37.804560+0000","last_became_active":"2021-02-10T04:33:05.150529+0000","last_became_peered":"2021-02-10T04:33:05.150529+0000","last_unstale":"2021-02-10T04:46:37.804560+0000","last_undegraded":"2021-02-10T04:46:37.804560+0000","last_fullsized":"2021-02-10T04:46:37.804560+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:22.990486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:46:22.990486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"1021","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.398274+0000","last_change":"2021-02-10T04:46:09.628000+0000","last_active":"2021-02-10T04:46:43.398274+0000","last_peered":"2021-02-10T04:46:43.398274+0000","last_clean":"2021-02-10T04:46:43.398274+0000","last_became_active":"2021-02-10T04:25:53.043277+0000","last_became_peered":"2021-02-10T04:25:53.043277+0000","last_unstale":"2021-02-10T04:46:43.398274+0000","last_undegraded":"2021-02-10T04:46:43.398274+0000","last_fullsized":"2021-02-10T04:46:43.398274+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:09.627905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:09.627905+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"1002'1657","reported_seq":"4971","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.397888+0000","last_change":"2021-02-10T04:44:35.602106+0000","last_active":"2021-02-10T04:46:43.397888+0000","last_peered":"2021-02-10T04:46:43.397888+0000","last_clean":"2021-02-10T04:46:43.397888+0000","last_became_active":"2021-02-10T04:44:35.600743+0000","last_became_peered":"2021-02-10T04:44:35.600743+0000","last_unstale":"2021-02-10T04:46:43.397888+0000","last_undegraded":"2021-02-10T04:46:43.397888+0000","last_fullsized":"2021-02-10T04:46:43.397888+0000","mapping_epoch":1015,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1016,"parent":"0.0","parent_split_bits":5,"last_scrub":"687'1095","last_scrub_stamp":"2021-02-10T04:38:27.895743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:38:27.895743+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":42793067,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":424,"num_read_kb":437674,"num_write":319,"num_write_kb":118798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":17936938,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"1048'20","reported_seq":"1185","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.397204+0000","last_change":"2021-02-10T04:38:22.919477+0000","last_active":"2021-02-10T04:46:43.397204+0000","last_peered":"2021-02-10T04:46:43.397204+0000","last_clean":"2021-02-10T04:46:43.397204+0000","last_became_active":"2021-02-10T04:26:39.005616+0000","last_became_peered":"2021-02-10T04:26:39.005616+0000","last_unstale":"2021-02-10T04:46:43.397204+0000","last_undegraded":"2021-02-10T04:46:43.397204+0000","last_fullsized":"2021-02-10T04:46:43.397204+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'16","last_scrub_stamp":"2021-02-10T04:38:22.919377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:22.919377+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":42,"num_read_kb":24,"num_write":20,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"1040","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.803866+0000","last_change":"2021-02-10T04:44:50.573117+0000","last_active":"2021-02-10T04:46:37.803866+0000","last_peered":"2021-02-10T04:46:37.803866+0000","last_clean":"2021-02-10T04:46:37.803866+0000","last_became_active":"2021-02-10T04:37:37.229852+0000","last_became_peered":"2021-02-10T04:37:37.229852+0000","last_unstale":"2021-02-10T04:46:37.803866+0000","last_undegraded":"2021-02-10T04:46:37.803866+0000","last_fullsized":"2021-02-10T04:46:37.803866+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:50.573048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:44:50.573048+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"1023","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.403152+0000","last_change":"2021-02-10T04:44:47.730663+0000","last_active":"2021-02-10T04:46:38.403152+0000","last_peered":"2021-02-10T04:46:38.403152+0000","last_clean":"2021-02-10T04:46:38.403152+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:46:38.403152+0000","last_undegraded":"2021-02-10T04:46:38.403152+0000","last_fullsized":"2021-02-10T04:46:38.403152+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:47.730571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:44:47.730571+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"436","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.803550+0000","last_change":"2021-02-10T04:46:37.748357+0000","last_active":"2021-02-10T04:46:37.803550+0000","last_peered":"2021-02-10T04:46:37.803550+0000","last_clean":"2021-02-10T04:46:37.803550+0000","last_became_active":"2021-02-10T04:37:37.244082+0000","last_became_peered":"2021-02-10T04:37:37.244082+0000","last_unstale":"2021-02-10T04:46:37.803550+0000","last_undegraded":"2021-02-10T04:46:37.803550+0000","last_fullsized":"2021-02-10T04:46:37.803550+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:37.748261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:46:37.748261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.12","version":"998'1009","reported_seq":"4709","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.403014+0000","last_change":"2021-02-10T04:45:45.518262+0000","last_active":"2021-02-10T04:46:38.403014+0000","last_peered":"2021-02-10T04:46:38.403014+0000","last_clean":"2021-02-10T04:46:38.403014+0000","last_became_active":"2021-02-10T04:45:45.516982+0000","last_became_peered":"2021-02-10T04:45:45.516982+0000","last_unstale":"2021-02-10T04:46:38.403014+0000","last_undegraded":"2021-02-10T04:46:38.403014+0000","last_fullsized":"2021-02-10T04:46:38.403014+0000","mapping_epoch":1042,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1043,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24710414,"num_objects":11,"num_object_clones":10,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":239,"num_read_kb":249403,"num_write":155,"num_write_kb":50104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":81817654,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"1049'37","reported_seq":"1090","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.398701+0000","last_change":"2021-02-10T04:44:44.573935+0000","last_active":"2021-02-10T04:46:43.398701+0000","last_peered":"2021-02-10T04:46:43.398701+0000","last_clean":"2021-02-10T04:46:43.398701+0000","last_became_active":"2021-02-10T04:25:49.007049+0000","last_became_peered":"2021-02-10T04:25:49.007049+0000","last_unstale":"2021-02-10T04:46:43.398701+0000","last_undegraded":"2021-02-10T04:46:43.398701+0000","last_fullsized":"2021-02-10T04:46:43.398701+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'29","last_scrub_stamp":"2021-02-10T04:44:44.573821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:44:44.573821+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":65,"num_read_kb":51,"num_write":37,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"1172","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.405107+0000","last_change":"2021-02-10T04:38:31.166652+0000","last_active":"2021-02-10T04:46:38.405107+0000","last_peered":"2021-02-10T04:46:38.405107+0000","last_clean":"2021-02-10T04:46:38.405107+0000","last_became_active":"2021-02-10T04:37:37.228840+0000","last_became_peered":"2021-02-10T04:37:37.228840+0000","last_unstale":"2021-02-10T04:46:38.405107+0000","last_undegraded":"2021-02-10T04:46:38.405107+0000","last_fullsized":"2021-02-10T04:46:38.405107+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:38:31.166586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:31.166586+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"372","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.397432+0000","last_change":"2021-02-10T04:44:54.669664+0000","last_active":"2021-02-10T04:46:43.397432+0000","last_peered":"2021-02-10T04:46:43.397432+0000","last_clean":"2021-02-10T04:46:43.397432+0000","last_became_active":"2021-02-10T04:38:39.246046+0000","last_became_peered":"2021-02-10T04:38:39.246046+0000","last_unstale":"2021-02-10T04:46:43.397432+0000","last_undegraded":"2021-02-10T04:46:43.397432+0000","last_fullsized":"2021-02-10T04:46:43.397432+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:54.669597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:44:54.669597+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"1016","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.400290+0000","last_change":"2021-02-10T04:33:50.809533+0000","last_active":"2021-02-10T04:46:43.400290+0000","last_peered":"2021-02-10T04:46:43.400290+0000","last_clean":"2021-02-10T04:46:43.400290+0000","last_became_active":"2021-02-10T04:25:53.150845+0000","last_became_peered":"2021-02-10T04:25:53.150845+0000","last_unstale":"2021-02-10T04:46:43.400290+0000","last_undegraded":"2021-02-10T04:46:43.400290+0000","last_fullsized":"2021-02-10T04:46:43.400290+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:50.809468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_clean_scrub_stamp":"2021-02-10T04:33:50.809468+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"1049'42","reported_seq":"1103","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.400336+0000","last_change":"2021-02-10T04:46:13.849861+0000","last_active":"2021-02-10T04:46:43.400336+0000","last_peered":"2021-02-10T04:46:43.400336+0000","last_clean":"2021-02-10T04:46:43.400336+0000","last_became_active":"2021-02-10T04:25:49.009177+0000","last_became_peered":"2021-02-10T04:25:49.009177+0000","last_unstale":"2021-02-10T04:46:43.400336+0000","last_undegraded":"2021-02-10T04:46:43.400336+0000","last_fullsized":"2021-02-10T04:46:43.400336+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'42","last_scrub_stamp":"2021-02-10T04:46:13.849755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:13.849755+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":75,"num_read_kb":57,"num_write":42,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"1040","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.800752+0000","last_change":"2021-02-10T04:44:38.636641+0000","last_active":"2021-02-10T04:46:37.800752+0000","last_peered":"2021-02-10T04:46:37.800752+0000","last_clean":"2021-02-10T04:46:37.800752+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:46:37.800752+0000","last_undegraded":"2021-02-10T04:46:37.800752+0000","last_fullsized":"2021-02-10T04:46:37.800752+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:38.636558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:44:38.636558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"58'2","reported_seq":"1276","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:42.934544+0000","last_change":"2021-02-10T04:44:50.716644+0000","last_active":"2021-02-10T04:46:42.934544+0000","last_peered":"2021-02-10T04:46:42.934544+0000","last_clean":"2021-02-10T04:46:42.934544+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:46:42.934544+0000","last_undegraded":"2021-02-10T04:46:42.934544+0000","last_fullsized":"2021-02-10T04:46:42.934544+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:44:50.716551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:44:50.716551+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"965","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.806463+0000","last_change":"2021-02-10T04:46:34.914405+0000","last_active":"2021-02-10T04:46:37.806463+0000","last_peered":"2021-02-10T04:46:37.806463+0000","last_clean":"2021-02-10T04:46:37.806463+0000","last_became_active":"2021-02-10T04:38:39.251336+0000","last_became_peered":"2021-02-10T04:38:39.251336+0000","last_unstale":"2021-02-10T04:46:37.806463+0000","last_undegraded":"2021-02-10T04:46:37.806463+0000","last_fullsized":"2021-02-10T04:46:37.806463+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:34.914288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:34.914288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"1049'51","reported_seq":"1112","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.405100+0000","last_change":"2021-02-10T04:44:56.738861+0000","last_active":"2021-02-10T04:46:38.405100+0000","last_peered":"2021-02-10T04:46:38.405100+0000","last_clean":"2021-02-10T04:46:38.405100+0000","last_became_active":"2021-02-10T04:25:49.015760+0000","last_became_peered":"2021-02-10T04:25:49.015760+0000","last_unstale":"2021-02-10T04:46:38.405100+0000","last_undegraded":"2021-02-10T04:46:38.405100+0000","last_fullsized":"2021-02-10T04:46:38.405100+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'43","last_scrub_stamp":"2021-02-10T04:44:56.738770+0000","last_deep_scrub":"76'35","last_deep_scrub_stamp":"2021-02-10T04:26:58.198641+0000","last_clean_scrub_stamp":"2021-02-10T04:44:56.738770+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":74,"num_read_kb":54,"num_write":40,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"1032","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.398400+0000","last_change":"2021-02-10T04:46:42.537588+0000","last_active":"2021-02-10T04:46:43.398400+0000","last_peered":"2021-02-10T04:46:43.398400+0000","last_clean":"2021-02-10T04:46:43.398400+0000","last_became_active":"2021-02-10T04:25:34.324981+0000","last_became_peered":"2021-02-10T04:25:34.324981+0000","last_unstale":"2021-02-10T04:46:43.398400+0000","last_undegraded":"2021-02-10T04:46:43.398400+0000","last_fullsized":"2021-02-10T04:46:43.398400+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:42.537443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:42.537443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"1033","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.400397+0000","last_change":"2021-02-10T04:44:37.827187+0000","last_active":"2021-02-10T04:46:43.400397+0000","last_peered":"2021-02-10T04:46:43.400397+0000","last_clean":"2021-02-10T04:46:43.400397+0000","last_became_active":"2021-02-10T04:37:37.246998+0000","last_became_peered":"2021-02-10T04:37:37.246998+0000","last_unstale":"2021-02-10T04:46:43.400397+0000","last_undegraded":"2021-02-10T04:46:43.400397+0000","last_fullsized":"2021-02-10T04:46:43.400397+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:37.827101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:44:37.827101+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"705","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.803340+0000","last_change":"2021-02-10T04:46:20.029857+0000","last_active":"2021-02-10T04:46:37.803340+0000","last_peered":"2021-02-10T04:46:37.803340+0000","last_clean":"2021-02-10T04:46:37.803340+0000","last_became_active":"2021-02-10T04:33:05.141911+0000","last_became_peered":"2021-02-10T04:33:05.141911+0000","last_unstale":"2021-02-10T04:46:37.803340+0000","last_undegraded":"2021-02-10T04:46:37.803340+0000","last_fullsized":"2021-02-10T04:46:37.803340+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:20.029748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:20.029748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"1049'39","reported_seq":"1011","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.400900+0000","last_change":"2021-02-10T04:44:46.552488+0000","last_active":"2021-02-10T04:46:43.400900+0000","last_peered":"2021-02-10T04:46:43.400900+0000","last_clean":"2021-02-10T04:46:43.400900+0000","last_became_active":"2021-02-10T04:37:37.257293+0000","last_became_peered":"2021-02-10T04:37:37.257293+0000","last_unstale":"2021-02-10T04:46:43.400900+0000","last_undegraded":"2021-02-10T04:46:43.400900+0000","last_fullsized":"2021-02-10T04:46:43.400900+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'27","last_scrub_stamp":"2021-02-10T04:44:46.552383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:44:46.552383+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"1035","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.403826+0000","last_change":"2021-02-10T04:38:32.193334+0000","last_active":"2021-02-10T04:46:38.403826+0000","last_peered":"2021-02-10T04:46:38.403826+0000","last_clean":"2021-02-10T04:46:38.403826+0000","last_became_active":"2021-02-10T04:25:36.336442+0000","last_became_peered":"2021-02-10T04:25:36.336442+0000","last_unstale":"2021-02-10T04:46:38.403826+0000","last_undegraded":"2021-02-10T04:46:38.403826+0000","last_fullsized":"2021-02-10T04:46:38.403826+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:38:32.193240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:32.193240+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"704","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.805502+0000","last_change":"2021-02-10T04:46:25.888326+0000","last_active":"2021-02-10T04:46:37.805502+0000","last_peered":"2021-02-10T04:46:37.805502+0000","last_clean":"2021-02-10T04:46:37.805502+0000","last_became_active":"2021-02-10T04:33:05.146838+0000","last_became_peered":"2021-02-10T04:33:05.146838+0000","last_unstale":"2021-02-10T04:46:37.805502+0000","last_undegraded":"2021-02-10T04:46:37.805502+0000","last_fullsized":"2021-02-10T04:46:37.805502+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:25.888229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:46:25.888229+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"1019","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.806294+0000","last_change":"2021-02-10T04:46:36.790576+0000","last_active":"2021-02-10T04:46:37.806294+0000","last_peered":"2021-02-10T04:46:37.806294+0000","last_clean":"2021-02-10T04:46:37.806294+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:46:37.806294+0000","last_undegraded":"2021-02-10T04:46:37.806294+0000","last_fullsized":"2021-02-10T04:46:37.806294+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:36.790487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:36.790487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"1049'34","reported_seq":"1084","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.401091+0000","last_change":"2021-02-10T04:38:25.860322+0000","last_active":"2021-02-10T04:46:43.401091+0000","last_peered":"2021-02-10T04:46:43.401091+0000","last_clean":"2021-02-10T04:46:43.401091+0000","last_became_active":"2021-02-10T04:25:49.008997+0000","last_became_peered":"2021-02-10T04:25:49.008997+0000","last_unstale":"2021-02-10T04:46:43.401091+0000","last_undegraded":"2021-02-10T04:46:43.401091+0000","last_fullsized":"2021-02-10T04:46:43.401091+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'28","last_scrub_stamp":"2021-02-10T04:38:25.860247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:25.860247+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":67,"num_read_kb":49,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"1044","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.398935+0000","last_change":"2021-02-10T04:44:36.642074+0000","last_active":"2021-02-10T04:46:43.398935+0000","last_peered":"2021-02-10T04:46:43.398935+0000","last_clean":"2021-02-10T04:46:43.398935+0000","last_became_active":"2021-02-10T04:25:36.336893+0000","last_became_peered":"2021-02-10T04:25:36.336893+0000","last_unstale":"2021-02-10T04:46:43.398935+0000","last_undegraded":"2021-02-10T04:46:43.398935+0000","last_fullsized":"2021-02-10T04:46:43.398935+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:36.641977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:26:46.425288+0000","last_clean_scrub_stamp":"2021-02-10T04:44:36.641977+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"1025","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.804991+0000","last_change":"2021-02-10T04:37:37.243679+0000","last_active":"2021-02-10T04:46:37.804991+0000","last_peered":"2021-02-10T04:46:37.804991+0000","last_clean":"2021-02-10T04:46:37.804991+0000","last_became_active":"2021-02-10T04:37:37.243275+0000","last_became_peered":"2021-02-10T04:37:37.243275+0000","last_unstale":"2021-02-10T04:46:37.804991+0000","last_undegraded":"2021-02-10T04:46:37.804991+0000","last_fullsized":"2021-02-10T04:46:37.804991+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:52.643099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:33:52.643099+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"1021","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.401394+0000","last_change":"2021-02-10T04:46:26.577039+0000","last_active":"2021-02-10T04:46:43.401394+0000","last_peered":"2021-02-10T04:46:43.401394+0000","last_clean":"2021-02-10T04:46:43.401394+0000","last_became_active":"2021-02-10T04:25:53.147831+0000","last_became_peered":"2021-02-10T04:25:53.147831+0000","last_unstale":"2021-02-10T04:46:43.401394+0000","last_undegraded":"2021-02-10T04:46:43.401394+0000","last_fullsized":"2021-02-10T04:46:43.401394+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:26.576960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:26.576960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"1003'739","reported_seq":"3726","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.402486+0000","last_change":"2021-02-10T04:44:30.581309+0000","last_active":"2021-02-10T04:46:38.402486+0000","last_peered":"2021-02-10T04:46:38.402486+0000","last_clean":"2021-02-10T04:46:38.402486+0000","last_became_active":"2021-02-10T04:44:30.579847+0000","last_became_peered":"2021-02-10T04:44:30.579847+0000","last_unstale":"2021-02-10T04:46:38.402486+0000","last_undegraded":"2021-02-10T04:46:38.402486+0000","last_fullsized":"2021-02-10T04:46:38.402486+0000","mapping_epoch":1012,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1013,"parent":"0.0","parent_split_bits":5,"last_scrub":"154'63","last_scrub_stamp":"2021-02-10T04:29:39.700115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:29:39.700115+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24059889,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":130,"num_read_kb":109544,"num_write":102,"num_write_kb":46174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":18429397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"1048'37","reported_seq":"1097","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.803690+0000","last_change":"2021-02-10T04:44:38.877308+0000","last_active":"2021-02-10T04:46:37.803690+0000","last_peered":"2021-02-10T04:46:37.803690+0000","last_clean":"2021-02-10T04:46:37.803690+0000","last_became_active":"2021-02-10T04:25:49.000911+0000","last_became_peered":"2021-02-10T04:25:49.000911+0000","last_unstale":"2021-02-10T04:46:37.803690+0000","last_undegraded":"2021-02-10T04:46:37.803690+0000","last_fullsized":"2021-02-10T04:46:37.803690+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'31","last_scrub_stamp":"2021-02-10T04:44:38.877241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:44:38.877241+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":71,"num_read_kb":55,"num_write":38,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"951","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.806743+0000","last_change":"2021-02-10T04:37:37.242121+0000","last_active":"2021-02-10T04:46:37.806743+0000","last_peered":"2021-02-10T04:46:37.806743+0000","last_clean":"2021-02-10T04:46:37.806743+0000","last_became_active":"2021-02-10T04:37:37.241751+0000","last_became_peered":"2021-02-10T04:37:37.241751+0000","last_unstale":"2021-02-10T04:46:37.806743+0000","last_undegraded":"2021-02-10T04:46:37.806743+0000","last_fullsized":"2021-02-10T04:46:37.806743+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:33:48.646802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:33:48.646802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"1031","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.802345+0000","last_change":"2021-02-10T04:44:51.575527+0000","last_active":"2021-02-10T04:46:37.802345+0000","last_peered":"2021-02-10T04:46:37.802345+0000","last_clean":"2021-02-10T04:46:37.802345+0000","last_became_active":"2021-02-10T04:38:39.253122+0000","last_became_peered":"2021-02-10T04:38:39.253122+0000","last_unstale":"2021-02-10T04:46:37.802345+0000","last_undegraded":"2021-02-10T04:46:37.802345+0000","last_fullsized":"2021-02-10T04:46:37.802345+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:51.575462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:44:51.575462+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"1027","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.397123+0000","last_change":"2021-02-10T04:46:24.697867+0000","last_active":"2021-02-10T04:46:43.397123+0000","last_peered":"2021-02-10T04:46:43.397123+0000","last_clean":"2021-02-10T04:46:43.397123+0000","last_became_active":"2021-02-10T04:25:53.039844+0000","last_became_peered":"2021-02-10T04:25:53.039844+0000","last_unstale":"2021-02-10T04:46:43.397123+0000","last_undegraded":"2021-02-10T04:46:43.397123+0000","last_fullsized":"2021-02-10T04:46:43.397123+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:24.697786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:24.697786+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"259'74","reported_seq":"60","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.396399+0000","last_change":"2021-02-10T04:44:27.547465+0000","last_active":"2021-02-10T04:46:43.396399+0000","last_peered":"2021-02-10T04:46:43.396399+0000","last_clean":"2021-02-10T04:46:43.396399+0000","last_became_active":"2021-02-10T04:44:27.546590+0000","last_became_peered":"2021-02-10T04:44:27.546590+0000","last_unstale":"2021-02-10T04:46:43.396399+0000","last_undegraded":"2021-02-10T04:46:43.396399+0000","last_fullsized":"2021-02-10T04:46:43.396399+0000","mapping_epoch":1009,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1010,"parent":"0.0","parent_split_bits":5,"last_scrub":"191'62","last_scrub_stamp":"2021-02-10T04:30:10.905905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:10.905905+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1218524,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":28228,"num_write":19,"num_write_kb":7022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"1048'16","reported_seq":"1051","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.804286+0000","last_change":"2021-02-10T04:30:13.089576+0000","last_active":"2021-02-10T04:46:37.804286+0000","last_peered":"2021-02-10T04:46:37.804286+0000","last_clean":"2021-02-10T04:46:37.804286+0000","last_became_active":"2021-02-10T04:25:49.013196+0000","last_became_peered":"2021-02-10T04:25:49.013196+0000","last_unstale":"2021-02-10T04:46:37.804286+0000","last_undegraded":"2021-02-10T04:46:37.804286+0000","last_fullsized":"2021-02-10T04:46:37.804286+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2021-02-10T04:30:13.089494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:30:13.089494+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":17,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"948","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.403831+0000","last_change":"2021-02-10T04:46:10.064529+0000","last_active":"2021-02-10T04:46:38.403831+0000","last_peered":"2021-02-10T04:46:38.403831+0000","last_clean":"2021-02-10T04:46:38.403831+0000","last_became_active":"2021-02-10T04:29:07.157944+0000","last_became_peered":"2021-02-10T04:29:07.157944+0000","last_unstale":"2021-02-10T04:46:38.403831+0000","last_undegraded":"2021-02-10T04:46:38.403831+0000","last_fullsized":"2021-02-10T04:46:38.403831+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:10.064468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:46:10.064468+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"941","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.803849+0000","last_change":"2021-02-10T04:46:15.814062+0000","last_active":"2021-02-10T04:46:37.803849+0000","last_peered":"2021-02-10T04:46:37.803849+0000","last_clean":"2021-02-10T04:46:37.803849+0000","last_became_active":"2021-02-10T04:29:07.146525+0000","last_became_peered":"2021-02-10T04:29:07.146525+0000","last_unstale":"2021-02-10T04:46:37.803849+0000","last_undegraded":"2021-02-10T04:46:37.803849+0000","last_fullsized":"2021-02-10T04:46:37.803849+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:15.813941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:46:15.813941+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"1020","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.801537+0000","last_change":"2021-02-10T04:46:15.009831+0000","last_active":"2021-02-10T04:46:37.801537+0000","last_peered":"2021-02-10T04:46:37.801537+0000","last_clean":"2021-02-10T04:46:37.801537+0000","last_became_active":"2021-02-10T04:25:53.042601+0000","last_became_peered":"2021-02-10T04:25:53.042601+0000","last_unstale":"2021-02-10T04:46:37.801537+0000","last_undegraded":"2021-02-10T04:46:37.801537+0000","last_fullsized":"2021-02-10T04:46:37.801537+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:15.009742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:15.009742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"1048'44","reported_seq":"1095","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.802403+0000","last_change":"2021-02-10T04:46:08.846352+0000","last_active":"2021-02-10T04:46:37.802403+0000","last_peered":"2021-02-10T04:46:37.802403+0000","last_clean":"2021-02-10T04:46:37.802403+0000","last_became_active":"2021-02-10T04:25:49.010481+0000","last_became_peered":"2021-02-10T04:25:49.010481+0000","last_unstale":"2021-02-10T04:46:37.802403+0000","last_undegraded":"2021-02-10T04:46:37.802403+0000","last_fullsized":"2021-02-10T04:46:37.802403+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'44","last_scrub_stamp":"2021-02-10T04:46:08.846219+0000","last_deep_scrub":"78'32","last_deep_scrub_stamp":"2021-02-10T04:27:09.170428+0000","last_clean_scrub_stamp":"2021-02-10T04:46:08.846219+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":65,"num_read_kb":46,"num_write":34,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"1032","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.401627+0000","last_change":"2021-02-10T04:46:33.543681+0000","last_active":"2021-02-10T04:46:43.401627+0000","last_peered":"2021-02-10T04:46:43.401627+0000","last_clean":"2021-02-10T04:46:43.401627+0000","last_became_active":"2021-02-10T04:25:34.327162+0000","last_became_peered":"2021-02-10T04:25:34.327162+0000","last_unstale":"2021-02-10T04:46:43.401627+0000","last_undegraded":"2021-02-10T04:46:43.401627+0000","last_fullsized":"2021-02-10T04:46:43.401627+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:33.543559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:46:33.543559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"705","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.801309+0000","last_change":"2021-02-10T04:44:45.577005+0000","last_active":"2021-02-10T04:46:37.801309+0000","last_peered":"2021-02-10T04:46:37.801309+0000","last_clean":"2021-02-10T04:46:37.801309+0000","last_became_active":"2021-02-10T04:33:05.169159+0000","last_became_peered":"2021-02-10T04:33:05.169159+0000","last_unstale":"2021-02-10T04:46:37.801309+0000","last_undegraded":"2021-02-10T04:46:37.801309+0000","last_fullsized":"2021-02-10T04:46:37.801309+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:45.576804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:44:45.576804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"1100","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.803637+0000","last_change":"2021-02-10T04:38:23.319098+0000","last_active":"2021-02-10T04:46:37.803637+0000","last_peered":"2021-02-10T04:46:37.803637+0000","last_clean":"2021-02-10T04:46:37.803637+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:46:37.803637+0000","last_undegraded":"2021-02-10T04:46:37.803637+0000","last_fullsized":"2021-02-10T04:46:37.803637+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:23.319023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:38:23.319023+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"1049'68","reported_seq":"1194","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.398790+0000","last_change":"2021-02-10T04:38:21.961539+0000","last_active":"2021-02-10T04:46:43.398790+0000","last_peered":"2021-02-10T04:46:43.398790+0000","last_clean":"2021-02-10T04:46:43.398790+0000","last_became_active":"2021-02-10T04:37:37.266384+0000","last_became_peered":"2021-02-10T04:37:37.266384+0000","last_unstale":"2021-02-10T04:46:43.398790+0000","last_undegraded":"2021-02-10T04:46:43.398790+0000","last_fullsized":"2021-02-10T04:46:43.398790+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":641,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'56","last_scrub_stamp":"2021-02-10T04:38:21.961470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:38:21.961470+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":125,"num_read_kb":102,"num_write":69,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":890,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"1042","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.401187+0000","last_change":"2021-02-10T04:44:39.845478+0000","last_active":"2021-02-10T04:46:43.401187+0000","last_peered":"2021-02-10T04:46:43.401187+0000","last_clean":"2021-02-10T04:46:43.401187+0000","last_became_active":"2021-02-10T04:25:36.336112+0000","last_became_peered":"2021-02-10T04:25:36.336112+0000","last_unstale":"2021-02-10T04:46:43.401187+0000","last_undegraded":"2021-02-10T04:46:43.401187+0000","last_fullsized":"2021-02-10T04:46:43.401187+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:39.845409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:44:39.845409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"367","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.396949+0000","last_change":"2021-02-10T04:38:39.244331+0000","last_active":"2021-02-10T04:46:43.396949+0000","last_peered":"2021-02-10T04:46:43.396949+0000","last_clean":"2021-02-10T04:46:43.396949+0000","last_became_active":"2021-02-10T04:38:39.243965+0000","last_became_peered":"2021-02-10T04:38:39.243965+0000","last_unstale":"2021-02-10T04:46:43.396949+0000","last_undegraded":"2021-02-10T04:46:43.396949+0000","last_fullsized":"2021-02-10T04:46:43.396949+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:33.199521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:38:33.199521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"1021","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.398858+0000","last_change":"2021-02-10T04:46:14.549225+0000","last_active":"2021-02-10T04:46:43.398858+0000","last_peered":"2021-02-10T04:46:43.398858+0000","last_clean":"2021-02-10T04:46:43.398858+0000","last_became_active":"2021-02-10T04:25:53.150917+0000","last_became_peered":"2021-02-10T04:25:53.150917+0000","last_unstale":"2021-02-10T04:46:43.398858+0000","last_undegraded":"2021-02-10T04:46:43.398858+0000","last_fullsized":"2021-02-10T04:46:43.398858+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:14.549136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:14.549136+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":"717","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.805377+0000","last_change":"2021-02-10T04:45:49.551648+0000","last_active":"2021-02-10T04:46:37.805377+0000","last_peered":"2021-02-10T04:46:37.805377+0000","last_clean":"2021-02-10T04:46:37.805377+0000","last_became_active":"2021-02-10T04:45:49.551288+0000","last_became_peered":"2021-02-10T04:45:49.551288+0000","last_unstale":"2021-02-10T04:46:37.805377+0000","last_undegraded":"2021-02-10T04:46:37.805377+0000","last_fullsized":"2021-02-10T04:46:37.805377+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:23.098681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:23.098681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.20","version":"0'0","reported_seq":"1005","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.800932+0000","last_change":"2021-02-10T04:30:11.492843+0000","last_active":"2021-02-10T04:46:37.800932+0000","last_peered":"2021-02-10T04:46:37.800932+0000","last_clean":"2021-02-10T04:46:37.800932+0000","last_became_active":"2021-02-10T04:29:07.148937+0000","last_became_peered":"2021-02-10T04:29:07.148937+0000","last_unstale":"2021-02-10T04:46:37.800932+0000","last_undegraded":"2021-02-10T04:46:37.800932+0000","last_fullsized":"2021-02-10T04:46:37.800932+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:30:11.492788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:41.375276+0000","last_clean_scrub_stamp":"2021-02-10T04:30:11.492788+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":"965","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.403646+0000","last_change":"2021-02-10T04:45:49.552398+0000","last_active":"2021-02-10T04:46:38.403646+0000","last_peered":"2021-02-10T04:46:38.403646+0000","last_clean":"2021-02-10T04:46:38.403646+0000","last_became_active":"2021-02-10T04:45:49.551858+0000","last_became_peered":"2021-02-10T04:45:49.551858+0000","last_unstale":"2021-02-10T04:46:38.403646+0000","last_undegraded":"2021-02-10T04:46:38.403646+0000","last_fullsized":"2021-02-10T04:46:38.403646+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:44:45.683902+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":"525","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.398594+0000","last_change":"2021-02-10T04:46:23.691562+0000","last_active":"2021-02-10T04:46:43.398594+0000","last_peered":"2021-02-10T04:46:43.398594+0000","last_clean":"2021-02-10T04:46:43.398594+0000","last_became_active":"2021-02-10T04:37:37.246865+0000","last_became_peered":"2021-02-10T04:37:37.246865+0000","last_unstale":"2021-02-10T04:46:43.398594+0000","last_undegraded":"2021-02-10T04:46:43.398594+0000","last_fullsized":"2021-02-10T04:46:43.398594+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:23.691476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:23.691476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":"19","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.803430+0000","last_change":"2021-02-10T04:45:49.552753+0000","last_active":"2021-02-10T04:46:37.803430+0000","last_peered":"2021-02-10T04:46:37.803430+0000","last_clean":"2021-02-10T04:46:37.803430+0000","last_became_active":"2021-02-10T04:45:49.552420+0000","last_became_peered":"2021-02-10T04:45:49.552420+0000","last_unstale":"2021-02-10T04:46:37.803430+0000","last_undegraded":"2021-02-10T04:46:37.803430+0000","last_fullsized":"2021-02-10T04:46:37.803430+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.395943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.395943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":"1046","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.403226+0000","last_change":"2021-02-10T04:46:32.031164+0000","last_active":"2021-02-10T04:46:38.403226+0000","last_peered":"2021-02-10T04:46:38.403226+0000","last_clean":"2021-02-10T04:46:38.403226+0000","last_became_active":"2021-02-10T04:28:23.129832+0000","last_became_peered":"2021-02-10T04:28:23.129832+0000","last_unstale":"2021-02-10T04:46:38.403226+0000","last_undegraded":"2021-02-10T04:46:38.403226+0000","last_fullsized":"2021-02-10T04:46:38.403226+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:32.031098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:32.031098+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":"20","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.404855+0000","last_change":"2021-02-10T04:45:49.551540+0000","last_active":"2021-02-10T04:46:38.404855+0000","last_peered":"2021-02-10T04:46:38.404855+0000","last_clean":"2021-02-10T04:46:38.404855+0000","last_became_active":"2021-02-10T04:45:49.551121+0000","last_became_peered":"2021-02-10T04:45:49.551121+0000","last_unstale":"2021-02-10T04:46:38.404855+0000","last_undegraded":"2021-02-10T04:46:38.404855+0000","last_fullsized":"2021-02-10T04:46:38.404855+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:34.667847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:44:34.667847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":"986","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.398751+0000","last_change":"2021-02-10T04:44:51.620170+0000","last_active":"2021-02-10T04:46:43.398751+0000","last_peered":"2021-02-10T04:46:43.398751+0000","last_clean":"2021-02-10T04:46:43.398751+0000","last_became_active":"2021-02-10T04:28:23.129523+0000","last_became_peered":"2021-02-10T04:28:23.129523+0000","last_unstale":"2021-02-10T04:46:43.398751+0000","last_undegraded":"2021-02-10T04:46:43.398751+0000","last_fullsized":"2021-02-10T04:46:43.398751+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:51.620102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:44:51.620102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":"20","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.402575+0000","last_change":"2021-02-10T04:45:49.759610+0000","last_active":"2021-02-10T04:46:38.402575+0000","last_peered":"2021-02-10T04:46:38.402575+0000","last_clean":"2021-02-10T04:46:38.402575+0000","last_became_active":"2021-02-10T04:45:49.759179+0000","last_became_peered":"2021-02-10T04:45:49.759179+0000","last_unstale":"2021-02-10T04:46:38.402575+0000","last_undegraded":"2021-02-10T04:46:38.402575+0000","last_fullsized":"2021-02-10T04:46:38.402575+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:26.124805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:38:26.124805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":"990","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.403332+0000","last_change":"2021-02-10T04:46:37.153814+0000","last_active":"2021-02-10T04:46:38.403332+0000","last_peered":"2021-02-10T04:46:38.403332+0000","last_clean":"2021-02-10T04:46:38.403332+0000","last_became_active":"2021-02-10T04:28:23.126368+0000","last_became_peered":"2021-02-10T04:28:23.126368+0000","last_unstale":"2021-02-10T04:46:38.403332+0000","last_undegraded":"2021-02-10T04:46:38.403332+0000","last_fullsized":"2021-02-10T04:46:38.403332+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:37.153685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:37.153685+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":"19","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.803088+0000","last_change":"2021-02-10T04:45:49.759600+0000","last_active":"2021-02-10T04:46:37.803088+0000","last_peered":"2021-02-10T04:46:37.803088+0000","last_clean":"2021-02-10T04:46:37.803088+0000","last_became_active":"2021-02-10T04:45:49.759224+0000","last_became_peered":"2021-02-10T04:45:49.759224+0000","last_unstale":"2021-02-10T04:46:37.803088+0000","last_undegraded":"2021-02-10T04:46:37.803088+0000","last_fullsized":"2021-02-10T04:46:37.803088+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:40.414601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:38:40.414601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.25","version":"53'2","reported_seq":"1001","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.805186+0000","last_change":"2021-02-10T04:44:39.917303+0000","last_active":"2021-02-10T04:46:37.805186+0000","last_peered":"2021-02-10T04:46:37.805186+0000","last_clean":"2021-02-10T04:46:37.805186+0000","last_became_active":"2021-02-10T04:28:23.467770+0000","last_became_peered":"2021-02-10T04:28:23.467770+0000","last_unstale":"2021-02-10T04:46:37.805186+0000","last_undegraded":"2021-02-10T04:46:37.805186+0000","last_fullsized":"2021-02-10T04:46:37.805186+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:44:39.917235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:44:39.917235+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":"1038","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.404542+0000","last_change":"2021-02-10T04:45:49.553481+0000","last_active":"2021-02-10T04:46:38.404542+0000","last_peered":"2021-02-10T04:46:38.404542+0000","last_clean":"2021-02-10T04:46:38.404542+0000","last_became_active":"2021-02-10T04:45:49.553084+0000","last_became_peered":"2021-02-10T04:45:49.553084+0000","last_unstale":"2021-02-10T04:46:38.404542+0000","last_undegraded":"2021-02-10T04:46:38.404542+0000","last_fullsized":"2021-02-10T04:46:38.404542+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:27.926662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:44:27.926662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.26","version":"53'1","reported_seq":"441","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.805566+0000","last_change":"2021-02-10T04:37:37.233712+0000","last_active":"2021-02-10T04:46:37.805566+0000","last_peered":"2021-02-10T04:46:37.805566+0000","last_clean":"2021-02-10T04:46:37.805566+0000","last_became_active":"2021-02-10T04:37:37.231081+0000","last_became_peered":"2021-02-10T04:37:37.231081+0000","last_unstale":"2021-02-10T04:46:37.805566+0000","last_undegraded":"2021-02-10T04:46:37.805566+0000","last_fullsized":"2021-02-10T04:46:37.805566+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:29:22.801929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:29:22.801929+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":423,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":"974","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.404142+0000","last_change":"2021-02-10T04:45:49.552654+0000","last_active":"2021-02-10T04:46:38.404142+0000","last_peered":"2021-02-10T04:46:38.404142+0000","last_clean":"2021-02-10T04:46:38.404142+0000","last_became_active":"2021-02-10T04:45:49.552026+0000","last_became_peered":"2021-02-10T04:45:49.552026+0000","last_unstale":"2021-02-10T04:46:38.404142+0000","last_undegraded":"2021-02-10T04:46:38.404142+0000","last_fullsized":"2021-02-10T04:46:38.404142+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:52.779754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:44:52.779754+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":"1012","reported_epoch":"1055","state":"active+clean","last_fresh":"2021-02-10T04:46:37.805560+0000","last_change":"2021-02-10T04:38:37.219069+0000","last_active":"2021-02-10T04:46:37.805560+0000","last_peered":"2021-02-10T04:46:37.805560+0000","last_clean":"2021-02-10T04:46:37.805560+0000","last_became_active":"2021-02-10T04:29:07.145907+0000","last_became_peered":"2021-02-10T04:29:07.145907+0000","last_unstale":"2021-02-10T04:46:37.805560+0000","last_undegraded":"2021-02-10T04:46:37.805560+0000","last_fullsized":"2021-02-10T04:46:37.805560+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:37.218968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:38:37.218968+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":"986","reported_epoch":"1057","state":"active+clean","last_fresh":"2021-02-10T04:46:43.398352+0000","last_change":"2021-02-10T04:46:12.555737+0000","last_active":"2021-02-10T04:46:43.398352+0000","last_peered":"2021-02-10T04:46:43.398352+0000","last_clean":"2021-02-10T04:46:43.398352+0000","last_became_active":"2021-02-10T04:28:23.129469+0000","last_became_peered":"2021-02-10T04:28:23.129469+0000","last_unstale":"2021-02-10T04:46:43.398352+0000","last_undegraded":"2021-02-10T04:46:43.398352+0000","last_fullsized":"2021-02-10T04:46:43.398352+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:12.555666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:46:12.555666+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":"985","reported_epoch":"1056","state":"active+clean","last_fresh":"2021-02-10T04:46:38.402894+0000","last_change":"2021-02-10T04:38:22.136377+0000","last_active":"2021-02-10T04:46:38.402894+0000","last_peered":"2021-02-10T04:46:38.402894+0000","last_clean":"2021-02-10T04:46:38.402894+0000","last_became_active":"2021-02-10T04:28:23.126795+0000","last_became_peered":"2021-02-10T04:28:23.126795+0000","last_unstale":"2021-02-10T04:46:38.402894+0000","last_undegraded":"2021-02-10T04:46:38.402894+0000","last_fullsized":"2021-02-10T04:46:38.402894+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:38:22.136290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_clean_scrub_stamp":"2021-02-10T04:38:22.136290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":885313522,"num_objects":365,"num_object_clones":316,"num_object_copies":1095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":365,"num_whiteouts":11,"num_read":7236,"num_read_kb":7389805,"num_write":5246,"num_write_kb":2015748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":564,"num_bytes_recovered":255526606,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1931636736,"data_stored":2559497134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23656,"ondisk_log_size":23656,"up":78,"acting":78,"num_store_stats":8},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":2368,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":18,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":6565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":126,"acting":126,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":552,"num_read_kb":547,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":2305,"num_read_kb":1778,"num_write":1306,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":8770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":75722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1354,"ondisk_log_size":1354,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21,"ondisk_log_size":21,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":126,"acting":126,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365056,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":226828,"kb_used_data":192964,"kb_used_omap":0,"kb_used_meta":33856,"kb_avail":93518324,"statfs":{"total":95995035648,"available":95762763776,"internally_reserved":0,"allocated":197595136,"data_stored":268825858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":34668544},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:46:12 2021","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.878,"15min":1.427},"min":{"1min":0.57299999999999995,"5min":0.501,"15min":0.495},"max":{"1min":2.1150000000000002,"5min":2.169,"15min":72.805999999999997},"last":0.97999999999999998},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.1000000000000001,"15min":1.6579999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.52000000000000002},"max":{"1min":2.839,"5min":5.1420000000000003,"15min":72.721999999999994},"last":0.72799999999999998}]},{"osd":1,"last update":"Wed Feb 10 04:46:12 2021","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":1.0249999999999999,"15min":1.014},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.48399999999999999},"max":{"1min":2.0030000000000001,"5min":5.1349999999999998,"15min":10.140000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":1.024,"5min":0.95999999999999996,"15min":0.97699999999999998},"min":{"1min":0.623,"5min":0.60599999999999998,"15min":0.56499999999999995},"max":{"1min":1.712,"5min":1.712,"15min":7.1520000000000001},"last":0.65700000000000003}]},{"osd":2,"last update":"Wed Feb 10 04:46:12 2021","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.036,"15min":1.587},"min":{"1min":0.73999999999999999,"5min":0.628,"15min":0.58299999999999996},"max":{"1min":1.6779999999999999,"5min":1.772,"15min":66.534000000000006},"last":0.78600000000000003},{"interface":"front","average":{"1min":1.087,"5min":1.2250000000000001,"15min":1.825},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.59099999999999997},"max":{"1min":2.198,"5min":4.9900000000000002,"15min":66.781999999999996},"last":0.92900000000000005}]},{"osd":3,"last update":"Wed Feb 10 04:46:12 2021","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.329,"15min":1.802},"min":{"1min":0.73599999999999999,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":2.3279999999999998,"5min":5.2599999999999998,"15min":68.305000000000007},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.216,"15min":1.7350000000000001},"min":{"1min":0.83399999999999996,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":2.2879999999999998,"5min":2.2879999999999998,"15min":69.331999999999994},"last":1.2430000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:46:12 2021","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":1.478,"15min":1.734},"min":{"1min":0.85399999999999998,"5min":0.70699999999999996,"15min":0.64600000000000002},"max":{"1min":2.9460000000000002,"5min":7.7859999999999996,"15min":97.733999999999995},"last":1.032},{"interface":"front","average":{"1min":1.302,"5min":1.3680000000000001,"15min":1.6819999999999999},"min":{"1min":0.84599999999999997,"5min":0.73399999999999999,"15min":0.65100000000000002},"max":{"1min":2.2370000000000001,"5min":6.4400000000000004,"15min":97.691000000000003},"last":1.2130000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:46:12 2021","interfaces":[{"interface":"back","average":{"1min":1.51,"5min":1.407,"15min":1.6359999999999999},"min":{"1min":0.88600000000000001,"5min":0.76000000000000001,"15min":0.66700000000000004},"max":{"1min":2.5670000000000002,"5min":3.387,"15min":35.283000000000001},"last":1.117},{"interface":"front","average":{"1min":1.635,"5min":1.704,"15min":1.9430000000000001},"min":{"1min":0.96299999999999997,"5min":0.86099999999999999,"15min":0.753},"max":{"1min":2.9870000000000001,"5min":7.0570000000000004,"15min":35.353999999999999},"last":1.27}]},{"osd":6,"last update":"Wed Feb 10 04:46:12 2021","interfaces":[{"interface":"back","average":{"1min":1.7829999999999999,"5min":1.9079999999999999,"15min":2.0110000000000001},"min":{"1min":1.0109999999999999,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":5.0359999999999996,"5min":7.79,"15min":32.616},"last":1.141},{"interface":"front","average":{"1min":1.736,"5min":1.774,"15min":1.889},"min":{"1min":1.1200000000000001,"5min":0.98999999999999999,"15min":0.79500000000000004},"max":{"1min":3.6349999999999998,"5min":8.0739999999999998,"15min":32.424999999999997},"last":1.0920000000000001}]}]},{"osd":6,"up_from":43,"seq":184683593988,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":358928,"kb_used_data":307080,"kb_used_omap":0,"kb_used_meta":51840,"kb_avail":93386224,"statfs":{"total":95995035648,"available":95627493376,"internally_reserved":0,"allocated":314449920,"data_stored":402184866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53084160},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:46:01 2021","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":1.0840000000000001,"15min":1.113},"min":{"1min":0.58899999999999997,"5min":0.46899999999999997,"15min":0.38800000000000001},"max":{"1min":1.7529999999999999,"5min":13.393000000000001,"15min":40.652000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0700000000000001,"15min":1.502},"min":{"1min":0.629,"5min":0.50900000000000001,"15min":0.47399999999999998},"max":{"1min":1.885,"5min":2.379,"15min":51.039999999999999},"last":1.1279999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:46:01 2021","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.1579999999999999,"15min":1.0649999999999999},"min":{"1min":0.60999999999999999,"5min":0.54400000000000004,"15min":0.443},"max":{"1min":1.4099999999999999,"5min":12.978999999999999,"15min":12.978999999999999},"last":0.92500000000000004},{"interface":"front","average":{"1min":1.329,"5min":1.169,"15min":1.1140000000000001},"min":{"1min":0.76300000000000001,"5min":0.60899999999999999,"15min":0.57799999999999996},"max":{"1min":2.6230000000000002,"5min":3.4199999999999999,"15min":6.0960000000000001},"last":1.028}]},{"osd":2,"last update":"Wed Feb 10 04:46:01 2021","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.2829999999999999,"15min":1.3080000000000001},"min":{"1min":0.95499999999999996,"5min":0.55600000000000005,"15min":0.54100000000000004},"max":{"1min":2.1269999999999998,"5min":2.645,"15min":24.725000000000001},"last":1.208},{"interface":"front","average":{"1min":1.175,"5min":1.24,"15min":1.2809999999999999},"min":{"1min":0.877,"5min":0.59899999999999998,"15min":0.55300000000000005},"max":{"1min":1.8680000000000001,"5min":2.2330000000000001,"15min":24.611999999999998},"last":1.1839999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:46:01 2021","interfaces":[{"interface":"back","average":{"1min":5.5620000000000003,"5min":2.2440000000000002,"15min":2.1720000000000002},"min":{"1min":1.069,"5min":0.76400000000000001,"15min":0.63200000000000001},"max":{"1min":68.653999999999996,"5min":68.653999999999996,"15min":114.17700000000001},"last":1.3100000000000001},{"interface":"front","average":{"1min":5.3010000000000002,"5min":2.3149999999999999,"15min":2.1789999999999998},"min":{"1min":0.96599999999999997,"5min":0.745,"15min":0.71699999999999997},"max":{"1min":68.361000000000004,"5min":68.361000000000004,"15min":116.929},"last":1.3420000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:46:01 2021","interfaces":[{"interface":"back","average":{"1min":1.5860000000000001,"5min":1.498,"15min":1.4199999999999999},"min":{"1min":1.022,"5min":0.72399999999999998,"15min":0.60599999999999998},"max":{"1min":2.016,"5min":3.141,"15min":7.5389999999999997},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.1799999999999999,"15min":1.101},"min":{"1min":0.78700000000000003,"5min":0.67400000000000004,"15min":0.57999999999999996},"max":{"1min":2.952,"5min":5.0640000000000001,"15min":5.0640000000000001},"last":1.2789999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:46:01 2021","interfaces":[{"interface":"back","average":{"1min":1.593,"5min":2.2799999999999998,"15min":1.6559999999999999},"min":{"1min":0.80500000000000005,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":3.2759999999999998,"5min":81.540000000000006,"15min":81.540000000000006},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.631,"5min":2.5190000000000001,"15min":1.849},"min":{"1min":1.0369999999999999,"5min":0.78000000000000003,"15min":0.66600000000000004},"max":{"1min":2.4060000000000001,"5min":81.492999999999995,"15min":81.492999999999995},"last":1.534}]},{"osd":7,"last update":"Wed Feb 10 04:46:23 2021","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.369,"15min":1.3520000000000001},"min":{"1min":0.95499999999999996,"5min":0.77400000000000002,"15min":0.755},"max":{"1min":2.2010000000000001,"5min":5.1820000000000004,"15min":5.1820000000000004},"last":1.157},{"interface":"front","average":{"1min":1.552,"5min":1.744,"15min":1.617},"min":{"1min":0.98399999999999999,"5min":0.96899999999999997,"15min":0.77500000000000002},"max":{"1min":2.3700000000000001,"5min":13.315,"15min":13.315},"last":1.4099999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575125,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":342032,"kb_used_data":292488,"kb_used_omap":0,"kb_used_meta":49536,"kb_avail":93403120,"statfs":{"total":95995035648,"available":95644794880,"internally_reserved":0,"allocated":299507712,"data_stored":381994862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":50724864},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:46:32 2021","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":1.4770000000000001,"15min":1.5700000000000001},"min":{"1min":0.53400000000000003,"5min":0.53000000000000003,"15min":0.502},"max":{"1min":1.7270000000000001,"5min":31.198,"15min":55.475999999999999},"last":0.627},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.482,"15min":1.637},"min":{"1min":0.63600000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.6279999999999999,"5min":31.558,"15min":55.368000000000002},"last":0.996}]},{"osd":2,"last update":"Wed Feb 10 04:45:51 2021","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.421,"15min":1.4730000000000001},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.49399999999999999},"max":{"1min":1.9870000000000001,"5min":30.585000000000001,"15min":40.268000000000001},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.466,"15min":1.5109999999999999},"min":{"1min":0.61499999999999999,"5min":0.59899999999999998,"15min":0.46800000000000003},"max":{"1min":2.3079999999999998,"5min":30.167999999999999,"15min":40.651000000000003},"last":0.85399999999999998}]},{"osd":3,"last update":"Wed Feb 10 04:46:10 2021","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.536,"15min":1.784},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.54700000000000004},"max":{"1min":2.1669999999999998,"5min":30.274999999999999,"15min":40.314},"last":1.359},{"interface":"front","average":{"1min":1.335,"5min":1.5429999999999999,"15min":1.762},"min":{"1min":0.88400000000000001,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":2.2290000000000001,"5min":31.443000000000001,"15min":40.973999999999997},"last":1.917}]},{"osd":4,"last update":"Wed Feb 10 04:46:32 2021","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1970000000000001,"15min":1.2509999999999999},"min":{"1min":0.66100000000000003,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":2.419,"5min":3.3849999999999998,"15min":6.3099999999999996},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.329,"5min":1.534,"15min":1.4370000000000001},"min":{"1min":0.95499999999999996,"5min":0.73199999999999998,"15min":0.70999999999999996},"max":{"1min":1.7350000000000001,"5min":29.547999999999998,"15min":29.547999999999998},"last":1.3320000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:45:45 2021","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.1950000000000001,"15min":1.5229999999999999},"min":{"1min":0.79000000000000004,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.754,"5min":4.6289999999999996,"15min":48.686999999999998},"last":1.647},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.7589999999999999,"15min":1.8220000000000001},"min":{"1min":0.871,"5min":0.84599999999999997,"15min":0.69699999999999995},"max":{"1min":2.2949999999999999,"5min":28.678000000000001,"15min":48.905000000000001},"last":1.671}]},{"osd":6,"last update":"Wed Feb 10 04:45:51 2021","interfaces":[{"interface":"back","average":{"1min":1.5780000000000001,"5min":1.8720000000000001,"15min":1.8120000000000001},"min":{"1min":1.173,"5min":0.82199999999999995,"15min":0.68400000000000005},"max":{"1min":2.6499999999999999,"5min":29.686,"15min":48.878999999999998},"last":1.446},{"interface":"front","average":{"1min":1.262,"5min":1.651,"15min":1.694},"min":{"1min":0.94599999999999995,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.865,"5min":28.475000000000001,"15min":49.011000000000003},"last":1.4159999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:46:10 2021","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.7370000000000001,"15min":1.6120000000000001},"min":{"1min":1.034,"5min":0.93999999999999995,"15min":0.81799999999999995},"max":{"1min":2.516,"5min":28.573,"15min":28.573},"last":1.139},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.825,"15min":1.6639999999999999},"min":{"1min":1.159,"5min":0.92600000000000005,"15min":0.92600000000000005},"max":{"1min":1.8779999999999999,"5min":29.408999999999999,"15min":29.408999999999999},"last":1.5489999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738651,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":394288,"kb_used_data":334888,"kb_used_omap":0,"kb_used_meta":59392,"kb_avail":93350864,"statfs":{"total":95995035648,"available":95591284736,"internally_reserved":0,"allocated":342925312,"data_stored":468056880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":60817408},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 04:46:24 2021","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.0229999999999999,"15min":1.0920000000000001},"min":{"1min":0.53800000000000003,"5min":0.53100000000000003,"15min":0.48199999999999998},"max":{"1min":2.0230000000000001,"5min":2.4159999999999999,"15min":28.946999999999999},"last":0.497},{"interface":"front","average":{"1min":0.997,"5min":1.087,"15min":1.01},"min":{"1min":0.55100000000000005,"5min":0.52700000000000002,"15min":0.52100000000000002},"max":{"1min":1.742,"5min":3.7050000000000001,"15min":3.7050000000000001},"last":0.91400000000000003}]},{"osd":2,"last update":"Wed Feb 10 04:45:58 2021","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.1799999999999999,"15min":1.3280000000000001},"min":{"1min":0.69499999999999995,"5min":0.57399999999999995,"15min":0.51400000000000001},"max":{"1min":1.74,"5min":3.903,"15min":25.984999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.167,"15min":1.2070000000000001},"min":{"1min":0.70199999999999996,"5min":0.56999999999999995,"15min":0.52800000000000002},"max":{"1min":1.625,"5min":3.4580000000000002,"15min":16.829000000000001},"last":0.79500000000000004}]},{"osd":3,"last update":"Wed Feb 10 04:46:13 2021","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.165,"15min":1.429},"min":{"1min":0.67100000000000004,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.8899999999999999,"5min":4.5830000000000002,"15min":29.692},"last":0.747},{"interface":"front","average":{"1min":1.484,"5min":1.514,"15min":1.72},"min":{"1min":0.90200000000000002,"5min":0.78500000000000003,"15min":0.54200000000000004},"max":{"1min":2.1869999999999998,"5min":5.8369999999999997,"15min":29.814},"last":0.93899999999999995}]},{"osd":4,"last update":"Wed Feb 10 04:46:24 2021","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.377,"15min":1.351},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.59399999999999997},"max":{"1min":2.0419999999999998,"5min":3.214,"15min":5.1680000000000001},"last":1.129},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.1970000000000001,"15min":1.2949999999999999},"min":{"1min":0.80100000000000005,"5min":0.72299999999999998,"15min":0.63800000000000001},"max":{"1min":1.6220000000000001,"5min":2.8690000000000002,"15min":14.672000000000001},"last":0.871}]},{"osd":5,"last update":"Wed Feb 10 04:45:43 2021","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.661,"15min":1.544},"min":{"1min":0.71999999999999997,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.77,"5min":25.047000000000001,"15min":25.047000000000001},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.641,"5min":1.861,"15min":1.843},"min":{"1min":1.0860000000000001,"5min":1.0580000000000001,"15min":0.88800000000000001},"max":{"1min":2.4319999999999999,"5min":24.972999999999999,"15min":28.827999999999999},"last":0.72299999999999998}]},{"osd":6,"last update":"Wed Feb 10 04:46:07 2021","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.786,"15min":1.881},"min":{"1min":1.022,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":2.4239999999999999,"5min":24.837,"15min":35.872999999999998},"last":1.2569999999999999},{"interface":"front","average":{"1min":1.819,"5min":1.966,"15min":2.0419999999999998},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":0.83099999999999996},"max":{"1min":2.5990000000000002,"5min":24.870999999999999,"15min":35.911000000000001},"last":1.006}]},{"osd":7,"last update":"Wed Feb 10 04:46:13 2021","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.603,"15min":1.5600000000000001},"min":{"1min":0.98299999999999998,"5min":0.80000000000000004,"15min":0.73499999999999999},"max":{"1min":2.1299999999999999,"5min":3.3330000000000002,"15min":5.6050000000000004},"last":1.347},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.5389999999999999,"15min":1.7110000000000001},"min":{"1min":1.0580000000000001,"5min":0.97299999999999998,"15min":0.94199999999999995},"max":{"1min":2.214,"5min":3.4300000000000002,"15min":16.545000000000002},"last":1.1599999999999999}]}]},{"osd":2,"up_from":0,"seq":77309411604,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":25,"seq":107374182670,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":386876,"kb_used_data":319668,"kb_used_omap":0,"kb_used_meta":67200,"kb_avail":93358276,"statfs":{"total":95995035648,"available":95598874624,"internally_reserved":0,"allocated":327340032,"data_stored":451995798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":68812800},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:46:10 2021","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.95399999999999996,"15min":1.0309999999999999},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.47599999999999998},"max":{"1min":1.333,"5min":2.113,"15min":7.899},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.117,"15min":1.107},"min":{"1min":0.57799999999999996,"5min":0.53100000000000003,"15min":0.52300000000000002},"max":{"1min":1.8999999999999999,"5min":3.0779999999999998,"15min":3.7330000000000001},"last":1.3100000000000001}]},{"osd":1,"last update":"Wed Feb 10 04:46:10 2021","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":1.0740000000000001,"15min":1.1080000000000001},"min":{"1min":0.66500000000000004,"5min":0.56899999999999995,"15min":0.56599999999999995},"max":{"1min":1.8660000000000001,"5min":3.153,"15min":8.9190000000000005},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.86699999999999999,"5min":1.02,"15min":1.036},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52100000000000002},"max":{"1min":1.2989999999999999,"5min":2.3740000000000001,"15min":4.6769999999999996},"last":0.83199999999999996}]},{"osd":2,"last update":"Wed Feb 10 04:46:10 2021","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.3220000000000001,"15min":1.2929999999999999},"min":{"1min":0.745,"5min":0.68100000000000005,"15min":0.56799999999999995},"max":{"1min":1.998,"5min":3.29,"15min":5.024},"last":0.68600000000000005},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.204,"15min":1.214},"min":{"1min":0.79900000000000004,"5min":0.61299999999999999,"15min":0.60799999999999998},"max":{"1min":2.1469999999999998,"5min":2.5449999999999999,"15min":4.5880000000000001},"last":1.1890000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:46:21 2021","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.3500000000000001,"15min":1.4670000000000001},"min":{"1min":0.88300000000000001,"5min":0.85099999999999998,"15min":0.629},"max":{"1min":1.698,"5min":2.5390000000000001,"15min":30.751999999999999},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.325,"5min":1.4550000000000001,"15min":1.5600000000000001},"min":{"1min":0.84499999999999997,"5min":0.749,"15min":0.65700000000000003},"max":{"1min":2.181,"5min":2.8559999999999999,"15min":30.928000000000001},"last":1.0009999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:46:40 2021","interfaces":[{"interface":"back","average":{"1min":1.5489999999999999,"5min":2.0390000000000001,"15min":1.9399999999999999},"min":{"1min":1.137,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":2.4129999999999998,"5min":26.183,"15min":35.616999999999997},"last":1.23},{"interface":"front","average":{"1min":1.5109999999999999,"5min":2.1379999999999999,"15min":2.0459999999999998},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":2.5449999999999999,"5min":26.331,"15min":35.661999999999999},"last":1.4810000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:46:03 2021","interfaces":[{"interface":"back","average":{"1min":1.6299999999999999,"5min":1.895,"15min":1.978},"min":{"1min":1.2649999999999999,"5min":0.88500000000000001,"15min":0.88500000000000001},"max":{"1min":2.1000000000000001,"5min":18.614999999999998,"15min":35.518000000000001},"last":1.7310000000000001},{"interface":"front","average":{"1min":1.5409999999999999,"5min":1.9339999999999999,"15min":2.2429999999999999},"min":{"1min":1.087,"5min":0.99099999999999999,"15min":0.92400000000000004},"max":{"1min":2.105,"5min":18.036999999999999,"15min":53.965000000000003},"last":1.2729999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:46:10 2021","interfaces":[{"interface":"back","average":{"1min":1.5700000000000001,"5min":1.7030000000000001,"15min":1.728},"min":{"1min":1.1319999999999999,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":2.0249999999999999,"5min":3.1850000000000001,"15min":4.3460000000000001},"last":1.387},{"interface":"front","average":{"1min":1.617,"5min":1.8069999999999999,"15min":1.8200000000000001},"min":{"1min":1.0169999999999999,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":2.3109999999999999,"5min":5.0270000000000001,"15min":5.2549999999999999},"last":1.1419999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986443,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":115624,"kb_used_data":71776,"kb_used_omap":0,"kb_used_meta":43840,"kb_avail":93629528,"statfs":{"total":95995035648,"available":95876636672,"internally_reserved":0,"allocated":73498624,"data_stored":93869923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":44892160},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4],"upper_bound":524288},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:46:25 2021","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":1.0249999999999999,"15min":1.1799999999999999},"min":{"1min":0.60499999999999998,"5min":0.503,"15min":0.501},"max":{"1min":2.1800000000000002,"5min":6.0880000000000001,"15min":43.939999999999998},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.997,"5min":0.999,"15min":1.2090000000000001},"min":{"1min":0.65600000000000003,"5min":0.52400000000000002,"15min":0.499},"max":{"1min":1.79,"5min":6.1479999999999997,"15min":43.829999999999998},"last":0.77000000000000002}]},{"osd":1,"last update":"Wed Feb 10 04:46:25 2021","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.872,"15min":0.92400000000000004},"min":{"1min":0.57799999999999996,"5min":0.56000000000000005,"15min":0.52800000000000002},"max":{"1min":1.036,"5min":2.085,"15min":24.295000000000002},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.026,"5min":1.167,"15min":1.1910000000000001},"min":{"1min":0.69499999999999995,"5min":0.60399999999999998,"15min":0.59999999999999998},"max":{"1min":1.923,"5min":6.5060000000000002,"15min":24.943000000000001},"last":1.024}]},{"osd":2,"last update":"Wed Feb 10 04:46:25 2021","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.2310000000000001,"15min":1.1419999999999999},"min":{"1min":0.82199999999999995,"5min":0.63800000000000001,"15min":0.56499999999999995},"max":{"1min":2.0510000000000002,"5min":5.7489999999999997,"15min":7.048},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.224,"15min":1.1899999999999999},"min":{"1min":0.76200000000000001,"5min":0.68200000000000005,"15min":0.51900000000000002},"max":{"1min":1.9410000000000001,"5min":2.7679999999999998,"15min":3.8919999999999999},"last":1.095}]},{"osd":3,"last update":"Wed Feb 10 04:46:25 2021","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.413,"15min":1.407},"min":{"1min":0.92600000000000005,"5min":0.627,"15min":0.60699999999999998},"max":{"1min":2.508,"5min":8.2119999999999997,"15min":24.530999999999999},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.3360000000000001,"15min":1.351},"min":{"1min":0.86399999999999999,"5min":0.74299999999999999,"15min":0.70299999999999996},"max":{"1min":1.742,"5min":8.2469999999999999,"15min":24.669},"last":1.1240000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:46:42 2021","interfaces":[{"interface":"back","average":{"1min":1.641,"5min":1.508,"15min":1.4410000000000001},"min":{"1min":0.90900000000000003,"5min":0.68200000000000005,"15min":0.60999999999999999},"max":{"1min":5.0270000000000001,"5min":5.0270000000000001,"15min":12.773999999999999},"last":1.931},{"interface":"front","average":{"1min":1.589,"5min":1.4419999999999999,"15min":1.3580000000000001},"min":{"1min":0.78400000000000003,"5min":0.70299999999999996,"15min":0.58799999999999997},"max":{"1min":4.8570000000000002,"5min":4.8570000000000002,"15min":13.038},"last":1.746}]},{"osd":6,"last update":"Wed Feb 10 04:45:51 2021","interfaces":[{"interface":"back","average":{"1min":1.7669999999999999,"5min":1.627,"15min":1.71},"min":{"1min":1.1259999999999999,"5min":0.82599999999999996,"15min":0.70299999999999996},"max":{"1min":3.3330000000000002,"5min":3.3330000000000002,"15min":42.088999999999999},"last":1.341},{"interface":"front","average":{"1min":1.9079999999999999,"5min":1.7050000000000001,"15min":1.8080000000000001},"min":{"1min":1.2270000000000001,"5min":0.97099999999999997,"15min":0.83899999999999997},"max":{"1min":4.6970000000000001,"5min":4.6970000000000001,"15min":42.027000000000001},"last":1.377}]},{"osd":7,"last update":"Wed Feb 10 04:46:18 2021","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.5509999999999999,"15min":1.4610000000000001},"min":{"1min":0.97799999999999998,"5min":0.874,"15min":0.84899999999999998},"max":{"1min":2.5350000000000001,"5min":5.8920000000000003,"15min":6.9199999999999999},"last":1.228},{"interface":"front","average":{"1min":1.6599999999999999,"5min":1.619,"15min":1.55},"min":{"1min":1.1299999999999999,"5min":0.93799999999999994,"15min":0.71499999999999997},"max":{"1min":2.319,"5min":4.5499999999999998,"15min":4.5499999999999998},"last":1.516}]}]},{"osd":5,"up_from":37,"seq":158913790216,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":382072,"kb_used_data":345328,"kb_used_omap":0,"kb_used_meta":36736,"kb_avail":93363080,"statfs":{"total":95995035648,"available":95603793920,"internally_reserved":0,"allocated":353615872,"data_stored":447843917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37617664},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:45:42 2021","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.97799999999999998,"15min":1.3939999999999999},"min":{"1min":0.57599999999999996,"5min":0.51800000000000002,"15min":0.40500000000000003},"max":{"1min":1.2789999999999999,"5min":4.5720000000000001,"15min":104.04300000000001},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.006,"5min":1.085,"15min":1.466},"min":{"1min":0.57099999999999995,"5min":0.501,"15min":0.46400000000000002},"max":{"1min":1.5720000000000001,"5min":15.539999999999999,"15min":107.776},"last":0.77500000000000002}]},{"osd":1,"last update":"Wed Feb 10 04:45:42 2021","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.006,"15min":1.319},"min":{"1min":0.57199999999999995,"5min":0.51900000000000002,"15min":0.504},"max":{"1min":1.6850000000000001,"5min":5.29,"15min":103.904},"last":0.71899999999999997},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2490000000000001,"15min":1.504},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.50800000000000001},"max":{"1min":2.097,"5min":11.875999999999999,"15min":104.315},"last":0.66400000000000003}]},{"osd":2,"last update":"Wed Feb 10 04:45:42 2021","interfaces":[{"interface":"back","average":{"1min":2.157,"5min":1.4339999999999999,"15min":2.0430000000000001},"min":{"1min":0.65100000000000002,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":17.693999999999999,"5min":17.693999999999999,"15min":103.65600000000001},"last":1.018},{"interface":"front","average":{"1min":2.1539999999999999,"5min":1.407,"15min":2.0390000000000001},"min":{"1min":0.66600000000000004,"5min":0.64000000000000001,"15min":0.63500000000000001},"max":{"1min":17.640999999999998,"5min":17.640999999999998,"15min":104.136},"last":0.97999999999999998}]},{"osd":3,"last update":"Wed Feb 10 04:45:42 2021","interfaces":[{"interface":"back","average":{"1min":3.8839999999999999,"5min":1.8740000000000001,"15min":2.1440000000000001},"min":{"1min":0.80600000000000005,"5min":0.66100000000000003,"15min":0.624},"max":{"1min":45.643000000000001,"5min":45.643000000000001,"15min":106.18000000000001},"last":1.3839999999999999},{"interface":"front","average":{"1min":3.9729999999999999,"5min":2.1030000000000002,"15min":2.1709999999999998},"min":{"1min":0.95799999999999996,"5min":0.754,"15min":0.64500000000000002},"max":{"1min":45.517000000000003,"5min":45.517000000000003,"15min":106.822},"last":1.5029999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:45:42 2021","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.325,"15min":1.242},"min":{"1min":0.85299999999999998,"5min":0.57099999999999995,"15min":0.55400000000000005},"max":{"1min":2.395,"5min":6.1639999999999997,"15min":6.1639999999999997},"last":0.91200000000000003},{"interface":"front","average":{"1min":1.504,"5min":1.446,"15min":1.3859999999999999},"min":{"1min":1.0429999999999999,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":2.052,"5min":3.0209999999999999,"15min":8.1449999999999996},"last":0.88700000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:45:52 2021","interfaces":[{"interface":"back","average":{"1min":1.7110000000000001,"5min":5.1580000000000004,"15min":2.8599999999999999},"min":{"1min":1.1679999999999999,"5min":0.65800000000000003,"15min":0.63500000000000001},"max":{"1min":2.27,"5min":319.51999999999998,"15min":319.51999999999998},"last":1.462},{"interface":"front","average":{"1min":1.5449999999999999,"5min":5.0720000000000001,"15min":2.7189999999999999},"min":{"1min":1.006,"5min":0.80500000000000005,"15min":0.74399999999999999},"max":{"1min":2.1520000000000001,"5min":319.22300000000001,"15min":319.22300000000001},"last":1.333}]},{"osd":7,"last update":"Wed Feb 10 04:46:15 2021","interfaces":[{"interface":"back","average":{"1min":1.621,"5min":1.474,"15min":1.4399999999999999},"min":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.752},"max":{"1min":2.4809999999999999,"5min":2.4809999999999999,"15min":5.4770000000000003},"last":1.046},{"interface":"front","average":{"1min":1.605,"5min":1.6080000000000001,"15min":1.6100000000000001},"min":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.83499999999999996},"max":{"1min":2.3809999999999998,"5min":2.992,"15min":8.7840000000000007},"last":1.2589999999999999}]}]}]}} 2021-02-10T04:47:03.793 INFO:tasks.thrashosds.thrasher:Setting 4.1c to [6, 5, 0] 2021-02-10T04:47:03.794 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '4.1c', '6', '5', '0'] 2021-02-10T04:47:03.794 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pg-upmap 4.1c 6 5 0 2021-02-10T04:47:04.106 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:47:04.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:03 smithi164 conmon[33677]: debug 2021-02-10T04:47:03.764+0000 7fbec19a1700 -1 osd.4 1063 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:04.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:03 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:47:04.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:03 smithi164 conmon[29288]: 02-10T04:47:02.331157+0000 mgr.y (mgr.14140) 1042 : cluster [DBG] pgmap v1815: 188 pgs: 188 active+clean; 887 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:04.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:03 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:47:04.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:03 smithi164 conmon[29288]: 02-10T04:47:02.738169+0000 osd.4 (osd.4) 2018 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:04.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:03 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:47:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:03 smithi164 conmon[29288]: 02-10T04:47:02.738178+0000 osd.4 2021-02-10T04:47:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:03 smithi164 conmon[29288]: (osd.4) 2019 : cluster [WRN] 2021-02-10T04:47:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:03 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:03 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:47:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:03 smithi164 conmon[29288]: 10T04:47:02.738184+0000 osd.4 (osd 2021-02-10T04:47:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:03 smithi164 conmon[29288]: .4) 2020 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:03 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:47:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:03 smithi164 conmon[29288]: 10T04:47:02.738192+0000 osd.4 2021-02-10T04:47:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:03 smithi164 conmon[29288]: (osd.4) 2021 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:03 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:47:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:03 smithi164 conmon[29288]: 02-10T04:47:02 2021-02-10T04:47:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:03 smithi164 conmon[29288]: .791474+0000 osd.5 (osd. 2021-02-10T04:47:04.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:03 smithi164 conmon[29288]: 5) 123 : cluster [DBG] 2021-02-10T04:47:04.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:03 smithi164 conmon[29288]: 2.c scrub ok 2021-02-10T04:47:04.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:03 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:47:04.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:03 smithi164 conmon[29288]: -10T04:47:02 2021-02-10T04:47:04.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:03 smithi164 conmon[29288]: .875913+0000 osd.0 (osd. 2021-02-10T04:47:04.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:03 smithi164 conmon[29288]: 0) 134 : cluster [DBG] 7.6 scrub ok 2021-02-10T04:47:04.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:03 smithi164 conmon[29288]: 2021-02-10T04:47:04.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:03 smithi085 conmon[29319]: cluster 2021-02-10T04:47:02.331157+0000 mgr.y (mgr.14140) 1042 : cluster [DBG] pgmap v1815: 188 pgs: 188 active+clean; 887 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:04.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:03 smithi085 conmon[29319]: cluster 2021-02-10T04:47:02.738169+0000 osd.4 (osd.4) 2018 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:04.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:03 smithi085 conmon[29319]: cluster 2021-02-10T04:47:02.738178+0000 osd.4 (osd.4) 2019 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:04.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:03 smithi085 conmon[29319]: cluster 2021-02-10T04:47:02.738184+0000 osd.4 (osd.4) 2020 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:04.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:03 smithi085 conmon[29319]: cluster 2021-02-10T04:47:02.738192+0000 osd.4 (osd.4) 2021 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:04.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:03 smithi085 conmon[29319]: cluster 2021-02-10T04:47:02.791474+0000 osd.5 (osd.5) 123 : cluster [DBG] 2.c scrub ok 2021-02-10T04:47:04.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:03 smithi085 conmon[29319]: cluster 2021-02-10T04:47:02.875913+0000 osd.0 (osd.0) 134 : cluster [DBG] 7.6 scrub ok 2021-02-10T04:47:04.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:03 smithi085 conmon[35961]: cluster 2021-02-10T04:47:02.331157 2021-02-10T04:47:04.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:03 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1042 : cluster [DBG] pgmap v1815: 188 pgs: 188 active+clean; 887 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:04.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:03 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:47:04.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:03 smithi085 conmon[35961]: -10T04:47:02.738169+0000 osd.4 (osd.4) 2018 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:04.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:03 smithi085 conmon[35961]: 2021-02-10T04:47:04.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:03 smithi085 conmon[35961]: cluster 2021-02-10T04:47:02 2021-02-10T04:47:04.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:03 smithi085 conmon[35961]: .738178+0000 osd.4 (osd.4) 2019 : cluster 2021-02-10T04:47:04.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:03 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:04.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:03 smithi085 conmon[35961]: cluster 2021-02-10T04:47: 2021-02-10T04:47:04.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:03 smithi085 conmon[35961]: 02.738184+0000 osd.4 (osd.4) 2020 : 2021-02-10T04:47:04.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:03 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:04.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:03 smithi085 conmon[35961]: cluster 2021-02-10T04:47 2021-02-10T04:47:04.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:03 smithi085 conmon[35961]: :02.738192+0000 osd.4 (osd.4) 2021 2021-02-10T04:47:04.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:03 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:04.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:03 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:47:04.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:03 smithi085 conmon[35961]: :47:02.791474+0000 osd.5 (osd. 2021-02-10T04:47:04.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:03 smithi085 conmon[35961]: 5) 123 : cluster [DBG] 2.c scrub ok 2021-02-10T04:47:04.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:03 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:47:04.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:03 smithi085 conmon[35961]: -10T04:47:02.875913+0000 osd.0 2021-02-10T04:47:04.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:03 smithi085 conmon[35961]: (osd.0) 134 : cluster [DBG] 7.6 scrub ok 2021-02-10T04:47:05.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:04 smithi164 conmon[33677]: debug 2021-02-10T04:47:04.755+0000 7fbec19a1700 -1 osd.4 1063 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:04 smithi164 conmon[29288]: cluster 2021-02-10T04:47:03.764944+0000 osd.4 (osd 2021-02-10T04:47:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:04 smithi164 conmon[29288]: .4) 2022 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:04 smithi164 conmon[29288]: cluster 2021-02-10T04:47:03.764951+0000 osd.4 (osd.4) 2023 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:04 smithi164 conmon[29288]: cluster 2021-02-10T04:47:03.764956+0000 osd.4 ( 2021-02-10T04:47:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:04 smithi164 conmon[29288]: osd.4) 2024 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:05.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:04 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:47:05.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:04 smithi164 conmon[29288]: 02-10T04:47:03.764960+0000 2021-02-10T04:47:05.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:04 smithi164 conmon[29288]: osd.4 (osd.4) 2025 : cluster [WRN] 2021-02-10T04:47:05.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:04 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:05.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:04 smithi164 conmon[29288]: cluster 2021-02-10T04:47 2021-02-10T04:47:05.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:04 smithi164 conmon[29288]: :03.951722+0000 osd.6 (osd.6) 124 2021-02-10T04:47:05.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:04 smithi164 conmon[29288]: : cluster [DBG] 7.19 scrub ok 2021-02-10T04:47:05.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:04 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:47:05.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:04 smithi164 conmon[29288]: -10T04:47:04.591690+0000 osd.3 2021-02-10T04:47:05.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:04 smithi164 conmon[29288]: (osd.3) 127 : cluster [DBG] 2.1 scrub ok 2021-02-10T04:47:05.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:04 smithi085 conmon[29319]: cluster 2021-02-10T04:47:03. 2021-02-10T04:47:05.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:04 smithi085 conmon[29319]: 764944+0000 osd.4 (osd.4) 2022 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:05.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:04 smithi085 conmon[29319]: cluster 2021-02-10T04:47:03.764951+0000 osd.4 (osd.4) 2023 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:05.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:04 smithi085 conmon[29319]: cluster 2021-02-10T04:47:03.764956+0000 osd.4 (osd.4) 2024 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:05.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:04 smithi085 conmon[29319]: cluster 2021-02-10T04:47:03.764960+0000 osd.4 (osd.4) 2025 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:05.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:04 smithi085 conmon[29319]: cluster 2021-02-10T04:47:03 2021-02-10T04:47:05.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:04 smithi085 conmon[29319]: .951722+0000 osd.6 (osd.6) 124 : cluster [DBG] 7.19 scrub ok 2021-02-10T04:47:05.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:04 smithi085 conmon[29319]: cluster 2021-02-10T04:47:04 2021-02-10T04:47:05.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:04 smithi085 conmon[29319]: .591690+0000 osd.3 (osd.3) 127 : cluster [DBG] 2.1 scrub ok 2021-02-10T04:47:05.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:04 smithi085 conmon[35961]: cluster 2021-02-10T04:47:03 2021-02-10T04:47:05.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:04 smithi085 conmon[35961]: .764944+0000 osd.4 (osd.4) 2022 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:05.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:04 smithi085 conmon[35961]: cluster 2021-02-10T04:47:03.764951+0000 osd.4 (osd.4) 2023 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:05.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:04 smithi085 conmon[35961]: cluster 2021-02-10T04:47:03. 2021-02-10T04:47:05.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:04 smithi085 conmon[35961]: 764956+0000 osd.4 (osd.4) 2024 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:05.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:04 smithi085 conmon[35961]: cluster 2021-02-10T04:47:03.764960+0000 osd.4 (osd.4) 2021-02-10T04:47:05.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:04 smithi085 conmon[35961]: 2025 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:05.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:04 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:47:05.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:04 smithi085 conmon[35961]: -02-10T04:47:03.951722+0000 osd.6 (osd.6) 124 2021-02-10T04:47:05.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:04 smithi085 conmon[35961]: : cluster [DBG] 7.19 scrub ok 2021-02-10T04:47:05.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:04 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:47:05.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:04 smithi085 conmon[35961]: 47:04.591690+0000 osd.3 (osd. 2021-02-10T04:47:05.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:04 smithi085 conmon[35961]: 3) 127 : cluster [DBG] 2.1 scrub ok 2021-02-10T04:47:05.503 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:47:05.540 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:47:05.714 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:47:05.760 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.7 dump_ops_in_flight 2021-02-10T04:47:05.801 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:47:05.823 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set noscrub 2021-02-10T04:47:05.907 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:47:05.937 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:47:06.018 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:47:05 smithi164 conmon[37263]: debug 2021-02-10T04:47:05.772+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:47:06.018 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:05 smithi164 conmon[29288]: cluster 2021-02-10T04:47:04.331824+0000 mgr.y 2021-02-10T04:47:06.018 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:05 smithi164 conmon[29288]: (mgr.14140) 1043 : cluster [DBG] pgmap v1816: 188 pgs: 188 active+clean; 886 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:06.018 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:05 smithi164 conmon[29288]: cluster 2021-02-10T04:47:04.756413+0000 osd.4 (osd.4) 2021-02-10T04:47:06.018 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:05 smithi164 conmon[29288]: 2026 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:06.018 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:05 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:47:06.018 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:05 smithi164 conmon[29288]: 04:47:04.756422+0000 osd.4 (osd.4) 2027 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:06.018 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:05 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:47:06.018 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:05 smithi164 conmon[29288]: -10T04:47:04.756429+0000 osd.4 2021-02-10T04:47:06.019 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:05 smithi164 conmon[29288]: (osd.4) 2028 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:06.019 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:05 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:47:06.019 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:05 smithi164 conmon[29288]: 10T04:47:04.756436+0000 osd.4 (osd 2021-02-10T04:47:06.019 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:05 smithi164 conmon[29288]: .4) 2029 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:06.019 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:05 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:47:06.019 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:05 smithi164 conmon[29288]: 04:47:04.957066+0000 osd.6 (osd.6) 125 : cluster 2021-02-10T04:47:06.020 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:05 smithi164 conmon[29288]: [DBG] 2.7 scrub ok 2021-02-10T04:47:06.020 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:05 smithi164 conmon[29288]: cluster 2021-02-10T04:47:06.020 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:05 smithi164 conmon[29288]: 2021-02-10T04:47:05.589574+0000 2021-02-10T04:47:06.020 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:05 smithi164 conmon[29288]: osd.3 (osd.3) 128 : cluster [DBG] 4.17 deep-scrub ok 2021-02-10T04:47:06.020 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:05 smithi164 conmon[29288]: 2021-02-10T04:47:06.021 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:05 smithi164 conmon[33677]: debug 2021-02-10T04:47:05.797+0000 7fbec19a1700 -1 osd.4 1063 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:06.021 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:06 smithi164 conmon[33677]: debug 2021-02-10T04:47:06.016+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:47:06.029 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:47:06.042 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:47:06.148 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:47:06.151 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:47:06.155 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:47:06.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[29319]: cluster 2021-02-10T04:47:04.331824+0000 mgr.y (mgr 2021-02-10T04:47:06.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[29319]: .14140) 1043 : cluster [DBG] pgmap v1816: 188 pgs: 188 active+clean; 886 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:06.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[29319]: cluster 2021-02-10T04:47:04.756413+0000 osd.4 (osd.4) 2026 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:06.181 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[29319]: cluster 2021-02-10T04:47:04.756422+0000 osd.4 (osd.4) 2027 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:06.182 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[29319]: cluster 2021-02-10T04:47:04.756429+0000 osd.4 (osd.4) 2028 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:06.182 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[29319]: cluster 2021-02-10T04:47:04.756436+0000 osd.4 (osd.4) 2029 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:06.182 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[29319]: cluster 2021-02-10T04:47:04.957066+0000 osd.6 (osd.6) 125 : cluster [DBG] 2.7 scrub ok 2021-02-10T04:47:06.182 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[29319]: cluster 2021-02-10T04:47:05.589574+0000 osd.3 (osd.3) 128 : cluster [DBG] 4.17 deep-scrub ok 2021-02-10T04:47:06.182 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: cluster 2021-02-10T04:47:06.182 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 2021-02-10T04:47 2021-02-10T04:47:06.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: :04.331824+0000 mgr.y (mgr.14140) 1043 : cluster [DBG] pgmap v1816: 188 pgs: 188 active+clean; 886 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:06.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:47:06.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 02-10T04:47:04.756413+0000 osd.4 ( 2021-02-10T04:47:06.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: osd.4) 2026 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:06.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:47:06.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: -02-10T04:47:04.756422+0000 2021-02-10T04:47:06.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: osd.4 (osd.4) 2027 : cluster [WRN] 2021-02-10T04:47:06.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:06.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: cluster 2021-02-10T04:47:04. 2021-02-10T04:47:06.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 756429+0000 osd.4 (osd.4) 2028 : cluster [WRN] 2021-02-10T04:47:06.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:06.183 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: cluster 2021-02-10T04:47: 2021-02-10T04:47:06.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 04.756436+0000 osd.4 (osd.4) 2029 : 2021-02-10T04:47:06.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: cluster 2021-02-10T04:47:06.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 2021-02-10T04:47:06.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: [WRN] 2021-02-10T04:47:06.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 2021-02-10T04:47:06.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:06.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 2021-02-10T04:47:06.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: cluster 2021-02-10T04:47:06.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 2021-02-10T04:47:06.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 2021 2021-02-10T04:47:06.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: - 2021-02-10T04:47:06.184 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 02 2021-02-10T04:47:06.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: - 2021-02-10T04:47:06.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 10 2021-02-10T04:47:06.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: T 2021-02-10T04:47:06.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 04 2021-02-10T04:47:06.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: : 2021-02-10T04:47:06.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 47 2021-02-10T04:47:06.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: : 2021-02-10T04:47:06.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 04 2021-02-10T04:47:06.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: . 2021-02-10T04:47:06.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 957066 2021-02-10T04:47:06.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: +0000 2021-02-10T04:47:06.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 2021-02-10T04:47:06.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: osd.6 2021-02-10T04:47:06.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: ( 2021-02-10T04:47:06.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: osd 2021-02-10T04:47:06.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: . 2021-02-10T04:47:06.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 6 2021-02-10T04:47:06.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: ) 2021-02-10T04:47:06.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 125 2021-02-10T04:47:06.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: : 2021-02-10T04:47:06.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: cluster 2021-02-10T04:47:06.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 2021-02-10T04:47:06.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: [DBG] 2021-02-10T04:47:06.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 2021-02-10T04:47:06.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 2.7 scrub ok 2021-02-10T04:47:06.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 2021-02-10T04:47:06.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: cluster 2021-02-10T04:47:06.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 2021-02-10T04:47:06.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 2021 2021-02-10T04:47:06.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: - 2021-02-10T04:47:06.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 02 2021-02-10T04:47:06.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: - 2021-02-10T04:47:06.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 10 2021-02-10T04:47:06.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: T 2021-02-10T04:47:06.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 04 2021-02-10T04:47:06.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: : 2021-02-10T04:47:06.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 47 2021-02-10T04:47:06.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: : 2021-02-10T04:47:06.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 05 2021-02-10T04:47:06.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: . 2021-02-10T04:47:06.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 589574 2021-02-10T04:47:06.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: +0000 2021-02-10T04:47:06.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 2021-02-10T04:47:06.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: osd.3 2021-02-10T04:47:06.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: ( 2021-02-10T04:47:06.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: osd 2021-02-10T04:47:06.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: . 2021-02-10T04:47:06.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:05 smithi085 conmon[35961]: 3) 128 : cluster [DBG] 4.17 deep-scrub ok 2021-02-10T04:47:06.190 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:47:06.814 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:06 smithi164 conmon[33677]: debug 2021-02-10T04:47:06.811+0000 7fbec19a1700 -1 osd.4 1063 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:07.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:06 smithi164 conmon[29288]: cluster 2021-02-10T04:47:05.797705+0000 2021-02-10T04:47:09.619 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:06 smithi164 conmon[29288]: osd.4 (osd.4) 2030 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:09.619 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:06 smithi164 conmon[29288]: cluster 2021-02-10T04:47:05.797714 2021-02-10T04:47:09.619 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:06 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2031 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:09.619 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:06 smithi164 conmon[29288]: cluster 2021-02-10T04:47:09.619 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:06 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:47:09.619 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:06 smithi164 conmon[29288]: 47:05.797728+0000 osd.4 (osd.4) 2032 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:09.620 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:06 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:47:09.620 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:06 smithi164 conmon[29288]: 02-10T04:47:05.797735+0000 2021-02-10T04:47:09.620 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:06 smithi164 conmon[29288]: osd.4 (osd.4) 2033 : cluster [WRN] 2021-02-10T04:47:09.620 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:06 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:09.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:06 smithi085 conmon[35961]: cluster 2021-02-10T04:47:05.797705+0000 osd.4 2021-02-10T04:47:09.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:06 smithi085 conmon[35961]: (osd.4) 2030 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:09.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:06 smithi085 conmon[35961]: cluster 2021-02-10T04:47:05.797714+0000 2021-02-10T04:47:09.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:06 smithi085 conmon[35961]: osd.4 (osd.4) 2031 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:09.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:06 smithi085 conmon[35961]: cluster 2021-02-10T04:47:09.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:06 smithi085 conmon[35961]: 2021-02-10T04:47:05.797728+0000 osd.4 (osd.4) 2032 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:09.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:06 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:47:09.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:06 smithi085 conmon[35961]: :47:05.797735+0000 osd.4 (osd. 2021-02-10T04:47:09.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:06 smithi085 conmon[35961]: 4) 2033 : cluster [WRN] 2021-02-10T04:47:09.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:06 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:09.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[35961]: cluster 2021-02-10T04:47:09.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[35961]: 2021-02-10T04:47:09.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:47:09.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[35961]: :47:06.332488+0000 mgr.y (mgr.14140) 1044 : cluster [DBG] pgmap v1817: 188 pgs: 188 active+clean; 889 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:09.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[35961]: cluster 2021-02-10T04:47:09.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[35961]: 2021-02-10T04:47 2021-02-10T04:47:09.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[35961]: :06.812288+0000 osd.4 (osd.4) 2034 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:09.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:47:09.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[35961]: 02-10T04:47:06.812297+0000 osd.4 (osd.4 2021-02-10T04:47:09.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[35961]: ) 2035 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:09.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:47:09.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[35961]: -10T04:47:06.812303+0000 osd.4 (osd.4 2021-02-10T04:47:09.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[35961]: ) 2036 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:09.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:47:09.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[35961]: -10T04:47:06.812309+0000 osd.4 (osd.4) 2037 2021-02-10T04:47:09.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:09.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:47:09.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[35961]: T04:47:06.861803+0000 osd.5 (osd.5) 124 : cluster 2021-02-10T04:47:09.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[35961]: [DBG] 7.e deep-scrub ok 2021-02-10T04:47:09.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[35961]: cluster 2021-02-10T04:47:09.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[35961]: 2021-02-10T04:47:09.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[35961]: 2021 2021-02-10T04:47:09.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[35961]: - 2021-02-10T04:47:09.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[35961]: 02 2021-02-10T04:47:09.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[35961]: -10T04:47:07.765369+0000 2021-02-10T04:47:09.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[35961]: osd.4 (osd.4) 2038 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:09.626 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[35961]: cluster 2021-02-10T04:47:07.765378+0000 osd.4 (osd.4) 2039 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:09.626 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[35961]: cluster 2021-02-10T04:47:07.765382+0000 osd.4 2021-02-10T04:47:09.626 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[35961]: (osd.4) 2040 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:09.626 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[35961]: cluster 2021-02-10T04:47: 2021-02-10T04:47:09.626 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[35961]: 07.765385+0000 osd.4 (osd.4) 2041 : cluster 2021-02-10T04:47:09.626 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:09.626 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:47:09.626 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[35961]: :47:07.914055+0000 osd.0 (osd.0 2021-02-10T04:47:09.626 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[35961]: ) 135 : cluster [DBG] 6.0 scrub ok 2021-02-10T04:47:09.627 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:47:09.627 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[35961]: 10T04:47:08.041012+0000 osd.5 ( 2021-02-10T04:47:09.627 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[35961]: osd.5) 125 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:47:09.627 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:47:09.627 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[35961]: 02-10T04:47:08.044262+0000 2021-02-10T04:47:09.627 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[35961]: osd.5 (osd.5) 126 : cluster [DBG] 2021-02-10T04:47:09.627 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:47:09.629 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[35961]: cluster 2021-02-10T04:47: 2021-02-10T04:47:09.629 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[35961]: 08.401795+0000 mon.a (mon.0) 2126 : 2021-02-10T04:47:09.629 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[35961]: cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 157 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:47:09.630 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:06 smithi085 conmon[29319]: cluster 2021-02-10T04:47:05.797705+0000 osd.4 2021-02-10T04:47:09.630 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:06 smithi085 conmon[29319]: (osd.4) 2030 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:09.630 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:06 smithi085 conmon[29319]: cluster 2021-02-10T04:47:05.797714+0000 osd.4 (osd.4) 2031 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:09.630 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:06 smithi085 conmon[29319]: cluster 2021-02-10T04:47:05.797728+0000 osd.4 (osd.4) 2032 : cluster [WRN] 2021-02-10T04:47:09.630 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:06 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:09.630 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:06 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:47:09.630 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:06 smithi085 conmon[29319]: -02-10T04:47:05.797735+0000 osd.4 (osd.4) 2033 : cluster [WRN] 2021-02-10T04:47:09.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:06 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:09.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[29319]: cluster 2021-02-10T04:47 2021-02-10T04:47:09.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[29319]: :06.332488+0000 mgr.y (mgr.14140) 1044 : cluster [DBG] pgmap v1817: 188 pgs: 188 active+clean; 889 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:09.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[29319]: cluster 2021-02-10T04:47:06.812288+0000 osd.4 ( 2021-02-10T04:47:09.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[29319]: osd.4) 2034 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:09.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[29319]: cluster 2021-02-10T04:47:09.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[29319]: 2021-02-10T04:47:06.812297+0000 osd.4 (osd.4) 2035 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:09.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[29319]: cluster 2021-02-10T04:47 2021-02-10T04:47:09.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[29319]: :06.812303+0000 osd.4 (osd.4) 2036 2021-02-10T04:47:09.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:09.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:47:09.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[29319]: 47:06.812309+0000 osd.4 (osd.4) 2037 : cluster [WRN] 2021-02-10T04:47:09.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:09.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:47:09.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[29319]: -10T04:47:06.861803+0000 osd.5 (osd.5) 124 : 2021-02-10T04:47:09.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:07 smithi085 conmon[29319]: cluster [DBG] 7.e deep-scrub ok 2021-02-10T04:47:09.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[29319]: cluster 2021-02-10T04:47:07.765369+0000 osd.4 (osd.4 2021-02-10T04:47:09.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[29319]: ) 2038 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:09.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[29319]: cluster 2021-02-10T04:47:07.765378+0000 osd.4 (osd.4) 2039 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:09.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[29319]: cluster 2021-02-10T04:47:09.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[29319]: 2021-02-10T04:47:07.765382 2021-02-10T04:47:09.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2040 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:09.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:47:09.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[29319]: 02-10T04:47:07.765385+0000 osd.4 (osd.4) 2041 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:09.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[29319]: 2021-02-10T04:47:09.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[29319]: cluster 2021-02-10T04:47:09.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[29319]: 2021-02-10T04:47:07.914055+0000 osd.0 2021-02-10T04:47:09.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[29319]: (osd.0) 135 : cluster [DBG] 6.0 scrub ok 2021-02-10T04:47:09.634 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:47:09.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[29319]: -02-10T04:47:08.041012+0000 osd.5 2021-02-10T04:47:09.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[29319]: (osd.5) 125 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:47:09.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:47:09.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[29319]: 02-10T04:47:08.044262+0000 osd.5 ( 2021-02-10T04:47:09.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[29319]: osd.5) 126 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:47:09.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:47:09.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[29319]: 10T04:47:08.401795+0000 mon.a (mon.0 2021-02-10T04:47:09.635 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:08 smithi085 conmon[29319]: ) 2126 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 157 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:47:09.636 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:07 smithi164 conmon[29288]: cluster 2021-02-10T04:47:06.332488+0000 mgr.y ( 2021-02-10T04:47:09.636 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:07 smithi164 conmon[29288]: mgr.14140) 1044 : cluster [DBG] pgmap v1817: 188 pgs: 188 active+clean; 889 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:09.636 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:07 smithi164 conmon[29288]: cluster 2021-02-10T04:47:06.812288+0000 osd.4 (osd.4) 2034 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:09.636 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:07 smithi164 conmon[29288]: 2021-02-10T04:47:09.636 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:07 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:47:09.636 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:07 smithi164 conmon[29288]: -02-10T04:47:06.812297+0000 osd.4 (osd.4 2021-02-10T04:47:09.636 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:07 smithi164 conmon[29288]: ) 2035 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:09.637 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:07 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:47:09.637 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:07 smithi164 conmon[29288]: -10T04:47:06.812303+0000 osd.4 ( 2021-02-10T04:47:09.637 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:07 smithi164 conmon[29288]: osd.4) 2036 : cluster 2021-02-10T04:47:09.637 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:07 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:09.637 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:07 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:47:09.637 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:07 smithi164 conmon[29288]: 47:06.812309+0000 osd.4 (osd.4) 2021-02-10T04:47:09.637 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:07 smithi164 conmon[29288]: 2037 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:09.637 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:07 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:47:09.637 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:07 smithi164 conmon[29288]: T04:47:06.861803+0000 osd.5 (osd 2021-02-10T04:47:09.638 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:07 smithi164 conmon[29288]: .5) 124 : cluster [DBG] 7.e deep-scrub ok 2021-02-10T04:47:09.638 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: cluster 2021-02-10T04:47:09.638 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: 2021-02-10T04:47:09.638 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: 2021 2021-02-10T04:47:09.638 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: - 2021-02-10T04:47:09.638 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: 02 2021-02-10T04:47:09.638 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: - 2021-02-10T04:47:09.639 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: 10 2021-02-10T04:47:09.639 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: T 2021-02-10T04:47:09.639 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: 04 2021-02-10T04:47:09.639 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: : 2021-02-10T04:47:09.639 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: 47 2021-02-10T04:47:09.639 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: : 2021-02-10T04:47:09.639 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: 07.765369+0000 osd.4 2021-02-10T04:47:09.639 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: (osd.4) 2038 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:09.639 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: cluster 2021-02-10T04:47:09.640 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: 2021-02-10T04:47:07.765378+0000 2021-02-10T04:47:09.640 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: osd.4 (osd.4) 2039 : cluster [WRN] 2021-02-10T04:47:09.640 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:09.640 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: cluster 2021-02-10T04:47:07 2021-02-10T04:47:09.640 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: .765382+0000 osd.4 (osd.4) 2040 : cluster 2021-02-10T04:47:09.640 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:09.641 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: cluster 2021-02-10T04:47: 2021-02-10T04:47:09.641 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: 07.765385+0000 osd.4 (osd.4) 2041 : 2021-02-10T04:47:09.641 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:09.641 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:47:09.641 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: 47:07.914055+0000 osd.0 (osd.0) 2021-02-10T04:47:09.641 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: 135 : cluster [DBG] 6.0 scrub ok 2021-02-10T04:47:09.641 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:47:09.641 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: 04:47:08.041012+0000 osd.5 (osd 2021-02-10T04:47:09.641 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: .5) 125 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:47:09.641 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:47:09.642 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: 02-10T04:47:08.044262+0000 osd.5 2021-02-10T04:47:09.642 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: (osd.5) 126 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:47:09.642 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:47:09.642 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: 02-10T04:47:08.401795+0000 mon.a (mon 2021-02-10T04:47:09.642 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[29288]: .0) 2126 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 157 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:47:09.642 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:07 smithi164 conmon[33677]: debug 2021-02-10T04:47:07.764+0000 7fbec19a1700 -1 osd.4 1063 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:09.643 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[33677]: debug 2021-02-10T04:47:08.781+0000 7fbec19a1700 -1 osd.4 1063 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:09.643 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:47:08 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:47:09.998 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:47:10.135 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:47:10.143 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[45443]: debug 2021-02-10T04:47:09.852+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:47:10.144 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[29319]: cluster 2021-02-10T04:47:08.333094+0000 2021-02-10T04:47:10.144 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[29319]: mgr.y (mgr.14140) 1045 : cluster [DBG] pgmap v1818: 188 pgs: 188 active+clean; 889 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:10.144 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[29319]: cluster 2021-02-10T04:47:08. 2021-02-10T04:47:10.144 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[29319]: 700062+0000 osd.5 (osd.5) 127 : cluster [DBG] 2.15 scrub ok 2021-02-10T04:47:10.144 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:47:10.144 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[29319]: 02-10T04:47:08.782544+0000 osd.4 (osd.4) 2042 : cluster 2021-02-10T04:47:10.144 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:10.144 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:47:10.144 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[29319]: -02-10T04:47:08.782554+0000 osd.4 (osd.4) 2043 2021-02-10T04:47:10.144 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:10.144 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:47:10.144 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[29319]: :47:08.782561+0000 osd.4 (osd.4) 2044 2021-02-10T04:47:10.145 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:10.145 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[29319]: cluster 2021-02-10T04:47:08.782567+0000 osd.4 (osd 2021-02-10T04:47:10.145 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[29319]: .4) 2045 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:10.145 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[29319]: cluster 2021-02-10T04:47:08.784587+0000 2021-02-10T04:47:10.145 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[29319]: osd.4 (osd.4) 2046 : cluster [DBG] 2.f scrub ok 2021-02-10T04:47:10.145 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[29319]: audit 2021-02-10T04:47:08.803045+0000 2021-02-10T04:47:10.145 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[29319]: mgr.y (mgr.14140) 1046 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:47:10.145 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[29319]: 2021-02-10T04:47:10.145 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:47:10.145 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[29319]: -02-10T04:47:08.945799 2021-02-10T04:47:10.145 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2127 2021-02-10T04:47:10.145 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[29319]: : cluster [DBG] osdmap e1064: 8 total, 8 up, 7 in 2021-02-10T04:47:10.145 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:47:10.146 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[29319]: T04:47:09.608877+0000 osd.3 ( 2021-02-10T04:47:10.146 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[29319]: osd.3) 129 : cluster [DBG] 4.19 scrub ok 2021-02-10T04:47:10.146 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[35961]: cluster 2021-02-10T04:47:08.333094 2021-02-10T04:47:10.146 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1045 : cluster [DBG] pgmap v1818: 188 pgs: 188 active+clean; 889 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:10.146 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:47:10.146 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[35961]: 47:08.700062+0000 osd.5 (osd.5) 127 2021-02-10T04:47:10.146 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[35961]: : cluster [DBG] 2.15 scrub ok 2021-02-10T04:47:10.146 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:47:10.146 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[35961]: -10T04:47:08.782544+0000 osd.4 (osd 2021-02-10T04:47:10.146 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[35961]: .4) 2042 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:10.146 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:47:10.146 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[35961]: :47:08.782554+0000 osd.4 (osd.4) 2043 : 2021-02-10T04:47:10.151 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:10.152 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:47:10.152 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[35961]: 04:47:08.782561+0000 osd.4 (osd 2021-02-10T04:47:10.152 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[35961]: .4) 2044 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:10.152 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:47:10.152 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[35961]: 02-10T04:47:08.782567+0000 2021-02-10T04:47:10.152 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[35961]: osd.4 (osd.4) 2045 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:10.152 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[35961]: cluster 2021-02-10T04:47:10.152 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[35961]: 2021-02-10T04:47:08.784587+0000 2021-02-10T04:47:10.152 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[35961]: osd.4 (osd.4) 2046 : cluster [DBG] 2021-02-10T04:47:10.152 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[35961]: 2.f scrub ok 2021-02-10T04:47:10.153 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:47:10.153 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[35961]: :47:08.803045+0000 mgr.y (mgr.14140) 2021-02-10T04:47:10.153 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[35961]: 1046 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:47:10.153 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:47:10.153 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[35961]: 10T04:47:08.945799+0000 mon.a (mon 2021-02-10T04:47:10.153 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[35961]: .0) 2127 : cluster [DBG] osdmap e1064: 8 total, 8 up, 7 in 2021-02-10T04:47:10.153 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[35961]: cluster 2021-02-10T04:47:09 2021-02-10T04:47:10.153 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[35961]: .608877+0000 osd.3 (osd.3) 129 : cluster [DBG] 2021-02-10T04:47:10.153 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:09 smithi085 conmon[35961]: 4.19 scrub ok 2021-02-10T04:47:10.154 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[33677]: debug 2021-02-10T04:47:10.154 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[33677]: 2021-02-10T04:47:09.825+0000 7fbec19a1700 -1 osd.4 1064 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:10.154 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[29288]: cluster 2021-02-10T04:47:08.333094 2021-02-10T04:47:10.154 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1045 : cluster [DBG] pgmap v1818: 188 pgs: 188 active+clean; 889 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:10.155 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:47:10.155 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[29288]: 47:08.700062+0000 osd.5 (osd.5) 127 : cluster [DBG] 2.15 scrub ok 2021-02-10T04:47:10.155 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:47:10.155 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[29288]: T04:47:08.782544+0000 osd.4 (osd. 2021-02-10T04:47:10.155 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[29288]: 4) 2042 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:10.155 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:47:10.155 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[29288]: 10T04:47:08.782554+0000 osd.4 2021-02-10T04:47:10.155 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[29288]: (osd.4) 2043 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:10.155 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:47:10.155 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[29288]: 02-10T04:47:08.782561 2021-02-10T04:47:10.155 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2044 : cluster 2021-02-10T04:47:10.156 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:10.156 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:47:10.156 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[29288]: 04:47:08.782567+0000 osd.4 ( 2021-02-10T04:47:10.156 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[29288]: osd.4) 2045 : cluster [WRN] 2021-02-10T04:47:10.156 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:10.156 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[29288]: cluster 2021-02-10T04:47:10.156 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[29288]: 2021-02-10T04:47:08.784587 2021-02-10T04:47:10.156 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2046 : cluster 2021-02-10T04:47:10.156 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[29288]: [DBG] 2.f scrub ok 2021-02-10T04:47:10.156 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[29288]: audit 2021-02-10T04:47:08. 2021-02-10T04:47:10.157 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[29288]: 803045+0000 mgr.y (mgr.14140) 1046 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:47:10.157 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:47:10.157 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[29288]: 10T04:47:08.945799+0000 mon.a 2021-02-10T04:47:10.157 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[29288]: (mon.0) 2127 : cluster [DBG] osdmap e1064: 8 total, 8 up, 7 in 2021-02-10T04:47:10.157 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:47:10.157 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[29288]: :47:09.608877+0000 osd.3 (osd.3) 129 : 2021-02-10T04:47:10.157 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:09 smithi164 conmon[29288]: cluster [DBG] 4.19 scrub ok 2021-02-10T04:47:10.178 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:47:10.393 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[49106]: debug 2021-02-10T04:47:10.393 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[49106]: 2021-02-10T04:47:10.391+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:47:10.423 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:47:10.530 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:47:10.573 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:47:11.143 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:10 smithi164 conmon[33677]: debug 2021-02-10T04:47:11.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:10 smithi164 conmon[33677]: 2021-02-10T04:47:10.777+0000 7fbec19a1700 -1 osd.4 1064 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:11.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:10 smithi164 conmon[29288]: cluster 2021-02-10T04:47:09.826151+0000 2021-02-10T04:47:11.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:10 smithi164 conmon[29288]: osd.4 (osd.4) 2047 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:11.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:10 smithi164 conmon[29288]: cluster 2021-02-10T04:47:09.826160+0000 2021-02-10T04:47:11.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:10 smithi164 conmon[29288]: osd.4 (osd.4) 2048 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:11.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:10 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:47:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:10 smithi164 conmon[29288]: T04:47:09.826168+0000 osd.4 ( 2021-02-10T04:47:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:10 smithi164 conmon[29288]: osd.4) 2049 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:10 smithi164 conmon[29288]: cluster 2021-02-10T04:47:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:10 smithi164 conmon[29288]: 2021-02-10T04:47:09.826174+0000 2021-02-10T04:47:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:10 smithi164 conmon[29288]: osd.4 (osd.4) 2050 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:10 smithi164 conmon[29288]: 2021-02-10T04:47:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:10 smithi164 conmon[29288]: cluster 2021-02-10T04:47:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:10 smithi164 conmon[29288]: 2021-02-10T04:47:09.869772+0000 osd.6 ( 2021-02-10T04:47:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:10 smithi164 conmon[29288]: osd.6) 126 : cluster [DBG] 1.7 scrub ok 2021-02-10T04:47:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:10 smithi164 conmon[29288]: cluster 2021-02-10T04:47:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:10 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:47:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:10 smithi164 conmon[29288]: :47:09.922783+0000 osd.0 (osd.0) 136 2021-02-10T04:47:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:10 smithi164 conmon[29288]: : cluster [DBG] 6.9 scrub ok 2021-02-10T04:47:11.251 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[29319]: cluster 2021-02-10T04:47:11.251 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:47:11.251 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[29319]: 04:47:09.826151+0000 osd.4 (osd.4) 2047 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:11.251 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:47:11.251 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[29319]: 10T04:47:09.826160+0000 osd.4 (osd.4) 2048 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:11.251 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[29319]: 2021-02-10T04:47:11.251 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[29319]: cluster 2021-02-10T04:47:09.826168+0000 2021-02-10T04:47:11.252 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[29319]: osd.4 (osd.4) 2049 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:11.252 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:47:11.252 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[29319]: T04:47:09.826174+0000 osd.4 (osd. 2021-02-10T04:47:11.252 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[29319]: 4) 2050 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:11.252 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:47:11.252 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[29319]: T04:47:09.869772+0000 osd.6 (osd.6) 2021-02-10T04:47:11.252 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[29319]: 126 : cluster [DBG] 1.7 scrub ok 2021-02-10T04:47:11.252 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:47:11.252 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[29319]: 47:09.922783+0000 osd.0 (osd.0) 136 2021-02-10T04:47:11.252 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[29319]: : cluster [DBG] 6.9 scrub ok 2021-02-10T04:47:11.252 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[35961]: cluster 2021-02-10T04:47 2021-02-10T04:47:11.252 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[35961]: :09.826151+0000 osd.4 (osd.4) 2047 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:11.253 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[35961]: cluster 2021-02-10T04:47:09.826160+0000 osd.4 (osd.4) 2048 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:11.253 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:47:11.253 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[35961]: 47:09.826168+0000 2021-02-10T04:47:11.253 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[35961]: osd.4 (osd.4) 2049 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:11.253 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:47:11.253 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[35961]: -02-10T04:47:09. 2021-02-10T04:47:11.253 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[35961]: 826174+0000 osd.4 (osd.4) 2050 : 2021-02-10T04:47:11.253 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:11.253 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:47:11.253 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[35961]: 02-10T04:47:09. 2021-02-10T04:47:11.253 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[35961]: 869772+0000 osd.6 (osd.6) 126 : 2021-02-10T04:47:11.253 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[35961]: cluster [DBG] 1.7 scrub ok 2021-02-10T04:47:11.254 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:47:11.254 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[35961]: :47:09.922783+0000 osd.0 (osd.0 2021-02-10T04:47:11.254 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[35961]: ) 136 : cluster [DBG] 6.9 scrub ok 2021-02-10T04:47:11.254 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:47:10 smithi085 conmon[41540]: debug 2021-02-10T04:47:10.984+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:47:11.972 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:47:12.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:11 smithi164 conmon[33677]: debug 2021-02-10T04:47:12.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:11 smithi164 conmon[33677]: 2021-02-10T04:47:11.806+0000 7fbec19a1700 -1 osd.4 1064 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:12.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:11 smithi164 conmon[33677]: 2021-02-10T04:47:12.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:11 smithi164 conmon[29288]: cluster 2021-02-10T04:47:10.333797+0000 mgr.y (mgr.14140) 1047 : cluster [DBG] pgmap v1820: 188 pgs: 188 active+clean; 889 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:12.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:11 smithi164 conmon[29288]: cluster 2021-02-10T04:47:10.777996+0000 osd.4 (osd.4) 2051 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:12.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:11 smithi164 conmon[29288]: cluster 2021-02-10T04:47:10.778008+0000 osd.4 (osd.4) 2052 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:12.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:11 smithi164 conmon[29288]: cluster 2021-02-10T04:47:10.778015+0000 osd.4 (osd. 2021-02-10T04:47:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:11 smithi164 conmon[29288]: 4) 2053 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:11 smithi164 conmon[29288]: cluster 2021-02-10T04:47:10.778021+0000 2021-02-10T04:47:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:11 smithi164 conmon[29288]: osd.4 (osd.4) 2054 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:11 smithi164 conmon[29288]: 2021-02-10T04:47:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:11 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:47:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:11 smithi164 conmon[29288]: -10T04:47:10.907524+0000 osd.0 (osd.0) 137 2021-02-10T04:47:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:11 smithi164 conmon[29288]: : cluster [DBG] 3.1 deep-scrub ok 2021-02-10T04:47:12.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[29319]: cluster 2021-02-10T04:47:10.333797+0000 mgr.y (mgr. 2021-02-10T04:47:12.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[29319]: 14140) 1047 : cluster [DBG] pgmap v1820: 188 pgs: 188 active+clean; 889 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:12.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[29319]: cluster 2021-02-10T04:47:10.777996+0000 osd.4 (osd.4) 2051 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:12.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[29319]: cluster 2021-02-10T04:47:10.778008+0000 osd.4 (osd.4) 2052 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:12.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[29319]: cluster 2021-02-10T04:47:10.778015+0000 osd.4 (osd.4) 2053 : cluster [WRN] 2021-02-10T04:47:12.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:12.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[29319]: cluster 2021-02-10T04:47:12.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[29319]: 2021-02-10T04:47:10.778021+0000 osd.4 (osd 2021-02-10T04:47:12.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[29319]: .4) 2054 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:12.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:47:12.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[29319]: 10T04:47:10.907524+0000 osd.0 ( 2021-02-10T04:47:12.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[29319]: osd.0) 137 : cluster [DBG] 3.1 deep-scrub ok 2021-02-10T04:47:12.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[35961]: cluster 2021-02-10T04:47:12.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[35961]: 2021-02-10T04:47 2021-02-10T04:47:12.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[35961]: :10.333797+0000 mgr.y (mgr.14140) 1047 : cluster [DBG] pgmap v1820: 188 pgs: 188 active+clean; 889 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:12.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[35961]: cluster 2021-02-10T04:47 2021-02-10T04:47:12.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[35961]: :10.777996+0000 osd.4 (osd.4 2021-02-10T04:47:12.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[35961]: ) 2051 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:12.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[35961]: cluster 2021-02-10T04:47:12.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[35961]: 2021-02-10T04:47:10 2021-02-10T04:47:12.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[35961]: .778008+0000 osd.4 (osd.4) 2052 : cluster [WRN] 2021-02-10T04:47:12.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:12.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[35961]: cluster 2021-02-10T04:47:12.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[35961]: 2021-02-10T04:47:10.778015+0000 osd.4 (osd.4 2021-02-10T04:47:12.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[35961]: ) 2053 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:12.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:47:12.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[35961]: :47:10.778021+0000 osd.4 (osd.4) 2021-02-10T04:47:12.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[35961]: 2054 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:12.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[35961]: cluster 2021-02-10T04:47: 2021-02-10T04:47:12.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[35961]: 10.907524+0000 osd.0 (osd.0) 137 : cluster 2021-02-10T04:47:12.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:11 smithi085 conmon[35961]: [DBG] 3.1 deep-scrub ok 2021-02-10T04:47:13.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:12 smithi164 conmon[33677]: debug 2021-02-10T04:47:12.797+0000 7fbec19a1700 -1 osd.4 1064 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:12 smithi164 conmon[29288]: cluster 2021-02-10T04:47:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:12 smithi164 conmon[29288]: 2021-02-10T04:47:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:12 smithi164 conmon[29288]: 2021 2021-02-10T04:47:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:12 smithi164 conmon[29288]: - 2021-02-10T04:47:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:12 smithi164 conmon[29288]: 02 2021-02-10T04:47:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:12 smithi164 conmon[29288]: - 2021-02-10T04:47:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:12 smithi164 conmon[29288]: 10 2021-02-10T04:47:13.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:12 smithi164 conmon[29288]: T 2021-02-10T04:47:13.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:12 smithi164 conmon[29288]: 04:47:11.806729+0000 osd.4 ( 2021-02-10T04:47:13.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:12 smithi164 conmon[29288]: osd.4) 2055 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:13.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:12 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:47:13.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:12 smithi164 conmon[29288]: 04:47:11. 2021-02-10T04:47:13.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:12 smithi164 conmon[29288]: 806741+0000 osd.4 (osd.4) 2056 : cluster [WRN] 2021-02-10T04:47:13.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:12 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:13.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:12 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:47:13.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:12 smithi164 conmon[29288]: :47:11.806748+0000 osd.4 (osd.4) 2057 2021-02-10T04:47:13.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:12 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:13.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:12 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:47:13.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:12 smithi164 conmon[29288]: 10T04:47:11.806755+0000 osd.4 (osd. 2021-02-10T04:47:13.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:12 smithi164 conmon[29288]: 4) 2058 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:13.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:12 smithi164 conmon[29288]: cluster 2021-02-10T04:47 2021-02-10T04:47:13.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:12 smithi164 conmon[29288]: :11.808545+0000 osd.4 (osd.4) 2059 : cluster 2021-02-10T04:47:13.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:12 smithi164 conmon[29288]: [DBG] 2.26 scrub ok 2021-02-10T04:47:13.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:12 smithi085 conmon[29319]: cluster 2021-02-10T04:47:11.806729+0000 osd.4 (osd.4) 2055 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:12 smithi085 conmon[29319]: cluster 2021-02-10T04:47:11.806741+0000 osd.4 (osd.4) 2056 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:12 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:47:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:12 smithi085 conmon[29319]: -02-10T04:47:11.806748+0000 osd.4 (osd.4) 2057 : cluster 2021-02-10T04:47:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:12 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:12 smithi085 conmon[29319]: cluster 2021-02-10T04:47:11. 2021-02-10T04:47:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:12 smithi085 conmon[29319]: 806755+0000 osd.4 (osd.4) 2058 : cluster 2021-02-10T04:47:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:12 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:12 smithi085 conmon[29319]: cluster 2021-02-10T04:47:11 2021-02-10T04:47:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:12 smithi085 conmon[29319]: .808545+0000 osd.4 (osd.4) 2059 : cluster 2021-02-10T04:47:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:12 smithi085 conmon[29319]: [DBG] 2.26 scrub ok 2021-02-10T04:47:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:12 smithi085 conmon[35961]: cluster 2021-02-10T04:47:11.806729+0000 osd.4 ( 2021-02-10T04:47:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:12 smithi085 conmon[35961]: osd.4) 2055 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:12 smithi085 conmon[35961]: cluster 2021-02-10T04:47:11.806741+0000 osd.4 (osd.4) 2056 : 2021-02-10T04:47:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:12 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:12 smithi085 conmon[35961]: cluster 2021-02-10T04:47:11.806748+0000 2021-02-10T04:47:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:12 smithi085 conmon[35961]: osd.4 (osd.4) 2057 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:12 smithi085 conmon[35961]: cluster 2021-02-10T04:47:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:12 smithi085 conmon[35961]: 2021-02-10T04:47:11.806755+0000 osd.4 (osd.4) 2058 : cluster 2021-02-10T04:47:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:12 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:12 smithi085 conmon[35961]: cluster 2021-02-10T04:47:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:12 smithi085 conmon[35961]: 2021-02-10T04:47:11.808545+0000 osd.4 (osd. 2021-02-10T04:47:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:12 smithi085 conmon[35961]: 4) 2059 : cluster [DBG] 2.26 scrub ok 2021-02-10T04:47:14.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[33677]: debug 2021-02-10T04:47:13.779+0000 7fbec19a1700 -1 osd.4 1064 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: cluster 2021-02-10T04:47:12.334351+0000 mgr.y (mgr. 2021-02-10T04:47:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: 14140) 1048 : cluster [DBG] pgmap v1821: 188 pgs: 188 active+clean; 889 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: cluster 2021-02-10T04:47:12.728015+0000 osd.3 (osd.3) 130 : cluster [DBG] 4.10 scrub ok 2021-02-10T04:47:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: cluster 2021-02-10T04:47:12 2021-02-10T04:47:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: .797973+0000 osd.4 (osd.4) 2060 : cluster [WRN] 2021-02-10T04:47:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:47:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: 10T04:47:12.797982+0000 osd.4 (osd. 2021-02-10T04:47:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: 4) 2061 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: cluster 2021-02-10T04:47:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: 2021-02-10T04:47:12.797988+0000 2021-02-10T04:47:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: osd.4 (osd.4) 2062 : cluster 2021-02-10T04:47:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: cluster 2021-02-10T04:47:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:47:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: 47:12.797994+0000 osd.4 (osd.4) 2063 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: cluster 2021-02-10T04:47:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:47:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: 47:12.800140+0000 osd.4 (osd.4) 2021-02-10T04:47:14.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: 2064 : cluster [DBG] 6.1a scrub ok 2021-02-10T04:47:14.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:47:14.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: -10T04:47:12.944339+0000 osd.0 (osd.0) 2021-02-10T04:47:14.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: 138 : cluster [DBG] 6.29 scrub ok 2021-02-10T04:47:14.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: cluster 2021-02-10T04:47:14.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: 2021-02-10T04:47:13. 2021-02-10T04:47:14.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: 032499+0000 osd.6 2021-02-10T04:47:14.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: (osd.6) 127 : cluster [DBG] 7.12 scrub ok 2021-02-10T04:47:14.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:47:14.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: -10T04:47:13.403627+0000 mon.a (mon 2021-02-10T04:47:14.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: .0) 2128 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 167 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:47:14.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: audit 2021-02-10T04:47:14.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: 2021-02-10T04:47:13.461468+0000 2021-02-10T04:47:14.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: mon.c (mon.1) 77 : audit [INF] 2021-02-10T04:47:14.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/1083684787' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.1c", "id": [6, 5, 0]}]: dispatch 2021-02-10T04:47:14.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: audit 2021-02-10T04:47: 2021-02-10T04:47:14.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: 13.461923+0000 mon.a (mon.0) 2129 : 2021-02-10T04:47:14.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:13 smithi164 conmon[29288]: audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.1c", "id": [6, 5, 0]}]: dispatch 2021-02-10T04:47:14.417 INFO:teuthology.orchestra.run.smithi085.stderr:set 4.1c pg_upmap mapping to [6,5,0] 2021-02-10T04:47:14.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: cluster 2021-02-10T04:47:14.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:47:14.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: 04:47:12.334351+0000 mgr.y (mgr.14140) 1048 : cluster [DBG] pgmap v1821: 188 pgs: 188 active+clean; 889 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:14.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: cluster 2021-02-10T04:47:14.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: 2021-02-10T04:47: 2021-02-10T04:47:14.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: 12.728015+0000 osd.3 (osd.3) 130 : cluster 2021-02-10T04:47:14.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: [DBG] 4.10 scrub ok 2021-02-10T04:47:14.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:47:14.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: 02-10T04:47:12.797973+0000 osd.4 ( 2021-02-10T04:47:14.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: osd.4) 2060 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:14.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: cluster 2021-02-10T04:47:14.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:47:14.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: T04:47:12.797982+0000 osd.4 (osd.4 2021-02-10T04:47:14.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: ) 2061 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:14.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: cluster 2021-02-10T04:47:14.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:47:14.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: :47:12.797988+0000 osd.4 (osd.4 2021-02-10T04:47:14.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: ) 2062 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:14.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:47:14.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: -02-10T04:47:12.797994+0000 2021-02-10T04:47:14.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: osd.4 (osd.4) 2063 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:14.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: 2021-02-10T04:47:14.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: cluster 2021-02-10T04:47:14.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:47:14.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: 04:47:12.800140+0000 2021-02-10T04:47:14.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: osd.4 (osd.4) 2064 2021-02-10T04:47:14.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: : cluster [DBG] 6.1a scrub ok 2021-02-10T04:47:14.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:47:14.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: -02-10T04 2021-02-10T04:47:14.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: :47:12.944339+0000 osd.0 2021-02-10T04:47:14.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: (osd.0) 138 : cluster [DBG] 6.29 scrub ok 2021-02-10T04:47:14.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:47:14.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: 02-10T04:47:13.032499+0000 osd.6 (osd 2021-02-10T04:47:14.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: .6) 127 : cluster [DBG] 7.12 scrub ok 2021-02-10T04:47:14.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:47:14.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: -02-10T04:47:13.403627+0000 2021-02-10T04:47:14.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: mon.a (mon.0) 2128 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 167 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:47:14.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: audit 2021-02-10T04:47:14.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: 2021-02-10T04:47:13.461468+0000 2021-02-10T04:47:14.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: mon.c (mon.1) 77 : audit [INF] 2021-02-10T04:47:14.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/1083684787' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.1c", "id": [6, 5, 0]}]: dispatch 2021-02-10T04:47:14.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: audit 2021-02-10T04:47:13. 2021-02-10T04:47:14.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: 461923+0000 mon.a (mon.0) 2129 : audit [INF] 2021-02-10T04:47:14.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[35961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.1c", "id": [6, 5, 0]}]: dispatch 2021-02-10T04:47:14.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: cluster 2021-02-10T04:47:14.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: 2021-02-10T04:47:14.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: 2021-02-10T04:47:12. 2021-02-10T04:47:14.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: 334351+0000 mgr.y (mgr.14140) 1048 : cluster [DBG] pgmap v1821: 188 pgs: 188 active+clean; 889 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:14.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:47:14.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: 47:12.728015+0000 osd.3 (osd.3) 130 : cluster 2021-02-10T04:47:14.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: [DBG] 4.10 scrub ok 2021-02-10T04:47:14.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:47:14.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: 02-10T04:47:12.797973+0000 osd.4 ( 2021-02-10T04:47:14.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: osd.4) 2060 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:14.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:47:14.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: 02-10T04:47:12.797982+0000 2021-02-10T04:47:14.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: osd.4 (osd.4) 2061 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:14.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: cluster 2021-02-10T04:47:14.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: 2021-02-10T04:47:12.797988+0000 2021-02-10T04:47:14.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: osd.4 (osd.4) 2062 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:14.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:47:14.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: :47:12.797994+0000 osd.4 (osd.4 2021-02-10T04:47:14.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: ) 2063 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:14.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:47:14.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: -02-10T04:47:12.800140+0000 2021-02-10T04:47:14.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: osd.4 (osd.4) 2064 : cluster 2021-02-10T04:47:14.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: [DBG] 6.1a scrub ok 2021-02-10T04:47:14.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:47:14.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: 02-10T04:47:12 2021-02-10T04:47:14.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: .944339+0000 osd.0 ( 2021-02-10T04:47:14.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: osd.0) 138 : cluster [DBG] 6.29 scrub ok 2021-02-10T04:47:14.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:47:14.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: -10T04:47:13.032499+0000 osd.6 (osd.6 2021-02-10T04:47:14.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: ) 127 : cluster [DBG] 7.12 scrub ok 2021-02-10T04:47:14.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: cluster 2021-02-10T04:47:13.403627 2021-02-10T04:47:14.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2128 : cluster [WRN] 2021-02-10T04:47:14.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: Health check update: 4 slow ops, oldest one blocked for 167 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:47:14.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: audit 2021 2021-02-10T04:47:14.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: -02-10T04:47:13.461468+0000 2021-02-10T04:47:14.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: mon.c (mon.1) 77 : audit [INF] from='client.? 172.21.15.85:0/1083684787' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.1c", "id": [6, 5, 0]}]: dispatch 2021-02-10T04:47:14.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: audit 2021-02-10T04: 2021-02-10T04:47:14.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:13 smithi085 conmon[29319]: 47:13.461923+0000 mon.a (mon.0) 2129 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.1c", "id": [6, 5, 0]}]: dispatch 2021-02-10T04:47:14.979 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[35961]: cluster 2021-02-10T04:47:13.579156+0000 2021-02-10T04:47:14.979 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[35961]: osd.5 (osd.5) 128 : cluster [DBG] 2.2 scrub ok 2021-02-10T04:47:14.980 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[35961]: cluster 2021-02-10T04:47:14.980 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[35961]: 2021-02-10T04:47:13.727233+0000 2021-02-10T04:47:14.980 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[35961]: osd.3 (osd.3) 131 : cluster [DBG] 2021-02-10T04:47:14.980 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[35961]: 6.f deep-scrub ok 2021-02-10T04:47:14.980 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[35961]: 2021-02-10T04:47:14.980 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:47:14.980 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[35961]: :47:13.779872+0000 osd.4 (osd. 2021-02-10T04:47:14.980 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[35961]: 4) 2065 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:14.980 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:47:14.980 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[35961]: 02-10T04:47:13.779881+0000 osd.4 ( 2021-02-10T04:47:14.980 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[35961]: osd.4) 2066 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:14.981 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:47:14.981 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[35961]: 02-10T04:47:13.779888+0000 2021-02-10T04:47:14.981 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[35961]: osd.4 (osd.4) 2067 : cluster [WRN] 2021-02-10T04:47:14.981 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:14.981 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[35961]: cluster 2021-02-10T04:47:14.981 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[35961]: 2021-02-10T04:47:13.779894+0000 osd.4 ( 2021-02-10T04:47:14.981 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[35961]: osd.4) 2068 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:14.981 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[35961]: cluster 2021-02-10T04:47 2021-02-10T04:47:14.981 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[35961]: :13.991392+0000 osd.6 (osd.6) 128 : cluster [DBG] 2021-02-10T04:47:14.981 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[35961]: 6.5 scrub ok 2021-02-10T04:47:14.981 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[35961]: audit 2021 2021-02-10T04:47:14.981 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[35961]: -02- 2021-02-10T04:47:14.982 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[35961]: 10T04:47:14.411114+0000 mon.a ( 2021-02-10T04:47:14.982 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[35961]: mon.0) 2130 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.1c", "id": [6, 5, 0]}]': finished 2021-02-10T04:47:14.982 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[35961]: cluster 2021-02-10T04:47: 2021-02-10T04:47:14.982 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[35961]: 14.411218+0000 mon.a (mon.0) 2131 : cluster [DBG] osdmap e1065: 8 total, 8 up, 7 in 2021-02-10T04:47:14.982 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[29319]: cluster 2021-02-10T04:47:14.982 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[29319]: 2021-02-10T04:47: 2021-02-10T04:47:14.982 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[29319]: 13.579156+0000 osd.5 (osd.5) 128 : cluster [DBG] 2.2 scrub ok 2021-02-10T04:47:14.982 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:47:14.982 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[29319]: -02-10T04:47:13.727233+0000 osd.3 (osd.3) 131 : cluster [DBG] 6.f deep-scrub ok 2021-02-10T04:47:14.982 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[29319]: 2021-02-10T04:47:14.982 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[29319]: cluster 2021-02-10T04:47:14.983 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[29319]: 2021-02-10T04:47:13.779872+0000 osd.4 (osd. 2021-02-10T04:47:14.983 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[29319]: 4) 2065 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:14.983 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[29319]: cluster 2021-02-10T04:47 2021-02-10T04:47:14.983 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[29319]: :13.779881+0000 osd.4 (osd.4) 2066 : cluster 2021-02-10T04:47:14.983 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:14.983 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[29319]: cluster 2021-02-10T04:47:13.779888 2021-02-10T04:47:14.983 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2067 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:14.983 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:47:14.983 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[29319]: 02-10T04:47:13.779894+0000 osd.4 2021-02-10T04:47:14.983 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[29319]: (osd.4) 2068 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:14.983 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:47:14.983 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[29319]: :47:13.991392+0000 osd.6 (osd.6 2021-02-10T04:47:14.984 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[29319]: ) 128 : cluster [DBG] 6.5 scrub ok 2021-02-10T04:47:14.984 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:47:14.984 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[29319]: :47:14.411114+0000 mon.a ( 2021-02-10T04:47:14.984 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[29319]: mon.0) 2130 : audit [INF] 2021-02-10T04:47:14.984 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[29319]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.1c", "id": [6, 5, 0]}]': finished 2021-02-10T04:47:14.984 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[29319]: cluster 2021-02-10T04:47:14.984 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[29319]: 2021-02-10T04:47:14.411218+0000 mon.a (mon 2021-02-10T04:47:14.984 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[29319]: .0) 2131 : cluster [DBG] osdmap e1065: 8 total, 8 up, 7 in 2021-02-10T04:47:14.984 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:14 smithi085 conmon[29319]: 2021-02-10T04:47:15.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:14 smithi164 conmon[33677]: debug 2021-02-10T04:47:14.751+0000 7fbec19a1700 -1 osd.4 1065 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:14 smithi164 conmon[29288]: cluster 2021-02-10T04:47:13.579156+0000 osd.5 (osd.5) 128 : cluster 2021-02-10T04:47:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:14 smithi164 conmon[29288]: [DBG] 2.2 scrub ok 2021-02-10T04:47:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:14 smithi164 conmon[29288]: cluster 2021-02-10T04:47:13.727233+0000 osd.3 (osd.3) 131 : cluster [DBG] 6.f deep-scrub ok 2021-02-10T04:47:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:14 smithi164 conmon[29288]: cluster 2021-02-10T04:47:13.779872+0000 osd.4 (osd.4) 2065 : 2021-02-10T04:47:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:14 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:14 smithi164 conmon[29288]: cluster 2021-02-10T04:47:13.779881+0000 2021-02-10T04:47:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:14 smithi164 conmon[29288]: osd.4 (osd.4) 2066 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:14 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:47:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:14 smithi164 conmon[29288]: -02-10T04:47:13.779888+0000 2021-02-10T04:47:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:14 smithi164 conmon[29288]: osd.4 (osd.4) 2067 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:14 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:47:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:14 smithi164 conmon[29288]: 02-10T04:47:13.779894+0000 osd.4 2021-02-10T04:47:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:14 smithi164 conmon[29288]: (osd.4) 2068 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:15.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:14 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:47:15.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:14 smithi164 conmon[29288]: -02-10T04:47:13.991392+0000 osd.6 2021-02-10T04:47:15.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:14 smithi164 conmon[29288]: (osd.6) 128 : cluster [DBG] 6.5 scrub ok 2021-02-10T04:47:15.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:14 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:47:15.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:14 smithi164 conmon[29288]: 02-10T04:47:14.411114+0000 mon.a ( 2021-02-10T04:47:15.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:14 smithi164 conmon[29288]: mon.0) 2130 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.1c", "id": [6, 5, 0]}]': finished 2021-02-10T04:47:15.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:14 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:47:15.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:14 smithi164 conmon[29288]: -02-10T04:47:14.411218+0000 mon.a 2021-02-10T04:47:15.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:14 smithi164 conmon[29288]: (mon.0) 2131 : cluster [DBG] osdmap e1065: 8 total, 8 up, 7 in 2021-02-10T04:47:15.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:14 smithi164 conmon[29288]: 2021-02-10T04:47:16.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[33677]: debug 2021-02-10T04:47:16.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[33677]: 2021-02-10T04:47:15.734+0000 7fbec19a1700 -1 osd.4 1066 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:16.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[33677]: 2021-02-10T04:47:16.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[29288]: cluster 2021-02-10T04:47:14.335086+0000 mgr.y (mgr 2021-02-10T04:47:16.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[29288]: .14140) 1049 : cluster [DBG] pgmap v1822: 188 pgs: 188 active+clean; 912 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:16.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[29288]: cluster 2021-02-10T04:47:14.611342+0000 osd.5 (osd.5) 129 : cluster [DBG] 7.0 scrub ok 2021-02-10T04:47:16.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[29288]: cluster 2021-02-10T04:47:14. 2021-02-10T04:47:16.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[29288]: 752449+0000 osd.4 (osd.4) 2069 : cluster [WRN] 2021-02-10T04:47:16.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:16.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[29288]: cluster 2021-02-10T04:47:14. 2021-02-10T04:47:16.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[29288]: 752457+0000 osd.4 (osd.4) 2070 : cluster 2021-02-10T04:47:16.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:16.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[29288]: cluster 2021-02-10T04:47:14.752464 2021-02-10T04:47:16.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2071 : cluster [WRN] 2021-02-10T04:47:16.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:16.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[29288]: cluster 2021-02-10T04:47:14. 2021-02-10T04:47:16.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[29288]: 752469+0000 osd.4 (osd.4) 2072 : cluster 2021-02-10T04:47:16.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:16.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[29288]: cluster 2021-02-10T04:47: 2021-02-10T04:47:16.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[29288]: 14.955538+0000 osd.0 (osd.0) 139 : 2021-02-10T04:47:16.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[29288]: cluster [DBG] 1.2 scrub ok 2021-02-10T04:47:16.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:47:16.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[29288]: :47:15.037657+0000 osd.6 (osd.6 2021-02-10T04:47:16.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[29288]: ) 129 : cluster [DBG] 6.1 scrub ok 2021-02-10T04:47:16.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:47:16.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[29288]: 10T04:47:15.292216+0000 osd.2 (osd.2) 2021-02-10T04:47:16.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[29288]: 1951 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:47:16.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:47:16.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[29288]: T04:47:15.292669+0000 osd.2 (osd.2) 2021-02-10T04:47:16.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[29288]: 1952 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:47:16.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:47:16.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[29288]: 04:47:15.411850+0000 mon.a (mon. 2021-02-10T04:47:16.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:15 smithi164 conmon[29288]: 0) 2132 : cluster [DBG] osdmap e1066: 8 total, 8 up, 7 in 2021-02-10T04:47:16.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[29319]: cluster 2021-02-10T04:47:14.335086+0000 mgr.y 2021-02-10T04:47:16.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[29319]: (mgr.14140) 1049 : cluster [DBG] pgmap v1822: 188 pgs: 188 active+clean; 912 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:16.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[29319]: cluster 2021-02-10T04:47:14.611342+0000 osd.5 (osd.5) 129 : cluster [DBG] 2021-02-10T04:47:16.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[29319]: 7.0 scrub ok 2021-02-10T04:47:16.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:47:16.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[29319]: 10T04:47:14.752449+0000 osd.4 (osd.4) 2069 : cluster 2021-02-10T04:47:16.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:16.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:47:16.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[29319]: 47:14.752457+0000 osd.4 (osd.4) 2070 : 2021-02-10T04:47:16.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:16.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[29319]: cluster 2021-02-10T04:47:14 2021-02-10T04:47:16.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[29319]: .752464+0000 osd.4 (osd.4) 2071 : cluster 2021-02-10T04:47:16.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:16.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[29319]: cluster 2021-02-10T04:47:14.752469 2021-02-10T04:47:16.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2072 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:16.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[29319]: 2021-02-10T04:47:16.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[29319]: cluster 2021-02-10T04:47:16.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[29319]: 2021-02-10T04:47:14.955538+0000 osd.0 (osd.0) 139 : cluster 2021-02-10T04:47:16.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[29319]: [DBG] 1.2 scrub ok 2021-02-10T04:47:16.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[29319]: cluster 2021-02-10T04:47:15.037657+0000 2021-02-10T04:47:16.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[29319]: osd.6 (osd.6) 129 : cluster [DBG] 6.1 scrub ok 2021-02-10T04:47:16.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:47:16.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[29319]: 02-10T04:47:15.292216+0000 osd.2 ( 2021-02-10T04:47:16.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[29319]: osd.2) 1951 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:47:16.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:47:16.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[29319]: 10T04:47:15.292669+0000 osd.2 (osd.2 2021-02-10T04:47:16.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[29319]: ) 1952 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:47:16.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[29319]: cluster 2021-02-10T04:47:16.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[29319]: 2021-02-10T04:47:15.411850+0000 mon.a (mon 2021-02-10T04:47:16.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[29319]: .0) 2132 : cluster [DBG] osdmap e1066: 8 total, 8 up, 7 in 2021-02-10T04:47:16.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: cluster 2021-02-10T04:47:16.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: 2021-02-10T04:47:16.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: 2021 2021-02-10T04:47:16.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: - 2021-02-10T04:47:16.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: 02 2021-02-10T04:47:16.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: -10T04:47:14.335086+0000 mgr.y (mgr.14140) 1049 2021-02-10T04:47:16.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: : cluster [DBG] pgmap v1822: 188 pgs: 188 active+clean; 912 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:16.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:47:16.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: 10T04:47:14.611342+0000 osd.5 (osd.5) 129 : cluster [DBG] 7.0 scrub ok 2021-02-10T04:47:16.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:47:16.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: 47:14.752449+0000 osd.4 (osd.4) 2069 : cluster 2021-02-10T04:47:16.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:16.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: cluster 2021-02-10T04:47 2021-02-10T04:47:16.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: :14.752457+0000 osd.4 (osd.4) 2070 : 2021-02-10T04:47:16.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:16.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:47:16.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: :47:14.752464+0000 osd.4 (osd.4) 2071 2021-02-10T04:47:16.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:16.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: cluster 2021-02-10T04:47: 2021-02-10T04:47:16.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: 14.752469+0000 osd.4 (osd.4) 2072 : cluster [WRN] 2021-02-10T04:47:16.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:16.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: cluster 2021-02-10T04:47:14.955538+0000 2021-02-10T04:47:16.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: osd.0 (osd.0) 139 : cluster [DBG] 1.2 scrub ok 2021-02-10T04:47:16.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: 2021-02-10T04:47:16.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: cluster 2021-02-10T04:47:16.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: 2021-02-10T04:47:15.037657+0000 osd.6 (osd 2021-02-10T04:47:16.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: .6) 129 : cluster [DBG] 6.1 scrub ok 2021-02-10T04:47:16.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:47:16.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: -10T04:47:15.292216+0000 osd.2 ( 2021-02-10T04:47:16.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: osd.2) 1951 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:47:16.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:47:16.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: 10T04:47:15.292669+0000 osd.2 (osd 2021-02-10T04:47:16.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: .2) 1952 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:47:16.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:47:16.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: 10T04:47:15.411850+0000 mon.a (mon. 2021-02-10T04:47:16.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:15 smithi085 conmon[35961]: 0) 2132 : cluster [DBG] osdmap e1066: 8 total, 8 up, 7 in 2021-02-10T04:47:16.979 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:16 smithi164 conmon[33677]: debug 2021-02-10T04:47:16.702+0000 7fbec19a1700 -1 osd.4 1066 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:16.979 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:16 smithi164 conmon[29288]: cluster 2021-02-10T04:47:15.735921 2021-02-10T04:47:16.979 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:16 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2073 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:16.979 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:16 smithi164 conmon[29288]: cluster 2021-02-10T04:47:15.735928+0000 2021-02-10T04:47:16.979 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:16 smithi164 conmon[29288]: osd.4 (osd.4) 2074 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:16.979 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:16 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:47:16.979 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:16 smithi164 conmon[29288]: -10T04:47:15.735942+0000 2021-02-10T04:47:16.979 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:16 smithi164 conmon[29288]: osd.4 (osd.4) 2075 : 2021-02-10T04:47:16.979 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:16 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:16.980 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:16 smithi164 conmon[29288]: cluster 2021-02-10T04:47:15.735949 2021-02-10T04:47:16.980 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:16 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2076 : cluster 2021-02-10T04:47:16.980 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:16 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:16.980 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:16 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:47:16.980 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:16 smithi164 conmon[29288]: -02-10T04:47:15.739694+0000 2021-02-10T04:47:16.980 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:16 smithi164 conmon[29288]: osd.4 (osd.4) 2077 2021-02-10T04:47:16.980 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:16 smithi164 conmon[29288]: : cluster [DBG] 4.13 scrub ok 2021-02-10T04:47:16.980 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:16 smithi164 conmon[29288]: cluster 2021-02-10T04:47:16.980 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:16 smithi164 conmon[29288]: 2021-02-10T04:47:16.980 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:16 smithi164 conmon[29288]: 2021 2021-02-10T04:47:16.980 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:16 smithi164 conmon[29288]: -02-10T04:47:16 2021-02-10T04:47:16.980 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:16 smithi164 conmon[29288]: .010872+0000 osd.6 (osd.6) 130 : cluster [DBG] 6.3 scrub ok 2021-02-10T04:47:17.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[29319]: cluster 2021-02-10T04:47:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[29319]: 2021-02-10T04:47: 2021-02-10T04:47:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[29319]: 15.735921+0000 osd.4 (osd.4) 2073 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:47:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[29319]: 02-10T04:47:15.735928+0000 osd.4 (osd.4) 2074 : cluster 2021-02-10T04:47:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:47:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[29319]: T04:47:15.735942+0000 osd.4 (osd.4) 2075 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:47:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[29319]: -10T04:47:15.735949+0000 osd.4 (osd.4) 2076 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[29319]: cluster 2021-02-10T04:47:15.739694+0000 osd.4 (osd.4 2021-02-10T04:47:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[29319]: ) 2077 : cluster [DBG] 4.13 scrub ok 2021-02-10T04:47:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:47:17.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[29319]: 10T04:47:16.010872+0000 osd.6 (osd.6) 130 : 2021-02-10T04:47:17.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[29319]: cluster [DBG] 6.3 scrub ok 2021-02-10T04:47:17.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[35961]: cluster 2021-02-10T04:47:17.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[35961]: 2021-02-10T04:47:17.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:47:17.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[35961]: :47:15.735921+0000 osd.4 (osd.4) 2073 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:17.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[35961]: 2021-02-10T04:47:17.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:47:17.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[35961]: 47:15.735928+0000 osd.4 (osd.4) 2074 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:17.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[35961]: cluster 2021-02-10T04:47:17.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[35961]: 2021-02-10T04:47: 2021-02-10T04:47:17.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[35961]: 15.735942+0000 osd.4 (osd. 2021-02-10T04:47:17.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[35961]: 4) 2075 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:17.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[35961]: 2021-02-10T04:47:17.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[35961]: cluster 2021-02-10T04:47:17.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:47:17.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[35961]: 47:15.735949+0000 osd.4 2021-02-10T04:47:17.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[35961]: (osd.4) 2076 : cluster 2021-02-10T04:47:17.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:17.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[35961]: cluster 2021-02-10T04:47:17.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[35961]: 2021-02-10T04:47:15. 2021-02-10T04:47:17.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[35961]: 739694+0000 osd.4 (osd.4) 2077 : cluster 2021-02-10T04:47:17.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[35961]: [DBG] 4.13 scrub ok 2021-02-10T04:47:17.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[35961]: cluster 2021-02-10T04:47: 2021-02-10T04:47:17.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[35961]: 16.010872+0000 osd.6 (osd.6) 130 2021-02-10T04:47:17.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:16 smithi085 conmon[35961]: : cluster [DBG] 6.3 scrub ok 2021-02-10T04:47:17.982 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:17 smithi164 conmon[33677]: debug 2021-02-10T04:47:17.982 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:17 smithi164 conmon[33677]: 2021-02-10T04:47:17.653+0000 7fbec19a1700 -1 osd.4 1066 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:17.982 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:17 smithi164 conmon[33677]: 2021-02-10T04:47:17.982 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:17 smithi164 conmon[29288]: cluster 2021-02-10T04:47:16.335488+0000 mgr.y (mgr.14140) 2021-02-10T04:47:17.982 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:17 smithi164 conmon[29288]: 1050 : cluster [DBG] pgmap v1825: 188 pgs: 188 active+clean; 912 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:17.982 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:17 smithi164 conmon[29288]: cluster 2021-02-10T04:47:16.641609+0000 osd.5 (osd.5) 130 : cluster [DBG] 7.17 scrub ok 2021-02-10T04:47:17.983 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:17 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:47:17.983 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:17 smithi164 conmon[29288]: T04:47:16.703930+0000 osd.4 (osd.4) 2078 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:17.983 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:17 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:47:17.983 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:17 smithi164 conmon[29288]: 10T04:47:16.703941+0000 osd.4 (osd.4) 2079 : cluster [WRN] 2021-02-10T04:47:17.983 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:17 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:17.983 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:17 smithi164 conmon[29288]: cluster 2021-02-10T04:47:16 2021-02-10T04:47:17.983 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:17 smithi164 conmon[29288]: .703947+0000 osd.4 (osd.4) 2080 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:17.983 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:17 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:47:17.983 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:17 smithi164 conmon[29288]: -10T04:47:16.703952+0000 osd.4 (osd 2021-02-10T04:47:17.983 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:17 smithi164 conmon[29288]: .4) 2081 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:17.983 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:17 smithi164 conmon[29288]: cluster 2021-02-10T04:47:17.984 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:17 smithi164 conmon[29288]: 2021-02-10T04:47:16.706411 2021-02-10T04:47:17.984 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:17 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2082 : cluster [DBG] 2.27 scrub ok 2021-02-10T04:47:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[29319]: cluster 2021-02-10T04:47:16.335488+0000 mgr.y 2021-02-10T04:47:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[29319]: (mgr.14140) 1050 : cluster [DBG] pgmap v1825: 188 pgs: 188 active+clean; 912 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[29319]: cluster 2021-02-10T04:47:16.641609+0000 osd.5 ( 2021-02-10T04:47:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[29319]: osd.5) 130 : cluster [DBG] 7.17 scrub ok 2021-02-10T04:47:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:47:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[29319]: 02-10T04:47:16.703930+0000 osd.4 (osd 2021-02-10T04:47:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[29319]: .4) 2078 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:47:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[29319]: T04:47:16.703941+0000 osd.4 2021-02-10T04:47:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[29319]: (osd.4) 2079 : cluster 2021-02-10T04:47:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:47:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[29319]: T04:47:16.703947+0000 osd.4 ( 2021-02-10T04:47:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[29319]: osd.4) 2080 : cluster [WRN] 2021-02-10T04:47:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:47:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[29319]: -10T04:47:16.703952+0000 2021-02-10T04:47:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[29319]: osd.4 (osd.4) 2081 2021-02-10T04:47:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[29319]: cluster 2021-02-10T04:47:16 2021-02-10T04:47:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[29319]: .706411+0000 osd.4 (osd.4) 2082 : cluster 2021-02-10T04:47:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[29319]: [DBG] 2.27 scrub ok 2021-02-10T04:47:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: cluster 2021-02-10T04:47:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: 2021-02-10T04:47:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: 2021 2021-02-10T04:47:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: - 2021-02-10T04:47:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: 02 2021-02-10T04:47:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: - 2021-02-10T04:47:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: 10 2021-02-10T04:47:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: T 2021-02-10T04:47:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: 04 2021-02-10T04:47:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: :47:16.335488+0000 mgr.y (mgr.14140) 1050 : cluster [DBG] pgmap v1825: 188 pgs: 188 active+clean; 912 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: 2021-02-10T04:47:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:47:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: T04:47:16.641609+0000 osd.5 (osd.5) 130 : cluster [DBG] 7.17 scrub ok 2021-02-10T04:47:18.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: cluster 2021-02-10T04:47:16.703930+0000 osd.4 ( 2021-02-10T04:47:18.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: osd.4) 2078 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:18.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: cluster 2021-02-10T04:47:18.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: 2021 2021-02-10T04:47:18.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: - 2021-02-10T04:47:18.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: 02 2021-02-10T04:47:18.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: - 2021-02-10T04:47:18.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: 10 2021-02-10T04:47:18.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: T 2021-02-10T04:47:18.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: 04 2021-02-10T04:47:18.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: : 2021-02-10T04:47:18.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: 47 2021-02-10T04:47:18.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: : 2021-02-10T04:47:18.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: 16 2021-02-10T04:47:18.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: . 2021-02-10T04:47:18.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: 703941 2021-02-10T04:47:18.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: +0000 2021-02-10T04:47:18.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: 2021-02-10T04:47:18.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: osd.4 2021-02-10T04:47:18.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: ( 2021-02-10T04:47:18.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: osd 2021-02-10T04:47:18.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: . 2021-02-10T04:47:18.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: 4 2021-02-10T04:47:18.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: ) 2021-02-10T04:47:18.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: 2079 2021-02-10T04:47:18.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: : 2021-02-10T04:47:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: cluster 2021-02-10T04:47:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: 2021-02-10T04:47:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: [WRN] 2021-02-10T04:47:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: 2021-02-10T04:47:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: 2021-02-10T04:47:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: cluster 2021-02-10T04:47:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: 2021-02-10T04:47:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: 2021 2021-02-10T04:47:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: -02-10T04:47:16 2021-02-10T04:47:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: .703947+0000 osd.4 (osd.4) 2080 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: cluster 2021-02-10T04:47:16. 2021-02-10T04:47:18.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: 703952+0000 osd.4 (osd.4) 2081 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:18.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: 2021-02-10T04:47:18.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: cluster 2021-02-10T04:47:18.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: 2021-02-10T04:47:16.706411+0000 osd.4 (osd 2021-02-10T04:47:18.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:17 smithi085 conmon[35961]: .4) 2082 : cluster [DBG] 2.27 scrub ok 2021-02-10T04:47:18.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:47:18 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:47:18.988 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:18 smithi164 conmon[33677]: debug 2021-02-10T04:47:18.693+0000 7fbec19a1700 -1 osd.4 1066 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:18.989 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:18 smithi164 conmon[29288]: cluster 2021-02-10T04:47:17. 2021-02-10T04:47:18.989 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:18 smithi164 conmon[29288]: 598105+0000 osd.5 (osd.5) 131 : cluster [DBG] 6.4 deep-scrub ok 2021-02-10T04:47:18.989 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:18 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:47:18.989 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:18 smithi164 conmon[29288]: 02-10T04:47:17.655405+0000 osd.4 2021-02-10T04:47:18.989 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:18 smithi164 conmon[29288]: (osd.4) 2083 : cluster 2021-02-10T04:47:18.989 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:18 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:18.989 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:18 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:47:18.989 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:18 smithi164 conmon[29288]: -02-10T04:47:17.655413 2021-02-10T04:47:18.989 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:18 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2084 : cluster 2021-02-10T04:47:18.989 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:18 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:18.989 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:18 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:47:18.990 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:18 smithi164 conmon[29288]: T04:47:17.655421+0000 osd.4 ( 2021-02-10T04:47:18.990 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:18 smithi164 conmon[29288]: osd.4) 2085 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:18.990 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:18 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:47:18.990 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:18 smithi164 conmon[29288]: -10T04:47:17.655428+0000 osd.4 (osd 2021-02-10T04:47:18.990 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:18 smithi164 conmon[29288]: .4) 2086 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:18.990 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:18 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:47:18.990 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:18 smithi164 conmon[29288]: -02-10T04:47:17.919739+0000 osd.0 2021-02-10T04:47:18.990 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:18 smithi164 conmon[29288]: (osd.0) 140 : cluster [DBG] 2.1f scrub ok 2021-02-10T04:47:18.990 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:18 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:47:18.990 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:18 smithi164 conmon[29288]: 02-10T04:47:18.026475+0000 2021-02-10T04:47:18.990 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:18 smithi164 conmon[29288]: osd.6 (osd.6) 131 : cluster [DBG] 6.28 scrub ok 2021-02-10T04:47:19.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[29319]: cluster 2021-02-10T04:47:17.598105+0000 2021-02-10T04:47:19.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[29319]: osd.5 (osd.5) 131 : cluster [DBG] 6.4 deep-scrub ok 2021-02-10T04:47:19.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[29319]: cluster 2021-02-10T04:47:17.655405+0000 osd.4 (osd.4) 2083 2021-02-10T04:47:19.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:19.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:47:19.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[29319]: -10T04:47:17.655413+0000 osd.4 (osd.4) 2084 2021-02-10T04:47:19.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[29319]: : cluster 2021-02-10T04:47:19.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:19.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:47:19.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[29319]: T04:47: 2021-02-10T04:47:19.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[29319]: 17.655421+0000 osd.4 (osd.4) 2085 : cluster 2021-02-10T04:47:19.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:19.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:47:19.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[29319]: 47:17.655428+0000 osd.4 (osd.4) 2086 : 2021-02-10T04:47:19.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:19.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[29319]: cluster 2021-02-10T04:47:19.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[29319]: 2021-02-10T04:47:17.919739+0000 osd.0 (osd.0) 140 2021-02-10T04:47:19.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[29319]: : cluster [DBG] 2.1f scrub ok 2021-02-10T04:47:19.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[29319]: cluster 2021-02-10T04:47:19.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[29319]: 2021-02-10T04:47:18.026475+0000 osd.6 (osd.6) 131 : 2021-02-10T04:47:19.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[29319]: cluster [DBG] 6.28 scrub ok 2021-02-10T04:47:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[35961]: cluster 2021-02-10T04:47:17.598105 2021-02-10T04:47:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[35961]: +0000 osd.5 (osd.5) 131 : cluster [DBG] 6.4 deep-scrub ok 2021-02-10T04:47:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:47:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[35961]: -02-10T04:47:17.655405+0000 osd.4 (osd.4) 2083 : cluster [WRN] 2021-02-10T04:47:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[35961]: cluster 2021-02-10T04:47:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[35961]: 2021-02-10T04:47:17.655413+0000 osd.4 (osd.4) 2084 : cluster 2021-02-10T04:47:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[35961]: cluster 2021-02-10T04:47 2021-02-10T04:47:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[35961]: :17.655421+0000 osd.4 (osd.4) 2085 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:47:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[35961]: -02-10T04:47:17.655428+0000 osd.4 (osd.4) 2086 : cluster 2021-02-10T04:47:19.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:19.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:47:19.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[35961]: :47:17.919739+0000 osd.0 (osd. 2021-02-10T04:47:19.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[35961]: 0) 140 : cluster [DBG] 2.1f scrub ok 2021-02-10T04:47:19.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:47:19.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[35961]: :47:18.026475+0000 osd.6 (osd.6) 131 2021-02-10T04:47:19.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:18 smithi085 conmon[35961]: : cluster [DBG] 6.28 scrub ok 2021-02-10T04:47:19.672 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:47:19.715 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:47:19.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:19 smithi164 conmon[33677]: debug 2021-02-10T04:47:19.650+0000 7fbec19a1700 -1 osd.4 1066 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:20.220 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:47:20.327 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:47:20.354 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:19 smithi164 conmon[29288]: cluster 2021-02-10T04:47:20.355 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:19 smithi164 conmon[29288]: 2021- 2021-02-10T04:47:20.355 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:19 smithi164 conmon[29288]: 02-10T04:47:18.336113+0000 mgr.y (mgr.14140) 1051 : cluster [DBG] pgmap v1826: 188 pgs: 188 active+clean; 897 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:20.355 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:19 smithi164 conmon[29288]: cluster 2021-02-10T04:47:20.355 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:19 smithi164 conmon[29288]: 2021-02-10T04:47:18. 2021-02-10T04:47:20.355 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:19 smithi164 conmon[29288]: 695291+0000 osd.4 (osd.4) 2087 : cluster 2021-02-10T04:47:20.355 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:19 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:20.355 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:19 smithi164 conmon[29288]: cluster 2021-02-10T04:47:18. 2021-02-10T04:47:20.355 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:19 smithi164 conmon[29288]: 695298+0000 osd.4 (osd.4) 2088 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:20.355 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:19 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:47:20.355 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:19 smithi164 conmon[29288]: 10T04:47:18.695303+0000 osd.4 (osd.4) 2021-02-10T04:47:20.355 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:19 smithi164 conmon[29288]: 2089 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:20.356 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:19 smithi164 conmon[29288]: cluster 2021-02-10T04:47:20.356 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:19 smithi164 conmon[29288]: 2021-02-10T04:47:18. 2021-02-10T04:47:20.356 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:19 smithi164 conmon[29288]: 695308+0000 osd.4 (osd.4) 2090 : cluster 2021-02-10T04:47:20.356 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:19 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:20.356 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:19 smithi164 conmon[29288]: cluster 2021-02-10T04:47:20.356 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:19 smithi164 conmon[29288]: 2021-02 2021-02-10T04:47:20.356 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:19 smithi164 conmon[29288]: -10T04:47:18.806638+0000 osd.3 2021-02-10T04:47:20.356 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:19 smithi164 conmon[29288]: (osd.3) 132 : cluster [DBG] 4.1d deep-scrub ok 2021-02-10T04:47:20.356 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:19 smithi164 conmon[29288]: audit 2021-02-10T04:47 2021-02-10T04:47:20.356 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:19 smithi164 conmon[29288]: :18.813049+0000 mgr.y (mgr.14140) 1052 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:47:20.356 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:19 smithi164 conmon[29288]: cluster 2021-02-10T04:47:20.356 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:19 smithi164 conmon[29288]: 2021-02-10T04:47:18.979778+0000 mon.a (mon.0) 2133 : cluster 2021-02-10T04:47:20.356 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:19 smithi164 conmon[29288]: [WRN] Health check update: 4 slow ops, oldest one blocked for 172 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:47:20.357 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:20 smithi164 conmon[33677]: debug 2021-02-10T04:47:20.009+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:47:20.357 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:47:20.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[35961]: cluster 2021-02-10T04:47:20.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[35961]: 2021-02-10T04:47:20.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[35961]: 2021 2021-02-10T04:47:20.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[35961]: -02-10T04:47:18. 2021-02-10T04:47:20.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[35961]: 336113+0000 mgr.y (mgr.14140) 1051 : cluster [DBG] pgmap v1826: 188 pgs: 188 active+clean; 897 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:20.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:47:20.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[35961]: 02-10T04:47:18.695291+0000 osd.4 ( 2021-02-10T04:47:20.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[35961]: osd.4) 2087 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:20.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:47:20.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[35961]: 10T04:47:18.695298+0000 osd.4 (osd. 2021-02-10T04:47:20.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[35961]: 4) 2088 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:20.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[35961]: cluster 2021-02-10T04:47:20.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[35961]: 2021-02-10T04:47:18.695303+0000 osd.4 (osd.4) 2089 : cluster 2021-02-10T04:47:20.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:20.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[35961]: cluster 2021-02-10T04:47:18.695308 2021-02-10T04:47:20.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2090 : cluster [WRN] 2021-02-10T04:47:20.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:20.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:47:20.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[35961]: :47:18.806638+0000 osd.3 (osd.3 2021-02-10T04:47:20.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[35961]: ) 132 : cluster [DBG] 4.1d deep-scrub ok 2021-02-10T04:47:20.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:47:20.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[35961]: 04:47:18.813049+0000 mgr.y (mgr.14140 2021-02-10T04:47:20.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[35961]: ) 1052 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:47:20.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:47:20.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[35961]: 04:47:18.979778+0000 mon.a (mon.0) 2021-02-10T04:47:20.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[35961]: 2133 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 172 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:47:20.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[29319]: cluster 2021-02-10T04:47:18.336113+0000 mgr.y (mgr.14140) 1051 : cluster [DBG] 2021-02-10T04:47:20.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[29319]: pgmap v1826: 188 pgs: 188 active+clean; 897 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:20.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[29319]: cluster 2021-02-10T04:47:18.695291+0000 osd.4 (osd.4) 2087 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:20.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[29319]: cluster 2021-02-10T04:47:20.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[29319]: 2021-02-10T04:47:18.695298+0000 osd.4 (osd.4) 2088 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:20.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[29319]: cluster 2021-02-10T04:47: 2021-02-10T04:47:20.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[29319]: 18.695303+0000 osd.4 (osd.4 2021-02-10T04:47:20.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[29319]: ) 2089 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:20.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[29319]: cluster 2021-02-10T04:47:18.695308+0000 2021-02-10T04:47:20.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[29319]: osd.4 (osd.4) 2090 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:20.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[29319]: cluster 2021-02-10T04:47:18.806638+0000 osd.3 (osd.3 2021-02-10T04:47:20.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[29319]: ) 132 : cluster [DBG] 4.1d deep-scrub ok 2021-02-10T04:47:20.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:47:20.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[29319]: -10T04:47:18.813049+0000 mgr.y (mgr.14140) 1052 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:47:20.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:47:20.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[29319]: 04:47:18.979778+0000 mon.a (mon.0) 2133 : cluster [WRN] 2021-02-10T04:47:20.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:19 smithi085 conmon[29319]: Health check update: 4 slow ops, oldest one blocked for 172 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:47:20.644 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:47:20 smithi164 conmon[40892]: debug 2021-02-10T04:47:20.428+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:47:20.740 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:47:20.846 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:47:20.880 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:47:20.998 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:20 smithi164 conmon[33677]: debug 2021-02-10T04:47:20.668+0000 7fbec19a1700 -1 osd.4 1066 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:21.000 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:20 smithi164 conmon[29288]: cluster 2021-02-10T04:47:21.000 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:20 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:47:21.000 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:20 smithi164 conmon[29288]: 47:19.637872+0000 osd.5 (osd.5) 132 : cluster [DBG] 6.7 scrub ok 2021-02-10T04:47:21.000 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:20 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:47:21.000 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:20 smithi164 conmon[29288]: 02-10T04:47:19.652194+0000 osd.4 (osd 2021-02-10T04:47:21.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:20 smithi164 conmon[29288]: .4) 2091 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:21.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:20 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:47:21.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:20 smithi164 conmon[29288]: -10T04:47:19.652203+0000 osd.4 (osd 2021-02-10T04:47:21.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:20 smithi164 conmon[29288]: .4) 2092 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:21.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:20 smithi164 conmon[29288]: cluster 2021-02-10T04:47:21.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:20 smithi164 conmon[29288]: 2021-02-10T04:47:19.652210+0000 osd.4 ( 2021-02-10T04:47:21.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:20 smithi164 conmon[29288]: osd.4) 2093 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:21.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:20 smithi164 conmon[29288]: 2021-02-10T04:47:21.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:20 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:47:21.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:20 smithi164 conmon[29288]: 02-10T04:47:19.652316+0000 2021-02-10T04:47:21.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:20 smithi164 conmon[29288]: osd.4 (osd.4) 2094 : cluster 2021-02-10T04:47:21.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:20 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:21.001 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:20 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:47:21.002 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:20 smithi164 conmon[29288]: :47:19.755710+0000 osd.3 (osd.3) 133 : cluster 2021-02-10T04:47:21.002 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:20 smithi164 conmon[29288]: [DBG] 5.18 deep-scrub ok 2021-02-10T04:47:21.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[29319]: cluster 2021-02-10T04:47:21.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[29319]: 2021- 2021-02-10T04:47:21.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[29319]: 02-10T04:47:19.637872+0000 osd.5 (osd.5) 132 : cluster 2021-02-10T04:47:21.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[29319]: [DBG] 6.7 scrub ok 2021-02-10T04:47:21.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[29319]: cluster 2021-02-10T04:47:19.652194+0000 2021-02-10T04:47:21.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[29319]: osd.4 (osd.4) 2091 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:21.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:47:21.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[29319]: -10T04:47:19.652203+0000 osd.4 ( 2021-02-10T04:47:21.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[29319]: osd.4) 2092 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:21.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:47:21.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[29319]: 02-10T04:47:19.652210+0000 osd.4 2021-02-10T04:47:21.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[29319]: (osd.4) 2093 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:21.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:47:21.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[29319]: -02-10T04:47:19.652316+0000 2021-02-10T04:47:21.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[29319]: osd.4 (osd.4) 2094 : cluster [WRN] 2021-02-10T04:47:21.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:21.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[29319]: cluster 2021-02-10T04:47:19. 2021-02-10T04:47:21.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[29319]: 755710+0000 osd.3 (osd.3) 133 : cluster 2021-02-10T04:47:21.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[29319]: [DBG] 5.18 deep-scrub ok 2021-02-10T04:47:21.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[35961]: cluster 2021-02-10T04:47:19.637872+0000 osd.5 (osd.5) 2021-02-10T04:47:21.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[35961]: 132 : cluster [DBG] 6.7 scrub ok 2021-02-10T04:47:21.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[35961]: cluster 2021-02-10T04:47:19.652194+0000 osd.4 (osd.4) 2091 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:21.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[35961]: cluster 2021-02-10T04:47:19.652203+0000 osd.4 (osd.4) 2092 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:21.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[35961]: cluster 2021-02-10T04:47:19.652210+0000 osd.4 ( 2021-02-10T04:47:21.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[35961]: osd.4) 2093 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:21.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[35961]: cluster 2021-02-10T04:47:21.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[35961]: 2021-02-10T04:47:19.652316+0000 osd.4 (osd.4 2021-02-10T04:47:21.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[35961]: ) 2094 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:21.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[35961]: 2021-02-10T04:47:21.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[35961]: cluster 2021-02-10T04:47 2021-02-10T04:47:21.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:20 smithi085 conmon[35961]: :19.755710+0000 osd.3 (osd.3) 133 : cluster [DBG] 5.18 deep-scrub ok 2021-02-10T04:47:21.196 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:47:21 smithi085 conmon[41540]: debug 2021-02-10T04:47:21.196 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:47:21 smithi085 conmon[41540]: 2021-02-10T04:47:21.007+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:47:21.249 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:47:21.843 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:47:21.843 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2021-02-10T04:47:21.844 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 0 2021-02-10T04:47:21.844 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:47:21.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:21 smithi164 conmon[33677]: debug 2021-02-10T04:47:21.638+0000 7fbec19a1700 -1 osd.4 1066 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:22.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:21 smithi164 conmon[29288]: cluster 2021-02-10T04:47:20.336910+0000 mgr.y 2021-02-10T04:47:22.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:22 smithi164 conmon[29288]: (mgr.14140) 1053 : cluster [DBG] pgmap v1827: 188 pgs: 188 active+clean; 897 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:22.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:22 smithi164 conmon[29288]: cluster 2021-02-10T04:47:20.635530+0000 osd.5 (osd.5) 133 : cluster [DBG] 6.24 deep-scrub ok 2021-02-10T04:47:22.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:22 smithi164 conmon[29288]: cluster 2021-02-10T04:47:20.670265+0000 osd.4 (osd.4) 2095 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:22.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:22 smithi164 conmon[29288]: cluster 2021-02-10T04:47:22.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:22 smithi164 conmon[29288]: 2021-02-10T04:47:20.670276+0000 osd.4 (osd.4) 2096 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:22.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:22 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:47:22.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:22 smithi164 conmon[29288]: -02-10T04:47:20.670308+0000 osd.4 (osd 2021-02-10T04:47:22.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:22 smithi164 conmon[29288]: .4) 2097 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:22.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:22 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:47:22.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:22 smithi164 conmon[29288]: -10T04:47:20.670316+0000 osd.4 (osd.4 2021-02-10T04:47:22.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:22 smithi164 conmon[29288]: ) 2098 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:22.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:22 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:47:22.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:22 smithi164 conmon[29288]: T04:47:20.802728+0000 osd.3 (osd.3 2021-02-10T04:47:22.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:22 smithi164 conmon[29288]: ) 134 : cluster [DBG] 7.2 deep-scrub ok 2021-02-10T04:47:22.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:21 smithi085 conmon[29319]: cluster 2021-02-10T04:47:20.336910+0000 2021-02-10T04:47:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:21 smithi085 conmon[29319]: mgr.y (mgr.14140) 1053 : cluster [DBG] pgmap v1827: 188 pgs: 188 active+clean; 897 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:21 smithi085 conmon[29319]: cluster 2021-02-10T04:47:20.635530+0000 osd.5 (osd.5) 133 : cluster [DBG] 6.24 deep-scrub ok 2021-02-10T04:47:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:21 smithi085 conmon[29319]: cluster 2021-02-10T04:47:20.670265+0000 osd.4 (osd.4) 2095 : cluster 2021-02-10T04:47:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:21 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:21 smithi085 conmon[29319]: cluster 2021-02-10T04:47:20.670276+0000 osd.4 ( 2021-02-10T04:47:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:21 smithi085 conmon[29319]: osd.4) 2096 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:21 smithi085 conmon[29319]: cluster 2021-02-10T04:47 2021-02-10T04:47:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:21 smithi085 conmon[29319]: :20.670308+0000 osd.4 (osd.4) 2097 : cluster [WRN] 2021-02-10T04:47:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:21 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:21 smithi085 conmon[29319]: cluster 2021-02-10T04:47:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:21 smithi085 conmon[29319]: 2021-02-10T04:47:20.670316+0000 osd.4 (osd.4) 2098 : cluster 2021-02-10T04:47:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:21 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:22.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:21 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:47:22.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:21 smithi085 conmon[29319]: -10T04:47:20.802728+0000 osd.3 (osd.3) 134 : cluster [DBG] 7.2 deep-scrub ok 2021-02-10T04:47:22.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:21 smithi085 conmon[35961]: cluster 2021-02-10T04:47:20.336910+0000 mgr.y ( 2021-02-10T04:47:22.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:21 smithi085 conmon[35961]: mgr.14140) 1053 : cluster [DBG] pgmap v1827: 188 pgs: 188 active+clean; 897 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:22.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:21 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:47:22.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:21 smithi085 conmon[35961]: T04:47:20.635530+0000 osd.5 (osd.5) 133 : cluster [DBG] 6.24 deep-scrub ok 2021-02-10T04:47:22.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:21 smithi085 conmon[35961]: cluster 2021-02-10T04:47:20. 2021-02-10T04:47:22.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:21 smithi085 conmon[35961]: 670265+0000 osd.4 (osd.4) 2095 : cluster [WRN] 2021-02-10T04:47:22.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:21 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:22.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:21 smithi085 conmon[35961]: cluster 2021-02-10T04:47:20. 2021-02-10T04:47:22.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:21 smithi085 conmon[35961]: 670276+0000 osd.4 (osd.4) 2096 : cluster 2021-02-10T04:47:22.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:21 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:22.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:21 smithi085 conmon[35961]: cluster 2021-02-10T04:47 2021-02-10T04:47:22.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:21 smithi085 conmon[35961]: :20.670308+0000 osd.4 (osd.4) 2097 : 2021-02-10T04:47:22.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:22 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:22.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:22 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:47:22.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:22 smithi085 conmon[35961]: 47:20.670316+0000 osd.4 (osd.4) 2021-02-10T04:47:22.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:22 smithi085 conmon[35961]: 2098 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:22.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:22 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:47:22.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:22 smithi085 conmon[35961]: T04:47:20.802728+0000 osd.3 (osd 2021-02-10T04:47:22.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:22 smithi085 conmon[35961]: .3) 134 : cluster [DBG] 7.2 deep-scrub ok 2021-02-10T04:47:23.009 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:23 smithi164 conmon[29288]: cluster 2021-02-10T04:47:21.640646+0000 osd.4 (osd 2021-02-10T04:47:24.683 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:23 smithi164 conmon[29288]: .4) 2099 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:24.683 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:23 smithi164 conmon[29288]: cluster 2021-02-10T04:47:21.640654+0000 osd.4 (osd.4) 2100 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:24.683 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:23 smithi164 conmon[29288]: cluster 2021-02-10T04:47:21.640671+0000 osd.4 (osd.4 2021-02-10T04:47:24.684 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:23 smithi164 conmon[29288]: ) 2101 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:24.684 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:23 smithi164 conmon[29288]: 2021-02-10T04:47:24.684 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:23 smithi164 conmon[29288]: cluster 2021-02-10T04:47:21.640678 2021-02-10T04:47:24.684 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:23 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2102 : cluster [WRN] 2021-02-10T04:47:24.684 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:23 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:24.684 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:22 smithi164 conmon[33677]: debug 2021-02-10T04:47:24.684 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:22 smithi164 conmon[33677]: 2021-02-10T04:47:22.654+0000 7fbec19a1700 -1 osd.4 1066 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:24.686 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:23 smithi085 conmon[35961]: cluster 2021-02-10T04:47:21.640646 2021-02-10T04:47:24.686 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:23 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2099 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:24.686 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:23 smithi085 conmon[35961]: cluster 2021-02-10T04:47:24.687 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:23 smithi085 conmon[35961]: 2021-02-10T04:47 2021-02-10T04:47:24.687 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:23 smithi085 conmon[35961]: :21.640654+0000 osd.4 (osd.4) 2100 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:24.687 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:23 smithi085 conmon[35961]: cluster 2021-02-10T04:47:24.687 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:23 smithi085 conmon[35961]: 2021-02-10T04:47:21.640671+0000 osd.4 (osd.4) 2101 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:24.688 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:23 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:47:24.688 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:23 smithi085 conmon[35961]: 47:21.640678+0000 osd.4 (osd.4) 2102 2021-02-10T04:47:24.688 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:23 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:24.688 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[35961]: cluster 2021-02-10T04:47:22.337473+0000 mgr.y ( 2021-02-10T04:47:24.688 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[35961]: mgr.14140) 1054 : cluster [DBG] pgmap v1828: 188 pgs: 188 active+clean; 897 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:24.688 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[35961]: cluster 2021-02-10T04:47:24.688 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[35961]: 2021-02-10T04:47:22.656448+0000 osd.4 (osd.4) 2103 2021-02-10T04:47:24.688 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:24.688 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:47:24.689 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[35961]: -10T04:47:22.656461+0000 osd.4 ( 2021-02-10T04:47:24.689 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[35961]: osd.4) 2104 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:24.689 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:47:24.689 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[35961]: 02-10T04:47:22.656468+0000 osd.4 2021-02-10T04:47:24.689 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[35961]: (osd.4) 2105 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:24.689 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:47:24.689 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[35961]: -02-10T04:47:22.656476+0000 2021-02-10T04:47:24.690 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[35961]: osd.4 (osd.4) 2106 : cluster [WRN] 2021-02-10T04:47:24.690 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:24.690 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:47:24.690 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[35961]: 04:47:22.855738+0000 osd.3 (osd 2021-02-10T04:47:24.690 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[35961]: .3) 135 : cluster [DBG] 7.3 scrub ok 2021-02-10T04:47:24.690 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:47:24.690 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[35961]: 04:47:22.953351+0000 osd.6 2021-02-10T04:47:24.690 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[35961]: (osd.6) 132 : cluster [DBG] 6.23 scrub ok 2021-02-10T04:47:24.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:23 smithi085 conmon[29319]: cluster 2021-02-10T04:47:21.640646+0000 osd.4 ( 2021-02-10T04:47:24.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:23 smithi085 conmon[29319]: osd.4) 2099 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:24.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:23 smithi085 conmon[29319]: cluster 2021-02-10T04:47:21.640654+0000 osd.4 (osd.4) 2100 : cluster 2021-02-10T04:47:24.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:23 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:24.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:23 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:47:24.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:23 smithi085 conmon[29319]: 10T04:47:21.640671+0000 osd.4 (osd.4) 2101 : cluster [WRN] 2021-02-10T04:47:24.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:23 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:24.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:23 smithi085 conmon[29319]: cluster 2021-02-10T04:47: 2021-02-10T04:47:24.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:23 smithi085 conmon[29319]: 21.640678+0000 osd.4 (osd.4) 2102 : cluster [WRN] 2021-02-10T04:47:24.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:23 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:24.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[29319]: cluster 2021-02-10T04:47:22.337473+0000 mgr.y (mgr.14140) 1054 2021-02-10T04:47:24.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[29319]: : cluster [DBG] pgmap v1828: 188 pgs: 188 active+clean; 897 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:24.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[29319]: cluster 2021-02-10T04:47:22.656448+0000 osd.4 (osd.4) 2103 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:24.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:47:24.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[29319]: 04:47:22.656461+0000 osd.4 (osd.4) 2104 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:24.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[29319]: cluster 2021-02-10T04:47:22.656468 2021-02-10T04:47:24.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2105 : cluster [WRN] 2021-02-10T04:47:24.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:24.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[29319]: cluster 2021-02-10T04:47:22.656476+0000 2021-02-10T04:47:24.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[29319]: osd.4 (osd.4) 2106 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:24.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[29319]: 2021-02-10T04:47:24.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:47:24.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[29319]: 02-10T04:47:22.855738+0000 osd.3 (osd.3) 135 : cluster [DBG] 7.3 scrub ok 2021-02-10T04:47:24.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:47:24.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[29319]: 10T04:47:22.953351+0000 osd.6 (osd.6) 132 : cluster [DBG] 6.23 scrub ok 2021-02-10T04:47:24.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:24 smithi085 conmon[29319]: 2021-02-10T04:47:24.694 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:23 smithi164 conmon[33677]: debug 2021-02-10T04:47:23.635+0000 7fbec19a1700 -1 osd.4 1066 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:24.694 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:24 smithi164 conmon[29288]: cluster 2021-02-10T04:47:22.337473+0000 mgr.y ( 2021-02-10T04:47:24.694 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:24 smithi164 conmon[29288]: mgr.14140) 1054 : cluster [DBG] pgmap v1828: 188 pgs: 188 active+clean; 897 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:24.695 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:24 smithi164 conmon[29288]: cluster 2021-02-10T04:47:22.656448+0000 2021-02-10T04:47:24.695 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:24 smithi164 conmon[29288]: osd.4 (osd.4) 2103 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:24.695 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:24 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:47:24.695 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:24 smithi164 conmon[29288]: 10T04:47:22.656461+0000 osd.4 2021-02-10T04:47:24.695 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:24 smithi164 conmon[29288]: (osd.4) 2104 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:24.696 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:24 smithi164 conmon[29288]: 2021-02-10T04:47:24.696 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:24 smithi164 conmon[29288]: cluster 2021-02-10T04:47:24.696 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:24 smithi164 conmon[29288]: 2021-02-10T04:47:22.656468+0000 osd.4 2021-02-10T04:47:24.696 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:24 smithi164 conmon[29288]: (osd.4) 2105 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:24.696 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:24 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:47:24.696 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:24 smithi164 conmon[29288]: 47:22.656476+0000 osd.4 (osd.4) 2106 : cluster [WRN] 2021-02-10T04:47:24.696 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:24 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:24.696 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:24 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:47:24.696 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:24 smithi164 conmon[29288]: 04:47:22.855738+0000 osd.3 (osd.3) 135 : cluster [DBG] 7.3 scrub ok 2021-02-10T04:47:24.696 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:24 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:47:24.696 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:24 smithi164 conmon[29288]: :47:22.953351+0000 osd.6 (osd.6 2021-02-10T04:47:24.696 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:24 smithi164 conmon[29288]: ) 132 : cluster [DBG] 6.23 scrub ok 2021-02-10T04:47:24.795 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:47:24.825 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:47:24.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:24 smithi164 conmon[33677]: debug 2021-02-10T04:47:24.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:24 smithi164 conmon[33677]: 2021-02-10T04:47:24.610+0000 7fbec19a1700 -1 osd.4 1066 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:24.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:24 smithi164 conmon[33677]: 2021-02-10T04:47:25.309 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:25 smithi085 conmon[29319]: cluster 2021-02-10T04:47:23.637567 2021-02-10T04:47:27.629 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:25 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2107 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:27.630 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:25 smithi085 conmon[29319]: cluster 2021-02-10T04:47:23.637579+0000 osd.4 (osd.4) 2108 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:27.630 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:25 smithi085 conmon[29319]: cluster 2021-02-10T04:47:23.637598+0000 osd.4 (osd.4) 2109 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:27.630 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:25 smithi085 conmon[29319]: cluster 2021-02-10T04:47:23.637609+0000 osd.4 (osd.4) 2110 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:27.630 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:25 smithi085 conmon[29319]: cluster 2021-02-10T04:47:23.904203+0000 osd.0 (osd.0) 141 : cluster 2021-02-10T04:47:27.630 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:25 smithi085 conmon[29319]: [DBG] 2.13 scrub ok 2021-02-10T04:47:27.630 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:25 smithi085 conmon[35961]: cluster 2021-02-10T04:47:23.637567+0000 osd.4 (osd 2021-02-10T04:47:27.630 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:25 smithi085 conmon[35961]: .4) 2107 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:27.630 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:25 smithi085 conmon[35961]: cluster 2021-02-10T04:47:27.631 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:25 smithi085 conmon[35961]: 2021 2021-02-10T04:47:27.631 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:25 smithi085 conmon[35961]: -02-10T04:47:23. 2021-02-10T04:47:27.631 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:25 smithi085 conmon[35961]: 637579+0000 osd.4 (osd.4) 2108 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:27.631 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:25 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:47:27.631 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:25 smithi085 conmon[35961]: 47:23.637598+0000 osd.4 (osd.4) 2109 2021-02-10T04:47:27.631 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:25 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:27.631 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:25 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:47:27.631 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:25 smithi085 conmon[35961]: 47:23.637609+0000 osd.4 (osd.4) 2110 : 2021-02-10T04:47:27.631 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:25 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:27.631 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:25 smithi085 conmon[35961]: cluster 2021-02-10T04:47:23 2021-02-10T04:47:27.631 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:25 smithi085 conmon[35961]: .904203+0000 osd.0 (osd.0) 141 : cluster 2021-02-10T04:47:27.631 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:25 smithi085 conmon[35961]: [DBG] 2.13 scrub ok 2021-02-10T04:47:27.632 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:47:25 smithi085 conmon[41540]: debug 2021-02-10T04:47:27.632 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:47:25 smithi085 conmon[41540]: 2021-02-10T04:47:25.307+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:47:27.632 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:47:25 smithi085 conmon[41540]: 2021-02-10T04:47:27.633 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:25 smithi164 conmon[29288]: cluster 2021-02-10T04:47:23.637567 2021-02-10T04:47:27.633 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:25 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2107 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:27.634 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:25 smithi164 conmon[29288]: cluster 2021-02-10T04:47:23.637579+0000 osd.4 (osd.4) 2108 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:27.634 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:25 smithi164 conmon[29288]: cluster 2021-02-10T04:47:23.637598+0000 osd.4 2021-02-10T04:47:27.634 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:25 smithi164 conmon[29288]: (osd.4) 2109 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:27.634 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:25 smithi164 conmon[29288]: cluster 2021-02-10T04:47:23 2021-02-10T04:47:27.634 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:25 smithi164 conmon[29288]: .637609+0000 osd.4 (osd.4) 2110 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:27.634 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:25 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:47:27.634 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:25 smithi164 conmon[29288]: 04:47:23.904203+0000 osd.0 (osd.0) 141 : cluster 2021-02-10T04:47:27.634 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:25 smithi164 conmon[29288]: [DBG] 2.13 scrub ok 2021-02-10T04:47:27.634 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:26 smithi164 conmon[29288]: cluster 2021-02-10T04:47: 2021-02-10T04:47:27.634 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:26 smithi164 conmon[29288]: 24.338109+0000 mgr.y (mgr.14140) 1055 : cluster [DBG] pgmap v1829: 188 pgs: 188 active+clean; 897 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:27.634 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:26 smithi164 conmon[29288]: cluster 2021-02-10T04:47:24.611731+0000 osd.4 (osd.4 2021-02-10T04:47:27.634 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:26 smithi164 conmon[29288]: ) 2111 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:27.635 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:26 smithi164 conmon[29288]: cluster 2021-02-10T04:47:24.611739+0000 osd.4 ( 2021-02-10T04:47:27.635 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:26 smithi164 conmon[29288]: osd.4) 2112 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:27.635 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:26 smithi164 conmon[29288]: cluster 2021-02-10T04:47:27.635 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:26 smithi164 conmon[29288]: 2021-02-10T04:47:24.611746+0000 osd.4 (osd.4) 2021-02-10T04:47:27.635 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:26 smithi164 conmon[29288]: 2113 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:27.635 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:26 smithi164 conmon[29288]: cluster 2021-02-10T04:47: 2021-02-10T04:47:27.635 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:26 smithi164 conmon[29288]: 24.611752+0000 osd.4 (osd.4) 2114 : cluster 2021-02-10T04:47:27.635 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:26 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:27.635 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:26 smithi164 conmon[29288]: cluster 2021-02-10T04:47:24.897676+0000 2021-02-10T04:47:27.635 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:26 smithi164 conmon[29288]: osd.3 (osd.3) 136 : cluster [DBG] 7.14 scrub ok 2021-02-10T04:47:27.635 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:26 smithi164 conmon[29288]: 2021-02-10T04:47:27.635 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:27 smithi164 conmon[29288]: cluster 2021-02-10T04:47:25.568084+0000 osd.4 (osd 2021-02-10T04:47:27.635 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:27 smithi164 conmon[29288]: .4) 2115 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:27.636 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:27 smithi164 conmon[29288]: cluster 2021-02-10T04:47:25.568092+0000 osd.4 (osd.4) 2116 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:27.636 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:27 smithi164 conmon[29288]: cluster 2021-02-10T04:47:25.568099+0000 osd.4 (osd.4) 2117 2021-02-10T04:47:27.636 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:27 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:27.636 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:27 smithi164 conmon[29288]: cluster 2021-02-10T04:47:25.568105+0000 osd.4 (osd.4) 2118 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:27.636 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:27 smithi164 conmon[29288]: cluster 2021-02-10T04:47:26.211136+0000 2021-02-10T04:47:27.636 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:27 smithi164 conmon[29288]: osd.6 (osd.6) 133 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:47:27.636 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:27 smithi164 conmon[29288]: cluster 2021-02-10T04:47:27.636 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:27 smithi164 conmon[29288]: 2021-02-10T04:47:26.214579+0000 osd.6 (osd.6) 134 : 2021-02-10T04:47:27.636 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:27 smithi164 conmon[29288]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:47:27.636 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:25 smithi164 conmon[33677]: debug 2021-02-10T04:47:25.566+0000 7fbec19a1700 -1 osd.4 1066 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:27.636 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:26 smithi164 conmon[33677]: debug 2021-02-10T04:47:26.544+0000 7fbec19a1700 -1 osd.4 1066 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:27.637 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:47:27.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:26 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:47:27.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:26 smithi085 conmon[29319]: 04:47:24.338109+0000 mgr.y (mgr.14140) 1055 : cluster [DBG] pgmap v1829: 188 pgs: 188 active+clean; 897 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:27.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:26 smithi085 conmon[29319]: cluster 2021-02-10T04:47:24.611731+0000 osd.4 (osd.4) 2111 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:27.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:26 smithi085 conmon[29319]: cluster 2021-02-10T04:47:24.611739+0000 osd.4 (osd.4) 2112 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:27.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:26 smithi085 conmon[29319]: cluster 2021-02-10T04:47:24.611746+0000 osd.4 (osd.4) 2113 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:27.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:26 smithi085 conmon[29319]: cluster 2021-02-10T04:47:24.611752+0000 osd.4 (osd.4) 2114 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:27.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:26 smithi085 conmon[29319]: cluster 2021-02-10T04:47:24.897676+0000 osd.3 (osd.3) 136 : cluster [DBG] 7.14 scrub ok 2021-02-10T04:47:27.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:27 smithi085 conmon[29319]: cluster 2021-02-10T04:47:25.568084+0000 osd.4 (osd.4) 2115 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:27.637 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:27 smithi085 conmon[29319]: cluster 2021-02-10T04:47:25.568092+0000 osd.4 (osd.4) 2116 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:27.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:27 smithi085 conmon[29319]: cluster 2021-02-10T04:47:25.568099+0000 osd.4 (osd.4) 2117 : 2021-02-10T04:47:27.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:27 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:27.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:27 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:47:27.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:27 smithi085 conmon[29319]: 47:25.568105+0000 osd.4 (osd.4) 2118 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:27.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:27 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:47:27.638 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:27 smithi085 conmon[29319]: 02-10T04:47:26.211136+0000 osd.6 (osd. 2021-02-10T04:47:27.639 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:27 smithi085 conmon[29319]: 6) 133 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:47:27.639 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:27 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:47:27.639 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:27 smithi085 conmon[29319]: :47:26.214579+0000 osd.6 (osd.6) 134 2021-02-10T04:47:27.639 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:27 smithi085 conmon[29319]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:47:27.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:26 smithi085 conmon[35961]: cluster 2021-02-10T04:47:27.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:26 smithi085 conmon[35961]: 2021-02-10T04:47:27.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:26 smithi085 conmon[35961]: 2021-02-10T04:47:24.338109 2021-02-10T04:47:27.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:26 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1055 : cluster [DBG] pgmap v1829: 188 pgs: 188 active+clean; 897 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:27.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:26 smithi085 conmon[35961]: cluster 2021-02-10T04:47:24 2021-02-10T04:47:27.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:26 smithi085 conmon[35961]: .611731+0000 osd.4 (osd.4) 2111 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:27.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:26 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:47:27.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:26 smithi085 conmon[35961]: 02-10T04:47:24.611739+0000 osd.4 (osd.4) 2112 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:27.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:26 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:47:27.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:26 smithi085 conmon[35961]: -10T04:47:24.611746+0000 osd.4 (osd 2021-02-10T04:47:27.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:26 smithi085 conmon[35961]: .4) 2113 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:27.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:26 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:47:27.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:26 smithi085 conmon[35961]: 02-10T04:47:24.611752+0000 2021-02-10T04:47:27.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:26 smithi085 conmon[35961]: osd.4 (osd.4) 2114 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:27.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:26 smithi085 conmon[35961]: 2021-02-10T04:47:27.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:26 smithi085 conmon[35961]: cluster 2021-02-10T04:47:24 2021-02-10T04:47:27.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:26 smithi085 conmon[35961]: .897676+0000 osd.3 (osd.3) 136 : cluster [DBG] 2021-02-10T04:47:27.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:26 smithi085 conmon[35961]: 7.14 scrub ok 2021-02-10T04:47:27.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:27 smithi085 conmon[35961]: cluster 2021-02-10T04:47:25.568084+0000 osd.4 (osd 2021-02-10T04:47:27.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:27 smithi085 conmon[35961]: .4) 2115 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:27.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:27 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:47:27.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:27 smithi085 conmon[35961]: T04:47:25.568092+0000 osd.4 (osd.4) 2116 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:27.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:27 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:47:27.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:27 smithi085 conmon[35961]: 02-10T04:47:25.568099+0000 osd.4 (osd 2021-02-10T04:47:27.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:27 smithi085 conmon[35961]: .4) 2117 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:27.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:27 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:47:27.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:27 smithi085 conmon[35961]: 02-10T04:47:25.568105+0000 2021-02-10T04:47:27.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:27 smithi085 conmon[35961]: osd.4 (osd.4) 2118 : cluster 2021-02-10T04:47:27.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:27 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:27.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:27 smithi085 conmon[35961]: cluster 2021-02-10T04:47:26 2021-02-10T04:47:27.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:27 smithi085 conmon[35961]: .211136+0000 osd.6 (osd.6) 133 : cluster 2021-02-10T04:47:27.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:27 smithi085 conmon[35961]: [DBG] purged_snaps scrub starts 2021-02-10T04:47:27.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:27 smithi085 conmon[35961]: cluster 2021-02-10T04:47: 2021-02-10T04:47:27.641 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:27 smithi085 conmon[35961]: 26.214579+0000 osd.6 (osd.6) 134 2021-02-10T04:47:27.642 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:27 smithi085 conmon[35961]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:47:27.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:27 smithi164 conmon[33677]: debug 2021-02-10T04:47:27.508+0000 7fbec19a1700 -1 osd.4 1067 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:28 smithi164 conmon[29288]: cluster 2021-02-10T04:47:26.338779 2021-02-10T04:47:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:28 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1056 : cluster [DBG] pgmap v1830: 188 pgs: 188 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:28 smithi164 conmon[29288]: cluster 2021-02-10T04:47:26. 2021-02-10T04:47:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:28 smithi164 conmon[29288]: 546097+0000 osd.4 (osd.4) 2119 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:28 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:47:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:28 smithi164 conmon[29288]: -10T04:47:26.546105+0000 osd.4 (osd. 2021-02-10T04:47:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:28 smithi164 conmon[29288]: 4) 2120 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:28 smithi164 conmon[29288]: cluster 2021-02-10T04:47:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:28 smithi164 conmon[29288]: 2021-02-10T04:47:26. 2021-02-10T04:47:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:28 smithi164 conmon[29288]: 546112+0000 osd.4 (osd.4) 2121 : cluster 2021-02-10T04:47:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:28 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:28 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:47:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:28 smithi164 conmon[29288]: 10T04:47:26.546117+0000 osd.4 2021-02-10T04:47:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:28 smithi164 conmon[29288]: (osd.4) 2122 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:28 smithi164 conmon[29288]: 2021-02-10T04:47:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:28 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:47:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:28 smithi164 conmon[29288]: -10T04:47:27.027275+0000 mon.a (mon. 2021-02-10T04:47:28.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:28 smithi164 conmon[29288]: 0) 2134 : cluster [DBG] osdmap e1067: 8 total, 8 up, 7 in 2021-02-10T04:47:28.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[29319]: cluster 2021-02-10T04:47:26.338779+0000 mgr.y (mgr. 2021-02-10T04:47:28.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[29319]: 14140) 1056 : cluster [DBG] pgmap v1830: 188 pgs: 188 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:28.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[29319]: cluster 2021-02-10T04:47:26.546097+0000 osd.4 (osd.4) 2119 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:28.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[29319]: cluster 2021-02-10T04:47:26.546105+0000 osd.4 (osd.4) 2120 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:28.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:47:28.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[29319]: :47:26.546112+0000 osd.4 (osd.4) 2121 : cluster 2021-02-10T04:47:28.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:28.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:47:28.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[29319]: -02-10T04:47:26.546117+0000 osd.4 (osd.4) 2122 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:28.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[29319]: 2021-02-10T04:47:28.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[29319]: cluster 2021-02-10T04:47:27.027275+0000 mon.a (mon.0) 2134 2021-02-10T04:47:28.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[29319]: : cluster [DBG] osdmap e1067: 8 total, 8 up, 7 in 2021-02-10T04:47:28.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[35961]: cluster 2021-02-10T04:47:28.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[35961]: 2021-02 2021-02-10T04:47:28.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[35961]: -10T04:47:26.338779+0000 mgr.y (mgr.14140) 1056 : cluster [DBG] pgmap v1830: 188 pgs: 188 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:28.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:47:28.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[35961]: -02-10T04:47:26.546097+0000 osd.4 (osd 2021-02-10T04:47:28.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[35961]: .4) 2119 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:28.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[35961]: cluster 2021-02-10T04:47: 2021-02-10T04:47:28.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[35961]: 26.546105+0000 osd.4 (osd.4) 2120 : 2021-02-10T04:47:28.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:28.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[35961]: cluster 2021-02-10T04:47:28.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[35961]: 2021-02-10T04:47:26.546112+0000 osd.4 ( 2021-02-10T04:47:28.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[35961]: osd.4) 2121 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:28.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:47:28.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[35961]: -10T04:47:26.546117+0000 osd.4 ( 2021-02-10T04:47:28.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[35961]: osd.4) 2122 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:28.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:47:28.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[35961]: 10T04:47:27.027275+0000 mon.a (mon. 2021-02-10T04:47:28.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[35961]: 0) 2134 : cluster [DBG] osdmap e1067: 8 total, 8 up, 7 in 2021-02-10T04:47:28.539 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:47:28.573 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:47:28.829 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:28 smithi164 conmon[33677]: debug 2021-02-10T04:47:28.554+0000 7fbec19a1700 -1 osd.4 1067 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:28.829 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:47:28 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:47:28.938 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:47:29.043 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:47:29.081 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:47:29.144 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:47:28 smithi164 conmon[40892]: debug 2021-02-10T04:47:29.144 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:47:28 smithi164 conmon[40892]: 2021-02-10T04:47:28.935+0000 7f3037cca700 -1 osd.6 pg_epoch: 1067 pg[6.22( empty local-lis/les=1046/1047 n=0 ec=1044/59 lis/c=1046/1046 les/c/f=1047/1047/0 sis=1046) [6,3,1] r=0 lpr=1046 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.22) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 3 (0x557a64f9e6e0) 2021-02-10T04:47:29.144 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:47:28 smithi164 conmon[40892]: 2021-02-10T04:47:29.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:29 smithi164 conmon[29288]: cluster 2021-02-10T04:47:27.510454+0000 2021-02-10T04:47:29.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:29 smithi164 conmon[29288]: osd.4 (osd.4) 2123 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:29.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:29 smithi164 conmon[29288]: cluster 2021-02-10T04:47:27.510464+0000 osd.4 (osd.4) 2124 : cluster [WRN] 2021-02-10T04:47:29.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:29 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:29.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:29 smithi164 conmon[29288]: cluster 2021-02-10T04:47:29.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:29 smithi164 conmon[29288]: 2021-02-10T04:47:27. 2021-02-10T04:47:29.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:29 smithi164 conmon[29288]: 510471+0000 osd.4 (osd.4) 2021-02-10T04:47:29.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:29 smithi164 conmon[29288]: 2125 : cluster 2021-02-10T04:47:29.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:29 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:29.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:29 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:47:29.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:29 smithi164 conmon[29288]: -02-10T04:47:27.510477+0000 2021-02-10T04:47:29.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:29 smithi164 conmon[29288]: osd.4 (osd.4) 2126 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:29.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:29 smithi164 conmon[29288]: 2021-02-10T04:47:29.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:29 smithi164 conmon[29288]: cluster 2021-02-10T04:47:29.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:29 smithi164 conmon[29288]: 2021-02-10T04:47:27. 2021-02-10T04:47:29.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:29 smithi164 conmon[29288]: 908216+0000 osd.0 (osd.0) 142 : cluster 2021-02-10T04:47:29.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:29 smithi164 conmon[29288]: [DBG] 5.5 scrub ok 2021-02-10T04:47:29.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:29 smithi164 conmon[29288]: cluster 2021-02-10T04:47:29.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:29 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:47:29.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:29 smithi164 conmon[29288]: :47:28.407471+0000 mon.a (mon.0) 2135 2021-02-10T04:47:29.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:29 smithi164 conmon[29288]: : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 177 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:47:29.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:47:28 smithi085 conmon[49106]: debug 2021-02-10T04:47:28.906+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:47:29.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:29 smithi085 conmon[29319]: cluster 2021-02-10T04:47:27. 2021-02-10T04:47:29.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:29 smithi085 conmon[29319]: 510454+0000 osd.4 (osd.4) 2123 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:29.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:29 smithi085 conmon[29319]: cluster 2021-02-10T04:47:27.510464+0000 osd.4 (osd.4) 2124 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:29.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:29 smithi085 conmon[29319]: cluster 2021-02-10T04:47:27.510471+0000 osd.4 (osd.4) 2125 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:29.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:29 smithi085 conmon[29319]: cluster 2021-02-10T04:47:27.510477+0000 osd.4 (osd.4) 2126 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:29.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:29 smithi085 conmon[29319]: cluster 2021-02-10T04:47:27.908216+0000 osd.0 (osd.0) 142 : cluster [DBG] 5.5 scrub ok 2021-02-10T04:47:29.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:29 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:47:29.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:29 smithi085 conmon[29319]: 10T04:47:28.407471+0000 mon.a (mon.0) 2135 : cluster 2021-02-10T04:47:29.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:29 smithi085 conmon[29319]: [WRN] Health check update: 4 slow ops, oldest one blocked for 177 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:47:29.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:29 smithi085 conmon[35961]: cluster 2021-02-10T04:47:27. 2021-02-10T04:47:29.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:29 smithi085 conmon[35961]: 510454+0000 osd.4 (osd.4) 2123 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:29.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:29 smithi085 conmon[35961]: cluster 2021-02-10T04:47:29.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:29 smithi085 conmon[35961]: 2021-02-10T04:47:27.510464+0000 osd.4 (osd.4) 2124 : cluster 2021-02-10T04:47:29.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:29 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:29.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:29 smithi085 conmon[35961]: cluster 2021-02-10T04:47 2021-02-10T04:47:29.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:29 smithi085 conmon[35961]: :27.510471+0000 osd.4 (osd.4) 2125 2021-02-10T04:47:29.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:29 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:29.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:29 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:47:29.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:29 smithi085 conmon[35961]: :47:27.510477+0000 osd.4 (osd.4 2021-02-10T04:47:29.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:29 smithi085 conmon[35961]: ) 2126 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:29.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:29 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:47:29.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:29 smithi085 conmon[35961]: T04:47:27.908216+0000 osd.0 (osd 2021-02-10T04:47:29.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:29 smithi085 conmon[35961]: .0) 142 : cluster [DBG] 5.5 scrub ok 2021-02-10T04:47:29.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:29 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:47:29.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:29 smithi085 conmon[35961]: :47:28.407471+0000 mon.a (mon.0) 2135 2021-02-10T04:47:29.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:29 smithi085 conmon[35961]: : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 177 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:47:29.530 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:47:29.567 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:47:29 smithi164 conmon[40892]: debug 2021-02-10T04:47:29.183+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:47:29.893 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:47:29.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:29 smithi164 conmon[33677]: debug 2021-02-10T04:47:29.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:47:29 smithi164 conmon[33677]: 2021-02-10T04:47:29.564+0000 7fbec19a1700 -1 osd.4 1067 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:47:29.923 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:47:29.955 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:47:29.987 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:47:30.016 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:47:30.041 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:47:30.067 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:47:30.094 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:47:30.096 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:47:30.096 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:47:30.096 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:47:30.096 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:47:30.096 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:47:30.096 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:47:30.096 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:47:30.097 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:47:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:47:30.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: :47:28.339420+0000 mgr.y (mgr.14140) 1057 : cluster [DBG] pgmap v1832: 188 pgs: 188 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:30.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: cluster 2021-02-10T04:47:28.555970+0000 osd.4 2021-02-10T04:47:30.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: (osd.4) 2127 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:30.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:47:30.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: 10T04:47:28.555978+0000 osd.4 (osd.4) 2128 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:30.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: cluster 2021-02-10T04:47:30.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: 2021-02-10T04:47:30.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: 2021 2021-02-10T04:47:30.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: - 2021-02-10T04:47:30.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: 02-10T04:47:28.555982+0000 2021-02-10T04:47:30.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: osd.4 (osd.4) 2129 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:30.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:47:30.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: :47:28.555986+0000 osd.4 2021-02-10T04:47:30.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: (osd.4) 2130 : cluster 2021-02-10T04:47:30.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:30.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: cluster 2021-02-10T04:47:30.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: 2021-02-10T04:47:30.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: 2021 2021-02-10T04:47:30.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: -02-10T04:47:28.706366 2021-02-10T04:47:30.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: +0000 osd.5 (osd.5) 134 : cluster [DBG] 6.26 scrub ok 2021-02-10T04:47:30.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: audit 2021-02-10T04:47:30.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: 2021-02 2021-02-10T04:47:30.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: -10T04:47:28. 2021-02-10T04:47:30.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: 828282+0000 mgr.y (mgr.14140) 1058 2021-02-10T04:47:30.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:47:30.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:47:30.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: T04:47:28.938723+0000 2021-02-10T04:47:30.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: osd.0 (osd.0) 143 : cluster [DBG] 2021-02-10T04:47:30.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: 5.1e scrub ok 2021-02-10T04:47:30.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:47:30.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: T04:47:29.864348+0000 mon.c (mon. 2021-02-10T04:47:30.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:30 smithi164 conmon[29288]: 1) 78 : audit [DBG] from='client.? 172.21.15.85:0/3750055764' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:47:30.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[29319]: cluster 2021-02-10T04:47:28.339420+0000 2021-02-10T04:47:30.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[29319]: mgr.y (mgr.14140) 1057 : cluster [DBG] pgmap v1832: 188 pgs: 188 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:30.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[29319]: cluster 2021-02-10T04:47:28.555970+0000 osd.4 (osd.4) 2127 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[29319]: cluster 2021-02-10T04:47:28.555978+0000 osd.4 (osd.4) 2128 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[29319]: cluster 2021-02-10T04:47:28.555982+0000 osd.4 (osd.4) 2129 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[29319]: cluster 2021-02-10T04:47:28.555986+0000 osd.4 (osd.4) 2130 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[29319]: cluster 2021-02-10T04:47:28.706366+0000 osd.5 (osd.5) 134 : cluster [DBG] 6.26 scrub ok 2021-02-10T04:47:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[29319]: audit 2021-02-10T04:47:28.828282+0000 mgr.y (mgr.14140) 1058 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:47:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[29319]: cluster 2021-02-10T04:47:28.938723+0000 osd.0 (osd.0) 143 : cluster [DBG] 5.1e scrub ok 2021-02-10T04:47:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[29319]: audit 2021-02-10T04:47:29.864348+0000 mon.c (mon.1) 78 : audit [DBG] from='client.? 172.21.15.85:0/3750055764' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:47:30.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[35961]: cluster 2021-02-10T04:47:28.339420 2021-02-10T04:47:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1057 : cluster [DBG] pgmap v1832: 188 pgs: 188 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[35961]: cluster 2021-02-10T04:47:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[35961]: 2021-02-10T04:47 2021-02-10T04:47:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[35961]: :28.555970+0000 osd.4 (osd.4) 2021-02-10T04:47:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[35961]: 2127 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[35961]: cluster 2021-02-10T04:47:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[35961]: 2021-02-10T04:47:28.555978 2021-02-10T04:47:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2128 : cluster 2021-02-10T04:47:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:47:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[35961]: T04:47:28.555982+0000 osd.4 ( 2021-02-10T04:47:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[35961]: osd.4) 2129 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:30.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:47:30.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[35961]: 10T04:47:28.555986+0000 osd.4 (osd 2021-02-10T04:47:30.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[35961]: .4) 2130 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:30.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[35961]: cluster 2021-02-10T04:47:30.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[35961]: 2021-02-10T04:47:28 2021-02-10T04:47:30.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[35961]: .706366+0000 osd.5 (osd.5) 134 : cluster 2021-02-10T04:47:30.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[35961]: [DBG] 6.26 scrub ok 2021-02-10T04:47:30.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[35961]: audit 2021 2021-02-10T04:47:30.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[35961]: -02-10T04:47:28.828282+0000 mgr.y 2021-02-10T04:47:30.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[35961]: (mgr.14140) 1058 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:47:30.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[35961]: cluster 2021-02-10T04:47:28.938723+0000 osd.0 (osd.0) 143 : cluster [DBG] 5.1e scrub ok 2021-02-10T04:47:30.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:30 smithi085 conmon[35961]: audit 2021-02-10T04:47:29.864348+0000 mon.c (mon.1) 78 : audit [DBG] from='client.? 172.21.15.85:0/3750055764' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:47:31.054 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is set 2021-02-10T04:47:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:31 smithi164 conmon[29288]: cluster 2021-02-10T04:47:29.566673+0000 osd.4 (osd 2021-02-10T04:47:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:31 smithi164 conmon[29288]: .4) 2131 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:31 smithi164 conmon[29288]: cluster 2021-02-10T04:47:29.566682+0000 osd.4 (osd.4) 2132 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:31 smithi164 conmon[29288]: cluster 2021-02-10T04:47:29.566689+0000 2021-02-10T04:47:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:31 smithi164 conmon[29288]: osd.4 (osd.4) 2133 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:31 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:47:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:31 smithi164 conmon[29288]: :47:29.566695+0000 osd.4 (osd.4) 2134 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:31 smithi164 conmon[29288]: cluster 2021-02-10T04:47:29. 2021-02-10T04:47:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:31 smithi164 conmon[29288]: 707325+0000 osd.5 (osd.5) 135 : cluster [DBG] 2.3 scrub ok 2021-02-10T04:47:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:31 smithi164 conmon[29288]: cluster 2021-02-10T04:47:29.965136+0000 2021-02-10T04:47:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:31 smithi164 conmon[29288]: osd.0 (osd.0) 144 : cluster [DBG] 4.16 scrub ok 2021-02-10T04:47:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:31 smithi164 conmon[29288]: audit 2021-02-10T04:47:30.122789+0000 2021-02-10T04:47:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:31 smithi164 conmon[29288]: mon.a (mon.0) 2136 : audit [INF] from='client.? 172.21.15.85:0/3186968771' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:47:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[29319]: cluster 2021-02-10T04:47:29.566673+0000 osd.4 (osd 2021-02-10T04:47:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[29319]: .4) 2131 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[29319]: cluster 2021-02-10T04:47:29.566682+0000 osd.4 (osd.4) 2132 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[29319]: cluster 2021-02-10T04:47:29.566689+0000 osd.4 (osd.4) 2133 2021-02-10T04:47:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:47:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[29319]: :47:29.566695+0000 osd.4 (osd. 2021-02-10T04:47:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[29319]: 4) 2134 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[29319]: cluster 2021-02-10T04:47:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[29319]: 2021-02-10T04:47:29. 2021-02-10T04:47:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[29319]: 707325+0000 osd.5 (osd.5) 135 : 2021-02-10T04:47:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[29319]: cluster [DBG] 2.3 scrub ok 2021-02-10T04:47:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[29319]: cluster 2021-02-10T04:47:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[29319]: 2021-02-10T04:47 2021-02-10T04:47:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[29319]: :29.965136+0000 osd.0 (osd.0) 144 2021-02-10T04:47:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[29319]: : cluster [DBG] 4.16 scrub ok 2021-02-10T04:47:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:47:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[29319]: -10T04:47:30.122789+0000 mon.a (mon. 2021-02-10T04:47:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[29319]: 0) 2136 : audit [INF] from='client.? 172.21.15.85:0/3186968771' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:47:31.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[35961]: cluster 2021-02-10T04:47:29.566673+0000 2021-02-10T04:47:31.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[35961]: osd.4 (osd.4) 2131 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:47:31.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[35961]: cluster 2021-02-10T04:47:29 2021-02-10T04:47:31.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[35961]: .566682+0000 osd.4 (osd.4) 2132 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:47:31.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[35961]: cluster 2021-02-10T04:47:29.566689 2021-02-10T04:47:31.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2133 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:47:31.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[35961]: 2021-02-10T04:47:31.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:47:31.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[35961]: 02-10T04:47:29.566695+0000 osd.4 (osd.4) 2134 : cluster 2021-02-10T04:47:31.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:47:31.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[35961]: cluster 2021-02-10T04:47:31.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[35961]: 2021-02-10T04:47:29.707325+0000 osd.5 (osd.5 2021-02-10T04:47:31.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[35961]: ) 135 : cluster [DBG] 2.3 scrub ok 2021-02-10T04:47:31.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[35961]: cluster 2021-02-10T04:47 2021-02-10T04:47:31.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[35961]: :29.965136+0000 osd.0 (osd.0) 144 : cluster [DBG] 2021-02-10T04:47:31.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[35961]: 4.16 scrub ok 2021-02-10T04:47:31.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[35961]: audit 2021-02-10T04:47:31.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[35961]: 2021-02-10T04:47:30.122789+0000 mon.a (mon.0) 2021-02-10T04:47:31.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:31 smithi085 conmon[35961]: 2136 : audit [INF] from='client.? 172.21.15.85:0/3186968771' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:47:32.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:32 smithi164 conmon[29288]: cluster 2021-02-10T04:47:30.340097+0000 mgr.y ( 2021-02-10T04:47:32.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:32 smithi164 conmon[29288]: mgr.14140) 1059 : cluster [DBG] pgmap v1833: 188 pgs: 188 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:32.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:32 smithi164 conmon[29288]: cluster 2021-02-10T04:47:31.036926+0000 mon.a (mon.0) 2137 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:47:32.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:32 smithi164 conmon[29288]: audit 2021-02-10T04:47:31.046376+0000 2021-02-10T04:47:32.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:32 smithi164 conmon[29288]: mon.a (mon.0) 2138 : audit [INF] from='client.? 172.21.15.85:0/3186968771' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:47:32.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:32 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:47:32.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:32 smithi164 conmon[29288]: 02-10T04:47:31.046508+0000 mon.a ( 2021-02-10T04:47:32.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:32 smithi164 conmon[29288]: mon.0) 2139 : cluster [DBG] osdmap e1068: 8 total, 8 up, 7 in 2021-02-10T04:47:32.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:32 smithi085 conmon[29319]: cluster 2021-02-10T04:47:30.340097+0000 mgr.y (mgr.14140) 1059 2021-02-10T04:47:32.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:32 smithi085 conmon[29319]: : cluster [DBG] pgmap v1833: 188 pgs: 188 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:32.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:32 smithi085 conmon[29319]: cluster 2021-02-10T04:47:31.036926+0000 mon.a (mon.0) 2137 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:47:32.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:32 smithi085 conmon[29319]: 2021-02-10T04:47:32.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:32 smithi085 conmon[29319]: audit 2021-02-10T04: 2021-02-10T04:47:32.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:32 smithi085 conmon[29319]: 47:31.046376+0000 mon.a (mon.0) 2138 : audit [INF] from='client.? 172.21.15.85:0/3186968771' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:47:32.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:32 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:47:32.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:32 smithi085 conmon[29319]: 02-10T04:47:31.046508+0000 2021-02-10T04:47:32.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:32 smithi085 conmon[29319]: mon.a (mon.0) 2139 : cluster [DBG] 2021-02-10T04:47:32.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:32 smithi085 conmon[29319]: osdmap e1068: 8 total, 8 up, 7 in 2021-02-10T04:47:32.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:32 smithi085 conmon[35961]: cluster 2021-02-10T04:47: 2021-02-10T04:47:32.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:32 smithi085 conmon[35961]: 30.340097+0000 mgr.y (mgr.14140) 1059 : cluster [DBG] pgmap v1833: 188 pgs: 188 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:32.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:32 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:47:32.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:32 smithi085 conmon[35961]: 02-10T04:47:31.036926+0000 mon.a ( 2021-02-10T04:47:32.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:32 smithi085 conmon[35961]: mon.0) 2137 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:47:32.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:32 smithi085 conmon[35961]: audit 2021-02-10T04:47:32.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:32 smithi085 conmon[35961]: 2021-02-10T04:47:31.046376 2021-02-10T04:47:32.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:32 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2138 2021-02-10T04:47:32.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:32 smithi085 conmon[35961]: : audit [INF] from='client.? 172.21.15.85:0/3186968771' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:47:32.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:32 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:47:32.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:32 smithi085 conmon[35961]: -02-10T04:47:31.046508+0000 2021-02-10T04:47:32.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:32 smithi085 conmon[35961]: mon.a (mon.0) 2139 : cluster [DBG] osdmap e1068: 8 total, 8 up, 7 in 2021-02-10T04:47:32.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:32 smithi085 conmon[35961]: 2021-02-10T04:47:32.596 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:47:32.633 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:47:32.941 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:47:32 smithi085 conmon[45443]: debug 2021-02-10T04:47:32.715+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:47:33.021 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:47:33.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:33 smithi164 conmon[29288]: cluster 2021-02-10T04:47:33.044235+0000 mon.a ( 2021-02-10T04:47:33.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:33 smithi164 conmon[29288]: mon.0) 2140 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 182 sec, osd.4 has slow ops) 2021-02-10T04:47:33.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:33 smithi085 conmon[29319]: cluster 2021-02-10T04:47:33.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:33 smithi085 conmon[29319]: 2021-02-10T04:47:33.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:33 smithi085 conmon[29319]: 2021 2021-02-10T04:47:33.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:33 smithi085 conmon[29319]: - 2021-02-10T04:47:33.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:33 smithi085 conmon[29319]: 02 2021-02-10T04:47:33.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:33 smithi085 conmon[29319]: - 2021-02-10T04:47:33.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:33 smithi085 conmon[29319]: 10T04:47:33.044235+0000 mon.a (mon. 2021-02-10T04:47:33.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:33 smithi085 conmon[29319]: 0) 2140 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 182 sec, osd.4 has slow ops) 2021-02-10T04:47:33.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:33 smithi085 conmon[35961]: cluster 2021-02-10T04:47:33. 2021-02-10T04:47:33.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:33 smithi085 conmon[35961]: 044235+0000 mon.a (mon.0) 2140 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 182 sec, osd.4 has slow ops) 2021-02-10T04:47:33.744 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.7 dump_blocked_ops 2021-02-10T04:47:33.899 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-02-10T04:47:33.900 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:47:33.900 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd dump -f json-pretty 2021-02-10T04:47:34.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:34 smithi085 conmon[29319]: cluster 2021-02-10T04:47:32.340641+0000 mgr.y ( 2021-02-10T04:47:34.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:34 smithi085 conmon[29319]: mgr.14140) 1060 : cluster [DBG] pgmap v1835: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:34.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:34 smithi085 conmon[29319]: cluster 2021-02-10T04:47:33.393500+0000 osd.3 (osd.3) 137 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:47:34.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:34 smithi085 conmon[29319]: cluster 2021-02-10T04:47:33.397596+0000 osd.3 (osd.3) 138 : 2021-02-10T04:47:34.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:34 smithi085 conmon[29319]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:47:34.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:34 smithi085 conmon[35961]: cluster 2021-02-10T04:47:32.340641+0000 2021-02-10T04:47:34.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:34 smithi085 conmon[35961]: mgr.y (mgr.14140) 1060 : cluster [DBG] pgmap v1835: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:34.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:34 smithi085 conmon[35961]: cluster 2021-02-10T04:47:34.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:34 smithi085 conmon[35961]: 2021-02-10T04:47:33.393500+0000 osd.3 (osd.3) 137 2021-02-10T04:47:34.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:34 smithi085 conmon[35961]: : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:47:34.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:34 smithi085 conmon[35961]: 2021-02-10T04:47:34.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:34 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:47:34.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:34 smithi085 conmon[35961]: 10T04:47:33.397596+0000 osd.3 ( 2021-02-10T04:47:34.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:34 smithi085 conmon[35961]: osd.3) 138 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:47:34.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:34 smithi164 conmon[29288]: cluster 2021-02-10T04:47:32.340641+0000 2021-02-10T04:47:34.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:34 smithi164 conmon[29288]: mgr.y (mgr.14140) 1060 : cluster [DBG] pgmap v1835: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2021-02-10T04:47:34.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:34 smithi164 conmon[29288]: cluster 2021-02-10T04:47:33.393500+0000 osd.3 (osd.3) 137 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:47:34.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:34 smithi164 conmon[29288]: cluster 2021-02-10T04:47:33.397596+0000 osd.3 (osd.3) 138 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:47:34.422 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:47:35.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:35 smithi164 conmon[29288]: cluster 2021-02-10T04:47:34.105276+0000 mon.a ( 2021-02-10T04:47:35.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:35 smithi164 conmon[29288]: mon.0) 2141 : cluster [DBG] osdmap e1069: 8 total, 8 up, 7 in 2021-02-10T04:47:35.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:35 smithi085 conmon[29319]: cluster 2021-02-10T04:47:34.105276+0000 mon.a ( 2021-02-10T04:47:35.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:35 smithi085 conmon[29319]: mon.0) 2141 : cluster [DBG] osdmap e1069: 8 total, 8 up, 7 in 2021-02-10T04:47:35.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:35 smithi085 conmon[35961]: cluster 2021-02-10T04:47:34.105276+0000 2021-02-10T04:47:35.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:35 smithi085 conmon[35961]: mon.a (mon.0) 2141 : cluster [DBG] osdmap e1069: 8 total, 8 up, 7 in 2021-02-10T04:47:35.548 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:47:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:36 smithi164 conmon[29288]: cluster 2021-02-10T04:47:34.341110+0000 mgr.y (mgr.14140) 2021-02-10T04:47:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:36 smithi164 conmon[29288]: 1061 : cluster [DBG] pgmap v1837: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-02-10T04:47:36.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:36 smithi085 conmon[29319]: cluster 2021-02-10T04:47:34.341110+0000 mgr.y 2021-02-10T04:47:36.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:36 smithi085 conmon[29319]: (mgr.14140) 1061 : cluster [DBG] pgmap v1837: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-02-10T04:47:36.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:36 smithi085 conmon[35961]: cluster 2021-02-10T04:47: 2021-02-10T04:47:36.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:36 smithi085 conmon[35961]: 34.341110+0000 mgr.y (mgr.14140) 1061 : cluster [DBG] pgmap v1837: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-02-10T04:47:37.950 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:47:38.462 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:38 smithi085 conmon[29319]: cluster 2021-02-10T04:47:36.341759+0000 2021-02-10T04:47:38.462 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:38 smithi085 conmon[29319]: mgr.y (mgr.14140) 1062 : cluster [DBG] pgmap v1838: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:47:38.463 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:38 smithi085 conmon[29319]: cluster 2021-02-10T04:47:37.167439+0000 osd.0 (osd.0) 145 : cluster 2021-02-10T04:47:38.463 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:38 smithi085 conmon[29319]: [DBG] purged_snaps scrub starts 2021-02-10T04:47:38.463 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:38 smithi085 conmon[29319]: cluster 2021-02-10T04:47:38.463 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:38 smithi085 conmon[29319]: 2021-02-10T04:47:37.171416+0000 osd.0 (osd. 2021-02-10T04:47:38.463 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:38 smithi085 conmon[29319]: 0) 146 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:47:38.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:38 smithi085 conmon[35961]: cluster 2021-02-10T04:47:38.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:38 smithi085 conmon[35961]: 2021-02-10T04:47:38.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:38 smithi085 conmon[35961]: 2021-02-10T04:47:36 2021-02-10T04:47:38.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:38 smithi085 conmon[35961]: .341759+0000 mgr.y (mgr.14140) 1062 : cluster [DBG] pgmap v1838: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:47:38.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:38 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:47:38.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:38 smithi085 conmon[35961]: 10T04:47:37.167439+0000 osd.0 2021-02-10T04:47:38.464 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:38 smithi085 conmon[35961]: (osd.0) 145 : cluster [DBG] 2021-02-10T04:47:38.464 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:38 smithi085 conmon[35961]: purged_snaps scrub starts 2021-02-10T04:47:38.464 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:38 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:47:38.464 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:38 smithi085 conmon[35961]: 47:37.171416+0000 osd.0 (osd.0) 146 : cluster [DBG] 2021-02-10T04:47:38.464 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:38 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:47:38.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:38 smithi164 conmon[29288]: cluster 2021-02-10T04:47:36.341759+0000 mgr.y (mgr.14140) 1062 : cluster [DBG] pgmap v1838: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:47:38.464 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:38 smithi164 conmon[29288]: cluster 2021-02-10T04:47:37.167439+0000 osd.0 (osd.0) 145 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:47:38.465 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:38 smithi164 conmon[29288]: cluster 2021-02-10T04:47:37.171416+0000 osd.0 (osd.0) 146 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:47:38.490 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:47:39.123 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:47:38 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:47:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:39 smithi164 conmon[29288]: cluster 2021-02-10T04:47:38.119320+0000 mon.a (mon.0) 2142 : cluster [DBG] osdmap e1070: 8 total, 8 up, 7 in 2021-02-10T04:47:41.867 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:39 smithi164 conmon[29288]: audit 2021-02-10T04:47:38.179823+0000 mon.a (mon.0) 2143 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:47:41.867 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:39 smithi164 conmon[29288]: audit 2021-02-10T04:47:41.867 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:39 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:47:41.867 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:39 smithi164 conmon[29288]: 47:38.180009+0000 mon.a (mon.0) 2144 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:47:41.869 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:39 smithi085 conmon[35961]: cluster 2021-02-10T04:47:38.119320+0000 mon.a ( 2021-02-10T04:47:41.869 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:39 smithi085 conmon[35961]: mon.0) 2142 : cluster [DBG] osdmap e1070: 8 total, 8 up, 7 in 2021-02-10T04:47:41.869 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:39 smithi085 conmon[35961]: audit 2021-02-10T04:47:38.179823+0000 mon.a (mon.0) 2143 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:47:41.869 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:39 smithi085 conmon[35961]: audit 2021-02-10T04:47:38.180009+0000 mon.a (mon 2021-02-10T04:47:41.869 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:39 smithi085 conmon[35961]: .0) 2144 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:47:41.869 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:40 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:47:41.869 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:40 smithi085 conmon[35961]: 10T04:47:38.342425+0000 mgr.y (mgr.14140) 1063 : cluster [DBG] pgmap v1840: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 701 B/s rd, 0 op/s 2021-02-10T04:47:41.870 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:40 smithi085 conmon[35961]: audit 2021-02-10T04:47: 2021-02-10T04:47:41.870 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:40 smithi085 conmon[35961]: 38.838018+0000 mgr.y (mgr.14140) 1064 : audit 2021-02-10T04:47:41.870 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:40 smithi085 conmon[35961]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:47:41.870 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:39 smithi085 conmon[29319]: cluster 2021-02-10T04:47:38.119320+0000 mon.a (mon.0) 2142 : cluster [DBG] osdmap e1070: 8 total, 8 up, 7 in 2021-02-10T04:47:41.870 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:39 smithi085 conmon[29319]: audit 2021-02-10T04:47:38.179823+0000 mon.a (mon.0) 2143 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:47:41.870 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:39 smithi085 conmon[29319]: audit 2021-02-10T04:47:38.180009+0000 mon.a (mon.0) 2144 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:47:41.870 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:40 smithi085 conmon[29319]: cluster 2021-02-10T04:47:41.870 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:40 smithi085 conmon[29319]: 2021-02-10T04:47:41.870 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:40 smithi085 conmon[29319]: 2021 2021-02-10T04:47:41.870 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:40 smithi085 conmon[29319]: - 2021-02-10T04:47:41.870 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:40 smithi085 conmon[29319]: 02 2021-02-10T04:47:41.870 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:40 smithi085 conmon[29319]: - 2021-02-10T04:47:41.871 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:40 smithi085 conmon[29319]: 10 2021-02-10T04:47:41.871 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:40 smithi085 conmon[29319]: T04:47:38.342425+0000 mgr.y ( 2021-02-10T04:47:41.871 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:40 smithi085 conmon[29319]: mgr.14140) 1063 : cluster [DBG] pgmap v1840: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 701 B/s rd, 0 op/s 2021-02-10T04:47:41.871 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:40 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:47:41.871 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:40 smithi085 conmon[29319]: 02-10T04:47:38.838018+0000 2021-02-10T04:47:41.871 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:40 smithi085 conmon[29319]: mgr.y (mgr.14140) 1064 : 2021-02-10T04:47:41.871 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:40 smithi085 conmon[29319]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:47:41.871 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:47:41.872 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:40 smithi164 conmon[29288]: cluster 2021-02-10T04:47:38.342425+0000 mgr.y (mgr.14140 2021-02-10T04:47:41.872 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:40 smithi164 conmon[29288]: ) 1063 : cluster [DBG] pgmap v1840: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 701 B/s rd, 0 op/s 2021-02-10T04:47:41.872 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:40 smithi164 conmon[29288]: audit 2021-02-10T04:47:38.838018+0000 mgr.y (mgr.14140) 1064 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:47:41.972 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:47:42.009 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:47:42.116 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:47:41 smithi085 conmon[52836]: debug 2021-02-10T04:47:41.762+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:47:42.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:42 smithi164 conmon[29288]: cluster 2021-02-10T04:47:40.343118 2021-02-10T04:47:42.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:42 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1065 : cluster [DBG] pgmap v1841: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:47:42.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:47:42 smithi164 conmon[40892]: debug 2021-02-10T04:47:42.091+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:47:42.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:42 smithi085 conmon[29319]: cluster 2021-02-10T04:47:40.343118 2021-02-10T04:47:42.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:42 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 1065 : cluster [DBG] pgmap v1841: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:47:42.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:42 smithi085 conmon[35961]: cluster 2021-02-10T04:47: 2021-02-10T04:47:42.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:42 smithi085 conmon[35961]: 40.343118+0000 mgr.y (mgr.14140) 1065 : cluster [DBG] pgmap v1841: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:47:42.714 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:47:43.873 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:47:43.929 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set nodeep-scrub 2021-02-10T04:47:44.182 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:47:44.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:44 smithi085 conmon[29319]: cluster 2021-02-10T04:47:44.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:44 smithi085 conmon[29319]: 2021- 2021-02-10T04:47:44.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:44 smithi085 conmon[29319]: 02-10T04:47:42.343702+0000 mgr.y (mgr.14140) 1066 : cluster [DBG] pgmap v1842: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 621 B/s rd, 0 op/s 2021-02-10T04:47:44.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:44 smithi085 conmon[35961]: cluster 2021-02-10T04:47:42.343702 2021-02-10T04:47:44.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:44 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1066 : cluster [DBG] pgmap v1842: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 621 B/s rd, 0 op/s 2021-02-10T04:47:44.271 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:47:44.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:44 smithi164 conmon[29288]: cluster 2021-02-10T04:47:42.343702+0000 2021-02-10T04:47:44.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:44 smithi164 conmon[29288]: mgr.y (mgr.14140) 1066 : cluster [DBG] pgmap v1842: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 621 B/s rd, 0 op/s 2021-02-10T04:47:45.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:45 smithi164 conmon[29288]: cluster 2021-02-10T04:47:43.953623+0000 osd.7 (osd. 2021-02-10T04:47:45.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:45 smithi164 conmon[29288]: 7) 76 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:47:45.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:45 smithi164 conmon[29288]: cluster 2021-02-10T04:47:43.955579+0000 osd.7 (osd.7) 77 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:47:45.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:45 smithi164 conmon[29288]: cluster 2021-02-10T04:47:44. 2021-02-10T04:47:45.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:45 smithi164 conmon[29288]: 423066+0000 mon.a (mon. 2021-02-10T04:47:45.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:45 smithi164 conmon[29288]: 0) 2145 : cluster [DBG] 2021-02-10T04:47:45.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:45 smithi164 conmon[29288]: osdmap e1071: 8 total, 8 up, 7 in 2021-02-10T04:47:45.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:45 smithi085 conmon[29319]: cluster 2021-02-10T04:47:43.953623+0000 osd.7 (osd.7) 76 : cluster 2021-02-10T04:47:45.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:45 smithi085 conmon[29319]: [DBG] purged_snaps scrub starts 2021-02-10T04:47:45.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:45 smithi085 conmon[29319]: cluster 2021-02-10T04:47:43.955579+0000 osd.7 (osd.7) 77 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:47:45.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:45 smithi085 conmon[29319]: cluster 2021-02-10T04:47:44.423066+0000 mon.a (mon. 2021-02-10T04:47:45.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:45 smithi085 conmon[29319]: 0) 2145 : cluster [DBG] osdmap e1071: 8 total, 8 up, 7 in 2021-02-10T04:47:45.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:45 smithi085 conmon[35961]: cluster 2021-02-10T04:47:43.953623+0000 osd.7 (osd.7) 76 2021-02-10T04:47:45.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:45 smithi085 conmon[35961]: : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:47:45.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:45 smithi085 conmon[35961]: cluster 2021-02-10T04:47:43.955579+0000 osd.7 (osd.7) 77 2021-02-10T04:47:45.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:45 smithi085 conmon[35961]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:47:45.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:45 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:47:45.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:45 smithi085 conmon[35961]: 02-10T04:47:44.423066+0000 mon.a (mon.0) 2145 : cluster [DBG] osdmap e1071: 8 total, 8 up, 7 in 2021-02-10T04:47:46.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:46 smithi164 conmon[29288]: cluster 2021-02-10T04:47:44.344369+0000 mgr.y (mgr 2021-02-10T04:47:46.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:46 smithi164 conmon[29288]: .14140) 1067 : cluster [DBG] pgmap v1843: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:47:46.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:46 smithi085 conmon[35961]: cluster 2021-02-10T04:47:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:46 smithi085 conmon[35961]: 2021-02-10T04:47:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:46 smithi085 conmon[35961]: 2021 2021-02-10T04:47:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:46 smithi085 conmon[35961]: -02-10T04:47:44.344369 2021-02-10T04:47:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:46 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1067 : cluster [DBG] pgmap v1843: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:47:46.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:46 smithi085 conmon[29319]: cluster 2021-02-10T04:47:46.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:46 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:47:46.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:46 smithi085 conmon[29319]: 04:47:44.344369+0000 mgr.y (mgr.14140) 1067 : cluster [DBG] pgmap v1843: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:47:47.816 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:47:47.929 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:47:48.095 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:47:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:48 smithi164 conmon[29288]: cluster 2021-02-10T04:47:46.345121+0000 mgr.y (mgr 2021-02-10T04:47:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:48 smithi164 conmon[29288]: .14140) 1068 : cluster [DBG] pgmap v1845: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:47:48.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:48 smithi085 conmon[29319]: cluster 2021-02-10T04:47:46.345121+0000 mgr.y (mgr.14140) 1068 : cluster [DBG] pgmap v1845: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:47:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:48 smithi085 conmon[35961]: cluster 2021-02-10T04:47:46.345121 2021-02-10T04:47:48.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:48 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1068 : cluster [DBG] pgmap v1845: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:47:48.193 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:47:48 smithi085 conmon[41540]: debug 2021-02-10T04:47:48.009+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:47:48.207 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:47:48.239 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:47:49.143 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:47:48 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:47:49.144 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:47:48 smithi164 conmon[40892]: debug 2021-02-10T04:47:49.144 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:47:48 smithi164 conmon[40892]: 2021-02-10T04:47:48.926+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:47:49.144 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:47:48 smithi164 conmon[40892]: 2021-02-10T04:47:49.223 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:47:49.330 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:47:49.388 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:47:49.692 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:47:49 smithi085 conmon[45443]: debug 2021-02-10T04:47:49.692 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:47:49 smithi085 conmon[45443]: 2021-02-10T04:47:49.458+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:47:49.692 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:47:49 smithi085 conmon[45443]: 2021-02-10T04:47:49.844 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:47:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:50 smithi085 conmon[35961]: cluster 2021-02-10T04:47: 2021-02-10T04:47:50.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:50 smithi085 conmon[35961]: 48.345919+0000 mgr.y (mgr.14140) 1069 : cluster [DBG] pgmap v1846: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:47:50.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:50 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:47:50.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:50 smithi085 conmon[35961]: T04:47:48.848033+0000 2021-02-10T04:47:50.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:50 smithi085 conmon[35961]: mgr.y (mgr.14140) 1070 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:47:50.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:50 smithi085 conmon[35961]: 2021-02-10T04:47:50.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:50 smithi085 conmon[29319]: cluster 2021-02-10T04:47:48.345919+0000 2021-02-10T04:47:50.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:50 smithi085 conmon[29319]: mgr.y (mgr.14140) 1069 : cluster [DBG] pgmap v1846: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:47:50.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:50 smithi085 conmon[29319]: audit 2021-02-10T04:47:48.848033+0000 mgr.y (mgr.14140) 1070 2021-02-10T04:47:50.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:50 smithi085 conmon[29319]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:47:50.591 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:50 smithi164 conmon[29288]: cluster 2021-02-10T04:47:48.345919+0000 mgr.y (mgr.14140) 2021-02-10T04:47:50.591 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:50 smithi164 conmon[29288]: 1069 : cluster [DBG] pgmap v1846: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:47:50.591 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:50 smithi164 conmon[29288]: audit 2021-02-10T04:47:48.848033+0000 mgr.y (mgr.14140) 1070 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:47:51.346 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:47:51.347 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:47:52.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:52 smithi164 conmon[29288]: cluster 2021-02-10T04:47:50.346672+0000 mgr.y (mgr.14140) 1071 : cluster [DBG] pgmap v1847: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:47:59.425 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:52 smithi164 conmon[29288]: cluster 2021-02-10T04:47:50.733366+0000 osd.4 (osd.4) 2135 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:47:59.425 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:52 smithi164 conmon[29288]: cluster 2021-02-10T04:47:50.734820+0000 osd.4 (osd.4) 2136 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:47:59.426 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:52 smithi164 conmon[29288]: cluster 2021-02-10T04:47:51.160532+0000 mon.a (mon.0) 2146 : cluster [DBG] osdmap e1072: 8 total, 8 up, 7 in 2021-02-10T04:47:59.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:52 smithi085 conmon[29319]: cluster 2021-02-10T04:47:50.346672+0000 mgr.y ( 2021-02-10T04:47:59.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:52 smithi085 conmon[29319]: mgr.14140) 1071 : cluster [DBG] pgmap v1847: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:47:59.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:52 smithi085 conmon[29319]: cluster 2021-02-10T04:47:50.733366+0000 osd.4 (osd.4) 2135 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:47:59.429 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:52 smithi085 conmon[29319]: cluster 2021-02-10T04:47:50.734820+0000 osd.4 (osd.4) 2136 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:47:59.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:52 smithi085 conmon[29319]: cluster 2021-02-10T04:47:51.160532+0000 mon.a (mon 2021-02-10T04:47:59.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:52 smithi085 conmon[29319]: .0) 2146 : cluster [DBG] osdmap e1072: 8 total, 8 up, 7 in 2021-02-10T04:47:59.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:53 smithi085 conmon[29319]: audit 2021-02-10T04:47:52.815732+0000 mon.a (mon 2021-02-10T04:47:59.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:53 smithi085 conmon[29319]: .0) 2147 : audit [DBG] from='client.? 172.21.15.85:0/66875775' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-02-10T04:47:59.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:54 smithi085 conmon[29319]: cluster 2021-02-10T04:47:52.347263+0000 2021-02-10T04:47:59.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:54 smithi085 conmon[29319]: mgr.y (mgr.14140) 1072 : cluster [DBG] pgmap v1849: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:47:59.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[29319]: cluster 2021-02-10T04:47:59.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[29319]: 2021-02-10T04:47:59.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[29319]: 2021 2021-02-10T04:47:59.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[29319]: - 2021-02-10T04:47:59.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[29319]: 02-10T04:47:54. 2021-02-10T04:47:59.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[29319]: 347892+0000 mgr.y (mgr.14140) 1073 : cluster [DBG] pgmap v1850: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:47:59.430 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:57 smithi085 conmon[29319]: cluster 2021-02-10T04:47:55.971087+0000 osd.1 2021-02-10T04:47:59.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:57 smithi085 conmon[29319]: (osd.1) 473 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:47:59.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:57 smithi085 conmon[29319]: cluster 2021-02-10T04:47:55.974482+0000 osd.1 (osd.1) 474 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:47:59.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:57 smithi085 conmon[29319]: cluster 2021-02-10T04:47:56.175901 2021-02-10T04:47:59.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:57 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2148 : cluster [DBG] 2021-02-10T04:47:59.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:57 smithi085 conmon[29319]: osdmap e1073: 8 total, 8 up, 7 in 2021-02-10T04:47:59.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:58 smithi085 conmon[29319]: cluster 2021-02-10T04:47:56.348289+0000 mgr.y ( 2021-02-10T04:47:59.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:47:58 smithi085 conmon[29319]: mgr.14140) 1074 : cluster [DBG] pgmap v1852: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:47:59.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:52 smithi085 conmon[35961]: cluster 2021-02-10T04:47:50.346672+0000 mgr.y (mgr 2021-02-10T04:47:59.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:52 smithi085 conmon[35961]: .14140) 1071 : cluster [DBG] pgmap v1847: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:47:59.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:52 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:47:59.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:52 smithi085 conmon[35961]: 47:50.733366+0000 osd.4 (osd.4) 2135 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:47:59.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:52 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:47:59.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:52 smithi085 conmon[35961]: 02-10T04:47:50.734820+0000 osd.4 (osd.4) 2021-02-10T04:47:59.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:52 smithi085 conmon[35961]: 2136 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:47:59.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:52 smithi085 conmon[35961]: cluster 2021-02-10T04:47:51.160532+0000 mon.a 2021-02-10T04:47:59.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:52 smithi085 conmon[35961]: (mon.0) 2146 : cluster [DBG] osdmap e1072: 8 total, 8 up, 7 in 2021-02-10T04:47:59.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:53 smithi085 conmon[35961]: audit 2021-02-10T04:47:52.815732+0000 mon.a (mon.0) 2147 2021-02-10T04:47:59.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:53 smithi085 conmon[35961]: : audit [DBG] from='client.? 172.21.15.85:0/66875775' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-02-10T04:47:59.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:54 smithi085 conmon[35961]: cluster 2021-02-10T04:47:52.347263+0000 mgr.y 2021-02-10T04:47:59.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:54 smithi085 conmon[35961]: (mgr.14140) 1072 : cluster [DBG] pgmap v1849: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:47:59.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: cluster 2021-02-10T04:47:59.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: 2021-02-10T04:47:59.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: 2021 2021-02-10T04:47:59.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: - 2021-02-10T04:47:59.432 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: 02 2021-02-10T04:47:59.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: - 2021-02-10T04:47:59.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: 10 2021-02-10T04:47:59.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: T 2021-02-10T04:47:59.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: 04 2021-02-10T04:47:59.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: : 2021-02-10T04:47:59.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: 47 2021-02-10T04:47:59.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: : 2021-02-10T04:47:59.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: 54 2021-02-10T04:47:59.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: . 2021-02-10T04:47:59.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: 347892 2021-02-10T04:47:59.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: +0000 2021-02-10T04:47:59.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: 2021-02-10T04:47:59.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: mgr.y 2021-02-10T04:47:59.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: ( 2021-02-10T04:47:59.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: mgr 2021-02-10T04:47:59.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: . 2021-02-10T04:47:59.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: 14140 2021-02-10T04:47:59.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: ) 2021-02-10T04:47:59.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: 1073 2021-02-10T04:47:59.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: : 2021-02-10T04:47:59.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: cluster 2021-02-10T04:47:59.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: 2021-02-10T04:47:59.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: [DBG] 2021-02-10T04:47:59.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: 2021-02-10T04:47:59.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: pgmap v1850: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:47:59.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:56 smithi085 conmon[35961]: 2021-02-10T04:47:59.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:57 smithi085 conmon[35961]: cluster 2021-02-10T04:47:55.971087+0000 2021-02-10T04:47:59.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:57 smithi085 conmon[35961]: osd.1 (osd.1) 473 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:47:59.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:57 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:47:59.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:57 smithi085 conmon[35961]: -10T04:47:55.974482+0000 osd.1 (osd.1) 474 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:47:59.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:57 smithi085 conmon[35961]: 2021-02-10T04:47:59.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:57 smithi085 conmon[35961]: cluster 2021-02-10T04:47:59.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:57 smithi085 conmon[35961]: 2021-02-10T04:47:56.175901+0000 mon.a (mon. 2021-02-10T04:47:59.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:57 smithi085 conmon[35961]: 0) 2148 : cluster [DBG] osdmap e1073: 8 total, 8 up, 7 in 2021-02-10T04:47:59.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:58 smithi085 conmon[35961]: cluster 2021-02-10T04:47:56.348289 2021-02-10T04:47:59.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:47:58 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1074 : cluster [DBG] pgmap v1852: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:47:59.435 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:47:59.435 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:47:59.436 INFO:teuthology.orchestra.run.smithi085.stdout: "epoch": 1072, 2021-02-10T04:47:59.436 INFO:teuthology.orchestra.run.smithi085.stdout: "fsid": "42224fce-6b57-11eb-8feb-001a4aab830c", 2021-02-10T04:47:59.436 INFO:teuthology.orchestra.run.smithi085.stdout: "created": "2021-02-10T04:20:57.933279+0000", 2021-02-10T04:47:59.436 INFO:teuthology.orchestra.run.smithi085.stdout: "modified": "2021-02-10T04:47:51.145838+0000", 2021-02-10T04:47:59.436 INFO:teuthology.orchestra.run.smithi085.stdout: "last_up_change": "2021-02-10T04:25:35.312604+0000", 2021-02-10T04:47:59.436 INFO:teuthology.orchestra.run.smithi085.stdout: "last_in_change": "2021-02-10T04:37:36.200492+0000", 2021-02-10T04:47:59.436 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T04:47:59.436 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_num": 5801984, 2021-02-10T04:47:59.436 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_set": [ 2021-02-10T04:47:59.436 INFO:teuthology.orchestra.run.smithi085.stdout: "noscrub", 2021-02-10T04:47:59.436 INFO:teuthology.orchestra.run.smithi085.stdout: "pglog_hardlimit", 2021-02-10T04:47:59.436 INFO:teuthology.orchestra.run.smithi085.stdout: "purged_snapdirs", 2021-02-10T04:47:59.436 INFO:teuthology.orchestra.run.smithi085.stdout: "recovery_deletes", 2021-02-10T04:47:59.437 INFO:teuthology.orchestra.run.smithi085.stdout: "sortbitwise" 2021-02-10T04:47:59.437 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:47:59.437 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_version": 18, 2021-02-10T04:47:59.437 INFO:teuthology.orchestra.run.smithi085.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T04:47:59.437 INFO:teuthology.orchestra.run.smithi085.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T04:47:59.437 INFO:teuthology.orchestra.run.smithi085.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T04:47:59.437 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_snapshot": "", 2021-02-10T04:47:59.437 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_max": 7, 2021-02-10T04:47:59.437 INFO:teuthology.orchestra.run.smithi085.stdout: "max_osd": 8, 2021-02-10T04:47:59.437 INFO:teuthology.orchestra.run.smithi085.stdout: "require_min_compat_client": "luminous", 2021-02-10T04:47:59.437 INFO:teuthology.orchestra.run.smithi085.stdout: "min_compat_client": "luminous", 2021-02-10T04:47:59.437 INFO:teuthology.orchestra.run.smithi085.stdout: "require_osd_release": "quincy", 2021-02-10T04:47:59.437 INFO:teuthology.orchestra.run.smithi085.stdout: "pools": [ 2021-02-10T04:47:59.437 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.438 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 1, 2021-02-10T04:47:59.438 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "device_health_metrics", 2021-02-10T04:47:59.438 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:24:05.258028+0000", 2021-02-10T04:47:59.438 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:47:59.438 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:47:59.438 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:47:59.438 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:47:59.438 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:47:59.438 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:47:59.438 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:47:59.438 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:47:59.438 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:47:59.438 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:47:59.438 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:47:59.439 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:47:59.439 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 11, 2021-02-10T04:47:59.439 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 11, 2021-02-10T04:47:59.439 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 11, 2021-02-10T04:47:59.439 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 11, 2021-02-10T04:47:59.439 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 11, 2021-02-10T04:47:59.439 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:47:59.439 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:47:59.439 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:47:59.439 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:47:59.439 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:47:59.439 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:47:59.439 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:47:59.440 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.440 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "112", 2021-02-10T04:47:59.440 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:47:59.440 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:47:59.440 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "110", 2021-02-10T04:47:59.440 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:47:59.440 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:47:59.440 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:47:59.440 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:47:59.440 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:47:59.440 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:47:59.440 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:47:59.440 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:47:59.440 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:47:59.441 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:47:59.441 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:47:59.441 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:47:59.441 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:47:59.441 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:47:59.441 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:47:59.441 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:47:59.441 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:47:59.441 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:47:59.441 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:47:59.441 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:47:59.441 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:47:59.441 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:47:59.441 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:47:59.442 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.442 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:47:59.442 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:47:59.442 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:47:59.442 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:47:59.442 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:47:59.442 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:47:59.442 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:47:59.442 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:47:59.442 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:47:59.442 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:47:59.442 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:47:59.442 INFO:teuthology.orchestra.run.smithi085.stdout: "options": { 2021-02-10T04:47:59.443 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_min": 1 2021-02-10T04:47:59.443 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.443 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:47:59.443 INFO:teuthology.orchestra.run.smithi085.stdout: "mgr_devicehealth": {} 2021-02-10T04:47:59.443 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.443 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.443 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.443 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 2, 2021-02-10T04:47:59.443 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": ".rgw.root", 2021-02-10T04:47:59.443 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:32.688554+0000", 2021-02-10T04:47:59.443 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:47:59.443 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:47:59.443 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:47:59.444 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:47:59.444 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:47:59.444 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:47:59.444 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:47:59.444 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:47:59.444 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:47:59.444 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:47:59.444 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:47:59.444 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:47:59.444 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 42, 2021-02-10T04:47:59.444 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 42, 2021-02-10T04:47:59.444 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 42, 2021-02-10T04:47:59.444 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 42, 2021-02-10T04:47:59.445 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 42, 2021-02-10T04:47:59.445 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:47:59.445 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:47:59.445 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:47:59.445 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:47:59.445 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:47:59.445 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:47:59.445 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:47:59.445 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.445 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "91", 2021-02-10T04:47:59.445 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:47:59.445 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:47:59.445 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "89", 2021-02-10T04:47:59.446 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:47:59.446 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:47:59.446 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:47:59.446 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:47:59.446 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:47:59.446 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:47:59.446 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:47:59.446 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:47:59.446 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:47:59.446 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:47:59.446 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:47:59.446 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:47:59.446 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:47:59.447 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:47:59.447 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:47:59.447 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:47:59.447 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:47:59.447 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:47:59.447 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:47:59.447 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:47:59.447 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:47:59.447 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:47:59.447 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:47:59.447 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.447 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:47:59.447 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:47:59.448 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:47:59.448 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:47:59.448 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:47:59.448 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:47:59.448 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:47:59.448 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:47:59.448 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:47:59.448 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:47:59.448 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:47:59.448 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:47:59.448 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:47:59.448 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T04:47:59.448 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.449 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.449 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.449 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 3, 2021-02-10T04:47:59.449 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "iscsi", 2021-02-10T04:47:59.449 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:47.658613+0000", 2021-02-10T04:47:59.449 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:47:59.449 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:47:59.449 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:47:59.449 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:47:59.449 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:47:59.449 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:47:59.449 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:47:59.449 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:47:59.449 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:47:59.450 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:47:59.450 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:47:59.450 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:47:59.450 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 3, 2021-02-10T04:47:59.450 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 3, 2021-02-10T04:47:59.450 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 3, 2021-02-10T04:47:59.450 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 3, 2021-02-10T04:47:59.450 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 3, 2021-02-10T04:47:59.450 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:47:59.450 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:47:59.450 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:47:59.450 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:47:59.451 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:47:59.451 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:47:59.451 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:47:59.451 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.451 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "58", 2021-02-10T04:47:59.451 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:47:59.451 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:47:59.451 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T04:47:59.451 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:47:59.451 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:47:59.451 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:47:59.451 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:47:59.451 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:47:59.452 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:47:59.452 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:47:59.452 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:47:59.452 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:47:59.452 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:47:59.452 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:47:59.452 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:47:59.452 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:47:59.452 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:47:59.452 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:47:59.452 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:47:59.452 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:47:59.452 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:47:59.452 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:47:59.453 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:47:59.453 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:47:59.453 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:47:59.453 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:47:59.453 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.453 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:47:59.453 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:47:59.453 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:47:59.453 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:47:59.453 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:47:59.453 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:47:59.453 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:47:59.453 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:47:59.454 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:47:59.454 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:47:59.454 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:47:59.454 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:47:59.454 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:47:59.454 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd": {} 2021-02-10T04:47:59.454 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.454 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.454 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.454 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 4, 2021-02-10T04:47:59.454 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "zone.rgw.log", 2021-02-10T04:47:59.454 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:47.968727+0000", 2021-02-10T04:47:59.454 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:47:59.455 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:47:59.455 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:47:59.455 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:47:59.455 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:47:59.455 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:47:59.455 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:47:59.455 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:47:59.455 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:47:59.455 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:47:59.455 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:47:59.455 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:47:59.455 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 32, 2021-02-10T04:47:59.455 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 32, 2021-02-10T04:47:59.456 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 32, 2021-02-10T04:47:59.456 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 32, 2021-02-10T04:47:59.456 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 32, 2021-02-10T04:47:59.456 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:47:59.456 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:47:59.456 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:47:59.456 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:47:59.456 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:47:59.456 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:47:59.456 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:47:59.456 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.456 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "57", 2021-02-10T04:47:59.456 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:47:59.457 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:47:59.457 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T04:47:59.457 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:47:59.457 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:47:59.457 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:47:59.457 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:47:59.457 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:47:59.457 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:47:59.457 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:47:59.457 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:47:59.457 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:47:59.457 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:47:59.457 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:47:59.458 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:47:59.458 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:47:59.458 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:47:59.458 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:47:59.458 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:47:59.458 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:47:59.458 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:47:59.458 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:47:59.458 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:47:59.458 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:47:59.458 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:47:59.458 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:47:59.458 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.459 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:47:59.459 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:47:59.459 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:47:59.459 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:47:59.459 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:47:59.459 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:47:59.459 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:47:59.459 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:47:59.459 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:47:59.459 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:47:59.459 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:47:59.459 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:47:59.459 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:47:59.459 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T04:47:59.460 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.460 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.460 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.460 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 5, 2021-02-10T04:47:59.460 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "zone.rgw.control", 2021-02-10T04:47:59.460 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:49.278260+0000", 2021-02-10T04:47:59.460 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:47:59.460 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:47:59.460 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:47:59.460 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:47:59.460 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:47:59.460 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:47:59.460 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:47:59.461 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:47:59.461 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:47:59.461 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:47:59.461 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:47:59.461 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:47:59.461 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 32, 2021-02-10T04:47:59.461 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 32, 2021-02-10T04:47:59.461 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 32, 2021-02-10T04:47:59.461 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 32, 2021-02-10T04:47:59.461 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 32, 2021-02-10T04:47:59.461 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:47:59.461 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:47:59.461 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:47:59.461 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:47:59.462 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:47:59.462 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:47:59.462 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:47:59.462 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.462 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "59", 2021-02-10T04:47:59.462 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:47:59.462 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:47:59.462 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T04:47:59.462 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:47:59.462 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:47:59.462 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:47:59.462 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:47:59.462 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:47:59.463 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:47:59.463 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:47:59.463 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:47:59.463 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:47:59.463 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:47:59.463 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:47:59.463 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:47:59.463 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:47:59.463 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:47:59.463 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:47:59.463 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:47:59.463 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:47:59.463 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:47:59.464 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:47:59.464 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:47:59.464 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:47:59.464 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:47:59.464 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:47:59.464 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.464 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:47:59.464 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:47:59.464 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:47:59.464 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:47:59.464 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:47:59.464 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:47:59.464 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:47:59.464 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:47:59.465 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:47:59.465 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:47:59.465 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:47:59.465 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:47:59.465 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:47:59.465 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T04:47:59.465 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.465 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.465 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.465 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 6, 2021-02-10T04:47:59.465 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "zone.rgw.meta", 2021-02-10T04:47:59.465 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:51.792514+0000", 2021-02-10T04:47:59.465 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:47:59.466 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:47:59.466 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:47:59.466 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:47:59.466 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:47:59.466 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:47:59.466 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:47:59.466 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:47:59.466 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:47:59.466 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:47:59.466 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:47:59.466 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:47:59.466 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 42, 2021-02-10T04:47:59.466 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 42, 2021-02-10T04:47:59.467 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 42, 2021-02-10T04:47:59.467 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 42, 2021-02-10T04:47:59.467 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 42, 2021-02-10T04:47:59.467 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:47:59.467 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:47:59.467 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:47:59.467 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:47:59.467 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:47:59.467 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:47:59.467 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:47:59.467 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.467 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "1046", 2021-02-10T04:47:59.467 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:47:59.467 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:47:59.468 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "1044", 2021-02-10T04:47:59.468 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:47:59.468 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:47:59.468 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:47:59.468 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:47:59.468 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:47:59.468 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:47:59.468 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:47:59.468 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:47:59.468 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:47:59.468 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:47:59.468 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:47:59.469 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:47:59.469 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:47:59.469 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:47:59.469 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:47:59.469 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:47:59.469 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:47:59.469 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:47:59.469 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:47:59.469 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:47:59.469 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:47:59.469 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:47:59.469 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:47:59.469 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.470 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:47:59.470 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:47:59.470 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:47:59.470 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:47:59.470 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:47:59.470 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:47:59.470 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:47:59.470 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:47:59.470 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:47:59.470 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:47:59.470 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:47:59.470 INFO:teuthology.orchestra.run.smithi085.stdout: "options": { 2021-02-10T04:47:59.470 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_bias": 4, 2021-02-10T04:47:59.471 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_min": 8 2021-02-10T04:47:59.471 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.471 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:47:59.471 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T04:47:59.471 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.471 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.471 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.471 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 7, 2021-02-10T04:47:59.471 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "unique_pool_0", 2021-02-10T04:47:59.471 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:28:23.932391+0000", 2021-02-10T04:47:59.471 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 8193, 2021-02-10T04:47:59.471 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2021-02-10T04:47:59.471 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:47:59.472 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:47:59.472 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:47:59.472 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:47:59.472 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:47:59.472 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:47:59.472 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:47:59.472 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:47:59.472 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:47:59.472 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:47:59.472 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 26, 2021-02-10T04:47:59.472 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 26, 2021-02-10T04:47:59.472 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 26, 2021-02-10T04:47:59.472 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 26, 2021-02-10T04:47:59.472 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 26, 2021-02-10T04:47:59.473 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:47:59.473 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:47:59.473 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:47:59.473 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:47:59.473 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:47:59.473 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:47:59.473 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:47:59.473 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.473 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "1009", 2021-02-10T04:47:59.473 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:47:59.473 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:47:59.473 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "1007", 2021-02-10T04:47:59.473 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:47:59.474 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:47:59.474 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 888, 2021-02-10T04:47:59.474 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 994, 2021-02-10T04:47:59.474 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:47:59.474 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:47:59.474 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:47:59.474 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:47:59.474 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:47:59.474 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:47:59.474 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:47:59.474 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:47:59.474 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:47:59.475 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:47:59.475 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:47:59.475 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:47:59.475 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:47:59.475 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:47:59.475 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:47:59.475 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:47:59.475 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:47:59.475 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:47:59.475 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:47:59.475 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.476 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:47:59.476 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:47:59.476 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:47:59.476 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:47:59.476 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:47:59.476 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:47:59.476 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:47:59.476 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:47:59.476 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:47:59.476 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:47:59.476 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:47:59.477 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:47:59.477 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:47:59.477 INFO:teuthology.orchestra.run.smithi085.stdout: "rados": {} 2021-02-10T04:47:59.477 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.477 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.477 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:47:59.477 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T04:47:59.478 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.478 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 0, 2021-02-10T04:47:59.478 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "45d0bb5f-35fa-4596-964f-75abcee53bee", 2021-02-10T04:47:59.478 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:47:59.478 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:47:59.478 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:47:59.478 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:47:59.478 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:47:59.478 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:47:59.478 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 8, 2021-02-10T04:47:59.478 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1046, 2021-02-10T04:47:59.479 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:47:59.479 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:47:59.479 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:47:59.479 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.479 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.479 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.479 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6801", 2021-02-10T04:47:59.479 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.479 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.479 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.479 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.480 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:47:59.480 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.480 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.480 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.480 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6802", 2021-02-10T04:47:59.480 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.480 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.480 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.480 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.480 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:47:59.480 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.480 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.481 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.481 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6804", 2021-02-10T04:47:59.481 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.481 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.481 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.481 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.481 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:47:59.481 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.481 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.481 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.481 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6803", 2021-02-10T04:47:59.481 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.482 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.482 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.482 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.482 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6801/7", 2021-02-10T04:47:59.482 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6802/7", 2021-02-10T04:47:59.482 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6804/7", 2021-02-10T04:47:59.482 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6803/7", 2021-02-10T04:47:59.482 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:47:59.482 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:47:59.482 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:47:59.483 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.483 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.483 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.483 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 1, 2021-02-10T04:47:59.483 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "3421a5ce-872e-4b52-adf7-6aaa595871a4", 2021-02-10T04:47:59.483 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:47:59.483 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:47:59.483 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:47:59.483 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 0, 2021-02-10T04:47:59.483 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:47:59.483 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:47:59.484 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 13, 2021-02-10T04:47:59.484 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1039, 2021-02-10T04:47:59.484 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:47:59.484 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:47:59.484 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:47:59.484 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.484 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.484 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.484 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6805", 2021-02-10T04:47:59.484 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.484 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.485 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.485 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.485 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:47:59.485 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.485 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.485 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.485 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6806", 2021-02-10T04:47:59.485 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.485 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.485 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.485 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.485 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:47:59.486 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.486 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.486 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.486 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6808", 2021-02-10T04:47:59.486 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.486 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.486 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.486 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.486 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:47:59.486 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.487 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.487 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.487 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6807", 2021-02-10T04:47:59.487 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.487 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.487 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.487 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.487 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6805/7", 2021-02-10T04:47:59.487 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6806/7", 2021-02-10T04:47:59.487 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6808/7", 2021-02-10T04:47:59.487 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6807/7", 2021-02-10T04:47:59.487 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:47:59.488 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:47:59.488 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:47:59.488 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.488 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.488 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.488 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 2, 2021-02-10T04:47:59.488 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "96e1952a-1042-4316-9fc9-86813e88b912", 2021-02-10T04:47:59.488 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:47:59.488 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 0, 2021-02-10T04:47:59.488 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 0, 2021-02-10T04:47:59.489 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:47:59.489 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:47:59.489 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:47:59.489 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 18, 2021-02-10T04:47:59.489 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 367, 2021-02-10T04:47:59.489 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:47:59.489 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:47:59.489 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:47:59.489 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.490 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.490 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.490 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6809", 2021-02-10T04:47:59.490 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.490 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.490 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.490 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.490 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:47:59.490 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.490 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.490 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.491 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6810", 2021-02-10T04:47:59.491 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.491 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.491 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.491 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.491 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:47:59.491 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.491 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.491 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.491 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6812", 2021-02-10T04:47:59.491 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.491 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.492 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.492 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.492 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:47:59.492 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.492 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.492 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.492 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6811", 2021-02-10T04:47:59.492 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.492 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.492 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.492 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.493 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6809/7", 2021-02-10T04:47:59.493 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6810/7", 2021-02-10T04:47:59.493 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6812/7", 2021-02-10T04:47:59.493 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6811/7", 2021-02-10T04:47:59.493 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:47:59.493 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:47:59.493 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:47:59.493 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.493 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.493 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.493 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 3, 2021-02-10T04:47:59.494 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "e019f49f-5dd4-4104-9ced-94c710616642", 2021-02-10T04:47:59.494 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:47:59.494 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:47:59.494 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:47:59.494 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:47:59.494 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:47:59.494 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:47:59.494 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 25, 2021-02-10T04:47:59.494 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1044, 2021-02-10T04:47:59.494 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:47:59.494 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:47:59.495 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:47:59.495 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.495 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.495 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.495 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6813", 2021-02-10T04:47:59.495 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.495 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.495 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.495 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.495 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:47:59.496 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.496 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.496 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.496 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6814", 2021-02-10T04:47:59.496 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.496 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.496 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.496 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.496 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:47:59.497 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.497 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.497 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.497 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6816", 2021-02-10T04:47:59.497 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.497 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.497 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.497 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.497 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:47:59.497 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.498 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.498 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.498 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6815", 2021-02-10T04:47:59.498 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.498 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.498 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.498 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.498 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6813/7", 2021-02-10T04:47:59.499 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6814/7", 2021-02-10T04:47:59.499 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6816/7", 2021-02-10T04:47:59.499 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6815/7", 2021-02-10T04:47:59.499 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:47:59.499 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:47:59.499 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:47:59.499 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.499 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.500 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.500 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 4, 2021-02-10T04:47:59.500 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "91c8a80a-37b6-4c30-a32a-647b2e6d801f", 2021-02-10T04:47:59.500 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:47:59.500 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:47:59.500 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:47:59.500 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:47:59.501 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:47:59.501 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:47:59.501 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 31, 2021-02-10T04:47:59.501 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1046, 2021-02-10T04:47:59.501 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:47:59.501 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:47:59.501 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:47:59.502 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.502 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.502 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.502 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6800", 2021-02-10T04:47:59.502 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.502 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.502 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.502 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.502 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:47:59.502 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.502 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.502 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.503 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6801", 2021-02-10T04:47:59.503 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.503 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.503 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.503 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.503 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:47:59.503 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.503 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.503 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.503 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6803", 2021-02-10T04:47:59.503 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.504 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.504 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.504 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.504 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:47:59.504 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.504 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.504 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.504 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6802", 2021-02-10T04:47:59.504 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.504 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.504 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.505 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.505 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6800/7", 2021-02-10T04:47:59.505 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6801/7", 2021-02-10T04:47:59.505 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6803/7", 2021-02-10T04:47:59.505 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6802/7", 2021-02-10T04:47:59.505 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:47:59.505 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:47:59.505 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:47:59.505 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.505 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.505 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.506 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 5, 2021-02-10T04:47:59.506 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e", 2021-02-10T04:47:59.506 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:47:59.506 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:47:59.506 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:47:59.506 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:47:59.506 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:47:59.506 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:47:59.506 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 37, 2021-02-10T04:47:59.506 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1046, 2021-02-10T04:47:59.506 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:47:59.507 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:47:59.507 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:47:59.507 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.507 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.507 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.507 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6804", 2021-02-10T04:47:59.507 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.507 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.507 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.507 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.507 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:47:59.507 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.508 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.508 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.508 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6805", 2021-02-10T04:47:59.508 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.508 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.508 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.508 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.508 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:47:59.508 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.508 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.508 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.509 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6807", 2021-02-10T04:47:59.509 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.509 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.509 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.509 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.509 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:47:59.509 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.509 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.509 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.509 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6806", 2021-02-10T04:47:59.509 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.510 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.510 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.510 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.510 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6804/7", 2021-02-10T04:47:59.510 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6805/7", 2021-02-10T04:47:59.510 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6807/7", 2021-02-10T04:47:59.510 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6806/7", 2021-02-10T04:47:59.510 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:47:59.510 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:47:59.510 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:47:59.510 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.511 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.511 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.511 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 6, 2021-02-10T04:47:59.511 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "adc40430-3eeb-4003-a824-343021243077", 2021-02-10T04:47:59.511 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:47:59.511 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:47:59.511 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:47:59.511 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:47:59.511 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:47:59.511 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:47:59.511 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 43, 2021-02-10T04:47:59.511 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1065, 2021-02-10T04:47:59.512 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:47:59.512 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:47:59.512 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:47:59.512 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.512 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.512 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.512 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6808", 2021-02-10T04:47:59.512 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.512 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.512 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.512 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.512 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:47:59.513 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.513 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.513 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.513 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6809", 2021-02-10T04:47:59.513 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.513 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.513 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.513 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.513 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:47:59.513 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.514 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.514 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.514 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6811", 2021-02-10T04:47:59.514 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.514 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.514 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.514 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.514 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:47:59.514 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.514 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.515 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.515 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6810", 2021-02-10T04:47:59.515 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.515 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.515 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.515 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.515 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6808/7", 2021-02-10T04:47:59.515 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6809/7", 2021-02-10T04:47:59.515 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6811/7", 2021-02-10T04:47:59.515 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6810/7", 2021-02-10T04:47:59.516 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:47:59.516 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:47:59.516 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:47:59.516 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.516 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.516 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.516 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 7, 2021-02-10T04:47:59.516 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37", 2021-02-10T04:47:59.516 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:47:59.516 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:47:59.516 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:47:59.516 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 0, 2021-02-10T04:47:59.517 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:47:59.517 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:47:59.517 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 50, 2021-02-10T04:47:59.517 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 112, 2021-02-10T04:47:59.517 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:47:59.517 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:47:59.517 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:47:59.517 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.517 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.517 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.517 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6812", 2021-02-10T04:47:59.518 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.518 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.518 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.518 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.518 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:47:59.518 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.518 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.518 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.518 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6813", 2021-02-10T04:47:59.518 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.518 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.519 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.519 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.519 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:47:59.519 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.519 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.519 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.519 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6815", 2021-02-10T04:47:59.519 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.519 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.519 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.519 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.519 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:47:59.520 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:47:59.520 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.520 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:47:59.520 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6814", 2021-02-10T04:47:59.520 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:47:59.520 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.520 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.520 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.520 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6812/7", 2021-02-10T04:47:59.521 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6813/7", 2021-02-10T04:47:59.521 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6815/7", 2021-02-10T04:47:59.521 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6814/7", 2021-02-10T04:47:59.521 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:47:59.521 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:47:59.521 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:47:59.521 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.521 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.521 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:47:59.521 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_xinfo": [ 2021-02-10T04:47:59.521 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.522 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 0, 2021-02-10T04:47:59.522 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:47:59.522 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:47:59.522 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:47:59.522 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:47:59.522 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:47:59.522 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:47:37.171420+0000", 2021-02-10T04:47:59.522 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:47:59.522 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.522 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.522 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 1, 2021-02-10T04:47:59.523 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:47:59.523 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:47:59.523 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:47:59.523 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:47:59.523 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:47:59.523 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:46:55.631100+0000", 2021-02-10T04:47:59.523 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:47:59.523 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.523 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.523 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 2, 2021-02-10T04:47:59.524 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:47:59.524 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:47:59.524 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:47:59.524 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:47:59.524 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 65536, 2021-02-10T04:47:59.524 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:47:15.292671+0000", 2021-02-10T04:47:59.524 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:47:59.524 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.524 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.525 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 3, 2021-02-10T04:47:59.525 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:47:59.525 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:47:59.525 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:47:59.525 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:47:59.525 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:47:59.525 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:47:33.397599+0000", 2021-02-10T04:47:59.525 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:47:59.525 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.525 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.526 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 4, 2021-02-10T04:47:59.526 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:47:59.526 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:47:59.526 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:47:59.526 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:47:59.526 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:47:59.527 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:47:50.734823+0000", 2021-02-10T04:47:59.527 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:47:59.527 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.527 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.527 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 5, 2021-02-10T04:47:59.527 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:47:59.528 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:47:59.528 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:47:59.528 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:47:59.528 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:47:59.528 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:47:08.044265+0000", 2021-02-10T04:47:59.528 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:47:59.528 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.528 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.528 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 6, 2021-02-10T04:47:59.528 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:47:59.528 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:47:59.529 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:47:59.529 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:47:59.529 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:47:59.529 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:47:26.214582+0000", 2021-02-10T04:47:59.529 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:47:59.529 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.529 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.529 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 7, 2021-02-10T04:47:59.529 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:47:59.529 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:47:59.529 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:47:59.529 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:47:59.530 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:47:59.530 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:47:43.955581+0000", 2021-02-10T04:47:59.530 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:47:59.530 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.530 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:47:59.530 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_upmap": [ 2021-02-10T04:47:59.530 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.530 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.1c", 2021-02-10T04:47:59.530 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T04:47:59.530 INFO:teuthology.orchestra.run.smithi085.stdout: 6, 2021-02-10T04:47:59.530 INFO:teuthology.orchestra.run.smithi085.stdout: 5, 2021-02-10T04:47:59.531 INFO:teuthology.orchestra.run.smithi085.stdout: 0 2021-02-10T04:47:59.531 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.531 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.531 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:47:59.531 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_upmap_items": [ 2021-02-10T04:47:59.531 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.531 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "2.7", 2021-02-10T04:47:59.531 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:47:59.531 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.531 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:47:59.531 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 4 2021-02-10T04:47:59.532 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.532 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.532 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.532 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.532 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "2.21", 2021-02-10T04:47:59.532 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:47:59.532 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.532 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:47:59.533 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 1 2021-02-10T04:47:59.533 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.533 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.533 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.533 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.533 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.9", 2021-02-10T04:47:59.533 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:47:59.533 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.534 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 1, 2021-02-10T04:47:59.534 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 6 2021-02-10T04:47:59.534 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.534 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.534 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 3, 2021-02-10T04:47:59.534 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 7 2021-02-10T04:47:59.534 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.535 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.535 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.535 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.535 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.b", 2021-02-10T04:47:59.535 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:47:59.535 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.535 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 1, 2021-02-10T04:47:59.535 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 5 2021-02-10T04:47:59.535 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.535 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.535 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.535 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.536 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.10", 2021-02-10T04:47:59.536 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:47:59.536 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.536 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 1, 2021-02-10T04:47:59.536 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 5 2021-02-10T04:47:59.536 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.536 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.536 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.536 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.536 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.6", 2021-02-10T04:47:59.536 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:47:59.536 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.536 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 5, 2021-02-10T04:47:59.537 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 6 2021-02-10T04:47:59.537 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.537 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.537 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.537 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.537 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.8", 2021-02-10T04:47:59.537 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:47:59.537 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.537 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 1, 2021-02-10T04:47:59.537 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 3 2021-02-10T04:47:59.537 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.537 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.537 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.538 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.538 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.10", 2021-02-10T04:47:59.538 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:47:59.538 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.538 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:47:59.538 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 3 2021-02-10T04:47:59.538 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.538 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.538 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.538 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.538 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.18", 2021-02-10T04:47:59.538 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:47:59.538 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.539 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 4, 2021-02-10T04:47:59.539 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 3 2021-02-10T04:47:59.539 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.539 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.539 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.539 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.539 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.1b", 2021-02-10T04:47:59.539 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:47:59.539 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.539 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 0, 2021-02-10T04:47:59.539 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 3 2021-02-10T04:47:59.539 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.539 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.540 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.540 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.540 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "6.1c", 2021-02-10T04:47:59.540 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:47:59.540 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.540 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 3, 2021-02-10T04:47:59.540 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 0 2021-02-10T04:47:59.540 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.540 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.540 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.540 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.540 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "7.6", 2021-02-10T04:47:59.540 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:47:59.541 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.541 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 3, 2021-02-10T04:47:59.541 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 1 2021-02-10T04:47:59.541 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.541 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.541 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.541 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.541 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "7.10", 2021-02-10T04:47:59.541 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:47:59.541 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.541 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:47:59.541 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 5 2021-02-10T04:47:59.541 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.542 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.542 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.542 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.542 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "7.12", 2021-02-10T04:47:59.542 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:47:59.542 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.542 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:47:59.542 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 5 2021-02-10T04:47:59.542 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.542 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.542 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.542 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.542 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "7.15", 2021-02-10T04:47:59.543 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:47:59.543 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:47:59.543 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:47:59.543 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 4 2021-02-10T04:47:59.543 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.543 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:47:59.543 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.543 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:47:59.543 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_temp": [], 2021-02-10T04:47:59.543 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_temp": [], 2021-02-10T04:47:59.543 INFO:teuthology.orchestra.run.smithi085.stdout: "blocklist": { 2021-02-10T04:47:59.543 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/801393150": "2021-02-11T04:21:37.851607+0000", 2021-02-10T04:47:59.544 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/3049671696": "2021-02-11T04:21:37.851607+0000", 2021-02-10T04:47:59.544 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/4056505602": "2021-02-11T04:21:20.700478+0000", 2021-02-10T04:47:59.544 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:6800/7": "2021-02-11T04:21:20.700478+0000", 2021-02-10T04:47:59.544 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/1879026062": "2021-02-11T04:21:20.700478+0000" 2021-02-10T04:47:59.544 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.544 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profiles": { 2021-02-10T04:47:59.544 INFO:teuthology.orchestra.run.smithi085.stdout: "default": { 2021-02-10T04:47:59.544 INFO:teuthology.orchestra.run.smithi085.stdout: "crush-failure-domain": "osd", 2021-02-10T04:47:59.544 INFO:teuthology.orchestra.run.smithi085.stdout: "k": "2", 2021-02-10T04:47:59.544 INFO:teuthology.orchestra.run.smithi085.stdout: "m": "1", 2021-02-10T04:47:59.544 INFO:teuthology.orchestra.run.smithi085.stdout: "plugin": "jerasure", 2021-02-10T04:47:59.544 INFO:teuthology.orchestra.run.smithi085.stdout: "ruleset-failure-domain": "osd", 2021-02-10T04:47:59.544 INFO:teuthology.orchestra.run.smithi085.stdout: "technique": "reed_sol_van" 2021-02-10T04:47:59.545 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.545 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:47:59.545 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps_queue": [], 2021-02-10T04:47:59.545 INFO:teuthology.orchestra.run.smithi085.stdout: "new_removed_snaps": [], 2021-02-10T04:47:59.545 INFO:teuthology.orchestra.run.smithi085.stdout: "new_purged_snaps": [], 2021-02-10T04:47:59.545 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_node_flags": {}, 2021-02-10T04:47:59.545 INFO:teuthology.orchestra.run.smithi085.stdout: "device_class_flags": {}, 2021-02-10T04:47:59.545 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_mode": { 2021-02-10T04:47:59.545 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_mode_enabled": false, 2021-02-10T04:47:59.545 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_bucket_count": 0, 2021-02-10T04:47:59.545 INFO:teuthology.orchestra.run.smithi085.stdout: "degraded_stretch_mode": 0, 2021-02-10T04:47:59.545 INFO:teuthology.orchestra.run.smithi085.stdout: "recovering_stretch_mode": 0, 2021-02-10T04:47:59.545 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_mode_bucket": 0 2021-02-10T04:47:59.546 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:47:59.546 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:47:59.546 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:53 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:47:59.546 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:53 smithi164 conmon[29288]: 04:47:52.815732+0000 mon.a (mon.0) 2147 : audit [DBG] from='client.? 172.21.15.85:0/66875775' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-02-10T04:47:59.546 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:54 smithi164 conmon[29288]: cluster 2021-02-10T04:47:59.546 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:54 smithi164 conmon[29288]: 2021-02-10T04:47:59.546 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:54 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:47:59.546 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:54 smithi164 conmon[29288]: :47:52.347263+0000 mgr.y (mgr.14140) 1072 : cluster [DBG] pgmap v1849: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:47:59.546 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:56 smithi164 conmon[29288]: cluster 2021-02-10T04:47:54.347892+0000 mgr.y (mgr. 2021-02-10T04:47:59.547 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:56 smithi164 conmon[29288]: 14140) 1073 : cluster [DBG] pgmap v1850: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:47:59.547 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:57 smithi164 conmon[29288]: cluster 2021-02-10T04:47:55.971087+0000 osd.1 (osd 2021-02-10T04:47:59.547 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:57 smithi164 conmon[29288]: .1) 473 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:47:59.547 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:57 smithi164 conmon[29288]: cluster 2021-02-10T04:47:55.974482+0000 osd.1 (osd.1) 474 : cluster 2021-02-10T04:47:59.547 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:57 smithi164 conmon[29288]: [DBG] purged_snaps scrub ok 2021-02-10T04:47:59.547 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:57 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:47:59.547 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:57 smithi164 conmon[29288]: 10T04:47:56.175901+0000 mon.a (mon.0) 2148 : cluster [DBG] osdmap e1073: 8 total, 8 up, 7 in 2021-02-10T04:47:59.547 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:58 smithi164 conmon[29288]: cluster 2021-02-10T04:47:56.348289+0000 mgr.y ( 2021-02-10T04:47:59.547 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:47:58 smithi164 conmon[29288]: mgr.14140) 1074 : cluster [DBG] pgmap v1852: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:47:59.547 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:47:58 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:47:59.647 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:48:00.065 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:48:00.387 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:48:00.441 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:48:00 smithi085 conmon[52836]: debug 2021-02-10T04:48:00.159+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:48:00.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:00 smithi085 conmon[29319]: cluster 2021-02-10T04:47:58.348966+0000 mgr.y (mgr 2021-02-10T04:48:00.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:00 smithi085 conmon[29319]: .14140) 1075 : cluster [DBG] pgmap v1853: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:48:00.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:00 smithi085 conmon[29319]: audit 2021-02-10T04:47:58.858051+0000 mgr.y (mgr.14140) 1076 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:48:00.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:00 smithi085 conmon[35961]: cluster 2021-02-10T04:47:58.348966 2021-02-10T04:48:00.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:00 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1075 : cluster [DBG] pgmap v1853: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:48:00.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:00 smithi085 conmon[35961]: audit 2021-02-10T04:47:58.858051+0000 2021-02-10T04:48:00.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:00 smithi085 conmon[35961]: mgr.y (mgr.14140) 1076 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:48:00.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:00 smithi085 conmon[35961]: 2021-02-10T04:48:00.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:00 smithi164 conmon[29288]: cluster 2021-02-10T04:47:58.348966+0000 mgr.y ( 2021-02-10T04:48:00.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:00 smithi164 conmon[29288]: mgr.14140) 1075 : cluster [DBG] pgmap v1853: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:48:00.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:00 smithi164 conmon[29288]: audit 2021-02-10T04:47:58.858051+0000 mgr.y (mgr.14140) 2021-02-10T04:48:00.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:00 smithi164 conmon[29288]: 1076 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:48:02.152 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.7 dump_historic_ops 2021-02-10T04:48:02.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:02 smithi164 conmon[29288]: cluster 2021-02-10T04:48:02.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:02 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:48:02.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:02 smithi164 conmon[29288]: 48:00.349651+0000 mgr.y (mgr.14140) 1077 : cluster [DBG] pgmap v1854: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:48:02.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:02 smithi085 conmon[29319]: cluster 2021-02-10T04:48:00.349651+0000 mgr.y (mgr 2021-02-10T04:48:02.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:02 smithi085 conmon[29319]: .14140) 1077 : cluster [DBG] pgmap v1854: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:48:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:02 smithi085 conmon[35961]: cluster 2021-02-10T04:48:00.349651+0000 mgr.y (mgr 2021-02-10T04:48:02.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:02 smithi085 conmon[35961]: .14140) 1077 : cluster [DBG] pgmap v1854: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:48:03.282 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:48:03.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:03 smithi164 conmon[29288]: cephadm 2021-02-10T04:48:02.244741+0000 mgr.y 2021-02-10T04:48:03.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:03 smithi164 conmon[29288]: (mgr.14140) 1078 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:48:03.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:03 smithi164 conmon[29288]: cephadm 2021-02-10T04:48:02 2021-02-10T04:48:03.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:03 smithi164 conmon[29288]: .245201+0000 mgr.y (mgr.14140) 1079 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:48:03.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:03 smithi085 conmon[29319]: cephadm 2021-02-10T04:48:02.244741+0000 2021-02-10T04:48:03.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:03 smithi085 conmon[29319]: mgr.y (mgr.14140) 1078 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:48:03.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:03 smithi085 conmon[29319]: cephadm 2021-02-10T04:48:02.245201+0000 mgr.y (mgr.14140) 1079 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:48:03.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:03 smithi085 conmon[35961]: cephadm 2021-02-10T04:48:02.244741+0000 mgr.y 2021-02-10T04:48:03.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:03 smithi085 conmon[35961]: (mgr.14140) 1078 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:48:03.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:03 smithi085 conmon[35961]: cephadm 2021-02-10T04:48:02.245201 2021-02-10T04:48:03.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:03 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1079 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:48:04.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:04 smithi164 conmon[29288]: cluster 2021-02-10T04:48:02.350168+0000 mgr.y (mgr.14140 2021-02-10T04:48:04.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:04 smithi164 conmon[29288]: ) 1080 : cluster [DBG] pgmap v1855: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:48:04.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:04 smithi164 conmon[29288]: audit 2021-02-10T04:48:03.739252+0000 mon.a (mon.0) 2149 : audit 2021-02-10T04:48:04.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:04 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:48:04.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:04 smithi085 conmon[29319]: cluster 2021-02-10T04:48:02.350168+0000 mgr.y (mgr.14140) 1080 : cluster [DBG] 2021-02-10T04:48:04.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:04 smithi085 conmon[29319]: pgmap v1855: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:48:04.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:04 smithi085 conmon[29319]: audit 2021-02-10T04:48:03.739252+0000 mon.a (mon.0) 2149 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:48:04.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:04 smithi085 conmon[35961]: cluster 2021-02-10T04:48:02.350168+0000 mgr.y (mgr.14140) 1080 : cluster [DBG] pgmap v1855: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:48:04.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:04 smithi085 conmon[35961]: audit 2021-02-10T04:48:03.739252+0000 mon.a (mon.0) 2149 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:48:06.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:06 smithi164 conmon[29288]: cluster 2021-02-10T04:48:06.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:06 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:48:06.469 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:06 smithi164 conmon[29288]: :48:04.350863+0000 mgr.y (mgr.14140) 1081 : cluster [DBG] pgmap v1856: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:48:06.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:06 smithi085 conmon[29319]: cluster 2021-02-10T04:48:04.350863+0000 mgr.y ( 2021-02-10T04:48:06.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:06 smithi085 conmon[29319]: mgr.14140) 1081 : cluster [DBG] pgmap v1856: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:48:06.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:06 smithi085 conmon[35961]: cluster 2021-02-10T04:48:06.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:06 smithi085 conmon[35961]: 2021-02-10T04:48:06.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:06 smithi085 conmon[35961]: 2021 2021-02-10T04:48:06.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:06 smithi085 conmon[35961]: - 2021-02-10T04:48:06.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:06 smithi085 conmon[35961]: 02 2021-02-10T04:48:06.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:06 smithi085 conmon[35961]: - 2021-02-10T04:48:06.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:06 smithi085 conmon[35961]: 10 2021-02-10T04:48:06.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:06 smithi085 conmon[35961]: T 2021-02-10T04:48:06.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:06 smithi085 conmon[35961]: 04 2021-02-10T04:48:06.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:06 smithi085 conmon[35961]: : 2021-02-10T04:48:06.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:06 smithi085 conmon[35961]: 48 2021-02-10T04:48:06.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:06 smithi085 conmon[35961]: : 2021-02-10T04:48:06.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:06 smithi085 conmon[35961]: 04 2021-02-10T04:48:06.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:06 smithi085 conmon[35961]: . 2021-02-10T04:48:06.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:06 smithi085 conmon[35961]: 350863 2021-02-10T04:48:06.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:06 smithi085 conmon[35961]: +0000 2021-02-10T04:48:06.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:06 smithi085 conmon[35961]: 2021-02-10T04:48:06.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:06 smithi085 conmon[35961]: mgr.y 2021-02-10T04:48:06.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:06 smithi085 conmon[35961]: ( 2021-02-10T04:48:06.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:06 smithi085 conmon[35961]: mgr 2021-02-10T04:48:06.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:06 smithi085 conmon[35961]: .14140) 1081 : cluster [DBG] pgmap v1856: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:48:06.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:06 smithi085 conmon[35961]: 2021-02-10T04:48:07.676 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:48:07.708 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:48:08.140 INFO:tasks.thrashosds.thrasher:j is {'epoch': 1072, 'fsid': '42224fce-6b57-11eb-8feb-001a4aab830c', 'created': '2021-02-10T04:20:57.933279+0000', 'modified': '2021-02-10T04:47:51.145838+0000', 'last_up_change': '2021-02-10T04:25:35.312604+0000', 'last_in_change': '2021-02-10T04:37:36.200492+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-02-10T04:24:05.258028+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '110', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2021-02-10T04:25:32.688554+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 42, 'pg_placement_num': 42, 'pg_placement_num_target': 42, 'pg_num_target': 42, 'pg_num_pending': 42, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '89', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2021-02-10T04:25:47.658613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'zone.rgw.log', 'create_time': '2021-02-10T04:25:47.968727+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'zone.rgw.control', 'create_time': '2021-02-10T04:25:49.278260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'zone.rgw.meta', 'create_time': '2021-02-10T04:25:51.792514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 42, 'pg_placement_num': 42, 'pg_placement_num_target': 42, 'pg_num_target': 42, 'pg_num_pending': 42, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1046', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '1044', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 8}, 'application_metadata': {'rgw': {}}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2021-02-10T04:28:23.932391+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1009', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '1007', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 888, 'snap_epoch': 994, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '45d0bb5f-35fa-4596-964f-75abcee53bee', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 1046, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6801', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6802', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6804', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6803', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6801/7', 'cluster_addr': '172.21.15.85:6802/7', 'heartbeat_back_addr': '172.21.15.85:6804/7', 'heartbeat_front_addr': '172.21.15.85:6803/7', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '3421a5ce-872e-4b52-adf7-6aaa595871a4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 1039, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6805', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6806', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6808', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6807', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6805/7', 'cluster_addr': '172.21.15.85:6806/7', 'heartbeat_back_addr': '172.21.15.85:6808/7', 'heartbeat_front_addr': '172.21.15.85:6807/7', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '96e1952a-1042-4316-9fc9-86813e88b912', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 367, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6809', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6810', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6812', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6811', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6809/7', 'cluster_addr': '172.21.15.85:6810/7', 'heartbeat_back_addr': '172.21.15.85:6812/7', 'heartbeat_front_addr': '172.21.15.85:6811/7', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'e019f49f-5dd4-4104-9ced-94c710616642', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 1044, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6813', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6814', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6816', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6815', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6813/7', 'cluster_addr': '172.21.15.85:6814/7', 'heartbeat_back_addr': '172.21.15.85:6816/7', 'heartbeat_front_addr': '172.21.15.85:6815/7', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '91c8a80a-37b6-4c30-a32a-647b2e6d801f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 1046, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6800', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6801', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6803', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6802', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6800/7', 'cluster_addr': '172.21.15.164:6801/7', 'heartbeat_back_addr': '172.21.15.164:6803/7', 'heartbeat_front_addr': '172.21.15.164:6802/7', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 1046, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6804', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6805', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6807', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6806', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6804/7', 'cluster_addr': '172.21.15.164:6805/7', 'heartbeat_back_addr': '172.21.15.164:6807/7', 'heartbeat_front_addr': '172.21.15.164:6806/7', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'adc40430-3eeb-4003-a824-343021243077', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 43, 'up_thru': 1065, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6808', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6809', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6811', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6810', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6808/7', 'cluster_addr': '172.21.15.164:6809/7', 'heartbeat_back_addr': '172.21.15.164:6811/7', 'heartbeat_front_addr': '172.21.15.164:6810/7', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 112, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6812', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6813', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6815', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6814', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6812/7', 'cluster_addr': '172.21.15.164:6813/7', 'heartbeat_back_addr': '172.21.15.164:6815/7', 'heartbeat_front_addr': '172.21.15.164:6814/7', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:47:37.171420+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:46:55.631100+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 65536, 'last_purged_snaps_scrub': '2021-02-10T04:47:15.292671+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:47:33.397599+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:47:50.734823+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:47:08.044265+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:47:26.214582+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:47:43.955581+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '4.1c', 'osds': [6, 5, 0]}], 'pg_upmap_items': [{'pgid': '2.7', 'mappings': [{'from': 7, 'to': 4}]}, {'pgid': '2.21', 'mappings': [{'from': 7, 'to': 1}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 6}, {'from': 3, 'to': 7}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.10', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '5.6', 'mappings': [{'from': 5, 'to': 6}]}, {'pgid': '5.8', 'mappings': [{'from': 1, 'to': 3}]}, {'pgid': '5.10', 'mappings': [{'from': 7, 'to': 3}]}, {'pgid': '5.18', 'mappings': [{'from': 4, 'to': 3}]}, {'pgid': '5.1b', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '6.1c', 'mappings': [{'from': 3, 'to': 0}]}, {'pgid': '7.6', 'mappings': [{'from': 3, 'to': 1}]}, {'pgid': '7.10', 'mappings': [{'from': 7, 'to': 5}]}, {'pgid': '7.12', 'mappings': [{'from': 7, 'to': 5}]}, {'pgid': '7.15', 'mappings': [{'from': 7, 'to': 4}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.85:0/801393150': '2021-02-11T04:21:37.851607+0000', '172.21.15.85:0/3049671696': '2021-02-11T04:21:37.851607+0000', '172.21.15.85:0/4056505602': '2021-02-11T04:21:20.700478+0000', '172.21.15.85:6800/7': '2021-02-11T04:21:20.700478+0000', '172.21.15.85:0/1879026062': '2021-02-11T04:21:20.700478+0000'}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2021-02-10T04:48:08.141 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 4.1c 2021-02-10T04:48:08.141 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd rm-pg-upmap 4.1c 2021-02-10T04:48:08.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:07 smithi164 conmon[33677]: debug 2021-02-10T04:48:08.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:07 smithi164 conmon[33677]: 2021-02-10T04:48:07.786+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:48:08.233 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:48:08.432 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:48:08.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:08 smithi085 conmon[29319]: cluster 2021-02-10T04:48:06.351503 2021-02-10T04:48:08.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:08 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 1082 : cluster [DBG] pgmap v1857: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1006 B/s rd, 0 op/s 2021-02-10T04:48:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:08 smithi085 conmon[35961]: cluster 2021-02-10T04:48:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:08 smithi085 conmon[35961]: 2021-02-10T04:48:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:08 smithi085 conmon[35961]: 2021 2021-02-10T04:48:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:08 smithi085 conmon[35961]: - 2021-02-10T04:48:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:08 smithi085 conmon[35961]: 02 2021-02-10T04:48:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:08 smithi085 conmon[35961]: - 2021-02-10T04:48:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:08 smithi085 conmon[35961]: 10T04:48:06.351503+0000 2021-02-10T04:48:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:08 smithi085 conmon[35961]: mgr.y (mgr.14140) 1082 : cluster [DBG] pgmap v1857: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1006 B/s rd, 0 op/s 2021-02-10T04:48:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:08 smithi164 conmon[29288]: cluster 2021-02-10T04:48:06.351503+0000 mgr.y ( 2021-02-10T04:48:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:08 smithi164 conmon[29288]: mgr.14140) 1082 : cluster [DBG] pgmap v1857: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1006 B/s rd, 0 op/s 2021-02-10T04:48:09.144 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:48:08 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:48:09.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:09 smithi164 conmon[29288]: cluster 2021-02-10T04:48:08.184368+0000 osd.5 (osd.5) 2021-02-10T04:48:20.577 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:09 smithi164 conmon[29288]: 136 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:48:20.578 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:09 smithi164 conmon[29288]: cluster 2021-02-10T04:48:08.187048+0000 osd.5 (osd.5) 137 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:48:20.578 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:09 smithi164 conmon[29288]: cluster 2021-02-10T04:48:08.218434+0000 2021-02-10T04:48:20.578 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:09 smithi164 conmon[29288]: mon.a (mon.0) 2150 : cluster [DBG] osdmap e1074: 8 total, 8 up, 7 in 2021-02-10T04:48:20.589 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[29319]: cluster 2021-02-10T04:48:20.589 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[29319]: 2021-02-10T04:48:20.590 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[29319]: 2021 2021-02-10T04:48:20.590 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[29319]: -02-10T04 2021-02-10T04:48:20.590 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[29319]: :48:08.184368+0000 osd.5 (osd.5) 136 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:48:20.590 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[29319]: 2021-02-10T04:48:20.590 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:48:20.590 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[29319]: -02-10T04:48:08.187048+0000 osd.5 2021-02-10T04:48:20.591 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[29319]: (osd.5) 137 : cluster 2021-02-10T04:48:20.591 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[29319]: [DBG] purged_snaps scrub ok 2021-02-10T04:48:20.591 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[29319]: cluster 2021-02-10T04:48:20.591 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[29319]: 2021-02-10T04:48:08.218434 2021-02-10T04:48:20.591 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2150 : cluster 2021-02-10T04:48:20.591 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[29319]: [DBG] osdmap e1074: 8 total, 8 up, 7 in 2021-02-10T04:48:20.591 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:10 smithi085 conmon[29319]: cluster 2021-02-10T04:48:08.351885+0000 2021-02-10T04:48:20.591 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:10 smithi085 conmon[29319]: mgr.y (mgr.14140) 1083 : cluster [DBG] pgmap v1859: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:48:20.592 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:10 smithi085 conmon[29319]: audit 2021-02-10T04:48:08.868142+0000 mgr.y (mgr.14140) 1084 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:48:20.592 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:11 smithi085 conmon[29319]: audit 2021-02-10T04:48:10.496863+0000 mon.b (mon. 2021-02-10T04:48:20.592 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:11 smithi085 conmon[29319]: 2) 47 : audit [DBG] from='client.? 172.21.15.85:0/2459199104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:48:20.592 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:11 smithi085 conmon[29319]: audit 2021-02-10T04:48:10.812014+0000 mon.c (mon.1) 79 : audit [INF] from='client.? 172.21.15.85:0/3053369715' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:48:20.592 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:11 smithi085 conmon[29319]: audit 2021-02-10T04:48:10.812457 2021-02-10T04:48:20.592 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:11 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2151 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:48:20.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:11 smithi085 conmon[29319]: 2021-02-10T04:48:20.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[29319]: cluster 2021-02-10T04:48:10.352606+0000 mgr.y (mgr. 2021-02-10T04:48:20.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[29319]: 14140) 1085 : cluster [DBG] pgmap v1860: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:48:20.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[29319]: cluster 2021-02-10T04:48:11.225811+0000 mon.a (mon.0) 2152 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:48:20.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[29319]: audit 2021-02-10T04:48:11.239895 2021-02-10T04:48:20.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2153 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:48:20.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:48:20.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[29319]: -10T04:48:11.240087+0000 mon.a ( 2021-02-10T04:48:20.593 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[29319]: mon.0) 2154 : cluster [DBG] osdmap e1075: 8 total, 8 up, 7 in 2021-02-10T04:48:20.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:48:20.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[29319]: 02-10T04:48:11.612854+0000 osd.4 2021-02-10T04:48:20.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[29319]: (osd.4) 2137 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:20.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[29319]: 2021-02-10T04:48:20.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[29319]: cluster 2021-02-10T04:48:20.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[29319]: 2021-02-10T04:48:11.612859+0000 osd.4 (osd 2021-02-10T04:48:20.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[29319]: .4) 2138 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:20.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:48:20.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[29319]: -10T04:48:11.612868+0000 osd.4 2021-02-10T04:48:20.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[29319]: (osd.4) 2139 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:20.594 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:48:20.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[29319]: -10T04:48:11.612873+0000 osd.4 (osd.4) 2140 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:20.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:13 smithi085 conmon[29319]: cluster 2021-02-10T04:48 2021-02-10T04:48:20.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:13 smithi085 conmon[29319]: :12.634190+0000 osd.4 (osd.4) 2141 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:20.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:13 smithi085 conmon[29319]: cluster 2021-02-10T04:48:12.634199+0000 osd.4 (osd.4 2021-02-10T04:48:20.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:13 smithi085 conmon[29319]: ) 2142 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:20.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:13 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:48:20.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:13 smithi085 conmon[29319]: -02-10T04:48:12.634214+0000 osd.4 (osd.4) 2143 : 2021-02-10T04:48:20.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:13 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:20.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:13 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:48:20.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:13 smithi085 conmon[29319]: 48:12.634220+0000 osd.4 (osd.4 2021-02-10T04:48:20.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:13 smithi085 conmon[29319]: ) 2144 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:20.595 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[29319]: cluster 2021-02-10T04:48:12.353100+0000 mgr.y 2021-02-10T04:48:20.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[29319]: (mgr.14140) 1086 : cluster [DBG] pgmap v1862: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:48:20.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[29319]: cluster 2021-02-10T04:48:13.236034+0000 mon.a (mon.0) 2155 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 227 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:48:20.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[29319]: cluster 2021-02-10T04:48:13.632185+0000 2021-02-10T04:48:20.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[29319]: osd.4 (osd.4) 2145 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:20.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:48:20.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[29319]: 10T04:48:13.632194+0000 osd.4 (osd. 2021-02-10T04:48:20.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[29319]: 4) 2146 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:20.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[29319]: cluster 2021-02-10T04:48:20.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[29319]: 2021-02 2021-02-10T04:48:20.596 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[29319]: -10T04:48:13.632201+0000 osd.4 (osd.4) 2147 2021-02-10T04:48:20.597 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:20.597 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[29319]: cluster 2021-02-10T04:48:13.632208+0000 osd.4 (osd 2021-02-10T04:48:20.597 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[29319]: .4) 2148 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:20.598 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:15 smithi085 conmon[29319]: cluster 2021-02-10T04:48:14.671169+0000 osd.4 2021-02-10T04:48:20.598 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:15 smithi085 conmon[29319]: (osd.4) 2149 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:20.598 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:15 smithi085 conmon[29319]: cluster 2021-02-10T04:48:14.671179+0000 osd.4 (osd.4) 2150 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:20.598 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:15 smithi085 conmon[29319]: 2021-02-10T04:48:20.598 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:15 smithi085 conmon[29319]: cluster 2021-02-10T04:48:14.671185+0000 osd.4 (osd.4) 2151 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:20.598 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:15 smithi085 conmon[29319]: cluster 2021-02-10T04:48:14.671190+0000 osd.4 (osd 2021-02-10T04:48:20.598 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:15 smithi085 conmon[29319]: .4) 2152 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:20.599 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:16 smithi085 conmon[29319]: cluster 2021-02-10T04:48:14.353713+0000 mgr.y (mgr.14140) 1087 : cluster [DBG] 2021-02-10T04:48:20.599 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:16 smithi085 conmon[29319]: pgmap v1863: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:48:20.599 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:16 smithi085 conmon[29319]: cluster 2021-02-10T04:48:15.677339+0000 osd.4 (osd.4) 2153 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:20.599 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:16 smithi085 conmon[29319]: cluster 2021-02-10T04:48:15.677347+0000 osd.4 (osd 2021-02-10T04:48:20.599 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:16 smithi085 conmon[29319]: .4) 2154 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:20.599 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:16 smithi085 conmon[29319]: cluster 2021-02-10T04:48:15.677355 2021-02-10T04:48:20.599 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:16 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2155 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:20.599 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:16 smithi085 conmon[29319]: cluster 2021-02-10T04:48:15.677361+0000 osd.4 (osd.4 2021-02-10T04:48:20.599 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:16 smithi085 conmon[29319]: ) 2156 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:20.599 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[29319]: cluster 2021-02-10T04:48:15.875179+0000 2021-02-10T04:48:20.599 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[29319]: osd.2 (osd.2) 1953 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:48:20.599 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[29319]: cluster 2021-02-10T04:48:15.875867+0000 osd.2 (osd.2) 1954 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:48:20.600 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[29319]: cluster 2021-02-10T04:48:16.258318+0000 2021-02-10T04:48:20.600 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[29319]: mon.a (mon.0) 2156 : cluster [DBG] osdmap e1076: 8 total, 8 up, 7 in 2021-02-10T04:48:20.600 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:48:20.600 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[29319]: 04:48:16.720848+0000 osd.4 (osd.4) 2157 : cluster [WRN] 2021-02-10T04:48:20.600 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:20.600 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:48:20.600 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[29319]: T04:48:16.720863+0000 osd.4 (osd.4) 2158 : cluster [WRN] 2021-02-10T04:48:20.601 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:20.601 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[29319]: cluster 2021-02-10T04:48:20.601 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[29319]: 2021-02-10T04:48:16.720873+0000 osd.4 (osd.4) 2021-02-10T04:48:20.601 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[29319]: 2159 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:20.601 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:48:20.601 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[29319]: T04:48:16.720883+0000 osd.4 (osd.4 2021-02-10T04:48:20.601 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[29319]: ) 2160 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:20.601 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:18 smithi085 conmon[29319]: cluster 2021-02-10T04:48:16.354097 2021-02-10T04:48:20.601 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:18 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 1088 : cluster [DBG] pgmap v1865: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:48:20.601 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:18 smithi085 conmon[29319]: cluster 2021-02-10T04:48:17.751332+0000 osd.4 (osd.4) 2161 : cluster [WRN] 2021-02-10T04:48:20.601 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:18 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:20.602 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:18 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:48:20.602 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:18 smithi085 conmon[29319]: -02-10T04:48:17.751340+0000 osd.4 2021-02-10T04:48:20.602 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:18 smithi085 conmon[29319]: (osd.4) 2162 : cluster [WRN] 2021-02-10T04:48:20.602 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:18 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:20.602 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:18 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:48:20.602 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:18 smithi085 conmon[29319]: -02-10T04:48:17.751348+0000 2021-02-10T04:48:20.602 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:18 smithi085 conmon[29319]: osd.4 (osd.4) 2163 : cluster 2021-02-10T04:48:20.602 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:18 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:20.602 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:18 smithi085 conmon[29319]: cluster 2021-02-10T04:48:20.602 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:18 smithi085 conmon[29319]: 2021-02-10T04:48 2021-02-10T04:48:20.602 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:18 smithi085 conmon[29319]: :17.751355+0000 osd.4 (osd. 2021-02-10T04:48:20.603 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:18 smithi085 conmon[29319]: 4) 2164 : cluster [WRN] 2021-02-10T04:48:20.603 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:18 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:20.603 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[29319]: cluster 2021-02-10T04:48:18.354655+0000 mgr.y ( 2021-02-10T04:48:20.603 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[29319]: mgr.14140) 1089 : cluster [DBG] pgmap v1866: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:48:20.603 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[29319]: cluster 2021-02-10T04:48:18.718117+0000 osd.4 (osd.4) 2165 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:20.603 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:48:20.603 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[29319]: 04:48:18.718126+0000 osd.4 (osd.4 2021-02-10T04:48:20.603 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[29319]: ) 2166 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:20.603 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:48:20.603 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[29319]: -02-10T04:48:18.718132+0000 osd.4 (osd.4) 2167 : cluster [WRN] 2021-02-10T04:48:20.603 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:20.603 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:48:20.604 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[29319]: 04:48:18.718138+0000 2021-02-10T04:48:20.604 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[29319]: osd.4 (osd.4) 2168 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:20.604 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[29319]: 2021-02-10T04:48:20.604 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[29319]: audit 2021-02-10T04:48:18 2021-02-10T04:48:20.604 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[29319]: .878058+0000 mgr.y (mgr.14140) 1090 : 2021-02-10T04:48:20.604 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[29319]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:48:20.604 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:48:20.604 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[29319]: T04:48:19.266104+0000 mon.a ( 2021-02-10T04:48:20.604 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[29319]: mon.0) 2157 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 232 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:48:20.604 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: cluster 2021-02-10T04:48:20.604 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: 2021-02-10T04:48:20.605 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: 2021 2021-02-10T04:48:20.605 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: - 2021-02-10T04:48:20.605 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: 02 2021-02-10T04:48:20.605 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: - 2021-02-10T04:48:20.605 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: 10 2021-02-10T04:48:20.605 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: T 2021-02-10T04:48:20.605 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: 04 2021-02-10T04:48:20.605 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: : 2021-02-10T04:48:20.605 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: 48 2021-02-10T04:48:20.605 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: : 2021-02-10T04:48:20.605 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: 08 2021-02-10T04:48:20.605 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: . 2021-02-10T04:48:20.606 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: 184368 2021-02-10T04:48:20.606 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: +0000 2021-02-10T04:48:20.606 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: 2021-02-10T04:48:20.606 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: osd.5 2021-02-10T04:48:20.606 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: ( 2021-02-10T04:48:20.606 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: osd 2021-02-10T04:48:20.606 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: . 2021-02-10T04:48:20.606 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: 5 2021-02-10T04:48:20.606 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: ) 2021-02-10T04:48:20.606 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: 136 2021-02-10T04:48:20.606 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: : 2021-02-10T04:48:20.606 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: cluster [DBG] purged_snaps scrub starts 2021-02-10T04:48:20.607 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:48:20.607 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: 02-10T04:48:08.187048+0000 osd.5 (osd 2021-02-10T04:48:20.607 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: .5) 137 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:48:20.607 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:48:20.607 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: -02-10T04:48:08.218434+0000 mon.a (mon. 2021-02-10T04:48:20.607 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:09 smithi085 conmon[35961]: 0) 2150 : cluster [DBG] osdmap e1074: 8 total, 8 up, 7 in 2021-02-10T04:48:20.607 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:10 smithi085 conmon[35961]: cluster 2021-02-10T04:48:20.607 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:10 smithi085 conmon[35961]: 2021-02-10T04:48:20.607 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:10 smithi085 conmon[35961]: 2021 2021-02-10T04:48:20.607 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:10 smithi085 conmon[35961]: - 2021-02-10T04:48:20.607 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:10 smithi085 conmon[35961]: 02 2021-02-10T04:48:20.607 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:10 smithi085 conmon[35961]: - 2021-02-10T04:48:20.608 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:10 smithi085 conmon[35961]: 10 2021-02-10T04:48:20.608 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:10 smithi085 conmon[35961]: T04:48:08.351885 2021-02-10T04:48:20.608 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:10 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1083 : cluster [DBG] pgmap v1859: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:48:20.608 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:10 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:48:20.608 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:10 smithi085 conmon[35961]: 02-10T04:48:08.868142+0000 mgr.y (mgr.14140) 1084 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:48:20.608 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:10 smithi085 conmon[35961]: 2021-02-10T04:48:20.608 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:11 smithi085 conmon[35961]: audit 2021-02-10T04:48:10.496863+0000 mon.b (mon. 2021-02-10T04:48:20.608 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:11 smithi085 conmon[35961]: 2) 47 : audit [DBG] from='client.? 172.21.15.85:0/2459199104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:48:20.608 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:11 smithi085 conmon[35961]: audit 2021-02-10T04:48:10.812014+0000 mon.c (mon.1) 79 : audit [INF] from='client.? 172.21.15.85:0/3053369715' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:48:20.608 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:11 smithi085 conmon[35961]: audit 2021-02-10T04:48:10.812457+0000 mon.a (mon.0 2021-02-10T04:48:20.608 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:11 smithi085 conmon[35961]: ) 2151 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:48:20.609 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[35961]: cluster 2021-02-10T04:48:10.352606+0000 2021-02-10T04:48:20.609 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[35961]: mgr.y (mgr.14140) 1085 : cluster [DBG] pgmap v1860: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:48:20.609 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[35961]: cluster 2021-02-10T04:48:20.609 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[35961]: 2021-02-10T04:48 2021-02-10T04:48:20.609 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[35961]: :11.225811+0000 mon.a (mon.0) 2152 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:48:20.609 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:48:20.609 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[35961]: -10T04:48:11.239895+0000 mon.a (mon.0) 2153 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:48:20.610 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[35961]: 2021-02-10T04:48:20.610 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:48:20.610 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[35961]: 02-10T04:48:11.240087+0000 mon.a (mon.0 2021-02-10T04:48:20.610 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[35961]: ) 2154 : cluster [DBG] osdmap e1075: 8 total, 8 up, 7 in 2021-02-10T04:48:20.610 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:48:20.610 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[35961]: 02-10T04:48:11.612854+0000 osd.4 (osd.4 2021-02-10T04:48:20.610 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[35961]: ) 2137 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:20.610 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:48:20.610 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[35961]: 02-10T04:48:11.612859+0000 osd.4 (osd. 2021-02-10T04:48:20.610 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[35961]: 4) 2138 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:20.610 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:48:20.610 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[35961]: 02-10T04:48:11.612868+0000 osd.4 (osd. 2021-02-10T04:48:20.611 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[35961]: 4) 2139 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:20.611 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:48:20.611 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[35961]: 02-10T04:48:11.612873+0000 osd.4 (osd. 2021-02-10T04:48:20.611 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:12 smithi085 conmon[35961]: 4) 2140 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:20.611 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:13 smithi085 conmon[35961]: cluster 2021-02-10T04:48:12.634190+0000 osd.4 (osd. 2021-02-10T04:48:20.611 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:13 smithi085 conmon[35961]: 4) 2141 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:20.611 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:13 smithi085 conmon[35961]: cluster 2021-02-10T04:48:20.611 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:13 smithi085 conmon[35961]: 2021 2021-02-10T04:48:20.611 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:13 smithi085 conmon[35961]: -02-10T04:48: 2021-02-10T04:48:20.611 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:13 smithi085 conmon[35961]: 12.634199+0000 osd.4 (osd.4) 2142 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:20.611 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:13 smithi085 conmon[35961]: cluster 2021-02-10T04:48 2021-02-10T04:48:20.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:13 smithi085 conmon[35961]: :12.634214+0000 osd.4 (osd.4) 2143 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:20.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:13 smithi085 conmon[35961]: cluster 2021-02-10T04:48:20.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:13 smithi085 conmon[35961]: 2021-02-10T04:48:12. 2021-02-10T04:48:20.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:13 smithi085 conmon[35961]: 634220+0000 osd.4 (osd.4) 2144 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:20.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: cluster 2021-02-10T04:48:20.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: 2021-02-10T04:48:20.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: 2021 2021-02-10T04:48:20.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: - 2021-02-10T04:48:20.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: 02 2021-02-10T04:48:20.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: - 2021-02-10T04:48:20.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: 10 2021-02-10T04:48:20.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: T 2021-02-10T04:48:20.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: 04 2021-02-10T04:48:20.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: : 2021-02-10T04:48:20.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: 48 2021-02-10T04:48:20.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: : 2021-02-10T04:48:20.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: 12 2021-02-10T04:48:20.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: . 2021-02-10T04:48:20.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: 353100 2021-02-10T04:48:20.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: +0000 2021-02-10T04:48:20.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: 2021-02-10T04:48:20.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: mgr.y (mgr.14140) 1086 : cluster 2021-02-10T04:48:20.613 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: [DBG] pgmap v1862: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:48:20.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: cluster 2021-02-10T04:48:20.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:48:20.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: 04:48:13.236034+0000 mon.a (mon.0) 2155 : cluster 2021-02-10T04:48:20.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: [WRN] Health check failed: 4 slow ops, oldest one blocked for 227 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:48:20.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:48:20.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: -02-10T04:48:13.632185+0000 2021-02-10T04:48:20.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: osd.4 (osd.4) 2145 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:20.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:48:20.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: 02-10T04:48:13. 2021-02-10T04:48:20.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: 632194+0000 osd.4 (osd.4) 2146 : cluster 2021-02-10T04:48:20.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:20.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: cluster 2021-02-10T04:48:20.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: 2021-02-10T04:48:13. 2021-02-10T04:48:20.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: 632201+0000 osd.4 (osd.4) 2147 : cluster [WRN] 2021-02-10T04:48:20.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:20.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: cluster 2021-02-10T04:48 2021-02-10T04:48:20.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: :13.632208+0000 osd.4 (osd.4) 2148 2021-02-10T04:48:20.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:14 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:20.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:15 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:48:20.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:15 smithi085 conmon[35961]: :48:14.671169+0000 osd.4 (osd.4) 2149 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:20.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:15 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:48:20.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:15 smithi085 conmon[35961]: T04:48:14.671179+0000 osd.4 (osd 2021-02-10T04:48:20.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:15 smithi085 conmon[35961]: .4) 2150 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:20.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:15 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:48:20.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:15 smithi085 conmon[35961]: 02-10T04:48: 2021-02-10T04:48:20.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:15 smithi085 conmon[35961]: 14.671185+0000 osd.4 (osd.4) 2151 : 2021-02-10T04:48:20.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:15 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:20.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:15 smithi085 conmon[35961]: cluster 2021-02-10T04:48:20.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:15 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:48:20.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:15 smithi085 conmon[35961]: 48:14.671190+0000 osd.4 2021-02-10T04:48:20.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:15 smithi085 conmon[35961]: (osd.4) 2152 : cluster 2021-02-10T04:48:20.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:15 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:20.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:16 smithi085 conmon[35961]: cluster 2021-02-10T04:48: 2021-02-10T04:48:20.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:16 smithi085 conmon[35961]: 14.353713+0000 mgr.y (mgr.14140) 1087 : cluster [DBG] pgmap v1863: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:48:20.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:16 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:48:20.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:16 smithi085 conmon[35961]: T04:48:15.677339+0000 osd.4 2021-02-10T04:48:20.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:16 smithi085 conmon[35961]: (osd.4) 2153 : cluster 2021-02-10T04:48:20.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:16 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:20.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:16 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:48:20.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:16 smithi085 conmon[35961]: 02-10T04:48:15.677347+0000 osd.4 (osd 2021-02-10T04:48:20.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:16 smithi085 conmon[35961]: .4) 2154 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:20.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:16 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:48:20.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:16 smithi085 conmon[35961]: 48:15.677355+0000 osd.4 (osd.4) 2155 : 2021-02-10T04:48:20.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:16 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:20.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:16 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:48:20.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:16 smithi085 conmon[35961]: :48:15.677361+0000 osd.4 (osd.4) 2021-02-10T04:48:20.617 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:16 smithi085 conmon[35961]: 2156 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:20.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[35961]: cluster 2021-02-10T04:48:20.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[35961]: 2021-02-10T04:48:20.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[35961]: 2021-02-10T04:48:15.875179+0000 osd.2 2021-02-10T04:48:20.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[35961]: (osd.2) 1953 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:48:20.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[35961]: cluster 2021-02-10T04:48:15. 2021-02-10T04:48:20.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[35961]: 875867+0000 osd.2 (osd.2) 1954 : cluster [DBG] 2021-02-10T04:48:20.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:48:20.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[35961]: cluster 2021-02-10T04:48:16 2021-02-10T04:48:20.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[35961]: .258318+0000 mon.a (mon.0) 2156 : cluster 2021-02-10T04:48:20.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[35961]: [DBG] osdmap e1076: 8 total, 8 up, 7 in 2021-02-10T04:48:20.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[35961]: cluster 2021-02-10T04:48:20.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[35961]: 2021-02-10T04:48:20.618 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[35961]: 2021-02-10T04:48 2021-02-10T04:48:20.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[35961]: :16.720848+0000 osd.4 (osd.4) 2157 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:20.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:48:20.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[35961]: 10T04:48:16.720863+0000 osd.4 (osd.4) 2021-02-10T04:48:20.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[35961]: 2158 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:20.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[35961]: cluster 2021-02-10T04:48:20.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[35961]: 2021-02-10T04:48 2021-02-10T04:48:20.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[35961]: :16.720873+0000 osd.4 (osd 2021-02-10T04:48:20.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[35961]: .4) 2159 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:20.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[35961]: 2021-02-10T04:48:20.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:48:20.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[35961]: :48:16.720883+0000 osd.4 (osd. 2021-02-10T04:48:20.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:17 smithi085 conmon[35961]: 4) 2160 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:20.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:18 smithi085 conmon[35961]: cluster 2021-02-10T04:48:16.354097 2021-02-10T04:48:20.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:18 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1088 : cluster [DBG] pgmap v1865: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:48:20.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:18 smithi085 conmon[35961]: cluster 2021-02-10T04:48:17.751332 2021-02-10T04:48:20.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:18 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2161 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:20.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:18 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:48:20.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:18 smithi085 conmon[35961]: :48:17.751340+0000 osd.4 (osd. 2021-02-10T04:48:20.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:18 smithi085 conmon[35961]: 4) 2162 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:20.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:18 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:48:20.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:18 smithi085 conmon[35961]: 04:48:17.751348+0000 osd.4 (osd 2021-02-10T04:48:20.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:18 smithi085 conmon[35961]: .4) 2163 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:20.620 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:18 smithi085 conmon[35961]: cluster 2021-02-10T04:48 2021-02-10T04:48:20.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:18 smithi085 conmon[35961]: :17.751355+0000 osd.4 ( 2021-02-10T04:48:20.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:18 smithi085 conmon[35961]: osd.4) 2164 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:20.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[35961]: cluster 2021-02-10T04:48:20.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[35961]: 2021-02-10T04:48:20.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[35961]: 2021 2021-02-10T04:48:20.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[35961]: - 2021-02-10T04:48:20.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[35961]: 02 2021-02-10T04:48:20.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[35961]: - 2021-02-10T04:48:20.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[35961]: 10T04:48:18.354655+0000 2021-02-10T04:48:20.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[35961]: mgr.y (mgr.14140) 1089 : cluster [DBG] pgmap v1866: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:48:20.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[35961]: cluster 2021-02-10T04:48: 2021-02-10T04:48:20.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[35961]: 18.718117+0000 osd.4 (osd.4) 2165 : cluster 2021-02-10T04:48:20.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:20.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[35961]: cluster 2021-02-10T04:48: 2021-02-10T04:48:20.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[35961]: 18.718126+0000 osd.4 (osd.4) 2166 : cluster 2021-02-10T04:48:20.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:20.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:48:20.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[35961]: :48:18.718132+0000 osd.4 (osd.4 2021-02-10T04:48:20.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[35961]: ) 2167 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:20.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[35961]: cluster 2021-02-10T04:48: 2021-02-10T04:48:20.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[35961]: 18.718138+0000 osd.4 (osd.4) 2168 : cluster 2021-02-10T04:48:20.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:20.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[35961]: audit 2021-02-10T04:48:20.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[35961]: 2021-02-10T04:48:18.878058+0000 mgr.y (mgr.14140) 1090 : audit 2021-02-10T04:48:20.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[35961]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:48:20.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[35961]: cluster 2021-02-10T04:48:19 2021-02-10T04:48:20.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[35961]: .266104+0000 mon.a (mon.0) 2157 : cluster 2021-02-10T04:48:20.623 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:19 smithi085 conmon[35961]: [WRN] Health check update: 4 slow ops, oldest one blocked for 232 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:48:20.623 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:48:20.623 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:48:20.623 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:48:20.623 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:48:20.623 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:48:20.624 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:48:20.624 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:48:20.624 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:48:20.624 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:48:20.624 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:48:20.624 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:48:20.624 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:48:20.624 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:48:20.624 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:48:20.624 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:48:20.624 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:48:20.625 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is set 2021-02-10T04:48:20.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:10 smithi164 conmon[29288]: cluster 2021-02-10T04:48:08.351885 2021-02-10T04:48:20.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:10 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1083 : cluster [DBG] pgmap v1859: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:48:20.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:10 smithi164 conmon[29288]: audit 2021-02-10T04:48:08.868142+0000 2021-02-10T04:48:20.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:10 smithi164 conmon[29288]: mgr.y (mgr.14140) 1084 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:48:20.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:11 smithi164 conmon[29288]: audit 2021-02-10T04:48:10.496863+0000 mon.b (mon.2) 47 : audit [DBG] from='client.? 172.21.15.85:0/2459199104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:48:20.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:11 smithi164 conmon[29288]: audit 2021-02-10T04:48:10.812014+0000 mon.c (mon.1) 79 : audit [INF] from='client.? 172.21.15.85:0/3053369715' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:48:20.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:11 smithi164 conmon[29288]: audit 2021-02-10T04:48:10.812457+0000 mon.a (mon.0) 2151 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:48:20.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:12 smithi164 conmon[29288]: cluster 2021-02-10T04:48:10.352606+0000 mgr.y ( 2021-02-10T04:48:20.625 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:12 smithi164 conmon[29288]: mgr.14140) 1085 : cluster [DBG] pgmap v1860: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:48:20.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:12 smithi164 conmon[29288]: cluster 2021-02-10T04:48:11.225811+0000 mon.a (mon.0) 2152 : cluster 2021-02-10T04:48:20.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:12 smithi164 conmon[29288]: [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:48:20.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:12 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:48:20.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:12 smithi164 conmon[29288]: T04:48:11.239895+0000 mon.a (mon.0) 2153 : audit 2021-02-10T04:48:20.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:12 smithi164 conmon[29288]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:48:20.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:12 smithi164 conmon[29288]: cluster 2021-02-10T04:48 2021-02-10T04:48:20.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:12 smithi164 conmon[29288]: :11.240087+0000 mon.a (mon.0) 2154 2021-02-10T04:48:20.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:12 smithi164 conmon[29288]: : cluster [DBG] osdmap e1075: 8 total, 8 up, 7 in 2021-02-10T04:48:20.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:12 smithi164 conmon[29288]: cluster 2021-02-10T04:48:20.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:12 smithi164 conmon[29288]: 2021-02-10 2021-02-10T04:48:20.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:12 smithi164 conmon[29288]: T04:48:11.612854+0000 osd.4 (osd 2021-02-10T04:48:20.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:12 smithi164 conmon[29288]: .4) 2137 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:20.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:12 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:48:20.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:12 smithi164 conmon[29288]: 02-10T04:48:11.612859+0000 osd.4 ( 2021-02-10T04:48:20.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:12 smithi164 conmon[29288]: osd.4) 2138 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:20.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:12 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:48:20.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:12 smithi164 conmon[29288]: -10T04:48:11.612868+0000 osd.4 (osd 2021-02-10T04:48:20.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:12 smithi164 conmon[29288]: .4) 2139 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:20.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:12 smithi164 conmon[29288]: cluster 2021-02-10T04:48:20.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:12 smithi164 conmon[29288]: 2021-02-10T04:48:11. 2021-02-10T04:48:20.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:12 smithi164 conmon[29288]: 612873+0000 osd.4 (osd.4) 2140 : cluster 2021-02-10T04:48:20.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:12 smithi164 conmon[29288]: [WRN] 2021-02-10T04:48:20.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:12 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:20.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:13 smithi164 conmon[29288]: cluster 2021-02-10T04:48:12.634190+0000 osd.4 2021-02-10T04:48:20.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:13 smithi164 conmon[29288]: (osd.4) 2141 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:20.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:13 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:48:20.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:13 smithi164 conmon[29288]: 48:12.634199+0000 osd.4 (osd.4) 2142 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:20.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:13 smithi164 conmon[29288]: cluster 2021-02-10T04:48:12.634214+0000 osd.4 2021-02-10T04:48:20.629 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:13 smithi164 conmon[29288]: (osd.4) 2143 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:20.629 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:13 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:48:20.629 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:13 smithi164 conmon[29288]: 04:48:12.634220+0000 osd.4 (osd.4) 2021-02-10T04:48:20.629 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:13 smithi164 conmon[29288]: 2144 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:20.629 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:14 smithi164 conmon[29288]: cluster 2021-02-10T04:48:12.353100 2021-02-10T04:48:20.629 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:14 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1086 : cluster [DBG] pgmap v1862: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:48:20.630 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:14 smithi164 conmon[29288]: cluster 2021-02-10T04:48:13. 2021-02-10T04:48:20.630 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:14 smithi164 conmon[29288]: 236034+0000 mon.a (mon.0) 2155 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 227 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:48:20.631 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:14 smithi164 conmon[29288]: cluster 2021-02-10T04:48:20.631 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:14 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:48:20.631 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:14 smithi164 conmon[29288]: 48:13.632185+0000 osd.4 (osd.4) 2021-02-10T04:48:20.631 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:14 smithi164 conmon[29288]: 2145 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:20.631 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:14 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:48:20.632 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:14 smithi164 conmon[29288]: 02-10T04:48:13.632194+0000 2021-02-10T04:48:20.632 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:14 smithi164 conmon[29288]: osd.4 (osd.4) 2146 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:20.632 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:14 smithi164 conmon[29288]: cluster 2021-02-10T04:48:20.632 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:14 smithi164 conmon[29288]: 2021-02-10T04:48: 2021-02-10T04:48:20.632 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:14 smithi164 conmon[29288]: 13.632201+0000 osd.4 (osd.4) 2147 : 2021-02-10T04:48:20.632 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:14 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:20.633 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:14 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:48:20.633 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:14 smithi164 conmon[29288]: :48:13.632208+0000 osd.4 (osd.4) 2148 : cluster 2021-02-10T04:48:20.633 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:14 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:20.633 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:15 smithi164 conmon[29288]: cluster 2021-02-10T04:48:20.633 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:15 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:48:20.634 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:15 smithi164 conmon[29288]: 48:14.671169+0000 osd.4 (osd.4) 2149 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:20.634 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:15 smithi164 conmon[29288]: cluster 2021-02-10T04:48:14.671179 2021-02-10T04:48:20.634 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:15 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2150 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:20.634 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:15 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:48:20.634 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:15 smithi164 conmon[29288]: 04:48:14.671185 2021-02-10T04:48:20.634 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:15 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2151 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:20.635 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:15 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:48:20.635 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:15 smithi164 conmon[29288]: :48:14.671190+0000 osd.4 (osd.4) 2152 : cluster 2021-02-10T04:48:20.635 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:15 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:20.635 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:16 smithi164 conmon[29288]: cluster 2021-02-10T04:48:14.353713+0000 mgr.y (mgr.14140) 1087 : cluster [DBG] 2021-02-10T04:48:20.635 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:16 smithi164 conmon[29288]: pgmap v1863: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:48:20.636 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:16 smithi164 conmon[29288]: cluster 2021-02-10T04:48:15.677339+0000 osd.4 (osd.4) 2153 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:20.636 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:16 smithi164 conmon[29288]: cluster 2021-02-10T04:48:15.677347+0000 2021-02-10T04:48:20.636 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:16 smithi164 conmon[29288]: osd.4 (osd.4) 2154 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:20.636 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:16 smithi164 conmon[29288]: cluster 2021-02-10T04:48:15. 2021-02-10T04:48:20.636 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:16 smithi164 conmon[29288]: 677355+0000 osd.4 (osd.4) 2155 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:20.637 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:16 smithi164 conmon[29288]: cluster 2021-02-10T04:48:15.677361 2021-02-10T04:48:20.637 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:16 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2156 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:20.637 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:17 smithi164 conmon[29288]: cluster 2021-02-10T04:48:15.875179+0000 osd.2 (osd. 2021-02-10T04:48:20.637 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:17 smithi164 conmon[29288]: 2) 1953 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:48:20.637 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:17 smithi164 conmon[29288]: cluster 2021-02-10T04:48:15.875867+0000 osd.2 (osd.2) 1954 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:48:20.637 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:17 smithi164 conmon[29288]: 2021-02-10T04:48:20.638 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:17 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:48:20.638 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:17 smithi164 conmon[29288]: T04:48:16.258318+0000 mon.a (mon 2021-02-10T04:48:20.638 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:17 smithi164 conmon[29288]: .0) 2156 : cluster [DBG] 2021-02-10T04:48:20.638 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:17 smithi164 conmon[29288]: osdmap e1076: 8 total, 8 up, 7 in 2021-02-10T04:48:20.638 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:17 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:48:20.639 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:17 smithi164 conmon[29288]: 02-10T04:48: 2021-02-10T04:48:20.639 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:17 smithi164 conmon[29288]: 16.720848+0000 osd.4 (osd. 2021-02-10T04:48:20.639 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:17 smithi164 conmon[29288]: 4) 2157 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:20.639 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:17 smithi164 conmon[29288]: cluster 2021-02-10T04:48 2021-02-10T04:48:20.639 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:17 smithi164 conmon[29288]: :16.720863+0000 osd.4 (osd.4) 2158 : cluster 2021-02-10T04:48:20.640 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:17 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:20.640 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:17 smithi164 conmon[29288]: cluster 2021-02-10T04:48:16.720873 2021-02-10T04:48:20.640 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:17 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2159 : cluster [WRN] 2021-02-10T04:48:20.640 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:17 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:20.640 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:17 smithi164 conmon[29288]: cluster 2021-02-10T04:48:20.640 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:17 smithi164 conmon[29288]: 2021-02-10T04:48:16.720883+0000 osd.4 (osd.4) 2021-02-10T04:48:20.641 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:17 smithi164 conmon[29288]: 2160 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:20.641 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:18 smithi164 conmon[29288]: cluster 2021-02-10T04:48:16.354097+0000 2021-02-10T04:48:20.641 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:18 smithi164 conmon[29288]: mgr.y (mgr.14140) 1088 : cluster [DBG] pgmap v1865: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:48:20.641 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:18 smithi164 conmon[29288]: cluster 2021-02-10T04:48:17.751332+0000 osd.4 (osd.4) 2161 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:20.641 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:18 smithi164 conmon[29288]: cluster 2021-02-10T04:48:17.751340+0000 osd.4 (osd.4) 2162 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:20.642 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:18 smithi164 conmon[29288]: cluster 2021-02-10T04:48:17.751348+0000 osd.4 (osd.4) 2163 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:20.642 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:18 smithi164 conmon[29288]: cluster 2021-02-10T04:48:17.751355+0000 osd.4 (osd.4) 2021-02-10T04:48:20.642 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:18 smithi164 conmon[29288]: 2164 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:20.642 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:19 smithi164 conmon[29288]: cluster 2021-02-10T04:48:18. 2021-02-10T04:48:20.642 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:19 smithi164 conmon[29288]: 354655+0000 mgr.y (mgr.14140) 1089 : cluster [DBG] pgmap v1866: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:48:20.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:19 smithi164 conmon[29288]: cluster 2021-02-10T04:48:18.718117+0000 osd.4 (osd.4) 2165 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:20.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:19 smithi164 conmon[29288]: cluster 2021-02-10T04:48:18.718126+0000 osd.4 (osd.4) 2166 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:20.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:19 smithi164 conmon[29288]: cluster 2021-02-10T04:48:18.718132+0000 osd.4 (osd.4) 2167 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:20.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:19 smithi164 conmon[29288]: 2021-02-10T04:48:20.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:19 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:48:20.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:19 smithi164 conmon[29288]: T04:48:18.718138+0000 osd.4 2021-02-10T04:48:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:19 smithi164 conmon[29288]: (osd.4) 2168 : cluster [WRN] 2021-02-10T04:48:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:19 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:19 smithi164 conmon[29288]: audit 2021-02-10T04:48:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:19 smithi164 conmon[29288]: 2021-02-10T04:48:18. 2021-02-10T04:48:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:19 smithi164 conmon[29288]: 878058+0000 mgr.y (mgr.14140) 1090 : audit [DBG] 2021-02-10T04:48:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:19 smithi164 conmon[29288]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:48:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:19 smithi164 conmon[29288]: cluster 2021-02-10T04:48:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:19 smithi164 conmon[29288]: 2021-02-10T04:48:19.266104+0000 mon.a (mon.0) 2157 : 2021-02-10T04:48:20.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:19 smithi164 conmon[29288]: cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 232 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:48:20.645 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:11 smithi164 conmon[33677]: debug 2021-02-10T04:48:20.646 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:11 smithi164 conmon[33677]: 2021-02-10T04:48:11.612+0000 7fbec19a1700 -1 osd.4 1075 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:20.646 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:11 smithi164 conmon[33677]: 2021-02-10T04:48:20.646 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:12 smithi164 conmon[33677]: debug 2021-02-10T04:48:12.633+0000 7fbec19a1700 -1 osd.4 1075 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:20.646 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:13 smithi164 conmon[33677]: debug 2021-02-10T04:48:20.646 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:13 smithi164 conmon[33677]: 2021-02-10T04:48:13.631+0000 7fbec19a1700 -1 osd.4 1075 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:20.646 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:13 smithi164 conmon[33677]: 2021-02-10T04:48:20.647 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:14 smithi164 conmon[33677]: debug 2021-02-10T04:48:14.669+0000 7fbec19a1700 -1 osd.4 1075 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:20.647 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:15 smithi164 conmon[33677]: debug 2021-02-10T04:48:20.647 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:15 smithi164 conmon[33677]: 2021-02-10T04:48:15.675+0000 7fbec19a1700 -1 osd.4 1075 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:20.647 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:15 smithi164 conmon[33677]: 2021-02-10T04:48:20.647 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:16 smithi164 conmon[33677]: debug 2021-02-10T04:48:16.719+0000 7fbec19a1700 -1 osd.4 1076 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:20.648 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:17 smithi164 conmon[33677]: debug 2021-02-10T04:48:17.749+0000 7fbec19a1700 -1 osd.4 1076 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:20.648 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:18 smithi164 conmon[33677]: debug 2021-02-10T04:48:20.648 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:18 smithi164 conmon[33677]: 2021-02-10T04:48:18.716+0000 7fbec19a1700 -1 osd.4 1076 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:20.648 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:19 smithi164 conmon[33677]: debug 2021-02-10T04:48:19.668+0000 7fbec19a1700 -1 osd.4 1076 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:20.649 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:48:18 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:48:20.649 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:48:20.649 INFO:teuthology.orchestra.run.smithi164.stdout: "size": 20, 2021-02-10T04:48:20.650 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 600, 2021-02-10T04:48:20.650 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [ 2021-02-10T04:48:20.650 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.650 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:7435 7.f e749/93 7:fe141921:::smithi08575397-2:head v 749'717, mlcod=746'714)", 2021-02-10T04:48:20.650 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:39:28.284404+0000", 2021-02-10T04:48:20.650 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 531.07939434599996, 2021-02-10T04:48:20.651 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.026541647000000002, 2021-02-10T04:48:20.651 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:48:20.651 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:48:20.651 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:48:20.651 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.652 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:48:20.652 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:28.284404+0000", 2021-02-10T04:48:20.652 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:48:20.652 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.652 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.653 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:48:20.653 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:28.284404+0000", 2021-02-10T04:48:20.653 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.6696e-05 2021-02-10T04:48:20.653 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.653 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.653 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:48:20.654 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:28.284430+0000", 2021-02-10T04:48:20.654 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.002e-06 2021-02-10T04:48:20.654 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.654 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.654 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:48:20.655 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:28.284437+0000", 2021-02-10T04:48:20.655 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1416999999999999e-05 2021-02-10T04:48:20.655 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.655 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.655 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:48:20.655 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:28.284449+0000", 2021-02-10T04:48:20.656 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.576e-06 2021-02-10T04:48:20.656 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.656 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.656 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:48:20.656 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:28.284456+0000", 2021-02-10T04:48:20.657 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.8678999999999998e-05 2021-02-10T04:48:20.657 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.657 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.657 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:48:20.657 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:28.284515+0000", 2021-02-10T04:48:20.657 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1319000000000003e-05 2021-02-10T04:48:20.657 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.658 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.658 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:48:20.658 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:28.284546+0000", 2021-02-10T04:48:20.658 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.026302322 2021-02-10T04:48:20.658 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.658 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.658 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:48:20.658 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:28.310849+0000", 2021-02-10T04:48:20.658 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.6636000000000001e-05 2021-02-10T04:48:20.658 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.658 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.659 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:48:20.659 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:28.310945+0000", 2021-02-10T04:48:20.659 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.026541647000000002 2021-02-10T04:48:20.659 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.659 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:48:20.659 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.660 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.660 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.660 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:7436 7.f e749/93 7:fe141921:::smithi08575397-2:head v 749'718, mlcod=746'714)", 2021-02-10T04:48:20.660 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:39:28.285462+0000", 2021-02-10T04:48:20.660 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 531.07833611499996, 2021-02-10T04:48:20.660 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.027332538999999999, 2021-02-10T04:48:20.661 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:48:20.661 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:48:20.661 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:48:20.661 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.661 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:48:20.661 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:28.285462+0000", 2021-02-10T04:48:20.661 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:48:20.661 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.661 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.661 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:48:20.661 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:28.285462+0000", 2021-02-10T04:48:20.661 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5160000000000001e-05 2021-02-10T04:48:20.661 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.661 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.662 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:48:20.662 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:28.285507+0000", 2021-02-10T04:48:20.662 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.9439999999999994e-06 2021-02-10T04:48:20.662 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.662 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.662 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:48:20.662 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:28.285516+0000", 2021-02-10T04:48:20.662 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.38e-05 2021-02-10T04:48:20.662 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.662 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.662 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:48:20.663 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:28.285530+0000", 2021-02-10T04:48:20.663 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.8350000000000004e-06 2021-02-10T04:48:20.663 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.663 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.663 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:48:20.663 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:28.285538+0000", 2021-02-10T04:48:20.663 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9601000000000002e-05 2021-02-10T04:48:20.663 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.663 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.663 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:48:20.663 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:28.285587+0000", 2021-02-10T04:48:20.663 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9930999999999999e-05 2021-02-10T04:48:20.663 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.664 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.664 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:48:20.664 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:28.285607+0000", 2021-02-10T04:48:20.664 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.027126727 2021-02-10T04:48:20.664 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.664 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.664 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:48:20.664 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:28.312734+0000", 2021-02-10T04:48:20.664 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0541000000000002e-05 2021-02-10T04:48:20.664 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.664 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.664 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:48:20.664 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:39:28.312794+0000", 2021-02-10T04:48:20.665 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.027332538999999999 2021-02-10T04:48:20.665 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.665 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:48:20.665 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.665 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.665 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.665 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:7991 7.5 e804/638 7:a5580f12:::smithi08575397-18:head v 804'253, mlcod=794'249)", 2021-02-10T04:48:20.665 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:40:23.369866+0000", 2021-02-10T04:48:20.665 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 475.99393221499997, 2021-02-10T04:48:20.665 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.019389575999999999, 2021-02-10T04:48:20.665 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:48:20.665 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:48:20.665 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:48:20.666 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.666 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:48:20.666 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:23.369866+0000", 2021-02-10T04:48:20.666 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:48:20.666 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.666 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.666 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:48:20.666 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:23.369866+0000", 2021-02-10T04:48:20.666 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8666999999999998e-05 2021-02-10T04:48:20.666 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.666 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.666 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:48:20.667 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:23.369895+0000", 2021-02-10T04:48:20.667 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.8740000000000003e-06 2021-02-10T04:48:20.667 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.667 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.667 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:48:20.667 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:23.369899+0000", 2021-02-10T04:48:20.667 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.7959999999999998e-06 2021-02-10T04:48:20.667 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.667 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.667 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:48:20.667 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:23.369906+0000", 2021-02-10T04:48:20.667 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.2410000000000003e-06 2021-02-10T04:48:20.667 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.668 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.668 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:48:20.668 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:23.369911+0000", 2021-02-10T04:48:20.668 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.4506999999999998e-05 2021-02-10T04:48:20.668 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.668 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.668 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:48:20.668 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:23.369966+0000", 2021-02-10T04:48:20.668 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7526000000000001e-05 2021-02-10T04:48:20.668 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.668 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.668 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:48:20.668 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:23.369983+0000", 2021-02-10T04:48:20.669 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.019220391999999999 2021-02-10T04:48:20.669 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.669 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.669 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:48:20.669 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:23.389204+0000", 2021-02-10T04:48:20.669 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.1573000000000002e-05 2021-02-10T04:48:20.669 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.669 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.669 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:48:20.669 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:23.389255+0000", 2021-02-10T04:48:20.669 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.019389575999999999 2021-02-10T04:48:20.669 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.669 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:48:20.669 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.670 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.670 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.670 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:7992 7.5 e804/638 7:a5580f12:::smithi08575397-18:head v 804'254, mlcod=794'249)", 2021-02-10T04:48:20.670 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:40:23.369937+0000", 2021-02-10T04:48:20.670 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 475.99386099999998, 2021-02-10T04:48:20.670 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.019364454, 2021-02-10T04:48:20.670 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:48:20.670 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:48:20.670 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:48:20.670 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.670 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:48:20.671 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:23.369937+0000", 2021-02-10T04:48:20.671 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:48:20.671 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.671 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.671 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:48:20.671 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:23.369937+0000", 2021-02-10T04:48:20.671 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.0436000000000001e-05 2021-02-10T04:48:20.671 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.671 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.671 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:48:20.672 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:23.369967+0000", 2021-02-10T04:48:20.672 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.6440000000000002e-06 2021-02-10T04:48:20.672 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.672 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.672 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:48:20.672 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:23.369971+0000", 2021-02-10T04:48:20.672 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.2310000000000004e-06 2021-02-10T04:48:20.672 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.672 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.673 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:48:20.673 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:23.369976+0000", 2021-02-10T04:48:20.673 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.605e-06 2021-02-10T04:48:20.673 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.673 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.673 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:48:20.673 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:23.369979+0000", 2021-02-10T04:48:20.673 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00060132399999999998 2021-02-10T04:48:20.673 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.673 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.674 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:48:20.674 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:23.370580+0000", 2021-02-10T04:48:20.674 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9156000000000001e-05 2021-02-10T04:48:20.674 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.674 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.674 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:48:20.674 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:23.370599+0000", 2021-02-10T04:48:20.674 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.018659319000000001 2021-02-10T04:48:20.674 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.675 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.675 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:48:20.675 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:23.389259+0000", 2021-02-10T04:48:20.675 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2738999999999999e-05 2021-02-10T04:48:20.675 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.675 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.675 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:48:20.675 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:23.389302+0000", 2021-02-10T04:48:20.676 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.019364454 2021-02-10T04:48:20.676 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.676 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:48:20.676 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.676 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.677 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.677 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:7993 7.5 e804/638 7:a5580f12:::smithi08575397-18:head v 804'255, mlcod=794'249)", 2021-02-10T04:48:20.677 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:40:23.371073+0000", 2021-02-10T04:48:20.677 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 475.992724646, 2021-02-10T04:48:20.677 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.018279558000000001, 2021-02-10T04:48:20.678 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:48:20.678 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:48:20.678 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:48:20.678 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.678 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:48:20.679 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:23.371073+0000", 2021-02-10T04:48:20.679 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:48:20.679 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.679 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.679 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:48:20.679 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:23.371073+0000", 2021-02-10T04:48:20.680 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.7103000000000001e-05 2021-02-10T04:48:20.680 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.680 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.680 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:48:20.680 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:23.371101+0000", 2021-02-10T04:48:20.681 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.8550000000000004e-06 2021-02-10T04:48:20.681 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.681 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.681 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:48:20.681 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:23.371104+0000", 2021-02-10T04:48:20.681 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.3439999999999997e-06 2021-02-10T04:48:20.682 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.682 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.682 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:48:20.682 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:23.371111+0000", 2021-02-10T04:48:20.682 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.3119999999999996e-06 2021-02-10T04:48:20.683 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.683 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.683 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:48:20.683 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:23.371118+0000", 2021-02-10T04:48:20.683 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.9079999999999999e-05 2021-02-10T04:48:20.684 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.684 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.684 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:48:20.684 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:23.371157+0000", 2021-02-10T04:48:20.684 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.1063000000000001e-05 2021-02-10T04:48:20.684 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.685 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.685 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:48:20.685 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:23.371178+0000", 2021-02-10T04:48:20.685 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.018125255 2021-02-10T04:48:20.685 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.685 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.686 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:48:20.686 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:23.389303+0000", 2021-02-10T04:48:20.686 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9546000000000003e-05 2021-02-10T04:48:20.686 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.686 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.687 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:48:20.687 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:23.389353+0000", 2021-02-10T04:48:20.687 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.018279558000000001 2021-02-10T04:48:20.687 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.687 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:48:20.688 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.688 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.688 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.688 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:8108 7.f e817/93 7:fe141921:::smithi08575397-2:head v 817'820, mlcod=814'816)", 2021-02-10T04:48:20.688 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:40:36.497126+0000", 2021-02-10T04:48:20.688 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 462.86667180400002, 2021-02-10T04:48:20.689 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.015567384, 2021-02-10T04:48:20.689 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:48:20.689 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:48:20.689 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:48:20.689 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.690 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:48:20.690 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.497126+0000", 2021-02-10T04:48:20.690 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:48:20.690 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.690 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.690 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:48:20.691 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.497126+0000", 2021-02-10T04:48:20.691 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0089076650000000004 2021-02-10T04:48:20.691 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.691 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.691 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:48:20.692 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.506034+0000", 2021-02-10T04:48:20.692 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.8670000000000006e-06 2021-02-10T04:48:20.692 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.692 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.692 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:48:20.693 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.506044+0000", 2021-02-10T04:48:20.693 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3040000000000001e-05 2021-02-10T04:48:20.693 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.693 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.693 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:48:20.693 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.506057+0000", 2021-02-10T04:48:20.694 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1589999999999993e-06 2021-02-10T04:48:20.694 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.694 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.694 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:48:20.694 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.506065+0000", 2021-02-10T04:48:20.695 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.1674e-05 2021-02-10T04:48:20.695 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.695 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.695 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:48:20.695 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.506117+0000", 2021-02-10T04:48:20.695 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1068000000000001e-05 2021-02-10T04:48:20.695 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.696 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.696 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:48:20.696 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.506148+0000", 2021-02-10T04:48:20.696 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0064803609999999996 2021-02-10T04:48:20.696 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.697 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.697 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:48:20.697 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.512628+0000", 2021-02-10T04:48:20.697 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.5549999999999994e-05 2021-02-10T04:48:20.697 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.697 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.698 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:48:20.698 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:40:36.512694+0000", 2021-02-10T04:48:20.698 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.015567384 2021-02-10T04:48:20.698 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.698 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:48:20.698 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.699 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.699 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.699 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:9910 7.f e975/93 7:fc15f6da:::smithi08575397-35:head v 975'1043, mlcod=971'1040)", 2021-02-10T04:48:20.699 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:43:15.610734+0000", 2021-02-10T04:48:20.699 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 303.75306382399998, 2021-02-10T04:48:20.700 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010925664999999999, 2021-02-10T04:48:20.700 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:48:20.700 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:48:20.700 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:48:20.700 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.700 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:48:20.701 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:15.610734+0000", 2021-02-10T04:48:20.701 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:48:20.701 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.701 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.701 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:48:20.702 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:15.610734+0000", 2021-02-10T04:48:20.702 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0048416709999999996 2021-02-10T04:48:20.702 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.702 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.702 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:48:20.702 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:15.615576+0000", 2021-02-10T04:48:20.702 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.6550000000000004e-06 2021-02-10T04:48:20.702 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.702 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.703 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:48:20.703 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:15.615584+0000", 2021-02-10T04:48:20.703 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.0580000000000004e-06 2021-02-10T04:48:20.703 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.703 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.703 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:48:20.703 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:15.615593+0000", 2021-02-10T04:48:20.703 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.066e-06 2021-02-10T04:48:20.703 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.703 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.703 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:48:20.703 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:15.615599+0000", 2021-02-10T04:48:20.703 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.4864999999999998e-05 2021-02-10T04:48:20.704 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.704 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.704 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:48:20.704 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:15.615644+0000", 2021-02-10T04:48:20.704 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0954e-05 2021-02-10T04:48:20.704 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.704 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.704 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:48:20.704 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:15.615665+0000", 2021-02-10T04:48:20.704 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0059269789999999998 2021-02-10T04:48:20.704 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.704 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.704 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:48:20.705 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:15.621591+0000", 2021-02-10T04:48:20.705 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.8417000000000006e-05 2021-02-10T04:48:20.705 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.705 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.705 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:48:20.705 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:15.621660+0000", 2021-02-10T04:48:20.705 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010925664999999999 2021-02-10T04:48:20.705 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.705 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:48:20.705 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.705 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.705 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.705 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:9942 7.f e980/93 7:fc15f6da:::smithi08575397-35:head v 980'1049, mlcod=978'1045)", 2021-02-10T04:48:20.706 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:43:20.370960+0000", 2021-02-10T04:48:20.706 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 298.99283774700001, 2021-02-10T04:48:20.706 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012039129000000001, 2021-02-10T04:48:20.706 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:48:20.706 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:48:20.706 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:48:20.706 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.706 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:48:20.706 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:20.370960+0000", 2021-02-10T04:48:20.706 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:48:20.706 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.706 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.706 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:48:20.707 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:20.370960+0000", 2021-02-10T04:48:20.707 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0056447700000000003 2021-02-10T04:48:20.707 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.707 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.707 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:48:20.707 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:20.376605+0000", 2021-02-10T04:48:20.707 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0487e-05 2021-02-10T04:48:20.707 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.707 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.707 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:48:20.707 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:20.376616+0000", 2021-02-10T04:48:20.707 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3200999999999999e-05 2021-02-10T04:48:20.707 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.708 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.708 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:48:20.708 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:20.376629+0000", 2021-02-10T04:48:20.708 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.3880000000000002e-06 2021-02-10T04:48:20.708 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.708 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.708 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:48:20.708 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:20.376636+0000", 2021-02-10T04:48:20.708 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9981000000000001e-05 2021-02-10T04:48:20.708 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.708 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.708 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:48:20.708 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:20.376686+0000", 2021-02-10T04:48:20.708 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2497999999999999e-05 2021-02-10T04:48:20.709 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.709 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.709 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:48:20.709 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:20.376729+0000", 2021-02-10T04:48:20.709 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0062084000000000002 2021-02-10T04:48:20.709 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.709 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.709 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:48:20.709 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:20.382937+0000", 2021-02-10T04:48:20.709 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.2403999999999994e-05 2021-02-10T04:48:20.709 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.709 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.709 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:48:20.709 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:43:20.382999+0000", 2021-02-10T04:48:20.710 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.012039129000000001 2021-02-10T04:48:20.710 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.710 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:48:20.710 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.710 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.711 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.711 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.19 1011/1010 [PushOp(7:9fd44245:::smithi08575397-20:212, version: 776'510, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:9fd44245:::smithi08575397-20:212@776'510, size: 2756513, copy_subset: [0~2756513], clone_subset: {}, snapset: 372=[]:{212=[203],27c=[27c],2cb=[2c8,2ae],309=[308,300,2f5,2f0,2ee,2ed,2eb,2ea,2d9],320=[31a],32b=[32a,328],351=[34e],358=[354,353],372=[372,370,36f,36b,368,366,35f]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:48:20.712 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:28.644072+0000", 2021-02-10T04:48:20.712 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 230.71972586000001, 2021-02-10T04:48:20.712 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.01035735, 2021-02-10T04:48:20.712 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:48:20.712 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:48:20.712 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:48:20.712 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.712 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:48:20.712 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.644072+0000", 2021-02-10T04:48:20.712 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:48:20.712 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.712 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.712 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:48:20.712 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.644072+0000", 2021-02-10T04:48:20.713 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001079015 2021-02-10T04:48:20.713 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.713 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.713 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:48:20.713 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.645151+0000", 2021-02-10T04:48:20.713 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1058e-05 2021-02-10T04:48:20.713 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.713 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.713 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:48:20.713 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.645182+0000", 2021-02-10T04:48:20.713 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.4929999999999999e-05 2021-02-10T04:48:20.713 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.713 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.714 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:48:20.714 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.645217+0000", 2021-02-10T04:48:20.714 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7141000000000001e-05 2021-02-10T04:48:20.714 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.714 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.714 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:48:20.714 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.645234+0000", 2021-02-10T04:48:20.714 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0085741570000000007 2021-02-10T04:48:20.714 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.714 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.714 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:48:20.714 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.653808+0000", 2021-02-10T04:48:20.714 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7589999999999997e-06 2021-02-10T04:48:20.714 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.715 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.715 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:48:20.715 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.653816+0000", 2021-02-10T04:48:20.715 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00061328999999999995 2021-02-10T04:48:20.715 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.715 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.715 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:48:20.715 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.654430+0000", 2021-02-10T04:48:20.715 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.01035735 2021-02-10T04:48:20.715 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.715 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:48:20.715 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.715 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.716 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.716 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.19 1011/1010 [PushOp(7:9fd44245:::smithi08575397-20:212, version: 776'510, data_included: [2097152~659361], data_size: 659361, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:9fd44245:::smithi08575397-20:212@776'510, size: 2756513, copy_subset: [0~2756513], clone_subset: {}, snapset: 372=[]:{212=[203],27c=[27c],2cb=[2c8,2ae],309=[308,300,2f5,2f0,2ee,2ed,2eb,2ea,2d9],320=[31a],32b=[32a,328],351=[34e],358=[354,353],372=[372,370,36f,36b,368,366,35f]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2756513, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:48:20.716 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:28.656618+0000", 2021-02-10T04:48:20.716 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 230.707179693, 2021-02-10T04:48:20.716 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0099259329999999996, 2021-02-10T04:48:20.716 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:48:20.716 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:48:20.716 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:48:20.716 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.716 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:48:20.716 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.656618+0000", 2021-02-10T04:48:20.716 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:48:20.716 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.717 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.717 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:48:20.717 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.656618+0000", 2021-02-10T04:48:20.717 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0010645349999999999 2021-02-10T04:48:20.717 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.717 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.717 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:48:20.717 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.657683+0000", 2021-02-10T04:48:20.717 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.6270000000000001e-05 2021-02-10T04:48:20.717 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.717 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.717 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:48:20.717 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.657709+0000", 2021-02-10T04:48:20.718 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.0611999999999997e-05 2021-02-10T04:48:20.718 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.718 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.718 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:48:20.718 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.657750+0000", 2021-02-10T04:48:20.718 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8016000000000002e-05 2021-02-10T04:48:20.718 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.718 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.718 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:48:20.718 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.657768+0000", 2021-02-10T04:48:20.718 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0079422280000000008 2021-02-10T04:48:20.718 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.718 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.719 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:48:20.719 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.665710+0000", 2021-02-10T04:48:20.719 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5824e-05 2021-02-10T04:48:20.719 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.719 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.719 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:48:20.719 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.665726+0000", 2021-02-10T04:48:20.719 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00081844800000000005 2021-02-10T04:48:20.719 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.719 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.719 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:48:20.719 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.666544+0000", 2021-02-10T04:48:20.719 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0099259329999999996 2021-02-10T04:48:20.719 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.720 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:48:20.720 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.720 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.720 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.720 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.19 1011/1010 [PushOp(7:9fd44245:::smithi08575397-20:320, version: 947'661, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:9fd44245:::smithi08575397-20:320@947'661, size: 2269242, copy_subset: [0~2269242], clone_subset: {7:9fd44245:::smithi08575397-20:27c=[]}, snapset: 372=[]:{212=[203],27c=[27c],2cb=[2c8,2ae],309=[308,300,2f5,2f0,2ee,2ed,2eb,2ea,2d9],320=[31a],32b=[32a,328],351=[34e],358=[354,353],372=[372,370,36f,36b,368,366,35f]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:48:20.720 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:28.697025+0000", 2021-02-10T04:48:20.720 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 230.66677333600001, 2021-02-10T04:48:20.720 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010329339, 2021-02-10T04:48:20.720 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:48:20.720 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:48:20.720 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:48:20.720 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.720 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:48:20.721 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.697025+0000", 2021-02-10T04:48:20.721 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:48:20.721 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.721 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.721 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:48:20.721 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.697025+0000", 2021-02-10T04:48:20.721 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0014287740000000001 2021-02-10T04:48:20.721 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.721 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.721 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:48:20.721 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.698453+0000", 2021-02-10T04:48:20.721 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.1725e-05 2021-02-10T04:48:20.721 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.722 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.722 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:48:20.722 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.698475+0000", 2021-02-10T04:48:20.722 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5849000000000001e-05 2021-02-10T04:48:20.722 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.722 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.722 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:48:20.722 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.698521+0000", 2021-02-10T04:48:20.722 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4253999999999999e-05 2021-02-10T04:48:20.722 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.722 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.722 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:48:20.722 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.698535+0000", 2021-02-10T04:48:20.723 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0081349270000000001 2021-02-10T04:48:20.723 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.723 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.723 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:48:20.723 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.706670+0000", 2021-02-10T04:48:20.723 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6552e-05 2021-02-10T04:48:20.723 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.723 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.723 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:48:20.723 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.706687+0000", 2021-02-10T04:48:20.723 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000667258 2021-02-10T04:48:20.723 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.723 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.723 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:48:20.724 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.707354+0000", 2021-02-10T04:48:20.724 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010329339 2021-02-10T04:48:20.724 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.724 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:48:20.724 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.724 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.724 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.724 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.19 1011/1010 [PushOp(7:9fd44245:::smithi08575397-20:32b, version: 953'667, data_included: [2097152~793916], data_size: 793916, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:9fd44245:::smithi08575397-20:32b@953'667, size: 2891068, copy_subset: [0~2891068], clone_subset: {7:9fd44245:::smithi08575397-20:320=[]}, snapset: 372=[]:{212=[203],27c=[27c],2cb=[2c8,2ae],309=[308,300,2f5,2f0,2ee,2ed,2eb,2ea,2d9],320=[31a],32b=[32a,328],351=[34e],358=[354,353],372=[372,370,36f,36b,368,366,35f]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2891068, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:48:20.724 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:28.737408+0000", 2021-02-10T04:48:20.724 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 230.626390077, 2021-02-10T04:48:20.724 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010315126000000001, 2021-02-10T04:48:20.724 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:48:20.724 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:48:20.725 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:48:20.725 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.725 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:48:20.725 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.737408+0000", 2021-02-10T04:48:20.725 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:48:20.725 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.725 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.725 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:48:20.725 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.737408+0000", 2021-02-10T04:48:20.726 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.002619101 2021-02-10T04:48:20.726 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.726 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.726 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:48:20.726 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.740027+0000", 2021-02-10T04:48:20.726 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.3180000000000002e-05 2021-02-10T04:48:20.726 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.726 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.726 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:48:20.726 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.740050+0000", 2021-02-10T04:48:20.726 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.4010000000000001e-05 2021-02-10T04:48:20.726 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.726 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.727 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:48:20.727 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.740084+0000", 2021-02-10T04:48:20.727 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.52e-05 2021-02-10T04:48:20.727 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.727 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.727 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:48:20.727 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.740099+0000", 2021-02-10T04:48:20.727 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0067203999999999996 2021-02-10T04:48:20.727 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.727 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.727 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:48:20.727 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.746820+0000", 2021-02-10T04:48:20.727 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1915e-05 2021-02-10T04:48:20.728 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.728 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.728 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:48:20.728 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.746832+0000", 2021-02-10T04:48:20.728 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00089132000000000002 2021-02-10T04:48:20.728 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.728 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.728 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:48:20.728 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.747723+0000", 2021-02-10T04:48:20.728 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010315126000000001 2021-02-10T04:48:20.728 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.728 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:48:20.728 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.729 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.729 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.729 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.19 1011/1010 [PushOp(7:9fd44245:::smithi08575397-20:2cb, version: 986'714, data_included: [1913836~641026], data_size: 641026, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:9fd44245:::smithi08575397-20:2cb@986'714, size: 2554862, copy_subset: [538121~865189,1730449~824413], clone_subset: {7:9fd44245:::smithi08575397-20:27c=[],7:9fd44245:::smithi08575397-20:320=[0~538121,1403310~327139]}, snapset: 372=[]:{212=[203],27c=[27c],2cb=[2c8,2ae],309=[308,300,2f5,2f0,2ee,2ed,2eb,2ea,2d9],320=[31a],32b=[32a,328],351=[34e],358=[354,353],372=[372,370,36f,36b,368,366,35f]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2554862, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1913836, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:48:20.729 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:28.761961+0000", 2021-02-10T04:48:20.729 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 230.601837378, 2021-02-10T04:48:20.729 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0098334979999999995, 2021-02-10T04:48:20.729 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:48:20.729 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:48:20.729 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:48:20.729 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.729 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:48:20.729 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.761961+0000", 2021-02-10T04:48:20.729 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:48:20.730 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.730 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.730 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:48:20.730 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.761961+0000", 2021-02-10T04:48:20.730 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0009811659999999999 2021-02-10T04:48:20.730 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.730 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.730 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:48:20.730 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.762942+0000", 2021-02-10T04:48:20.730 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.2175000000000003e-05 2021-02-10T04:48:20.730 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.730 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.730 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:48:20.731 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.762974+0000", 2021-02-10T04:48:20.731 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.2935000000000001e-05 2021-02-10T04:48:20.731 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.731 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.731 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:48:20.731 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.763027+0000", 2021-02-10T04:48:20.731 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8119999999999999e-05 2021-02-10T04:48:20.731 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.731 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.731 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:48:20.731 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.763045+0000", 2021-02-10T04:48:20.731 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0076169890000000002 2021-02-10T04:48:20.731 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.732 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.732 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:48:20.732 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.770662+0000", 2021-02-10T04:48:20.732 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.3100000000000006e-06 2021-02-10T04:48:20.732 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.732 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.732 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:48:20.732 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.770671+0000", 2021-02-10T04:48:20.732 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0011228029999999999 2021-02-10T04:48:20.732 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.732 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.732 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:48:20.732 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.771794+0000", 2021-02-10T04:48:20.732 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0098334979999999995 2021-02-10T04:48:20.733 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.733 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:48:20.733 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.733 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.733 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.733 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.19 1011/1010 [PushOp(7:9fd44245:::smithi08575397-20:372, version: 988'716, data_included: [3145728~652951], data_size: 652951, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:9fd44245:::smithi08575397-20:372@988'716, size: 3798679, copy_subset: [0~3798679], clone_subset: {7:9fd44245:::smithi08575397-20:32b=[]}, snapset: 372=[]:{212=[203],27c=[27c],2cb=[2c8,2ae],309=[308,300,2f5,2f0,2ee,2ed,2eb,2ea,2d9],320=[31a],32b=[32a,328],351=[34e],358=[354,353],372=[372,370,36f,36b,368,366,35f]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3798679, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:48:20.733 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:28.809203+0000", 2021-02-10T04:48:20.733 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 230.55459523600001, 2021-02-10T04:48:20.733 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010095241, 2021-02-10T04:48:20.733 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:48:20.733 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:48:20.733 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:48:20.733 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.734 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:48:20.734 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.809203+0000", 2021-02-10T04:48:20.734 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:48:20.734 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.734 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.734 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:48:20.734 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.809203+0000", 2021-02-10T04:48:20.734 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00063922499999999999 2021-02-10T04:48:20.734 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.734 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.734 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:48:20.734 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.809842+0000", 2021-02-10T04:48:20.734 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.5457999999999999e-05 2021-02-10T04:48:20.735 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.735 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.735 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:48:20.735 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.809868+0000", 2021-02-10T04:48:20.735 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8017e-05 2021-02-10T04:48:20.735 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.735 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.735 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:48:20.735 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.809896+0000", 2021-02-10T04:48:20.735 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6266999999999999e-05 2021-02-10T04:48:20.735 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.735 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.735 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:48:20.735 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.809912+0000", 2021-02-10T04:48:20.736 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0083131160000000006 2021-02-10T04:48:20.736 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.736 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.736 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:48:20.736 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.818225+0000", 2021-02-10T04:48:20.736 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.6239999999999997e-06 2021-02-10T04:48:20.736 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.736 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.736 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:48:20.736 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.818235+0000", 2021-02-10T04:48:20.736 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0010635340000000001 2021-02-10T04:48:20.736 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.736 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.737 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:48:20.737 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.819298+0000", 2021-02-10T04:48:20.737 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010095241 2021-02-10T04:48:20.737 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.737 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:48:20.737 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.737 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.737 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.737 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.13 1013/1010 [PushOp(7:cf4c8c3b:::smithi08575397-6:203, version: 675'1073, data_included: [2246743~16786,2771268~573490], data_size: 590276, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:cf4c8c3b:::smithi08575397-6:203@675'1073, size: 3344758, copy_subset: [0~3344758], clone_subset: {}, snapset: 372=[]:{203=[203],27c=[27c],2bf=[2ae],2cb=[2c8],2f1=[2f0,2ee,2ed,2eb,2ea],308=[308,300,2f5],32c=[32a,328,31a],335=[335,332,32f],351=[34e,34d,34b,349,345,344,343,33e,33c,33a],364=[35f,354,353],372=[372,370,36f,36b,368,366]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3344758, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2246743, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:48:20.737 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:31.239214+0000", 2021-02-10T04:48:20.737 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 228.12458415500001, 2021-02-10T04:48:20.737 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010704108, 2021-02-10T04:48:20.737 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:48:20.737 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:48:20.738 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:48:20.738 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.738 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:48:20.738 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:31.239214+0000", 2021-02-10T04:48:20.738 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:48:20.738 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.738 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.738 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:48:20.738 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:31.239214+0000", 2021-02-10T04:48:20.738 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00052990299999999997 2021-02-10T04:48:20.738 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.738 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.738 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:48:20.739 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:31.239744+0000", 2021-02-10T04:48:20.739 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4080999999999999e-05 2021-02-10T04:48:20.739 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.739 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.739 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:48:20.739 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:31.239768+0000", 2021-02-10T04:48:20.739 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2466000000000003e-05 2021-02-10T04:48:20.739 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.739 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.739 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:48:20.739 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:31.239810+0000", 2021-02-10T04:48:20.739 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7711999999999999e-05 2021-02-10T04:48:20.740 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.740 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.740 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:48:20.740 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:31.239828+0000", 2021-02-10T04:48:20.740 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0083525189999999992 2021-02-10T04:48:20.740 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.740 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.740 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:48:20.740 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:31.248181+0000", 2021-02-10T04:48:20.740 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.2199999999999992e-06 2021-02-10T04:48:20.740 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.740 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.740 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:48:20.740 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:31.248189+0000", 2021-02-10T04:48:20.741 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001729207 2021-02-10T04:48:20.741 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.741 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.741 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:48:20.741 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:31.249918+0000", 2021-02-10T04:48:20.741 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010704108 2021-02-10T04:48:20.741 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.741 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:48:20.741 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.741 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.741 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.741 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.13 1014/1010 [PushOp(7:cd29c741:::smithi08575397-46:309, version: 997'1632, data_included: [2487424~241133,3163563~515291], data_size: 756424, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:cd29c741:::smithi08575397-46:309@997'1632, size: 3949909, copy_subset: [749909~671631,2110479~1839430], clone_subset: {7:cd29c741:::smithi08575397-46:304=[0~749909,1421540~688939]}, snapset: 369=[]:{2a2=[27c],2b6=[2ae],304=[300,2f5,2f0,2ee,2ed,2eb,2ea,2d9,2c8],309=[308],33e=[33e,33c],343=[343],364=[35f,354,353,34e,34d,34b,349,345,344]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3949909, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2487424, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:48:20.741 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:32.869894+0000", 2021-02-10T04:48:20.742 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 226.493904284, 2021-02-10T04:48:20.742 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010419648, 2021-02-10T04:48:20.742 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:48:20.742 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:48:20.742 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:48:20.742 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.742 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:48:20.742 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:32.869894+0000", 2021-02-10T04:48:20.742 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:48:20.742 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.742 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.742 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:48:20.742 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:32.869894+0000", 2021-02-10T04:48:20.743 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00079295799999999997 2021-02-10T04:48:20.743 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.743 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.743 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:48:20.743 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:32.870687+0000", 2021-02-10T04:48:20.743 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4919e-05 2021-02-10T04:48:20.743 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.743 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.743 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:48:20.743 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:32.870712+0000", 2021-02-10T04:48:20.743 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2763999999999999e-05 2021-02-10T04:48:20.743 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.743 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.743 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:48:20.744 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:32.870754+0000", 2021-02-10T04:48:20.744 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8715e-05 2021-02-10T04:48:20.744 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.744 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.744 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:48:20.744 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:32.870773+0000", 2021-02-10T04:48:20.744 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0082588120000000008 2021-02-10T04:48:20.744 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.744 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.744 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:48:20.744 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:32.879032+0000", 2021-02-10T04:48:20.744 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1678000000000001e-05 2021-02-10T04:48:20.744 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.745 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.745 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:48:20.745 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:32.879044+0000", 2021-02-10T04:48:20.745 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001269802 2021-02-10T04:48:20.745 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.745 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.745 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:48:20.745 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:32.880313+0000", 2021-02-10T04:48:20.745 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010419648 2021-02-10T04:48:20.745 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.745 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:48:20.745 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.745 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.745 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.746 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.13 1014/1010 [PushOp(7:cd29c741:::smithi08575397-46:33e, version: 998'1638, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:cd29c741:::smithi08575397-46:33e@998'1638, size: 3193627, copy_subset: [0~3193627], clone_subset: {7:cd29c741:::smithi08575397-46:309=[]}, snapset: 369=[]:{2a2=[27c],2b6=[2ae],304=[300,2f5,2f0,2ee,2ed,2eb,2ea,2d9,2c8],309=[308],33e=[33e,33c],343=[343],364=[35f,354,353,34e,34d,34b,349,345,344]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:48:20.746 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:33.012334+0000", 2021-02-10T04:48:20.746 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 226.351463878, 2021-02-10T04:48:20.746 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0098818410000000006, 2021-02-10T04:48:20.746 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:48:20.746 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:48:20.746 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:48:20.746 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.746 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:48:20.746 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.012334+0000", 2021-02-10T04:48:20.746 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:48:20.746 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.746 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.747 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:48:20.747 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.012334+0000", 2021-02-10T04:48:20.747 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001088105 2021-02-10T04:48:20.747 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.747 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.747 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:48:20.747 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.013422+0000", 2021-02-10T04:48:20.747 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.3787000000000001e-05 2021-02-10T04:48:20.747 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.747 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.747 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:48:20.747 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.013446+0000", 2021-02-10T04:48:20.747 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.8970000000000001e-05 2021-02-10T04:48:20.747 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.748 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.748 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:48:20.748 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.013485+0000", 2021-02-10T04:48:20.748 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7571000000000002e-05 2021-02-10T04:48:20.748 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.748 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.748 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:48:20.748 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.013503+0000", 2021-02-10T04:48:20.748 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0081431379999999994 2021-02-10T04:48:20.748 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.748 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.748 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:48:20.748 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.021646+0000", 2021-02-10T04:48:20.749 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.6649999999999998e-06 2021-02-10T04:48:20.749 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.749 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.749 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:48:20.749 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.021655+0000", 2021-02-10T04:48:20.749 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00056060500000000004 2021-02-10T04:48:20.749 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.749 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.749 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:48:20.749 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.022216+0000", 2021-02-10T04:48:20.749 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0098818410000000006 2021-02-10T04:48:20.749 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.749 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:48:20.750 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.750 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.750 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.750 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.13 1014/1010 [PushOp(7:cd29c741:::smithi08575397-46:33e, version: 998'1638, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:cd29c741:::smithi08575397-46:33e@998'1638, size: 3193627, copy_subset: [0~3193627], clone_subset: {7:cd29c741:::smithi08575397-46:309=[]}, snapset: 369=[]:{2a2=[27c],2b6=[2ae],304=[300,2f5,2f0,2ee,2ed,2eb,2ea,2d9,2c8],309=[308],33e=[33e,33c],343=[343],364=[35f,354,353,34e,34d,34b,349,345,344]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:48:20.750 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:33.024117+0000", 2021-02-10T04:48:20.750 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 226.33968128199999, 2021-02-10T04:48:20.750 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010067222000000001, 2021-02-10T04:48:20.750 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:48:20.750 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:48:20.750 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:48:20.750 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.750 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:48:20.750 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.024117+0000", 2021-02-10T04:48:20.751 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:48:20.751 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.751 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.751 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:48:20.751 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.024117+0000", 2021-02-10T04:48:20.751 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0012032010000000001 2021-02-10T04:48:20.751 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.751 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.751 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:48:20.751 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.025320+0000", 2021-02-10T04:48:20.751 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8218e-05 2021-02-10T04:48:20.751 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.751 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.752 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:48:20.752 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.025348+0000", 2021-02-10T04:48:20.752 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5818000000000001e-05 2021-02-10T04:48:20.752 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.752 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.752 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:48:20.752 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.025394+0000", 2021-02-10T04:48:20.752 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5656000000000001e-05 2021-02-10T04:48:20.752 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.752 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.752 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:48:20.752 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.025410+0000", 2021-02-10T04:48:20.752 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0080926039999999998 2021-02-10T04:48:20.753 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.753 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.753 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:48:20.753 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.033502+0000", 2021-02-10T04:48:20.753 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.3239999999999992e-06 2021-02-10T04:48:20.753 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.753 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.753 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:48:20.753 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.033512+0000", 2021-02-10T04:48:20.753 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00067240099999999999 2021-02-10T04:48:20.753 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.753 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.754 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:48:20.754 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.034184+0000", 2021-02-10T04:48:20.754 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010067222000000001 2021-02-10T04:48:20.754 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.754 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:48:20.754 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.754 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.754 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.754 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.13 1014/1010 [PushOp(7:cf4c8c3b:::smithi08575397-6:351, version: 1000'1646, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:cf4c8c3b:::smithi08575397-6:351@1000'1646, size: 2302856, copy_subset: [0~2302856], clone_subset: {7:cf4c8c3b:::smithi08575397-6:335=[],7:cf4c8c3b:::smithi08575397-6:372=[]}, snapset: 372=[]:{203=[203],27c=[27c],2bf=[2ae],2cb=[2c8],2f1=[2f0,2ee,2ed,2eb,2ea],308=[308,300,2f5],32c=[32a,328,31a],335=[335,332,32f],351=[34e,34d,34b,349,345,344,343,33e,33c,33a],364=[35f,354,353],372=[372,370,36f,36b,368,366]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:48:20.754 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:33.196545+0000", 2021-02-10T04:48:20.754 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 226.16725257799999, 2021-02-10T04:48:20.754 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010517324, 2021-02-10T04:48:20.754 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:48:20.755 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:48:20.755 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:48:20.755 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.755 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:48:20.755 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.196545+0000", 2021-02-10T04:48:20.755 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:48:20.755 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.755 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.755 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:48:20.755 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.196545+0000", 2021-02-10T04:48:20.755 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001506042 2021-02-10T04:48:20.755 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.755 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.755 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:48:20.756 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.198052+0000", 2021-02-10T04:48:20.756 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1785000000000001e-05 2021-02-10T04:48:20.756 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.756 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.756 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:48:20.756 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.198083+0000", 2021-02-10T04:48:20.756 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.9489000000000001e-05 2021-02-10T04:48:20.756 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.756 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.756 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:48:20.756 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.198123+0000", 2021-02-10T04:48:20.756 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0149e-05 2021-02-10T04:48:20.756 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.757 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.757 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:48:20.757 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.198143+0000", 2021-02-10T04:48:20.757 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0082678630000000003 2021-02-10T04:48:20.757 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.757 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.757 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:48:20.757 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.206411+0000", 2021-02-10T04:48:20.757 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2583e-05 2021-02-10T04:48:20.757 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.757 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.757 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:48:20.757 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.206423+0000", 2021-02-10T04:48:20.757 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00063941300000000003 2021-02-10T04:48:20.758 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.758 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.758 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:48:20.758 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.207063+0000", 2021-02-10T04:48:20.758 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010517324 2021-02-10T04:48:20.758 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.758 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:48:20.758 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.758 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.758 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.758 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(osd.3.0:2969 4.17 e1063/638 4:e9187ab8:reshard::reshard.0000000000:head v 1063'26, mlcod=1049'24)", 2021-02-10T04:48:20.758 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:47:05.585474+0000", 2021-02-10T04:48:20.759 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 73.778323756000006, 2021-02-10T04:48:20.759 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0033504899999999998, 2021-02-10T04:48:20.759 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:48:20.759 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:48:20.759 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:48:20.759 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.759 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:48:20.759 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:47:05.585474+0000", 2021-02-10T04:48:20.759 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:48:20.759 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.759 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.759 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:48:20.759 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:47:05.585474+0000", 2021-02-10T04:48:20.760 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1758000000000003e-05 2021-02-10T04:48:20.760 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.760 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.760 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:48:20.760 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:47:05.585506+0000", 2021-02-10T04:48:20.760 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.8409999999999996e-06 2021-02-10T04:48:20.760 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.760 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.760 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:48:20.760 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:47:05.585512+0000", 2021-02-10T04:48:20.760 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7399999999999999e-06 2021-02-10T04:48:20.760 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.760 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.760 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:48:20.761 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:47:05.585522+0000", 2021-02-10T04:48:20.761 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2400000000000001e-06 2021-02-10T04:48:20.761 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.761 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.761 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:48:20.761 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:47:05.585526+0000", 2021-02-10T04:48:20.761 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.0031999999999999e-05 2021-02-10T04:48:20.761 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.761 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.761 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:48:20.761 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:47:05.585566+0000", 2021-02-10T04:48:20.761 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0579000000000001e-05 2021-02-10T04:48:20.761 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.761 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.762 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:48:20.762 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:47:05.585586+0000", 2021-02-10T04:48:20.762 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0031840599999999998 2021-02-10T04:48:20.762 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.762 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.762 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:48:20.762 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:47:05.588771+0000", 2021-02-10T04:48:20.762 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.4240000000000002e-05 2021-02-10T04:48:20.762 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:48:20.762 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:48:20.762 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:48:20.762 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:47:05.588825+0000", 2021-02-10T04:48:20.763 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0033504899999999998 2021-02-10T04:48:20.763 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.763 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:48:20.763 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.763 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:48:20.763 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:48:20.763 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:48:20.763 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:48:20.765 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:20 smithi085 conmon[29319]: cluster 2021-02-10T04:48:19. 2021-02-10T04:48:20.765 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:20 smithi085 conmon[29319]: 670110+0000 osd.4 (osd.4) 2169 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:20.765 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:20 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:48:20.765 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:20 smithi085 conmon[29319]: :48:19.670117+0000 osd.4 (osd.4) 2170 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:20.765 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:20 smithi085 conmon[29319]: cluster 2021-02-10T04:48 2021-02-10T04:48:20.765 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:20 smithi085 conmon[29319]: :19.670121+0000 osd.4 (osd.4) 2171 : cluster [WRN] 2021-02-10T04:48:20.765 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:20 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:20.765 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:20 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:48:20.765 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:20 smithi085 conmon[29319]: 02-10T04:48:19.670125+0000 osd.4 (osd.4) 2172 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:20.765 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:20 smithi085 conmon[35961]: cluster 2021-02-10T04:48:19.670110+0000 2021-02-10T04:48:20.766 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:20 smithi085 conmon[35961]: osd.4 (osd.4) 2169 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:20.766 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:20 smithi085 conmon[35961]: cluster 2021-02-10T04:48:20.766 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:20 smithi085 conmon[35961]: 2021-02-10T04:48:19.670117 2021-02-10T04:48:20.766 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:20 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2170 : cluster 2021-02-10T04:48:20.766 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:20 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:20.766 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:20 smithi085 conmon[35961]: cluster 2021-02-10T04:48:19.670121+0000 osd.4 2021-02-10T04:48:20.766 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:20 smithi085 conmon[35961]: (osd.4) 2171 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:20.766 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:20 smithi085 conmon[35961]: cluster 2021-02-10T04:48:19.670125+0000 osd.4 (osd.4 2021-02-10T04:48:20.766 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:20 smithi085 conmon[35961]: ) 2172 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:20.766 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:20 smithi164 conmon[29288]: cluster 2021-02-10T04:48:19.670110+0000 osd.4 2021-02-10T04:48:20.766 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:20 smithi164 conmon[29288]: (osd.4) 2169 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:20.767 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:20 smithi164 conmon[29288]: cluster 2021-02-10T04:48:19.670117+0000 osd.4 (osd.4) 2170 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:20.767 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:20 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:48:20.767 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:20 smithi164 conmon[29288]: 02-10T04:48:19.670121+0000 osd.4 (osd.4) 2171 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:20.767 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:20 smithi164 conmon[29288]: cluster 2021-02-10T04:48:19.670125+0000 2021-02-10T04:48:20.767 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:20 smithi164 conmon[29288]: osd.4 (osd.4) 2172 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:20.804 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:48:21.145 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:48:20 smithi164 conmon[37263]: debug 2021-02-10T04:48:21.145 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:48:20 smithi164 conmon[37263]: 2021-02-10T04:48:20.902+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:48:21.145 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:20 smithi164 conmon[33677]: debug 2021-02-10T04:48:21.145 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:20 smithi164 conmon[33677]: 2021-02-10T04:48:20.706+0000 7fbec19a1700 -1 osd.4 1076 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:21.145 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:20 smithi164 conmon[33677]: 2021-02-10T04:48:21.355 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:48:21.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:21 smithi085 conmon[29319]: cluster 2021-02-10T04:48:20.355300+0000 mgr.y ( 2021-02-10T04:48:21.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:21 smithi085 conmon[29319]: mgr.14140) 1091 : cluster [DBG] pgmap v1867: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:48:21.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:21 smithi085 conmon[29319]: cluster 2021-02-10T04:48:20.708483+0000 osd.4 (osd.4) 2173 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:21.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:21 smithi085 conmon[29319]: cluster 2021-02-10T04:48:20.708492+0000 osd.4 ( 2021-02-10T04:48:21.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:21 smithi085 conmon[29319]: osd.4) 2174 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:21.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:21 smithi085 conmon[29319]: cluster 2021-02-10T04:48:20.708499+0000 osd.4 (osd.4) 2175 : cluster [WRN] 2021-02-10T04:48:21.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:21 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:21.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:21 smithi085 conmon[29319]: cluster 2021-02-10T04:48:20.708506+0000 osd.4 (osd.4) 2176 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:21.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:21 smithi085 conmon[35961]: cluster 2021-02-10T04:48:20.355300+0000 2021-02-10T04:48:21.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:21 smithi085 conmon[35961]: mgr.y (mgr.14140) 1091 : cluster [DBG] pgmap v1867: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:48:21.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:21 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:48:21.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:21 smithi085 conmon[35961]: 02-10T04:48:20.708483+0000 osd.4 ( 2021-02-10T04:48:21.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:21 smithi085 conmon[35961]: osd.4) 2173 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:21.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:21 smithi085 conmon[35961]: cluster 2021-02-10T04:48:21.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:21 smithi085 conmon[35961]: 2021-02-10T04:48:20.708492+0000 2021-02-10T04:48:21.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:21 smithi085 conmon[35961]: osd.4 (osd.4) 2174 : cluster [WRN] 2021-02-10T04:48:21.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:21 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:21.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:21 smithi085 conmon[35961]: cluster 2021-02-10T04:48:20 2021-02-10T04:48:21.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:21 smithi085 conmon[35961]: .708499+0000 osd.4 (osd.4) 2175 : 2021-02-10T04:48:21.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:21 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:21.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:21 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:48:21.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:21 smithi085 conmon[35961]: 48:20.708506+0000 osd.4 (osd.4 2021-02-10T04:48:21.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:21 smithi085 conmon[35961]: ) 2176 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:21.694 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:21 smithi164 conmon[29288]: cluster 2021-02-10T04:48:20.355300+0000 mgr.y (mgr 2021-02-10T04:48:21.694 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:21 smithi164 conmon[29288]: .14140) 1091 : cluster [DBG] pgmap v1867: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:48:21.694 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:21 smithi164 conmon[29288]: cluster 2021-02-10T04:48:20.708483+0000 osd.4 (osd.4) 2173 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:21.694 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:21 smithi164 conmon[29288]: cluster 2021-02-10T04:48:20.708492+0000 osd.4 (osd.4) 2174 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:21.694 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:21 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:48:21.694 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:21 smithi164 conmon[29288]: T04:48:20.708499+0000 osd.4 (osd.4 2021-02-10T04:48:21.694 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:21 smithi164 conmon[29288]: ) 2175 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:21.695 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:21 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:48:21.695 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:21 smithi164 conmon[29288]: 04:48:20.708506+0000 osd.4 (osd. 2021-02-10T04:48:21.695 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:21 smithi164 conmon[29288]: 4) 2176 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:21.695 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:21 smithi164 conmon[33677]: debug 2021-02-10T04:48:21.690+0000 7fbec19a1700 -1 osd.4 1076 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:22 smithi085 conmon[29319]: cluster 2021-02-10T04:48:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:22 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:48:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:22 smithi085 conmon[29319]: :48:21.692118+0000 osd.4 (osd.4) 2177 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:22 smithi085 conmon[29319]: cluster 2021-02-10T04:48:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:22 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:48:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:22 smithi085 conmon[29319]: 48:21.692124+0000 osd.4 (osd.4) 2178 2021-02-10T04:48:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:22 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:22 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:48:22.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:22 smithi085 conmon[29319]: 10T04:48:21.692128+0000 osd.4 ( 2021-02-10T04:48:22.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:22 smithi085 conmon[29319]: osd.4) 2179 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:22.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:22 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:48:22.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:22 smithi085 conmon[29319]: T04:48:21.692133+0000 osd.4 (osd. 2021-02-10T04:48:22.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:22 smithi085 conmon[29319]: 4) 2180 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:22 smithi085 conmon[35961]: cluster 2021-02-10T04:48:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:22 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:48:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:22 smithi085 conmon[35961]: T04:48:21.692118+0000 osd.4 (osd.4) 2177 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:22 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:48:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:22 smithi085 conmon[35961]: 02-10T04:48:21.692124+0000 2021-02-10T04:48:22.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:22 smithi085 conmon[35961]: osd.4 (osd.4) 2178 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:22.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:22 smithi085 conmon[35961]: cluster 2021-02-10T04:48:22.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:22 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:48:22.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:22 smithi085 conmon[35961]: T04:48:21.692128+0000 osd.4 (osd.4 2021-02-10T04:48:22.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:22 smithi085 conmon[35961]: ) 2179 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:22.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:22 smithi085 conmon[35961]: cluster 2021-02-10T04:48:22.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:22 smithi085 conmon[35961]: 2021-02-10T04:48 2021-02-10T04:48:22.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:22 smithi085 conmon[35961]: :21.692133+0000 osd.4 (osd.4) 2180 2021-02-10T04:48:22.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:22 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:22.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:22 smithi164 conmon[29288]: cluster 2021-02-10T04:48 2021-02-10T04:48:22.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:22 smithi164 conmon[29288]: :21.692118+0000 osd.4 (osd.4) 2177 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:22.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:22 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:48:22.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:22 smithi164 conmon[29288]: 04:48:21.692124+0000 osd.4 (osd.4) 2178 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:22.712 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:22 smithi164 conmon[29288]: 2021-02-10T04:48:22.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:22 smithi164 conmon[29288]: cluster 2021-02-10T04:48:21. 2021-02-10T04:48:22.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:22 smithi164 conmon[29288]: 692128+0000 osd.4 (osd.4) 2179 : cluster [WRN] 2021-02-10T04:48:22.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:22 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:22.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:22 smithi164 conmon[29288]: cluster 2021-02-10T04:48:22.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:22 smithi164 conmon[29288]: 2021-02-10T04:48:21.692133+0000 osd.4 (osd 2021-02-10T04:48:22.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:22 smithi164 conmon[29288]: .4) 2180 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:23 smithi085 conmon[29319]: cluster 2021-02-10T04:48:22.355815+0000 2021-02-10T04:48:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:23 smithi085 conmon[29319]: mgr.y (mgr.14140) 1092 : cluster [DBG] pgmap v1868: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:48:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:23 smithi085 conmon[29319]: cluster 2021-02-10T04:48:22.710663+0000 osd.4 (osd.4) 2181 : cluster 2021-02-10T04:48:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:23 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:23 smithi085 conmon[29319]: cluster 2021-02-10T04:48 2021-02-10T04:48:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:23 smithi085 conmon[29319]: :22.710673+0000 osd.4 (osd.4) 2182 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:23 smithi085 conmon[29319]: cluster 2021-02-10T04:48:22.710679+0000 2021-02-10T04:48:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:23 smithi085 conmon[29319]: osd.4 (osd.4) 2183 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:23 smithi085 conmon[29319]: cluster 2021-02-10T04:48: 2021-02-10T04:48:23.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:23 smithi085 conmon[29319]: 22.710685+0000 osd.4 (osd.4) 2184 : cluster [WRN] 2021-02-10T04:48:23.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:23 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:23 smithi085 conmon[35961]: cluster 2021-02-10T04:48:22.355815+0000 2021-02-10T04:48:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:23 smithi085 conmon[35961]: mgr.y (mgr.14140) 1092 : cluster [DBG] pgmap v1868: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:48:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:23 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:48:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:23 smithi085 conmon[35961]: -10T04:48:22.710663+0000 osd.4 (osd.4) 2181 : 2021-02-10T04:48:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:23 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:23 smithi085 conmon[35961]: cluster 2021-02-10T04:48 2021-02-10T04:48:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:23 smithi085 conmon[35961]: :22.710673+0000 osd.4 (osd. 2021-02-10T04:48:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:23 smithi085 conmon[35961]: 4) 2182 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:23 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:48:23.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:23 smithi085 conmon[35961]: 02-10T04:48:22. 2021-02-10T04:48:23.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:23 smithi085 conmon[35961]: 710679+0000 osd.4 (osd.4) 2183 : cluster 2021-02-10T04:48:23.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:23 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:23.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:23 smithi085 conmon[35961]: cluster 2021-02-10T04:48 2021-02-10T04:48:23.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:23 smithi085 conmon[35961]: :22.710685+0000 osd.4 (osd.4) 2184 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:23.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:23 smithi164 conmon[29288]: cluster 2021-02-10T04:48:22.355815+0000 mgr.y 2021-02-10T04:48:23.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:23 smithi164 conmon[29288]: (mgr.14140) 1092 : cluster [DBG] pgmap v1868: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:48:23.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:23 smithi164 conmon[29288]: cluster 2021-02-10T04:48:22.710663+0000 osd.4 (osd.4) 2181 : cluster 2021-02-10T04:48:23.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:23 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:23.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:23 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:48:23.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:23 smithi164 conmon[29288]: -02-10T04:48:22.710673+0000 osd.4 (osd.4) 2182 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:23.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:23 smithi164 conmon[29288]: cluster 2021-02-10T04:48:22.710679+0000 osd.4 2021-02-10T04:48:23.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:23 smithi164 conmon[29288]: (osd.4) 2183 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:23.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:23 smithi164 conmon[29288]: cluster 2021-02-10T04:48:23.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:23 smithi164 conmon[29288]: 2021-02-10T04:48:23.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:23 smithi164 conmon[29288]: 2021-02-10 2021-02-10T04:48:23.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:23 smithi164 conmon[29288]: T04:48:22.710685+0000 osd.4 (osd.4) 2184 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:23.896 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:22 smithi164 conmon[33677]: debug 2021-02-10T04:48:22.708+0000 7fbec19a1700 -1 osd.4 1076 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:23.896 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:23 smithi164 conmon[33677]: debug 2021-02-10T04:48:23.680+0000 7fbec19a1700 -1 osd.4 1076 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:24.697 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:24 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:48:24.698 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:24 smithi164 conmon[29288]: -10T04:48:23.681937+0000 osd.4 (osd.4) 2185 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:24.698 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:24 smithi164 conmon[29288]: cluster 2021-02-10T04:48:23.681947+0000 osd.4 (osd.4) 2186 : cluster [WRN] 2021-02-10T04:48:24.698 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:24 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:24.698 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:24 smithi164 conmon[29288]: cluster 2021-02-10T04:48: 2021-02-10T04:48:24.698 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:24 smithi164 conmon[29288]: 23.681952+0000 osd.4 (osd.4) 2187 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:24.699 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:24 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:48:24.699 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:24 smithi164 conmon[29288]: T04:48:23.681956+0000 osd.4 (osd 2021-02-10T04:48:24.699 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:24 smithi164 conmon[29288]: .4) 2188 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:24.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[29319]: cluster 2021-02-10T04:48:23.681937 2021-02-10T04:48:24.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2185 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:24.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[29319]: cluster 2021-02-10T04:48:23.681947+0000 osd.4 (osd.4) 2186 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:24.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[29319]: cluster 2021-02-10T04:48:23.681952+0000 osd.4 (osd.4) 2187 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:24.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:48:24.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[29319]: T04:48:23.681956+0000 osd.4 (osd.4) 2188 : cluster 2021-02-10T04:48:24.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:24.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: cluster 2021-02-10T04:48:24.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: 2021-02-10T04:48:24.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: 2021 2021-02-10T04:48:24.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: - 2021-02-10T04:48:24.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: 02 2021-02-10T04:48:24.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: - 2021-02-10T04:48:24.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: 10 2021-02-10T04:48:24.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: T 2021-02-10T04:48:24.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: 04 2021-02-10T04:48:24.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: : 2021-02-10T04:48:24.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: 48 2021-02-10T04:48:24.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: : 2021-02-10T04:48:24.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: 23 2021-02-10T04:48:24.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: . 2021-02-10T04:48:24.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: 681937 2021-02-10T04:48:24.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: +0000 2021-02-10T04:48:24.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: 2021-02-10T04:48:24.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: osd.4 2021-02-10T04:48:24.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: ( 2021-02-10T04:48:24.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: osd 2021-02-10T04:48:24.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: . 2021-02-10T04:48:24.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: 4 2021-02-10T04:48:24.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: ) 2021-02-10T04:48:24.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: 2185 2021-02-10T04:48:24.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: : 2021-02-10T04:48:24.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: cluster 2021-02-10T04:48:24.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: 2021-02-10T04:48:24.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: [WRN] 2021-02-10T04:48:24.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: 2021-02-10T04:48:24.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:24.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: 2021-02-10T04:48:24.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: cluster 2021-02-10T04:48:24.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: 2021-02-10T04:48:24.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: 2021 2021-02-10T04:48:24.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: - 2021-02-10T04:48:24.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: 02 2021-02-10T04:48:24.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: - 2021-02-10T04:48:24.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: 10 2021-02-10T04:48:24.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: T 2021-02-10T04:48:24.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: 04 2021-02-10T04:48:24.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: :48:23.681947+0000 osd.4 (osd.4) 2186 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:24.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: cluster 2021-02-10T04:48:24.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: 2021 2021-02-10T04:48:24.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: - 2021-02-10T04:48:24.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: 02 2021-02-10T04:48:24.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: - 2021-02-10T04:48:24.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: 10 2021-02-10T04:48:24.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: T 2021-02-10T04:48:24.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: 04 2021-02-10T04:48:24.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: : 2021-02-10T04:48:24.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: 48 2021-02-10T04:48:24.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: :23.681952+0000 osd.4 ( 2021-02-10T04:48:24.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: osd.4) 2187 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:24.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: cluster 2021-02-10T04:48:24.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:48:24.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: 48:23.681956+0000 osd.4 (osd.4) 2188 2021-02-10T04:48:24.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:24 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:25.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:24 smithi164 conmon[33677]: debug 2021-02-10T04:48:25.145 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:24 smithi164 conmon[33677]: 2021-02-10T04:48:24.694+0000 7fbec19a1700 -1 osd.4 1076 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:25.730 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:25 smithi164 conmon[29288]: cluster 2021-02-10T04:48:24.356403+0000 mgr.y (mgr.14140) 1093 : cluster 2021-02-10T04:48:25.731 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:25 smithi164 conmon[29288]: [DBG] pgmap v1869: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:48:25.731 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:25 smithi164 conmon[29288]: cluster 2021-02-10T04:48:24.696512+0000 osd.4 (osd.4) 2189 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:25.731 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:25 smithi164 conmon[29288]: cluster 2021-02-10T04:48:24.696524+0000 osd.4 (osd.4) 2190 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:25.731 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:25 smithi164 conmon[29288]: cluster 2021-02-10T04:48:24.696531+0000 osd.4 (osd.4) 2191 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:25.731 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:25 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:48:25.731 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:25 smithi164 conmon[29288]: 02-10T04:48:24.696537+0000 osd.4 (osd.4) 2192 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:25.754 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:25 smithi085 conmon[29319]: cluster 2021-02-10T04:48:24.356403+0000 mgr.y (mgr.14140) 1093 : cluster [DBG] pgmap v1869: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:48:25.754 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:25 smithi085 conmon[29319]: cluster 2021-02-10T04:48:24.696512+0000 osd.4 (osd.4) 2189 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:25.754 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:25 smithi085 conmon[29319]: cluster 2021-02-10T04:48:24.696524+0000 osd.4 (osd.4) 2190 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:25.754 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:25 smithi085 conmon[29319]: cluster 2021-02-10T04:48:24.696531+0000 osd.4 (osd.4) 2191 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:25.754 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:25 smithi085 conmon[29319]: cluster 2021-02-10T04:48:24.696537+0000 osd.4 (osd.4) 2192 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:25.754 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:25 smithi085 conmon[35961]: cluster 2021-02-10T04:48:24.356403+0000 mgr.y (mgr.14140) 1093 : cluster [DBG] pgmap v1869: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:48:25.755 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:25 smithi085 conmon[35961]: cluster 2021-02-10T04:48:24.696512+0000 osd.4 (osd.4) 2189 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:25.755 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:25 smithi085 conmon[35961]: cluster 2021-02-10T04:48:24.696524+0000 osd.4 (osd.4) 2190 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:25.755 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:25 smithi085 conmon[35961]: cluster 2021-02-10T04:48:24.696531+0000 osd.4 (osd.4) 2191 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:25.755 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:25 smithi085 conmon[35961]: cluster 2021-02-10T04:48:24.696537+0000 osd.4 (osd.4) 2192 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:26.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:25 smithi164 conmon[33677]: debug 2021-02-10T04:48:26.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:25 smithi164 conmon[33677]: 2021-02-10T04:48:25.728+0000 7fbec19a1700 -1 osd.4 1076 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:26.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:25 smithi164 conmon[33677]: 2021-02-10T04:48:26.456 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:26 smithi085 conmon[29319]: cluster 2021-02-10T04:48:25.730055+0000 osd.4 (osd.4) 2193 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:26.456 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:26 smithi085 conmon[29319]: cluster 2021-02-10T04:48:25.730073+0000 osd.4 (osd.4) 2194 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:26.456 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:26 smithi085 conmon[29319]: cluster 2021-02-10T04:48:25.730088+0000 osd.4 (osd.4) 2195 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:26.456 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:26 smithi085 conmon[29319]: cluster 2021-02-10T04:48:25.730094+0000 osd.4 (osd.4) 2196 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:26.750 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:26 smithi164 conmon[29288]: cluster 2021-02-10T04:48:25.730055+0000 osd.4 (osd. 2021-02-10T04:48:26.750 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:26 smithi164 conmon[29288]: 4) 2193 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:26.750 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:26 smithi164 conmon[29288]: cluster 2021-02-10T04:48:25.730073+0000 osd.4 (osd.4) 2194 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:26.750 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:26 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:48:26.750 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:26 smithi164 conmon[29288]: :48:25.730088+0000 osd.4 ( 2021-02-10T04:48:26.750 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:26 smithi164 conmon[29288]: osd. 2021-02-10T04:48:26.751 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:26 smithi164 conmon[29288]: 4 2021-02-10T04:48:26.751 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:26 smithi164 conmon[29288]: ) 2195 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:26.751 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:26 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:48:26.751 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:26 smithi164 conmon[29288]: 02-10T04:48:25.730094+0000 osd.4 ( 2021-02-10T04:48:26.751 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:26 smithi164 conmon[29288]: osd.4) 2196 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:26.751 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:26 smithi164 conmon[33677]: debug 2021-02-10T04:48:26.748+0000 7fbec19a1700 -1 osd.4 1077 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:26.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:26 smithi085 conmon[35961]: cluster 2021-02-10T04:48:26.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:26 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:48:26.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:26 smithi085 conmon[35961]: 48:25.730055+0000 osd.4 (osd.4) 2193 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:26.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:26 smithi085 conmon[35961]: cluster 2021-02-10T04:48:26.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:26 smithi085 conmon[35961]: 2021-02-10T04:48:26.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:26 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:48:26.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:26 smithi085 conmon[35961]: 48:25.730073+0000 osd.4 (osd.4) 2194 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:26.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:26 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:48:26.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:26 smithi085 conmon[35961]: 02-10T04:48:25.730088+0000 osd.4 2021-02-10T04:48:26.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:26 smithi085 conmon[35961]: (osd.4) 2195 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:26.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:26 smithi085 conmon[35961]: cluster 2021-02-10T04:48:26.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:26 smithi085 conmon[35961]: 2021-02-10T04:48:25.730094+0000 2021-02-10T04:48:26.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:26 smithi085 conmon[35961]: osd.4 (osd.4) 2196 : cluster [WRN] 2021-02-10T04:48:26.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:26 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:27.477 INFO:teuthology.orchestra.run.smithi085.stderr:clear 4.1c pg_upmap mapping 2021-02-10T04:48:27.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[33677]: debug 2021-02-10T04:48:27.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[33677]: 2021-02-10T04:48:27.702+0000 7fbec19a1700 -1 osd.4 1078 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:27.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: cluster 2021-02-10T04:48:26.323198 2021-02-10T04:48:27.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: +0000 osd.6 (osd.6) 135 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:48:27.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: cluster 2021-02-10T04:48:27.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: 2021 2021-02-10T04:48:27.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: - 2021-02-10T04:48:27.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: 02-10T04:48:26.327662+0000 2021-02-10T04:48:27.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: osd.6 (osd.6) 136 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:48:27.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:48:27.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: -02-10T04:48:26.357055+0000 mgr.y ( 2021-02-10T04:48:27.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: mgr.14140) 1094 : cluster [DBG] pgmap v1870: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1013 B/s rd, 0 op/s 2021-02-10T04:48:27.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: cluster 2021-02-10T04:48:27.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: 2021-02-10T04:48:26.459645+0000 2021-02-10T04:48:27.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: mon.a (mon.0) 2158 : cluster [DBG] 2021-02-10T04:48:27.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: osdmap e1077: 8 total, 8 up, 7 in 2021-02-10T04:48:27.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:48:27.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: -02-10T04:48:26. 2021-02-10T04:48:27.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: 749727+0000 osd.4 (osd.4) 2197 : cluster [WRN] 2021-02-10T04:48:27.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:27.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:48:27.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: :48:26. 2021-02-10T04:48:27.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: 749737+0000 osd.4 (osd.4) 2198 : cluster [WRN] 2021-02-10T04:48:27.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:27.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:48:27.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: T04:48:26.749744+0000 osd.4 (osd 2021-02-10T04:48:27.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: .4) 2199 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:27.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: cluster 2021-02-10T04:48:27.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:48:27.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: 48:26.749750+0000 osd.4 (osd.4) 2200 : 2021-02-10T04:48:27.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:27.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:48:27.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: 02-10T04:48:27.100723+0000 2021-02-10T04:48:27.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:27 smithi164 conmon[29288]: mon.a (mon.0) 2159 : audit [INF] from='client.? 172.21.15.85:0/1301717988' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "4.1c"}]: dispatch 2021-02-10T04:48:27.916 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:48:27.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[29319]: cluster 2021-02-10T04:48:26.323198+0000 osd.6 (osd.6) 135 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:48:27.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[29319]: cluster 2021-02-10T04:48:26.327662+0000 osd.6 (osd.6) 136 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:48:27.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:48:27.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[29319]: -10T04:48:26.357055+0000 2021-02-10T04:48:27.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[29319]: mgr.y (mgr.14140) 1094 : cluster [DBG] pgmap v1870: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1013 B/s rd, 0 op/s 2021-02-10T04:48:27.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[29319]: 2021-02-10T04:48:27.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[29319]: cluster 2021-02-10T04:48:26 2021-02-10T04:48:27.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[29319]: .459645+0000 mon.a (mon.0) 2158 : cluster 2021-02-10T04:48:27.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[29319]: [DBG] 2021-02-10T04:48:27.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[29319]: osdmap e1077: 8 total, 8 up, 7 in 2021-02-10T04:48:27.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[29319]: cluster 2021-02-10T04:48:26. 2021-02-10T04:48:27.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[29319]: 749727+0000 osd.4 (osd.4) 2197 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:27.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[29319]: cluster 2021-02-10T04:48:26.749737+0000 2021-02-10T04:48:27.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[29319]: osd.4 (osd.4) 2198 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:27.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[29319]: cluster 2021-02-10T04:48:26.749744+0000 osd.4 (osd 2021-02-10T04:48:27.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[29319]: .4) 2199 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:27.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:48:27.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[29319]: -02-10T04:48:26.749750+0000 osd.4 (osd.4) 2200 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:27.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[29319]: audit 2021-02-10T04:48:27.100723+0000 mon.a 2021-02-10T04:48:27.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[29319]: (mon.0) 2159 : audit [INF] from='client.? 172.21.15.85:0/1301717988' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "4.1c"}]: dispatch 2021-02-10T04:48:27.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[35961]: cluster 2021-02-10T04:48:27.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[35961]: 2021-02-10T04:48:27.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[35961]: 2021 2021-02-10T04:48:27.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[35961]: -02-10T04:48:26.323198 2021-02-10T04:48:27.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[35961]: +0000 osd.6 (osd.6) 135 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:48:27.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:48:27.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[35961]: T04:48:26.327662+0000 osd.6 (osd 2021-02-10T04:48:27.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[35961]: .6) 136 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:48:27.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:48:27.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[35961]: 10T04:48:26.357055+0000 mgr.y 2021-02-10T04:48:27.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[35961]: (mgr.14140) 1094 : cluster [DBG] pgmap v1870: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1013 B/s rd, 0 op/s 2021-02-10T04:48:27.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[35961]: 2021-02-10T04:48:27.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:48:27.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[35961]: -02-10T04:48:26.459645+0000 mon.a (mon.0 2021-02-10T04:48:27.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[35961]: ) 2158 : cluster [DBG] osdmap e1077: 8 total, 8 up, 7 in 2021-02-10T04:48:27.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:48:27.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[35961]: -10T04:48:26.749727+0000 osd.4 ( 2021-02-10T04:48:27.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[35961]: osd.4) 2197 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:27.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[35961]: cluster 2021-02-10T04:48:26 2021-02-10T04:48:27.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[35961]: .749737+0000 osd.4 (osd.4) 2198 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:27.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[35961]: cluster 2021-02-10T04:48:26.749744+0000 2021-02-10T04:48:27.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[35961]: osd.4 (osd.4) 2199 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:27.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[35961]: cluster 2021-02-10T04:48:26.749750+0000 2021-02-10T04:48:27.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[35961]: osd.4 (osd.4) 2200 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:27.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[35961]: audit 2021-02-10T04:48:27.100723+0000 mon.a 2021-02-10T04:48:27.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:27 smithi085 conmon[35961]: (mon.0) 2159 : audit [INF] from='client.? 172.21.15.85:0/1301717988' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "4.1c"}]: dispatch 2021-02-10T04:48:27.952 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset noscrub 2021-02-10T04:48:28.608 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:48:28.614 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:48:28.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[29319]: audit 2021-02-10T04:48:27.461471+0000 mon.a (mon. 2021-02-10T04:48:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[29319]: 0) 2160 : audit [INF] from='client.? 172.21.15.85:0/1301717988' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "4.1c"}]': finished 2021-02-10T04:48:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[29319]: cluster 2021-02-10T04:48:27.461533+0000 mon.a (mon.0) 2161 : cluster [DBG] osdmap e1078: 8 total, 8 up, 7 in 2021-02-10T04:48:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[29319]: cluster 2021-02-10T04:48:27.703878+0000 osd.4 (osd.4) 2201 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:28.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[29319]: cluster 2021-02-10T04:48:27.703887+0000 osd.4 (osd.4) 2202 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:28.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[29319]: cluster 2021-02-10T04:48:27.703892+0000 osd.4 (osd.4) 2203 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:28.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[29319]: cluster 2021-02-10T04:48:27.703898+0000 2021-02-10T04:48:28.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[29319]: osd.4 (osd.4) 2204 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:28.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[29319]: cluster 2021-02-10T04:48:28.421627+0000 mon.a (mon.0) 2021-02-10T04:48:28.696 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[29319]: 2162 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 237 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:48:28.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: audit 2021-02-10T04:48:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 2021-02-10T04:48:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 2021 2021-02-10T04:48:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: - 2021-02-10T04:48:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 02-10T04:48:27.461471 2021-02-10T04:48:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2160 : audit [INF] from='client.? 172.21.15.85:0/1301717988' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "4.1c"}]': finished 2021-02-10T04:48:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:48:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 10T04:48:27.461533+0000 mon.a (mon.0) 2161 : cluster [DBG] osdmap e1078: 8 total, 8 up, 7 in 2021-02-10T04:48:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 2021-02-10T04:48:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: cluster 2021-02-10T04:48:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 2021-02-10T04:48:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 2021 2021-02-10T04:48:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: - 2021-02-10T04:48:28.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 02 2021-02-10T04:48:28.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: - 2021-02-10T04:48:28.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 10 2021-02-10T04:48:28.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: T 2021-02-10T04:48:28.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 04 2021-02-10T04:48:28.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: : 2021-02-10T04:48:28.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 48 2021-02-10T04:48:28.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: : 2021-02-10T04:48:28.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 27 2021-02-10T04:48:28.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: . 2021-02-10T04:48:28.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 703878 2021-02-10T04:48:28.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: +0000 2021-02-10T04:48:28.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 2021-02-10T04:48:28.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: osd.4 2021-02-10T04:48:28.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: ( 2021-02-10T04:48:28.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: osd 2021-02-10T04:48:28.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: . 2021-02-10T04:48:28.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 4 2021-02-10T04:48:28.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: ) 2021-02-10T04:48:28.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 2201 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:28.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: cluster 2021-02-10T04:48:28.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 2021-02-10T04:48:28.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 2021 2021-02-10T04:48:28.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: - 2021-02-10T04:48:28.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 02 2021-02-10T04:48:28.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: - 2021-02-10T04:48:28.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 10 2021-02-10T04:48:28.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: T 2021-02-10T04:48:28.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 04 2021-02-10T04:48:28.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: : 2021-02-10T04:48:28.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 48 2021-02-10T04:48:28.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: : 2021-02-10T04:48:28.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 27 2021-02-10T04:48:28.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: . 2021-02-10T04:48:28.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 703887 2021-02-10T04:48:28.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: +0000 2021-02-10T04:48:28.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: osd.4 (osd.4) 2202 2021-02-10T04:48:28.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:28.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: cluster 2021-02-10T04:48:28.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 2021-02-10T04:48:28.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 2021 2021-02-10T04:48:28.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: - 2021-02-10T04:48:28.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 02 2021-02-10T04:48:28.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: - 2021-02-10T04:48:28.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 10 2021-02-10T04:48:28.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: T 2021-02-10T04:48:28.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 04 2021-02-10T04:48:28.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: : 2021-02-10T04:48:28.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 48 2021-02-10T04:48:28.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: : 2021-02-10T04:48:28.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 27 2021-02-10T04:48:28.701 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: . 2021-02-10T04:48:28.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 703892 2021-02-10T04:48:28.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: +0000 2021-02-10T04:48:28.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 2021-02-10T04:48:28.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: osd.4 2021-02-10T04:48:28.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: ( 2021-02-10T04:48:28.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: osd 2021-02-10T04:48:28.702 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: . 2021-02-10T04:48:28.703 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 4 2021-02-10T04:48:28.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: ) 2021-02-10T04:48:28.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 2203 2021-02-10T04:48:28.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: : 2021-02-10T04:48:28.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: cluster 2021-02-10T04:48:28.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 2021-02-10T04:48:28.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: [WRN] 2021-02-10T04:48:28.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 2021-02-10T04:48:28.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:28.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 2021-02-10T04:48:28.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: cluster 2021-02-10T04:48:28.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:48:28.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: T04:48:27.703898+0000 osd.4 (osd.4) 2204 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:28.704 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 2021-02-10T04:48:28.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:48:28.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 02-10T04:48:28.421627+0000 mon.a (mon. 2021-02-10T04:48:28.705 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:28 smithi085 conmon[35961]: 0) 2162 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 237 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:48:28.888 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:28 smithi164 conmon[33677]: debug 2021-02-10T04:48:28.731+0000 7fbec19a1700 -1 osd.4 1079 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:28.889 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:48:28 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:48:28.889 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:28 smithi164 conmon[29288]: audit 2021-02-10T04:48:27.461471+0000 mon.a (mon.0) 2160 : 2021-02-10T04:48:28.889 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:28 smithi164 conmon[29288]: audit [INF] from='client.? 172.21.15.85:0/1301717988' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "4.1c"}]': finished 2021-02-10T04:48:28.889 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:28 smithi164 conmon[29288]: cluster 2021-02-10T04:48:27.461533+0000 mon.a (mon.0) 2161 : cluster [DBG] osdmap e1078: 8 total, 8 up, 7 in 2021-02-10T04:48:28.889 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:28 smithi164 conmon[29288]: cluster 2021-02-10T04:48:27.703878+0000 2021-02-10T04:48:28.890 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:28 smithi164 conmon[29288]: osd.4 (osd.4) 2201 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:28.890 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:28 smithi164 conmon[29288]: cluster 2021-02-10T04:48 2021-02-10T04:48:28.890 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:28 smithi164 conmon[29288]: :27.703887+0000 osd.4 (osd.4) 2202 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:28.890 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:28 smithi164 conmon[29288]: cluster 2021-02-10T04:48:27.703892+0000 osd.4 ( 2021-02-10T04:48:28.890 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:28 smithi164 conmon[29288]: osd.4) 2203 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:28.890 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:28 smithi164 conmon[29288]: cluster 2021-02-10T04:48:27.703898+0000 2021-02-10T04:48:28.890 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:28 smithi164 conmon[29288]: osd.4 (osd.4) 2204 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:28.890 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:28 smithi164 conmon[29288]: cluster 2021-02-10T04:48: 2021-02-10T04:48:28.890 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:28 smithi164 conmon[29288]: 28.421627+0000 mon.a (mon.0) 2162 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 237 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:48:29.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:29 smithi164 conmon[29288]: cluster 2021-02-10T04:48:28. 2021-02-10T04:48:29.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:29 smithi164 conmon[29288]: 357680+0000 mgr.y (mgr.14140) 1095 : cluster [DBG] pgmap v1873: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:48:29.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:29 smithi164 conmon[29288]: cluster 2021-02-10T04:48:28.473000+0000 mon.a (mon.0) 2163 : cluster 2021-02-10T04:48:29.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:29 smithi164 conmon[29288]: [DBG] osdmap e1079: 8 total, 8 up, 7 in 2021-02-10T04:48:29.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:29 smithi164 conmon[29288]: cluster 2021-02-10T04:48:29.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:29 smithi164 conmon[29288]: 2021-02-10T04:48: 2021-02-10T04:48:29.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:29 smithi164 conmon[29288]: 28.733127+0000 osd.4 (osd.4) 2205 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:29 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:48:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:29 smithi164 conmon[29288]: 48:28.733134+0000 osd.4 (osd.4) 2206 : cluster 2021-02-10T04:48:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:29 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:29 smithi164 conmon[29288]: cluster 2021-02-10T04:48:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:29 smithi164 conmon[29288]: 2021-02-10T04:48:28 2021-02-10T04:48:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:29 smithi164 conmon[29288]: .733139+0000 osd.4 ( 2021-02-10T04:48:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:29 smithi164 conmon[29288]: osd.4) 2207 : cluster 2021-02-10T04:48:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:29 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:29 smithi164 conmon[29288]: cluster 2021-02-10T04:48:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:29 smithi164 conmon[29288]: 2021-02-10T04:48:28.733142+0000 2021-02-10T04:48:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:29 smithi164 conmon[29288]: osd.4 (osd.4) 2208 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:29 smithi164 conmon[29288]: 2021-02-10T04:48:29.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:29 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:48:29.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:29 smithi164 conmon[29288]: -10T04:48:28. 2021-02-10T04:48:29.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:29 smithi164 conmon[29288]: 888049+0000 mgr.y (mgr.14140) 1096 2021-02-10T04:48:29.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:29 smithi164 conmon[29288]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:48:29.896 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:29 smithi164 conmon[33677]: debug 2021-02-10T04:48:29.733+0000 7fbec19a1700 -1 osd.4 1079 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:29.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[35961]: cluster 2021-02-10T04:48 2021-02-10T04:48:29.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[35961]: :28.357680+0000 mgr.y (mgr.14140) 1095 : cluster [DBG] pgmap v1873: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:48:29.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:48:29.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[35961]: 48:28.473000+0000 mon.a (mon.0) 2163 : cluster [DBG] osdmap e1079: 8 total, 8 up, 7 in 2021-02-10T04:48:29.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:48:29.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[35961]: 02-10T04:48:28.733127+0000 osd.4 (osd 2021-02-10T04:48:29.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[35961]: .4) 2205 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:29.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[35961]: cluster 2021-02-10T04:48:29.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:48:29.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[35961]: 04:48:28.733134+0000 osd.4 (osd.4) 2206 : cluster 2021-02-10T04:48:29.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:29.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[35961]: cluster 2021-02-10T04:48:29.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[35961]: 2021-02 2021-02-10T04:48:29.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[35961]: -10T04:48:28.733139+0000 osd.4 2021-02-10T04:48:29.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[35961]: (osd.4) 2207 2021-02-10T04:48:29.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:29.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:48:29.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[35961]: 02-10T04:48:28.733142+0000 osd.4 (osd 2021-02-10T04:48:29.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[35961]: .4) 2208 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:29.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[35961]: audit 2021-02-10T04:48:29.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:48:29.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[35961]: 04:48:28.888049+0000 mgr.y (mgr.14140) 1096 : audit [DBG] 2021-02-10T04:48:29.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[35961]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:48:29.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[29319]: cluster 2021-02-10T04:48:29.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:48:29.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[29319]: 10T04:48:28.357680+0000 mgr.y (mgr.14140) 1095 : cluster [DBG] pgmap v1873: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:48:29.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:48:29.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[29319]: 48:28.473000+0000 mon.a (mon.0) 2163 2021-02-10T04:48:29.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[29319]: : cluster [DBG] osdmap e1079: 8 total, 8 up, 7 in 2021-02-10T04:48:29.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:48:29.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[29319]: -10T04:48:28.733127+0000 osd.4 (osd 2021-02-10T04:48:29.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[29319]: .4) 2205 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:29.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[29319]: cluster 2021-02-10T04:48 2021-02-10T04:48:29.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[29319]: :28.733134+0000 osd.4 (osd.4) 2206 2021-02-10T04:48:29.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:29.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:48:29.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[29319]: 04:48:28.733139+0000 osd.4 (osd. 2021-02-10T04:48:29.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[29319]: 4) 2207 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:29.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:48:29.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[29319]: 10T04:48:28.733142 2021-02-10T04:48:29.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2208 : cluster [WRN] 2021-02-10T04:48:29.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:29.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[29319]: audit 2021-02-10T04:48:28.888049+0000 mgr.y (mgr.14140) 2021-02-10T04:48:29.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:29 smithi085 conmon[29319]: 1096 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:48:30.183 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:48:30.212 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:48:30.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:30 smithi085 conmon[35961]: cluster 2021-02-10T04:48: 2021-02-10T04:48:30.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:30 smithi085 conmon[35961]: 29.735052+0000 osd.4 (osd.4) 2209 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:30.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:30 smithi085 conmon[35961]: cluster 2021-02-10T04:48:30.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:30 smithi085 conmon[35961]: 2021-02-10T04:48:29.735058+0000 osd.4 ( 2021-02-10T04:48:30.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:30 smithi085 conmon[35961]: osd.4) 2210 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:30.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:30 smithi085 conmon[35961]: cluster 2021-02-10T04:48:30.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:30 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:48:30.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:30 smithi085 conmon[35961]: :48:29.735062+0000 osd.4 (osd.4) 2211 2021-02-10T04:48:30.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:30 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:30.541 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:30 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:48:30.542 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:30 smithi085 conmon[35961]: 02-10T04:48:29.735065+0000 osd.4 ( 2021-02-10T04:48:30.542 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:30 smithi085 conmon[35961]: osd.4) 2212 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:30 smithi164 conmon[29288]: cluster 2021-02-10T04:48:29.735052+0000 osd.4 (osd.4) 2209 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:30 smithi164 conmon[29288]: cluster 2021-02-10T04:48:29.735058+0000 osd.4 (osd.4) 2210 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:30 smithi164 conmon[29288]: cluster 2021-02-10T04:48:29.735062+0000 osd.4 ( 2021-02-10T04:48:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:30 smithi164 conmon[29288]: osd.4) 2211 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:30 smithi164 conmon[29288]: cluster 2021-02-10T04:48: 2021-02-10T04:48:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:30 smithi164 conmon[29288]: 29.735065+0000 osd.4 (osd.4) 2212 : cluster 2021-02-10T04:48:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:30 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:30.895 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:30 smithi164 conmon[33677]: debug 2021-02-10T04:48:30.895 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:30 smithi164 conmon[33677]: 2021-02-10T04:48:30.738+0000 7fbec19a1700 -1 osd.4 1079 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:30.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:30 smithi085 conmon[29319]: cluster 2021-02-10T04:48:30.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:30 smithi085 conmon[29319]: 2021-02-10T04:48:30.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:30 smithi085 conmon[29319]: 2021 2021-02-10T04:48:30.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:30 smithi085 conmon[29319]: - 2021-02-10T04:48:30.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:30 smithi085 conmon[29319]: 02 2021-02-10T04:48:30.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:30 smithi085 conmon[29319]: -10T04:48:29.735052+0000 2021-02-10T04:48:30.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:30 smithi085 conmon[29319]: osd.4 (osd.4) 2209 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:30.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:30 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:48:30.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:30 smithi085 conmon[29319]: 02-10T04:48:29.735058+0000 osd.4 (osd.4) 2021-02-10T04:48:30.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:30 smithi085 conmon[29319]: 2210 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:30.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:30 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:48:30.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:30 smithi085 conmon[29319]: 10T04:48:29.735062+0000 osd.4 (osd. 2021-02-10T04:48:30.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:30 smithi085 conmon[29319]: 4) 2211 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:30.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:30 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:48:30.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:30 smithi085 conmon[29319]: -02-10T04:48:29.735065+0000 2021-02-10T04:48:30.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:30 smithi085 conmon[29319]: osd.4 (osd.4) 2212 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:31.429 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.0 dump_ops_in_flight 2021-02-10T04:48:31.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:31 smithi164 conmon[29288]: cluster 2021-02-10T04:48:30.358462+0000 2021-02-10T04:48:31.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:31 smithi164 conmon[29288]: mgr.y (mgr.14140) 1097 : cluster [DBG] pgmap v1875: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:48:31.720 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:31 smithi164 conmon[29288]: cluster 2021-02-10T04:48:30.739091+0000 osd.4 (osd.4) 2213 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:31.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:31 smithi164 conmon[29288]: cluster 2021-02-10T04:48:30.739103+0000 osd.4 2021-02-10T04:48:31.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:31 smithi164 conmon[29288]: (osd.4) 2214 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:31.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:31 smithi164 conmon[29288]: cluster 2021-02-10T04:48:30.739110+0000 osd.4 2021-02-10T04:48:31.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:31 smithi164 conmon[29288]: (osd.4) 2215 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:31.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:31 smithi164 conmon[29288]: cluster 2021-02-10T04:48:31.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:31 smithi164 conmon[29288]: 2021-02-10T04:48:30.739116+0000 osd.4 (osd.4 2021-02-10T04:48:31.721 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:31 smithi164 conmon[29288]: ) 2216 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:31.732 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:48:31.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:31 smithi085 conmon[29319]: cluster 2021-02-10T04:48:31.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:31 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:48:31.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:31 smithi085 conmon[29319]: T04:48:30.358462+0000 mgr.y (mgr.14140) 1097 : cluster [DBG] pgmap v1875: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:48:31.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:31 smithi085 conmon[29319]: cluster 2021-02-10T04:48:31.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:31 smithi085 conmon[29319]: 2021-02-10T04:48:31.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:31 smithi085 conmon[29319]: 2021-02 2021-02-10T04:48:31.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:31 smithi085 conmon[29319]: -10T04:48:30.739091+0000 osd.4 (osd. 2021-02-10T04:48:31.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:31 smithi085 conmon[29319]: 4) 2213 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:31.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:31 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:48:31.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:31 smithi085 conmon[29319]: 02-10T04:48:30.739103+0000 osd.4 (osd.4) 2021-02-10T04:48:31.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:31 smithi085 conmon[29319]: 2214 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:31.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:31 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:48:31.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:31 smithi085 conmon[29319]: 48:30.739110+0000 osd.4 (osd.4) 2215 2021-02-10T04:48:31.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:31 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:31.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:31 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:48:31.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:31 smithi085 conmon[29319]: :48:30.739116+0000 osd.4 (osd.4) 2021-02-10T04:48:31.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:31 smithi085 conmon[29319]: 2216 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:31.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:31 smithi085 conmon[35961]: cluster 2021-02-10T04:48:30.358462+0000 mgr.y (mgr.14140) 1097 : 2021-02-10T04:48:31.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:31 smithi085 conmon[35961]: cluster [DBG] pgmap v1875: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:48:31.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:31 smithi085 conmon[35961]: cluster 2021-02-10T04:48: 2021-02-10T04:48:31.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:31 smithi085 conmon[35961]: 30.739091+0000 osd.4 (osd.4) 2213 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:31.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:31 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:48:31.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:31 smithi085 conmon[35961]: -10T04:48:30.739103+0000 osd.4 (osd.4) 2214 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:31.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:31 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:48:31.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:31 smithi085 conmon[35961]: 04:48:30.739110+0000 osd.4 (osd.4) 2215 : cluster 2021-02-10T04:48:31.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:31 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:31.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:31 smithi085 conmon[35961]: cluster 2021-02-10T04:48:31.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:31 smithi085 conmon[35961]: 2021-02-10T04:48:30.739116+0000 osd.4 (osd.4) 2216 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:32.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:31 smithi164 conmon[33677]: debug 2021-02-10T04:48:31.719+0000 7fbec19a1700 -1 osd.4 1079 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:32.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:32 smithi164 conmon[33677]: debug 2021-02-10T04:48:32.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:32 smithi164 conmon[33677]: 2021-02-10T04:48:32.742+0000 7fbec19a1700 -1 osd.4 1079 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:32.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:32 smithi164 conmon[33677]: 2021-02-10T04:48:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:32 smithi164 conmon[29288]: cluster 2021-02-10T04:48:31.720051+0000 osd.4 (osd. 2021-02-10T04:48:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:32 smithi164 conmon[29288]: 4) 2217 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:32 smithi164 conmon[29288]: cluster 2021-02-10T04:48:31.720063+0000 osd.4 (osd.4) 2218 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:32 smithi164 conmon[29288]: cluster 2021-02-10T04:48: 2021-02-10T04:48:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:32 smithi164 conmon[29288]: 31.720070+0000 osd.4 (osd.4) 2219 : cluster 2021-02-10T04:48:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:32 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:32 smithi164 conmon[29288]: cluster 2021-02-10T04:48:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:32 smithi164 conmon[29288]: 2021-02-10T04:48:31. 2021-02-10T04:48:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:32 smithi164 conmon[29288]: 720079+0000 osd.4 (osd.4) 2220 : cluster 2021-02-10T04:48:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:32 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:32.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:32 smithi085 conmon[29319]: cluster 2021-02-10T04:48:31.720051+0000 osd.4 ( 2021-02-10T04:48:32.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:32 smithi085 conmon[29319]: osd.4) 2217 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:32.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:32 smithi085 conmon[29319]: cluster 2021-02-10T04:48:31.720063+0000 osd.4 (osd.4) 2218 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:32.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:32 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:48:32.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:32 smithi085 conmon[29319]: 48:31.720070+0000 osd.4 (osd.4) 2219 2021-02-10T04:48:32.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:32 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:32.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:32 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:48:32.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:32 smithi085 conmon[29319]: 48:31.720079+0000 osd.4 (osd.4) 2220 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:32.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:32 smithi085 conmon[35961]: cluster 2021-02-10T04:48:32.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:32 smithi085 conmon[35961]: 2021-02-10T04:48:32.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:32 smithi085 conmon[35961]: 2021-02-10T04:48: 2021-02-10T04:48:32.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:32 smithi085 conmon[35961]: 31.720051+0000 osd.4 (osd.4) 2217 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:32.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:32 smithi085 conmon[35961]: cluster 2021-02-10T04:48:32.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:32 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:48:32.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:32 smithi085 conmon[35961]: 10T04:48:31.720063+0000 osd.4 (osd.4 2021-02-10T04:48:32.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:32 smithi085 conmon[35961]: ) 2218 : cluster [WRN] 2021-02-10T04:48:32.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:32 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:32.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:32 smithi085 conmon[35961]: cluster 2021-02-10T04:48:32.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:32 smithi085 conmon[35961]: 2021-02-10T04:48:32.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:32 smithi085 conmon[35961]: 2021 2021-02-10T04:48:32.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:32 smithi085 conmon[35961]: - 2021-02-10T04:48:32.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:32 smithi085 conmon[35961]: 02-10T04:48:31.720070+0000 2021-02-10T04:48:32.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:32 smithi085 conmon[35961]: osd.4 (osd.4) 2219 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:32.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:32 smithi085 conmon[35961]: cluster 2021-02-10T04:48:32.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:32 smithi085 conmon[35961]: 2021-02-10T04:48:32.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:32 smithi085 conmon[35961]: 2021 2021-02-10T04:48:32.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:32 smithi085 conmon[35961]: - 2021-02-10T04:48:32.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:32 smithi085 conmon[35961]: 02-10T04:48:31 2021-02-10T04:48:32.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:32 smithi085 conmon[35961]: .720079+0000 osd.4 (osd.4) 2220 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:33 smithi164 conmon[29288]: cluster 2021-02-10T04:48:32.359025+0000 mgr.y ( 2021-02-10T04:48:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:33 smithi164 conmon[29288]: mgr.14140) 1098 : cluster [DBG] pgmap v1876: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:48:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:33 smithi164 conmon[29288]: cluster 2021-02-10T04:48:32.743227+0000 osd.4 (osd.4) 2221 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:33 smithi164 conmon[29288]: cluster 2021-02-10T04:48:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:33 smithi164 conmon[29288]: 2021-02-10T04:48:32.743237+0000 osd.4 (osd.4) 2222 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:33 smithi164 conmon[29288]: cluster 2021-02-10T04:48 2021-02-10T04:48:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:33 smithi164 conmon[29288]: :32.743244+0000 osd.4 (osd.4) 2223 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:33 smithi164 conmon[29288]: 2021-02-10T04:48:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:33 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:48:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:33 smithi164 conmon[29288]: -02-10T04:48:32.743250+0000 osd.4 (osd.4) 2224 2021-02-10T04:48:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:33 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:33 smithi164 conmon[29288]: cluster 2021-02-10T04:48:33. 2021-02-10T04:48:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:33 smithi164 conmon[29288]: 424025+0000 mon.a (mon.0) 2164 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 247 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:48:33.895 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:33 smithi164 conmon[33677]: debug 2021-02-10T04:48:33.730+0000 7fbec19a1700 -1 osd.4 1079 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:33.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[29319]: cluster 2021-02-10T04:48:33.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[29319]: 2021-02-10T04:48:33.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[29319]: 2021 2021-02-10T04:48:33.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[29319]: - 2021-02-10T04:48:33.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[29319]: 02 2021-02-10T04:48:33.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[29319]: - 2021-02-10T04:48:33.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[29319]: 10 2021-02-10T04:48:33.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[29319]: T04:48:32 2021-02-10T04:48:33.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[29319]: .359025+0000 mgr.y ( 2021-02-10T04:48:33.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[29319]: mgr.14140) 1098 : cluster [DBG] pgmap v1876: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:48:33.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[29319]: cluster 2021-02-10T04:48:33.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[29319]: 2021-02-10T04:48:33.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[29319]: 2021 2021-02-10T04:48:33.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[29319]: -02-10T04:48:32.743227 2021-02-10T04:48:33.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2221 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:33.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[29319]: cluster 2021-02-10T04:48:33.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[29319]: 2021-02-10T04:48:32.743237+0000 osd.4 (osd.4) 2222 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:33.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[29319]: cluster 2021-02-10T04:48: 2021-02-10T04:48:33.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[29319]: 32.743244+0000 osd.4 (osd.4) 2223 : cluster 2021-02-10T04:48:33.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:33.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[29319]: cluster 2021-02-10T04:48:32.743250+0000 osd.4 2021-02-10T04:48:33.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[29319]: (osd.4) 2224 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:33.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:48:33.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[29319]: T04:48:33.424025+0000 mon.a (mon. 2021-02-10T04:48:33.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[29319]: 0) 2164 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 247 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:48:33.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[35961]: cluster 2021-02-10T04:48:32.359025+0000 2021-02-10T04:48:33.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[35961]: mgr.y (mgr.14140) 1098 : cluster [DBG] pgmap v1876: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:48:33.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:48:33.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[35961]: 10T04:48:32.743227+0000 osd.4 (osd.4) 2221 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:33.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:48:33.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[35961]: -02-10T04:48:32.743237 2021-02-10T04:48:33.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2222 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:33.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[35961]: cluster 2021-02-10T04:48:32.743244+0000 osd.4 (osd.4) 2223 : cluster 2021-02-10T04:48:33.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:33.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[35961]: cluster 2021-02-10T04:48:32.743250+0000 osd.4 (osd 2021-02-10T04:48:33.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[35961]: .4) 2224 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:33.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[35961]: cluster 2021-02-10T04:48:33.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:33 smithi085 conmon[35961]: 2021-02-10T04:48:33.424025+0000 mon.a (mon.0) 2164 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 247 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:48:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:34 smithi164 conmon[29288]: cluster 2021-02-10T04:48:33.730778 2021-02-10T04:48:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:34 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2225 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:34 smithi164 conmon[29288]: cluster 2021-02-10T04:48:33.730789+0000 osd.4 (osd 2021-02-10T04:48:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:34 smithi164 conmon[29288]: .4) 2226 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:34 smithi164 conmon[29288]: cluster 2021-02-10T04:48:33 2021-02-10T04:48:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:34 smithi164 conmon[29288]: .730798+0000 osd.4 (osd.4) 2227 : cluster [WRN] 2021-02-10T04:48:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:34 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:34 smithi164 conmon[29288]: cluster 2021-02-10T04:48:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:34 smithi164 conmon[29288]: 2021-02-10T04:48:33.730804+0000 osd.4 (osd.4) 2228 2021-02-10T04:48:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:34 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:34.895 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:34 smithi164 conmon[33677]: debug 2021-02-10T04:48:34.895 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:34 smithi164 conmon[33677]: 2021-02-10T04:48:34.774+0000 7fbec19a1700 -1 osd.4 1080 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:34.895 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:34 smithi164 conmon[33677]: 2021-02-10T04:48:34.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[29319]: cluster 2021-02-10T04:48:33.730778+0000 osd.4 (osd.4) 2021-02-10T04:48:34.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[29319]: 2225 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:34.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[29319]: cluster 2021-02-10T04:48:33.730789+0000 osd.4 (osd.4) 2226 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:34.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[29319]: cluster 2021-02-10T04:48:33.730798+0000 osd.4 (osd.4) 2227 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:34.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[29319]: 2021-02-10T04:48:34.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[29319]: cluster 2021-02-10T04:48:34.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[29319]: 2021-02-10T04:48:33.730804+0000 osd.4 (osd.4) 2228 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:34.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: cluster 2021-02-10T04:48:34.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 2021-02-10T04:48:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 2021 2021-02-10T04:48:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: - 2021-02-10T04:48:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 02 2021-02-10T04:48:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: - 2021-02-10T04:48:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 10 2021-02-10T04:48:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: T 2021-02-10T04:48:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 04 2021-02-10T04:48:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: : 2021-02-10T04:48:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 48 2021-02-10T04:48:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: : 2021-02-10T04:48:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 33 2021-02-10T04:48:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: . 2021-02-10T04:48:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 730778 2021-02-10T04:48:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: +0000 2021-02-10T04:48:34.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 2021-02-10T04:48:34.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: osd.4 2021-02-10T04:48:34.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: ( 2021-02-10T04:48:34.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: osd 2021-02-10T04:48:34.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: . 2021-02-10T04:48:34.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 4 2021-02-10T04:48:34.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: ) 2021-02-10T04:48:34.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 2225 2021-02-10T04:48:34.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: : 2021-02-10T04:48:34.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: cluster 2021-02-10T04:48:34.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 2021-02-10T04:48:34.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: [WRN] 2021-02-10T04:48:34.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 2021-02-10T04:48:34.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:34.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 2021-02-10T04:48:34.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: cluster 2021-02-10T04:48:34.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 2021-02-10T04:48:34.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 2021 2021-02-10T04:48:34.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: - 2021-02-10T04:48:34.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 02 2021-02-10T04:48:34.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: - 2021-02-10T04:48:34.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 10 2021-02-10T04:48:34.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: T 2021-02-10T04:48:34.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 04 2021-02-10T04:48:34.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: : 2021-02-10T04:48:34.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 48 2021-02-10T04:48:34.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: : 2021-02-10T04:48:34.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 33 2021-02-10T04:48:34.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: . 2021-02-10T04:48:34.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 730789 2021-02-10T04:48:34.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: +0000 2021-02-10T04:48:34.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 2021-02-10T04:48:34.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: osd.4 2021-02-10T04:48:34.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: ( 2021-02-10T04:48:34.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: osd 2021-02-10T04:48:34.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: . 2021-02-10T04:48:34.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 4 2021-02-10T04:48:34.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: ) 2021-02-10T04:48:34.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 2226 2021-02-10T04:48:34.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: : 2021-02-10T04:48:34.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: cluster 2021-02-10T04:48:34.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 2021-02-10T04:48:34.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: [WRN] 2021-02-10T04:48:34.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 2021-02-10T04:48:34.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:34.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 2021-02-10T04:48:34.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: cluster 2021-02-10T04:48:34.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 2021-02-10T04:48:34.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 2021 2021-02-10T04:48:34.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: - 2021-02-10T04:48:34.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 02 2021-02-10T04:48:34.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: - 2021-02-10T04:48:34.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 10 2021-02-10T04:48:34.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: T 2021-02-10T04:48:34.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 04 2021-02-10T04:48:34.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: : 2021-02-10T04:48:34.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 48 2021-02-10T04:48:34.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: : 2021-02-10T04:48:34.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 33 2021-02-10T04:48:34.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: . 2021-02-10T04:48:34.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 730798 2021-02-10T04:48:34.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: +0000 2021-02-10T04:48:34.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 2021-02-10T04:48:34.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: osd.4 2021-02-10T04:48:34.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: ( 2021-02-10T04:48:34.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: osd 2021-02-10T04:48:34.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: . 2021-02-10T04:48:34.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 4 2021-02-10T04:48:34.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: ) 2021-02-10T04:48:34.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 2227 2021-02-10T04:48:34.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: : 2021-02-10T04:48:34.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: cluster 2021-02-10T04:48:34.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 2021-02-10T04:48:34.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: [WRN] 2021-02-10T04:48:34.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 2021-02-10T04:48:34.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:34.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 2021-02-10T04:48:34.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: cluster 2021-02-10T04:48:34.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 2021-02-10T04:48:34.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 2021 2021-02-10T04:48:34.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: - 2021-02-10T04:48:34.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 02 2021-02-10T04:48:34.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: - 2021-02-10T04:48:34.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 10 2021-02-10T04:48:34.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: T 2021-02-10T04:48:34.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 04 2021-02-10T04:48:34.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: : 2021-02-10T04:48:34.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 48 2021-02-10T04:48:34.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: : 2021-02-10T04:48:34.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 33 2021-02-10T04:48:34.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: . 2021-02-10T04:48:34.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 730804 2021-02-10T04:48:34.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: +0000 2021-02-10T04:48:34.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 2021-02-10T04:48:34.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: osd.4 2021-02-10T04:48:34.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: ( 2021-02-10T04:48:34.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: osd 2021-02-10T04:48:34.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: . 2021-02-10T04:48:34.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 4 2021-02-10T04:48:34.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: ) 2021-02-10T04:48:34.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 2228 2021-02-10T04:48:34.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: : 2021-02-10T04:48:34.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: cluster 2021-02-10T04:48:34.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 2021-02-10T04:48:34.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: [WRN] 2021-02-10T04:48:34.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 2021-02-10T04:48:34.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:34.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:34 smithi085 conmon[35961]: 2021-02-10T04:48:35.135 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:48:35.322 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:48:35.356 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[45443]: debug 2021-02-10T04:48:35.074+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:48:35.370 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:48:35.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[29319]: cluster 2021-02-10T04:48:34.346807+0000 osd.3 (osd.3 2021-02-10T04:48:35.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[29319]: ) 139 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:48:35.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[29319]: cluster 2021-02-10T04:48:34.351224+0000 osd.3 (osd.3) 140 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:48:35.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[29319]: cluster 2021-02-10T04:48:34.359718+0000 mgr.y (mgr.14140) 2021-02-10T04:48:35.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[29319]: 1099 : cluster [DBG] pgmap v1877: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:48:35.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[29319]: cluster 2021-02-10T04:48:35.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[29319]: 2021-02-10T04:48:34. 2021-02-10T04:48:35.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[29319]: 559503+0000 mon.a (mon.0) 2165 : 2021-02-10T04:48:35.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[29319]: cluster [DBG] osdmap e1080: 8 total, 8 up, 7 in 2021-02-10T04:48:35.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:48:35.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[29319]: -10T04:48:34.775680+0000 osd.4 (osd.4) 2229 : 2021-02-10T04:48:35.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:35.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:48:35.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[29319]: T04:48:34.775688+0000 osd.4 (osd.4) 2230 : 2021-02-10T04:48:35.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:35.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[29319]: cluster 2021-02-10T04:48:34.775693 2021-02-10T04:48:35.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2231 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:35.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:48:35.614 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[29319]: 02-10T04:48:34.775696+0000 osd.4 (osd.4) 2232 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:35.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[35961]: cluster 2021-02-10T04:48:34.346807+0000 2021-02-10T04:48:35.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[35961]: osd.3 (osd.3) 139 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:48:35.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[35961]: cluster 2021-02-10T04:48:35.614 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[35961]: 2021-02-10T04:48:34.351224+0000 osd.3 (osd.3) 2021-02-10T04:48:35.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[35961]: 140 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:48:35.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[35961]: cluster 2021-02-10T04:48:35.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[35961]: 2021-02-10T04:48 2021-02-10T04:48:35.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[35961]: :34.359718+0000 mgr.y ( 2021-02-10T04:48:35.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[35961]: mgr.14140) 1099 : cluster 2021-02-10T04:48:35.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[35961]: [DBG] pgmap v1877: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:48:35.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[35961]: cluster 2021-02-10T04:48:35.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[35961]: 2021- 2021-02-10T04:48:35.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[35961]: 02-10T04:48:34 2021-02-10T04:48:35.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[35961]: .559503+0000 mon.a (mon. 2021-02-10T04:48:35.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[35961]: 0) 2165 : cluster [DBG] osdmap e1080: 8 total, 8 up, 7 in 2021-02-10T04:48:35.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[35961]: 2021-02-10T04:48:35.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:48:35.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[35961]: :48:34.775680+0000 osd.4 2021-02-10T04:48:35.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[35961]: (osd.4) 2229 : cluster 2021-02-10T04:48:35.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:35.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[35961]: cluster 2021-02-10T04:48:35.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[35961]: 2021-02-10T04:48:34.775688+0000 2021-02-10T04:48:35.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[35961]: osd.4 (osd.4) 2230 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:35.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[35961]: 2021-02-10T04:48:35.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[35961]: cluster 2021-02-10T04:48:35.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[35961]: 2021-02-10T04:48:34.775693+0000 osd.4 ( 2021-02-10T04:48:35.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[35961]: osd.4) 2231 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:35.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:48:35.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[35961]: -02-10T04:48:34.775696+0000 osd.4 (osd. 2021-02-10T04:48:35.616 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[35961]: 4) 2232 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:35.616 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[41540]: debug 2021-02-10T04:48:35.617 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:48:35 smithi085 conmon[41540]: 2021-02-10T04:48:35.611+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:48:35.643 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:48:35.748 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:48:35.785 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:48:35.824 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:35 smithi164 conmon[29288]: cluster 2021-02-10T04:48:34.346807 2021-02-10T04:48:35.824 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:35 smithi164 conmon[29288]: +0000 osd.3 (osd.3) 139 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:48:35.825 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:35 smithi164 conmon[29288]: cluster 2021-02-10T04:48:34.351224+0000 osd.3 (osd.3) 140 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:48:35.825 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:35 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:48:35.825 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:35 smithi164 conmon[29288]: -10T04:48:34.359718+0000 mgr.y (mgr.14140) 1099 : cluster [DBG] pgmap v1877: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:48:35.825 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:35 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:48:35.825 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:35 smithi164 conmon[29288]: :48:34.559503+0000 mon.a (mon.0) 2165 : cluster [DBG] 2021-02-10T04:48:35.825 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:35 smithi164 conmon[29288]: osdmap e1080: 8 total, 8 up, 7 in 2021-02-10T04:48:35.825 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:35 smithi164 conmon[29288]: cluster 2021-02-10T04:48: 2021-02-10T04:48:35.825 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:35 smithi164 conmon[29288]: 34.775680+0000 osd.4 (osd.4) 2229 : 2021-02-10T04:48:35.825 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:35 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:35.825 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:35 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:48:35.825 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:35 smithi164 conmon[29288]: T04:48:34.775688+0000 osd.4 (osd. 2021-02-10T04:48:35.825 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:35 smithi164 conmon[29288]: 4) 2230 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:35.825 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:35 smithi164 conmon[29288]: cluster 2021-02-10T04:48:35.825 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:35 smithi164 conmon[29288]: 2021-02-10T04:48:34.775693 2021-02-10T04:48:35.826 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:35 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2231 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:35.826 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:35 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:48:35.826 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:35 smithi164 conmon[29288]: :48:34.775696+0000 osd.4 (osd.4) 2232 : cluster 2021-02-10T04:48:35.826 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:35 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:35.826 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:35 smithi164 conmon[33677]: debug 2021-02-10T04:48:35.822+0000 7fbec19a1700 -1 osd.4 1080 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:36.143 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:48:35 smithi164 conmon[37263]: debug 2021-02-10T04:48:36.144 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:48:35 smithi164 conmon[37263]: 2021-02-10T04:48:35.868+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:48:36.144 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:48:35 smithi164 conmon[37263]: 2021-02-10T04:48:36.194 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:48:36.300 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:48:36.334 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:48:36.840 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:36 smithi164 conmon[29288]: cluster 2021-02-10T04:48:35 2021-02-10T04:48:36.841 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:36 smithi164 conmon[29288]: .823619+0000 osd.4 (osd.4) 2233 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:36.841 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:36 smithi164 conmon[29288]: cluster 2021-02-10T04:48:36.841 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:36 smithi164 conmon[29288]: 2021-02-10T04:48:35.823626+0000 osd.4 2021-02-10T04:48:36.841 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:36 smithi164 conmon[29288]: (osd.4) 2234 : cluster 2021-02-10T04:48:36.841 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:36 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:36.841 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:36 smithi164 conmon[29288]: cluster 2021-02-10T04:48:35.823632+0000 osd.4 (osd.4) 2235 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:36.842 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:36 smithi164 conmon[29288]: cluster 2021-02-10T04:48:36.842 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:36 smithi164 conmon[29288]: 2021-02-10T04:48:35.823635+0000 2021-02-10T04:48:36.842 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:36 smithi164 conmon[29288]: osd.4 (osd.4) 2236 2021-02-10T04:48:36.842 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:36 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:36.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:36 smithi085 conmon[29319]: cluster 2021-02-10T04:48:36.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:36 smithi085 conmon[29319]: 2021-02-10T04:48:36.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:36 smithi085 conmon[29319]: 2021 2021-02-10T04:48:36.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:36 smithi085 conmon[29319]: - 2021-02-10T04:48:36.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:36 smithi085 conmon[29319]: 02-10T04:48:35.823619+0000 osd.4 ( 2021-02-10T04:48:36.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:36 smithi085 conmon[29319]: osd.4) 2233 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:36.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:36 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:48:36.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:36 smithi085 conmon[29319]: -10T04:48:35.823626+0000 osd.4 (osd.4 2021-02-10T04:48:36.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:36 smithi085 conmon[29319]: ) 2234 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:36.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:36 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:48:36.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:36 smithi085 conmon[29319]: -02-10T04:48:35.823632+0000 osd.4 (osd 2021-02-10T04:48:36.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:36 smithi085 conmon[29319]: .4) 2235 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:36.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:36 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:48:36.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:36 smithi085 conmon[29319]: -02-10T04:48:35.823635+0000 2021-02-10T04:48:36.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:36 smithi085 conmon[29319]: osd.4 (osd.4) 2236 : cluster [WRN] 2021-02-10T04:48:36.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:36 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:36.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:36 smithi085 conmon[35961]: cluster 2021-02-10T04:48:35.823619+0000 osd.4 (osd.4) 2233 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:36.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:36 smithi085 conmon[35961]: cluster 2021-02-10T04:48:35.823626+0000 osd.4 (osd.4) 2234 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:36.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:36 smithi085 conmon[35961]: cluster 2021-02-10T04:48:35.823632+0000 osd.4 (osd.4) 2235 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:36.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:36 smithi085 conmon[35961]: cluster 2021-02-10T04:48:35.823635+0000 osd.4 (osd.4) 2236 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:37.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:36 smithi164 conmon[33677]: debug 2021-02-10T04:48:36.837+0000 7fbec19a1700 -1 osd.4 1080 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:37.860 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:37 smithi164 conmon[29288]: cluster 2021-02-10T04:48:36.360486+0000 mgr.y (mgr.14140) 1100 : cluster [DBG] pgmap v1879: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:48:37.860 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:37 smithi164 conmon[29288]: 2021-02-10T04:48:37.860 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:37 smithi164 conmon[29288]: cluster 2021-02-10T04:48:36.838600+0000 osd.4 (osd.4) 2237 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:37.860 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:37 smithi164 conmon[29288]: cluster 2021-02-10T04:48:36 2021-02-10T04:48:37.860 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:37 smithi164 conmon[29288]: .838609+0000 osd.4 (osd.4) 2238 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:37.861 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:37 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:48:37.861 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:37 smithi164 conmon[29288]: -10T04:48:36.838615+0000 2021-02-10T04:48:37.861 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:37 smithi164 conmon[29288]: osd.4 (osd.4) 2239 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:37.861 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:37 smithi164 conmon[29288]: 2021-02-10T04:48:37.861 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:37 smithi164 conmon[29288]: cluster 2021-02-10T04:48:37.861 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:37 smithi164 conmon[29288]: 2021-02-10T04:48:36.838622+0000 osd.4 (osd 2021-02-10T04:48:37.861 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:37 smithi164 conmon[29288]: .4) 2240 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:37.861 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:37 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:48:37.861 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:37 smithi164 conmon[29288]: 04:48:37.194108+0000 osd.0 (osd. 2021-02-10T04:48:37.861 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:37 smithi164 conmon[29288]: 0) 147 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:48:37.861 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:37 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:48:37.861 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:37 smithi164 conmon[29288]: :48:37.198365+0000 osd.0 (osd. 2021-02-10T04:48:37.861 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:37 smithi164 conmon[29288]: 0) 148 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:48:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[35961]: cluster 2021-02-10T04:48:36.360486+0000 mgr.y (mgr. 2021-02-10T04:48:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[35961]: 14140) 1100 : cluster [DBG] pgmap v1879: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:48:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[35961]: cluster 2021-02-10T04:48:36.838600+0000 osd.4 (osd 2021-02-10T04:48:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[35961]: .4) 2237 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:48:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[35961]: T04:48:36.838609+0000 osd.4 (osd.4) 2238 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[35961]: cluster 2021-02-10T04:48:36.838615+0000 osd.4 (osd 2021-02-10T04:48:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[35961]: .4) 2239 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[35961]: cluster 2021-02-10T04:48:36. 2021-02-10T04:48:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[35961]: 838622+0000 osd.4 (osd.4) 2240 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:48:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[35961]: T04:48:37.194108+0000 osd.0 (osd.0) 147 : cluster 2021-02-10T04:48:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[35961]: [DBG] purged_snaps scrub starts 2021-02-10T04:48:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[35961]: cluster 2021-02-10T04:48:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[35961]: 2021-02-10T04:48:37.198365+0000 osd.0 (osd 2021-02-10T04:48:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[35961]: .0) 148 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:48:37.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[29319]: cluster 2021-02-10T04:48:36.360486+0000 mgr.y (mgr. 2021-02-10T04:48:37.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[29319]: 14140) 1100 : cluster [DBG] pgmap v1879: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:48:37.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:48:37.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[29319]: T04:48:36.838600+0000 osd.4 (osd.4) 2237 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:37.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[29319]: cluster 2021-02-10T04:48:36 2021-02-10T04:48:37.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[29319]: .838609+0000 osd.4 (osd.4) 2238 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:37.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:48:37.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[29319]: -10T04:48:36.838615+0000 osd.4 (osd.4) 2239 2021-02-10T04:48:37.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:37.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[29319]: cluster 2021-02-10T04:48:36 2021-02-10T04:48:37.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[29319]: .838622+0000 osd.4 (osd.4) 2240 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:37.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[29319]: cluster 2021-02-10T04:48 2021-02-10T04:48:37.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[29319]: :37.194108+0000 osd.0 (osd.0) 147 : 2021-02-10T04:48:37.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[29319]: cluster [DBG] purged_snaps scrub starts 2021-02-10T04:48:37.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:48:37.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:37 smithi085 conmon[29319]: 48:37.198365+0000 osd.0 (osd.0) 148 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:48:38.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:37 smithi164 conmon[33677]: debug 2021-02-10T04:48:37.859+0000 7fbec19a1700 -1 osd.4 1081 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:38.818 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:48:38.862 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:38 smithi164 conmon[33677]: debug 2021-02-10T04:48:38.860+0000 7fbec19a1700 -1 osd.4 1081 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:38.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:38 smithi164 conmon[29288]: cluster 2021-02-10T04:48:37.579506+0000 mon.a (mon.0) 2166 : cluster [DBG] 2021-02-10T04:48:38.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:38 smithi164 conmon[29288]: osdmap e1081: 8 total, 8 up, 7 in 2021-02-10T04:48:38.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:38 smithi164 conmon[29288]: cluster 2021-02-10T04:48:37.859860+0000 osd.4 (osd.4) 2241 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:38.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:38 smithi164 conmon[29288]: cluster 2021-02-10T04:48:38.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:38 smithi164 conmon[29288]: 2021-02-10T04:48:37.859869+0000 osd.4 (osd.4) 2242 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:38.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:38 smithi164 conmon[29288]: cluster 2021-02-10T04:48:37.859878 2021-02-10T04:48:38.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:38 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2243 : cluster [WRN] 2021-02-10T04:48:38.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:38 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:38.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:38 smithi164 conmon[29288]: cluster 2021-02-10T04:48:37.859883 2021-02-10T04:48:38.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:38 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2244 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:38.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:38 smithi164 conmon[29288]: audit 2021-02-10T04:48:38.186665+0000 mon.a ( 2021-02-10T04:48:38.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:38 smithi164 conmon[29288]: mon.0) 2167 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:48:38.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:38 smithi164 conmon[29288]: audit 2021 2021-02-10T04:48:38.864 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:38 smithi164 conmon[29288]: -02-10T04:48:38.186969+0000 mon.a (mon.0) 2168 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:48:38.941 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[45443]: debug 2021-02-10T04:48:38.582+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:48:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: cluster 2021-02-10T04:48:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: 2021-02-10T04:48:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: 2021 2021-02-10T04:48:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: - 2021-02-10T04:48:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: 02 2021-02-10T04:48:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: - 2021-02-10T04:48:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: 10 2021-02-10T04:48:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: T 2021-02-10T04:48:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: 04:48:37.579506+0000 2021-02-10T04:48:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: mon.a (mon.0) 2166 : cluster [DBG] osdmap e1081: 8 total, 8 up, 7 in 2021-02-10T04:48:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:48:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: -10T04:48:37.859860+0000 osd.4 2021-02-10T04:48:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: (osd.4) 2241 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: 2021-02-10T04:48:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: cluster 2021-02-10T04:48:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: 2021-02-10T04:48:37.859869+0000 osd.4 (osd 2021-02-10T04:48:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: .4) 2242 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:48:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: -10T04:48:37.859878+0000 osd.4 (osd 2021-02-10T04:48:38.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: .4) 2243 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:38.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:48:38.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: 02-10T04:48:37.859883+0000 2021-02-10T04:48:38.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: osd.4 (osd.4) 2244 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:38.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: 2021-02-10T04:48:38.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: audit 2021-02-10T04:48:38.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: 2021-02-10T04:48:38.186665+0000 mon.a 2021-02-10T04:48:38.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: (mon.0) 2167 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:48:38.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: 2021-02-10T04:48:38.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: audit 2021-02-10T04:48:38.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: 2021-02-10T04:48:38.186969+0000 mon.a ( 2021-02-10T04:48:38.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[29319]: mon.0) 2168 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:48:38.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: cluster 2021-02-10T04:48: 2021-02-10T04:48:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: 37.579506+0000 mon.a (mon.0) 2166 : cluster [DBG] osdmap e1081: 8 total, 8 up, 7 in 2021-02-10T04:48:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: cluster 2021-02-10T04:48:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:48:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: 48:37.859860+0000 osd.4 2021-02-10T04:48:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: (osd.4) 2241 : cluster 2021-02-10T04:48:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:48:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: 02-10T04:48:37. 2021-02-10T04:48:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: 859869+0000 osd.4 (osd.4) 2021-02-10T04:48:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: 2242 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:48:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: -02-10T04:48: 2021-02-10T04:48:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: 37.859878+0000 osd.4 (osd. 2021-02-10T04:48:38.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: 4) 2243 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:38.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: 2021-02-10T04:48:38.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: cluster 2021-02-10T04:48:38.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: 2021-02-10T04:48 2021-02-10T04:48:38.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: :37.859883+0000 osd.4 ( 2021-02-10T04:48:38.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: osd.4) 2244 : cluster 2021-02-10T04:48:38.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:38.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: audit 2021 2021-02-10T04:48:38.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: -02-10T04:48:38 2021-02-10T04:48:38.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: .186665+0000 mon.a (mon. 2021-02-10T04:48:38.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: 0) 2167 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:48:38.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: 2021-02-10T04:48:38.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: audit 2021 2021-02-10T04:48:38.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: -02-10T04:48: 2021-02-10T04:48:38.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: 38.186969+0000 mon.a (mon 2021-02-10T04:48:38.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: .0) 2168 : audit [INF] 2021-02-10T04:48:38.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:38 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:48:39.144 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:48:38 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:48:39.840 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:39 smithi164 conmon[33677]: debug 2021-02-10T04:48:39.838+0000 7fbec19a1700 -1 osd.4 1081 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:39.840 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:39 smithi164 conmon[29288]: cluster 2021-02-10T04:48:38.361046+0000 mgr.y ( 2021-02-10T04:48:39.840 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:39 smithi164 conmon[29288]: mgr.14140) 1101 : cluster [DBG] pgmap v1881: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:48:39.841 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:39 smithi164 conmon[29288]: cluster 2021-02-10T04:48:38.572033+0000 mon.a (mon.0) 2169 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 252 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:48:39.841 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:39 smithi164 conmon[29288]: cluster 2021-02-10T04:48:38.861046+0000 2021-02-10T04:48:39.841 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:39 smithi164 conmon[29288]: osd.4 (osd.4) 2245 : cluster [WRN] 2021-02-10T04:48:39.841 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:39 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:39.841 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:39 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:48:39.842 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:39 smithi164 conmon[29288]: -02-10T04:48:38.861052+0000 2021-02-10T04:48:39.842 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:39 smithi164 conmon[29288]: osd.4 (osd.4) 2246 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:39.842 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:39 smithi164 conmon[29288]: cluster 2021-02-10T04:48:39.842 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:39 smithi164 conmon[29288]: 2021-02-10T04:48:38.861055+0000 2021-02-10T04:48:39.842 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:39 smithi164 conmon[29288]: osd.4 (osd.4) 2247 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:39.843 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:39 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:48:39.843 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:39 smithi164 conmon[29288]: -10T04:48:38.861059+0000 osd.4 (osd.4) 2021-02-10T04:48:39.843 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:39 smithi164 conmon[29288]: 2248 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:39.843 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:39 smithi164 conmon[29288]: audit 2021 2021-02-10T04:48:39.843 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:39 smithi164 conmon[29288]: -02-10T04:48:38.898053+0000 mgr.y 2021-02-10T04:48:39.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:39 smithi164 conmon[29288]: (mgr.14140) 1102 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:48:39.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[35961]: cluster 2021-02-10T04:48:38.361046 2021-02-10T04:48:39.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1101 : cluster [DBG] pgmap v1881: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:48:39.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[35961]: cluster 2021-02-10T04:48:38.572033+0000 2021-02-10T04:48:39.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[35961]: mon.a (mon.0) 2169 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 252 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:48:39.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:48:39.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[35961]: -02-10T04:48:38.861046+0000 2021-02-10T04:48:39.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[35961]: osd.4 (osd.4) 2245 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:39.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[35961]: 2021-02-10T04:48:39.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:48:39.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[35961]: 48:38.861052+0000 osd.4 (osd.4) 2021-02-10T04:48:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[35961]: 2246 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[35961]: cluster 2021-02-10T04:48:38.861055+0000 2021-02-10T04:48:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[35961]: osd.4 (osd.4) 2247 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[35961]: cluster 2021-02-10T04:48:38.861059+0000 osd.4 (osd.4 2021-02-10T04:48:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[35961]: ) 2248 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:48:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[35961]: 02-10T04:48:38.898053+0000 mgr.y (mgr.14140) 1102 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:48:39.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[29319]: cluster 2021-02-10T04:48:38.361046+0000 mgr.y ( 2021-02-10T04:48:39.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[29319]: mgr.14140) 1101 : cluster [DBG] pgmap v1881: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:48:39.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[29319]: cluster 2021-02-10T04:48:39.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[29319]: 2021-02-10T04:48:38 2021-02-10T04:48:39.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[29319]: .572033+0000 mon.a (mon.0) 2169 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 252 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:48:39.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[29319]: cluster 2021-02-10T04:48:39.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[29319]: 2021 2021-02-10T04:48:39.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[29319]: - 2021-02-10T04:48:39.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[29319]: 02-10T04:48:38.861046+0000 2021-02-10T04:48:39.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[29319]: osd.4 (osd.4) 2245 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:39.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[29319]: cluster 2021-02-10T04:48:38.861052+0000 osd.4 2021-02-10T04:48:39.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[29319]: (osd.4) 2246 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:39.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:48:39.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[29319]: -10T04:48:38.861055+0000 osd.4 2021-02-10T04:48:39.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[29319]: (osd.4) 2247 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:39.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[29319]: 2021-02-10T04:48:39.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[29319]: cluster 2021-02-10T04:48 2021-02-10T04:48:39.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[29319]: :38.861059+0000 osd.4 (osd.4) 2248 : cluster [WRN] 2021-02-10T04:48:39.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:39.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[29319]: audit 2021-02-10T04:48: 2021-02-10T04:48:39.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[29319]: 38.898053+0000 mgr.y (mgr.14140) 1102 : 2021-02-10T04:48:39.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:39 smithi085 conmon[29319]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:48:40.388 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-02-10T04:48:40.388 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 0 1 2021-02-10T04:48:40.672 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:48:40.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:40 smithi085 conmon[29319]: cluster 2021-02-10T04:48:39.838799+0000 osd.4 2021-02-10T04:48:40.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:40 smithi085 conmon[29319]: (osd.4) 2249 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:40.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:40 smithi085 conmon[29319]: cluster 2021-02-10T04:48:39.838805+0000 osd.4 (osd.4) 2250 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:40.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:40 smithi085 conmon[29319]: cluster 2021-02-10T04:48:39.838810+0000 osd.4 (osd.4) 2251 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:40.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:40 smithi085 conmon[29319]: cluster 2021-02-10T04:48:39.838814+0000 osd.4 (osd.4) 2252 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:40.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:40 smithi085 conmon[35961]: cluster 2021-02-10T04:48:39.838799 2021-02-10T04:48:40.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:40 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2249 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:40.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:40 smithi085 conmon[35961]: cluster 2021-02-10T04:48: 2021-02-10T04:48:40.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:40 smithi085 conmon[35961]: 39.838805+0000 osd.4 (osd.4) 2250 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:40.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:40 smithi085 conmon[35961]: cluster 2021-02-10T04:48:39.838810 2021-02-10T04:48:40.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:40 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2251 : 2021-02-10T04:48:40.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:40 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:40.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:40 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:48:40.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:40 smithi085 conmon[35961]: 48:39.838814+0000 osd.4 (osd.4) 2021-02-10T04:48:40.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:40 smithi085 conmon[35961]: 2252 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:40.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:40 smithi164 conmon[33677]: debug 2021-02-10T04:48:40.797+0000 7fbec19a1700 -1 osd.4 1081 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:40.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:40 smithi164 conmon[29288]: cluster 2021-02-10T04:48:40.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:40 smithi164 conmon[29288]: 2021-02-10T04:48:40.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:40 smithi164 conmon[29288]: 2021 2021-02-10T04:48:40.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:40 smithi164 conmon[29288]: - 2021-02-10T04:48:40.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:40 smithi164 conmon[29288]: 02 2021-02-10T04:48:40.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:40 smithi164 conmon[29288]: - 2021-02-10T04:48:40.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:40 smithi164 conmon[29288]: 10 2021-02-10T04:48:40.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:40 smithi164 conmon[29288]: T 2021-02-10T04:48:40.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:40 smithi164 conmon[29288]: 04 2021-02-10T04:48:40.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:40 smithi164 conmon[29288]: : 2021-02-10T04:48:40.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:40 smithi164 conmon[29288]: 48:39.838799+0000 osd.4 (osd 2021-02-10T04:48:40.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:40 smithi164 conmon[29288]: .4) 2249 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:40.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:40 smithi164 conmon[29288]: cluster 2021-02-10T04:48:39 2021-02-10T04:48:40.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:40 smithi164 conmon[29288]: .838805+0000 osd.4 (osd.4) 2250 : cluster [WRN] 2021-02-10T04:48:40.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:40 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:40.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:40 smithi164 conmon[29288]: cluster 2021-02-10T04:48: 2021-02-10T04:48:40.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:40 smithi164 conmon[29288]: 39.838810+0000 osd.4 (osd.4) 2251 : cluster 2021-02-10T04:48:40.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:40 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:40.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:40 smithi164 conmon[29288]: cluster 2021-02-10T04:48:39. 2021-02-10T04:48:40.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:40 smithi164 conmon[29288]: 838814+0000 osd.4 (osd.4) 2252 : cluster [WRN] 2021-02-10T04:48:40.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:40 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:41.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:41 smithi164 conmon[33677]: debug 2021-02-10T04:48:41.803+0000 7fbec19a1700 -1 osd.4 1081 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:41.937 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:41 smithi164 conmon[29288]: cluster 2021-02-10T04:48:40.361843+0000 mgr.y (mgr 2021-02-10T04:48:41.937 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:41 smithi164 conmon[29288]: .14140) 1103 : cluster [DBG] pgmap v1882: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:48:41.938 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:41 smithi164 conmon[29288]: cluster 2021-02-10T04:48:40.798630+0000 osd.4 (osd.4) 2253 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:41.938 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:41 smithi164 conmon[29288]: cluster 2021-02-10T04:48:40.798638+0000 osd.4 2021-02-10T04:48:41.938 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:41 smithi164 conmon[29288]: (osd.4) 2254 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:41.938 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:41 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:48:41.938 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:41 smithi164 conmon[29288]: 10T04:48:40.798642+0000 osd.4 (osd 2021-02-10T04:48:41.938 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:41 smithi164 conmon[29288]: .4) 2255 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:41.939 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:41 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:48:41.939 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:41 smithi164 conmon[29288]: 02-10T04:48:40.798646+0000 osd.4 ( 2021-02-10T04:48:41.939 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:41 smithi164 conmon[29288]: osd.4) 2256 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[29319]: cluster 2021-02-10T04:48:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[29319]: 2021-02-10T04:48: 2021-02-10T04:48:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[29319]: 40.361843+0000 mgr.y (mgr.14140) 1103 : cluster [DBG] pgmap v1882: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:48:41.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[29319]: cluster 2021-02-10T04:48:41.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[29319]: 2021-02-10T04:48:40.798630+0000 osd.4 (osd.4) 2253 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:41.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:48:41.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[29319]: 04:48:40.798638+0000 osd.4 (osd.4) 2254 2021-02-10T04:48:41.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:41.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:48:41.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[29319]: 10T04:48:40.798642+0000 2021-02-10T04:48:41.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[29319]: osd.4 (osd.4) 2255 : cluster [WRN] 2021-02-10T04:48:41.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:41.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:48:41.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[29319]: 04:48:40.798646+0000 osd.4 (osd.4) 2256 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[35961]: cluster 2021-02-10T04:48:41.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[35961]: 2021-02-10T04:48:41.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:48:41.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[35961]: :48:40.361843+0000 mgr.y (mgr.14140) 1103 : cluster [DBG] pgmap v1882: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:48:41.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:48:41.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[35961]: -02-10T04:48:40.798630+0000 osd.4 ( 2021-02-10T04:48:41.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[35961]: osd.4) 2253 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:41.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[35961]: cluster 2021-02-10T04:48:41.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[35961]: 2021-02-10T04:48:41.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[35961]: 2021-02-10T04:48:40 2021-02-10T04:48:41.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[35961]: .798638+0000 osd.4 (osd.4) 2254 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:41.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:48:41.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[35961]: 10T04:48:40.798642+0000 2021-02-10T04:48:41.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[35961]: osd.4 (osd.4) 2255 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:41.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:48:41.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[35961]: 02-10T04:48:40.798646+0000 osd.4 ( 2021-02-10T04:48:41.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[35961]: osd.4) 2256 : cluster 2021-02-10T04:48:41.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:41 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:42 smithi164 conmon[29288]: cluster 2021-02-10T04:48:41.804608 2021-02-10T04:48:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:42 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2257 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:42 smithi164 conmon[29288]: cluster 2021-02-10T04:48:41.804616+0000 osd.4 (osd.4 2021-02-10T04:48:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:42 smithi164 conmon[29288]: ) 2258 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:42 smithi164 conmon[29288]: cluster 2021-02-10T04:48:41.804622 2021-02-10T04:48:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:42 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2259 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:42 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:48:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:42 smithi164 conmon[29288]: :48:41.804625+0000 osd.4 (osd.4) 2260 : cluster [WRN] 2021-02-10T04:48:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:42 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:42.895 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:42 smithi164 conmon[33677]: debug 2021-02-10T04:48:42.895 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:42 smithi164 conmon[33677]: 2021-02-10T04:48:42.764+0000 7fbec19a1700 -1 osd.4 1081 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:42.895 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:42 smithi164 conmon[33677]: 2021-02-10T04:48:42.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:42 smithi085 conmon[35961]: cluster 2021-02-10T04:48:41.804608+0000 2021-02-10T04:48:42.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:42 smithi085 conmon[35961]: osd.4 (osd.4) 2257 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:42.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:42 smithi085 conmon[35961]: cluster 2021-02-10T04:48:41.804616+0000 2021-02-10T04:48:42.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:42 smithi085 conmon[35961]: osd.4 (osd.4) 2258 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:42.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:42 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:48:42.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:42 smithi085 conmon[35961]: -10T04:48:41.804622+0000 osd.4 (osd 2021-02-10T04:48:42.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:42 smithi085 conmon[35961]: .4) 2259 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:42.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:42 smithi085 conmon[35961]: 2021-02-10T04:48:42.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:42 smithi085 conmon[35961]: cluster 2021-02-10T04:48:42.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:42 smithi085 conmon[35961]: 2021-02-10T04:48:41.804625+0000 osd.4 2021-02-10T04:48:42.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:42 smithi085 conmon[35961]: (osd.4) 2260 : cluster 2021-02-10T04:48:42.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:42 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:42 smithi085 conmon[29319]: cluster 2021-02-10T04:48:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:42 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:48:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:42 smithi085 conmon[29319]: 48:41.804608+0000 osd.4 (osd.4) 2257 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:42 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:48:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:42 smithi085 conmon[29319]: 02-10T04:48:41.804616 2021-02-10T04:48:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:42 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2258 : 2021-02-10T04:48:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:42 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:42 smithi085 conmon[29319]: cluster 2021-02-10T04:48:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:42 smithi085 conmon[29319]: 2021-02-10T04:48: 2021-02-10T04:48:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:42 smithi085 conmon[29319]: 41.804622+0000 osd.4 (osd 2021-02-10T04:48:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:42 smithi085 conmon[29319]: .4) 2259 : cluster [WRN] 2021-02-10T04:48:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:42 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:42.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:42 smithi085 conmon[29319]: cluster 2021-02-10T04:48:42.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:42 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:48:42.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:42 smithi085 conmon[29319]: 48:41.804625+0000 osd.4 ( 2021-02-10T04:48:42.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:42 smithi085 conmon[29319]: osd.4) 2260 : cluster [WRN] 2021-02-10T04:48:42.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:42 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:43.192 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:48:43.226 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:48:43.596 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:48:43 smithi164 conmon[37263]: debug 2021-02-10T04:48:43.316+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:48:43.739 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:48:43.846 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:48:43.876 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:43 smithi164 conmon[29288]: cluster 2021-02-10T04:48:42.362308+0000 2021-02-10T04:48:43.876 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:43 smithi164 conmon[29288]: mgr.y (mgr.14140) 1104 : cluster [DBG] pgmap v1883: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:48:43.876 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:43 smithi164 conmon[29288]: cluster 2021-02-10T04:48:43.876 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:43 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:48:43.876 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:43 smithi164 conmon[29288]: 48:42.765159+0000 osd.4 (osd.4) 2261 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:43.876 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:43 smithi164 conmon[29288]: 2021-02-10T04:48:43.876 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:43 smithi164 conmon[29288]: cluster 2021-02-10T04:48:43.876 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:43 smithi164 conmon[29288]: 2021-02-10T04:48:42. 2021-02-10T04:48:43.876 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:43 smithi164 conmon[29288]: 765167+0000 osd.4 (osd.4) 2262 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:43.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:43 smithi164 conmon[29288]: cluster 2021-02-10T04:48:42.765173+0000 2021-02-10T04:48:43.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:43 smithi164 conmon[29288]: osd.4 (osd.4) 2263 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:43.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:43 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:48:43.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:43 smithi164 conmon[29288]: 10T04:48:42.765178+0000 osd.4 (osd.4) 2021-02-10T04:48:43.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:43 smithi164 conmon[29288]: 2264 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:43.877 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:43 smithi164 conmon[33677]: debug 2021-02-10T04:48:43.742+0000 7fbec19a1700 -1 osd.4 1081 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:43.877 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:48:43.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:43 smithi085 conmon[35961]: cluster 2021-02-10T04:48:42.362308+0000 2021-02-10T04:48:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:43 smithi085 conmon[35961]: mgr.y (mgr.14140) 1104 : cluster [DBG] pgmap v1883: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:48:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:43 smithi085 conmon[35961]: cluster 2021-02-10T04:48 2021-02-10T04:48:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:43 smithi085 conmon[35961]: :42.765159+0000 osd.4 (osd.4) 2261 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:43 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:48:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:43 smithi085 conmon[35961]: 48:42.765167+0000 osd.4 (osd.4) 2262 2021-02-10T04:48:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:43 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:43 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:48:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:43 smithi085 conmon[35961]: 48:42.765173+0000 osd.4 (osd. 2021-02-10T04:48:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:43 smithi085 conmon[35961]: 4) 2263 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:43 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:48:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:43 smithi085 conmon[35961]: 10T04:48:42.765178+0000 osd.4 (osd 2021-02-10T04:48:43.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:43 smithi085 conmon[35961]: .4) 2264 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:43.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:43 smithi085 conmon[29319]: cluster 2021-02-10T04:48:42.362308+0000 mgr.y ( 2021-02-10T04:48:43.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:43 smithi085 conmon[29319]: mgr.14140) 1104 : cluster [DBG] pgmap v1883: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:48:43.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:43 smithi085 conmon[29319]: cluster 2021-02-10T04:48:42.765159+0000 osd.4 (osd.4) 2261 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:43.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:43 smithi085 conmon[29319]: cluster 2021-02-10T04:48:42.765167+0000 osd.4 (osd.4) 2262 2021-02-10T04:48:43.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:43 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:43.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:43 smithi085 conmon[29319]: cluster 2021-02-10T04:48:43.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:43 smithi085 conmon[29319]: 2021- 2021-02-10T04:48:43.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:43 smithi085 conmon[29319]: 02-10T04:48:42.765173+0000 2021-02-10T04:48:43.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:43 smithi085 conmon[29319]: osd.4 (osd.4) 2263 : cluster [WRN] 2021-02-10T04:48:43.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:43 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:43.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:43 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:48:43.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:43 smithi085 conmon[29319]: -10T04:48:42.765178+0000 osd.4 (osd.4) 2021-02-10T04:48:43.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:43 smithi085 conmon[29319]: 2264 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:44.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:43 smithi164 conmon[33677]: debug 2021-02-10T04:48:43.939+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:48:44.222 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:48:44.328 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:48:44.362 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:48:44.602 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:48:44 smithi164 conmon[40892]: debug 2021-02-10T04:48:44.440+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:48:44.845 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:48:44.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:44 smithi164 conmon[33677]: debug 2021-02-10T04:48:44.780+0000 7fbec19a1700 -1 osd.4 1081 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:44.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:44 smithi164 conmon[29288]: cluster 2021-02-10T04:48:43.743635+0000 osd.4 (osd 2021-02-10T04:48:44.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:44 smithi164 conmon[29288]: .4) 2265 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:44.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:44 smithi164 conmon[29288]: cluster 2021-02-10T04:48:43.743646+0000 osd.4 (osd.4) 2266 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:44.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:44 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:48:44.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:44 smithi164 conmon[29288]: 48:43.743653+0000 osd.4 (osd.4) 2267 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:44.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:44 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:48:44.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:44 smithi164 conmon[29288]: 04:48:43.743659+0000 osd.4 (osd 2021-02-10T04:48:44.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:44 smithi164 conmon[29288]: .4) 2268 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:44.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:44 smithi085 conmon[29319]: cluster 2021-02-10T04:48: 2021-02-10T04:48:44.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:44 smithi085 conmon[29319]: 43.743635+0000 osd.4 (osd.4) 2265 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:44.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:44 smithi085 conmon[29319]: cluster 2021-02-10T04:48:43.743646+0000 2021-02-10T04:48:44.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:44 smithi085 conmon[29319]: osd.4 (osd.4) 2266 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:44.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:44 smithi085 conmon[29319]: cluster 2021-02-10T04:48:43.743653+0000 osd.4 (osd.4) 2021-02-10T04:48:44.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:44 smithi085 conmon[29319]: 2267 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:44.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:44 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:48:44.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:44 smithi085 conmon[29319]: 48:43.743659+0000 osd.4 (osd.4) 2268 : cluster 2021-02-10T04:48:44.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:44 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:44.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:44 smithi085 conmon[35961]: cluster 2021-02-10T04:48:43.743635+0000 2021-02-10T04:48:44.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:44 smithi085 conmon[35961]: osd.4 (osd.4) 2265 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:44.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:44 smithi085 conmon[35961]: cluster 2021-02-10T04:48: 2021-02-10T04:48:44.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:44 smithi085 conmon[35961]: 43.743646+0000 osd.4 (osd. 2021-02-10T04:48:44.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:44 smithi085 conmon[35961]: 4) 2266 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:44.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:44 smithi085 conmon[35961]: cluster 2021-02-10T04:48:44.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:44 smithi085 conmon[35961]: 2021-02-10T04:48:43.743653+0000 2021-02-10T04:48:44.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:44 smithi085 conmon[35961]: osd.4 (osd.4) 2267 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:44.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:44 smithi085 conmon[35961]: 2021-02-10T04:48:44.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:44 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:48:44.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:44 smithi085 conmon[35961]: T04:48:43.743659+0000 2021-02-10T04:48:44.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:44 smithi085 conmon[35961]: osd.4 (osd.4) 2268 : 2021-02-10T04:48:44.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:44 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:44.950 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.6 2021-02-10T04:48:44.977 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:48:45.293 INFO:teuthology.orchestra.run.smithi164.stdout:1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T04:48:45.299 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:48:45 smithi164 conmon[40892]: debug 2021-02-10T04:48:45.036+0000 7f30545c5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:48:45.400 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:48:45.605 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:45 smithi164 conmon[29288]: cluster 2021-02-10T04:48:44.362932+0000 2021-02-10T04:48:45.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:45 smithi164 conmon[29288]: mgr.y (mgr.14140) 1105 : cluster [DBG] pgmap v1884: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:48:45.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:45 smithi164 conmon[29288]: cluster 2021-02-10T04:48:44.781130+0000 osd.4 (osd.4) 2269 2021-02-10T04:48:45.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:45 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:45.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:45 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:48:45.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:45 smithi164 conmon[29288]: 02-10T04:48:44.781136+0000 osd.4 (osd.4) 2270 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:45.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:45 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:48:45.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:45 smithi164 conmon[29288]: 02-10T04:48:44.781142+0000 2021-02-10T04:48:45.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:45 smithi164 conmon[29288]: osd.4 (osd.4) 2271 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:45.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:45 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:48:45.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:45 smithi164 conmon[29288]: 02-10T04:48:44.781145+0000 2021-02-10T04:48:45.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:45 smithi164 conmon[29288]: osd.4 (osd.4) 2272 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:45.607 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:45 smithi164 conmon[29288]: 2021-02-10T04:48:45.607 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:45 smithi164 conmon[29288]: cluster 2021-02-10T04:48:45.607 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:45 smithi164 conmon[29288]: 2021-02-10T04:48:44.786668+0000 2021-02-10T04:48:45.607 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:45 smithi164 conmon[29288]: osd.7 (osd.7) 78 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:48:45.607 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:45 smithi164 conmon[29288]: cluster 2021-02-10T04:48:45.607 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:45 smithi164 conmon[29288]: 2021-02 2021-02-10T04:48:45.607 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:45 smithi164 conmon[29288]: -10T04:48:44.789208+0000 osd.7 (osd.7) 79 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:48:45.632 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:48:45.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:45 smithi164 conmon[33677]: debug 2021-02-10T04:48:45.791+0000 7fbec19a1700 -1 osd.4 1082 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[29319]: cluster 2021-02-10T04:48:44.362932+0000 mgr.y ( 2021-02-10T04:48:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[29319]: mgr.14140) 1105 : cluster [DBG] pgmap v1884: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:48:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[29319]: cluster 2021-02-10T04:48:44.781130+0000 osd.4 (osd.4) 2269 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[29319]: cluster 2021-02-10T04:48:44.781136+0000 osd.4 (osd.4 2021-02-10T04:48:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[29319]: ) 2270 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:48:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[29319]: 48:44.781142+0000 osd.4 (osd.4) 2271 : cluster 2021-02-10T04:48:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:45.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:48:45.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[29319]: 02-10T04:48:44.781145+0000 2021-02-10T04:48:45.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[29319]: osd.4 (osd.4) 2272 : cluster 2021-02-10T04:48:45.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:45.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:48:45.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[29319]: -10T04:48:44.786668+0000 2021-02-10T04:48:45.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[29319]: osd.7 (osd.7) 78 : 2021-02-10T04:48:45.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[29319]: cluster [DBG] purged_snaps scrub starts 2021-02-10T04:48:45.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:48:45.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[29319]: 10T04:48:44.789208+0000 osd.7 (osd.7) 2021-02-10T04:48:45.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[29319]: 79 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:48:45.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[35961]: cluster 2021-02-10T04:48:44.362932+0000 mgr.y (mgr.14140) 1105 : cluster 2021-02-10T04:48:45.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[35961]: [DBG] pgmap v1884: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:48:45.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[35961]: cluster 2021-02-10T04:48:44.781130+0000 osd.4 (osd.4) 2021-02-10T04:48:45.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[35961]: 2269 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:45.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:48:45.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[35961]: 48:44.781136+0000 osd.4 (osd.4) 2270 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:45.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:48:45.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[35961]: -10T04:48 2021-02-10T04:48:45.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[35961]: :44.781142+0000 osd.4 (osd.4) 2021-02-10T04:48:45.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[35961]: 2271 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:45.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:48:45.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[35961]: 10T04:48: 2021-02-10T04:48:45.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[35961]: 44.781145+0000 osd.4 (osd.4) 2272 2021-02-10T04:48:45.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:45.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:48:45.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[35961]: -10T04:48:44.786668+0000 osd.7 (osd.7 2021-02-10T04:48:45.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[35961]: ) 78 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:48:45.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:48:45.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[35961]: :48:44.789208+0000 osd.7 (osd.7) 79 : cluster 2021-02-10T04:48:45.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:45 smithi085 conmon[35961]: [DBG] purged_snaps scrub ok 2021-02-10T04:48:46.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:46 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:48:46.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:46 smithi164 conmon[29288]: :48:45.615888+0000 mon.a (mon.0) 2170 : cluster [DBG] osdmap e1082: 8 total, 8 up, 7 in 2021-02-10T04:48:46.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:46 smithi164 conmon[29288]: cluster 2021-02-10T04:48:45. 2021-02-10T04:48:46.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:46 smithi164 conmon[29288]: 792524+0000 osd.4 (osd.4) 2273 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:46.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:46 smithi164 conmon[29288]: cluster 2021-02-10T04:48: 2021-02-10T04:48:46.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:46 smithi164 conmon[29288]: 45.792535+0000 osd.4 (osd.4) 2274 : cluster 2021-02-10T04:48:46.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:46 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:46.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:46 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:48:46.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:46 smithi164 conmon[29288]: :48:45.792541+0000 osd.4 (osd.4) 2021-02-10T04:48:46.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:46 smithi164 conmon[29288]: 2275 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:46.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:46 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:48:46.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:46 smithi164 conmon[29288]: :48:45.792547+0000 osd.4 (osd.4 2021-02-10T04:48:46.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:46 smithi164 conmon[29288]: ) 2276 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:46.895 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:46 smithi164 conmon[33677]: debug 2021-02-10T04:48:46.763+0000 7fbec19a1700 -1 osd.4 1082 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:46.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:46 smithi085 conmon[29319]: cluster 2021-02-10T04:48:45.615888+0000 mon.a (mon.0) 2170 2021-02-10T04:48:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:46 smithi085 conmon[29319]: : cluster [DBG] osdmap e1082: 8 total, 8 up, 7 in 2021-02-10T04:48:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:46 smithi085 conmon[29319]: cluster 2021-02-10T04:48:45.792524+0000 osd.4 (osd.4) 2273 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:46 smithi085 conmon[29319]: cluster 2021-02-10T04:48:45.792535+0000 osd.4 (osd.4) 2274 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:46 smithi085 conmon[29319]: cluster 2021-02-10T04:48:45.792541+0000 osd.4 (osd.4) 2275 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:46 smithi085 conmon[29319]: cluster 2021-02-10T04:48:45.792547+0000 osd.4 (osd.4) 2276 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:46.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:46 smithi085 conmon[35961]: cluster 2021-02-10T04:48:45. 2021-02-10T04:48:46.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:46 smithi085 conmon[35961]: 615888+0000 mon.a (mon.0) 2170 : cluster [DBG] osdmap e1082: 8 total, 8 up, 7 in 2021-02-10T04:48:46.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:46 smithi085 conmon[35961]: cluster 2021-02-10T04:48:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:46 smithi085 conmon[35961]: 2021-02-10T04:48:45.792524+0000 osd.4 (osd.4) 2273 : 2021-02-10T04:48:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:46 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:46 smithi085 conmon[35961]: cluster 2021-02-10T04:48:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:46 smithi085 conmon[35961]: 2021-02-10T04:48:45.792535+0000 osd.4 2021-02-10T04:48:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:46 smithi085 conmon[35961]: (osd.4) 2274 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:46 smithi085 conmon[35961]: cluster 2021-02-10T04:48:45. 2021-02-10T04:48:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:46 smithi085 conmon[35961]: 792541+0000 osd.4 (osd.4) 2275 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:46 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:48:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:46 smithi085 conmon[35961]: -10T04:48:45.792547+0000 2021-02-10T04:48:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:46 smithi085 conmon[35961]: osd.4 (osd.4) 2276 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:46 smithi085 conmon[35961]: 2021-02-10T04:48:47.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:47 smithi164 conmon[33677]: debug 2021-02-10T04:48:47.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:47 smithi164 conmon[33677]: 2021-02-10T04:48:47.751+0000 7fbec19a1700 -1 osd.4 1082 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:47.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:47 smithi164 conmon[33677]: 2021-02-10T04:48:47.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:47 smithi164 conmon[29288]: cluster 2021-02-10T04:48:46.363340 2021-02-10T04:48:47.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:47 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1106 : cluster [DBG] pgmap v1886: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:48:47.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:47 smithi164 conmon[29288]: cluster 2021-02-10T04:48:46.764408+0000 osd.4 (osd.4 2021-02-10T04:48:47.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:47 smithi164 conmon[29288]: ) 2277 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:47.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:47 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:48:47.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:47 smithi164 conmon[29288]: 10T04:48:46.764414+0000 osd.4 (osd.4) 2278 : cluster [WRN] 2021-02-10T04:48:47.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:47 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:47.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:47 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:48:47.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:47 smithi164 conmon[29288]: 04:48:46.764419+0000 osd.4 (osd.4 2021-02-10T04:48:47.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:47 smithi164 conmon[29288]: ) 2279 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:47.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:47 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:48:47.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:47 smithi164 conmon[29288]: T04:48:46.764430+0000 osd.4 (osd.4 2021-02-10T04:48:47.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:47 smithi164 conmon[29288]: ) 2280 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:47.941 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[45443]: debug 2021-02-10T04:48:47.701+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:48:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[29319]: cluster 2021-02-10T04:48:46.363340+0000 2021-02-10T04:48:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[29319]: mgr.y (mgr.14140) 1106 : cluster [DBG] pgmap v1886: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:48:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:48:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[29319]: 48:46.764408+0000 osd.4 (osd.4) 2277 : cluster [WRN] 2021-02-10T04:48:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[29319]: cluster 2021-02-10T04:48:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[29319]: 2021-02-10T04:48: 2021-02-10T04:48:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[29319]: 46.764414+0000 osd.4 (osd 2021-02-10T04:48:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[29319]: .4) 2278 : cluster [WRN] 2021-02-10T04:48:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:48:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[29319]: -02-10T04:48: 2021-02-10T04:48:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[29319]: 46.764419+0000 osd.4 2021-02-10T04:48:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[29319]: (osd.4) 2279 : cluster 2021-02-10T04:48:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[29319]: 2021-02-10T04:48:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:48:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[29319]: -02-10T04:48:46 2021-02-10T04:48:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[29319]: .764430+0000 osd.4 (osd.4 2021-02-10T04:48:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[29319]: ) 2280 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:47.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[29319]: 2021-02-10T04:48:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[35961]: cluster 2021-02-10T04:48 2021-02-10T04:48:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[35961]: :46.363340+0000 mgr.y (mgr.14140) 1106 : cluster [DBG] pgmap v1886: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:48:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:48:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[35961]: 02-10T04:48: 2021-02-10T04:48:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[35961]: 46.764408+0000 osd.4 ( 2021-02-10T04:48:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[35961]: osd.4) 2277 : cluster 2021-02-10T04:48:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:48:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[35961]: T04:48:46.764414 2021-02-10T04:48:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2278 2021-02-10T04:48:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:48:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[35961]: 02-10T04:48:46.764419+0000 2021-02-10T04:48:47.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[35961]: osd.4 (osd.4) 2279 : 2021-02-10T04:48:47.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:47.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:48:47.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[35961]: 02-10T04:48:46.764430+0000 osd.4 ( 2021-02-10T04:48:47.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[35961]: osd.4) 2280 : cluster [WRN] 2021-02-10T04:48:47.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:47 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:47.953 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:48:48.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[33677]: debug 2021-02-10T04:48:48.769+0000 7fbec19a1700 -1 osd.4 1082 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: cluster 2021-02-10T04:48:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: 2021-02-10T04:48:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:48:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: 48:47.752396+0000 osd.4 (osd.4) 2281 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: 2021-02-10T04:48:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: cluster 2021-02-10T04:48:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: 2021-02-10T04:48:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: 2021 2021-02-10T04:48:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: - 2021-02-10T04:48:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: 02 2021-02-10T04:48:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: - 2021-02-10T04:48:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: 10 2021-02-10T04:48:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: T 2021-02-10T04:48:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: 04 2021-02-10T04:48:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: : 2021-02-10T04:48:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: 48 2021-02-10T04:48:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: : 2021-02-10T04:48:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: 47 2021-02-10T04:48:48.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: . 2021-02-10T04:48:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: 752401 2021-02-10T04:48:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2282 : cluster 2021-02-10T04:48:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: cluster 2021-02-10T04:48:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: 2021-02-10T04:48:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: 2021 2021-02-10T04:48:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: - 2021-02-10T04:48:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: 02 2021-02-10T04:48:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: - 2021-02-10T04:48:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: 10 2021-02-10T04:48:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: T 2021-02-10T04:48:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: 04 2021-02-10T04:48:48.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: : 2021-02-10T04:48:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: 48 2021-02-10T04:48:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: : 2021-02-10T04:48:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: 47 2021-02-10T04:48:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: . 2021-02-10T04:48:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: 752405 2021-02-10T04:48:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: +0000 2021-02-10T04:48:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: 2021-02-10T04:48:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: osd.4 2021-02-10T04:48:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: ( 2021-02-10T04:48:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: osd 2021-02-10T04:48:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: . 2021-02-10T04:48:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: 4 2021-02-10T04:48:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: ) 2021-02-10T04:48:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: 2283 2021-02-10T04:48:48.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:48.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:48:48.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: -10T04:48:47.752409+0000 osd.4 (osd.4) 2284 : cluster 2021-02-10T04:48:48.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:48.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:48:48.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: -10T04:48:48.428122+0000 mon.a (mon.0) 2171 2021-02-10T04:48:48.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[29288]: : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 257 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:48:48.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:48 smithi085 conmon[29319]: cluster 2021-02-10T04:48:47.752396+0000 osd.4 (osd.4) 2281 : 2021-02-10T04:48:48.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:48 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:48.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:48 smithi085 conmon[29319]: cluster 2021-02-10T04:48:47.752401+0000 osd.4 (osd.4) 2282 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:48.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:48 smithi085 conmon[29319]: cluster 2021-02-10T04:48:47.752405+0000 osd.4 (osd. 2021-02-10T04:48:48.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:48 smithi085 conmon[29319]: 4) 2283 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:48.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:48 smithi085 conmon[29319]: cluster 2021-02-10T04:48:47.752409+0000 2021-02-10T04:48:48.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:48 smithi085 conmon[29319]: osd.4 (osd.4) 2284 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:48.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:48 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:48:48.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:48 smithi085 conmon[29319]: :48:48.428122+0000 mon.a (mon.0) 2171 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 257 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:48:48.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:48 smithi085 conmon[35961]: cluster 2021-02-10T04:48:47.752396+0000 2021-02-10T04:48:48.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:48 smithi085 conmon[35961]: osd.4 (osd.4) 2281 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:48.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:48 smithi085 conmon[35961]: cluster 2021-02-10T04:48:48.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:48 smithi085 conmon[35961]: 2021-02-10T04:48:47.752401+0000 osd.4 ( 2021-02-10T04:48:48.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:48 smithi085 conmon[35961]: osd.4) 2282 : cluster [WRN] 2021-02-10T04:48:48.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:48 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:48.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:48 smithi085 conmon[35961]: cluster 2021-02-10T04:48:48.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:48 smithi085 conmon[35961]: 2021-02-10T04:48:47.752405+0000 osd.4 (osd.4) 2283 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:48.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:48 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:48:48.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:48 smithi085 conmon[35961]: 04:48:47.752409+0000 osd.4 (osd. 2021-02-10T04:48:48.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:48 smithi085 conmon[35961]: 4) 2284 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:48.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:48 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:48:48.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:48 smithi085 conmon[35961]: 10T04:48:48.428122+0000 mon.a (mon.0 2021-02-10T04:48:48.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:48 smithi085 conmon[35961]: ) 2171 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 257 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:48:49.394 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:48:48 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:48:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:49 smithi164 conmon[29288]: cluster 2021-02-10T04:48:48.363981+0000 mgr.y (mgr.14140) 2021-02-10T04:48:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:49 smithi164 conmon[29288]: 1107 : cluster [DBG] pgmap v1887: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:48:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:49 smithi164 conmon[29288]: cluster 2021-02-10T04:48:48.770535+0000 osd.4 (osd.4) 2285 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:49 smithi164 conmon[29288]: cluster 2021-02-10T04:48:48.770544+0000 osd.4 (osd.4) 2286 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:49 smithi164 conmon[29288]: cluster 2021-02-10T04:48:48.770551+0000 osd.4 (osd.4) 2287 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:49 smithi164 conmon[29288]: cluster 2021-02-10T04:48:48.770561+0000 osd.4 (osd.4) 2288 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:49 smithi164 conmon[29288]: audit 2021-02-10T04:48:48.907991+0000 mgr.y (mgr.14140) 1108 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:48:49.895 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:49 smithi164 conmon[33677]: debug 2021-02-10T04:48:49.773+0000 7fbec19a1700 -1 osd.4 1082 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:49.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[35961]: cluster 2021-02-10T04:48:48.363981+0000 2021-02-10T04:48:49.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[35961]: mgr.y (mgr.14140) 1107 : cluster [DBG] pgmap v1887: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:48:49.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:48:49.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[35961]: 10T04:48:48.770535+0000 osd.4 (osd.4) 2285 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:49.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:48:49.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[35961]: T04:48:48.770544+0000 osd.4 (osd.4) 2286 : cluster [WRN] 2021-02-10T04:48:49.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:49.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[35961]: cluster 2021-02-10T04:48:49.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[35961]: 2021-02-10T04:48:48.770551+0000 osd.4 (osd. 2021-02-10T04:48:49.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[35961]: 4) 2287 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:49.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:48:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[35961]: :48:48.770561+0000 osd.4 (osd.4 2021-02-10T04:48:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[35961]: ) 2288 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:48:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[35961]: -10T04:48:48.907991+0000 mgr.y 2021-02-10T04:48:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[35961]: (mgr.14140) 1108 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:48:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[29319]: cluster 2021-02-10T04:48 2021-02-10T04:48:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[29319]: :48.363981+0000 mgr.y (mgr.14140) 1107 : cluster [DBG] pgmap v1887: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:48:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:48:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[29319]: 10T04:48:48.770535 2021-02-10T04:48:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2285 2021-02-10T04:48:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:48:49.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[29319]: T04:48:48.770544+0000 2021-02-10T04:48:49.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[29319]: osd.4 2021-02-10T04:48:49.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[29319]: (osd.4 2021-02-10T04:48:49.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[29319]: ) 2286 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:49.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:48:49.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[29319]: 10T04:48:48.770551+0000 osd.4 (osd.4) 2287 2021-02-10T04:48:49.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:49.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:48:49.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[29319]: 10T04:48:48.770561+0000 osd.4 (osd.4) 2288 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:49.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[29319]: audit 2021-02-10T04:48:49.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:48:49.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[29319]: 04:48:48.907991+0000 2021-02-10T04:48:49.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[29319]: mgr.y (mgr.14140) 1108 2021-02-10T04:48:49.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:49 smithi085 conmon[29319]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:48:50.317 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:48:50.465 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:48:50.738 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:48:50.838 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:48:50 smithi085 conmon[52836]: debug 2021-02-10T04:48:50.655+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:48:50.839 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:50 smithi085 conmon[29319]: cluster 2021-02-10T04:48:49.774101+0000 osd.4 ( 2021-02-10T04:48:50.839 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:50 smithi085 conmon[29319]: osd.4) 2289 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:50.839 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:50 smithi085 conmon[29319]: cluster 2021-02-10T04:48:49.774109+0000 osd.4 (osd.4) 2290 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:50.839 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:50 smithi085 conmon[29319]: cluster 2021-02-10T04:48:49.774116+0000 osd.4 ( 2021-02-10T04:48:50.839 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:50 smithi085 conmon[29319]: osd.4) 2291 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:50.839 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:50 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:48:50.839 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:50 smithi085 conmon[29319]: 02-10T04:48:49.774121+0000 osd.4 (osd 2021-02-10T04:48:50.839 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:50 smithi085 conmon[29319]: .4) 2292 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:50.839 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:50 smithi085 conmon[29319]: 2021-02-10T04:48:50.840 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:50 smithi085 conmon[35961]: cluster 2021-02-10T04:48:49.774101+0000 osd.4 (osd. 2021-02-10T04:48:50.840 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:50 smithi085 conmon[35961]: 4) 2289 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:50.840 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:50 smithi085 conmon[35961]: cluster 2021-02-10T04:48:49.774109+0000 osd.4 (osd.4) 2290 : cluster [WRN] 2021-02-10T04:48:50.840 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:50 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:50.840 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:50 smithi085 conmon[35961]: cluster 2021-02-10T04:48:49.774116+0000 osd.4 (osd.4) 2291 : cluster 2021-02-10T04:48:50.840 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:50 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:50.840 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:50 smithi085 conmon[35961]: cluster 2021-02-10T04:48:49.774121 2021-02-10T04:48:50.840 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:50 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2292 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:50.845 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:48:50.880 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:48:50.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:50 smithi164 conmon[33677]: debug 2021-02-10T04:48:50.785+0000 7fbec19a1700 -1 osd.4 1082 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:50.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:50 smithi164 conmon[29288]: cluster 2021-02-10T04:48:49.774101+0000 osd.4 (osd. 2021-02-10T04:48:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:50 smithi164 conmon[29288]: 4) 2289 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:50 smithi164 conmon[29288]: cluster 2021-02-10T04:48:49.774109+0000 osd.4 (osd.4) 2290 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:50 smithi164 conmon[29288]: cluster 2021-02-10T04:48:49.774116+0000 osd.4 (osd.4 2021-02-10T04:48:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:50 smithi164 conmon[29288]: ) 2291 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:50 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:48:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:50 smithi164 conmon[29288]: 10T04:48:49.774121+0000 osd.4 (osd.4) 2292 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:50.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:50 smithi164 conmon[29288]: 2021-02-10T04:48:50.997 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:48:51.099 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:48:50 smithi085 conmon[41540]: debug 2021-02-10T04:48:50.963+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:48:51.201 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:48:51.242 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:48:51.608 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:48:51.643 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:48:51 smithi085 conmon[52836]: debug 2021-02-10T04:48:51.381+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:48:51.644 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:51 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:48:51.644 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:51 smithi085 conmon[29319]: 04:48:50.364627+0000 mgr.y (mgr.14140) 1109 : cluster [DBG] pgmap v1888: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:48:51.644 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:51 smithi085 conmon[29319]: cluster 2021-02-10T04:48:50.786563+0000 osd.4 (osd.4) 2293 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:51.644 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:51 smithi085 conmon[29319]: cluster 2021-02-10T04:48:50.786572+0000 osd.4 (osd.4) 2294 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:51.644 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:51 smithi085 conmon[29319]: cluster 2021-02-10T04:48:50.786577+0000 osd.4 (osd.4) 2295 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:51.644 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:51 smithi085 conmon[29319]: cluster 2021-02-10T04:48:50.786581+0000 osd.4 (osd.4) 2296 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:51.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:51 smithi085 conmon[35961]: cluster 2021-02-10T04:48:51.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:51 smithi085 conmon[35961]: 2021-02-10T04:48: 2021-02-10T04:48:51.644 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:51 smithi085 conmon[35961]: 50.364627+0000 mgr.y (mgr.14140) 1109 : cluster [DBG] pgmap v1888: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:48:51.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:51 smithi085 conmon[35961]: cluster 2021-02-10T04:48:51.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:51 smithi085 conmon[35961]: 2021-02-10T04:48:50. 2021-02-10T04:48:51.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:51 smithi085 conmon[35961]: 786563+0000 osd.4 (osd.4) 2293 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:51.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:51 smithi085 conmon[35961]: 2021-02-10T04:48:51.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:51 smithi085 conmon[35961]: cluster 2021-02-10T04:48:51.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:51 smithi085 conmon[35961]: 2021-02-10T04:48:50 2021-02-10T04:48:51.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:51 smithi085 conmon[35961]: .786572+0000 osd.4 (osd.4) 2294 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:51.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:51 smithi085 conmon[35961]: cluster 2021-02-10T04:48 2021-02-10T04:48:51.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:51 smithi085 conmon[35961]: :50.786577+0000 osd.4 (osd.4) 2295 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:51.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:51 smithi085 conmon[35961]: cluster 2021-02-10T04:48:51.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:51 smithi085 conmon[35961]: 2021-02-10T04:48:50.786581+0000 osd.4 (osd.4) 2296 : cluster 2021-02-10T04:48:51.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:51 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:51.712 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:48:51.740 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:48:51.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:51 smithi164 conmon[29288]: cluster 2021-02-10T04:48:51.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:51 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:48:51.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:51 smithi164 conmon[29288]: 04:48:50.364627+0000 mgr.y (mgr.14140) 1109 : cluster [DBG] pgmap v1888: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:48:51.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:51 smithi164 conmon[29288]: cluster 2021-02-10T04:48:51.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:51 smithi164 conmon[29288]: 2021-02-10T04:48:50 2021-02-10T04:48:51.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:51 smithi164 conmon[29288]: .786563+0000 osd.4 (osd.4) 2293 2021-02-10T04:48:51.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:51 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:51.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:51 smithi164 conmon[29288]: cluster 2021-02-10T04:48:51.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:51 smithi164 conmon[29288]: 2021-02-10 2021-02-10T04:48:51.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:51 smithi164 conmon[29288]: T04:48:50.786572+0000 osd.4 (osd.4) 2294 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:51.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:51 smithi164 conmon[29288]: cluster 2021-02-10T04:48:50. 2021-02-10T04:48:51.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:51 smithi164 conmon[29288]: 786577+0000 osd.4 (osd.4) 2295 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:51.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:51 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:48:51.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:51 smithi164 conmon[29288]: -02-10T04:48:50.786581+0000 osd.4 2021-02-10T04:48:51.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:51 smithi164 conmon[29288]: (osd.4) 2296 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:51.895 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:48:51 smithi164 conmon[44544]: debug 2021-02-10T04:48:51.895 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:48:51 smithi164 conmon[44544]: 2021-02-10T04:48:51.817+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:48:51.895 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:48:51 smithi164 conmon[44544]: 2021-02-10T04:48:51.896 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:51 smithi164 conmon[33677]: debug 2021-02-10T04:48:51.751+0000 7fbec19a1700 -1 osd.4 1082 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:52.408 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:48:52.515 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.4 2021-02-10T04:48:52.548 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:48:52.793 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:52 smithi164 conmon[29288]: cluster 2021-02-10T04:48:51.674670+0000 osd.4 (osd 2021-02-10T04:48:52.793 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:52 smithi164 conmon[29288]: .4) 2297 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:48:52.793 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:52 smithi164 conmon[29288]: cluster 2021-02-10T04:48:51.675836+0000 osd.4 (osd.4) 2298 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:48:52.793 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:52 smithi164 conmon[29288]: cluster 2021-02-10T04:48:51.752059+0000 osd.4 (osd.4) 2299 : 2021-02-10T04:48:52.793 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:52 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:52.793 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:52 smithi164 conmon[29288]: cluster 2021-02-10T04:48: 2021-02-10T04:48:52.793 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:52 smithi164 conmon[29288]: 51.752069+0000 osd.4 (osd.4) 2300 : 2021-02-10T04:48:52.793 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:52 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:52.793 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:52 smithi164 conmon[29288]: cluster 2021-02-10T04:48:51.752076+0000 osd.4 (osd. 2021-02-10T04:48:52.793 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:52 smithi164 conmon[29288]: 4) 2301 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:52.794 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:52 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:48:52.794 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:52 smithi164 conmon[29288]: T04:48:51.752082+0000 osd.4 (osd. 2021-02-10T04:48:52.794 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:52 smithi164 conmon[29288]: 4) 2302 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:52.794 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:52 smithi164 conmon[33677]: debug 2021-02-10T04:48:52.628+0000 7fbec6a28700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:48:52.794 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:52 smithi164 conmon[33677]: debug 2021-02-10T04:48:52.794 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:52 smithi164 conmon[33677]: 2021-02-10T04:48:52.791+0000 7fbec19a1700 -1 osd.4 1083 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:52.795 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:52 smithi164 conmon[33677]: 2021-02-10T04:48:52.879 INFO:teuthology.orchestra.run.smithi164.stdout:3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T04:48:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[29319]: cluster 2021-02-10T04:48:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[29319]: 2021-02-10T04:48:51 2021-02-10T04:48:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[29319]: .674670+0000 osd.4 (osd.4) 2297 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:48:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[29319]: cluster 2021-02-10T04:48:51.675836 2021-02-10T04:48:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2298 : cluster [DBG] 2021-02-10T04:48:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[29319]: purged_snaps scrub ok 2021-02-10T04:48:52.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:48:52.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[29319]: 02-10T04:48:51.752059+0000 2021-02-10T04:48:52.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[29319]: osd.4 (osd.4) 2299 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:52.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[29319]: cluster 2021-02-10T04:48:52.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[29319]: 2021-02-10T04:48:52.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[29319]: 2021 2021-02-10T04:48:52.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[29319]: - 2021-02-10T04:48:52.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[29319]: 02 2021-02-10T04:48:52.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[29319]: - 2021-02-10T04:48:52.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[29319]: 10 2021-02-10T04:48:52.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[29319]: T 2021-02-10T04:48:52.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[29319]: 04 2021-02-10T04:48:52.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[29319]: : 2021-02-10T04:48:52.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[29319]: 48 2021-02-10T04:48:52.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[29319]: : 2021-02-10T04:48:52.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[29319]: 51.752069+0000 osd.4 (osd.4) 2300 : cluster [WRN] 2021-02-10T04:48:52.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:52.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[29319]: cluster 2021-02-10T04:48:51.752076+0000 2021-02-10T04:48:52.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[29319]: osd.4 (osd.4) 2301 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:52.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[29319]: cluster 2021-02-10T04:48:52.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[29319]: 2021-02-10T04:48:51.752082 2021-02-10T04:48:52.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2302 : cluster [WRN] 2021-02-10T04:48:52.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:52.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[35961]: cluster 2021-02-10T04:48:52.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[35961]: 2021-02-10T04:48:52.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[35961]: 2021 2021-02-10T04:48:52.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[35961]: - 2021-02-10T04:48:52.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[35961]: 02 2021-02-10T04:48:52.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[35961]: - 2021-02-10T04:48:52.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[35961]: 10 2021-02-10T04:48:52.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[35961]: T04:48:51.674670+0000 osd.4 2021-02-10T04:48:52.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[35961]: (osd.4) 2297 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:48:52.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:48:52.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[35961]: :48:51.675836+0000 osd.4 (osd.4) 2298 : cluster 2021-02-10T04:48:52.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[35961]: [DBG] purged_snaps scrub ok 2021-02-10T04:48:52.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:48:52.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[35961]: 10T04:48:51.752059+0000 osd.4 (osd. 2021-02-10T04:48:52.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[35961]: 4) 2299 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:52.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:48:52.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[35961]: 02-10T04:48:51.752069+0000 osd.4 ( 2021-02-10T04:48:52.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[35961]: osd.4) 2300 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:52.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:48:52.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[35961]: T04:48:51.752076+0000 osd.4 (osd. 2021-02-10T04:48:52.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[35961]: 4) 2301 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:52.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:48:52.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[35961]: 04:48:51.752082+0000 osd.4 (osd. 2021-02-10T04:48:52.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:52 smithi085 conmon[35961]: 4) 2302 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:52.984 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:48:53.013 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:48:53.090 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:48:53 smithi164 conmon[44544]: debug 2021-02-10T04:48:53.088+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:48:53.392 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:48:53.498 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:48:53.535 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:48:53.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[29319]: cluster 2021-02-10T04:48:52.365333+0000 mgr.y 2021-02-10T04:48:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[29319]: (mgr.14140) 1110 : cluster [DBG] pgmap v1889: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:48:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[29319]: cluster 2021-02-10T04:48:52.653287+0000 mon.a (mon.0) 2172 : cluster [DBG] osdmap e1083: 8 total, 8 up, 7 in 2021-02-10T04:48:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[29319]: cluster 2021-02-10T04:48:52.792297+0000 osd.4 (osd.4) 2303 : cluster 2021-02-10T04:48:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[29319]: cluster 2021-02-10T04:48:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[29319]: 2021-02-10T04:48:52.792305 2021-02-10T04:48:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2304 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:48:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[29319]: 02-10T04:48:52.792312+0000 osd.4 (osd. 2021-02-10T04:48:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[29319]: 4) 2305 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[29319]: cluster 2021-02-10T04:48:52.792318+0000 2021-02-10T04:48:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[29319]: osd.4 (osd.4) 2306 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:53.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[29319]: 2021-02-10T04:48:53.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[29319]: cluster 2021-02-10T04:48:53.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[29319]: 2021-02-10T04:48:53.429741+0000 mon.a (mon.0 2021-02-10T04:48:53.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[29319]: ) 2173 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 267 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:48:53.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[35961]: cluster 2021-02-10T04:48:53.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[35961]: 2021-02-10T04:48:53.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[35961]: 2021-02-10T04:48:52.365333+0000 2021-02-10T04:48:53.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[35961]: mgr.y (mgr.14140) 1110 : cluster [DBG] pgmap v1889: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:48:53.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[35961]: cluster 2021-02-10T04:48:53.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[35961]: 2021-02-10T04:48:52.653287+0000 mon.a (mon.0) 2172 : cluster [DBG] 2021-02-10T04:48:53.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[35961]: osdmap e1083: 8 total, 8 up, 7 in 2021-02-10T04:48:53.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[35961]: cluster 2021-02-10T04:48: 2021-02-10T04:48:53.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[35961]: 52.792297+0000 osd.4 (osd.4) 2303 : cluster 2021-02-10T04:48:53.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:53.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:48:53.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[35961]: 48:52.792305+0000 osd.4 (osd.4) 2304 2021-02-10T04:48:53.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:53.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[35961]: cluster 2021-02-10T04:48 2021-02-10T04:48:53.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[35961]: :52.792312+0000 osd.4 (osd.4) 2305 2021-02-10T04:48:53.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:53.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:48:53.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[35961]: :48:52.792318+0000 osd.4 (osd.4) 2021-02-10T04:48:53.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[35961]: 2306 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:53.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:48:53.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[35961]: 04:48:53.429741+0000 mon.a (mon. 2021-02-10T04:48:53.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:53 smithi085 conmon[35961]: 0) 2173 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 267 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:48:53.979 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:48:53.980 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [], 2021-02-10T04:48:53.980 INFO:teuthology.orchestra.run.smithi085.stdout: "num_ops": 0 2021-02-10T04:48:53.980 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:48:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:53 smithi164 conmon[29288]: cluster 2021-02-10T04:48:52.365333+0000 mgr.y (mgr.14140) 1110 : cluster [DBG] 2021-02-10T04:48:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:53 smithi164 conmon[29288]: pgmap v1889: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:48:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:53 smithi164 conmon[29288]: cluster 2021-02-10T04:48:52.653287+0000 mon.a (mon.0) 2172 : cluster [DBG] osdmap e1083: 8 total, 8 up, 7 in 2021-02-10T04:48:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:53 smithi164 conmon[29288]: cluster 2021-02-10T04:48 2021-02-10T04:48:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:53 smithi164 conmon[29288]: :52.792297+0000 osd.4 (osd.4) 2303 2021-02-10T04:48:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:53 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:53 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:48:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:53 smithi164 conmon[29288]: :48:52.792305+0000 osd.4 (osd.4) 2304 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:53 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:48:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:53 smithi164 conmon[29288]: 10T04:48:52.792312+0000 2021-02-10T04:48:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:53 smithi164 conmon[29288]: osd.4 (osd.4) 2305 : 2021-02-10T04:48:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:53 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:53 smithi164 conmon[29288]: cluster 2021-02-10T04:48:52.792318+0000 osd.4 (osd.4) 2306 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:53 smithi164 conmon[29288]: cluster 2021-02-10T04:48:53.429741+0000 mon.a (mon.0) 2173 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 267 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:48:54.145 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:53 smithi164 conmon[33677]: debug 2021-02-10T04:48:54.145 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:53 smithi164 conmon[33677]: 2021-02-10T04:48:53.801+0000 7fbec19a1700 -1 osd.4 1083 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:54.145 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:53 smithi164 conmon[33677]: 2021-02-10T04:48:54.668 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is unset 2021-02-10T04:48:54.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:54 smithi085 conmon[35961]: audit 2021-02-10T04:48:53 2021-02-10T04:48:54.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:54 smithi085 conmon[35961]: .693515+0000 mon.a (mon.0) 2174 : audit [INF] from='client.? 172.21.15.85:0/2658296459' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:48:54.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:54 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:48:54.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:54 smithi085 conmon[35961]: 02-10T04:48:53.802097+0000 osd.4 (osd 2021-02-10T04:48:54.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:54 smithi085 conmon[35961]: .4) 2307 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:54.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:54 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:48:54.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:54 smithi085 conmon[35961]: T04:48:53.802104+0000 osd.4 (osd.4) 2308 : cluster 2021-02-10T04:48:54.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:54 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:54.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:54 smithi085 conmon[35961]: cluster 2021-02-10T04:48:53.802110+0000 osd.4 (osd. 2021-02-10T04:48:54.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:54 smithi085 conmon[35961]: 4) 2309 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:54.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:54 smithi085 conmon[35961]: cluster 2021-02-10T04:48:54.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:54 smithi085 conmon[35961]: 2021-02-10T04:48:54.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:54 smithi085 conmon[35961]: 2021 2021-02-10T04:48:54.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:54 smithi085 conmon[35961]: -02-10T04:48:53 2021-02-10T04:48:54.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:54 smithi085 conmon[35961]: .802116+0000 osd.4 (osd.4) 2310 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:54.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:54 smithi085 conmon[29319]: audit 2021-02-10T04:48:53.693515+0000 mon.a (mon 2021-02-10T04:48:54.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:54 smithi085 conmon[29319]: .0) 2174 : audit [INF] from='client.? 172.21.15.85:0/2658296459' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:48:54.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:54 smithi085 conmon[29319]: cluster 2021-02-10T04:48:53.802097+0000 osd.4 (osd.4) 2307 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:54.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:54 smithi085 conmon[29319]: cluster 2021-02-10T04:48:53.802104+0000 osd.4 (osd.4) 2308 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:54.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:54 smithi085 conmon[29319]: cluster 2021-02-10T04:48:53.802110+0000 osd.4 (osd.4) 2309 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:54.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:54 smithi085 conmon[29319]: cluster 2021-02-10T04:48:53.802116+0000 osd.4 (osd.4) 2310 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:55.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:54 smithi164 conmon[33677]: debug 2021-02-10T04:48:55.145 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:54 smithi164 conmon[33677]: 2021-02-10T04:48:54.752+0000 7fbec19a1700 -1 osd.4 1084 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:55.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:54 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:48:55.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:54 smithi164 conmon[29288]: :48:53.693515+0000 mon.a (mon.0) 2174 : audit [INF] from='client.? 172.21.15.85:0/2658296459' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:48:55.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:54 smithi164 conmon[29288]: cluster 2021-02-10T04:48:53 2021-02-10T04:48:55.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:54 smithi164 conmon[29288]: .802097+0000 osd.4 (osd.4) 2307 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:55.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:54 smithi164 conmon[29288]: cluster 2021-02-10T04:48:53. 2021-02-10T04:48:55.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:54 smithi164 conmon[29288]: 802104+0000 osd.4 (osd.4) 2308 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:55.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:54 smithi164 conmon[29288]: 2021-02-10T04:48:55.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:54 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:48:55.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:54 smithi164 conmon[29288]: 02-10T04:48:53.802110+0000 osd.4 (osd. 2021-02-10T04:48:55.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:54 smithi164 conmon[29288]: 4) 2309 2021-02-10T04:48:55.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:54 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:55.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:54 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:48:55.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:54 smithi164 conmon[29288]: 02-10T04:48:53.802116 2021-02-10T04:48:55.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:54 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2310 : cluster [WRN] 2021-02-10T04:48:55.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:54 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:55.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[29319]: cluster 2021-02-10T04:48:54.365975+0000 2021-02-10T04:48:55.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[29319]: mgr.y (mgr.14140) 1111 : cluster [DBG] pgmap v1891: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:48:55.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[29319]: cluster 2021-02-10T04:48:54.651279+0000 mon.a (mon.0) 2175 : cluster 2021-02-10T04:48:55.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[29319]: [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:48:55.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[29319]: audit 2021-02-10T04:48:54.664200+0000 mon.a (mon.0) 2021-02-10T04:48:55.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[29319]: 2176 : audit [INF] from='client.? 172.21.15.85:0/2658296459' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:48:55.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:48:55.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[29319]: 04:48:54.664276+0000 mon.a (mon.0) 2177 : cluster [DBG] 2021-02-10T04:48:55.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[29319]: osdmap e1084: 8 total, 8 up, 7 in 2021-02-10T04:48:55.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:48:55.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[29319]: -02-10T04:48: 2021-02-10T04:48:55.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[29319]: 54.753343+0000 osd.4 (osd.4) 2311 2021-02-10T04:48:55.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:55.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:48:55.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[29319]: 04:48:54.753352+0000 osd.4 2021-02-10T04:48:55.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[29319]: (osd.4) 2312 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:55.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:48:55.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[29319]: -02-10T04:48:54. 2021-02-10T04:48:55.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[29319]: 753358+0000 osd.4 (osd 2021-02-10T04:48:55.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[29319]: .4) 2313 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:55.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[29319]: cluster 2021-02-10T04:48:54.753363+0000 2021-02-10T04:48:55.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[29319]: osd.4 (osd.4) 2314 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:55.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:48:55.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[29319]: -10T04:48:54.943230+0000 osd.5 2021-02-10T04:48:55.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[29319]: (osd.5) 138 : cluster [DBG] 2.16 scrub ok 2021-02-10T04:48:55.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[29319]: 2021-02-10T04:48:55.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[35961]: cluster 2021-02-10T04:48:54. 2021-02-10T04:48:55.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[35961]: 365975+0000 mgr.y (mgr.14140) 1111 : cluster [DBG] pgmap v1891: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:48:55.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:48:55.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[35961]: T04:48:54.651279+0000 mon.a (mon.0) 2175 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:48:55.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:48:55.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[35961]: -10T04:48:54.664200+0000 mon.a (mon 2021-02-10T04:48:55.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[35961]: .0) 2176 : audit [INF] from='client.? 172.21.15.85:0/2658296459' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:48:55.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:48:55.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[35961]: -10T04:48:54.664276+0000 mon.a (mon 2021-02-10T04:48:55.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[35961]: .0) 2177 : cluster [DBG] osdmap e1084: 8 total, 8 up, 7 in 2021-02-10T04:48:55.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:48:55.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[35961]: :48:54.753343+0000 osd.4 (osd.4) 2311 : cluster [WRN] 2021-02-10T04:48:55.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:55.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:48:55.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[35961]: 04:48:54.753352+0000 osd.4 (osd.4) 2312 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:55.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[35961]: cluster 2021-02-10T04:48:54.753358+0000 osd.4 (osd.4 2021-02-10T04:48:55.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[35961]: ) 2313 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:55.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[35961]: cluster 2021-02-10T04:48: 2021-02-10T04:48:55.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[35961]: 54.753363+0000 osd.4 (osd.4) 2314 : cluster 2021-02-10T04:48:55.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:55.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[35961]: cluster 2021-02-10T04:48: 2021-02-10T04:48:55.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[35961]: 54.943230+0000 osd.5 (osd. 2021-02-10T04:48:55.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[35961]: 5) 138 : cluster [DBG] 2021-02-10T04:48:55.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:55 smithi085 conmon[35961]: 2.16 scrub ok 2021-02-10T04:48:56.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:55 smithi164 conmon[33677]: debug 2021-02-10T04:48:55.791+0000 7fbec19a1700 -1 osd.4 1084 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:55 smithi164 conmon[29288]: cluster 2021-02-10T04:48:54.365975+0000 2021-02-10T04:48:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:55 smithi164 conmon[29288]: mgr.y (mgr.14140) 1111 : cluster [DBG] pgmap v1891: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:48:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:55 smithi164 conmon[29288]: cluster 2021-02-10T04:48:54.651279+0000 mon.a (mon.0) 2175 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:48:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:55 smithi164 conmon[29288]: audit 2021-02-10T04:48:54. 2021-02-10T04:48:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:55 smithi164 conmon[29288]: 664200+0000 mon.a (mon.0) 2176 : audit [INF] from='client.? 172.21.15.85:0/2658296459' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:48:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:55 smithi164 conmon[29288]: cluster 2021-02-10T04:48: 2021-02-10T04:48:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:55 smithi164 conmon[29288]: 54.664276+0000 mon.a (mon.0) 2177 : cluster [DBG] osdmap e1084: 8 total, 8 up, 7 in 2021-02-10T04:48:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:55 smithi164 conmon[29288]: cluster 2021-02-10T04:48:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:55 smithi164 conmon[29288]: 2021-02-10T04:48:54 2021-02-10T04:48:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:55 smithi164 conmon[29288]: .753343+0000 osd.4 (osd.4) 2311 : cluster 2021-02-10T04:48:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:55 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:55 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:48:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:55 smithi164 conmon[29288]: :48:54.753352+0000 osd.4 (osd.4) 2312 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:55 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:48:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:55 smithi164 conmon[29288]: :48:54.753358+0000 osd.4 (osd.4) 2313 : 2021-02-10T04:48:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:55 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:56.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:55 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:48:56.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:55 smithi164 conmon[29288]: :48:54.753363+0000 osd.4 (osd.4) 2314 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:56.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:55 smithi164 conmon[29288]: cluster 2021-02-10T04:48:56.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:55 smithi164 conmon[29288]: 2021-02-10T04:48:56.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:55 smithi164 conmon[29288]: 2021 2021-02-10T04:48:56.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:55 smithi164 conmon[29288]: - 2021-02-10T04:48:56.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:55 smithi164 conmon[29288]: 02-10T04:48:54 2021-02-10T04:48:56.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:55 smithi164 conmon[29288]: .943230+0000 osd.5 (osd.5) 138 : cluster [DBG] 2.16 scrub ok 2021-02-10T04:48:56.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:55 smithi164 conmon[29288]: 2021-02-10T04:48:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:48:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[29319]: 48:55.792618+0000 osd.4 2021-02-10T04:48:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[29319]: (osd.4) 2315 : 2021-02-10T04:48:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[29319]: cluster 2021-02-10T04:48:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[29319]: 2021-02-10T04:48:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:48:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[29319]: :48:55.792626+0000 osd.4 (osd.4) 2316 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[29319]: cluster 2021-02-10T04:48:55.792632+0000 osd.4 (osd 2021-02-10T04:48:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[29319]: .4) 2317 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[29319]: cluster 2021-02-10T04:48:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[29319]: 2021-02-10T04:48:55.792638+0000 osd.4 (osd. 2021-02-10T04:48:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[29319]: 4) 2318 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[29319]: cluster 2021-02-10T04:48:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:48:56.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[29319]: T04:48:56.530265+0000 osd.1 (osd.1) 475 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:48:56.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[29319]: 2021-02-10T04:48:56.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[29319]: cluster 2021-02-10T04:48:56.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[29319]: 2021-02-10T04:48:56.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[29319]: 2021 2021-02-10T04:48:56.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[29319]: - 2021-02-10T04:48:56.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[29319]: 02 2021-02-10T04:48:56.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[29319]: -10T04:48:56. 2021-02-10T04:48:56.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[29319]: 533656+0000 osd.1 (osd.1) 476 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:48:56.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[35961]: cluster 2021-02-10T04:48:56.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[35961]: 2021-02-10T04:48:56.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[35961]: 2021 2021-02-10T04:48:56.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[35961]: - 2021-02-10T04:48:56.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[35961]: 02 2021-02-10T04:48:56.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[35961]: - 2021-02-10T04:48:56.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[35961]: 10 2021-02-10T04:48:56.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[35961]: T 2021-02-10T04:48:56.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[35961]: 04 2021-02-10T04:48:56.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[35961]: : 2021-02-10T04:48:56.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[35961]: 48 2021-02-10T04:48:56.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[35961]: : 2021-02-10T04:48:56.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[35961]: 55 2021-02-10T04:48:56.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[35961]: .792618+0000 osd.4 ( 2021-02-10T04:48:56.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[35961]: osd.4) 2315 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:56.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:48:56.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[35961]: 02-10T04:48:55.792626+0000 osd.4 (osd 2021-02-10T04:48:56.949 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[35961]: .4) 2316 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:56.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:48:56.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[35961]: 02-10T04:48:55.792632+0000 osd.4 ( 2021-02-10T04:48:56.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[35961]: osd.4) 2317 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:56.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:48:56.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[35961]: 02-10T04:48:55.792638+0000 osd.4 ( 2021-02-10T04:48:56.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[35961]: osd.4) 2318 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:56.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:48:56.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[35961]: -10T04:48:56.530265+0000 osd.1 (osd 2021-02-10T04:48:56.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[35961]: .1) 475 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:48:56.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:48:56.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[35961]: 02-10T04:48:56.533656+0000 osd.1 ( 2021-02-10T04:48:56.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:56 smithi085 conmon[35961]: osd.1) 476 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:48:57.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:56 smithi164 conmon[33677]: debug 2021-02-10T04:48:56.745+0000 7fbec19a1700 -1 osd.4 1085 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:56 smithi164 conmon[29288]: cluster 2021-02-10T04:48:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:56 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:48:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:56 smithi164 conmon[29288]: 48:55.792618+0000 osd.4 (osd.4) 2315 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:56 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:48:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:56 smithi164 conmon[29288]: 10T04:48:55.792626+0000 osd.4 (osd 2021-02-10T04:48:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:56 smithi164 conmon[29288]: .4) 2316 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:56 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:48:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:56 smithi164 conmon[29288]: -02-10T04:48:55.792632+0000 osd.4 2021-02-10T04:48:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:56 smithi164 conmon[29288]: (osd.4) 2317 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:56 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:48:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:56 smithi164 conmon[29288]: 02-10T04:48:55.792638+0000 osd.4 ( 2021-02-10T04:48:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:56 smithi164 conmon[29288]: osd.4) 2318 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:56 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:48:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:56 smithi164 conmon[29288]: 02-10T04:48:56.530265+0000 2021-02-10T04:48:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:56 smithi164 conmon[29288]: osd.1 (osd.1) 475 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:48:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:56 smithi164 conmon[29288]: 2021-02-10T04:48:57.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:56 smithi164 conmon[29288]: cluster 2021-02-10T04:48:57.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:56 smithi164 conmon[29288]: 2021-02-10T04:48:56.533656+0000 osd.1 (osd 2021-02-10T04:48:57.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:56 smithi164 conmon[29288]: .1) 476 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:48:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[35961]: cluster 2021-02-10T04:48:56.366541+0000 mgr.y (mgr. 2021-02-10T04:48:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[35961]: 14140) 1112 : cluster [DBG] pgmap v1893: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:48:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:48:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[35961]: 48:56.672711+0000 mon.a (mon.0) 2178 : cluster [DBG] osdmap e1085: 8 total, 8 up, 7 in 2021-02-10T04:48:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[35961]: cluster 2021-02-10T04:48: 2021-02-10T04:48:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[35961]: 56.745923+0000 osd.4 (osd.4) 2319 : cluster [WRN] 2021-02-10T04:48:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[35961]: cluster 2021-02-10T04:48:56.745933+0000 osd.4 2021-02-10T04:48:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[35961]: (osd.4) 2320 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[35961]: cluster 2021-02-10T04:48:56.745939+0000 2021-02-10T04:48:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[35961]: osd.4 (osd.4) 2321 : cluster [WRN] 2021-02-10T04:48:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:48:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[35961]: -10T04:48:56 2021-02-10T04:48:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[35961]: .745945+0000 osd.4 (osd.4) 2021-02-10T04:48:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[35961]: 2322 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:57.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[29319]: cluster 2021-02-10T04:48:56.366541 2021-02-10T04:48:57.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 1112 : cluster [DBG] pgmap v1893: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:48:57.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[29319]: cluster 2021-02-10T04:48:56.672711 2021-02-10T04:48:57.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2178 : cluster [DBG] osdmap e1085: 8 total, 8 up, 7 in 2021-02-10T04:48:57.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:48:57.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[29319]: -10T04:48:56.745923+0000 2021-02-10T04:48:57.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[29319]: osd.4 (osd.4) 2319 : 2021-02-10T04:48:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[29319]: cluster 2021-02-10T04:48 2021-02-10T04:48:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[29319]: :56.745933+0000 osd.4 (osd. 2021-02-10T04:48:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[29319]: 4) 2320 : cluster [WRN] 2021-02-10T04:48:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[29319]: cluster 2021-02-10T04:48:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:48:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[29319]: 04:48:56.745939+0000 osd.4 (osd.4) 2321 : cluster 2021-02-10T04:48:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:48:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[29319]: 02-10T04:48:56.745945+0000 osd.4 (osd. 2021-02-10T04:48:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[29319]: 4) 2322 : cluster [WRN] 2021-02-10T04:48:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:57 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:58.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:57 smithi164 conmon[33677]: debug 2021-02-10T04:48:57.759+0000 7fbec19a1700 -1 osd.4 1085 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:57 smithi164 conmon[29288]: cluster 2021-02-10T04:48:56.366541+0000 mgr.y (mgr.14140) 1112 : 2021-02-10T04:48:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:57 smithi164 conmon[29288]: cluster [DBG] pgmap v1893: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:48:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:57 smithi164 conmon[29288]: cluster 2021-02-10T04:48:56.672711+0000 mon.a (mon.0) 2178 : cluster [DBG] osdmap e1085: 8 total, 8 up, 7 in 2021-02-10T04:48:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:57 smithi164 conmon[29288]: cluster 2021-02-10T04:48:56. 2021-02-10T04:48:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:57 smithi164 conmon[29288]: 745923+0000 osd.4 (osd.4) 2319 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:57 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:48:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:57 smithi164 conmon[29288]: 10T04:48:56.745933+0000 osd.4 (osd.4) 2320 2021-02-10T04:48:58.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:57 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:58.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:57 smithi164 conmon[29288]: cluster 2021-02-10T04:48:56 2021-02-10T04:48:58.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:57 smithi164 conmon[29288]: .745939+0000 osd.4 (osd.4) 2321 : 2021-02-10T04:48:58.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:57 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:58.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:57 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:48:58.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:57 smithi164 conmon[29288]: 10T04:48:56.745945+0000 osd.4 2021-02-10T04:48:58.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:57 smithi164 conmon[29288]: (osd.4) 2322 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:58.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:57 smithi164 conmon[29288]: 2021-02-10T04:48:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:58 smithi085 conmon[29319]: cluster 2021-02-10T04:48:57. 2021-02-10T04:48:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:58 smithi085 conmon[29319]: 759855+0000 osd.4 (osd.4) 2323 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:58 smithi085 conmon[29319]: cluster 2021-02-10T04:48 2021-02-10T04:48:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:58 smithi085 conmon[29319]: :57.759866+0000 osd.4 (osd.4) 2324 : cluster [WRN] 2021-02-10T04:48:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:58 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:58 smithi085 conmon[29319]: cluster 2021-02-10T04:48:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:58 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:48:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:58 smithi085 conmon[29319]: 48:57.759875+0000 osd.4 ( 2021-02-10T04:48:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:58 smithi085 conmon[29319]: osd.4) 2325 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:58 smithi085 conmon[29319]: 2021-02-10T04:48:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:58 smithi085 conmon[29319]: cluster 2021-02-10T04:48:58.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:58 smithi085 conmon[29319]: 2021-02-10T04:48:57 2021-02-10T04:48:58.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:58 smithi085 conmon[29319]: .759879+0000 osd.4 (osd.4) 2326 : 2021-02-10T04:48:58.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:58 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:58 smithi085 conmon[35961]: cluster 2021-02-10T04:48:57.759855+0000 2021-02-10T04:48:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:58 smithi085 conmon[35961]: osd.4 (osd.4) 2323 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:58 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:48:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:58 smithi085 conmon[35961]: 02-10T04:48:57.759866+0000 osd.4 (osd.4) 2324 : 2021-02-10T04:48:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:58 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:58 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:48:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:58 smithi085 conmon[35961]: 04:48:57.759875+0000 osd.4 (osd 2021-02-10T04:48:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:58 smithi085 conmon[35961]: .4) 2325 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:58 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:48:58.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:58 smithi085 conmon[35961]: -10T04:48:57.759879+0000 osd.4 ( 2021-02-10T04:48:58.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:58 smithi085 conmon[35961]: osd.4) 2326 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:59.144 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:48:58 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:48:59.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:58 smithi164 conmon[33677]: debug 2021-02-10T04:48:58.729+0000 7fbec19a1700 -1 osd.4 1085 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:48:59.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:58 smithi164 conmon[29288]: cluster 2021-02-10T04:48:57.759855+0000 osd.4 (osd. 2021-02-10T04:48:59.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:58 smithi164 conmon[29288]: 4) 2323 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:59.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:58 smithi164 conmon[29288]: cluster 2021-02-10T04:48:57.759866+0000 osd.4 (osd.4) 2324 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:58 smithi164 conmon[29288]: cluster 2021-02-10T04:48:57.759875+0000 osd.4 (osd.4) 2325 : 2021-02-10T04:48:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:58 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:58 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:48:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:58 smithi164 conmon[29288]: :48:57.759879+0000 osd.4 (osd.4) 2326 2021-02-10T04:48:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:58 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[29319]: cluster 2021-02-10T04:48:58.367381+0000 mgr.y (mgr.14140) 2021-02-10T04:48:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[29319]: 1113 : cluster [DBG] pgmap v1895: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:48:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[29319]: cluster 2021-02-10T04:48:58.665926+0000 mon.a (mon.0) 2179 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 272 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:48:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[29319]: cluster 2021-02-10T04:48: 2021-02-10T04:48:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[29319]: 58.730685+0000 osd.4 (osd.4) 2327 : cluster 2021-02-10T04:48:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:48:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[29319]: -10T04:48:58.730697+0000 osd.4 2021-02-10T04:48:59.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[29319]: (osd.4) 2328 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:59.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[29319]: cluster 2021-02-10T04:48:59.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[29319]: 2021-02-10T04:48:58. 2021-02-10T04:48:59.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[29319]: 730706+0000 osd.4 (osd.4) 2329 : cluster 2021-02-10T04:48:59.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:59.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[29319]: cluster 2021-02-10T04:48:59.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:48:59.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[29319]: 10T04:48:58.730714+0000 osd.4 ( 2021-02-10T04:48:59.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[29319]: osd.4) 2330 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:59.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[29319]: audit 2021-02-10T04:48:59.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[29319]: 2021-02-10T04:48 2021-02-10T04:48:59.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[29319]: :58.918061+0000 mgr.y (mgr.14140) 1114 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:48:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: cluster 2021-02-10T04:48:58.367381+0000 2021-02-10T04:48:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: mgr.y (mgr.14140) 1113 : cluster [DBG] pgmap v1895: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:48:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:48:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: -10T04:48:58.665926+0000 2021-02-10T04:48:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: mon.a (mon.0) 2179 : cluster 2021-02-10T04:48:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: [WRN] Health check update: 4 slow ops, oldest one blocked for 272 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:48:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:48:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: -10T04:48:58.730685+0000 osd.4 ( 2021-02-10T04:48:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: osd.4) 2327 : cluster [WRN] 2021-02-10T04:48:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:48:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:48:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: 04:48:58.730697+0000 osd.4 (osd.4) 2021-02-10T04:48:59.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: 2328 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:48:59.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: cluster 2021-02-10T04:48:59.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: 2021-02-10T04:48:59.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: 2021 2021-02-10T04:48:59.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: - 2021-02-10T04:48:59.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: 02 2021-02-10T04:48:59.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: - 2021-02-10T04:48:59.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: 10 2021-02-10T04:48:59.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: T 2021-02-10T04:48:59.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: 04:48:58.730706+0000 2021-02-10T04:48:59.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: osd.4 (osd.4) 2329 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:48:59.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: cluster 2021-02-10T04:48:59.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: 2021-02-10T04:48:59.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: 2021 2021-02-10T04:48:59.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: - 2021-02-10T04:48:59.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: 02-10T04:48:58.730714+0000 osd.4 (osd.4) 2330 2021-02-10T04:48:59.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:48:59.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: audit 2021-02-10T04:48:59.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: 2021-02-10T04:48:58.918061+0000 mgr.y (mgr 2021-02-10T04:48:59.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:48:59 smithi085 conmon[35961]: .14140) 1114 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:49:00.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:48:59 smithi164 conmon[33677]: debug 2021-02-10T04:48:59.722+0000 7fbec19a1700 -1 osd.4 1085 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:49:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:59 smithi164 conmon[29288]: cluster 2021-02-10T04:48:58.367381+0000 mgr.y (mgr.14140 2021-02-10T04:49:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:59 smithi164 conmon[29288]: ) 1113 : cluster [DBG] pgmap v1895: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:49:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:59 smithi164 conmon[29288]: cluster 2021-02-10T04:48:58.665926+0000 mon.a (mon.0) 2179 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 272 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:49:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:59 smithi164 conmon[29288]: cluster 2021-02-10T04:48 2021-02-10T04:49:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:59 smithi164 conmon[29288]: :58.730685+0000 osd.4 (osd.4) 2327 : cluster 2021-02-10T04:49:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:59 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:59 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:49:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:59 smithi164 conmon[29288]: 48:58.730697+0000 osd.4 (osd.4) 2328 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:59 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:49:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:59 smithi164 conmon[29288]: 04:48:58.730706+0000 osd.4 (osd.4) 2021-02-10T04:49:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:59 smithi164 conmon[29288]: 2329 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:59 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:49:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:59 smithi164 conmon[29288]: :48:58.730714+0000 osd.4 ( 2021-02-10T04:49:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:59 smithi164 conmon[29288]: osd.4) 2330 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:59 smithi164 conmon[29288]: 2021-02-10T04:49:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:59 smithi164 conmon[29288]: audit 2021-02-10T04:49:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:59 smithi164 conmon[29288]: 2021-02-10T04:48:58.918061+0000 mgr.y (mgr. 2021-02-10T04:49:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:48:59 smithi164 conmon[29288]: 14140) 1114 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:49:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:00 smithi085 conmon[29319]: cluster 2021-02-10T04:48:59.723348+0000 osd.4 ( 2021-02-10T04:49:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:00 smithi085 conmon[29319]: osd.4) 2331 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:00 smithi085 conmon[29319]: cluster 2021-02-10T04:48:59.723357+0000 osd.4 (osd.4) 2332 : cluster [WRN] 2021-02-10T04:49:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:00 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:00.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:00 smithi085 conmon[29319]: cluster 2021-02-10T04:49:00.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:00 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:49:00.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:00 smithi085 conmon[29319]: 10T04:48:59.723364+0000 osd.4 (osd. 2021-02-10T04:49:00.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:00 smithi085 conmon[29319]: 4) 2333 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:00.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:00 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:49:00.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:00 smithi085 conmon[29319]: 02-10T04:48:59.723372+0000 osd.4 (osd 2021-02-10T04:49:00.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:00 smithi085 conmon[29319]: .4) 2334 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:00.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:00 smithi085 conmon[35961]: cluster 2021-02-10T04:49:00.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:00 smithi085 conmon[35961]: 2021-02-10T04:48 2021-02-10T04:49:00.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:00 smithi085 conmon[35961]: :59.723348+0000 osd.4 (osd.4) 2331 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:00.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:00 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:49:00.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:00 smithi085 conmon[35961]: 04:48:59.723357+0000 osd.4 (osd 2021-02-10T04:49:00.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:00 smithi085 conmon[35961]: .4) 2332 : cluster 2021-02-10T04:49:00.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:00 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:00.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:00 smithi085 conmon[35961]: cluster 2021-02-10T04:48: 2021-02-10T04:49:00.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:00 smithi085 conmon[35961]: 59.723364+0000 osd.4 (osd.4) 2333 : cluster 2021-02-10T04:49:00.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:00 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:00.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:00 smithi085 conmon[35961]: cluster 2021-02-10T04:48:59 2021-02-10T04:49:00.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:00 smithi085 conmon[35961]: .723372+0000 osd.4 (osd.4) 2334 : cluster 2021-02-10T04:49:00.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:00 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:00 smithi164 conmon[29288]: cluster 2021-02-10T04:49:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:00 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:49:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:00 smithi164 conmon[29288]: 48:59.723348+0000 osd.4 (osd.4) 2331 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:00 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:49:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:00 smithi164 conmon[29288]: 02-10T04:48:59.723357 2021-02-10T04:49:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:00 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2332 : cluster [WRN] 2021-02-10T04:49:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:00 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:00 smithi164 conmon[29288]: cluster 2021-02-10T04:48 2021-02-10T04:49:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:00 smithi164 conmon[29288]: :59.723364+0000 osd.4 (osd.4) 2333 : 2021-02-10T04:49:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:00 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:00 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:49:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:00 smithi164 conmon[29288]: -02-10T04:48:59.723372+0000 osd.4 ( 2021-02-10T04:49:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:00 smithi164 conmon[29288]: osd.4) 2334 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:01.145 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:49:00 smithi164 conmon[33677]: debug 2021-02-10T04:49:01.145 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:49:00 smithi164 conmon[33677]: 2021-02-10T04:49:00.753+0000 7fbec19a1700 -1 osd.4 1085 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:49:01.145 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:49:00 smithi164 conmon[33677]: 2021-02-10T04:49:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:01 smithi085 conmon[29319]: cluster 2021-02-10T04:49:00.368351+0000 2021-02-10T04:49:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:01 smithi085 conmon[29319]: mgr.y (mgr.14140) 1115 : cluster [DBG] pgmap v1896: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:49:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:01 smithi085 conmon[29319]: cluster 2021-02-10T04:49:00.754044+0000 osd.4 (osd.4) 2335 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:01 smithi085 conmon[29319]: cluster 2021-02-10T04:49:00.754053+0000 osd.4 (osd.4) 2336 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:01 smithi085 conmon[29319]: cluster 2021-02-10T04:49:00.754060+0000 osd.4 (osd.4) 2337 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:01 smithi085 conmon[29319]: cluster 2021-02-10T04:49:00.754067+0000 osd.4 (osd.4) 2338 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:01.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:01 smithi085 conmon[35961]: cluster 2021-02-10T04:49:01.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:01 smithi085 conmon[35961]: 2021-02-10T04:49:01.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:01 smithi085 conmon[35961]: 2021-02-10T04:49:00. 2021-02-10T04:49:01.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:01 smithi085 conmon[35961]: 368351+0000 mgr.y (mgr.14140) 1115 : cluster [DBG] pgmap v1896: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:49:01.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:01 smithi085 conmon[35961]: cluster 2021-02-10T04:49 2021-02-10T04:49:01.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:01 smithi085 conmon[35961]: :00.754044+0000 osd.4 (osd.4) 2335 : cluster 2021-02-10T04:49:01.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:01 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:01.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:01 smithi085 conmon[35961]: cluster 2021-02-10T04:49 2021-02-10T04:49:01.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:01 smithi085 conmon[35961]: :00.754053+0000 osd.4 (osd.4) 2336 2021-02-10T04:49:01.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:01 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:01.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:01 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:49:01.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:01 smithi085 conmon[35961]: 10T04:49:00.754060+0000 osd.4 (osd 2021-02-10T04:49:01.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:01 smithi085 conmon[35961]: .4) 2337 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:01.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:01 smithi085 conmon[35961]: cluster 2021-02-10T04:49:01.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:01 smithi085 conmon[35961]: 2021-02-10T04:49:00.754067 2021-02-10T04:49:01.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:01 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2338 : cluster [WRN] 2021-02-10T04:49:01.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:01 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:02.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:49:01 smithi164 conmon[33677]: debug 2021-02-10T04:49:01.800+0000 7fbec19a1700 -1 osd.4 1085 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:49:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:01 smithi164 conmon[29288]: cluster 2021-02-10T04:49:00.368351+0000 mgr.y ( 2021-02-10T04:49:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:01 smithi164 conmon[29288]: mgr.14140) 1115 : cluster [DBG] pgmap v1896: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:49:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:01 smithi164 conmon[29288]: cluster 2021-02-10T04:49:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:01 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:49:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:01 smithi164 conmon[29288]: 49:00.754044+0000 osd.4 (osd.4) 2335 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:01 smithi164 conmon[29288]: cluster 2021-02-10T04:49:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:01 smithi164 conmon[29288]: 2021-02-10T04:49:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:01 smithi164 conmon[29288]: 2021-02-10T04:49:00. 2021-02-10T04:49:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:01 smithi164 conmon[29288]: 754053+0000 osd.4 (osd.4) 2336 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:01 smithi164 conmon[29288]: cluster 2021-02-10T04:49:00. 2021-02-10T04:49:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:01 smithi164 conmon[29288]: 754060+0000 osd.4 (osd.4) 2337 : cluster 2021-02-10T04:49:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:01 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:01 smithi164 conmon[29288]: cluster 2021-02-10T04:49:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:01 smithi164 conmon[29288]: 2021-02-10T04:49:00.754067+0000 osd.4 (osd.4) 2338 2021-02-10T04:49:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:01 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:02.699 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:49:02 smithi085 conmon[41540]: debug 2021-02-10T04:49:02.441+0000 7fb94a360700 -1 osd.0 pg_epoch: 1085 pg[1.0( empty local-lis/les=130/131 n=0 ec=19/19 lis/c=130/130 les/c/f=131/131/0 sis=130) [0,7,6] r=0 lpr=130 crt=0'0 mlcod 0'0 active+clean] scrubber pg(1.0) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6 (0x555d32c87340) 2021-02-10T04:49:02.954 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:02 smithi085 conmon[29319]: cluster 2021-02-10T04:49:02.954 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:02 smithi085 conmon[29319]: 2021-02-10T04:49:02.954 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:02 smithi085 conmon[29319]: 2021-02-10T04:49:01. 2021-02-10T04:49:02.955 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:02 smithi085 conmon[29319]: 800704+0000 osd.4 (osd.4) 2339 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:02.955 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:02 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:49:02.955 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:02 smithi085 conmon[29319]: 10T04:49:01.800725+0000 osd.4 (osd.4) 2021-02-10T04:49:02.955 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:02 smithi085 conmon[29319]: 2340 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:02.955 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:02 smithi085 conmon[29319]: cluster 2021-02-10T04:49:02.956 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:02 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:49:02.956 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:02 smithi085 conmon[29319]: T04:49:01.800733+0000 osd.4 ( 2021-02-10T04:49:02.956 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:02 smithi085 conmon[29319]: osd.4) 2341 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:02.956 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:02 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:49:02.956 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:02 smithi085 conmon[29319]: 02-10T04:49:01.800739+0000 osd.4 (osd. 2021-02-10T04:49:02.957 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:02 smithi085 conmon[29319]: 4) 2342 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:02.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:02 smithi085 conmon[35961]: cluster 2021-02-10T04:49:01.800704 2021-02-10T04:49:02.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:02 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2339 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:02.957 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:02 smithi085 conmon[35961]: cluster 2021-02-10T04:49:01.800725+0000 2021-02-10T04:49:02.958 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:02 smithi085 conmon[35961]: osd.4 (osd.4) 2340 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:02.958 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:02 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:49:02.958 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:02 smithi085 conmon[35961]: 02-10T04:49:01.800733+0000 osd.4 (osd 2021-02-10T04:49:02.958 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:02 smithi085 conmon[35961]: .4) 2341 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:02.958 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:02 smithi085 conmon[35961]: cluster 2021-02-10T04:49:02.959 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:02 smithi085 conmon[35961]: 2021 2021-02-10T04:49:02.959 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:02 smithi085 conmon[35961]: - 2021-02-10T04:49:02.959 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:02 smithi085 conmon[35961]: 02-10T04:49 2021-02-10T04:49:02.959 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:02 smithi085 conmon[35961]: :01.800739+0000 osd.4 (osd.4) 2342 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:03.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:49:02 smithi164 conmon[33677]: debug 2021-02-10T04:49:02.814+0000 7fbec19a1700 -1 osd.4 1085 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:49:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:02 smithi164 conmon[29288]: cluster 2021-02-10T04:49:01.800704+0000 osd.4 (osd.4 2021-02-10T04:49:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:02 smithi164 conmon[29288]: ) 2339 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:02 smithi164 conmon[29288]: cluster 2021-02-10T04:49:01.800725+0000 osd.4 (osd.4) 2340 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:02 smithi164 conmon[29288]: cluster 2021-02-10T04:49:01. 2021-02-10T04:49:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:02 smithi164 conmon[29288]: 800733+0000 osd.4 (osd.4) 2341 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:02 smithi164 conmon[29288]: cluster 2021-02-10T04:49:01.800739+0000 osd.4 (osd.4 2021-02-10T04:49:03.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:02 smithi164 conmon[29288]: ) 2342 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:03.644 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:49:03 smithi164 conmon[40892]: debug 2021-02-10T04:49:03.182+0000 7f3033cc2700 -1 osd.6 pg_epoch: 1085 pg[6.22( empty local-lis/les=1046/1047 n=0 ec=1044/59 lis/c=1046/1046 les/c/f=1047/1047/0 sis=1046) [6,3,1] r=0 lpr=1046 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.22) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x557a7375fce0) 2021-02-10T04:49:03.973 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[29319]: cluster 2021-02-10T04:49:03.973 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[29319]: 2021-02-10T04:49:03.973 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[29319]: 2021 2021-02-10T04:49:03.973 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[29319]: - 2021-02-10T04:49:03.973 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[29319]: 02 2021-02-10T04:49:03.973 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[29319]: - 2021-02-10T04:49:03.973 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[29319]: 10T04:49:02.368941+0000 2021-02-10T04:49:03.973 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[29319]: mgr.y (mgr.14140) 1116 : cluster [DBG] pgmap v1897: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:49:03.974 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[29319]: cluster 2021-02-10T04:49:03.974 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:49:03.974 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[29319]: 04:49:02.815304+0000 osd.4 (osd. 2021-02-10T04:49:03.974 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[29319]: 4) 2343 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:03.974 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[29319]: cluster 2021-02-10T04:49:03.974 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[29319]: 2021-02-10T04:49:03.974 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[29319]: 2021-02-10T04:49 2021-02-10T04:49:03.974 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[29319]: :02.815312+0000 osd.4 (osd.4) 2344 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:03.974 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:49:03.974 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[29319]: 49:02.815316+0000 osd.4 (osd.4) 2345 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:03.974 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[29319]: cluster 2021-02-10T04:49:02.815332+0000 2021-02-10T04:49:03.974 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[29319]: osd.4 (osd.4) 2346 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:03.975 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[35961]: cluster 2021-02-10T04:49:02.368941+0000 mgr.y (mgr 2021-02-10T04:49:03.975 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[35961]: .14140) 1116 : cluster [DBG] pgmap v1897: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:49:03.975 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[35961]: cluster 2021-02-10T04:49: 2021-02-10T04:49:03.975 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[35961]: 02.815304+0000 osd.4 (osd.4) 2343 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:03.975 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:49:03.975 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[35961]: 02-10T04:49:02.815312+0000 osd.4 ( 2021-02-10T04:49:03.975 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[35961]: osd.4) 2344 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:03.975 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:49:03.975 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[35961]: 02-10T04:49:02.815316+0000 osd.4 ( 2021-02-10T04:49:03.975 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[35961]: osd.4) 2345 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:03.975 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:49:03.975 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[35961]: 49:02.815332+0000 osd.4 (osd.4) 2346 2021-02-10T04:49:03.976 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:04.145 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:49:03 smithi164 conmon[33677]: debug 2021-02-10T04:49:03.776+0000 7fbec19a1700 -1 osd.4 1085 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:49:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:03 smithi164 conmon[29288]: cluster 2021-02-10T04:49:02.368941+0000 mgr.y (mgr 2021-02-10T04:49:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:03 smithi164 conmon[29288]: .14140) 1116 : cluster [DBG] pgmap v1897: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:49:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:03 smithi164 conmon[29288]: cluster 2021-02-10T04:49:02.815304+0000 osd.4 (osd.4) 2343 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:03 smithi164 conmon[29288]: cluster 2021-02-10T04:49:02.815312+0000 osd.4 (osd.4) 2344 : cluster [WRN] 2021-02-10T04:49:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:03 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:03 smithi164 conmon[29288]: cluster 2021-02-10T04:49:02.815316 2021-02-10T04:49:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:03 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2345 : cluster [WRN] 2021-02-10T04:49:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:03 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:04.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:03 smithi164 conmon[29288]: cluster 2021-02-10T04:49:02.815332 2021-02-10T04:49:04.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:03 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2346 : cluster [WRN] 2021-02-10T04:49:04.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:03 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:04.256 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:49:04.441 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:49:03 smithi085 conmon[45443]: debug 2021-02-10T04:49:03.969+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:49:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:04 smithi164 conmon[29288]: audit 2021-02-10T04:49:03.746579 2021-02-10T04:49:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:04 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2180 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:49:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:04 smithi164 conmon[29288]: audit 2021-02-10T04:49:03.746837+0000 mon.a (mon.0) 2181 : audit 2021-02-10T04:49:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:04 smithi164 conmon[29288]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:49:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:04 smithi164 conmon[29288]: cluster 2021-02-10T04:49:03.776850+0000 2021-02-10T04:49:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:04 smithi164 conmon[29288]: osd.4 (osd.4) 2347 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:04 smithi164 conmon[29288]: cluster 2021-02-10T04:49:03.776859+0000 osd.4 (osd 2021-02-10T04:49:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:04 smithi164 conmon[29288]: .4) 2348 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:05.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:04 smithi164 conmon[29288]: cluster 2021-02-10T04:49:03.776865 2021-02-10T04:49:05.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:04 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2349 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:05.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:04 smithi164 conmon[29288]: cluster 2021-02-10T04:49:03.776871+0000 osd.4 2021-02-10T04:49:05.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:04 smithi164 conmon[29288]: (osd.4) 2350 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:05.145 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:49:04 smithi164 conmon[33677]: debug 2021-02-10T04:49:04.766+0000 7fbec19a1700 -1 osd.4 1085 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:49:05.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[35961]: audit 2021-02-10T04:49:03 2021-02-10T04:49:05.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[35961]: .746579+0000 mon.a (mon.0) 2180 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:49:05.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:49:05.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[35961]: :49:03.746837+0000 mon.a (mon.0) 2181 : 2021-02-10T04:49:05.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[35961]: audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:49:05.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[35961]: cluster 2021-02-10T04:49:03.776850 2021-02-10T04:49:05.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2347 : cluster [WRN] 2021-02-10T04:49:05.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:05.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[35961]: cluster 2021-02-10T04:49:05.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[35961]: 2021-02-10T04:49:03.776859+0000 osd.4 (osd. 2021-02-10T04:49:05.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[35961]: 4) 2348 : cluster 2021-02-10T04:49:05.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:05.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:49:05.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[35961]: 04:49:03.776865+0000 osd.4 (osd 2021-02-10T04:49:05.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[35961]: .4) 2349 2021-02-10T04:49:05.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[35961]: : cluster 2021-02-10T04:49:05.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:05.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[35961]: cluster 2021-02-10T04:49:05.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[35961]: 2021-02 2021-02-10T04:49:05.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[35961]: -10T04:49:03.776871+0000 osd.4 2021-02-10T04:49:05.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[35961]: (osd.4) 2350 : cluster [WRN] 2021-02-10T04:49:05.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:05.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[29319]: audit 2021-02-10T04:49:03.746579+0000 2021-02-10T04:49:05.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[29319]: mon.a (mon.0) 2180 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:49:05.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[29319]: audit 2021-02-10T04:49:03.746837+0000 mon.a (mon.0) 2181 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:49:05.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[29319]: cluster 2021-02-10T04:49:03.776850+0000 osd.4 (osd.4) 2347 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:05.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[29319]: cluster 2021-02-10T04:49:03.776859+0000 osd.4 (osd.4) 2348 : 2021-02-10T04:49:05.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:05.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[29319]: cluster 2021-02-10T04:49:03.776865+0000 2021-02-10T04:49:05.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[29319]: osd.4 (osd.4) 2349 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:05.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:49:05.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:04 smithi085 conmon[29319]: -02-10T04:49:03.776871+0000 osd.4 (osd.4) 2350 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:06.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:49:05 smithi164 conmon[33677]: debug 2021-02-10T04:49:05.788+0000 7fbec19a1700 -1 osd.4 1085 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:49:06.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:05 smithi164 conmon[29288]: cluster 2021-02-10T04:49:04.369586 2021-02-10T04:49:06.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:05 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1117 : cluster [DBG] pgmap v1898: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:49:06.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:05 smithi164 conmon[29288]: cluster 2021-02-10T04:49:04.767585+0000 osd.4 (osd.4) 2351 2021-02-10T04:49:06.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:05 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:06.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:05 smithi164 conmon[29288]: cluster 2021-02-10T04:49:06.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:05 smithi164 conmon[29288]: 2021-02-10T04:49:06.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:05 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:49:06.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:05 smithi164 conmon[29288]: 49:04.767591+0000 osd.4 (osd.4) 2352 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:06.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:05 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:49:06.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:05 smithi164 conmon[29288]: 04:49:04.767603+0000 osd.4 (osd.4) 2021-02-10T04:49:06.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:05 smithi164 conmon[29288]: 2353 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:06.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:05 smithi164 conmon[29288]: cluster 2021-02-10T04:49 2021-02-10T04:49:06.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:05 smithi164 conmon[29288]: :04.767607+0000 osd.4 (osd.4) 2354 : 2021-02-10T04:49:06.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:05 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:06.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[29319]: cluster 2021-02-10T04:49:04.369586+0000 mgr.y ( 2021-02-10T04:49:06.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[29319]: mgr.14140) 1117 : cluster [DBG] pgmap v1898: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:49:06.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[29319]: cluster 2021-02-10T04:49:04.767585+0000 osd.4 (osd.4) 2351 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:06.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:49:06.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[29319]: 10T04:49:04.767591+0000 osd.4 (osd.4) 2352 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:06.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[29319]: cluster 2021-02-10T04:49:06.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:49:06.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[29319]: 10T04:49:04.767603+0000 osd.4 (osd. 2021-02-10T04:49:06.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[29319]: 4) 2353 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:06.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[29319]: cluster 2021-02-10T04:49:06.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[29319]: 2021-02-10T04:49:06.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[29319]: 2021-02-10T04:49:04. 2021-02-10T04:49:06.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[29319]: 767607+0000 osd.4 (osd.4) 2354 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:06.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[35961]: cluster 2021-02-10T04:49:06.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[35961]: 2021-02-10T04:49:06.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[35961]: 2021 2021-02-10T04:49:06.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[35961]: - 2021-02-10T04:49:06.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[35961]: 02-10T04:49:04. 2021-02-10T04:49:06.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[35961]: 369586+0000 mgr.y (mgr.14140) 1117 : cluster [DBG] pgmap v1898: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:49:06.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:49:06.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[35961]: 02-10T04:49:04.767585+0000 osd.4 ( 2021-02-10T04:49:06.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[35961]: osd.4) 2351 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:06.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:49:06.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[35961]: 04:49:04.767591+0000 2021-02-10T04:49:06.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[35961]: osd.4 (osd.4) 2352 2021-02-10T04:49:06.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:06.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[35961]: cluster 2021-02-10T04:49:06.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:49:06.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[35961]: 49:04.767603+0000 osd.4 2021-02-10T04:49:06.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[35961]: (osd.4) 2353 : cluster 2021-02-10T04:49:06.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:06.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[35961]: cluster 2021-02-10T04:49:06.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:49:06.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[35961]: 04:49: 2021-02-10T04:49:06.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[35961]: 04.767607+0000 osd.4 (osd 2021-02-10T04:49:06.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[35961]: .4) 2354 : cluster [WRN] 2021-02-10T04:49:06.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:05 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:07.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:49:06 smithi164 conmon[33677]: debug 2021-02-10T04:49:06.800+0000 7fbec19a1700 -1 osd.4 1085 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:49:07.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:06 smithi164 conmon[29288]: cluster 2021-02-10T04:49:05.789318+0000 osd.4 (osd. 2021-02-10T04:49:07.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:06 smithi164 conmon[29288]: 4) 2355 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:07.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:06 smithi164 conmon[29288]: cluster 2021-02-10T04:49:05.789326+0000 osd.4 (osd.4) 2356 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:07.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:06 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:49:07.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:06 smithi164 conmon[29288]: -02-10T04:49:05.789333+0000 osd.4 (osd.4) 2357 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:07.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:06 smithi164 conmon[29288]: cluster 2021-02-10T04:49:05.789339+0000 osd.4 (osd.4 2021-02-10T04:49:07.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:06 smithi164 conmon[29288]: ) 2358 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:07.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:06 smithi085 conmon[35961]: cluster 2021-02-10T04:49:07.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:06 smithi085 conmon[35961]: 2021-02-10T04:49:07.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:06 smithi085 conmon[35961]: 2021 2021-02-10T04:49:07.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:06 smithi085 conmon[35961]: - 2021-02-10T04:49:07.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:06 smithi085 conmon[35961]: 02 2021-02-10T04:49:07.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:06 smithi085 conmon[35961]: - 2021-02-10T04:49:07.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:06 smithi085 conmon[35961]: 10T04:49:05.789318+0000 2021-02-10T04:49:07.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:06 smithi085 conmon[35961]: osd.4 (osd.4) 2355 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:07.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:06 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:49:07.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:06 smithi085 conmon[35961]: 49:05.789326+0000 osd.4 (osd.4) 2356 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:07.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:06 smithi085 conmon[35961]: 2021-02-10T04:49:07.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:06 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:49:07.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:06 smithi085 conmon[35961]: 02-10T04:49:05.789333+0000 osd.4 (osd.4) 2357 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:07.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:06 smithi085 conmon[35961]: 2021-02-10T04:49:07.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:06 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:49:07.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:06 smithi085 conmon[35961]: 10T04:49:05.789339+0000 osd.4 (osd.4) 2358 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:07.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:06 smithi085 conmon[35961]: 2021-02-10T04:49:07.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:06 smithi085 conmon[29319]: cluster 2021-02-10T04:49:07.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:06 smithi085 conmon[29319]: 2021-02-10T04:49:07.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:06 smithi085 conmon[29319]: 2021-02-10T04:49:05.789318 2021-02-10T04:49:07.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:06 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2355 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:07.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:06 smithi085 conmon[29319]: cluster 2021-02-10T04:49:05.789326+0000 osd.4 2021-02-10T04:49:07.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:06 smithi085 conmon[29319]: (osd.4) 2356 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:07.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:06 smithi085 conmon[29319]: cluster 2021-02-10T04:49:05.789333+0000 osd.4 (osd.4) 2357 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:07.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:06 smithi085 conmon[29319]: cluster 2021-02-10T04:49:05.789339+0000 osd.4 (osd.4) 2358 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:08.018 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[29319]: cluster 2021-02-10T04:49:08.018 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[29319]: 2021-02-10T04:49:08.018 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[29319]: 2021 2021-02-10T04:49:08.018 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[29319]: - 2021-02-10T04:49:08.018 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[29319]: 02 2021-02-10T04:49:08.018 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[29319]: - 2021-02-10T04:49:08.018 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[29319]: 10 2021-02-10T04:49:08.018 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[29319]: T 2021-02-10T04:49:08.019 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[29319]: 04 2021-02-10T04:49:08.019 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[29319]: : 2021-02-10T04:49:08.019 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[29319]: 49 2021-02-10T04:49:08.019 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[29319]: : 2021-02-10T04:49:08.019 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[29319]: 06.370229+0000 mgr.y (mgr.14140) 1118 2021-02-10T04:49:08.019 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[29319]: : cluster [DBG] pgmap v1899: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:08.019 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[29319]: cluster 2021-02-10T04:49:06 2021-02-10T04:49:08.019 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[29319]: .801318+0000 osd.4 (osd.4) 2359 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:08.019 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:49:08.019 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[29319]: :49:06.801323+0000 osd.4 (osd.4) 2360 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:08.019 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:49:08.019 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[29319]: :49:06.801327+0000 osd.4 (osd.4 2021-02-10T04:49:08.020 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[29319]: ) 2361 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:08.020 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:49:08.020 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[29319]: T04:49:06.801330+0000 osd.4 ( 2021-02-10T04:49:08.020 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[29319]: osd.4) 2362 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:08.021 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: cluster 2021-02-10T04:49:08.021 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:49:08.021 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: 04:49:06.370229+0000 mgr.y (mgr.14140) 1118 : cluster [DBG] pgmap v1899: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:08.021 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: cluster 2021-02-10T04:49:08.021 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: 2021-02-10T04:49:08.021 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: 2021 2021-02-10T04:49:08.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: - 2021-02-10T04:49:08.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: 02 2021-02-10T04:49:08.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: - 2021-02-10T04:49:08.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: 10 2021-02-10T04:49:08.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: T 2021-02-10T04:49:08.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: 04 2021-02-10T04:49:08.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: : 2021-02-10T04:49:08.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: 49 2021-02-10T04:49:08.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: : 2021-02-10T04:49:08.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: 06 2021-02-10T04:49:08.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: . 2021-02-10T04:49:08.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: 801318 2021-02-10T04:49:08.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: +0000 2021-02-10T04:49:08.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: 2021-02-10T04:49:08.023 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: osd.4 2021-02-10T04:49:08.023 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: ( 2021-02-10T04:49:08.023 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: osd 2021-02-10T04:49:08.023 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: . 2021-02-10T04:49:08.023 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: 4 2021-02-10T04:49:08.023 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: ) 2021-02-10T04:49:08.023 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: 2359 2021-02-10T04:49:08.023 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: : 2021-02-10T04:49:08.023 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: cluster 2021-02-10T04:49:08.023 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: 2021-02-10T04:49:08.023 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: [WRN] 2021-02-10T04:49:08.024 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: 2021-02-10T04:49:08.024 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:08.024 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: 2021-02-10T04:49:08.024 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: cluster 2021-02-10T04:49:08.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: 2021-02-10T04:49:08.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: 2021 2021-02-10T04:49:08.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: -02-10T04:49: 2021-02-10T04:49:08.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: 06.801323+0000 osd.4 (osd.4) 2360 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:08.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:49:08.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: 10T04:49:06.801327+0000 osd.4 2021-02-10T04:49:08.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: (osd.4) 2361 : cluster 2021-02-10T04:49:08.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:08.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: cluster 2021-02-10T04:49:06. 2021-02-10T04:49:08.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: 801330+0000 osd.4 (osd.4) 2362 : cluster [WRN] 2021-02-10T04:49:08.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:07 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:08.127 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:49:08.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:49:07 smithi164 conmon[33677]: debug 2021-02-10T04:49:07.812+0000 7fbec19a1700 -1 osd.4 1085 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:49:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:07 smithi164 conmon[29288]: cluster 2021-02-10T04:49:06.370229 2021-02-10T04:49:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:07 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1118 : cluster [DBG] pgmap v1899: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:07 smithi164 conmon[29288]: cluster 2021-02-10T04:49:06.801318+0000 osd.4 (osd.4) 2359 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:07 smithi164 conmon[29288]: cluster 2021-02-10T04:49:06.801323+0000 osd.4 (osd.4) 2360 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:08.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:07 smithi164 conmon[29288]: cluster 2021-02-10T04:49:06.801327+0000 osd.4 (osd.4) 2361 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:08.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:07 smithi164 conmon[29288]: cluster 2021-02-10T04:49:06.801330+0000 osd.4 (osd.4) 2362 : cluster 2021-02-10T04:49:08.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:07 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:08.153 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:49:08.566 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.0 dump_blocked_ops 2021-02-10T04:49:08.720 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:49:08.856 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:49:08.892 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[52836]: debug 2021-02-10T04:49:08.892 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[52836]: 2021-02-10T04:49:08.639+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:49:08.893 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[29319]: cluster 2021-02-10T04:49:08.893 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[29319]: 2021-02-10T04:49 2021-02-10T04:49:08.893 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[29319]: :07.813107+0000 osd.4 (osd.4) 2363 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:08.893 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:49:08.893 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[29319]: -10T04:49:07.813116+0000 osd.4 2021-02-10T04:49:08.894 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[29319]: (osd.4) 2364 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:08.894 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[29319]: cluster 2021-02-10T04:49:08.894 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[29319]: 2021-02-10T04:49:07.813123+0000 2021-02-10T04:49:08.894 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[29319]: osd.4 (osd.4) 2365 : cluster [WRN] 2021-02-10T04:49:08.894 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:08.895 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:49:08.895 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[29319]: 02-10T04:49: 2021-02-10T04:49:08.895 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[29319]: 07.813130+0000 osd.4 (osd 2021-02-10T04:49:08.895 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[29319]: .4) 2366 : cluster [WRN] 2021-02-10T04:49:08.896 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:08.896 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[29319]: cluster 2021-02-10T04:49:08.896 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[29319]: 2021-02-10T04:49:08.435589+0000 2021-02-10T04:49:08.896 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[29319]: mon.a (mon.0) 2182 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 277 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:49:08.896 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[29319]: 2021-02-10T04:49:08.897 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[35961]: cluster 2021-02-10T04:49:07.813107+0000 2021-02-10T04:49:08.897 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[35961]: osd.4 (osd.4) 2363 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:08.897 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:49:08.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[35961]: 10T04:49:07.813116+0000 osd.4 (osd.4) 2364 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:08.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[35961]: 2021-02-10T04:49:08.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:49:08.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[35961]: 02-10T04:49:07.813123+0000 osd.4 (osd.4) 2021-02-10T04:49:08.898 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[35961]: 2365 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:08.899 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:49:08.899 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[35961]: 04:49:07.813130+0000 osd.4 (osd 2021-02-10T04:49:08.899 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[35961]: .4) 2366 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:08.899 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[35961]: cluster 2021-02-10T04:49:08.899 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[35961]: 2021-02-10T04:49:08. 2021-02-10T04:49:08.900 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[35961]: 435589+0000 mon.a (mon.0) 2182 : cluster 2021-02-10T04:49:08.900 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:08 smithi085 conmon[35961]: [WRN] Health check update: 4 slow ops, oldest one blocked for 277 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:49:08.906 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:49:08.932 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:49:09.096 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:49:09.144 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:49:08 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:49:09.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:08 smithi164 conmon[29288]: cluster 2021-02-10T04:49:07.813107+0000 osd.4 (osd.4) 2363 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:09.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:08 smithi164 conmon[29288]: 2021-02-10T04:49:09.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:08 smithi164 conmon[29288]: cluster 2021-02-10T04:49:07.813116+0000 osd.4 (osd.4) 2364 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:09.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:08 smithi164 conmon[29288]: cluster 2021-02-10T04:49:07.813123+0000 osd.4 (osd.4) 2365 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:09.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:08 smithi164 conmon[29288]: cluster 2021-02-10T04:49:07.813130+0000 osd.4 (osd.4) 2366 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:09.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:08 smithi164 conmon[29288]: cluster 2021-02-10T04:49:08.435589+0000 mon.a (mon.0) 2182 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 277 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:49:09.145 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:49:08 smithi164 conmon[33677]: debug 2021-02-10T04:49:08.849+0000 7fbec19a1700 -1 osd.4 1086 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:49:09.192 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[41540]: debug 2021-02-10T04:49:09.035+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:49:09.198 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:49:09.241 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:49:09.348 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:49:09.454 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:49:09.495 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:49:09.506 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:49:09.541 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:49:09.578 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:49:09.607 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:49:09.643 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[37263]: debug 2021-02-10T04:49:09.303+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:49:09.644 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:49:09.678 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:49:09.708 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:49:09.746 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:49:09.748 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:49:09.751 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:49:09.751 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:49:09.751 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:49:09.751 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:49:09.751 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:49:09.751 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:49:09.751 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:49:09.751 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:49:09.855 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:49:09.890 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:49:09.957 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[52836]: debug 2021-02-10T04:49:09.703+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:49:09.958 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: cluster 2021-02-10T04:49:09.958 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: 2021-02-10T04:49:09.958 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: 2021 2021-02-10T04:49:09.958 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: - 2021-02-10T04:49:09.959 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: 02 2021-02-10T04:49:09.959 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: - 2021-02-10T04:49:09.959 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: 10 2021-02-10T04:49:09.959 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: T 2021-02-10T04:49:09.959 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: 04 2021-02-10T04:49:09.959 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: : 2021-02-10T04:49:09.959 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: 49 2021-02-10T04:49:09.959 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: : 2021-02-10T04:49:09.959 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: 08 2021-02-10T04:49:09.959 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: . 2021-02-10T04:49:09.959 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: 206982 2021-02-10T04:49:09.959 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: +0000 2021-02-10T04:49:09.959 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: 2021-02-10T04:49:09.960 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: osd.5 (osd.5) 139 : cluster [DBG] 2021-02-10T04:49:09.960 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: purged_snaps scrub starts 2021-02-10T04:49:09.960 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:49:09.960 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: 10T04:49:08.210740+0000 osd.5 (osd 2021-02-10T04:49:09.960 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: .5) 140 : cluster [DBG] 2021-02-10T04:49:09.960 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: purged_snaps scrub ok 2021-02-10T04:49:09.960 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: cluster 2021-02-10T04:49:09.960 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:49:09.960 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: :49:08.370886+0000 mgr.y (mgr.14140) 2021-02-10T04:49:09.960 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: 1119 : cluster [DBG] pgmap v1900: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 875 B/s rd, 0 op/s 2021-02-10T04:49:09.960 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:49:09.960 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: -10T04:49:08.731716+0000 mon.a 2021-02-10T04:49:09.960 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: (mon.0) 2183 : cluster [DBG] osdmap e1086: 8 total, 8 up, 7 in 2021-02-10T04:49:09.961 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: cluster 2021-02-10T04:49:09.961 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:49:09.961 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: T04:49:08.849710+0000 osd.4 ( 2021-02-10T04:49:09.961 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: osd.4) 2367 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:09.961 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:49:09.961 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: -02-10T04:49:08.849725+0000 2021-02-10T04:49:09.961 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: osd.4 (osd.4) 2368 : cluster [WRN] 2021-02-10T04:49:09.961 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:09.961 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: cluster 2021-02-10T04:49:08 2021-02-10T04:49:09.961 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: .849733+0000 osd.4 (osd.4) 2369 : cluster 2021-02-10T04:49:09.961 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:09.961 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: cluster 2021-02-10T04:49: 2021-02-10T04:49:09.961 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: 08.849740+0000 osd.4 (osd.4) 2370 : cluster 2021-02-10T04:49:09.962 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:09.962 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: audit 2021-02-10T04:49:09.962 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: 2021-02-10T04:49:08 2021-02-10T04:49:09.962 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: .930023+0000 mgr.y (mgr.14140) 1120 : audit 2021-02-10T04:49:09.962 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:49:09.962 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: audit 2021-02-10T 2021-02-10T04:49:09.962 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: 04:49:09.456878+0000 mon.c (mon.1 2021-02-10T04:49:09.962 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[29319]: ) 80 : audit [DBG] from='client.? 172.21.15.85:0/3500917410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:49:09.962 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: cluster 2021-02-10T04:49:08.206982+0000 2021-02-10T04:49:09.963 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: osd.5 (osd.5) 139 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:49:09.963 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: cluster 2021-02-10T04:49:08 2021-02-10T04:49:09.963 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: .210740+0000 osd.5 (osd.5) 140 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:49:09.963 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:49:09.963 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: 02-10T04:49:08.370886+0000 mgr.y ( 2021-02-10T04:49:09.963 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: mgr.14140) 1119 : cluster [DBG] 2021-02-10T04:49:09.963 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: pgmap v1900: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 875 B/s rd, 0 op/s 2021-02-10T04:49:09.963 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:49:09.963 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: 02-10T04:49: 2021-02-10T04:49:09.964 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: 08.731716+0000 mon.a (mon. 2021-02-10T04:49:09.964 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: 0) 2183 : cluster [DBG] 2021-02-10T04:49:09.964 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: osdmap e1086: 8 total, 8 up, 7 in 2021-02-10T04:49:09.964 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:49:09.964 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: 10T04:49:08. 2021-02-10T04:49:09.964 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: 849710+0000 osd.4 (osd.4) 2367 2021-02-10T04:49:09.964 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:09.964 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: cluster 2021-02-10T04:49:08 2021-02-10T04:49:09.964 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: .849725+0000 osd.4 (osd.4) 2368 : cluster 2021-02-10T04:49:09.964 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:09.964 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: cluster 2021-02-10T04:49:09.964 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: 2021-02-10T04:49:08.849733+0000 2021-02-10T04:49:09.965 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: osd.4 (osd.4) 2369 : cluster [WRN] 2021-02-10T04:49:09.965 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:09.965 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: cluster 2021-02-10T04:49:09.965 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: 2021-02-10T04:49:08.849740+0000 2021-02-10T04:49:09.965 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: osd.4 (osd.4) 2370 2021-02-10T04:49:09.965 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:09.965 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: audit 2021 2021-02-10T04:49:09.966 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: -02-10T04:49 2021-02-10T04:49:09.966 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: :08.930023+0000 mgr.y (mgr 2021-02-10T04:49:09.966 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: .14140) 1120 : audit [DBG] 2021-02-10T04:49:09.966 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:49:09.966 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:49:09.966 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: 10T04:49:09. 2021-02-10T04:49:09.966 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: 456878+0000 mon.c (mon.1) 2021-02-10T04:49:09.966 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:09 smithi085 conmon[35961]: 80 : audit [DBG] from='client.? 172.21.15.85:0/3500917410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:49:10.006 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:49:10.011 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: cluster 2021-02-10T04:49:08.206982+0000 osd.5 (osd 2021-02-10T04:49:10.011 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: .5) 139 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:49:10.011 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: cluster 2021-02-10T04:49:08.210740+0000 osd.5 (osd. 2021-02-10T04:49:10.011 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: 5) 140 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:49:10.012 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: cluster 2021-02-10T04:49:08 2021-02-10T04:49:10.012 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: .370886+0000 mgr.y (mgr.14140) 1119 : cluster [DBG] 2021-02-10T04:49:10.012 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: pgmap v1900: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 875 B/s rd, 0 op/s 2021-02-10T04:49:10.012 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:49:10.012 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: -02-10T04:49:08 2021-02-10T04:49:10.012 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: .731716+0000 mon.a (mon.0) 2183 : 2021-02-10T04:49:10.013 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: cluster [DBG] osdmap e1086: 8 total, 8 up, 7 in 2021-02-10T04:49:10.013 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:49:10.013 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: 02-10T04:49:08.849710+0000 osd.4 2021-02-10T04:49:10.013 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: (osd.4) 2367 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:49:10.013 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:49:10.013 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: -10T04:49:08.849725+0000 osd.4 (osd 2021-02-10T04:49:10.014 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: .4) 2368 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:49:10.014 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: cluster 2021-02-10T04:49:10.014 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:49:10.014 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: :49:08.849733+0000 osd.4 2021-02-10T04:49:10.014 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: (osd.4) 2369 : cluster 2021-02-10T04:49:10.015 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:49:10.015 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:49:10.015 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: 02-10T04:49:08.849740+0000 2021-02-10T04:49:10.015 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: osd.4 (osd.4) 2370 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:49:10.015 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: 2021-02-10T04:49:10.015 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:49:10.016 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: 02-10T04:49:08.930023+0000 mgr.y 2021-02-10T04:49:10.016 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: (mgr.14140) 1120 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:49:10.016 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:49:10.016 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: 02-10T04:49:09.456878+0000 2021-02-10T04:49:10.016 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[29288]: mon.c (mon.1) 80 : audit [DBG] from='client.? 172.21.15.85:0/3500917410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:49:10.016 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:49:09 smithi164 conmon[44544]: debug 2021-02-10T04:49:09.977+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:49:10.117 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:49:10.146 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:49:10.591 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset nodeep-scrub 2021-02-10T04:49:10.746 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.0 to 1 (10000) 2021-02-10T04:49:10.910 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:49:10.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:10 smithi085 conmon[29319]: audit 2021-02-10T04:49:09.806747+0000 mon.c (mon.1) 81 : audit [INF] from='client.? 172.21.15.85:0/531565303' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T04:49:10.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:10 smithi085 conmon[29319]: 2021-02-10T04:49:10.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:10 smithi085 conmon[29319]: audit 2021-02-10T04:49:09.807179+0000 mon.a (mon.0) 2184 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T04:49:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:10 smithi085 conmon[35961]: audit 2021-02-10T04:49:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:10 smithi085 conmon[35961]: 2021-02-10T04:49:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:10 smithi085 conmon[35961]: 2021 2021-02-10T04:49:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:10 smithi085 conmon[35961]: -02-10T04:49:09. 2021-02-10T04:49:10.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:10 smithi085 conmon[35961]: 806747+0000 mon.c (mon.1) 81 : audit [INF] from='client.? 172.21.15.85:0/531565303' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T04:49:10.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:10 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:49:10.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:10 smithi085 conmon[35961]: :49:09.807179+0000 mon.a (mon.0 2021-02-10T04:49:10.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:10 smithi085 conmon[35961]: ) 2184 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T04:49:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:10 smithi164 conmon[29288]: audit 2021-02-10T04:49:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:10 smithi164 conmon[29288]: 2021-02-10T04:49 2021-02-10T04:49:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:10 smithi164 conmon[29288]: :09.806747+0000 mon.c (mon.1) 81 : audit [INF] from='client.? 172.21.15.85:0/531565303' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T04:49:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:10 smithi164 conmon[29288]: audit 2021-02-10T04:49:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:10 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:49:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:10 smithi164 conmon[29288]: 49:09.807179+0000 mon.a (mon.0) 2184 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T04:49:12.083 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:11 smithi164 conmon[29288]: cluster 2021-02-10T04:49:10.371703+0000 2021-02-10T04:49:12.083 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:11 smithi164 conmon[29288]: mgr.y (mgr.14140) 1121 : cluster [DBG] pgmap v1902: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:12.083 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:11 smithi164 conmon[29288]: audit 2021-02-10T04:49:10.734760+0000 mon.a (mon.0) 2185 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2021-02-10T04:49:12.083 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:11 smithi164 conmon[29288]: cluster 2021-02-10T04:49:10.734834+0000 mon.a (mon.0) 2186 : cluster [DBG] osdmap e1087: 8 total, 8 up, 7 in 2021-02-10T04:49:12.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:11 smithi085 conmon[29319]: cluster 2021-02-10T04:49: 2021-02-10T04:49:12.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:11 smithi085 conmon[29319]: 10.371703+0000 mgr.y (mgr.14140) 1121 : cluster [DBG] pgmap v1902: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:12.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:11 smithi085 conmon[29319]: audit 2021-02-10T04:49:10.734760+0000 mon.a (mon.0) 2185 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2021-02-10T04:49:12.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:11 smithi085 conmon[29319]: cluster 2021-02-10T04:49:10.734834+0000 mon.a (mon.0) 2186 : cluster [DBG] osdmap e1087: 8 total, 8 up, 7 in 2021-02-10T04:49:12.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:11 smithi085 conmon[35961]: cluster 2021-02-10T04:49:12.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:11 smithi085 conmon[35961]: 2021-02-10T04:49:12.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:11 smithi085 conmon[35961]: 2021 2021-02-10T04:49:12.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:11 smithi085 conmon[35961]: -02-10T04:49:10. 2021-02-10T04:49:12.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:11 smithi085 conmon[35961]: 371703+0000 mgr.y (mgr.14140) 1121 : cluster [DBG] pgmap v1902: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:12.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:11 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:49:12.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:11 smithi085 conmon[35961]: 02-10T04:49:10.734760+0000 mon.a 2021-02-10T04:49:12.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:11 smithi085 conmon[35961]: (mon.0) 2185 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2021-02-10T04:49:12.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:11 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:49:12.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:11 smithi085 conmon[35961]: 02-10T04:49:10.734834+0000 2021-02-10T04:49:12.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:11 smithi085 conmon[35961]: mon.a (mon.0) 2186 : cluster [DBG] 2021-02-10T04:49:12.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:11 smithi085 conmon[35961]: osdmap e1087: 8 total, 8 up, 7 in 2021-02-10T04:49:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:12 smithi164 conmon[29288]: cluster 2021-02-10T04:49:12. 2021-02-10T04:49:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:12 smithi164 conmon[29288]: 741561+0000 mon.a (mon.0) 2187 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 282 sec, osd.4 has slow ops) 2021-02-10T04:49:13.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:12 smithi085 conmon[29319]: cluster 2021-02-10T04:49:12.741561+0000 2021-02-10T04:49:13.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:12 smithi085 conmon[29319]: mon.a (mon.0) 2187 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 282 sec, osd.4 has slow ops) 2021-02-10T04:49:13.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:12 smithi085 conmon[35961]: cluster 2021-02-10T04:49:12.741561 2021-02-10T04:49:13.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:12 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2187 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 282 sec, osd.4 has slow ops) 2021-02-10T04:49:13.801 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:49:13 smithi085 conmon[41540]: debug 2021-02-10T04:49:13.517+0000 7fb94e368700 -1 osd.0 pg_epoch: 1087 pg[1.0( empty local-lis/les=130/131 n=0 ec=19/19 lis/c=130/130 les/c/f=131/131/0 sis=130) [0,7,6] r=0 lpr=130 crt=0'0 mlcod 0'0 active+clean] scrubber pg(1.0) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6 (0x555d392dd760) 2021-02-10T04:49:13.801 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:13 smithi085 conmon[29319]: cluster 2021-02-10T04:49:12.372232+0000 2021-02-10T04:49:13.801 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:13 smithi085 conmon[29319]: mgr.y (mgr.14140) 1122 : cluster [DBG] pgmap v1904: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:49:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:13 smithi164 conmon[29288]: cluster 2021-02-10T04:49:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:13 smithi164 conmon[29288]: 2021-02-10T04:49:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:13 smithi164 conmon[29288]: 2021 2021-02-10T04:49:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:13 smithi164 conmon[29288]: - 2021-02-10T04:49:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:13 smithi164 conmon[29288]: 02 2021-02-10T04:49:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:13 smithi164 conmon[29288]: - 2021-02-10T04:49:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:13 smithi164 conmon[29288]: 10 2021-02-10T04:49:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:13 smithi164 conmon[29288]: T04:49:12.372232+0000 mgr.y 2021-02-10T04:49:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:13 smithi164 conmon[29288]: (mgr.14140) 1122 : cluster [DBG] pgmap v1904: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:49:14.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:13 smithi085 conmon[35961]: cluster 2021-02-10T04:49:14.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:13 smithi085 conmon[35961]: 2021-02-10T04:49:14.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:13 smithi085 conmon[35961]: 2021 2021-02-10T04:49:14.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:13 smithi085 conmon[35961]: - 2021-02-10T04:49:14.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:13 smithi085 conmon[35961]: 02 2021-02-10T04:49:14.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:13 smithi085 conmon[35961]: - 2021-02-10T04:49:14.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:13 smithi085 conmon[35961]: 10 2021-02-10T04:49:14.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:13 smithi085 conmon[35961]: T 2021-02-10T04:49:14.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:13 smithi085 conmon[35961]: 04 2021-02-10T04:49:14.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:13 smithi085 conmon[35961]: : 2021-02-10T04:49:14.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:13 smithi085 conmon[35961]: 49 2021-02-10T04:49:14.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:13 smithi085 conmon[35961]: : 2021-02-10T04:49:14.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:13 smithi085 conmon[35961]: 12 2021-02-10T04:49:14.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:13 smithi085 conmon[35961]: .372232+0000 mgr.y (mgr.14140) 2021-02-10T04:49:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:13 smithi085 conmon[35961]: 1122 : cluster [DBG] pgmap v1904: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:49:16.121 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:15 smithi164 conmon[29288]: cluster 2021-02-10T04:49:14.372949+0000 mgr.y (mgr. 2021-02-10T04:49:16.122 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:15 smithi164 conmon[29288]: 14140) 1123 : cluster [DBG] pgmap v1905: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:49:16.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:15 smithi085 conmon[29319]: cluster 2021-02-10T04:49:16.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:15 smithi085 conmon[29319]: 2021-02-10T04:49:16.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:15 smithi085 conmon[29319]: 2021-02-10T04:49:14.372949 2021-02-10T04:49:16.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:15 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 1123 : cluster [DBG] pgmap v1905: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:49:16.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:15 smithi085 conmon[35961]: cluster 2021-02-10T04:49:14. 2021-02-10T04:49:16.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:15 smithi085 conmon[35961]: 372949+0000 mgr.y (mgr.14140) 1123 : cluster [DBG] pgmap v1905: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:49:16.815 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:16 smithi164 conmon[29288]: cluster 2021-02-10T04:49:15.921483+0000 osd.2 (osd.2) 1955 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:49:16.815 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:16 smithi164 conmon[29288]: cluster 2021-02-10T04:49:15.922410+0000 osd.2 (osd.2) 1956 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:49:17.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[35961]: cluster 2021-02-10T04:49:17.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[35961]: 2021-02-10T04:49:17.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[35961]: 2021 2021-02-10T04:49:17.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[35961]: - 2021-02-10T04:49:17.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[35961]: 02 2021-02-10T04:49:17.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[35961]: -10T04:49:15.921483+0000 2021-02-10T04:49:17.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[35961]: osd.2 (osd.2) 1955 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:49:17.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:49:17.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[35961]: 49:15.922410+0000 osd.2 (osd.2) 1956 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:49:17.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[29319]: cluster 2021-02-10T04:49:17.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[29319]: 2021-02-10T04:49:17.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[29319]: 2021 2021-02-10T04:49:17.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[29319]: - 2021-02-10T04:49:17.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[29319]: 02 2021-02-10T04:49:17.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[29319]: - 2021-02-10T04:49:17.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[29319]: 10 2021-02-10T04:49:17.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[29319]: T 2021-02-10T04:49:17.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[29319]: 04 2021-02-10T04:49:17.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[29319]: : 2021-02-10T04:49:17.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[29319]: 49 2021-02-10T04:49:17.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[29319]: : 2021-02-10T04:49:17.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[29319]: 15 2021-02-10T04:49:17.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[29319]: . 2021-02-10T04:49:17.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[29319]: 921483 2021-02-10T04:49:17.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[29319]: +0000 2021-02-10T04:49:17.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[29319]: 2021-02-10T04:49:17.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[29319]: osd.2 2021-02-10T04:49:17.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[29319]: ( 2021-02-10T04:49:17.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[29319]: osd 2021-02-10T04:49:17.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[29319]: . 2021-02-10T04:49:17.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[29319]: 2) 1955 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:49:17.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:49:17.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[29319]: 04:49:15.922410+0000 osd.2 (osd.2) 1956 2021-02-10T04:49:17.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:16 smithi085 conmon[29319]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:49:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:17 smithi164 conmon[29288]: cluster 2021-02-10T04:49:16.373672+0000 mgr.y 2021-02-10T04:49:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:17 smithi164 conmon[29288]: (mgr.14140) 1124 : cluster [DBG] pgmap v1906: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:49:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:17 smithi164 conmon[29288]: cluster 2021-02-10T04:49:16.816274+0000 mon.a (mon.0) 2188 : cluster [DBG] osdmap e1088: 8 total, 8 up, 7 in 2021-02-10T04:49:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:17 smithi164 conmon[29288]: audit 2021-02-10T04:49:16.988791+0000 mon.a (mon.0) 2189 2021-02-10T04:49:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:17 smithi164 conmon[29288]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:49:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:17 smithi164 conmon[29288]: cephadm 2021-02- 2021-02-10T04:49:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:17 smithi164 conmon[29288]: 10T04:49:16.989455+0000 mgr.y (mgr.14140) 1125 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:49:18.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:17 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:49:18.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:17 smithi085 conmon[29319]: 04:49:16.373672+0000 mgr.y (mgr.14140) 1124 : cluster [DBG] pgmap v1906: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:49:18.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:17 smithi085 conmon[29319]: cluster 2021-02-10T04:49:16.816274+0000 mon.a (mon.0 2021-02-10T04:49:18.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:17 smithi085 conmon[29319]: ) 2188 : cluster [DBG] osdmap e1088: 8 total, 8 up, 7 in 2021-02-10T04:49:18.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:17 smithi085 conmon[29319]: audit 2021-02-10T04:49: 2021-02-10T04:49:18.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:17 smithi085 conmon[29319]: 16.988791+0000 mon.a (mon.0) 2189 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:49:18.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:17 smithi085 conmon[29319]: cephadm 2021-02-10T04:49:16.989455+0000 2021-02-10T04:49:18.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:17 smithi085 conmon[29319]: mgr.y (mgr.14140) 1125 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:49:18.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:17 smithi085 conmon[35961]: cluster 2021-02-10T04:49:18.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:17 smithi085 conmon[35961]: 2021-02-10T04:49:18.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:17 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:49:18.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:17 smithi085 conmon[35961]: :49:16.373672+0000 mgr.y (mgr.14140) 1124 : cluster [DBG] pgmap v1906: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:49:18.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:17 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:49:18.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:17 smithi085 conmon[35961]: T04:49:16.816274+0000 mon.a (mon.0 2021-02-10T04:49:18.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:17 smithi085 conmon[35961]: ) 2188 : cluster [DBG] osdmap e1088: 8 total, 8 up, 7 in 2021-02-10T04:49:18.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:17 smithi085 conmon[35961]: audit 2021 2021-02-10T04:49:18.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:17 smithi085 conmon[35961]: -02-10T04:49:16.988791+0000 2021-02-10T04:49:18.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:17 smithi085 conmon[35961]: mon.a (mon.0) 2189 : audit [INF] 2021-02-10T04:49:18.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:17 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:49:18.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:17 smithi085 conmon[35961]: cephadm 2021-02-10T04:49:16 2021-02-10T04:49:18.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:17 smithi085 conmon[35961]: .989455+0000 mgr.y (mgr.14140) 1125 : cephadm 2021-02-10T04:49:18.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:17 smithi085 conmon[35961]: [INF] Refreshing smithi164 facts 2021-02-10T04:49:18.692 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:49:18 smithi085 conmon[41540]: debug 2021-02-10T04:49:18.692 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:49:18 smithi085 conmon[41540]: 2021-02-10T04:49:18.402+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:49:18.692 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:49:18 smithi085 conmon[41540]: 2021-02-10T04:49:18.723 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:49:19.394 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:49:18 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:49:20.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:19 smithi164 conmon[29288]: cluster 2021-02-10T04:49:20.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:19 smithi164 conmon[29288]: 2021-02-10T04:49:20.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:19 smithi164 conmon[29288]: 2021 2021-02-10T04:49:20.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:19 smithi164 conmon[29288]: - 2021-02-10T04:49:20.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:19 smithi164 conmon[29288]: 02 2021-02-10T04:49:20.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:19 smithi164 conmon[29288]: - 2021-02-10T04:49:20.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:19 smithi164 conmon[29288]: 10T04:49:18.374339+0000 mgr.y (mgr.14140) 1126 : 2021-02-10T04:49:20.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:19 smithi164 conmon[29288]: cluster [DBG] pgmap v1908: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:49:20.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:19 smithi164 conmon[29288]: audit 2021-02-10T04:49:20.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:19 smithi164 conmon[29288]: 2021-02-10T04:49:18.940051+0000 mgr.y (mgr.14140) 2021-02-10T04:49:20.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:19 smithi164 conmon[29288]: 1127 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:49:20.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:19 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:49:20.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:19 smithi085 conmon[35961]: T04:49:18.374339+0000 mgr.y (mgr.14140) 1126 : cluster [DBG] pgmap v1908: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:49:20.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:19 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:49:20.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:19 smithi085 conmon[35961]: 02-10T04:49:18.940051+0000 2021-02-10T04:49:20.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:19 smithi085 conmon[35961]: mgr.y (mgr.14140) 1127 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:49:20.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:19 smithi085 conmon[29319]: cluster 2021-02-10T04:49:20.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:19 smithi085 conmon[29319]: 2021-02 2021-02-10T04:49:20.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:19 smithi085 conmon[29319]: -10T04:49:18.374339+0000 mgr.y (mgr.14140) 1126 : cluster [DBG] pgmap v1908: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:49:20.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:19 smithi085 conmon[29319]: audit 2021-02-10T04:49:20.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:19 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:49:20.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:19 smithi085 conmon[29319]: T04:49:18.940051+0000 mgr.y (mgr.14140) 1127 : audit [DBG] 2021-02-10T04:49:20.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:19 smithi085 conmon[29319]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:49:21.820 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:49:21 smithi085 conmon[41540]: debug 2021-02-10T04:49:21.559+0000 7fb94a360700 -1 osd.0 pg_epoch: 1088 pg[1.0( empty local-lis/les=130/131 n=0 ec=19/19 lis/c=130/130 les/c/f=131/131/0 sis=130) [0,7,6] r=0 lpr=130 crt=0'0 mlcod 0'0 active+clean] scrubber pg(1.0) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6 (0x555d3284e420) 2021-02-10T04:49:22.102 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:21 smithi085 conmon[35961]: cluster 2021-02-10T04:49:22.102 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:21 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:49:22.102 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:21 smithi085 conmon[35961]: :49:20.375161+0000 mgr.y (mgr.14140) 1128 : cluster [DBG] pgmap v1909: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:49:22.102 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:21 smithi085 conmon[29319]: cluster 2021-02-10T04:49:20.375161+0000 mgr.y (mgr 2021-02-10T04:49:22.103 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:21 smithi085 conmon[29319]: .14140) 1128 : cluster [DBG] pgmap v1909: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:49:22.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:21 smithi164 conmon[29288]: cluster 2021-02-10T04:49:20.375161+0000 mgr.y (mgr.14140) 2021-02-10T04:49:22.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:21 smithi164 conmon[29288]: 1128 : cluster [DBG] pgmap v1909: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:49:22.941 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:49:22 smithi085 conmon[41540]: debug 2021-02-10T04:49:22.941 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:49:22 smithi085 conmon[41540]: 2021-02-10T04:49:22.533+0000 7fb94e368700 -1 osd.0 pg_epoch: 1088 pg[1.0( empty local-lis/les=130/131 n=0 ec=19/19 lis/c=130/130 les/c/f=131/131/0 sis=130) [0,7,6] r=0 lpr=130 crt=0'0 mlcod 0'0 active+clean] scrubber pg(1.0) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6 (0x555d327fc840) 2021-02-10T04:49:22.942 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:49:22 smithi085 conmon[41540]: 2021-02-10T04:49:24.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:23 smithi164 conmon[29288]: cluster 2021-02-10T04:49:22.375788+0000 mgr.y ( 2021-02-10T04:49:24.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:23 smithi164 conmon[29288]: mgr.14140) 1129 : cluster [DBG] pgmap v1910: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:24.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[29319]: cluster 2021-02-10T04:49:22.375788+0000 mgr.y (mgr.14140 2021-02-10T04:49:24.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[29319]: ) 1129 : cluster [DBG] pgmap v1910: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:24.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: cluster 2021-02-10T04:49:24.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: 2021-02-10T04:49:24.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: 2021 2021-02-10T04:49:24.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: - 2021-02-10T04:49:24.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: 02 2021-02-10T04:49:24.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: - 2021-02-10T04:49:24.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: 10 2021-02-10T04:49:24.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: T 2021-02-10T04:49:24.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: 04 2021-02-10T04:49:24.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: : 2021-02-10T04:49:24.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: 49 2021-02-10T04:49:24.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: : 2021-02-10T04:49:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: 22 2021-02-10T04:49:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: . 2021-02-10T04:49:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: 375788 2021-02-10T04:49:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: +0000 2021-02-10T04:49:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: 2021-02-10T04:49:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: mgr.y 2021-02-10T04:49:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: ( 2021-02-10T04:49:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: mgr 2021-02-10T04:49:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: . 2021-02-10T04:49:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: 14140 2021-02-10T04:49:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: ) 2021-02-10T04:49:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: 1129 2021-02-10T04:49:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: : 2021-02-10T04:49:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: cluster 2021-02-10T04:49:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: 2021-02-10T04:49:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: [DBG] 2021-02-10T04:49:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: 2021-02-10T04:49:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: pgmap v1910: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:23 smithi085 conmon[35961]: 2021-02-10T04:49:24.987 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:49:25.009 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 1 1 2021-02-10T04:49:25.021 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:49:25.446 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:49:26.107 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[29319]: cluster 2021-02-10T04:49:24.376416+0000 mgr.y (mgr.14140) 1130 : cluster [DBG] pgmap v1911: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:26.107 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: cluster 2021-02-10T04:49:26.107 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: 2021-02-10T04:49:26.107 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: 2021 2021-02-10T04:49:26.107 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: - 2021-02-10T04:49:26.107 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: 02 2021-02-10T04:49:26.107 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: - 2021-02-10T04:49:26.107 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: 10 2021-02-10T04:49:26.107 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: T 2021-02-10T04:49:26.108 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: 04 2021-02-10T04:49:26.108 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: : 2021-02-10T04:49:26.108 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: 49 2021-02-10T04:49:26.108 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: : 2021-02-10T04:49:26.108 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: 24 2021-02-10T04:49:26.108 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: . 2021-02-10T04:49:26.108 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: 376416 2021-02-10T04:49:26.108 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: +0000 2021-02-10T04:49:26.108 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: 2021-02-10T04:49:26.108 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: mgr.y 2021-02-10T04:49:26.108 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: ( 2021-02-10T04:49:26.108 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: mgr 2021-02-10T04:49:26.108 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: . 2021-02-10T04:49:26.108 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: 14140 2021-02-10T04:49:26.109 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: ) 2021-02-10T04:49:26.109 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: 1130 2021-02-10T04:49:26.109 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: : 2021-02-10T04:49:26.109 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: cluster 2021-02-10T04:49:26.109 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: 2021-02-10T04:49:26.109 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: [DBG] 2021-02-10T04:49:26.109 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: 2021-02-10T04:49:26.109 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: pgmap v1911: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:26.109 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:25 smithi085 conmon[35961]: 2021-02-10T04:49:26.144 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:49:25 smithi164 conmon[44544]: debug 2021-02-10T04:49:25.761+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:49:26.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:25 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:49:26.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:25 smithi164 conmon[29288]: T04:49:24.376416+0000 mgr.y (mgr.14140) 1130 : cluster [DBG] pgmap v1911: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:26.382 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:49:26.488 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:49:26.519 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:49:27.002 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:49:27.029 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:49:27.135 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:49:27.143 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:49:26 smithi164 conmon[44544]: debug 2021-02-10T04:49:26.790+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:49:27.168 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:49:27.424 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:49:27.473 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:49:27.473 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:49:27.536 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:49:27.541 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:49:27 smithi164 conmon[37263]: debug 2021-02-10T04:49:27.314+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:49:27.642 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.7 2021-02-10T04:49:27.676 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:49:28.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:27 smithi164 conmon[29288]: cluster 2021-02-10T04:49:26.377099+0000 mgr.y (mgr.14140) 1131 : cluster 2021-02-10T04:49:28.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:27 smithi164 conmon[29288]: [DBG] pgmap v1912: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:28.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:27 smithi164 conmon[29288]: cluster 2021-02-10T04:49:26.417196+0000 osd.6 (osd.6) 137 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:49:28.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:27 smithi164 conmon[29288]: cluster 2021-02-10T04:49:26.419490+0000 osd.6 (osd.6) 138 : cluster [DBG] 2021-02-10T04:49:28.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:27 smithi164 conmon[29288]: purged_snaps scrub ok 2021-02-10T04:49:28.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:27 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:49:28.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:27 smithi164 conmon[29288]: 04:49:26.836848+0000 mon.a (mon.0) 2021-02-10T04:49:28.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:27 smithi164 conmon[29288]: 2190 : cluster [DBG] osdmap e1089: 8 total, 8 up, 7 in 2021-02-10T04:49:28.144 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:49:27 smithi164 conmon[44544]: debug 2021-02-10T04:49:27.915+0000 7f64cb997700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:49:28.169 INFO:teuthology.orchestra.run.smithi164.stdout:252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T04:49:28.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:27 smithi085 conmon[35961]: cluster 2021-02-10T04:49:26.377099+0000 mgr.y (mgr.14140) 1131 : cluster [DBG] pgmap v1912: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:28.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:27 smithi085 conmon[35961]: cluster 2021-02-10T04:49:26.417196+0000 osd.6 (osd.6) 137 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:49:28.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:27 smithi085 conmon[35961]: cluster 2021-02-10T04:49:26.419490+0000 osd.6 (osd.6) 138 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:49:28.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:27 smithi085 conmon[35961]: cluster 2021-02-10T04:49:26.836848+0000 mon.a (mon.0) 2190 : cluster [DBG] osdmap e1089: 8 total, 8 up, 7 in 2021-02-10T04:49:28.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:27 smithi085 conmon[29319]: cluster 2021-02-10T04:49:28.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:27 smithi085 conmon[29319]: 2021-02-10T04:49:28.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:27 smithi085 conmon[29319]: 2021 2021-02-10T04:49:28.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:27 smithi085 conmon[29319]: - 2021-02-10T04:49:28.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:27 smithi085 conmon[29319]: 02 2021-02-10T04:49:28.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:27 smithi085 conmon[29319]: - 2021-02-10T04:49:28.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:27 smithi085 conmon[29319]: 10T04:49:26.377099+0000 mgr.y 2021-02-10T04:49:28.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:27 smithi085 conmon[29319]: (mgr.14140) 1131 : cluster [DBG] pgmap v1912: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:28.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:27 smithi085 conmon[29319]: cluster 2021-02-10T04:49:28.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:27 smithi085 conmon[29319]: 2021 2021-02-10T04:49:28.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:27 smithi085 conmon[29319]: - 2021-02-10T04:49:28.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:27 smithi085 conmon[29319]: 02 2021-02-10T04:49:28.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:27 smithi085 conmon[29319]: - 2021-02-10T04:49:28.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:27 smithi085 conmon[29319]: 10T04:49:26.417196 2021-02-10T04:49:28.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:27 smithi085 conmon[29319]: +0000 osd.6 (osd.6) 137 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:49:28.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:27 smithi085 conmon[29319]: cluster 2021-02-10T04:49:28.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:27 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:49:28.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:27 smithi085 conmon[29319]: T04:49:26.419490+0000 osd.6 (osd. 2021-02-10T04:49:28.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:27 smithi085 conmon[29319]: 6) 138 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:49:28.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:27 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:49:28.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:27 smithi085 conmon[29319]: 02-10T04:49:26.836848+0000 mon.a (mon. 2021-02-10T04:49:28.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:27 smithi085 conmon[29319]: 0) 2190 : cluster [DBG] osdmap e1089: 8 total, 8 up, 7 in 2021-02-10T04:49:28.276 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.0 2021-02-10T04:49:28.366 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:49:29.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:49:29 smithi164 conmon[40892]: debug 2021-02-10T04:49:29.085+0000 7f3037cca700 -1 osd.6 pg_epoch: 1089 pg[6.22( empty local-lis/les=1046/1047 n=0 ec=1044/59 lis/c=1046/1046 les/c/f=1047/1047/0 sis=1046) [6,3,1] r=0 lpr=1046 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.22) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x557a7842a580) 2021-02-10T04:49:29.394 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:49:28 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:49:30.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:29 smithi164 conmon[29288]: cluster 2021-02-10T04:49:28.377732+0000 2021-02-10T04:49:30.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:29 smithi164 conmon[29288]: mgr.y (mgr.14140) 1132 : cluster [DBG] pgmap v1914: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:30.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:29 smithi164 conmon[29288]: audit 2021-02-10T04:49:28.950008 2021-02-10T04:49:30.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:29 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1133 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:49:30.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:29 smithi085 conmon[35961]: cluster 2021-02-10T04:49:28.377732 2021-02-10T04:49:30.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:29 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1132 : cluster [DBG] pgmap v1914: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:30.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:29 smithi085 conmon[35961]: audit 2021-02-10T04:49 2021-02-10T04:49:30.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:29 smithi085 conmon[35961]: :28.950008+0000 mgr.y (mgr.14140) 1133 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:49:30.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:29 smithi085 conmon[29319]: cluster 2021-02-10T04:49:28.377732+0000 mgr.y (mgr.14140) 1132 : cluster [DBG] pgmap v1914: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:30.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:29 smithi085 conmon[29319]: 2021-02-10T04:49:30.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:29 smithi085 conmon[29319]: audit 2021-02-10T04:49:28.950008+0000 mgr.y (mgr.14140) 1133 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:49:32.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:31 smithi164 conmon[29288]: cluster 2021-02-10T04:49:30.378481+0000 mgr.y (mgr.14140) 1134 : cluster [DBG] pgmap v1915: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:32.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:31 smithi085 conmon[29319]: cluster 2021-02-10T04:49:32.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:31 smithi085 conmon[29319]: 2021-02-10T04:49:32.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:31 smithi085 conmon[29319]: 2021 2021-02-10T04:49:32.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:31 smithi085 conmon[29319]: - 2021-02-10T04:49:32.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:31 smithi085 conmon[29319]: 02 2021-02-10T04:49:32.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:31 smithi085 conmon[29319]: - 2021-02-10T04:49:32.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:31 smithi085 conmon[29319]: 10T04:49:30.378481+0000 2021-02-10T04:49:32.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:31 smithi085 conmon[29319]: mgr.y (mgr.14140) 1134 : cluster [DBG] pgmap v1915: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:32.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:31 smithi085 conmon[35961]: cluster 2021-02-10T04:49:30. 2021-02-10T04:49:32.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:31 smithi085 conmon[35961]: 378481+0000 mgr.y (mgr.14140) 1134 : cluster [DBG] pgmap v1915: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:33.993 INFO:teuthology.orchestra.run.smithi085.stdout:e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:49:34.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:33 smithi164 conmon[29288]: cluster 2021-02-10T04:49:32.379029+0000 mgr.y 2021-02-10T04:49:34.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:33 smithi164 conmon[29288]: (mgr.14140) 1135 : cluster [DBG] pgmap v1916: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:34.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:33 smithi085 conmon[29319]: cluster 2021-02-10T04:49:32.379029+0000 mgr.y (mgr. 2021-02-10T04:49:34.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:33 smithi085 conmon[29319]: 14140) 1135 : cluster [DBG] pgmap v1916: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:34.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:33 smithi085 conmon[35961]: cluster 2021-02-10T04:49:32 2021-02-10T04:49:34.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:33 smithi085 conmon[35961]: .379029+0000 mgr.y (mgr.14140) 1135 : cluster [DBG] pgmap v1916: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:34.193 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:49:33 smithi085 conmon[41540]: debug 2021-02-10T04:49:33.743+0000 7fb96845e700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:49:34.644 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 04:49:34 smithi164 conmon[40892]: debug 2021-02-10T04:49:34.151+0000 7f3033cc2700 -1 osd.6 pg_epoch: 1089 pg[6.22( empty local-lis/les=1046/1047 n=0 ec=1044/59 lis/c=1046/1046 les/c/f=1047/1047/0 sis=1046) [6,3,1] r=0 lpr=1046 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.22) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x557a7842adc0) 2021-02-10T04:49:34.942 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:49:34 smithi085 conmon[41540]: debug 2021-02-10T04:49:34.942 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:49:34 smithi085 conmon[41540]: 2021-02-10T04:49:34.473+0000 7fb94e368700 -1 osd.0 pg_epoch: 1089 pg[1.0( empty local-lis/les=130/131 n=0 ec=19/19 lis/c=130/130 les/c/f=131/131/0 sis=130) [0,7,6] r=0 lpr=130 crt=0'0 mlcod 0'0 active+clean] scrubber pg(1.0) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6 (0x555d3ad74dc0) 2021-02-10T04:49:34.942 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:49:34 smithi085 conmon[41540]: 2021-02-10T04:49:36.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:35 smithi164 conmon[29288]: cluster 2021-02-10T04:49:36.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:35 smithi164 conmon[29288]: 2021-02-10T04:49: 2021-02-10T04:49:36.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:35 smithi164 conmon[29288]: 34.379722+0000 mgr.y (mgr.14140) 1136 : cluster [DBG] pgmap v1917: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:36.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:35 smithi164 conmon[29288]: cluster 2021-02-10T04:49:36.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:35 smithi164 conmon[29288]: 2021-02-10T04:49:34.819709+0000 osd.3 (osd.3) 141 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:49:36.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:35 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:49:36.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:35 smithi164 conmon[29288]: 49:34.824097+0000 osd.3 (osd.3) 142 : cluster 2021-02-10T04:49:36.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:35 smithi164 conmon[29288]: [DBG] purged_snaps scrub ok 2021-02-10T04:49:36.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:35 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:49:36.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:35 smithi164 conmon[29288]: 04:49:34.856877+0000 mon.a (mon.0 2021-02-10T04:49:36.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:35 smithi164 conmon[29288]: ) 2191 : cluster [DBG] osdmap e1090: 8 total, 8 up, 7 in 2021-02-10T04:49:36.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: cluster 2021-02-10T04:49:36.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: 2021-02-10T04:49:36.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: 2021 2021-02-10T04:49:36.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: - 2021-02-10T04:49:36.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: 02 2021-02-10T04:49:36.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: - 2021-02-10T04:49:36.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: 10 2021-02-10T04:49:36.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: T 2021-02-10T04:49:36.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: 04 2021-02-10T04:49:36.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: :49:34.379722+0000 mgr.y (mgr. 2021-02-10T04:49:36.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: 14140) 1136 : cluster [DBG] pgmap v1917: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:36.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: cluster 2021-02-10T04:49:34.819709 2021-02-10T04:49:36.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: +0000 osd.3 (osd.3) 141 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:49:36.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: cluster 2021-02-10T04:49:36.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: 2021-02-10T04:49:36.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: 2021 2021-02-10T04:49:36.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: - 2021-02-10T04:49:36.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: 02 2021-02-10T04:49:36.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: -10T04:49:34. 2021-02-10T04:49:36.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: 824097+0000 osd.3 (osd.3) 142 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:49:36.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: cluster 2021-02-10T04:49:36.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: 2021-02-10T04:49:36.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: 2021 2021-02-10T04:49:36.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: - 2021-02-10T04:49:36.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: 02 2021-02-10T04:49:36.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: - 2021-02-10T04:49:36.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: 10 2021-02-10T04:49:36.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: T 2021-02-10T04:49:36.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: 04 2021-02-10T04:49:36.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: : 2021-02-10T04:49:36.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: 49 2021-02-10T04:49:36.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: : 2021-02-10T04:49:36.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: 34 2021-02-10T04:49:36.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: . 2021-02-10T04:49:36.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: 856877 2021-02-10T04:49:36.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: +0000 2021-02-10T04:49:36.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: 2021-02-10T04:49:36.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: mon.a 2021-02-10T04:49:36.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: ( 2021-02-10T04:49:36.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: mon 2021-02-10T04:49:36.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: . 2021-02-10T04:49:36.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: 0 2021-02-10T04:49:36.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: ) 2021-02-10T04:49:36.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: 2191 2021-02-10T04:49:36.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: : 2021-02-10T04:49:36.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: cluster 2021-02-10T04:49:36.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: 2021-02-10T04:49:36.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: [DBG] 2021-02-10T04:49:36.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: 2021-02-10T04:49:36.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: osdmap e1090: 8 total, 8 up, 7 in 2021-02-10T04:49:36.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[35961]: 2021-02-10T04:49:36.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[29319]: cluster 2021-02-10T04:49:34.379722+0000 mgr.y (mgr 2021-02-10T04:49:36.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[29319]: .14140) 1136 : cluster [DBG] pgmap v1917: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:36.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[29319]: cluster 2021-02-10T04:49:34.819709+0000 2021-02-10T04:49:36.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[29319]: osd.3 (osd.3) 141 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:49:36.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[29319]: cluster 2021-02-10T04:49:34.824097+0000 osd.3 2021-02-10T04:49:36.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[29319]: (osd.3) 142 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:49:36.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[29319]: cluster 2021-02-10T04:49:34. 2021-02-10T04:49:36.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:35 smithi085 conmon[29319]: 856877+0000 mon.a (mon.0) 2191 : cluster [DBG] osdmap e1090: 8 total, 8 up, 7 in 2021-02-10T04:49:38.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:37 smithi164 conmon[29288]: cluster 2021-02-10T04:49:36.380304+0000 mgr.y 2021-02-10T04:49:38.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:37 smithi164 conmon[29288]: (mgr.14140) 1137 : cluster [DBG] pgmap v1919: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:49:38.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:37 smithi164 conmon[29288]: cluster 2021-02-10T04:49:37.369958+0000 osd.0 (osd.0) 149 : cluster [DBG] 2021-02-10T04:49:38.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:37 smithi164 conmon[29288]: purged_snaps scrub starts 2021-02-10T04:49:38.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:37 smithi164 conmon[29288]: cluster 2021-02-10T04:49: 2021-02-10T04:49:38.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:37 smithi164 conmon[29288]: 37.372387+0000 osd.0 (osd.0) 150 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:49:38.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:37 smithi085 conmon[29319]: cluster 2021-02-10T04:49:36.380304+0000 mgr.y (mgr.14140) 1137 : cluster 2021-02-10T04:49:38.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:37 smithi085 conmon[29319]: [DBG] pgmap v1919: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:49:38.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:37 smithi085 conmon[29319]: cluster 2021-02-10T04:49:37.369958+0000 osd.0 (osd.0) 149 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:49:38.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:37 smithi085 conmon[29319]: cluster 2021-02-10T04:49:37.372387+0000 osd.0 (osd.0) 150 2021-02-10T04:49:38.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:37 smithi085 conmon[29319]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:49:38.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:37 smithi085 conmon[35961]: cluster 2021-02-10T04:49:36.380304+0000 2021-02-10T04:49:38.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:37 smithi085 conmon[35961]: mgr.y (mgr.14140) 1137 : cluster [DBG] pgmap v1919: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:49:38.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:37 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:49:38.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:37 smithi085 conmon[35961]: :49:37.369958+0000 osd.0 (osd.0) 149 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:49:38.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:37 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:49:38.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:37 smithi085 conmon[35961]: -02-10T04:49:37.372387+0000 2021-02-10T04:49:38.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:37 smithi085 conmon[35961]: osd.0 (osd.0) 150 : cluster 2021-02-10T04:49:38.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:37 smithi085 conmon[35961]: [DBG] purged_snaps scrub ok 2021-02-10T04:49:39.144 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:49:38 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:49:39.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:38 smithi164 conmon[29288]: cluster 2021-02-10T04:49:39.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:38 smithi164 conmon[29288]: 2021-02-10T04:49:39.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:38 smithi164 conmon[29288]: 2021 2021-02-10T04:49:39.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:38 smithi164 conmon[29288]: - 2021-02-10T04:49:39.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:38 smithi164 conmon[29288]: 02-10T04:49 2021-02-10T04:49:39.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:38 smithi164 conmon[29288]: :37.861294+0000 mon.a (mon.0) 2192 : cluster [DBG] osdmap e1091: 8 total, 8 up, 7 in 2021-02-10T04:49:39.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:38 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:49:39.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:38 smithi164 conmon[29288]: 02-10T04:49:38.197576+0000 mon.a (mon.0) 2193 2021-02-10T04:49:39.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:38 smithi164 conmon[29288]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:49:39.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:38 smithi164 conmon[29288]: audit 2021-02-10T04:49:39.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:38 smithi164 conmon[29288]: 2021-02-10T04:49: 2021-02-10T04:49:39.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:38 smithi164 conmon[29288]: 38.197912+0000 mon.a (mon.0) 2194 : 2021-02-10T04:49:39.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:38 smithi164 conmon[29288]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:49:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:38 smithi085 conmon[29319]: cluster 2021-02-10T04:49:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:38 smithi085 conmon[29319]: 2021-02 2021-02-10T04:49:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:38 smithi085 conmon[29319]: -10T04:49:37.861294+0000 mon.a (mon.0) 2192 : cluster [DBG] osdmap e1091: 8 total, 8 up, 7 in 2021-02-10T04:49:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:38 smithi085 conmon[29319]: 2021-02-10T04:49:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:38 smithi085 conmon[29319]: audit 2021-02-10T04:49:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:38 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:49:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:38 smithi085 conmon[29319]: 04:49:38.197576+0000 mon.a (mon 2021-02-10T04:49:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:38 smithi085 conmon[29319]: .0) 2193 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:49:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:38 smithi085 conmon[29319]: audit 2021-02-10T04:49:39.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:38 smithi085 conmon[29319]: 2021-02 2021-02-10T04:49:39.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:38 smithi085 conmon[29319]: -10T04:49:38.197912+0000 mon.a (mon.0) 2021-02-10T04:49:39.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:38 smithi085 conmon[29319]: 2194 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:49:39.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:38 smithi085 conmon[35961]: cluster 2021-02-10T04:49:39.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:38 smithi085 conmon[35961]: 2021-02-10T04:49:39.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:38 smithi085 conmon[35961]: 2021 2021-02-10T04:49:39.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:38 smithi085 conmon[35961]: - 2021-02-10T04:49:39.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:38 smithi085 conmon[35961]: 02-10T04:49 2021-02-10T04:49:39.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:38 smithi085 conmon[35961]: :37.861294+0000 mon.a (mon.0) 2192 : cluster [DBG] osdmap e1091: 8 total, 8 up, 7 in 2021-02-10T04:49:39.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:38 smithi085 conmon[35961]: audit 2021 2021-02-10T04:49:39.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:38 smithi085 conmon[35961]: -02-10T04:49:38.197576+0000 mon.a (mon. 2021-02-10T04:49:39.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:38 smithi085 conmon[35961]: 0) 2193 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:49:39.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:38 smithi085 conmon[35961]: audit 2021 2021-02-10T04:49:39.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:38 smithi085 conmon[35961]: -02-10T04:49:38.197912+0000 2021-02-10T04:49:39.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:38 smithi085 conmon[35961]: mon.a (mon.0) 2194 : audit [INF] 2021-02-10T04:49:39.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:38 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:49:39.691 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:49:39 smithi085 conmon[41540]: debug 2021-02-10T04:49:39.500+0000 7fb94e368700 -1 osd.0 pg_epoch: 1091 pg[1.0( empty local-lis/les=130/131 n=0 ec=19/19 lis/c=130/130 les/c/f=131/131/0 sis=130) [0,7,6] r=0 lpr=130 crt=0'0 mlcod 0'0 active+clean] scrubber pg(1.0) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6 (0x555d32c37b80) 2021-02-10T04:49:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:39 smithi164 conmon[29288]: cluster 2021-02-10T04:49:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:39 smithi164 conmon[29288]: 2021-02-10T04:49:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:39 smithi164 conmon[29288]: 2021-02-10T04:49: 2021-02-10T04:49:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:39 smithi164 conmon[29288]: 38.380905+0000 mgr.y (mgr.14140) 1138 : cluster [DBG] pgmap v1921: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:49:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:39 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:49:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:39 smithi164 conmon[29288]: 02-10T04:49:38.959999+0000 mgr.y (mgr.14140) 1139 : audit 2021-02-10T04:49:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:39 smithi164 conmon[29288]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:49:40.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:39 smithi085 conmon[29319]: cluster 2021-02-10T04:49:38.380905+0000 mgr.y (mgr. 2021-02-10T04:49:40.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:39 smithi085 conmon[29319]: 14140) 1138 : cluster [DBG] pgmap v1921: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:49:40.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:39 smithi085 conmon[29319]: audit 2021-02-10T04:49:38.959999+0000 mgr.y (mgr.14140) 1139 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:49:40.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:39 smithi085 conmon[35961]: cluster 2021-02-10T04:49:40.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:39 smithi085 conmon[35961]: 2021-02-10T04:49:40.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:39 smithi085 conmon[35961]: 2021 2021-02-10T04:49:40.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:39 smithi085 conmon[35961]: - 2021-02-10T04:49:40.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:39 smithi085 conmon[35961]: 02 2021-02-10T04:49:40.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:39 smithi085 conmon[35961]: - 2021-02-10T04:49:40.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:39 smithi085 conmon[35961]: 10 2021-02-10T04:49:40.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:39 smithi085 conmon[35961]: T 2021-02-10T04:49:40.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:39 smithi085 conmon[35961]: 04 2021-02-10T04:49:40.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:39 smithi085 conmon[35961]: : 2021-02-10T04:49:40.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:39 smithi085 conmon[35961]: 49 2021-02-10T04:49:40.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:39 smithi085 conmon[35961]: : 2021-02-10T04:49:40.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:39 smithi085 conmon[35961]: 38 2021-02-10T04:49:40.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:39 smithi085 conmon[35961]: . 2021-02-10T04:49:40.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:39 smithi085 conmon[35961]: 380905 2021-02-10T04:49:40.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:39 smithi085 conmon[35961]: +0000 mgr.y (mgr 2021-02-10T04:49:40.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:39 smithi085 conmon[35961]: .14140) 1138 : cluster [DBG] pgmap v1921: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:49:40.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:39 smithi085 conmon[35961]: audit 2021-02-10T04:49 2021-02-10T04:49:40.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:39 smithi085 conmon[35961]: :38.959999+0000 mgr.y (mgr.14140) 1139 : audit 2021-02-10T04:49:40.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:39 smithi085 conmon[35961]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:49:41.875 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:49:41 smithi085 conmon[41540]: debug 2021-02-10T04:49:41.463+0000 7fb94a360700 -1 osd.0 pg_epoch: 1091 pg[1.0( empty local-lis/les=130/131 n=0 ec=19/19 lis/c=130/130 les/c/f=131/131/0 sis=130) [0,7,6] r=0 lpr=130 crt=0'0 mlcod 0'0 active+clean] scrubber pg(1.0) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6 (0x555d35d629a0) 2021-02-10T04:49:41.876 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:41 smithi085 conmon[29319]: cluster 2021-02-10T04:49:40.381552+0000 mgr.y (mgr.14140) 2021-02-10T04:49:41.876 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:41 smithi085 conmon[29319]: 1140 : cluster [DBG] pgmap v1922: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:49:41.876 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:41 smithi085 conmon[35961]: cluster 2021-02-10T04:49:40.381552+0000 2021-02-10T04:49:41.876 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:41 smithi085 conmon[35961]: mgr.y (mgr.14140) 1140 : cluster [DBG] pgmap v1922: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:49:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:41 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:49:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:41 smithi164 conmon[29288]: T04:49:40.381552+0000 mgr.y (mgr.14140) 1140 : cluster [DBG] pgmap v1922: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:49:42.700 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:49:42.748 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:49:42.835 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.0 dump_historic_ops 2021-02-10T04:49:43.185 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:49:43.191 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:49:43 smithi085 conmon[52836]: debug 2021-02-10T04:49:43.082+0000 7fba938b7700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:49:43.209 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:49:43.346 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:49:43.390 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:49:43.617 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:49:43.749 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.1 2021-02-10T04:49:43.780 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:49:43 smithi085 conmon[49106]: debug 2021-02-10T04:49:43.503+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:49:43.786 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:49:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:43 smithi164 conmon[29288]: cluster 2021-02-10T04:49:42.382126+0000 2021-02-10T04:49:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:43 smithi164 conmon[29288]: mgr.y (mgr.14140) 1141 : cluster [DBG] pgmap v1923: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:49:44.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:43 smithi085 conmon[29319]: cluster 2021-02-10T04:49:42.382126+0000 2021-02-10T04:49:44.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:43 smithi085 conmon[29319]: mgr.y (mgr.14140) 1141 : cluster [DBG] pgmap v1923: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:49:44.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:43 smithi085 conmon[35961]: cluster 2021-02-10T04:49:42.382126+0000 mgr.y (mgr.14140 2021-02-10T04:49:44.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:43 smithi085 conmon[35961]: ) 1141 : cluster [DBG] pgmap v1923: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:49:44.708 INFO:teuthology.orchestra.run.smithi085.stdout:28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 2021-02-10T04:49:44.942 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:49:44 smithi085 conmon[45443]: debug 2021-02-10T04:49:44.942 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:49:44 smithi085 conmon[45443]: 2021-02-10T04:49:44.542+0000 7f71435f5700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:49:44.942 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:49:44 smithi085 conmon[45443]: 2021-02-10T04:49:45.899 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is unset 2021-02-10T04:49:45.906 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.1 to 1 (10000) 2021-02-10T04:49:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:45 smithi164 conmon[29288]: cluster 2021-02-10T04:49:44.382891+0000 mgr.y ( 2021-02-10T04:49:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:45 smithi164 conmon[29288]: mgr.14140) 1142 : cluster [DBG] pgmap v1924: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:49:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:45 smithi164 conmon[29288]: audit 2021-02-10T04:49:45.788549+0000 mon.c (mon.1) 82 : audit [INF] from='client.? 172.21.15.85:0/753494368' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:49:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:45 smithi164 conmon[29288]: audit 2021-02-10T04:49:45.788964+0000 2021-02-10T04:49:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:45 smithi164 conmon[29288]: mon.a (mon.0) 2195 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:49:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:45 smithi164 conmon[29288]: audit 2021-02-10T04:49:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:45 smithi164 conmon[29288]: 2021-02-10T04:49:45 2021-02-10T04:49:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:45 smithi164 conmon[29288]: .797481+0000 mon.a (mon.0) 2196 : audit [INF] from='client.? 172.21.15.85:0/3986657332' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-02-10T04:49:46.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:45 smithi085 conmon[29319]: cluster 2021-02-10T04:49:44.382891+0000 2021-02-10T04:49:46.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:45 smithi085 conmon[29319]: mgr.y (mgr.14140) 1142 : cluster [DBG] pgmap v1924: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:49:46.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:45 smithi085 conmon[29319]: audit 2021-02-10T04:49:45.788549+0000 mon.c (mon.1) 82 : audit 2021-02-10T04:49:46.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:45 smithi085 conmon[29319]: [INF] from='client.? 172.21.15.85:0/753494368' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:49:46.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:45 smithi085 conmon[29319]: audit 2021 2021-02-10T04:49:46.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:45 smithi085 conmon[29319]: -02-10T04:49:45.788964+0000 mon.a (mon.0) 2195 2021-02-10T04:49:46.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:45 smithi085 conmon[29319]: : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:49:46.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:45 smithi085 conmon[29319]: audit 2021-02-10T04:49 2021-02-10T04:49:46.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:45 smithi085 conmon[29319]: :45.797481+0000 mon.a (mon.0) 2196 : audit [INF] 2021-02-10T04:49:46.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:45 smithi085 conmon[29319]: from='client.? 172.21.15.85:0/3986657332' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-02-10T04:49:46.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:45 smithi085 conmon[35961]: cluster 2021-02-10T04:49:44.382891+0000 mgr.y (mgr.14140) 1142 : cluster [DBG] pgmap v1924: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:49:46.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:45 smithi085 conmon[35961]: audit 2021-02-10T04:49:45.788549+0000 mon.c (mon.1) 82 : audit [INF] from='client.? 172.21.15.85:0/753494368' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:49:46.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:45 smithi085 conmon[35961]: audit 2021-02-10T04:49:45.788964+0000 mon.a (mon.0) 2021-02-10T04:49:46.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:45 smithi085 conmon[35961]: 2195 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:49:46.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:45 smithi085 conmon[35961]: audit 2021-02-10T04:49:45.797481+0000 mon.a (mon.0) 2196 : audit 2021-02-10T04:49:46.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:45 smithi085 conmon[35961]: [INF] from='client.? 172.21.15.85:0/3986657332' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-02-10T04:49:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:46 smithi164 conmon[29288]: cluster 2021-02-10T04:49:45.682019 2021-02-10T04:49:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:46 smithi164 conmon[29288]: +0000 osd.7 (osd.7) 80 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:49:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:46 smithi164 conmon[29288]: cluster 2021-02-10T04:49:45.684439+0000 osd.7 (osd.7) 2021-02-10T04:49:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:46 smithi164 conmon[29288]: 81 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:49:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:46 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:49:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:46 smithi164 conmon[29288]: -02-10T04:49:45.879370+0000 mon.a (mon.0) 2197 : cluster [INF] 2021-02-10T04:49:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:46 smithi164 conmon[29288]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:49:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:46 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:49:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:46 smithi164 conmon[29288]: 10T04:49:45.879403+0000 mon.a (mon.0) 2198 2021-02-10T04:49:47.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:46 smithi164 conmon[29288]: : cluster [INF] Cluster is now healthy 2021-02-10T04:49:47.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:46 smithi164 conmon[29288]: audit 2021-02-10T04:49:47.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:46 smithi164 conmon[29288]: 2021-02-10T04:49:45.891215+0000 mon.a (mon.0) 2199 2021-02-10T04:49:47.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:46 smithi164 conmon[29288]: : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:49:47.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:46 smithi164 conmon[29288]: audit 2021-02-10T04:49:47.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:46 smithi164 conmon[29288]: 2021-02-10T04:49:45.891396 2021-02-10T04:49:47.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:46 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2200 : audit [INF] 2021-02-10T04:49:47.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:46 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/3986657332' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-02-10T04:49:47.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:46 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:49:47.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:46 smithi164 conmon[29288]: 04:49:45.891503+0000 2021-02-10T04:49:47.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:46 smithi164 conmon[29288]: mon.a (mon.0) 2201 : cluster 2021-02-10T04:49:47.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:46 smithi164 conmon[29288]: [DBG] osdmap e1092: 8 total, 8 up, 7 in 2021-02-10T04:49:47.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: cluster 2021-02-10T04:49:47.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: 2021-02-10T04:49:47.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: 2021 2021-02-10T04:49:47.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: - 2021-02-10T04:49:47.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: 02-10T04:49:45 2021-02-10T04:49:47.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: .682019+0000 osd.7 (osd.7) 80 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:49:47.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: cluster 2021-02-10T04:49:47.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: 2021-02-10T04:49:47.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: 2021 2021-02-10T04:49:47.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: - 2021-02-10T04:49:47.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: 02 2021-02-10T04:49:47.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: - 2021-02-10T04:49:47.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: 10 2021-02-10T04:49:47.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: T 2021-02-10T04:49:47.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: 04 2021-02-10T04:49:47.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: : 2021-02-10T04:49:47.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: 49 2021-02-10T04:49:47.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: : 2021-02-10T04:49:47.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: 45 2021-02-10T04:49:47.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: . 2021-02-10T04:49:47.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: 684439 2021-02-10T04:49:47.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: +0000 2021-02-10T04:49:47.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: osd.7 (osd.7) 81 : cluster 2021-02-10T04:49:47.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: [DBG] purged_snaps scrub ok 2021-02-10T04:49:47.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: cluster 2021-02-10T04:49:47.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: 2021-02-10T04:49:45 2021-02-10T04:49:47.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: .879370+0000 mon.a (mon.0) 2197 : cluster [INF] 2021-02-10T04:49:47.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:49:47.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:49:47.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: -10T04:49:45.879403+0000 mon.a ( 2021-02-10T04:49:47.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: mon.0) 2198 : cluster [INF] 2021-02-10T04:49:47.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: Cluster is now healthy 2021-02-10T04:49:47.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: audit 2021-02-10T 2021-02-10T04:49:47.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: 04:49:45.891215+0000 mon.a (mon.0) 2021-02-10T04:49:47.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: 2199 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:49:47.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:49:47.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: -10T04:49:45.891396+0000 2021-02-10T04:49:47.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: mon.a (mon.0) 2200 : audit [INF] from='client.? 172.21.15.85:0/3986657332' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-02-10T04:49:47.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: 2021-02-10T04:49:47.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: cluster 2021-02-10T04:49:47.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: 2021-02-10T04:49:45.891503+0000 mon.a 2021-02-10T04:49:47.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[29319]: (mon.0) 2201 : cluster [DBG] osdmap e1092: 8 total, 8 up, 7 in 2021-02-10T04:49:47.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[35961]: cluster 2021-02-10T04:49:45. 2021-02-10T04:49:47.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[35961]: 682019+0000 osd.7 (osd.7) 80 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:49:47.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[35961]: cluster 2021-02-10T04:49:47.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[35961]: 2021 2021-02-10T04:49:47.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[35961]: - 2021-02-10T04:49:47.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[35961]: 02-10T04:49:45. 2021-02-10T04:49:47.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[35961]: 684439+0000 osd.7 (osd.7) 81 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:49:47.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:49:47.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[35961]: 10T04:49:45.879370+0000 mon.a (mon.0) 2021-02-10T04:49:47.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[35961]: 2197 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:49:47.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:49:47.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[35961]: 02-10T04:49:45.879403+0000 2021-02-10T04:49:47.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[35961]: mon.a (mon. 2021-02-10T04:49:47.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[35961]: 0) 2198 : cluster [INF] Cluster is now healthy 2021-02-10T04:49:47.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:49:47.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[35961]: 02-10T04:49:45.891215+0000 mon.a (mon. 2021-02-10T04:49:47.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[35961]: 0) 2199 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:49:47.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:49:47.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[35961]: 02-10T04:49:45.891396+0000 mon.a ( 2021-02-10T04:49:47.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[35961]: mon.0) 2200 : audit [INF] from='client.? 172.21.15.85:0/3986657332' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-02-10T04:49:47.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[35961]: cluster 2021-02-10T04:49:47.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[35961]: 2021-02-10T04:49 2021-02-10T04:49:47.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[35961]: :45.891503+0000 mon.a (mon 2021-02-10T04:49:47.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[35961]: .0) 2201 : cluster 2021-02-10T04:49:47.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:46 smithi085 conmon[35961]: [DBG] osdmap e1092: 8 total, 8 up, 7 in 2021-02-10T04:49:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:47 smithi164 conmon[29288]: cluster 2021-02-10T04:49:46.383322+0000 mgr.y (mgr 2021-02-10T04:49:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:47 smithi164 conmon[29288]: .14140) 1143 : cluster [DBG] pgmap v1926: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:49:48.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:47 smithi085 conmon[35961]: cluster 2021-02-10T04:49:46.383322+0000 2021-02-10T04:49:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:47 smithi085 conmon[35961]: mgr.y (mgr.14140) 1143 : cluster [DBG] pgmap v1926: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:49:48.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:47 smithi085 conmon[29319]: cluster 2021-02-10T04:49:48.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:47 smithi085 conmon[29319]: 2021-02-10T04:49:48.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:47 smithi085 conmon[29319]: 2021 2021-02-10T04:49:48.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:47 smithi085 conmon[29319]: -02-10T04: 2021-02-10T04:49:48.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:47 smithi085 conmon[29319]: 49:46.383322+0000 mgr.y (mgr.14140) 1143 : cluster [DBG] pgmap v1926: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:49:48.691 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:49:48 smithi085 conmon[41540]: debug 2021-02-10T04:49:48.437+0000 7fb94e368700 -1 osd.0 pg_epoch: 1092 pg[1.0( empty local-lis/les=130/131 n=0 ec=19/19 lis/c=130/130 les/c/f=131/131/0 sis=130) [0,7,6] r=0 lpr=130 crt=0'0 mlcod 0'0 active+clean] scrubber pg(1.0) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6 (0x555d3c8aa2c0) 2021-02-10T04:49:49.394 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:49:48 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:49:50.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:49 smithi085 conmon[35961]: cluster 2021-02-10T04:49:48.384062 2021-02-10T04:49:50.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:49 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1144 : cluster [DBG] pgmap v1927: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:50.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:49 smithi085 conmon[35961]: audit 2021-02-10T04:49:48.970025+0000 mgr.y (mgr.14140 2021-02-10T04:49:50.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:49 smithi085 conmon[35961]: ) 1145 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:49:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:49 smithi085 conmon[29319]: cluster 2021-02-10T04:49:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:49 smithi085 conmon[29319]: 2021-02-10T04:49:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:49 smithi085 conmon[29319]: 2021-02-10T04:49:48.384062+0000 mgr.y 2021-02-10T04:49:50.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:49 smithi085 conmon[29319]: (mgr.14140) 1144 : cluster [DBG] pgmap v1927: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:50.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:49 smithi085 conmon[29319]: audit 2021-02-10T04:49:50.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:49 smithi085 conmon[29319]: 2021-02 2021-02-10T04:49:50.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:49 smithi085 conmon[29319]: -10T04:49:48.970025+0000 mgr.y (mgr.14140) 1145 : 2021-02-10T04:49:50.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:49 smithi085 conmon[29319]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:49:50.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:49 smithi164 conmon[29288]: cluster 2021-02-10T04:49:48.384062+0000 mgr.y 2021-02-10T04:49:50.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:49 smithi164 conmon[29288]: (mgr.14140) 1144 : cluster [DBG] pgmap v1927: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:50.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:49 smithi164 conmon[29288]: audit 2021-02-10T04:49:48.970025+0000 mgr.y (mgr.14140) 1145 : audit [DBG] 2021-02-10T04:49:50.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:49 smithi164 conmon[29288]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:49:52.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:51 smithi085 conmon[35961]: cluster 2021-02-10T04:49:50.384866+0000 2021-02-10T04:49:52.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:51 smithi085 conmon[35961]: mgr.y (mgr.14140) 1146 : cluster [DBG] pgmap v1928: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:52.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:51 smithi085 conmon[29319]: cluster 2021-02-10T04:49:50.384866+0000 mgr.y ( 2021-02-10T04:49:52.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:51 smithi085 conmon[29319]: mgr.14140) 1146 : cluster [DBG] pgmap v1928: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:52.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:51 smithi164 conmon[29288]: cluster 2021-02-10T04:49:50.384866 2021-02-10T04:49:52.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:51 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1146 : cluster [DBG] pgmap v1928: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:53.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:52 smithi085 conmon[35961]: cluster 2021-02-10T04:49:51.956933 2021-02-10T04:49:53.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:52 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2371 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:49:53.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:52 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:49:53.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:52 smithi085 conmon[35961]: :49:51.958475+0000 osd.4 (osd.4) 2372 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:49:53.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:52 smithi085 conmon[29319]: cluster 2021-02-10T04:49:51.956933+0000 osd.4 (osd. 2021-02-10T04:49:53.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:52 smithi085 conmon[29319]: 4) 2371 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:49:53.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:52 smithi085 conmon[29319]: cluster 2021-02-10T04:49:51.958475+0000 osd.4 (osd.4) 2372 : cluster 2021-02-10T04:49:53.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:52 smithi085 conmon[29319]: [DBG] purged_snaps scrub ok 2021-02-10T04:49:53.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:52 smithi164 conmon[29288]: cluster 2021-02-10T04:49:53.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:52 smithi164 conmon[29288]: 2021-02-10T04:49:53.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:52 smithi164 conmon[29288]: 2021 2021-02-10T04:49:53.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:52 smithi164 conmon[29288]: - 2021-02-10T04:49:53.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:52 smithi164 conmon[29288]: 02-10T04:49:51.956933+0000 2021-02-10T04:49:53.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:52 smithi164 conmon[29288]: osd.4 (osd.4) 2371 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:49:53.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:52 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:49:53.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:52 smithi164 conmon[29288]: 49:51.958475+0000 osd.4 (osd.4) 2372 : 2021-02-10T04:49:53.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:52 smithi164 conmon[29288]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:49:54.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:53 smithi085 conmon[35961]: cluster 2021-02-10T04:49:54.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:53 smithi085 conmon[35961]: 2021-02-10T04:49:54.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:53 smithi085 conmon[35961]: 2021 2021-02-10T04:49:54.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:53 smithi085 conmon[35961]: -02-10T04: 2021-02-10T04:49:54.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:53 smithi085 conmon[35961]: 49:52.385442+0000 mgr.y (mgr.14140) 1147 : cluster [DBG] pgmap v1929: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:54.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:53 smithi085 conmon[35961]: cluster 2021-02-10T04:49:54.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:53 smithi085 conmon[35961]: 2021-02-10T04:49 2021-02-10T04:49:54.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:53 smithi085 conmon[35961]: :52.909338+0000 mon.a (mon.0) 2202 : cluster [DBG] osdmap e1093: 8 total, 8 up, 7 in 2021-02-10T04:49:54.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:53 smithi085 conmon[29319]: cluster 2021-02-10T04:49:52.385442+0000 mgr.y (mgr.14140) 1147 : cluster [DBG] pgmap v1929: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:54.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:53 smithi085 conmon[29319]: cluster 2021-02-10T04:49:52.909338+0000 mon.a (mon.0) 2202 : cluster [DBG] osdmap e1093: 8 total, 8 up, 7 in 2021-02-10T04:49:54.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:53 smithi164 conmon[29288]: cluster 2021-02-10T04:49:52.385442+0000 2021-02-10T04:49:54.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:53 smithi164 conmon[29288]: mgr.y (mgr.14140) 1147 : cluster [DBG] pgmap v1929: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:54.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:53 smithi164 conmon[29288]: cluster 2021-02-10T04:49:52.909338+0000 mon.a (mon.0) 2202 : cluster 2021-02-10T04:49:54.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:53 smithi164 conmon[29288]: [DBG] osdmap e1093: 8 total, 8 up, 7 in 2021-02-10T04:49:54.942 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:49:54 smithi085 conmon[41540]: debug 2021-02-10T04:49:54.942 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:49:54 smithi085 conmon[41540]: 2021-02-10T04:49:54.487+0000 7fb94a360700 -1 osd.0 pg_epoch: 1093 pg[1.0( empty local-lis/les=130/131 n=0 ec=19/19 lis/c=130/130 les/c/f=131/131/0 sis=130) [0,7,6] r=0 lpr=130 crt=0'0 mlcod 0'0 active+clean] scrubber pg(1.0) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6 (0x555d3ad758c0) 2021-02-10T04:49:54.942 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:49:54 smithi085 conmon[41540]: 2021-02-10T04:49:56.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:55 smithi085 conmon[29319]: cluster 2021-02-10T04:49:54.386046+0000 2021-02-10T04:49:56.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:55 smithi085 conmon[29319]: mgr.y (mgr.14140) 1148 : cluster [DBG] pgmap v1931: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:49:56.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:55 smithi085 conmon[35961]: cluster 2021-02-10T04:49:56.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:55 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:49:56.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:55 smithi085 conmon[35961]: 04:49:54.386046+0000 mgr.y (mgr.14140) 1148 : cluster [DBG] pgmap v1931: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:49:56.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:55 smithi164 conmon[29288]: cluster 2021-02-10T04:49:54.386046+0000 mgr.y (mgr 2021-02-10T04:49:56.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:55 smithi164 conmon[29288]: .14140) 1148 : cluster [DBG] pgmap v1931: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:49:56.566 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:49:56.604 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:49:57.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:56 smithi085 conmon[29319]: cluster 2021-02-10T04:49:57.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:56 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:49:57.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:56 smithi085 conmon[29319]: 49:56.561031+0000 osd.1 (osd.1) 477 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:49:57.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:56 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:49:57.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:56 smithi085 conmon[29319]: 10T04:49:56.564413+0000 osd.1 (osd.1) 478 : cluster [DBG] 2021-02-10T04:49:57.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:56 smithi085 conmon[29319]: purged_snaps scrub ok 2021-02-10T04:49:57.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:56 smithi085 conmon[35961]: cluster 2021-02-10T04:49:57.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:56 smithi085 conmon[35961]: 2021-02-10T04:49:57.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:56 smithi085 conmon[35961]: 2021 2021-02-10T04:49:57.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:56 smithi085 conmon[35961]: - 2021-02-10T04:49:57.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:56 smithi085 conmon[35961]: 02 2021-02-10T04:49:57.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:56 smithi085 conmon[35961]: - 2021-02-10T04:49:57.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:56 smithi085 conmon[35961]: 10 2021-02-10T04:49:57.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:56 smithi085 conmon[35961]: T04:49:56.561031+0000 osd.1 2021-02-10T04:49:57.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:56 smithi085 conmon[35961]: (osd.1) 477 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:49:57.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:56 smithi085 conmon[35961]: cluster 2021-02-10T04:49 2021-02-10T04:49:57.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:56 smithi085 conmon[35961]: :56.564413+0000 osd.1 (osd.1) 478 : cluster 2021-02-10T04:49:57.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:56 smithi085 conmon[35961]: [DBG] purged_snaps scrub ok 2021-02-10T04:49:57.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:56 smithi164 conmon[29288]: cluster 2021-02-10T04:49:56.561031+0000 osd.1 (osd.1) 2021-02-10T04:49:57.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:56 smithi164 conmon[29288]: 477 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:49:57.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:56 smithi164 conmon[29288]: cluster 2021-02-10T04:49:56.564413+0000 osd.1 (osd.1) 478 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:49:58.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:57 smithi085 conmon[29319]: cluster 2021-02-10T04:49:56.386784+0000 2021-02-10T04:49:58.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:57 smithi085 conmon[29319]: mgr.y (mgr.14140) 1149 : cluster [DBG] pgmap v1932: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:58.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:57 smithi085 conmon[29319]: cluster 2021-02-10T04:49:56.936054+0000 mon.a (mon.0) 2203 : cluster [DBG] osdmap e1094: 8 total, 8 up, 7 in 2021-02-10T04:49:58.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:57 smithi085 conmon[35961]: cluster 2021-02-10T04:49:56.386784 2021-02-10T04:49:58.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:57 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1149 : cluster [DBG] pgmap v1932: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:58.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:57 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:49:58.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:57 smithi085 conmon[35961]: T04:49:56.936054+0000 mon.a (mon.0) 2203 : cluster [DBG] osdmap e1094: 8 total, 8 up, 7 in 2021-02-10T04:49:58.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:57 smithi164 conmon[29288]: cluster 2021-02-10T04:49:56.386784+0000 mgr.y ( 2021-02-10T04:49:58.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:57 smithi164 conmon[29288]: mgr.14140) 1149 : cluster [DBG] pgmap v1932: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:49:58.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:57 smithi164 conmon[29288]: cluster 2021-02-10T04:49:56.936054+0000 mon.a (mon.0) 2203 : cluster [DBG] osdmap e1094: 8 total, 8 up, 7 in 2021-02-10T04:49:59.394 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:49:58 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:50:00.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:59 smithi085 conmon[35961]: cluster 2021-02-10T04:49:58.387510+0000 2021-02-10T04:50:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:59 smithi085 conmon[35961]: mgr.y (mgr.14140) 1150 : cluster [DBG] pgmap v1934: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:50:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:59 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:50:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:59 smithi085 conmon[35961]: -10T04:49:58.979998+0000 mgr.y (mgr.14140) 1151 : audit 2021-02-10T04:50:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:49:59 smithi085 conmon[35961]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:50:00.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:59 smithi085 conmon[29319]: cluster 2021-02-10T04:50:00.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:59 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:50:00.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:59 smithi085 conmon[29319]: 49:58.387510+0000 mgr.y (mgr.14140) 1150 : cluster [DBG] pgmap v1934: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:50:00.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:59 smithi085 conmon[29319]: 2021-02-10T04:50:00.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:59 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:50:00.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:59 smithi085 conmon[29319]: 02-10T04:49:58.979998+0000 mgr.y (mgr.14140) 1151 : audit [DBG] 2021-02-10T04:50:00.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:49:59 smithi085 conmon[29319]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:50:00.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:59 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:50:00.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:59 smithi164 conmon[29288]: 04:49:58.387510+0000 mgr.y (mgr.14140) 1150 : cluster [DBG] pgmap v1934: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:50:00.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:59 smithi164 conmon[29288]: audit 2021-02-10T04:49:58.979998 2021-02-10T04:50:00.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:49:59 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1151 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:50:01.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:00 smithi164 conmon[29288]: cluster 2021-02-10T04:50:01.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:00 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:50:01.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:00 smithi164 conmon[29288]: 04:50:00.000094+0000 mon.a (mon.0) 2204 : cluster [INF] overall HEALTH_OK 2021-02-10T04:50:01.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:00 smithi085 conmon[29319]: cluster 2021-02-10T04:50:01.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:00 smithi085 conmon[29319]: 2021-02-10T04:50 2021-02-10T04:50:01.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:00 smithi085 conmon[29319]: :00.000094+0000 mon.a (mon.0) 2204 : cluster [INF] overall HEALTH_OK 2021-02-10T04:50:01.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:00 smithi085 conmon[35961]: cluster 2021-02-10T04:50:00.000094+0000 mon.a (mon.0 2021-02-10T04:50:01.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:00 smithi085 conmon[35961]: ) 2204 : cluster [INF] overall HEALTH_OK 2021-02-10T04:50:02.125 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: *** Caught signal (Segmentation fault) ** 2021-02-10T04:50:02.125 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: in thread 7fba950ba700 thread_name:msgr-worker-2 2021-02-10T04:50:02.125 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: ceph version 17.0.0-681-gc1ea6241 (c1ea624123d412aff8b9d1430e36cb45fcab76b8) quincy (dev) 2021-02-10T04:50:02.125 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fba9a0abb20] 2021-02-10T04:50:02.125 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 2: (std::_Rb_tree, boost::intrusive_ptr, std::_Identity >, std::less >, std::allocator > >::find(boost::intrusive_ptr const&) const+0x2c) [0x55fea8bcc48c] 2021-02-10T04:50:02.126 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 3: (AsyncConnection::_stop()+0xab) [0x55fea8bc663b] 2021-02-10T04:50:02.126 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 4: (ProtocolV2::stop()+0x8f) [0x55fea8bf171f] 2021-02-10T04:50:02.126 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 5: (ProtocolV2::handle_existing_connection(boost::intrusive_ptr const&)+0x742) [0x55fea8c06e62] 2021-02-10T04:50:02.126 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 6: (ProtocolV2::handle_client_ident(ceph::buffer::v15_2_0::list&)+0xeef) [0x55fea8c086ff] 2021-02-10T04:50:02.126 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 7: (ProtocolV2::handle_frame_payload()+0x20b) [0x55fea8c08d0b] 2021-02-10T04:50:02.127 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 8: (ProtocolV2::handle_read_frame_dispatch()+0x160) [0x55fea8c08f90] 2021-02-10T04:50:02.127 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 9: (ProtocolV2::_handle_read_frame_epilogue_main()+0x95) [0x55fea8c09185] 2021-02-10T04:50:02.127 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 10: (ProtocolV2::_handle_read_frame_segment()+0x92) [0x55fea8c09232] 2021-02-10T04:50:02.127 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 11: (ProtocolV2::handle_read_frame_segment(std::unique_ptr&&, int)+0x201) [0x55fea8c0a381] 2021-02-10T04:50:02.127 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 12: (ProtocolV2::run_continuation(Ct&)+0x3c) [0x55fea8bf25bc] 2021-02-10T04:50:02.128 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 13: (AsyncConnection::process()+0x789) [0x55fea8bc96d9] 2021-02-10T04:50:02.128 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 14: (EventCenter::process_events(unsigned int, std::chrono::duration >*)+0xcb7) [0x55fea8a19e37] 2021-02-10T04:50:02.128 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 15: /usr/bin/ceph-osd(+0xe8a95c) [0x55fea8a1d95c] 2021-02-10T04:50:02.128 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 16: /lib64/libstdc++.so.6(+0xc2ba3) [0x7fba996fbba3] 2021-02-10T04:50:02.128 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 17: /lib64/libpthread.so.0(+0x814a) [0x7fba9a0a114a] 2021-02-10T04:50:02.128 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 18: clone() 2021-02-10T04:50:02.129 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: debug 2021-02-10T04:50:01.886+0000 7fba950ba700 -1 *** Caught signal (Segmentation fault) ** 2021-02-10T04:50:02.129 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: in thread 7fba950ba700 thread_name:msgr-worker-2 2021-02-10T04:50:02.129 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 2021-02-10T04:50:02.129 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: ceph version 17.0.0-681-gc1ea6241 (c1ea624123d412aff8b9d1430e36cb45fcab76b8) quincy (dev) 2021-02-10T04:50:02.129 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fba9a0abb20] 2021-02-10T04:50:02.130 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 2: (std::_Rb_tree, boost::intrusive_ptr, std::_Identity >, std::less >, std::allocator > >::find(boost::intrusive_ptr const&) const+0x2c) [0x55fea8bcc48c] 2021-02-10T04:50:02.130 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 3: (AsyncConnection::_stop()+0xab) [0x55fea8bc663b] 2021-02-10T04:50:02.130 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 4: (ProtocolV2::stop()+0x8f) [0x55fea8bf171f] 2021-02-10T04:50:02.130 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 5: (ProtocolV2::handle_existing_connection(boost::intrusive_ptr const&)+0x742) [0x55fea8c06e62] 2021-02-10T04:50:02.130 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 6: (ProtocolV2::handle_client_ident(ceph::buffer::v15_2_0::list&)+0xeef) [0x55fea8c086ff] 2021-02-10T04:50:02.130 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 7: (ProtocolV2::handle_frame_payload()+0x20b) [0x55fea8c08d0b] 2021-02-10T04:50:02.131 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 8: (ProtocolV2::handle_read_frame_dispatch()+0x160) [0x55fea8c08f90] 2021-02-10T04:50:02.131 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 9: (ProtocolV2::_handle_read_frame_epilogue_main()+0x95) [0x55fea8c09185] 2021-02-10T04:50:02.131 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 10: (ProtocolV2::_handle_read_frame_segment()+0x92) [0x55fea8c09232] 2021-02-10T04:50:02.131 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 11: (ProtocolV2::handle_read_frame_segment(std::unique_ptr&&, int)+0x201) [0x55fea8c0a381] 2021-02-10T04:50:02.131 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 12: (ProtocolV2::run_continuation(Ct&)+0x3c) [0x55fea8bf25bc] 2021-02-10T04:50:02.132 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 13: (AsyncConnection::process()+0x789) [0x55fea8bc96d9] 2021-02-10T04:50:02.132 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 14: (EventCenter::process_events(unsigned int, std::chrono::duration >*)+0xcb7) [0x55fea8a19e37] 2021-02-10T04:50:02.132 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 15: /usr/bin/ceph-osd(+0xe8a95c) [0x55fea8a1d95c] 2021-02-10T04:50:02.132 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 16: /lib64/libstdc++.so.6(+0xc2ba3) [0x7fba996fbba3] 2021-02-10T04:50:02.132 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 17: /lib64/libpthread.so.0(+0x814a) [0x7fba9a0a114a] 2021-02-10T04:50:02.132 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 18: clone() 2021-02-10T04:50:02.133 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-02-10T04:50:02.133 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 2021-02-10T04:50:02.133 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: debug -10> 2021-02-10T04:50:01.886+0000 7fba950ba700 -1 *** Caught signal (Segmentation fault) ** 2021-02-10T04:50:02.133 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: in thread 7fba950ba700 thread_name:msgr-worker-2 2021-02-10T04:50:02.133 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 2021-02-10T04:50:02.134 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: ceph version 17.0.0-681-gc1ea6241 (c1ea624123d412aff8b9d1430e36cb45fcab76b8) quincy (dev) 2021-02-10T04:50:02.134 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fba9a0abb20] 2021-02-10T04:50:02.134 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 2: (std::_Rb_tree, boost::intrusive_ptr, std::_Identity >, std::less >, std::allocator > >::find(boost::intrusive_ptr const&) const+0x2c) [0x55fea8bcc48c] 2021-02-10T04:50:02.134 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 3: (AsyncConnection::_stop()+0xab) [0x55fea8bc663b] 2021-02-10T04:50:02.134 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 4: (ProtocolV2::stop()+0x8f) [0x55fea8bf171f] 2021-02-10T04:50:02.134 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 5: (ProtocolV2::handle_existing_connection(boost::intrusive_ptr const&)+0x742) [0x55fea8c06e62] 2021-02-10T04:50:02.135 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 6: (ProtocolV2::handle_client_ident(ceph::buffer::v15_2_0::list&)+0xeef) [0x55fea8c086ff] 2021-02-10T04:50:02.135 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 7: (ProtocolV2::handle_frame_payload()+0x20b) [0x55fea8c08d0b] 2021-02-10T04:50:02.135 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 8: (ProtocolV2::handle_read_frame_dispatch()+0x160) [0x55fea8c08f90] 2021-02-10T04:50:02.135 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 9: (ProtocolV2::_handle_read_frame_epilogue_main()+0x95) [0x55fea8c09185] 2021-02-10T04:50:02.135 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 10: (ProtocolV2::_handle_read_frame_segment()+0x92) [0x55fea8c09232] 2021-02-10T04:50:02.136 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 11: (ProtocolV2::handle_read_frame_segment(std::unique_ptr&&, int)+0x201) [0x55fea8c0a381] 2021-02-10T04:50:02.136 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 12: (ProtocolV2::run_continuation(Ct&)+0x3c) [0x55fea8bf25bc] 2021-02-10T04:50:02.136 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 13: (AsyncConnection::process()+0x789) [0x55fea8bc96d9] 2021-02-10T04:50:02.136 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 14: (EventCenter::process_events(unsigned int, std::chrono::duration >*)+0xcb7) [0x55fea8a19e37] 2021-02-10T04:50:02.136 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 15: /usr/bin/ceph-osd(+0xe8a95c) [0x55fea8a1d95c] 2021-02-10T04:50:02.136 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 16: /lib64/libstdc++.so.6(+0xc2ba3) [0x7fba996fbba3] 2021-02-10T04:50:02.137 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 17: /lib64/libpthread.so.0(+0x814a) [0x7fba9a0a114a] 2021-02-10T04:50:02.137 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 18: clone() 2021-02-10T04:50:02.137 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-02-10T04:50:02.137 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 2021-02-10T04:50:02.137 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: debug -9999> 2021-02-10T04:50:01.886+0000 7fba950ba700 -1 *** Caught signal (Segmentation fault) ** 2021-02-10T04:50:02.138 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: in thread 7fba950ba700 thread_name:msgr-worker-2 2021-02-10T04:50:02.138 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 2021-02-10T04:50:02.138 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: ceph version 17.0.0-681-gc1ea6241 (c1ea624123d412aff8b9d1430e36cb45fcab76b8) quincy (dev) 2021-02-10T04:50:02.138 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fba9a0abb20] 2021-02-10T04:50:02.138 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 2: (std::_Rb_tree, boost::intrusive_ptr, std::_Identity >, std::less >, std::allocator > >::find(boost::intrusive_ptr const&) const+0x2c) [0x55fea8bcc48c] 2021-02-10T04:50:02.138 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 3: (AsyncConnection::_stop()+0xab) [0x55fea8bc663b] 2021-02-10T04:50:02.139 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 4: (ProtocolV2::stop()+0x8f) [0x55fea8bf171f] 2021-02-10T04:50:02.139 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 5: (ProtocolV2::handle_existing_connection(boost::intrusive_ptr const&)+0x742) [0x55fea8c06e62] 2021-02-10T04:50:02.139 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 6: (ProtocolV2::handle_client_ident(ceph::buffer::v15_2_0::list&)+0xeef) [0x55fea8c086ff] 2021-02-10T04:50:02.139 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 7: (ProtocolV2::handle_frame_payload()+0x20b) [0x55fea8c08d0b] 2021-02-10T04:50:02.139 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 8: (ProtocolV2::handle_read_frame_dispatch()+0x160) [0x55fea8c08f90] 2021-02-10T04:50:02.140 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 9: (ProtocolV2::_handle_read_frame_epilogue_main()+0x95) [0x55fea8c09185] 2021-02-10T04:50:02.140 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 10: (ProtocolV2::_handle_read_frame_segment()+0x92) [0x55fea8c09232] 2021-02-10T04:50:02.140 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 11: (ProtocolV2::handle_read_frame_segment(std::unique_ptr&&, int)+0x201) [0x55fea8c0a381] 2021-02-10T04:50:02.140 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 12: (ProtocolV2::run_continuation(Ct&)+0x3c) [0x55fea8bf25bc] 2021-02-10T04:50:02.140 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 13: (AsyncConnection::process()+0x789) [0x55fea8bc96d9] 2021-02-10T04:50:02.140 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 14: (EventCenter::process_events(unsigned int, std::chrono::duration >*)+0xcb7) [0x55fea8a19e37] 2021-02-10T04:50:02.141 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 15: /usr/bin/ceph-osd(+0xe8a95c) [0x55fea8a1d95c] 2021-02-10T04:50:02.141 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 16: /lib64/libstdc++.so.6(+0xc2ba3) [0x7fba996fbba3] 2021-02-10T04:50:02.141 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 17: /lib64/libpthread.so.0(+0x814a) [0x7fba9a0a114a] 2021-02-10T04:50:02.141 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 18: clone() 2021-02-10T04:50:02.141 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-02-10T04:50:02.142 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[52836]: 2021-02-10T04:50:02.142 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[29319]: cluster 2021-02-10T04:50:00 2021-02-10T04:50:02.142 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[29319]: .388334+0000 mgr.y (mgr.14140) 1152 : cluster [DBG] pgmap v1935: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:50:02.142 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[35961]: cluster 2021-02-10T04:50:00.388334+0000 mgr.y 2021-02-10T04:50:02.143 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:01 smithi085 conmon[35961]: (mgr.14140) 1152 : cluster [DBG] pgmap v1935: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:50:02.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:01 smithi164 conmon[29288]: cluster 2021-02-10T04:50:00.388334+0000 mgr.y (mgr.14140) 1152 : cluster [DBG] pgmap v1935: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:50:02.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:01 smithi164 conmon[29288]: 2021-02-10T04:50:02.475 INFO:teuthology.orchestra.run.smithi085.stdout:f14cf16c59c986d262ad12064a4385559674d8d460ef7bf160701a8280d5fac6 2021-02-10T04:50:02.941 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[41540]: debug 2021-02-10T04:50:02.467+0000 7fb94a360700 -1 osd.0 pg_epoch: 1094 pg[1.0( empty local-lis/les=130/131 n=0 ec=19/19 lis/c=130/130 les/c/f=131/131/0 sis=130) [0,7,6] r=0 lpr=130 crt=0'0 mlcod 0'0 active+clean] scrubber pg(1.0) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6 (0x555d327d2f20) 2021-02-10T04:50:03.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: cluster 2021-02-10T04:50:02.146002+0000 mon.a (mon 2021-02-10T04:50:03.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: .0) 2205 : cluster [DBG] osd.3 reported immediately failed by osd.0 2021-02-10T04:50:03.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: cluster 2021-02-10T04:50:02.146048+0000 mon.a (mon.0) 2206 : cluster [INF] osd.3 failed (root=default,host=smithi085) (connection refused reported by osd.0) 2021-02-10T04:50:03.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 2021-02-10T04:50:03.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: cluster 2021-02-10T04:50:02. 2021-02-10T04:50:03.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 345959+0000 mon.a (mon.0) 2207 : cluster [DBG] osd.3 reported immediately failed by osd.0 2021-02-10T04:50:03.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:50:03.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 02-10T04:50:02.346089+0000 2021-02-10T04:50:03.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: mon.a (mon.0) 2208 : cluster [DBG] osd.3 reported immediately failed by osd.2 2021-02-10T04:50:03.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 2021-02-10T04:50:03.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:50:03.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: -02-10T04:50: 2021-02-10T04:50:03.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 02.346199+0000 mon.a (mon 2021-02-10T04:50:03.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: .0) 2209 : cluster [DBG] 2021-02-10T04:50:03.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: osd.3 reported immediately failed by osd.2 2021-02-10T04:50:03.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: cluster 2021-02-10T04:50:03.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:50:03.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 50:02.346388+0000 mon.a 2021-02-10T04:50:03.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: (mon.0) 2210 : cluster 2021-02-10T04:50:03.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: [DBG] osd.3 reported immediately failed by osd.1 2021-02-10T04:50:03.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: cluster 2021-02-10T04:50:03.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 2021-02-10T04:50:02.346549+0000 mon.a (mon.0) 2211 : cluster [DBG] osd.3 reported immediately failed by osd.5 2021-02-10T04:50:03.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: cluster 2021-02-10T04:50: 2021-02-10T04:50:03.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 02.346671+0000 mon.a (mon.0) 2212 : cluster [DBG] osd.3 reported immediately failed by osd.1 2021-02-10T04:50:03.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:50:03.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 50:02.346807+0000 mon.a (mon.0) 2213 : cluster [DBG] 2021-02-10T04:50:03.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: osd.3 reported immediately failed by osd.4 2021-02-10T04:50:03.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: cluster 2021-02-10T04:50:03.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 2021-02-10T04:50:02.346931+0000 2021-02-10T04:50:03.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: mon.a (mon.0) 2214 : cluster [DBG] osd.3 reported immediately failed by osd.5 2021-02-10T04:50:03.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 2021-02-10T04:50:03.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: cluster 2021-02-10T04:50:03.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 2021-02-10T04:50:02 2021-02-10T04:50:03.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: .347045+0000 mon.a (mon.0) 2215 : cluster 2021-02-10T04:50:03.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: [DBG] osd.3 reported immediately failed by osd.6 2021-02-10T04:50:03.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:50:03.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: -10T04:50:02.347147+0000 2021-02-10T04:50:03.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: mon.a (mon.0) 2216 : cluster [DBG] 2021-02-10T04:50:03.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: osd.3 reported immediately failed by osd.7 2021-02-10T04:50:03.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: cluster 2021-02-10T04:50:03.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 2021-02-10T04:50: 2021-02-10T04:50:03.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 02.347263+0000 mon.a (mon.0) 2217 : cluster [DBG] 2021-02-10T04:50:03.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: osd.3 reported immediately failed by osd.4 2021-02-10T04:50:03.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:50:03.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: -02-10T04:50:02.347376+0000 mon.a (mon.0) 2218 : cluster 2021-02-10T04:50:03.403 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: [DBG] osd.3 reported immediately failed by osd.6 2021-02-10T04:50:03.403 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:50:03.403 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 50:02.347489+0000 mon.a (mon.0) 2021-02-10T04:50:03.403 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 2219 : cluster [DBG] osd.3 reported immediately failed by osd.7 2021-02-10T04:50:03.403 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:50:03.404 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 10T04:50:02.347698+0000 mon.a (mon.0) 2220 : cluster 2021-02-10T04:50:03.404 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: [DBG] osd.3 reported immediately failed by osd.0 2021-02-10T04:50:03.404 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: cluster 2021-02-10T04:50:02 2021-02-10T04:50:03.404 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: .747038+0000 mon.a (mon.0) 2221 : cluster [DBG] osd.3 reported immediately failed by osd.2 2021-02-10T04:50:03.404 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:50:03.404 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: :50:02.747170+0000 mon.a (mon.0 2021-02-10T04:50:03.405 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: ) 2222 : cluster [DBG] osd.3 reported immediately failed by osd.0 2021-02-10T04:50:03.405 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:50:03.405 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: -02-10T04:50:02.747251+0000 2021-02-10T04:50:03.405 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: mon.a (mon.0) 2223 : cluster [DBG] osd.3 reported immediately failed by osd.2 2021-02-10T04:50:03.405 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 2021-02-10T04:50:03.405 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: cluster 2021-02-10T04:50:03.406 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 2021-02-10T04:50:02.747373+0000 mon.a ( 2021-02-10T04:50:03.406 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: mon.0) 2224 : cluster [DBG] osd.3 reported immediately failed by osd.1 2021-02-10T04:50:03.406 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: cluster 2021-02-10T04:50:03.406 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:50:03.406 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 04:50:02.747495+0000 mon.a (mon. 2021-02-10T04:50:03.407 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 0) 2225 : cluster [DBG] osd.3 reported immediately failed by osd.6 2021-02-10T04:50:03.407 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: cluster 2021-02-10T04:50:03.407 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:50:03.407 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: :50:02.747681+0000 mon.a (mon.0) 2021-02-10T04:50:03.408 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 2226 : cluster [DBG] osd.3 reported immediately failed by osd.1 2021-02-10T04:50:03.408 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:50:03.408 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 02-10T04:50:02.747834+0000 mon.a 2021-02-10T04:50:03.408 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: (mon.0) 2227 : cluster [DBG] osd.3 reported immediately failed by osd.4 2021-02-10T04:50:03.408 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:50:03.409 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 02-10T04:50:02.747968+0000 2021-02-10T04:50:03.409 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: mon.a (mon.0) 2228 : cluster [DBG] osd.3 reported immediately failed by osd.0 2021-02-10T04:50:03.409 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:50:03.409 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 10T04:50:02.748100 2021-02-10T04:50:03.409 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2021-02-10T04:50:03.409 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 2229 : cluster [DBG] osd.3 reported immediately failed by osd.7 2021-02-10T04:50:03.410 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:50:03.410 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 10T04:50:02.748240+0000 mon.a ( 2021-02-10T04:50:03.410 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: mon.0) 2230 : cluster [DBG] osd.3 reported immediately failed by osd.4 2021-02-10T04:50:03.410 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: cluster 2021-02-10T04:50:03.410 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 2021-02-10T04:50:02.748372+0000 2021-02-10T04:50:03.411 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: mon.a (mon.0) 2231 : cluster [DBG] osd.3 reported immediately failed by osd.7 2021-02-10T04:50:03.411 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 2021-02-10T04:50:03.411 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: cluster 2021-02-10T04:50:03.411 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: 2021-02-10T04:50:02.748540+0000 mon.a ( 2021-02-10T04:50:03.411 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: mon.0) 2232 : cluster [DBG] osd.3 reported immediately failed by osd.5 2021-02-10T04:50:03.411 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:50:03.411 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: T04:50:02.748694+0000 mon.a (mon 2021-02-10T04:50:03.412 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: .0) 2233 : cluster [DBG] osd.3 reported immediately failed by osd.5 2021-02-10T04:50:03.412 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:50:03.412 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: T04:50:02.748868+0000 mon.a (mon 2021-02-10T04:50:03.412 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:02 smithi164 conmon[29288]: .0) 2234 : cluster [DBG] osd.3 reported immediately failed by osd.6 2021-02-10T04:50:03.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster 2021-02-10T04:50:02 2021-02-10T04:50:03.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: .146002+0000 mon.a (mon.0) 2205 : cluster [DBG] osd.3 reported immediately failed by osd.0 2021-02-10T04:50:03.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster 2021-02-10T04:50:02.146048+0000 mon.a (mon. 2021-02-10T04:50:03.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: 0) 2206 : cluster [INF] osd.3 failed (root=default,host=smithi085) (connection refused reported by osd.0) 2021-02-10T04:50:03.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:50:03.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: 50:02.345959+0000 mon.a (mon.0) 2207 : cluster [DBG] osd.3 reported immediately failed by osd.0 2021-02-10T04:50:03.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster 2021-02-10T04:50:02.346089 2021-02-10T04:50:03.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2208 : cluster [DBG] 2021-02-10T04:50:03.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: osd.3 reported immediately failed by osd.2 2021-02-10T04:50:03.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:50:03.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: -02-10T04:50:02.346199+0000 mon.a (mon.0) 2021-02-10T04:50:03.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: 2209 : cluster [DBG] osd.3 reported immediately failed by osd.2 2021-02-10T04:50:03.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:50:03.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: 04:50:02.346388+0000 mon.a (mon.0 2021-02-10T04:50:03.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: ) 2210 : cluster [DBG] osd.3 reported immediately failed by osd.1 2021-02-10T04:50:03.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:50:03.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: 50:02.346549+0000 mon.a (mon.0) 2211 : 2021-02-10T04:50:03.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster [DBG] osd.3 reported immediately failed by osd.5 2021-02-10T04:50:03.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:50:03.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: 10T04:50:02.346671+0000 mon.a (mon 2021-02-10T04:50:03.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: .0) 2212 : cluster [DBG] osd.3 reported immediately failed by osd.1 2021-02-10T04:50:03.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: 2021-02-10T04:50:03.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:50:03.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: 10T04:50:02.346807+0000 mon.a (mon.0) 2213 : cluster 2021-02-10T04:50:03.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: [DBG] osd.3 reported immediately failed by osd.4 2021-02-10T04:50:03.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:50:03.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: 10T04:50:02.346931+0000 mon.a (mon.0 2021-02-10T04:50:03.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: ) 2214 : cluster [DBG] osd.3 reported immediately failed by osd.5 2021-02-10T04:50:03.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:50:03.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: 50:02.347045+0000 mon.a (mon.0) 2215 : 2021-02-10T04:50:03.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster [DBG] osd.3 reported immediately failed by osd.6 2021-02-10T04:50:03.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster 2021-02-10T04:50:03.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: 2021-02-10T04:50:02.347147+0000 mon.a (mon.0 2021-02-10T04:50:03.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: ) 2216 : cluster [DBG] osd.3 reported immediately failed by osd.7 2021-02-10T04:50:03.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:50:03.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: -02-10T04:50:02.347263 2021-02-10T04:50:03.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2217 : cluster [DBG] osd.3 reported immediately failed by osd.4 2021-02-10T04:50:03.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: 2021-02-10T04:50:03.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster 2021-02-10T04:50:03.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: 2021-02-10T04:50:02.347376+0000 mon.a ( 2021-02-10T04:50:03.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: mon.0) 2218 : cluster [DBG] 2021-02-10T04:50:03.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: osd.3 reported immediately failed by osd.6 2021-02-10T04:50:03.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:50:03.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: -02-10T04:50: 2021-02-10T04:50:03.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: 02.347489+0000 mon.a (mon 2021-02-10T04:50:03.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: .0) 2219 : cluster [DBG] osd.3 reported immediately failed by osd.7 2021-02-10T04:50:03.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: 2021-02-10T04:50:03.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:50:03.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: 04:50:02.347698+0000 2021-02-10T04:50:03.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: mon.a (mon.0) 2220 : cluster 2021-02-10T04:50:03.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: [DBG] osd.3 reported immediately failed by osd.0 2021-02-10T04:50:03.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:50:03.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: -10T04:50:02.747038+0000 mon.a 2021-02-10T04:50:03.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: (mon.0) 2221 : cluster [DBG] 2021-02-10T04:50:03.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: osd.3 reported immediately failed by osd.2 2021-02-10T04:50:03.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:50:03.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: -02-10T04:50 2021-02-10T04:50:03.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: :02.747170+0000 mon.a (mon 2021-02-10T04:50:03.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: .0) 2222 : cluster [DBG] 2021-02-10T04:50:03.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: osd.3 reported immediately failed by osd.0 2021-02-10T04:50:03.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:50:03.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: 02-10T04:50: 2021-02-10T04:50:03.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: 02.747251+0000 mon.a (mon. 2021-02-10T04:50:03.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: 0) 2223 : cluster [DBG] osd.3 reported immediately failed by osd.2 2021-02-10T04:50:03.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: 2021-02-10T04:50:03.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster 2021-02-10T04:50 2021-02-10T04:50:03.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: :02.747373+0000 mon.a (mon.0) 2224 : 2021-02-10T04:50:03.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster [DBG] osd.3 reported immediately failed by osd.1 2021-02-10T04:50:03.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster 2021-02-10T04:50:03.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: 2021-02-10T04:50: 2021-02-10T04:50:03.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: 02.747495+0000 mon.a (mon. 2021-02-10T04:50:03.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: 0) 2225 : cluster [DBG] osd.3 reported immediately failed by osd.6 2021-02-10T04:50:03.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:50:03.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: T04:50:02.747681+0000 mon.a (mon.0) 2226 2021-02-10T04:50:03.450 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: : cluster [DBG] osd.3 reported immediately failed by osd.1 2021-02-10T04:50:03.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:50:03.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: 02-10T04:50:02.747834 2021-02-10T04:50:03.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2227 2021-02-10T04:50:03.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: : cluster [DBG] osd.3 reported immediately failed by osd.4 2021-02-10T04:50:03.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:50:03.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: -02-10T04:50: 2021-02-10T04:50:03.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: 02.747968+0000 mon.a (mon.0 2021-02-10T04:50:03.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: ) 2228 : cluster [DBG] osd.3 reported immediately failed by osd.0 2021-02-10T04:50:03.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:50:03.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: T04:50:02.748100+0000 mon.a (mon.0) 2229 : cluster [DBG] osd.3 reported immediately failed by osd.7 2021-02-10T04:50:03.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: 2021-02-10T04:50:03.451 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:50:03.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: -02-10T04:50:02 2021-02-10T04:50:03.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: .748240+0000 mon.a ( 2021-02-10T04:50:03.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: mon.0) 2230 : cluster [DBG] osd.3 reported immediately failed by osd.4 2021-02-10T04:50:03.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster 2021-02-10T04:50:03.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: 2021-02-10T04:50:02. 2021-02-10T04:50:03.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: 748372+0000 mon.a (mon.0 2021-02-10T04:50:03.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: ) 2231 : cluster [DBG] osd.3 reported immediately failed by osd.7 2021-02-10T04:50:03.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster 2021-02-10T04:50 2021-02-10T04:50:03.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: :02.748540+0000 mon.a (mon.0) 2232 2021-02-10T04:50:03.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: : cluster [DBG] osd.3 reported immediately failed by osd.5 2021-02-10T04:50:03.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:50:03.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: 02-10T04:50:02.748694+0000 mon.a (mon 2021-02-10T04:50:03.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: .0) 2233 : cluster [DBG] osd.3 reported immediately failed by osd.5 2021-02-10T04:50:03.452 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:50:03.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: T04:50:02.748868+0000 mon.a (mon.0) 2021-02-10T04:50:03.453 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[29319]: 2234 : cluster [DBG] osd.3 reported immediately failed by osd.6 2021-02-10T04:50:03.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster 2021-02-10T04:50:02 2021-02-10T04:50:03.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: .146002+0000 mon.a (mon.0) 2205 : cluster [DBG] osd.3 reported immediately failed by osd.0 2021-02-10T04:50:03.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster 2021-02-10T04:50:03.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:50:03.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: 50:02.146048+0000 mon.a (mon.0) 2206 : cluster [INF] osd.3 failed (root=default,host=smithi085) (connection refused reported by osd.0) 2021-02-10T04:50:03.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:50:03.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: 02-10T04:50:02.345959+0000 mon.a (mon. 2021-02-10T04:50:03.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: 0) 2207 : cluster [DBG] osd.3 reported immediately failed by osd.0 2021-02-10T04:50:03.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster 2021-02-10T04:50:02.346089 2021-02-10T04:50:03.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2208 : cluster [DBG] osd.3 reported immediately failed by osd.2 2021-02-10T04:50:03.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:50:03.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: 04:50:02.346199+0000 mon.a (mon.0) 2209 : 2021-02-10T04:50:03.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster [DBG] osd.3 reported immediately failed by osd.2 2021-02-10T04:50:03.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster 2021-02-10T04:50:03.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: 2021-02-10T04:50:02.346388+0000 mon.a (mon.0) 2210 2021-02-10T04:50:03.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: : cluster [DBG] osd.3 reported immediately failed by osd.1 2021-02-10T04:50:03.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster 2021-02-10T04:50:03.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: 2021-02-10T04:50:02.346549+0000 mon.a (mon.0) 2211 : 2021-02-10T04:50:03.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster [DBG] osd.3 reported immediately failed by osd.5 2021-02-10T04:50:03.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:50:03.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: -02-10T04:50:02.346671+0000 mon.a (mon.0) 2212 : cluster [DBG] 2021-02-10T04:50:03.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: osd.3 reported immediately failed by osd.1 2021-02-10T04:50:03.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:50:03.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: -10T04:50:02.346807+0000 mon.a (mon.0) 2213 : cluster [DBG] osd.3 reported immediately failed by osd.4 2021-02-10T04:50:03.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:50:03.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: T04:50:02.346931+0000 mon.a (mon.0) 2214 : cluster 2021-02-10T04:50:03.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: [DBG] osd.3 reported immediately failed by osd.5 2021-02-10T04:50:03.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:50:03.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: 02-10T04:50:02.347045+0000 mon.a (mon.0) 2215 : cluster [DBG] 2021-02-10T04:50:03.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: osd.3 reported immediately failed by osd.6 2021-02-10T04:50:03.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:50:03.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: -10T04:50:02.347147+0000 mon.a (mon.0) 2216 : cluster [DBG] osd.3 reported immediately failed by osd.7 2021-02-10T04:50:03.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:50:03.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: T04:50:02.347263+0000 mon.a (mon.0) 2217 : cluster [DBG] osd.3 reported immediately failed by osd.4 2021-02-10T04:50:03.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster 2021-02-10T04:50: 2021-02-10T04:50:03.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: 02.347376+0000 mon.a (mon.0) 2218 : cluster [DBG] osd.3 reported immediately failed by osd.6 2021-02-10T04:50:03.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:50:03.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: T04:50:02.347489+0000 mon.a (mon.0) 2219 : 2021-02-10T04:50:03.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster [DBG] osd.3 reported immediately failed by osd.7 2021-02-10T04:50:03.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:50:03.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: -02-10T04:50:02.347698+0000 mon.a (mon.0) 2220 : cluster 2021-02-10T04:50:03.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: [DBG] osd.3 reported immediately failed by osd.0 2021-02-10T04:50:03.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:50:03.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: 02-10T04:50:02.747038+0000 mon.a (mon.0) 2221 : cluster [DBG] 2021-02-10T04:50:03.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: osd.3 reported immediately failed by osd.2 2021-02-10T04:50:03.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:50:03.456 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: -10T04:50:02.747170+0000 mon.a (mon.0) 2222 : cluster [DBG] 2021-02-10T04:50:03.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: osd.3 reported immediately failed by osd.0 2021-02-10T04:50:03.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:50:03.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: 02-10T04:50:02.747251+0000 mon.a (mon.0) 2223 : cluster [DBG] 2021-02-10T04:50:03.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: osd.3 reported immediately failed by osd.2 2021-02-10T04:50:03.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster 2021-02-10T04:50:02 2021-02-10T04:50:03.457 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: .747373+0000 mon.a (mon.0) 2224 : cluster [DBG] 2021-02-10T04:50:03.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: osd.3 reported immediately failed by osd.1 2021-02-10T04:50:03.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:50:03.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: 02-10T04:50:02.747495+0000 mon.a (mon.0) 2225 : cluster 2021-02-10T04:50:03.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: [DBG] osd.3 reported immediately failed by osd.6 2021-02-10T04:50:03.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster 2021-02-10T04:50 2021-02-10T04:50:03.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: :02.747681+0000 mon.a (mon.0) 2226 : cluster 2021-02-10T04:50:03.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: [DBG] osd.3 reported immediately failed by osd.1 2021-02-10T04:50:03.458 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster 2021-02-10T04:50:02 2021-02-10T04:50:03.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: .747834+0000 mon.a (mon.0) 2227 : cluster 2021-02-10T04:50:03.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: [DBG] osd.3 reported immediately failed by osd.4 2021-02-10T04:50:03.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster 2021-02-10T04:50:02. 2021-02-10T04:50:03.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: 747968+0000 mon.a (mon.0) 2228 : cluster [DBG] 2021-02-10T04:50:03.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: osd.3 reported immediately failed by osd.0 2021-02-10T04:50:03.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:50:03.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: T04:50:02.748100+0000 mon.a (mon.0) 2229 : cluster 2021-02-10T04:50:03.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: [DBG] osd.3 reported immediately failed by osd.7 2021-02-10T04:50:03.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster 2021-02-10T04:50:03.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: 2021-02-10T04:50:02.748240+0000 mon.a ( 2021-02-10T04:50:03.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: mon.0) 2230 : cluster [DBG] osd.3 reported immediately failed by osd.4 2021-02-10T04:50:03.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:50:03.459 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: 04:50:02.748372+0000 mon.a (mon.0) 2021-02-10T04:50:03.460 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: 2231 : cluster [DBG] osd.3 reported immediately failed by osd.7 2021-02-10T04:50:03.460 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:50:03.460 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: :50:02.748540+0000 mon.a (mon.0) 2021-02-10T04:50:03.460 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: 2232 : cluster [DBG] osd.3 reported immediately failed by osd.5 2021-02-10T04:50:03.460 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster 2021-02-10T04:50: 2021-02-10T04:50:03.460 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: 02.748694+0000 mon.a (mon.0) 2233 : cluster 2021-02-10T04:50:03.460 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: [DBG] osd.3 reported immediately failed by osd.5 2021-02-10T04:50:03.460 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: cluster 2021-02-10T04:50:02. 2021-02-10T04:50:03.460 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: 748868+0000 mon.a (mon.0) 2234 : cluster [DBG] osd.3 reported immediately failed by osd.6 2021-02-10T04:50:03.460 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:02 smithi085 conmon[35961]: 2021-02-10T04:50:04.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:03 smithi085 conmon[35961]: cluster 2021-02-10T04:50:02.388802 2021-02-10T04:50:04.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:03 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1153 : cluster [DBG] pgmap v1936: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:50:04.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:03 smithi085 conmon[35961]: cluster 2021-02-10T04:50:04.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:03 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:50:04.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:03 smithi085 conmon[35961]: :50:02.948784+0000 mon.a (mon.0) 2235 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2021-02-10T04:50:04.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:03 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:50:04.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:03 smithi085 conmon[35961]: 10T04:50:02.965913+0000 mon.a (mon.0) 2236 2021-02-10T04:50:04.290 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:03 smithi085 conmon[35961]: : cluster [DBG] osdmap e1095: 8 total, 7 up, 7 in 2021-02-10T04:50:04.290 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:03 smithi085 conmon[29319]: cluster 2021-02-10T04:50:02.388802+0000 mgr.y 2021-02-10T04:50:04.290 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:03 smithi085 conmon[29319]: (mgr.14140) 1153 : cluster [DBG] pgmap v1936: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:50:04.290 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:03 smithi085 conmon[29319]: cluster 2021-02-10T04:50:02.948784+0000 mon.a (mon.0) 2235 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2021-02-10T04:50:04.290 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:03 smithi085 conmon[29319]: cluster 2021-02-10T04:50:02.965913 2021-02-10T04:50:04.290 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:03 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2236 : cluster 2021-02-10T04:50:04.290 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:03 smithi085 conmon[29319]: [DBG] osdmap e1095: 8 total, 7 up, 7 in 2021-02-10T04:50:04.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: cluster 2021-02-10T04:50:04.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: 2021-02-10T04:50:04.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: 2021 2021-02-10T04:50:04.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: - 2021-02-10T04:50:04.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: 02 2021-02-10T04:50:04.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: - 2021-02-10T04:50:04.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: 10 2021-02-10T04:50:04.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: T 2021-02-10T04:50:04.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: 04 2021-02-10T04:50:04.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: : 2021-02-10T04:50:04.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: 50 2021-02-10T04:50:04.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: : 2021-02-10T04:50:04.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: 02 2021-02-10T04:50:04.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: . 2021-02-10T04:50:04.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: 388802 2021-02-10T04:50:04.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: +0000 2021-02-10T04:50:04.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: 2021-02-10T04:50:04.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: mgr.y 2021-02-10T04:50:04.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: ( 2021-02-10T04:50:04.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: mgr 2021-02-10T04:50:04.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: . 2021-02-10T04:50:04.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: 14140 2021-02-10T04:50:04.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: ) 2021-02-10T04:50:04.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: 1153 2021-02-10T04:50:04.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: : 2021-02-10T04:50:04.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: cluster 2021-02-10T04:50:04.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: 2021-02-10T04:50:04.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: [DBG] 2021-02-10T04:50:04.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: 2021-02-10T04:50:04.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: pgmap v1936: 188 pgs: 1 active+clean+scrubbing, 187 active+clean; 883 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:50:04.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: 2021-02-10T04:50:04.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: cluster 2021-02-10T04:50:04.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: 2021-02-10T04:50:04.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: 2021 2021-02-10T04:50:04.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: - 2021-02-10T04:50:04.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: 02 2021-02-10T04:50:04.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: - 2021-02-10T04:50:04.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: 10 2021-02-10T04:50:04.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: T 2021-02-10T04:50:04.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: 04 2021-02-10T04:50:04.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: : 2021-02-10T04:50:04.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: 50 2021-02-10T04:50:04.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: : 2021-02-10T04:50:04.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: 02 2021-02-10T04:50:04.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: . 2021-02-10T04:50:04.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: 948784+0000 mon.a (mon 2021-02-10T04:50:04.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: .0) 2235 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2021-02-10T04:50:04.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:50:04.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: -02-10T04:50:02.965913+0000 2021-02-10T04:50:04.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: mon.a (mon.0) 2236 : cluster [DBG] osdmap e1095: 8 total, 7 up, 7 in 2021-02-10T04:50:04.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:03 smithi164 conmon[29288]: 2021-02-10T04:50:04.898 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 3 1 2021-02-10T04:50:04.899 DEBUG:teuthology.orchestra.run.smithi164:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.5 2021-02-10T04:50:04.946 INFO:teuthology.orchestra.run.smithi164.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:50:05.042 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:04 smithi085 conmon[35961]: cluster 2021-02-10T04:50:03.756239+0000 osd.4 (osd.4) 2373 : cluster [DBG] 7.16 scrub ok 2021-02-10T04:50:05.042 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:04 smithi085 conmon[35961]: cluster 2021-02-10T04:50:03.959110+0000 mon.a (mon.0) 2237 : cluster [DBG] osdmap e1096: 8 total, 7 up, 7 in 2021-02-10T04:50:05.042 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:04 smithi085 conmon[29319]: cluster 2021-02-10T04:50:03.756239+0000 2021-02-10T04:50:05.042 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:04 smithi085 conmon[29319]: osd.4 (osd.4) 2373 : cluster [DBG] 7.16 scrub ok 2021-02-10T04:50:05.043 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:04 smithi085 conmon[29319]: cluster 2021-02-10T04:50:03.959110+0000 mon.a (mon.0) 2237 : cluster [DBG] osdmap e1096: 8 total, 7 up, 7 in 2021-02-10T04:50:05.175 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:50:05.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:04 smithi164 conmon[29288]: cluster 2021-02-10T04:50:03.756239+0000 osd.4 ( 2021-02-10T04:50:05.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:04 smithi164 conmon[29288]: osd.4) 2373 : cluster [DBG] 7.16 scrub ok 2021-02-10T04:50:05.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:04 smithi164 conmon[29288]: cluster 2021-02-10T04:50:03.959110+0000 2021-02-10T04:50:05.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:04 smithi164 conmon[29288]: mon.a 2021-02-10T04:50:05.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:04 smithi164 conmon[29288]: (mon.0) 2237 : cluster 2021-02-10T04:50:05.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:04 smithi164 conmon[29288]: [DBG] osdmap e1096: 8 total, 7 up, 7 in 2021-02-10T04:50:05.982 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:05 smithi164 conmon[29288]: cluster 2021-02-10T04:50:04.209999+0000 2021-02-10T04:50:05.982 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:05 smithi164 conmon[29288]: osd.6 (osd.6) 139 : cluster [DBG] 6.25 scrub ok 2021-02-10T04:50:05.982 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:05 smithi164 conmon[29288]: cluster 2021-02-10T04:50:04.287107+0000 osd.5 (osd.5) 141 : cluster [DBG] 2.18 scrub ok 2021-02-10T04:50:05.982 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:05 smithi164 conmon[29288]: cluster 2021-02-10T04:50:04.389381+0000 mgr.y (mgr.14140) 1154 : cluster [DBG] pgmap v1939: 188 pgs: 6 active+undersized+degraded, 16 active+undersized, 26 peering, 18 stale+active+clean, 1 active+clean+scrubbing, 121 active+clean; 893 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 49 MiB/s rd, 14 MiB/s wr, 90 op/s; 86/2223 objects degraded (3.869%); 1.1 MiB/s, 3 objects/s recovering 2021-02-10T04:50:05.982 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:05 smithi164 conmon[29288]: cluster 2021-02-10T04:50:04.767442+0000 osd.4 (osd.4) 2374 : cluster [DBG] 4.e scrub ok 2021-02-10T04:50:05.982 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:05 smithi164 conmon[29288]: cluster 2021-02-10T04:50:04.967356+0000 mon.a (mon.0) 2238 : cluster [WRN] Health check failed: Reduced data availability: 9 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2021-02-10T04:50:05.982 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:05 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:50:05.983 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:05 smithi164 conmon[29288]: 50:04.967402+0000 mon.a (mon 2021-02-10T04:50:05.983 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:05 smithi164 conmon[29288]: .0) 2239 : cluster [WRN] 2021-02-10T04:50:05.983 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:05 smithi164 conmon[29288]: Health check failed: Degraded data redundancy: 86/2223 objects degraded (3.869%), 6 pgs degraded (PG_DEGRADED) 2021-02-10T04:50:06.256 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:05 smithi085 conmon[29319]: cluster 2021-02-10T04:50:04.209999+0000 osd.6 (osd.6) 139 : cluster [DBG] 6.25 scrub ok 2021-02-10T04:50:06.256 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:05 smithi085 conmon[29319]: cluster 2021-02-10T04:50:04.287107+0000 osd.5 (osd.5) 141 : cluster [DBG] 2.18 scrub ok 2021-02-10T04:50:06.256 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:05 smithi085 conmon[29319]: cluster 2021-02-10T04:50:04.389381+0000 mgr.y (mgr.14140) 1154 : cluster [DBG] pgmap v1939: 188 pgs: 6 active+undersized+degraded, 16 active+undersized, 26 peering, 18 stale+active+clean, 1 active+clean+scrubbing, 121 active+clean; 893 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 49 MiB/s rd, 14 MiB/s wr, 90 op/s; 86/2223 objects degraded (3.869%); 1.1 MiB/s, 3 objects/s recovering 2021-02-10T04:50:06.256 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:05 smithi085 conmon[29319]: cluster 2021-02-10T04:50:04.767442+0000 osd.4 (osd.4) 2374 : cluster [DBG] 4.e scrub ok 2021-02-10T04:50:06.256 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:05 smithi085 conmon[29319]: cluster 2021-02-10T04:50:04.967356+0000 mon.a (mon.0) 2238 : cluster [WRN] Health check failed: Reduced data availability: 9 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2021-02-10T04:50:06.256 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:05 smithi085 conmon[29319]: cluster 2021-02-10T04:50:04.967402+0000 mon.a (mon.0) 2239 : cluster [WRN] Health check failed: Degraded data redundancy: 86/2223 objects degraded (3.869%), 6 pgs degraded (PG_DEGRADED) 2021-02-10T04:50:06.256 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:05 smithi085 conmon[35961]: cluster 2021-02-10T04:50:04.209999+0000 osd.6 (osd.6) 139 : cluster [DBG] 6.25 scrub ok 2021-02-10T04:50:06.256 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:05 smithi085 conmon[35961]: cluster 2021-02-10T04:50:04.287107+0000 osd.5 (osd.5) 141 : cluster [DBG] 2.18 scrub ok 2021-02-10T04:50:06.257 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:05 smithi085 conmon[35961]: cluster 2021-02-10T04:50:04.389381+0000 mgr.y (mgr.14140) 1154 : cluster [DBG] pgmap v1939: 188 pgs: 6 active+undersized+degraded, 16 active+undersized, 26 peering, 18 stale+active+clean, 1 active+clean+scrubbing, 121 active+clean; 893 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 49 MiB/s rd, 14 MiB/s wr, 90 op/s; 86/2223 objects degraded (3.869%); 1.1 MiB/s, 3 objects/s recovering 2021-02-10T04:50:06.257 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:05 smithi085 conmon[35961]: cluster 2021-02-10T04:50:04.767442+0000 osd.4 (osd.4) 2374 : cluster [DBG] 4.e scrub ok 2021-02-10T04:50:06.257 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:05 smithi085 conmon[35961]: cluster 2021-02-10T04:50:04.967356+0000 mon.a (mon.0) 2238 : cluster [WRN] Health check failed: Reduced data availability: 9 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2021-02-10T04:50:06.257 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:05 smithi085 conmon[35961]: cluster 2021-02-10T04:50:04.967402+0000 mon.a (mon.0) 2239 : cluster [WRN] Health check failed: Degraded data redundancy: 86/2223 objects degraded (3.869%), 6 pgs degraded (PG_DEGRADED) 2021-02-10T04:50:06.643 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:50:06 smithi164 conmon[37263]: debug 2021-02-10T04:50:06.280+0000 7f1c89ae9700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:50:06.691 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:06 smithi085 systemd[1]: ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.3.service: Main process exited, code=exited, status=139/n/a 2021-02-10T04:50:06.784 INFO:teuthology.orchestra.run.smithi164.stdout:83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 2021-02-10T04:50:06.890 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.2 2021-02-10T04:50:06.925 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:50:07.068 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set noscrub 2021-02-10T04:50:07.095 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:06 smithi085 conmon[29319]: cluster 2021-02-10T04:50:05.424255+0000 2021-02-10T04:50:07.095 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:06 smithi085 conmon[29319]: osd.0 (osd.0) 151 : cluster [DBG] 1.0 deep-scrub ok 2021-02-10T04:50:07.095 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:06 smithi085 conmon[29319]: cluster 2021-02-10T04:50:05.815128+0000 osd.4 (osd.4) 2375 : cluster [DBG] 7.15 scrub ok 2021-02-10T04:50:07.096 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:06 smithi085 conmon[35961]: cluster 2021-02-10T04:50:05.424255+0000 osd.0 2021-02-10T04:50:07.096 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:06 smithi085 conmon[35961]: (osd.0) 151 : cluster [DBG] 1.0 deep-scrub ok 2021-02-10T04:50:07.096 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:06 smithi085 conmon[35961]: cluster 2021-02-10T04:50:07.096 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:06 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:50:07.096 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:06 smithi085 conmon[35961]: 04:50:05.815128+0000 2021-02-10T04:50:07.096 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:06 smithi085 conmon[35961]: osd.4 (osd.4) 2375 : cluster [DBG] 7.15 scrub ok 2021-02-10T04:50:07.096 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:06 smithi085 conmon[35961]: 2021-02-10T04:50:07.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:06 smithi164 conmon[29288]: cluster 2021-02-10T04:50:05.424255+0000 osd.0 (osd. 2021-02-10T04:50:07.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:06 smithi164 conmon[29288]: 0) 151 : cluster [DBG] 1.0 deep-scrub ok 2021-02-10T04:50:07.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:06 smithi164 conmon[29288]: cluster 2021-02-10T04:50:07.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:06 smithi164 conmon[29288]: 2021-02- 2021-02-10T04:50:07.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:06 smithi164 conmon[29288]: 10T04:50:05.815128+0000 osd.4 (osd.4 2021-02-10T04:50:07.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:06 smithi164 conmon[29288]: ) 2375 : cluster [DBG] 7.15 scrub ok 2021-02-10T04:50:07.271 INFO:teuthology.orchestra.run.smithi085.stdout:1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T04:50:07.369 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:50:07.377 DEBUG:teuthology.orchestra.run.smithi085:> sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 2021-02-10T04:50:07.407 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:50:07 smithi085 conmon[49106]: debug 2021-02-10T04:50:07.216+0000 7f77a8f1b700 -1 received signal: Hangup from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:50:07.410 INFO:teuthology.orchestra.run.smithi085.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2021-02-10T04:50:07.586 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:50:07.608 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:50:07.623 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:50:07.623 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 20, 2021-02-10T04:50:07.623 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 600, 2021-02-10T04:50:07.623 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [ 2021-02-10T04:50:07.623 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.623 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(osd.6.43:137 7.14 7.68825154 (undecoded) ondisk+read+known_if_redirected e1010)", 2021-02-10T04:50:07.623 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:44:27.547799+0000", 2021-02-10T04:50:07.623 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 340.06952734800001, 2021-02-10T04:50:07.623 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.037435434, 2021-02-10T04:50:07.624 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:07.624 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:50:07.624 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:07.624 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.624 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:07.624 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.547799+0000", 2021-02-10T04:50:07.624 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:07.624 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.624 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.624 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:07.624 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.547799+0000", 2021-02-10T04:50:07.624 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.7039000000000001e-05 2021-02-10T04:50:07.624 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.625 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.625 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:07.625 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.547847+0000", 2021-02-10T04:50:07.625 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.011e-06 2021-02-10T04:50:07.625 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.625 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.625 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:07.625 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.547855+0000", 2021-02-10T04:50:07.625 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.3619e-05 2021-02-10T04:50:07.625 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.625 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.625 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:07.625 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.547868+0000", 2021-02-10T04:50:07.626 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.4290000000000004e-06 2021-02-10T04:50:07.626 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.626 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.626 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:07.626 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.547876+0000", 2021-02-10T04:50:07.626 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.6455999999999997e-05 2021-02-10T04:50:07.626 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.626 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.626 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.626 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.547922+0000", 2021-02-10T04:50:07.627 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0771000000000001e-05 2021-02-10T04:50:07.627 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.627 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.627 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for peered", 2021-02-10T04:50:07.627 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.547933+0000", 2021-02-10T04:50:07.627 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.037287318 2021-02-10T04:50:07.627 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.627 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.627 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.627 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.585220+0000", 2021-02-10T04:50:07.627 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.4791e-05 2021-02-10T04:50:07.627 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.627 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.628 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:07.628 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.585235+0000", 2021-02-10T04:50:07.628 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.037435434 2021-02-10T04:50:07.628 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.628 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:07.628 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.628 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.628 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.628 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(osd.6.43:137 7.14 7.68825154 (undecoded) ondisk+retry+read+known_if_redirected e1010)", 2021-02-10T04:50:07.628 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:44:27.550423+0000", 2021-02-10T04:50:07.628 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 340.066903524, 2021-02-10T04:50:07.628 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0351207659999999, 2021-02-10T04:50:07.628 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:07.628 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:50:07.629 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:07.629 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.629 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:07.629 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.550423+0000", 2021-02-10T04:50:07.629 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:07.629 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.629 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.629 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:07.629 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.550423+0000", 2021-02-10T04:50:07.629 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.8334999999999999e-05 2021-02-10T04:50:07.629 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.629 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.629 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:07.629 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.550452+0000", 2021-02-10T04:50:07.630 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.831e-06 2021-02-10T04:50:07.630 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.630 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.630 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:07.630 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.550456+0000", 2021-02-10T04:50:07.630 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0564000000000001e-05 2021-02-10T04:50:07.630 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.630 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.630 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:07.630 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.550467+0000", 2021-02-10T04:50:07.630 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.8599999999999998e-06 2021-02-10T04:50:07.630 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.630 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.631 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:07.631 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.550473+0000", 2021-02-10T04:50:07.631 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.5661999999999998e-05 2021-02-10T04:50:07.631 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.631 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.631 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.631 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.550519+0000", 2021-02-10T04:50:07.631 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.8610000000000004e-06 2021-02-10T04:50:07.631 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.631 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.631 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for peered", 2021-02-10T04:50:07.631 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.550527+0000", 2021-02-10T04:50:07.631 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0350097140000001 2021-02-10T04:50:07.631 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.632 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.632 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.632 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.585537+0000", 2021-02-10T04:50:07.632 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.939e-06 2021-02-10T04:50:07.632 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.632 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.632 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:07.632 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.585544+0000", 2021-02-10T04:50:07.632 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0351207659999999 2021-02-10T04:50:07.632 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.632 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:07.632 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.632 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.632 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.633 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(osd.6.43:137 7.14 7.68825154 (undecoded) ondisk+retry+read+known_if_redirected e1010)", 2021-02-10T04:50:07.633 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:44:27.553162+0000", 2021-02-10T04:50:07.633 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 340.064165307, 2021-02-10T04:50:07.633 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0324232959999999, 2021-02-10T04:50:07.633 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:07.633 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:50:07.633 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:07.633 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.633 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:07.633 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.553162+0000", 2021-02-10T04:50:07.633 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:07.633 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.633 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.634 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:07.634 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.553162+0000", 2021-02-10T04:50:07.634 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.8466000000000001e-05 2021-02-10T04:50:07.634 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.634 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.634 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:07.634 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.553200+0000", 2021-02-10T04:50:07.634 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.878e-06 2021-02-10T04:50:07.634 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.634 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.634 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:07.634 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.553207+0000", 2021-02-10T04:50:07.634 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2311e-05 2021-02-10T04:50:07.634 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.635 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.635 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:07.635 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.553219+0000", 2021-02-10T04:50:07.635 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.4740000000000002e-06 2021-02-10T04:50:07.635 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.635 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.635 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:07.635 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.553226+0000", 2021-02-10T04:50:07.635 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9032000000000001e-05 2021-02-10T04:50:07.635 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.635 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.635 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.636 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.553275+0000", 2021-02-10T04:50:07.636 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.2520000000000005e-06 2021-02-10T04:50:07.636 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.636 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.636 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for peered", 2021-02-10T04:50:07.636 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.553284+0000", 2021-02-10T04:50:07.636 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.032281432 2021-02-10T04:50:07.636 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.636 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.636 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.636 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.585565+0000", 2021-02-10T04:50:07.636 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.9451000000000001e-05 2021-02-10T04:50:07.637 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.637 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.637 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:07.637 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.585585+0000", 2021-02-10T04:50:07.637 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0324232959999999 2021-02-10T04:50:07.637 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.637 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:07.637 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.637 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.637 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.637 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(osd.6.43:137 7.14 7.68825154 (undecoded) ondisk+retry+read+known_if_redirected e1010)", 2021-02-10T04:50:07.638 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:44:27.555324+0000", 2021-02-10T04:50:07.638 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 340.06200273100001, 2021-02-10T04:50:07.638 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.030283651, 2021-02-10T04:50:07.638 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:07.638 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:50:07.638 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:07.638 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.638 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:07.638 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.555324+0000", 2021-02-10T04:50:07.638 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:07.639 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.639 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.639 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:07.639 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.555324+0000", 2021-02-10T04:50:07.639 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.2412000000000001e-05 2021-02-10T04:50:07.639 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.639 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.639 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:07.639 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.555356+0000", 2021-02-10T04:50:07.639 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.934e-06 2021-02-10T04:50:07.639 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.639 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.639 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:07.639 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.555363+0000", 2021-02-10T04:50:07.640 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2578000000000001e-05 2021-02-10T04:50:07.640 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.640 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.640 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:07.640 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.555376+0000", 2021-02-10T04:50:07.640 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.6250000000000001e-06 2021-02-10T04:50:07.640 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.640 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.640 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:07.640 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.555383+0000", 2021-02-10T04:50:07.640 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.6804000000000002e-05 2021-02-10T04:50:07.640 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.640 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.641 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.641 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.555419+0000", 2021-02-10T04:50:07.641 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.3980000000000004e-06 2021-02-10T04:50:07.641 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.641 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.641 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for peered", 2021-02-10T04:50:07.641 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.555426+0000", 2021-02-10T04:50:07.641 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0301757359999999 2021-02-10T04:50:07.641 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.641 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.641 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.641 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.585602+0000", 2021-02-10T04:50:07.641 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.1639999999999998e-06 2021-02-10T04:50:07.641 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.642 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.642 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:07.642 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.585608+0000", 2021-02-10T04:50:07.642 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.030283651 2021-02-10T04:50:07.642 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.642 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:07.642 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.642 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.642 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.642 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(osd.6.43:137 7.14 7.68825154 (undecoded) ondisk+retry+read+known_if_redirected e1010)", 2021-02-10T04:50:07.642 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:44:27.560120+0000", 2021-02-10T04:50:07.642 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 340.057206506, 2021-02-10T04:50:07.642 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.027132121, 2021-02-10T04:50:07.642 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:07.643 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:50:07.643 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:07.643 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.643 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:07.643 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.560120+0000", 2021-02-10T04:50:07.643 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:07.643 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.643 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.643 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:07.643 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.560120+0000", 2021-02-10T04:50:07.643 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.0791000000000003e-05 2021-02-10T04:50:07.643 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.644 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.644 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:07.644 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.560161+0000", 2021-02-10T04:50:07.644 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.2800000000000003e-06 2021-02-10T04:50:07.644 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.644 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.644 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:07.644 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.560166+0000", 2021-02-10T04:50:07.644 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0981999999999999e-05 2021-02-10T04:50:07.644 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.645 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.645 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:07.645 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.560177+0000", 2021-02-10T04:50:07.645 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.2509999999999997e-06 2021-02-10T04:50:07.645 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.645 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.645 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:07.645 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.560185+0000", 2021-02-10T04:50:07.645 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.7989000000000002e-05 2021-02-10T04:50:07.646 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.646 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.646 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.646 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.560263+0000", 2021-02-10T04:50:07.646 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0079e-05 2021-02-10T04:50:07.646 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.646 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.646 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for peered", 2021-02-10T04:50:07.646 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.560273+0000", 2021-02-10T04:50:07.646 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0253518049999999 2021-02-10T04:50:07.646 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.646 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.646 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.647 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.585624+0000", 2021-02-10T04:50:07.647 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0016279440000000001 2021-02-10T04:50:07.647 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.647 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.647 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:07.647 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.587252+0000", 2021-02-10T04:50:07.647 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.027132121 2021-02-10T04:50:07.647 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.647 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:07.647 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.647 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.647 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.647 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(osd.6.43:137 7.14 7.68825154 (undecoded) ondisk+retry+read+known_if_redirected e1010)", 2021-02-10T04:50:07.647 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:44:27.563551+0000", 2021-02-10T04:50:07.648 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 340.05377534899998, 2021-02-10T04:50:07.648 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0237599079999999, 2021-02-10T04:50:07.648 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:07.648 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:50:07.648 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:07.648 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.648 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:07.648 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.563551+0000", 2021-02-10T04:50:07.648 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:07.648 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.648 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.648 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:07.648 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.563551+0000", 2021-02-10T04:50:07.649 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.626e-05 2021-02-10T04:50:07.649 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.649 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.649 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:07.649 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.563608+0000", 2021-02-10T04:50:07.649 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.6689999999999995e-06 2021-02-10T04:50:07.649 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.649 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.649 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:07.649 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.563616+0000", 2021-02-10T04:50:07.649 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.471e-05 2021-02-10T04:50:07.649 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.649 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.649 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:07.650 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.563631+0000", 2021-02-10T04:50:07.650 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.8939999999999993e-06 2021-02-10T04:50:07.650 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.650 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.650 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:07.650 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.563639+0000", 2021-02-10T04:50:07.650 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0067e-05 2021-02-10T04:50:07.650 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.650 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.650 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.650 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.563689+0000", 2021-02-10T04:50:07.650 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0200999999999999e-05 2021-02-10T04:50:07.650 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.650 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.651 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for peered", 2021-02-10T04:50:07.651 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.563699+0000", 2021-02-10T04:50:07.651 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.023602031 2021-02-10T04:50:07.651 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.651 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.651 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.651 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.587301+0000", 2021-02-10T04:50:07.651 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0076e-05 2021-02-10T04:50:07.651 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.651 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.651 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:07.651 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.587311+0000", 2021-02-10T04:50:07.651 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0237599079999999 2021-02-10T04:50:07.652 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.652 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:07.652 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.652 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.652 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.652 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(osd.6.43:137 7.14 7.68825154 (undecoded) ondisk+retry+read+known_if_redirected e1010)", 2021-02-10T04:50:07.652 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:44:27.565840+0000", 2021-02-10T04:50:07.652 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 340.05148639599997, 2021-02-10T04:50:07.652 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0215073610000001, 2021-02-10T04:50:07.652 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:07.652 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:50:07.652 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:07.652 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.652 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:07.653 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.565840+0000", 2021-02-10T04:50:07.653 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:07.653 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.653 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.653 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:07.653 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.565840+0000", 2021-02-10T04:50:07.653 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.1137999999999999e-05 2021-02-10T04:50:07.653 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.653 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.653 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:07.653 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.565872+0000", 2021-02-10T04:50:07.653 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.6530000000000002e-06 2021-02-10T04:50:07.653 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.654 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.654 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:07.654 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.565876+0000", 2021-02-10T04:50:07.654 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.9910000000000002e-06 2021-02-10T04:50:07.654 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.654 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.654 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:07.654 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.565884+0000", 2021-02-10T04:50:07.654 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.5560000000000001e-06 2021-02-10T04:50:07.654 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.654 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.654 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:07.654 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.565889+0000", 2021-02-10T04:50:07.655 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0800999999999997e-05 2021-02-10T04:50:07.655 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.655 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.655 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.655 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.565940+0000", 2021-02-10T04:50:07.655 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.49e-06 2021-02-10T04:50:07.655 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.655 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.655 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for peered", 2021-02-10T04:50:07.655 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.565948+0000", 2021-02-10T04:50:07.655 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0213906049999999 2021-02-10T04:50:07.655 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.655 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.656 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.656 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.587339+0000", 2021-02-10T04:50:07.656 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.1269999999999992e-06 2021-02-10T04:50:07.656 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.656 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.656 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:07.656 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.587348+0000", 2021-02-10T04:50:07.656 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0215073610000001 2021-02-10T04:50:07.656 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.656 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:07.656 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.656 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.656 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.656 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(osd.6.43:137 7.14 7.68825154 (undecoded) ondisk+retry+read+known_if_redirected e1010)", 2021-02-10T04:50:07.657 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:44:27.567849+0000", 2021-02-10T04:50:07.657 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 340.04947736399998, 2021-02-10T04:50:07.657 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.019533646, 2021-02-10T04:50:07.657 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:07.657 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:50:07.657 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:07.657 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.657 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:07.657 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.567849+0000", 2021-02-10T04:50:07.657 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:07.658 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.658 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.658 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:07.658 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.567849+0000", 2021-02-10T04:50:07.658 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.3035999999999999e-05 2021-02-10T04:50:07.658 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.658 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.658 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:07.658 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.567892+0000", 2021-02-10T04:50:07.658 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.7530000000000001e-06 2021-02-10T04:50:07.658 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.658 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.659 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:07.659 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.567897+0000", 2021-02-10T04:50:07.659 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.9100000000000005e-06 2021-02-10T04:50:07.659 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.659 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.659 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:07.659 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.567905+0000", 2021-02-10T04:50:07.659 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4220000000000002e-06 2021-02-10T04:50:07.659 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.659 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.659 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:07.660 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.567910+0000", 2021-02-10T04:50:07.660 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.2613e-05 2021-02-10T04:50:07.660 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.660 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.660 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.660 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.567952+0000", 2021-02-10T04:50:07.660 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.6309999999999992e-06 2021-02-10T04:50:07.661 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.661 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.661 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for peered", 2021-02-10T04:50:07.661 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.567961+0000", 2021-02-10T04:50:07.661 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0194131420000001 2021-02-10T04:50:07.661 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.661 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.661 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.662 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.587374+0000", 2021-02-10T04:50:07.662 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.1390000000000001e-06 2021-02-10T04:50:07.662 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.662 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.662 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:07.662 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.587383+0000", 2021-02-10T04:50:07.662 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.019533646 2021-02-10T04:50:07.662 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.662 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:07.662 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.662 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.662 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.662 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(osd.6.43:137 7.14 7.68825154 (undecoded) ondisk+retry+read+known_if_redirected e1010)", 2021-02-10T04:50:07.663 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:44:27.569666+0000", 2021-02-10T04:50:07.663 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 340.04766049099999, 2021-02-10T04:50:07.663 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.01777161, 2021-02-10T04:50:07.663 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:07.663 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:50:07.663 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:07.663 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.663 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:07.663 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.569666+0000", 2021-02-10T04:50:07.663 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:07.663 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.663 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.663 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:07.664 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.569666+0000", 2021-02-10T04:50:07.664 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.3389000000000001e-05 2021-02-10T04:50:07.664 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.664 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.664 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:07.664 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.569710+0000", 2021-02-10T04:50:07.664 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.6579999999999996e-06 2021-02-10T04:50:07.664 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.664 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.664 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:07.664 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.569716+0000", 2021-02-10T04:50:07.664 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2147000000000001e-05 2021-02-10T04:50:07.664 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.665 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.665 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:07.665 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.569729+0000", 2021-02-10T04:50:07.665 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.8510000000000004e-06 2021-02-10T04:50:07.665 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.665 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.665 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:07.665 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.569734+0000", 2021-02-10T04:50:07.665 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.5423e-05 2021-02-10T04:50:07.665 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.665 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.665 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.665 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.569800+0000", 2021-02-10T04:50:07.665 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.7770000000000008e-06 2021-02-10T04:50:07.666 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.666 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.666 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for peered", 2021-02-10T04:50:07.666 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.569809+0000", 2021-02-10T04:50:07.666 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.017604669 2021-02-10T04:50:07.666 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.666 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.666 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.666 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.587413+0000", 2021-02-10T04:50:07.666 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.4695999999999999e-05 2021-02-10T04:50:07.666 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.666 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.666 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:07.667 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.587438+0000", 2021-02-10T04:50:07.667 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.01777161 2021-02-10T04:50:07.667 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.667 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:07.667 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.667 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.667 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.667 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(osd.6.43:137 7.14 7.68825154 (undecoded) ondisk+retry+read+known_if_redirected e1010)", 2021-02-10T04:50:07.667 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:44:27.571749+0000", 2021-02-10T04:50:07.667 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 340.04557749700001, 2021-02-10T04:50:07.667 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.015727853, 2021-02-10T04:50:07.667 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:07.667 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:50:07.668 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:07.668 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.668 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:07.668 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.571749+0000", 2021-02-10T04:50:07.668 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:07.668 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.668 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.668 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:07.668 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.571749+0000", 2021-02-10T04:50:07.668 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.8561e-05 2021-02-10T04:50:07.668 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.668 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.668 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:07.668 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.571778+0000", 2021-02-10T04:50:07.669 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.3879999999999999e-06 2021-02-10T04:50:07.669 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.669 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.669 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:07.669 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.571783+0000", 2021-02-10T04:50:07.669 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.4066999999999999e-05 2021-02-10T04:50:07.669 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.669 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.669 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:07.669 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.571797+0000", 2021-02-10T04:50:07.669 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.72e-06 2021-02-10T04:50:07.669 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.669 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.670 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:07.670 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.571804+0000", 2021-02-10T04:50:07.670 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.1192999999999997e-05 2021-02-10T04:50:07.670 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.670 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.670 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.670 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.571845+0000", 2021-02-10T04:50:07.670 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.1969999999999996e-06 2021-02-10T04:50:07.670 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.670 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.670 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for peered", 2021-02-10T04:50:07.670 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.571853+0000", 2021-02-10T04:50:07.670 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0156152270000001 2021-02-10T04:50:07.671 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.671 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.671 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.671 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.587469+0000", 2021-02-10T04:50:07.671 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.4999999999999999e-06 2021-02-10T04:50:07.671 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.671 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.671 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:07.671 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.587477+0000", 2021-02-10T04:50:07.671 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.015727853 2021-02-10T04:50:07.671 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.671 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:07.671 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.671 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.672 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.672 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(osd.6.43:137 7.14 7.68825154 (undecoded) ondisk+retry+read+known_if_redirected e1010)", 2021-02-10T04:50:07.672 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:44:27.573747+0000", 2021-02-10T04:50:07.672 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 340.043580131, 2021-02-10T04:50:07.672 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0137800969999999, 2021-02-10T04:50:07.672 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:07.672 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:50:07.672 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:07.672 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.672 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:07.672 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.573747+0000", 2021-02-10T04:50:07.672 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:07.672 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.673 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.673 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:07.673 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.573747+0000", 2021-02-10T04:50:07.673 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4252000000000003e-05 2021-02-10T04:50:07.673 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.673 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.673 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:07.673 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.573791+0000", 2021-02-10T04:50:07.673 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.3690000000000001e-06 2021-02-10T04:50:07.673 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.673 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.673 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:07.673 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.573798+0000", 2021-02-10T04:50:07.674 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.314e-05 2021-02-10T04:50:07.674 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.674 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.674 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:07.674 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.573811+0000", 2021-02-10T04:50:07.674 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.3990000000000001e-06 2021-02-10T04:50:07.674 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.674 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.674 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:07.674 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.573818+0000", 2021-02-10T04:50:07.674 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.3263999999999997e-05 2021-02-10T04:50:07.674 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.674 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.675 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.675 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.573871+0000", 2021-02-10T04:50:07.675 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.8459999999999997e-06 2021-02-10T04:50:07.675 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.675 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.675 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for peered", 2021-02-10T04:50:07.675 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.573880+0000", 2021-02-10T04:50:07.675 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0136308359999999 2021-02-10T04:50:07.675 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.675 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.675 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.675 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.587511+0000", 2021-02-10T04:50:07.675 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.5991e-05 2021-02-10T04:50:07.676 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.676 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.676 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:07.676 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.587527+0000", 2021-02-10T04:50:07.676 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0137800969999999 2021-02-10T04:50:07.676 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.676 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:07.676 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.676 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.676 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.676 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(osd.6.43:137 7.14 7.68825154 (undecoded) ondisk+retry+read+known_if_redirected e1010)", 2021-02-10T04:50:07.676 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:44:27.575985+0000", 2021-02-10T04:50:07.676 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 340.041341606, 2021-02-10T04:50:07.677 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0115965309999999, 2021-02-10T04:50:07.677 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:07.677 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:50:07.677 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:07.677 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.677 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:07.677 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.575985+0000", 2021-02-10T04:50:07.677 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:07.677 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.677 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.677 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:07.677 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.575985+0000", 2021-02-10T04:50:07.677 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.6108e-05 2021-02-10T04:50:07.677 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.678 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.678 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:07.678 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.576031+0000", 2021-02-10T04:50:07.678 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.6340000000000003e-06 2021-02-10T04:50:07.678 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.678 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.678 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:07.678 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.576038+0000", 2021-02-10T04:50:07.678 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2289e-05 2021-02-10T04:50:07.678 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.678 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.678 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:07.678 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.576050+0000", 2021-02-10T04:50:07.679 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.2389999999999999e-06 2021-02-10T04:50:07.679 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.679 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.679 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:07.679 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.576056+0000", 2021-02-10T04:50:07.679 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.5979000000000001e-05 2021-02-10T04:50:07.679 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.679 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.679 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.679 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.576102+0000", 2021-02-10T04:50:07.679 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.4419999999999998e-06 2021-02-10T04:50:07.679 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.679 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.679 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for peered", 2021-02-10T04:50:07.680 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.576111+0000", 2021-02-10T04:50:07.680 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.01146161 2021-02-10T04:50:07.680 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.680 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.680 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.680 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.587573+0000", 2021-02-10T04:50:07.680 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.2299999999999997e-06 2021-02-10T04:50:07.680 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.680 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.680 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:07.680 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.587582+0000", 2021-02-10T04:50:07.680 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0115965309999999 2021-02-10T04:50:07.680 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.681 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:07.681 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.681 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.681 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.681 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(osd.6.43:137 7.14 7.68825154 (undecoded) ondisk+retry+read+known_if_redirected e1010)", 2021-02-10T04:50:07.681 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:44:27.577902+0000", 2021-02-10T04:50:07.681 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 340.03942432299999, 2021-02-10T04:50:07.681 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0097169429999999, 2021-02-10T04:50:07.681 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:07.681 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:50:07.681 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:07.681 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.681 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:07.681 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.577902+0000", 2021-02-10T04:50:07.682 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:07.682 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.682 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.682 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:07.682 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.577902+0000", 2021-02-10T04:50:07.682 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.9561999999999999e-05 2021-02-10T04:50:07.682 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.682 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.682 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:07.682 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.577932+0000", 2021-02-10T04:50:07.682 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.51e-06 2021-02-10T04:50:07.682 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.683 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.683 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:07.683 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.577937+0000", 2021-02-10T04:50:07.683 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.8339999999999999e-06 2021-02-10T04:50:07.683 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.683 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.683 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:07.683 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.577944+0000", 2021-02-10T04:50:07.683 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4939999999999997e-06 2021-02-10T04:50:07.683 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.683 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.683 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:07.683 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.577949+0000", 2021-02-10T04:50:07.684 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.8974999999999999e-05 2021-02-10T04:50:07.684 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.684 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.684 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.684 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.577988+0000", 2021-02-10T04:50:07.684 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.1029999999999998e-06 2021-02-10T04:50:07.684 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.684 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.684 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for peered", 2021-02-10T04:50:07.684 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.577994+0000", 2021-02-10T04:50:07.684 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0096165399999999 2021-02-10T04:50:07.684 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.684 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.684 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.685 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.587611+0000", 2021-02-10T04:50:07.685 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.9250000000000001e-06 2021-02-10T04:50:07.685 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.685 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.685 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:07.685 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.587619+0000", 2021-02-10T04:50:07.685 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0097169429999999 2021-02-10T04:50:07.685 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.685 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:07.685 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.685 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.685 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.685 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(osd.6.43:137 7.14 7.68825154 (undecoded) ondisk+retry+read+known_if_redirected e1010)", 2021-02-10T04:50:07.686 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:44:27.579734+0000", 2021-02-10T04:50:07.686 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 340.03759301000002, 2021-02-10T04:50:07.686 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.009397015, 2021-02-10T04:50:07.686 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:07.686 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:50:07.686 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:07.686 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.686 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:07.686 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.579734+0000", 2021-02-10T04:50:07.686 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:07.686 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.686 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.686 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:07.687 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.579734+0000", 2021-02-10T04:50:07.687 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.4297e-05 2021-02-10T04:50:07.687 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.687 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.687 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:07.687 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.579768+0000", 2021-02-10T04:50:07.687 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4909999999999999e-06 2021-02-10T04:50:07.687 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.687 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.687 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:07.687 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.579773+0000", 2021-02-10T04:50:07.687 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.2099999999999993e-06 2021-02-10T04:50:07.687 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.687 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.688 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:07.688 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.579781+0000", 2021-02-10T04:50:07.688 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.3959999999999999e-06 2021-02-10T04:50:07.688 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.688 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.688 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:07.688 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.579785+0000", 2021-02-10T04:50:07.688 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.5219000000000001e-05 2021-02-10T04:50:07.688 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.688 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.688 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.688 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.579840+0000", 2021-02-10T04:50:07.688 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.4879e-05 2021-02-10T04:50:07.689 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.689 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.689 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for peered", 2021-02-10T04:50:07.689 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.579865+0000", 2021-02-10T04:50:07.689 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.007779502 2021-02-10T04:50:07.689 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.689 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.689 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.689 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.587645+0000", 2021-02-10T04:50:07.689 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0014860209999999999 2021-02-10T04:50:07.689 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.689 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.689 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:07.689 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.589131+0000", 2021-02-10T04:50:07.690 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.009397015 2021-02-10T04:50:07.690 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.690 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:07.690 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.690 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.690 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.690 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(osd.6.43:137 7.14 7.68825154 (undecoded) ondisk+retry+read+known_if_redirected e1010)", 2021-02-10T04:50:07.690 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:44:27.581717+0000", 2021-02-10T04:50:07.690 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 340.03560978399997, 2021-02-10T04:50:07.690 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.006032839, 2021-02-10T04:50:07.690 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:07.690 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:50:07.690 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:07.691 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.691 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:07.691 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.581717+0000", 2021-02-10T04:50:07.691 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:07.691 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.691 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.691 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:07.691 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.581717+0000", 2021-02-10T04:50:07.691 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.1039999999999997e-05 2021-02-10T04:50:07.691 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.691 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.691 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:07.691 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.581798+0000", 2021-02-10T04:50:07.691 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.0620000000000003e-06 2021-02-10T04:50:07.692 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.692 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.692 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:07.692 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.581805+0000", 2021-02-10T04:50:07.692 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2835e-05 2021-02-10T04:50:07.692 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.692 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.692 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:07.692 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.581818+0000", 2021-02-10T04:50:07.692 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.4548e-05 2021-02-10T04:50:07.692 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.692 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.692 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:07.693 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.581833+0000", 2021-02-10T04:50:07.693 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.2203000000000001e-05 2021-02-10T04:50:07.693 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.693 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.693 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.693 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.581885+0000", 2021-02-10T04:50:07.693 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.4319999999999999e-06 2021-02-10T04:50:07.693 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.693 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.693 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for peered", 2021-02-10T04:50:07.693 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.581893+0000", 2021-02-10T04:50:07.693 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0058462859999999 2021-02-10T04:50:07.693 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.693 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.694 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.694 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.587739+0000", 2021-02-10T04:50:07.694 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0433e-05 2021-02-10T04:50:07.694 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.694 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.694 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:07.694 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.587750+0000", 2021-02-10T04:50:07.694 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.006032839 2021-02-10T04:50:07.694 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.694 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:07.694 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.694 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.694 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.695 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(osd.6.43:137 7.14 7.68825154 (undecoded) ondisk+retry+read+known_if_redirected e1010)", 2021-02-10T04:50:07.695 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:44:27.583485+0000", 2021-02-10T04:50:07.695 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 340.03384166500001, 2021-02-10T04:50:07.695 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.004303422, 2021-02-10T04:50:07.695 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:07.695 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:50:07.695 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:07.695 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.695 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:07.695 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.583485+0000", 2021-02-10T04:50:07.695 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:07.695 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.695 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.695 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:07.696 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.583485+0000", 2021-02-10T04:50:07.696 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.727e-05 2021-02-10T04:50:07.696 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.696 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.696 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:07.696 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.583532+0000", 2021-02-10T04:50:07.696 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.9319999999999999e-06 2021-02-10T04:50:07.696 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.696 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.696 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:07.696 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.583539+0000", 2021-02-10T04:50:07.696 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.3356999999999999e-05 2021-02-10T04:50:07.696 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.696 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.697 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:07.697 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.583553+0000", 2021-02-10T04:50:07.697 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.2609999999999999e-06 2021-02-10T04:50:07.697 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.697 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.697 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:07.697 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.583559+0000", 2021-02-10T04:50:07.697 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.7971000000000004e-05 2021-02-10T04:50:07.697 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.697 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.697 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.697 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.583627+0000", 2021-02-10T04:50:07.697 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.6219999999999998e-06 2021-02-10T04:50:07.698 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.698 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.698 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for peered", 2021-02-10T04:50:07.698 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.583636+0000", 2021-02-10T04:50:07.698 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0041441870000001 2021-02-10T04:50:07.698 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.698 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.698 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.698 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.587780+0000", 2021-02-10T04:50:07.698 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.8219999999999996e-06 2021-02-10T04:50:07.698 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.698 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.698 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:07.699 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.587789+0000", 2021-02-10T04:50:07.699 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.004303422 2021-02-10T04:50:07.699 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.699 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:07.699 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.699 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.699 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.699 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(osd.6.43:137 7.14 7.68825154 (undecoded) ondisk+retry+read+known_if_redirected e1010)", 2021-02-10T04:50:07.699 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:44:27.585678+0000", 2021-02-10T04:50:07.699 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 340.031649168, 2021-02-10T04:50:07.699 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0021417589999999, 2021-02-10T04:50:07.699 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:07.699 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:50:07.700 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:07.700 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.700 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:07.700 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.585678+0000", 2021-02-10T04:50:07.700 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:07.700 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.700 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.700 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:07.700 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.585678+0000", 2021-02-10T04:50:07.700 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.4491e-05 2021-02-10T04:50:07.700 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.700 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.700 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:07.700 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.585702+0000", 2021-02-10T04:50:07.701 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.6210000000000003e-06 2021-02-10T04:50:07.701 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.701 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.701 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:07.701 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.585707+0000", 2021-02-10T04:50:07.701 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.0080000000000002e-06 2021-02-10T04:50:07.701 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.701 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.701 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:07.701 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.585715+0000", 2021-02-10T04:50:07.701 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.3420000000000001e-06 2021-02-10T04:50:07.701 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.701 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.702 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:07.702 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.585719+0000", 2021-02-10T04:50:07.702 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.5000000000000003e-05 2021-02-10T04:50:07.702 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.702 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.702 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.702 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.585764+0000", 2021-02-10T04:50:07.702 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0179e-05 2021-02-10T04:50:07.702 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.702 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.702 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for peered", 2021-02-10T04:50:07.702 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.585774+0000", 2021-02-10T04:50:07.702 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0020361630000001 2021-02-10T04:50:07.703 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.703 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.703 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.703 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.587810+0000", 2021-02-10T04:50:07.703 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.9549999999999998e-06 2021-02-10T04:50:07.703 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.703 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.703 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:07.703 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.587819+0000", 2021-02-10T04:50:07.703 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0021417589999999 2021-02-10T04:50:07.703 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.703 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:07.703 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.703 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.704 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.704 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(osd.6.43:137 7.14 7.68825154 (undecoded) ondisk+retry+read+known_if_redirected e1010)", 2021-02-10T04:50:07.704 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:44:27.588001+0000", 2021-02-10T04:50:07.704 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 340.02932593100002, 2021-02-10T04:50:07.704 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0011517860000001, 2021-02-10T04:50:07.704 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:07.704 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:50:07.704 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:07.704 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.704 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:07.704 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.588001+0000", 2021-02-10T04:50:07.704 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:07.704 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.704 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.705 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:07.705 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.588001+0000", 2021-02-10T04:50:07.705 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.3899000000000001e-05 2021-02-10T04:50:07.705 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.705 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.705 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:07.705 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.588045+0000", 2021-02-10T04:50:07.705 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.0770000000000002e-06 2021-02-10T04:50:07.705 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.705 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.705 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:07.705 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.588052+0000", 2021-02-10T04:50:07.705 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2018000000000001e-05 2021-02-10T04:50:07.706 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.706 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.706 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:07.706 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.588064+0000", 2021-02-10T04:50:07.706 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.5040000000000004e-06 2021-02-10T04:50:07.706 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.706 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.706 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:07.706 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.588068+0000", 2021-02-10T04:50:07.706 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.5649999999999998e-05 2021-02-10T04:50:07.706 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.706 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.706 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.707 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.588114+0000", 2021-02-10T04:50:07.707 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.4470000000000006e-06 2021-02-10T04:50:07.707 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.707 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.707 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for peered", 2021-02-10T04:50:07.707 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.588122+0000", 2021-02-10T04:50:07.707 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.99994406599999996 2021-02-10T04:50:07.707 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.707 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.707 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.707 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.588067+0000", 2021-02-10T04:50:07.707 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0010861250000000001 2021-02-10T04:50:07.707 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.708 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.708 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:07.708 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.589153+0000", 2021-02-10T04:50:07.708 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0011517860000001 2021-02-10T04:50:07.708 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.708 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:07.708 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.708 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.708 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.708 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(osd.6.43:137 7.14 7.68825154 (undecoded) ondisk+retry+read+known_if_redirected e1010)", 2021-02-10T04:50:07.708 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:44:27.590022+0000", 2021-02-10T04:50:07.708 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 340.02730454800002, 2021-02-10T04:50:07.708 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.99916800299999997, 2021-02-10T04:50:07.708 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:07.709 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:50:07.709 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:07.709 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.709 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:07.709 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.590022+0000", 2021-02-10T04:50:07.709 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:07.709 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.709 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.709 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:07.709 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.590022+0000", 2021-02-10T04:50:07.709 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.0212000000000001e-05 2021-02-10T04:50:07.709 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.709 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.710 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:07.710 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.590052+0000", 2021-02-10T04:50:07.710 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.7780000000000002e-06 2021-02-10T04:50:07.710 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.710 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.710 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:07.710 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.590059+0000", 2021-02-10T04:50:07.710 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2772000000000001e-05 2021-02-10T04:50:07.710 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.710 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.710 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:07.710 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.590072+0000", 2021-02-10T04:50:07.710 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.2670000000000003e-06 2021-02-10T04:50:07.711 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.711 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.711 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:07.711 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.590078+0000", 2021-02-10T04:50:07.711 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.2014999999999997e-05 2021-02-10T04:50:07.711 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.711 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.711 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.711 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.590120+0000", 2021-02-10T04:50:07.711 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.558e-06 2021-02-10T04:50:07.711 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.711 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.711 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for peered", 2021-02-10T04:50:07.712 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.590126+0000", 2021-02-10T04:50:07.712 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.99905493000000001 2021-02-10T04:50:07.712 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.712 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.712 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.712 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.589181+0000", 2021-02-10T04:50:07.712 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.4709999999999996e-06 2021-02-10T04:50:07.712 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.712 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.712 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:07.712 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.589190+0000", 2021-02-10T04:50:07.712 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.99916800299999997 2021-02-10T04:50:07.712 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.712 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:07.713 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.713 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.713 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.713 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(osd.6.43:137 7.14 7.68825154 (undecoded) ondisk+retry+read+known_if_redirected e1010)", 2021-02-10T04:50:07.713 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:44:27.591710+0000", 2021-02-10T04:50:07.713 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 340.02561707900003, 2021-02-10T04:50:07.713 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.99767575500000005, 2021-02-10T04:50:07.713 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:07.713 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:50:07.713 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:07.713 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.713 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:07.713 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.591710+0000", 2021-02-10T04:50:07.713 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:07.714 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.714 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.714 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:07.714 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.591710+0000", 2021-02-10T04:50:07.714 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.2529999999999999e-05 2021-02-10T04:50:07.714 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.714 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.714 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:07.714 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.591732+0000", 2021-02-10T04:50:07.714 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4399999999999998e-06 2021-02-10T04:50:07.714 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.714 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.715 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:07.715 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.591737+0000", 2021-02-10T04:50:07.715 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.0679999999999997e-06 2021-02-10T04:50:07.715 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.715 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.715 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:07.715 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.591745+0000", 2021-02-10T04:50:07.715 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.3460000000000004e-06 2021-02-10T04:50:07.715 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.715 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.715 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:07.715 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.591749+0000", 2021-02-10T04:50:07.715 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.2129999999999999e-05 2021-02-10T04:50:07.715 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.716 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.716 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.716 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.591781+0000", 2021-02-10T04:50:07.716 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.6720000000000001e-06 2021-02-10T04:50:07.716 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.716 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.716 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for peered", 2021-02-10T04:50:07.716 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:27.591787+0000", 2021-02-10T04:50:07.716 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.99758909900000003 2021-02-10T04:50:07.716 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.716 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.716 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:07.717 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.589376+0000", 2021-02-10T04:50:07.717 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.4700000000000008e-06 2021-02-10T04:50:07.717 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:07.717 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:07.717 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:07.717 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:44:28.589385+0000", 2021-02-10T04:50:07.717 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.99767575500000005 2021-02-10T04:50:07.717 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.717 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:07.717 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.717 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:07.717 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:07.717 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:50:07.718 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:50:07.718 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:50:07.719 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:50:07.719 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:50:07.727 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:50:07.728 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:50:07.728 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:50:07.728 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:50:07.728 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.3 2021-02-10T04:50:07.728 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: problem getting command descriptions from osd.3 2021-02-10T04:50:07.728 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:50:07.728 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:50:07.728 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:50:07.728 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:50:08.024 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:07 smithi164 conmon[33677]: debug 2021-02-10T04:50:07.764+0000 7fbec19a1700 -1 osd.4 1096 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:50:08.081 INFO:teuthology.orchestra.run.smithi085.stderr:Error: no container with name or ID ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3 found: no such container 2021-02-10T04:50:08.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2021-02-10T04:50:08.097 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_c1ea624123d412aff8b9d1430e36cb45fcab76b8/qa/tasks/ceph_manager.py", line 116, in wrapper return func(self) File "/home/teuthworker/src/git.ceph.com_ceph-c_c1ea624123d412aff8b9d1430e36cb45fcab76b8/qa/tasks/ceph_manager.py", line 1089, in do_sighup self.ceph_manager.signal_osd(osd, signal.SIGHUP, silent=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_c1ea624123d412aff8b9d1430e36cb45fcab76b8/qa/tasks/ceph_manager.py", line 2804, in signal_osd self.cluster).signal(sig, silent=silent) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/daemon/cephadmunit.py", line 123, in signal self.remote.sh(self.kill_cmd(sig)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/remote.py", line 259, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi085 with status 125: 'sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3' 2021-02-10T04:50:08.100 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_ceph-c_c1ea624123d412aff8b9d1430e36cb45fcab76b8/qa/tasks/ceph_manager.py", line 116, in wrapper return func(self) File "/home/teuthworker/src/git.ceph.com_ceph-c_c1ea624123d412aff8b9d1430e36cb45fcab76b8/qa/tasks/ceph_manager.py", line 1089, in do_sighup self.ceph_manager.signal_osd(osd, signal.SIGHUP, silent=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_c1ea624123d412aff8b9d1430e36cb45fcab76b8/qa/tasks/ceph_manager.py", line 2804, in signal_osd self.cluster).signal(sig, silent=silent) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/daemon/cephadmunit.py", line 123, in signal self.remote.sh(self.kill_cmd(sig)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/remote.py", line 259, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi085 with status 125: 'sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3' 2021-02-10T04:50:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:08 smithi164 conmon[29288]: cluster 2021-02-10T04:50:06.195228+0000 osd.5 (osd.5) 2021-02-10T04:50:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:08 smithi164 conmon[29288]: 142 : cluster [DBG] 5.19 scrub ok 2021-02-10T04:50:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:08 smithi164 conmon[29288]: cluster 2021-02-10T04:50:06.390047+0000 mgr.y (mgr.14140) 1155 : cluster [DBG] pgmap v1940: 188 pgs: 14 active+undersized+degraded, 28 active+undersized, 26 peering, 7 stale+active+clean, 1 active+clean+scrubbing, 112 active+clean; 893 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 46 MiB/s rd, 13 MiB/s wr, 83 op/s; 188/2223 objects degraded (8.457%); 1012 KiB/s, 1 objects/s recovering 2021-02-10T04:50:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:08 smithi164 conmon[29288]: audit 2021-02-10T04:50:07.023088+0000 mon.a 2021-02-10T04:50:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:08 smithi164 conmon[29288]: (mon.0) 2240 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:50:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:08 smithi164 conmon[29288]: cephadm 2021-02-10T04:50:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:08 smithi164 conmon[29288]: 2021-02-10T04:50:07.024772+0000 mgr.y (mgr.14140 2021-02-10T04:50:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:08 smithi164 conmon[29288]: ) 1156 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:50:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:08 smithi164 conmon[29288]: audit 2021-02-10T04:50:07.555065+0000 mon.b ( 2021-02-10T04:50:08.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:08 smithi164 conmon[29288]: mon.2) 48 : audit [DBG] from='client.? 172.21.15.85:0/2056747421' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:50:08.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:08 smithi164 conmon[29288]: audit 2021-02-10T04:50:07 2021-02-10T04:50:08.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:08 smithi164 conmon[29288]: .572684+0000 mon.a (mon.0) 2241 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:50:08.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:08 smithi085 conmon[35961]: cluster 2021-02-10T04:50:06.195228+0000 osd.5 (osd 2021-02-10T04:50:08.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:08 smithi085 conmon[35961]: .5) 142 : cluster [DBG] 5.19 scrub ok 2021-02-10T04:50:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:08 smithi085 conmon[35961]: cluster 2021-02-10T04:50:06.390047+0000 mgr.y (mgr.14140) 1155 : cluster [DBG] pgmap v1940: 188 pgs: 14 active+undersized+degraded, 28 active+undersized, 26 peering, 7 stale+active+clean, 1 active+clean+scrubbing, 112 active+clean; 893 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 46 MiB/s rd, 13 MiB/s wr, 83 op/s; 188/2223 objects degraded (8.457%); 1012 KiB/s, 1 objects/s recovering 2021-02-10T04:50:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:08 smithi085 conmon[35961]: audit 2021-02-10T04:50:07.023088+0000 mon.a (mon.0) 2240 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:50:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:08 smithi085 conmon[35961]: cephadm 2021-02-10T04:50:07.024772+0000 mgr.y (mgr.14140) 1156 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:50:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:08 smithi085 conmon[35961]: audit 2021-02-10T04:50:07.555065+0000 mon.b (mon.2) 48 : audit [DBG] from='client.? 172.21.15.85:0/2056747421' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:50:08.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:08 smithi085 conmon[35961]: audit 2021-02-10T04:50:07.572684+0000 mon.a (mon.0) 2241 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:50:08.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:08 smithi085 conmon[29319]: cluster 2021-02-10T04:50:06.195228+0000 osd.5 (osd.5) 2021-02-10T04:50:08.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:08 smithi085 conmon[29319]: 142 : cluster [DBG] 5.19 scrub ok 2021-02-10T04:50:08.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:08 smithi085 conmon[29319]: cluster 2021-02-10T04:50:06.390047+0000 mgr.y (mgr.14140) 1155 : cluster [DBG] pgmap v1940: 188 pgs: 14 active+undersized+degraded, 28 active+undersized, 26 peering, 7 stale+active+clean, 1 active+clean+scrubbing, 112 active+clean; 893 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 46 MiB/s rd, 13 MiB/s wr, 83 op/s; 188/2223 objects degraded (8.457%); 1012 KiB/s, 1 objects/s recovering 2021-02-10T04:50:08.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:08 smithi085 conmon[29319]: audit 2021-02-10T04:50:07.023088+0000 mon.a (mon.0) 2240 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:50:08.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:08 smithi085 conmon[29319]: cephadm 2021-02-10T04:50:07.024772+0000 mgr.y (mgr.14140) 1156 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:50:08.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:08 smithi085 conmon[29319]: audit 2021-02-10T04:50:07.555065+0000 mon.b (mon.2) 48 : audit [DBG] from='client.? 172.21.15.85:0/2056747421' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:50:08.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:08 smithi085 conmon[29319]: audit 2021-02-10T04:50:07.572684+0000 mon.a (mon.0) 2241 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:50:09.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:09 smithi164 conmon[29288]: cluster 2021-02-10T04:50:09.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:09 smithi164 conmon[29288]: 2021-02-10T04:50:09.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:09 smithi164 conmon[29288]: 2021-02-10T04:50:07. 2021-02-10T04:50:09.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:09 smithi164 conmon[29288]: 765111+0000 osd.4 (osd.4) 2376 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:09.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:09 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:50:09.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:09 smithi164 conmon[29288]: 50:07.765119+0000 osd.4 (osd.4) 2377 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:09.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:09 smithi164 conmon[29288]: 2021-02-10T04:50:09.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:09 smithi164 conmon[29288]: cluster 2021-02-10T04:50:09.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:09 smithi164 conmon[29288]: 2021-02-10T04:50:07.765125+0000 osd.4 (osd.4 2021-02-10T04:50:09.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:09 smithi164 conmon[29288]: ) 2378 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:09.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:09 smithi164 conmon[29288]: cluster 2021-02-10T04:50 2021-02-10T04:50:09.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:09 smithi164 conmon[29288]: :07.765131+0000 osd.4 (osd.4) 2379 : cluster 2021-02-10T04:50:09.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:09 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:09.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:09 smithi164 conmon[29288]: cluster 2021-02-10T04:50:07.767244+0000 osd.4 (osd 2021-02-10T04:50:09.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:09 smithi164 conmon[29288]: .4) 2380 : cluster [DBG] 6.21 deep-scrub ok 2021-02-10T04:50:09.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:09 smithi164 conmon[29288]: cluster 2021-02-10T04:50:08.271972+0000 osd.6 (osd. 2021-02-10T04:50:09.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:09 smithi164 conmon[29288]: 6) 140 : cluster [DBG] 6.20 deep-scrub ok 2021-02-10T04:50:09.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:09 smithi164 conmon[29288]: cluster 2021-02-10T04:50:09.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:09 smithi164 conmon[29288]: 2021-02-10T04:50:08.358296+0000 osd.0 (osd 2021-02-10T04:50:09.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:09 smithi164 conmon[29288]: .0) 152 : cluster [DBG] 2.8 scrub ok 2021-02-10T04:50:09.146 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:08 smithi164 conmon[33677]: debug 2021-02-10T04:50:08.801+0000 7fbec19a1700 -1 osd.4 1096 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:50:09.146 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:50:08 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:50:09.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[35961]: cluster 2021-02-10T04:50:07.765111 2021-02-10T04:50:09.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2376 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:09.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:50:09.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[35961]: T04:50:07.765119+0000 osd.4 (osd.4) 2377 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:09.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[35961]: 2021-02-10T04:50:09.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[35961]: cluster 2021-02-10T04:50:09.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[35961]: 2021-02-10T04:50:07.765125+0000 osd.4 ( 2021-02-10T04:50:09.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[35961]: osd.4) 2378 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:09.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:50:09.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[35961]: 02-10T04:50:07.765131+0000 osd.4 2021-02-10T04:50:09.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[35961]: (osd.4) 2379 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:09.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[35961]: 2021-02-10T04:50:09.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[35961]: cluster 2021-02-10T04:50:09.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[35961]: 2021-02-10T04:50:07.767244+0000 osd.4 (osd 2021-02-10T04:50:09.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[35961]: .4) 2380 : cluster [DBG] 6.21 deep-scrub ok 2021-02-10T04:50:09.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:50:09.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[35961]: 10T04:50:08.271972+0000 osd.6 (osd 2021-02-10T04:50:09.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[35961]: .6) 140 : cluster [DBG] 6.20 deep-scrub ok 2021-02-10T04:50:09.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:50:09.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[35961]: 02-10T04:50:08.358296+0000 2021-02-10T04:50:09.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[35961]: osd.0 (osd.0) 152 : cluster [DBG] 2.8 scrub ok 2021-02-10T04:50:09.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[35961]: 2021-02-10T04:50:09.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[29319]: cluster 2021-02-10T04:50:07.765111+0000 osd.4 (osd 2021-02-10T04:50:09.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[29319]: .4) 2376 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:09.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[29319]: cluster 2021-02-10T04:50:07.765119+0000 osd.4 (osd. 2021-02-10T04:50:09.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[29319]: 4) 2377 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:09.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[29319]: cluster 2021-02-10T04:50 2021-02-10T04:50:09.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[29319]: :07.765125+0000 osd.4 (osd.4) 2378 : 2021-02-10T04:50:09.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:09.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:50:09.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[29319]: -10T04:50:07.765131+0000 osd.4 2021-02-10T04:50:09.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[29319]: (osd.4) 2379 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:09.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[29319]: cluster 2021-02-10T04:50:09.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[29319]: 2021-02-10T04:50:07.767244+0000 osd.4 2021-02-10T04:50:09.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[29319]: (osd.4) 2380 : cluster [DBG] 6.21 deep-scrub ok 2021-02-10T04:50:09.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[29319]: cluster 2021-02-10T04:50:09.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:50:09.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[29319]: :50:08.271972+0000 osd.6 (osd. 2021-02-10T04:50:09.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[29319]: 6) 140 : cluster [DBG] 2021-02-10T04:50:09.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[29319]: 6.20 deep-scrub ok 2021-02-10T04:50:09.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:50:09.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[29319]: -02-10T04:50:08. 2021-02-10T04:50:09.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[29319]: 358296+0000 osd.0 (osd.0) 152 : cluster 2021-02-10T04:50:09.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:09 smithi085 conmon[29319]: [DBG] 2.8 scrub ok 2021-02-10T04:50:10.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:09 smithi164 conmon[33677]: debug 2021-02-10T04:50:09.812+0000 7fbec19a1700 -1 osd.4 1096 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:50:10.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:10 smithi164 conmon[29288]: cluster 2021-02-10T04:50:08.390849+0000 mgr.y ( 2021-02-10T04:50:10.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:10 smithi164 conmon[29288]: mgr.14140) 1157 : cluster [DBG] pgmap v1941: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 73 MiB/s rd, 20 MiB/s wr, 128 op/s; 381/2229 objects degraded (17.093%); 0 B/s, 1 objects/s recovering 2021-02-10T04:50:10.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:10 smithi164 conmon[29288]: cluster 2021-02-10T04:50:08.801931+0000 osd.4 (osd.4) 2381 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:10.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:10 smithi164 conmon[29288]: cluster 2021-02-10T04:50:08.801938+0000 2021-02-10T04:50:10.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:10 smithi164 conmon[29288]: osd.4 (osd.4) 2382 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:10.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:10 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:50:10.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:10 smithi164 conmon[29288]: -02-10T04:50:08.801943+0000 osd.4 2021-02-10T04:50:10.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:10 smithi164 conmon[29288]: (osd.4) 2383 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:10.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:10 smithi164 conmon[29288]: cluster 2021-02-10T04:50:10.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:10 smithi164 conmon[29288]: 2021-02-10T04:50:08. 2021-02-10T04:50:10.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:10 smithi164 conmon[29288]: 801946+0000 osd.4 (osd.4) 2384 : 2021-02-10T04:50:10.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:10 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:10.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:10 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:50:10.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:10 smithi164 conmon[29288]: :50:08.989972+0000 mgr.y (mgr. 2021-02-10T04:50:10.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:10 smithi164 conmon[29288]: 14140) 1158 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:50:10.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:10 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:50:10.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:10 smithi164 conmon[29288]: -02-10T04:50:09.024261+0000 mon.a (mon. 2021-02-10T04:50:10.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:10 smithi164 conmon[29288]: 0) 2242 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 13 pgs peering) 2021-02-10T04:50:10.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:10 smithi164 conmon[29288]: cluster 2021-02-10T04:50:10.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:10 smithi164 conmon[29288]: 2021-02-10T04:50:09.045492+0000 osd.5 (osd.5) 143 : cluster [DBG] 2021-02-10T04:50:10.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:10 smithi164 conmon[29288]: purged_snaps scrub starts 2021-02-10T04:50:10.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:10 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:50:10.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:10 smithi164 conmon[29288]: 10T04:50:09.048834+0000 osd.5 2021-02-10T04:50:10.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:10 smithi164 conmon[29288]: (osd.5) 144 : cluster 2021-02-10T04:50:10.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:10 smithi164 conmon[29288]: [DBG] purged_snaps scrub ok 2021-02-10T04:50:10.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[29319]: cluster 2021-02-10T04:50:08.390849+0000 mgr.y (mgr.14140) 1157 : cluster [DBG] pgmap v1941: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 73 MiB/s rd, 20 MiB/s wr, 128 op/s; 381/2229 objects degraded (17.093%); 0 B/s, 1 objects/s recovering 2021-02-10T04:50:10.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[29319]: cluster 2021-02-10T04:50:08.801931+0000 osd.4 (osd.4) 2381 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:10.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[29319]: cluster 2021-02-10T04:50:08.801938+0000 osd.4 (osd.4) 2382 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:10.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[29319]: cluster 2021-02-10T04:50:08.801943+0000 osd.4 (osd.4) 2383 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:10.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[29319]: cluster 2021-02-10T04:50: 2021-02-10T04:50:10.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[29319]: 08.801946+0000 osd.4 (osd.4) 2384 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:10.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:50:10.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[29319]: T04:50:08.989972+0000 mgr.y (mgr. 2021-02-10T04:50:10.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[29319]: 14140) 1158 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:50:10.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[29319]: cluster 2021-02-10T04:50:10.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[29319]: 2021-02-10T04:50:09.024261+0000 mon.a (mon.0) 2242 : 2021-02-10T04:50:10.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[29319]: cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 13 pgs peering) 2021-02-10T04:50:10.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:50:10.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[29319]: 02-10T04:50:09.045492+0000 osd.5 (osd.5) 143 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:50:10.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:50:10.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[29319]: 04:50:09.048834+0000 osd.5 (osd.5) 144 : cluster 2021-02-10T04:50:10.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[29319]: [DBG] purged_snaps scrub ok 2021-02-10T04:50:10.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[35961]: cluster 2021-02-10T04:50:08.390849 2021-02-10T04:50:10.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1157 : cluster [DBG] pgmap v1941: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 73 MiB/s rd, 20 MiB/s wr, 128 op/s; 381/2229 objects degraded (17.093%); 0 B/s, 1 objects/s recovering 2021-02-10T04:50:10.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[35961]: cluster 2021-02-10T04:50:08.801931 2021-02-10T04:50:10.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2381 : cluster [WRN] 2021-02-10T04:50:10.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:10.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[35961]: cluster 2021-02-10T04:50:10.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[35961]: 2021-02-10T04:50:08.801938+0000 2021-02-10T04:50:10.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[35961]: osd.4 (osd.4) 2382 : cluster [WRN] 2021-02-10T04:50:10.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:10.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:50:10.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[35961]: T04:50:08.801943+0000 osd.4 (osd 2021-02-10T04:50:10.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[35961]: .4) 2383 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:10.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[35961]: cluster 2021-02-10T04:50:10.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[35961]: 2021-02-10T04:50:08.801946+0000 2021-02-10T04:50:10.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[35961]: osd.4 (osd.4) 2384 : cluster [WRN] 2021-02-10T04:50:10.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:10.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:50:10.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[35961]: :50:08.989972+0000 mgr.y (mgr. 2021-02-10T04:50:10.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[35961]: 14140) 1158 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:50:10.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:50:10.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[35961]: 02-10T04:50:09.024261+0000 mon.a (mon.0 2021-02-10T04:50:10.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[35961]: ) 2242 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 13 pgs peering) 2021-02-10T04:50:10.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:50:10.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[35961]: 10T04:50:09.045492+0000 osd.5 ( 2021-02-10T04:50:10.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[35961]: osd.5) 143 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:50:10.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[35961]: cluster 2021-02-10T04:50 2021-02-10T04:50:10.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[35961]: :09.048834+0000 osd.5 (osd.5) 144 2021-02-10T04:50:10.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:10 smithi085 conmon[35961]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:50:10.896 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:50:10.896 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:50:11.134 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:11 smithi085 conmon[29319]: cluster 2021-02-10T04:50:09.813041+0000 osd.4 ( 2021-02-10T04:50:11.134 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:11 smithi085 conmon[29319]: osd.4) 2385 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:11.135 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:11 smithi085 conmon[29319]: cluster 2021-02-10T04:50:09.813046+0000 osd.4 (osd.4) 2386 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:11.135 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:11 smithi085 conmon[29319]: cluster 2021-02-10T04:50:09.813050+0000 osd.4 (osd.4) 2387 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:11.135 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:11 smithi085 conmon[29319]: cluster 2021-02-10T04:50:09.813053+0000 osd.4 (osd.4) 2388 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:11.135 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:11 smithi085 conmon[29319]: cluster 2021-02-10T04:50:10.040969+0000 mon.a (mon.0) 2243 : cluster [DBG] osdmap e1097: 8 total, 7 up, 7 in 2021-02-10T04:50:11.135 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:11 smithi085 conmon[29319]: cluster 2021-02-10T04:50:10.308279+0000 2021-02-10T04:50:11.135 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:11 smithi085 conmon[29319]: osd.6 (osd.6) 141 : cluster [DBG] 6.27 scrub ok 2021-02-10T04:50:11.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:11 smithi085 conmon[35961]: cluster 2021-02-10T04:50:09.813041 2021-02-10T04:50:11.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:11 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2385 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:11.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:11 smithi085 conmon[35961]: cluster 2021-02-10T04:50: 2021-02-10T04:50:11.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:11 smithi085 conmon[35961]: 09.813046+0000 osd.4 (osd.4) 2386 : cluster 2021-02-10T04:50:11.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:11 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:11.137 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:11 smithi085 conmon[35961]: cluster 2021-02-10T04:50:11.137 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:11 smithi085 conmon[35961]: 2021-02-10T04:50 2021-02-10T04:50:11.137 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:11 smithi085 conmon[35961]: :09.813050+0000 osd.4 (osd.4) 2387 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:11.137 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:11 smithi085 conmon[35961]: cluster 2021-02-10T04:50:11.137 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:11 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:50:11.138 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:11 smithi085 conmon[35961]: T04:50:09.813053+0000 osd.4 (osd.4) 2388 2021-02-10T04:50:11.138 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:11 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:11.138 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:11 smithi085 conmon[35961]: cluster 2021-02-10T04:50:11.138 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:11 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:50:11.138 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:11 smithi085 conmon[35961]: 04:50:10.040969+0000 mon.a (mon.0) 2243 : cluster [DBG] osdmap e1097: 8 total, 7 up, 7 in 2021-02-10T04:50:11.138 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:11 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:50:11.139 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:11 smithi085 conmon[35961]: -02-10T04:50:10.308279+0000 2021-02-10T04:50:11.139 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:11 smithi085 conmon[35961]: osd.6 (osd.6) 141 : cluster [DBG] 6.27 scrub ok 2021-02-10T04:50:11.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:10 smithi164 conmon[33677]: debug 2021-02-10T04:50:11.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:10 smithi164 conmon[33677]: 2021-02-10T04:50:10.808+0000 7fbec19a1700 -1 osd.4 1097 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:50:11.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:11 smithi164 conmon[29288]: cluster 2021-02-10T04:50:09.813041+0000 osd.4 (osd.4) 2021-02-10T04:50:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:11 smithi164 conmon[29288]: 2385 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:11 smithi164 conmon[29288]: cluster 2021-02-10T04:50:09.813046+0000 osd.4 (osd.4) 2386 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:11 smithi164 conmon[29288]: cluster 2021-02-10T04:50:09.813050+0000 osd.4 (osd.4) 2387 : cluster 2021-02-10T04:50:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:11 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:11 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:50:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:11 smithi164 conmon[29288]: -02-10T04:50:09.813053+0000 osd.4 (osd.4) 2388 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:11.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:11 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:50:11.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:11 smithi164 conmon[29288]: 02-10T04:50:10.040969+0000 mon.a (mon.0) 2243 : cluster [DBG] osdmap e1097: 8 total, 7 up, 7 in 2021-02-10T04:50:11.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:11 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:50:11.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:11 smithi164 conmon[29288]: T04:50:10.308279+0000 osd.6 (osd.6) 141 : 2021-02-10T04:50:11.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:11 smithi164 conmon[29288]: cluster [DBG] 6.27 scrub ok 2021-02-10T04:50:11.356 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight 2021-02-10T04:50:11.593 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:50:12.047 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:11 smithi164 conmon[33677]: debug 2021-02-10T04:50:11.768+0000 7fbec19a1700 -1 osd.4 1097 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:50:12.048 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:12 smithi085 conmon[29319]: cluster 2021-02-10T04:50:10.176872 2021-02-10T04:50:12.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:12 smithi085 conmon[29319]: +0000 osd.5 (osd.5) 145 : cluster [DBG] 6.d scrub ok 2021-02-10T04:50:12.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:12 smithi085 conmon[29319]: cluster 2021-02-10T04:50:10.391462+0000 mgr.y (mgr.14140) 1159 : cluster [DBG] pgmap v1943: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 79 MiB/s rd, 22 MiB/s wr, 138 op/s; 381/2229 objects degraded (17.093%); 0 B/s, 2 objects/s recovering 2021-02-10T04:50:12.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:12 smithi085 conmon[29319]: cluster 2021-02-10T04:50:10.809980+0000 osd.4 (osd.4) 2389 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:12.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:12 smithi085 conmon[29319]: cluster 2021-02-10T04:50:10.809986+0000 osd.4 (osd.4) 2390 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:12.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:12 smithi085 conmon[29319]: cluster 2021-02-10T04:50:10.809992+0000 osd.4 (osd.4) 2391 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:12.049 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:12 smithi085 conmon[29319]: cluster 2021-02-10T04:50:10.810001+0000 osd.4 (osd.4) 2392 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:12.049 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:12 smithi164 conmon[29288]: cluster 2021-02-10T04:50:10.176872+0000 osd.5 (osd. 2021-02-10T04:50:12.049 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:12 smithi164 conmon[29288]: 5) 145 : cluster [DBG] 6.d scrub ok 2021-02-10T04:50:12.050 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:12 smithi164 conmon[29288]: cluster 2021-02-10T04:50:10.391462+0000 mgr.y (mgr.14140) 1159 : 2021-02-10T04:50:12.050 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:12 smithi164 conmon[29288]: cluster [DBG] pgmap v1943: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 79 MiB/s rd, 22 MiB/s wr, 138 op/s; 381/2229 objects degraded (17.093%); 0 B/s, 2 objects/s recovering 2021-02-10T04:50:12.050 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:12 smithi164 conmon[29288]: cluster 2021-02-10T04:50:10.809980+0000 osd.4 ( 2021-02-10T04:50:12.050 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:12 smithi164 conmon[29288]: osd.4) 2389 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:12.050 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:12 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:50:12.050 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:12 smithi164 conmon[29288]: :50:10.809986+0000 osd.4 (osd.4) 2390 : 2021-02-10T04:50:12.050 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:12 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:12.050 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:12 smithi164 conmon[29288]: cluster 2021-02-10T04:50:12.050 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:12 smithi164 conmon[29288]: 2021-02-10T04:50:10.809992 2021-02-10T04:50:12.050 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:12 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2391 2021-02-10T04:50:12.050 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:12 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:12.050 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:12 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:50:12.050 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:12 smithi164 conmon[29288]: 02-10T04:50:10.810001+0000 osd.4 (osd.4) 2392 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:12.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:12 smithi085 conmon[35961]: cluster 2021-02-10T04:50:10.176872+0000 2021-02-10T04:50:12.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:12 smithi085 conmon[35961]: osd.5 (osd.5) 145 : cluster [DBG] 6.d scrub ok 2021-02-10T04:50:12.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:12 smithi085 conmon[35961]: cluster 2021-02-10T04:50:10.391462+0000 mgr.y (mgr.14140) 1159 : cluster [DBG] pgmap v1943: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 79 MiB/s rd, 22 MiB/s wr, 138 op/s; 381/2229 objects degraded (17.093%); 0 B/s, 2 objects/s recovering 2021-02-10T04:50:12.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:12 smithi085 conmon[35961]: cluster 2021-02-10T04:50:10.809980+0000 osd.4 (osd.4 2021-02-10T04:50:12.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:12 smithi085 conmon[35961]: ) 2389 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:12.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:12 smithi085 conmon[35961]: cluster 2021-02-10T04:50:10.809986+0000 osd.4 (osd.4) 2390 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:12.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:12 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:50:12.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:12 smithi085 conmon[35961]: T04:50:10.809992+0000 osd.4 (osd.4) 2391 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:12.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:12 smithi085 conmon[35961]: 2021-02-10T04:50:12.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:12 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:50:12.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:12 smithi085 conmon[35961]: -10T04:50:10.810001+0000 osd.4 (osd.4) 2392 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:12.897 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:50:13.056 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:12 smithi164 conmon[33677]: debug 2021-02-10T04:50:13.056 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:12 smithi164 conmon[33677]: 2021-02-10T04:50:12.767+0000 7fbec19a1700 -1 osd.4 1097 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:50:13.056 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:13 smithi164 conmon[29288]: cluster 2021-02-10T04:50:11.770513+0000 osd.4 ( 2021-02-10T04:50:13.057 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:13 smithi164 conmon[29288]: osd.4) 2393 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:13.057 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:13 smithi164 conmon[29288]: cluster 2021-02-10T04:50:11.770525+0000 osd.4 (osd.4) 2394 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:13.057 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:13 smithi164 conmon[29288]: cluster 2021-02-10T04:50:11.770532+0000 osd.4 (osd.4) 2395 2021-02-10T04:50:13.057 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:13 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:13.057 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:13 smithi164 conmon[29288]: cluster 2021-02-10T04:50 2021-02-10T04:50:13.057 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:13 smithi164 conmon[29288]: :11.770538+0000 osd.4 (osd.4) 2396 : cluster 2021-02-10T04:50:13.057 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:13 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:13.079 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:13 smithi085 conmon[29319]: cluster 2021-02-10T04:50:13.079 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:13 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:50:13.079 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:13 smithi085 conmon[29319]: 04:50:11.770513+0000 osd.4 (osd.4) 2393 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:13.079 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:13 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:50:13.079 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:13 smithi085 conmon[29319]: -02-10T04:50:11.770525+0000 osd.4 (osd.4) 2021-02-10T04:50:13.080 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:13 smithi085 conmon[29319]: 2394 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:13.080 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:13 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:50:13.080 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:13 smithi085 conmon[29319]: 50:11.770532+0000 osd.4 (osd.4) 2021-02-10T04:50:13.080 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:13 smithi085 conmon[29319]: 2395 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:13.080 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:13 smithi085 conmon[29319]: cluster 2021-02-10T04:50:13.080 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:13 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:50:13.080 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:13 smithi085 conmon[29319]: :50:11.770538+0000 osd.4 (osd.4 2021-02-10T04:50:13.080 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:13 smithi085 conmon[29319]: ) 2396 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:13.080 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:13 smithi085 conmon[35961]: cluster 2021-02-10T04:50:11.770513+0000 osd.4 (osd.4) 2393 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:13.081 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:13 smithi085 conmon[35961]: cluster 2021-02-10T04:50:11.770525+0000 osd.4 (osd.4) 2394 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:13.081 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:13 smithi085 conmon[35961]: cluster 2021-02-10T04:50:11.770532+0000 osd.4 (osd.4) 2395 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:13.081 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:13 smithi085 conmon[35961]: cluster 2021-02-10T04:50:11.770538+0000 osd.4 (osd.4) 2396 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:13.166 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:50:14.054 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:13 smithi164 conmon[33677]: debug 2021-02-10T04:50:13.751+0000 7fbec19a1700 -1 osd.4 1097 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:50:14.055 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: cluster 2021-02-10T04:50:14.056 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[29319]: cluster 2021-02-10T04:50:12.392196+0000 mgr.y (mgr.14140 2021-02-10T04:50:14.056 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[29319]: ) 1160 : cluster [DBG] pgmap v1944: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 26 MiB/s rd, 7.3 MiB/s wr, 43 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:14.056 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[29319]: cluster 2021-02-10T04:50:12.768815+0000 osd.4 (osd.4) 2397 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:14.057 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[29319]: cluster 2021-02-10T04:50:12.768823+0000 osd.4 (osd.4) 2398 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:14.057 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[29319]: cluster 2021-02-10T04:50:12.768830+0000 osd.4 (osd.4) 2399 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:14.057 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[29319]: cluster 2021-02-10T04:50:12.768833+0000 osd.4 (osd.4) 2400 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:14.057 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[29319]: cluster 2021-02-10T04:50:12.772080+0000 osd.4 (osd.4) 2401 : cluster [DBG] 2.25 scrub ok 2021-02-10T04:50:14.057 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[29319]: cluster 2021-02-10T04:50:13.047043+0000 mon.a (mon.0) 2244 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 347 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:50:14.057 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[29319]: cluster 2021-02-10T04:50:13.454022+0000 mon.a (mon.0) 2245 : cluster [WRN] Health check update: Degraded data redundancy: 381/2229 objects degraded (17.093%), 32 pgs degraded (PG_DEGRADED) 2021-02-10T04:50:14.367 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[35961]: cluster 2021-02-10T04:50:12. 2021-02-10T04:50:14.367 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[35961]: 392196+0000 mgr.y (mgr.14140) 1160 : cluster [DBG] pgmap v1944: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 26 MiB/s rd, 7.3 MiB/s wr, 43 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:14.367 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[35961]: cluster 2021-02-10T04:50:14.367 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[35961]: 2021-02-10T04:50:12.768815 2021-02-10T04:50:14.367 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2397 : cluster [WRN] 2021-02-10T04:50:14.367 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:14.367 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:50:14.367 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[35961]: 02-10T04:50:12.768823+0000 osd.4 ( 2021-02-10T04:50:14.367 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[35961]: osd.4) 2398 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:14.367 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[35961]: cluster 2021-02-10T04:50:14.368 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:50:14.368 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[35961]: :50:12.768830+0000 osd.4 (osd.4) 2399 2021-02-10T04:50:14.368 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:14.368 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:50:14.368 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[35961]: -10T04:50:12.768833+0000 osd.4 2021-02-10T04:50:14.368 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[35961]: (osd.4) 2400 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:14.368 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:50:14.368 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[35961]: 02-10T04:50:12.772080+0000 osd.4 2021-02-10T04:50:14.368 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[35961]: (osd.4) 2401 : 2021-02-10T04:50:14.368 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[35961]: cluster [DBG] 2.25 scrub ok 2021-02-10T04:50:14.368 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:50:14.369 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[35961]: -02-10T04:50:13.047043 2021-02-10T04:50:14.369 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2244 : cluster 2021-02-10T04:50:14.369 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[35961]: [WRN] Health check failed: 4 slow ops, oldest one blocked for 347 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:50:14.369 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:50:14.369 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[35961]: 02-10T04:50:13.454022+0000 2021-02-10T04:50:14.369 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[35961]: mon.a (mon.0) 2245 : cluster [WRN] 2021-02-10T04:50:14.369 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:14 smithi085 conmon[35961]: Health check update: Degraded data redundancy: 381/2229 objects degraded (17.093%), 32 pgs degraded (PG_DEGRADED) 2021-02-10T04:50:14.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: 2021-02-10T04:50:14.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: 2021 2021-02-10T04:50:14.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: - 2021-02-10T04:50:14.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: 02 2021-02-10T04:50:14.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: - 2021-02-10T04:50:14.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: 10T04:50:12.392196+0000 2021-02-10T04:50:14.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: mgr.y (mgr.14140) 1160 : cluster [DBG] pgmap v1944: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 26 MiB/s rd, 7.3 MiB/s wr, 43 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:14.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: cluster 2021-02-10T04:50:14.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: 2021-02-10T04:50: 2021-02-10T04:50:14.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: 12.768815+0000 osd.4 (osd.4 2021-02-10T04:50:14.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: ) 2397 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:14.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: 2021-02-10T04:50:14.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: cluster 2021-02-10T04:50:14.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: 2021-02-10T04:50 2021-02-10T04:50:14.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: :12.768823+0000 osd.4 (osd.4) 2398 : cluster 2021-02-10T04:50:14.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:14.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: cluster 2021-02-10T04:50:14.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:50:14.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: :50:12.768830+0000 2021-02-10T04:50:14.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: osd.4 (osd.4) 2399 : 2021-02-10T04:50:14.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:14.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: cluster 2021-02-10T04:50:14.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: 2021-02-10T04:50:12.768833+0000 osd.4 2021-02-10T04:50:14.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: (osd.4) 2400 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:14.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:50:14.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: 50:12.772080+0000 osd.4 (osd.4) 2401 : cluster [DBG] 2.25 scrub ok 2021-02-10T04:50:14.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: cluster 2021-02-10T04:50:13.047043+0000 2021-02-10T04:50:14.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: mon.a (mon.0) 2244 : cluster [WRN] 2021-02-10T04:50:14.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: Health check failed: 4 slow ops, oldest one blocked for 347 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:50:14.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: cluster 2021-02-10T04:50:14.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: 2021-02-10T04:50:13.454022 2021-02-10T04:50:14.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2245 : cluster [WRN] Health check update: Degraded data redundancy: 381/2229 objects degraded (17.093%), 32 pgs degraded (PG_DEGRADED) 2021-02-10T04:50:14.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[29288]: 2021-02-10T04:50:15.063 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.3 to 1 (10000) 2021-02-10T04:50:15.064 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:14 smithi164 conmon[33677]: debug 2021-02-10T04:50:14.734+0000 7fbec19a1700 -1 osd.4 1097 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:50:15.064 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:15 smithi164 conmon[29288]: cluster 2021-02-10T04:50:13.752851+0000 osd.4 ( 2021-02-10T04:50:15.064 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:15 smithi164 conmon[29288]: osd.4) 2402 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:15.064 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:15 smithi164 conmon[29288]: cluster 2021-02-10T04:50:13.752861+0000 osd.4 (osd.4) 2403 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:15.064 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:15 smithi164 conmon[29288]: cluster 2021-02-10T04:50:13.752867+0000 osd.4 (osd 2021-02-10T04:50:15.064 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:15 smithi164 conmon[29288]: .4) 2404 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:15.064 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:15 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:50:15.065 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:15 smithi164 conmon[29288]: 10T04:50:13.752873+0000 osd.4 (osd 2021-02-10T04:50:15.065 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:15 smithi164 conmon[29288]: .4) 2405 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:15.065 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:15 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:50:15.065 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:15 smithi164 conmon[29288]: 02-10T04:50:14.481969+0000 mon.a 2021-02-10T04:50:15.065 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:15 smithi164 conmon[29288]: (mon.0) 2246 : audit [INF] from='client.? 172.21.15.85:0/2837806240' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-02-10T04:50:15.065 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:15 smithi164 conmon[29288]: 2021-02-10T04:50:15.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:15 smithi085 conmon[29319]: cluster 2021-02-10T04:50:13.752851 2021-02-10T04:50:15.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:15 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2402 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:15.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:15 smithi085 conmon[29319]: cluster 2021-02-10T04:50:13.752861+0000 osd.4 (osd.4) 2403 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:15.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:15 smithi085 conmon[29319]: cluster 2021-02-10T04:50:13.752867+0000 osd.4 (osd.4) 2404 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:15.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:15 smithi085 conmon[29319]: cluster 2021-02-10T04:50:13.752873+0000 osd.4 (osd.4) 2405 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:15.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:15 smithi085 conmon[29319]: audit 2021-02-10T04:50:14.481969+0000 mon.a (mon.0) 2246 : audit [INF] from='client.? 172.21.15.85:0/2837806240' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-02-10T04:50:15.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:15 smithi085 conmon[35961]: cluster 2021-02-10T04:50:13. 2021-02-10T04:50:15.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:15 smithi085 conmon[35961]: 752851+0000 osd.4 (osd.4) 2402 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:15.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:15 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:50:15.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:15 smithi085 conmon[35961]: 04:50:13.752861+0000 osd.4 (osd.4) 2403 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:15.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:15 smithi085 conmon[35961]: cluster 2021-02-10T04:50:15.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:15 smithi085 conmon[35961]: 2021-02-10T04:50:13.752867+0000 2021-02-10T04:50:15.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:15 smithi085 conmon[35961]: osd.4 (osd.4) 2404 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:15.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:15 smithi085 conmon[35961]: 2021-02-10T04:50:15.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:15 smithi085 conmon[35961]: cluster 2021-02-10T04:50: 2021-02-10T04:50:15.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:15 smithi085 conmon[35961]: 13.752873+0000 osd.4 (osd.4) 2405 2021-02-10T04:50:15.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:15 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:15.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:15 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:50:15.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:15 smithi085 conmon[35961]: :50:14.481969+0000 mon.a (mon.0 2021-02-10T04:50:15.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:15 smithi085 conmon[35961]: ) 2246 : audit [INF] from='client.? 172.21.15.85:0/2837806240' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-02-10T04:50:16.065 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is set 2021-02-10T04:50:16.067 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:15 smithi164 conmon[33677]: debug 2021-02-10T04:50:16.068 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:15 smithi164 conmon[33677]: 2021-02-10T04:50:15.723+0000 7fbec19a1700 -1 osd.4 1098 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:50:16.068 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:15 smithi164 conmon[33677]: 2021-02-10T04:50:16.068 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: cluster 2021-02-10T04:50:14.158654+0000 osd.5 2021-02-10T04:50:16.068 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: (osd.5) 146 : cluster [DBG] 4.2 scrub ok 2021-02-10T04:50:16.068 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: cluster 2021-02-10T04:50:14.392982+0000 mgr.y (mgr.14140) 1161 : cluster [DBG] pgmap v1945: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 22 MiB/s rd, 6.1 MiB/s wr, 36 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:16.068 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: cluster 2021-02-10T04:50:14.735836+0000 osd.4 (osd.4) 2021-02-10T04:50:16.068 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: 2406 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:16.069 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: cluster 2021-02-10T04:50:14. 2021-02-10T04:50:16.069 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: 735844+0000 osd.4 (osd.4) 2407 : cluster [WRN] 2021-02-10T04:50:16.069 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:16.069 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: cluster 2021-02-10T04:50:16.069 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: 2021-02-10T04:50:14.735849+0000 osd.4 (osd. 2021-02-10T04:50:16.069 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: 4) 2408 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:16.069 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:50:16.069 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: 04:50:14.735854+0000 osd.4 (osd 2021-02-10T04:50:16.069 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: .4) 2409 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:16.069 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:50:16.069 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: 02-10T04:50:15.054978+0000 2021-02-10T04:50:16.069 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: mon.a (mon.0) 2247 : audit [INF] from='client.? 172.21.15.85:0/253912181' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:50:16.069 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: 2021-02-10T04:50:16.070 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:50:16.070 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: 10T04:50:15.058529+0000 mon.a (mon.0) 2248 2021-02-10T04:50:16.070 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: : audit [INF] from='client.? 172.21.15.85:0/2837806240' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2021-02-10T04:50:16.070 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: cluster 2021-02-10T04:50:15 2021-02-10T04:50:16.070 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: .058731+0000 mon.a (mon.0) 2249 : cluster 2021-02-10T04:50:16.070 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: [DBG] osdmap e1098: 8 total, 7 up, 7 in 2021-02-10T04:50:16.070 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: cluster 2021-02-10T04:50:16.070 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: 2021-02-10T04:50:15.255132+0000 osd.6 ( 2021-02-10T04:50:16.070 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: osd.6) 142 : cluster [DBG] 5.1f deep-scrub ok 2021-02-10T04:50:16.070 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:50:16.070 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: 04:50:16.055350+0000 mon.a (mon.0 2021-02-10T04:50:16.070 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: ) 2250 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:50:16.070 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: audit 2021-02-10T04: 2021-02-10T04:50:16.071 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: 50:16.059646+0000 mon.a (mon.0) 2021-02-10T04:50:16.071 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: 2251 : audit [INF] from='client.? 172.21.15.85:0/253912181' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:50:16.071 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: cluster 2021-02-10T04:50 2021-02-10T04:50:16.071 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: :16.059729+0000 mon.a (mon.0) 2252 : cluster 2021-02-10T04:50:16.071 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[29288]: [DBG] osdmap e1099: 8 total, 7 up, 7 in 2021-02-10T04:50:16.343 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: cluster 2021-02-10T04:50:14.158654+0000 osd.5 (osd.5) 146 2021-02-10T04:50:16.344 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: : cluster [DBG] 4.2 scrub ok 2021-02-10T04:50:16.344 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: cluster 2021-02-10T04:50:14.392982+0000 mgr.y (mgr.14140) 1161 : cluster [DBG] pgmap v1945: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 22 MiB/s rd, 6.1 MiB/s wr, 36 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:16.344 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:50:16.344 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: -10T04:50:14.735836+0000 osd.4 (osd.4) 2406 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:16.344 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: 2021-02-10T04:50:16.344 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: cluster 2021-02-10T04:50:16.344 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:50:16.344 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: T04:50:14.735844+0000 osd.4 ( 2021-02-10T04:50:16.344 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: osd.4) 2407 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:16.344 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: 2021-02-10T04:50:16.344 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:50:16.345 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: 02-10T04:50:14.735849+0000 osd.4 (osd. 2021-02-10T04:50:16.345 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: 4) 2408 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:16.345 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: cluster 2021-02-10T04:50:16.345 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: 2021-02-10T04:50 2021-02-10T04:50:16.345 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: :14.735854+0000 osd.4 (osd. 2021-02-10T04:50:16.345 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: 4) 2409 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:16.345 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: audit 2021-02-10T04:50:16.345 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:50:16.345 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: 10T04:50:15.054978 2021-02-10T04:50:16.345 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2247 : audit 2021-02-10T04:50:16.345 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: [INF] from='client.? 172.21.15.85:0/253912181' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:50:16.345 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:50:16.345 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: 02-10T04:50:15.058529+0000 2021-02-10T04:50:16.346 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: mon.a (mon.0) 2248 : audit 2021-02-10T04:50:16.346 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: [INF] from='client.? 172.21.15.85:0/2837806240' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2021-02-10T04:50:16.346 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:50:16.346 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: 02-10T04:50:15.058731+0000 mon.a ( 2021-02-10T04:50:16.346 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: mon.0) 2249 : cluster [DBG] osdmap e1098: 8 total, 7 up, 7 in 2021-02-10T04:50:16.346 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:50:16.346 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: 02-10T04:50:15.255132+0000 osd.6 (osd. 2021-02-10T04:50:16.346 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: 6) 142 : cluster [DBG] 5.1f deep-scrub ok 2021-02-10T04:50:16.346 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: cluster 2021-02-10T04:50:16.055350+0000 2021-02-10T04:50:16.346 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: mon.a (mon.0) 2250 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:50:16.346 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: audit 2021-02-10T04:50: 2021-02-10T04:50:16.346 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: 16.059646+0000 mon.a (mon.0) 2251 : 2021-02-10T04:50:16.346 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: audit [INF] from='client.? 172.21.15.85:0/253912181' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:50:16.347 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: cluster 2021-02-10T04:50:16.347 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: 2021-02-10T04:50 2021-02-10T04:50:16.347 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: :16.059729+0000 mon.a ( 2021-02-10T04:50:16.347 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: mon.0) 2252 : cluster [DBG] 2021-02-10T04:50:16.347 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[29319]: osdmap e1099: 8 total, 7 up, 7 in 2021-02-10T04:50:16.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: cluster 2021-02-10T04:50:14.158654 2021-02-10T04:50:16.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: +0000 osd.5 (osd.5) 146 : cluster [DBG] 4.2 scrub ok 2021-02-10T04:50:16.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:50:16.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: 10T04:50:14.392982+0000 2021-02-10T04:50:16.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: mgr.y (mgr.14140) 1161 : cluster 2021-02-10T04:50:16.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: [DBG] pgmap v1945: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 22 MiB/s rd, 6.1 MiB/s wr, 36 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:16.347 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:50:16.348 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: 02-10T04:50:14.735836+0000 osd.4 (osd.4 2021-02-10T04:50:16.348 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: ) 2406 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:16.348 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:50:16.348 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: 50:14.735844+0000 osd.4 (osd.4 2021-02-10T04:50:16.348 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: ) 2407 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:16.348 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:50:16.348 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: :50:14.735849+0000 osd.4 (osd. 2021-02-10T04:50:16.348 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: 4) 2408 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:16.348 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:50:16.348 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: T04:50:14.735854+0000 osd.4 2021-02-10T04:50:16.348 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: (osd.4) 2409 : cluster 2021-02-10T04:50:16.348 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:16.348 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:50:16.348 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: 04:50:15.054978+0000 mon.a (mon. 2021-02-10T04:50:16.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: 0) 2247 : audit [INF] from='client.? 172.21.15.85:0/253912181' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:50:16.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:50:16.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: 04:50:15.058529+0000 mon.a (mon. 2021-02-10T04:50:16.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: 0) 2248 : audit [INF] from='client.? 172.21.15.85:0/2837806240' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2021-02-10T04:50:16.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:50:16.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: 10T04:50:15.058731+0000 2021-02-10T04:50:16.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: mon.a (mon.0) 2249 : cluster 2021-02-10T04:50:16.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: [DBG] osdmap e1098: 8 total, 7 up, 7 in 2021-02-10T04:50:16.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:50:16.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: 02-10T04:50:15.255132+0000 osd.6 ( 2021-02-10T04:50:16.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: osd.6) 142 : cluster [DBG] 5.1f deep-scrub ok 2021-02-10T04:50:16.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:50:16.349 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: -10T04:50:16.055350+0000 mon.a (mon. 2021-02-10T04:50:16.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: 0) 2250 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:50:16.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: audit 2021 2021-02-10T04:50:16.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: -02-10T04:50 2021-02-10T04:50:16.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: :16.059646+0000 mon.a (mon.0) 2021-02-10T04:50:16.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: 2251 : audit [INF] from='client.? 172.21.15.85:0/253912181' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:50:16.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:50:16.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: 02-10T04:50: 2021-02-10T04:50:16.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: 16.059729+0000 mon.a (mon. 2021-02-10T04:50:16.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: 0) 2252 : cluster [DBG] osdmap e1099: 8 total, 7 up, 7 in 2021-02-10T04:50:16.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:16 smithi085 conmon[35961]: 2021-02-10T04:50:17.078 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:16 smithi164 conmon[33677]: debug 2021-02-10T04:50:16.757+0000 7fbec19a1700 -1 osd.4 1099 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:50:17.079 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: cluster 2021-02-10T04:50:17.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 2021-02-10T04:50:17.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 2021 2021-02-10T04:50:17.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: - 2021-02-10T04:50:17.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 02 2021-02-10T04:50:17.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: - 2021-02-10T04:50:17.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 10 2021-02-10T04:50:17.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: T 2021-02-10T04:50:17.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 04 2021-02-10T04:50:17.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: : 2021-02-10T04:50:17.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 50 2021-02-10T04:50:17.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: : 2021-02-10T04:50:17.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 15 2021-02-10T04:50:17.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: . 2021-02-10T04:50:17.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 724788 2021-02-10T04:50:17.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: +0000 2021-02-10T04:50:17.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 2021-02-10T04:50:17.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: osd.4 2021-02-10T04:50:17.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: ( 2021-02-10T04:50:17.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: osd 2021-02-10T04:50:17.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: . 2021-02-10T04:50:17.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 4 2021-02-10T04:50:17.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: ) 2021-02-10T04:50:17.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 2410 2021-02-10T04:50:17.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: : 2021-02-10T04:50:17.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: cluster 2021-02-10T04:50:17.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 2021-02-10T04:50:17.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: [WRN] 2021-02-10T04:50:17.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 2021-02-10T04:50:17.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:17.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 2021-02-10T04:50:17.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: cluster 2021-02-10T04:50:17.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 2021-02-10T04:50:17.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 2021 2021-02-10T04:50:17.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: - 2021-02-10T04:50:17.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 02 2021-02-10T04:50:17.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: - 2021-02-10T04:50:17.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 10 2021-02-10T04:50:17.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: T 2021-02-10T04:50:17.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 04 2021-02-10T04:50:17.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: : 2021-02-10T04:50:17.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 50 2021-02-10T04:50:17.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: : 2021-02-10T04:50:17.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 15 2021-02-10T04:50:17.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: . 2021-02-10T04:50:17.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 724798 2021-02-10T04:50:17.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: +0000 2021-02-10T04:50:17.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 2021-02-10T04:50:17.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: osd.4 2021-02-10T04:50:17.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: ( 2021-02-10T04:50:17.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: osd 2021-02-10T04:50:17.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: . 2021-02-10T04:50:17.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 4 2021-02-10T04:50:17.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: ) 2021-02-10T04:50:17.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 2411 2021-02-10T04:50:17.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: : 2021-02-10T04:50:17.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: cluster 2021-02-10T04:50:17.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 2021-02-10T04:50:17.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: [WRN] 2021-02-10T04:50:17.403 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 2021-02-10T04:50:17.403 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:17.403 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 2021-02-10T04:50:17.403 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: cluster 2021-02-10T04:50:17.403 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 2021-02-10T04:50:17.403 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 2021 2021-02-10T04:50:17.403 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: - 2021-02-10T04:50:17.403 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 02 2021-02-10T04:50:17.403 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: - 2021-02-10T04:50:17.403 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 10 2021-02-10T04:50:17.403 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: T 2021-02-10T04:50:17.403 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 04 2021-02-10T04:50:17.403 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: : 2021-02-10T04:50:17.404 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 50 2021-02-10T04:50:17.404 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: : 2021-02-10T04:50:17.404 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 15 2021-02-10T04:50:17.404 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: . 2021-02-10T04:50:17.404 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 724805 2021-02-10T04:50:17.404 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: +0000 2021-02-10T04:50:17.404 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 2021-02-10T04:50:17.404 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: osd.4 2021-02-10T04:50:17.404 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: ( 2021-02-10T04:50:17.404 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: osd 2021-02-10T04:50:17.404 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: . 2021-02-10T04:50:17.404 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 4 2021-02-10T04:50:17.404 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: ) 2021-02-10T04:50:17.404 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 2412 2021-02-10T04:50:17.405 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: : 2021-02-10T04:50:17.405 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: cluster 2021-02-10T04:50:17.405 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 2021-02-10T04:50:17.405 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: [WRN] 2021-02-10T04:50:17.405 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 2021-02-10T04:50:17.405 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:17.405 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 2021-02-10T04:50:17.405 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: cluster 2021-02-10T04:50:17.405 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 2021-02-10T04:50:17.405 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 2021 2021-02-10T04:50:17.405 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: - 2021-02-10T04:50:17.405 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 02 2021-02-10T04:50:17.405 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: - 2021-02-10T04:50:17.406 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 10T04:50:15.724810+0000 osd.4 (osd.4) 2413 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:17.406 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: cluster 2021-02-10T04:50:17.406 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 2021-02-10T04:50:16.499477+0000 osd.2 (osd.2) 1957 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:50:17.406 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: cluster 2021-02-10T04:50:17.406 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 2021-02-10T04:50:17.406 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[29288]: 2021-02-10T04:50:16.499886+0000 osd.2 (osd.2) 1958 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:50:17.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[29319]: cluster 2021-02-10T04:50:15.724788+0000 osd.4 (osd. 2021-02-10T04:50:17.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[29319]: 4) 2410 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[29319]: cluster 2021-02-10T04:50:15.724798 2021-02-10T04:50:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2411 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:50:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[29319]: T04:50:15.724805+0000 osd.4 (osd.4) 2412 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[29319]: cluster 2021-02-10T04:50:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[29319]: 2021-02-10T04:50:15. 2021-02-10T04:50:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[29319]: 724810+0000 osd.4 (osd.4) 2413 : cluster 2021-02-10T04:50:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:50:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[29319]: :50:16.499477+0000 osd.2 (osd.2 2021-02-10T04:50:17.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[29319]: ) 1957 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:50:17.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[29319]: cluster 2021-02-10T04:50:17.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[29319]: 2021-02-10T04:50 2021-02-10T04:50:17.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[29319]: :16.499886+0000 osd.2 (osd.2) 2021-02-10T04:50:17.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[29319]: 1958 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:50:17.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[35961]: cluster 2021-02-10T04:50:15.724788+0000 2021-02-10T04:50:17.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[35961]: osd.4 (osd.4) 2410 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:17.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[35961]: cluster 2021-02-10T04:50:17.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[35961]: 2021-02-10T04:50:15.724798+0000 2021-02-10T04:50:17.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[35961]: osd.4 (osd.4) 2411 : cluster 2021-02-10T04:50:17.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:17.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[35961]: cluster 2021-02-10T04:50:17.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[35961]: 2021-02 2021-02-10T04:50:17.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[35961]: -10T04:50:15.724805+0000 osd.4 2021-02-10T04:50:17.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[35961]: (osd.4) 2412 : cluster 2021-02-10T04:50:17.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:17.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[35961]: cluster 2021-02-10T04:50:17.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[35961]: 2021-02-10T04:50:15.724810 2021-02-10T04:50:17.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2413 2021-02-10T04:50:17.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:17.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:50:17.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[35961]: -02-10T04:50:16.499477+0000 2021-02-10T04:50:17.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[35961]: osd.2 (osd.2) 1957 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:50:17.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:50:17.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[35961]: -10T04:50:16.499886+0000 osd.2 2021-02-10T04:50:17.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[35961]: (osd.2) 1958 : cluster [DBG] 2021-02-10T04:50:17.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:17 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:50:18.076 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:17 smithi164 conmon[33677]: debug 2021-02-10T04:50:17.799+0000 7fbec19a1700 -1 osd.4 1100 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:50:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:18 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:50:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:18 smithi164 conmon[29288]: :50:16.393631+0000 mgr.y (mgr.14140) 1162 : cluster [DBG] pgmap v1948: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 805 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:18 smithi164 conmon[29288]: cluster 2021-02-10T04:50 2021-02-10T04:50:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:18 smithi164 conmon[29288]: :16.758752+0000 osd.4 (osd.4) 2414 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:18 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:50:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:18 smithi164 conmon[29288]: 10T04:50:16.758763+0000 osd.4 (osd 2021-02-10T04:50:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:18 smithi164 conmon[29288]: .4) 2415 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:18.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:18 smithi164 conmon[29288]: cluster 2021-02-10T04:50:18.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:18 smithi164 conmon[29288]: 2021-02-10T04:50:18.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:18 smithi164 conmon[29288]: 2021 2021-02-10T04:50:18.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:18 smithi164 conmon[29288]: -02-10T04:50: 2021-02-10T04:50:18.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:18 smithi164 conmon[29288]: 16.758772+0000 osd.4 (osd.4) 2416 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:18.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:18 smithi164 conmon[29288]: cluster 2021-02-10T04:50:16.758778+0000 osd.4 (osd.4) 2021-02-10T04:50:18.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:18 smithi164 conmon[29288]: 2417 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:18.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:18 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:50:18.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:18 smithi164 conmon[29288]: 02-10T04:50:17.069816+0000 mon.a (mon.0) 2253 : 2021-02-10T04:50:18.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:18 smithi164 conmon[29288]: cluster [DBG] osdmap e1100: 8 total, 7 up, 7 in 2021-02-10T04:50:18.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:18 smithi085 conmon[29319]: cluster 2021-02-10T04:50:16.393631+0000 mgr.y (mgr.14140 2021-02-10T04:50:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:18 smithi085 conmon[29319]: ) 1162 : cluster [DBG] pgmap v1948: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 805 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:18 smithi085 conmon[29319]: cluster 2021-02-10T04:50:16.758752+0000 osd.4 (osd.4) 2414 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:18 smithi085 conmon[29319]: cluster 2021-02-10T04:50:16.758763+0000 osd.4 (osd.4) 2415 : cluster [WRN] 2021-02-10T04:50:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:18 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:18 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:50:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:18 smithi085 conmon[29319]: 02-10T04:50:16.758772+0000 osd.4 (osd.4) 2416 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:18 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:50:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:18 smithi085 conmon[29319]: :50:16.758778+0000 osd.4 (osd.4) 2417 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:18 smithi085 conmon[29319]: cluster 2021-02-10T04:50:17.069816+0000 mon.a (mon. 2021-02-10T04:50:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:18 smithi085 conmon[29319]: 0) 2253 : cluster [DBG] osdmap e1100: 8 total, 7 up, 7 in 2021-02-10T04:50:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:18 smithi085 conmon[35961]: cluster 2021-02-10T04:50:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:18 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:50:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:18 smithi085 conmon[35961]: :50:16.393631+0000 mgr.y (mgr.14140) 1162 : cluster [DBG] pgmap v1948: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 805 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:18 smithi085 conmon[35961]: 2021-02-10T04:50:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:18 smithi085 conmon[35961]: cluster 2021-02-10T04:50:16.758752+0000 osd.4 ( 2021-02-10T04:50:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:18 smithi085 conmon[35961]: osd.4) 2414 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:18 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:50:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:18 smithi085 conmon[35961]: 02-10T04:50:16.758763+0000 osd.4 2021-02-10T04:50:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:18 smithi085 conmon[35961]: (osd.4) 2415 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:18 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:50:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:18 smithi085 conmon[35961]: 02-10T04:50:16.758772+0000 osd.4 (osd.4) 2416 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:18 smithi085 conmon[35961]: cluster 2021-02-10T04:50:16.758778+0000 2021-02-10T04:50:18.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:18 smithi085 conmon[35961]: osd.4 (osd.4) 2417 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:18 smithi085 conmon[35961]: cluster 2021-02-10T04:50:17.069816+0000 mon.a 2021-02-10T04:50:18.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:18 smithi085 conmon[35961]: (mon.0) 2253 : cluster [DBG] osdmap e1100: 8 total, 7 up, 7 in 2021-02-10T04:50:19.009 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 4 1 2021-02-10T04:50:19.083 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:50:19 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:50:19.084 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:18 smithi164 conmon[33677]: debug 2021-02-10T04:50:19.084 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:18 smithi164 conmon[33677]: 2021-02-10T04:50:18.778+0000 7fbec19a1700 -1 osd.4 1100 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:50:19.084 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:18 smithi164 conmon[33677]: 2021-02-10T04:50:19.231 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:50:19.372 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:19 smithi085 conmon[35961]: cluster 2021-02-10T04:50:17.801569+0000 2021-02-10T04:50:19.372 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:19 smithi085 conmon[35961]: osd.4 (osd.4) 2418 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:19.372 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:19 smithi085 conmon[35961]: cluster 2021-02-10T04:50:17.801579+0000 osd.4 (osd.4) 2419 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:19.372 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:19 smithi085 conmon[35961]: cluster 2021-02-10T04:50:17.801585+0000 osd.4 (osd.4) 2420 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:19.372 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:19 smithi085 conmon[35961]: cluster 2021-02-10T04:50:17.801593+0000 osd.4 (osd.4) 2421 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:19.373 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:19 smithi085 conmon[29319]: cluster 2021-02-10T04:50:17.801569+0000 osd.4 (osd.4) 2418 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:19.373 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:19 smithi085 conmon[29319]: cluster 2021-02-10T04:50:17.801579+0000 osd.4 (osd.4) 2419 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:19.373 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:19 smithi085 conmon[29319]: cluster 2021-02-10T04:50:17.801585+0000 osd.4 (osd.4) 2420 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:19.373 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:19 smithi085 conmon[29319]: cluster 2021-02-10T04:50:17.801593+0000 osd.4 (osd.4) 2421 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:19.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:19 smithi164 conmon[29288]: cluster 2021-02-10T04:50:19.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:19 smithi164 conmon[29288]: 2021-02-10T04:50:19.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:19 smithi164 conmon[29288]: 2021 2021-02-10T04:50:19.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:19 smithi164 conmon[29288]: - 2021-02-10T04:50:19.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:19 smithi164 conmon[29288]: 02-10T04:50:17.801569 2021-02-10T04:50:19.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:19 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2418 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:19.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:19 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:50:19.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:19 smithi164 conmon[29288]: -02-10T04:50:17.801579+0000 osd.4 2021-02-10T04:50:19.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:19 smithi164 conmon[29288]: (osd.4) 2419 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:19.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:19 smithi164 conmon[29288]: cluster 2021-02-10T04:50:19.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:19 smithi164 conmon[29288]: 2021-02-10T04:50:19.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:19 smithi164 conmon[29288]: 2021 2021-02-10T04:50:19.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:19 smithi164 conmon[29288]: - 2021-02-10T04:50:19.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:19 smithi164 conmon[29288]: 02 2021-02-10T04:50:19.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:19 smithi164 conmon[29288]: - 2021-02-10T04:50:19.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:19 smithi164 conmon[29288]: 10T04:50:17.801585 2021-02-10T04:50:19.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:19 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2420 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:19.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:19 smithi164 conmon[29288]: cluster 2021-02-10T04:50:19.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:19 smithi164 conmon[29288]: 2021-02-10 2021-02-10T04:50:19.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:19 smithi164 conmon[29288]: T04:50:17.801593+0000 osd.4 (osd.4) 2421 : 2021-02-10T04:50:19.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:19 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:20.084 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:19 smithi164 conmon[33677]: debug 2021-02-10T04:50:19.737+0000 7fbec19a1700 -1 osd.4 1100 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:50:20.085 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:20 smithi164 conmon[29288]: cluster 2021-02-10T04:50:18.394435+0000 2021-02-10T04:50:20.085 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:20 smithi164 conmon[29288]: mgr.y (mgr.14140) 1163 : cluster [DBG] pgmap v1950: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:20.085 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:20 smithi164 conmon[29288]: cluster 2021-02-10T04:50:18.780593+0000 osd.4 (osd.4) 2422 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:20.085 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:20 smithi164 conmon[29288]: cluster 2021-02-10T04:50:20.085 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:20 smithi164 conmon[29288]: 2021-02 2021-02-10T04:50:20.085 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:20 smithi164 conmon[29288]: -10T04:50:18.780599+0000 osd.4 2021-02-10T04:50:20.085 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:20 smithi164 conmon[29288]: (osd.4) 2423 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:20.085 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:20 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:50:20.085 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:20 smithi164 conmon[29288]: 02-10T04:50:18.780603+0000 osd.4 (osd. 2021-02-10T04:50:20.085 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:20 smithi164 conmon[29288]: 4) 2424 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:20.085 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:20 smithi164 conmon[29288]: cluster 2021-02-10T04:50:20.085 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:20 smithi164 conmon[29288]: 2021 2021-02-10T04:50:20.085 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:20 smithi164 conmon[29288]: -02-10T04:50:18.780606+0000 2021-02-10T04:50:20.086 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:20 smithi164 conmon[29288]: osd.4 (osd.4) 2425 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:20.086 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:20 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:50:20.086 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:20 smithi164 conmon[29288]: -10T04:50:19.000047+0000 mgr.y (mgr. 2021-02-10T04:50:20.086 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:20 smithi164 conmon[29288]: 14140) 1164 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:50:20.086 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:20 smithi164 conmon[29288]: cluster 2021-02-10T04:50:20.086 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:20 smithi164 conmon[29288]: 2021-02-10T04:50:19.074770 2021-02-10T04:50:20.086 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:20 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2254 : cluster [WRN] 2021-02-10T04:50:20.086 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:20 smithi164 conmon[29288]: Health check update: 4 slow ops, oldest one blocked for 352 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:50:20.323 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[29319]: cluster 2021-02-10T04:50:18.394435+0000 mgr.y (mgr.14140) 2021-02-10T04:50:20.323 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[29319]: 1163 : cluster [DBG] pgmap v1950: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:20.323 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[29319]: cluster 2021-02-10T04:50:18.780593+0000 osd.4 (osd.4) 2422 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:20.323 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[29319]: cluster 2021-02-10T04:50:18.780599+0000 osd.4 (osd.4) 2423 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:20.323 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[29319]: cluster 2021-02-10T04:50:18.780603+0000 osd.4 (osd.4) 2424 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:20.323 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[29319]: cluster 2021-02-10T04:50:18.780606+0000 osd.4 (osd.4) 2425 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:20.323 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[29319]: audit 2021-02-10T04:50:19.000047+0000 mgr.y (mgr.14140) 1164 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:50:20.324 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[29319]: cluster 2021-02-10T04:50:19.074770+0000 mon.a (mon.0) 2254 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 352 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:50:20.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[35961]: cluster 2021-02-10T04:50:20.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[35961]: 2021-02-10T04:50:20.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[35961]: 2021 2021-02-10T04:50:20.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[35961]: - 2021-02-10T04:50:20.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[35961]: 02 2021-02-10T04:50:20.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[35961]: -10T04:50:18. 2021-02-10T04:50:20.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[35961]: 394435+0000 mgr.y (mgr.14140) 1163 : cluster [DBG] pgmap v1950: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:20.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:50:20.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[35961]: -10T04:50:18.780593+0000 osd.4 (osd. 2021-02-10T04:50:20.324 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[35961]: 4) 2422 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:20.325 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:50:20.325 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[35961]: :50:18.780599+0000 osd.4 (osd.4 2021-02-10T04:50:20.325 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[35961]: ) 2423 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:20.325 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[35961]: cluster 2021-02-10T04:50 2021-02-10T04:50:20.325 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[35961]: :18.780603+0000 osd.4 (osd.4) 2424 : 2021-02-10T04:50:20.325 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:20.325 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[35961]: cluster 2021-02-10T04:50: 2021-02-10T04:50:20.325 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[35961]: 18.780606+0000 osd.4 (osd.4) 2425 2021-02-10T04:50:20.325 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:20.325 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[35961]: audit 2021-02-10T04:50: 2021-02-10T04:50:20.325 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[35961]: 19.000047+0000 mgr.y (mgr.14140) 1164 : audit 2021-02-10T04:50:20.325 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[35961]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:50:20.325 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[35961]: cluster 2021-02-10T04:50:19. 2021-02-10T04:50:20.325 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[35961]: 074770+0000 mon.a (mon.0) 2254 : cluster [WRN] 2021-02-10T04:50:20.326 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:20 smithi085 conmon[35961]: Health check update: 4 slow ops, oldest one blocked for 352 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:50:20.530 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:50:20.550 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:50:20.575 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:50:20.577 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:50:20.577 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:50:20.598 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:50:20.618 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:50:20.641 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:50:20.665 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:50:20.665 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:50:20.665 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:50:20.665 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:50:20.665 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.3 2021-02-10T04:50:20.665 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: problem getting command descriptions from osd.3 2021-02-10T04:50:20.665 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:50:20.665 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:50:20.665 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:50:20.665 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:50:20.713 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set nodeep-scrub 2021-02-10T04:50:21.212 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:50:21.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:21 smithi164 conmon[29288]: cluster 2021-02-10T04:50:21.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:21 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:50:21.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:21 smithi164 conmon[29288]: :50:19.739456+0000 osd.4 (osd.4) 2426 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:21.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:21 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:50:21.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:21 smithi164 conmon[29288]: -02-10T04:50:19.739469+0000 osd.4 (osd 2021-02-10T04:50:21.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:21 smithi164 conmon[29288]: .4) 2427 : cluster [WRN] 2021-02-10T04:50:21.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:21 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:21.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:21 smithi164 conmon[29288]: cluster 2021-02-10T04:50:21.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:21 smithi164 conmon[29288]: 2021-02-10T04:50:19.739476+0000 2021-02-10T04:50:21.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:21 smithi164 conmon[29288]: osd.4 (osd.4) 2428 : 2021-02-10T04:50:21.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:21 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:21.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:21 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:50:21.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:21 smithi164 conmon[29288]: 02-10T04:50:19.739493+0000 2021-02-10T04:50:21.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:21 smithi164 conmon[29288]: osd.4 (osd.4) 2429 : 2021-02-10T04:50:21.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:21 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:21.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:21 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:50:21.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:21 smithi164 conmon[29288]: 02-10T04:50:20.504629+0000 2021-02-10T04:50:21.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:21 smithi164 conmon[29288]: mon.a (mon.0) 2255 : 2021-02-10T04:50:21.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:21 smithi164 conmon[29288]: audit [DBG] from='client.? 172.21.15.85:0/248766158' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:50:21.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:21 smithi085 conmon[29319]: cluster 2021-02-10T04:50:19.739456+0000 osd.4 (osd.4) 2426 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:21.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:21 smithi085 conmon[29319]: cluster 2021-02-10T04:50:19.739469+0000 osd.4 (osd.4) 2427 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:21.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:21 smithi085 conmon[29319]: cluster 2021-02-10T04:50:19.739476+0000 osd.4 (osd.4) 2428 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:21.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:21 smithi085 conmon[29319]: cluster 2021-02-10T04:50:19.739493+0000 osd.4 (osd.4) 2429 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:21.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:21 smithi085 conmon[29319]: audit 2021-02-10T04:50:20.504629+0000 mon.a (mon.0) 2255 : audit [DBG] from='client.? 172.21.15.85:0/248766158' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:50:21.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:21 smithi085 conmon[35961]: cluster 2021-02-10T04:50:19.739456+0000 osd.4 (osd.4) 2426 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:21.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:21 smithi085 conmon[35961]: cluster 2021-02-10T04:50:19.739469+0000 osd.4 (osd.4) 2427 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:21.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:21 smithi085 conmon[35961]: cluster 2021-02-10T04:50:19.739476+0000 osd.4 (osd.4) 2428 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:21.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:21 smithi085 conmon[35961]: cluster 2021-02-10T04:50:19.739493+0000 osd.4 (osd.4) 2429 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:21.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:21 smithi085 conmon[35961]: audit 2021-02-10T04:50:20 2021-02-10T04:50:21.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:21 smithi085 conmon[35961]: .504629+0000 mon.a (mon.0) 2255 : audit [DBG] from='client.? 172.21.15.85:0/248766158' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:50:22.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:22 smithi164 conmon[29288]: cluster 2021-02-10T04:50:20.395138+0000 mgr.y (mgr.14140) 1165 2021-02-10T04:50:22.727 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:22 smithi164 conmon[29288]: : cluster [DBG] pgmap v1951: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:22.728 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:22 smithi085 conmon[29319]: cluster 2021-02-10T04:50:20.395138+0000 mgr.y (mgr.14140) 1165 : cluster 2021-02-10T04:50:22.728 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:22 smithi085 conmon[29319]: [DBG] pgmap v1951: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:22.729 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:22 smithi085 conmon[35961]: cluster 2021-02-10T04:50:22.729 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:22 smithi085 conmon[35961]: 2021-02-10T04:50 2021-02-10T04:50:22.729 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:22 smithi085 conmon[35961]: :20.395138+0000 mgr.y (mgr.14140) 1165 : cluster [DBG] pgmap v1951: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:23.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:23 smithi085 conmon[29319]: cluster 2021-02-10T04:50:23.093456+0000 mon.a (mon. 2021-02-10T04:50:23.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:23 smithi085 conmon[29319]: 0) 2256 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 352 sec, osd.4 has slow ops) 2021-02-10T04:50:23.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:23 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:50:23.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:23 smithi085 conmon[35961]: 50:23.093456+0000 mon.a (mon.0) 2256 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 352 sec, osd.4 has slow ops) 2021-02-10T04:50:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:23 smithi164 conmon[29288]: cluster 2021-02-10T04:50:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:23 smithi164 conmon[29288]: 2021-02-10T04:50:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:23 smithi164 conmon[29288]: 2021 2021-02-10T04:50:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:23 smithi164 conmon[29288]: -02-10T04:50:23. 2021-02-10T04:50:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:23 smithi164 conmon[29288]: 093456+0000 mon.a (mon.0) 2256 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 352 sec, osd.4 has slow ops) 2021-02-10T04:50:24.176 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.1 dump_blocked_ops 2021-02-10T04:50:24.299 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:50:24.299 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:50:24.433 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:50:24.438 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:24 smithi085 conmon[29319]: cluster 2021-02-10T04:50:22.395912+0000 mgr.y ( 2021-02-10T04:50:24.438 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:24 smithi085 conmon[29319]: mgr.14140) 1166 : cluster [DBG] pgmap v1952: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 697 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:24.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:24 smithi085 conmon[35961]: cluster 2021-02-10T04:50:22.395912 2021-02-10T04:50:24.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:24 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1166 : cluster [DBG] pgmap v1952: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 697 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:24.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:24 smithi164 conmon[29288]: cluster 2021-02-10T04:50:24.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:24 smithi164 conmon[29288]: 2021-02-10T04:50:24.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:24 smithi164 conmon[29288]: 2021-02-10T04:50:22 2021-02-10T04:50:24.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:24 smithi164 conmon[29288]: .395912+0000 mgr.y (mgr.14140) 1166 : cluster [DBG] pgmap v1952: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 697 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:26.300 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:50:26.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:26 smithi085 conmon[29319]: cluster 2021-02-10T04:50:24.396904 2021-02-10T04:50:26.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:26 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 1167 : cluster [DBG] pgmap v1953: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:26 smithi085 conmon[35961]: cluster 2021-02-10T04:50:24.396904 2021-02-10T04:50:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:26 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1167 : cluster [DBG] pgmap v1953: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:26.553 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:50:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:26 smithi164 conmon[29288]: cluster 2021-02-10T04:50:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:26 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:50:26.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:26 smithi164 conmon[29288]: 04:50:24.396904+0000 mgr.y (mgr.14140) 1167 : cluster [DBG] pgmap v1953: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:27.941 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:27 smithi085 systemd[1]: ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.3.service: Failed with result 'exit-code'. 2021-02-10T04:50:28.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:28 smithi085 conmon[35961]: cluster 2021-02-10T04:50: 2021-02-10T04:50:28.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:28 smithi085 conmon[35961]: 26.397612+0000 mgr.y (mgr.14140) 1168 : cluster [DBG] pgmap v1954: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:28.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:28 smithi085 conmon[35961]: cluster 2021-02-10T04:50:27.252591+0000 osd.6 (osd.6) 143 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:50:28.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:28 smithi085 conmon[35961]: cluster 2021-02-10T04:50:27.255567+0000 osd.6 (osd.6) 144 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:50:28.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:28 smithi085 conmon[29319]: cluster 2021-02-10T04:50:26.397612+0000 mgr.y (mgr.14140) 1168 : cluster [DBG] pgmap v1954: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:28.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:28 smithi085 conmon[29319]: cluster 2021-02-10T04:50:27.252591+0000 osd.6 (osd.6) 143 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:50:28.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:28 smithi085 conmon[29319]: cluster 2021-02-10T04:50:27.255567+0000 osd.6 (osd.6) 144 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:50:28.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:28 smithi164 conmon[29288]: cluster 2021-02-10T04:50:26.397612+0000 mgr.y 2021-02-10T04:50:28.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:28 smithi164 conmon[29288]: (mgr.14140) 1168 : cluster [DBG] pgmap v1954: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:28.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:28 smithi164 conmon[29288]: cluster 2021-02-10T04:50:27.252591+0000 osd.6 (osd.6) 143 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:50:28.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:28 smithi164 conmon[29288]: cluster 2021-02-10T04:50:27.255567+0000 osd.6 (osd.6) 144 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:50:29.169 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.4 to 1 (10000) 2021-02-10T04:50:29.394 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:50:29 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:50:29.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:29 smithi164 conmon[29288]: cluster 2021-02-10T04:50:28.158377 2021-02-10T04:50:29.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:29 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2257 : cluster [DBG] osdmap e1101: 8 total, 7 up, 7 in 2021-02-10T04:50:29.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:29 smithi164 conmon[29288]: audit 2021-02-10T04:50:28.244652+0000 mon.c (mon.1) 83 : audit [INF] 2021-02-10T04:50:29.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:29 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/670313092' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-02-10T04:50:29.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:29 smithi164 conmon[29288]: audit 2021-02-10T04:50:28.244975 2021-02-10T04:50:29.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:29 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2258 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-02-10T04:50:29.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[29319]: cluster 2021-02-10T04:50:29.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[29319]: 2021-02-10T04:50:29.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[29319]: 2021 2021-02-10T04:50:29.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[29319]: -02-10T04:50:28.158377 2021-02-10T04:50:29.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2257 : cluster [DBG] osdmap e1101: 8 total, 7 up, 7 in 2021-02-10T04:50:29.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[29319]: audit 2021-02-10T04:50:29.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[29319]: 2021-02-10T04:50:29.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[29319]: 2021 2021-02-10T04:50:29.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[29319]: - 2021-02-10T04:50:29.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[29319]: 02 2021-02-10T04:50:29.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[29319]: - 2021-02-10T04:50:29.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[29319]: 10 2021-02-10T04:50:29.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[29319]: T 2021-02-10T04:50:29.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[29319]: 04 2021-02-10T04:50:29.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[29319]: : 2021-02-10T04:50:29.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[29319]: 50:28.244652+0000 mon.c ( 2021-02-10T04:50:29.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[29319]: mon.1) 83 : audit [INF] from='client.? 172.21.15.85:0/670313092' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-02-10T04:50:29.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[29319]: audit 2021-02-10T04:50 2021-02-10T04:50:29.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[29319]: :28.244975+0000 mon.a (mon.0) 2258 : 2021-02-10T04:50:29.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[29319]: audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-02-10T04:50:29.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[35961]: cluster 2021-02-10T04:50:29.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[35961]: 2021-02-10T04:50:29.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[35961]: 2021 2021-02-10T04:50:29.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[35961]: - 2021-02-10T04:50:29.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[35961]: 02 2021-02-10T04:50:29.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[35961]: - 2021-02-10T04:50:29.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[35961]: 10 2021-02-10T04:50:29.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[35961]: T04:50:28.158377+0000 mon.a (mon 2021-02-10T04:50:29.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[35961]: .0) 2257 : cluster [DBG] osdmap e1101: 8 total, 7 up, 7 in 2021-02-10T04:50:29.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[35961]: audit 2021-02-10T04:50:29.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[35961]: 2021-02-10T04:50:28.244652+0000 mon.c (mon 2021-02-10T04:50:29.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[35961]: .1) 83 : audit [INF] from='client.? 172.21.15.85:0/670313092' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-02-10T04:50:29.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[35961]: audit 2021 2021-02-10T04:50:29.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[35961]: -02-10T04:50:28.244975 2021-02-10T04:50:29.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2258 : audit [INF] 2021-02-10T04:50:29.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:29 smithi085 conmon[35961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-02-10T04:50:30.179 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is set 2021-02-10T04:50:30.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[29319]: cluster 2021-02-10T04:50: 2021-02-10T04:50:30.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[29319]: 28.398299+0000 mgr.y (mgr.14140) 1169 : cluster [DBG] pgmap v1956: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:30.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[29319]: audit 2021-02-10T04:50:29.009983+0000 mgr.y (mgr.14140) 1170 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:50:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[29319]: audit 2021-02-10T04:50:29.164536+0000 mon.a (mon.0) 2259 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-02-10T04:50:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[29319]: cluster 2021-02-10T04:50:29.164619+0000 mon.a (mon.0) 2260 : cluster [DBG] osdmap e1102: 8 total, 7 up, 7 in 2021-02-10T04:50:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:50:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[29319]: T04:50:29.693797+0000 mon.c (mon.1) 84 : audit [INF] from='client.? 172.21.15.85:0/623114102' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:50:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:50:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[29319]: :50:29.694145+0000 mon.a (mon.0) 2261 : audit 2021-02-10T04:50:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[29319]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:50:30.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[35961]: cluster 2021-02-10T04:50:30.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[35961]: 2021 2021-02-10T04:50:30.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[35961]: -02-10T04:50:28.398299+0000 mgr.y (mgr 2021-02-10T04:50:30.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[35961]: .14140) 1169 : cluster [DBG] pgmap v1956: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:50:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[35961]: 02-10T04:50:29.009983+0000 mgr.y ( 2021-02-10T04:50:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[35961]: mgr.14140) 1170 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:50:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[35961]: 2021-02-10T04:50:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:50:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[35961]: 02-10T04:50:29.164536+0000 mon.a ( 2021-02-10T04:50:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[35961]: mon.0) 2259 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-02-10T04:50:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[35961]: 2021-02-10T04:50:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:50:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[35961]: -02-10T04:50:29.164619+0000 2021-02-10T04:50:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[35961]: mon.a (mon.0) 2260 : cluster [DBG] osdmap e1102: 8 total, 7 up, 7 in 2021-02-10T04:50:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[35961]: 2021-02-10T04:50:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[35961]: audit 2021-02-10T04:50:30.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[35961]: 2021-02-10T04:50:29.693797+0000 mon.c ( 2021-02-10T04:50:30.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[35961]: mon.1) 84 : audit [INF] from='client.? 172.21.15.85:0/623114102' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:50:30.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:50:30.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[35961]: 02-10T04:50:29.694145+0000 mon.a 2021-02-10T04:50:30.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:30 smithi085 conmon[35961]: (mon.0) 2261 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:50:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:30 smithi164 conmon[29288]: cluster 2021-02-10T04:50:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:30 smithi164 conmon[29288]: 2021-02-10T04:50:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:30 smithi164 conmon[29288]: 2021-02-10T04:50: 2021-02-10T04:50:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:30 smithi164 conmon[29288]: 28.398299+0000 mgr.y (mgr.14140) 1169 : cluster [DBG] pgmap v1956: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:30 smithi164 conmon[29288]: audit 2021 2021-02-10T04:50:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:30 smithi164 conmon[29288]: -02-10T04:50:29 2021-02-10T04:50:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:30 smithi164 conmon[29288]: .009983+0000 mgr.y (mgr.14140) 1170 : 2021-02-10T04:50:30.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:30 smithi164 conmon[29288]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:50:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:30 smithi164 conmon[29288]: audit 2021-02-10T04:50:29.164536+0000 mon.a (mon.0) 2259 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-02-10T04:50:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:30 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:50:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:30 smithi164 conmon[29288]: 10T04:50:29.164619+0000 mon.a (mon.0) 2021-02-10T04:50:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:30 smithi164 conmon[29288]: 2260 : cluster [DBG] osdmap e1102: 8 total, 7 up, 7 in 2021-02-10T04:50:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:30 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:50:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:30 smithi164 conmon[29288]: 02-10T04:50:29. 2021-02-10T04:50:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:30 smithi164 conmon[29288]: 693797+0000 mon.c (mon.1) 84 : audit 2021-02-10T04:50:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:30 smithi164 conmon[29288]: [INF] from='client.? 172.21.15.85:0/623114102' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:50:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:30 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:50:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:30 smithi164 conmon[29288]: 10T04:50:29.694145+0000 2021-02-10T04:50:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:30 smithi164 conmon[29288]: mon.a (mon.0) 2261 : audit [INF] 2021-02-10T04:50:30.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:30 smithi164 conmon[29288]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:50:30.713 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 5 1 2021-02-10T04:50:30.992 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:50:31.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:31 smithi085 conmon[29319]: cluster 2021-02-10T04:50:30.162134+0000 mon.a (mon.0) 2262 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:50:31.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:31 smithi085 conmon[29319]: audit 2021-02-10T04:50:30.173491+0000 mon.a (mon.0) 2263 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:50:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:31 smithi085 conmon[29319]: cluster 2021-02-10T04:50:30.173543+0000 mon.a (mon.0) 2264 : cluster [DBG] osdmap e1103: 8 total, 7 up, 7 in 2021-02-10T04:50:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:31 smithi085 conmon[35961]: cluster 2021-02-10T04:50:30.162134+0000 mon.a (mon.0) 2262 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:50:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:31 smithi085 conmon[35961]: audit 2021-02-10T04:50:30.173491+0000 mon.a (mon.0) 2263 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:50:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:31 smithi085 conmon[35961]: cluster 2021-02-10T04:50:30.173543+0000 mon.a (mon.0) 2264 : cluster [DBG] osdmap e1103: 8 total, 7 up, 7 in 2021-02-10T04:50:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:31 smithi164 conmon[29288]: cluster 2021-02-10T04:50:30.162134+0000 mon.a 2021-02-10T04:50:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:31 smithi164 conmon[29288]: (mon.0) 2262 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:50:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:31 smithi164 conmon[29288]: audit 2021-02-10T04:50:30.173491+0000 mon.a (mon.0) 2263 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:50:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:31 smithi164 conmon[29288]: cluster 2021-02-10T04:50:30.173543+0000 mon.a (mon.0) 2021-02-10T04:50:31.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:31 smithi164 conmon[29288]: 2264 : cluster [DBG] osdmap e1103: 8 total, 7 up, 7 in 2021-02-10T04:50:32.174 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:32 smithi085 conmon[35961]: cluster 2021-02-10T04:50:30.398813+0000 2021-02-10T04:50:32.174 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:32 smithi085 conmon[35961]: mgr.y (mgr.14140) 1171 : cluster [DBG] pgmap v1959: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 852 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:32.268 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:50:32.268 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:50:32.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:32 smithi085 conmon[29319]: cluster 2021-02-10T04:50:30.398813+0000 mgr.y (mgr.14140) 1171 : 2021-02-10T04:50:32.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:32 smithi085 conmon[29319]: cluster [DBG] pgmap v1959: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 852 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:32 smithi164 conmon[29288]: cluster 2021-02-10T04:50:30.398813+0000 mgr.y (mgr 2021-02-10T04:50:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:32 smithi164 conmon[29288]: .14140) 1171 : cluster [DBG] pgmap v1959: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 852 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:34.035 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.1 dump_historic_ops 2021-02-10T04:50:34.314 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:34 smithi085 conmon[35961]: cluster 2021-02-10T04:50:34.314 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:34 smithi085 conmon[35961]: 2021-02-10T04:50:34.314 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:34 smithi085 conmon[35961]: 2021 2021-02-10T04:50:34.314 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:34 smithi085 conmon[35961]: - 2021-02-10T04:50:34.314 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:34 smithi085 conmon[35961]: 02 2021-02-10T04:50:34.315 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:34 smithi085 conmon[35961]: - 2021-02-10T04:50:34.315 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:34 smithi085 conmon[35961]: 10 2021-02-10T04:50:34.315 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:34 smithi085 conmon[35961]: T 2021-02-10T04:50:34.315 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:34 smithi085 conmon[35961]: 04 2021-02-10T04:50:34.315 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:34 smithi085 conmon[35961]: : 2021-02-10T04:50:34.315 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:34 smithi085 conmon[35961]: 50 2021-02-10T04:50:34.315 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:34 smithi085 conmon[35961]: : 2021-02-10T04:50:34.315 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:34 smithi085 conmon[35961]: 32 2021-02-10T04:50:34.315 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:34 smithi085 conmon[35961]: . 2021-02-10T04:50:34.315 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:34 smithi085 conmon[35961]: 399388 2021-02-10T04:50:34.315 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:34 smithi085 conmon[35961]: +0000 2021-02-10T04:50:34.315 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:34 smithi085 conmon[35961]: 2021-02-10T04:50:34.315 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:34 smithi085 conmon[35961]: mgr.y 2021-02-10T04:50:34.316 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:34 smithi085 conmon[35961]: ( 2021-02-10T04:50:34.316 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:34 smithi085 conmon[35961]: mgr 2021-02-10T04:50:34.316 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:34 smithi085 conmon[35961]: .14140) 1172 : cluster [DBG] pgmap v1960: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:34.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:34 smithi085 conmon[29319]: cluster 2021-02-10T04:50:32.399388+0000 mgr.y (mgr 2021-02-10T04:50:34.316 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:34 smithi085 conmon[29319]: .14140) 1172 : cluster [DBG] pgmap v1960: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:34.336 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:50:34.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:34 smithi164 conmon[29288]: cluster 2021-02-10T04:50:32.399388+0000 mgr.y (mgr 2021-02-10T04:50:34.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:34 smithi164 conmon[29288]: .14140) 1172 : cluster [DBG] pgmap v1960: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:35.644 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset noscrub 2021-02-10T04:50:35.806 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:50:35.826 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:50:35.851 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:50:35.879 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:50:35.893 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:50:35.903 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:50:35.932 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:50:35.955 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:50:35.956 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:50:35.956 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:50:35.956 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:50:35.956 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.3 2021-02-10T04:50:35.956 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: problem getting command descriptions from osd.3 2021-02-10T04:50:35.956 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:50:35.956 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:50:35.956 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:50:35.956 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:50:36.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:36 smithi085 conmon[29319]: cluster 2021-02-10T04:50:34.400252+0000 mgr.y (mgr.14140) 1173 : cluster [DBG] pgmap v1961: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.6 KiB/s rd, 1 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:36.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:36 smithi085 conmon[29319]: audit 2021-02-10T04:50:35.783565+0000 mon.a (mon.0) 2265 : audit [DBG] from='client.? 172.21.15.85:0/3335483791' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:50:36.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:36 smithi085 conmon[35961]: cluster 2021-02-10T04:50:34.400252+0000 mgr.y (mgr.14140) 1173 : cluster [DBG] pgmap v1961: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.6 KiB/s rd, 1 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:36.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:36 smithi085 conmon[35961]: audit 2021-02-10T04:50:35.783565+0000 mon.a (mon.0) 2265 : audit [DBG] from='client.? 172.21.15.85:0/3335483791' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:50:36.604 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:36 smithi164 conmon[33677]: debug 2021-02-10T04:50:36.605 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:36 smithi164 conmon[33677]: 2021-02-10T04:50:36.603+0000 7fbec19a1700 -1 osd.4 1103 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:50:36.605 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:36 smithi164 conmon[33677]: 2021-02-10T04:50:36.605 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:36 smithi164 conmon[29288]: cluster 2021-02-10T04:50:34.400252 2021-02-10T04:50:36.605 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:36 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1173 : cluster [DBG] pgmap v1961: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.6 KiB/s rd, 1 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:36.605 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:36 smithi164 conmon[29288]: audit 2021-02-10T04:50:35.783565+0000 mon.a (mon.0) 2265 : audit [DBG] 2021-02-10T04:50:36.605 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:36 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/3335483791' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:50:37.812 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:37 smithi085 systemd[1]: ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.3.service: Service RestartSec=10s expired, scheduling restart. 2021-02-10T04:50:37.813 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:37 smithi085 systemd[1]: ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.3.service: Scheduled restart job, restart counter is at 1. 2021-02-10T04:50:37.813 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:37 smithi085 systemd[1]: Stopped Ceph osd.3 for 42224fce-6b57-11eb-8feb-001a4aab830c. 2021-02-10T04:50:37.813 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:37 smithi085 systemd[1]: Starting Ceph osd.3 for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T04:50:37.813 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:50:37.813 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:50:37.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:37 smithi164 conmon[33677]: debug 2021-02-10T04:50:37.609+0000 7fbec19a1700 -1 osd.4 1103 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:50:38.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:38 smithi085 conmon[29319]: cluster 2021-02-10T04:50:38.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:38 smithi085 conmon[29319]: 2021-02-10T04:50:36 2021-02-10T04:50:38.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:38 smithi085 conmon[29319]: .400855+0000 mgr.y (mgr.14140) 1174 : cluster [DBG] pgmap v1962: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:38.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:38 smithi085 conmon[29319]: cluster 2021-02-10T04:50 2021-02-10T04:50:38.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:38 smithi085 conmon[29319]: :36.603892+0000 osd.4 (osd.4) 2430 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:38.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:38 smithi085 conmon[29319]: 2021-02-10T04:50:38.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:38 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:50:38.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:38 smithi085 conmon[29319]: -02-10T04:50:36.603900+0000 osd.4 (osd.4) 2431 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:38.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:38 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:50:38.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:38 smithi085 conmon[29319]: -10T04:50:36.603909+0000 osd.4 (osd.4) 2432 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:38.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:38 smithi085 conmon[29319]: cluster 2021-02-10T04:50 2021-02-10T04:50:38.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:38 smithi085 conmon[29319]: :36.603917+0000 osd.4 (osd.4) 2433 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:38.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:38 smithi085 conmon[29319]: 2021-02-10T04:50:38.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:38 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:50:38.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:38 smithi085 conmon[29319]: 04:50:37.392240+0000 osd.0 (osd.0) 153 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:50:38.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:38 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:50:38.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:38 smithi085 conmon[29319]: :50:37.394721+0000 osd.0 (osd.0) 154 : cluster 2021-02-10T04:50:38.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:38 smithi085 conmon[29319]: [DBG] purged_snaps scrub ok 2021-02-10T04:50:38.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:38 smithi085 conmon[35961]: cluster 2021-02-10T04:50:36.400855+0000 2021-02-10T04:50:38.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:38 smithi085 conmon[35961]: mgr.y (mgr.14140) 1174 : cluster [DBG] pgmap v1962: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:38.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:38 smithi085 conmon[35961]: cluster 2021-02-10T04:50:36.603892+0000 osd.4 (osd.4) 2430 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:38.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:38 smithi085 conmon[35961]: cluster 2021-02-10T04:50:36.603900+0000 osd.4 (osd.4) 2431 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:38.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:38 smithi085 conmon[35961]: cluster 2021-02-10T04:50:36.603909+0000 osd.4 (osd.4) 2432 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:38.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:38 smithi085 conmon[35961]: cluster 2021-02-10T04:50:36.603917+0000 osd.4 (osd.4) 2433 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:38.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:38 smithi085 conmon[35961]: cluster 2021-02-10T04:50:37.392240+0000 osd.0 (osd.0) 153 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:50:38.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:38 smithi085 conmon[35961]: cluster 2021-02-10T04:50:37.394721+0000 osd.0 (osd.0) 154 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:50:38.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:38 smithi164 conmon[29288]: cluster 2021-02-10T04:50:38.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:38 smithi164 conmon[29288]: 2021-02-10T04:50:38.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:38 smithi164 conmon[29288]: 2021-02-10T04:50:36.400855+0000 mgr.y (mgr.14140) 1174 : cluster [DBG] pgmap v1962: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:38.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:38 smithi164 conmon[29288]: 2021-02-10T04:50:38.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:38 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:50:38.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:38 smithi164 conmon[29288]: :50:36.603892+0000 osd.4 (osd.4) 2430 : 2021-02-10T04:50:38.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:38 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:38.627 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:38 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:50:38.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:38 smithi164 conmon[29288]: 50:36.603900+0000 osd.4 (osd.4) 2021-02-10T04:50:38.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:38 smithi164 conmon[29288]: 2431 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:38.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:38 smithi164 conmon[29288]: cluster 2021-02-10T04:50:36 2021-02-10T04:50:38.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:38 smithi164 conmon[29288]: .603909+0000 osd.4 (osd.4) 2432 : cluster [WRN] 2021-02-10T04:50:38.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:38 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:38.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:38 smithi164 conmon[29288]: cluster 2021-02-10T04:50:38.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:38 smithi164 conmon[29288]: 2021-02-10T04:50:36.603917 2021-02-10T04:50:38.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:38 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2433 : cluster [WRN] 2021-02-10T04:50:38.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:38 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:38.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:38 smithi164 conmon[29288]: cluster 2021-02-10T04:50:37.392240+0000 2021-02-10T04:50:38.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:38 smithi164 conmon[29288]: osd.0 (osd.0) 153 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:50:38.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:38 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:50:38.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:38 smithi164 conmon[29288]: -10T04 2021-02-10T04:50:38.628 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:38 smithi164 conmon[29288]: :50:37.394721+0000 osd.0 (osd.0) 2021-02-10T04:50:38.629 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:38 smithi164 conmon[29288]: 154 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:50:38.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:38 smithi164 conmon[33677]: debug 2021-02-10T04:50:38.625+0000 7fbec19a1700 -1 osd.4 1104 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:50:39.394 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:50:39 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:50:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:39 smithi164 conmon[29288]: cluster 2021-02-10T04:50:37.610023+0000 osd.4 (osd.4) 2434 2021-02-10T04:50:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:39 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:39 smithi164 conmon[29288]: cluster 2021-02-10T04:50:37.610032+0000 osd.4 (osd.4) 2435 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:39 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:50:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:39 smithi164 conmon[29288]: 10T04:50:37.610038+0000 osd.4 (osd.4) 2436 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:39 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:50:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:39 smithi164 conmon[29288]: :50:37.610044+0000 osd.4 (osd.4) 2437 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:39 smithi164 conmon[29288]: cluster 2021-02-10T04:50 2021-02-10T04:50:39.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:39 smithi164 conmon[29288]: :38.189186+0000 mon.a (mon.0) 2266 : cluster 2021-02-10T04:50:39.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:39 smithi164 conmon[29288]: [DBG] osdmap e1104: 8 total, 7 up, 7 in 2021-02-10T04:50:39.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:39 smithi164 conmon[29288]: audit 2021 2021-02-10T04:50:39.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:39 smithi164 conmon[29288]: -02-10T04:50:38.204206+0000 mon.a (mon.0) 2267 : audit 2021-02-10T04:50:39.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:39 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:50:39.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:39 smithi164 conmon[29288]: audit 2021-02-10T04:50: 2021-02-10T04:50:39.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:39 smithi164 conmon[29288]: 38.204384+0000 mon.a (mon.0) 2268 : audit 2021-02-10T04:50:39.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:39 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:50:39.659 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[29319]: cluster 2021-02-10T04:50:37.610023+0000 osd.4 (osd. 2021-02-10T04:50:39.659 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[29319]: 4) 2434 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:39.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[29319]: cluster 2021-02-10T04:50:37.610032+0000 osd.4 ( 2021-02-10T04:50:39.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[29319]: osd.4) 2435 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:39.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:50:39.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[29319]: T04:50:37.610038+0000 osd.4 (osd. 2021-02-10T04:50:39.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[29319]: 4) 2436 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:39.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[29319]: 2021-02-10T04:50:39.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:50:39.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[29319]: -02-10T04:50:37.610044+0000 osd.4 (osd.4 2021-02-10T04:50:39.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[29319]: ) 2437 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:39.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:50:39.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[29319]: T04:50:38.189186+0000 mon.a (mon 2021-02-10T04:50:39.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[29319]: .0) 2266 : cluster [DBG] osdmap e1104: 8 total, 7 up, 7 in 2021-02-10T04:50:39.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[29319]: audit 2021-02-10T04:50:38.204206+0000 mon.a (mon.0) 2021-02-10T04:50:39.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[29319]: 2267 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:50:39.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:50:39.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[29319]: :50:38.204384+0000 mon.a (mon. 2021-02-10T04:50:39.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[29319]: 0) 2268 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:50:39.661 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[35961]: cluster 2021-02-10T04:50:39.661 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[35961]: 2021-02-10T04:50:39.661 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[35961]: 2021 2021-02-10T04:50:39.661 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[35961]: -02-10T04:50:37 2021-02-10T04:50:39.661 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[35961]: .610023+0000 osd.4 (osd.4) 2434 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:39.661 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[35961]: cluster 2021-02-10T04:50: 2021-02-10T04:50:39.661 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[35961]: 37.610032+0000 osd.4 (osd.4) 2435 : cluster 2021-02-10T04:50:39.662 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:39.662 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[35961]: cluster 2021-02-10T04:50:37.610038 2021-02-10T04:50:39.662 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2436 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:39.662 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[35961]: 2021-02-10T04:50:39.662 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[35961]: cluster 2021-02-10T04:50:37. 2021-02-10T04:50:39.662 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[35961]: 610044+0000 osd.4 (osd.4) 2437 : 2021-02-10T04:50:39.662 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:39.662 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:50:39.662 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[35961]: -10T04:50:38.189186+0000 2021-02-10T04:50:39.662 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[35961]: mon.a (mon.0) 2266 : cluster [DBG] osdmap e1104: 8 total, 7 up, 7 in 2021-02-10T04:50:39.662 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:50:39.662 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[35961]: 10T04:50:38.204206+0000 mon.a (mon. 2021-02-10T04:50:39.663 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[35961]: 0) 2267 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:50:39.663 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:50:39.663 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[35961]: T04:50:38.204384+0000 mon.a (mon.0) 2268 2021-02-10T04:50:39.663 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:39 smithi085 conmon[35961]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:50:39.815 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:50:39.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:39 smithi164 conmon[33677]: debug 2021-02-10T04:50:39.624+0000 7fbec19a1700 -1 osd.4 1104 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:50:40.128 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:50:40.202 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.5 to 1 (10000) 2021-02-10T04:50:40.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:40 smithi085 conmon[29319]: cluster 2021-02-10T04:50:38.401472+0000 mgr.y (mgr.14140) 2021-02-10T04:50:40.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:40 smithi085 conmon[29319]: 1175 : cluster [DBG] pgmap v1964: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 622 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:40.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:40 smithi085 conmon[29319]: cluster 2021-02-10T04:50:38.626107+0000 osd.4 (osd.4) 2438 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:40.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:40 smithi085 conmon[29319]: cluster 2021-02-10T04:50:38.626117+0000 osd.4 (osd.4) 2439 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:40.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:40 smithi085 conmon[29319]: cluster 2021-02-10T04:50:38.626122+0000 osd.4 (osd.4) 2440 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:40.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:40 smithi085 conmon[29319]: cluster 2021-02-10T04:50:38.626125+0000 osd.4 (osd.4) 2441 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:40.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:40 smithi085 conmon[29319]: audit 2021-02-10T04:50:39.020019+0000 mgr.y (mgr.14140) 1176 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:50:40.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:40 smithi085 conmon[29319]: cluster 2021-02-10T04:50:39.187047+0000 mon.a (mon.0) 2269 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 372 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:50:40.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:40 smithi085 conmon[29319]: audit 2021-02-10T04:50:39.617387+0000 mon.c (mon.1) 85 : audit [INF] from='client.? 172.21.15.85:0/3360884430' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-02-10T04:50:40.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:40 smithi085 conmon[29319]: audit 2021-02-10T04:50:39.617734+0000 mon.a (mon.0) 2270 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-02-10T04:50:40.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:40 smithi085 conmon[35961]: cluster 2021-02-10T04:50:38.401472+0000 mgr.y (mgr.14140 2021-02-10T04:50:40.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:40 smithi085 conmon[35961]: ) 1175 : cluster [DBG] pgmap v1964: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 622 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:40.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:40 smithi085 conmon[35961]: cluster 2021-02-10T04:50:38.626107+0000 osd.4 (osd.4) 2438 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:40.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:40 smithi085 conmon[35961]: cluster 2021-02-10T04:50:38.626117+0000 osd.4 (osd.4) 2439 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:40.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:40 smithi085 conmon[35961]: cluster 2021-02-10T04:50:38.626122+0000 osd.4 (osd.4) 2440 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:40.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:40 smithi085 conmon[35961]: cluster 2021-02-10T04:50:38.626125+0000 osd.4 (osd.4) 2441 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:40.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:40 smithi085 conmon[35961]: audit 2021-02-10T04:50:39.020019+0000 mgr.y (mgr.14140) 1176 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:50:40.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:40 smithi085 conmon[35961]: cluster 2021-02-10T04:50:39.187047+0000 mon.a (mon.0) 2269 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 372 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:50:40.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:40 smithi085 conmon[35961]: audit 2021-02-10T04:50:39.617387+0000 mon.c (mon.1) 85 : audit [INF] from='client.? 172.21.15.85:0/3360884430' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-02-10T04:50:40.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:40 smithi085 conmon[35961]: audit 2021-02-10T04:50:39.617734+0000 mon.a (mon.0) 2270 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-02-10T04:50:40.614 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:40 smithi164 conmon[29288]: cluster 2021-02-10T04:50:38.401472+0000 mgr.y (mgr.14140) 1175 : cluster 2021-02-10T04:50:40.614 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:40 smithi164 conmon[29288]: [DBG] pgmap v1964: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 622 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:40.614 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:40 smithi164 conmon[29288]: cluster 2021-02-10T04:50:38.626107+0000 osd.4 (osd.4) 2438 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:40.614 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:40 smithi164 conmon[29288]: cluster 2021-02-10T04:50:38.626117+0000 2021-02-10T04:50:40.614 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:40 smithi164 conmon[29288]: osd.4 (osd.4) 2439 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:40.614 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:40 smithi164 conmon[29288]: cluster 2021-02-10T04:50: 2021-02-10T04:50:40.614 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:40 smithi164 conmon[29288]: 38.626122+0000 osd.4 (osd.4) 2440 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:40.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:40 smithi164 conmon[29288]: cluster 2021-02-10T04:50 2021-02-10T04:50:40.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:40 smithi164 conmon[29288]: :38.626125+0000 osd.4 (osd.4) 2441 : cluster [WRN] 2021-02-10T04:50:40.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:40 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:40.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:40 smithi164 conmon[29288]: audit 2021-02-10T04:50:40.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:40 smithi164 conmon[29288]: 2021-02-10T04:50:39.020019+0000 mgr.y (mgr.14140) 1176 : 2021-02-10T04:50:40.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:40 smithi164 conmon[29288]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:50:40.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:40 smithi164 conmon[29288]: cluster 2021-02-10T04:50:40.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:40 smithi164 conmon[29288]: 2021-02-10T04:50:39.187047+0000 mon.a (mon.0) 2021-02-10T04:50:40.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:40 smithi164 conmon[29288]: 2269 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 372 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:50:40.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:40 smithi164 conmon[29288]: audit 2021-02-10T04:50:39 2021-02-10T04:50:40.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:40 smithi164 conmon[29288]: .617387+0000 mon.c (mon.1) 85 : audit [INF] 2021-02-10T04:50:40.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:40 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/3360884430' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-02-10T04:50:40.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:40 smithi164 conmon[29288]: audit 2021-02-10T04:50:40.615 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:40 smithi164 conmon[29288]: 2021-02-10T04:50:39.617734+0000 mon.a (mon.0) 2021-02-10T04:50:40.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:40 smithi164 conmon[29288]: 2270 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-02-10T04:50:40.616 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:40 smithi164 conmon[33677]: debug 2021-02-10T04:50:40.616 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:40 smithi164 conmon[33677]: 2021-02-10T04:50:40.612+0000 7fbec19a1700 -1 osd.4 1105 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:50:40.616 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:40 smithi164 conmon[33677]: 2021-02-10T04:50:41.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:41 smithi164 conmon[29288]: cluster 2021-02-10T04:50:39.625265+0000 osd.4 (osd.4) 2021-02-10T04:50:41.616 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:41 smithi164 conmon[29288]: 2442 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:41.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:41 smithi164 conmon[29288]: cluster 2021-02-10T04:50:39.625274+0000 osd.4 (osd.4) 2443 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:41.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:41 smithi164 conmon[29288]: cluster 2021-02-10T04:50:39.625278+0000 osd.4 (osd.4) 2021-02-10T04:50:41.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:41 smithi164 conmon[29288]: 2444 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:41.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:41 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:50:41.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:41 smithi164 conmon[29288]: T04:50:39.625281+0000 2021-02-10T04:50:41.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:41 smithi164 conmon[29288]: osd.4 (osd.4) 2445 : cluster 2021-02-10T04:50:41.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:41 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:41.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:41 smithi164 conmon[29288]: audit 2021-02-10T04:50:41.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:41 smithi164 conmon[29288]: 2021-02-10T04:50: 2021-02-10T04:50:41.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:41 smithi164 conmon[29288]: 40.197108+0000 mon.a (mon 2021-02-10T04:50:41.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:41 smithi164 conmon[29288]: .0) 2271 : audit [INF] 2021-02-10T04:50:41.618 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:41 smithi164 conmon[29288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-02-10T04:50:41.618 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:41 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:50:41.618 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:41 smithi164 conmon[29288]: 10T04:50:40.197164 2021-02-10T04:50:41.618 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:41 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2272 2021-02-10T04:50:41.618 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:41 smithi164 conmon[29288]: : cluster [DBG] osdmap e1105: 8 total, 7 up, 7 in 2021-02-10T04:50:41.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[29319]: cluster 2021-02-10T04:50:39.625265+0000 osd.4 (osd. 2021-02-10T04:50:41.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[29319]: 4) 2442 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:41.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[29319]: cluster 2021-02-10T04:50:39.625274+0000 osd.4 (osd.4) 2443 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:41.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[29319]: cluster 2021-02-10T04:50:39.625278+0000 osd.4 (osd.4) 2444 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:41.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[29319]: cluster 2021-02-10T04:50:39.625281+0000 osd.4 (osd 2021-02-10T04:50:41.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[29319]: .4) 2445 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:41.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[29319]: audit 2021-02-10T04:50:41.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[29319]: 2021-02-10T04:50: 2021-02-10T04:50:41.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[29319]: 40.197108+0000 mon.a (mon.0) 2271 2021-02-10T04:50:41.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[29319]: : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-02-10T04:50:41.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[29319]: cluster 2021-02-10T04:50:41.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[29319]: 2021-02-10T04:50:40 2021-02-10T04:50:41.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[29319]: .197164+0000 mon.a (mon.0) 2272 : 2021-02-10T04:50:41.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[29319]: cluster [DBG] osdmap e1105: 8 total, 7 up, 7 in 2021-02-10T04:50:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[35961]: cluster 2021-02-10T04:50:39.625265+0000 osd.4 (osd.4 2021-02-10T04:50:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[35961]: ) 2442 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[35961]: cluster 2021-02-10T04:50:39.625274+0000 osd.4 (osd.4) 2443 : cluster [WRN] 2021-02-10T04:50:41.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:41.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[35961]: cluster 2021-02-10T04:50: 2021-02-10T04:50:41.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[35961]: 39.625278+0000 osd.4 (osd.4 2021-02-10T04:50:41.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[35961]: ) 2444 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:41.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:50:41.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[35961]: -02-10T04 2021-02-10T04:50:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[35961]: :50:39.625281+0000 osd.4 (osd.4 2021-02-10T04:50:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[35961]: ) 2445 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:50:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[35961]: 10T04:50:40.197108+0000 mon.a (mon 2021-02-10T04:50:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[35961]: .0) 2271 : audit 2021-02-10T04:50:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[35961]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-02-10T04:50:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[35961]: cluster 2021-02-10T04:50:40 2021-02-10T04:50:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[35961]: .197164+0000 mon.a (mon.0 2021-02-10T04:50:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[35961]: ) 2272 : cluster [DBG] osdmap e1105: 8 total, 7 up, 7 in 2021-02-10T04:50:41.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:41 smithi085 conmon[35961]: 2021-02-10T04:50:41.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:41 smithi164 conmon[33677]: debug 2021-02-10T04:50:41.615+0000 7fbec19a1700 -1 osd.4 1105 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:50:42.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:42 smithi085 conmon[29319]: cluster 2021-02-10T04:50:40.401915+0000 mgr.y (mgr. 2021-02-10T04:50:42.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:42 smithi085 conmon[29319]: 14140) 1177 : cluster [DBG] pgmap v1966: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:42.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:42 smithi085 conmon[29319]: cluster 2021-02-10T04:50:40.613341+0000 osd.4 (osd.4) 2446 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:42.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:42 smithi085 conmon[29319]: cluster 2021-02-10T04:50:40.613351+0000 osd.4 (osd.4) 2447 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:42.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:42 smithi085 conmon[29319]: cluster 2021-02-10T04:50:40.613359+0000 osd.4 (osd.4) 2448 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:42.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:42 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:50:42.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:42 smithi085 conmon[29319]: T04:50:40.613364+0000 osd.4 (osd.4) 2449 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:42.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:42 smithi085 conmon[35961]: cluster 2021-02-10T04:50:40 2021-02-10T04:50:42.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:42 smithi085 conmon[35961]: .401915+0000 mgr.y (mgr.14140) 1177 : cluster [DBG] pgmap v1966: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:42.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:42 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:50:42.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:42 smithi085 conmon[35961]: -02-10T04:50:40.613341+0000 osd.4 (osd.4) 2446 : cluster 2021-02-10T04:50:42.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:42 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:42.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:42 smithi085 conmon[35961]: cluster 2021-02-10T04:50:40.613351+0000 osd.4 (osd.4 2021-02-10T04:50:42.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:42 smithi085 conmon[35961]: ) 2447 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:42.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:42 smithi085 conmon[35961]: cluster 2021-02-10T04:50:42.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:42 smithi085 conmon[35961]: 2021-02-10T04:50:40.613359+0000 osd.4 (osd.4) 2448 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:42.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:42 smithi085 conmon[35961]: cluster 2021-02-10T04:50:40.613364+0000 2021-02-10T04:50:42.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:42 smithi085 conmon[35961]: osd.4 (osd.4) 2449 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:42.604 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:42 smithi164 conmon[33677]: debug 2021-02-10T04:50:42.603+0000 7fbec19a1700 -1 osd.4 1105 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:50:42.605 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:42 smithi164 conmon[29288]: cluster 2021-02-10T04:50:40.401915+0000 mgr.y (mgr.14140) 1177 : cluster 2021-02-10T04:50:42.605 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:42 smithi164 conmon[29288]: [DBG] pgmap v1966: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:42.605 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:42 smithi164 conmon[29288]: cluster 2021-02-10T04:50:40.613341+0000 osd.4 (osd.4) 2446 2021-02-10T04:50:42.605 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:42 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:42.605 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:42 smithi164 conmon[29288]: cluster 2021-02-10T04:50:42.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:42 smithi164 conmon[29288]: 2021-02-10T04:50:40.613351+0000 osd.4 (osd.4) 2447 : 2021-02-10T04:50:42.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:42 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:42.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:42 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:50:42.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:42 smithi164 conmon[29288]: 04:50:40.613359+0000 osd.4 (osd.4) 2021-02-10T04:50:42.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:42 smithi164 conmon[29288]: 2448 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:42.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:42 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:50:42.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:42 smithi164 conmon[29288]: 10T04:50:40.613364+0000 osd.4 (osd.4) 2449 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:42.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:42 smithi164 conmon[29288]: 2021-02-10T04:50:42.690 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 6 1 2021-02-10T04:50:42.953 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:50:43.195 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:50:43.195 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 20, 2021-02-10T04:50:43.195 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 600, 2021-02-10T04:50:43.196 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [ 2021-02-10T04:50:43.196 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.196 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(client.14817.0:8302 7.b e834/367 7:d39fd0bb:::smithi08575397-9:head v 834'1285, mlcod=833'1282)", 2021-02-10T04:50:43.196 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:40:53.314623+0000", 2021-02-10T04:50:43.196 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 589.87695997100002, 2021-02-10T04:50:43.196 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.063025197000000005, 2021-02-10T04:50:43.196 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:43.196 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:50:43.196 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:43.196 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.196 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:43.196 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.314623+0000", 2021-02-10T04:50:43.197 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:43.197 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.197 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.197 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:43.197 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.314623+0000", 2021-02-10T04:50:43.197 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00048649199999999999 2021-02-10T04:50:43.197 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.197 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.197 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:43.197 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.315110+0000", 2021-02-10T04:50:43.197 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1936999999999999e-05 2021-02-10T04:50:43.197 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.197 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.197 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:43.198 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.315122+0000", 2021-02-10T04:50:43.198 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.519e-05 2021-02-10T04:50:43.198 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.198 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.198 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:43.198 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.315137+0000", 2021-02-10T04:50:43.198 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.6339999999999996e-06 2021-02-10T04:50:43.198 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.198 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.198 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:43.198 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.315147+0000", 2021-02-10T04:50:43.198 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.7534999999999997e-05 2021-02-10T04:50:43.198 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.199 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.199 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:43.199 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.315204+0000", 2021-02-10T04:50:43.199 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.8254000000000001e-05 2021-02-10T04:50:43.199 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.199 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.199 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:50:43.199 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.315232+0000", 2021-02-10T04:50:43.199 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.062346727999999997 2021-02-10T04:50:43.199 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.199 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.199 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:50:43.199 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.377579+0000", 2021-02-10T04:50:43.199 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.9426999999999998e-05 2021-02-10T04:50:43.200 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.200 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.200 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:43.200 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.377649+0000", 2021-02-10T04:50:43.200 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.063025197000000005 2021-02-10T04:50:43.200 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.200 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:43.200 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.200 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.200 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.200 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(client.14817.0:8303 7.b e834/367 7:d39fd0bb:::smithi08575397-9:head v 834'1286, mlcod=833'1282)", 2021-02-10T04:50:43.200 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:40:53.322501+0000", 2021-02-10T04:50:43.200 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 589.86908230400002, 2021-02-10T04:50:43.201 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.057640922999999997, 2021-02-10T04:50:43.201 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:43.201 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:50:43.201 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:43.201 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.201 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:43.201 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.322501+0000", 2021-02-10T04:50:43.201 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:43.201 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.201 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.201 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:43.201 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.322501+0000", 2021-02-10T04:50:43.201 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000396993 2021-02-10T04:50:43.201 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.202 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.202 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:43.202 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.322898+0000", 2021-02-10T04:50:43.202 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1368e-05 2021-02-10T04:50:43.202 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.202 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.202 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:43.202 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.322909+0000", 2021-02-10T04:50:43.202 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.7588000000000002e-05 2021-02-10T04:50:43.202 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.202 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.202 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:43.202 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.322927+0000", 2021-02-10T04:50:43.203 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1759e-05 2021-02-10T04:50:43.203 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.203 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.203 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:43.203 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.322939+0000", 2021-02-10T04:50:43.203 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.5894000000000001e-05 2021-02-10T04:50:43.203 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.203 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.203 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:43.203 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.323025+0000", 2021-02-10T04:50:43.203 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.2357999999999997e-05 2021-02-10T04:50:43.203 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.203 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.203 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:50:43.204 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.323057+0000", 2021-02-10T04:50:43.204 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.056999723000000002 2021-02-10T04:50:43.204 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.204 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.204 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:50:43.204 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.380057+0000", 2021-02-10T04:50:43.204 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.5240000000000004e-05 2021-02-10T04:50:43.204 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.204 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.204 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:43.204 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.380142+0000", 2021-02-10T04:50:43.204 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.057640922999999997 2021-02-10T04:50:43.204 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.204 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:43.205 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.205 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.205 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.205 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(client.14817.0:8304 7.b e834/367 7:d39fd0bb:::smithi08575397-9:head v 834'1287, mlcod=833'1282)", 2021-02-10T04:50:43.205 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:40:53.327417+0000", 2021-02-10T04:50:43.205 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 589.86416630300005, 2021-02-10T04:50:43.205 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.052763271, 2021-02-10T04:50:43.205 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:43.205 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:50:43.205 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:43.205 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.205 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:43.205 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.327417+0000", 2021-02-10T04:50:43.205 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:43.206 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.206 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.206 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:43.206 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.327417+0000", 2021-02-10T04:50:43.206 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000121675 2021-02-10T04:50:43.206 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.206 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.206 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:43.206 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.327539+0000", 2021-02-10T04:50:43.206 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.1749999999999999e-06 2021-02-10T04:50:43.206 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.206 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.207 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:43.207 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.327546+0000", 2021-02-10T04:50:43.207 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.3279999999999995e-06 2021-02-10T04:50:43.207 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.207 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.207 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:43.207 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.327555+0000", 2021-02-10T04:50:43.207 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.6000000000000001e-06 2021-02-10T04:50:43.207 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.207 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.207 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:43.208 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.327563+0000", 2021-02-10T04:50:43.208 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000100009 2021-02-10T04:50:43.208 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.208 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.208 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:43.208 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.327663+0000", 2021-02-10T04:50:43.208 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.6676999999999999e-05 2021-02-10T04:50:43.208 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.208 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.208 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:50:43.208 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.327689+0000", 2021-02-10T04:50:43.208 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.052456869000000003 2021-02-10T04:50:43.208 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.209 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.209 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:50:43.209 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.380146+0000", 2021-02-10T04:50:43.209 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.3937999999999999e-05 2021-02-10T04:50:43.209 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.209 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.209 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:43.209 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.380180+0000", 2021-02-10T04:50:43.209 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.052763271 2021-02-10T04:50:43.209 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.209 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:43.209 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.209 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.210 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.210 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(client.14817.0:8305 7.b e834/367 7:d39fd0bb:::smithi08575397-9:head v 834'1288, mlcod=833'1282)", 2021-02-10T04:50:43.210 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:40:53.328710+0000", 2021-02-10T04:50:43.210 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 589.86287293999999, 2021-02-10T04:50:43.210 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.051500733999999999, 2021-02-10T04:50:43.210 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:43.210 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:50:43.210 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:43.210 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.210 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:43.210 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.328710+0000", 2021-02-10T04:50:43.210 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:43.210 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.211 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.211 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:43.211 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.328710+0000", 2021-02-10T04:50:43.211 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4453999999999998e-05 2021-02-10T04:50:43.211 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.211 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.211 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:43.211 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.328755+0000", 2021-02-10T04:50:43.211 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.3810000000000004e-06 2021-02-10T04:50:43.211 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.211 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.211 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:43.211 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.328761+0000", 2021-02-10T04:50:43.212 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.2520000000000002e-06 2021-02-10T04:50:43.212 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.212 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.212 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:43.212 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.328768+0000", 2021-02-10T04:50:43.212 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0471e-05 2021-02-10T04:50:43.212 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.212 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.212 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:43.212 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.328779+0000", 2021-02-10T04:50:43.212 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.2276999999999998e-05 2021-02-10T04:50:43.212 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.212 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.212 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:43.213 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.328861+0000", 2021-02-10T04:50:43.213 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2559e-05 2021-02-10T04:50:43.213 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.213 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.213 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:50:43.213 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.328874+0000", 2021-02-10T04:50:43.213 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.051308193000000002 2021-02-10T04:50:43.213 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.213 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.213 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:50:43.213 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.380182+0000", 2021-02-10T04:50:43.213 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.9147000000000001e-05 2021-02-10T04:50:43.213 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.214 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.214 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:43.214 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.380211+0000", 2021-02-10T04:50:43.214 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.051500733999999999 2021-02-10T04:50:43.214 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.214 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:43.214 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.214 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.214 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.214 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(client.14817.0:8307 7.b e834/367 7:d58f4058:::smithi08575397-42:head v 834'1290, mlcod=833'1282)", 2021-02-10T04:50:43.214 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:40:53.334687+0000", 2021-02-10T04:50:43.214 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 589.85689672299998, 2021-02-10T04:50:43.215 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.045555432, 2021-02-10T04:50:43.215 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:43.215 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:50:43.215 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:43.215 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.215 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:43.215 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.334687+0000", 2021-02-10T04:50:43.215 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:43.215 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.216 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.216 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:43.216 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.334687+0000", 2021-02-10T04:50:43.216 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00033472299999999999 2021-02-10T04:50:43.216 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.216 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.216 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:43.216 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.335021+0000", 2021-02-10T04:50:43.216 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0244999999999999e-05 2021-02-10T04:50:43.216 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.216 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.217 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:43.217 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.335032+0000", 2021-02-10T04:50:43.217 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.6330000000000001e-05 2021-02-10T04:50:43.217 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.217 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.217 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:43.217 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.335048+0000", 2021-02-10T04:50:43.217 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.0410000000000003e-06 2021-02-10T04:50:43.217 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.217 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.217 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:43.217 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.335057+0000", 2021-02-10T04:50:43.217 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0460000000000001e-05 2021-02-10T04:50:43.217 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.218 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.218 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:43.218 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.335107+0000", 2021-02-10T04:50:43.218 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.2478000000000001e-05 2021-02-10T04:50:43.218 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.218 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.218 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:50:43.218 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.335130+0000", 2021-02-10T04:50:43.218 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.045082983 2021-02-10T04:50:43.218 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.218 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.218 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:50:43.218 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.380213+0000", 2021-02-10T04:50:43.219 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.9172000000000001e-05 2021-02-10T04:50:43.219 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.219 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.219 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:43.219 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.380242+0000", 2021-02-10T04:50:43.219 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.045555432 2021-02-10T04:50:43.219 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.219 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:43.219 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.219 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.219 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.219 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(client.14817.0:8308 7.b e834/367 7:d58f4058:::smithi08575397-42:head v 834'1291, mlcod=833'1282)", 2021-02-10T04:50:43.219 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:40:53.343550+0000", 2021-02-10T04:50:43.219 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 589.84803312999998, 2021-02-10T04:50:43.220 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.036724358999999998, 2021-02-10T04:50:43.220 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:43.220 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:50:43.220 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:43.220 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.220 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:43.220 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.343550+0000", 2021-02-10T04:50:43.220 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:43.220 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.220 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.220 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:43.220 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.343550+0000", 2021-02-10T04:50:43.220 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00059002499999999999 2021-02-10T04:50:43.220 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.221 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.221 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:43.221 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.344140+0000", 2021-02-10T04:50:43.221 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0667e-05 2021-02-10T04:50:43.221 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.221 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.221 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:43.221 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.344151+0000", 2021-02-10T04:50:43.221 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.7839999999999999e-05 2021-02-10T04:50:43.221 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.221 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.221 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:43.221 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.344169+0000", 2021-02-10T04:50:43.221 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.077e-05 2021-02-10T04:50:43.222 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.222 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.222 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:43.222 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.344179+0000", 2021-02-10T04:50:43.222 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.9289999999999998e-05 2021-02-10T04:50:43.222 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.222 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.222 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:43.222 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.344219+0000", 2021-02-10T04:50:43.222 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.0673e-05 2021-02-10T04:50:43.222 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.222 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.222 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:50:43.223 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.344239+0000", 2021-02-10T04:50:43.223 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.036005257999999998 2021-02-10T04:50:43.223 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.223 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.223 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:50:43.223 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.380245+0000", 2021-02-10T04:50:43.223 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.9836e-05 2021-02-10T04:50:43.223 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.223 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.223 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:43.223 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.380275+0000", 2021-02-10T04:50:43.223 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.036724358999999998 2021-02-10T04:50:43.223 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.223 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:43.224 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.224 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.224 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.224 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(client.14817.0:8309 7.b e834/367 7:d58f4058:::smithi08575397-42:head v 834'1292, mlcod=833'1282)", 2021-02-10T04:50:43.224 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:40:53.346148+0000", 2021-02-10T04:50:43.224 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 589.84543481399999, 2021-02-10T04:50:43.224 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.034158948000000001, 2021-02-10T04:50:43.224 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:43.224 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:50:43.224 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:43.224 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.224 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:43.224 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.346148+0000", 2021-02-10T04:50:43.224 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:43.225 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.225 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.225 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:43.225 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.346148+0000", 2021-02-10T04:50:43.225 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000240881 2021-02-10T04:50:43.225 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.225 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.225 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:43.225 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.346389+0000", 2021-02-10T04:50:43.225 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.3119999999999994e-06 2021-02-10T04:50:43.225 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.225 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.225 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:43.226 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.346398+0000", 2021-02-10T04:50:43.226 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.4073000000000001e-05 2021-02-10T04:50:43.226 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.226 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.226 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:43.226 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.346412+0000", 2021-02-10T04:50:43.226 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.9070000000000004e-06 2021-02-10T04:50:43.226 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.226 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.226 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:43.226 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.346418+0000", 2021-02-10T04:50:43.226 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4362e-05 2021-02-10T04:50:43.226 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.227 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.227 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:43.227 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.346462+0000", 2021-02-10T04:50:43.227 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.6574999999999999e-05 2021-02-10T04:50:43.227 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.227 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.227 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:50:43.227 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.346489+0000", 2021-02-10T04:50:43.227 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.033787728000000003 2021-02-10T04:50:43.227 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.227 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.227 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:50:43.227 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.380276+0000", 2021-02-10T04:50:43.227 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.1109999999999999e-05 2021-02-10T04:50:43.228 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.228 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.228 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:43.228 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:53.380307+0000", 2021-02-10T04:50:43.228 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.034158948000000001 2021-02-10T04:50:43.228 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.228 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:43.228 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.228 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.228 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.228 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(client.14817.0:8318 7.a e835/649 7:531b0702:::smithi08575397-3:head v 835'1247, mlcod=831'1243)", 2021-02-10T04:50:43.228 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:40:54.312824+0000", 2021-02-10T04:50:43.228 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 588.87875884799996, 2021-02-10T04:50:43.229 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.062836660000000003, 2021-02-10T04:50:43.229 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:43.229 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:50:43.229 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:43.229 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.229 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:43.229 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:54.312824+0000", 2021-02-10T04:50:43.229 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:43.229 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.229 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.229 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:43.229 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:54.312824+0000", 2021-02-10T04:50:43.229 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00024957400000000002 2021-02-10T04:50:43.230 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.230 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.230 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:43.230 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:54.313074+0000", 2021-02-10T04:50:43.230 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.5179999999999998e-06 2021-02-10T04:50:43.230 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.230 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.230 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:43.230 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:54.313082+0000", 2021-02-10T04:50:43.230 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0461e-05 2021-02-10T04:50:43.231 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.231 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.231 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:43.231 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:54.313092+0000", 2021-02-10T04:50:43.231 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.9230000000000002e-06 2021-02-10T04:50:43.231 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.231 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.231 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:43.231 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:54.313100+0000", 2021-02-10T04:50:43.231 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0118341 2021-02-10T04:50:43.231 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.231 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.231 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:43.232 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:54.324934+0000", 2021-02-10T04:50:43.232 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.2119000000000002e-05 2021-02-10T04:50:43.232 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.232 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.232 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:50:43.232 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:54.324966+0000", 2021-02-10T04:50:43.232 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.050618436000000003 2021-02-10T04:50:43.232 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.232 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.232 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:50:43.232 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:54.375585+0000", 2021-02-10T04:50:43.232 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.6528999999999999e-05 2021-02-10T04:50:43.232 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.232 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.233 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:43.233 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:40:54.375661+0000", 2021-02-10T04:50:43.233 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.062836660000000003 2021-02-10T04:50:43.233 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.233 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:43.233 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.233 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.233 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.233 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(client.14817.0:8914 7.9 e889/93 7:93db45b2:::smithi08575397-44:head v 889'614, mlcod=887'611)", 2021-02-10T04:50:43.233 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:41:48.329333+0000", 2021-02-10T04:50:43.233 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 534.86225036400003, 2021-02-10T04:50:43.233 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.033602931000000003, 2021-02-10T04:50:43.233 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:43.233 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:50:43.234 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:43.234 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.234 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:43.234 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.329333+0000", 2021-02-10T04:50:43.234 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:43.234 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.234 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.234 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:43.234 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.329333+0000", 2021-02-10T04:50:43.234 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00043033699999999998 2021-02-10T04:50:43.234 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.234 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.234 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:43.235 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.329763+0000", 2021-02-10T04:50:43.235 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2962000000000001e-05 2021-02-10T04:50:43.235 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.235 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.235 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:43.235 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.329776+0000", 2021-02-10T04:50:43.235 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.4295e-05 2021-02-10T04:50:43.235 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.235 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.235 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:43.235 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.329791+0000", 2021-02-10T04:50:43.235 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.4949999999999997e-06 2021-02-10T04:50:43.235 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.235 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.236 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:43.236 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.329800+0000", 2021-02-10T04:50:43.236 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00013405200000000001 2021-02-10T04:50:43.236 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.236 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.236 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:43.236 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.329934+0000", 2021-02-10T04:50:43.236 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.5145000000000001e-05 2021-02-10T04:50:43.236 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.236 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.236 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:50:43.236 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.329959+0000", 2021-02-10T04:50:43.236 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.032893328999999999 2021-02-10T04:50:43.236 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.237 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.237 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:50:43.237 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.362853+0000", 2021-02-10T04:50:43.237 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.3316e-05 2021-02-10T04:50:43.237 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.237 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.237 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:43.237 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.362936+0000", 2021-02-10T04:50:43.237 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.033602931000000003 2021-02-10T04:50:43.237 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.237 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:43.237 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.237 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.237 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.238 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(client.14817.0:8917 7.b e889/367 7:db4ad6c0:::smithi08575397-12:head v 889'1420, mlcod=886'1416)", 2021-02-10T04:50:43.238 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:41:48.337365+0000", 2021-02-10T04:50:43.238 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 534.85421869899994, 2021-02-10T04:50:43.238 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.046911662, 2021-02-10T04:50:43.238 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:43.238 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:50:43.238 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:43.238 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.238 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:43.238 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.337365+0000", 2021-02-10T04:50:43.238 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:43.238 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.238 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.239 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:43.239 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.337365+0000", 2021-02-10T04:50:43.239 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00053747699999999996 2021-02-10T04:50:43.239 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.239 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.239 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:43.239 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.337902+0000", 2021-02-10T04:50:43.239 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.5529999999999998e-06 2021-02-10T04:50:43.239 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.239 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.239 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:43.239 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.337912+0000", 2021-02-10T04:50:43.239 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.5463999999999998e-05 2021-02-10T04:50:43.239 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.240 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.240 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:43.240 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.337947+0000", 2021-02-10T04:50:43.240 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0145e-05 2021-02-10T04:50:43.240 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.240 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.240 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:43.240 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.337957+0000", 2021-02-10T04:50:43.240 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.4820000000000005e-05 2021-02-10T04:50:43.240 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.240 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.240 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:43.240 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.338032+0000", 2021-02-10T04:50:43.241 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.1603999999999998e-05 2021-02-10T04:50:43.241 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.241 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.241 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:50:43.241 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.338064+0000", 2021-02-10T04:50:43.241 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.046139105 2021-02-10T04:50:43.241 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.241 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.241 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:50:43.241 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.384203+0000", 2021-02-10T04:50:43.241 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.3493999999999998e-05 2021-02-10T04:50:43.241 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.241 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.242 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:43.242 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.384276+0000", 2021-02-10T04:50:43.242 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.046911662 2021-02-10T04:50:43.242 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.242 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:43.242 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.242 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.242 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.242 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(client.14817.0:8918 7.b e889/367 7:db4ad6c0:::smithi08575397-12:head v 889'1421, mlcod=886'1416)", 2021-02-10T04:50:43.242 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:41:48.344067+0000", 2021-02-10T04:50:43.242 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 534.847516563, 2021-02-10T04:50:43.242 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.043465726000000003, 2021-02-10T04:50:43.242 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:43.243 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:50:43.243 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:43.243 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.243 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:43.243 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.344067+0000", 2021-02-10T04:50:43.243 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:43.243 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.243 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.243 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:43.243 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.344067+0000", 2021-02-10T04:50:43.243 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000350527 2021-02-10T04:50:43.243 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.243 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.243 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:43.244 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.344417+0000", 2021-02-10T04:50:43.244 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.5670000000000001e-06 2021-02-10T04:50:43.244 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.244 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.244 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:43.244 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.344427+0000", 2021-02-10T04:50:43.244 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.6121e-05 2021-02-10T04:50:43.244 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.244 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.244 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:43.244 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.344443+0000", 2021-02-10T04:50:43.244 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.8960000000000003e-06 2021-02-10T04:50:43.244 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.244 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.245 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:43.245 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.344451+0000", 2021-02-10T04:50:43.245 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00016986799999999999 2021-02-10T04:50:43.245 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.245 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.245 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:43.245 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.344621+0000", 2021-02-10T04:50:43.245 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.9335e-05 2021-02-10T04:50:43.245 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.245 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.245 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:50:43.245 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.344650+0000", 2021-02-10T04:50:43.245 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.042803167000000003 2021-02-10T04:50:43.245 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.246 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.246 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:50:43.246 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.387453+0000", 2021-02-10T04:50:43.246 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.9245000000000004e-05 2021-02-10T04:50:43.246 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.246 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.246 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:43.246 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.387532+0000", 2021-02-10T04:50:43.246 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.043465726000000003 2021-02-10T04:50:43.246 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.246 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:43.246 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.246 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.247 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.247 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(client.14817.0:8919 7.b e889/367 7:db4ad6c0:::smithi08575397-12:head v 889'1422, mlcod=886'1416)", 2021-02-10T04:50:43.247 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:41:48.345754+0000", 2021-02-10T04:50:43.247 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 534.84582939699999, 2021-02-10T04:50:43.247 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.041928624999999997, 2021-02-10T04:50:43.247 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:43.247 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:50:43.247 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:43.247 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.247 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:43.247 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.345754+0000", 2021-02-10T04:50:43.247 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:43.247 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.247 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.248 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:43.248 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.345754+0000", 2021-02-10T04:50:43.248 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.5686999999999998e-05 2021-02-10T04:50:43.248 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.248 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.248 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:43.248 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.345800+0000", 2021-02-10T04:50:43.248 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.1189999999999997e-06 2021-02-10T04:50:43.248 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.248 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.248 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:43.248 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.345808+0000", 2021-02-10T04:50:43.248 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1678999999999999e-05 2021-02-10T04:50:43.249 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.249 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.249 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:43.249 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.345819+0000", 2021-02-10T04:50:43.249 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.6919999999999999e-06 2021-02-10T04:50:43.249 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.249 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.249 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:43.249 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.345825+0000", 2021-02-10T04:50:43.249 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00011239399999999999 2021-02-10T04:50:43.249 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.249 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.249 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:43.249 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.345937+0000", 2021-02-10T04:50:43.250 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.1392000000000001e-05 2021-02-10T04:50:43.250 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.250 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.250 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:50:43.250 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.345959+0000", 2021-02-10T04:50:43.250 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.041578454000000001 2021-02-10T04:50:43.250 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.250 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.250 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:50:43.250 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.387537+0000", 2021-02-10T04:50:43.250 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000145208 2021-02-10T04:50:43.250 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.250 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.251 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:43.251 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:41:48.387683+0000", 2021-02-10T04:50:43.251 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.041928624999999997 2021-02-10T04:50:43.251 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.251 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:43.251 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.251 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.251 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.251 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(client.14817.0:9992 7.0 e988/93 7:06aed406:::smithi08575397-5:head v 988'1051, mlcod=987'1047)", 2021-02-10T04:50:43.251 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:43:28.373155+0000", 2021-02-10T04:50:43.251 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 434.81842852800003, 2021-02-10T04:50:43.251 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.048602003999999997, 2021-02-10T04:50:43.251 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:43.252 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:50:43.252 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:43.252 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.252 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:43.252 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:43:28.373155+0000", 2021-02-10T04:50:43.252 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:43.252 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.252 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.252 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:43.252 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:43:28.373155+0000", 2021-02-10T04:50:43.252 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00060382100000000005 2021-02-10T04:50:43.252 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.252 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.252 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:43.253 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:43:28.373759+0000", 2021-02-10T04:50:43.253 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.0079999999999996e-06 2021-02-10T04:50:43.253 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.253 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.253 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:43.253 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:43:28.373766+0000", 2021-02-10T04:50:43.253 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.807e-06 2021-02-10T04:50:43.253 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.253 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.253 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:43.253 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:43:28.373773+0000", 2021-02-10T04:50:43.253 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.3300000000000004e-06 2021-02-10T04:50:43.253 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.253 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.254 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:43.254 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:43:28.373780+0000", 2021-02-10T04:50:43.254 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.4379000000000001e-05 2021-02-10T04:50:43.254 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.254 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.254 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:43.254 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:43:28.373844+0000", 2021-02-10T04:50:43.254 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.552e-05 2021-02-10T04:50:43.254 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.254 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.254 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:50:43.254 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:43:28.373870+0000", 2021-02-10T04:50:43.254 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.047817531000000003 2021-02-10T04:50:43.255 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.255 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.255 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:50:43.255 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:43:28.421687+0000", 2021-02-10T04:50:43.255 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.9608000000000002e-05 2021-02-10T04:50:43.255 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.255 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.255 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:43.255 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:43:28.421757+0000", 2021-02-10T04:50:43.255 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.048602003999999997 2021-02-10T04:50:43.255 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.255 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:43.255 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.255 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.256 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.256 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(client.14817.0:9996 7.a e988/649 7:52017b30:::smithi08575397-43:head v 988'1498, mlcod=987'1494)", 2021-02-10T04:50:43.256 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:43:28.382245+0000", 2021-02-10T04:50:43.256 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 434.80933819400002, 2021-02-10T04:50:43.256 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.041573013999999998, 2021-02-10T04:50:43.256 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:43.256 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:50:43.256 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:43.256 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.256 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:43.256 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:43:28.382245+0000", 2021-02-10T04:50:43.256 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:43.256 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.256 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.257 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:43.257 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:43:28.382245+0000", 2021-02-10T04:50:43.257 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.8621e-05 2021-02-10T04:50:43.257 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.257 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.257 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:43.257 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:43:28.382304+0000", 2021-02-10T04:50:43.257 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.0219999999999993e-06 2021-02-10T04:50:43.257 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.257 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.257 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:43.257 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:43:28.382313+0000", 2021-02-10T04:50:43.257 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.6319000000000001e-05 2021-02-10T04:50:43.257 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.258 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.258 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:43.258 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:43:28.382329+0000", 2021-02-10T04:50:43.258 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.3459999999999994e-06 2021-02-10T04:50:43.258 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.258 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.258 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:43.258 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:43:28.382337+0000", 2021-02-10T04:50:43.258 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.6677e-05 2021-02-10T04:50:43.258 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.258 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.258 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:43.258 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:43:28.382394+0000", 2021-02-10T04:50:43.259 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.3331000000000001e-05 2021-02-10T04:50:43.259 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.259 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.259 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:50:43.259 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:43:28.382417+0000", 2021-02-10T04:50:43.259 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.041341581000000002 2021-02-10T04:50:43.259 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.259 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.259 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:50:43.259 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:43:28.423759+0000", 2021-02-10T04:50:43.259 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.9117000000000003e-05 2021-02-10T04:50:43.259 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.259 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.259 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:43.260 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:43:28.423818+0000", 2021-02-10T04:50:43.260 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.041573013999999998 2021-02-10T04:50:43.260 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.260 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:43.260 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.260 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.260 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.260 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPush(7.6 1038/1037 [PushOp(7:649ff6e5:::smithi08575397-4:30d, version: 996'481, data_included: [2470657~460040,3474633~442916], data_size: 902956, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:649ff6e5:::smithi08575397-4:30d@996'481, size: 3966716, copy_subset: [0~3966716], clone_subset: {7:649ff6e5:::smithi08575397-4:2ed=[],7:649ff6e5:::smithi08575397-4:31d=[]}, snapset: 377=[]:{21f=[203],2ed=[2ed,2eb,2ea,2d9,2c8,2ae],30d=[308],31d=[31a],345=[345,344,343,33e,33c,33a,335,332,32f],372=[372,370,36f,36b,368,366,35f,354,353,34e,34d,34b,349],377=[377,374,373]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3966716, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2470657, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:50:43.260 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:45:40.526611+0000", 2021-02-10T04:50:43.260 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 302.664972277, 2021-02-10T04:50:43.260 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.013833715999999999, 2021-02-10T04:50:43.260 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:43.260 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:50:43.261 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:43.261 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.261 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:43.261 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:45:40.526611+0000", 2021-02-10T04:50:43.261 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:43.261 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.261 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.261 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:43.261 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:45:40.526611+0000", 2021-02-10T04:50:43.261 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00050879799999999998 2021-02-10T04:50:43.261 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.261 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.261 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:43.261 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:45:40.527120+0000", 2021-02-10T04:50:43.262 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.4562999999999998e-05 2021-02-10T04:50:43.262 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.262 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.262 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:43.262 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:45:40.527144+0000", 2021-02-10T04:50:43.262 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.1081000000000002e-05 2021-02-10T04:50:43.262 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.262 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.262 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:43.262 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:45:40.527185+0000", 2021-02-10T04:50:43.262 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.0604000000000001e-05 2021-02-10T04:50:43.262 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.262 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.263 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:43.263 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:45:40.527206+0000", 2021-02-10T04:50:43.263 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0097080310000000006 2021-02-10T04:50:43.263 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.263 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.263 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:43.263 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:45:40.536914+0000", 2021-02-10T04:50:43.263 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.3061000000000001e-05 2021-02-10T04:50:43.263 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.263 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.263 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:50:43.263 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:45:40.536927+0000", 2021-02-10T04:50:43.263 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0035175779999999999 2021-02-10T04:50:43.263 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.264 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.264 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:43.264 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:45:40.540445+0000", 2021-02-10T04:50:43.264 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.013833715999999999 2021-02-10T04:50:43.264 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.264 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:43.264 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.264 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.264 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.264 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(osd.3.0:2980 4.1d e1066/55 4:bc2c97bb:::data_log.34:head v 1066'41, mlcod=1049'33)", 2021-02-10T04:50:43.264 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:47:18.804198+0000", 2021-02-10T04:50:43.264 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 204.38738483099999, 2021-02-10T04:50:43.264 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00077521000000000003, 2021-02-10T04:50:43.264 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:43.265 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:50:43.265 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:43.265 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.265 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:43.265 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:47:18.804198+0000", 2021-02-10T04:50:43.265 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:43.265 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.265 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.265 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:43.265 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:47:18.804198+0000", 2021-02-10T04:50:43.265 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.6962000000000002e-05 2021-02-10T04:50:43.265 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.265 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.266 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:43.266 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:47:18.804245+0000", 2021-02-10T04:50:43.266 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.0430000000000002e-06 2021-02-10T04:50:43.266 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.266 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.266 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:43.266 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:47:18.804252+0000", 2021-02-10T04:50:43.266 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0798000000000001e-05 2021-02-10T04:50:43.266 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.266 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.266 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:43.266 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:47:18.804263+0000", 2021-02-10T04:50:43.266 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.977e-06 2021-02-10T04:50:43.266 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.267 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.267 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:43.267 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:47:18.804268+0000", 2021-02-10T04:50:43.267 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000169982 2021-02-10T04:50:43.267 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.267 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.267 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:43.267 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:47:18.804438+0000", 2021-02-10T04:50:43.267 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.8516e-05 2021-02-10T04:50:43.267 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.267 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.267 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:50:43.267 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:47:18.804457+0000", 2021-02-10T04:50:43.268 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00047023200000000001 2021-02-10T04:50:43.268 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.268 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.268 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T04:50:43.268 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:47:18.804927+0000", 2021-02-10T04:50:43.268 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.6699999999999997e-05 2021-02-10T04:50:43.268 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.268 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.268 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:43.268 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:47:18.804974+0000", 2021-02-10T04:50:43.268 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00077521000000000003 2021-02-10T04:50:43.268 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.268 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:43.268 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.269 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.269 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.269 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPush(4.1c 1079/1078 [PushOp(4:3e426232:gc::gc.9:head, version: 62'9, data_included: [0~110], data_size: 110, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(4:3e426232:gc::gc.9:head@62'9, size: 110, copy_subset: [0~110], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:110, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2021-02-10T04:50:43.269 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:48:28.608721+0000", 2021-02-10T04:50:43.269 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 134.58286214099999, 2021-02-10T04:50:43.269 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00054460999999999995, 2021-02-10T04:50:43.269 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:43.269 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:50:43.269 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:43.269 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.269 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:43.269 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:48:28.608721+0000", 2021-02-10T04:50:43.269 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:43.270 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.270 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.270 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:43.270 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:48:28.608721+0000", 2021-02-10T04:50:43.270 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.9780999999999997e-05 2021-02-10T04:50:43.270 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.270 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.270 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:43.270 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:48:28.608761+0000", 2021-02-10T04:50:43.270 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.3810000000000001e-05 2021-02-10T04:50:43.270 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.270 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.271 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:43.271 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:48:28.608785+0000", 2021-02-10T04:50:43.271 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.5257999999999999e-05 2021-02-10T04:50:43.271 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.271 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.271 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:43.271 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:48:28.608830+0000", 2021-02-10T04:50:43.271 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1161999999999999e-05 2021-02-10T04:50:43.271 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.271 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.271 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:43.271 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:48:28.608841+0000", 2021-02-10T04:50:43.271 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000111196 2021-02-10T04:50:43.271 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.272 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.272 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:43.272 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:48:28.608952+0000", 2021-02-10T04:50:43.272 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2441e-05 2021-02-10T04:50:43.272 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.272 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.272 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:50:43.272 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:48:28.608965+0000", 2021-02-10T04:50:43.272 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000300962 2021-02-10T04:50:43.272 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.272 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.272 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:43.272 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:48:28.609266+0000", 2021-02-10T04:50:43.272 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00054460999999999995 2021-02-10T04:50:43.273 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.273 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:43.273 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.273 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.273 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.273 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPush(4.1c 1079/1078 [PushOp(4:3d9e841d:::obj_delete_at_hint.0000000041:head, version: 1048'35, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:3d9e841d:::obj_delete_at_hint.0000000041:head@1048'35, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2021-02-10T04:50:43.273 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:48:28.988028+0000", 2021-02-10T04:50:43.273 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 134.20355538600001, 2021-02-10T04:50:43.273 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00042702099999999998, 2021-02-10T04:50:43.273 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:43.273 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:50:43.273 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:43.273 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.273 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:43.274 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:48:28.988028+0000", 2021-02-10T04:50:43.274 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:43.274 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.274 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.274 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:43.274 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:48:28.988028+0000", 2021-02-10T04:50:43.274 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.8088999999999999e-05 2021-02-10T04:50:43.274 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.274 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.274 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:43.274 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:48:28.988086+0000", 2021-02-10T04:50:43.274 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.1464999999999999e-05 2021-02-10T04:50:43.274 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.275 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.275 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:43.275 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:48:28.988107+0000", 2021-02-10T04:50:43.275 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.9485999999999999e-05 2021-02-10T04:50:43.275 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.275 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.275 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:43.275 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:48:28.988137+0000", 2021-02-10T04:50:43.275 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.4999e-05 2021-02-10T04:50:43.275 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.275 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.275 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:43.275 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:48:28.988152+0000", 2021-02-10T04:50:43.275 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.2948999999999994e-05 2021-02-10T04:50:43.276 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.276 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.276 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:43.276 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:48:28.988245+0000", 2021-02-10T04:50:43.276 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0227000000000001e-05 2021-02-10T04:50:43.276 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.276 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.276 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:50:43.276 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:48:28.988255+0000", 2021-02-10T04:50:43.276 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00019980599999999999 2021-02-10T04:50:43.276 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.276 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.276 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:43.276 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:48:28.988455+0000", 2021-02-10T04:50:43.277 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00042702099999999998 2021-02-10T04:50:43.277 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.277 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:43.277 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.277 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.277 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.277 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPush(4.1c 1079/1078 [PushOp(4:38283b28:::obj_delete_at_hint.0000000089:head, version: 1049'37, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:38283b28:::obj_delete_at_hint.0000000089:head@1049'37, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2021-02-10T04:50:43.277 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:48:29.083242+0000", 2021-02-10T04:50:43.277 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 134.10834131799999, 2021-02-10T04:50:43.277 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00043425499999999998, 2021-02-10T04:50:43.277 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:43.277 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:50:43.277 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:43.277 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.278 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:43.278 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:48:29.083242+0000", 2021-02-10T04:50:43.278 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:43.278 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.278 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.278 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:43.278 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:48:29.083242+0000", 2021-02-10T04:50:43.278 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.8297000000000001e-05 2021-02-10T04:50:43.278 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.278 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.278 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:43.278 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:48:29.083300+0000", 2021-02-10T04:50:43.278 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.3322999999999999e-05 2021-02-10T04:50:43.279 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.279 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.279 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:43.279 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:48:29.083314+0000", 2021-02-10T04:50:43.279 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.5347999999999999e-05 2021-02-10T04:50:43.279 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.279 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.279 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:43.279 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:48:29.083339+0000", 2021-02-10T04:50:43.279 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.5336e-05 2021-02-10T04:50:43.279 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.279 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.279 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:43.279 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:48:29.083354+0000", 2021-02-10T04:50:43.280 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.6234000000000001e-05 2021-02-10T04:50:43.280 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.280 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.280 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:43.280 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:48:29.083451+0000", 2021-02-10T04:50:43.280 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1936000000000001e-05 2021-02-10T04:50:43.280 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.280 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.280 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:50:43.280 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:48:29.083462+0000", 2021-02-10T04:50:43.280 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00021378099999999999 2021-02-10T04:50:43.280 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.280 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.281 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:43.281 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:48:29.083676+0000", 2021-02-10T04:50:43.281 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00043425499999999998 2021-02-10T04:50:43.281 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.281 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:43.281 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.281 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.281 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.281 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDScrubReserve(6.22 RELEASE e1085)", 2021-02-10T04:50:43.281 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:49:01.203787+0000", 2021-02-10T04:50:43.281 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 101.987796102, 2021-02-10T04:50:43.281 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0026857389999999999, 2021-02-10T04:50:43.281 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:50:43.281 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:50:43.282 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:50:43.282 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.282 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:50:43.282 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:49:01.203787+0000", 2021-02-10T04:50:43.282 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:50:43.282 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.282 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.282 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:50:43.282 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:49:01.203787+0000", 2021-02-10T04:50:43.282 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.1279000000000003e-05 2021-02-10T04:50:43.282 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.282 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.282 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:50:43.282 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:49:01.203818+0000", 2021-02-10T04:50:43.283 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.6819999999999997e-06 2021-02-10T04:50:43.283 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.283 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.283 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:50:43.283 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:49:01.203825+0000", 2021-02-10T04:50:43.283 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0288e-05 2021-02-10T04:50:43.283 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.283 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.283 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:50:43.283 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:49:01.203835+0000", 2021-02-10T04:50:43.283 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.3090000000000001e-06 2021-02-10T04:50:43.283 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.283 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.283 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:50:43.284 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:49:01.203842+0000", 2021-02-10T04:50:43.284 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0026159040000000001 2021-02-10T04:50:43.284 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.284 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.284 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:50:43.284 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:49:01.206458+0000", 2021-02-10T04:50:43.284 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1985e-05 2021-02-10T04:50:43.284 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.284 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.284 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:50:43.284 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:49:01.206470+0000", 2021-02-10T04:50:43.284 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.292e-06 2021-02-10T04:50:43.284 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:50:43.285 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:50:43.285 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:50:43.285 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:49:01.206473+0000", 2021-02-10T04:50:43.285 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0026857389999999999 2021-02-10T04:50:43.285 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.285 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:43.285 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.285 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:50:43.285 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:50:43.285 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:50:43.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:43 smithi085 conmon[29319]: cluster 2021-02-10T04:50:41.615854+0000 osd.4 (osd.4) 2450 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:43.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:43 smithi085 conmon[29319]: cluster 2021-02-10T04:50:41.615862+0000 osd.4 (osd.4) 2451 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:43.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:43 smithi085 conmon[29319]: cluster 2021-02-10T04:50:41.615868+0000 osd.4 (osd.4) 2452 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:43.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:43 smithi085 conmon[29319]: cluster 2021-02-10T04:50:41.615874+0000 osd.4 (osd.4) 2453 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:43.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:43 smithi085 conmon[35961]: cluster 2021-02-10T04:50:43.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:43 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:50:43.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:43 smithi085 conmon[35961]: 04:50:41.615854+0000 osd.4 (osd.4) 2450 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:43.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:43 smithi085 conmon[35961]: cluster 2021-02-10T04:50:43.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:43 smithi085 conmon[35961]: 2021-02-10T04:50:43.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:43 smithi085 conmon[35961]: 2021-02-10T04:50:41 2021-02-10T04:50:43.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:43 smithi085 conmon[35961]: .615862+0000 osd.4 (osd.4) 2451 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:43.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:43 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:50:43.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:43 smithi085 conmon[35961]: 04:50:41.615868+0000 osd.4 (osd.4) 2452 2021-02-10T04:50:43.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:43 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:43.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:43 smithi085 conmon[35961]: cluster 2021-02-10T04:50:41. 2021-02-10T04:50:43.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:43 smithi085 conmon[35961]: 615874+0000 osd.4 (osd.4) 2453 : cluster [WRN] 2021-02-10T04:50:43.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:43 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:43.600 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:43 smithi164 conmon[33677]: debug 2021-02-10T04:50:43.599+0000 7fbec19a1700 -1 osd.4 1105 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:50:43.601 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:43 smithi164 conmon[29288]: cluster 2021-02-10T04:50:41.615854+0000 2021-02-10T04:50:43.601 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:43 smithi164 conmon[29288]: osd.4 (osd.4) 2450 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:43.601 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:43 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:50:43.601 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:43 smithi164 conmon[29288]: T04:50:41.615862+0000 osd.4 (osd.4) 2451 : cluster [WRN] 2021-02-10T04:50:43.601 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:43 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:43.601 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:43 smithi164 conmon[29288]: cluster 2021-02-10T04:50:41. 2021-02-10T04:50:43.601 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:43 smithi164 conmon[29288]: 615868+0000 osd.4 (osd.4) 2452 : cluster [WRN] 2021-02-10T04:50:43.601 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:43 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:43.601 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:43 smithi164 conmon[29288]: cluster 2021-02-10T04:50:41.615874+0000 2021-02-10T04:50:43.601 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:43 smithi164 conmon[29288]: osd.4 (osd.4) 2453 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:43.601 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:43 smithi164 conmon[29288]: 2021-02-10T04:50:44.498 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[35961]: cluster 2021-02-10T04:50:44.498 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:50:44.498 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[35961]: 04:50:42.402544+0000 mgr.y (mgr.14140) 1178 : cluster [DBG] pgmap v1967: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:44.498 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:50:44.498 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[35961]: 10T04:50:42.603959+0000 osd.4 (osd.4) 2454 : 2021-02-10T04:50:44.498 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:44.498 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[35961]: cluster 2021-02-10T04:50:42.603970+0000 2021-02-10T04:50:44.498 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[35961]: osd.4 (osd.4) 2455 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:44.498 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:50:44.498 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[35961]: :50:42.603979+0000 osd.4 (osd.4) 2021-02-10T04:50:44.499 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[35961]: 2456 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:44.499 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:50:44.499 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[35961]: 04:50:42.603986+0000 osd.4 (osd.4) 2021-02-10T04:50:44.499 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[35961]: 2457 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:44.499 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:50:44.499 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[35961]: :50:43.943086+0000 mon.c (mon.1) 86 : 2021-02-10T04:50:44.499 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[35961]: audit [INF] from='client.? 172.21.15.85:0/2464202791' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:50:44.499 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:50:44.499 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[35961]: 50:43.943493+0000 mon.a (mon.0 2021-02-10T04:50:44.499 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[35961]: ) 2273 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:50:44.500 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[29319]: cluster 2021-02-10T04:50:42 2021-02-10T04:50:44.500 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[29319]: .402544+0000 mgr.y (mgr.14140) 1178 : cluster [DBG] pgmap v1967: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:44.500 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[29319]: cluster 2021-02-10T04:50:42.603959+0000 osd.4 (osd.4) 2454 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:44.500 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[29319]: cluster 2021-02-10T04:50:42.603970+0000 osd.4 (osd.4) 2455 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:44.500 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:50:44.500 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[29319]: :50:42.603979+0000 osd.4 (osd.4) 2456 : cluster 2021-02-10T04:50:44.500 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:44.500 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:50:44.500 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[29319]: T04:50:42.603986+0000 osd.4 (osd 2021-02-10T04:50:44.500 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[29319]: .4) 2457 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:44.500 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[29319]: audit 2021-02-10T04:50:44.501 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[29319]: 2021-02-10T04:50:43.943086+0000 2021-02-10T04:50:44.501 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[29319]: mon.c (mon.1) 86 : audit [INF] from='client.? 172.21.15.85:0/2464202791' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:50:44.501 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:50:44.501 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[29319]: T04:50:43.943493+0000 mon.a (mon. 2021-02-10T04:50:44.501 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:44 smithi085 conmon[29319]: 0) 2273 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:50:44.585 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is unset 2021-02-10T04:50:44.621 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:44 smithi164 conmon[29288]: cluster 2021-02-10T04:50:42.402544+0000 mgr.y (mgr.14140) 1178 : cluster 2021-02-10T04:50:44.621 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:44 smithi164 conmon[29288]: [DBG] pgmap v1967: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:44.622 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:44 smithi164 conmon[29288]: cluster 2021-02-10T04:50:42.603959+0000 osd.4 (osd.4) 2454 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:44.622 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:44 smithi164 conmon[29288]: cluster 2021-02-10T04:50:42.603970 2021-02-10T04:50:44.622 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:44 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2455 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:44.622 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:44 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:50:44.622 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:44 smithi164 conmon[29288]: 50:42.603979+0000 osd.4 (osd.4) 2456 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:44.622 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:44 smithi164 conmon[29288]: cluster 2021-02-10T04:50:42.603986+0000 osd.4 (osd.4 2021-02-10T04:50:44.622 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:44 smithi164 conmon[29288]: ) 2457 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:44.622 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:44 smithi164 conmon[29288]: audit 2021 2021-02-10T04:50:44.622 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:44 smithi164 conmon[29288]: -02-10T04:50:43.943086+0000 mon.c (mon.1) 86 : audit [INF] from='client.? 172.21.15.85:0/2464202791' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:50:44.622 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:44 smithi164 conmon[29288]: audit 2021-02-10T04:50:43.943493+0000 2021-02-10T04:50:44.622 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:44 smithi164 conmon[29288]: mon.a (mon.0) 2273 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:50:44.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:44 smithi164 conmon[33677]: debug 2021-02-10T04:50:44.620+0000 7fbec19a1700 -1 osd.4 1106 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:50:45.622 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:45 smithi164 conmon[29288]: cluster 2021-02-10T04:50:43.600029+0000 osd.4 2021-02-10T04:50:45.622 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:45 smithi164 conmon[29288]: (osd.4) 2458 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:45.623 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:45 smithi164 conmon[29288]: cluster 2021-02-10T04:50:43.600041+0000 osd.4 (osd.4) 2459 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:45.623 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:45 smithi164 conmon[29288]: 2021-02-10T04:50:45.623 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:45 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:50:45.623 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:45 smithi164 conmon[29288]: T04:50:43.600051+0000 osd.4 ( 2021-02-10T04:50:45.623 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:45 smithi164 conmon[29288]: osd.4) 2460 : cluster [WRN] 2021-02-10T04:50:45.623 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:45 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:45.623 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:45 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:50:45.623 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:45 smithi164 conmon[29288]: -02-10T04:50 2021-02-10T04:50:45.623 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:45 smithi164 conmon[29288]: :43.600058+0000 osd.4 (osd 2021-02-10T04:50:45.623 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:45 smithi164 conmon[29288]: .4) 2461 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:45.623 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:45 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:50:45.623 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:45 smithi164 conmon[29288]: -10T04:50:44.464203+0000 mon.a ( 2021-02-10T04:50:45.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:45 smithi164 conmon[29288]: mon.0) 2274 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:50:45.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:45 smithi164 conmon[29288]: audit 2021-02-10T04:50:45.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:45 smithi164 conmon[29288]: 2021-02-10T04:50:44.468383 2021-02-10T04:50:45.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:45 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2275 : audit [INF] 2021-02-10T04:50:45.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:45 smithi164 conmon[29288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:50:45.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:45 smithi164 conmon[29288]: cluster 2021-02-10T04:50:45.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:45 smithi164 conmon[29288]: 2021-02-10T04:50:44. 2021-02-10T04:50:45.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:45 smithi164 conmon[29288]: 468456+0000 mon.a (mon.0) 2276 : cluster [DBG] 2021-02-10T04:50:45.624 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:45 smithi164 conmon[29288]: osdmap e1106: 8 total, 7 up, 7 in 2021-02-10T04:50:45.645 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[29319]: cluster 2021-02-10T04:50:43.600029+0000 osd.4 ( 2021-02-10T04:50:45.646 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[29319]: osd.4) 2458 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:45.646 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[29319]: cluster 2021-02-10T04:50:43.600041+0000 osd.4 (osd.4) 2459 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:45.646 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:50:45.646 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[29319]: 50:43.600051+0000 osd.4 (osd 2021-02-10T04:50:45.646 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[29319]: .4) 2460 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:45.646 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:50:45.646 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[29319]: -02-10T04:50:43.600058+0000 2021-02-10T04:50:45.646 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[29319]: osd.4 (osd.4) 2461 : cluster [WRN] 2021-02-10T04:50:45.646 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:45.646 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[29319]: cluster 2021-02-10T04:50 2021-02-10T04:50:45.646 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[29319]: :44.464203+0000 mon.a (mon.0) 2274 2021-02-10T04:50:45.647 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[29319]: : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:50:45.647 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:50:45.647 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[29319]: T04:50:44.468383+0000 mon.a ( 2021-02-10T04:50:45.647 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[29319]: mon.0) 2275 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:50:45.647 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[29319]: cluster 2021-02-10T04:50:45.647 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[29319]: 2021-02-10T04:50 2021-02-10T04:50:45.647 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[29319]: :44.468456+0000 mon.a (mon 2021-02-10T04:50:45.647 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[29319]: .0) 2276 : cluster [DBG] osdmap e1106: 8 total, 7 up, 7 in 2021-02-10T04:50:45.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[35961]: cluster 2021-02-10T04:50:43.600029+0000 osd.4 (osd.4 2021-02-10T04:50:45.647 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[35961]: ) 2458 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:45.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[35961]: cluster 2021-02-10T04:50:43.600041+0000 osd.4 (osd.4) 2021-02-10T04:50:45.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[35961]: 2459 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:45.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[35961]: cluster 2021-02-10T04:50 2021-02-10T04:50:45.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[35961]: :43.600051+0000 osd.4 (osd.4) 2021-02-10T04:50:45.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[35961]: 2460 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:45.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[35961]: cluster 2021-02-10T04:50:45.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:50:45.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[35961]: 04:50:43.600058+0000 osd.4 ( 2021-02-10T04:50:45.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[35961]: osd.4) 2461 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:45.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:50:45.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[35961]: 02-10T04:50:44.464203+0000 mon.a 2021-02-10T04:50:45.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[35961]: (mon.0) 2274 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:50:45.648 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[35961]: 2021-02-10T04:50:45.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[35961]: audit 2021-02-10T04:50:45.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[35961]: 2021-02-10T04:50:44 2021-02-10T04:50:45.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[35961]: .468383+0000 mon.a (mon.0) 2275 2021-02-10T04:50:45.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[35961]: : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:50:45.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:50:45.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[35961]: -10T04:50:44.468456+0000 mon.a 2021-02-10T04:50:45.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[35961]: (mon.0) 2276 : cluster [DBG] osdmap e1106: 8 total, 7 up, 7 in 2021-02-10T04:50:45.649 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:45 smithi085 conmon[35961]: 2021-02-10T04:50:45.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:45 smithi164 conmon[33677]: debug 2021-02-10T04:50:45.620+0000 7fbec19a1700 -1 osd.4 1106 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:50:46.576 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:46 smithi085 bash[146287]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2021-02-10T04:50:46.576 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:46 smithi085 bash[146287]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-3 --no-mon-config 2021-02-10T04:50:46.576 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:46 smithi085 bash[146287]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3/block 2021-02-10T04:50:46.576 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:46 smithi085 bash[146287]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block 2021-02-10T04:50:46.577 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:46 smithi085 bash[146287]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2021-02-10T04:50:46.577 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:46 smithi085 bash[146287]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2021-02-10T04:50:46.577 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:46 smithi085 bash[146287]: --> ceph-volume lvm activate successful for osd ID: 3 2021-02-10T04:50:46.577 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[29319]: cluster 2021-02-10T04:50:44. 2021-02-10T04:50:46.577 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[29319]: 403233+0000 mgr.y (mgr.14140) 1179 : cluster [DBG] pgmap v1968: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:46.577 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[29319]: cluster 2021-02-10T04:50:44.621248+0000 osd.4 (osd. 2021-02-10T04:50:46.577 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[29319]: 4) 2462 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:46.578 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[29319]: cluster 2021-02-10T04:50: 2021-02-10T04:50:46.578 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[29319]: 44.621258+0000 osd.4 (osd.4) 2463 2021-02-10T04:50:46.578 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:46.578 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:50:46.578 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[29319]: 10T04:50:44.621265+0000 osd.4 2021-02-10T04:50:46.578 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[29319]: (osd.4) 2464 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:46.578 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:50:46.578 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[29319]: 04:50:44.621271+0000 osd.4 (osd.4) 2465 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:46.578 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[29319]: cluster 2021-02-10T04:50:46.578 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[29319]: 2021-02-10T04:50:44.624497+0000 osd.4 (osd.4) 2466 : cluster [DBG] 4.9 scrub ok 2021-02-10T04:50:46.578 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:50:46.578 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[29319]: -10T04:50:45.362407+0000 osd.6 (osd.6) 145 : cluster [DBG] 2021-02-10T04:50:46.579 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[29319]: 6.15 scrub ok 2021-02-10T04:50:46.579 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[35961]: cluster 2021-02-10T04:50:44 2021-02-10T04:50:46.579 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[35961]: .403233+0000 mgr.y (mgr.14140) 1179 : cluster [DBG] pgmap v1968: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:46.579 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[35961]: cluster 2021-02-10T04:50:44.621248+0000 2021-02-10T04:50:46.579 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[35961]: osd.4 (osd.4) 2462 : cluster [WRN] 2021-02-10T04:50:46.579 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:46.579 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[35961]: cluster 2021-02-10T04:50:46.579 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[35961]: 2021 2021-02-10T04:50:46.579 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[35961]: -02-10T04:50:44 2021-02-10T04:50:46.579 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[35961]: .621258+0000 osd.4 (osd 2021-02-10T04:50:46.579 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[35961]: .4) 2463 : 2021-02-10T04:50:46.579 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[35961]: cluster [WRN] 2021-02-10T04:50:46.579 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:46.580 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[35961]: cluster 2021-02-10T04:50:46.580 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:50:46.580 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[35961]: T04:50:44.621265+0000 osd.4 2021-02-10T04:50:46.580 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[35961]: (osd.4) 2464 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:46.580 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:50:46.580 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[35961]: 02-10T04:50:44.621271+0000 osd.4 (osd.4 2021-02-10T04:50:46.580 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[35961]: ) 2465 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:46.580 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:50:46.580 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[35961]: 02-10T04:50:44.624497+0000 2021-02-10T04:50:46.580 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[35961]: osd.4 2021-02-10T04:50:46.580 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[35961]: (osd.4) 2466 2021-02-10T04:50:46.580 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[35961]: : cluster [DBG] 4.9 scrub ok 2021-02-10T04:50:46.580 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:50:46.581 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[35961]: 02-10T04: 2021-02-10T04:50:46.581 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[35961]: 50:45.362407+0000 osd.6 2021-02-10T04:50:46.581 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[35961]: (osd.6) 145 : cluster 2021-02-10T04:50:46.581 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:46 smithi085 conmon[35961]: [DBG] 6.15 scrub ok 2021-02-10T04:50:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:46 smithi164 conmon[29288]: cluster 2021-02-10T04:50:44.403233+0000 mgr.y ( 2021-02-10T04:50:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:46 smithi164 conmon[29288]: mgr.14140) 1179 : cluster [DBG] pgmap v1968: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:46 smithi164 conmon[29288]: cluster 2021-02-10T04:50:44.621248+0000 osd.4 (osd.4) 2462 : cluster [WRN] 2021-02-10T04:50:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:46 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:46 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:50:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:46 smithi164 conmon[29288]: 04:50:44.621258+0000 osd.4 (osd.4) 2463 : cluster [WRN] 2021-02-10T04:50:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:46 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:46 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:50:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:46 smithi164 conmon[29288]: 02-10T04:50:44.621265+0000 osd.4 (osd 2021-02-10T04:50:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:46 smithi164 conmon[29288]: .4) 2464 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:46 smithi164 conmon[29288]: cluster 2021-02-10T04:50:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:46 smithi164 conmon[29288]: 2021-02-10T04:50:44.621271+0000 osd.4 2021-02-10T04:50:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:46 smithi164 conmon[29288]: (osd.4) 2465 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:46 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:50:46.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:46 smithi164 conmon[29288]: -10T04:50:44.624497+0000 osd.4 ( 2021-02-10T04:50:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:46 smithi164 conmon[29288]: osd.4) 2466 : cluster [DBG] 4.9 scrub ok 2021-02-10T04:50:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:46 smithi164 conmon[29288]: cluster 2021-02-10T04:50:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:46 smithi164 conmon[29288]: 2021-02-10T04:50:45.362407+0000 2021-02-10T04:50:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:46 smithi164 conmon[29288]: osd.6 (osd.6) 145 : cluster [DBG] 2021-02-10T04:50:46.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:46 smithi164 conmon[29288]: 6.15 scrub ok 2021-02-10T04:50:47.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:46 smithi164 conmon[33677]: debug 2021-02-10T04:50:46.643+0000 7fbec19a1700 -1 osd.4 1106 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:50:47.565 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[29319]: cluster 2021-02-10T04:50:45.621537 2021-02-10T04:50:47.565 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2467 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:47.565 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[29319]: cluster 2021-02-10T04:50:45.621544+0000 osd.4 (osd. 2021-02-10T04:50:47.565 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[29319]: 4) 2468 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:47.565 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:50:47.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[29319]: 02-10T04:50:45.621553+0000 osd.4 (osd.4) 2469 : cluster 2021-02-10T04:50:47.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:47.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[29319]: cluster 2021-02-10T04:50 2021-02-10T04:50:47.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[29319]: :45.621556+0000 osd.4 (osd.4) 2470 : cluster 2021-02-10T04:50:47.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:47.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[29319]: cluster 2021-02-10T04:50: 2021-02-10T04:50:47.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[29319]: 45.623637+0000 osd.4 (osd.4) 2471 : cluster [DBG] 2.b scrub ok 2021-02-10T04:50:47.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[29319]: 2021-02-10T04:50:47.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[29319]: cluster 2021-02-10T04:50:47.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[29319]: 2021-02-10T04:50:46.380281+0000 osd.6 (osd 2021-02-10T04:50:47.566 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[29319]: .6) 146 : cluster [DBG] 5.6 scrub ok 2021-02-10T04:50:47.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[35961]: cluster 2021-02-10T04:50: 2021-02-10T04:50:47.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[35961]: 45.621537+0000 osd.4 (osd.4) 2467 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:47.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[35961]: cluster 2021-02-10T04:50: 2021-02-10T04:50:47.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[35961]: 45.621544+0000 osd.4 (osd.4) 2468 2021-02-10T04:50:47.567 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:47.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[35961]: cluster 2021-02-10T04:50:45. 2021-02-10T04:50:47.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[35961]: 621553+0000 osd.4 (osd.4) 2469 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:47.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[35961]: 2021-02-10T04:50:47.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[35961]: cluster 2021-02-10T04:50:47.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[35961]: 2021-02-10T04:50:45 2021-02-10T04:50:47.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[35961]: .621556+0000 osd.4 (osd.4 2021-02-10T04:50:47.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[35961]: ) 2470 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:47.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[35961]: 2021-02-10T04:50:47.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:50:47.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[35961]: 04:50:45.623637+0000 2021-02-10T04:50:47.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[35961]: osd.4 (osd.4) 2471 : 2021-02-10T04:50:47.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[35961]: cluster [DBG] 2.b scrub ok 2021-02-10T04:50:47.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:50:47.568 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[35961]: -02-10T04:50: 2021-02-10T04:50:47.569 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[35961]: 46.380281+0000 osd.6 (osd 2021-02-10T04:50:47.569 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[35961]: .6) 146 : cluster [DBG] 2021-02-10T04:50:47.569 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:47 smithi085 conmon[35961]: 5.6 scrub ok 2021-02-10T04:50:47.569 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.2 dump_ops_in_flight 2021-02-10T04:50:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:47 smithi164 conmon[29288]: cluster 2021-02-10T04:50:45.621537 2021-02-10T04:50:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:47 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2467 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:47 smithi164 conmon[29288]: cluster 2021-02-10T04:50:45.621544+0000 osd.4 (osd.4) 2468 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:47.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:47 smithi164 conmon[29288]: cluster 2021-02-10T04:50:45.621553+0000 osd.4 (osd.4 2021-02-10T04:50:47.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:47 smithi164 conmon[29288]: ) 2469 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:47.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:47 smithi164 conmon[29288]: cluster 2021-02-10T04:50:45.621556+0000 osd.4 (osd.4) 2470 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:47.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:47 smithi164 conmon[29288]: cluster 2021-02-10T04:50:45.623637+0000 osd.4 (osd.4) 2471 : cluster [DBG] 2.b scrub ok 2021-02-10T04:50:47.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:47 smithi164 conmon[29288]: cluster 2021-02-10T04:50:46.380281+0000 osd.6 ( 2021-02-10T04:50:47.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:47 smithi164 conmon[29288]: osd.6) 146 : cluster [DBG] 5.6 scrub ok 2021-02-10T04:50:47.816 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:50:48.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:47 smithi164 conmon[33677]: debug 2021-02-10T04:50:47.658+0000 7fbec19a1700 -1 osd.4 1107 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:50:48.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:48 smithi085 conmon[29319]: cluster 2021-02-10T04:50:46.332500+0000 osd.7 (osd.7) 82 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:50:48.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:48 smithi085 conmon[29319]: cluster 2021-02-10T04:50:46.334674+0000 osd.7 (osd.7) 83 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:50:48.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:48 smithi085 conmon[29319]: cluster 2021-02-10T04:50:46.403812+0000 mgr.y (mgr.14140) 1180 : cluster [DBG] pgmap v1970: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:48.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:48 smithi085 conmon[29319]: cluster 2021-02-10T04:50:46.434622+0000 osd.0 (osd.0) 155 : cluster [DBG] 2.5 scrub ok 2021-02-10T04:50:48.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:48 smithi085 conmon[29319]: cluster 2021-02-10T04:50:46.644643+0000 osd.4 (osd.4) 2472 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:48.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:48 smithi085 conmon[29319]: cluster 2021-02-10T04:50:46.644657+0000 osd.4 (osd.4) 2473 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:48.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:48 smithi085 conmon[29319]: cluster 2021-02-10T04:50:46.644665+0000 osd.4 (osd.4) 2474 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:48.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:48 smithi085 conmon[29319]: cluster 2021-02-10T04:50 2021-02-10T04:50:48.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:48 smithi085 conmon[29319]: :46.644672+0000 osd.4 (osd.4) 2475 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:48.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:48 smithi085 conmon[29319]: cluster 2021-02-10T04:50:47.236727+0000 mon.a (mon.0) 2277 : cluster [DBG] osdmap e1107: 8 total, 7 up, 7 in 2021-02-10T04:50:48.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:48 smithi085 conmon[29319]: cluster 2021-02-10T04:50:47.407553+0000 osd.5 (osd.5) 147 : cluster [DBG] 5.15 scrub ok 2021-02-10T04:50:48.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:48 smithi085 conmon[35961]: cluster 2021-02-10T04:50:46.332500+0000 osd.7 (osd.7) 82 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:50:48.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:48 smithi085 conmon[35961]: cluster 2021-02-10T04:50:46.334674+0000 osd.7 (osd.7) 83 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:50:48.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:48 smithi085 conmon[35961]: cluster 2021-02-10T04:50:46.403812+0000 mgr.y (mgr.14140) 1180 : cluster [DBG] pgmap v1970: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:48.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:48 smithi085 conmon[35961]: cluster 2021-02-10T04:50:46.434622+0000 osd.0 (osd.0) 155 : cluster [DBG] 2.5 scrub ok 2021-02-10T04:50:48.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:48 smithi085 conmon[35961]: cluster 2021-02-10T04:50:46.644643+0000 osd.4 (osd.4) 2472 : cluster 2021-02-10T04:50:48.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:48 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:48.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:48 smithi085 conmon[35961]: cluster 2021-02-10T04:50:46.644657+0000 osd.4 (osd.4) 2473 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:48.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:48 smithi085 conmon[35961]: cluster 2021-02-10T04:50:46.644665+0000 osd.4 (osd.4) 2474 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:48.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:48 smithi085 conmon[35961]: cluster 2021-02-10T04:50:46.644672+0000 osd.4 (osd.4) 2475 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:48.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:48 smithi085 conmon[35961]: cluster 2021-02-10T04:50:47.236727+0000 mon.a (mon.0) 2277 : cluster [DBG] osdmap e1107: 8 total, 7 up, 7 in 2021-02-10T04:50:48.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:48 smithi085 conmon[35961]: cluster 2021-02-10T04:50:47.407553+0000 osd.5 (osd.5) 147 : cluster [DBG] 5.15 scrub ok 2021-02-10T04:50:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: cluster 2021-02-10T04:50:46. 2021-02-10T04:50:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: 332500+0000 osd.7 (osd.7) 82 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:50:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: cluster 2021-02-10T04:50:46.334674+0000 2021-02-10T04:50:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: osd.7 (osd.7) 83 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:50:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: cluster 2021-02-10T04:50 2021-02-10T04:50:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: :46.403812+0000 mgr.y (mgr.14140) 1180 : cluster 2021-02-10T04:50:48.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: [DBG] pgmap v1970: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:48.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:50:48.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: 02-10T04:50:46. 2021-02-10T04:50:48.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: 434622+0000 osd.0 (osd.0) 155 : cluster 2021-02-10T04:50:48.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: [DBG] 2.5 scrub ok 2021-02-10T04:50:48.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: cluster 2021-02-10T04:50:48.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: 2021-02-10T04:50: 2021-02-10T04:50:48.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: 46.644643+0000 osd.4 (osd.4) 2472 : cluster 2021-02-10T04:50:48.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:48.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: cluster 2021-02-10T04:50 2021-02-10T04:50:48.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: :46.644657+0000 osd.4 (osd.4) 2473 : cluster 2021-02-10T04:50:48.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:48.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: cluster 2021-02-10T04:50: 2021-02-10T04:50:48.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: 46.644665+0000 osd.4 (osd.4) 2474 : cluster 2021-02-10T04:50:48.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:48.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: cluster 2021-02-10T04:50:48.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: 2021-02-10T04:50 2021-02-10T04:50:48.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: :46.644672+0000 osd.4 ( 2021-02-10T04:50:48.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: osd.4) 2475 : cluster [WRN] 2021-02-10T04:50:48.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:48.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:50:48.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: 02-10T04:50:47.236727+0000 mon.a ( 2021-02-10T04:50:48.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: mon.0) 2277 : cluster [DBG] osdmap e1107: 8 total, 7 up, 7 in 2021-02-10T04:50:48.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:50:48.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: -02-10T04:50:47.407553+0000 osd.5 2021-02-10T04:50:48.648 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[29288]: (osd.5) 147 : cluster [DBG] 5.15 scrub ok 2021-02-10T04:50:49.031 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:50:49 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:50:49.031 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:48 smithi164 conmon[33677]: debug 2021-02-10T04:50:48.664+0000 7fbec19a1700 -1 osd.4 1107 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:50:49.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:49 smithi164 conmon[29288]: cluster 2021-02-10T04:50:49.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:49 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:50:49.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:49 smithi164 conmon[29288]: 04:50:47.481679+0000 osd.0 (osd.0) 156 : 2021-02-10T04:50:49.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:49 smithi164 conmon[29288]: cluster [DBG] 2.21 scrub ok 2021-02-10T04:50:49.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:49 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:50:49.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:49 smithi164 conmon[29288]: 02-10T04:50:47.658736+0000 osd.4 2021-02-10T04:50:49.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:49 smithi164 conmon[29288]: (osd.4) 2476 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:49.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:49 smithi164 conmon[29288]: cluster 2021-02-10T04:50:49.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:49 smithi164 conmon[29288]: 2021-02-10T04:50:47. 2021-02-10T04:50:49.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:49 smithi164 conmon[29288]: 658746+0000 osd.4 (osd.4) 2477 : cluster [WRN] 2021-02-10T04:50:49.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:49 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:49.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:49 smithi164 conmon[29288]: cluster 2021-02-10T04:50: 2021-02-10T04:50:49.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:49 smithi164 conmon[29288]: 47.658757+0000 osd.4 (osd.4) 2478 : cluster 2021-02-10T04:50:49.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:49 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:49.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:49 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:50:49.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:49 smithi164 conmon[29288]: 04:50:47.658764+0000 2021-02-10T04:50:49.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:49 smithi164 conmon[29288]: osd.4 (osd.4) 2479 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:49.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:49 smithi164 conmon[29288]: 2021-02-10T04:50:49.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:49 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:50:49.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:49 smithi164 conmon[29288]: 02-10T04:50:47.662371+0000 2021-02-10T04:50:49.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:49 smithi164 conmon[29288]: osd.4 (osd.4) 2480 : cluster [DBG] 4.d scrub ok 2021-02-10T04:50:49.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:49 smithi164 conmon[29288]: cluster 2021-02-10T04:50:49.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:49 smithi164 conmon[29288]: 2021-02-10T04:50:48.408744 2021-02-10T04:50:49.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:49 smithi164 conmon[29288]: +0000 osd.5 (osd.5) 148 : cluster [DBG] 6.a scrub ok 2021-02-10T04:50:49.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:49 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:50:49.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:49 smithi164 conmon[29288]: 02-10T04:50:48.463424+0000 mon.a (mon 2021-02-10T04:50:49.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:49 smithi164 conmon[29288]: .0) 2278 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 377 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:50:49.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:49 smithi085 conmon[35961]: cluster 2021-02-10T04:50:47.481679+0000 osd.0 (osd. 2021-02-10T04:50:49.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:49 smithi085 conmon[35961]: 0) 156 : cluster [DBG] 2.21 scrub ok 2021-02-10T04:50:49.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:49 smithi085 conmon[35961]: cluster 2021-02-10T04:50:47.658736+0000 osd.4 (osd.4) 2476 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:49.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:49 smithi085 conmon[35961]: cluster 2021-02-10T04:50:47.658746+0000 osd.4 (osd.4) 2477 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:49.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:49 smithi085 conmon[35961]: cluster 2021-02-10T04:50:47.658757+0000 osd.4 (osd.4) 2478 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:49.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:49 smithi085 conmon[35961]: cluster 2021-02-10T04:50:47.658764+0000 osd.4 (osd.4) 2479 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:49.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:49 smithi085 conmon[35961]: cluster 2021-02-10T04:50:47.662371+0000 osd.4 (osd.4) 2480 : cluster [DBG] 4.d scrub ok 2021-02-10T04:50:49.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:49 smithi085 conmon[35961]: cluster 2021-02-10T04:50:48.408744+0000 osd.5 (osd.5) 148 : cluster [DBG] 6.a scrub ok 2021-02-10T04:50:49.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:49 smithi085 conmon[35961]: cluster 2021-02-10T04:50:48.463424+0000 mon.a (mon.0) 2278 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 377 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:50:49.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:49 smithi085 conmon[29319]: cluster 2021-02-10T04:50:47.481679 2021-02-10T04:50:49.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:49 smithi085 conmon[29319]: +0000 osd.0 (osd.0) 156 : cluster [DBG] 2.21 scrub ok 2021-02-10T04:50:49.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:49 smithi085 conmon[29319]: cluster 2021-02-10T04:50:47.658736+0000 osd.4 (osd.4) 2476 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:49.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:49 smithi085 conmon[29319]: cluster 2021-02-10T04:50:47.658746+0000 osd.4 (osd.4) 2477 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:49.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:49 smithi085 conmon[29319]: cluster 2021-02-10T04:50:47.658757+0000 osd.4 (osd.4) 2478 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:49.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:49 smithi085 conmon[29319]: cluster 2021-02-10T04:50:47.658764+0000 osd.4 (osd.4) 2479 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:49.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:49 smithi085 conmon[29319]: cluster 2021-02-10T04:50:47.662371+0000 osd.4 (osd.4) 2480 : cluster [DBG] 4.d scrub ok 2021-02-10T04:50:49.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:49 smithi085 conmon[29319]: cluster 2021-02-10T04:50:48.408744+0000 osd.5 (osd.5) 148 : cluster [DBG] 6.a scrub ok 2021-02-10T04:50:49.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:49 smithi085 conmon[29319]: cluster 2021-02-10T04:50:48.463424+0000 mon.a (mon.0) 2278 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 377 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:50:50.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:49 smithi164 conmon[33677]: debug 2021-02-10T04:50:49.669+0000 7fbec19a1700 -1 osd.4 1107 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:50:50.529 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset nodeep-scrub 2021-02-10T04:50:50.556 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:50 smithi085 conmon[29319]: cluster 2021-02-10T04:50:48. 2021-02-10T04:50:50.556 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:50 smithi085 conmon[29319]: 404411+0000 mgr.y (mgr.14140) 1181 : cluster [DBG] pgmap v1972: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:50.556 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:50 smithi085 conmon[29319]: cluster 2021-02-10T04:50:48.664704+0000 osd.4 (osd.4) 2481 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:50.556 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:50 smithi085 conmon[29319]: cluster 2021-02-10T04:50:48.664727+0000 osd.4 (osd.4) 2482 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:50.556 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:50 smithi085 conmon[29319]: cluster 2021-02-10T04:50:48.664735+0000 osd.4 (osd.4) 2483 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:50.557 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:50 smithi085 conmon[29319]: cluster 2021-02-10T04:50:48.664740+0000 osd.4 (osd.4) 2484 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:50.557 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:50 smithi085 conmon[29319]: cluster 2021-02-10T04:50:48.668070+0000 osd.4 (osd.4) 2485 : cluster [DBG] 4.18 scrub ok 2021-02-10T04:50:50.557 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:50 smithi085 conmon[29319]: audit 2021-02-10T04:50:49.030010+0000 mgr.y (mgr.14140) 1182 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:50:50.557 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:50 smithi085 conmon[29319]: cluster 2021-02-10T04:50:49.391898+0000 osd.5 (osd.5) 149 : cluster [DBG] 6.1d scrub ok 2021-02-10T04:50:50.557 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:50 smithi085 conmon[35961]: cluster 2021-02-10T04:50:48.404411+0000 mgr.y ( 2021-02-10T04:50:50.557 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:50 smithi085 conmon[35961]: mgr.14140) 1181 : cluster [DBG] pgmap v1972: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:50.557 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:50 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:50:50.557 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:50 smithi085 conmon[35961]: 50:48.664704+0000 osd.4 (osd.4) 2481 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:50.557 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:50 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:50:50.557 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:50 smithi085 conmon[35961]: 50:48.664727+0000 osd.4 (osd.4) 2482 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:50.557 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:50 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:50:50.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:50 smithi085 conmon[35961]: :50:48.664735+0000 osd.4 (osd.4 2021-02-10T04:50:50.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:50 smithi085 conmon[35961]: ) 2483 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:50.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:50 smithi085 conmon[35961]: cluster 2021-02-10T04:50:48.664740+0000 osd.4 2021-02-10T04:50:50.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:50 smithi085 conmon[35961]: (osd.4) 2484 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:50.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:50 smithi085 conmon[35961]: cluster 2021-02-10T04:50:48.668070 2021-02-10T04:50:50.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:50 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2485 : cluster [DBG] 4.18 scrub ok 2021-02-10T04:50:50.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:50 smithi085 conmon[35961]: audit 2021-02-10T04:50:49.030010+0000 mgr.y (mgr.14140) 1182 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:50:50.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:50 smithi085 conmon[35961]: cluster 2021-02-10T04:50:49. 2021-02-10T04:50:50.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:50 smithi085 conmon[35961]: 391898+0000 osd.5 (osd.5) 149 2021-02-10T04:50:50.558 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:50 smithi085 conmon[35961]: : cluster [DBG] 6.1d scrub ok 2021-02-10T04:50:50.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: cluster 2021-02-10T04:50:50.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: 2021-02-10T04:50:50.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: 2021 2021-02-10T04:50:50.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: - 2021-02-10T04:50:50.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: 02 2021-02-10T04:50:50.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: - 2021-02-10T04:50:50.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: 10T04:50:48.404411+0000 2021-02-10T04:50:50.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: mgr.y (mgr.14140) 1181 : cluster [DBG] pgmap v1972: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:50.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:50:50.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: -02-10T04: 2021-02-10T04:50:50.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: 50:48.664704+0000 osd.4 (osd. 2021-02-10T04:50:50.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: 4) 2481 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:50.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: cluster 2021-02-10T04:50:50.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: 2021-02-10T04:50:48 2021-02-10T04:50:50.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: .664727+0000 2021-02-10T04:50:50.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: osd.4 (osd.4) 2482 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:50.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:50:50.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: 02-10T04:50:48.664735 2021-02-10T04:50:50.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: +0000 osd.4 2021-02-10T04:50:50.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: (osd.4) 2483 : cluster 2021-02-10T04:50:50.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:50.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: cluster 2021-02-10T04:50:50.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: 2021-02- 2021-02-10T04:50:50.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: 10T04:50:48.664740 2021-02-10T04:50:50.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2484 : cluster 2021-02-10T04:50:50.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:50.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:50:50.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: T04:50:48.668070+0000 osd.4 (osd. 2021-02-10T04:50:50.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: 4) 2485 : cluster [DBG] 4.18 scrub ok 2021-02-10T04:50:50.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:50:50.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: -10T04:50:49.030010+0000 2021-02-10T04:50:50.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: mgr.y (mgr.14140) 1182 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:50:50.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: 2021-02-10T04:50:50.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:50:50.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: 02-10T 2021-02-10T04:50:50.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: 04:50:49.391898+0000 2021-02-10T04:50:50.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: osd.5 (osd.5) 149 : 2021-02-10T04:50:50.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: cluster [DBG] 6.1d scrub ok 2021-02-10T04:50:50.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[29288]: 2021-02-10T04:50:50.837 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:50:50.839 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:50:50.863 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:50:50.881 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:50:50.905 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:50:50.931 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:50:50.956 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:50:50.982 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:50:50.982 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:50:50.982 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:50:50.982 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:50:50.982 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.3 2021-02-10T04:50:50.983 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: problem getting command descriptions from osd.3 2021-02-10T04:50:50.983 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:50:50.983 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:50:50.983 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:50:50.983 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:50:51.143 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[33677]: debug 2021-02-10T04:50:51.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[33677]: 2021-02-10T04:50:50.694+0000 7fbec19a1700 -1 osd.4 1107 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:50:51.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:50:50 smithi164 conmon[33677]: 2021-02-10T04:50:51.247 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[29319]: cluster 2021-02-10T04:50:49.447957+0000 osd.0 (osd.0) 2021-02-10T04:50:51.248 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[29319]: 157 : cluster [DBG] 2.4 scrub ok 2021-02-10T04:50:51.248 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[29319]: cluster 2021-02-10T04:50:49.669744+0000 osd.4 (osd.4) 2486 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:51.248 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[29319]: cluster 2021-02-10T04:50:49.669752+0000 osd.4 (osd.4) 2487 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:51.248 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:50:51.248 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[29319]: T04:50:49.669759+0000 osd.4 (osd.4) 2488 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:51.248 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:50:51.248 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[29319]: 04:50:49.669763+0000 osd.4 (osd.4) 2489 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:51.248 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:50:51.248 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[29319]: -10T04:50:50.399345+0000 2021-02-10T04:50:51.248 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[29319]: osd.0 (osd.0) 158 : cluster 2021-02-10T04:50:51.248 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[29319]: [DBG] 2.0 scrub ok 2021-02-10T04:50:51.248 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:50:51.249 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[29319]: 02-10T04:50:50 2021-02-10T04:50:51.249 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[29319]: .813010+0000 mon.c (mon.1) 87 2021-02-10T04:50:51.249 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[29319]: : audit [DBG] from='client.? 172.21.15.85:0/3430261834' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:50:51.249 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:51 smithi164 conmon[29288]: cluster 2021-02-10T04:50:49.447957+0000 osd.0 (osd.0) 157 : cluster 2021-02-10T04:50:51.249 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:51 smithi164 conmon[29288]: [DBG] 2.4 scrub ok 2021-02-10T04:50:51.249 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:51 smithi164 conmon[29288]: cluster 2021-02-10T04:50:49.669744+0000 osd.4 (osd.4) 2486 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:51.250 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:51 smithi164 conmon[29288]: cluster 2021-02-10T04:50:49.669752+0000 osd.4 2021-02-10T04:50:51.250 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:51 smithi164 conmon[29288]: (osd.4) 2487 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:51.250 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:51 smithi164 conmon[29288]: cluster 2021-02-10T04:50:49.669759+0000 osd.4 (osd.4) 2488 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:51.250 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:51 smithi164 conmon[29288]: 2021-02-10T04:50:51.250 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:51 smithi164 conmon[29288]: cluster 2021-02-10T04:50 2021-02-10T04:50:51.250 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[35961]: cluster 2021-02-10T04:50:49. 2021-02-10T04:50:51.250 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[35961]: 447957+0000 osd.0 (osd.0) 157 : cluster [DBG] 2.4 scrub ok 2021-02-10T04:50:51.250 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[35961]: cluster 2021-02-10T04:50:49.669744 2021-02-10T04:50:51.250 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2486 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:51.250 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:50:51.251 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[35961]: 50:49.669752+0000 osd.4 (osd.4) 2487 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:51.251 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[35961]: cluster 2021-02-10T04:50:51.251 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[35961]: 2021-02-10T04:50:49.669759+0000 2021-02-10T04:50:51.251 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[35961]: osd.4 (osd.4) 2488 : 2021-02-10T04:50:51.251 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:51.251 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:50:51.251 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[35961]: 02-10T04:50:49.669763+0000 2021-02-10T04:50:51.251 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[35961]: osd.4 (osd.4) 2489 : 2021-02-10T04:50:51.251 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:51.251 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:50:51.251 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[35961]: 02-10T04:50:50.399345 2021-02-10T04:50:51.251 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[35961]: +0000 osd.0 (osd.0) 2021-02-10T04:50:51.251 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[35961]: 158 : cluster [DBG] 2.0 scrub ok 2021-02-10T04:50:51.252 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:50:51.252 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[35961]: 02-10T04:50:50.813010 2021-02-10T04:50:51.252 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[35961]: +0000 mon.c (mon.1) 2021-02-10T04:50:51.252 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:51 smithi085 conmon[35961]: 87 : audit [DBG] from='client.? 172.21.15.85:0/3430261834' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:50:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:51 smithi164 conmon[29288]: :49.669763+0000 osd.4 (osd.4) 2489 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:51 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:50:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:51 smithi164 conmon[29288]: -10T04:50:50.399345+0000 osd.0 ( 2021-02-10T04:50:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:51 smithi164 conmon[29288]: osd.0) 158 : cluster [DBG] 2.0 scrub ok 2021-02-10T04:50:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:51 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:50:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:51 smithi164 conmon[29288]: -10T04:50:50.813010+0000 mon.c (mon 2021-02-10T04:50:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:51 smithi164 conmon[29288]: .1) 87 : audit [DBG] from='client.? 172.21.15.85:0/3430261834' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:50:52.253 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[29319]: cluster 2021-02-10T04:50:50.405192+0000 mgr.y (mgr.14140) 1183 2021-02-10T04:50:52.253 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[29319]: : cluster [DBG] pgmap v1973: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:52.253 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[29319]: cluster 2021-02-10T04:50:50.694728+0000 osd.4 (osd.4) 2490 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:52.253 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[29319]: cluster 2021-02-10T04:50:50.694752+0000 osd.4 (osd.4) 2491 : 2021-02-10T04:50:52.253 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:52.254 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[29319]: cluster 2021-02-10T04:50:50.694759+0000 osd.4 (osd.4) 2492 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:52.254 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[29319]: cluster 2021-02-10T04:50:50.694764+0000 osd.4 (osd.4) 2493 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:52.254 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[35961]: cluster 2021-02-10T04:50:50.405192 2021-02-10T04:50:52.254 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1183 : cluster [DBG] pgmap v1973: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:52.254 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[35961]: cluster 2021-02-10T04:50:50.694728+0000 2021-02-10T04:50:52.254 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[35961]: osd.4 (osd.4) 2490 : cluster [WRN] 2021-02-10T04:50:52.254 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:52.254 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[35961]: cluster 2021-02-10T04:50:50. 2021-02-10T04:50:52.255 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[35961]: 694752+0000 osd.4 (osd.4) 2491 : cluster 2021-02-10T04:50:52.255 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:52.255 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[35961]: cluster 2021-02-10T04:50:50 2021-02-10T04:50:52.255 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[35961]: .694759+0000 osd.4 (osd.4) 2492 : 2021-02-10T04:50:52.255 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:52.255 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:50:52.255 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[35961]: 50:50.694764+0000 osd.4 (osd.4) 2021-02-10T04:50:52.255 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[35961]: 2493 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:52 smithi164 conmon[29288]: cluster 2021-02-10T04:50:50.405192+0000 mgr.y 2021-02-10T04:50:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:52 smithi164 conmon[29288]: (mgr.14140) 1183 : cluster [DBG] pgmap v1973: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:52 smithi164 conmon[29288]: cluster 2021-02-10T04:50:50.694728+0000 osd.4 (osd.4) 2490 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:50:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:52 smithi164 conmon[29288]: 2021-02-10T04:50:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:52 smithi164 conmon[29288]: cluster 2021-02-10T04:50:50.694752+0000 2021-02-10T04:50:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:52 smithi164 conmon[29288]: osd.4 (osd.4) 2491 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:50:52.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:52 smithi164 conmon[29288]: cluster 2021-02-10T04:50:50.694759 2021-02-10T04:50:52.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:52 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2492 : cluster [WRN] 2021-02-10T04:50:52.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:52 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:50:52.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:52 smithi164 conmon[29288]: cluster 2021-02-10T04:50:52.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:52 smithi164 conmon[29288]: 2021-02-10T04:50:50.694764+0000 osd.4 (osd.4 2021-02-10T04:50:52.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:52 smithi164 conmon[29288]: ) 2493 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:50:52.998 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:52 smithi085 bash[146287]: 63f0ddb1ac253738c578074d3651b1a5c0ae11f7b9d0e5968d8e739f894fd01f 2021-02-10T04:50:52.999 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:52 smithi085 systemd[1]: Started Ceph osd.3 for 42224fce-6b57-11eb-8feb-001a4aab830c. 2021-02-10T04:50:53.044 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:50:53.096 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:50:53.274 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[147676]: debug 2021-02-10T04:50:52.994+0000 7fb04dd9a700 20 -- 172.21.15.85:0/2448712107 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x557f50edb000 msgr2=0x557f50e45e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2021-02-10T04:50:53.275 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[147676]: debug 2021-02-10T04:50:52.994+0000 7fb04dd9a700 20 -- 172.21.15.85:0/2448712107 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x557f50edb000 msgr2=0x557f50e45e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=96 2021-02-10T04:50:53.275 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[147676]: debug 2021-02-10T04:50:52.994+0000 7fb04dd9a700 20 --2- 172.21.15.85:0/2448712107 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x557f50edb000 0x557f50e45e00 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x557f50e5bd40 tx=0x557f50e8cd80).handle_read_frame_preamble_main r=0 2021-02-10T04:50:53.275 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[147676]: debug 2021-02-10T04:50:52.994+0000 7fb04dd9a700 20 --2- 172.21.15.85:0/2448712107 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x557f50edb000 0x557f50e45e00 secure :-1 s=THROTTLE_MESSAGE pgs=222 cs=0 l=1 rev1=1 rx=0x557f50e5bd40 tx=0x557f50e8cd80).throttle_message 2021-02-10T04:50:53.275 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[147676]: debug 2021-02-10T04:50:52.994+0000 7fb04dd9a700 20 --2- 172.21.15.85:0/2448712107 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x557f50edb000 0x557f50e45e00 secure :-1 s=THROTTLE_BYTES pgs=222 cs=0 l=1 rev1=1 rx=0x557f50e5bd40 tx=0x557f50e8cd80).throttle_bytes 2021-02-10T04:50:53.275 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[147676]: debug 2021-02-10T04:50:52.994+0000 7fb04dd9a700 20 --2- 172.21.15.85:0/2448712107 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x557f50edb000 0x557f50e45e00 secure :-1 s=THROTTLE_DISPATCH_QUEUE pgs=222 cs=0 l=1 rev1=1 rx=0x557f50e5bd40 tx=0x557f50e8cd80).throttle_dispatch_queue 2021-02-10T04:50:53.275 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[147676]: debug 2021-02-10T04:50:52.994+0000 7fb04dd9a700 20 --2- 172.21.15.85:0/2448712107 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x557f50edb000 0x557f50e45e00 secure :-1 s=THROTTLE_DONE pgs=222 cs=0 l=1 rev1=1 rx=0x557f50e5bd40 tx=0x557f50e8cd80).read_frame_segment seg_idx=0 2021-02-10T04:50:53.275 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[147676]: debug 2021-02-10T04:50:52.994+0000 7fb04dd9a700 20 --2- 172.21.15.85:0/2448712107 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x557f50edb000 0x557f50e45e00 secure :-1 s=THROTTLE_DONE pgs=222 cs=0 l=1 rev1=1 rx=0x557f50e5bd40 tx=0x557f50e8cd80).read_frame_segment seg_idx=1 2021-02-10T04:50:53.275 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[147676]: debug 2021-02-10T04:50:52.994+0000 7fb04dd9a700 20 -- 172.21.15.85:0/2448712107 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x557f50edb000 msgr2=0x557f50e45e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=208 2021-02-10T04:50:53.275 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[147676]: debug 2021-02-10T04:50:52.994+0000 7fb04dd9a700 20 --2- 172.21.15.85:0/2448712107 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x557f50edb000 0x557f50e45e00 secure :-1 s=THROTTLE_DONE pgs=222 cs=0 l=1 rev1=1 rx=0x557f50e5bd40 tx=0x557f50e8cd80).handle_read_frame_segment r=0 2021-02-10T04:50:53.275 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[147676]: debug 2021-02-10T04:50:52.994+0000 7fb04dd9a700 20 -- 172.21.15.85:0/2448712107 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x557f50edb000 msgr2=0x557f50e45e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=32 2021-02-10T04:50:53.276 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[147676]: debug 2021-02-10T04:50:52.994+0000 7fb04dd9a700 20 --2- 172.21.15.85:0/2448712107 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x557f50edb000 0x557f50e45e00 secure :-1 s=THROTTLE_DONE pgs=222 cs=0 l=1 rev1=1 rx=0x557f50e5bd40 tx=0x557f50e8cd80).handle_read_frame_epilogue_main r=0 2021-02-10T04:50:53.276 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[147676]: debug 2021-02-10T04:50:52.994+0000 7fb04dd9a700 10 --2- 172.21.15.85:0/2448712107 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x557f50edb000 0x557f50e45e00 secure :-1 s=THROTTLE_DONE pgs=222 cs=0 l=1 rev1=1 rx=0x557f50e5bd40 tx=0x557f50e8cd80).handle_read_frame_dispatch tag=17 2021-02-10T04:50:53.276 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[147676]: debug 2021-02-10T04:50:52.994+0000 7fb04dd9a700 20 --2- 172.21.15.85:0/2448712107 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x557f50edb000 0x557f50e45e00 secure :-1 s=THROTTLE_DONE pgs=222 cs=0 l=1 rev1=1 rx=0x557f50e5bd40 tx=0x557f50e8cd80).handle_message 2021-02-10T04:50:53.276 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[147676]: debug 2021-02-10T04:50:52.994+0000 7fb04dd9a700 5 --2- 172.21.15.85:0/2448712107 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x557f50edb000 0x557f50e45e00 secure :-1 s=THROTTLE_DONE pgs=222 cs=0 l=1 rev1=1 rx=0x557f50e5bd40 tx=0x557f50e8cd80).handle_message got 194 + 0 + 0 byte message. envelope type=18 src mon.1 off 0 2021-02-10T04:50:53.276 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[147676]: debug 2021-02-10T04:50:52.994+0000 7fb04dd9a700 5 --2- 172.21.15.85:0/2448712107 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x557f50edb000 0x557f50e45e00 secure :-1 s=READ_MESSAGE_COMPLETE pgs=222 cs=0 l=1 rev1=1 rx=0x557f50e5bd40 tx=0x557f50e8cd80).handle_message received message m=0x557f50e5e4e0 seq=4 from=mon.1 type=18 auth_reply(proto 2 0 (0) Success) v1 2021-02-10T04:50:53.276 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[147676]: debug 2021-02-10T04:50:52.994+0000 7fb04dd9a700 20 -- 172.21.15.85:0/2448712107 queue 0x557f50e5e4e0 prio 196 2021-02-10T04:50:53.276 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[147676]: debug 2021-02-10T04:50:52.994+0000 7fb04dd9a700 20 --2- 172.21.15.85:0/2448712107 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x557f50edb000 0x557f50e45e00 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x557f50e5bd40 tx=0x557f50e8cd80).read_frame 2021-02-10T04:50:53.276 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[147676]: debug 2021-02-10T04:50:52.994+0000 7fb04dd9a700 20 -- 172.21.15.85:0/2448712107 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x557f50edb000 msgr2=0x557f50e45e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=96 2021-02-10T04:50:53.276 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[147676]: debug 2021-02-10T04:50:52.994+0000 7fb04d599700 1 -- 172.21.15.85:0/2448712107 <== mon.1 v2:172.21.15.85:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557f50e5e4e0 con 0x557f50edb000 2021-02-10T04:50:53.276 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[147676]: debug 2021-02-10T04:50:52.994+0000 7fb04d599700 10 -- 172.21.15.85:0/2448712107 dispatch_throttle_release 194 to dispatch throttler 194/104857600 2021-02-10T04:50:53.276 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:52 smithi085 conmon[147676]: debug 2021-02-10T04:50:52.994+0000 7fb04d599700 20 -- 172.21.15.85:0/2448712107 done calling dispatch on 0x557f50e5e4e0 2021-02-10T04:50:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:53 smithi164 conmon[29288]: cluster 2021-02-10T04:50:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:53 smithi164 conmon[29288]: 2021 2021-02-10T04:50:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:53 smithi164 conmon[29288]: -02-10T04:50:52.405797+0000 mgr.y (mgr.14140) 1184 : cluster [DBG] pgmap v1974: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:53 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:50:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:53 smithi164 conmon[29288]: 04:50:53.251481+0000 mon.a (mon.0) 2279 2021-02-10T04:50:53.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:53 smithi164 conmon[29288]: : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 382 sec, osd.4 has slow ops) 2021-02-10T04:50:53.691 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:53 smithi085 conmon[147676]: debug 2021-02-10T04:50:53.514+0000 7fb055024200 -1 Falling back to public interface 2021-02-10T04:50:53.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:53 smithi085 conmon[29319]: cluster 2021-02-10T04:50:52.405797+0000 mgr.y 2021-02-10T04:50:53.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:53 smithi085 conmon[29319]: (mgr.14140) 1184 : cluster [DBG] pgmap v1974: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:53.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:53 smithi085 conmon[29319]: cluster 2021-02-10T04:50:53.251481+0000 mon.a (mon.0) 2279 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 382 sec, osd.4 has slow ops) 2021-02-10T04:50:53.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:53 smithi085 conmon[35961]: cluster 2021-02-10T04:50:52.405797 2021-02-10T04:50:53.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:53 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1184 : cluster [DBG] pgmap v1974: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:53.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:53 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:50:53.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:53 smithi085 conmon[35961]: 10T04:50:53.251481+0000 mon.a (mon.0) 2279 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 382 sec, osd.4 has slow ops) 2021-02-10T04:50:54.475 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.6 to 1 (10000) 2021-02-10T04:50:54.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:54 smithi164 conmon[29288]: cluster 2021-02-10T04:50:52. 2021-02-10T04:50:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:54 smithi164 conmon[29288]: 341524+0000 osd.5 (osd.5) 150 : cluster [DBG] 1.1 scrub ok 2021-02-10T04:50:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:54 smithi164 conmon[29288]: cluster 2021-02-10T04:50:52.421790+0000 osd.0 2021-02-10T04:50:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:54 smithi164 conmon[29288]: (osd.0) 159 : cluster [DBG] 5.b scrub ok 2021-02-10T04:50:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:54 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:50:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:54 smithi164 conmon[29288]: 02-10T04:50:52.896371+0000 osd.4 (osd.4) 2021-02-10T04:50:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:54 smithi164 conmon[29288]: 2494 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:50:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:54 smithi164 conmon[29288]: cluster 2021-02-10T04:50:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:54 smithi164 conmon[29288]: 2021-02-10 2021-02-10T04:50:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:54 smithi164 conmon[29288]: T04:50:52.897463+0000 osd.4 (osd.4) 2495 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:50:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:54 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:50:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:54 smithi164 conmon[29288]: -10T04:50:53.153243+0000 osd.6 2021-02-10T04:50:54.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:54 smithi164 conmon[29288]: (osd.6) 147 : cluster [DBG] 4.1f scrub ok 2021-02-10T04:50:54.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:54 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:50:54.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:54 smithi164 conmon[29288]: 02-10T04:50:53.327240+0000 mon.a (mon 2021-02-10T04:50:54.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:54 smithi164 conmon[29288]: .0) 2280 : cluster [DBG] osdmap e1108: 8 total, 7 up, 7 in 2021-02-10T04:50:54.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:54 smithi164 conmon[29288]: audit 2021-02-10T04:50:54.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:54 smithi164 conmon[29288]: 2021-02-10 2021-02-10T04:50:54.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:54 smithi164 conmon[29288]: T04:50:54.107949+0000 mon.a (mon.0) 2281 2021-02-10T04:50:54.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:54 smithi164 conmon[29288]: : audit [INF] from='client.? 172.21.15.85:0/3637513251' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T04:50:54.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[29319]: cluster 2021-02-10T04:50:52.341524+0000 osd.5 (osd. 2021-02-10T04:50:54.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[29319]: 5) 150 : cluster [DBG] 1.1 scrub ok 2021-02-10T04:50:54.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[29319]: cluster 2021-02-10T04:50:52.421790+0000 osd.0 (osd.0) 159 : cluster [DBG] 5.b scrub ok 2021-02-10T04:50:54.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[29319]: cluster 2021-02-10T04:50:52.896371+0000 osd.4 2021-02-10T04:50:54.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[29319]: (osd.4) 2494 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:50:54.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:50:54.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[29319]: -02-10T04:50:52.897463+0000 2021-02-10T04:50:54.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[29319]: osd.4 (osd.4) 2495 : cluster [DBG] 2021-02-10T04:50:54.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[29319]: purged_snaps scrub ok 2021-02-10T04:50:54.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[29319]: cluster 2021-02-10T04:50 2021-02-10T04:50:54.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[29319]: :53.153243+0000 osd.6 (osd.6) 2021-02-10T04:50:54.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[29319]: 147 : cluster [DBG] 4.1f scrub ok 2021-02-10T04:50:54.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:50:54.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[29319]: 02-10T04:50:53.327240+0000 2021-02-10T04:50:54.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[29319]: mon.a (mon.0) 2280 : cluster [DBG] 2021-02-10T04:50:54.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[29319]: osdmap e1108: 8 total, 7 up, 7 in 2021-02-10T04:50:54.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:50:54.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[29319]: :50:54.107949+0000 mon.a (mon.0 2021-02-10T04:50:54.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[29319]: ) 2281 : audit [INF] from='client.? 172.21.15.85:0/3637513251' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T04:50:54.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[35961]: cluster 2021-02-10T04:50:52. 2021-02-10T04:50:54.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[35961]: 341524+0000 osd.5 (osd.5) 150 : cluster [DBG] 1.1 scrub ok 2021-02-10T04:50:54.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[35961]: cluster 2021-02-10T04:50:52 2021-02-10T04:50:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[35961]: .421790+0000 osd.0 (osd.0) 159 : 2021-02-10T04:50:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[35961]: cluster [DBG] 5.b scrub ok 2021-02-10T04:50:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[35961]: cluster 2021-02-10T04:50:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[35961]: 2021-02-10T04:50 2021-02-10T04:50:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[35961]: :52.896371+0000 osd.4 (osd.4) 2494 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:50:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:50:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[35961]: 02-10T04:50:52.897463+0000 osd.4 (osd. 2021-02-10T04:50:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[35961]: 4) 2495 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:50:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:50:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[35961]: -02-10T04:50:53.153243+0000 2021-02-10T04:50:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[35961]: osd.6 (osd.6) 147 : cluster [DBG] 2021-02-10T04:50:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[35961]: 4.1f scrub ok 2021-02-10T04:50:54.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[35961]: cluster 2021-02-10T04:50 2021-02-10T04:50:54.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[35961]: :53.327240+0000 mon.a (mon.0) 2280 : 2021-02-10T04:50:54.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[35961]: cluster [DBG] osdmap e1108: 8 total, 7 up, 7 in 2021-02-10T04:50:54.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[35961]: audit 2021-02-10T04:50: 2021-02-10T04:50:54.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[35961]: 54.107949+0000 mon.a (mon.0) 2281 : audit [INF] 2021-02-10T04:50:54.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:54 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/3637513251' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T04:50:55.097 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:50:55.101 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:50:55.101 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:50:55.350 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:50:55.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:55 smithi164 conmon[29288]: cluster 2021-02-10T04:50:54.154428+0000 2021-02-10T04:50:55.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:55 smithi164 conmon[29288]: osd.6 (osd.6) 148 : cluster [DBG] 4.a scrub ok 2021-02-10T04:50:55.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:55 smithi164 conmon[29288]: cluster 2021-02-10T04:50:54.406479+0000 mgr.y (mgr.14140) 1185 2021-02-10T04:50:55.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:55 smithi164 conmon[29288]: : cluster [DBG] pgmap v1976: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:55.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:55 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:50:55.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:55 smithi164 conmon[29288]: 02-10T04:50:54.471419+0000 mon.a (mon.0) 2282 : 2021-02-10T04:50:55.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:55 smithi164 conmon[29288]: audit [INF] from='client.? 172.21.15.85:0/3637513251' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2021-02-10T04:50:55.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:55 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:50:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:55 smithi164 conmon[29288]: :50:54.471469+0000 mon.a (mon. 2021-02-10T04:50:55.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:55 smithi164 conmon[29288]: 0) 2283 : cluster [DBG] osdmap e1109: 8 total, 7 up, 7 in 2021-02-10T04:50:55.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:55 smithi085 conmon[29319]: cluster 2021-02-10T04:50:54.154428+0000 osd.6 (osd.6) 148 : cluster [DBG] 4.a scrub ok 2021-02-10T04:50:55.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:55 smithi085 conmon[29319]: cluster 2021-02-10T04:50:54.406479+0000 mgr.y (mgr.14140) 1185 : cluster [DBG] pgmap v1976: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:55.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:55 smithi085 conmon[29319]: audit 2021-02-10T04:50:54.471419+0000 mon.a (mon.0) 2282 : audit [INF] from='client.? 172.21.15.85:0/3637513251' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2021-02-10T04:50:55.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:55 smithi085 conmon[29319]: cluster 2021-02-10T04:50:54.471469+0000 mon.a (mon.0) 2283 : cluster [DBG] osdmap e1109: 8 total, 7 up, 7 in 2021-02-10T04:50:55.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:55 smithi085 conmon[35961]: cluster 2021-02-10T04:50:54.154428 2021-02-10T04:50:55.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:55 smithi085 conmon[35961]: +0000 osd.6 (osd.6) 148 : cluster [DBG] 4.a scrub ok 2021-02-10T04:50:55.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:55 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:50:55.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:55 smithi085 conmon[35961]: 50:54.406479+0000 mgr.y (mgr.14140) 1185 : cluster [DBG] pgmap v1976: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:55.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:55 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:50:55.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:55 smithi085 conmon[35961]: T04:50:54.471419+0000 mon.a (mon. 2021-02-10T04:50:55.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:55 smithi085 conmon[35961]: 0) 2282 : audit [INF] from='client.? 172.21.15.85:0/3637513251' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2021-02-10T04:50:55.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:55 smithi085 conmon[35961]: cluster 2021-02-10T04:50:55.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:55 smithi085 conmon[35961]: 2021-02-10T04:50: 2021-02-10T04:50:55.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:55 smithi085 conmon[35961]: 54.471469+0000 mon.a (mon.0) 2283 : 2021-02-10T04:50:55.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:55 smithi085 conmon[35961]: cluster [DBG] osdmap e1109: 8 total, 7 up, 7 in 2021-02-10T04:50:56.731 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:56 smithi085 conmon[29319]: cluster 2021-02-10T04:50:54.466255+0000 osd.0 ( 2021-02-10T04:50:56.731 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:56 smithi085 conmon[29319]: osd.0) 160 : cluster [DBG] 5.3 scrub ok 2021-02-10T04:50:56.731 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:56 smithi085 conmon[29319]: cluster 2021-02-10T04:50:55.142473+0000 osd.6 (osd.6) 149 : cluster [DBG] 4.12 scrub ok 2021-02-10T04:50:56.731 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:56 smithi085 conmon[29319]: cluster 2021-02-10T04:50:55.473341+0000 osd.5 (osd 2021-02-10T04:50:56.731 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:56 smithi085 conmon[29319]: .5) 151 : cluster [DBG] 5.7 scrub ok 2021-02-10T04:50:56.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:56 smithi085 conmon[35961]: cluster 2021-02-10T04:50:54 2021-02-10T04:50:56.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:56 smithi085 conmon[35961]: .466255+0000 osd.0 (osd.0) 160 : cluster [DBG] 5.3 scrub ok 2021-02-10T04:50:56.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:56 smithi085 conmon[35961]: cluster 2021-02-10T04:50:55 2021-02-10T04:50:56.732 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:56 smithi085 conmon[35961]: .142473+0000 osd.6 (osd.6) 149 : cluster [DBG] 2021-02-10T04:50:56.732 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:56 smithi085 conmon[35961]: 4.12 scrub ok 2021-02-10T04:50:56.732 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:56 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:50:56.732 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:56 smithi085 conmon[35961]: 50:55.473341+0000 osd.5 (osd.5) 151 2021-02-10T04:50:56.732 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:56 smithi085 conmon[35961]: : cluster [DBG] 5.7 scrub ok 2021-02-10T04:50:56.755 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 7 1 2021-02-10T04:50:56.801 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.2 dump_blocked_ops 2021-02-10T04:50:56.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:56 smithi164 conmon[29288]: cluster 2021-02-10T04:50:54.466255+0000 osd.0 2021-02-10T04:50:56.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:56 smithi164 conmon[29288]: (osd.0) 160 : cluster [DBG] 5.3 scrub ok 2021-02-10T04:50:56.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:56 smithi164 conmon[29288]: cluster 2021-02-10T04:50:55.142473+0000 osd.6 (osd.6) 149 : cluster [DBG] 4.12 scrub ok 2021-02-10T04:50:56.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:56 smithi164 conmon[29288]: cluster 2021-02-10T04:50:55.473341 2021-02-10T04:50:56.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:56 smithi164 conmon[29288]: +0000 osd.5 (osd.5) 151 : cluster [DBG] 5.7 scrub ok 2021-02-10T04:50:56.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:56 smithi164 conmon[29288]: 2021-02-10T04:50:57.309 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:50:57.311 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:50:57.691 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[147676]: debug 2021-02-10T04:50:57.449+0000 7fb055024200 -1 osd.3 1094 log_to_monitors {default=true} 2021-02-10T04:50:57.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[29319]: cluster 2021-02-10T04:50:56.099648+0000 osd.6 2021-02-10T04:50:57.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[29319]: (osd.6) 150 : cluster [DBG] 5.11 scrub ok 2021-02-10T04:50:57.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[29319]: cluster 2021-02-10T04:50:56.407170+0000 mgr.y (mgr.14140) 1186 : cluster [DBG] 2021-02-10T04:50:57.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[29319]: pgmap v1978: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:57.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:50:57.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[29319]: :50:57.455347+0000 mon.c (mon.1) 88 : audit [INF] 2021-02-10T04:50:57.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[29319]: from='osd.3 v2:172.21.15.85:6813/7' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-02-10T04:50:57.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[29319]: audit 2021-02-10T04:50:57.455713+0000 mon.a (mon 2021-02-10T04:50:57.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[29319]: .0) 2284 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-02-10T04:50:57.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[29319]: audit 2021 2021-02-10T04:50:57.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[29319]: -02-10T04:50:57.456577+0000 2021-02-10T04:50:57.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[29319]: mon.c (mon.1) 89 : audit [INF] 2021-02-10T04:50:57.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[29319]: from='osd.3 v2:172.21.15.85:6813/7' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:50:57.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[29319]: audit 2021-02-10T04:50: 2021-02-10T04:50:57.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[29319]: 57.456897+0000 mon.a (mon.0) 2285 : audit 2021-02-10T04:50:57.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[29319]: [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:50:57.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[35961]: cluster 2021-02-10T04:50:56.099648+0000 osd.6 (osd. 2021-02-10T04:50:57.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[35961]: 6) 150 : cluster [DBG] 5.11 scrub ok 2021-02-10T04:50:57.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[35961]: cluster 2021-02-10T04:50:56.407170+0000 mgr.y (mgr.14140) 1186 : cluster [DBG] pgmap v1978: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:57.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[35961]: audit 2021-02-10T04:50:57.455347+0000 mon.c (mon.1) 88 : audit [INF] 2021-02-10T04:50:57.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[35961]: from='osd.3 v2:172.21.15.85:6813/7' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-02-10T04:50:57.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:50:57.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[35961]: 02-10T04:50:57.455713+0000 mon.a (mon.0) 2284 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-02-10T04:50:57.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[35961]: 2021-02-10T04:50:57.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[35961]: audit 2021-02-10T04:50:57.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[35961]: 2021-02-10T04:50 2021-02-10T04:50:57.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[35961]: :57.456577+0000 mon.c ( 2021-02-10T04:50:57.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[35961]: mon.1) 89 : audit [INF] 2021-02-10T04:50:57.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[35961]: from='osd.3 v2:172.21.15.85:6813/7' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:50:57.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:50:57.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[35961]: 10T04:50:57.456897+0000 mon.a (mon.0) 2285 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:50:57.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:57 smithi085 conmon[35961]: 2021-02-10T04:50:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:57 smithi164 conmon[29288]: cluster 2021-02-10T04:50:56.099648+0000 2021-02-10T04:50:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:57 smithi164 conmon[29288]: osd.6 (osd.6) 150 : cluster [DBG] 5.11 scrub ok 2021-02-10T04:50:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:57 smithi164 conmon[29288]: cluster 2021-02-10T04:50:56.407170+0000 mgr.y (mgr.14140) 1186 : cluster [DBG] pgmap v1978: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:57 smithi164 conmon[29288]: audit 2021-02-10T04:50:57.455347 2021-02-10T04:50:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:57 smithi164 conmon[29288]: +0000 mon.c (mon.1) 88 : audit [INF] 2021-02-10T04:50:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:57 smithi164 conmon[29288]: from='osd.3 v2:172.21.15.85:6813/7' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-02-10T04:50:57.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:57 smithi164 conmon[29288]: audit 2021-02-10T04:50:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:57 smithi164 conmon[29288]: 2021-02-10T04:50:57.455713+0000 2021-02-10T04:50:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:57 smithi164 conmon[29288]: mon.a (mon.0) 2284 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-02-10T04:50:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:57 smithi164 conmon[29288]: 2021-02-10T04:50:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:57 smithi164 conmon[29288]: audit 2021-02-10T04:50:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:57 smithi164 conmon[29288]: 2021-02-10T04:50:57.456577+0000 mon.c ( 2021-02-10T04:50:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:57 smithi164 conmon[29288]: mon.1) 89 : audit [INF] from='osd.3 v2:172.21.15.85:6813/7' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:50:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:57 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:50:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:57 smithi164 conmon[29288]: -10T04:50:57.456897+0000 2021-02-10T04:50:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:57 smithi164 conmon[29288]: mon.a (mon.0) 2285 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:50:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:57 smithi164 conmon[29288]: 2021-02-10T04:50:58.772 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:58 smithi085 conmon[29319]: cluster 2021-02-10T04:50:56.688620+0000 osd.1 ( 2021-02-10T04:50:58.772 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:58 smithi085 conmon[29319]: osd.1) 479 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:50:58.772 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:58 smithi085 conmon[29319]: cluster 2021-02-10T04:50:56.691279+0000 osd.1 (osd.1) 480 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:50:58.772 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:58 smithi085 conmon[29319]: cluster 2021-02-10T04:50:57.483509+0000 mon.a (mon.0) 2286 : cluster [DBG] osdmap e1110: 8 total, 7 up, 7 in 2021-02-10T04:50:58.773 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:58 smithi085 conmon[29319]: cluster 2021-02-10T04:50:57.568538 2021-02-10T04:50:58.773 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:58 smithi085 conmon[29319]: +0000 osd.0 (osd.0) 2021-02-10T04:50:58.773 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:58 smithi085 conmon[29319]: 161 : cluster [DBG] 6.10 scrub ok 2021-02-10T04:50:58.773 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:58 smithi085 conmon[35961]: cluster 2021-02-10T04:50:56.688620+0000 2021-02-10T04:50:58.773 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:58 smithi085 conmon[35961]: osd.1 (osd.1) 479 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:50:58.773 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:58 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:50:58.773 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:58 smithi085 conmon[35961]: 04:50:56.691279+0000 osd.1 (osd.1) 480 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:50:58.773 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:58 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:50:58.773 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:58 smithi085 conmon[35961]: -02-10T04:50:57.483509+0000 2021-02-10T04:50:58.774 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:58 smithi085 conmon[35961]: mon.a (mon.0) 2286 : cluster [DBG] osdmap e1110: 8 total, 7 up, 7 in 2021-02-10T04:50:58.774 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:58 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:50:58.774 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:58 smithi085 conmon[35961]: 02-10T04:50:57.568538+0000 2021-02-10T04:50:58.774 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:58 smithi085 conmon[35961]: osd.0 (osd.0) 161 : cluster [DBG] 6.10 scrub ok 2021-02-10T04:50:58.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: cluster 2021-02-10T04:50:58.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: 2021-02-10T04:50:58.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: 2021 2021-02-10T04:50:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: - 2021-02-10T04:50:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: 02 2021-02-10T04:50:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: - 2021-02-10T04:50:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: 10 2021-02-10T04:50:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: T 2021-02-10T04:50:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: 04 2021-02-10T04:50:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: : 2021-02-10T04:50:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: 50 2021-02-10T04:50:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: : 2021-02-10T04:50:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: 56 2021-02-10T04:50:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: . 2021-02-10T04:50:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: 688620 2021-02-10T04:50:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: +0000 2021-02-10T04:50:58.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: 2021-02-10T04:50:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: osd.1 2021-02-10T04:50:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: ( 2021-02-10T04:50:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: osd 2021-02-10T04:50:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: . 2021-02-10T04:50:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: 1 2021-02-10T04:50:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: ) 2021-02-10T04:50:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: 479 2021-02-10T04:50:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: : 2021-02-10T04:50:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: cluster 2021-02-10T04:50:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: 2021-02-10T04:50:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: [DBG] 2021-02-10T04:50:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: 2021-02-10T04:50:58.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: purged_snaps scrub starts 2021-02-10T04:50:58.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: 2021-02-10T04:50:58.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: cluster 2021-02-10T04:50:56.691279+0000 2021-02-10T04:50:58.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: osd.1 (osd.1) 480 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:50:58.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:50:58.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: 04:50:57.483509+0000 mon.a (mon.0 2021-02-10T04:50:58.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: ) 2286 : cluster [DBG] osdmap e1110: 8 total, 7 up, 7 in 2021-02-10T04:50:58.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:50:58.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: 10T04:50:57.568538+0000 osd.0 (osd 2021-02-10T04:50:58.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:58 smithi164 conmon[29288]: .0) 161 : cluster [DBG] 6.10 scrub ok 2021-02-10T04:50:59.393 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:50:59.515 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is unset 2021-02-10T04:50:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[29319]: cluster 2021-02-10T04:50:58.142520 2021-02-10T04:50:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[29319]: +0000 osd.6 (osd.6) 151 : cluster [DBG] 5.2 scrub ok 2021-02-10T04:50:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[29319]: cluster 2021-02-10T04:50:58.407766+0000 mgr.y (mgr.14140) 1187 : cluster [DBG] pgmap v1980: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[29319]: cluster 2021-02-10T04:50:58.490743+0000 mon.a (mon.0) 2287 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2021-02-10T04:50:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[29319]: cluster 2021-02-10T04:50:58.496818+0000 mon.a (mon.0) 2288 : cluster [INF] osd.3 v2:172.21.15.85:6813/7 boot 2021-02-10T04:50:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[29319]: cluster 2021-02-10T04:50:58.496886+0000 mon.a (mon.0) 2289 : cluster [DBG] osdmap e1111: 8 total, 8 up, 7 in 2021-02-10T04:50:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[29319]: audit 2021-02-10T04:50:58.500741+0000 mon.a (mon.0) 2290 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-02-10T04:50:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[29319]: audit 2021-02-10T04:50:58.856699+0000 mon.c (mon.1) 90 : audit [INF] from='client.? 172.21.15.85:0/3396778516' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:50:59.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[29319]: audit 2021-02-10T04:50:58.857091+0000 mon.a (mon.0) 2291 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:50:59.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[29319]: audit 2021-02-10T04:50:59.040015+0000 mgr.y (mgr.14140) 1188 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:50:59.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[29319]: cluster 2021-02-10T04:50:59.491801+0000 mon.a (mon.0) 2292 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:50:59.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[29319]: audit 2021-02-10T04:50:59.498377+0000 mon.a (mon.0) 2293 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:50:59.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[29319]: cluster 2021-02-10T04:50:59.498502+0000 mon.a (mon.0) 2294 : cluster [DBG] osdmap e1112: 8 total, 8 up, 7 in 2021-02-10T04:50:59.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: cluster 2021-02-10T04:50:59.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: 2021-02-10T04:50:59.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: 2021 2021-02-10T04:50:59.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: - 2021-02-10T04:50:59.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: 02 2021-02-10T04:50:59.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: - 2021-02-10T04:50:59.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: 10 2021-02-10T04:50:59.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: T 2021-02-10T04:50:59.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: 04 2021-02-10T04:50:59.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: : 2021-02-10T04:50:59.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: 50 2021-02-10T04:50:59.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: : 2021-02-10T04:50:59.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: 58 2021-02-10T04:50:59.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: . 2021-02-10T04:50:59.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: 142520 2021-02-10T04:50:59.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: +0000 2021-02-10T04:50:59.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: 2021-02-10T04:50:59.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: osd.6 2021-02-10T04:50:59.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: ( 2021-02-10T04:50:59.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: osd 2021-02-10T04:50:59.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: . 2021-02-10T04:50:59.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: 6 2021-02-10T04:50:59.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: ) 2021-02-10T04:50:59.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: 151 : cluster [DBG] 5.2 scrub ok 2021-02-10T04:50:59.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: 2021-02-10T04:50:59.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:50:59.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: :50:58.407766+0000 mgr.y (mgr. 2021-02-10T04:50:59.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: 14140) 1187 : cluster [DBG] pgmap v1980: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:59.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:50:59.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: 02-10T04:50:58.490743+0000 2021-02-10T04:50:59.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: mon.a (mon.0) 2287 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2021-02-10T04:50:59.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: 2021-02-10T04:50:59.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: cluster 2021-02-10T04:50:58.496818 2021-02-10T04:50:59.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2288 : cluster [INF] osd.3 v2:172.21.15.85:6813/7 boot 2021-02-10T04:50:59.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: 2021-02-10T04:50:59.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: cluster 2021-02-10T04:50:58 2021-02-10T04:50:59.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: .496886+0000 mon.a (mon.0) 2289 : 2021-02-10T04:50:59.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: cluster [DBG] osdmap e1111: 8 total, 8 up, 7 in 2021-02-10T04:50:59.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:50:59.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: 04:50:58.500741+0000 mon.a (mon. 2021-02-10T04:50:59.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: 0) 2290 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-02-10T04:50:59.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:50:59.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: -10T04:50:58.856699+0000 mon.c 2021-02-10T04:50:59.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: (mon.1) 90 : audit [INF] from='client.? 172.21.15.85:0/3396778516' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:50:59.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:50:59.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: 02-10T04:50:58.857091+0000 2021-02-10T04:50:59.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: mon.a (mon.0) 2291 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:50:59.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: 2021-02-10T04:50:59.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: audit 2021-02-10T04:50:59.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: 2021-02-10T04:50:59.040015+0000 mgr.y ( 2021-02-10T04:50:59.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: mgr.14140) 1188 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:50:59.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:50:59.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: -02-10T04:50:59.491801+0000 2021-02-10T04:50:59.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: mon.a (mon.0) 2292 : cluster [INF] 2021-02-10T04:50:59.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:50:59.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: audit 2021-02-10T04:50:59 2021-02-10T04:50:59.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: .498377+0000 mon.a (mon.0) 2293 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:50:59.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:50:59.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: -10T04:50:59.498502+0000 mon.a 2021-02-10T04:50:59.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: (mon.0) 2294 : cluster [DBG] osdmap e1112: 8 total, 8 up, 7 in 2021-02-10T04:50:59.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:50:59 smithi085 conmon[35961]: 2021-02-10T04:50:59.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: cluster 2021-02-10T04:50:59.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: 2021-02-10T04:50:59.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: 2021 2021-02-10T04:50:59.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: - 2021-02-10T04:50:59.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: 02-10T04:50: 2021-02-10T04:50:59.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: 58.142520+0000 osd.6 (osd.6) 151 : cluster [DBG] 5.2 scrub ok 2021-02-10T04:50:59.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:50:59.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: -02-10T04:50:58.407766+0000 mgr.y (mgr.14140 2021-02-10T04:50:59.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: ) 1187 : cluster [DBG] pgmap v1980: 188 pgs: 32 active+undersized+degraded, 54 active+undersized, 102 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 381/2229 objects degraded (17.093%) 2021-02-10T04:50:59.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: cluster 2021-02-10T04:50:59.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: 2021-02-10T04:50 2021-02-10T04:50:59.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: :58.490743+0000 mon.a (mon.0) 2287 2021-02-10T04:50:59.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2021-02-10T04:50:59.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: cluster 2021-02-10T04:50:59.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: 2021-02-10T04:50 2021-02-10T04:50:59.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: :58.496818+0000 mon.a (mon.0) 2288 2021-02-10T04:50:59.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: : cluster [INF] osd.3 v2:172.21.15.85:6813/7 boot 2021-02-10T04:50:59.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: cluster 2021-02-10T04:50:59.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:50:59.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: 50:58.496886+0000 mon.a (mon.0) 2021-02-10T04:50:59.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: 2289 : cluster [DBG] osdmap e1111: 8 total, 8 up, 7 in 2021-02-10T04:50:59.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: audit 2021-02-10T04:50:59.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: 2021-02-10T04:50 2021-02-10T04:50:59.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: :58.500741+0000 mon.a (mon.0) 2021-02-10T04:50:59.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: 2290 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-02-10T04:50:59.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: audit 2021-02-10T04:50:59.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:50:59.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: 50:58.856699+0000 mon.c (mon.1) 2021-02-10T04:50:59.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: 90 : audit [INF] from='client.? 172.21.15.85:0/3396778516' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:50:59.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: audit 2021-02-10T04:50:59.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:50:59.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: 04:50:58.857091+0000 mon.a (mon. 2021-02-10T04:50:59.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: 0) 2291 : audit 2021-02-10T04:50:59.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:50:59.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:50:59.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: -10T04:50:59.040015+0000 mgr.y (mgr.14140) 1188 : audit 2021-02-10T04:50:59.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:50:59.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: cluster 2021-02-10T04:50:59.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: 2021-02-10T04:50:59. 2021-02-10T04:50:59.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: 491801+0000 mon.a (mon.0) 2292 : cluster 2021-02-10T04:50:59.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:50:59.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: audit 2021-02-10T04:50:59.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: 2021-02-10T04:50:59. 2021-02-10T04:50:59.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: 498377+0000 mon.a (mon.0) 2293 : audit 2021-02-10T04:50:59.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:50:59.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:50:59.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: 02-10T04:50:59.498502+0000 mon.a 2021-02-10T04:50:59.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: (mon.0) 2294 : cluster [DBG] osdmap e1112: 8 total, 8 up, 7 in 2021-02-10T04:50:59.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:50:59 smithi164 conmon[29288]: 2021-02-10T04:51:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:00 smithi164 conmon[29288]: cluster 2021-02-10T04:50:58.409336 2021-02-10T04:51:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:00 smithi164 conmon[29288]: +0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:51:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:00 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:51:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:00 smithi164 conmon[29288]: 02-10T04:50:58.418845+0000 osd.3 ( 2021-02-10T04:51:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:00 smithi164 conmon[29288]: osd.3) 2 : cluster [DBG] 2021-02-10T04:51:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:00 smithi164 conmon[29288]: purged_snaps scrub ok 2021-02-10T04:51:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:00 smithi164 conmon[29288]: cluster 2021-02-10T04:51:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:00 smithi164 conmon[29288]: 2021-02-10T04:50:59. 2021-02-10T04:51:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:00 smithi164 conmon[29288]: 167907+0000 osd.6 (osd.6) 152 : cluster [DBG] 2021-02-10T04:51:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:00 smithi164 conmon[29288]: 2.1d scrub ok 2021-02-10T04:51:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:00 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:51:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:00 smithi164 conmon[29288]: -10T04:50:59.824370 2021-02-10T04:51:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:00 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2496 : cluster [DBG] 2021-02-10T04:51:00.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:00 smithi164 conmon[29288]: 2.d scrub ok 2021-02-10T04:51:00.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:00 smithi085 conmon[29319]: cluster 2021-02-10T04:50:58.409336+0000 2021-02-10T04:51:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:00 smithi085 conmon[29319]: osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:51:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:00 smithi085 conmon[29319]: cluster 2021-02-10T04:50:58.418845+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:51:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:00 smithi085 conmon[29319]: cluster 2021-02-10T04:51:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:00 smithi085 conmon[29319]: 2021-02-10T04:50:59.167907+0000 osd.6 (osd.6) 152 : cluster 2021-02-10T04:51:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:00 smithi085 conmon[29319]: [DBG] 2.1d scrub ok 2021-02-10T04:51:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:00 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:51:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:00 smithi085 conmon[29319]: -10T04:50:59. 2021-02-10T04:51:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:00 smithi085 conmon[29319]: 824370+0000 osd.4 (osd.4) 2496 : cluster 2021-02-10T04:51:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:00 smithi085 conmon[29319]: [DBG] 2.d scrub ok 2021-02-10T04:51:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:00 smithi085 conmon[35961]: cluster 2021-02-10T04:50:58.409336+0000 2021-02-10T04:51:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:00 smithi085 conmon[35961]: osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:51:00.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:00 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:51:00.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:00 smithi085 conmon[35961]: T04:50:58.418845+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:51:00.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:00 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:51:00.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:00 smithi085 conmon[35961]: :50:59.167907+0000 osd.6 (osd.6) 152 : cluster 2021-02-10T04:51:00.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:00 smithi085 conmon[35961]: [DBG] 2.1d scrub ok 2021-02-10T04:51:00.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:00 smithi085 conmon[35961]: cluster 2021-02-10T04:50:59.824370+0000 osd.4 ( 2021-02-10T04:51:00.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:00 smithi085 conmon[35961]: osd.4) 2496 : cluster [DBG] 2.d scrub ok 2021-02-10T04:51:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:01 smithi164 conmon[29288]: cluster 2021-02-10T04:51:00.195514+0000 osd.6 (osd 2021-02-10T04:51:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:01 smithi164 conmon[29288]: .6) 153 : cluster [DBG] 1.6 deep-scrub ok 2021-02-10T04:51:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:01 smithi164 conmon[29288]: cluster 2021-02-10T04:51:00.408395+0000 mgr.y (mgr. 2021-02-10T04:51:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:01 smithi164 conmon[29288]: 14140) 1189 : cluster [DBG] pgmap v1983: 188 pgs: 11 peering, 26 active+undersized+degraded, 40 active+undersized, 111 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 862 B/s rd, 0 op/s; 260/2229 objects degraded (11.664%) 2021-02-10T04:51:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:01 smithi164 conmon[29288]: cluster 2021-02-10T04:51 2021-02-10T04:51:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:01 smithi164 conmon[29288]: :00.509637+0000 2021-02-10T04:51:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:01 smithi164 conmon[29288]: mon.a (mon.0) 2295 : cluster [WRN] Health check update: Degraded data redundancy: 260/2229 objects degraded (11.664%), 26 pgs degraded (PG_DEGRADED) 2021-02-10T04:51:01.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:01 smithi164 conmon[29288]: 2021-02-10T04:51:01.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:01 smithi085 conmon[29319]: cluster 2021-02-10T04:51:00.195514+0000 osd.6 (osd.6) 153 : cluster [DBG] 1.6 deep-scrub ok 2021-02-10T04:51:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:01 smithi085 conmon[29319]: cluster 2021-02-10T04:51:00.408395+0000 mgr.y (mgr.14140) 1189 : cluster [DBG] pgmap v1983: 188 pgs: 11 peering, 26 active+undersized+degraded, 40 active+undersized, 111 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 862 B/s rd, 0 op/s; 260/2229 objects degraded (11.664%) 2021-02-10T04:51:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:01 smithi085 conmon[29319]: cluster 2021-02-10T04:51:00.509637+0000 mon.a (mon.0) 2295 : cluster [WRN] Health check update: Degraded data redundancy: 260/2229 objects degraded (11.664%), 26 pgs degraded (PG_DEGRADED) 2021-02-10T04:51:01.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:01 smithi085 conmon[35961]: cluster 2021-02-10T04:51:00.195514+0000 osd.6 (osd.6) 153 : cluster [DBG] 1.6 deep-scrub ok 2021-02-10T04:51:01.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:01 smithi085 conmon[35961]: cluster 2021-02-10T04:51:00.408395+0000 mgr.y (mgr.14140) 1189 : cluster [DBG] pgmap v1983: 188 pgs: 11 peering, 26 active+undersized+degraded, 40 active+undersized, 111 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 862 B/s rd, 0 op/s; 260/2229 objects degraded (11.664%) 2021-02-10T04:51:01.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:01 smithi085 conmon[35961]: cluster 2021-02-10T04:51:00.509637+0000 mon.a (mon.0) 2295 : cluster [WRN] Health check update: Degraded data redundancy: 260/2229 objects degraded (11.664%), 26 pgs degraded (PG_DEGRADED) 2021-02-10T04:51:02.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:02 smithi164 conmon[29288]: cluster 2021-02-10T04:51:01.185816+0000 osd.6 (osd.6 2021-02-10T04:51:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:02 smithi164 conmon[29288]: ) 154 : cluster [DBG] 5.9 scrub ok 2021-02-10T04:51:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:02 smithi164 conmon[29288]: cluster 2021-02-10T04:51:01.450040+0000 osd.5 (osd.5) 152 : cluster [DBG] 4.15 deep-scrub ok 2021-02-10T04:51:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:02 smithi164 conmon[29288]: cluster 2021-02-10T04:51:01.466308+0000 2021-02-10T04:51:02.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:02 smithi164 conmon[29288]: osd.3 (osd.3) 3 : cluster [DBG] 7.a scrub ok 2021-02-10T04:51:02.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:02 smithi085 conmon[29319]: cluster 2021-02-10T04:51:01.185816+0000 osd.6 (osd.6) 154 : cluster 2021-02-10T04:51:02.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:02 smithi085 conmon[29319]: [DBG] 5.9 scrub ok 2021-02-10T04:51:02.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:02 smithi085 conmon[29319]: cluster 2021-02-10T04:51:01.450040+0000 osd.5 (osd.5) 152 : cluster [DBG] 4.15 deep-scrub ok 2021-02-10T04:51:02.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:02 smithi085 conmon[29319]: cluster 2021-02-10T04:51:01.466308+0000 osd.3 (osd.3) 3 : cluster [DBG] 7.a scrub ok 2021-02-10T04:51:02.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:02 smithi085 conmon[35961]: cluster 2021-02-10T04:51:01.185816+0000 2021-02-10T04:51:02.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:02 smithi085 conmon[35961]: osd.6 (osd.6) 154 : cluster [DBG] 5.9 scrub ok 2021-02-10T04:51:02.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:02 smithi085 conmon[35961]: cluster 2021-02-10T04:51:01.450040+0000 osd.5 (osd.5) 152 2021-02-10T04:51:02.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:02 smithi085 conmon[35961]: : cluster [DBG] 4.15 deep-scrub ok 2021-02-10T04:51:02.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:02 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:51:02.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:02 smithi085 conmon[35961]: T04:51:01.466308+0000 osd.3 (osd.3) 3 : cluster [DBG] 7.a scrub ok 2021-02-10T04:51:03.413 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:51:03.435 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:51:03.455 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:51:03.479 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:51:03.507 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:51:03.538 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:51:03.561 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:51:03.587 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:51:03.591 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:51:03.591 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:51:03.591 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:51:03.591 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:51:03.591 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:51:03.591 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:51:03.591 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:51:03.591 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:51:03.616 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:51:03.616 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [], 2021-02-10T04:51:03.616 INFO:teuthology.orchestra.run.smithi085.stdout: "complaint_time": 30, 2021-02-10T04:51:03.616 INFO:teuthology.orchestra.run.smithi085.stdout: "num_blocked_ops": 0 2021-02-10T04:51:03.616 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:51:03.864 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:03 smithi164 conmon[33677]: debug 2021-02-10T04:51:03.863+0000 7fbec19a1700 -1 osd.4 1112 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:03.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:03 smithi164 conmon[29288]: cluster 2021-02-10T04:51:02.394432+0000 osd.5 (osd.5) 153 : cluster 2021-02-10T04:51:03.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:03 smithi164 conmon[29288]: [DBG] 5.1b scrub ok 2021-02-10T04:51:03.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:03 smithi164 conmon[29288]: cluster 2021-02-10T04:51:02.409024+0000 mgr.y (mgr.14140) 1190 : cluster [DBG] pgmap v1984: 188 pgs: 11 peering, 22 active+undersized+degraded, 33 active+undersized, 122 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 237/2229 objects degraded (10.633%) 2021-02-10T04:51:03.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:03 smithi164 conmon[29288]: cluster 2021-02-10T04:51:02.455695+0000 osd.3 (osd.3) 4 : cluster [DBG] 2.1b deep-scrub ok 2021-02-10T04:51:03.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:03 smithi164 conmon[29288]: cluster 2021-02-10T04:51:03.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:03 smithi164 conmon[29288]: 2021-02-10T04:51:02.860826+0000 osd.4 (osd.4) 2497 : cluster [DBG] 4.1a deep-scrub ok 2021-02-10T04:51:03.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:03 smithi164 conmon[29288]: audit 2021-02-10T04:51:03.390724+0000 mon.a (mon.0) 2296 2021-02-10T04:51:03.865 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:03 smithi164 conmon[29288]: : audit [DBG] from='client.? 172.21.15.85:0/3454094194' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:51:03.923 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:03 smithi085 conmon[29319]: cluster 2021-02-10T04:51:02.394432+0000 2021-02-10T04:51:03.923 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:03 smithi085 conmon[29319]: osd.5 (osd.5) 153 : cluster [DBG] 5.1b scrub ok 2021-02-10T04:51:03.923 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:03 smithi085 conmon[29319]: cluster 2021-02-10T04:51:02.409024+0000 mgr.y (mgr.14140) 1190 : cluster [DBG] pgmap v1984: 188 pgs: 11 peering, 22 active+undersized+degraded, 33 active+undersized, 122 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 237/2229 objects degraded (10.633%) 2021-02-10T04:51:03.924 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:03 smithi085 conmon[29319]: cluster 2021-02-10T04:51:02.455695+0000 osd.3 (osd.3) 4 : cluster [DBG] 2.1b deep-scrub ok 2021-02-10T04:51:03.924 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:03 smithi085 conmon[29319]: cluster 2021-02-10T04:51:02.860826+0000 osd.4 (osd.4) 2497 : cluster [DBG] 4.1a deep-scrub ok 2021-02-10T04:51:03.924 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:03 smithi085 conmon[29319]: audit 2021-02-10T04:51:03.390724+0000 mon.a (mon.0) 2296 : audit [DBG] from='client.? 172.21.15.85:0/3454094194' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:51:03.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:03 smithi085 conmon[35961]: cluster 2021-02-10T04:51:02. 2021-02-10T04:51:03.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:03 smithi085 conmon[35961]: 394432+0000 osd.5 (osd.5) 153 : cluster [DBG] 5.1b scrub ok 2021-02-10T04:51:03.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:03 smithi085 conmon[35961]: cluster 2021-02-10T04:51:03.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:03 smithi085 conmon[35961]: 2021-02-10T04:51:02.409024+0000 mgr.y (mgr.14140) 1190 2021-02-10T04:51:03.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:03 smithi085 conmon[35961]: : cluster [DBG] pgmap v1984: 188 pgs: 11 peering, 22 active+undersized+degraded, 33 active+undersized, 122 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 237/2229 objects degraded (10.633%) 2021-02-10T04:51:03.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:03 smithi085 conmon[35961]: cluster 2021-02-10T04:51: 2021-02-10T04:51:03.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:03 smithi085 conmon[35961]: 02.455695+0000 osd.3 (osd.3) 4 2021-02-10T04:51:03.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:03 smithi085 conmon[35961]: : cluster [DBG] 2.1b deep-scrub ok 2021-02-10T04:51:03.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:03 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:51:03.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:03 smithi085 conmon[35961]: 02-10T04:51:02.860826+0000 2021-02-10T04:51:03.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:03 smithi085 conmon[35961]: osd.4 (osd.4) 2497 : 2021-02-10T04:51:03.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:03 smithi085 conmon[35961]: cluster [DBG] 4.1a deep-scrub ok 2021-02-10T04:51:03.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:03 smithi085 conmon[35961]: audit 2021-02-10T04:51:03.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:03 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:51:03.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:03 smithi085 conmon[35961]: 51:03.390724+0000 mon.a ( 2021-02-10T04:51:03.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:03 smithi085 conmon[35961]: mon.0) 2296 : audit 2021-02-10T04:51:03.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:03 smithi085 conmon[35961]: [DBG] from='client.? 172.21.15.85:0/3454094194' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:51:04.579 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.7 to 1 (10000) 2021-02-10T04:51:04.598 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set noscrub 2021-02-10T04:51:04.828 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:51:04.876 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:04 smithi164 conmon[33677]: debug 2021-02-10T04:51:04.876 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:04 smithi164 conmon[33677]: 2021-02-10T04:51:04.873+0000 7fbec19a1700 -1 osd.4 1113 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:04.876 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:04 smithi164 conmon[33677]: 2021-02-10T04:51:04.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:04 smithi164 conmon[29288]: cluster 2021-02-10T04:51:04.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:04 smithi164 conmon[29288]: 2021-02-10T04:51:04.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:04 smithi164 conmon[29288]: 2021 2021-02-10T04:51:04.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:04 smithi164 conmon[29288]: -02-10T04:51:03.221245+0000 2021-02-10T04:51:04.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:04 smithi164 conmon[29288]: osd.6 (osd.6) 155 : cluster [DBG] 7.d scrub ok 2021-02-10T04:51:04.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:04 smithi164 conmon[29288]: audit 2021-02-10T04:51:03.624313 2021-02-10T04:51:04.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:04 smithi164 conmon[29288]: +0000 mon.c (mon.1) 91 : audit [INF] from='client.? 172.21.15.85:0/2662377754' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T04:51:04.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:04 smithi164 conmon[29288]: audit 2021-02-10T04:51: 2021-02-10T04:51:04.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:04 smithi164 conmon[29288]: 03.624694+0000 mon.a (mon.0) 2297 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T04:51:04.877 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:04 smithi164 conmon[29288]: cluster 2021-02-10T04:51:04.878 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:04 smithi164 conmon[29288]: 2021-02-10T04:51:03.863788+0000 osd.4 (osd.4) 2021-02-10T04:51:04.878 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:04 smithi164 conmon[29288]: 2498 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:04.878 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:04 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:51:04.878 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:04 smithi164 conmon[29288]: 02-10T04:51:03.863800+0000 osd.4 (osd 2021-02-10T04:51:04.878 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:04 smithi164 conmon[29288]: .4) 2499 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:04.878 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:04 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:51:04.878 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:04 smithi164 conmon[29288]: 02-10T04:51:03.863810+0000 osd.4 (osd.4) 2021-02-10T04:51:04.878 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:04 smithi164 conmon[29288]: 2500 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:04.878 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:04 smithi164 conmon[29288]: cluster 2021-02-10T04:51:03.863819+0000 2021-02-10T04:51:04.878 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:04 smithi164 conmon[29288]: osd.4 (osd.4) 2501 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:04.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[35961]: cluster 2021-02-10T04:51:03. 2021-02-10T04:51:04.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[35961]: 221245+0000 osd.6 (osd.6) 155 : cluster [DBG] 7.d scrub ok 2021-02-10T04:51:04.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[35961]: audit 2021 2021-02-10T04:51:04.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[35961]: -02-10T04:51:03.624313+0000 mon.c (mon.1) 91 : audit [INF] 2021-02-10T04:51:04.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/2662377754' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T04:51:04.896 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[35961]: audit 2021-02-10T04:51:03.624694 2021-02-10T04:51:04.896 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2297 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T04:51:04.896 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:51:04.896 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[35961]: -02-10T04:51: 2021-02-10T04:51:04.896 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[35961]: 03.863788+0000 osd.4 (osd.4) 2498 : cluster [WRN] 2021-02-10T04:51:04.896 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:04.896 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[35961]: cluster 2021-02-10T04:51:03. 2021-02-10T04:51:04.897 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[35961]: 863800+0000 osd.4 (osd.4) 2499 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:04.897 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:51:04.897 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[35961]: 51:03.863810+0000 osd.4 (osd.4) 2500 : 2021-02-10T04:51:04.897 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:04.897 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[35961]: cluster 2021-02-10T04:51: 2021-02-10T04:51:04.897 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[35961]: 03.863819+0000 osd.4 (osd.4) 2501 : cluster 2021-02-10T04:51:04.897 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:04.898 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[29319]: cluster 2021-02-10T04:51:04.898 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[29319]: 2021-02-10T04:51:04.898 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[29319]: 2021-02-10T04:51:03 2021-02-10T04:51:04.898 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[29319]: .221245+0000 osd.6 (osd.6) 155 : cluster [DBG] 7.d scrub ok 2021-02-10T04:51:04.898 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:51:04.898 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[29319]: 02-10T04:51:03.624313+0000 mon.c (mon. 2021-02-10T04:51:04.898 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[29319]: 1) 91 : audit [INF] from='client.? 172.21.15.85:0/2662377754' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T04:51:04.899 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:51:04.899 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[29319]: -10T04:51:03.624694+0000 2021-02-10T04:51:04.899 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[29319]: mon.a (mon.0) 2297 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T04:51:04.899 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[29319]: 2021-02-10T04:51:04.899 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:51:04.899 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[29319]: -02-10T04:51:03.863788+0000 osd.4 (osd 2021-02-10T04:51:04.899 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[29319]: .4) 2498 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:04.900 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[29319]: cluster 2021-02-10T04:51:03 2021-02-10T04:51:04.900 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[29319]: .863800+0000 osd.4 (osd.4) 2499 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:04.900 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[29319]: 2021-02-10T04:51:04.900 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:51:04.900 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[29319]: 10T04:51:03.863810+0000 osd.4 (osd.4 2021-02-10T04:51:04.900 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[29319]: ) 2500 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:04.900 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:51:04.901 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[29319]: 51:03.863819+0000 osd.4 (osd.4) 2501 : cluster [WRN] 2021-02-10T04:51:04.901 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:04 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:05.829 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:05 smithi164 conmon[33677]: debug 2021-02-10T04:51:05.828+0000 7fbec19a1700 -1 osd.4 1113 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:05.830 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:05 smithi164 conmon[29288]: cluster 2021-02-10T04:51:04.409680+0000 mgr.y (mgr.14140) 1191 : cluster 2021-02-10T04:51:05.830 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:05 smithi164 conmon[29288]: [DBG] pgmap v1985: 188 pgs: 42 active+clean+wait, 146 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s; 2.8 MiB/s, 8 objects/s recovering 2021-02-10T04:51:05.830 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:05 smithi164 conmon[29288]: cluster 2021-02-10T04:51:04.565092+0000 mon.a (mon.0) 2298 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 237/2229 objects degraded (10.633%), 22 pgs degraded) 2021-02-10T04:51:05.830 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:05 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:51:05.830 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:05 smithi164 conmon[29288]: T04:51:04.565116+0000 mon.a (mon.0) 2299 : cluster [INF] Cluster is now healthy 2021-02-10T04:51:05.830 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:05 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:51:05.831 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:05 smithi164 conmon[29288]: T04:51:04.573972+0000 mon.a (mon 2021-02-10T04:51:05.831 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:05 smithi164 conmon[29288]: .0) 2300 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-02-10T04:51:05.831 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:05 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:51:05.831 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:05 smithi164 conmon[29288]: -02-10T04:51:04.574043+0000 2021-02-10T04:51:05.831 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:05 smithi164 conmon[29288]: mon.a (mon.0) 2301 : cluster [DBG] 2021-02-10T04:51:05.831 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:05 smithi164 conmon[29288]: osdmap e1113: 8 total, 8 up, 7 in 2021-02-10T04:51:05.831 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:05 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:51:05.831 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:05 smithi164 conmon[29288]: 04:51:04.873992+0000 osd.4 (osd. 2021-02-10T04:51:05.831 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:05 smithi164 conmon[29288]: 4) 2502 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:05.831 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:05 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:51:05.831 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:05 smithi164 conmon[29288]: -02-10T04:51:04.874001+0000 2021-02-10T04:51:05.831 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:05 smithi164 conmon[29288]: osd.4 (osd.4) 2503 : cluster [WRN] 2021-02-10T04:51:05.832 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:05 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:05.832 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:05 smithi164 conmon[29288]: cluster 2021-02-10T04:51 2021-02-10T04:51:05.832 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:05 smithi164 conmon[29288]: :04.874007+0000 osd.4 (osd.4) 2021-02-10T04:51:05.832 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:05 smithi164 conmon[29288]: 2504 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:05.832 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:05 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:51:05.832 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:05 smithi164 conmon[29288]: T04:51:04.874014+0000 osd.4 ( 2021-02-10T04:51:05.832 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:05 smithi164 conmon[29288]: osd.4) 2505 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:05.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:05 smithi085 conmon[29319]: cluster 2021-02-10T04:51:04.409680+0000 mgr.y 2021-02-10T04:51:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:05 smithi085 conmon[29319]: (mgr.14140) 1191 : cluster [DBG] pgmap v1985: 188 pgs: 42 active+clean+wait, 146 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s; 2.8 MiB/s, 8 objects/s recovering 2021-02-10T04:51:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:05 smithi085 conmon[29319]: cluster 2021-02-10T04:51:04.565092+0000 mon.a (mon.0) 2298 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 237/2229 objects degraded (10.633%), 22 pgs degraded) 2021-02-10T04:51:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:05 smithi085 conmon[29319]: cluster 2021-02-10T04:51:04.565116+0000 mon.a (mon.0) 2299 : cluster [INF] Cluster is now healthy 2021-02-10T04:51:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:05 smithi085 conmon[29319]: audit 2021-02-10T04:51:04.573972+0000 mon.a (mon.0) 2300 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-02-10T04:51:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:05 smithi085 conmon[29319]: cluster 2021-02-10T04:51:04.574043+0000 mon.a (mon.0) 2301 : cluster [DBG] osdmap e1113: 8 total, 8 up, 7 in 2021-02-10T04:51:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:05 smithi085 conmon[29319]: cluster 2021-02-10T04:51:04.873992+0000 osd.4 (osd.4) 2502 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:05 smithi085 conmon[29319]: cluster 2021-02-10T04:51:04.874001+0000 osd.4 (osd.4) 2503 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:05 smithi085 conmon[29319]: cluster 2021-02-10T04:51:04.874007+0000 osd.4 (osd.4) 2504 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:05 smithi085 conmon[29319]: cluster 2021-02-10T04:51:04.874014+0000 osd.4 (osd.4) 2505 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:05.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:05 smithi085 conmon[35961]: cluster 2021-02-10T04:51:04 2021-02-10T04:51:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:05 smithi085 conmon[35961]: .409680+0000 mgr.y (mgr.14140) 1191 : cluster [DBG] pgmap v1985: 188 pgs: 42 active+clean+wait, 146 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s; 2.8 MiB/s, 8 objects/s recovering 2021-02-10T04:51:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:05 smithi085 conmon[35961]: cluster 2021-02-10T04:51:04.565092+0000 mon.a (mon.0) 2298 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 237/2229 objects degraded (10.633%), 22 pgs degraded) 2021-02-10T04:51:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:05 smithi085 conmon[35961]: cluster 2021-02-10T04:51:04.565116+0000 mon.a (mon.0) 2299 : cluster [INF] Cluster is now healthy 2021-02-10T04:51:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:05 smithi085 conmon[35961]: audit 2021-02-10T04:51:04.573972+0000 mon.a (mon.0) 2300 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-02-10T04:51:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:05 smithi085 conmon[35961]: cluster 2021-02-10T04:51:04.574043+0000 mon.a (mon.0) 2301 : cluster [DBG] osdmap e1113: 8 total, 8 up, 7 in 2021-02-10T04:51:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:05 smithi085 conmon[35961]: cluster 2021-02-10T04:51:04.873992+0000 osd.4 (osd.4) 2502 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:05 smithi085 conmon[35961]: cluster 2021-02-10T04:51:04.874001+0000 osd.4 (osd.4) 2503 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:05 smithi085 conmon[35961]: cluster 2021-02-10T04:51:04.874007+0000 osd.4 (osd.4) 2504 : 2021-02-10T04:51:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:05 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:05 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:51:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:05 smithi085 conmon[35961]: -02-10T04:51:04.874014+0000 2021-02-10T04:51:05.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:05 smithi085 conmon[35961]: osd.4 (osd.4) 2505 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:06.504 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.2 dump_historic_ops 2021-02-10T04:51:06.510 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2021-02-10T04:51:06.510 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:51:06.763 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:51:06.796 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:51:06.843 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:06 smithi164 conmon[29288]: cluster 2021-02-10T04:51:05.242684 2021-02-10T04:51:06.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:06 smithi164 conmon[29288]: +0000 osd.6 (osd.6) 156 : cluster [DBG] 1.3 scrub ok 2021-02-10T04:51:06.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:06 smithi164 conmon[29288]: cluster 2021-02-10T04:51:05. 2021-02-10T04:51:06.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:06 smithi164 conmon[29288]: 475138+0000 osd.5 (osd.5) 154 : cluster [DBG] 5.12 scrub ok 2021-02-10T04:51:06.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:06 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:51:06.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:06 smithi164 conmon[29288]: -02-10T04:51:05.828862 2021-02-10T04:51:06.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:06 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2021-02-10T04:51:06.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:06 smithi164 conmon[29288]: 2506 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:06.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:06 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:51:06.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:06 smithi164 conmon[29288]: -02-10T04:51:05 2021-02-10T04:51:06.844 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:06 smithi164 conmon[29288]: .828871+0000 osd.4 (osd. 2021-02-10T04:51:06.845 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:06 smithi164 conmon[29288]: 4) 2507 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:06.845 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:06 smithi164 conmon[29288]: 2021-02-10T04:51:06.845 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:06 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:51:06.845 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:06 smithi164 conmon[29288]: T04:51:05.828877+0000 osd.4 ( 2021-02-10T04:51:06.845 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:06 smithi164 conmon[29288]: osd.4) 2508 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:06.845 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:06 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:51:06.845 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:06 smithi164 conmon[29288]: -02-10T04:51:05 2021-02-10T04:51:06.845 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:06 smithi164 conmon[29288]: .828883+0000 osd.4 (osd.4 2021-02-10T04:51:06.845 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:06 smithi164 conmon[29288]: ) 2509 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:06.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:06 smithi085 conmon[29319]: cluster 2021-02-10T04:51:05.242684+0000 osd.6 (osd.6) 156 : cluster [DBG] 1.3 scrub ok 2021-02-10T04:51:06.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:06 smithi085 conmon[29319]: cluster 2021-02-10T04:51:05.475138+0000 osd.5 (osd.5) 154 : cluster [DBG] 5.12 scrub ok 2021-02-10T04:51:06.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:06 smithi085 conmon[29319]: cluster 2021-02-10T04:51:05.828862+0000 osd.4 (osd.4) 2506 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:06.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:06 smithi085 conmon[29319]: cluster 2021-02-10T04:51:05.828871+0000 osd.4 (osd.4) 2507 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:06.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:06 smithi085 conmon[29319]: cluster 2021-02-10T04:51:05.828877+0000 osd.4 (osd.4) 2508 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:06.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:06 smithi085 conmon[29319]: cluster 2021-02-10T04:51:05.828883+0000 osd.4 (osd.4) 2509 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:06.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:06 smithi085 conmon[35961]: cluster 2021-02-10T04:51:05.242684+0000 osd.6 (osd.6) 156 : cluster [DBG] 1.3 scrub ok 2021-02-10T04:51:06.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:06 smithi085 conmon[35961]: cluster 2021-02-10T04:51:05.475138+0000 osd.5 (osd.5) 154 : cluster [DBG] 5.12 scrub ok 2021-02-10T04:51:06.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:06 smithi085 conmon[35961]: cluster 2021-02-10T04:51:05.828862+0000 osd.4 (osd.4) 2506 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:06.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:06 smithi085 conmon[35961]: cluster 2021-02-10T04:51:05.828871+0000 osd.4 (osd.4) 2507 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:06.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:06 smithi085 conmon[35961]: cluster 2021-02-10T04:51:05.828877+0000 osd.4 (osd.4) 2508 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:06.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:06 smithi085 conmon[35961]: cluster 2021-02-10T04:51:05.828883+0000 osd.4 (osd.4) 2509 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:07.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:06 smithi164 conmon[33677]: debug 2021-02-10T04:51:06.842+0000 7fbec19a1700 -1 osd.4 1113 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:07.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:07 smithi085 conmon[35961]: cluster 2021-02-10T04:51:06.410149+0000 mgr.y ( 2021-02-10T04:51:07.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:07 smithi085 conmon[35961]: mgr.14140) 1192 : cluster [DBG] pgmap v1987: 188 pgs: 42 active+clean+wait, 146 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s; 2.4 MiB/s, 7 objects/s recovering 2021-02-10T04:51:07.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:07 smithi085 conmon[35961]: cluster 2021-02-10T04:51:06.843498+0000 osd.4 (osd.4) 2510 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:07.621 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:07 smithi085 conmon[35961]: cluster 2021-02-10T04:51:06.843506+0000 osd.4 (osd.4) 2511 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:07.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:07 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:51:07.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:07 smithi085 conmon[35961]: 02-10T04:51:06.843513+0000 osd.4 (osd 2021-02-10T04:51:07.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:07 smithi085 conmon[35961]: .4) 2512 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:07.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:07 smithi085 conmon[35961]: cluster 2021-02-10T04:51:07.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:07 smithi085 conmon[35961]: 2021-02-10T04:51:06.843519+0000 osd.4 ( 2021-02-10T04:51:07.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:07 smithi085 conmon[35961]: osd.4) 2513 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:07.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:07 smithi085 conmon[35961]: cluster 2021-02-10T04:51: 2021-02-10T04:51:07.622 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:07 smithi085 conmon[35961]: 06.846843+0000 osd.4 (osd.4) 2514 : cluster [DBG] 4.c scrub ok 2021-02-10T04:51:07.623 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:07 smithi085 conmon[29319]: cluster 2021-02-10T04:51:06.410149 2021-02-10T04:51:07.623 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:07 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 1192 : cluster [DBG] pgmap v1987: 188 pgs: 42 active+clean+wait, 146 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s; 2.4 MiB/s, 7 objects/s recovering 2021-02-10T04:51:07.623 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:07 smithi085 conmon[29319]: cluster 2021-02-10T04:51:06.843498+0000 osd.4 (osd.4) 2510 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:07.623 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:07 smithi085 conmon[29319]: cluster 2021-02-10T04:51:06.843506+0000 osd.4 (osd.4) 2511 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:07.623 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:07 smithi085 conmon[29319]: cluster 2021-02-10T04:51:06.843513+0000 osd.4 (osd.4) 2512 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:07.623 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:07 smithi085 conmon[29319]: cluster 2021-02-10T04:51:06.843519+0000 osd.4 (osd.4) 2513 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:07.623 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:07 smithi085 conmon[29319]: cluster 2021-02-10T04:51:06.846843+0000 osd.4 (osd.4) 2514 : cluster [DBG] 4.c scrub ok 2021-02-10T04:51:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:07 smithi164 conmon[29288]: cluster 2021-02-10T04:51:06.410149+0000 2021-02-10T04:51:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:07 smithi164 conmon[29288]: mgr.y (mgr.14140) 1192 : cluster [DBG] pgmap v1987: 188 pgs: 42 active+clean+wait, 146 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s; 2.4 MiB/s, 7 objects/s recovering 2021-02-10T04:51:07.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:07 smithi164 conmon[29288]: cluster 2021-02-10T04:51:06.843498+0000 osd.4 (osd.4) 2510 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:07 smithi164 conmon[29288]: cluster 2021-02-10T04:51:06.843506+0000 osd.4 (osd.4) 2511 : cluster 2021-02-10T04:51:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:07 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:07 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:51:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:07 smithi164 conmon[29288]: 04:51:06.843513+0000 osd.4 (osd. 2021-02-10T04:51:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:07 smithi164 conmon[29288]: 4) 2512 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:07.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:07 smithi164 conmon[29288]: cluster 2021-02-10T04:51:07.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:07 smithi164 conmon[29288]: 2021-02-10T04:51:07.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:07 smithi164 conmon[29288]: 2021 2021-02-10T04:51:07.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:07 smithi164 conmon[29288]: - 2021-02-10T04:51:07.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:07 smithi164 conmon[29288]: 02 2021-02-10T04:51:07.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:07 smithi164 conmon[29288]: - 2021-02-10T04:51:07.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:07 smithi164 conmon[29288]: 10T04:51:06.843519 2021-02-10T04:51:07.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:07 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2513 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:07.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:07 smithi164 conmon[29288]: cluster 2021-02-10T04:51:07.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:07 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:51:07.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:07 smithi164 conmon[29288]: 04:51:06.846843+0000 osd.4 (osd.4) 2514 : cluster 2021-02-10T04:51:07.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:07 smithi164 conmon[29288]: [DBG] 4.c scrub ok 2021-02-10T04:51:07.898 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:07 smithi164 conmon[33677]: debug 2021-02-10T04:51:07.801+0000 7fbec19a1700 -1 osd.4 1113 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:08.458 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:51:08.717 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:51:08.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: cluster 2021-02-10T04:51:07.182672 2021-02-10T04:51:08.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: +0000 osd.6 (osd.6) 157 : cluster [DBG] 2.24 scrub ok 2021-02-10T04:51:08.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: cephadm 2021-02- 2021-02-10T04:51:08.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: 10T04:51:07.579610+0000 mgr.y (mgr.14140) 2021-02-10T04:51:08.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: 1193 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:51:08.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: cephadm 2021- 2021-02-10T04:51:08.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: 02-10T04:51:07.580084 2021-02-10T04:51:08.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 2021-02-10T04:51:08.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: 1194 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:51:08.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:51:08.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: -02-10T04:51:07 2021-02-10T04:51:08.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: .609494+0000 osd.0 (osd.0) 162 : cluster 2021-02-10T04:51:08.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: [DBG] 5.8 scrub ok 2021-02-10T04:51:08.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: cluster 2021-02-10T04:51:08.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:51:08.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: 51:07.802878+0000 osd.4 2021-02-10T04:51:08.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: (osd.4) 2515 : cluster 2021-02-10T04:51:08.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:08.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:51:08.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: -02-10T04:51: 2021-02-10T04:51:08.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: 07.802888+0000 osd.4 (osd. 2021-02-10T04:51:08.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: 4) 2516 : cluster [WRN] 2021-02-10T04:51:08.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:08.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: cluster 2021-02-10T04:51:08.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: 2021-02-10T04:51:07. 2021-02-10T04:51:08.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: 802894+0000 osd.4 (osd.4) 2021-02-10T04:51:08.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: 2517 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:08.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:51:08.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: -02-10T04:51:07.802900+0000 2021-02-10T04:51:08.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: osd.4 (osd.4) 2518 : cluster [WRN] 2021-02-10T04:51:08.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:08.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: cluster 2021-02-10T04:51: 2021-02-10T04:51:08.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: 08.433551+0000 osd.3 (osd.3) 5 : cluster [DBG] 4.1c scrub ok 2021-02-10T04:51:08.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[29288]: 2021-02-10T04:51:08.897 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:08 smithi164 conmon[33677]: debug 2021-02-10T04:51:08.754+0000 7fbec19a1700 -1 osd.4 1113 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[29319]: cluster 2021-02-10T04:51:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[29319]: 2021-02 2021-02-10T04:51:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[29319]: -10T04:51:07.182672+0000 osd.6 (osd.6) 157 : cluster [DBG] 2.24 scrub ok 2021-02-10T04:51:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[29319]: cephadm 2021-02 2021-02-10T04:51:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[29319]: -10T04:51:07.579610+0000 mgr.y (mgr.14140 2021-02-10T04:51:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[29319]: ) 1193 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:51:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[29319]: cephadm 2021-02- 2021-02-10T04:51:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[29319]: 10T04:51:07.580084+0000 mgr.y (mgr 2021-02-10T04:51:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[29319]: .14140) 1194 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:51:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:51:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[29319]: 02-10T04:51:07.609494+0000 osd.0 2021-02-10T04:51:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[29319]: (osd.0) 162 : cluster [DBG] 5.8 scrub ok 2021-02-10T04:51:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[29319]: 2021-02-10T04:51:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[29319]: cluster 2021-02-10T04:51:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[29319]: 2021-02-10T04:51:07.802878+0000 osd.4 ( 2021-02-10T04:51:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[29319]: osd.4) 2515 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:51:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[29319]: 02-10T04:51:07.802888+0000 2021-02-10T04:51:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[29319]: osd.4 (osd.4) 2516 : cluster [WRN] 2021-02-10T04:51:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[29319]: cluster 2021-02-10T04:51:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[29319]: 2021-02-10T04:51:07.802894+0000 osd.4 (osd 2021-02-10T04:51:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[29319]: .4) 2517 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:08.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:51:08.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[29319]: 02-10T04:51:07.802900+0000 2021-02-10T04:51:08.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[29319]: osd.4 (osd.4) 2518 : cluster [WRN] 2021-02-10T04:51:08.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:08.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[29319]: cluster 2021-02-10T04:51:08.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[29319]: 2021-02-10T04:51:08.433551+0000 osd.3 2021-02-10T04:51:08.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[29319]: (osd.3) 5 : cluster [DBG] 4.1c scrub ok 2021-02-10T04:51:08.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: cluster 2021-02-10T04:51:07.182672+0000 2021-02-10T04:51:08.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: osd.6 (osd.6) 157 : cluster [DBG] 2.24 scrub ok 2021-02-10T04:51:08.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: cephadm 2021-02-10T04:51:07.579610+0000 mgr.y ( 2021-02-10T04:51:08.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: mgr.14140) 1193 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:51:08.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: cephadm 2021- 2021-02-10T04:51:08.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: 02-10T04:51:07 2021-02-10T04:51:08.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: .580084+0000 mgr.y (mgr.14140) 2021-02-10T04:51:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: 1194 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:51:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: cluster 2021-02-10T04:51:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: 2021-02-10T04:51 2021-02-10T04:51:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: :07.609494+0000 osd.0 (osd 2021-02-10T04:51:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: .0) 162 : cluster [DBG] 5.8 scrub ok 2021-02-10T04:51:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: 2021-02-10T04:51:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: cluster 2021-02-10T04:51:07.802878 2021-02-10T04:51:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2515 : 2021-02-10T04:51:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:51:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: -02-10T04:51: 2021-02-10T04:51:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: 07.802888+0000 osd.4 (osd. 2021-02-10T04:51:08.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: 4) 2516 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: 2021-02-10T04:51:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:51:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: -10T04:51:07.802894 2021-02-10T04:51:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2517 2021-02-10T04:51:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: cluster 2021-02-10T04:51:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: 2021-02-10T04:51 2021-02-10T04:51:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: :07.802900+0000 osd.4 ( 2021-02-10T04:51:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: osd.4) 2518 : cluster 2021-02-10T04:51:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:51:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: -02-10T04:51 2021-02-10T04:51:08.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: :08.433551+0000 osd.3 2021-02-10T04:51:08.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: (osd.3) 5 : cluster 2021-02-10T04:51:08.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:08 smithi085 conmon[35961]: [DBG] 4.1c scrub ok 2021-02-10T04:51:09.394 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:51:09 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:51:09.895 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:09 smithi164 conmon[33677]: debug 2021-02-10T04:51:09.797+0000 7fbec19a1700 -1 osd.4 1114 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:09.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:09 smithi164 conmon[29288]: cluster 2021-02-10T04:51:08.220468+0000 osd.6 (osd.6) 158 : cluster [DBG] 7.c scrub ok 2021-02-10T04:51:09.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:09 smithi164 conmon[29288]: cluster 2021-02-10T04:51:08.410837+0000 mgr.y (mgr.14140) 1195 : cluster [DBG] pgmap v1988: 188 pgs: 42 active+clean+wait, 146 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 574 B/s rd, 0 op/s; 2.1 MiB/s, 6 objects/s recovering 2021-02-10T04:51:09.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:09 smithi164 conmon[29288]: cluster 2021-02-10T04:51:08.451285+0000 osd.5 (osd.5) 155 : cluster [DBG] 5.16 scrub ok 2021-02-10T04:51:09.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:09 smithi164 conmon[29288]: cluster 2021-02-10T04:51:08.580720+0000 mon.a (mon.0) 2302 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 402 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:51:09.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:09 smithi164 conmon[29288]: cluster 2021-02-10T04:51:08.639778+0000 osd.0 (osd.0) 163 : cluster [DBG] 2.10 scrub ok 2021-02-10T04:51:09.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:09 smithi164 conmon[29288]: cluster 2021-02-10T04:51:08.756085+0000 osd.4 (osd.4) 2519 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:09.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:09 smithi164 conmon[29288]: cluster 2021-02-10T04:51:08.756096+0000 osd.4 (osd.4) 2520 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:09.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:09 smithi164 conmon[29288]: cluster 2021-02-10T04:51:08.756103+0000 osd.4 (osd.4) 2521 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:09.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:09 smithi164 conmon[29288]: cluster 2021-02-10T04:51:08.756109+0000 osd.4 (osd.4) 2522 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:09.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:09 smithi164 conmon[29288]: cluster 2021-02-10T04:51:09.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:09 smithi164 conmon[29288]: 2021-02-10T04:51:08.758125+0000 osd.4 (osd.4) 2523 : cluster [DBG] 5.1d scrub ok 2021-02-10T04:51:09.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:09 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:51:09.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:09 smithi164 conmon[29288]: 02-10T04:51:09. 2021-02-10T04:51:09.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:09 smithi164 conmon[29288]: 050034+0000 mgr.y (mgr.14140) 2021-02-10T04:51:09.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:09 smithi164 conmon[29288]: 1196 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:51:09.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:09 smithi164 conmon[29288]: audit 2021 2021-02-10T04:51:09.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:09 smithi164 conmon[29288]: -02-10T04:51:09.349675+0000 mon.a (mon.0) 2303 2021-02-10T04:51:09.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:09 smithi164 conmon[29288]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:51:09.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[29319]: cluster 2021-02-10T04:51:08 2021-02-10T04:51:09.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[29319]: .220468+0000 osd.6 (osd.6) 158 : cluster [DBG] 7.c scrub ok 2021-02-10T04:51:09.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[29319]: cluster 2021-02-10T04:51:08.410837+0000 mgr.y (mgr.14140) 1195 : cluster [DBG] pgmap v1988: 188 pgs: 42 active+clean+wait, 146 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 574 B/s rd, 0 op/s; 2.1 MiB/s, 6 objects/s recovering 2021-02-10T04:51:09.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[29319]: cluster 2021-02-10T04:51:08.451285+0000 osd.5 (osd.5) 155 : cluster [DBG] 5.16 scrub ok 2021-02-10T04:51:09.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[29319]: cluster 2021-02-10T04:51:08.580720+0000 mon.a (mon.0) 2302 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 402 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:51:09.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[29319]: cluster 2021-02-10T04:51:08.639778+0000 osd.0 (osd.0) 163 : cluster 2021-02-10T04:51:09.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[29319]: [DBG] 2.10 scrub ok 2021-02-10T04:51:09.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[29319]: cluster 2021-02-10T04:51:08.756085+0000 osd.4 (osd.4) 2519 2021-02-10T04:51:09.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:09.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[29319]: cluster 2021-02-10T04:51:09.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[29319]: 2021-02-10T04:51:08.756096+0000 osd.4 (osd 2021-02-10T04:51:09.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[29319]: .4) 2520 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[29319]: cluster 2021-02-10T04:51:08.756103+0000 osd.4 (osd.4) 2521 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:51:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[29319]: -02-10T04:51:08.756109+0000 osd.4 (osd.4) 2522 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:51:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[29319]: 10 2021-02-10T04:51:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[29319]: T04: 2021-02-10T04:51:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[29319]: 51:08.758125+0000 osd.4 (osd.4) 2021-02-10T04:51:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[29319]: 2523 : cluster [DBG] 5.1d scrub ok 2021-02-10T04:51:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[29319]: audit 2021 2021-02-10T04:51:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[29319]: -02-10T04:51:09.050034+0000 mgr.y ( 2021-02-10T04:51:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[29319]: mgr.14140) 1196 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:51:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[29319]: audit 2021 2021-02-10T04:51:09.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[29319]: -02-10T04:51:09.349675+0000 2021-02-10T04:51:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[29319]: mon.a (mon.0) 2021-02-10T04:51:09.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[29319]: 2303 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:51:09.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: cluster 2021-02-10T04:51:09.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:51:09.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: 51:08.220468+0000 osd.6 (osd.6) 158 : cluster [DBG] 7.c scrub ok 2021-02-10T04:51:09.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:51:09.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: 02-10T04:51:08.410837+0000 mgr.y (mgr 2021-02-10T04:51:09.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: .14140) 1195 : cluster [DBG] pgmap v1988: 188 pgs: 42 active+clean+wait, 146 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 574 B/s rd, 0 op/s; 2.1 MiB/s, 6 objects/s recovering 2021-02-10T04:51:09.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:51:09.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: 02-10T04:51:08.451285 2021-02-10T04:51:09.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: +0000 osd.5 (osd.5) 155 2021-02-10T04:51:09.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: : cluster [DBG] 5.16 scrub ok 2021-02-10T04:51:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:51:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: -02-10T04:51:08.580720+0000 mon.a 2021-02-10T04:51:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: (mon.0) 2302 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 402 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:51:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:51:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: -02-10T04:51:08.639778 2021-02-10T04:51:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: +0000 osd.0 (osd.0) 163 : cluster [DBG] 2021-02-10T04:51:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: 2.10 scrub ok 2021-02-10T04:51:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: cluster 2021-02-10T04:51:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: 2021-02-10T04:51: 2021-02-10T04:51:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: 08.756085+0000 osd.4 (osd.4) 2519 : cluster 2021-02-10T04:51:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:51:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: 10T04:51:08.756096+0000 osd.4 (osd.4 2021-02-10T04:51:09.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: ) 2520 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:09.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:51:09.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: 10T04:51:08.756103+0000 osd.4 (osd. 2021-02-10T04:51:09.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: 4) 2521 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:09.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:51:09.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: 02-10T04:51:08.756109+0000 osd.4 2021-02-10T04:51:09.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: (osd.4) 2522 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:09.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:51:09.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: :51:08.758125+0000 osd.4 (osd.4) 2523 : 2021-02-10T04:51:09.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: cluster [DBG] 5.1d scrub ok 2021-02-10T04:51:09.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: audit 2021-02-10T04:51:09.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:51:09.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: 10T04:51:09.050034+0000 mgr.y (mgr.14140) 1196 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:51:09.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: 2021-02-10T04:51:09.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:51:09.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: 51:09.349675+0000 mon.a (mon.0 2021-02-10T04:51:09.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:09 smithi085 conmon[35961]: ) 2303 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:51:10.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:10 smithi164 conmon[33677]: debug 2021-02-10T04:51:10.818+0000 7fbec19a1700 -1 osd.4 1114 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:10.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:10 smithi164 conmon[29288]: cluster 2021-02-10T04:51:09.173445+0000 osd.5 2021-02-10T04:51:10.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:10 smithi164 conmon[29288]: (osd.5) 156 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:51:10.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:10 smithi164 conmon[29288]: cluster 2021-02-10T04:51:09.176608+0000 osd.5 (osd.5) 157 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:51:10.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:10 smithi164 conmon[29288]: cluster 2021-02-10T04:51:09.433787 2021-02-10T04:51:10.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:10 smithi164 conmon[29288]: +0000 osd.5 (osd.5) 158 : cluster [DBG] 6.19 deep-scrub ok 2021-02-10T04:51:10.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:10 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:51:10.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:10 smithi164 conmon[29288]: 04:51:09.475056+0000 osd.3 (osd.3) 2021-02-10T04:51:10.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:10 smithi164 conmon[29288]: 6 : cluster [DBG] 5.17 scrub ok 2021-02-10T04:51:10.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:10 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:51:10.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:10 smithi164 conmon[29288]: -10T04:51:09.599542+0000 mon.a ( 2021-02-10T04:51:10.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:10 smithi164 conmon[29288]: mon.0) 2304 : cluster [DBG] osdmap e1114: 8 total, 8 up, 7 in 2021-02-10T04:51:10.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:10 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:51:10.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:10 smithi164 conmon[29288]: 02-10T04:51:09.799275+0000 osd.4 2021-02-10T04:51:10.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:10 smithi164 conmon[29288]: (osd.4) 2524 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:10.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:10 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:51:10.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:10 smithi164 conmon[29288]: 02-10T04:51:09.799283+0000 osd.4 2021-02-10T04:51:10.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:10 smithi164 conmon[29288]: (osd.4) 2525 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:10.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:10 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:51:10.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:10 smithi164 conmon[29288]: -10T04:51:09.799289+0000 osd.4 ( 2021-02-10T04:51:10.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:10 smithi164 conmon[29288]: osd.4) 2526 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:10.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:10 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:51:10.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:10 smithi164 conmon[29288]: 02-10T04:51:09.799295+0000 osd.4 2021-02-10T04:51:10.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:10 smithi164 conmon[29288]: (osd.4) 2527 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:10.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[35961]: cluster 2021-02-10T04:51:09.173445+0000 osd.5 (osd.5) 156 : cluster 2021-02-10T04:51:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[35961]: [DBG] purged_snaps scrub starts 2021-02-10T04:51:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[35961]: cluster 2021-02-10T04:51:09.176608+0000 2021-02-10T04:51:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[35961]: osd.5 (osd.5) 157 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:51:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[35961]: cluster 2021-02-10T04:51 2021-02-10T04:51:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[35961]: :09.433787+0000 osd.5 (osd.5) 158 : cluster 2021-02-10T04:51:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[35961]: [DBG] 6.19 deep-scrub ok 2021-02-10T04:51:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[35961]: cluster 2021-02-10T04:51:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[35961]: 2021-02-10T04:51:09.475056+0000 osd.3 (osd.3) 6 2021-02-10T04:51:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[35961]: : cluster [DBG] 5.17 scrub ok 2021-02-10T04:51:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[35961]: cluster 2021-02-10T04:51: 2021-02-10T04:51:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[35961]: 09.599542+0000 mon.a (mon.0) 2304 : cluster 2021-02-10T04:51:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[35961]: [DBG] osdmap e1114: 8 total, 8 up, 7 in 2021-02-10T04:51:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:51:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[35961]: 10T04:51:09.799275+0000 osd.4 (osd.4) 2524 : 2021-02-10T04:51:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[35961]: cluster 2021-02-10T04:51:09. 2021-02-10T04:51:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[35961]: 799283+0000 osd.4 (osd.4) 2525 : cluster 2021-02-10T04:51:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[35961]: cluster 2021-02-10T04:51:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[35961]: 2021-02-10T04:51:09.799289+0000 osd.4 (osd. 2021-02-10T04:51:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[35961]: 4) 2526 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:51:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[35961]: :51:09.799295+0000 osd.4 (osd.4 2021-02-10T04:51:10.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[35961]: ) 2527 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[29319]: cluster 2021-02-10T04:51:09.173445+0000 osd.5 (osd.5) 156 2021-02-10T04:51:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[29319]: : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:51:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[29319]: cluster 2021-02-10T04:51:09.176608+0000 osd.5 (osd.5) 157 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:51:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:51:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[29319]: :51:09. 2021-02-10T04:51:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[29319]: 433787+0000 osd.5 (osd. 2021-02-10T04:51:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[29319]: 5) 158 : cluster [DBG] 6.19 deep-scrub ok 2021-02-10T04:51:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:51:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[29319]: -02-10T04:51:09.475056+0000 osd.3 (osd.3 2021-02-10T04:51:10.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[29319]: ) 6 : cluster [DBG] 5.17 scrub ok 2021-02-10T04:51:10.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[29319]: cluster 2021-02-10T04:51:10.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[29319]: 2021-02-10T04:51:09.599542+0000 mon.a (mon.0) 2021-02-10T04:51:10.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[29319]: 2304 : cluster [DBG] osdmap e1114: 8 total, 8 up, 7 in 2021-02-10T04:51:10.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[29319]: cluster 2021-02-10T04:51: 2021-02-10T04:51:10.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[29319]: 09.799275+0000 osd.4 (osd.4) 2524 : cluster 2021-02-10T04:51:10.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:10.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:51:10.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[29319]: 51:09.799283+0000 osd.4 (osd.4) 2021-02-10T04:51:10.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[29319]: 2525 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:10.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:51:10.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[29319]: 10T04:51:09.799289+0000 osd.4 (osd.4 2021-02-10T04:51:10.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[29319]: ) 2526 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:10.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:51:10.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[29319]: :51:09.799295+0000 osd.4 (osd.4) 2527 : cluster [WRN] 2021-02-10T04:51:10.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:10 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:11.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:11 smithi164 conmon[33677]: debug 2021-02-10T04:51:11.776+0000 7fbec19a1700 -1 osd.4 1114 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:11.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:11 smithi164 conmon[29288]: cluster 2021-02-10T04:51:10.398471 2021-02-10T04:51:11.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:11 smithi164 conmon[29288]: +0000 osd.5 (osd.5) 159 : cluster [DBG] 5.4 scrub ok 2021-02-10T04:51:11.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:11 smithi164 conmon[29288]: cluster 2021-02-10T04:51: 2021-02-10T04:51:11.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:11 smithi164 conmon[29288]: 10.411260+0000 mgr.y (mgr.14140) 1197 : cluster [DBG] pgmap v1990: 188 pgs: 42 active+clean+wait, 146 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2.4 MiB/s, 7 objects/s recovering 2021-02-10T04:51:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:11 smithi164 conmon[29288]: cluster 2021-02-10T04:51:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:11 smithi164 conmon[29288]: 2021-02-10T04:51: 2021-02-10T04:51:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:11 smithi164 conmon[29288]: 10.819691+0000 osd.4 (osd. 2021-02-10T04:51:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:11 smithi164 conmon[29288]: 4) 2528 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:11 smithi164 conmon[29288]: 2021-02-10T04:51:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:11 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:51:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:11 smithi164 conmon[29288]: -02-10T04:51:10.819701 2021-02-10T04:51:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:11 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2529 2021-02-10T04:51:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:11 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:11 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:51:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:11 smithi164 conmon[29288]: 02-10T04:51:10.819709+0000 osd.4 2021-02-10T04:51:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:11 smithi164 conmon[29288]: (osd.4) 2530 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:11 smithi164 conmon[29288]: cluster 2021-02-10T04:51:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:11 smithi164 conmon[29288]: 2021-02-10T04:51:10.819716+0000 2021-02-10T04:51:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:11 smithi164 conmon[29288]: osd.4 (osd.4) 2531 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:11 smithi164 conmon[29288]: 2021-02-10T04:51:11.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:11 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:51:11.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:11 smithi085 conmon[29319]: 10T04:51:10.398471+0000 osd.5 (osd.5) 159 : cluster [DBG] 5.4 scrub ok 2021-02-10T04:51:11.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:11 smithi085 conmon[29319]: cluster 2021-02-10T04:51:10.411260+0000 mgr.y (mgr.14140) 1197 : cluster [DBG] pgmap v1990: 188 pgs: 42 active+clean+wait, 146 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2.4 MiB/s, 7 objects/s recovering 2021-02-10T04:51:11.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:11 smithi085 conmon[29319]: cluster 2021-02-10T04:51:10.819691+0000 osd.4 (osd.4) 2528 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:11.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:11 smithi085 conmon[29319]: cluster 2021-02-10T04:51:10.819701+0000 osd.4 (osd.4) 2529 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:11.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:11 smithi085 conmon[29319]: cluster 2021-02-10T04:51:10.819709+0000 osd.4 (osd.4) 2530 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:11.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:11 smithi085 conmon[29319]: cluster 2021-02-10T04:51:10.819716+0000 osd.4 (osd.4) 2531 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:11 smithi085 conmon[35961]: cluster 2021-02-10T04:51: 2021-02-10T04:51:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:11 smithi085 conmon[35961]: 10.398471+0000 osd.5 (osd.5) 159 : cluster [DBG] 5.4 scrub ok 2021-02-10T04:51:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:11 smithi085 conmon[35961]: cluster 2021-02-10T04:51:10.411260+0000 mgr.y (mgr.14140) 1197 : cluster [DBG] pgmap v1990: 188 pgs: 42 active+clean+wait, 146 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2.4 MiB/s, 7 objects/s recovering 2021-02-10T04:51:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:11 smithi085 conmon[35961]: cluster 2021-02-10T04:51:10.819691+0000 osd.4 (osd.4) 2528 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:11 smithi085 conmon[35961]: cluster 2021-02-10T04:51:10.819701+0000 osd.4 (osd.4) 2529 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:11 smithi085 conmon[35961]: cluster 2021-02-10T04:51:10.819709+0000 osd.4 (osd.4) 2530 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:11.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:11 smithi085 conmon[35961]: cluster 2021-02-10T04:51:10.819716+0000 osd.4 (osd.4) 2531 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:12.610 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:12 smithi085 conmon[29319]: cluster 2021-02-10T04:51:11.778040+0000 osd.4 (osd 2021-02-10T04:51:12.610 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:12 smithi085 conmon[29319]: .4) 2532 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:12.610 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:12 smithi085 conmon[29319]: cluster 2021-02-10T04:51:11.778061+0000 osd.4 (osd.4) 2533 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:12.611 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:12 smithi085 conmon[29319]: cluster 2021-02-10T04:51:11.778072+0000 osd.4 (osd.4) 2534 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:12.611 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:12 smithi085 conmon[29319]: cluster 2021-02-10T04:51:11.778078+0000 osd.4 (osd.4) 2535 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:12.611 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:12 smithi085 conmon[29319]: cluster 2021-02-10T04:51:11.779773+0000 osd.4 (osd.4) 2536 : cluster [DBG] 6.17 scrub ok 2021-02-10T04:51:12.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:12 smithi164 conmon[33677]: debug 2021-02-10T04:51:12.768+0000 7fbec19a1700 -1 osd.4 1114 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:12.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:12 smithi164 conmon[29288]: cluster 2021-02-10T04:51:11.778040+0000 osd.4 ( 2021-02-10T04:51:12.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:12 smithi164 conmon[29288]: osd.4) 2532 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:12.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:12 smithi164 conmon[29288]: cluster 2021-02-10T04:51:11.778061+0000 osd.4 (osd.4) 2533 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:12.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:12 smithi164 conmon[29288]: cluster 2021-02-10T04:51:11.778072+0000 osd.4 (osd 2021-02-10T04:51:12.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:12 smithi164 conmon[29288]: .4) 2534 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:12.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:12 smithi164 conmon[29288]: cluster 2021-02-10T04:51:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:12 smithi164 conmon[29288]: 2021-02-10T04:51:11.778078+0000 2021-02-10T04:51:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:12 smithi164 conmon[29288]: osd.4 (osd.4) 2535 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:12 smithi164 conmon[29288]: 2021-02-10T04:51:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:12 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:51:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:12 smithi164 conmon[29288]: 10T04:51:11.779773+0000 osd.4 ( 2021-02-10T04:51:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:12 smithi164 conmon[29288]: osd.4) 2536 : cluster [DBG] 6.17 scrub ok 2021-02-10T04:51:12.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:12 smithi085 conmon[35961]: cluster 2021-02-10T04:51:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:12 smithi085 conmon[35961]: 2021-02-10T04:51:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:12 smithi085 conmon[35961]: 2021 2021-02-10T04:51:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:12 smithi085 conmon[35961]: - 2021-02-10T04:51:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:12 smithi085 conmon[35961]: 02 2021-02-10T04:51:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:12 smithi085 conmon[35961]: - 2021-02-10T04:51:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:12 smithi085 conmon[35961]: 10T04:51:11.778040+0000 osd.4 2021-02-10T04:51:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:12 smithi085 conmon[35961]: (osd.4) 2532 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:12 smithi085 conmon[35961]: cluster 2021-02-10T04:51:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:12 smithi085 conmon[35961]: 2021-02-10T04:51:11 2021-02-10T04:51:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:12 smithi085 conmon[35961]: .778061+0000 osd.4 (osd.4) 2533 : cluster 2021-02-10T04:51:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:12 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:12.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:12 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:51:12.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:12 smithi085 conmon[35961]: -02-10T04:51:11.778072+0000 2021-02-10T04:51:12.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:12 smithi085 conmon[35961]: osd.4 (osd.4) 2534 : cluster [WRN] 2021-02-10T04:51:12.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:12 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:12.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:12 smithi085 conmon[35961]: cluster 2021-02-10T04:51:12.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:12 smithi085 conmon[35961]: 2021-02-10T04:51:11.778078 2021-02-10T04:51:12.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:12 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2021-02-10T04:51:12.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:12 smithi085 conmon[35961]: 2535 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:12.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:12 smithi085 conmon[35961]: cluster 2021-02-10T04:51:12.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:12 smithi085 conmon[35961]: 2021-02-10T04:51:11.779773 2021-02-10T04:51:12.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:12 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2536 : cluster [DBG] 6.17 scrub ok 2021-02-10T04:51:13.288 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:51:13.288 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 20, 2021-02-10T04:51:13.288 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 600, 2021-02-10T04:51:13.288 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [] 2021-02-10T04:51:13.288 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:51:13.611 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[29319]: cluster 2021-02-10T04:51:13.611 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[29319]: 2021-02-10T04:51:12. 2021-02-10T04:51:13.611 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[29319]: 389265+0000 osd.5 (osd.5) 160 : cluster [DBG] 1.a scrub ok 2021-02-10T04:51:13.611 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[29319]: cluster 2021-02-10T04:51:12.411817+0000 mgr.y 2021-02-10T04:51:13.611 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[29319]: (mgr.14140) 1198 : cluster [DBG] pgmap v1991: 188 pgs: 42 active+clean+wait, 146 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:51:13.611 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:51:13.611 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[29319]: 10T04:51:12.770027+0000 osd.4 (osd.4) 2021-02-10T04:51:13.611 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[29319]: 2537 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:13.611 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:51:13.611 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[29319]: T04:51:12.770037+0000 osd.4 ( 2021-02-10T04:51:13.611 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[29319]: osd.4) 2538 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:13.612 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:51:13.612 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[29319]: 02-10T04:51:12.770046+0000 2021-02-10T04:51:13.612 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[29319]: osd.4 (osd.4) 2539 : cluster [WRN] 2021-02-10T04:51:13.612 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:13.612 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:51:13.612 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[29319]: T04:51:12.770051+0000 osd.4 ( 2021-02-10T04:51:13.612 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[29319]: osd.4) 2540 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:13.612 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:51:13.612 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[29319]: 02-10T04:51:13.032139+0000 2021-02-10T04:51:13.612 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[29319]: mon.c (mon.1) 92 : 2021-02-10T04:51:13.612 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[29319]: audit [INF] from='client.? 172.21.15.85:0/430899663' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:51:13.612 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[29319]: audit 2021 2021-02-10T04:51:13.612 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[29319]: -02-10T04:51 2021-02-10T04:51:13.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[29319]: :13.032470+0000 mon.a (mon 2021-02-10T04:51:13.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[29319]: .0) 2305 : audit [INF] 2021-02-10T04:51:13.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[29319]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:51:13.623 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is set 2021-02-10T04:51:13.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:13 smithi164 conmon[33677]: debug 2021-02-10T04:51:13.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:13 smithi164 conmon[33677]: 2021-02-10T04:51:13.760+0000 7fbec19a1700 -1 osd.4 1115 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:13.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:13 smithi164 conmon[33677]: 2021-02-10T04:51:13.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:13 smithi164 conmon[29288]: cluster 2021-02-10T04:51:13.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:13 smithi164 conmon[29288]: 2021-02-10T04:51:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:13 smithi164 conmon[29288]: 2021 2021-02-10T04:51:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:13 smithi164 conmon[29288]: -02-10T04:51:12.389265 2021-02-10T04:51:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:13 smithi164 conmon[29288]: +0000 osd.5 (osd.5) 160 : cluster [DBG] 1.a scrub ok 2021-02-10T04:51:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:13 smithi164 conmon[29288]: cluster 2021-02-10T04:51: 2021-02-10T04:51:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:13 smithi164 conmon[29288]: 12.411817+0000 mgr.y (mgr.14140) 1198 : cluster [DBG] pgmap v1991: 188 pgs: 42 active+clean+wait, 146 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:51:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:13 smithi164 conmon[29288]: cluster 2021-02-10T04:51:12. 2021-02-10T04:51:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:13 smithi164 conmon[29288]: 770027+0000 osd.4 (osd.4) 2537 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:13 smithi164 conmon[29288]: 2021-02-10T04:51:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:13 smithi164 conmon[29288]: cluster 2021-02-10T04:51:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:13 smithi164 conmon[29288]: 2021-02-10T04:51:12.770037+0000 osd.4 (osd 2021-02-10T04:51:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:13 smithi164 conmon[29288]: .4) 2538 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:13 smithi164 conmon[29288]: cluster 2021-02-10T04:51:13.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:13 smithi164 conmon[29288]: 2021-02-10T04:51:12.770046+0000 2021-02-10T04:51:13.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:13 smithi164 conmon[29288]: osd.4 (osd.4) 2539 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:13.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:13 smithi164 conmon[29288]: cluster 2021-02-10T04:51:13.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:13 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:51:13.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:13 smithi164 conmon[29288]: 51:12.770051+0000 osd.4 (osd.4) 2540 2021-02-10T04:51:13.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:13 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:13.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:13 smithi164 conmon[29288]: audit 2021-02-10T04:51:13.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:13 smithi164 conmon[29288]: 2021-02-10T04:51:13.032139+0000 2021-02-10T04:51:13.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:13 smithi164 conmon[29288]: mon.c (mon.1) 92 : audit [INF] from='client.? 172.21.15.85:0/430899663' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:51:13.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:13 smithi164 conmon[29288]: 2021-02-10T04:51:13.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:13 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:51:13.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:13 smithi164 conmon[29288]: -10T04:51:13.032470+0000 2021-02-10T04:51:13.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:13 smithi164 conmon[29288]: mon.a (mon.0) 2305 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:51:13.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[35961]: cluster 2021-02-10T04:51:12.389265+0000 osd.5 (osd 2021-02-10T04:51:13.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[35961]: .5) 160 : cluster [DBG] 1.a scrub ok 2021-02-10T04:51:13.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[35961]: cluster 2021-02-10T04:51:12.411817+0000 mgr.y (mgr 2021-02-10T04:51:13.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[35961]: .14140) 1198 : cluster [DBG] pgmap v1991: 188 pgs: 42 active+clean+wait, 146 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:51:13.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:51:13.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[35961]: 02-10T04:51:12.770027+0000 osd.4 (osd.4) 2537 : cluster 2021-02-10T04:51:13.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:13.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:51:13.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[35961]: -10T04:51:12.770037+0000 osd.4 (osd.4) 2538 : cluster [WRN] 2021-02-10T04:51:13.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[35961]: cluster 2021-02-10T04:51:12.770046 2021-02-10T04:51:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2539 : cluster [WRN] 2021-02-10T04:51:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[35961]: cluster 2021-02-10T04:51: 2021-02-10T04:51:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[35961]: 12.770051+0000 osd.4 (osd.4) 2540 : 2021-02-10T04:51:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[35961]: audit 2021-02-10T04:51:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[35961]: 2021-02-10T04:51 2021-02-10T04:51:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[35961]: :13.032139+0000 mon.c ( 2021-02-10T04:51:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[35961]: mon.1) 92 : audit [INF] 2021-02-10T04:51:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/430899663' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:51:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:51:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[35961]: 51:13.032470+0000 mon.a (mon.0) 2305 2021-02-10T04:51:13.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:13 smithi085 conmon[35961]: : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:51:14.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:14 smithi164 conmon[33677]: debug 2021-02-10T04:51:14.729+0000 7fbec19a1700 -1 osd.4 1115 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:14 smithi164 conmon[29288]: cluster 2021-02-10T04:51:13.607782+0000 mon.a 2021-02-10T04:51:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:14 smithi164 conmon[29288]: (mon.0) 2306 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:51:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:14 smithi164 conmon[29288]: audit 2021-02-10T04:51:13.619793+0000 mon.a (mon. 2021-02-10T04:51:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:14 smithi164 conmon[29288]: 0) 2307 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:51:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:14 smithi164 conmon[29288]: cluster 2021-02-10T04:51:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:14 smithi164 conmon[29288]: 2021-02-10T04:51:13.619894+0000 mon.a ( 2021-02-10T04:51:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:14 smithi164 conmon[29288]: mon.0) 2308 : cluster [DBG] 2021-02-10T04:51:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:14 smithi164 conmon[29288]: osdmap e1115: 8 total, 8 up, 7 in 2021-02-10T04:51:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:14 smithi164 conmon[29288]: cluster 2021-02-10T04:51:13.762551+0000 osd.4 (osd.4) 2541 : cluster 2021-02-10T04:51:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:14 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:14 smithi164 conmon[29288]: cluster 2021-02-10T04:51:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:14 smithi164 conmon[29288]: 2021-02-10T04:51:13.762559+0000 2021-02-10T04:51:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:14 smithi164 conmon[29288]: osd.4 (osd.4) 2021-02-10T04:51:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:14 smithi164 conmon[29288]: 2542 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:14 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:51:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:14 smithi164 conmon[29288]: 10T04:51:13.762566+0000 osd.4 (osd.4) 2543 : 2021-02-10T04:51:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:14 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:14 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:51:14.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:14 smithi164 conmon[29288]: 02-10T04:51:13.762570+0000 osd.4 (osd.4) 2544 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:14.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:14 smithi164 conmon[29288]: 2021-02-10T04:51:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[29319]: cluster 2021-02-10T04:51:13.607782+0000 2021-02-10T04:51:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[29319]: mon.a (mon.0) 2306 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:51:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[29319]: audit 2021-02-10T04:51:13.619793+0000 mon.a (mon.0) 2307 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:51:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[29319]: cluster 2021-02-10T04:51:13. 2021-02-10T04:51:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[29319]: 619894+0000 mon.a (mon.0) 2308 : cluster [DBG] osdmap e1115: 8 total, 8 up, 7 in 2021-02-10T04:51:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:51:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[29319]: -02-10T04:51:13.762551+0000 osd.4 ( 2021-02-10T04:51:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[29319]: osd.4) 2541 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:14.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[29319]: cluster 2021-02-10T04:51: 2021-02-10T04:51:14.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[29319]: 13.762559+0000 osd.4 (osd.4) 2542 2021-02-10T04:51:14.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:14.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:51:14.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[29319]: 02-10T04:51:13. 2021-02-10T04:51:14.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[29319]: 762566+0000 osd.4 (osd.4) 2543 2021-02-10T04:51:14.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:14.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:51:14.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[29319]: -02-10T04:51: 2021-02-10T04:51:14.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[29319]: 13.762570+0000 osd.4 (osd.4 2021-02-10T04:51:14.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[29319]: ) 2544 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:14.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[35961]: cluster 2021-02-10T04:51:13.607782 2021-02-10T04:51:14.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2306 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:51:14.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:51:14.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[35961]: 51:13.619793+0000 mon.a (mon.0) 2307 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:51:14.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:51:14.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[35961]: 02-10T04:51:13.619894+0000 mon.a (mon.0) 2308 : cluster [DBG] osdmap e1115: 8 total, 8 up, 7 in 2021-02-10T04:51:14.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[35961]: cluster 2021-02-10T04:51:13.762551+0000 osd.4 2021-02-10T04:51:14.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[35961]: (osd.4) 2541 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:14.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[35961]: cluster 2021-02-10T04:51:14.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[35961]: 2021-02-10T04:51:13.762559+0000 osd.4 (osd.4) 2542 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:14.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[35961]: 2021-02-10T04:51:14.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[35961]: cluster 2021-02-10T04:51:13.762566+0000 osd.4 (osd.4) 2543 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:14.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:51:14.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:14 smithi085 conmon[35961]: 10T04:51:13.762570+0000 osd.4 (osd.4) 2544 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:15.593 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.3 dump_ops_in_flight 2021-02-10T04:51:15.858 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:51:15.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:15 smithi164 conmon[33677]: debug 2021-02-10T04:51:15.757+0000 7fbec19a1700 -1 osd.4 1115 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:15.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:15 smithi164 conmon[29288]: cluster 2021-02-10T04:51:14.412403+0000 mgr.y (mgr.14140) 1199 2021-02-10T04:51:15.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:15 smithi164 conmon[29288]: : cluster [DBG] pgmap v1993: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:51:15.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:15 smithi164 conmon[29288]: cluster 2021-02-10T04:51:14.730801+0000 osd.4 (osd.4) 2545 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:15.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:15 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:51:15.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:15 smithi164 conmon[29288]: T04:51:14.730809+0000 osd.4 (osd.4) 2546 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:15.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:15 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:51:15.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:15 smithi164 conmon[29288]: 04:51:14.730813+0000 osd.4 (osd. 2021-02-10T04:51:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:15 smithi164 conmon[29288]: 4) 2547 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:15 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:51:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:15 smithi164 conmon[29288]: 02-10T04:51:14.730816+0000 2021-02-10T04:51:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:15 smithi164 conmon[29288]: osd.4 (osd.4) 2548 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:15 smithi164 conmon[29288]: 2021-02-10T04:51:15.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:15 smithi085 conmon[35961]: cluster 2021-02-10T04:51:14.412403 2021-02-10T04:51:15.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:15 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1199 : cluster [DBG] pgmap v1993: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:51:15.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:15 smithi085 conmon[35961]: cluster 2021-02-10T04:51:14.730801+0000 osd.4 (osd.4) 2545 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:15.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:15 smithi085 conmon[35961]: cluster 2021-02-10T04:51:14.730809+0000 osd.4 (osd.4) 2546 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:15.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:15 smithi085 conmon[35961]: cluster 2021-02-10T04:51:14.730813+0000 osd.4 (osd.4) 2547 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:15.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:15 smithi085 conmon[35961]: cluster 2021-02-10T04:51:14.730816+0000 osd.4 (osd.4) 2548 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:15.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:15 smithi085 conmon[29319]: cluster 2021-02-10T04:51:15.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:15 smithi085 conmon[29319]: 2021-02-10T04:51:15.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:15 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:51:15.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:15 smithi085 conmon[29319]: :51:14.412403+0000 mgr.y (mgr.14140) 1199 : cluster [DBG] pgmap v1993: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:51:15.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:15 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:51:15.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:15 smithi085 conmon[29319]: 02-10T04:51:14.730801 2021-02-10T04:51:15.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:15 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2545 : cluster [WRN] 2021-02-10T04:51:15.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:15 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:15.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:15 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:51:15.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:15 smithi085 conmon[29319]: 02-10T04:51:14.730809+0000 2021-02-10T04:51:15.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:15 smithi085 conmon[29319]: osd.4 (osd.4) 2546 : cluster [WRN] 2021-02-10T04:51:15.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:15 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:15.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:15 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:51:15.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:15 smithi085 conmon[29319]: :51:14.730813+0000 osd.4 (osd.4 2021-02-10T04:51:15.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:15 smithi085 conmon[29319]: ) 2547 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:15.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:15 smithi085 conmon[29319]: cluster 2021-02-10T04:51:15.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:15 smithi085 conmon[29319]: 2021-02-10T04:51:14.730816 2021-02-10T04:51:15.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:15 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2548 : cluster 2021-02-10T04:51:15.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:15 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:16.239 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:51:16.256 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:51:16.259 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:51:16.264 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:51:16.289 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:51:16.323 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:51:16.349 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:51:16.376 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:51:16.396 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:51:16.418 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:51:16.420 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:51:16.420 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:51:16.420 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:51:16.420 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:51:16.420 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:51:16.420 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:51:16.420 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:51:16.420 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:51:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: cluster 2021-02-10T04:51:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 2021-02-10T04:51:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 2021 2021-02-10T04:51:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: - 2021-02-10T04:51:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 02 2021-02-10T04:51:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: -10T04:51: 2021-02-10T04:51:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 15.759484+0000 osd.4 (osd.4) 2549 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: cluster 2021-02-10T04:51:16.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 2021-02-10T04:51:16.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 2021 2021-02-10T04:51:16.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: - 2021-02-10T04:51:16.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 02 2021-02-10T04:51:16.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: - 2021-02-10T04:51:16.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 10 2021-02-10T04:51:16.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: T 2021-02-10T04:51:16.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 04 2021-02-10T04:51:16.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: : 2021-02-10T04:51:16.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 51 2021-02-10T04:51:16.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: : 2021-02-10T04:51:16.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 15 2021-02-10T04:51:16.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: . 2021-02-10T04:51:16.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 759496 2021-02-10T04:51:16.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: +0000 2021-02-10T04:51:16.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 2021-02-10T04:51:16.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: osd.4 2021-02-10T04:51:16.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: ( 2021-02-10T04:51:16.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: osd 2021-02-10T04:51:16.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: . 2021-02-10T04:51:16.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 4 2021-02-10T04:51:16.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: ) 2021-02-10T04:51:16.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 2550 2021-02-10T04:51:16.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: : 2021-02-10T04:51:16.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: cluster 2021-02-10T04:51:16.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 2021-02-10T04:51:16.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: [WRN] 2021-02-10T04:51:16.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 2021-02-10T04:51:16.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:16.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 2021-02-10T04:51:16.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: cluster 2021-02-10T04:51:16.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 2021-02-10T04:51:16.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 2021 2021-02-10T04:51:16.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: - 2021-02-10T04:51:16.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 02 2021-02-10T04:51:16.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: - 2021-02-10T04:51:16.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 10 2021-02-10T04:51:16.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: T 2021-02-10T04:51:16.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 04 2021-02-10T04:51:16.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: : 2021-02-10T04:51:16.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 51 2021-02-10T04:51:16.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: : 2021-02-10T04:51:16.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 15 2021-02-10T04:51:16.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: . 2021-02-10T04:51:16.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 759502 2021-02-10T04:51:16.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: +0000 2021-02-10T04:51:16.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 2021-02-10T04:51:16.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: osd.4 2021-02-10T04:51:16.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: ( 2021-02-10T04:51:16.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: osd 2021-02-10T04:51:16.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: . 2021-02-10T04:51:16.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 4 2021-02-10T04:51:16.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: ) 2021-02-10T04:51:16.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 2551 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:16.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:51:16.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: :51:15.759509+0000 osd.4 (osd.4) 2552 : 2021-02-10T04:51:16.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:16.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:51:16.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: :51:16.215113+0000 mon.c 2021-02-10T04:51:16.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: (mon.1) 93 : audit 2021-02-10T04:51:16.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: [DBG] from='client.? 172.21.15.85:0/3680285070' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:51:16.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: audit 2021-02-10T04:51:16.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:51:16.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 51:16.235082+0000 mgr.y 2021-02-10T04:51:16.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: (mgr.14140) 1200 : audit [DBG] from='client.15624 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:51:16.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:16 smithi164 conmon[29288]: 2021-02-10T04:51:16.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:16 smithi085 conmon[29319]: cluster 2021-02-10T04:51:15.759484+0000 osd.4 (osd.4) 2549 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:16.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:16 smithi085 conmon[29319]: cluster 2021-02-10T04:51:15.759496+0000 osd.4 (osd.4) 2550 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:16.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:16 smithi085 conmon[29319]: cluster 2021-02-10T04:51:15.759502+0000 osd.4 (osd.4) 2551 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:16.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:16 smithi085 conmon[29319]: cluster 2021-02-10T04:51:15.759509+0000 osd.4 (osd.4) 2552 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:16.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:16 smithi085 conmon[29319]: audit 2021-02-10T04:51:16.215113+0000 mon.c (mon.1) 93 : audit [DBG] from='client.? 172.21.15.85:0/3680285070' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:51:16.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:16 smithi085 conmon[29319]: audit 2021-02-10T04:51:16.235082+0000 mgr.y (mgr.14140) 1200 : audit [DBG] from='client.15624 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:51:16.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:16 smithi085 conmon[35961]: cluster 2021-02-10T04:51:16.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:16 smithi085 conmon[35961]: 2021-02-10T04:51:15.759484 2021-02-10T04:51:16.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:16 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2549 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:16.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:16 smithi085 conmon[35961]: cluster 2021-02-10T04:51:16.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:16 smithi085 conmon[35961]: 2021-02-10T04:51:15.759496+0000 osd.4 (osd.4) 2550 : cluster 2021-02-10T04:51:16.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:16 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:16.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:16 smithi085 conmon[35961]: cluster 2021-02-10T04:51:16.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:16 smithi085 conmon[35961]: 2021-02-10T04:51:15.759502+0000 osd.4 (osd.4) 2021-02-10T04:51:16.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:16 smithi085 conmon[35961]: 2551 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:16.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:16 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:51:16.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:16 smithi085 conmon[35961]: -02-10T04:51:15.759509+0000 2021-02-10T04:51:16.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:16 smithi085 conmon[35961]: osd.4 (osd.4) 2552 : cluster [WRN] 2021-02-10T04:51:16.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:16 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:16.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:16 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:51:16.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:16 smithi085 conmon[35961]: 04:51:16.215113+0000 mon.c (mon 2021-02-10T04:51:16.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:16 smithi085 conmon[35961]: .1) 93 : audit [DBG] 2021-02-10T04:51:16.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:16 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/3680285070' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:51:16.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:16 smithi085 conmon[35961]: audit 2021 2021-02-10T04:51:16.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:16 smithi085 conmon[35961]: -02-10T04:51:16.235082 2021-02-10T04:51:16.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:16 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1200 : audit 2021-02-10T04:51:16.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:16 smithi085 conmon[35961]: [DBG] from='client.15624 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:51:17.697 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set nodeep-scrub 2021-02-10T04:51:17.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:17 smithi164 conmon[29288]: cluster 2021-02-10T04:51:16.412977+0000 mgr.y (mgr 2021-02-10T04:51:17.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:17 smithi164 conmon[29288]: .14140) 1201 : cluster [DBG] pgmap v1994: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:51:17.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:17 smithi164 conmon[29288]: cluster 2021-02-10T04:51:16.554462+0000 osd.2 ( 2021-02-10T04:51:17.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:17 smithi164 conmon[29288]: osd.2) 1959 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:51:17.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:17 smithi164 conmon[29288]: cluster 2021-02-10T04:51:16. 2021-02-10T04:51:17.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:17 smithi164 conmon[29288]: 555091+0000 osd.2 (osd.2) 1960 : cluster [DBG] 2021-02-10T04:51:17.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:17 smithi164 conmon[29288]: purged_snaps scrub ok 2021-02-10T04:51:17.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:17 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:51:17.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:17 smithi164 conmon[29288]: 51:16.637781+0000 2021-02-10T04:51:17.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:17 smithi164 conmon[29288]: mon.a (mon.0) 2309 : cluster [DBG] osdmap e1116: 8 total, 8 up, 7 in 2021-02-10T04:51:17.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:17 smithi164 conmon[29288]: 2021-02-10T04:51:17.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:17 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:51:17.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:17 smithi085 conmon[29319]: 04:51:16.412977+0000 mgr.y (mgr.14140) 1201 : cluster [DBG] pgmap v1994: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:51:17.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:17 smithi085 conmon[29319]: cluster 2021-02-10T04:51:16.554462+0000 osd.2 (osd.2) 1959 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:51:17.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:17 smithi085 conmon[29319]: cluster 2021-02-10T04:51:16.555091+0000 osd.2 (osd.2) 1960 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:51:17.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:17 smithi085 conmon[29319]: cluster 2021-02-10T04:51:17.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:17 smithi085 conmon[29319]: 2021-02-10T04:51:16.637781+0000 mon.a (mon.0) 2309 : cluster [DBG] osdmap e1116: 8 total, 8 up, 7 in 2021-02-10T04:51:17.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:17 smithi085 conmon[35961]: cluster 2021-02-10T04:51:16.412977+0000 mgr.y ( 2021-02-10T04:51:17.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:17 smithi085 conmon[35961]: mgr.14140) 1201 : cluster [DBG] pgmap v1994: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:51:17.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:17 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:51:17.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:17 smithi085 conmon[35961]: -10T04:51:16.554462+0000 osd.2 2021-02-10T04:51:17.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:17 smithi085 conmon[35961]: (osd.2) 1959 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:51:17.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:17 smithi085 conmon[35961]: 2021-02-10T04:51:17.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:17 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:51:17.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:17 smithi085 conmon[35961]: -10T04:51:16.555091 2021-02-10T04:51:17.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:17 smithi085 conmon[35961]: +0000 osd.2 (osd.2 2021-02-10T04:51:17.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:17 smithi085 conmon[35961]: ) 1960 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:51:17.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:17 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:51:17.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:17 smithi085 conmon[35961]: -10T04:51:16.637781+0000 mon.a 2021-02-10T04:51:17.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:17 smithi085 conmon[35961]: (mon.0) 2309 : cluster [DBG] osdmap e1116: 8 total, 8 up, 7 in 2021-02-10T04:51:18.064 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:51:18.630 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":1993,"stamp":"2021-02-10T04:51:14.411992+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":940031637,"num_objects":743,"num_object_clones":336,"num_object_copies":2229,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":743,"num_whiteouts":12,"num_read":7693,"num_read_kb":4771900,"num_write":4628,"num_write_kb":1265219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":395,"num_bytes_recovered":101346816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25130,"ondisk_log_size":25130,"up":564,"acting":564,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":564,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2175532,"kb_used_data":1863004,"kb_used_omap":386,"kb_used_meta":311485,"kb_avail":654040532,"statfs":{"total":671965249536,"available":669737504768,"internally_reserved":0,"allocated":1907716096,"data_stored":2513820182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":396036,"internal_metadata":318960892},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0,4],"upper_bound":1048576},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002154"},"pg_stats":[{"pgid":"6.29","version":"0'0","reported_seq":"1111","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595662+0000","last_change":"2021-02-10T04:47:12.944476+0000","last_active":"2021-02-10T04:51:04.595662+0000","last_peered":"2021-02-10T04:51:04.595662+0000","last_clean":"2021-02-10T04:51:04.595662+0000","last_became_active":"2021-02-10T04:45:49.759065+0000","last_became_peered":"2021-02-10T04:45:49.759065+0000","last_unstale":"2021-02-10T04:51:04.595662+0000","last_undegraded":"2021-02-10T04:51:04.595662+0000","last_fullsized":"2021-02-10T04:51:04.595662+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:12.944345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:12.944345+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":"505","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.599973+0000","last_change":"2021-02-10T04:47:18.026548+0000","last_active":"2021-02-10T04:51:09.599973+0000","last_peered":"2021-02-10T04:51:09.599973+0000","last_clean":"2021-02-10T04:51:09.599973+0000","last_became_active":"2021-02-10T04:45:49.761512+0000","last_became_peered":"2021-02-10T04:45:49.761512+0000","last_unstale":"2021-02-10T04:51:09.599973+0000","last_undegraded":"2021-02-10T04:51:09.599973+0000","last_fullsized":"2021-02-10T04:51:09.599973+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:18.026478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:18.026478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"131'3","reported_seq":"1286","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:12.989256+0000","last_change":"2021-02-10T04:51:01.185887+0000","last_active":"2021-02-10T04:51:12.989256+0000","last_peered":"2021-02-10T04:51:12.989256+0000","last_clean":"2021-02-10T04:51:12.989256+0000","last_became_active":"2021-02-10T04:29:07.145525+0000","last_became_peered":"2021-02-10T04:29:07.145525+0000","last_unstale":"2021-02-10T04:51:12.989256+0000","last_undegraded":"2021-02-10T04:51:12.989256+0000","last_fullsized":"2021-02-10T04:51:12.989256+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'3","last_scrub_stamp":"2021-02-10T04:51:01.185820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.185820+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"1049'54","reported_seq":"1184","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.575344+0000","last_change":"2021-02-10T04:46:51.840139+0000","last_active":"2021-02-10T04:51:04.575344+0000","last_peered":"2021-02-10T04:51:04.575344+0000","last_clean":"2021-02-10T04:51:04.575344+0000","last_became_active":"2021-02-10T04:25:49.002058+0000","last_became_peered":"2021-02-10T04:25:49.002058+0000","last_unstale":"2021-02-10T04:51:04.575344+0000","last_undegraded":"2021-02-10T04:51:04.575344+0000","last_fullsized":"2021-02-10T04:51:04.575344+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:46:51.840049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:51.840049+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":95,"num_read_kb":78,"num_write":55,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"474","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:08.005652+0000","last_change":"2021-02-10T04:51:08.005652+0000","last_active":"2021-02-10T04:51:08.005652+0000","last_peered":"2021-02-10T04:51:08.005652+0000","last_clean":"2021-02-10T04:51:08.005652+0000","last_became_active":"2021-02-10T04:50:59.548518+0000","last_became_peered":"2021-02-10T04:50:59.548518+0000","last_unstale":"2021-02-10T04:51:08.005652+0000","last_undegraded":"2021-02-10T04:51:08.005652+0000","last_fullsized":"2021-02-10T04:51:08.005652+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:46:34.577684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:34.577684+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"1083","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.576183+0000","last_change":"2021-02-10T04:50:48.408819+0000","last_active":"2021-02-10T04:51:04.576183+0000","last_peered":"2021-02-10T04:51:04.576183+0000","last_clean":"2021-02-10T04:51:04.576183+0000","last_became_active":"2021-02-10T04:25:53.148090+0000","last_became_peered":"2021-02-10T04:25:53.148090+0000","last_unstale":"2021-02-10T04:51:04.576183+0000","last_undegraded":"2021-02-10T04:51:04.576183+0000","last_fullsized":"2021-02-10T04:51:04.576183+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:48.408748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:50:48.408748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"1003'1689","reported_seq":"4651","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.575925+0000","last_change":"2021-02-10T04:46:54.814105+0000","last_active":"2021-02-10T04:51:04.575925+0000","last_peered":"2021-02-10T04:51:04.575925+0000","last_clean":"2021-02-10T04:51:04.575925+0000","last_became_active":"2021-02-10T04:33:05.144488+0000","last_became_peered":"2021-02-10T04:33:05.144488+0000","last_unstale":"2021-02-10T04:51:04.575925+0000","last_undegraded":"2021-02-10T04:51:04.575925+0000","last_fullsized":"2021-02-10T04:51:04.575925+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:46:54.814020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:54.814020+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":126521476,"num_objects":49,"num_object_clones":44,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":773,"num_read_kb":819582,"num_write":561,"num_write_kb":221117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"1049'32","reported_seq":"1177","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.596309+0000","last_change":"2021-02-10T04:46:31.614762+0000","last_active":"2021-02-10T04:51:04.596309+0000","last_peered":"2021-02-10T04:51:04.596309+0000","last_clean":"2021-02-10T04:51:04.596309+0000","last_became_active":"2021-02-10T04:38:39.243413+0000","last_became_peered":"2021-02-10T04:38:39.243413+0000","last_unstale":"2021-02-10T04:51:04.596309+0000","last_undegraded":"2021-02-10T04:51:04.596309+0000","last_fullsized":"2021-02-10T04:51:04.596309+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'32","last_scrub_stamp":"2021-02-10T04:46:31.614697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:31.614697+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":62,"num_read_kb":45,"num_write":33,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":894,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"782","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595832+0000","last_change":"2021-02-10T04:50:59.824441+0000","last_active":"2021-02-10T04:51:04.595832+0000","last_peered":"2021-02-10T04:51:04.595832+0000","last_clean":"2021-02-10T04:51:04.595832+0000","last_became_active":"2021-02-10T04:50:59.526936+0000","last_became_peered":"2021-02-10T04:50:59.526936+0000","last_unstale":"2021-02-10T04:51:04.595832+0000","last_undegraded":"2021-02-10T04:51:04.595832+0000","last_fullsized":"2021-02-10T04:51:04.595832+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:59.824373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:59.824373+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"506","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595531+0000","last_change":"2021-02-10T04:50:59.571325+0000","last_active":"2021-02-10T04:51:04.595531+0000","last_peered":"2021-02-10T04:51:04.595531+0000","last_clean":"2021-02-10T04:51:04.595531+0000","last_became_active":"2021-02-10T04:50:59.570943+0000","last_became_peered":"2021-02-10T04:50:59.570943+0000","last_unstale":"2021-02-10T04:51:04.595531+0000","last_undegraded":"2021-02-10T04:51:04.595531+0000","last_fullsized":"2021-02-10T04:51:04.595531+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:46.664985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:46:46.664985+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"1118","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.596846+0000","last_change":"2021-02-10T04:50:59.577404+0000","last_active":"2021-02-10T04:51:04.596846+0000","last_peered":"2021-02-10T04:51:04.596846+0000","last_clean":"2021-02-10T04:51:04.596846+0000","last_became_active":"2021-02-10T04:50:59.577097+0000","last_became_peered":"2021-02-10T04:50:59.577097+0000","last_unstale":"2021-02-10T04:51:04.596846+0000","last_undegraded":"2021-02-10T04:51:04.596846+0000","last_fullsized":"2021-02-10T04:51:04.596846+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:09.922789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:09.922789+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"259'74","reported_seq":"2031","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595120+0000","last_change":"2021-02-10T04:50:59.556045+0000","last_active":"2021-02-10T04:51:04.595120+0000","last_peered":"2021-02-10T04:51:04.595120+0000","last_clean":"2021-02-10T04:51:04.595120+0000","last_became_active":"2021-02-10T04:50:59.555212+0000","last_became_peered":"2021-02-10T04:50:59.555212+0000","last_unstale":"2021-02-10T04:51:04.595120+0000","last_undegraded":"2021-02-10T04:51:04.595120+0000","last_fullsized":"2021-02-10T04:51:04.595120+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'62","last_scrub_stamp":"2021-02-10T04:30:10.905905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:10.905905+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1218524,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":28227,"num_write":19,"num_write_kb":7021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":567285,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"1049'61","reported_seq":"1198","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.601014+0000","last_change":"2021-02-10T04:50:54.154530+0000","last_active":"2021-02-10T04:51:09.601014+0000","last_peered":"2021-02-10T04:51:09.601014+0000","last_clean":"2021-02-10T04:51:09.601014+0000","last_became_active":"2021-02-10T04:25:49.015913+0000","last_became_peered":"2021-02-10T04:25:49.015913+0000","last_unstale":"2021-02-10T04:51:09.601014+0000","last_undegraded":"2021-02-10T04:51:09.601014+0000","last_fullsized":"2021-02-10T04:51:09.601014+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'61","last_scrub_stamp":"2021-02-10T04:50:54.154436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:50:54.154436+0000","log_size":61,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":104,"num_read_kb":84,"num_write":62,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"642'3","reported_seq":"686","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.601375+0000","last_change":"2021-02-10T04:47:02.791566+0000","last_active":"2021-02-10T04:51:09.601375+0000","last_peered":"2021-02-10T04:51:09.601375+0000","last_clean":"2021-02-10T04:51:09.601375+0000","last_became_active":"2021-02-10T04:37:37.234230+0000","last_became_peered":"2021-02-10T04:37:37.234230+0000","last_unstale":"2021-02-10T04:51:09.601375+0000","last_undegraded":"2021-02-10T04:51:09.601375+0000","last_fullsized":"2021-02-10T04:51:09.601375+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'3","last_scrub_stamp":"2021-02-10T04:47:02.791478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:02.791478+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"502","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595990+0000","last_change":"2021-02-10T04:50:52.421852+0000","last_active":"2021-02-10T04:51:04.595990+0000","last_peered":"2021-02-10T04:51:04.595990+0000","last_clean":"2021-02-10T04:51:04.595990+0000","last_became_active":"2021-02-10T04:37:37.230818+0000","last_became_peered":"2021-02-10T04:37:37.230818+0000","last_unstale":"2021-02-10T04:51:04.595990+0000","last_undegraded":"2021-02-10T04:51:04.595990+0000","last_fullsized":"2021-02-10T04:51:04.595990+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:52.421793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:52.421793+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"488","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.998889+0000","last_change":"2021-02-10T04:51:07.998889+0000","last_active":"2021-02-10T04:51:07.998889+0000","last_peered":"2021-02-10T04:51:07.998889+0000","last_clean":"2021-02-10T04:51:07.998889+0000","last_became_active":"2021-02-10T04:50:59.549316+0000","last_became_peered":"2021-02-10T04:50:59.549316+0000","last_unstale":"2021-02-10T04:51:07.998889+0000","last_undegraded":"2021-02-10T04:51:07.998889+0000","last_fullsized":"2021-02-10T04:51:07.998889+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.731458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.731458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"1003'739","reported_seq":"3779","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595812+0000","last_change":"2021-02-10T04:46:47.806184+0000","last_active":"2021-02-10T04:51:04.595812+0000","last_peered":"2021-02-10T04:51:04.595812+0000","last_clean":"2021-02-10T04:51:04.595812+0000","last_became_active":"2021-02-10T04:44:25.545483+0000","last_became_peered":"2021-02-10T04:44:25.545483+0000","last_unstale":"2021-02-10T04:51:04.595812+0000","last_undegraded":"2021-02-10T04:51:04.595812+0000","last_fullsized":"2021-02-10T04:51:04.595812+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:46:47.806105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:47.806105+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25888931,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"1049'61","reported_seq":"1211","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595928+0000","last_change":"2021-02-10T04:50:47.662471+0000","last_active":"2021-02-10T04:51:04.595928+0000","last_peered":"2021-02-10T04:51:04.595928+0000","last_clean":"2021-02-10T04:51:04.595928+0000","last_became_active":"2021-02-10T04:37:37.237215+0000","last_became_peered":"2021-02-10T04:37:37.237215+0000","last_unstale":"2021-02-10T04:51:04.595928+0000","last_undegraded":"2021-02-10T04:51:04.595928+0000","last_fullsized":"2021-02-10T04:51:04.595928+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'61","last_scrub_stamp":"2021-02-10T04:50:47.662378+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:50:47.662378+0000","log_size":61,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":83,"num_read_kb":65,"num_write":50,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"1004","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595289+0000","last_change":"2021-02-10T04:50:45.623700+0000","last_active":"2021-02-10T04:51:04.595289+0000","last_peered":"2021-02-10T04:51:04.595289+0000","last_clean":"2021-02-10T04:51:04.595289+0000","last_became_active":"2021-02-10T04:29:07.145616+0000","last_became_peered":"2021-02-10T04:29:07.145616+0000","last_unstale":"2021-02-10T04:51:04.595289+0000","last_undegraded":"2021-02-10T04:51:04.595289+0000","last_fullsized":"2021-02-10T04:51:04.595289+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:45.623640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:45.623640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"762","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595276+0000","last_change":"2021-02-10T04:46:43.762837+0000","last_active":"2021-02-10T04:51:04.595276+0000","last_peered":"2021-02-10T04:51:04.595276+0000","last_clean":"2021-02-10T04:51:04.595276+0000","last_became_active":"2021-02-10T04:33:05.146783+0000","last_became_peered":"2021-02-10T04:33:05.146783+0000","last_unstale":"2021-02-10T04:51:04.595276+0000","last_undegraded":"2021-02-10T04:51:04.595276+0000","last_fullsized":"2021-02-10T04:51:04.595276+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:43.762742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:46:43.762742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"462","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.991134+0000","last_change":"2021-02-10T04:51:07.991134+0000","last_active":"2021-02-10T04:51:07.991134+0000","last_peered":"2021-02-10T04:51:07.991134+0000","last_clean":"2021-02-10T04:51:07.991134+0000","last_became_active":"2021-02-10T04:50:59.585315+0000","last_became_peered":"2021-02-10T04:50:59.585315+0000","last_unstale":"2021-02-10T04:51:07.991134+0000","last_undegraded":"2021-02-10T04:51:07.991134+0000","last_fullsized":"2021-02-10T04:51:07.991134+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:13.727241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:13.727241+0000","last_clean_scrub_stamp":"2021-02-10T04:47:13.727241+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"1063'654","reported_seq":"4324","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.576934+0000","last_change":"2021-02-10T04:50:59.541622+0000","last_active":"2021-02-10T04:51:04.576934+0000","last_peered":"2021-02-10T04:51:04.576934+0000","last_clean":"2021-02-10T04:51:04.576934+0000","last_became_active":"2021-02-10T04:50:59.540629+0000","last_became_peered":"2021-02-10T04:50:59.540629+0000","last_unstale":"2021-02-10T04:51:04.576934+0000","last_undegraded":"2021-02-10T04:51:04.576934+0000","last_fullsized":"2021-02-10T04:51:04.576934+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:47:06.861811+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49401261,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":2,"num_read":342,"num_read_kb":364544,"num_write":223,"num_write_kb":67454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":11919937,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"1045","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.575348+0000","last_change":"2021-02-10T04:50:59.539568+0000","last_active":"2021-02-10T04:51:04.575348+0000","last_peered":"2021-02-10T04:51:04.575348+0000","last_clean":"2021-02-10T04:51:04.575348+0000","last_became_active":"2021-02-10T04:50:59.539264+0000","last_became_peered":"2021-02-10T04:50:59.539264+0000","last_unstale":"2021-02-10T04:51:04.575348+0000","last_undegraded":"2021-02-10T04:51:04.575348+0000","last_fullsized":"2021-02-10T04:51:04.575348+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:33.963005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:46:33.963005+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"1066","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.992892+0000","last_change":"2021-02-10T04:51:07.992892+0000","last_active":"2021-02-10T04:51:07.992892+0000","last_peered":"2021-02-10T04:51:07.992892+0000","last_clean":"2021-02-10T04:51:07.992892+0000","last_became_active":"2021-02-10T04:50:59.550989+0000","last_became_peered":"2021-02-10T04:50:59.550989+0000","last_unstale":"2021-02-10T04:51:07.992892+0000","last_undegraded":"2021-02-10T04:51:07.992892+0000","last_fullsized":"2021-02-10T04:51:07.992892+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:53.528227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:46:53.528227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"1049'33","reported_seq":"1152","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:06.847026+0000","last_change":"2021-02-10T04:51:06.846957+0000","last_active":"2021-02-10T04:51:06.847026+0000","last_peered":"2021-02-10T04:51:06.847026+0000","last_clean":"2021-02-10T04:51:06.847026+0000","last_became_active":"2021-02-10T04:50:59.584833+0000","last_became_peered":"2021-02-10T04:50:59.584833+0000","last_unstale":"2021-02-10T04:51:06.847026+0000","last_undegraded":"2021-02-10T04:51:06.847026+0000","last_fullsized":"2021-02-10T04:51:06.847026+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'33","last_scrub_stamp":"2021-02-10T04:51:06.846850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:06.846850+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":43,"num_write":33,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"763","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.983976+0000","last_change":"2021-02-10T04:51:07.983976+0000","last_active":"2021-02-10T04:51:07.983976+0000","last_peered":"2021-02-10T04:51:07.983976+0000","last_clean":"2021-02-10T04:51:07.983976+0000","last_became_active":"2021-02-10T04:50:59.565075+0000","last_became_peered":"2021-02-10T04:50:59.565075+0000","last_unstale":"2021-02-10T04:51:07.983976+0000","last_undegraded":"2021-02-10T04:51:07.983976+0000","last_fullsized":"2021-02-10T04:51:07.983976+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:19.573103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:19.573103+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"639'3","reported_seq":"674","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.602572+0000","last_change":"2021-02-10T04:47:01.757109+0000","last_active":"2021-02-10T04:51:09.602572+0000","last_peered":"2021-02-10T04:51:09.602572+0000","last_clean":"2021-02-10T04:51:09.602572+0000","last_became_active":"2021-02-10T04:37:37.233738+0000","last_became_peered":"2021-02-10T04:37:37.233738+0000","last_unstale":"2021-02-10T04:51:09.602572+0000","last_undegraded":"2021-02-10T04:51:09.602572+0000","last_fullsized":"2021-02-10T04:51:09.602572+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"639'3","last_scrub_stamp":"2021-02-10T04:47:01.756908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:47:01.756908+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"1086","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595864+0000","last_change":"2021-02-10T04:46:48.602911+0000","last_active":"2021-02-10T04:51:04.595864+0000","last_peered":"2021-02-10T04:51:04.595864+0000","last_clean":"2021-02-10T04:51:04.595864+0000","last_became_active":"2021-02-10T04:37:37.233109+0000","last_became_peered":"2021-02-10T04:37:37.233109+0000","last_unstale":"2021-02-10T04:51:04.595864+0000","last_undegraded":"2021-02-10T04:51:04.595864+0000","last_fullsized":"2021-02-10T04:51:04.595864+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:48.602819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:48.602819+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"1004'1093","reported_seq":"6352","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.596331+0000","last_change":"2021-02-10T04:50:59.597444+0000","last_active":"2021-02-10T04:51:04.596331+0000","last_peered":"2021-02-10T04:51:04.596331+0000","last_clean":"2021-02-10T04:51:04.596331+0000","last_became_active":"2021-02-10T04:50:59.596040+0000","last_became_peered":"2021-02-10T04:50:59.596040+0000","last_unstale":"2021-02-10T04:51:04.596331+0000","last_undegraded":"2021-02-10T04:51:04.596331+0000","last_fullsized":"2021-02-10T04:51:04.596331+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:46:19.757486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:19.757486+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":78869393,"num_objects":34,"num_object_clones":30,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":1,"num_read":558,"num_read_kb":622116,"num_write":371,"num_write_kb":133580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"1035","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:12.389350+0000","last_change":"2021-02-10T04:51:12.389350+0000","last_active":"2021-02-10T04:51:12.389350+0000","last_peered":"2021-02-10T04:51:12.389350+0000","last_clean":"2021-02-10T04:51:12.389350+0000","last_became_active":"2021-02-10T04:37:37.229648+0000","last_became_peered":"2021-02-10T04:37:37.229648+0000","last_unstale":"2021-02-10T04:51:12.389350+0000","last_undegraded":"2021-02-10T04:51:12.389350+0000","last_fullsized":"2021-02-10T04:51:12.389350+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:12.389272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:12.389272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"1049'41","reported_seq":"837","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.985033+0000","last_change":"2021-02-10T04:51:07.985033+0000","last_active":"2021-02-10T04:51:07.985033+0000","last_peered":"2021-02-10T04:51:07.985033+0000","last_clean":"2021-02-10T04:51:07.985033+0000","last_became_active":"2021-02-10T04:50:59.587994+0000","last_became_peered":"2021-02-10T04:50:59.587994+0000","last_unstale":"2021-02-10T04:51:07.985033+0000","last_undegraded":"2021-02-10T04:51:07.985033+0000","last_fullsized":"2021-02-10T04:51:07.985033+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'41","last_scrub_stamp":"2021-02-10T04:46:28.515513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:28.515513+0000","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":66,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"786","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.984726+0000","last_change":"2021-02-10T04:51:07.984726+0000","last_active":"2021-02-10T04:51:07.984726+0000","last_peered":"2021-02-10T04:51:07.984726+0000","last_clean":"2021-02-10T04:51:07.984726+0000","last_became_active":"2021-02-10T04:50:59.561519+0000","last_became_peered":"2021-02-10T04:50:59.561519+0000","last_unstale":"2021-02-10T04:51:07.984726+0000","last_undegraded":"2021-02-10T04:51:07.984726+0000","last_fullsized":"2021-02-10T04:51:07.984726+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:47.605360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_clean_scrub_stamp":"2021-02-10T04:46:47.605360+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"78'3","reported_seq":"1386","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.988269+0000","last_change":"2021-02-10T04:46:17.816956+0000","last_active":"2021-02-10T04:51:07.988269+0000","last_peered":"2021-02-10T04:51:07.988269+0000","last_clean":"2021-02-10T04:51:07.988269+0000","last_became_active":"2021-02-10T04:25:51.025218+0000","last_became_peered":"2021-02-10T04:25:51.025218+0000","last_unstale":"2021-02-10T04:51:07.988269+0000","last_undegraded":"2021-02-10T04:51:07.988269+0000","last_fullsized":"2021-02-10T04:51:07.988269+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:46:17.816879+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:46:17.816879+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"1083","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.577465+0000","last_change":"2021-02-10T04:50:10.176968+0000","last_active":"2021-02-10T04:51:04.577465+0000","last_peered":"2021-02-10T04:51:04.577465+0000","last_clean":"2021-02-10T04:51:04.577465+0000","last_became_active":"2021-02-10T04:25:53.042697+0000","last_became_peered":"2021-02-10T04:25:53.042697+0000","last_unstale":"2021-02-10T04:51:04.577465+0000","last_undegraded":"2021-02-10T04:51:04.577465+0000","last_fullsized":"2021-02-10T04:51:04.577465+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:10.176878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:50:10.176878+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"1002'1090","reported_seq":"5298","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.602266+0000","last_change":"2021-02-10T04:51:08.220559+0000","last_active":"2021-02-10T04:51:09.602266+0000","last_peered":"2021-02-10T04:51:09.602266+0000","last_clean":"2021-02-10T04:51:09.602266+0000","last_became_active":"2021-02-10T04:50:59.585773+0000","last_became_peered":"2021-02-10T04:50:59.585773+0000","last_unstale":"2021-02-10T04:51:09.602266+0000","last_undegraded":"2021-02-10T04:51:09.602266+0000","last_fullsized":"2021-02-10T04:51:09.602266+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:51:08.220472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.220472+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":48821292,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":491,"num_read_kb":552336,"num_write":351,"num_write_kb":129921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"1049'47","reported_seq":"1162","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595679+0000","last_change":"2021-02-10T04:50:04.767513+0000","last_active":"2021-02-10T04:51:04.595679+0000","last_peered":"2021-02-10T04:51:04.595679+0000","last_clean":"2021-02-10T04:51:04.595679+0000","last_became_active":"2021-02-10T04:25:49.011093+0000","last_became_peered":"2021-02-10T04:25:49.011093+0000","last_unstale":"2021-02-10T04:51:04.595679+0000","last_undegraded":"2021-02-10T04:51:04.595679+0000","last_fullsized":"2021-02-10T04:51:04.595679+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'47","last_scrub_stamp":"2021-02-10T04:50:04.767446+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:50:04.767446+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":68,"num_read_kb":52,"num_write":38,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"1068","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595661+0000","last_change":"2021-02-10T04:50:08.358363+0000","last_active":"2021-02-10T04:51:04.595661+0000","last_peered":"2021-02-10T04:51:04.595661+0000","last_clean":"2021-02-10T04:51:04.595661+0000","last_became_active":"2021-02-10T04:29:07.160593+0000","last_became_peered":"2021-02-10T04:29:07.160593+0000","last_unstale":"2021-02-10T04:51:04.595661+0000","last_undegraded":"2021-02-10T04:51:04.595661+0000","last_fullsized":"2021-02-10T04:51:04.595661+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:08.358300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:50:08.358300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"1085","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.577255+0000","last_change":"2021-02-10T04:46:48.842056+0000","last_active":"2021-02-10T04:51:04.577255+0000","last_peered":"2021-02-10T04:51:04.577255+0000","last_clean":"2021-02-10T04:51:04.577255+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:51:04.577255+0000","last_undegraded":"2021-02-10T04:51:04.577255+0000","last_fullsized":"2021-02-10T04:51:04.577255+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:48.841970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:46:48.841970+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"1041","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:08.003846+0000","last_change":"2021-02-10T04:51:08.003846+0000","last_active":"2021-02-10T04:51:08.003846+0000","last_peered":"2021-02-10T04:51:08.003846+0000","last_clean":"2021-02-10T04:51:08.003846+0000","last_became_active":"2021-02-10T04:50:59.545501+0000","last_became_peered":"2021-02-10T04:50:59.545501+0000","last_unstale":"2021-02-10T04:51:08.003846+0000","last_undegraded":"2021-02-10T04:51:08.003846+0000","last_fullsized":"2021-02-10T04:51:08.003846+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:01.565998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:01.565998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"1001'1165","reported_seq":"4262","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.602222+0000","last_change":"2021-02-10T04:51:03.221365+0000","last_active":"2021-02-10T04:51:09.602222+0000","last_peered":"2021-02-10T04:51:09.602222+0000","last_clean":"2021-02-10T04:51:09.602222+0000","last_became_active":"2021-02-10T04:28:25.152030+0000","last_became_peered":"2021-02-10T04:28:25.152030+0000","last_unstale":"2021-02-10T04:51:09.602222+0000","last_undegraded":"2021-02-10T04:51:09.602222+0000","last_fullsized":"2021-02-10T04:51:09.602222+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":59203690,"num_objects":25,"num_object_clones":21,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":573,"num_read_kb":599059,"num_write":400,"num_write_kb":178261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"1113'5","reported_seq":"1288","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.988308+0000","last_change":"2021-02-10T04:51:07.979631+0000","last_active":"2021-02-10T04:51:07.988308+0000","last_peered":"2021-02-10T04:51:07.988308+0000","last_clean":"2021-02-10T04:51:07.988308+0000","last_became_active":"2021-02-10T04:50:59.569578+0000","last_became_peered":"2021-02-10T04:50:59.569578+0000","last_unstale":"2021-02-10T04:51:07.988308+0000","last_undegraded":"2021-02-10T04:51:07.988308+0000","last_fullsized":"2021-02-10T04:51:07.988308+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'3","last_scrub_stamp":"2021-02-10T04:46:36.557037+0000","last_deep_scrub":"158'3","last_deep_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_clean_scrub_stamp":"2021-02-10T04:46:36.557037+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"1049'54","reported_seq":"1171","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595187+0000","last_change":"2021-02-10T04:46:38.776905+0000","last_active":"2021-02-10T04:51:04.595187+0000","last_peered":"2021-02-10T04:51:04.595187+0000","last_clean":"2021-02-10T04:51:04.595187+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:51:04.595187+0000","last_undegraded":"2021-02-10T04:51:04.595187+0000","last_fullsized":"2021-02-10T04:51:04.595187+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:46:38.776809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:38.776809+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":93,"num_read_kb":73,"num_write":54,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"1151","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.602513+0000","last_change":"2021-02-10T04:50:59.591464+0000","last_active":"2021-02-10T04:51:09.602513+0000","last_peered":"2021-02-10T04:51:09.602513+0000","last_clean":"2021-02-10T04:51:09.602513+0000","last_became_active":"2021-02-10T04:50:59.591146+0000","last_became_peered":"2021-02-10T04:50:59.591146+0000","last_unstale":"2021-02-10T04:51:09.602513+0000","last_undegraded":"2021-02-10T04:51:09.602513+0000","last_fullsized":"2021-02-10T04:51:09.602513+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:04.957072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:04.957072+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"1029","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.602122+0000","last_change":"2021-02-10T04:47:16.010935+0000","last_active":"2021-02-10T04:51:09.602122+0000","last_peered":"2021-02-10T04:51:09.602122+0000","last_clean":"2021-02-10T04:51:09.602122+0000","last_became_active":"2021-02-10T04:45:47.533648+0000","last_became_peered":"2021-02-10T04:45:47.533648+0000","last_unstale":"2021-02-10T04:51:09.602122+0000","last_undegraded":"2021-02-10T04:51:09.602122+0000","last_fullsized":"2021-02-10T04:51:09.602122+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1045,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:16.010876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:16.010876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"998'1009","reported_seq":"4579","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.984082+0000","last_change":"2021-02-10T04:51:07.984082+0000","last_active":"2021-02-10T04:51:07.984082+0000","last_peered":"2021-02-10T04:51:07.984082+0000","last_clean":"2021-02-10T04:51:07.984082+0000","last_became_active":"2021-02-10T04:50:59.567798+0000","last_became_peered":"2021-02-10T04:50:59.567798+0000","last_unstale":"2021-02-10T04:51:07.984082+0000","last_undegraded":"2021-02-10T04:51:07.984082+0000","last_fullsized":"2021-02-10T04:51:07.984082+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24710413,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":239,"num_read_kb":249403,"num_write":154,"num_write_kb":50104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":8290058,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"980","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.976066+0000","last_change":"2021-02-10T04:51:07.976066+0000","last_active":"2021-02-10T04:51:07.976066+0000","last_peered":"2021-02-10T04:51:07.976066+0000","last_clean":"2021-02-10T04:51:07.976066+0000","last_became_active":"2021-02-10T04:50:59.570854+0000","last_became_peered":"2021-02-10T04:50:59.570854+0000","last_unstale":"2021-02-10T04:51:07.976066+0000","last_undegraded":"2021-02-10T04:51:07.976066+0000","last_fullsized":"2021-02-10T04:51:07.976066+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:46.592377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:46:46.592377+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"1094","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595528+0000","last_change":"2021-02-10T04:50:59.527897+0000","last_active":"2021-02-10T04:51:04.595528+0000","last_peered":"2021-02-10T04:51:04.595528+0000","last_clean":"2021-02-10T04:51:04.595528+0000","last_became_active":"2021-02-10T04:50:59.527553+0000","last_became_peered":"2021-02-10T04:50:59.527553+0000","last_unstale":"2021-02-10T04:51:04.595528+0000","last_undegraded":"2021-02-10T04:51:04.595528+0000","last_fullsized":"2021-02-10T04:51:04.595528+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:35.785517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:46:35.785517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"1049'72","reported_seq":"1197","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.988809+0000","last_change":"2021-02-10T04:51:07.988809+0000","last_active":"2021-02-10T04:51:07.988809+0000","last_peered":"2021-02-10T04:51:07.988809+0000","last_clean":"2021-02-10T04:51:07.988809+0000","last_became_active":"2021-02-10T04:50:59.561344+0000","last_became_peered":"2021-02-10T04:50:59.561344+0000","last_unstale":"2021-02-10T04:51:07.988809+0000","last_undegraded":"2021-02-10T04:51:07.988809+0000","last_fullsized":"2021-02-10T04:51:07.988809+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'72","last_scrub_stamp":"2021-02-10T04:46:30.515026+0000","last_deep_scrub":"152'48","last_deep_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_clean_scrub_stamp":"2021-02-10T04:46:30.515026+0000","log_size":72,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":105,"num_read_kb":83,"num_write":60,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"777","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.602351+0000","last_change":"2021-02-10T04:46:22.069341+0000","last_active":"2021-02-10T04:51:09.602351+0000","last_peered":"2021-02-10T04:51:09.602351+0000","last_clean":"2021-02-10T04:51:09.602351+0000","last_became_active":"2021-02-10T04:33:05.141066+0000","last_became_peered":"2021-02-10T04:33:05.141066+0000","last_unstale":"2021-02-10T04:51:09.602351+0000","last_undegraded":"2021-02-10T04:51:09.602351+0000","last_fullsized":"2021-02-10T04:51:09.602351+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:46:22.069247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:22.069247+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"1110","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595585+0000","last_change":"2021-02-10T04:50:59.571151+0000","last_active":"2021-02-10T04:51:04.595585+0000","last_peered":"2021-02-10T04:51:04.595585+0000","last_clean":"2021-02-10T04:51:04.595585+0000","last_became_active":"2021-02-10T04:50:59.570328+0000","last_became_peered":"2021-02-10T04:50:59.570328+0000","last_unstale":"2021-02-10T04:51:04.595585+0000","last_undegraded":"2021-02-10T04:51:04.595585+0000","last_fullsized":"2021-02-10T04:51:04.595585+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:27.926662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:44:27.926662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"1002'1657","reported_seq":"4994","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.996273+0000","last_change":"2021-02-10T04:51:07.996273+0000","last_active":"2021-02-10T04:51:07.996273+0000","last_peered":"2021-02-10T04:51:07.996273+0000","last_clean":"2021-02-10T04:51:07.996273+0000","last_became_active":"2021-02-10T04:50:59.574139+0000","last_became_peered":"2021-02-10T04:50:59.574139+0000","last_unstale":"2021-02-10T04:51:07.996273+0000","last_undegraded":"2021-02-10T04:51:07.996273+0000","last_fullsized":"2021-02-10T04:51:07.996273+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":42793066,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":424,"num_read_kb":437673,"num_write":318,"num_write_kb":118797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":17936938,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"1029","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595534+0000","last_change":"2021-02-10T04:46:30.848110+0000","last_active":"2021-02-10T04:51:04.595534+0000","last_peered":"2021-02-10T04:51:04.595534+0000","last_clean":"2021-02-10T04:51:04.595534+0000","last_became_active":"2021-02-10T04:37:37.233158+0000","last_became_peered":"2021-02-10T04:37:37.233158+0000","last_unstale":"2021-02-10T04:51:04.595534+0000","last_undegraded":"2021-02-10T04:51:04.595534+0000","last_fullsized":"2021-02-10T04:51:04.595534+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:30.848045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:46:30.848045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"1157","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.601187+0000","last_change":"2021-02-10T04:50:58.142613+0000","last_active":"2021-02-10T04:51:09.601187+0000","last_peered":"2021-02-10T04:51:09.601187+0000","last_clean":"2021-02-10T04:51:09.601187+0000","last_became_active":"2021-02-10T04:25:51.025636+0000","last_became_peered":"2021-02-10T04:25:51.025636+0000","last_unstale":"2021-02-10T04:51:09.601187+0000","last_undegraded":"2021-02-10T04:51:09.601187+0000","last_fullsized":"2021-02-10T04:51:09.601187+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:58.142526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:58.142526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"1049'47","reported_seq":"1172","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595908+0000","last_change":"2021-02-10T04:46:29.672214+0000","last_active":"2021-02-10T04:51:04.595908+0000","last_peered":"2021-02-10T04:51:04.595908+0000","last_clean":"2021-02-10T04:51:04.595908+0000","last_became_active":"2021-02-10T04:25:49.002937+0000","last_became_peered":"2021-02-10T04:25:49.002937+0000","last_unstale":"2021-02-10T04:51:04.595908+0000","last_undegraded":"2021-02-10T04:51:04.595908+0000","last_fullsized":"2021-02-10T04:51:04.595908+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'47","last_scrub_stamp":"2021-02-10T04:46:29.672141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:29.672141+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":87,"num_read_kb":68,"num_write":50,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"1021","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.596603+0000","last_change":"2021-02-10T04:50:46.434726+0000","last_active":"2021-02-10T04:51:04.596603+0000","last_peered":"2021-02-10T04:51:04.596603+0000","last_clean":"2021-02-10T04:51:04.596603+0000","last_became_active":"2021-02-10T04:29:07.161341+0000","last_became_peered":"2021-02-10T04:29:07.161341+0000","last_unstale":"2021-02-10T04:51:04.596603+0000","last_undegraded":"2021-02-10T04:51:04.596603+0000","last_fullsized":"2021-02-10T04:51:04.596603+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:50:46.434629+0000","last_deep_scrub":"53'2","last_deep_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_clean_scrub_stamp":"2021-02-10T04:50:46.434629+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"787","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.601884+0000","last_change":"2021-02-10T04:47:15.037885+0000","last_active":"2021-02-10T04:51:09.601884+0000","last_peered":"2021-02-10T04:51:09.601884+0000","last_clean":"2021-02-10T04:51:09.601884+0000","last_became_active":"2021-02-10T04:45:47.528080+0000","last_became_peered":"2021-02-10T04:45:47.528080+0000","last_unstale":"2021-02-10T04:51:09.601884+0000","last_undegraded":"2021-02-10T04:51:09.601884+0000","last_fullsized":"2021-02-10T04:51:09.601884+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1045,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:15.037663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:15.037663+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"1000'1071","reported_seq":"4649","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.575581+0000","last_change":"2021-02-10T04:50:59.597606+0000","last_active":"2021-02-10T04:51:04.575581+0000","last_peered":"2021-02-10T04:51:04.575581+0000","last_clean":"2021-02-10T04:51:04.575581+0000","last_became_active":"2021-02-10T04:50:59.596326+0000","last_became_peered":"2021-02-10T04:50:59.596326+0000","last_unstale":"2021-02-10T04:51:04.575581+0000","last_undegraded":"2021-02-10T04:51:04.575581+0000","last_fullsized":"2021-02-10T04:51:04.575581+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:47:14.611347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:14.611347+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46572736,"num_objects":18,"num_object_clones":15,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"1123","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.602463+0000","last_change":"2021-02-10T04:51:00.195591+0000","last_active":"2021-02-10T04:51:09.602463+0000","last_peered":"2021-02-10T04:51:09.602463+0000","last_clean":"2021-02-10T04:51:09.602463+0000","last_became_active":"2021-02-10T04:50:59.543222+0000","last_became_peered":"2021-02-10T04:50:59.543222+0000","last_unstale":"2021-02-10T04:51:09.602463+0000","last_undegraded":"2021-02-10T04:51:09.602463+0000","last_fullsized":"2021-02-10T04:51:09.602463+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_clean_scrub_stamp":"2021-02-10T04:51:00.195518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"1085","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.577328+0000","last_change":"2021-02-10T04:50:55.473455+0000","last_active":"2021-02-10T04:51:04.577328+0000","last_peered":"2021-02-10T04:51:04.577328+0000","last_clean":"2021-02-10T04:51:04.577328+0000","last_became_active":"2021-02-10T04:25:51.032201+0000","last_became_peered":"2021-02-10T04:25:51.032201+0000","last_unstale":"2021-02-10T04:51:04.577328+0000","last_undegraded":"2021-02-10T04:51:04.577328+0000","last_fullsized":"2021-02-10T04:51:04.577328+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:55.473347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:50:55.473347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"1049'40","reported_seq":"1180","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.596437+0000","last_change":"2021-02-10T04:46:18.777738+0000","last_active":"2021-02-10T04:51:04.596437+0000","last_peered":"2021-02-10T04:51:04.596437+0000","last_clean":"2021-02-10T04:51:04.596437+0000","last_became_active":"2021-02-10T04:37:37.247141+0000","last_became_peered":"2021-02-10T04:37:37.247141+0000","last_unstale":"2021-02-10T04:51:04.596437+0000","last_undegraded":"2021-02-10T04:51:04.596437+0000","last_fullsized":"2021-02-10T04:51:04.596437+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'40","last_scrub_stamp":"2021-02-10T04:46:18.777661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:18.777661+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":52,"num_write":40,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"824","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.596102+0000","last_change":"2021-02-10T04:50:50.399431+0000","last_active":"2021-02-10T04:51:04.596102+0000","last_peered":"2021-02-10T04:51:04.596102+0000","last_clean":"2021-02-10T04:51:04.596102+0000","last_became_active":"2021-02-10T04:33:05.166368+0000","last_became_peered":"2021-02-10T04:33:05.166368+0000","last_unstale":"2021-02-10T04:51:04.596102+0000","last_undegraded":"2021-02-10T04:51:04.596102+0000","last_fullsized":"2021-02-10T04:51:04.596102+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:50.399350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:50.399350+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"1102","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.596897+0000","last_change":"2021-02-10T04:50:59.577926+0000","last_active":"2021-02-10T04:51:04.596897+0000","last_peered":"2021-02-10T04:51:04.596897+0000","last_clean":"2021-02-10T04:51:04.596897+0000","last_became_active":"2021-02-10T04:50:59.576153+0000","last_became_peered":"2021-02-10T04:50:59.576153+0000","last_unstale":"2021-02-10T04:51:04.596897+0000","last_undegraded":"2021-02-10T04:51:04.596897+0000","last_fullsized":"2021-02-10T04:51:04.596897+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"787","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.576872+0000","last_change":"2021-02-10T04:50:59.595978+0000","last_active":"2021-02-10T04:51:04.576872+0000","last_peered":"2021-02-10T04:51:04.576872+0000","last_clean":"2021-02-10T04:51:04.576872+0000","last_became_active":"2021-02-10T04:50:59.595667+0000","last_became_peered":"2021-02-10T04:50:59.595667+0000","last_unstale":"2021-02-10T04:51:04.576872+0000","last_undegraded":"2021-02-10T04:51:04.576872+0000","last_fullsized":"2021-02-10T04:51:04.576872+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:17.598113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:17.598113+0000","last_clean_scrub_stamp":"2021-02-10T04:47:17.598113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.5","version":"1000'446","reported_seq":"2034","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.596226+0000","last_change":"2021-02-10T04:46:38.774890+0000","last_active":"2021-02-10T04:51:04.596226+0000","last_peered":"2021-02-10T04:51:04.596226+0000","last_clean":"2021-02-10T04:51:04.596226+0000","last_became_active":"2021-02-10T04:44:25.549930+0000","last_became_peered":"2021-02-10T04:44:25.549930+0000","last_unstale":"2021-02-10T04:51:04.596226+0000","last_undegraded":"2021-02-10T04:51:04.596226+0000","last_fullsized":"2021-02-10T04:51:04.596226+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:46:38.774795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:38.774795+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19615419,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"848","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.601547+0000","last_change":"2021-02-10T04:51:05.242796+0000","last_active":"2021-02-10T04:51:09.601547+0000","last_peered":"2021-02-10T04:51:09.601547+0000","last_clean":"2021-02-10T04:51:09.601547+0000","last_became_active":"2021-02-10T04:33:05.140966+0000","last_became_peered":"2021-02-10T04:33:05.140966+0000","last_unstale":"2021-02-10T04:51:09.601547+0000","last_undegraded":"2021-02-10T04:51:09.601547+0000","last_fullsized":"2021-02-10T04:51:09.601547+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:05.242690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:05.242690+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"500","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:10.398593+0000","last_change":"2021-02-10T04:51:10.398593+0000","last_active":"2021-02-10T04:51:10.398593+0000","last_peered":"2021-02-10T04:51:10.398593+0000","last_clean":"2021-02-10T04:51:10.398593+0000","last_became_active":"2021-02-10T04:37:37.234644+0000","last_became_peered":"2021-02-10T04:37:37.234644+0000","last_unstale":"2021-02-10T04:51:10.398593+0000","last_undegraded":"2021-02-10T04:51:10.398593+0000","last_fullsized":"2021-02-10T04:51:10.398593+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:10.398477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:10.398477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"1049'39","reported_seq":"1148","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.603053+0000","last_change":"2021-02-10T04:46:34.039385+0000","last_active":"2021-02-10T04:51:09.603053+0000","last_peered":"2021-02-10T04:51:09.603053+0000","last_clean":"2021-02-10T04:51:09.603053+0000","last_became_active":"2021-02-10T04:25:49.002471+0000","last_became_peered":"2021-02-10T04:25:49.002471+0000","last_unstale":"2021-02-10T04:51:09.603053+0000","last_undegraded":"2021-02-10T04:51:09.603053+0000","last_fullsized":"2021-02-10T04:51:09.603053+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'39","last_scrub_stamp":"2021-02-10T04:46:34.039273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:34.039273+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":51,"num_write":39,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"1132","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595745+0000","last_change":"2021-02-10T04:50:05.424349+0000","last_active":"2021-02-10T04:51:04.595745+0000","last_peered":"2021-02-10T04:51:04.595745+0000","last_clean":"2021-02-10T04:51:04.595745+0000","last_became_active":"2021-02-10T04:29:07.158075+0000","last_became_peered":"2021-02-10T04:29:07.158075+0000","last_unstale":"2021-02-10T04:51:04.595745+0000","last_undegraded":"2021-02-10T04:51:04.595745+0000","last_fullsized":"2021-02-10T04:51:04.595745+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_clean_scrub_stamp":"2021-02-10T04:50:05.424262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"1270","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.576142+0000","last_change":"2021-02-10T04:50:59.575837+0000","last_active":"2021-02-10T04:51:04.576142+0000","last_peered":"2021-02-10T04:51:04.576142+0000","last_clean":"2021-02-10T04:51:04.576142+0000","last_became_active":"2021-02-10T04:50:59.575479+0000","last_became_peered":"2021-02-10T04:50:59.575479+0000","last_unstale":"2021-02-10T04:51:04.576142+0000","last_undegraded":"2021-02-10T04:51:04.576142+0000","last_fullsized":"2021-02-10T04:51:04.576142+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:29.707329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:29.707329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"1077","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:08.002847+0000","last_change":"2021-02-10T04:51:08.002847+0000","last_active":"2021-02-10T04:51:08.002847+0000","last_peered":"2021-02-10T04:51:08.002847+0000","last_clean":"2021-02-10T04:51:08.002847+0000","last_became_active":"2021-02-10T04:50:59.587343+0000","last_became_peered":"2021-02-10T04:50:59.587343+0000","last_unstale":"2021-02-10T04:51:08.002847+0000","last_undegraded":"2021-02-10T04:51:08.002847+0000","last_fullsized":"2021-02-10T04:51:08.002847+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:17.537072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:17.537072+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"1107","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.577372+0000","last_change":"2021-02-10T04:50:59.538442+0000","last_active":"2021-02-10T04:51:04.577372+0000","last_peered":"2021-02-10T04:51:04.577372+0000","last_clean":"2021-02-10T04:51:04.577372+0000","last_became_active":"2021-02-10T04:50:59.538009+0000","last_became_peered":"2021-02-10T04:50:59.538009+0000","last_unstale":"2021-02-10T04:51:04.577372+0000","last_undegraded":"2021-02-10T04:51:04.577372+0000","last_fullsized":"2021-02-10T04:51:04.577372+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:19.637875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:19.637875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"1000'490","reported_seq":"3649","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.596487+0000","last_change":"2021-02-10T04:47:02.876009+0000","last_active":"2021-02-10T04:51:04.596487+0000","last_peered":"2021-02-10T04:51:04.596487+0000","last_clean":"2021-02-10T04:51:04.596487+0000","last_became_active":"2021-02-10T04:45:42.422424+0000","last_became_peered":"2021-02-10T04:45:42.422424+0000","last_unstale":"2021-02-10T04:51:04.596487+0000","last_undegraded":"2021-02-10T04:51:04.596487+0000","last_fullsized":"2021-02-10T04:51:04.596487+0000","mapping_epoch":1039,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1040,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:47:02.875920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:02.875920+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22588464,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"434","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.602093+0000","last_change":"2021-02-10T04:50:46.380356+0000","last_active":"2021-02-10T04:51:09.602093+0000","last_peered":"2021-02-10T04:51:09.602093+0000","last_clean":"2021-02-10T04:51:09.602093+0000","last_became_active":"2021-02-10T04:38:39.252266+0000","last_became_peered":"2021-02-10T04:38:39.252266+0000","last_unstale":"2021-02-10T04:51:09.602093+0000","last_undegraded":"2021-02-10T04:51:09.602093+0000","last_fullsized":"2021-02-10T04:51:09.602093+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:46.380285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:50:46.380285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"1049'37","reported_seq":"857","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.577545+0000","last_change":"2021-02-10T04:46:36.919290+0000","last_active":"2021-02-10T04:51:04.577545+0000","last_peered":"2021-02-10T04:51:04.577545+0000","last_clean":"2021-02-10T04:51:04.577545+0000","last_became_active":"2021-02-10T04:33:05.146695+0000","last_became_peered":"2021-02-10T04:33:05.146695+0000","last_unstale":"2021-02-10T04:51:04.577545+0000","last_undegraded":"2021-02-10T04:51:04.577545+0000","last_fullsized":"2021-02-10T04:51:04.577545+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:46:36.919205+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:46:36.919205+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"515","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.973588+0000","last_change":"2021-02-10T04:51:07.973588+0000","last_active":"2021-02-10T04:51:07.973588+0000","last_peered":"2021-02-10T04:51:07.973588+0000","last_clean":"2021-02-10T04:51:07.973588+0000","last_became_active":"2021-02-10T04:50:59.565239+0000","last_became_peered":"2021-02-10T04:50:59.565239+0000","last_unstale":"2021-02-10T04:51:07.973588+0000","last_undegraded":"2021-02-10T04:51:07.973588+0000","last_fullsized":"2021-02-10T04:51:07.973588+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:04.591695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:04.591695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"1340","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:13.409491+0000","last_change":"2021-02-10T04:46:53.011176+0000","last_active":"2021-02-10T04:51:13.409491+0000","last_peered":"2021-02-10T04:51:13.409491+0000","last_clean":"2021-02-10T04:51:13.409491+0000","last_became_active":"2021-02-10T04:37:37.249591+0000","last_became_peered":"2021-02-10T04:37:37.249591+0000","last_unstale":"2021-02-10T04:51:13.409491+0000","last_undegraded":"2021-02-10T04:51:13.409491+0000","last_fullsized":"2021-02-10T04:51:13.409491+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:46:53.011080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:53.011080+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":822,"num_read_kb":817,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"1035","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.602103+0000","last_change":"2021-02-10T04:50:59.529934+0000","last_active":"2021-02-10T04:51:09.602103+0000","last_peered":"2021-02-10T04:51:09.602103+0000","last_clean":"2021-02-10T04:51:09.602103+0000","last_became_active":"2021-02-10T04:50:59.529649+0000","last_became_peered":"2021-02-10T04:50:59.529649+0000","last_unstale":"2021-02-10T04:51:09.602103+0000","last_undegraded":"2021-02-10T04:51:09.602103+0000","last_fullsized":"2021-02-10T04:51:09.602103+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:13.991399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:47:13.991399+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.4","version":"1000'620","reported_seq":"4283","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595729+0000","last_change":"2021-02-10T04:46:40.753673+0000","last_active":"2021-02-10T04:51:04.595729+0000","last_peered":"2021-02-10T04:51:04.595729+0000","last_clean":"2021-02-10T04:51:04.595729+0000","last_became_active":"2021-02-10T04:44:25.537632+0000","last_became_peered":"2021-02-10T04:44:25.537632+0000","last_unstale":"2021-02-10T04:51:04.595729+0000","last_undegraded":"2021-02-10T04:51:04.595729+0000","last_fullsized":"2021-02-10T04:51:04.595729+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:46:40.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:40.753581+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"1100","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595928+0000","last_change":"2021-02-10T04:47:14.955640+0000","last_active":"2021-02-10T04:51:04.595928+0000","last_peered":"2021-02-10T04:51:04.595928+0000","last_clean":"2021-02-10T04:51:04.595928+0000","last_became_active":"2021-02-10T04:29:07.159607+0000","last_became_peered":"2021-02-10T04:29:07.159607+0000","last_unstale":"2021-02-10T04:51:04.595928+0000","last_undegraded":"2021-02-10T04:51:04.595928+0000","last_fullsized":"2021-02-10T04:51:04.595928+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:14.955544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:47:14.955544+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"1201","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.596674+0000","last_change":"2021-02-10T04:47:27.908308+0000","last_active":"2021-02-10T04:51:04.596674+0000","last_peered":"2021-02-10T04:51:04.596674+0000","last_clean":"2021-02-10T04:51:04.596674+0000","last_became_active":"2021-02-10T04:25:51.027624+0000","last_became_peered":"2021-02-10T04:25:51.027624+0000","last_unstale":"2021-02-10T04:51:04.596674+0000","last_undegraded":"2021-02-10T04:51:04.596674+0000","last_fullsized":"2021-02-10T04:51:04.596674+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:27.908221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:47:27.908221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"1049'55","reported_seq":"781","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.993003+0000","last_change":"2021-02-10T04:51:07.993003+0000","last_active":"2021-02-10T04:51:07.993003+0000","last_peered":"2021-02-10T04:51:07.993003+0000","last_clean":"2021-02-10T04:51:07.993003+0000","last_became_active":"2021-02-10T04:50:59.563323+0000","last_became_peered":"2021-02-10T04:50:59.563323+0000","last_unstale":"2021-02-10T04:51:07.993003+0000","last_undegraded":"2021-02-10T04:51:07.993003+0000","last_fullsized":"2021-02-10T04:51:07.993003+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'55","last_scrub_stamp":"2021-02-10T04:46:55.516761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:55.516761+0000","log_size":55,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":98,"num_read_kb":74,"num_write":58,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"1103","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.576307+0000","last_change":"2021-02-10T04:47:13.579252+0000","last_active":"2021-02-10T04:51:04.576307+0000","last_peered":"2021-02-10T04:51:04.576307+0000","last_clean":"2021-02-10T04:51:04.576307+0000","last_became_active":"2021-02-10T04:28:21.125541+0000","last_became_peered":"2021-02-10T04:28:21.125541+0000","last_unstale":"2021-02-10T04:51:04.576307+0000","last_undegraded":"2021-02-10T04:51:04.576307+0000","last_fullsized":"2021-02-10T04:51:04.576307+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:13.579163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:13.579163+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"1070","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.984424+0000","last_change":"2021-02-10T04:51:07.984424+0000","last_active":"2021-02-10T04:51:07.984424+0000","last_peered":"2021-02-10T04:51:07.984424+0000","last_clean":"2021-02-10T04:51:07.984424+0000","last_became_active":"2021-02-10T04:50:59.555689+0000","last_became_peered":"2021-02-10T04:50:59.555689+0000","last_unstale":"2021-02-10T04:51:07.984424+0000","last_undegraded":"2021-02-10T04:51:07.984424+0000","last_fullsized":"2021-02-10T04:51:07.984424+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.395943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.395943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"1002'2042","reported_seq":"9330","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.602437+0000","last_change":"2021-02-10T04:50:59.593597+0000","last_active":"2021-02-10T04:51:09.602437+0000","last_peered":"2021-02-10T04:51:09.602437+0000","last_clean":"2021-02-10T04:51:09.602437+0000","last_became_active":"2021-02-10T04:50:59.591769+0000","last_became_peered":"2021-02-10T04:50:59.591769+0000","last_unstale":"2021-02-10T04:51:09.602437+0000","last_undegraded":"2021-02-10T04:51:09.602437+0000","last_fullsized":"2021-02-10T04:51:09.602437+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:46:57.054230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:57.054230+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":85678898,"num_objects":35,"num_object_clones":29,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"1035","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.575834+0000","last_change":"2021-02-10T04:50:52.341596+0000","last_active":"2021-02-10T04:51:04.575834+0000","last_peered":"2021-02-10T04:51:04.575834+0000","last_clean":"2021-02-10T04:51:04.575834+0000","last_became_active":"2021-02-10T04:37:37.219422+0000","last_became_peered":"2021-02-10T04:37:37.219422+0000","last_unstale":"2021-02-10T04:51:04.575834+0000","last_undegraded":"2021-02-10T04:51:04.575834+0000","last_fullsized":"2021-02-10T04:51:04.575834+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:52.341527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:50:52.341527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"58'2","reported_seq":"1396","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.988252+0000","last_change":"2021-02-10T04:50:54.466333+0000","last_active":"2021-02-10T04:51:07.988252+0000","last_peered":"2021-02-10T04:51:07.988252+0000","last_clean":"2021-02-10T04:51:07.988252+0000","last_became_active":"2021-02-10T04:25:51.026251+0000","last_became_peered":"2021-02-10T04:25:51.026251+0000","last_unstale":"2021-02-10T04:51:07.988252+0000","last_undegraded":"2021-02-10T04:51:07.988252+0000","last_fullsized":"2021-02-10T04:51:07.988252+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:54.466259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:54.466259+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"1049'54","reported_seq":"873","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.576515+0000","last_change":"2021-02-10T04:50:14.158728+0000","last_active":"2021-02-10T04:51:04.576515+0000","last_peered":"2021-02-10T04:51:04.576515+0000","last_clean":"2021-02-10T04:51:04.576515+0000","last_became_active":"2021-02-10T04:33:05.141901+0000","last_became_peered":"2021-02-10T04:33:05.141901+0000","last_unstale":"2021-02-10T04:51:04.576515+0000","last_undegraded":"2021-02-10T04:51:04.576515+0000","last_fullsized":"2021-02-10T04:51:04.576515+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:50:14.158659+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:50:14.158659+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":75,"num_read_kb":57,"num_write":42,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"850","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595505+0000","last_change":"2021-02-10T04:50:49.448042+0000","last_active":"2021-02-10T04:51:04.595505+0000","last_peered":"2021-02-10T04:51:04.595505+0000","last_clean":"2021-02-10T04:51:04.595505+0000","last_became_active":"2021-02-10T04:37:37.247405+0000","last_became_peered":"2021-02-10T04:37:37.247405+0000","last_unstale":"2021-02-10T04:51:04.595505+0000","last_undegraded":"2021-02-10T04:51:04.595505+0000","last_fullsized":"2021-02-10T04:51:04.595505+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:49.447961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:49.447961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"1116","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595575+0000","last_change":"2021-02-10T04:50:59.577707+0000","last_active":"2021-02-10T04:51:04.595575+0000","last_peered":"2021-02-10T04:51:04.595575+0000","last_clean":"2021-02-10T04:51:04.595575+0000","last_became_active":"2021-02-10T04:50:59.577351+0000","last_became_peered":"2021-02-10T04:50:59.577351+0000","last_unstale":"2021-02-10T04:51:04.595575+0000","last_undegraded":"2021-02-10T04:51:04.595575+0000","last_fullsized":"2021-02-10T04:51:04.595575+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:07.914061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:07.914061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"293'77","reported_seq":"2884","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595067+0000","last_change":"2021-02-10T04:50:59.527961+0000","last_active":"2021-02-10T04:51:04.595067+0000","last_peered":"2021-02-10T04:51:04.595067+0000","last_clean":"2021-02-10T04:51:04.595067+0000","last_became_active":"2021-02-10T04:50:59.526860+0000","last_became_peered":"2021-02-10T04:50:59.526860+0000","last_unstale":"2021-02-10T04:51:04.595067+0000","last_undegraded":"2021-02-10T04:51:04.595067+0000","last_fullsized":"2021-02-10T04:51:04.595067+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3773171,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":36,"num_read_kb":37174,"num_write":18,"num_write_kb":6631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"1123","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.602354+0000","last_change":"2021-02-10T04:50:59.589648+0000","last_active":"2021-02-10T04:51:09.602354+0000","last_peered":"2021-02-10T04:51:09.602354+0000","last_clean":"2021-02-10T04:51:09.602354+0000","last_became_active":"2021-02-10T04:50:59.589417+0000","last_became_peered":"2021-02-10T04:50:59.589417+0000","last_unstale":"2021-02-10T04:51:09.602354+0000","last_undegraded":"2021-02-10T04:51:09.602354+0000","last_fullsized":"2021-02-10T04:51:09.602354+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:09.869777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:47:09.869777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"521","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.609699+0000","last_change":"2021-02-10T04:51:07.609624+0000","last_active":"2021-02-10T04:51:07.609699+0000","last_peered":"2021-02-10T04:51:07.609699+0000","last_clean":"2021-02-10T04:51:07.609699+0000","last_became_active":"2021-02-10T04:50:59.576664+0000","last_became_peered":"2021-02-10T04:50:59.576664+0000","last_unstale":"2021-02-10T04:51:07.609699+0000","last_undegraded":"2021-02-10T04:51:07.609699+0000","last_fullsized":"2021-02-10T04:51:07.609699+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.609500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.609500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"1048'39","reported_seq":"1204","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595283+0000","last_change":"2021-02-10T04:50:44.624567+0000","last_active":"2021-02-10T04:51:04.595283+0000","last_peered":"2021-02-10T04:51:04.595283+0000","last_clean":"2021-02-10T04:51:04.595283+0000","last_became_active":"2021-02-10T04:39:02.560739+0000","last_became_peered":"2021-02-10T04:39:02.560739+0000","last_unstale":"2021-02-10T04:51:04.595283+0000","last_undegraded":"2021-02-10T04:51:04.595283+0000","last_fullsized":"2021-02-10T04:51:04.595283+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'39","last_scrub_stamp":"2021-02-10T04:50:44.624501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:50:44.624501+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":76,"num_read_kb":59,"num_write":41,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"1212","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.596062+0000","last_change":"2021-02-10T04:47:08.784658+0000","last_active":"2021-02-10T04:51:04.596062+0000","last_peered":"2021-02-10T04:51:04.596062+0000","last_clean":"2021-02-10T04:51:04.596062+0000","last_became_active":"2021-02-10T04:37:37.238701+0000","last_became_peered":"2021-02-10T04:37:37.238701+0000","last_unstale":"2021-02-10T04:51:04.596062+0000","last_undegraded":"2021-02-10T04:51:04.596062+0000","last_fullsized":"2021-02-10T04:51:04.596062+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:47:08.784593+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:47:08.784593+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"1072","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.995649+0000","last_change":"2021-02-10T04:51:07.995649+0000","last_active":"2021-02-10T04:51:07.995649+0000","last_peered":"2021-02-10T04:51:07.995649+0000","last_clean":"2021-02-10T04:51:07.995649+0000","last_became_active":"2021-02-10T04:50:59.561436+0000","last_became_peered":"2021-02-10T04:50:59.561436+0000","last_unstale":"2021-02-10T04:51:07.995649+0000","last_undegraded":"2021-02-10T04:51:07.995649+0000","last_fullsized":"2021-02-10T04:51:07.995649+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:15.536661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:15.536661+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"1001'1528","reported_seq":"5735","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.982619+0000","last_change":"2021-02-10T04:51:07.982619+0000","last_active":"2021-02-10T04:51:07.982619+0000","last_peered":"2021-02-10T04:51:07.982619+0000","last_clean":"2021-02-10T04:51:07.982619+0000","last_became_active":"2021-02-10T04:50:59.587009+0000","last_became_peered":"2021-02-10T04:50:59.587009+0000","last_unstale":"2021-02-10T04:51:07.982619+0000","last_undegraded":"2021-02-10T04:51:07.982619+0000","last_fullsized":"2021-02-10T04:51:07.982619+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":93888272,"num_objects":33,"num_object_clones":30,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":1,"num_read":716,"num_read_kb":703869,"num_write":572,"num_write_kb":222710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":42623785,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.15","version":"1000'446","reported_seq":"1985","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595837+0000","last_change":"2021-02-10T04:50:05.815243+0000","last_active":"2021-02-10T04:51:04.595837+0000","last_peered":"2021-02-10T04:51:04.595837+0000","last_clean":"2021-02-10T04:51:04.595837+0000","last_became_active":"2021-02-10T04:45:43.499003+0000","last_became_peered":"2021-02-10T04:45:43.499003+0000","last_unstale":"2021-02-10T04:51:04.595837+0000","last_undegraded":"2021-02-10T04:51:04.595837+0000","last_fullsized":"2021-02-10T04:51:04.595837+0000","mapping_epoch":1040,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1041,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:50:05.815135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:50:05.815135+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24896285,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"1048'40","reported_seq":"1180","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.596538+0000","last_change":"2021-02-10T04:50:59.578612+0000","last_active":"2021-02-10T04:51:04.596538+0000","last_peered":"2021-02-10T04:51:04.596538+0000","last_clean":"2021-02-10T04:51:04.596538+0000","last_became_active":"2021-02-10T04:50:59.578193+0000","last_became_peered":"2021-02-10T04:50:59.578193+0000","last_unstale":"2021-02-10T04:51:04.596538+0000","last_undegraded":"2021-02-10T04:51:04.596538+0000","last_fullsized":"2021-02-10T04:51:04.596538+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'40","last_scrub_stamp":"2021-02-10T04:47:29.965140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:47:29.965140+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":80,"num_read_kb":58,"num_write":41,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"515","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:08.639942+0000","last_change":"2021-02-10T04:51:08.639876+0000","last_active":"2021-02-10T04:51:08.639942+0000","last_peered":"2021-02-10T04:51:08.639942+0000","last_clean":"2021-02-10T04:51:08.639942+0000","last_became_active":"2021-02-10T04:37:37.234649+0000","last_became_peered":"2021-02-10T04:37:37.234649+0000","last_unstale":"2021-02-10T04:51:08.639942+0000","last_undegraded":"2021-02-10T04:51:08.639942+0000","last_fullsized":"2021-02-10T04:51:08.639942+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:51:08.639784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.639784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":247,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"1046","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:09.475198+0000","last_change":"2021-02-10T04:51:09.475150+0000","last_active":"2021-02-10T04:51:09.475198+0000","last_peered":"2021-02-10T04:51:09.475198+0000","last_clean":"2021-02-10T04:51:09.475198+0000","last_became_active":"2021-02-10T04:50:59.585917+0000","last_became_peered":"2021-02-10T04:50:59.585917+0000","last_unstale":"2021-02-10T04:51:09.475198+0000","last_undegraded":"2021-02-10T04:51:09.475198+0000","last_fullsized":"2021-02-10T04:51:09.475198+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:09.475063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:09.475063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"491","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595408+0000","last_change":"2021-02-10T04:46:27.889189+0000","last_active":"2021-02-10T04:51:04.595408+0000","last_peered":"2021-02-10T04:51:04.595408+0000","last_clean":"2021-02-10T04:51:04.595408+0000","last_became_active":"2021-02-10T04:37:37.238049+0000","last_became_peered":"2021-02-10T04:37:37.238049+0000","last_unstale":"2021-02-10T04:51:04.595408+0000","last_undegraded":"2021-02-10T04:51:04.595408+0000","last_fullsized":"2021-02-10T04:51:04.595408+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:27.889132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:27.889132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"1000'620","reported_seq":"4259","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.977588+0000","last_change":"2021-02-10T04:51:07.977588+0000","last_active":"2021-02-10T04:51:07.977588+0000","last_peered":"2021-02-10T04:51:07.977588+0000","last_clean":"2021-02-10T04:51:07.977588+0000","last_became_active":"2021-02-10T04:50:59.591216+0000","last_became_peered":"2021-02-10T04:50:59.591216+0000","last_unstale":"2021-02-10T04:51:07.977588+0000","last_undegraded":"2021-02-10T04:51:07.977588+0000","last_fullsized":"2021-02-10T04:51:07.977588+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:47:24.897680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:24.897680+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14265195,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":144,"num_read_kb":128483,"num_write":129,"num_write_kb":53444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":19999269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"1063'30","reported_seq":"1141","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.995751+0000","last_change":"2021-02-10T04:51:07.995751+0000","last_active":"2021-02-10T04:51:07.995751+0000","last_peered":"2021-02-10T04:51:07.995751+0000","last_clean":"2021-02-10T04:51:07.995751+0000","last_became_active":"2021-02-10T04:50:59.587330+0000","last_became_peered":"2021-02-10T04:50:59.587330+0000","last_unstale":"2021-02-10T04:51:07.995751+0000","last_undegraded":"2021-02-10T04:51:07.995751+0000","last_fullsized":"2021-02-10T04:51:07.995751+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1063'30","last_scrub_stamp":"2021-02-10T04:47:05.589582+0000","last_deep_scrub":"1063'30","last_deep_scrub_stamp":"2021-02-10T04:47:05.589582+0000","last_clean_scrub_stamp":"2021-02-10T04:47:05.589582+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"1097","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.602184+0000","last_change":"2021-02-10T04:46:55.046228+0000","last_active":"2021-02-10T04:51:09.602184+0000","last_peered":"2021-02-10T04:51:09.602184+0000","last_clean":"2021-02-10T04:51:09.602184+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:51:09.602184+0000","last_undegraded":"2021-02-10T04:51:09.602184+0000","last_fullsized":"2021-02-10T04:51:09.602184+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:46:55.046158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:55.046158+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"1100","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:08.451405+0000","last_change":"2021-02-10T04:51:08.451362+0000","last_active":"2021-02-10T04:51:08.451405+0000","last_peered":"2021-02-10T04:51:08.451405+0000","last_clean":"2021-02-10T04:51:08.451405+0000","last_became_active":"2021-02-10T04:50:59.540375+0000","last_became_peered":"2021-02-10T04:50:59.540375+0000","last_unstale":"2021-02-10T04:51:08.451405+0000","last_undegraded":"2021-02-10T04:51:08.451405+0000","last_fullsized":"2021-02-10T04:51:08.451405+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:08.451289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.451289+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"1011","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.602305+0000","last_change":"2021-02-10T04:50:45.362633+0000","last_active":"2021-02-10T04:51:09.602305+0000","last_peered":"2021-02-10T04:51:09.602305+0000","last_clean":"2021-02-10T04:51:09.602305+0000","last_became_active":"2021-02-10T04:29:07.147249+0000","last_became_peered":"2021-02-10T04:29:07.147249+0000","last_unstale":"2021-02-10T04:51:09.602305+0000","last_undegraded":"2021-02-10T04:51:09.602305+0000","last_fullsized":"2021-02-10T04:51:09.602305+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:45.362537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:45.362537+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.17","version":"1002'2042","reported_seq":"9273","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.577052+0000","last_change":"2021-02-10T04:47:16.641750+0000","last_active":"2021-02-10T04:51:04.577052+0000","last_peered":"2021-02-10T04:51:04.577052+0000","last_clean":"2021-02-10T04:51:04.577052+0000","last_became_active":"2021-02-10T04:44:30.657433+0000","last_became_peered":"2021-02-10T04:44:30.657433+0000","last_unstale":"2021-02-10T04:51:04.577052+0000","last_undegraded":"2021-02-10T04:51:04.577052+0000","last_fullsized":"2021-02-10T04:51:04.577052+0000","mapping_epoch":1012,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1013,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:47:16.641616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:16.641616+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6921254,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"1049'46","reported_seq":"1265","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.988787+0000","last_change":"2021-02-10T04:51:07.988787+0000","last_active":"2021-02-10T04:51:07.988787+0000","last_peered":"2021-02-10T04:51:07.988787+0000","last_clean":"2021-02-10T04:51:07.988787+0000","last_became_active":"2021-02-10T04:50:59.569915+0000","last_became_peered":"2021-02-10T04:50:59.569915+0000","last_unstale":"2021-02-10T04:51:07.988787+0000","last_undegraded":"2021-02-10T04:51:07.988787+0000","last_fullsized":"2021-02-10T04:51:07.988787+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'46","last_scrub_stamp":"2021-02-10T04:46:56.512292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:56.512292+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":65,"num_write":46,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"84'7","reported_seq":"1124","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.576228+0000","last_change":"2021-02-10T04:50:59.576334+0000","last_active":"2021-02-10T04:51:04.576228+0000","last_peered":"2021-02-10T04:51:04.576228+0000","last_clean":"2021-02-10T04:51:04.576228+0000","last_became_active":"2021-02-10T04:50:59.575924+0000","last_became_peered":"2021-02-10T04:50:59.575924+0000","last_unstale":"2021-02-10T04:51:04.576228+0000","last_undegraded":"2021-02-10T04:51:04.576228+0000","last_fullsized":"2021-02-10T04:51:04.576228+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'7","last_scrub_stamp":"2021-02-10T04:46:49.869504+0000","last_deep_scrub":"84'7","last_deep_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_clean_scrub_stamp":"2021-02-10T04:46:49.869504+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"58'2","reported_seq":"1392","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.602512+0000","last_change":"2021-02-10T04:50:47.407626+0000","last_active":"2021-02-10T04:51:09.602512+0000","last_peered":"2021-02-10T04:51:09.602512+0000","last_clean":"2021-02-10T04:51:09.602512+0000","last_became_active":"2021-02-10T04:25:51.032328+0000","last_became_peered":"2021-02-10T04:25:51.032328+0000","last_unstale":"2021-02-10T04:51:09.602512+0000","last_undegraded":"2021-02-10T04:51:09.602512+0000","last_fullsized":"2021-02-10T04:51:09.602512+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:47.407557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:47.407557+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"1098","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.596358+0000","last_change":"2021-02-10T04:50:59.576884+0000","last_active":"2021-02-10T04:51:04.596358+0000","last_peered":"2021-02-10T04:51:04.596358+0000","last_clean":"2021-02-10T04:51:04.596358+0000","last_became_active":"2021-02-10T04:50:59.576536+0000","last_became_peered":"2021-02-10T04:50:59.576536+0000","last_unstale":"2021-02-10T04:51:04.596358+0000","last_undegraded":"2021-02-10T04:51:04.596358+0000","last_fullsized":"2021-02-10T04:51:04.596358+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:33.677438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_clean_scrub_stamp":"2021-02-10T04:46:33.677438+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"1000'490","reported_seq":"3615","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595916+0000","last_change":"2021-02-10T04:50:03.756349+0000","last_active":"2021-02-10T04:51:04.595916+0000","last_peered":"2021-02-10T04:51:04.595916+0000","last_clean":"2021-02-10T04:51:04.595916+0000","last_became_active":"2021-02-10T04:44:30.702606+0000","last_became_peered":"2021-02-10T04:44:30.702606+0000","last_unstale":"2021-02-10T04:51:04.595916+0000","last_undegraded":"2021-02-10T04:51:04.595916+0000","last_fullsized":"2021-02-10T04:51:04.595916+0000","mapping_epoch":1012,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1013,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:50:03.756248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:50:03.756248+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16154790,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"1112'40","reported_seq":"1168","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.577294+0000","last_change":"2021-02-10T04:51:01.450124+0000","last_active":"2021-02-10T04:51:04.577294+0000","last_peered":"2021-02-10T04:51:04.577294+0000","last_clean":"2021-02-10T04:51:04.577294+0000","last_became_active":"2021-02-10T04:50:59.539529+0000","last_became_peered":"2021-02-10T04:50:59.539529+0000","last_unstale":"2021-02-10T04:51:04.577294+0000","last_undegraded":"2021-02-10T04:51:04.577294+0000","last_fullsized":"2021-02-10T04:51:04.577294+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1112'40","last_scrub_stamp":"2021-02-10T04:51:01.450050+0000","last_deep_scrub":"1112'40","last_deep_scrub_stamp":"2021-02-10T04:51:01.450050+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.450050+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":59,"num_read_kb":44,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"1114","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.596158+0000","last_change":"2021-02-10T04:50:59.579777+0000","last_active":"2021-02-10T04:51:04.596158+0000","last_peered":"2021-02-10T04:51:04.596158+0000","last_clean":"2021-02-10T04:51:04.596158+0000","last_became_active":"2021-02-10T04:50:59.579485+0000","last_became_peered":"2021-02-10T04:50:59.579485+0000","last_unstale":"2021-02-10T04:51:04.596158+0000","last_undegraded":"2021-02-10T04:51:04.596158+0000","last_fullsized":"2021-02-10T04:51:04.596158+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:23.904210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:47:23.904210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"1113'5","reported_seq":"1303","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.997635+0000","last_change":"2021-02-10T04:51:07.996294+0000","last_active":"2021-02-10T04:51:07.997635+0000","last_peered":"2021-02-10T04:51:07.997635+0000","last_clean":"2021-02-10T04:51:07.997635+0000","last_became_active":"2021-02-10T04:50:59.581263+0000","last_became_peered":"2021-02-10T04:50:59.581263+0000","last_unstale":"2021-02-10T04:51:07.997635+0000","last_undegraded":"2021-02-10T04:51:07.997635+0000","last_fullsized":"2021-02-10T04:51:07.997635+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"639'3","last_scrub_stamp":"2021-02-10T04:46:39.502896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:46:39.502896+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"1091","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:11.779876+0000","last_change":"2021-02-10T04:51:11.779876+0000","last_active":"2021-02-10T04:51:11.779876+0000","last_peered":"2021-02-10T04:51:11.779876+0000","last_clean":"2021-02-10T04:51:11.779876+0000","last_became_active":"2021-02-10T04:37:37.238360+0000","last_became_peered":"2021-02-10T04:37:37.238360+0000","last_unstale":"2021-02-10T04:51:11.779876+0000","last_undegraded":"2021-02-10T04:51:11.779876+0000","last_fullsized":"2021-02-10T04:51:11.779876+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:11.779779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:11.779779+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"1032'89","reported_seq":"880","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.602142+0000","last_change":"2021-02-10T04:50:59.528845+0000","last_active":"2021-02-10T04:51:09.602142+0000","last_peered":"2021-02-10T04:51:09.602142+0000","last_clean":"2021-02-10T04:51:09.602142+0000","last_became_active":"2021-02-10T04:50:59.528302+0000","last_became_peered":"2021-02-10T04:50:59.528302+0000","last_unstale":"2021-02-10T04:51:09.602142+0000","last_undegraded":"2021-02-10T04:51:09.602142+0000","last_fullsized":"2021-02-10T04:51:09.602142+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":5,"last_scrub":"1032'89","last_scrub_stamp":"2021-02-10T04:47:00.950114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:00.950114+0000","log_size":89,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5949711,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"1048'27","reported_seq":"859","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.601361+0000","last_change":"2021-02-10T04:50:55.142571+0000","last_active":"2021-02-10T04:51:09.601361+0000","last_peered":"2021-02-10T04:51:09.601361+0000","last_clean":"2021-02-10T04:51:09.601361+0000","last_became_active":"2021-02-10T04:37:37.249007+0000","last_became_peered":"2021-02-10T04:37:37.249007+0000","last_unstale":"2021-02-10T04:51:09.601361+0000","last_undegraded":"2021-02-10T04:51:09.601361+0000","last_fullsized":"2021-02-10T04:51:09.601361+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'27","last_scrub_stamp":"2021-02-10T04:50:55.142479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:50:55.142479+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":34,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"1110","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.602006+0000","last_change":"2021-02-10T04:50:59.543884+0000","last_active":"2021-02-10T04:51:09.602006+0000","last_peered":"2021-02-10T04:51:09.602006+0000","last_clean":"2021-02-10T04:51:09.602006+0000","last_became_active":"2021-02-10T04:50:59.543531+0000","last_became_peered":"2021-02-10T04:50:59.543531+0000","last_unstale":"2021-02-10T04:51:09.602006+0000","last_undegraded":"2021-02-10T04:51:09.602006+0000","last_fullsized":"2021-02-10T04:51:09.602006+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:42.217450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:42.217450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"1043","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.973746+0000","last_change":"2021-02-10T04:51:07.973746+0000","last_active":"2021-02-10T04:51:07.973746+0000","last_peered":"2021-02-10T04:51:07.973746+0000","last_clean":"2021-02-10T04:51:07.973746+0000","last_became_active":"2021-02-10T04:50:59.567699+0000","last_became_peered":"2021-02-10T04:50:59.567699+0000","last_unstale":"2021-02-10T04:51:07.973746+0000","last_undegraded":"2021-02-10T04:51:07.973746+0000","last_fullsized":"2021-02-10T04:51:07.973746+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:45.524973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:44:45.524973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"1088","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595560+0000","last_change":"2021-02-10T04:50:57.568614+0000","last_active":"2021-02-10T04:51:04.595560+0000","last_peered":"2021-02-10T04:51:04.595560+0000","last_clean":"2021-02-10T04:51:04.595560+0000","last_became_active":"2021-02-10T04:25:53.039322+0000","last_became_peered":"2021-02-10T04:25:53.039322+0000","last_unstale":"2021-02-10T04:51:04.595560+0000","last_undegraded":"2021-02-10T04:51:04.595560+0000","last_fullsized":"2021-02-10T04:51:04.595560+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:57.568541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:57.568541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"1000'1071","reported_seq":"4598","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.975248+0000","last_change":"2021-02-10T04:51:07.975248+0000","last_active":"2021-02-10T04:51:07.975248+0000","last_peered":"2021-02-10T04:51:07.975248+0000","last_clean":"2021-02-10T04:51:07.975248+0000","last_became_active":"2021-02-10T04:50:59.570053+0000","last_became_peered":"2021-02-10T04:50:59.570053+0000","last_unstale":"2021-02-10T04:51:07.975248+0000","last_undegraded":"2021-02-10T04:51:07.975248+0000","last_fullsized":"2021-02-10T04:51:07.975248+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":5,"last_scrub":"412'357","last_scrub_stamp":"2021-02-10T04:33:50.582002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:50.582002+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23286368,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":242,"num_read_kb":226835,"num_write":185,"num_write_kb":76005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"1049'31","reported_seq":"1132","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595377+0000","last_change":"2021-02-10T04:47:15.739807+0000","last_active":"2021-02-10T04:51:04.595377+0000","last_peered":"2021-02-10T04:51:04.595377+0000","last_clean":"2021-02-10T04:51:04.595377+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:51:04.595377+0000","last_undegraded":"2021-02-10T04:51:04.595377+0000","last_fullsized":"2021-02-10T04:51:04.595377+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'31","last_scrub_stamp":"2021-02-10T04:47:15.739700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:47:15.739700+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":41,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"779","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.577351+0000","last_change":"2021-02-10T04:47:08.700162+0000","last_active":"2021-02-10T04:51:04.577351+0000","last_peered":"2021-02-10T04:51:04.577351+0000","last_clean":"2021-02-10T04:51:04.577351+0000","last_became_active":"2021-02-10T04:33:05.147956+0000","last_became_peered":"2021-02-10T04:33:05.147956+0000","last_unstale":"2021-02-10T04:51:04.577351+0000","last_undegraded":"2021-02-10T04:51:04.577351+0000","last_fullsized":"2021-02-10T04:51:04.577351+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:47:08.700069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:08.700069+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"783","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:05.475299+0000","last_change":"2021-02-10T04:51:05.475233+0000","last_active":"2021-02-10T04:51:05.475299+0000","last_peered":"2021-02-10T04:51:05.475299+0000","last_clean":"2021-02-10T04:51:05.475299+0000","last_became_active":"2021-02-10T04:50:59.575796+0000","last_became_peered":"2021-02-10T04:50:59.575796+0000","last_unstale":"2021-02-10T04:51:05.475299+0000","last_undegraded":"2021-02-10T04:51:05.475299+0000","last_fullsized":"2021-02-10T04:51:05.475299+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:05.475145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:51:05.475145+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"1072","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.979066+0000","last_change":"2021-02-10T04:51:07.979066+0000","last_active":"2021-02-10T04:51:07.979066+0000","last_peered":"2021-02-10T04:51:07.979066+0000","last_clean":"2021-02-10T04:51:07.979066+0000","last_became_active":"2021-02-10T04:50:59.557657+0000","last_became_peered":"2021-02-10T04:50:59.557657+0000","last_unstale":"2021-02-10T04:51:07.979066+0000","last_undegraded":"2021-02-10T04:51:07.979066+0000","last_fullsized":"2021-02-10T04:51:07.979066+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:09.627905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:09.627905+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"1002'1657","reported_seq":"5032","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595672+0000","last_change":"2021-02-10T04:46:54.881511+0000","last_active":"2021-02-10T04:51:04.595672+0000","last_peered":"2021-02-10T04:51:04.595672+0000","last_clean":"2021-02-10T04:51:04.595672+0000","last_became_active":"2021-02-10T04:44:35.600743+0000","last_became_peered":"2021-02-10T04:44:35.600743+0000","last_unstale":"2021-02-10T04:51:04.595672+0000","last_undegraded":"2021-02-10T04:51:04.595672+0000","last_fullsized":"2021-02-10T04:51:04.595672+0000","mapping_epoch":1015,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1016,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:46:54.881412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:54.881412+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":47335497,"num_objects":20,"num_object_clones":18,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"1048'20","reported_seq":"1211","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:08.002869+0000","last_change":"2021-02-10T04:51:08.002869+0000","last_active":"2021-02-10T04:51:08.002869+0000","last_peered":"2021-02-10T04:51:08.002869+0000","last_clean":"2021-02-10T04:51:08.002869+0000","last_became_active":"2021-02-10T04:50:59.544484+0000","last_became_peered":"2021-02-10T04:50:59.544484+0000","last_unstale":"2021-02-10T04:51:08.002869+0000","last_undegraded":"2021-02-10T04:51:08.002869+0000","last_fullsized":"2021-02-10T04:51:08.002869+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'20","last_scrub_stamp":"2021-02-10T04:47:12.728019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:47:12.728019+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":20,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"1103","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.577195+0000","last_change":"2021-02-10T04:48:54.943305+0000","last_active":"2021-02-10T04:51:04.577195+0000","last_peered":"2021-02-10T04:51:04.577195+0000","last_clean":"2021-02-10T04:51:04.577195+0000","last_became_active":"2021-02-10T04:37:37.229852+0000","last_became_peered":"2021-02-10T04:37:37.229852+0000","last_unstale":"2021-02-10T04:51:04.577195+0000","last_undegraded":"2021-02-10T04:51:04.577195+0000","last_fullsized":"2021-02-10T04:51:04.577195+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:48:54.943234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:48:54.943234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"1086","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.602053+0000","last_change":"2021-02-10T04:50:56.099710+0000","last_active":"2021-02-10T04:51:09.602053+0000","last_peered":"2021-02-10T04:51:09.602053+0000","last_clean":"2021-02-10T04:51:09.602053+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:51:09.602053+0000","last_undegraded":"2021-02-10T04:51:09.602053+0000","last_fullsized":"2021-02-10T04:51:09.602053+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:56.099651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:50:56.099651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"509","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595138+0000","last_change":"2021-02-10T04:50:59.570603+0000","last_active":"2021-02-10T04:51:04.595138+0000","last_peered":"2021-02-10T04:51:04.595138+0000","last_clean":"2021-02-10T04:51:04.595138+0000","last_became_active":"2021-02-10T04:50:59.569846+0000","last_became_peered":"2021-02-10T04:50:59.569846+0000","last_unstale":"2021-02-10T04:51:04.595138+0000","last_undegraded":"2021-02-10T04:51:04.595138+0000","last_fullsized":"2021-02-10T04:51:04.595138+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:37.748261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:46:37.748261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.12","version":"998'1009","reported_seq":"4772","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.600634+0000","last_change":"2021-02-10T04:47:13.032602+0000","last_active":"2021-02-10T04:51:09.600634+0000","last_peered":"2021-02-10T04:51:09.600634+0000","last_clean":"2021-02-10T04:51:09.600634+0000","last_became_active":"2021-02-10T04:45:45.516982+0000","last_became_peered":"2021-02-10T04:45:45.516982+0000","last_unstale":"2021-02-10T04:51:09.600634+0000","last_undegraded":"2021-02-10T04:51:09.600634+0000","last_fullsized":"2021-02-10T04:51:09.600634+0000","mapping_epoch":1042,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1043,"parent":"0.0","parent_split_bits":5,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:13.032506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:13.032506+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49420827,"num_objects":21,"num_object_clones":19,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"1049'37","reported_seq":"1139","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:08.006429+0000","last_change":"2021-02-10T04:51:08.006429+0000","last_active":"2021-02-10T04:51:08.006429+0000","last_peered":"2021-02-10T04:51:08.006429+0000","last_clean":"2021-02-10T04:51:08.006429+0000","last_became_active":"2021-02-10T04:50:59.549412+0000","last_became_peered":"2021-02-10T04:50:59.549412+0000","last_unstale":"2021-02-10T04:51:08.006429+0000","last_undegraded":"2021-02-10T04:51:08.006429+0000","last_fullsized":"2021-02-10T04:51:08.006429+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'29","last_scrub_stamp":"2021-02-10T04:44:44.573821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:44:44.573821+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":64,"num_read_kb":50,"num_write":37,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"1235","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.602762+0000","last_change":"2021-02-10T04:46:44.241805+0000","last_active":"2021-02-10T04:51:09.602762+0000","last_peered":"2021-02-10T04:51:09.602762+0000","last_clean":"2021-02-10T04:51:09.602762+0000","last_became_active":"2021-02-10T04:37:37.228840+0000","last_became_peered":"2021-02-10T04:37:37.228840+0000","last_unstale":"2021-02-10T04:51:09.602762+0000","last_undegraded":"2021-02-10T04:51:09.602762+0000","last_fullsized":"2021-02-10T04:51:09.602762+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:46:44.241683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:44.241683+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"395","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.988411+0000","last_change":"2021-02-10T04:51:07.988411+0000","last_active":"2021-02-10T04:51:07.988411+0000","last_peered":"2021-02-10T04:51:07.988411+0000","last_clean":"2021-02-10T04:51:07.988411+0000","last_became_active":"2021-02-10T04:50:59.564977+0000","last_became_peered":"2021-02-10T04:50:59.564977+0000","last_unstale":"2021-02-10T04:51:07.988411+0000","last_undegraded":"2021-02-10T04:51:07.988411+0000","last_fullsized":"2021-02-10T04:51:07.988411+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:54.669597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:44:54.669597+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"1041","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.998582+0000","last_change":"2021-02-10T04:51:07.998582+0000","last_active":"2021-02-10T04:51:07.998582+0000","last_peered":"2021-02-10T04:51:07.998582+0000","last_clean":"2021-02-10T04:51:07.998582+0000","last_became_active":"2021-02-10T04:50:59.563229+0000","last_became_peered":"2021-02-10T04:50:59.563229+0000","last_unstale":"2021-02-10T04:51:07.998582+0000","last_undegraded":"2021-02-10T04:51:07.998582+0000","last_fullsized":"2021-02-10T04:51:07.998582+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:58.514318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_clean_scrub_stamp":"2021-02-10T04:46:58.514318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"1049'42","reported_seq":"1174","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.596722+0000","last_change":"2021-02-10T04:50:59.578533+0000","last_active":"2021-02-10T04:51:04.596722+0000","last_peered":"2021-02-10T04:51:04.596722+0000","last_clean":"2021-02-10T04:51:04.596722+0000","last_became_active":"2021-02-10T04:50:59.577949+0000","last_became_peered":"2021-02-10T04:50:59.577949+0000","last_unstale":"2021-02-10T04:51:04.596722+0000","last_undegraded":"2021-02-10T04:51:04.596722+0000","last_fullsized":"2021-02-10T04:51:04.596722+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'42","last_scrub_stamp":"2021-02-10T04:46:13.849755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:13.849755+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":75,"num_read_kb":57,"num_write":42,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"1153","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.575425+0000","last_change":"2021-02-10T04:50:04.287211+0000","last_active":"2021-02-10T04:51:04.575425+0000","last_peered":"2021-02-10T04:51:04.575425+0000","last_clean":"2021-02-10T04:51:04.575425+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:51:04.575425+0000","last_undegraded":"2021-02-10T04:51:04.575425+0000","last_fullsized":"2021-02-10T04:51:04.575425+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:04.287114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:04.287114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"1098'3","reported_seq":"1394","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:12.989112+0000","last_change":"2021-02-10T04:50:15.255211+0000","last_active":"2021-02-10T04:51:12.989112+0000","last_peered":"2021-02-10T04:51:12.989112+0000","last_clean":"2021-02-10T04:51:12.989112+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:51:12.989112+0000","last_undegraded":"2021-02-10T04:51:12.989112+0000","last_fullsized":"2021-02-10T04:51:12.989112+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"1098'3","last_scrub_stamp":"2021-02-10T04:50:15.255137+0000","last_deep_scrub":"1098'3","last_deep_scrub_stamp":"2021-02-10T04:50:15.255137+0000","last_clean_scrub_stamp":"2021-02-10T04:50:15.255137+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"1023","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.576730+0000","last_change":"2021-02-10T04:46:34.914405+0000","last_active":"2021-02-10T04:51:04.576730+0000","last_peered":"2021-02-10T04:51:04.576730+0000","last_clean":"2021-02-10T04:51:04.576730+0000","last_became_active":"2021-02-10T04:38:39.251336+0000","last_became_peered":"2021-02-10T04:38:39.251336+0000","last_unstale":"2021-02-10T04:51:04.576730+0000","last_undegraded":"2021-02-10T04:51:04.576730+0000","last_fullsized":"2021-02-10T04:51:04.576730+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:34.914288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:34.914288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"1049'51","reported_seq":"1175","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.602848+0000","last_change":"2021-02-10T04:50:53.153340+0000","last_active":"2021-02-10T04:51:09.602848+0000","last_peered":"2021-02-10T04:51:09.602848+0000","last_clean":"2021-02-10T04:51:09.602848+0000","last_became_active":"2021-02-10T04:25:49.015760+0000","last_became_peered":"2021-02-10T04:25:49.015760+0000","last_unstale":"2021-02-10T04:51:09.602848+0000","last_undegraded":"2021-02-10T04:51:09.602848+0000","last_fullsized":"2021-02-10T04:51:09.602848+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'51","last_scrub_stamp":"2021-02-10T04:50:53.153251+0000","last_deep_scrub":"76'35","last_deep_scrub_stamp":"2021-02-10T04:26:58.198641+0000","last_clean_scrub_stamp":"2021-02-10T04:50:53.153251+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":74,"num_read_kb":54,"num_write":40,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"1045","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.997058+0000","last_change":"2021-02-10T04:51:07.997058+0000","last_active":"2021-02-10T04:51:07.997058+0000","last_peered":"2021-02-10T04:51:07.997058+0000","last_clean":"2021-02-10T04:51:07.997058+0000","last_became_active":"2021-02-10T04:50:59.555151+0000","last_became_peered":"2021-02-10T04:50:59.555151+0000","last_unstale":"2021-02-10T04:51:07.997058+0000","last_undegraded":"2021-02-10T04:51:07.997058+0000","last_fullsized":"2021-02-10T04:51:07.997058+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:42.537443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:42.537443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"1094","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.596755+0000","last_change":"2021-02-10T04:47:28.938832+0000","last_active":"2021-02-10T04:51:04.596755+0000","last_peered":"2021-02-10T04:51:04.596755+0000","last_clean":"2021-02-10T04:51:04.596755+0000","last_became_active":"2021-02-10T04:37:37.246998+0000","last_became_peered":"2021-02-10T04:37:37.246998+0000","last_unstale":"2021-02-10T04:51:04.596755+0000","last_undegraded":"2021-02-10T04:51:04.596755+0000","last_fullsized":"2021-02-10T04:51:04.596755+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.938731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.938731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"768","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.577515+0000","last_change":"2021-02-10T04:50:49.391976+0000","last_active":"2021-02-10T04:51:04.577515+0000","last_peered":"2021-02-10T04:51:04.577515+0000","last_clean":"2021-02-10T04:51:04.577515+0000","last_became_active":"2021-02-10T04:33:05.141911+0000","last_became_peered":"2021-02-10T04:33:05.141911+0000","last_unstale":"2021-02-10T04:51:04.577515+0000","last_undegraded":"2021-02-10T04:51:04.577515+0000","last_fullsized":"2021-02-10T04:51:04.577515+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:49.391902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:49.391902+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"1049'39","reported_seq":"54","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:08.433710+0000","last_change":"2021-02-10T04:51:08.433663+0000","last_active":"2021-02-10T04:51:08.433710+0000","last_peered":"2021-02-10T04:51:08.433710+0000","last_clean":"2021-02-10T04:51:08.433710+0000","last_became_active":"2021-02-10T04:50:59.567778+0000","last_became_peered":"2021-02-10T04:50:59.567778+0000","last_unstale":"2021-02-10T04:51:08.433710+0000","last_undegraded":"2021-02-10T04:51:08.433710+0000","last_fullsized":"2021-02-10T04:51:08.433710+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'39","last_scrub_stamp":"2021-02-10T04:51:08.433562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.433562+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"1098","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.601096+0000","last_change":"2021-02-10T04:46:47.143081+0000","last_active":"2021-02-10T04:51:09.601096+0000","last_peered":"2021-02-10T04:51:09.601096+0000","last_clean":"2021-02-10T04:51:09.601096+0000","last_became_active":"2021-02-10T04:25:36.336442+0000","last_became_peered":"2021-02-10T04:25:36.336442+0000","last_unstale":"2021-02-10T04:51:09.601096+0000","last_undegraded":"2021-02-10T04:51:09.601096+0000","last_fullsized":"2021-02-10T04:51:09.601096+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:46:47.142978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:47.142978+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"767","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:08.758290+0000","last_change":"2021-02-10T04:51:08.758225+0000","last_active":"2021-02-10T04:51:08.758290+0000","last_peered":"2021-02-10T04:51:08.758290+0000","last_clean":"2021-02-10T04:51:08.758290+0000","last_became_active":"2021-02-10T04:33:05.146838+0000","last_became_peered":"2021-02-10T04:33:05.146838+0000","last_unstale":"2021-02-10T04:51:08.758290+0000","last_undegraded":"2021-02-10T04:51:08.758290+0000","last_fullsized":"2021-02-10T04:51:08.758290+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:08.758129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.758129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"1077","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595736+0000","last_change":"2021-02-10T04:46:36.790576+0000","last_active":"2021-02-10T04:51:04.595736+0000","last_peered":"2021-02-10T04:51:04.595736+0000","last_clean":"2021-02-10T04:51:04.595736+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:51:04.595736+0000","last_undegraded":"2021-02-10T04:51:04.595736+0000","last_fullsized":"2021-02-10T04:51:04.595736+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:36.790487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:36.790487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"1066'44","reported_seq":"1122","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.985331+0000","last_change":"2021-02-10T04:51:07.985331+0000","last_active":"2021-02-10T04:51:07.985331+0000","last_peered":"2021-02-10T04:51:07.985331+0000","last_clean":"2021-02-10T04:51:07.985331+0000","last_became_active":"2021-02-10T04:50:59.563388+0000","last_became_peered":"2021-02-10T04:50:59.563388+0000","last_unstale":"2021-02-10T04:51:07.985331+0000","last_undegraded":"2021-02-10T04:51:07.985331+0000","last_fullsized":"2021-02-10T04:51:07.985331+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1066'44","last_scrub_stamp":"2021-02-10T04:47:18.806645+0000","last_deep_scrub":"1066'44","last_deep_scrub_stamp":"2021-02-10T04:47:18.806645+0000","last_clean_scrub_stamp":"2021-02-10T04:47:18.806645+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":67,"num_read_kb":49,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"1066","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:08.006463+0000","last_change":"2021-02-10T04:51:08.006463+0000","last_active":"2021-02-10T04:51:08.006463+0000","last_peered":"2021-02-10T04:51:08.006463+0000","last_clean":"2021-02-10T04:51:08.006463+0000","last_became_active":"2021-02-10T04:50:59.563657+0000","last_became_peered":"2021-02-10T04:50:59.563657+0000","last_unstale":"2021-02-10T04:51:08.006463+0000","last_undegraded":"2021-02-10T04:51:08.006463+0000","last_fullsized":"2021-02-10T04:51:08.006463+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_clean_scrub_stamp":"2021-02-10T04:51:02.455702+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"1103","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595500+0000","last_change":"2021-02-10T04:50:59.585214+0000","last_active":"2021-02-10T04:51:04.595500+0000","last_peered":"2021-02-10T04:51:04.595500+0000","last_clean":"2021-02-10T04:51:04.595500+0000","last_became_active":"2021-02-10T04:50:59.584846+0000","last_became_peered":"2021-02-10T04:50:59.584846+0000","last_unstale":"2021-02-10T04:51:04.595500+0000","last_undegraded":"2021-02-10T04:51:04.595500+0000","last_fullsized":"2021-02-10T04:51:04.595500+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:58.666264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:46:58.666264+0000","last_clean_scrub_stamp":"2021-02-10T04:46:58.666264+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"1073","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:08.002745+0000","last_change":"2021-02-10T04:51:08.002745+0000","last_active":"2021-02-10T04:51:08.002745+0000","last_peered":"2021-02-10T04:51:08.002745+0000","last_clean":"2021-02-10T04:51:08.002745+0000","last_became_active":"2021-02-10T04:50:59.539582+0000","last_became_peered":"2021-02-10T04:50:59.539582+0000","last_unstale":"2021-02-10T04:51:08.002745+0000","last_undegraded":"2021-02-10T04:51:08.002745+0000","last_fullsized":"2021-02-10T04:51:08.002745+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:26.576960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:26.576960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"1003'739","reported_seq":"3789","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.599421+0000","last_change":"2021-02-10T04:47:03.951820+0000","last_active":"2021-02-10T04:51:09.599421+0000","last_peered":"2021-02-10T04:51:09.599421+0000","last_clean":"2021-02-10T04:51:09.599421+0000","last_became_active":"2021-02-10T04:44:30.579847+0000","last_became_peered":"2021-02-10T04:44:30.579847+0000","last_unstale":"2021-02-10T04:51:09.599421+0000","last_undegraded":"2021-02-10T04:51:09.599421+0000","last_fullsized":"2021-02-10T04:51:09.599421+0000","mapping_epoch":1012,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1013,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:47:03.951729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:03.951729+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22230846,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"1112'47","reported_seq":"1185","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595390+0000","last_change":"2021-02-10T04:51:02.860932+0000","last_active":"2021-02-10T04:51:04.595390+0000","last_peered":"2021-02-10T04:51:04.595390+0000","last_clean":"2021-02-10T04:51:04.595390+0000","last_became_active":"2021-02-10T04:50:59.570786+0000","last_became_peered":"2021-02-10T04:50:59.570786+0000","last_unstale":"2021-02-10T04:51:04.595390+0000","last_undegraded":"2021-02-10T04:51:04.595390+0000","last_fullsized":"2021-02-10T04:51:04.595390+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1112'47","last_scrub_stamp":"2021-02-10T04:51:02.860835+0000","last_deep_scrub":"1112'47","last_deep_scrub_stamp":"2021-02-10T04:51:02.860835+0000","last_clean_scrub_stamp":"2021-02-10T04:51:02.860835+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":71,"num_read_kb":55,"num_write":38,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"1014","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.576941+0000","last_change":"2021-02-10T04:46:59.760348+0000","last_active":"2021-02-10T04:51:04.576941+0000","last_peered":"2021-02-10T04:51:04.576941+0000","last_clean":"2021-02-10T04:51:04.576941+0000","last_became_active":"2021-02-10T04:37:37.241751+0000","last_became_peered":"2021-02-10T04:37:37.241751+0000","last_unstale":"2021-02-10T04:51:04.576941+0000","last_undegraded":"2021-02-10T04:51:04.576941+0000","last_fullsized":"2021-02-10T04:51:04.576941+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:59.760263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:59.760263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"1109","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.576064+0000","last_change":"2021-02-10T04:51:02.394551+0000","last_active":"2021-02-10T04:51:04.576064+0000","last_peered":"2021-02-10T04:51:04.576064+0000","last_clean":"2021-02-10T04:51:04.576064+0000","last_became_active":"2021-02-10T04:50:59.575340+0000","last_became_peered":"2021-02-10T04:50:59.575340+0000","last_unstale":"2021-02-10T04:51:04.576064+0000","last_undegraded":"2021-02-10T04:51:04.576064+0000","last_fullsized":"2021-02-10T04:51:04.576064+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:02.394440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:02.394440+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"1083","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.596194+0000","last_change":"2021-02-10T04:46:24.697867+0000","last_active":"2021-02-10T04:51:04.596194+0000","last_peered":"2021-02-10T04:51:04.596194+0000","last_clean":"2021-02-10T04:51:04.596194+0000","last_became_active":"2021-02-10T04:25:53.039844+0000","last_became_peered":"2021-02-10T04:25:53.039844+0000","last_unstale":"2021-02-10T04:51:04.596194+0000","last_undegraded":"2021-02-10T04:51:04.596194+0000","last_fullsized":"2021-02-10T04:51:04.596194+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:24.697786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:24.697786+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"259'74","reported_seq":"121","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595496+0000","last_change":"2021-02-10T04:46:49.745214+0000","last_active":"2021-02-10T04:51:04.595496+0000","last_peered":"2021-02-10T04:51:04.595496+0000","last_clean":"2021-02-10T04:51:04.595496+0000","last_became_active":"2021-02-10T04:44:27.546590+0000","last_became_peered":"2021-02-10T04:44:27.546590+0000","last_unstale":"2021-02-10T04:51:04.595496+0000","last_undegraded":"2021-02-10T04:51:04.595496+0000","last_fullsized":"2021-02-10T04:51:04.595496+0000","mapping_epoch":1009,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1010,"parent":"0.0","parent_split_bits":5,"last_scrub":"259'74","last_scrub_stamp":"2021-02-10T04:46:49.745104+0000","last_deep_scrub":"259'74","last_deep_scrub_stamp":"2021-02-10T04:46:49.745104+0000","last_clean_scrub_stamp":"2021-02-10T04:46:49.745104+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"1061'20","reported_seq":"1133","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595316+0000","last_change":"2021-02-10T04:50:59.569616+0000","last_active":"2021-02-10T04:51:04.595316+0000","last_peered":"2021-02-10T04:51:04.595316+0000","last_clean":"2021-02-10T04:51:04.595316+0000","last_became_active":"2021-02-10T04:50:59.569344+0000","last_became_peered":"2021-02-10T04:50:59.569344+0000","last_unstale":"2021-02-10T04:51:04.595316+0000","last_undegraded":"2021-02-10T04:51:04.595316+0000","last_fullsized":"2021-02-10T04:51:04.595316+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1061'20","last_scrub_stamp":"2021-02-10T04:46:55.646261+0000","last_deep_scrub":"1061'20","last_deep_scrub_stamp":"2021-02-10T04:46:55.646261+0000","last_clean_scrub_stamp":"2021-02-10T04:46:55.646261+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":17,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"1011","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.602008+0000","last_change":"2021-02-10T04:50:59.167977+0000","last_active":"2021-02-10T04:51:09.602008+0000","last_peered":"2021-02-10T04:51:09.602008+0000","last_clean":"2021-02-10T04:51:09.602008+0000","last_became_active":"2021-02-10T04:29:07.157944+0000","last_became_peered":"2021-02-10T04:29:07.157944+0000","last_unstale":"2021-02-10T04:51:09.602008+0000","last_undegraded":"2021-02-10T04:51:09.602008+0000","last_fullsized":"2021-02-10T04:51:09.602008+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:59.167911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:50:59.167911+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"999","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595458+0000","last_change":"2021-02-10T04:46:15.814062+0000","last_active":"2021-02-10T04:51:04.595458+0000","last_peered":"2021-02-10T04:51:04.595458+0000","last_clean":"2021-02-10T04:51:04.595458+0000","last_became_active":"2021-02-10T04:29:07.146525+0000","last_became_peered":"2021-02-10T04:29:07.146525+0000","last_unstale":"2021-02-10T04:51:04.595458+0000","last_undegraded":"2021-02-10T04:51:04.595458+0000","last_fullsized":"2021-02-10T04:51:04.595458+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:15.813941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:46:15.813941+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"1098","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:09.433980+0000","last_change":"2021-02-10T04:51:09.433892+0000","last_active":"2021-02-10T04:51:09.433980+0000","last_peered":"2021-02-10T04:51:09.433980+0000","last_clean":"2021-02-10T04:51:09.433980+0000","last_became_active":"2021-02-10T04:50:59.539256+0000","last_became_peered":"2021-02-10T04:50:59.539256+0000","last_unstale":"2021-02-10T04:51:09.433980+0000","last_undegraded":"2021-02-10T04:51:09.433980+0000","last_fullsized":"2021-02-10T04:51:09.433980+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:09.433793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:09.433793+0000","last_clean_scrub_stamp":"2021-02-10T04:51:09.433793+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"1048'44","reported_seq":"1158","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.594957+0000","last_change":"2021-02-10T04:50:48.668186+0000","last_active":"2021-02-10T04:51:04.594957+0000","last_peered":"2021-02-10T04:51:04.594957+0000","last_clean":"2021-02-10T04:51:04.594957+0000","last_became_active":"2021-02-10T04:25:49.010481+0000","last_became_peered":"2021-02-10T04:25:49.010481+0000","last_unstale":"2021-02-10T04:51:04.594957+0000","last_undegraded":"2021-02-10T04:51:04.594957+0000","last_fullsized":"2021-02-10T04:51:04.594957+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'44","last_scrub_stamp":"2021-02-10T04:50:48.668079+0000","last_deep_scrub":"78'32","last_deep_scrub_stamp":"2021-02-10T04:27:09.170428+0000","last_clean_scrub_stamp":"2021-02-10T04:50:48.668079+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":65,"num_read_kb":46,"num_write":34,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"1084","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.979394+0000","last_change":"2021-02-10T04:51:07.979394+0000","last_active":"2021-02-10T04:51:07.979394+0000","last_peered":"2021-02-10T04:51:07.979394+0000","last_clean":"2021-02-10T04:51:07.979394+0000","last_became_active":"2021-02-10T04:50:59.557394+0000","last_became_peered":"2021-02-10T04:50:59.557394+0000","last_unstale":"2021-02-10T04:51:07.979394+0000","last_undegraded":"2021-02-10T04:51:07.979394+0000","last_fullsized":"2021-02-10T04:51:07.979394+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:33.543559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:46:33.543559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"768","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.575756+0000","last_change":"2021-02-10T04:50:06.195327+0000","last_active":"2021-02-10T04:51:04.575756+0000","last_peered":"2021-02-10T04:51:04.575756+0000","last_clean":"2021-02-10T04:51:04.575756+0000","last_became_active":"2021-02-10T04:33:05.169159+0000","last_became_peered":"2021-02-10T04:33:05.169159+0000","last_unstale":"2021-02-10T04:51:04.575756+0000","last_undegraded":"2021-02-10T04:51:04.575756+0000","last_fullsized":"2021-02-10T04:51:04.575756+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:06.195234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:06.195234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"1163","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595288+0000","last_change":"2021-02-10T04:47:12.800261+0000","last_active":"2021-02-10T04:51:04.595288+0000","last_peered":"2021-02-10T04:51:04.595288+0000","last_clean":"2021-02-10T04:51:04.595288+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:51:04.595288+0000","last_undegraded":"2021-02-10T04:51:04.595288+0000","last_fullsized":"2021-02-10T04:51:04.595288+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:12.800147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:12.800147+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"1049'68","reported_seq":"1220","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.979350+0000","last_change":"2021-02-10T04:51:07.979350+0000","last_active":"2021-02-10T04:51:07.979350+0000","last_peered":"2021-02-10T04:51:07.979350+0000","last_clean":"2021-02-10T04:51:07.979350+0000","last_became_active":"2021-02-10T04:50:59.561153+0000","last_became_peered":"2021-02-10T04:50:59.561153+0000","last_unstale":"2021-02-10T04:51:07.979350+0000","last_undegraded":"2021-02-10T04:51:07.979350+0000","last_fullsized":"2021-02-10T04:51:07.979350+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'68","last_scrub_stamp":"2021-02-10T04:47:09.608881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:47:09.608881+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":124,"num_read_kb":101,"num_write":69,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":890,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"1103","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.596253+0000","last_change":"2021-02-10T04:47:17.919840+0000","last_active":"2021-02-10T04:51:04.596253+0000","last_peered":"2021-02-10T04:51:04.596253+0000","last_clean":"2021-02-10T04:51:04.596253+0000","last_became_active":"2021-02-10T04:25:36.336112+0000","last_became_peered":"2021-02-10T04:25:36.336112+0000","last_unstale":"2021-02-10T04:51:04.596253+0000","last_undegraded":"2021-02-10T04:51:04.596253+0000","last_fullsized":"2021-02-10T04:51:04.596253+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:17.919745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:17.919745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"395","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.970205+0000","last_change":"2021-02-10T04:51:07.970205+0000","last_active":"2021-02-10T04:51:07.970205+0000","last_peered":"2021-02-10T04:51:07.970205+0000","last_clean":"2021-02-10T04:51:07.970205+0000","last_became_active":"2021-02-10T04:50:59.572077+0000","last_became_peered":"2021-02-10T04:50:59.572077+0000","last_unstale":"2021-02-10T04:51:07.970205+0000","last_undegraded":"2021-02-10T04:51:07.970205+0000","last_fullsized":"2021-02-10T04:51:07.970205+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:19.755718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:19.755718+0000","last_clean_scrub_stamp":"2021-02-10T04:47:19.755718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"1072","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:08.006421+0000","last_change":"2021-02-10T04:51:08.006421+0000","last_active":"2021-02-10T04:51:08.006421+0000","last_peered":"2021-02-10T04:51:08.006421+0000","last_clean":"2021-02-10T04:51:08.006421+0000","last_became_active":"2021-02-10T04:50:59.563156+0000","last_became_peered":"2021-02-10T04:50:59.563156+0000","last_unstale":"2021-02-10T04:51:08.006421+0000","last_undegraded":"2021-02-10T04:51:08.006421+0000","last_fullsized":"2021-02-10T04:51:08.006421+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:14.549136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:14.549136+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":"795","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.576609+0000","last_change":"2021-02-10T04:50:59.595747+0000","last_active":"2021-02-10T04:51:04.576609+0000","last_peered":"2021-02-10T04:51:04.576609+0000","last_clean":"2021-02-10T04:51:04.576609+0000","last_became_active":"2021-02-10T04:50:59.595454+0000","last_became_peered":"2021-02-10T04:50:59.595454+0000","last_unstale":"2021-02-10T04:51:04.576609+0000","last_undegraded":"2021-02-10T04:51:04.576609+0000","last_fullsized":"2021-02-10T04:51:04.576609+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:20.635536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:20.635536+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.635536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.20","version":"0'0","reported_seq":"1068","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.575505+0000","last_change":"2021-02-10T04:46:50.859426+0000","last_active":"2021-02-10T04:51:04.575505+0000","last_peered":"2021-02-10T04:51:04.575505+0000","last_clean":"2021-02-10T04:51:04.575505+0000","last_became_active":"2021-02-10T04:29:07.148937+0000","last_became_peered":"2021-02-10T04:29:07.148937+0000","last_unstale":"2021-02-10T04:51:04.575505+0000","last_undegraded":"2021-02-10T04:51:04.575505+0000","last_fullsized":"2021-02-10T04:51:04.575505+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:50.859195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:41.375276+0000","last_clean_scrub_stamp":"2021-02-10T04:46:50.859195+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":"1028","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.601924+0000","last_change":"2021-02-10T04:50:04.210063+0000","last_active":"2021-02-10T04:51:09.601924+0000","last_peered":"2021-02-10T04:51:09.601924+0000","last_clean":"2021-02-10T04:51:09.601924+0000","last_became_active":"2021-02-10T04:45:49.551858+0000","last_became_peered":"2021-02-10T04:45:49.551858+0000","last_unstale":"2021-02-10T04:51:09.601924+0000","last_undegraded":"2021-02-10T04:51:09.601924+0000","last_fullsized":"2021-02-10T04:51:09.601924+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:04.210002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:50:04.210002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":"586","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595897+0000","last_change":"2021-02-10T04:50:47.481762+0000","last_active":"2021-02-10T04:51:04.595897+0000","last_peered":"2021-02-10T04:51:04.595897+0000","last_clean":"2021-02-10T04:51:04.595897+0000","last_became_active":"2021-02-10T04:37:37.246865+0000","last_became_peered":"2021-02-10T04:37:37.246865+0000","last_unstale":"2021-02-10T04:51:04.595897+0000","last_undegraded":"2021-02-10T04:51:04.595897+0000","last_fullsized":"2021-02-10T04:51:04.595897+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:47.481686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:47.481686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":"82","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.576805+0000","last_change":"2021-02-10T04:47:28.706460+0000","last_active":"2021-02-10T04:51:04.576805+0000","last_peered":"2021-02-10T04:51:04.576805+0000","last_clean":"2021-02-10T04:51:04.576805+0000","last_became_active":"2021-02-10T04:45:49.552420+0000","last_became_peered":"2021-02-10T04:45:49.552420+0000","last_unstale":"2021-02-10T04:51:04.576805+0000","last_undegraded":"2021-02-10T04:51:04.576805+0000","last_fullsized":"2021-02-10T04:51:04.576805+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.706372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.706372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":"1119","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.600829+0000","last_change":"2021-02-10T04:50:59.567255+0000","last_active":"2021-02-10T04:51:09.600829+0000","last_peered":"2021-02-10T04:51:09.600829+0000","last_clean":"2021-02-10T04:51:09.600829+0000","last_became_active":"2021-02-10T04:50:59.566827+0000","last_became_peered":"2021-02-10T04:50:59.566827+0000","last_unstale":"2021-02-10T04:51:09.600829+0000","last_undegraded":"2021-02-10T04:51:09.600829+0000","last_fullsized":"2021-02-10T04:51:09.600829+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:32.031098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:32.031098+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":"83","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.602273+0000","last_change":"2021-02-10T04:50:10.308350+0000","last_active":"2021-02-10T04:51:09.602273+0000","last_peered":"2021-02-10T04:51:09.602273+0000","last_clean":"2021-02-10T04:51:09.602273+0000","last_became_active":"2021-02-10T04:45:49.551121+0000","last_became_peered":"2021-02-10T04:45:49.551121+0000","last_unstale":"2021-02-10T04:51:09.602273+0000","last_undegraded":"2021-02-10T04:51:09.602273+0000","last_fullsized":"2021-02-10T04:51:09.602273+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:10.308282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:10.308282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":"1008","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.989063+0000","last_change":"2021-02-10T04:51:07.989063+0000","last_active":"2021-02-10T04:51:07.989063+0000","last_peered":"2021-02-10T04:51:07.989063+0000","last_clean":"2021-02-10T04:51:07.989063+0000","last_became_active":"2021-02-10T04:50:59.561442+0000","last_became_peered":"2021-02-10T04:50:59.561442+0000","last_unstale":"2021-02-10T04:51:07.989063+0000","last_undegraded":"2021-02-10T04:51:07.989063+0000","last_fullsized":"2021-02-10T04:51:07.989063+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:51.620102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:44:51.620102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":"83","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.599548+0000","last_change":"2021-02-10T04:50:08.272047+0000","last_active":"2021-02-10T04:51:09.599548+0000","last_peered":"2021-02-10T04:51:09.599548+0000","last_clean":"2021-02-10T04:51:09.599548+0000","last_became_active":"2021-02-10T04:45:49.759179+0000","last_became_peered":"2021-02-10T04:45:49.759179+0000","last_unstale":"2021-02-10T04:51:09.599548+0000","last_undegraded":"2021-02-10T04:51:09.599548+0000","last_fullsized":"2021-02-10T04:51:09.599548+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:08.271977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:08.271977+0000","last_clean_scrub_stamp":"2021-02-10T04:50:08.271977+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":"1053","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.602171+0000","last_change":"2021-02-10T04:51:07.182775+0000","last_active":"2021-02-10T04:51:09.602171+0000","last_peered":"2021-02-10T04:51:09.602171+0000","last_clean":"2021-02-10T04:51:09.602171+0000","last_became_active":"2021-02-10T04:28:23.126368+0000","last_became_peered":"2021-02-10T04:28:23.126368+0000","last_unstale":"2021-02-10T04:51:09.602171+0000","last_undegraded":"2021-02-10T04:51:09.602171+0000","last_fullsized":"2021-02-10T04:51:09.602171+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.182678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.182678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":"82","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595181+0000","last_change":"2021-02-10T04:50:07.767334+0000","last_active":"2021-02-10T04:51:04.595181+0000","last_peered":"2021-02-10T04:51:04.595181+0000","last_clean":"2021-02-10T04:51:04.595181+0000","last_became_active":"2021-02-10T04:45:49.759224+0000","last_became_peered":"2021-02-10T04:45:49.759224+0000","last_unstale":"2021-02-10T04:51:04.595181+0000","last_undegraded":"2021-02-10T04:51:04.595181+0000","last_fullsized":"2021-02-10T04:51:04.595181+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:07.767250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:07.767250+0000","last_clean_scrub_stamp":"2021-02-10T04:50:07.767250+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.25","version":"53'2","reported_seq":"1064","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595627+0000","last_change":"2021-02-10T04:50:12.772167+0000","last_active":"2021-02-10T04:51:04.595627+0000","last_peered":"2021-02-10T04:51:04.595627+0000","last_clean":"2021-02-10T04:51:04.595627+0000","last_became_active":"2021-02-10T04:28:23.467770+0000","last_became_peered":"2021-02-10T04:28:23.467770+0000","last_unstale":"2021-02-10T04:51:04.595627+0000","last_undegraded":"2021-02-10T04:51:04.595627+0000","last_fullsized":"2021-02-10T04:51:04.595627+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:50:12.772086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:12.772086+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":"1155","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.600548+0000","last_change":"2021-02-10T04:50:59.567042+0000","last_active":"2021-02-10T04:51:09.600548+0000","last_peered":"2021-02-10T04:51:09.600548+0000","last_clean":"2021-02-10T04:51:09.600548+0000","last_became_active":"2021-02-10T04:50:59.566548+0000","last_became_peered":"2021-02-10T04:50:59.566548+0000","last_unstale":"2021-02-10T04:51:09.600548+0000","last_undegraded":"2021-02-10T04:51:09.600548+0000","last_fullsized":"2021-02-10T04:51:09.600548+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:27.926662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:44:27.926662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.26","version":"53'1","reported_seq":"504","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595631+0000","last_change":"2021-02-10T04:47:11.808609+0000","last_active":"2021-02-10T04:51:04.595631+0000","last_peered":"2021-02-10T04:51:04.595631+0000","last_clean":"2021-02-10T04:51:04.595631+0000","last_became_active":"2021-02-10T04:37:37.231081+0000","last_became_peered":"2021-02-10T04:37:37.231081+0000","last_unstale":"2021-02-10T04:51:04.595631+0000","last_undegraded":"2021-02-10T04:51:04.595631+0000","last_fullsized":"2021-02-10T04:51:04.595631+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:47:11.808550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:11.808550+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":"1037","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.601468+0000","last_change":"2021-02-10T04:47:22.953445+0000","last_active":"2021-02-10T04:51:09.601468+0000","last_peered":"2021-02-10T04:51:09.601468+0000","last_clean":"2021-02-10T04:51:09.601468+0000","last_became_active":"2021-02-10T04:45:49.552026+0000","last_became_peered":"2021-02-10T04:45:49.552026+0000","last_unstale":"2021-02-10T04:51:09.601468+0000","last_undegraded":"2021-02-10T04:51:09.601468+0000","last_fullsized":"2021-02-10T04:51:09.601468+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:22.953356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.953356+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":"1075","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:04.595966+0000","last_change":"2021-02-10T04:47:16.706513+0000","last_active":"2021-02-10T04:51:04.595966+0000","last_peered":"2021-02-10T04:51:04.595966+0000","last_clean":"2021-02-10T04:51:04.595966+0000","last_became_active":"2021-02-10T04:29:07.145907+0000","last_became_peered":"2021-02-10T04:29:07.145907+0000","last_unstale":"2021-02-10T04:51:04.595966+0000","last_undegraded":"2021-02-10T04:51:04.595966+0000","last_fullsized":"2021-02-10T04:51:04.595966+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:16.706416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:16.706416+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":"1037","reported_epoch":"1113","state":"active+clean","last_fresh":"2021-02-10T04:51:07.984986+0000","last_change":"2021-02-10T04:51:07.984986+0000","last_active":"2021-02-10T04:51:07.984986+0000","last_peered":"2021-02-10T04:51:07.984986+0000","last_clean":"2021-02-10T04:51:07.984986+0000","last_became_active":"2021-02-10T04:50:59.569162+0000","last_became_peered":"2021-02-10T04:50:59.569162+0000","last_unstale":"2021-02-10T04:51:07.984986+0000","last_undegraded":"2021-02-10T04:51:07.984986+0000","last_fullsized":"2021-02-10T04:51:07.984986+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:12.555666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:46:12.555666+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":"1063","reported_epoch":"1114","state":"active+clean","last_fresh":"2021-02-10T04:51:09.601805+0000","last_change":"2021-02-10T04:50:59.528569+0000","last_active":"2021-02-10T04:51:09.601805+0000","last_peered":"2021-02-10T04:51:09.601805+0000","last_clean":"2021-02-10T04:51:09.601805+0000","last_became_active":"2021-02-10T04:50:59.527945+0000","last_became_peered":"2021-02-10T04:50:59.527945+0000","last_unstale":"2021-02-10T04:51:09.601805+0000","last_undegraded":"2021-02-10T04:51:09.601805+0000","last_fullsized":"2021-02-10T04:51:09.601805+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:50.057162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_clean_scrub_stamp":"2021-02-10T04:46:50.057162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":940005779,"num_objects":386,"num_object_clones":336,"num_object_copies":1158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":386,"num_whiteouts":12,"num_read":4562,"num_read_kb":4769301,"num_write":3301,"num_write_kb":1265045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":101337272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1931636736,"data_stored":2559497134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23681,"ondisk_log_size":23681,"up":78,"acting":78,"num_store_stats":8},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":1945,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":17,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":6565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":126,"acting":126,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":822,"num_read_kb":817,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":2297,"num_read_kb":1770,"num_write":1306,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":8770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":75722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1392,"ondisk_log_size":1392,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":126,"acting":126,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365109,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":227704,"kb_used_data":192880,"kb_used_omap":0,"kb_used_meta":34816,"kb_avail":93517448,"statfs":{"total":95995035648,"available":95761866752,"internally_reserved":0,"allocated":197509120,"data_stored":268739112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35651584},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:50:27 2021","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":1.2969999999999999,"15min":1.288},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":2.1379999999999999,"5min":34.299999999999997,"15min":67.277000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.516,"15min":1.548},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":2.4260000000000002,"5min":34.652999999999999,"15min":66.933999999999997},"last":1.208}]},{"osd":1,"last update":"Wed Feb 10 04:50:27 2021","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.91100000000000003,"15min":0.99399999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.48399999999999999},"max":{"1min":2.1829999999999998,"5min":2.3420000000000001,"15min":10.140000000000001},"last":1.5169999999999999},{"interface":"front","average":{"1min":0.97799999999999998,"5min":1.048,"15min":1.01},"min":{"1min":0.66200000000000003,"5min":0.623,"15min":0.57299999999999995},"max":{"1min":2.5409999999999999,"5min":2.6549999999999998,"15min":7.1520000000000001},"last":0.89200000000000002}]},{"osd":2,"last update":"Wed Feb 10 04:50:27 2021","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.1100000000000001,"15min":1.462},"min":{"1min":0.61899999999999999,"5min":0.61399999999999999,"15min":0.58299999999999996},"max":{"1min":3.0630000000000002,"5min":3.0630000000000002,"15min":66.534000000000006},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.21,"5min":1.218,"15min":1.712},"min":{"1min":0.67900000000000005,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.8660000000000001,"5min":6.7930000000000001,"15min":66.781999999999996},"last":1.4790000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:49:21 2021","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.2749999999999999,"15min":1.587},"min":{"1min":0.73099999999999998,"5min":0.72999999999999998,"15min":0.69599999999999995},"max":{"1min":2.3210000000000002,"5min":5.2599999999999998,"15min":68.305000000000007},"last":1.728},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.3129999999999999,"15min":1.552},"min":{"1min":0.95899999999999996,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":2.0310000000000001,"5min":2.4249999999999998,"15min":69.331999999999994},"last":1.0860000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:50:27 2021","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.454,"15min":1.78},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.64600000000000002},"max":{"1min":2.778,"5min":7.0640000000000001,"15min":97.733999999999995},"last":1.8600000000000001},{"interface":"front","average":{"1min":1.208,"5min":1.343,"15min":1.6899999999999999},"min":{"1min":0.74299999999999999,"5min":0.69099999999999995,"15min":0.65100000000000002},"max":{"1min":2.903,"5min":4.6269999999999998,"15min":97.691000000000003},"last":1.899}]},{"osd":5,"last update":"Wed Feb 10 04:50:27 2021","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.6459999999999999,"15min":1.6459999999999999},"min":{"1min":0.875,"5min":0.85699999999999998,"15min":0.66700000000000004},"max":{"1min":3.1819999999999999,"5min":14.972,"15min":35.283000000000001},"last":1.179},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.8109999999999999,"15min":1.958},"min":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.85299999999999998},"max":{"1min":3.101,"5min":15.218999999999999,"15min":35.353999999999999},"last":2.0840000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:50:27 2021","interfaces":[{"interface":"back","average":{"1min":1.6459999999999999,"5min":1.895,"15min":1.899},"min":{"1min":0.97399999999999998,"5min":0.97199999999999998,"15min":0.89100000000000001},"max":{"1min":3.2879999999999998,"5min":16.050000000000001,"15min":16.050000000000001},"last":1.9750000000000001},{"interface":"front","average":{"1min":1.49,"5min":1.839,"15min":1.7889999999999999},"min":{"1min":1.077,"5min":0.94799999999999995,"15min":0.94799999999999995},"max":{"1min":3.3119999999999998,"5min":15.960000000000001,"15min":15.960000000000001},"last":2.3069999999999999}]}]},{"osd":6,"up_from":43,"seq":184683594042,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":359892,"kb_used_data":306900,"kb_used_omap":0,"kb_used_meta":52864,"kb_avail":93385260,"statfs":{"total":95995035648,"available":95626506240,"internally_reserved":0,"allocated":314265600,"data_stored":402042958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54132736},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:10 2021","interfaces":[{"interface":"back","average":{"1min":6.6200000000000001,"5min":2.121,"15min":1.3799999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.44700000000000001},"max":{"1min":137.60599999999999,"5min":137.60599999999999,"15min":137.60599999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":6.5789999999999997,"5min":2.0840000000000001,"15min":1.5960000000000001},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":137.66,"5min":137.66,"15min":137.66},"last":0.56599999999999995}]},{"osd":1,"last update":"Wed Feb 10 04:51:10 2021","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.0509999999999999,"15min":1.0580000000000001},"min":{"1min":0.53000000000000003,"5min":0.499,"15min":0.47699999999999998},"max":{"1min":5.2160000000000002,"5min":5.2160000000000002,"15min":12.978999999999999},"last":1.155},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.115,"15min":1.1359999999999999},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":3.6219999999999999,"5min":3.6219999999999999,"15min":6.0960000000000001},"last":0.94399999999999995}]},{"osd":2,"last update":"Wed Feb 10 04:51:10 2021","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.246,"15min":1.3340000000000001},"min":{"1min":0.61099999999999999,"5min":0.56200000000000006,"15min":0.54100000000000004},"max":{"1min":5.0990000000000002,"5min":5.0990000000000002,"15min":24.725000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.236,"5min":1.2989999999999999,"15min":1.3240000000000001},"min":{"1min":0.76900000000000002,"5min":0.66200000000000003,"15min":0.59899999999999998},"max":{"1min":5.3140000000000001,"5min":5.3140000000000001,"15min":24.611999999999998},"last":1.266}]},{"osd":3,"last update":"Wed Feb 10 04:49:04 2021","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":2.948,"15min":2.4060000000000001},"min":{"1min":0.89600000000000002,"5min":0.66700000000000004,"15min":0.63200000000000001},"max":{"1min":2.1429999999999998,"5min":75.897000000000006,"15min":114.17700000000001},"last":1.3799999999999999},{"interface":"front","average":{"1min":1.5469999999999999,"5min":2.9300000000000002,"15min":2.4489999999999998},"min":{"1min":1.0409999999999999,"5min":0.745,"15min":0.745},"max":{"1min":2.1680000000000001,"5min":75.994,"15min":116.929},"last":1.0589999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:51:10 2021","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.4099999999999999,"15min":1.456},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.60599999999999998},"max":{"1min":7.1459999999999999,"5min":7.1459999999999999,"15min":7.5389999999999997},"last":1.181},{"interface":"front","average":{"1min":1.319,"5min":1.109,"15min":1.1359999999999999},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":11.997999999999999,"5min":11.997999999999999,"15min":11.997999999999999},"last":1.411}]},{"osd":5,"last update":"Wed Feb 10 04:51:10 2021","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.2589999999999999,"15min":1.6870000000000001},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.56899999999999995},"max":{"1min":4.2249999999999996,"5min":4.2249999999999996,"15min":81.540000000000006},"last":1.51},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.595,"15min":1.9330000000000001},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.66600000000000004},"max":{"1min":5.5069999999999997,"5min":5.5069999999999997,"15min":81.492999999999995},"last":1.6739999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:50:31 2021","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.3220000000000001,"15min":1.3779999999999999},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.665,"5min":2.2010000000000001,"15min":5.1820000000000004},"last":1.6559999999999999},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.625,"15min":1.6679999999999999},"min":{"1min":1.008,"5min":0.98399999999999999,"15min":0.77500000000000002},"max":{"1min":2.2799999999999998,"5min":3.2349999999999999,"15min":13.315},"last":1.605}]}]},{"osd":1,"up_from":13,"seq":55834575179,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":347028,"kb_used_data":292308,"kb_used_omap":0,"kb_used_meta":54592,"kb_avail":93398124,"statfs":{"total":95995035648,"available":95639678976,"internally_reserved":0,"allocated":299323392,"data_stored":381852954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55902208},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:50:45 2021","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":2.484,"15min":1.8859999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":115.622,"5min":115.622,"15min":115.622},"last":0.56999999999999995},{"interface":"front","average":{"1min":8.2690000000000001,"5min":2.4289999999999998,"15min":1.9119999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":115.73099999999999,"5min":115.73099999999999,"15min":115.73099999999999},"last":0.72599999999999998}]},{"osd":2,"last update":"Wed Feb 10 04:51:04 2021","interfaces":[{"interface":"back","average":{"1min":6.8689999999999998,"5min":2.1139999999999999,"15min":1.6619999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":115.59699999999999,"5min":115.59699999999999,"15min":115.59699999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":6.9539999999999997,"5min":2.1899999999999999,"15min":1.704},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.46800000000000003},"max":{"1min":115.753,"5min":115.753,"15min":115.753},"last":0.64200000000000002}]},{"osd":3,"last update":"Wed Feb 10 04:49:15 2021","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.2609999999999999,"15min":1.5740000000000001},"min":{"1min":0.80900000000000005,"5min":0.58899999999999997,"15min":0.57899999999999996},"max":{"1min":2.262,"5min":4.9909999999999997,"15min":32.432000000000002},"last":0.753},{"interface":"front","average":{"1min":1.258,"5min":1.351,"15min":1.601},"min":{"1min":0.629,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":2.226,"5min":5.9720000000000004,"15min":32.811},"last":0.89800000000000002}]},{"osd":4,"last update":"Wed Feb 10 04:50:45 2021","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":1.169,"15min":1.2330000000000001},"min":{"1min":0.67900000000000005,"5min":0.66100000000000003,"15min":0.55200000000000005},"max":{"1min":1.8899999999999999,"5min":4.899,"15min":6.3099999999999996},"last":1.173},{"interface":"front","average":{"1min":1.075,"5min":1.198,"15min":1.3939999999999999},"min":{"1min":0.79500000000000004,"5min":0.747,"15min":0.70999999999999996},"max":{"1min":2.4649999999999999,"5min":2.4649999999999999,"15min":29.547999999999998},"last":1.4610000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:50:52 2021","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.2030000000000001,"15min":1.4590000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":2.266,"5min":5.0300000000000002,"15min":48.686999999999998},"last":1.333},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.411,"15min":1.778},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.7050000000000001,"5min":5.0389999999999997,"15min":48.905000000000001},"last":1.1000000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:51:04 2021","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":1.26,"15min":1.6930000000000001},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.7030000000000001,"5min":2.1659999999999999,"15min":48.878999999999998},"last":2.0089999999999999},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.329,"15min":1.609},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":2.661,"5min":5.1429999999999998,"15min":49.011000000000003},"last":1.4239999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:50:19 2021","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.464,"15min":1.5900000000000001},"min":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.81799999999999995},"max":{"1min":1.706,"5min":4.9770000000000003,"15min":28.573},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.353,"5min":1.4430000000000001,"15min":1.6299999999999999},"min":{"1min":0.87,"5min":0.87,"15min":0.87},"max":{"1min":2.298,"5min":2.298,"15min":29.408999999999999},"last":1.3600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738704,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":395112,"kb_used_data":334696,"kb_used_omap":0,"kb_used_meta":60288,"kb_avail":93350040,"statfs":{"total":95995035648,"available":95590440960,"internally_reserved":0,"allocated":342728704,"data_stored":467908108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":61734912},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 04:50:30 2021","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.88500000000000001,"15min":1.0800000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.704,"5min":2.2930000000000001,"15min":28.946999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.94699999999999995,"15min":1.022},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.534,"5min":2.0670000000000002,"15min":3.7050000000000001},"last":1.1819999999999999}]},{"osd":2,"last update":"Wed Feb 10 04:51:07 2021","interfaces":[{"interface":"back","average":{"1min":7.0490000000000004,"5min":2.2650000000000001,"15min":1.6870000000000001},"min":{"1min":0.57299999999999995,"5min":0.52100000000000002,"15min":0.51400000000000001},"max":{"1min":97.983999999999995,"5min":97.983999999999995,"15min":97.983999999999995},"last":0.59799999999999998},{"interface":"front","average":{"1min":6.992,"5min":2.2389999999999999,"15min":1.591},"min":{"1min":0.51300000000000001,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":97.844999999999999,"5min":97.844999999999999,"15min":97.844999999999999},"last":0.747}]},{"osd":3,"last update":"Wed Feb 10 04:49:20 2021","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.1870000000000001,"15min":1.4099999999999999},"min":{"1min":0.55300000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":3.3420000000000001,"5min":3.3420000000000001,"15min":29.692},"last":1.046},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.401,"15min":1.712},"min":{"1min":0.83999999999999997,"5min":0.753,"15min":0.68600000000000005},"max":{"1min":2.6379999999999999,"5min":4.0700000000000003,"15min":29.814},"last":0.71699999999999997}]},{"osd":4,"last update":"Wed Feb 10 04:50:30 2021","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":1.1819999999999999,"15min":1.3280000000000001},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.7330000000000001,"5min":2.206,"15min":5.1680000000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":1.099,"5min":1.2230000000000001,"15min":1.286},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.8420000000000001,"5min":3.359,"15min":14.672000000000001},"last":0.88800000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:50:53 2021","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":1.429,"15min":1.5429999999999999},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.4370000000000001,"5min":6.5990000000000002,"15min":25.047000000000001},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.042,"5min":1.4870000000000001,"15min":1.796},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.758,"5min":3.7850000000000001,"15min":28.827999999999999},"last":1.3220000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:50:20 2021","interfaces":[{"interface":"back","average":{"1min":3.6930000000000001,"5min":2.0139999999999998,"15min":1.8799999999999999},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":45.012,"5min":45.012,"15min":45.012},"last":1.339},{"interface":"front","average":{"1min":3.7509999999999999,"5min":2.097,"15min":2.069},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":45.225000000000001,"5min":45.225000000000001,"15min":45.225000000000001},"last":1.1359999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:50:23 2021","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.4399999999999999,"15min":1.5589999999999999},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.8999999999999999,"5min":2.8700000000000001,"15min":5.6050000000000004},"last":1.282},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.554,"15min":1.671},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":2.1320000000000001,"5min":3.4689999999999999,"15min":16.545000000000002},"last":1.159}]}]},{"osd":2,"up_from":0,"seq":77309411657,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":1111,"seq":4771708665860,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":338292,"kb_used_data":319476,"kb_used_omap":386,"kb_used_meta":18301,"kb_avail":93406860,"statfs":{"total":95995035648,"available":95648624640,"internally_reserved":0,"allocated":327143424,"data_stored":451847026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":396036,"internal_metadata":18740476},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4950000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0059999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]}]},{"osd":4,"up_from":31,"seq":133143986497,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":120556,"kb_used_data":71596,"kb_used_omap":0,"kb_used_meta":48832,"kb_avail":93624596,"statfs":{"total":95995035648,"available":95871586304,"internally_reserved":0,"allocated":73314304,"data_stored":93728015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":50003968},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4],"upper_bound":1048576},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:50:32 2021","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":2.1219999999999999,"15min":1.6120000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":3.2370000000000001,"5min":95.721999999999994,"15min":95.721999999999994},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.77500000000000002,"5min":2.1509999999999998,"15min":1.6180000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.637,"5min":95.344999999999999,"15min":95.344999999999999},"last":0.93700000000000006}]},{"osd":1,"last update":"Wed Feb 10 04:50:32 2021","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":0.94299999999999995,"15min":0.96199999999999997},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":7.883,"5min":7.883,"15min":24.295000000000002},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.081,"5min":1.119,"15min":1.24},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.60399999999999998},"max":{"1min":3.1160000000000001,"5min":4.3449999999999998,"15min":24.943000000000001},"last":1.3540000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:50:32 2021","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.208,"15min":1.1879999999999999},"min":{"1min":0.69199999999999995,"5min":0.627,"15min":0.61299999999999999},"max":{"1min":3.359,"5min":4.2279999999999998,"15min":7.048},"last":1.411},{"interface":"front","average":{"1min":1.054,"5min":1.232,"15min":1.2050000000000001},"min":{"1min":0.73399999999999999,"5min":0.60299999999999998,"15min":0.51900000000000002},"max":{"1min":1.663,"5min":4.3840000000000003,"15min":4.3840000000000003},"last":1.1319999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:49:28 2021","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":2.581,"15min":1.8220000000000001},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":2.1920000000000002,"5min":95.533000000000001,"15min":95.533000000000001},"last":1.669},{"interface":"front","average":{"1min":1.2450000000000001,"5min":2.508,"15min":1.7769999999999999},"min":{"1min":0.875,"5min":0.74299999999999999,"15min":0.70299999999999996},"max":{"1min":2.5800000000000001,"5min":95.426000000000002,"15min":95.426000000000002},"last":1.8500000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:50:47 2021","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":1.845,"15min":1.597},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.60999999999999999},"max":{"1min":8.1509999999999998,"5min":35.774000000000001,"15min":35.774000000000001},"last":1.232},{"interface":"front","average":{"1min":1.413,"5min":1.8,"15min":1.5389999999999999},"min":{"1min":0.76200000000000001,"5min":0.67100000000000004,"15min":0.58799999999999997},"max":{"1min":7.9950000000000001,"5min":35.606000000000002,"15min":35.606000000000002},"last":1.165}]},{"osd":6,"last update":"Wed Feb 10 04:50:59 2021","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":3.8809999999999998,"15min":2.54},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.70299999999999996},"max":{"1min":4.5270000000000001,"5min":201.626,"15min":201.626},"last":1.5409999999999999},{"interface":"front","average":{"1min":1.3089999999999999,"5min":3.9359999999999999,"15min":2.617},"min":{"1min":0.90200000000000002,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":1.855,"5min":201.577,"15min":201.577},"last":1.3029999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:50:25 2021","interfaces":[{"interface":"back","average":{"1min":1.6910000000000001,"5min":1.512,"15min":1.5189999999999999},"min":{"1min":1.0349999999999999,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":4.7439999999999998,"5min":4.7439999999999998,"15min":6.9199999999999999},"last":1.7569999999999999},{"interface":"front","average":{"1min":1.397,"5min":1.6060000000000001,"15min":1.5800000000000001},"min":{"1min":0.91400000000000003,"5min":0.72899999999999998,"15min":0.71499999999999997},"max":{"1min":1.831,"5min":5.149,"15min":5.149},"last":1.794}]}]},{"osd":5,"up_from":37,"seq":158913790270,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":386948,"kb_used_data":345148,"kb_used_omap":0,"kb_used_meta":41792,"kb_avail":93358204,"statfs":{"total":95995035648,"available":95598800896,"internally_reserved":0,"allocated":353431552,"data_stored":447702009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42795008},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:50:48 2021","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.87,"15min":0.93600000000000005},"min":{"1min":0.443,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.984,"5min":2.2509999999999999,"15min":4.5720000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.98699999999999999,"15min":1.0309999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46400000000000002},"max":{"1min":1.5800000000000001,"5min":3.0169999999999999,"15min":15.539999999999999},"last":0.70899999999999996}]},{"osd":1,"last update":"Wed Feb 10 04:50:48 2021","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.878,"15min":0.92600000000000005},"min":{"1min":0.55200000000000005,"5min":0.52800000000000002,"15min":0.504},"max":{"1min":2.1179999999999999,"5min":2.1179999999999999,"15min":5.29},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.095,"5min":1.1040000000000001,"15min":1.137},"min":{"1min":0.63700000000000001,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":2.0609999999999999,"5min":3.9180000000000001,"15min":11.875999999999999},"last":1.075}]},{"osd":2,"last update":"Wed Feb 10 04:50:48 2021","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.119,"15min":1.3580000000000001},"min":{"1min":0.627,"5min":0.53700000000000003,"15min":0.50600000000000001},"max":{"1min":1.7849999999999999,"5min":2.141,"15min":44.462000000000003},"last":1.222},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.1240000000000001,"15min":1.365},"min":{"1min":0.67800000000000005,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":2.2189999999999999,"5min":2.3420000000000001,"15min":44.398000000000003},"last":1.6619999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:49:47 2021","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.855,"15min":1.6559999999999999},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.0129999999999999,"5min":45.643000000000001,"15min":45.643000000000001},"last":1.339},{"interface":"front","average":{"1min":1.139,"5min":1.8169999999999999,"15min":1.702},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.8380000000000001,"5min":45.517000000000003,"15min":45.517000000000003},"last":1.5189999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:50:48 2021","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.2709999999999999,"15min":1.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.55400000000000005},"max":{"1min":1.849,"5min":3.1309999999999998,"15min":6.1639999999999997},"last":1.3129999999999999},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.3240000000000001,"15min":1.393},"min":{"1min":0.68100000000000005,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.9279999999999999,"5min":2.5699999999999998,"15min":3.0209999999999999},"last":1.4550000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:51:01 2021","interfaces":[{"interface":"back","average":{"1min":1.5800000000000001,"5min":2.294,"15min":3.036},"min":{"1min":0.85099999999999998,"5min":0.81200000000000006,"15min":0.63500000000000001},"max":{"1min":5.8879999999999999,"5min":38.014000000000003,"15min":319.51999999999998},"last":1.581},{"interface":"front","average":{"1min":1.427,"5min":2.1890000000000001,"15min":2.9300000000000002},"min":{"1min":0.80400000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":5.5750000000000002,"5min":37.896999999999998,"15min":319.22300000000001},"last":1.7889999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:50:23 2021","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.508,"15min":1.4810000000000001},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.752},"max":{"1min":2.0350000000000001,"5min":3.3479999999999999,"15min":4.6699999999999999},"last":1.633},{"interface":"front","average":{"1min":1.554,"5min":1.583,"15min":1.611},"min":{"1min":0.92500000000000004,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.2210000000000001,"5min":2.9279999999999999,"15min":3.1040000000000001},"last":1.8600000000000001}]}]}]}} 2021-02-10T04:51:18.638 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:51:18.892 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:51:18.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:18 smithi164 conmon[29288]: cluster 2021-02-10T04:51:18.472526+0000 mon.a (mon. 2021-02-10T04:51:18.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:18 smithi164 conmon[29288]: 0) 2310 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 407 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:51:18.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[29319]: cluster 2021-02-10T04:51:18.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[29319]: 2021-02-10T04:51:18.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[29319]: 2021 2021-02-10T04:51:18.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[29319]: -02-10T04:51:18.472526 2021-02-10T04:51:18.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2310 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 407 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:51:18.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: cluster 2021-02-10T04:51:18.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: 2021-02-10T04:51:18.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: 2021 2021-02-10T04:51:18.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: - 2021-02-10T04:51:18.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: 02 2021-02-10T04:51:18.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: - 2021-02-10T04:51:18.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: 10 2021-02-10T04:51:18.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: T 2021-02-10T04:51:18.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: 04 2021-02-10T04:51:18.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: : 2021-02-10T04:51:18.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: 51 2021-02-10T04:51:18.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: : 2021-02-10T04:51:18.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: 18 2021-02-10T04:51:18.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: . 2021-02-10T04:51:18.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: 472526 2021-02-10T04:51:18.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: +0000 2021-02-10T04:51:18.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: 2021-02-10T04:51:18.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: mon.a 2021-02-10T04:51:18.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: ( 2021-02-10T04:51:18.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: mon 2021-02-10T04:51:18.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: . 2021-02-10T04:51:18.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: 0 2021-02-10T04:51:18.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: ) 2021-02-10T04:51:18.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: 2310 2021-02-10T04:51:18.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: : 2021-02-10T04:51:18.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: cluster 2021-02-10T04:51:18.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: 2021-02-10T04:51:18.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: [WRN] 2021-02-10T04:51:18.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: 2021-02-10T04:51:18.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: Health check update: 4 slow ops, oldest one blocked for 407 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:51:18.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:18 smithi085 conmon[35961]: 2021-02-10T04:51:19.393 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:51:19 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:51:19.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[35961]: cluster 2021-02-10T04:51:18.413957+0000 mgr.y (mgr.14140 2021-02-10T04:51:19.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[35961]: ) 1202 : cluster [DBG] pgmap v1996: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:51:19.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[35961]: cluster 2021-02-10T04:51:18.635725+0000 mon.a (mon.0 2021-02-10T04:51:19.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[35961]: ) 2311 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 407 sec, osd.4 has slow ops) 2021-02-10T04:51:19.645 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[35961]: audit 2021-02-10T04:51:19.061108 2021-02-10T04:51:19.646 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1203 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:51:19.649 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: cluster 2021-02-10T04:51:19.649 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: 2021-02-10T04:51:19.649 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: 2021 2021-02-10T04:51:19.649 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: - 2021-02-10T04:51:19.649 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: 02 2021-02-10T04:51:19.650 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: - 2021-02-10T04:51:19.650 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: 10 2021-02-10T04:51:19.650 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: T 2021-02-10T04:51:19.650 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: 04 2021-02-10T04:51:19.650 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: : 2021-02-10T04:51:19.650 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: 51 2021-02-10T04:51:19.650 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: : 2021-02-10T04:51:19.650 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: 18 2021-02-10T04:51:19.650 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: . 2021-02-10T04:51:19.650 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: 413957 2021-02-10T04:51:19.650 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: +0000 2021-02-10T04:51:19.650 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: 2021-02-10T04:51:19.651 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: mgr.y 2021-02-10T04:51:19.651 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: ( 2021-02-10T04:51:19.651 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: mgr 2021-02-10T04:51:19.651 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: . 2021-02-10T04:51:19.651 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: 14140 2021-02-10T04:51:19.651 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: ) 2021-02-10T04:51:19.651 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: 1202 2021-02-10T04:51:19.651 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: : 2021-02-10T04:51:19.651 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: cluster 2021-02-10T04:51:19.651 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: 2021-02-10T04:51:19.651 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: [DBG] 2021-02-10T04:51:19.651 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: 2021-02-10T04:51:19.651 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: pgmap v1996: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:51:19.652 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: 2021-02-10T04:51:19.652 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: cluster 2021-02-10T04:51:19.652 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: 2021-02-10T04:51:19.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: 2021 2021-02-10T04:51:19.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: - 2021-02-10T04:51:19.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: 02 2021-02-10T04:51:19.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: - 2021-02-10T04:51:19.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: 10 2021-02-10T04:51:19.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: T 2021-02-10T04:51:19.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: 04 2021-02-10T04:51:19.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: : 2021-02-10T04:51:19.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: 51 2021-02-10T04:51:19.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: : 2021-02-10T04:51:19.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: 18 2021-02-10T04:51:19.653 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: . 2021-02-10T04:51:19.654 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: 635725 2021-02-10T04:51:19.654 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: +0000 2021-02-10T04:51:19.654 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: 2021-02-10T04:51:19.654 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: mon.a 2021-02-10T04:51:19.654 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: (mon.0) 2311 : cluster [INF] 2021-02-10T04:51:19.654 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 407 sec, osd.4 has slow ops) 2021-02-10T04:51:19.654 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: audit 2021-02-10T04:51: 2021-02-10T04:51:19.654 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: 19.061108+0000 mgr.y (mgr.14140) 1203 : audit 2021-02-10T04:51:19.654 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:19 smithi085 conmon[29319]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:51:20.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:19 smithi164 conmon[29288]: cluster 2021-02-10T04:51:18.413957+0000 mgr.y (mgr.14140) 1202 : cluster [DBG] 2021-02-10T04:51:20.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:19 smithi164 conmon[29288]: pgmap v1996: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:51:20.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:19 smithi164 conmon[29288]: cluster 2021-02-10T04:51:18.635725+0000 mon.a (mon.0) 2311 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 407 sec, osd.4 has slow ops) 2021-02-10T04:51:20.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:19 smithi164 conmon[29288]: audit 2021-02-10T04:51:19.061108+0000 mgr.y (mgr.14140) 1203 : audit 2021-02-10T04:51:20.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:19 smithi164 conmon[29288]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:51:20.740 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:51:20.740 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:51:21.301 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:51:21.565 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:51:21.655 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:21 smithi085 conmon[29319]: cluster 2021-02-10T04:51:20.414513+0000 mgr.y (mgr.14140) 1204 : cluster [DBG] pgmap v1997: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:51:21.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:21 smithi085 conmon[35961]: cluster 2021-02-10T04:51:20.414513+0000 mgr.y (mgr.14140) 1204 : cluster [DBG] pgmap v1997: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:51:22.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:21 smithi164 conmon[29288]: cluster 2021-02-10T04:51:20. 2021-02-10T04:51:22.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:21 smithi164 conmon[29288]: 414513+0000 mgr.y (mgr.14140) 1204 : cluster [DBG] pgmap v1997: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:51:23.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:23 smithi085 conmon[29319]: cluster 2021-02-10T04:51:22.414965+0000 mgr.y (mgr.14140) 1205 : cluster [DBG] pgmap v1998: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:51:23.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:23 smithi085 conmon[29319]: 2021-02-10T04:51:23.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:23 smithi085 conmon[35961]: cluster 2021-02-10T04:51:22.414965+0000 mgr.y (mgr. 2021-02-10T04:51:23.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:23 smithi085 conmon[35961]: 14140) 1205 : cluster [DBG] pgmap v1998: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:51:24.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:23 smithi164 conmon[29288]: cluster 2021-02-10T04:51:22.414965+0000 mgr.y ( 2021-02-10T04:51:24.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:23 smithi164 conmon[29288]: mgr.14140) 1205 : cluster [DBG] pgmap v1998: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:51:24.153 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.3 dump_blocked_ops 2021-02-10T04:51:24.409 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:51:25.657 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:25 smithi085 conmon[29319]: cluster 2021-02-10T04:51:24.415651+0000 mgr.y (mgr.14140) 1206 : cluster [DBG] pgmap v1999: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:51:25.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:25 smithi085 conmon[35961]: cluster 2021-02-10T04:51:25.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:25 smithi085 conmon[35961]: 2021-02-10T04:51:25.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:25 smithi085 conmon[35961]: 2021 2021-02-10T04:51:25.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:25 smithi085 conmon[35961]: -02-10T04:51:24. 2021-02-10T04:51:25.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:25 smithi085 conmon[35961]: 415651+0000 mgr.y (mgr.14140) 1206 : cluster [DBG] pgmap v1999: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:51:26.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:25 smithi164 conmon[29288]: cluster 2021-02-10T04:51:26.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:25 smithi164 conmon[29288]: 2021-02-10T04:51:26.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:25 smithi164 conmon[29288]: 2021-02-10T04:51:24. 2021-02-10T04:51:26.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:25 smithi164 conmon[29288]: 415651+0000 mgr.y (mgr.14140) 1206 : cluster [DBG] pgmap v1999: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:51:26.822 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:51:26.825 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:51:27.658 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:27 smithi085 conmon[35961]: cluster 2021-02-10T04:51:26 2021-02-10T04:51:27.658 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:27 smithi085 conmon[35961]: .416256+0000 mgr.y (mgr.14140) 1207 : cluster [DBG] pgmap v2000: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:51:27.658 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:27 smithi085 conmon[35961]: audit 2021-02-10T04:51:27.658 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:27 smithi085 conmon[35961]: 2021-02-10T04:51: 2021-02-10T04:51:27.658 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:27 smithi085 conmon[35961]: 26.800694+0000 mgr.y (mgr.14140 2021-02-10T04:51:27.659 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:27 smithi085 conmon[35961]: ) 1208 : audit [DBG] from='client.25546 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:51:27.659 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:27 smithi085 conmon[35961]: audit 2021 2021-02-10T04:51:27.659 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:27 smithi085 conmon[35961]: -02-10T04:51:26.805987+0000 mon.a (mon 2021-02-10T04:51:27.659 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:27 smithi085 conmon[35961]: .0) 2312 : audit [INF] from='client.? 172.21.15.85:0/798252887' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:51:27.667 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is set 2021-02-10T04:51:27.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:27 smithi085 conmon[29319]: cluster 2021-02-10T04:51:26.416256+0000 mgr.y (mgr.14140) 1207 2021-02-10T04:51:27.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:27 smithi085 conmon[29319]: : cluster [DBG] pgmap v2000: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:51:27.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:27 smithi085 conmon[29319]: audit 2021-02-10T04:51:26.800694+0000 mgr.y (mgr.14140) 1208 : audit [DBG] from='client.25546 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:51:27.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:27 smithi085 conmon[29319]: audit 2021-02-10T04:51:26.805987+0000 mon.a (mon.0) 2312 : audit [INF] from='client.? 172.21.15.85:0/798252887' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:51:28.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:27 smithi164 conmon[29288]: cluster 2021-02-10T04:51:26.416256+0000 mgr.y (mgr 2021-02-10T04:51:28.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:27 smithi164 conmon[29288]: .14140) 1207 : cluster [DBG] pgmap v2000: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:51:28.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:27 smithi164 conmon[29288]: audit 2021-02-10T04:51:26.800694+0000 mgr.y (mgr.14140) 1208 : audit [DBG] from='client.25546 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:51:28.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:27 smithi164 conmon[29288]: audit 2021-02-10T04:51:26.805987+0000 2021-02-10T04:51:28.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:27 smithi164 conmon[29288]: mon.a (mon.0) 2312 : audit [INF] from='client.? 172.21.15.85:0/798252887' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:51:28.853 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:28 smithi085 conmon[35961]: cluster 2021-02-10T04:51:27.404663+0000 2021-02-10T04:51:28.854 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:28 smithi085 conmon[35961]: osd.6 (osd.6) 159 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:51:28.854 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:28 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:51:28.854 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:28 smithi085 conmon[35961]: :51:27.406664+0000 osd.6 (osd.6) 160 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:51:28.854 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:28 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:51:28.854 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:28 smithi085 conmon[35961]: 02-10T04:51:27.656059+0000 mon.a (mon.0) 2313 2021-02-10T04:51:28.854 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:28 smithi085 conmon[35961]: : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:51:28.854 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:28 smithi085 conmon[35961]: audit 2021-02-10T04:51 2021-02-10T04:51:28.854 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:28 smithi085 conmon[35961]: :27.663731+0000 mon.a (mon.0) 2314 : 2021-02-10T04:51:28.854 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:28 smithi085 conmon[35961]: audit [INF] from='client.? 172.21.15.85:0/798252887' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:51:28.854 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:28 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:51:28.855 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:28 smithi085 conmon[35961]: 02-10T04:51:27.663789+0000 mon.a (mon.0) 2315 : cluster 2021-02-10T04:51:28.855 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:28 smithi085 conmon[35961]: [DBG] osdmap e1117: 8 total, 8 up, 7 in 2021-02-10T04:51:28.855 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:28 smithi085 conmon[29319]: cluster 2021-02-10T04:51:27. 2021-02-10T04:51:28.855 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:28 smithi085 conmon[29319]: 404663+0000 osd.6 (osd.6) 159 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:51:28.855 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:28 smithi085 conmon[29319]: cluster 2021-02-10T04:51:27.406664+0000 osd.6 (osd.6) 160 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:51:28.855 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:28 smithi085 conmon[29319]: cluster 2021-02-10T04:51:27.656059+0000 mon.a (mon.0) 2313 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:51:28.855 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:28 smithi085 conmon[29319]: audit 2021-02-10T04:51:28.855 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:28 smithi085 conmon[29319]: 2021-02-10T04:51:27.663731+0000 mon.a (mon.0) 2314 : audit [INF] from='client.? 172.21.15.85:0/798252887' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:51:28.855 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:28 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:51:28.855 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:28 smithi085 conmon[29319]: 10T04:51:27.663789+0000 mon.a (mon.0) 2315 : cluster 2021-02-10T04:51:28.856 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:28 smithi085 conmon[29319]: [DBG] osdmap e1117: 8 total, 8 up, 7 in 2021-02-10T04:51:29.072 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:28 smithi164 conmon[29288]: cluster 2021-02-10T04:51:27.404663+0000 osd.6 (osd.6) 159 2021-02-10T04:51:29.072 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:28 smithi164 conmon[29288]: : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:51:29.072 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:28 smithi164 conmon[29288]: cluster 2021-02-10T04:51:27.406664+0000 osd.6 (osd.6) 160 : cluster 2021-02-10T04:51:29.072 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:28 smithi164 conmon[29288]: [DBG] purged_snaps scrub ok 2021-02-10T04:51:29.072 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:28 smithi164 conmon[29288]: cluster 2021-02-10T04:51 2021-02-10T04:51:29.072 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:28 smithi164 conmon[29288]: :27.656059+0000 mon.a (mon.0) 2313 : 2021-02-10T04:51:29.072 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:28 smithi164 conmon[29288]: cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:51:29.072 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:28 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:51:29.072 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:28 smithi164 conmon[29288]: 04:51:27.663731+0000 mon.a (mon.0 2021-02-10T04:51:29.073 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:28 smithi164 conmon[29288]: ) 2314 : audit [INF] from='client.? 172.21.15.85:0/798252887' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:51:29.073 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:28 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:51:29.073 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:28 smithi164 conmon[29288]: 04:51:27.663789+0000 mon.a (mon.0 2021-02-10T04:51:29.073 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:28 smithi164 conmon[29288]: ) 2315 : cluster [DBG] osdmap e1117: 8 total, 8 up, 7 in 2021-02-10T04:51:29.394 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:51:29 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:51:29.453 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":2000,"stamp":"2021-02-10T04:51:26.415859+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":940031637,"num_objects":743,"num_object_clones":336,"num_object_copies":2229,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":743,"num_whiteouts":12,"num_read":7703,"num_read_kb":4771910,"num_write":4628,"num_write_kb":1265219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":395,"num_bytes_recovered":101346816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25130,"ondisk_log_size":25130,"up":564,"acting":564,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":564,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2175184,"kb_used_data":1863064,"kb_used_omap":386,"kb_used_meta":311677,"kb_avail":654040880,"statfs":{"total":671965249536,"available":669737861120,"internally_reserved":0,"allocated":1907777536,"data_stored":2513854602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":396036,"internal_metadata":319157500},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4],"upper_bound":1048576},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003276"},"pg_stats":[{"pgid":"6.29","version":"0'0","reported_seq":"1113","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.858970+0000","last_change":"2021-02-10T04:47:12.944476+0000","last_active":"2021-02-10T04:51:14.858970+0000","last_peered":"2021-02-10T04:51:14.858970+0000","last_clean":"2021-02-10T04:51:14.858970+0000","last_became_active":"2021-02-10T04:45:49.759065+0000","last_became_peered":"2021-02-10T04:45:49.759065+0000","last_unstale":"2021-02-10T04:51:14.858970+0000","last_undegraded":"2021-02-10T04:51:14.858970+0000","last_fullsized":"2021-02-10T04:51:14.858970+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:12.944345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:12.944345+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":"506","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.626122+0000","last_change":"2021-02-10T04:47:18.026548+0000","last_active":"2021-02-10T04:51:13.626122+0000","last_peered":"2021-02-10T04:51:13.626122+0000","last_clean":"2021-02-10T04:51:13.626122+0000","last_became_active":"2021-02-10T04:45:49.761512+0000","last_became_peered":"2021-02-10T04:45:49.761512+0000","last_unstale":"2021-02-10T04:51:13.626122+0000","last_undegraded":"2021-02-10T04:51:13.626122+0000","last_fullsized":"2021-02-10T04:51:13.626122+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:18.026478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:18.026478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"131'3","reported_seq":"1289","reported_epoch":"1116","state":"active+clean","last_fresh":"2021-02-10T04:51:17.990317+0000","last_change":"2021-02-10T04:51:01.185887+0000","last_active":"2021-02-10T04:51:17.990317+0000","last_peered":"2021-02-10T04:51:17.990317+0000","last_clean":"2021-02-10T04:51:17.990317+0000","last_became_active":"2021-02-10T04:29:07.145525+0000","last_became_peered":"2021-02-10T04:29:07.145525+0000","last_unstale":"2021-02-10T04:51:17.990317+0000","last_undegraded":"2021-02-10T04:51:17.990317+0000","last_fullsized":"2021-02-10T04:51:17.990317+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'3","last_scrub_stamp":"2021-02-10T04:51:01.185820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.185820+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"1049'54","reported_seq":"1186","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.620975+0000","last_change":"2021-02-10T04:46:51.840139+0000","last_active":"2021-02-10T04:51:13.620975+0000","last_peered":"2021-02-10T04:51:13.620975+0000","last_clean":"2021-02-10T04:51:13.620975+0000","last_became_active":"2021-02-10T04:25:49.002058+0000","last_became_peered":"2021-02-10T04:25:49.002058+0000","last_unstale":"2021-02-10T04:51:13.620975+0000","last_undegraded":"2021-02-10T04:51:13.620975+0000","last_fullsized":"2021-02-10T04:51:13.620975+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:46:51.840049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:51.840049+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":95,"num_read_kb":78,"num_write":55,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"476","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.619707+0000","last_change":"2021-02-10T04:51:08.005652+0000","last_active":"2021-02-10T04:51:13.619707+0000","last_peered":"2021-02-10T04:51:13.619707+0000","last_clean":"2021-02-10T04:51:13.619707+0000","last_became_active":"2021-02-10T04:50:59.548518+0000","last_became_peered":"2021-02-10T04:50:59.548518+0000","last_unstale":"2021-02-10T04:51:13.619707+0000","last_undegraded":"2021-02-10T04:51:13.619707+0000","last_fullsized":"2021-02-10T04:51:13.619707+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:46:34.577684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:34.577684+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"1085","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.622188+0000","last_change":"2021-02-10T04:50:48.408819+0000","last_active":"2021-02-10T04:51:13.622188+0000","last_peered":"2021-02-10T04:51:13.622188+0000","last_clean":"2021-02-10T04:51:13.622188+0000","last_became_active":"2021-02-10T04:25:53.148090+0000","last_became_peered":"2021-02-10T04:25:53.148090+0000","last_unstale":"2021-02-10T04:51:13.622188+0000","last_undegraded":"2021-02-10T04:51:13.622188+0000","last_fullsized":"2021-02-10T04:51:13.622188+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:48.408748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:50:48.408748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"1003'1689","reported_seq":"4653","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.622858+0000","last_change":"2021-02-10T04:46:54.814105+0000","last_active":"2021-02-10T04:51:13.622858+0000","last_peered":"2021-02-10T04:51:13.622858+0000","last_clean":"2021-02-10T04:51:13.622858+0000","last_became_active":"2021-02-10T04:33:05.144488+0000","last_became_peered":"2021-02-10T04:33:05.144488+0000","last_unstale":"2021-02-10T04:51:13.622858+0000","last_undegraded":"2021-02-10T04:51:13.622858+0000","last_fullsized":"2021-02-10T04:51:13.622858+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:46:54.814020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:54.814020+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":126521476,"num_objects":49,"num_object_clones":44,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":773,"num_read_kb":819582,"num_write":561,"num_write_kb":221117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"1049'32","reported_seq":"1179","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.859563+0000","last_change":"2021-02-10T04:46:31.614762+0000","last_active":"2021-02-10T04:51:14.859563+0000","last_peered":"2021-02-10T04:51:14.859563+0000","last_clean":"2021-02-10T04:51:14.859563+0000","last_became_active":"2021-02-10T04:38:39.243413+0000","last_became_peered":"2021-02-10T04:38:39.243413+0000","last_unstale":"2021-02-10T04:51:14.859563+0000","last_undegraded":"2021-02-10T04:51:14.859563+0000","last_fullsized":"2021-02-10T04:51:14.859563+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'32","last_scrub_stamp":"2021-02-10T04:46:31.614697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:31.614697+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":62,"num_read_kb":45,"num_write":33,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":894,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"784","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.622597+0000","last_change":"2021-02-10T04:50:59.824441+0000","last_active":"2021-02-10T04:51:13.622597+0000","last_peered":"2021-02-10T04:51:13.622597+0000","last_clean":"2021-02-10T04:51:13.622597+0000","last_became_active":"2021-02-10T04:50:59.526936+0000","last_became_peered":"2021-02-10T04:50:59.526936+0000","last_unstale":"2021-02-10T04:51:13.622597+0000","last_undegraded":"2021-02-10T04:51:13.622597+0000","last_fullsized":"2021-02-10T04:51:13.622597+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:59.824373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:59.824373+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"508","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.620483+0000","last_change":"2021-02-10T04:50:59.571325+0000","last_active":"2021-02-10T04:51:13.620483+0000","last_peered":"2021-02-10T04:51:13.620483+0000","last_clean":"2021-02-10T04:51:13.620483+0000","last_became_active":"2021-02-10T04:50:59.570943+0000","last_became_peered":"2021-02-10T04:50:59.570943+0000","last_unstale":"2021-02-10T04:51:13.620483+0000","last_undegraded":"2021-02-10T04:51:13.620483+0000","last_fullsized":"2021-02-10T04:51:13.620483+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:46.664985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:46:46.664985+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"1120","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.859354+0000","last_change":"2021-02-10T04:50:59.577404+0000","last_active":"2021-02-10T04:51:14.859354+0000","last_peered":"2021-02-10T04:51:14.859354+0000","last_clean":"2021-02-10T04:51:14.859354+0000","last_became_active":"2021-02-10T04:50:59.577097+0000","last_became_peered":"2021-02-10T04:50:59.577097+0000","last_unstale":"2021-02-10T04:51:14.859354+0000","last_undegraded":"2021-02-10T04:51:14.859354+0000","last_fullsized":"2021-02-10T04:51:14.859354+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:09.922789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:09.922789+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"259'74","reported_seq":"2033","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.619818+0000","last_change":"2021-02-10T04:50:59.556045+0000","last_active":"2021-02-10T04:51:13.619818+0000","last_peered":"2021-02-10T04:51:13.619818+0000","last_clean":"2021-02-10T04:51:13.619818+0000","last_became_active":"2021-02-10T04:50:59.555212+0000","last_became_peered":"2021-02-10T04:50:59.555212+0000","last_unstale":"2021-02-10T04:51:13.619818+0000","last_undegraded":"2021-02-10T04:51:13.619818+0000","last_fullsized":"2021-02-10T04:51:13.619818+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'62","last_scrub_stamp":"2021-02-10T04:30:10.905905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:10.905905+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1218524,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":28227,"num_write":19,"num_write_kb":7021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":567285,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"1049'61","reported_seq":"1199","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.626649+0000","last_change":"2021-02-10T04:50:54.154530+0000","last_active":"2021-02-10T04:51:13.626649+0000","last_peered":"2021-02-10T04:51:13.626649+0000","last_clean":"2021-02-10T04:51:13.626649+0000","last_became_active":"2021-02-10T04:25:49.015913+0000","last_became_peered":"2021-02-10T04:25:49.015913+0000","last_unstale":"2021-02-10T04:51:13.626649+0000","last_undegraded":"2021-02-10T04:51:13.626649+0000","last_fullsized":"2021-02-10T04:51:13.626649+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'61","last_scrub_stamp":"2021-02-10T04:50:54.154436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:50:54.154436+0000","log_size":61,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":104,"num_read_kb":84,"num_write":62,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"642'3","reported_seq":"690","reported_epoch":"1116","state":"active+clean","last_fresh":"2021-02-10T04:51:16.637523+0000","last_change":"2021-02-10T04:47:02.791566+0000","last_active":"2021-02-10T04:51:16.637523+0000","last_peered":"2021-02-10T04:51:16.637523+0000","last_clean":"2021-02-10T04:51:16.637523+0000","last_became_active":"2021-02-10T04:37:37.234230+0000","last_became_peered":"2021-02-10T04:37:37.234230+0000","last_unstale":"2021-02-10T04:51:16.637523+0000","last_undegraded":"2021-02-10T04:51:16.637523+0000","last_fullsized":"2021-02-10T04:51:16.637523+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'3","last_scrub_stamp":"2021-02-10T04:47:02.791478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:02.791478+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"504","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.859480+0000","last_change":"2021-02-10T04:50:52.421852+0000","last_active":"2021-02-10T04:51:14.859480+0000","last_peered":"2021-02-10T04:51:14.859480+0000","last_clean":"2021-02-10T04:51:14.859480+0000","last_became_active":"2021-02-10T04:37:37.230818+0000","last_became_peered":"2021-02-10T04:37:37.230818+0000","last_unstale":"2021-02-10T04:51:14.859480+0000","last_undegraded":"2021-02-10T04:51:14.859480+0000","last_fullsized":"2021-02-10T04:51:14.859480+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:52.421793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:52.421793+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"490","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.618806+0000","last_change":"2021-02-10T04:51:07.998889+0000","last_active":"2021-02-10T04:51:13.618806+0000","last_peered":"2021-02-10T04:51:13.618806+0000","last_clean":"2021-02-10T04:51:13.618806+0000","last_became_active":"2021-02-10T04:50:59.549316+0000","last_became_peered":"2021-02-10T04:50:59.549316+0000","last_unstale":"2021-02-10T04:51:13.618806+0000","last_undegraded":"2021-02-10T04:51:13.618806+0000","last_fullsized":"2021-02-10T04:51:13.618806+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.731458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.731458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"1003'739","reported_seq":"3781","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.859046+0000","last_change":"2021-02-10T04:46:47.806184+0000","last_active":"2021-02-10T04:51:14.859046+0000","last_peered":"2021-02-10T04:51:14.859046+0000","last_clean":"2021-02-10T04:51:14.859046+0000","last_became_active":"2021-02-10T04:44:25.545483+0000","last_became_peered":"2021-02-10T04:44:25.545483+0000","last_unstale":"2021-02-10T04:51:14.859046+0000","last_undegraded":"2021-02-10T04:51:14.859046+0000","last_fullsized":"2021-02-10T04:51:14.859046+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:46:47.806105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:47.806105+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25888931,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"1049'61","reported_seq":"1213","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.623185+0000","last_change":"2021-02-10T04:50:47.662471+0000","last_active":"2021-02-10T04:51:13.623185+0000","last_peered":"2021-02-10T04:51:13.623185+0000","last_clean":"2021-02-10T04:51:13.623185+0000","last_became_active":"2021-02-10T04:37:37.237215+0000","last_became_peered":"2021-02-10T04:37:37.237215+0000","last_unstale":"2021-02-10T04:51:13.623185+0000","last_undegraded":"2021-02-10T04:51:13.623185+0000","last_fullsized":"2021-02-10T04:51:13.623185+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'61","last_scrub_stamp":"2021-02-10T04:50:47.662378+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:50:47.662378+0000","log_size":61,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":83,"num_read_kb":65,"num_write":50,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"1006","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.620768+0000","last_change":"2021-02-10T04:50:45.623700+0000","last_active":"2021-02-10T04:51:13.620768+0000","last_peered":"2021-02-10T04:51:13.620768+0000","last_clean":"2021-02-10T04:51:13.620768+0000","last_became_active":"2021-02-10T04:29:07.145616+0000","last_became_peered":"2021-02-10T04:29:07.145616+0000","last_unstale":"2021-02-10T04:51:13.620768+0000","last_undegraded":"2021-02-10T04:51:13.620768+0000","last_fullsized":"2021-02-10T04:51:13.620768+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:45.623640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:45.623640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"764","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.621865+0000","last_change":"2021-02-10T04:46:43.762837+0000","last_active":"2021-02-10T04:51:13.621865+0000","last_peered":"2021-02-10T04:51:13.621865+0000","last_clean":"2021-02-10T04:51:13.621865+0000","last_became_active":"2021-02-10T04:33:05.146783+0000","last_became_peered":"2021-02-10T04:33:05.146783+0000","last_unstale":"2021-02-10T04:51:13.621865+0000","last_undegraded":"2021-02-10T04:51:13.621865+0000","last_fullsized":"2021-02-10T04:51:13.621865+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:43.762742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:46:43.762742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"464","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.619987+0000","last_change":"2021-02-10T04:51:07.991134+0000","last_active":"2021-02-10T04:51:13.619987+0000","last_peered":"2021-02-10T04:51:13.619987+0000","last_clean":"2021-02-10T04:51:13.619987+0000","last_became_active":"2021-02-10T04:50:59.585315+0000","last_became_peered":"2021-02-10T04:50:59.585315+0000","last_unstale":"2021-02-10T04:51:13.619987+0000","last_undegraded":"2021-02-10T04:51:13.619987+0000","last_fullsized":"2021-02-10T04:51:13.619987+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:13.727241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:13.727241+0000","last_clean_scrub_stamp":"2021-02-10T04:47:13.727241+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"1063'654","reported_seq":"4326","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.624070+0000","last_change":"2021-02-10T04:50:59.541622+0000","last_active":"2021-02-10T04:51:13.624070+0000","last_peered":"2021-02-10T04:51:13.624070+0000","last_clean":"2021-02-10T04:51:13.624070+0000","last_became_active":"2021-02-10T04:50:59.540629+0000","last_became_peered":"2021-02-10T04:50:59.540629+0000","last_unstale":"2021-02-10T04:51:13.624070+0000","last_undegraded":"2021-02-10T04:51:13.624070+0000","last_fullsized":"2021-02-10T04:51:13.624070+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:47:06.861811+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49401261,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":2,"num_read":342,"num_read_kb":364544,"num_write":223,"num_write_kb":67454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":11919937,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"1047","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.621320+0000","last_change":"2021-02-10T04:50:59.539568+0000","last_active":"2021-02-10T04:51:13.621320+0000","last_peered":"2021-02-10T04:51:13.621320+0000","last_clean":"2021-02-10T04:51:13.621320+0000","last_became_active":"2021-02-10T04:50:59.539264+0000","last_became_peered":"2021-02-10T04:50:59.539264+0000","last_unstale":"2021-02-10T04:51:13.621320+0000","last_undegraded":"2021-02-10T04:51:13.621320+0000","last_fullsized":"2021-02-10T04:51:13.621320+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:33.963005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:46:33.963005+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"1068","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.619451+0000","last_change":"2021-02-10T04:51:07.992892+0000","last_active":"2021-02-10T04:51:13.619451+0000","last_peered":"2021-02-10T04:51:13.619451+0000","last_clean":"2021-02-10T04:51:13.619451+0000","last_became_active":"2021-02-10T04:50:59.550989+0000","last_became_peered":"2021-02-10T04:50:59.550989+0000","last_unstale":"2021-02-10T04:51:13.619451+0000","last_undegraded":"2021-02-10T04:51:13.619451+0000","last_fullsized":"2021-02-10T04:51:13.619451+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:53.528227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:46:53.528227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"1049'33","reported_seq":"1154","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.621887+0000","last_change":"2021-02-10T04:51:06.846957+0000","last_active":"2021-02-10T04:51:13.621887+0000","last_peered":"2021-02-10T04:51:13.621887+0000","last_clean":"2021-02-10T04:51:13.621887+0000","last_became_active":"2021-02-10T04:50:59.584833+0000","last_became_peered":"2021-02-10T04:50:59.584833+0000","last_unstale":"2021-02-10T04:51:13.621887+0000","last_undegraded":"2021-02-10T04:51:13.621887+0000","last_fullsized":"2021-02-10T04:51:13.621887+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'33","last_scrub_stamp":"2021-02-10T04:51:06.846850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:06.846850+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":43,"num_write":33,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"765","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.619526+0000","last_change":"2021-02-10T04:51:07.983976+0000","last_active":"2021-02-10T04:51:13.619526+0000","last_peered":"2021-02-10T04:51:13.619526+0000","last_clean":"2021-02-10T04:51:13.619526+0000","last_became_active":"2021-02-10T04:50:59.565075+0000","last_became_peered":"2021-02-10T04:50:59.565075+0000","last_unstale":"2021-02-10T04:51:13.619526+0000","last_undegraded":"2021-02-10T04:51:13.619526+0000","last_fullsized":"2021-02-10T04:51:13.619526+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:19.573103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:19.573103+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"639'3","reported_seq":"678","reported_epoch":"1116","state":"active+clean","last_fresh":"2021-02-10T04:51:17.990729+0000","last_change":"2021-02-10T04:47:01.757109+0000","last_active":"2021-02-10T04:51:17.990729+0000","last_peered":"2021-02-10T04:51:17.990729+0000","last_clean":"2021-02-10T04:51:17.990729+0000","last_became_active":"2021-02-10T04:37:37.233738+0000","last_became_peered":"2021-02-10T04:37:37.233738+0000","last_unstale":"2021-02-10T04:51:17.990729+0000","last_undegraded":"2021-02-10T04:51:17.990729+0000","last_fullsized":"2021-02-10T04:51:17.990729+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"639'3","last_scrub_stamp":"2021-02-10T04:47:01.756908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:47:01.756908+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"1088","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.622456+0000","last_change":"2021-02-10T04:46:48.602911+0000","last_active":"2021-02-10T04:51:13.622456+0000","last_peered":"2021-02-10T04:51:13.622456+0000","last_clean":"2021-02-10T04:51:13.622456+0000","last_became_active":"2021-02-10T04:37:37.233109+0000","last_became_peered":"2021-02-10T04:37:37.233109+0000","last_unstale":"2021-02-10T04:51:13.622456+0000","last_undegraded":"2021-02-10T04:51:13.622456+0000","last_fullsized":"2021-02-10T04:51:13.622456+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:48.602819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:48.602819+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"1004'1093","reported_seq":"6354","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.860106+0000","last_change":"2021-02-10T04:50:59.597444+0000","last_active":"2021-02-10T04:51:14.860106+0000","last_peered":"2021-02-10T04:51:14.860106+0000","last_clean":"2021-02-10T04:51:14.860106+0000","last_became_active":"2021-02-10T04:50:59.596040+0000","last_became_peered":"2021-02-10T04:50:59.596040+0000","last_unstale":"2021-02-10T04:51:14.860106+0000","last_undegraded":"2021-02-10T04:51:14.860106+0000","last_fullsized":"2021-02-10T04:51:14.860106+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:46:19.757486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:19.757486+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":78869393,"num_objects":34,"num_object_clones":30,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":1,"num_read":558,"num_read_kb":622116,"num_write":371,"num_write_kb":133580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"1037","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.622048+0000","last_change":"2021-02-10T04:51:12.389350+0000","last_active":"2021-02-10T04:51:13.622048+0000","last_peered":"2021-02-10T04:51:13.622048+0000","last_clean":"2021-02-10T04:51:13.622048+0000","last_became_active":"2021-02-10T04:37:37.229648+0000","last_became_peered":"2021-02-10T04:37:37.229648+0000","last_unstale":"2021-02-10T04:51:13.622048+0000","last_undegraded":"2021-02-10T04:51:13.622048+0000","last_fullsized":"2021-02-10T04:51:13.622048+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:12.389272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:12.389272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"1049'41","reported_seq":"839","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.620167+0000","last_change":"2021-02-10T04:51:07.985033+0000","last_active":"2021-02-10T04:51:13.620167+0000","last_peered":"2021-02-10T04:51:13.620167+0000","last_clean":"2021-02-10T04:51:13.620167+0000","last_became_active":"2021-02-10T04:50:59.587994+0000","last_became_peered":"2021-02-10T04:50:59.587994+0000","last_unstale":"2021-02-10T04:51:13.620167+0000","last_undegraded":"2021-02-10T04:51:13.620167+0000","last_fullsized":"2021-02-10T04:51:13.620167+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'41","last_scrub_stamp":"2021-02-10T04:46:28.515513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:28.515513+0000","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":66,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"788","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.619356+0000","last_change":"2021-02-10T04:51:07.984726+0000","last_active":"2021-02-10T04:51:13.619356+0000","last_peered":"2021-02-10T04:51:13.619356+0000","last_clean":"2021-02-10T04:51:13.619356+0000","last_became_active":"2021-02-10T04:50:59.561519+0000","last_became_peered":"2021-02-10T04:50:59.561519+0000","last_unstale":"2021-02-10T04:51:13.619356+0000","last_undegraded":"2021-02-10T04:51:13.619356+0000","last_fullsized":"2021-02-10T04:51:13.619356+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:47.605360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_clean_scrub_stamp":"2021-02-10T04:46:47.605360+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"78'3","reported_seq":"1390","reported_epoch":"1116","state":"active+clean","last_fresh":"2021-02-10T04:51:16.638787+0000","last_change":"2021-02-10T04:46:17.816956+0000","last_active":"2021-02-10T04:51:16.638787+0000","last_peered":"2021-02-10T04:51:16.638787+0000","last_clean":"2021-02-10T04:51:16.638787+0000","last_became_active":"2021-02-10T04:25:51.025218+0000","last_became_peered":"2021-02-10T04:25:51.025218+0000","last_unstale":"2021-02-10T04:51:16.638787+0000","last_undegraded":"2021-02-10T04:51:16.638787+0000","last_fullsized":"2021-02-10T04:51:16.638787+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:46:17.816879+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:46:17.816879+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"1085","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.624401+0000","last_change":"2021-02-10T04:50:10.176968+0000","last_active":"2021-02-10T04:51:13.624401+0000","last_peered":"2021-02-10T04:51:13.624401+0000","last_clean":"2021-02-10T04:51:13.624401+0000","last_became_active":"2021-02-10T04:25:53.042697+0000","last_became_peered":"2021-02-10T04:25:53.042697+0000","last_unstale":"2021-02-10T04:51:13.624401+0000","last_undegraded":"2021-02-10T04:51:13.624401+0000","last_fullsized":"2021-02-10T04:51:13.624401+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:10.176878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:50:10.176878+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"1002'1090","reported_seq":"5299","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.627858+0000","last_change":"2021-02-10T04:51:08.220559+0000","last_active":"2021-02-10T04:51:13.627858+0000","last_peered":"2021-02-10T04:51:13.627858+0000","last_clean":"2021-02-10T04:51:13.627858+0000","last_became_active":"2021-02-10T04:50:59.585773+0000","last_became_peered":"2021-02-10T04:50:59.585773+0000","last_unstale":"2021-02-10T04:51:13.627858+0000","last_undegraded":"2021-02-10T04:51:13.627858+0000","last_fullsized":"2021-02-10T04:51:13.627858+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:51:08.220472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.220472+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":48821292,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":491,"num_read_kb":552336,"num_write":351,"num_write_kb":129921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"1049'47","reported_seq":"1164","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.622195+0000","last_change":"2021-02-10T04:50:04.767513+0000","last_active":"2021-02-10T04:51:13.622195+0000","last_peered":"2021-02-10T04:51:13.622195+0000","last_clean":"2021-02-10T04:51:13.622195+0000","last_became_active":"2021-02-10T04:25:49.011093+0000","last_became_peered":"2021-02-10T04:25:49.011093+0000","last_unstale":"2021-02-10T04:51:13.622195+0000","last_undegraded":"2021-02-10T04:51:13.622195+0000","last_fullsized":"2021-02-10T04:51:13.622195+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'47","last_scrub_stamp":"2021-02-10T04:50:04.767446+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:50:04.767446+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":68,"num_read_kb":52,"num_write":38,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"1070","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.858995+0000","last_change":"2021-02-10T04:50:08.358363+0000","last_active":"2021-02-10T04:51:14.858995+0000","last_peered":"2021-02-10T04:51:14.858995+0000","last_clean":"2021-02-10T04:51:14.858995+0000","last_became_active":"2021-02-10T04:29:07.160593+0000","last_became_peered":"2021-02-10T04:29:07.160593+0000","last_unstale":"2021-02-10T04:51:14.858995+0000","last_undegraded":"2021-02-10T04:51:14.858995+0000","last_fullsized":"2021-02-10T04:51:14.858995+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:08.358300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:50:08.358300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"1087","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.624804+0000","last_change":"2021-02-10T04:46:48.842056+0000","last_active":"2021-02-10T04:51:13.624804+0000","last_peered":"2021-02-10T04:51:13.624804+0000","last_clean":"2021-02-10T04:51:13.624804+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:51:13.624804+0000","last_undegraded":"2021-02-10T04:51:13.624804+0000","last_fullsized":"2021-02-10T04:51:13.624804+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:48.841970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:46:48.841970+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"1043","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.619812+0000","last_change":"2021-02-10T04:51:08.003846+0000","last_active":"2021-02-10T04:51:13.619812+0000","last_peered":"2021-02-10T04:51:13.619812+0000","last_clean":"2021-02-10T04:51:13.619812+0000","last_became_active":"2021-02-10T04:50:59.545501+0000","last_became_peered":"2021-02-10T04:50:59.545501+0000","last_unstale":"2021-02-10T04:51:13.619812+0000","last_undegraded":"2021-02-10T04:51:13.619812+0000","last_fullsized":"2021-02-10T04:51:13.619812+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:01.565998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:01.565998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"1001'1165","reported_seq":"4263","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.627402+0000","last_change":"2021-02-10T04:51:03.221365+0000","last_active":"2021-02-10T04:51:13.627402+0000","last_peered":"2021-02-10T04:51:13.627402+0000","last_clean":"2021-02-10T04:51:13.627402+0000","last_became_active":"2021-02-10T04:28:25.152030+0000","last_became_peered":"2021-02-10T04:28:25.152030+0000","last_unstale":"2021-02-10T04:51:13.627402+0000","last_undegraded":"2021-02-10T04:51:13.627402+0000","last_fullsized":"2021-02-10T04:51:13.627402+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":59203690,"num_objects":25,"num_object_clones":21,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":573,"num_read_kb":599059,"num_write":400,"num_write_kb":178261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"1113'5","reported_seq":"1292","reported_epoch":"1116","state":"active+clean","last_fresh":"2021-02-10T04:51:16.639040+0000","last_change":"2021-02-10T04:51:07.979631+0000","last_active":"2021-02-10T04:51:16.639040+0000","last_peered":"2021-02-10T04:51:16.639040+0000","last_clean":"2021-02-10T04:51:16.639040+0000","last_became_active":"2021-02-10T04:50:59.569578+0000","last_became_peered":"2021-02-10T04:50:59.569578+0000","last_unstale":"2021-02-10T04:51:16.639040+0000","last_undegraded":"2021-02-10T04:51:16.639040+0000","last_fullsized":"2021-02-10T04:51:16.639040+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'3","last_scrub_stamp":"2021-02-10T04:46:36.557037+0000","last_deep_scrub":"158'3","last_deep_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_clean_scrub_stamp":"2021-02-10T04:46:36.557037+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"1049'54","reported_seq":"1173","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.621470+0000","last_change":"2021-02-10T04:46:38.776905+0000","last_active":"2021-02-10T04:51:13.621470+0000","last_peered":"2021-02-10T04:51:13.621470+0000","last_clean":"2021-02-10T04:51:13.621470+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:51:13.621470+0000","last_undegraded":"2021-02-10T04:51:13.621470+0000","last_fullsized":"2021-02-10T04:51:13.621470+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:46:38.776809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:38.776809+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":93,"num_read_kb":73,"num_write":54,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"1152","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.625824+0000","last_change":"2021-02-10T04:50:59.591464+0000","last_active":"2021-02-10T04:51:13.625824+0000","last_peered":"2021-02-10T04:51:13.625824+0000","last_clean":"2021-02-10T04:51:13.625824+0000","last_became_active":"2021-02-10T04:50:59.591146+0000","last_became_peered":"2021-02-10T04:50:59.591146+0000","last_unstale":"2021-02-10T04:51:13.625824+0000","last_undegraded":"2021-02-10T04:51:13.625824+0000","last_fullsized":"2021-02-10T04:51:13.625824+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:04.957072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:04.957072+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"1030","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.626851+0000","last_change":"2021-02-10T04:47:16.010935+0000","last_active":"2021-02-10T04:51:13.626851+0000","last_peered":"2021-02-10T04:51:13.626851+0000","last_clean":"2021-02-10T04:51:13.626851+0000","last_became_active":"2021-02-10T04:45:47.533648+0000","last_became_peered":"2021-02-10T04:45:47.533648+0000","last_unstale":"2021-02-10T04:51:13.626851+0000","last_undegraded":"2021-02-10T04:51:13.626851+0000","last_fullsized":"2021-02-10T04:51:13.626851+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1045,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:16.010876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:16.010876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"998'1009","reported_seq":"4581","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.619360+0000","last_change":"2021-02-10T04:51:07.984082+0000","last_active":"2021-02-10T04:51:13.619360+0000","last_peered":"2021-02-10T04:51:13.619360+0000","last_clean":"2021-02-10T04:51:13.619360+0000","last_became_active":"2021-02-10T04:50:59.567798+0000","last_became_peered":"2021-02-10T04:50:59.567798+0000","last_unstale":"2021-02-10T04:51:13.619360+0000","last_undegraded":"2021-02-10T04:51:13.619360+0000","last_fullsized":"2021-02-10T04:51:13.619360+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24710413,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":239,"num_read_kb":249403,"num_write":154,"num_write_kb":50104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":8290058,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"982","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.620171+0000","last_change":"2021-02-10T04:51:07.976066+0000","last_active":"2021-02-10T04:51:13.620171+0000","last_peered":"2021-02-10T04:51:13.620171+0000","last_clean":"2021-02-10T04:51:13.620171+0000","last_became_active":"2021-02-10T04:50:59.570854+0000","last_became_peered":"2021-02-10T04:50:59.570854+0000","last_unstale":"2021-02-10T04:51:13.620171+0000","last_undegraded":"2021-02-10T04:51:13.620171+0000","last_fullsized":"2021-02-10T04:51:13.620171+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:46.592377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:46:46.592377+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"1096","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.621759+0000","last_change":"2021-02-10T04:50:59.527897+0000","last_active":"2021-02-10T04:51:13.621759+0000","last_peered":"2021-02-10T04:51:13.621759+0000","last_clean":"2021-02-10T04:51:13.621759+0000","last_became_active":"2021-02-10T04:50:59.527553+0000","last_became_peered":"2021-02-10T04:50:59.527553+0000","last_unstale":"2021-02-10T04:51:13.621759+0000","last_undegraded":"2021-02-10T04:51:13.621759+0000","last_fullsized":"2021-02-10T04:51:13.621759+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:35.785517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:46:35.785517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"1049'72","reported_seq":"1199","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.618734+0000","last_change":"2021-02-10T04:51:07.988809+0000","last_active":"2021-02-10T04:51:13.618734+0000","last_peered":"2021-02-10T04:51:13.618734+0000","last_clean":"2021-02-10T04:51:13.618734+0000","last_became_active":"2021-02-10T04:50:59.561344+0000","last_became_peered":"2021-02-10T04:50:59.561344+0000","last_unstale":"2021-02-10T04:51:13.618734+0000","last_undegraded":"2021-02-10T04:51:13.618734+0000","last_fullsized":"2021-02-10T04:51:13.618734+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'72","last_scrub_stamp":"2021-02-10T04:46:30.515026+0000","last_deep_scrub":"152'48","last_deep_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_clean_scrub_stamp":"2021-02-10T04:46:30.515026+0000","log_size":72,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":105,"num_read_kb":83,"num_write":60,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"778","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.626303+0000","last_change":"2021-02-10T04:46:22.069341+0000","last_active":"2021-02-10T04:51:13.626303+0000","last_peered":"2021-02-10T04:51:13.626303+0000","last_clean":"2021-02-10T04:51:13.626303+0000","last_became_active":"2021-02-10T04:33:05.141066+0000","last_became_peered":"2021-02-10T04:33:05.141066+0000","last_unstale":"2021-02-10T04:51:13.626303+0000","last_undegraded":"2021-02-10T04:51:13.626303+0000","last_fullsized":"2021-02-10T04:51:13.626303+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:46:22.069247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:22.069247+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"1112","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.620482+0000","last_change":"2021-02-10T04:50:59.571151+0000","last_active":"2021-02-10T04:51:13.620482+0000","last_peered":"2021-02-10T04:51:13.620482+0000","last_clean":"2021-02-10T04:51:13.620482+0000","last_became_active":"2021-02-10T04:50:59.570328+0000","last_became_peered":"2021-02-10T04:50:59.570328+0000","last_unstale":"2021-02-10T04:51:13.620482+0000","last_undegraded":"2021-02-10T04:51:13.620482+0000","last_fullsized":"2021-02-10T04:51:13.620482+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:27.926662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:44:27.926662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"1002'1657","reported_seq":"4996","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.619106+0000","last_change":"2021-02-10T04:51:07.996273+0000","last_active":"2021-02-10T04:51:13.619106+0000","last_peered":"2021-02-10T04:51:13.619106+0000","last_clean":"2021-02-10T04:51:13.619106+0000","last_became_active":"2021-02-10T04:50:59.574139+0000","last_became_peered":"2021-02-10T04:50:59.574139+0000","last_unstale":"2021-02-10T04:51:13.619106+0000","last_undegraded":"2021-02-10T04:51:13.619106+0000","last_fullsized":"2021-02-10T04:51:13.619106+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":42793066,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":424,"num_read_kb":437673,"num_write":318,"num_write_kb":118797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":17936938,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"1031","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.622027+0000","last_change":"2021-02-10T04:46:30.848110+0000","last_active":"2021-02-10T04:51:13.622027+0000","last_peered":"2021-02-10T04:51:13.622027+0000","last_clean":"2021-02-10T04:51:13.622027+0000","last_became_active":"2021-02-10T04:37:37.233158+0000","last_became_peered":"2021-02-10T04:37:37.233158+0000","last_unstale":"2021-02-10T04:51:13.622027+0000","last_undegraded":"2021-02-10T04:51:13.622027+0000","last_fullsized":"2021-02-10T04:51:13.622027+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:30.848045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:46:30.848045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"1158","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.626824+0000","last_change":"2021-02-10T04:50:58.142613+0000","last_active":"2021-02-10T04:51:13.626824+0000","last_peered":"2021-02-10T04:51:13.626824+0000","last_clean":"2021-02-10T04:51:13.626824+0000","last_became_active":"2021-02-10T04:25:51.025636+0000","last_became_peered":"2021-02-10T04:25:51.025636+0000","last_unstale":"2021-02-10T04:51:13.626824+0000","last_undegraded":"2021-02-10T04:51:13.626824+0000","last_fullsized":"2021-02-10T04:51:13.626824+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:58.142526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:58.142526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"1049'47","reported_seq":"1174","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.859438+0000","last_change":"2021-02-10T04:46:29.672214+0000","last_active":"2021-02-10T04:51:14.859438+0000","last_peered":"2021-02-10T04:51:14.859438+0000","last_clean":"2021-02-10T04:51:14.859438+0000","last_became_active":"2021-02-10T04:25:49.002937+0000","last_became_peered":"2021-02-10T04:25:49.002937+0000","last_unstale":"2021-02-10T04:51:14.859438+0000","last_undegraded":"2021-02-10T04:51:14.859438+0000","last_fullsized":"2021-02-10T04:51:14.859438+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'47","last_scrub_stamp":"2021-02-10T04:46:29.672141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:29.672141+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":87,"num_read_kb":68,"num_write":50,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"1023","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.859884+0000","last_change":"2021-02-10T04:50:46.434726+0000","last_active":"2021-02-10T04:51:14.859884+0000","last_peered":"2021-02-10T04:51:14.859884+0000","last_clean":"2021-02-10T04:51:14.859884+0000","last_became_active":"2021-02-10T04:29:07.161341+0000","last_became_peered":"2021-02-10T04:29:07.161341+0000","last_unstale":"2021-02-10T04:51:14.859884+0000","last_undegraded":"2021-02-10T04:51:14.859884+0000","last_fullsized":"2021-02-10T04:51:14.859884+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:50:46.434629+0000","last_deep_scrub":"53'2","last_deep_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_clean_scrub_stamp":"2021-02-10T04:50:46.434629+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"788","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.626088+0000","last_change":"2021-02-10T04:47:15.037885+0000","last_active":"2021-02-10T04:51:13.626088+0000","last_peered":"2021-02-10T04:51:13.626088+0000","last_clean":"2021-02-10T04:51:13.626088+0000","last_became_active":"2021-02-10T04:45:47.528080+0000","last_became_peered":"2021-02-10T04:45:47.528080+0000","last_unstale":"2021-02-10T04:51:13.626088+0000","last_undegraded":"2021-02-10T04:51:13.626088+0000","last_fullsized":"2021-02-10T04:51:13.626088+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1045,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:15.037663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:15.037663+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"1000'1071","reported_seq":"4651","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.621699+0000","last_change":"2021-02-10T04:50:59.597606+0000","last_active":"2021-02-10T04:51:13.621699+0000","last_peered":"2021-02-10T04:51:13.621699+0000","last_clean":"2021-02-10T04:51:13.621699+0000","last_became_active":"2021-02-10T04:50:59.596326+0000","last_became_peered":"2021-02-10T04:50:59.596326+0000","last_unstale":"2021-02-10T04:51:13.621699+0000","last_undegraded":"2021-02-10T04:51:13.621699+0000","last_fullsized":"2021-02-10T04:51:13.621699+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:47:14.611347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:14.611347+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46572736,"num_objects":18,"num_object_clones":15,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"1124","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.626621+0000","last_change":"2021-02-10T04:51:00.195591+0000","last_active":"2021-02-10T04:51:13.626621+0000","last_peered":"2021-02-10T04:51:13.626621+0000","last_clean":"2021-02-10T04:51:13.626621+0000","last_became_active":"2021-02-10T04:50:59.543222+0000","last_became_peered":"2021-02-10T04:50:59.543222+0000","last_unstale":"2021-02-10T04:51:13.626621+0000","last_undegraded":"2021-02-10T04:51:13.626621+0000","last_fullsized":"2021-02-10T04:51:13.626621+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_clean_scrub_stamp":"2021-02-10T04:51:00.195518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"1087","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.624906+0000","last_change":"2021-02-10T04:50:55.473455+0000","last_active":"2021-02-10T04:51:13.624906+0000","last_peered":"2021-02-10T04:51:13.624906+0000","last_clean":"2021-02-10T04:51:13.624906+0000","last_became_active":"2021-02-10T04:25:51.032201+0000","last_became_peered":"2021-02-10T04:25:51.032201+0000","last_unstale":"2021-02-10T04:51:13.624906+0000","last_undegraded":"2021-02-10T04:51:13.624906+0000","last_fullsized":"2021-02-10T04:51:13.624906+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:55.473347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:50:55.473347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"1049'40","reported_seq":"1182","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.859999+0000","last_change":"2021-02-10T04:46:18.777738+0000","last_active":"2021-02-10T04:51:14.859999+0000","last_peered":"2021-02-10T04:51:14.859999+0000","last_clean":"2021-02-10T04:51:14.859999+0000","last_became_active":"2021-02-10T04:37:37.247141+0000","last_became_peered":"2021-02-10T04:37:37.247141+0000","last_unstale":"2021-02-10T04:51:14.859999+0000","last_undegraded":"2021-02-10T04:51:14.859999+0000","last_fullsized":"2021-02-10T04:51:14.859999+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'40","last_scrub_stamp":"2021-02-10T04:46:18.777661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:18.777661+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":52,"num_write":40,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"826","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.859273+0000","last_change":"2021-02-10T04:50:50.399431+0000","last_active":"2021-02-10T04:51:14.859273+0000","last_peered":"2021-02-10T04:51:14.859273+0000","last_clean":"2021-02-10T04:51:14.859273+0000","last_became_active":"2021-02-10T04:33:05.166368+0000","last_became_peered":"2021-02-10T04:33:05.166368+0000","last_unstale":"2021-02-10T04:51:14.859273+0000","last_undegraded":"2021-02-10T04:51:14.859273+0000","last_fullsized":"2021-02-10T04:51:14.859273+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:50.399350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:50.399350+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"1104","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.859262+0000","last_change":"2021-02-10T04:50:59.577926+0000","last_active":"2021-02-10T04:51:14.859262+0000","last_peered":"2021-02-10T04:51:14.859262+0000","last_clean":"2021-02-10T04:51:14.859262+0000","last_became_active":"2021-02-10T04:50:59.576153+0000","last_became_peered":"2021-02-10T04:50:59.576153+0000","last_unstale":"2021-02-10T04:51:14.859262+0000","last_undegraded":"2021-02-10T04:51:14.859262+0000","last_fullsized":"2021-02-10T04:51:14.859262+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"789","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.623745+0000","last_change":"2021-02-10T04:50:59.595978+0000","last_active":"2021-02-10T04:51:13.623745+0000","last_peered":"2021-02-10T04:51:13.623745+0000","last_clean":"2021-02-10T04:51:13.623745+0000","last_became_active":"2021-02-10T04:50:59.595667+0000","last_became_peered":"2021-02-10T04:50:59.595667+0000","last_unstale":"2021-02-10T04:51:13.623745+0000","last_undegraded":"2021-02-10T04:51:13.623745+0000","last_fullsized":"2021-02-10T04:51:13.623745+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:17.598113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:17.598113+0000","last_clean_scrub_stamp":"2021-02-10T04:47:17.598113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.5","version":"1000'446","reported_seq":"2036","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.859750+0000","last_change":"2021-02-10T04:46:38.774890+0000","last_active":"2021-02-10T04:51:14.859750+0000","last_peered":"2021-02-10T04:51:14.859750+0000","last_clean":"2021-02-10T04:51:14.859750+0000","last_became_active":"2021-02-10T04:44:25.549930+0000","last_became_peered":"2021-02-10T04:44:25.549930+0000","last_unstale":"2021-02-10T04:51:14.859750+0000","last_undegraded":"2021-02-10T04:51:14.859750+0000","last_fullsized":"2021-02-10T04:51:14.859750+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:46:38.774795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:38.774795+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19615419,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"849","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.626278+0000","last_change":"2021-02-10T04:51:05.242796+0000","last_active":"2021-02-10T04:51:13.626278+0000","last_peered":"2021-02-10T04:51:13.626278+0000","last_clean":"2021-02-10T04:51:13.626278+0000","last_became_active":"2021-02-10T04:33:05.140966+0000","last_became_peered":"2021-02-10T04:33:05.140966+0000","last_unstale":"2021-02-10T04:51:13.626278+0000","last_undegraded":"2021-02-10T04:51:13.626278+0000","last_fullsized":"2021-02-10T04:51:13.626278+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:05.242690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:05.242690+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"502","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.624125+0000","last_change":"2021-02-10T04:51:10.398593+0000","last_active":"2021-02-10T04:51:13.624125+0000","last_peered":"2021-02-10T04:51:13.624125+0000","last_clean":"2021-02-10T04:51:13.624125+0000","last_became_active":"2021-02-10T04:37:37.234644+0000","last_became_peered":"2021-02-10T04:37:37.234644+0000","last_unstale":"2021-02-10T04:51:13.624125+0000","last_undegraded":"2021-02-10T04:51:13.624125+0000","last_fullsized":"2021-02-10T04:51:13.624125+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:10.398477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:10.398477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"1049'39","reported_seq":"1149","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.627015+0000","last_change":"2021-02-10T04:46:34.039385+0000","last_active":"2021-02-10T04:51:13.627015+0000","last_peered":"2021-02-10T04:51:13.627015+0000","last_clean":"2021-02-10T04:51:13.627015+0000","last_became_active":"2021-02-10T04:25:49.002471+0000","last_became_peered":"2021-02-10T04:25:49.002471+0000","last_unstale":"2021-02-10T04:51:13.627015+0000","last_undegraded":"2021-02-10T04:51:13.627015+0000","last_fullsized":"2021-02-10T04:51:13.627015+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'39","last_scrub_stamp":"2021-02-10T04:46:34.039273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:34.039273+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":51,"num_write":39,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"1134","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.859221+0000","last_change":"2021-02-10T04:50:05.424349+0000","last_active":"2021-02-10T04:51:14.859221+0000","last_peered":"2021-02-10T04:51:14.859221+0000","last_clean":"2021-02-10T04:51:14.859221+0000","last_became_active":"2021-02-10T04:29:07.158075+0000","last_became_peered":"2021-02-10T04:29:07.158075+0000","last_unstale":"2021-02-10T04:51:14.859221+0000","last_undegraded":"2021-02-10T04:51:14.859221+0000","last_fullsized":"2021-02-10T04:51:14.859221+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_clean_scrub_stamp":"2021-02-10T04:50:05.424262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"1272","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.623073+0000","last_change":"2021-02-10T04:50:59.575837+0000","last_active":"2021-02-10T04:51:13.623073+0000","last_peered":"2021-02-10T04:51:13.623073+0000","last_clean":"2021-02-10T04:51:13.623073+0000","last_became_active":"2021-02-10T04:50:59.575479+0000","last_became_peered":"2021-02-10T04:50:59.575479+0000","last_unstale":"2021-02-10T04:51:13.623073+0000","last_undegraded":"2021-02-10T04:51:13.623073+0000","last_fullsized":"2021-02-10T04:51:13.623073+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:29.707329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:29.707329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"1079","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.619436+0000","last_change":"2021-02-10T04:51:08.002847+0000","last_active":"2021-02-10T04:51:13.619436+0000","last_peered":"2021-02-10T04:51:13.619436+0000","last_clean":"2021-02-10T04:51:13.619436+0000","last_became_active":"2021-02-10T04:50:59.587343+0000","last_became_peered":"2021-02-10T04:50:59.587343+0000","last_unstale":"2021-02-10T04:51:13.619436+0000","last_undegraded":"2021-02-10T04:51:13.619436+0000","last_fullsized":"2021-02-10T04:51:13.619436+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:17.537072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:17.537072+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"1109","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.624991+0000","last_change":"2021-02-10T04:50:59.538442+0000","last_active":"2021-02-10T04:51:13.624991+0000","last_peered":"2021-02-10T04:51:13.624991+0000","last_clean":"2021-02-10T04:51:13.624991+0000","last_became_active":"2021-02-10T04:50:59.538009+0000","last_became_peered":"2021-02-10T04:50:59.538009+0000","last_unstale":"2021-02-10T04:51:13.624991+0000","last_undegraded":"2021-02-10T04:51:13.624991+0000","last_fullsized":"2021-02-10T04:51:13.624991+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:19.637875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:19.637875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"1000'490","reported_seq":"3651","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.860019+0000","last_change":"2021-02-10T04:47:02.876009+0000","last_active":"2021-02-10T04:51:14.860019+0000","last_peered":"2021-02-10T04:51:14.860019+0000","last_clean":"2021-02-10T04:51:14.860019+0000","last_became_active":"2021-02-10T04:45:42.422424+0000","last_became_peered":"2021-02-10T04:45:42.422424+0000","last_unstale":"2021-02-10T04:51:14.860019+0000","last_undegraded":"2021-02-10T04:51:14.860019+0000","last_fullsized":"2021-02-10T04:51:14.860019+0000","mapping_epoch":1039,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1040,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:47:02.875920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:02.875920+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22588464,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"435","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.626577+0000","last_change":"2021-02-10T04:50:46.380356+0000","last_active":"2021-02-10T04:51:13.626577+0000","last_peered":"2021-02-10T04:51:13.626577+0000","last_clean":"2021-02-10T04:51:13.626577+0000","last_became_active":"2021-02-10T04:38:39.252266+0000","last_became_peered":"2021-02-10T04:38:39.252266+0000","last_unstale":"2021-02-10T04:51:13.626577+0000","last_undegraded":"2021-02-10T04:51:13.626577+0000","last_fullsized":"2021-02-10T04:51:13.626577+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:46.380285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:50:46.380285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"1049'37","reported_seq":"859","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.624699+0000","last_change":"2021-02-10T04:46:36.919290+0000","last_active":"2021-02-10T04:51:13.624699+0000","last_peered":"2021-02-10T04:51:13.624699+0000","last_clean":"2021-02-10T04:51:13.624699+0000","last_became_active":"2021-02-10T04:33:05.146695+0000","last_became_peered":"2021-02-10T04:33:05.146695+0000","last_unstale":"2021-02-10T04:51:13.624699+0000","last_undegraded":"2021-02-10T04:51:13.624699+0000","last_fullsized":"2021-02-10T04:51:13.624699+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:46:36.919205+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:46:36.919205+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"517","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.618985+0000","last_change":"2021-02-10T04:51:07.973588+0000","last_active":"2021-02-10T04:51:13.618985+0000","last_peered":"2021-02-10T04:51:13.618985+0000","last_clean":"2021-02-10T04:51:13.618985+0000","last_became_active":"2021-02-10T04:50:59.565239+0000","last_became_peered":"2021-02-10T04:50:59.565239+0000","last_unstale":"2021-02-10T04:51:13.618985+0000","last_undegraded":"2021-02-10T04:51:13.618985+0000","last_fullsized":"2021-02-10T04:51:13.618985+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:04.591695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:04.591695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"1351","reported_epoch":"1116","state":"active+clean","last_fresh":"2021-02-10T04:51:22.419230+0000","last_change":"2021-02-10T04:46:53.011176+0000","last_active":"2021-02-10T04:51:22.419230+0000","last_peered":"2021-02-10T04:51:22.419230+0000","last_clean":"2021-02-10T04:51:22.419230+0000","last_became_active":"2021-02-10T04:37:37.249591+0000","last_became_peered":"2021-02-10T04:37:37.249591+0000","last_unstale":"2021-02-10T04:51:22.419230+0000","last_undegraded":"2021-02-10T04:51:22.419230+0000","last_fullsized":"2021-02-10T04:51:22.419230+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:46:53.011080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:53.011080+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":832,"num_read_kb":827,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"1036","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.627313+0000","last_change":"2021-02-10T04:50:59.529934+0000","last_active":"2021-02-10T04:51:13.627313+0000","last_peered":"2021-02-10T04:51:13.627313+0000","last_clean":"2021-02-10T04:51:13.627313+0000","last_became_active":"2021-02-10T04:50:59.529649+0000","last_became_peered":"2021-02-10T04:50:59.529649+0000","last_unstale":"2021-02-10T04:51:13.627313+0000","last_undegraded":"2021-02-10T04:51:13.627313+0000","last_fullsized":"2021-02-10T04:51:13.627313+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:13.991399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:47:13.991399+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.4","version":"1000'620","reported_seq":"4285","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.859549+0000","last_change":"2021-02-10T04:46:40.753673+0000","last_active":"2021-02-10T04:51:14.859549+0000","last_peered":"2021-02-10T04:51:14.859549+0000","last_clean":"2021-02-10T04:51:14.859549+0000","last_became_active":"2021-02-10T04:44:25.537632+0000","last_became_peered":"2021-02-10T04:44:25.537632+0000","last_unstale":"2021-02-10T04:51:14.859549+0000","last_undegraded":"2021-02-10T04:51:14.859549+0000","last_fullsized":"2021-02-10T04:51:14.859549+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:46:40.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:40.753581+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"1102","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.859236+0000","last_change":"2021-02-10T04:47:14.955640+0000","last_active":"2021-02-10T04:51:14.859236+0000","last_peered":"2021-02-10T04:51:14.859236+0000","last_clean":"2021-02-10T04:51:14.859236+0000","last_became_active":"2021-02-10T04:29:07.159607+0000","last_became_peered":"2021-02-10T04:29:07.159607+0000","last_unstale":"2021-02-10T04:51:14.859236+0000","last_undegraded":"2021-02-10T04:51:14.859236+0000","last_fullsized":"2021-02-10T04:51:14.859236+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:14.955544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:47:14.955544+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"1203","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.859898+0000","last_change":"2021-02-10T04:47:27.908308+0000","last_active":"2021-02-10T04:51:14.859898+0000","last_peered":"2021-02-10T04:51:14.859898+0000","last_clean":"2021-02-10T04:51:14.859898+0000","last_became_active":"2021-02-10T04:25:51.027624+0000","last_became_peered":"2021-02-10T04:25:51.027624+0000","last_unstale":"2021-02-10T04:51:14.859898+0000","last_undegraded":"2021-02-10T04:51:14.859898+0000","last_fullsized":"2021-02-10T04:51:14.859898+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:27.908221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:47:27.908221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"1049'55","reported_seq":"783","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.620268+0000","last_change":"2021-02-10T04:51:07.993003+0000","last_active":"2021-02-10T04:51:13.620268+0000","last_peered":"2021-02-10T04:51:13.620268+0000","last_clean":"2021-02-10T04:51:13.620268+0000","last_became_active":"2021-02-10T04:50:59.563323+0000","last_became_peered":"2021-02-10T04:50:59.563323+0000","last_unstale":"2021-02-10T04:51:13.620268+0000","last_undegraded":"2021-02-10T04:51:13.620268+0000","last_fullsized":"2021-02-10T04:51:13.620268+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'55","last_scrub_stamp":"2021-02-10T04:46:55.516761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:55.516761+0000","log_size":55,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":98,"num_read_kb":74,"num_write":58,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"1105","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.622343+0000","last_change":"2021-02-10T04:47:13.579252+0000","last_active":"2021-02-10T04:51:13.622343+0000","last_peered":"2021-02-10T04:51:13.622343+0000","last_clean":"2021-02-10T04:51:13.622343+0000","last_became_active":"2021-02-10T04:28:21.125541+0000","last_became_peered":"2021-02-10T04:28:21.125541+0000","last_unstale":"2021-02-10T04:51:13.622343+0000","last_undegraded":"2021-02-10T04:51:13.622343+0000","last_fullsized":"2021-02-10T04:51:13.622343+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:13.579163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:13.579163+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"1072","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.619592+0000","last_change":"2021-02-10T04:51:07.984424+0000","last_active":"2021-02-10T04:51:13.619592+0000","last_peered":"2021-02-10T04:51:13.619592+0000","last_clean":"2021-02-10T04:51:13.619592+0000","last_became_active":"2021-02-10T04:50:59.555689+0000","last_became_peered":"2021-02-10T04:50:59.555689+0000","last_unstale":"2021-02-10T04:51:13.619592+0000","last_undegraded":"2021-02-10T04:51:13.619592+0000","last_fullsized":"2021-02-10T04:51:13.619592+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.395943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.395943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"1002'2042","reported_seq":"9331","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.625741+0000","last_change":"2021-02-10T04:50:59.593597+0000","last_active":"2021-02-10T04:51:13.625741+0000","last_peered":"2021-02-10T04:51:13.625741+0000","last_clean":"2021-02-10T04:51:13.625741+0000","last_became_active":"2021-02-10T04:50:59.591769+0000","last_became_peered":"2021-02-10T04:50:59.591769+0000","last_unstale":"2021-02-10T04:51:13.625741+0000","last_undegraded":"2021-02-10T04:51:13.625741+0000","last_fullsized":"2021-02-10T04:51:13.625741+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:46:57.054230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:57.054230+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":85678898,"num_objects":35,"num_object_clones":29,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"1037","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.624885+0000","last_change":"2021-02-10T04:50:52.341596+0000","last_active":"2021-02-10T04:51:13.624885+0000","last_peered":"2021-02-10T04:51:13.624885+0000","last_clean":"2021-02-10T04:51:13.624885+0000","last_became_active":"2021-02-10T04:37:37.219422+0000","last_became_peered":"2021-02-10T04:37:37.219422+0000","last_unstale":"2021-02-10T04:51:13.624885+0000","last_undegraded":"2021-02-10T04:51:13.624885+0000","last_fullsized":"2021-02-10T04:51:13.624885+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:52.341527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:50:52.341527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"58'2","reported_seq":"1401","reported_epoch":"1116","state":"active+clean","last_fresh":"2021-02-10T04:51:17.990089+0000","last_change":"2021-02-10T04:50:54.466333+0000","last_active":"2021-02-10T04:51:17.990089+0000","last_peered":"2021-02-10T04:51:17.990089+0000","last_clean":"2021-02-10T04:51:17.990089+0000","last_became_active":"2021-02-10T04:25:51.026251+0000","last_became_peered":"2021-02-10T04:25:51.026251+0000","last_unstale":"2021-02-10T04:51:17.990089+0000","last_undegraded":"2021-02-10T04:51:17.990089+0000","last_fullsized":"2021-02-10T04:51:17.990089+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:54.466259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:54.466259+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"1049'54","reported_seq":"875","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.622625+0000","last_change":"2021-02-10T04:50:14.158728+0000","last_active":"2021-02-10T04:51:13.622625+0000","last_peered":"2021-02-10T04:51:13.622625+0000","last_clean":"2021-02-10T04:51:13.622625+0000","last_became_active":"2021-02-10T04:33:05.141901+0000","last_became_peered":"2021-02-10T04:33:05.141901+0000","last_unstale":"2021-02-10T04:51:13.622625+0000","last_undegraded":"2021-02-10T04:51:13.622625+0000","last_fullsized":"2021-02-10T04:51:13.622625+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:50:14.158659+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:50:14.158659+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":75,"num_read_kb":57,"num_write":42,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"852","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.859419+0000","last_change":"2021-02-10T04:50:49.448042+0000","last_active":"2021-02-10T04:51:14.859419+0000","last_peered":"2021-02-10T04:51:14.859419+0000","last_clean":"2021-02-10T04:51:14.859419+0000","last_became_active":"2021-02-10T04:37:37.247405+0000","last_became_peered":"2021-02-10T04:37:37.247405+0000","last_unstale":"2021-02-10T04:51:14.859419+0000","last_undegraded":"2021-02-10T04:51:14.859419+0000","last_fullsized":"2021-02-10T04:51:14.859419+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:49.447961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:49.447961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"1118","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.858999+0000","last_change":"2021-02-10T04:50:59.577707+0000","last_active":"2021-02-10T04:51:14.858999+0000","last_peered":"2021-02-10T04:51:14.858999+0000","last_clean":"2021-02-10T04:51:14.858999+0000","last_became_active":"2021-02-10T04:50:59.577351+0000","last_became_peered":"2021-02-10T04:50:59.577351+0000","last_unstale":"2021-02-10T04:51:14.858999+0000","last_undegraded":"2021-02-10T04:51:14.858999+0000","last_fullsized":"2021-02-10T04:51:14.858999+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:07.914061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:07.914061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"293'77","reported_seq":"2886","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.621193+0000","last_change":"2021-02-10T04:50:59.527961+0000","last_active":"2021-02-10T04:51:13.621193+0000","last_peered":"2021-02-10T04:51:13.621193+0000","last_clean":"2021-02-10T04:51:13.621193+0000","last_became_active":"2021-02-10T04:50:59.526860+0000","last_became_peered":"2021-02-10T04:50:59.526860+0000","last_unstale":"2021-02-10T04:51:13.621193+0000","last_undegraded":"2021-02-10T04:51:13.621193+0000","last_fullsized":"2021-02-10T04:51:13.621193+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3773171,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":36,"num_read_kb":37174,"num_write":18,"num_write_kb":6631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"1124","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.625704+0000","last_change":"2021-02-10T04:50:59.589648+0000","last_active":"2021-02-10T04:51:13.625704+0000","last_peered":"2021-02-10T04:51:13.625704+0000","last_clean":"2021-02-10T04:51:13.625704+0000","last_became_active":"2021-02-10T04:50:59.589417+0000","last_became_peered":"2021-02-10T04:50:59.589417+0000","last_unstale":"2021-02-10T04:51:13.625704+0000","last_undegraded":"2021-02-10T04:51:13.625704+0000","last_fullsized":"2021-02-10T04:51:13.625704+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:09.869777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:47:09.869777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"523","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.859368+0000","last_change":"2021-02-10T04:51:07.609624+0000","last_active":"2021-02-10T04:51:14.859368+0000","last_peered":"2021-02-10T04:51:14.859368+0000","last_clean":"2021-02-10T04:51:14.859368+0000","last_became_active":"2021-02-10T04:50:59.576664+0000","last_became_peered":"2021-02-10T04:50:59.576664+0000","last_unstale":"2021-02-10T04:51:14.859368+0000","last_undegraded":"2021-02-10T04:51:14.859368+0000","last_fullsized":"2021-02-10T04:51:14.859368+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.609500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.609500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"1048'39","reported_seq":"1206","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.621563+0000","last_change":"2021-02-10T04:50:44.624567+0000","last_active":"2021-02-10T04:51:13.621563+0000","last_peered":"2021-02-10T04:51:13.621563+0000","last_clean":"2021-02-10T04:51:13.621563+0000","last_became_active":"2021-02-10T04:39:02.560739+0000","last_became_peered":"2021-02-10T04:39:02.560739+0000","last_unstale":"2021-02-10T04:51:13.621563+0000","last_undegraded":"2021-02-10T04:51:13.621563+0000","last_fullsized":"2021-02-10T04:51:13.621563+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'39","last_scrub_stamp":"2021-02-10T04:50:44.624501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:50:44.624501+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":76,"num_read_kb":59,"num_write":41,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"1214","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.622782+0000","last_change":"2021-02-10T04:47:08.784658+0000","last_active":"2021-02-10T04:51:13.622782+0000","last_peered":"2021-02-10T04:51:13.622782+0000","last_clean":"2021-02-10T04:51:13.622782+0000","last_became_active":"2021-02-10T04:37:37.238701+0000","last_became_peered":"2021-02-10T04:37:37.238701+0000","last_unstale":"2021-02-10T04:51:13.622782+0000","last_undegraded":"2021-02-10T04:51:13.622782+0000","last_fullsized":"2021-02-10T04:51:13.622782+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:47:08.784593+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:47:08.784593+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"1074","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.619673+0000","last_change":"2021-02-10T04:51:07.995649+0000","last_active":"2021-02-10T04:51:13.619673+0000","last_peered":"2021-02-10T04:51:13.619673+0000","last_clean":"2021-02-10T04:51:13.619673+0000","last_became_active":"2021-02-10T04:50:59.561436+0000","last_became_peered":"2021-02-10T04:50:59.561436+0000","last_unstale":"2021-02-10T04:51:13.619673+0000","last_undegraded":"2021-02-10T04:51:13.619673+0000","last_fullsized":"2021-02-10T04:51:13.619673+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:15.536661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:15.536661+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"1001'1528","reported_seq":"5737","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.619312+0000","last_change":"2021-02-10T04:51:07.982619+0000","last_active":"2021-02-10T04:51:13.619312+0000","last_peered":"2021-02-10T04:51:13.619312+0000","last_clean":"2021-02-10T04:51:13.619312+0000","last_became_active":"2021-02-10T04:50:59.587009+0000","last_became_peered":"2021-02-10T04:50:59.587009+0000","last_unstale":"2021-02-10T04:51:13.619312+0000","last_undegraded":"2021-02-10T04:51:13.619312+0000","last_fullsized":"2021-02-10T04:51:13.619312+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":93888272,"num_objects":33,"num_object_clones":30,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":1,"num_read":716,"num_read_kb":703869,"num_write":572,"num_write_kb":222710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":42623785,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.15","version":"1000'446","reported_seq":"1987","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.622446+0000","last_change":"2021-02-10T04:50:05.815243+0000","last_active":"2021-02-10T04:51:13.622446+0000","last_peered":"2021-02-10T04:51:13.622446+0000","last_clean":"2021-02-10T04:51:13.622446+0000","last_became_active":"2021-02-10T04:45:43.499003+0000","last_became_peered":"2021-02-10T04:45:43.499003+0000","last_unstale":"2021-02-10T04:51:13.622446+0000","last_undegraded":"2021-02-10T04:51:13.622446+0000","last_fullsized":"2021-02-10T04:51:13.622446+0000","mapping_epoch":1040,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1041,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:50:05.815135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:50:05.815135+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24896285,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"1048'40","reported_seq":"1182","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.860092+0000","last_change":"2021-02-10T04:50:59.578612+0000","last_active":"2021-02-10T04:51:14.860092+0000","last_peered":"2021-02-10T04:51:14.860092+0000","last_clean":"2021-02-10T04:51:14.860092+0000","last_became_active":"2021-02-10T04:50:59.578193+0000","last_became_peered":"2021-02-10T04:50:59.578193+0000","last_unstale":"2021-02-10T04:51:14.860092+0000","last_undegraded":"2021-02-10T04:51:14.860092+0000","last_fullsized":"2021-02-10T04:51:14.860092+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'40","last_scrub_stamp":"2021-02-10T04:47:29.965140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:47:29.965140+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":80,"num_read_kb":58,"num_write":41,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"517","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.859092+0000","last_change":"2021-02-10T04:51:08.639876+0000","last_active":"2021-02-10T04:51:14.859092+0000","last_peered":"2021-02-10T04:51:14.859092+0000","last_clean":"2021-02-10T04:51:14.859092+0000","last_became_active":"2021-02-10T04:37:37.234649+0000","last_became_peered":"2021-02-10T04:37:37.234649+0000","last_unstale":"2021-02-10T04:51:14.859092+0000","last_undegraded":"2021-02-10T04:51:14.859092+0000","last_fullsized":"2021-02-10T04:51:14.859092+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:51:08.639784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.639784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":247,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"1048","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.620543+0000","last_change":"2021-02-10T04:51:09.475150+0000","last_active":"2021-02-10T04:51:13.620543+0000","last_peered":"2021-02-10T04:51:13.620543+0000","last_clean":"2021-02-10T04:51:13.620543+0000","last_became_active":"2021-02-10T04:50:59.585917+0000","last_became_peered":"2021-02-10T04:50:59.585917+0000","last_unstale":"2021-02-10T04:51:13.620543+0000","last_undegraded":"2021-02-10T04:51:13.620543+0000","last_fullsized":"2021-02-10T04:51:13.620543+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:09.475063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:09.475063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"493","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.622014+0000","last_change":"2021-02-10T04:46:27.889189+0000","last_active":"2021-02-10T04:51:13.622014+0000","last_peered":"2021-02-10T04:51:13.622014+0000","last_clean":"2021-02-10T04:51:13.622014+0000","last_became_active":"2021-02-10T04:37:37.238049+0000","last_became_peered":"2021-02-10T04:37:37.238049+0000","last_unstale":"2021-02-10T04:51:13.622014+0000","last_undegraded":"2021-02-10T04:51:13.622014+0000","last_fullsized":"2021-02-10T04:51:13.622014+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:27.889132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:27.889132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"1000'620","reported_seq":"4261","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.619899+0000","last_change":"2021-02-10T04:51:07.977588+0000","last_active":"2021-02-10T04:51:13.619899+0000","last_peered":"2021-02-10T04:51:13.619899+0000","last_clean":"2021-02-10T04:51:13.619899+0000","last_became_active":"2021-02-10T04:50:59.591216+0000","last_became_peered":"2021-02-10T04:50:59.591216+0000","last_unstale":"2021-02-10T04:51:13.619899+0000","last_undegraded":"2021-02-10T04:51:13.619899+0000","last_fullsized":"2021-02-10T04:51:13.619899+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:47:24.897680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:24.897680+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14265195,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":144,"num_read_kb":128483,"num_write":129,"num_write_kb":53444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":19999269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"1063'30","reported_seq":"1143","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.620097+0000","last_change":"2021-02-10T04:51:07.995751+0000","last_active":"2021-02-10T04:51:13.620097+0000","last_peered":"2021-02-10T04:51:13.620097+0000","last_clean":"2021-02-10T04:51:13.620097+0000","last_became_active":"2021-02-10T04:50:59.587330+0000","last_became_peered":"2021-02-10T04:50:59.587330+0000","last_unstale":"2021-02-10T04:51:13.620097+0000","last_undegraded":"2021-02-10T04:51:13.620097+0000","last_fullsized":"2021-02-10T04:51:13.620097+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1063'30","last_scrub_stamp":"2021-02-10T04:47:05.589582+0000","last_deep_scrub":"1063'30","last_deep_scrub_stamp":"2021-02-10T04:47:05.589582+0000","last_clean_scrub_stamp":"2021-02-10T04:47:05.589582+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"1098","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.626685+0000","last_change":"2021-02-10T04:46:55.046228+0000","last_active":"2021-02-10T04:51:13.626685+0000","last_peered":"2021-02-10T04:51:13.626685+0000","last_clean":"2021-02-10T04:51:13.626685+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:51:13.626685+0000","last_undegraded":"2021-02-10T04:51:13.626685+0000","last_fullsized":"2021-02-10T04:51:13.626685+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:46:55.046158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:55.046158+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"1102","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.624150+0000","last_change":"2021-02-10T04:51:08.451362+0000","last_active":"2021-02-10T04:51:13.624150+0000","last_peered":"2021-02-10T04:51:13.624150+0000","last_clean":"2021-02-10T04:51:13.624150+0000","last_became_active":"2021-02-10T04:50:59.540375+0000","last_became_peered":"2021-02-10T04:50:59.540375+0000","last_unstale":"2021-02-10T04:51:13.624150+0000","last_undegraded":"2021-02-10T04:51:13.624150+0000","last_fullsized":"2021-02-10T04:51:13.624150+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:08.451289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.451289+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"1012","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.627524+0000","last_change":"2021-02-10T04:50:45.362633+0000","last_active":"2021-02-10T04:51:13.627524+0000","last_peered":"2021-02-10T04:51:13.627524+0000","last_clean":"2021-02-10T04:51:13.627524+0000","last_became_active":"2021-02-10T04:29:07.147249+0000","last_became_peered":"2021-02-10T04:29:07.147249+0000","last_unstale":"2021-02-10T04:51:13.627524+0000","last_undegraded":"2021-02-10T04:51:13.627524+0000","last_fullsized":"2021-02-10T04:51:13.627524+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:45.362537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:45.362537+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.17","version":"1002'2042","reported_seq":"9275","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.624465+0000","last_change":"2021-02-10T04:47:16.641750+0000","last_active":"2021-02-10T04:51:13.624465+0000","last_peered":"2021-02-10T04:51:13.624465+0000","last_clean":"2021-02-10T04:51:13.624465+0000","last_became_active":"2021-02-10T04:44:30.657433+0000","last_became_peered":"2021-02-10T04:44:30.657433+0000","last_unstale":"2021-02-10T04:51:13.624465+0000","last_undegraded":"2021-02-10T04:51:13.624465+0000","last_fullsized":"2021-02-10T04:51:13.624465+0000","mapping_epoch":1012,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1013,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:47:16.641616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:16.641616+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6921254,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"1049'46","reported_seq":"1267","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.620119+0000","last_change":"2021-02-10T04:51:07.988787+0000","last_active":"2021-02-10T04:51:13.620119+0000","last_peered":"2021-02-10T04:51:13.620119+0000","last_clean":"2021-02-10T04:51:13.620119+0000","last_became_active":"2021-02-10T04:50:59.569915+0000","last_became_peered":"2021-02-10T04:50:59.569915+0000","last_unstale":"2021-02-10T04:51:13.620119+0000","last_undegraded":"2021-02-10T04:51:13.620119+0000","last_fullsized":"2021-02-10T04:51:13.620119+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'46","last_scrub_stamp":"2021-02-10T04:46:56.512292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:56.512292+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":65,"num_write":46,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"84'7","reported_seq":"1126","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.622269+0000","last_change":"2021-02-10T04:50:59.576334+0000","last_active":"2021-02-10T04:51:13.622269+0000","last_peered":"2021-02-10T04:51:13.622269+0000","last_clean":"2021-02-10T04:51:13.622269+0000","last_became_active":"2021-02-10T04:50:59.575924+0000","last_became_peered":"2021-02-10T04:50:59.575924+0000","last_unstale":"2021-02-10T04:51:13.622269+0000","last_undegraded":"2021-02-10T04:51:13.622269+0000","last_fullsized":"2021-02-10T04:51:13.622269+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'7","last_scrub_stamp":"2021-02-10T04:46:49.869504+0000","last_deep_scrub":"84'7","last_deep_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_clean_scrub_stamp":"2021-02-10T04:46:49.869504+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"58'2","reported_seq":"1396","reported_epoch":"1116","state":"active+clean","last_fresh":"2021-02-10T04:51:17.990141+0000","last_change":"2021-02-10T04:50:47.407626+0000","last_active":"2021-02-10T04:51:17.990141+0000","last_peered":"2021-02-10T04:51:17.990141+0000","last_clean":"2021-02-10T04:51:17.990141+0000","last_became_active":"2021-02-10T04:25:51.032328+0000","last_became_peered":"2021-02-10T04:25:51.032328+0000","last_unstale":"2021-02-10T04:51:17.990141+0000","last_undegraded":"2021-02-10T04:51:17.990141+0000","last_fullsized":"2021-02-10T04:51:17.990141+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:47.407557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:47.407557+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"1100","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.859933+0000","last_change":"2021-02-10T04:50:59.576884+0000","last_active":"2021-02-10T04:51:14.859933+0000","last_peered":"2021-02-10T04:51:14.859933+0000","last_clean":"2021-02-10T04:51:14.859933+0000","last_became_active":"2021-02-10T04:50:59.576536+0000","last_became_peered":"2021-02-10T04:50:59.576536+0000","last_unstale":"2021-02-10T04:51:14.859933+0000","last_undegraded":"2021-02-10T04:51:14.859933+0000","last_fullsized":"2021-02-10T04:51:14.859933+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:33.677438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_clean_scrub_stamp":"2021-02-10T04:46:33.677438+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"1000'490","reported_seq":"3617","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.622494+0000","last_change":"2021-02-10T04:50:03.756349+0000","last_active":"2021-02-10T04:51:13.622494+0000","last_peered":"2021-02-10T04:51:13.622494+0000","last_clean":"2021-02-10T04:51:13.622494+0000","last_became_active":"2021-02-10T04:44:30.702606+0000","last_became_peered":"2021-02-10T04:44:30.702606+0000","last_unstale":"2021-02-10T04:51:13.622494+0000","last_undegraded":"2021-02-10T04:51:13.622494+0000","last_fullsized":"2021-02-10T04:51:13.622494+0000","mapping_epoch":1012,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1013,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:50:03.756248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:50:03.756248+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16154790,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"1112'40","reported_seq":"1170","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.623844+0000","last_change":"2021-02-10T04:51:01.450124+0000","last_active":"2021-02-10T04:51:13.623844+0000","last_peered":"2021-02-10T04:51:13.623844+0000","last_clean":"2021-02-10T04:51:13.623844+0000","last_became_active":"2021-02-10T04:50:59.539529+0000","last_became_peered":"2021-02-10T04:50:59.539529+0000","last_unstale":"2021-02-10T04:51:13.623844+0000","last_undegraded":"2021-02-10T04:51:13.623844+0000","last_fullsized":"2021-02-10T04:51:13.623844+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1112'40","last_scrub_stamp":"2021-02-10T04:51:01.450050+0000","last_deep_scrub":"1112'40","last_deep_scrub_stamp":"2021-02-10T04:51:01.450050+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.450050+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":59,"num_read_kb":44,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"1116","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.859518+0000","last_change":"2021-02-10T04:50:59.579777+0000","last_active":"2021-02-10T04:51:14.859518+0000","last_peered":"2021-02-10T04:51:14.859518+0000","last_clean":"2021-02-10T04:51:14.859518+0000","last_became_active":"2021-02-10T04:50:59.579485+0000","last_became_peered":"2021-02-10T04:50:59.579485+0000","last_unstale":"2021-02-10T04:51:14.859518+0000","last_undegraded":"2021-02-10T04:51:14.859518+0000","last_fullsized":"2021-02-10T04:51:14.859518+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:23.904210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:47:23.904210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"1113'5","reported_seq":"1307","reported_epoch":"1116","state":"active+clean","last_fresh":"2021-02-10T04:51:16.640455+0000","last_change":"2021-02-10T04:51:07.996294+0000","last_active":"2021-02-10T04:51:16.640455+0000","last_peered":"2021-02-10T04:51:16.640455+0000","last_clean":"2021-02-10T04:51:16.640455+0000","last_became_active":"2021-02-10T04:50:59.581263+0000","last_became_peered":"2021-02-10T04:50:59.581263+0000","last_unstale":"2021-02-10T04:51:16.640455+0000","last_undegraded":"2021-02-10T04:51:16.640455+0000","last_fullsized":"2021-02-10T04:51:16.640455+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"639'3","last_scrub_stamp":"2021-02-10T04:46:39.502896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:46:39.502896+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"1093","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.623131+0000","last_change":"2021-02-10T04:51:11.779876+0000","last_active":"2021-02-10T04:51:13.623131+0000","last_peered":"2021-02-10T04:51:13.623131+0000","last_clean":"2021-02-10T04:51:13.623131+0000","last_became_active":"2021-02-10T04:37:37.238360+0000","last_became_peered":"2021-02-10T04:37:37.238360+0000","last_unstale":"2021-02-10T04:51:13.623131+0000","last_undegraded":"2021-02-10T04:51:13.623131+0000","last_fullsized":"2021-02-10T04:51:13.623131+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:11.779779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:11.779779+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"1032'89","reported_seq":"881","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.626783+0000","last_change":"2021-02-10T04:50:59.528845+0000","last_active":"2021-02-10T04:51:13.626783+0000","last_peered":"2021-02-10T04:51:13.626783+0000","last_clean":"2021-02-10T04:51:13.626783+0000","last_became_active":"2021-02-10T04:50:59.528302+0000","last_became_peered":"2021-02-10T04:50:59.528302+0000","last_unstale":"2021-02-10T04:51:13.626783+0000","last_undegraded":"2021-02-10T04:51:13.626783+0000","last_fullsized":"2021-02-10T04:51:13.626783+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":5,"last_scrub":"1032'89","last_scrub_stamp":"2021-02-10T04:47:00.950114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:00.950114+0000","log_size":89,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5949711,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"1048'27","reported_seq":"860","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.626954+0000","last_change":"2021-02-10T04:50:55.142571+0000","last_active":"2021-02-10T04:51:13.626954+0000","last_peered":"2021-02-10T04:51:13.626954+0000","last_clean":"2021-02-10T04:51:13.626954+0000","last_became_active":"2021-02-10T04:37:37.249007+0000","last_became_peered":"2021-02-10T04:37:37.249007+0000","last_unstale":"2021-02-10T04:51:13.626954+0000","last_undegraded":"2021-02-10T04:51:13.626954+0000","last_fullsized":"2021-02-10T04:51:13.626954+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'27","last_scrub_stamp":"2021-02-10T04:50:55.142479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:50:55.142479+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":34,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"1111","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.627626+0000","last_change":"2021-02-10T04:50:59.543884+0000","last_active":"2021-02-10T04:51:13.627626+0000","last_peered":"2021-02-10T04:51:13.627626+0000","last_clean":"2021-02-10T04:51:13.627626+0000","last_became_active":"2021-02-10T04:50:59.543531+0000","last_became_peered":"2021-02-10T04:50:59.543531+0000","last_unstale":"2021-02-10T04:51:13.627626+0000","last_undegraded":"2021-02-10T04:51:13.627626+0000","last_fullsized":"2021-02-10T04:51:13.627626+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:42.217450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:42.217450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"1045","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.619208+0000","last_change":"2021-02-10T04:51:07.973746+0000","last_active":"2021-02-10T04:51:13.619208+0000","last_peered":"2021-02-10T04:51:13.619208+0000","last_clean":"2021-02-10T04:51:13.619208+0000","last_became_active":"2021-02-10T04:50:59.567699+0000","last_became_peered":"2021-02-10T04:50:59.567699+0000","last_unstale":"2021-02-10T04:51:13.619208+0000","last_undegraded":"2021-02-10T04:51:13.619208+0000","last_fullsized":"2021-02-10T04:51:13.619208+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:45.524973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:44:45.524973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"1090","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.858913+0000","last_change":"2021-02-10T04:50:57.568614+0000","last_active":"2021-02-10T04:51:14.858913+0000","last_peered":"2021-02-10T04:51:14.858913+0000","last_clean":"2021-02-10T04:51:14.858913+0000","last_became_active":"2021-02-10T04:25:53.039322+0000","last_became_peered":"2021-02-10T04:25:53.039322+0000","last_unstale":"2021-02-10T04:51:14.858913+0000","last_undegraded":"2021-02-10T04:51:14.858913+0000","last_fullsized":"2021-02-10T04:51:14.858913+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:57.568541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:57.568541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"1000'1071","reported_seq":"4600","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.618423+0000","last_change":"2021-02-10T04:51:07.975248+0000","last_active":"2021-02-10T04:51:13.618423+0000","last_peered":"2021-02-10T04:51:13.618423+0000","last_clean":"2021-02-10T04:51:13.618423+0000","last_became_active":"2021-02-10T04:50:59.570053+0000","last_became_peered":"2021-02-10T04:50:59.570053+0000","last_unstale":"2021-02-10T04:51:13.618423+0000","last_undegraded":"2021-02-10T04:51:13.618423+0000","last_fullsized":"2021-02-10T04:51:13.618423+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":5,"last_scrub":"412'357","last_scrub_stamp":"2021-02-10T04:33:50.582002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:50.582002+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23286368,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":242,"num_read_kb":226835,"num_write":185,"num_write_kb":76005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"1049'31","reported_seq":"1134","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.620852+0000","last_change":"2021-02-10T04:47:15.739807+0000","last_active":"2021-02-10T04:51:13.620852+0000","last_peered":"2021-02-10T04:51:13.620852+0000","last_clean":"2021-02-10T04:51:13.620852+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:51:13.620852+0000","last_undegraded":"2021-02-10T04:51:13.620852+0000","last_fullsized":"2021-02-10T04:51:13.620852+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'31","last_scrub_stamp":"2021-02-10T04:47:15.739700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:47:15.739700+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":41,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"781","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.624631+0000","last_change":"2021-02-10T04:47:08.700162+0000","last_active":"2021-02-10T04:51:13.624631+0000","last_peered":"2021-02-10T04:51:13.624631+0000","last_clean":"2021-02-10T04:51:13.624631+0000","last_became_active":"2021-02-10T04:33:05.147956+0000","last_became_peered":"2021-02-10T04:33:05.147956+0000","last_unstale":"2021-02-10T04:51:13.624631+0000","last_undegraded":"2021-02-10T04:51:13.624631+0000","last_fullsized":"2021-02-10T04:51:13.624631+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:47:08.700069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:08.700069+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"785","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.622540+0000","last_change":"2021-02-10T04:51:05.475233+0000","last_active":"2021-02-10T04:51:13.622540+0000","last_peered":"2021-02-10T04:51:13.622540+0000","last_clean":"2021-02-10T04:51:13.622540+0000","last_became_active":"2021-02-10T04:50:59.575796+0000","last_became_peered":"2021-02-10T04:50:59.575796+0000","last_unstale":"2021-02-10T04:51:13.622540+0000","last_undegraded":"2021-02-10T04:51:13.622540+0000","last_fullsized":"2021-02-10T04:51:13.622540+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:05.475145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:51:05.475145+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"1074","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.618848+0000","last_change":"2021-02-10T04:51:07.979066+0000","last_active":"2021-02-10T04:51:13.618848+0000","last_peered":"2021-02-10T04:51:13.618848+0000","last_clean":"2021-02-10T04:51:13.618848+0000","last_became_active":"2021-02-10T04:50:59.557657+0000","last_became_peered":"2021-02-10T04:50:59.557657+0000","last_unstale":"2021-02-10T04:51:13.618848+0000","last_undegraded":"2021-02-10T04:51:13.618848+0000","last_fullsized":"2021-02-10T04:51:13.618848+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:09.627905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:09.627905+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"1002'1657","reported_seq":"5034","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.859251+0000","last_change":"2021-02-10T04:46:54.881511+0000","last_active":"2021-02-10T04:51:14.859251+0000","last_peered":"2021-02-10T04:51:14.859251+0000","last_clean":"2021-02-10T04:51:14.859251+0000","last_became_active":"2021-02-10T04:44:35.600743+0000","last_became_peered":"2021-02-10T04:44:35.600743+0000","last_unstale":"2021-02-10T04:51:14.859251+0000","last_undegraded":"2021-02-10T04:51:14.859251+0000","last_fullsized":"2021-02-10T04:51:14.859251+0000","mapping_epoch":1015,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1016,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:46:54.881412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:54.881412+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":47335497,"num_objects":20,"num_object_clones":18,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"1048'20","reported_seq":"1213","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.618677+0000","last_change":"2021-02-10T04:51:08.002869+0000","last_active":"2021-02-10T04:51:13.618677+0000","last_peered":"2021-02-10T04:51:13.618677+0000","last_clean":"2021-02-10T04:51:13.618677+0000","last_became_active":"2021-02-10T04:50:59.544484+0000","last_became_peered":"2021-02-10T04:50:59.544484+0000","last_unstale":"2021-02-10T04:51:13.618677+0000","last_undegraded":"2021-02-10T04:51:13.618677+0000","last_fullsized":"2021-02-10T04:51:13.618677+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'20","last_scrub_stamp":"2021-02-10T04:47:12.728019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:47:12.728019+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":20,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"1105","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.624358+0000","last_change":"2021-02-10T04:48:54.943305+0000","last_active":"2021-02-10T04:51:13.624358+0000","last_peered":"2021-02-10T04:51:13.624358+0000","last_clean":"2021-02-10T04:51:13.624358+0000","last_became_active":"2021-02-10T04:37:37.229852+0000","last_became_peered":"2021-02-10T04:37:37.229852+0000","last_unstale":"2021-02-10T04:51:13.624358+0000","last_undegraded":"2021-02-10T04:51:13.624358+0000","last_fullsized":"2021-02-10T04:51:13.624358+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:48:54.943234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:48:54.943234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"1087","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.626291+0000","last_change":"2021-02-10T04:50:56.099710+0000","last_active":"2021-02-10T04:51:13.626291+0000","last_peered":"2021-02-10T04:51:13.626291+0000","last_clean":"2021-02-10T04:51:13.626291+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:51:13.626291+0000","last_undegraded":"2021-02-10T04:51:13.626291+0000","last_fullsized":"2021-02-10T04:51:13.626291+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:56.099651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:50:56.099651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"511","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.620123+0000","last_change":"2021-02-10T04:50:59.570603+0000","last_active":"2021-02-10T04:51:13.620123+0000","last_peered":"2021-02-10T04:51:13.620123+0000","last_clean":"2021-02-10T04:51:13.620123+0000","last_became_active":"2021-02-10T04:50:59.569846+0000","last_became_peered":"2021-02-10T04:50:59.569846+0000","last_unstale":"2021-02-10T04:51:13.620123+0000","last_undegraded":"2021-02-10T04:51:13.620123+0000","last_fullsized":"2021-02-10T04:51:13.620123+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:37.748261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:46:37.748261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.12","version":"998'1009","reported_seq":"4773","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.626280+0000","last_change":"2021-02-10T04:47:13.032602+0000","last_active":"2021-02-10T04:51:13.626280+0000","last_peered":"2021-02-10T04:51:13.626280+0000","last_clean":"2021-02-10T04:51:13.626280+0000","last_became_active":"2021-02-10T04:45:45.516982+0000","last_became_peered":"2021-02-10T04:45:45.516982+0000","last_unstale":"2021-02-10T04:51:13.626280+0000","last_undegraded":"2021-02-10T04:51:13.626280+0000","last_fullsized":"2021-02-10T04:51:13.626280+0000","mapping_epoch":1042,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1043,"parent":"0.0","parent_split_bits":5,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:13.032506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:13.032506+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49420827,"num_objects":21,"num_object_clones":19,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"1049'37","reported_seq":"1141","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.619214+0000","last_change":"2021-02-10T04:51:08.006429+0000","last_active":"2021-02-10T04:51:13.619214+0000","last_peered":"2021-02-10T04:51:13.619214+0000","last_clean":"2021-02-10T04:51:13.619214+0000","last_became_active":"2021-02-10T04:50:59.549412+0000","last_became_peered":"2021-02-10T04:50:59.549412+0000","last_unstale":"2021-02-10T04:51:13.619214+0000","last_undegraded":"2021-02-10T04:51:13.619214+0000","last_fullsized":"2021-02-10T04:51:13.619214+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'29","last_scrub_stamp":"2021-02-10T04:44:44.573821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:44:44.573821+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":64,"num_read_kb":50,"num_write":37,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"1236","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.627213+0000","last_change":"2021-02-10T04:46:44.241805+0000","last_active":"2021-02-10T04:51:13.627213+0000","last_peered":"2021-02-10T04:51:13.627213+0000","last_clean":"2021-02-10T04:51:13.627213+0000","last_became_active":"2021-02-10T04:37:37.228840+0000","last_became_peered":"2021-02-10T04:37:37.228840+0000","last_unstale":"2021-02-10T04:51:13.627213+0000","last_undegraded":"2021-02-10T04:51:13.627213+0000","last_fullsized":"2021-02-10T04:51:13.627213+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:46:44.241683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:44.241683+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"397","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.618746+0000","last_change":"2021-02-10T04:51:07.988411+0000","last_active":"2021-02-10T04:51:13.618746+0000","last_peered":"2021-02-10T04:51:13.618746+0000","last_clean":"2021-02-10T04:51:13.618746+0000","last_became_active":"2021-02-10T04:50:59.564977+0000","last_became_peered":"2021-02-10T04:50:59.564977+0000","last_unstale":"2021-02-10T04:51:13.618746+0000","last_undegraded":"2021-02-10T04:51:13.618746+0000","last_fullsized":"2021-02-10T04:51:13.618746+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:54.669597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:44:54.669597+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"1043","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.619134+0000","last_change":"2021-02-10T04:51:07.998582+0000","last_active":"2021-02-10T04:51:13.619134+0000","last_peered":"2021-02-10T04:51:13.619134+0000","last_clean":"2021-02-10T04:51:13.619134+0000","last_became_active":"2021-02-10T04:50:59.563229+0000","last_became_peered":"2021-02-10T04:50:59.563229+0000","last_unstale":"2021-02-10T04:51:13.619134+0000","last_undegraded":"2021-02-10T04:51:13.619134+0000","last_fullsized":"2021-02-10T04:51:13.619134+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:58.514318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_clean_scrub_stamp":"2021-02-10T04:46:58.514318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"1049'42","reported_seq":"1176","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.860244+0000","last_change":"2021-02-10T04:50:59.578533+0000","last_active":"2021-02-10T04:51:14.860244+0000","last_peered":"2021-02-10T04:51:14.860244+0000","last_clean":"2021-02-10T04:51:14.860244+0000","last_became_active":"2021-02-10T04:50:59.577949+0000","last_became_peered":"2021-02-10T04:50:59.577949+0000","last_unstale":"2021-02-10T04:51:14.860244+0000","last_undegraded":"2021-02-10T04:51:14.860244+0000","last_fullsized":"2021-02-10T04:51:14.860244+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'42","last_scrub_stamp":"2021-02-10T04:46:13.849755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:13.849755+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":75,"num_read_kb":57,"num_write":42,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"1155","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.621407+0000","last_change":"2021-02-10T04:50:04.287211+0000","last_active":"2021-02-10T04:51:13.621407+0000","last_peered":"2021-02-10T04:51:13.621407+0000","last_clean":"2021-02-10T04:51:13.621407+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:51:13.621407+0000","last_undegraded":"2021-02-10T04:51:13.621407+0000","last_fullsized":"2021-02-10T04:51:13.621407+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:04.287114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:04.287114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"1098'3","reported_seq":"1397","reported_epoch":"1116","state":"active+clean","last_fresh":"2021-02-10T04:51:17.990176+0000","last_change":"2021-02-10T04:50:15.255211+0000","last_active":"2021-02-10T04:51:17.990176+0000","last_peered":"2021-02-10T04:51:17.990176+0000","last_clean":"2021-02-10T04:51:17.990176+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:51:17.990176+0000","last_undegraded":"2021-02-10T04:51:17.990176+0000","last_fullsized":"2021-02-10T04:51:17.990176+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"1098'3","last_scrub_stamp":"2021-02-10T04:50:15.255137+0000","last_deep_scrub":"1098'3","last_deep_scrub_stamp":"2021-02-10T04:50:15.255137+0000","last_clean_scrub_stamp":"2021-02-10T04:50:15.255137+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"1025","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.623570+0000","last_change":"2021-02-10T04:46:34.914405+0000","last_active":"2021-02-10T04:51:13.623570+0000","last_peered":"2021-02-10T04:51:13.623570+0000","last_clean":"2021-02-10T04:51:13.623570+0000","last_became_active":"2021-02-10T04:38:39.251336+0000","last_became_peered":"2021-02-10T04:38:39.251336+0000","last_unstale":"2021-02-10T04:51:13.623570+0000","last_undegraded":"2021-02-10T04:51:13.623570+0000","last_fullsized":"2021-02-10T04:51:13.623570+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:34.914288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:34.914288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"1049'51","reported_seq":"1176","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.627275+0000","last_change":"2021-02-10T04:50:53.153340+0000","last_active":"2021-02-10T04:51:13.627275+0000","last_peered":"2021-02-10T04:51:13.627275+0000","last_clean":"2021-02-10T04:51:13.627275+0000","last_became_active":"2021-02-10T04:25:49.015760+0000","last_became_peered":"2021-02-10T04:25:49.015760+0000","last_unstale":"2021-02-10T04:51:13.627275+0000","last_undegraded":"2021-02-10T04:51:13.627275+0000","last_fullsized":"2021-02-10T04:51:13.627275+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'51","last_scrub_stamp":"2021-02-10T04:50:53.153251+0000","last_deep_scrub":"76'35","last_deep_scrub_stamp":"2021-02-10T04:26:58.198641+0000","last_clean_scrub_stamp":"2021-02-10T04:50:53.153251+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":74,"num_read_kb":54,"num_write":40,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"1047","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.619166+0000","last_change":"2021-02-10T04:51:07.997058+0000","last_active":"2021-02-10T04:51:13.619166+0000","last_peered":"2021-02-10T04:51:13.619166+0000","last_clean":"2021-02-10T04:51:13.619166+0000","last_became_active":"2021-02-10T04:50:59.555151+0000","last_became_peered":"2021-02-10T04:50:59.555151+0000","last_unstale":"2021-02-10T04:51:13.619166+0000","last_undegraded":"2021-02-10T04:51:13.619166+0000","last_fullsized":"2021-02-10T04:51:13.619166+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:42.537443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:42.537443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"1096","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.860244+0000","last_change":"2021-02-10T04:47:28.938832+0000","last_active":"2021-02-10T04:51:14.860244+0000","last_peered":"2021-02-10T04:51:14.860244+0000","last_clean":"2021-02-10T04:51:14.860244+0000","last_became_active":"2021-02-10T04:37:37.246998+0000","last_became_peered":"2021-02-10T04:37:37.246998+0000","last_unstale":"2021-02-10T04:51:14.860244+0000","last_undegraded":"2021-02-10T04:51:14.860244+0000","last_fullsized":"2021-02-10T04:51:14.860244+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.938731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.938731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"770","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.624485+0000","last_change":"2021-02-10T04:50:49.391976+0000","last_active":"2021-02-10T04:51:13.624485+0000","last_peered":"2021-02-10T04:51:13.624485+0000","last_clean":"2021-02-10T04:51:13.624485+0000","last_became_active":"2021-02-10T04:33:05.141911+0000","last_became_peered":"2021-02-10T04:33:05.141911+0000","last_unstale":"2021-02-10T04:51:13.624485+0000","last_undegraded":"2021-02-10T04:51:13.624485+0000","last_fullsized":"2021-02-10T04:51:13.624485+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:49.391902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:49.391902+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"1049'39","reported_seq":"56","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.620078+0000","last_change":"2021-02-10T04:51:08.433663+0000","last_active":"2021-02-10T04:51:13.620078+0000","last_peered":"2021-02-10T04:51:13.620078+0000","last_clean":"2021-02-10T04:51:13.620078+0000","last_became_active":"2021-02-10T04:50:59.567778+0000","last_became_peered":"2021-02-10T04:50:59.567778+0000","last_unstale":"2021-02-10T04:51:13.620078+0000","last_undegraded":"2021-02-10T04:51:13.620078+0000","last_fullsized":"2021-02-10T04:51:13.620078+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'39","last_scrub_stamp":"2021-02-10T04:51:08.433562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.433562+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"1099","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.626745+0000","last_change":"2021-02-10T04:46:47.143081+0000","last_active":"2021-02-10T04:51:13.626745+0000","last_peered":"2021-02-10T04:51:13.626745+0000","last_clean":"2021-02-10T04:51:13.626745+0000","last_became_active":"2021-02-10T04:25:36.336442+0000","last_became_peered":"2021-02-10T04:25:36.336442+0000","last_unstale":"2021-02-10T04:51:13.626745+0000","last_undegraded":"2021-02-10T04:51:13.626745+0000","last_fullsized":"2021-02-10T04:51:13.626745+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:46:47.142978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:47.142978+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"769","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.622285+0000","last_change":"2021-02-10T04:51:08.758225+0000","last_active":"2021-02-10T04:51:13.622285+0000","last_peered":"2021-02-10T04:51:13.622285+0000","last_clean":"2021-02-10T04:51:13.622285+0000","last_became_active":"2021-02-10T04:33:05.146838+0000","last_became_peered":"2021-02-10T04:33:05.146838+0000","last_unstale":"2021-02-10T04:51:13.622285+0000","last_undegraded":"2021-02-10T04:51:13.622285+0000","last_fullsized":"2021-02-10T04:51:13.622285+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:08.758129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.758129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"1079","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.622285+0000","last_change":"2021-02-10T04:46:36.790576+0000","last_active":"2021-02-10T04:51:13.622285+0000","last_peered":"2021-02-10T04:51:13.622285+0000","last_clean":"2021-02-10T04:51:13.622285+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:51:13.622285+0000","last_undegraded":"2021-02-10T04:51:13.622285+0000","last_fullsized":"2021-02-10T04:51:13.622285+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:36.790487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:36.790487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"1066'44","reported_seq":"1124","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.619458+0000","last_change":"2021-02-10T04:51:07.985331+0000","last_active":"2021-02-10T04:51:13.619458+0000","last_peered":"2021-02-10T04:51:13.619458+0000","last_clean":"2021-02-10T04:51:13.619458+0000","last_became_active":"2021-02-10T04:50:59.563388+0000","last_became_peered":"2021-02-10T04:50:59.563388+0000","last_unstale":"2021-02-10T04:51:13.619458+0000","last_undegraded":"2021-02-10T04:51:13.619458+0000","last_fullsized":"2021-02-10T04:51:13.619458+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1066'44","last_scrub_stamp":"2021-02-10T04:47:18.806645+0000","last_deep_scrub":"1066'44","last_deep_scrub_stamp":"2021-02-10T04:47:18.806645+0000","last_clean_scrub_stamp":"2021-02-10T04:47:18.806645+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":67,"num_read_kb":49,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"1068","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.619738+0000","last_change":"2021-02-10T04:51:08.006463+0000","last_active":"2021-02-10T04:51:13.619738+0000","last_peered":"2021-02-10T04:51:13.619738+0000","last_clean":"2021-02-10T04:51:13.619738+0000","last_became_active":"2021-02-10T04:50:59.563657+0000","last_became_peered":"2021-02-10T04:50:59.563657+0000","last_unstale":"2021-02-10T04:51:13.619738+0000","last_undegraded":"2021-02-10T04:51:13.619738+0000","last_fullsized":"2021-02-10T04:51:13.619738+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_clean_scrub_stamp":"2021-02-10T04:51:02.455702+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"1105","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.622097+0000","last_change":"2021-02-10T04:50:59.585214+0000","last_active":"2021-02-10T04:51:13.622097+0000","last_peered":"2021-02-10T04:51:13.622097+0000","last_clean":"2021-02-10T04:51:13.622097+0000","last_became_active":"2021-02-10T04:50:59.584846+0000","last_became_peered":"2021-02-10T04:50:59.584846+0000","last_unstale":"2021-02-10T04:51:13.622097+0000","last_undegraded":"2021-02-10T04:51:13.622097+0000","last_fullsized":"2021-02-10T04:51:13.622097+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:58.666264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:46:58.666264+0000","last_clean_scrub_stamp":"2021-02-10T04:46:58.666264+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"1075","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.620314+0000","last_change":"2021-02-10T04:51:08.002745+0000","last_active":"2021-02-10T04:51:13.620314+0000","last_peered":"2021-02-10T04:51:13.620314+0000","last_clean":"2021-02-10T04:51:13.620314+0000","last_became_active":"2021-02-10T04:50:59.539582+0000","last_became_peered":"2021-02-10T04:50:59.539582+0000","last_unstale":"2021-02-10T04:51:13.620314+0000","last_undegraded":"2021-02-10T04:51:13.620314+0000","last_fullsized":"2021-02-10T04:51:13.620314+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:26.576960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:26.576960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"1003'739","reported_seq":"3790","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.625573+0000","last_change":"2021-02-10T04:47:03.951820+0000","last_active":"2021-02-10T04:51:13.625573+0000","last_peered":"2021-02-10T04:51:13.625573+0000","last_clean":"2021-02-10T04:51:13.625573+0000","last_became_active":"2021-02-10T04:44:30.579847+0000","last_became_peered":"2021-02-10T04:44:30.579847+0000","last_unstale":"2021-02-10T04:51:13.625573+0000","last_undegraded":"2021-02-10T04:51:13.625573+0000","last_fullsized":"2021-02-10T04:51:13.625573+0000","mapping_epoch":1012,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1013,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:47:03.951729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:03.951729+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22230846,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"1112'47","reported_seq":"1187","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.620269+0000","last_change":"2021-02-10T04:51:02.860932+0000","last_active":"2021-02-10T04:51:13.620269+0000","last_peered":"2021-02-10T04:51:13.620269+0000","last_clean":"2021-02-10T04:51:13.620269+0000","last_became_active":"2021-02-10T04:50:59.570786+0000","last_became_peered":"2021-02-10T04:50:59.570786+0000","last_unstale":"2021-02-10T04:51:13.620269+0000","last_undegraded":"2021-02-10T04:51:13.620269+0000","last_fullsized":"2021-02-10T04:51:13.620269+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1112'47","last_scrub_stamp":"2021-02-10T04:51:02.860835+0000","last_deep_scrub":"1112'47","last_deep_scrub_stamp":"2021-02-10T04:51:02.860835+0000","last_clean_scrub_stamp":"2021-02-10T04:51:02.860835+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":71,"num_read_kb":55,"num_write":38,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"1016","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.623829+0000","last_change":"2021-02-10T04:46:59.760348+0000","last_active":"2021-02-10T04:51:13.623829+0000","last_peered":"2021-02-10T04:51:13.623829+0000","last_clean":"2021-02-10T04:51:13.623829+0000","last_became_active":"2021-02-10T04:37:37.241751+0000","last_became_peered":"2021-02-10T04:37:37.241751+0000","last_unstale":"2021-02-10T04:51:13.623829+0000","last_undegraded":"2021-02-10T04:51:13.623829+0000","last_fullsized":"2021-02-10T04:51:13.623829+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:59.760263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:59.760263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"1111","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.622995+0000","last_change":"2021-02-10T04:51:02.394551+0000","last_active":"2021-02-10T04:51:13.622995+0000","last_peered":"2021-02-10T04:51:13.622995+0000","last_clean":"2021-02-10T04:51:13.622995+0000","last_became_active":"2021-02-10T04:50:59.575340+0000","last_became_peered":"2021-02-10T04:50:59.575340+0000","last_unstale":"2021-02-10T04:51:13.622995+0000","last_undegraded":"2021-02-10T04:51:13.622995+0000","last_fullsized":"2021-02-10T04:51:13.622995+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:02.394440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:02.394440+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"1085","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.859326+0000","last_change":"2021-02-10T04:46:24.697867+0000","last_active":"2021-02-10T04:51:14.859326+0000","last_peered":"2021-02-10T04:51:14.859326+0000","last_clean":"2021-02-10T04:51:14.859326+0000","last_became_active":"2021-02-10T04:25:53.039844+0000","last_became_peered":"2021-02-10T04:25:53.039844+0000","last_unstale":"2021-02-10T04:51:14.859326+0000","last_undegraded":"2021-02-10T04:51:14.859326+0000","last_fullsized":"2021-02-10T04:51:14.859326+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:24.697786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:24.697786+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"259'74","reported_seq":"123","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.858839+0000","last_change":"2021-02-10T04:46:49.745214+0000","last_active":"2021-02-10T04:51:14.858839+0000","last_peered":"2021-02-10T04:51:14.858839+0000","last_clean":"2021-02-10T04:51:14.858839+0000","last_became_active":"2021-02-10T04:44:27.546590+0000","last_became_peered":"2021-02-10T04:44:27.546590+0000","last_unstale":"2021-02-10T04:51:14.858839+0000","last_undegraded":"2021-02-10T04:51:14.858839+0000","last_fullsized":"2021-02-10T04:51:14.858839+0000","mapping_epoch":1009,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1010,"parent":"0.0","parent_split_bits":5,"last_scrub":"259'74","last_scrub_stamp":"2021-02-10T04:46:49.745104+0000","last_deep_scrub":"259'74","last_deep_scrub_stamp":"2021-02-10T04:46:49.745104+0000","last_clean_scrub_stamp":"2021-02-10T04:46:49.745104+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"1061'20","reported_seq":"1135","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.620686+0000","last_change":"2021-02-10T04:50:59.569616+0000","last_active":"2021-02-10T04:51:13.620686+0000","last_peered":"2021-02-10T04:51:13.620686+0000","last_clean":"2021-02-10T04:51:13.620686+0000","last_became_active":"2021-02-10T04:50:59.569344+0000","last_became_peered":"2021-02-10T04:50:59.569344+0000","last_unstale":"2021-02-10T04:51:13.620686+0000","last_undegraded":"2021-02-10T04:51:13.620686+0000","last_fullsized":"2021-02-10T04:51:13.620686+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1061'20","last_scrub_stamp":"2021-02-10T04:46:55.646261+0000","last_deep_scrub":"1061'20","last_deep_scrub_stamp":"2021-02-10T04:46:55.646261+0000","last_clean_scrub_stamp":"2021-02-10T04:46:55.646261+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":17,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"1012","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.627224+0000","last_change":"2021-02-10T04:50:59.167977+0000","last_active":"2021-02-10T04:51:13.627224+0000","last_peered":"2021-02-10T04:51:13.627224+0000","last_clean":"2021-02-10T04:51:13.627224+0000","last_became_active":"2021-02-10T04:29:07.157944+0000","last_became_peered":"2021-02-10T04:29:07.157944+0000","last_unstale":"2021-02-10T04:51:13.627224+0000","last_undegraded":"2021-02-10T04:51:13.627224+0000","last_fullsized":"2021-02-10T04:51:13.627224+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:59.167911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:50:59.167911+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"1001","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.620342+0000","last_change":"2021-02-10T04:46:15.814062+0000","last_active":"2021-02-10T04:51:13.620342+0000","last_peered":"2021-02-10T04:51:13.620342+0000","last_clean":"2021-02-10T04:51:13.620342+0000","last_became_active":"2021-02-10T04:29:07.146525+0000","last_became_peered":"2021-02-10T04:29:07.146525+0000","last_unstale":"2021-02-10T04:51:13.620342+0000","last_undegraded":"2021-02-10T04:51:13.620342+0000","last_fullsized":"2021-02-10T04:51:13.620342+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:15.813941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:46:15.813941+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"1100","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.620029+0000","last_change":"2021-02-10T04:51:09.433892+0000","last_active":"2021-02-10T04:51:13.620029+0000","last_peered":"2021-02-10T04:51:13.620029+0000","last_clean":"2021-02-10T04:51:13.620029+0000","last_became_active":"2021-02-10T04:50:59.539256+0000","last_became_peered":"2021-02-10T04:50:59.539256+0000","last_unstale":"2021-02-10T04:51:13.620029+0000","last_undegraded":"2021-02-10T04:51:13.620029+0000","last_fullsized":"2021-02-10T04:51:13.620029+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:09.433793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:09.433793+0000","last_clean_scrub_stamp":"2021-02-10T04:51:09.433793+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"1048'44","reported_seq":"1160","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.619502+0000","last_change":"2021-02-10T04:50:48.668186+0000","last_active":"2021-02-10T04:51:13.619502+0000","last_peered":"2021-02-10T04:51:13.619502+0000","last_clean":"2021-02-10T04:51:13.619502+0000","last_became_active":"2021-02-10T04:25:49.010481+0000","last_became_peered":"2021-02-10T04:25:49.010481+0000","last_unstale":"2021-02-10T04:51:13.619502+0000","last_undegraded":"2021-02-10T04:51:13.619502+0000","last_fullsized":"2021-02-10T04:51:13.619502+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'44","last_scrub_stamp":"2021-02-10T04:50:48.668079+0000","last_deep_scrub":"78'32","last_deep_scrub_stamp":"2021-02-10T04:27:09.170428+0000","last_clean_scrub_stamp":"2021-02-10T04:50:48.668079+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":65,"num_read_kb":46,"num_write":34,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"1086","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.619804+0000","last_change":"2021-02-10T04:51:07.979394+0000","last_active":"2021-02-10T04:51:13.619804+0000","last_peered":"2021-02-10T04:51:13.619804+0000","last_clean":"2021-02-10T04:51:13.619804+0000","last_became_active":"2021-02-10T04:50:59.557394+0000","last_became_peered":"2021-02-10T04:50:59.557394+0000","last_unstale":"2021-02-10T04:51:13.619804+0000","last_undegraded":"2021-02-10T04:51:13.619804+0000","last_fullsized":"2021-02-10T04:51:13.619804+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:33.543559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:46:33.543559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"770","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.619890+0000","last_change":"2021-02-10T04:50:06.195327+0000","last_active":"2021-02-10T04:51:13.619890+0000","last_peered":"2021-02-10T04:51:13.619890+0000","last_clean":"2021-02-10T04:51:13.619890+0000","last_became_active":"2021-02-10T04:33:05.169159+0000","last_became_peered":"2021-02-10T04:33:05.169159+0000","last_unstale":"2021-02-10T04:51:13.619890+0000","last_undegraded":"2021-02-10T04:51:13.619890+0000","last_fullsized":"2021-02-10T04:51:13.619890+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:06.195234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:06.195234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"1165","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.620226+0000","last_change":"2021-02-10T04:47:12.800261+0000","last_active":"2021-02-10T04:51:13.620226+0000","last_peered":"2021-02-10T04:51:13.620226+0000","last_clean":"2021-02-10T04:51:13.620226+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:51:13.620226+0000","last_undegraded":"2021-02-10T04:51:13.620226+0000","last_fullsized":"2021-02-10T04:51:13.620226+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:12.800147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:12.800147+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"1049'68","reported_seq":"1222","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.618942+0000","last_change":"2021-02-10T04:51:07.979350+0000","last_active":"2021-02-10T04:51:13.618942+0000","last_peered":"2021-02-10T04:51:13.618942+0000","last_clean":"2021-02-10T04:51:13.618942+0000","last_became_active":"2021-02-10T04:50:59.561153+0000","last_became_peered":"2021-02-10T04:50:59.561153+0000","last_unstale":"2021-02-10T04:51:13.618942+0000","last_undegraded":"2021-02-10T04:51:13.618942+0000","last_fullsized":"2021-02-10T04:51:13.618942+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'68","last_scrub_stamp":"2021-02-10T04:47:09.608881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:47:09.608881+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":124,"num_read_kb":101,"num_write":69,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":890,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"1105","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.860045+0000","last_change":"2021-02-10T04:47:17.919840+0000","last_active":"2021-02-10T04:51:14.860045+0000","last_peered":"2021-02-10T04:51:14.860045+0000","last_clean":"2021-02-10T04:51:14.860045+0000","last_became_active":"2021-02-10T04:25:36.336112+0000","last_became_peered":"2021-02-10T04:25:36.336112+0000","last_unstale":"2021-02-10T04:51:14.860045+0000","last_undegraded":"2021-02-10T04:51:14.860045+0000","last_fullsized":"2021-02-10T04:51:14.860045+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:17.919745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:17.919745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"397","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.618381+0000","last_change":"2021-02-10T04:51:07.970205+0000","last_active":"2021-02-10T04:51:13.618381+0000","last_peered":"2021-02-10T04:51:13.618381+0000","last_clean":"2021-02-10T04:51:13.618381+0000","last_became_active":"2021-02-10T04:50:59.572077+0000","last_became_peered":"2021-02-10T04:50:59.572077+0000","last_unstale":"2021-02-10T04:51:13.618381+0000","last_undegraded":"2021-02-10T04:51:13.618381+0000","last_fullsized":"2021-02-10T04:51:13.618381+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:19.755718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:19.755718+0000","last_clean_scrub_stamp":"2021-02-10T04:47:19.755718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"1074","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.619419+0000","last_change":"2021-02-10T04:51:08.006421+0000","last_active":"2021-02-10T04:51:13.619419+0000","last_peered":"2021-02-10T04:51:13.619419+0000","last_clean":"2021-02-10T04:51:13.619419+0000","last_became_active":"2021-02-10T04:50:59.563156+0000","last_became_peered":"2021-02-10T04:50:59.563156+0000","last_unstale":"2021-02-10T04:51:13.619419+0000","last_undegraded":"2021-02-10T04:51:13.619419+0000","last_fullsized":"2021-02-10T04:51:13.619419+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:14.549136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:14.549136+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":"797","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.619713+0000","last_change":"2021-02-10T04:50:59.595747+0000","last_active":"2021-02-10T04:51:13.619713+0000","last_peered":"2021-02-10T04:51:13.619713+0000","last_clean":"2021-02-10T04:51:13.619713+0000","last_became_active":"2021-02-10T04:50:59.595454+0000","last_became_peered":"2021-02-10T04:50:59.595454+0000","last_unstale":"2021-02-10T04:51:13.619713+0000","last_undegraded":"2021-02-10T04:51:13.619713+0000","last_fullsized":"2021-02-10T04:51:13.619713+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:20.635536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:20.635536+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.635536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.20","version":"0'0","reported_seq":"1070","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.621614+0000","last_change":"2021-02-10T04:46:50.859426+0000","last_active":"2021-02-10T04:51:13.621614+0000","last_peered":"2021-02-10T04:51:13.621614+0000","last_clean":"2021-02-10T04:51:13.621614+0000","last_became_active":"2021-02-10T04:29:07.148937+0000","last_became_peered":"2021-02-10T04:29:07.148937+0000","last_unstale":"2021-02-10T04:51:13.621614+0000","last_undegraded":"2021-02-10T04:51:13.621614+0000","last_fullsized":"2021-02-10T04:51:13.621614+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:50.859195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:41.375276+0000","last_clean_scrub_stamp":"2021-02-10T04:46:50.859195+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":"1029","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.626811+0000","last_change":"2021-02-10T04:50:04.210063+0000","last_active":"2021-02-10T04:51:13.626811+0000","last_peered":"2021-02-10T04:51:13.626811+0000","last_clean":"2021-02-10T04:51:13.626811+0000","last_became_active":"2021-02-10T04:45:49.551858+0000","last_became_peered":"2021-02-10T04:45:49.551858+0000","last_unstale":"2021-02-10T04:51:13.626811+0000","last_undegraded":"2021-02-10T04:51:13.626811+0000","last_fullsized":"2021-02-10T04:51:13.626811+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:04.210002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:50:04.210002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":"588","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:14.859059+0000","last_change":"2021-02-10T04:50:47.481762+0000","last_active":"2021-02-10T04:51:14.859059+0000","last_peered":"2021-02-10T04:51:14.859059+0000","last_clean":"2021-02-10T04:51:14.859059+0000","last_became_active":"2021-02-10T04:37:37.246865+0000","last_became_peered":"2021-02-10T04:37:37.246865+0000","last_unstale":"2021-02-10T04:51:14.859059+0000","last_undegraded":"2021-02-10T04:51:14.859059+0000","last_fullsized":"2021-02-10T04:51:14.859059+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:47.481686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:47.481686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":"84","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.623944+0000","last_change":"2021-02-10T04:47:28.706460+0000","last_active":"2021-02-10T04:51:13.623944+0000","last_peered":"2021-02-10T04:51:13.623944+0000","last_clean":"2021-02-10T04:51:13.623944+0000","last_became_active":"2021-02-10T04:45:49.552420+0000","last_became_peered":"2021-02-10T04:45:49.552420+0000","last_unstale":"2021-02-10T04:51:13.623944+0000","last_undegraded":"2021-02-10T04:51:13.623944+0000","last_fullsized":"2021-02-10T04:51:13.623944+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.706372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.706372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":"1120","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.626446+0000","last_change":"2021-02-10T04:50:59.567255+0000","last_active":"2021-02-10T04:51:13.626446+0000","last_peered":"2021-02-10T04:51:13.626446+0000","last_clean":"2021-02-10T04:51:13.626446+0000","last_became_active":"2021-02-10T04:50:59.566827+0000","last_became_peered":"2021-02-10T04:50:59.566827+0000","last_unstale":"2021-02-10T04:51:13.626446+0000","last_undegraded":"2021-02-10T04:51:13.626446+0000","last_fullsized":"2021-02-10T04:51:13.626446+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:32.031098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:32.031098+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":"84","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.625634+0000","last_change":"2021-02-10T04:50:10.308350+0000","last_active":"2021-02-10T04:51:13.625634+0000","last_peered":"2021-02-10T04:51:13.625634+0000","last_clean":"2021-02-10T04:51:13.625634+0000","last_became_active":"2021-02-10T04:45:49.551121+0000","last_became_peered":"2021-02-10T04:45:49.551121+0000","last_unstale":"2021-02-10T04:51:13.625634+0000","last_undegraded":"2021-02-10T04:51:13.625634+0000","last_fullsized":"2021-02-10T04:51:13.625634+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:10.308282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:10.308282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":"1010","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.619757+0000","last_change":"2021-02-10T04:51:07.989063+0000","last_active":"2021-02-10T04:51:13.619757+0000","last_peered":"2021-02-10T04:51:13.619757+0000","last_clean":"2021-02-10T04:51:13.619757+0000","last_became_active":"2021-02-10T04:50:59.561442+0000","last_became_peered":"2021-02-10T04:50:59.561442+0000","last_unstale":"2021-02-10T04:51:13.619757+0000","last_undegraded":"2021-02-10T04:51:13.619757+0000","last_fullsized":"2021-02-10T04:51:13.619757+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:51.620102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:44:51.620102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":"84","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.625344+0000","last_change":"2021-02-10T04:50:08.272047+0000","last_active":"2021-02-10T04:51:13.625344+0000","last_peered":"2021-02-10T04:51:13.625344+0000","last_clean":"2021-02-10T04:51:13.625344+0000","last_became_active":"2021-02-10T04:45:49.759179+0000","last_became_peered":"2021-02-10T04:45:49.759179+0000","last_unstale":"2021-02-10T04:51:13.625344+0000","last_undegraded":"2021-02-10T04:51:13.625344+0000","last_fullsized":"2021-02-10T04:51:13.625344+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:08.271977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:08.271977+0000","last_clean_scrub_stamp":"2021-02-10T04:50:08.271977+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":"1054","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.627778+0000","last_change":"2021-02-10T04:51:07.182775+0000","last_active":"2021-02-10T04:51:13.627778+0000","last_peered":"2021-02-10T04:51:13.627778+0000","last_clean":"2021-02-10T04:51:13.627778+0000","last_became_active":"2021-02-10T04:28:23.126368+0000","last_became_peered":"2021-02-10T04:28:23.126368+0000","last_unstale":"2021-02-10T04:51:13.627778+0000","last_undegraded":"2021-02-10T04:51:13.627778+0000","last_fullsized":"2021-02-10T04:51:13.627778+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.182678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.182678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":"84","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.621385+0000","last_change":"2021-02-10T04:50:07.767334+0000","last_active":"2021-02-10T04:51:13.621385+0000","last_peered":"2021-02-10T04:51:13.621385+0000","last_clean":"2021-02-10T04:51:13.621385+0000","last_became_active":"2021-02-10T04:45:49.759224+0000","last_became_peered":"2021-02-10T04:45:49.759224+0000","last_unstale":"2021-02-10T04:51:13.621385+0000","last_undegraded":"2021-02-10T04:51:13.621385+0000","last_fullsized":"2021-02-10T04:51:13.621385+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:07.767250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:07.767250+0000","last_clean_scrub_stamp":"2021-02-10T04:50:07.767250+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.25","version":"53'2","reported_seq":"1066","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.622348+0000","last_change":"2021-02-10T04:50:12.772167+0000","last_active":"2021-02-10T04:51:13.622348+0000","last_peered":"2021-02-10T04:51:13.622348+0000","last_clean":"2021-02-10T04:51:13.622348+0000","last_became_active":"2021-02-10T04:28:23.467770+0000","last_became_peered":"2021-02-10T04:28:23.467770+0000","last_unstale":"2021-02-10T04:51:13.622348+0000","last_undegraded":"2021-02-10T04:51:13.622348+0000","last_fullsized":"2021-02-10T04:51:13.622348+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:50:12.772086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:12.772086+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":"1156","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.626236+0000","last_change":"2021-02-10T04:50:59.567042+0000","last_active":"2021-02-10T04:51:13.626236+0000","last_peered":"2021-02-10T04:51:13.626236+0000","last_clean":"2021-02-10T04:51:13.626236+0000","last_became_active":"2021-02-10T04:50:59.566548+0000","last_became_peered":"2021-02-10T04:50:59.566548+0000","last_unstale":"2021-02-10T04:51:13.626236+0000","last_undegraded":"2021-02-10T04:51:13.626236+0000","last_fullsized":"2021-02-10T04:51:13.626236+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:27.926662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:44:27.926662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.26","version":"53'1","reported_seq":"506","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.622189+0000","last_change":"2021-02-10T04:47:11.808609+0000","last_active":"2021-02-10T04:51:13.622189+0000","last_peered":"2021-02-10T04:51:13.622189+0000","last_clean":"2021-02-10T04:51:13.622189+0000","last_became_active":"2021-02-10T04:37:37.231081+0000","last_became_peered":"2021-02-10T04:37:37.231081+0000","last_unstale":"2021-02-10T04:51:13.622189+0000","last_undegraded":"2021-02-10T04:51:13.622189+0000","last_fullsized":"2021-02-10T04:51:13.622189+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:47:11.808550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:11.808550+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":"1038","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.626204+0000","last_change":"2021-02-10T04:47:22.953445+0000","last_active":"2021-02-10T04:51:13.626204+0000","last_peered":"2021-02-10T04:51:13.626204+0000","last_clean":"2021-02-10T04:51:13.626204+0000","last_became_active":"2021-02-10T04:45:49.552026+0000","last_became_peered":"2021-02-10T04:45:49.552026+0000","last_unstale":"2021-02-10T04:51:13.626204+0000","last_undegraded":"2021-02-10T04:51:13.626204+0000","last_fullsized":"2021-02-10T04:51:13.626204+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:22.953356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.953356+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":"1077","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.622242+0000","last_change":"2021-02-10T04:47:16.706513+0000","last_active":"2021-02-10T04:51:13.622242+0000","last_peered":"2021-02-10T04:51:13.622242+0000","last_clean":"2021-02-10T04:51:13.622242+0000","last_became_active":"2021-02-10T04:29:07.145907+0000","last_became_peered":"2021-02-10T04:29:07.145907+0000","last_unstale":"2021-02-10T04:51:13.622242+0000","last_undegraded":"2021-02-10T04:51:13.622242+0000","last_fullsized":"2021-02-10T04:51:13.622242+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:16.706416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:16.706416+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":"1039","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.618493+0000","last_change":"2021-02-10T04:51:07.984986+0000","last_active":"2021-02-10T04:51:13.618493+0000","last_peered":"2021-02-10T04:51:13.618493+0000","last_clean":"2021-02-10T04:51:13.618493+0000","last_became_active":"2021-02-10T04:50:59.569162+0000","last_became_peered":"2021-02-10T04:50:59.569162+0000","last_unstale":"2021-02-10T04:51:13.618493+0000","last_undegraded":"2021-02-10T04:51:13.618493+0000","last_fullsized":"2021-02-10T04:51:13.618493+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:12.555666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:46:12.555666+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":"1064","reported_epoch":"1115","state":"active+clean","last_fresh":"2021-02-10T04:51:13.625837+0000","last_change":"2021-02-10T04:50:59.528569+0000","last_active":"2021-02-10T04:51:13.625837+0000","last_peered":"2021-02-10T04:51:13.625837+0000","last_clean":"2021-02-10T04:51:13.625837+0000","last_became_active":"2021-02-10T04:50:59.527945+0000","last_became_peered":"2021-02-10T04:50:59.527945+0000","last_unstale":"2021-02-10T04:51:13.625837+0000","last_undegraded":"2021-02-10T04:51:13.625837+0000","last_fullsized":"2021-02-10T04:51:13.625837+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:50.057162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_clean_scrub_stamp":"2021-02-10T04:46:50.057162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":940005779,"num_objects":386,"num_object_clones":336,"num_object_copies":1158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":386,"num_whiteouts":12,"num_read":4562,"num_read_kb":4769301,"num_write":3301,"num_write_kb":1265045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":101337272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1931636736,"data_stored":2559497134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23681,"ondisk_log_size":23681,"up":78,"acting":78,"num_store_stats":8},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":1945,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":17,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":6565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":126,"acting":126,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":832,"num_read_kb":827,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":2297,"num_read_kb":1770,"num_write":1306,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":8770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":75722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1392,"ondisk_log_size":1392,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":126,"acting":126,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365112,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":227684,"kb_used_data":192796,"kb_used_omap":0,"kb_used_meta":34880,"kb_avail":93517468,"statfs":{"total":95995035648,"available":95761887232,"internally_reserved":0,"allocated":197423104,"data_stored":268690864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35717120},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:50:27 2021","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":1.2969999999999999,"15min":1.288},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":2.1379999999999999,"5min":34.299999999999997,"15min":67.277000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.516,"15min":1.548},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":2.4260000000000002,"5min":34.652999999999999,"15min":66.933999999999997},"last":0.63600000000000001}]},{"osd":1,"last update":"Wed Feb 10 04:50:27 2021","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.91100000000000003,"15min":0.99399999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.48399999999999999},"max":{"1min":2.1829999999999998,"5min":2.3420000000000001,"15min":10.140000000000001},"last":0.874},{"interface":"front","average":{"1min":0.97799999999999998,"5min":1.048,"15min":1.01},"min":{"1min":0.66200000000000003,"5min":0.623,"15min":0.57299999999999995},"max":{"1min":2.5409999999999999,"5min":2.6549999999999998,"15min":7.1520000000000001},"last":0.79000000000000004}]},{"osd":2,"last update":"Wed Feb 10 04:50:27 2021","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.1100000000000001,"15min":1.462},"min":{"1min":0.61899999999999999,"5min":0.61399999999999999,"15min":0.58299999999999996},"max":{"1min":3.0630000000000002,"5min":3.0630000000000002,"15min":66.534000000000006},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.21,"5min":1.218,"15min":1.712},"min":{"1min":0.67900000000000005,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.8660000000000001,"5min":6.7930000000000001,"15min":66.781999999999996},"last":0.84799999999999998}]},{"osd":3,"last update":"Wed Feb 10 04:49:21 2021","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.2749999999999999,"15min":1.587},"min":{"1min":0.73099999999999998,"5min":0.72999999999999998,"15min":0.69599999999999995},"max":{"1min":2.3210000000000002,"5min":5.2599999999999998,"15min":68.305000000000007},"last":1.0409999999999999},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.3129999999999999,"15min":1.552},"min":{"1min":0.95899999999999996,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":2.0310000000000001,"5min":2.4249999999999998,"15min":69.331999999999994},"last":1.101}]},{"osd":4,"last update":"Wed Feb 10 04:50:27 2021","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.454,"15min":1.78},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.64600000000000002},"max":{"1min":2.778,"5min":7.0640000000000001,"15min":97.733999999999995},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.343,"15min":1.6899999999999999},"min":{"1min":0.74299999999999999,"5min":0.69099999999999995,"15min":0.65100000000000002},"max":{"1min":2.903,"5min":4.6269999999999998,"15min":97.691000000000003},"last":1.1339999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:50:27 2021","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.6459999999999999,"15min":1.6459999999999999},"min":{"1min":0.875,"5min":0.85699999999999998,"15min":0.66700000000000004},"max":{"1min":3.1819999999999999,"5min":14.972,"15min":35.283000000000001},"last":1.9590000000000001},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.8109999999999999,"15min":1.958},"min":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.85299999999999998},"max":{"1min":3.101,"5min":15.218999999999999,"15min":35.353999999999999},"last":0.95599999999999996}]},{"osd":6,"last update":"Wed Feb 10 04:50:27 2021","interfaces":[{"interface":"back","average":{"1min":1.6459999999999999,"5min":1.895,"15min":1.899},"min":{"1min":0.97399999999999998,"5min":0.97199999999999998,"15min":0.89100000000000001},"max":{"1min":3.2879999999999998,"5min":16.050000000000001,"15min":16.050000000000001},"last":1.181},{"interface":"front","average":{"1min":1.49,"5min":1.839,"15min":1.7889999999999999},"min":{"1min":1.077,"5min":0.94799999999999995,"15min":0.94799999999999995},"max":{"1min":3.3119999999999998,"5min":15.960000000000001,"15min":15.960000000000001},"last":1.3560000000000001}]}]},{"osd":6,"up_from":43,"seq":184683594044,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":359796,"kb_used_data":306924,"kb_used_omap":0,"kb_used_meta":52864,"kb_avail":93385356,"statfs":{"total":95995035648,"available":95626604544,"internally_reserved":0,"allocated":314290176,"data_stored":402056736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54132736},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:10 2021","interfaces":[{"interface":"back","average":{"1min":6.6200000000000001,"5min":2.121,"15min":1.3799999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.44700000000000001},"max":{"1min":137.60599999999999,"5min":137.60599999999999,"15min":137.60599999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":6.5789999999999997,"5min":2.0840000000000001,"15min":1.5960000000000001},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":137.66,"5min":137.66,"15min":137.66},"last":0.67100000000000004}]},{"osd":1,"last update":"Wed Feb 10 04:51:10 2021","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.0509999999999999,"15min":1.0580000000000001},"min":{"1min":0.53000000000000003,"5min":0.499,"15min":0.47699999999999998},"max":{"1min":5.2160000000000002,"5min":5.2160000000000002,"15min":12.978999999999999},"last":1.143},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.115,"15min":1.1359999999999999},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":3.6219999999999999,"5min":3.6219999999999999,"15min":6.0960000000000001},"last":1.119}]},{"osd":2,"last update":"Wed Feb 10 04:51:10 2021","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.246,"15min":1.3340000000000001},"min":{"1min":0.61099999999999999,"5min":0.56200000000000006,"15min":0.54100000000000004},"max":{"1min":5.0990000000000002,"5min":5.0990000000000002,"15min":24.725000000000001},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.236,"5min":1.2989999999999999,"15min":1.3240000000000001},"min":{"1min":0.76900000000000002,"5min":0.66200000000000003,"15min":0.59899999999999998},"max":{"1min":5.3140000000000001,"5min":5.3140000000000001,"15min":24.611999999999998},"last":1.0349999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:49:04 2021","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":2.948,"15min":2.4060000000000001},"min":{"1min":0.89600000000000002,"5min":0.66700000000000004,"15min":0.63200000000000001},"max":{"1min":2.1429999999999998,"5min":75.897000000000006,"15min":114.17700000000001},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.5469999999999999,"5min":2.9300000000000002,"15min":2.4489999999999998},"min":{"1min":1.0409999999999999,"5min":0.745,"15min":0.745},"max":{"1min":2.1680000000000001,"5min":75.994,"15min":116.929},"last":1.262}]},{"osd":4,"last update":"Wed Feb 10 04:51:10 2021","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.4099999999999999,"15min":1.456},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.60599999999999998},"max":{"1min":7.1459999999999999,"5min":7.1459999999999999,"15min":7.5389999999999997},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.319,"5min":1.109,"15min":1.1359999999999999},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":11.997999999999999,"5min":11.997999999999999,"15min":11.997999999999999},"last":1.4019999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:51:10 2021","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.2589999999999999,"15min":1.6870000000000001},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.56899999999999995},"max":{"1min":4.2249999999999996,"5min":4.2249999999999996,"15min":81.540000000000006},"last":1.657},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.595,"15min":1.9330000000000001},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.66600000000000004},"max":{"1min":5.5069999999999997,"5min":5.5069999999999997,"15min":81.492999999999995},"last":1.51}]},{"osd":7,"last update":"Wed Feb 10 04:50:31 2021","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.3220000000000001,"15min":1.3779999999999999},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.665,"5min":2.2010000000000001,"15min":5.1820000000000004},"last":1.5389999999999999},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.625,"15min":1.6679999999999999},"min":{"1min":1.008,"5min":0.98399999999999999,"15min":0.77500000000000002},"max":{"1min":2.2799999999999998,"5min":3.2349999999999999,"15min":13.315},"last":1.627}]}]},{"osd":1,"up_from":13,"seq":55834575181,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":346996,"kb_used_data":292332,"kb_used_omap":0,"kb_used_meta":54656,"kb_avail":93398156,"statfs":{"total":95995035648,"available":95639711744,"internally_reserved":0,"allocated":299347968,"data_stored":381866732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55967744},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:50:45 2021","interfaces":[{"interface":"back","average":{"1min":8.1839999999999993,"5min":2.484,"15min":1.8859999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":115.622,"5min":115.622,"15min":115.622},"last":0.58699999999999997},{"interface":"front","average":{"1min":8.2690000000000001,"5min":2.4289999999999998,"15min":1.9119999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":115.73099999999999,"5min":115.73099999999999,"15min":115.73099999999999},"last":0.93200000000000005}]},{"osd":2,"last update":"Wed Feb 10 04:51:04 2021","interfaces":[{"interface":"back","average":{"1min":6.8689999999999998,"5min":2.1139999999999999,"15min":1.6619999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":115.59699999999999,"5min":115.59699999999999,"15min":115.59699999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":6.9539999999999997,"5min":2.1899999999999999,"15min":1.704},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.46800000000000003},"max":{"1min":115.753,"5min":115.753,"15min":115.753},"last":0.81899999999999995}]},{"osd":3,"last update":"Wed Feb 10 04:49:15 2021","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.2609999999999999,"15min":1.5740000000000001},"min":{"1min":0.80900000000000005,"5min":0.58899999999999997,"15min":0.57899999999999996},"max":{"1min":2.262,"5min":4.9909999999999997,"15min":32.432000000000002},"last":0.71399999999999997},{"interface":"front","average":{"1min":1.258,"5min":1.351,"15min":1.601},"min":{"1min":0.629,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":2.226,"5min":5.9720000000000004,"15min":32.811},"last":1.0189999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:50:45 2021","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":1.169,"15min":1.2330000000000001},"min":{"1min":0.67900000000000005,"5min":0.66100000000000003,"15min":0.55200000000000005},"max":{"1min":1.8899999999999999,"5min":4.899,"15min":6.3099999999999996},"last":0.96899999999999997},{"interface":"front","average":{"1min":1.075,"5min":1.198,"15min":1.3939999999999999},"min":{"1min":0.79500000000000004,"5min":0.747,"15min":0.70999999999999996},"max":{"1min":2.4649999999999999,"5min":2.4649999999999999,"15min":29.547999999999998},"last":1.613}]},{"osd":5,"last update":"Wed Feb 10 04:50:52 2021","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.2030000000000001,"15min":1.4590000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":2.266,"5min":5.0300000000000002,"15min":48.686999999999998},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.411,"15min":1.778},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.7050000000000001,"5min":5.0389999999999997,"15min":48.905000000000001},"last":1.0469999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:51:04 2021","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":1.26,"15min":1.6930000000000001},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.7030000000000001,"5min":2.1659999999999999,"15min":48.878999999999998},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.329,"15min":1.609},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":2.661,"5min":5.1429999999999998,"15min":49.011000000000003},"last":0.95399999999999996}]},{"osd":7,"last update":"Wed Feb 10 04:51:20 2021","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.4179999999999999,"15min":1.5629999999999999},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":3.1139999999999999,"5min":4.9770000000000003,"15min":28.573},"last":1.121},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.3959999999999999,"15min":1.597},"min":{"1min":0.88800000000000001,"5min":0.87,"15min":0.87},"max":{"1min":1.6080000000000001,"5min":2.298,"15min":29.408999999999999},"last":1.5189999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738707,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":395080,"kb_used_data":334720,"kb_used_omap":0,"kb_used_meta":60352,"kb_avail":93350072,"statfs":{"total":95995035648,"available":95590473728,"internally_reserved":0,"allocated":342753280,"data_stored":467921886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":61800448},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 04:50:30 2021","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.88500000000000001,"15min":1.0800000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.704,"5min":2.2930000000000001,"15min":28.946999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.94699999999999995,"15min":1.022},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.534,"5min":2.0670000000000002,"15min":3.7050000000000001},"last":0.90700000000000003}]},{"osd":2,"last update":"Wed Feb 10 04:51:07 2021","interfaces":[{"interface":"back","average":{"1min":7.0490000000000004,"5min":2.2650000000000001,"15min":1.6870000000000001},"min":{"1min":0.57299999999999995,"5min":0.52100000000000002,"15min":0.51400000000000001},"max":{"1min":97.983999999999995,"5min":97.983999999999995,"15min":97.983999999999995},"last":0.69699999999999995},{"interface":"front","average":{"1min":6.992,"5min":2.2389999999999999,"15min":1.591},"min":{"1min":0.51300000000000001,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":97.844999999999999,"5min":97.844999999999999,"15min":97.844999999999999},"last":0.66200000000000003}]},{"osd":3,"last update":"Wed Feb 10 04:49:20 2021","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.1870000000000001,"15min":1.4099999999999999},"min":{"1min":0.55300000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":3.3420000000000001,"5min":3.3420000000000001,"15min":29.692},"last":0.78800000000000003},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.401,"15min":1.712},"min":{"1min":0.83999999999999997,"5min":0.753,"15min":0.68600000000000005},"max":{"1min":2.6379999999999999,"5min":4.0700000000000003,"15min":29.814},"last":1.038}]},{"osd":4,"last update":"Wed Feb 10 04:50:30 2021","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":1.1819999999999999,"15min":1.3280000000000001},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.7330000000000001,"5min":2.206,"15min":5.1680000000000001},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.099,"5min":1.2230000000000001,"15min":1.286},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.8420000000000001,"5min":3.359,"15min":14.672000000000001},"last":1.1639999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:50:53 2021","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":1.429,"15min":1.5429999999999999},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.4370000000000001,"5min":6.5990000000000002,"15min":25.047000000000001},"last":1.714},{"interface":"front","average":{"1min":1.042,"5min":1.4870000000000001,"15min":1.796},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.758,"5min":3.7850000000000001,"15min":28.827999999999999},"last":0.96299999999999997}]},{"osd":6,"last update":"Wed Feb 10 04:51:21 2021","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.9339999999999999,"15min":1.8420000000000001},"min":{"1min":0.81200000000000006,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.663,"5min":45.012,"15min":45.012},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.9590000000000001,"15min":2.024},"min":{"1min":0.873,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.5449999999999999,"5min":45.225000000000001,"15min":45.225000000000001},"last":1.069}]},{"osd":7,"last update":"Wed Feb 10 04:50:23 2021","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.4399999999999999,"15min":1.5589999999999999},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.8999999999999999,"5min":2.8700000000000001,"15min":5.6050000000000004},"last":1.333},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.554,"15min":1.671},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":2.1320000000000001,"5min":3.4689999999999999,"15min":16.545000000000002},"last":1.385}]}]},{"osd":2,"up_from":0,"seq":77309411660,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":1111,"seq":4771708665862,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":338196,"kb_used_data":319500,"kb_used_omap":386,"kb_used_meta":18301,"kb_avail":93406956,"statfs":{"total":95995035648,"available":95648722944,"internally_reserved":0,"allocated":327168000,"data_stored":451860804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":396036,"internal_metadata":18740476},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0860000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93999999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1910000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7280000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.506}]}]},{"osd":4,"up_from":31,"seq":133143986499,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":120460,"kb_used_data":71620,"kb_used_omap":0,"kb_used_meta":48832,"kb_avail":93624692,"statfs":{"total":95995035648,"available":95871684608,"internally_reserved":0,"allocated":73338880,"data_stored":93741793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":50003968},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4],"upper_bound":1048576},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:50:32 2021","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":2.1219999999999999,"15min":1.6120000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":3.2370000000000001,"5min":95.721999999999994,"15min":95.721999999999994},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.77500000000000002,"5min":2.1509999999999998,"15min":1.6180000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.637,"5min":95.344999999999999,"15min":95.344999999999999},"last":0.56299999999999994}]},{"osd":1,"last update":"Wed Feb 10 04:50:32 2021","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":0.94299999999999995,"15min":0.96199999999999997},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":7.883,"5min":7.883,"15min":24.295000000000002},"last":0.67300000000000004},{"interface":"front","average":{"1min":1.081,"5min":1.119,"15min":1.24},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.60399999999999998},"max":{"1min":3.1160000000000001,"5min":4.3449999999999998,"15min":24.943000000000001},"last":0.89600000000000002}]},{"osd":2,"last update":"Wed Feb 10 04:50:32 2021","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.208,"15min":1.1879999999999999},"min":{"1min":0.69199999999999995,"5min":0.627,"15min":0.61299999999999999},"max":{"1min":3.359,"5min":4.2279999999999998,"15min":7.048},"last":0.78100000000000003},{"interface":"front","average":{"1min":1.054,"5min":1.232,"15min":1.2050000000000001},"min":{"1min":0.73399999999999999,"5min":0.60299999999999998,"15min":0.51900000000000002},"max":{"1min":1.663,"5min":4.3840000000000003,"15min":4.3840000000000003},"last":0.83099999999999996}]},{"osd":3,"last update":"Wed Feb 10 04:49:28 2021","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":2.581,"15min":1.8220000000000001},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":2.1920000000000002,"5min":95.533000000000001,"15min":95.533000000000001},"last":1.014},{"interface":"front","average":{"1min":1.2450000000000001,"5min":2.508,"15min":1.7769999999999999},"min":{"1min":0.875,"5min":0.74299999999999999,"15min":0.70299999999999996},"max":{"1min":2.5800000000000001,"5min":95.426000000000002,"15min":95.426000000000002},"last":1.125}]},{"osd":5,"last update":"Wed Feb 10 04:50:47 2021","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":1.845,"15min":1.597},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.60999999999999999},"max":{"1min":8.1509999999999998,"5min":35.774000000000001,"15min":35.774000000000001},"last":1.177},{"interface":"front","average":{"1min":1.413,"5min":1.8,"15min":1.5389999999999999},"min":{"1min":0.76200000000000001,"5min":0.67100000000000004,"15min":0.58799999999999997},"max":{"1min":7.9950000000000001,"5min":35.606000000000002,"15min":35.606000000000002},"last":0.98099999999999998}]},{"osd":6,"last update":"Wed Feb 10 04:50:59 2021","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":3.8809999999999998,"15min":2.54},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.70299999999999996},"max":{"1min":4.5270000000000001,"5min":201.626,"15min":201.626},"last":1.6819999999999999},{"interface":"front","average":{"1min":1.3089999999999999,"5min":3.9359999999999999,"15min":2.617},"min":{"1min":0.90200000000000002,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":1.855,"5min":201.577,"15min":201.577},"last":1.7090000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:50:25 2021","interfaces":[{"interface":"back","average":{"1min":1.6910000000000001,"5min":1.512,"15min":1.5189999999999999},"min":{"1min":1.0349999999999999,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":4.7439999999999998,"5min":4.7439999999999998,"15min":6.9199999999999999},"last":1.456},{"interface":"front","average":{"1min":1.397,"5min":1.6060000000000001,"15min":1.5800000000000001},"min":{"1min":0.91400000000000003,"5min":0.72899999999999998,"15min":0.71499999999999997},"max":{"1min":1.831,"5min":5.149,"15min":5.149},"last":1.0669999999999999}]}]},{"osd":5,"up_from":37,"seq":158913790272,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":386972,"kb_used_data":345172,"kb_used_omap":0,"kb_used_meta":41792,"kb_avail":93358180,"statfs":{"total":95995035648,"available":95598776320,"internally_reserved":0,"allocated":353456128,"data_stored":447715787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42795008},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:50:48 2021","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.87,"15min":0.93600000000000005},"min":{"1min":0.443,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.984,"5min":2.2509999999999999,"15min":4.5720000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.98699999999999999,"15min":1.0309999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46400000000000002},"max":{"1min":1.5800000000000001,"5min":3.0169999999999999,"15min":15.539999999999999},"last":1.1679999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:50:48 2021","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.878,"15min":0.92600000000000005},"min":{"1min":0.55200000000000005,"5min":0.52800000000000002,"15min":0.504},"max":{"1min":2.1179999999999999,"5min":2.1179999999999999,"15min":5.29},"last":1.196},{"interface":"front","average":{"1min":1.095,"5min":1.1040000000000001,"15min":1.137},"min":{"1min":0.63700000000000001,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":2.0609999999999999,"5min":3.9180000000000001,"15min":11.875999999999999},"last":1.343}]},{"osd":2,"last update":"Wed Feb 10 04:50:48 2021","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.119,"15min":1.3580000000000001},"min":{"1min":0.627,"5min":0.53700000000000003,"15min":0.50600000000000001},"max":{"1min":1.7849999999999999,"5min":2.141,"15min":44.462000000000003},"last":1.01},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.1240000000000001,"15min":1.365},"min":{"1min":0.67800000000000005,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":2.2189999999999999,"5min":2.3420000000000001,"15min":44.398000000000003},"last":1.415}]},{"osd":3,"last update":"Wed Feb 10 04:49:47 2021","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.855,"15min":1.6559999999999999},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.0129999999999999,"5min":45.643000000000001,"15min":45.643000000000001},"last":1.127},{"interface":"front","average":{"1min":1.139,"5min":1.8169999999999999,"15min":1.702},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.8380000000000001,"5min":45.517000000000003,"15min":45.517000000000003},"last":1.282}]},{"osd":4,"last update":"Wed Feb 10 04:50:48 2021","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.2709999999999999,"15min":1.2869999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.55400000000000005},"max":{"1min":1.849,"5min":3.1309999999999998,"15min":6.1639999999999997},"last":1.4610000000000001},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.3240000000000001,"15min":1.393},"min":{"1min":0.68100000000000005,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.9279999999999999,"5min":2.5699999999999998,"15min":3.0209999999999999},"last":1.3149999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:51:01 2021","interfaces":[{"interface":"back","average":{"1min":1.5800000000000001,"5min":2.294,"15min":3.036},"min":{"1min":0.85099999999999998,"5min":0.81200000000000006,"15min":0.63500000000000001},"max":{"1min":5.8879999999999999,"5min":38.014000000000003,"15min":319.51999999999998},"last":1.3959999999999999},{"interface":"front","average":{"1min":1.427,"5min":2.1890000000000001,"15min":2.9300000000000002},"min":{"1min":0.80400000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":5.5750000000000002,"5min":37.896999999999998,"15min":319.22300000000001},"last":1.512}]},{"osd":7,"last update":"Wed Feb 10 04:50:23 2021","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.508,"15min":1.4810000000000001},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.752},"max":{"1min":2.0350000000000001,"5min":3.3479999999999999,"15min":4.6699999999999999},"last":1.784},{"interface":"front","average":{"1min":1.554,"5min":1.583,"15min":1.611},"min":{"1min":0.92500000000000004,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.2210000000000001,"5min":2.9279999999999999,"15min":3.1040000000000001},"last":1.484}]}]}]}} 2021-02-10T04:51:29.460 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2021-02-10T04:51:29.461 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:51:29.461 INFO:tasks.thrashosds.thrasher:Adding osd 2 2021-02-10T04:51:29.461 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd in 2 2021-02-10T04:51:29.713 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:51:29.869 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:29 smithi085 conmon[29319]: cluster 2021-02-10T04:51:29.870 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:29 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:51:29.870 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:29 smithi085 conmon[29319]: 04:51:28.416741+0000 mgr.y (mgr.14140) 1209 : cluster [DBG] pgmap v2002: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:51:29.870 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:29 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:51:29.870 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:29 smithi085 conmon[29319]: 02-10T04:51:29.070968+0000 mgr.y (mgr.14140 2021-02-10T04:51:29.870 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:29 smithi085 conmon[29319]: ) 1210 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:51:29.870 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:29 smithi085 conmon[35961]: cluster 2021-02-10T04:51:28.416741+0000 mgr.y (mgr.14140) 1209 : cluster [DBG] pgmap v2002: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:51:29.870 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:29 smithi085 conmon[35961]: audit 2021-02-10T04:51:29.070968+0000 mgr.y (mgr.14140) 1210 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:51:30.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:29 smithi164 conmon[29288]: cluster 2021-02-10T04:51:28.416741+0000 mgr.y (mgr.14140) 1209 : cluster 2021-02-10T04:51:30.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:29 smithi164 conmon[29288]: [DBG] pgmap v2002: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:51:30.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:29 smithi164 conmon[29288]: audit 2021-02-10T04:51:29.070968+0000 mgr.y (mgr.14140) 1210 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:51:30.523 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:51:30.523 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:51:30.669 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:51:30.690 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:51:30.711 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:51:30.732 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:51:30.755 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:51:30.777 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:51:30.807 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:51:30.831 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:51:30.833 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:51:30.833 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:51:30.833 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:51:30.833 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:51:30.834 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:51:30.834 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:51:30.834 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:51:30.834 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:51:30.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:30 smithi085 conmon[29319]: audit 2021-02-10T04:51:30.625228+0000 mon.c (mon 2021-02-10T04:51:30.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:30 smithi085 conmon[29319]: .1) 94 : audit [DBG] from='client.? 172.21.15.85:0/4251430884' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:51:30.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:30 smithi085 conmon[35961]: audit 2021-02-10T04:51:30.625228 2021-02-10T04:51:30.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:30 smithi085 conmon[35961]: +0000 mon.c (mon.1) 94 : audit [DBG] from='client.? 172.21.15.85:0/4251430884' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:51:31.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:30 smithi164 conmon[29288]: audit 2021-02-10T04:51:31.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:30 smithi164 conmon[29288]: 2021-02-10T04:51:31.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:30 smithi164 conmon[29288]: 2021-02-10T04:51:30 2021-02-10T04:51:31.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:30 smithi164 conmon[29288]: .625228+0000 mon.c (mon.1) 94 : audit [DBG] from='client.? 172.21.15.85:0/4251430884' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:51:31.932 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset noscrub 2021-02-10T04:51:31.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:31 smithi085 conmon[29319]: cluster 2021-02-10T04:51:30.417371+0000 mgr.y (mgr.14140) 1211 : 2021-02-10T04:51:31.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:31 smithi085 conmon[29319]: cluster [DBG] pgmap v2003: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:51:31.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:31 smithi085 conmon[35961]: cluster 2021-02-10T04:51:30.417371+0000 2021-02-10T04:51:31.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:31 smithi085 conmon[35961]: mgr.y (mgr.14140) 1211 : cluster [DBG] pgmap v2003: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:51:32.138 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:51:32.143 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:31 smithi164 conmon[33677]: debug 2021-02-10T04:51:31.699+0000 7fbec19a1700 -1 osd.4 1117 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:32.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:31 smithi164 conmon[29288]: cluster 2021-02-10T04:51:32.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:31 smithi164 conmon[29288]: 2021-02-10T04:51:32.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:31 smithi164 conmon[29288]: 2021 2021-02-10T04:51:32.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:31 smithi164 conmon[29288]: - 2021-02-10T04:51:32.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:31 smithi164 conmon[29288]: 02 2021-02-10T04:51:32.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:31 smithi164 conmon[29288]: - 2021-02-10T04:51:32.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:31 smithi164 conmon[29288]: 10 2021-02-10T04:51:32.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:31 smithi164 conmon[29288]: T04:51:30.417371+0000 2021-02-10T04:51:32.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:31 smithi164 conmon[29288]: mgr.y (mgr.14140) 1211 : cluster [DBG] pgmap v2003: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:51:33.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:32 smithi164 conmon[33677]: debug 2021-02-10T04:51:32.720+0000 7fbec19a1700 -1 osd.4 1117 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:33.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:32 smithi164 conmon[29288]: cluster 2021-02-10T04:51:31.700344+0000 osd.4 (osd.4) 2553 2021-02-10T04:51:33.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:32 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:33.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:32 smithi164 conmon[29288]: cluster 2021-02-10T04:51:31.700350+0000 osd.4 (osd.4) 2554 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:33.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:32 smithi164 conmon[29288]: cluster 2021-02-10T04:51:31.700356+0000 osd.4 (osd.4) 2555 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:33.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:32 smithi164 conmon[29288]: cluster 2021-02-10T04:51:31.700363+0000 osd.4 (osd.4) 2556 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:33.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:32 smithi085 conmon[35961]: cluster 2021-02-10T04:51:31.700344+0000 osd.4 (osd.4) 2553 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:33.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:32 smithi085 conmon[35961]: cluster 2021-02-10T04:51:31.700350+0000 osd.4 (osd.4) 2554 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:33.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:32 smithi085 conmon[35961]: cluster 2021-02-10T04:51:31.700356+0000 osd.4 (osd.4) 2555 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:33.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:32 smithi085 conmon[35961]: cluster 2021-02-10T04:51:31.700363+0000 osd.4 (osd.4) 2556 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:33.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:32 smithi085 conmon[29319]: cluster 2021-02-10T04:51:31.700344+0000 osd.4 (osd.4) 2553 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:33.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:32 smithi085 conmon[29319]: cluster 2021-02-10T04:51:31.700350+0000 osd.4 (osd.4) 2554 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:33.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:32 smithi085 conmon[29319]: cluster 2021-02-10T04:51:31.700356+0000 osd.4 (osd.4) 2555 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:33.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:32 smithi085 conmon[29319]: cluster 2021-02-10T04:51:31.700363+0000 osd.4 (osd.4) 2556 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:33.846 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.3 dump_historic_ops 2021-02-10T04:51:33.971 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[29319]: cluster 2021-02-10T04:51:33.971 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[29319]: 2021-02-10T04:51:32.417875 2021-02-10T04:51:33.971 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 1212 : cluster [DBG] pgmap v2004: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:51:33.971 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[29319]: cluster 2021-02-10T04:51:32.683063+0000 2021-02-10T04:51:33.971 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[29319]: mon.a (mon.0) 2316 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 427 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:51:33.971 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[29319]: cluster 2021-02-10T04:51:33.971 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[29319]: 2021-02-10T04:51:32.721633+0000 2021-02-10T04:51:33.971 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[29319]: osd.4 (osd.4) 2557 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:33.971 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[29319]: 2021-02-10T04:51:33.971 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:51:33.971 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[29319]: :51:32.721642+0000 osd.4 (osd. 2021-02-10T04:51:33.972 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[29319]: 4) 2558 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:33.972 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[29319]: cluster 2021-02-10T04:51:32 2021-02-10T04:51:33.972 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[29319]: .721648+0000 osd.4 (osd.4) 2559 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:33.972 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[29319]: 2021-02-10T04:51:33.972 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:51:33.972 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[29319]: -10T04:51:32.721661+0000 osd.4 (osd.4) 2560 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:33.972 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[29319]: 2021-02-10T04:51:33.972 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[35961]: cluster 2021-02-10T04:51: 2021-02-10T04:51:33.972 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[35961]: 32.417875+0000 mgr.y (mgr.14140) 1212 : cluster [DBG] pgmap v2004: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:51:33.972 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[35961]: cluster 2021-02-10T04:51 2021-02-10T04:51:33.972 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[35961]: :32.683063+0000 mon.a (mon.0) 2316 : cluster 2021-02-10T04:51:33.973 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[35961]: [WRN] Health check failed: 4 slow ops, oldest one blocked for 427 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:51:33.973 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[35961]: cluster 2021-02-10T04:51:33.973 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:51:33.973 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[35961]: 10T04:51:32.721633 2021-02-10T04:51:33.973 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2557 : cluster [WRN] 2021-02-10T04:51:33.973 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:33.973 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:51:33.973 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[35961]: 02-10T04:51: 2021-02-10T04:51:33.973 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[35961]: 32.721642+0000 osd.4 (osd. 2021-02-10T04:51:33.973 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[35961]: 4) 2558 : cluster [WRN] 2021-02-10T04:51:33.973 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:33.973 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:51:33.973 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[35961]: -10T04:51:32.721648 2021-02-10T04:51:33.974 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2559 : cluster 2021-02-10T04:51:33.974 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:33.974 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:51:33.974 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[35961]: T04:51:32.721661+0000 osd.4 ( 2021-02-10T04:51:33.974 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:33 smithi085 conmon[35961]: osd.4) 2560 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:34.114 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:51:34.143 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:33 smithi164 conmon[33677]: debug 2021-02-10T04:51:33.700+0000 7fbec19a1700 -1 osd.4 1117 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:34.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:33 smithi164 conmon[29288]: cluster 2021-02-10T04:51:34.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:33 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:51:34.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:33 smithi164 conmon[29288]: 51:32.417875+0000 mgr.y (mgr.14140) 1212 : cluster [DBG] pgmap v2004: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:51:34.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:33 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:51:34.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:33 smithi164 conmon[29288]: 02-10T04:51:32.683063+0000 mon.a (mon 2021-02-10T04:51:34.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:33 smithi164 conmon[29288]: .0) 2316 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 427 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:51:34.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:33 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:51:34.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:33 smithi164 conmon[29288]: -10T04:51:32.721633+0000 osd.4 (osd. 2021-02-10T04:51:34.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:33 smithi164 conmon[29288]: 4) 2557 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:34.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:33 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:51:34.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:33 smithi164 conmon[29288]: 02-10T04:51:32.721642+0000 osd.4 2021-02-10T04:51:34.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:33 smithi164 conmon[29288]: (osd.4) 2558 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:34.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:33 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:51:34.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:33 smithi164 conmon[29288]: -10T04:51:32.721648+0000 osd.4 ( 2021-02-10T04:51:34.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:33 smithi164 conmon[29288]: osd.4) 2559 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:34.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:33 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:51:34.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:33 smithi164 conmon[29288]: -02-10T04:51:32.721661+0000 2021-02-10T04:51:34.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:33 smithi164 conmon[29288]: osd.4 (osd.4) 2560 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:34.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:33 smithi164 conmon[29288]: 2021-02-10T04:51:34.925 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:34 smithi085 conmon[29319]: cluster 2021-02-10T04:51:33.701647 2021-02-10T04:51:34.925 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:34 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2561 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:34.925 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:34 smithi085 conmon[29319]: cluster 2021-02-10T04:51:33.701652+0000 osd.4 (osd.4) 2562 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:34.925 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:34 smithi085 conmon[29319]: cluster 2021-02-10T04:51:33.701656+0000 osd.4 (osd.4) 2563 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:34.925 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:34 smithi085 conmon[29319]: cluster 2021-02-10T04:51:33.701660+0000 osd.4 (osd.4) 2564 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:34.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:34 smithi085 conmon[35961]: cluster 2021-02-10T04:51:34.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:34 smithi085 conmon[35961]: 2021-02-10T04:51:34.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:34 smithi085 conmon[35961]: 2021-02-10T04:51:33. 2021-02-10T04:51:34.926 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:34 smithi085 conmon[35961]: 701647+0000 osd.4 (osd.4) 2561 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:34.926 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:34 smithi085 conmon[35961]: cluster 2021-02-10T04:51:34.926 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:34 smithi085 conmon[35961]: 2021-02-10T04:51:33.701652 2021-02-10T04:51:34.926 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:34 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2562 2021-02-10T04:51:34.926 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:34 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:34.926 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:34 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:51:34.926 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:34 smithi085 conmon[35961]: -02-10T04:51:33.701656+0000 2021-02-10T04:51:34.926 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:34 smithi085 conmon[35961]: osd.4 (osd.4) 2563 : cluster 2021-02-10T04:51:34.926 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:34 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:34.926 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:34 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:51:34.926 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:34 smithi085 conmon[35961]: -02-10T04:51:33.701660+0000 2021-02-10T04:51:34.926 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:34 smithi085 conmon[35961]: osd.4 (osd.4) 2564 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:34.927 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:34 smithi085 conmon[35961]: 2021-02-10T04:51:35.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:34 smithi164 conmon[33677]: debug 2021-02-10T04:51:34.665+0000 7fbec19a1700 -1 osd.4 1117 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:35.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:34 smithi164 conmon[29288]: cluster 2021-02-10T04:51:33.701647+0000 osd.4 2021-02-10T04:51:35.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:34 smithi164 conmon[29288]: (osd.4) 2561 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:35.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:34 smithi164 conmon[29288]: cluster 2021-02-10T04:51:33.701652+0000 osd.4 (osd.4) 2562 : 2021-02-10T04:51:35.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:34 smithi164 conmon[29288]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:35.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:34 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:51:35.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:34 smithi164 conmon[29288]: 02-10T04:51:33.701656+0000 osd.4 (osd.4) 2563 : cluster 2021-02-10T04:51:35.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:34 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:35.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:34 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:51:35.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:34 smithi164 conmon[29288]: 04:51:33.701660+0000 osd.4 (osd.4 2021-02-10T04:51:35.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:34 smithi164 conmon[29288]: ) 2564 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:35.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[29319]: debug 2021-02-10T04:51:35.325+0000 7f9e105d6700 -1 mon.a@0(leader).osd e1117 definitely_dead 0 2021-02-10T04:51:35.728 INFO:teuthology.orchestra.run.smithi085.stderr:marked in osd.2. 2021-02-10T04:51:35.993 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:51:36.020 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[29319]: cluster 2021-02-10T04:51:34.418550+0000 mgr.y 2021-02-10T04:51:36.021 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[29319]: (mgr.14140) 1213 : cluster [DBG] pgmap v2005: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:51:36.021 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[29319]: cluster 2021-02-10T04:51:34.665725+0000 osd.4 (osd.4) 2565 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:36.021 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[29319]: cluster 2021-02-10T04:51:34.665732+0000 2021-02-10T04:51:36.021 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[29319]: osd.4 (osd.4) 2566 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:36.021 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[29319]: cluster 2021-02-10T04:51:36.021 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[29319]: 2021-02-10T04:51:34.665737+0000 osd.4 (osd.4) 2567 : 2021-02-10T04:51:36.021 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:36.021 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:51:36.021 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[29319]: :51:34.665740+0000 osd.4 (osd.4) 2568 2021-02-10T04:51:36.021 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:36.022 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:51:36.022 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[29319]: 10T04:51:35.325553+0000 mon.a ( 2021-02-10T04:51:36.022 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[29319]: mon.0) 2317 : audit [DBG] from='client.? 172.21.15.85:0/1613390899' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2021-02-10T04:51:36.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: cluster 2021-02-10T04:51:36.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: 2021-02-10T04:51:36.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: 2021 2021-02-10T04:51:36.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: - 2021-02-10T04:51:36.023 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: 02 2021-02-10T04:51:36.023 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: - 2021-02-10T04:51:36.023 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: 10 2021-02-10T04:51:36.023 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: T 2021-02-10T04:51:36.023 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: 04 2021-02-10T04:51:36.023 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: : 2021-02-10T04:51:36.023 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: 51 2021-02-10T04:51:36.023 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: : 2021-02-10T04:51:36.023 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: 34 2021-02-10T04:51:36.023 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: . 2021-02-10T04:51:36.023 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: 418550 2021-02-10T04:51:36.024 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: +0000 2021-02-10T04:51:36.024 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: 2021-02-10T04:51:36.024 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: mgr.y 2021-02-10T04:51:36.024 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: ( 2021-02-10T04:51:36.024 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: mgr 2021-02-10T04:51:36.024 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: . 2021-02-10T04:51:36.024 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: 14140 2021-02-10T04:51:36.024 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: ) 2021-02-10T04:51:36.024 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: 1213 2021-02-10T04:51:36.024 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: : 2021-02-10T04:51:36.024 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: cluster 2021-02-10T04:51:36.024 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: 2021-02-10T04:51:36.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: [DBG] 2021-02-10T04:51:36.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: 2021-02-10T04:51:36.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: pgmap v2005: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:51:36.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: 2021-02-10T04:51:36.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: cluster 2021-02-10T04:51:36.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: 2021-02-10T04:51:36.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: 2021 2021-02-10T04:51:36.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: - 2021-02-10T04:51:36.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: 02 2021-02-10T04:51:36.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: - 2021-02-10T04:51:36.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: 10 2021-02-10T04:51:36.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: T 2021-02-10T04:51:36.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: 04 2021-02-10T04:51:36.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: : 2021-02-10T04:51:36.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: 51 2021-02-10T04:51:36.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: : 2021-02-10T04:51:36.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: 34.665725+0000 osd.4 (osd.4) 2565 : cluster 2021-02-10T04:51:36.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:36.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:51:36.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: T04:51:34.665732+0000 osd.4 (osd.4) 2566 : cluster 2021-02-10T04:51:36.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:36.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:51:36.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: 02-10T04:51: 2021-02-10T04:51:36.027 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: 34.665737+0000 osd.4 (osd 2021-02-10T04:51:36.027 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: .4) 2567 : cluster [WRN] 2021-02-10T04:51:36.027 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:36.027 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:51:36.027 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: 02-10T04:51:34.665740+0000 2021-02-10T04:51:36.027 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: osd.4 (osd.4) 2568 : cluster 2021-02-10T04:51:36.027 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:36.027 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:51:36.027 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: 02-10T04:51: 2021-02-10T04:51:36.027 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: 35.325553+0000 mon.a ( 2021-02-10T04:51:36.027 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: mon.0) 2317 : audit [DBG] 2021-02-10T04:51:36.027 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:35 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/1613390899' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2021-02-10T04:51:36.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:35 smithi164 conmon[33677]: debug 2021-02-10T04:51:35.654+0000 7fbec19a1700 -1 osd.4 1117 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:36.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:35 smithi164 conmon[29288]: cluster 2021-02-10T04:51:36.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:35 smithi164 conmon[29288]: 2021-02-10T04:51:36.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:35 smithi164 conmon[29288]: 2021 2021-02-10T04:51:36.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:35 smithi164 conmon[29288]: -02-10T04:51:34. 2021-02-10T04:51:36.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:35 smithi164 conmon[29288]: 418550+0000 mgr.y (mgr.14140) 1213 : cluster [DBG] pgmap v2005: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:51:36.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:35 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:51:36.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:35 smithi164 conmon[29288]: 04:51:34.665725+0000 osd.4 (osd.4) 2021-02-10T04:51:36.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:35 smithi164 conmon[29288]: 2565 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:36.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:35 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:51:36.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:35 smithi164 conmon[29288]: T04:51:34.665732+0000 osd.4 (osd. 2021-02-10T04:51:36.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:35 smithi164 conmon[29288]: 4) 2566 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:36.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:35 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:51:36.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:35 smithi164 conmon[29288]: -10T04:51:34.665737+0000 osd.4 ( 2021-02-10T04:51:36.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:35 smithi164 conmon[29288]: osd.4) 2567 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:36.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:35 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:51:36.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:35 smithi164 conmon[29288]: 02-10T04:51:34.665740+0000 osd.4 ( 2021-02-10T04:51:36.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:35 smithi164 conmon[29288]: osd.4) 2568 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:36.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:35 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:51:36.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:35 smithi164 conmon[29288]: 02-10T04:51:35.325553+0000 mon.a ( 2021-02-10T04:51:36.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:35 smithi164 conmon[29288]: mon.0) 2317 : audit [DBG] from='client.? 172.21.15.85:0/1613390899' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2021-02-10T04:51:36.220 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:51:37.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:36 smithi164 conmon[33677]: debug 2021-02-10T04:51:36.678+0000 7fbec19a1700 -1 osd.4 1118 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:37.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:36 smithi164 conmon[29288]: cluster 2021-02-10T04:51:35.655623+0000 osd.4 (osd. 2021-02-10T04:51:37.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:36 smithi164 conmon[29288]: 4) 2569 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:37.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:36 smithi164 conmon[29288]: cluster 2021-02-10T04:51:35.655632+0000 osd.4 (osd.4) 2570 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:37.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:36 smithi164 conmon[29288]: cluster 2021-02-10T04:51:35.655640+0000 osd.4 (osd.4) 2571 : cluster [WRN] 2021-02-10T04:51:37.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:36 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:37.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:36 smithi164 conmon[29288]: cluster 2021-02-10T04:51:35.655646 2021-02-10T04:51:37.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:36 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2572 : cluster [WRN] 2021-02-10T04:51:37.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:36 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:37.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:36 smithi164 conmon[29288]: audit 2021-02-10T04:51:35.709252 2021-02-10T04:51:37.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:36 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2318 : audit [INF] 2021-02-10T04:51:37.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:36 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/1613390899' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2021-02-10T04:51:37.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:36 smithi164 conmon[29288]: cluster 2021-02-10T04:51:35. 2021-02-10T04:51:37.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:36 smithi164 conmon[29288]: 709318+0000 mon.a (mon.0) 2319 : cluster 2021-02-10T04:51:37.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:36 smithi164 conmon[29288]: [DBG] osdmap e1118: 8 total, 8 up, 8 in 2021-02-10T04:51:37.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:36 smithi085 conmon[29319]: cluster 2021-02-10T04:51:35.655623+0000 osd.4 (osd.4) 2569 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:37.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:36 smithi085 conmon[29319]: cluster 2021-02-10T04:51:35.655632+0000 osd.4 (osd.4) 2570 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:37.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:36 smithi085 conmon[29319]: cluster 2021-02-10T04:51:35.655640+0000 osd.4 (osd.4) 2571 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:37.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:36 smithi085 conmon[29319]: cluster 2021-02-10T04:51:35.655646+0000 osd.4 (osd.4) 2572 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:37.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:36 smithi085 conmon[29319]: audit 2021-02-10T04:51:35.709252+0000 mon.a (mon.0) 2318 : audit [INF] from='client.? 172.21.15.85:0/1613390899' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2021-02-10T04:51:37.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:36 smithi085 conmon[29319]: cluster 2021-02-10T04:51:35.709318+0000 mon.a (mon.0) 2319 : cluster [DBG] osdmap e1118: 8 total, 8 up, 8 in 2021-02-10T04:51:37.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:36 smithi085 conmon[35961]: cluster 2021-02-10T04:51:35.655623+0000 2021-02-10T04:51:37.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:36 smithi085 conmon[35961]: osd.4 (osd.4) 2569 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:37.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:36 smithi085 conmon[35961]: cluster 2021-02-10T04:51: 2021-02-10T04:51:37.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:36 smithi085 conmon[35961]: 35.655632+0000 osd.4 (osd.4) 2570 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:37.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:36 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:51:37.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:36 smithi085 conmon[35961]: 02-10T04:51:35.655640+0000 osd.4 2021-02-10T04:51:37.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:36 smithi085 conmon[35961]: (osd.4) 2571 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520629+0000 currently started 2021-02-10T04:51:37.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:36 smithi085 conmon[35961]: 2021-02-10T04:51:37.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:36 smithi085 conmon[35961]: cluster 2021-02-10T04:51:37.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:36 smithi085 conmon[35961]: 2021-02-10T04:51:37.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:36 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:51:37.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:36 smithi085 conmon[35961]: :51:35.655646+0000 osd.4 (osd.4) 2021-02-10T04:51:37.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:36 smithi085 conmon[35961]: 2572 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520750+0000 currently delayed 2021-02-10T04:51:37.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:36 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:51:37.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:36 smithi085 conmon[35961]: 10T04:51:35.709252+0000 mon.a (mon. 2021-02-10T04:51:37.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:36 smithi085 conmon[35961]: 0) 2318 : audit [INF] from='client.? 172.21.15.85:0/1613390899' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2021-02-10T04:51:37.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:36 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:51:37.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:36 smithi085 conmon[35961]: 02-10T04:51:35.709318+0000 mon.a 2021-02-10T04:51:37.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:36 smithi085 conmon[35961]: (mon.0) 2319 : cluster [DBG] osdmap e1118: 8 total, 8 up, 8 in 2021-02-10T04:51:38.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[33677]: debug 2021-02-10T04:51:38.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[33677]: 2021-02-10T04:51:37.719+0000 7fbec19a1700 -1 osd.4 1119 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:38.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: cluster 2021-02-10T04:51:38.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: 2021-02-10T04:51:38.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: 2021 2021-02-10T04:51:38.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: - 2021-02-10T04:51:38.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: 02 2021-02-10T04:51:38.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: - 2021-02-10T04:51:38.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: 10 2021-02-10T04:51:38.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: T 2021-02-10T04:51:38.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: 04 2021-02-10T04:51:38.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: : 2021-02-10T04:51:38.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: 51 2021-02-10T04:51:38.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: : 2021-02-10T04:51:38.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: 36 2021-02-10T04:51:38.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: . 2021-02-10T04:51:38.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: 419038 2021-02-10T04:51:38.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: +0000 2021-02-10T04:51:38.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: 2021-02-10T04:51:38.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: mgr.y 2021-02-10T04:51:38.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: ( 2021-02-10T04:51:38.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: mgr 2021-02-10T04:51:38.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: . 2021-02-10T04:51:38.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: 14140 2021-02-10T04:51:38.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: ) 2021-02-10T04:51:38.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: 1214 : cluster [DBG] pgmap v2007: 188 pgs: 23 peering, 165 active+clean; 895 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:51:38.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: 2021-02-10T04:51:38.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:51:38.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: :51:36.678807+0000 osd.4 (osd.4 2021-02-10T04:51:38.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: ) 2573 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:38.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:51:38.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: -10T04:51:36.678815+0000 osd.4 ( 2021-02-10T04:51:38.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: osd.4) 2574 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:38.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:51:38.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: 02-10T04:51:36.825087+0000 mon.a 2021-02-10T04:51:38.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:37 smithi164 conmon[29288]: (mon.0) 2320 : cluster [DBG] osdmap e1119: 8 total, 8 up, 8 in 2021-02-10T04:51:38.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: cluster 2021-02-10T04:51:38.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 2021-02-10T04:51:38.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 2021 2021-02-10T04:51:38.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: - 2021-02-10T04:51:38.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 02 2021-02-10T04:51:38.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: - 2021-02-10T04:51:38.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 10 2021-02-10T04:51:38.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: T 2021-02-10T04:51:38.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 04 2021-02-10T04:51:38.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: : 2021-02-10T04:51:38.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 51 2021-02-10T04:51:38.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: : 2021-02-10T04:51:38.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 36 2021-02-10T04:51:38.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: . 2021-02-10T04:51:38.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 419038 2021-02-10T04:51:38.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: +0000 2021-02-10T04:51:38.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 2021-02-10T04:51:38.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: mgr.y 2021-02-10T04:51:38.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: ( 2021-02-10T04:51:38.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: mgr 2021-02-10T04:51:38.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: . 2021-02-10T04:51:38.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 14140 2021-02-10T04:51:38.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: ) 2021-02-10T04:51:38.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 1214 2021-02-10T04:51:38.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: : 2021-02-10T04:51:38.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: cluster 2021-02-10T04:51:38.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 2021-02-10T04:51:38.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: [DBG] 2021-02-10T04:51:38.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 2021-02-10T04:51:38.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: pgmap v2007: 188 pgs: 23 peering, 165 active+clean; 895 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:51:38.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 2021-02-10T04:51:38.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: cluster 2021-02-10T04:51:38.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 2021-02-10T04:51:38.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 2021 2021-02-10T04:51:38.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: - 2021-02-10T04:51:38.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 02 2021-02-10T04:51:38.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: - 2021-02-10T04:51:38.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 10 2021-02-10T04:51:38.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: T 2021-02-10T04:51:38.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 04 2021-02-10T04:51:38.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: : 2021-02-10T04:51:38.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 51 2021-02-10T04:51:38.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: : 2021-02-10T04:51:38.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 36 2021-02-10T04:51:38.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: . 2021-02-10T04:51:38.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 678807 2021-02-10T04:51:38.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: +0000 2021-02-10T04:51:38.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 2021-02-10T04:51:38.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: osd.4 2021-02-10T04:51:38.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: ( 2021-02-10T04:51:38.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: osd 2021-02-10T04:51:38.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: . 2021-02-10T04:51:38.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 4 2021-02-10T04:51:38.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: ) 2021-02-10T04:51:38.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 2573 2021-02-10T04:51:38.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: : 2021-02-10T04:51:38.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: cluster 2021-02-10T04:51:38.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 2021-02-10T04:51:38.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: [WRN] 2021-02-10T04:51:38.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 2021-02-10T04:51:38.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:38.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 2021-02-10T04:51:38.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: cluster 2021-02-10T04:51:38.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 2021-02-10T04:51:38.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 2021 2021-02-10T04:51:38.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: - 2021-02-10T04:51:38.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 02 2021-02-10T04:51:38.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: - 2021-02-10T04:51:38.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 10T04:51:36. 2021-02-10T04:51:38.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 678815+0000 osd.4 (osd.4) 2574 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:38.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:51:38.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 10T04:51:36.825087+0000 mon.a (mon.0) 2320 : cluster [DBG] osdmap e1119: 8 total, 8 up, 8 in 2021-02-10T04:51:38.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[29319]: 2021-02-10T04:51:38.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: cluster 2021-02-10T04:51:38.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 2021-02-10T04:51:38.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 2021 2021-02-10T04:51:38.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: - 2021-02-10T04:51:38.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 02 2021-02-10T04:51:38.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: - 2021-02-10T04:51:38.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 10 2021-02-10T04:51:38.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: T 2021-02-10T04:51:38.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 04 2021-02-10T04:51:38.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: : 2021-02-10T04:51:38.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 51 2021-02-10T04:51:38.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: : 2021-02-10T04:51:38.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 36 2021-02-10T04:51:38.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: . 2021-02-10T04:51:38.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 419038 2021-02-10T04:51:38.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: +0000 2021-02-10T04:51:38.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 2021-02-10T04:51:38.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: mgr.y 2021-02-10T04:51:38.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: ( 2021-02-10T04:51:38.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: mgr 2021-02-10T04:51:38.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: . 2021-02-10T04:51:38.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 14140 2021-02-10T04:51:38.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: ) 2021-02-10T04:51:38.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 1214 2021-02-10T04:51:38.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: : 2021-02-10T04:51:38.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: cluster 2021-02-10T04:51:38.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 2021-02-10T04:51:38.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: [DBG] 2021-02-10T04:51:38.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 2021-02-10T04:51:38.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: pgmap v2007: 188 pgs: 23 peering, 165 active+clean; 895 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:51:38.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 2021-02-10T04:51:38.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: cluster 2021-02-10T04:51:38.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 2021-02-10T04:51:38.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 2021 2021-02-10T04:51:38.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: - 2021-02-10T04:51:38.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 02 2021-02-10T04:51:38.201 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: - 2021-02-10T04:51:38.201 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 10 2021-02-10T04:51:38.201 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: T 2021-02-10T04:51:38.201 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 04 2021-02-10T04:51:38.201 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: : 2021-02-10T04:51:38.201 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 51 2021-02-10T04:51:38.201 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: : 2021-02-10T04:51:38.201 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 36 2021-02-10T04:51:38.201 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: . 2021-02-10T04:51:38.201 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 678807 2021-02-10T04:51:38.201 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: +0000 2021-02-10T04:51:38.201 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 2021-02-10T04:51:38.201 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: osd.4 2021-02-10T04:51:38.201 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: ( 2021-02-10T04:51:38.202 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: osd 2021-02-10T04:51:38.202 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: . 2021-02-10T04:51:38.202 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 4 2021-02-10T04:51:38.202 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: ) 2021-02-10T04:51:38.202 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 2573 2021-02-10T04:51:38.202 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: : 2021-02-10T04:51:38.202 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: cluster 2021-02-10T04:51:38.202 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 2021-02-10T04:51:38.202 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: [WRN] 2021-02-10T04:51:38.202 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 2021-02-10T04:51:38.202 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:38.202 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 2021-02-10T04:51:38.202 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: cluster 2021-02-10T04:51:38.203 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 2021-02-10T04:51:38.203 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 2021 2021-02-10T04:51:38.203 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: - 2021-02-10T04:51:38.203 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 02 2021-02-10T04:51:38.203 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: - 2021-02-10T04:51:38.203 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 10 2021-02-10T04:51:38.203 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: T 2021-02-10T04:51:38.203 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 04 2021-02-10T04:51:38.203 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: : 2021-02-10T04:51:38.203 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 51 2021-02-10T04:51:38.203 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: : 2021-02-10T04:51:38.203 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 36 2021-02-10T04:51:38.203 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: . 2021-02-10T04:51:38.203 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 678815 2021-02-10T04:51:38.204 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: +0000 2021-02-10T04:51:38.204 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 2021-02-10T04:51:38.204 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: osd.4 2021-02-10T04:51:38.204 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: ( 2021-02-10T04:51:38.204 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: osd 2021-02-10T04:51:38.204 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: . 2021-02-10T04:51:38.204 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 4 2021-02-10T04:51:38.204 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: ) 2021-02-10T04:51:38.204 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 2574 2021-02-10T04:51:38.204 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: : 2021-02-10T04:51:38.204 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: cluster 2021-02-10T04:51:38.204 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 2021-02-10T04:51:38.204 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: [WRN] 2021-02-10T04:51:38.205 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 2021-02-10T04:51:38.205 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:38.205 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 2021-02-10T04:51:38.205 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: cluster 2021-02-10T04:51:38.205 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 2021-02-10T04:51:38.205 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 2021 2021-02-10T04:51:38.205 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: - 2021-02-10T04:51:38.205 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 02 2021-02-10T04:51:38.205 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: - 2021-02-10T04:51:38.205 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 10 2021-02-10T04:51:38.205 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: T 2021-02-10T04:51:38.205 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 04 2021-02-10T04:51:38.205 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: : 2021-02-10T04:51:38.205 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 51 2021-02-10T04:51:38.206 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: : 2021-02-10T04:51:38.206 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 36 2021-02-10T04:51:38.206 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: . 2021-02-10T04:51:38.206 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 825087 2021-02-10T04:51:38.206 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: +0000 2021-02-10T04:51:38.206 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 2021-02-10T04:51:38.206 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: mon.a 2021-02-10T04:51:38.206 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: ( 2021-02-10T04:51:38.206 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: mon 2021-02-10T04:51:38.206 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: . 2021-02-10T04:51:38.206 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 0 2021-02-10T04:51:38.206 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: ) 2021-02-10T04:51:38.206 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 2320 2021-02-10T04:51:38.207 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: : 2021-02-10T04:51:38.207 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: cluster 2021-02-10T04:51:38.207 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 2021-02-10T04:51:38.207 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: [DBG] 2021-02-10T04:51:38.207 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 2021-02-10T04:51:38.207 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: osdmap e1119: 8 total, 8 up, 8 in 2021-02-10T04:51:38.207 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:37 smithi085 conmon[35961]: 2021-02-10T04:51:38.861 INFO:tasks.thrashosds.thrasher:Added osd 2 2021-02-10T04:51:39.031 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[29319]: cluster 2021-02-10T04:51:37.720094+0000 2021-02-10T04:51:39.031 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[29319]: osd.4 (osd.4) 2575 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:39.031 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[29319]: cluster 2021-02-10T04:51:37.720104+0000 osd.4 (osd.4) 2576 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:39.031 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[29319]: cluster 2021-02-10T04:51:37.824369+0000 mon.a (mon.0) 2321 : cluster [DBG] osdmap e1120: 8 total, 8 up, 8 in 2021-02-10T04:51:39.031 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[29319]: cluster 2021-02-10T04:51:38.081982+0000 osd.0 (osd.0) 164 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:51:39.031 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[29319]: cluster 2021-02-10T04:51:38.084699+0000 osd.0 (osd.0) 165 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:51:39.031 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[29319]: audit 2021-02-10T04:51:38.210196+0000 mon.a (mon.0) 2322 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:51:39.032 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[29319]: audit 2021-02-10T04:51:38.245150+0000 mon.a (mon.0) 2323 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:51:39.032 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:51:39.032 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[29319]: -02-10T04:51:38.483955+0000 mon.a (mon.0) 2021-02-10T04:51:39.032 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[29319]: 2324 : cluster [DBG] osdmap e1121: 8 total, 8 up, 8 in 2021-02-10T04:51:39.032 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: cluster 2021-02-10T04:51:39.032 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: 2021-02-10T04:51:39.032 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: 2021 2021-02-10T04:51:39.032 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: - 2021-02-10T04:51:39.032 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: 02 2021-02-10T04:51:39.033 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: - 2021-02-10T04:51:39.033 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: 10 2021-02-10T04:51:39.033 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: T 2021-02-10T04:51:39.033 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: 04 2021-02-10T04:51:39.033 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: : 2021-02-10T04:51:39.033 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: 51 2021-02-10T04:51:39.033 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: : 2021-02-10T04:51:39.033 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: 37 2021-02-10T04:51:39.033 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: . 2021-02-10T04:51:39.033 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: 720094 2021-02-10T04:51:39.033 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: +0000 2021-02-10T04:51:39.033 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: 2021-02-10T04:51:39.033 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: osd.4 2021-02-10T04:51:39.034 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: ( 2021-02-10T04:51:39.034 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: osd 2021-02-10T04:51:39.034 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: . 2021-02-10T04:51:39.034 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: 4 2021-02-10T04:51:39.034 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: ) 2021-02-10T04:51:39.034 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: 2575 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:39.034 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:51:39.034 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: T04:51:37.720104+0000 osd.4 (osd. 2021-02-10T04:51:39.034 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: 4) 2576 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:39.034 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:51:39.034 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: -10T04:51:37.824369+0000 mon.a 2021-02-10T04:51:39.034 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: (mon.0) 2321 : cluster [DBG] 2021-02-10T04:51:39.034 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: osdmap e1120: 8 total, 8 up, 8 in 2021-02-10T04:51:39.035 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:51:39.035 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: 04:51:38.081982+0000 osd.0 (osd. 2021-02-10T04:51:39.035 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: 0) 164 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:51:39.035 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: cluster 2021-02-10T04:51:39.035 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: 2021-02-10T04:51:38.084699+0000 osd.0 ( 2021-02-10T04:51:39.035 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: osd.0) 165 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:51:39.035 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: audit 2021-02-10T04:51:39.035 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: 2021-02-10T04:51:38 2021-02-10T04:51:39.035 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: .210196+0000 mon.a (mon.0) 2322 : audit [INF] 2021-02-10T04:51:39.035 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:51:39.035 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:51:39.035 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: -10T04:51:38.245150+0000 mon.a (mon.0) 2323 : 2021-02-10T04:51:39.035 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:51:39.036 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: cluster 2021-02-10T04:51: 2021-02-10T04:51:39.036 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: 38.483955+0000 mon.a (mon.0) 2324 2021-02-10T04:51:39.036 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:38 smithi085 conmon[35961]: : cluster [DBG] osdmap e1121: 8 total, 8 up, 8 in 2021-02-10T04:51:39.084 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:38 smithi164 conmon[33677]: debug 2021-02-10T04:51:39.084 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:38 smithi164 conmon[33677]: 2021-02-10T04:51:38.727+0000 7fbec19a1700 -1 osd.4 1121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:39.084 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:38 smithi164 conmon[33677]: 2021-02-10T04:51:39.084 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:38 smithi164 conmon[29288]: cluster 2021-02-10T04:51:37.720094+0000 osd.4 (osd 2021-02-10T04:51:39.084 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:38 smithi164 conmon[29288]: .4) 2575 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:39.084 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:38 smithi164 conmon[29288]: cluster 2021-02-10T04:51:37.720104+0000 osd.4 (osd 2021-02-10T04:51:39.084 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:38 smithi164 conmon[29288]: .4) 2576 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:39.084 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:38 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:51:39.084 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:38 smithi164 conmon[29288]: T04:51:37.824369+0000 mon.a (mon.0 2021-02-10T04:51:39.085 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:38 smithi164 conmon[29288]: ) 2321 : cluster [DBG] osdmap e1120: 8 total, 8 up, 8 in 2021-02-10T04:51:39.085 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:38 smithi164 conmon[29288]: cluster 2021-02-10T04:51:38. 2021-02-10T04:51:39.085 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:38 smithi164 conmon[29288]: 081982+0000 osd.0 (osd.0) 164 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:51:39.085 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:38 smithi164 conmon[29288]: 2021-02-10T04:51:39.085 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:38 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:51:39.085 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:38 smithi164 conmon[29288]: 02-10T04:51:38.084699+0000 osd.0 (osd.0) 2021-02-10T04:51:39.085 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:38 smithi164 conmon[29288]: 165 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:51:39.085 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:38 smithi164 conmon[29288]: audit 2021-02-10T04:51:38.210196+0000 mon.a 2021-02-10T04:51:39.085 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:38 smithi164 conmon[29288]: (mon.0) 2322 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:51:39.085 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:38 smithi164 conmon[29288]: audit 2021-02-10T04:51: 2021-02-10T04:51:39.085 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:38 smithi164 conmon[29288]: 38.245150+0000 mon.a (mon.0) 2021-02-10T04:51:39.085 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:38 smithi164 conmon[29288]: 2323 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:51:39.085 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:38 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:51:39.086 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:38 smithi164 conmon[29288]: -10T04:51:38.483955 2021-02-10T04:51:39.086 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:38 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2324 2021-02-10T04:51:39.086 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:38 smithi164 conmon[29288]: : cluster [DBG] osdmap e1121: 8 total, 8 up, 8 in 2021-02-10T04:51:39.086 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:51:39 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:51:39.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:39 smithi085 conmon[29319]: cluster 2021-02-10T04:51:38.419474+0000 mgr.y (mgr. 2021-02-10T04:51:39.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:39 smithi085 conmon[29319]: 14140) 1215 : cluster [DBG] pgmap v2010: 188 pgs: 1 active+recovering+undersized+remapped, 6 remapped+peering, 1 active+recovering, 23 peering, 157 active+clean; 895 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 10/2151 objects misplaced (0.465%) 2021-02-10T04:51:39.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:39 smithi085 conmon[29319]: cluster 2021-02-10T04:51:38.728207+0000 osd.4 (osd.4) 2577 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:39.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:39 smithi085 conmon[29319]: cluster 2021-02-10T04:51:38.728216+0000 osd.4 (osd.4) 2578 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:39 smithi085 conmon[29319]: cluster 2021-02-10T04:51:38.826493+0000 mon.a (mon.0) 2325 : cluster [WRN] Health check update: 2 slow ops, oldest one blocked for 432 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:51:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:39 smithi085 conmon[29319]: audit 2021-02-10T04:51:39.083013+0000 mgr.y (mgr.14140) 1216 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:51:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:39 smithi085 conmon[29319]: cluster 2021-02-10T04:51:39.501083+0000 mon.a (mon.0) 2326 : cluster [DBG] osdmap e1122: 8 total, 8 up, 8 in 2021-02-10T04:51:39.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:39 smithi085 conmon[35961]: cluster 2021-02-10T04:51:38.419474+0000 mgr.y (mgr.14140) 1215 : cluster 2021-02-10T04:51:39.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:39 smithi085 conmon[35961]: [DBG] pgmap v2010: 188 pgs: 1 active+recovering+undersized+remapped, 6 remapped+peering, 1 active+recovering, 23 peering, 157 active+clean; 895 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 10/2151 objects misplaced (0.465%) 2021-02-10T04:51:39.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:39 smithi085 conmon[35961]: cluster 2021-02-10T04:51:38.728207+0000 osd.4 (osd.4) 2577 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:39.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:39 smithi085 conmon[35961]: cluster 2021-02-10T04:51:38.728216+0000 osd.4 (osd.4) 2578 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:39.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:39 smithi085 conmon[35961]: cluster 2021-02-10T04:51:38.826493+0000 mon.a (mon.0) 2325 : cluster [WRN] Health check update: 2 slow ops, oldest one blocked for 432 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:51:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:39 smithi085 conmon[35961]: audit 2021-02-10T04:51:39.083013+0000 mgr.y (mgr.14140) 1216 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:51:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:39 smithi085 conmon[35961]: cluster 2021-02-10T04:51:39.501083+0000 mon.a (mon.0) 2326 : cluster [DBG] osdmap e1122: 8 total, 8 up, 8 in 2021-02-10T04:51:40.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:39 smithi164 conmon[33677]: debug 2021-02-10T04:51:39.715+0000 7fbec19a1700 -1 osd.4 1122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:39 smithi164 conmon[29288]: cluster 2021-02-10T04:51:38.419474+0000 mgr.y (mgr.14140) 1215 2021-02-10T04:51:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:39 smithi164 conmon[29288]: : cluster [DBG] pgmap v2010: 188 pgs: 1 active+recovering+undersized+remapped, 6 remapped+peering, 1 active+recovering, 23 peering, 157 active+clean; 895 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 10/2151 objects misplaced (0.465%) 2021-02-10T04:51:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:39 smithi164 conmon[29288]: cluster 2021-02-10T04:51:38.728207+0000 osd.4 (osd.4) 2577 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:39 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:51:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:39 smithi164 conmon[29288]: T04:51:38.728216+0000 osd.4 (osd.4) 2578 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:39 smithi164 conmon[29288]: cluster 2021-02-10T04:51:38 2021-02-10T04:51:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:39 smithi164 conmon[29288]: .826493+0000 mon.a (mon.0) 2325 : cluster 2021-02-10T04:51:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:39 smithi164 conmon[29288]: [WRN] Health check update: 2 slow ops, oldest one blocked for 432 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:51:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:39 smithi164 conmon[29288]: audit 2021-02-10T04:51: 2021-02-10T04:51:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:39 smithi164 conmon[29288]: 39.083013+0000 mgr.y (mgr.14140) 1216 : 2021-02-10T04:51:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:39 smithi164 conmon[29288]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:51:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:39 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:51:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:39 smithi164 conmon[29288]: :51:39.501083+0000 mon.a (mon.0) 2021-02-10T04:51:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:39 smithi164 conmon[29288]: 2326 : cluster [DBG] osdmap e1122: 8 total, 8 up, 8 in 2021-02-10T04:51:41.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:40 smithi164 conmon[33677]: debug 2021-02-10T04:51:40.693+0000 7fbec19a1700 -1 osd.4 1123 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:41.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:40 smithi164 conmon[29288]: cluster 2021-02-10T04:51:39.715957+0000 osd.4 (osd.4) 2579 2021-02-10T04:51:41.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:40 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:41.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:40 smithi164 conmon[29288]: cluster 2021-02-10T04:51:39.715963+0000 osd.4 (osd.4) 2580 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:41.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:40 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:51:41.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:40 smithi164 conmon[29288]: -02-10T04:51:40.505304+0000 mon.a (mon.0) 2327 : cluster [DBG] osdmap e1123: 8 total, 8 up, 8 in 2021-02-10T04:51:41.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:40 smithi085 conmon[29319]: cluster 2021-02-10T04:51:39.715957+0000 osd.4 2021-02-10T04:51:41.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:40 smithi085 conmon[29319]: (osd.4) 2579 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:41.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:40 smithi085 conmon[29319]: cluster 2021-02-10T04:51:39 2021-02-10T04:51:41.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:40 smithi085 conmon[29319]: .715963+0000 osd.4 (osd.4) 2580 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:41.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:40 smithi085 conmon[29319]: cluster 2021-02-10T04:51:41.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:40 smithi085 conmon[29319]: 2021-02-10T04:51:40.505304+0000 mon.a (mon.0) 2021-02-10T04:51:41.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:40 smithi085 conmon[29319]: 2327 : cluster [DBG] osdmap e1123: 8 total, 8 up, 8 in 2021-02-10T04:51:41.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:40 smithi085 conmon[35961]: cluster 2021-02-10T04:51:39.715957+0000 2021-02-10T04:51:41.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:40 smithi085 conmon[35961]: osd.4 (osd.4) 2579 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:41.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:40 smithi085 conmon[35961]: cluster 2021-02-10T04:51:39.715963 2021-02-10T04:51:41.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:40 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2580 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:41.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:40 smithi085 conmon[35961]: cluster 2021-02-10T04:51:40.505304 2021-02-10T04:51:41.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:40 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2327 : cluster [DBG] 2021-02-10T04:51:41.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:40 smithi085 conmon[35961]: osdmap e1123: 8 total, 8 up, 8 in 2021-02-10T04:51:41.395 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:51:41.395 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 20, 2021-02-10T04:51:41.395 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 600, 2021-02-10T04:51:41.395 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [ 2021-02-10T04:51:41.395 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.395 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:3431 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94187755107840 gen 3] snapc 0=[] ondisk+write+known_if_redirected e1112)", 2021-02-10T04:51:41.395 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:02.988562+0000", 2021-02-10T04:51:41.396 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 38.404256611999998, 2021-02-10T04:51:41.396 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9928285949999998, 2021-02-10T04:51:41.396 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:51:41.396 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:51:41.396 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:51:41.396 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:51:41.396 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:51:41.396 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 3431 2021-02-10T04:51:41.396 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.396 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:51:41.396 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.396 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:51:41.397 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988562+0000", 2021-02-10T04:51:41.397 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:51:41.397 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.397 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.397 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:51:41.397 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988562+0000", 2021-02-10T04:51:41.397 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.1761000000000001e-05 2021-02-10T04:51:41.397 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.397 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.397 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:51:41.397 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988583+0000", 2021-02-10T04:51:41.397 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.5929999999999996e-06 2021-02-10T04:51:41.397 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.397 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.398 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:51:41.398 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988589+0000", 2021-02-10T04:51:41.398 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.8294000000000001e-05 2021-02-10T04:51:41.398 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.398 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.398 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:51:41.398 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988607+0000", 2021-02-10T04:51:41.398 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.6950000000000003e-06 2021-02-10T04:51:41.398 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.398 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.398 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:51:41.398 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988614+0000", 2021-02-10T04:51:41.399 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.5459999999999999e-05 2021-02-10T04:51:41.399 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.399 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.399 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:51:41.399 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988689+0000", 2021-02-10T04:51:41.399 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.0969e-05 2021-02-10T04:51:41.399 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.399 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.399 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for readable", 2021-02-10T04:51:41.399 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988730+0000", 2021-02-10T04:51:41.399 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9912170979999999 2021-02-10T04:51:41.399 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.400 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.400 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:51:41.400 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.979947+0000", 2021-02-10T04:51:41.400 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00020239699999999999 2021-02-10T04:51:41.400 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.400 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.400 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:51:41.400 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.980150+0000", 2021-02-10T04:51:41.400 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001240328 2021-02-10T04:51:41.400 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.400 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.400 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:51:41.400 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.981390+0000", 2021-02-10T04:51:41.400 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9928285949999998 2021-02-10T04:51:41.401 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.401 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:51:41.401 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.401 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.401 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.401 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:3432 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 94187770080640 gen 4] snapc 0=[] ondisk+write+known_if_redirected e1112)", 2021-02-10T04:51:41.401 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:02.988669+0000", 2021-02-10T04:51:41.401 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 38.404148902000003, 2021-02-10T04:51:41.401 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0090249269999996, 2021-02-10T04:51:41.401 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:51:41.401 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:51:41.401 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:51:41.401 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:51:41.402 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:51:41.402 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 3432 2021-02-10T04:51:41.402 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.402 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:51:41.402 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.402 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:51:41.402 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988669+0000", 2021-02-10T04:51:41.402 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:51:41.402 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.402 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.402 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:51:41.403 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988669+0000", 2021-02-10T04:51:41.403 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.0630999999999999e-05 2021-02-10T04:51:41.403 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.403 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.403 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:51:41.403 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988700+0000", 2021-02-10T04:51:41.403 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.3149999999999997e-06 2021-02-10T04:51:41.403 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.403 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.403 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:51:41.403 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988706+0000", 2021-02-10T04:51:41.403 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0995000000000001e-05 2021-02-10T04:51:41.404 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.404 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.404 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:51:41.404 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988717+0000", 2021-02-10T04:51:41.404 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.3070000000000002e-06 2021-02-10T04:51:41.404 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.404 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.404 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:51:41.404 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988722+0000", 2021-02-10T04:51:41.404 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.5676e-05 2021-02-10T04:51:41.404 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.404 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.405 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:51:41.405 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988778+0000", 2021-02-10T04:51:41.405 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.2759999999999998e-05 2021-02-10T04:51:41.405 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.405 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.405 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for readable", 2021-02-10T04:51:41.405 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988811+0000", 2021-02-10T04:51:41.405 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0076071410000003 2021-02-10T04:51:41.405 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.405 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.405 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:51:41.405 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.996418+0000", 2021-02-10T04:51:41.405 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000159133 2021-02-10T04:51:41.406 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.406 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.406 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:51:41.406 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.996577+0000", 2021-02-10T04:51:41.406 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001116969 2021-02-10T04:51:41.406 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.406 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.406 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:51:41.406 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.997694+0000", 2021-02-10T04:51:41.406 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0090249269999996 2021-02-10T04:51:41.406 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.407 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:51:41.407 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.407 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.407 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.407 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:3435 5.14 5:2b04a3e9:::notify.6:head [watch ping cookie 94187770080640 gen 4] snapc 0=[] ondisk+write+known_if_redirected e1113)", 2021-02-10T04:51:41.407 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:07.988123+0000", 2021-02-10T04:51:41.407 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 33.404694751999997, 2021-02-10T04:51:41.407 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0089928860000000003, 2021-02-10T04:51:41.407 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:51:41.407 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:51:41.407 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:51:41.407 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:51:41.408 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:51:41.408 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 3435 2021-02-10T04:51:41.408 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.408 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:51:41.408 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.408 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:51:41.408 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.988123+0000", 2021-02-10T04:51:41.408 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:51:41.408 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.408 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.408 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:51:41.408 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.988123+0000", 2021-02-10T04:51:41.408 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.8739999999999999e-05 2021-02-10T04:51:41.408 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.409 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.409 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:51:41.409 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.988152+0000", 2021-02-10T04:51:41.409 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.4450000000000001e-06 2021-02-10T04:51:41.409 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.409 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.409 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:51:41.409 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.988159+0000", 2021-02-10T04:51:41.409 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0798999999999999e-05 2021-02-10T04:51:41.409 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.409 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.409 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:51:41.409 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.988169+0000", 2021-02-10T04:51:41.410 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.2630000000000003e-06 2021-02-10T04:51:41.410 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.410 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.410 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:51:41.410 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.988175+0000", 2021-02-10T04:51:41.410 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.1480000000000002e-05 2021-02-10T04:51:41.410 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.410 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.410 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:51:41.410 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.988226+0000", 2021-02-10T04:51:41.410 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.1520999999999997e-05 2021-02-10T04:51:41.410 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.411 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.411 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for readable", 2021-02-10T04:51:41.411 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.988278+0000", 2021-02-10T04:51:41.411 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0086211429999999995 2021-02-10T04:51:41.411 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.411 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.411 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:51:41.411 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.996899+0000", 2021-02-10T04:51:41.411 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.0494000000000006e-05 2021-02-10T04:51:41.411 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.411 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.411 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:51:41.412 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.996979+0000", 2021-02-10T04:51:41.412 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00013700100000000001 2021-02-10T04:51:41.412 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.412 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.412 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:51:41.412 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.997116+0000", 2021-02-10T04:51:41.412 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0089928860000000003 2021-02-10T04:51:41.412 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.412 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:51:41.412 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.412 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.412 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.412 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:3481 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 94187770080640 gen 5] snapc 0=[] ondisk+write+known_if_redirected e1118)", 2021-02-10T04:51:41.413 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:35.712201+0000", 2021-02-10T04:51:41.413 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 5.6806171059999997, 2021-02-10T04:51:41.413 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.172105363, 2021-02-10T04:51:41.413 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:51:41.413 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:51:41.413 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:51:41.413 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:51:41.413 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:51:41.413 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 3481 2021-02-10T04:51:41.413 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.413 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:51:41.413 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.413 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:51:41.413 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:35.712201+0000", 2021-02-10T04:51:41.414 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:51:41.414 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.414 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.414 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:51:41.414 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:35.712201+0000", 2021-02-10T04:51:41.414 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.2008000000000003e-05 2021-02-10T04:51:41.414 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.414 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.414 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:51:41.414 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:35.712233+0000", 2021-02-10T04:51:41.414 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.0690000000000002e-06 2021-02-10T04:51:41.414 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.415 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.415 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:51:41.415 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:35.712241+0000", 2021-02-10T04:51:41.415 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2391999999999999e-05 2021-02-10T04:51:41.415 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.415 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.415 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:51:41.415 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:35.712253+0000", 2021-02-10T04:51:41.415 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.5499999999999997e-06 2021-02-10T04:51:41.415 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.415 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.415 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:51:41.416 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:35.712261+0000", 2021-02-10T04:51:41.416 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0015999999999997e-05 2021-02-10T04:51:41.416 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.416 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.416 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:51:41.416 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:35.712311+0000", 2021-02-10T04:51:41.416 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0231000000000001e-05 2021-02-10T04:51:41.416 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.416 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.416 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op must wait for map", 2021-02-10T04:51:41.416 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:35.712321+0000", 2021-02-10T04:51:41.416 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.02825248 2021-02-10T04:51:41.416 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.416 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.417 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:51:41.417 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:35.740574+0000", 2021-02-10T04:51:41.417 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0251999999999999e-05 2021-02-10T04:51:41.417 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.417 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.417 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for peered", 2021-02-10T04:51:41.417 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:35.740584+0000", 2021-02-10T04:51:41.417 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.129581339 2021-02-10T04:51:41.417 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.417 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.417 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:51:41.417 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:36.870165+0000", 2021-02-10T04:51:41.417 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00034935599999999999 2021-02-10T04:51:41.417 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.418 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.418 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for degraded object", 2021-02-10T04:51:41.418 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:36.870515+0000", 2021-02-10T04:51:41.418 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.012278183999999999 2021-02-10T04:51:41.418 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.418 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.418 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:51:41.418 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:36.882793+0000", 2021-02-10T04:51:41.418 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000120188 2021-02-10T04:51:41.418 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.418 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.418 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:51:41.418 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:36.882913+0000", 2021-02-10T04:51:41.419 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0013932980000000001 2021-02-10T04:51:41.419 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.419 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.419 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:51:41.419 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:36.884306+0000", 2021-02-10T04:51:41.419 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.172105363 2021-02-10T04:51:41.419 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.419 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:51:41.419 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.419 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.419 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.419 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.14 1121/1119 [PushReplyOp(7:2dcefd69:::smithi08575397-28:333)])", 2021-02-10T04:51:41.419 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:39.199259+0000", 2021-02-10T04:51:41.420 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 2.1935589590000002, 2021-02-10T04:51:41.420 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0030838609999999998, 2021-02-10T04:51:41.420 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:51:41.420 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:51:41.420 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:51:41.420 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.420 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:51:41.420 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.199259+0000", 2021-02-10T04:51:41.420 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:51:41.420 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.420 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.420 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:51:41.420 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.199259+0000", 2021-02-10T04:51:41.420 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.2369999999999997e-05 2021-02-10T04:51:41.421 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.421 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.421 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:51:41.421 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.199292+0000", 2021-02-10T04:51:41.421 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.072e-06 2021-02-10T04:51:41.421 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.421 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.421 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:51:41.421 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.199300+0000", 2021-02-10T04:51:41.421 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.4684999999999999e-05 2021-02-10T04:51:41.421 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.421 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.422 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:51:41.422 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.199314+0000", 2021-02-10T04:51:41.422 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.5300000000000002e-06 2021-02-10T04:51:41.422 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.422 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.422 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:51:41.422 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.199321+0000", 2021-02-10T04:51:41.422 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9565999999999999e-05 2021-02-10T04:51:41.422 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.423 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.423 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:51:41.423 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.199370+0000", 2021-02-10T04:51:41.423 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0029726380000000001 2021-02-10T04:51:41.423 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.423 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.423 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:51:41.423 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.202343+0000", 2021-02-10T04:51:41.423 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0030838609999999998 2021-02-10T04:51:41.423 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.423 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:51:41.423 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.424 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.424 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.424 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.14 1121/1119 [PushReplyOp(7:2dcefd69:::smithi08575397-28:368)])", 2021-02-10T04:51:41.424 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:39.217537+0000", 2021-02-10T04:51:41.424 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 2.175281204, 2021-02-10T04:51:41.424 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031144010000000001, 2021-02-10T04:51:41.424 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:51:41.424 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:51:41.424 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:51:41.424 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.424 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:51:41.424 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.217537+0000", 2021-02-10T04:51:41.424 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:51:41.425 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.425 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.425 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:51:41.425 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.217537+0000", 2021-02-10T04:51:41.425 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.5524000000000001e-05 2021-02-10T04:51:41.425 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.425 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.425 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:51:41.425 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.217572+0000", 2021-02-10T04:51:41.425 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1260999999999999e-05 2021-02-10T04:51:41.425 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.425 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.425 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:51:41.426 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.217584+0000", 2021-02-10T04:51:41.426 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.4684999999999999e-05 2021-02-10T04:51:41.426 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.426 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.426 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:51:41.426 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.217598+0000", 2021-02-10T04:51:41.426 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.6000000000000007e-06 2021-02-10T04:51:41.426 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.426 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.426 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:51:41.426 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.217607+0000", 2021-02-10T04:51:41.427 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.8575000000000002e-05 2021-02-10T04:51:41.427 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.427 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.427 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:51:41.427 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.217656+0000", 2021-02-10T04:51:41.427 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.002995756 2021-02-10T04:51:41.427 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.427 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.427 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:51:41.427 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.220651+0000", 2021-02-10T04:51:41.427 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031144010000000001 2021-02-10T04:51:41.427 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.427 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:51:41.428 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.428 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.428 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.428 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52017b30:::smithi08575397-43:20a)])", 2021-02-10T04:51:41.428 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:39.887305+0000", 2021-02-10T04:51:41.428 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 1.5055132470000001, 2021-02-10T04:51:41.428 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003468512, 2021-02-10T04:51:41.428 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:51:41.428 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:51:41.428 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:51:41.428 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.428 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:51:41.429 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.887305+0000", 2021-02-10T04:51:41.429 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:51:41.429 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.429 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.429 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:51:41.429 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.887305+0000", 2021-02-10T04:51:41.429 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.1676e-05 2021-02-10T04:51:41.429 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.429 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.429 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:51:41.429 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.887337+0000", 2021-02-10T04:51:41.429 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.9230000000000008e-06 2021-02-10T04:51:41.429 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.430 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.430 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:51:41.430 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.887345+0000", 2021-02-10T04:51:41.430 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2656999999999999e-05 2021-02-10T04:51:41.430 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.430 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.430 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:51:41.430 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.887358+0000", 2021-02-10T04:51:41.430 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.0090000000000001e-06 2021-02-10T04:51:41.430 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.430 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.430 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:51:41.430 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.887365+0000", 2021-02-10T04:51:41.431 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.7717000000000003e-05 2021-02-10T04:51:41.431 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.431 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.431 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:51:41.431 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.887423+0000", 2021-02-10T04:51:41.431 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00335053 2021-02-10T04:51:41.431 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.431 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.431 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:51:41.431 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.890773+0000", 2021-02-10T04:51:41.432 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003468512 2021-02-10T04:51:41.432 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.432 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:51:41.432 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.432 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.432 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.432 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.3 1122/1119 [PushReplyOp(7:c765b4cc:::smithi08575397-30:309)])", 2021-02-10T04:51:41.432 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:39.889361+0000", 2021-02-10T04:51:41.432 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 1.503457536, 2021-02-10T04:51:41.432 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0033870189999999998, 2021-02-10T04:51:41.433 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:51:41.433 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:51:41.433 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:51:41.433 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.433 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:51:41.433 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.889361+0000", 2021-02-10T04:51:41.433 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:51:41.433 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.433 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.433 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:51:41.433 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.889361+0000", 2021-02-10T04:51:41.434 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.9168999999999999e-05 2021-02-10T04:51:41.434 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.434 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.434 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:51:41.434 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.889380+0000", 2021-02-10T04:51:41.434 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.5779999999999998e-06 2021-02-10T04:51:41.434 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.434 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.434 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:51:41.434 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.889385+0000", 2021-02-10T04:51:41.434 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.8790000000000003e-06 2021-02-10T04:51:41.434 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.435 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.435 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:51:41.435 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.889393+0000", 2021-02-10T04:51:41.435 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.972e-06 2021-02-10T04:51:41.435 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.435 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.435 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:51:41.435 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.889398+0000", 2021-02-10T04:51:41.435 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4211000000000001e-05 2021-02-10T04:51:41.435 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.435 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.435 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:51:41.435 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.889442+0000", 2021-02-10T04:51:41.436 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0033052099999999998 2021-02-10T04:51:41.436 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.436 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.436 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:51:41.436 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.892748+0000", 2021-02-10T04:51:41.436 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0033870189999999998 2021-02-10T04:51:41.436 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.436 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:51:41.436 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.436 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.436 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.436 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52cf5dd7:::smithi08575397-7:20a)])", 2021-02-10T04:51:41.436 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:39.897118+0000", 2021-02-10T04:51:41.436 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 1.4956996279999999, 2021-02-10T04:51:41.437 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0045733759999999997, 2021-02-10T04:51:41.437 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:51:41.437 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:51:41.437 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:51:41.437 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.437 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:51:41.437 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.897118+0000", 2021-02-10T04:51:41.437 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:51:41.437 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.437 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.437 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:51:41.437 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.897118+0000", 2021-02-10T04:51:41.437 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.0298000000000001e-05 2021-02-10T04:51:41.438 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.438 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.438 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:51:41.438 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.897139+0000", 2021-02-10T04:51:41.438 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.7610000000000007e-06 2021-02-10T04:51:41.438 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.438 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.438 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:51:41.438 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.897147+0000", 2021-02-10T04:51:41.438 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2027e-05 2021-02-10T04:51:41.438 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.438 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.438 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:51:41.439 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.897159+0000", 2021-02-10T04:51:41.439 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.0929999999999999e-06 2021-02-10T04:51:41.439 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.439 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.439 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:51:41.439 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.897165+0000", 2021-02-10T04:51:41.439 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000100141 2021-02-10T04:51:41.439 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.439 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.439 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:51:41.439 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.897265+0000", 2021-02-10T04:51:41.439 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0044270560000000004 2021-02-10T04:51:41.440 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.440 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.440 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:51:41.440 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.901692+0000", 2021-02-10T04:51:41.440 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0045733759999999997 2021-02-10T04:51:41.440 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.440 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:51:41.440 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.440 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.440 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.440 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52017b30:::smithi08575397-43:2d9)])", 2021-02-10T04:51:41.440 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:39.946105+0000", 2021-02-10T04:51:41.440 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 1.446713031, 2021-02-10T04:51:41.440 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0035851820000000001, 2021-02-10T04:51:41.441 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:51:41.441 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:51:41.441 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:51:41.441 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.441 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:51:41.441 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946105+0000", 2021-02-10T04:51:41.441 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:51:41.441 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.441 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.441 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:51:41.441 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946105+0000", 2021-02-10T04:51:41.441 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.7836e-05 2021-02-10T04:51:41.441 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.442 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.442 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:51:41.442 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946123+0000", 2021-02-10T04:51:41.442 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.9430000000000003e-06 2021-02-10T04:51:41.442 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.442 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.442 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:51:41.442 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946130+0000", 2021-02-10T04:51:41.442 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.7319999999999993e-06 2021-02-10T04:51:41.442 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.442 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.442 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:51:41.442 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946140+0000", 2021-02-10T04:51:41.443 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.277e-06 2021-02-10T04:51:41.443 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.443 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.443 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:51:41.443 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946147+0000", 2021-02-10T04:51:41.443 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.4368000000000004e-05 2021-02-10T04:51:41.443 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.443 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.443 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:51:41.443 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946211+0000", 2021-02-10T04:51:41.443 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0034790260000000001 2021-02-10T04:51:41.443 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.443 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.444 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:51:41.444 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.949690+0000", 2021-02-10T04:51:41.444 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0035851820000000001 2021-02-10T04:51:41.444 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.444 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:51:41.444 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.444 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.444 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.444 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.3 1122/1119 [PushReplyOp(7:c1415b8e:::smithi08575397-38:35f)])", 2021-02-10T04:51:41.444 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:39.946563+0000", 2021-02-10T04:51:41.444 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 1.4462552609999999, 2021-02-10T04:51:41.444 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031900539999999999, 2021-02-10T04:51:41.444 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:51:41.444 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:51:41.445 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:51:41.445 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.445 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:51:41.445 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946563+0000", 2021-02-10T04:51:41.445 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:51:41.445 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.445 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.445 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:51:41.445 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946563+0000", 2021-02-10T04:51:41.445 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.2059e-05 2021-02-10T04:51:41.445 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.445 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.445 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:51:41.446 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946585+0000", 2021-02-10T04:51:41.446 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9450000000000001e-06 2021-02-10T04:51:41.446 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.446 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.446 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:51:41.446 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946590+0000", 2021-02-10T04:51:41.446 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.9759999999999998e-06 2021-02-10T04:51:41.446 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.446 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.446 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:51:41.446 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946597+0000", 2021-02-10T04:51:41.446 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.9949999999999999e-06 2021-02-10T04:51:41.446 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.446 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.447 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:51:41.447 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946601+0000", 2021-02-10T04:51:41.447 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4035000000000001e-05 2021-02-10T04:51:41.447 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.447 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.447 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:51:41.447 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946645+0000", 2021-02-10T04:51:41.447 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031080439999999999 2021-02-10T04:51:41.447 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.447 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.447 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:51:41.447 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.949753+0000", 2021-02-10T04:51:41.447 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031900539999999999 2021-02-10T04:51:41.448 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.448 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:51:41.448 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.448 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.448 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.448 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52017b30:::smithi08575397-43:2d9)])", 2021-02-10T04:51:41.448 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:39.951097+0000", 2021-02-10T04:51:41.448 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 1.4417209879999999, 2021-02-10T04:51:41.448 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00327655, 2021-02-10T04:51:41.448 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:51:41.448 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:51:41.448 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:51:41.448 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.449 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:51:41.449 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.951097+0000", 2021-02-10T04:51:41.449 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:51:41.449 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.449 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.449 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:51:41.449 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.951097+0000", 2021-02-10T04:51:41.449 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.2909e-05 2021-02-10T04:51:41.449 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.449 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.449 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:51:41.449 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.951120+0000", 2021-02-10T04:51:41.449 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.2708999999999999e-05 2021-02-10T04:51:41.450 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.450 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.450 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:51:41.450 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.951143+0000", 2021-02-10T04:51:41.450 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.9159999999999996e-06 2021-02-10T04:51:41.450 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.450 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.450 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:51:41.450 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.951153+0000", 2021-02-10T04:51:41.450 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9230000000000001e-06 2021-02-10T04:51:41.450 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.450 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.450 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:51:41.451 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.951158+0000", 2021-02-10T04:51:41.451 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.9799000000000003e-05 2021-02-10T04:51:41.451 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.451 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.451 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:51:41.451 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.951197+0000", 2021-02-10T04:51:41.451 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031762940000000001 2021-02-10T04:51:41.451 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.451 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.451 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:51:41.451 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.954374+0000", 2021-02-10T04:51:41.451 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00327655 2021-02-10T04:51:41.452 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.452 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:51:41.452 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.452 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.452 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.452 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:531b0702:::smithi08575397-3:300)])", 2021-02-10T04:51:41.452 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:39.963275+0000", 2021-02-10T04:51:41.452 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 1.429543585, 2021-02-10T04:51:41.452 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003308152, 2021-02-10T04:51:41.452 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:51:41.452 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:51:41.452 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:51:41.452 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.453 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:51:41.453 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.963275+0000", 2021-02-10T04:51:41.453 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:51:41.453 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.453 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.453 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:51:41.453 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.963275+0000", 2021-02-10T04:51:41.453 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.0713e-05 2021-02-10T04:51:41.453 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.453 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.453 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:51:41.453 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.963295+0000", 2021-02-10T04:51:41.453 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.6899999999999997e-06 2021-02-10T04:51:41.453 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.454 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.454 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:51:41.454 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.963301+0000", 2021-02-10T04:51:41.454 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1399e-05 2021-02-10T04:51:41.454 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.454 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.454 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:51:41.454 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.963312+0000", 2021-02-10T04:51:41.454 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.3340000000000003e-06 2021-02-10T04:51:41.454 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.454 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.454 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:51:41.454 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.963317+0000", 2021-02-10T04:51:41.455 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.8989000000000002e-05 2021-02-10T04:51:41.455 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.455 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.455 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:51:41.455 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.963356+0000", 2021-02-10T04:51:41.455 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0032270269999999999 2021-02-10T04:51:41.455 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.455 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.455 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:51:41.455 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.966583+0000", 2021-02-10T04:51:41.455 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003308152 2021-02-10T04:51:41.455 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.455 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:51:41.455 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.456 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.456 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.456 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52017b30:::smithi08575397-43:2cb)])", 2021-02-10T04:51:41.456 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:40.051461+0000", 2021-02-10T04:51:41.456 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 1.3413566219999999, 2021-02-10T04:51:41.456 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0030824229999999999, 2021-02-10T04:51:41.456 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:51:41.456 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:51:41.456 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:51:41.456 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.456 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:51:41.456 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.051461+0000", 2021-02-10T04:51:41.456 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:51:41.456 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.457 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.457 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:51:41.457 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.051461+0000", 2021-02-10T04:51:41.457 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.1915999999999997e-05 2021-02-10T04:51:41.457 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.457 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.457 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:51:41.457 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.051493+0000", 2021-02-10T04:51:41.457 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.9529999999999999e-06 2021-02-10T04:51:41.457 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.457 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.457 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:51:41.458 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.051501+0000", 2021-02-10T04:51:41.458 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.3309000000000001e-05 2021-02-10T04:51:41.458 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.458 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.458 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:51:41.458 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.051515+0000", 2021-02-10T04:51:41.458 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0247e-05 2021-02-10T04:51:41.458 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.458 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.458 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:51:41.458 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.051525+0000", 2021-02-10T04:51:41.459 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.0068999999999999e-05 2021-02-10T04:51:41.459 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.459 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.459 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:51:41.459 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.051565+0000", 2021-02-10T04:51:41.459 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.002978929 2021-02-10T04:51:41.459 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.459 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.459 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:51:41.459 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.054544+0000", 2021-02-10T04:51:41.459 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0030824229999999999 2021-02-10T04:51:41.459 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.459 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:51:41.460 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.460 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.460 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.460 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52017b30:::smithi08575397-43:372)])", 2021-02-10T04:51:41.460 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:40.124429+0000", 2021-02-10T04:51:41.460 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 1.2683886740000001, 2021-02-10T04:51:41.460 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0033528249999999998, 2021-02-10T04:51:41.460 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:51:41.460 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:51:41.460 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:51:41.460 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.460 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:51:41.460 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.124429+0000", 2021-02-10T04:51:41.460 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:51:41.461 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.461 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.461 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:51:41.461 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.124429+0000", 2021-02-10T04:51:41.461 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.7447000000000001e-05 2021-02-10T04:51:41.461 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.461 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.461 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:51:41.461 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.124447+0000", 2021-02-10T04:51:41.461 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0030000000000002e-06 2021-02-10T04:51:41.461 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.461 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.461 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:51:41.462 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.124452+0000", 2021-02-10T04:51:41.462 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.2669999999999995e-06 2021-02-10T04:51:41.462 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.462 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.462 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:51:41.462 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.124461+0000", 2021-02-10T04:51:41.462 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.2230000000000001e-06 2021-02-10T04:51:41.462 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.462 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.462 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:51:41.462 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.124465+0000", 2021-02-10T04:51:41.463 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.5660000000000001e-05 2021-02-10T04:51:41.463 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.463 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.463 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:51:41.463 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.124501+0000", 2021-02-10T04:51:41.463 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003281225 2021-02-10T04:51:41.463 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.463 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.463 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:51:41.463 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.127782+0000", 2021-02-10T04:51:41.463 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0033528249999999998 2021-02-10T04:51:41.463 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.464 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:51:41.464 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.464 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.464 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.464 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52017b30:::smithi08575397-43:33a)])", 2021-02-10T04:51:41.464 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:40.158898+0000", 2021-02-10T04:51:41.464 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 1.233919851, 2021-02-10T04:51:41.464 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003157211, 2021-02-10T04:51:41.464 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:51:41.464 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:51:41.464 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:51:41.464 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.464 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:51:41.464 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.158898+0000", 2021-02-10T04:51:41.465 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:51:41.465 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.465 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.465 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:51:41.465 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.158898+0000", 2021-02-10T04:51:41.465 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.8318999999999999e-05 2021-02-10T04:51:41.465 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.465 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.465 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:51:41.465 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.158917+0000", 2021-02-10T04:51:41.465 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.2360000000000004e-06 2021-02-10T04:51:41.465 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.466 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.466 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:51:41.466 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.158922+0000", 2021-02-10T04:51:41.466 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.7189999999999996e-06 2021-02-10T04:51:41.466 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.466 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.466 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:51:41.466 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.158932+0000", 2021-02-10T04:51:41.466 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.5459999999999999e-06 2021-02-10T04:51:41.466 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.466 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.466 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:51:41.466 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.158937+0000", 2021-02-10T04:51:41.467 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.625e-05 2021-02-10T04:51:41.467 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.467 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.467 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:51:41.467 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.158973+0000", 2021-02-10T04:51:41.467 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003082141 2021-02-10T04:51:41.467 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.467 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.467 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:51:41.467 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.162055+0000", 2021-02-10T04:51:41.467 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003157211 2021-02-10T04:51:41.467 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.467 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:51:41.468 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.468 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.468 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.468 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52017b30:::smithi08575397-43:33a)])", 2021-02-10T04:51:41.468 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:40.163313+0000", 2021-02-10T04:51:41.468 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 1.2295052799999999, 2021-02-10T04:51:41.468 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0032056989999999998, 2021-02-10T04:51:41.468 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:51:41.468 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:51:41.468 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:51:41.468 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.468 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:51:41.468 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.163313+0000", 2021-02-10T04:51:41.469 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:51:41.469 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.469 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.469 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:51:41.469 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.163313+0000", 2021-02-10T04:51:41.469 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.8493999999999999e-05 2021-02-10T04:51:41.469 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.469 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.469 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:51:41.469 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.163331+0000", 2021-02-10T04:51:41.469 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.1150000000000002e-06 2021-02-10T04:51:41.469 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.469 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.470 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:51:41.470 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.163336+0000", 2021-02-10T04:51:41.470 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.4429999999999992e-06 2021-02-10T04:51:41.470 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.470 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.470 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:51:41.470 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.163346+0000", 2021-02-10T04:51:41.470 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.767e-06 2021-02-10T04:51:41.470 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.470 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.470 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:51:41.470 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.163352+0000", 2021-02-10T04:51:41.470 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.5129000000000002e-05 2021-02-10T04:51:41.471 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.471 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.471 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:51:41.471 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.163387+0000", 2021-02-10T04:51:41.471 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031317509999999999 2021-02-10T04:51:41.471 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.471 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.471 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:51:41.471 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.166519+0000", 2021-02-10T04:51:41.471 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0032056989999999998 2021-02-10T04:51:41.471 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.471 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:51:41.471 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.472 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.472 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.472 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52cf5dd7:::smithi08575397-7:33e)])", 2021-02-10T04:51:41.472 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:40.173135+0000", 2021-02-10T04:51:41.472 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 1.2196831800000001, 2021-02-10T04:51:41.472 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003144015, 2021-02-10T04:51:41.472 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:51:41.472 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:51:41.472 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:51:41.472 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.472 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:51:41.472 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.173135+0000", 2021-02-10T04:51:41.472 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:51:41.472 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.473 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.473 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:51:41.473 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.173135+0000", 2021-02-10T04:51:41.473 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.7708999999999999e-05 2021-02-10T04:51:41.473 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.473 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.473 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:51:41.473 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.173153+0000", 2021-02-10T04:51:41.473 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.921e-06 2021-02-10T04:51:41.473 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.473 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.473 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:51:41.473 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.173158+0000", 2021-02-10T04:51:41.474 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.3769999999999996e-06 2021-02-10T04:51:41.474 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.474 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.474 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:51:41.474 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.173166+0000", 2021-02-10T04:51:41.474 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.4409999999999999e-06 2021-02-10T04:51:41.474 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.474 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.474 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:51:41.474 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.173175+0000", 2021-02-10T04:51:41.474 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.6347999999999998e-05 2021-02-10T04:51:41.474 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.475 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.475 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:51:41.475 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.173212+0000", 2021-02-10T04:51:41.475 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0030672189999999999 2021-02-10T04:51:41.475 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.475 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.475 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:51:41.475 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.176279+0000", 2021-02-10T04:51:41.475 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003144015 2021-02-10T04:51:41.475 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.475 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:51:41.475 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.475 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.475 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.476 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52cf5dd7:::smithi08575397-7:33e)])", 2021-02-10T04:51:41.476 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:40.180475+0000", 2021-02-10T04:51:41.476 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 1.2123433370000001, 2021-02-10T04:51:41.476 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031979669999999999, 2021-02-10T04:51:41.476 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:51:41.476 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:51:41.476 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:51:41.476 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.476 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:51:41.476 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.180475+0000", 2021-02-10T04:51:41.476 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:51:41.476 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.476 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.477 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:51:41.477 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.180475+0000", 2021-02-10T04:51:41.477 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.9508e-05 2021-02-10T04:51:41.477 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.477 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.477 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:51:41.477 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.180494+0000", 2021-02-10T04:51:41.477 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.7320000000000003e-06 2021-02-10T04:51:41.477 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.477 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.477 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:51:41.477 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.180500+0000", 2021-02-10T04:51:41.477 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.4909999999999994e-06 2021-02-10T04:51:41.477 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.478 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.478 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:51:41.478 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.180510+0000", 2021-02-10T04:51:41.478 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.3980000000000004e-06 2021-02-10T04:51:41.478 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.478 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.478 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:51:41.478 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.180516+0000", 2021-02-10T04:51:41.478 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.1493999999999999e-05 2021-02-10T04:51:41.478 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.478 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.478 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:51:41.478 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.180557+0000", 2021-02-10T04:51:41.479 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031153439999999999 2021-02-10T04:51:41.479 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.479 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.479 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:51:41.479 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.183673+0000", 2021-02-10T04:51:41.479 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031979669999999999 2021-02-10T04:51:41.479 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.479 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:51:41.479 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.479 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.479 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.479 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52017b30:::smithi08575397-43:349)])", 2021-02-10T04:51:41.479 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:40.214713+0000", 2021-02-10T04:51:41.479 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 1.178104883, 2021-02-10T04:51:41.480 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031374720000000001, 2021-02-10T04:51:41.480 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:51:41.480 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:51:41.480 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:51:41.480 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.480 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:51:41.480 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.214713+0000", 2021-02-10T04:51:41.480 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:51:41.480 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.480 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.480 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:51:41.480 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.214713+0000", 2021-02-10T04:51:41.480 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.8167000000000001e-05 2021-02-10T04:51:41.480 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.481 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.481 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:51:41.481 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.214731+0000", 2021-02-10T04:51:41.481 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0479999999999998e-06 2021-02-10T04:51:41.481 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.481 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.481 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:51:41.481 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.214736+0000", 2021-02-10T04:51:41.481 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.4749999999999999e-06 2021-02-10T04:51:41.481 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.481 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.481 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:51:41.482 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.214746+0000", 2021-02-10T04:51:41.482 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.7610000000000004e-06 2021-02-10T04:51:41.482 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.482 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.482 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:51:41.482 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.214752+0000", 2021-02-10T04:51:41.482 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.621e-05 2021-02-10T04:51:41.482 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.482 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.482 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:51:41.482 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.214798+0000", 2021-02-10T04:51:41.482 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0030528109999999999 2021-02-10T04:51:41.482 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:51:41.482 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:51:41.483 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:51:41.483 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.217851+0000", 2021-02-10T04:51:41.483 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031374720000000001 2021-02-10T04:51:41.483 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.483 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:51:41.483 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.483 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:51:41.483 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:51:41.483 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:51:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[29319]: cluster 2021-02-10T04:51:40.419938+0000 2021-02-10T04:51:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[29319]: mgr.y (mgr.14140) 1217 : cluster [DBG] pgmap v2013: 188 pgs: 1 active+recovering+undersized+remapped, 6 remapped+peering, 2 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 23 peering, 154 active+clean; 895 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 78/2151 objects degraded (3.626%); 10/2151 objects misplaced (0.465%); 14 MiB/s, 39 objects/s recovering 2021-02-10T04:51:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[29319]: cluster 2021-02-10T04:51:40.694423+0000 osd.4 (osd.4) 2581 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[29319]: cluster 2021-02-10T04:51:40.694430+0000 osd.4 (osd.4) 2582 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[29319]: cluster 2021-02-10T04:51:40.833202+0000 mon.a (mon.0) 2328 : cluster [WRN] Health check failed: Degraded data redundancy: 78/2151 objects degraded (3.626%), 1 pg degraded (PG_DEGRADED) 2021-02-10T04:51:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[29319]: cluster 2021-02-10T04:51:41.503758 2021-02-10T04:51:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2329 : cluster [DBG] osdmap e1124: 8 total, 8 up, 8 in 2021-02-10T04:51:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[29319]: audit 2021-02-10T04:51:41. 2021-02-10T04:51:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[29319]: 567529+0000 mon.c (mon.1) 95 : audit [INF] from='client.? 172.21.15.85:0/4227049558' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:51:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[29319]: audit 2021-02-10T04:51:41.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[29319]: 2021-02-10T04:51:41.567905+0000 2021-02-10T04:51:41.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[29319]: mon.a (mon.0) 2330 : audit [INF] 2021-02-10T04:51:41.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[29319]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:51:41.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:51:41.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[35961]: 51:40.419938+0000 mgr.y (mgr.14140) 1217 : cluster [DBG] pgmap v2013: 188 pgs: 1 active+recovering+undersized+remapped, 6 remapped+peering, 2 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 23 peering, 154 active+clean; 895 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 78/2151 objects degraded (3.626%); 10/2151 objects misplaced (0.465%); 14 MiB/s, 39 objects/s recovering 2021-02-10T04:51:41.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[35961]: cluster 2021-02-10T04:51:41.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[35961]: 2021-02-10T04:51:40.694423+0000 osd.4 (osd.4) 2581 : 2021-02-10T04:51:41.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:41.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:51:41.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[35961]: -10T04:51:40.694430+0000 2021-02-10T04:51:41.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[35961]: osd.4 (osd.4) 2582 : cluster [WRN] 2021-02-10T04:51:41.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[35961]: cluster 2021-02-10T04:51:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:51:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[35961]: :51:40.833202+0000 mon.a 2021-02-10T04:51:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[35961]: (mon.0) 2328 : cluster [WRN] Health check failed: Degraded data redundancy: 78/2151 objects degraded (3.626%), 1 pg degraded (PG_DEGRADED) 2021-02-10T04:51:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:51:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[35961]: 02-10T04:51:41.503758+0000 mon.a 2021-02-10T04:51:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[35961]: (mon.0) 2329 : cluster [DBG] osdmap e1124: 8 total, 8 up, 8 in 2021-02-10T04:51:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:51:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[35961]: 02-10T04:51:41.567529+0000 2021-02-10T04:51:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[35961]: mon.c (mon.1) 95 : audit [INF] from='client.? 172.21.15.85:0/4227049558' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:51:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:51:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[35961]: 10T04:51:41.567905+0000 mon.a (mon 2021-02-10T04:51:41.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:41 smithi085 conmon[35961]: .0) 2330 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:51:42.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:41 smithi164 conmon[33677]: debug 2021-02-10T04:51:41.741+0000 7fbec19a1700 -1 osd.4 1124 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:41 smithi164 conmon[29288]: cluster 2021-02-10T04:51:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:41 smithi164 conmon[29288]: 2021-02-10T04:51:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:41 smithi164 conmon[29288]: 2021 2021-02-10T04:51:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:41 smithi164 conmon[29288]: - 2021-02-10T04:51:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:41 smithi164 conmon[29288]: 02-10T04:51: 2021-02-10T04:51:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:41 smithi164 conmon[29288]: 40.419938+0000 mgr.y (mgr.14140) 1217 : cluster [DBG] pgmap v2013: 188 pgs: 1 active+recovering+undersized+remapped, 6 remapped+peering, 2 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 23 peering, 154 active+clean; 895 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 78/2151 objects degraded (3.626%); 10/2151 objects misplaced (0.465%); 14 MiB/s, 39 objects/s recovering 2021-02-10T04:51:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:41 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:51:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:41 smithi164 conmon[29288]: 02-10T04:51:40.694423+0000 osd.4 ( 2021-02-10T04:51:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:41 smithi164 conmon[29288]: osd.4) 2581 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:41 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:51:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:41 smithi164 conmon[29288]: 02-10T04:51:40.694430+0000 osd.4 ( 2021-02-10T04:51:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:41 smithi164 conmon[29288]: osd.4) 2582 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:41 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:51:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:41 smithi164 conmon[29288]: -02-10T04:51:40.833202+0000 2021-02-10T04:51:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:41 smithi164 conmon[29288]: mon.a (mon.0) 2328 : cluster [WRN] Health check failed: Degraded data redundancy: 78/2151 objects degraded (3.626%), 1 pg degraded (PG_DEGRADED) 2021-02-10T04:51:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:41 smithi164 conmon[29288]: 2021-02-10T04:51:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:41 smithi164 conmon[29288]: cluster 2021-02-10T04:51:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:41 smithi164 conmon[29288]: 2021-02-10T04:51:41.503758+0000 mon.a (mon 2021-02-10T04:51:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:41 smithi164 conmon[29288]: .0) 2329 : cluster [DBG] osdmap e1124: 8 total, 8 up, 8 in 2021-02-10T04:51:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:41 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:51:42.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:41 smithi164 conmon[29288]: -10T04:51:41.567529+0000 mon.c ( 2021-02-10T04:51:42.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:41 smithi164 conmon[29288]: mon.1) 95 : audit [INF] from='client.? 172.21.15.85:0/4227049558' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:51:42.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:41 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:51:42.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:41 smithi164 conmon[29288]: 02-10T04:51:41.567905+0000 mon.a 2021-02-10T04:51:42.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:41 smithi164 conmon[29288]: (mon.0) 2330 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:51:42.506 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is unset 2021-02-10T04:51:43.055 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:51:43.055 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:51:43.056 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:51:43.056 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:51:43.056 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:51:43.056 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:51:43.060 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:51:43.084 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:51:43.086 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:51:43.086 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:51:43.086 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:51:43.086 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:51:43.086 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:51:43.086 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:51:43.086 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:51:43.086 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:51:43.111 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:42 smithi085 conmon[29319]: cluster 2021-02-10T04:51:41.742633+0000 osd.4 (osd.4) 2583 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:43.111 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:42 smithi085 conmon[29319]: cluster 2021-02-10T04:51:41.742640+0000 osd.4 (osd.4) 2584 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:43.111 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:42 smithi085 conmon[29319]: cluster 2021-02-10T04:51:42.495957+0000 mon.a (mon.0) 2331 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:51:43.111 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:42 smithi085 conmon[29319]: audit 2021-02-10T04:51:42.499860+0000 mon.a (mon.0) 2332 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:51:43.111 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:42 smithi085 conmon[29319]: cluster 2021-02-10T04:51:42.499931+0000 mon.a (mon.0) 2333 : cluster [DBG] osdmap e1125: 8 total, 8 up, 8 in 2021-02-10T04:51:43.112 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:42 smithi085 conmon[35961]: cluster 2021-02-10T04:51:41 2021-02-10T04:51:43.112 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:42 smithi085 conmon[35961]: .742633+0000 osd.4 (osd.4) 2583 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:43.112 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:42 smithi085 conmon[35961]: cluster 2021-02-10T04:51:43.112 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:42 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:51:43.112 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:42 smithi085 conmon[35961]: :51:41.742640+0000 osd.4 (osd.4) 2584 : cluster [WRN] 2021-02-10T04:51:43.112 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:42 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:43.112 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:42 smithi085 conmon[35961]: cluster 2021-02-10T04:51:43.112 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:42 smithi085 conmon[35961]: 2021-02-10T04:51:42. 2021-02-10T04:51:43.112 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:42 smithi085 conmon[35961]: 495957+0000 mon.a (mon.0) 2331 : cluster 2021-02-10T04:51:43.112 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:42 smithi085 conmon[35961]: [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:51:43.112 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:42 smithi085 conmon[35961]: audit 2021-02-10T04:51:43.112 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:42 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:51:43.112 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:42 smithi085 conmon[35961]: 04:51:42.499860+0000 mon.a (mon.0) 2332 : 2021-02-10T04:51:43.113 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:42 smithi085 conmon[35961]: audit [INF] 2021-02-10T04:51:43.113 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:42 smithi085 conmon[35961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:51:43.113 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:42 smithi085 conmon[35961]: cluster 2021-02-10T04:51:43.113 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:42 smithi085 conmon[35961]: 2021 2021-02-10T04:51:43.113 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:42 smithi085 conmon[35961]: -02-10T04:51:42.499931+0000 mon.a (mon.0 2021-02-10T04:51:43.113 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:42 smithi085 conmon[35961]: ) 2333 : cluster [DBG] osdmap e1125: 8 total, 8 up, 8 in 2021-02-10T04:51:43.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:42 smithi164 conmon[33677]: debug 2021-02-10T04:51:42.782+0000 7fbec19a1700 -1 osd.4 1125 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:43.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:42 smithi164 conmon[29288]: cluster 2021-02-10T04:51:43.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:42 smithi164 conmon[29288]: 2021-02 2021-02-10T04:51:43.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:42 smithi164 conmon[29288]: -10T04:51:41.742633+0000 osd.4 (osd.4) 2583 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:43.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:42 smithi164 conmon[29288]: cluster 2021-02-10T04:51:43.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:42 smithi164 conmon[29288]: 2021-02-10T04:51:41. 2021-02-10T04:51:43.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:42 smithi164 conmon[29288]: 742640+0000 osd.4 (osd.4) 2584 : cluster 2021-02-10T04:51:43.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:42 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:43.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:42 smithi164 conmon[29288]: cluster 2021-02-10T04:51:42.495957+0000 mon.a (mon.0) 2331 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:51:43.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:42 smithi164 conmon[29288]: audit 2021 2021-02-10T04:51:43.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:42 smithi164 conmon[29288]: -02-10T04:51:42.499860+0000 mon.a 2021-02-10T04:51:43.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:42 smithi164 conmon[29288]: (mon.0) 2332 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:51:43.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:42 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:51:43.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:42 smithi164 conmon[29288]: -02-10T04:51:42.499931+0000 2021-02-10T04:51:43.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:42 smithi164 conmon[29288]: mon.a (mon.0) 2333 : cluster [DBG] osdmap e1125: 8 total, 8 up, 8 in 2021-02-10T04:51:43.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:42 smithi164 conmon[29288]: 2021-02-10T04:51:43.631 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.4 dump_ops_in_flight 2021-02-10T04:51:43.861 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-02-10T04:51:43.861 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:51:43.861 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.865674 2021-02-10T04:51:43.862 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd primary-affinity 1 0.8656744204695512 2021-02-10T04:51:44.103 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:51:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:43 smithi164 conmon[29288]: cluster 2021-02-10T04:51:42.420414+0000 mgr.y (mgr.14140) 2021-02-10T04:51:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:43 smithi164 conmon[29288]: 1218 : cluster [DBG] pgmap v2016: 188 pgs: 1 active+recovering+undersized+remapped, 5 remapped+peering, 2 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 179 active+clean; 896 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.7 MiB/s wr, 79 op/s; 78/2229 objects degraded (3.499%); 5/2229 objects misplaced (0.224%); 32 MiB/s, 69 objects/s recovering 2021-02-10T04:51:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:43 smithi164 conmon[29288]: cluster 2021-02-10T04:51:42.783454+0000 osd.4 (osd.4) 2585 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:43 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:51:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:43 smithi164 conmon[29288]: :51:42.783459+0000 osd.4 (osd.4) 2586 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:43 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:51:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:43 smithi164 conmon[29288]: 10T04:51:42.892852+0000 mon.c (mon.1) 96 : audit [DBG] from='client.? 172.21.15.85:0/1154031384' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:51:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:43 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:51:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:43 smithi164 conmon[29288]: -10T04:51:43.485489+0000 mon.a (mon.0) 2334 : cluster 2021-02-10T04:51:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:43 smithi164 conmon[29288]: [DBG] osdmap e1126: 8 total, 8 up, 8 in 2021-02-10T04:51:44.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:43 smithi085 conmon[29319]: cluster 2021-02-10T04:51:42.420414+0000 mgr.y (mgr.14140) 1218 : cluster [DBG] pgmap v2016: 188 pgs: 1 active+recovering+undersized+remapped, 5 remapped+peering, 2 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 179 active+clean; 896 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.7 MiB/s wr, 79 op/s; 78/2229 objects degraded (3.499%); 5/2229 objects misplaced (0.224%); 32 MiB/s, 69 objects/s recovering 2021-02-10T04:51:44.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:43 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:51:44.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:43 smithi085 conmon[29319]: T04:51:42.783454+0000 osd.4 (osd.4) 2585 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:44.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:43 smithi085 conmon[29319]: cluster 2021-02-10T04:51:42.783459+0000 osd.4 (osd.4) 2586 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:44.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:43 smithi085 conmon[29319]: audit 2021-02-10T04:51:42.892852+0000 mon.c (mon.1) 96 : audit [DBG] from='client.? 172.21.15.85:0/1154031384' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:51:44.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:43 smithi085 conmon[29319]: cluster 2021-02-10T04:51:43.485489+0000 mon.a (mon. 2021-02-10T04:51:44.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:43 smithi085 conmon[29319]: 0) 2334 : cluster [DBG] osdmap e1126: 8 total, 8 up, 8 in 2021-02-10T04:51:44.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:43 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:51:44.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:43 smithi085 conmon[35961]: :51:42.420414+0000 mgr.y (mgr.14140) 1218 : cluster [DBG] pgmap v2016: 188 pgs: 1 active+recovering+undersized+remapped, 5 remapped+peering, 2 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 179 active+clean; 896 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.7 MiB/s wr, 79 op/s; 78/2229 objects degraded (3.499%); 5/2229 objects misplaced (0.224%); 32 MiB/s, 69 objects/s recovering 2021-02-10T04:51:44.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:43 smithi085 conmon[35961]: cluster 2021-02-10T04:51:44.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:43 smithi085 conmon[35961]: 2021-02-10T04:51:42.783454+0000 osd.4 (osd.4) 2585 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:44.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:43 smithi085 conmon[35961]: cluster 2021-02-10T04:51:42.783459+0000 osd.4 ( 2021-02-10T04:51:44.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:43 smithi085 conmon[35961]: osd.4) 2586 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:44.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:43 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:51:44.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:43 smithi085 conmon[35961]: 04:51:42.892852+0000 mon.c 2021-02-10T04:51:44.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:43 smithi085 conmon[35961]: (mon.1) 96 : audit 2021-02-10T04:51:44.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:43 smithi085 conmon[35961]: [DBG] from='client.? 172.21.15.85:0/1154031384' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:51:44.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:43 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:51:44.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:43 smithi085 conmon[35961]: -02-10T04:51:43.485489+0000 mon.a 2021-02-10T04:51:44.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:43 smithi085 conmon[35961]: (mon.0) 2334 : cluster [DBG] osdmap e1126: 8 total, 8 up, 8 in 2021-02-10T04:51:45.123 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:44 smithi085 conmon[29319]: cluster 2021-02-10T04:51:45.123 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:44 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:51:45.123 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:44 smithi085 conmon[29319]: 10T04:51:43.663758+0000 osd.3 (osd.3) 7 : cluster [DBG] 5.13 scrub ok 2021-02-10T04:51:45.123 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:44 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:51:45.123 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:44 smithi085 conmon[29319]: -10T04:51:44.492294+0000 mon.a (mon 2021-02-10T04:51:45.123 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:44 smithi085 conmon[29319]: .0) 2335 : cluster [DBG] osdmap e1127: 8 total, 8 up, 8 in 2021-02-10T04:51:45.123 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:44 smithi085 conmon[29319]: 2021-02-10T04:51:45.123 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:44 smithi085 conmon[35961]: cluster 2021-02-10T04:51:43.663758+0000 2021-02-10T04:51:45.123 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:44 smithi085 conmon[35961]: osd.3 (osd.3) 7 : cluster [DBG] 5.13 scrub ok 2021-02-10T04:51:45.124 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:44 smithi085 conmon[35961]: cluster 2021-02-10T04:51:44.492294+0000 mon.a (mon.0) 2335 : cluster [DBG] osdmap e1127: 8 total, 8 up, 8 in 2021-02-10T04:51:45.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:44 smithi164 conmon[29288]: cluster 2021-02-10T04:51:43.663758+0000 osd.3 ( 2021-02-10T04:51:45.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:44 smithi164 conmon[29288]: osd.3) 7 : cluster [DBG] 5.13 scrub ok 2021-02-10T04:51:45.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:44 smithi164 conmon[29288]: cluster 2021-02-10T04:51:44.492294+0000 mon.a (mon.0) 2335 : cluster [DBG] osdmap e1127: 8 total, 8 up, 8 in 2021-02-10T04:51:45.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[35961]: cluster 2021-02-10T04:51:44.420869 2021-02-10T04:51:45.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1219 : cluster [DBG] pgmap v2019: 188 pgs: 1 peering, 1 active+undersized+remapped, 186 active+clean; 896 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.7 MiB/s wr, 79 op/s; 57 MiB/s, 135 objects/s recovering 2021-02-10T04:51:45.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[35961]: cluster 2021-02-10T04:51:45.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[35961]: 2021-02-10T04:51:44.694706+0000 osd.3 (osd.3) 8 : cluster [DBG] 4.11 scrub ok 2021-02-10T04:51:45.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[35961]: 2021-02-10T04:51:45.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[35961]: cluster 2021-02-10T04:51:45.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[35961]: 2021-02-10T04:51:44.799421+0000 osd.4 (osd.4 2021-02-10T04:51:45.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[35961]: ) 2587 : cluster [DBG] 6.1e scrub ok 2021-02-10T04:51:45.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[35961]: cluster 2021-02-10T04:51:45.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[35961]: 2021-02-10T04:51:44.850150+0000 mon.a (mon.0) 2336 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 78/2229 objects degraded (3.499%), 1 pg degraded) 2021-02-10T04:51:45.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:51:45.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[35961]: -02-10T04:51:44.970506+0000 osd.2 (osd.2) 1961 : cluster [DBG] 6.14 scrub ok 2021-02-10T04:51:45.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:51:45.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[35961]: 51:45.336761+0000 osd.0 (osd. 2021-02-10T04:51:45.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[35961]: 0) 166 : cluster [DBG] 4.1e scrub ok 2021-02-10T04:51:45.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[29319]: cluster 2021-02-10T04:51:44.420869+0000 mgr.y ( 2021-02-10T04:51:45.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[29319]: mgr.14140) 1219 : cluster [DBG] pgmap v2019: 188 pgs: 1 peering, 1 active+undersized+remapped, 186 active+clean; 896 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.7 MiB/s wr, 79 op/s; 57 MiB/s, 135 objects/s recovering 2021-02-10T04:51:45.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[29319]: cluster 2021-02-10T04:51:44.694706+0000 osd.3 (osd.3) 8 : cluster [DBG] 2021-02-10T04:51:45.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[29319]: 4.11 scrub ok 2021-02-10T04:51:45.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[29319]: cluster 2021-02-10T04:51:45.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[29319]: 2021-02-10T04:51:44.799421+0000 osd.4 (osd. 2021-02-10T04:51:45.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[29319]: 4) 2587 : cluster [DBG] 6.1e scrub ok 2021-02-10T04:51:45.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[29319]: cluster 2021-02-10T04:51:45.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[29319]: 2021-02-10T04:51:44.850150+0000 2021-02-10T04:51:45.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[29319]: mon.a (mon.0) 2336 : cluster [INF] 2021-02-10T04:51:45.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[29319]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 78/2229 objects degraded (3.499%), 1 pg degraded) 2021-02-10T04:51:45.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[29319]: cluster 2021-02-10T04:51 2021-02-10T04:51:45.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[29319]: :44.970506+0000 osd.2 (osd.2) 1961 2021-02-10T04:51:45.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[29319]: : cluster [DBG] 6.14 scrub ok 2021-02-10T04:51:45.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:51:45.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[29319]: -10T04:51:45.336761+0000 osd.0 2021-02-10T04:51:45.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[29319]: (osd.0) 166 : cluster [DBG] 4.1e scrub ok 2021-02-10T04:51:45.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:45 smithi085 conmon[29319]: 2021-02-10T04:51:46.080 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:51:46.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:45 smithi164 conmon[29288]: cluster 2021-02-10T04:51:44.420869+0000 mgr.y 2021-02-10T04:51:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:45 smithi164 conmon[29288]: (mgr.14140) 1219 : cluster [DBG] pgmap v2019: 188 pgs: 1 peering, 1 active+undersized+remapped, 186 active+clean; 896 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 6.9 MiB/s rd, 1.7 MiB/s wr, 79 op/s; 57 MiB/s, 135 objects/s recovering 2021-02-10T04:51:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:45 smithi164 conmon[29288]: cluster 2021-02-10T04:51:44.694706+0000 osd.3 (osd.3) 8 : cluster [DBG] 4.11 scrub ok 2021-02-10T04:51:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:45 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:51:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:45 smithi164 conmon[29288]: -02-10T04:51:44.799421+0000 osd.4 (osd.4) 2587 : cluster [DBG] 6.1e scrub ok 2021-02-10T04:51:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:45 smithi164 conmon[29288]: cluster 2021-02-10T04:51:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:45 smithi164 conmon[29288]: 2021-02-10T04:51:44.850150+0000 mon.a (mon.0) 2336 2021-02-10T04:51:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:45 smithi164 conmon[29288]: : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 78/2229 objects degraded (3.499%), 1 pg degraded) 2021-02-10T04:51:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:45 smithi164 conmon[29288]: cluster 2021-02-10T04:51:44.970506+0000 2021-02-10T04:51:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:45 smithi164 conmon[29288]: osd.2 (osd.2) 1961 : cluster [DBG] 6.14 scrub ok 2021-02-10T04:51:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:45 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:51:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:45 smithi164 conmon[29288]: 04:51:45.336761+0000 osd.0 (osd.0 2021-02-10T04:51:46.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:45 smithi164 conmon[29288]: ) 166 : cluster [DBG] 4.1e scrub ok 2021-02-10T04:51:46.950 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset nodeep-scrub 2021-02-10T04:51:46.981 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:51:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:46 smithi164 conmon[29288]: cluster 2021-02-10T04:51:45.813673+0000 osd.4 ( 2021-02-10T04:51:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:46 smithi164 conmon[29288]: osd.4) 2588 : cluster [DBG] 5.1a scrub ok 2021-02-10T04:51:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:46 smithi164 conmon[29288]: cluster 2021-02-10T04:51:45.964026+0000 osd.2 (osd.2) 1962 : cluster [DBG] 7.5 scrub ok 2021-02-10T04:51:47.150 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:46 smithi085 conmon[29319]: cluster 2021-02-10T04:51:45.813673+0000 osd.4 ( 2021-02-10T04:51:47.150 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:46 smithi085 conmon[29319]: osd.4) 2588 : cluster [DBG] 5.1a scrub ok 2021-02-10T04:51:47.150 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:46 smithi085 conmon[29319]: cluster 2021-02-10T04:51:45.964026+0000 osd.2 (osd.2) 1962 : cluster [DBG] 7.5 scrub ok 2021-02-10T04:51:47.151 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:46 smithi085 conmon[35961]: cluster 2021-02-10T04:51:45.813673+0000 osd.4 (osd.4) 2588 : cluster [DBG] 5.1a scrub ok 2021-02-10T04:51:47.151 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:46 smithi085 conmon[35961]: cluster 2021-02-10T04:51:45.964026+0000 osd.2 (osd.2) 1962 : cluster [DBG] 7.5 scrub ok 2021-02-10T04:51:47.214 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:51:47.253 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:51:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:47 smithi164 conmon[29288]: cluster 2021-02-10T04:51:46.421387+0000 2021-02-10T04:51:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:47 smithi164 conmon[29288]: mgr.y (mgr.14140) 1220 : cluster [DBG] pgmap v2021: 188 pgs: 1 peering, 187 active+clean; 896 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 36 MiB/s, 85 objects/s recovering 2021-02-10T04:51:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:47 smithi164 conmon[29288]: cluster 2021-02-10T04:51:47.121526+0000 osd.7 (osd.7) 84 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:51:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:47 smithi164 conmon[29288]: 2021-02-10T04:51:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:47 smithi164 conmon[29288]: cluster 2021-02-10T04:51:47.124026+0000 osd.7 (osd.7) 2021-02-10T04:51:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:47 smithi164 conmon[29288]: 85 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:51:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:47 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:51:48.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:47 smithi164 conmon[29288]: T04:51:47.281924+0000 osd.0 (osd.0) 167 : cluster [DBG] 4.6 scrub ok 2021-02-10T04:51:48.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:47 smithi085 conmon[29319]: cluster 2021-02-10T04:51:46.421387+0000 mgr.y (mgr.14140) 1220 : cluster [DBG] pgmap v2021: 188 pgs: 1 peering, 187 active+clean; 896 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 36 MiB/s, 85 objects/s recovering 2021-02-10T04:51:48.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:47 smithi085 conmon[29319]: cluster 2021-02-10T04:51:47.121526+0000 osd.7 (osd.7) 84 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:51:48.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:47 smithi085 conmon[29319]: cluster 2021-02-10T04:51:47.124026+0000 osd.7 (osd.7) 85 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:51:48.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:47 smithi085 conmon[29319]: cluster 2021-02-10T04:51:47.281924+0000 osd.0 (osd.0) 167 : cluster [DBG] 4.6 scrub ok 2021-02-10T04:51:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:47 smithi085 conmon[35961]: cluster 2021-02-10T04:51:46.421387+0000 mgr.y (mgr.14140) 1220 : cluster [DBG] pgmap v2021: 188 pgs: 1 peering, 187 active+clean; 896 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 36 MiB/s, 85 objects/s recovering 2021-02-10T04:51:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:47 smithi085 conmon[35961]: cluster 2021-02-10T04:51:47.121526+0000 osd.7 (osd.7) 84 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:51:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:47 smithi085 conmon[35961]: cluster 2021-02-10T04:51:47.124026+0000 osd.7 (osd.7) 85 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:51:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:47 smithi085 conmon[35961]: cluster 2021-02-10T04:51:47.281924+0000 osd.0 (osd.0) 167 : cluster [DBG] 4.6 scrub ok 2021-02-10T04:51:48.511 INFO:teuthology.orchestra.run.smithi085.stderr:set osd.1 primary-affinity to 0.865674 (8567322) 2021-02-10T04:51:48.881 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:48 smithi085 conmon[29319]: cluster 2021-02-10T04:51:47.322324 2021-02-10T04:51:48.881 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:48 smithi085 conmon[29319]: +0000 osd.6 (osd.6) 161 : cluster [DBG] 4.5 scrub ok 2021-02-10T04:51:48.881 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:48 smithi085 conmon[29319]: cluster 2021-02-10T04:51:47.493052+0000 osd.5 (osd.5) 161 : cluster [DBG] 6.1c scrub ok 2021-02-10T04:51:48.882 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:48 smithi085 conmon[29319]: cluster 2021-02-10T04:51:47.855279+0000 osd.4 (osd.4) 2589 : cluster [DBG] 5.e scrub ok 2021-02-10T04:51:48.882 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:48 smithi085 conmon[29319]: cluster 2021-02-10T04:51:47.866742+0000 mon.a (mon.0) 2337 : cluster [DBG] osdmap e1128: 8 total, 8 up, 8 in 2021-02-10T04:51:48.882 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:48 smithi085 conmon[29319]: audit 2021-02-10T04:51:48.355967+0000 mon.a (mon.0) 2338 : audit [INF] from='client.? 172.21.15.85:0/1078311320' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.8656744204695512}]: dispatch 2021-02-10T04:51:48.882 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:48 smithi085 conmon[29319]: cluster 2021-02-10T04:51:48.481118+0000 mon.a (mon.0) 2339 : cluster [WRN] Health check update: 2 slow ops, oldest one blocked for 437 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:51:48.882 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:48 smithi085 conmon[29319]: audit 2021-02-10T04:51:48.487831+0000 mon.a (mon.0) 2340 : audit [INF] from='client.? 172.21.15.85:0/1078311320' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.8656744204695512}]': finished 2021-02-10T04:51:48.882 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:48 smithi085 conmon[29319]: cluster 2021-02-10T04:51:48.487919+0000 mon.a (mon.0) 2341 : cluster [DBG] osdmap e1129: 8 total, 8 up, 8 in 2021-02-10T04:51:48.882 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:48 smithi085 conmon[35961]: cluster 2021-02-10T04:51:47. 2021-02-10T04:51:48.882 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:48 smithi085 conmon[35961]: 322324+0000 osd.6 (osd.6) 161 : cluster [DBG] 4.5 scrub ok 2021-02-10T04:51:48.882 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:48 smithi085 conmon[35961]: cluster 2021-02-10T04:51:47.493052+0000 osd.5 (osd.5) 161 : cluster [DBG] 6.1c scrub ok 2021-02-10T04:51:48.882 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:48 smithi085 conmon[35961]: cluster 2021-02-10T04:51:47.855279+0000 osd.4 (osd.4) 2589 : cluster [DBG] 5.e scrub ok 2021-02-10T04:51:48.882 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:48 smithi085 conmon[35961]: cluster 2021-02-10T04:51:47.866742+0000 mon.a (mon.0) 2337 : cluster [DBG] osdmap e1128: 8 total, 8 up, 8 in 2021-02-10T04:51:48.883 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:48 smithi085 conmon[35961]: audit 2021-02-10T04:51:48.355967+0000 mon.a (mon.0) 2338 : audit [INF] from='client.? 172.21.15.85:0/1078311320' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.8656744204695512}]: dispatch 2021-02-10T04:51:48.883 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:48 smithi085 conmon[35961]: cluster 2021-02-10T04:51:48.481118+0000 mon.a (mon.0) 2339 : cluster [WRN] Health check update: 2 slow ops, oldest one blocked for 437 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:51:48.883 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:48 smithi085 conmon[35961]: audit 2021-02-10T04:51:48.487831+0000 mon.a (mon.0) 2340 : audit [INF] from='client.? 172.21.15.85:0/1078311320' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.8656744204695512}]': finished 2021-02-10T04:51:48.883 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:48 smithi085 conmon[35961]: cluster 2021-02-10T04:51:48.487919+0000 mon.a (mon.0) 2341 : cluster [DBG] osdmap e1129: 8 total, 8 up, 8 in 2021-02-10T04:51:49.144 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:51:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:48 smithi164 conmon[29288]: cluster 2021-02-10T04:51:47.322324+0000 2021-02-10T04:51:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:48 smithi164 conmon[29288]: osd.6 (osd.6) 161 : cluster [DBG] 4.5 scrub ok 2021-02-10T04:51:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:48 smithi164 conmon[29288]: cluster 2021-02-10T04:51:47.493052+0000 osd.5 (osd.5) 161 : cluster [DBG] 6.1c scrub ok 2021-02-10T04:51:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:48 smithi164 conmon[29288]: cluster 2021-02-10T04:51:47.855279+0000 osd.4 (osd.4) 2589 : cluster [DBG] 5.e scrub ok 2021-02-10T04:51:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:48 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:51:49.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:48 smithi164 conmon[29288]: 51:47.866742+0000 mon.a (mon.0) 2337 : cluster [DBG] osdmap e1128: 8 total, 8 up, 8 in 2021-02-10T04:51:49.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:48 smithi164 conmon[29288]: audit 2021-02-10T04:51:49.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:48 smithi164 conmon[29288]: 2021-02-10T04:51:48.355967+0000 mon.a ( 2021-02-10T04:51:49.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:48 smithi164 conmon[29288]: mon.0) 2338 : audit [INF] from='client.? 172.21.15.85:0/1078311320' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.8656744204695512}]: dispatch 2021-02-10T04:51:49.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:48 smithi164 conmon[29288]: cluster 2021-02-10T04:51:49.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:48 smithi164 conmon[29288]: 2021-02-10T04:51:48.481118 2021-02-10T04:51:49.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:48 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2339 : cluster [WRN] Health check update: 2 slow ops, oldest one blocked for 437 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:51:49.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:48 smithi164 conmon[29288]: 2021-02-10T04:51:49.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:48 smithi164 conmon[29288]: audit 2021-02-10T04:51:49.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:48 smithi164 conmon[29288]: 2021-02-10T04:51:48. 2021-02-10T04:51:49.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:48 smithi164 conmon[29288]: 487831+0000 mon.a (mon.0) 2340 : audit [INF] from='client.? 172.21.15.85:0/1078311320' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.8656744204695512}]': finished 2021-02-10T04:51:49.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:48 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:51:49.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:48 smithi164 conmon[29288]: -10T04:51:48.487919+0000 mon.a (mon.0) 2341 : cluster 2021-02-10T04:51:49.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:48 smithi164 conmon[29288]: [DBG] osdmap e1129: 8 total, 8 up, 8 in 2021-02-10T04:51:49.881 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: cluster 2021-02-10T04:51:49.881 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:51:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: :51:48.274952+0000 osd.6 (osd.6) 162 : cluster [DBG] 2.6 scrub ok 2021-02-10T04:51:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: cluster 2021-02-10T04:51:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: 2021-02-10 2021-02-10T04:51:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: T04:51:48.421931+0000 mgr.y (mgr.14140) 1221 : cluster 2021-02-10T04:51:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: [DBG] pgmap v2023: 188 pgs: 1 peering, 187 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 864 B/s rd, 0 op/s; 26 MiB/s, 66 objects/s recovering 2021-02-10T04:51:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:51:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: 10T04:51:48.872237+0000 mon.a ( 2021-02-10T04:51:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: mon.0) 2342 : cluster [INF] 2021-02-10T04:51:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 437 sec, osd.4 has slow ops) 2021-02-10T04:51:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: cluster 2021-02-10T04:51:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: 2021-02-10T04:51 2021-02-10T04:51:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: :48.872832+0000 osd.4 ( 2021-02-10T04:51:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: osd.4) 2590 : cluster [DBG] 2021-02-10T04:51:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: 4.1 scrub ok 2021-02-10T04:51:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: cluster 2021-02-10T04:51:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:51:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: 51:48.990054+0000 osd.2 ( 2021-02-10T04:51:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: osd.2) 1963 : cluster 2021-02-10T04:51:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: [DBG] 2.e scrub ok 2021-02-10T04:51:50.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: audit 2021-02-10T04:51:50.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:51:50.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: :51:49.093051+0000 mgr.y (mgr.14140) 1222 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:51:50.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: cluster 2021-02-10T04:51:50.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: 2021-02-10T04:51:50.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: 2021 2021-02-10T04:51:50.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: -02-10T04:51 2021-02-10T04:51:50.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: :49.332991+0000 osd.0 (osd.0) 168 : cluster [DBG] 2021-02-10T04:51:50.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: 7.f scrub ok 2021-02-10T04:51:50.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: cluster 2021-02-10T04:51:50.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: 2021-02 2021-02-10T04:51:50.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: -10T04:51:49.487654+0000 mon.a 2021-02-10T04:51:50.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: (mon.0) 2343 : cluster 2021-02-10T04:51:50.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:49 smithi164 conmon[29288]: [DBG] osdmap e1130: 8 total, 8 up, 8 in 2021-02-10T04:51:50.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[29319]: cluster 2021-02-10T04:51:48.274952+0000 osd.6 (osd.6 2021-02-10T04:51:50.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[29319]: ) 162 : cluster [DBG] 2.6 scrub ok 2021-02-10T04:51:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[29319]: cluster 2021-02-10T04:51:48.421931+0000 mgr.y (mgr.14140) 1221 : cluster [DBG] pgmap v2023: 188 pgs: 1 peering, 187 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 864 B/s rd, 0 op/s; 26 MiB/s, 66 objects/s recovering 2021-02-10T04:51:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[29319]: cluster 2021-02-10T04:51:48.872237+0000 mon.a ( 2021-02-10T04:51:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[29319]: mon.0) 2342 : cluster [INF] Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 437 sec, osd.4 has slow ops) 2021-02-10T04:51:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[29319]: cluster 2021-02-10T04:51 2021-02-10T04:51:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[29319]: :48.872832+0000 osd.4 (osd.4) 2590 : cluster [DBG] 2021-02-10T04:51:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[29319]: 4.1 scrub ok 2021-02-10T04:51:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[29319]: cluster 2021-02-10T04:51:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[29319]: 2021-02-10T04:51:48.990054+0000 osd.2 (osd.2 2021-02-10T04:51:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[29319]: ) 1963 : cluster [DBG] 2.e scrub ok 2021-02-10T04:51:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[29319]: audit 2021-02-10T04:51: 2021-02-10T04:51:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[29319]: 49.093051+0000 mgr.y (mgr.14140) 1222 : audit 2021-02-10T04:51:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[29319]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:51:50.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[29319]: cluster 2021-02-10T04:51:50.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[29319]: 2021-02-10T04:51:49.332991+0000 osd.0 (osd.0) 2021-02-10T04:51:50.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[29319]: 168 : cluster [DBG] 7.f scrub ok 2021-02-10T04:51:50.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[29319]: cluster 2021-02-10T04:51:50.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:51:50.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[29319]: 51:49.487654+0000 mon.a (mon.0) 2021-02-10T04:51:50.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[29319]: 2343 : cluster [DBG] osdmap e1130: 8 total, 8 up, 8 in 2021-02-10T04:51:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[35961]: cluster 2021-02-10T04:51:48.274952+0000 osd.6 (osd.6) 162 : 2021-02-10T04:51:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[35961]: cluster [DBG] 2.6 scrub ok 2021-02-10T04:51:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[35961]: cluster 2021-02-10T04:51: 2021-02-10T04:51:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[35961]: 48.421931+0000 mgr.y (mgr.14140) 1221 : cluster [DBG] pgmap v2023: 188 pgs: 1 peering, 187 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 864 B/s rd, 0 op/s; 26 MiB/s, 66 objects/s recovering 2021-02-10T04:51:50.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[35961]: cluster 2021-02-10T04:51:50.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:51:50.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[35961]: 10T04:51:48.872237+0000 mon.a (mon 2021-02-10T04:51:50.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[35961]: .0) 2342 : cluster [INF] Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 437 sec, osd.4 has slow ops) 2021-02-10T04:51:50.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:51:50.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[35961]: 02-10T04:51:48.872832+0000 osd.4 ( 2021-02-10T04:51:50.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[35961]: osd.4) 2590 : cluster [DBG] 4.1 scrub ok 2021-02-10T04:51:50.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[35961]: cluster 2021-02-10T04:51 2021-02-10T04:51:50.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[35961]: :48.990054+0000 osd.2 (osd.2) 1963 : cluster [DBG] 2.e scrub ok 2021-02-10T04:51:50.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[35961]: audit 2021-02-10T04:51 2021-02-10T04:51:50.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[35961]: :49.093051+0000 mgr.y (mgr.14140) 1222 : 2021-02-10T04:51:50.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[35961]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:51:50.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:51:50.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[35961]: -10T04:51:49.332991+0000 osd.0 ( 2021-02-10T04:51:50.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[35961]: osd.0) 168 : cluster [DBG] 7.f scrub ok 2021-02-10T04:51:50.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[35961]: 2021-02-10T04:51:50.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[35961]: cluster 2021-02-10T04:51:50.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:51:50.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[35961]: 10T04:51:49. 2021-02-10T04:51:50.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[35961]: 487654+0000 mon.a (mon.0) 2021-02-10T04:51:50.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:49 smithi085 conmon[35961]: 2343 : cluster [DBG] osdmap e1130: 8 total, 8 up, 8 in 2021-02-10T04:51:51.954 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:51:51.975 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:51:51.995 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:51:52.014 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:51:52.040 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:51:52.066 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:51:52.089 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:51:52.110 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:51:52.111 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:51:52.111 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:51:52.111 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:51:52.111 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:51:52.112 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:51:52.112 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:51:52.112 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:51:52.112 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:51:52.134 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:51 smithi085 conmon[29319]: cluster 2021-02-10T04:51:50.036109+0000 2021-02-10T04:51:52.134 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:51 smithi085 conmon[29319]: osd.2 (osd.2) 1964 : cluster [DBG] 6.12 scrub ok 2021-02-10T04:51:52.134 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:51 smithi085 conmon[29319]: cluster 2021-02-10T04:51:50.422397+0000 mgr.y (mgr.14140) 1223 : cluster [DBG] pgmap v2026: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 863 B/s rd, 0 op/s 2021-02-10T04:51:52.134 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:51 smithi085 conmon[29319]: cluster 2021-02-10T04:51:50.432987+0000 osd.5 (osd.5) 162 : cluster [DBG] 1.8 scrub ok 2021-02-10T04:51:52.134 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:51 smithi085 conmon[29319]: cluster 2021-02-10T04:51:50.895364+0000 osd.4 (osd.4) 2591 : cluster [DBG] 5.1 scrub ok 2021-02-10T04:51:52.135 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:51 smithi085 conmon[29319]: cluster 2021-02-10T04:51:51.382035+0000 osd.0 (osd.0) 169 : 2021-02-10T04:51:52.135 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:51 smithi085 conmon[29319]: cluster [DBG] 4.b scrub ok 2021-02-10T04:51:52.135 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:51 smithi085 conmon[35961]: cluster 2021-02-10T04:51: 2021-02-10T04:51:52.135 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:51 smithi085 conmon[35961]: 50.036109+0000 osd.2 (osd.2) 1964 : cluster [DBG] 6.12 scrub ok 2021-02-10T04:51:52.135 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:51 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:51:52.135 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:51 smithi085 conmon[35961]: 02-10T04:51:50. 2021-02-10T04:51:52.135 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:51 smithi085 conmon[35961]: 422397+0000 mgr.y (mgr.14140) 2021-02-10T04:51:52.135 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:51 smithi085 conmon[35961]: 1223 : cluster [DBG] 2021-02-10T04:51:52.135 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:51 smithi085 conmon[35961]: pgmap v2026: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 863 B/s rd, 0 op/s 2021-02-10T04:51:52.135 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:51 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:51:52.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:51 smithi085 conmon[35961]: 02-10T04:51:50.432987 2021-02-10T04:51:52.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:51 smithi085 conmon[35961]: +0000 osd.5 (osd.5) 162 : cluster 2021-02-10T04:51:52.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:51 smithi085 conmon[35961]: [DBG] 1.8 scrub ok 2021-02-10T04:51:52.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:51 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:51:52.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:51 smithi085 conmon[35961]: 02-10T04:51:50.895364+0000 osd.4 2021-02-10T04:51:52.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:51 smithi085 conmon[35961]: (osd.4) 2591 : cluster 2021-02-10T04:51:52.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:51 smithi085 conmon[35961]: [DBG] 5.1 scrub ok 2021-02-10T04:51:52.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:51 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:51:52.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:51 smithi085 conmon[35961]: 10T04:51:51.382035+0000 2021-02-10T04:51:52.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:51 smithi085 conmon[35961]: osd.0 (osd.0) 169 : 2021-02-10T04:51:52.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:51 smithi085 conmon[35961]: cluster [DBG] 4.b scrub ok 2021-02-10T04:51:52.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:51 smithi164 conmon[29288]: cluster 2021-02-10T04:51:50.036109+0000 2021-02-10T04:51:52.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:51 smithi164 conmon[29288]: osd.2 (osd.2) 1964 : cluster [DBG] 6.12 scrub ok 2021-02-10T04:51:52.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:51 smithi164 conmon[29288]: cluster 2021-02-10T04:51:50.422397+0000 mgr.y (mgr.14140) 1223 : 2021-02-10T04:51:52.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:51 smithi164 conmon[29288]: cluster [DBG] pgmap v2026: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 863 B/s rd, 0 op/s 2021-02-10T04:51:52.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:51 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:51:52.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:51 smithi164 conmon[29288]: -10T04:51:50.432987+0000 osd.5 (osd.5) 162 : cluster [DBG] 2021-02-10T04:51:52.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:51 smithi164 conmon[29288]: 1.8 scrub ok 2021-02-10T04:51:52.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:51 smithi164 conmon[29288]: cluster 2021-02-10T04:51 2021-02-10T04:51:52.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:51 smithi164 conmon[29288]: :50.895364+0000 osd.4 (osd.4) 2591 : cluster [DBG] 2021-02-10T04:51:52.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:51 smithi164 conmon[29288]: 5.1 scrub ok 2021-02-10T04:51:52.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:51 smithi164 conmon[29288]: cluster 2021-02-10T04:51:52.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:51 smithi164 conmon[29288]: 2021-02-10T04:51:51.382035+0000 osd.0 (osd.0) 169 2021-02-10T04:51:52.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:51 smithi164 conmon[29288]: : cluster [DBG] 4.b scrub ok 2021-02-10T04:51:52.898 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is unset 2021-02-10T04:51:53.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:52 smithi164 conmon[33677]: debug 2021-02-10T04:51:53.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:52 smithi164 conmon[33677]: 2021-02-10T04:51:52.863+0000 7fbec19a1700 -1 osd.4 1130 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:53.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:52 smithi164 conmon[33677]: 2021-02-10T04:51:53.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:52 smithi164 conmon[29288]: cluster 2021-02-10T04:51:51.836631+0000 2021-02-10T04:51:53.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:52 smithi164 conmon[29288]: osd.3 (osd.3) 9 : cluster [DBG] 5.10 scrub ok 2021-02-10T04:51:53.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:52 smithi164 conmon[29288]: cluster 2021-02-10T04:51:51.911847+0000 osd.4 (osd.4) 2592 : cluster [DBG] 1.5 scrub ok 2021-02-10T04:51:53.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:52 smithi164 conmon[29288]: 2021-02-10T04:51:53.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:52 smithi164 conmon[29288]: audit 2021-02-10T04:51: 2021-02-10T04:51:53.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:52 smithi164 conmon[29288]: 51.927254+0000 mon.a (mon.0) 2344 : audit [DBG] from='client.? 172.21.15.85:0/307102792' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:51:53.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:52 smithi164 conmon[29288]: 2021-02-10T04:51:53.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:52 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:51:53.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:52 smithi164 conmon[29288]: -02-10T04:51:52.350701+0000 osd.1 (osd. 2021-02-10T04:51:53.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:52 smithi164 conmon[29288]: 1) 481 : cluster [DBG] 2.a scrub ok 2021-02-10T04:51:53.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:52 smithi164 conmon[29288]: audit 2021-02-10T04:51:53.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:52 smithi164 conmon[29288]: 2021-02-10T04:51:52.368189 2021-02-10T04:51:53.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:52 smithi164 conmon[29288]: +0000 mon.c (mon.1) 97 : audit [INF] 2021-02-10T04:51:53.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:52 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/1086280459' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:51:53.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:52 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:51:53.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:52 smithi164 conmon[29288]: 04:51:52.368567+0000 mon.a (mon.0) 2021-02-10T04:51:53.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:52 smithi164 conmon[29288]: 2345 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:51:53.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:52 smithi085 conmon[29319]: cluster 2021-02-10T04:51:51.836631+0000 osd.3 (osd.3) 9 : 2021-02-10T04:51:53.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:52 smithi085 conmon[29319]: cluster [DBG] 5.10 scrub ok 2021-02-10T04:51:53.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:52 smithi085 conmon[29319]: cluster 2021-02-10T04:51:51.911847+0000 osd.4 (osd.4) 2592 : cluster [DBG] 1.5 scrub ok 2021-02-10T04:51:53.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:52 smithi085 conmon[29319]: audit 2021-02-10T04:51:51.927254+0000 mon.a (mon.0) 2344 : audit [DBG] from='client.? 172.21.15.85:0/307102792' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:51:53.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:52 smithi085 conmon[29319]: cluster 2021-02-10T04:51:52.350701+0000 osd.1 (osd.1) 481 : cluster [DBG] 2.a scrub ok 2021-02-10T04:51:53.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:52 smithi085 conmon[29319]: audit 2021-02-10T04:51:52.368189+0000 mon.c (mon.1) 97 : audit [INF] from='client.? 172.21.15.85:0/1086280459' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:51:53.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:52 smithi085 conmon[29319]: audit 2021-02-10T04:51:52.368567+0000 mon.a (mon.0) 2345 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:51:53.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:52 smithi085 conmon[35961]: cluster 2021-02-10T04:51:51.836631+0000 osd.3 2021-02-10T04:51:53.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:52 smithi085 conmon[35961]: (osd.3) 9 : cluster [DBG] 5.10 scrub ok 2021-02-10T04:51:53.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:52 smithi085 conmon[35961]: cluster 2021-02-10T04:51:51. 2021-02-10T04:51:53.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:52 smithi085 conmon[35961]: 911847+0000 osd.4 (osd.4) 2592 : cluster [DBG] 1.5 scrub ok 2021-02-10T04:51:53.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:52 smithi085 conmon[35961]: audit 2021-02-10T04:51:51 2021-02-10T04:51:53.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:52 smithi085 conmon[35961]: .927254+0000 mon.a (mon.0) 2344 : audit [DBG] from='client.? 172.21.15.85:0/307102792' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:51:53.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:52 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:51:53.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:52 smithi085 conmon[35961]: -02-10T04:51:52.350701+0000 osd.1 (osd.1) 481 : cluster [DBG] 2021-02-10T04:51:53.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:52 smithi085 conmon[35961]: 2.a scrub ok 2021-02-10T04:51:53.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:52 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:51:53.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:52 smithi085 conmon[35961]: 02-10T04:51:52.368189+0000 mon.c 2021-02-10T04:51:53.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:52 smithi085 conmon[35961]: (mon.1) 97 : audit [INF] from='client.? 172.21.15.85:0/1086280459' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:51:53.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:52 smithi085 conmon[35961]: 2021-02-10T04:51:53.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:52 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:51:53.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:52 smithi085 conmon[35961]: T04:51:52.368567+0000 mon.a (mon 2021-02-10T04:51:53.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:52 smithi085 conmon[35961]: .0) 2345 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:51:53.893 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[29319]: cluster 2021-02-10T04:51:52.422945+0000 mgr.y (mgr.14140) 1224 : cluster [DBG] pgmap v2027: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:51:53.893 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[29319]: cluster 2021-02-10T04:51:52.498150+0000 osd.5 (osd.5) 163 : cluster [DBG] 2.12 scrub ok 2021-02-10T04:51:53.893 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[29319]: cluster 2021-02-10T04:51:52.863845+0000 osd.4 (osd.4) 2593 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:53.893 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[29319]: cluster 2021-02-10T04:51:52.863852+0000 osd.4 (osd.4) 2594 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:53.893 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[29319]: cluster 2021-02-10T04:51:52.883011+0000 mon.a (mon.0) 2346 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:51:53.893 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[29319]: cluster 2021-02-10T04:51:52.883037+0000 mon.a (mon.0) 2347 : cluster [INF] Cluster is now healthy 2021-02-10T04:51:53.893 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[29319]: audit 2021-02-10T04:51:52.892688+0000 mon.a (mon.0) 2348 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:51:53.894 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[29319]: cluster 2021-02-10T04:51:52.892766+0000 mon.a (mon.0) 2349 : cluster [DBG] osdmap e1131: 8 total, 8 up, 8 in 2021-02-10T04:51:53.894 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[29319]: cluster 2021-02-10T04:51:52.962174+0000 osd.4 (osd.4) 2595 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:51:53.894 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[29319]: cluster 2021-02-10T04:51:52.963473+0000 osd.4 (osd.4) 2596 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:51:53.894 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[29319]: cluster 2021-02-10T04:51:53.894 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[29319]: 2021-02-10T04:51:53.038831+0000 osd.2 (osd.2) 1965 : cluster [DBG] 5.a scrub ok 2021-02-10T04:51:53.894 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[29319]: audit 2021-02-10T04:51:53.099415+0000 mon.a (mon.0) 2350 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:51:53.894 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[29319]: cluster 2021-02-10T04:51:53.487754+0000 mon.a (mon.0) 2351 : cluster [DBG] osdmap e1132: 8 total, 8 up, 8 in 2021-02-10T04:51:53.894 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: cluster 2021-02-10T04:51:52.422945+0000 mgr.y (mgr.14140) 1224 : cluster [DBG] pgmap v2027: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:51:53.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: cluster 2021-02-10T04:51:52.498150+0000 osd.5 (osd.5) 163 : cluster [DBG] 2.12 scrub ok 2021-02-10T04:51:53.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: cluster 2021-02-10T04:51:52.863845+0000 osd.4 (osd.4) 2593 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:53.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: cluster 2021-02-10T04:51:52.863852+0000 osd.4 (osd.4) 2594 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:53.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: cluster 2021-02-10T04:51:52.883011+0000 mon.a ( 2021-02-10T04:51:53.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: mon.0) 2346 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:51:53.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: cluster 2021-02-10T04:51:52.883037+0000 mon.a (mon.0) 2347 2021-02-10T04:51:53.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: : cluster [INF] Cluster is now healthy 2021-02-10T04:51:53.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:51:53.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: -10T04:51:52.892688+0000 mon.a (mon.0) 2021-02-10T04:51:53.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: 2348 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:51:53.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: cluster 2021-02-10T04:51:53.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:51:53.896 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: 04:51:52.892766+0000 mon.a (mon.0 2021-02-10T04:51:53.896 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: ) 2021-02-10T04:51:53.896 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: 2349 : cluster [DBG] osdmap e1131: 8 total, 8 up, 8 in 2021-02-10T04:51:53.896 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:51:53.896 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: -02-10T04:51:52.962174 2021-02-10T04:51:53.896 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2595 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:51:53.896 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:51:53.896 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: T04:51:52.963473+0000 osd.4 (osd.4) 2596 : 2021-02-10T04:51:53.896 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:51:53.896 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: cluster 2021-02-10T04:51:53.896 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: 2021-02-10T04:51:53.038831+0000 osd.2 (osd.2) 2021-02-10T04:51:53.896 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: 1965 : cluster [DBG] 5.a scrub ok 2021-02-10T04:51:53.896 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:51:53.896 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: 10T04:51:53.099415+0000 2021-02-10T04:51:53.897 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: mon.a (mon.0) 2350 : audit [INF] 2021-02-10T04:51:53.897 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:51:53.897 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:51:53.897 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: 10T04:51:53.487754 2021-02-10T04:51:53.897 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2351 2021-02-10T04:51:53.897 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:53 smithi085 conmon[35961]: : cluster [DBG] osdmap e1132: 8 total, 8 up, 8 in 2021-02-10T04:51:54.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[33677]: debug 2021-02-10T04:51:53.839+0000 7fbec19a1700 -1 osd.4 1132 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[29288]: cluster 2021-02-10T04:51:52.422945+0000 mgr.y 2021-02-10T04:51:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[29288]: (mgr.14140) 1224 : cluster [DBG] pgmap v2027: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:51:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[29288]: cluster 2021-02-10T04:51:52.498150+0000 osd.5 (osd.5) 163 : cluster [DBG] 2.12 scrub ok 2021-02-10T04:51:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[29288]: cluster 2021-02-10T04:51:52. 2021-02-10T04:51:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[29288]: 863845+0000 osd.4 (osd.4) 2593 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:51:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[29288]: T04:51:52.863852+0000 osd.4 (osd 2021-02-10T04:51:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[29288]: .4) 2594 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:51:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[29288]: -10T04:51:52.883011+0000 mon.a ( 2021-02-10T04:51:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[29288]: mon.0) 2346 : cluster 2021-02-10T04:51:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[29288]: [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:51:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:51:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[29288]: 51:52.883037+0000 mon.a (mon.0) 2021-02-10T04:51:54.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[29288]: 2347 : cluster [INF] Cluster is now healthy 2021-02-10T04:51:54.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[29288]: audit 2021-02-10T04:51:52.892688+0000 mon.a (mon.0) 2348 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:51:54.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[29288]: cluster 2021-02-10T04:51:52.892766+0000 mon.a (mon.0) 2349 : cluster [DBG] osdmap e1131: 8 total, 8 up, 8 in 2021-02-10T04:51:54.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[29288]: cluster 2021-02-10T04:51:52.962174+0000 osd.4 (osd.4) 2595 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:51:54.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:51:54.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[29288]: :51:52.963473+0000 osd.4 (osd.4) 2596 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:51:54.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[29288]: cluster 2021-02-10T04:51:54.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[29288]: 2021-02-10T04:51:53.038831+0000 osd.2 2021-02-10T04:51:54.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[29288]: (osd.2) 1965 : cluster [DBG] 5.a scrub ok 2021-02-10T04:51:54.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:51:54.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[29288]: 02-10T04:51:53.099415+0000 mon.a 2021-02-10T04:51:54.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[29288]: (mon.0) 2350 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:51:54.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:51:54.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[29288]: -02-10T04:51:53.487754+0000 2021-02-10T04:51:54.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[29288]: mon.a (mon.0) 2351 : cluster [DBG] 2021-02-10T04:51:54.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:53 smithi164 conmon[29288]: osdmap e1132: 8 total, 8 up, 8 in 2021-02-10T04:51:55.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:54 smithi164 conmon[29288]: cluster 2021-02-10T04:51:53.839711+0000 osd.4 ( 2021-02-10T04:51:55.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:54 smithi164 conmon[29288]: osd.4) 2597 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:55.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:54 smithi164 conmon[29288]: cluster 2021-02-10T04:51:53.839729+0000 osd.4 (osd.4) 2598 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:55.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:54 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:51:55.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:54 smithi164 conmon[29288]: 02-10T04:51:54.444283+0000 osd.0 (osd.0) 170 : cluster [DBG] 7.4 scrub ok 2021-02-10T04:51:55.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:54 smithi164 conmon[33677]: debug 2021-02-10T04:51:54.846+0000 7fbec19a1700 -1 osd.4 1132 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:55.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:54 smithi085 conmon[29319]: cluster 2021-02-10T04:51:53.839711+0000 osd.4 (osd. 2021-02-10T04:51:55.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:54 smithi085 conmon[29319]: 4) 2597 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:55.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:54 smithi085 conmon[29319]: cluster 2021-02-10T04:51:53.839729+0000 osd.4 (osd.4) 2598 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:55.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:54 smithi085 conmon[29319]: cluster 2021-02-10T04:51:54.444283+0000 osd.0 (osd 2021-02-10T04:51:55.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:54 smithi085 conmon[29319]: .0) 170 : cluster [DBG] 7.4 scrub ok 2021-02-10T04:51:55.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:54 smithi085 conmon[35961]: cluster 2021-02-10T04:51:53.839711+0000 2021-02-10T04:51:55.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:54 smithi085 conmon[35961]: osd.4 (osd.4) 2597 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:55.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:54 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:51:55.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:54 smithi085 conmon[35961]: -02-10T04:51:53.839729+0000 osd.4 (osd.4) 2598 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:55.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:54 smithi085 conmon[35961]: cluster 2021-02-10T04:51:55.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:54 smithi085 conmon[35961]: 2021-02-10T04:51:54.444283+0000 osd.0 (osd 2021-02-10T04:51:55.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:54 smithi085 conmon[35961]: .0) 170 : cluster [DBG] 7.4 scrub ok 2021-02-10T04:51:55.502 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:51:55.746 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:51:55.896 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:55 smithi085 conmon[29319]: cluster 2021-02-10T04:51:54.423396+0000 mgr.y (mgr.14140) 2021-02-10T04:51:55.896 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:55 smithi085 conmon[29319]: 1225 : cluster [DBG] pgmap v2030: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 71 KiB/s rd, 71 op/s 2021-02-10T04:51:55.897 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:55 smithi085 conmon[29319]: cluster 2021-02-10T04:51:54.847521+0000 osd.4 (osd.4) 2599 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:56.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:55 smithi164 conmon[33677]: debug 2021-02-10T04:51:56.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:55 smithi164 conmon[33677]: 2021-02-10T04:51:55.800+0000 7fbec19a1700 -1 osd.4 1132 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:55 smithi164 conmon[29288]: cluster 2021-02-10T04:51:54.423396+0000 2021-02-10T04:51:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:55 smithi164 conmon[29288]: mgr.y (mgr.14140) 1225 : cluster [DBG] pgmap v2030: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 71 KiB/s rd, 71 op/s 2021-02-10T04:51:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:55 smithi164 conmon[29288]: cluster 2021-02-10T04:51:54.847521+0000 osd.4 (osd.4) 2599 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:55 smithi164 conmon[29288]: cluster 2021-02-10T04:51:54.847529+0000 osd.4 (osd 2021-02-10T04:51:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:55 smithi164 conmon[29288]: .4) 2600 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:56.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:55 smithi085 conmon[29319]: cluster 2021-02-10T04:51:54.847529+0000 osd.4 (osd.4) 2600 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:56.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:55 smithi085 conmon[35961]: cluster 2021-02-10T04:51:54. 2021-02-10T04:51:56.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:55 smithi085 conmon[35961]: 423396+0000 mgr.y (mgr.14140) 1225 : cluster [DBG] pgmap v2030: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 71 KiB/s rd, 71 op/s 2021-02-10T04:51:56.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:55 smithi085 conmon[35961]: cluster 2021-02-10T04:51 2021-02-10T04:51:56.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:55 smithi085 conmon[35961]: :54.847521+0000 osd.4 (osd.4) 2599 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:56.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:55 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:51:56.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:55 smithi085 conmon[35961]: 04:51:54.847529+0000 osd.4 (osd. 2021-02-10T04:51:56.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:55 smithi085 conmon[35961]: 4) 2600 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:56.465 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set noscrub 2021-02-10T04:51:56.465 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-02-10T04:51:56.466 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:51:56.466 INFO:tasks.thrashosds.thrasher:inject_pause on 2 2021-02-10T04:51:56.466 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2021-02-10T04:51:56.466 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2021-02-10T04:51:56.466 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.2 config set filestore_inject_stall 3 2021-02-10T04:51:56.724 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:51:56.730 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:51:57.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:56 smithi164 conmon[33677]: debug 2021-02-10T04:51:56.786+0000 7fbec19a1700 -1 osd.4 1132 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:56 smithi164 conmon[29288]: cluster 2021-02-10T04:51:55.430392+0000 osd.5 ( 2021-02-10T04:51:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:56 smithi164 conmon[29288]: osd.5) 164 : cluster [DBG] 4.8 deep-scrub ok 2021-02-10T04:51:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:56 smithi164 conmon[29288]: cluster 2021-02-10T04:51:55.801496+0000 osd.4 (osd.4) 2601 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:56 smithi164 conmon[29288]: cluster 2021-02-10T04:51:55.801506+0000 osd.4 (osd.4) 2021-02-10T04:51:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:56 smithi164 conmon[29288]: 2602 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:56 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:51:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:56 smithi164 conmon[29288]: T04:51:55.814480+0000 osd.3 (osd. 2021-02-10T04:51:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:56 smithi164 conmon[29288]: 3) 10 : cluster [DBG] 6.11 scrub ok 2021-02-10T04:51:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:56 smithi164 conmon[29288]: cluster 2021-02-10T04:51:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:56 smithi164 conmon[29288]: 2021- 2021-02-10T04:51:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:56 smithi164 conmon[29288]: 02-10T04:51:56.431179+0000 osd.1 (osd.1 2021-02-10T04:51:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:56 smithi164 conmon[29288]: ) 482 : cluster [DBG] 4.7 scrub ok 2021-02-10T04:51:57.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:56 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:51:57.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:56 smithi085 conmon[29319]: :51:55.430392+0000 osd.5 (osd.5) 164 : cluster [DBG] 4.8 deep-scrub ok 2021-02-10T04:51:57.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:56 smithi085 conmon[29319]: cluster 2021-02-10T04:51:55.801496+0000 osd.4 (osd.4) 2601 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:57.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:56 smithi085 conmon[29319]: cluster 2021-02-10T04:51:55.801506+0000 osd.4 (osd.4) 2602 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:57.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:56 smithi085 conmon[29319]: cluster 2021-02-10T04:51:55.814480+0000 osd.3 (osd.3) 10 : cluster [DBG] 6.11 scrub ok 2021-02-10T04:51:57.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:56 smithi085 conmon[29319]: cluster 2021-02-10T04:51:56.431179+0000 osd.1 (osd.1) 482 : cluster [DBG] 4.7 scrub ok 2021-02-10T04:51:57.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:56 smithi085 conmon[35961]: cluster 2021-02-10T04:51:57.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:56 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:51:57.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:56 smithi085 conmon[35961]: :51:55.430392+0000 osd.5 (osd.5) 164 : cluster [DBG] 4.8 deep-scrub ok 2021-02-10T04:51:57.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:56 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:51:57.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:56 smithi085 conmon[35961]: 02-10T04:51:55.801496+0000 osd.4 2021-02-10T04:51:57.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:56 smithi085 conmon[35961]: (osd.4) 2601 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:57.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:56 smithi085 conmon[35961]: 2021-02-10T04:51:57.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:56 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:51:57.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:56 smithi085 conmon[35961]: -10T04:51:55.801506+0000 osd.4 (osd.4) 2602 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:57.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:56 smithi085 conmon[35961]: 2021-02-10T04:51:57.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:56 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:51:57.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:56 smithi085 conmon[35961]: 51:55.814480+0000 osd.3 (osd.3) 10 : cluster [DBG] 6.11 scrub ok 2021-02-10T04:51:57.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:56 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:51:57.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:56 smithi085 conmon[35961]: :51:56.431179+0000 osd.1 (osd.1) 482 : 2021-02-10T04:51:57.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:56 smithi085 conmon[35961]: cluster [DBG] 4.7 scrub ok 2021-02-10T04:51:58.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:57 smithi164 conmon[33677]: debug 2021-02-10T04:51:57.745+0000 7fbec19a1700 -1 osd.4 1132 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:57 smithi164 conmon[29288]: cluster 2021-02-10T04:51:56.423902+0000 mgr.y (mgr.14140) 2021-02-10T04:51:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:57 smithi164 conmon[29288]: 1226 : cluster [DBG] pgmap v2031: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 61 KiB/s rd, 60 op/s 2021-02-10T04:51:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:57 smithi164 conmon[29288]: cluster 2021-02-10T04:51:56.449077+0000 osd.5 (osd.5) 165 : cluster [DBG] 2.20 scrub ok 2021-02-10T04:51:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:57 smithi164 conmon[29288]: cluster 2021-02-10T04:51:56.787109+0000 osd.4 (osd.4) 2603 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:58.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:57 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:51:58.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:57 smithi164 conmon[29288]: -10T04:51:56.787118+0000 osd.4 (osd.4) 2604 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:58.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:57 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:51:58.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:57 smithi164 conmon[29288]: 04:51:57.121834+0000 osd.2 (osd.2) 1966 : cluster [DBG] 7.e scrub ok 2021-02-10T04:51:58.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:57 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:51:58.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:57 smithi164 conmon[29288]: 02-10T04:51:57.488041+0000 osd.1 (osd. 2021-02-10T04:51:58.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:57 smithi164 conmon[29288]: 1) 483 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:51:58.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:57 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:51:58.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:57 smithi164 conmon[29288]: 04:51:57.490319+0000 osd.1 (osd. 2021-02-10T04:51:58.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:57 smithi164 conmon[29288]: 1) 484 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:51:58.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[35961]: cluster 2021-02-10T04:51:56.423902+0000 mgr.y (mgr 2021-02-10T04:51:58.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[35961]: .14140) 1226 : cluster [DBG] pgmap v2031: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 61 KiB/s rd, 60 op/s 2021-02-10T04:51:58.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[35961]: cluster 2021-02-10T04:51:56.449077+0000 osd.5 (osd.5) 165 : cluster [DBG] 2.20 scrub ok 2021-02-10T04:51:58.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[35961]: 2021-02-10T04:51:58.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[35961]: cluster 2021-02-10T04:51:56.787109+0000 osd.4 (osd.4) 2603 : cluster 2021-02-10T04:51:58.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:58.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[35961]: cluster 2021-02-10T04:51:56.787118+0000 2021-02-10T04:51:58.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[35961]: osd.4 (osd.4) 2604 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:58.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[35961]: cluster 2021-02-10T04:51:58.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[35961]: 2021-02-10T04:51:57.121834+0000 osd.2 (osd.2) 1966 : cluster [DBG] 7.e scrub ok 2021-02-10T04:51:58.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[35961]: 2021-02-10T04:51:58.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:51:58.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[35961]: :51:57.488041+0000 osd.1 (osd.1) 483 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:51:58.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[35961]: cluster 2021-02-10T04:51:58.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[35961]: 2021-02-10T04:51:57.490319 2021-02-10T04:51:58.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 484 2021-02-10T04:51:58.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[35961]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:51:58.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[29319]: cluster 2021-02-10T04:51:58.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:51:58.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[29319]: 51:56.423902+0000 mgr.y (mgr.14140) 1226 : cluster [DBG] pgmap v2031: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 61 KiB/s rd, 60 op/s 2021-02-10T04:51:58.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[29319]: cluster 2021-02-10T04:51: 2021-02-10T04:51:58.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[29319]: 56.449077+0000 osd.5 (osd.5) 165 : cluster [DBG] 2.20 scrub ok 2021-02-10T04:51:58.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[29319]: 2021-02-10T04:51:58.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[29319]: cluster 2021-02-10T04:51:56.787109+0000 osd.4 ( 2021-02-10T04:51:58.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[29319]: osd.4) 2603 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:58.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:51:58.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[29319]: 51:56.787118+0000 osd.4 (osd.4) 2604 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:58.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[29319]: 2021-02-10T04:51:58.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:51:58.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[29319]: 51:57.121834+0000 osd.2 (osd.2) 1966 : cluster [DBG] 7.e scrub ok 2021-02-10T04:51:58.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[29319]: cluster 2021-02-10T04:51:57 2021-02-10T04:51:58.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[29319]: .488041+0000 osd.1 (osd.1) 483 : cluster 2021-02-10T04:51:58.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[29319]: [DBG] purged_snaps scrub starts 2021-02-10T04:51:58.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:51:58.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[29319]: 10T04:51:57.490319+0000 2021-02-10T04:51:58.198 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[29319]: osd.1 (osd.1) 484 : cluster 2021-02-10T04:51:58.198 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:57 smithi085 conmon[29319]: [DBG] purged_snaps scrub ok 2021-02-10T04:51:59.103 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:58 smithi164 conmon[29288]: cluster 2021-02-10T04:51:57.287357+0000 2021-02-10T04:51:59.103 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:58 smithi164 conmon[29288]: osd.6 (osd.6) 163 : cluster [DBG] 2.22 scrub ok 2021-02-10T04:51:59.103 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:58 smithi164 conmon[29288]: cluster 2021-02-10T04:51:57.436855+0000 osd.5 (osd.5) 166 : cluster [DBG] 5.f scrub ok 2021-02-10T04:51:59.103 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:58 smithi164 conmon[29288]: cluster 2021-02-10T04:51:57.746107+0000 2021-02-10T04:51:59.104 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:58 smithi164 conmon[29288]: osd.4 (osd.4) 2605 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:59.104 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:58 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:51:59.104 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:58 smithi164 conmon[29288]: :51:57.746115+0000 osd.4 (osd.4) 2606 : cluster [WRN] 2021-02-10T04:51:59.104 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:58 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:59.104 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:58 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:51:59.104 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:58 smithi164 conmon[29288]: -10T04:51:57.769911+0000 osd.3 (osd.3) 11 : cluster [DBG] 3.2 scrub ok 2021-02-10T04:51:59.104 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:58 smithi164 conmon[29288]: cluster 2021-02-10T04:51:57 2021-02-10T04:51:59.104 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:58 smithi164 conmon[29288]: .909582+0000 mon.a (mon.0) 2352 : cluster [DBG] osdmap e1133: 8 total, 8 up, 8 in 2021-02-10T04:51:59.104 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:58 smithi164 conmon[29288]: cluster 2021-02-10T04:51:58.353656+0000 2021-02-10T04:51:59.104 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:58 smithi164 conmon[29288]: osd.1 (osd.1) 485 : cluster [DBG] 4.f scrub ok 2021-02-10T04:51:59.104 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:58 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:51:59.104 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:58 smithi164 conmon[29288]: T04:51:58.491422+0000 mon.a (mon.0) 2353 : cluster 2021-02-10T04:51:59.104 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:58 smithi164 conmon[29288]: [DBG] osdmap e1134: 8 total, 8 up, 8 in 2021-02-10T04:51:59.105 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:58 smithi164 conmon[33677]: debug 2021-02-10T04:51:58.751+0000 7fbec19a1700 -1 osd.4 1133 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:51:59.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:58 smithi085 conmon[29319]: cluster 2021-02-10T04:51:57 2021-02-10T04:51:59.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:58 smithi085 conmon[29319]: .287357+0000 osd.6 (osd.6) 163 : cluster [DBG] 2.22 scrub ok 2021-02-10T04:51:59.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:58 smithi085 conmon[29319]: cluster 2021-02-10T04:51:57.436855+0000 osd.5 (osd.5) 166 : cluster [DBG] 5.f scrub ok 2021-02-10T04:51:59.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:58 smithi085 conmon[29319]: cluster 2021-02-10T04:51:57.746107+0000 osd.4 (osd.4) 2605 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:59.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:58 smithi085 conmon[29319]: cluster 2021-02-10T04:51:57.746115+0000 osd.4 (osd.4) 2606 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:59.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:58 smithi085 conmon[29319]: cluster 2021-02-10T04:51:57.769911+0000 osd.3 (osd.3) 11 : cluster [DBG] 3.2 scrub ok 2021-02-10T04:51:59.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:58 smithi085 conmon[29319]: cluster 2021-02-10T04:51:57.909582+0000 mon.a (mon.0) 2352 : cluster [DBG] osdmap e1133: 8 total, 8 up, 8 in 2021-02-10T04:51:59.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:58 smithi085 conmon[29319]: cluster 2021-02-10T04:51:58.353656+0000 osd.1 (osd.1) 485 : cluster [DBG] 4.f scrub ok 2021-02-10T04:51:59.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:58 smithi085 conmon[29319]: cluster 2021-02-10T04:51:58.491422+0000 mon.a (mon.0) 2353 : cluster [DBG] osdmap e1134: 8 total, 8 up, 8 in 2021-02-10T04:51:59.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:58 smithi085 conmon[35961]: cluster 2021-02-10T04:51:57 2021-02-10T04:51:59.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:58 smithi085 conmon[35961]: .287357+0000 osd.6 (osd.6) 163 : cluster [DBG] 2.22 scrub ok 2021-02-10T04:51:59.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:58 smithi085 conmon[35961]: cluster 2021-02-10T04:51:57.436855+0000 osd.5 (osd.5) 166 : cluster [DBG] 5.f scrub ok 2021-02-10T04:51:59.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:58 smithi085 conmon[35961]: cluster 2021-02-10T04:51:57.746107+0000 osd.4 (osd.4) 2605 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:51:59.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:58 smithi085 conmon[35961]: cluster 2021-02-10T04:51:57.746115+0000 osd.4 (osd.4) 2606 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:51:59.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:58 smithi085 conmon[35961]: cluster 2021-02-10T04:51:57.769911+0000 osd.3 (osd.3) 11 : cluster [DBG] 3.2 scrub ok 2021-02-10T04:51:59.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:58 smithi085 conmon[35961]: cluster 2021-02-10T04:51:57.909582+0000 mon.a (mon.0) 2352 : cluster [DBG] osdmap e1133: 8 total, 8 up, 8 in 2021-02-10T04:51:59.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:58 smithi085 conmon[35961]: cluster 2021-02-10T04:51:58.353656+0000 osd.1 (osd.1) 485 : cluster [DBG] 4.f scrub ok 2021-02-10T04:51:59.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:58 smithi085 conmon[35961]: cluster 2021-02-10T04:51:58.491422+0000 mon.a (mon.0) 2353 : cluster [DBG] osdmap e1134: 8 total, 8 up, 8 in 2021-02-10T04:51:59.394 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:52:00.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[33677]: debug 2021-02-10T04:51:59.751+0000 7fbec19a1700 -1 osd.4 1134 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:52:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[29288]: cluster 2021-02-10T04:51:58.424378+0000 mgr.y 2021-02-10T04:52:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[29288]: (mgr.14140) 1227 : cluster [DBG] pgmap v2033: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 71 KiB/s rd, 71 op/s 2021-02-10T04:52:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[29288]: cluster 2021-02-10T04:51:58.558131+0000 osd.3 (osd.3) 12 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:52:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[29288]: cluster 2021-02-10T04:51:58. 2021-02-10T04:52:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[29288]: 562360+0000 osd.3 (osd.3) 13 2021-02-10T04:52:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[29288]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:52:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:52:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[29288]: -02-10T04:51: 2021-02-10T04:52:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[29288]: 58.752648+0000 osd.4 (osd.4 2021-02-10T04:52:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[29288]: ) 2607 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:00.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:52:00.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[29288]: -02-10T04:51:58 2021-02-10T04:52:00.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[29288]: .752657+0000 osd.4 (osd.4 2021-02-10T04:52:00.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[29288]: ) 2608 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:00.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:52:00.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[29288]: -02-10T04:51:58 2021-02-10T04:52:00.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[29288]: .912860+0000 mon.a ( 2021-02-10T04:52:00.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[29288]: mon.0) 2354 : cluster [WRN] 2021-02-10T04:52:00.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[29288]: Health check failed: 2 slow ops, oldest one blocked for 452 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:52:00.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[29288]: audit 2021-02-10T04:52:00.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[29288]: 2021-02-10T04:51 2021-02-10T04:52:00.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[29288]: :59.102991+0000 mgr.y (mgr 2021-02-10T04:52:00.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[29288]: .14140) 1228 : audit [DBG] 2021-02-10T04:52:00.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[29288]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:52:00.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:52:00.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[29288]: -02-10T04:51: 2021-02-10T04:52:00.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[29288]: 59.498380+0000 mon.a (mon 2021-02-10T04:52:00.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[29288]: .0) 2355 : cluster [DBG] 2021-02-10T04:52:00.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:51:59 smithi164 conmon[29288]: osdmap e1135: 8 total, 8 up, 8 in 2021-02-10T04:52:00.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:59 smithi085 conmon[29319]: cluster 2021-02-10T04:51:58.424378+0000 mgr.y (mgr.14140) 1227 : cluster [DBG] pgmap v2033: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 71 KiB/s rd, 71 op/s 2021-02-10T04:52:00.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:59 smithi085 conmon[29319]: cluster 2021-02-10T04:51:58.558131+0000 osd.3 (osd.3) 12 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:52:00.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:59 smithi085 conmon[29319]: cluster 2021-02-10T04:51:58.562360+0000 osd.3 (osd.3) 13 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:52:00.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:59 smithi085 conmon[29319]: cluster 2021-02-10T04:51:58.752648+0000 osd.4 (osd.4) 2607 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:00.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:59 smithi085 conmon[29319]: cluster 2021-02-10T04:51:58.752657+0000 osd.4 (osd.4) 2608 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:00.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:59 smithi085 conmon[29319]: cluster 2021-02-10T04:51:58.912860+0000 mon.a (mon.0) 2354 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 452 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:52:00.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:59 smithi085 conmon[29319]: audit 2021-02-10T04:51:59.102991+0000 mgr.y (mgr.14140) 1228 : audit 2021-02-10T04:52:00.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:59 smithi085 conmon[29319]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:52:00.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:51:59 smithi085 conmon[29319]: cluster 2021-02-10T04:51:59.498380+0000 mon.a (mon.0) 2355 : cluster [DBG] osdmap e1135: 8 total, 8 up, 8 in 2021-02-10T04:52:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:59 smithi085 conmon[35961]: cluster 2021-02-10T04:51:58.424378+0000 mgr.y (mgr.14140) 1227 : cluster [DBG] pgmap v2033: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 71 KiB/s rd, 71 op/s 2021-02-10T04:52:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:59 smithi085 conmon[35961]: cluster 2021-02-10T04:51:58.558131+0000 osd.3 (osd.3) 12 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:52:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:59 smithi085 conmon[35961]: cluster 2021-02-10T04:51:58.562360+0000 osd.3 (osd.3) 13 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:52:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:59 smithi085 conmon[35961]: cluster 2021-02-10T04:51:58.752648+0000 osd.4 (osd.4) 2607 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:59 smithi085 conmon[35961]: cluster 2021-02-10T04:51:58.752657+0000 osd.4 (osd.4) 2021-02-10T04:52:00.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:59 smithi085 conmon[35961]: 2608 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:00.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:59 smithi085 conmon[35961]: cluster 2021-02-10T04:51:58.912860+0000 mon.a (mon.0) 2354 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 452 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:52:00.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:59 smithi085 conmon[35961]: audit 2021-02-10T04:51:59.102991+0000 mgr.y 2021-02-10T04:52:00.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:59 smithi085 conmon[35961]: (mgr.14140) 1228 : audit 2021-02-10T04:52:00.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:59 smithi085 conmon[35961]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:52:00.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:59 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:52:00.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:59 smithi085 conmon[35961]: 02-10T04:51:59. 2021-02-10T04:52:00.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:59 smithi085 conmon[35961]: 498380+0000 mon.a (mon.0) 2355 : cluster 2021-02-10T04:52:00.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:51:59 smithi085 conmon[35961]: [DBG] osdmap e1135: 8 total, 8 up, 8 in 2021-02-10T04:52:00.932 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:00 smithi085 conmon[29319]: cluster 2021-02-10T04:51:59.420082+0000 osd.5 (osd.5) 167 : cluster 2021-02-10T04:52:00.933 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:00 smithi085 conmon[29319]: [DBG] 2.2 scrub ok 2021-02-10T04:52:00.933 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:00 smithi085 conmon[29319]: cluster 2021-02-10T04:51:59.752096+0000 osd.4 (osd.4) 2609 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:00.933 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:00 smithi085 conmon[29319]: cluster 2021-02-10T04:51:59.752105+0000 osd.4 (osd 2021-02-10T04:52:00.933 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:00 smithi085 conmon[29319]: .4) 2610 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:00.933 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:00 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:52:00.933 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:00 smithi085 conmon[29319]: 10T04:51:59.838786+0000 osd.3 (osd.3 2021-02-10T04:52:00.933 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:00 smithi085 conmon[29319]: ) 14 : cluster [DBG] 6.1b scrub ok 2021-02-10T04:52:00.933 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:00 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:52:00.933 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:00 smithi085 conmon[29319]: T04:52:00.501121+0000 osd.0 (osd.0 2021-02-10T04:52:00.933 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:00 smithi085 conmon[29319]: ) 171 : cluster [DBG] 6.18 scrub ok 2021-02-10T04:52:00.933 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:00 smithi085 conmon[35961]: cluster 2021-02-10T04:51:59. 2021-02-10T04:52:00.934 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:00 smithi085 conmon[35961]: 420082+0000 osd.5 (osd.5) 167 : cluster [DBG] 2.2 scrub ok 2021-02-10T04:52:00.934 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:00 smithi085 conmon[35961]: cluster 2021-02-10T04:52:00.934 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:00 smithi085 conmon[35961]: 2021-02-10T04:51:59.752096+0000 osd.4 2021-02-10T04:52:00.934 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:00 smithi085 conmon[35961]: (osd.4) 2609 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:00.934 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:00 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:52:00.934 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:00 smithi085 conmon[35961]: 04:51:59.752105+0000 osd.4 (osd.4) 2610 : cluster 2021-02-10T04:52:00.934 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:00 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:00.934 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:00 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:52:00.934 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:00 smithi085 conmon[35961]: :51:59.838786+0000 osd.3 (osd.3 2021-02-10T04:52:00.934 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:00 smithi085 conmon[35961]: ) 14 : cluster [DBG] 6.1b scrub ok 2021-02-10T04:52:00.934 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:00 smithi085 conmon[35961]: cluster 2021-02-10T04:52:00.934 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:00 smithi085 conmon[35961]: 2021-02-10T04:52:00. 2021-02-10T04:52:00.934 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:00 smithi085 conmon[35961]: 501121+0000 osd.0 (osd.0) 171 : cluster 2021-02-10T04:52:00.935 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:00 smithi085 conmon[35961]: [DBG] 6.18 scrub ok 2021-02-10T04:52:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:00 smithi164 conmon[29288]: cluster 2021-02-10T04:51:59.420082+0000 osd.5 2021-02-10T04:52:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:00 smithi164 conmon[29288]: (osd.5) 167 : cluster [DBG] 2.2 scrub ok 2021-02-10T04:52:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:00 smithi164 conmon[29288]: cluster 2021-02-10T04:51:59.752096+0000 osd.4 (osd.4) 2609 2021-02-10T04:52:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:00 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:00 smithi164 conmon[29288]: cluster 2021-02-10T04:51:59.752105 2021-02-10T04:52:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:00 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2610 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:00 smithi164 conmon[29288]: 2021-02-10T04:52:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:00 smithi164 conmon[29288]: cluster 2021-02-10T04:52:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:00 smithi164 conmon[29288]: 2021-02-10T04:51:59.838786+0000 osd.3 ( 2021-02-10T04:52:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:00 smithi164 conmon[29288]: osd.3) 14 : cluster [DBG] 6.1b scrub ok 2021-02-10T04:52:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:00 smithi164 conmon[29288]: cluster 2021-02-10T04:52:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:00 smithi164 conmon[29288]: 2021-02-10T04:52:00.501121+0000 2021-02-10T04:52:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:00 smithi164 conmon[29288]: osd.0 (osd.0) 171 : cluster [DBG] 2021-02-10T04:52:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:00 smithi164 conmon[29288]: 6.18 scrub ok 2021-02-10T04:52:01.145 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:52:00 smithi164 conmon[33677]: debug 2021-02-10T04:52:00.753+0000 7fbec19a1700 -1 osd.4 1135 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:52:01.518 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:52:01.538 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:52:01.557 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:52:01.578 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:52:01.603 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:52:01.623 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:52:01.644 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:52:01.666 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:52:01.667 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:52:01.667 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:52:01.667 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:52:01.667 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:52:01.667 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:52:01.667 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:52:01.667 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:52:01.667 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:52:02.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[29319]: cluster 2021-02-10T04:52:02.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:52:02.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[29319]: 04:52:00.424851+0000 mgr.y (mgr.14140) 1229 : cluster [DBG] pgmap v2036: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:52:02.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[29319]: cluster 2021-02-10T04:52: 2021-02-10T04:52:02.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[29319]: 00.754196+0000 osd.4 (osd.4) 2611 : cluster 2021-02-10T04:52:02.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:02.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:52:02.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[29319]: -02-10T04:52:00.754205+0000 osd.4 (osd.4) 2612 2021-02-10T04:52:02.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:02.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:52:02.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[29319]: -10T04:52:01.121377+0000 2021-02-10T04:52:02.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[29319]: osd.2 (osd.2) 1967 : cluster [DBG] 2021-02-10T04:52:02.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[29319]: 2.c scrub ok 2021-02-10T04:52:02.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:52:02.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[29319]: 04:52:01.483062+0000 osd.0 ( 2021-02-10T04:52:02.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[29319]: osd.0) 172 : cluster [DBG] 4.3 deep-scrub ok 2021-02-10T04:52:02.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[29319]: audit 2021-02-10T04:52:02.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[29319]: 2021-02-10T04:52:01. 2021-02-10T04:52:02.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[29319]: 494770+0000 mon.a (mon.0) 2356 : 2021-02-10T04:52:02.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[29319]: audit [DBG] from='client.? 172.21.15.85:0/886064744' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:52:02.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[35961]: cluster 2021-02-10T04:52:00.424851+0000 mgr.y (mgr. 2021-02-10T04:52:02.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[35961]: 14140) 1229 : cluster [DBG] pgmap v2036: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:52:02.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[35961]: cluster 2021-02-10T04:52:00.754196 2021-02-10T04:52:02.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2611 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:02.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:52:02.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[35961]: 02-10T04:52:00.754205+0000 osd.4 (osd 2021-02-10T04:52:02.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[35961]: .4) 2612 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:02.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[35961]: cluster 2021-02-10T04:52:01.121377 2021-02-10T04:52:02.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1967 : cluster [DBG] 2.c scrub ok 2021-02-10T04:52:02.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[35961]: cluster 2021-02-10T04:52:01. 2021-02-10T04:52:02.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[35961]: 483062+0000 osd.0 (osd.0) 172 : cluster [DBG] 4.3 deep-scrub ok 2021-02-10T04:52:02.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:52:02.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[35961]: 52:01.494770+0000 mon.a (mon.0) 2356 : audit 2021-02-10T04:52:02.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:01 smithi085 conmon[35961]: [DBG] from='client.? 172.21.15.85:0/886064744' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:52:02.252 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:01 smithi164 conmon[29288]: cluster 2021-02-10T04:52:00.424851+0000 mgr.y (mgr.14140) 1229 2021-02-10T04:52:02.252 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:01 smithi164 conmon[29288]: : cluster [DBG] pgmap v2036: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:52:02.252 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:01 smithi164 conmon[29288]: cluster 2021-02-10T04:52:00.754196+0000 osd.4 (osd.4) 2611 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:02.252 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:01 smithi164 conmon[29288]: cluster 2021-02-10T04:52:00.754205+0000 osd.4 (osd.4) 2612 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:02.252 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:01 smithi164 conmon[29288]: cluster 2021-02-10T04:52:01.121377+0000 osd.2 (osd.2) 1967 : cluster [DBG] 2.c scrub ok 2021-02-10T04:52:02.252 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:01 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:52:02.252 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:01 smithi164 conmon[29288]: -10T04:52:01.483062+0000 osd.0 (osd.0) 172 : cluster [DBG] 4.3 deep-scrub ok 2021-02-10T04:52:02.252 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:01 smithi164 conmon[29288]: audit 2021-02-10T04:52:02.252 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:01 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:52:02.252 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:01 smithi164 conmon[29288]: 52:01.494770+0000 mon.a (mon.0) 2356 2021-02-10T04:52:02.253 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:01 smithi164 conmon[29288]: : audit [DBG] from='client.? 172.21.15.85:0/886064744' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:52:02.320 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:52:02.320 INFO:teuthology.orchestra.run.smithi085.stdout: "success": "" 2021-02-10T04:52:02.320 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:52:02.853 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:52:02.854 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:52:02.946 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is set 2021-02-10T04:52:03.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:02 smithi164 conmon[29288]: cluster 2021-02-10T04:52:02.162848 2021-02-10T04:52:03.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:02 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1968 : cluster [DBG] 6.f scrub ok 2021-02-10T04:52:03.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:02 smithi164 conmon[29288]: audit 2021-02-10T04:52:02.189888+0000 mon.a (mon.0) 2357 : audit [INF] from='client.? 172.21.15.85:0/2380308267' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:52:03.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:02 smithi085 conmon[29319]: cluster 2021-02-10T04:52:02.162848+0000 2021-02-10T04:52:03.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:02 smithi085 conmon[29319]: osd.2 (osd.2) 1968 : cluster [DBG] 6.f scrub ok 2021-02-10T04:52:03.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:02 smithi085 conmon[29319]: audit 2021-02-10T04:52:02.189888+0000 mon.a (mon.0) 2357 : audit [INF] from='client.? 172.21.15.85:0/2380308267' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:52:03.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:02 smithi085 conmon[35961]: cluster 2021-02-10T04:52:02 2021-02-10T04:52:03.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:02 smithi085 conmon[35961]: .162848+0000 osd.2 (osd.2) 1968 : cluster [DBG] 6.f scrub ok 2021-02-10T04:52:03.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:02 smithi085 conmon[35961]: audit 2021 2021-02-10T04:52:03.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:02 smithi085 conmon[35961]: -02-10T04:52:02.189888+0000 mon.a (mon.0) 2357 : 2021-02-10T04:52:03.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:02 smithi085 conmon[35961]: audit [INF] from='client.? 172.21.15.85:0/2380308267' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:52:03.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[35961]: cluster 2021-02-10T04:52:03.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[35961]: 2021-02 2021-02-10T04:52:03.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[35961]: -10T04:52:02.425355+0000 mgr.y (mgr.14140) 1230 : cluster [DBG] 2021-02-10T04:52:03.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[35961]: pgmap v2037: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:52:03.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:52:03.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[35961]: 04:52:02.933584+0000 mon.a (mon. 2021-02-10T04:52:03.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[35961]: 0) 2358 : cluster [INF] Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 452 sec, osd.4 has slow ops) 2021-02-10T04:52:03.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:52:03.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[35961]: 02-10T04:52:02.933616+0000 mon.a 2021-02-10T04:52:03.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[35961]: (mon.0) 2359 : cluster [INF] Cluster is now healthy 2021-02-10T04:52:03.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[35961]: 2021-02-10T04:52:03.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[35961]: cluster 2021-02-10T04:52:02 2021-02-10T04:52:03.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[35961]: .934248+0000 mon.a (mon.0) 2360 : 2021-02-10T04:52:03.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[35961]: cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:52:03.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[35961]: audit 2021-02-10T04:52:02.941949+0000 mon.a ( 2021-02-10T04:52:03.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[35961]: mon.0) 2361 : audit [INF] from='client.? 172.21.15.85:0/2380308267' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:52:03.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[35961]: cluster 2021-02-10T04:52 2021-02-10T04:52:03.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[35961]: :02.941983+0000 mon.a (mon.0) 2021-02-10T04:52:03.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[35961]: 2362 : cluster [DBG] osdmap e1136: 8 total, 8 up, 8 in 2021-02-10T04:52:04.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[29319]: cluster 2021-02-10T04:52:02.425355+0000 mgr.y (mgr.14140) 1230 : 2021-02-10T04:52:04.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[29319]: cluster [DBG] pgmap v2037: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:52:04.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[29319]: cluster 2021-02-10T04:52:02.933584+0000 mon.a (mon.0) 2358 : cluster [INF] Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 452 sec, osd.4 has slow ops) 2021-02-10T04:52:04.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[29319]: cluster 2021-02-10T04:52:02.933616 2021-02-10T04:52:04.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2359 : cluster [INF] Cluster is now healthy 2021-02-10T04:52:04.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:52:04.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[29319]: 10T04:52:02.934248+0000 mon.a (mon.0) 2360 : cluster 2021-02-10T04:52:04.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[29319]: [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:52:04.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[29319]: audit 2021-02-10T04: 2021-02-10T04:52:04.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[29319]: 52:02.941949+0000 mon.a (mon.0) 2361 : audit [INF] from='client.? 172.21.15.85:0/2380308267' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:52:04.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[29319]: cluster 2021-02-10T04:52:04.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[29319]: 2021-02-10T04:52:02.941983+0000 mon.a (mon.0) 2021-02-10T04:52:04.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:03 smithi085 conmon[29319]: 2362 : cluster [DBG] osdmap e1136: 8 total, 8 up, 8 in 2021-02-10T04:52:04.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:03 smithi164 conmon[29288]: cluster 2021-02-10T04:52:04.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:03 smithi164 conmon[29288]: 2021-02-10T04:52 2021-02-10T04:52:04.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:03 smithi164 conmon[29288]: :02.425355+0000 mgr.y (mgr.14140) 1230 : cluster [DBG] pgmap v2037: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:52:04.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:03 smithi164 conmon[29288]: cluster 2021-02-10T04:52 2021-02-10T04:52:04.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:03 smithi164 conmon[29288]: :02.933584+0000 mon.a (mon.0) 2358 : cluster 2021-02-10T04:52:04.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:03 smithi164 conmon[29288]: [INF] Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 452 sec, osd.4 has slow ops) 2021-02-10T04:52:04.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:03 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:52:04.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:03 smithi164 conmon[29288]: -02-10T04:52:02.933616+0000 2021-02-10T04:52:04.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:03 smithi164 conmon[29288]: mon.a (mon.0) 2359 : cluster [INF] 2021-02-10T04:52:04.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:03 smithi164 conmon[29288]: Cluster is now healthy 2021-02-10T04:52:04.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:03 smithi164 conmon[29288]: cluster 2021-02-10T04:52:02. 2021-02-10T04:52:04.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:03 smithi164 conmon[29288]: 934248+0000 mon.a (mon.0) 2360 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:52:04.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:03 smithi164 conmon[29288]: 2021-02-10T04:52:04.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:03 smithi164 conmon[29288]: audit 2021-02-10T04:52:02.941949 2021-02-10T04:52:04.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:03 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2361 : audit 2021-02-10T04:52:04.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:03 smithi164 conmon[29288]: [INF] from='client.? 172.21.15.85:0/2380308267' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:52:04.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:03 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:52:04.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:03 smithi164 conmon[29288]: -10T04:52:02.941983 2021-02-10T04:52:04.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:03 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2362 : 2021-02-10T04:52:04.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:03 smithi164 conmon[29288]: cluster [DBG] osdmap e1136: 8 total, 8 up, 8 in 2021-02-10T04:52:06.260 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:52:06.288 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:05 smithi085 conmon[29319]: cluster 2021-02-10T04:52:06.288 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:05 smithi085 conmon[29319]: 2021-02-10T04:52: 2021-02-10T04:52:06.288 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:05 smithi085 conmon[29319]: 04.425990+0000 mgr.y (mgr.14140) 1231 : cluster [DBG] pgmap v2039: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:52:06.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:05 smithi085 conmon[35961]: cluster 2021-02-10T04:52:04. 2021-02-10T04:52:06.289 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:05 smithi085 conmon[35961]: 425990+0000 mgr.y (mgr.14140) 1231 : cluster [DBG] pgmap v2039: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:52:06.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:05 smithi164 conmon[29288]: cluster 2021-02-10T04:52:04.425990+0000 mgr.y (mgr.14140) 2021-02-10T04:52:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:05 smithi164 conmon[29288]: 1231 : cluster [DBG] pgmap v2039: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:52:06.504 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:52:07.192 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set nodeep-scrub 2021-02-10T04:52:07.428 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:52:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:07 smithi164 conmon[29288]: cluster 2021-02-10T04:52:06.426492+0000 mgr.y (mgr.14140) 2021-02-10T04:52:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:07 smithi164 conmon[29288]: 1232 : cluster [DBG] pgmap v2040: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 645 B/s rd, 0 op/s 2021-02-10T04:52:08.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:07 smithi085 conmon[29319]: cluster 2021-02-10T04:52:06.426492+0000 mgr.y (mgr.14140) 1232 : cluster [DBG] pgmap v2040: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 645 B/s rd, 0 op/s 2021-02-10T04:52:08.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:07 smithi085 conmon[35961]: cluster 2021-02-10T04:52:06.426492+0000 mgr.y (mgr.14140) 1232 : cluster [DBG] pgmap v2040: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 645 B/s rd, 0 op/s 2021-02-10T04:52:09.394 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:52:09 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:52:09.935 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-02-10T04:52:09.935 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:52:09.935 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2021-02-10T04:52:09.936 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:52:10.334 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:52:10.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:09 smithi164 conmon[29288]: cluster 2021-02-10T04:52:08.427117+0000 mgr.y (mgr.14140 2021-02-10T04:52:10.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:09 smithi164 conmon[29288]: ) 1233 : cluster [DBG] pgmap v2041: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:52:10.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:09 smithi164 conmon[29288]: audit 2021-02-10T04:52:09.113033+0000 mgr.y (mgr.14140) 1234 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:52:10.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:09 smithi164 conmon[29288]: audit 2021-02-10T04:52:09.769221+0000 mon.a (mon.0) 2363 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:52:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:09 smithi085 conmon[29319]: cluster 2021-02-10T04:52:08.427117+0000 mgr.y (mgr 2021-02-10T04:52:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:09 smithi085 conmon[29319]: .14140) 1233 : cluster [DBG] pgmap v2041: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:52:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:09 smithi085 conmon[29319]: audit 2021-02-10T04:52:09.113033+0000 mgr.y (mgr.14140) 1234 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:52:10.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:09 smithi085 conmon[29319]: audit 2021-02-10T04:52:09.769221+0000 mon.a (mon.0) 2363 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:52:10.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:09 smithi085 conmon[35961]: cluster 2021-02-10T04:52:08.427117 2021-02-10T04:52:10.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:09 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1233 : cluster [DBG] pgmap v2041: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:52:10.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:09 smithi085 conmon[35961]: audit 2021-02-10T04:52:10.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:09 smithi085 conmon[35961]: 2021-02-10T04:52:09.113033+0000 mgr.y (mgr.14140) 1234 : audit 2021-02-10T04:52:10.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:09 smithi085 conmon[35961]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:52:10.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:09 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:52:10.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:09 smithi085 conmon[35961]: -10T04:52:09.769221+0000 mon.a 2021-02-10T04:52:10.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:09 smithi085 conmon[35961]: (mon.0) 2363 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:52:11.117 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:10 smithi085 conmon[29319]: cephadm 2021-02-10T04:52:09.355696+0000 mgr.y (mgr. 2021-02-10T04:52:11.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:10 smithi085 conmon[29319]: 14140) 1235 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:52:11.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:10 smithi085 conmon[29319]: cephadm 2021-02-10T04:52:09.769881+0000 mgr.y (mgr.14140) 1236 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:52:11.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:10 smithi085 conmon[29319]: cluster 2021-02-10T04:52:09.905163+0000 osd.5 (osd.5) 168 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:52:11.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:10 smithi085 conmon[29319]: cluster 2021-02-10T04:52:09.907899+0000 osd.5 (osd.5) 169 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:52:11.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:10 smithi085 conmon[29319]: cluster 2021-02-10T04:52:10.771967+0000 mon.a (mon.0) 2364 : cluster [DBG] osdmap e1137: 8 total, 8 up, 8 in 2021-02-10T04:52:11.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:10 smithi085 conmon[29319]: audit 2021-02-10T04:52:10.899616+0000 mon.a (mon.0) 2365 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:52:11.118 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:10 smithi085 conmon[35961]: cephadm 2021-02-10T04:52:09.355696+0000 2021-02-10T04:52:11.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:10 smithi085 conmon[35961]: mgr.y (mgr.14140) 1235 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:52:11.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:10 smithi085 conmon[35961]: cephadm 2021 2021-02-10T04:52:11.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:10 smithi085 conmon[35961]: -02-10T04:52:09.769881+0000 mgr.y (mgr.14140) 1236 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:52:11.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:10 smithi085 conmon[35961]: 2021-02-10T04:52:11.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:10 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:52:11.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:10 smithi085 conmon[35961]: :52:09.905163+0000 osd.5 (osd.5) 2021-02-10T04:52:11.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:10 smithi085 conmon[35961]: 168 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:52:11.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:10 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:52:11.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:10 smithi085 conmon[35961]: 10T04:52:09.907899+0000 osd.5 2021-02-10T04:52:11.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:10 smithi085 conmon[35961]: (osd.5) 169 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:52:11.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:10 smithi085 conmon[35961]: 2021-02-10T04:52:11.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:10 smithi085 conmon[35961]: cluster 2021-02-10T04:52: 2021-02-10T04:52:11.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:10 smithi085 conmon[35961]: 10.771967+0000 mon.a (mon.0) 2364 : 2021-02-10T04:52:11.120 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:10 smithi085 conmon[35961]: cluster [DBG] osdmap e1137: 8 total, 8 up, 8 in 2021-02-10T04:52:11.120 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:10 smithi085 conmon[35961]: audit 2021-02-10T04:52: 2021-02-10T04:52:11.120 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:10 smithi085 conmon[35961]: 10.899616+0000 mon.a (mon.0) 2365 : audit 2021-02-10T04:52:11.120 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:10 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:52:11.322 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:52:11.347 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:52:11.372 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:52:11.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:10 smithi164 conmon[29288]: cephadm 2021-02-10T04:52:09.355696+0000 mgr.y (mgr.14140) 1235 : cephadm 2021-02-10T04:52:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:10 smithi164 conmon[29288]: [INF] Refreshing smithi164 facts 2021-02-10T04:52:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:10 smithi164 conmon[29288]: cephadm 2021-02-10T04:52:09.769881+0000 mgr.y (mgr.14140) 1236 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:52:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:10 smithi164 conmon[29288]: cluster 2021-02-10T04:52:09.905163+0000 osd.5 (osd.5) 168 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:52:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:10 smithi164 conmon[29288]: cluster 2021-02-10T04:52:09.907899+0000 osd.5 (osd.5) 169 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:52:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:10 smithi164 conmon[29288]: cluster 2021-02-10T04:52:10.771967+0000 mon.a (mon.0) 2364 : cluster [DBG] osdmap e1137: 8 total, 8 up, 8 in 2021-02-10T04:52:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:10 smithi164 conmon[29288]: audit 2021-02-10T04:52:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:10 smithi164 conmon[29288]: 2021-02 2021-02-10T04:52:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:10 smithi164 conmon[29288]: -10T04:52:10.899616 2021-02-10T04:52:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:10 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2365 : audit [INF] 2021-02-10T04:52:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:10 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:52:11.400 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:52:11.426 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:52:11.449 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:52:11.472 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:52:11.491 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:52:11.493 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:52:11.493 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:52:11.493 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:52:11.493 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:52:11.493 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:52:11.493 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:52:11.493 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:52:11.493 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:52:11.965 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is set 2021-02-10T04:52:12.084 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:11 smithi085 conmon[29319]: cluster 2021-02-10T04:52:10.427640+0000 mgr.y (mgr.14140) 1237 : cluster [DBG] pgmap v2042: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:52:12.085 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:11 smithi085 conmon[29319]: audit 2021-02-10T04:52:11.297473+0000 mon.a (mon.0) 2366 : audit [DBG] from='client.? 172.21.15.85:0/2945651085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:52:12.085 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:11 smithi085 conmon[29319]: audit 2021-02-10T04:52:11.823076+0000 mon.a (mon.0) 2367 : audit [INF] from='client.? 172.21.15.85:0/2204171674' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:52:12.085 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:11 smithi085 conmon[35961]: cluster 2021-02-10T04:52:10.427640+0000 mgr.y (mgr.14140) 1237 : cluster [DBG] pgmap v2042: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:52:12.085 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:11 smithi085 conmon[35961]: audit 2021-02-10T04:52:11.297473+0000 mon.a (mon.0) 2366 : audit [DBG] from='client.? 172.21.15.85:0/2945651085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:52:12.085 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:11 smithi085 conmon[35961]: audit 2021-02-10T04:52:11.823076+0000 mon.a (mon.0) 2367 : audit [INF] from='client.? 172.21.15.85:0/2204171674' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:52:12.145 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:52:11 smithi164 conmon[33677]: debug 2021-02-10T04:52:11.742+0000 7fbec19a1700 -1 osd.4 1137 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:52:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:11 smithi164 conmon[29288]: cluster 2021-02-10T04:52:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:11 smithi164 conmon[29288]: 2021-02- 2021-02-10T04:52:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:11 smithi164 conmon[29288]: 10T04:52:10.427640+0000 mgr.y (mgr.14140) 1237 : cluster [DBG] pgmap v2042: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:52:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:11 smithi164 conmon[29288]: audit 2021-02-10T04:52:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:11 smithi164 conmon[29288]: 2021- 2021-02-10T04:52:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:11 smithi164 conmon[29288]: 02-10T04:52:11.297473+0000 mon.a (mon.0) 2021-02-10T04:52:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:11 smithi164 conmon[29288]: 2366 : audit [DBG] from='client.? 172.21.15.85:0/2945651085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:52:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:11 smithi164 conmon[29288]: audit 2021-02-10T04:52:12.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:11 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:52:12.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:11 smithi164 conmon[29288]: 04:52:11.823076+0000 mon.a (mon. 2021-02-10T04:52:12.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:11 smithi164 conmon[29288]: 0) 2367 : audit [INF] from='client.? 172.21.15.85:0/2204171674' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:52:12.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[49106]: debug 2021-02-10T04:52:12.083+0000 7f77a3e94700 -1 osd.2 1138 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) 2021-02-10T04:52:13.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:52:12 smithi164 conmon[33677]: debug 2021-02-10T04:52:12.779+0000 7fbec19a1700 -1 osd.4 1138 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:52:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:12 smithi164 conmon[29288]: cluster 2021-02-10T04:52:11.744537+0000 2021-02-10T04:52:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:12 smithi164 conmon[29288]: osd.4 (osd.4) 2613 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:12 smithi164 conmon[29288]: cluster 2021-02-10T04:52:11.744546+0000 osd.4 (osd.4) 2614 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:12 smithi164 conmon[29288]: cluster 2021-02-10T04:52:11.953813 2021-02-10T04:52:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:12 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2368 : 2021-02-10T04:52:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:12 smithi164 conmon[29288]: cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:52:13.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:12 smithi164 conmon[29288]: audit 2021-02-10T04:52:11.960896+0000 2021-02-10T04:52:13.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:12 smithi164 conmon[29288]: mon.a (mon.0) 2369 : audit [INF] from='client.? 172.21.15.85:0/2204171674' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:52:13.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:12 smithi164 conmon[29288]: cluster 2021-02-10T04:52:11.960968+0000 mon.a ( 2021-02-10T04:52:13.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:12 smithi164 conmon[29288]: mon.0) 2370 : cluster [DBG] osdmap e1138: 8 total, 8 up, 8 in 2021-02-10T04:52:13.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:12 smithi164 conmon[29288]: cluster 2021-02-10T04:52:12.084364+0000 osd.2 (osd.2 2021-02-10T04:52:13.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:12 smithi164 conmon[29288]: ) 1969 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:13.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:12 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:52:13.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:12 smithi164 conmon[29288]: 10T04:52:12.084375+0000 osd.2 (osd.2) 1970 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:13.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[49106]: debug 2021-02-10T04:52:13.080+0000 7f77a3e94700 -1 osd.2 1138 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) 2021-02-10T04:52:13.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[29319]: cluster 2021-02-10T04:52:11.744537+0000 osd.4 (osd. 2021-02-10T04:52:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[29319]: 4) 2613 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[29319]: cluster 2021-02-10T04:52:11.744546+0000 osd.4 (osd.4) 2614 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[29319]: cluster 2021-02-10T04:52:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[29319]: 2021- 2021-02-10T04:52:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[29319]: 02-10T04:52: 2021-02-10T04:52:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[29319]: 11.953813+0000 mon.a (mon.0 2021-02-10T04:52:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[29319]: ) 2368 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:52:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[29319]: audit 2021 2021-02-10T04:52:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[29319]: -02-10T04:52:11 2021-02-10T04:52:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[29319]: .960896+0000 mon.a (mon. 2021-02-10T04:52:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[29319]: 0) 2369 : audit [INF] 2021-02-10T04:52:13.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[29319]: from='client.? 172.21.15.85:0/2204171674' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:52:13.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:52:13.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[29319]: -02-10T04:52:11.960968+0000 mon.a 2021-02-10T04:52:13.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[29319]: (mon.0) 2370 : cluster [DBG] osdmap e1138: 8 total, 8 up, 8 in 2021-02-10T04:52:13.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:52:13.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[29319]: 02-10T04:52:12.084364+0000 osd.2 2021-02-10T04:52:13.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[29319]: (osd.2) 1969 : cluster 2021-02-10T04:52:13.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:13.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:52:13.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[29319]: 02-10T04:52:12.084375+0000 osd.2 2021-02-10T04:52:13.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[29319]: (osd.2) 1970 : cluster [WRN] 2021-02-10T04:52:13.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:13.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[35961]: cluster 2021-02-10T04:52: 2021-02-10T04:52:13.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[35961]: 11.744537+0000 osd.4 (osd.4) 2613 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:13.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:52:13.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[35961]: T04:52:11.744546+0000 osd.4 (osd. 2021-02-10T04:52:13.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[35961]: 4) 2614 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:13.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:52:13.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[35961]: T04:52:11.953813+0000 mon.a ( 2021-02-10T04:52:13.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[35961]: mon.0) 2368 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:52:13.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[35961]: 2021-02-10T04:52:13.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[35961]: audit 2021-02-10T04:52:13.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[35961]: 2021-02-10T04:52:11.960896+0000 mon.a ( 2021-02-10T04:52:13.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[35961]: mon.0) 2369 : audit [INF] from='client.? 172.21.15.85:0/2204171674' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:52:13.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:52:13.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[35961]: 10T04:52:11.960968+0000 mon.a 2021-02-10T04:52:13.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[35961]: (mon.0) 2370 : cluster [DBG] 2021-02-10T04:52:13.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[35961]: osdmap e1138: 8 total, 8 up, 8 in 2021-02-10T04:52:13.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[35961]: cluster 2021-02-10T04:52:13.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[35961]: 2021-02-10T04:52:12.084364+0000 osd.2 ( 2021-02-10T04:52:13.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[35961]: osd.2) 1969 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:13.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:52:13.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[35961]: -10T04:52:12. 2021-02-10T04:52:13.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[35961]: 084375+0000 osd.2 (osd.2 2021-02-10T04:52:13.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[35961]: ) 1970 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:13.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:12 smithi085 conmon[35961]: 2021-02-10T04:52:13.759 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.4 dump_blocked_ops 2021-02-10T04:52:14.064 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:52:14.071 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[35961]: cluster 2021-02-10T04:52:12.428145 2021-02-10T04:52:14.071 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1238 : cluster [DBG] pgmap v2045: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:52:14.071 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[35961]: cluster 2021-02-10T04:52:12. 2021-02-10T04:52:14.071 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[35961]: 780856+0000 osd.4 (osd.4) 2615 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:14.071 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:52:14.071 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[35961]: 52:12.780866+0000 osd.4 (osd.4 2021-02-10T04:52:14.071 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[35961]: ) 2616 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:14.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:52:14.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[35961]: T04:52:12.958852+0000 mon.a ( 2021-02-10T04:52:14.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[35961]: mon.0) 2371 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 467 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:52:14.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:52:14.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[35961]: T04:52:13.080819+0000 osd.2 ( 2021-02-10T04:52:14.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[35961]: osd.2) 1971 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:14.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:52:14.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[35961]: -10T04:52:13.080826+0000 2021-02-10T04:52:14.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[35961]: osd.2 (osd.2) 1972 : cluster [WRN] 2021-02-10T04:52:14.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:14.073 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[29319]: cluster 2021-02-10T04:52:12.428145+0000 mgr.y (mgr.14140) 1238 : cluster 2021-02-10T04:52:14.073 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[29319]: [DBG] pgmap v2045: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:52:14.073 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[29319]: cluster 2021-02-10T04:52:12.780856+0000 osd.4 (osd.4) 2615 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:14.073 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[29319]: cluster 2021-02-10T04:52: 2021-02-10T04:52:14.073 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[29319]: 12.780866+0000 osd.4 (osd.4) 2616 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:14.073 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[29319]: cluster 2021-02-10T04:52:12.958852+0000 2021-02-10T04:52:14.073 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[29319]: mon.a (mon.0) 2371 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 467 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:52:14.073 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:52:14.073 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[29319]: :52:13.080819+0000 2021-02-10T04:52:14.073 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[29319]: osd.2 (osd.2) 1971 : cluster 2021-02-10T04:52:14.073 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:14.074 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:52:14.074 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[29319]: 10T04:52:13. 2021-02-10T04:52:14.074 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[29319]: 080826+0000 osd.2 (osd.2) 2021-02-10T04:52:14.074 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:13 smithi085 conmon[29319]: 1972 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:14.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:52:13 smithi164 conmon[33677]: debug 2021-02-10T04:52:14.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:52:13 smithi164 conmon[33677]: 2021-02-10T04:52:13.802+0000 7fbec19a1700 -1 osd.4 1138 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:52:14.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:52:13 smithi164 conmon[33677]: 2021-02-10T04:52:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:13 smithi164 conmon[29288]: cluster 2021-02-10T04:52:12.428145+0000 mgr.y 2021-02-10T04:52:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:13 smithi164 conmon[29288]: (mgr.14140) 1238 : cluster [DBG] pgmap v2045: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:52:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:13 smithi164 conmon[29288]: cluster 2021-02-10T04:52:12.780856+0000 osd.4 (osd.4) 2021-02-10T04:52:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:13 smithi164 conmon[29288]: 2615 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:13 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:52:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:13 smithi164 conmon[29288]: 10T04:52:12.780866+0000 osd.4 (osd.4) 2616 : cluster [WRN] 2021-02-10T04:52:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:13 smithi164 conmon[29288]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:13 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:52:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:13 smithi164 conmon[29288]: 02-10T04:52:12 2021-02-10T04:52:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:13 smithi164 conmon[29288]: .958852+0000 mon.a (mon.0 2021-02-10T04:52:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:13 smithi164 conmon[29288]: ) 2371 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 467 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:52:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:13 smithi164 conmon[29288]: 2021-02-10T04:52:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:13 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:52:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:13 smithi164 conmon[29288]: -10T04:52:13 2021-02-10T04:52:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:13 smithi164 conmon[29288]: .080819+0000 osd.2 (osd.2 2021-02-10T04:52:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:13 smithi164 conmon[29288]: ) 1971 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:13 smithi164 conmon[29288]: 2021-02-10T04:52:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:13 smithi164 conmon[29288]: cluster 2021-02-10T04:52:14.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:13 smithi164 conmon[29288]: 2021-02-10T04:52:13.080826+0000 osd.2 (osd 2021-02-10T04:52:14.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:13 smithi164 conmon[29288]: .2) 1972 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:14.359 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:52:14 smithi085 conmon[49106]: debug 2021-02-10T04:52:14.069+0000 7f77a3e94700 -1 osd.2 1138 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) 2021-02-10T04:52:15.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:52:14 smithi164 conmon[33677]: debug 2021-02-10T04:52:15.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:52:14 smithi164 conmon[33677]: 2021-02-10T04:52:14.793+0000 7fbec19a1700 -1 osd.4 1138 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:52:15.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:52:14 smithi164 conmon[33677]: 2021-02-10T04:52:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:14 smithi164 conmon[29288]: cluster 2021-02-10T04:52:13.804124+0000 2021-02-10T04:52:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:14 smithi164 conmon[29288]: osd.4 (osd.4) 2617 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:14 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:52:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:14 smithi164 conmon[29288]: :52:13.804133+0000 osd.4 (osd.4) 2618 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:14 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:52:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:14 smithi164 conmon[29288]: -02-10T04:52:14.069629+0000 2021-02-10T04:52:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:14 smithi164 conmon[29288]: osd.2 (osd.2) 1973 : cluster [WRN] 2021-02-10T04:52:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:14 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:14 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:52:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:14 smithi164 conmon[29288]: 52:14.069634+0000 osd.2 (osd.2) 1974 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:15.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:52:15 smithi085 conmon[49106]: debug 2021-02-10T04:52:15.061+0000 7f77a3e94700 -1 osd.2 1138 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) 2021-02-10T04:52:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:14 smithi085 conmon[29319]: cluster 2021-02-10T04:52:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:14 smithi085 conmon[29319]: 2021-02-10T04:52:13 2021-02-10T04:52:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:14 smithi085 conmon[29319]: .804124+0000 osd.4 (osd.4) 2617 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:14 smithi085 conmon[29319]: cluster 2021-02-10T04:52:13.804133 2021-02-10T04:52:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:14 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2618 : cluster [WRN] 2021-02-10T04:52:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:14 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:14 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:52:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:14 smithi085 conmon[29319]: 04:52:14.069629+0000 osd.2 (osd 2021-02-10T04:52:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:14 smithi085 conmon[29319]: . 2021-02-10T04:52:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:14 smithi085 conmon[29319]: 2) 1973 : cluster 2021-02-10T04:52:15.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:14 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:15.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:14 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:52:15.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:14 smithi085 conmon[29319]: 02-10T04:52:14.069634+0000 2021-02-10T04:52:15.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:14 smithi085 conmon[29319]: osd.2 (osd.2) 1974 : cluster 2021-02-10T04:52:15.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:14 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:14 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:52:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:14 smithi085 conmon[35961]: 04:52:13.804124+0000 osd.4 (osd.4) 2617 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:14 smithi085 conmon[35961]: cluster 2021-02-10T04:52:13.804133+0000 osd.4 (osd.4) 2618 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:14 smithi085 conmon[35961]: cluster 2021-02-10T04:52: 2021-02-10T04:52:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:14 smithi085 conmon[35961]: 14.069629+0000 osd.2 (osd.2) 1973 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:14 smithi085 conmon[35961]: cluster 2021-02-10T04:52:15.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:14 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:52:15.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:14 smithi085 conmon[35961]: T04:52:14.069634 2021-02-10T04:52:15.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:14 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 2021-02-10T04:52:15.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:14 smithi085 conmon[35961]: 1974 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:15.519 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:52:15.522 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:52:15.754 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:52:15 smithi164 conmon[33677]: debug 2021-02-10T04:52:15.750+0000 7fbec19a1700 -1 osd.4 1138 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:52:16.098 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:15 smithi164 conmon[29288]: cluster 2021-02-10T04:52:14.428806+0000 2021-02-10T04:52:16.098 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:15 smithi164 conmon[29288]: mgr.y (mgr.14140) 1239 : cluster [DBG] pgmap v2046: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:52:16.098 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:15 smithi164 conmon[29288]: cluster 2021-02-10T04:52:14.795375+0000 osd.4 (osd.4) 2619 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:16.098 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:15 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:52:16.098 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:15 smithi164 conmon[29288]: -10T04:52:14.795386+0000 osd.4 (osd.4) 2620 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:16.098 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:15 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:52:16.098 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:15 smithi164 conmon[29288]: 02-10T04:52:15.061902+0000 osd.2 ( 2021-02-10T04:52:16.098 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:15 smithi164 conmon[29288]: osd.2) 1975 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:16.099 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:15 smithi164 conmon[29288]: cluster 2021-02-10T04:52:16.099 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:15 smithi164 conmon[29288]: 2021-02-10T04:52: 2021-02-10T04:52:16.099 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:15 smithi164 conmon[29288]: 15.061910+0000 osd.2 (osd.2) 1976 2021-02-10T04:52:16.099 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:15 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:16.361 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset noscrub 2021-02-10T04:52:16.363 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:52:16.392 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[49106]: debug 2021-02-10T04:52:16.064+0000 7f77a3e94700 -1 osd.2 1138 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) 2021-02-10T04:52:16.393 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:15 smithi085 conmon[29319]: cluster 2021-02-10T04:52:14.428806 2021-02-10T04:52:16.393 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:15 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 1239 : cluster [DBG] pgmap v2046: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:52:16.393 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:15 smithi085 conmon[29319]: cluster 2021-02-10T04:52:14.795375+0000 osd.4 (osd.4) 2619 : 2021-02-10T04:52:16.394 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:15 smithi085 conmon[29319]: cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:16.394 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:15 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:52:16.394 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:15 smithi085 conmon[29319]: :52:14.795386+0000 osd.4 (osd.4) 2620 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:16.394 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:15 smithi085 conmon[29319]: 2021-02-10T04:52:16.394 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:15 smithi085 conmon[29319]: cluster 2021-02-10T04:52:15. 2021-02-10T04:52:16.394 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:15 smithi085 conmon[29319]: 061902+0000 osd.2 (osd.2) 1975 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:16.394 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:15 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:52:16.394 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:15 smithi085 conmon[29319]: 04:52:15.061910+0000 osd.2 (osd 2021-02-10T04:52:16.394 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:15 smithi085 conmon[29319]: .2) 1976 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:16.394 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:15 smithi085 conmon[35961]: cluster 2021-02-10T04:52:14. 2021-02-10T04:52:16.394 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:15 smithi085 conmon[35961]: 428806+0000 mgr.y (mgr.14140) 1239 : cluster [DBG] pgmap v2046: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:52:16.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:15 smithi085 conmon[35961]: cluster 2021-02-10T04:52:16.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:15 smithi085 conmon[35961]: 2021-02-10T04:52:14. 2021-02-10T04:52:16.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:15 smithi085 conmon[35961]: 795375+0000 osd.4 ( 2021-02-10T04:52:16.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:15 smithi085 conmon[35961]: osd.4) 2619 : cluster 2021-02-10T04:52:16.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:15 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:16.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:15 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:52:16.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:15 smithi085 conmon[35961]: 10T04:52:14.795386+0000 osd.4 (osd.4) 2021-02-10T04:52:16.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:15 smithi085 conmon[35961]: 2620 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:16.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:15 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:52:16.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:15 smithi085 conmon[35961]: 02-10T04:52:15.061902+0000 2021-02-10T04:52:16.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:15 smithi085 conmon[35961]: osd.2 (osd.2) 1975 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:16.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:15 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:52:16.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:15 smithi085 conmon[35961]: -02-10T04:52:15.061910 2021-02-10T04:52:16.396 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:15 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 1976 : cluster [WRN] 2021-02-10T04:52:16.396 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:15 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:16.622 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:52:16.629 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:52:16.953 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":2046,"stamp":"2021-02-10T04:52:14.428300+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":940031637,"num_objects":743,"num_object_clones":336,"num_object_copies":2229,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":743,"num_whiteouts":12,"num_read":7330,"num_read_kb":4771537,"num_write":4628,"num_write_kb":1265219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1044,"num_bytes_recovered":397940035,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25156,"ondisk_log_size":25156,"up":564,"acting":564,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":564,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2242944,"kb_used_data":1889544,"kb_used_omap":386,"kb_used_meta":351293,"kb_avail":747718272,"statfs":{"total":767960285184,"available":765663510528,"internally_reserved":0,"allocated":1934893056,"data_stored":2514453560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":396036,"internal_metadata":359724284},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,2],"upper_bound":1048576},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002156"},"pg_stats":[{"pgid":"6.29","version":"0'0","reported_seq":"28","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.943988+0000","last_change":"2021-02-10T04:51:36.854728+0000","last_active":"2021-02-10T04:52:02.943988+0000","last_peered":"2021-02-10T04:52:02.943988+0000","last_clean":"2021-02-10T04:52:02.943988+0000","last_became_active":"2021-02-10T04:51:36.853034+0000","last_became_peered":"2021-02-10T04:51:36.853034+0000","last_unstale":"2021-02-10T04:52:02.943988+0000","last_undegraded":"2021-02-10T04:52:02.943988+0000","last_fullsized":"2021-02-10T04:52:02.943988+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:12.944345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:12.944345+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":"28","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.944216+0000","last_change":"2021-02-10T04:51:36.865312+0000","last_active":"2021-02-10T04:52:02.944216+0000","last_peered":"2021-02-10T04:52:02.944216+0000","last_clean":"2021-02-10T04:52:02.944216+0000","last_became_active":"2021-02-10T04:51:36.864926+0000","last_became_peered":"2021-02-10T04:51:36.864926+0000","last_unstale":"2021-02-10T04:52:02.944216+0000","last_undegraded":"2021-02-10T04:52:02.944216+0000","last_fullsized":"2021-02-10T04:52:02.944216+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:18.026478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:18.026478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"131'3","reported_seq":"1321","reported_epoch":"1138","state":"active+clean","last_fresh":"2021-02-10T04:52:11.968387+0000","last_change":"2021-02-10T04:51:01.185887+0000","last_active":"2021-02-10T04:52:11.968387+0000","last_peered":"2021-02-10T04:52:11.968387+0000","last_clean":"2021-02-10T04:52:11.968387+0000","last_became_active":"2021-02-10T04:29:07.145525+0000","last_became_peered":"2021-02-10T04:29:07.145525+0000","last_unstale":"2021-02-10T04:52:11.968387+0000","last_undegraded":"2021-02-10T04:52:11.968387+0000","last_fullsized":"2021-02-10T04:52:11.968387+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'3","last_scrub_stamp":"2021-02-10T04:51:01.185820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.185820+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"1132'66","reported_seq":"1225","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.770788+0000","last_change":"2021-02-10T04:51:55.430490+0000","last_active":"2021-02-10T04:52:10.770788+0000","last_peered":"2021-02-10T04:52:10.770788+0000","last_clean":"2021-02-10T04:52:10.770788+0000","last_became_active":"2021-02-10T04:25:49.002058+0000","last_became_peered":"2021-02-10T04:25:49.002058+0000","last_unstale":"2021-02-10T04:52:10.770788+0000","last_undegraded":"2021-02-10T04:52:10.770788+0000","last_fullsized":"2021-02-10T04:52:10.770788+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1132'66","last_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_deep_scrub":"1132'66","last_deep_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.430399+0000","log_size":66,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":95,"num_read_kb":78,"num_write":55,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"37","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.944591+0000","last_change":"2021-02-10T04:51:48.990145+0000","last_active":"2021-02-10T04:52:02.944591+0000","last_peered":"2021-02-10T04:52:02.944591+0000","last_clean":"2021-02-10T04:52:02.944591+0000","last_became_active":"2021-02-10T04:51:36.875000+0000","last_became_peered":"2021-02-10T04:51:36.875000+0000","last_unstale":"2021-02-10T04:52:02.944591+0000","last_undegraded":"2021-02-10T04:52:02.944591+0000","last_fullsized":"2021-02-10T04:52:02.944591+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:51:48.990062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.990062+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"1107","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.771105+0000","last_change":"2021-02-10T04:50:48.408819+0000","last_active":"2021-02-10T04:52:10.771105+0000","last_peered":"2021-02-10T04:52:10.771105+0000","last_clean":"2021-02-10T04:52:10.771105+0000","last_became_active":"2021-02-10T04:25:53.148090+0000","last_became_peered":"2021-02-10T04:25:53.148090+0000","last_unstale":"2021-02-10T04:52:10.771105+0000","last_undegraded":"2021-02-10T04:52:10.771105+0000","last_fullsized":"2021-02-10T04:52:10.771105+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:48.408748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:50:48.408748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"1003'1689","reported_seq":"4544","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:11.181456+0000","last_change":"2021-02-10T04:51:49.499841+0000","last_active":"2021-02-10T04:52:11.181456+0000","last_peered":"2021-02-10T04:52:11.181456+0000","last_clean":"2021-02-10T04:52:11.181456+0000","last_became_active":"2021-02-10T04:51:49.497472+0000","last_became_peered":"2021-02-10T04:51:49.497472+0000","last_unstale":"2021-02-10T04:52:11.181456+0000","last_undegraded":"2021-02-10T04:52:11.181456+0000","last_fullsized":"2021-02-10T04:52:11.181456+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:46:54.814020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:54.814020+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":126521476,"num_objects":49,"num_object_clones":44,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":773,"num_read_kb":819582,"num_write":561,"num_write_kb":221117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"1049'32","reported_seq":"1203","reported_epoch":"1135","state":"active+clean","last_fresh":"2021-02-10T04:52:00.499756+0000","last_change":"2021-02-10T04:51:51.382109+0000","last_active":"2021-02-10T04:52:00.499756+0000","last_peered":"2021-02-10T04:52:00.499756+0000","last_clean":"2021-02-10T04:52:00.499756+0000","last_became_active":"2021-02-10T04:38:39.243413+0000","last_became_peered":"2021-02-10T04:38:39.243413+0000","last_unstale":"2021-02-10T04:52:00.499756+0000","last_undegraded":"2021-02-10T04:52:00.499756+0000","last_fullsized":"2021-02-10T04:52:00.499756+0000","mapping_epoch":700,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'32","last_scrub_stamp":"2021-02-10T04:51:51.382040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.382040+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":62,"num_read_kb":45,"num_write":33,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":894,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"786","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:11.182086+0000","last_change":"2021-02-10T04:51:49.502917+0000","last_active":"2021-02-10T04:52:11.182086+0000","last_peered":"2021-02-10T04:52:11.182086+0000","last_clean":"2021-02-10T04:52:11.182086+0000","last_became_active":"2021-02-10T04:51:49.502576+0000","last_became_peered":"2021-02-10T04:51:49.502576+0000","last_unstale":"2021-02-10T04:52:11.182086+0000","last_undegraded":"2021-02-10T04:52:11.182086+0000","last_fullsized":"2021-02-10T04:52:11.182086+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:59.824373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:59.824373+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"33","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.942177+0000","last_change":"2021-02-10T04:51:53.038893+0000","last_active":"2021-02-10T04:52:02.942177+0000","last_peered":"2021-02-10T04:52:02.942177+0000","last_clean":"2021-02-10T04:52:02.942177+0000","last_became_active":"2021-02-10T04:51:36.855507+0000","last_became_peered":"2021-02-10T04:51:36.855507+0000","last_unstale":"2021-02-10T04:52:02.942177+0000","last_undegraded":"2021-02-10T04:52:02.942177+0000","last_fullsized":"2021-02-10T04:52:02.942177+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:53.038834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:53.038834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"1148","reported_epoch":"1135","state":"active+clean","last_fresh":"2021-02-10T04:52:00.499816+0000","last_change":"2021-02-10T04:51:36.854385+0000","last_active":"2021-02-10T04:52:00.499816+0000","last_peered":"2021-02-10T04:52:00.499816+0000","last_clean":"2021-02-10T04:52:00.499816+0000","last_became_active":"2021-02-10T04:51:36.853896+0000","last_became_peered":"2021-02-10T04:51:36.853896+0000","last_unstale":"2021-02-10T04:52:00.499816+0000","last_undegraded":"2021-02-10T04:52:00.499816+0000","last_fullsized":"2021-02-10T04:52:00.499816+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:09.922789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:09.922789+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"259'74","reported_seq":"44","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.944174+0000","last_change":"2021-02-10T04:51:37.539961+0000","last_active":"2021-02-10T04:52:02.944174+0000","last_peered":"2021-02-10T04:52:02.944174+0000","last_clean":"2021-02-10T04:52:02.944174+0000","last_became_active":"2021-02-10T04:51:36.859215+0000","last_became_peered":"2021-02-10T04:51:36.859215+0000","last_unstale":"2021-02-10T04:52:02.944174+0000","last_undegraded":"2021-02-10T04:52:02.944174+0000","last_fullsized":"2021-02-10T04:52:02.944174+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'62","last_scrub_stamp":"2021-02-10T04:30:10.905905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:30:10.905905+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1218524,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":28227,"num_write":19,"num_write_kb":7021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1134570,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"1049'61","reported_seq":"1221","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774418+0000","last_change":"2021-02-10T04:50:54.154530+0000","last_active":"2021-02-10T04:52:10.774418+0000","last_peered":"2021-02-10T04:52:10.774418+0000","last_clean":"2021-02-10T04:52:10.774418+0000","last_became_active":"2021-02-10T04:25:49.015913+0000","last_became_peered":"2021-02-10T04:25:49.015913+0000","last_unstale":"2021-02-10T04:52:10.774418+0000","last_undegraded":"2021-02-10T04:52:10.774418+0000","last_fullsized":"2021-02-10T04:52:10.774418+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'61","last_scrub_stamp":"2021-02-10T04:50:54.154436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:50:54.154436+0000","log_size":61,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":104,"num_read_kb":84,"num_write":62,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"1119'4","reported_seq":"45","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:10.440937+0000","last_change":"2021-02-10T04:52:01.121444+0000","last_active":"2021-02-10T04:52:10.440937+0000","last_peered":"2021-02-10T04:52:10.440937+0000","last_clean":"2021-02-10T04:52:10.440937+0000","last_became_active":"2021-02-10T04:51:36.863322+0000","last_became_peered":"2021-02-10T04:51:36.863322+0000","last_unstale":"2021-02-10T04:52:10.440937+0000","last_undegraded":"2021-02-10T04:52:10.440937+0000","last_fullsized":"2021-02-10T04:52:10.440937+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'4","last_scrub_stamp":"2021-02-10T04:52:01.121380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.121380+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"28","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.944081+0000","last_change":"2021-02-10T04:51:36.861919+0000","last_active":"2021-02-10T04:52:02.944081+0000","last_peered":"2021-02-10T04:52:02.944081+0000","last_clean":"2021-02-10T04:52:02.944081+0000","last_became_active":"2021-02-10T04:51:36.860476+0000","last_became_peered":"2021-02-10T04:51:36.860476+0000","last_unstale":"2021-02-10T04:52:02.944081+0000","last_undegraded":"2021-02-10T04:52:02.944081+0000","last_fullsized":"2021-02-10T04:52:02.944081+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:52.421793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:52.421793+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"28","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.944053+0000","last_change":"2021-02-10T04:51:36.852665+0000","last_active":"2021-02-10T04:52:02.944053+0000","last_peered":"2021-02-10T04:52:02.944053+0000","last_clean":"2021-02-10T04:52:02.944053+0000","last_became_active":"2021-02-10T04:51:36.851957+0000","last_became_peered":"2021-02-10T04:51:36.851957+0000","last_unstale":"2021-02-10T04:52:02.944053+0000","last_undegraded":"2021-02-10T04:52:02.944053+0000","last_fullsized":"2021-02-10T04:52:02.944053+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:24.731458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:24.731458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"1003'739","reported_seq":"3800","reported_epoch":"1135","state":"active+clean","last_fresh":"2021-02-10T04:52:00.499213+0000","last_change":"2021-02-10T04:46:47.806184+0000","last_active":"2021-02-10T04:52:00.499213+0000","last_peered":"2021-02-10T04:52:00.499213+0000","last_clean":"2021-02-10T04:52:00.499213+0000","last_became_active":"2021-02-10T04:44:25.545483+0000","last_became_peered":"2021-02-10T04:44:25.545483+0000","last_unstale":"2021-02-10T04:52:00.499213+0000","last_undegraded":"2021-02-10T04:52:00.499213+0000","last_fullsized":"2021-02-10T04:52:00.499213+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:46:47.806105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:47.806105+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25888931,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"1049'61","reported_seq":"1269","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.944578+0000","last_change":"2021-02-10T04:51:37.439479+0000","last_active":"2021-02-10T04:52:02.944578+0000","last_peered":"2021-02-10T04:52:02.944578+0000","last_clean":"2021-02-10T04:52:02.944578+0000","last_became_active":"2021-02-10T04:51:36.849289+0000","last_became_peered":"2021-02-10T04:51:36.849289+0000","last_unstale":"2021-02-10T04:52:02.944578+0000","last_undegraded":"2021-02-10T04:52:02.944578+0000","last_fullsized":"2021-02-10T04:52:02.944578+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'61","last_scrub_stamp":"2021-02-10T04:50:47.662378+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:50:47.662378+0000","log_size":61,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":83,"num_read_kb":65,"num_write":50,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"1027","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.943514+0000","last_change":"2021-02-10T04:50:45.623700+0000","last_active":"2021-02-10T04:52:02.943514+0000","last_peered":"2021-02-10T04:52:02.943514+0000","last_clean":"2021-02-10T04:52:02.943514+0000","last_became_active":"2021-02-10T04:29:07.145616+0000","last_became_peered":"2021-02-10T04:29:07.145616+0000","last_unstale":"2021-02-10T04:52:02.943514+0000","last_undegraded":"2021-02-10T04:52:02.943514+0000","last_fullsized":"2021-02-10T04:52:02.943514+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:45.623640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:45.623640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"346","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:11.181845+0000","last_change":"2021-02-10T04:51:49.629658+0000","last_active":"2021-02-10T04:52:11.181845+0000","last_peered":"2021-02-10T04:52:11.181845+0000","last_clean":"2021-02-10T04:52:11.181845+0000","last_became_active":"2021-02-10T04:51:49.629360+0000","last_became_peered":"2021-02-10T04:51:49.629360+0000","last_unstale":"2021-02-10T04:52:11.181845+0000","last_undegraded":"2021-02-10T04:52:11.181845+0000","last_fullsized":"2021-02-10T04:52:11.181845+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:43.762742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:46:43.762742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"33","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.944472+0000","last_change":"2021-02-10T04:52:02.162917+0000","last_active":"2021-02-10T04:52:02.944472+0000","last_peered":"2021-02-10T04:52:02.944472+0000","last_clean":"2021-02-10T04:52:02.944472+0000","last_became_active":"2021-02-10T04:51:36.855744+0000","last_became_peered":"2021-02-10T04:51:36.855744+0000","last_unstale":"2021-02-10T04:52:02.944472+0000","last_undegraded":"2021-02-10T04:52:02.944472+0000","last_fullsized":"2021-02-10T04:52:02.944472+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:02.162851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:13.727241+0000","last_clean_scrub_stamp":"2021-02-10T04:52:02.162851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"1063'654","reported_seq":"4287","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.944523+0000","last_change":"2021-02-10T04:51:57.121903+0000","last_active":"2021-02-10T04:52:02.944523+0000","last_peered":"2021-02-10T04:52:02.944523+0000","last_clean":"2021-02-10T04:52:02.944523+0000","last_became_active":"2021-02-10T04:51:40.514397+0000","last_became_peered":"2021-02-10T04:51:40.514397+0000","last_unstale":"2021-02-10T04:52:02.944523+0000","last_undegraded":"2021-02-10T04:52:02.944523+0000","last_fullsized":"2021-02-10T04:52:02.944523+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":0,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:51:57.121838+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.121838+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49401261,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":2,"num_read":342,"num_read_kb":364544,"num_write":223,"num_write_kb":67454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":52698047,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"1083","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.770835+0000","last_change":"2021-02-10T04:51:50.433056+0000","last_active":"2021-02-10T04:52:10.770835+0000","last_peered":"2021-02-10T04:52:10.770835+0000","last_clean":"2021-02-10T04:52:10.770835+0000","last_became_active":"2021-02-10T04:51:36.843165+0000","last_became_peered":"2021-02-10T04:51:36.843165+0000","last_unstale":"2021-02-10T04:52:10.770835+0000","last_undegraded":"2021-02-10T04:52:10.770835+0000","last_fullsized":"2021-02-10T04:52:10.770835+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.432990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.432990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"1099","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774207+0000","last_change":"2021-02-10T04:51:36.851443+0000","last_active":"2021-02-10T04:52:10.774207+0000","last_peered":"2021-02-10T04:52:10.774207+0000","last_clean":"2021-02-10T04:52:10.774207+0000","last_became_active":"2021-02-10T04:51:36.851075+0000","last_became_peered":"2021-02-10T04:51:36.851075+0000","last_unstale":"2021-02-10T04:52:10.774207+0000","last_undegraded":"2021-02-10T04:52:10.774207+0000","last_fullsized":"2021-02-10T04:52:10.774207+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:53.528227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:46:53.528227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"1049'33","reported_seq":"1175","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.943615+0000","last_change":"2021-02-10T04:51:06.846957+0000","last_active":"2021-02-10T04:52:02.943615+0000","last_peered":"2021-02-10T04:52:02.943615+0000","last_clean":"2021-02-10T04:52:02.943615+0000","last_became_active":"2021-02-10T04:50:59.584833+0000","last_became_peered":"2021-02-10T04:50:59.584833+0000","last_unstale":"2021-02-10T04:52:02.943615+0000","last_undegraded":"2021-02-10T04:52:02.943615+0000","last_fullsized":"2021-02-10T04:52:02.943615+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'33","last_scrub_stamp":"2021-02-10T04:51:06.846850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:06.846850+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":43,"num_write":33,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"776","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:11.181316+0000","last_change":"2021-02-10T04:51:52.350764+0000","last_active":"2021-02-10T04:52:11.181316+0000","last_peered":"2021-02-10T04:52:11.181316+0000","last_clean":"2021-02-10T04:52:11.181316+0000","last_became_active":"2021-02-10T04:51:49.567784+0000","last_became_peered":"2021-02-10T04:51:49.567784+0000","last_unstale":"2021-02-10T04:52:11.181316+0000","last_undegraded":"2021-02-10T04:52:11.181316+0000","last_fullsized":"2021-02-10T04:52:11.181316+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:52.350706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.350706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"1119'4","reported_seq":"40","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:07.999080+0000","last_change":"2021-02-10T04:51:36.907926+0000","last_active":"2021-02-10T04:52:07.999080+0000","last_peered":"2021-02-10T04:52:07.999080+0000","last_clean":"2021-02-10T04:52:07.999080+0000","last_became_active":"2021-02-10T04:51:36.853647+0000","last_became_peered":"2021-02-10T04:51:36.853647+0000","last_unstale":"2021-02-10T04:52:07.999080+0000","last_undegraded":"2021-02-10T04:52:07.999080+0000","last_fullsized":"2021-02-10T04:52:07.999080+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"639'3","last_scrub_stamp":"2021-02-10T04:47:01.756908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:47:01.756908+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"1118","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.943995+0000","last_change":"2021-02-10T04:51:36.849935+0000","last_active":"2021-02-10T04:52:02.943995+0000","last_peered":"2021-02-10T04:52:02.943995+0000","last_clean":"2021-02-10T04:52:02.943995+0000","last_became_active":"2021-02-10T04:51:36.849503+0000","last_became_peered":"2021-02-10T04:51:36.849503+0000","last_unstale":"2021-02-10T04:52:02.943995+0000","last_undegraded":"2021-02-10T04:52:02.943995+0000","last_fullsized":"2021-02-10T04:52:02.943995+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:48.602819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:48.602819+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"1004'1093","reported_seq":"6378","reported_epoch":"1135","state":"active+clean","last_fresh":"2021-02-10T04:52:00.500140+0000","last_change":"2021-02-10T04:51:49.333087+0000","last_active":"2021-02-10T04:52:00.500140+0000","last_peered":"2021-02-10T04:52:00.500140+0000","last_clean":"2021-02-10T04:52:00.500140+0000","last_became_active":"2021-02-10T04:50:59.596040+0000","last_became_peered":"2021-02-10T04:50:59.596040+0000","last_unstale":"2021-02-10T04:52:00.500140+0000","last_undegraded":"2021-02-10T04:52:00.500140+0000","last_fullsized":"2021-02-10T04:52:00.500140+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":78869393,"num_objects":34,"num_object_clones":30,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":1,"num_read":558,"num_read_kb":622116,"num_write":371,"num_write_kb":133580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"1068","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.770923+0000","last_change":"2021-02-10T04:51:36.841978+0000","last_active":"2021-02-10T04:52:10.770923+0000","last_peered":"2021-02-10T04:52:10.770923+0000","last_clean":"2021-02-10T04:52:10.770923+0000","last_became_active":"2021-02-10T04:51:36.841582+0000","last_became_peered":"2021-02-10T04:51:36.841582+0000","last_unstale":"2021-02-10T04:52:10.770923+0000","last_undegraded":"2021-02-10T04:52:10.770923+0000","last_fullsized":"2021-02-10T04:52:10.770923+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:12.389272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:12.389272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"1049'41","reported_seq":"794","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:11.181960+0000","last_change":"2021-02-10T04:51:58.353746+0000","last_active":"2021-02-10T04:52:11.181960+0000","last_peered":"2021-02-10T04:52:11.181960+0000","last_clean":"2021-02-10T04:52:11.181960+0000","last_became_active":"2021-02-10T04:51:49.502935+0000","last_became_peered":"2021-02-10T04:51:49.502935+0000","last_unstale":"2021-02-10T04:52:11.181960+0000","last_undegraded":"2021-02-10T04:52:11.181960+0000","last_fullsized":"2021-02-10T04:52:11.181960+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'41","last_scrub_stamp":"2021-02-10T04:51:58.353660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:58.353660+0000","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":66,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"794","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:11.181083+0000","last_change":"2021-02-10T04:51:49.568270+0000","last_active":"2021-02-10T04:52:11.181083+0000","last_peered":"2021-02-10T04:52:11.181083+0000","last_clean":"2021-02-10T04:52:11.181083+0000","last_became_active":"2021-02-10T04:51:49.567883+0000","last_became_peered":"2021-02-10T04:51:49.567883+0000","last_unstale":"2021-02-10T04:52:11.181083+0000","last_undegraded":"2021-02-10T04:52:11.181083+0000","last_fullsized":"2021-02-10T04:52:11.181083+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:47.605360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_clean_scrub_stamp":"2021-02-10T04:46:47.605360+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"78'3","reported_seq":"1426","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:07.999125+0000","last_change":"2021-02-10T04:51:47.855391+0000","last_active":"2021-02-10T04:52:07.999125+0000","last_peered":"2021-02-10T04:52:07.999125+0000","last_clean":"2021-02-10T04:52:07.999125+0000","last_became_active":"2021-02-10T04:25:51.025218+0000","last_became_peered":"2021-02-10T04:25:51.025218+0000","last_unstale":"2021-02-10T04:52:07.999125+0000","last_undegraded":"2021-02-10T04:52:07.999125+0000","last_fullsized":"2021-02-10T04:52:07.999125+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:51:47.855283+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.855283+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"1107","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.772115+0000","last_change":"2021-02-10T04:50:10.176968+0000","last_active":"2021-02-10T04:52:10.772115+0000","last_peered":"2021-02-10T04:52:10.772115+0000","last_clean":"2021-02-10T04:52:10.772115+0000","last_became_active":"2021-02-10T04:25:53.042697+0000","last_became_peered":"2021-02-10T04:25:53.042697+0000","last_unstale":"2021-02-10T04:52:10.772115+0000","last_undegraded":"2021-02-10T04:52:10.772115+0000","last_fullsized":"2021-02-10T04:52:10.772115+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:10.176878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:50:10.176878+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"1002'1090","reported_seq":"5321","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774889+0000","last_change":"2021-02-10T04:51:08.220559+0000","last_active":"2021-02-10T04:52:10.774889+0000","last_peered":"2021-02-10T04:52:10.774889+0000","last_clean":"2021-02-10T04:52:10.774889+0000","last_became_active":"2021-02-10T04:50:59.585773+0000","last_became_peered":"2021-02-10T04:50:59.585773+0000","last_unstale":"2021-02-10T04:52:10.774889+0000","last_undegraded":"2021-02-10T04:52:10.774889+0000","last_fullsized":"2021-02-10T04:52:10.774889+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:51:08.220472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.220472+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":48821292,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":491,"num_read_kb":552336,"num_write":351,"num_write_kb":129921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"1049'47","reported_seq":"1185","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.943670+0000","last_change":"2021-02-10T04:50:04.767513+0000","last_active":"2021-02-10T04:52:02.943670+0000","last_peered":"2021-02-10T04:52:02.943670+0000","last_clean":"2021-02-10T04:52:02.943670+0000","last_became_active":"2021-02-10T04:25:49.011093+0000","last_became_peered":"2021-02-10T04:25:49.011093+0000","last_unstale":"2021-02-10T04:52:02.943670+0000","last_undegraded":"2021-02-10T04:52:02.943670+0000","last_fullsized":"2021-02-10T04:52:02.943670+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'47","last_scrub_stamp":"2021-02-10T04:50:04.767446+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:50:04.767446+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":68,"num_read_kb":52,"num_write":38,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"1089","reported_epoch":"1135","state":"active+clean","last_fresh":"2021-02-10T04:52:00.499071+0000","last_change":"2021-02-10T04:50:08.358363+0000","last_active":"2021-02-10T04:52:00.499071+0000","last_peered":"2021-02-10T04:52:00.499071+0000","last_clean":"2021-02-10T04:52:00.499071+0000","last_became_active":"2021-02-10T04:29:07.160593+0000","last_became_peered":"2021-02-10T04:29:07.160593+0000","last_unstale":"2021-02-10T04:52:00.499071+0000","last_undegraded":"2021-02-10T04:52:00.499071+0000","last_fullsized":"2021-02-10T04:52:00.499071+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:08.358300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:50:08.358300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"1114","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.772435+0000","last_change":"2021-02-10T04:51:57.436961+0000","last_active":"2021-02-10T04:52:10.772435+0000","last_peered":"2021-02-10T04:52:10.772435+0000","last_clean":"2021-02-10T04:52:10.772435+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:52:10.772435+0000","last_undegraded":"2021-02-10T04:52:10.772435+0000","last_fullsized":"2021-02-10T04:52:10.772435+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.436862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.436862+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"1065","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.773808+0000","last_change":"2021-02-10T04:51:08.003846+0000","last_active":"2021-02-10T04:52:10.773808+0000","last_peered":"2021-02-10T04:52:10.773808+0000","last_clean":"2021-02-10T04:52:10.773808+0000","last_became_active":"2021-02-10T04:50:59.545501+0000","last_became_peered":"2021-02-10T04:50:59.545501+0000","last_unstale":"2021-02-10T04:52:10.773808+0000","last_undegraded":"2021-02-10T04:52:10.773808+0000","last_fullsized":"2021-02-10T04:52:10.773808+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:01.565998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:01.565998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"1001'1165","reported_seq":"4285","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774796+0000","last_change":"2021-02-10T04:51:03.221365+0000","last_active":"2021-02-10T04:52:10.774796+0000","last_peered":"2021-02-10T04:52:10.774796+0000","last_clean":"2021-02-10T04:52:10.774796+0000","last_became_active":"2021-02-10T04:28:25.152030+0000","last_became_peered":"2021-02-10T04:28:25.152030+0000","last_unstale":"2021-02-10T04:52:10.774796+0000","last_undegraded":"2021-02-10T04:52:10.774796+0000","last_fullsized":"2021-02-10T04:52:10.774796+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":59203690,"num_objects":25,"num_object_clones":21,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":573,"num_read_kb":599059,"num_write":400,"num_write_kb":178261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"1113'5","reported_seq":"1324","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774801+0000","last_change":"2021-02-10T04:51:07.979631+0000","last_active":"2021-02-10T04:52:10.774801+0000","last_peered":"2021-02-10T04:52:10.774801+0000","last_clean":"2021-02-10T04:52:10.774801+0000","last_became_active":"2021-02-10T04:50:59.569578+0000","last_became_peered":"2021-02-10T04:50:59.569578+0000","last_unstale":"2021-02-10T04:52:10.774801+0000","last_undegraded":"2021-02-10T04:52:10.774801+0000","last_fullsized":"2021-02-10T04:52:10.774801+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'3","last_scrub_stamp":"2021-02-10T04:46:36.557037+0000","last_deep_scrub":"158'3","last_deep_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_clean_scrub_stamp":"2021-02-10T04:46:36.557037+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"1049'54","reported_seq":"1199","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.943321+0000","last_change":"2021-02-10T04:51:48.872937+0000","last_active":"2021-02-10T04:52:02.943321+0000","last_peered":"2021-02-10T04:52:02.943321+0000","last_clean":"2021-02-10T04:52:02.943321+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:52:02.943321+0000","last_undegraded":"2021-02-10T04:52:02.943321+0000","last_fullsized":"2021-02-10T04:52:02.943321+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:51:48.872839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.872839+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":93,"num_read_kb":73,"num_write":54,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"1183","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774866+0000","last_change":"2021-02-10T04:51:36.871181+0000","last_active":"2021-02-10T04:52:10.774866+0000","last_peered":"2021-02-10T04:52:10.774866+0000","last_clean":"2021-02-10T04:52:10.774866+0000","last_became_active":"2021-02-10T04:51:36.869931+0000","last_became_peered":"2021-02-10T04:51:36.869931+0000","last_unstale":"2021-02-10T04:52:10.774866+0000","last_undegraded":"2021-02-10T04:52:10.774866+0000","last_fullsized":"2021-02-10T04:52:10.774866+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:04.957072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:04.957072+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"1061","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774817+0000","last_change":"2021-02-10T04:51:36.870536+0000","last_active":"2021-02-10T04:52:10.774817+0000","last_peered":"2021-02-10T04:52:10.774817+0000","last_clean":"2021-02-10T04:52:10.774817+0000","last_became_active":"2021-02-10T04:51:36.869649+0000","last_became_peered":"2021-02-10T04:51:36.869649+0000","last_unstale":"2021-02-10T04:52:10.774817+0000","last_undegraded":"2021-02-10T04:52:10.774817+0000","last_fullsized":"2021-02-10T04:52:10.774817+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:16.010876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:16.010876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"998'1009","reported_seq":"4620","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774622+0000","last_change":"2021-02-10T04:51:39.509051+0000","last_active":"2021-02-10T04:52:10.774622+0000","last_peered":"2021-02-10T04:52:10.774622+0000","last_clean":"2021-02-10T04:52:10.774622+0000","last_became_active":"2021-02-10T04:51:39.508602+0000","last_became_peered":"2021-02-10T04:51:39.508602+0000","last_unstale":"2021-02-10T04:52:10.774622+0000","last_undegraded":"2021-02-10T04:52:10.774622+0000","last_fullsized":"2021-02-10T04:52:10.774622+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":0,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24710413,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":239,"num_read_kb":249403,"num_write":154,"num_write_kb":50104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":8290058,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"1004","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774385+0000","last_change":"2021-02-10T04:51:07.976066+0000","last_active":"2021-02-10T04:52:10.774385+0000","last_peered":"2021-02-10T04:52:10.774385+0000","last_clean":"2021-02-10T04:52:10.774385+0000","last_became_active":"2021-02-10T04:50:59.570854+0000","last_became_peered":"2021-02-10T04:50:59.570854+0000","last_unstale":"2021-02-10T04:52:10.774385+0000","last_undegraded":"2021-02-10T04:52:10.774385+0000","last_fullsized":"2021-02-10T04:52:10.774385+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:46.592377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:46:46.592377+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"1122","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.943550+0000","last_change":"2021-02-10T04:51:50.895471+0000","last_active":"2021-02-10T04:52:02.943550+0000","last_peered":"2021-02-10T04:52:02.943550+0000","last_clean":"2021-02-10T04:52:02.943550+0000","last_became_active":"2021-02-10T04:50:59.527553+0000","last_became_peered":"2021-02-10T04:50:59.527553+0000","last_unstale":"2021-02-10T04:52:02.943550+0000","last_undegraded":"2021-02-10T04:52:02.943550+0000","last_fullsized":"2021-02-10T04:52:02.943550+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.895370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.895370+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"1049'72","reported_seq":"1221","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.775289+0000","last_change":"2021-02-10T04:51:07.988809+0000","last_active":"2021-02-10T04:52:10.775289+0000","last_peered":"2021-02-10T04:52:10.775289+0000","last_clean":"2021-02-10T04:52:10.775289+0000","last_became_active":"2021-02-10T04:50:59.561344+0000","last_became_peered":"2021-02-10T04:50:59.561344+0000","last_unstale":"2021-02-10T04:52:10.775289+0000","last_undegraded":"2021-02-10T04:52:10.775289+0000","last_fullsized":"2021-02-10T04:52:10.775289+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'72","last_scrub_stamp":"2021-02-10T04:46:30.515026+0000","last_deep_scrub":"152'48","last_deep_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_clean_scrub_stamp":"2021-02-10T04:46:30.515026+0000","log_size":72,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":105,"num_read_kb":83,"num_write":60,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"369","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:11.181809+0000","last_change":"2021-02-10T04:51:49.504222+0000","last_active":"2021-02-10T04:52:11.181809+0000","last_peered":"2021-02-10T04:52:11.181809+0000","last_clean":"2021-02-10T04:52:11.181809+0000","last_became_active":"2021-02-10T04:51:49.503801+0000","last_became_peered":"2021-02-10T04:51:49.503801+0000","last_unstale":"2021-02-10T04:52:11.181809+0000","last_undegraded":"2021-02-10T04:52:11.181809+0000","last_fullsized":"2021-02-10T04:52:11.181809+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:51:48.274960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.274960+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"1142","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.944164+0000","last_change":"2021-02-10T04:51:36.849004+0000","last_active":"2021-02-10T04:52:02.944164+0000","last_peered":"2021-02-10T04:52:02.944164+0000","last_clean":"2021-02-10T04:52:02.944164+0000","last_became_active":"2021-02-10T04:51:36.848614+0000","last_became_peered":"2021-02-10T04:51:36.848614+0000","last_unstale":"2021-02-10T04:52:02.944164+0000","last_undegraded":"2021-02-10T04:52:02.944164+0000","last_fullsized":"2021-02-10T04:52:02.944164+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:27.926662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:44:27.926662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"1002'1657","reported_seq":"5189","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.773386+0000","last_change":"2021-02-10T04:51:41.513951+0000","last_active":"2021-02-10T04:52:10.773386+0000","last_peered":"2021-02-10T04:52:10.773386+0000","last_clean":"2021-02-10T04:52:10.773386+0000","last_became_active":"2021-02-10T04:51:41.512410+0000","last_became_peered":"2021-02-10T04:51:41.512410+0000","last_unstale":"2021-02-10T04:52:10.773386+0000","last_undegraded":"2021-02-10T04:52:10.773386+0000","last_fullsized":"2021-02-10T04:52:10.773386+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1124,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":42793066,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":424,"num_read_kb":437673,"num_write":318,"num_write_kb":118797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":44652029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"1066","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.943350+0000","last_change":"2021-02-10T04:51:51.911945+0000","last_active":"2021-02-10T04:52:02.943350+0000","last_peered":"2021-02-10T04:52:02.943350+0000","last_clean":"2021-02-10T04:52:02.943350+0000","last_became_active":"2021-02-10T04:51:36.850015+0000","last_became_peered":"2021-02-10T04:51:36.850015+0000","last_unstale":"2021-02-10T04:52:02.943350+0000","last_undegraded":"2021-02-10T04:52:02.943350+0000","last_fullsized":"2021-02-10T04:52:02.943350+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.911851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.911851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"1180","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774743+0000","last_change":"2021-02-10T04:50:58.142613+0000","last_active":"2021-02-10T04:52:10.774743+0000","last_peered":"2021-02-10T04:52:10.774743+0000","last_clean":"2021-02-10T04:52:10.774743+0000","last_became_active":"2021-02-10T04:25:51.025636+0000","last_became_peered":"2021-02-10T04:25:51.025636+0000","last_unstale":"2021-02-10T04:52:10.774743+0000","last_undegraded":"2021-02-10T04:52:10.774743+0000","last_fullsized":"2021-02-10T04:52:10.774743+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:58.142526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:58.142526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"1135'58","reported_seq":"1209","reported_epoch":"1135","state":"active+clean","last_fresh":"2021-02-10T04:52:01.483215+0000","last_change":"2021-02-10T04:52:01.483143+0000","last_active":"2021-02-10T04:52:01.483215+0000","last_peered":"2021-02-10T04:52:01.483215+0000","last_clean":"2021-02-10T04:52:01.483215+0000","last_became_active":"2021-02-10T04:25:49.002937+0000","last_became_peered":"2021-02-10T04:25:49.002937+0000","last_unstale":"2021-02-10T04:52:01.483215+0000","last_undegraded":"2021-02-10T04:52:01.483215+0000","last_fullsized":"2021-02-10T04:52:01.483215+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1135'58","last_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_deep_scrub":"1135'58","last_deep_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.483067+0000","log_size":58,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":87,"num_read_kb":68,"num_write":50,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"1042","reported_epoch":"1135","state":"active+clean","last_fresh":"2021-02-10T04:52:00.499590+0000","last_change":"2021-02-10T04:50:46.434726+0000","last_active":"2021-02-10T04:52:00.499590+0000","last_peered":"2021-02-10T04:52:00.499590+0000","last_clean":"2021-02-10T04:52:00.499590+0000","last_became_active":"2021-02-10T04:29:07.161341+0000","last_became_peered":"2021-02-10T04:29:07.161341+0000","last_unstale":"2021-02-10T04:52:00.499590+0000","last_undegraded":"2021-02-10T04:52:00.499590+0000","last_fullsized":"2021-02-10T04:52:00.499590+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:50:46.434629+0000","last_deep_scrub":"53'2","last_deep_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_clean_scrub_stamp":"2021-02-10T04:50:46.434629+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"811","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:11.181190+0000","last_change":"2021-02-10T04:51:49.497296+0000","last_active":"2021-02-10T04:52:11.181190+0000","last_peered":"2021-02-10T04:52:11.181190+0000","last_clean":"2021-02-10T04:52:11.181190+0000","last_became_active":"2021-02-10T04:51:49.496237+0000","last_became_peered":"2021-02-10T04:51:49.496237+0000","last_unstale":"2021-02-10T04:52:11.181190+0000","last_undegraded":"2021-02-10T04:52:11.181190+0000","last_fullsized":"2021-02-10T04:52:11.181190+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:15.037663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:15.037663+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"1000'1071","reported_seq":"4673","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.771098+0000","last_change":"2021-02-10T04:50:59.597606+0000","last_active":"2021-02-10T04:52:10.771098+0000","last_peered":"2021-02-10T04:52:10.771098+0000","last_clean":"2021-02-10T04:52:10.771098+0000","last_became_active":"2021-02-10T04:50:59.596326+0000","last_became_peered":"2021-02-10T04:50:59.596326+0000","last_unstale":"2021-02-10T04:52:10.771098+0000","last_undegraded":"2021-02-10T04:52:10.771098+0000","last_fullsized":"2021-02-10T04:52:10.771098+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:47:14.611347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:14.611347+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46572736,"num_objects":18,"num_object_clones":15,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"1146","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.775117+0000","last_change":"2021-02-10T04:51:00.195591+0000","last_active":"2021-02-10T04:52:10.775117+0000","last_peered":"2021-02-10T04:52:10.775117+0000","last_clean":"2021-02-10T04:52:10.775117+0000","last_became_active":"2021-02-10T04:50:59.543222+0000","last_became_peered":"2021-02-10T04:50:59.543222+0000","last_unstale":"2021-02-10T04:52:10.775117+0000","last_undegraded":"2021-02-10T04:52:10.775117+0000","last_fullsized":"2021-02-10T04:52:10.775117+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_clean_scrub_stamp":"2021-02-10T04:51:00.195518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"1109","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.773058+0000","last_change":"2021-02-10T04:50:55.473455+0000","last_active":"2021-02-10T04:52:10.773058+0000","last_peered":"2021-02-10T04:52:10.773058+0000","last_clean":"2021-02-10T04:52:10.773058+0000","last_became_active":"2021-02-10T04:25:51.032201+0000","last_became_peered":"2021-02-10T04:25:51.032201+0000","last_unstale":"2021-02-10T04:52:10.773058+0000","last_undegraded":"2021-02-10T04:52:10.773058+0000","last_fullsized":"2021-02-10T04:52:10.773058+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:55.473347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:50:55.473347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"1049'40","reported_seq":"1237","reported_epoch":"1135","state":"active+clean","last_fresh":"2021-02-10T04:52:00.499796+0000","last_change":"2021-02-10T04:51:47.282005+0000","last_active":"2021-02-10T04:52:00.499796+0000","last_peered":"2021-02-10T04:52:00.499796+0000","last_clean":"2021-02-10T04:52:00.499796+0000","last_became_active":"2021-02-10T04:51:36.853217+0000","last_became_peered":"2021-02-10T04:51:36.853217+0000","last_unstale":"2021-02-10T04:52:00.499796+0000","last_undegraded":"2021-02-10T04:52:00.499796+0000","last_fullsized":"2021-02-10T04:52:00.499796+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'40","last_scrub_stamp":"2021-02-10T04:51:47.281929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.281929+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":52,"num_write":40,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1260,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"324","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:11.181187+0000","last_change":"2021-02-10T04:51:49.629022+0000","last_active":"2021-02-10T04:52:11.181187+0000","last_peered":"2021-02-10T04:52:11.181187+0000","last_clean":"2021-02-10T04:52:11.181187+0000","last_became_active":"2021-02-10T04:51:49.628679+0000","last_became_peered":"2021-02-10T04:51:49.628679+0000","last_unstale":"2021-02-10T04:52:11.181187+0000","last_undegraded":"2021-02-10T04:52:11.181187+0000","last_fullsized":"2021-02-10T04:52:11.181187+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:50.399350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:50.399350+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"1123","reported_epoch":"1135","state":"active+clean","last_fresh":"2021-02-10T04:52:00.499426+0000","last_change":"2021-02-10T04:50:59.577926+0000","last_active":"2021-02-10T04:52:00.499426+0000","last_peered":"2021-02-10T04:52:00.499426+0000","last_clean":"2021-02-10T04:52:00.499426+0000","last_became_active":"2021-02-10T04:50:59.576153+0000","last_became_peered":"2021-02-10T04:50:59.576153+0000","last_unstale":"2021-02-10T04:52:00.499426+0000","last_undegraded":"2021-02-10T04:52:00.499426+0000","last_fullsized":"2021-02-10T04:52:00.499426+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"796","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:11.181858+0000","last_change":"2021-02-10T04:51:49.501742+0000","last_active":"2021-02-10T04:52:11.181858+0000","last_peered":"2021-02-10T04:52:11.181858+0000","last_clean":"2021-02-10T04:52:11.181858+0000","last_became_active":"2021-02-10T04:51:49.501405+0000","last_became_peered":"2021-02-10T04:51:49.501405+0000","last_unstale":"2021-02-10T04:52:11.181858+0000","last_undegraded":"2021-02-10T04:52:11.181858+0000","last_fullsized":"2021-02-10T04:52:11.181858+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:17.598113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:17.598113+0000","last_clean_scrub_stamp":"2021-02-10T04:47:17.598113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"1000'446","reported_seq":"2072","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.944147+0000","last_change":"2021-02-10T04:51:45.964115+0000","last_active":"2021-02-10T04:52:02.944147+0000","last_peered":"2021-02-10T04:52:02.944147+0000","last_clean":"2021-02-10T04:52:02.944147+0000","last_became_active":"2021-02-10T04:51:39.510577+0000","last_became_peered":"2021-02-10T04:51:39.510577+0000","last_unstale":"2021-02-10T04:52:02.944147+0000","last_undegraded":"2021-02-10T04:52:02.944147+0000","last_fullsized":"2021-02-10T04:52:02.944147+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:51:45.964029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.964029+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19615419,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14193373,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"287","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:11.181529+0000","last_change":"2021-02-10T04:51:49.499632+0000","last_active":"2021-02-10T04:52:11.181529+0000","last_peered":"2021-02-10T04:52:11.181529+0000","last_clean":"2021-02-10T04:52:11.181529+0000","last_became_active":"2021-02-10T04:51:49.499252+0000","last_became_peered":"2021-02-10T04:51:49.499252+0000","last_unstale":"2021-02-10T04:52:11.181529+0000","last_undegraded":"2021-02-10T04:52:11.181529+0000","last_fullsized":"2021-02-10T04:52:11.181529+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:05.242690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:05.242690+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"28","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.944008+0000","last_change":"2021-02-10T04:51:36.859470+0000","last_active":"2021-02-10T04:52:02.944008+0000","last_peered":"2021-02-10T04:52:02.944008+0000","last_clean":"2021-02-10T04:52:02.944008+0000","last_became_active":"2021-02-10T04:51:36.858966+0000","last_became_peered":"2021-02-10T04:51:36.858966+0000","last_unstale":"2021-02-10T04:52:02.944008+0000","last_undegraded":"2021-02-10T04:52:02.944008+0000","last_fullsized":"2021-02-10T04:52:02.944008+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:10.398477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:10.398477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"1049'39","reported_seq":"1176","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774628+0000","last_change":"2021-02-10T04:51:47.322426+0000","last_active":"2021-02-10T04:52:10.774628+0000","last_peered":"2021-02-10T04:52:10.774628+0000","last_clean":"2021-02-10T04:52:10.774628+0000","last_became_active":"2021-02-10T04:25:49.002471+0000","last_became_peered":"2021-02-10T04:25:49.002471+0000","last_unstale":"2021-02-10T04:52:10.774628+0000","last_undegraded":"2021-02-10T04:52:10.774628+0000","last_fullsized":"2021-02-10T04:52:10.774628+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'39","last_scrub_stamp":"2021-02-10T04:51:47.322331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.322331+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":51,"num_write":39,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"1153","reported_epoch":"1135","state":"active+clean","last_fresh":"2021-02-10T04:52:00.499119+0000","last_change":"2021-02-10T04:50:05.424349+0000","last_active":"2021-02-10T04:52:00.499119+0000","last_peered":"2021-02-10T04:52:00.499119+0000","last_clean":"2021-02-10T04:52:00.499119+0000","last_became_active":"2021-02-10T04:29:07.158075+0000","last_became_peered":"2021-02-10T04:29:07.158075+0000","last_unstale":"2021-02-10T04:52:00.499119+0000","last_undegraded":"2021-02-10T04:52:00.499119+0000","last_fullsized":"2021-02-10T04:52:00.499119+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_clean_scrub_stamp":"2021-02-10T04:50:05.424262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"1303","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.771789+0000","last_change":"2021-02-10T04:51:36.843835+0000","last_active":"2021-02-10T04:52:10.771789+0000","last_peered":"2021-02-10T04:52:10.771789+0000","last_clean":"2021-02-10T04:52:10.771789+0000","last_became_active":"2021-02-10T04:51:36.843553+0000","last_became_peered":"2021-02-10T04:51:36.843553+0000","last_unstale":"2021-02-10T04:52:10.771789+0000","last_undegraded":"2021-02-10T04:52:10.771789+0000","last_fullsized":"2021-02-10T04:52:10.771789+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:29.707329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:29.707329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"1106","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774638+0000","last_change":"2021-02-10T04:51:57.769992+0000","last_active":"2021-02-10T04:52:10.774638+0000","last_peered":"2021-02-10T04:52:10.774638+0000","last_clean":"2021-02-10T04:52:10.774638+0000","last_became_active":"2021-02-10T04:50:59.587343+0000","last_became_peered":"2021-02-10T04:50:59.587343+0000","last_unstale":"2021-02-10T04:52:10.774638+0000","last_undegraded":"2021-02-10T04:52:10.774638+0000","last_fullsized":"2021-02-10T04:52:10.774638+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"1131","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.772613+0000","last_change":"2021-02-10T04:50:59.538442+0000","last_active":"2021-02-10T04:52:10.772613+0000","last_peered":"2021-02-10T04:52:10.772613+0000","last_clean":"2021-02-10T04:52:10.772613+0000","last_became_active":"2021-02-10T04:50:59.538009+0000","last_became_peered":"2021-02-10T04:50:59.538009+0000","last_unstale":"2021-02-10T04:52:10.772613+0000","last_undegraded":"2021-02-10T04:52:10.772613+0000","last_fullsized":"2021-02-10T04:52:10.772613+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:19.637875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:19.637875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"1000'490","reported_seq":"3670","reported_epoch":"1135","state":"active+clean","last_fresh":"2021-02-10T04:52:00.499844+0000","last_change":"2021-02-10T04:47:02.876009+0000","last_active":"2021-02-10T04:52:00.499844+0000","last_peered":"2021-02-10T04:52:00.499844+0000","last_clean":"2021-02-10T04:52:00.499844+0000","last_became_active":"2021-02-10T04:45:42.422424+0000","last_became_peered":"2021-02-10T04:45:42.422424+0000","last_unstale":"2021-02-10T04:52:00.499844+0000","last_undegraded":"2021-02-10T04:52:00.499844+0000","last_fullsized":"2021-02-10T04:52:00.499844+0000","mapping_epoch":1039,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1040,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:47:02.875920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:02.875920+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22588464,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"28","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.944668+0000","last_change":"2021-02-10T04:51:36.864828+0000","last_active":"2021-02-10T04:52:02.944668+0000","last_peered":"2021-02-10T04:52:02.944668+0000","last_clean":"2021-02-10T04:52:02.944668+0000","last_became_active":"2021-02-10T04:51:36.863976+0000","last_became_peered":"2021-02-10T04:51:36.863976+0000","last_unstale":"2021-02-10T04:52:02.944668+0000","last_undegraded":"2021-02-10T04:52:02.944668+0000","last_fullsized":"2021-02-10T04:52:02.944668+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:46.380285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:50:46.380285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"1049'37","reported_seq":"810","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:11.181963+0000","last_change":"2021-02-10T04:51:56.431263+0000","last_active":"2021-02-10T04:52:11.181963+0000","last_peered":"2021-02-10T04:52:11.181963+0000","last_clean":"2021-02-10T04:52:11.181963+0000","last_became_active":"2021-02-10T04:51:49.630474+0000","last_became_peered":"2021-02-10T04:51:49.630474+0000","last_unstale":"2021-02-10T04:52:11.181963+0000","last_undegraded":"2021-02-10T04:52:11.181963+0000","last_fullsized":"2021-02-10T04:52:11.181963+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:56.431187+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.431187+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":59,"num_read_kb":43,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"28","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.943678+0000","last_change":"2021-02-10T04:51:36.876034+0000","last_active":"2021-02-10T04:52:02.943678+0000","last_peered":"2021-02-10T04:52:02.943678+0000","last_clean":"2021-02-10T04:52:02.943678+0000","last_became_active":"2021-02-10T04:51:36.875715+0000","last_became_peered":"2021-02-10T04:51:36.875715+0000","last_unstale":"2021-02-10T04:52:02.943678+0000","last_undegraded":"2021-02-10T04:52:02.943678+0000","last_fullsized":"2021-02-10T04:52:02.943678+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:04.591695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:04.591695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"807","reported_epoch":"1138","state":"active+clean","last_fresh":"2021-02-10T04:52:12.270646+0000","last_change":"2021-02-10T04:51:49.497189+0000","last_active":"2021-02-10T04:52:12.270646+0000","last_peered":"2021-02-10T04:52:12.270646+0000","last_clean":"2021-02-10T04:52:12.270646+0000","last_became_active":"2021-02-10T04:51:49.496092+0000","last_became_peered":"2021-02-10T04:51:49.496092+0000","last_unstale":"2021-02-10T04:52:12.270646+0000","last_undegraded":"2021-02-10T04:52:12.270646+0000","last_fullsized":"2021-02-10T04:52:12.270646+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:46:53.011080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:53.011080+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":455,"num_read_kb":450,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"1058","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774740+0000","last_change":"2021-02-10T04:50:59.529934+0000","last_active":"2021-02-10T04:52:10.774740+0000","last_peered":"2021-02-10T04:52:10.774740+0000","last_clean":"2021-02-10T04:52:10.774740+0000","last_became_active":"2021-02-10T04:50:59.529649+0000","last_became_peered":"2021-02-10T04:50:59.529649+0000","last_unstale":"2021-02-10T04:52:10.774740+0000","last_undegraded":"2021-02-10T04:52:10.774740+0000","last_fullsized":"2021-02-10T04:52:10.774740+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:13.991399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:47:13.991399+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.4","version":"1000'620","reported_seq":"4309","reported_epoch":"1135","state":"active+clean","last_fresh":"2021-02-10T04:52:00.499566+0000","last_change":"2021-02-10T04:51:54.444365+0000","last_active":"2021-02-10T04:52:00.499566+0000","last_peered":"2021-02-10T04:52:00.499566+0000","last_clean":"2021-02-10T04:52:00.499566+0000","last_became_active":"2021-02-10T04:44:25.537632+0000","last_became_peered":"2021-02-10T04:44:25.537632+0000","last_unstale":"2021-02-10T04:52:00.499566+0000","last_undegraded":"2021-02-10T04:52:00.499566+0000","last_fullsized":"2021-02-10T04:52:00.499566+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:51:54.444287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:54.444287+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"1121","reported_epoch":"1135","state":"active+clean","last_fresh":"2021-02-10T04:52:00.499310+0000","last_change":"2021-02-10T04:47:14.955640+0000","last_active":"2021-02-10T04:52:00.499310+0000","last_peered":"2021-02-10T04:52:00.499310+0000","last_clean":"2021-02-10T04:52:00.499310+0000","last_became_active":"2021-02-10T04:29:07.159607+0000","last_became_peered":"2021-02-10T04:29:07.159607+0000","last_unstale":"2021-02-10T04:52:00.499310+0000","last_undegraded":"2021-02-10T04:52:00.499310+0000","last_fullsized":"2021-02-10T04:52:00.499310+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:14.955544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:47:14.955544+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"1222","reported_epoch":"1135","state":"active+clean","last_fresh":"2021-02-10T04:52:00.499615+0000","last_change":"2021-02-10T04:47:27.908308+0000","last_active":"2021-02-10T04:52:00.499615+0000","last_peered":"2021-02-10T04:52:00.499615+0000","last_clean":"2021-02-10T04:52:00.499615+0000","last_became_active":"2021-02-10T04:25:51.027624+0000","last_became_peered":"2021-02-10T04:25:51.027624+0000","last_unstale":"2021-02-10T04:52:00.499615+0000","last_undegraded":"2021-02-10T04:52:00.499615+0000","last_fullsized":"2021-02-10T04:52:00.499615+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:27.908221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:47:27.908221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"1049'55","reported_seq":"767","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:11.181691+0000","last_change":"2021-02-10T04:51:49.502196+0000","last_active":"2021-02-10T04:52:11.181691+0000","last_peered":"2021-02-10T04:52:11.181691+0000","last_clean":"2021-02-10T04:52:11.181691+0000","last_became_active":"2021-02-10T04:51:49.501622+0000","last_became_peered":"2021-02-10T04:51:49.501622+0000","last_unstale":"2021-02-10T04:52:11.181691+0000","last_undegraded":"2021-02-10T04:52:11.181691+0000","last_fullsized":"2021-02-10T04:52:11.181691+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'55","last_scrub_stamp":"2021-02-10T04:46:55.516761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:55.516761+0000","log_size":55,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":98,"num_read_kb":74,"num_write":58,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"1132","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.771190+0000","last_change":"2021-02-10T04:51:59.420149+0000","last_active":"2021-02-10T04:52:10.771190+0000","last_peered":"2021-02-10T04:52:10.771190+0000","last_clean":"2021-02-10T04:52:10.771190+0000","last_became_active":"2021-02-10T04:28:21.125541+0000","last_became_peered":"2021-02-10T04:28:21.125541+0000","last_unstale":"2021-02-10T04:52:10.771190+0000","last_undegraded":"2021-02-10T04:52:10.771190+0000","last_fullsized":"2021-02-10T04:52:10.771190+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.420086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.420086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"1094","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774338+0000","last_change":"2021-02-10T04:51:07.984424+0000","last_active":"2021-02-10T04:52:10.774338+0000","last_peered":"2021-02-10T04:52:10.774338+0000","last_clean":"2021-02-10T04:52:10.774338+0000","last_became_active":"2021-02-10T04:50:59.555689+0000","last_became_peered":"2021-02-10T04:50:59.555689+0000","last_unstale":"2021-02-10T04:52:10.774338+0000","last_undegraded":"2021-02-10T04:52:10.774338+0000","last_fullsized":"2021-02-10T04:52:10.774338+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.395943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.395943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"1002'2042","reported_seq":"9353","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774843+0000","last_change":"2021-02-10T04:50:59.593597+0000","last_active":"2021-02-10T04:52:10.774843+0000","last_peered":"2021-02-10T04:52:10.774843+0000","last_clean":"2021-02-10T04:52:10.774843+0000","last_became_active":"2021-02-10T04:50:59.591769+0000","last_became_peered":"2021-02-10T04:50:59.591769+0000","last_unstale":"2021-02-10T04:52:10.774843+0000","last_undegraded":"2021-02-10T04:52:10.774843+0000","last_fullsized":"2021-02-10T04:52:10.774843+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:46:57.054230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:57.054230+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":85678898,"num_objects":35,"num_object_clones":29,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"1068","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.772501+0000","last_change":"2021-02-10T04:51:36.851659+0000","last_active":"2021-02-10T04:52:10.772501+0000","last_peered":"2021-02-10T04:52:10.772501+0000","last_clean":"2021-02-10T04:52:10.772501+0000","last_became_active":"2021-02-10T04:51:36.851393+0000","last_became_peered":"2021-02-10T04:51:36.851393+0000","last_unstale":"2021-02-10T04:52:10.772501+0000","last_undegraded":"2021-02-10T04:52:10.772501+0000","last_fullsized":"2021-02-10T04:52:10.772501+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:52.341527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:50:52.341527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"58'2","reported_seq":"1429","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.998302+0000","last_change":"2021-02-10T04:50:54.466333+0000","last_active":"2021-02-10T04:52:02.998302+0000","last_peered":"2021-02-10T04:52:02.998302+0000","last_clean":"2021-02-10T04:52:02.998302+0000","last_became_active":"2021-02-10T04:25:51.026251+0000","last_became_peered":"2021-02-10T04:25:51.026251+0000","last_unstale":"2021-02-10T04:52:02.998302+0000","last_undegraded":"2021-02-10T04:52:02.998302+0000","last_fullsized":"2021-02-10T04:52:02.998302+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:54.466259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:54.466259+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"1049'54","reported_seq":"816","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:11.181433+0000","last_change":"2021-02-10T04:51:49.498890+0000","last_active":"2021-02-10T04:52:11.181433+0000","last_peered":"2021-02-10T04:52:11.181433+0000","last_clean":"2021-02-10T04:52:11.181433+0000","last_became_active":"2021-02-10T04:51:49.498066+0000","last_became_peered":"2021-02-10T04:51:49.498066+0000","last_unstale":"2021-02-10T04:52:11.181433+0000","last_undegraded":"2021-02-10T04:52:11.181433+0000","last_fullsized":"2021-02-10T04:52:11.181433+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:50:14.158659+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:50:14.158659+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":74,"num_read_kb":56,"num_write":42,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"382","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:11.181742+0000","last_change":"2021-02-10T04:51:49.631085+0000","last_active":"2021-02-10T04:52:11.181742+0000","last_peered":"2021-02-10T04:52:11.181742+0000","last_clean":"2021-02-10T04:52:11.181742+0000","last_became_active":"2021-02-10T04:51:49.630731+0000","last_became_peered":"2021-02-10T04:51:49.630731+0000","last_unstale":"2021-02-10T04:52:11.181742+0000","last_undegraded":"2021-02-10T04:52:11.181742+0000","last_fullsized":"2021-02-10T04:52:11.181742+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:49.447961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:49.447961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"1146","reported_epoch":"1135","state":"active+clean","last_fresh":"2021-02-10T04:52:00.499041+0000","last_change":"2021-02-10T04:51:36.853231+0000","last_active":"2021-02-10T04:52:00.499041+0000","last_peered":"2021-02-10T04:52:00.499041+0000","last_clean":"2021-02-10T04:52:00.499041+0000","last_became_active":"2021-02-10T04:51:36.852821+0000","last_became_peered":"2021-02-10T04:51:36.852821+0000","last_unstale":"2021-02-10T04:52:00.499041+0000","last_undegraded":"2021-02-10T04:52:00.499041+0000","last_fullsized":"2021-02-10T04:52:00.499041+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:07.914061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:07.914061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"293'77","reported_seq":"2907","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.943177+0000","last_change":"2021-02-10T04:50:59.527961+0000","last_active":"2021-02-10T04:52:02.943177+0000","last_peered":"2021-02-10T04:52:02.943177+0000","last_clean":"2021-02-10T04:52:02.943177+0000","last_became_active":"2021-02-10T04:50:59.526860+0000","last_became_peered":"2021-02-10T04:50:59.526860+0000","last_unstale":"2021-02-10T04:52:02.943177+0000","last_undegraded":"2021-02-10T04:52:02.943177+0000","last_fullsized":"2021-02-10T04:52:02.943177+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3773171,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":36,"num_read_kb":37174,"num_write":18,"num_write_kb":6631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"1146","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774786+0000","last_change":"2021-02-10T04:50:59.589648+0000","last_active":"2021-02-10T04:52:10.774786+0000","last_peered":"2021-02-10T04:52:10.774786+0000","last_clean":"2021-02-10T04:52:10.774786+0000","last_became_active":"2021-02-10T04:50:59.589417+0000","last_became_peered":"2021-02-10T04:50:59.589417+0000","last_unstale":"2021-02-10T04:52:10.774786+0000","last_undegraded":"2021-02-10T04:52:10.774786+0000","last_fullsized":"2021-02-10T04:52:10.774786+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:09.869777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:47:09.869777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"28","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.943762+0000","last_change":"2021-02-10T04:51:36.861848+0000","last_active":"2021-02-10T04:52:02.943762+0000","last_peered":"2021-02-10T04:52:02.943762+0000","last_clean":"2021-02-10T04:52:02.943762+0000","last_became_active":"2021-02-10T04:51:36.860737+0000","last_became_peered":"2021-02-10T04:51:36.860737+0000","last_unstale":"2021-02-10T04:52:02.943762+0000","last_undegraded":"2021-02-10T04:52:02.943762+0000","last_fullsized":"2021-02-10T04:52:02.943762+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.609500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.609500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"1048'39","reported_seq":"1227","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.943549+0000","last_change":"2021-02-10T04:50:44.624567+0000","last_active":"2021-02-10T04:52:02.943549+0000","last_peered":"2021-02-10T04:52:02.943549+0000","last_clean":"2021-02-10T04:52:02.943549+0000","last_became_active":"2021-02-10T04:39:02.560739+0000","last_became_peered":"2021-02-10T04:39:02.560739+0000","last_unstale":"2021-02-10T04:52:02.943549+0000","last_undegraded":"2021-02-10T04:52:02.943549+0000","last_fullsized":"2021-02-10T04:52:02.943549+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'39","last_scrub_stamp":"2021-02-10T04:50:44.624501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:50:44.624501+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":76,"num_read_kb":59,"num_write":41,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"1235","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.945076+0000","last_change":"2021-02-10T04:47:08.784658+0000","last_active":"2021-02-10T04:52:02.945076+0000","last_peered":"2021-02-10T04:52:02.945076+0000","last_clean":"2021-02-10T04:52:02.945076+0000","last_became_active":"2021-02-10T04:37:37.238701+0000","last_became_peered":"2021-02-10T04:37:37.238701+0000","last_unstale":"2021-02-10T04:52:02.945076+0000","last_undegraded":"2021-02-10T04:52:02.945076+0000","last_fullsized":"2021-02-10T04:52:02.945076+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:47:08.784593+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:47:08.784593+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"1096","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.773905+0000","last_change":"2021-02-10T04:51:07.995649+0000","last_active":"2021-02-10T04:52:10.773905+0000","last_peered":"2021-02-10T04:52:10.773905+0000","last_clean":"2021-02-10T04:52:10.773905+0000","last_became_active":"2021-02-10T04:50:59.561436+0000","last_became_peered":"2021-02-10T04:50:59.561436+0000","last_unstale":"2021-02-10T04:52:10.773905+0000","last_undegraded":"2021-02-10T04:52:10.773905+0000","last_fullsized":"2021-02-10T04:52:10.773905+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:15.536661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:15.536661+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"1001'1528","reported_seq":"6076","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774552+0000","last_change":"2021-02-10T04:51:41.519986+0000","last_active":"2021-02-10T04:52:10.774552+0000","last_peered":"2021-02-10T04:52:10.774552+0000","last_clean":"2021-02-10T04:52:10.774552+0000","last_became_active":"2021-02-10T04:51:41.517363+0000","last_became_peered":"2021-02-10T04:51:41.517363+0000","last_unstale":"2021-02-10T04:52:10.774552+0000","last_undegraded":"2021-02-10T04:52:10.774552+0000","last_fullsized":"2021-02-10T04:52:10.774552+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1124,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":93888272,"num_objects":33,"num_object_clones":30,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":1,"num_read":716,"num_read_kb":703869,"num_write":572,"num_write_kb":222710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":117937665,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.15","version":"1000'446","reported_seq":"2069","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.944424+0000","last_change":"2021-02-10T04:51:40.511548+0000","last_active":"2021-02-10T04:52:02.944424+0000","last_peered":"2021-02-10T04:52:02.944424+0000","last_clean":"2021-02-10T04:52:02.944424+0000","last_became_active":"2021-02-10T04:51:40.510899+0000","last_became_peered":"2021-02-10T04:51:40.510899+0000","last_unstale":"2021-02-10T04:52:02.944424+0000","last_undegraded":"2021-02-10T04:52:02.944424+0000","last_fullsized":"2021-02-10T04:52:02.944424+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:50:05.815135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:50:05.815135+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24896285,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":18983408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"1048'40","reported_seq":"1201","reported_epoch":"1135","state":"active+clean","last_fresh":"2021-02-10T04:52:00.499943+0000","last_change":"2021-02-10T04:50:59.578612+0000","last_active":"2021-02-10T04:52:00.499943+0000","last_peered":"2021-02-10T04:52:00.499943+0000","last_clean":"2021-02-10T04:52:00.499943+0000","last_became_active":"2021-02-10T04:50:59.578193+0000","last_became_peered":"2021-02-10T04:50:59.578193+0000","last_unstale":"2021-02-10T04:52:00.499943+0000","last_undegraded":"2021-02-10T04:52:00.499943+0000","last_fullsized":"2021-02-10T04:52:00.499943+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'40","last_scrub_stamp":"2021-02-10T04:47:29.965140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:47:29.965140+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":80,"num_read_kb":58,"num_write":41,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"32","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.944120+0000","last_change":"2021-02-10T04:51:37.254347+0000","last_active":"2021-02-10T04:52:02.944120+0000","last_peered":"2021-02-10T04:52:02.944120+0000","last_clean":"2021-02-10T04:52:02.944120+0000","last_became_active":"2021-02-10T04:51:36.868351+0000","last_became_peered":"2021-02-10T04:51:36.868351+0000","last_unstale":"2021-02-10T04:52:02.944120+0000","last_undegraded":"2021-02-10T04:52:02.944120+0000","last_fullsized":"2021-02-10T04:52:02.944120+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:51:08.639784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.639784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"1070","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.775225+0000","last_change":"2021-02-10T04:51:09.475150+0000","last_active":"2021-02-10T04:52:10.775225+0000","last_peered":"2021-02-10T04:52:10.775225+0000","last_clean":"2021-02-10T04:52:10.775225+0000","last_became_active":"2021-02-10T04:50:59.585917+0000","last_became_peered":"2021-02-10T04:50:59.585917+0000","last_unstale":"2021-02-10T04:52:10.775225+0000","last_undegraded":"2021-02-10T04:52:10.775225+0000","last_fullsized":"2021-02-10T04:52:10.775225+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:09.475063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:09.475063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"33","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.944222+0000","last_change":"2021-02-10T04:51:44.970615+0000","last_active":"2021-02-10T04:52:02.944222+0000","last_peered":"2021-02-10T04:52:02.944222+0000","last_clean":"2021-02-10T04:52:02.944222+0000","last_became_active":"2021-02-10T04:51:36.859707+0000","last_became_peered":"2021-02-10T04:51:36.859707+0000","last_unstale":"2021-02-10T04:52:02.944222+0000","last_undegraded":"2021-02-10T04:52:02.944222+0000","last_fullsized":"2021-02-10T04:52:02.944222+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:44.970511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.970511+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"1000'620","reported_seq":"4420","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774056+0000","last_change":"2021-02-10T04:51:40.511426+0000","last_active":"2021-02-10T04:52:10.774056+0000","last_peered":"2021-02-10T04:52:10.774056+0000","last_clean":"2021-02-10T04:52:10.774056+0000","last_became_active":"2021-02-10T04:51:40.510114+0000","last_became_peered":"2021-02-10T04:51:40.510114+0000","last_unstale":"2021-02-10T04:52:10.774056+0000","last_undegraded":"2021-02-10T04:52:10.774056+0000","last_fullsized":"2021-02-10T04:52:10.774056+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:47:24.897680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:24.897680+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14265195,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":144,"num_read_kb":128483,"num_write":129,"num_write_kb":53444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":39998538,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"1063'30","reported_seq":"1165","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.775024+0000","last_change":"2021-02-10T04:51:07.995751+0000","last_active":"2021-02-10T04:52:10.775024+0000","last_peered":"2021-02-10T04:52:10.775024+0000","last_clean":"2021-02-10T04:52:10.775024+0000","last_became_active":"2021-02-10T04:50:59.587330+0000","last_became_peered":"2021-02-10T04:50:59.587330+0000","last_unstale":"2021-02-10T04:52:10.775024+0000","last_undegraded":"2021-02-10T04:52:10.775024+0000","last_fullsized":"2021-02-10T04:52:10.775024+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1063'30","last_scrub_stamp":"2021-02-10T04:47:05.589582+0000","last_deep_scrub":"1063'30","last_deep_scrub_stamp":"2021-02-10T04:47:05.589582+0000","last_clean_scrub_stamp":"2021-02-10T04:47:05.589582+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"1120","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.773692+0000","last_change":"2021-02-10T04:46:55.046228+0000","last_active":"2021-02-10T04:52:10.773692+0000","last_peered":"2021-02-10T04:52:10.773692+0000","last_clean":"2021-02-10T04:52:10.773692+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:52:10.773692+0000","last_undegraded":"2021-02-10T04:52:10.773692+0000","last_fullsized":"2021-02-10T04:52:10.773692+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:46:55.046158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:55.046158+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"1124","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.771424+0000","last_change":"2021-02-10T04:51:08.451362+0000","last_active":"2021-02-10T04:52:10.771424+0000","last_peered":"2021-02-10T04:52:10.771424+0000","last_clean":"2021-02-10T04:52:10.771424+0000","last_became_active":"2021-02-10T04:50:59.540375+0000","last_became_peered":"2021-02-10T04:50:59.540375+0000","last_unstale":"2021-02-10T04:52:10.771424+0000","last_undegraded":"2021-02-10T04:52:10.771424+0000","last_fullsized":"2021-02-10T04:52:10.771424+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:08.451289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.451289+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"1034","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774821+0000","last_change":"2021-02-10T04:50:45.362633+0000","last_active":"2021-02-10T04:52:10.774821+0000","last_peered":"2021-02-10T04:52:10.774821+0000","last_clean":"2021-02-10T04:52:10.774821+0000","last_became_active":"2021-02-10T04:29:07.147249+0000","last_became_peered":"2021-02-10T04:29:07.147249+0000","last_unstale":"2021-02-10T04:52:10.774821+0000","last_undegraded":"2021-02-10T04:52:10.774821+0000","last_fullsized":"2021-02-10T04:52:10.774821+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:45.362537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:45.362537+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.17","version":"1002'2042","reported_seq":"9388","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.771676+0000","last_change":"2021-02-10T04:51:41.510539+0000","last_active":"2021-02-10T04:52:10.771676+0000","last_peered":"2021-02-10T04:52:10.771676+0000","last_clean":"2021-02-10T04:52:10.771676+0000","last_became_active":"2021-02-10T04:51:41.509707+0000","last_became_peered":"2021-02-10T04:51:41.509707+0000","last_unstale":"2021-02-10T04:52:10.771676+0000","last_undegraded":"2021-02-10T04:52:10.771676+0000","last_fullsized":"2021-02-10T04:52:10.771676+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1124,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:47:16.641616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:16.641616+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6921254,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":5403183,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"1049'46","reported_seq":"1289","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774239+0000","last_change":"2021-02-10T04:51:07.988787+0000","last_active":"2021-02-10T04:52:10.774239+0000","last_peered":"2021-02-10T04:52:10.774239+0000","last_clean":"2021-02-10T04:52:10.774239+0000","last_became_active":"2021-02-10T04:50:59.569915+0000","last_became_peered":"2021-02-10T04:50:59.569915+0000","last_unstale":"2021-02-10T04:52:10.774239+0000","last_undegraded":"2021-02-10T04:52:10.774239+0000","last_fullsized":"2021-02-10T04:52:10.774239+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'46","last_scrub_stamp":"2021-02-10T04:46:56.512292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:46:56.512292+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":65,"num_write":46,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"84'7","reported_seq":"1153","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.771163+0000","last_change":"2021-02-10T04:51:52.498251+0000","last_active":"2021-02-10T04:52:10.771163+0000","last_peered":"2021-02-10T04:52:10.771163+0000","last_clean":"2021-02-10T04:52:10.771163+0000","last_became_active":"2021-02-10T04:50:59.575924+0000","last_became_peered":"2021-02-10T04:50:59.575924+0000","last_unstale":"2021-02-10T04:52:10.771163+0000","last_undegraded":"2021-02-10T04:52:10.771163+0000","last_fullsized":"2021-02-10T04:52:10.771163+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'7","last_scrub_stamp":"2021-02-10T04:51:52.498158+0000","last_deep_scrub":"84'7","last_deep_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.498158+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"58'2","reported_seq":"1428","reported_epoch":"1138","state":"active+clean","last_fresh":"2021-02-10T04:52:11.966947+0000","last_change":"2021-02-10T04:50:47.407626+0000","last_active":"2021-02-10T04:52:11.966947+0000","last_peered":"2021-02-10T04:52:11.966947+0000","last_clean":"2021-02-10T04:52:11.966947+0000","last_became_active":"2021-02-10T04:25:51.032328+0000","last_became_peered":"2021-02-10T04:25:51.032328+0000","last_unstale":"2021-02-10T04:52:11.966947+0000","last_undegraded":"2021-02-10T04:52:11.966947+0000","last_fullsized":"2021-02-10T04:52:11.966947+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:47.407557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:47.407557+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"1119","reported_epoch":"1135","state":"active+clean","last_fresh":"2021-02-10T04:52:00.499757+0000","last_change":"2021-02-10T04:50:59.576884+0000","last_active":"2021-02-10T04:52:00.499757+0000","last_peered":"2021-02-10T04:52:00.499757+0000","last_clean":"2021-02-10T04:52:00.499757+0000","last_became_active":"2021-02-10T04:50:59.576536+0000","last_became_peered":"2021-02-10T04:50:59.576536+0000","last_unstale":"2021-02-10T04:52:00.499757+0000","last_undegraded":"2021-02-10T04:52:00.499757+0000","last_fullsized":"2021-02-10T04:52:00.499757+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:33.677438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_clean_scrub_stamp":"2021-02-10T04:46:33.677438+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"1000'490","reported_seq":"3638","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.944075+0000","last_change":"2021-02-10T04:50:03.756349+0000","last_active":"2021-02-10T04:52:02.944075+0000","last_peered":"2021-02-10T04:52:02.944075+0000","last_clean":"2021-02-10T04:52:02.944075+0000","last_became_active":"2021-02-10T04:44:30.702606+0000","last_became_peered":"2021-02-10T04:44:30.702606+0000","last_unstale":"2021-02-10T04:52:02.944075+0000","last_undegraded":"2021-02-10T04:52:02.944075+0000","last_fullsized":"2021-02-10T04:52:02.944075+0000","mapping_epoch":1012,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1013,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:50:03.756248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:50:03.756248+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16154790,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"1112'40","reported_seq":"1192","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.771967+0000","last_change":"2021-02-10T04:51:01.450124+0000","last_active":"2021-02-10T04:52:10.771967+0000","last_peered":"2021-02-10T04:52:10.771967+0000","last_clean":"2021-02-10T04:52:10.771967+0000","last_became_active":"2021-02-10T04:50:59.539529+0000","last_became_peered":"2021-02-10T04:50:59.539529+0000","last_unstale":"2021-02-10T04:52:10.771967+0000","last_undegraded":"2021-02-10T04:52:10.771967+0000","last_fullsized":"2021-02-10T04:52:10.771967+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1112'40","last_scrub_stamp":"2021-02-10T04:51:01.450050+0000","last_deep_scrub":"1112'40","last_deep_scrub_stamp":"2021-02-10T04:51:01.450050+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.450050+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":59,"num_read_kb":44,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"1135","reported_epoch":"1135","state":"active+clean","last_fresh":"2021-02-10T04:52:00.499685+0000","last_change":"2021-02-10T04:50:59.579777+0000","last_active":"2021-02-10T04:52:00.499685+0000","last_peered":"2021-02-10T04:52:00.499685+0000","last_clean":"2021-02-10T04:52:00.499685+0000","last_became_active":"2021-02-10T04:50:59.579485+0000","last_became_peered":"2021-02-10T04:50:59.579485+0000","last_unstale":"2021-02-10T04:52:00.499685+0000","last_undegraded":"2021-02-10T04:52:00.499685+0000","last_fullsized":"2021-02-10T04:52:00.499685+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:23.904210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:47:23.904210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"1119'6","reported_seq":"1353","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774444+0000","last_change":"2021-02-10T04:51:36.890787+0000","last_active":"2021-02-10T04:52:10.774444+0000","last_peered":"2021-02-10T04:52:10.774444+0000","last_clean":"2021-02-10T04:52:10.774444+0000","last_became_active":"2021-02-10T04:51:36.869733+0000","last_became_peered":"2021-02-10T04:51:36.869733+0000","last_unstale":"2021-02-10T04:52:10.774444+0000","last_undegraded":"2021-02-10T04:52:10.774444+0000","last_fullsized":"2021-02-10T04:52:10.774444+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"639'3","last_scrub_stamp":"2021-02-10T04:46:39.502896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:46:39.502896+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"1123","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.945434+0000","last_change":"2021-02-10T04:51:36.849933+0000","last_active":"2021-02-10T04:52:02.945434+0000","last_peered":"2021-02-10T04:52:02.945434+0000","last_clean":"2021-02-10T04:52:02.945434+0000","last_became_active":"2021-02-10T04:51:36.849184+0000","last_became_peered":"2021-02-10T04:51:36.849184+0000","last_unstale":"2021-02-10T04:52:02.945434+0000","last_undegraded":"2021-02-10T04:52:02.945434+0000","last_fullsized":"2021-02-10T04:52:02.945434+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:11.779779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:11.779779+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"1032'89","reported_seq":"903","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.773786+0000","last_change":"2021-02-10T04:50:59.528845+0000","last_active":"2021-02-10T04:52:10.773786+0000","last_peered":"2021-02-10T04:52:10.773786+0000","last_clean":"2021-02-10T04:52:10.773786+0000","last_became_active":"2021-02-10T04:50:59.528302+0000","last_became_peered":"2021-02-10T04:50:59.528302+0000","last_unstale":"2021-02-10T04:52:10.773786+0000","last_undegraded":"2021-02-10T04:52:10.773786+0000","last_fullsized":"2021-02-10T04:52:10.773786+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":5,"last_scrub":"1032'89","last_scrub_stamp":"2021-02-10T04:47:00.950114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:00.950114+0000","log_size":89,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5949711,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"1048'27","reported_seq":"805","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:11.181386+0000","last_change":"2021-02-10T04:51:49.497961+0000","last_active":"2021-02-10T04:52:11.181386+0000","last_peered":"2021-02-10T04:52:11.181386+0000","last_clean":"2021-02-10T04:52:11.181386+0000","last_became_active":"2021-02-10T04:51:49.496294+0000","last_became_peered":"2021-02-10T04:51:49.496294+0000","last_unstale":"2021-02-10T04:52:11.181386+0000","last_undegraded":"2021-02-10T04:52:11.181386+0000","last_fullsized":"2021-02-10T04:52:11.181386+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'27","last_scrub_stamp":"2021-02-10T04:50:55.142479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:50:55.142479+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":33,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"1133","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.773252+0000","last_change":"2021-02-10T04:50:59.543884+0000","last_active":"2021-02-10T04:52:10.773252+0000","last_peered":"2021-02-10T04:52:10.773252+0000","last_clean":"2021-02-10T04:52:10.773252+0000","last_became_active":"2021-02-10T04:50:59.543531+0000","last_became_peered":"2021-02-10T04:50:59.543531+0000","last_unstale":"2021-02-10T04:52:10.773252+0000","last_undegraded":"2021-02-10T04:52:10.773252+0000","last_fullsized":"2021-02-10T04:52:10.773252+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:42.217450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:42.217450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"1074","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.773626+0000","last_change":"2021-02-10T04:51:43.663830+0000","last_active":"2021-02-10T04:52:10.773626+0000","last_peered":"2021-02-10T04:52:10.773626+0000","last_clean":"2021-02-10T04:52:10.773626+0000","last_became_active":"2021-02-10T04:50:59.567699+0000","last_became_peered":"2021-02-10T04:50:59.567699+0000","last_unstale":"2021-02-10T04:52:10.773626+0000","last_undegraded":"2021-02-10T04:52:10.773626+0000","last_fullsized":"2021-02-10T04:52:10.773626+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:43.663762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:51:43.663762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"1109","reported_epoch":"1135","state":"active+clean","last_fresh":"2021-02-10T04:52:00.498993+0000","last_change":"2021-02-10T04:50:57.568614+0000","last_active":"2021-02-10T04:52:00.498993+0000","last_peered":"2021-02-10T04:52:00.498993+0000","last_clean":"2021-02-10T04:52:00.498993+0000","last_became_active":"2021-02-10T04:25:53.039322+0000","last_became_peered":"2021-02-10T04:25:53.039322+0000","last_unstale":"2021-02-10T04:52:00.498993+0000","last_undegraded":"2021-02-10T04:52:00.498993+0000","last_fullsized":"2021-02-10T04:52:00.498993+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:57.568541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:57.568541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"1000'1071","reported_seq":"4639","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.773106+0000","last_change":"2021-02-10T04:51:39.509148+0000","last_active":"2021-02-10T04:52:10.773106+0000","last_peered":"2021-02-10T04:52:10.773106+0000","last_clean":"2021-02-10T04:52:10.773106+0000","last_became_active":"2021-02-10T04:51:39.508551+0000","last_became_peered":"2021-02-10T04:51:39.508551+0000","last_unstale":"2021-02-10T04:52:10.773106+0000","last_undegraded":"2021-02-10T04:52:10.773106+0000","last_fullsized":"2021-02-10T04:52:10.773106+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":5,"last_scrub":"412'357","last_scrub_stamp":"2021-02-10T04:33:50.582002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:50.582002+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23286368,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":242,"num_read_kb":226835,"num_write":185,"num_write_kb":76005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"1049'31","reported_seq":"1155","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.943590+0000","last_change":"2021-02-10T04:47:15.739807+0000","last_active":"2021-02-10T04:52:02.943590+0000","last_peered":"2021-02-10T04:52:02.943590+0000","last_clean":"2021-02-10T04:52:02.943590+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:52:02.943590+0000","last_undegraded":"2021-02-10T04:52:02.943590+0000","last_fullsized":"2021-02-10T04:52:02.943590+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'31","last_scrub_stamp":"2021-02-10T04:47:15.739700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:47:15.739700+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":41,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"364","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:11.182003+0000","last_change":"2021-02-10T04:51:49.629897+0000","last_active":"2021-02-10T04:52:11.182003+0000","last_peered":"2021-02-10T04:52:11.182003+0000","last_clean":"2021-02-10T04:52:11.182003+0000","last_became_active":"2021-02-10T04:51:49.629375+0000","last_became_peered":"2021-02-10T04:51:49.629375+0000","last_unstale":"2021-02-10T04:52:11.182003+0000","last_undegraded":"2021-02-10T04:52:11.182003+0000","last_fullsized":"2021-02-10T04:52:11.182003+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:47:08.700069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:08.700069+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"787","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:11.181462+0000","last_change":"2021-02-10T04:51:49.501612+0000","last_active":"2021-02-10T04:52:11.181462+0000","last_peered":"2021-02-10T04:52:11.181462+0000","last_clean":"2021-02-10T04:52:11.181462+0000","last_became_active":"2021-02-10T04:51:49.501254+0000","last_became_peered":"2021-02-10T04:51:49.501254+0000","last_unstale":"2021-02-10T04:52:11.181462+0000","last_undegraded":"2021-02-10T04:52:11.181462+0000","last_fullsized":"2021-02-10T04:52:11.181462+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:05.475145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:51:05.475145+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"1101","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.773497+0000","last_change":"2021-02-10T04:51:55.814554+0000","last_active":"2021-02-10T04:52:10.773497+0000","last_peered":"2021-02-10T04:52:10.773497+0000","last_clean":"2021-02-10T04:52:10.773497+0000","last_became_active":"2021-02-10T04:50:59.557657+0000","last_became_peered":"2021-02-10T04:50:59.557657+0000","last_unstale":"2021-02-10T04:52:10.773497+0000","last_undegraded":"2021-02-10T04:52:10.773497+0000","last_fullsized":"2021-02-10T04:52:10.773497+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:55.814486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.814486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"1002'1657","reported_seq":"5242","reported_epoch":"1135","state":"active+clean","last_fresh":"2021-02-10T04:52:00.499311+0000","last_change":"2021-02-10T04:51:41.510994+0000","last_active":"2021-02-10T04:52:00.499311+0000","last_peered":"2021-02-10T04:52:00.499311+0000","last_clean":"2021-02-10T04:52:00.499311+0000","last_became_active":"2021-02-10T04:51:41.509378+0000","last_became_peered":"2021-02-10T04:51:41.509378+0000","last_unstale":"2021-02-10T04:52:00.499311+0000","last_undegraded":"2021-02-10T04:52:00.499311+0000","last_fullsized":"2021-02-10T04:52:00.499311+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1124,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:46:54.881412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:54.881412+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":47335497,"num_objects":20,"num_object_clones":18,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":39443766,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"1048'20","reported_seq":"1235","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.775159+0000","last_change":"2021-02-10T04:51:08.002869+0000","last_active":"2021-02-10T04:52:10.775159+0000","last_peered":"2021-02-10T04:52:10.775159+0000","last_clean":"2021-02-10T04:52:10.775159+0000","last_became_active":"2021-02-10T04:50:59.544484+0000","last_became_peered":"2021-02-10T04:50:59.544484+0000","last_unstale":"2021-02-10T04:52:10.775159+0000","last_undegraded":"2021-02-10T04:52:10.775159+0000","last_fullsized":"2021-02-10T04:52:10.775159+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'20","last_scrub_stamp":"2021-02-10T04:47:12.728019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:47:12.728019+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":20,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"1136","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.771561+0000","last_change":"2021-02-10T04:51:36.844960+0000","last_active":"2021-02-10T04:52:10.771561+0000","last_peered":"2021-02-10T04:52:10.771561+0000","last_clean":"2021-02-10T04:52:10.771561+0000","last_became_active":"2021-02-10T04:51:36.844573+0000","last_became_peered":"2021-02-10T04:51:36.844573+0000","last_unstale":"2021-02-10T04:52:10.771561+0000","last_undegraded":"2021-02-10T04:52:10.771561+0000","last_fullsized":"2021-02-10T04:52:10.771561+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:48:54.943234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:48:54.943234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"1109","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.773607+0000","last_change":"2021-02-10T04:50:56.099710+0000","last_active":"2021-02-10T04:52:10.773607+0000","last_peered":"2021-02-10T04:52:10.773607+0000","last_clean":"2021-02-10T04:52:10.773607+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:52:10.773607+0000","last_undegraded":"2021-02-10T04:52:10.773607+0000","last_fullsized":"2021-02-10T04:52:10.773607+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:56.099651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:50:56.099651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"33","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.942237+0000","last_change":"2021-02-10T04:51:50.036214+0000","last_active":"2021-02-10T04:52:02.942237+0000","last_peered":"2021-02-10T04:52:02.942237+0000","last_clean":"2021-02-10T04:52:02.942237+0000","last_became_active":"2021-02-10T04:51:36.855910+0000","last_became_peered":"2021-02-10T04:51:36.855910+0000","last_unstale":"2021-02-10T04:52:02.942237+0000","last_undegraded":"2021-02-10T04:52:02.942237+0000","last_fullsized":"2021-02-10T04:52:02.942237+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.036117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.036117+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"998'1009","reported_seq":"4970","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774058+0000","last_change":"2021-02-10T04:51:44.507022+0000","last_active":"2021-02-10T04:52:10.774058+0000","last_peered":"2021-02-10T04:52:10.774058+0000","last_clean":"2021-02-10T04:52:10.774058+0000","last_became_active":"2021-02-10T04:51:44.505646+0000","last_became_peered":"2021-02-10T04:51:44.505646+0000","last_unstale":"2021-02-10T04:52:10.774058+0000","last_undegraded":"2021-02-10T04:52:10.774058+0000","last_fullsized":"2021-02-10T04:52:10.774058+0000","mapping_epoch":1126,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1127,"parent":"0.0","parent_split_bits":5,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:13.032506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:13.032506+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49420827,"num_objects":21,"num_object_clones":19,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":36763798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"1049'37","reported_seq":"1168","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.773995+0000","last_change":"2021-02-10T04:51:44.694779+0000","last_active":"2021-02-10T04:52:10.773995+0000","last_peered":"2021-02-10T04:52:10.773995+0000","last_clean":"2021-02-10T04:52:10.773995+0000","last_became_active":"2021-02-10T04:50:59.549412+0000","last_became_peered":"2021-02-10T04:50:59.549412+0000","last_unstale":"2021-02-10T04:52:10.773995+0000","last_undegraded":"2021-02-10T04:52:10.773995+0000","last_fullsized":"2021-02-10T04:52:10.773995+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:44.694711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.694711+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":64,"num_read_kb":50,"num_write":37,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"1271","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774965+0000","last_change":"2021-02-10T04:51:36.893936+0000","last_active":"2021-02-10T04:52:10.774965+0000","last_peered":"2021-02-10T04:52:10.774965+0000","last_clean":"2021-02-10T04:52:10.774965+0000","last_became_active":"2021-02-10T04:51:36.870573+0000","last_became_peered":"2021-02-10T04:51:36.870573+0000","last_unstale":"2021-02-10T04:52:10.774965+0000","last_undegraded":"2021-02-10T04:52:10.774965+0000","last_fullsized":"2021-02-10T04:52:10.774965+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:46:44.241683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:44.241683+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"424","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.775234+0000","last_change":"2021-02-10T04:51:51.836723+0000","last_active":"2021-02-10T04:52:10.775234+0000","last_peered":"2021-02-10T04:52:10.775234+0000","last_clean":"2021-02-10T04:52:10.775234+0000","last_became_active":"2021-02-10T04:50:59.564977+0000","last_became_peered":"2021-02-10T04:50:59.564977+0000","last_unstale":"2021-02-10T04:52:10.775234+0000","last_undegraded":"2021-02-10T04:52:10.775234+0000","last_fullsized":"2021-02-10T04:52:10.775234+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.836638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.836638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"1065","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.773529+0000","last_change":"2021-02-10T04:51:07.998582+0000","last_active":"2021-02-10T04:52:10.773529+0000","last_peered":"2021-02-10T04:52:10.773529+0000","last_clean":"2021-02-10T04:52:10.773529+0000","last_became_active":"2021-02-10T04:50:59.563229+0000","last_became_peered":"2021-02-10T04:50:59.563229+0000","last_unstale":"2021-02-10T04:52:10.773529+0000","last_undegraded":"2021-02-10T04:52:10.773529+0000","last_fullsized":"2021-02-10T04:52:10.773529+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:58.514318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_clean_scrub_stamp":"2021-02-10T04:46:58.514318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"1049'42","reported_seq":"1200","reported_epoch":"1135","state":"active+clean","last_fresh":"2021-02-10T04:52:00.500098+0000","last_change":"2021-02-10T04:51:45.336870+0000","last_active":"2021-02-10T04:52:00.500098+0000","last_peered":"2021-02-10T04:52:00.500098+0000","last_clean":"2021-02-10T04:52:00.500098+0000","last_became_active":"2021-02-10T04:50:59.577949+0000","last_became_peered":"2021-02-10T04:50:59.577949+0000","last_unstale":"2021-02-10T04:52:00.500098+0000","last_undegraded":"2021-02-10T04:52:00.500098+0000","last_fullsized":"2021-02-10T04:52:00.500098+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'42","last_scrub_stamp":"2021-02-10T04:51:45.336772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.336772+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":75,"num_read_kb":57,"num_write":42,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"1177","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.770880+0000","last_change":"2021-02-10T04:50:04.287211+0000","last_active":"2021-02-10T04:52:10.770880+0000","last_peered":"2021-02-10T04:52:10.770880+0000","last_clean":"2021-02-10T04:52:10.770880+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:52:10.770880+0000","last_undegraded":"2021-02-10T04:52:10.770880+0000","last_fullsized":"2021-02-10T04:52:10.770880+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:04.287114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:04.287114+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"1098'3","reported_seq":"1429","reported_epoch":"1138","state":"active+clean","last_fresh":"2021-02-10T04:52:11.969830+0000","last_change":"2021-02-10T04:50:15.255211+0000","last_active":"2021-02-10T04:52:11.969830+0000","last_peered":"2021-02-10T04:52:11.969830+0000","last_clean":"2021-02-10T04:52:11.969830+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:52:11.969830+0000","last_undegraded":"2021-02-10T04:52:11.969830+0000","last_fullsized":"2021-02-10T04:52:11.969830+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"1098'3","last_scrub_stamp":"2021-02-10T04:50:15.255137+0000","last_deep_scrub":"1098'3","last_deep_scrub_stamp":"2021-02-10T04:50:15.255137+0000","last_clean_scrub_stamp":"2021-02-10T04:50:15.255137+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"1061","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.771343+0000","last_change":"2021-02-10T04:51:47.493125+0000","last_active":"2021-02-10T04:52:10.771343+0000","last_peered":"2021-02-10T04:52:10.771343+0000","last_clean":"2021-02-10T04:52:10.771343+0000","last_became_active":"2021-02-10T04:51:36.852591+0000","last_became_peered":"2021-02-10T04:51:36.852591+0000","last_unstale":"2021-02-10T04:52:10.771343+0000","last_undegraded":"2021-02-10T04:52:10.771343+0000","last_fullsized":"2021-02-10T04:52:10.771343+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:47.493057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.493057+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"1049'51","reported_seq":"1198","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774996+0000","last_change":"2021-02-10T04:50:53.153340+0000","last_active":"2021-02-10T04:52:10.774996+0000","last_peered":"2021-02-10T04:52:10.774996+0000","last_clean":"2021-02-10T04:52:10.774996+0000","last_became_active":"2021-02-10T04:25:49.015760+0000","last_became_peered":"2021-02-10T04:25:49.015760+0000","last_unstale":"2021-02-10T04:52:10.774996+0000","last_undegraded":"2021-02-10T04:52:10.774996+0000","last_fullsized":"2021-02-10T04:52:10.774996+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'51","last_scrub_stamp":"2021-02-10T04:50:53.153251+0000","last_deep_scrub":"76'35","last_deep_scrub_stamp":"2021-02-10T04:26:58.198641+0000","last_clean_scrub_stamp":"2021-02-10T04:50:53.153251+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":74,"num_read_kb":54,"num_write":40,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"1069","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.773842+0000","last_change":"2021-02-10T04:51:07.997058+0000","last_active":"2021-02-10T04:52:10.773842+0000","last_peered":"2021-02-10T04:52:10.773842+0000","last_clean":"2021-02-10T04:52:10.773842+0000","last_became_active":"2021-02-10T04:50:59.555151+0000","last_became_peered":"2021-02-10T04:50:59.555151+0000","last_unstale":"2021-02-10T04:52:10.773842+0000","last_undegraded":"2021-02-10T04:52:10.773842+0000","last_fullsized":"2021-02-10T04:52:10.773842+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:42.537443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:42.537443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"1124","reported_epoch":"1135","state":"active+clean","last_fresh":"2021-02-10T04:52:00.500106+0000","last_change":"2021-02-10T04:51:36.859111+0000","last_active":"2021-02-10T04:52:00.500106+0000","last_peered":"2021-02-10T04:52:00.500106+0000","last_clean":"2021-02-10T04:52:00.500106+0000","last_became_active":"2021-02-10T04:51:36.858767+0000","last_became_peered":"2021-02-10T04:51:36.858767+0000","last_unstale":"2021-02-10T04:52:00.500106+0000","last_undegraded":"2021-02-10T04:52:00.500106+0000","last_fullsized":"2021-02-10T04:52:00.500106+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.938731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.938731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"344","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:11.181912+0000","last_change":"2021-02-10T04:51:49.499732+0000","last_active":"2021-02-10T04:52:11.181912+0000","last_peered":"2021-02-10T04:52:11.181912+0000","last_clean":"2021-02-10T04:52:11.181912+0000","last_became_active":"2021-02-10T04:51:49.499348+0000","last_became_peered":"2021-02-10T04:51:49.499348+0000","last_unstale":"2021-02-10T04:52:11.181912+0000","last_undegraded":"2021-02-10T04:52:11.181912+0000","last_fullsized":"2021-02-10T04:52:11.181912+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:49.391902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:49.391902+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"1049'39","reported_seq":"44","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.943995+0000","last_change":"2021-02-10T04:51:37.360371+0000","last_active":"2021-02-10T04:52:02.943995+0000","last_peered":"2021-02-10T04:52:02.943995+0000","last_clean":"2021-02-10T04:52:02.943995+0000","last_became_active":"2021-02-10T04:51:36.876346+0000","last_became_peered":"2021-02-10T04:51:36.876346+0000","last_unstale":"2021-02-10T04:52:02.943995+0000","last_undegraded":"2021-02-10T04:52:02.943995+0000","last_fullsized":"2021-02-10T04:52:02.943995+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'39","last_scrub_stamp":"2021-02-10T04:51:08.433562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.433562+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"1121","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774450+0000","last_change":"2021-02-10T04:46:47.143081+0000","last_active":"2021-02-10T04:52:10.774450+0000","last_peered":"2021-02-10T04:52:10.774450+0000","last_clean":"2021-02-10T04:52:10.774450+0000","last_became_active":"2021-02-10T04:25:36.336442+0000","last_became_peered":"2021-02-10T04:25:36.336442+0000","last_unstale":"2021-02-10T04:52:10.774450+0000","last_undegraded":"2021-02-10T04:52:10.774450+0000","last_fullsized":"2021-02-10T04:52:10.774450+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:46:47.142978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:47.142978+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"346","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:11.182170+0000","last_change":"2021-02-10T04:51:49.630855+0000","last_active":"2021-02-10T04:52:11.182170+0000","last_peered":"2021-02-10T04:52:11.182170+0000","last_clean":"2021-02-10T04:52:11.182170+0000","last_became_active":"2021-02-10T04:51:49.630513+0000","last_became_peered":"2021-02-10T04:51:49.630513+0000","last_unstale":"2021-02-10T04:52:11.182170+0000","last_undegraded":"2021-02-10T04:52:11.182170+0000","last_fullsized":"2021-02-10T04:52:11.182170+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:08.758129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.758129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"1105","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.943759+0000","last_change":"2021-02-10T04:51:44.799512+0000","last_active":"2021-02-10T04:52:02.943759+0000","last_peered":"2021-02-10T04:52:02.943759+0000","last_clean":"2021-02-10T04:52:02.943759+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:52:02.943759+0000","last_undegraded":"2021-02-10T04:52:02.943759+0000","last_fullsized":"2021-02-10T04:52:02.943759+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:44.799429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.799429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"1066'44","reported_seq":"1146","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774154+0000","last_change":"2021-02-10T04:51:07.985331+0000","last_active":"2021-02-10T04:52:10.774154+0000","last_peered":"2021-02-10T04:52:10.774154+0000","last_clean":"2021-02-10T04:52:10.774154+0000","last_became_active":"2021-02-10T04:50:59.563388+0000","last_became_peered":"2021-02-10T04:50:59.563388+0000","last_unstale":"2021-02-10T04:52:10.774154+0000","last_undegraded":"2021-02-10T04:52:10.774154+0000","last_fullsized":"2021-02-10T04:52:10.774154+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1066'44","last_scrub_stamp":"2021-02-10T04:47:18.806645+0000","last_deep_scrub":"1066'44","last_deep_scrub_stamp":"2021-02-10T04:47:18.806645+0000","last_clean_scrub_stamp":"2021-02-10T04:47:18.806645+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":67,"num_read_kb":49,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"1090","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.773954+0000","last_change":"2021-02-10T04:51:08.006463+0000","last_active":"2021-02-10T04:52:10.773954+0000","last_peered":"2021-02-10T04:52:10.773954+0000","last_clean":"2021-02-10T04:52:10.773954+0000","last_became_active":"2021-02-10T04:50:59.563657+0000","last_became_peered":"2021-02-10T04:50:59.563657+0000","last_unstale":"2021-02-10T04:52:10.773954+0000","last_undegraded":"2021-02-10T04:52:10.773954+0000","last_fullsized":"2021-02-10T04:52:10.773954+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_clean_scrub_stamp":"2021-02-10T04:51:02.455702+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"1135","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.943588+0000","last_change":"2021-02-10T04:51:36.860480+0000","last_active":"2021-02-10T04:52:02.943588+0000","last_peered":"2021-02-10T04:52:02.943588+0000","last_clean":"2021-02-10T04:52:02.943588+0000","last_became_active":"2021-02-10T04:51:36.860129+0000","last_became_peered":"2021-02-10T04:51:36.860129+0000","last_unstale":"2021-02-10T04:52:02.943588+0000","last_undegraded":"2021-02-10T04:52:02.943588+0000","last_fullsized":"2021-02-10T04:52:02.943588+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:58.666264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:46:58.666264+0000","last_clean_scrub_stamp":"2021-02-10T04:46:58.666264+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"1097","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.775078+0000","last_change":"2021-02-10T04:51:08.002745+0000","last_active":"2021-02-10T04:52:10.775078+0000","last_peered":"2021-02-10T04:52:10.775078+0000","last_clean":"2021-02-10T04:52:10.775078+0000","last_became_active":"2021-02-10T04:50:59.539582+0000","last_became_peered":"2021-02-10T04:50:59.539582+0000","last_unstale":"2021-02-10T04:52:10.775078+0000","last_undegraded":"2021-02-10T04:52:10.775078+0000","last_fullsized":"2021-02-10T04:52:10.775078+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:26.576960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:46:26.576960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"1003'739","reported_seq":"3881","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.773178+0000","last_change":"2021-02-10T04:51:40.512224+0000","last_active":"2021-02-10T04:52:10.773178+0000","last_peered":"2021-02-10T04:52:10.773178+0000","last_clean":"2021-02-10T04:52:10.773178+0000","last_became_active":"2021-02-10T04:51:40.511505+0000","last_became_peered":"2021-02-10T04:51:40.511505+0000","last_unstale":"2021-02-10T04:52:10.773178+0000","last_undegraded":"2021-02-10T04:52:10.773178+0000","last_fullsized":"2021-02-10T04:52:10.773178+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:47:03.951729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:03.951729+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22230846,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":18429397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"1112'47","reported_seq":"1208","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.943820+0000","last_change":"2021-02-10T04:51:02.860932+0000","last_active":"2021-02-10T04:52:02.943820+0000","last_peered":"2021-02-10T04:52:02.943820+0000","last_clean":"2021-02-10T04:52:02.943820+0000","last_became_active":"2021-02-10T04:50:59.570786+0000","last_became_peered":"2021-02-10T04:50:59.570786+0000","last_unstale":"2021-02-10T04:52:02.943820+0000","last_undegraded":"2021-02-10T04:52:02.943820+0000","last_fullsized":"2021-02-10T04:52:02.943820+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1112'47","last_scrub_stamp":"2021-02-10T04:51:02.860835+0000","last_deep_scrub":"1112'47","last_deep_scrub_stamp":"2021-02-10T04:51:02.860835+0000","last_clean_scrub_stamp":"2021-02-10T04:51:02.860835+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":71,"num_read_kb":55,"num_write":38,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"1047","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.771546+0000","last_change":"2021-02-10T04:51:36.858788+0000","last_active":"2021-02-10T04:52:10.771546+0000","last_peered":"2021-02-10T04:52:10.771546+0000","last_clean":"2021-02-10T04:52:10.771546+0000","last_became_active":"2021-02-10T04:51:36.858446+0000","last_became_peered":"2021-02-10T04:51:36.858446+0000","last_unstale":"2021-02-10T04:52:10.771546+0000","last_undegraded":"2021-02-10T04:52:10.771546+0000","last_fullsized":"2021-02-10T04:52:10.771546+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:59.760263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:59.760263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"1133","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.771699+0000","last_change":"2021-02-10T04:51:02.394551+0000","last_active":"2021-02-10T04:52:10.771699+0000","last_peered":"2021-02-10T04:52:10.771699+0000","last_clean":"2021-02-10T04:52:10.771699+0000","last_became_active":"2021-02-10T04:50:59.575340+0000","last_became_peered":"2021-02-10T04:50:59.575340+0000","last_unstale":"2021-02-10T04:52:10.771699+0000","last_undegraded":"2021-02-10T04:52:10.771699+0000","last_fullsized":"2021-02-10T04:52:10.771699+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:02.394440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:02.394440+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"1109","reported_epoch":"1135","state":"active+clean","last_fresh":"2021-02-10T04:52:00.501257+0000","last_change":"2021-02-10T04:52:00.501188+0000","last_active":"2021-02-10T04:52:00.501257+0000","last_peered":"2021-02-10T04:52:00.501257+0000","last_clean":"2021-02-10T04:52:00.501257+0000","last_became_active":"2021-02-10T04:25:53.039844+0000","last_became_peered":"2021-02-10T04:25:53.039844+0000","last_unstale":"2021-02-10T04:52:00.501257+0000","last_undegraded":"2021-02-10T04:52:00.501257+0000","last_fullsized":"2021-02-10T04:52:00.501257+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:00.501127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:00.501127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"259'74","reported_seq":"142","reported_epoch":"1135","state":"active+clean","last_fresh":"2021-02-10T04:52:00.498968+0000","last_change":"2021-02-10T04:46:49.745214+0000","last_active":"2021-02-10T04:52:00.498968+0000","last_peered":"2021-02-10T04:52:00.498968+0000","last_clean":"2021-02-10T04:52:00.498968+0000","last_became_active":"2021-02-10T04:44:27.546590+0000","last_became_peered":"2021-02-10T04:44:27.546590+0000","last_unstale":"2021-02-10T04:52:00.498968+0000","last_undegraded":"2021-02-10T04:52:00.498968+0000","last_fullsized":"2021-02-10T04:52:00.498968+0000","mapping_epoch":1009,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1010,"parent":"0.0","parent_split_bits":5,"last_scrub":"259'74","last_scrub_stamp":"2021-02-10T04:46:49.745104+0000","last_deep_scrub":"259'74","last_deep_scrub_stamp":"2021-02-10T04:46:49.745104+0000","last_clean_scrub_stamp":"2021-02-10T04:46:49.745104+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"1061'20","reported_seq":"1156","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.943508+0000","last_change":"2021-02-10T04:50:59.569616+0000","last_active":"2021-02-10T04:52:02.943508+0000","last_peered":"2021-02-10T04:52:02.943508+0000","last_clean":"2021-02-10T04:52:02.943508+0000","last_became_active":"2021-02-10T04:50:59.569344+0000","last_became_peered":"2021-02-10T04:50:59.569344+0000","last_unstale":"2021-02-10T04:52:02.943508+0000","last_undegraded":"2021-02-10T04:52:02.943508+0000","last_fullsized":"2021-02-10T04:52:02.943508+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1061'20","last_scrub_stamp":"2021-02-10T04:46:55.646261+0000","last_deep_scrub":"1061'20","last_deep_scrub_stamp":"2021-02-10T04:46:55.646261+0000","last_clean_scrub_stamp":"2021-02-10T04:46:55.646261+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":17,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"1034","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774700+0000","last_change":"2021-02-10T04:50:59.167977+0000","last_active":"2021-02-10T04:52:10.774700+0000","last_peered":"2021-02-10T04:52:10.774700+0000","last_clean":"2021-02-10T04:52:10.774700+0000","last_became_active":"2021-02-10T04:29:07.157944+0000","last_became_peered":"2021-02-10T04:29:07.157944+0000","last_unstale":"2021-02-10T04:52:10.774700+0000","last_undegraded":"2021-02-10T04:52:10.774700+0000","last_fullsized":"2021-02-10T04:52:10.774700+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:59.167911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:50:59.167911+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"1027","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.943909+0000","last_change":"2021-02-10T04:51:45.813755+0000","last_active":"2021-02-10T04:52:02.943909+0000","last_peered":"2021-02-10T04:52:02.943909+0000","last_clean":"2021-02-10T04:52:02.943909+0000","last_became_active":"2021-02-10T04:29:07.146525+0000","last_became_peered":"2021-02-10T04:29:07.146525+0000","last_unstale":"2021-02-10T04:52:02.943909+0000","last_undegraded":"2021-02-10T04:52:02.943909+0000","last_fullsized":"2021-02-10T04:52:02.943909+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:45.813676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.813676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"1122","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.772609+0000","last_change":"2021-02-10T04:51:09.433892+0000","last_active":"2021-02-10T04:52:10.772609+0000","last_peered":"2021-02-10T04:52:10.772609+0000","last_clean":"2021-02-10T04:52:10.772609+0000","last_became_active":"2021-02-10T04:50:59.539256+0000","last_became_peered":"2021-02-10T04:50:59.539256+0000","last_unstale":"2021-02-10T04:52:10.772609+0000","last_undegraded":"2021-02-10T04:52:10.772609+0000","last_fullsized":"2021-02-10T04:52:10.772609+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:09.433793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:09.433793+0000","last_clean_scrub_stamp":"2021-02-10T04:51:09.433793+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"1048'44","reported_seq":"1181","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.943073+0000","last_change":"2021-02-10T04:50:48.668186+0000","last_active":"2021-02-10T04:52:02.943073+0000","last_peered":"2021-02-10T04:52:02.943073+0000","last_clean":"2021-02-10T04:52:02.943073+0000","last_became_active":"2021-02-10T04:25:49.010481+0000","last_became_peered":"2021-02-10T04:25:49.010481+0000","last_unstale":"2021-02-10T04:52:02.943073+0000","last_undegraded":"2021-02-10T04:52:02.943073+0000","last_fullsized":"2021-02-10T04:52:02.943073+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'44","last_scrub_stamp":"2021-02-10T04:50:48.668079+0000","last_deep_scrub":"78'32","last_deep_scrub_stamp":"2021-02-10T04:27:09.170428+0000","last_clean_scrub_stamp":"2021-02-10T04:50:48.668079+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":65,"num_read_kb":46,"num_write":34,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"1108","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774707+0000","last_change":"2021-02-10T04:51:07.979394+0000","last_active":"2021-02-10T04:52:10.774707+0000","last_peered":"2021-02-10T04:52:10.774707+0000","last_clean":"2021-02-10T04:52:10.774707+0000","last_became_active":"2021-02-10T04:50:59.557394+0000","last_became_peered":"2021-02-10T04:50:59.557394+0000","last_unstale":"2021-02-10T04:52:10.774707+0000","last_undegraded":"2021-02-10T04:52:10.774707+0000","last_fullsized":"2021-02-10T04:52:10.774707+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:33.543559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:46:33.543559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"792","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.770831+0000","last_change":"2021-02-10T04:50:06.195327+0000","last_active":"2021-02-10T04:52:10.770831+0000","last_peered":"2021-02-10T04:52:10.770831+0000","last_clean":"2021-02-10T04:52:10.770831+0000","last_became_active":"2021-02-10T04:33:05.169159+0000","last_became_peered":"2021-02-10T04:33:05.169159+0000","last_unstale":"2021-02-10T04:52:10.770831+0000","last_undegraded":"2021-02-10T04:52:10.770831+0000","last_fullsized":"2021-02-10T04:52:10.770831+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:06.195234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:06.195234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"1186","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.943698+0000","last_change":"2021-02-10T04:47:12.800261+0000","last_active":"2021-02-10T04:52:02.943698+0000","last_peered":"2021-02-10T04:52:02.943698+0000","last_clean":"2021-02-10T04:52:02.943698+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:52:02.943698+0000","last_undegraded":"2021-02-10T04:52:02.943698+0000","last_fullsized":"2021-02-10T04:52:02.943698+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:12.800147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:12.800147+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"1049'68","reported_seq":"1287","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.773697+0000","last_change":"2021-02-10T04:51:36.907653+0000","last_active":"2021-02-10T04:52:10.773697+0000","last_peered":"2021-02-10T04:52:10.773697+0000","last_clean":"2021-02-10T04:52:10.773697+0000","last_became_active":"2021-02-10T04:51:36.851354+0000","last_became_peered":"2021-02-10T04:51:36.851354+0000","last_unstale":"2021-02-10T04:52:10.773697+0000","last_undegraded":"2021-02-10T04:52:10.773697+0000","last_fullsized":"2021-02-10T04:52:10.773697+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'68","last_scrub_stamp":"2021-02-10T04:47:09.608881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:47:09.608881+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":124,"num_read_kb":101,"num_write":69,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"1124","reported_epoch":"1135","state":"active+clean","last_fresh":"2021-02-10T04:52:00.500057+0000","last_change":"2021-02-10T04:47:17.919840+0000","last_active":"2021-02-10T04:52:00.500057+0000","last_peered":"2021-02-10T04:52:00.500057+0000","last_clean":"2021-02-10T04:52:00.500057+0000","last_became_active":"2021-02-10T04:25:36.336112+0000","last_became_peered":"2021-02-10T04:25:36.336112+0000","last_unstale":"2021-02-10T04:52:00.500057+0000","last_undegraded":"2021-02-10T04:52:00.500057+0000","last_fullsized":"2021-02-10T04:52:00.500057+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:17.919745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:17.919745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"419","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.773018+0000","last_change":"2021-02-10T04:51:07.970205+0000","last_active":"2021-02-10T04:52:10.773018+0000","last_peered":"2021-02-10T04:52:10.773018+0000","last_clean":"2021-02-10T04:52:10.773018+0000","last_became_active":"2021-02-10T04:50:59.572077+0000","last_became_peered":"2021-02-10T04:50:59.572077+0000","last_unstale":"2021-02-10T04:52:10.773018+0000","last_undegraded":"2021-02-10T04:52:10.773018+0000","last_fullsized":"2021-02-10T04:52:10.773018+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:19.755718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:19.755718+0000","last_clean_scrub_stamp":"2021-02-10T04:47:19.755718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"1101","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.773836+0000","last_change":"2021-02-10T04:51:59.838856+0000","last_active":"2021-02-10T04:52:10.773836+0000","last_peered":"2021-02-10T04:52:10.773836+0000","last_clean":"2021-02-10T04:52:10.773836+0000","last_became_active":"2021-02-10T04:50:59.563156+0000","last_became_peered":"2021-02-10T04:50:59.563156+0000","last_unstale":"2021-02-10T04:52:10.773836+0000","last_undegraded":"2021-02-10T04:52:10.773836+0000","last_fullsized":"2021-02-10T04:52:10.773836+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.838790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.838790+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":"828","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.771179+0000","last_change":"2021-02-10T04:51:36.852559+0000","last_active":"2021-02-10T04:52:10.771179+0000","last_peered":"2021-02-10T04:52:10.771179+0000","last_clean":"2021-02-10T04:52:10.771179+0000","last_became_active":"2021-02-10T04:51:36.852058+0000","last_became_peered":"2021-02-10T04:51:36.852058+0000","last_unstale":"2021-02-10T04:52:10.771179+0000","last_undegraded":"2021-02-10T04:52:10.771179+0000","last_fullsized":"2021-02-10T04:52:10.771179+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:20.635536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:20.635536+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.635536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.20","version":"0'0","reported_seq":"1097","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.771017+0000","last_change":"2021-02-10T04:51:56.449184+0000","last_active":"2021-02-10T04:52:10.771017+0000","last_peered":"2021-02-10T04:52:10.771017+0000","last_clean":"2021-02-10T04:52:10.771017+0000","last_became_active":"2021-02-10T04:29:07.148937+0000","last_became_peered":"2021-02-10T04:29:07.148937+0000","last_unstale":"2021-02-10T04:52:10.771017+0000","last_undegraded":"2021-02-10T04:52:10.771017+0000","last_fullsized":"2021-02-10T04:52:10.771017+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:56.449085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:41.375276+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.449085+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":"1060","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774613+0000","last_change":"2021-02-10T04:51:36.870420+0000","last_active":"2021-02-10T04:52:10.774613+0000","last_peered":"2021-02-10T04:52:10.774613+0000","last_clean":"2021-02-10T04:52:10.774613+0000","last_became_active":"2021-02-10T04:51:36.869785+0000","last_became_peered":"2021-02-10T04:51:36.869785+0000","last_unstale":"2021-02-10T04:52:10.774613+0000","last_undegraded":"2021-02-10T04:52:10.774613+0000","last_fullsized":"2021-02-10T04:52:10.774613+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:04.210002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:50:04.210002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":"28","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.943763+0000","last_change":"2021-02-10T04:51:36.868469+0000","last_active":"2021-02-10T04:52:02.943763+0000","last_peered":"2021-02-10T04:52:02.943763+0000","last_clean":"2021-02-10T04:52:02.943763+0000","last_became_active":"2021-02-10T04:51:36.867396+0000","last_became_peered":"2021-02-10T04:51:36.867396+0000","last_unstale":"2021-02-10T04:52:02.943763+0000","last_undegraded":"2021-02-10T04:52:02.943763+0000","last_fullsized":"2021-02-10T04:52:02.943763+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:47.481686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:47.481686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":"21","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:11.181705+0000","last_change":"2021-02-10T04:51:49.629284+0000","last_active":"2021-02-10T04:52:11.181705+0000","last_peered":"2021-02-10T04:52:11.181705+0000","last_clean":"2021-02-10T04:52:11.181705+0000","last_became_active":"2021-02-10T04:51:49.628920+0000","last_became_peered":"2021-02-10T04:51:49.628920+0000","last_unstale":"2021-02-10T04:52:11.181705+0000","last_undegraded":"2021-02-10T04:52:11.181705+0000","last_fullsized":"2021-02-10T04:52:11.181705+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.706372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.706372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":"1147","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774215+0000","last_change":"2021-02-10T04:51:57.287452+0000","last_active":"2021-02-10T04:52:10.774215+0000","last_peered":"2021-02-10T04:52:10.774215+0000","last_clean":"2021-02-10T04:52:10.774215+0000","last_became_active":"2021-02-10T04:50:59.566827+0000","last_became_peered":"2021-02-10T04:50:59.566827+0000","last_unstale":"2021-02-10T04:52:10.774215+0000","last_undegraded":"2021-02-10T04:52:10.774215+0000","last_fullsized":"2021-02-10T04:52:10.774215+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.287363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.287363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":"106","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774642+0000","last_change":"2021-02-10T04:50:10.308350+0000","last_active":"2021-02-10T04:52:10.774642+0000","last_peered":"2021-02-10T04:52:10.774642+0000","last_clean":"2021-02-10T04:52:10.774642+0000","last_became_active":"2021-02-10T04:45:49.551121+0000","last_became_peered":"2021-02-10T04:45:49.551121+0000","last_unstale":"2021-02-10T04:52:10.774642+0000","last_undegraded":"2021-02-10T04:52:10.774642+0000","last_fullsized":"2021-02-10T04:52:10.774642+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:10.308282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:10.308282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":"1032","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.773955+0000","last_change":"2021-02-10T04:51:07.989063+0000","last_active":"2021-02-10T04:52:10.773955+0000","last_peered":"2021-02-10T04:52:10.773955+0000","last_clean":"2021-02-10T04:52:10.773955+0000","last_became_active":"2021-02-10T04:50:59.561442+0000","last_became_peered":"2021-02-10T04:50:59.561442+0000","last_unstale":"2021-02-10T04:52:10.773955+0000","last_undegraded":"2021-02-10T04:52:10.773955+0000","last_fullsized":"2021-02-10T04:52:10.773955+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:51.620102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:44:51.620102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":"28","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.943887+0000","last_change":"2021-02-10T04:51:36.867457+0000","last_active":"2021-02-10T04:52:02.943887+0000","last_peered":"2021-02-10T04:52:02.943887+0000","last_clean":"2021-02-10T04:52:02.943887+0000","last_became_active":"2021-02-10T04:51:36.867061+0000","last_became_peered":"2021-02-10T04:51:36.867061+0000","last_unstale":"2021-02-10T04:52:02.943887+0000","last_undegraded":"2021-02-10T04:52:02.943887+0000","last_fullsized":"2021-02-10T04:52:02.943887+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:08.271977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:08.271977+0000","last_clean_scrub_stamp":"2021-02-10T04:50:08.271977+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":"1076","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774807+0000","last_change":"2021-02-10T04:51:07.182775+0000","last_active":"2021-02-10T04:52:10.774807+0000","last_peered":"2021-02-10T04:52:10.774807+0000","last_clean":"2021-02-10T04:52:10.774807+0000","last_became_active":"2021-02-10T04:28:23.126368+0000","last_became_peered":"2021-02-10T04:28:23.126368+0000","last_unstale":"2021-02-10T04:52:10.774807+0000","last_undegraded":"2021-02-10T04:52:10.774807+0000","last_fullsized":"2021-02-10T04:52:10.774807+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.182678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.182678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":"28","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.944173+0000","last_change":"2021-02-10T04:51:36.855468+0000","last_active":"2021-02-10T04:52:02.944173+0000","last_peered":"2021-02-10T04:52:02.944173+0000","last_clean":"2021-02-10T04:52:02.944173+0000","last_became_active":"2021-02-10T04:51:36.855101+0000","last_became_peered":"2021-02-10T04:51:36.855101+0000","last_unstale":"2021-02-10T04:52:02.944173+0000","last_undegraded":"2021-02-10T04:52:02.944173+0000","last_fullsized":"2021-02-10T04:52:02.944173+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:07.767250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:07.767250+0000","last_clean_scrub_stamp":"2021-02-10T04:50:07.767250+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.25","version":"53'2","reported_seq":"1087","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.943807+0000","last_change":"2021-02-10T04:50:12.772167+0000","last_active":"2021-02-10T04:52:02.943807+0000","last_peered":"2021-02-10T04:52:02.943807+0000","last_clean":"2021-02-10T04:52:02.943807+0000","last_became_active":"2021-02-10T04:28:23.467770+0000","last_became_peered":"2021-02-10T04:28:23.467770+0000","last_unstale":"2021-02-10T04:52:02.943807+0000","last_undegraded":"2021-02-10T04:52:02.943807+0000","last_fullsized":"2021-02-10T04:52:02.943807+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:50:12.772086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:12.772086+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":"1178","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.773978+0000","last_change":"2021-02-10T04:50:59.567042+0000","last_active":"2021-02-10T04:52:10.773978+0000","last_peered":"2021-02-10T04:52:10.773978+0000","last_clean":"2021-02-10T04:52:10.773978+0000","last_became_active":"2021-02-10T04:50:59.566548+0000","last_became_peered":"2021-02-10T04:50:59.566548+0000","last_unstale":"2021-02-10T04:52:10.773978+0000","last_undegraded":"2021-02-10T04:52:10.773978+0000","last_fullsized":"2021-02-10T04:52:10.773978+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:27.926662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:44:27.926662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.26","version":"53'1","reported_seq":"527","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.943578+0000","last_change":"2021-02-10T04:47:11.808609+0000","last_active":"2021-02-10T04:52:02.943578+0000","last_peered":"2021-02-10T04:52:02.943578+0000","last_clean":"2021-02-10T04:52:02.943578+0000","last_became_active":"2021-02-10T04:37:37.231081+0000","last_became_peered":"2021-02-10T04:37:37.231081+0000","last_unstale":"2021-02-10T04:52:02.943578+0000","last_undegraded":"2021-02-10T04:52:02.943578+0000","last_fullsized":"2021-02-10T04:52:02.943578+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:47:11.808550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:11.808550+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":"1060","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774476+0000","last_change":"2021-02-10T04:47:22.953445+0000","last_active":"2021-02-10T04:52:10.774476+0000","last_peered":"2021-02-10T04:52:10.774476+0000","last_clean":"2021-02-10T04:52:10.774476+0000","last_became_active":"2021-02-10T04:45:49.552026+0000","last_became_peered":"2021-02-10T04:45:49.552026+0000","last_unstale":"2021-02-10T04:52:10.774476+0000","last_undegraded":"2021-02-10T04:52:10.774476+0000","last_fullsized":"2021-02-10T04:52:10.774476+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:22.953356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.953356+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":"1098","reported_epoch":"1136","state":"active+clean","last_fresh":"2021-02-10T04:52:02.944882+0000","last_change":"2021-02-10T04:47:16.706513+0000","last_active":"2021-02-10T04:52:02.944882+0000","last_peered":"2021-02-10T04:52:02.944882+0000","last_clean":"2021-02-10T04:52:02.944882+0000","last_became_active":"2021-02-10T04:29:07.145907+0000","last_became_peered":"2021-02-10T04:29:07.145907+0000","last_unstale":"2021-02-10T04:52:02.944882+0000","last_undegraded":"2021-02-10T04:52:02.944882+0000","last_fullsized":"2021-02-10T04:52:02.944882+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:16.706416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:16.706416+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":"1061","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.774894+0000","last_change":"2021-02-10T04:51:07.984986+0000","last_active":"2021-02-10T04:52:10.774894+0000","last_peered":"2021-02-10T04:52:10.774894+0000","last_clean":"2021-02-10T04:52:10.774894+0000","last_became_active":"2021-02-10T04:50:59.569162+0000","last_became_peered":"2021-02-10T04:50:59.569162+0000","last_unstale":"2021-02-10T04:52:10.774894+0000","last_undegraded":"2021-02-10T04:52:10.774894+0000","last_fullsized":"2021-02-10T04:52:10.774894+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:12.555666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:46:12.555666+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":"1086","reported_epoch":"1137","state":"active+clean","last_fresh":"2021-02-10T04:52:10.773390+0000","last_change":"2021-02-10T04:50:59.528569+0000","last_active":"2021-02-10T04:52:10.773390+0000","last_peered":"2021-02-10T04:52:10.773390+0000","last_clean":"2021-02-10T04:52:10.773390+0000","last_became_active":"2021-02-10T04:50:59.527945+0000","last_became_peered":"2021-02-10T04:50:59.527945+0000","last_unstale":"2021-02-10T04:52:10.773390+0000","last_undegraded":"2021-02-10T04:52:10.773390+0000","last_fullsized":"2021-02-10T04:52:10.773390+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:50.057162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_clean_scrub_stamp":"2021-02-10T04:46:50.057162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":940005779,"num_objects":386,"num_object_clones":336,"num_object_copies":1158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":386,"num_whiteouts":12,"num_read":4562,"num_read_kb":4769301,"num_write":3301,"num_write_kb":1265045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":858,"num_bytes_recovered":397927832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1927012352,"data_stored":2511133359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23681,"ondisk_log_size":23681,"up":78,"acting":78,"num_store_stats":8},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":1945,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":17,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1069,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":5835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":126,"acting":126,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":455,"num_read_kb":450,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":2294,"num_read_kb":1767,"num_write":1306,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":11134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1415,"ondisk_log_size":1415,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":126,"acting":126,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365121,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":228292,"kb_used_data":193020,"kb_used_omap":0,"kb_used_meta":35264,"kb_avail":93516860,"statfs":{"total":95995035648,"available":95761264640,"internally_reserved":0,"allocated":197652480,"data_stored":268829609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36110336},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:27 2021","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":1.2370000000000001,"15min":1.284},"min":{"1min":0.50800000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1220000000000001,"5min":34.299999999999997,"15min":67.277000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.5109999999999999,"15min":1.548},"min":{"1min":0.46300000000000002,"5min":0.438,"15min":0.438},"max":{"1min":2.8690000000000002,"5min":34.652999999999999,"15min":66.933999999999997},"last":0.59199999999999997}]},{"osd":1,"last update":"Wed Feb 10 04:51:27 2021","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.89300000000000002,"15min":0.995},"min":{"1min":0.497,"5min":0.497,"15min":0.48399999999999999},"max":{"1min":1.5169999999999999,"5min":2.3420000000000001,"15min":10.140000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.91300000000000003,"5min":1.026,"15min":1.0129999999999999},"min":{"1min":0.64000000000000001,"5min":0.628,"15min":0.57299999999999995},"max":{"1min":1.2390000000000001,"5min":2.6549999999999998,"15min":7.1520000000000001},"last":0.69299999999999995}]},{"osd":2,"last update":"Wed Feb 10 04:51:27 2021","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.095,"15min":1.3089999999999999},"min":{"1min":0.68600000000000005,"5min":0.61399999999999999,"15min":0.58299999999999996},"max":{"1min":1.405,"5min":3.0630000000000002,"15min":66.534000000000006},"last":1.133},{"interface":"front","average":{"1min":1.306,"5min":1.262,"15min":1.5609999999999999},"min":{"1min":0.64800000000000002,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":3.8820000000000001,"5min":6.7930000000000001,"15min":66.781999999999996},"last":1.7849999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:51:58 2021","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":4.5890000000000004,"5min":4.5890000000000004,"15min":4.5890000000000004},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"min":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":4.6609999999999996,"5min":4.6609999999999996,"15min":4.6609999999999996},"last":0.83899999999999997}]},{"osd":4,"last update":"Wed Feb 10 04:51:27 2021","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.4550000000000001,"15min":1.7869999999999999},"min":{"1min":0.82699999999999996,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":2.9729999999999999,"5min":7.0640000000000001,"15min":97.733999999999995},"last":1.8520000000000001},{"interface":"front","average":{"1min":1.171,"5min":1.3169999999999999,"15min":1.694},"min":{"1min":0.69699999999999995,"5min":0.69099999999999995,"15min":0.65100000000000002},"max":{"1min":2.008,"5min":4.6269999999999998,"15min":97.691000000000003},"last":0.78100000000000003}]},{"osd":5,"last update":"Wed Feb 10 04:51:27 2021","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.6160000000000001,"15min":1.6499999999999999},"min":{"1min":0.99199999999999999,"5min":0.85699999999999998,"15min":0.66700000000000004},"max":{"1min":2.21,"5min":14.972,"15min":35.283000000000001},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.798,"15min":1.9610000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":4.1470000000000002,"5min":15.218999999999999,"15min":35.353999999999999},"last":1.9610000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:51:27 2021","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":1.8680000000000001,"15min":1.8999999999999999},"min":{"1min":0.98799999999999999,"5min":0.97199999999999998,"15min":0.89100000000000001},"max":{"1min":4.0380000000000003,"5min":16.050000000000001,"15min":16.050000000000001},"last":2.044},{"interface":"front","average":{"1min":1.577,"5min":1.8080000000000001,"15min":1.802},"min":{"1min":0.876,"5min":0.876,"15min":0.876},"max":{"1min":3.9159999999999999,"5min":15.960000000000001,"15min":15.960000000000001},"last":0.98999999999999999}]}]},{"osd":6,"up_from":43,"seq":184683594054,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":341372,"kb_used_data":283680,"kb_used_omap":0,"kb_used_meta":57408,"kb_avail":93403780,"statfs":{"total":95995035648,"available":95645470720,"internally_reserved":0,"allocated":290488320,"data_stored":372767197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58785792},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:52:12 2021","interfaces":[{"interface":"back","average":{"1min":4.016,"5min":2.7400000000000002,"15min":1.595},"min":{"1min":0.52000000000000002,"5min":0.48099999999999998,"15min":0.44700000000000001},"max":{"1min":61.768000000000001,"5min":137.60599999999999,"15min":137.60599999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":4.0270000000000001,"5min":2.694,"15min":1.7989999999999999},"min":{"1min":0.504,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":62.683,"5min":137.66,"15min":137.66},"last":1.0649999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:52:12 2021","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":1.0600000000000001,"15min":1.0609999999999999},"min":{"1min":0.53800000000000003,"5min":0.499,"15min":0.499},"max":{"1min":1.335,"5min":5.2160000000000002,"15min":12.978999999999999},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.1279999999999999,"15min":1.1339999999999999},"min":{"1min":0.54700000000000004,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.611,"5min":3.6219999999999999,"15min":6.0960000000000001},"last":1.421}]},{"osd":2,"last update":"Wed Feb 10 04:52:12 2021","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.266,"15min":1.2649999999999999},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.54100000000000004},"max":{"1min":3.6629999999999998,"5min":5.0990000000000002,"15min":7.8140000000000001},"last":1.569},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.286,"15min":1.2649999999999999},"min":{"1min":0.77100000000000002,"5min":0.66200000000000003,"15min":0.59899999999999998},"max":{"1min":1.774,"5min":5.3140000000000001,"15min":5.3140000000000001},"last":1.407}]},{"osd":3,"last update":"Wed Feb 10 04:52:00 2021","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.6080000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":1.4630000000000001},{"interface":"front","average":{"1min":1.125,"5min":1.125,"15min":1.125},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.9079999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:52:12 2021","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.415,"15min":1.462},"min":{"1min":0.68300000000000005,"5min":0.64600000000000002,"15min":0.60599999999999998},"max":{"1min":2.661,"5min":7.1459999999999999,"15min":7.5389999999999997},"last":1.6839999999999999},{"interface":"front","average":{"1min":0.97599999999999998,"5min":1.0940000000000001,"15min":1.1379999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.4299999999999999,"5min":11.997999999999999,"15min":11.997999999999999},"last":1.0109999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:52:12 2021","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.244,"15min":1.6859999999999999},"min":{"1min":0.81999999999999995,"5min":0.61699999999999999,"15min":0.56899999999999995},"max":{"1min":2.0099999999999998,"5min":4.2249999999999996,"15min":81.540000000000006},"last":1.325},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.5760000000000001,"15min":1.946},"min":{"1min":0.96699999999999997,"5min":0.81000000000000005,"15min":0.78000000000000003},"max":{"1min":1.9870000000000001,"5min":5.5069999999999997,"15min":81.492999999999995},"last":1.595}]},{"osd":7,"last update":"Wed Feb 10 04:51:36 2021","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.3140000000000001,"15min":1.375},"min":{"1min":0.81399999999999995,"5min":0.747,"15min":0.747},"max":{"1min":4.0919999999999996,"5min":4.0919999999999996,"15min":5.1820000000000004},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.5720000000000001,"5min":1.629,"15min":1.6659999999999999},"min":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.77500000000000002},"max":{"1min":5.4109999999999996,"5min":5.4109999999999996,"15min":13.315},"last":1.7150000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575191,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":267524,"kb_used_data":212008,"kb_used_omap":0,"kb_used_meta":55232,"kb_avail":93477628,"statfs":{"total":95995035648,"available":95721091072,"internally_reserved":0,"allocated":217096192,"data_stored":277108917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":56557568},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:48 2021","interfaces":[{"interface":"back","average":{"1min":1.696,"5min":2.6320000000000001,"15min":1.921},"min":{"1min":0.44700000000000001,"5min":0.437,"15min":0.437},"max":{"1min":14.053000000000001,"5min":115.622,"15min":115.622},"last":0.55500000000000005},{"interface":"front","average":{"1min":1.728,"5min":2.585,"15min":1.948},"min":{"1min":0.49199999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":14.109,"5min":115.73099999999999,"15min":115.73099999999999},"last":0.64300000000000002}]},{"osd":2,"last update":"Wed Feb 10 04:52:04 2021","interfaces":[{"interface":"back","average":{"1min":2.746,"5min":2.4689999999999999,"15min":1.776},"min":{"1min":0.61799999999999999,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":38.762999999999998,"5min":115.59699999999999,"15min":115.59699999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":2.7410000000000001,"5min":2.5339999999999998,"15min":1.8109999999999999},"min":{"1min":0.64200000000000002,"5min":0.49099999999999999,"15min":0.46800000000000003},"max":{"1min":38.866999999999997,"5min":115.753,"15min":115.753},"last":0.80200000000000005}]},{"osd":3,"last update":"Wed Feb 10 04:52:00 2021","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":3.077,"5min":3.077,"15min":3.077},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.052,"5min":1.052,"15min":1.052},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":1.012}]},{"osd":4,"last update":"Wed Feb 10 04:51:48 2021","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.157,"15min":1.2270000000000001},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.55200000000000005},"max":{"1min":2.9670000000000001,"5min":4.899,"15min":6.3099999999999996},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.173,"15min":1.3779999999999999},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73199999999999998},"max":{"1min":1.6659999999999999,"5min":2.4649999999999999,"15min":29.547999999999998},"last":1.127}]},{"osd":5,"last update":"Wed Feb 10 04:51:54 2021","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.234,"15min":1.4079999999999999},"min":{"1min":0.75,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":2.7959999999999998,"5min":5.0300000000000002,"15min":48.686999999999998},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.4019999999999999,"15min":1.736},"min":{"1min":0.84599999999999997,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":2.8199999999999998,"5min":5.0389999999999997,"15min":48.905000000000001},"last":0.89600000000000002}]},{"osd":6,"last update":"Wed Feb 10 04:52:04 2021","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.264,"15min":1.6890000000000001},"min":{"1min":0.95099999999999996,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":2.0089999999999999,"5min":2.1659999999999999,"15min":48.878999999999998},"last":1.202},{"interface":"front","average":{"1min":1.282,"5min":1.3140000000000001,"15min":1.6020000000000001},"min":{"1min":0.82199999999999995,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.8160000000000001,"5min":5.1429999999999998,"15min":49.011000000000003},"last":1.4399999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:51:20 2021","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.4179999999999999,"15min":1.5629999999999999},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":3.1139999999999999,"5min":4.9770000000000003,"15min":28.573},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.3959999999999999,"15min":1.597},"min":{"1min":0.88800000000000001,"5min":0.87,"15min":0.87},"max":{"1min":1.6080000000000001,"5min":2.298,"15min":29.408999999999999},"last":1.284}]}]},{"osd":0,"up_from":8,"seq":34359738716,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":376052,"kb_used_data":315308,"kb_used_omap":0,"kb_used_meta":60736,"kb_avail":93369100,"statfs":{"total":95995035648,"available":95609958400,"internally_reserved":0,"allocated":322875392,"data_stored":447092007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62193664},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 04:51:32 2021","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.81100000000000005,"15min":1.0089999999999999},"min":{"1min":0.47099999999999997,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.0329999999999999,"5min":2.2930000000000001,"15min":28.946999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.874,"5min":0.92300000000000004,"15min":1.0149999999999999},"min":{"1min":0.68000000000000005,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.1819999999999999,"5min":2.0670000000000002,"15min":3.7050000000000001},"last":0.89800000000000002}]},{"osd":2,"last update":"Wed Feb 10 04:52:09 2021","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":2.2389999999999999,"15min":1.613},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0700000000000001,"5min":97.983999999999995,"15min":97.983999999999995},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.749,"5min":2.153,"15min":1.5169999999999999},"min":{"1min":0.51900000000000002,"5min":0.51300000000000001,"15min":0.47699999999999998},"max":{"1min":1.1319999999999999,"5min":97.844999999999999,"15min":97.844999999999999},"last":0.58499999999999996}]},{"osd":3,"last update":"Wed Feb 10 04:51:59 2021","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.87,"15min":0.87},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.254,"5min":1.254,"15min":1.254},"last":0.95999999999999996}]},{"osd":4,"last update":"Wed Feb 10 04:51:32 2021","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":1.119,"15min":1.306},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.3200000000000001,"5min":2.206,"15min":5.1680000000000001},"last":1.028},{"interface":"front","average":{"1min":0.96499999999999997,"5min":1.179,"15min":1.252},"min":{"1min":0.64200000000000002,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.607,"5min":3.359,"15min":14.672000000000001},"last":1.0580000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:51:54 2021","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.325,"15min":1.512},"min":{"1min":0.84599999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.714,"5min":3.944,"15min":25.047000000000001},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.413,"15min":1.7090000000000001},"min":{"1min":0.79400000000000004,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":2.1669999999999998,"5min":3.7850000000000001,"15min":28.827999999999999},"last":1.1830000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:51:21 2021","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.9339999999999999,"15min":1.8420000000000001},"min":{"1min":0.81200000000000006,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.663,"5min":45.012,"15min":45.012},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.9590000000000001,"15min":2.024},"min":{"1min":0.873,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.5449999999999999,"5min":45.225000000000001,"15min":45.225000000000001},"last":1.387}]},{"osd":7,"last update":"Wed Feb 10 04:51:25 2021","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.3480000000000001,"15min":1.53},"min":{"1min":0.82499999999999996,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.417,"5min":2.8700000000000001,"15min":5.6050000000000004},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.4770000000000001,"15min":1.5800000000000001},"min":{"1min":0.82399999999999995,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.55,"5min":3.4689999999999999,"15min":13.968999999999999},"last":1.268}]}]},{"osd":2,"up_from":18,"seq":77309411669,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":336828,"kb_used_data":292000,"kb_used_omap":0,"kb_used_meta":44544,"kb_avail":93408324,"statfs":{"total":95995035648,"available":95650123776,"internally_reserved":0,"allocated":299008000,"data_stored":347159040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45613056},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2],"upper_bound":131072},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:50 2021","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.99099999999999999,"15min":1.718},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":3.698,"5min":3.996,"15min":163.15199999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.91300000000000003,"15min":1.655},"min":{"1min":0.51100000000000001,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.9710000000000001,"5min":2.2349999999999999,"15min":162.959},"last":0.56899999999999995}]},{"osd":1,"last update":"Wed Feb 10 04:51:50 2021","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.002,"15min":1.0800000000000001},"min":{"1min":0.59499999999999997,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":3.8159999999999998,"5min":3.8159999999999998,"15min":5.0229999999999997},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.83499999999999996,"15min":0.94999999999999996},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.2010000000000001,"5min":1.895,"15min":5.726},"last":0.59399999999999997}]},{"osd":3,"last update":"Wed Feb 10 04:52:03 2021","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":4.2380000000000004,"5min":4.2380000000000004,"15min":4.2380000000000004},"last":1.0660000000000001},{"interface":"front","average":{"1min":1.331,"5min":1.331,"15min":1.331},"min":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":4.54,"5min":4.54,"15min":4.54},"last":1.2909999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:51:29 2021","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.264,"15min":1.369},"min":{"1min":0.83099999999999996,"5min":0.79500000000000004,"15min":0.67900000000000005},"max":{"1min":4.1310000000000002,"5min":4.1310000000000002,"15min":4.3319999999999999},"last":1.149},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.0600000000000001,"15min":1.167},"min":{"1min":0.68700000000000006,"5min":0.66800000000000004,"15min":0.65900000000000003},"max":{"1min":3.411,"5min":3.411,"15min":4.4489999999999998},"last":0.69499999999999995}]},{"osd":5,"last update":"Wed Feb 10 04:51:50 2021","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":2.081,"15min":1.7769999999999999},"min":{"1min":0.754,"5min":0.754,"15min":0.74399999999999999},"max":{"1min":4.3410000000000002,"5min":33.018999999999998,"15min":33.018999999999998},"last":1.349},{"interface":"front","average":{"1min":1.163,"5min":1.706,"15min":1.5449999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.71699999999999997},"max":{"1min":1.9910000000000001,"5min":33.043999999999997,"15min":33.043999999999997},"last":1.175}]},{"osd":6,"last update":"Wed Feb 10 04:51:13 2021","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.5509999999999999,"15min":1.7250000000000001},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.67700000000000005},"max":{"1min":1.6799999999999999,"5min":20.007000000000001,"15min":55.145000000000003},"last":1.258},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.72,"15min":1.8979999999999999},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.75800000000000001},"max":{"1min":1.627,"5min":19.417999999999999,"15min":55.101999999999997},"last":1.411}]},{"osd":7,"last update":"Wed Feb 10 04:51:22 2021","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.5449999999999999,"15min":1.702},"min":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"max":{"1min":3.3730000000000002,"5min":3.3730000000000002,"15min":5.2439999999999998},"last":1.228},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.2090000000000001,"15min":1.3819999999999999},"min":{"1min":0.85199999999999998,"5min":0.85099999999999998,"15min":0.82199999999999995},"max":{"1min":1.4550000000000001,"5min":1.95,"15min":4.9189999999999996},"last":0.97699999999999998}]}]},{"osd":3,"up_from":1111,"seq":4771708665872,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311760,"kb_used_data":288180,"kb_used_omap":386,"kb_used_meta":22909,"kb_avail":93433392,"statfs":{"total":95995035648,"available":95675793408,"internally_reserved":0,"allocated":295096320,"data_stored":397240028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":396036,"internal_metadata":23459068},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:58 2021","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":0.67300000000000004}]},{"osd":1,"last update":"Wed Feb 10 04:51:58 2021","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.83399999999999996}]},{"osd":2,"last update":"Wed Feb 10 04:51:58 2021","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":6.5460000000000003,"5min":6.5460000000000003,"15min":6.5460000000000003},"last":0.76200000000000001},{"interface":"front","average":{"1min":1.121,"5min":1.121,"15min":1.121},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":6.5049999999999999,"5min":6.5049999999999999,"15min":6.5049999999999999},"last":1.1870000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:51:58 2021","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":0.91500000000000004},{"interface":"front","average":{"1min":1.129,"5min":1.129,"15min":1.129},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.8999999999999999,"5min":1.8999999999999999,"15min":1.8999999999999999},"last":1.0449999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:51:58 2021","interfaces":[{"interface":"back","average":{"1min":2.7480000000000002,"5min":2.7480000000000002,"15min":2.7480000000000002},"min":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"max":{"1min":31.867000000000001,"5min":31.867000000000001,"15min":31.867000000000001},"last":1.367},{"interface":"front","average":{"1min":2.5430000000000001,"5min":2.5430000000000001,"15min":2.5430000000000001},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":31.733000000000001,"5min":31.733000000000001,"15min":31.733000000000001},"last":0.872}]},{"osd":6,"last update":"Wed Feb 10 04:51:58 2021","interfaces":[{"interface":"back","average":{"1min":5.7270000000000003,"5min":5.7270000000000003,"15min":5.7270000000000003},"min":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"max":{"1min":64.376999999999995,"5min":64.376999999999995,"15min":64.376999999999995},"last":1.2370000000000001},{"interface":"front","average":{"1min":5.6849999999999996,"5min":5.6849999999999996,"15min":5.6849999999999996},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":64.438999999999993,"5min":64.438999999999993,"15min":64.438999999999993},"last":1.3360000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:51:58 2021","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.506,"5min":1.506,"15min":1.506},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"max":{"1min":1.599,"5min":1.599,"15min":1.599},"last":1.077}]}]},{"osd":4,"up_from":31,"seq":133143986509,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":104644,"kb_used_data":71592,"kb_used_omap":0,"kb_used_meta":32768,"kb_avail":93640508,"statfs":{"total":95995035648,"available":95887880192,"internally_reserved":0,"allocated":73310208,"data_stored":93728581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33554432},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2],"upper_bound":1048576},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:36 2021","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.0899999999999999,"15min":1.593},"min":{"1min":0.45800000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.7799999999999998,"5min":95.721999999999994,"15min":95.721999999999994},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.67700000000000005,"5min":2.0870000000000002,"15min":1.6000000000000001},"min":{"1min":0.46000000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.002,"5min":95.344999999999999,"15min":95.344999999999999},"last":1.194}]},{"osd":1,"last update":"Wed Feb 10 04:51:36 2021","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.93500000000000005,"15min":0.95099999999999996},"min":{"1min":0.53000000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.0269999999999999,"5min":7.883,"15min":24.295000000000002},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.101,"15min":1.2210000000000001},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.60399999999999998},"max":{"1min":1.3540000000000001,"5min":4.3449999999999998,"15min":24.943000000000001},"last":1.3109999999999999}]},{"osd":2,"last update":"Wed Feb 10 04:51:36 2021","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.1910000000000001,"15min":1.1839999999999999},"min":{"1min":0.73499999999999999,"5min":0.627,"15min":0.61299999999999999},"max":{"1min":2.8929999999999998,"5min":4.2279999999999998,"15min":7.048},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.087,"5min":1.1919999999999999,"15min":1.198},"min":{"1min":0.73099999999999998,"5min":0.60299999999999998,"15min":0.51900000000000002},"max":{"1min":2.738,"5min":4.3840000000000003,"15min":4.3840000000000003},"last":1.3440000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:51:59 2021","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"min":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":2.1429999999999998,"5min":2.1429999999999998,"15min":2.1429999999999998},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.53,"5min":1.53,"15min":1.53},"min":{"1min":1.099,"5min":1.099,"15min":1.099},"max":{"1min":2.2559999999999998,"5min":2.2559999999999998,"15min":2.2559999999999998},"last":1.4450000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:51:49 2021","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.7869999999999999,"15min":1.554},"min":{"1min":0.68799999999999994,"5min":0.67600000000000005,"15min":0.60999999999999999},"max":{"1min":2.3300000000000001,"5min":35.774000000000001,"15min":35.774000000000001},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.244,"5min":1.7310000000000001,"15min":1.4850000000000001},"min":{"1min":0.73099999999999998,"5min":0.67100000000000004,"15min":0.58799999999999997},"max":{"1min":2.613,"5min":35.606000000000002,"15min":35.606000000000002},"last":1.278}]},{"osd":6,"last update":"Wed Feb 10 04:51:59 2021","interfaces":[{"interface":"back","average":{"1min":1.768,"5min":3.8919999999999999,"15min":2.492},"min":{"1min":1.1830000000000001,"5min":0.88900000000000001,"15min":0.76600000000000001},"max":{"1min":3.117,"5min":201.626,"15min":201.626},"last":2.0459999999999998},{"interface":"front","average":{"1min":1.629,"5min":3.8740000000000001,"15min":2.552},"min":{"1min":0.92800000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":3.0680000000000001,"5min":201.577,"15min":201.577},"last":2.093}]},{"osd":7,"last update":"Wed Feb 10 04:51:26 2021","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.486,"15min":1.488},"min":{"1min":0.90500000000000003,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.907,"5min":4.7439999999999998,"15min":6.9199999999999999},"last":1.6000000000000001},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.5289999999999999,"15min":1.5680000000000001},"min":{"1min":0.95199999999999996,"5min":0.72899999999999998,"15min":0.71499999999999997},"max":{"1min":2.2250000000000001,"5min":5.149,"15min":5.149},"last":1.466}]}]},{"osd":5,"up_from":37,"seq":158913790282,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":276472,"kb_used_data":233756,"kb_used_omap":0,"kb_used_meta":42432,"kb_avail":93468680,"statfs":{"total":95995035648,"available":95711928320,"internally_reserved":0,"allocated":239366144,"data_stored":310528181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43450368},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:51 2021","interfaces":[{"interface":"back","average":{"1min":5.0590000000000002,"5min":1.714,"15min":1.204},"min":{"1min":0.45700000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":58.624000000000002,"5min":58.624000000000002,"15min":58.624000000000002},"last":0.61799999999999999},{"interface":"front","average":{"1min":5.04,"5min":1.784,"15min":1.3009999999999999},"min":{"1min":0.56599999999999995,"5min":0.46600000000000003,"15min":0.46400000000000002},"max":{"1min":57.880000000000003,"5min":57.880000000000003,"15min":57.880000000000003},"last":0.71499999999999997}]},{"osd":1,"last update":"Wed Feb 10 04:51:51 2021","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":0.93000000000000005,"15min":0.93999999999999995},"min":{"1min":0.58699999999999997,"5min":0.52800000000000002,"15min":0.504},"max":{"1min":4.4000000000000004,"5min":4.4000000000000004,"15min":5.29},"last":0.747},{"interface":"front","average":{"1min":1.421,"5min":1.157,"15min":1.1639999999999999},"min":{"1min":0.627,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":5.7300000000000004,"5min":5.7300000000000004,"15min":11.875999999999999},"last":1.052}]},{"osd":2,"last update":"Wed Feb 10 04:51:51 2021","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.1930000000000001,"15min":1.3819999999999999},"min":{"1min":0.67000000000000004,"5min":0.53700000000000003,"15min":0.50600000000000001},"max":{"1min":5.5119999999999996,"5min":5.5119999999999996,"15min":44.462000000000003},"last":1.391},{"interface":"front","average":{"1min":1.411,"5min":1.1870000000000001,"15min":1.383},"min":{"1min":0.67300000000000004,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":5.4669999999999996,"5min":5.4669999999999996,"15min":44.398000000000003},"last":1.1299999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:51:59 2021","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.335,"15min":1.335},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":2.2509999999999999,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":1.5069999999999999},{"interface":"front","average":{"1min":1.181,"5min":1.181,"15min":1.181},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.9690000000000001,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.2410000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:51:51 2021","interfaces":[{"interface":"back","average":{"1min":1.6240000000000001,"5min":1.3340000000000001,"15min":1.3200000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.57099999999999995},"max":{"1min":5.944,"5min":5.944,"15min":6.1639999999999997},"last":1.216},{"interface":"front","average":{"1min":1.714,"5min":1.3999999999999999,"15min":1.4159999999999999},"min":{"1min":1.0820000000000001,"5min":0.68100000000000005,"15min":0.67500000000000004},"max":{"1min":5.6180000000000003,"5min":5.6180000000000003,"15min":5.6180000000000003},"last":1.641}]},{"osd":6,"last update":"Wed Feb 10 04:52:02 2021","interfaces":[{"interface":"back","average":{"1min":1.5669999999999999,"5min":2.3010000000000002,"15min":3.048},"min":{"1min":0.98499999999999999,"5min":0.85099999999999998,"15min":0.65800000000000003},"max":{"1min":2.331,"5min":38.014000000000003,"15min":319.51999999999998},"last":1.768},{"interface":"front","average":{"1min":1.458,"5min":2.1869999999999998,"15min":2.9380000000000002},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":2.3820000000000001,"5min":37.896999999999998,"15min":319.22300000000001},"last":1.619}]},{"osd":7,"last update":"Wed Feb 10 04:51:28 2021","interfaces":[{"interface":"back","average":{"1min":1.7270000000000001,"5min":1.5289999999999999,"15min":1.496},"min":{"1min":0.83499999999999996,"5min":0.75600000000000001,"15min":0.752},"max":{"1min":5.843,"5min":5.843,"15min":5.843},"last":1.347},{"interface":"front","average":{"1min":1.7629999999999999,"5min":1.6140000000000001,"15min":1.6200000000000001},"min":{"1min":0.99299999999999999,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":6.0369999999999999,"5min":6.0369999999999999,"15min":6.0369999999999999},"last":1.879}]}]}]}} 2021-02-10T04:52:16.960 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2021-02-10T04:52:16.961 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pool set unique_pool_0 pg_num 36 2021-02-10T04:52:17.145 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:52:16 smithi164 conmon[33677]: debug 2021-02-10T04:52:16.771+0000 7fbec19a1700 -1 osd.4 1138 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:52:17.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:16 smithi164 conmon[29288]: audit 2021-02-10T04:52:15.499099+0000 mgr.y (mgr 2021-02-10T04:52:17.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:16 smithi164 conmon[29288]: .14140) 1240 : audit [DBG] from='client.25618 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:52:17.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:16 smithi164 conmon[29288]: cluster 2021-02-10T04:52:15.752266+0000 osd.4 (osd.4) 2621 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:17.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:16 smithi164 conmon[29288]: cluster 2021-02-10T04:52:15.752278+0000 osd.4 (osd.4) 2622 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:17.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:16 smithi164 conmon[29288]: cluster 2021-02-10T04:52:16.064988+0000 osd.2 (osd.2) 1977 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:17.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:16 smithi164 conmon[29288]: cluster 2021-02-10T04:52:16.064997+0000 osd.2 (osd.2) 1978 : cluster [WRN] 2021-02-10T04:52:17.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:16 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:17.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:16 smithi164 conmon[29288]: cluster 2021-02-10T04:52:16.785304+0000 osd.2 (osd.2) 1979 2021-02-10T04:52:17.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:16 smithi164 conmon[29288]: : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:52:17.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:16 smithi164 conmon[29288]: cluster 2021-02-10T04:52:16. 2021-02-10T04:52:17.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:16 smithi164 conmon[29288]: 787076+0000 osd.2 (osd.2) 1980 : cluster [DBG] 2021-02-10T04:52:17.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:16 smithi164 conmon[29288]: purged_snaps scrub ok 2021-02-10T04:52:17.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:52:17 smithi085 conmon[49106]: debug 2021-02-10T04:52:17.080+0000 7f77a3e94700 -1 osd.2 1139 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) 2021-02-10T04:52:17.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[29319]: audit 2021-02-10T04:52:15.499099+0000 mgr.y (mgr.14140) 1240 : audit [DBG] from='client.25618 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:52:17.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[29319]: cluster 2021-02-10T04:52:15.752266+0000 osd.4 (osd.4) 2621 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:17.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[29319]: cluster 2021-02-10T04:52:15.752278+0000 osd.4 (osd.4) 2622 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:17.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[29319]: cluster 2021-02-10T04:52:16.064988+0000 osd.2 (osd.2) 1977 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:17.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[29319]: cluster 2021-02-10T04:52:16.064997+0000 osd.2 (osd.2) 1978 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:17.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[29319]: cluster 2021-02-10T04:52:16.785304+0000 osd.2 (osd.2) 1979 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:52:17.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[29319]: cluster 2021-02-10T04:52:16.787076+0000 osd.2 (osd 2021-02-10T04:52:17.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[29319]: .2) 1980 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:52:17.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[35961]: audit 2021-02-10T04:52:15. 2021-02-10T04:52:17.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[35961]: 499099+0000 mgr.y (mgr.14140) 1240 : audit [DBG] from='client.25618 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:52:17.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[35961]: cluster 2021-02-10T04:52:15.752266+0000 2021-02-10T04:52:17.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[35961]: osd.4 (osd.4) 2621 : cluster [WRN] 2021-02-10T04:52:17.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[35961]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:17.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[35961]: cluster 2021-02-10T04:52:17.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[35961]: 2021-02-10T04:52:15.752278+0000 osd.4 2021-02-10T04:52:17.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[35961]: (osd.4) 2622 : cluster [WRN] 2021-02-10T04:52:17.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:17.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[35961]: cluster 2021-02-10T04:52:17.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[35961]: 2021-02-10T04:52:16.064988+0000 2021-02-10T04:52:17.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[35961]: osd.2 (osd.2) 1977 : cluster [WRN] 2021-02-10T04:52:17.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:17.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[35961]: cluster 2021-02-10T04:52 2021-02-10T04:52:17.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[35961]: :16.064997+0000 osd.2 (osd.2 2021-02-10T04:52:17.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[35961]: ) 1978 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:17.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:52:17.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[35961]: -10T04:52:16.785304+0000 2021-02-10T04:52:17.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[35961]: osd.2 (osd.2) 1979 : cluster [DBG] 2021-02-10T04:52:17.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[35961]: purged_snaps scrub starts 2021-02-10T04:52:17.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[35961]: cluster 2021-02-10T04:52 2021-02-10T04:52:17.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[35961]: :16.787076+0000 osd.2 (osd.2 2021-02-10T04:52:17.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:16 smithi085 conmon[35961]: ) 1980 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:52:17.222 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:52:18.002 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:17 smithi085 conmon[29319]: cluster 2021-02-10T04:52:16. 2021-02-10T04:52:18.002 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:17 smithi085 conmon[29319]: 429311+0000 mgr.y (mgr.14140) 1241 : cluster [DBG] pgmap v2047: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:52:18.002 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:17 smithi085 conmon[29319]: cluster 2021-02-10T04:52:16.773269+0000 osd.4 (osd.4) 2623 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:18.002 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:17 smithi085 conmon[29319]: cluster 2021-02-10T04:52:16.773279+0000 osd.4 (osd.4) 2624 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:18.002 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:17 smithi085 conmon[29319]: cluster 2021-02-10T04:52:16.977878+0000 mon.a (mon.0) 2372 : cluster [DBG] osdmap e1139: 8 total, 8 up, 8 in 2021-02-10T04:52:18.002 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:17 smithi085 conmon[29319]: cluster 2021-02-10T04:52:17.081446+0000 osd.2 (osd.2) 1981 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:18.002 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:17 smithi085 conmon[29319]: cluster 2021-02-10T04:52:17.081452+0000 osd.2 (osd.2) 1982 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:18.003 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:17 smithi085 conmon[35961]: cluster 2021-02-10T04:52:16.429311 2021-02-10T04:52:18.003 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:17 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1241 : cluster [DBG] pgmap v2047: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:52:18.003 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:17 smithi085 conmon[35961]: cluster 2021-02-10T04:52:16.773269+0000 osd.4 (osd.4) 2623 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:18.003 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:17 smithi085 conmon[35961]: cluster 2021-02-10T04:52:16.773279+0000 osd.4 (osd.4) 2624 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:18.003 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:17 smithi085 conmon[35961]: cluster 2021-02-10T04:52:16.977878+0000 mon.a (mon.0) 2372 : cluster [DBG] osdmap e1139: 8 total, 8 up, 8 in 2021-02-10T04:52:18.003 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:17 smithi085 conmon[35961]: cluster 2021-02-10T04:52:17.081446+0000 osd.2 (osd.2) 1981 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:18.003 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:17 smithi085 conmon[35961]: cluster 2021-02-10T04:52:17.081452+0000 osd.2 (osd.2) 1982 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:18.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:52:17 smithi164 conmon[33677]: debug 2021-02-10T04:52:17.754+0000 7fbec19a1700 -1 osd.4 1139 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:52:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:17 smithi164 conmon[29288]: cluster 2021-02-10T04:52:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:17 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:52:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:17 smithi164 conmon[29288]: 52:16.429311+0000 mgr.y (mgr.14140) 1241 : cluster [DBG] pgmap v2047: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:52:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:17 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:52:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:17 smithi164 conmon[29288]: :52:16.773269+0000 osd.4 (osd.4) 2021-02-10T04:52:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:17 smithi164 conmon[29288]: 2623 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:18.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:17 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:52:18.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:17 smithi164 conmon[29288]: 10T04:52:16.773279+0000 osd.4 (osd. 2021-02-10T04:52:18.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:17 smithi164 conmon[29288]: 4) 2624 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:18.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:17 smithi164 conmon[29288]: cluster 2021-02-10T04:52:18.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:17 smithi164 conmon[29288]: 2021-02-10T04:52:16.977878+0000 2021-02-10T04:52:18.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:17 smithi164 conmon[29288]: mon.a (mon.0) 2372 : cluster [DBG] osdmap e1139: 8 total, 8 up, 8 in 2021-02-10T04:52:18.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:17 smithi164 conmon[29288]: cluster 2021-02-10T04:52:18.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:17 smithi164 conmon[29288]: 2021-02-10T04:52:18.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:17 smithi164 conmon[29288]: 2021-02-10T04:52 2021-02-10T04:52:18.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:17 smithi164 conmon[29288]: :17.081446+0000 osd.2 (osd.2) 1981 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:18.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:17 smithi164 conmon[29288]: cluster 2021-02-10T04:52:18.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:17 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:52:18.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:17 smithi164 conmon[29288]: 04:52:17.081452+0000 osd.2 (osd.2) 2021-02-10T04:52:18.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:17 smithi164 conmon[29288]: 1982 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:18.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:52:18 smithi085 conmon[49106]: debug 2021-02-10T04:52:18.070+0000 7f77a3e94700 -1 osd.2 1139 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) 2021-02-10T04:52:19.125 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:52:19 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:52:19.125 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:52:18 smithi164 conmon[33677]: debug 2021-02-10T04:52:18.777+0000 7fbec19a1700 -1 osd.4 1139 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:52:19.125 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:18 smithi164 conmon[29288]: cluster 2021-02-10T04:52:17.756029+0000 2021-02-10T04:52:19.125 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:18 smithi164 conmon[29288]: osd.4 (osd.4) 2625 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:19.125 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:18 smithi164 conmon[29288]: cluster 2021-02-10T04:52:17.756038+0000 osd.4 2021-02-10T04:52:19.126 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:18 smithi164 conmon[29288]: (osd.4) 2626 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:19.126 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:18 smithi164 conmon[29288]: cluster 2021-02-10T04:52:18.070807 2021-02-10T04:52:19.126 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:18 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 1983 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:19.126 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:18 smithi164 conmon[29288]: cluster 2021-02-10T04:52:19.126 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:18 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:52:19.126 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:18 smithi164 conmon[29288]: 04:52:18.070812+0000 osd.2 (osd.2 2021-02-10T04:52:19.126 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:18 smithi164 conmon[29288]: ) 1984 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:19.126 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:18 smithi164 conmon[29288]: cluster 2021-02-10T04:52:19.126 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:18 smithi164 conmon[29288]: 2021-02-10T04:52 2021-02-10T04:52:19.126 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:18 smithi164 conmon[29288]: :18.490672+0000 mon.a (mon.0) 2373 : 2021-02-10T04:52:19.126 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:18 smithi164 conmon[29288]: cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 467 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:52:19.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[49106]: debug 2021-02-10T04:52:19.084+0000 7f77a3e94700 -1 osd.2 1139 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) 2021-02-10T04:52:19.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:18 smithi085 conmon[29319]: cluster 2021-02-10T04:52:17.756029+0000 osd.4 (osd.4) 2625 : cluster 2021-02-10T04:52:19.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:18 smithi085 conmon[29319]: [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:19.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:18 smithi085 conmon[29319]: cluster 2021-02-10T04:52:17.756038+0000 osd.4 (osd.4) 2626 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:19.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:18 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:52:19.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:18 smithi085 conmon[29319]: :52:18.070807+0000 osd.2 (osd.2) 1983 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:19.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:18 smithi085 conmon[29319]: 2021-02-10T04:52:19.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:18 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:52:19.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:18 smithi085 conmon[29319]: -10T04:52:18.070812+0000 osd.2 (osd.2) 1984 2021-02-10T04:52:19.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:18 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:19.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:18 smithi085 conmon[29319]: cluster 2021-02-10T04:52:19.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:18 smithi085 conmon[29319]: 2021-02-10T04:52:18.490672+0000 mon.a ( 2021-02-10T04:52:19.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:18 smithi085 conmon[29319]: mon.0) 2373 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 467 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:52:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:18 smithi085 conmon[35961]: cluster 2021-02-10T04:52:17.756029 2021-02-10T04:52:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:18 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2625 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:18 smithi085 conmon[35961]: cluster 2021-02-10T04:52:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:18 smithi085 conmon[35961]: 2021-02-10T04:52:17.756038+0000 osd.4 2021-02-10T04:52:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:18 smithi085 conmon[35961]: (osd.4) 2626 : cluster 2021-02-10T04:52:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:18 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:18 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:52:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:18 smithi085 conmon[35961]: 10T04:52:18.070807+0000 osd.2 (osd.2) 1983 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:18 smithi085 conmon[35961]: cluster 2021-02-10T04:52:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:18 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:52:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:18 smithi085 conmon[35961]: 04:52:18.070812+0000 2021-02-10T04:52:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:18 smithi085 conmon[35961]: osd.2 (osd.2) 1984 : cluster 2021-02-10T04:52:19.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:18 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:18 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:52:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:18 smithi085 conmon[35961]: 10T04:52:18.490672+0000 mon.a (mon 2021-02-10T04:52:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:18 smithi085 conmon[35961]: .0) 2373 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 467 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:52:19.985 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:52:19 smithi164 conmon[33677]: debug 2021-02-10T04:52:19.733+0000 7fbec19a1700 -1 osd.4 1139 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:52:19.986 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:19 smithi164 conmon[29288]: cluster 2021-02-10T04:52:18.429970+0000 2021-02-10T04:52:19.986 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:19 smithi164 conmon[29288]: mgr.y (mgr.14140) 1242 : cluster [DBG] pgmap v2049: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:52:19.986 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:19 smithi164 conmon[29288]: cluster 2021-02-10T04:52:18.778996+0000 osd.4 (osd.4) 2627 : cluster [WRN] 2021-02-10T04:52:19.986 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:19 smithi164 conmon[29288]: slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:19.986 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:19 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:52:19.986 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:19 smithi164 conmon[29288]: -02-10T04:52:18.779006+0000 2021-02-10T04:52:19.986 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:19 smithi164 conmon[29288]: osd.4 (osd.4) 2628 : cluster 2021-02-10T04:52:19.986 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:19 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:19.986 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:19 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:52:19.986 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:19 smithi164 conmon[29288]: 02-10T04:52: 2021-02-10T04:52:19.987 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:19 smithi164 conmon[29288]: 19.084695+0000 osd.2 (osd.2) 1985 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:19.987 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:19 smithi164 conmon[29288]: 2021-02-10T04:52:19.987 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:19 smithi164 conmon[29288]: cluster 2021-02-10T04:52:19.987 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:19 smithi164 conmon[29288]: 2021-02-10T04:52:19.084704+0000 osd.2 (osd. 2021-02-10T04:52:19.987 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:19 smithi164 conmon[29288]: 2) 1986 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:19.987 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:19 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:52:19.987 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:19 smithi164 conmon[29288]: :52:19.123958+0000 mgr.y (mgr.14140) 1243 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:52:20.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:52:20 smithi085 conmon[49106]: debug 2021-02-10T04:52:20.058+0000 7f77a3e94700 -1 osd.2 1139 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) 2021-02-10T04:52:20.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[29319]: cluster 2021-02-10T04:52:18.429970+0000 mgr.y (mgr.14140) 1242 : cluster 2021-02-10T04:52:20.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[29319]: [DBG] pgmap v2049: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:52:20.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[29319]: cluster 2021-02-10T04:52:18.778996+0000 osd.4 (osd.4) 2627 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:20.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[29319]: cluster 2021-02-10T04:52:18.779006+0000 osd.4 (osd.4) 2628 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:20.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[29319]: 2021-02-10T04:52:20.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[29319]: cluster 2021-02-10T04:52:19.084695+0000 2021-02-10T04:52:20.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[29319]: osd.2 (osd.2) 1985 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:20.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[29319]: cluster 2021-02-10T04:52:20.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[29319]: 2021-02-10T04:52:19.084704+0000 osd.2 (osd.2) 1986 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:20.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:52:20.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[29319]: :52:19.123958+0000 mgr.y (mgr.14140) 1243 : 2021-02-10T04:52:20.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[29319]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:52:20.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[35961]: cluster 2021-02-10T04:52:18.429970+0000 mgr.y ( 2021-02-10T04:52:20.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[35961]: mgr.14140) 1242 : cluster [DBG] pgmap v2049: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:52:20.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:52:20.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[35961]: -10T04:52:18.778996+0000 osd.4 (osd.4) 2627 2021-02-10T04:52:20.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:20.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:52:20.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[35961]: -10T04:52:18.779006+0000 2021-02-10T04:52:20.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[35961]: osd.4 (osd.4) 2628 : cluster [WRN] 2021-02-10T04:52:20.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[35961]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:20.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[35961]: cluster 2021-02-10T04:52: 2021-02-10T04:52:20.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[35961]: 19.084695+0000 osd.2 (osd.2) 1985 2021-02-10T04:52:20.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[35961]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:20.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:52:20.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[35961]: T04:52:19.084704+0000 osd.2 ( 2021-02-10T04:52:20.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[35961]: osd.2) 1986 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:20.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:52:20.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[35961]: 02-10T04:52:19.123958+0000 2021-02-10T04:52:20.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[35961]: mgr.y (mgr.14140) 1243 : audit [DBG] 2021-02-10T04:52:20.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:19 smithi085 conmon[35961]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:52:20.755 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:52:20 smithi164 conmon[33677]: debug 2021-02-10T04:52:20.751+0000 7fbec19a1700 -1 osd.4 1139 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:52:20.872 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:52:20.872 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [ 2021-02-10T04:52:20.873 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.873 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007)", 2021-02-10T04:52:20.873 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:24.520446+0000", 2021-02-10T04:52:20.873 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 476.35126626599998, 2021-02-10T04:52:20.873 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 476.35129603399997, 2021-02-10T04:52:20.873 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:52:20.873 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:52:20.873 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:52:20.873 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:52:20.873 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:52:20.873 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 910 2021-02-10T04:52:20.873 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.874 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:52:20.874 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.874 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:52:20.874 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520446+0000", 2021-02-10T04:52:20.874 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:52:20.874 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.874 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.874 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:52:20.874 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520446+0000", 2021-02-10T04:52:20.874 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.3349000000000001e-05 2021-02-10T04:52:20.874 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.874 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.874 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:52:20.875 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520490+0000", 2021-02-10T04:52:20.875 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6949999999999997e-06 2021-02-10T04:52:20.875 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.875 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.875 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:52:20.875 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520495+0000", 2021-02-10T04:52:20.875 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3142000000000001e-05 2021-02-10T04:52:20.875 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.875 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.875 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:52:20.875 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520508+0000", 2021-02-10T04:52:20.875 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9640000000000002e-06 2021-02-10T04:52:20.875 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.876 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.876 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:52:20.876 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520513+0000", 2021-02-10T04:52:20.876 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.038756196999999999 2021-02-10T04:52:20.876 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.876 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.876 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:20.876 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.559270+0000", 2021-02-10T04:52:20.876 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6949999999999997e-06 2021-02-10T04:52:20.876 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.876 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.876 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for peered", 2021-02-10T04:52:20.876 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.559275+0000", 2021-02-10T04:52:20.877 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.98370379500000005 2021-02-10T04:52:20.877 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.877 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.877 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:20.877 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:25.542979+0000", 2021-02-10T04:52:20.877 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000126095 2021-02-10T04:52:20.877 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.877 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.877 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:52:20.877 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:25.543105+0000", 2021-02-10T04:52:20.877 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 337.420295028 2021-02-10T04:52:20.877 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.877 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.878 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:20.878 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:02.963400+0000", 2021-02-10T04:52:20.878 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1430999999999999e-05 2021-02-10T04:52:20.878 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.878 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.878 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for peered", 2021-02-10T04:52:20.878 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:02.963412+0000", 2021-02-10T04:52:20.878 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0132677919999999 2021-02-10T04:52:20.878 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.878 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.878 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:20.878 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:03.976679+0000", 2021-02-10T04:52:20.879 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000111842 2021-02-10T04:52:20.879 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.879 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.879 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:52:20.879 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:03.976791+0000", 2021-02-10T04:52:20.879 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 54.524946901 2021-02-10T04:52:20.879 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.879 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.879 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:20.879 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:58.501738+0000", 2021-02-10T04:52:20.879 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1005000000000001e-05 2021-02-10T04:52:20.879 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.879 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.880 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for peered", 2021-02-10T04:52:20.880 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:58.501749+0000", 2021-02-10T04:52:20.880 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0263936469999999 2021-02-10T04:52:20.880 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.880 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.880 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:20.880 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:59.528143+0000", 2021-02-10T04:52:20.880 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00014606100000000001 2021-02-10T04:52:20.880 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.880 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.880 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:52:20.880 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:59.528289+0000", 2021-02-10T04:52:20.880 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 395.00784263899999 2021-02-10T04:52:20.880 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:20.881 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:52:20.881 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:20.881 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.881 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.881 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007)", 2021-02-10T04:52:20.881 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:24.520546+0000", 2021-02-10T04:52:20.881 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 476.35116656700001, 2021-02-10T04:52:20.881 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 476.35133133199997, 2021-02-10T04:52:20.881 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:52:20.881 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "delayed", 2021-02-10T04:52:20.881 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:52:20.881 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:52:20.881 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:52:20.882 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 911 2021-02-10T04:52:20.882 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.882 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:52:20.882 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.882 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:52:20.882 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520546+0000", 2021-02-10T04:52:20.882 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:52:20.882 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.882 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.882 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:52:20.882 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520546+0000", 2021-02-10T04:52:20.882 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.0148e-05 2021-02-10T04:52:20.883 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.883 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.883 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:52:20.883 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520576+0000", 2021-02-10T04:52:20.883 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.4709999999999998e-06 2021-02-10T04:52:20.883 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.883 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.883 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:52:20.883 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520582+0000", 2021-02-10T04:52:20.883 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0281e-05 2021-02-10T04:52:20.883 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.883 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.883 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:52:20.884 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520592+0000", 2021-02-10T04:52:20.884 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5079999999999999e-06 2021-02-10T04:52:20.884 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.884 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.884 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:52:20.884 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520596+0000", 2021-02-10T04:52:20.884 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.038699253000000003 2021-02-10T04:52:20.884 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.884 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.884 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:20.884 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.559296+0000", 2021-02-10T04:52:20.884 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6760000000000004e-06 2021-02-10T04:52:20.884 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.885 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.885 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for peered", 2021-02-10T04:52:20.885 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.559301+0000", 2021-02-10T04:52:20.885 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.98411211399999998 2021-02-10T04:52:20.885 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.885 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.885 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:20.885 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:25.543413+0000", 2021-02-10T04:52:20.885 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.6785e-05 2021-02-10T04:52:20.885 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.885 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.885 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:52:20.885 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:25.543460+0000", 2021-02-10T04:52:20.885 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 337.41998265900003 2021-02-10T04:52:20.886 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.886 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.886 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:20.886 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:02.963443+0000", 2021-02-10T04:52:20.886 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.8140000000000003e-06 2021-02-10T04:52:20.886 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.886 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.886 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for peered", 2021-02-10T04:52:20.886 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:02.963450+0000", 2021-02-10T04:52:20.886 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0138036189999999 2021-02-10T04:52:20.886 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.886 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.886 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:20.887 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:03.977253+0000", 2021-02-10T04:52:20.887 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.6387999999999997e-05 2021-02-10T04:52:20.887 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.887 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.887 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:52:20.887 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:03.977300+0000", 2021-02-10T04:52:20.887 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 54.524484158 2021-02-10T04:52:20.887 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.887 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.887 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:20.887 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:58.501784+0000", 2021-02-10T04:52:20.887 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.0169999999999996e-06 2021-02-10T04:52:20.887 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.888 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.888 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for peered", 2021-02-10T04:52:20.888 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:58.501792+0000", 2021-02-10T04:52:20.888 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.026636855 2021-02-10T04:52:20.888 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.888 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.888 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:20.888 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:59.528429+0000", 2021-02-10T04:52:20.888 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.7280999999999998e-05 2021-02-10T04:52:20.888 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:20.888 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:20.888 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:52:20.888 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:59.528466+0000", 2021-02-10T04:52:20.889 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 395.00792002700001 2021-02-10T04:52:20.889 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:20.889 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:52:20.889 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:20.889 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:20.889 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2021-02-10T04:52:20.889 INFO:teuthology.orchestra.run.smithi164.stdout: "complaint_time": 30, 2021-02-10T04:52:20.889 INFO:teuthology.orchestra.run.smithi164.stdout: "num_blocked_ops": 2 2021-02-10T04:52:20.889 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:52:21.116 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[49106]: debug 2021-02-10T04:52:21.067+0000 7f77a3e94700 -1 osd.2 1139 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) 2021-02-10T04:52:21.117 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:20 smithi085 conmon[35961]: cluster 2021-02-10T04:52:19.735013+0000 2021-02-10T04:52:21.117 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:20 smithi085 conmon[35961]: osd.4 (osd.4) 2629 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:21.117 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:20 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:52:21.117 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:20 smithi085 conmon[35961]: 04:52:19 2021-02-10T04:52:21.117 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:20 smithi085 conmon[35961]: .735027+0000 osd.4 (osd.4) 2630 : 2021-02-10T04:52:21.117 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:20 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:21.117 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:20 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:52:21.117 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:20 smithi085 conmon[35961]: 02-10T04:52:20 2021-02-10T04:52:21.117 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:20 smithi085 conmon[35961]: .058846+0000 osd.2 (osd.2) 2021-02-10T04:52:21.118 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:20 smithi085 conmon[35961]: 1987 : cluster 2021-02-10T04:52:21.118 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:20 smithi085 conmon[35961]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:21.118 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:20 smithi085 conmon[35961]: cluster 2021-02-10T04:52:20.058855+0000 2021-02-10T04:52:21.118 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:20 smithi085 conmon[35961]: osd.2 (osd.2) 1988 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:21.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:20 smithi085 conmon[29319]: cluster 2021-02-10T04:52:19.735013+0000 osd.4 ( 2021-02-10T04:52:21.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:20 smithi085 conmon[29319]: osd.4) 2629 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:21.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:20 smithi085 conmon[29319]: cluster 2021-02-10T04:52:19.735027+0000 osd.4 (osd.4) 2630 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:21.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:20 smithi085 conmon[29319]: cluster 2021-02-10T04:52:20.058846+0000 osd.2 (osd.2) 1987 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:21.119 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:20 smithi085 conmon[29319]: cluster 2021-02-10T04:52:20.058855+0000 osd.2 (osd.2) 1988 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:21.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:20 smithi164 conmon[29288]: cluster 2021-02-10T04:52:19.735013+0000 osd.4 (osd.4) 2021-02-10T04:52:21.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:20 smithi164 conmon[29288]: 2629 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:21.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:20 smithi164 conmon[29288]: cluster 2021-02-10T04:52:19.735027+0000 osd.4 (osd.4) 2630 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:21.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:20 smithi164 conmon[29288]: 2021-02-10T04:52:21.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:20 smithi164 conmon[29288]: cluster 2021-02-10T04:52:20 2021-02-10T04:52:21.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:20 smithi164 conmon[29288]: .058846+0000 osd.2 (osd.2) 1987 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:21.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:20 smithi164 conmon[29288]: cluster 2021-02-10T04:52: 2021-02-10T04:52:21.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:20 smithi164 conmon[29288]: 20.058855+0000 osd.2 (osd.2) 1988 : cluster [WRN] 2021-02-10T04:52:21.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:20 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:21.894 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 04:52:21 smithi164 conmon[33677]: debug 2021-02-10T04:52:21.753+0000 7fbec19a1700 -1 osd.4 1139 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) 2021-02-10T04:52:22.041 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:52:22.065 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:52:22.101 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:52:22.132 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:52:22.157 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:52:22.180 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:52:22.215 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:52:22.238 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:52:22.241 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:52:22.241 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:52:22.241 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:52:22.241 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:52:22.241 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:52:22.241 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:52:22.241 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:52:22.242 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:52:22.264 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[49106]: debug 2021-02-10T04:52:22.265 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[49106]: 2021-02-10T04:52:22.039+0000 7f77a3e94700 -1 osd.2 1139 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) 2021-02-10T04:52:22.265 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[49106]: 2021-02-10T04:52:22.265 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[29319]: cluster 2021-02-10T04:52:22.265 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:52:22.265 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[29319]: 52:20.430563+0000 mgr.y (mgr.14140) 1244 : cluster [DBG] pgmap v2050: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:52:22.266 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:52:22.266 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[29319]: 02-10T04:52:20.753469+0000 osd.4 (osd. 2021-02-10T04:52:22.266 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[29319]: 4) 2631 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:22.266 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:52:22.266 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[29319]: -02-10T04:52:20.753478+0000 2021-02-10T04:52:22.266 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[29319]: osd.4 (osd.4) 2632 : cluster [WRN] 2021-02-10T04:52:22.266 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[29319]: slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:22.266 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[29319]: cluster 2021-02-10T04:52 2021-02-10T04:52:22.266 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[29319]: :21.068445+0000 osd.2 (osd.2) 2021-02-10T04:52:22.266 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[29319]: 1989 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:22.266 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:52:22.266 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[29319]: 10T04:52:21.068451+0000 osd.2 2021-02-10T04:52:22.266 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[29319]: (osd.2) 1990 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:22.267 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[29319]: 2021-02-10T04:52:22.267 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[35961]: cluster 2021-02-10T04:52:22.267 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[35961]: 2021-02-10T04:52:22.267 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[35961]: 2021 2021-02-10T04:52:22.267 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[35961]: - 2021-02-10T04:52:22.267 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[35961]: 02 2021-02-10T04:52:22.267 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[35961]: - 2021-02-10T04:52:22.267 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[35961]: 10 2021-02-10T04:52:22.268 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[35961]: T04:52:20.430563+0000 2021-02-10T04:52:22.268 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[35961]: mgr.y (mgr.14140) 1244 : cluster [DBG] pgmap v2050: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:52:22.268 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[35961]: cluster 2021-02-10T04:52:22.268 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:52:22.268 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[35961]: T04:52:20.753469+0000 osd.4 ( 2021-02-10T04:52:22.268 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[35961]: osd.4) 2631 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:22.268 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:52:22.268 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[35961]: 02-10T04:52:20.753478+0000 2021-02-10T04:52:22.268 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[35961]: osd.4 (osd.4) 2632 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:22.268 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:52:22.268 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[35961]: -02-10T04:52: 2021-02-10T04:52:22.268 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[35961]: 21.068445+0000 osd.2 (osd 2021-02-10T04:52:22.268 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[35961]: .2) 1989 : cluster [WRN] 2021-02-10T04:52:22.269 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:22.269 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:52:22.269 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[35961]: 02-10T04:52:21.068451+0000 2021-02-10T04:52:22.269 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[35961]: osd.2 (osd.2) 1990 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:22.269 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:21 smithi085 conmon[35961]: 2021-02-10T04:52:22.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:21 smithi164 conmon[29288]: cluster 2021-02-10T04:52:20.430563+0000 mgr.y (mgr.14140) 1244 : cluster [DBG] pgmap v2050: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:52:22.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:21 smithi164 conmon[29288]: cluster 2021-02-10T04:52:20.753469+0000 osd.4 (osd.4) 2631 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:22.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:21 smithi164 conmon[29288]: cluster 2021-02-10T04:52 2021-02-10T04:52:22.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:21 smithi164 conmon[29288]: :20.753478+0000 osd.4 (osd.4) 2632 : cluster 2021-02-10T04:52:22.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:21 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:22.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:21 smithi164 conmon[29288]: cluster 2021-02-10T04:52:22.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:21 smithi164 conmon[29288]: 2021-02-10T04:52:21.068445+0000 osd.2 (osd.2) 1989 2021-02-10T04:52:22.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:21 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:22.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:21 smithi164 conmon[29288]: cluster 2021-02-10T04:52:21 2021-02-10T04:52:22.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:21 smithi164 conmon[29288]: .068451+0000 osd.2 (osd.2) 1990 : cluster 2021-02-10T04:52:22.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:21 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:23.011 INFO:teuthology.orchestra.run.smithi085.stderr:set pool 7 pg_num to 36 2021-02-10T04:52:23.011 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is unset 2021-02-10T04:52:23.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: cluster 2021-02-10T04:52:23.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: 2021-02-10T04:52:23.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: 2021 2021-02-10T04:52:23.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: - 2021-02-10T04:52:23.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: 02 2021-02-10T04:52:23.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: -10T04:52:21.753799 2021-02-10T04:52:23.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2633 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:23.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:52:23.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: -02-10T04:52:21. 2021-02-10T04:52:23.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: 753812+0000 osd.4 (osd.4) 2021-02-10T04:52:23.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: 2634 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:23.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:52:23.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: 02-10T04:52:22.014095+0000 mon.a (mon.0) 2374 : audit [DBG] from='client.? 172.21.15.85:0/3508549920' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:52:23.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: cluster 2021-02-10T04:52 2021-02-10T04:52:23.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: :22.040478+0000 osd.2 (osd.2) 1991 : cluster 2021-02-10T04:52:23.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:23.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:52:23.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: 10T04:52:22.040486+0000 osd.2 (osd.2) 1992 : cluster 2021-02-10T04:52:23.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:23.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: audit 2021-02-10T04:52: 2021-02-10T04:52:23.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: 22.243640+0000 mon.c (mon.1) 98 : audit 2021-02-10T04:52:23.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: [INF] from='client.? 172.21.15.85:0/3663205625' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]: dispatch 2021-02-10T04:52:23.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: audit 2021-02-10T04:52:23.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: 2021-02-10T04:52 2021-02-10T04:52:23.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: :22.244092+0000 mon.a (mon. 2021-02-10T04:52:23.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: 0) 2375 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]: dispatch 2021-02-10T04:52:23.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:52:23.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: 10T04:52:22.296145+0000 mon.c (mon 2021-02-10T04:52:23.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: .1) 99 : audit [INF] from='client.? 172.21.15.85:0/1905112108' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:52:23.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:52:23.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: 10T04:52:22.296525+0000 mon.a (mon.0 2021-02-10T04:52:23.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:22 smithi164 conmon[29288]: ) 2376 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:52:23.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[29319]: cluster 2021-02-10T04:52:21.753799+0000 osd.4 (osd.4) 2633 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:23.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[29319]: cluster 2021-02-10T04:52:21.753812+0000 osd.4 (osd.4) 2634 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:23.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[29319]: audit 2021-02-10T04:52:22.014095+0000 mon.a (mon.0) 2374 : audit [DBG] from='client.? 172.21.15.85:0/3508549920' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:52:23.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[29319]: cluster 2021-02-10T04:52:22.040478+0000 osd.2 (osd.2) 1991 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:23.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[29319]: cluster 2021-02-10T04:52:22.040486+0000 osd.2 (osd.2) 1992 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:23.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[29319]: audit 2021-02-10T04:52:22.243640+0000 mon.c (mon.1) 98 : audit [INF] from='client.? 172.21.15.85:0/3663205625' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]: dispatch 2021-02-10T04:52:23.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[29319]: audit 2021-02-10T04:52:22.244092+0000 mon.a (mon.0) 2375 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]: dispatch 2021-02-10T04:52:23.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[29319]: audit 2021-02-10T04:52:22.296145+0000 mon.c (mon.1) 99 : audit [INF] from='client.? 172.21.15.85:0/1905112108' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:52:23.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[29319]: audit 2021-02-10T04:52:22.296525+0000 mon.a (mon.0) 2376 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:52:23.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:52:23.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[35961]: 10T04:52:21.753799+0000 osd.4 (osd.4) 2633 : cluster [WRN] slow request osd_op(client.14817.0:910 7.1 7:8132b738:::smithi08575397-33:head [copy-from ver 110 in=55b] snapc 3b=[3b,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520446+0000 currently started 2021-02-10T04:52:23.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[35961]: cluster 2021-02-10T04:52:23.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:52:23.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[35961]: 52:21.753812+0000 osd.4 (osd.4) 2021-02-10T04:52:23.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[35961]: 2634 : cluster [WRN] slow request osd_op(client.14817.0:911 7.1 7:8132b738:::smithi08575397-33:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007) initiated 2021-02-10T04:44:24.520546+0000 currently delayed 2021-02-10T04:52:23.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:52:23.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[35961]: -10T04:52:22.014095+0000 mon.a (mon. 2021-02-10T04:52:23.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[35961]: 0) 2374 : audit [DBG] from='client.? 172.21.15.85:0/3508549920' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:52:23.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[35961]: cluster 2021-02-10T04:52:23.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[35961]: 2021-02-10T04:52: 2021-02-10T04:52:23.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[35961]: 22.040478+0000 osd.2 (osd.2) 1991 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:23.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[35961]: cluster 2021-02-10T04:52:22 2021-02-10T04:52:23.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[35961]: .040486+0000 osd.2 (osd.2) 1992 : cluster [WRN] 2021-02-10T04:52:23.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[35961]: slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:23.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[35961]: audit 2021-02-10T04:52:22.243640 2021-02-10T04:52:23.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[35961]: +0000 mon.c (mon.1) 98 : audit [INF] from='client.? 172.21.15.85:0/3663205625' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]: dispatch 2021-02-10T04:52:23.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[35961]: 2021-02-10T04:52:23.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[35961]: audit 2021-02-10T04:52:23.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[35961]: 2021-02-10T04:52:22.244092+0000 mon.a (mon 2021-02-10T04:52:23.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[35961]: .0) 2375 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]: dispatch 2021-02-10T04:52:23.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:52:23.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[35961]: 02-10T04:52:22.296145+0000 mon.c ( 2021-02-10T04:52:23.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[35961]: mon.1) 99 : audit [INF] from='client.? 172.21.15.85:0/1905112108' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:52:23.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:52:23.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[35961]: 02-10T04:52:22.296525+0000 mon.a ( 2021-02-10T04:52:23.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:22 smithi085 conmon[35961]: mon.0) 2376 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:52:24.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[29319]: cluster 2021-02-10T04:52:22.431181+0000 mgr.y (mgr.14140) 1245 : cluster [DBG] pgmap v2051: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:52:24.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[29319]: cluster 2021-02-10T04:52:22.994400+0000 mon.a (mon.0) 2377 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:52:24.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[29319]: audit 2021-02-10T04:52:23.005145+0000 mon.a (mon 2021-02-10T04:52:24.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[29319]: .0) 2378 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]': finished 2021-02-10T04:52:24.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:52:24.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[29319]: :52:23.005433+0000 mon.a (mon.0) 2379 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:52:24.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[29319]: cluster 2021-02-10T04:52:23.005486+0000 mon.a (mon.0) 2021-02-10T04:52:24.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[29319]: 2380 : cluster [DBG] osdmap e1140: 8 total, 8 up, 8 in 2021-02-10T04:52:24.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[29319]: cluster 2021-02-10T04:52:24.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[29319]: 2021-02-10T04:52:23.492256+0000 mon.a (mon.0) 2381 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 477 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:52:24.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[29319]: 2021-02-10T04:52:24.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:52:24.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[29319]: :52:23.566017+0000 osd.0 (osd.0) 173 2021-02-10T04:52:24.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[29319]: : cluster [DBG] 6.16 scrub ok 2021-02-10T04:52:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[35961]: cluster 2021-02-10T04:52:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[35961]: 2021-02-10T04:52:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[35961]: 2021-02-10T04:52:22. 2021-02-10T04:52:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[35961]: 431181+0000 mgr.y (mgr.14140) 1245 : cluster [DBG] pgmap v2051: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:52:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:52:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[35961]: 10T04:52:22.994400+0000 2021-02-10T04:52:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[35961]: mon.a (mon.0) 2377 : cluster [WRN] 2021-02-10T04:52:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[35961]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:52:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:52:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[35961]: -10T04:52:23.005145+0000 2021-02-10T04:52:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[35961]: mon.a (mon.0) 2378 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]': finished 2021-02-10T04:52:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[35961]: 2021-02-10T04:52:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[35961]: audit 2021-02-10T04:52:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[35961]: 2021-02-10T04:52: 2021-02-10T04:52:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[35961]: 23.005433+0000 mon.a (mon.0) 2379 2021-02-10T04:52:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[35961]: : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:52:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:52:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[35961]: 02-10T04:52:23 2021-02-10T04:52:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[35961]: .005486+0000 mon.a (mon.0) 2380 : cluster 2021-02-10T04:52:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[35961]: [DBG] osdmap e1140: 8 total, 8 up, 8 in 2021-02-10T04:52:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:52:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[35961]: 10T04:52:23.492256+0000 2021-02-10T04:52:24.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[35961]: mon.a (mon.0) 2381 : cluster [WRN] 2021-02-10T04:52:24.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[35961]: Health check update: 4 slow ops, oldest one blocked for 477 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:52:24.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:52:24.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[35961]: T04:52:23.566017+0000 2021-02-10T04:52:24.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[35961]: osd.0 (osd.0) 173 : cluster [DBG] 2021-02-10T04:52:24.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:23 smithi085 conmon[35961]: 6.16 scrub ok 2021-02-10T04:52:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:23 smithi164 conmon[29288]: cluster 2021-02-10T04:52:22.431181+0000 mgr.y 2021-02-10T04:52:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:23 smithi164 conmon[29288]: (mgr.14140) 1245 : cluster [DBG] pgmap v2051: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-02-10T04:52:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:23 smithi164 conmon[29288]: cluster 2021-02-10T04:52:22.994400+0000 mon.a (mon.0) 2377 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:52:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:23 smithi164 conmon[29288]: audit 2021-02-10T04:52:23.005145+0000 mon.a 2021-02-10T04:52:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:23 smithi164 conmon[29288]: (mon.0) 2378 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]': finished 2021-02-10T04:52:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:23 smithi164 conmon[29288]: audit 2021-02-10T04:52:23.005433+0000 mon.a (mon 2021-02-10T04:52:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:23 smithi164 conmon[29288]: .0) 2379 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:52:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:23 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:52:24.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:23 smithi164 conmon[29288]: 04:52:23.005486+0000 mon.a ( 2021-02-10T04:52:24.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:23 smithi164 conmon[29288]: mon.0) 2380 : cluster [DBG] osdmap e1140: 8 total, 8 up, 8 in 2021-02-10T04:52:24.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:23 smithi164 conmon[29288]: cluster 2021-02-10T04:52:24.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:23 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:52:24.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:23 smithi164 conmon[29288]: 52:23.492256+0000 mon.a (mon.0) 2021-02-10T04:52:24.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:23 smithi164 conmon[29288]: 2381 : cluster [WRN] Health check update: 4 slow ops, oldest one blocked for 477 sec, daemons [osd.2,osd.4] have slow ops. (SLOW_OPS) 2021-02-10T04:52:24.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:23 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:52:24.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:23 smithi164 conmon[29288]: 02-10T04:52:23.566017+0000 osd.0 2021-02-10T04:52:24.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:23 smithi164 conmon[29288]: (osd.0) 173 : cluster [DBG] 6.16 scrub ok 2021-02-10T04:52:25.283 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:25 smithi085 conmon[29319]: cluster 2021-02-10T04:52:23.095582+0000 osd.6 (osd.6 2021-02-10T04:52:25.283 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:25 smithi085 conmon[29319]: ) 164 : cluster [DBG] 2.14 scrub ok 2021-02-10T04:52:25.283 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:25 smithi085 conmon[29319]: cluster 2021-02-10T04:52:24.121680+0000 osd.2 (osd.2) 1993 : cluster [DBG] 6.29 scrub ok 2021-02-10T04:52:25.283 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:25 smithi085 conmon[29319]: audit 2021-02-10T04:52:24.432685+0000 mon.a ( 2021-02-10T04:52:25.284 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:25 smithi085 conmon[29319]: mon.0) 2382 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]: dispatch 2021-02-10T04:52:25.284 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:25 smithi085 conmon[35961]: cluster 2021-02-10T04:52:23.095582+0000 osd.6 2021-02-10T04:52:25.284 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:25 smithi085 conmon[35961]: (osd.6) 164 : cluster [DBG] 2.14 scrub ok 2021-02-10T04:52:25.284 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:25 smithi085 conmon[35961]: cluster 2021-02-10T04:52:24.121680+0000 osd.2 (osd 2021-02-10T04:52:25.284 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:25 smithi085 conmon[35961]: .2) 1993 : cluster [DBG] 6.29 scrub ok 2021-02-10T04:52:25.284 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:25 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:52:25.284 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:25 smithi085 conmon[35961]: :52:24.432685+0000 mon.a (mon.0 2021-02-10T04:52:25.284 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:25 smithi085 conmon[35961]: ) 2382 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]: dispatch 2021-02-10T04:52:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:25 smithi164 conmon[29288]: cluster 2021-02-10T04:52:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:25 smithi164 conmon[29288]: 2021-02-10T04:52:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:25 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:52:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:25 smithi164 conmon[29288]: :52:23.095582+0000 osd.6 (osd.6) 164 : cluster [DBG] 2.14 scrub ok 2021-02-10T04:52:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:25 smithi164 conmon[29288]: cluster 2021-02-10T04:52:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:25 smithi164 conmon[29288]: 2021-02 2021-02-10T04:52:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:25 smithi164 conmon[29288]: -10T04:52:24.121680+0000 osd.2 (osd.2) 1993 2021-02-10T04:52:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:25 smithi164 conmon[29288]: : cluster [DBG] 6.29 scrub ok 2021-02-10T04:52:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:25 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:52:25.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:25 smithi164 conmon[29288]: 02-10T04:52:24.432685+0000 mon.a 2021-02-10T04:52:25.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:25 smithi164 conmon[29288]: (mon.0) 2382 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]: dispatch 2021-02-10T04:52:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: cluster 2021-02-10T04:52:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: 2021-02-10T04:52:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: 2021 2021-02-10T04:52:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: - 2021-02-10T04:52:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: 02 2021-02-10T04:52:26.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: - 2021-02-10T04:52:26.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: 10 2021-02-10T04:52:26.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: T 2021-02-10T04:52:26.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: 04 2021-02-10T04:52:26.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: : 2021-02-10T04:52:26.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: 52 2021-02-10T04:52:26.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: : 2021-02-10T04:52:26.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: 24 2021-02-10T04:52:26.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: . 2021-02-10T04:52:26.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: 071909 2021-02-10T04:52:26.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: +0000 2021-02-10T04:52:26.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: 2021-02-10T04:52:26.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: osd.6 2021-02-10T04:52:26.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: ( 2021-02-10T04:52:26.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: osd 2021-02-10T04:52:26.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: . 2021-02-10T04:52:26.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: 6 2021-02-10T04:52:26.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: ) 2021-02-10T04:52:26.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: 165 2021-02-10T04:52:26.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: : 2021-02-10T04:52:26.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: cluster 2021-02-10T04:52:26.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: 2021-02-10T04:52:26.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: [DBG] 2021-02-10T04:52:26.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: 2.29 scrub ok 2021-02-10T04:52:26.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:52:26.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: 10T04:52:24.431836+0000 mgr.y (mgr.14140) 1246 2021-02-10T04:52:26.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: : cluster [DBG] pgmap v2053: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:52:26.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: audit 2021-02-10T04:52:26.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: 2021-02-10 2021-02-10T04:52:26.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: T04:52:25.001014+0000 mon.a (mon 2021-02-10T04:52:26.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: .0) 2383 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]': finished 2021-02-10T04:52:26.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: cluster 2021-02-10T04:52:26.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: 2021-02-10T04:52:25.001299+0000 2021-02-10T04:52:26.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: mon.a (mon.0) 2384 : cluster [DBG] 2021-02-10T04:52:26.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: osdmap e1141: 8 total, 8 up, 8 in 2021-02-10T04:52:26.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:52:26.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: :52:25.030274+0000 osd.3 (osd.3 2021-02-10T04:52:26.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: ) 15 : cluster [DBG] 6.1f scrub ok 2021-02-10T04:52:26.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:52:26.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: -10T04:52:25.358024+0000 osd.1 2021-02-10T04:52:26.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:26 smithi164 conmon[29288]: (osd.1) 486 : cluster [DBG] 3.0 scrub ok 2021-02-10T04:52:26.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:26 smithi085 conmon[29319]: cluster 2021-02-10T04:52:24.071909+0000 osd.6 ( 2021-02-10T04:52:26.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:26 smithi085 conmon[29319]: osd.6) 165 : cluster [DBG] 2.29 scrub ok 2021-02-10T04:52:26.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:26 smithi085 conmon[29319]: cluster 2021-02-10T04:52:24.431836+0000 mgr.y (mgr.14140) 1246 : cluster [DBG] pgmap v2053: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:52:26.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:26 smithi085 conmon[29319]: audit 2021-02-10T04:52:25.001014+0000 mon.a (mon.0) 2383 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]': finished 2021-02-10T04:52:26.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:26 smithi085 conmon[29319]: cluster 2021-02-10T04:52:25.001299+0000 mon.a (mon.0) 2384 : cluster [DBG] osdmap e1141: 8 total, 8 up, 8 in 2021-02-10T04:52:26.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:26 smithi085 conmon[29319]: cluster 2021-02-10T04:52:25.030274+0000 osd.3 (osd.3) 15 : cluster [DBG] 6.1f scrub ok 2021-02-10T04:52:26.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:26 smithi085 conmon[29319]: cluster 2021-02-10T04:52:25.358024+0000 osd.1 (osd.1) 486 : cluster [DBG] 3.0 scrub ok 2021-02-10T04:52:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:26 smithi085 conmon[35961]: cluster 2021-02-10T04:52:24.071909+0000 2021-02-10T04:52:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:26 smithi085 conmon[35961]: osd.6 (osd.6) 165 : cluster [DBG] 2.29 scrub ok 2021-02-10T04:52:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:26 smithi085 conmon[35961]: cluster 2021-02-10T04:52 2021-02-10T04:52:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:26 smithi085 conmon[35961]: :24.431836+0000 mgr.y (mgr.14140) 1246 : cluster [DBG] pgmap v2053: 188 pgs: 188 active+clean; 896 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:52:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:26 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:52:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:26 smithi085 conmon[35961]: 02-10T04:52:25.001014+0000 2021-02-10T04:52:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:26 smithi085 conmon[35961]: mon.a (mon.0) 2383 : audit [INF] 2021-02-10T04:52:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:26 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]': finished 2021-02-10T04:52:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:26 smithi085 conmon[35961]: cluster 2021-02-10T04:52:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:26 smithi085 conmon[35961]: 2021-02-10T04:52:25.001299+0000 mon.a ( 2021-02-10T04:52:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:26 smithi085 conmon[35961]: mon.0) 2384 : cluster [DBG] osdmap e1141: 8 total, 8 up, 8 in 2021-02-10T04:52:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:26 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:52:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:26 smithi085 conmon[35961]: -02-10T04:52:25.030274+0000 osd.3 2021-02-10T04:52:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:26 smithi085 conmon[35961]: (osd.3) 15 : cluster [DBG] 6.1f scrub ok 2021-02-10T04:52:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:26 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:52:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:26 smithi085 conmon[35961]: :52:25.358024+0000 osd.1 (osd.1) 486 : cluster 2021-02-10T04:52:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:26 smithi085 conmon[35961]: [DBG] 3.0 scrub ok 2021-02-10T04:52:27.255 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset nodeep-scrub 2021-02-10T04:52:27.293 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:52:27.298 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[29319]: cluster 2021-02-10T04:52:25.745570+0000 osd.4 2021-02-10T04:52:27.298 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[29319]: (osd.4) 2635 : cluster [DBG] 6.e scrub ok 2021-02-10T04:52:27.298 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[29319]: cluster 2021-02-10T04:52:26.015661+0000 mon.a (mon. 2021-02-10T04:52:27.298 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[29319]: 0) 2385 : cluster [DBG] osdmap e1142: 8 total, 8 up, 8 in 2021-02-10T04:52:27.298 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:52:27.298 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[29319]: T04:52:26.311951+0000 osd.1 (osd. 2021-02-10T04:52:27.298 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[29319]: 1) 487 : cluster [DBG] 5.c scrub ok 2021-02-10T04:52:27.298 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:52:27.298 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[29319]: 02-10T04:52:26.433334+0000 2021-02-10T04:52:27.299 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[29319]: mon.a (mon.0) 2386 2021-02-10T04:52:27.299 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[29319]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]: dispatch 2021-02-10T04:52:27.299 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:52:27.299 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[29319]: 02-10T04:52:26.597663 2021-02-10T04:52:27.299 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[29319]: +0000 osd.0 (osd.0) 174 : cluster [DBG] 2021-02-10T04:52:27.299 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[29319]: 7.18 scrub ok 2021-02-10T04:52:27.299 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[35961]: cluster 2021-02-10T04:52:25.745570 2021-02-10T04:52:27.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2635 : cluster [DBG] 6.e scrub ok 2021-02-10T04:52:27.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:52:27.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[35961]: -02-10T04:52:26.015661+0000 mon.a (mon.0) 2385 : cluster [DBG] 2021-02-10T04:52:27.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[35961]: osdmap e1142: 8 total, 8 up, 8 in 2021-02-10T04:52:27.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[35961]: cluster 2021-02-10T04:52: 2021-02-10T04:52:27.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[35961]: 26.311951+0000 osd.1 (osd. 2021-02-10T04:52:27.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[35961]: 1) 487 : cluster [DBG] 5.c scrub ok 2021-02-10T04:52:27.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[35961]: audit 2021 2021-02-10T04:52:27.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[35961]: -02-10T04:52 2021-02-10T04:52:27.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[35961]: :26. 2021-02-10T04:52:27.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[35961]: 433334+0000 mon.a (mon.0) 2386 : audit 2021-02-10T04:52:27.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]: dispatch 2021-02-10T04:52:27.300 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:52:27.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[35961]: -02-10T04:52:26.597663+0000 2021-02-10T04:52:27.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[35961]: osd.0 (osd.0) 174 : cluster [DBG] 2021-02-10T04:52:27.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:27 smithi085 conmon[35961]: 7.18 scrub ok 2021-02-10T04:52:27.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:27 smithi164 conmon[29288]: cluster 2021-02-10T04:52:25.745570+0000 2021-02-10T04:52:27.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:27 smithi164 conmon[29288]: osd.4 (osd.4) 2635 : cluster [DBG] 6.e scrub ok 2021-02-10T04:52:27.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:27 smithi164 conmon[29288]: cluster 2021-02-10T04:52:27.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:27 smithi164 conmon[29288]: 2021-02-10T04:52:26.015661+0000 mon.a (mon.0) 2385 : cluster [DBG] osdmap e1142: 8 total, 8 up, 8 in 2021-02-10T04:52:27.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:27 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:52:27.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:27 smithi164 conmon[29288]: 02-10T04:52:26.311951+0000 osd.1 (osd.1) 487 2021-02-10T04:52:27.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:27 smithi164 conmon[29288]: : cluster [DBG] 5.c scrub ok 2021-02-10T04:52:27.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:27 smithi164 conmon[29288]: audit 2021-02-10T04:52:26.433334 2021-02-10T04:52:27.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:27 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2386 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]: dispatch 2021-02-10T04:52:27.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:27 smithi164 conmon[29288]: cluster 2021-02-10T04:52:27.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:27 smithi164 conmon[29288]: 2021-02-10T04:52:26. 2021-02-10T04:52:27.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:27 smithi164 conmon[29288]: 597663+0000 osd.0 (osd.0) 174 : cluster [DBG] 7.18 scrub ok 2021-02-10T04:52:27.537 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:52:27.562 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:52:28.081 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[29319]: cluster 2021-02-10T04:52:28.081 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[29319]: 2021-02-10T04:52:28.081 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[29319]: 2021 2021-02-10T04:52:28.081 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[29319]: - 2021-02-10T04:52:28.081 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[29319]: 02 2021-02-10T04:52:28.081 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[29319]: - 2021-02-10T04:52:28.081 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[29319]: 10 2021-02-10T04:52:28.081 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[29319]: T04:52:26. 2021-02-10T04:52:28.081 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[29319]: 408264+0000 osd.5 (osd.5) 170 : cluster [DBG] 2.1c scrub ok 2021-02-10T04:52:28.081 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[29319]: cluster 2021-02-10T04:52:26.432336 2021-02-10T04:52:28.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 1247 : cluster [DBG] pgmap v2056: 198 pgs: 1 peering, 9 unknown, 188 active+clean; 859 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:52:28.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[29319]: audit 2021-02-10T 2021-02-10T04:52:28.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[29319]: 04:52:27.035408+0000 mon.a (mon.0) 2387 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]': finished 2021-02-10T04:52:28.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:52:28.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[29319]: 02-10T04:52:27.035488+0000 mon.a ( 2021-02-10T04:52:28.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[29319]: mon.0) 2388 : cluster [DBG] osdmap e1143: 8 total, 8 up, 8 in 2021-02-10T04:52:28.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:52:28.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[29319]: -02-10T04:52:27.126592+0000 osd.2 (osd.2 2021-02-10T04:52:28.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[29319]: ) 1994 : cluster [DBG] 5.d scrub ok 2021-02-10T04:52:28.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:52:28.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[29319]: 02-10T04:52:27.602001+0000 osd.0 ( 2021-02-10T04:52:28.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[29319]: osd.0) 175 : cluster [DBG] 7.9 scrub ok 2021-02-10T04:52:28.083 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[35961]: cluster 2021-02-10T04:52:28.083 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[35961]: 2021-02-10T04:52:28.083 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[35961]: 2021 2021-02-10T04:52:28.083 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[35961]: -02-10T04:52:26.408264+0000 osd.5 (osd 2021-02-10T04:52:28.083 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[35961]: .5) 170 : cluster [DBG] 2.1c scrub ok 2021-02-10T04:52:28.083 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[35961]: cluster 2021-02-10T04:52:26.432336+0000 mgr.y (mgr.14140) 1247 2021-02-10T04:52:28.083 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[35961]: : cluster [DBG] pgmap v2056: 198 pgs: 1 peering, 9 unknown, 188 active+clean; 859 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:52:28.083 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[35961]: audit 2021-02-10T04:52:27.035408 2021-02-10T04:52:28.083 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2387 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]': finished 2021-02-10T04:52:28.083 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[35961]: 2021-02-10T04:52:28.083 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:52:28.083 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[35961]: 52:27.035488+0000 mon.a (mon.0) 2388 : cluster [DBG] osdmap e1143: 8 total, 8 up, 8 in 2021-02-10T04:52:28.083 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:52:28.084 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[35961]: 02-10T04:52:27.126592+0000 osd.2 (osd.2) 2021-02-10T04:52:28.084 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[35961]: 1994 : cluster [DBG] 5.d scrub ok 2021-02-10T04:52:28.084 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:52:28.084 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[35961]: 02-10T04:52:27.602001+0000 osd.0 2021-02-10T04:52:28.084 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:28 smithi085 conmon[35961]: (osd.0) 175 : cluster [DBG] 7.9 scrub ok 2021-02-10T04:52:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:28 smithi164 conmon[29288]: cluster 2021-02-10T04:52:26.408264+0000 osd.5 ( 2021-02-10T04:52:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:28 smithi164 conmon[29288]: osd.5) 170 : cluster [DBG] 2.1c scrub ok 2021-02-10T04:52:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:28 smithi164 conmon[29288]: cluster 2021-02-10T04:52:26.432336+0000 mgr.y (mgr.14140) 1247 : cluster [DBG] pgmap v2056: 198 pgs: 1 peering, 9 unknown, 188 active+clean; 859 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:52:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:28 smithi164 conmon[29288]: audit 2021-02-10T04:52:27.035408+0000 mon.a ( 2021-02-10T04:52:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:28 smithi164 conmon[29288]: mon.0) 2387 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]': finished 2021-02-10T04:52:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:28 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:52:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:28 smithi164 conmon[29288]: 04:52:27.035488+0000 mon.a (mon.0) 2021-02-10T04:52:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:28 smithi164 conmon[29288]: 2388 : cluster [DBG] osdmap e1143: 8 total, 8 up, 8 in 2021-02-10T04:52:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:28 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:52:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:28 smithi164 conmon[29288]: :52:27.126592+0000 osd.2 (osd.2) 1994 : cluster 2021-02-10T04:52:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:28 smithi164 conmon[29288]: [DBG] 5.d scrub ok 2021-02-10T04:52:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:28 smithi164 conmon[29288]: cluster 2021-02-10T04:52:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:28 smithi164 conmon[29288]: 2021-02-10T04:52:27.602001+0000 osd.0 ( 2021-02-10T04:52:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:28 smithi164 conmon[29288]: osd.0) 175 : cluster [DBG] 7.9 scrub ok 2021-02-10T04:52:28.541 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.4 dump_historic_ops 2021-02-10T04:52:28.971 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:52:29.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[29319]: cluster 2021-02-10T04:52:27.451604+0000 2021-02-10T04:52:29.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[29319]: osd.6 (osd.6) 166 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:52:29.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[29319]: cluster 2021-02-10T04:52:27.455391+0000 osd.6 (osd.6) 167 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:52:29.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[29319]: cluster 2021-02-10T04:52:28.037396+0000 mon.a (mon.0) 2389 : cluster [DBG] osdmap e1144: 8 total, 8 up, 8 in 2021-02-10T04:52:29.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[29319]: cluster 2021-02-10T04:52:28.092284+0000 osd.2 (osd.2) 1995 : cluster [DBG] 6.21 scrub ok 2021-02-10T04:52:29.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[29319]: cluster 2021-02-10T04:52:28.493416+0000 mon.a (mon.0) 2390 : cluster [WRN] Health check update: 2 slow ops, oldest one blocked for 477 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:52:29.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[29319]: cluster 2021-02-10T04:52:29.036760+0000 mon.a (mon.0) 2391 : cluster [DBG] osdmap e1145: 8 total, 8 up, 8 in 2021-02-10T04:52:29.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: cluster 2021-02-10T04:52:29.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: 2021-02-10T04:52:29.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: 2021 2021-02-10T04:52:29.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: - 2021-02-10T04:52:29.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: 02 2021-02-10T04:52:29.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: - 2021-02-10T04:52:29.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: 10 2021-02-10T04:52:29.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: T 2021-02-10T04:52:29.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: 04 2021-02-10T04:52:29.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: : 2021-02-10T04:52:29.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: 52 2021-02-10T04:52:29.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: : 2021-02-10T04:52:29.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: 27 2021-02-10T04:52:29.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: . 2021-02-10T04:52:29.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: 451604 2021-02-10T04:52:29.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: +0000 2021-02-10T04:52:29.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: 2021-02-10T04:52:29.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: osd.6 2021-02-10T04:52:29.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: ( 2021-02-10T04:52:29.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: osd 2021-02-10T04:52:29.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: . 2021-02-10T04:52:29.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: 6 2021-02-10T04:52:29.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: ) 2021-02-10T04:52:29.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: 166 2021-02-10T04:52:29.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: : 2021-02-10T04:52:29.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: cluster 2021-02-10T04:52:29.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: 2021-02-10T04:52:29.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: [DBG] purged_snaps scrub starts 2021-02-10T04:52:29.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: cluster 2021-02-10T04:52:27.455391+0000 2021-02-10T04:52:29.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: osd.6 (osd.6) 167 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:52:29.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:52:29.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: 02-10T04:52:28.037396+0000 mon.a 2021-02-10T04:52:29.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: (mon.0) 2389 : cluster 2021-02-10T04:52:29.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: [DBG] osdmap e1144: 8 total, 8 up, 8 in 2021-02-10T04:52:29.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: cluster 2021-02-10T04:52:29.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: 2021 2021-02-10T04:52:29.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: -02-10T04 2021-02-10T04:52:29.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: :52:28.092284+0000 osd.2 (osd.2 2021-02-10T04:52:29.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: ) 1995 : cluster [DBG] 6.21 scrub ok 2021-02-10T04:52:29.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: 2021-02-10T04:52:29.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: cluster 2021-02-10T04:52:29.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:52:29.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: 52:28.493416+0000 mon.a ( 2021-02-10T04:52:29.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: mon.0) 2390 : cluster 2021-02-10T04:52:29.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: [WRN] Health check update: 2 slow ops, oldest one blocked for 477 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:52:29.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:52:29.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: 02-10T04:52:29.036760+0000 2021-02-10T04:52:29.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: mon.a (mon.0) 2391 2021-02-10T04:52:29.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:29 smithi085 conmon[35961]: : cluster [DBG] osdmap e1145: 8 total, 8 up, 8 in 2021-02-10T04:52:29.394 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:52:29 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:52:29.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:29 smithi164 conmon[29288]: cluster 2021-02-10T04:52:27.451604+0000 osd.6 (osd. 2021-02-10T04:52:29.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:29 smithi164 conmon[29288]: 6) 166 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:52:29.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:29 smithi164 conmon[29288]: cluster 2021-02-10T04:52:27.455391+0000 osd.6 (osd.6) 167 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:52:29.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:29 smithi164 conmon[29288]: cluster 2021-02-10T04:52:28.037396+0000 mon.a (mon.0) 2389 : cluster [DBG] osdmap e1144: 8 total, 8 up, 8 in 2021-02-10T04:52:29.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:29 smithi164 conmon[29288]: cluster 2021-02-10T04:52:28.092284+0000 osd.2 (osd.2) 1995 : cluster [DBG] 6.21 scrub ok 2021-02-10T04:52:29.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:29 smithi164 conmon[29288]: cluster 2021-02-10T04:52:28.493416+0000 mon.a (mon.0) 2390 : cluster [WRN] Health check update: 2 slow ops, oldest one blocked for 477 sec, osd.4 has slow ops (SLOW_OPS) 2021-02-10T04:52:29.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:29 smithi164 conmon[29288]: cluster 2021-02-10T04:52:29.036760+0000 mon.a (mon.0) 2391 : cluster [DBG] osdmap e1145: 8 total, 8 up, 8 in 2021-02-10T04:52:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:30 smithi164 conmon[29288]: cluster 2021-02-10T04:52:28. 2021-02-10T04:52:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:30 smithi164 conmon[29288]: 432896+0000 mgr.y (mgr.14140) 1248 : cluster [DBG] pgmap v2059: 198 pgs: 1 activating, 3 remapped+peering, 3 peering, 3 unknown, 188 active+clean; 811 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:52:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:30 smithi164 conmon[29288]: cluster 2021-02-10T04:52:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:30 smithi164 conmon[29288]: 2021-02-10T04:52:29.041849+0000 mon.a (mon.0) 2392 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2021-02-10T04:52:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:30 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:52:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:30 smithi164 conmon[29288]: 02-10T04:52:29.041882+0000 mon.a (mon 2021-02-10T04:52:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:30 smithi164 conmon[29288]: .0) 2393 : cluster [INF] Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 477 sec, osd.4 has slow ops) 2021-02-10T04:52:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:30 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:52:30.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:30 smithi164 conmon[29288]: 02-10T04:52:29.134034+0000 mgr.y 2021-02-10T04:52:30.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:30 smithi164 conmon[29288]: (mgr.14140) 1249 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:52:30.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[29319]: cluster 2021-02-10T04:52:28.432896+0000 mgr.y (mgr.14140) 2021-02-10T04:52:30.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[29319]: 1248 : cluster [DBG] pgmap v2059: 198 pgs: 1 activating, 3 remapped+peering, 3 peering, 3 unknown, 188 active+clean; 811 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:52:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[29319]: cluster 2021-02-10T04:52:29.041849+0000 mon.a (mon.0) 2392 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2021-02-10T04:52:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[29319]: cluster 2021-02-10T04:52:29.041882+0000 mon.a (mon.0) 2393 : cluster [INF] Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 477 sec, osd.4 has slow ops) 2021-02-10T04:52:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:52:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[29319]: 02-10T04:52:29.134034+0000 2021-02-10T04:52:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[29319]: mgr.y (mgr.14140) 1249 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:52:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[29319]: 2021-02-10T04:52:30.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[35961]: cluster 2021-02-10T04:52:30.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[35961]: 2021-02-10T04:52:30.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[35961]: 2021 2021-02-10T04:52:30.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[35961]: - 2021-02-10T04:52:30.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[35961]: 02-10T04:52:28. 2021-02-10T04:52:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[35961]: 432896+0000 mgr.y (mgr.14140) 1248 : cluster [DBG] pgmap v2059: 198 pgs: 1 activating, 3 remapped+peering, 3 peering, 3 unknown, 188 active+clean; 811 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:52:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[35961]: cluster 2021-02-10T04:52:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[35961]: 2021-02-10T04:52:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[35961]: 2021 2021-02-10T04:52:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[35961]: - 2021-02-10T04:52:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[35961]: 02-10T04:52:29 2021-02-10T04:52:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[35961]: .041849+0000 mon.a (mon.0) 2392 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2021-02-10T04:52:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:52:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[35961]: -10T04:52:29.041882+0000 2021-02-10T04:52:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[35961]: mon.a (mon.0) 2393 : cluster 2021-02-10T04:52:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[35961]: [INF] Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 477 sec, osd.4 has slow ops) 2021-02-10T04:52:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[35961]: 2021-02-10T04:52:30.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[35961]: audit 2021-02-10T04:52:30.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[35961]: 2021-02-10T04:52:29 2021-02-10T04:52:30.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[35961]: .134034+0000 mgr.y (mgr.14140) 1249 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:52:30.554 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-02-10T04:52:30.554 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:52:30.554 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2021-02-10T04:52:30.554 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2021-02-10T04:52:30.554 DEBUG:teuthology.orchestra.run.smithi085:> sudo systemctl stop ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.0 2021-02-10T04:52:30.942 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:52:30 smithi085 systemd[1]: Stopping Ceph osd.0 for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T04:52:30.942 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[41540]: debug 2021-02-10T04:52:30.669+0000 7fb96845e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:52:30.942 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[41540]: debug 2021-02-10T04:52:30.669+0000 7fb96845e700 -1 osd.0 1146 *** Got signal Terminated *** 2021-02-10T04:52:30.942 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:52:30 smithi085 conmon[41540]: debug 2021-02-10T04:52:30.669+0000 7fb96845e700 -1 osd.0 1146 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-02-10T04:52:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: cluster 2021-02-10T04:52:30.019092+0000 osd.3 2021-02-10T04:52:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: (osd.3) 16 : cluster [DBG] 6.b scrub ok 2021-02-10T04:52:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: cluster 2021-02-10T04:52:30.056773+0000 mon.a (mon.0) 2394 : cluster [DBG] osdmap e1146: 8 total, 8 up, 8 in 2021-02-10T04:52:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: cluster 2021-02-10T04:52:30.133040+0000 osd.2 (osd.2) 1996 : cluster [DBG] 6.28 scrub ok 2021-02-10T04:52:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: cluster 2021-02-10T04:52:30.276934+0000 osd.1 (osd.1) 488 : cluster [DBG] 4.4 scrub ok 2021-02-10T04:52:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: cluster 2021-02-10T04:52:30.695533+0000 2021-02-10T04:52:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: mon.a (mon.0) 2395 : cluster [DBG] osd.0 reported immediately failed by osd.1 2021-02-10T04:52:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: cluster 2021-02-10T04:52:30.695562+0000 mon.a ( 2021-02-10T04:52:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: mon.0) 2396 : cluster [INF] osd.0 failed (root=default,host=smithi085) (connection refused reported by osd.1) 2021-02-10T04:52:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: cluster 2021-02-10T04:52: 2021-02-10T04:52:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: 30.695745+0000 mon.a (mon.0) 2397 : cluster 2021-02-10T04:52:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: [DBG] osd.0 reported immediately failed by osd.1 2021-02-10T04:52:31.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: cluster 2021-02-10T04:52:30 2021-02-10T04:52:31.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: .695926+0000 mon.a (mon.0) 2398 : cluster 2021-02-10T04:52:31.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:52:31.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: cluster 2021-02-10T04:52 2021-02-10T04:52:31.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: :30.696069+0000 mon.a (mon.0) 2021-02-10T04:52:31.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: 2399 : cluster [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:52:31.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:52:31.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: 10T04:52:30.696205+0000 mon.a (mon. 2021-02-10T04:52:31.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: 0) 2400 : cluster [DBG] osd.0 reported immediately failed by osd.5 2021-02-10T04:52:31.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:52:31.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: -10T04:52:30.696325+0000 mon.a ( 2021-02-10T04:52:31.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: mon.0) 2401 : cluster [DBG] osd.0 reported immediately failed by osd.5 2021-02-10T04:52:31.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:52:31.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: 04:52:30.718024+0000 mon.a (mon. 2021-02-10T04:52:31.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: 0) 2402 : cluster [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:52:31.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:52:31.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: T04:52:30.896132+0000 mon.a (mon. 2021-02-10T04:52:31.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: 0) 2403 : cluster [DBG] osd.0 reported immediately failed by osd.2 2021-02-10T04:52:31.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: cluster 2021-02-10T04:52:31.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: 2021-02-10T04:52:30.896277 2021-02-10T04:52:31.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2404 : cluster [DBG] 2021-02-10T04:52:31.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: osd.0 reported immediately failed by osd.2 2021-02-10T04:52:31.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: cluster 2021-02-10T04:52:30.896412 2021-02-10T04:52:31.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2405 : cluster [DBG] osd.0 reported immediately failed by osd.1 2021-02-10T04:52:31.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:52:31.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: -02-10T04:52:30.896537+0000 2021-02-10T04:52:31.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: mon.a (mon.0) 2406 : cluster [DBG] osd.0 reported immediately failed by osd.3 2021-02-10T04:52:31.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:52:31.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: -02-10T04:52:30.896669+0000 2021-02-10T04:52:31.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: mon.a (mon.0) 2407 : cluster [DBG] 2021-02-10T04:52:31.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: osd.0 reported immediately failed by osd.1 2021-02-10T04:52:31.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:52:31.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: :52:30.896791+0000 mon.a (mon.0) 2021-02-10T04:52:31.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: 2408 : cluster [DBG] osd.0 reported immediately failed by osd.5 2021-02-10T04:52:31.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:52:31.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: 04:52:30.896891+0000 mon.a (mon. 2021-02-10T04:52:31.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: 0) 2409 : cluster [DBG] osd.0 reported immediately failed by osd.3 2021-02-10T04:52:31.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:52:31.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: 04:52:30.897020+0000 mon.a (mon.0) 2021-02-10T04:52:31.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: 2410 : cluster [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:52:31.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:52:31.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: 04:52:30.897118+0000 mon.a (mon.0) 2021-02-10T04:52:31.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: 2411 : cluster [DBG] osd.0 reported immediately failed by osd.7 2021-02-10T04:52:31.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:52:31.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: 10T04:52:30.897210+0000 mon.a (mon.0) 2412 : cluster [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:52:31.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: 2021-02-10T04:52:31.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: cluster 2021-02-10T04:52:31.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: 2021-02-10T04:52:30.897312+0000 2021-02-10T04:52:31.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: mon.a (mon.0) 2413 : cluster [DBG] osd.0 reported immediately failed by osd.5 2021-02-10T04:52:31.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: 2021-02-10T04:52:31.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:52:31.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: :52:30.897404+0000 mon.a (mon.0 2021-02-10T04:52:31.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: ) 2414 : cluster [DBG] osd.0 reported immediately failed by osd.6 2021-02-10T04:52:31.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:52:31.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: -02-10T04:52:30.897499 2021-02-10T04:52:31.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2415 : cluster 2021-02-10T04:52:31.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: [DBG] osd.0 reported immediately failed by osd.7 2021-02-10T04:52:31.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:52:31.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: -10T04:52:30.903419+0000 2021-02-10T04:52:31.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: mon.a (mon.0) 2416 : cluster [DBG] 2021-02-10T04:52:31.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: osd.0 reported immediately failed by osd.6 2021-02-10T04:52:31.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:52:31.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: T04:52:30.918357+0000 mon.a (mon 2021-02-10T04:52:31.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:31 smithi164 conmon[29288]: .0) 2417 : cluster [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:52:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: cluster 2021-02-10T04:52:30.019092+0000 osd.3 (osd. 2021-02-10T04:52:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: 3) 16 : cluster [DBG] 6.b scrub ok 2021-02-10T04:52:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: cluster 2021-02-10T04:52:30.056773+0000 mon.a (mon.0) 2394 : cluster [DBG] osdmap e1146: 8 total, 8 up, 8 in 2021-02-10T04:52:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: cluster 2021-02-10T04:52:30.133040+0000 osd.2 (osd.2) 1996 : cluster [DBG] 6.28 scrub ok 2021-02-10T04:52:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: cluster 2021-02-10T04:52:30.276934+0000 2021-02-10T04:52:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: osd.1 (osd.1) 488 : cluster [DBG] 4.4 scrub ok 2021-02-10T04:52:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:52:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: 10T04:52:30.695533+0000 mon.a (mon.0) 2395 : cluster [DBG] osd.0 reported immediately failed by osd.1 2021-02-10T04:52:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:52:31.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: :52:30.695562+0000 mon.a (mon.0) 2396 : cluster 2021-02-10T04:52:31.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: [INF] osd.0 failed (root=default,host=smithi085) (connection refused reported by osd.1) 2021-02-10T04:52:31.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:52:31.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: 02-10T04:52:30.695745+0000 mon.a (mon.0) 2397 : cluster [DBG] osd.0 reported immediately failed by osd.1 2021-02-10T04:52:31.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: 2021-02-10T04:52:31.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:52:31.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: 02-10T04:52:30.695926+0000 mon.a (mon.0 2021-02-10T04:52:31.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: ) 2398 : cluster [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:52:31.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:52:31.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: :52:30.696069+0000 mon.a (mon.0) 2399 : cluster 2021-02-10T04:52:31.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:52:31.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:52:31.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: 10T04:52:30.696205+0000 mon.a ( 2021-02-10T04:52:31.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: mon.0) 2400 : cluster [DBG] osd.0 reported immediately failed by osd.5 2021-02-10T04:52:31.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:52:31.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: 02-10T04:52:30.696325+0000 2021-02-10T04:52:31.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: mon.a (mon.0) 2401 : cluster [DBG] 2021-02-10T04:52:31.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: osd.0 reported immediately failed by osd.5 2021-02-10T04:52:31.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:52:31.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: -02-10T04:52:30.718024+0000 mon.a (mon. 2021-02-10T04:52:31.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: 0) 2402 : cluster [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:52:31.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:52:31.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: 02-10T04:52:30.896132+0000 2021-02-10T04:52:31.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: mon.a (mon.0) 2403 : cluster [DBG] 2021-02-10T04:52:31.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: osd.0 reported immediately failed by osd.2 2021-02-10T04:52:31.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: cluster 2021-02-10T04:52:31.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: 2021-02-10T04:52:30.896277+0000 mon.a ( 2021-02-10T04:52:31.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: mon.0) 2404 : cluster [DBG] osd.0 reported immediately failed by osd.2 2021-02-10T04:52:31.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:52:31.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: -10T04:52:30.896412+0000 2021-02-10T04:52:31.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: mon.a (mon.0) 2405 : cluster [DBG] osd.0 reported immediately failed by osd.1 2021-02-10T04:52:31.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: 2021-02-10T04:52:31.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:52:31.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: 04:52:30.896537+0000 mon.a (mon 2021-02-10T04:52:31.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: .0) 2406 : cluster [DBG] osd.0 reported immediately failed by osd.3 2021-02-10T04:52:31.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:52:31.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: -10T04:52:30.896669+0000 2021-02-10T04:52:31.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: mon.a (mon.0) 2407 : cluster [DBG] 2021-02-10T04:52:31.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: osd.0 reported immediately failed by osd.1 2021-02-10T04:52:31.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: cluster 2021-02-10T04:52:31.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: 2021-02-10T04:52:30.896791+0000 mon.a 2021-02-10T04:52:31.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: (mon.0) 2408 : cluster [DBG] osd.0 reported immediately failed by osd.5 2021-02-10T04:52:31.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:52:31.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: -10T04:52:30.896891+0000 2021-02-10T04:52:31.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: mon.a (mon.0) 2409 : cluster [DBG] osd.0 reported immediately failed by osd.3 2021-02-10T04:52:31.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: 2021-02-10T04:52:31.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:52:31.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: -10T04:52:30.897020+0000 mon.a (mon. 2021-02-10T04:52:31.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: 0) 2410 : cluster [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:52:31.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:52:31.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: T04:52:30.897118+0000 mon.a ( 2021-02-10T04:52:31.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: mon.0) 2411 : cluster [DBG] osd.0 reported immediately failed by osd.7 2021-02-10T04:52:31.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:52:31.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: 10T04:52:30.897210+0000 mon.a 2021-02-10T04:52:31.447 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: (mon.0) 2412 : cluster 2021-02-10T04:52:31.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:52:31.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: cluster 2021-02-10T04:52:30 2021-02-10T04:52:31.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: .897312+0000 mon.a (mon.0) 2413 : cluster [DBG] osd.0 reported immediately failed by osd.5 2021-02-10T04:52:31.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: 2021-02-10T04:52:31.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: cluster 2021-02-10T04:52: 2021-02-10T04:52:31.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: 30.897404+0000 mon.a (mon.0) 2414 : cluster [DBG] osd.0 reported immediately failed by osd.6 2021-02-10T04:52:31.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: cluster 2021-02-10T04:52:31.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: 2021-02-10T04:52:30.897499+0000 mon.a (mon.0) 2415 2021-02-10T04:52:31.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: : cluster [DBG] osd.0 reported immediately failed by osd.7 2021-02-10T04:52:31.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:52:31.448 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: :52:30.903419+0000 mon.a (mon.0 2021-02-10T04:52:31.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: ) 2416 : cluster [DBG] osd.0 reported immediately failed by osd.6 2021-02-10T04:52:31.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:52:31.449 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[29319]: 10T04:52:30.918357+0000 mon.a (mon.0) 2417 : cluster [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:52:31.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: cluster 2021-02-10T04:52:31.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 2021-02-10T04:52:31.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 2021 2021-02-10T04:52:31.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: - 2021-02-10T04:52:31.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 02-10T04:52:30.019092+0000 osd.3 2021-02-10T04:52:31.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: (osd.3) 16 : cluster [DBG] 6.b scrub ok 2021-02-10T04:52:31.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: cluster 2021-02-10T04:52: 2021-02-10T04:52:31.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 30.056773+0000 mon.a (mon.0) 2394 : cluster 2021-02-10T04:52:31.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: [DBG] osdmap e1146: 8 total, 8 up, 8 in 2021-02-10T04:52:31.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: cluster 2021-02-10T04:52:31.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 2021-02-10T04:52:30.133040+0000 osd.2 (osd. 2021-02-10T04:52:31.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 2) 1996 : cluster [DBG] 6.28 scrub ok 2021-02-10T04:52:31.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:52:31.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: -10T04:52:30.276934+0000 osd.1 ( 2021-02-10T04:52:31.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: osd.1) 488 : cluster [DBG] 4.4 scrub ok 2021-02-10T04:52:31.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:52:31.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: -10T04:52:30.695533+0000 mon.a 2021-02-10T04:52:31.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: (mon.0) 2395 : cluster [DBG] osd.0 reported immediately failed by osd.1 2021-02-10T04:52:31.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 2021-02-10T04:52:31.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: cluster 2021-02-10T04:52:31.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 2021-02-10T04:52:30.695562+0000 mon.a (mon. 2021-02-10T04:52:31.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 0) 2396 : cluster [INF] osd.0 failed (root=default,host=smithi085) (connection refused reported by osd.1) 2021-02-10T04:52:31.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:52:31.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: :52:30.695745+0000 mon.a (mon. 2021-02-10T04:52:31.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 0) 2397 : cluster [DBG] osd.0 reported immediately failed by osd.1 2021-02-10T04:52:31.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:52:31.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 04:52:30.695926+0000 mon.a (mon. 2021-02-10T04:52:31.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 0) 2398 : cluster [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:52:31.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:52:31.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 10T04:52:30.696069+0000 mon.a 2021-02-10T04:52:31.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: (mon.0) 2399 : cluster [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:52:31.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:52:31.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 10T04:52:30.696205+0000 mon.a ( 2021-02-10T04:52:31.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: mon.0) 2400 : cluster [DBG] osd.0 reported immediately failed by osd.5 2021-02-10T04:52:31.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:52:31.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: -10T04:52:30.696325+0000 2021-02-10T04:52:31.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: mon.a (mon.0) 2401 : cluster [DBG] osd.0 reported immediately failed by osd.5 2021-02-10T04:52:31.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 2021-02-10T04:52:31.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:52:31.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: -10T04:52:30.718024+0000 mon.a (mon.0) 2402 : cluster [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:52:31.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:52:31.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 10T04:52:30.896132+0000 mon.a (mon 2021-02-10T04:52:31.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: .0) 2403 : cluster [DBG] osd.0 reported immediately failed by osd.2 2021-02-10T04:52:31.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:52:31.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 04:52:30.896277+0000 mon.a (mon. 2021-02-10T04:52:31.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 0) 2404 : cluster [DBG] osd.0 reported immediately failed by osd.2 2021-02-10T04:52:31.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:52:31.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: -10T04:52:30.896412+0000 mon.a ( 2021-02-10T04:52:31.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: mon.0) 2405 : cluster [DBG] osd.0 reported immediately failed by osd.1 2021-02-10T04:52:31.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:52:31.452 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 02-10T04:52:30.896537+0000 2021-02-10T04:52:31.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: mon.a (mon.0) 2406 : cluster [DBG] 2021-02-10T04:52:31.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: osd.0 reported immediately failed by osd.3 2021-02-10T04:52:31.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: cluster 2021-02-10T04:52:30 2021-02-10T04:52:31.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: .896669+0000 mon.a (mon.0) 2407 : cluster 2021-02-10T04:52:31.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: [DBG] 2021-02-10T04:52:31.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 2021-02-10T04:52:31.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: osd.0 reported immediately failed by osd.1 2021-02-10T04:52:31.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 2021-02-10T04:52:31.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: cluster 2021-02-10T04:52:31.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 2021-02-10T04:52:31.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 2021 2021-02-10T04:52:31.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: - 2021-02-10T04:52:31.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 02-10T04:52:30.896791 2021-02-10T04:52:31.453 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2408 : cluster [DBG] osd.0 reported immediately failed by osd.5 2021-02-10T04:52:31.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:52:31.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 02-10T04:52:30.896891+0000 mon.a ( 2021-02-10T04:52:31.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: mon.0) 2409 : cluster [DBG] osd.0 reported immediately failed by osd.3 2021-02-10T04:52:31.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:52:31.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 02-10T04:52:30.897020+0000 mon.a (mon.0) 2410 : cluster [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:52:31.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:52:31.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 02-10T04:52:30.897118+0000 2021-02-10T04:52:31.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: mon.a (mon.0) 2411 : cluster [DBG] osd.0 reported immediately failed by osd.7 2021-02-10T04:52:31.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: cluster 2021-02-10T04:52:31.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 2021-02-10T04:52:30.897210+0000 2021-02-10T04:52:31.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: mon.a (mon.0) 2412 : cluster [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:52:31.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 2021-02-10T04:52:31.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:52:31.454 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 02-10T04:52:30.897312+0000 mon.a ( 2021-02-10T04:52:31.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: mon.0) 2413 : cluster [DBG] osd.0 reported immediately failed by osd.5 2021-02-10T04:52:31.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:52:31.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 02-10T04:52:30.897404+0000 mon.a ( 2021-02-10T04:52:31.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: mon.0) 2414 : cluster [DBG] osd.0 reported immediately failed by osd.6 2021-02-10T04:52:31.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:52:31.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: -02-10T04:52:30.897499+0000 mon.a ( 2021-02-10T04:52:31.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: mon.0) 2415 : cluster [DBG] osd.0 reported immediately failed by osd.7 2021-02-10T04:52:31.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:52:31.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 02-10T04:52:30.903419+0000 2021-02-10T04:52:31.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: mon.a (mon.0) 2416 : cluster [DBG] 2021-02-10T04:52:31.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: osd.0 reported immediately failed by osd.6 2021-02-10T04:52:31.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: cluster 2021-02-10T04:52: 2021-02-10T04:52:31.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: 30.918357+0000 mon.a (mon.0) 2417 : cluster 2021-02-10T04:52:31.455 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:31 smithi085 conmon[35961]: [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:52:32.062 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:32 smithi164 conmon[29288]: cluster 2021-02-10T04:52:30.405102+0000 osd.5 (osd.5) 171 : cluster 2021-02-10T04:52:32.062 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:32 smithi164 conmon[29288]: [DBG] 6.24 scrub ok 2021-02-10T04:52:32.062 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:32 smithi164 conmon[29288]: cluster 2021-02-10T04:52:30.433364+0000 mgr.y (mgr.14140) 1250 : cluster [DBG] pgmap v2062: 198 pgs: 1 activating, 2 remapped+peering, 3 peering, 3 unknown, 189 active+clean; 759 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:52:32.062 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:32 smithi164 conmon[29288]: cluster 2021-02-10T04:52:32.062 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:32 smithi164 conmon[29288]: 2021-02-10T04:52:31.055458 2021-02-10T04:52:32.062 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:32 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2418 : 2021-02-10T04:52:32.062 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:32 smithi164 conmon[29288]: cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2021-02-10T04:52:32.062 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:32 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:52:32.062 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:32 smithi164 conmon[29288]: 10T04:52:31.065368+0000 mon.a (mon.0) 2021-02-10T04:52:32.062 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:32 smithi164 conmon[29288]: 2419 : cluster [DBG] osdmap e1147: 8 total, 7 up, 8 in 2021-02-10T04:52:32.062 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:32 smithi164 conmon[29288]: cluster 2021-02-10T04:52:32.063 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:32 smithi164 conmon[29288]: 2021-02-10T04:52: 2021-02-10T04:52:32.063 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:32 smithi164 conmon[29288]: 31.272966+0000 osd.1 (osd.1) 489 : cluster [DBG] 6.4 scrub ok 2021-02-10T04:52:32.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:32 smithi085 conmon[35961]: cluster 2021-02-10T04:52:30.405102+0000 osd.5 2021-02-10T04:52:32.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:32 smithi085 conmon[35961]: (osd.5) 171 : cluster [DBG] 6.24 scrub ok 2021-02-10T04:52:32.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:32 smithi085 conmon[35961]: cluster 2021-02-10T04:52:30. 2021-02-10T04:52:32.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:32 smithi085 conmon[35961]: 433364+0000 mgr.y (mgr.14140) 1250 : cluster [DBG] pgmap v2062: 198 pgs: 1 activating, 2 remapped+peering, 3 peering, 3 unknown, 189 active+clean; 759 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:52:32.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:32 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:52:32.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:32 smithi085 conmon[35961]: -10T04:52:31.055458+0000 mon.a (mon.0) 2418 : cluster 2021-02-10T04:52:32.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:32 smithi085 conmon[35961]: [WRN] Health check failed: 1 osds down (OSD_DOWN) 2021-02-10T04:52:32.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:32 smithi085 conmon[35961]: cluster 2021-02-10T04:52 2021-02-10T04:52:32.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:32 smithi085 conmon[35961]: :31.065368+0000 mon.a (mon.0) 2419 : cluster [DBG] osdmap e1147: 8 total, 7 up, 8 in 2021-02-10T04:52:32.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:32 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:52:32.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:32 smithi085 conmon[35961]: -10T04:52:31.272966+0000 2021-02-10T04:52:32.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:32 smithi085 conmon[35961]: osd.1 (osd.1) 489 : cluster [DBG] 2021-02-10T04:52:32.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:32 smithi085 conmon[35961]: 6.4 scrub ok 2021-02-10T04:52:32.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:32 smithi085 conmon[29319]: cluster 2021-02-10T04:52:30.405102+0000 osd.5 (osd. 2021-02-10T04:52:32.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:32 smithi085 conmon[29319]: 5) 171 : cluster [DBG] 6.24 scrub ok 2021-02-10T04:52:32.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:32 smithi085 conmon[29319]: cluster 2021-02-10T04:52:30.433364+0000 mgr.y (mgr.14140) 1250 : cluster [DBG] pgmap v2062: 198 pgs: 1 activating, 2 remapped+peering, 3 peering, 3 unknown, 189 active+clean; 759 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:52:32.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:32 smithi085 conmon[29319]: cluster 2021-02-10T04:52:32.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:32 smithi085 conmon[29319]: 2021-02-10T04:52:31.055458+0000 mon.a (mon.0) 2418 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2021-02-10T04:52:32.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:32 smithi085 conmon[29319]: cluster 2021-02-10T04:52:31.065368+0000 mon.a (mon. 2021-02-10T04:52:32.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:32 smithi085 conmon[29319]: 0) 2419 : cluster [DBG] osdmap e1147: 8 total, 7 up, 8 in 2021-02-10T04:52:32.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:32 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:52:32.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:32 smithi085 conmon[29319]: 52:31.272966+0000 osd.1 (osd.1) 489 : cluster 2021-02-10T04:52:32.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:32 smithi085 conmon[29319]: [DBG] 6.4 scrub ok 2021-02-10T04:52:32.941 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:52:32 smithi085 podman[157088]: e3e5902a7adc1d6c2993dcd3fc4c15ff2d74ad4bdf6c27f047a0b544bba8e669 2021-02-10T04:52:33.069 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:33 smithi164 conmon[29288]: cluster 2021-02-10T04:52:31.403818+0000 osd.5 2021-02-10T04:52:33.069 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:33 smithi164 conmon[29288]: (osd.5) 172 : cluster [DBG] 6.7 scrub ok 2021-02-10T04:52:33.069 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:33 smithi164 conmon[29288]: cluster 2021-02-10T04:52:31.770899+0000 osd.4 (osd.4) 2636 : cluster [DBG] 4.1b scrub ok 2021-02-10T04:52:33.069 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:33 smithi164 conmon[29288]: cluster 2021-02-10T04:52:33.069 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:33 smithi164 conmon[29288]: 2021-02-10T04:52:32.073967+0000 2021-02-10T04:52:33.069 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:33 smithi164 conmon[29288]: mon.a (mon.0) 2420 : cluster [DBG] osdmap e1148: 8 total, 7 up, 8 in 2021-02-10T04:52:33.069 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:33 smithi164 conmon[29288]: 2021-02-10T04:52:33.069 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:33 smithi164 conmon[29288]: cluster 2021-02-10T04:52:33.070 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:33 smithi164 conmon[29288]: 2021-02-10T04:52:32 2021-02-10T04:52:33.070 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:33 smithi164 conmon[29288]: .124898+0000 osd.2 (osd.2 2021-02-10T04:52:33.070 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:33 smithi164 conmon[29288]: ) 1997 : cluster [DBG] 6.20 scrub ok 2021-02-10T04:52:33.353 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:33 smithi085 conmon[35961]: cluster 2021-02-10T04:52:31. 2021-02-10T04:52:33.353 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:33 smithi085 conmon[35961]: 403818+0000 osd.5 (osd.5) 172 : cluster [DBG] 6.7 scrub ok 2021-02-10T04:52:33.353 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:33 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:52:33.353 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:33 smithi085 conmon[35961]: -10T04:52:31.770899+0000 osd.4 (osd.4) 2021-02-10T04:52:33.353 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:33 smithi085 conmon[35961]: 2636 : cluster [DBG] 4.1b scrub ok 2021-02-10T04:52:33.353 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:33 smithi085 conmon[35961]: cluster 2021-02-10T04:52:33.353 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:33 smithi085 conmon[35961]: 2021-02-10T04:52 2021-02-10T04:52:33.353 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:33 smithi085 conmon[35961]: :32.073967+0000 mon.a (mon 2021-02-10T04:52:33.353 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:33 smithi085 conmon[35961]: .0) 2420 : cluster [DBG] osdmap e1148: 8 total, 7 up, 8 in 2021-02-10T04:52:33.354 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:33 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:52:33.354 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:33 smithi085 conmon[35961]: -10T04:52:32.124898+0000 2021-02-10T04:52:33.354 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:33 smithi085 conmon[35961]: osd.2 (osd.2) 1997 : 2021-02-10T04:52:33.354 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:33 smithi085 conmon[35961]: cluster [DBG] 6.20 scrub ok 2021-02-10T04:52:33.354 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:33 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:52:33.354 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:33 smithi085 conmon[29319]: -10T04:52:31.403818+0000 osd.5 (osd.5) 172 : cluster [DBG] 6.7 scrub ok 2021-02-10T04:52:33.354 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:33 smithi085 conmon[29319]: cluster 2021-02-10T04:52:31.770899+0000 osd.4 (osd.4) 2636 : cluster [DBG] 4.1b scrub ok 2021-02-10T04:52:33.354 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:33 smithi085 conmon[29319]: cluster 2021-02-10T04:52:32.073967+0000 mon.a (mon.0) 2420 : cluster [DBG] osdmap e1148: 8 total, 7 up, 8 in 2021-02-10T04:52:33.354 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:33 smithi085 conmon[29319]: cluster 2021-02-10T04:52:32.124898+0000 osd.2 (osd.2) 1997 : cluster [DBG] 6.20 scrub ok 2021-02-10T04:52:33.992 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:52:34.009 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:52:34.028 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:52:34.062 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:52:34.087 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:52:34.106 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:52:34.127 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:52:34.128 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-02-10T04:52:34.128 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: problem getting command descriptions from osd.0 2021-02-10T04:52:34.128 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:52:34.128 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:52:34.128 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:52:34.128 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:52:34.128 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:52:34.128 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:52:34.128 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:52:34.393 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:52:34 smithi085 conmon[49106]: debug 2021-02-10T04:52:34.139+0000 7f77a3e94700 -1 osd.2 1149 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) 2021-02-10T04:52:34.393 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:34 smithi085 conmon[29319]: cluster 2021-02-10T04:52:32.433916 2021-02-10T04:52:34.393 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:34 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 1251 : cluster [DBG] pgmap v2065: 198 pgs: 5 active+undersized+degraded, 5 active+undersized, 16 stale+active+clean, 1 activating, 15 peering, 3 unknown, 153 active+clean; 818 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s wr, 9 op/s; 23/2115 objects degraded (1.087%) 2021-02-10T04:52:34.393 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:34 smithi085 conmon[29319]: cluster 2021-02-10T04:52:33.066578+0000 mon.a (mon.0) 2421 : cluster [WRN] Health check failed: Degraded data redundancy: 23/2115 objects degraded (1.087%), 5 pgs degraded (PG_DEGRADED) 2021-02-10T04:52:34.393 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:34 smithi085 conmon[29319]: cluster 2021-02-10T04:52:33.074892+0000 mon.a (mon.0) 2422 : cluster [DBG] osdmap e1149: 8 total, 7 up, 8 in 2021-02-10T04:52:34.393 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:34 smithi085 conmon[29319]: cluster 2021-02-10T04:52:33.088422+0000 osd.6 (osd.6) 168 : cluster [DBG] 2.17 scrub ok 2021-02-10T04:52:34.393 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:34 smithi085 conmon[29319]: audit 2021-02-10T04:52:33.970277+0000 mon.c (mon.1) 100 : audit [DBG] from='client.? 172.21.15.85:0/1349051992' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:52:34.393 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:34 smithi085 conmon[35961]: cluster 2021-02-10T04:52:34.394 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:34 smithi085 conmon[35961]: 2021-02-10T04:52:34.394 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:34 smithi085 conmon[35961]: 2021 2021-02-10T04:52:34.394 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:34 smithi085 conmon[35961]: -02-10T04:52:32.433916+0000 mgr.y ( 2021-02-10T04:52:34.394 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:34 smithi085 conmon[35961]: mgr.14140) 1251 : cluster [DBG] pgmap v2065: 198 pgs: 5 active+undersized+degraded, 5 active+undersized, 16 stale+active+clean, 1 activating, 15 peering, 3 unknown, 153 active+clean; 818 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s wr, 9 op/s; 23/2115 objects degraded (1.087%) 2021-02-10T04:52:34.394 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:34 smithi085 conmon[35961]: cluster 2021-02-10T04:52 2021-02-10T04:52:34.394 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:34 smithi085 conmon[35961]: :33.066578+0000 mon.a (mon.0) 2421 : cluster [WRN] Health check failed: Degraded data redundancy: 23/2115 objects degraded (1.087%), 5 pgs degraded (PG_DEGRADED) 2021-02-10T04:52:34.394 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:34 smithi085 conmon[35961]: cluster 2021-02-10T04:52:34.394 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:34 smithi085 conmon[35961]: 2021-02-10T04:52:33.074892+0000 mon.a (mon. 2021-02-10T04:52:34.394 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:34 smithi085 conmon[35961]: 0) 2422 : cluster [DBG] osdmap e1149: 8 total, 7 up, 8 in 2021-02-10T04:52:34.394 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:34 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:52:34.394 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:34 smithi085 conmon[35961]: 10T04:52:33.088422+0000 osd.6 (osd 2021-02-10T04:52:34.394 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:34 smithi085 conmon[35961]: .6) 168 : cluster [DBG] 2021-02-10T04:52:34.394 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:34 smithi085 conmon[35961]: 2.17 scrub ok 2021-02-10T04:52:34.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:34 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:52:34.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:34 smithi085 conmon[35961]: -10T04:52:33.970277+0000 mon.c (mon.1) 100 : audit 2021-02-10T04:52:34.395 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:34 smithi085 conmon[35961]: [DBG] from='client.? 172.21.15.85:0/1349051992' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:52:34.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:34 smithi164 conmon[29288]: cluster 2021-02-10T04:52:32.433916+0000 mgr.y (mgr.14140) 1251 : cluster 2021-02-10T04:52:34.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:34 smithi164 conmon[29288]: [DBG] pgmap v2065: 198 pgs: 5 active+undersized+degraded, 5 active+undersized, 16 stale+active+clean, 1 activating, 15 peering, 3 unknown, 153 active+clean; 818 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s wr, 9 op/s; 23/2115 objects degraded (1.087%) 2021-02-10T04:52:34.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:34 smithi164 conmon[29288]: cluster 2021-02-10T04:52:33.066578+0000 mon.a (mon.0) 2421 : cluster [WRN] Health check failed: Degraded data redundancy: 23/2115 objects degraded (1.087%), 5 pgs degraded (PG_DEGRADED) 2021-02-10T04:52:34.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:34 smithi164 conmon[29288]: cluster 2021-02-10T04:52:33.074892+0000 mon.a (mon.0) 2422 : cluster [DBG] osdmap e1149: 8 total, 7 up, 8 in 2021-02-10T04:52:34.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:34 smithi164 conmon[29288]: cluster 2021-02-10T04:52:33.088422+0000 osd.6 (osd.6) 168 : cluster [DBG] 2.17 scrub ok 2021-02-10T04:52:34.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:34 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:52:34.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:34 smithi164 conmon[29288]: -10T04:52:33.970277+0000 mon.c (mon.1) 100 : audit [DBG] from='client.? 172.21.15.85:0/1349051992' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:52:35.096 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is unset 2021-02-10T04:52:35.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:35 smithi164 conmon[29288]: cluster 2021-02-10T04:52:33.741704+0000 osd.4 2021-02-10T04:52:35.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:35 smithi164 conmon[29288]: (osd.4) 2637 : cluster [DBG] 5.1c scrub ok 2021-02-10T04:52:35.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:35 smithi164 conmon[29288]: cluster 2021-02-10T04:52:34.047348+0000 osd.3 (osd.3) 17 : cluster [DBG] 5.0 scrub ok 2021-02-10T04:52:35.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:35 smithi164 conmon[29288]: cluster 2021-02-10T04:52:34.089785+0000 osd.6 (osd.6) 2021-02-10T04:52:35.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:35 smithi164 conmon[29288]: 169 : cluster [DBG] 7.11 scrub ok 2021-02-10T04:52:35.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:35 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:52:35.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:35 smithi164 conmon[29288]: 10T04:52:34.193342+0000 mon.a (mon 2021-02-10T04:52:35.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:35 smithi164 conmon[29288]: .0) 2423 : audit [INF] from='client.? 172.21.15.85:0/3823981985' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:52:35.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:52:35 smithi085 conmon[49106]: debug 2021-02-10T04:52:35.167+0000 7f77a3e94700 -1 osd.2 1150 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) 2021-02-10T04:52:35.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:35 smithi085 conmon[29319]: cluster 2021-02-10T04:52:33.741704+0000 osd.4 (osd.4 2021-02-10T04:52:35.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:35 smithi085 conmon[29319]: ) 2637 : cluster [DBG] 5.1c scrub ok 2021-02-10T04:52:35.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:35 smithi085 conmon[29319]: cluster 2021-02-10T04:52:34.047348+0000 osd.3 (osd.3) 17 : cluster [DBG] 5.0 scrub ok 2021-02-10T04:52:35.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:35 smithi085 conmon[29319]: cluster 2021-02-10T04:52:34.089785+0000 osd.6 (osd.6) 169 : cluster 2021-02-10T04:52:35.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:35 smithi085 conmon[29319]: [DBG] 7.11 scrub ok 2021-02-10T04:52:35.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:35 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:52:35.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:35 smithi085 conmon[29319]: :52:34.193342+0000 mon.a (mon.0) 2423 : audit [INF] from='client.? 172.21.15.85:0/3823981985' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:52:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:35 smithi085 conmon[35961]: cluster 2021-02-10T04:52:33.741704+0000 2021-02-10T04:52:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:35 smithi085 conmon[35961]: osd.4 (osd.4) 2637 : cluster [DBG] 5.1c scrub ok 2021-02-10T04:52:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:35 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:52:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:35 smithi085 conmon[35961]: -10T04:52:34.047348+0000 osd.3 (osd.3) 17 : cluster [DBG] 5.0 scrub ok 2021-02-10T04:52:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:35 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:52:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:35 smithi085 conmon[35961]: -02-10T04:52:34.089785+0000 osd.6 ( 2021-02-10T04:52:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:35 smithi085 conmon[35961]: osd.6) 169 : cluster [DBG] 7.11 scrub ok 2021-02-10T04:52:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:35 smithi085 conmon[35961]: audit 2021-02-10T04:52:34.193342 2021-02-10T04:52:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:35 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2423 : audit [INF] from='client.? 172.21.15.85:0/3823981985' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:52:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:35 smithi085 conmon[35961]: 2021-02-10T04:52:36.089 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:52:36.089 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:52:36.387 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[49106]: debug 2021-02-10T04:52:36.162+0000 7f77a3e94700 -1 osd.2 1150 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) 2021-02-10T04:52:36.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[29319]: cluster 2021-02-10T04:52:34.140269+0000 2021-02-10T04:52:36.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[29319]: osd.2 (osd.2) 1998 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:36.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[29319]: cluster 2021-02-10T04:52:34.140274+0000 osd.2 (osd.2) 1999 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:36.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[29319]: cluster 2021-02-10T04:52:34.434567+0000 mgr.y (mgr.14140) 1252 : cluster [DBG] pgmap v2067: 198 pgs: 23 active+undersized+degraded, 30 active+undersized, 14 peering, 131 active+clean; 895 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 351 MiB/s rd, 98 MiB/s wr, 611 op/s; 225/2232 objects degraded (10.081%); 12 MiB/s, 22 objects/s recovering 2021-02-10T04:52:36.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[29319]: cluster 2021-02-10T04:52:35.068646+0000 mon.a (mon.0) 2424 : cluster [WRN] Health check update: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2021-02-10T04:52:36.388 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[29319]: cluster 2021-02-10T04:52:35.068894+0000 mon.a (mon.0 2021-02-10T04:52:36.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[29319]: ) 2425 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:52:36.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[29319]: audit 2021-02-10T04:52:35.080892+0000 mon.a (mon.0 2021-02-10T04:52:36.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[29319]: ) 2426 : audit [INF] from='client.? 172.21.15.85:0/3823981985' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:52:36.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[29319]: cluster 2021-02-10T04:52 2021-02-10T04:52:36.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[29319]: :35.080963+0000 mon.a (mon.0) 2427 2021-02-10T04:52:36.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[29319]: : cluster [DBG] osdmap e1150: 8 total, 7 up, 8 in 2021-02-10T04:52:36.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:52:36.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[29319]: -10T04:52:35.091366+0000 osd.3 ( 2021-02-10T04:52:36.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[29319]: osd.3) 18 : cluster [DBG] 4.14 scrub ok 2021-02-10T04:52:36.389 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[35961]: cluster 2021-02-10T04:52:34.140269+0000 osd.2 (osd 2021-02-10T04:52:36.389 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[35961]: .2) 1998 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:36.389 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[35961]: cluster 2021-02-10T04:52:34.140274+0000 osd.2 (osd.2) 1999 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:36.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[35961]: cluster 2021-02-10T04:52:34.434567+0000 2021-02-10T04:52:36.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[35961]: mgr.y (mgr.14140) 1252 : cluster [DBG] pgmap v2067: 198 pgs: 23 active+undersized+degraded, 30 active+undersized, 14 peering, 131 active+clean; 895 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 351 MiB/s rd, 98 MiB/s wr, 611 op/s; 225/2232 objects degraded (10.081%); 12 MiB/s, 22 objects/s recovering 2021-02-10T04:52:36.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[35961]: 2021-02-10T04:52:36.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:52:36.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[35961]: -10T04:52:35.068646 2021-02-10T04:52:36.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2424 : cluster 2021-02-10T04:52:36.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[35961]: [WRN] Health check update: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2021-02-10T04:52:36.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:52:36.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[35961]: T04:52:35.068894+0000 mon.a (mon.0) 2425 : 2021-02-10T04:52:36.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[35961]: cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:52:36.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[35961]: audit 2021 2021-02-10T04:52:36.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[35961]: -02-10T04:52:35.080892+0000 2021-02-10T04:52:36.390 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[35961]: mon.a (mon.0) 2426 : audit 2021-02-10T04:52:36.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[35961]: [INF] from='client.? 172.21.15.85:0/3823981985' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:52:36.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:52:36.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[35961]: -02-10T04:52:35.080963+0000 2021-02-10T04:52:36.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[35961]: mon.a (mon.0) 2427 : cluster [DBG] osdmap e1150: 8 total, 7 up, 8 in 2021-02-10T04:52:36.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:52:36.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[35961]: 04:52:35.091366+0000 osd.3 (osd.3) 18 : cluster [DBG] 2021-02-10T04:52:36.391 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:36 smithi085 conmon[35961]: 4.14 scrub ok 2021-02-10T04:52:36.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:36 smithi164 conmon[29288]: cluster 2021-02-10T04:52:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:36 smithi164 conmon[29288]: 2021-02-10T04:52:34 2021-02-10T04:52:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:36 smithi164 conmon[29288]: .140269+0000 osd.2 (osd.2) 1998 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:36 smithi164 conmon[29288]: cluster 2021-02-10T04:52:34.140274+0000 osd.2 (osd.2) 1999 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:36 smithi164 conmon[29288]: cluster 2021-02-10T04:52: 2021-02-10T04:52:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:36 smithi164 conmon[29288]: 34.434567+0000 mgr.y (mgr.14140) 1252 : cluster [DBG] pgmap v2067: 198 pgs: 23 active+undersized+degraded, 30 active+undersized, 14 peering, 131 active+clean; 895 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 351 MiB/s rd, 98 MiB/s wr, 611 op/s; 225/2232 objects degraded (10.081%); 12 MiB/s, 22 objects/s recovering 2021-02-10T04:52:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:36 smithi164 conmon[29288]: cluster 2021-02-10T04:52:35.068646+0000 2021-02-10T04:52:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:36 smithi164 conmon[29288]: mon.a (mon.0) 2424 : cluster [WRN] Health check update: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2021-02-10T04:52:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:36 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:52:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:36 smithi164 conmon[29288]: -02-10T04:52:35.068894+0000 mon.a 2021-02-10T04:52:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:36 smithi164 conmon[29288]: (mon.0) 2425 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:52:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:36 smithi164 conmon[29288]: audit 2021-02-10T04:52:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:36 smithi164 conmon[29288]: 2021-02-10T04:52:35.080892+0000 mon.a (mon.0) 2021-02-10T04:52:36.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:36 smithi164 conmon[29288]: 2426 : audit [INF] from='client.? 172.21.15.85:0/3823981985' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:52:36.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:36 smithi164 conmon[29288]: cluster 2021-02-10T04:52:35. 2021-02-10T04:52:36.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:36 smithi164 conmon[29288]: 080963+0000 mon.a (mon.0) 2427 : cluster [DBG] osdmap e1150: 8 total, 7 up, 8 in 2021-02-10T04:52:36.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:36 smithi164 conmon[29288]: cluster 2021-02-10T04:52:35.091366+0000 2021-02-10T04:52:36.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:36 smithi164 conmon[29288]: osd.3 (osd.3) 18 : cluster [DBG] 4.14 scrub ok 2021-02-10T04:52:37.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:37 smithi164 conmon[29288]: cluster 2021-02-10T04:52:35.168244 2021-02-10T04:52:37.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:37 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 2000 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:37.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:37 smithi164 conmon[29288]: cluster 2021-02-10T04:52:35.168255+0000 osd.2 (osd.2) 2001 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:37.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:37 smithi164 conmon[29288]: cluster 2021-02-10T04:52:35.170154+0000 osd.2 (osd.2) 2002 : cluster [DBG] 6.8 scrub ok 2021-02-10T04:52:37.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:37 smithi164 conmon[29288]: cluster 2021-02-10T04:52:36.083777+0000 osd.6 (osd.6) 170 : cluster [DBG] 2.1a scrub ok 2021-02-10T04:52:37.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:52:37 smithi085 conmon[49106]: debug 2021-02-10T04:52:37.140+0000 7f77a3e94700 -1 osd.2 1150 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) 2021-02-10T04:52:37.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:37 smithi085 conmon[35961]: cluster 2021-02-10T04:52:37.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:37 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:52:37.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:37 smithi085 conmon[35961]: :52:35.168244+0000 osd.2 (osd.2) 2000 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:37.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:37 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:52:37.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:37 smithi085 conmon[35961]: T04:52:35.168255+0000 osd.2 (osd.2) 2001 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:37.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:37 smithi085 conmon[35961]: cluster 2021-02-10T04:52:37.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:37 smithi085 conmon[35961]: 2021-02-10T04:52:35.170154+0000 osd.2 (osd.2) 2002 : cluster [DBG] 2021-02-10T04:52:37.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:37 smithi085 conmon[35961]: 6.8 scrub ok 2021-02-10T04:52:37.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:37 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:52:37.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:37 smithi085 conmon[35961]: 02-10T04:52:36.083777+0000 osd.6 (osd.6) 170 : cluster 2021-02-10T04:52:37.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:37 smithi085 conmon[35961]: [DBG] 2.1a scrub ok 2021-02-10T04:52:37.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:37 smithi085 conmon[29319]: cluster 2021-02-10T04:52:35.168244+0000 osd.2 ( 2021-02-10T04:52:37.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:37 smithi085 conmon[29319]: osd.2) 2000 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:37.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:37 smithi085 conmon[29319]: cluster 2021-02-10T04:52:35.168255+0000 osd.2 (osd. 2021-02-10T04:52:37.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:37 smithi085 conmon[29319]: 2) 2001 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:37.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:37 smithi085 conmon[29319]: cluster 2021-02-10T04:52:37.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:37 smithi085 conmon[29319]: 2021-02-10T04:52:35.170154+0000 osd.2 (osd.2 2021-02-10T04:52:37.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:37 smithi085 conmon[29319]: ) 2002 : cluster [DBG] 6.8 scrub ok 2021-02-10T04:52:37.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:37 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:52:37.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:37 smithi085 conmon[29319]: -02-10T04:52:36.083777+0000 2021-02-10T04:52:37.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:37 smithi085 conmon[29319]: osd.6 (osd.6) 170 : cluster [DBG] 2021-02-10T04:52:37.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:37 smithi085 conmon[29319]: 2.1a scrub ok 2021-02-10T04:52:37.571 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:52:37.572 INFO:teuthology.orchestra.run.smithi164.stdout: "size": 20, 2021-02-10T04:52:37.572 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 600, 2021-02-10T04:52:37.572 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [ 2021-02-10T04:52:37.572 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.572 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:865 7.11 7:8e9d3756:::smithi08575397-17:head [copy-from ver 29] snapc 37=[36,32,2e] RETRY=1 ondisk+retry+write+known_if_redirected e1007)", 2021-02-10T04:52:37.572 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:24.520229+0000", 2021-02-10T04:52:37.572 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 493.04800383100002, 2021-02-10T04:52:37.572 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0090932100000001, 2021-02-10T04:52:37.572 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:52:37.572 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:52:37.573 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:52:37.573 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:52:37.573 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:52:37.573 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 865 2021-02-10T04:52:37.573 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.573 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:52:37.573 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.573 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:52:37.573 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520229+0000", 2021-02-10T04:52:37.573 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:52:37.573 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.573 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.574 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:52:37.574 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520229+0000", 2021-02-10T04:52:37.574 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5864e-05 2021-02-10T04:52:37.574 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.574 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.574 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:52:37.574 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520274+0000", 2021-02-10T04:52:37.574 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.2700000000000004e-06 2021-02-10T04:52:37.574 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.574 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.574 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:52:37.574 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520283+0000", 2021-02-10T04:52:37.574 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4335000000000001e-05 2021-02-10T04:52:37.575 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.575 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.575 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:52:37.575 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520297+0000", 2021-02-10T04:52:37.575 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1950000000000003e-06 2021-02-10T04:52:37.575 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.575 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.575 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:52:37.575 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520305+0000", 2021-02-10T04:52:37.575 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.038909046000000003 2021-02-10T04:52:37.575 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.575 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.575 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.576 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.559214+0000", 2021-02-10T04:52:37.576 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1300000000000001e-06 2021-02-10T04:52:37.576 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.576 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.576 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for peered", 2021-02-10T04:52:37.576 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.559222+0000", 2021-02-10T04:52:37.576 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0032650030000001 2021-02-10T04:52:37.576 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.576 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.576 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.576 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:25.562487+0000", 2021-02-10T04:52:37.576 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00013934400000000001 2021-02-10T04:52:37.576 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.576 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.577 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:52:37.577 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:25.562627+0000", 2021-02-10T04:52:37.577 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.96669502299999999 2021-02-10T04:52:37.577 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.577 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.577 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:52:37.577 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.529322+0000", 2021-02-10T04:52:37.577 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0090932100000001 2021-02-10T04:52:37.577 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.577 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:52:37.577 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.577 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.577 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.578 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:866 7.11 7:8e9d3756:::smithi08575397-17:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007)", 2021-02-10T04:52:37.578 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:24.520353+0000", 2021-02-10T04:52:37.578 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 493.04787968699998, 2021-02-10T04:52:37.578 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0096650170000001, 2021-02-10T04:52:37.578 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:52:37.578 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T04:52:37.578 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:52:37.578 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:52:37.578 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:52:37.578 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 866 2021-02-10T04:52:37.578 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.578 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:52:37.578 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.579 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:52:37.579 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520353+0000", 2021-02-10T04:52:37.579 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:52:37.579 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.579 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.579 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:52:37.579 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520353+0000", 2021-02-10T04:52:37.579 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.6482000000000001e-05 2021-02-10T04:52:37.579 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.579 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.579 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:52:37.579 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520389+0000", 2021-02-10T04:52:37.579 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.7499999999999997e-06 2021-02-10T04:52:37.580 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.580 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.580 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:52:37.580 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520396+0000", 2021-02-10T04:52:37.580 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2269999999999999e-05 2021-02-10T04:52:37.580 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.580 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.580 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:52:37.580 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520408+0000", 2021-02-10T04:52:37.580 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.5300000000000004e-06 2021-02-10T04:52:37.580 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.580 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.580 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:52:37.581 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520414+0000", 2021-02-10T04:52:37.581 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.038829508999999998 2021-02-10T04:52:37.581 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.581 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.581 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.581 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.559243+0000", 2021-02-10T04:52:37.581 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.3689999999999998e-06 2021-02-10T04:52:37.581 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.581 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.581 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for peered", 2021-02-10T04:52:37.581 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.559249+0000", 2021-02-10T04:52:37.581 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.003527509 2021-02-10T04:52:37.581 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.582 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.582 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.582 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:25.562776+0000", 2021-02-10T04:52:37.582 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5167999999999999e-05 2021-02-10T04:52:37.582 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.582 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.582 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:52:37.582 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:25.562821+0000", 2021-02-10T04:52:37.582 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.967184818 2021-02-10T04:52:37.582 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.582 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.582 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.583 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.530006+0000", 2021-02-10T04:52:37.583 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1612e-05 2021-02-10T04:52:37.583 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.583 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.583 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:52:37.583 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.530018+0000", 2021-02-10T04:52:37.583 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0096650170000001 2021-02-10T04:52:37.583 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.583 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:52:37.583 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.583 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.583 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.584 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=2 ondisk+retry+write+known_if_redirected e1007)", 2021-02-10T04:52:37.584 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:24.520629+0000", 2021-02-10T04:52:37.584 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 493.04760317300003, 2021-02-10T04:52:37.584 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 431.18813977600001, 2021-02-10T04:52:37.584 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:52:37.584 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:52:37.584 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:52:37.584 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:52:37.584 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:52:37.584 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 1822 2021-02-10T04:52:37.585 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.585 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:52:37.585 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.585 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:52:37.585 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520629+0000", 2021-02-10T04:52:37.585 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:52:37.585 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.585 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.585 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:52:37.585 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520629+0000", 2021-02-10T04:52:37.585 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.4091999999999997e-05 2021-02-10T04:52:37.585 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.585 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.586 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:52:37.586 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520673+0000", 2021-02-10T04:52:37.586 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.5389999999999998e-06 2021-02-10T04:52:37.586 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.586 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.586 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:52:37.586 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520679+0000", 2021-02-10T04:52:37.586 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2472e-05 2021-02-10T04:52:37.586 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.586 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.586 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:52:37.586 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520691+0000", 2021-02-10T04:52:37.587 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.9959999999999996e-06 2021-02-10T04:52:37.587 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.587 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.587 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:52:37.587 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520698+0000", 2021-02-10T04:52:37.587 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.002521695 2021-02-10T04:52:37.587 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.587 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.587 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.587 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.523220+0000", 2021-02-10T04:52:37.587 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.5290000000000008e-06 2021-02-10T04:52:37.587 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.588 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.588 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for peered", 2021-02-10T04:52:37.588 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.523229+0000", 2021-02-10T04:52:37.588 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0220689999999999 2021-02-10T04:52:37.588 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.588 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.588 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.588 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:25.545298+0000", 2021-02-10T04:52:37.588 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000116295 2021-02-10T04:52:37.588 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.588 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.588 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:52:37.588 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:25.545414+0000", 2021-02-10T04:52:37.588 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 337.42098471399999 2021-02-10T04:52:37.589 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.589 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.589 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.589 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:02.966399+0000", 2021-02-10T04:52:37.589 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0069e-05 2021-02-10T04:52:37.589 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.589 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.589 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for peered", 2021-02-10T04:52:37.589 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:02.966409+0000", 2021-02-10T04:52:37.589 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0227805050000001 2021-02-10T04:52:37.589 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.589 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.590 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.590 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:03.989189+0000", 2021-02-10T04:52:37.590 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000110087 2021-02-10T04:52:37.590 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.590 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.590 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:52:37.590 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:03.989299+0000", 2021-02-10T04:52:37.590 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 54.512301180000001 2021-02-10T04:52:37.590 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.590 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.590 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.590 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:58.501600+0000", 2021-02-10T04:52:37.590 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3611000000000001e-05 2021-02-10T04:52:37.591 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.591 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.591 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for peered", 2021-02-10T04:52:37.591 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:58.501614+0000", 2021-02-10T04:52:37.591 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0539863439999999 2021-02-10T04:52:37.591 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.591 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.591 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.591 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:59.555600+0000", 2021-02-10T04:52:37.591 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00012568799999999999 2021-02-10T04:52:37.591 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.591 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.591 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:52:37.592 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:59.555726+0000", 2021-02-10T04:52:37.592 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 36.153042960000001 2021-02-10T04:52:37.592 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.592 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.592 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:52:37.592 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:35.708769+0000", 2021-02-10T04:52:37.592 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 431.18813977600001 2021-02-10T04:52:37.592 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.592 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:52:37.592 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.592 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.592 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.592 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected e1007)", 2021-02-10T04:52:37.593 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:24.520750+0000", 2021-02-10T04:52:37.593 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 493.04748267000002, 2021-02-10T04:52:37.593 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 431.189838533, 2021-02-10T04:52:37.593 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:52:37.593 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T04:52:37.593 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:52:37.593 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:52:37.593 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:52:37.593 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 1823 2021-02-10T04:52:37.593 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.593 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:52:37.593 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.593 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:52:37.594 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520750+0000", 2021-02-10T04:52:37.594 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:52:37.594 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.594 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.594 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:52:37.594 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520750+0000", 2021-02-10T04:52:37.594 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8719000000000001e-05 2021-02-10T04:52:37.594 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.594 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.594 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:52:37.594 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520778+0000", 2021-02-10T04:52:37.594 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.8170000000000004e-06 2021-02-10T04:52:37.595 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.595 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.595 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:52:37.595 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520784+0000", 2021-02-10T04:52:37.595 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4408e-05 2021-02-10T04:52:37.595 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.595 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.595 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:52:37.595 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520799+0000", 2021-02-10T04:52:37.595 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.7829999999999998e-06 2021-02-10T04:52:37.595 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.595 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.595 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:52:37.596 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520803+0000", 2021-02-10T04:52:37.596 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0025132660000000001 2021-02-10T04:52:37.596 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.596 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.596 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.596 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.523317+0000", 2021-02-10T04:52:37.596 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.469e-06 2021-02-10T04:52:37.596 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.596 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.596 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for peered", 2021-02-10T04:52:37.596 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.523324+0000", 2021-02-10T04:52:37.596 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0222044850000001 2021-02-10T04:52:37.596 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.597 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.597 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.597 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:25.545529+0000", 2021-02-10T04:52:37.597 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.0717999999999999e-05 2021-02-10T04:52:37.597 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.597 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.597 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:52:37.597 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:25.545569+0000", 2021-02-10T04:52:37.597 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 337.42086338299998 2021-02-10T04:52:37.597 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.597 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.598 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.598 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:02.966433+0000", 2021-02-10T04:52:37.598 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.9880000000000001e-06 2021-02-10T04:52:37.598 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.598 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.598 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for peered", 2021-02-10T04:52:37.598 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:02.966439+0000", 2021-02-10T04:52:37.598 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0229873359999999 2021-02-10T04:52:37.598 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.598 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.598 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.598 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:03.989426+0000", 2021-02-10T04:52:37.599 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.8639000000000001e-05 2021-02-10T04:52:37.599 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.599 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.599 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:52:37.599 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:03.989465+0000", 2021-02-10T04:52:37.599 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 54.512182013 2021-02-10T04:52:37.599 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.599 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.599 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.599 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:58.501647+0000", 2021-02-10T04:52:37.599 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.4399999999999999e-06 2021-02-10T04:52:37.599 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.599 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.600 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for peered", 2021-02-10T04:52:37.600 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:58.501654+0000", 2021-02-10T04:52:37.600 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0541827640000001 2021-02-10T04:52:37.600 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.600 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.600 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.600 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:59.555837+0000", 2021-02-10T04:52:37.600 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.1437000000000002e-05 2021-02-10T04:52:37.600 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.600 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.600 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:52:37.600 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:59.555858+0000", 2021-02-10T04:52:37.600 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 36.154705172 2021-02-10T04:52:37.601 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.601 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.601 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.601 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:35.710564+0000", 2021-02-10T04:52:37.601 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4695999999999999e-05 2021-02-10T04:52:37.601 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.601 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.601 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:52:37.601 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:35.710588+0000", 2021-02-10T04:52:37.601 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 431.189838533 2021-02-10T04:52:37.601 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.602 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:52:37.602 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.602 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.602 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.603 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:2821 7.11 7:8eb2e6b5:::smithi08575397-27:head [copy-from ver 168] snapc cc=[cc,cb,c9] RETRY=1 ondisk+retry+write+known_if_redirected e1007)", 2021-02-10T04:52:37.603 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:24.520839+0000", 2021-02-10T04:52:37.603 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 493.04739362599997, 2021-02-10T04:52:37.603 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0085211759999999, 2021-02-10T04:52:37.603 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:52:37.603 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:52:37.603 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:52:37.603 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:52:37.603 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:52:37.603 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 2821 2021-02-10T04:52:37.603 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.603 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:52:37.603 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.604 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:52:37.604 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520839+0000", 2021-02-10T04:52:37.604 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:52:37.604 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.604 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.604 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:52:37.604 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520839+0000", 2021-02-10T04:52:37.604 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2160999999999998e-05 2021-02-10T04:52:37.604 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.604 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.604 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:52:37.604 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520881+0000", 2021-02-10T04:52:37.604 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.9889999999999997e-06 2021-02-10T04:52:37.605 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.605 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.605 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:52:37.605 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520887+0000", 2021-02-10T04:52:37.605 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2689e-05 2021-02-10T04:52:37.605 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.605 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.605 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:52:37.605 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520900+0000", 2021-02-10T04:52:37.605 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.0610000000000003e-06 2021-02-10T04:52:37.605 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.605 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.605 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:52:37.606 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520905+0000", 2021-02-10T04:52:37.606 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.038414679 2021-02-10T04:52:37.606 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.606 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.606 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.606 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.559319+0000", 2021-02-10T04:52:37.606 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.2280000000000003e-06 2021-02-10T04:52:37.606 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.606 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.606 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for peered", 2021-02-10T04:52:37.606 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.559326+0000", 2021-02-10T04:52:37.606 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0035212650000001 2021-02-10T04:52:37.607 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.607 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.607 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.607 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:25.562847+0000", 2021-02-10T04:52:37.607 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000111644 2021-02-10T04:52:37.607 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.607 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.607 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:52:37.607 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:25.562959+0000", 2021-02-10T04:52:37.607 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.96640146000000005 2021-02-10T04:52:37.608 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.608 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.608 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:52:37.608 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.529360+0000", 2021-02-10T04:52:37.608 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0085211759999999 2021-02-10T04:52:37.608 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.608 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:52:37.608 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.608 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.608 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.608 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14817.0:2822 7.11 7:8eb2e6b5:::smithi08575397-27:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e1007)", 2021-02-10T04:52:37.608 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:24.520938+0000", 2021-02-10T04:52:37.609 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 493.04729482099998, 2021-02-10T04:52:37.609 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0089547369999998, 2021-02-10T04:52:37.609 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:52:37.609 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T04:52:37.609 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:52:37.609 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14817", 2021-02-10T04:52:37.609 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/2026569549", 2021-02-10T04:52:37.609 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 2822 2021-02-10T04:52:37.609 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.609 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:52:37.609 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.609 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:52:37.609 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520938+0000", 2021-02-10T04:52:37.610 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:52:37.610 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.610 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.610 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:52:37.610 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520938+0000", 2021-02-10T04:52:37.610 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1031999999999997e-05 2021-02-10T04:52:37.610 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.610 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.610 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:52:37.610 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520969+0000", 2021-02-10T04:52:37.610 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6210000000000001e-06 2021-02-10T04:52:37.610 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.610 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.610 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:52:37.611 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520974+0000", 2021-02-10T04:52:37.611 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4473e-05 2021-02-10T04:52:37.611 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.611 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.611 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:52:37.611 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520989+0000", 2021-02-10T04:52:37.611 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.921e-06 2021-02-10T04:52:37.611 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.611 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.611 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:52:37.611 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.520994+0000", 2021-02-10T04:52:37.611 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.038350782 2021-02-10T04:52:37.611 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.612 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.612 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.612 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.559344+0000", 2021-02-10T04:52:37.612 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0140000000000004e-06 2021-02-10T04:52:37.612 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.612 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.612 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for peered", 2021-02-10T04:52:37.612 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:24.559350+0000", 2021-02-10T04:52:37.612 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0037162040000001 2021-02-10T04:52:37.612 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.612 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.612 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.613 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:25.563067+0000", 2021-02-10T04:52:37.613 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.1128999999999999e-05 2021-02-10T04:52:37.613 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.613 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.613 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for blocked object", 2021-02-10T04:52:37.613 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:25.563108+0000", 2021-02-10T04:52:37.613 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.966768399 2021-02-10T04:52:37.613 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.613 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.613 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.614 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.529876+0000", 2021-02-10T04:52:37.614 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6161999999999999e-05 2021-02-10T04:52:37.614 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.614 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.614 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:52:37.614 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:26.529892+0000", 2021-02-10T04:52:37.614 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0089547369999998 2021-02-10T04:52:37.614 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.614 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:52:37.614 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.614 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.614 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.614 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.16 1011/1010 [PushOp(7:6cacee86:::smithi08575397-26:2f7, version: 988'471, data_included: [1559253~178321,2089081~578711], data_size: 757032, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:6cacee86:::smithi08575397-26:2f7@988'471, size: 2756612, copy_subset: [0~2756612], clone_subset: {7:6cacee86:::smithi08575397-26:2b9=[]}, snapset: 374=[]:{2b9=[2ae,27c],2f7=[2f5,2f0,2ee,2ed,2eb,2ea,2d9],30d=[308,300],333=[332,32f,32a,328,31a],374=[374,373,372,370,36f,36b,368,366,35f,354,353,34e,34d,34b,349,345,344,343]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2756612, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1559253, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:52:37.615 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:28.656373+0000", 2021-02-10T04:52:37.615 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 488.91185899099997, 2021-02-10T04:52:37.615 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0099672589999999991, 2021-02-10T04:52:37.615 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:52:37.615 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:52:37.615 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:52:37.615 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.615 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:52:37.615 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.656373+0000", 2021-02-10T04:52:37.615 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:52:37.615 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.615 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.615 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:52:37.616 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.656373+0000", 2021-02-10T04:52:37.616 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0013001670000000001 2021-02-10T04:52:37.616 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.616 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.616 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:52:37.616 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.657674+0000", 2021-02-10T04:52:37.616 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0499000000000001e-05 2021-02-10T04:52:37.616 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.616 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.616 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:52:37.617 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.657694+0000", 2021-02-10T04:52:37.617 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.3170999999999998e-05 2021-02-10T04:52:37.617 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.617 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.618 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:52:37.618 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.657727+0000", 2021-02-10T04:52:37.618 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8783999999999998e-05 2021-02-10T04:52:37.618 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.618 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.618 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:52:37.618 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.657746+0000", 2021-02-10T04:52:37.618 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0073734930000000001 2021-02-10T04:52:37.618 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.618 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.618 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.619 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.665120+0000", 2021-02-10T04:52:37.619 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0694e-05 2021-02-10T04:52:37.619 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.619 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.619 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:52:37.619 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.665130+0000", 2021-02-10T04:52:37.619 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001210451 2021-02-10T04:52:37.619 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.619 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.619 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:52:37.620 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.666341+0000", 2021-02-10T04:52:37.620 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0099672589999999991 2021-02-10T04:52:37.620 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.620 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:52:37.620 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.620 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.620 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.620 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.16 1011/1010 [PushOp(7:6cacee86:::smithi08575397-26:30d, version: 995'479, data_included: [2021108~91444,2557501~785880], data_size: 877324, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:6cacee86:::smithi08575397-26:30d@995'479, size: 3680997, copy_subset: [480997~434941,1407473~705079,2557501~1123496], clone_subset: {7:6cacee86:::smithi08575397-26:2f7=[0~480997,915938~491535,2112552~444949]}, snapset: 374=[]:{2b9=[2ae,27c],2f7=[2f5,2f0,2ee,2ed,2eb,2ea,2d9],30d=[308,300],333=[332,32f,32a,328,31a],374=[374,373,372,370,36f,36b,368,366,35f,354,353,34e,34d,34b,349,345,344,343]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3680997, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2021108, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:52:37.620 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:28.680110+0000", 2021-02-10T04:52:37.620 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 488.88812256799997, 2021-02-10T04:52:37.620 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0099816599999999998, 2021-02-10T04:52:37.620 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:52:37.620 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:52:37.621 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:52:37.621 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.621 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:52:37.621 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.680110+0000", 2021-02-10T04:52:37.621 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:52:37.621 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.621 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.621 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:52:37.621 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.680110+0000", 2021-02-10T04:52:37.621 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0011890309999999999 2021-02-10T04:52:37.621 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.621 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.621 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:52:37.622 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.681299+0000", 2021-02-10T04:52:37.622 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4763e-05 2021-02-10T04:52:37.622 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.622 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.622 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:52:37.622 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.681324+0000", 2021-02-10T04:52:37.622 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.0151999999999998e-05 2021-02-10T04:52:37.622 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.622 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.622 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:52:37.622 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.681364+0000", 2021-02-10T04:52:37.622 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0234e-05 2021-02-10T04:52:37.622 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.623 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.623 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:52:37.623 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.681384+0000", 2021-02-10T04:52:37.623 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0075636619999999996 2021-02-10T04:52:37.623 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.623 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.623 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.623 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.688948+0000", 2021-02-10T04:52:37.623 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0399e-05 2021-02-10T04:52:37.623 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.623 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.623 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:52:37.623 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.688958+0000", 2021-02-10T04:52:37.624 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0011334190000000001 2021-02-10T04:52:37.624 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.624 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.624 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:52:37.624 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.690092+0000", 2021-02-10T04:52:37.624 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0099816599999999998 2021-02-10T04:52:37.624 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.624 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:52:37.624 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.624 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.624 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.624 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.16 1011/1010 [PushOp(7:6cacee86:::smithi08575397-26:333, version: 996'485, data_included: [686049~409317,1847413~438636], data_size: 847953, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:6cacee86:::smithi08575397-26:333@996'485, size: 2286049, copy_subset: [686049~409317,1847413~438636], clone_subset: {7:6cacee86:::smithi08575397-26:30d=[0~686049,1095366~752047]}, snapset: 374=[]:{2b9=[2ae,27c],2f7=[2f5,2f0,2ee,2ed,2eb,2ea,2d9],30d=[308,300],333=[332,32f,32a,328,31a],374=[374,373,372,370,36f,36b,368,366,35f,354,353,34e,34d,34b,349,345,344,343]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2286049, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2021-02-10T04:52:37.624 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:28.691816+0000", 2021-02-10T04:52:37.625 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 488.87641683300001, 2021-02-10T04:52:37.625 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010082539, 2021-02-10T04:52:37.625 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:52:37.625 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:52:37.625 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:52:37.625 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.625 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:52:37.625 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.691816+0000", 2021-02-10T04:52:37.625 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:52:37.625 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.625 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.625 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:52:37.625 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.691816+0000", 2021-02-10T04:52:37.625 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0011081540000000001 2021-02-10T04:52:37.626 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.626 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.626 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:52:37.626 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.692924+0000", 2021-02-10T04:52:37.626 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8170000000000001e-05 2021-02-10T04:52:37.626 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.626 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.626 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:52:37.626 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.692942+0000", 2021-02-10T04:52:37.626 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1245000000000003e-05 2021-02-10T04:52:37.626 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.626 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.627 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:52:37.627 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.692973+0000", 2021-02-10T04:52:37.627 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8142999999999999e-05 2021-02-10T04:52:37.627 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.627 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.627 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:52:37.627 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.692991+0000", 2021-02-10T04:52:37.627 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0078590580000000004 2021-02-10T04:52:37.627 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.627 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.627 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.627 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.700850+0000", 2021-02-10T04:52:37.627 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1731e-05 2021-02-10T04:52:37.628 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.628 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.628 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:52:37.628 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.700862+0000", 2021-02-10T04:52:37.628 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001036038 2021-02-10T04:52:37.628 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.628 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.628 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:52:37.628 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.701898+0000", 2021-02-10T04:52:37.628 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010082539 2021-02-10T04:52:37.628 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.628 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:52:37.628 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.629 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.629 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.629 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.16 1011/1010 [PushOp(7:6cacee86:::smithi08575397-26:head, version: 1000'490, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:6cacee86:::smithi08575397-26:head@1000'490, size: 3697837, copy_subset: [0~3697837], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:52:37.629 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:28.731732+0000", 2021-02-10T04:52:37.629 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 488.83650045600001, 2021-02-10T04:52:37.629 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0099831009999999994, 2021-02-10T04:52:37.629 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:52:37.629 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:52:37.629 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:52:37.629 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.629 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:52:37.629 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.731732+0000", 2021-02-10T04:52:37.629 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:52:37.630 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.630 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.630 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:52:37.630 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.731732+0000", 2021-02-10T04:52:37.630 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0010008879999999999 2021-02-10T04:52:37.630 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.630 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.630 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:52:37.630 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.732733+0000", 2021-02-10T04:52:37.630 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6137000000000002e-05 2021-02-10T04:52:37.630 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.630 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.630 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:52:37.631 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.732749+0000", 2021-02-10T04:52:37.631 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.5225e-05 2021-02-10T04:52:37.631 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.631 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.631 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:52:37.631 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.732774+0000", 2021-02-10T04:52:37.631 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2556e-05 2021-02-10T04:52:37.631 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.631 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.631 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:52:37.631 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.732787+0000", 2021-02-10T04:52:37.631 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0082074980000000006 2021-02-10T04:52:37.632 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.632 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.632 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.632 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.740994+0000", 2021-02-10T04:52:37.632 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4891e-05 2021-02-10T04:52:37.632 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.632 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.632 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:52:37.632 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.741009+0000", 2021-02-10T04:52:37.632 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00070590599999999996 2021-02-10T04:52:37.632 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.633 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.633 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:52:37.633 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.741715+0000", 2021-02-10T04:52:37.633 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0099831009999999994 2021-02-10T04:52:37.633 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.633 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:52:37.633 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.633 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.633 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.633 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.16 1011/1010 [PushOp(7:6cacee86:::smithi08575397-26:head, version: 1000'490, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:6cacee86:::smithi08575397-26:head@1000'490, size: 3697837, copy_subset: [0~3697837], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:52:37.633 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:28.743396+0000", 2021-02-10T04:52:37.633 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 488.82483677900001, 2021-02-10T04:52:37.633 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010392541999999999, 2021-02-10T04:52:37.634 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:52:37.634 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:52:37.634 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:52:37.634 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.634 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:52:37.634 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.743396+0000", 2021-02-10T04:52:37.634 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:52:37.634 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.634 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.634 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:52:37.634 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.743396+0000", 2021-02-10T04:52:37.634 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00097603700000000002 2021-02-10T04:52:37.634 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.635 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.635 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:52:37.635 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.744372+0000", 2021-02-10T04:52:37.635 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4667999999999999e-05 2021-02-10T04:52:37.635 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.635 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.635 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:52:37.635 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.744386+0000", 2021-02-10T04:52:37.635 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.6957999999999998e-05 2021-02-10T04:52:37.635 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.635 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.635 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:52:37.635 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.744413+0000", 2021-02-10T04:52:37.636 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2369e-05 2021-02-10T04:52:37.636 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.636 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.636 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:52:37.636 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.744426+0000", 2021-02-10T04:52:37.636 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00859854 2021-02-10T04:52:37.636 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.636 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.636 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.636 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.753024+0000", 2021-02-10T04:52:37.636 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7890000000000006e-06 2021-02-10T04:52:37.636 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.636 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.637 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:52:37.637 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.753034+0000", 2021-02-10T04:52:37.637 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00075418100000000004 2021-02-10T04:52:37.637 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.637 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.637 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:52:37.637 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.753788+0000", 2021-02-10T04:52:37.637 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010392541999999999 2021-02-10T04:52:37.637 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.637 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:52:37.637 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.637 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.637 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.637 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.15 1038/1037 [PushOp(7:adfc1887:::smithi08575397-25:218, version: 778'234, data_included: [2097152~793652], data_size: 793652, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:adfc1887:::smithi08575397-25:218@778'234, size: 2890804, copy_subset: [0~2890804], clone_subset: {}, snapset: 374=[]:{218=[203],27c=[27c],2cb=[2c8,2ae],2ff=[2f5],328=[328],335=[335,332,32f,32a],33e=[33e,33c,33a],343=[343],374=[374,373,372,370,36f,36b,368,366,35f,354,353,34e,34d,34b,349,345,344]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2890804, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:52:37.638 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:45:40.761541+0000", 2021-02-10T04:52:37.638 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 416.80669113900001, 2021-02-10T04:52:37.638 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010475346, 2021-02-10T04:52:37.638 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:52:37.638 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:52:37.638 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:52:37.638 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.639 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:52:37.639 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:40.761541+0000", 2021-02-10T04:52:37.639 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:52:37.639 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.639 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.639 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:52:37.639 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:40.761541+0000", 2021-02-10T04:52:37.639 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00093548000000000004 2021-02-10T04:52:37.639 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.639 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.639 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:52:37.639 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:40.762477+0000", 2021-02-10T04:52:37.639 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.3981e-05 2021-02-10T04:52:37.640 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.640 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.640 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:52:37.640 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:40.762501+0000", 2021-02-10T04:52:37.640 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.7656e-05 2021-02-10T04:52:37.640 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.640 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.640 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:52:37.640 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:40.762538+0000", 2021-02-10T04:52:37.640 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6436999999999999e-05 2021-02-10T04:52:37.640 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.640 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.640 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:52:37.641 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:40.762555+0000", 2021-02-10T04:52:37.641 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0086926 2021-02-10T04:52:37.641 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.641 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.641 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.641 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:40.771247+0000", 2021-02-10T04:52:37.641 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.9180000000000006e-06 2021-02-10T04:52:37.641 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.641 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.641 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:52:37.641 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:40.771257+0000", 2021-02-10T04:52:37.641 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00075927399999999999 2021-02-10T04:52:37.641 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.641 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.642 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:52:37.642 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:40.772017+0000", 2021-02-10T04:52:37.642 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010475346 2021-02-10T04:52:37.642 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.642 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:52:37.642 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.642 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.642 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.642 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.15 1038/1037 [PushOp(7:adfc1887:::smithi08575397-25:33e, version: 961'409, data_included: [1486901~894536,2801290~154040], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:adfc1887:::smithi08575397-25:33e@961'409, size: 3311276, copy_subset: [0~3311276], clone_subset: {7:adfc1887:::smithi08575397-25:328=[]}, snapset: 374=[]:{218=[203],27c=[27c],2cb=[2c8,2ae],2ff=[2f5],328=[328],335=[335,332,32f,32a],33e=[33e,33c,33a],343=[343],374=[374,373,372,370,36f,36b,368,366,35f,354,353,34e,34d,34b,349,345,344]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2955330, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1486901, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:52:37.642 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:45:41.084420+0000", 2021-02-10T04:52:37.642 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 416.48381209600001, 2021-02-10T04:52:37.642 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010577269, 2021-02-10T04:52:37.643 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:52:37.643 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:52:37.643 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:52:37.643 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.643 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:52:37.643 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.084420+0000", 2021-02-10T04:52:37.643 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:52:37.643 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.643 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.643 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:52:37.643 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.084420+0000", 2021-02-10T04:52:37.643 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000988224 2021-02-10T04:52:37.643 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.644 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.644 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:52:37.644 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.085409+0000", 2021-02-10T04:52:37.644 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4181000000000001e-05 2021-02-10T04:52:37.644 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.644 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.644 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:52:37.644 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.085433+0000", 2021-02-10T04:52:37.644 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.2017000000000002e-05 2021-02-10T04:52:37.644 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.644 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.644 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:52:37.644 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.085465+0000", 2021-02-10T04:52:37.644 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6639000000000001e-05 2021-02-10T04:52:37.645 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.645 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.645 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:52:37.645 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.085481+0000", 2021-02-10T04:52:37.645 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0079138650000000008 2021-02-10T04:52:37.645 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.645 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.645 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.645 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.093395+0000", 2021-02-10T04:52:37.645 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2374999999999999e-05 2021-02-10T04:52:37.645 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.645 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.646 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:52:37.646 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.093408+0000", 2021-02-10T04:52:37.646 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001589968 2021-02-10T04:52:37.646 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.646 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.646 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:52:37.646 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.094998+0000", 2021-02-10T04:52:37.646 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010577269 2021-02-10T04:52:37.646 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.646 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:52:37.646 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.646 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.647 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.647 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.15 1038/1037 [PushOp(7:adfc1887:::smithi08575397-25:2ff, version: 995'435, data_included: [2154604~68996,2888554~512829], data_size: 581825, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:adfc1887:::smithi08575397-25:2ff@995'435, size: 3496120, copy_subset: [0~3496120], clone_subset: {7:adfc1887:::smithi08575397-25:2cb=[],7:adfc1887:::smithi08575397-25:328=[]}, snapset: 374=[]:{218=[203],27c=[27c],2cb=[2c8,2ae],2ff=[2f5],328=[328],335=[335,332,32f,32a],33e=[33e,33c,33a],343=[343],374=[374,373,372,370,36f,36b,368,366,35f,354,353,34e,34d,34b,349,345,344]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3496120, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2154604, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:52:37.647 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:45:41.177698+0000", 2021-02-10T04:52:37.647 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 416.39053470200002, 2021-02-10T04:52:37.647 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.013292897, 2021-02-10T04:52:37.647 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:52:37.647 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:52:37.647 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:52:37.647 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.647 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:52:37.647 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.177698+0000", 2021-02-10T04:52:37.647 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:52:37.647 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.647 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.648 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:52:37.648 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.177698+0000", 2021-02-10T04:52:37.648 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00050759299999999998 2021-02-10T04:52:37.648 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.648 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.648 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:52:37.648 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.178205+0000", 2021-02-10T04:52:37.648 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.7773999999999999e-05 2021-02-10T04:52:37.648 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.648 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.648 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:52:37.648 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.178233+0000", 2021-02-10T04:52:37.648 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2228999999999998e-05 2021-02-10T04:52:37.649 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.649 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.649 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:52:37.649 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.178275+0000", 2021-02-10T04:52:37.649 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9355e-05 2021-02-10T04:52:37.649 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.649 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.649 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:52:37.649 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.178295+0000", 2021-02-10T04:52:37.649 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0086878040000000004 2021-02-10T04:52:37.649 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.649 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.649 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.649 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.186982+0000", 2021-02-10T04:52:37.650 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.8190000000000003e-06 2021-02-10T04:52:37.650 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.650 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.650 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:52:37.650 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.186992+0000", 2021-02-10T04:52:37.650 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0039983229999999998 2021-02-10T04:52:37.650 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.650 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.650 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:52:37.650 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.190991+0000", 2021-02-10T04:52:37.650 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.013292897 2021-02-10T04:52:37.650 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.650 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:52:37.651 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.651 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.651 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.651 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.15 1038/1037 [PushOp(7:adfc1887:::smithi08575397-25:343, version: 998'441, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:adfc1887:::smithi08575397-25:343@998'441, size: 3529767, copy_subset: [0~3529767], clone_subset: {7:adfc1887:::smithi08575397-25:33e=[]}, snapset: 374=[]:{218=[203],27c=[27c],2cb=[2c8,2ae],2ff=[2f5],328=[328],335=[335,332,32f,32a],33e=[33e,33c,33a],343=[343],374=[374,373,372,370,36f,36b,368,366,35f,354,353,34e,34d,34b,349,345,344]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:52:37.651 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:45:41.284850+0000", 2021-02-10T04:52:37.651 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 416.283382808, 2021-02-10T04:52:37.651 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0099639719999999998, 2021-02-10T04:52:37.651 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:52:37.651 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:52:37.651 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:52:37.651 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.651 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:52:37.651 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.284850+0000", 2021-02-10T04:52:37.652 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:52:37.652 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.652 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.652 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:52:37.652 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.284850+0000", 2021-02-10T04:52:37.652 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00104846 2021-02-10T04:52:37.652 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.652 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.652 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:52:37.652 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.285898+0000", 2021-02-10T04:52:37.652 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8198999999999999e-05 2021-02-10T04:52:37.652 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.652 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.653 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:52:37.653 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.285926+0000", 2021-02-10T04:52:37.653 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.4586999999999998e-05 2021-02-10T04:52:37.653 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.653 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.653 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:52:37.653 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.285971+0000", 2021-02-10T04:52:37.653 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6572e-05 2021-02-10T04:52:37.653 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.653 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.653 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:52:37.653 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.285987+0000", 2021-02-10T04:52:37.653 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0081604750000000004 2021-02-10T04:52:37.654 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.654 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.654 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.654 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.294148+0000", 2021-02-10T04:52:37.654 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1707e-05 2021-02-10T04:52:37.654 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.654 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.654 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:52:37.654 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.294160+0000", 2021-02-10T04:52:37.654 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00065397199999999995 2021-02-10T04:52:37.654 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.654 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.655 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:52:37.655 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.294814+0000", 2021-02-10T04:52:37.655 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0099639719999999998 2021-02-10T04:52:37.655 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.655 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:52:37.655 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.655 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.655 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.655 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.15 1038/1037 [PushOp(7:adfc1887:::smithi08575397-25:374, version: 1000'443, data_included: [2051410~13328,2496305~525139,3472104~292961], data_size: 831428, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:adfc1887:::smithi08575397-25:374@1000'443, size: 3765065, copy_subset: [565065~427465,1430299~634439,2496305~525139,3472104~292961], clone_subset: {7:adfc1887:::smithi08575397-25:343=[0~565065,992530~437769,2064738~431567,3021444~450660]}, snapset: 374=[]:{218=[203],27c=[27c],2cb=[2c8,2ae],2ff=[2f5],328=[328],335=[335,332,32f,32a],33e=[33e,33c,33a],343=[343],374=[374,373,372,370,36f,36b,368,366,35f,354,353,34e,34d,34b,349,345,344]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3765065, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2051410, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:52:37.655 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:45:41.320627+0000", 2021-02-10T04:52:37.655 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 416.24760579100001, 2021-02-10T04:52:37.655 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.013224167, 2021-02-10T04:52:37.656 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:52:37.656 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:52:37.656 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:52:37.656 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.656 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:52:37.656 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.320627+0000", 2021-02-10T04:52:37.656 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:52:37.656 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.656 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.656 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:52:37.656 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.320627+0000", 2021-02-10T04:52:37.657 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000743616 2021-02-10T04:52:37.657 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.657 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.657 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:52:37.657 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.321370+0000", 2021-02-10T04:52:37.657 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.4378999999999997e-05 2021-02-10T04:52:37.657 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.657 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.657 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:52:37.657 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.321405+0000", 2021-02-10T04:52:37.657 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0112000000000003e-05 2021-02-10T04:52:37.657 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.657 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.657 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:52:37.658 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.321465+0000", 2021-02-10T04:52:37.658 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2058000000000001e-05 2021-02-10T04:52:37.658 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.658 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.658 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:52:37.658 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.321487+0000", 2021-02-10T04:52:37.658 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0083593250000000008 2021-02-10T04:52:37.658 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.658 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.658 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.658 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.329846+0000", 2021-02-10T04:52:37.658 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1922999999999999e-05 2021-02-10T04:52:37.658 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.659 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.659 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:52:37.659 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.329858+0000", 2021-02-10T04:52:37.659 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0039927540000000003 2021-02-10T04:52:37.659 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.659 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.659 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:52:37.659 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.333851+0000", 2021-02-10T04:52:37.659 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.013224167 2021-02-10T04:52:37.659 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.659 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:52:37.659 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.659 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.660 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.660 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPushReply(7.15 1121/1119 [PushReplyOp(7:adfc1887:::smithi08575397-25:33e)])", 2021-02-10T04:52:37.660 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:51:38.576167+0000", 2021-02-10T04:52:37.660 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 58.992065814, 2021-02-10T04:52:37.660 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010982059000000001, 2021-02-10T04:52:37.660 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:52:37.660 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T04:52:37.660 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:52:37.660 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.660 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:52:37.660 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.576167+0000", 2021-02-10T04:52:37.660 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:52:37.660 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.661 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.661 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:52:37.661 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.576167+0000", 2021-02-10T04:52:37.661 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.3866999999999999e-05 2021-02-10T04:52:37.661 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.661 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.661 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:52:37.661 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.576200+0000", 2021-02-10T04:52:37.661 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7049999999999999e-06 2021-02-10T04:52:37.661 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.662 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.662 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:52:37.662 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.576208+0000", 2021-02-10T04:52:37.662 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1736000000000001e-05 2021-02-10T04:52:37.662 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.662 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.662 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:52:37.662 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.576220+0000", 2021-02-10T04:52:37.662 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.1890000000000004e-06 2021-02-10T04:52:37.662 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.662 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.662 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:52:37.662 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.576226+0000", 2021-02-10T04:52:37.663 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0097831849999999998 2021-02-10T04:52:37.663 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.663 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.663 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.663 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.586009+0000", 2021-02-10T04:52:37.663 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001139377 2021-02-10T04:52:37.663 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.663 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.663 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:52:37.663 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.587149+0000", 2021-02-10T04:52:37.663 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010982059000000001 2021-02-10T04:52:37.663 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.664 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:52:37.664 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.664 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.664 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.664 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPushReply(7.15 1121/1119 [PushReplyOp(7:adfc1887:::smithi08575397-25:2cb)])", 2021-02-10T04:52:37.664 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:51:38.601205+0000", 2021-02-10T04:52:37.664 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 58.967027825000002, 2021-02-10T04:52:37.664 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0090249059999999992, 2021-02-10T04:52:37.664 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:52:37.664 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T04:52:37.664 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:52:37.664 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.665 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:52:37.665 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.601205+0000", 2021-02-10T04:52:37.665 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:52:37.665 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.665 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.665 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:52:37.665 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.601205+0000", 2021-02-10T04:52:37.665 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.6381999999999998e-05 2021-02-10T04:52:37.665 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.665 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.666 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:52:37.666 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.601241+0000", 2021-02-10T04:52:37.666 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.4120000000000001e-06 2021-02-10T04:52:37.666 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.666 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.666 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:52:37.666 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.601249+0000", 2021-02-10T04:52:37.666 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3551e-05 2021-02-10T04:52:37.666 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.666 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.667 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:52:37.667 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.601263+0000", 2021-02-10T04:52:37.667 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.3390000000000004e-06 2021-02-10T04:52:37.667 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.667 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.667 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:52:37.667 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.601270+0000", 2021-02-10T04:52:37.667 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0085583159999999998 2021-02-10T04:52:37.667 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.667 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.667 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.667 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.609829+0000", 2021-02-10T04:52:37.668 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00040090600000000003 2021-02-10T04:52:37.668 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.668 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.668 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:52:37.668 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.610229+0000", 2021-02-10T04:52:37.668 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0090249059999999992 2021-02-10T04:52:37.668 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.668 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:52:37.668 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.668 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.668 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.668 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPushReply(7.15 1121/1119 [PushReplyOp(7:adfc1887:::smithi08575397-25:343)])", 2021-02-10T04:52:37.668 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:51:38.760603+0000", 2021-02-10T04:52:37.669 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 58.807629528, 2021-02-10T04:52:37.669 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0094503050000000009, 2021-02-10T04:52:37.669 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:52:37.669 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T04:52:37.669 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:52:37.669 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.669 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:52:37.669 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.760603+0000", 2021-02-10T04:52:37.669 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:52:37.669 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.669 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.669 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:52:37.669 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.760603+0000", 2021-02-10T04:52:37.669 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1231e-05 2021-02-10T04:52:37.670 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.670 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.670 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:52:37.670 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.760634+0000", 2021-02-10T04:52:37.670 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7470000000000005e-06 2021-02-10T04:52:37.670 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.670 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.670 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:52:37.670 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.760642+0000", 2021-02-10T04:52:37.670 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2286000000000001e-05 2021-02-10T04:52:37.670 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.670 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.671 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:52:37.671 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.760654+0000", 2021-02-10T04:52:37.671 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.4500000000000001e-06 2021-02-10T04:52:37.671 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.671 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.671 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:52:37.671 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.760661+0000", 2021-02-10T04:52:37.671 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0090138119999999995 2021-02-10T04:52:37.671 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.671 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.671 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.671 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.769674+0000", 2021-02-10T04:52:37.671 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00037877899999999999 2021-02-10T04:52:37.671 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.672 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.672 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:52:37.672 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.770053+0000", 2021-02-10T04:52:37.672 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0094503050000000009 2021-02-10T04:52:37.672 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.672 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:52:37.672 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.672 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.673 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.673 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.24373.0:3511 5.e 5:7759931f:::notify.3:head [watch ping cookie 94187770077184] snapc 0=[] ondisk+write+known_if_redirected e1128)", 2021-02-10T04:52:37.673 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:51:52.996715+0000", 2021-02-10T04:52:37.673 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 44.571517610999997, 2021-02-10T04:52:37.673 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00044475200000000003, 2021-02-10T04:52:37.673 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:52:37.673 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:52:37.673 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:52:37.673 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.24373", 2021-02-10T04:52:37.673 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:52:37.673 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 3511 2021-02-10T04:52:37.673 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.673 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:52:37.674 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.674 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:52:37.674 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:52.996715+0000", 2021-02-10T04:52:37.674 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:52:37.674 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.674 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.674 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:52:37.674 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:52.996715+0000", 2021-02-10T04:52:37.674 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.0203000000000001e-05 2021-02-10T04:52:37.674 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.674 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.674 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:52:37.674 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:52.996755+0000", 2021-02-10T04:52:37.675 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.1379999999999996e-06 2021-02-10T04:52:37.675 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.675 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.675 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:52:37.675 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:52.996764+0000", 2021-02-10T04:52:37.675 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.507e-05 2021-02-10T04:52:37.675 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.675 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.675 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:52:37.675 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:52.996779+0000", 2021-02-10T04:52:37.675 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.9789999999999999e-06 2021-02-10T04:52:37.675 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.675 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.676 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:52:37.676 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:52.996788+0000", 2021-02-10T04:52:37.676 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00017783799999999999 2021-02-10T04:52:37.676 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.676 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.676 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:52:37.676 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:52.996966+0000", 2021-02-10T04:52:37.676 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.5011000000000002e-05 2021-02-10T04:52:37.676 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.676 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.676 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:52:37.676 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:52.997061+0000", 2021-02-10T04:52:37.676 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.8512999999999997e-05 2021-02-10T04:52:37.676 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:52:37.677 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:52:37.677 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:52:37.677 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:52.997160+0000", 2021-02-10T04:52:37.677 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00044475200000000003 2021-02-10T04:52:37.677 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.677 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:52:37.677 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.677 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:52:37.677 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:52:37.677 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:52:38.090 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:52:38.297 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:52:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:38 smithi164 conmon[29288]: cluster 2021-02-10T04:52:36.163991+0000 osd.2 2021-02-10T04:52:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:38 smithi164 conmon[29288]: (osd.2) 2003 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:38 smithi164 conmon[29288]: cluster 2021-02-10T04:52:36.163999+0000 osd.2 (osd.2) 2004 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:38 smithi164 conmon[29288]: cluster 2021-02-10T04:52:36.435186+0000 mgr.y (mgr.14140) 1253 : cluster [DBG] pgmap v2069: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 895 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 286 MiB/s rd, 80 MiB/s wr, 498 op/s; 278/2232 objects degraded (12.455%); 25/2232 objects misplaced (1.120%); 14 MiB/s, 25 objects/s recovering 2021-02-10T04:52:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:38 smithi164 conmon[29288]: cluster 2021-02-10T04:52:36.777481+0000 osd.4 (osd.4) 2638 : 2021-02-10T04:52:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:38 smithi164 conmon[29288]: cluster [DBG] 6.1a deep-scrub ok 2021-02-10T04:52:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:38 smithi164 conmon[29288]: cluster 2021-02-10T04:52:37.071130+0000 mon.a ( 2021-02-10T04:52:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:38 smithi164 conmon[29288]: mon.0) 2428 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 59 sec, osd.2 has slow ops (SLOW_OPS) 2021-02-10T04:52:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:38 smithi164 conmon[29288]: cluster 2021-02-10T04:52: 2021-02-10T04:52:38.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:38 smithi164 conmon[29288]: 37.071166+0000 mon.a (mon.0) 2429 : cluster 2021-02-10T04:52:38.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:38 smithi164 conmon[29288]: [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2021-02-10T04:52:38.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:38 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:52:38.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:38 smithi164 conmon[29288]: -02-10T04:52: 2021-02-10T04:52:38.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:38 smithi164 conmon[29288]: 37.142468+0000 osd.2 (osd. 2021-02-10T04:52:38.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:38 smithi164 conmon[29288]: 2) 2005 : cluster [WRN] 2021-02-10T04:52:38.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:38 smithi164 conmon[29288]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:38.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:38 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:52:38.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:38 smithi164 conmon[29288]: -02-10T04:52: 2021-02-10T04:52:38.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:38 smithi164 conmon[29288]: 37.142473+0000 osd.2 (osd. 2021-02-10T04:52:38.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:38 smithi164 conmon[29288]: 2) 2006 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:38.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:38 smithi164 conmon[29288]: 2021-02-10T04:52:38.407 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set noscrub 2021-02-10T04:52:38.429 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[49106]: debug 2021-02-10T04:52:38.186+0000 7f77a3e94700 -1 osd.2 1150 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) 2021-02-10T04:52:38.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[35961]: cluster 2021-02-10T04:52:36.163991+0000 osd.2 (osd.2) 2003 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:38.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[35961]: cluster 2021-02-10T04:52:36.163999+0000 osd.2 (osd.2) 2004 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:38.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[35961]: cluster 2021-02-10T04:52:36.435186+0000 mgr.y (mgr.14140) 1253 : cluster [DBG] pgmap v2069: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 895 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 286 MiB/s rd, 80 MiB/s wr, 498 op/s; 278/2232 objects degraded (12.455%); 25/2232 objects misplaced (1.120%); 14 MiB/s, 25 objects/s recovering 2021-02-10T04:52:38.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[35961]: cluster 2021-02-10T04:52:36.777481+0000 osd.4 (osd.4) 2638 : cluster [DBG] 6.1a deep-scrub ok 2021-02-10T04:52:38.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[35961]: cluster 2021-02-10T04:52:37.071130+0000 mon.a (mon.0) 2428 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 59 sec, osd.2 has slow ops (SLOW_OPS) 2021-02-10T04:52:38.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[35961]: cluster 2021-02-10T04:52:37.071166+0000 mon.a (mon.0) 2429 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2021-02-10T04:52:38.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[35961]: cluster 2021-02-10T04:52:37.142468+0000 osd.2 (osd.2) 2005 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:38.430 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[35961]: cluster 2021-02-10T04:52:37.142473+0000 osd.2 (osd.2) 2006 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:38.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[29319]: cluster 2021-02-10T04:52:36.163991+0000 2021-02-10T04:52:38.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[29319]: osd.2 (osd.2) 2003 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:38.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[29319]: cluster 2021-02-10T04:52:36.163999+0000 osd.2 (osd.2) 2004 2021-02-10T04:52:38.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[29319]: : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:38.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:52:38.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[29319]: -02-10T04:52:36.435186+0000 mgr.y (mgr.14140) 1253 2021-02-10T04:52:38.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[29319]: : cluster [DBG] pgmap v2069: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 895 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 286 MiB/s rd, 80 MiB/s wr, 498 op/s; 278/2232 objects degraded (12.455%); 25/2232 objects misplaced (1.120%); 14 MiB/s, 25 objects/s recovering 2021-02-10T04:52:38.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:52:38.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[29319]: 04:52:36.777481+0000 osd.4 (osd.4 2021-02-10T04:52:38.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[29319]: ) 2638 : cluster [DBG] 6.1a deep-scrub ok 2021-02-10T04:52:38.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:52:38.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[29319]: 02-10T04:52:37.071130+0000 mon.a 2021-02-10T04:52:38.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[29319]: (mon.0) 2428 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 59 sec, osd.2 has slow ops (SLOW_OPS) 2021-02-10T04:52:38.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:52:38.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[29319]: 02-10T04:52:37.071166+0000 mon.a (mon 2021-02-10T04:52:38.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[29319]: .0) 2429 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2021-02-10T04:52:38.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[29319]: cluster 2021-02-10T04:52:38.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[29319]: 2021-02-10T04:52:37.142468+0000 2021-02-10T04:52:38.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[29319]: osd.2 (osd.2) 2005 : cluster [WRN] 2021-02-10T04:52:38.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:38.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[29319]: cluster 2021-02-10T04:52 2021-02-10T04:52:38.432 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[29319]: :37.142473+0000 osd.2 (osd.2) 2021-02-10T04:52:38.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:38 smithi085 conmon[29319]: 2006 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:38.491 DEBUG:teuthology.orchestra.run.smithi085:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.0.service' 2021-02-10T04:52:38.598 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-02-10T04:52:38.599 INFO:tasks.cephadm.osd.0:Stopped osd.0 2021-02-10T04:52:38.599 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.0 2021-02-10T04:52:38.599 INFO:tasks.thrashosds.thrasher:remote for osd 0 is ubuntu@smithi085.front.sepia.ceph.com 2021-02-10T04:52:38.599 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -n osd.0 --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op list-pgs 2021-02-10T04:52:38.651 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:52:38.878 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/osd.0/config 2021-02-10T04:52:39.230 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[49106]: debug 2021-02-10T04:52:39.188+0000 7f77a3e94700 -1 osd.2 1150 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) 2021-02-10T04:52:39.230 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[29319]: cluster 2021-02-10T04:52:37.324036+0000 osd.5 (osd.5) 2021-02-10T04:52:39.230 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[29319]: 173 : cluster [DBG] 2.3 scrub ok 2021-02-10T04:52:39.230 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[29319]: cluster 2021-02-10T04:52:38.188543+0000 osd.2 (osd.2) 2007 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:39.230 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[29319]: cluster 2021-02-10T04:52:38.188553+0000 osd.2 (osd.2) 2008 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:39.230 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[29319]: audit 2021-02-10T04:52:38.214472+0000 mon.a (mon.0) 2430 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:52:39.230 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[29319]: audit 2021-02-10T04:52:38.247658+0000 mon.a (mon.0) 2431 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:52:39.230 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[29319]: cluster 2021-02-10T04:52:38.278903+0000 osd.1 (osd.1) 490 : cluster [DBG] 6.1 deep-scrub ok 2021-02-10T04:52:39.231 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[29319]: cluster 2021-02-10T04:52:38.496642+0000 mon.a (mon.0) 2432 : cluster [WRN] Health check update: Degraded data redundancy: 278/2232 objects degraded (12.455%), 29 pgs degraded (PG_DEGRADED) 2021-02-10T04:52:39.231 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[35961]: cluster 2021-02-10T04:52:37.324036+0000 osd.5 (osd 2021-02-10T04:52:39.231 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[35961]: .5) 173 : cluster [DBG] 2.3 scrub ok 2021-02-10T04:52:39.231 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:52:39.231 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[35961]: T04:52:38.188543+0000 osd.2 (osd.2) 2007 : 2021-02-10T04:52:39.231 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[35961]: cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:39.231 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:52:39.231 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[35961]: 52:38.188553+0000 osd.2 (osd.2 2021-02-10T04:52:39.231 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[35961]: ) 2008 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:39.232 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:52:39.232 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[35961]: -10T04:52:38.214472+0000 mon.a (mon. 2021-02-10T04:52:39.232 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[35961]: 0) 2430 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:52:39.232 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:52:39.232 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[35961]: 10T04:52:38.247658+0000 mon.a 2021-02-10T04:52:39.232 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[35961]: (mon.0) 2431 : audit 2021-02-10T04:52:39.232 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:52:39.232 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:52:39.232 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[35961]: -02-10T04:52:38.278903 2021-02-10T04:52:39.232 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 490 : cluster [DBG] 2021-02-10T04:52:39.232 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[35961]: 6.1 deep-scrub ok 2021-02-10T04:52:39.232 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[35961]: cluster 2021-02-10T04:52: 2021-02-10T04:52:39.232 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[35961]: 38.496642+0000 mon.a (mon.0) 2432 : cluster 2021-02-10T04:52:39.233 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:39 smithi085 conmon[35961]: [WRN] Health check update: Degraded data redundancy: 278/2232 objects degraded (12.455%), 29 pgs degraded (PG_DEGRADED) 2021-02-10T04:52:39.394 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:52:39 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:52:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:39 smithi164 conmon[29288]: cluster 2021-02-10T04:52:37.324036+0000 osd.5 (osd.5) 173 : 2021-02-10T04:52:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:39 smithi164 conmon[29288]: cluster [DBG] 2.3 scrub ok 2021-02-10T04:52:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:39 smithi164 conmon[29288]: cluster 2021-02-10T04:52:38.188543+0000 osd.2 (osd.2) 2007 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:39 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:52:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:39 smithi164 conmon[29288]: T04:52:38.188553+0000 osd.2 (osd.2) 2008 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:39.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:39 smithi164 conmon[29288]: audit 2021-02-10T04:52:38.214472+0000 mon.a (mon.0) 2430 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:52:39.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:39 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:52:39.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:39 smithi164 conmon[29288]: T04:52:38.247658+0000 mon.a ( 2021-02-10T04:52:39.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:39 smithi164 conmon[29288]: mon.0) 2431 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:52:39.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:39 smithi164 conmon[29288]: cluster 2021-02-10T04:52:39.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:39 smithi164 conmon[29288]: 2021-02-10T04:52: 2021-02-10T04:52:39.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:39 smithi164 conmon[29288]: 38.278903+0000 osd.1 (osd.1) 490 : 2021-02-10T04:52:39.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:39 smithi164 conmon[29288]: cluster [DBG] 6.1 deep-scrub ok 2021-02-10T04:52:39.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:39 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:52:39.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:39 smithi164 conmon[29288]: T04:52:38.496642+0000 mon.a ( 2021-02-10T04:52:39.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:39 smithi164 conmon[29288]: mon.0) 2432 : cluster [WRN] Health check update: Degraded data redundancy: 278/2232 objects degraded (12.455%), 29 pgs degraded (PG_DEGRADED) 2021-02-10T04:52:40.084 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:40 smithi085 conmon[29319]: cluster 2021-02-10T04:52:38.435955+0000 mgr.y (mgr 2021-02-10T04:52:40.084 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:40 smithi085 conmon[29319]: .14140) 1254 : cluster [DBG] pgmap v2070: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 895 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 241 MiB/s rd, 68 MiB/s wr, 421 op/s; 278/2232 objects degraded (12.455%); 25/2232 objects misplaced (1.120%); 18 MiB/s, 36 objects/s recovering 2021-02-10T04:52:40.084 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:40 smithi085 conmon[29319]: audit 2021-02-10T04:52:39.143996+0000 mgr.y (mgr.14140) 1255 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:52:40.084 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:40 smithi085 conmon[29319]: cluster 2021-02-10T04:52:39.190395+0000 osd.2 (osd.2) 2009 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:40.084 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:40 smithi085 conmon[29319]: cluster 2021-02-10T04:52:39.190401+0000 osd.2 (osd.2) 2010 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:40.085 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:40 smithi085 conmon[35961]: cluster 2021-02-10T04:52:38.435955+0000 2021-02-10T04:52:40.085 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:40 smithi085 conmon[35961]: mgr.y (mgr.14140) 1254 : cluster [DBG] pgmap v2070: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 895 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 241 MiB/s rd, 68 MiB/s wr, 421 op/s; 278/2232 objects degraded (12.455%); 25/2232 objects misplaced (1.120%); 18 MiB/s, 36 objects/s recovering 2021-02-10T04:52:40.085 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:40 smithi085 conmon[35961]: audit 2021-02-10T04:52:40.085 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:40 smithi085 conmon[35961]: 2021-02-10T04:52:39.143996+0000 mgr.y (mgr.14140) 1255 : 2021-02-10T04:52:40.085 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:40 smithi085 conmon[35961]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:52:40.085 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:40 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:52:40.085 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:40 smithi085 conmon[35961]: T04:52:39.190395+0000 osd.2 (osd 2021-02-10T04:52:40.085 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:40 smithi085 conmon[35961]: .2) 2009 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:40.085 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:40 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:52:40.085 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:40 smithi085 conmon[35961]: 10T04:52:39.190401+0000 osd.2 (osd 2021-02-10T04:52:40.085 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:40 smithi085 conmon[35961]: .2) 2010 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:40.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:40 smithi164 conmon[29288]: cluster 2021-02-10T04:52:38.435955+0000 mgr.y 2021-02-10T04:52:40.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:40 smithi164 conmon[29288]: (mgr.14140) 1254 : cluster [DBG] pgmap v2070: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 895 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 241 MiB/s rd, 68 MiB/s wr, 421 op/s; 278/2232 objects degraded (12.455%); 25/2232 objects misplaced (1.120%); 18 MiB/s, 36 objects/s recovering 2021-02-10T04:52:40.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:40 smithi164 conmon[29288]: audit 2021-02-10T04:52:39.143996+0000 2021-02-10T04:52:40.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:40 smithi164 conmon[29288]: mgr.y (mgr.14140) 1255 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:52:40.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:40 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:52:40.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:40 smithi164 conmon[29288]: :52:39.190395+0000 osd.2 (osd.2) 2009 2021-02-10T04:52:40.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:40 smithi164 conmon[29288]: : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:40.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:40 smithi164 conmon[29288]: cluster 2021-02-10T04:52 2021-02-10T04:52:40.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:40 smithi164 conmon[29288]: :39.190401+0000 osd.2 (osd.2) 2010 : cluster 2021-02-10T04:52:40.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:40 smithi164 conmon[29288]: [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:40.399 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:52:40 smithi085 conmon[49106]: debug 2021-02-10T04:52:40.157+0000 7f77a3e94700 -1 osd.2 1150 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) 2021-02-10T04:52:41.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:41 smithi164 conmon[29288]: cluster 2021-02-10T04:52:39.695228+0000 osd.4 (osd.4) 2639 : cluster 2021-02-10T04:52:41.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:41 smithi164 conmon[29288]: [DBG] 4.13 scrub ok 2021-02-10T04:52:41.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:41 smithi164 conmon[29288]: cluster 2021-02-10T04:52:40.038549+0000 osd.3 (osd.3) 19 : cluster [DBG] 5.14 scrub ok 2021-02-10T04:52:41.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:41 smithi164 conmon[29288]: cluster 2021-02-10T04:52:40.066347+0000 2021-02-10T04:52:41.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:41 smithi164 conmon[29288]: osd.6 (osd.6) 171 : cluster [DBG] 6.5 scrub ok 2021-02-10T04:52:41.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:41 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:52:41.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:41 smithi164 conmon[29288]: :52:40.159518+0000 osd.2 (osd.2) 2011 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:41.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:41 smithi164 conmon[29288]: cluster 2021-02-10T04:52:40.159526+0000 osd.2 ( 2021-02-10T04:52:41.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:41 smithi164 conmon[29288]: osd.2) 2012 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:41.434 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:52:41 smithi085 conmon[49106]: debug 2021-02-10T04:52:41.157+0000 7f77a3e94700 -1 osd.2 1150 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) 2021-02-10T04:52:41.434 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:41 smithi085 conmon[29319]: cluster 2021-02-10T04:52:39.695228 2021-02-10T04:52:41.434 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:41 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2639 : cluster [DBG] 4.13 scrub ok 2021-02-10T04:52:41.434 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:41 smithi085 conmon[29319]: cluster 2021-02-10T04:52:40.038549+0000 osd.3 (osd.3) 19 : cluster [DBG] 5.14 scrub ok 2021-02-10T04:52:41.435 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:41 smithi085 conmon[29319]: cluster 2021-02-10T04:52:40.066347+0000 osd.6 (osd.6) 171 : cluster [DBG] 6.5 scrub ok 2021-02-10T04:52:41.435 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:41 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:52:41.435 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:41 smithi085 conmon[29319]: -10T04:52:40.159518 2021-02-10T04:52:41.435 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:41 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 2011 : cluster [WRN] 2021-02-10T04:52:41.435 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:41 smithi085 conmon[29319]: slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:41.435 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:41 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:52:41.435 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:41 smithi085 conmon[29319]: :52:40.159526+0000 osd.2 (osd.2) 2021-02-10T04:52:41.435 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:41 smithi085 conmon[29319]: 2012 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:41.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:41 smithi085 conmon[35961]: cluster 2021-02-10T04:52:39.695228+0000 osd.4 2021-02-10T04:52:41.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:41 smithi085 conmon[35961]: (osd.4) 2639 : cluster [DBG] 4.13 scrub ok 2021-02-10T04:52:41.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:41 smithi085 conmon[35961]: cluster 2021-02-10T04:52:40.038549+0000 osd.3 (osd 2021-02-10T04:52:41.435 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:41 smithi085 conmon[35961]: .3) 19 : cluster [DBG] 5.14 scrub ok 2021-02-10T04:52:41.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:41 smithi085 conmon[35961]: cluster 2021-02-10T04:52:41.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:41 smithi085 conmon[35961]: 2021-02-10T04:52:40.066347 2021-02-10T04:52:41.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:41 smithi085 conmon[35961]: +0000 osd.6 (osd.6) 171 : cluster [DBG] 2021-02-10T04:52:41.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:41 smithi085 conmon[35961]: 6.5 scrub ok 2021-02-10T04:52:41.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:41 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:52:41.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:41 smithi085 conmon[35961]: T04:52:40.159518+0000 osd.2 2021-02-10T04:52:41.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:41 smithi085 conmon[35961]: (osd.2) 2011 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:41.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:41 smithi085 conmon[35961]: 2021-02-10T04:52:41.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:41 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:52:41.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:41 smithi085 conmon[35961]: -02-10T04:52:40.159526+0000 osd.2 2021-02-10T04:52:41.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:41 smithi085 conmon[35961]: (osd.2) 2012 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:41.436 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:41 smithi085 conmon[35961]: 2021-02-10T04:52:41.708 INFO:tasks.rados.rados.0.smithi085.stdout:685: finishing copy_from to smithi08575397-39 2021-02-10T04:52:41.708 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 39 v 158 (ObjNum 229 snap 51 seq_num 229) dirty exists 2021-02-10T04:52:41.708 INFO:tasks.rados.rados.0.smithi085.stdout:685: finishing copy_from racing read to smithi08575397-39 2021-02-10T04:52:41.708 INFO:tasks.rados.rados.0.smithi085.stdout:685: done (1 left) 2021-02-10T04:52:41.708 INFO:tasks.rados.rados.0.smithi085.stdout: waiting on 1 2021-02-10T04:52:41.770 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.6 dump_ops_in_flight 2021-02-10T04:52:42.071 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:52:42.089 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:42 smithi164 conmon[29288]: cluster 2021-02-10T04:52:40.436679+0000 mgr.y (mgr. 2021-02-10T04:52:42.089 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:42 smithi164 conmon[29288]: 14140) 1256 : cluster [DBG] pgmap v2071: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 895 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 133 MiB/s rd, 34 MiB/s wr, 220 op/s; 278/2232 objects degraded (12.455%); 25/2232 objects misplaced (1.120%); 12 MiB/s, 22 objects/s recovering 2021-02-10T04:52:42.089 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:42 smithi164 conmon[29288]: cluster 2021-02-10T04:52:41.041546+0000 osd.3 (osd.3) 20 : cluster [DBG] 1.9 scrub ok 2021-02-10T04:52:42.089 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:42 smithi164 conmon[29288]: cluster 2021-02-10T04:52:41.059156+0000 osd.6 (osd.6) 172 : cluster [DBG] 2.11 scrub ok 2021-02-10T04:52:42.089 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:42 smithi164 conmon[29288]: cluster 2021-02-10T04:52:41.159078+0000 osd.2 (osd.2) 2013 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:42.090 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:42 smithi164 conmon[29288]: cluster 2021-02-10T04:52:41.159091+0000 osd.2 (osd.2) 2014 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:42.291 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:42 smithi085 conmon[29319]: cluster 2021-02-10T04:52:40.436679+0000 mgr.y (mgr.14140) 1256 2021-02-10T04:52:42.291 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:42 smithi085 conmon[29319]: : cluster [DBG] pgmap v2071: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 895 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 133 MiB/s rd, 34 MiB/s wr, 220 op/s; 278/2232 objects degraded (12.455%); 25/2232 objects misplaced (1.120%); 12 MiB/s, 22 objects/s recovering 2021-02-10T04:52:42.291 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:42 smithi085 conmon[29319]: cluster 2021-02-10T04:52:41.041546+0000 osd.3 (osd.3) 20 : cluster [DBG] 1.9 scrub ok 2021-02-10T04:52:42.291 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:42 smithi085 conmon[29319]: cluster 2021-02-10T04:52:41.059156+0000 osd.6 (osd.6) 172 : cluster [DBG] 2.11 scrub ok 2021-02-10T04:52:42.291 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:42 smithi085 conmon[29319]: cluster 2021-02-10T04:52:41.159078+0000 osd.2 (osd.2) 2013 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:42.291 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:42 smithi085 conmon[29319]: cluster 2021-02-10T04:52:41.159091+0000 osd.2 (osd.2) 2014 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:42.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:42 smithi085 conmon[35961]: cluster 2021-02-10T04:52:40 2021-02-10T04:52:42.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:42 smithi085 conmon[35961]: .436679+0000 mgr.y (mgr.14140) 1256 : cluster [DBG] pgmap v2071: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 895 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 133 MiB/s rd, 34 MiB/s wr, 220 op/s; 278/2232 objects degraded (12.455%); 25/2232 objects misplaced (1.120%); 12 MiB/s, 22 objects/s recovering 2021-02-10T04:52:42.291 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:42 smithi085 conmon[35961]: cluster 2021-02-10T04:52:41.041546+0000 osd.3 (osd.3) 20 : cluster [DBG] 1.9 scrub ok 2021-02-10T04:52:42.292 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:42 smithi085 conmon[35961]: cluster 2021-02-10T04:52:41.059156+0000 osd.6 (osd.6) 172 : cluster [DBG] 2.11 scrub ok 2021-02-10T04:52:42.292 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:42 smithi085 conmon[35961]: cluster 2021-02-10T04:52:41.159078+0000 osd.2 (osd.2) 2013 : cluster [WRN] slow request osd_op(client.14817.0:1822 7.8 7:10dac65a:::smithi08575397-39:head [copy-from ver 158 in=55b] snapc 83=[83,82,80,7e,78] RETRY=3 ondisk+retry+write+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707799+0000 currently started 2021-02-10T04:52:42.292 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:42 smithi085 conmon[35961]: cluster 2021-02-10T04:52:41.159091+0000 osd.2 (osd.2) 2014 : cluster [WRN] slow request osd_op(client.14817.0:1823 7.8 7:10dac65a:::smithi08575397-39:head [stat] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected e1118) initiated 2021-02-10T04:51:35.707924+0000 currently delayed 2021-02-10T04:52:42.576 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:52:42.593 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:52:42.610 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:52:42.628 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:52:42.645 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:52:42.690 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:52:42.708 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:52:42.709 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-02-10T04:52:42.709 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: problem getting command descriptions from osd.0 2021-02-10T04:52:42.709 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:52:42.709 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:52:42.709 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:52:42.709 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:52:42.709 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:52:42.709 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:52:42.709 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:52:43.095 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:43 smithi085 conmon[29319]: cluster 2021-02-10T04:52:41.285999+0000 2021-02-10T04:52:43.204 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:43 smithi085 conmon[29319]: osd.5 (osd.5) 174 : cluster [DBG] 7.17 scrub ok 2021-02-10T04:52:43.204 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:43 smithi085 conmon[29319]: cluster 2021-02-10T04:52:41.758680+0000 osd.4 (osd.4) 2640 : cluster [DBG] 2.27 scrub ok 2021-02-10T04:52:43.204 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:43 smithi085 conmon[29319]: audit 2021-02-10T04:52:42.555328+0000 mon.c (mon.1) 101 : audit [DBG] from='client.? 172.21.15.85:0/4040206045' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:52:43.204 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:43 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:52:43.204 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:43 smithi085 conmon[29319]: 02-10T04:52:42.727655+0000 mon.c (mon.1) 102 : audit [INF] from='client.? 172.21.15.85:0/1440808244' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:52:43.204 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:43 smithi085 conmon[29319]: audit 2021 2021-02-10T04:52:43.204 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:43 smithi085 conmon[29319]: -02-10T04:52: 2021-02-10T04:52:43.204 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:43 smithi085 conmon[29319]: 42.727952+0000 mon.a (mon. 2021-02-10T04:52:43.204 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:43 smithi085 conmon[29319]: 0) 2433 : audit [INF] 2021-02-10T04:52:43.204 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:43 smithi085 conmon[29319]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:52:43.205 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is set 2021-02-10T04:52:43.373 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:43 smithi164 conmon[29288]: cluster 2021-02-10T04:52:41.285999+0000 osd.5 (osd.5 2021-02-10T04:52:43.373 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:43 smithi164 conmon[29288]: ) 174 : cluster [DBG] 7.17 scrub ok 2021-02-10T04:52:43.373 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:43 smithi164 conmon[29288]: cluster 2021-02-10T04:52:41.758680+0000 osd.4 ( 2021-02-10T04:52:43.373 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:43 smithi164 conmon[29288]: osd.4) 2640 : cluster [DBG] 2.27 scrub ok 2021-02-10T04:52:43.373 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:43 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:52:43.374 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:43 smithi164 conmon[29288]: :52:42.555328+0000 mon.c (mon.1) 2021-02-10T04:52:43.374 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:43 smithi164 conmon[29288]: 101 : audit [DBG] from='client.? 172.21.15.85:0/4040206045' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:52:43.374 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:43 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:52:43.374 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:43 smithi164 conmon[29288]: 04:52:42.727655+0000 mon.c (mon.1) 2021-02-10T04:52:43.374 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:43 smithi164 conmon[29288]: 102 : audit [INF] from='client.? 172.21.15.85:0/1440808244' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:52:43.375 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:43 smithi164 conmon[29288]: audit 2021-02-10T04:52:43.375 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:43 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:52:43.375 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:43 smithi164 conmon[29288]: 52:42.727952+0000 mon.a ( 2021-02-10T04:52:43.375 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:43 smithi164 conmon[29288]: mon.0) 2433 : audit [INF] 2021-02-10T04:52:43.375 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:43 smithi164 conmon[29288]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:52:43.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:43 smithi085 conmon[35961]: cluster 2021-02-10T04:52:43.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:43 smithi085 conmon[35961]: 2021-02-10T04:52:43.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:43 smithi085 conmon[35961]: 2021-02-10T04:52:41. 2021-02-10T04:52:43.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:43 smithi085 conmon[35961]: 285999+0000 osd.5 (osd.5) 174 : cluster [DBG] 7.17 scrub ok 2021-02-10T04:52:43.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:43 smithi085 conmon[35961]: cluster 2021-02-10T04:52 2021-02-10T04:52:43.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:43 smithi085 conmon[35961]: :41.758680+0000 osd.4 (osd.4) 2640 : 2021-02-10T04:52:43.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:43 smithi085 conmon[35961]: cluster [DBG] 2.27 scrub ok 2021-02-10T04:52:43.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:43 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:52:43.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:43 smithi085 conmon[35961]: :52:42.555328+0000 mon.c (mon.1 2021-02-10T04:52:43.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:43 smithi085 conmon[35961]: ) 101 : audit [DBG] from='client.? 172.21.15.85:0/4040206045' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:52:43.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:43 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:52:43.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:43 smithi085 conmon[35961]: 04:52:42.727655+0000 mon.c (mon. 2021-02-10T04:52:43.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:43 smithi085 conmon[35961]: 1) 102 : audit [INF] from='client.? 172.21.15.85:0/1440808244' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:52:43.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:43 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:52:43.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:43 smithi085 conmon[35961]: T04:52:42.727952+0000 mon.a (mon 2021-02-10T04:52:43.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:43 smithi085 conmon[35961]: .0) 2433 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:52:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:44 smithi164 conmon[29288]: cluster 2021-02-10T04:52:42.437351+0000 mgr.y (mgr.14140) 1257 2021-02-10T04:52:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:44 smithi164 conmon[29288]: : cluster [DBG] pgmap v2072: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 895 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 87 MiB/s rd, 21 MiB/s wr, 139 op/s; 278/2232 objects degraded (12.455%); 25/2232 objects misplaced (1.120%); 9.9 MiB/s, 19 objects/s recovering 2021-02-10T04:52:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:44 smithi164 conmon[29288]: cluster 2021-02-10T04:52:43.087168+0000 mon.a (mon.0) 2434 : cluster [WRN] Health check update: 2 slow ops, oldest one blocked for 64 sec, osd.2 has slow ops (SLOW_OPS) 2021-02-10T04:52:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:44 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:52:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:44 smithi164 conmon[29288]: -02-10T04:52:43.087752+0000 mon.a ( 2021-02-10T04:52:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:44 smithi164 conmon[29288]: mon.0) 2435 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:52:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:44 smithi164 conmon[29288]: audit 2021-02-10T04:52 2021-02-10T04:52:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:44 smithi164 conmon[29288]: :43.097264+0000 mon.a (mon.0) 2436 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:52:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:44 smithi164 conmon[29288]: cluster 2021-02-10T04:52 2021-02-10T04:52:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:44 smithi164 conmon[29288]: :43.097356+0000 mon.a (mon 2021-02-10T04:52:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:44 smithi164 conmon[29288]: .0) 2437 : cluster [DBG] 2021-02-10T04:52:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:44 smithi164 conmon[29288]: osdmap e1151: 8 total, 7 up, 8 in 2021-02-10T04:52:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:44 smithi164 conmon[29288]: cluster 2021-02-10T04:52:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:44 smithi164 conmon[29288]: 2021-02- 2021-02-10T04:52:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:44 smithi164 conmon[29288]: 10T04:52:43.102155+0000 osd.3 (osd 2021-02-10T04:52:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:44 smithi164 conmon[29288]: .3) 21 : cluster [DBG] 4.17 scrub ok 2021-02-10T04:52:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:44 smithi164 conmon[29288]: cluster 2021-02-10T04:52:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:44 smithi164 conmon[29288]: 2021- 2021-02-10T04:52:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:44 smithi164 conmon[29288]: 02-10T04:52:43.502930+0000 2021-02-10T04:52:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:44 smithi164 conmon[29288]: mon.a (mon.0) 2438 : cluster [DBG] osdmap e1152: 8 total, 7 up, 8 in 2021-02-10T04:52:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:44 smithi164 conmon[29288]: 2021-02-10T04:52:44.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[29319]: cluster 2021-02-10T04:52:42.437351+0000 mgr.y (mgr.14140) 1257 : cluster [DBG] pgmap v2072: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 895 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 87 MiB/s rd, 21 MiB/s wr, 139 op/s; 278/2232 objects degraded (12.455%); 25/2232 objects misplaced (1.120%); 9.9 MiB/s, 19 objects/s recovering 2021-02-10T04:52:44.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[29319]: cluster 2021-02-10T04:52:43.087168+0000 mon.a (mon.0) 2434 : cluster [WRN] Health check update: 2 slow ops, oldest one blocked for 64 sec, osd.2 has slow ops (SLOW_OPS) 2021-02-10T04:52:44.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[29319]: cluster 2021-02-10T04:52:43.087752+0000 mon.a 2021-02-10T04:52:44.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[29319]: (mon.0) 2435 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:52:44.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[29319]: 2021-02-10T04:52:44.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[29319]: audit 2021-02-10T04:52:43.097264+0000 mon.a (mon.0) 2436 : audit [INF] 2021-02-10T04:52:44.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[29319]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:52:44.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[29319]: cluster 2021-02-10T04:52:43 2021-02-10T04:52:44.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[29319]: .097356+0000 mon.a (mon.0) 2437 : cluster [DBG] 2021-02-10T04:52:44.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[29319]: osdmap e1151: 8 total, 7 up, 8 in 2021-02-10T04:52:44.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:52:44.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[29319]: 02-10T04:52:43.102155+0000 osd.3 (osd.3) 21 2021-02-10T04:52:44.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[29319]: : cluster [DBG] 4.17 scrub ok 2021-02-10T04:52:44.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[29319]: cluster 2021-02-10T04:52:43.502930 2021-02-10T04:52:44.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2438 : cluster [DBG] osdmap e1152: 8 total, 7 up, 8 in 2021-02-10T04:52:44.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[35961]: cluster 2021-02-10T04:52:44.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[35961]: 2021- 2021-02-10T04:52:44.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[35961]: 02-10T04:52:42.437351+0000 mgr.y (mgr.14140) 1257 : cluster [DBG] pgmap v2072: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 895 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 87 MiB/s rd, 21 MiB/s wr, 139 op/s; 278/2232 objects degraded (12.455%); 25/2232 objects misplaced (1.120%); 9.9 MiB/s, 19 objects/s recovering 2021-02-10T04:52:44.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:52:44.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[35961]: 02-10T04:52:43.087168+0000 mon.a 2021-02-10T04:52:44.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[35961]: (mon.0) 2434 : cluster [WRN] Health check update: 2 slow ops, oldest one blocked for 64 sec, osd.2 has slow ops (SLOW_OPS) 2021-02-10T04:52:44.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[35961]: 2021-02-10T04:52:44.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[35961]: cluster 2021-02-10T04:52:43.087752+0000 mon.a (mon. 2021-02-10T04:52:44.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[35961]: 0) 2435 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:52:44.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:52:44.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[35961]: -10T04:52:43.097264+0000 mon.a 2021-02-10T04:52:44.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[35961]: (mon.0) 2436 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:52:44.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[35961]: 2021-02-10T04:52:44.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[35961]: cluster 2021-02-10T04:52:44.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[35961]: 2021-02-10T04:52:43.097356+0000 mon.a ( 2021-02-10T04:52:44.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[35961]: mon.0) 2437 : cluster [DBG] osdmap e1151: 8 total, 7 up, 8 in 2021-02-10T04:52:44.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:52:44.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[35961]: -02-10T04:52:43.102155+0000 2021-02-10T04:52:44.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[35961]: osd.3 (osd.3) 21 : cluster [DBG] 2021-02-10T04:52:44.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[35961]: 4.17 scrub ok 2021-02-10T04:52:44.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[35961]: cluster 2021-02-10T04:52:43 2021-02-10T04:52:44.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[35961]: .502930+0000 mon.a (mon.0) 2438 : 2021-02-10T04:52:44.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:44 smithi085 conmon[35961]: cluster [DBG] osdmap e1152: 8 total, 7 up, 8 in 2021-02-10T04:52:44.448 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:52:44.448 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:52:44.553 INFO:teuthology.orchestra.run.smithi085.stdout:7.f 2021-02-10T04:52:44.553 INFO:teuthology.orchestra.run.smithi085.stdout:7.e 2021-02-10T04:52:44.553 INFO:teuthology.orchestra.run.smithi085.stdout:7.d 2021-02-10T04:52:44.553 INFO:teuthology.orchestra.run.smithi085.stdout:7.5 2021-02-10T04:52:44.553 INFO:teuthology.orchestra.run.smithi085.stdout:7.1f 2021-02-10T04:52:44.553 INFO:teuthology.orchestra.run.smithi085.stdout:7.1d 2021-02-10T04:52:44.553 INFO:teuthology.orchestra.run.smithi085.stdout:7.18 2021-02-10T04:52:44.554 INFO:teuthology.orchestra.run.smithi085.stdout:7.1e 2021-02-10T04:52:44.554 INFO:teuthology.orchestra.run.smithi085.stdout:7.16 2021-02-10T04:52:44.554 INFO:teuthology.orchestra.run.smithi085.stdout:7.13 2021-02-10T04:52:44.554 INFO:teuthology.orchestra.run.smithi085.stdout:7.4 2021-02-10T04:52:44.554 INFO:teuthology.orchestra.run.smithi085.stdout:6.9 2021-02-10T04:52:44.554 INFO:teuthology.orchestra.run.smithi085.stdout:6.26 2021-02-10T04:52:44.554 INFO:teuthology.orchestra.run.smithi085.stdout:6.18 2021-02-10T04:52:44.554 INFO:teuthology.orchestra.run.smithi085.stdout:6.16 2021-02-10T04:52:44.554 INFO:teuthology.orchestra.run.smithi085.stdout:6.13 2021-02-10T04:52:44.554 INFO:teuthology.orchestra.run.smithi085.stdout:6.11 2021-02-10T04:52:44.554 INFO:teuthology.orchestra.run.smithi085.stdout:7.7 2021-02-10T04:52:44.554 INFO:teuthology.orchestra.run.smithi085.stdout:6.10 2021-02-10T04:52:44.555 INFO:teuthology.orchestra.run.smithi085.stdout:7.9 2021-02-10T04:52:44.555 INFO:teuthology.orchestra.run.smithi085.stdout:6.27 2021-02-10T04:52:44.555 INFO:teuthology.orchestra.run.smithi085.stdout:5.e 2021-02-10T04:52:44.555 INFO:teuthology.orchestra.run.smithi085.stdout:5.c 2021-02-10T04:52:44.555 INFO:teuthology.orchestra.run.smithi085.stdout:7.6 2021-02-10T04:52:44.555 INFO:teuthology.orchestra.run.smithi085.stdout:5.7 2021-02-10T04:52:44.555 INFO:teuthology.orchestra.run.smithi085.stdout:2.4 2021-02-10T04:52:44.555 INFO:teuthology.orchestra.run.smithi085.stdout:2.21 2021-02-10T04:52:44.555 INFO:teuthology.orchestra.run.smithi085.stdout:7.12 2021-02-10T04:52:44.555 INFO:teuthology.orchestra.run.smithi085.stdout:2.1d 2021-02-10T04:52:44.555 INFO:teuthology.orchestra.run.smithi085.stdout:2.10 2021-02-10T04:52:44.555 INFO:teuthology.orchestra.run.smithi085.stdout:6.d 2021-02-10T04:52:44.556 INFO:teuthology.orchestra.run.smithi085.stdout:2.15 2021-02-10T04:52:44.556 INFO:teuthology.orchestra.run.smithi085.stdout:2.1 2021-02-10T04:52:44.556 INFO:teuthology.orchestra.run.smithi085.stdout:6.29 2021-02-10T04:52:44.556 INFO:teuthology.orchestra.run.smithi085.stdout:2.1f 2021-02-10T04:52:44.556 INFO:teuthology.orchestra.run.smithi085.stdout:1.1 2021-02-10T04:52:44.556 INFO:teuthology.orchestra.run.smithi085.stdout:2.1e 2021-02-10T04:52:44.556 INFO:teuthology.orchestra.run.smithi085.stdout:1.0 2021-02-10T04:52:44.556 INFO:teuthology.orchestra.run.smithi085.stdout:2.5 2021-02-10T04:52:44.556 INFO:teuthology.orchestra.run.smithi085.stdout:7.1b 2021-02-10T04:52:44.556 INFO:teuthology.orchestra.run.smithi085.stdout:2.13 2021-02-10T04:52:44.556 INFO:teuthology.orchestra.run.smithi085.stdout:2.0 2021-02-10T04:52:44.556 INFO:teuthology.orchestra.run.smithi085.stdout:2.c 2021-02-10T04:52:44.557 INFO:teuthology.orchestra.run.smithi085.stdout:4.7 2021-02-10T04:52:44.557 INFO:teuthology.orchestra.run.smithi085.stdout:1.6 2021-02-10T04:52:44.557 INFO:teuthology.orchestra.run.smithi085.stdout:2.f 2021-02-10T04:52:44.557 INFO:teuthology.orchestra.run.smithi085.stdout:5.8 2021-02-10T04:52:44.557 INFO:teuthology.orchestra.run.smithi085.stdout:4.19 2021-02-10T04:52:44.557 INFO:teuthology.orchestra.run.smithi085.stdout:3.1 2021-02-10T04:52:44.557 INFO:teuthology.orchestra.run.smithi085.stdout:1.2 2021-02-10T04:52:44.557 INFO:teuthology.orchestra.run.smithi085.stdout:4.0 2021-02-10T04:52:44.557 INFO:teuthology.orchestra.run.smithi085.stdout:4.16 2021-02-10T04:52:44.557 INFO:teuthology.orchestra.run.smithi085.stdout:4.1e 2021-02-10T04:52:44.557 INFO:teuthology.orchestra.run.smithi085.stdout:2.19 2021-02-10T04:52:44.557 INFO:teuthology.orchestra.run.smithi085.stdout:4.1a 2021-02-10T04:52:44.557 INFO:teuthology.orchestra.run.smithi085.stdout:4.3 2021-02-10T04:52:44.557 INFO:teuthology.orchestra.run.smithi085.stdout:4.6 2021-02-10T04:52:44.558 INFO:teuthology.orchestra.run.smithi085.stdout:4.b 2021-02-10T04:52:44.558 INFO:teuthology.orchestra.run.smithi085.stdout:6.a 2021-02-10T04:52:44.558 INFO:teuthology.orchestra.run.smithi085.stdout:5.2 2021-02-10T04:52:44.558 INFO:teuthology.orchestra.run.smithi085.stdout:5.1d 2021-02-10T04:52:44.558 INFO:teuthology.orchestra.run.smithi085.stdout:6.0 2021-02-10T04:52:44.558 INFO:teuthology.orchestra.run.smithi085.stdout:5.13 2021-02-10T04:52:44.558 INFO:teuthology.orchestra.run.smithi085.stdout:5.3 2021-02-10T04:52:44.558 INFO:teuthology.orchestra.run.smithi085.stdout:5.15 2021-02-10T04:52:44.558 INFO:teuthology.orchestra.run.smithi085.stdout:5.1e 2021-02-10T04:52:44.558 INFO:teuthology.orchestra.run.smithi085.stdout:5.b 2021-02-10T04:52:44.558 INFO:teuthology.orchestra.run.smithi085.stdout:2.8 2021-02-10T04:52:44.558 INFO:teuthology.orchestra.run.smithi085.stdout:5.5 2021-02-10T04:52:45.935 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -n osd.0 --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op export --pgid 2.1 --file /var/log/ceph/exp.2.1.0 2021-02-10T04:52:46.183 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/osd.0/config 2021-02-10T04:52:46.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:46 smithi085 conmon[35961]: cluster 2021-02-10T04:52:44. 2021-02-10T04:52:46.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:46 smithi085 conmon[35961]: 437948+0000 mgr.y (mgr.14140) 1258 : cluster [DBG] pgmap v2075: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 895 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/2232 objects degraded (12.455%); 25/2232 objects misplaced (1.120%) 2021-02-10T04:52:46.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:46 smithi085 conmon[29319]: cluster 2021-02-10T04:52:44.437948 2021-02-10T04:52:46.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:46 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 1258 : cluster [DBG] pgmap v2075: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 895 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/2232 objects degraded (12.455%); 25/2232 objects misplaced (1.120%) 2021-02-10T04:52:46.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:46 smithi164 conmon[29288]: cluster 2021-02-10T04:52:44.437948+0000 mgr.y (mgr.14140) 1258 : cluster [DBG] 2021-02-10T04:52:46.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:46 smithi164 conmon[29288]: pgmap v2075: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 895 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 278/2232 objects degraded (12.455%); 25/2232 objects misplaced (1.120%) 2021-02-10T04:52:46.450 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:52:46.670 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:52:46.721 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set nodeep-scrub 2021-02-10T04:52:46.955 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:52:47.305 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:47 smithi085 conmon[29319]: cluster 2021-02-10T04:52:47.098708+0000 mon.a (mon.0) 2439 : cluster [WRN] 2021-02-10T04:52:47.305 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:47 smithi085 conmon[29319]: Health check update: Degraded data redundancy: 278/2235 objects degraded (12.438%), 29 pgs degraded (PG_DEGRADED) 2021-02-10T04:52:47.305 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:47 smithi085 conmon[29319]: cluster 2021-02-10T04:52:47.098736+0000 mon.a (mon.0) 2440 : cluster [INF] Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 64 sec, osd.2 has slow ops) 2021-02-10T04:52:47.305 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:47 smithi085 conmon[35961]: cluster 2021-02-10T04:52:47. 2021-02-10T04:52:47.305 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:47 smithi085 conmon[35961]: 098708+0000 mon.a (mon.0) 2439 : cluster [WRN] Health check update: Degraded data redundancy: 278/2235 objects degraded (12.438%), 29 pgs degraded (PG_DEGRADED) 2021-02-10T04:52:47.305 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:47 smithi085 conmon[35961]: cluster 2021-02-10T04:52:47.305 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:47 smithi085 conmon[35961]: 2021-02-10T04:52:47.098736+0000 mon.a (mon.0) 2440 : 2021-02-10T04:52:47.306 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:47 smithi085 conmon[35961]: cluster [INF] Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 64 sec, osd.2 has slow ops) 2021-02-10T04:52:47.605 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:47 smithi164 conmon[29288]: cluster 2021-02-10T04:52:47.098708 2021-02-10T04:52:47.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:47 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2439 : cluster [WRN] Health check update: Degraded data redundancy: 278/2235 objects degraded (12.438%), 29 pgs degraded (PG_DEGRADED) 2021-02-10T04:52:47.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:47 smithi164 conmon[29288]: cluster 2021-02-10T04:52:47.098736+0000 mon.a (mon. 2021-02-10T04:52:47.606 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:47 smithi164 conmon[29288]: 0) 2440 : cluster [INF] Health check cleared: SLOW_OPS (was: 2 slow ops, oldest one blocked for 64 sec, osd.2 has slow ops) 2021-02-10T04:52:48.094 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:52:48.094 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:52:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:48 smithi164 conmon[29288]: cluster 2021-02-10T04:52:46.438590+0000 mgr.y (mgr.14140) 1259 : cluster [DBG] pgmap v2076: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 899 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 470 KiB/s wr, 0 op/s; 278/2235 objects degraded (12.438%); 25/2235 objects misplaced (1.119%) 2021-02-10T04:52:48.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:48 smithi085 conmon[29319]: cluster 2021-02-10T04:52:46.438590+0000 mgr.y (mgr.14140 2021-02-10T04:52:48.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:48 smithi085 conmon[29319]: ) 1259 : cluster [DBG] pgmap v2076: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 899 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 470 KiB/s wr, 0 op/s; 278/2235 objects degraded (12.438%); 25/2235 objects misplaced (1.119%) 2021-02-10T04:52:48.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:48 smithi085 conmon[35961]: cluster 2021-02-10T04:52:46.438590+0000 mgr.y 2021-02-10T04:52:48.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:48 smithi085 conmon[35961]: (mgr.14140) 1259 : cluster [DBG] pgmap v2076: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 899 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 470 KiB/s wr, 0 op/s; 278/2235 objects degraded (12.438%); 25/2235 objects misplaced (1.119%) 2021-02-10T04:52:49.364 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:49 smithi085 conmon[35961]: cluster 2021-02-10T04:52:47.580245+0000 osd.7 (osd.7) 2021-02-10T04:52:49.365 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:49 smithi085 conmon[35961]: 86 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:52:49.365 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:49 smithi085 conmon[35961]: cluster 2021-02-10T04:52:47.581752+0000 osd.7 (osd.7) 87 : cluster [DBG] purged_snaps scrub found 3 strays 2021-02-10T04:52:49.365 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:49 smithi085 conmon[35961]: cluster 2021-02-10T04:52:48.160196+0000 mon.a (mon.0) 2441 : cluster [DBG] osdmap e1153: 8 total, 7 up, 8 in 2021-02-10T04:52:49.365 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:49 smithi085 conmon[35961]: cluster 2021-02-10T04:52:48.509595+0000 mon.a (mon.0) 2442 : cluster [DBG] osdmap e1154: 8 total, 7 up, 8 in 2021-02-10T04:52:49.365 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:49 smithi085 conmon[29319]: cluster 2021-02-10T04:52:47.580245+0000 osd.7 (osd. 2021-02-10T04:52:49.365 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:49 smithi085 conmon[29319]: 7) 86 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:52:49.365 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:49 smithi085 conmon[29319]: cluster 2021-02-10T04:52:47.581752+0000 osd.7 (osd.7) 87 : cluster [DBG] purged_snaps scrub found 3 strays 2021-02-10T04:52:49.365 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:49 smithi085 conmon[29319]: cluster 2021-02-10T04:52:48.160196+0000 mon.a (mon.0) 2441 : cluster [DBG] osdmap e1153: 8 total, 7 up, 8 in 2021-02-10T04:52:49.366 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:49 smithi085 conmon[29319]: cluster 2021-02-10T04:52:48.509595+0000 mon.a (mon.0) 2442 : cluster [DBG] osdmap e1154: 8 total, 7 up, 8 in 2021-02-10T04:52:49.644 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:52:49 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:52:49.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:49 smithi164 conmon[29288]: cluster 2021-02-10T04:52:49.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:49 smithi164 conmon[29288]: 2021-02-10T04:52 2021-02-10T04:52:49.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:49 smithi164 conmon[29288]: :47.580245+0000 osd.7 (osd.7) 86 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:52:49.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:49 smithi164 conmon[29288]: cluster 2021-02-10T04:52: 2021-02-10T04:52:49.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:49 smithi164 conmon[29288]: 47.581752+0000 osd.7 (osd.7) 87 : cluster [DBG] purged_snaps scrub found 3 strays 2021-02-10T04:52:49.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:49 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:52:49.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:49 smithi164 conmon[29288]: 10T04:52:48.160196+0000 mon.a (mon.0) 2441 2021-02-10T04:52:49.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:49 smithi164 conmon[29288]: : cluster [DBG] osdmap e1153: 8 total, 7 up, 8 in 2021-02-10T04:52:49.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:49 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:52:49.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:49 smithi164 conmon[29288]: :52:48.509595+0000 mon.a (mon.0) 2021-02-10T04:52:49.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:49 smithi164 conmon[29288]: 2442 : cluster [DBG] osdmap e1154: 8 total, 7 up, 8 in 2021-02-10T04:52:50.371 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:50 smithi085 conmon[29319]: cluster 2021-02-10T04:52:48.439145+0000 mgr.y (mgr.14140) 1260 : cluster [DBG] pgmap v2078: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 899 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 626 KiB/s wr, 1 op/s; 278/2235 objects degraded (12.438%); 25/2235 objects misplaced (1.119%) 2021-02-10T04:52:50.372 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:50 smithi085 conmon[29319]: audit 2021-02-10T04:52:49.154160+0000 mgr.y (mgr.14140) 1261 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:52:50.372 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:50 smithi085 conmon[35961]: cluster 2021-02-10T04:52:50.372 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:50 smithi085 conmon[35961]: 2021-02-10T04:52:50.372 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:50 smithi085 conmon[35961]: 2021-02-10T04:52: 2021-02-10T04:52:50.372 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:50 smithi085 conmon[35961]: 48.439145+0000 mgr.y (mgr.14140) 1260 : cluster [DBG] pgmap v2078: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 899 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 626 KiB/s wr, 1 op/s; 278/2235 objects degraded (12.438%); 25/2235 objects misplaced (1.119%) 2021-02-10T04:52:50.372 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:50 smithi085 conmon[35961]: audit 2021-02-10T04:52:49 2021-02-10T04:52:50.372 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:50 smithi085 conmon[35961]: .154160+0000 mgr.y (mgr.14140) 1261 : audit 2021-02-10T04:52:50.372 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:50 smithi085 conmon[35961]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:52:50.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:50 smithi164 conmon[29288]: cluster 2021-02-10T04:52:50.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:50 smithi164 conmon[29288]: 2021-02-10T04:52:50.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:50 smithi164 conmon[29288]: 2021-02-10T04:52:48.439145 2021-02-10T04:52:50.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:50 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1260 : cluster [DBG] pgmap v2078: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 899 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 626 KiB/s wr, 1 op/s; 278/2235 objects degraded (12.438%); 25/2235 objects misplaced (1.119%) 2021-02-10T04:52:50.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:50 smithi164 conmon[29288]: audit 2021-02-10T04:52 2021-02-10T04:52:50.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:50 smithi164 conmon[29288]: :49.154160+0000 mgr.y (mgr.14140) 1261 : audit 2021-02-10T04:52:50.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:50 smithi164 conmon[29288]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:52:50.834 INFO:teuthology.orchestra.run.smithi085.stderr:Exporting 2.1 info 2.1( empty local-lis/les=1118/1119 n=0 ec=48/48 lis/c=1118/1118 les/c/f=1119/1119/0 sis=1118) 2021-02-10T04:52:50.834 INFO:teuthology.orchestra.run.smithi085.stderr:Export successful 2021-02-10T04:52:50.870 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:52:50.887 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:52:50.909 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:52:50.927 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:52:50.945 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:52:50.969 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:52:50.993 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:52:50.993 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-02-10T04:52:50.993 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: problem getting command descriptions from osd.0 2021-02-10T04:52:50.993 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:52:50.993 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:52:50.993 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:52:50.993 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:52:50.993 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:52:50.994 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:52:50.994 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:52:51.195 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is set 2021-02-10T04:52:51.364 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:51 smithi085 conmon[29319]: audit 2021-02-10T04:52:50.845582 2021-02-10T04:52:51.364 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:51 smithi085 conmon[29319]: +0000 mon.c (mon.1) 103 : audit [DBG] from='client.? 172.21.15.85:0/3425497421' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:52:51.364 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:51 smithi085 conmon[29319]: audit 2021-02-10T04:52:51.072718+0000 mon.a (mon.0) 2443 : audit [INF] from='client.? 172.21.15.85:0/3186580094' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:52:51.364 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:51 smithi085 conmon[35961]: audit 2021-02-10T04:52:50.845582 2021-02-10T04:52:51.364 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:51 smithi085 conmon[35961]: +0000 mon.c (mon.1) 103 : audit [DBG] from='client.? 172.21.15.85:0/3425497421' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:52:51.364 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:51 smithi085 conmon[35961]: audit 2021-02-10T04:52: 2021-02-10T04:52:51.365 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:51 smithi085 conmon[35961]: 51.072718+0000 mon.a (mon.0) 2443 : audit [INF] from='client.? 172.21.15.85:0/3186580094' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:52:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:51 smithi164 conmon[29288]: audit 2021-02-10T04:52:50.845582+0000 mon.c ( 2021-02-10T04:52:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:51 smithi164 conmon[29288]: mon.1) 103 : audit [DBG] from='client.? 172.21.15.85:0/3425497421' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:52:51.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:51 smithi164 conmon[29288]: audit 2021-02-10T04:52:51.072718+0000 mon.a (mon.0) 2443 : audit [INF] from='client.? 172.21.15.85:0/3186580094' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:52:51.773 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.6 dump_blocked_ops 2021-02-10T04:52:52.151 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:52:52.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:52 smithi085 conmon[29319]: cluster 2021-02-10T04:52:50.439927+0000 mgr.y 2021-02-10T04:52:52.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:52 smithi085 conmon[29319]: (mgr.14140) 1262 : cluster [DBG] pgmap v2080: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 899 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 737 B/s rd, 542 KiB/s wr, 1 op/s; 278/2235 objects degraded (12.438%); 25/2235 objects misplaced (1.119%) 2021-02-10T04:52:52.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:52 smithi085 conmon[29319]: cluster 2021-02-10T04:52:51.165953+0000 mon.a ( 2021-02-10T04:52:52.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:52 smithi085 conmon[29319]: mon.0) 2444 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:52:52.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:52 smithi085 conmon[29319]: audit 2021 2021-02-10T04:52:52.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:52 smithi085 conmon[29319]: -02-10T04:52:51 2021-02-10T04:52:52.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:52 smithi085 conmon[29319]: .175554+0000 mon.a (mon. 2021-02-10T04:52:52.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:52 smithi085 conmon[29319]: 0) 2445 : audit [INF] from='client.? 172.21.15.85:0/3186580094' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:52:52.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:52 smithi085 conmon[29319]: 2021-02-10T04:52:52.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:52 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:52:52.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:52 smithi085 conmon[29319]: -02-10T04:52: 2021-02-10T04:52:52.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:52 smithi085 conmon[29319]: 51.175760+0000 mon.a (mon 2021-02-10T04:52:52.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:52 smithi085 conmon[29319]: .0) 2446 : cluster [DBG] 2021-02-10T04:52:52.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:52 smithi085 conmon[29319]: osdmap e1155: 8 total, 7 up, 8 in 2021-02-10T04:52:52.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:52 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:52:52.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:52 smithi085 conmon[35961]: :52:50.439927+0000 mgr.y (mgr.14140) 1262 : cluster [DBG] pgmap v2080: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 899 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 737 B/s rd, 542 KiB/s wr, 1 op/s; 278/2235 objects degraded (12.438%); 25/2235 objects misplaced (1.119%) 2021-02-10T04:52:52.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:52 smithi085 conmon[35961]: 2021-02-10T04:52:52.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:52 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:52:52.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:52 smithi085 conmon[35961]: 02-10T04:52:51.165953+0000 mon.a ( 2021-02-10T04:52:52.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:52 smithi085 conmon[35961]: mon.0) 2444 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:52:52.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:52 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:52:52.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:52 smithi085 conmon[35961]: T04:52:51.175554+0000 2021-02-10T04:52:52.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:52 smithi085 conmon[35961]: mon.a (mon.0) 2445 : audit [INF] from='client.? 172.21.15.85:0/3186580094' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:52:52.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:52 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:52:52.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:52 smithi085 conmon[35961]: T04:52:51.175760+0000 mon.a (mon.0) 2446 : cluster 2021-02-10T04:52:52.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:52 smithi085 conmon[35961]: [DBG] osdmap e1155: 8 total, 7 up, 8 in 2021-02-10T04:52:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:52 smithi164 conmon[29288]: cluster 2021-02-10T04:52:50.439927+0000 2021-02-10T04:52:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:52 smithi164 conmon[29288]: mgr.y (mgr.14140) 1262 : cluster [DBG] pgmap v2080: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 899 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 737 B/s rd, 542 KiB/s wr, 1 op/s; 278/2235 objects degraded (12.438%); 25/2235 objects misplaced (1.119%) 2021-02-10T04:52:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:52 smithi164 conmon[29288]: cluster 2021-02-10T04:52:51.165953+0000 mon.a (mon.0) 2444 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:52:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:52 smithi164 conmon[29288]: audit 2021-02-10T04:52:51.175554+0000 mon.a (mon.0) 2445 : audit [INF] from='client.? 172.21.15.85:0/3186580094' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:52:52.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:52 smithi164 conmon[29288]: cluster 2021-02-10T04:52:51.175760+0000 mon.a (mon.0) 2446 : cluster [DBG] osdmap e1155: 8 total, 7 up, 8 in 2021-02-10T04:52:52.728 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:52:52.728 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:52:53.376 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -n osd.0 --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --force --op remove --pgid 2.1 2021-02-10T04:52:53.605 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/osd.0/config 2021-02-10T04:52:54.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:54 smithi085 conmon[35961]: cluster 2021-02-10T04:52:52.440486+0000 mgr.y (mgr.14140) 1263 : cluster [DBG] pgmap v2082: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 899 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 278/2235 objects degraded (12.438%); 25/2235 objects misplaced (1.119%) 2021-02-10T04:52:54.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:54 smithi085 conmon[35961]: cluster 2021-02-10T04:52:53.275170+0000 osd.4 (osd.4) 2641 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:52:54.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:54 smithi085 conmon[35961]: cluster 2021-02-10T04:52:53.276094+0000 osd.4 (osd.4) 2642 : cluster [DBG] purged_snaps scrub found 3 strays 2021-02-10T04:52:54.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:54 smithi085 conmon[35961]: cluster 2021-02-10T04:52:53.511984+0000 mon.a (mon.0) 2447 : cluster [DBG] osdmap e1156: 8 total, 7 up, 8 in 2021-02-10T04:52:54.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:54 smithi085 conmon[29319]: cluster 2021-02-10T04:52:52.440486+0000 mgr.y (mgr.14140) 1263 : cluster [DBG] pgmap v2082: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 899 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 278/2235 objects degraded (12.438%); 25/2235 objects misplaced (1.119%) 2021-02-10T04:52:54.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:54 smithi085 conmon[29319]: cluster 2021-02-10T04:52:53.275170+0000 osd.4 (osd.4) 2641 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:52:54.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:54 smithi085 conmon[29319]: cluster 2021-02-10T04:52:53.276094+0000 osd.4 (osd.4) 2642 : cluster [DBG] purged_snaps scrub found 3 strays 2021-02-10T04:52:54.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:54 smithi085 conmon[29319]: cluster 2021-02-10T04:52:53.511984+0000 mon.a (mon.0) 2447 : cluster [DBG] osdmap e1156: 8 total, 7 up, 8 in 2021-02-10T04:52:54.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:54 smithi164 conmon[29288]: cluster 2021-02-10T04:52:54.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:54 smithi164 conmon[29288]: 2021- 2021-02-10T04:52:54.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:54 smithi164 conmon[29288]: 02-10T04:52:52.440486+0000 mgr.y (mgr.14140) 1263 : cluster [DBG] pgmap v2082: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 899 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 278/2235 objects degraded (12.438%); 25/2235 objects misplaced (1.119%) 2021-02-10T04:52:54.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:54 smithi164 conmon[29288]: cluster 2021-02-10T04:52:54.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:54 smithi164 conmon[29288]: 2021-02-10T04:52 2021-02-10T04:52:54.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:54 smithi164 conmon[29288]: :53.275170+0000 osd.4 (osd.4) 2641 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:52:54.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:54 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:52:54.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:54 smithi164 conmon[29288]: 52:53.276094+0000 osd.4 (osd.4) 2642 2021-02-10T04:52:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:54 smithi164 conmon[29288]: : cluster [DBG] purged_snaps scrub found 3 strays 2021-02-10T04:52:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:54 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:52:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:54 smithi164 conmon[29288]: 10T04:52:53.511984+0000 mon.a (mon.0 2021-02-10T04:52:54.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:54 smithi164 conmon[29288]: ) 2447 : cluster [DBG] osdmap e1156: 8 total, 7 up, 8 in 2021-02-10T04:52:54.729 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:52:54.940 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:52:55.080 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset noscrub 2021-02-10T04:52:55.311 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:52:55.859 INFO:teuthology.orchestra.run.smithi085.stdout: marking collection for removal 2021-02-10T04:52:55.859 INFO:teuthology.orchestra.run.smithi085.stdout:setting '_remove' omap key 2021-02-10T04:52:55.864 INFO:teuthology.orchestra.run.smithi085.stdout:finish_remove_pgs 2.1_head removing 2.1 2021-02-10T04:52:55.865 INFO:teuthology.orchestra.run.smithi085.stdout:Remove successful 2021-02-10T04:52:55.930 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:52:55.930 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2021-02-10T04:52:55.930 INFO:teuthology.orchestra.run.smithi164.stdout: "complaint_time": 30, 2021-02-10T04:52:55.930 INFO:teuthology.orchestra.run.smithi164.stdout: "num_blocked_ops": 0 2021-02-10T04:52:55.931 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:52:56.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:56 smithi164 conmon[29288]: cluster 2021-02-10T04:52:54.441020+0000 mgr.y (mgr.14140) 1264 : cluster [DBG] pgmap v2084: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 899 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 278/2235 objects degraded (12.438%); 25/2235 objects misplaced (1.119%) 2021-02-10T04:52:56.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:56 smithi085 conmon[29319]: cluster 2021-02-10T04:52:54.441020+0000 mgr.y (mgr.14140) 1264 : cluster [DBG] pgmap v2084: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 899 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 278/2235 objects degraded (12.438%); 25/2235 objects misplaced (1.119%) 2021-02-10T04:52:56.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:56 smithi085 conmon[35961]: cluster 2021-02-10T04:52:54.441020+0000 mgr.y (mgr.14140) 1264 : cluster [DBG] pgmap v2084: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 899 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 278/2235 objects degraded (12.438%); 25/2235 objects misplaced (1.119%) 2021-02-10T04:52:58.215 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -n osd.0 --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.1.0 2021-02-10T04:52:58.432 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/osd.0/config 2021-02-10T04:52:58.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:58 smithi085 conmon[35961]: cluster 2021-02-10T04:52: 2021-02-10T04:52:58.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:58 smithi085 conmon[35961]: 56.441807+0000 mgr.y (mgr.14140) 1265 : cluster [DBG] pgmap v2085: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 899 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 645 B/s rd, 0 op/s; 278/2235 objects degraded (12.438%); 25/2235 objects misplaced (1.119%) 2021-02-10T04:52:58.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:58 smithi085 conmon[29319]: cluster 2021-02-10T04:52:56.441807 2021-02-10T04:52:58.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:58 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 1265 : cluster [DBG] pgmap v2085: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 899 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 645 B/s rd, 0 op/s; 278/2235 objects degraded (12.438%); 25/2235 objects misplaced (1.119%) 2021-02-10T04:52:58.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: cluster 2021-02-10T04:52:58.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: 2021-02-10T04:52:58.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: 2021 2021-02-10T04:52:58.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: - 2021-02-10T04:52:58.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: 02 2021-02-10T04:52:58.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: - 2021-02-10T04:52:58.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: 10 2021-02-10T04:52:58.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: T 2021-02-10T04:52:58.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: 04 2021-02-10T04:52:58.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: : 2021-02-10T04:52:58.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: 52 2021-02-10T04:52:58.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: : 2021-02-10T04:52:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: 56 2021-02-10T04:52:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: . 2021-02-10T04:52:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: 441807 2021-02-10T04:52:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: +0000 2021-02-10T04:52:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: 2021-02-10T04:52:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: mgr.y 2021-02-10T04:52:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: ( 2021-02-10T04:52:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: mgr 2021-02-10T04:52:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: . 2021-02-10T04:52:58.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: 14140 2021-02-10T04:52:58.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: ) 2021-02-10T04:52:58.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: 1265 2021-02-10T04:52:58.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: : 2021-02-10T04:52:58.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: cluster 2021-02-10T04:52:58.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: 2021-02-10T04:52:58.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: [DBG] 2021-02-10T04:52:58.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: 2021-02-10T04:52:58.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: pgmap v2085: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 899 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 645 B/s rd, 0 op/s; 278/2235 objects degraded (12.438%); 25/2235 objects misplaced (1.119%) 2021-02-10T04:52:58.647 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:58 smithi164 conmon[29288]: 2021-02-10T04:52:58.769 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:52:58.804 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:52:58.840 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:52:58.861 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:52:58.882 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:52:58.902 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:52:58.924 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:52:58.924 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-02-10T04:52:58.924 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: problem getting command descriptions from osd.0 2021-02-10T04:52:58.924 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:52:58.924 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:52:58.924 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:52:58.924 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:52:58.924 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:52:58.924 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:52:58.924 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:52:59.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:59 smithi085 conmon[29319]: cluster 2021-02-10T04:52:57.759188+0000 osd.1 (osd.1) 491 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:52:59.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:59 smithi085 conmon[29319]: cluster 2021-02-10T04:52:57.760951+0000 osd.1 (osd.1) 492 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:52:59.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:59 smithi085 conmon[29319]: cluster 2021-02-10T04:52:58.183082+0000 mon.a (mon.0) 2448 : cluster [DBG] osdmap e1157: 8 total, 7 up, 8 in 2021-02-10T04:52:59.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:59 smithi085 conmon[29319]: audit 2021-02-10T04:52:58.742264+0000 mon.c (mon.1) 104 : audit [DBG] from='client.? 172.21.15.85:0/3581874365' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:52:59.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:59 smithi085 conmon[29319]: audit 2021-02-10T04:52:59.115625+0000 mon.c (mon.1) 105 : audit [INF] from='client.? 172.21.15.85:0/2592890273' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:52:59.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:52:59 smithi085 conmon[29319]: audit 2021-02-10T04:52:59.115933+0000 mon.a (mon.0) 2449 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:52:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:59 smithi085 conmon[35961]: cluster 2021-02-10T04:52:57.759188+0000 osd.1 (osd.1) 491 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:52:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:59 smithi085 conmon[35961]: cluster 2021-02-10T04:52:57.760951+0000 osd.1 (osd.1) 492 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:52:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:59 smithi085 conmon[35961]: cluster 2021-02-10T04:52:58.183082+0000 mon.a (mon.0) 2448 : cluster [DBG] osdmap e1157: 8 total, 7 up, 8 in 2021-02-10T04:52:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:59 smithi085 conmon[35961]: audit 2021-02-10T04:52:58.742264+0000 mon.c (mon.1) 104 : audit [DBG] from='client.? 172.21.15.85:0/3581874365' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:52:59.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:59 smithi085 conmon[35961]: audit 2021-02-10T04:52:59.115625+0000 mon.c (mon.1) 105 : audit [INF] from='client.? 172.21.15.85:0/2592890273' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:52:59.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:52:59 smithi085 conmon[35961]: audit 2021-02-10T04:52:59.115933+0000 mon.a (mon.0) 2449 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:52:59.516 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is unset 2021-02-10T04:52:59.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:52:59 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:52:59.537 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:59 smithi164 conmon[29288]: cluster 2021-02-10T04:52:57.759188 2021-02-10T04:52:59.537 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:59 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 491 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:52:59.537 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:59 smithi164 conmon[29288]: cluster 2021-02-10T04:52:59.537 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:59 smithi164 conmon[29288]: 2021-02-10T04:52:57.760951 2021-02-10T04:52:59.537 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:59 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 492 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:52:59.537 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:59 smithi164 conmon[29288]: 2021-02-10T04:52:59.537 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:59 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:52:59.538 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:59 smithi164 conmon[29288]: T04:52:58.183082+0000 mon.a (mon. 2021-02-10T04:52:59.538 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:59 smithi164 conmon[29288]: 0) 2448 : cluster [DBG] osdmap e1157: 8 total, 7 up, 8 in 2021-02-10T04:52:59.538 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:59 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:52:59.538 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:59 smithi164 conmon[29288]: 02-10T04:52:58.742264+0000 mon.c (mon.1) 2021-02-10T04:52:59.538 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:59 smithi164 conmon[29288]: 104 : audit [DBG] from='client.? 172.21.15.85:0/3581874365' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:52:59.538 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:59 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:52:59.538 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:59 smithi164 conmon[29288]: 02-10T04:52:59.115625+0000 mon.c ( 2021-02-10T04:52:59.538 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:59 smithi164 conmon[29288]: mon.1) 105 : audit [INF] from='client.? 172.21.15.85:0/2592890273' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:52:59.538 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:59 smithi164 conmon[29288]: audit 2021-02-10T04:52:59.538 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:59 smithi164 conmon[29288]: 2021-02-10T04:52:59.115933+0000 2021-02-10T04:52:59.538 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:52:59 smithi164 conmon[29288]: mon.a (mon.0) 2449 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:52:59.541 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.6 dump_historic_ops 2021-02-10T04:52:59.838 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:53:00.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:00 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:53:00.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:00 smithi164 conmon[29288]: 10T04:52:58.442613+0000 mgr.y (mgr.14140) 1266 : cluster [DBG] pgmap v2087: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 899 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 278/2235 objects degraded (12.438%); 25/2235 objects misplaced (1.119%) 2021-02-10T04:53:00.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:00 smithi164 conmon[29288]: cluster 2021-02-10T04:52:58.726393+0000 osd.3 (osd.3) 22 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:53:00.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:00 smithi164 conmon[29288]: cluster 2021-02-10T04:52:58.730385+0000 osd.3 (osd.3) 23 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:53:00.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:00 smithi164 conmon[29288]: audit 2021-02-10T04:52:59.164098+0000 mgr.y (mgr.14140) 1267 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:53:00.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:00 smithi164 conmon[29288]: cluster 2021-02-10T04:52:59.505920+0000 mon.a (mon.0) 2450 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:53:00.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:00 smithi164 conmon[29288]: audit 2021-02-10T04:52:59.511556+0000 mon.a (mon.0) 2451 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:53:00.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:00 smithi164 conmon[29288]: cluster 2021-02-10T04:52:59.511693+0000 mon.a (mon.0) 2452 : cluster [DBG] osdmap e1158: 8 total, 7 up, 8 in 2021-02-10T04:53:00.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[29319]: cluster 2021-02-10T04:52:58.442613+0000 mgr.y (mgr. 2021-02-10T04:53:00.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[29319]: 14140) 1266 : cluster [DBG] pgmap v2087: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 899 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 278/2235 objects degraded (12.438%); 25/2235 objects misplaced (1.119%) 2021-02-10T04:53:00.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[29319]: cluster 2021-02-10T04:52:58.726393+0000 osd.3 (osd.3) 22 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:53:00.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[29319]: cluster 2021-02-10T04:52:58.730385+0000 osd.3 (osd.3) 23 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:53:00.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[29319]: audit 2021-02-10T04:52:59.164098+0000 mgr.y (mgr.14140) 1267 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:53:00.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[29319]: cluster 2021-02-10T04:52:59.505920+0000 mon.a (mon.0) 2450 : cluster 2021-02-10T04:53:00.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[29319]: [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:53:00.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[29319]: audit 2021-02-10T 2021-02-10T04:53:00.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[29319]: 04:52:59.511556+0000 mon.a (mon. 2021-02-10T04:53:00.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[29319]: 0) 2451 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:53:00.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[29319]: cluster 2021-02-10T04:53:00.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[29319]: 2021-02-10T04:52 2021-02-10T04:53:00.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[29319]: :59.511693+0000 mon.a (mon.0) 2021-02-10T04:53:00.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[29319]: 2452 : cluster [DBG] osdmap e1158: 8 total, 7 up, 8 in 2021-02-10T04:53:00.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[35961]: cluster 2021-02-10T04:53:00.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[35961]: 2021-02 2021-02-10T04:53:00.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[35961]: -10T04:52:58.442613+0000 mgr.y (mgr.14140) 1266 : cluster [DBG] 2021-02-10T04:53:00.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[35961]: pgmap v2087: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 899 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 278/2235 objects degraded (12.438%); 25/2235 objects misplaced (1.119%) 2021-02-10T04:53:00.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[35961]: cluster 2021-02-10T04:53:00.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[35961]: 2021-02-10T04:52: 2021-02-10T04:53:00.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[35961]: 58.726393+0000 osd.3 (osd.3) 22 2021-02-10T04:53:00.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[35961]: : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:53:00.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[35961]: cluster 2021-02-10T04:53:00.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[35961]: 2021-02-10T04:52: 2021-02-10T04:53:00.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[35961]: 58.730385+0000 osd.3 (osd.3) 23 : 2021-02-10T04:53:00.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[35961]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:53:00.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[35961]: audit 2021 2021-02-10T04:53:00.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[35961]: -02-10T04:52:59.164098+0000 2021-02-10T04:53:00.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[35961]: mgr.y (mgr.14140) 1267 : audit [DBG] 2021-02-10T04:53:00.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[35961]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:53:00.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[35961]: cluster 2021-02-10T04:52 2021-02-10T04:53:00.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[35961]: :59.505920+0000 mon.a (mon.0) 2021-02-10T04:53:00.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[35961]: 2450 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:53:00.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[35961]: audit 2021 2021-02-10T04:53:00.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[35961]: -02-10T04:52:59.511556+0000 2021-02-10T04:53:00.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[35961]: mon.a (mon.0) 2451 : audit [INF] 2021-02-10T04:53:00.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[35961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:53:00.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:53:00.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[35961]: 04:52:59.511693+0000 mon.a (mon 2021-02-10T04:53:00.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:00 smithi085 conmon[35961]: .0) 2452 : cluster [DBG] osdmap e1158: 8 total, 7 up, 8 in 2021-02-10T04:53:01.036 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:53:01.036 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:53:01.359 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:01 smithi085 conmon[29319]: cluster 2021-02-10T04:53:00.964485 2021-02-10T04:53:01.359 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:01 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 2015 : cluster [DBG] 7.8 scrub ok 2021-02-10T04:53:01.359 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:01 smithi085 conmon[35961]: cluster 2021-02-10T04:53:00.964485+0000 osd.2 (osd 2021-02-10T04:53:01.359 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:01 smithi085 conmon[35961]: .2) 2015 : cluster [DBG] 7.8 scrub ok 2021-02-10T04:53:01.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:01 smithi164 conmon[29288]: cluster 2021-02-10T04:53:00.964485+0000 2021-02-10T04:53:01.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:01 smithi164 conmon[29288]: osd.2 (osd.2) 2015 : cluster [DBG] 7.8 scrub ok 2021-02-10T04:53:02.279 INFO:teuthology.orchestra.run.smithi085.stdout:get_pg_num_history pg_num_history pg_num_history(e1146 pg_nums {1={110=11},2={89=42},3={55=3},4={55=32},5={57=32},6={1044=42},7={1007=26,1141=36}} deleted_pools ) 2021-02-10T04:53:02.279 INFO:teuthology.orchestra.run.smithi085.stdout:Importing pgid 2.1 2021-02-10T04:53:02.305 INFO:teuthology.orchestra.run.smithi085.stdout:write_pg epoch 1119 info 2.1( empty local-lis/les=1118/1119 n=0 ec=48/48 lis/c=1118/1118 les/c/f=1119/1119/0 sis=1118) 2021-02-10T04:53:02.315 INFO:teuthology.orchestra.run.smithi085.stdout:Import successful 2021-02-10T04:53:02.579 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:02 smithi164 conmon[29288]: cluster 2021-02-10T04:53:00.343224+0000 2021-02-10T04:53:02.579 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:02 smithi164 conmon[29288]: osd.5 (osd.5) 175 : cluster [DBG] 2.16 scrub ok 2021-02-10T04:53:02.580 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:02 smithi164 conmon[29288]: cluster 2021-02-10T04:53:00.443122+0000 mgr.y (mgr.14140) 1268 : cluster [DBG] pgmap v2089: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 899 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 738 B/s rd, 0 op/s; 278/2235 objects degraded (12.438%); 25/2235 objects misplaced (1.119%) 2021-02-10T04:53:02.580 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:02 smithi164 conmon[29288]: cluster 2021-02-10T04:53:01.304622+0000 osd.1 (osd.1) 493 : cluster [DBG] 4.2 scrub ok 2021-02-10T04:53:02.580 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:02 smithi164 conmon[29288]: 2021-02-10T04:53:02.580 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:02 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:53:02.580 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:02 smithi164 conmon[29288]: -10T04:53:01.929162+0000 osd.6 (osd.6) 173 : cluster [DBG] 7.19 scrub ok 2021-02-10T04:53:02.581 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:02 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:53:02.581 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:02 smithi164 conmon[29288]: 10T04:53:01.972907+0000 osd.2 (osd.2) 2016 : 2021-02-10T04:53:02.581 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:02 smithi164 conmon[29288]: cluster [DBG] 5.6 scrub ok 2021-02-10T04:53:02.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:02 smithi085 conmon[29319]: cluster 2021-02-10T04:53:00.343224+0000 osd.5 (osd.5) 175 : cluster 2021-02-10T04:53:02.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:02 smithi085 conmon[29319]: [DBG] 2.16 scrub ok 2021-02-10T04:53:02.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:02 smithi085 conmon[29319]: cluster 2021-02-10T04:53:00.443122+0000 mgr.y (mgr.14140) 1268 : cluster [DBG] pgmap v2089: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 899 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 738 B/s rd, 0 op/s; 278/2235 objects degraded (12.438%); 25/2235 objects misplaced (1.119%) 2021-02-10T04:53:02.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:02 smithi085 conmon[29319]: cluster 2021-02-10T04:53:01.304622+0000 osd.1 (osd.1) 493 : cluster [DBG] 4.2 scrub ok 2021-02-10T04:53:02.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:02 smithi085 conmon[29319]: cluster 2021-02-10T04:53:01.929162+0000 osd.6 (osd.6) 173 : cluster [DBG] 7.19 scrub ok 2021-02-10T04:53:02.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:02 smithi085 conmon[29319]: cluster 2021-02-10T04:53:01.972907+0000 osd.2 (osd.2) 2016 : cluster [DBG] 5.6 scrub ok 2021-02-10T04:53:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:02 smithi085 conmon[35961]: cluster 2021-02-10T04:53: 2021-02-10T04:53:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:02 smithi085 conmon[35961]: 00.343224+0000 osd.5 (osd.5) 175 : cluster [DBG] 2.16 scrub ok 2021-02-10T04:53:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:02 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:53:02.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:02 smithi085 conmon[35961]: -10T04:53:00.443122+0000 mgr.y ( 2021-02-10T04:53:02.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:02 smithi085 conmon[35961]: mgr.14140) 1268 : cluster [DBG] pgmap v2089: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 899 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 738 B/s rd, 0 op/s; 278/2235 objects degraded (12.438%); 25/2235 objects misplaced (1.119%) 2021-02-10T04:53:02.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:02 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:53:02.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:02 smithi085 conmon[35961]: -02-10T04:53:01.304622+0000 2021-02-10T04:53:02.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:02 smithi085 conmon[35961]: osd.1 (osd.1) 493 : cluster [DBG] 2021-02-10T04:53:02.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:02 smithi085 conmon[35961]: 4.2 scrub ok 2021-02-10T04:53:02.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:02 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:53:02.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:02 smithi085 conmon[35961]: :53:01.929162+0000 osd.6 (osd. 2021-02-10T04:53:02.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:02 smithi085 conmon[35961]: 6) 173 : cluster [DBG] 7.19 scrub ok 2021-02-10T04:53:02.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:02 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:53:02.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:02 smithi085 conmon[35961]: -02-10T04:53:01.972907 2021-02-10T04:53:02.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:02 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 2016 : cluster 2021-02-10T04:53:02.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:02 smithi085 conmon[35961]: [DBG] 5.6 scrub ok 2021-02-10T04:53:03.036 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:53:03.276 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:53:03.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:03 smithi085 conmon[29319]: cluster 2021-02-10T04:53:01.331463+0000 2021-02-10T04:53:03.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:03 smithi085 conmon[29319]: osd.5 (osd.5) 176 : cluster [DBG] 2.18 scrub ok 2021-02-10T04:53:03.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:03 smithi085 conmon[29319]: cluster 2021-02-10T04:53:02.112528+0000 osd.3 (osd.3) 24 : cluster [DBG] 6.c scrub ok 2021-02-10T04:53:03.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:03 smithi085 conmon[29319]: cluster 2021-02-10T04:53:02.924660+0000 osd.2 (osd.2) 2017 : cluster [DBG] 4.1c scrub ok 2021-02-10T04:53:03.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:03 smithi085 conmon[35961]: cluster 2021-02-10T04:53:01.331463+0000 osd.5 (osd. 2021-02-10T04:53:03.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:03 smithi085 conmon[35961]: 5) 176 : cluster [DBG] 2.18 scrub ok 2021-02-10T04:53:03.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:03 smithi085 conmon[35961]: cluster 2021-02-10T04:53:02.112528+0000 osd.3 (osd.3) 24 : cluster [DBG] 6.c scrub ok 2021-02-10T04:53:03.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:03 smithi085 conmon[35961]: cluster 2021-02-10T04:53:02.924660+0000 osd.2 (osd.2) 2017 : cluster [DBG] 4.1c scrub ok 2021-02-10T04:53:03.443 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset nodeep-scrub 2021-02-10T04:53:03.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:03 smithi164 conmon[29288]: cluster 2021-02-10T04:53:01.331463+0000 osd.5 (osd.5) 176 : cluster 2021-02-10T04:53:03.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:03 smithi164 conmon[29288]: [DBG] 2.18 scrub ok 2021-02-10T04:53:03.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:03 smithi164 conmon[29288]: cluster 2021-02-10T04:53:02.112528+0000 osd.3 (osd.3) 24 : cluster [DBG] 6.c scrub ok 2021-02-10T04:53:03.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:03 smithi164 conmon[29288]: cluster 2021-02-10T04:53:02.924660+0000 osd.2 (osd.2) 2017 : cluster [DBG] 4.1c scrub ok 2021-02-10T04:53:03.689 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:53:04.023 DEBUG:teuthology.orchestra.run.smithi085:> sudo rm -f /var/log/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/exp.2.1.0 2021-02-10T04:53:04.317 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:04 smithi085 conmon[29319]: cluster 2021-02-10T04:53:02.443729+0000 mgr.y (mgr.14140) 1269 : cluster [DBG] pgmap v2090: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 900 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 278/2238 objects degraded (12.422%); 25/2238 objects misplaced (1.117%) 2021-02-10T04:53:04.318 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:04 smithi085 conmon[29319]: cluster 2021-02-10T04:53:03.202178+0000 mon.a (mon.0) 2453 : cluster [WRN] Health check update: Degraded data redundancy: 278/2238 objects degraded (12.422%), 29 pgs degraded (PG_DEGRADED) 2021-02-10T04:53:04.318 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:04 smithi085 conmon[29319]: cluster 2021-02-10T04:53:03.228176+0000 osd.1 (osd.1) 494 : cluster [DBG] 1.3 scrub ok 2021-02-10T04:53:04.318 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:04 smithi085 conmon[29319]: cluster 2021-02-10T04:53:03.511993+0000 mon.a (mon.0) 2454 : cluster [DBG] osdmap e1159: 8 total, 7 up, 8 in 2021-02-10T04:53:04.318 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:04 smithi085 conmon[29319]: cluster 2021-02-10T04:53:03.915400+0000 osd.2 (osd.2) 2018 : cluster [DBG] 7.20 scrub ok 2021-02-10T04:53:04.318 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:04 smithi085 conmon[35961]: cluster 2021-02-10T04:53:02.443729 2021-02-10T04:53:04.318 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:04 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1269 : cluster [DBG] pgmap v2090: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 900 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 278/2238 objects degraded (12.422%); 25/2238 objects misplaced (1.117%) 2021-02-10T04:53:04.318 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:04 smithi085 conmon[35961]: cluster 2021-02-10T04:53:03.202178+0000 mon.a 2021-02-10T04:53:04.318 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:04 smithi085 conmon[35961]: (mon.0) 2453 : cluster [WRN] Health check update: Degraded data redundancy: 278/2238 objects degraded (12.422%), 29 pgs degraded (PG_DEGRADED) 2021-02-10T04:53:04.318 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:04 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:53:04.318 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:04 smithi085 conmon[35961]: -02-10T04:53:03.228176+0000 osd.1 (osd.1) 494 : cluster 2021-02-10T04:53:04.319 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:04 smithi085 conmon[35961]: [DBG] 1.3 scrub ok 2021-02-10T04:53:04.319 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:04 smithi085 conmon[35961]: cluster 2021-02-10T04:53:04.319 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:04 smithi085 conmon[35961]: 2021-02-10T04:53:03.511993+0000 mon.a ( 2021-02-10T04:53:04.319 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:04 smithi085 conmon[35961]: mon.0) 2454 : cluster [DBG] osdmap e1159: 8 total, 7 up, 8 in 2021-02-10T04:53:04.319 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:04 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:53:04.319 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:04 smithi085 conmon[35961]: 10T04:53:03.915400+0000 osd.2 (osd 2021-02-10T04:53:04.319 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:04 smithi085 conmon[35961]: .2) 2018 : cluster [DBG] 7.20 scrub ok 2021-02-10T04:53:04.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:04 smithi164 conmon[29288]: cluster 2021-02-10T04:53:02.443729+0000 mgr.y (mgr.14140) 2021-02-10T04:53:04.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:04 smithi164 conmon[29288]: 1269 : cluster [DBG] pgmap v2090: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 900 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 278/2238 objects degraded (12.422%); 25/2238 objects misplaced (1.117%) 2021-02-10T04:53:04.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:04 smithi164 conmon[29288]: cluster 2021-02-10T04:53:03.202178+0000 mon.a (mon.0) 2453 : cluster [WRN] Health check update: Degraded data redundancy: 278/2238 objects degraded (12.422%), 29 pgs degraded (PG_DEGRADED) 2021-02-10T04:53:04.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:04 smithi164 conmon[29288]: cluster 2021-02-10T04:53:03.228176+0000 osd.1 2021-02-10T04:53:04.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:04 smithi164 conmon[29288]: (osd.1) 494 : cluster [DBG] 1.3 scrub ok 2021-02-10T04:53:04.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:04 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:53:04.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:04 smithi164 conmon[29288]: T04:53:03.511993+0000 mon.a (mon.0) 2021-02-10T04:53:04.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:04 smithi164 conmon[29288]: 2454 : cluster [DBG] osdmap e1159: 8 total, 7 up, 8 in 2021-02-10T04:53:04.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:04 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:53:04.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:04 smithi164 conmon[29288]: -02-10T04:53:03.915400+0000 2021-02-10T04:53:04.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:04 smithi164 conmon[29288]: osd.2 (osd.2) 2018 : cluster [DBG] 7.20 scrub ok 2021-02-10T04:53:05.487 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:05 smithi085 conmon[29319]: cluster 2021-02-10T04:53:03.933903+0000 2021-02-10T04:53:05.487 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:05 smithi085 conmon[29319]: osd.4 (osd.4) 2643 : cluster [DBG] 2.26 scrub ok 2021-02-10T04:53:05.487 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:05 smithi085 conmon[29319]: cluster 2021-02-10T04:53:04.058242+0000 osd.3 (osd.3) 25 : cluster [DBG] 4.10 scrub ok 2021-02-10T04:53:05.487 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:05 smithi085 conmon[29319]: cluster 2021-02-10T04:53:04.354401+0000 osd.5 (osd.5) 177 : cluster [DBG] 5.19 scrub ok 2021-02-10T04:53:05.488 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:05 smithi085 conmon[35961]: cluster 2021-02-10T04:53:03.933903+0000 osd.4 (osd.4) 2021-02-10T04:53:05.488 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:05 smithi085 conmon[35961]: 2643 : cluster [DBG] 2.26 scrub ok 2021-02-10T04:53:05.488 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:05 smithi085 conmon[35961]: cluster 2021-02-10T04:53:04.058242+0000 osd.3 (osd.3) 25 : cluster [DBG] 4.10 scrub ok 2021-02-10T04:53:05.488 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:05 smithi085 conmon[35961]: cluster 2021-02-10T04:53:04.354401+0000 osd.5 (osd.5) 177 : cluster [DBG] 5.19 scrub ok 2021-02-10T04:53:05.545 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:05 smithi164 conmon[29288]: cluster 2021-02-10T04:53:03.933903+0000 2021-02-10T04:53:05.545 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:05 smithi164 conmon[29288]: osd.4 (osd.4) 2643 : cluster [DBG] 2.26 scrub ok 2021-02-10T04:53:05.545 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:05 smithi164 conmon[29288]: cluster 2021-02-10T04:53:04.058242 2021-02-10T04:53:05.545 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:05 smithi164 conmon[29288]: +0000 osd.3 (osd.3) 25 : cluster [DBG] 4.10 scrub ok 2021-02-10T04:53:05.545 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:05 smithi164 conmon[29288]: cluster 2021-02-10T04:53:05.546 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:05 smithi164 conmon[29288]: 2021-02-10T04:53:04.354401+0000 2021-02-10T04:53:05.546 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:05 smithi164 conmon[29288]: osd.5 (osd.5) 177 : cluster [DBG] 5.19 scrub ok 2021-02-10T04:53:06.403 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:06 smithi085 conmon[29319]: cluster 2021-02-10T04:53:04.444265+0000 mgr.y ( 2021-02-10T04:53:06.404 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:06 smithi085 conmon[29319]: mgr.14140) 1270 : cluster [DBG] pgmap v2092: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 900 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 278/2238 objects degraded (12.422%); 25/2238 objects misplaced (1.117%) 2021-02-10T04:53:06.404 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:06 smithi085 conmon[29319]: cluster 2021-02-10T04:53:05.984653+0000 osd.6 (osd.6) 174 : cluster [DBG] 2.7 scrub ok 2021-02-10T04:53:06.404 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:06 smithi085 conmon[35961]: cluster 2021-02-10T04:53:04.444265+0000 mgr.y (mgr.14140 2021-02-10T04:53:06.404 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:06 smithi085 conmon[35961]: ) 1270 : cluster [DBG] pgmap v2092: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 900 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 278/2238 objects degraded (12.422%); 25/2238 objects misplaced (1.117%) 2021-02-10T04:53:06.404 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:06 smithi085 conmon[35961]: cluster 2021-02-10T04:53:05.984653+0000 osd.6 ( 2021-02-10T04:53:06.404 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:06 smithi085 conmon[35961]: osd.6) 174 : cluster [DBG] 2.7 scrub ok 2021-02-10T04:53:06.432 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:53:06.432 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:53:06.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:06 smithi164 conmon[29288]: cluster 2021-02-10T04:53:04.444265+0000 mgr.y (mgr.14140) 1270 : cluster [DBG] pgmap v2092: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 900 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 278/2238 objects degraded (12.422%); 25/2238 objects misplaced (1.117%) 2021-02-10T04:53:06.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:06 smithi164 conmon[29288]: cluster 2021-02-10T04:53:05.984653+0000 osd.6 (osd.6) 174 : cluster [DBG] 2.7 scrub ok 2021-02-10T04:53:07.024 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:53:07.045 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:53:07.072 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:53:07.099 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:53:07.123 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:53:07.141 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:53:07.158 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:53:07.158 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-02-10T04:53:07.158 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: problem getting command descriptions from osd.0 2021-02-10T04:53:07.158 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:53:07.159 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:53:07.159 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:53:07.159 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:53:07.159 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:53:07.159 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:53:07.159 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:53:07.226 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is unset 2021-02-10T04:53:07.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[29319]: cluster 2021-02-10T04:53:06.077712+0000 2021-02-10T04:53:07.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[29319]: osd.3 (osd.3) 26 : cluster [DBG] 4.1d scrub ok 2021-02-10T04:53:07.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[29319]: audit 2021-02-10T04:53:07.003190+0000 mon.a (mon.0) 2455 : audit [DBG] from='client.? 172.21.15.85:0/2340265084' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:53:07.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[29319]: cluster 2021-02-10T04:53:07.024973+0000 osd.6 (osd.6) 175 : cluster [DBG] 1.7 scrub ok 2021-02-10T04:53:07.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[29319]: audit 2021-02-10T04:53:07.187806+0000 mon.c (mon.1) 106 : audit [INF] from='client.? 172.21.15.85:0/3498718327' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:53:07.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[29319]: audit 2021-02-10T04:53:07. 2021-02-10T04:53:07.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[29319]: 188129+0000 mon.a (mon.0) 2456 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:53:07.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[35961]: cluster 2021-02-10T04:53:07.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[35961]: 2021-02-10T04:53:07.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[35961]: 2021 2021-02-10T04:53:07.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[35961]: - 2021-02-10T04:53:07.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[35961]: 02 2021-02-10T04:53:07.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[35961]: - 2021-02-10T04:53:07.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[35961]: 10 2021-02-10T04:53:07.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[35961]: T 2021-02-10T04:53:07.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[35961]: 04 2021-02-10T04:53:07.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[35961]: : 2021-02-10T04:53:07.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[35961]: 53 2021-02-10T04:53:07.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[35961]: : 2021-02-10T04:53:07.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[35961]: 06 2021-02-10T04:53:07.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[35961]: . 2021-02-10T04:53:07.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[35961]: 077712 2021-02-10T04:53:07.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[35961]: +0000 2021-02-10T04:53:07.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[35961]: osd.3 (osd.3) 26 : cluster 2021-02-10T04:53:07.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[35961]: [DBG] 4.1d scrub ok 2021-02-10T04:53:07.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[35961]: audit 2021-02-10T04:53:07. 2021-02-10T04:53:07.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[35961]: 003190+0000 mon.a (mon.0) 2455 : audit [DBG] 2021-02-10T04:53:07.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/2340265084' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:53:07.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[35961]: cluster 2021-02-10T04:53:07. 2021-02-10T04:53:07.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[35961]: 024973+0000 osd.6 (osd.6) 175 : cluster [DBG] 2021-02-10T04:53:07.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[35961]: 1.7 scrub ok 2021-02-10T04:53:07.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[35961]: audit 2021-02-10T04:53:07 2021-02-10T04:53:07.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[35961]: .187806+0000 mon.c (mon.1) 106 : audit [INF] 2021-02-10T04:53:07.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/3498718327' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:53:07.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[35961]: audit 2021-02-10T04:53:07.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[35961]: 2021-02-10T04:53:07.188129+0000 mon.a (mon.0) 2456 : 2021-02-10T04:53:07.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:07 smithi085 conmon[35961]: audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:53:07.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:07 smithi164 conmon[29288]: cluster 2021-02-10T04:53:07.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:07 smithi164 conmon[29288]: 2021-02 2021-02-10T04:53:07.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:07 smithi164 conmon[29288]: -10T04:53:06.077712+0000 osd.3 (osd.3) 26 : cluster [DBG] 4.1d scrub ok 2021-02-10T04:53:07.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:07 smithi164 conmon[29288]: audit 2021-02-10T04:53:07.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:07 smithi164 conmon[29288]: 2021- 2021-02-10T04:53:07.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:07 smithi164 conmon[29288]: 02-10T04:53:07.003190+0000 mon.a (mon.0) 2021-02-10T04:53:07.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:07 smithi164 conmon[29288]: 2455 : audit [DBG] from='client.? 172.21.15.85:0/2340265084' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:53:07.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:07 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:53:07.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:07 smithi164 conmon[29288]: -02-10T04:53:07.024973+0000 2021-02-10T04:53:07.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:07 smithi164 conmon[29288]: osd.6 (osd.6) 175 : cluster [DBG] 1.7 scrub ok 2021-02-10T04:53:07.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:07 smithi164 conmon[29288]: 2021-02-10T04:53:07.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:07 smithi164 conmon[29288]: audit 2021-02-10T04:53:07.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:07 smithi164 conmon[29288]: 2021-02-10T04:53:07.187806+0000 mon.c ( 2021-02-10T04:53:07.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:07 smithi164 conmon[29288]: mon.1) 106 : audit [INF] from='client.? 172.21.15.85:0/3498718327' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:53:07.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:07 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:53:07.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:07 smithi164 conmon[29288]: 02-10T04:53:07.188129+0000 2021-02-10T04:53:07.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:07 smithi164 conmon[29288]: mon.a (mon.0) 2456 : audit [INF] 2021-02-10T04:53:07.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:07 smithi164 conmon[29288]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:53:08.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[29319]: cluster 2021-02-10T04:53:06.444841+0000 2021-02-10T04:53:08.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[29319]: mgr.y (mgr.14140) 1271 : cluster [DBG] pgmap v2093: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 905 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 278/2241 objects degraded (12.405%); 25/2241 objects misplaced (1.116%) 2021-02-10T04:53:08.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:53:08.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[29319]: :53:07.091944+0000 osd.3 (osd.3) 27 : cluster [DBG] 2021-02-10T04:53:08.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[29319]: 5.18 scrub ok 2021-02-10T04:53:08.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:53:08.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[29319]: 02-10T04:53:07. 2021-02-10T04:53:08.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[29319]: 213247+0000 mon.a (mon.0) 2457 : cluster 2021-02-10T04:53:08.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[29319]: [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:53:08.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[29319]: audit 2021-02-10T04:53:07 2021-02-10T04:53:08.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[29319]: .220242+0000 mon.a (mon.0) 2458 : audit [INF] 2021-02-10T04:53:08.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[29319]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:53:08.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:53:08.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[29319]: 53:07.220303+0000 mon.a (mon.0) 2459 : cluster [DBG] 2021-02-10T04:53:08.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[29319]: osdmap e1160: 8 total, 7 up, 8 in 2021-02-10T04:53:08.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:53:08.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[29319]: 10T04:53:07.324558+0000 osd.5 (osd.5) 2021-02-10T04:53:08.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[29319]: 178 : cluster [DBG] 5.1b scrub ok 2021-02-10T04:53:08.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[35961]: cluster 2021-02-10T04:53:06.444841+0000 mgr.y (mgr. 2021-02-10T04:53:08.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[35961]: 14140) 1271 : cluster [DBG] pgmap v2093: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 905 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 278/2241 objects degraded (12.405%); 25/2241 objects misplaced (1.116%) 2021-02-10T04:53:08.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[35961]: cluster 2021-02-10T04:53:07.091944+0000 osd.3 (osd.3 2021-02-10T04:53:08.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[35961]: ) 27 : cluster [DBG] 5.18 scrub ok 2021-02-10T04:53:08.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:53:08.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[35961]: T04:53:07.213247+0000 mon.a (mon.0) 2457 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:53:08.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:53:08.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[35961]: 10T04:53:07.220242+0000 mon.a ( 2021-02-10T04:53:08.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[35961]: mon.0) 2458 : audit [INF] 2021-02-10T04:53:08.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[35961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:53:08.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:53:08.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[35961]: -10T04:53:07.220303+0000 2021-02-10T04:53:08.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[35961]: mon.a (mon.0) 2459 : cluster [DBG] 2021-02-10T04:53:08.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[35961]: osdmap e1160: 8 total, 7 up, 8 in 2021-02-10T04:53:08.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[35961]: cluster 2021-02-10T04:53:08.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[35961]: 2021-02-10T04:53 2021-02-10T04:53:08.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[35961]: :07.324558+0000 osd.5 ( 2021-02-10T04:53:08.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:08 smithi085 conmon[35961]: osd.5) 178 : cluster [DBG] 5.1b scrub ok 2021-02-10T04:53:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:08 smithi164 conmon[29288]: cluster 2021-02-10T04:53:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:08 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:53:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:08 smithi164 conmon[29288]: 53:06.444841+0000 mgr.y (mgr.14140) 1271 : cluster [DBG] pgmap v2093: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 905 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 278/2241 objects degraded (12.405%); 25/2241 objects misplaced (1.116%) 2021-02-10T04:53:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:08 smithi164 conmon[29288]: cluster 2021-02-10T04:53:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:08 smithi164 conmon[29288]: 2021-02-10T04:53:07.091944+0000 osd.3 (osd.3) 27 : cluster [DBG] 5.18 scrub ok 2021-02-10T04:53:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:08 smithi164 conmon[29288]: cluster 2021-02-10T04:53:07. 2021-02-10T04:53:08.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:08 smithi164 conmon[29288]: 213247+0000 mon.a (mon.0) 2457 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:53:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:08 smithi164 conmon[29288]: audit 2021-02-10T04:53:07.220242+0000 2021-02-10T04:53:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:08 smithi164 conmon[29288]: mon.a (mon.0) 2458 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:53:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:08 smithi164 conmon[29288]: cluster 2021-02-10T04:53:07.220303+0000 mon.a (mon.0) 2459 : cluster [DBG] osdmap e1160: 8 total, 7 up, 8 in 2021-02-10T04:53:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:08 smithi164 conmon[29288]: cluster 2021-02-10T04:53:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:08 smithi164 conmon[29288]: 2021-02-10T04:53:07.324558+0000 osd.5 (osd.5) 178 : 2021-02-10T04:53:08.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:08 smithi164 conmon[29288]: cluster [DBG] 5.1b scrub ok 2021-02-10T04:53:09.003 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:53:09.003 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:53:09.068 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2021-02-10T04:53:09.069 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:53:09.069 INFO:tasks.thrashosds.thrasher:Shrinking pool zone.rgw.log 2021-02-10T04:53:09.069 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool zone.rgw.log by 10 min 8 2021-02-10T04:53:09.069 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:53:09.302 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:53:09.559 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:09 smithi085 conmon[29319]: cluster 2021-02-10T04:53:08.506731+0000 mon.a (mon.0) 2460 : cluster [WRN] Health check update: Degraded data redundancy: 278/2241 objects degraded (12.405%), 29 pgs degraded (PG_DEGRADED) 2021-02-10T04:53:09.560 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:09 smithi085 conmon[29319]: cluster 2021-02-10T04:53:08.511477+0000 mon.a (mon.0) 2461 : cluster [DBG] osdmap e1161: 8 total, 7 up, 8 in 2021-02-10T04:53:09.560 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:09 smithi085 conmon[29319]: cluster 2021-02-10T04:53:09.005418+0000 osd.6 (osd.6) 176 : cluster [DBG] 6.3 scrub ok 2021-02-10T04:53:09.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:09 smithi085 conmon[35961]: cluster 2021-02-10T04:53:08.506731+0000 mon.a (mon.0) 2460 : cluster [WRN] Health check update: Degraded data redundancy: 278/2241 objects degraded (12.405%), 29 pgs degraded (PG_DEGRADED) 2021-02-10T04:53:09.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:09 smithi085 conmon[35961]: cluster 2021-02-10T04:53:08.511477+0000 mon.a (mon.0) 2461 : cluster [DBG] osdmap e1161: 8 total, 7 up, 8 in 2021-02-10T04:53:09.560 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:09 smithi085 conmon[35961]: cluster 2021-02-10T04:53:09.005418+0000 osd.6 (osd.6) 176 : cluster [DBG] 6.3 scrub ok 2021-02-10T04:53:09.644 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:53:09 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:53:09.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:09 smithi164 conmon[29288]: cluster 2021-02-10T04:53: 2021-02-10T04:53:09.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:09 smithi164 conmon[29288]: 08.506731+0000 mon.a (mon.0) 2460 : cluster [WRN] Health check update: Degraded data redundancy: 278/2241 objects degraded (12.405%), 29 pgs degraded (PG_DEGRADED) 2021-02-10T04:53:09.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:09 smithi164 conmon[29288]: cluster 2021-02-10T04:53:08.511477+0000 mon.a (mon.0) 2461 : cluster [DBG] osdmap e1161: 8 total, 7 up, 8 in 2021-02-10T04:53:09.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:09 smithi164 conmon[29288]: 2021-02-10T04:53:09.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:09 smithi164 conmon[29288]: cluster 2021-02-10T04:53:09.005418+0000 osd.6 (osd.6) 176 : 2021-02-10T04:53:09.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:09 smithi164 conmon[29288]: cluster [DBG] 6.3 scrub ok 2021-02-10T04:53:10.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:10 smithi164 conmon[29288]: cluster 2021-02-10T04:53:08.445661+0000 mgr.y 2021-02-10T04:53:10.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:10 smithi164 conmon[29288]: (mgr.14140) 1272 : cluster [DBG] pgmap v2095: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 905 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 278/2241 objects degraded (12.405%); 25/2241 objects misplaced (1.116%) 2021-02-10T04:53:10.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:10 smithi164 conmon[29288]: cluster 2021-02-10T04:53:09.002364+0000 osd.4 (osd.4) 2644 : cluster [DBG] 4.e scrub ok 2021-02-10T04:53:10.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:10 smithi164 conmon[29288]: cluster 2021-02-10T04:53:09.121292+0000 osd.3 (osd.3) 28 : cluster [DBG] 2.23 scrub ok 2021-02-10T04:53:10.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:10 smithi164 conmon[29288]: audit 2021-02-10T04:53:09.173985+0000 mgr.y (mgr.14140) 1273 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:53:10.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:10 smithi164 conmon[29288]: cluster 2021-02-10T04:53:10.028357+0000 osd.5 (osd.5) 179 : cluster [DBG] 2021-02-10T04:53:10.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:10 smithi164 conmon[29288]: purged_snaps scrub starts 2021-02-10T04:53:10.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:10 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:53:10.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:10 smithi164 conmon[29288]: 04:53:10.031216+0000 osd.5 (osd 2021-02-10T04:53:10.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:10 smithi164 conmon[29288]: .5) 180 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:53:10.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:10 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:53:10.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:10 smithi164 conmon[29288]: -02-10T04:53:10.075874+0000 2021-02-10T04:53:10.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:10 smithi164 conmon[29288]: osd.2 (osd.2) 2019 : cluster [DBG] 2021-02-10T04:53:10.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:10 smithi164 conmon[29288]: 7.1b scrub ok 2021-02-10T04:53:10.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[29319]: cluster 2021-02-10T04:53:08.445661+0000 mgr.y (mgr.14140 2021-02-10T04:53:10.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[29319]: ) 1272 : cluster [DBG] pgmap v2095: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 905 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 278/2241 objects degraded (12.405%); 25/2241 objects misplaced (1.116%) 2021-02-10T04:53:10.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[29319]: cluster 2021-02-10T04:53:09.002364+0000 osd.4 (osd.4) 2644 : cluster [DBG] 4.e scrub ok 2021-02-10T04:53:10.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[29319]: cluster 2021-02-10T04:53:09.121292+0000 osd.3 (osd.3) 28 : cluster [DBG] 2.23 scrub ok 2021-02-10T04:53:10.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[29319]: audit 2021-02-10T04:53:09.173985+0000 mgr.y (mgr.14140) 1273 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:53:10.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[29319]: cluster 2021-02-10T04:53:10.028357+0000 osd.5 (osd.5) 179 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:53:10.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[29319]: cluster 2021-02-10T04:53:10.031216+0000 osd.5 (osd.5) 180 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:53:10.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[29319]: cluster 2021-02-10T04:53:10.075874+0000 osd.2 (osd.2) 2019 : cluster [DBG] 7.1b scrub ok 2021-02-10T04:53:10.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: cluster 2021-02-10T04:53:10.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 2021-02-10T04:53:10.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 2021 2021-02-10T04:53:10.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: - 2021-02-10T04:53:10.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 02 2021-02-10T04:53:10.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: - 2021-02-10T04:53:10.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 10 2021-02-10T04:53:10.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: T 2021-02-10T04:53:10.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 04 2021-02-10T04:53:10.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: : 2021-02-10T04:53:10.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 53 2021-02-10T04:53:10.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: : 2021-02-10T04:53:10.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 08 2021-02-10T04:53:10.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: . 2021-02-10T04:53:10.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 445661 2021-02-10T04:53:10.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: +0000 2021-02-10T04:53:10.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 2021-02-10T04:53:10.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: mgr.y 2021-02-10T04:53:10.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: ( 2021-02-10T04:53:10.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: mgr 2021-02-10T04:53:10.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: . 2021-02-10T04:53:10.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 14140 2021-02-10T04:53:10.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: ) 2021-02-10T04:53:10.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 1272 2021-02-10T04:53:10.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: : 2021-02-10T04:53:10.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: cluster 2021-02-10T04:53:10.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 2021-02-10T04:53:10.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: [DBG] 2021-02-10T04:53:10.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 2021-02-10T04:53:10.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: pgmap v2095: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 905 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 278/2241 objects degraded (12.405%); 25/2241 objects misplaced (1.116%) 2021-02-10T04:53:10.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 2021-02-10T04:53:10.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: cluster 2021-02-10T04:53:10.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 2021-02-10T04:53:10.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 2021 2021-02-10T04:53:10.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: - 2021-02-10T04:53:10.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 02 2021-02-10T04:53:10.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: - 2021-02-10T04:53:10.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 10 2021-02-10T04:53:10.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: T 2021-02-10T04:53:10.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 04 2021-02-10T04:53:10.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: : 2021-02-10T04:53:10.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 53 2021-02-10T04:53:10.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: : 2021-02-10T04:53:10.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 09 2021-02-10T04:53:10.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: . 2021-02-10T04:53:10.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 002364 2021-02-10T04:53:10.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: +0000 2021-02-10T04:53:10.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 2021-02-10T04:53:10.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: osd.4 2021-02-10T04:53:10.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: ( 2021-02-10T04:53:10.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: osd 2021-02-10T04:53:10.696 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: . 2021-02-10T04:53:10.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 4 2021-02-10T04:53:10.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: ) 2021-02-10T04:53:10.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 2644 2021-02-10T04:53:10.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: : 2021-02-10T04:53:10.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: cluster 2021-02-10T04:53:10.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 2021-02-10T04:53:10.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: [DBG] 2021-02-10T04:53:10.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 2021-02-10T04:53:10.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 4.e scrub ok 2021-02-10T04:53:10.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 2021-02-10T04:53:10.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: cluster 2021-02-10T04:53:10.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 2021-02-10T04:53:10.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 2021 2021-02-10T04:53:10.697 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: - 2021-02-10T04:53:10.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 02 2021-02-10T04:53:10.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: - 2021-02-10T04:53:10.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 10 2021-02-10T04:53:10.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: T 2021-02-10T04:53:10.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 04 2021-02-10T04:53:10.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: : 2021-02-10T04:53:10.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 53 2021-02-10T04:53:10.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: : 2021-02-10T04:53:10.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 09 2021-02-10T04:53:10.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: . 2021-02-10T04:53:10.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 121292 2021-02-10T04:53:10.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: +0000 2021-02-10T04:53:10.698 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 2021-02-10T04:53:10.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: osd.3 2021-02-10T04:53:10.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: ( 2021-02-10T04:53:10.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: osd 2021-02-10T04:53:10.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: .3) 28 : cluster [DBG] 2.23 scrub ok 2021-02-10T04:53:10.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: audit 2021-02-10T04:53:09 2021-02-10T04:53:10.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: .173985+0000 mgr.y (mgr.14140) 1273 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:53:10.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 2021-02-10T04:53:10.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: cluster 2021-02-10T04:53:10.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 2021-02-10T04:53: 2021-02-10T04:53:10.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 10.028357+0000 osd.5 (osd 2021-02-10T04:53:10.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: .5) 179 : cluster [DBG] 2021-02-10T04:53:10.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: purged_snaps scrub starts 2021-02-10T04:53:10.699 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: cluster 2021-02-10T04:53:10.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 2021-02-10T04:53:10.031216 2021-02-10T04:53:10.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: +0000 osd.5 (osd.5) 180 : cluster 2021-02-10T04:53:10.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: [DBG] purged_snaps scrub ok 2021-02-10T04:53:10.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:53:10.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: 02-10T04:53:10.075874+0000 2021-02-10T04:53:10.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: osd.2 (osd.2) 2019 2021-02-10T04:53:10.700 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:10 smithi085 conmon[35961]: : cluster [DBG] 7.1b scrub ok 2021-02-10T04:53:10.944 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set noscrub 2021-02-10T04:53:11.004 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:53:11.224 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:53:11.268 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:53:11.309 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.7 dump_ops_in_flight 2021-02-10T04:53:11.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:11 smithi085 conmon[29319]: cluster 2021-02-10T04:53:10.226782+0000 mon.a (mon 2021-02-10T04:53:11.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:11 smithi085 conmon[29319]: .0) 2462 : cluster [DBG] osdmap e1162: 8 total, 7 up, 8 in 2021-02-10T04:53:11.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:11 smithi085 conmon[29319]: cluster 2021-02-10T04:53:10.276350+0000 osd.5 (osd.5) 181 : cluster [DBG] 4.15 scrub ok 2021-02-10T04:53:11.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:11 smithi085 conmon[29319]: cluster 2021-02-10T04:53:10.958572+0000 osd.6 (osd.6) 177 : cluster [DBG] 6.23 scrub ok 2021-02-10T04:53:11.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:11 smithi085 conmon[35961]: cluster 2021-02-10T04:53:10.226782+0000 mon.a 2021-02-10T04:53:11.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:11 smithi085 conmon[35961]: (mon.0) 2462 : cluster [DBG] osdmap e1162: 8 total, 7 up, 8 in 2021-02-10T04:53:11.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:11 smithi085 conmon[35961]: cluster 2021-02-10T04:53:10.276350+0000 osd.5 (osd.5) 181 2021-02-10T04:53:11.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:11 smithi085 conmon[35961]: : cluster [DBG] 4.15 scrub ok 2021-02-10T04:53:11.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:11 smithi085 conmon[35961]: cluster 2021-02-10T04:53:10. 2021-02-10T04:53:11.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:11 smithi085 conmon[35961]: 958572+0000 osd.6 (osd.6) 177 : cluster [DBG] 6.23 scrub ok 2021-02-10T04:53:11.631 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:53:11.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:11 smithi164 conmon[29288]: cluster 2021-02-10T04:53:10.226782 2021-02-10T04:53:11.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:11 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2462 : cluster [DBG] osdmap e1162: 8 total, 7 up, 8 in 2021-02-10T04:53:11.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:11 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:53:11.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:11 smithi164 conmon[29288]: T04:53:10.276350+0000 osd.5 (osd.5) 181 : cluster 2021-02-10T04:53:11.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:11 smithi164 conmon[29288]: [DBG] 4.15 scrub ok 2021-02-10T04:53:11.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:11 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:53:11.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:11 smithi164 conmon[29288]: 02-10T04:53:10.958572+0000 osd.6 (osd 2021-02-10T04:53:11.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:11 smithi164 conmon[29288]: .6) 177 : cluster [DBG] 6.23 scrub ok 2021-02-10T04:53:12.327 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:53:12.331 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:53:12.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:12 smithi164 conmon[29288]: cluster 2021-02-10T04:53:10.446181+0000 mgr.y (mgr.14140) 1274 : 2021-02-10T04:53:12.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:12 smithi164 conmon[29288]: cluster [DBG] pgmap v2098: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 905 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 278/2241 objects degraded (12.405%); 25/2241 objects misplaced (1.116%) 2021-02-10T04:53:12.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:12 smithi164 conmon[29288]: cephadm 2021-02-10T04:53:10.904683+0000 mgr.y (mgr.14140) 1275 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:53:12.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:12 smithi164 conmon[29288]: cluster 2021-02-10T04:53:11.051021+0000 osd.4 (osd.4) 2645 : cluster [DBG] 7.15 scrub ok 2021-02-10T04:53:12.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:12 smithi164 conmon[29288]: cluster 2021-02-10T04:53:11.139595+0000 osd.3 (osd.3) 29 : cluster [DBG] 2.28 scrub ok 2021-02-10T04:53:12.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:12 smithi164 conmon[29288]: cluster 2021-02-10T04:53:11.271453+0000 osd.1 (osd.1) 495 : cluster [DBG] 2.9 scrub ok 2021-02-10T04:53:12.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:12 smithi164 conmon[29288]: audit 2021-02-10T04:53:11.412345+0000 mon.a (mon.0) 2463 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:53:12.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:12 smithi164 conmon[29288]: audit 2021-02-10T04:53:11.910674+0000 mon.a (mon.0) 2464 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:53:12.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:12 smithi085 conmon[29319]: cluster 2021-02-10T04:53:10.446181+0000 mgr.y (mgr.14140) 1274 : cluster [DBG] pgmap v2098: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 905 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 278/2241 objects degraded (12.405%); 25/2241 objects misplaced (1.116%) 2021-02-10T04:53:12.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:12 smithi085 conmon[29319]: cephadm 2021-02-10T04:53:10.904683+0000 mgr.y (mgr.14140) 1275 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:53:12.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:12 smithi085 conmon[29319]: cluster 2021-02-10T04:53:11.051021+0000 osd.4 (osd.4) 2645 : cluster [DBG] 7.15 scrub ok 2021-02-10T04:53:12.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:12 smithi085 conmon[29319]: cluster 2021-02-10T04:53:11.139595+0000 osd.3 (osd.3) 29 : cluster [DBG] 2.28 scrub ok 2021-02-10T04:53:12.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:12 smithi085 conmon[29319]: cluster 2021-02-10T04:53:11.271453+0000 osd.1 (osd.1) 495 : cluster [DBG] 2.9 scrub ok 2021-02-10T04:53:12.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:12 smithi085 conmon[29319]: audit 2021-02-10T04:53:11.412345+0000 mon.a (mon.0) 2463 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:53:12.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:12 smithi085 conmon[29319]: audit 2021-02-10T04:53:11.910674+0000 mon.a (mon.0) 2464 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:53:12.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:12 smithi085 conmon[35961]: cluster 2021-02-10T04:53:10.446181+0000 mgr.y (mgr.14140) 1274 : cluster [DBG] pgmap v2098: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 905 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 278/2241 objects degraded (12.405%); 25/2241 objects misplaced (1.116%) 2021-02-10T04:53:12.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:12 smithi085 conmon[35961]: cephadm 2021-02-10T04:53:10.904683+0000 mgr.y (mgr.14140) 1275 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:53:12.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:12 smithi085 conmon[35961]: cluster 2021-02-10T04:53:11.051021+0000 osd.4 (osd.4) 2645 : cluster [DBG] 7.15 scrub ok 2021-02-10T04:53:12.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:12 smithi085 conmon[35961]: cluster 2021-02-10T04:53:11.139595+0000 osd.3 (osd.3) 29 : cluster [DBG] 2021-02-10T04:53:12.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:12 smithi085 conmon[35961]: 2.28 scrub ok 2021-02-10T04:53:12.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:12 smithi085 conmon[35961]: cluster 2021-02-10T04:53:11.271453+0000 osd.1 (osd.1) 495 : cluster [DBG] 2.9 scrub ok 2021-02-10T04:53:12.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:12 smithi085 conmon[35961]: audit 2021-02-10T04:53:11.412345+0000 mon.a (mon.0) 2463 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:53:12.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:12 smithi085 conmon[35961]: audit 2021-02-10T04:53:11.910674+0000 mon.a (mon.0) 2464 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:53:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:13 smithi164 conmon[29288]: cephadm 2021-02-10T04:53:11.412913+0000 mgr.y (mgr.14140) 2021-02-10T04:53:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:13 smithi164 conmon[29288]: 1276 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:53:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:13 smithi164 conmon[29288]: cluster 2021-02-10T04:53:12.187149+0000 osd.3 (osd.3) 30 : cluster [DBG] 6.6 scrub ok 2021-02-10T04:53:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:13 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:53:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:13 smithi164 conmon[29288]: :53:12.308040+0000 mgr.y (mgr.14140) 1277 : audit [DBG] from='client.15795 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:53:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:13 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:53:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:13 smithi164 conmon[29288]: 10T04:53:12.332226+0000 osd.5 (osd.5) 182 : cluster [DBG] 5.16 scrub ok 2021-02-10T04:53:13.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:13 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:53:13.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:13 smithi164 conmon[29288]: 04:53:12.951522+0000 osd.6 (osd.6) 178 2021-02-10T04:53:13.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:13 smithi164 conmon[29288]: : cluster [DBG] 6.22 scrub ok 2021-02-10T04:53:13.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:13 smithi085 conmon[29319]: cephadm 2021-02-10T04:53:11.412913+0000 mgr.y (mgr 2021-02-10T04:53:13.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:13 smithi085 conmon[29319]: .14140) 1276 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:53:13.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:13 smithi085 conmon[29319]: cluster 2021-02-10T04:53:12.187149+0000 osd.3 (osd.3) 30 : cluster [DBG] 6.6 scrub ok 2021-02-10T04:53:13.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:13 smithi085 conmon[29319]: audit 2021-02-10T04:53:12.308040+0000 2021-02-10T04:53:13.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:13 smithi085 conmon[29319]: mgr.y (mgr.14140) 1277 : audit [DBG] from='client.15795 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:53:13.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:13 smithi085 conmon[29319]: cluster 2021-02-10T04:53:12.332226+0000 osd.5 (osd. 2021-02-10T04:53:13.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:13 smithi085 conmon[29319]: 5) 182 : cluster [DBG] 5.16 scrub ok 2021-02-10T04:53:13.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:13 smithi085 conmon[29319]: cluster 2021-02-10T04:53: 2021-02-10T04:53:13.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:13 smithi085 conmon[29319]: 12.951522+0000 osd.6 (osd.6) 178 : cluster [DBG] 6.22 scrub ok 2021-02-10T04:53:13.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:13 smithi085 conmon[35961]: cephadm 2021-02-10T04:53:11.412913+0000 mgr.y (mgr.14140) 1276 2021-02-10T04:53:13.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:13 smithi085 conmon[35961]: : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:53:13.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:13 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:53:13.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:13 smithi085 conmon[35961]: 02-10T04:53:12.187149+0000 osd.3 (osd.3) 30 : cluster 2021-02-10T04:53:13.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:13 smithi085 conmon[35961]: [DBG] 6.6 scrub ok 2021-02-10T04:53:13.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:13 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:53:13.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:13 smithi085 conmon[35961]: 53:12.308040+0000 mgr.y (mgr.14140) 1277 : audit [DBG] from='client.15795 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:53:13.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:13 smithi085 conmon[35961]: 2021-02-10T04:53:13.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:13 smithi085 conmon[35961]: cluster 2021-02-10T04:53: 2021-02-10T04:53:13.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:13 smithi085 conmon[35961]: 12.332226+0000 osd.5 (osd.5) 182 : cluster [DBG] 5.16 scrub ok 2021-02-10T04:53:13.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:13 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:53:13.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:13 smithi085 conmon[35961]: -02-10T04:53:12.951522+0000 osd.6 (osd.6) 178 : cluster [DBG] 6.22 scrub ok 2021-02-10T04:53:14.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:14 smithi085 conmon[35961]: cluster 2021-02-10T04:53:12.446887+0000 2021-02-10T04:53:14.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:14 smithi085 conmon[35961]: mgr.y (mgr.14140) 1278 : cluster [DBG] pgmap v2099: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 870 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 278/2202 objects degraded (12.625%); 12/2202 objects misplaced (0.545%) 2021-02-10T04:53:14.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:14 smithi085 conmon[35961]: cluster 2021-02-10T04:53:13.099224+0000 osd.4 (osd.4) 2646 : cluster [DBG] 6.2 scrub ok 2021-02-10T04:53:14.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:14 smithi085 conmon[35961]: cluster 2021-02-10T04:53:13.508426+0000 mon.a (mon.0) 2465 : cluster [WRN] Health check update: Degraded data redundancy: 278/2202 objects degraded (12.625%), 29 pgs degraded (PG_DEGRADED) 2021-02-10T04:53:14.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:14 smithi085 conmon[35961]: cluster 2021-02-10T04:53:13.514073+0000 mon.a (mon.0) 2466 : cluster [DBG] osdmap e1163: 8 total, 7 up, 8 in 2021-02-10T04:53:14.374 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:14 smithi085 conmon[29319]: cluster 2021-02-10T04:53:12.446887+0000 mgr.y (mgr.14140) 1278 : cluster [DBG] 2021-02-10T04:53:14.374 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:14 smithi085 conmon[29319]: pgmap v2099: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 870 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 278/2202 objects degraded (12.625%); 12/2202 objects misplaced (0.545%) 2021-02-10T04:53:14.374 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:14 smithi085 conmon[29319]: cluster 2021-02-10T04:53:13.099224+0000 osd.4 (osd.4) 2646 : cluster [DBG] 6.2 scrub ok 2021-02-10T04:53:14.375 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:14 smithi085 conmon[29319]: cluster 2021-02-10T04:53:13.508426+0000 mon.a (mon.0) 2465 : cluster [WRN] Health check update: Degraded data redundancy: 278/2202 objects degraded (12.625%), 29 pgs degraded (PG_DEGRADED) 2021-02-10T04:53:14.375 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:14 smithi085 conmon[29319]: cluster 2021-02-10T04:53:13.514073+0000 mon.a (mon.0) 2466 : cluster [DBG] osdmap e1163: 8 total, 7 up, 8 in 2021-02-10T04:53:14.483 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":2098,"stamp":"2021-02-10T04:53:10.445814+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":948649351,"num_objects":747,"num_object_clones":337,"num_object_copies":2241,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":278,"num_objects_misplaced":25,"num_objects_unfound":0,"num_objects_dirty":747,"num_whiteouts":12,"num_read":7794,"num_read_kb":5154760,"num_write":4945,"num_write_kb":1396569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":981,"num_bytes_recovered":394303146,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36195,"ondisk_log_size":36195,"up":527,"acting":528,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":597,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2368996,"kb_used_data":1994596,"kb_used_omap":386,"kb_used_meta":373949,"kb_avail":747592220,"statfs":{"total":767960285184,"available":765534433280,"internally_reserved":0,"allocated":2042466304,"data_stored":2620367475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":396036,"internal_metadata":382924028},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4755709,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":-116,"num_read_kb":-113412,"num_write":-93,"num_write_kb":-38002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001940"},"pg_stats":[{"pgid":"6.29","version":"0'0","reported_seq":"61","reported_epoch":"1158","state":"active+undersized","last_fresh":"2021-02-10T04:52:59.513373+0000","last_change":"2021-02-10T04:52:32.089564+0000","last_active":"2021-02-10T04:52:59.513373+0000","last_peered":"2021-02-10T04:52:59.513373+0000","last_clean":"2021-02-10T04:52:30.064708+0000","last_became_active":"2021-02-10T04:52:32.089564+0000","last_became_peered":"2021-02-10T04:52:32.089564+0000","last_unstale":"2021-02-10T04:52:59.513373+0000","last_undegraded":"2021-02-10T04:52:59.513373+0000","last_fullsized":"2021-02-10T04:52:32.077911+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:24.121684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:24.121684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":"55","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.513625+0000","last_change":"2021-02-10T04:52:30.133105+0000","last_active":"2021-02-10T04:52:59.513625+0000","last_peered":"2021-02-10T04:52:59.513625+0000","last_clean":"2021-02-10T04:52:59.513625+0000","last_became_active":"2021-02-10T04:51:36.864926+0000","last_became_peered":"2021-02-10T04:51:36.864926+0000","last_unstale":"2021-02-10T04:52:59.513625+0000","last_undegraded":"2021-02-10T04:52:59.513625+0000","last_fullsized":"2021-02-10T04:52:59.513625+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.133043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.133043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"131'3","reported_seq":"1355","reported_epoch":"1160","state":"active+clean","last_fresh":"2021-02-10T04:53:08.010684+0000","last_change":"2021-02-10T04:51:01.185887+0000","last_active":"2021-02-10T04:53:08.010684+0000","last_peered":"2021-02-10T04:53:08.010684+0000","last_clean":"2021-02-10T04:53:08.010684+0000","last_became_active":"2021-02-10T04:29:07.145525+0000","last_became_peered":"2021-02-10T04:29:07.145525+0000","last_unstale":"2021-02-10T04:53:08.010684+0000","last_undegraded":"2021-02-10T04:53:08.010684+0000","last_fullsized":"2021-02-10T04:53:08.010684+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'3","last_scrub_stamp":"2021-02-10T04:51:01.185820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.185820+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"1132'66","reported_seq":"1247","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.513948+0000","last_change":"2021-02-10T04:51:55.430490+0000","last_active":"2021-02-10T04:53:03.513948+0000","last_peered":"2021-02-10T04:53:03.513948+0000","last_clean":"2021-02-10T04:53:03.513948+0000","last_became_active":"2021-02-10T04:25:49.002058+0000","last_became_peered":"2021-02-10T04:25:49.002058+0000","last_unstale":"2021-02-10T04:53:03.513948+0000","last_undegraded":"2021-02-10T04:53:03.513948+0000","last_fullsized":"2021-02-10T04:53:03.513948+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1132'66","last_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_deep_scrub":"1132'66","last_deep_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.430399+0000","log_size":66,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":95,"num_read_kb":78,"num_write":55,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"59","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.515594+0000","last_change":"2021-02-10T04:51:48.990145+0000","last_active":"2021-02-10T04:52:59.515594+0000","last_peered":"2021-02-10T04:52:59.515594+0000","last_clean":"2021-02-10T04:52:59.515594+0000","last_became_active":"2021-02-10T04:51:36.875000+0000","last_became_peered":"2021-02-10T04:51:36.875000+0000","last_unstale":"2021-02-10T04:52:59.515594+0000","last_undegraded":"2021-02-10T04:52:59.515594+0000","last_fullsized":"2021-02-10T04:52:59.515594+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:51:48.990062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.990062+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"1135","reported_epoch":"1159","state":"active+undersized","last_fresh":"2021-02-10T04:53:03.510856+0000","last_change":"2021-02-10T04:52:32.077502+0000","last_active":"2021-02-10T04:53:03.510856+0000","last_peered":"2021-02-10T04:53:03.510856+0000","last_clean":"2021-02-10T04:52:30.059220+0000","last_became_active":"2021-02-10T04:52:32.077502+0000","last_became_peered":"2021-02-10T04:52:32.077502+0000","last_unstale":"2021-02-10T04:53:03.510856+0000","last_undegraded":"2021-02-10T04:53:03.510856+0000","last_fullsized":"2021-02-10T04:52:32.069476+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:48.408748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:50:48.408748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"1003'1689","reported_seq":"4576","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.685332+0000","last_change":"2021-02-10T04:52:26.031857+0000","last_active":"2021-02-10T04:53:03.685332+0000","last_peered":"2021-02-10T04:53:03.685332+0000","last_clean":"2021-02-10T04:53:03.685332+0000","last_became_active":"2021-02-10T04:52:26.030105+0000","last_became_peered":"2021-02-10T04:52:26.030105+0000","last_unstale":"2021-02-10T04:53:03.685332+0000","last_undegraded":"2021-02-10T04:53:03.685332+0000","last_fullsized":"2021-02-10T04:53:03.685332+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:46:54.814020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:54.814020+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":63260738,"num_objects":24,"num_object_clones":22,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":386,"num_read_kb":409791,"num_write":280,"num_write_kb":110558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"1049'32","reported_seq":"1126","reported_epoch":"1159","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:03.511421+0000","last_change":"2021-02-10T04:52:32.079965+0000","last_active":"2021-02-10T04:53:03.511421+0000","last_peered":"2021-02-10T04:53:03.511421+0000","last_clean":"2021-02-10T04:45:56.690905+0000","last_became_active":"2021-02-10T04:52:32.079965+0000","last_became_peered":"2021-02-10T04:52:32.079965+0000","last_unstale":"2021-02-10T04:53:03.511421+0000","last_undegraded":"2021-02-10T04:52:32.069399+0000","last_fullsized":"2021-02-10T04:52:32.069289+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'32","last_scrub_stamp":"2021-02-10T04:51:51.382040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.382040+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":61,"num_read_kb":44,"num_write":33,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":894,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":10}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"808","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.686344+0000","last_change":"2021-02-10T04:51:49.502917+0000","last_active":"2021-02-10T04:53:03.686344+0000","last_peered":"2021-02-10T04:53:03.686344+0000","last_clean":"2021-02-10T04:53:03.686344+0000","last_became_active":"2021-02-10T04:51:49.502576+0000","last_became_peered":"2021-02-10T04:51:49.502576+0000","last_unstale":"2021-02-10T04:53:03.686344+0000","last_undegraded":"2021-02-10T04:53:03.686344+0000","last_fullsized":"2021-02-10T04:53:03.686344+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:59.824373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:59.824373+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"55","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.512710+0000","last_change":"2021-02-10T04:51:53.038893+0000","last_active":"2021-02-10T04:52:59.512710+0000","last_peered":"2021-02-10T04:52:59.512710+0000","last_clean":"2021-02-10T04:52:59.512710+0000","last_became_active":"2021-02-10T04:51:36.855507+0000","last_became_peered":"2021-02-10T04:51:36.855507+0000","last_unstale":"2021-02-10T04:52:59.512710+0000","last_undegraded":"2021-02-10T04:52:59.512710+0000","last_fullsized":"2021-02-10T04:52:59.512710+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:53.038834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:53.038834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"1143","reported_epoch":"1158","state":"active+undersized","last_fresh":"2021-02-10T04:52:59.512476+0000","last_change":"2021-02-10T04:52:32.089886+0000","last_active":"2021-02-10T04:52:59.512476+0000","last_peered":"2021-02-10T04:52:59.512476+0000","last_clean":"2021-02-10T04:51:27.993389+0000","last_became_active":"2021-02-10T04:52:32.089886+0000","last_became_peered":"2021-02-10T04:52:32.089886+0000","last_unstale":"2021-02-10T04:52:59.512476+0000","last_undegraded":"2021-02-10T04:52:59.512476+0000","last_fullsized":"2021-02-10T04:52:32.083725+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:09.922789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:09.922789+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.8","version":"1150'76","reported_seq":"73","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:53:00.964614+0000","last_change":"2021-02-10T04:53:00.964563+0000","last_active":"2021-02-10T04:53:00.964614+0000","last_peered":"2021-02-10T04:53:00.964614+0000","last_clean":"2021-02-10T04:53:00.964614+0000","last_became_active":"2021-02-10T04:51:36.859215+0000","last_became_peered":"2021-02-10T04:51:36.859215+0000","last_unstale":"2021-02-10T04:53:00.964614+0000","last_undegraded":"2021-02-10T04:53:00.964614+0000","last_fullsized":"2021-02-10T04:53:00.964614+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1150'76","last_scrub_stamp":"2021-02-10T04:53:00.964499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:00.964499+0000","log_size":76,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6285851,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"1049'61","reported_seq":"1244","reported_epoch":"1160","state":"active+clean","last_fresh":"2021-02-10T04:53:07.222278+0000","last_change":"2021-02-10T04:50:54.154530+0000","last_active":"2021-02-10T04:53:07.222278+0000","last_peered":"2021-02-10T04:53:07.222278+0000","last_clean":"2021-02-10T04:53:07.222278+0000","last_became_active":"2021-02-10T04:25:49.015913+0000","last_became_peered":"2021-02-10T04:25:49.015913+0000","last_unstale":"2021-02-10T04:53:07.222278+0000","last_undegraded":"2021-02-10T04:53:07.222278+0000","last_fullsized":"2021-02-10T04:53:07.222278+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'61","last_scrub_stamp":"2021-02-10T04:50:54.154436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:50:54.154436+0000","log_size":61,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":104,"num_read_kb":84,"num_write":62,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"1150'5","reported_seq":"84","reported_epoch":"1159","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:03.513991+0000","last_change":"2021-02-10T04:52:32.093814+0000","last_active":"2021-02-10T04:53:03.513991+0000","last_peered":"2021-02-10T04:53:03.513991+0000","last_clean":"2021-02-10T04:52:30.445006+0000","last_became_active":"2021-02-10T04:52:32.093814+0000","last_became_peered":"2021-02-10T04:52:32.093814+0000","last_unstale":"2021-02-10T04:53:03.513991+0000","last_undegraded":"2021-02-10T04:52:32.079897+0000","last_fullsized":"2021-02-10T04:52:32.079762+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'4","last_scrub_stamp":"2021-02-10T04:52:01.121380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.121380+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"56","reported_epoch":"1158","state":"active+undersized","last_fresh":"2021-02-10T04:52:59.513137+0000","last_change":"2021-02-10T04:52:32.093831+0000","last_active":"2021-02-10T04:52:59.513137+0000","last_peered":"2021-02-10T04:52:59.513137+0000","last_clean":"2021-02-10T04:52:30.057710+0000","last_became_active":"2021-02-10T04:52:32.093831+0000","last_became_peered":"2021-02-10T04:52:32.093831+0000","last_unstale":"2021-02-10T04:52:59.513137+0000","last_undegraded":"2021-02-10T04:52:59.513137+0000","last_fullsized":"2021-02-10T04:52:32.081168+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:52.421793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:52.421793+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"55","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.513685+0000","last_change":"2021-02-10T04:52:35.170221+0000","last_active":"2021-02-10T04:52:59.513685+0000","last_peered":"2021-02-10T04:52:59.513685+0000","last_clean":"2021-02-10T04:52:59.513685+0000","last_became_active":"2021-02-10T04:51:36.851957+0000","last_became_peered":"2021-02-10T04:51:36.851957+0000","last_unstale":"2021-02-10T04:52:59.513685+0000","last_undegraded":"2021-02-10T04:52:59.513685+0000","last_fullsized":"2021-02-10T04:52:59.513685+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:35.170157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:35.170157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"1003'739","reported_seq":"3664","reported_epoch":"1159","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:03.684114+0000","last_change":"2021-02-10T04:52:32.084576+0000","last_active":"2021-02-10T04:53:03.684114+0000","last_peered":"2021-02-10T04:53:03.684114+0000","last_clean":"2021-02-10T04:43:58.648858+0000","last_became_active":"2021-02-10T04:52:32.084576+0000","last_became_peered":"2021-02-10T04:52:32.084576+0000","last_unstale":"2021-02-10T04:53:03.684114+0000","last_undegraded":"2021-02-10T04:52:32.072214+0000","last_fullsized":"2021-02-10T04:52:32.072101+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:52:27.602006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:27.602006+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24059888,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":129,"num_read_kb":109544,"num_write":102,"num_write_kb":46173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"1049'61","reported_seq":"1291","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.527592+0000","last_change":"2021-02-10T04:51:37.439479+0000","last_active":"2021-02-10T04:52:59.527592+0000","last_peered":"2021-02-10T04:52:59.527592+0000","last_clean":"2021-02-10T04:52:59.527592+0000","last_became_active":"2021-02-10T04:51:36.849289+0000","last_became_peered":"2021-02-10T04:51:36.849289+0000","last_unstale":"2021-02-10T04:52:59.527592+0000","last_undegraded":"2021-02-10T04:52:59.527592+0000","last_fullsized":"2021-02-10T04:52:59.527592+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'61","last_scrub_stamp":"2021-02-10T04:50:47.662378+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:50:47.662378+0000","log_size":61,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":83,"num_read_kb":65,"num_write":50,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"1049","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.529847+0000","last_change":"2021-02-10T04:50:45.623700+0000","last_active":"2021-02-10T04:52:59.529847+0000","last_peered":"2021-02-10T04:52:59.529847+0000","last_clean":"2021-02-10T04:52:59.529847+0000","last_became_active":"2021-02-10T04:29:07.145616+0000","last_became_peered":"2021-02-10T04:29:07.145616+0000","last_unstale":"2021-02-10T04:52:59.529847+0000","last_undegraded":"2021-02-10T04:52:59.529847+0000","last_fullsized":"2021-02-10T04:52:59.529847+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:45.623640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:45.623640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"379","reported_epoch":"1159","state":"active+undersized","last_fresh":"2021-02-10T04:53:03.686232+0000","last_change":"2021-02-10T04:52:32.083894+0000","last_active":"2021-02-10T04:53:03.686232+0000","last_peered":"2021-02-10T04:53:03.686232+0000","last_clean":"2021-02-10T04:52:30.060966+0000","last_became_active":"2021-02-10T04:52:32.083894+0000","last_became_peered":"2021-02-10T04:52:32.083894+0000","last_unstale":"2021-02-10T04:53:03.686232+0000","last_undegraded":"2021-02-10T04:53:03.686232+0000","last_fullsized":"2021-02-10T04:52:32.079256+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:26.311958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.311958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"55","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.515308+0000","last_change":"2021-02-10T04:52:02.162917+0000","last_active":"2021-02-10T04:52:59.515308+0000","last_peered":"2021-02-10T04:52:59.515308+0000","last_clean":"2021-02-10T04:52:59.515308+0000","last_became_active":"2021-02-10T04:51:36.855744+0000","last_became_peered":"2021-02-10T04:51:36.855744+0000","last_unstale":"2021-02-10T04:52:59.515308+0000","last_undegraded":"2021-02-10T04:52:59.515308+0000","last_fullsized":"2021-02-10T04:52:59.515308+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:02.162851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:13.727241+0000","last_clean_scrub_stamp":"2021-02-10T04:52:02.162851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"1063'654","reported_seq":"4325","reported_epoch":"1158","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:52:59.515454+0000","last_change":"2021-02-10T04:52:32.093941+0000","last_active":"2021-02-10T04:52:59.515454+0000","last_peered":"2021-02-10T04:52:59.515454+0000","last_clean":"2021-02-10T04:52:30.062769+0000","last_became_active":"2021-02-10T04:52:32.093941+0000","last_became_peered":"2021-02-10T04:52:32.093941+0000","last_unstale":"2021-02-10T04:52:59.515454+0000","last_undegraded":"2021-02-10T04:52:32.085816+0000","last_fullsized":"2021-02-10T04:52:32.085711+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:51:57.121838+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.121838+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24700630,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":171,"num_read_kb":182272,"num_write":111,"num_write_kb":33727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":26349023,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":12}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"1105","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.509342+0000","last_change":"2021-02-10T04:51:50.433056+0000","last_active":"2021-02-10T04:53:03.509342+0000","last_peered":"2021-02-10T04:53:03.509342+0000","last_clean":"2021-02-10T04:53:03.509342+0000","last_became_active":"2021-02-10T04:51:36.843165+0000","last_became_peered":"2021-02-10T04:51:36.843165+0000","last_unstale":"2021-02-10T04:53:03.509342+0000","last_undegraded":"2021-02-10T04:53:03.509342+0000","last_fullsized":"2021-02-10T04:53:03.509342+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.432990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.432990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"1125","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.513694+0000","last_change":"2021-02-10T04:52:41.041630+0000","last_active":"2021-02-10T04:53:03.513694+0000","last_peered":"2021-02-10T04:53:03.513694+0000","last_clean":"2021-02-10T04:53:03.513694+0000","last_became_active":"2021-02-10T04:51:36.851075+0000","last_became_peered":"2021-02-10T04:51:36.851075+0000","last_unstale":"2021-02-10T04:53:03.513694+0000","last_undegraded":"2021-02-10T04:53:03.513694+0000","last_fullsized":"2021-02-10T04:53:03.513694+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:41.041553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.041553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"1049'33","reported_seq":"1197","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.526551+0000","last_change":"2021-02-10T04:51:06.846957+0000","last_active":"2021-02-10T04:52:59.526551+0000","last_peered":"2021-02-10T04:52:59.526551+0000","last_clean":"2021-02-10T04:52:59.526551+0000","last_became_active":"2021-02-10T04:50:59.584833+0000","last_became_peered":"2021-02-10T04:50:59.584833+0000","last_unstale":"2021-02-10T04:52:59.526551+0000","last_undegraded":"2021-02-10T04:52:59.526551+0000","last_fullsized":"2021-02-10T04:52:59.526551+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'33","last_scrub_stamp":"2021-02-10T04:51:06.846850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:06.846850+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":43,"num_write":33,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"798","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.684657+0000","last_change":"2021-02-10T04:51:52.350764+0000","last_active":"2021-02-10T04:53:03.684657+0000","last_peered":"2021-02-10T04:53:03.684657+0000","last_clean":"2021-02-10T04:53:03.684657+0000","last_became_active":"2021-02-10T04:51:49.567784+0000","last_became_peered":"2021-02-10T04:51:49.567784+0000","last_unstale":"2021-02-10T04:53:03.684657+0000","last_undegraded":"2021-02-10T04:53:03.684657+0000","last_fullsized":"2021-02-10T04:53:03.684657+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:52.350706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.350706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"1119'4","reported_seq":"78","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:53:03.009237+0000","last_change":"2021-02-10T04:52:27.126670+0000","last_active":"2021-02-10T04:53:03.009237+0000","last_peered":"2021-02-10T04:53:03.009237+0000","last_clean":"2021-02-10T04:53:03.009237+0000","last_became_active":"2021-02-10T04:51:36.853647+0000","last_became_peered":"2021-02-10T04:51:36.853647+0000","last_unstale":"2021-02-10T04:53:03.009237+0000","last_undegraded":"2021-02-10T04:53:03.009237+0000","last_fullsized":"2021-02-10T04:53:03.009237+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'4","last_scrub_stamp":"2021-02-10T04:52:27.126599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:27.126599+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"1145","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.528321+0000","last_change":"2021-02-10T04:52:25.745660+0000","last_active":"2021-02-10T04:52:59.528321+0000","last_peered":"2021-02-10T04:52:59.528321+0000","last_clean":"2021-02-10T04:52:59.528321+0000","last_became_active":"2021-02-10T04:51:36.849503+0000","last_became_peered":"2021-02-10T04:51:36.849503+0000","last_unstale":"2021-02-10T04:52:59.528321+0000","last_undegraded":"2021-02-10T04:52:59.528321+0000","last_fullsized":"2021-02-10T04:52:59.528321+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:25.745577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.745577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"1004'1093","reported_seq":"6200","reported_epoch":"1159","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:03.515792+0000","last_change":"2021-02-10T04:52:32.090286+0000","last_active":"2021-02-10T04:53:03.515792+0000","last_peered":"2021-02-10T04:53:03.515792+0000","last_clean":"2021-02-10T04:44:07.998404+0000","last_became_active":"2021-02-10T04:52:32.090286+0000","last_became_peered":"2021-02-10T04:52:32.090286+0000","last_unstale":"2021-02-10T04:53:03.515792+0000","last_undegraded":"2021-02-10T04:52:32.075231+0000","last_fullsized":"2021-02-10T04:52:32.075115+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39434696,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":279,"num_read_kb":311058,"num_write":185,"num_write_kb":66790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":17}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"1090","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.510619+0000","last_change":"2021-02-10T04:51:36.841978+0000","last_active":"2021-02-10T04:53:03.510619+0000","last_peered":"2021-02-10T04:53:03.510619+0000","last_clean":"2021-02-10T04:53:03.510619+0000","last_became_active":"2021-02-10T04:51:36.841582+0000","last_became_peered":"2021-02-10T04:51:36.841582+0000","last_unstale":"2021-02-10T04:53:03.510619+0000","last_undegraded":"2021-02-10T04:53:03.510619+0000","last_fullsized":"2021-02-10T04:53:03.510619+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:12.389272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:12.389272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"1049'41","reported_seq":"816","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.684992+0000","last_change":"2021-02-10T04:51:58.353746+0000","last_active":"2021-02-10T04:53:03.684992+0000","last_peered":"2021-02-10T04:53:03.684992+0000","last_clean":"2021-02-10T04:53:03.684992+0000","last_became_active":"2021-02-10T04:51:49.502935+0000","last_became_peered":"2021-02-10T04:51:49.502935+0000","last_unstale":"2021-02-10T04:53:03.684992+0000","last_undegraded":"2021-02-10T04:53:03.684992+0000","last_fullsized":"2021-02-10T04:53:03.684992+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'41","last_scrub_stamp":"2021-02-10T04:51:58.353660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:58.353660+0000","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":66,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"816","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.684038+0000","last_change":"2021-02-10T04:51:49.568270+0000","last_active":"2021-02-10T04:53:03.684038+0000","last_peered":"2021-02-10T04:53:03.684038+0000","last_clean":"2021-02-10T04:53:03.684038+0000","last_became_active":"2021-02-10T04:51:49.567883+0000","last_became_peered":"2021-02-10T04:51:49.567883+0000","last_unstale":"2021-02-10T04:53:03.684038+0000","last_undegraded":"2021-02-10T04:53:03.684038+0000","last_fullsized":"2021-02-10T04:53:03.684038+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:47.605360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_clean_scrub_stamp":"2021-02-10T04:46:47.605360+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"1148'4","reported_seq":"1465","reported_epoch":"1158","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:03.009536+0000","last_change":"2021-02-10T04:52:32.088315+0000","last_active":"2021-02-10T04:53:03.009536+0000","last_peered":"2021-02-10T04:53:03.009536+0000","last_clean":"2021-02-10T04:52:30.062243+0000","last_became_active":"2021-02-10T04:52:32.088315+0000","last_became_peered":"2021-02-10T04:52:32.088315+0000","last_unstale":"2021-02-10T04:53:03.009536+0000","last_undegraded":"2021-02-10T04:52:32.069861+0000","last_fullsized":"2021-02-10T04:52:32.069742+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:51:47.855283+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.855283+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"1135","reported_epoch":"1159","state":"active+undersized","last_fresh":"2021-02-10T04:53:03.514893+0000","last_change":"2021-02-10T04:52:32.094688+0000","last_active":"2021-02-10T04:53:03.514893+0000","last_peered":"2021-02-10T04:53:03.514893+0000","last_clean":"2021-02-10T04:52:30.059376+0000","last_became_active":"2021-02-10T04:52:32.094688+0000","last_became_peered":"2021-02-10T04:52:32.094688+0000","last_unstale":"2021-02-10T04:53:03.514893+0000","last_undegraded":"2021-02-10T04:53:03.514893+0000","last_fullsized":"2021-02-10T04:52:32.079768+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:10.176878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:50:10.176878+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"1002'1090","reported_seq":"5354","reported_epoch":"1160","state":"active+clean","last_fresh":"2021-02-10T04:53:07.222343+0000","last_change":"2021-02-10T04:52:26.030198+0000","last_active":"2021-02-10T04:53:07.222343+0000","last_peered":"2021-02-10T04:53:07.222343+0000","last_clean":"2021-02-10T04:53:07.222343+0000","last_became_active":"2021-02-10T04:52:26.028182+0000","last_became_peered":"2021-02-10T04:52:26.028182+0000","last_unstale":"2021-02-10T04:53:07.222343+0000","last_undegraded":"2021-02-10T04:53:07.222343+0000","last_fullsized":"2021-02-10T04:53:07.222343+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:51:08.220472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.220472+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24410646,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":245,"num_read_kb":276168,"num_write":175,"num_write_kb":64960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"1049'47","reported_seq":"1207","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.527916+0000","last_change":"2021-02-10T04:50:04.767513+0000","last_active":"2021-02-10T04:52:59.527916+0000","last_peered":"2021-02-10T04:52:59.527916+0000","last_clean":"2021-02-10T04:52:59.527916+0000","last_became_active":"2021-02-10T04:25:49.011093+0000","last_became_peered":"2021-02-10T04:25:49.011093+0000","last_unstale":"2021-02-10T04:52:59.527916+0000","last_undegraded":"2021-02-10T04:52:59.527916+0000","last_fullsized":"2021-02-10T04:52:59.527916+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'47","last_scrub_stamp":"2021-02-10T04:50:04.767446+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:50:04.767446+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":68,"num_read_kb":52,"num_write":38,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"82","reported_epoch":"1159","state":"active+undersized","last_fresh":"2021-02-10T04:53:03.684246+0000","last_change":"2021-02-10T04:52:32.092388+0000","last_active":"2021-02-10T04:53:03.684246+0000","last_peered":"2021-02-10T04:53:03.684246+0000","last_clean":"2021-02-10T04:28:18.347728+0000","last_became_active":"2021-02-10T04:52:32.092388+0000","last_became_peered":"2021-02-10T04:52:32.092388+0000","last_unstale":"2021-02-10T04:53:03.684246+0000","last_undegraded":"2021-02-10T04:53:03.684246+0000","last_fullsized":"2021-02-10T04:52:32.078707+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:08.358300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:50:08.358300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"1136","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.514378+0000","last_change":"2021-02-10T04:51:57.436961+0000","last_active":"2021-02-10T04:53:03.514378+0000","last_peered":"2021-02-10T04:53:03.514378+0000","last_clean":"2021-02-10T04:53:03.514378+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:53:03.514378+0000","last_undegraded":"2021-02-10T04:53:03.514378+0000","last_fullsized":"2021-02-10T04:53:03.514378+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.436862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.436862+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"1091","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.514705+0000","last_change":"2021-02-10T04:53:02.112612+0000","last_active":"2021-02-10T04:53:03.514705+0000","last_peered":"2021-02-10T04:53:03.514705+0000","last_clean":"2021-02-10T04:53:03.514705+0000","last_became_active":"2021-02-10T04:50:59.545501+0000","last_became_peered":"2021-02-10T04:50:59.545501+0000","last_unstale":"2021-02-10T04:53:03.514705+0000","last_undegraded":"2021-02-10T04:53:03.514705+0000","last_fullsized":"2021-02-10T04:53:03.514705+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:02.112533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:02.112533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"1001'1165","reported_seq":"4324","reported_epoch":"1160","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:07.222898+0000","last_change":"2021-02-10T04:52:32.082334+0000","last_active":"2021-02-10T04:53:07.222898+0000","last_peered":"2021-02-10T04:53:07.222898+0000","last_clean":"2021-02-10T04:52:30.068280+0000","last_became_active":"2021-02-10T04:52:32.082334+0000","last_became_peered":"2021-02-10T04:52:32.082334+0000","last_unstale":"2021-02-10T04:53:07.222898+0000","last_undegraded":"2021-02-10T04:52:32.079317+0000","last_fullsized":"2021-02-10T04:52:32.078663+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29601845,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":286,"num_read_kb":299529,"num_write":200,"num_write_kb":89130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":12}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"1113'5","reported_seq":"1361","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.511891+0000","last_change":"2021-02-10T04:52:34.047474+0000","last_active":"2021-02-10T04:53:03.511891+0000","last_peered":"2021-02-10T04:53:03.511891+0000","last_clean":"2021-02-10T04:53:03.511891+0000","last_became_active":"2021-02-10T04:50:59.569578+0000","last_became_peered":"2021-02-10T04:50:59.569578+0000","last_unstale":"2021-02-10T04:53:03.511891+0000","last_undegraded":"2021-02-10T04:53:03.511891+0000","last_fullsized":"2021-02-10T04:53:03.511891+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1113'5","last_scrub_stamp":"2021-02-10T04:52:34.047358+0000","last_deep_scrub":"158'3","last_deep_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_clean_scrub_stamp":"2021-02-10T04:52:34.047358+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"1049'54","reported_seq":"1221","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.525244+0000","last_change":"2021-02-10T04:51:48.872937+0000","last_active":"2021-02-10T04:52:59.525244+0000","last_peered":"2021-02-10T04:52:59.525244+0000","last_clean":"2021-02-10T04:52:59.525244+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:52:59.525244+0000","last_undegraded":"2021-02-10T04:52:59.525244+0000","last_fullsized":"2021-02-10T04:52:59.525244+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:51:48.872839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.872839+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":93,"num_read_kb":73,"num_write":54,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"1211","reported_epoch":"1160","state":"active+clean","last_fresh":"2021-02-10T04:53:07.223562+0000","last_change":"2021-02-10T04:53:05.984747+0000","last_active":"2021-02-10T04:53:07.223562+0000","last_peered":"2021-02-10T04:53:07.223562+0000","last_clean":"2021-02-10T04:53:07.223562+0000","last_became_active":"2021-02-10T04:51:36.869931+0000","last_became_peered":"2021-02-10T04:51:36.869931+0000","last_unstale":"2021-02-10T04:53:07.223562+0000","last_undegraded":"2021-02-10T04:53:07.223562+0000","last_fullsized":"2021-02-10T04:53:07.223562+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:05.984659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:05.984659+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"1089","reported_epoch":"1161","state":"active+clean","last_fresh":"2021-02-10T04:53:09.005480+0000","last_change":"2021-02-10T04:53:09.005480+0000","last_active":"2021-02-10T04:53:09.005480+0000","last_peered":"2021-02-10T04:53:09.005480+0000","last_clean":"2021-02-10T04:53:09.005480+0000","last_became_active":"2021-02-10T04:51:36.869649+0000","last_became_peered":"2021-02-10T04:51:36.869649+0000","last_unstale":"2021-02-10T04:53:09.005480+0000","last_undegraded":"2021-02-10T04:53:09.005480+0000","last_fullsized":"2021-02-10T04:53:09.005480+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:09.005421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.005421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"998'1009","reported_seq":"4650","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.514204+0000","last_change":"2021-02-10T04:52:26.037227+0000","last_active":"2021-02-10T04:53:03.514204+0000","last_peered":"2021-02-10T04:53:03.514204+0000","last_clean":"2021-02-10T04:53:03.514204+0000","last_became_active":"2021-02-10T04:52:26.036723+0000","last_became_peered":"2021-02-10T04:52:26.036723+0000","last_unstale":"2021-02-10T04:53:03.514204+0000","last_undegraded":"2021-02-10T04:53:03.514204+0000","last_fullsized":"2021-02-10T04:53:03.514204+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12355206,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":119,"num_read_kb":124701,"num_write":77,"num_write_kb":25052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4145029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"1025","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.515208+0000","last_change":"2021-02-10T04:51:07.976066+0000","last_active":"2021-02-10T04:53:03.515208+0000","last_peered":"2021-02-10T04:53:03.515208+0000","last_clean":"2021-02-10T04:53:03.515208+0000","last_became_active":"2021-02-10T04:50:59.570854+0000","last_became_peered":"2021-02-10T04:50:59.570854+0000","last_unstale":"2021-02-10T04:53:03.515208+0000","last_undegraded":"2021-02-10T04:53:03.515208+0000","last_fullsized":"2021-02-10T04:53:03.515208+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:46.592377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:46:46.592377+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"1144","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.525605+0000","last_change":"2021-02-10T04:51:50.895471+0000","last_active":"2021-02-10T04:52:59.525605+0000","last_peered":"2021-02-10T04:52:59.525605+0000","last_clean":"2021-02-10T04:52:59.525605+0000","last_became_active":"2021-02-10T04:50:59.527553+0000","last_became_peered":"2021-02-10T04:50:59.527553+0000","last_unstale":"2021-02-10T04:52:59.525605+0000","last_undegraded":"2021-02-10T04:52:59.525605+0000","last_fullsized":"2021-02-10T04:52:59.525605+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.895370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.895370+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"1049'72","reported_seq":"1248","reported_epoch":"1159","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:03.512418+0000","last_change":"2021-02-10T04:52:32.090145+0000","last_active":"2021-02-10T04:53:03.512418+0000","last_peered":"2021-02-10T04:53:03.512418+0000","last_clean":"2021-02-10T04:52:30.063735+0000","last_became_active":"2021-02-10T04:52:32.090145+0000","last_became_peered":"2021-02-10T04:52:32.090145+0000","last_unstale":"2021-02-10T04:53:03.512418+0000","last_undegraded":"2021-02-10T04:52:32.073169+0000","last_fullsized":"2021-02-10T04:52:32.073062+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'72","last_scrub_stamp":"2021-02-10T04:46:30.515026+0000","last_deep_scrub":"152'48","last_deep_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_clean_scrub_stamp":"2021-02-10T04:46:30.515026+0000","log_size":72,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":105,"num_read_kb":83,"num_write":60,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":14}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"391","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.686205+0000","last_change":"2021-02-10T04:51:49.504222+0000","last_active":"2021-02-10T04:53:03.686205+0000","last_peered":"2021-02-10T04:53:03.686205+0000","last_clean":"2021-02-10T04:53:03.686205+0000","last_became_active":"2021-02-10T04:51:49.503801+0000","last_became_peered":"2021-02-10T04:51:49.503801+0000","last_unstale":"2021-02-10T04:53:03.686205+0000","last_undegraded":"2021-02-10T04:53:03.686205+0000","last_fullsized":"2021-02-10T04:53:03.686205+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:51:48.274960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.274960+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"1164","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.526393+0000","last_change":"2021-02-10T04:51:36.849004+0000","last_active":"2021-02-10T04:52:59.526393+0000","last_peered":"2021-02-10T04:52:59.526393+0000","last_clean":"2021-02-10T04:52:59.526393+0000","last_became_active":"2021-02-10T04:51:36.848614+0000","last_became_peered":"2021-02-10T04:51:36.848614+0000","last_unstale":"2021-02-10T04:52:59.526393+0000","last_undegraded":"2021-02-10T04:52:59.526393+0000","last_fullsized":"2021-02-10T04:52:59.526393+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:27.926662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:44:27.926662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"1002'1657","reported_seq":"5220","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.514511+0000","last_change":"2021-02-10T04:52:26.036806+0000","last_active":"2021-02-10T04:53:03.514511+0000","last_peered":"2021-02-10T04:53:03.514511+0000","last_clean":"2021-02-10T04:53:03.514511+0000","last_became_active":"2021-02-10T04:52:26.032363+0000","last_became_peered":"2021-02-10T04:52:26.032363+0000","last_unstale":"2021-02-10T04:53:03.514511+0000","last_undegraded":"2021-02-10T04:53:03.514511+0000","last_fullsized":"2021-02-10T04:53:03.514511+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21396533,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":212,"num_read_kb":218836,"num_write":159,"num_write_kb":59398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":22326014,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"1088","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.526820+0000","last_change":"2021-02-10T04:51:51.911945+0000","last_active":"2021-02-10T04:52:59.526820+0000","last_peered":"2021-02-10T04:52:59.526820+0000","last_clean":"2021-02-10T04:52:59.526820+0000","last_became_active":"2021-02-10T04:51:36.850015+0000","last_became_peered":"2021-02-10T04:51:36.850015+0000","last_unstale":"2021-02-10T04:52:59.526820+0000","last_undegraded":"2021-02-10T04:52:59.526820+0000","last_fullsized":"2021-02-10T04:52:59.526820+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.911851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.911851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"1209","reported_epoch":"1160","state":"active+undersized","last_fresh":"2021-02-10T04:53:07.222375+0000","last_change":"2021-02-10T04:52:32.089163+0000","last_active":"2021-02-10T04:53:07.222375+0000","last_peered":"2021-02-10T04:53:07.222375+0000","last_clean":"2021-02-10T04:52:30.062078+0000","last_became_active":"2021-02-10T04:52:32.089163+0000","last_became_peered":"2021-02-10T04:52:32.089163+0000","last_unstale":"2021-02-10T04:53:07.222375+0000","last_undegraded":"2021-02-10T04:53:07.222375+0000","last_fullsized":"2021-02-10T04:52:32.072413+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:58.142526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:58.142526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"1135'58","reported_seq":"1214","reported_epoch":"1159","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:03.511859+0000","last_change":"2021-02-10T04:52:32.074386+0000","last_active":"2021-02-10T04:53:03.511859+0000","last_peered":"2021-02-10T04:53:03.511859+0000","last_clean":"2021-02-10T04:52:01.474364+0000","last_became_active":"2021-02-10T04:52:32.074386+0000","last_became_peered":"2021-02-10T04:52:32.074386+0000","last_unstale":"2021-02-10T04:53:03.511859+0000","last_undegraded":"2021-02-10T04:52:32.071237+0000","last_fullsized":"2021-02-10T04:52:32.070967+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1135'58","last_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_deep_scrub":"1135'58","last_deep_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.483067+0000","log_size":58,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":87,"num_read_kb":68,"num_write":50,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":11}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"31","reported_epoch":"1158","state":"active+undersized","last_fresh":"2021-02-10T04:52:59.527235+0000","last_change":"2021-02-10T04:52:32.076740+0000","last_active":"2021-02-10T04:52:59.527235+0000","last_peered":"2021-02-10T04:52:59.527235+0000","last_clean":"2021-02-10T04:25:40.828974+0000","last_became_active":"2021-02-10T04:52:32.076740+0000","last_became_peered":"2021-02-10T04:52:32.076740+0000","last_unstale":"2021-02-10T04:52:59.527235+0000","last_undegraded":"2021-02-10T04:52:59.527235+0000","last_fullsized":"2021-02-10T04:52:32.070328+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:50:46.434629+0000","last_deep_scrub":"53'2","last_deep_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_clean_scrub_stamp":"2021-02-10T04:50:46.434629+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"838","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.684108+0000","last_change":"2021-02-10T04:52:38.278980+0000","last_active":"2021-02-10T04:53:03.684108+0000","last_peered":"2021-02-10T04:53:03.684108+0000","last_clean":"2021-02-10T04:53:03.684108+0000","last_became_active":"2021-02-10T04:51:49.496237+0000","last_became_peered":"2021-02-10T04:51:49.496237+0000","last_unstale":"2021-02-10T04:53:03.684108+0000","last_undegraded":"2021-02-10T04:53:03.684108+0000","last_fullsized":"2021-02-10T04:53:03.684108+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:38.278910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:52:38.278910+0000","last_clean_scrub_stamp":"2021-02-10T04:52:38.278910+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"1000'1071","reported_seq":"4705","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.509884+0000","last_change":"2021-02-10T04:52:26.032390+0000","last_active":"2021-02-10T04:53:03.509884+0000","last_peered":"2021-02-10T04:53:03.509884+0000","last_clean":"2021-02-10T04:53:03.509884+0000","last_became_active":"2021-02-10T04:52:26.031050+0000","last_became_peered":"2021-02-10T04:52:26.031050+0000","last_unstale":"2021-02-10T04:53:03.509884+0000","last_undegraded":"2021-02-10T04:53:03.509884+0000","last_fullsized":"2021-02-10T04:53:03.509884+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:47:14.611347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:14.611347+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23286368,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"1175","reported_epoch":"1160","state":"active+undersized","last_fresh":"2021-02-10T04:53:07.223138+0000","last_change":"2021-02-10T04:52:32.081955+0000","last_active":"2021-02-10T04:53:07.223138+0000","last_peered":"2021-02-10T04:53:07.223138+0000","last_clean":"2021-02-10T04:52:30.068208+0000","last_became_active":"2021-02-10T04:52:32.081955+0000","last_became_peered":"2021-02-10T04:52:32.081955+0000","last_unstale":"2021-02-10T04:53:07.223138+0000","last_undegraded":"2021-02-10T04:53:07.223138+0000","last_fullsized":"2021-02-10T04:52:32.077723+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_clean_scrub_stamp":"2021-02-10T04:51:00.195518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"1137","reported_epoch":"1159","state":"active+undersized","last_fresh":"2021-02-10T04:53:03.514517+0000","last_change":"2021-02-10T04:52:32.097628+0000","last_active":"2021-02-10T04:53:03.514517+0000","last_peered":"2021-02-10T04:53:03.514517+0000","last_clean":"2021-02-10T04:52:30.061275+0000","last_became_active":"2021-02-10T04:52:32.097628+0000","last_became_peered":"2021-02-10T04:52:32.097628+0000","last_unstale":"2021-02-10T04:53:03.514517+0000","last_undegraded":"2021-02-10T04:53:03.514517+0000","last_fullsized":"2021-02-10T04:52:32.086329+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:55.473347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:50:55.473347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"1049'40","reported_seq":"1205","reported_epoch":"1158","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:52:59.515307+0000","last_change":"2021-02-10T04:52:32.094738+0000","last_active":"2021-02-10T04:52:59.515307+0000","last_peered":"2021-02-10T04:52:59.515307+0000","last_clean":"2021-02-10T04:51:27.993802+0000","last_became_active":"2021-02-10T04:52:32.094738+0000","last_became_peered":"2021-02-10T04:52:32.094738+0000","last_unstale":"2021-02-10T04:52:59.515307+0000","last_undegraded":"2021-02-10T04:52:32.085496+0000","last_fullsized":"2021-02-10T04:52:32.085393+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'40","last_scrub_stamp":"2021-02-10T04:51:47.281929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.281929+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":52,"num_write":40,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":10}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"352","reported_epoch":"1159","state":"active+undersized","last_fresh":"2021-02-10T04:53:03.684018+0000","last_change":"2021-02-10T04:52:32.091929+0000","last_active":"2021-02-10T04:53:03.684018+0000","last_peered":"2021-02-10T04:53:03.684018+0000","last_clean":"2021-02-10T04:52:30.068091+0000","last_became_active":"2021-02-10T04:52:32.091929+0000","last_became_peered":"2021-02-10T04:52:32.091929+0000","last_unstale":"2021-02-10T04:53:03.684018+0000","last_undegraded":"2021-02-10T04:53:03.684018+0000","last_fullsized":"2021-02-10T04:52:32.078333+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:50.399350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:50.399350+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"1111","reported_epoch":"1158","state":"active+undersized","last_fresh":"2021-02-10T04:52:59.525480+0000","last_change":"2021-02-10T04:52:32.081196+0000","last_active":"2021-02-10T04:52:59.525480+0000","last_peered":"2021-02-10T04:52:59.525480+0000","last_clean":"2021-02-10T04:49:56.935460+0000","last_became_active":"2021-02-10T04:52:32.081196+0000","last_became_peered":"2021-02-10T04:52:32.081196+0000","last_unstale":"2021-02-10T04:52:59.525480+0000","last_undegraded":"2021-02-10T04:52:59.525480+0000","last_fullsized":"2021-02-10T04:52:32.068851+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"823","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.686350+0000","last_change":"2021-02-10T04:52:31.273037+0000","last_active":"2021-02-10T04:53:03.686350+0000","last_peered":"2021-02-10T04:53:03.686350+0000","last_clean":"2021-02-10T04:53:03.686350+0000","last_became_active":"2021-02-10T04:51:49.501405+0000","last_became_peered":"2021-02-10T04:51:49.501405+0000","last_unstale":"2021-02-10T04:53:03.686350+0000","last_undegraded":"2021-02-10T04:53:03.686350+0000","last_fullsized":"2021-02-10T04:53:03.686350+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:31.272971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:17.598113+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.272971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"1000'446","reported_seq":"2100","reported_epoch":"1158","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:52:59.514877+0000","last_change":"2021-02-10T04:52:32.087644+0000","last_active":"2021-02-10T04:52:59.514877+0000","last_peered":"2021-02-10T04:52:59.514877+0000","last_clean":"2021-02-10T04:52:30.064188+0000","last_became_active":"2021-02-10T04:52:32.087644+0000","last_became_peered":"2021-02-10T04:52:32.087644+0000","last_unstale":"2021-02-10T04:52:59.514877+0000","last_undegraded":"2021-02-10T04:52:32.081292+0000","last_fullsized":"2021-02-10T04:52:32.081161+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:51:45.964029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.964029+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19615419,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14193373,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":9}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"314","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.685289+0000","last_change":"2021-02-10T04:53:03.228260+0000","last_active":"2021-02-10T04:53:03.685289+0000","last_peered":"2021-02-10T04:53:03.685289+0000","last_clean":"2021-02-10T04:53:03.685289+0000","last_became_active":"2021-02-10T04:51:49.499252+0000","last_became_peered":"2021-02-10T04:51:49.499252+0000","last_unstale":"2021-02-10T04:53:03.685289+0000","last_undegraded":"2021-02-10T04:53:03.685289+0000","last_fullsized":"2021-02-10T04:53:03.685289+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:03.228182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.228182+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"50","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.514131+0000","last_change":"2021-02-10T04:51:36.859470+0000","last_active":"2021-02-10T04:52:59.514131+0000","last_peered":"2021-02-10T04:52:59.514131+0000","last_clean":"2021-02-10T04:52:59.514131+0000","last_became_active":"2021-02-10T04:51:36.858966+0000","last_became_peered":"2021-02-10T04:51:36.858966+0000","last_unstale":"2021-02-10T04:52:59.514131+0000","last_undegraded":"2021-02-10T04:52:59.514131+0000","last_fullsized":"2021-02-10T04:52:59.514131+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:10.398477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:10.398477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"1049'39","reported_seq":"1199","reported_epoch":"1160","state":"active+clean","last_fresh":"2021-02-10T04:53:07.222618+0000","last_change":"2021-02-10T04:51:47.322426+0000","last_active":"2021-02-10T04:53:07.222618+0000","last_peered":"2021-02-10T04:53:07.222618+0000","last_clean":"2021-02-10T04:53:07.222618+0000","last_became_active":"2021-02-10T04:25:49.002471+0000","last_became_peered":"2021-02-10T04:25:49.002471+0000","last_unstale":"2021-02-10T04:53:07.222618+0000","last_undegraded":"2021-02-10T04:53:07.222618+0000","last_fullsized":"2021-02-10T04:53:07.222618+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'39","last_scrub_stamp":"2021-02-10T04:51:47.322331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.322331+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":51,"num_write":39,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"105","reported_epoch":"1160","state":"active+undersized","last_fresh":"2021-02-10T04:53:07.221671+0000","last_change":"2021-02-10T04:52:32.083919+0000","last_active":"2021-02-10T04:53:07.221671+0000","last_peered":"2021-02-10T04:53:07.221671+0000","last_clean":"2021-02-10T04:28:45.095539+0000","last_became_active":"2021-02-10T04:52:32.083919+0000","last_became_peered":"2021-02-10T04:52:32.083919+0000","last_unstale":"2021-02-10T04:53:07.221671+0000","last_undegraded":"2021-02-10T04:53:07.221671+0000","last_fullsized":"2021-02-10T04:52:32.067706+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_clean_scrub_stamp":"2021-02-10T04:50:05.424262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"1330","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.511757+0000","last_change":"2021-02-10T04:52:37.324122+0000","last_active":"2021-02-10T04:53:03.511757+0000","last_peered":"2021-02-10T04:53:03.511757+0000","last_clean":"2021-02-10T04:53:03.511757+0000","last_became_active":"2021-02-10T04:51:36.843553+0000","last_became_peered":"2021-02-10T04:51:36.843553+0000","last_unstale":"2021-02-10T04:53:03.511757+0000","last_undegraded":"2021-02-10T04:53:03.511757+0000","last_fullsized":"2021-02-10T04:53:03.511757+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:37.324040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:37.324040+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"1127","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.514280+0000","last_change":"2021-02-10T04:51:57.769992+0000","last_active":"2021-02-10T04:53:03.514280+0000","last_peered":"2021-02-10T04:53:03.514280+0000","last_clean":"2021-02-10T04:53:03.514280+0000","last_became_active":"2021-02-10T04:50:59.587343+0000","last_became_peered":"2021-02-10T04:50:59.587343+0000","last_unstale":"2021-02-10T04:53:03.514280+0000","last_undegraded":"2021-02-10T04:53:03.514280+0000","last_fullsized":"2021-02-10T04:53:03.514280+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"1158","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.514644+0000","last_change":"2021-02-10T04:52:31.403904+0000","last_active":"2021-02-10T04:53:03.514644+0000","last_peered":"2021-02-10T04:53:03.514644+0000","last_clean":"2021-02-10T04:53:03.514644+0000","last_became_active":"2021-02-10T04:50:59.538009+0000","last_became_peered":"2021-02-10T04:50:59.538009+0000","last_unstale":"2021-02-10T04:53:03.514644+0000","last_undegraded":"2021-02-10T04:53:03.514644+0000","last_fullsized":"2021-02-10T04:53:03.514644+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:31.403824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.403824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"1000'490","reported_seq":"3547","reported_epoch":"1159","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:03.686075+0000","last_change":"2021-02-10T04:52:32.091003+0000","last_active":"2021-02-10T04:53:03.686075+0000","last_peered":"2021-02-10T04:53:03.686075+0000","last_clean":"2021-02-10T04:45:36.401519+0000","last_became_active":"2021-02-10T04:52:32.091003+0000","last_became_peered":"2021-02-10T04:52:32.091003+0000","last_unstale":"2021-02-10T04:53:03.686075+0000","last_undegraded":"2021-02-10T04:52:32.076920+0000","last_fullsized":"2021-02-10T04:52:32.076862+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1040,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:47:02.875920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:02.875920+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19371627,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":141,"num_read_kb":148568,"num_write":85,"num_write_kb":31732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"55","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:53:01.973063+0000","last_change":"2021-02-10T04:53:01.972991+0000","last_active":"2021-02-10T04:53:01.973063+0000","last_peered":"2021-02-10T04:53:01.973063+0000","last_clean":"2021-02-10T04:53:01.973063+0000","last_became_active":"2021-02-10T04:51:36.863976+0000","last_became_peered":"2021-02-10T04:51:36.863976+0000","last_unstale":"2021-02-10T04:53:01.973063+0000","last_undegraded":"2021-02-10T04:53:01.973063+0000","last_fullsized":"2021-02-10T04:53:01.973063+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:01.972913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.972913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"1049'37","reported_seq":"838","reported_epoch":"1159","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:03.684846+0000","last_change":"2021-02-10T04:52:32.095888+0000","last_active":"2021-02-10T04:53:03.684846+0000","last_peered":"2021-02-10T04:53:03.684846+0000","last_clean":"2021-02-10T04:52:30.062113+0000","last_became_active":"2021-02-10T04:52:32.095888+0000","last_became_peered":"2021-02-10T04:52:32.095888+0000","last_unstale":"2021-02-10T04:53:03.684846+0000","last_undegraded":"2021-02-10T04:52:32.076227+0000","last_fullsized":"2021-02-10T04:52:32.076110+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:56.431187+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.431187+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":59,"num_read_kb":43,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"56","reported_epoch":"1158","state":"active+undersized","last_fresh":"2021-02-10T04:52:59.513033+0000","last_change":"2021-02-10T04:52:32.089886+0000","last_active":"2021-02-10T04:52:59.513033+0000","last_peered":"2021-02-10T04:52:59.513033+0000","last_clean":"2021-02-10T04:52:30.064491+0000","last_became_active":"2021-02-10T04:52:32.089886+0000","last_became_peered":"2021-02-10T04:52:32.089886+0000","last_unstale":"2021-02-10T04:52:59.513033+0000","last_undegraded":"2021-02-10T04:52:59.513033+0000","last_fullsized":"2021-02-10T04:52:32.078850+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:04.591695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:04.591695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"889","reported_epoch":"1160","state":"active+clean","last_fresh":"2021-02-10T04:53:07.335266+0000","last_change":"2021-02-10T04:52:25.358146+0000","last_active":"2021-02-10T04:53:07.335266+0000","last_peered":"2021-02-10T04:53:07.335266+0000","last_clean":"2021-02-10T04:53:07.335266+0000","last_became_active":"2021-02-10T04:51:49.496092+0000","last_became_peered":"2021-02-10T04:51:49.496092+0000","last_unstale":"2021-02-10T04:53:07.335266+0000","last_undegraded":"2021-02-10T04:53:07.335266+0000","last_fullsized":"2021-02-10T04:53:07.335266+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":510,"num_read_kb":505,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"1086","reported_epoch":"1160","state":"active+clean","last_fresh":"2021-02-10T04:53:07.222806+0000","last_change":"2021-02-10T04:52:40.066424+0000","last_active":"2021-02-10T04:53:07.222806+0000","last_peered":"2021-02-10T04:53:07.222806+0000","last_clean":"2021-02-10T04:53:07.222806+0000","last_became_active":"2021-02-10T04:50:59.529649+0000","last_became_peered":"2021-02-10T04:50:59.529649+0000","last_unstale":"2021-02-10T04:53:07.222806+0000","last_undegraded":"2021-02-10T04:53:07.222806+0000","last_fullsized":"2021-02-10T04:53:07.222806+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:40.066354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:52:40.066354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.4","version":"1000'620","reported_seq":"4165","reported_epoch":"1158","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:52:59.526350+0000","last_change":"2021-02-10T04:52:32.082414+0000","last_active":"2021-02-10T04:52:59.526350+0000","last_peered":"2021-02-10T04:52:59.526350+0000","last_clean":"2021-02-10T04:43:51.960482+0000","last_became_active":"2021-02-10T04:52:32.082414+0000","last_became_peered":"2021-02-10T04:52:32.082414+0000","last_unstale":"2021-02-10T04:52:59.526350+0000","last_undegraded":"2021-02-10T04:52:32.068324+0000","last_fullsized":"2021-02-10T04:52:32.068200+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:51:54.444287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:54.444287+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14265195,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":144,"num_read_kb":128482,"num_write":129,"num_write_kb":53443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"113","reported_epoch":"1158","state":"active+undersized","last_fresh":"2021-02-10T04:52:59.525746+0000","last_change":"2021-02-10T04:52:32.076404+0000","last_active":"2021-02-10T04:52:59.525746+0000","last_peered":"2021-02-10T04:52:59.525746+0000","last_clean":"2021-02-10T04:28:47.053193+0000","last_became_active":"2021-02-10T04:52:32.076404+0000","last_became_peered":"2021-02-10T04:52:32.076404+0000","last_unstale":"2021-02-10T04:52:59.525746+0000","last_undegraded":"2021-02-10T04:52:59.525746+0000","last_fullsized":"2021-02-10T04:52:32.068279+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:14.955544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:47:14.955544+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"21","reported_epoch":"1159","state":"active+undersized","last_fresh":"2021-02-10T04:53:03.686575+0000","last_change":"2021-02-10T04:52:32.084350+0000","last_active":"2021-02-10T04:53:03.686575+0000","last_peered":"2021-02-10T04:53:03.686575+0000","last_clean":"2021-02-10T04:25:49.999907+0000","last_became_active":"2021-02-10T04:52:32.084350+0000","last_became_peered":"2021-02-10T04:52:32.084350+0000","last_unstale":"2021-02-10T04:53:03.686575+0000","last_undegraded":"2021-02-10T04:53:03.686575+0000","last_fullsized":"2021-02-10T04:52:32.073004+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:27.908221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:47:27.908221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"1049'55","reported_seq":"794","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.685945+0000","last_change":"2021-02-10T04:52:30.277009+0000","last_active":"2021-02-10T04:53:03.685945+0000","last_peered":"2021-02-10T04:53:03.685945+0000","last_clean":"2021-02-10T04:53:03.685945+0000","last_became_active":"2021-02-10T04:51:49.501622+0000","last_became_peered":"2021-02-10T04:51:49.501622+0000","last_unstale":"2021-02-10T04:53:03.685945+0000","last_undegraded":"2021-02-10T04:53:03.685945+0000","last_fullsized":"2021-02-10T04:53:03.685945+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'55","last_scrub_stamp":"2021-02-10T04:52:30.276938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.276938+0000","log_size":55,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":98,"num_read_kb":74,"num_write":58,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"1154","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.511069+0000","last_change":"2021-02-10T04:51:59.420149+0000","last_active":"2021-02-10T04:53:03.511069+0000","last_peered":"2021-02-10T04:53:03.511069+0000","last_clean":"2021-02-10T04:53:03.511069+0000","last_became_active":"2021-02-10T04:28:21.125541+0000","last_became_peered":"2021-02-10T04:28:21.125541+0000","last_unstale":"2021-02-10T04:53:03.511069+0000","last_undegraded":"2021-02-10T04:53:03.511069+0000","last_fullsized":"2021-02-10T04:53:03.511069+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.420086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.420086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"1115","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.514833+0000","last_change":"2021-02-10T04:51:07.984424+0000","last_active":"2021-02-10T04:53:03.514833+0000","last_peered":"2021-02-10T04:53:03.514833+0000","last_clean":"2021-02-10T04:53:03.514833+0000","last_became_active":"2021-02-10T04:50:59.555689+0000","last_became_peered":"2021-02-10T04:50:59.555689+0000","last_unstale":"2021-02-10T04:53:03.514833+0000","last_undegraded":"2021-02-10T04:53:03.514833+0000","last_fullsized":"2021-02-10T04:53:03.514833+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:29:36.395943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:29:36.395943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"1002'2042","reported_seq":"9383","reported_epoch":"1160","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:07.223439+0000","last_change":"2021-02-10T04:52:32.083034+0000","last_active":"2021-02-10T04:53:07.223439+0000","last_peered":"2021-02-10T04:53:07.223439+0000","last_clean":"2021-02-10T04:52:30.068473+0000","last_became_active":"2021-02-10T04:52:32.083034+0000","last_became_peered":"2021-02-10T04:52:32.083034+0000","last_unstale":"2021-02-10T04:53:07.223439+0000","last_undegraded":"2021-02-10T04:52:32.077334+0000","last_fullsized":"2021-02-10T04:52:32.077220+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:46:57.054230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:57.054230+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":85678898,"num_objects":35,"num_object_clones":29,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":35}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"1096","reported_epoch":"1159","state":"active+undersized","last_fresh":"2021-02-10T04:53:03.510161+0000","last_change":"2021-02-10T04:52:32.094777+0000","last_active":"2021-02-10T04:53:03.510161+0000","last_peered":"2021-02-10T04:53:03.510161+0000","last_clean":"2021-02-10T04:52:30.061448+0000","last_became_active":"2021-02-10T04:52:32.094777+0000","last_became_peered":"2021-02-10T04:52:32.094777+0000","last_unstale":"2021-02-10T04:53:03.510161+0000","last_undegraded":"2021-02-10T04:53:03.510161+0000","last_fullsized":"2021-02-10T04:52:32.069238+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:52.341527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:50:52.341527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"1148'3","reported_seq":"39","reported_epoch":"1160","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:08.010709+0000","last_change":"2021-02-10T04:52:32.089249+0000","last_active":"2021-02-10T04:53:08.010709+0000","last_peered":"2021-02-10T04:53:08.010709+0000","last_clean":"2021-02-10T04:25:51.042416+0000","last_became_active":"2021-02-10T04:52:32.089249+0000","last_became_peered":"2021-02-10T04:52:32.089249+0000","last_unstale":"2021-02-10T04:53:08.010709+0000","last_undegraded":"2021-02-10T04:52:32.073980+0000","last_fullsized":"2021-02-10T04:52:32.073877+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:54.466259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:54.466259+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"1049'54","reported_seq":"843","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.684873+0000","last_change":"2021-02-10T04:53:01.304693+0000","last_active":"2021-02-10T04:53:03.684873+0000","last_peered":"2021-02-10T04:53:03.684873+0000","last_clean":"2021-02-10T04:53:03.684873+0000","last_became_active":"2021-02-10T04:51:49.498066+0000","last_became_peered":"2021-02-10T04:51:49.498066+0000","last_unstale":"2021-02-10T04:53:03.684873+0000","last_undegraded":"2021-02-10T04:53:03.684873+0000","last_fullsized":"2021-02-10T04:53:03.684873+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:53:01.304626+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.304626+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":74,"num_read_kb":56,"num_write":42,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"410","reported_epoch":"1159","state":"active+undersized","last_fresh":"2021-02-10T04:53:03.686037+0000","last_change":"2021-02-10T04:52:32.091144+0000","last_active":"2021-02-10T04:53:03.686037+0000","last_peered":"2021-02-10T04:53:03.686037+0000","last_clean":"2021-02-10T04:52:30.060819+0000","last_became_active":"2021-02-10T04:52:32.091144+0000","last_became_peered":"2021-02-10T04:52:32.091144+0000","last_unstale":"2021-02-10T04:53:03.686037+0000","last_undegraded":"2021-02-10T04:53:03.686037+0000","last_fullsized":"2021-02-10T04:52:32.079969+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:49.447961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:49.447961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"1141","reported_epoch":"1159","state":"active+undersized","last_fresh":"2021-02-10T04:53:03.511768+0000","last_change":"2021-02-10T04:52:32.088766+0000","last_active":"2021-02-10T04:53:03.511768+0000","last_peered":"2021-02-10T04:53:03.511768+0000","last_clean":"2021-02-10T04:51:27.993073+0000","last_became_active":"2021-02-10T04:52:32.088766+0000","last_became_peered":"2021-02-10T04:52:32.088766+0000","last_unstale":"2021-02-10T04:53:03.511768+0000","last_undegraded":"2021-02-10T04:53:03.511768+0000","last_fullsized":"2021-02-10T04:52:32.071883+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:07.914061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:07.914061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"293'77","reported_seq":"2939","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.524991+0000","last_change":"2021-02-10T04:52:26.025283+0000","last_active":"2021-02-10T04:52:59.524991+0000","last_peered":"2021-02-10T04:52:59.524991+0000","last_clean":"2021-02-10T04:52:59.524991+0000","last_became_active":"2021-02-10T04:52:26.024812+0000","last_became_peered":"2021-02-10T04:52:26.024812+0000","last_unstale":"2021-02-10T04:52:59.524991+0000","last_undegraded":"2021-02-10T04:52:59.524991+0000","last_fullsized":"2021-02-10T04:52:59.524991+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1886585,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":18587,"num_write":9,"num_write_kb":3315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"1174","reported_epoch":"1160","state":"active+clean","last_fresh":"2021-02-10T04:53:07.223351+0000","last_change":"2021-02-10T04:53:07.025068+0000","last_active":"2021-02-10T04:53:07.223351+0000","last_peered":"2021-02-10T04:53:07.223351+0000","last_clean":"2021-02-10T04:53:07.223351+0000","last_became_active":"2021-02-10T04:50:59.589417+0000","last_became_peered":"2021-02-10T04:50:59.589417+0000","last_unstale":"2021-02-10T04:53:07.223351+0000","last_undegraded":"2021-02-10T04:53:07.223351+0000","last_fullsized":"2021-02-10T04:53:07.223351+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.024980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.024980+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"56","reported_epoch":"1158","state":"active+undersized","last_fresh":"2021-02-10T04:52:59.513238+0000","last_change":"2021-02-10T04:52:32.082945+0000","last_active":"2021-02-10T04:52:59.513238+0000","last_peered":"2021-02-10T04:52:59.513238+0000","last_clean":"2021-02-10T04:52:30.056402+0000","last_became_active":"2021-02-10T04:52:32.082945+0000","last_became_peered":"2021-02-10T04:52:32.082945+0000","last_unstale":"2021-02-10T04:52:59.513238+0000","last_undegraded":"2021-02-10T04:52:59.513238+0000","last_fullsized":"2021-02-10T04:52:32.077773+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.609500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.609500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"1048'39","reported_seq":"1249","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.525327+0000","last_change":"2021-02-10T04:50:44.624567+0000","last_active":"2021-02-10T04:52:59.525327+0000","last_peered":"2021-02-10T04:52:59.525327+0000","last_clean":"2021-02-10T04:52:59.525327+0000","last_became_active":"2021-02-10T04:39:02.560739+0000","last_became_peered":"2021-02-10T04:39:02.560739+0000","last_unstale":"2021-02-10T04:52:59.525327+0000","last_undegraded":"2021-02-10T04:52:59.525327+0000","last_fullsized":"2021-02-10T04:52:59.525327+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'39","last_scrub_stamp":"2021-02-10T04:50:44.624501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:50:44.624501+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":76,"num_read_kb":59,"num_write":41,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"1264","reported_epoch":"1158","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:52:59.529105+0000","last_change":"2021-02-10T04:52:32.084961+0000","last_active":"2021-02-10T04:52:59.529105+0000","last_peered":"2021-02-10T04:52:59.529105+0000","last_clean":"2021-02-10T04:52:30.813009+0000","last_became_active":"2021-02-10T04:52:32.084961+0000","last_became_peered":"2021-02-10T04:52:32.084961+0000","last_unstale":"2021-02-10T04:52:59.529105+0000","last_undegraded":"2021-02-10T04:52:32.070468+0000","last_fullsized":"2021-02-10T04:52:32.070358+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:47:08.784593+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:47:08.784593+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"1122","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.514915+0000","last_change":"2021-02-10T04:52:30.019225+0000","last_active":"2021-02-10T04:53:03.514915+0000","last_peered":"2021-02-10T04:53:03.514915+0000","last_clean":"2021-02-10T04:53:03.514915+0000","last_became_active":"2021-02-10T04:50:59.561436+0000","last_became_peered":"2021-02-10T04:50:59.561436+0000","last_unstale":"2021-02-10T04:53:03.514915+0000","last_undegraded":"2021-02-10T04:53:03.514915+0000","last_fullsized":"2021-02-10T04:53:03.514915+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.019102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.019102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"1001'1528","reported_seq":"6107","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.514065+0000","last_change":"2021-02-10T04:52:26.036366+0000","last_active":"2021-02-10T04:53:03.514065+0000","last_peered":"2021-02-10T04:53:03.514065+0000","last_clean":"2021-02-10T04:53:03.514065+0000","last_became_active":"2021-02-10T04:52:26.034028+0000","last_became_peered":"2021-02-10T04:52:26.034028+0000","last_unstale":"2021-02-10T04:53:03.514065+0000","last_undegraded":"2021-02-10T04:53:03.514065+0000","last_fullsized":"2021-02-10T04:53:03.514065+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46944136,"num_objects":16,"num_object_clones":15,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":358,"num_read_kb":351934,"num_write":286,"num_write_kb":111355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":58968832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.15","version":"1000'446","reported_seq":"2091","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.527433+0000","last_change":"2021-02-10T04:51:40.511548+0000","last_active":"2021-02-10T04:52:59.527433+0000","last_peered":"2021-02-10T04:52:59.527433+0000","last_clean":"2021-02-10T04:52:59.527433+0000","last_became_active":"2021-02-10T04:51:40.510899+0000","last_became_peered":"2021-02-10T04:51:40.510899+0000","last_unstale":"2021-02-10T04:52:59.527433+0000","last_undegraded":"2021-02-10T04:52:59.527433+0000","last_fullsized":"2021-02-10T04:52:59.527433+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:50:05.815135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:50:05.815135+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24896285,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":18983408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"1048'40","reported_seq":"1112","reported_epoch":"1159","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:03.515454+0000","last_change":"2021-02-10T04:52:32.086896+0000","last_active":"2021-02-10T04:53:03.515454+0000","last_peered":"2021-02-10T04:53:03.515454+0000","last_clean":"2021-02-10T04:45:56.447986+0000","last_became_active":"2021-02-10T04:52:32.086896+0000","last_became_peered":"2021-02-10T04:52:32.086896+0000","last_unstale":"2021-02-10T04:53:03.515454+0000","last_undegraded":"2021-02-10T04:52:32.076620+0000","last_fullsized":"2021-02-10T04:52:32.076502+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'40","last_scrub_stamp":"2021-02-10T04:47:29.965140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:47:29.965140+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":79,"num_read_kb":57,"num_write":41,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":13}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"60","reported_epoch":"1158","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:52:59.513722+0000","last_change":"2021-02-10T04:52:32.094875+0000","last_active":"2021-02-10T04:52:59.513722+0000","last_peered":"2021-02-10T04:52:59.513722+0000","last_clean":"2021-02-10T04:52:30.056787+0000","last_became_active":"2021-02-10T04:52:32.094875+0000","last_became_peered":"2021-02-10T04:52:32.094875+0000","last_unstale":"2021-02-10T04:52:59.513722+0000","last_undegraded":"2021-02-10T04:52:32.083818+0000","last_fullsized":"2021-02-10T04:52:32.083707+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:51:08.639784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.639784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"1091","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.515548+0000","last_change":"2021-02-10T04:51:09.475150+0000","last_active":"2021-02-10T04:53:03.515548+0000","last_peered":"2021-02-10T04:53:03.515548+0000","last_clean":"2021-02-10T04:53:03.515548+0000","last_became_active":"2021-02-10T04:50:59.585917+0000","last_became_peered":"2021-02-10T04:50:59.585917+0000","last_unstale":"2021-02-10T04:53:03.515548+0000","last_undegraded":"2021-02-10T04:53:03.515548+0000","last_fullsized":"2021-02-10T04:53:03.515548+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:09.475063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:09.475063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"55","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.514210+0000","last_change":"2021-02-10T04:51:44.970615+0000","last_active":"2021-02-10T04:52:59.514210+0000","last_peered":"2021-02-10T04:52:59.514210+0000","last_clean":"2021-02-10T04:52:59.514210+0000","last_became_active":"2021-02-10T04:51:36.859707+0000","last_became_peered":"2021-02-10T04:51:36.859707+0000","last_unstale":"2021-02-10T04:52:59.514210+0000","last_undegraded":"2021-02-10T04:52:59.514210+0000","last_fullsized":"2021-02-10T04:52:59.514210+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:44.970511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.970511+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"1000'620","reported_seq":"4441","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.514788+0000","last_change":"2021-02-10T04:51:40.511426+0000","last_active":"2021-02-10T04:53:03.514788+0000","last_peered":"2021-02-10T04:53:03.514788+0000","last_clean":"2021-02-10T04:53:03.514788+0000","last_became_active":"2021-02-10T04:51:40.510114+0000","last_became_peered":"2021-02-10T04:51:40.510114+0000","last_unstale":"2021-02-10T04:53:03.514788+0000","last_undegraded":"2021-02-10T04:53:03.514788+0000","last_fullsized":"2021-02-10T04:53:03.514788+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:47:24.897680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:24.897680+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14265195,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":144,"num_read_kb":128483,"num_write":129,"num_write_kb":53444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":39998538,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"1063'30","reported_seq":"1191","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.515190+0000","last_change":"2021-02-10T04:52:43.102272+0000","last_active":"2021-02-10T04:53:03.515190+0000","last_peered":"2021-02-10T04:53:03.515190+0000","last_clean":"2021-02-10T04:53:03.515190+0000","last_became_active":"2021-02-10T04:50:59.587330+0000","last_became_peered":"2021-02-10T04:50:59.587330+0000","last_unstale":"2021-02-10T04:53:03.515190+0000","last_undegraded":"2021-02-10T04:53:03.515190+0000","last_fullsized":"2021-02-10T04:53:03.515190+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1063'30","last_scrub_stamp":"2021-02-10T04:52:43.102165+0000","last_deep_scrub":"1063'30","last_deep_scrub_stamp":"2021-02-10T04:47:05.589582+0000","last_clean_scrub_stamp":"2021-02-10T04:52:43.102165+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"1148","reported_epoch":"1160","state":"active+clean","last_fresh":"2021-02-10T04:53:07.222241+0000","last_change":"2021-02-10T04:52:41.059269+0000","last_active":"2021-02-10T04:53:07.222241+0000","last_peered":"2021-02-10T04:53:07.222241+0000","last_clean":"2021-02-10T04:53:07.222241+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:53:07.222241+0000","last_undegraded":"2021-02-10T04:53:07.222241+0000","last_fullsized":"2021-02-10T04:53:07.222241+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:52:41.059159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.059159+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"1146","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.519888+0000","last_change":"2021-02-10T04:51:08.451362+0000","last_active":"2021-02-10T04:53:03.519888+0000","last_peered":"2021-02-10T04:53:03.519888+0000","last_clean":"2021-02-10T04:53:03.519888+0000","last_became_active":"2021-02-10T04:50:59.540375+0000","last_became_peered":"2021-02-10T04:50:59.540375+0000","last_unstale":"2021-02-10T04:53:03.519888+0000","last_undegraded":"2021-02-10T04:53:03.519888+0000","last_fullsized":"2021-02-10T04:53:03.519888+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:08.451289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.451289+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"1057","reported_epoch":"1160","state":"active+clean","last_fresh":"2021-02-10T04:53:07.223028+0000","last_change":"2021-02-10T04:50:45.362633+0000","last_active":"2021-02-10T04:53:07.223028+0000","last_peered":"2021-02-10T04:53:07.223028+0000","last_clean":"2021-02-10T04:53:07.223028+0000","last_became_active":"2021-02-10T04:29:07.147249+0000","last_became_peered":"2021-02-10T04:29:07.147249+0000","last_unstale":"2021-02-10T04:53:07.223028+0000","last_undegraded":"2021-02-10T04:53:07.223028+0000","last_fullsized":"2021-02-10T04:53:07.223028+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:45.362537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:45.362537+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.17","version":"1002'2042","reported_seq":"9415","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.509337+0000","last_change":"2021-02-10T04:52:41.286103+0000","last_active":"2021-02-10T04:53:03.509337+0000","last_peered":"2021-02-10T04:53:03.509337+0000","last_clean":"2021-02-10T04:53:03.509337+0000","last_became_active":"2021-02-10T04:51:41.509707+0000","last_became_peered":"2021-02-10T04:51:41.509707+0000","last_unstale":"2021-02-10T04:53:03.509337+0000","last_undegraded":"2021-02-10T04:53:03.509337+0000","last_fullsized":"2021-02-10T04:53:03.509337+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1124,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:52:41.286006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.286006+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6921254,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":5403183,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"1049'46","reported_seq":"1315","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.515119+0000","last_change":"2021-02-10T04:52:35.091494+0000","last_active":"2021-02-10T04:53:03.515119+0000","last_peered":"2021-02-10T04:53:03.515119+0000","last_clean":"2021-02-10T04:53:03.515119+0000","last_became_active":"2021-02-10T04:50:59.569915+0000","last_became_peered":"2021-02-10T04:50:59.569915+0000","last_unstale":"2021-02-10T04:53:03.515119+0000","last_undegraded":"2021-02-10T04:53:03.515119+0000","last_fullsized":"2021-02-10T04:53:03.515119+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'46","last_scrub_stamp":"2021-02-10T04:52:35.091374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:35.091374+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":65,"num_write":46,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"84'7","reported_seq":"1175","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.510985+0000","last_change":"2021-02-10T04:51:52.498251+0000","last_active":"2021-02-10T04:53:03.510985+0000","last_peered":"2021-02-10T04:53:03.510985+0000","last_clean":"2021-02-10T04:53:03.510985+0000","last_became_active":"2021-02-10T04:50:59.575924+0000","last_became_peered":"2021-02-10T04:50:59.575924+0000","last_unstale":"2021-02-10T04:53:03.510985+0000","last_undegraded":"2021-02-10T04:53:03.510985+0000","last_fullsized":"2021-02-10T04:53:03.510985+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'7","last_scrub_stamp":"2021-02-10T04:51:52.498158+0000","last_deep_scrub":"84'7","last_deep_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.498158+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"1148'3","reported_seq":"1467","reported_epoch":"1160","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:07.227020+0000","last_change":"2021-02-10T04:52:32.094670+0000","last_active":"2021-02-10T04:53:07.227020+0000","last_peered":"2021-02-10T04:53:07.227020+0000","last_clean":"2021-02-10T04:52:30.059221+0000","last_became_active":"2021-02-10T04:52:32.094670+0000","last_became_peered":"2021-02-10T04:52:32.094670+0000","last_unstale":"2021-02-10T04:53:07.227020+0000","last_undegraded":"2021-02-10T04:52:32.078919+0000","last_fullsized":"2021-02-10T04:52:32.078798+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:47.407557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:47.407557+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"1107","reported_epoch":"1159","state":"active+undersized","last_fresh":"2021-02-10T04:53:03.515322+0000","last_change":"2021-02-10T04:52:32.085745+0000","last_active":"2021-02-10T04:53:03.515322+0000","last_peered":"2021-02-10T04:53:03.515322+0000","last_clean":"2021-02-10T04:49:56.939276+0000","last_became_active":"2021-02-10T04:52:32.085745+0000","last_became_peered":"2021-02-10T04:52:32.085745+0000","last_unstale":"2021-02-10T04:53:03.515322+0000","last_undegraded":"2021-02-10T04:53:03.515322+0000","last_fullsized":"2021-02-10T04:52:32.076058+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:23.566027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_clean_scrub_stamp":"2021-02-10T04:52:23.566027+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.16","version":"1000'490","reported_seq":"3666","reported_epoch":"1158","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:52:59.528422+0000","last_change":"2021-02-10T04:52:32.077228+0000","last_active":"2021-02-10T04:52:59.528422+0000","last_peered":"2021-02-10T04:52:59.528422+0000","last_clean":"2021-02-10T04:52:30.062406+0000","last_became_active":"2021-02-10T04:52:32.077228+0000","last_became_peered":"2021-02-10T04:52:32.077228+0000","last_unstale":"2021-02-10T04:52:59.528422+0000","last_undegraded":"2021-02-10T04:52:32.071522+0000","last_fullsized":"2021-02-10T04:52:32.071453+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1013,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:50:03.756248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:50:03.756248+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16154790,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"1112'40","reported_seq":"1214","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.510663+0000","last_change":"2021-02-10T04:51:01.450124+0000","last_active":"2021-02-10T04:53:03.510663+0000","last_peered":"2021-02-10T04:53:03.510663+0000","last_clean":"2021-02-10T04:53:03.510663+0000","last_became_active":"2021-02-10T04:50:59.539529+0000","last_became_peered":"2021-02-10T04:50:59.539529+0000","last_unstale":"2021-02-10T04:53:03.510663+0000","last_undegraded":"2021-02-10T04:53:03.510663+0000","last_fullsized":"2021-02-10T04:53:03.510663+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1112'40","last_scrub_stamp":"2021-02-10T04:51:01.450050+0000","last_deep_scrub":"1112'40","last_deep_scrub_stamp":"2021-02-10T04:51:01.450050+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.450050+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":59,"num_read_kb":44,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"1123","reported_epoch":"1158","state":"active+undersized","last_fresh":"2021-02-10T04:52:59.529625+0000","last_change":"2021-02-10T04:52:32.080868+0000","last_active":"2021-02-10T04:52:59.529625+0000","last_peered":"2021-02-10T04:52:59.529625+0000","last_clean":"2021-02-10T04:49:56.937464+0000","last_became_active":"2021-02-10T04:52:32.080868+0000","last_became_peered":"2021-02-10T04:52:32.080868+0000","last_unstale":"2021-02-10T04:52:59.529625+0000","last_undegraded":"2021-02-10T04:52:59.529625+0000","last_fullsized":"2021-02-10T04:52:32.068376+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:23.904210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:47:23.904210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"1119'6","reported_seq":"1390","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.515549+0000","last_change":"2021-02-10T04:52:40.038646+0000","last_active":"2021-02-10T04:53:03.515549+0000","last_peered":"2021-02-10T04:53:03.515549+0000","last_clean":"2021-02-10T04:53:03.515549+0000","last_became_active":"2021-02-10T04:51:36.869733+0000","last_became_peered":"2021-02-10T04:51:36.869733+0000","last_unstale":"2021-02-10T04:53:03.515549+0000","last_undegraded":"2021-02-10T04:53:03.515549+0000","last_fullsized":"2021-02-10T04:53:03.515549+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'6","last_scrub_stamp":"2021-02-10T04:52:40.038554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:40.038554+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"1145","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.529488+0000","last_change":"2021-02-10T04:51:36.849933+0000","last_active":"2021-02-10T04:52:59.529488+0000","last_peered":"2021-02-10T04:52:59.529488+0000","last_clean":"2021-02-10T04:52:59.529488+0000","last_became_active":"2021-02-10T04:51:36.849184+0000","last_became_peered":"2021-02-10T04:51:36.849184+0000","last_unstale":"2021-02-10T04:52:59.529488+0000","last_undegraded":"2021-02-10T04:52:59.529488+0000","last_fullsized":"2021-02-10T04:52:59.529488+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:11.779779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:11.779779+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"1032'89","reported_seq":"931","reported_epoch":"1160","state":"active+clean","last_fresh":"2021-02-10T04:53:07.222333+0000","last_change":"2021-02-10T04:52:34.089878+0000","last_active":"2021-02-10T04:53:07.222333+0000","last_peered":"2021-02-10T04:53:07.222333+0000","last_clean":"2021-02-10T04:53:07.222333+0000","last_became_active":"2021-02-10T04:50:59.528302+0000","last_became_peered":"2021-02-10T04:50:59.528302+0000","last_unstale":"2021-02-10T04:53:07.222333+0000","last_undegraded":"2021-02-10T04:53:07.222333+0000","last_fullsized":"2021-02-10T04:53:07.222333+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":5,"last_scrub":"1032'89","last_scrub_stamp":"2021-02-10T04:52:34.089789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:34.089789+0000","log_size":89,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5949711,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"1048'27","reported_seq":"827","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.684791+0000","last_change":"2021-02-10T04:51:49.497961+0000","last_active":"2021-02-10T04:53:03.684791+0000","last_peered":"2021-02-10T04:53:03.684791+0000","last_clean":"2021-02-10T04:53:03.684791+0000","last_became_active":"2021-02-10T04:51:49.496294+0000","last_became_peered":"2021-02-10T04:51:49.496294+0000","last_unstale":"2021-02-10T04:53:03.684791+0000","last_undegraded":"2021-02-10T04:53:03.684791+0000","last_fullsized":"2021-02-10T04:53:03.684791+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'27","last_scrub_stamp":"2021-02-10T04:50:55.142479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:50:55.142479+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":33,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"1161","reported_epoch":"1160","state":"active+clean","last_fresh":"2021-02-10T04:53:07.222118+0000","last_change":"2021-02-10T04:52:23.095651+0000","last_active":"2021-02-10T04:53:07.222118+0000","last_peered":"2021-02-10T04:53:07.222118+0000","last_clean":"2021-02-10T04:53:07.222118+0000","last_became_active":"2021-02-10T04:50:59.543531+0000","last_became_peered":"2021-02-10T04:50:59.543531+0000","last_unstale":"2021-02-10T04:53:07.222118+0000","last_undegraded":"2021-02-10T04:53:07.222118+0000","last_fullsized":"2021-02-10T04:53:07.222118+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:23.095588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:23.095588+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"1101","reported_epoch":"1159","state":"active+undersized","last_fresh":"2021-02-10T04:53:03.514598+0000","last_change":"2021-02-10T04:52:32.085370+0000","last_active":"2021-02-10T04:53:03.514598+0000","last_peered":"2021-02-10T04:53:03.514598+0000","last_clean":"2021-02-10T04:52:30.067066+0000","last_became_active":"2021-02-10T04:52:32.085370+0000","last_became_peered":"2021-02-10T04:52:32.085370+0000","last_unstale":"2021-02-10T04:53:03.514598+0000","last_undegraded":"2021-02-10T04:53:03.514598+0000","last_fullsized":"2021-02-10T04:52:32.074435+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:43.663762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:51:43.663762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"21","reported_epoch":"1159","state":"active+undersized","last_fresh":"2021-02-10T04:53:03.509536+0000","last_change":"2021-02-10T04:52:32.091922+0000","last_active":"2021-02-10T04:53:03.509536+0000","last_peered":"2021-02-10T04:53:03.509536+0000","last_clean":"2021-02-10T04:25:52.015519+0000","last_became_active":"2021-02-10T04:52:32.091922+0000","last_became_peered":"2021-02-10T04:52:32.091922+0000","last_unstale":"2021-02-10T04:53:03.509536+0000","last_undegraded":"2021-02-10T04:53:03.509536+0000","last_fullsized":"2021-02-10T04:52:32.069065+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:57.568541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:57.568541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.10","version":"1000'1071","reported_seq":"4660","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.511677+0000","last_change":"2021-02-10T04:51:39.509148+0000","last_active":"2021-02-10T04:53:03.511677+0000","last_peered":"2021-02-10T04:53:03.511677+0000","last_clean":"2021-02-10T04:53:03.511677+0000","last_became_active":"2021-02-10T04:51:39.508551+0000","last_became_peered":"2021-02-10T04:51:39.508551+0000","last_unstale":"2021-02-10T04:53:03.511677+0000","last_undegraded":"2021-02-10T04:53:03.511677+0000","last_fullsized":"2021-02-10T04:53:03.511677+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":5,"last_scrub":"412'357","last_scrub_stamp":"2021-02-10T04:33:50.582002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:50.582002+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23286368,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":242,"num_read_kb":226835,"num_write":185,"num_write_kb":76005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"1049'31","reported_seq":"1182","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.529934+0000","last_change":"2021-02-10T04:52:39.695340+0000","last_active":"2021-02-10T04:52:59.529934+0000","last_peered":"2021-02-10T04:52:59.529934+0000","last_clean":"2021-02-10T04:52:59.529934+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:52:59.529934+0000","last_undegraded":"2021-02-10T04:52:59.529934+0000","last_fullsized":"2021-02-10T04:52:59.529934+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'31","last_scrub_stamp":"2021-02-10T04:52:39.695237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:39.695237+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":41,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"392","reported_epoch":"1159","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:03.685985+0000","last_change":"2021-02-10T04:52:32.094351+0000","last_active":"2021-02-10T04:53:03.685985+0000","last_peered":"2021-02-10T04:53:03.685985+0000","last_clean":"2021-02-10T04:52:30.061022+0000","last_became_active":"2021-02-10T04:52:32.094351+0000","last_became_peered":"2021-02-10T04:52:32.094351+0000","last_unstale":"2021-02-10T04:53:03.685985+0000","last_undegraded":"2021-02-10T04:52:32.074350+0000","last_fullsized":"2021-02-10T04:52:32.074239+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:47:08.700069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:08.700069+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"809","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.684951+0000","last_change":"2021-02-10T04:51:49.501612+0000","last_active":"2021-02-10T04:53:03.684951+0000","last_peered":"2021-02-10T04:53:03.684951+0000","last_clean":"2021-02-10T04:53:03.684951+0000","last_became_active":"2021-02-10T04:51:49.501254+0000","last_became_peered":"2021-02-10T04:51:49.501254+0000","last_unstale":"2021-02-10T04:53:03.684951+0000","last_undegraded":"2021-02-10T04:53:03.684951+0000","last_fullsized":"2021-02-10T04:53:03.684951+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:05.475145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:51:05.475145+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"1128","reported_epoch":"1159","state":"active+undersized","last_fresh":"2021-02-10T04:53:03.512805+0000","last_change":"2021-02-10T04:52:32.087218+0000","last_active":"2021-02-10T04:53:03.512805+0000","last_peered":"2021-02-10T04:53:03.512805+0000","last_clean":"2021-02-10T04:52:30.062886+0000","last_became_active":"2021-02-10T04:52:32.087218+0000","last_became_peered":"2021-02-10T04:52:32.087218+0000","last_unstale":"2021-02-10T04:53:03.512805+0000","last_undegraded":"2021-02-10T04:53:03.512805+0000","last_fullsized":"2021-02-10T04:52:32.072167+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:55.814486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.814486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"1002'1657","reported_seq":"5059","reported_epoch":"1158","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:52:59.513476+0000","last_change":"2021-02-10T04:52:32.088477+0000","last_active":"2021-02-10T04:52:59.513476+0000","last_peered":"2021-02-10T04:52:59.513476+0000","last_clean":"2021-02-10T04:51:27.993324+0000","last_became_active":"2021-02-10T04:52:32.088477+0000","last_became_peered":"2021-02-10T04:52:32.088477+0000","last_unstale":"2021-02-10T04:52:59.513476+0000","last_undegraded":"2021-02-10T04:52:32.081970+0000","last_fullsized":"2021-02-10T04:52:32.081848+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1124,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:46:54.881412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:54.881412+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":47335497,"num_objects":20,"num_object_clones":18,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":20}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"1048'20","reported_seq":"1261","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:04.058398+0000","last_change":"2021-02-10T04:53:04.058324+0000","last_active":"2021-02-10T04:53:04.058398+0000","last_peered":"2021-02-10T04:53:04.058398+0000","last_clean":"2021-02-10T04:53:04.058398+0000","last_became_active":"2021-02-10T04:50:59.544484+0000","last_became_peered":"2021-02-10T04:50:59.544484+0000","last_unstale":"2021-02-10T04:53:04.058398+0000","last_undegraded":"2021-02-10T04:53:04.058398+0000","last_fullsized":"2021-02-10T04:53:04.058398+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'20","last_scrub_stamp":"2021-02-10T04:53:04.058248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:04.058248+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":20,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"1163","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.519985+0000","last_change":"2021-02-10T04:53:00.343307+0000","last_active":"2021-02-10T04:53:03.519985+0000","last_peered":"2021-02-10T04:53:03.519985+0000","last_clean":"2021-02-10T04:53:03.519985+0000","last_became_active":"2021-02-10T04:51:36.844573+0000","last_became_peered":"2021-02-10T04:51:36.844573+0000","last_unstale":"2021-02-10T04:53:03.519985+0000","last_undegraded":"2021-02-10T04:53:03.519985+0000","last_fullsized":"2021-02-10T04:53:03.519985+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:00.343228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:00.343228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"1132","reported_epoch":"1160","state":"active+clean","last_fresh":"2021-02-10T04:53:07.222156+0000","last_change":"2021-02-10T04:50:56.099710+0000","last_active":"2021-02-10T04:53:07.222156+0000","last_peered":"2021-02-10T04:53:07.222156+0000","last_clean":"2021-02-10T04:53:07.222156+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:53:07.222156+0000","last_undegraded":"2021-02-10T04:53:07.222156+0000","last_fullsized":"2021-02-10T04:53:07.222156+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:56.099651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:50:56.099651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"55","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.512793+0000","last_change":"2021-02-10T04:51:50.036214+0000","last_active":"2021-02-10T04:52:59.512793+0000","last_peered":"2021-02-10T04:52:59.512793+0000","last_clean":"2021-02-10T04:52:59.512793+0000","last_became_active":"2021-02-10T04:51:36.855910+0000","last_became_peered":"2021-02-10T04:51:36.855910+0000","last_unstale":"2021-02-10T04:52:59.512793+0000","last_undegraded":"2021-02-10T04:52:59.512793+0000","last_fullsized":"2021-02-10T04:52:59.512793+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.036117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.036117+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"998'1009","reported_seq":"4999","reported_epoch":"1160","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:07.221768+0000","last_change":"2021-02-10T04:52:32.090443+0000","last_active":"2021-02-10T04:53:07.221768+0000","last_peered":"2021-02-10T04:53:07.221768+0000","last_clean":"2021-02-10T04:52:30.061024+0000","last_became_active":"2021-02-10T04:52:32.090443+0000","last_became_peered":"2021-02-10T04:52:32.090443+0000","last_unstale":"2021-02-10T04:53:07.221768+0000","last_undegraded":"2021-02-10T04:52:32.076916+0000","last_fullsized":"2021-02-10T04:52:32.076790+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1127,"parent":"0.0","parent_split_bits":5,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:13.032506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:13.032506+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49420827,"num_objects":21,"num_object_clones":19,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":36763798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":21}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"1049'37","reported_seq":"1189","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.513498+0000","last_change":"2021-02-10T04:51:44.694779+0000","last_active":"2021-02-10T04:53:03.513498+0000","last_peered":"2021-02-10T04:53:03.513498+0000","last_clean":"2021-02-10T04:53:03.513498+0000","last_became_active":"2021-02-10T04:50:59.549412+0000","last_became_peered":"2021-02-10T04:50:59.549412+0000","last_unstale":"2021-02-10T04:53:03.513498+0000","last_undegraded":"2021-02-10T04:53:03.513498+0000","last_fullsized":"2021-02-10T04:53:03.513498+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:44.694711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.694711+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":64,"num_read_kb":50,"num_write":37,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"1299","reported_epoch":"1160","state":"active+clean","last_fresh":"2021-02-10T04:53:07.223714+0000","last_change":"2021-02-10T04:52:33.088525+0000","last_active":"2021-02-10T04:53:07.223714+0000","last_peered":"2021-02-10T04:53:07.223714+0000","last_clean":"2021-02-10T04:53:07.223714+0000","last_became_active":"2021-02-10T04:51:36.870573+0000","last_became_peered":"2021-02-10T04:51:36.870573+0000","last_unstale":"2021-02-10T04:53:07.223714+0000","last_undegraded":"2021-02-10T04:53:07.223714+0000","last_fullsized":"2021-02-10T04:53:07.223714+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:52:33.088428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:33.088428+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"445","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.512320+0000","last_change":"2021-02-10T04:51:51.836723+0000","last_active":"2021-02-10T04:53:03.512320+0000","last_peered":"2021-02-10T04:53:03.512320+0000","last_clean":"2021-02-10T04:53:03.512320+0000","last_became_active":"2021-02-10T04:50:59.564977+0000","last_became_peered":"2021-02-10T04:50:59.564977+0000","last_unstale":"2021-02-10T04:53:03.512320+0000","last_undegraded":"2021-02-10T04:53:03.512320+0000","last_fullsized":"2021-02-10T04:53:03.512320+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.836638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.836638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"1092","reported_epoch":"1159","state":"active+undersized","last_fresh":"2021-02-10T04:53:03.514580+0000","last_change":"2021-02-10T04:52:32.081484+0000","last_active":"2021-02-10T04:53:03.514580+0000","last_peered":"2021-02-10T04:53:03.514580+0000","last_clean":"2021-02-10T04:52:30.067390+0000","last_became_active":"2021-02-10T04:52:32.081484+0000","last_became_peered":"2021-02-10T04:52:32.081484+0000","last_unstale":"2021-02-10T04:53:03.514580+0000","last_undegraded":"2021-02-10T04:53:03.514580+0000","last_fullsized":"2021-02-10T04:52:32.074112+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:58.514318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_clean_scrub_stamp":"2021-02-10T04:46:58.514318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1d","version":"1001'1165","reported_seq":"4153","reported_epoch":"1159","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:03.686147+0000","last_change":"2021-02-10T04:52:32.087778+0000","last_active":"2021-02-10T04:53:03.686147+0000","last_peered":"2021-02-10T04:53:03.686147+0000","last_clean":"2021-02-10T04:52:30.061103+0000","last_became_active":"2021-02-10T04:52:32.087778+0000","last_became_peered":"2021-02-10T04:52:32.087778+0000","last_unstale":"2021-02-10T04:53:03.686147+0000","last_undegraded":"2021-02-10T04:52:32.074694+0000","last_fullsized":"2021-02-10T04:52:32.074582+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1144,"parent":"0.0","parent_split_bits":5,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29601845,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":287,"num_read_kb":299530,"num_write":200,"num_write_kb":89131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":13}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"1049'42","reported_seq":"1107","reported_epoch":"1159","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:03.515686+0000","last_change":"2021-02-10T04:52:32.088591+0000","last_active":"2021-02-10T04:53:03.515686+0000","last_peered":"2021-02-10T04:53:03.515686+0000","last_clean":"2021-02-10T04:45:56.729967+0000","last_became_active":"2021-02-10T04:52:32.088591+0000","last_became_peered":"2021-02-10T04:52:32.088591+0000","last_unstale":"2021-02-10T04:53:03.515686+0000","last_undegraded":"2021-02-10T04:52:32.077147+0000","last_fullsized":"2021-02-10T04:52:32.077086+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'42","last_scrub_stamp":"2021-02-10T04:51:45.336772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.336772+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":74,"num_read_kb":56,"num_write":42,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":12}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"1204","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.509435+0000","last_change":"2021-02-10T04:53:01.331566+0000","last_active":"2021-02-10T04:53:03.509435+0000","last_peered":"2021-02-10T04:53:03.509435+0000","last_clean":"2021-02-10T04:53:03.509435+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:53:03.509435+0000","last_undegraded":"2021-02-10T04:53:03.509435+0000","last_fullsized":"2021-02-10T04:53:03.509435+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:01.331466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.331466+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"1098'3","reported_seq":"1463","reported_epoch":"1160","state":"active+clean","last_fresh":"2021-02-10T04:53:08.010701+0000","last_change":"2021-02-10T04:50:15.255211+0000","last_active":"2021-02-10T04:53:08.010701+0000","last_peered":"2021-02-10T04:53:08.010701+0000","last_clean":"2021-02-10T04:53:08.010701+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:53:08.010701+0000","last_undegraded":"2021-02-10T04:53:08.010701+0000","last_fullsized":"2021-02-10T04:53:08.010701+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"1098'3","last_scrub_stamp":"2021-02-10T04:50:15.255137+0000","last_deep_scrub":"1098'3","last_deep_scrub_stamp":"2021-02-10T04:50:15.255137+0000","last_clean_scrub_stamp":"2021-02-10T04:50:15.255137+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"1083","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.514207+0000","last_change":"2021-02-10T04:51:47.493125+0000","last_active":"2021-02-10T04:53:03.514207+0000","last_peered":"2021-02-10T04:53:03.514207+0000","last_clean":"2021-02-10T04:53:03.514207+0000","last_became_active":"2021-02-10T04:51:36.852591+0000","last_became_peered":"2021-02-10T04:51:36.852591+0000","last_unstale":"2021-02-10T04:53:03.514207+0000","last_undegraded":"2021-02-10T04:53:03.514207+0000","last_fullsized":"2021-02-10T04:53:03.514207+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:47.493057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.493057+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1c","version":"1002'1090","reported_seq":"5179","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.513908+0000","last_change":"2021-02-10T04:52:31.088797+0000","last_active":"2021-02-10T04:53:03.513908+0000","last_peered":"2021-02-10T04:53:03.513908+0000","last_clean":"2021-02-10T04:53:03.513908+0000","last_became_active":"2021-02-10T04:52:31.088265+0000","last_became_peered":"2021-02-10T04:52:31.088265+0000","last_unstale":"2021-02-10T04:53:03.513908+0000","last_undegraded":"2021-02-10T04:53:03.513908+0000","last_fullsized":"2021-02-10T04:53:03.513908+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:51:08.220472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.220472+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24410646,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":246,"num_read_kb":276168,"num_write":176,"num_write_kb":64961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"1049'51","reported_seq":"1221","reported_epoch":"1160","state":"active+clean","last_fresh":"2021-02-10T04:53:07.223775+0000","last_change":"2021-02-10T04:50:53.153340+0000","last_active":"2021-02-10T04:53:07.223775+0000","last_peered":"2021-02-10T04:53:07.223775+0000","last_clean":"2021-02-10T04:53:07.223775+0000","last_became_active":"2021-02-10T04:25:49.015760+0000","last_became_peered":"2021-02-10T04:25:49.015760+0000","last_unstale":"2021-02-10T04:53:07.223775+0000","last_undegraded":"2021-02-10T04:53:07.223775+0000","last_fullsized":"2021-02-10T04:53:07.223775+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'51","last_scrub_stamp":"2021-02-10T04:50:53.153251+0000","last_deep_scrub":"76'35","last_deep_scrub_stamp":"2021-02-10T04:26:58.198641+0000","last_clean_scrub_stamp":"2021-02-10T04:50:53.153251+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":74,"num_read_kb":54,"num_write":40,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"1096","reported_epoch":"1159","state":"active+undersized","last_fresh":"2021-02-10T04:53:03.513242+0000","last_change":"2021-02-10T04:52:32.079189+0000","last_active":"2021-02-10T04:53:03.513242+0000","last_peered":"2021-02-10T04:53:03.513242+0000","last_clean":"2021-02-10T04:52:30.063079+0000","last_became_active":"2021-02-10T04:52:32.079189+0000","last_became_peered":"2021-02-10T04:52:32.079189+0000","last_unstale":"2021-02-10T04:53:03.513242+0000","last_undegraded":"2021-02-10T04:53:03.513242+0000","last_fullsized":"2021-02-10T04:52:32.071674+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:42.537443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:42.537443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"1119","reported_epoch":"1158","state":"active+undersized","last_fresh":"2021-02-10T04:52:59.515107+0000","last_change":"2021-02-10T04:52:32.088503+0000","last_active":"2021-02-10T04:52:59.515107+0000","last_peered":"2021-02-10T04:52:59.515107+0000","last_clean":"2021-02-10T04:51:27.994110+0000","last_became_active":"2021-02-10T04:52:32.088503+0000","last_became_peered":"2021-02-10T04:52:32.088503+0000","last_unstale":"2021-02-10T04:52:59.515107+0000","last_undegraded":"2021-02-10T04:52:59.515107+0000","last_fullsized":"2021-02-10T04:52:32.082093+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.938731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.938731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"366","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.685790+0000","last_change":"2021-02-10T04:51:49.499732+0000","last_active":"2021-02-10T04:53:03.685790+0000","last_peered":"2021-02-10T04:53:03.685790+0000","last_clean":"2021-02-10T04:53:03.685790+0000","last_became_active":"2021-02-10T04:51:49.499348+0000","last_became_peered":"2021-02-10T04:51:49.499348+0000","last_unstale":"2021-02-10T04:53:03.685790+0000","last_undegraded":"2021-02-10T04:53:03.685790+0000","last_fullsized":"2021-02-10T04:53:03.685790+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:49.391902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:49.391902+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1f","version":"1004'1093","reported_seq":"6422","reported_epoch":"1160","state":"active+clean","last_fresh":"2021-02-10T04:53:07.222051+0000","last_change":"2021-02-10T04:52:31.089952+0000","last_active":"2021-02-10T04:53:07.222051+0000","last_peered":"2021-02-10T04:53:07.222051+0000","last_clean":"2021-02-10T04:53:07.222051+0000","last_became_active":"2021-02-10T04:52:31.088634+0000","last_became_peered":"2021-02-10T04:52:31.088634+0000","last_unstale":"2021-02-10T04:53:07.222051+0000","last_undegraded":"2021-02-10T04:53:07.222051+0000","last_fullsized":"2021-02-10T04:53:07.222051+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39434697,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":279,"num_read_kb":311058,"num_write":186,"num_write_kb":66790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":53995913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"1049'39","reported_seq":"71","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:53:02.924777+0000","last_change":"2021-02-10T04:53:02.924733+0000","last_active":"2021-02-10T04:53:02.924777+0000","last_peered":"2021-02-10T04:53:02.924777+0000","last_clean":"2021-02-10T04:53:02.924777+0000","last_became_active":"2021-02-10T04:51:36.876346+0000","last_became_peered":"2021-02-10T04:51:36.876346+0000","last_unstale":"2021-02-10T04:53:02.924777+0000","last_undegraded":"2021-02-10T04:53:02.924777+0000","last_fullsized":"2021-02-10T04:53:02.924777+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'39","last_scrub_stamp":"2021-02-10T04:53:02.924664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:02.924664+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"1149","reported_epoch":"1160","state":"active+clean","last_fresh":"2021-02-10T04:53:07.222339+0000","last_change":"2021-02-10T04:52:36.083849+0000","last_active":"2021-02-10T04:53:07.222339+0000","last_peered":"2021-02-10T04:53:07.222339+0000","last_clean":"2021-02-10T04:53:07.222339+0000","last_became_active":"2021-02-10T04:25:36.336442+0000","last_became_peered":"2021-02-10T04:25:36.336442+0000","last_unstale":"2021-02-10T04:53:07.222339+0000","last_undegraded":"2021-02-10T04:53:07.222339+0000","last_fullsized":"2021-02-10T04:53:07.222339+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:52:36.083784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:36.083784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"374","reported_epoch":"1159","state":"active+undersized","last_fresh":"2021-02-10T04:53:03.686499+0000","last_change":"2021-02-10T04:52:32.084893+0000","last_active":"2021-02-10T04:53:03.686499+0000","last_peered":"2021-02-10T04:53:03.686499+0000","last_clean":"2021-02-10T04:52:30.061372+0000","last_became_active":"2021-02-10T04:52:32.084893+0000","last_became_peered":"2021-02-10T04:52:32.084893+0000","last_unstale":"2021-02-10T04:53:03.686499+0000","last_undegraded":"2021-02-10T04:53:03.686499+0000","last_fullsized":"2021-02-10T04:52:32.073304+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:08.758129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.758129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"1127","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.528002+0000","last_change":"2021-02-10T04:51:44.799512+0000","last_active":"2021-02-10T04:52:59.528002+0000","last_peered":"2021-02-10T04:52:59.528002+0000","last_clean":"2021-02-10T04:52:59.528002+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:52:59.528002+0000","last_undegraded":"2021-02-10T04:52:59.528002+0000","last_fullsized":"2021-02-10T04:52:59.528002+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:44.799429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.799429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1e","version":"1063'654","reported_seq":"4399","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.686469+0000","last_change":"2021-02-10T04:52:31.088116+0000","last_active":"2021-02-10T04:53:03.686469+0000","last_peered":"2021-02-10T04:53:03.686469+0000","last_clean":"2021-02-10T04:53:03.686469+0000","last_became_active":"2021-02-10T04:52:31.087662+0000","last_became_peered":"2021-02-10T04:52:31.087662+0000","last_unstale":"2021-02-10T04:53:03.686469+0000","last_undegraded":"2021-02-10T04:53:03.686469+0000","last_fullsized":"2021-02-10T04:53:03.686469+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:51:57.121838+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.121838+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24700631,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":171,"num_read_kb":182272,"num_write":112,"num_write_kb":33727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":26349024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"1066'44","reported_seq":"1172","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:06.077840+0000","last_change":"2021-02-10T04:53:06.077789+0000","last_active":"2021-02-10T04:53:06.077840+0000","last_peered":"2021-02-10T04:53:06.077840+0000","last_clean":"2021-02-10T04:53:06.077840+0000","last_became_active":"2021-02-10T04:50:59.563388+0000","last_became_peered":"2021-02-10T04:50:59.563388+0000","last_unstale":"2021-02-10T04:53:06.077840+0000","last_undegraded":"2021-02-10T04:53:06.077840+0000","last_fullsized":"2021-02-10T04:53:06.077840+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1066'44","last_scrub_stamp":"2021-02-10T04:53:06.077718+0000","last_deep_scrub":"1066'44","last_deep_scrub_stamp":"2021-02-10T04:47:18.806645+0000","last_clean_scrub_stamp":"2021-02-10T04:53:06.077718+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":67,"num_read_kb":49,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"1111","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.515000+0000","last_change":"2021-02-10T04:51:08.006463+0000","last_active":"2021-02-10T04:53:03.515000+0000","last_peered":"2021-02-10T04:53:03.515000+0000","last_clean":"2021-02-10T04:53:03.515000+0000","last_became_active":"2021-02-10T04:50:59.563657+0000","last_became_peered":"2021-02-10T04:50:59.563657+0000","last_unstale":"2021-02-10T04:53:03.515000+0000","last_undegraded":"2021-02-10T04:53:03.515000+0000","last_fullsized":"2021-02-10T04:53:03.515000+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_clean_scrub_stamp":"2021-02-10T04:51:02.455702+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"1162","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.526699+0000","last_change":"2021-02-10T04:52:33.741801+0000","last_active":"2021-02-10T04:52:59.526699+0000","last_peered":"2021-02-10T04:52:59.526699+0000","last_clean":"2021-02-10T04:52:59.526699+0000","last_became_active":"2021-02-10T04:51:36.860129+0000","last_became_peered":"2021-02-10T04:51:36.860129+0000","last_unstale":"2021-02-10T04:52:59.526699+0000","last_undegraded":"2021-02-10T04:52:59.526699+0000","last_fullsized":"2021-02-10T04:52:59.526699+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:33.741711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:46:58.666264+0000","last_clean_scrub_stamp":"2021-02-10T04:52:33.741711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"1123","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.515276+0000","last_change":"2021-02-10T04:52:25.030419+0000","last_active":"2021-02-10T04:53:03.515276+0000","last_peered":"2021-02-10T04:53:03.515276+0000","last_clean":"2021-02-10T04:53:03.515276+0000","last_became_active":"2021-02-10T04:50:59.539582+0000","last_became_peered":"2021-02-10T04:50:59.539582+0000","last_unstale":"2021-02-10T04:53:03.515276+0000","last_undegraded":"2021-02-10T04:53:03.515276+0000","last_fullsized":"2021-02-10T04:53:03.515276+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:25.030281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.030281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"1003'739","reported_seq":"3909","reported_epoch":"1160","state":"active+clean","last_fresh":"2021-02-10T04:53:07.221494+0000","last_change":"2021-02-10T04:53:01.929261+0000","last_active":"2021-02-10T04:53:07.221494+0000","last_peered":"2021-02-10T04:53:07.221494+0000","last_clean":"2021-02-10T04:53:07.221494+0000","last_became_active":"2021-02-10T04:51:40.511505+0000","last_became_peered":"2021-02-10T04:51:40.511505+0000","last_unstale":"2021-02-10T04:53:07.221494+0000","last_undegraded":"2021-02-10T04:53:07.221494+0000","last_fullsized":"2021-02-10T04:53:07.221494+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:53:01.929169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.929169+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22230846,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":18429397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"1112'47","reported_seq":"1236","reported_epoch":"1158","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:52:59.525966+0000","last_change":"2021-02-10T04:52:32.080987+0000","last_active":"2021-02-10T04:52:59.525966+0000","last_peered":"2021-02-10T04:52:59.525966+0000","last_clean":"2021-02-10T04:52:30.059333+0000","last_became_active":"2021-02-10T04:52:32.080987+0000","last_became_peered":"2021-02-10T04:52:32.080987+0000","last_unstale":"2021-02-10T04:52:59.525966+0000","last_undegraded":"2021-02-10T04:52:32.068699+0000","last_fullsized":"2021-02-10T04:52:32.068563+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1112'47","last_scrub_stamp":"2021-02-10T04:51:02.860835+0000","last_deep_scrub":"1112'47","last_deep_scrub_stamp":"2021-02-10T04:51:02.860835+0000","last_clean_scrub_stamp":"2021-02-10T04:51:02.860835+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":71,"num_read_kb":55,"num_write":38,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":10}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"1074","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.514403+0000","last_change":"2021-02-10T04:52:26.408468+0000","last_active":"2021-02-10T04:53:03.514403+0000","last_peered":"2021-02-10T04:53:03.514403+0000","last_clean":"2021-02-10T04:53:03.514403+0000","last_became_active":"2021-02-10T04:51:36.858446+0000","last_became_peered":"2021-02-10T04:51:36.858446+0000","last_unstale":"2021-02-10T04:53:03.514403+0000","last_undegraded":"2021-02-10T04:53:03.514403+0000","last_fullsized":"2021-02-10T04:53:03.514403+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:26.408326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.408326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"1160","reported_epoch":"1160","state":"active+clean","last_fresh":"2021-02-10T04:53:07.324650+0000","last_change":"2021-02-10T04:53:07.324650+0000","last_active":"2021-02-10T04:53:07.324650+0000","last_peered":"2021-02-10T04:53:07.324650+0000","last_clean":"2021-02-10T04:53:07.324650+0000","last_became_active":"2021-02-10T04:50:59.575340+0000","last_became_peered":"2021-02-10T04:50:59.575340+0000","last_unstale":"2021-02-10T04:53:07.324650+0000","last_undegraded":"2021-02-10T04:53:07.324650+0000","last_fullsized":"2021-02-10T04:53:07.324650+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.324564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.324564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"21","reported_epoch":"1159","state":"active+undersized","last_fresh":"2021-02-10T04:53:03.683990+0000","last_change":"2021-02-10T04:52:32.091240+0000","last_active":"2021-02-10T04:53:03.683990+0000","last_peered":"2021-02-10T04:53:03.683990+0000","last_clean":"2021-02-10T04:25:52.015519+0000","last_became_active":"2021-02-10T04:52:32.091240+0000","last_became_peered":"2021-02-10T04:52:32.091240+0000","last_unstale":"2021-02-10T04:53:03.683990+0000","last_undegraded":"2021-02-10T04:53:03.683990+0000","last_fullsized":"2021-02-10T04:52:32.077955+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:00.501127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:00.501127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.18","version":"259'74","reported_seq":"23","reported_epoch":"1160","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:07.221511+0000","last_change":"2021-02-10T04:52:32.083683+0000","last_active":"2021-02-10T04:53:07.221511+0000","last_peered":"2021-02-10T04:53:07.221511+0000","last_clean":"2021-02-10T04:31:15.165244+0000","last_became_active":"2021-02-10T04:52:32.083683+0000","last_became_peered":"2021-02-10T04:52:32.083683+0000","last_unstale":"2021-02-10T04:53:07.221511+0000","last_undegraded":"2021-02-10T04:52:32.067776+0000","last_fullsized":"2021-02-10T04:52:32.067560+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1010,"parent":"0.0","parent_split_bits":5,"last_scrub":"259'74","last_scrub_stamp":"2021-02-10T04:52:26.597668+0000","last_deep_scrub":"259'74","last_deep_scrub_stamp":"2021-02-10T04:46:49.745104+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.597668+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1218524,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":28228,"num_write":19,"num_write_kb":7022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"1061'20","reported_seq":"1183","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.529767+0000","last_change":"2021-02-10T04:52:31.771000+0000","last_active":"2021-02-10T04:52:59.529767+0000","last_peered":"2021-02-10T04:52:59.529767+0000","last_clean":"2021-02-10T04:52:59.529767+0000","last_became_active":"2021-02-10T04:50:59.569344+0000","last_became_peered":"2021-02-10T04:50:59.569344+0000","last_unstale":"2021-02-10T04:52:59.529767+0000","last_undegraded":"2021-02-10T04:52:59.529767+0000","last_fullsized":"2021-02-10T04:52:59.529767+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1061'20","last_scrub_stamp":"2021-02-10T04:52:31.770907+0000","last_deep_scrub":"1061'20","last_deep_scrub_stamp":"2021-02-10T04:46:55.646261+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.770907+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":17,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"1063","reported_epoch":"1160","state":"active+undersized","last_fresh":"2021-02-10T04:53:07.222687+0000","last_change":"2021-02-10T04:52:32.083801+0000","last_active":"2021-02-10T04:53:07.222687+0000","last_peered":"2021-02-10T04:53:07.222687+0000","last_clean":"2021-02-10T04:52:30.068110+0000","last_became_active":"2021-02-10T04:52:32.083801+0000","last_became_peered":"2021-02-10T04:52:32.083801+0000","last_unstale":"2021-02-10T04:53:07.222687+0000","last_undegraded":"2021-02-10T04:53:07.222687+0000","last_fullsized":"2021-02-10T04:52:32.078249+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:59.167911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:50:59.167911+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"1049","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.526156+0000","last_change":"2021-02-10T04:51:45.813755+0000","last_active":"2021-02-10T04:52:59.526156+0000","last_peered":"2021-02-10T04:52:59.526156+0000","last_clean":"2021-02-10T04:52:59.526156+0000","last_became_active":"2021-02-10T04:29:07.146525+0000","last_became_peered":"2021-02-10T04:29:07.146525+0000","last_unstale":"2021-02-10T04:52:59.526156+0000","last_undegraded":"2021-02-10T04:52:59.526156+0000","last_fullsized":"2021-02-10T04:52:59.526156+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:45.813676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.813676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"1144","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.510356+0000","last_change":"2021-02-10T04:51:09.433892+0000","last_active":"2021-02-10T04:53:03.510356+0000","last_peered":"2021-02-10T04:53:03.510356+0000","last_clean":"2021-02-10T04:53:03.510356+0000","last_became_active":"2021-02-10T04:50:59.539256+0000","last_became_peered":"2021-02-10T04:50:59.539256+0000","last_unstale":"2021-02-10T04:53:03.510356+0000","last_undegraded":"2021-02-10T04:53:03.510356+0000","last_fullsized":"2021-02-10T04:53:03.510356+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:09.433793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:09.433793+0000","last_clean_scrub_stamp":"2021-02-10T04:51:09.433793+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1b","version":"1003'1689","reported_seq":"4584","reported_epoch":"1158","state":"active+clean+remapped","last_fresh":"2021-02-10T04:52:59.513012+0000","last_change":"2021-02-10T04:52:33.087998+0000","last_active":"2021-02-10T04:52:59.513012+0000","last_peered":"2021-02-10T04:52:59.513012+0000","last_clean":"2021-02-10T04:52:59.513012+0000","last_became_active":"2021-02-10T04:52:33.087105+0000","last_became_peered":"2021-02-10T04:52:33.087105+0000","last_unstale":"2021-02-10T04:52:59.513012+0000","last_undegraded":"2021-02-10T04:52:59.513012+0000","last_fullsized":"2021-02-10T04:52:59.513012+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:46:54.814020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:54.814020+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":63260738,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":25,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":387,"num_read_kb":409791,"num_write":281,"num_write_kb":110559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":22449943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3,1],"avail_no_missing":["2","1","3"],"object_location_counts":[{"shards":"1,2,3","objects":25}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"1048'44","reported_seq":"1203","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.513085+0000","last_change":"2021-02-10T04:50:48.668186+0000","last_active":"2021-02-10T04:52:59.513085+0000","last_peered":"2021-02-10T04:52:59.513085+0000","last_clean":"2021-02-10T04:52:59.513085+0000","last_became_active":"2021-02-10T04:25:49.010481+0000","last_became_peered":"2021-02-10T04:25:49.010481+0000","last_unstale":"2021-02-10T04:52:59.513085+0000","last_undegraded":"2021-02-10T04:52:59.513085+0000","last_fullsized":"2021-02-10T04:52:59.513085+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'44","last_scrub_stamp":"2021-02-10T04:50:48.668079+0000","last_deep_scrub":"78'32","last_deep_scrub_stamp":"2021-02-10T04:27:09.170428+0000","last_clean_scrub_stamp":"2021-02-10T04:50:48.668079+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":65,"num_read_kb":46,"num_write":34,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"1135","reported_epoch":"1159","state":"active+undersized","last_fresh":"2021-02-10T04:53:03.515557+0000","last_change":"2021-02-10T04:52:32.087531+0000","last_active":"2021-02-10T04:53:03.515557+0000","last_peered":"2021-02-10T04:53:03.515557+0000","last_clean":"2021-02-10T04:52:30.066743+0000","last_became_active":"2021-02-10T04:52:32.087531+0000","last_became_peered":"2021-02-10T04:52:32.087531+0000","last_unstale":"2021-02-10T04:53:03.515557+0000","last_undegraded":"2021-02-10T04:53:03.515557+0000","last_fullsized":"2021-02-10T04:52:32.076804+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:33.543559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:46:33.543559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"819","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:04.354581+0000","last_change":"2021-02-10T04:53:04.354508+0000","last_active":"2021-02-10T04:53:04.354581+0000","last_peered":"2021-02-10T04:53:04.354581+0000","last_clean":"2021-02-10T04:53:04.354581+0000","last_became_active":"2021-02-10T04:33:05.169159+0000","last_became_peered":"2021-02-10T04:33:05.169159+0000","last_unstale":"2021-02-10T04:53:04.354581+0000","last_undegraded":"2021-02-10T04:53:04.354581+0000","last_fullsized":"2021-02-10T04:53:04.354581+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:04.354409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:04.354409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"1213","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.525811+0000","last_change":"2021-02-10T04:52:36.777554+0000","last_active":"2021-02-10T04:52:59.525811+0000","last_peered":"2021-02-10T04:52:59.525811+0000","last_clean":"2021-02-10T04:52:59.525811+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:52:59.525811+0000","last_undegraded":"2021-02-10T04:52:59.525811+0000","last_fullsized":"2021-02-10T04:52:59.525811+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:36.777485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:52:36.777485+0000","last_clean_scrub_stamp":"2021-02-10T04:52:36.777485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1a","version":"1001'1528","reported_seq":"5770","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.524928+0000","last_change":"2021-02-10T04:52:31.086406+0000","last_active":"2021-02-10T04:52:59.524928+0000","last_peered":"2021-02-10T04:52:59.524928+0000","last_clean":"2021-02-10T04:52:59.524928+0000","last_became_active":"2021-02-10T04:52:31.085959+0000","last_became_peered":"2021-02-10T04:52:31.085959+0000","last_unstale":"2021-02-10T04:52:59.524928+0000","last_undegraded":"2021-02-10T04:52:59.524928+0000","last_fullsized":"2021-02-10T04:52:59.524928+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46944136,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":358,"num_read_kb":351935,"num_write":286,"num_write_kb":111355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":21311893,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"1049'68","reported_seq":"1314","reported_epoch":"1159","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:03.513050+0000","last_change":"2021-02-10T04:52:32.091082+0000","last_active":"2021-02-10T04:53:03.513050+0000","last_peered":"2021-02-10T04:53:03.513050+0000","last_clean":"2021-02-10T04:52:30.063018+0000","last_became_active":"2021-02-10T04:52:32.091082+0000","last_became_peered":"2021-02-10T04:52:32.091082+0000","last_unstale":"2021-02-10T04:53:03.513050+0000","last_undegraded":"2021-02-10T04:52:32.072802+0000","last_fullsized":"2021-02-10T04:52:32.072695+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'68","last_scrub_stamp":"2021-02-10T04:47:09.608881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:47:09.608881+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":124,"num_read_kb":101,"num_write":69,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":16}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"24","reported_epoch":"1158","state":"active+undersized","last_fresh":"2021-02-10T04:52:59.528958+0000","last_change":"2021-02-10T04:52:32.084553+0000","last_active":"2021-02-10T04:52:59.528958+0000","last_peered":"2021-02-10T04:52:59.528958+0000","last_clean":"2021-02-10T04:25:33.303710+0000","last_became_active":"2021-02-10T04:52:32.084553+0000","last_became_peered":"2021-02-10T04:52:32.084553+0000","last_unstale":"2021-02-10T04:52:59.528958+0000","last_undegraded":"2021-02-10T04:52:59.528958+0000","last_fullsized":"2021-02-10T04:52:32.070030+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:17.919745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:17.919745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"445","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:07.092105+0000","last_change":"2021-02-10T04:53:07.092032+0000","last_active":"2021-02-10T04:53:07.092105+0000","last_peered":"2021-02-10T04:53:07.092105+0000","last_clean":"2021-02-10T04:53:07.092105+0000","last_became_active":"2021-02-10T04:50:59.572077+0000","last_became_peered":"2021-02-10T04:50:59.572077+0000","last_unstale":"2021-02-10T04:53:07.092105+0000","last_undegraded":"2021-02-10T04:53:07.092105+0000","last_fullsized":"2021-02-10T04:53:07.092105+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.091949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:19.755718+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.091949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"1122","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.514787+0000","last_change":"2021-02-10T04:51:59.838856+0000","last_active":"2021-02-10T04:53:03.514787+0000","last_peered":"2021-02-10T04:53:03.514787+0000","last_clean":"2021-02-10T04:53:03.514787+0000","last_became_active":"2021-02-10T04:50:59.563156+0000","last_became_peered":"2021-02-10T04:50:59.563156+0000","last_unstale":"2021-02-10T04:53:03.514787+0000","last_undegraded":"2021-02-10T04:53:03.514787+0000","last_fullsized":"2021-02-10T04:53:03.514787+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.838790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.838790+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":"855","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.513825+0000","last_change":"2021-02-10T04:52:30.405170+0000","last_active":"2021-02-10T04:53:03.513825+0000","last_peered":"2021-02-10T04:53:03.513825+0000","last_clean":"2021-02-10T04:53:03.513825+0000","last_became_active":"2021-02-10T04:51:36.852058+0000","last_became_peered":"2021-02-10T04:51:36.852058+0000","last_unstale":"2021-02-10T04:53:03.513825+0000","last_undegraded":"2021-02-10T04:53:03.513825+0000","last_fullsized":"2021-02-10T04:53:03.513825+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.405106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:20.635536+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.405106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.20","version":"0'0","reported_seq":"1119","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.509796+0000","last_change":"2021-02-10T04:51:56.449184+0000","last_active":"2021-02-10T04:53:03.509796+0000","last_peered":"2021-02-10T04:53:03.509796+0000","last_clean":"2021-02-10T04:53:03.509796+0000","last_became_active":"2021-02-10T04:29:07.148937+0000","last_became_peered":"2021-02-10T04:29:07.148937+0000","last_unstale":"2021-02-10T04:53:03.509796+0000","last_undegraded":"2021-02-10T04:53:03.509796+0000","last_fullsized":"2021-02-10T04:53:03.509796+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:56.449085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:41.375276+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.449085+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":"1083","reported_epoch":"1160","state":"active+clean","last_fresh":"2021-02-10T04:53:07.222520+0000","last_change":"2021-02-10T04:51:36.870420+0000","last_active":"2021-02-10T04:53:07.222520+0000","last_peered":"2021-02-10T04:53:07.222520+0000","last_clean":"2021-02-10T04:53:07.222520+0000","last_became_active":"2021-02-10T04:51:36.869785+0000","last_became_peered":"2021-02-10T04:51:36.869785+0000","last_unstale":"2021-02-10T04:53:07.222520+0000","last_undegraded":"2021-02-10T04:53:07.222520+0000","last_fullsized":"2021-02-10T04:53:07.222520+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:04.210002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:50:04.210002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":"56","reported_epoch":"1158","state":"active+undersized","last_fresh":"2021-02-10T04:52:59.513167+0000","last_change":"2021-02-10T04:52:32.094666+0000","last_active":"2021-02-10T04:52:59.513167+0000","last_peered":"2021-02-10T04:52:59.513167+0000","last_clean":"2021-02-10T04:52:30.064564+0000","last_became_active":"2021-02-10T04:52:32.094666+0000","last_became_peered":"2021-02-10T04:52:32.094666+0000","last_unstale":"2021-02-10T04:52:59.513167+0000","last_undegraded":"2021-02-10T04:52:59.513167+0000","last_fullsized":"2021-02-10T04:52:32.084052+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:47.481686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:47.481686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":"49","reported_epoch":"1159","state":"active+undersized","last_fresh":"2021-02-10T04:53:03.685936+0000","last_change":"2021-02-10T04:52:32.096027+0000","last_active":"2021-02-10T04:53:03.685936+0000","last_peered":"2021-02-10T04:53:03.685936+0000","last_clean":"2021-02-10T04:52:30.061012+0000","last_became_active":"2021-02-10T04:52:32.096027+0000","last_became_peered":"2021-02-10T04:52:32.096027+0000","last_unstale":"2021-02-10T04:53:03.685936+0000","last_undegraded":"2021-02-10T04:53:03.685936+0000","last_fullsized":"2021-02-10T04:52:32.076629+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.706372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.706372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":"1170","reported_epoch":"1160","state":"active+clean","last_fresh":"2021-02-10T04:53:07.222031+0000","last_change":"2021-02-10T04:51:57.287452+0000","last_active":"2021-02-10T04:53:07.222031+0000","last_peered":"2021-02-10T04:53:07.222031+0000","last_clean":"2021-02-10T04:53:07.222031+0000","last_became_active":"2021-02-10T04:50:59.566827+0000","last_became_peered":"2021-02-10T04:50:59.566827+0000","last_unstale":"2021-02-10T04:53:07.222031+0000","last_undegraded":"2021-02-10T04:53:07.222031+0000","last_fullsized":"2021-02-10T04:53:07.222031+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.287363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.287363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":"135","reported_epoch":"1160","state":"active+undersized","last_fresh":"2021-02-10T04:53:07.223880+0000","last_change":"2021-02-10T04:52:32.082221+0000","last_active":"2021-02-10T04:53:07.223880+0000","last_peered":"2021-02-10T04:53:07.223880+0000","last_clean":"2021-02-10T04:52:30.068335+0000","last_became_active":"2021-02-10T04:52:32.082221+0000","last_became_peered":"2021-02-10T04:52:32.082221+0000","last_unstale":"2021-02-10T04:53:07.223880+0000","last_undegraded":"2021-02-10T04:53:07.223880+0000","last_fullsized":"2021-02-10T04:52:32.077664+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:10.308282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:10.308282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":"1053","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.515085+0000","last_change":"2021-02-10T04:51:07.989063+0000","last_active":"2021-02-10T04:53:03.515085+0000","last_peered":"2021-02-10T04:53:03.515085+0000","last_clean":"2021-02-10T04:53:03.515085+0000","last_became_active":"2021-02-10T04:50:59.561442+0000","last_became_peered":"2021-02-10T04:50:59.561442+0000","last_unstale":"2021-02-10T04:53:03.515085+0000","last_undegraded":"2021-02-10T04:53:03.515085+0000","last_fullsized":"2021-02-10T04:53:03.515085+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:51.620102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:44:51.620102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.21","version":"293'77","reported_seq":"730","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.513366+0000","last_change":"2021-02-10T04:52:28.062923+0000","last_active":"2021-02-10T04:53:03.513366+0000","last_peered":"2021-02-10T04:53:03.513366+0000","last_clean":"2021-02-10T04:53:03.513366+0000","last_became_active":"2021-02-10T04:52:28.062463+0000","last_became_peered":"2021-02-10T04:52:28.062463+0000","last_unstale":"2021-02-10T04:53:03.513366+0000","last_undegraded":"2021-02-10T04:53:03.513366+0000","last_fullsized":"2021-02-10T04:53:03.513366+0000","mapping_epoch":1143,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1144,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1886586,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":15163,"num_write":9,"num_write_kb":3316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":"55","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.513422+0000","last_change":"2021-02-10T04:52:32.124955+0000","last_active":"2021-02-10T04:52:59.513422+0000","last_peered":"2021-02-10T04:52:59.513422+0000","last_clean":"2021-02-10T04:52:59.513422+0000","last_became_active":"2021-02-10T04:51:36.867061+0000","last_became_peered":"2021-02-10T04:51:36.867061+0000","last_unstale":"2021-02-10T04:52:59.513422+0000","last_undegraded":"2021-02-10T04:52:59.513422+0000","last_fullsized":"2021-02-10T04:52:59.513422+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:32.124901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:08.271977+0000","last_clean_scrub_stamp":"2021-02-10T04:52:32.124901+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":"1099","reported_epoch":"1160","state":"active+clean","last_fresh":"2021-02-10T04:53:07.222258+0000","last_change":"2021-02-10T04:51:07.182775+0000","last_active":"2021-02-10T04:53:07.222258+0000","last_peered":"2021-02-10T04:53:07.222258+0000","last_clean":"2021-02-10T04:53:07.222258+0000","last_became_active":"2021-02-10T04:28:23.126368+0000","last_became_peered":"2021-02-10T04:28:23.126368+0000","last_unstale":"2021-02-10T04:53:07.222258+0000","last_undegraded":"2021-02-10T04:53:07.222258+0000","last_fullsized":"2021-02-10T04:53:07.222258+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.182678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.182678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.20","version":"1000'1071","reported_seq":"4503","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.915465+0000","last_change":"2021-02-10T04:53:03.915465+0000","last_active":"2021-02-10T04:53:03.915465+0000","last_peered":"2021-02-10T04:53:03.915465+0000","last_clean":"2021-02-10T04:53:03.915465+0000","last_became_active":"2021-02-10T04:52:31.088690+0000","last_became_peered":"2021-02-10T04:52:31.088690+0000","last_unstale":"2021-02-10T04:53:03.915465+0000","last_undegraded":"2021-02-10T04:53:03.915465+0000","last_fullsized":"2021-02-10T04:53:03.915465+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:03.915403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.915403+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16398893,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":"55","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.513503+0000","last_change":"2021-02-10T04:52:28.092351+0000","last_active":"2021-02-10T04:52:59.513503+0000","last_peered":"2021-02-10T04:52:59.513503+0000","last_clean":"2021-02-10T04:52:59.513503+0000","last_became_active":"2021-02-10T04:51:36.855101+0000","last_became_peered":"2021-02-10T04:51:36.855101+0000","last_unstale":"2021-02-10T04:52:59.513503+0000","last_undegraded":"2021-02-10T04:52:59.513503+0000","last_fullsized":"2021-02-10T04:52:59.513503+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:28.092288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:07.767250+0000","last_clean_scrub_stamp":"2021-02-10T04:52:28.092288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.25","version":"53'2","reported_seq":"1109","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.526908+0000","last_change":"2021-02-10T04:50:12.772167+0000","last_active":"2021-02-10T04:52:59.526908+0000","last_peered":"2021-02-10T04:52:59.526908+0000","last_clean":"2021-02-10T04:52:59.526908+0000","last_became_active":"2021-02-10T04:28:23.467770+0000","last_became_peered":"2021-02-10T04:28:23.467770+0000","last_unstale":"2021-02-10T04:52:59.526908+0000","last_undegraded":"2021-02-10T04:52:59.526908+0000","last_fullsized":"2021-02-10T04:52:59.526908+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:50:12.772086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:12.772086+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.23","version":"1002'1657","reported_seq":"5031","reported_epoch":"1160","state":"active+clean","last_fresh":"2021-02-10T04:53:07.221671+0000","last_change":"2021-02-10T04:52:31.093874+0000","last_active":"2021-02-10T04:53:07.221671+0000","last_peered":"2021-02-10T04:53:07.221671+0000","last_clean":"2021-02-10T04:53:07.221671+0000","last_became_active":"2021-02-10T04:52:31.092996+0000","last_became_peered":"2021-02-10T04:52:31.092996+0000","last_unstale":"2021-02-10T04:53:07.221671+0000","last_undegraded":"2021-02-10T04:53:07.221671+0000","last_fullsized":"2021-02-10T04:53:07.221671+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21396533,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":212,"num_read_kb":218837,"num_write":159,"num_write_kb":59399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":20479176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":"1201","reported_epoch":"1160","state":"active+clean","last_fresh":"2021-02-10T04:53:07.221651+0000","last_change":"2021-02-10T04:50:59.567042+0000","last_active":"2021-02-10T04:53:07.221651+0000","last_peered":"2021-02-10T04:53:07.221651+0000","last_clean":"2021-02-10T04:53:07.221651+0000","last_became_active":"2021-02-10T04:50:59.566548+0000","last_became_peered":"2021-02-10T04:50:59.566548+0000","last_unstale":"2021-02-10T04:53:07.221651+0000","last_undegraded":"2021-02-10T04:53:07.221651+0000","last_fullsized":"2021-02-10T04:53:07.221651+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:44:27.926662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:44:27.926662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.26","version":"53'1","reported_seq":"554","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.934004+0000","last_change":"2021-02-10T04:53:03.934004+0000","last_active":"2021-02-10T04:53:03.934004+0000","last_peered":"2021-02-10T04:53:03.934004+0000","last_clean":"2021-02-10T04:53:03.934004+0000","last_became_active":"2021-02-10T04:37:37.231081+0000","last_became_peered":"2021-02-10T04:37:37.231081+0000","last_unstale":"2021-02-10T04:53:03.934004+0000","last_undegraded":"2021-02-10T04:53:03.934004+0000","last_fullsized":"2021-02-10T04:53:03.934004+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:53:03.933910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.933910+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.22","version":"998'1009","reported_seq":"4521","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.525016+0000","last_change":"2021-02-10T04:52:31.086563+0000","last_active":"2021-02-10T04:52:59.525016+0000","last_peered":"2021-02-10T04:52:59.525016+0000","last_clean":"2021-02-10T04:52:59.525016+0000","last_became_active":"2021-02-10T04:52:31.085288+0000","last_became_peered":"2021-02-10T04:52:31.085288+0000","last_unstale":"2021-02-10T04:52:59.525016+0000","last_undegraded":"2021-02-10T04:52:59.525016+0000","last_fullsized":"2021-02-10T04:52:59.525016+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12355207,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":120,"num_read_kb":124702,"num_write":77,"num_write_kb":25052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4145029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":"1083","reported_epoch":"1160","state":"active+clean","last_fresh":"2021-02-10T04:53:07.221640+0000","last_change":"2021-02-10T04:47:22.953445+0000","last_active":"2021-02-10T04:53:07.221640+0000","last_peered":"2021-02-10T04:53:07.221640+0000","last_clean":"2021-02-10T04:53:07.221640+0000","last_became_active":"2021-02-10T04:45:49.552026+0000","last_became_peered":"2021-02-10T04:45:49.552026+0000","last_unstale":"2021-02-10T04:53:07.221640+0000","last_undegraded":"2021-02-10T04:53:07.221640+0000","last_fullsized":"2021-02-10T04:53:07.221640+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:22.953356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.953356+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":"1125","reported_epoch":"1158","state":"active+clean","last_fresh":"2021-02-10T04:52:59.528804+0000","last_change":"2021-02-10T04:52:41.758800+0000","last_active":"2021-02-10T04:52:59.528804+0000","last_peered":"2021-02-10T04:52:59.528804+0000","last_clean":"2021-02-10T04:52:59.528804+0000","last_became_active":"2021-02-10T04:29:07.145907+0000","last_became_peered":"2021-02-10T04:29:07.145907+0000","last_unstale":"2021-02-10T04:52:59.528804+0000","last_undegraded":"2021-02-10T04:52:59.528804+0000","last_fullsized":"2021-02-10T04:52:59.528804+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:41.758684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.758684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":"1082","reported_epoch":"1159","state":"active+clean","last_fresh":"2021-02-10T04:53:03.511979+0000","last_change":"2021-02-10T04:51:07.984986+0000","last_active":"2021-02-10T04:53:03.511979+0000","last_peered":"2021-02-10T04:53:03.511979+0000","last_clean":"2021-02-10T04:53:03.511979+0000","last_became_active":"2021-02-10T04:50:59.569162+0000","last_became_peered":"2021-02-10T04:50:59.569162+0000","last_unstale":"2021-02-10T04:53:03.511979+0000","last_undegraded":"2021-02-10T04:53:03.511979+0000","last_fullsized":"2021-02-10T04:53:03.511979+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:12.555666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:46:12.555666+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":"1114","reported_epoch":"1160","state":"active+clean","last_fresh":"2021-02-10T04:53:07.221949+0000","last_change":"2021-02-10T04:52:24.072005+0000","last_active":"2021-02-10T04:53:07.221949+0000","last_peered":"2021-02-10T04:53:07.221949+0000","last_clean":"2021-02-10T04:53:07.221949+0000","last_became_active":"2021-02-10T04:50:59.527945+0000","last_became_peered":"2021-02-10T04:50:59.527945+0000","last_unstale":"2021-02-10T04:53:07.221949+0000","last_undegraded":"2021-02-10T04:53:07.221949+0000","last_fullsized":"2021-02-10T04:53:07.221949+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:24.071915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_clean_scrub_stamp":"2021-02-10T04:52:24.071915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":948623470,"num_objects":390,"num_object_clones":337,"num_object_copies":1170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":167,"num_objects_misplaced":25,"num_objects_unfound":0,"num_objects_dirty":390,"num_whiteouts":12,"num_read":4974,"num_read_kb":5152472,"num_write":3617,"num_write_kb":1396394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":805,"num_bytes_recovered":394291573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2033135616,"data_stored":2616221515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34716,"ondisk_log_size":34716,"up":94,"acting":95,"num_store_stats":8},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":1968,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":18,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1069,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":5835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":111,"acting":111,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":29,"acting":29,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":510,"num_read_kb":505,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":8,"acting":8,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":2291,"num_read_kb":1764,"num_write":1306,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":10504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1415,"ondisk_log_size":1415,"up":87,"acting":87,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31,"ondisk_log_size":31,"up":84,"acting":84,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":114,"acting":114,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365132,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":189580,"kb_used_data":149124,"kb_used_omap":0,"kb_used_meta":40448,"kb_avail":93555572,"statfs":{"total":95995035648,"available":95800905728,"internally_reserved":0,"allocated":152702976,"data_stored":202795234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41418752},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:27 2021","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":1.2370000000000001,"15min":1.284},"min":{"1min":0.50800000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1220000000000001,"5min":34.299999999999997,"15min":67.277000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.5109999999999999,"15min":1.548},"min":{"1min":0.46300000000000002,"5min":0.438,"15min":0.438},"max":{"1min":2.8690000000000002,"5min":34.652999999999999,"15min":66.933999999999997},"last":0.71399999999999997}]},{"osd":1,"last update":"Wed Feb 10 04:52:31 2021","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.92000000000000004,"15min":0.995},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":2.254,"5min":2.3420000000000001,"15min":10.140000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.0069999999999999,"15min":1.018},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.536,"5min":2.5409999999999999,"15min":7.1520000000000001},"last":0.78200000000000003}]},{"osd":2,"last update":"Wed Feb 10 04:52:31 2021","interfaces":[{"interface":"back","average":{"1min":1.7869999999999999,"5min":1.2350000000000001,"15min":1.367},"min":{"1min":0.627,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":11.468,"5min":11.468,"15min":66.534000000000006},"last":0.64700000000000002},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.2609999999999999,"15min":1.5589999999999999},"min":{"1min":0.69499999999999995,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.8779999999999999,"5min":6.7930000000000001,"15min":66.781999999999996},"last":0.59499999999999997}]},{"osd":3,"last update":"Wed Feb 10 04:52:58 2021","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.3089999999999999,"15min":1.3480000000000001},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":2.2749999999999999,"5min":4.5890000000000004,"15min":4.5890000000000004},"last":0.71199999999999997},{"interface":"front","average":{"1min":1.042,"5min":1.3959999999999999,"15min":1.4550000000000001},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.698,"5min":4.6609999999999996,"15min":4.6609999999999996},"last":0.95399999999999996}]},{"osd":4,"last update":"Wed Feb 10 04:52:31 2021","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.474,"15min":1.516},"min":{"1min":0.78900000000000003,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":2.7970000000000002,"5min":7.0640000000000001,"15min":10.781000000000001},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.444,"5min":1.3700000000000001,"15min":1.413},"min":{"1min":0.76600000000000001,"5min":0.69099999999999995,"15min":0.65100000000000002},"max":{"1min":3.4049999999999998,"5min":4.6269999999999998,"15min":10.326000000000001},"last":0.98099999999999998}]},{"osd":5,"last update":"Wed Feb 10 04:52:31 2021","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.488,"15min":1.653},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.66700000000000004},"max":{"1min":1.9399999999999999,"5min":4.0570000000000004,"15min":35.283000000000001},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.6819999999999999,"5min":1.6739999999999999,"15min":1.9590000000000001},"min":{"1min":0.97099999999999997,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":3.8769999999999998,"5min":6.6760000000000002,"15min":35.353999999999999},"last":1.319}]},{"osd":6,"last update":"Wed Feb 10 04:52:31 2021","interfaces":[{"interface":"back","average":{"1min":1.581,"5min":1.7290000000000001,"15min":1.901},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.89100000000000001},"max":{"1min":4.0339999999999998,"5min":6.9409999999999998,"15min":16.050000000000001},"last":1.036},{"interface":"front","average":{"1min":1.698,"5min":1.6910000000000001,"15min":1.804},"min":{"1min":0.98999999999999999,"5min":0.876,"15min":0.876},"max":{"1min":3.6480000000000001,"5min":4.7439999999999998,"15min":15.960000000000001},"last":1.1080000000000001}]}]},{"osd":6,"up_from":43,"seq":184683594065,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":401316,"kb_used_data":341084,"kb_used_omap":0,"kb_used_meta":60224,"kb_avail":93343836,"statfs":{"total":95995035648,"available":95584088064,"internally_reserved":0,"allocated":349270016,"data_stored":442637948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":61669376},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:52:12 2021","interfaces":[{"interface":"back","average":{"1min":4.016,"5min":2.7400000000000002,"15min":1.595},"min":{"1min":0.52000000000000002,"5min":0.48099999999999998,"15min":0.44700000000000001},"max":{"1min":61.768000000000001,"5min":137.60599999999999,"15min":137.60599999999999},"last":0.97999999999999998},{"interface":"front","average":{"1min":4.0270000000000001,"5min":2.694,"15min":1.7989999999999999},"min":{"1min":0.504,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":62.683,"5min":137.66,"15min":137.66},"last":0.82099999999999995}]},{"osd":1,"last update":"Wed Feb 10 04:52:12 2021","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":1.0600000000000001,"15min":1.0609999999999999},"min":{"1min":0.53800000000000003,"5min":0.499,"15min":0.499},"max":{"1min":1.335,"5min":5.2160000000000002,"15min":12.978999999999999},"last":0.622},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.1279999999999999,"15min":1.1339999999999999},"min":{"1min":0.54700000000000004,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.611,"5min":3.6219999999999999,"15min":6.0960000000000001},"last":1.0800000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:52:12 2021","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.266,"15min":1.2649999999999999},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.54100000000000004},"max":{"1min":3.6629999999999998,"5min":5.0990000000000002,"15min":7.8140000000000001},"last":163.81700000000001},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.286,"15min":1.2649999999999999},"min":{"1min":0.77100000000000002,"5min":0.66200000000000003,"15min":0.59899999999999998},"max":{"1min":1.774,"5min":5.3140000000000001,"15min":5.3140000000000001},"last":164.56800000000001}]},{"osd":3,"last update":"Wed Feb 10 04:53:00 2021","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.1890000000000001,"15min":1.2090000000000001},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":1.23,"5min":1.1459999999999999,"15min":1.1319999999999999},"min":{"1min":0.76000000000000001,"5min":0.749,"15min":0.749},"max":{"1min":2.363,"5min":2.363,"15min":2.363},"last":1.1850000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:52:12 2021","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.415,"15min":1.462},"min":{"1min":0.68300000000000005,"5min":0.64600000000000002,"15min":0.60599999999999998},"max":{"1min":2.661,"5min":7.1459999999999999,"15min":7.5389999999999997},"last":1.252},{"interface":"front","average":{"1min":0.97599999999999998,"5min":1.0940000000000001,"15min":1.1379999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.4299999999999999,"5min":11.997999999999999,"15min":11.997999999999999},"last":0.72099999999999997}]},{"osd":5,"last update":"Wed Feb 10 04:52:12 2021","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.244,"15min":1.6859999999999999},"min":{"1min":0.81999999999999995,"5min":0.61699999999999999,"15min":0.56899999999999995},"max":{"1min":2.0099999999999998,"5min":4.2249999999999996,"15min":81.540000000000006},"last":0.873},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.5760000000000001,"15min":1.946},"min":{"1min":0.96699999999999997,"5min":0.81000000000000005,"15min":0.78000000000000003},"max":{"1min":1.9870000000000001,"5min":5.5069999999999997,"15min":81.492999999999995},"last":1.163}]},{"osd":7,"last update":"Wed Feb 10 04:52:39 2021","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.2729999999999999,"15min":1.3660000000000001},"min":{"1min":0.79200000000000004,"5min":0.747,"15min":0.747},"max":{"1min":1.5820000000000001,"5min":4.0919999999999996,"15min":5.1820000000000004},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.6100000000000001,"15min":1.675},"min":{"1min":0.97699999999999998,"5min":0.95999999999999996,"15min":0.91000000000000003},"max":{"1min":2.448,"5min":5.4109999999999996,"15min":13.315},"last":1.2330000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575202,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":284684,"kb_used_data":223556,"kb_used_omap":0,"kb_used_meta":61120,"kb_avail":93460468,"statfs":{"total":95995035648,"available":95703519232,"internally_reserved":0,"allocated":228921344,"data_stored":288773076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62586880},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:48 2021","interfaces":[{"interface":"back","average":{"1min":1.696,"5min":2.6320000000000001,"15min":1.921},"min":{"1min":0.44700000000000001,"5min":0.437,"15min":0.437},"max":{"1min":14.053000000000001,"5min":115.622,"15min":115.622},"last":0.69599999999999995},{"interface":"front","average":{"1min":1.728,"5min":2.585,"15min":1.948},"min":{"1min":0.49199999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":14.109,"5min":115.73099999999999,"15min":115.73099999999999},"last":0.72899999999999998}]},{"osd":2,"last update":"Wed Feb 10 04:53:05 2021","interfaces":[{"interface":"back","average":{"1min":3.8109999999999999,"5min":3.0350000000000001,"15min":1.956},"min":{"1min":0.495,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":53.884999999999998,"5min":115.59699999999999,"15min":115.59699999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":3.8220000000000001,"5min":3.0760000000000001,"15min":1.992},"min":{"1min":0.49299999999999999,"5min":0.49099999999999999,"15min":0.46800000000000003},"max":{"1min":52.862000000000002,"5min":115.753,"15min":115.753},"last":0.64200000000000002}]},{"osd":3,"last update":"Wed Feb 10 04:53:00 2021","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.92500000000000004,"15min":0.95399999999999996},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.2250000000000001,"5min":3.077,"15min":3.077},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.92500000000000004,"5min":1.026,"15min":1.0429999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.70599999999999996}]},{"osd":4,"last update":"Wed Feb 10 04:52:51 2021","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.081,"15min":1.196},"min":{"1min":0.60499999999999998,"5min":0.56699999999999995,"15min":0.55200000000000005},"max":{"1min":1.4710000000000001,"5min":2.9670000000000001,"15min":4.899},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.165,"5min":1.1559999999999999,"15min":1.3500000000000001},"min":{"1min":0.78200000000000003,"5min":0.73599999999999999,"15min":0.73199999999999998},"max":{"1min":2.5270000000000001,"5min":2.5270000000000001,"15min":29.547999999999998},"last":0.83999999999999997}]},{"osd":5,"last update":"Wed Feb 10 04:52:55 2021","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.1679999999999999,"15min":1.228},"min":{"1min":0.64500000000000002,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.895,"5min":2.7959999999999998,"15min":7.8049999999999997},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.2889999999999999,"15min":1.5389999999999999},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.571,"5min":2.8199999999999998,"15min":28.678000000000001},"last":0.75800000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:53:05 2021","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.218,"15min":1.5129999999999999},"min":{"1min":0.75700000000000001,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.7010000000000001,"5min":2.1659999999999999,"15min":29.686},"last":1.202},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.2390000000000001,"15min":1.4419999999999999},"min":{"1min":0.86799999999999999,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.75,"5min":2.7789999999999999,"15min":28.475000000000001},"last":0.99299999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:52:21 2021","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.375,"15min":1.5509999999999999},"min":{"1min":0.90500000000000003,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.4570000000000001,"5min":4.9770000000000003,"15min":28.573},"last":0.82499999999999996},{"interface":"front","average":{"1min":1.389,"5min":1.371,"15min":1.5840000000000001},"min":{"1min":1.159,"5min":0.87,"15min":0.87},"max":{"1min":1.6779999999999999,"5min":2.298,"15min":29.408999999999999},"last":1.0149999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738720,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":399164,"kb_used_data":337140,"kb_used_omap":0,"kb_used_meta":62016,"kb_avail":93345988,"statfs":{"total":95995035648,"available":95586291712,"internally_reserved":0,"allocated":345231360,"data_stored":470653300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":63504384},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 04:51:32 2021","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.81100000000000005,"15min":1.0089999999999999},"min":{"1min":0.47099999999999997,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.0329999999999999,"5min":2.2930000000000001,"15min":28.946999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.874,"5min":0.92300000000000004,"15min":1.0149999999999999},"min":{"1min":0.68000000000000005,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.1819999999999999,"5min":2.0670000000000002,"15min":3.7050000000000001},"last":0.89200000000000002}]},{"osd":2,"last update":"Wed Feb 10 04:52:09 2021","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":2.2389999999999999,"15min":1.613},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0700000000000001,"5min":97.983999999999995,"15min":97.983999999999995},"last":1.056},{"interface":"front","average":{"1min":0.749,"5min":2.153,"15min":1.5169999999999999},"min":{"1min":0.51900000000000002,"5min":0.51300000000000001,"15min":0.47699999999999998},"max":{"1min":1.1319999999999999,"5min":97.844999999999999,"15min":97.844999999999999},"last":1.296}]},{"osd":3,"last update":"Wed Feb 10 04:51:59 2021","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.87,"15min":0.87},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":1.002},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.254,"5min":1.254,"15min":1.254},"last":0.871}]},{"osd":4,"last update":"Wed Feb 10 04:51:32 2021","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":1.119,"15min":1.306},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.3200000000000001,"5min":2.206,"15min":5.1680000000000001},"last":1.1120000000000001},{"interface":"front","average":{"1min":0.96499999999999997,"5min":1.179,"15min":1.252},"min":{"1min":0.64200000000000002,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.607,"5min":3.359,"15min":14.672000000000001},"last":1.0920000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:51:54 2021","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.325,"15min":1.512},"min":{"1min":0.84599999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.714,"5min":3.944,"15min":25.047000000000001},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.413,"15min":1.7090000000000001},"min":{"1min":0.79400000000000004,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":2.1669999999999998,"5min":3.7850000000000001,"15min":28.827999999999999},"last":1.226}]},{"osd":6,"last update":"Wed Feb 10 04:52:23 2021","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.845,"15min":1.8280000000000001},"min":{"1min":1.0109999999999999,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.7889999999999999,"5min":45.012,"15min":45.012},"last":1.587},{"interface":"front","average":{"1min":1.327,"5min":1.891,"15min":1.95},"min":{"1min":0.88800000000000001,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.0979999999999999,"5min":45.225000000000001,"15min":45.225000000000001},"last":1.3620000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:52:28 2021","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.3080000000000001,"15min":1.5069999999999999},"min":{"1min":0.96099999999999997,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.605,"5min":2.8700000000000001,"15min":5.6050000000000004},"last":1.532},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.3959999999999999,"15min":1.5669999999999999},"min":{"1min":1.0129999999999999,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.794,"5min":3.1259999999999999,"15min":13.968999999999999},"last":1.5129999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411680,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":364656,"kb_used_data":318312,"kb_used_omap":0,"kb_used_meta":46336,"kb_avail":93380496,"statfs":{"total":95995035648,"available":95621627904,"internally_reserved":0,"allocated":325951488,"data_stored":375280961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":47448064},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:50 2021","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.99099999999999999,"15min":1.718},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":3.698,"5min":3.996,"15min":163.15199999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.91300000000000003,"15min":1.655},"min":{"1min":0.51100000000000001,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.9710000000000001,"5min":2.2349999999999999,"15min":162.959},"last":0.85399999999999998}]},{"osd":1,"last update":"Wed Feb 10 04:52:52 2021","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.95899999999999996,"15min":1.0580000000000001},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.3029999999999999,"5min":3.8159999999999998,"15min":5.0229999999999997},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.81399999999999995,"15min":0.91700000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.177,"5min":1.895,"15min":5.726},"last":0.64200000000000002}]},{"osd":3,"last update":"Wed Feb 10 04:53:03 2021","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":1.099,"15min":1.143},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.4279999999999999,"5min":4.2380000000000004,"15min":4.2380000000000004},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.05,"5min":1.274,"15min":1.3120000000000001},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.607,"5min":4.54,"15min":4.54},"last":1.5049999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:52:30 2021","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.2450000000000001,"15min":1.3480000000000001},"min":{"1min":0.81299999999999994,"5min":0.80200000000000005,"15min":0.77000000000000002},"max":{"1min":1.8169999999999999,"5min":4.1310000000000002,"15min":4.3319999999999999},"last":1.6020000000000001},{"interface":"front","average":{"1min":1.018,"5min":1.056,"15min":1.145},"min":{"1min":0.69499999999999995,"5min":0.66800000000000004,"15min":0.65900000000000003},"max":{"1min":1.3580000000000001,"5min":3.411,"15min":4.4489999999999998},"last":0.86899999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:52:52 2021","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.8120000000000001,"15min":1.748},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.75,"5min":33.018999999999998,"15min":33.018999999999998},"last":1.7949999999999999},{"interface":"front","average":{"1min":0.95699999999999996,"5min":1.625,"15min":1.512},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.4530000000000001,"5min":33.043999999999997,"15min":33.043999999999997},"last":0.96599999999999997}]},{"osd":6,"last update":"Wed Feb 10 04:52:16 2021","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.508,"15min":1.526},"min":{"1min":0.84799999999999998,"5min":0.81599999999999995,"15min":0.67700000000000005},"max":{"1min":1.6910000000000001,"5min":20.007000000000001,"15min":20.007000000000001},"last":0.84399999999999997},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.6759999999999999,"15min":1.706},"min":{"1min":0.95399999999999996,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":4.415,"5min":19.417999999999999,"15min":19.417999999999999},"last":1.669}]},{"osd":7,"last update":"Wed Feb 10 04:52:27 2021","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.532,"15min":1.6719999999999999},"min":{"1min":1.0129999999999999,"5min":0.93400000000000005,"15min":0.93400000000000005},"max":{"1min":4.6699999999999999,"5min":4.6699999999999999,"15min":4.6699999999999999},"last":1.73},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.208,"15min":1.355},"min":{"1min":0.97699999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":1.6719999999999999,"5min":1.95,"15min":4.9189999999999996},"last":0.99199999999999999}]}]},{"osd":3,"up_from":1111,"seq":4771708665883,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":325716,"kb_used_data":300044,"kb_used_omap":386,"kb_used_meta":25277,"kb_avail":93419436,"statfs":{"total":95995035648,"available":95661502464,"internally_reserved":0,"allocated":307245056,"data_stored":409071244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":396036,"internal_metadata":25883900},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:58 2021","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":0.67200000000000004}]},{"osd":1,"last update":"Wed Feb 10 04:53:03 2021","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.65500000000000003,"15min":0.65100000000000002},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.86699999999999999,"15min":0.88500000000000001},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":2.137,"5min":2.137,"15min":2.137},"last":0.45600000000000002}]},{"osd":2,"last update":"Wed Feb 10 04:53:03 2021","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":1.0740000000000001,"15min":1.1240000000000001},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.206,"5min":6.5460000000000003,"15min":6.5460000000000003},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.97599999999999998,"5min":1.0920000000000001,"15min":1.111},"min":{"1min":0.623,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.865,"5min":6.5049999999999999,"15min":6.5049999999999999},"last":0.50800000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:53:03 2021","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.96399999999999997,"15min":0.96799999999999997},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.96,"5min":1.96,"15min":1.96},"last":0.628},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.0840000000000001,"15min":1.1140000000000001},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.5049999999999999,"5min":1.8999999999999999,"15min":1.8999999999999999},"last":0.59299999999999997}]},{"osd":5,"last update":"Wed Feb 10 04:53:03 2021","interfaces":[{"interface":"back","average":{"1min":3.9529999999999998,"5min":2.9889999999999999,"15min":2.8279999999999998},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":52.353000000000002,"5min":52.353000000000002,"15min":52.353000000000002},"last":0.66100000000000003},{"interface":"front","average":{"1min":3.7730000000000001,"5min":2.7890000000000001,"15min":2.625},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":52.301000000000002,"5min":52.301000000000002,"15min":52.301000000000002},"last":0.71999999999999997}]},{"osd":6,"last update":"Wed Feb 10 04:53:03 2021","interfaces":[{"interface":"back","average":{"1min":3.9119999999999999,"5min":5.3639999999999999,"15min":5.6059999999999999},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":51.914999999999999,"5min":64.376999999999995,"15min":64.376999999999995},"last":0.83699999999999997},{"interface":"front","average":{"1min":4.0410000000000004,"5min":5.3559999999999999,"15min":5.5750000000000002},"min":{"1min":0.96499999999999997,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":51.896000000000001,"5min":64.438999999999993,"15min":64.438999999999993},"last":0.85599999999999998}]},{"osd":7,"last update":"Wed Feb 10 04:53:03 2021","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.105,"15min":1.119},"min":{"1min":0.82699999999999996,"5min":0.754,"15min":0.754},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.143,"15min":1.143},"min":{"1min":0.90800000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"max":{"1min":2.044,"5min":2.044,"15min":2.044},"last":0.79900000000000004}]}]},{"osd":4,"up_from":31,"seq":133143986520,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":154504,"kb_used_data":120320,"kb_used_omap":0,"kb_used_meta":34176,"kb_avail":93590648,"statfs":{"total":95995035648,"available":95836823552,"internally_reserved":0,"allocated":123207680,"data_stored":156104419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":34996224},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:36 2021","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.0899999999999999,"15min":1.593},"min":{"1min":0.45800000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.7799999999999998,"5min":95.721999999999994,"15min":95.721999999999994},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.67700000000000005,"5min":2.0870000000000002,"15min":1.6000000000000001},"min":{"1min":0.46000000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.002,"5min":95.344999999999999,"15min":95.344999999999999},"last":0.70599999999999996}]},{"osd":1,"last update":"Wed Feb 10 04:52:39 2021","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.89600000000000002,"15min":0.96799999999999997},"min":{"1min":0.50900000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":3.5529999999999999,"5min":7.883,"15min":24.295000000000002},"last":0.52600000000000002},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.081,"15min":1.232},"min":{"1min":0.63800000000000001,"5min":0.61599999999999999,"15min":0.60399999999999998},"max":{"1min":3.8090000000000002,"5min":3.8090000000000002,"15min":24.943000000000001},"last":0.55200000000000005}]},{"osd":2,"last update":"Wed Feb 10 04:52:39 2021","interfaces":[{"interface":"back","average":{"1min":8.5600000000000005,"5min":2.625,"15min":1.6830000000000001},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":120.39,"5min":120.39,"15min":120.39},"last":0.95199999999999996},{"interface":"front","average":{"1min":8.6189999999999998,"5min":2.629,"15min":1.694},"min":{"1min":0.66100000000000003,"5min":0.60299999999999998,"15min":0.51900000000000002},"max":{"1min":120.583,"5min":120.583,"15min":120.583},"last":0.77400000000000002}]},{"osd":3,"last update":"Wed Feb 10 04:52:59 2021","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.2709999999999999,"15min":1.2949999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.657,"5min":2.1429999999999998,"15min":2.1429999999999998},"last":0.73199999999999998},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.484,"15min":1.514},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":3.5699999999999998,"5min":3.5699999999999998,"15min":3.5699999999999998},"last":0.82099999999999995}]},{"osd":5,"last update":"Wed Feb 10 04:52:50 2021","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.724,"15min":1.54},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.60999999999999999},"max":{"1min":1.9390000000000001,"5min":35.774000000000001,"15min":35.774000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":1.121,"5min":1.669,"15min":1.474},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.58799999999999997},"max":{"1min":1.9430000000000001,"5min":35.606000000000002,"15min":35.606000000000002},"last":0.88500000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:52:59 2021","interfaces":[{"interface":"back","average":{"1min":2.0339999999999998,"5min":3.9470000000000001,"15min":2.5299999999999998},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.76600000000000001},"max":{"1min":12.488,"5min":201.626,"15min":201.626},"last":1.014},{"interface":"front","average":{"1min":2.1179999999999999,"5min":3.907,"15min":2.5840000000000001},"min":{"1min":0.91200000000000003,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":12.84,"5min":201.577,"15min":201.577},"last":1.0409999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:52:26 2021","interfaces":[{"interface":"back","average":{"1min":1.649,"5min":1.518,"15min":1.512},"min":{"1min":1.01,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.4399999999999999,"5min":4.7439999999999998,"15min":6.9199999999999999},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.444,"5min":1.4890000000000001,"15min":1.569},"min":{"1min":0.84599999999999997,"5min":0.72899999999999998,"15min":0.71499999999999997},"max":{"1min":2.1960000000000002,"5min":2.8010000000000002,"15min":5.149},"last":0.91600000000000004}]}]},{"osd":5,"up_from":37,"seq":158913790293,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":249376,"kb_used_data":205016,"kb_used_omap":0,"kb_used_meta":44352,"kb_avail":93495776,"statfs":{"total":95995035648,"available":95739674624,"internally_reserved":0,"allocated":209936384,"data_stored":275051293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45416448},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:51 2021","interfaces":[{"interface":"back","average":{"1min":5.0590000000000002,"5min":1.714,"15min":1.204},"min":{"1min":0.45700000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":58.624000000000002,"5min":58.624000000000002,"15min":58.624000000000002},"last":0.64600000000000002},{"interface":"front","average":{"1min":5.04,"5min":1.784,"15min":1.3009999999999999},"min":{"1min":0.56599999999999995,"5min":0.46600000000000003,"15min":0.46400000000000002},"max":{"1min":57.880000000000003,"5min":57.880000000000003,"15min":57.880000000000003},"last":1.284}]},{"osd":1,"last update":"Wed Feb 10 04:52:51 2021","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.90600000000000003,"15min":0.94499999999999995},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.6200000000000001,"5min":4.4000000000000004,"15min":5.29},"last":0.57099999999999995},{"interface":"front","average":{"1min":1.573,"5min":1.238,"15min":1.198},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":11.818,"5min":11.818,"15min":11.875999999999999},"last":0.51400000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:52:51 2021","interfaces":[{"interface":"back","average":{"1min":3.7919999999999998,"5min":1.706,"15min":1.5600000000000001},"min":{"1min":0.53800000000000003,"5min":0.53700000000000003,"15min":0.50600000000000001},"max":{"1min":50.948999999999998,"5min":50.948999999999998,"15min":50.948999999999998},"last":0.81599999999999995},{"interface":"front","average":{"1min":3.9359999999999999,"5min":1.732,"15min":1.5760000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":50.149000000000001,"5min":50.149000000000001,"15min":50.149000000000001},"last":0.68600000000000005}]},{"osd":3,"last update":"Wed Feb 10 04:53:03 2021","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.2769999999999999,"15min":1.3149999999999999},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.5069999999999999,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.21,"5min":1.1859999999999999,"15min":1.1819999999999999},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.9199999999999999,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":0.90600000000000003}]},{"osd":4,"last update":"Wed Feb 10 04:52:51 2021","interfaces":[{"interface":"back","average":{"1min":1.6160000000000001,"5min":1.367,"15min":1.353},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.57099999999999995},"max":{"1min":9.4890000000000008,"5min":9.4890000000000008,"15min":9.4890000000000008},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.288,"5min":1.3680000000000001,"15min":1.409},"min":{"1min":0.85599999999999998,"5min":0.68100000000000005,"15min":0.67500000000000004},"max":{"1min":1.6890000000000001,"5min":5.6180000000000003,"15min":5.6180000000000003},"last":1.125}]},{"osd":6,"last update":"Wed Feb 10 04:53:03 2021","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":2.2799999999999998,"15min":3.0350000000000001},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.65800000000000003},"max":{"1min":2.7559999999999998,"5min":38.014000000000003,"15min":319.51999999999998},"last":1.226},{"interface":"front","average":{"1min":1.7250000000000001,"5min":2.2170000000000001,"15min":2.956},"min":{"1min":0.71399999999999997,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":4.8840000000000003,"5min":37.896999999999998,"15min":319.22300000000001},"last":0.79300000000000004}]},{"osd":7,"last update":"Wed Feb 10 04:52:32 2021","interfaces":[{"interface":"back","average":{"1min":1.8899999999999999,"5min":1.5860000000000001,"15min":1.5349999999999999},"min":{"1min":0.83599999999999997,"5min":0.75600000000000001,"15min":0.752},"max":{"1min":11.897,"5min":11.897,"15min":11.897},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.5600000000000001,"5min":1.6220000000000001,"15min":1.619},"min":{"1min":1.107,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.2229999999999999,"5min":6.0369999999999999,"15min":6.0369999999999999},"last":1.2909999999999999}]}]}]}} 2021-02-10T04:53:14.492 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2021-02-10T04:53:14.492 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pool set zone.rgw.log pg_num 22 2021-02-10T04:53:14.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:14 smithi164 conmon[29288]: cluster 2021-02-10T04:53:12.446887+0000 mgr.y (mgr.14140) 1278 : cluster [DBG] 2021-02-10T04:53:14.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:14 smithi164 conmon[29288]: pgmap v2099: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 870 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 278/2202 objects degraded (12.625%); 12/2202 objects misplaced (0.545%) 2021-02-10T04:53:14.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:14 smithi164 conmon[29288]: cluster 2021-02-10T04:53:13.099224+0000 osd.4 (osd.4) 2646 : cluster [DBG] 6.2 scrub ok 2021-02-10T04:53:14.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:14 smithi164 conmon[29288]: cluster 2021-02-10T04:53:13.508426+0000 mon.a (mon.0) 2465 : cluster [WRN] Health check update: Degraded data redundancy: 278/2202 objects degraded (12.625%), 29 pgs degraded (PG_DEGRADED) 2021-02-10T04:53:14.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:14 smithi164 conmon[29288]: cluster 2021-02-10T04:53:13.514073+0000 mon.a (mon.0) 2466 : cluster [DBG] osdmap e1163: 8 total, 7 up, 8 in 2021-02-10T04:53:14.720 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:53:15.086 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:53:15.104 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:53:15.123 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:53:15.142 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:53:15.177 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:53:15.211 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:53:15.231 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:53:15.231 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-02-10T04:53:15.231 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: problem getting command descriptions from osd.0 2021-02-10T04:53:15.231 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:53:15.231 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:53:15.232 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:53:15.232 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:53:15.232 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:53:15.232 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:53:15.232 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:53:15.257 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is set 2021-02-10T04:53:15.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:15 smithi085 conmon[29319]: cluster 2021-02-10T04:53:14.202441+0000 osd.1 (osd.1) 496 : cluster [DBG] 4.12 deep-scrub ok 2021-02-10T04:53:15.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:15 smithi085 conmon[29319]: audit 2021-02-10T04:53:15.061491+0000 mon.a (mon.0) 2467 : audit [DBG] from='client.? 172.21.15.85:0/3950698383' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:53:15.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:15 smithi085 conmon[29319]: audit 2021-02-10T04:53:15.210780+0000 mon.a (mon.0) 2468 : audit [INF] from='client.? 172.21.15.85:0/3861106876' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:53:15.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:15 smithi085 conmon[35961]: cluster 2021-02-10T04:53:14.202441+0000 osd.1 (osd.1) 496 : cluster [DBG] 4.12 deep-scrub ok 2021-02-10T04:53:15.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:15 smithi085 conmon[35961]: audit 2021-02-10T04:53:15.061491+0000 mon.a (mon.0) 2467 : audit [DBG] from='client.? 172.21.15.85:0/3950698383' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:53:15.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:15 smithi085 conmon[35961]: audit 2021-02-10T04:53:15.210780+0000 mon.a (mon.0) 2468 : audit [INF] from='client.? 172.21.15.85:0/3861106876' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:53:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:15 smithi164 conmon[29288]: cluster 2021-02-10T04:53:14.202441+0000 osd.1 (osd 2021-02-10T04:53:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:15 smithi164 conmon[29288]: .1) 496 : cluster [DBG] 4.12 deep-scrub ok 2021-02-10T04:53:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:15 smithi164 conmon[29288]: audit 2021-02-10T04:53:15.061491+0000 mon.a (mon.0) 2467 : audit [DBG] from='client.? 172.21.15.85:0/3950698383' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:53:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:15 smithi164 conmon[29288]: audit 2021-02-10T04:53:15.210780+0000 mon.a ( 2021-02-10T04:53:15.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:15 smithi164 conmon[29288]: mon.0) 2468 : audit [INF] from='client.? 172.21.15.85:0/3861106876' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:53:16.611 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:16 smithi085 conmon[35961]: cluster 2021-02-10T04:53:14.447427 2021-02-10T04:53:16.611 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:16 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1279 : cluster [DBG] pgmap v2101: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 870 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 278/2202 objects degraded (12.625%); 12/2202 objects misplaced (0.545%) 2021-02-10T04:53:16.611 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:16 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:53:16.611 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:16 smithi085 conmon[35961]: 02-10T04:53:15.185682+0000 osd.1 (osd.1) 497 : cluster [DBG] 2021-02-10T04:53:16.611 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:16 smithi085 conmon[35961]: 6.1d scrub ok 2021-02-10T04:53:16.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:16 smithi085 conmon[35961]: cluster 2021-02-10T04:53: 2021-02-10T04:53:16.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:16 smithi085 conmon[35961]: 15.246172+0000 mon.a (mon.0) 2469 2021-02-10T04:53:16.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:16 smithi085 conmon[35961]: : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:53:16.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:16 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:53:16.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:16 smithi085 conmon[35961]: -10T04:53:15. 2021-02-10T04:53:16.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:16 smithi085 conmon[35961]: 252868+0000 mon.a (mon.0) 2021-02-10T04:53:16.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:16 smithi085 conmon[35961]: 2470 : audit [INF] from='client.? 172.21.15.85:0/3861106876' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:53:16.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:16 smithi085 conmon[35961]: cluster 2021-02-10T04:53: 2021-02-10T04:53:16.612 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:16 smithi085 conmon[35961]: 15.252933+0000 mon.a (mon.0) 2471 : cluster [DBG] osdmap e1164: 8 total, 7 up, 8 in 2021-02-10T04:53:16.612 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:16 smithi085 conmon[29319]: cluster 2021-02-10T04:53:14.447427+0000 mgr.y (mgr.14140) 1279 : cluster [DBG] 2021-02-10T04:53:16.612 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:16 smithi085 conmon[29319]: pgmap v2101: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 870 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 278/2202 objects degraded (12.625%); 12/2202 objects misplaced (0.545%) 2021-02-10T04:53:16.612 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:16 smithi085 conmon[29319]: cluster 2021-02-10T04:53:15.185682+0000 osd.1 (osd.1) 497 : cluster [DBG] 6.1d scrub ok 2021-02-10T04:53:16.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:16 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:53:16.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:16 smithi085 conmon[29319]: 10T04:53:15.246172+0000 mon.a (mon. 2021-02-10T04:53:16.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:16 smithi085 conmon[29319]: 0) 2469 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:53:16.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:16 smithi085 conmon[29319]: audit 2021-02-10T04:53:16.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:16 smithi085 conmon[29319]: 2021-02-10T04:53:15.252868 2021-02-10T04:53:16.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:16 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2470 : audit [INF] 2021-02-10T04:53:16.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:16 smithi085 conmon[29319]: from='client.? 172.21.15.85:0/3861106876' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:53:16.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:16 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:53:16.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:16 smithi085 conmon[29319]: -10T04:53:15.252933+0000 2021-02-10T04:53:16.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:16 smithi085 conmon[29319]: mon.a (mon.0) 2471 : cluster [DBG] osdmap e1164: 8 total, 7 up, 8 in 2021-02-10T04:53:16.613 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:16 smithi085 conmon[29319]: 2021-02-10T04:53:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:16 smithi164 conmon[29288]: cluster 2021-02-10T04:53:14.447427 2021-02-10T04:53:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:16 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1279 : cluster [DBG] pgmap v2101: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 870 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 278/2202 objects degraded (12.625%); 12/2202 objects misplaced (0.545%) 2021-02-10T04:53:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:16 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:53:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:16 smithi164 conmon[29288]: :53:15.185682+0000 osd.1 (osd.1) 497 : cluster [DBG] 2021-02-10T04:53:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:16 smithi164 conmon[29288]: 6.1d scrub ok 2021-02-10T04:53:16.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:16 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:53:16.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:16 smithi164 conmon[29288]: -10T04:53:15.246172+0000 mon.a 2021-02-10T04:53:16.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:16 smithi164 conmon[29288]: (mon.0) 2469 : cluster 2021-02-10T04:53:16.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:16 smithi164 conmon[29288]: [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:53:16.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:16 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:53:16.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:16 smithi164 conmon[29288]: 02-10T04:53:15.252868+0000 mon.a (mon.0 2021-02-10T04:53:16.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:16 smithi164 conmon[29288]: ) 2470 : audit [INF] from='client.? 172.21.15.85:0/3861106876' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:53:16.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:16 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:53:16.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:16 smithi164 conmon[29288]: -02-10T04:53:15.252933+0000 2021-02-10T04:53:16.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:16 smithi164 conmon[29288]: mon.a (mon.0) 2471 : cluster [DBG] osdmap e1164: 8 total, 7 up, 8 in 2021-02-10T04:53:17.015 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:53:17.015 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:53:18.272 INFO:teuthology.orchestra.run.smithi085.stderr:set pool 4 pg_num to 22 2021-02-10T04:53:18.535 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:18 smithi164 conmon[29288]: cluster 2021-02-10T04:53:18.535 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:18 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:53:18.536 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:18 smithi164 conmon[29288]: 04:53:16.448040+0000 mgr.y (mgr.14140) 1280 : cluster [DBG] pgmap v2103: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 870 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 278/2202 objects degraded (12.625%); 12/2202 objects misplaced (0.545%) 2021-02-10T04:53:18.536 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:18 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:53:18.536 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:18 smithi164 conmon[29288]: -10T04:53:17.315754+0000 2021-02-10T04:53:18.536 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:18 smithi164 conmon[29288]: osd.2 (osd.2) 2020 : 2021-02-10T04:53:18.536 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:18 smithi164 conmon[29288]: cluster [DBG] purged_snaps scrub starts 2021-02-10T04:53:18.536 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:18 smithi164 conmon[29288]: cluster 2021-02-10T04:53:18.536 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:18 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:53:18.536 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:18 smithi164 conmon[29288]: 53:17.317355+0000 osd.2 ( 2021-02-10T04:53:18.536 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:18 smithi164 conmon[29288]: osd.2) 2021 : cluster [DBG] 2021-02-10T04:53:18.536 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:18 smithi164 conmon[29288]: purged_snaps scrub found 3 strays 2021-02-10T04:53:18.536 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:18 smithi164 conmon[29288]: audit 2021-02-10T04:53:18.537 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:18 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:53:18.537 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:18 smithi164 conmon[29288]: 53:18.154775+0000 mon.a ( 2021-02-10T04:53:18.537 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:18 smithi164 conmon[29288]: mon.0) 2472 : audit [INF] 2021-02-10T04:53:18.537 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:18 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/2494550881' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num", "val": "22"}]: dispatch 2021-02-10T04:53:18.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:18 smithi085 conmon[29319]: cluster 2021-02-10T04:53:16.448040+0000 mgr.y (mgr.14140 2021-02-10T04:53:18.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:18 smithi085 conmon[29319]: ) 1280 : cluster [DBG] pgmap v2103: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 870 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 278/2202 objects degraded (12.625%); 12/2202 objects misplaced (0.545%) 2021-02-10T04:53:18.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:18 smithi085 conmon[29319]: cluster 2021-02-10T04:53:17.315754+0000 osd.2 (osd.2) 2020 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:53:18.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:18 smithi085 conmon[29319]: cluster 2021-02-10T04:53:17.317355+0000 osd.2 (osd.2) 2021 : cluster [DBG] purged_snaps scrub found 3 strays 2021-02-10T04:53:18.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:18 smithi085 conmon[29319]: audit 2021-02-10T04:53:18.154775+0000 mon.a (mon.0) 2472 : audit [INF] from='client.? 172.21.15.85:0/2494550881' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num", "val": "22"}]: dispatch 2021-02-10T04:53:18.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:18 smithi085 conmon[35961]: cluster 2021-02-10T04:53:16.448040 2021-02-10T04:53:18.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:18 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1280 : cluster [DBG] pgmap v2103: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 870 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 278/2202 objects degraded (12.625%); 12/2202 objects misplaced (0.545%) 2021-02-10T04:53:18.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:18 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:53:18.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:18 smithi085 conmon[35961]: -10T04:53:17.315754+0000 osd.2 (osd.2) 2020 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:53:18.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:18 smithi085 conmon[35961]: 2021-02-10T04:53:18.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:18 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:53:18.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:18 smithi085 conmon[35961]: 04:53:17.317355+0000 osd.2 (osd.2) 2021 : cluster [DBG] 2021-02-10T04:53:18.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:18 smithi085 conmon[35961]: purged_snaps scrub found 3 strays 2021-02-10T04:53:18.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:18 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:53:18.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:18 smithi085 conmon[35961]: 53:18.154775+0000 mon.a (mon.0) 2472 : audit [INF] 2021-02-10T04:53:18.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:18 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/2494550881' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num", "val": "22"}]: dispatch 2021-02-10T04:53:19.017 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:53:19.248 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:53:19.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[29319]: audit 2021-02-10T04:53:19.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[29319]: 2021-02-10T04:53:19.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[29319]: 2021 2021-02-10T04:53:19.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[29319]: - 2021-02-10T04:53:19.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[29319]: 02 2021-02-10T04:53:19.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[29319]: - 2021-02-10T04:53:19.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[29319]: 10 2021-02-10T04:53:19.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[29319]: T 2021-02-10T04:53:19.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[29319]: 04 2021-02-10T04:53:19.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[29319]: : 2021-02-10T04:53:19.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[29319]: 53 2021-02-10T04:53:19.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[29319]: : 2021-02-10T04:53:19.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[29319]: 18 2021-02-10T04:53:19.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[29319]: . 2021-02-10T04:53:19.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[29319]: 263813 2021-02-10T04:53:19.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2473 : 2021-02-10T04:53:19.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[29319]: audit [INF] from='client.? 172.21.15.85:0/2494550881' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num", "val": "22"}]': finished 2021-02-10T04:53:19.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[29319]: cluster 2021-02-10T04:53: 2021-02-10T04:53:19.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[29319]: 18.263950+0000 mon.a (mon.0) 2474 : cluster 2021-02-10T04:53:19.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[29319]: [DBG] osdmap e1165: 8 total, 7 up, 8 in 2021-02-10T04:53:19.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:53:19.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[29319]: T04:53:18.449288+0000 mon.a (mon. 2021-02-10T04:53:19.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[29319]: 0) 2475 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pgp_num_actual", "val": "22"}]: dispatch 2021-02-10T04:53:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: audit 2021-02-10T04:53:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 2021-02-10T04:53:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 2021 2021-02-10T04:53:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: - 2021-02-10T04:53:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 02 2021-02-10T04:53:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: - 2021-02-10T04:53:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 10 2021-02-10T04:53:19.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: T 2021-02-10T04:53:19.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 04 2021-02-10T04:53:19.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: : 2021-02-10T04:53:19.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 53 2021-02-10T04:53:19.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: : 2021-02-10T04:53:19.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 18 2021-02-10T04:53:19.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: . 2021-02-10T04:53:19.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 263813 2021-02-10T04:53:19.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: +0000 2021-02-10T04:53:19.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 2021-02-10T04:53:19.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: mon.a 2021-02-10T04:53:19.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: ( 2021-02-10T04:53:19.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: mon 2021-02-10T04:53:19.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: . 2021-02-10T04:53:19.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 0 2021-02-10T04:53:19.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: ) 2021-02-10T04:53:19.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 2473 2021-02-10T04:53:19.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: : 2021-02-10T04:53:19.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: audit 2021-02-10T04:53:19.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 2021-02-10T04:53:19.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: [INF] 2021-02-10T04:53:19.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 2021-02-10T04:53:19.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/2494550881' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num", "val": "22"}]': finished 2021-02-10T04:53:19.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 2021-02-10T04:53:19.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: cluster 2021-02-10T04:53:19.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 2021-02-10T04:53:19.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 2021 2021-02-10T04:53:19.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: - 2021-02-10T04:53:19.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 02 2021-02-10T04:53:19.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: - 2021-02-10T04:53:19.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 10 2021-02-10T04:53:19.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: T 2021-02-10T04:53:19.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 04 2021-02-10T04:53:19.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: : 2021-02-10T04:53:19.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 53 2021-02-10T04:53:19.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: : 2021-02-10T04:53:19.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 18 2021-02-10T04:53:19.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: . 2021-02-10T04:53:19.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 263950 2021-02-10T04:53:19.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: +0000 2021-02-10T04:53:19.447 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 2021-02-10T04:53:19.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: mon.a 2021-02-10T04:53:19.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: ( 2021-02-10T04:53:19.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: mon 2021-02-10T04:53:19.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: . 2021-02-10T04:53:19.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 0 2021-02-10T04:53:19.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: ) 2021-02-10T04:53:19.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 2474 2021-02-10T04:53:19.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: : 2021-02-10T04:53:19.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: cluster 2021-02-10T04:53:19.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 2021-02-10T04:53:19.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: [DBG] 2021-02-10T04:53:19.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 2021-02-10T04:53:19.448 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: osdmap e1165: 8 total, 7 up, 8 in 2021-02-10T04:53:19.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 2021-02-10T04:53:19.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: audit 2021-02-10T04:53:19.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 2021-02-10T04:53:19.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 2021 2021-02-10T04:53:19.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: - 2021-02-10T04:53:19.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 02 2021-02-10T04:53:19.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: - 2021-02-10T04:53:19.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 10 2021-02-10T04:53:19.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: T 2021-02-10T04:53:19.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 04 2021-02-10T04:53:19.449 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: : 2021-02-10T04:53:19.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 53 2021-02-10T04:53:19.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: : 2021-02-10T04:53:19.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 18 2021-02-10T04:53:19.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: . 2021-02-10T04:53:19.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 449288 2021-02-10T04:53:19.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: +0000 2021-02-10T04:53:19.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 2021-02-10T04:53:19.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: mon.a 2021-02-10T04:53:19.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: ( 2021-02-10T04:53:19.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: mon 2021-02-10T04:53:19.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: . 2021-02-10T04:53:19.450 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 0 2021-02-10T04:53:19.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: ) 2021-02-10T04:53:19.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 2475 2021-02-10T04:53:19.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: : 2021-02-10T04:53:19.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: audit 2021-02-10T04:53:19.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 2021-02-10T04:53:19.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: [INF] 2021-02-10T04:53:19.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 2021-02-10T04:53:19.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pgp_num_actual", "val": "22"}]: dispatch 2021-02-10T04:53:19.451 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:19 smithi085 conmon[35961]: 2021-02-10T04:53:19.536 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set nodeep-scrub 2021-02-10T04:53:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:19 smithi164 conmon[29288]: audit 2021-02-10T04:53:18.263813+0000 mon.a ( 2021-02-10T04:53:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:19 smithi164 conmon[29288]: mon.0) 2473 : audit [INF] from='client.? 172.21.15.85:0/2494550881' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num", "val": "22"}]': finished 2021-02-10T04:53:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:19 smithi164 conmon[29288]: cluster 2021-02-10T04:53:18.263950+0000 mon.a (mon.0) 2474 : cluster [DBG] osdmap e1165: 8 total, 7 up, 8 in 2021-02-10T04:53:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:19 smithi164 conmon[29288]: audit 2021-02-10T04:53:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:19 smithi164 conmon[29288]: 2021-02-10T04:53:18.449288+0000 mon.a (mon.0) 2475 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pgp_num_actual", "val": "22"}]: dispatch 2021-02-10T04:53:19.644 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:53:19 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:53:19.770 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:53:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:20 smithi164 conmon[29288]: cluster 2021-02-10T04:53:18.448486+0000 2021-02-10T04:53:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:20 smithi164 conmon[29288]: mgr.y (mgr.14140) 1281 : cluster [DBG] pgmap v2105: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 870 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 278/2202 objects degraded (12.625%); 12/2202 objects misplaced (0.545%) 2021-02-10T04:53:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:20 smithi164 conmon[29288]: audit 2021-02-10T04:53:19.183999+0000 mgr.y (mgr.14140) 1282 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:53:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:20 smithi164 conmon[29288]: audit 2021-02-10T04:53:19.264518+0000 mon.a (mon.0) 2476 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pgp_num_actual", "val": "22"}]': finished 2021-02-10T04:53:20.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:20 smithi164 conmon[29288]: cluster 2021-02-10T04:53:19.264583+0000 mon.a (mon.0) 2477 : cluster [DBG] osdmap e1166: 8 total, 7 up, 8 in 2021-02-10T04:53:20.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:20 smithi085 conmon[29319]: cluster 2021-02-10T04:53:18.448486+0000 mgr.y (mgr.14140) 1281 : cluster [DBG] pgmap v2105: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 870 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 278/2202 objects degraded (12.625%); 12/2202 objects misplaced (0.545%) 2021-02-10T04:53:20.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:20 smithi085 conmon[29319]: audit 2021-02-10T04:53:19.183999+0000 mgr.y (mgr.14140) 1282 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:53:20.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:20 smithi085 conmon[29319]: audit 2021-02-10T04:53:19.264518+0000 mon.a (mon.0) 2476 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pgp_num_actual", "val": "22"}]': finished 2021-02-10T04:53:20.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:20 smithi085 conmon[29319]: cluster 2021-02-10T04:53:19.264583+0000 mon.a (mon.0) 2477 : cluster [DBG] osdmap e1166: 8 total, 7 up, 8 in 2021-02-10T04:53:20.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:20 smithi085 conmon[35961]: cluster 2021-02-10T04:53:18.448486+0000 mgr.y (mgr.14140) 1281 : cluster [DBG] pgmap v2105: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 870 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 278/2202 objects degraded (12.625%); 12/2202 objects misplaced (0.545%) 2021-02-10T04:53:20.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:20 smithi085 conmon[35961]: audit 2021-02-10T04:53:19.183999+0000 mgr.y (mgr.14140) 1282 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:53:20.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:20 smithi085 conmon[35961]: audit 2021-02-10T04:53:19.264518+0000 mon.a (mon.0) 2476 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pgp_num_actual", "val": "22"}]': finished 2021-02-10T04:53:20.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:20 smithi085 conmon[35961]: cluster 2021-02-10T04:53:19.264583+0000 mon.a (mon 2021-02-10T04:53:20.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:20 smithi085 conmon[35961]: .0) 2477 : cluster [DBG] osdmap e1166: 8 total, 7 up, 8 in 2021-02-10T04:53:21.130 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:53:21.130 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:53:21.361 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:53:21.385 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:53:21.408 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:53:21.431 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:53:21.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:21 smithi085 conmon[29319]: cluster 2021-02-10T04:53:20.280202+0000 mon.a ( 2021-02-10T04:53:21.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:21 smithi085 conmon[29319]: mon.0) 2478 : cluster [DBG] osdmap e1167: 8 total, 7 up, 8 in 2021-02-10T04:53:21.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:21 smithi085 conmon[35961]: cluster 2021-02-10T04:53:20.280202 2021-02-10T04:53:21.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:21 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2478 : cluster [DBG] osdmap e1167: 8 total, 7 up, 8 in 2021-02-10T04:53:21.452 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:53:21.478 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:53:21.503 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:53:21.503 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-02-10T04:53:21.503 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: problem getting command descriptions from osd.0 2021-02-10T04:53:21.503 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:53:21.503 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:53:21.504 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:53:21.504 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:53:21.504 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:53:21.504 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:53:21.504 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:53:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:21 smithi164 conmon[29288]: cluster 2021-02-10T04:53:20.280202+0000 mon.a (mon 2021-02-10T04:53:21.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:21 smithi164 conmon[29288]: .0) 2478 : cluster [DBG] osdmap e1167: 8 total, 7 up, 8 in 2021-02-10T04:53:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:22 smithi164 conmon[29288]: cluster 2021-02-10T04:53:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:22 smithi164 conmon[29288]: 2021-02-10T04:53:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:22 smithi164 conmon[29288]: 2021-02-10T04:53: 2021-02-10T04:53:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:22 smithi164 conmon[29288]: 20.449174+0000 mgr.y (mgr.14140) 1283 : cluster [DBG] pgmap v2108: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 870 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 984 B/s rd, 0 op/s; 278/2202 objects degraded (12.625%); 12/2202 objects misplaced (0.545%) 2021-02-10T04:53:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:22 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:53:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:22 smithi164 conmon[29288]: 10T04:53:21.292749+0000 mon.a (mon. 2021-02-10T04:53:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:22 smithi164 conmon[29288]: 0) 2479 : cluster [DBG] osdmap e1168: 8 total, 7 up, 8 in 2021-02-10T04:53:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:22 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:53:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:22 smithi164 conmon[29288]: T04:53:21.326769+0000 mon.a (mon.0 2021-02-10T04:53:22.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:22 smithi164 conmon[29288]: ) 2480 : audit [DBG] from='client.? 172.21.15.85:0/616776226' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:53:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:22 smithi085 conmon[29319]: cluster 2021-02-10T04:53:20.449174+0000 mgr.y (mgr.14140) 1283 : cluster 2021-02-10T04:53:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:22 smithi085 conmon[29319]: [DBG] pgmap v2108: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 870 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 984 B/s rd, 0 op/s; 278/2202 objects degraded (12.625%); 12/2202 objects misplaced (0.545%) 2021-02-10T04:53:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:22 smithi085 conmon[29319]: cluster 2021-02-10T04:53:21.292749+0000 mon.a (mon.0) 2479 : cluster [DBG] osdmap e1168: 8 total, 7 up, 8 in 2021-02-10T04:53:22.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:22 smithi085 conmon[29319]: audit 2021-02-10T04:53:21.326769+0000 mon.a (mon.0) 2480 : audit [DBG] from='client.? 172.21.15.85:0/616776226' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:53:22.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:22 smithi085 conmon[35961]: cluster 2021-02-10T04:53:20.449174+0000 mgr.y 2021-02-10T04:53:22.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:22 smithi085 conmon[35961]: (mgr.14140) 1283 : cluster [DBG] pgmap v2108: 198 pgs: 1 active+clean+remapped, 29 active+undersized+degraded, 37 active+undersized, 131 active+clean; 870 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 984 B/s rd, 0 op/s; 278/2202 objects degraded (12.625%); 12/2202 objects misplaced (0.545%) 2021-02-10T04:53:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:22 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:53:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:22 smithi085 conmon[35961]: 02-10T04:53:21.292749+0000 mon.a (mon.0) 2479 : cluster [DBG] osdmap e1168: 8 total, 7 up, 8 in 2021-02-10T04:53:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:22 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:53:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:22 smithi085 conmon[35961]: -10T04:53:21.326769+0000 mon.a (mon. 2021-02-10T04:53:22.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:22 smithi085 conmon[35961]: 0) 2480 : audit [DBG] from='client.? 172.21.15.85:0/616776226' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:53:22.866 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:53:22.866 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:53:23.373 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:23 smithi085 conmon[29319]: cluster 2021-02-10T04:53:22.449807+0000 mgr.y 2021-02-10T04:53:23.374 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:23 smithi085 conmon[29319]: (mgr.14140) 1284 : cluster [DBG] pgmap v2110: 198 pgs: 1 remapped+peering, 1 active+clean+remapped, 1 active+clean+snaptrim, 26 active+undersized+degraded, 37 active+undersized, 132 active+clean; 870 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 237/2163 objects degraded (10.957%); 12/2163 objects misplaced (0.555%); 424 B/s, 5 objects/s recovering 2021-02-10T04:53:23.374 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:23 smithi085 conmon[29319]: cluster 2021-02-10T04:53:23.287523+0000 mon.a (mon.0) 2481 : cluster [WRN] Health check update: Degraded data redundancy: 237/2163 objects degraded (10.957%), 26 pgs degraded (PG_DEGRADED) 2021-02-10T04:53:23.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:23 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:53:23.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:23 smithi085 conmon[35961]: 53:22.449807+0000 mgr.y (mgr.14140) 1284 : cluster [DBG] pgmap v2110: 198 pgs: 1 remapped+peering, 1 active+clean+remapped, 1 active+clean+snaptrim, 26 active+undersized+degraded, 37 active+undersized, 132 active+clean; 870 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 237/2163 objects degraded (10.957%); 12/2163 objects misplaced (0.555%); 424 B/s, 5 objects/s recovering 2021-02-10T04:53:23.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:23 smithi085 conmon[35961]: cluster 2021-02-10T04:53:23.287523+0000 mon.a (mon.0) 2481 : cluster [WRN] Health check update: Degraded data redundancy: 237/2163 objects degraded (10.957%), 26 pgs degraded (PG_DEGRADED) 2021-02-10T04:53:23.576 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.7 dump_blocked_ops 2021-02-10T04:53:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:23 smithi164 conmon[29288]: cluster 2021-02-10T04:53:22.449807+0000 mgr.y (mgr.14140) 1284 : cluster 2021-02-10T04:53:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:23 smithi164 conmon[29288]: [DBG] pgmap v2110: 198 pgs: 1 remapped+peering, 1 active+clean+remapped, 1 active+clean+snaptrim, 26 active+undersized+degraded, 37 active+undersized, 132 active+clean; 870 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 237/2163 objects degraded (10.957%); 12/2163 objects misplaced (0.555%); 424 B/s, 5 objects/s recovering 2021-02-10T04:53:23.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:23 smithi164 conmon[29288]: cluster 2021-02-10T04:53:23.287523+0000 mon.a (mon.0) 2481 : cluster [WRN] Health check update: Degraded data redundancy: 237/2163 objects degraded (10.957%), 26 pgs degraded (PG_DEGRADED) 2021-02-10T04:53:23.877 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:53:24.245 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2021-02-10T04:53:24.245 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:53:24.245 INFO:tasks.thrashosds.thrasher:fixing pg num pool zone.rgw.log 2021-02-10T04:53:24.246 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:53:24.464 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:53:24.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:24 smithi085 conmon[35961]: cluster 2021-02-10T04:53:23.520635 2021-02-10T04:53:24.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:24 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2482 : cluster [DBG] osdmap e1169: 8 total, 7 up, 8 in 2021-02-10T04:53:24.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:24 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:53:24.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:24 smithi085 conmon[35961]: 53:23.661178+0000 mon.c (mon.1) 107 : audit [INF] from='client.? 172.21.15.85:0/3916094499' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:53:24.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:24 smithi085 conmon[35961]: audit 2021-02-10T04:53:23.661479+0000 mon.a (mon 2021-02-10T04:53:24.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:24 smithi085 conmon[35961]: .0) 2483 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:53:24.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:24 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:53:24.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:24 smithi085 conmon[35961]: 02-10T04:53:24.451361+0000 2021-02-10T04:53:24.522 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:24 smithi085 conmon[35961]: mon.a (mon.0) 2484 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "31"}]: dispatch 2021-02-10T04:53:24.542 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is set 2021-02-10T04:53:24.618 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:24 smithi164 conmon[29288]: cluster 2021-02-10T04:53:23.520635+0000 2021-02-10T04:53:24.619 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:24 smithi164 conmon[29288]: mon.a (mon.0) 2482 : cluster [DBG] osdmap e1169: 8 total, 7 up, 8 in 2021-02-10T04:53:24.619 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:24 smithi164 conmon[29288]: audit 2021-02-10T04:53:23.661178+0000 mon.c (mon.1) 107 : audit [INF] from='client.? 172.21.15.85:0/3916094499' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:53:24.619 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:24 smithi164 conmon[29288]: audit 2021-02-10T04:53:23.661479+0000 mon.a (mon.0) 2483 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:53:24.619 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:24 smithi164 conmon[29288]: audit 2021-02-10T04:53:24.451361+0000 mon.a (mon.0) 2484 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "31"}]: dispatch 2021-02-10T04:53:24.867 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:53:24.889 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:24 smithi085 conmon[29319]: cluster 2021-02-10T04:53:24.889 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:24 smithi085 conmon[29319]: 2021- 2021-02-10T04:53:24.889 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:24 smithi085 conmon[29319]: 02-10T04:53:23.520635+0000 mon.a (mon.0) 2482 : cluster [DBG] osdmap e1169: 8 total, 7 up, 8 in 2021-02-10T04:53:24.889 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:24 smithi085 conmon[29319]: audit 2021-02-10T04:53:24.890 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:24 smithi085 conmon[29319]: 2021-02-10T04:53:23.661178+0000 2021-02-10T04:53:24.890 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:24 smithi085 conmon[29319]: mon.c (mon.1) 107 : audit [INF] 2021-02-10T04:53:24.890 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:24 smithi085 conmon[29319]: from='client.? 172.21.15.85:0/3916094499' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:53:24.890 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:24 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:53:24.890 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:24 smithi085 conmon[29319]: -10T04:53:23.661479+0000 2021-02-10T04:53:24.890 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:24 smithi085 conmon[29319]: mon.a (mon.0) 2483 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:53:24.890 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:24 smithi085 conmon[29319]: 2021-02-10T04:53:24.890 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:24 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:53:24.890 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:24 smithi085 conmon[29319]: :53:24.451361+0000 mon.a (mon. 2021-02-10T04:53:24.890 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:24 smithi085 conmon[29319]: 0) 2484 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "31"}]: dispatch 2021-02-10T04:53:25.077 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:53:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:25 smithi164 conmon[29288]: cluster 2021-02-10T04:53:24.450440+0000 mgr.y ( 2021-02-10T04:53:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:25 smithi164 conmon[29288]: mgr.14140) 1285 : cluster [DBG] pgmap v2112: 198 pgs: 1 remapped+peering, 3 active+clean+remapped, 1 active+clean+snaptrim, 25 active+undersized+degraded, 37 active+undersized, 131 active+clean; 870 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 227/2163 objects degraded (10.495%); 21/2163 objects misplaced (0.971%); 917 B/s, 11 objects/s recovering 2021-02-10T04:53:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:25 smithi164 conmon[29288]: cluster 2021-02-10T04:53:24.525238+0000 mon.a (mon.0) 2485 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:53:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:25 smithi164 conmon[29288]: audit 2021-02-10T04:53:24.532996+0000 mon.a (mon.0) 2486 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:53:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:25 smithi164 conmon[29288]: audit 2021-02-10T04:53:24.533073+0000 mon.a (mon.0) 2487 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "31"}]': finished 2021-02-10T04:53:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:25 smithi164 conmon[29288]: cluster 2021-02-10T04:53:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:25 smithi164 conmon[29288]: 2021-02-10T04:53:24. 2021-02-10T04:53:25.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:25 smithi164 conmon[29288]: 533119+0000 mon.a (mon.0) 2488 : cluster [DBG] 2021-02-10T04:53:25.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:25 smithi164 conmon[29288]: osdmap e1170: 8 total, 7 up, 8 in 2021-02-10T04:53:25.780 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:25 smithi085 conmon[29319]: cluster 2021-02-10T04:53:24.450440+0000 mgr.y ( 2021-02-10T04:53:25.781 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:25 smithi085 conmon[29319]: mgr.14140) 1285 : cluster [DBG] pgmap v2112: 198 pgs: 1 remapped+peering, 3 active+clean+remapped, 1 active+clean+snaptrim, 25 active+undersized+degraded, 37 active+undersized, 131 active+clean; 870 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 227/2163 objects degraded (10.495%); 21/2163 objects misplaced (0.971%); 917 B/s, 11 objects/s recovering 2021-02-10T04:53:25.781 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:25 smithi085 conmon[29319]: cluster 2021-02-10T04:53:24.525238+0000 mon.a (mon.0) 2485 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:53:25.781 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:25 smithi085 conmon[29319]: audit 2021-02-10T04:53:24.532996+0000 mon.a (mon.0) 2486 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:53:25.781 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:25 smithi085 conmon[29319]: audit 2021-02-10T04:53:24.533073+0000 mon.a (mon.0) 2487 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "31"}]': finished 2021-02-10T04:53:25.781 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:25 smithi085 conmon[29319]: cluster 2021-02-10T04:53:24.533119+0000 mon.a (mon.0) 2488 : cluster [DBG] osdmap e1170: 8 total, 7 up, 8 in 2021-02-10T04:53:25.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:25 smithi085 conmon[35961]: cluster 2021-02-10T04:53:24.450440+0000 2021-02-10T04:53:25.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:25 smithi085 conmon[35961]: mgr.y (mgr.14140) 1285 : cluster [DBG] pgmap v2112: 198 pgs: 1 remapped+peering, 3 active+clean+remapped, 1 active+clean+snaptrim, 25 active+undersized+degraded, 37 active+undersized, 131 active+clean; 870 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 227/2163 objects degraded (10.495%); 21/2163 objects misplaced (0.971%); 917 B/s, 11 objects/s recovering 2021-02-10T04:53:25.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:25 smithi085 conmon[35961]: cluster 2021-02-10T04:53:25.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:25 smithi085 conmon[35961]: 2021-02-10T04:53:24.525238+0000 mon.a (mon.0) 2485 : cluster 2021-02-10T04:53:25.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:25 smithi085 conmon[35961]: [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:53:25.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:25 smithi085 conmon[35961]: audit 2021-02-10T04:53 2021-02-10T04:53:25.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:25 smithi085 conmon[35961]: :24.532996+0000 mon.a (mon.0) 2486 : audit 2021-02-10T04:53:25.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:25 smithi085 conmon[35961]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:53:25.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:25 smithi085 conmon[35961]: audit 2021-02-10T04:53:24 2021-02-10T04:53:25.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:25 smithi085 conmon[35961]: .533073+0000 mon.a (mon.0) 2487 : audit 2021-02-10T04:53:25.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:25 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "31"}]': finished 2021-02-10T04:53:25.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:25 smithi085 conmon[35961]: cluster 2021-02-10T04:53:24.533119+0000 2021-02-10T04:53:25.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:25 smithi085 conmon[35961]: mon.a (mon.0) 2488 : cluster [DBG] 2021-02-10T04:53:25.782 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:25 smithi085 conmon[35961]: osdmap e1170: 8 total, 7 up, 8 in 2021-02-10T04:53:26.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:26 smithi164 conmon[29288]: cluster 2021-02-10T04:53:26.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:26 smithi164 conmon[29288]: 2021-02-10T04:53:26.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:26 smithi164 conmon[29288]: 2021 2021-02-10T04:53:26.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:26 smithi164 conmon[29288]: - 2021-02-10T04:53:26.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:26 smithi164 conmon[29288]: 02-10T04:53 2021-02-10T04:53:26.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:26 smithi164 conmon[29288]: :25.533376+0000 mon.a (mon.0) 2489 : cluster [DBG] osdmap e1171: 8 total, 7 up, 8 in 2021-02-10T04:53:26.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:26 smithi085 conmon[35961]: cluster 2021-02-10T04:53:25.533376+0000 2021-02-10T04:53:26.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:26 smithi085 conmon[35961]: mon.a (mon.0) 2489 : cluster [DBG] osdmap e1171: 8 total, 7 up, 8 in 2021-02-10T04:53:26.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:26 smithi085 conmon[29319]: cluster 2021-02-10T04:53:25.533376 2021-02-10T04:53:26.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:26 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2489 : cluster [DBG] osdmap e1171: 8 total, 7 up, 8 in 2021-02-10T04:53:27.784 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset noscrub 2021-02-10T04:53:27.813 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:27 smithi085 conmon[29319]: cluster 2021-02-10T04:53:26. 2021-02-10T04:53:27.813 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:27 smithi085 conmon[29319]: 451052+0000 mgr.y (mgr.14140) 1286 : cluster [DBG] pgmap v2115: 198 pgs: 4 active+clean+remapped, 25 active+undersized+degraded, 37 active+undersized, 132 active+clean; 868 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 1.5 KiB/s wr, 23 op/s; 227/2199 objects degraded (10.323%); 34/2199 objects misplaced (1.546%); 1.1 KiB/s, 14 objects/s recovering 2021-02-10T04:53:27.813 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:27 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:53:27.813 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:27 smithi085 conmon[29319]: 53:26.536722+0000 mon.a (mon.0) 2490 : cluster [DBG] osdmap e1172: 8 total, 7 up, 8 in 2021-02-10T04:53:27.813 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:27 smithi085 conmon[35961]: cluster 2021-02-10T04:53:26.451052+0000 mgr.y 2021-02-10T04:53:27.813 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:27 smithi085 conmon[35961]: (mgr.14140) 1286 : cluster [DBG] pgmap v2115: 198 pgs: 4 active+clean+remapped, 25 active+undersized+degraded, 37 active+undersized, 132 active+clean; 868 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 1.5 KiB/s wr, 23 op/s; 227/2199 objects degraded (10.323%); 34/2199 objects misplaced (1.546%); 1.1 KiB/s, 14 objects/s recovering 2021-02-10T04:53:27.814 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:27 smithi085 conmon[35961]: cluster 2021-02-10T04:53:26.536722+0000 mon.a (mon.0) 2490 2021-02-10T04:53:27.814 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:27 smithi085 conmon[35961]: : cluster [DBG] osdmap e1172: 8 total, 7 up, 8 in 2021-02-10T04:53:27.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:27 smithi164 conmon[29288]: cluster 2021-02-10T04:53:26.451052+0000 mgr.y (mgr.14140 2021-02-10T04:53:27.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:27 smithi164 conmon[29288]: ) 1286 : cluster [DBG] pgmap v2115: 198 pgs: 4 active+clean+remapped, 25 active+undersized+degraded, 37 active+undersized, 132 active+clean; 868 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 1.5 KiB/s wr, 23 op/s; 227/2199 objects degraded (10.323%); 34/2199 objects misplaced (1.546%); 1.1 KiB/s, 14 objects/s recovering 2021-02-10T04:53:27.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:27 smithi164 conmon[29288]: cluster 2021-02-10T04:53:26.536722+0000 mon.a (mon.0) 2490 : cluster [DBG] osdmap e1172: 8 total, 7 up, 8 in 2021-02-10T04:53:28.025 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:53:28.878 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:53:28.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:28 smithi164 conmon[29288]: cluster 2021-02-10T04:53:27.546932+0000 mon.a (mon 2021-02-10T04:53:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:28 smithi164 conmon[29288]: .0) 2491 : cluster [DBG] osdmap e1173: 8 total, 7 up, 8 in 2021-02-10T04:53:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:28 smithi164 conmon[29288]: audit 2021-02-10T04:53:28.452515+0000 mon.a (mon.0) 2492 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "30"}]: dispatch 2021-02-10T04:53:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:28 smithi164 conmon[29288]: cluster 2021-02-10T04:53:28.513352+0000 2021-02-10T04:53:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:28 smithi164 conmon[29288]: mon.a (mon.0) 2493 : cluster [WRN] Health check update: Degraded data redundancy: 227/2199 objects degraded (10.323%), 25 pgs degraded (PG_DEGRADED) 2021-02-10T04:53:28.901 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:53:28.922 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:53:28.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:28 smithi085 conmon[29319]: cluster 2021-02-10T04:53:27.546932+0000 mon.a (mon.0) 2491 : cluster [DBG] osdmap e1173: 8 total, 7 up, 8 in 2021-02-10T04:53:28.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:28 smithi085 conmon[29319]: audit 2021-02-10T04:53:28.452515+0000 mon.a (mon.0) 2492 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "30"}]: dispatch 2021-02-10T04:53:28.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:28 smithi085 conmon[29319]: cluster 2021-02-10T04:53:28.513352+0000 mon.a (mon.0) 2493 : cluster [WRN] Health check update: Degraded data redundancy: 227/2199 objects degraded (10.323%), 25 pgs degraded (PG_DEGRADED) 2021-02-10T04:53:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:28 smithi085 conmon[35961]: cluster 2021-02-10T04:53:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:28 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:53:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:28 smithi085 conmon[35961]: 53:27.546932+0000 mon.a (mon.0) 2491 : cluster [DBG] osdmap e1173: 8 total, 7 up, 8 in 2021-02-10T04:53:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:28 smithi085 conmon[35961]: audit 2021-02-10T04:53:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:28 smithi085 conmon[35961]: 2021 2021-02-10T04:53:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:28 smithi085 conmon[35961]: -02-10T04: 2021-02-10T04:53:28.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:28 smithi085 conmon[35961]: 53:28.452515+0000 mon.a (mon.0) 2492 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "30"}]: dispatch 2021-02-10T04:53:28.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:28 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:53:28.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:28 smithi085 conmon[35961]: 02-10T04:53:28.513352+0000 mon.a (mon.0 2021-02-10T04:53:28.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:28 smithi085 conmon[35961]: ) 2493 : cluster [WRN] Health check update: Degraded data redundancy: 227/2199 objects degraded (10.323%), 25 pgs degraded (PG_DEGRADED) 2021-02-10T04:53:28.944 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:53:28.978 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:53:29.012 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:53:29.034 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:53:29.034 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-02-10T04:53:29.034 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: problem getting command descriptions from osd.0 2021-02-10T04:53:29.034 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:53:29.034 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:53:29.034 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:53:29.034 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:53:29.034 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:53:29.034 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:53:29.035 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:53:29.075 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:53:29.079 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:53:29.600 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:53:29.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: cluster 2021-02-10T04:53:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: 2021-02-10T04:53:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: 2021 2021-02-10T04:53:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: - 2021-02-10T04:53:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: 02 2021-02-10T04:53:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: - 2021-02-10T04:53:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: 10 2021-02-10T04:53:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: T 2021-02-10T04:53:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: 04:53:28.252054+0000 osd.6 2021-02-10T04:53:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: (osd.6) 179 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:53:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: cluster 2021-02-10T04:53:29.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: 2021-02-10T04:53:29.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: 2021 2021-02-10T04:53:29.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: - 2021-02-10T04:53:29.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: 02 2021-02-10T04:53:29.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: - 2021-02-10T04:53:29.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: 10 2021-02-10T04:53:29.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: T 2021-02-10T04:53:29.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: 04 2021-02-10T04:53:29.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: : 2021-02-10T04:53:29.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: 53:28.255698+0000 2021-02-10T04:53:29.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: osd.6 (osd.6) 180 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:53:29.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:53:29.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: -02-10T04:53:28.451677+0000 2021-02-10T04:53:29.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: mgr.y (mgr.14140) 1287 : cluster [DBG] pgmap v2118: 197 pgs: 4 active+clean+remapped, 25 active+undersized+degraded, 37 active+undersized, 131 active+clean; 868 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 2.0 KiB/s wr, 31 op/s; 227/2199 objects degraded (10.323%); 34/2199 objects misplaced (1.546%); 301 B/s, 3 objects/s recovering 2021-02-10T04:53:29.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: 2021-02-10T04:53:29.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: audit 2021-02-10T04:53:28. 2021-02-10T04:53:29.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: 590046+0000 mon.a (mon.0) 2494 : audit 2021-02-10T04:53:29.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "30"}]': finished 2021-02-10T04:53:29.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: cluster 2021-02-10T04:53 2021-02-10T04:53:29.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: :28.590163+0000 mon.a (mon.0) 2495 2021-02-10T04:53:29.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: : cluster [DBG] osdmap e1174: 8 total, 7 up, 8 in 2021-02-10T04:53:29.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:53:29.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: T04:53:28.848370+0000 mon.c (mon. 2021-02-10T04:53:29.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: 1) 108 : audit [DBG] from='client.? 172.21.15.85:0/4024792241' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:53:29.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:53:29.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: -10T04:53:29.056050+0000 mgr.y ( 2021-02-10T04:53:29.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: mgr.14140) 1288 : audit [DBG] from='client.25738 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:53:29.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: audit 2021 2021-02-10T04:53:29.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: -02-10T04:53:29.193988+0000 2021-02-10T04:53:29.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: mgr.y (mgr.14140) 1289 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:53:29.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:29 smithi164 conmon[29288]: 2021-02-10T04:53:29.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[29319]: cluster 2021-02-10T04:53:28.252054+0000 osd.6 2021-02-10T04:53:29.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[29319]: (osd.6) 179 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:53:29.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[29319]: cluster 2021-02-10T04:53:28.255698+0000 osd.6 (osd.6) 180 : cluster [DBG] 2021-02-10T04:53:29.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[29319]: purged_snaps scrub ok 2021-02-10T04:53:29.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:53:29.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[29319]: 10T04:53:28.451677+0000 mgr.y (mgr.14140) 1287 : cluster 2021-02-10T04:53:29.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[29319]: [DBG] pgmap v2118: 197 pgs: 4 active+clean+remapped, 25 active+undersized+degraded, 37 active+undersized, 131 active+clean; 868 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 2.0 KiB/s wr, 31 op/s; 227/2199 objects degraded (10.323%); 34/2199 objects misplaced (1.546%); 301 B/s, 3 objects/s recovering 2021-02-10T04:53:29.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[29319]: audit 2021-02-10T04: 2021-02-10T04:53:29.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[29319]: 53:28.590046+0000 mon.a (mon.0 2021-02-10T04:53:29.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[29319]: ) 2494 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "30"}]': finished 2021-02-10T04:53:29.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:53:29.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[29319]: 02-10T04:53:28.590163+0000 2021-02-10T04:53:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[29319]: mon.a (mon.0) 2495 : cluster [DBG] 2021-02-10T04:53:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[29319]: osdmap e1174: 8 total, 7 up, 8 in 2021-02-10T04:53:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:53:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[29319]: :53:28.848370+0000 mon.c (mon.1 2021-02-10T04:53:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[29319]: ) 108 : audit [DBG] from='client.? 172.21.15.85:0/4024792241' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:53:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:53:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[29319]: 02-10T04:53:29.056050+0000 2021-02-10T04:53:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[29319]: mgr.y (mgr.14140) 1288 : audit [DBG] 2021-02-10T04:53:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[29319]: from='client.25738 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:53:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:53:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[29319]: :53:29.193988+0000 mgr.y (mgr.14140 2021-02-10T04:53:29.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[29319]: ) 1289 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:53:29.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[35961]: cluster 2021-02-10T04:53:28.252054 2021-02-10T04:53:29.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[35961]: +0000 osd.6 (osd.6) 179 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:53:29.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:53:29.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[35961]: 04:53:28.255698+0000 osd.6 (osd.6) 180 : cluster 2021-02-10T04:53:29.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[35961]: [DBG] purged_snaps scrub ok 2021-02-10T04:53:29.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:53:29.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[35961]: :53:28.451677+0000 mgr.y (mgr. 2021-02-10T04:53:29.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[35961]: 14140) 1287 : cluster [DBG] pgmap v2118: 197 pgs: 4 active+clean+remapped, 25 active+undersized+degraded, 37 active+undersized, 131 active+clean; 868 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 2.0 KiB/s wr, 31 op/s; 227/2199 objects degraded (10.323%); 34/2199 objects misplaced (1.546%); 301 B/s, 3 objects/s recovering 2021-02-10T04:53:29.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:53:29.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[35961]: 02-10T04:53:28.590046+0000 2021-02-10T04:53:29.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[35961]: mon.a (mon.0) 2494 : audit [INF] 2021-02-10T04:53:29.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "30"}]': finished 2021-02-10T04:53:29.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:53:29.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[35961]: 53:28.590163+0000 mon.a (mon. 2021-02-10T04:53:29.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[35961]: 0) 2495 : cluster [DBG] osdmap e1174: 8 total, 7 up, 8 in 2021-02-10T04:53:29.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:53:29.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[35961]: 02-10T04:53:28.848370+0000 2021-02-10T04:53:29.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[35961]: mon.c (mon.1) 108 : audit [DBG] 2021-02-10T04:53:29.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/4024792241' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:53:29.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:53:29.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[35961]: :53:29.056050+0000 mgr.y (mgr. 2021-02-10T04:53:29.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[35961]: 14140) 1288 : audit [DBG] from='client.25738 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:53:29.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[35961]: audit 2021-02-10T04:53:29.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[35961]: 2021-02-10T04:53:29.193988 2021-02-10T04:53:29.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1289 : audit 2021-02-10T04:53:29.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:29 smithi085 conmon[35961]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:53:30.086 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:53:30.086 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:53:30.816 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:53:30.817 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:53:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:30 smithi164 conmon[29288]: cluster 2021-02-10T04:53:29.593044 2021-02-10T04:53:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:30 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2496 : cluster [DBG] osdmap e1175: 8 total, 7 up, 8 in 2021-02-10T04:53:30.924 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:30 smithi085 conmon[29319]: cluster 2021-02-10T04:53:29.593044+0000 mon.a ( 2021-02-10T04:53:30.924 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:30 smithi085 conmon[29319]: mon.0) 2496 : cluster [DBG] osdmap e1175: 8 total, 7 up, 8 in 2021-02-10T04:53:30.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:30 smithi085 conmon[35961]: cluster 2021-02-10T04:53:29.593044+0000 2021-02-10T04:53:30.924 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:30 smithi085 conmon[35961]: mon.a (mon.0) 2496 : cluster [DBG] osdmap e1175: 8 total, 7 up, 8 in 2021-02-10T04:53:30.940 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":2119,"stamp":"2021-02-10T04:53:28.587111+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":910111060,"num_objects":733,"num_object_clones":325,"num_object_copies":2199,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":227,"num_objects_misplaced":34,"num_objects_unfound":0,"num_objects_dirty":733,"num_whiteouts":12,"num_read":7425,"num_read_kb":4744987,"num_write":4664,"num_write_kb":1286010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1018,"num_bytes_recovered":371859167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36119,"ondisk_log_size":36119,"up":525,"acting":529,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":599,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2368568,"kb_used_data":1992312,"kb_used_omap":386,"kb_used_meta":375805,"kb_avail":747592648,"statfs":{"total":767960285184,"available":765534871552,"internally_reserved":0,"allocated":2040127488,"data_stored":2617539120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":396036,"internal_metadata":384824572},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2435842,"num_objects":12,"num_object_clones":-1,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":84,"num_read_kb":62,"num_write":41,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1206,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.056889"},"pg_stats":[{"pgid":"6.29","version":"0'0","reported_seq":"73","reported_epoch":"1170","state":"active+undersized","last_fresh":"2021-02-10T04:53:24.536172+0000","last_change":"2021-02-10T04:52:32.089564+0000","last_active":"2021-02-10T04:53:24.536172+0000","last_peered":"2021-02-10T04:53:24.536172+0000","last_clean":"2021-02-10T04:52:30.064708+0000","last_became_active":"2021-02-10T04:52:32.089564+0000","last_became_peered":"2021-02-10T04:52:32.089564+0000","last_unstale":"2021-02-10T04:53:24.536172+0000","last_undegraded":"2021-02-10T04:53:24.536172+0000","last_fullsized":"2021-02-10T04:52:32.077911+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:24.121684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:24.121684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":"67","reported_epoch":"1170","state":"active+clean","last_fresh":"2021-02-10T04:53:24.535514+0000","last_change":"2021-02-10T04:52:30.133105+0000","last_active":"2021-02-10T04:53:24.535514+0000","last_peered":"2021-02-10T04:53:24.535514+0000","last_clean":"2021-02-10T04:53:24.535514+0000","last_became_active":"2021-02-10T04:51:36.864926+0000","last_became_peered":"2021-02-10T04:51:36.864926+0000","last_unstale":"2021-02-10T04:53:24.535514+0000","last_undegraded":"2021-02-10T04:53:24.535514+0000","last_fullsized":"2021-02-10T04:53:24.535514+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.133043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.133043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"131'3","reported_seq":"1366","reported_epoch":"1168","state":"active+clean","last_fresh":"2021-02-10T04:53:23.013613+0000","last_change":"2021-02-10T04:51:01.185887+0000","last_active":"2021-02-10T04:53:23.013613+0000","last_peered":"2021-02-10T04:53:23.013613+0000","last_clean":"2021-02-10T04:53:23.013613+0000","last_became_active":"2021-02-10T04:29:07.145525+0000","last_became_peered":"2021-02-10T04:29:07.145525+0000","last_unstale":"2021-02-10T04:53:23.013613+0000","last_undegraded":"2021-02-10T04:53:23.013613+0000","last_fullsized":"2021-02-10T04:53:23.013613+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'3","last_scrub_stamp":"2021-02-10T04:51:01.185820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.185820+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"1132'66","reported_seq":"1260","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.555742+0000","last_change":"2021-02-10T04:51:55.430490+0000","last_active":"2021-02-10T04:53:26.555742+0000","last_peered":"2021-02-10T04:53:26.555742+0000","last_clean":"2021-02-10T04:53:26.555742+0000","last_became_active":"2021-02-10T04:25:49.002058+0000","last_became_peered":"2021-02-10T04:25:49.002058+0000","last_unstale":"2021-02-10T04:53:26.555742+0000","last_undegraded":"2021-02-10T04:53:26.555742+0000","last_fullsized":"2021-02-10T04:53:26.555742+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1132'66","last_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_deep_scrub":"1132'66","last_deep_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.430399+0000","log_size":66,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":95,"num_read_kb":78,"num_write":55,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"71","reported_epoch":"1170","state":"active+clean","last_fresh":"2021-02-10T04:53:24.537577+0000","last_change":"2021-02-10T04:51:48.990145+0000","last_active":"2021-02-10T04:53:24.537577+0000","last_peered":"2021-02-10T04:53:24.537577+0000","last_clean":"2021-02-10T04:53:24.537577+0000","last_became_active":"2021-02-10T04:51:36.875000+0000","last_became_peered":"2021-02-10T04:51:36.875000+0000","last_unstale":"2021-02-10T04:53:24.537577+0000","last_undegraded":"2021-02-10T04:53:24.537577+0000","last_fullsized":"2021-02-10T04:53:24.537577+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:51:48.990062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.990062+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"1148","reported_epoch":"1172","state":"active+undersized","last_fresh":"2021-02-10T04:53:26.555615+0000","last_change":"2021-02-10T04:52:32.077502+0000","last_active":"2021-02-10T04:53:26.555615+0000","last_peered":"2021-02-10T04:53:26.555615+0000","last_clean":"2021-02-10T04:52:30.059220+0000","last_became_active":"2021-02-10T04:52:32.077502+0000","last_became_peered":"2021-02-10T04:52:32.077502+0000","last_unstale":"2021-02-10T04:53:26.555615+0000","last_undegraded":"2021-02-10T04:53:26.555615+0000","last_fullsized":"2021-02-10T04:52:32.069476+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:48.408748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:50:48.408748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"1003'1689","reported_seq":"4589","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.540292+0000","last_change":"2021-02-10T04:52:26.031857+0000","last_active":"2021-02-10T04:53:26.540292+0000","last_peered":"2021-02-10T04:53:26.540292+0000","last_clean":"2021-02-10T04:53:26.540292+0000","last_became_active":"2021-02-10T04:52:26.030105+0000","last_became_peered":"2021-02-10T04:52:26.030105+0000","last_unstale":"2021-02-10T04:53:26.540292+0000","last_undegraded":"2021-02-10T04:53:26.540292+0000","last_fullsized":"2021-02-10T04:53:26.540292+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:46:54.814020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:54.814020+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":63260738,"num_objects":24,"num_object_clones":22,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":386,"num_read_kb":409791,"num_write":280,"num_write_kb":110558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"1049'32","reported_seq":"1139","reported_epoch":"1172","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:26.555533+0000","last_change":"2021-02-10T04:52:32.079965+0000","last_active":"2021-02-10T04:53:26.555533+0000","last_peered":"2021-02-10T04:53:26.555533+0000","last_clean":"2021-02-10T04:45:56.690905+0000","last_became_active":"2021-02-10T04:52:32.079965+0000","last_became_peered":"2021-02-10T04:52:32.079965+0000","last_unstale":"2021-02-10T04:53:26.555533+0000","last_undegraded":"2021-02-10T04:52:32.069399+0000","last_fullsized":"2021-02-10T04:52:32.069289+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":701,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'32","last_scrub_stamp":"2021-02-10T04:51:51.382040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.382040+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":61,"num_read_kb":44,"num_write":33,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":894,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":10}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"821","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.548281+0000","last_change":"2021-02-10T04:51:49.502917+0000","last_active":"2021-02-10T04:53:26.548281+0000","last_peered":"2021-02-10T04:53:26.548281+0000","last_clean":"2021-02-10T04:53:26.548281+0000","last_became_active":"2021-02-10T04:51:49.502576+0000","last_became_peered":"2021-02-10T04:51:49.502576+0000","last_unstale":"2021-02-10T04:53:26.548281+0000","last_undegraded":"2021-02-10T04:53:26.548281+0000","last_fullsized":"2021-02-10T04:53:26.548281+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:59.824373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:59.824373+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"67","reported_epoch":"1170","state":"active+clean","last_fresh":"2021-02-10T04:53:24.536009+0000","last_change":"2021-02-10T04:51:53.038893+0000","last_active":"2021-02-10T04:53:24.536009+0000","last_peered":"2021-02-10T04:53:24.536009+0000","last_clean":"2021-02-10T04:53:24.536009+0000","last_became_active":"2021-02-10T04:51:36.855507+0000","last_became_peered":"2021-02-10T04:51:36.855507+0000","last_unstale":"2021-02-10T04:53:24.536009+0000","last_undegraded":"2021-02-10T04:53:24.536009+0000","last_fullsized":"2021-02-10T04:53:24.536009+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:53.038834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:53.038834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"1155","reported_epoch":"1170","state":"active+undersized","last_fresh":"2021-02-10T04:53:24.535052+0000","last_change":"2021-02-10T04:52:32.089886+0000","last_active":"2021-02-10T04:53:24.535052+0000","last_peered":"2021-02-10T04:53:24.535052+0000","last_clean":"2021-02-10T04:51:27.993389+0000","last_became_active":"2021-02-10T04:52:32.089886+0000","last_became_peered":"2021-02-10T04:52:32.089886+0000","last_unstale":"2021-02-10T04:53:24.535052+0000","last_undegraded":"2021-02-10T04:53:24.535052+0000","last_fullsized":"2021-02-10T04:52:32.083725+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:09.922789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:09.922789+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.8","version":"1168'82","reported_seq":"103","reported_epoch":"1170","state":"active+clean","last_fresh":"2021-02-10T04:53:24.535842+0000","last_change":"2021-02-10T04:53:23.326363+0000","last_active":"2021-02-10T04:53:24.535842+0000","last_peered":"2021-02-10T04:53:24.535842+0000","last_clean":"2021-02-10T04:53:24.535842+0000","last_became_active":"2021-02-10T04:51:36.859215+0000","last_became_peered":"2021-02-10T04:51:36.859215+0000","last_unstale":"2021-02-10T04:53:24.535842+0000","last_undegraded":"2021-02-10T04:53:24.535842+0000","last_fullsized":"2021-02-10T04:53:24.535842+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1150'76","last_scrub_stamp":"2021-02-10T04:53:00.964499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:00.964499+0000","log_size":82,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3848803,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"1049'61","reported_seq":"1252","reported_epoch":"1168","state":"active+clean","last_fresh":"2021-02-10T04:53:21.291992+0000","last_change":"2021-02-10T04:50:54.154530+0000","last_active":"2021-02-10T04:53:21.291992+0000","last_peered":"2021-02-10T04:53:21.291992+0000","last_clean":"2021-02-10T04:53:21.291992+0000","last_became_active":"2021-02-10T04:25:49.015913+0000","last_became_peered":"2021-02-10T04:25:49.015913+0000","last_unstale":"2021-02-10T04:53:21.291992+0000","last_undegraded":"2021-02-10T04:53:21.291992+0000","last_fullsized":"2021-02-10T04:53:21.291992+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'61","last_scrub_stamp":"2021-02-10T04:50:54.154436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:50:54.154436+0000","log_size":61,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":104,"num_read_kb":84,"num_write":62,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"1150'5","reported_seq":"100","reported_epoch":"1170","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:25.456042+0000","last_change":"2021-02-10T04:52:32.093814+0000","last_active":"2021-02-10T04:53:25.456042+0000","last_peered":"2021-02-10T04:53:25.456042+0000","last_clean":"2021-02-10T04:52:30.445006+0000","last_became_active":"2021-02-10T04:52:32.093814+0000","last_became_peered":"2021-02-10T04:52:32.093814+0000","last_unstale":"2021-02-10T04:53:25.456042+0000","last_undegraded":"2021-02-10T04:52:32.079897+0000","last_fullsized":"2021-02-10T04:52:32.079762+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'4","last_scrub_stamp":"2021-02-10T04:52:01.121380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.121380+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"68","reported_epoch":"1170","state":"active+undersized","last_fresh":"2021-02-10T04:53:24.536226+0000","last_change":"2021-02-10T04:52:32.093831+0000","last_active":"2021-02-10T04:53:24.536226+0000","last_peered":"2021-02-10T04:53:24.536226+0000","last_clean":"2021-02-10T04:52:30.057710+0000","last_became_active":"2021-02-10T04:52:32.093831+0000","last_became_peered":"2021-02-10T04:52:32.093831+0000","last_unstale":"2021-02-10T04:53:24.536226+0000","last_undegraded":"2021-02-10T04:53:24.536226+0000","last_fullsized":"2021-02-10T04:52:32.081168+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:52.421793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:52.421793+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"67","reported_epoch":"1170","state":"active+clean","last_fresh":"2021-02-10T04:53:24.535603+0000","last_change":"2021-02-10T04:52:35.170221+0000","last_active":"2021-02-10T04:53:24.535603+0000","last_peered":"2021-02-10T04:53:24.535603+0000","last_clean":"2021-02-10T04:53:24.535603+0000","last_became_active":"2021-02-10T04:51:36.851957+0000","last_became_peered":"2021-02-10T04:51:36.851957+0000","last_unstale":"2021-02-10T04:53:24.535603+0000","last_undegraded":"2021-02-10T04:53:24.535603+0000","last_fullsized":"2021-02-10T04:53:24.535603+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:35.170157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:35.170157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"1003'739","reported_seq":"3677","reported_epoch":"1172","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:26.539813+0000","last_change":"2021-02-10T04:52:32.084576+0000","last_active":"2021-02-10T04:53:26.539813+0000","last_peered":"2021-02-10T04:53:26.539813+0000","last_clean":"2021-02-10T04:43:58.648858+0000","last_became_active":"2021-02-10T04:52:32.084576+0000","last_became_peered":"2021-02-10T04:52:32.084576+0000","last_unstale":"2021-02-10T04:53:26.539813+0000","last_undegraded":"2021-02-10T04:52:32.072214+0000","last_fullsized":"2021-02-10T04:52:32.072101+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:52:27.602006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:27.602006+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24059888,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":129,"num_read_kb":109544,"num_write":102,"num_write_kb":46173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"1049'61","reported_seq":"1304","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.538339+0000","last_change":"2021-02-10T04:51:37.439479+0000","last_active":"2021-02-10T04:53:25.538339+0000","last_peered":"2021-02-10T04:53:25.538339+0000","last_clean":"2021-02-10T04:53:25.538339+0000","last_became_active":"2021-02-10T04:51:36.849289+0000","last_became_peered":"2021-02-10T04:51:36.849289+0000","last_unstale":"2021-02-10T04:53:25.538339+0000","last_undegraded":"2021-02-10T04:53:25.538339+0000","last_fullsized":"2021-02-10T04:53:25.538339+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'61","last_scrub_stamp":"2021-02-10T04:50:47.662378+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:50:47.662378+0000","log_size":61,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":83,"num_read_kb":65,"num_write":50,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"1062","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.534252+0000","last_change":"2021-02-10T04:50:45.623700+0000","last_active":"2021-02-10T04:53:25.534252+0000","last_peered":"2021-02-10T04:53:25.534252+0000","last_clean":"2021-02-10T04:53:25.534252+0000","last_became_active":"2021-02-10T04:29:07.145616+0000","last_became_peered":"2021-02-10T04:29:07.145616+0000","last_unstale":"2021-02-10T04:53:25.534252+0000","last_undegraded":"2021-02-10T04:53:25.534252+0000","last_fullsized":"2021-02-10T04:53:25.534252+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:45.623640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:45.623640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"392","reported_epoch":"1172","state":"active+undersized","last_fresh":"2021-02-10T04:53:26.540807+0000","last_change":"2021-02-10T04:52:32.083894+0000","last_active":"2021-02-10T04:53:26.540807+0000","last_peered":"2021-02-10T04:53:26.540807+0000","last_clean":"2021-02-10T04:52:30.060966+0000","last_became_active":"2021-02-10T04:52:32.083894+0000","last_became_peered":"2021-02-10T04:52:32.083894+0000","last_unstale":"2021-02-10T04:53:26.540807+0000","last_undegraded":"2021-02-10T04:53:26.540807+0000","last_fullsized":"2021-02-10T04:52:32.079256+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:26.311958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.311958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"67","reported_epoch":"1170","state":"active+clean","last_fresh":"2021-02-10T04:53:24.537260+0000","last_change":"2021-02-10T04:52:02.162917+0000","last_active":"2021-02-10T04:53:24.537260+0000","last_peered":"2021-02-10T04:53:24.537260+0000","last_clean":"2021-02-10T04:53:24.537260+0000","last_became_active":"2021-02-10T04:51:36.855744+0000","last_became_peered":"2021-02-10T04:51:36.855744+0000","last_unstale":"2021-02-10T04:53:24.537260+0000","last_undegraded":"2021-02-10T04:53:24.537260+0000","last_fullsized":"2021-02-10T04:53:24.537260+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:02.162851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:13.727241+0000","last_clean_scrub_stamp":"2021-02-10T04:52:02.162851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"1063'654","reported_seq":"4337","reported_epoch":"1170","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:24.537480+0000","last_change":"2021-02-10T04:52:32.093941+0000","last_active":"2021-02-10T04:53:24.537480+0000","last_peered":"2021-02-10T04:53:24.537480+0000","last_clean":"2021-02-10T04:52:30.062769+0000","last_became_active":"2021-02-10T04:52:32.093941+0000","last_became_peered":"2021-02-10T04:52:32.093941+0000","last_unstale":"2021-02-10T04:53:24.537480+0000","last_undegraded":"2021-02-10T04:52:32.085816+0000","last_fullsized":"2021-02-10T04:52:32.085711+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:51:57.121838+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.121838+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24700630,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":171,"num_read_kb":182272,"num_write":111,"num_write_kb":33727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":26349023,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":12}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"1118","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.555822+0000","last_change":"2021-02-10T04:51:50.433056+0000","last_active":"2021-02-10T04:53:26.555822+0000","last_peered":"2021-02-10T04:53:26.555822+0000","last_clean":"2021-02-10T04:53:26.555822+0000","last_became_active":"2021-02-10T04:51:36.843165+0000","last_became_peered":"2021-02-10T04:51:36.843165+0000","last_unstale":"2021-02-10T04:53:26.555822+0000","last_undegraded":"2021-02-10T04:53:26.555822+0000","last_fullsized":"2021-02-10T04:53:26.555822+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.432990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.432990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"1137","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.545497+0000","last_change":"2021-02-10T04:52:41.041630+0000","last_active":"2021-02-10T04:53:25.545497+0000","last_peered":"2021-02-10T04:53:25.545497+0000","last_clean":"2021-02-10T04:53:25.545497+0000","last_became_active":"2021-02-10T04:51:36.851075+0000","last_became_peered":"2021-02-10T04:51:36.851075+0000","last_unstale":"2021-02-10T04:53:25.545497+0000","last_undegraded":"2021-02-10T04:53:25.545497+0000","last_fullsized":"2021-02-10T04:53:25.545497+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:41.041553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.041553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"1049'33","reported_seq":"1210","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.537279+0000","last_change":"2021-02-10T04:51:06.846957+0000","last_active":"2021-02-10T04:53:25.537279+0000","last_peered":"2021-02-10T04:53:25.537279+0000","last_clean":"2021-02-10T04:53:25.537279+0000","last_became_active":"2021-02-10T04:50:59.584833+0000","last_became_peered":"2021-02-10T04:50:59.584833+0000","last_unstale":"2021-02-10T04:53:25.537279+0000","last_undegraded":"2021-02-10T04:53:25.537279+0000","last_fullsized":"2021-02-10T04:53:25.537279+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'33","last_scrub_stamp":"2021-02-10T04:51:06.846850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:06.846850+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":43,"num_write":33,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"811","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.540100+0000","last_change":"2021-02-10T04:51:52.350764+0000","last_active":"2021-02-10T04:53:26.540100+0000","last_peered":"2021-02-10T04:53:26.540100+0000","last_clean":"2021-02-10T04:53:26.540100+0000","last_became_active":"2021-02-10T04:51:49.567784+0000","last_became_peered":"2021-02-10T04:51:49.567784+0000","last_unstale":"2021-02-10T04:53:26.540100+0000","last_undegraded":"2021-02-10T04:53:26.540100+0000","last_fullsized":"2021-02-10T04:53:26.540100+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:52.350706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.350706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"1119'4","reported_seq":"94","reported_epoch":"1170","state":"active+clean","last_fresh":"2021-02-10T04:53:24.537008+0000","last_change":"2021-02-10T04:52:27.126670+0000","last_active":"2021-02-10T04:53:24.537008+0000","last_peered":"2021-02-10T04:53:24.537008+0000","last_clean":"2021-02-10T04:53:24.537008+0000","last_became_active":"2021-02-10T04:51:36.853647+0000","last_became_peered":"2021-02-10T04:51:36.853647+0000","last_unstale":"2021-02-10T04:53:24.537008+0000","last_undegraded":"2021-02-10T04:53:24.537008+0000","last_fullsized":"2021-02-10T04:53:24.537008+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'4","last_scrub_stamp":"2021-02-10T04:52:27.126599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:27.126599+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"1158","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.542065+0000","last_change":"2021-02-10T04:52:25.745660+0000","last_active":"2021-02-10T04:53:25.542065+0000","last_peered":"2021-02-10T04:53:25.542065+0000","last_clean":"2021-02-10T04:53:25.542065+0000","last_became_active":"2021-02-10T04:51:36.849503+0000","last_became_peered":"2021-02-10T04:51:36.849503+0000","last_unstale":"2021-02-10T04:53:25.542065+0000","last_undegraded":"2021-02-10T04:53:25.542065+0000","last_fullsized":"2021-02-10T04:53:25.542065+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:25.745577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.745577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"1004'1093","reported_seq":"6212","reported_epoch":"1171","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:25.551928+0000","last_change":"2021-02-10T04:52:32.090286+0000","last_active":"2021-02-10T04:53:25.551928+0000","last_peered":"2021-02-10T04:53:25.551928+0000","last_clean":"2021-02-10T04:44:07.998404+0000","last_became_active":"2021-02-10T04:52:32.090286+0000","last_became_peered":"2021-02-10T04:52:32.090286+0000","last_unstale":"2021-02-10T04:53:25.551928+0000","last_undegraded":"2021-02-10T04:52:32.075231+0000","last_fullsized":"2021-02-10T04:52:32.075115+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39434696,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":279,"num_read_kb":311058,"num_write":185,"num_write_kb":66790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":17}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"1103","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.555328+0000","last_change":"2021-02-10T04:51:36.841978+0000","last_active":"2021-02-10T04:53:26.555328+0000","last_peered":"2021-02-10T04:53:26.555328+0000","last_clean":"2021-02-10T04:53:26.555328+0000","last_became_active":"2021-02-10T04:51:36.841582+0000","last_became_peered":"2021-02-10T04:51:36.841582+0000","last_unstale":"2021-02-10T04:53:26.555328+0000","last_undegraded":"2021-02-10T04:53:26.555328+0000","last_fullsized":"2021-02-10T04:53:26.555328+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:12.389272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:12.389272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"1049'51","reported_seq":"847","reported_epoch":"1173","state":"active+clean","last_fresh":"2021-02-10T04:53:27.560299+0000","last_change":"2021-02-10T04:53:27.560299+0000","last_active":"2021-02-10T04:53:27.560299+0000","last_peered":"2021-02-10T04:53:27.560299+0000","last_clean":"2021-02-10T04:53:27.560299+0000","last_became_active":"2021-02-10T04:53:27.559886+0000","last_became_peered":"2021-02-10T04:53:27.559886+0000","last_unstale":"2021-02-10T04:53:27.560299+0000","last_undegraded":"2021-02-10T04:53:27.560299+0000","last_fullsized":"2021-02-10T04:53:27.560299+0000","mapping_epoch":1129,"log_start":"1049'51","ondisk_log_start":"1049'51","created":55,"last_epoch_clean":1173,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'41","last_scrub_stamp":"2021-02-10T04:51:58.353660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:58.353660+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1472,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":139,"num_read_kb":106,"num_write":82,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"834","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.539559+0000","last_change":"2021-02-10T04:53:11.271557+0000","last_active":"2021-02-10T04:53:26.539559+0000","last_peered":"2021-02-10T04:53:26.539559+0000","last_clean":"2021-02-10T04:53:26.539559+0000","last_became_active":"2021-02-10T04:51:49.567883+0000","last_became_peered":"2021-02-10T04:51:49.567883+0000","last_unstale":"2021-02-10T04:53:26.539559+0000","last_undegraded":"2021-02-10T04:53:26.539559+0000","last_fullsized":"2021-02-10T04:53:26.539559+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:11.271459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.271459+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"1148'4","reported_seq":"1482","reported_epoch":"1171","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:25.542023+0000","last_change":"2021-02-10T04:52:32.088315+0000","last_active":"2021-02-10T04:53:25.542023+0000","last_peered":"2021-02-10T04:53:25.542023+0000","last_clean":"2021-02-10T04:52:30.062243+0000","last_became_active":"2021-02-10T04:52:32.088315+0000","last_became_peered":"2021-02-10T04:52:32.088315+0000","last_unstale":"2021-02-10T04:53:25.542023+0000","last_undegraded":"2021-02-10T04:52:32.069861+0000","last_fullsized":"2021-02-10T04:52:32.069742+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:51:47.855283+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.855283+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"1148","reported_epoch":"1172","state":"active+undersized","last_fresh":"2021-02-10T04:53:26.557419+0000","last_change":"2021-02-10T04:52:32.094688+0000","last_active":"2021-02-10T04:53:26.557419+0000","last_peered":"2021-02-10T04:53:26.557419+0000","last_clean":"2021-02-10T04:52:30.059376+0000","last_became_active":"2021-02-10T04:52:32.094688+0000","last_became_peered":"2021-02-10T04:52:32.094688+0000","last_unstale":"2021-02-10T04:53:26.557419+0000","last_undegraded":"2021-02-10T04:53:26.557419+0000","last_fullsized":"2021-02-10T04:52:32.079768+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:10.176878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:50:10.176878+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"1002'1090","reported_seq":"5362","reported_epoch":"1168","state":"active+clean","last_fresh":"2021-02-10T04:53:21.293022+0000","last_change":"2021-02-10T04:52:26.030198+0000","last_active":"2021-02-10T04:53:21.293022+0000","last_peered":"2021-02-10T04:53:21.293022+0000","last_clean":"2021-02-10T04:53:21.293022+0000","last_became_active":"2021-02-10T04:52:26.028182+0000","last_became_peered":"2021-02-10T04:52:26.028182+0000","last_unstale":"2021-02-10T04:53:21.293022+0000","last_undegraded":"2021-02-10T04:53:21.293022+0000","last_fullsized":"2021-02-10T04:53:21.293022+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:51:08.220472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.220472+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24410646,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":245,"num_read_kb":276168,"num_write":175,"num_write_kb":64960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"1049'47","reported_seq":"1225","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.541827+0000","last_change":"2021-02-10T04:53:09.002441+0000","last_active":"2021-02-10T04:53:25.541827+0000","last_peered":"2021-02-10T04:53:25.541827+0000","last_clean":"2021-02-10T04:53:25.541827+0000","last_became_active":"2021-02-10T04:25:49.011093+0000","last_became_peered":"2021-02-10T04:25:49.011093+0000","last_unstale":"2021-02-10T04:53:25.541827+0000","last_undegraded":"2021-02-10T04:53:25.541827+0000","last_fullsized":"2021-02-10T04:53:25.541827+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'47","last_scrub_stamp":"2021-02-10T04:53:09.002368+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.002368+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":68,"num_read_kb":52,"num_write":38,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"95","reported_epoch":"1172","state":"active+undersized","last_fresh":"2021-02-10T04:53:26.539639+0000","last_change":"2021-02-10T04:52:32.092388+0000","last_active":"2021-02-10T04:53:26.539639+0000","last_peered":"2021-02-10T04:53:26.539639+0000","last_clean":"2021-02-10T04:28:18.347728+0000","last_became_active":"2021-02-10T04:52:32.092388+0000","last_became_peered":"2021-02-10T04:52:32.092388+0000","last_unstale":"2021-02-10T04:53:26.539639+0000","last_undegraded":"2021-02-10T04:53:26.539639+0000","last_fullsized":"2021-02-10T04:52:32.078707+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:08.358300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:50:08.358300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"1149","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.563287+0000","last_change":"2021-02-10T04:51:57.436961+0000","last_active":"2021-02-10T04:53:26.563287+0000","last_peered":"2021-02-10T04:53:26.563287+0000","last_clean":"2021-02-10T04:53:26.563287+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:53:26.563287+0000","last_undegraded":"2021-02-10T04:53:26.563287+0000","last_fullsized":"2021-02-10T04:53:26.563287+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.436862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.436862+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"1103","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.545510+0000","last_change":"2021-02-10T04:53:02.112612+0000","last_active":"2021-02-10T04:53:25.545510+0000","last_peered":"2021-02-10T04:53:25.545510+0000","last_clean":"2021-02-10T04:53:25.545510+0000","last_became_active":"2021-02-10T04:50:59.545501+0000","last_became_peered":"2021-02-10T04:50:59.545501+0000","last_unstale":"2021-02-10T04:53:25.545510+0000","last_undegraded":"2021-02-10T04:53:25.545510+0000","last_fullsized":"2021-02-10T04:53:25.545510+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:02.112533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:02.112533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"1001'1165","reported_seq":"4332","reported_epoch":"1168","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:21.300548+0000","last_change":"2021-02-10T04:52:32.082334+0000","last_active":"2021-02-10T04:53:21.300548+0000","last_peered":"2021-02-10T04:53:21.300548+0000","last_clean":"2021-02-10T04:52:30.068280+0000","last_became_active":"2021-02-10T04:52:32.082334+0000","last_became_peered":"2021-02-10T04:52:32.082334+0000","last_unstale":"2021-02-10T04:53:21.300548+0000","last_undegraded":"2021-02-10T04:52:32.079317+0000","last_fullsized":"2021-02-10T04:52:32.078663+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29601845,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":286,"num_read_kb":299529,"num_write":200,"num_write_kb":89130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":12}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"1113'5","reported_seq":"1377","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.544775+0000","last_change":"2021-02-10T04:52:34.047474+0000","last_active":"2021-02-10T04:53:25.544775+0000","last_peered":"2021-02-10T04:53:25.544775+0000","last_clean":"2021-02-10T04:53:25.544775+0000","last_became_active":"2021-02-10T04:50:59.569578+0000","last_became_peered":"2021-02-10T04:50:59.569578+0000","last_unstale":"2021-02-10T04:53:25.544775+0000","last_undegraded":"2021-02-10T04:53:25.544775+0000","last_fullsized":"2021-02-10T04:53:25.544775+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1113'5","last_scrub_stamp":"2021-02-10T04:52:34.047358+0000","last_deep_scrub":"158'3","last_deep_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_clean_scrub_stamp":"2021-02-10T04:52:34.047358+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"1049'54","reported_seq":"1234","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.532890+0000","last_change":"2021-02-10T04:51:48.872937+0000","last_active":"2021-02-10T04:53:25.532890+0000","last_peered":"2021-02-10T04:53:25.532890+0000","last_clean":"2021-02-10T04:53:25.532890+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:53:25.532890+0000","last_undegraded":"2021-02-10T04:53:25.532890+0000","last_fullsized":"2021-02-10T04:53:25.532890+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:51:48.872839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.872839+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":93,"num_read_kb":73,"num_write":54,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"1219","reported_epoch":"1168","state":"active+clean","last_fresh":"2021-02-10T04:53:21.300996+0000","last_change":"2021-02-10T04:53:05.984747+0000","last_active":"2021-02-10T04:53:21.300996+0000","last_peered":"2021-02-10T04:53:21.300996+0000","last_clean":"2021-02-10T04:53:21.300996+0000","last_became_active":"2021-02-10T04:51:36.869931+0000","last_became_peered":"2021-02-10T04:51:36.869931+0000","last_unstale":"2021-02-10T04:53:21.300996+0000","last_undegraded":"2021-02-10T04:53:21.300996+0000","last_fullsized":"2021-02-10T04:53:21.300996+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:05.984659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:05.984659+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"1097","reported_epoch":"1168","state":"active+clean","last_fresh":"2021-02-10T04:53:21.293263+0000","last_change":"2021-02-10T04:53:09.005480+0000","last_active":"2021-02-10T04:53:21.293263+0000","last_peered":"2021-02-10T04:53:21.293263+0000","last_clean":"2021-02-10T04:53:21.293263+0000","last_became_active":"2021-02-10T04:51:36.869649+0000","last_became_peered":"2021-02-10T04:51:36.869649+0000","last_unstale":"2021-02-10T04:53:21.293263+0000","last_undegraded":"2021-02-10T04:53:21.293263+0000","last_fullsized":"2021-02-10T04:53:21.293263+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:09.005421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.005421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"998'1009","reported_seq":"4662","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.545047+0000","last_change":"2021-02-10T04:52:26.037227+0000","last_active":"2021-02-10T04:53:25.545047+0000","last_peered":"2021-02-10T04:53:25.545047+0000","last_clean":"2021-02-10T04:53:25.545047+0000","last_became_active":"2021-02-10T04:52:26.036723+0000","last_became_peered":"2021-02-10T04:52:26.036723+0000","last_unstale":"2021-02-10T04:53:25.545047+0000","last_undegraded":"2021-02-10T04:53:25.545047+0000","last_fullsized":"2021-02-10T04:53:25.545047+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12355206,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":119,"num_read_kb":124701,"num_write":77,"num_write_kb":25052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4145029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"1037","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.546048+0000","last_change":"2021-02-10T04:51:07.976066+0000","last_active":"2021-02-10T04:53:25.546048+0000","last_peered":"2021-02-10T04:53:25.546048+0000","last_clean":"2021-02-10T04:53:25.546048+0000","last_became_active":"2021-02-10T04:50:59.570854+0000","last_became_peered":"2021-02-10T04:50:59.570854+0000","last_unstale":"2021-02-10T04:53:25.546048+0000","last_undegraded":"2021-02-10T04:53:25.546048+0000","last_fullsized":"2021-02-10T04:53:25.546048+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:46.592377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:46:46.592377+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"1157","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.533221+0000","last_change":"2021-02-10T04:51:50.895471+0000","last_active":"2021-02-10T04:53:25.533221+0000","last_peered":"2021-02-10T04:53:25.533221+0000","last_clean":"2021-02-10T04:53:25.533221+0000","last_became_active":"2021-02-10T04:50:59.527553+0000","last_became_peered":"2021-02-10T04:50:59.527553+0000","last_unstale":"2021-02-10T04:53:25.533221+0000","last_undegraded":"2021-02-10T04:53:25.533221+0000","last_fullsized":"2021-02-10T04:53:25.533221+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.895370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.895370+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"1049'72","reported_seq":"1260","reported_epoch":"1171","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:25.545314+0000","last_change":"2021-02-10T04:52:32.090145+0000","last_active":"2021-02-10T04:53:25.545314+0000","last_peered":"2021-02-10T04:53:25.545314+0000","last_clean":"2021-02-10T04:52:30.063735+0000","last_became_active":"2021-02-10T04:52:32.090145+0000","last_became_peered":"2021-02-10T04:52:32.090145+0000","last_unstale":"2021-02-10T04:53:25.545314+0000","last_undegraded":"2021-02-10T04:52:32.073169+0000","last_fullsized":"2021-02-10T04:52:32.073062+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'72","last_scrub_stamp":"2021-02-10T04:46:30.515026+0000","last_deep_scrub":"152'48","last_deep_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_clean_scrub_stamp":"2021-02-10T04:46:30.515026+0000","log_size":72,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":105,"num_read_kb":83,"num_write":60,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":14}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"404","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.547892+0000","last_change":"2021-02-10T04:51:49.504222+0000","last_active":"2021-02-10T04:53:26.547892+0000","last_peered":"2021-02-10T04:53:26.547892+0000","last_clean":"2021-02-10T04:53:26.547892+0000","last_became_active":"2021-02-10T04:51:49.503801+0000","last_became_peered":"2021-02-10T04:51:49.503801+0000","last_unstale":"2021-02-10T04:53:26.547892+0000","last_undegraded":"2021-02-10T04:53:26.547892+0000","last_fullsized":"2021-02-10T04:53:26.547892+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:51:48.274960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.274960+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"1182","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.534003+0000","last_change":"2021-02-10T04:53:13.099288+0000","last_active":"2021-02-10T04:53:25.534003+0000","last_peered":"2021-02-10T04:53:25.534003+0000","last_clean":"2021-02-10T04:53:25.534003+0000","last_became_active":"2021-02-10T04:51:36.848614+0000","last_became_peered":"2021-02-10T04:51:36.848614+0000","last_unstale":"2021-02-10T04:53:25.534003+0000","last_undegraded":"2021-02-10T04:53:25.534003+0000","last_fullsized":"2021-02-10T04:53:25.534003+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:13.099228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:13.099228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"1002'1657","reported_seq":"5232","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.545631+0000","last_change":"2021-02-10T04:52:26.036806+0000","last_active":"2021-02-10T04:53:25.545631+0000","last_peered":"2021-02-10T04:53:25.545631+0000","last_clean":"2021-02-10T04:53:25.545631+0000","last_became_active":"2021-02-10T04:52:26.032363+0000","last_became_peered":"2021-02-10T04:52:26.032363+0000","last_unstale":"2021-02-10T04:53:25.545631+0000","last_undegraded":"2021-02-10T04:53:25.545631+0000","last_fullsized":"2021-02-10T04:53:25.545631+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21396533,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":212,"num_read_kb":218836,"num_write":159,"num_write_kb":59398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":22326014,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"1101","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.537539+0000","last_change":"2021-02-10T04:51:51.911945+0000","last_active":"2021-02-10T04:53:25.537539+0000","last_peered":"2021-02-10T04:53:25.537539+0000","last_clean":"2021-02-10T04:53:25.537539+0000","last_became_active":"2021-02-10T04:51:36.850015+0000","last_became_peered":"2021-02-10T04:51:36.850015+0000","last_unstale":"2021-02-10T04:53:25.537539+0000","last_undegraded":"2021-02-10T04:53:25.537539+0000","last_fullsized":"2021-02-10T04:53:25.537539+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.911851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.911851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"1217","reported_epoch":"1168","state":"active+undersized","last_fresh":"2021-02-10T04:53:21.295974+0000","last_change":"2021-02-10T04:52:32.089163+0000","last_active":"2021-02-10T04:53:21.295974+0000","last_peered":"2021-02-10T04:53:21.295974+0000","last_clean":"2021-02-10T04:52:30.062078+0000","last_became_active":"2021-02-10T04:52:32.089163+0000","last_became_peered":"2021-02-10T04:52:32.089163+0000","last_unstale":"2021-02-10T04:53:21.295974+0000","last_undegraded":"2021-02-10T04:53:21.295974+0000","last_fullsized":"2021-02-10T04:52:32.072413+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:58.142526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:58.142526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"1135'58","reported_seq":"1227","reported_epoch":"1172","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:26.556543+0000","last_change":"2021-02-10T04:52:32.074386+0000","last_active":"2021-02-10T04:53:26.556543+0000","last_peered":"2021-02-10T04:53:26.556543+0000","last_clean":"2021-02-10T04:52:01.474364+0000","last_became_active":"2021-02-10T04:52:32.074386+0000","last_became_peered":"2021-02-10T04:52:32.074386+0000","last_unstale":"2021-02-10T04:53:26.556543+0000","last_undegraded":"2021-02-10T04:52:32.071237+0000","last_fullsized":"2021-02-10T04:52:32.070967+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1135'58","last_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_deep_scrub":"1135'58","last_deep_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.483067+0000","log_size":58,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":87,"num_read_kb":68,"num_write":50,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":11}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"44","reported_epoch":"1171","state":"active+undersized","last_fresh":"2021-02-10T04:53:25.538006+0000","last_change":"2021-02-10T04:52:32.076740+0000","last_active":"2021-02-10T04:53:25.538006+0000","last_peered":"2021-02-10T04:53:25.538006+0000","last_clean":"2021-02-10T04:25:40.828974+0000","last_became_active":"2021-02-10T04:52:32.076740+0000","last_became_peered":"2021-02-10T04:52:32.076740+0000","last_unstale":"2021-02-10T04:53:25.538006+0000","last_undegraded":"2021-02-10T04:53:25.538006+0000","last_fullsized":"2021-02-10T04:52:32.070328+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:50:46.434629+0000","last_deep_scrub":"53'2","last_deep_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_clean_scrub_stamp":"2021-02-10T04:50:46.434629+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"851","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.539706+0000","last_change":"2021-02-10T04:52:38.278980+0000","last_active":"2021-02-10T04:53:26.539706+0000","last_peered":"2021-02-10T04:53:26.539706+0000","last_clean":"2021-02-10T04:53:26.539706+0000","last_became_active":"2021-02-10T04:51:49.496237+0000","last_became_peered":"2021-02-10T04:51:49.496237+0000","last_unstale":"2021-02-10T04:53:26.539706+0000","last_undegraded":"2021-02-10T04:53:26.539706+0000","last_fullsized":"2021-02-10T04:53:26.539706+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:38.278910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:52:38.278910+0000","last_clean_scrub_stamp":"2021-02-10T04:52:38.278910+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"1000'1071","reported_seq":"4718","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.556247+0000","last_change":"2021-02-10T04:52:26.032390+0000","last_active":"2021-02-10T04:53:26.556247+0000","last_peered":"2021-02-10T04:53:26.556247+0000","last_clean":"2021-02-10T04:53:26.556247+0000","last_became_active":"2021-02-10T04:52:26.031050+0000","last_became_peered":"2021-02-10T04:52:26.031050+0000","last_unstale":"2021-02-10T04:53:26.556247+0000","last_undegraded":"2021-02-10T04:53:26.556247+0000","last_fullsized":"2021-02-10T04:53:26.556247+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:47:14.611347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:14.611347+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23286368,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"1183","reported_epoch":"1168","state":"active+undersized","last_fresh":"2021-02-10T04:53:21.301846+0000","last_change":"2021-02-10T04:52:32.081955+0000","last_active":"2021-02-10T04:53:21.301846+0000","last_peered":"2021-02-10T04:53:21.301846+0000","last_clean":"2021-02-10T04:52:30.068208+0000","last_became_active":"2021-02-10T04:52:32.081955+0000","last_became_peered":"2021-02-10T04:52:32.081955+0000","last_unstale":"2021-02-10T04:53:21.301846+0000","last_undegraded":"2021-02-10T04:53:21.301846+0000","last_fullsized":"2021-02-10T04:52:32.077723+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_clean_scrub_stamp":"2021-02-10T04:51:00.195518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"1150","reported_epoch":"1172","state":"active+undersized","last_fresh":"2021-02-10T04:53:26.563443+0000","last_change":"2021-02-10T04:52:32.097628+0000","last_active":"2021-02-10T04:53:26.563443+0000","last_peered":"2021-02-10T04:53:26.563443+0000","last_clean":"2021-02-10T04:52:30.061275+0000","last_became_active":"2021-02-10T04:52:32.097628+0000","last_became_peered":"2021-02-10T04:52:32.097628+0000","last_unstale":"2021-02-10T04:53:26.563443+0000","last_undegraded":"2021-02-10T04:53:26.563443+0000","last_fullsized":"2021-02-10T04:52:32.086329+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:55.473347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:50:55.473347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"1049'40","reported_seq":"1217","reported_epoch":"1170","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:24.537404+0000","last_change":"2021-02-10T04:52:32.094738+0000","last_active":"2021-02-10T04:53:24.537404+0000","last_peered":"2021-02-10T04:53:24.537404+0000","last_clean":"2021-02-10T04:51:27.993802+0000","last_became_active":"2021-02-10T04:52:32.094738+0000","last_became_peered":"2021-02-10T04:52:32.094738+0000","last_unstale":"2021-02-10T04:53:24.537404+0000","last_undegraded":"2021-02-10T04:52:32.085496+0000","last_fullsized":"2021-02-10T04:52:32.085393+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'40","last_scrub_stamp":"2021-02-10T04:51:47.281929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.281929+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":52,"num_write":40,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":10}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"365","reported_epoch":"1172","state":"active+undersized","last_fresh":"2021-02-10T04:53:26.539521+0000","last_change":"2021-02-10T04:52:32.091929+0000","last_active":"2021-02-10T04:53:26.539521+0000","last_peered":"2021-02-10T04:53:26.539521+0000","last_clean":"2021-02-10T04:52:30.068091+0000","last_became_active":"2021-02-10T04:52:32.091929+0000","last_became_peered":"2021-02-10T04:52:32.091929+0000","last_unstale":"2021-02-10T04:53:26.539521+0000","last_undegraded":"2021-02-10T04:53:26.539521+0000","last_fullsized":"2021-02-10T04:52:32.078333+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:50.399350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:50.399350+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"1124","reported_epoch":"1171","state":"active+undersized","last_fresh":"2021-02-10T04:53:25.533129+0000","last_change":"2021-02-10T04:52:32.081196+0000","last_active":"2021-02-10T04:53:25.533129+0000","last_peered":"2021-02-10T04:53:25.533129+0000","last_clean":"2021-02-10T04:49:56.935460+0000","last_became_active":"2021-02-10T04:52:32.081196+0000","last_became_peered":"2021-02-10T04:52:32.081196+0000","last_unstale":"2021-02-10T04:53:25.533129+0000","last_undegraded":"2021-02-10T04:53:25.533129+0000","last_fullsized":"2021-02-10T04:52:32.068851+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"836","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.540929+0000","last_change":"2021-02-10T04:52:31.273037+0000","last_active":"2021-02-10T04:53:26.540929+0000","last_peered":"2021-02-10T04:53:26.540929+0000","last_clean":"2021-02-10T04:53:26.540929+0000","last_became_active":"2021-02-10T04:51:49.501405+0000","last_became_peered":"2021-02-10T04:51:49.501405+0000","last_unstale":"2021-02-10T04:53:26.540929+0000","last_undegraded":"2021-02-10T04:53:26.540929+0000","last_fullsized":"2021-02-10T04:53:26.540929+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:31.272971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:17.598113+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.272971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"1000'446","reported_seq":"2112","reported_epoch":"1170","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:24.537083+0000","last_change":"2021-02-10T04:52:32.087644+0000","last_active":"2021-02-10T04:53:24.537083+0000","last_peered":"2021-02-10T04:53:24.537083+0000","last_clean":"2021-02-10T04:52:30.064188+0000","last_became_active":"2021-02-10T04:52:32.087644+0000","last_became_peered":"2021-02-10T04:52:32.087644+0000","last_unstale":"2021-02-10T04:53:24.537083+0000","last_undegraded":"2021-02-10T04:52:32.081292+0000","last_fullsized":"2021-02-10T04:52:32.081161+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:51:45.964029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.964029+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19615419,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14193373,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":9}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"327","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.540207+0000","last_change":"2021-02-10T04:53:03.228260+0000","last_active":"2021-02-10T04:53:26.540207+0000","last_peered":"2021-02-10T04:53:26.540207+0000","last_clean":"2021-02-10T04:53:26.540207+0000","last_became_active":"2021-02-10T04:51:49.499252+0000","last_became_peered":"2021-02-10T04:51:49.499252+0000","last_unstale":"2021-02-10T04:53:26.540207+0000","last_undegraded":"2021-02-10T04:53:26.540207+0000","last_fullsized":"2021-02-10T04:53:26.540207+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:03.228182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.228182+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"62","reported_epoch":"1170","state":"active+clean","last_fresh":"2021-02-10T04:53:24.536774+0000","last_change":"2021-02-10T04:51:36.859470+0000","last_active":"2021-02-10T04:53:24.536774+0000","last_peered":"2021-02-10T04:53:24.536774+0000","last_clean":"2021-02-10T04:53:24.536774+0000","last_became_active":"2021-02-10T04:51:36.858966+0000","last_became_peered":"2021-02-10T04:51:36.858966+0000","last_unstale":"2021-02-10T04:53:24.536774+0000","last_undegraded":"2021-02-10T04:53:24.536774+0000","last_fullsized":"2021-02-10T04:53:24.536774+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:10.398477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:10.398477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"1049'39","reported_seq":"1207","reported_epoch":"1168","state":"active+clean","last_fresh":"2021-02-10T04:53:21.298648+0000","last_change":"2021-02-10T04:51:47.322426+0000","last_active":"2021-02-10T04:53:21.298648+0000","last_peered":"2021-02-10T04:53:21.298648+0000","last_clean":"2021-02-10T04:53:21.298648+0000","last_became_active":"2021-02-10T04:25:49.002471+0000","last_became_peered":"2021-02-10T04:25:49.002471+0000","last_unstale":"2021-02-10T04:53:21.298648+0000","last_undegraded":"2021-02-10T04:53:21.298648+0000","last_fullsized":"2021-02-10T04:53:21.298648+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'39","last_scrub_stamp":"2021-02-10T04:51:47.322331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.322331+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":51,"num_write":39,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"113","reported_epoch":"1168","state":"active+undersized","last_fresh":"2021-02-10T04:53:21.290909+0000","last_change":"2021-02-10T04:52:32.083919+0000","last_active":"2021-02-10T04:53:21.290909+0000","last_peered":"2021-02-10T04:53:21.290909+0000","last_clean":"2021-02-10T04:28:45.095539+0000","last_became_active":"2021-02-10T04:52:32.083919+0000","last_became_peered":"2021-02-10T04:52:32.083919+0000","last_unstale":"2021-02-10T04:53:21.290909+0000","last_undegraded":"2021-02-10T04:53:21.290909+0000","last_fullsized":"2021-02-10T04:52:32.067706+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_clean_scrub_stamp":"2021-02-10T04:50:05.424262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"1343","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.556434+0000","last_change":"2021-02-10T04:52:37.324122+0000","last_active":"2021-02-10T04:53:26.556434+0000","last_peered":"2021-02-10T04:53:26.556434+0000","last_clean":"2021-02-10T04:53:26.556434+0000","last_became_active":"2021-02-10T04:51:36.843553+0000","last_became_peered":"2021-02-10T04:51:36.843553+0000","last_unstale":"2021-02-10T04:53:26.556434+0000","last_undegraded":"2021-02-10T04:53:26.556434+0000","last_fullsized":"2021-02-10T04:53:26.556434+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:37.324040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:37.324040+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"1139","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.545150+0000","last_change":"2021-02-10T04:51:57.769992+0000","last_active":"2021-02-10T04:53:25.545150+0000","last_peered":"2021-02-10T04:53:25.545150+0000","last_clean":"2021-02-10T04:53:25.545150+0000","last_became_active":"2021-02-10T04:50:59.587343+0000","last_became_peered":"2021-02-10T04:50:59.587343+0000","last_unstale":"2021-02-10T04:53:25.545150+0000","last_undegraded":"2021-02-10T04:53:25.545150+0000","last_fullsized":"2021-02-10T04:53:25.545150+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"1171","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.563564+0000","last_change":"2021-02-10T04:52:31.403904+0000","last_active":"2021-02-10T04:53:26.563564+0000","last_peered":"2021-02-10T04:53:26.563564+0000","last_clean":"2021-02-10T04:53:26.563564+0000","last_became_active":"2021-02-10T04:50:59.538009+0000","last_became_peered":"2021-02-10T04:50:59.538009+0000","last_unstale":"2021-02-10T04:53:26.563564+0000","last_undegraded":"2021-02-10T04:53:26.563564+0000","last_fullsized":"2021-02-10T04:53:26.563564+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:31.403824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.403824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"1000'490","reported_seq":"3560","reported_epoch":"1172","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:26.547857+0000","last_change":"2021-02-10T04:52:32.091003+0000","last_active":"2021-02-10T04:53:26.547857+0000","last_peered":"2021-02-10T04:53:26.547857+0000","last_clean":"2021-02-10T04:45:36.401519+0000","last_became_active":"2021-02-10T04:52:32.091003+0000","last_became_peered":"2021-02-10T04:52:32.091003+0000","last_unstale":"2021-02-10T04:53:26.547857+0000","last_undegraded":"2021-02-10T04:52:32.076920+0000","last_fullsized":"2021-02-10T04:52:32.076862+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1040,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:47:02.875920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:02.875920+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19371627,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":141,"num_read_kb":148568,"num_write":85,"num_write_kb":31732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"67","reported_epoch":"1170","state":"active+clean","last_fresh":"2021-02-10T04:53:24.537728+0000","last_change":"2021-02-10T04:53:01.972991+0000","last_active":"2021-02-10T04:53:24.537728+0000","last_peered":"2021-02-10T04:53:24.537728+0000","last_clean":"2021-02-10T04:53:24.537728+0000","last_became_active":"2021-02-10T04:51:36.863976+0000","last_became_peered":"2021-02-10T04:51:36.863976+0000","last_unstale":"2021-02-10T04:53:24.537728+0000","last_undegraded":"2021-02-10T04:53:24.537728+0000","last_fullsized":"2021-02-10T04:53:24.537728+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:01.972913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.972913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"1049'37","reported_seq":"851","reported_epoch":"1172","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:26.547942+0000","last_change":"2021-02-10T04:52:32.095888+0000","last_active":"2021-02-10T04:53:26.547942+0000","last_peered":"2021-02-10T04:53:26.547942+0000","last_clean":"2021-02-10T04:52:30.062113+0000","last_became_active":"2021-02-10T04:52:32.095888+0000","last_became_peered":"2021-02-10T04:52:32.095888+0000","last_unstale":"2021-02-10T04:53:26.547942+0000","last_undegraded":"2021-02-10T04:52:32.076227+0000","last_fullsized":"2021-02-10T04:52:32.076110+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:56.431187+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.431187+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":59,"num_read_kb":43,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"68","reported_epoch":"1170","state":"active+undersized","last_fresh":"2021-02-10T04:53:24.535379+0000","last_change":"2021-02-10T04:52:32.089886+0000","last_active":"2021-02-10T04:53:24.535379+0000","last_peered":"2021-02-10T04:53:24.535379+0000","last_clean":"2021-02-10T04:52:30.064491+0000","last_became_active":"2021-02-10T04:52:32.089886+0000","last_became_peered":"2021-02-10T04:52:32.089886+0000","last_unstale":"2021-02-10T04:53:24.535379+0000","last_undegraded":"2021-02-10T04:53:24.535379+0000","last_fullsized":"2021-02-10T04:52:32.078850+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:04.591695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:04.591695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"922","reported_epoch":"1173","state":"active+clean","last_fresh":"2021-02-10T04:53:27.550790+0000","last_change":"2021-02-10T04:52:25.358146+0000","last_active":"2021-02-10T04:53:27.550790+0000","last_peered":"2021-02-10T04:53:27.550790+0000","last_clean":"2021-02-10T04:53:27.550790+0000","last_became_active":"2021-02-10T04:51:49.496092+0000","last_became_peered":"2021-02-10T04:51:49.496092+0000","last_unstale":"2021-02-10T04:53:27.550790+0000","last_undegraded":"2021-02-10T04:53:27.550790+0000","last_fullsized":"2021-02-10T04:53:27.550790+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":530,"num_read_kb":525,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"1094","reported_epoch":"1168","state":"active+clean","last_fresh":"2021-02-10T04:53:21.300282+0000","last_change":"2021-02-10T04:52:40.066424+0000","last_active":"2021-02-10T04:53:21.300282+0000","last_peered":"2021-02-10T04:53:21.300282+0000","last_clean":"2021-02-10T04:53:21.300282+0000","last_became_active":"2021-02-10T04:50:59.529649+0000","last_became_peered":"2021-02-10T04:50:59.529649+0000","last_unstale":"2021-02-10T04:53:21.300282+0000","last_undegraded":"2021-02-10T04:53:21.300282+0000","last_fullsized":"2021-02-10T04:53:21.300282+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:40.066354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:52:40.066354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.4","version":"1000'620","reported_seq":"4178","reported_epoch":"1171","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:25.537070+0000","last_change":"2021-02-10T04:52:32.082414+0000","last_active":"2021-02-10T04:53:25.537070+0000","last_peered":"2021-02-10T04:53:25.537070+0000","last_clean":"2021-02-10T04:43:51.960482+0000","last_became_active":"2021-02-10T04:52:32.082414+0000","last_became_peered":"2021-02-10T04:52:32.082414+0000","last_unstale":"2021-02-10T04:53:25.537070+0000","last_undegraded":"2021-02-10T04:52:32.068324+0000","last_fullsized":"2021-02-10T04:52:32.068200+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:51:54.444287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:54.444287+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14265195,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":144,"num_read_kb":128482,"num_write":129,"num_write_kb":53443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"126","reported_epoch":"1171","state":"active+undersized","last_fresh":"2021-02-10T04:53:25.533658+0000","last_change":"2021-02-10T04:52:32.076404+0000","last_active":"2021-02-10T04:53:25.533658+0000","last_peered":"2021-02-10T04:53:25.533658+0000","last_clean":"2021-02-10T04:28:47.053193+0000","last_became_active":"2021-02-10T04:52:32.076404+0000","last_became_peered":"2021-02-10T04:52:32.076404+0000","last_unstale":"2021-02-10T04:53:25.533658+0000","last_undegraded":"2021-02-10T04:53:25.533658+0000","last_fullsized":"2021-02-10T04:52:32.068279+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:14.955544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:47:14.955544+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"34","reported_epoch":"1172","state":"active+undersized","last_fresh":"2021-02-10T04:53:26.548390+0000","last_change":"2021-02-10T04:52:32.084350+0000","last_active":"2021-02-10T04:53:26.548390+0000","last_peered":"2021-02-10T04:53:26.548390+0000","last_clean":"2021-02-10T04:25:49.999907+0000","last_became_active":"2021-02-10T04:52:32.084350+0000","last_became_peered":"2021-02-10T04:52:32.084350+0000","last_unstale":"2021-02-10T04:53:26.548390+0000","last_undegraded":"2021-02-10T04:53:26.548390+0000","last_fullsized":"2021-02-10T04:52:32.073004+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:27.908221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:47:27.908221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"1049'55","reported_seq":"807","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.539863+0000","last_change":"2021-02-10T04:52:30.277009+0000","last_active":"2021-02-10T04:53:26.539863+0000","last_peered":"2021-02-10T04:53:26.539863+0000","last_clean":"2021-02-10T04:53:26.539863+0000","last_became_active":"2021-02-10T04:51:49.501622+0000","last_became_peered":"2021-02-10T04:51:49.501622+0000","last_unstale":"2021-02-10T04:53:26.539863+0000","last_undegraded":"2021-02-10T04:53:26.539863+0000","last_fullsized":"2021-02-10T04:53:26.539863+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'55","last_scrub_stamp":"2021-02-10T04:52:30.276938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.276938+0000","log_size":55,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":98,"num_read_kb":74,"num_write":58,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"1167","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.555822+0000","last_change":"2021-02-10T04:51:59.420149+0000","last_active":"2021-02-10T04:53:26.555822+0000","last_peered":"2021-02-10T04:53:26.555822+0000","last_clean":"2021-02-10T04:53:26.555822+0000","last_became_active":"2021-02-10T04:28:21.125541+0000","last_became_peered":"2021-02-10T04:28:21.125541+0000","last_unstale":"2021-02-10T04:53:26.555822+0000","last_undegraded":"2021-02-10T04:53:26.555822+0000","last_fullsized":"2021-02-10T04:53:26.555822+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.420086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.420086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"1132","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.546860+0000","last_change":"2021-02-10T04:53:12.187225+0000","last_active":"2021-02-10T04:53:25.546860+0000","last_peered":"2021-02-10T04:53:25.546860+0000","last_clean":"2021-02-10T04:53:25.546860+0000","last_became_active":"2021-02-10T04:50:59.555689+0000","last_became_peered":"2021-02-10T04:50:59.555689+0000","last_unstale":"2021-02-10T04:53:25.546860+0000","last_undegraded":"2021-02-10T04:53:25.546860+0000","last_fullsized":"2021-02-10T04:53:25.546860+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.187154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.187154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"1002'2042","reported_seq":"9391","reported_epoch":"1168","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:21.300868+0000","last_change":"2021-02-10T04:52:32.083034+0000","last_active":"2021-02-10T04:53:21.300868+0000","last_peered":"2021-02-10T04:53:21.300868+0000","last_clean":"2021-02-10T04:52:30.068473+0000","last_became_active":"2021-02-10T04:52:32.083034+0000","last_became_peered":"2021-02-10T04:52:32.083034+0000","last_unstale":"2021-02-10T04:53:21.300868+0000","last_undegraded":"2021-02-10T04:52:32.077334+0000","last_fullsized":"2021-02-10T04:52:32.077220+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:46:57.054230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:57.054230+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":85678898,"num_objects":35,"num_object_clones":29,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":35}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"1109","reported_epoch":"1172","state":"active+undersized","last_fresh":"2021-02-10T04:53:26.555417+0000","last_change":"2021-02-10T04:52:32.094777+0000","last_active":"2021-02-10T04:53:26.555417+0000","last_peered":"2021-02-10T04:53:26.555417+0000","last_clean":"2021-02-10T04:52:30.061448+0000","last_became_active":"2021-02-10T04:52:32.094777+0000","last_became_peered":"2021-02-10T04:52:32.094777+0000","last_unstale":"2021-02-10T04:53:26.555417+0000","last_undegraded":"2021-02-10T04:53:26.555417+0000","last_fullsized":"2021-02-10T04:52:32.069238+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:52.341527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:50:52.341527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"1148'3","reported_seq":"50","reported_epoch":"1168","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:23.013287+0000","last_change":"2021-02-10T04:52:32.089249+0000","last_active":"2021-02-10T04:53:23.013287+0000","last_peered":"2021-02-10T04:53:23.013287+0000","last_clean":"2021-02-10T04:25:51.042416+0000","last_became_active":"2021-02-10T04:52:32.089249+0000","last_became_peered":"2021-02-10T04:52:32.089249+0000","last_unstale":"2021-02-10T04:53:23.013287+0000","last_undegraded":"2021-02-10T04:52:32.073980+0000","last_fullsized":"2021-02-10T04:52:32.073877+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:54.466259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:54.466259+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"1049'54","reported_seq":"856","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.540233+0000","last_change":"2021-02-10T04:53:01.304693+0000","last_active":"2021-02-10T04:53:26.540233+0000","last_peered":"2021-02-10T04:53:26.540233+0000","last_clean":"2021-02-10T04:53:26.540233+0000","last_became_active":"2021-02-10T04:51:49.498066+0000","last_became_peered":"2021-02-10T04:51:49.498066+0000","last_unstale":"2021-02-10T04:53:26.540233+0000","last_undegraded":"2021-02-10T04:53:26.540233+0000","last_fullsized":"2021-02-10T04:53:26.540233+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:53:01.304626+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.304626+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":74,"num_read_kb":56,"num_write":42,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"423","reported_epoch":"1172","state":"active+undersized","last_fresh":"2021-02-10T04:53:26.539957+0000","last_change":"2021-02-10T04:52:32.091144+0000","last_active":"2021-02-10T04:53:26.539957+0000","last_peered":"2021-02-10T04:53:26.539957+0000","last_clean":"2021-02-10T04:52:30.060819+0000","last_became_active":"2021-02-10T04:52:32.091144+0000","last_became_peered":"2021-02-10T04:52:32.091144+0000","last_unstale":"2021-02-10T04:53:26.539957+0000","last_undegraded":"2021-02-10T04:53:26.539957+0000","last_fullsized":"2021-02-10T04:52:32.079969+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:49.447961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:49.447961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"1153","reported_epoch":"1171","state":"active+undersized","last_fresh":"2021-02-10T04:53:25.544813+0000","last_change":"2021-02-10T04:52:32.088766+0000","last_active":"2021-02-10T04:53:25.544813+0000","last_peered":"2021-02-10T04:53:25.544813+0000","last_clean":"2021-02-10T04:51:27.993073+0000","last_became_active":"2021-02-10T04:52:32.088766+0000","last_became_peered":"2021-02-10T04:52:32.088766+0000","last_unstale":"2021-02-10T04:53:25.544813+0000","last_undegraded":"2021-02-10T04:53:25.544813+0000","last_fullsized":"2021-02-10T04:52:32.071883+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:07.914061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:07.914061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"293'77","reported_seq":"2952","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.532516+0000","last_change":"2021-02-10T04:52:26.025283+0000","last_active":"2021-02-10T04:53:25.532516+0000","last_peered":"2021-02-10T04:53:25.532516+0000","last_clean":"2021-02-10T04:53:25.532516+0000","last_became_active":"2021-02-10T04:52:26.024812+0000","last_became_peered":"2021-02-10T04:52:26.024812+0000","last_unstale":"2021-02-10T04:53:25.532516+0000","last_undegraded":"2021-02-10T04:53:25.532516+0000","last_fullsized":"2021-02-10T04:53:25.532516+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1886585,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":18587,"num_write":9,"num_write_kb":3315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"1182","reported_epoch":"1168","state":"active+clean","last_fresh":"2021-02-10T04:53:21.300771+0000","last_change":"2021-02-10T04:53:07.025068+0000","last_active":"2021-02-10T04:53:21.300771+0000","last_peered":"2021-02-10T04:53:21.300771+0000","last_clean":"2021-02-10T04:53:21.300771+0000","last_became_active":"2021-02-10T04:50:59.589417+0000","last_became_peered":"2021-02-10T04:50:59.589417+0000","last_unstale":"2021-02-10T04:53:21.300771+0000","last_undegraded":"2021-02-10T04:53:21.300771+0000","last_fullsized":"2021-02-10T04:53:21.300771+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.024980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.024980+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"68","reported_epoch":"1170","state":"active+undersized","last_fresh":"2021-02-10T04:53:24.535243+0000","last_change":"2021-02-10T04:52:32.082945+0000","last_active":"2021-02-10T04:53:24.535243+0000","last_peered":"2021-02-10T04:53:24.535243+0000","last_clean":"2021-02-10T04:52:30.056402+0000","last_became_active":"2021-02-10T04:52:32.082945+0000","last_became_peered":"2021-02-10T04:52:32.082945+0000","last_unstale":"2021-02-10T04:53:24.535243+0000","last_undegraded":"2021-02-10T04:53:24.535243+0000","last_fullsized":"2021-02-10T04:52:32.077773+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.609500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.609500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"1048'39","reported_seq":"1262","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.532976+0000","last_change":"2021-02-10T04:50:44.624567+0000","last_active":"2021-02-10T04:53:25.532976+0000","last_peered":"2021-02-10T04:53:25.532976+0000","last_clean":"2021-02-10T04:53:25.532976+0000","last_became_active":"2021-02-10T04:39:02.560739+0000","last_became_peered":"2021-02-10T04:39:02.560739+0000","last_unstale":"2021-02-10T04:53:25.532976+0000","last_undegraded":"2021-02-10T04:53:25.532976+0000","last_fullsized":"2021-02-10T04:53:25.532976+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'39","last_scrub_stamp":"2021-02-10T04:50:44.624501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:50:44.624501+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":76,"num_read_kb":59,"num_write":41,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"1277","reported_epoch":"1171","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:25.542410+0000","last_change":"2021-02-10T04:52:32.084961+0000","last_active":"2021-02-10T04:53:25.542410+0000","last_peered":"2021-02-10T04:53:25.542410+0000","last_clean":"2021-02-10T04:52:30.813009+0000","last_became_active":"2021-02-10T04:52:32.084961+0000","last_became_peered":"2021-02-10T04:52:32.084961+0000","last_unstale":"2021-02-10T04:53:25.542410+0000","last_undegraded":"2021-02-10T04:52:32.070468+0000","last_fullsized":"2021-02-10T04:52:32.070358+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:47:08.784593+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:47:08.784593+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"1134","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.546227+0000","last_change":"2021-02-10T04:52:30.019225+0000","last_active":"2021-02-10T04:53:25.546227+0000","last_peered":"2021-02-10T04:53:25.546227+0000","last_clean":"2021-02-10T04:53:25.546227+0000","last_became_active":"2021-02-10T04:50:59.561436+0000","last_became_peered":"2021-02-10T04:50:59.561436+0000","last_unstale":"2021-02-10T04:53:25.546227+0000","last_undegraded":"2021-02-10T04:53:25.546227+0000","last_fullsized":"2021-02-10T04:53:25.546227+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.019102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.019102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"1001'1528","reported_seq":"6119","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.544952+0000","last_change":"2021-02-10T04:52:26.036366+0000","last_active":"2021-02-10T04:53:25.544952+0000","last_peered":"2021-02-10T04:53:25.544952+0000","last_clean":"2021-02-10T04:53:25.544952+0000","last_became_active":"2021-02-10T04:52:26.034028+0000","last_became_peered":"2021-02-10T04:52:26.034028+0000","last_unstale":"2021-02-10T04:53:25.544952+0000","last_undegraded":"2021-02-10T04:53:25.544952+0000","last_fullsized":"2021-02-10T04:53:25.544952+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46944136,"num_objects":16,"num_object_clones":15,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":358,"num_read_kb":351934,"num_write":286,"num_write_kb":111355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":58968832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.15","version":"1000'446","reported_seq":"2109","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.538208+0000","last_change":"2021-02-10T04:53:11.051124+0000","last_active":"2021-02-10T04:53:25.538208+0000","last_peered":"2021-02-10T04:53:25.538208+0000","last_clean":"2021-02-10T04:53:25.538208+0000","last_became_active":"2021-02-10T04:51:40.510899+0000","last_became_peered":"2021-02-10T04:51:40.510899+0000","last_unstale":"2021-02-10T04:53:25.538208+0000","last_undegraded":"2021-02-10T04:53:25.538208+0000","last_fullsized":"2021-02-10T04:53:25.538208+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:53:11.051031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.051031+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24896285,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":18983408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"1048'40","reported_seq":"56","reported_epoch":"1170","state":"active+clean+remapped","last_fresh":"2021-02-10T04:53:24.537122+0000","last_change":"2021-02-10T04:53:22.691257+0000","last_active":"2021-02-10T04:53:24.537122+0000","last_peered":"2021-02-10T04:53:24.537122+0000","last_clean":"2021-02-10T04:53:24.537122+0000","last_became_active":"2021-02-10T04:53:21.308544+0000","last_became_peered":"2021-02-10T04:53:21.308544+0000","last_unstale":"2021-02-10T04:53:24.537122+0000","last_undegraded":"2021-02-10T04:53:24.537122+0000","last_fullsized":"2021-02-10T04:53:24.537122+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'40","last_scrub_stamp":"2021-02-10T04:47:29.965140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:47:29.965140+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":79,"num_read_kb":57,"num_write":41,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1206,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1,3],"avail_no_missing":["2","1","3"],"object_location_counts":[{"shards":"1,2,3","objects":13}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"72","reported_epoch":"1170","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:24.535701+0000","last_change":"2021-02-10T04:52:32.094875+0000","last_active":"2021-02-10T04:53:24.535701+0000","last_peered":"2021-02-10T04:53:24.535701+0000","last_clean":"2021-02-10T04:52:30.056787+0000","last_became_active":"2021-02-10T04:52:32.094875+0000","last_became_peered":"2021-02-10T04:52:32.094875+0000","last_unstale":"2021-02-10T04:53:24.535701+0000","last_undegraded":"2021-02-10T04:52:32.083818+0000","last_fullsized":"2021-02-10T04:52:32.083707+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:51:08.639784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.639784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"1103","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.552132+0000","last_change":"2021-02-10T04:51:09.475150+0000","last_active":"2021-02-10T04:53:25.552132+0000","last_peered":"2021-02-10T04:53:25.552132+0000","last_clean":"2021-02-10T04:53:25.552132+0000","last_became_active":"2021-02-10T04:50:59.585917+0000","last_became_peered":"2021-02-10T04:50:59.585917+0000","last_unstale":"2021-02-10T04:53:25.552132+0000","last_undegraded":"2021-02-10T04:53:25.552132+0000","last_fullsized":"2021-02-10T04:53:25.552132+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:09.475063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:09.475063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"67","reported_epoch":"1170","state":"active+clean","last_fresh":"2021-02-10T04:53:24.536834+0000","last_change":"2021-02-10T04:51:44.970615+0000","last_active":"2021-02-10T04:53:24.536834+0000","last_peered":"2021-02-10T04:53:24.536834+0000","last_clean":"2021-02-10T04:53:24.536834+0000","last_became_active":"2021-02-10T04:51:36.859707+0000","last_became_peered":"2021-02-10T04:51:36.859707+0000","last_unstale":"2021-02-10T04:53:24.536834+0000","last_undegraded":"2021-02-10T04:53:24.536834+0000","last_fullsized":"2021-02-10T04:53:24.536834+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:44.970511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.970511+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"1000'620","reported_seq":"4453","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.545597+0000","last_change":"2021-02-10T04:51:40.511426+0000","last_active":"2021-02-10T04:53:25.545597+0000","last_peered":"2021-02-10T04:53:25.545597+0000","last_clean":"2021-02-10T04:53:25.545597+0000","last_became_active":"2021-02-10T04:51:40.510114+0000","last_became_peered":"2021-02-10T04:51:40.510114+0000","last_unstale":"2021-02-10T04:53:25.545597+0000","last_undegraded":"2021-02-10T04:53:25.545597+0000","last_fullsized":"2021-02-10T04:53:25.545597+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:47:24.897680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:24.897680+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14265195,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":144,"num_read_kb":128483,"num_write":129,"num_write_kb":53444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":39998538,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"1063'30","reported_seq":"1148","reported_epoch":"1172","state":"active+clean+remapped","last_fresh":"2021-02-10T04:53:26.548367+0000","last_change":"2021-02-10T04:53:21.312523+0000","last_active":"2021-02-10T04:53:26.548367+0000","last_peered":"2021-02-10T04:53:26.548367+0000","last_clean":"2021-02-10T04:53:26.548367+0000","last_became_active":"2021-02-10T04:53:21.302364+0000","last_became_peered":"2021-02-10T04:53:21.302364+0000","last_unstale":"2021-02-10T04:53:26.548367+0000","last_undegraded":"2021-02-10T04:53:26.548367+0000","last_fullsized":"2021-02-10T04:53:26.548367+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"1063'30","last_scrub_stamp":"2021-02-10T04:52:43.102165+0000","last_deep_scrub":"1063'30","last_deep_scrub_stamp":"2021-02-10T04:47:05.589582+0000","last_clean_scrub_stamp":"2021-02-10T04:52:43.102165+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5,3],"avail_no_missing":["1","3","5"],"object_location_counts":[{"shards":"1,3,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"1156","reported_epoch":"1168","state":"active+clean","last_fresh":"2021-02-10T04:53:21.291235+0000","last_change":"2021-02-10T04:52:41.059269+0000","last_active":"2021-02-10T04:53:21.291235+0000","last_peered":"2021-02-10T04:53:21.291235+0000","last_clean":"2021-02-10T04:53:21.291235+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:53:21.291235+0000","last_undegraded":"2021-02-10T04:53:21.291235+0000","last_fullsized":"2021-02-10T04:53:21.291235+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:52:41.059159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.059159+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"1164","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.562886+0000","last_change":"2021-02-10T04:53:12.332890+0000","last_active":"2021-02-10T04:53:26.562886+0000","last_peered":"2021-02-10T04:53:26.562886+0000","last_clean":"2021-02-10T04:53:26.562886+0000","last_became_active":"2021-02-10T04:50:59.540375+0000","last_became_peered":"2021-02-10T04:50:59.540375+0000","last_unstale":"2021-02-10T04:53:26.562886+0000","last_undegraded":"2021-02-10T04:53:26.562886+0000","last_fullsized":"2021-02-10T04:53:26.562886+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.332233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.332233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"1065","reported_epoch":"1168","state":"active+clean","last_fresh":"2021-02-10T04:53:21.300616+0000","last_change":"2021-02-10T04:50:45.362633+0000","last_active":"2021-02-10T04:53:21.300616+0000","last_peered":"2021-02-10T04:53:21.300616+0000","last_clean":"2021-02-10T04:53:21.300616+0000","last_became_active":"2021-02-10T04:29:07.147249+0000","last_became_peered":"2021-02-10T04:29:07.147249+0000","last_unstale":"2021-02-10T04:53:21.300616+0000","last_undegraded":"2021-02-10T04:53:21.300616+0000","last_fullsized":"2021-02-10T04:53:21.300616+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:45.362537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:45.362537+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.17","version":"1002'2042","reported_seq":"9428","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.562921+0000","last_change":"2021-02-10T04:52:41.286103+0000","last_active":"2021-02-10T04:53:26.562921+0000","last_peered":"2021-02-10T04:53:26.562921+0000","last_clean":"2021-02-10T04:53:26.562921+0000","last_became_active":"2021-02-10T04:51:41.509707+0000","last_became_peered":"2021-02-10T04:51:41.509707+0000","last_unstale":"2021-02-10T04:53:26.562921+0000","last_undegraded":"2021-02-10T04:53:26.562921+0000","last_fullsized":"2021-02-10T04:53:26.562921+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1124,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:52:41.286006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.286006+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6921254,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":5403183,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"1049'46","reported_seq":"1327","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.545955+0000","last_change":"2021-02-10T04:52:35.091494+0000","last_active":"2021-02-10T04:53:25.545955+0000","last_peered":"2021-02-10T04:53:25.545955+0000","last_clean":"2021-02-10T04:53:25.545955+0000","last_became_active":"2021-02-10T04:50:59.569915+0000","last_became_peered":"2021-02-10T04:50:59.569915+0000","last_unstale":"2021-02-10T04:53:25.545955+0000","last_undegraded":"2021-02-10T04:53:25.545955+0000","last_fullsized":"2021-02-10T04:53:25.545955+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'46","last_scrub_stamp":"2021-02-10T04:52:35.091374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:35.091374+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":65,"num_write":46,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"84'7","reported_seq":"1188","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.555744+0000","last_change":"2021-02-10T04:51:52.498251+0000","last_active":"2021-02-10T04:53:26.555744+0000","last_peered":"2021-02-10T04:53:26.555744+0000","last_clean":"2021-02-10T04:53:26.555744+0000","last_became_active":"2021-02-10T04:50:59.575924+0000","last_became_peered":"2021-02-10T04:50:59.575924+0000","last_unstale":"2021-02-10T04:53:26.555744+0000","last_undegraded":"2021-02-10T04:53:26.555744+0000","last_fullsized":"2021-02-10T04:53:26.555744+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'7","last_scrub_stamp":"2021-02-10T04:51:52.498158+0000","last_deep_scrub":"84'7","last_deep_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.498158+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"1148'3","reported_seq":"1484","reported_epoch":"1173","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:27.546488+0000","last_change":"2021-02-10T04:52:32.094670+0000","last_active":"2021-02-10T04:53:27.546488+0000","last_peered":"2021-02-10T04:53:27.546488+0000","last_clean":"2021-02-10T04:52:30.059221+0000","last_became_active":"2021-02-10T04:52:32.094670+0000","last_became_peered":"2021-02-10T04:52:32.094670+0000","last_unstale":"2021-02-10T04:53:27.546488+0000","last_undegraded":"2021-02-10T04:52:32.078919+0000","last_fullsized":"2021-02-10T04:52:32.078798+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:47.407557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:47.407557+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"1119","reported_epoch":"1171","state":"active+undersized","last_fresh":"2021-02-10T04:53:25.546949+0000","last_change":"2021-02-10T04:52:32.085745+0000","last_active":"2021-02-10T04:53:25.546949+0000","last_peered":"2021-02-10T04:53:25.546949+0000","last_clean":"2021-02-10T04:49:56.939276+0000","last_became_active":"2021-02-10T04:52:32.085745+0000","last_became_peered":"2021-02-10T04:52:32.085745+0000","last_unstale":"2021-02-10T04:53:25.546949+0000","last_undegraded":"2021-02-10T04:53:25.546949+0000","last_fullsized":"2021-02-10T04:52:32.076058+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:23.566027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_clean_scrub_stamp":"2021-02-10T04:52:23.566027+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.16","version":"1000'490","reported_seq":"3679","reported_epoch":"1171","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:25.542114+0000","last_change":"2021-02-10T04:52:32.077228+0000","last_active":"2021-02-10T04:53:25.542114+0000","last_peered":"2021-02-10T04:53:25.542114+0000","last_clean":"2021-02-10T04:52:30.062406+0000","last_became_active":"2021-02-10T04:52:32.077228+0000","last_became_peered":"2021-02-10T04:52:32.077228+0000","last_unstale":"2021-02-10T04:53:25.542114+0000","last_undegraded":"2021-02-10T04:52:32.071522+0000","last_fullsized":"2021-02-10T04:52:32.071453+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1013,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:50:03.756248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:50:03.756248+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16154790,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"1112'40","reported_seq":"1232","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.557056+0000","last_change":"2021-02-10T04:53:10.276449+0000","last_active":"2021-02-10T04:53:26.557056+0000","last_peered":"2021-02-10T04:53:26.557056+0000","last_clean":"2021-02-10T04:53:26.557056+0000","last_became_active":"2021-02-10T04:50:59.539529+0000","last_became_peered":"2021-02-10T04:50:59.539529+0000","last_unstale":"2021-02-10T04:53:26.557056+0000","last_undegraded":"2021-02-10T04:53:26.557056+0000","last_fullsized":"2021-02-10T04:53:26.557056+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1112'40","last_scrub_stamp":"2021-02-10T04:53:10.276358+0000","last_deep_scrub":"1112'40","last_deep_scrub_stamp":"2021-02-10T04:51:01.450050+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.276358+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":59,"num_read_kb":44,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"1136","reported_epoch":"1171","state":"active+undersized","last_fresh":"2021-02-10T04:53:25.534071+0000","last_change":"2021-02-10T04:52:32.080868+0000","last_active":"2021-02-10T04:53:25.534071+0000","last_peered":"2021-02-10T04:53:25.534071+0000","last_clean":"2021-02-10T04:49:56.937464+0000","last_became_active":"2021-02-10T04:52:32.080868+0000","last_became_peered":"2021-02-10T04:52:32.080868+0000","last_unstale":"2021-02-10T04:53:25.534071+0000","last_undegraded":"2021-02-10T04:53:25.534071+0000","last_fullsized":"2021-02-10T04:52:32.068376+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:23.904210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:47:23.904210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"1119'6","reported_seq":"1406","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.546247+0000","last_change":"2021-02-10T04:52:40.038646+0000","last_active":"2021-02-10T04:53:25.546247+0000","last_peered":"2021-02-10T04:53:25.546247+0000","last_clean":"2021-02-10T04:53:25.546247+0000","last_became_active":"2021-02-10T04:51:36.869733+0000","last_became_peered":"2021-02-10T04:51:36.869733+0000","last_unstale":"2021-02-10T04:53:25.546247+0000","last_undegraded":"2021-02-10T04:53:25.546247+0000","last_fullsized":"2021-02-10T04:53:25.546247+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'6","last_scrub_stamp":"2021-02-10T04:52:40.038554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:40.038554+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"1158","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.543081+0000","last_change":"2021-02-10T04:51:36.849933+0000","last_active":"2021-02-10T04:53:25.543081+0000","last_peered":"2021-02-10T04:53:25.543081+0000","last_clean":"2021-02-10T04:53:25.543081+0000","last_became_active":"2021-02-10T04:51:36.849184+0000","last_became_peered":"2021-02-10T04:51:36.849184+0000","last_unstale":"2021-02-10T04:53:25.543081+0000","last_undegraded":"2021-02-10T04:53:25.543081+0000","last_fullsized":"2021-02-10T04:53:25.543081+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:11.779779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:11.779779+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"1032'89","reported_seq":"939","reported_epoch":"1168","state":"active+clean","last_fresh":"2021-02-10T04:53:21.291364+0000","last_change":"2021-02-10T04:52:34.089878+0000","last_active":"2021-02-10T04:53:21.291364+0000","last_peered":"2021-02-10T04:53:21.291364+0000","last_clean":"2021-02-10T04:53:21.291364+0000","last_became_active":"2021-02-10T04:50:59.528302+0000","last_became_peered":"2021-02-10T04:50:59.528302+0000","last_unstale":"2021-02-10T04:53:21.291364+0000","last_undegraded":"2021-02-10T04:53:21.291364+0000","last_fullsized":"2021-02-10T04:53:21.291364+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":5,"last_scrub":"1032'89","last_scrub_stamp":"2021-02-10T04:52:34.089789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:34.089789+0000","log_size":89,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5949711,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"1163'37","reported_seq":"855","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.540213+0000","last_change":"2021-02-10T04:53:14.202505+0000","last_active":"2021-02-10T04:53:26.540213+0000","last_peered":"2021-02-10T04:53:26.540213+0000","last_clean":"2021-02-10T04:53:26.540213+0000","last_became_active":"2021-02-10T04:51:49.496294+0000","last_became_peered":"2021-02-10T04:51:49.496294+0000","last_unstale":"2021-02-10T04:53:26.540213+0000","last_undegraded":"2021-02-10T04:53:26.540213+0000","last_fullsized":"2021-02-10T04:53:26.540213+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1163'37","last_scrub_stamp":"2021-02-10T04:53:14.202445+0000","last_deep_scrub":"1163'37","last_deep_scrub_stamp":"2021-02-10T04:53:14.202445+0000","last_clean_scrub_stamp":"2021-02-10T04:53:14.202445+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":33,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"1169","reported_epoch":"1168","state":"active+clean","last_fresh":"2021-02-10T04:53:21.292780+0000","last_change":"2021-02-10T04:52:23.095651+0000","last_active":"2021-02-10T04:53:21.292780+0000","last_peered":"2021-02-10T04:53:21.292780+0000","last_clean":"2021-02-10T04:53:21.292780+0000","last_became_active":"2021-02-10T04:50:59.543531+0000","last_became_peered":"2021-02-10T04:50:59.543531+0000","last_unstale":"2021-02-10T04:53:21.292780+0000","last_undegraded":"2021-02-10T04:53:21.292780+0000","last_fullsized":"2021-02-10T04:53:21.292780+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:23.095588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:23.095588+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"1113","reported_epoch":"1171","state":"active+undersized","last_fresh":"2021-02-10T04:53:25.545862+0000","last_change":"2021-02-10T04:52:32.085370+0000","last_active":"2021-02-10T04:53:25.545862+0000","last_peered":"2021-02-10T04:53:25.545862+0000","last_clean":"2021-02-10T04:52:30.067066+0000","last_became_active":"2021-02-10T04:52:32.085370+0000","last_became_peered":"2021-02-10T04:52:32.085370+0000","last_unstale":"2021-02-10T04:53:25.545862+0000","last_undegraded":"2021-02-10T04:53:25.545862+0000","last_fullsized":"2021-02-10T04:52:32.074435+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:43.663762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:51:43.663762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"34","reported_epoch":"1172","state":"active+undersized","last_fresh":"2021-02-10T04:53:26.555982+0000","last_change":"2021-02-10T04:52:32.091922+0000","last_active":"2021-02-10T04:53:26.555982+0000","last_peered":"2021-02-10T04:53:26.555982+0000","last_clean":"2021-02-10T04:25:52.015519+0000","last_became_active":"2021-02-10T04:52:32.091922+0000","last_became_peered":"2021-02-10T04:52:32.091922+0000","last_unstale":"2021-02-10T04:53:26.555982+0000","last_undegraded":"2021-02-10T04:53:26.555982+0000","last_fullsized":"2021-02-10T04:52:32.069065+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:57.568541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:57.568541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.10","version":"1000'1071","reported_seq":"4672","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.544682+0000","last_change":"2021-02-10T04:51:39.509148+0000","last_active":"2021-02-10T04:53:25.544682+0000","last_peered":"2021-02-10T04:53:25.544682+0000","last_clean":"2021-02-10T04:53:25.544682+0000","last_became_active":"2021-02-10T04:51:39.508551+0000","last_became_peered":"2021-02-10T04:51:39.508551+0000","last_unstale":"2021-02-10T04:53:25.544682+0000","last_undegraded":"2021-02-10T04:53:25.544682+0000","last_fullsized":"2021-02-10T04:53:25.544682+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":5,"last_scrub":"412'357","last_scrub_stamp":"2021-02-10T04:33:50.582002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:33:50.582002+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23286368,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":242,"num_read_kb":226835,"num_write":185,"num_write_kb":76005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"1049'31","reported_seq":"1195","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.534340+0000","last_change":"2021-02-10T04:52:39.695340+0000","last_active":"2021-02-10T04:53:25.534340+0000","last_peered":"2021-02-10T04:53:25.534340+0000","last_clean":"2021-02-10T04:53:25.534340+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:53:25.534340+0000","last_undegraded":"2021-02-10T04:53:25.534340+0000","last_fullsized":"2021-02-10T04:53:25.534340+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'31","last_scrub_stamp":"2021-02-10T04:52:39.695237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:39.695237+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":41,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"405","reported_epoch":"1172","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:26.547997+0000","last_change":"2021-02-10T04:52:32.094351+0000","last_active":"2021-02-10T04:53:26.547997+0000","last_peered":"2021-02-10T04:53:26.547997+0000","last_clean":"2021-02-10T04:52:30.061022+0000","last_became_active":"2021-02-10T04:52:32.094351+0000","last_became_peered":"2021-02-10T04:52:32.094351+0000","last_unstale":"2021-02-10T04:53:26.547997+0000","last_undegraded":"2021-02-10T04:52:32.074350+0000","last_fullsized":"2021-02-10T04:52:32.074239+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:47:08.700069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:08.700069+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"822","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.540273+0000","last_change":"2021-02-10T04:51:49.501612+0000","last_active":"2021-02-10T04:53:26.540273+0000","last_peered":"2021-02-10T04:53:26.540273+0000","last_clean":"2021-02-10T04:53:26.540273+0000","last_became_active":"2021-02-10T04:51:49.501254+0000","last_became_peered":"2021-02-10T04:51:49.501254+0000","last_unstale":"2021-02-10T04:53:26.540273+0000","last_undegraded":"2021-02-10T04:53:26.540273+0000","last_fullsized":"2021-02-10T04:53:26.540273+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:05.475145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:51:05.475145+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"1140","reported_epoch":"1171","state":"active+undersized","last_fresh":"2021-02-10T04:53:25.544925+0000","last_change":"2021-02-10T04:52:32.087218+0000","last_active":"2021-02-10T04:53:25.544925+0000","last_peered":"2021-02-10T04:53:25.544925+0000","last_clean":"2021-02-10T04:52:30.062886+0000","last_became_active":"2021-02-10T04:52:32.087218+0000","last_became_peered":"2021-02-10T04:52:32.087218+0000","last_unstale":"2021-02-10T04:53:25.544925+0000","last_undegraded":"2021-02-10T04:53:25.544925+0000","last_fullsized":"2021-02-10T04:52:32.072167+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:55.814486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.814486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"1002'1657","reported_seq":"5071","reported_epoch":"1170","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:24.536467+0000","last_change":"2021-02-10T04:52:32.088477+0000","last_active":"2021-02-10T04:53:24.536467+0000","last_peered":"2021-02-10T04:53:24.536467+0000","last_clean":"2021-02-10T04:51:27.993324+0000","last_became_active":"2021-02-10T04:52:32.088477+0000","last_became_peered":"2021-02-10T04:52:32.088477+0000","last_unstale":"2021-02-10T04:53:24.536467+0000","last_undegraded":"2021-02-10T04:52:32.081970+0000","last_fullsized":"2021-02-10T04:52:32.081848+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1124,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:46:54.881412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:54.881412+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":47335497,"num_objects":20,"num_object_clones":18,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":20}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"1048'20","reported_seq":"1273","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.545104+0000","last_change":"2021-02-10T04:53:04.058324+0000","last_active":"2021-02-10T04:53:25.545104+0000","last_peered":"2021-02-10T04:53:25.545104+0000","last_clean":"2021-02-10T04:53:25.545104+0000","last_became_active":"2021-02-10T04:50:59.544484+0000","last_became_peered":"2021-02-10T04:50:59.544484+0000","last_unstale":"2021-02-10T04:53:25.545104+0000","last_undegraded":"2021-02-10T04:53:25.545104+0000","last_fullsized":"2021-02-10T04:53:25.545104+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'20","last_scrub_stamp":"2021-02-10T04:53:04.058248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:04.058248+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":20,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"1176","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.562991+0000","last_change":"2021-02-10T04:53:00.343307+0000","last_active":"2021-02-10T04:53:26.562991+0000","last_peered":"2021-02-10T04:53:26.562991+0000","last_clean":"2021-02-10T04:53:26.562991+0000","last_became_active":"2021-02-10T04:51:36.844573+0000","last_became_peered":"2021-02-10T04:51:36.844573+0000","last_unstale":"2021-02-10T04:53:26.562991+0000","last_undegraded":"2021-02-10T04:53:26.562991+0000","last_fullsized":"2021-02-10T04:53:26.562991+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:00.343228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:00.343228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"1140","reported_epoch":"1168","state":"active+clean","last_fresh":"2021-02-10T04:53:21.291146+0000","last_change":"2021-02-10T04:50:56.099710+0000","last_active":"2021-02-10T04:53:21.291146+0000","last_peered":"2021-02-10T04:53:21.291146+0000","last_clean":"2021-02-10T04:53:21.291146+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:53:21.291146+0000","last_undegraded":"2021-02-10T04:53:21.291146+0000","last_fullsized":"2021-02-10T04:53:21.291146+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:56.099651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:50:56.099651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"67","reported_epoch":"1170","state":"active+clean","last_fresh":"2021-02-10T04:53:24.536051+0000","last_change":"2021-02-10T04:51:50.036214+0000","last_active":"2021-02-10T04:53:24.536051+0000","last_peered":"2021-02-10T04:53:24.536051+0000","last_clean":"2021-02-10T04:53:24.536051+0000","last_became_active":"2021-02-10T04:51:36.855910+0000","last_became_peered":"2021-02-10T04:51:36.855910+0000","last_unstale":"2021-02-10T04:53:24.536051+0000","last_undegraded":"2021-02-10T04:53:24.536051+0000","last_fullsized":"2021-02-10T04:53:24.536051+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.036117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.036117+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"998'1009","reported_seq":"5007","reported_epoch":"1168","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:21.291071+0000","last_change":"2021-02-10T04:52:32.090443+0000","last_active":"2021-02-10T04:53:21.291071+0000","last_peered":"2021-02-10T04:53:21.291071+0000","last_clean":"2021-02-10T04:52:30.061024+0000","last_became_active":"2021-02-10T04:52:32.090443+0000","last_became_peered":"2021-02-10T04:52:32.090443+0000","last_unstale":"2021-02-10T04:53:21.291071+0000","last_undegraded":"2021-02-10T04:52:32.076916+0000","last_fullsized":"2021-02-10T04:52:32.076790+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1127,"parent":"0.0","parent_split_bits":5,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:13.032506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:13.032506+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49420827,"num_objects":21,"num_object_clones":19,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":36763798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":21}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"1049'37","reported_seq":"1201","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.545318+0000","last_change":"2021-02-10T04:51:44.694779+0000","last_active":"2021-02-10T04:53:25.545318+0000","last_peered":"2021-02-10T04:53:25.545318+0000","last_clean":"2021-02-10T04:53:25.545318+0000","last_became_active":"2021-02-10T04:50:59.549412+0000","last_became_peered":"2021-02-10T04:50:59.549412+0000","last_unstale":"2021-02-10T04:53:25.545318+0000","last_undegraded":"2021-02-10T04:53:25.545318+0000","last_fullsized":"2021-02-10T04:53:25.545318+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:44.694711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.694711+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":64,"num_read_kb":50,"num_write":37,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"1307","reported_epoch":"1168","state":"active+clean","last_fresh":"2021-02-10T04:53:21.301172+0000","last_change":"2021-02-10T04:52:33.088525+0000","last_active":"2021-02-10T04:53:21.301172+0000","last_peered":"2021-02-10T04:53:21.301172+0000","last_clean":"2021-02-10T04:53:21.301172+0000","last_became_active":"2021-02-10T04:51:36.870573+0000","last_became_peered":"2021-02-10T04:51:36.870573+0000","last_unstale":"2021-02-10T04:53:21.301172+0000","last_undegraded":"2021-02-10T04:53:21.301172+0000","last_fullsized":"2021-02-10T04:53:21.301172+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:52:33.088428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:33.088428+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"457","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.545185+0000","last_change":"2021-02-10T04:51:51.836723+0000","last_active":"2021-02-10T04:53:25.545185+0000","last_peered":"2021-02-10T04:53:25.545185+0000","last_clean":"2021-02-10T04:53:25.545185+0000","last_became_active":"2021-02-10T04:50:59.564977+0000","last_became_peered":"2021-02-10T04:50:59.564977+0000","last_unstale":"2021-02-10T04:53:25.545185+0000","last_undegraded":"2021-02-10T04:53:25.545185+0000","last_fullsized":"2021-02-10T04:53:25.545185+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.836638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.836638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"1104","reported_epoch":"1171","state":"active+undersized","last_fresh":"2021-02-10T04:53:25.545729+0000","last_change":"2021-02-10T04:52:32.081484+0000","last_active":"2021-02-10T04:53:25.545729+0000","last_peered":"2021-02-10T04:53:25.545729+0000","last_clean":"2021-02-10T04:52:30.067390+0000","last_became_active":"2021-02-10T04:52:32.081484+0000","last_became_peered":"2021-02-10T04:52:32.081484+0000","last_unstale":"2021-02-10T04:53:25.545729+0000","last_undegraded":"2021-02-10T04:53:25.545729+0000","last_fullsized":"2021-02-10T04:52:32.074112+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:58.514318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_clean_scrub_stamp":"2021-02-10T04:46:58.514318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1d","version":"1001'1165","reported_seq":"4166","reported_epoch":"1172","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:26.548157+0000","last_change":"2021-02-10T04:52:32.087778+0000","last_active":"2021-02-10T04:53:26.548157+0000","last_peered":"2021-02-10T04:53:26.548157+0000","last_clean":"2021-02-10T04:52:30.061103+0000","last_became_active":"2021-02-10T04:52:32.087778+0000","last_became_peered":"2021-02-10T04:52:32.087778+0000","last_unstale":"2021-02-10T04:53:26.548157+0000","last_undegraded":"2021-02-10T04:52:32.074694+0000","last_fullsized":"2021-02-10T04:52:32.074582+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1144,"parent":"0.0","parent_split_bits":5,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29601845,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":287,"num_read_kb":299530,"num_write":200,"num_write_kb":89131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":13}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"1049'42","reported_seq":"66","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.537056+0000","last_change":"2021-02-10T04:53:21.620902+0000","last_active":"2021-02-10T04:53:25.537056+0000","last_peered":"2021-02-10T04:53:25.537056+0000","last_clean":"2021-02-10T04:53:25.537056+0000","last_became_active":"2021-02-10T04:53:20.298497+0000","last_became_peered":"2021-02-10T04:53:20.298497+0000","last_unstale":"2021-02-10T04:53:25.537056+0000","last_undegraded":"2021-02-10T04:53:25.537056+0000","last_fullsized":"2021-02-10T04:53:25.537056+0000","mapping_epoch":1166,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'42","last_scrub_stamp":"2021-02-10T04:51:45.336772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.336772+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":74,"num_read_kb":56,"num_write":42,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":934,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"1217","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.555899+0000","last_change":"2021-02-10T04:53:01.331566+0000","last_active":"2021-02-10T04:53:26.555899+0000","last_peered":"2021-02-10T04:53:26.555899+0000","last_clean":"2021-02-10T04:53:26.555899+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:53:26.555899+0000","last_undegraded":"2021-02-10T04:53:26.555899+0000","last_fullsized":"2021-02-10T04:53:26.555899+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:01.331466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.331466+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"1098'3","reported_seq":"1474","reported_epoch":"1168","state":"active+clean","last_fresh":"2021-02-10T04:53:23.013254+0000","last_change":"2021-02-10T04:50:15.255211+0000","last_active":"2021-02-10T04:53:23.013254+0000","last_peered":"2021-02-10T04:53:23.013254+0000","last_clean":"2021-02-10T04:53:23.013254+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:53:23.013254+0000","last_undegraded":"2021-02-10T04:53:23.013254+0000","last_fullsized":"2021-02-10T04:53:23.013254+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"1098'3","last_scrub_stamp":"2021-02-10T04:50:15.255137+0000","last_deep_scrub":"1098'3","last_deep_scrub_stamp":"2021-02-10T04:50:15.255137+0000","last_clean_scrub_stamp":"2021-02-10T04:50:15.255137+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"1096","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.556530+0000","last_change":"2021-02-10T04:51:47.493125+0000","last_active":"2021-02-10T04:53:26.556530+0000","last_peered":"2021-02-10T04:53:26.556530+0000","last_clean":"2021-02-10T04:53:26.556530+0000","last_became_active":"2021-02-10T04:51:36.852591+0000","last_became_peered":"2021-02-10T04:51:36.852591+0000","last_unstale":"2021-02-10T04:53:26.556530+0000","last_undegraded":"2021-02-10T04:53:26.556530+0000","last_fullsized":"2021-02-10T04:53:26.556530+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:47.493057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.493057+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1c","version":"1002'1090","reported_seq":"5192","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.556385+0000","last_change":"2021-02-10T04:52:31.088797+0000","last_active":"2021-02-10T04:53:26.556385+0000","last_peered":"2021-02-10T04:53:26.556385+0000","last_clean":"2021-02-10T04:53:26.556385+0000","last_became_active":"2021-02-10T04:52:31.088265+0000","last_became_peered":"2021-02-10T04:52:31.088265+0000","last_unstale":"2021-02-10T04:53:26.556385+0000","last_undegraded":"2021-02-10T04:53:26.556385+0000","last_fullsized":"2021-02-10T04:53:26.556385+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:51:08.220472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.220472+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24410646,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":246,"num_read_kb":276168,"num_write":176,"num_write_kb":64961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"1108","reported_epoch":"1171","state":"active+undersized","last_fresh":"2021-02-10T04:53:25.545042+0000","last_change":"2021-02-10T04:52:32.079189+0000","last_active":"2021-02-10T04:53:25.545042+0000","last_peered":"2021-02-10T04:53:25.545042+0000","last_clean":"2021-02-10T04:52:30.063079+0000","last_became_active":"2021-02-10T04:52:32.079189+0000","last_became_peered":"2021-02-10T04:52:32.079189+0000","last_unstale":"2021-02-10T04:53:25.545042+0000","last_undegraded":"2021-02-10T04:53:25.545042+0000","last_fullsized":"2021-02-10T04:52:32.071674+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:42.537443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:42.537443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"1131","reported_epoch":"1170","state":"active+undersized","last_fresh":"2021-02-10T04:53:24.537290+0000","last_change":"2021-02-10T04:52:32.088503+0000","last_active":"2021-02-10T04:53:24.537290+0000","last_peered":"2021-02-10T04:53:24.537290+0000","last_clean":"2021-02-10T04:51:27.994110+0000","last_became_active":"2021-02-10T04:52:32.088503+0000","last_became_peered":"2021-02-10T04:52:32.088503+0000","last_unstale":"2021-02-10T04:53:24.537290+0000","last_undegraded":"2021-02-10T04:53:24.537290+0000","last_fullsized":"2021-02-10T04:52:32.082093+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.938731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.938731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"384","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.547830+0000","last_change":"2021-02-10T04:53:15.185748+0000","last_active":"2021-02-10T04:53:26.547830+0000","last_peered":"2021-02-10T04:53:26.547830+0000","last_clean":"2021-02-10T04:53:26.547830+0000","last_became_active":"2021-02-10T04:51:49.499348+0000","last_became_peered":"2021-02-10T04:51:49.499348+0000","last_unstale":"2021-02-10T04:53:26.547830+0000","last_undegraded":"2021-02-10T04:53:26.547830+0000","last_fullsized":"2021-02-10T04:53:26.547830+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:15.185686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:15.185686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1f","version":"1004'1093","reported_seq":"6430","reported_epoch":"1168","state":"active+clean","last_fresh":"2021-02-10T04:53:21.301414+0000","last_change":"2021-02-10T04:52:31.089952+0000","last_active":"2021-02-10T04:53:21.301414+0000","last_peered":"2021-02-10T04:53:21.301414+0000","last_clean":"2021-02-10T04:53:21.301414+0000","last_became_active":"2021-02-10T04:52:31.088634+0000","last_became_peered":"2021-02-10T04:52:31.088634+0000","last_unstale":"2021-02-10T04:53:21.301414+0000","last_undegraded":"2021-02-10T04:53:21.301414+0000","last_fullsized":"2021-02-10T04:53:21.301414+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39434697,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":279,"num_read_kb":311058,"num_write":186,"num_write_kb":66790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":53995913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"1049'39","reported_seq":"38","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.533750+0000","last_change":"2021-02-10T04:53:20.402799+0000","last_active":"2021-02-10T04:53:25.533750+0000","last_peered":"2021-02-10T04:53:25.533750+0000","last_clean":"2021-02-10T04:53:25.533750+0000","last_became_active":"2021-02-10T04:53:20.306525+0000","last_became_peered":"2021-02-10T04:53:20.306525+0000","last_unstale":"2021-02-10T04:53:25.533750+0000","last_undegraded":"2021-02-10T04:53:25.533750+0000","last_fullsized":"2021-02-10T04:53:25.533750+0000","mapping_epoch":1166,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1167,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'39","last_scrub_stamp":"2021-02-10T04:53:02.924664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:02.924664+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"1157","reported_epoch":"1168","state":"active+clean","last_fresh":"2021-02-10T04:53:21.298265+0000","last_change":"2021-02-10T04:52:36.083849+0000","last_active":"2021-02-10T04:53:21.298265+0000","last_peered":"2021-02-10T04:53:21.298265+0000","last_clean":"2021-02-10T04:53:21.298265+0000","last_became_active":"2021-02-10T04:25:36.336442+0000","last_became_peered":"2021-02-10T04:25:36.336442+0000","last_unstale":"2021-02-10T04:53:21.298265+0000","last_undegraded":"2021-02-10T04:53:21.298265+0000","last_fullsized":"2021-02-10T04:53:21.298265+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:52:36.083784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:36.083784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"387","reported_epoch":"1172","state":"active+undersized","last_fresh":"2021-02-10T04:53:26.548366+0000","last_change":"2021-02-10T04:52:32.084893+0000","last_active":"2021-02-10T04:53:26.548366+0000","last_peered":"2021-02-10T04:53:26.548366+0000","last_clean":"2021-02-10T04:52:30.061372+0000","last_became_active":"2021-02-10T04:52:32.084893+0000","last_became_peered":"2021-02-10T04:52:32.084893+0000","last_unstale":"2021-02-10T04:53:26.548366+0000","last_undegraded":"2021-02-10T04:53:26.548366+0000","last_fullsized":"2021-02-10T04:52:32.073304+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:08.758129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.758129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"1140","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.541929+0000","last_change":"2021-02-10T04:51:44.799512+0000","last_active":"2021-02-10T04:53:25.541929+0000","last_peered":"2021-02-10T04:53:25.541929+0000","last_clean":"2021-02-10T04:53:25.541929+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:53:25.541929+0000","last_undegraded":"2021-02-10T04:53:25.541929+0000","last_fullsized":"2021-02-10T04:53:25.541929+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:44.799429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.799429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1e","version":"1063'654","reported_seq":"4412","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.548084+0000","last_change":"2021-02-10T04:52:31.088116+0000","last_active":"2021-02-10T04:53:26.548084+0000","last_peered":"2021-02-10T04:53:26.548084+0000","last_clean":"2021-02-10T04:53:26.548084+0000","last_became_active":"2021-02-10T04:52:31.087662+0000","last_became_peered":"2021-02-10T04:52:31.087662+0000","last_unstale":"2021-02-10T04:53:26.548084+0000","last_undegraded":"2021-02-10T04:53:26.548084+0000","last_fullsized":"2021-02-10T04:53:26.548084+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:51:57.121838+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.121838+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24700631,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":171,"num_read_kb":182272,"num_write":112,"num_write_kb":33727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":26349024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"1066'44","reported_seq":"1154","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.538414+0000","last_change":"2021-02-10T04:53:20.807729+0000","last_active":"2021-02-10T04:53:25.538414+0000","last_peered":"2021-02-10T04:53:25.538414+0000","last_clean":"2021-02-10T04:53:25.538414+0000","last_became_active":"2021-02-10T04:53:20.297480+0000","last_became_peered":"2021-02-10T04:53:20.297480+0000","last_unstale":"2021-02-10T04:53:25.538414+0000","last_undegraded":"2021-02-10T04:53:25.538414+0000","last_fullsized":"2021-02-10T04:53:25.538414+0000","mapping_epoch":1166,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1167,"parent":"0.0","parent_split_bits":0,"last_scrub":"1066'44","last_scrub_stamp":"2021-02-10T04:53:06.077718+0000","last_deep_scrub":"1066'44","last_deep_scrub_stamp":"2021-02-10T04:47:18.806645+0000","last_clean_scrub_stamp":"2021-02-10T04:53:06.077718+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":67,"num_read_kb":49,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"1123","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.546336+0000","last_change":"2021-02-10T04:51:08.006463+0000","last_active":"2021-02-10T04:53:25.546336+0000","last_peered":"2021-02-10T04:53:25.546336+0000","last_clean":"2021-02-10T04:53:25.546336+0000","last_became_active":"2021-02-10T04:50:59.563657+0000","last_became_peered":"2021-02-10T04:50:59.563657+0000","last_unstale":"2021-02-10T04:53:25.546336+0000","last_undegraded":"2021-02-10T04:53:25.546336+0000","last_fullsized":"2021-02-10T04:53:25.546336+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_clean_scrub_stamp":"2021-02-10T04:51:02.455702+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"1175","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.537421+0000","last_change":"2021-02-10T04:52:33.741801+0000","last_active":"2021-02-10T04:53:25.537421+0000","last_peered":"2021-02-10T04:53:25.537421+0000","last_clean":"2021-02-10T04:53:25.537421+0000","last_became_active":"2021-02-10T04:51:36.860129+0000","last_became_peered":"2021-02-10T04:51:36.860129+0000","last_unstale":"2021-02-10T04:53:25.537421+0000","last_undegraded":"2021-02-10T04:53:25.537421+0000","last_fullsized":"2021-02-10T04:53:25.537421+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:33.741711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:46:58.666264+0000","last_clean_scrub_stamp":"2021-02-10T04:52:33.741711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"1135","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.551824+0000","last_change":"2021-02-10T04:52:25.030419+0000","last_active":"2021-02-10T04:53:25.551824+0000","last_peered":"2021-02-10T04:53:25.551824+0000","last_clean":"2021-02-10T04:53:25.551824+0000","last_became_active":"2021-02-10T04:50:59.539582+0000","last_became_peered":"2021-02-10T04:50:59.539582+0000","last_unstale":"2021-02-10T04:53:25.551824+0000","last_undegraded":"2021-02-10T04:53:25.551824+0000","last_fullsized":"2021-02-10T04:53:25.551824+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:25.030281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.030281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"1003'739","reported_seq":"3917","reported_epoch":"1168","state":"active+clean","last_fresh":"2021-02-10T04:53:21.290917+0000","last_change":"2021-02-10T04:53:01.929261+0000","last_active":"2021-02-10T04:53:21.290917+0000","last_peered":"2021-02-10T04:53:21.290917+0000","last_clean":"2021-02-10T04:53:21.290917+0000","last_became_active":"2021-02-10T04:51:40.511505+0000","last_became_peered":"2021-02-10T04:51:40.511505+0000","last_unstale":"2021-02-10T04:53:21.290917+0000","last_undegraded":"2021-02-10T04:53:21.290917+0000","last_fullsized":"2021-02-10T04:53:21.290917+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:53:01.929169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.929169+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22230846,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":18429397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"1112'47","reported_seq":"55","reported_epoch":"1168","state":"active+clean","last_fresh":"2021-02-10T04:53:21.334324+0000","last_change":"2021-02-10T04:53:21.334003+0000","last_active":"2021-02-10T04:53:21.334324+0000","last_peered":"2021-02-10T04:53:21.334324+0000","last_clean":"2021-02-10T04:53:21.334324+0000","last_became_active":"2021-02-10T04:53:20.297145+0000","last_became_peered":"2021-02-10T04:53:20.297145+0000","last_unstale":"2021-02-10T04:53:21.334324+0000","last_undegraded":"2021-02-10T04:53:21.334324+0000","last_fullsized":"2021-02-10T04:53:21.334324+0000","mapping_epoch":1166,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"1112'47","last_scrub_stamp":"2021-02-10T04:51:02.860835+0000","last_deep_scrub":"1112'47","last_deep_scrub_stamp":"2021-02-10T04:51:02.860835+0000","last_clean_scrub_stamp":"2021-02-10T04:51:02.860835+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":71,"num_read_kb":55,"num_write":38,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"1087","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.556741+0000","last_change":"2021-02-10T04:52:26.408468+0000","last_active":"2021-02-10T04:53:26.556741+0000","last_peered":"2021-02-10T04:53:26.556741+0000","last_clean":"2021-02-10T04:53:26.556741+0000","last_became_active":"2021-02-10T04:51:36.858446+0000","last_became_peered":"2021-02-10T04:51:36.858446+0000","last_unstale":"2021-02-10T04:53:26.556741+0000","last_undegraded":"2021-02-10T04:53:26.556741+0000","last_fullsized":"2021-02-10T04:53:26.556741+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:26.408326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.408326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"1173","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.556354+0000","last_change":"2021-02-10T04:53:07.324650+0000","last_active":"2021-02-10T04:53:26.556354+0000","last_peered":"2021-02-10T04:53:26.556354+0000","last_clean":"2021-02-10T04:53:26.556354+0000","last_became_active":"2021-02-10T04:50:59.575340+0000","last_became_peered":"2021-02-10T04:50:59.575340+0000","last_unstale":"2021-02-10T04:53:26.556354+0000","last_undegraded":"2021-02-10T04:53:26.556354+0000","last_fullsized":"2021-02-10T04:53:26.556354+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.324564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.324564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"34","reported_epoch":"1172","state":"active+undersized","last_fresh":"2021-02-10T04:53:26.539424+0000","last_change":"2021-02-10T04:52:32.091240+0000","last_active":"2021-02-10T04:53:26.539424+0000","last_peered":"2021-02-10T04:53:26.539424+0000","last_clean":"2021-02-10T04:25:52.015519+0000","last_became_active":"2021-02-10T04:52:32.091240+0000","last_became_peered":"2021-02-10T04:52:32.091240+0000","last_unstale":"2021-02-10T04:53:26.539424+0000","last_undegraded":"2021-02-10T04:53:26.539424+0000","last_fullsized":"2021-02-10T04:52:32.077955+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:00.501127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:00.501127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.18","version":"259'74","reported_seq":"31","reported_epoch":"1168","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:21.292020+0000","last_change":"2021-02-10T04:52:32.083683+0000","last_active":"2021-02-10T04:53:21.292020+0000","last_peered":"2021-02-10T04:53:21.292020+0000","last_clean":"2021-02-10T04:31:15.165244+0000","last_became_active":"2021-02-10T04:52:32.083683+0000","last_became_peered":"2021-02-10T04:52:32.083683+0000","last_unstale":"2021-02-10T04:53:21.292020+0000","last_undegraded":"2021-02-10T04:52:32.067776+0000","last_fullsized":"2021-02-10T04:52:32.067560+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1010,"parent":"0.0","parent_split_bits":5,"last_scrub":"259'74","last_scrub_stamp":"2021-02-10T04:52:26.597668+0000","last_deep_scrub":"259'74","last_deep_scrub_stamp":"2021-02-10T04:46:49.745104+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.597668+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1218524,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":28228,"num_write":19,"num_write_kb":7022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"1061'20","reported_seq":"1076","reported_epoch":"1172","state":"active+clean+remapped","last_fresh":"2021-02-10T04:53:26.540395+0000","last_change":"2021-02-10T04:53:21.301748+0000","last_active":"2021-02-10T04:53:26.540395+0000","last_peered":"2021-02-10T04:53:26.540395+0000","last_clean":"2021-02-10T04:53:26.540395+0000","last_became_active":"2021-02-10T04:53:21.301122+0000","last_became_peered":"2021-02-10T04:53:21.301122+0000","last_unstale":"2021-02-10T04:53:26.540395+0000","last_undegraded":"2021-02-10T04:53:26.540395+0000","last_fullsized":"2021-02-10T04:53:26.540395+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"1061'20","last_scrub_stamp":"2021-02-10T04:52:31.770907+0000","last_deep_scrub":"1061'20","last_deep_scrub_stamp":"2021-02-10T04:46:55.646261+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.770907+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":17,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,3],"avail_no_missing":["1","3","4"],"object_location_counts":[{"shards":"1,3,4","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"1071","reported_epoch":"1168","state":"active+undersized","last_fresh":"2021-02-10T04:53:21.298789+0000","last_change":"2021-02-10T04:52:32.083801+0000","last_active":"2021-02-10T04:53:21.298789+0000","last_peered":"2021-02-10T04:53:21.298789+0000","last_clean":"2021-02-10T04:52:30.068110+0000","last_became_active":"2021-02-10T04:52:32.083801+0000","last_became_peered":"2021-02-10T04:52:32.083801+0000","last_unstale":"2021-02-10T04:53:21.298789+0000","last_undegraded":"2021-02-10T04:53:21.298789+0000","last_fullsized":"2021-02-10T04:52:32.078249+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:59.167911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:50:59.167911+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"1062","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.533866+0000","last_change":"2021-02-10T04:51:45.813755+0000","last_active":"2021-02-10T04:53:25.533866+0000","last_peered":"2021-02-10T04:53:25.533866+0000","last_clean":"2021-02-10T04:53:25.533866+0000","last_became_active":"2021-02-10T04:29:07.146525+0000","last_became_peered":"2021-02-10T04:29:07.146525+0000","last_unstale":"2021-02-10T04:53:25.533866+0000","last_undegraded":"2021-02-10T04:53:25.533866+0000","last_fullsized":"2021-02-10T04:53:25.533866+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:45.813676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.813676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"1157","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.555556+0000","last_change":"2021-02-10T04:51:09.433892+0000","last_active":"2021-02-10T04:53:26.555556+0000","last_peered":"2021-02-10T04:53:26.555556+0000","last_clean":"2021-02-10T04:53:26.555556+0000","last_became_active":"2021-02-10T04:50:59.539256+0000","last_became_peered":"2021-02-10T04:50:59.539256+0000","last_unstale":"2021-02-10T04:53:26.555556+0000","last_undegraded":"2021-02-10T04:53:26.555556+0000","last_fullsized":"2021-02-10T04:53:26.555556+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:09.433793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:09.433793+0000","last_clean_scrub_stamp":"2021-02-10T04:51:09.433793+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1b","version":"1003'1689","reported_seq":"4601","reported_epoch":"1170","state":"active+clean+remapped","last_fresh":"2021-02-10T04:53:24.535809+0000","last_change":"2021-02-10T04:53:10.075976+0000","last_active":"2021-02-10T04:53:24.535809+0000","last_peered":"2021-02-10T04:53:24.535809+0000","last_clean":"2021-02-10T04:53:24.535809+0000","last_became_active":"2021-02-10T04:52:33.087105+0000","last_became_peered":"2021-02-10T04:52:33.087105+0000","last_unstale":"2021-02-10T04:53:24.535809+0000","last_undegraded":"2021-02-10T04:53:24.535809+0000","last_fullsized":"2021-02-10T04:53:24.535809+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:53:10.075879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.075879+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27159495,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3,1],"avail_no_missing":["2","1","3"],"object_location_counts":[{"shards":"1,2,3","objects":12}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"1048'44","reported_seq":"51","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.555090+0000","last_change":"2021-02-10T04:53:20.360411+0000","last_active":"2021-02-10T04:53:26.555090+0000","last_peered":"2021-02-10T04:53:26.555090+0000","last_clean":"2021-02-10T04:53:26.555090+0000","last_became_active":"2021-02-10T04:53:20.304544+0000","last_became_peered":"2021-02-10T04:53:20.304544+0000","last_unstale":"2021-02-10T04:53:26.555090+0000","last_undegraded":"2021-02-10T04:53:26.555090+0000","last_fullsized":"2021-02-10T04:53:26.555090+0000","mapping_epoch":1166,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1167,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'44","last_scrub_stamp":"2021-02-10T04:50:48.668079+0000","last_deep_scrub":"78'32","last_deep_scrub_stamp":"2021-02-10T04:27:09.170428+0000","last_clean_scrub_stamp":"2021-02-10T04:50:48.668079+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":64,"num_read_kb":45,"num_write":34,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"1147","reported_epoch":"1171","state":"active+undersized","last_fresh":"2021-02-10T04:53:25.547239+0000","last_change":"2021-02-10T04:52:32.087531+0000","last_active":"2021-02-10T04:53:25.547239+0000","last_peered":"2021-02-10T04:53:25.547239+0000","last_clean":"2021-02-10T04:52:30.066743+0000","last_became_active":"2021-02-10T04:52:32.087531+0000","last_became_peered":"2021-02-10T04:52:32.087531+0000","last_unstale":"2021-02-10T04:53:25.547239+0000","last_undegraded":"2021-02-10T04:53:25.547239+0000","last_fullsized":"2021-02-10T04:52:32.076804+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:33.543559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:46:33.543559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"832","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.555329+0000","last_change":"2021-02-10T04:53:04.354508+0000","last_active":"2021-02-10T04:53:26.555329+0000","last_peered":"2021-02-10T04:53:26.555329+0000","last_clean":"2021-02-10T04:53:26.555329+0000","last_became_active":"2021-02-10T04:33:05.169159+0000","last_became_peered":"2021-02-10T04:33:05.169159+0000","last_unstale":"2021-02-10T04:53:26.555329+0000","last_undegraded":"2021-02-10T04:53:26.555329+0000","last_fullsized":"2021-02-10T04:53:26.555329+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:04.354409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:04.354409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"1226","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.533794+0000","last_change":"2021-02-10T04:52:36.777554+0000","last_active":"2021-02-10T04:53:25.533794+0000","last_peered":"2021-02-10T04:53:25.533794+0000","last_clean":"2021-02-10T04:53:25.533794+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:53:25.533794+0000","last_undegraded":"2021-02-10T04:53:25.533794+0000","last_fullsized":"2021-02-10T04:53:25.533794+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:36.777485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:52:36.777485+0000","last_clean_scrub_stamp":"2021-02-10T04:52:36.777485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1a","version":"1001'1528","reported_seq":"5783","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.532509+0000","last_change":"2021-02-10T04:52:31.086406+0000","last_active":"2021-02-10T04:53:25.532509+0000","last_peered":"2021-02-10T04:53:25.532509+0000","last_clean":"2021-02-10T04:53:25.532509+0000","last_became_active":"2021-02-10T04:52:31.085959+0000","last_became_peered":"2021-02-10T04:52:31.085959+0000","last_unstale":"2021-02-10T04:53:25.532509+0000","last_undegraded":"2021-02-10T04:53:25.532509+0000","last_fullsized":"2021-02-10T04:53:25.532509+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46944136,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":358,"num_read_kb":351935,"num_write":286,"num_write_kb":111355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":21311893,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"1049'68","reported_seq":"65","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.532607+0000","last_change":"2021-02-10T04:53:20.381302+0000","last_active":"2021-02-10T04:53:25.532607+0000","last_peered":"2021-02-10T04:53:25.532607+0000","last_clean":"2021-02-10T04:53:25.532607+0000","last_became_active":"2021-02-10T04:53:20.296560+0000","last_became_peered":"2021-02-10T04:53:20.296560+0000","last_unstale":"2021-02-10T04:53:25.532607+0000","last_undegraded":"2021-02-10T04:53:25.532607+0000","last_fullsized":"2021-02-10T04:53:25.532607+0000","mapping_epoch":1166,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1167,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'68","last_scrub_stamp":"2021-02-10T04:47:09.608881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:47:09.608881+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":124,"num_read_kb":101,"num_write":69,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"37","reported_epoch":"1171","state":"active+undersized","last_fresh":"2021-02-10T04:53:25.542269+0000","last_change":"2021-02-10T04:52:32.084553+0000","last_active":"2021-02-10T04:53:25.542269+0000","last_peered":"2021-02-10T04:53:25.542269+0000","last_clean":"2021-02-10T04:25:33.303710+0000","last_became_active":"2021-02-10T04:52:32.084553+0000","last_became_peered":"2021-02-10T04:52:32.084553+0000","last_unstale":"2021-02-10T04:53:25.542269+0000","last_undegraded":"2021-02-10T04:53:25.542269+0000","last_fullsized":"2021-02-10T04:52:32.070030+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:17.919745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:17.919745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"457","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.544593+0000","last_change":"2021-02-10T04:53:07.092032+0000","last_active":"2021-02-10T04:53:25.544593+0000","last_peered":"2021-02-10T04:53:25.544593+0000","last_clean":"2021-02-10T04:53:25.544593+0000","last_became_active":"2021-02-10T04:50:59.572077+0000","last_became_peered":"2021-02-10T04:50:59.572077+0000","last_unstale":"2021-02-10T04:53:25.544593+0000","last_undegraded":"2021-02-10T04:53:25.544593+0000","last_fullsized":"2021-02-10T04:53:25.544593+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.091949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:19.755718+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.091949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"1134","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.546094+0000","last_change":"2021-02-10T04:51:59.838856+0000","last_active":"2021-02-10T04:53:25.546094+0000","last_peered":"2021-02-10T04:53:25.546094+0000","last_clean":"2021-02-10T04:53:25.546094+0000","last_became_active":"2021-02-10T04:50:59.563156+0000","last_became_peered":"2021-02-10T04:50:59.563156+0000","last_unstale":"2021-02-10T04:53:25.546094+0000","last_undegraded":"2021-02-10T04:53:25.546094+0000","last_fullsized":"2021-02-10T04:53:25.546094+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.838790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.838790+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":"868","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.556300+0000","last_change":"2021-02-10T04:52:30.405170+0000","last_active":"2021-02-10T04:53:26.556300+0000","last_peered":"2021-02-10T04:53:26.556300+0000","last_clean":"2021-02-10T04:53:26.556300+0000","last_became_active":"2021-02-10T04:51:36.852058+0000","last_became_peered":"2021-02-10T04:51:36.852058+0000","last_unstale":"2021-02-10T04:53:26.556300+0000","last_undegraded":"2021-02-10T04:53:26.556300+0000","last_fullsized":"2021-02-10T04:53:26.556300+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.405106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:20.635536+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.405106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.20","version":"0'0","reported_seq":"1132","reported_epoch":"1172","state":"active+clean","last_fresh":"2021-02-10T04:53:26.556165+0000","last_change":"2021-02-10T04:51:56.449184+0000","last_active":"2021-02-10T04:53:26.556165+0000","last_peered":"2021-02-10T04:53:26.556165+0000","last_clean":"2021-02-10T04:53:26.556165+0000","last_became_active":"2021-02-10T04:29:07.148937+0000","last_became_peered":"2021-02-10T04:29:07.148937+0000","last_unstale":"2021-02-10T04:53:26.556165+0000","last_undegraded":"2021-02-10T04:53:26.556165+0000","last_fullsized":"2021-02-10T04:53:26.556165+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:56.449085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:41.375276+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.449085+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":"1091","reported_epoch":"1168","state":"active+clean","last_fresh":"2021-02-10T04:53:21.298566+0000","last_change":"2021-02-10T04:51:36.870420+0000","last_active":"2021-02-10T04:53:21.298566+0000","last_peered":"2021-02-10T04:53:21.298566+0000","last_clean":"2021-02-10T04:53:21.298566+0000","last_became_active":"2021-02-10T04:51:36.869785+0000","last_became_peered":"2021-02-10T04:51:36.869785+0000","last_unstale":"2021-02-10T04:53:21.298566+0000","last_undegraded":"2021-02-10T04:53:21.298566+0000","last_fullsized":"2021-02-10T04:53:21.298566+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:04.210002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:50:04.210002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":"68","reported_epoch":"1170","state":"active+undersized","last_fresh":"2021-02-10T04:53:24.535466+0000","last_change":"2021-02-10T04:52:32.094666+0000","last_active":"2021-02-10T04:53:24.535466+0000","last_peered":"2021-02-10T04:53:24.535466+0000","last_clean":"2021-02-10T04:52:30.064564+0000","last_became_active":"2021-02-10T04:52:32.094666+0000","last_became_peered":"2021-02-10T04:52:32.094666+0000","last_unstale":"2021-02-10T04:53:24.535466+0000","last_undegraded":"2021-02-10T04:53:24.535466+0000","last_fullsized":"2021-02-10T04:52:32.084052+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:47.481686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:47.481686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":"62","reported_epoch":"1172","state":"active+undersized","last_fresh":"2021-02-10T04:53:26.547749+0000","last_change":"2021-02-10T04:52:32.096027+0000","last_active":"2021-02-10T04:53:26.547749+0000","last_peered":"2021-02-10T04:53:26.547749+0000","last_clean":"2021-02-10T04:52:30.061012+0000","last_became_active":"2021-02-10T04:52:32.096027+0000","last_became_peered":"2021-02-10T04:52:32.096027+0000","last_unstale":"2021-02-10T04:53:26.547749+0000","last_undegraded":"2021-02-10T04:53:26.547749+0000","last_fullsized":"2021-02-10T04:52:32.076629+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.706372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.706372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":"1178","reported_epoch":"1168","state":"active+clean","last_fresh":"2021-02-10T04:53:21.291418+0000","last_change":"2021-02-10T04:51:57.287452+0000","last_active":"2021-02-10T04:53:21.291418+0000","last_peered":"2021-02-10T04:53:21.291418+0000","last_clean":"2021-02-10T04:53:21.291418+0000","last_became_active":"2021-02-10T04:50:59.566827+0000","last_became_peered":"2021-02-10T04:50:59.566827+0000","last_unstale":"2021-02-10T04:53:21.291418+0000","last_undegraded":"2021-02-10T04:53:21.291418+0000","last_fullsized":"2021-02-10T04:53:21.291418+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.287363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.287363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":"143","reported_epoch":"1168","state":"active+undersized","last_fresh":"2021-02-10T04:53:21.300632+0000","last_change":"2021-02-10T04:52:32.082221+0000","last_active":"2021-02-10T04:53:21.300632+0000","last_peered":"2021-02-10T04:53:21.300632+0000","last_clean":"2021-02-10T04:52:30.068335+0000","last_became_active":"2021-02-10T04:52:32.082221+0000","last_became_peered":"2021-02-10T04:52:32.082221+0000","last_unstale":"2021-02-10T04:53:21.300632+0000","last_undegraded":"2021-02-10T04:53:21.300632+0000","last_fullsized":"2021-02-10T04:52:32.077664+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:10.308282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:10.308282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":"1070","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.546410+0000","last_change":"2021-02-10T04:53:09.121379+0000","last_active":"2021-02-10T04:53:25.546410+0000","last_peered":"2021-02-10T04:53:25.546410+0000","last_clean":"2021-02-10T04:53:25.546410+0000","last_became_active":"2021-02-10T04:50:59.561442+0000","last_became_peered":"2021-02-10T04:50:59.561442+0000","last_unstale":"2021-02-10T04:53:25.546410+0000","last_undegraded":"2021-02-10T04:53:25.546410+0000","last_fullsized":"2021-02-10T04:53:25.546410+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:09.121298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.121298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.21","version":"293'77","reported_seq":"742","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.545136+0000","last_change":"2021-02-10T04:52:28.062923+0000","last_active":"2021-02-10T04:53:25.545136+0000","last_peered":"2021-02-10T04:53:25.545136+0000","last_clean":"2021-02-10T04:53:25.545136+0000","last_became_active":"2021-02-10T04:52:28.062463+0000","last_became_peered":"2021-02-10T04:52:28.062463+0000","last_unstale":"2021-02-10T04:53:25.545136+0000","last_undegraded":"2021-02-10T04:53:25.545136+0000","last_fullsized":"2021-02-10T04:53:25.545136+0000","mapping_epoch":1143,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1144,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1886586,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":15163,"num_write":9,"num_write_kb":3316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":"67","reported_epoch":"1170","state":"active+clean","last_fresh":"2021-02-10T04:53:24.535435+0000","last_change":"2021-02-10T04:52:32.124955+0000","last_active":"2021-02-10T04:53:24.535435+0000","last_peered":"2021-02-10T04:53:24.535435+0000","last_clean":"2021-02-10T04:53:24.535435+0000","last_became_active":"2021-02-10T04:51:36.867061+0000","last_became_peered":"2021-02-10T04:51:36.867061+0000","last_unstale":"2021-02-10T04:53:24.535435+0000","last_undegraded":"2021-02-10T04:53:24.535435+0000","last_fullsized":"2021-02-10T04:53:24.535435+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:32.124901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:08.271977+0000","last_clean_scrub_stamp":"2021-02-10T04:52:32.124901+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":"1107","reported_epoch":"1168","state":"active+clean","last_fresh":"2021-02-10T04:53:21.292934+0000","last_change":"2021-02-10T04:51:07.182775+0000","last_active":"2021-02-10T04:53:21.292934+0000","last_peered":"2021-02-10T04:53:21.292934+0000","last_clean":"2021-02-10T04:53:21.292934+0000","last_became_active":"2021-02-10T04:28:23.126368+0000","last_became_peered":"2021-02-10T04:28:23.126368+0000","last_unstale":"2021-02-10T04:53:21.292934+0000","last_undegraded":"2021-02-10T04:53:21.292934+0000","last_fullsized":"2021-02-10T04:53:21.292934+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.182678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.182678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.20","version":"1000'1071","reported_seq":"4515","reported_epoch":"1170","state":"active+clean","last_fresh":"2021-02-10T04:53:24.531900+0000","last_change":"2021-02-10T04:53:03.915465+0000","last_active":"2021-02-10T04:53:24.531900+0000","last_peered":"2021-02-10T04:53:24.531900+0000","last_clean":"2021-02-10T04:53:24.531900+0000","last_became_active":"2021-02-10T04:52:31.088690+0000","last_became_peered":"2021-02-10T04:52:31.088690+0000","last_unstale":"2021-02-10T04:53:24.531900+0000","last_undegraded":"2021-02-10T04:53:24.531900+0000","last_fullsized":"2021-02-10T04:53:24.531900+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:03.915403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.915403+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16398893,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":"67","reported_epoch":"1170","state":"active+clean","last_fresh":"2021-02-10T04:53:24.536109+0000","last_change":"2021-02-10T04:52:28.092351+0000","last_active":"2021-02-10T04:53:24.536109+0000","last_peered":"2021-02-10T04:53:24.536109+0000","last_clean":"2021-02-10T04:53:24.536109+0000","last_became_active":"2021-02-10T04:51:36.855101+0000","last_became_peered":"2021-02-10T04:51:36.855101+0000","last_unstale":"2021-02-10T04:53:24.536109+0000","last_undegraded":"2021-02-10T04:53:24.536109+0000","last_fullsized":"2021-02-10T04:53:24.536109+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:28.092288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:07.767250+0000","last_clean_scrub_stamp":"2021-02-10T04:52:28.092288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.25","version":"53'2","reported_seq":"1122","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.537666+0000","last_change":"2021-02-10T04:50:12.772167+0000","last_active":"2021-02-10T04:53:25.537666+0000","last_peered":"2021-02-10T04:53:25.537666+0000","last_clean":"2021-02-10T04:53:25.537666+0000","last_became_active":"2021-02-10T04:28:23.467770+0000","last_became_peered":"2021-02-10T04:28:23.467770+0000","last_unstale":"2021-02-10T04:53:25.537666+0000","last_undegraded":"2021-02-10T04:53:25.537666+0000","last_fullsized":"2021-02-10T04:53:25.537666+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:50:12.772086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:12.772086+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.23","version":"1002'1657","reported_seq":"5039","reported_epoch":"1168","state":"active+clean","last_fresh":"2021-02-10T04:53:21.291384+0000","last_change":"2021-02-10T04:52:31.093874+0000","last_active":"2021-02-10T04:53:21.291384+0000","last_peered":"2021-02-10T04:53:21.291384+0000","last_clean":"2021-02-10T04:53:21.291384+0000","last_became_active":"2021-02-10T04:52:31.092996+0000","last_became_peered":"2021-02-10T04:52:31.092996+0000","last_unstale":"2021-02-10T04:53:21.291384+0000","last_undegraded":"2021-02-10T04:53:21.291384+0000","last_fullsized":"2021-02-10T04:53:21.291384+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21396533,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":212,"num_read_kb":218837,"num_write":159,"num_write_kb":59399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":20479176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":"1214","reported_epoch":"1168","state":"active+clean","last_fresh":"2021-02-10T04:53:21.290949+0000","last_change":"2021-02-10T04:53:12.951612+0000","last_active":"2021-02-10T04:53:21.290949+0000","last_peered":"2021-02-10T04:53:21.290949+0000","last_clean":"2021-02-10T04:53:21.290949+0000","last_became_active":"2021-02-10T04:50:59.566548+0000","last_became_peered":"2021-02-10T04:50:59.566548+0000","last_unstale":"2021-02-10T04:53:21.290949+0000","last_undegraded":"2021-02-10T04:53:21.290949+0000","last_fullsized":"2021-02-10T04:53:21.290949+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.951528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.951528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.26","version":"53'1","reported_seq":"567","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.541826+0000","last_change":"2021-02-10T04:53:03.934004+0000","last_active":"2021-02-10T04:53:25.541826+0000","last_peered":"2021-02-10T04:53:25.541826+0000","last_clean":"2021-02-10T04:53:25.541826+0000","last_became_active":"2021-02-10T04:37:37.231081+0000","last_became_peered":"2021-02-10T04:37:37.231081+0000","last_unstale":"2021-02-10T04:53:25.541826+0000","last_undegraded":"2021-02-10T04:53:25.541826+0000","last_fullsized":"2021-02-10T04:53:25.541826+0000","mapping_epoch":638,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:53:03.933910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.933910+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.22","version":"998'1009","reported_seq":"4534","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.533394+0000","last_change":"2021-02-10T04:52:31.086563+0000","last_active":"2021-02-10T04:53:25.533394+0000","last_peered":"2021-02-10T04:53:25.533394+0000","last_clean":"2021-02-10T04:53:25.533394+0000","last_became_active":"2021-02-10T04:52:31.085288+0000","last_became_peered":"2021-02-10T04:52:31.085288+0000","last_unstale":"2021-02-10T04:53:25.533394+0000","last_undegraded":"2021-02-10T04:53:25.533394+0000","last_fullsized":"2021-02-10T04:53:25.533394+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12355207,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":120,"num_read_kb":124702,"num_write":77,"num_write_kb":25052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4145029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":"1096","reported_epoch":"1168","state":"active+clean","last_fresh":"2021-02-10T04:53:21.291300+0000","last_change":"2021-02-10T04:53:10.958668+0000","last_active":"2021-02-10T04:53:21.291300+0000","last_peered":"2021-02-10T04:53:21.291300+0000","last_clean":"2021-02-10T04:53:21.291300+0000","last_became_active":"2021-02-10T04:45:49.552026+0000","last_became_peered":"2021-02-10T04:45:49.552026+0000","last_unstale":"2021-02-10T04:53:21.291300+0000","last_undegraded":"2021-02-10T04:53:21.291300+0000","last_fullsized":"2021-02-10T04:53:21.291300+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:10.958579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.958579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":"1138","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.542120+0000","last_change":"2021-02-10T04:52:41.758800+0000","last_active":"2021-02-10T04:53:25.542120+0000","last_peered":"2021-02-10T04:53:25.542120+0000","last_clean":"2021-02-10T04:53:25.542120+0000","last_became_active":"2021-02-10T04:29:07.145907+0000","last_became_peered":"2021-02-10T04:29:07.145907+0000","last_unstale":"2021-02-10T04:53:25.542120+0000","last_undegraded":"2021-02-10T04:53:25.542120+0000","last_fullsized":"2021-02-10T04:53:25.542120+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:41.758684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.758684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":"1099","reported_epoch":"1171","state":"active+clean","last_fresh":"2021-02-10T04:53:25.544857+0000","last_change":"2021-02-10T04:53:11.139690+0000","last_active":"2021-02-10T04:53:25.544857+0000","last_peered":"2021-02-10T04:53:25.544857+0000","last_clean":"2021-02-10T04:53:25.544857+0000","last_became_active":"2021-02-10T04:50:59.569162+0000","last_became_peered":"2021-02-10T04:50:59.569162+0000","last_unstale":"2021-02-10T04:53:25.544857+0000","last_undegraded":"2021-02-10T04:53:25.544857+0000","last_fullsized":"2021-02-10T04:53:25.544857+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:11.139602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.139602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":"1122","reported_epoch":"1168","state":"active+clean","last_fresh":"2021-02-10T04:53:21.291005+0000","last_change":"2021-02-10T04:52:24.072005+0000","last_active":"2021-02-10T04:53:21.291005+0000","last_peered":"2021-02-10T04:53:21.291005+0000","last_clean":"2021-02-10T04:53:21.291005+0000","last_became_active":"2021-02-10T04:50:59.527945+0000","last_became_peered":"2021-02-10T04:50:59.527945+0000","last_unstale":"2021-02-10T04:53:21.291005+0000","last_undegraded":"2021-02-10T04:53:21.291005+0000","last_fullsized":"2021-02-10T04:53:21.291005+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:24.071915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_clean_scrub_stamp":"2021-02-10T04:52:24.071915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":910085179,"num_objects":376,"num_object_clones":325,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":167,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":376,"num_whiteouts":12,"num_read":4587,"num_read_kb":4742681,"num_write":3336,"num_write_kb":1285835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":769,"num_bytes_recovered":371841630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2029686784,"data_stored":2612817805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34722,"ondisk_log_size":34722,"up":94,"acting":95,"num_store_stats":8},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":1968,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":18,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1069,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":5835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":111,"acting":111,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":29,"acting":29,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":530,"num_read_kb":525,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":8,"acting":8,"num_store_stats":7},{"poolid":4,"num_pg":31,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":53,"num_objects_misplaced":22,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":2289,"num_read_kb":1762,"num_write":1306,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":239,"num_bytes_recovered":16468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2068480,"data_stored":74336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1333,"ondisk_log_size":1333,"up":85,"acting":88,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31,"ondisk_log_size":31,"up":84,"acting":84,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":114,"acting":114,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365136,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":188820,"kb_used_data":148108,"kb_used_omap":0,"kb_used_meta":40704,"kb_avail":93556332,"statfs":{"total":95995035648,"available":95801683968,"internally_reserved":0,"allocated":151662592,"data_stored":201729466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41680896},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:27 2021","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":1.2370000000000001,"15min":1.284},"min":{"1min":0.50800000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1220000000000001,"5min":34.299999999999997,"15min":67.277000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.5109999999999999,"15min":1.548},"min":{"1min":0.46300000000000002,"5min":0.438,"15min":0.438},"max":{"1min":2.8690000000000002,"5min":34.652999999999999,"15min":66.933999999999997},"last":0.71399999999999997}]},{"osd":1,"last update":"Wed Feb 10 04:52:31 2021","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.92000000000000004,"15min":0.995},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":2.254,"5min":2.3420000000000001,"15min":10.140000000000001},"last":0.71499999999999997},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.0069999999999999,"15min":1.018},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.536,"5min":2.5409999999999999,"15min":7.1520000000000001},"last":0.505}]},{"osd":2,"last update":"Wed Feb 10 04:52:31 2021","interfaces":[{"interface":"back","average":{"1min":1.7869999999999999,"5min":1.2350000000000001,"15min":1.367},"min":{"1min":0.627,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":11.468,"5min":11.468,"15min":66.534000000000006},"last":0.626},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.2609999999999999,"15min":1.5589999999999999},"min":{"1min":0.69499999999999995,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.8779999999999999,"5min":6.7930000000000001,"15min":66.781999999999996},"last":0.89100000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:52:58 2021","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.3089999999999999,"15min":1.3480000000000001},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":2.2749999999999999,"5min":4.5890000000000004,"15min":4.5890000000000004},"last":0.58199999999999996},{"interface":"front","average":{"1min":1.042,"5min":1.3959999999999999,"15min":1.4550000000000001},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.698,"5min":4.6609999999999996,"15min":4.6609999999999996},"last":0.85399999999999998}]},{"osd":4,"last update":"Wed Feb 10 04:52:31 2021","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.474,"15min":1.516},"min":{"1min":0.78900000000000003,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":2.7970000000000002,"5min":7.0640000000000001,"15min":10.781000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.444,"5min":1.3700000000000001,"15min":1.413},"min":{"1min":0.76600000000000001,"5min":0.69099999999999995,"15min":0.65100000000000002},"max":{"1min":3.4049999999999998,"5min":4.6269999999999998,"15min":10.326000000000001},"last":0.83099999999999996}]},{"osd":5,"last update":"Wed Feb 10 04:52:31 2021","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.488,"15min":1.653},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.66700000000000004},"max":{"1min":1.9399999999999999,"5min":4.0570000000000004,"15min":35.283000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":1.6819999999999999,"5min":1.6739999999999999,"15min":1.9590000000000001},"min":{"1min":0.97099999999999997,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":3.8769999999999998,"5min":6.6760000000000002,"15min":35.353999999999999},"last":1.2330000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:52:31 2021","interfaces":[{"interface":"back","average":{"1min":1.581,"5min":1.7290000000000001,"15min":1.901},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.89100000000000001},"max":{"1min":4.0339999999999998,"5min":6.9409999999999998,"15min":16.050000000000001},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.698,"5min":1.6910000000000001,"15min":1.804},"min":{"1min":0.98999999999999999,"5min":0.876,"15min":0.876},"max":{"1min":3.6480000000000001,"5min":4.7439999999999998,"15min":15.960000000000001},"last":1.127}]}]},{"osd":6,"up_from":43,"seq":184683594068,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":401624,"kb_used_data":341200,"kb_used_omap":0,"kb_used_meta":60416,"kb_avail":93343528,"statfs":{"total":95995035648,"available":95583772672,"internally_reserved":0,"allocated":349388800,"data_stored":442694384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":61865984},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:52:12 2021","interfaces":[{"interface":"back","average":{"1min":4.016,"5min":2.7400000000000002,"15min":1.595},"min":{"1min":0.52000000000000002,"5min":0.48099999999999998,"15min":0.44700000000000001},"max":{"1min":61.768000000000001,"5min":137.60599999999999,"15min":137.60599999999999},"last":0.97999999999999998},{"interface":"front","average":{"1min":4.0270000000000001,"5min":2.694,"15min":1.7989999999999999},"min":{"1min":0.504,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":62.683,"5min":137.66,"15min":137.66},"last":0.82099999999999995}]},{"osd":1,"last update":"Wed Feb 10 04:53:14 2021","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.1040000000000001,"15min":1.087},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":13.18,"5min":13.18,"15min":13.18},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.1699999999999999,"15min":1.145},"min":{"1min":0.51800000000000002,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":11.041,"5min":11.041,"15min":11.041},"last":1.046}]},{"osd":2,"last update":"Wed Feb 10 04:53:14 2021","interfaces":[{"interface":"back","average":{"1min":8.8919999999999995,"5min":2.7930000000000001,"15min":1.7669999999999999},"min":{"1min":0.60999999999999999,"5min":0.57099999999999995,"15min":0.55600000000000005},"max":{"1min":163.81700000000001,"5min":163.81700000000001,"15min":163.81700000000001},"last":0.69899999999999995},{"interface":"front","average":{"1min":8.9610000000000003,"5min":2.7810000000000001,"15min":1.774},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.59899999999999998},"max":{"1min":164.56800000000001,"5min":164.56800000000001,"15min":164.56800000000001},"last":1.081}]},{"osd":3,"last update":"Wed Feb 10 04:53:00 2021","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.1890000000000001,"15min":1.2090000000000001},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.23,"5min":1.1459999999999999,"15min":1.1319999999999999},"min":{"1min":0.76000000000000001,"5min":0.749,"15min":0.749},"max":{"1min":2.363,"5min":2.363,"15min":2.363},"last":0.81899999999999995}]},{"osd":4,"last update":"Wed Feb 10 04:53:14 2021","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.341,"15min":1.4399999999999999},"min":{"1min":0.71399999999999997,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":2.2360000000000002,"5min":7.1459999999999999,"15min":7.1459999999999999},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.147,"5min":1.113,"15min":1.145},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":5.2750000000000004,"5min":11.997999999999999,"15min":11.997999999999999},"last":0.91400000000000003}]},{"osd":5,"last update":"Wed Feb 10 04:53:14 2021","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.3,"15min":1.6579999999999999},"min":{"1min":0.66800000000000004,"5min":0.61699999999999999,"15min":0.56899999999999995},"max":{"1min":12.045,"5min":12.045,"15min":81.540000000000006},"last":1.008},{"interface":"front","average":{"1min":1.8460000000000001,"5min":1.601,"15min":1.919},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78000000000000003},"max":{"1min":13.401,"5min":13.401,"15min":81.492999999999995},"last":1.288}]},{"osd":7,"last update":"Wed Feb 10 04:52:39 2021","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.2729999999999999,"15min":1.3660000000000001},"min":{"1min":0.79200000000000004,"5min":0.747,"15min":0.747},"max":{"1min":1.5820000000000001,"5min":4.0919999999999996,"15min":5.1820000000000004},"last":0.80000000000000004},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.6100000000000001,"15min":1.675},"min":{"1min":0.97699999999999998,"5min":0.95999999999999996,"15min":0.91000000000000003},"max":{"1min":2.448,"5min":5.4109999999999996,"15min":13.315},"last":1.397}]}]},{"osd":1,"up_from":13,"seq":55834575206,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":285304,"kb_used_data":223792,"kb_used_omap":0,"kb_used_meta":61504,"kb_avail":93459848,"statfs":{"total":95995035648,"available":95702884352,"internally_reserved":0,"allocated":229163008,"data_stored":288873348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62980096},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:48 2021","interfaces":[{"interface":"back","average":{"1min":1.696,"5min":2.6320000000000001,"15min":1.921},"min":{"1min":0.44700000000000001,"5min":0.437,"15min":0.437},"max":{"1min":14.053000000000001,"5min":115.622,"15min":115.622},"last":0.69599999999999995},{"interface":"front","average":{"1min":1.728,"5min":2.585,"15min":1.948},"min":{"1min":0.49199999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":14.109,"5min":115.73099999999999,"15min":115.73099999999999},"last":0.72899999999999998}]},{"osd":2,"last update":"Wed Feb 10 04:53:05 2021","interfaces":[{"interface":"back","average":{"1min":3.8109999999999999,"5min":3.0350000000000001,"15min":1.956},"min":{"1min":0.495,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":53.884999999999998,"5min":115.59699999999999,"15min":115.59699999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":3.8220000000000001,"5min":3.0760000000000001,"15min":1.992},"min":{"1min":0.49299999999999999,"5min":0.49099999999999999,"15min":0.46800000000000003},"max":{"1min":52.862000000000002,"5min":115.753,"15min":115.753},"last":0.61799999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:53:00 2021","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.92500000000000004,"15min":0.95399999999999996},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.2250000000000001,"5min":3.077,"15min":3.077},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.92500000000000004,"5min":1.026,"15min":1.0429999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.82299999999999995}]},{"osd":4,"last update":"Wed Feb 10 04:52:51 2021","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.081,"15min":1.196},"min":{"1min":0.60499999999999998,"5min":0.56699999999999995,"15min":0.55200000000000005},"max":{"1min":1.4710000000000001,"5min":2.9670000000000001,"15min":4.899},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.165,"5min":1.1559999999999999,"15min":1.3500000000000001},"min":{"1min":0.78200000000000003,"5min":0.73599999999999999,"15min":0.73199999999999998},"max":{"1min":2.5270000000000001,"5min":2.5270000000000001,"15min":29.547999999999998},"last":1.0449999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:52:55 2021","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.1679999999999999,"15min":1.228},"min":{"1min":0.64500000000000002,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.895,"5min":2.7959999999999998,"15min":7.8049999999999997},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.2889999999999999,"15min":1.5389999999999999},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.571,"5min":2.8199999999999998,"15min":28.678000000000001},"last":1.1859999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:53:05 2021","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.218,"15min":1.5129999999999999},"min":{"1min":0.75700000000000001,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.7010000000000001,"5min":2.1659999999999999,"15min":29.686},"last":1.21},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.2390000000000001,"15min":1.4419999999999999},"min":{"1min":0.86799999999999999,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.75,"5min":2.7789999999999999,"15min":28.475000000000001},"last":1.153}]},{"osd":7,"last update":"Wed Feb 10 04:53:23 2021","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.27,"15min":1.524},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.484,"5min":3.274,"15min":28.573},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.3120000000000001,"15min":1.5609999999999999},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.776,"5min":2.298,"15min":29.408999999999999},"last":1.302}]}]},{"osd":0,"up_from":8,"seq":34359738720,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":399164,"kb_used_data":337140,"kb_used_omap":0,"kb_used_meta":62016,"kb_avail":93345988,"statfs":{"total":95995035648,"available":95586291712,"internally_reserved":0,"allocated":345231360,"data_stored":470653300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":63504384},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 04:51:32 2021","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.81100000000000005,"15min":1.0089999999999999},"min":{"1min":0.47099999999999997,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.0329999999999999,"5min":2.2930000000000001,"15min":28.946999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.874,"5min":0.92300000000000004,"15min":1.0149999999999999},"min":{"1min":0.68000000000000005,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.1819999999999999,"5min":2.0670000000000002,"15min":3.7050000000000001},"last":0.89200000000000002}]},{"osd":2,"last update":"Wed Feb 10 04:52:09 2021","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":2.2389999999999999,"15min":1.613},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0700000000000001,"5min":97.983999999999995,"15min":97.983999999999995},"last":1.056},{"interface":"front","average":{"1min":0.749,"5min":2.153,"15min":1.5169999999999999},"min":{"1min":0.51900000000000002,"5min":0.51300000000000001,"15min":0.47699999999999998},"max":{"1min":1.1319999999999999,"5min":97.844999999999999,"15min":97.844999999999999},"last":1.296}]},{"osd":3,"last update":"Wed Feb 10 04:51:59 2021","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.87,"15min":0.87},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":1.002},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.254,"5min":1.254,"15min":1.254},"last":0.871}]},{"osd":4,"last update":"Wed Feb 10 04:51:32 2021","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":1.119,"15min":1.306},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.3200000000000001,"5min":2.206,"15min":5.1680000000000001},"last":1.1120000000000001},{"interface":"front","average":{"1min":0.96499999999999997,"5min":1.179,"15min":1.252},"min":{"1min":0.64200000000000002,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.607,"5min":3.359,"15min":14.672000000000001},"last":1.0920000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:51:54 2021","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.325,"15min":1.512},"min":{"1min":0.84599999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.714,"5min":3.944,"15min":25.047000000000001},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.413,"15min":1.7090000000000001},"min":{"1min":0.79400000000000004,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":2.1669999999999998,"5min":3.7850000000000001,"15min":28.827999999999999},"last":1.226}]},{"osd":6,"last update":"Wed Feb 10 04:52:23 2021","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.845,"15min":1.8280000000000001},"min":{"1min":1.0109999999999999,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.7889999999999999,"5min":45.012,"15min":45.012},"last":1.587},{"interface":"front","average":{"1min":1.327,"5min":1.891,"15min":1.95},"min":{"1min":0.88800000000000001,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.0979999999999999,"5min":45.225000000000001,"15min":45.225000000000001},"last":1.3620000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:52:28 2021","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.3080000000000001,"15min":1.5069999999999999},"min":{"1min":0.96099999999999997,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.605,"5min":2.8700000000000001,"15min":5.6050000000000004},"last":1.532},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.3959999999999999,"15min":1.5669999999999999},"min":{"1min":1.0129999999999999,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.794,"5min":3.1259999999999999,"15min":13.968999999999999},"last":1.5129999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411684,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":363964,"kb_used_data":317364,"kb_used_omap":0,"kb_used_meta":46592,"kb_avail":93381188,"statfs":{"total":95995035648,"available":95622336512,"internally_reserved":0,"allocated":324980736,"data_stored":374230933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":47710208},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:50 2021","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.99099999999999999,"15min":1.718},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":3.698,"5min":3.996,"15min":163.15199999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.91300000000000003,"15min":1.655},"min":{"1min":0.51100000000000001,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.9710000000000001,"5min":2.2349999999999999,"15min":162.959},"last":0.85399999999999998}]},{"osd":1,"last update":"Wed Feb 10 04:52:52 2021","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.95899999999999996,"15min":1.0580000000000001},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.3029999999999999,"5min":3.8159999999999998,"15min":5.0229999999999997},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.81399999999999995,"15min":0.91700000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.177,"5min":1.895,"15min":5.726},"last":0.55900000000000005}]},{"osd":3,"last update":"Wed Feb 10 04:53:03 2021","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":1.099,"15min":1.143},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.4279999999999999,"5min":4.2380000000000004,"15min":4.2380000000000004},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.05,"5min":1.274,"15min":1.3120000000000001},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.607,"5min":4.54,"15min":4.54},"last":1.0249999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:52:30 2021","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.2450000000000001,"15min":1.3480000000000001},"min":{"1min":0.81299999999999994,"5min":0.80200000000000005,"15min":0.77000000000000002},"max":{"1min":1.8169999999999999,"5min":4.1310000000000002,"15min":4.3319999999999999},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.018,"5min":1.056,"15min":1.145},"min":{"1min":0.69499999999999995,"5min":0.66800000000000004,"15min":0.65900000000000003},"max":{"1min":1.3580000000000001,"5min":3.411,"15min":4.4489999999999998},"last":0.95999999999999996}]},{"osd":5,"last update":"Wed Feb 10 04:52:52 2021","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.8120000000000001,"15min":1.748},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.75,"5min":33.018999999999998,"15min":33.018999999999998},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.95699999999999996,"5min":1.625,"15min":1.512},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.4530000000000001,"5min":33.043999999999997,"15min":33.043999999999997},"last":0.76800000000000002}]},{"osd":6,"last update":"Wed Feb 10 04:53:20 2021","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":1.2350000000000001,"15min":1.498},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.46,"5min":2.4609999999999999,"15min":20.007000000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.28,"5min":1.4530000000000001,"15min":1.6830000000000001},"min":{"1min":0.92000000000000004,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.8680000000000001,"5min":4.415,"15min":19.417999999999999},"last":1.264}]},{"osd":7,"last update":"Wed Feb 10 04:52:27 2021","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.532,"15min":1.6719999999999999},"min":{"1min":1.0129999999999999,"5min":0.93400000000000005,"15min":0.93400000000000005},"max":{"1min":4.6699999999999999,"5min":4.6699999999999999,"15min":4.6699999999999999},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.208,"15min":1.355},"min":{"1min":0.97699999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":1.6719999999999999,"5min":1.95,"15min":4.9189999999999996},"last":1.1719999999999999}]}]},{"osd":3,"up_from":1111,"seq":4771708665887,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":326088,"kb_used_data":300160,"kb_used_omap":386,"kb_used_meta":25533,"kb_avail":93419064,"statfs":{"total":95995035648,"available":95661121536,"internally_reserved":0,"allocated":307363840,"data_stored":409160521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":396036,"internal_metadata":26146044},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:58 2021","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":0.67200000000000004}]},{"osd":1,"last update":"Wed Feb 10 04:53:03 2021","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.65500000000000003,"15min":0.65100000000000002},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.86699999999999999,"15min":0.88500000000000001},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":2.137,"5min":2.137,"15min":2.137},"last":0.58099999999999996}]},{"osd":2,"last update":"Wed Feb 10 04:53:03 2021","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":1.0740000000000001,"15min":1.1240000000000001},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.206,"5min":6.5460000000000003,"15min":6.5460000000000003},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.97599999999999998,"5min":1.0920000000000001,"15min":1.111},"min":{"1min":0.623,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.865,"5min":6.5049999999999999,"15min":6.5049999999999999},"last":0.67900000000000005}]},{"osd":4,"last update":"Wed Feb 10 04:53:03 2021","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.96399999999999997,"15min":0.96799999999999997},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.96,"5min":1.96,"15min":1.96},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.0840000000000001,"15min":1.1140000000000001},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.5049999999999999,"5min":1.8999999999999999,"15min":1.8999999999999999},"last":0.81499999999999995}]},{"osd":5,"last update":"Wed Feb 10 04:53:03 2021","interfaces":[{"interface":"back","average":{"1min":3.9529999999999998,"5min":2.9889999999999999,"15min":2.8279999999999998},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":52.353000000000002,"5min":52.353000000000002,"15min":52.353000000000002},"last":0.96099999999999997},{"interface":"front","average":{"1min":3.7730000000000001,"5min":2.7890000000000001,"15min":2.625},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":52.301000000000002,"5min":52.301000000000002,"15min":52.301000000000002},"last":0.72699999999999998}]},{"osd":6,"last update":"Wed Feb 10 04:53:03 2021","interfaces":[{"interface":"back","average":{"1min":3.9119999999999999,"5min":5.3639999999999999,"15min":5.6059999999999999},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":51.914999999999999,"5min":64.376999999999995,"15min":64.376999999999995},"last":1.4730000000000001},{"interface":"front","average":{"1min":4.0410000000000004,"5min":5.3559999999999999,"15min":5.5750000000000002},"min":{"1min":0.96499999999999997,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":51.896000000000001,"5min":64.438999999999993,"15min":64.438999999999993},"last":1.8080000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:53:03 2021","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.105,"15min":1.119},"min":{"1min":0.82699999999999996,"5min":0.754,"15min":0.754},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.143,"15min":1.143},"min":{"1min":0.90800000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"max":{"1min":2.044,"5min":2.044,"15min":2.044},"last":0.93899999999999995}]}]},{"osd":4,"up_from":31,"seq":133143986524,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":153888,"kb_used_data":119384,"kb_used_omap":0,"kb_used_meta":34496,"kb_avail":93591264,"statfs":{"total":95995035648,"available":95837454336,"internally_reserved":0,"allocated":122249216,"data_stored":155061800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35323904},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:36 2021","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.0899999999999999,"15min":1.593},"min":{"1min":0.45800000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.7799999999999998,"5min":95.721999999999994,"15min":95.721999999999994},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.67700000000000005,"5min":2.0870000000000002,"15min":1.6000000000000001},"min":{"1min":0.46000000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.002,"5min":95.344999999999999,"15min":95.344999999999999},"last":0.70599999999999996}]},{"osd":1,"last update":"Wed Feb 10 04:52:39 2021","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.89600000000000002,"15min":0.96799999999999997},"min":{"1min":0.50900000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":3.5529999999999999,"5min":7.883,"15min":24.295000000000002},"last":0.51100000000000001},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.081,"15min":1.232},"min":{"1min":0.63800000000000001,"5min":0.61599999999999999,"15min":0.60399999999999998},"max":{"1min":3.8090000000000002,"5min":3.8090000000000002,"15min":24.943000000000001},"last":0.621}]},{"osd":2,"last update":"Wed Feb 10 04:52:39 2021","interfaces":[{"interface":"back","average":{"1min":8.5600000000000005,"5min":2.625,"15min":1.6830000000000001},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":120.39,"5min":120.39,"15min":120.39},"last":0.77000000000000002},{"interface":"front","average":{"1min":8.6189999999999998,"5min":2.629,"15min":1.694},"min":{"1min":0.66100000000000003,"5min":0.60299999999999998,"15min":0.51900000000000002},"max":{"1min":120.583,"5min":120.583,"15min":120.583},"last":0.56200000000000006}]},{"osd":3,"last update":"Wed Feb 10 04:52:59 2021","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.2709999999999999,"15min":1.2949999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.657,"5min":2.1429999999999998,"15min":2.1429999999999998},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.484,"15min":1.514},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":3.5699999999999998,"5min":3.5699999999999998,"15min":3.5699999999999998},"last":0.97899999999999998}]},{"osd":5,"last update":"Wed Feb 10 04:52:50 2021","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.724,"15min":1.54},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.60999999999999999},"max":{"1min":1.9390000000000001,"5min":35.774000000000001,"15min":35.774000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":1.121,"5min":1.669,"15min":1.474},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.58799999999999997},"max":{"1min":1.9430000000000001,"5min":35.606000000000002,"15min":35.606000000000002},"last":4.319}]},{"osd":6,"last update":"Wed Feb 10 04:52:59 2021","interfaces":[{"interface":"back","average":{"1min":2.0339999999999998,"5min":3.9470000000000001,"15min":2.5299999999999998},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.76600000000000001},"max":{"1min":12.488,"5min":201.626,"15min":201.626},"last":1.3360000000000001},{"interface":"front","average":{"1min":2.1179999999999999,"5min":3.907,"15min":2.5840000000000001},"min":{"1min":0.91200000000000003,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":12.84,"5min":201.577,"15min":201.577},"last":1.0840000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:52:26 2021","interfaces":[{"interface":"back","average":{"1min":1.649,"5min":1.518,"15min":1.512},"min":{"1min":1.01,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":2.4399999999999999,"5min":4.7439999999999998,"15min":6.9199999999999999},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.444,"5min":1.4890000000000001,"15min":1.569},"min":{"1min":0.84599999999999997,"5min":0.72899999999999998,"15min":0.71499999999999997},"max":{"1min":2.1960000000000002,"5min":2.8010000000000002,"15min":5.149},"last":1.157}]}]},{"osd":5,"up_from":37,"seq":158913790297,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":249716,"kb_used_data":205164,"kb_used_omap":0,"kb_used_meta":44544,"kb_avail":93495436,"statfs":{"total":95995035648,"available":95739326464,"internally_reserved":0,"allocated":210087936,"data_stored":275135368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45613056},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:51 2021","interfaces":[{"interface":"back","average":{"1min":5.0590000000000002,"5min":1.714,"15min":1.204},"min":{"1min":0.45700000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":58.624000000000002,"5min":58.624000000000002,"15min":58.624000000000002},"last":0.64600000000000002},{"interface":"front","average":{"1min":5.04,"5min":1.784,"15min":1.3009999999999999},"min":{"1min":0.56599999999999995,"5min":0.46600000000000003,"15min":0.46400000000000002},"max":{"1min":57.880000000000003,"5min":57.880000000000003,"15min":57.880000000000003},"last":1.284}]},{"osd":1,"last update":"Wed Feb 10 04:52:51 2021","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.90600000000000003,"15min":0.94499999999999995},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.6200000000000001,"5min":4.4000000000000004,"15min":5.29},"last":6.391},{"interface":"front","average":{"1min":1.573,"5min":1.238,"15min":1.198},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":11.818,"5min":11.818,"15min":11.875999999999999},"last":0.63900000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:52:51 2021","interfaces":[{"interface":"back","average":{"1min":3.7919999999999998,"5min":1.706,"15min":1.5600000000000001},"min":{"1min":0.53800000000000003,"5min":0.53700000000000003,"15min":0.50600000000000001},"max":{"1min":50.948999999999998,"5min":50.948999999999998,"15min":50.948999999999998},"last":0.70399999999999996},{"interface":"front","average":{"1min":3.9359999999999999,"5min":1.732,"15min":1.5760000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":50.149000000000001,"5min":50.149000000000001,"15min":50.149000000000001},"last":6.7130000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:53:03 2021","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.2769999999999999,"15min":1.3149999999999999},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.5069999999999999,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":1.034},{"interface":"front","average":{"1min":1.21,"5min":1.1859999999999999,"15min":1.1819999999999999},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.9199999999999999,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":6.5129999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:52:51 2021","interfaces":[{"interface":"back","average":{"1min":1.6160000000000001,"5min":1.367,"15min":1.353},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.57099999999999995},"max":{"1min":9.4890000000000008,"5min":9.4890000000000008,"15min":9.4890000000000008},"last":0.873},{"interface":"front","average":{"1min":1.288,"5min":1.3680000000000001,"15min":1.409},"min":{"1min":0.85599999999999998,"5min":0.68100000000000005,"15min":0.67500000000000004},"max":{"1min":1.6890000000000001,"5min":5.6180000000000003,"15min":5.6180000000000003},"last":0.93500000000000005}]},{"osd":6,"last update":"Wed Feb 10 04:53:03 2021","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":2.2799999999999998,"15min":3.0350000000000001},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.65800000000000003},"max":{"1min":2.7559999999999998,"5min":38.014000000000003,"15min":319.51999999999998},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.7250000000000001,"5min":2.2170000000000001,"15min":2.956},"min":{"1min":0.71399999999999997,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":4.8840000000000003,"5min":37.896999999999998,"15min":319.22300000000001},"last":6.8220000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:52:32 2021","interfaces":[{"interface":"back","average":{"1min":1.8899999999999999,"5min":1.5860000000000001,"15min":1.5349999999999999},"min":{"1min":0.83599999999999997,"5min":0.75600000000000001,"15min":0.752},"max":{"1min":11.897,"5min":11.897,"15min":11.897},"last":1.163},{"interface":"front","average":{"1min":1.5600000000000001,"5min":1.6220000000000001,"15min":1.619},"min":{"1min":1.107,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.2229999999999999,"5min":6.0369999999999999,"15min":6.0369999999999999},"last":1.415}]}]}]}} 2021-02-10T04:53:30.949 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pool set zone.rgw.log pgp_num 22 2021-02-10T04:53:31.176 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:53:31.617 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is unset 2021-02-10T04:53:31.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:31 smithi164 conmon[29288]: cluster 2021-02-10T04:53:30.452142+0000 mgr.y 2021-02-10T04:53:31.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:31 smithi164 conmon[29288]: (mgr.14140) 1290 : cluster [DBG] pgmap v2121: 197 pgs: 4 active+clean+remapped, 25 active+undersized+degraded, 37 active+undersized, 131 active+clean; 868 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 227/2199 objects degraded (10.323%); 34/2199 objects misplaced (1.546%) 2021-02-10T04:53:31.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:31 smithi164 conmon[29288]: cluster 2021-02-10T04:53:30.607722+0000 mon.a (mon.0) 2497 : cluster [DBG] osdmap e1176: 8 total, 7 up, 8 in 2021-02-10T04:53:31.617 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:31 smithi164 conmon[29288]: 2021-02-10T04:53:31.618 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:31 smithi164 conmon[29288]: audit 2021-02-10T04:53 2021-02-10T04:53:31.618 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:31 smithi164 conmon[29288]: :31.403707+0000 mon.a (mon.0) 2498 : audit [INF] from='client.? 172.21.15.85:0/4193054183' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:53:31.875 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:31 smithi085 conmon[29319]: cluster 2021-02-10T04:53:30.452142+0000 mgr.y (mgr.14140) 1290 : cluster [DBG] pgmap v2121: 197 pgs: 4 active+clean+remapped, 25 active+undersized+degraded, 37 active+undersized, 131 active+clean; 868 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 227/2199 objects degraded (10.323%); 34/2199 objects misplaced (1.546%) 2021-02-10T04:53:31.875 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:31 smithi085 conmon[29319]: cluster 2021-02-10T04:53:30.607722+0000 mon.a (mon.0) 2497 : cluster [DBG] osdmap e1176: 8 total, 7 up, 8 in 2021-02-10T04:53:31.875 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:31 smithi085 conmon[29319]: audit 2021-02-10T04:53:31.403707+0000 mon.a (mon.0) 2498 : audit [INF] from='client.? 172.21.15.85:0/4193054183' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:53:31.875 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:31 smithi085 conmon[35961]: cluster 2021-02-10T04:53:31.875 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:31 smithi085 conmon[35961]: 2021-02-10T04:53:30.452142+0000 2021-02-10T04:53:31.875 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:31 smithi085 conmon[35961]: mgr.y (mgr.14140) 1290 : cluster [DBG] pgmap v2121: 197 pgs: 4 active+clean+remapped, 25 active+undersized+degraded, 37 active+undersized, 131 active+clean; 868 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 227/2199 objects degraded (10.323%); 34/2199 objects misplaced (1.546%) 2021-02-10T04:53:31.875 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:31 smithi085 conmon[35961]: cluster 2021-02-10T04:53:30.607722+0000 2021-02-10T04:53:31.875 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:31 smithi085 conmon[35961]: mon.a (mon.0) 2497 : cluster [DBG] osdmap e1176: 8 total, 7 up, 8 in 2021-02-10T04:53:31.876 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:31 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:53:31.876 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:31 smithi085 conmon[35961]: 02-10T04:53:31.403707+0000 mon.a 2021-02-10T04:53:31.876 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:31 smithi085 conmon[35961]: (mon.0) 2498 : audit [INF] from='client.? 172.21.15.85:0/4193054183' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:53:32.631 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[35961]: cluster 2021-02-10T04:53:31.605328+0000 2021-02-10T04:53:32.631 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[35961]: mon.a (mon.0) 2499 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:53:32.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[29319]: cluster 2021-02-10T04:53:32.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[29319]: 2021-02-10T04:53:32.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[29319]: 2021 2021-02-10T04:53:32.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[29319]: - 2021-02-10T04:53:32.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[29319]: 02-10T04:53:31. 2021-02-10T04:53:32.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[29319]: 605328+0000 mon.a (mon.0) 2499 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:53:32.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[29319]: audit 2021 2021-02-10T04:53:32.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[29319]: -02-10T04:53: 2021-02-10T04:53:32.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[29319]: 31.608544+0000 mon.a (mon 2021-02-10T04:53:32.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[29319]: .0) 2500 : audit [INF] 2021-02-10T04:53:32.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[29319]: from='client.? 172.21.15.85:0/4193054183' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:53:32.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:53:32.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[29319]: 02-10T04:53: 2021-02-10T04:53:32.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[29319]: 31.608611+0000 mon.a ( 2021-02-10T04:53:32.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[29319]: mon.0) 2501 : cluster 2021-02-10T04:53:32.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[29319]: [DBG] osdmap e1177: 8 total, 7 up, 8 in 2021-02-10T04:53:32.632 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[29319]: audit 2021 2021-02-10T04:53:32.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[29319]: -02-10T04:53 2021-02-10T04:53:32.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[29319]: :32.453827+0000 mon.a ( 2021-02-10T04:53:32.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[29319]: mon.0) 2502 : audit 2021-02-10T04:53:32.633 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[29319]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "29"}]: dispatch 2021-02-10T04:53:32.817 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:53:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:32 smithi164 conmon[29288]: cluster 2021-02-10T04:53:31.605328+0000 2021-02-10T04:53:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:32 smithi164 conmon[29288]: mon.a (mon.0) 2499 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:53:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:32 smithi164 conmon[29288]: audit 2021-02-10T04:53:31.608544+0000 mon.a (mon.0) 2500 : audit [INF] from='client.? 172.21.15.85:0/4193054183' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:53:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:32 smithi164 conmon[29288]: cluster 2021-02-10T04:53:31.608611 2021-02-10T04:53:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:32 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2501 : cluster [DBG] 2021-02-10T04:53:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:32 smithi164 conmon[29288]: osdmap e1177: 8 total, 7 up, 8 in 2021-02-10T04:53:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:32 smithi164 conmon[29288]: audit 2021-02-10T04:53:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:32 smithi164 conmon[29288]: 2021-02-10T04:53:32.453827+0000 mon.a 2021-02-10T04:53:32.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:32 smithi164 conmon[29288]: (mon.0) 2502 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "29"}]: dispatch 2021-02-10T04:53:32.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:53:32.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[35961]: T04:53:31.608544+0000 mon.a (mon.0) 2500 : audit [INF] from='client.? 172.21.15.85:0/4193054183' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:53:32.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:53:32.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[35961]: 02-10T04:53:31.608611+0000 mon.a ( 2021-02-10T04:53:32.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[35961]: mon.0) 2501 : cluster [DBG] osdmap e1177: 8 total, 7 up, 8 in 2021-02-10T04:53:32.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[35961]: audit 2021 2021-02-10T04:53:32.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[35961]: -02-10T04:53:32.453827+0000 2021-02-10T04:53:32.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[35961]: mon.a (mon.0) 2502 : audit [INF] 2021-02-10T04:53:32.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:32 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "29"}]: dispatch 2021-02-10T04:53:33.030 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:53:33.552 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.7 dump_historic_ops 2021-02-10T04:53:33.779 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[35961]: cluster 2021-02-10T04:53:32.373323+0000 osd.5 2021-02-10T04:53:33.779 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[35961]: (osd.5) 183 : cluster [DBG] 4.18 scrub ok 2021-02-10T04:53:33.779 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:53:33.779 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[35961]: 53:32.452861+0000 mgr.y (mgr.14140) 1291 : cluster [DBG] pgmap v2124: 196 pgs: 4 active+clean+remapped, 25 active+undersized+degraded, 37 active+undersized, 130 active+clean; 868 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 227/2199 objects degraded (10.323%); 34/2199 objects misplaced (1.546%) 2021-02-10T04:53:33.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:53:33.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[35961]: -10T04:53:32.621218+0000 mon.a ( 2021-02-10T04:53:33.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[35961]: mon.0) 2503 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "29"}]': finished 2021-02-10T04:53:33.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:53:33.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[35961]: -02-10T04:53:32.621274+0000 2021-02-10T04:53:33.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[35961]: mon.a (mon.0) 2504 : cluster [DBG] 2021-02-10T04:53:33.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[35961]: osdmap e1178: 8 total, 7 up, 8 in 2021-02-10T04:53:33.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[35961]: cluster 2021-02-10T04:53:32 2021-02-10T04:53:33.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[35961]: .953690+0000 osd.2 (osd.2) 2022 : 2021-02-10T04:53:33.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[35961]: cluster [DBG] 4.16 scrub ok 2021-02-10T04:53:33.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:53:33.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[35961]: 53:33.305823+0000 osd.3 (osd.3) 2021-02-10T04:53:33.780 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[35961]: 31 : cluster [DBG] 1.4 scrub ok 2021-02-10T04:53:33.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:53:33.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[35961]: 10T04:53:33.515179+0000 mon.a (mon.0) 2505 : cluster [WRN] Health check update: Degraded data redundancy: 227/2199 objects degraded (10.323%), 25 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2021-02-10T04:53:33.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:53:33.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[35961]: 53:33.525848+0000 mon.a (mon.0 2021-02-10T04:53:33.781 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[35961]: ) 2506 : cluster [DBG] osdmap e1179: 8 total, 7 up, 8 in 2021-02-10T04:53:33.781 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[29319]: cluster 2021-02-10T04:53:32. 2021-02-10T04:53:33.781 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[29319]: 373323+0000 osd.5 (osd.5) 183 : cluster [DBG] 4.18 scrub ok 2021-02-10T04:53:33.781 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[29319]: cluster 2021-02-10T04:53:32.452861+0000 mgr.y (mgr.14140) 1291 : cluster [DBG] pgmap v2124: 196 pgs: 4 active+clean+remapped, 25 active+undersized+degraded, 37 active+undersized, 130 active+clean; 868 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 227/2199 objects degraded (10.323%); 34/2199 objects misplaced (1.546%) 2021-02-10T04:53:33.781 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[29319]: audit 2021-02-10T04:53:32.621218+0000 mon.a (mon.0) 2503 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "29"}]': finished 2021-02-10T04:53:33.781 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[29319]: cluster 2021-02-10T04:53: 2021-02-10T04:53:33.781 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[29319]: 32.621274+0000 mon.a (mon.0) 2504 : cluster [DBG] osdmap e1178: 8 total, 7 up, 8 in 2021-02-10T04:53:33.782 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:53:33.782 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[29319]: 10T04:53:32.953690+0000 2021-02-10T04:53:33.782 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[29319]: osd.2 (osd.2) 2022 : cluster [DBG] 2021-02-10T04:53:33.782 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[29319]: 4.16 scrub ok 2021-02-10T04:53:33.782 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:53:33.782 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[29319]: 04:53:33.305823+0000 osd.3 (osd.3 2021-02-10T04:53:33.782 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[29319]: ) 31 : cluster [DBG] 1.4 scrub ok 2021-02-10T04:53:33.782 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:53:33.782 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[29319]: -10T04:53:33.515179+0000 mon.a 2021-02-10T04:53:33.782 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[29319]: (mon.0) 2505 : cluster [WRN] Health check update: Degraded data redundancy: 227/2199 objects degraded (10.323%), 25 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2021-02-10T04:53:33.782 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:53:33.782 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[29319]: -10T04:53:33.525848+0000 mon.a 2021-02-10T04:53:33.783 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:33 smithi085 conmon[29319]: (mon.0) 2506 : cluster [DBG] osdmap e1179: 8 total, 7 up, 8 in 2021-02-10T04:53:33.842 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:53:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:33 smithi164 conmon[29288]: cluster 2021-02-10T04:53:32.373323+0000 osd.5 2021-02-10T04:53:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:33 smithi164 conmon[29288]: (osd.5) 183 : cluster [DBG] 4.18 scrub ok 2021-02-10T04:53:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:33 smithi164 conmon[29288]: cluster 2021-02-10T04:53:32.452861+0000 mgr.y (mgr.14140) 1291 : cluster [DBG] pgmap v2124: 196 pgs: 4 active+clean+remapped, 25 active+undersized+degraded, 37 active+undersized, 130 active+clean; 868 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 227/2199 objects degraded (10.323%); 34/2199 objects misplaced (1.546%) 2021-02-10T04:53:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:33 smithi164 conmon[29288]: audit 2021-02-10T04:53:32.621218+0000 mon.a (mon.0) 2503 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "29"}]': finished 2021-02-10T04:53:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:33 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:53:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:33 smithi164 conmon[29288]: :53:32.621274+0000 mon.a (mon.0) 2504 : cluster [DBG] osdmap e1178: 8 total, 7 up, 8 in 2021-02-10T04:53:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:33 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:53:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:33 smithi164 conmon[29288]: 04:53:32.953690+0000 osd.2 (osd 2021-02-10T04:53:33.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:33 smithi164 conmon[29288]: .2) 2022 : cluster [DBG] 4.16 scrub ok 2021-02-10T04:53:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:33 smithi164 conmon[29288]: 2021-02-10T04:53:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:33 smithi164 conmon[29288]: cluster 2021-02-10T04:53:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:33 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:53:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:33 smithi164 conmon[29288]: 53:33.305823+0000 osd.3 (osd.3) 31 : 2021-02-10T04:53:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:33 smithi164 conmon[29288]: cluster [DBG] 1.4 scrub ok 2021-02-10T04:53:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:33 smithi164 conmon[29288]: cluster 2021-02-10T04:53:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:33 smithi164 conmon[29288]: 2021-02-10T04:53 2021-02-10T04:53:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:33 smithi164 conmon[29288]: :33.515179+0000 mon.a (mon.0) 2505 : cluster 2021-02-10T04:53:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:33 smithi164 conmon[29288]: [WRN] Health check update: Degraded data redundancy: 227/2199 objects degraded (10.323%), 25 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2021-02-10T04:53:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:33 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:53:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:33 smithi164 conmon[29288]: 02-10T04:53:33.525848+0000 mon.a (mon. 2021-02-10T04:53:33.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:33 smithi164 conmon[29288]: 0) 2506 : cluster [DBG] osdmap e1179: 8 total, 7 up, 8 in 2021-02-10T04:53:34.547 INFO:teuthology.orchestra.run.smithi085.stderr:set pool 4 pgp_num to 22 2021-02-10T04:53:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:34 smithi164 conmon[29288]: cluster 2021-02-10T04:53:33.948342+0000 osd.2 (osd.2) 2023 2021-02-10T04:53:34.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:34 smithi164 conmon[29288]: : cluster [DBG] 5.4 scrub ok 2021-02-10T04:53:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:34 smithi164 conmon[29288]: cluster 2021-02-10T04:53:34.122148+0000 osd.4 (osd.4) 2647 : cluster [DBG] 4.19 scrub ok 2021-02-10T04:53:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:34 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:53:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:34 smithi164 conmon[29288]: 10T04:53:34.289308+0000 mon.c (mon.1) 109 : audit [INF] from='client.? 172.21.15.85:0/2596366384' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pgp_num", "val": "22"}]: dispatch 2021-02-10T04:53:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:34 smithi164 conmon[29288]: audit 2021-02-10T04:53:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:34 smithi164 conmon[29288]: 2021-02-10T04:53:34.289703+0000 mon.a (mon.0) 2507 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pgp_num", "val": "22"}]: dispatch 2021-02-10T04:53:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:34 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:53:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:34 smithi164 conmon[29288]: :53:34.534061+0000 mon.a (mon.0) 2021-02-10T04:53:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:34 smithi164 conmon[29288]: 2508 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pgp_num", "val": "22"}]': finished 2021-02-10T04:53:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:34 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:53:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:34 smithi164 conmon[29288]: 04:53:34.534140+0000 mon.a (mon 2021-02-10T04:53:34.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:34 smithi164 conmon[29288]: .0) 2509 : cluster [DBG] osdmap e1180: 8 total, 7 up, 8 in 2021-02-10T04:53:34.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[29319]: cluster 2021-02-10T04:53:33.948342+0000 osd.2 (osd.2) 2023 : cluster [DBG] 5.4 scrub ok 2021-02-10T04:53:34.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[29319]: cluster 2021-02-10T04:53:34.122148+0000 osd.4 (osd.4) 2647 : cluster [DBG] 4.19 scrub ok 2021-02-10T04:53:34.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[29319]: audit 2021-02-10T04:53:34.289308+0000 mon.c (mon.1) 109 : audit [INF] from='client.? 172.21.15.85:0/2596366384' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pgp_num", "val": "22"}]: dispatch 2021-02-10T04:53:34.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[29319]: audit 2021-02-10T04:53:34.289703+0000 mon.a (mon.0) 2507 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pgp_num", "val": "22"}]: dispatch 2021-02-10T04:53:34.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[29319]: audit 2021-02-10T04:53:34.534061+0000 mon.a (mon.0) 2508 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pgp_num", "val": "22"}]': finished 2021-02-10T04:53:34.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[29319]: cluster 2021-02-10T04:53:34.534140+0000 mon.a (mon.0) 2509 : cluster [DBG] osdmap e1180: 8 total, 7 up, 8 in 2021-02-10T04:53:34.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[35961]: cluster 2021-02-10T04:53:34.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:53:34.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[35961]: :53:33.948342+0000 osd.2 (osd.2) 2023 : cluster [DBG] 5.4 scrub ok 2021-02-10T04:53:34.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:53:34.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[35961]: T04:53:34.122148+0000 osd.4 (osd.4) 2647 : cluster [DBG] 2021-02-10T04:53:34.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[35961]: 4.19 scrub ok 2021-02-10T04:53:34.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:53:34.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[35961]: 04:53:34.289308+0000 mon.c 2021-02-10T04:53:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[35961]: (mon.1) 109 : audit 2021-02-10T04:53:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[35961]: [INF] from='client.? 172.21.15.85:0/2596366384' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pgp_num", "val": "22"}]: dispatch 2021-02-10T04:53:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[35961]: audit 2021 2021-02-10T04:53:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[35961]: -02-10T04:53:34 2021-02-10T04:53:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[35961]: .289703+0000 mon.a (mon. 2021-02-10T04:53:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[35961]: 0) 2507 : audit [INF] 2021-02-10T04:53:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[35961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pgp_num", "val": "22"}]: dispatch 2021-02-10T04:53:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[35961]: audit 2021 2021-02-10T04:53:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[35961]: -02-10T04:53:34 2021-02-10T04:53:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[35961]: .534061+0000 mon.a (mon. 2021-02-10T04:53:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[35961]: 0) 2508 : audit [INF] 2021-02-10T04:53:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[35961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pgp_num", "val": "22"}]': finished 2021-02-10T04:53:34.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[35961]: cluster 2021-02-10T04:53:34.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:53:34.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[35961]: 10T04:53:34.534140 2021-02-10T04:53:34.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2021-02-10T04:53:34.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:34 smithi085 conmon[35961]: 2509 : cluster [DBG] osdmap e1180: 8 total, 7 up, 8 in 2021-02-10T04:53:35.261 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset nodeep-scrub 2021-02-10T04:53:35.477 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:53:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:35 smithi164 conmon[29288]: cluster 2021-02-10T04:53:34.178944+0000 osd.1 (osd.1) 498 : cluster [DBG] 2.d scrub ok 2021-02-10T04:53:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:35 smithi164 conmon[29288]: cluster 2021-02-10T04:53:34.407780+0000 osd.5 (osd.5) 184 : cluster [DBG] 7.0 scrub ok 2021-02-10T04:53:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:35 smithi164 conmon[29288]: cluster 2021-02-10T04:53:34.453637+0000 mgr.y (mgr.14140) 1292 : cluster [DBG] pgmap v2127: 196 pgs: 4 active+clean+remapped, 25 active+undersized+degraded, 37 active+undersized, 130 active+clean; 868 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 227/2199 objects degraded (10.323%); 34/2199 objects misplaced (1.546%) 2021-02-10T04:53:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:35 smithi164 conmon[29288]: cluster 2021-02-10T04:53:34.865571+0000 osd.6 (osd.6) 181 : cluster [DBG] 5.1f scrub ok 2021-02-10T04:53:35.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:35 smithi164 conmon[29288]: cluster 2021-02-10T04:53:34.927420+0000 osd.2 (osd.2) 2024 : cluster [DBG] 6.14 scrub ok 2021-02-10T04:53:35.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:35 smithi164 conmon[29288]: cluster 2021-02-10T04:53:35.302773+0000 osd.3 (osd.3 2021-02-10T04:53:35.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:35 smithi164 conmon[29288]: ) 32 : cluster [DBG] 7.10 scrub ok 2021-02-10T04:53:35.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:35 smithi164 conmon[29288]: cluster 2021-02-10T04:53:35.535535+0000 mon.a (mon.0) 2510 : cluster [DBG] osdmap e1181: 8 total, 7 up, 8 in 2021-02-10T04:53:35.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:35 smithi085 conmon[29319]: cluster 2021-02-10T04:53:34.178944+0000 osd.1 ( 2021-02-10T04:53:35.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:35 smithi085 conmon[29319]: osd.1) 498 : cluster [DBG] 2.d scrub ok 2021-02-10T04:53:35.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:35 smithi085 conmon[29319]: cluster 2021-02-10T04:53:34.407780+0000 osd.5 (osd.5) 184 : cluster [DBG] 7.0 scrub ok 2021-02-10T04:53:35.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:35 smithi085 conmon[29319]: cluster 2021-02-10T04:53:34.453637+0000 mgr.y (mgr.14140) 1292 : cluster [DBG] pgmap v2127: 196 pgs: 4 active+clean+remapped, 25 active+undersized+degraded, 37 active+undersized, 130 active+clean; 868 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 227/2199 objects degraded (10.323%); 34/2199 objects misplaced (1.546%) 2021-02-10T04:53:35.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:35 smithi085 conmon[29319]: cluster 2021-02-10T04:53:34.865571+0000 osd.6 (osd.6) 181 : cluster [DBG] 5.1f scrub ok 2021-02-10T04:53:35.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:35 smithi085 conmon[29319]: cluster 2021-02-10T04:53:34.927420+0000 osd.2 (osd.2) 2024 : cluster [DBG] 6.14 scrub ok 2021-02-10T04:53:35.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:35 smithi085 conmon[29319]: cluster 2021-02-10T04:53:35.302773+0000 osd.3 (osd.3) 32 : cluster [DBG] 7.10 scrub ok 2021-02-10T04:53:35.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:35 smithi085 conmon[29319]: cluster 2021-02-10T04:53:35.535535+0000 mon.a (mon.0) 2510 : cluster [DBG] osdmap e1181: 8 total, 7 up, 8 in 2021-02-10T04:53:35.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:35 smithi085 conmon[35961]: cluster 2021-02-10T04:53:34.178944+0000 osd.1 (osd.1 2021-02-10T04:53:35.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:35 smithi085 conmon[35961]: ) 498 : cluster [DBG] 2.d scrub ok 2021-02-10T04:53:35.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:35 smithi085 conmon[35961]: cluster 2021-02-10T04:53:34.407780+0000 osd.5 (osd.5) 184 : cluster [DBG] 7.0 scrub ok 2021-02-10T04:53:35.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:35 smithi085 conmon[35961]: cluster 2021-02-10T04:53 2021-02-10T04:53:35.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:35 smithi085 conmon[35961]: :34.453637+0000 mgr.y (mgr.14140) 1292 : cluster [DBG] pgmap v2127: 196 pgs: 4 active+clean+remapped, 25 active+undersized+degraded, 37 active+undersized, 130 active+clean; 868 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 227/2199 objects degraded (10.323%); 34/2199 objects misplaced (1.546%) 2021-02-10T04:53:35.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:35 smithi085 conmon[35961]: cluster 2021-02-10T04:53:35.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:35 smithi085 conmon[35961]: 2021-02-10T04:53:34.865571+0000 osd.6 (osd.6) 181 : cluster [DBG] 2021-02-10T04:53:35.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:35 smithi085 conmon[35961]: 5.1f scrub ok 2021-02-10T04:53:35.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:35 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:53:35.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:35 smithi085 conmon[35961]: 02-10T04:53:34.927420+0000 osd.2 (osd. 2021-02-10T04:53:35.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:35 smithi085 conmon[35961]: 2) 2024 : cluster [DBG] 6.14 scrub ok 2021-02-10T04:53:35.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:35 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:53:35.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:35 smithi085 conmon[35961]: 02-10T04:53:35.302773+0000 2021-02-10T04:53:35.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:35 smithi085 conmon[35961]: osd.3 (osd.3) 32 : cluster [DBG] 2021-02-10T04:53:35.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:35 smithi085 conmon[35961]: 7.10 scrub ok 2021-02-10T04:53:35.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:35 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:53:35.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:35 smithi085 conmon[35961]: :53:35.535535+0000 mon.a (mon. 2021-02-10T04:53:35.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:35 smithi085 conmon[35961]: 0) 2510 : cluster [DBG] osdmap e1181: 8 total, 7 up, 8 in 2021-02-10T04:53:36.551 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:53:36.571 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:53:36.596 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:53:36.618 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:53:36.642 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:53:36.685 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:53:36.705 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:53:36.705 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-02-10T04:53:36.705 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: problem getting command descriptions from osd.0 2021-02-10T04:53:36.705 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:53:36.705 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:53:36.706 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:53:36.706 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:53:36.706 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:53:36.706 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:53:36.706 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:53:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:36 smithi164 conmon[29288]: cluster 2021-02-10T04:53:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:36 smithi164 conmon[29288]: 2021-02-10T04:53:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:36 smithi164 conmon[29288]: 2021 2021-02-10T04:53:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:36 smithi164 conmon[29288]: - 2021-02-10T04:53:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:36 smithi164 conmon[29288]: 02 2021-02-10T04:53:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:36 smithi164 conmon[29288]: -10T04:53: 2021-02-10T04:53:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:36 smithi164 conmon[29288]: 35.216386+0000 osd.1 (osd.1) 499 : cluster [DBG] 5.12 scrub ok 2021-02-10T04:53:36.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:36 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:53:36.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:36 smithi164 conmon[29288]: 10T04:53:35.858553+0000 osd.6 (osd.6 2021-02-10T04:53:36.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:36 smithi164 conmon[29288]: ) 182 : cluster [DBG] 6.25 scrub ok 2021-02-10T04:53:36.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:36 smithi164 conmon[29288]: cluster 2021-02-10T04:53:36.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:36 smithi164 conmon[29288]: 2021-02- 2021-02-10T04:53:36.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:36 smithi164 conmon[29288]: 10T04:53:35.876678+0000 2021-02-10T04:53:36.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:36 smithi164 conmon[29288]: osd.2 (osd.2) 2025 : cluster [DBG] 6.12 scrub ok 2021-02-10T04:53:36.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:36 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:53:36.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:36 smithi164 conmon[29288]: -10T04:53:36.102921+0000 osd.4 (osd.4) 2648 2021-02-10T04:53:36.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:36 smithi164 conmon[29288]: : cluster [DBG] 2.25 scrub ok 2021-02-10T04:53:36.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:36 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:53:36.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:36 smithi164 conmon[29288]: 10T04:53:36.455169+0000 mon.a (mon. 2021-02-10T04:53:36.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:36 smithi164 conmon[29288]: 0) 2511 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "28"}]: dispatch 2021-02-10T04:53:36.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:36 smithi164 conmon[29288]: audit 2021-02-10T04:53:36.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:36 smithi164 conmon[29288]: 2021-02-10T04:53 2021-02-10T04:53:36.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:36 smithi164 conmon[29288]: :36.531608+0000 mon.c (mon.1) 110 : audit 2021-02-10T04:53:36.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:36 smithi164 conmon[29288]: [DBG] from='client.? 172.21.15.85:0/3911672410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:53:36.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:36 smithi085 conmon[29319]: cluster 2021-02-10T04:53:35.216386+0000 osd.1 2021-02-10T04:53:36.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:36 smithi085 conmon[29319]: (osd.1) 499 : cluster [DBG] 5.12 scrub ok 2021-02-10T04:53:36.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:36 smithi085 conmon[29319]: cluster 2021-02-10T04:53:35.858553+0000 osd.6 (osd.6) 182 : cluster [DBG] 6.25 scrub ok 2021-02-10T04:53:36.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:36 smithi085 conmon[29319]: cluster 2021-02-10T04:53:35.876678+0000 osd.2 (osd.2) 2025 : cluster [DBG] 6.12 scrub ok 2021-02-10T04:53:36.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:36 smithi085 conmon[29319]: cluster 2021-02-10T04:53:36.102921+0000 osd.4 (osd.4) 2648 : cluster [DBG] 2.25 scrub ok 2021-02-10T04:53:36.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:36 smithi085 conmon[29319]: audit 2021-02-10T04:53:36.455169+0000 mon.a (mon.0) 2511 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "28"}]: dispatch 2021-02-10T04:53:36.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:36 smithi085 conmon[29319]: audit 2021-02-10T04:53:36.531608+0000 mon.c (mon.1) 110 : audit [DBG] from='client.? 172.21.15.85:0/3911672410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:53:36.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:36 smithi085 conmon[35961]: cluster 2021-02-10T04:53:36.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:36 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:53:36.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:36 smithi085 conmon[35961]: 04:53:35.216386+0000 osd.1 (osd.1) 499 : cluster [DBG] 5.12 scrub ok 2021-02-10T04:53:36.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:36 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:53:36.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:36 smithi085 conmon[35961]: 10T04:53:35.858553+0000 osd.6 2021-02-10T04:53:36.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:36 smithi085 conmon[35961]: (osd.6) 182 : cluster [DBG] 6.25 scrub ok 2021-02-10T04:53:36.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:36 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:53:36.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:36 smithi085 conmon[35961]: 02-10T04:53:35.876678+0000 osd.2 (osd 2021-02-10T04:53:36.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:36 smithi085 conmon[35961]: .2) 2025 : cluster [DBG] 6.12 scrub ok 2021-02-10T04:53:36.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:36 smithi085 conmon[35961]: cluster 2021-02-10T04:53:36.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:36 smithi085 conmon[35961]: 2021-02-10T04:53:36 2021-02-10T04:53:36.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:36 smithi085 conmon[35961]: .102921+0000 osd.4 (osd.4) 2648 : 2021-02-10T04:53:36.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:36 smithi085 conmon[35961]: cluster [DBG] 2.25 scrub ok 2021-02-10T04:53:36.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:36 smithi085 conmon[35961]: audit 2021-02-10T04:53:36.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:36 smithi085 conmon[35961]: 2021- 2021-02-10T04:53:36.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:36 smithi085 conmon[35961]: 02-10T04:53:36.455169+0000 2021-02-10T04:53:36.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:36 smithi085 conmon[35961]: mon.a (mon.0) 2511 : audit [INF] 2021-02-10T04:53:36.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:36 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "28"}]: dispatch 2021-02-10T04:53:36.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:36 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:53:36.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:36 smithi085 conmon[35961]: 10T04:53:36.531608+0000 mon.c (mon.1) 2021-02-10T04:53:36.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:36 smithi085 conmon[35961]: 110 : audit [DBG] from='client.? 172.21.15.85:0/3911672410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:53:37.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[29319]: cluster 2021-02-10T04:53:36 2021-02-10T04:53:37.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[29319]: .454303+0000 mgr.y (mgr.14140) 1293 : cluster [DBG] pgmap v2130: 195 pgs: 4 active+clean+remapped, 25 active+undersized+degraded, 37 active+undersized, 129 active+clean; 868 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 227/2169 objects degraded (10.466%); 34/2169 objects misplaced (1.568%) 2021-02-10T04:53:37.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[29319]: audit 2021-02-10T04:53:36 2021-02-10T04:53:37.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[29319]: .641891+0000 mon.a (mon.0) 2512 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "28"}]': finished 2021-02-10T04:53:37.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:53:37.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[29319]: 10T04:53:36.641971 2021-02-10T04:53:37.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2513 2021-02-10T04:53:37.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[29319]: : cluster [DBG] osdmap e1182: 8 total, 7 up, 8 in 2021-02-10T04:53:37.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:53:37.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[29319]: 53:36.836193+0000 osd.6 (osd.6) 183 : cluster [DBG] 2021-02-10T04:53:37.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[29319]: 4.a scrub ok 2021-02-10T04:53:37.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:53:37.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[29319]: :53:37.108673+0000 osd.4 (osd.4) 2649 : cluster [DBG] 4.d scrub ok 2021-02-10T04:53:37.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[29319]: cluster 2021-02-10T04:53:37. 2021-02-10T04:53:37.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[29319]: 384391+0000 osd.3 (osd.3) 33 : cluster [DBG] 7.14 scrub ok 2021-02-10T04:53:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: cluster 2021-02-10T04:53:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: 2021-02-10T04:53:36 2021-02-10T04:53:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: .454303+0000 mgr.y (mgr.14140) 1293 : cluster [DBG] pgmap v2130: 195 pgs: 4 active+clean+remapped, 25 active+undersized+degraded, 37 active+undersized, 129 active+clean; 868 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 227/2169 objects degraded (10.466%); 34/2169 objects misplaced (1.568%) 2021-02-10T04:53:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: audit 2021-02-10T04:53:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: 2021-02-10T04:53:36.641891 2021-02-10T04:53:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2512 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "28"}]': finished 2021-02-10T04:53:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:53:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: -02-10T04:53:36.641971+0000 2021-02-10T04:53:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: mon.a (mon.0) 2513 : cluster [DBG] osdmap e1182: 8 total, 7 up, 8 in 2021-02-10T04:53:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:53:37.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: 02-10T04:53:36.836193+0000 2021-02-10T04:53:37.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: osd.6 (osd.6) 183 : cluster [DBG] 4.a scrub ok 2021-02-10T04:53:37.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: cluster 2021-02-10T04:53:37.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: 2021-02-10T04:53:37.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: 2021 2021-02-10T04:53:37.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: - 2021-02-10T04:53:37.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: 02 2021-02-10T04:53:37.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: - 2021-02-10T04:53:37.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: 10 2021-02-10T04:53:37.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: T 2021-02-10T04:53:37.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: 04 2021-02-10T04:53:37.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: : 2021-02-10T04:53:37.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: 53 2021-02-10T04:53:37.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: : 2021-02-10T04:53:37.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: 37 2021-02-10T04:53:37.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: . 2021-02-10T04:53:37.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: 108673 2021-02-10T04:53:37.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: +0000 2021-02-10T04:53:37.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: 2021-02-10T04:53:37.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: osd.4 2021-02-10T04:53:37.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: ( 2021-02-10T04:53:37.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: osd 2021-02-10T04:53:37.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: . 2021-02-10T04:53:37.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: 4 2021-02-10T04:53:37.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: ) 2021-02-10T04:53:37.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: 2649 : cluster [DBG] 4.d scrub ok 2021-02-10T04:53:37.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: 2021-02-10T04:53:37.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:53:37.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: 53:37.384391+0000 osd.3 (osd.3) 2021-02-10T04:53:37.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:37 smithi085 conmon[35961]: 33 : cluster [DBG] 7.14 scrub ok 2021-02-10T04:53:38.123 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:37 smithi164 conmon[29288]: cluster 2021-02-10T04:53:38.123 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:37 smithi164 conmon[29288]: 2021-02-10T04:53:38.123 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:37 smithi164 conmon[29288]: 2021 2021-02-10T04:53:38.123 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:37 smithi164 conmon[29288]: - 2021-02-10T04:53:38.123 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:37 smithi164 conmon[29288]: 02 2021-02-10T04:53:38.123 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:37 smithi164 conmon[29288]: - 2021-02-10T04:53:38.123 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:37 smithi164 conmon[29288]: 10 2021-02-10T04:53:38.123 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:37 smithi164 conmon[29288]: T 2021-02-10T04:53:38.123 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:37 smithi164 conmon[29288]: 04 2021-02-10T04:53:38.124 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:37 smithi164 conmon[29288]: : 2021-02-10T04:53:38.124 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:37 smithi164 conmon[29288]: 53 2021-02-10T04:53:38.124 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:37 smithi164 conmon[29288]: : 2021-02-10T04:53:38.124 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:37 smithi164 conmon[29288]: 36 2021-02-10T04:53:38.124 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:37 smithi164 conmon[29288]: .454303+0000 mgr.y (mgr. 2021-02-10T04:53:38.124 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:37 smithi164 conmon[29288]: 14140) 1293 : cluster [DBG] pgmap v2130: 195 pgs: 4 active+clean+remapped, 25 active+undersized+degraded, 37 active+undersized, 129 active+clean; 868 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 227/2169 objects degraded (10.466%); 34/2169 objects misplaced (1.568%) 2021-02-10T04:53:38.124 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:37 smithi164 conmon[29288]: audit 2021-02-10T04:53 2021-02-10T04:53:38.124 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:37 smithi164 conmon[29288]: :36.641891+0000 mon.a (mon.0) 2512 : audit 2021-02-10T04:53:38.124 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:37 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "28"}]': finished 2021-02-10T04:53:38.124 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:37 smithi164 conmon[29288]: cluster 2021-02-10T04:53: 2021-02-10T04:53:38.124 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:37 smithi164 conmon[29288]: 36.641971+0000 mon.a (mon.0) 2513 : cluster 2021-02-10T04:53:38.124 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:37 smithi164 conmon[29288]: [DBG] osdmap e1182: 8 total, 7 up, 8 in 2021-02-10T04:53:38.124 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:37 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:53:38.125 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:37 smithi164 conmon[29288]: :53:36.836193+0000 osd.6 (osd. 2021-02-10T04:53:38.125 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:37 smithi164 conmon[29288]: 6) 183 : cluster [DBG] 4.a scrub ok 2021-02-10T04:53:38.125 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:37 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:53:38.125 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:37 smithi164 conmon[29288]: T04:53:37.108673+0000 osd.4 (osd 2021-02-10T04:53:38.125 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:37 smithi164 conmon[29288]: .4) 2649 : cluster [DBG] 4.d scrub ok 2021-02-10T04:53:38.125 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:37 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:53:38.125 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:37 smithi164 conmon[29288]: 10T04:53:37.384391+0000 osd.3 2021-02-10T04:53:38.125 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:37 smithi164 conmon[29288]: (osd.3) 33 : cluster [DBG] 7.14 scrub ok 2021-02-10T04:53:38.164 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:53:38.164 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:53:38.856 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:53:38.856 INFO:teuthology.orchestra.run.smithi164.stdout: "size": 20, 2021-02-10T04:53:38.856 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 600, 2021-02-10T04:53:38.856 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [ 2021-02-10T04:53:38.856 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.856 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.19 1011/1010 [PushOp(7:9fd44245:::smithi08575397-20:212, version: 776'510, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:9fd44245:::smithi08575397-20:212@776'510, size: 2756513, copy_subset: [0~2756513], clone_subset: {}, snapset: 372=[]:{212=[203],27c=[27c],2cb=[2c8,2ae],309=[308,300,2f5,2f0,2ee,2ed,2eb,2ea,2d9],320=[31a],32b=[32a,328],351=[34e],358=[354,353],372=[372,370,36f,36b,368,366,35f]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:53:38.856 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:28.644072+0000", 2021-02-10T04:53:38.856 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 550.20864910800003, 2021-02-10T04:53:38.857 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.01035735, 2021-02-10T04:53:38.857 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:53:38.857 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:53:38.857 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:53:38.857 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.857 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:53:38.857 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.644072+0000", 2021-02-10T04:53:38.857 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:53:38.857 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.857 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.857 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:53:38.857 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.644072+0000", 2021-02-10T04:53:38.858 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001079015 2021-02-10T04:53:38.858 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.858 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.858 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:53:38.858 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.645151+0000", 2021-02-10T04:53:38.858 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1058e-05 2021-02-10T04:53:38.858 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.858 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.858 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:53:38.858 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.645182+0000", 2021-02-10T04:53:38.858 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.4929999999999999e-05 2021-02-10T04:53:38.858 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.858 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.859 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:53:38.859 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.645217+0000", 2021-02-10T04:53:38.859 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7141000000000001e-05 2021-02-10T04:53:38.859 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.859 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.859 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:53:38.859 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.645234+0000", 2021-02-10T04:53:38.859 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0085741570000000007 2021-02-10T04:53:38.859 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.859 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.859 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:53:38.859 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.653808+0000", 2021-02-10T04:53:38.859 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7589999999999997e-06 2021-02-10T04:53:38.860 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.860 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.860 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:53:38.860 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.653816+0000", 2021-02-10T04:53:38.860 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00061328999999999995 2021-02-10T04:53:38.860 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.860 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.860 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:53:38.860 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.654430+0000", 2021-02-10T04:53:38.860 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.01035735 2021-02-10T04:53:38.860 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.860 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:53:38.860 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.861 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.861 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.861 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.19 1011/1010 [PushOp(7:9fd44245:::smithi08575397-20:212, version: 776'510, data_included: [2097152~659361], data_size: 659361, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:9fd44245:::smithi08575397-20:212@776'510, size: 2756513, copy_subset: [0~2756513], clone_subset: {}, snapset: 372=[]:{212=[203],27c=[27c],2cb=[2c8,2ae],309=[308,300,2f5,2f0,2ee,2ed,2eb,2ea,2d9],320=[31a],32b=[32a,328],351=[34e],358=[354,353],372=[372,370,36f,36b,368,366,35f]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2756513, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:53:38.861 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:28.656618+0000", 2021-02-10T04:53:38.861 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 550.19610294100005, 2021-02-10T04:53:38.861 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0099259329999999996, 2021-02-10T04:53:38.861 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:53:38.861 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:53:38.861 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:53:38.861 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.861 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:53:38.861 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.656618+0000", 2021-02-10T04:53:38.861 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:53:38.862 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.862 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.862 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:53:38.862 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.656618+0000", 2021-02-10T04:53:38.862 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0010645349999999999 2021-02-10T04:53:38.862 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.862 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.862 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:53:38.862 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.657683+0000", 2021-02-10T04:53:38.862 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.6270000000000001e-05 2021-02-10T04:53:38.862 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.862 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.863 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:53:38.863 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.657709+0000", 2021-02-10T04:53:38.863 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.0611999999999997e-05 2021-02-10T04:53:38.863 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.863 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.863 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:53:38.863 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.657750+0000", 2021-02-10T04:53:38.863 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8016000000000002e-05 2021-02-10T04:53:38.863 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.863 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.863 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:53:38.863 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.657768+0000", 2021-02-10T04:53:38.863 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0079422280000000008 2021-02-10T04:53:38.864 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.864 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.864 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:53:38.864 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.665710+0000", 2021-02-10T04:53:38.864 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5824e-05 2021-02-10T04:53:38.864 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.864 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.864 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:53:38.864 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.665726+0000", 2021-02-10T04:53:38.864 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00081844800000000005 2021-02-10T04:53:38.864 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.864 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.864 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:53:38.864 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.666544+0000", 2021-02-10T04:53:38.865 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0099259329999999996 2021-02-10T04:53:38.865 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.865 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:53:38.865 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.865 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.865 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.865 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.19 1011/1010 [PushOp(7:9fd44245:::smithi08575397-20:320, version: 947'661, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:9fd44245:::smithi08575397-20:320@947'661, size: 2269242, copy_subset: [0~2269242], clone_subset: {7:9fd44245:::smithi08575397-20:27c=[]}, snapset: 372=[]:{212=[203],27c=[27c],2cb=[2c8,2ae],309=[308,300,2f5,2f0,2ee,2ed,2eb,2ea,2d9],320=[31a],32b=[32a,328],351=[34e],358=[354,353],372=[372,370,36f,36b,368,366,35f]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:53:38.865 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:28.697025+0000", 2021-02-10T04:53:38.865 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 550.155696584, 2021-02-10T04:53:38.865 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010329339, 2021-02-10T04:53:38.865 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:53:38.865 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:53:38.865 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:53:38.866 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.866 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:53:38.866 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.697025+0000", 2021-02-10T04:53:38.866 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:53:38.866 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.866 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.866 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:53:38.866 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.697025+0000", 2021-02-10T04:53:38.866 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0014287740000000001 2021-02-10T04:53:38.866 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.866 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.866 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:53:38.866 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.698453+0000", 2021-02-10T04:53:38.867 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.1725e-05 2021-02-10T04:53:38.867 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.867 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.867 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:53:38.867 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.698475+0000", 2021-02-10T04:53:38.867 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5849000000000001e-05 2021-02-10T04:53:38.867 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.867 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.867 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:53:38.867 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.698521+0000", 2021-02-10T04:53:38.867 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4253999999999999e-05 2021-02-10T04:53:38.867 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.868 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.868 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:53:38.868 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.698535+0000", 2021-02-10T04:53:38.868 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0081349270000000001 2021-02-10T04:53:38.868 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.868 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.868 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:53:38.868 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.706670+0000", 2021-02-10T04:53:38.868 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6552e-05 2021-02-10T04:53:38.868 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.868 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.868 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:53:38.868 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.706687+0000", 2021-02-10T04:53:38.869 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000667258 2021-02-10T04:53:38.869 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.869 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.869 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:53:38.869 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.707354+0000", 2021-02-10T04:53:38.869 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010329339 2021-02-10T04:53:38.869 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.869 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:53:38.869 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.869 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.869 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.869 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.19 1011/1010 [PushOp(7:9fd44245:::smithi08575397-20:32b, version: 953'667, data_included: [2097152~793916], data_size: 793916, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:9fd44245:::smithi08575397-20:32b@953'667, size: 2891068, copy_subset: [0~2891068], clone_subset: {7:9fd44245:::smithi08575397-20:320=[]}, snapset: 372=[]:{212=[203],27c=[27c],2cb=[2c8,2ae],309=[308,300,2f5,2f0,2ee,2ed,2eb,2ea,2d9],320=[31a],32b=[32a,328],351=[34e],358=[354,353],372=[372,370,36f,36b,368,366,35f]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2891068, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:53:38.869 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:28.737408+0000", 2021-02-10T04:53:38.870 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 550.11531332499999, 2021-02-10T04:53:38.870 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010315126000000001, 2021-02-10T04:53:38.870 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:53:38.870 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:53:38.870 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:53:38.870 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.870 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:53:38.870 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.737408+0000", 2021-02-10T04:53:38.870 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:53:38.870 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.870 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.870 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:53:38.870 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.737408+0000", 2021-02-10T04:53:38.871 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.002619101 2021-02-10T04:53:38.871 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.871 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.871 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:53:38.871 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.740027+0000", 2021-02-10T04:53:38.871 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.3180000000000002e-05 2021-02-10T04:53:38.871 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.871 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.871 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:53:38.871 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.740050+0000", 2021-02-10T04:53:38.871 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.4010000000000001e-05 2021-02-10T04:53:38.871 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.871 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.871 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:53:38.872 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.740084+0000", 2021-02-10T04:53:38.872 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.52e-05 2021-02-10T04:53:38.872 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.872 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.872 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:53:38.872 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.740099+0000", 2021-02-10T04:53:38.872 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0067203999999999996 2021-02-10T04:53:38.872 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.872 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.872 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:53:38.872 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.746820+0000", 2021-02-10T04:53:38.872 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1915e-05 2021-02-10T04:53:38.872 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.873 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.873 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:53:38.873 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.746832+0000", 2021-02-10T04:53:38.873 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00089132000000000002 2021-02-10T04:53:38.873 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.873 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.873 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:53:38.873 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.747723+0000", 2021-02-10T04:53:38.873 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010315126000000001 2021-02-10T04:53:38.873 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.873 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:53:38.873 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.873 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.874 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.874 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.19 1011/1010 [PushOp(7:9fd44245:::smithi08575397-20:2cb, version: 986'714, data_included: [1913836~641026], data_size: 641026, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:9fd44245:::smithi08575397-20:2cb@986'714, size: 2554862, copy_subset: [538121~865189,1730449~824413], clone_subset: {7:9fd44245:::smithi08575397-20:27c=[],7:9fd44245:::smithi08575397-20:320=[0~538121,1403310~327139]}, snapset: 372=[]:{212=[203],27c=[27c],2cb=[2c8,2ae],309=[308,300,2f5,2f0,2ee,2ed,2eb,2ea,2d9],320=[31a],32b=[32a,328],351=[34e],358=[354,353],372=[372,370,36f,36b,368,366,35f]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2554862, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1913836, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:53:38.874 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:28.761961+0000", 2021-02-10T04:53:38.874 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 550.09076062600002, 2021-02-10T04:53:38.874 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0098334979999999995, 2021-02-10T04:53:38.874 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:53:38.874 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:53:38.874 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:53:38.874 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.874 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:53:38.874 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.761961+0000", 2021-02-10T04:53:38.874 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:53:38.874 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.875 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.875 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:53:38.875 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.761961+0000", 2021-02-10T04:53:38.875 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0009811659999999999 2021-02-10T04:53:38.875 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.875 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.875 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:53:38.875 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.762942+0000", 2021-02-10T04:53:38.875 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.2175000000000003e-05 2021-02-10T04:53:38.875 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.875 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.875 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:53:38.875 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.762974+0000", 2021-02-10T04:53:38.875 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.2935000000000001e-05 2021-02-10T04:53:38.876 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.876 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.876 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:53:38.876 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.763027+0000", 2021-02-10T04:53:38.876 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8119999999999999e-05 2021-02-10T04:53:38.876 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.876 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.876 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:53:38.876 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.763045+0000", 2021-02-10T04:53:38.876 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0076169890000000002 2021-02-10T04:53:38.876 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.876 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.876 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:53:38.877 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.770662+0000", 2021-02-10T04:53:38.877 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.3100000000000006e-06 2021-02-10T04:53:38.877 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.877 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.877 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:53:38.877 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.770671+0000", 2021-02-10T04:53:38.877 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0011228029999999999 2021-02-10T04:53:38.877 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.877 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.877 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:53:38.877 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.771794+0000", 2021-02-10T04:53:38.877 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0098334979999999995 2021-02-10T04:53:38.877 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.877 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:53:38.878 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.878 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.878 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.878 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.19 1011/1010 [PushOp(7:9fd44245:::smithi08575397-20:372, version: 988'716, data_included: [3145728~652951], data_size: 652951, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:9fd44245:::smithi08575397-20:372@988'716, size: 3798679, copy_subset: [0~3798679], clone_subset: {7:9fd44245:::smithi08575397-20:32b=[]}, snapset: 372=[]:{212=[203],27c=[27c],2cb=[2c8,2ae],309=[308,300,2f5,2f0,2ee,2ed,2eb,2ea,2d9],320=[31a],32b=[32a,328],351=[34e],358=[354,353],372=[372,370,36f,36b,368,366,35f]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3798679, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:53:38.878 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:28.809203+0000", 2021-02-10T04:53:38.878 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 550.04351848399995, 2021-02-10T04:53:38.878 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010095241, 2021-02-10T04:53:38.878 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:53:38.878 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:53:38.878 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:53:38.878 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.878 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:53:38.879 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.809203+0000", 2021-02-10T04:53:38.879 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:53:38.879 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.879 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.879 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:53:38.879 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.809203+0000", 2021-02-10T04:53:38.879 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00063922499999999999 2021-02-10T04:53:38.879 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.879 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.879 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:53:38.879 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.809842+0000", 2021-02-10T04:53:38.879 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.5457999999999999e-05 2021-02-10T04:53:38.880 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.880 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.880 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:53:38.880 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.809868+0000", 2021-02-10T04:53:38.880 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8017e-05 2021-02-10T04:53:38.880 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.880 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.880 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:53:38.880 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.809896+0000", 2021-02-10T04:53:38.880 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6266999999999999e-05 2021-02-10T04:53:38.880 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.880 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.881 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:53:38.881 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.809912+0000", 2021-02-10T04:53:38.881 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0083131160000000006 2021-02-10T04:53:38.881 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.881 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.881 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:53:38.881 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.818225+0000", 2021-02-10T04:53:38.881 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.6239999999999997e-06 2021-02-10T04:53:38.881 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.881 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.881 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:53:38.881 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.818235+0000", 2021-02-10T04:53:38.882 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0010635340000000001 2021-02-10T04:53:38.882 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.882 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.882 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:53:38.882 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:28.819298+0000", 2021-02-10T04:53:38.882 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010095241 2021-02-10T04:53:38.882 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.882 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:53:38.882 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.882 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.882 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.882 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.13 1013/1010 [PushOp(7:cf4c8c3b:::smithi08575397-6:203, version: 675'1073, data_included: [2246743~16786,2771268~573490], data_size: 590276, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:cf4c8c3b:::smithi08575397-6:203@675'1073, size: 3344758, copy_subset: [0~3344758], clone_subset: {}, snapset: 372=[]:{203=[203],27c=[27c],2bf=[2ae],2cb=[2c8],2f1=[2f0,2ee,2ed,2eb,2ea],308=[308,300,2f5],32c=[32a,328,31a],335=[335,332,32f],351=[34e,34d,34b,349,345,344,343,33e,33c,33a],364=[35f,354,353],372=[372,370,36f,36b,368,366]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3344758, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2246743, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:53:38.883 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:31.239214+0000", 2021-02-10T04:53:38.883 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 547.61350740299997, 2021-02-10T04:53:38.883 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010704108, 2021-02-10T04:53:38.883 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:53:38.883 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:53:38.883 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:53:38.883 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.883 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:53:38.883 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:31.239214+0000", 2021-02-10T04:53:38.883 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:53:38.883 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.883 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.883 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:53:38.884 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:31.239214+0000", 2021-02-10T04:53:38.884 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00052990299999999997 2021-02-10T04:53:38.884 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.884 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.884 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:53:38.884 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:31.239744+0000", 2021-02-10T04:53:38.884 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4080999999999999e-05 2021-02-10T04:53:38.884 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.884 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.884 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:53:38.884 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:31.239768+0000", 2021-02-10T04:53:38.884 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2466000000000003e-05 2021-02-10T04:53:38.884 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.884 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.885 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:53:38.885 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:31.239810+0000", 2021-02-10T04:53:38.885 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7711999999999999e-05 2021-02-10T04:53:38.885 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.885 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.885 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:53:38.885 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:31.239828+0000", 2021-02-10T04:53:38.885 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0083525189999999992 2021-02-10T04:53:38.885 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.885 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.885 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:53:38.885 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:31.248181+0000", 2021-02-10T04:53:38.885 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.2199999999999992e-06 2021-02-10T04:53:38.886 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.886 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.886 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:53:38.886 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:31.248189+0000", 2021-02-10T04:53:38.886 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001729207 2021-02-10T04:53:38.886 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.886 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.886 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:53:38.886 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:31.249918+0000", 2021-02-10T04:53:38.886 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010704108 2021-02-10T04:53:38.886 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.886 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:53:38.886 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.887 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.887 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.887 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.13 1014/1010 [PushOp(7:cd29c741:::smithi08575397-46:309, version: 997'1632, data_included: [2487424~241133,3163563~515291], data_size: 756424, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:cd29c741:::smithi08575397-46:309@997'1632, size: 3949909, copy_subset: [749909~671631,2110479~1839430], clone_subset: {7:cd29c741:::smithi08575397-46:304=[0~749909,1421540~688939]}, snapset: 369=[]:{2a2=[27c],2b6=[2ae],304=[300,2f5,2f0,2ee,2ed,2eb,2ea,2d9,2c8],309=[308],33e=[33e,33c],343=[343],364=[35f,354,353,34e,34d,34b,349,345,344]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3949909, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2487424, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:53:38.887 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:32.869894+0000", 2021-02-10T04:53:38.887 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 545.98282753199999, 2021-02-10T04:53:38.887 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010419648, 2021-02-10T04:53:38.887 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:53:38.887 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:53:38.887 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:53:38.887 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.887 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:53:38.887 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:32.869894+0000", 2021-02-10T04:53:38.887 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:53:38.888 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.888 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.888 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:53:38.888 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:32.869894+0000", 2021-02-10T04:53:38.888 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00079295799999999997 2021-02-10T04:53:38.888 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.888 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.888 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:53:38.888 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:32.870687+0000", 2021-02-10T04:53:38.888 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4919e-05 2021-02-10T04:53:38.888 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.888 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.888 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:53:38.889 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:32.870712+0000", 2021-02-10T04:53:38.889 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2763999999999999e-05 2021-02-10T04:53:38.889 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.889 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.889 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:53:38.889 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:32.870754+0000", 2021-02-10T04:53:38.889 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8715e-05 2021-02-10T04:53:38.889 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.889 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.889 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:53:38.889 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:32.870773+0000", 2021-02-10T04:53:38.889 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0082588120000000008 2021-02-10T04:53:38.889 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.890 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.890 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:53:38.890 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:32.879032+0000", 2021-02-10T04:53:38.890 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1678000000000001e-05 2021-02-10T04:53:38.890 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.890 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.890 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:53:38.890 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:32.879044+0000", 2021-02-10T04:53:38.890 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001269802 2021-02-10T04:53:38.890 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.890 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.890 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:53:38.890 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:32.880313+0000", 2021-02-10T04:53:38.891 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010419648 2021-02-10T04:53:38.891 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.891 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:53:38.891 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.891 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.891 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.891 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.13 1014/1010 [PushOp(7:cd29c741:::smithi08575397-46:33e, version: 998'1638, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:cd29c741:::smithi08575397-46:33e@998'1638, size: 3193627, copy_subset: [0~3193627], clone_subset: {7:cd29c741:::smithi08575397-46:309=[]}, snapset: 369=[]:{2a2=[27c],2b6=[2ae],304=[300,2f5,2f0,2ee,2ed,2eb,2ea,2d9,2c8],309=[308],33e=[33e,33c],343=[343],364=[35f,354,353,34e,34d,34b,349,345,344]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:53:38.891 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:33.012334+0000", 2021-02-10T04:53:38.891 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 545.840387126, 2021-02-10T04:53:38.891 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0098818410000000006, 2021-02-10T04:53:38.891 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:53:38.891 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:53:38.892 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:53:38.892 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.892 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:53:38.892 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.012334+0000", 2021-02-10T04:53:38.892 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:53:38.892 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.892 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.892 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:53:38.892 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.012334+0000", 2021-02-10T04:53:38.892 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001088105 2021-02-10T04:53:38.892 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.892 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.892 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:53:38.893 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.013422+0000", 2021-02-10T04:53:38.893 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.3787000000000001e-05 2021-02-10T04:53:38.893 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.893 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.893 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:53:38.893 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.013446+0000", 2021-02-10T04:53:38.893 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.8970000000000001e-05 2021-02-10T04:53:38.893 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.893 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.893 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:53:38.893 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.013485+0000", 2021-02-10T04:53:38.893 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7571000000000002e-05 2021-02-10T04:53:38.893 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.893 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.894 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:53:38.894 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.013503+0000", 2021-02-10T04:53:38.894 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0081431379999999994 2021-02-10T04:53:38.894 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.894 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.894 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:53:38.894 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.021646+0000", 2021-02-10T04:53:38.894 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.6649999999999998e-06 2021-02-10T04:53:38.894 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.894 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.894 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:53:38.894 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.021655+0000", 2021-02-10T04:53:38.895 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00056060500000000004 2021-02-10T04:53:38.895 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.895 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.895 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:53:38.895 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.022216+0000", 2021-02-10T04:53:38.895 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0098818410000000006 2021-02-10T04:53:38.895 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.895 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:53:38.895 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.895 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.895 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.895 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.13 1014/1010 [PushOp(7:cd29c741:::smithi08575397-46:33e, version: 998'1638, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:cd29c741:::smithi08575397-46:33e@998'1638, size: 3193627, copy_subset: [0~3193627], clone_subset: {7:cd29c741:::smithi08575397-46:309=[]}, snapset: 369=[]:{2a2=[27c],2b6=[2ae],304=[300,2f5,2f0,2ee,2ed,2eb,2ea,2d9,2c8],309=[308],33e=[33e,33c],343=[343],364=[35f,354,353,34e,34d,34b,349,345,344]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:53:38.895 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:33.024117+0000", 2021-02-10T04:53:38.896 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 545.82860453000001, 2021-02-10T04:53:38.896 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010067222000000001, 2021-02-10T04:53:38.896 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:53:38.896 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:53:38.896 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:53:38.896 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.896 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:53:38.896 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.024117+0000", 2021-02-10T04:53:38.896 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:53:38.896 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.896 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.896 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:53:38.896 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.024117+0000", 2021-02-10T04:53:38.896 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0012032010000000001 2021-02-10T04:53:38.897 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.897 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.897 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:53:38.897 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.025320+0000", 2021-02-10T04:53:38.897 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8218e-05 2021-02-10T04:53:38.897 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.897 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.897 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:53:38.897 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.025348+0000", 2021-02-10T04:53:38.897 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5818000000000001e-05 2021-02-10T04:53:38.897 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.897 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.897 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:53:38.898 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.025394+0000", 2021-02-10T04:53:38.898 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5656000000000001e-05 2021-02-10T04:53:38.898 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.898 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.898 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:53:38.898 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.025410+0000", 2021-02-10T04:53:38.898 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0080926039999999998 2021-02-10T04:53:38.898 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.898 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.898 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:53:38.898 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.033502+0000", 2021-02-10T04:53:38.898 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.3239999999999992e-06 2021-02-10T04:53:38.898 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.899 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.899 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:53:38.899 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.033512+0000", 2021-02-10T04:53:38.899 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00067240099999999999 2021-02-10T04:53:38.899 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.899 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.899 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:53:38.899 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.034184+0000", 2021-02-10T04:53:38.899 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010067222000000001 2021-02-10T04:53:38.899 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.899 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:53:38.899 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.899 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.900 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.900 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.13 1014/1010 [PushOp(7:cf4c8c3b:::smithi08575397-6:351, version: 1000'1646, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:cf4c8c3b:::smithi08575397-6:351@1000'1646, size: 2302856, copy_subset: [0~2302856], clone_subset: {7:cf4c8c3b:::smithi08575397-6:335=[],7:cf4c8c3b:::smithi08575397-6:372=[]}, snapset: 372=[]:{203=[203],27c=[27c],2bf=[2ae],2cb=[2c8],2f1=[2f0,2ee,2ed,2eb,2ea],308=[308,300,2f5],32c=[32a,328,31a],335=[335,332,32f],351=[34e,34d,34b,349,345,344,343,33e,33c,33a],364=[35f,354,353],372=[372,370,36f,36b,368,366]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:53:38.900 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:44:33.196545+0000", 2021-02-10T04:53:38.900 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 545.65617582599998, 2021-02-10T04:53:38.900 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010517324, 2021-02-10T04:53:38.900 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:53:38.900 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:53:38.900 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:53:38.900 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.900 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:53:38.900 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.196545+0000", 2021-02-10T04:53:38.900 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:53:38.900 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.901 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.901 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:53:38.901 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.196545+0000", 2021-02-10T04:53:38.901 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001506042 2021-02-10T04:53:38.901 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.901 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.901 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:53:38.901 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.198052+0000", 2021-02-10T04:53:38.901 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1785000000000001e-05 2021-02-10T04:53:38.901 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.901 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.901 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:53:38.901 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.198083+0000", 2021-02-10T04:53:38.902 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.9489000000000001e-05 2021-02-10T04:53:38.902 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.902 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.902 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:53:38.902 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.198123+0000", 2021-02-10T04:53:38.902 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0149e-05 2021-02-10T04:53:38.902 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.902 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.902 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:53:38.902 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.198143+0000", 2021-02-10T04:53:38.902 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0082678630000000003 2021-02-10T04:53:38.902 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.902 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.903 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:53:38.903 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.206411+0000", 2021-02-10T04:53:38.903 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2583e-05 2021-02-10T04:53:38.903 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.903 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.903 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:53:38.903 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.206423+0000", 2021-02-10T04:53:38.903 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00063941300000000003 2021-02-10T04:53:38.903 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.903 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.903 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:53:38.903 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:44:33.207063+0000", 2021-02-10T04:53:38.903 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010517324 2021-02-10T04:53:38.904 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.904 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:53:38.904 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.904 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.904 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.904 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(osd.3.0:2969 4.17 e1063/638 4:e9187ab8:reshard::reshard.0000000000:head v 1063'26, mlcod=1049'24)", 2021-02-10T04:53:38.904 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:47:05.585474+0000", 2021-02-10T04:53:38.904 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 393.26724700400001, 2021-02-10T04:53:38.904 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0033504899999999998, 2021-02-10T04:53:38.904 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:53:38.904 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:53:38.904 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:53:38.904 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.904 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:53:38.905 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:47:05.585474+0000", 2021-02-10T04:53:38.905 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:53:38.905 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.905 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.905 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:53:38.905 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:47:05.585474+0000", 2021-02-10T04:53:38.905 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1758000000000003e-05 2021-02-10T04:53:38.905 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.905 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.905 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:53:38.905 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:47:05.585506+0000", 2021-02-10T04:53:38.905 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.8409999999999996e-06 2021-02-10T04:53:38.905 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.906 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.906 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:53:38.906 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:47:05.585512+0000", 2021-02-10T04:53:38.906 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7399999999999999e-06 2021-02-10T04:53:38.906 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.906 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.906 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:53:38.906 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:47:05.585522+0000", 2021-02-10T04:53:38.906 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2400000000000001e-06 2021-02-10T04:53:38.906 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.906 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.906 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:53:38.907 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:47:05.585526+0000", 2021-02-10T04:53:38.907 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.0031999999999999e-05 2021-02-10T04:53:38.907 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.907 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.907 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:53:38.907 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:47:05.585566+0000", 2021-02-10T04:53:38.907 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0579000000000001e-05 2021-02-10T04:53:38.907 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.907 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.907 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:53:38.907 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:47:05.585586+0000", 2021-02-10T04:53:38.907 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0031840599999999998 2021-02-10T04:53:38.907 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.908 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.908 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:53:38.908 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:47:05.588771+0000", 2021-02-10T04:53:38.908 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.4240000000000002e-05 2021-02-10T04:53:38.908 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.908 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.908 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:53:38.908 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:47:05.588825+0000", 2021-02-10T04:53:38.908 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0033504899999999998 2021-02-10T04:53:38.908 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.908 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:53:38.908 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.908 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.908 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.909 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDScrubReserve(5.1f REQUEST e1098)", 2021-02-10T04:53:38.909 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:50:15.248416+0000", 2021-02-10T04:53:38.909 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 203.60430526299999, 2021-02-10T04:53:38.909 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001281117, 2021-02-10T04:53:38.909 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:53:38.909 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:53:38.909 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:53:38.909 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.909 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:53:38.909 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:15.248416+0000", 2021-02-10T04:53:38.909 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:53:38.909 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.909 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.910 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:53:38.910 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:15.248416+0000", 2021-02-10T04:53:38.910 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4530999999999999e-05 2021-02-10T04:53:38.910 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.910 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.910 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:53:38.910 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:15.248441+0000", 2021-02-10T04:53:38.910 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.4350000000000002e-06 2021-02-10T04:53:38.910 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.910 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.910 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:53:38.910 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:15.248447+0000", 2021-02-10T04:53:38.911 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.5650000000000002e-06 2021-02-10T04:53:38.911 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.911 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.911 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:53:38.911 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:15.248456+0000", 2021-02-10T04:53:38.911 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.345e-06 2021-02-10T04:53:38.911 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.911 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.911 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:53:38.911 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:15.248463+0000", 2021-02-10T04:53:38.911 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.4034000000000004e-05 2021-02-10T04:53:38.911 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.911 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.912 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:53:38.912 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:15.248527+0000", 2021-02-10T04:53:38.912 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.4339999999999998e-06 2021-02-10T04:53:38.912 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.912 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.912 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2021-02-10T04:53:38.912 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:15.248536+0000", 2021-02-10T04:53:38.912 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001105277 2021-02-10T04:53:38.912 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.912 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.912 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:53:38.912 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:15.249642+0000", 2021-02-10T04:53:38.912 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5092000000000001e-05 2021-02-10T04:53:38.912 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.913 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.913 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:53:38.913 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:15.249657+0000", 2021-02-10T04:53:38.913 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.0404000000000001e-05 2021-02-10T04:53:38.913 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.913 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.913 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:53:38.913 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:50:15.249697+0000", 2021-02-10T04:53:38.913 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001281117 2021-02-10T04:53:38.913 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.913 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:53:38.913 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.913 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.914 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.914 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(osd.5.0:2039 4.8 e1132/55 4:11fabdfc:::obj_delete_at_hint.0000000047:head v 1132'59, mlcod=1049'53)", 2021-02-10T04:53:38.914 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:51:55.425130+0000", 2021-02-10T04:53:38.914 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 103.42759169199999, 2021-02-10T04:53:38.914 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00072059100000000003, 2021-02-10T04:53:38.914 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:53:38.914 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:53:38.914 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:53:38.914 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.914 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:53:38.914 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:55.425130+0000", 2021-02-10T04:53:38.914 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:53:38.914 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.915 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.915 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:53:38.915 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:55.425130+0000", 2021-02-10T04:53:38.915 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.5641e-05 2021-02-10T04:53:38.915 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.915 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.915 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:53:38.915 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:55.425165+0000", 2021-02-10T04:53:38.915 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.1299999999999998e-06 2021-02-10T04:53:38.915 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.915 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.915 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:53:38.915 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:55.425171+0000", 2021-02-10T04:53:38.915 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.5589999999999994e-06 2021-02-10T04:53:38.916 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.916 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.916 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:53:38.916 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:55.425181+0000", 2021-02-10T04:53:38.916 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.7319999999999998e-06 2021-02-10T04:53:38.916 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.916 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.916 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:53:38.916 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:55.425186+0000", 2021-02-10T04:53:38.916 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000138806 2021-02-10T04:53:38.916 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.916 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.917 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:53:38.917 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:55.425324+0000", 2021-02-10T04:53:38.917 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5512999999999999e-05 2021-02-10T04:53:38.917 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.917 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.917 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:53:38.917 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:55.425340+0000", 2021-02-10T04:53:38.917 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00046465999999999999 2021-02-10T04:53:38.917 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.917 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.917 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:53:38.917 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:55.425805+0000", 2021-02-10T04:53:38.917 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5550000000000003e-05 2021-02-10T04:53:38.918 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.918 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.918 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:53:38.918 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:55.425850+0000", 2021-02-10T04:53:38.918 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00072059100000000003 2021-02-10T04:53:38.918 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.918 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:53:38.918 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.918 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.918 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.918 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDScrubReserve(6.1b RELEASE e1135)", 2021-02-10T04:53:38.918 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:51:59.838977+0000", 2021-02-10T04:53:38.918 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 99.013744375000002, 2021-02-10T04:53:38.919 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00091688999999999996, 2021-02-10T04:53:38.919 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:53:38.919 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:53:38.919 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:53:38.919 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.919 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:53:38.919 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:59.838977+0000", 2021-02-10T04:53:38.919 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:53:38.919 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.919 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.919 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:53:38.919 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:59.838977+0000", 2021-02-10T04:53:38.919 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.3427999999999999e-05 2021-02-10T04:53:38.920 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.920 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.920 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:53:38.920 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:59.839010+0000", 2021-02-10T04:53:38.920 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.5780000000000004e-06 2021-02-10T04:53:38.920 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.920 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.920 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:53:38.920 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:59.839017+0000", 2021-02-10T04:53:38.920 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1941e-05 2021-02-10T04:53:38.920 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.920 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.920 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:53:38.921 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:59.839029+0000", 2021-02-10T04:53:38.921 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0549999999999996e-06 2021-02-10T04:53:38.921 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.921 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.921 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:53:38.921 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:59.839035+0000", 2021-02-10T04:53:38.921 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.7172999999999997e-05 2021-02-10T04:53:38.921 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.921 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.921 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:53:38.921 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:59.839082+0000", 2021-02-10T04:53:38.921 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0451999999999999e-05 2021-02-10T04:53:38.921 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.922 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.922 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2021-02-10T04:53:38.922 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:59.839093+0000", 2021-02-10T04:53:38.922 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00078892700000000003 2021-02-10T04:53:38.922 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.922 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.922 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:53:38.922 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:59.839881+0000", 2021-02-10T04:53:38.922 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.3559999999999999e-06 2021-02-10T04:53:38.922 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.922 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.922 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:53:38.922 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:59.839891+0000", 2021-02-10T04:53:38.923 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.9799999999999998e-06 2021-02-10T04:53:38.923 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.923 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.923 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:53:38.923 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:59.839894+0000", 2021-02-10T04:53:38.923 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00091688999999999996 2021-02-10T04:53:38.923 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.923 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:53:38.923 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.923 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.923 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.923 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(osd.0.0:2227 4.3 e1135/55 4:c3a682d4:reshard::reshard.0000000004:head v 1135'54, mlcod=1049'46)", 2021-02-10T04:53:38.923 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:52:01.479836+0000", 2021-02-10T04:53:38.923 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 97.372885182000005, 2021-02-10T04:53:38.924 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00074421199999999996, 2021-02-10T04:53:38.924 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:53:38.924 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:53:38.924 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:53:38.924 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.924 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:53:38.924 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:01.479836+0000", 2021-02-10T04:53:38.924 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:53:38.924 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.924 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.924 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:53:38.924 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:01.479836+0000", 2021-02-10T04:53:38.924 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2742e-05 2021-02-10T04:53:38.925 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.925 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.925 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:53:38.925 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:01.479859+0000", 2021-02-10T04:53:38.925 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.9310000000000001e-06 2021-02-10T04:53:38.925 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.925 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.925 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:53:38.925 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:01.479863+0000", 2021-02-10T04:53:38.925 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.7769999999999999e-06 2021-02-10T04:53:38.925 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.925 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.925 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:53:38.926 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:01.479869+0000", 2021-02-10T04:53:38.926 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.2169999999999999e-06 2021-02-10T04:53:38.926 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.926 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.926 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:53:38.926 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:01.479872+0000", 2021-02-10T04:53:38.926 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000103602 2021-02-10T04:53:38.926 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.926 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.926 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:53:38.926 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:01.479975+0000", 2021-02-10T04:53:38.926 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9996999999999998e-05 2021-02-10T04:53:38.926 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.927 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.927 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:53:38.927 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:01.479995+0000", 2021-02-10T04:53:38.927 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00053611000000000002 2021-02-10T04:53:38.927 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.927 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.927 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:53:38.927 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:01.480531+0000", 2021-02-10T04:53:38.927 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.8835999999999998e-05 2021-02-10T04:53:38.927 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.927 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.927 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:53:38.927 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:01.480580+0000", 2021-02-10T04:53:38.928 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00074421199999999996 2021-02-10T04:53:38.928 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.928 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:53:38.928 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.928 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.928 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.928 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:1822 7.8 e1150/1118 7:10dac65a:::smithi08575397-39:head v 1150'76, mlcod=0'0)", 2021-02-10T04:53:38.928 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:52:41.692373+0000", 2021-02-10T04:53:38.928 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 57.160348022999997, 2021-02-10T04:53:38.928 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.015071337000000001, 2021-02-10T04:53:38.928 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:53:38.928 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:53:38.928 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:53:38.928 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.929 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:53:38.929 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.692373+0000", 2021-02-10T04:53:38.929 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:53:38.929 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.929 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.929 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:53:38.929 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.692373+0000", 2021-02-10T04:53:38.929 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0074515270000000003 2021-02-10T04:53:38.929 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.929 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.929 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:53:38.929 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.699825+0000", 2021-02-10T04:53:38.929 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1279e-05 2021-02-10T04:53:38.930 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.930 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.930 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:53:38.930 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.699836+0000", 2021-02-10T04:53:38.930 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5341000000000001e-05 2021-02-10T04:53:38.930 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.930 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.930 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:53:38.930 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.699851+0000", 2021-02-10T04:53:38.930 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.2879999999999998e-06 2021-02-10T04:53:38.930 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.930 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.930 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:53:38.931 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.699861+0000", 2021-02-10T04:53:38.931 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.9645999999999998e-05 2021-02-10T04:53:38.931 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.931 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.931 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:53:38.931 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.699920+0000", 2021-02-10T04:53:38.931 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.5575000000000002e-05 2021-02-10T04:53:38.931 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.931 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.931 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:53:38.931 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.699946+0000", 2021-02-10T04:53:38.931 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0074493550000000004 2021-02-10T04:53:38.931 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.932 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.932 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:53:38.932 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.707395+0000", 2021-02-10T04:53:38.932 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9326000000000002e-05 2021-02-10T04:53:38.932 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.932 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.932 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:53:38.932 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.707445+0000", 2021-02-10T04:53:38.932 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.015071337000000001 2021-02-10T04:53:38.932 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.932 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:53:38.932 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.932 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.933 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.933 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDScrubReserve(4.15 REQUEST e1162)", 2021-02-10T04:53:38.933 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:53:10.268558+0000", 2021-02-10T04:53:38.933 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 28.584163685, 2021-02-10T04:53:38.933 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0040140230000000002, 2021-02-10T04:53:38.933 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:53:38.933 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:53:38.933 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:53:38.933 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.933 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:53:38.933 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:10.268558+0000", 2021-02-10T04:53:38.933 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:53:38.933 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.934 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.934 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:53:38.934 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:10.268558+0000", 2021-02-10T04:53:38.934 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.0046e-05 2021-02-10T04:53:38.934 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.934 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.934 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:53:38.934 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:10.268588+0000", 2021-02-10T04:53:38.934 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.3389999999999998e-06 2021-02-10T04:53:38.934 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.934 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.934 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:53:38.934 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:10.268594+0000", 2021-02-10T04:53:38.935 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7350000000000008e-06 2021-02-10T04:53:38.935 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.935 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.935 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:53:38.935 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:10.268604+0000", 2021-02-10T04:53:38.935 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.0740000000000004e-06 2021-02-10T04:53:38.935 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.935 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.935 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:53:38.935 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:10.268611+0000", 2021-02-10T04:53:38.935 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6780000000000002e-05 2021-02-10T04:53:38.935 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.935 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.936 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:53:38.936 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:10.268668+0000", 2021-02-10T04:53:38.936 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.9339999999999995e-06 2021-02-10T04:53:38.936 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.936 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.936 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2021-02-10T04:53:38.936 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:10.268677+0000", 2021-02-10T04:53:38.936 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.003831119 2021-02-10T04:53:38.936 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.936 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.936 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:53:38.936 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:10.272508+0000", 2021-02-10T04:53:38.936 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5756999999999998e-05 2021-02-10T04:53:38.936 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.937 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.937 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:53:38.937 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:10.272523+0000", 2021-02-10T04:53:38.937 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.8238999999999997e-05 2021-02-10T04:53:38.937 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.937 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.937 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:53:38.937 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:10.272572+0000", 2021-02-10T04:53:38.937 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0040140230000000002 2021-02-10T04:53:38.937 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.937 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:53:38.937 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.938 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.938 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.938 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPull(4.16 e1168/1167 cost 1049576)", 2021-02-10T04:53:38.938 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:53:21.548255+0000", 2021-02-10T04:53:38.938 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 17.304466686000001, 2021-02-10T04:53:38.938 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00043738700000000002, 2021-02-10T04:53:38.938 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:53:38.938 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T04:53:38.938 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:53:38.938 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.938 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:53:38.938 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:21.548255+0000", 2021-02-10T04:53:38.938 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:53:38.939 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.939 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.939 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:53:38.939 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:21.548255+0000", 2021-02-10T04:53:38.939 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.0311000000000002e-05 2021-02-10T04:53:38.939 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.939 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.939 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:53:38.939 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:21.548285+0000", 2021-02-10T04:53:38.939 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5358000000000001e-05 2021-02-10T04:53:38.939 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.939 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.939 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:53:38.940 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:21.548300+0000", 2021-02-10T04:53:38.940 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1365999999999999e-05 2021-02-10T04:53:38.940 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.940 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.940 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:53:38.940 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:21.548312+0000", 2021-02-10T04:53:38.940 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.4389999999999999e-06 2021-02-10T04:53:38.940 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.940 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.940 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:53:38.940 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:21.548317+0000", 2021-02-10T04:53:38.940 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5556999999999998e-05 2021-02-10T04:53:38.940 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.941 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.941 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:53:38.941 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:21.548363+0000", 2021-02-10T04:53:38.941 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000329356 2021-02-10T04:53:38.941 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.941 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.941 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:53:38.941 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:21.548692+0000", 2021-02-10T04:53:38.941 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00043738700000000002 2021-02-10T04:53:38.941 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.941 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:53:38.941 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.941 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.942 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.942 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPull(4.16 e1168/1167 cost 1049576)", 2021-02-10T04:53:38.942 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:53:21.738599+0000", 2021-02-10T04:53:38.942 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 17.114122250000001, 2021-02-10T04:53:38.942 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00049746100000000004, 2021-02-10T04:53:38.942 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:53:38.942 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T04:53:38.942 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:53:38.942 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.942 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:53:38.942 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:21.738599+0000", 2021-02-10T04:53:38.942 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:53:38.942 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.943 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.943 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:53:38.943 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:21.738599+0000", 2021-02-10T04:53:38.943 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.5064999999999997e-05 2021-02-10T04:53:38.943 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.943 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.943 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:53:38.943 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:21.738634+0000", 2021-02-10T04:53:38.943 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.0034000000000002e-05 2021-02-10T04:53:38.943 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.943 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.943 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:53:38.943 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:21.738654+0000", 2021-02-10T04:53:38.944 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3062e-05 2021-02-10T04:53:38.944 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.944 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.944 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:53:38.944 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:21.738667+0000", 2021-02-10T04:53:38.944 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.2119999999999992e-06 2021-02-10T04:53:38.944 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.944 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.944 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:53:38.944 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:21.738676+0000", 2021-02-10T04:53:38.944 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.3748e-05 2021-02-10T04:53:38.944 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.944 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.944 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:53:38.945 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:21.738740+0000", 2021-02-10T04:53:38.945 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00035634000000000002 2021-02-10T04:53:38.945 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:53:38.945 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:53:38.945 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:53:38.945 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:21.739097+0000", 2021-02-10T04:53:38.945 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00049746100000000004 2021-02-10T04:53:38.945 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.945 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:53:38.945 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.945 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:53:38.945 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:53:38.945 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:53:38.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:38 smithi085 conmon[35961]: cluster 2021-02-10T04:53:37.642663+0000 mon.a (mon.0) 2514 : cluster [DBG] osdmap e1183: 8 total, 7 up, 8 in 2021-02-10T04:53:38.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:38 smithi085 conmon[35961]: cluster 2021-02-10T04:53:38.085557+0000 osd.4 (osd.4) 2650 : cluster [DBG] 4.9 scrub ok 2021-02-10T04:53:38.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:38 smithi085 conmon[35961]: cluster 2021-02-10T04:53:38.170790+0000 osd.1 (osd.1) 500 : cluster [DBG] 7.b scrub ok 2021-02-10T04:53:38.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:38 smithi085 conmon[35961]: audit 2021-02-10T04:53:38.249537+0000 mon.a (mon.0) 2515 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:53:38.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:38 smithi085 conmon[35961]: audit 2021-02-10T04:53:38.250857+0000 mon.a (mon.0) 2516 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:53:38.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:38 smithi085 conmon[35961]: cluster 2021-02-10T04:53:38.516952+0000 mon.a (mon.0) 2517 : cluster [WRN] Health check update: Degraded data redundancy: 227/2169 objects degraded (10.466%), 25 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2021-02-10T04:53:38.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:38 smithi085 conmon[29319]: cluster 2021-02-10T04:53:37.642663+0000 mon.a (mon.0) 2514 : cluster [DBG] osdmap e1183: 8 total, 7 up, 8 in 2021-02-10T04:53:38.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:38 smithi085 conmon[29319]: cluster 2021-02-10T04:53:38.085557+0000 osd.4 (osd.4) 2650 : cluster [DBG] 4.9 scrub ok 2021-02-10T04:53:38.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:38 smithi085 conmon[29319]: cluster 2021-02-10T04:53:38.170790+0000 osd.1 (osd.1) 500 : cluster [DBG] 7.b scrub ok 2021-02-10T04:53:38.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:38 smithi085 conmon[29319]: audit 2021-02-10T04:53:38.249537+0000 mon.a (mon.0) 2515 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:53:38.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:38 smithi085 conmon[29319]: audit 2021-02-10T04:53:38.250857+0000 mon.a (mon.0) 2516 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:53:38.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:38 smithi085 conmon[29319]: cluster 2021-02-10T04:53:38.516952+0000 mon.a (mon.0) 2517 : cluster [WRN] Health check update: Degraded data redundancy: 227/2169 objects degraded (10.466%), 25 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2021-02-10T04:53:39.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:38 smithi164 conmon[29288]: cluster 2021-02-10T04:53:37.642663+0000 mon.a ( 2021-02-10T04:53:39.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:38 smithi164 conmon[29288]: mon.0) 2514 : cluster [DBG] osdmap e1183: 8 total, 7 up, 8 in 2021-02-10T04:53:39.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:38 smithi164 conmon[29288]: cluster 2021-02-10T04:53:38.085557+0000 osd.4 (osd.4) 2650 : cluster [DBG] 4.9 scrub ok 2021-02-10T04:53:39.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:38 smithi164 conmon[29288]: cluster 2021-02-10T04:53:38.170790+0000 osd.1 (osd.1) 500 : cluster [DBG] 7.b scrub ok 2021-02-10T04:53:39.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:38 smithi164 conmon[29288]: audit 2021-02-10T04:53:38.249537+0000 mon.a (mon.0) 2515 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:53:39.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:38 smithi164 conmon[29288]: audit 2021-02-10T04:53:38.250857+0000 mon.a (mon.0) 2516 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:53:39.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:38 smithi164 conmon[29288]: cluster 2021-02-10T04:53:38.516952+0000 mon.a (mon.0) 2517 : cluster [WRN] Health check update: Degraded data redundancy: 227/2169 objects degraded (10.466%), 25 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2021-02-10T04:53:39.644 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:53:39 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:53:39.665 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is unset 2021-02-10T04:53:39.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[29319]: cluster 2021-02-10T04:53:38.363819+0000 osd.5 (osd.5 2021-02-10T04:53:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[29319]: ) 185 : cluster [DBG] 6.19 scrub ok 2021-02-10T04:53:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[29319]: cluster 2021-02-10T04:53:38.455089+0000 mgr.y (mgr.14140) 1294 : cluster [DBG] pgmap v2133: 195 pgs: 2 peering, 4 active+clean+remapped, 25 active+undersized+degraded, 37 active+undersized, 127 active+clean; 866 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 227/2196 objects degraded (10.337%); 34/2196 objects misplaced (1.548%) 2021-02-10T04:53:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[29319]: cluster 2021-02-10T04:53:38.654855+0000 mon.a (mon.0) 2518 : cluster [DBG] osdmap e1184: 8 total, 7 up, 8 in 2021-02-10T04:53:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[29319]: audit 2021-02-10T04:53:38.938116+0000 mon.c (mon.1) 111 : audit 2021-02-10T04:53:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[29319]: [INF] from='client.? 172.21.15.85:0/4067285670' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:53:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:53:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[29319]: -10T04:53:38.938442+0000 mon.a 2021-02-10T04:53:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[29319]: (mon.0) 2519 : audit 2021-02-10T04:53:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[29319]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:53:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[29319]: cluster 2021-02-10T04:53:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:53:39.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[29319]: T04:53:39.126257+0000 osd.4 (osd 2021-02-10T04:53:39.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[29319]: .4) 2651 : cluster [DBG] 2.b scrub ok 2021-02-10T04:53:39.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:53:39.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[29319]: 02-10T04:53:39.204177+0000 2021-02-10T04:53:39.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[29319]: mgr.y (mgr.14140) 1295 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:53:39.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[29319]: 2021-02-10T04:53:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:53:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[35961]: 53:38.363819+0000 osd.5 (osd.5) 185 : cluster [DBG] 6.19 scrub ok 2021-02-10T04:53:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:53:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[35961]: -02-10T04:53:38.455089+0000 2021-02-10T04:53:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[35961]: mgr.y (mgr.14140) 1294 : cluster [DBG] pgmap v2133: 195 pgs: 2 peering, 4 active+clean+remapped, 25 active+undersized+degraded, 37 active+undersized, 127 active+clean; 866 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 227/2196 objects degraded (10.337%); 34/2196 objects misplaced (1.548%) 2021-02-10T04:53:39.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[35961]: cluster 2021-02-10T04:53:39.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[35961]: 2021-02-10T04:53:38.654855+0000 2021-02-10T04:53:39.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[35961]: mon.a (mon.0) 2518 : cluster [DBG] osdmap e1184: 8 total, 7 up, 8 in 2021-02-10T04:53:39.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[35961]: 2021-02-10T04:53:39.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:53:39.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[35961]: 53:38.938116+0000 mon.c (mon.1) 111 : 2021-02-10T04:53:39.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[35961]: audit [INF] from='client.? 172.21.15.85:0/4067285670' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:53:39.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:53:39.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[35961]: -10T04:53:38.938442+0000 mon.a 2021-02-10T04:53:39.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[35961]: (mon.0) 2519 : audit [INF] 2021-02-10T04:53:39.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[35961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:53:39.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[35961]: cluster 2021-02-10T04:53:39 2021-02-10T04:53:39.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[35961]: .126257+0000 osd.4 (osd.4) 2651 : 2021-02-10T04:53:39.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[35961]: cluster [DBG] 2.b scrub ok 2021-02-10T04:53:39.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:53:39.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[35961]: 53:39.204177+0000 mgr.y (mgr.14140) 2021-02-10T04:53:39.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:39 smithi085 conmon[35961]: 1295 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:53:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:39 smithi164 conmon[29288]: cluster 2021-02-10T04:53:38.363819 2021-02-10T04:53:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:39 smithi164 conmon[29288]: +0000 osd.5 (osd.5) 185 : cluster [DBG] 6.19 scrub ok 2021-02-10T04:53:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:39 smithi164 conmon[29288]: cluster 2021-02-10T04:53:38.455089+0000 mgr.y (mgr.14140) 1294 : cluster [DBG] pgmap v2133: 195 pgs: 2 peering, 4 active+clean+remapped, 25 active+undersized+degraded, 37 active+undersized, 127 active+clean; 866 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 227/2196 objects degraded (10.337%); 34/2196 objects misplaced (1.548%) 2021-02-10T04:53:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:39 smithi164 conmon[29288]: cluster 2021-02-10T04:53:38.654855+0000 mon.a (mon.0) 2518 : cluster [DBG] osdmap e1184: 8 total, 7 up, 8 in 2021-02-10T04:53:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:39 smithi164 conmon[29288]: audit 2021-02-10T04:53:38.938116+0000 mon.c (mon.1) 111 : audit [INF] from='client.? 172.21.15.85:0/4067285670' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:53:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:39 smithi164 conmon[29288]: audit 2021-02-10T04:53:38.938442+0000 mon.a (mon.0) 2519 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:53:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:39 smithi164 conmon[29288]: cluster 2021-02-10T04:53:39.126257+0000 osd.4 (osd.4) 2651 : cluster [DBG] 2.b scrub ok 2021-02-10T04:53:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:39 smithi164 conmon[29288]: audit 2021-02-10T04:53:39.204177+0000 mgr.y (mgr.14140) 1295 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:53:40.165 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:53:40.374 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:53:40.723 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2021-02-10T04:53:40.723 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:53:40.723 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.846167 2021-02-10T04:53:40.723 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd primary-affinity 7 0.8461670652742835 2021-02-10T04:53:40.905 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:40 smithi085 conmon[29319]: cluster 2021-02-10T04:53:39.652317+0000 2021-02-10T04:53:40.905 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:40 smithi085 conmon[29319]: mon.a (mon.0) 2520 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:53:40.905 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:40 smithi085 conmon[29319]: audit 2021-02-10T04:53:39.659549+0000 mon.a (mon.0) 2521 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:53:40.905 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:40 smithi085 conmon[29319]: cluster 2021-02-10T04:53:39.659620+0000 mon.a (mon.0) 2522 : cluster [DBG] osdmap e1185: 8 total, 7 up, 8 in 2021-02-10T04:53:40.905 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:40 smithi085 conmon[29319]: cluster 2021-02-10T04:53:40.135360+0000 osd.1 (osd.1) 501 : cluster [DBG] 7.1e scrub ok 2021-02-10T04:53:40.905 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:40 smithi085 conmon[29319]: audit 2021-02-10T04:53:40.456648+0000 mon.a (mon.0) 2523 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "4.b"}]: dispatch 2021-02-10T04:53:40.905 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:40 smithi085 conmon[29319]: audit 2021-02-10T04:53:40.456880+0000 mon.a (mon.0) 2524 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.b"}]: dispatch 2021-02-10T04:53:40.906 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:40 smithi085 conmon[35961]: cluster 2021-02-10T04:53:39.652317+0000 2021-02-10T04:53:40.906 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:40 smithi085 conmon[35961]: mon.a (mon.0) 2520 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:53:40.906 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:40 smithi085 conmon[35961]: audit 2021-02-10T04:53:39.659549+0000 mon.a (mon.0) 2521 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:53:40.906 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:40 smithi085 conmon[35961]: cluster 2021-02-10T04:53:39.659620+0000 mon.a (mon.0) 2522 : cluster [DBG] osdmap e1185: 8 total, 7 up, 8 in 2021-02-10T04:53:40.906 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:40 smithi085 conmon[35961]: cluster 2021-02-10T04:53:40.135360+0000 osd.1 (osd.1) 501 : cluster [DBG] 7.1e scrub ok 2021-02-10T04:53:40.906 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:40 smithi085 conmon[35961]: audit 2021-02-10T04:53:40.456648+0000 mon.a (mon.0) 2523 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "4.b"}]: dispatch 2021-02-10T04:53:40.906 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:40 smithi085 conmon[35961]: audit 2021-02-10T04:53:40.456880+0000 mon.a (mon.0) 2524 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.b"}]: dispatch 2021-02-10T04:53:40.955 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:53:41.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:40 smithi164 conmon[29288]: cluster 2021-02-10T04:53:39.652317+0000 mon.a 2021-02-10T04:53:41.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:40 smithi164 conmon[29288]: (mon.0) 2520 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:53:41.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:40 smithi164 conmon[29288]: audit 2021-02-10T04:53:39.659549+0000 mon.a (mon.0) 2521 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:53:41.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:40 smithi164 conmon[29288]: cluster 2021-02-10T04:53:39.659620+0000 mon.a (mon.0 2021-02-10T04:53:41.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:40 smithi164 conmon[29288]: ) 2522 : cluster [DBG] osdmap e1185: 8 total, 7 up, 8 in 2021-02-10T04:53:41.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:40 smithi164 conmon[29288]: cluster 2021-02-10T04:53:41.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:40 smithi164 conmon[29288]: 2021-02-10T04:53:40.135360+0000 2021-02-10T04:53:41.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:40 smithi164 conmon[29288]: osd.1 (osd.1) 501 : cluster [DBG] 7.1e scrub ok 2021-02-10T04:53:41.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:40 smithi164 conmon[29288]: 2021-02-10T04:53:41.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:40 smithi164 conmon[29288]: audit 2021-02-10T04:53:41.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:40 smithi164 conmon[29288]: 2021-02-10T04:53:40.456648+0000 mon.a ( 2021-02-10T04:53:41.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:40 smithi164 conmon[29288]: mon.0) 2523 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "4.b"}]: dispatch 2021-02-10T04:53:41.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:40 smithi164 conmon[29288]: audit 2021 2021-02-10T04:53:41.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:40 smithi164 conmon[29288]: -02-10T04:53:40.456880+0000 2021-02-10T04:53:41.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:40 smithi164 conmon[29288]: mon.a (mon.0) 2524 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.b"}]: dispatch 2021-02-10T04:53:41.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:40 smithi164 conmon[29288]: 2021-02-10T04:53:41.664 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:41 smithi085 conmon[29319]: cluster 2021-02-10T04:53:40.455703 2021-02-10T04:53:41.664 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:41 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 1296 : cluster [DBG] pgmap v2136: 194 pgs: 1 peering, 4 active+clean+remapped, 25 active+undersized+degraded, 37 active+undersized, 127 active+clean; 866 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 227/2175 objects degraded (10.437%); 34/2175 objects misplaced (1.563%) 2021-02-10T04:53:41.664 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:41 smithi085 conmon[29319]: audit 2021-02-10T04:53:40.662853+0000 mon.a (mon.0) 2525 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "4.b"}]': finished 2021-02-10T04:53:41.664 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:41 smithi085 conmon[29319]: audit 2021-02-10T04:53:40.662914+0000 mon.a (mon.0) 2526 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.b"}]': finished 2021-02-10T04:53:41.664 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:41 smithi085 conmon[29319]: cluster 2021-02-10T04:53:40.662961+0000 mon.a (mon.0) 2527 : cluster [DBG] osdmap e1186: 8 total, 7 up, 8 in 2021-02-10T04:53:41.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:41 smithi085 conmon[35961]: cluster 2021-02-10T04:53:40.455703+0000 mgr.y (mgr 2021-02-10T04:53:41.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:41 smithi085 conmon[35961]: .14140) 1296 : cluster [DBG] pgmap v2136: 194 pgs: 1 peering, 4 active+clean+remapped, 25 active+undersized+degraded, 37 active+undersized, 127 active+clean; 866 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 227/2175 objects degraded (10.437%); 34/2175 objects misplaced (1.563%) 2021-02-10T04:53:41.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:41 smithi085 conmon[35961]: audit 2021-02-10T04:53:40.662853+0000 mon.a (mon.0) 2525 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "4.b"}]': finished 2021-02-10T04:53:41.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:41 smithi085 conmon[35961]: audit 2021-02-10T04:53:40.662914+0000 mon.a (mon.0) 2526 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.b"}]': finished 2021-02-10T04:53:41.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:41 smithi085 conmon[35961]: cluster 2021-02-10T04:53:40.662961+0000 mon.a (mon.0) 2527 : cluster [DBG] osdmap e1186: 8 total, 7 up, 8 in 2021-02-10T04:53:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:41 smithi164 conmon[29288]: cluster 2021-02-10T04:53:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:41 smithi164 conmon[29288]: 2021-02-10T04:53 2021-02-10T04:53:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:41 smithi164 conmon[29288]: :40.455703+0000 mgr.y (mgr.14140) 1296 : cluster [DBG] pgmap v2136: 194 pgs: 1 peering, 4 active+clean+remapped, 25 active+undersized+degraded, 37 active+undersized, 127 active+clean; 866 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 227/2175 objects degraded (10.437%); 34/2175 objects misplaced (1.563%) 2021-02-10T04:53:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:41 smithi164 conmon[29288]: audit 2021-02-10T04:53:40 2021-02-10T04:53:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:41 smithi164 conmon[29288]: .662853+0000 mon.a (mon.0) 2525 : audit [INF] 2021-02-10T04:53:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:41 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "4.b"}]': finished 2021-02-10T04:53:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:41 smithi164 conmon[29288]: audit 2021-02-10T04:53:40.662914+0000 2021-02-10T04:53:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:41 smithi164 conmon[29288]: mon.a (mon.0) 2526 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.b"}]': finished 2021-02-10T04:53:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:41 smithi164 conmon[29288]: 2021-02-10T04:53:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:41 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:53:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:41 smithi164 conmon[29288]: 02-10T04:53:40.662961+0000 mon.a (mon.0) 2527 : cluster 2021-02-10T04:53:42.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:41 smithi164 conmon[29288]: [DBG] osdmap e1186: 8 total, 7 up, 8 in 2021-02-10T04:53:42.688 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set noscrub 2021-02-10T04:53:42.721 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:42 smithi085 conmon[29319]: cluster 2021-02-10T04:53:41.353789+0000 osd.5 ( 2021-02-10T04:53:42.722 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:42 smithi085 conmon[29319]: osd.5) 186 : cluster [DBG] 7.1c scrub ok 2021-02-10T04:53:42.722 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:42 smithi085 conmon[29319]: cluster 2021-02-10T04:53:41.673259+0000 mon.a (mon.0) 2528 : cluster [DBG] osdmap e1187: 8 total, 7 up, 8 in 2021-02-10T04:53:42.722 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:42 smithi085 conmon[29319]: cluster 2021-02-10T04:53:41.722507+0000 osd.2 (osd.2) 2026 : cluster [DBG] 2.e scrub ok 2021-02-10T04:53:42.722 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:42 smithi085 conmon[29319]: cluster 2021-02-10T04:53:42.071095+0000 osd.4 (osd.4) 2652 : cluster [DBG] 4.c deep-scrub ok 2021-02-10T04:53:42.725 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:42 smithi085 conmon[35961]: cluster 2021-02-10T04:53:42.725 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:42 smithi085 conmon[35961]: 2021-02-10T04:53:42.725 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:42 smithi085 conmon[35961]: 2021 2021-02-10T04:53:42.725 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:42 smithi085 conmon[35961]: - 2021-02-10T04:53:42.725 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:42 smithi085 conmon[35961]: 02 2021-02-10T04:53:42.725 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:42 smithi085 conmon[35961]: - 2021-02-10T04:53:42.725 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:42 smithi085 conmon[35961]: 10 2021-02-10T04:53:42.725 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:42 smithi085 conmon[35961]: T 2021-02-10T04:53:42.726 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:42 smithi085 conmon[35961]: 04:53:41.353789+0000 osd.5 ( 2021-02-10T04:53:42.726 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:42 smithi085 conmon[35961]: osd.5) 186 : cluster [DBG] 7.1c scrub ok 2021-02-10T04:53:42.726 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:42 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:53:42.726 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:42 smithi085 conmon[35961]: T04:53:41.673259+0000 mon.a (mon 2021-02-10T04:53:42.726 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:42 smithi085 conmon[35961]: .0) 2528 : cluster [DBG] osdmap e1187: 8 total, 7 up, 8 in 2021-02-10T04:53:42.726 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:42 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:53:42.726 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:42 smithi085 conmon[35961]: 02-10T04:53:41.722507+0000 osd.2 2021-02-10T04:53:42.726 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:42 smithi085 conmon[35961]: (osd.2) 2026 : cluster [DBG] 2.e scrub ok 2021-02-10T04:53:42.726 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:42 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:53:42.726 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:42 smithi085 conmon[35961]: 02-10T04:53:42.071095+0000 2021-02-10T04:53:42.726 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:42 smithi085 conmon[35961]: osd.4 (osd.4) 2652 : cluster [DBG] 2021-02-10T04:53:42.726 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:42 smithi085 conmon[35961]: 4.c deep-scrub ok 2021-02-10T04:53:42.955 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:53:42.957 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:42 smithi164 conmon[29288]: cluster 2021-02-10T04:53:41.353789+0000 osd.5 ( 2021-02-10T04:53:42.957 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:42 smithi164 conmon[29288]: osd.5) 186 : cluster [DBG] 7.1c scrub ok 2021-02-10T04:53:42.957 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:42 smithi164 conmon[29288]: cluster 2021-02-10T04:53:41.673259+0000 mon.a (mon.0) 2528 : cluster [DBG] osdmap e1187: 8 total, 7 up, 8 in 2021-02-10T04:53:42.957 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:42 smithi164 conmon[29288]: cluster 2021-02-10T04:53:41. 2021-02-10T04:53:42.957 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:42 smithi164 conmon[29288]: 722507+0000 osd.2 (osd.2) 2026 : cluster [DBG] 2.e scrub ok 2021-02-10T04:53:42.957 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:42 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:53:42.957 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:42 smithi164 conmon[29288]: -02-10T04:53:42.071095+0000 osd.4 (osd.4) 2652 : cluster 2021-02-10T04:53:42.957 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:42 smithi164 conmon[29288]: [DBG] 4.c deep-scrub ok 2021-02-10T04:53:42.976 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight 2021-02-10T04:53:43.209 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:53:43.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:43 smithi085 conmon[35961]: cluster 2021-02-10T04:53:42.456456+0000 mgr.y (mgr.14140) 1297 : cluster [DBG] pgmap v2139: 194 pgs: 4 active+clean+remapped, 25 active+undersized+degraded, 37 active+undersized, 128 active+clean; 866 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 227/2196 objects degraded (10.337%); 34/2196 objects misplaced (1.548%) 2021-02-10T04:53:43.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:43 smithi085 conmon[35961]: cluster 2021-02-10T04:53:42.676856+0000 mon.a (mon.0) 2529 : cluster [DBG] osdmap e1188: 8 total, 7 up, 8 in 2021-02-10T04:53:43.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:43 smithi085 conmon[35961]: cluster 2021-02-10T04:53:43.519057+0000 mon.a (mon.0) 2530 : cluster [WRN] Health check update: Degraded data redundancy: 227/2196 objects degraded (10.337%), 25 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2021-02-10T04:53:43.948 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:43 smithi085 conmon[35961]: cluster 2021-02-10T04:53:43.523003+0000 mon.a (mon.0) 2531 : cluster [DBG] osdmap e1189: 8 total, 7 up, 8 in 2021-02-10T04:53:43.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:43 smithi085 conmon[29319]: cluster 2021-02-10T04:53:42.456456+0000 mgr.y (mgr.14140) 1297 : cluster [DBG] pgmap v2139: 194 pgs: 4 active+clean+remapped, 25 active+undersized+degraded, 37 active+undersized, 128 active+clean; 866 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 227/2196 objects degraded (10.337%); 34/2196 objects misplaced (1.548%) 2021-02-10T04:53:43.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:43 smithi085 conmon[29319]: cluster 2021-02-10T04:53:42.676856+0000 mon.a (mon.0) 2529 : cluster [DBG] osdmap e1188: 8 total, 7 up, 8 in 2021-02-10T04:53:43.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:43 smithi085 conmon[29319]: cluster 2021-02-10T04:53:43.519057+0000 mon.a (mon.0) 2530 : cluster [WRN] Health check update: Degraded data redundancy: 227/2196 objects degraded (10.337%), 25 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2021-02-10T04:53:43.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:43 smithi085 conmon[29319]: cluster 2021-02-10T04:53:43.523003+0000 mon.a (mon.0) 2531 : cluster [DBG] osdmap e1189: 8 total, 7 up, 8 in 2021-02-10T04:53:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:43 smithi164 conmon[29288]: cluster 2021-02-10T04:53:42.456456 2021-02-10T04:53:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:43 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1297 : cluster [DBG] pgmap v2139: 194 pgs: 4 active+clean+remapped, 25 active+undersized+degraded, 37 active+undersized, 128 active+clean; 866 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 227/2196 objects degraded (10.337%); 34/2196 objects misplaced (1.548%) 2021-02-10T04:53:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:43 smithi164 conmon[29288]: cluster 2021-02-10T04:53:42.676856 2021-02-10T04:53:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:43 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2529 : cluster [DBG] osdmap e1188: 8 total, 7 up, 8 in 2021-02-10T04:53:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:43 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:53:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:43 smithi164 conmon[29288]: 02-10T04:53:43.519057+0000 mon.a 2021-02-10T04:53:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:43 smithi164 conmon[29288]: (mon.0) 2530 : cluster 2021-02-10T04:53:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:43 smithi164 conmon[29288]: [WRN] Health check update: Degraded data redundancy: 227/2196 objects degraded (10.337%), 25 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2021-02-10T04:53:44.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:43 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:53:44.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:43 smithi164 conmon[29288]: T04:53:43.523003+0000 mon.a (mon.0) 2531 : cluster 2021-02-10T04:53:44.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:43 smithi164 conmon[29288]: [DBG] osdmap e1189: 8 total, 7 up, 8 in 2021-02-10T04:53:45.340 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:53:48.133 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:45 smithi164 conmon[29288]: cluster 2021-02-10T04:53:48.133 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:45 smithi164 conmon[29288]: 2021-02-10T04:53:48.134 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:45 smithi164 conmon[29288]: 2021 2021-02-10T04:53:48.134 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:45 smithi164 conmon[29288]: - 2021-02-10T04:53:48.134 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:45 smithi164 conmon[29288]: 02 2021-02-10T04:53:48.134 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:45 smithi164 conmon[29288]: -10T04:53:44.457164 2021-02-10T04:53:48.134 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:45 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1298 : cluster [DBG] pgmap v2142: 194 pgs: 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 128 active+clean; 854 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 217/2157 objects degraded (10.060%); 44/2157 objects misplaced (2.040%) 2021-02-10T04:53:48.134 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:45 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:53:48.134 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:45 smithi164 conmon[29288]: T04:53:45.306704+0000 mon.c (mon 2021-02-10T04:53:48.134 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:45 smithi164 conmon[29288]: .1) 112 : audit [DBG] from='client.? 172.21.15.85:0/988956103' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:53:48.134 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:45 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:53:48.134 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:45 smithi164 conmon[29288]: 02-10T04:53:45.528154+0000 mon.a 2021-02-10T04:53:48.134 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:45 smithi164 conmon[29288]: (mon.0) 2532 : audit [INF] from='client.? 172.21.15.85:0/852813929' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.8461670652742835}]: dispatch 2021-02-10T04:53:48.134 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:46 smithi164 conmon[29288]: audit 2021-02-10T04:53:45.682319+0000 mon.a (mon 2021-02-10T04:53:48.134 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:46 smithi164 conmon[29288]: .0) 2533 : audit [INF] from='client.? 172.21.15.85:0/852813929' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.8461670652742835}]': finished 2021-02-10T04:53:48.135 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:46 smithi164 conmon[29288]: cluster 2021-02-10T04:53:45.682370+0000 mon.a (mon.0) 2534 : cluster [DBG] osdmap e1190: 8 total, 7 up, 8 in 2021-02-10T04:53:48.135 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:53:48.135 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:53:48.135 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:53:48.135 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:53:48.135 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:53:48.135 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:53:48.135 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-02-10T04:53:48.135 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: problem getting command descriptions from osd.0 2021-02-10T04:53:48.135 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:53:48.135 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:53:48.136 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:53:48.136 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:53:48.136 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:53:48.136 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:53:48.136 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:53:48.136 INFO:teuthology.orchestra.run.smithi085.stderr:set osd.7 primary-affinity to 0.846167 (8554542) 2021-02-10T04:53:48.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:45 smithi085 conmon[35961]: cluster 2021-02-10T04:53:44.457164 2021-02-10T04:53:48.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:45 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1298 : cluster [DBG] pgmap v2142: 194 pgs: 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 128 active+clean; 854 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 217/2157 objects degraded (10.060%); 44/2157 objects misplaced (2.040%) 2021-02-10T04:53:48.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:45 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:53:48.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:45 smithi085 conmon[35961]: 02-10T04:53:45.306704+0000 mon.c (mon.1) 112 : audit 2021-02-10T04:53:48.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:45 smithi085 conmon[35961]: [DBG] from='client.? 172.21.15.85:0/988956103' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:53:48.136 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:45 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:53:48.137 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:45 smithi085 conmon[35961]: 53:45.528154+0000 mon.a (mon.0) 2021-02-10T04:53:48.137 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:45 smithi085 conmon[35961]: 2532 : audit [INF] from='client.? 172.21.15.85:0/852813929' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.8461670652742835}]: dispatch 2021-02-10T04:53:48.137 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:46 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:53:48.137 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:46 smithi085 conmon[35961]: 53:45.682319+0000 mon.a (mon.0) 2533 : audit [INF] from='client.? 172.21.15.85:0/852813929' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.8461670652742835}]': finished 2021-02-10T04:53:48.137 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:46 smithi085 conmon[35961]: cluster 2021-02-10T04:53:45. 2021-02-10T04:53:48.137 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:46 smithi085 conmon[35961]: 682370+0000 mon.a (mon.0) 2534 : cluster [DBG] 2021-02-10T04:53:48.137 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:46 smithi085 conmon[35961]: osdmap e1190: 8 total, 7 up, 8 in 2021-02-10T04:53:48.137 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:47 smithi085 conmon[35961]: cluster 2021-02-10T04:53: 2021-02-10T04:53:48.137 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:47 smithi085 conmon[35961]: 46.457884+0000 mgr.y (mgr.14140) 1299 : cluster [DBG] pgmap v2144: 194 pgs: 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 128 active+clean; 854 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 217/2157 objects degraded (10.060%); 44/2157 objects misplaced (2.040%) 2021-02-10T04:53:48.137 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:47 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:53:48.137 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:47 smithi085 conmon[35961]: -02-10T04:53:46. 2021-02-10T04:53:48.137 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:47 smithi085 conmon[35961]: 701479+0000 mon.a (mon.0 2021-02-10T04:53:48.137 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:47 smithi085 conmon[35961]: ) 2535 : cluster [DBG] osdmap e1191: 8 total, 7 up, 8 in 2021-02-10T04:53:48.137 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:47 smithi085 conmon[35961]: 2021-02-10T04:53:48.138 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:45 smithi085 conmon[29319]: cluster 2021-02-10T04:53:48.138 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:45 smithi085 conmon[29319]: 2021-02-10T04:53:48.138 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:45 smithi085 conmon[29319]: 2021 2021-02-10T04:53:48.138 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:45 smithi085 conmon[29319]: - 2021-02-10T04:53:48.138 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:45 smithi085 conmon[29319]: 02 2021-02-10T04:53:48.138 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:45 smithi085 conmon[29319]: - 2021-02-10T04:53:48.138 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:45 smithi085 conmon[29319]: 10T04:53:44.457164+0000 mgr.y (mgr.14140 2021-02-10T04:53:48.138 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:45 smithi085 conmon[29319]: ) 1298 : cluster [DBG] pgmap v2142: 194 pgs: 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 128 active+clean; 854 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 217/2157 objects degraded (10.060%); 44/2157 objects misplaced (2.040%) 2021-02-10T04:53:48.138 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:45 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:53:48.138 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:45 smithi085 conmon[29319]: T04:53:45.306704+0000 mon.c (mon 2021-02-10T04:53:48.138 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:45 smithi085 conmon[29319]: .1) 112 : audit [DBG] 2021-02-10T04:53:48.138 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:45 smithi085 conmon[29319]: from='client.? 172.21.15.85:0/988956103' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:53:48.139 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:45 smithi085 conmon[29319]: audit 2021-02-10T04:53:45.528154+0000 mon.a (mon.0) 2532 : 2021-02-10T04:53:48.139 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:45 smithi085 conmon[29319]: audit [INF] from='client.? 172.21.15.85:0/852813929' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.8461670652742835}]: dispatch 2021-02-10T04:53:48.139 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:46 smithi085 conmon[29319]: audit 2021-02-10T04:53:45.682319+0000 mon.a (mon.0) 2533 : audit [INF] from='client.? 172.21.15.85:0/852813929' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.8461670652742835}]': finished 2021-02-10T04:53:48.139 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:46 smithi085 conmon[29319]: cluster 2021-02-10T04:53:45.682370+0000 mon.a (mon.0) 2534 : cluster [DBG] osdmap e1190: 8 total, 7 up, 8 in 2021-02-10T04:53:48.139 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:47 smithi085 conmon[29319]: cluster 2021-02-10T04:53:46.457884+0000 2021-02-10T04:53:48.139 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:47 smithi085 conmon[29319]: mgr.y (mgr.14140) 1299 : cluster [DBG] pgmap v2144: 194 pgs: 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 128 active+clean; 854 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 217/2157 objects degraded (10.060%); 44/2157 objects misplaced (2.040%) 2021-02-10T04:53:48.139 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:47 smithi085 conmon[29319]: cluster 2021-02-10T04:53:46.701479+0000 mon.a (mon.0) 2535 : cluster [DBG] osdmap e1191: 8 total, 7 up, 8 in 2021-02-10T04:53:48.139 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:53:48.139 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:53:48.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:47 smithi164 conmon[29288]: cluster 2021-02-10T04:53:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:47 smithi164 conmon[29288]: 2021-02-10T04:53:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:47 smithi164 conmon[29288]: 2021 2021-02-10T04:53:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:47 smithi164 conmon[29288]: - 2021-02-10T04:53:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:47 smithi164 conmon[29288]: 02 2021-02-10T04:53:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:47 smithi164 conmon[29288]: - 2021-02-10T04:53:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:47 smithi164 conmon[29288]: 10 2021-02-10T04:53:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:47 smithi164 conmon[29288]: T 2021-02-10T04:53:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:47 smithi164 conmon[29288]: 04 2021-02-10T04:53:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:47 smithi164 conmon[29288]: : 2021-02-10T04:53:48.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:47 smithi164 conmon[29288]: 53 2021-02-10T04:53:48.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:47 smithi164 conmon[29288]: : 2021-02-10T04:53:48.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:47 smithi164 conmon[29288]: 46.457884+0000 mgr.y (mgr. 2021-02-10T04:53:48.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:47 smithi164 conmon[29288]: 14140) 1299 : cluster [DBG] pgmap v2144: 194 pgs: 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 128 active+clean; 854 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 217/2157 objects degraded (10.060%); 44/2157 objects misplaced (2.040%) 2021-02-10T04:53:48.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:47 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:53:48.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:47 smithi164 conmon[29288]: 53:46.701479+0000 mon.a (mon.0) 2535 : cluster [DBG] osdmap e1191: 8 total, 7 up, 8 in 2021-02-10T04:53:48.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:47 smithi164 conmon[29288]: 2021-02-10T04:53:48.191 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:53:48.192 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:53:48.643 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:53:48 smithi164 conmon[37263]: debug 2021-02-10T04:53:48.234+0000 7f1c679e3700 -1 osd.5 pg_epoch: 1191 pg[1.a( empty local-lis/les=1118/1119 n=0 ec=110/19 lis/c=1118/1118 les/c/f=1119/1119/0 sis=1118) [5,2,6] r=0 lpr=1118 crt=0'0 mlcod 0'0 active+clean] scrubber pg(1.a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 2 (0x5566506029a0) 2021-02-10T04:53:48.721 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is set 2021-02-10T04:53:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:48 smithi164 conmon[29288]: cluster 2021-02-10T04:53:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:48 smithi164 conmon[29288]: 2021-02- 2021-02-10T04:53:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:48 smithi164 conmon[29288]: 10T04:53:47.764369+0000 osd.7 (osd.7) 88 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:53:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:48 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:53:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:48 smithi164 conmon[29288]: 10T04:53:47.766213+0000 osd.7 (osd.7) 2021-02-10T04:53:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:48 smithi164 conmon[29288]: 89 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:53:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:48 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:53:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:48 smithi164 conmon[29288]: 10T04:53:47.901658 2021-02-10T04:53:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:48 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2536 : audit [INF] 2021-02-10T04:53:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:48 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/2038924691' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:53:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:48 smithi164 conmon[29288]: cluster 2021-02-10T04:53: 2021-02-10T04:53:49.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:48 smithi164 conmon[29288]: 48.520697+0000 mon.a (mon.0) 2537 : cluster [WRN] Health check update: Degraded data redundancy: 217/2157 objects degraded (10.060%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2021-02-10T04:53:49.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:48 smithi085 conmon[35961]: cluster 2021-02-10T04:53:47.764369+0000 2021-02-10T04:53:49.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:48 smithi085 conmon[35961]: osd.7 (osd.7) 88 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:53:49.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:48 smithi085 conmon[35961]: cluster 2021-02-10T04:53:47.766213 2021-02-10T04:53:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:48 smithi085 conmon[35961]: +0000 osd.7 (osd.7) 89 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:53:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:48 smithi085 conmon[35961]: audit 2021-02-10T04:53:47. 2021-02-10T04:53:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:48 smithi085 conmon[35961]: 901658+0000 mon.a (mon.0) 2536 : audit [INF] from='client.? 172.21.15.85:0/2038924691' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:53:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:48 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:53:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:48 smithi085 conmon[35961]: 02-10T04:53:48.520697+0000 mon.a (mon.0) 2537 2021-02-10T04:53:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:48 smithi085 conmon[35961]: : cluster [WRN] Health check update: Degraded data redundancy: 217/2157 objects degraded (10.060%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2021-02-10T04:53:49.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:48 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:53:49.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:48 smithi085 conmon[29319]: :53:47.764369+0000 osd.7 (osd.7) 88 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:53:49.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:48 smithi085 conmon[29319]: cluster 2021-02-10T04:53:47.766213+0000 osd.7 (osd.7) 89 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:53:49.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:48 smithi085 conmon[29319]: audit 2021-02-10T04:53:47.901658+0000 mon.a (mon.0) 2536 : audit [INF] from='client.? 172.21.15.85:0/2038924691' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:53:49.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:48 smithi085 conmon[29319]: cluster 2021-02-10T04:53:48.520697+0000 mon.a (mon.0) 2537 : cluster [WRN] Health check update: Degraded data redundancy: 217/2157 objects degraded (10.060%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2021-02-10T04:53:49.462 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.1 dump_blocked_ops 2021-02-10T04:53:49.643 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:53:49 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:53:49.668 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:53:49.710 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:49 smithi085 conmon[35961]: cluster 2021-02-10T04:53:48.458774+0000 mgr.y (mgr.14140 2021-02-10T04:53:49.711 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:49 smithi085 conmon[35961]: ) 1300 : cluster [DBG] pgmap v2146: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 854 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 217/2157 objects degraded (10.060%); 44/2157 objects misplaced (2.040%) 2021-02-10T04:53:49.711 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:49 smithi085 conmon[35961]: cluster 2021-02-10T04:53:48.698135+0000 mon.a (mon.0) 2538 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:53:49.711 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:49 smithi085 conmon[35961]: audit 2021-02-10T04:53:48.710854+0000 mon.a (mon.0) 2539 : audit [INF] from='client.? 172.21.15.85:0/2038924691' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:53:49.711 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:49 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:53:49.711 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:49 smithi085 conmon[35961]: 02-10T04:53:48.710921+0000 mon.a (mon.0) 2540 : cluster [DBG] osdmap e1192: 8 total, 7 up, 8 in 2021-02-10T04:53:49.711 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:49 smithi085 conmon[35961]: audit 2021-02-10T04:53:49.711 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:49 smithi085 conmon[35961]: 2021-02-10T04:53:49.213977+0000 mgr.y (mgr 2021-02-10T04:53:49.711 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:49 smithi085 conmon[35961]: .14140) 1301 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:53:50.135 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:49 smithi085 conmon[29319]: cluster 2021-02-10T04:53:48.458774 2021-02-10T04:53:50.135 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:49 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 1300 : cluster [DBG] pgmap v2146: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 854 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 217/2157 objects degraded (10.060%); 44/2157 objects misplaced (2.040%) 2021-02-10T04:53:50.135 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:49 smithi085 conmon[29319]: cluster 2021-02-10T04:53:48.698135+0000 mon.a (mon.0) 2538 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:53:50.135 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:49 smithi085 conmon[29319]: audit 2021-02-10T04:53:48.710854+0000 mon.a (mon.0) 2539 : audit [INF] from='client.? 172.21.15.85:0/2038924691' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:53:50.135 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:49 smithi085 conmon[29319]: cluster 2021-02-10T04:53:48.710921+0000 mon.a (mon.0) 2540 : cluster [DBG] osdmap e1192: 8 total, 7 up, 8 in 2021-02-10T04:53:50.135 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:49 smithi085 conmon[29319]: audit 2021-02-10T04:53:49.213977+0000 mgr.y (mgr.14140) 1301 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:53:50.140 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:53:50.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:49 smithi164 conmon[29288]: cluster 2021-02-10T04:53:48.458774+0000 2021-02-10T04:53:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:49 smithi164 conmon[29288]: mgr.y (mgr.14140) 1300 : cluster [DBG] pgmap v2146: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 854 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 217/2157 objects degraded (10.060%); 44/2157 objects misplaced (2.040%) 2021-02-10T04:53:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:49 smithi164 conmon[29288]: cluster 2021-02-10T04:53:48.698135+0000 mon.a (mon.0) 2021-02-10T04:53:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:49 smithi164 conmon[29288]: 2538 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:53:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:49 smithi164 conmon[29288]: audit 2021-02-10T04:53:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:49 smithi164 conmon[29288]: 2021-02-10T04:53:48. 2021-02-10T04:53:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:49 smithi164 conmon[29288]: 710854+0000 mon.a (mon.0) 2021-02-10T04:53:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:49 smithi164 conmon[29288]: 2539 : audit [INF] from='client.? 172.21.15.85:0/2038924691' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:53:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:49 smithi164 conmon[29288]: cluster 2021-02-10T04:53:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:49 smithi164 conmon[29288]: 2021-02-10T04:53:48.710921+0000 2021-02-10T04:53:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:49 smithi164 conmon[29288]: mon.a (mon.0) 2540 : cluster [DBG] osdmap e1192: 8 total, 7 up, 8 in 2021-02-10T04:53:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:49 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:53:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:49 smithi164 conmon[29288]: -10T04:53:49.213977+0000 mgr.y (mgr.14140 2021-02-10T04:53:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:49 smithi164 conmon[29288]: ) 1301 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:53:50.386 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:53:52.127 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:51 smithi085 conmon[29319]: cluster 2021-02-10T04:53:50.459445+0000 mgr.y (mgr.14140) 1302 : cluster [DBG] pgmap v2148: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 854 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 217/2157 objects degraded (10.060%); 44/2157 objects misplaced (2.040%) 2021-02-10T04:53:52.127 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:51 smithi085 conmon[35961]: cluster 2021-02-10T04:53:50.459445+0000 mgr.y (mgr.14140) 1302 : cluster [DBG] pgmap v2148: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 854 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 217/2157 objects degraded (10.060%); 44/2157 objects misplaced (2.040%) 2021-02-10T04:53:52.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:51 smithi164 conmon[29288]: cluster 2021-02-10T04:53:50.459445+0000 mgr.y ( 2021-02-10T04:53:52.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:51 smithi164 conmon[29288]: mgr.14140) 1302 : cluster [DBG] pgmap v2148: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 854 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 217/2157 objects degraded (10.060%); 44/2157 objects misplaced (2.040%) 2021-02-10T04:53:52.182 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set nodeep-scrub 2021-02-10T04:53:52.412 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:53:53.112 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:53:53.113 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:53:53.140 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2021-02-10T04:53:53.140 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:53:53.141 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd dump -f json-pretty 2021-02-10T04:53:53.268 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:53:53.303 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:53:53.338 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:53:53.357 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:53:53.377 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:53:53.396 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:53:53.400 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:53:53.422 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:53:53.422 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-02-10T04:53:53.422 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: problem getting command descriptions from osd.0 2021-02-10T04:53:53.422 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:53:53.422 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:53:53.422 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:53:53.422 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:53:53.422 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:53:53.422 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:53:53.422 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:53:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:53 smithi164 conmon[29288]: cluster 2021-02-10T04:53:52.460602+0000 2021-02-10T04:53:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:53 smithi164 conmon[29288]: mgr.y (mgr.14140) 1303 : cluster [DBG] pgmap v2149: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 154 MiB/s rd, 44 MiB/s wr, 279 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 281 KiB/s, 1 objects/s recovering 2021-02-10T04:53:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:53 smithi164 conmon[29288]: audit 2021-02-10T04:53 2021-02-10T04:53:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:53 smithi164 conmon[29288]: :53.244927+0000 mon.c (mon.1) 113 : audit [DBG] 2021-02-10T04:53:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:53 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/4256955129' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:53:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:53 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:53:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:53 smithi164 conmon[29288]: -10T04:53:53.522097+0000 mon.a (mon.0 2021-02-10T04:53:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:53 smithi164 conmon[29288]: ) 2541 : cluster [WRN] Health check update: Degraded data redundancy: 214/2196 objects degraded (9.745%), 24 pgs degraded, 49 pgs undersized (PG_DEGRADED) 2021-02-10T04:53:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:53 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:53:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:53 smithi164 conmon[29288]: -02-10T04:53:53 2021-02-10T04:53:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:53 smithi164 conmon[29288]: .527042+0000 mon.a (mon.0) 2542 : 2021-02-10T04:53:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:53 smithi164 conmon[29288]: cluster [DBG] osdmap e1193: 8 total, 7 up, 8 in 2021-02-10T04:53:54.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:53 smithi085 conmon[29319]: cluster 2021-02-10T04:53:52.460602+0000 mgr.y (mgr.14140) 1303 : cluster [DBG] pgmap v2149: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 154 MiB/s rd, 44 MiB/s wr, 279 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 281 KiB/s, 1 objects/s recovering 2021-02-10T04:53:54.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:53 smithi085 conmon[29319]: audit 2021-02-10T04:53:53.244927+0000 mon.c (mon.1) 113 : audit [DBG] from='client.? 172.21.15.85:0/4256955129' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:53:54.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:53 smithi085 conmon[29319]: cluster 2021-02-10T04:53:53.522097+0000 mon.a (mon.0) 2541 : cluster [WRN] Health check update: Degraded data redundancy: 214/2196 objects degraded (9.745%), 24 pgs degraded, 49 pgs undersized (PG_DEGRADED) 2021-02-10T04:53:54.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:53 smithi085 conmon[29319]: cluster 2021-02-10T04:53:53.527042+0000 mon.a (mon.0) 2542 : cluster [DBG] osdmap e1193: 8 total, 7 up, 8 in 2021-02-10T04:53:54.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:53 smithi085 conmon[35961]: cluster 2021-02-10T04:53:52.460602+0000 mgr.y (mgr.14140) 1303 : cluster [DBG] pgmap v2149: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 154 MiB/s rd, 44 MiB/s wr, 279 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 281 KiB/s, 1 objects/s recovering 2021-02-10T04:53:54.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:53 smithi085 conmon[35961]: audit 2021-02-10T04:53:53.244927+0000 mon.c (mon.1) 113 : audit [DBG] from='client.? 172.21.15.85:0/4256955129' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:53:54.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:53 smithi085 conmon[35961]: cluster 2021-02-10T04:53:53.522097+0000 mon.a (mon.0) 2541 : cluster [WRN] Health check update: Degraded data redundancy: 214/2196 objects degraded (9.745%), 24 pgs degraded, 49 pgs undersized (PG_DEGRADED) 2021-02-10T04:53:54.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:53 smithi085 conmon[35961]: cluster 2021-02-10T04:53:53.527042+0000 mon.a (mon.0) 2542 : cluster [DBG] osdmap e1193: 8 total, 7 up, 8 in 2021-02-10T04:53:54.730 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: cluster 2021-02-10T04:53:54.730 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: 2021-02-10T04:53:54.730 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: 2021 2021-02-10T04:53:54.730 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: - 2021-02-10T04:53:54.730 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: 02 2021-02-10T04:53:54.730 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: - 2021-02-10T04:53:54.730 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: 10 2021-02-10T04:53:54.730 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: T 2021-02-10T04:53:54.730 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: 04 2021-02-10T04:53:54.730 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: : 2021-02-10T04:53:54.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: 53 2021-02-10T04:53:54.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: : 2021-02-10T04:53:54.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: 54 2021-02-10T04:53:54.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: . 2021-02-10T04:53:54.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: 195902 2021-02-10T04:53:54.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: +0000 2021-02-10T04:53:54.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: 2021-02-10T04:53:54.731 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[29319]: cluster 2021-02-10T04:53:54.195902 2021-02-10T04:53:54.731 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2653 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:53:54.731 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[29319]: cluster 2021-02-10T04:53:54.197035+0000 osd.4 (osd.4) 2654 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:53:55.071 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: osd.4 2021-02-10T04:53:55.071 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: ( 2021-02-10T04:53:55.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: osd 2021-02-10T04:53:55.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: . 2021-02-10T04:53:55.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: 4 2021-02-10T04:53:55.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: ) 2021-02-10T04:53:55.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: 2653 2021-02-10T04:53:55.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: : 2021-02-10T04:53:55.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: cluster 2021-02-10T04:53:55.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: [DBG] purged_snaps scrub starts 2021-02-10T04:53:55.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: cluster 2021-02-10T04:53:55.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: 2021-02-10T04:53:55.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: 2021 2021-02-10T04:53:55.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: - 2021-02-10T04:53:55.072 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: 02 2021-02-10T04:53:55.073 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: - 2021-02-10T04:53:55.073 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: 10 2021-02-10T04:53:55.073 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: T 2021-02-10T04:53:55.073 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: 04 2021-02-10T04:53:55.073 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: : 2021-02-10T04:53:55.073 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: 53 2021-02-10T04:53:55.073 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: :54.197035+0000 2021-02-10T04:53:55.073 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:54 smithi085 conmon[35961]: osd.4 (osd.4) 2654 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:53:55.076 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:53:55.076 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:53:55.118 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.1 dump_historic_ops 2021-02-10T04:53:55.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:54 smithi164 conmon[29288]: cluster 2021-02-10T04:53:54.195902+0000 osd.4 (osd 2021-02-10T04:53:55.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:54 smithi164 conmon[29288]: .4) 2653 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:53:55.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:54 smithi164 conmon[29288]: cluster 2021-02-10T04:53:54.197035+0000 osd.4 (osd.4) 2654 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:53:55.330 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:53:56.021 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:55 smithi085 conmon[35961]: cluster 2021-02-10T04:53:54.461301+0000 mgr.y (mgr 2021-02-10T04:53:56.021 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:55 smithi085 conmon[35961]: .14140) 1304 : cluster [DBG] pgmap v2151: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 134 MiB/s rd, 39 MiB/s wr, 245 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 246 KiB/s, 1 objects/s recovering 2021-02-10T04:53:56.021 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:55 smithi085 conmon[35961]: cluster 2021-02-10T04:53:54.727066+0000 mon.a (mon. 2021-02-10T04:53:56.021 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:55 smithi085 conmon[35961]: 0) 2543 : cluster [DBG] osdmap e1194: 8 total, 7 up, 8 in 2021-02-10T04:53:56.021 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:55 smithi085 conmon[29319]: cluster 2021-02-10T04:53:54.461301+0000 mgr.y (mgr.14140) 1304 : 2021-02-10T04:53:56.022 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:55 smithi085 conmon[29319]: cluster [DBG] pgmap v2151: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 134 MiB/s rd, 39 MiB/s wr, 245 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 246 KiB/s, 1 objects/s recovering 2021-02-10T04:53:56.022 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:55 smithi085 conmon[29319]: cluster 2021-02-10T04:53:54.727066+0000 mon.a (mon.0) 2543 : cluster [DBG] osdmap e1194: 8 total, 7 up, 8 in 2021-02-10T04:53:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:55 smithi164 conmon[29288]: cluster 2021-02-10T04:53:54.461301+0000 mgr.y (mgr 2021-02-10T04:53:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:55 smithi164 conmon[29288]: .14140) 1304 : cluster [DBG] pgmap v2151: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 134 MiB/s rd, 39 MiB/s wr, 245 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 246 KiB/s, 1 objects/s recovering 2021-02-10T04:53:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:55 smithi164 conmon[29288]: cluster 2021-02-10T04:53:54.727066+0000 mon.a (mon.0) 2543 : cluster [DBG] osdmap e1194: 8 total, 7 up, 8 in 2021-02-10T04:53:57.077 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:53:57.289 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:53:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:57 smithi164 conmon[29288]: cluster 2021-02-10T04:53:56.462122+0000 mgr.y (mgr.14140) 1305 : cluster 2021-02-10T04:53:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:57 smithi164 conmon[29288]: [DBG] pgmap v2153: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 135 MiB/s rd, 39 MiB/s wr, 244 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 246 KiB/s, 1 objects/s recovering 2021-02-10T04:53:58.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:57 smithi085 conmon[29319]: cluster 2021-02-10T04:53:58.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:57 smithi085 conmon[29319]: 2021-02-10T04:53:56.462122+0000 mgr.y (mgr.14140) 1305 : cluster [DBG] pgmap v2153: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 135 MiB/s rd, 39 MiB/s wr, 244 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 246 KiB/s, 1 objects/s recovering 2021-02-10T04:53:58.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:57 smithi085 conmon[35961]: cluster 2021-02-10T04:53:56. 2021-02-10T04:53:58.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:57 smithi085 conmon[35961]: 462122+0000 mgr.y (mgr.14140) 1305 : cluster [DBG] pgmap v2153: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 135 MiB/s rd, 39 MiB/s wr, 244 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 246 KiB/s, 1 objects/s recovering 2021-02-10T04:53:58.536 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is set 2021-02-10T04:53:58.845 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:58 smithi085 conmon[29319]: cluster 2021-02-10T04:53:58.011016+0000 osd.1 (osd.1) 502 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:53:58.845 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:58 smithi085 conmon[29319]: cluster 2021-02-10T04:53:58.012832+0000 osd.1 (osd.1) 503 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:53:58.845 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:58 smithi085 conmon[29319]: audit 2021-02-10T04:53:58.358553+0000 mon.a (mon.0) 2544 : audit [INF] from='client.? 172.21.15.85:0/3067741764' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:53:58.845 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:58 smithi085 conmon[29319]: cluster 2021-02-10T04:53:58.524662+0000 mon.a (mon.0) 2545 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:53:58.845 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:58 smithi085 conmon[29319]: audit 2021-02-10T04:53:58.531505+0000 mon.a (mon.0) 2546 : audit [INF] from='client.? 172.21.15.85:0/3067741764' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:53:58.845 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:58 smithi085 conmon[29319]: cluster 2021-02-10T04:53:58.531665+0000 mon.a (mon.0) 2547 : cluster [DBG] osdmap e1195: 8 total, 7 up, 8 in 2021-02-10T04:53:58.845 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:58 smithi085 conmon[35961]: cluster 2021-02-10T04:53:58.011016+0000 2021-02-10T04:53:58.845 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:58 smithi085 conmon[35961]: osd.1 (osd.1) 502 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:53:58.846 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:58 smithi085 conmon[35961]: cluster 2021-02-10T04:53:58.012832+0000 osd.1 (osd.1) 503 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:53:58.846 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:58 smithi085 conmon[35961]: audit 2021-02-10T04:53:58.358553+0000 mon.a (mon.0) 2544 : audit [INF] from='client.? 172.21.15.85:0/3067741764' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:53:58.846 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:58 smithi085 conmon[35961]: cluster 2021-02-10T04:53:58.524662+0000 mon.a (mon.0) 2545 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:53:58.846 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:58 smithi085 conmon[35961]: audit 2021-02-10T04:53:58.531505+0000 mon.a (mon.0) 2546 : audit [INF] from='client.? 172.21.15.85:0/3067741764' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:53:58.846 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:58 smithi085 conmon[35961]: cluster 2021-02-10T04:53:58.531665+0000 mon.a (mon.0) 2547 : cluster [DBG] osdmap e1195: 8 total, 7 up, 8 in 2021-02-10T04:53:59.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:58 smithi164 conmon[29288]: cluster 2021-02-10T04:53:58.011016+0000 osd.1 (osd.1 2021-02-10T04:53:59.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:58 smithi164 conmon[29288]: ) 502 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:53:59.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:58 smithi164 conmon[29288]: cluster 2021-02-10T04:53:58.012832+0000 osd.1 (osd.1) 503 2021-02-10T04:53:59.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:58 smithi164 conmon[29288]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:53:59.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:58 smithi164 conmon[29288]: audit 2021-02-10T04:53:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:58 smithi164 conmon[29288]: 2021-02-10T04:53 2021-02-10T04:53:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:58 smithi164 conmon[29288]: :58.358553+0000 mon.a ( 2021-02-10T04:53:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:58 smithi164 conmon[29288]: mon.0) 2544 : audit [INF] 2021-02-10T04:53:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:58 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/3067741764' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:53:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:58 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:53:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:58 smithi164 conmon[29288]: -10T04:53:58.524662+0000 2021-02-10T04:53:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:58 smithi164 conmon[29288]: mon.a (mon.0) 2545 : cluster [WRN] 2021-02-10T04:53:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:58 smithi164 conmon[29288]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:53:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:58 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:53:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:58 smithi164 conmon[29288]: 04:53:58.531505+0000 mon.a ( 2021-02-10T04:53:59.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:58 smithi164 conmon[29288]: mon.0) 2546 : audit [INF] 2021-02-10T04:53:59.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:58 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/3067741764' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:53:59.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:58 smithi164 conmon[29288]: cluster 2021-02-10T04:53:59.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:58 smithi164 conmon[29288]: 2021-02-10T04:53 2021-02-10T04:53:59.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:58 smithi164 conmon[29288]: :58.531665+0000 mon.a ( 2021-02-10T04:53:59.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:58 smithi164 conmon[29288]: mon.0) 2547 : cluster [DBG] 2021-02-10T04:53:59.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:58 smithi164 conmon[29288]: osdmap e1195: 8 total, 7 up, 8 in 2021-02-10T04:53:59.644 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:53:59 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:53:59.994 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:53:59.994 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 20, 2021-02-10T04:53:59.994 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 600, 2021-02-10T04:53:59.995 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [ 2021-02-10T04:53:59.995 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:53:59.995 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPush(7.6 1038/1037 [PushOp(7:649ff6e5:::smithi08575397-4:30d, version: 996'481, data_included: [2470657~460040,3474633~442916], data_size: 902956, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:649ff6e5:::smithi08575397-4:30d@996'481, size: 3966716, copy_subset: [0~3966716], clone_subset: {7:649ff6e5:::smithi08575397-4:2ed=[],7:649ff6e5:::smithi08575397-4:31d=[]}, snapset: 377=[]:{21f=[203],2ed=[2ed,2eb,2ea,2d9,2c8,2ae],30d=[308],31d=[31a],345=[345,344,343,33e,33c,33a,335,332,32f],372=[372,370,36f,36b,368,366,35f,354,353,34e,34d,34b,349],377=[377,374,373]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3966716, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2470657, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:53:59.995 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:45:40.526611+0000", 2021-02-10T04:53:59.995 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 499.46548443500001, 2021-02-10T04:53:59.995 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.013833715999999999, 2021-02-10T04:53:59.995 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:53:59.995 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:53:59.995 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:53:59.995 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:53:59.995 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:53:59.995 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:45:40.526611+0000", 2021-02-10T04:53:59.996 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:53:59.996 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:53:59.996 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:53:59.996 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:53:59.996 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:45:40.526611+0000", 2021-02-10T04:53:59.996 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00050879799999999998 2021-02-10T04:53:59.996 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:53:59.996 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:53:59.996 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:53:59.996 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:45:40.527120+0000", 2021-02-10T04:53:59.996 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.4562999999999998e-05 2021-02-10T04:53:59.996 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:53:59.996 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:53:59.996 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:53:59.997 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:45:40.527144+0000", 2021-02-10T04:53:59.997 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.1081000000000002e-05 2021-02-10T04:53:59.997 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:53:59.997 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:53:59.997 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:53:59.997 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:45:40.527185+0000", 2021-02-10T04:53:59.997 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.0604000000000001e-05 2021-02-10T04:53:59.997 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:53:59.997 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:53:59.997 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:53:59.997 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:45:40.527206+0000", 2021-02-10T04:53:59.997 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0097080310000000006 2021-02-10T04:53:59.997 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:53:59.998 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:53:59.998 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:53:59.998 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:45:40.536914+0000", 2021-02-10T04:53:59.998 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.3061000000000001e-05 2021-02-10T04:53:59.998 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:53:59.998 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:53:59.998 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:53:59.998 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:45:40.536927+0000", 2021-02-10T04:53:59.998 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0035175779999999999 2021-02-10T04:53:59.998 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:53:59.998 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:53:59.998 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:53:59.998 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:45:40.540445+0000", 2021-02-10T04:53:59.999 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.013833715999999999 2021-02-10T04:53:59.999 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:53:59.999 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:53:59.999 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:53:59.999 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:53:59.999 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:53:59.999 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDScrubReserve(6.22 RELEASE e1085)", 2021-02-10T04:53:59.999 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:49:01.203787+0000", 2021-02-10T04:53:59.999 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 298.78830826000001, 2021-02-10T04:53:59.999 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0026857389999999999, 2021-02-10T04:53:59.999 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:53:59.999 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:53:59.999 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:53:59.999 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.000 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:00.000 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:49:01.203787+0000", 2021-02-10T04:54:00.000 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:00.000 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.000 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.000 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:00.000 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:49:01.203787+0000", 2021-02-10T04:54:00.000 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.1279000000000003e-05 2021-02-10T04:54:00.000 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.000 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.000 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:00.000 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:49:01.203818+0000", 2021-02-10T04:54:00.001 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.6819999999999997e-06 2021-02-10T04:54:00.001 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.001 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.001 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:00.001 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:49:01.203825+0000", 2021-02-10T04:54:00.001 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0288e-05 2021-02-10T04:54:00.001 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.002 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.002 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:00.002 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:49:01.203835+0000", 2021-02-10T04:54:00.002 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.3090000000000001e-06 2021-02-10T04:54:00.002 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.002 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.002 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:00.002 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:49:01.203842+0000", 2021-02-10T04:54:00.002 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0026159040000000001 2021-02-10T04:54:00.002 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.002 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.002 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:00.002 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:49:01.206458+0000", 2021-02-10T04:54:00.003 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1985e-05 2021-02-10T04:54:00.003 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.003 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.003 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:54:00.003 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:49:01.206470+0000", 2021-02-10T04:54:00.003 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.292e-06 2021-02-10T04:54:00.003 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.003 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.003 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:00.003 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:49:01.206473+0000", 2021-02-10T04:54:00.003 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0026857389999999999 2021-02-10T04:54:00.003 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.003 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:00.004 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.004 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.004 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.004 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDScrubReserve(5.13 REQUEST e1125)", 2021-02-10T04:54:00.004 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:42.707797+0000", 2021-02-10T04:54:00.004 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 137.28429855100001, 2021-02-10T04:54:00.004 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0026698830000000001, 2021-02-10T04:54:00.004 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:00.004 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:54:00.004 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:00.004 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.004 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:00.004 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:42.707797+0000", 2021-02-10T04:54:00.005 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:00.005 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.005 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.005 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:00.005 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:42.707797+0000", 2021-02-10T04:54:00.005 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.3185000000000002e-05 2021-02-10T04:54:00.005 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.005 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.005 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:00.005 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:42.707830+0000", 2021-02-10T04:54:00.005 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.1279999999999996e-06 2021-02-10T04:54:00.005 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.005 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.005 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:00.006 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:42.707836+0000", 2021-02-10T04:54:00.006 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2726e-05 2021-02-10T04:54:00.006 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.006 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.006 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:00.006 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:42.707849+0000", 2021-02-10T04:54:00.006 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.9969999999999993e-06 2021-02-10T04:54:00.006 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.006 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.006 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:00.006 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:42.707859+0000", 2021-02-10T04:54:00.006 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.9902999999999998e-05 2021-02-10T04:54:00.006 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.007 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.007 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:00.007 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:42.707929+0000", 2021-02-10T04:54:00.007 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.4039999999999995e-06 2021-02-10T04:54:00.007 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.007 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.007 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op must wait for map", 2021-02-10T04:54:00.007 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:42.707937+0000", 2021-02-10T04:54:00.007 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0024856959999999999 2021-02-10T04:54:00.007 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.007 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.007 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:00.007 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:42.710423+0000", 2021-02-10T04:54:00.007 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0876000000000001e-05 2021-02-10T04:54:00.008 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.008 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.008 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:54:00.008 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:42.710434+0000", 2021-02-10T04:54:00.008 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.2968e-05 2021-02-10T04:54:00.008 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.008 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.008 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:00.008 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:42.710467+0000", 2021-02-10T04:54:00.008 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0026698830000000001 2021-02-10T04:54:00.008 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.008 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:00.009 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.009 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.009 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.009 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDScrubReserve(4.f GRANT e1133)", 2021-02-10T04:54:00.009 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:58.349630+0000", 2021-02-10T04:54:00.009 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 121.642465417, 2021-02-10T04:54:00.009 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001509364, 2021-02-10T04:54:00.009 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:00.009 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:54:00.009 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:00.009 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.009 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:00.009 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:58.349630+0000", 2021-02-10T04:54:00.010 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:00.010 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.010 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.010 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:00.010 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:58.349630+0000", 2021-02-10T04:54:00.010 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.3569999999999998e-05 2021-02-10T04:54:00.010 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.010 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.010 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:00.011 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:58.349674+0000", 2021-02-10T04:54:00.011 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.453e-05 2021-02-10T04:54:00.011 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.011 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.011 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:00.011 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:58.349688+0000", 2021-02-10T04:54:00.011 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.5824999999999998e-05 2021-02-10T04:54:00.011 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.011 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.011 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:00.011 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:58.349704+0000", 2021-02-10T04:54:00.012 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.0309999999999998e-06 2021-02-10T04:54:00.012 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.012 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.012 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:00.012 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:58.349712+0000", 2021-02-10T04:54:00.012 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0014020409999999999 2021-02-10T04:54:00.012 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.012 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.012 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:00.012 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:58.351114+0000", 2021-02-10T04:54:00.012 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0040999999999999e-05 2021-02-10T04:54:00.012 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.013 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.013 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:54:00.013 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:58.351124+0000", 2021-02-10T04:54:00.013 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.7079999999999992e-06 2021-02-10T04:54:00.013 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.013 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.013 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:54:00.013 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:58.351134+0000", 2021-02-10T04:54:00.013 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.6180000000000003e-06 2021-02-10T04:54:00.013 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.013 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.013 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:00.013 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:58.351139+0000", 2021-02-10T04:54:00.014 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001509364 2021-02-10T04:54:00.014 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.014 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:00.014 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.014 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.014 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.014 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDScrubReserve(6.23 REQUEST e1162)", 2021-02-10T04:54:00.014 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:53:10.954648+0000", 2021-02-10T04:54:00.014 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 49.037447651000001, 2021-02-10T04:54:00.014 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0022612159999999999, 2021-02-10T04:54:00.014 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:00.014 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:54:00.014 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:00.014 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.015 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:00.015 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:10.954648+0000", 2021-02-10T04:54:00.015 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:00.015 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.015 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.015 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:00.015 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:10.954648+0000", 2021-02-10T04:54:00.015 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.8775000000000001e-05 2021-02-10T04:54:00.015 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.015 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.015 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:00.015 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:10.954677+0000", 2021-02-10T04:54:00.015 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.0900000000000001e-06 2021-02-10T04:54:00.016 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.016 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.016 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:00.016 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:10.954683+0000", 2021-02-10T04:54:00.016 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0467e-05 2021-02-10T04:54:00.016 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.016 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.016 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:00.016 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:10.954693+0000", 2021-02-10T04:54:00.016 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.4579999999999993e-06 2021-02-10T04:54:00.016 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.016 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.016 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:00.016 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:10.954702+0000", 2021-02-10T04:54:00.017 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.1063000000000005e-05 2021-02-10T04:54:00.017 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.017 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.017 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:00.017 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:10.954793+0000", 2021-02-10T04:54:00.017 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.6080000000000003e-06 2021-02-10T04:54:00.017 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.017 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.017 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op must wait for map", 2021-02-10T04:54:00.017 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:10.954798+0000", 2021-02-10T04:54:00.017 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0020520880000000001 2021-02-10T04:54:00.017 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.018 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.018 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:00.018 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:10.956850+0000", 2021-02-10T04:54:00.018 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.9319999999999999e-06 2021-02-10T04:54:00.018 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.018 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.018 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:54:00.018 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:10.956857+0000", 2021-02-10T04:54:00.018 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.1734999999999998e-05 2021-02-10T04:54:00.018 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.018 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.018 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:00.018 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:10.956909+0000", 2021-02-10T04:54:00.019 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0022612159999999999 2021-02-10T04:54:00.019 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.019 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:00.019 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.019 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.019 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.019 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "replica_scrub(pg: 7.15,from:0'0,to:1000'444,epoch:1162/1122,start:7:a8000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2021-02-10T04:54:00.019 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:53:11.048881+0000", 2021-02-10T04:54:00.019 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 48.943214357000002, 2021-02-10T04:54:00.019 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0011760589999999999, 2021-02-10T04:54:00.019 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:00.019 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:54:00.019 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:00.019 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.020 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:00.020 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:11.048881+0000", 2021-02-10T04:54:00.020 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:00.020 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.020 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.020 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:00.020 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:11.048881+0000", 2021-02-10T04:54:00.020 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.4139e-05 2021-02-10T04:54:00.020 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.020 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.020 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:00.020 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:11.048915+0000", 2021-02-10T04:54:00.020 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.4009999999999997e-06 2021-02-10T04:54:00.021 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.021 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.021 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:00.021 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:11.048924+0000", 2021-02-10T04:54:00.021 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.7982000000000002e-05 2021-02-10T04:54:00.021 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.021 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.021 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:00.021 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:11.048942+0000", 2021-02-10T04:54:00.021 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.0999999999999998e-06 2021-02-10T04:54:00.021 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.021 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.021 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:00.021 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:11.048949+0000", 2021-02-10T04:54:00.022 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00015702199999999999 2021-02-10T04:54:00.022 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.022 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.022 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:00.022 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:11.049106+0000", 2021-02-10T04:54:00.022 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00095141500000000005 2021-02-10T04:54:00.022 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.022 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.023 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:00.023 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:11.050057+0000", 2021-02-10T04:54:00.023 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0011760589999999999 2021-02-10T04:54:00.023 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.023 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:00.023 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.023 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.023 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.023 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop_reply(osd.1.0:255 4.12 e1163/1129 ondisk, result = 0)", 2021-02-10T04:54:00.023 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:53:14.198464+0000", 2021-02-10T04:54:00.023 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 45.793631263999998, 2021-02-10T04:54:00.023 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0026725379999999999, 2021-02-10T04:54:00.023 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:00.024 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:54:00.024 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:00.024 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.024 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:00.024 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.198464+0000", 2021-02-10T04:54:00.024 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:00.024 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.024 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.024 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:00.024 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.198464+0000", 2021-02-10T04:54:00.024 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.908e-05 2021-02-10T04:54:00.024 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.024 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.025 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:00.025 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.198483+0000", 2021-02-10T04:54:00.025 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9320000000000004e-06 2021-02-10T04:54:00.025 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.025 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.025 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:00.025 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.198488+0000", 2021-02-10T04:54:00.025 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2391e-05 2021-02-10T04:54:00.025 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.025 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.025 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:00.025 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.198501+0000", 2021-02-10T04:54:00.025 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.5099999999999998e-06 2021-02-10T04:54:00.025 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.026 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.026 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:00.026 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.198506+0000", 2021-02-10T04:54:00.026 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.002550189 2021-02-10T04:54:00.026 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.026 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.026 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:00.026 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201056+0000", 2021-02-10T04:54:00.026 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.1579999999999999e-06 2021-02-10T04:54:00.026 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.026 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.026 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:54:00.026 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201063+0000", 2021-02-10T04:54:00.027 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.3277999999999999e-05 2021-02-10T04:54:00.027 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.027 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.027 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:00.027 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201137+0000", 2021-02-10T04:54:00.027 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0026725379999999999 2021-02-10T04:54:00.027 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.027 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:00.027 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.027 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.027 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.027 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop_reply(osd.1.0:255 4.12 e1163/1129 ondisk, result = 0)", 2021-02-10T04:54:00.027 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:53:14.198781+0000", 2021-02-10T04:54:00.028 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 45.793314784000003, 2021-02-10T04:54:00.028 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00246247, 2021-02-10T04:54:00.028 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:00.028 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:54:00.028 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:00.028 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.028 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:00.028 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.198781+0000", 2021-02-10T04:54:00.028 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:00.028 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.028 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.028 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:00.028 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.198781+0000", 2021-02-10T04:54:00.028 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.2428000000000002e-05 2021-02-10T04:54:00.029 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.029 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.029 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:00.029 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.198813+0000", 2021-02-10T04:54:00.029 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.4359999999999999e-06 2021-02-10T04:54:00.029 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.029 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.029 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:00.029 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.198820+0000", 2021-02-10T04:54:00.029 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0390999999999999e-05 2021-02-10T04:54:00.029 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.029 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.029 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:00.030 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.198830+0000", 2021-02-10T04:54:00.030 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.5820000000000001e-06 2021-02-10T04:54:00.030 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.030 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.030 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:00.030 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.198835+0000", 2021-02-10T04:54:00.030 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0023251650000000001 2021-02-10T04:54:00.030 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.030 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.030 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:00.030 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201161+0000", 2021-02-10T04:54:00.030 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.4490000000000002e-06 2021-02-10T04:54:00.030 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.031 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.031 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:54:00.031 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201168+0000", 2021-02-10T04:54:00.031 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.5018999999999994e-05 2021-02-10T04:54:00.031 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.031 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.031 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:00.031 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201243+0000", 2021-02-10T04:54:00.031 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00246247 2021-02-10T04:54:00.031 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.031 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:00.031 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.032 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.032 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.032 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop_reply(osd.1.0:256 4.12 e1163/1129 ondisk, result = 0)", 2021-02-10T04:54:00.032 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:53:14.198896+0000", 2021-02-10T04:54:00.032 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 45.793199598999998, 2021-02-10T04:54:00.032 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0024031920000000002, 2021-02-10T04:54:00.032 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:00.032 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:54:00.032 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:00.033 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.033 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:00.033 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.198896+0000", 2021-02-10T04:54:00.033 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:00.033 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.033 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.033 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:00.033 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.198896+0000", 2021-02-10T04:54:00.033 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.7742000000000001e-05 2021-02-10T04:54:00.033 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.034 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.034 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:00.034 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.198914+0000", 2021-02-10T04:54:00.034 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.6219999999999998e-06 2021-02-10T04:54:00.034 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.034 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.034 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:00.034 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.198917+0000", 2021-02-10T04:54:00.034 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.9989999999999997e-06 2021-02-10T04:54:00.034 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.034 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.034 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:00.034 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.198923+0000", 2021-02-10T04:54:00.035 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.3270000000000001e-06 2021-02-10T04:54:00.035 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.035 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.035 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:00.035 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.198927+0000", 2021-02-10T04:54:00.035 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0023472319999999999 2021-02-10T04:54:00.035 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.035 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.035 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:00.035 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201274+0000", 2021-02-10T04:54:00.035 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.2879999999999998e-06 2021-02-10T04:54:00.035 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.035 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.035 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:54:00.036 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201280+0000", 2021-02-10T04:54:00.036 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.8981999999999999e-05 2021-02-10T04:54:00.036 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.036 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.036 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:00.036 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201299+0000", 2021-02-10T04:54:00.036 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0024031920000000002 2021-02-10T04:54:00.036 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.036 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:00.036 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.036 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.036 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.036 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop_reply(osd.1.0:257 4.12 e1163/1129 ondisk, result = 0)", 2021-02-10T04:54:00.037 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:53:14.199170+0000", 2021-02-10T04:54:00.037 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 45.792925341, 2021-02-10T04:54:00.037 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0022613949999999998, 2021-02-10T04:54:00.037 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:00.037 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:54:00.037 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:00.037 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.037 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:00.037 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199170+0000", 2021-02-10T04:54:00.037 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:00.037 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.037 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.037 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:00.038 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199170+0000", 2021-02-10T04:54:00.038 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.9148999999999999e-05 2021-02-10T04:54:00.038 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.038 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.038 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:00.038 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199189+0000", 2021-02-10T04:54:00.038 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.963e-06 2021-02-10T04:54:00.038 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.038 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.038 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:00.038 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199193+0000", 2021-02-10T04:54:00.038 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.2650000000000002e-06 2021-02-10T04:54:00.038 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.038 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.039 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:00.039 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199199+0000", 2021-02-10T04:54:00.039 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.8920000000000002e-06 2021-02-10T04:54:00.039 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.039 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.039 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:00.039 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199203+0000", 2021-02-10T04:54:00.039 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.002218406 2021-02-10T04:54:00.039 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.039 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.039 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:00.039 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201422+0000", 2021-02-10T04:54:00.039 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.5750000000000001e-06 2021-02-10T04:54:00.039 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.040 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.040 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:54:00.040 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201425+0000", 2021-02-10T04:54:00.040 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.1449999999999996e-06 2021-02-10T04:54:00.040 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.040 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.040 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:00.040 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201431+0000", 2021-02-10T04:54:00.040 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0022613949999999998 2021-02-10T04:54:00.040 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.040 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:00.040 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.040 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.041 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.041 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop_reply(osd.1.0:256 4.12 e1163/1129 ondisk, result = 0)", 2021-02-10T04:54:00.041 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:53:14.199380+0000", 2021-02-10T04:54:00.041 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 45.792715694000002, 2021-02-10T04:54:00.041 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.002014498, 2021-02-10T04:54:00.041 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:00.041 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:54:00.041 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:00.041 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.041 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:00.041 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199380+0000", 2021-02-10T04:54:00.041 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:00.041 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.042 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.042 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:00.042 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199380+0000", 2021-02-10T04:54:00.042 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.1387000000000003e-05 2021-02-10T04:54:00.042 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.042 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.042 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:00.042 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199411+0000", 2021-02-10T04:54:00.042 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.8690000000000001e-06 2021-02-10T04:54:00.042 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.043 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.043 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:00.043 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199417+0000", 2021-02-10T04:54:00.043 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.9839999999999996e-06 2021-02-10T04:54:00.043 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.043 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.043 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:00.043 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199427+0000", 2021-02-10T04:54:00.043 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.8180000000000003e-06 2021-02-10T04:54:00.043 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.043 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.043 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:00.044 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199432+0000", 2021-02-10T04:54:00.044 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0018845299999999999 2021-02-10T04:54:00.044 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.044 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.044 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:00.044 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201316+0000", 2021-02-10T04:54:00.044 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.263e-06 2021-02-10T04:54:00.044 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.044 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.044 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:54:00.044 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201323+0000", 2021-02-10T04:54:00.044 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.1647e-05 2021-02-10T04:54:00.044 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.044 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.045 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:00.045 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201394+0000", 2021-02-10T04:54:00.045 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.002014498 2021-02-10T04:54:00.045 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.045 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:00.045 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.045 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.045 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.045 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop_reply(osd.1.0:258 4.12 e1163/1129 ondisk, result = 0)", 2021-02-10T04:54:00.045 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:53:14.199517+0000", 2021-02-10T04:54:00.045 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 45.792578730999999, 2021-02-10T04:54:00.045 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.002049159, 2021-02-10T04:54:00.045 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:00.046 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:54:00.046 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:00.046 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.046 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:00.046 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199517+0000", 2021-02-10T04:54:00.046 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:00.046 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.046 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.046 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:00.046 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199517+0000", 2021-02-10T04:54:00.046 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.7507e-05 2021-02-10T04:54:00.046 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.046 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.047 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:00.047 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199534+0000", 2021-02-10T04:54:00.047 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.5080000000000001e-06 2021-02-10T04:54:00.047 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.047 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.047 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:00.047 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199538+0000", 2021-02-10T04:54:00.047 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.8660000000000002e-06 2021-02-10T04:54:00.047 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.047 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.047 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:00.047 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199544+0000", 2021-02-10T04:54:00.047 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.5190000000000001e-06 2021-02-10T04:54:00.047 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.048 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.048 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:00.048 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199547+0000", 2021-02-10T04:54:00.048 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0020049949999999999 2021-02-10T04:54:00.048 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.048 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.048 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:00.048 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201552+0000", 2021-02-10T04:54:00.048 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.9170000000000003e-06 2021-02-10T04:54:00.048 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.048 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.048 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:54:00.048 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201558+0000", 2021-02-10T04:54:00.049 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.8469999999999996e-06 2021-02-10T04:54:00.049 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.049 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.049 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:00.049 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201566+0000", 2021-02-10T04:54:00.049 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.002049159 2021-02-10T04:54:00.049 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.049 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:00.049 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.049 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.049 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.049 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop_reply(osd.1.0:257 4.12 e1163/1129 ondisk, result = 0)", 2021-02-10T04:54:00.049 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:53:14.199608+0000", 2021-02-10T04:54:00.049 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 45.792487440999999, 2021-02-10T04:54:00.050 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001919544, 2021-02-10T04:54:00.050 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:00.050 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:54:00.050 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:00.050 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.050 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:00.050 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199608+0000", 2021-02-10T04:54:00.050 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:00.050 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.050 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.050 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:00.050 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199608+0000", 2021-02-10T04:54:00.050 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.0032999999999999e-05 2021-02-10T04:54:00.051 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.051 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.051 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:00.051 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199638+0000", 2021-02-10T04:54:00.051 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.4809999999999998e-06 2021-02-10T04:54:00.051 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.051 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.051 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:00.051 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199642+0000", 2021-02-10T04:54:00.051 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.6890000000000001e-06 2021-02-10T04:54:00.051 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.051 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.051 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:00.051 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199647+0000", 2021-02-10T04:54:00.052 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.6710000000000001e-06 2021-02-10T04:54:00.052 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.052 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.052 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:00.052 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199651+0000", 2021-02-10T04:54:00.052 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0018102909999999999 2021-02-10T04:54:00.052 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.052 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.052 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:00.052 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201461+0000", 2021-02-10T04:54:00.052 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.7569999999999998e-06 2021-02-10T04:54:00.052 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.052 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.053 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:54:00.053 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201468+0000", 2021-02-10T04:54:00.053 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.9621999999999999e-05 2021-02-10T04:54:00.053 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.053 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.053 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:00.053 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201528+0000", 2021-02-10T04:54:00.053 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001919544 2021-02-10T04:54:00.053 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.053 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:00.054 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.054 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.054 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.054 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop_reply(osd.1.0:258 4.12 e1163/1129 ondisk, result = 0)", 2021-02-10T04:54:00.054 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:53:14.199763+0000", 2021-02-10T04:54:00.054 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 45.792332653000003, 2021-02-10T04:54:00.054 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0019015200000000001, 2021-02-10T04:54:00.054 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:00.054 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:54:00.054 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:00.054 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.054 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:00.054 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199763+0000", 2021-02-10T04:54:00.055 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:00.055 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.055 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.055 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:00.055 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199763+0000", 2021-02-10T04:54:00.055 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.8804000000000002e-05 2021-02-10T04:54:00.055 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.055 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.055 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:00.055 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199812+0000", 2021-02-10T04:54:00.055 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.049e-06 2021-02-10T04:54:00.055 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.055 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.055 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:00.056 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199818+0000", 2021-02-10T04:54:00.056 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.0044e-05 2021-02-10T04:54:00.056 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.056 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.056 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:00.056 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199838+0000", 2021-02-10T04:54:00.056 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0649999999999998e-06 2021-02-10T04:54:00.056 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.056 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.056 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:00.056 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199843+0000", 2021-02-10T04:54:00.056 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001754603 2021-02-10T04:54:00.056 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.057 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.057 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:00.057 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201597+0000", 2021-02-10T04:54:00.057 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.2190000000000001e-06 2021-02-10T04:54:00.057 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.057 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.057 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:54:00.057 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201604+0000", 2021-02-10T04:54:00.057 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.0736000000000003e-05 2021-02-10T04:54:00.057 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.057 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.057 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:00.057 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201664+0000", 2021-02-10T04:54:00.057 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0019015200000000001 2021-02-10T04:54:00.058 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.058 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:00.058 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.058 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.058 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.058 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop_reply(osd.1.0:259 4.12 e1163/1129 ondisk, result = 0)", 2021-02-10T04:54:00.058 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:53:14.199914+0000", 2021-02-10T04:54:00.058 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 45.792181266999997, 2021-02-10T04:54:00.058 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001777968, 2021-02-10T04:54:00.058 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:00.058 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:54:00.058 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:00.058 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.059 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:00.059 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199914+0000", 2021-02-10T04:54:00.059 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:00.059 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.059 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.059 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:00.059 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199914+0000", 2021-02-10T04:54:00.059 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.8846e-05 2021-02-10T04:54:00.059 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.059 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.059 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:00.059 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199953+0000", 2021-02-10T04:54:00.059 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.2650000000000002e-06 2021-02-10T04:54:00.060 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.060 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.060 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:00.060 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199959+0000", 2021-02-10T04:54:00.060 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.4309999999999994e-06 2021-02-10T04:54:00.060 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.060 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.060 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:00.060 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199968+0000", 2021-02-10T04:54:00.060 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.921e-06 2021-02-10T04:54:00.060 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.060 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.060 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:00.060 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.199973+0000", 2021-02-10T04:54:00.061 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0017110949999999999 2021-02-10T04:54:00.061 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.061 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.061 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:00.061 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201684+0000", 2021-02-10T04:54:00.061 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.9900000000000002e-06 2021-02-10T04:54:00.061 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.061 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.061 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:54:00.061 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201687+0000", 2021-02-10T04:54:00.061 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.4199999999999998e-06 2021-02-10T04:54:00.061 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.061 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.062 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:00.062 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201692+0000", 2021-02-10T04:54:00.062 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001777968 2021-02-10T04:54:00.062 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.062 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:00.062 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.063 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.063 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.063 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop_reply(osd.1.0:259 4.12 e1163/1129 ondisk, result = 0)", 2021-02-10T04:54:00.063 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:53:14.200163+0000", 2021-02-10T04:54:00.063 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 45.791932359999997, 2021-02-10T04:54:00.063 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001599065, 2021-02-10T04:54:00.064 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:00.064 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:54:00.064 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:00.064 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.064 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:00.065 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.200163+0000", 2021-02-10T04:54:00.065 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:00.065 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.065 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.065 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:00.065 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.200163+0000", 2021-02-10T04:54:00.066 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.0348000000000001e-05 2021-02-10T04:54:00.066 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.066 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.066 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:00.066 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.200193+0000", 2021-02-10T04:54:00.066 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.8039999999999998e-06 2021-02-10T04:54:00.067 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.067 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.067 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:00.067 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.200199+0000", 2021-02-10T04:54:00.067 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.8630000000000003e-06 2021-02-10T04:54:00.068 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.068 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.068 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:00.068 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.200209+0000", 2021-02-10T04:54:00.068 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9389999999999996e-06 2021-02-10T04:54:00.068 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.069 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.069 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:00.069 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.200214+0000", 2021-02-10T04:54:00.069 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001494776 2021-02-10T04:54:00.069 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.069 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.070 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:00.070 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201709+0000", 2021-02-10T04:54:00.070 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.6109999999999998e-06 2021-02-10T04:54:00.070 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.070 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.071 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:54:00.071 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201712+0000", 2021-02-10T04:54:00.071 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9724e-05 2021-02-10T04:54:00.071 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.071 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.071 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:00.072 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201762+0000", 2021-02-10T04:54:00.072 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001599065 2021-02-10T04:54:00.072 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.072 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:00.072 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.072 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.073 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.073 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop_reply(osd.1.0:260 4.12 e1163/1129 ondisk, result = 0)", 2021-02-10T04:54:00.073 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:53:14.200264+0000", 2021-02-10T04:54:00.073 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 45.791831328000001, 2021-02-10T04:54:00.073 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0015290620000000001, 2021-02-10T04:54:00.074 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:00.074 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:54:00.074 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:00.074 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.074 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:00.075 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.200264+0000", 2021-02-10T04:54:00.075 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:00.075 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.075 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.075 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:00.076 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.200264+0000", 2021-02-10T04:54:00.076 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.6348e-05 2021-02-10T04:54:00.076 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.076 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.076 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:00.076 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.200280+0000", 2021-02-10T04:54:00.077 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.4369999999999999e-06 2021-02-10T04:54:00.077 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.077 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.077 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:00.077 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.200284+0000", 2021-02-10T04:54:00.077 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.395e-06 2021-02-10T04:54:00.077 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.077 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.078 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:00.078 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.200289+0000", 2021-02-10T04:54:00.078 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.835e-06 2021-02-10T04:54:00.078 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.078 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.078 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:00.078 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.200292+0000", 2021-02-10T04:54:00.078 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0014918069999999999 2021-02-10T04:54:00.079 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.079 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.079 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:00.079 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201784+0000", 2021-02-10T04:54:00.079 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.8340000000000001e-06 2021-02-10T04:54:00.079 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.079 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.079 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:54:00.080 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201788+0000", 2021-02-10T04:54:00.080 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.4060000000000004e-06 2021-02-10T04:54:00.080 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.080 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.080 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:00.080 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201793+0000", 2021-02-10T04:54:00.080 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0015290620000000001 2021-02-10T04:54:00.081 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.081 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:00.081 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.081 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.081 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.081 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop_reply(osd.1.0:260 4.12 e1163/1129 ondisk, result = 0)", 2021-02-10T04:54:00.081 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:53:14.200414+0000", 2021-02-10T04:54:00.081 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 45.791681392000001, 2021-02-10T04:54:00.081 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0014352080000000001, 2021-02-10T04:54:00.082 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:00.082 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:54:00.082 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:00.082 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.082 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:00.082 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.200414+0000", 2021-02-10T04:54:00.082 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:00.082 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.082 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.082 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:00.082 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.200414+0000", 2021-02-10T04:54:00.082 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.4610999999999998e-05 2021-02-10T04:54:00.083 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.083 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.083 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:00.083 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.200449+0000", 2021-02-10T04:54:00.083 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.0090000000000004e-06 2021-02-10T04:54:00.083 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.083 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.083 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:00.083 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.200455+0000", 2021-02-10T04:54:00.083 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0550999999999999e-05 2021-02-10T04:54:00.083 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.083 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.083 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:00.084 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.200465+0000", 2021-02-10T04:54:00.084 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.8850000000000004e-06 2021-02-10T04:54:00.084 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.084 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.084 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:00.084 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.200471+0000", 2021-02-10T04:54:00.084 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00134201 2021-02-10T04:54:00.084 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.084 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.084 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:00.084 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201813+0000", 2021-02-10T04:54:00.084 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.1030000000000002e-06 2021-02-10T04:54:00.084 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.084 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.085 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:54:00.085 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201816+0000", 2021-02-10T04:54:00.085 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.3039e-05 2021-02-10T04:54:00.085 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.085 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.085 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:00.085 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:14.201849+0000", 2021-02-10T04:54:00.085 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0014352080000000001 2021-02-10T04:54:00.085 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.085 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:00.085 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.085 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.085 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.086 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "rep_scrubmap(7.1e e1147 from shard 5)", 2021-02-10T04:54:00.086 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:53:40.134201+0000", 2021-02-10T04:54:00.086 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 19.857894651999999, 2021-02-10T04:54:00.086 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00097472100000000001, 2021-02-10T04:54:00.086 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:00.086 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:54:00.086 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:00.086 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.086 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:00.086 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:40.134201+0000", 2021-02-10T04:54:00.086 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:00.086 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.087 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.087 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:00.087 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:40.134201+0000", 2021-02-10T04:54:00.087 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.8139e-05 2021-02-10T04:54:00.087 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.087 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.087 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:00.087 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:40.134259+0000", 2021-02-10T04:54:00.087 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.5190000000000003e-06 2021-02-10T04:54:00.087 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.087 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.087 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:00.088 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:40.134266+0000", 2021-02-10T04:54:00.088 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0139999999999999e-05 2021-02-10T04:54:00.088 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.088 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.088 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:00.089 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:40.134277+0000", 2021-02-10T04:54:00.089 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.4260000000000002e-06 2021-02-10T04:54:00.089 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.089 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.089 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:00.089 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:40.134282+0000", 2021-02-10T04:54:00.089 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00086885499999999997 2021-02-10T04:54:00.089 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.089 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.089 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:00.089 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:40.135151+0000", 2021-02-10T04:54:00.089 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.2489999999999998e-06 2021-02-10T04:54:00.090 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.090 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.090 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:54:00.090 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:40.135157+0000", 2021-02-10T04:54:00.090 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.8393000000000002e-05 2021-02-10T04:54:00.090 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.090 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.090 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:00.090 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:40.135176+0000", 2021-02-10T04:54:00.090 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00097472100000000001 2021-02-10T04:54:00.090 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.090 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:00.091 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.091 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.091 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.091 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPush(4.b 1188/1187 [PushOp(4:d5da380d:::data_log.25:head, version: 58'1, data_included: [0~156], data_size: 156, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:d5da380d:::data_log.25:head@58'1, size: 156, copy_subset: [0~156], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:156, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2021-02-10T04:54:00.091 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:53:42.696033+0000", 2021-02-10T04:54:00.091 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 17.296062753000001, 2021-02-10T04:54:00.091 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00061331899999999997, 2021-02-10T04:54:00.091 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:00.091 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:54:00.091 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:00.091 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.091 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:00.091 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:42.696033+0000", 2021-02-10T04:54:00.092 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:00.092 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.092 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.092 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:00.092 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:42.696033+0000", 2021-02-10T04:54:00.092 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.2459999999999998e-05 2021-02-10T04:54:00.092 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.092 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.092 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:00.092 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:42.696065+0000", 2021-02-10T04:54:00.092 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.6007000000000001e-05 2021-02-10T04:54:00.092 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.092 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.093 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:00.093 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:42.696081+0000", 2021-02-10T04:54:00.093 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.1549e-05 2021-02-10T04:54:00.093 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.093 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.093 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:00.093 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:42.696103+0000", 2021-02-10T04:54:00.093 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0981e-05 2021-02-10T04:54:00.093 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.093 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.093 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:00.093 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:42.696114+0000", 2021-02-10T04:54:00.093 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.6619000000000004e-05 2021-02-10T04:54:00.093 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.094 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.094 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:00.094 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:42.696210+0000", 2021-02-10T04:54:00.094 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.7610000000000002e-06 2021-02-10T04:54:00.094 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.094 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.094 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:54:00.094 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:42.696220+0000", 2021-02-10T04:54:00.094 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00042594200000000001 2021-02-10T04:54:00.094 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:00.094 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:00.094 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:00.095 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:42.696646+0000", 2021-02-10T04:54:00.095 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00061331899999999997 2021-02-10T04:54:00.095 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.095 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:00.095 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.095 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:00.095 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:00.095 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:54:00.096 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:59 smithi085 conmon[29319]: cluster 2021-02-10T04:53:58.463017 2021-02-10T04:54:00.096 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:59 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 1306 : cluster [DBG] pgmap v2154: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 130 MiB/s rd, 38 MiB/s wr, 237 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 238 KiB/s, 1 objects/s recovering 2021-02-10T04:54:00.096 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:59 smithi085 conmon[29319]: cluster 2021-02-10T04:53:58.850400+0000 osd.3 (osd.3) 34 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:54:00.096 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:59 smithi085 conmon[29319]: cluster 2021-02-10T04:53:58.855050+0000 osd.3 (osd.3) 35 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:54:00.096 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:59 smithi085 conmon[29319]: audit 2021-02-10T04:53:59.224024+0000 mgr.y (mgr.14140) 1307 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:54:00.096 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:53:59 smithi085 conmon[29319]: 2021-02-10T04:54:00.096 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:59 smithi085 conmon[35961]: cluster 2021-02-10T04:53:58 2021-02-10T04:54:00.097 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:59 smithi085 conmon[35961]: .463017+0000 mgr.y (mgr.14140) 1306 : cluster [DBG] pgmap v2154: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 130 MiB/s rd, 38 MiB/s wr, 237 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 238 KiB/s, 1 objects/s recovering 2021-02-10T04:54:00.097 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:59 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:54:00.097 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:59 smithi085 conmon[35961]: -02-10T04:53:58 2021-02-10T04:54:00.097 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:59 smithi085 conmon[35961]: .850400+0000 osd.3 (osd.3) 34 2021-02-10T04:54:00.097 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:59 smithi085 conmon[35961]: : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:54:00.097 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:59 smithi085 conmon[35961]: cluster 2021-02-10T04:54:00.097 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:59 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:54:00.097 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:59 smithi085 conmon[35961]: T04:53:58.855050+0000 osd.3 2021-02-10T04:54:00.097 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:59 smithi085 conmon[35961]: (osd.3) 35 : cluster 2021-02-10T04:54:00.097 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:59 smithi085 conmon[35961]: [DBG] purged_snaps scrub ok 2021-02-10T04:54:00.097 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:59 smithi085 conmon[35961]: audit 2021 2021-02-10T04:54:00.097 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:59 smithi085 conmon[35961]: -02-10T04:53: 2021-02-10T04:54:00.097 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:59 smithi085 conmon[35961]: 59.224024+0000 mgr.y (mgr 2021-02-10T04:54:00.098 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:59 smithi085 conmon[35961]: .14140) 1307 : audit [DBG] 2021-02-10T04:54:00.098 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:53:59 smithi085 conmon[35961]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:54:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:59 smithi164 conmon[29288]: cluster 2021-02-10T04:53:58.463017+0000 2021-02-10T04:54:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:59 smithi164 conmon[29288]: mgr.y (mgr.14140) 1306 : cluster [DBG] pgmap v2154: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 130 MiB/s rd, 38 MiB/s wr, 237 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 238 KiB/s, 1 objects/s recovering 2021-02-10T04:54:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:59 smithi164 conmon[29288]: cluster 2021-02-10T04:53 2021-02-10T04:54:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:59 smithi164 conmon[29288]: :58.850400+0000 osd.3 (osd.3) 34 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:54:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:59 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:54:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:59 smithi164 conmon[29288]: 10T04:53:58.855050+0000 osd.3 (osd 2021-02-10T04:54:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:59 smithi164 conmon[29288]: .3) 35 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:54:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:59 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:54:00.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:59 smithi164 conmon[29288]: -10T04:53:59.224024+0000 2021-02-10T04:54:00.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:59 smithi164 conmon[29288]: mgr.y (mgr.14140) 1307 : 2021-02-10T04:54:00.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:53:59 smithi164 conmon[29288]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:54:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:00 smithi164 conmon[29288]: cluster 2021-02-10T04:53:59.739743+0000 mon.a (mon.0) 2021-02-10T04:54:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:00 smithi164 conmon[29288]: 2548 : cluster [DBG] osdmap e1196: 8 total, 7 up, 8 in 2021-02-10T04:54:01.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:00 smithi085 conmon[35961]: cluster 2021-02-10T04:53:59.739743+0000 mon.a 2021-02-10T04:54:01.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:00 smithi085 conmon[35961]: (mon.0) 2548 : cluster [DBG] osdmap e1196: 8 total, 7 up, 8 in 2021-02-10T04:54:01.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:00 smithi085 conmon[29319]: cluster 2021-02-10T04:53:59.739743+0000 mon.a (mon. 2021-02-10T04:54:01.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:00 smithi085 conmon[29319]: 0) 2548 : cluster [DBG] osdmap e1196: 8 total, 7 up, 8 in 2021-02-10T04:54:01.455 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.2 dump_ops_in_flight 2021-02-10T04:54:01.701 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:54:01.904 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset noscrub 2021-02-10T04:54:01.963 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:54:01.963 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:54:01.963 INFO:teuthology.orchestra.run.smithi085.stdout: "epoch": 1196, 2021-02-10T04:54:01.963 INFO:teuthology.orchestra.run.smithi085.stdout: "fsid": "42224fce-6b57-11eb-8feb-001a4aab830c", 2021-02-10T04:54:01.963 INFO:teuthology.orchestra.run.smithi085.stdout: "created": "2021-02-10T04:20:57.933279+0000", 2021-02-10T04:54:01.963 INFO:teuthology.orchestra.run.smithi085.stdout: "modified": "2021-02-10T04:53:59.732821+0000", 2021-02-10T04:54:01.963 INFO:teuthology.orchestra.run.smithi085.stdout: "last_up_change": "2021-02-10T04:52:31.054977+0000", 2021-02-10T04:54:01.963 INFO:teuthology.orchestra.run.smithi085.stdout: "last_in_change": "2021-02-10T04:51:35.701776+0000", 2021-02-10T04:54:01.963 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T04:54:01.963 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_num": 5806080, 2021-02-10T04:54:01.964 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_set": [ 2021-02-10T04:54:01.964 INFO:teuthology.orchestra.run.smithi085.stdout: "nodeep-scrub", 2021-02-10T04:54:01.964 INFO:teuthology.orchestra.run.smithi085.stdout: "noscrub", 2021-02-10T04:54:01.964 INFO:teuthology.orchestra.run.smithi085.stdout: "pglog_hardlimit", 2021-02-10T04:54:01.964 INFO:teuthology.orchestra.run.smithi085.stdout: "purged_snapdirs", 2021-02-10T04:54:01.964 INFO:teuthology.orchestra.run.smithi085.stdout: "recovery_deletes", 2021-02-10T04:54:01.964 INFO:teuthology.orchestra.run.smithi085.stdout: "sortbitwise" 2021-02-10T04:54:01.964 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:54:01.964 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_version": 18, 2021-02-10T04:54:01.964 INFO:teuthology.orchestra.run.smithi085.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T04:54:01.964 INFO:teuthology.orchestra.run.smithi085.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T04:54:01.964 INFO:teuthology.orchestra.run.smithi085.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T04:54:01.964 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_snapshot": "", 2021-02-10T04:54:01.965 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_max": 7, 2021-02-10T04:54:01.965 INFO:teuthology.orchestra.run.smithi085.stdout: "max_osd": 8, 2021-02-10T04:54:01.965 INFO:teuthology.orchestra.run.smithi085.stdout: "require_min_compat_client": "luminous", 2021-02-10T04:54:01.965 INFO:teuthology.orchestra.run.smithi085.stdout: "min_compat_client": "luminous", 2021-02-10T04:54:01.965 INFO:teuthology.orchestra.run.smithi085.stdout: "require_osd_release": "quincy", 2021-02-10T04:54:01.965 INFO:teuthology.orchestra.run.smithi085.stdout: "pools": [ 2021-02-10T04:54:01.965 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:01.965 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 1, 2021-02-10T04:54:01.965 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "device_health_metrics", 2021-02-10T04:54:01.965 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:24:05.258028+0000", 2021-02-10T04:54:01.965 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:54:01.965 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:54:01.965 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:54:01.966 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:54:01.966 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:54:01.966 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:54:01.966 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:54:01.966 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:54:01.966 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:54:01.966 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:54:01.966 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:54:01.966 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:54:01.966 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 11, 2021-02-10T04:54:01.966 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 11, 2021-02-10T04:54:01.966 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 11, 2021-02-10T04:54:01.967 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 11, 2021-02-10T04:54:01.967 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 11, 2021-02-10T04:54:01.967 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:54:01.967 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:54:01.967 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:54:01.967 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:54:01.967 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:54:01.967 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:54:01.967 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:54:01.967 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:01.967 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "112", 2021-02-10T04:54:01.967 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:54:01.967 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:54:01.968 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "110", 2021-02-10T04:54:01.968 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:54:01.968 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:54:01.968 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:54:01.968 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:54:01.968 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:54:01.968 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:54:01.968 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:54:01.968 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:54:01.968 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:54:01.968 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:54:01.968 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:54:01.968 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:54:01.969 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:54:01.969 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:54:01.969 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:54:01.969 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:54:01.969 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:54:01.969 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:54:01.969 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:54:01.969 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:54:01.969 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:54:01.969 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:54:01.969 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:54:01.969 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:01.969 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:54:01.970 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:54:01.970 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:54:01.970 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:54:01.970 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:54:01.970 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:54:01.970 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:54:01.970 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:54:01.970 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:54:01.970 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:54:01.970 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:54:01.970 INFO:teuthology.orchestra.run.smithi085.stdout: "options": { 2021-02-10T04:54:01.970 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_min": 1 2021-02-10T04:54:01.970 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:01.971 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:54:01.971 INFO:teuthology.orchestra.run.smithi085.stdout: "mgr_devicehealth": {} 2021-02-10T04:54:01.971 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:01.971 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:01.971 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:01.971 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 2, 2021-02-10T04:54:01.971 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": ".rgw.root", 2021-02-10T04:54:01.971 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:32.688554+0000", 2021-02-10T04:54:01.971 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:54:01.971 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:54:01.971 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:54:01.971 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:54:01.972 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:54:01.972 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:54:01.972 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:54:01.972 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:54:01.972 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:54:01.972 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:54:01.972 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:54:01.972 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:54:01.972 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 42, 2021-02-10T04:54:01.972 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 42, 2021-02-10T04:54:01.972 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 42, 2021-02-10T04:54:01.972 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 42, 2021-02-10T04:54:01.972 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 42, 2021-02-10T04:54:01.972 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:54:01.973 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:54:01.973 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:54:01.973 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:54:01.973 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:54:01.973 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:54:01.973 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:54:01.973 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:01.973 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "91", 2021-02-10T04:54:01.973 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:54:01.973 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:54:01.973 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "89", 2021-02-10T04:54:01.973 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:54:01.973 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:54:01.974 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:54:01.974 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:54:01.974 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:54:01.974 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:54:01.974 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:54:01.974 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:54:01.974 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:54:01.974 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:54:01.974 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:54:01.974 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:54:01.974 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:54:01.975 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:54:01.975 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:54:01.975 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:54:01.975 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:54:01.975 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:54:01.975 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:54:01.975 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:54:01.975 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:54:01.975 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:54:01.975 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:54:01.975 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:01.975 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:54:01.975 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:54:01.975 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:54:01.976 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:54:01.976 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:54:01.976 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:54:01.976 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:54:01.976 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:54:01.976 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:54:01.976 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:54:01.976 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:54:01.976 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:54:01.976 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:54:01.976 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T04:54:01.976 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:01.976 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:01.977 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:01.977 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 3, 2021-02-10T04:54:01.977 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "iscsi", 2021-02-10T04:54:01.977 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:47.658613+0000", 2021-02-10T04:54:01.977 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:54:01.977 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:54:01.977 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:54:01.977 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:54:01.977 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:54:01.977 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:54:01.977 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:54:01.977 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:54:01.977 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:54:01.978 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:54:01.978 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:54:01.978 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:54:01.978 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 3, 2021-02-10T04:54:01.978 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 3, 2021-02-10T04:54:01.978 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 3, 2021-02-10T04:54:01.978 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 3, 2021-02-10T04:54:01.978 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 3, 2021-02-10T04:54:01.978 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:54:01.978 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:54:01.978 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:54:01.978 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:54:01.978 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:54:01.979 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:54:01.979 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:54:01.979 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:01.979 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "58", 2021-02-10T04:54:01.979 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:54:01.979 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:54:01.979 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T04:54:01.979 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:54:01.979 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:54:01.979 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:54:01.979 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:54:01.979 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:54:01.979 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:54:01.980 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:54:01.980 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:54:01.980 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:54:01.980 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:54:01.980 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:54:01.980 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:54:01.980 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:54:01.980 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:54:01.980 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:54:01.980 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:54:01.980 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:54:01.980 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:54:01.980 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:54:01.981 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:54:01.981 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:54:01.981 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:54:01.981 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:54:01.981 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:01.981 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:54:01.981 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:54:01.981 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:54:01.981 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:54:01.981 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:54:01.981 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:54:01.981 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:54:01.981 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:54:01.982 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:54:01.982 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:54:01.982 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:54:01.982 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:54:01.982 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:54:01.982 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd": {} 2021-02-10T04:54:01.982 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:01.982 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:01.982 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:01.982 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 4, 2021-02-10T04:54:01.982 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "zone.rgw.log", 2021-02-10T04:54:01.983 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:47.968727+0000", 2021-02-10T04:54:01.983 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:54:01.983 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:54:01.983 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:54:01.983 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:54:01.983 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:54:01.983 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:54:01.983 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:54:01.983 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:54:01.983 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:54:01.983 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:54:01.983 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:54:01.983 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:54:01.984 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 28, 2021-02-10T04:54:01.984 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 22, 2021-02-10T04:54:01.984 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 22, 2021-02-10T04:54:01.984 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 22, 2021-02-10T04:54:01.984 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 28, 2021-02-10T04:54:01.984 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:54:01.984 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "4.1c", 2021-02-10T04:54:01.984 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 1184, 2021-02-10T04:54:01.984 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 1183, 2021-02-10T04:54:01.984 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 1183, 2021-02-10T04:54:01.984 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "1049'39", 2021-02-10T04:54:01.984 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "1049'33" 2021-02-10T04:54:01.984 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:01.985 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "1184", 2021-02-10T04:54:01.985 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:54:01.985 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "1184", 2021-02-10T04:54:01.985 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "1182", 2021-02-10T04:54:01.985 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:54:01.985 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:54:01.985 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:54:01.985 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:54:01.985 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:54:01.985 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:54:01.985 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:54:01.985 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:54:01.985 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:54:01.986 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:54:01.986 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:54:01.986 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:54:01.986 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:54:01.986 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:54:01.986 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:54:01.986 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:54:01.986 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:54:01.986 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:54:01.986 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:54:01.986 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:54:01.986 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:54:01.986 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:54:01.987 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:54:01.987 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:01.987 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:54:01.987 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:54:01.987 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:54:01.987 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:54:01.987 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:54:01.987 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:54:01.987 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:54:01.987 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:54:01.987 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:54:01.987 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:54:01.987 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:54:01.988 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:54:01.988 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:54:01.988 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T04:54:01.988 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:01.988 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:01.988 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:01.988 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 5, 2021-02-10T04:54:01.988 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "zone.rgw.control", 2021-02-10T04:54:01.988 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:49.278260+0000", 2021-02-10T04:54:01.988 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:54:01.988 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:54:01.988 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:54:01.988 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:54:01.989 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:54:01.989 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:54:01.989 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:54:01.989 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:54:01.989 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:54:01.989 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:54:01.989 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:54:01.989 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:54:01.989 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 32, 2021-02-10T04:54:01.989 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 32, 2021-02-10T04:54:01.989 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 32, 2021-02-10T04:54:01.989 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 32, 2021-02-10T04:54:01.989 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 32, 2021-02-10T04:54:01.990 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:54:01.990 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:54:01.990 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:54:01.990 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:54:01.990 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:54:01.990 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:54:01.990 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:54:01.990 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:01.990 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "59", 2021-02-10T04:54:01.990 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:54:01.990 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:54:01.990 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T04:54:01.991 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:54:01.991 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:54:01.991 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:54:01.991 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:54:01.991 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:54:01.991 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:54:01.991 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:54:01.991 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:54:01.991 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:54:01.991 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:54:01.991 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:54:01.991 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:54:01.991 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:54:01.992 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:54:01.992 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:54:01.992 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:54:01.992 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:54:01.992 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:54:01.992 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:54:01.992 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:54:01.992 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:54:01.992 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:54:01.992 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:54:01.992 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:01.992 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:54:01.992 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:54:01.993 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:54:01.993 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:54:01.993 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:54:01.993 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:54:01.993 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:54:01.993 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:54:01.993 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:54:01.993 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:54:01.993 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:54:01.993 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:54:01.993 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:54:01.993 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T04:54:01.993 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:01.994 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:01.994 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:01.994 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 6, 2021-02-10T04:54:01.994 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "zone.rgw.meta", 2021-02-10T04:54:01.994 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:51.792514+0000", 2021-02-10T04:54:01.994 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:54:01.994 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:54:01.994 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:54:01.994 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:54:01.994 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:54:01.994 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:54:01.994 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:54:01.994 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:54:01.995 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:54:01.995 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:54:01.995 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:54:01.995 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:54:01.995 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 42, 2021-02-10T04:54:01.995 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 42, 2021-02-10T04:54:01.995 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 42, 2021-02-10T04:54:01.995 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 42, 2021-02-10T04:54:01.995 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 42, 2021-02-10T04:54:01.995 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:54:01.995 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:54:01.995 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:54:01.995 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:54:01.996 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:54:01.996 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:54:01.996 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:54:01.996 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:01.996 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "1046", 2021-02-10T04:54:01.996 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:54:01.996 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:54:01.996 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "1044", 2021-02-10T04:54:01.996 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:54:01.996 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:54:01.996 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:54:01.996 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:54:01.996 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:54:01.997 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:54:01.997 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:54:01.997 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:54:01.997 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:54:01.997 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:54:01.997 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:54:01.997 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:54:01.997 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:54:01.997 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:54:01.997 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:54:01.997 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:54:01.997 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:54:01.997 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:54:01.997 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:54:01.998 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:54:01.998 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:54:01.998 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:54:01.998 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:54:01.998 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:01.998 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:54:01.998 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:54:01.998 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:54:01.998 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:54:01.998 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:54:01.998 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:54:01.998 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:54:01.999 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:54:01.999 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:54:01.999 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:54:01.999 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:54:01.999 INFO:teuthology.orchestra.run.smithi085.stdout: "options": { 2021-02-10T04:54:01.999 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_bias": 4, 2021-02-10T04:54:01.999 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_min": 8 2021-02-10T04:54:01.999 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:01.999 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:54:01.999 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T04:54:01.999 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:01.999 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.000 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.000 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 7, 2021-02-10T04:54:02.000 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "unique_pool_0", 2021-02-10T04:54:02.000 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:28:23.932391+0000", 2021-02-10T04:54:02.000 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 8193, 2021-02-10T04:54:02.000 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2021-02-10T04:54:02.000 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:54:02.000 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:54:02.000 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:54:02.000 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:54:02.000 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:54:02.001 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:54:02.001 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:54:02.001 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:54:02.001 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:54:02.001 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:54:02.001 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 36, 2021-02-10T04:54:02.001 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 36, 2021-02-10T04:54:02.001 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 36, 2021-02-10T04:54:02.001 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 36, 2021-02-10T04:54:02.001 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 36, 2021-02-10T04:54:02.002 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:54:02.002 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:54:02.002 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:54:02.002 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:54:02.002 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:54:02.002 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:54:02.002 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:54:02.002 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.002 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "1143", 2021-02-10T04:54:02.002 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:54:02.002 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:54:02.002 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "1141", 2021-02-10T04:54:02.002 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:54:02.003 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:54:02.003 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 888, 2021-02-10T04:54:02.003 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 994, 2021-02-10T04:54:02.003 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:54:02.003 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:54:02.003 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:54:02.003 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:54:02.003 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:54:02.003 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:54:02.003 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:54:02.003 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:54:02.003 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:54:02.003 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:54:02.004 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:54:02.004 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:54:02.004 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:54:02.004 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:54:02.004 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:54:02.004 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:54:02.004 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:54:02.004 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:54:02.004 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:54:02.004 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.004 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:54:02.004 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:54:02.004 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:54:02.005 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:54:02.005 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:54:02.005 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:54:02.005 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:54:02.005 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:54:02.005 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:54:02.005 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:54:02.005 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:54:02.005 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:54:02.005 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:54:02.005 INFO:teuthology.orchestra.run.smithi085.stdout: "rados": {} 2021-02-10T04:54:02.005 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.005 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.006 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:54:02.006 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T04:54:02.006 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.006 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 0, 2021-02-10T04:54:02.006 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "45d0bb5f-35fa-4596-964f-75abcee53bee", 2021-02-10T04:54:02.006 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 0, 2021-02-10T04:54:02.006 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:54:02.006 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:54:02.006 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:54:02.006 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:54:02.006 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:54:02.006 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 8, 2021-02-10T04:54:02.007 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1146, 2021-02-10T04:54:02.007 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 1147, 2021-02-10T04:54:02.007 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:54:02.007 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:54:02.007 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.007 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.007 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.007 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6801", 2021-02-10T04:54:02.007 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.007 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.007 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.007 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.007 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:54:02.008 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.008 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.008 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.008 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6802", 2021-02-10T04:54:02.008 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.008 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.008 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.008 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.008 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:54:02.008 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.008 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.008 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.008 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6804", 2021-02-10T04:54:02.008 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.009 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.009 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.009 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.009 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:54:02.009 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.009 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.009 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.009 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6803", 2021-02-10T04:54:02.009 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.009 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.009 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.009 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.010 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6801/7", 2021-02-10T04:54:02.010 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6802/7", 2021-02-10T04:54:02.010 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6804/7", 2021-02-10T04:54:02.010 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6803/7", 2021-02-10T04:54:02.010 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:54:02.010 INFO:teuthology.orchestra.run.smithi085.stdout: "exists" 2021-02-10T04:54:02.010 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.010 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.010 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.010 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 1, 2021-02-10T04:54:02.010 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "3421a5ce-872e-4b52-adf7-6aaa595871a4", 2021-02-10T04:54:02.010 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:54:02.010 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:54:02.011 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:54:02.011 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 0.86566162109375, 2021-02-10T04:54:02.011 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:54:02.011 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:54:02.011 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 13, 2021-02-10T04:54:02.011 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1187, 2021-02-10T04:54:02.011 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:54:02.011 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:54:02.011 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:54:02.011 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.011 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.011 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.011 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6805", 2021-02-10T04:54:02.012 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.012 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.012 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.012 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.012 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:54:02.012 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.012 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.012 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.012 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6806", 2021-02-10T04:54:02.012 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.012 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.012 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.012 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.012 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:54:02.013 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.013 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.013 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.013 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6808", 2021-02-10T04:54:02.013 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.013 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.013 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.013 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.013 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:54:02.013 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.013 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.013 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.013 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6807", 2021-02-10T04:54:02.014 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.014 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.014 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.014 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.014 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6805/7", 2021-02-10T04:54:02.014 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6806/7", 2021-02-10T04:54:02.014 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6808/7", 2021-02-10T04:54:02.014 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6807/7", 2021-02-10T04:54:02.014 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:54:02.014 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:54:02.014 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:54:02.014 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.014 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.015 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.015 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 2, 2021-02-10T04:54:02.015 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "96e1952a-1042-4316-9fc9-86813e88b912", 2021-02-10T04:54:02.015 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:54:02.015 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:54:02.015 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:54:02.015 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:54:02.015 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:54:02.015 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:54:02.015 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 18, 2021-02-10T04:54:02.015 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1167, 2021-02-10T04:54:02.015 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:54:02.015 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:54:02.016 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:54:02.016 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.016 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.016 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.016 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6809", 2021-02-10T04:54:02.016 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.016 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.016 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.016 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.016 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:54:02.016 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.016 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.016 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.017 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6810", 2021-02-10T04:54:02.017 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.017 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.017 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.017 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.017 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:54:02.017 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.017 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.017 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.017 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6812", 2021-02-10T04:54:02.017 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.017 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.017 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.018 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.018 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:54:02.018 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.018 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.018 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.018 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6811", 2021-02-10T04:54:02.018 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.018 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.018 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.018 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.018 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6809/7", 2021-02-10T04:54:02.018 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6810/7", 2021-02-10T04:54:02.018 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6812/7", 2021-02-10T04:54:02.019 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6811/7", 2021-02-10T04:54:02.019 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:54:02.019 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:54:02.019 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:54:02.019 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.019 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.019 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.019 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 3, 2021-02-10T04:54:02.019 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "e019f49f-5dd4-4104-9ced-94c710616642", 2021-02-10T04:54:02.019 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:54:02.019 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:54:02.019 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:54:02.020 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:54:02.020 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 25, 2021-02-10T04:54:02.020 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 1094, 2021-02-10T04:54:02.020 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 1111, 2021-02-10T04:54:02.020 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1147, 2021-02-10T04:54:02.020 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 1095, 2021-02-10T04:54:02.020 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:54:02.020 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:54:02.020 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.021 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.021 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.021 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6813", 2021-02-10T04:54:02.021 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.021 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.021 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.021 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.021 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:54:02.021 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.021 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.021 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.021 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6814", 2021-02-10T04:54:02.021 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.022 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.022 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.022 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.022 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:54:02.022 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.022 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.022 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.022 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6816", 2021-02-10T04:54:02.022 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.022 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.022 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.022 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.023 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:54:02.023 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.023 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.023 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.023 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6815", 2021-02-10T04:54:02.023 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.023 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.023 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.023 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.023 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6813/7", 2021-02-10T04:54:02.023 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6814/7", 2021-02-10T04:54:02.023 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6816/7", 2021-02-10T04:54:02.023 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6815/7", 2021-02-10T04:54:02.024 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:54:02.024 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:54:02.024 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:54:02.024 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.024 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.024 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.024 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 4, 2021-02-10T04:54:02.024 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "91c8a80a-37b6-4c30-a32a-647b2e6d801f", 2021-02-10T04:54:02.024 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:54:02.024 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:54:02.024 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:54:02.024 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:54:02.024 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:54:02.025 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:54:02.025 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 31, 2021-02-10T04:54:02.025 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1184, 2021-02-10T04:54:02.025 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:54:02.025 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:54:02.025 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:54:02.025 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.025 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.025 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.025 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6800", 2021-02-10T04:54:02.025 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.025 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.026 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.026 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.026 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:54:02.026 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.026 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.026 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.026 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6801", 2021-02-10T04:54:02.026 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.026 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.026 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.026 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.026 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:54:02.027 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.027 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.027 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.027 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6803", 2021-02-10T04:54:02.027 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.027 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.027 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.027 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.027 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:54:02.027 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.027 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.027 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.027 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6802", 2021-02-10T04:54:02.028 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.028 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.028 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.028 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.028 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6800/7", 2021-02-10T04:54:02.028 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6801/7", 2021-02-10T04:54:02.028 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6803/7", 2021-02-10T04:54:02.028 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6802/7", 2021-02-10T04:54:02.028 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:54:02.028 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:54:02.028 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:54:02.028 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.028 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.029 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.029 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 5, 2021-02-10T04:54:02.029 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e", 2021-02-10T04:54:02.029 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:54:02.029 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:54:02.029 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:54:02.029 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:54:02.029 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:54:02.029 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:54:02.029 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 37, 2021-02-10T04:54:02.029 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1166, 2021-02-10T04:54:02.029 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:54:02.029 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:54:02.030 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:54:02.030 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.030 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.030 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.030 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6804", 2021-02-10T04:54:02.030 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.030 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.030 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.030 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.030 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:54:02.030 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.030 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.031 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.031 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6805", 2021-02-10T04:54:02.031 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.031 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.031 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.031 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.031 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:54:02.031 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.031 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.031 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.031 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6807", 2021-02-10T04:54:02.031 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.031 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.032 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.032 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.032 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:54:02.032 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.032 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.032 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.032 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6806", 2021-02-10T04:54:02.032 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.032 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.032 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.032 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.032 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6804/7", 2021-02-10T04:54:02.032 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6805/7", 2021-02-10T04:54:02.033 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6807/7", 2021-02-10T04:54:02.033 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6806/7", 2021-02-10T04:54:02.033 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:54:02.033 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:54:02.033 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:54:02.033 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.033 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.033 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.033 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 6, 2021-02-10T04:54:02.033 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "adc40430-3eeb-4003-a824-343021243077", 2021-02-10T04:54:02.033 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:54:02.033 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:54:02.033 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:54:02.034 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:54:02.034 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:54:02.034 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:54:02.034 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 43, 2021-02-10T04:54:02.034 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1166, 2021-02-10T04:54:02.034 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:54:02.034 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:54:02.034 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:54:02.035 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.035 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.035 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.035 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6808", 2021-02-10T04:54:02.035 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.036 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.036 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.036 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.036 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:54:02.036 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.036 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.037 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.038 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6809", 2021-02-10T04:54:02.038 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.038 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.038 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.039 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.039 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:54:02.039 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.039 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.039 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.039 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6811", 2021-02-10T04:54:02.039 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.039 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.039 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.039 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.039 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:54:02.040 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.040 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.040 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.040 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6810", 2021-02-10T04:54:02.040 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.040 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.040 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.040 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.040 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6808/7", 2021-02-10T04:54:02.040 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6809/7", 2021-02-10T04:54:02.041 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6811/7", 2021-02-10T04:54:02.041 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6810/7", 2021-02-10T04:54:02.041 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:54:02.041 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:54:02.041 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:54:02.041 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.041 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.041 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.041 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 7, 2021-02-10T04:54:02.041 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37", 2021-02-10T04:54:02.041 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:54:02.041 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:54:02.041 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:54:02.042 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 0.846160888671875, 2021-02-10T04:54:02.042 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:54:02.042 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:54:02.042 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 50, 2021-02-10T04:54:02.042 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1190, 2021-02-10T04:54:02.042 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:54:02.042 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:54:02.042 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:54:02.042 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.043 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.043 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.043 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6812", 2021-02-10T04:54:02.043 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.043 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.043 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.043 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.043 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:54:02.043 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.044 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.044 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.044 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6813", 2021-02-10T04:54:02.044 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.044 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.044 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.044 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.044 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:54:02.045 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.045 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.045 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.045 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6815", 2021-02-10T04:54:02.045 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.045 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.045 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.046 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.046 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:54:02.046 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:02.046 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.046 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:02.046 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6814", 2021-02-10T04:54:02.046 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:02.047 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.047 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.047 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.047 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6812/7", 2021-02-10T04:54:02.047 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6813/7", 2021-02-10T04:54:02.047 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6815/7", 2021-02-10T04:54:02.047 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6814/7", 2021-02-10T04:54:02.047 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:54:02.048 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:54:02.048 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:54:02.048 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.048 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.048 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:54:02.048 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_xinfo": [ 2021-02-10T04:54:02.048 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.048 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 0, 2021-02-10T04:54:02.049 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:54:02.049 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:54:02.049 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:54:02.049 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:54:02.049 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:54:02.049 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:51:38.084702+0000", 2021-02-10T04:54:02.049 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 1147 2021-02-10T04:54:02.050 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.050 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.050 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 1, 2021-02-10T04:54:02.050 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:54:02.050 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:54:02.050 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:54:02.051 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:54:02.051 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:54:02.051 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:53:58.012834+0000", 2021-02-10T04:54:02.051 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:54:02.051 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.051 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.051 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 2, 2021-02-10T04:54:02.052 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:54:02.052 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:54:02.052 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:54:02.052 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:54:02.052 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:54:02.052 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:53:17.317424+0000", 2021-02-10T04:54:02.052 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:54:02.052 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.053 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.053 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 3, 2021-02-10T04:54:02.053 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:54:02.053 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:54:02.053 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:54:02.053 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:54:02.053 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:54:02.054 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:53:58.855054+0000", 2021-02-10T04:54:02.054 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 1095 2021-02-10T04:54:02.054 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.054 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.054 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 4, 2021-02-10T04:54:02.054 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:54:02.054 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:54:02.054 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:54:02.055 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:54:02.055 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:54:02.055 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:53:54.197037+0000", 2021-02-10T04:54:02.055 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:54:02.055 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.055 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.055 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 5, 2021-02-10T04:54:02.055 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:54:02.055 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:54:02.055 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:54:02.056 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:54:02.056 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:54:02.056 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:53:10.031220+0000", 2021-02-10T04:54:02.056 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:54:02.056 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.056 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.056 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 6, 2021-02-10T04:54:02.056 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:54:02.056 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:54:02.056 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:54:02.056 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:54:02.056 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:54:02.056 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:53:28.255702+0000", 2021-02-10T04:54:02.057 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:54:02.057 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.057 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.057 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 7, 2021-02-10T04:54:02.057 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:54:02.057 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:54:02.057 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:54:02.057 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:54:02.057 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:54:02.057 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:53:47.766217+0000", 2021-02-10T04:54:02.058 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:54:02.058 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.058 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:54:02.058 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_upmap": [], 2021-02-10T04:54:02.058 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_upmap_items": [ 2021-02-10T04:54:02.058 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.059 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "2.7", 2021-02-10T04:54:02.059 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:54:02.059 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.059 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:54:02.059 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 4 2021-02-10T04:54:02.059 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.060 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.060 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.060 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.060 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "2.21", 2021-02-10T04:54:02.060 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:54:02.060 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.061 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:54:02.061 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 1 2021-02-10T04:54:02.061 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.061 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.061 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.061 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.061 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.9", 2021-02-10T04:54:02.062 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:54:02.062 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.062 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 1, 2021-02-10T04:54:02.062 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 6 2021-02-10T04:54:02.062 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.062 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.063 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 3, 2021-02-10T04:54:02.063 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 7 2021-02-10T04:54:02.063 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.063 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.063 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.064 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.064 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.10", 2021-02-10T04:54:02.064 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:54:02.064 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.064 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 1, 2021-02-10T04:54:02.064 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 5 2021-02-10T04:54:02.065 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.065 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.065 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.065 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.065 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.6", 2021-02-10T04:54:02.065 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:54:02.066 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.066 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 5, 2021-02-10T04:54:02.066 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 6 2021-02-10T04:54:02.066 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.066 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.066 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.067 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.067 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.8", 2021-02-10T04:54:02.067 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:54:02.067 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.067 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 1, 2021-02-10T04:54:02.067 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 3 2021-02-10T04:54:02.068 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.068 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.068 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.068 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.068 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.10", 2021-02-10T04:54:02.068 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:54:02.069 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.069 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:54:02.069 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 3 2021-02-10T04:54:02.069 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.069 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.070 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.070 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.070 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.18", 2021-02-10T04:54:02.070 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:54:02.070 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.070 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 4, 2021-02-10T04:54:02.071 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 3 2021-02-10T04:54:02.071 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.071 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.071 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.071 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.071 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.1b", 2021-02-10T04:54:02.072 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:54:02.072 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.072 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 0, 2021-02-10T04:54:02.072 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 3 2021-02-10T04:54:02.072 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.072 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.073 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.073 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.073 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "7.6", 2021-02-10T04:54:02.073 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:54:02.073 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.073 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 3, 2021-02-10T04:54:02.074 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 1 2021-02-10T04:54:02.074 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.075 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.075 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.075 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.075 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "7.15", 2021-02-10T04:54:02.075 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:54:02.075 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.075 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:54:02.075 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 4 2021-02-10T04:54:02.075 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.075 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.075 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.075 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:54:02.075 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_temp": [ 2021-02-10T04:54:02.076 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.076 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.b", 2021-02-10T04:54:02.076 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T04:54:02.076 INFO:teuthology.orchestra.run.smithi085.stdout: 1, 2021-02-10T04:54:02.076 INFO:teuthology.orchestra.run.smithi085.stdout: 4, 2021-02-10T04:54:02.076 INFO:teuthology.orchestra.run.smithi085.stdout: 5 2021-02-10T04:54:02.076 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.076 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.076 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.076 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.16", 2021-02-10T04:54:02.076 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T04:54:02.076 INFO:teuthology.orchestra.run.smithi085.stdout: 2, 2021-02-10T04:54:02.076 INFO:teuthology.orchestra.run.smithi085.stdout: 1, 2021-02-10T04:54:02.077 INFO:teuthology.orchestra.run.smithi085.stdout: 3 2021-02-10T04:54:02.077 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.077 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.077 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.077 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.17", 2021-02-10T04:54:02.077 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T04:54:02.077 INFO:teuthology.orchestra.run.smithi085.stdout: 1, 2021-02-10T04:54:02.077 INFO:teuthology.orchestra.run.smithi085.stdout: 5, 2021-02-10T04:54:02.077 INFO:teuthology.orchestra.run.smithi085.stdout: 3 2021-02-10T04:54:02.077 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.077 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.077 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.077 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.1b", 2021-02-10T04:54:02.078 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T04:54:02.078 INFO:teuthology.orchestra.run.smithi085.stdout: 1, 2021-02-10T04:54:02.078 INFO:teuthology.orchestra.run.smithi085.stdout: 4, 2021-02-10T04:54:02.078 INFO:teuthology.orchestra.run.smithi085.stdout: 3 2021-02-10T04:54:02.078 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.078 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.078 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:02.078 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "7.1b", 2021-02-10T04:54:02.078 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T04:54:02.078 INFO:teuthology.orchestra.run.smithi085.stdout: 2, 2021-02-10T04:54:02.078 INFO:teuthology.orchestra.run.smithi085.stdout: 3, 2021-02-10T04:54:02.078 INFO:teuthology.orchestra.run.smithi085.stdout: 1 2021-02-10T04:54:02.079 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:02.079 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.079 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:54:02.079 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_temp": [], 2021-02-10T04:54:02.079 INFO:teuthology.orchestra.run.smithi085.stdout: "blocklist": { 2021-02-10T04:54:02.079 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/801393150": "2021-02-11T04:21:37.851607+0000", 2021-02-10T04:54:02.079 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/3049671696": "2021-02-11T04:21:37.851607+0000", 2021-02-10T04:54:02.079 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/4056505602": "2021-02-11T04:21:20.700478+0000", 2021-02-10T04:54:02.079 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:6800/7": "2021-02-11T04:21:20.700478+0000", 2021-02-10T04:54:02.079 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/1879026062": "2021-02-11T04:21:20.700478+0000" 2021-02-10T04:54:02.079 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.079 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profiles": { 2021-02-10T04:54:02.079 INFO:teuthology.orchestra.run.smithi085.stdout: "default": { 2021-02-10T04:54:02.080 INFO:teuthology.orchestra.run.smithi085.stdout: "crush-failure-domain": "osd", 2021-02-10T04:54:02.080 INFO:teuthology.orchestra.run.smithi085.stdout: "k": "2", 2021-02-10T04:54:02.080 INFO:teuthology.orchestra.run.smithi085.stdout: "m": "1", 2021-02-10T04:54:02.080 INFO:teuthology.orchestra.run.smithi085.stdout: "plugin": "jerasure", 2021-02-10T04:54:02.080 INFO:teuthology.orchestra.run.smithi085.stdout: "ruleset-failure-domain": "osd", 2021-02-10T04:54:02.080 INFO:teuthology.orchestra.run.smithi085.stdout: "technique": "reed_sol_van" 2021-02-10T04:54:02.080 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.080 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:02.080 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps_queue": [], 2021-02-10T04:54:02.080 INFO:teuthology.orchestra.run.smithi085.stdout: "new_removed_snaps": [], 2021-02-10T04:54:02.080 INFO:teuthology.orchestra.run.smithi085.stdout: "new_purged_snaps": [], 2021-02-10T04:54:02.080 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_node_flags": {}, 2021-02-10T04:54:02.080 INFO:teuthology.orchestra.run.smithi085.stdout: "device_class_flags": {}, 2021-02-10T04:54:02.081 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_mode": { 2021-02-10T04:54:02.081 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_mode_enabled": false, 2021-02-10T04:54:02.081 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_bucket_count": 0, 2021-02-10T04:54:02.081 INFO:teuthology.orchestra.run.smithi085.stdout: "degraded_stretch_mode": 0, 2021-02-10T04:54:02.081 INFO:teuthology.orchestra.run.smithi085.stdout: "recovering_stretch_mode": 0, 2021-02-10T04:54:02.081 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_mode_bucket": 0 2021-02-10T04:54:02.081 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:02.081 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:54:02.093 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:54:02.120 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:54:02.142 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:54:02.143 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:54:02.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:01 smithi164 conmon[29288]: cluster 2021-02-10T04:54:00.463678+0000 mgr.y ( 2021-02-10T04:54:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:01 smithi164 conmon[29288]: mgr.14140) 1308 : cluster [DBG] pgmap v2157: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:02.146 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:01 smithi085 conmon[29319]: cluster 2021-02-10T04:54:00.463678+0000 mgr.y (mgr.14140) 1308 : cluster [DBG] pgmap v2157: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:02.146 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:01 smithi085 conmon[35961]: cluster 2021-02-10T04:54:00.463678 2021-02-10T04:54:02.147 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:01 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1308 : cluster [DBG] pgmap v2157: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:02.166 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:54:02.187 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:54:02.206 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:54:02.229 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:54:02.230 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-02-10T04:54:02.230 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: problem getting command descriptions from osd.0 2021-02-10T04:54:02.230 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:54:02.230 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:54:02.231 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:54:02.231 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:54:02.231 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:54:02.231 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:54:02.231 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:54:03.104 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:02 smithi085 conmon[35961]: audit 2021-02-10T04:54:01.960954+0000 mon.a (mon.0 2021-02-10T04:54:03.104 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:02 smithi085 conmon[35961]: ) 2549 : audit [DBG] from='client.? 172.21.15.85:0/3839939747' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-02-10T04:54:03.104 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:02 smithi085 conmon[35961]: audit 2021-02-10T04:54:02.057432+0000 mon.a (mon.0) 2550 : audit [DBG] from='client.? 172.21.15.85:0/2393963688' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:54:03.105 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:02 smithi085 conmon[29319]: audit 2021-02-10T04:54:01.960954+0000 mon.a (mon.0) 2549 2021-02-10T04:54:03.105 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:02 smithi085 conmon[29319]: : audit [DBG] from='client.? 172.21.15.85:0/3839939747' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-02-10T04:54:03.105 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:02 smithi085 conmon[29319]: audit 2021-02-10T04:54:02.057432+0000 mon.a (mon.0) 2550 : audit [DBG] from='client.? 172.21.15.85:0/2393963688' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:54:03.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:02 smithi164 conmon[29288]: audit 2021-02-10T04:54:01.960954+0000 2021-02-10T04:54:03.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:02 smithi164 conmon[29288]: mon.a (mon.0) 2549 : audit [DBG] from='client.? 172.21.15.85:0/3839939747' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-02-10T04:54:03.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:02 smithi164 conmon[29288]: audit 2021-02-10T04:54:02.057432+0000 mon.a (mon.0) 2550 : audit [DBG] from='client.? 172.21.15.85:0/2393963688' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:54:03.854 INFO:tasks.thrashosds.thrasher:j is {'epoch': 1196, 'fsid': '42224fce-6b57-11eb-8feb-001a4aab830c', 'created': '2021-02-10T04:20:57.933279+0000', 'modified': '2021-02-10T04:53:59.732821+0000', 'last_up_change': '2021-02-10T04:52:31.054977+0000', 'last_in_change': '2021-02-10T04:51:35.701776+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-02-10T04:24:05.258028+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '110', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2021-02-10T04:25:32.688554+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 42, 'pg_placement_num': 42, 'pg_placement_num_target': 42, 'pg_num_target': 42, 'pg_num_pending': 42, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '89', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2021-02-10T04:25:47.658613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'zone.rgw.log', 'create_time': '2021-02-10T04:25:47.968727+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 28, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 28, 'last_pg_merge_meta': {'source_pgid': '4.1c', 'ready_epoch': 1184, 'last_epoch_started': 1183, 'last_epoch_clean': 1183, 'source_version': "1049'39", 'target_version': "1049'33"}, 'last_change': '1184', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '1184', 'last_force_op_resend_preluminous': '1182', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'zone.rgw.control', 'create_time': '2021-02-10T04:25:49.278260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'zone.rgw.meta', 'create_time': '2021-02-10T04:25:51.792514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 42, 'pg_placement_num': 42, 'pg_placement_num_target': 42, 'pg_num_target': 42, 'pg_num_pending': 42, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1046', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '1044', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 8}, 'application_metadata': {'rgw': {}}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2021-02-10T04:28:23.932391+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 36, 'pg_placement_num': 36, 'pg_placement_num_target': 36, 'pg_num_target': 36, 'pg_num_pending': 36, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1143', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '1141', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 888, 'snap_epoch': 994, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '45d0bb5f-35fa-4596-964f-75abcee53bee', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 1146, 'down_at': 1147, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6801', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6802', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6804', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6803', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6801/7', 'cluster_addr': '172.21.15.85:6802/7', 'heartbeat_back_addr': '172.21.15.85:6804/7', 'heartbeat_front_addr': '172.21.15.85:6803/7', 'state': ['exists']}, {'osd': 1, 'uuid': '3421a5ce-872e-4b52-adf7-6aaa595871a4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.86566162109375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 1187, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6805', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6806', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6808', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6807', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6805/7', 'cluster_addr': '172.21.15.85:6806/7', 'heartbeat_back_addr': '172.21.15.85:6808/7', 'heartbeat_front_addr': '172.21.15.85:6807/7', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '96e1952a-1042-4316-9fc9-86813e88b912', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 1167, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6809', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6810', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6812', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6811', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6809/7', 'cluster_addr': '172.21.15.85:6810/7', 'heartbeat_back_addr': '172.21.15.85:6812/7', 'heartbeat_front_addr': '172.21.15.85:6811/7', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'e019f49f-5dd4-4104-9ced-94c710616642', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 25, 'last_clean_end': 1094, 'up_from': 1111, 'up_thru': 1147, 'down_at': 1095, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6813', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6814', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6816', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6815', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6813/7', 'cluster_addr': '172.21.15.85:6814/7', 'heartbeat_back_addr': '172.21.15.85:6816/7', 'heartbeat_front_addr': '172.21.15.85:6815/7', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '91c8a80a-37b6-4c30-a32a-647b2e6d801f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 1184, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6800', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6801', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6803', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6802', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6800/7', 'cluster_addr': '172.21.15.164:6801/7', 'heartbeat_back_addr': '172.21.15.164:6803/7', 'heartbeat_front_addr': '172.21.15.164:6802/7', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 1166, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6804', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6805', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6807', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6806', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6804/7', 'cluster_addr': '172.21.15.164:6805/7', 'heartbeat_back_addr': '172.21.15.164:6807/7', 'heartbeat_front_addr': '172.21.15.164:6806/7', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'adc40430-3eeb-4003-a824-343021243077', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 43, 'up_thru': 1166, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6808', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6809', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6811', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6810', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6808/7', 'cluster_addr': '172.21.15.164:6809/7', 'heartbeat_back_addr': '172.21.15.164:6811/7', 'heartbeat_front_addr': '172.21.15.164:6810/7', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.846160888671875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 1190, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6812', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6813', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6815', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6814', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6812/7', 'cluster_addr': '172.21.15.164:6813/7', 'heartbeat_back_addr': '172.21.15.164:6815/7', 'heartbeat_front_addr': '172.21.15.164:6814/7', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:51:38.084702+0000', 'dead_epoch': 1147}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:53:58.012834+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:53:17.317424+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:53:58.855054+0000', 'dead_epoch': 1095}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:53:54.197037+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:53:10.031220+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:53:28.255702+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:53:47.766217+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.7', 'mappings': [{'from': 7, 'to': 4}]}, {'pgid': '2.21', 'mappings': [{'from': 7, 'to': 1}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 6}, {'from': 3, 'to': 7}]}, {'pgid': '4.10', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '5.6', 'mappings': [{'from': 5, 'to': 6}]}, {'pgid': '5.8', 'mappings': [{'from': 1, 'to': 3}]}, {'pgid': '5.10', 'mappings': [{'from': 7, 'to': 3}]}, {'pgid': '5.18', 'mappings': [{'from': 4, 'to': 3}]}, {'pgid': '5.1b', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '7.6', 'mappings': [{'from': 3, 'to': 1}]}, {'pgid': '7.15', 'mappings': [{'from': 7, 'to': 4}]}], 'pg_temp': [{'pgid': '4.b', 'osds': [1, 4, 5]}, {'pgid': '4.16', 'osds': [2, 1, 3]}, {'pgid': '4.17', 'osds': [1, 5, 3]}, {'pgid': '4.1b', 'osds': [1, 4, 3]}, {'pgid': '7.1b', 'osds': [2, 3, 1]}], 'primary_temp': [], 'blocklist': {'172.21.15.85:0/801393150': '2021-02-11T04:21:37.851607+0000', '172.21.15.85:0/3049671696': '2021-02-11T04:21:37.851607+0000', '172.21.15.85:0/4056505602': '2021-02-11T04:21:20.700478+0000', '172.21.15.85:6800/7': '2021-02-11T04:21:20.700478+0000', '172.21.15.85:0/1879026062': '2021-02-11T04:21:20.700478+0000'}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2021-02-10T04:54:03.854 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:54:03.982 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:54:03.982 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:54:04.131 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:54:04.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:03 smithi164 conmon[29288]: cluster 2021-02-10T04:54:02.464476+0000 mgr.y ( 2021-02-10T04:54:04.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:03 smithi164 conmon[29288]: mgr.14140) 1309 : cluster [DBG] pgmap v2158: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 661 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:04.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:03 smithi164 conmon[29288]: cluster 2021-02-10T04:54:03.530081+0000 mon.a (mon.0) 2551 : cluster [DBG] osdmap e1197: 8 total, 7 up, 8 in 2021-02-10T04:54:04.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:03 smithi085 conmon[29319]: cluster 2021-02-10T04:54:02.464476+0000 mgr.y ( 2021-02-10T04:54:04.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:03 smithi085 conmon[29319]: mgr.14140) 1309 : cluster [DBG] pgmap v2158: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 661 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:04.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:03 smithi085 conmon[29319]: cluster 2021-02-10T04:54:03.530081+0000 mon.a (mon.0) 2551 : cluster 2021-02-10T04:54:04.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:03 smithi085 conmon[29319]: [DBG] osdmap e1197: 8 total, 7 up, 8 in 2021-02-10T04:54:04.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:03 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:54:04.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:03 smithi085 conmon[35961]: 54:02.464476+0000 mgr.y (mgr.14140) 1309 : cluster [DBG] pgmap v2158: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 661 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:04.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:03 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:54:04.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:03 smithi085 conmon[35961]: 10T04:54:03.530081+0000 mon.a (mon 2021-02-10T04:54:04.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:03 smithi085 conmon[35961]: .0) 2551 : cluster [DBG] osdmap e1197: 8 total, 7 up, 8 in 2021-02-10T04:54:05.984 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:54:06.009 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:05 smithi085 conmon[29319]: cluster 2021-02-10T04:54:04.465301+0000 mgr.y (mgr.14140) 1310 : cluster [DBG] pgmap v2160: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:06.009 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:05 smithi085 conmon[35961]: cluster 2021-02-10T04:54:04.465301+0000 mgr.y (mgr.14140) 1310 : cluster [DBG] pgmap v2160: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:06.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:05 smithi164 conmon[29288]: cluster 2021-02-10T04:54:04.465301+0000 mgr.y (mgr.14140) 1310 : cluster 2021-02-10T04:54:06.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:05 smithi164 conmon[29288]: [DBG] pgmap v2160: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:06.227 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:54:07.842 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:07 smithi085 conmon[29319]: cluster 2021-02-10T04:54:06.466066+0000 mgr.y (mgr.14140) 1311 : cluster 2021-02-10T04:54:07.842 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:07 smithi085 conmon[29319]: [DBG] pgmap v2161: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 645 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:07.842 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:07 smithi085 conmon[35961]: cluster 2021-02-10T04:54:06.466066+0000 mgr.y (mgr. 2021-02-10T04:54:07.842 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:07 smithi085 conmon[35961]: 14140) 1311 : cluster [DBG] pgmap v2161: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 645 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:07 smithi164 conmon[29288]: cluster 2021-02-10T04:54:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:07 smithi164 conmon[29288]: 2021-02-10T04:54:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:07 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:54:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:07 smithi164 conmon[29288]: :54:06.466066+0000 mgr.y (mgr.14140) 1311 : cluster [DBG] pgmap v2161: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 645 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:08.887 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:54:08.888 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:54:09.644 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:54:09 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:54:09.724 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:54:09.727 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:54:09.772 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is unset 2021-02-10T04:54:09.870 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.2 dump_blocked_ops 2021-02-10T04:54:10.120 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:54:10.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:09 smithi164 conmon[29288]: cluster 2021-02-10T04:54:08.466935+0000 mgr.y (mgr.14140) 1312 : cluster 2021-02-10T04:54:10.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:09 smithi164 conmon[29288]: [DBG] pgmap v2162: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:10.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:09 smithi164 conmon[29288]: audit 2021-02-10T04:54:09.040051+0000 mon.a (mon.0) 2552 : audit [INF] from='client.? 172.21.15.85:0/91990614' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:54:10.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:09 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:54:10.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:09 smithi164 conmon[29288]: -10T04:54:09.234546+0000 mgr.y (mgr.14140) 1313 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:54:10.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:09 smithi085 conmon[29319]: cluster 2021-02-10T04:54:08.466935+0000 mgr.y (mgr.14140) 1312 : cluster [DBG] pgmap v2162: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:10.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:09 smithi085 conmon[29319]: audit 2021-02-10T04:54:09.040051+0000 mon.a (mon.0) 2552 : audit [INF] from='client.? 172.21.15.85:0/91990614' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:54:10.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:09 smithi085 conmon[29319]: audit 2021-02-10T04:54:09.234546+0000 mgr.y (mgr.14140) 1313 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:54:10.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:09 smithi085 conmon[35961]: cluster 2021-02-10T04:54:08.466935+0000 mgr.y (mgr.14140) 1312 : cluster [DBG] pgmap v2162: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:10.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:09 smithi085 conmon[35961]: audit 2021-02-10T04:54:09.040051+0000 mon.a (mon.0) 2552 : audit [INF] from='client.? 172.21.15.85:0/91990614' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:54:10.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:09 smithi085 conmon[35961]: audit 2021-02-10T04:54:09.234546+0000 mgr.y (mgr.14140) 1313 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:54:10.319 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:54:10.341 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:54:10.358 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:54:10.376 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:54:10.395 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:54:10.421 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:54:10.439 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:54:10.439 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-02-10T04:54:10.440 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: problem getting command descriptions from osd.0 2021-02-10T04:54:10.440 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:54:10.440 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:54:10.440 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:54:10.440 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:54:10.440 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:54:10.440 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:54:10.440 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:54:11.090 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:10 smithi085 conmon[29319]: audit 2021-02-10T04:54:09.703701+0000 mgr.y (mgr. 2021-02-10T04:54:11.091 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:10 smithi085 conmon[29319]: 14140) 1314 : audit [DBG] from='client.25816 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:54:11.091 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:10 smithi085 conmon[29319]: cluster 2021-02-10T04:54:09.757804+0000 mon.a (mon.0) 2553 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:54:11.091 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:10 smithi085 conmon[29319]: audit 2021-02-10T04:54:09.766622+0000 mon.a (mon.0) 2554 : audit [INF] from='client.? 172.21.15.85:0/91990614' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:54:11.091 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:10 smithi085 conmon[29319]: cluster 2021-02-10T04:54:09.766684+0000 mon.a (mon.0) 2555 : cluster [DBG] osdmap e1198: 8 total, 7 up, 8 in 2021-02-10T04:54:11.091 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:10 smithi085 conmon[29319]: audit 2021-02-10T04:54:10.293027+0000 mon.a (mon.0) 2556 : audit [DBG] from='client.? 172.21.15.85:0/3718813164' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:54:11.091 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:10 smithi085 conmon[35961]: audit 2021-02-10T04:54:09.703701+0000 2021-02-10T04:54:11.091 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:10 smithi085 conmon[35961]: mgr.y (mgr.14140) 1314 : audit [DBG] from='client.25816 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:54:11.091 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:10 smithi085 conmon[35961]: cluster 2021-02-10T04:54:09.757804+0000 mon.a 2021-02-10T04:54:11.091 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:10 smithi085 conmon[35961]: (mon.0) 2553 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:54:11.091 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:10 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:54:11.091 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:10 smithi085 conmon[35961]: 54:09.766622+0000 mon.a (mon.0) 2021-02-10T04:54:11.092 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:10 smithi085 conmon[35961]: 2554 : audit [INF] from='client.? 172.21.15.85:0/91990614' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:54:11.092 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:10 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:54:11.092 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:10 smithi085 conmon[35961]: 04:54:09.766684+0000 mon.a (mon. 2021-02-10T04:54:11.092 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:10 smithi085 conmon[35961]: 0) 2555 : cluster [DBG] osdmap e1198: 8 total, 7 up, 8 in 2021-02-10T04:54:11.092 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:10 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:54:11.092 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:10 smithi085 conmon[35961]: -10T04:54:10.293027+0000 mon.a ( 2021-02-10T04:54:11.092 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:10 smithi085 conmon[35961]: mon.0) 2556 : audit [DBG] from='client.? 172.21.15.85:0/3718813164' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:54:11.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:10 smithi164 conmon[29288]: audit 2021-02-10T04:54:09.703701+0000 2021-02-10T04:54:11.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:10 smithi164 conmon[29288]: mgr.y (mgr.14140) 1314 : audit [DBG] from='client.25816 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:54:11.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:10 smithi164 conmon[29288]: cluster 2021-02-10T04:54:09.757804+0000 mon.a (mon.0) 2553 2021-02-10T04:54:11.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:10 smithi164 conmon[29288]: : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:54:11.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:10 smithi164 conmon[29288]: audit 2021-02-10T04:54:11.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:10 smithi164 conmon[29288]: 2021-02-10T04:54:09.766622+0000 2021-02-10T04:54:11.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:10 smithi164 conmon[29288]: mon.a (mon.0) 2554 : audit [INF] 2021-02-10T04:54:11.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:10 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/91990614' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:54:11.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:10 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:54:11.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:10 smithi164 conmon[29288]: -10T04:54:09. 2021-02-10T04:54:11.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:10 smithi164 conmon[29288]: 766684+0000 mon.a (mon.0) 2555 : cluster 2021-02-10T04:54:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:10 smithi164 conmon[29288]: [DBG] osdmap e1198: 8 total, 7 up, 8 in 2021-02-10T04:54:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:10 smithi164 conmon[29288]: audit 2021-02-10T04:54:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:10 smithi164 conmon[29288]: 2021-02-10 2021-02-10T04:54:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:10 smithi164 conmon[29288]: T04:54:10.293027+0000 mon.a (mon 2021-02-10T04:54:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:10 smithi164 conmon[29288]: .0) 2556 : audit [DBG] from='client.? 172.21.15.85:0/3718813164' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:54:11.243 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":2162,"stamp":"2021-02-10T04:54:08.466225+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":931713416,"num_objects":732,"num_object_clones":323,"num_object_copies":2196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":214,"num_objects_misplaced":44,"num_objects_unfound":0,"num_objects_dirty":732,"num_whiteouts":10,"num_read":7357,"num_read_kb":4589602,"num_write":4593,"num_write_kb":1255989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":889,"num_bytes_recovered":307466596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35869,"ondisk_log_size":35869,"up":516,"acting":521,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":590,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2377420,"kb_used_data":1994508,"kb_used_omap":386,"kb_used_meta":382461,"kb_avail":747583796,"statfs":{"total":767960285184,"available":765525807104,"internally_reserved":0,"allocated":2042376192,"data_stored":2618809545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":396036,"internal_metadata":391640316},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.728373"},"pg_stats":[{"pgid":"6.29","version":"0'0","reported_seq":"1045","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.772802+0000","last_change":"2021-02-10T04:53:46.719266+0000","last_active":"2021-02-10T04:53:59.772802+0000","last_peered":"2021-02-10T04:53:59.772802+0000","last_clean":"2021-02-10T04:52:30.064708+0000","last_became_active":"2021-02-10T04:53:46.719266+0000","last_became_peered":"2021-02-10T04:53:46.719266+0000","last_unstale":"2021-02-10T04:53:59.772802+0000","last_undegraded":"2021-02-10T04:53:59.772802+0000","last_fullsized":"2021-02-10T04:53:46.701117+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:24.121684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:24.121684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":"93","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.740115+0000","last_change":"2021-02-10T04:52:30.133105+0000","last_active":"2021-02-10T04:53:59.740115+0000","last_peered":"2021-02-10T04:53:59.740115+0000","last_clean":"2021-02-10T04:53:59.740115+0000","last_became_active":"2021-02-10T04:51:36.864926+0000","last_became_peered":"2021-02-10T04:51:36.864926+0000","last_unstale":"2021-02-10T04:53:59.740115+0000","last_undegraded":"2021-02-10T04:53:59.740115+0000","last_fullsized":"2021-02-10T04:53:59.740115+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.133043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.133043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"1191'4","reported_seq":"152","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:54:03.020507+0000","last_change":"2021-02-10T04:53:46.736164+0000","last_active":"2021-02-10T04:54:03.020507+0000","last_peered":"2021-02-10T04:54:03.020507+0000","last_clean":"2021-02-10T04:54:03.020507+0000","last_became_active":"2021-02-10T04:53:46.735711+0000","last_became_peered":"2021-02-10T04:53:46.735711+0000","last_unstale":"2021-02-10T04:54:03.020507+0000","last_undegraded":"2021-02-10T04:54:03.020507+0000","last_fullsized":"2021-02-10T04:54:03.020507+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'3","last_scrub_stamp":"2021-02-10T04:51:01.185820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.185820+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"1132'66","reported_seq":"1284","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.743340+0000","last_change":"2021-02-10T04:51:55.430490+0000","last_active":"2021-02-10T04:53:59.743340+0000","last_peered":"2021-02-10T04:53:59.743340+0000","last_clean":"2021-02-10T04:53:59.743340+0000","last_became_active":"2021-02-10T04:25:49.002058+0000","last_became_peered":"2021-02-10T04:25:49.002058+0000","last_unstale":"2021-02-10T04:53:59.743340+0000","last_undegraded":"2021-02-10T04:53:59.743340+0000","last_fullsized":"2021-02-10T04:53:59.743340+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1132'66","last_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_deep_scrub":"1132'66","last_deep_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.430399+0000","log_size":66,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":95,"num_read_kb":78,"num_write":55,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"102","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.742419+0000","last_change":"2021-02-10T04:53:41.722609+0000","last_active":"2021-02-10T04:53:59.742419+0000","last_peered":"2021-02-10T04:53:59.742419+0000","last_clean":"2021-02-10T04:53:59.742419+0000","last_became_active":"2021-02-10T04:51:36.875000+0000","last_became_peered":"2021-02-10T04:51:36.875000+0000","last_unstale":"2021-02-10T04:53:59.742419+0000","last_undegraded":"2021-02-10T04:53:59.742419+0000","last_fullsized":"2021-02-10T04:53:59.742419+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:53:41.722516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:41.722516+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"1172","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.743556+0000","last_change":"2021-02-10T04:52:32.077502+0000","last_active":"2021-02-10T04:53:59.743556+0000","last_peered":"2021-02-10T04:53:59.743556+0000","last_clean":"2021-02-10T04:52:30.059220+0000","last_became_active":"2021-02-10T04:52:32.077502+0000","last_became_peered":"2021-02-10T04:52:32.077502+0000","last_unstale":"2021-02-10T04:53:59.743556+0000","last_undegraded":"2021-02-10T04:53:59.743556+0000","last_fullsized":"2021-02-10T04:52:32.069476+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:48.408748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:50:48.408748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"1003'1689","reported_seq":"4618","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:54:00.502229+0000","last_change":"2021-02-10T04:53:38.170873+0000","last_active":"2021-02-10T04:54:00.502229+0000","last_peered":"2021-02-10T04:54:00.502229+0000","last_clean":"2021-02-10T04:54:00.502229+0000","last_became_active":"2021-02-10T04:52:26.030105+0000","last_became_peered":"2021-02-10T04:52:26.030105+0000","last_unstale":"2021-02-10T04:54:00.502229+0000","last_undegraded":"2021-02-10T04:54:00.502229+0000","last_fullsized":"2021-02-10T04:54:00.502229+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:53:38.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.170802+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":99361981,"num_objects":37,"num_object_clones":33,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"1049'32","reported_seq":"43","reported_epoch":"1196","state":"active+clean+remapped","last_fresh":"2021-02-10T04:54:00.501966+0000","last_change":"2021-02-10T04:53:42.716996+0000","last_active":"2021-02-10T04:54:00.501966+0000","last_peered":"2021-02-10T04:54:00.501966+0000","last_clean":"2021-02-10T04:54:00.501966+0000","last_became_active":"2021-02-10T04:53:42.693156+0000","last_became_peered":"2021-02-10T04:53:42.693156+0000","last_unstale":"2021-02-10T04:54:00.501966+0000","last_undegraded":"2021-02-10T04:54:00.501966+0000","last_fullsized":"2021-02-10T04:54:00.501966+0000","mapping_epoch":1187,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1188,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'32","last_scrub_stamp":"2021-02-10T04:51:51.382040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.382040+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":61,"num_read_kb":44,"num_write":33,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1788,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,5],"avail_no_missing":["1","4","5"],"object_location_counts":[{"shards":"1,4,5","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"850","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:54:00.503640+0000","last_change":"2021-02-10T04:53:34.179015+0000","last_active":"2021-02-10T04:54:00.503640+0000","last_peered":"2021-02-10T04:54:00.503640+0000","last_clean":"2021-02-10T04:54:00.503640+0000","last_became_active":"2021-02-10T04:51:49.502576+0000","last_became_peered":"2021-02-10T04:51:49.502576+0000","last_unstale":"2021-02-10T04:54:00.503640+0000","last_undegraded":"2021-02-10T04:54:00.503640+0000","last_fullsized":"2021-02-10T04:54:00.503640+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:34.178949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.178949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"93","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.741642+0000","last_change":"2021-02-10T04:51:53.038893+0000","last_active":"2021-02-10T04:53:59.741642+0000","last_peered":"2021-02-10T04:53:59.741642+0000","last_clean":"2021-02-10T04:53:59.741642+0000","last_became_active":"2021-02-10T04:51:36.855507+0000","last_became_peered":"2021-02-10T04:51:36.855507+0000","last_unstale":"2021-02-10T04:53:59.741642+0000","last_undegraded":"2021-02-10T04:53:59.741642+0000","last_fullsized":"2021-02-10T04:53:59.741642+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:53.038834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:53.038834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"1181","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.739235+0000","last_change":"2021-02-10T04:52:32.089886+0000","last_active":"2021-02-10T04:53:59.739235+0000","last_peered":"2021-02-10T04:53:59.739235+0000","last_clean":"2021-02-10T04:51:27.993389+0000","last_became_active":"2021-02-10T04:52:32.089886+0000","last_became_peered":"2021-02-10T04:52:32.089886+0000","last_unstale":"2021-02-10T04:53:59.739235+0000","last_undegraded":"2021-02-10T04:53:59.739235+0000","last_fullsized":"2021-02-10T04:52:32.083725+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:09.922789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:09.922789+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.8","version":"1168'82","reported_seq":"129","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.740202+0000","last_change":"2021-02-10T04:53:23.326363+0000","last_active":"2021-02-10T04:53:59.740202+0000","last_peered":"2021-02-10T04:53:59.740202+0000","last_clean":"2021-02-10T04:53:59.740202+0000","last_became_active":"2021-02-10T04:51:36.859215+0000","last_became_peered":"2021-02-10T04:51:36.859215+0000","last_unstale":"2021-02-10T04:53:59.740202+0000","last_undegraded":"2021-02-10T04:53:59.740202+0000","last_fullsized":"2021-02-10T04:53:59.740202+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1150'76","last_scrub_stamp":"2021-02-10T04:53:00.964499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:00.964499+0000","log_size":82,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3848803,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"1049'61","reported_seq":"1285","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.741274+0000","last_change":"2021-02-10T04:53:36.836307+0000","last_active":"2021-02-10T04:53:59.741274+0000","last_peered":"2021-02-10T04:53:59.741274+0000","last_clean":"2021-02-10T04:53:59.741274+0000","last_became_active":"2021-02-10T04:25:49.015913+0000","last_became_peered":"2021-02-10T04:25:49.015913+0000","last_unstale":"2021-02-10T04:53:59.741274+0000","last_undegraded":"2021-02-10T04:53:59.741274+0000","last_fullsized":"2021-02-10T04:53:59.741274+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'61","last_scrub_stamp":"2021-02-10T04:53:36.836202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:36.836202+0000","log_size":61,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":104,"num_read_kb":84,"num_write":62,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"1150'5","reported_seq":"134","reported_epoch":"1197","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:03.535112+0000","last_change":"2021-02-10T04:52:32.093814+0000","last_active":"2021-02-10T04:54:03.535112+0000","last_peered":"2021-02-10T04:54:03.535112+0000","last_clean":"2021-02-10T04:52:30.445006+0000","last_became_active":"2021-02-10T04:52:32.093814+0000","last_became_peered":"2021-02-10T04:52:32.093814+0000","last_unstale":"2021-02-10T04:54:03.535112+0000","last_undegraded":"2021-02-10T04:52:32.079897+0000","last_fullsized":"2021-02-10T04:52:32.079762+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'4","last_scrub_stamp":"2021-02-10T04:52:01.121380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.121380+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"94","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.741205+0000","last_change":"2021-02-10T04:52:32.093831+0000","last_active":"2021-02-10T04:53:59.741205+0000","last_peered":"2021-02-10T04:53:59.741205+0000","last_clean":"2021-02-10T04:52:30.057710+0000","last_became_active":"2021-02-10T04:52:32.093831+0000","last_became_peered":"2021-02-10T04:52:32.093831+0000","last_unstale":"2021-02-10T04:53:59.741205+0000","last_undegraded":"2021-02-10T04:53:59.741205+0000","last_fullsized":"2021-02-10T04:52:32.081168+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:52.421793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:52.421793+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"494","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.772584+0000","last_change":"2021-02-10T04:53:46.733236+0000","last_active":"2021-02-10T04:53:59.772584+0000","last_peered":"2021-02-10T04:53:59.772584+0000","last_clean":"2021-02-10T04:53:59.772584+0000","last_became_active":"2021-02-10T04:53:46.732890+0000","last_became_peered":"2021-02-10T04:53:46.732890+0000","last_unstale":"2021-02-10T04:53:59.772584+0000","last_undegraded":"2021-02-10T04:53:59.772584+0000","last_fullsized":"2021-02-10T04:53:59.772584+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:35.170157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:35.170157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"1003'739","reported_seq":"3701","reported_epoch":"1196","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:00.446376+0000","last_change":"2021-02-10T04:52:32.084576+0000","last_active":"2021-02-10T04:54:00.446376+0000","last_peered":"2021-02-10T04:54:00.446376+0000","last_clean":"2021-02-10T04:43:58.648858+0000","last_became_active":"2021-02-10T04:52:32.084576+0000","last_became_peered":"2021-02-10T04:52:32.084576+0000","last_unstale":"2021-02-10T04:54:00.446376+0000","last_undegraded":"2021-02-10T04:52:32.072214+0000","last_fullsized":"2021-02-10T04:52:32.072101+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:52:27.602006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:27.602006+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24059888,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":129,"num_read_kb":109544,"num_write":102,"num_write_kb":46173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"1066'44","reported_seq":"1352","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.741849+0000","last_change":"2021-02-10T04:53:37.108747+0000","last_active":"2021-02-10T04:53:59.741849+0000","last_peered":"2021-02-10T04:53:59.741849+0000","last_clean":"2021-02-10T04:53:59.741849+0000","last_became_active":"2021-02-10T04:53:35.551311+0000","last_became_peered":"2021-02-10T04:53:35.551311+0000","last_unstale":"2021-02-10T04:53:59.741849+0000","last_undegraded":"2021-02-10T04:53:59.741849+0000","last_fullsized":"2021-02-10T04:53:59.741849+0000","mapping_epoch":1118,"log_start":"1066'44","ondisk_log_start":"1066'44","created":55,"last_epoch_clean":1181,"parent":"0.0","parent_split_bits":0,"last_scrub":"1066'44","last_scrub_stamp":"2021-02-10T04:53:37.108677+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:53:37.108677+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1578,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":150,"num_read_kb":114,"num_write":86,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":2312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"1092","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.741019+0000","last_change":"2021-02-10T04:53:39.126356+0000","last_active":"2021-02-10T04:53:59.741019+0000","last_peered":"2021-02-10T04:53:59.741019+0000","last_clean":"2021-02-10T04:53:59.741019+0000","last_became_active":"2021-02-10T04:29:07.145616+0000","last_became_peered":"2021-02-10T04:29:07.145616+0000","last_unstale":"2021-02-10T04:53:59.741019+0000","last_undegraded":"2021-02-10T04:53:59.741019+0000","last_fullsized":"2021-02-10T04:53:59.741019+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:39.126261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:39.126261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"416","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:54:00.446405+0000","last_change":"2021-02-10T04:52:32.083894+0000","last_active":"2021-02-10T04:54:00.446405+0000","last_peered":"2021-02-10T04:54:00.446405+0000","last_clean":"2021-02-10T04:52:30.060966+0000","last_became_active":"2021-02-10T04:52:32.083894+0000","last_became_peered":"2021-02-10T04:52:32.083894+0000","last_unstale":"2021-02-10T04:54:00.446405+0000","last_undegraded":"2021-02-10T04:54:00.446405+0000","last_fullsized":"2021-02-10T04:52:32.079256+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:26.311958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.311958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"97","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.744873+0000","last_change":"2021-02-10T04:53:47.708333+0000","last_active":"2021-02-10T04:53:59.744873+0000","last_peered":"2021-02-10T04:53:59.744873+0000","last_clean":"2021-02-10T04:53:59.744873+0000","last_became_active":"2021-02-10T04:51:36.855744+0000","last_became_peered":"2021-02-10T04:51:36.855744+0000","last_unstale":"2021-02-10T04:53:59.744873+0000","last_undegraded":"2021-02-10T04:53:59.744873+0000","last_fullsized":"2021-02-10T04:53:59.744873+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:02.162851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:13.727241+0000","last_clean_scrub_stamp":"2021-02-10T04:52:02.162851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"1063'654","reported_seq":"4363","reported_epoch":"1196","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:59.742279+0000","last_change":"2021-02-10T04:52:32.093941+0000","last_active":"2021-02-10T04:53:59.742279+0000","last_peered":"2021-02-10T04:53:59.742279+0000","last_clean":"2021-02-10T04:52:30.062769+0000","last_became_active":"2021-02-10T04:52:32.093941+0000","last_became_peered":"2021-02-10T04:52:32.093941+0000","last_unstale":"2021-02-10T04:53:59.742279+0000","last_undegraded":"2021-02-10T04:52:32.085816+0000","last_fullsized":"2021-02-10T04:52:32.085711+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:51:57.121838+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.121838+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24700630,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":171,"num_read_kb":182272,"num_write":111,"num_write_kb":33727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":26349023,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":12}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"1142","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.743478+0000","last_change":"2021-02-10T04:51:50.433056+0000","last_active":"2021-02-10T04:53:59.743478+0000","last_peered":"2021-02-10T04:53:59.743478+0000","last_clean":"2021-02-10T04:53:59.743478+0000","last_became_active":"2021-02-10T04:51:36.843165+0000","last_became_peered":"2021-02-10T04:51:36.843165+0000","last_unstale":"2021-02-10T04:53:59.743478+0000","last_undegraded":"2021-02-10T04:53:59.743478+0000","last_fullsized":"2021-02-10T04:53:59.743478+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.432990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.432990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"1089","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.772936+0000","last_change":"2021-02-10T04:53:46.727888+0000","last_active":"2021-02-10T04:53:59.772936+0000","last_peered":"2021-02-10T04:53:59.772936+0000","last_clean":"2021-02-10T04:53:59.772936+0000","last_became_active":"2021-02-10T04:53:46.727670+0000","last_became_peered":"2021-02-10T04:53:46.727670+0000","last_unstale":"2021-02-10T04:53:59.772936+0000","last_undegraded":"2021-02-10T04:53:59.772936+0000","last_fullsized":"2021-02-10T04:53:59.772936+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:41.041553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.041553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"1187'54","reported_seq":"1273","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.741044+0000","last_change":"2021-02-10T04:53:42.071173+0000","last_active":"2021-02-10T04:53:59.741044+0000","last_peered":"2021-02-10T04:53:59.741044+0000","last_clean":"2021-02-10T04:53:59.741044+0000","last_became_active":"2021-02-10T04:53:39.679752+0000","last_became_peered":"2021-02-10T04:53:39.679752+0000","last_unstale":"2021-02-10T04:53:59.741044+0000","last_undegraded":"2021-02-10T04:53:59.741044+0000","last_fullsized":"2021-02-10T04:53:59.741044+0000","mapping_epoch":1111,"log_start":"1049'39","ondisk_log_start":"1049'39","created":55,"last_epoch_clean":1185,"parent":"0.0","parent_split_bits":0,"last_scrub":"1187'54","last_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_deep_scrub":"1187'54","last_deep_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_clean_scrub_stamp":"2021-02-10T04:53:42.071103+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":114,"num_read_kb":94,"num_write":73,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"835","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:54:00.502235+0000","last_change":"2021-02-10T04:51:52.350764+0000","last_active":"2021-02-10T04:54:00.502235+0000","last_peered":"2021-02-10T04:54:00.502235+0000","last_clean":"2021-02-10T04:54:00.502235+0000","last_became_active":"2021-02-10T04:51:49.567784+0000","last_became_peered":"2021-02-10T04:51:49.567784+0000","last_unstale":"2021-02-10T04:54:00.502235+0000","last_undegraded":"2021-02-10T04:54:00.502235+0000","last_fullsized":"2021-02-10T04:54:00.502235+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:52.350706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.350706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"1119'4","reported_seq":"128","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:54:03.020235+0000","last_change":"2021-02-10T04:52:27.126670+0000","last_active":"2021-02-10T04:54:03.020235+0000","last_peered":"2021-02-10T04:54:03.020235+0000","last_clean":"2021-02-10T04:54:03.020235+0000","last_became_active":"2021-02-10T04:51:36.853647+0000","last_became_peered":"2021-02-10T04:51:36.853647+0000","last_unstale":"2021-02-10T04:54:03.020235+0000","last_undegraded":"2021-02-10T04:54:03.020235+0000","last_fullsized":"2021-02-10T04:54:03.020235+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'4","last_scrub_stamp":"2021-02-10T04:52:27.126599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:27.126599+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"1183","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.741530+0000","last_change":"2021-02-10T04:52:25.745660+0000","last_active":"2021-02-10T04:53:59.741530+0000","last_peered":"2021-02-10T04:53:59.741530+0000","last_clean":"2021-02-10T04:53:59.741530+0000","last_became_active":"2021-02-10T04:51:36.849503+0000","last_became_peered":"2021-02-10T04:51:36.849503+0000","last_unstale":"2021-02-10T04:53:59.741530+0000","last_undegraded":"2021-02-10T04:53:59.741530+0000","last_fullsized":"2021-02-10T04:53:59.741530+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:25.745577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.745577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"1004'1093","reported_seq":"6237","reported_epoch":"1196","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:59.743029+0000","last_change":"2021-02-10T04:52:32.090286+0000","last_active":"2021-02-10T04:53:59.743029+0000","last_peered":"2021-02-10T04:53:59.743029+0000","last_clean":"2021-02-10T04:44:07.998404+0000","last_became_active":"2021-02-10T04:52:32.090286+0000","last_became_peered":"2021-02-10T04:52:32.090286+0000","last_unstale":"2021-02-10T04:53:59.743029+0000","last_undegraded":"2021-02-10T04:52:32.075231+0000","last_fullsized":"2021-02-10T04:52:32.075115+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39434696,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":279,"num_read_kb":311058,"num_write":185,"num_write_kb":66790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":17}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"1131","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.743336+0000","last_change":"2021-02-10T04:53:48.235488+0000","last_active":"2021-02-10T04:53:59.743336+0000","last_peered":"2021-02-10T04:53:59.743336+0000","last_clean":"2021-02-10T04:53:59.743336+0000","last_became_active":"2021-02-10T04:51:36.841582+0000","last_became_peered":"2021-02-10T04:51:36.841582+0000","last_unstale":"2021-02-10T04:53:59.743336+0000","last_undegraded":"2021-02-10T04:53:59.743336+0000","last_fullsized":"2021-02-10T04:53:59.743336+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:12.389272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:12.389272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"1049'51","reported_seq":"871","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:54:00.503725+0000","last_change":"2021-02-10T04:53:27.560299+0000","last_active":"2021-02-10T04:54:00.503725+0000","last_peered":"2021-02-10T04:54:00.503725+0000","last_clean":"2021-02-10T04:54:00.503725+0000","last_became_active":"2021-02-10T04:53:27.559886+0000","last_became_peered":"2021-02-10T04:53:27.559886+0000","last_unstale":"2021-02-10T04:54:00.503725+0000","last_undegraded":"2021-02-10T04:54:00.503725+0000","last_fullsized":"2021-02-10T04:54:00.503725+0000","mapping_epoch":1129,"log_start":"1049'51","ondisk_log_start":"1049'51","created":55,"last_epoch_clean":1173,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'41","last_scrub_stamp":"2021-02-10T04:51:58.353660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:58.353660+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1472,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":139,"num_read_kb":106,"num_write":82,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"858","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:54:00.501899+0000","last_change":"2021-02-10T04:53:11.271557+0000","last_active":"2021-02-10T04:54:00.501899+0000","last_peered":"2021-02-10T04:54:00.501899+0000","last_clean":"2021-02-10T04:54:00.501899+0000","last_became_active":"2021-02-10T04:51:49.567883+0000","last_became_peered":"2021-02-10T04:51:49.567883+0000","last_unstale":"2021-02-10T04:54:00.501899+0000","last_undegraded":"2021-02-10T04:54:00.501899+0000","last_fullsized":"2021-02-10T04:54:00.501899+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:11.271459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.271459+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"1148'4","reported_seq":"1515","reported_epoch":"1196","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:03.020299+0000","last_change":"2021-02-10T04:52:32.088315+0000","last_active":"2021-02-10T04:54:03.020299+0000","last_peered":"2021-02-10T04:54:03.020299+0000","last_clean":"2021-02-10T04:52:30.062243+0000","last_became_active":"2021-02-10T04:52:32.088315+0000","last_became_peered":"2021-02-10T04:52:32.088315+0000","last_unstale":"2021-02-10T04:54:03.020299+0000","last_undegraded":"2021-02-10T04:52:32.069861+0000","last_fullsized":"2021-02-10T04:52:32.069742+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:51:47.855283+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.855283+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"1172","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.744868+0000","last_change":"2021-02-10T04:52:32.094688+0000","last_active":"2021-02-10T04:53:59.744868+0000","last_peered":"2021-02-10T04:53:59.744868+0000","last_clean":"2021-02-10T04:52:30.059376+0000","last_became_active":"2021-02-10T04:52:32.094688+0000","last_became_peered":"2021-02-10T04:52:32.094688+0000","last_unstale":"2021-02-10T04:53:59.744868+0000","last_undegraded":"2021-02-10T04:53:59.744868+0000","last_fullsized":"2021-02-10T04:52:32.079768+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:10.176878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:50:10.176878+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"1002'1090","reported_seq":"5390","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.741537+0000","last_change":"2021-02-10T04:52:26.030198+0000","last_active":"2021-02-10T04:53:59.741537+0000","last_peered":"2021-02-10T04:53:59.741537+0000","last_clean":"2021-02-10T04:53:59.741537+0000","last_became_active":"2021-02-10T04:52:26.028182+0000","last_became_peered":"2021-02-10T04:52:26.028182+0000","last_unstale":"2021-02-10T04:53:59.741537+0000","last_undegraded":"2021-02-10T04:53:59.741537+0000","last_fullsized":"2021-02-10T04:53:59.741537+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:51:08.220472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.220472+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24410646,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":245,"num_read_kb":276168,"num_write":175,"num_write_kb":64960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"1049'47","reported_seq":"1268","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.741347+0000","last_change":"2021-02-10T04:53:31.624862+0000","last_active":"2021-02-10T04:53:59.741347+0000","last_peered":"2021-02-10T04:53:59.741347+0000","last_clean":"2021-02-10T04:53:59.741347+0000","last_became_active":"2021-02-10T04:53:31.624318+0000","last_became_peered":"2021-02-10T04:53:31.624318+0000","last_unstale":"2021-02-10T04:53:59.741347+0000","last_undegraded":"2021-02-10T04:53:59.741347+0000","last_fullsized":"2021-02-10T04:53:59.741347+0000","mapping_epoch":55,"log_start":"1049'47","ondisk_log_start":"1049'47","created":55,"last_epoch_clean":1177,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'47","last_scrub_stamp":"2021-02-10T04:53:09.002368+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.002368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1672,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":142,"num_read_kb":108,"num_write":80,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":934,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"125","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.772628+0000","last_change":"2021-02-10T04:53:46.729988+0000","last_active":"2021-02-10T04:53:59.772628+0000","last_peered":"2021-02-10T04:53:59.772628+0000","last_clean":"2021-02-10T04:28:18.347728+0000","last_became_active":"2021-02-10T04:53:46.729988+0000","last_became_peered":"2021-02-10T04:53:46.729988+0000","last_unstale":"2021-02-10T04:53:59.772628+0000","last_undegraded":"2021-02-10T04:53:59.772628+0000","last_fullsized":"2021-02-10T04:53:46.711213+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:08.358300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:50:08.358300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"1173","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.745127+0000","last_change":"2021-02-10T04:51:57.436961+0000","last_active":"2021-02-10T04:53:59.745127+0000","last_peered":"2021-02-10T04:53:59.745127+0000","last_clean":"2021-02-10T04:53:59.745127+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:53:59.745127+0000","last_undegraded":"2021-02-10T04:53:59.745127+0000","last_fullsized":"2021-02-10T04:53:59.745127+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.436862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.436862+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"1128","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.742233+0000","last_change":"2021-02-10T04:53:02.112612+0000","last_active":"2021-02-10T04:53:59.742233+0000","last_peered":"2021-02-10T04:53:59.742233+0000","last_clean":"2021-02-10T04:53:59.742233+0000","last_became_active":"2021-02-10T04:50:59.545501+0000","last_became_peered":"2021-02-10T04:50:59.545501+0000","last_unstale":"2021-02-10T04:53:59.742233+0000","last_undegraded":"2021-02-10T04:53:59.742233+0000","last_fullsized":"2021-02-10T04:53:59.742233+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:02.112533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:02.112533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"1001'1165","reported_seq":"4360","reported_epoch":"1196","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:59.742023+0000","last_change":"2021-02-10T04:52:32.082334+0000","last_active":"2021-02-10T04:53:59.742023+0000","last_peered":"2021-02-10T04:53:59.742023+0000","last_clean":"2021-02-10T04:52:30.068280+0000","last_became_active":"2021-02-10T04:52:32.082334+0000","last_became_peered":"2021-02-10T04:52:32.082334+0000","last_unstale":"2021-02-10T04:53:59.742023+0000","last_undegraded":"2021-02-10T04:52:32.079317+0000","last_fullsized":"2021-02-10T04:52:32.078663+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29601845,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":286,"num_read_kb":299529,"num_write":200,"num_write_kb":89130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":12}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"1113'5","reported_seq":"1410","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:54:03.020279+0000","last_change":"2021-02-10T04:52:34.047474+0000","last_active":"2021-02-10T04:54:03.020279+0000","last_peered":"2021-02-10T04:54:03.020279+0000","last_clean":"2021-02-10T04:54:03.020279+0000","last_became_active":"2021-02-10T04:50:59.569578+0000","last_became_peered":"2021-02-10T04:50:59.569578+0000","last_unstale":"2021-02-10T04:54:03.020279+0000","last_undegraded":"2021-02-10T04:54:03.020279+0000","last_fullsized":"2021-02-10T04:54:03.020279+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1113'5","last_scrub_stamp":"2021-02-10T04:52:34.047358+0000","last_deep_scrub":"158'3","last_deep_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_clean_scrub_stamp":"2021-02-10T04:52:34.047358+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"1049'54","reported_seq":"1259","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.740276+0000","last_change":"2021-02-10T04:51:48.872937+0000","last_active":"2021-02-10T04:53:59.740276+0000","last_peered":"2021-02-10T04:53:59.740276+0000","last_clean":"2021-02-10T04:53:59.740276+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:53:59.740276+0000","last_undegraded":"2021-02-10T04:53:59.740276+0000","last_fullsized":"2021-02-10T04:53:59.740276+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:51:48.872839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.872839+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":93,"num_read_kb":73,"num_write":54,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"1247","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.742062+0000","last_change":"2021-02-10T04:53:05.984747+0000","last_active":"2021-02-10T04:53:59.742062+0000","last_peered":"2021-02-10T04:53:59.742062+0000","last_clean":"2021-02-10T04:53:59.742062+0000","last_became_active":"2021-02-10T04:51:36.869931+0000","last_became_peered":"2021-02-10T04:51:36.869931+0000","last_unstale":"2021-02-10T04:53:59.742062+0000","last_undegraded":"2021-02-10T04:53:59.742062+0000","last_fullsized":"2021-02-10T04:53:59.742062+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:05.984659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:05.984659+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"1051","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.773429+0000","last_change":"2021-02-10T04:53:46.731709+0000","last_active":"2021-02-10T04:53:59.773429+0000","last_peered":"2021-02-10T04:53:59.773429+0000","last_clean":"2021-02-10T04:53:59.773429+0000","last_became_active":"2021-02-10T04:53:46.731340+0000","last_became_peered":"2021-02-10T04:53:46.731340+0000","last_unstale":"2021-02-10T04:53:59.773429+0000","last_undegraded":"2021-02-10T04:53:59.773429+0000","last_fullsized":"2021-02-10T04:53:59.773429+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:09.005421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.005421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"998'1009","reported_seq":"4687","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.741953+0000","last_change":"2021-02-10T04:52:26.037227+0000","last_active":"2021-02-10T04:53:59.741953+0000","last_peered":"2021-02-10T04:53:59.741953+0000","last_clean":"2021-02-10T04:53:59.741953+0000","last_became_active":"2021-02-10T04:52:26.036723+0000","last_became_peered":"2021-02-10T04:52:26.036723+0000","last_unstale":"2021-02-10T04:53:59.741953+0000","last_undegraded":"2021-02-10T04:53:59.741953+0000","last_fullsized":"2021-02-10T04:53:59.741953+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12355206,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":119,"num_read_kb":124701,"num_write":77,"num_write_kb":25052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4145029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"1067","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.742746+0000","last_change":"2021-02-10T04:53:33.305896+0000","last_active":"2021-02-10T04:53:59.742746+0000","last_peered":"2021-02-10T04:53:59.742746+0000","last_clean":"2021-02-10T04:53:59.742746+0000","last_became_active":"2021-02-10T04:50:59.570854+0000","last_became_peered":"2021-02-10T04:50:59.570854+0000","last_unstale":"2021-02-10T04:53:59.742746+0000","last_undegraded":"2021-02-10T04:53:59.742746+0000","last_fullsized":"2021-02-10T04:53:59.742746+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:33.305827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:33.305827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"1182","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.740401+0000","last_change":"2021-02-10T04:51:50.895471+0000","last_active":"2021-02-10T04:53:59.740401+0000","last_peered":"2021-02-10T04:53:59.740401+0000","last_clean":"2021-02-10T04:53:59.740401+0000","last_became_active":"2021-02-10T04:50:59.527553+0000","last_became_peered":"2021-02-10T04:50:59.527553+0000","last_unstale":"2021-02-10T04:53:59.740401+0000","last_undegraded":"2021-02-10T04:53:59.740401+0000","last_fullsized":"2021-02-10T04:53:59.740401+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.895370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.895370+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"1049'72","reported_seq":"1285","reported_epoch":"1196","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:59.741196+0000","last_change":"2021-02-10T04:52:32.090145+0000","last_active":"2021-02-10T04:53:59.741196+0000","last_peered":"2021-02-10T04:53:59.741196+0000","last_clean":"2021-02-10T04:52:30.063735+0000","last_became_active":"2021-02-10T04:52:32.090145+0000","last_became_peered":"2021-02-10T04:52:32.090145+0000","last_unstale":"2021-02-10T04:53:59.741196+0000","last_undegraded":"2021-02-10T04:52:32.073169+0000","last_fullsized":"2021-02-10T04:52:32.073062+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'72","last_scrub_stamp":"2021-02-10T04:46:30.515026+0000","last_deep_scrub":"152'48","last_deep_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_clean_scrub_stamp":"2021-02-10T04:46:30.515026+0000","log_size":72,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":105,"num_read_kb":83,"num_write":60,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":14}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"436","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:54:00.503674+0000","last_change":"2021-02-10T04:53:48.191514+0000","last_active":"2021-02-10T04:54:00.503674+0000","last_peered":"2021-02-10T04:54:00.503674+0000","last_clean":"2021-02-10T04:54:00.503674+0000","last_became_active":"2021-02-10T04:51:49.503801+0000","last_became_peered":"2021-02-10T04:51:49.503801+0000","last_unstale":"2021-02-10T04:54:00.503674+0000","last_undegraded":"2021-02-10T04:54:00.503674+0000","last_fullsized":"2021-02-10T04:54:00.503674+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:51:48.274960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.274960+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"1207","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.740642+0000","last_change":"2021-02-10T04:53:13.099288+0000","last_active":"2021-02-10T04:53:59.740642+0000","last_peered":"2021-02-10T04:53:59.740642+0000","last_clean":"2021-02-10T04:53:59.740642+0000","last_became_active":"2021-02-10T04:51:36.848614+0000","last_became_peered":"2021-02-10T04:51:36.848614+0000","last_unstale":"2021-02-10T04:53:59.740642+0000","last_undegraded":"2021-02-10T04:53:59.740642+0000","last_fullsized":"2021-02-10T04:53:59.740642+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:13.099228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:13.099228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"1002'1657","reported_seq":"5257","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.741993+0000","last_change":"2021-02-10T04:52:26.036806+0000","last_active":"2021-02-10T04:53:59.741993+0000","last_peered":"2021-02-10T04:53:59.741993+0000","last_clean":"2021-02-10T04:53:59.741993+0000","last_became_active":"2021-02-10T04:52:26.032363+0000","last_became_peered":"2021-02-10T04:52:26.032363+0000","last_unstale":"2021-02-10T04:53:59.741993+0000","last_undegraded":"2021-02-10T04:53:59.741993+0000","last_fullsized":"2021-02-10T04:53:59.741993+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21396533,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":212,"num_read_kb":218836,"num_write":159,"num_write_kb":59398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":22326014,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"1126","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.740962+0000","last_change":"2021-02-10T04:51:51.911945+0000","last_active":"2021-02-10T04:53:59.740962+0000","last_peered":"2021-02-10T04:53:59.740962+0000","last_clean":"2021-02-10T04:53:59.740962+0000","last_became_active":"2021-02-10T04:51:36.850015+0000","last_became_peered":"2021-02-10T04:51:36.850015+0000","last_unstale":"2021-02-10T04:53:59.740962+0000","last_undegraded":"2021-02-10T04:53:59.740962+0000","last_fullsized":"2021-02-10T04:53:59.740962+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.911851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.911851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"1245","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.741439+0000","last_change":"2021-02-10T04:52:32.089163+0000","last_active":"2021-02-10T04:53:59.741439+0000","last_peered":"2021-02-10T04:53:59.741439+0000","last_clean":"2021-02-10T04:52:30.062078+0000","last_became_active":"2021-02-10T04:52:32.089163+0000","last_became_peered":"2021-02-10T04:52:32.089163+0000","last_unstale":"2021-02-10T04:53:59.741439+0000","last_undegraded":"2021-02-10T04:53:59.741439+0000","last_fullsized":"2021-02-10T04:52:32.072413+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:58.142526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:58.142526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"1135'58","reported_seq":"1251","reported_epoch":"1196","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:59.744241+0000","last_change":"2021-02-10T04:52:32.074386+0000","last_active":"2021-02-10T04:53:59.744241+0000","last_peered":"2021-02-10T04:53:59.744241+0000","last_clean":"2021-02-10T04:52:01.474364+0000","last_became_active":"2021-02-10T04:52:32.074386+0000","last_became_peered":"2021-02-10T04:52:32.074386+0000","last_unstale":"2021-02-10T04:53:59.744241+0000","last_undegraded":"2021-02-10T04:52:32.071237+0000","last_fullsized":"2021-02-10T04:52:32.070967+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1135'58","last_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_deep_scrub":"1135'58","last_deep_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.483067+0000","log_size":58,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":87,"num_read_kb":68,"num_write":50,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":11}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"123","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.774496+0000","last_change":"2021-02-10T04:53:46.733605+0000","last_active":"2021-02-10T04:53:59.774496+0000","last_peered":"2021-02-10T04:53:59.774496+0000","last_clean":"2021-02-10T04:29:05.137445+0000","last_became_active":"2021-02-10T04:53:46.733605+0000","last_became_peered":"2021-02-10T04:53:46.733605+0000","last_unstale":"2021-02-10T04:53:59.774496+0000","last_undegraded":"2021-02-10T04:53:59.774496+0000","last_fullsized":"2021-02-10T04:53:46.710695+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:50:46.434629+0000","last_deep_scrub":"53'2","last_deep_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_clean_scrub_stamp":"2021-02-10T04:50:46.434629+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"875","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:54:00.502312+0000","last_change":"2021-02-10T04:52:38.278980+0000","last_active":"2021-02-10T04:54:00.502312+0000","last_peered":"2021-02-10T04:54:00.502312+0000","last_clean":"2021-02-10T04:54:00.502312+0000","last_became_active":"2021-02-10T04:51:49.496237+0000","last_became_peered":"2021-02-10T04:51:49.496237+0000","last_unstale":"2021-02-10T04:54:00.502312+0000","last_undegraded":"2021-02-10T04:54:00.502312+0000","last_fullsized":"2021-02-10T04:54:00.502312+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:38.278910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:52:38.278910+0000","last_clean_scrub_stamp":"2021-02-10T04:52:38.278910+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"1000'1071","reported_seq":"4747","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.743836+0000","last_change":"2021-02-10T04:53:34.407875+0000","last_active":"2021-02-10T04:53:59.743836+0000","last_peered":"2021-02-10T04:53:59.743836+0000","last_clean":"2021-02-10T04:53:59.743836+0000","last_became_active":"2021-02-10T04:52:26.031050+0000","last_became_peered":"2021-02-10T04:52:26.031050+0000","last_unstale":"2021-02-10T04:53:59.743836+0000","last_undegraded":"2021-02-10T04:53:59.743836+0000","last_fullsized":"2021-02-10T04:53:59.743836+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:34.407784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.407784+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30173843,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"1211","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.741569+0000","last_change":"2021-02-10T04:52:32.081955+0000","last_active":"2021-02-10T04:53:59.741569+0000","last_peered":"2021-02-10T04:53:59.741569+0000","last_clean":"2021-02-10T04:52:30.068208+0000","last_became_active":"2021-02-10T04:52:32.081955+0000","last_became_peered":"2021-02-10T04:52:32.081955+0000","last_unstale":"2021-02-10T04:53:59.741569+0000","last_undegraded":"2021-02-10T04:53:59.741569+0000","last_fullsized":"2021-02-10T04:52:32.077723+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_clean_scrub_stamp":"2021-02-10T04:51:00.195518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"1174","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.745208+0000","last_change":"2021-02-10T04:52:32.097628+0000","last_active":"2021-02-10T04:53:59.745208+0000","last_peered":"2021-02-10T04:53:59.745208+0000","last_clean":"2021-02-10T04:52:30.061275+0000","last_became_active":"2021-02-10T04:52:32.097628+0000","last_became_peered":"2021-02-10T04:52:32.097628+0000","last_unstale":"2021-02-10T04:53:59.745208+0000","last_undegraded":"2021-02-10T04:53:59.745208+0000","last_fullsized":"2021-02-10T04:52:32.086329+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:55.473347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:50:55.473347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"1049'40","reported_seq":"1243","reported_epoch":"1196","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:59.742135+0000","last_change":"2021-02-10T04:52:32.094738+0000","last_active":"2021-02-10T04:53:59.742135+0000","last_peered":"2021-02-10T04:53:59.742135+0000","last_clean":"2021-02-10T04:51:27.993802+0000","last_became_active":"2021-02-10T04:52:32.094738+0000","last_became_peered":"2021-02-10T04:52:32.094738+0000","last_unstale":"2021-02-10T04:53:59.742135+0000","last_undegraded":"2021-02-10T04:52:32.085496+0000","last_fullsized":"2021-02-10T04:52:32.085393+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'40","last_scrub_stamp":"2021-02-10T04:51:47.281929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.281929+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":52,"num_write":40,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":10}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"349","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.772688+0000","last_change":"2021-02-10T04:53:46.731083+0000","last_active":"2021-02-10T04:53:59.772688+0000","last_peered":"2021-02-10T04:53:59.772688+0000","last_clean":"2021-02-10T04:52:30.068091+0000","last_became_active":"2021-02-10T04:53:46.731083+0000","last_became_peered":"2021-02-10T04:53:46.731083+0000","last_unstale":"2021-02-10T04:53:59.772688+0000","last_undegraded":"2021-02-10T04:53:59.772688+0000","last_fullsized":"2021-02-10T04:53:46.711548+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:50.399350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:50.399350+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"1149","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.740384+0000","last_change":"2021-02-10T04:52:32.081196+0000","last_active":"2021-02-10T04:53:59.740384+0000","last_peered":"2021-02-10T04:53:59.740384+0000","last_clean":"2021-02-10T04:49:56.935460+0000","last_became_active":"2021-02-10T04:52:32.081196+0000","last_became_peered":"2021-02-10T04:52:32.081196+0000","last_unstale":"2021-02-10T04:53:59.740384+0000","last_undegraded":"2021-02-10T04:53:59.740384+0000","last_fullsized":"2021-02-10T04:52:32.068851+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"860","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:54:00.503437+0000","last_change":"2021-02-10T04:52:31.273037+0000","last_active":"2021-02-10T04:54:00.503437+0000","last_peered":"2021-02-10T04:54:00.503437+0000","last_clean":"2021-02-10T04:54:00.503437+0000","last_became_active":"2021-02-10T04:51:49.501405+0000","last_became_peered":"2021-02-10T04:51:49.501405+0000","last_unstale":"2021-02-10T04:54:00.503437+0000","last_undegraded":"2021-02-10T04:54:00.503437+0000","last_fullsized":"2021-02-10T04:54:00.503437+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:31.272971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:17.598113+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.272971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"1000'446","reported_seq":"1906","reported_epoch":"1196","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:59.774044+0000","last_change":"2021-02-10T04:53:46.723625+0000","last_active":"2021-02-10T04:53:59.774044+0000","last_peered":"2021-02-10T04:53:59.774044+0000","last_clean":"2021-02-10T04:43:52.634592+0000","last_became_active":"2021-02-10T04:53:46.723625+0000","last_became_peered":"2021-02-10T04:53:46.723625+0000","last_unstale":"2021-02-10T04:53:59.774044+0000","last_undegraded":"2021-02-10T04:53:46.707033+0000","last_fullsized":"2021-02-10T04:53:46.706921+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:51:45.964029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.964029+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22255852,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68435,"num_write":72,"num_write_kb":24345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":403693,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":9}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"351","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:54:00.502197+0000","last_change":"2021-02-10T04:53:03.228260+0000","last_active":"2021-02-10T04:54:00.502197+0000","last_peered":"2021-02-10T04:54:00.502197+0000","last_clean":"2021-02-10T04:54:00.502197+0000","last_became_active":"2021-02-10T04:51:49.499252+0000","last_became_peered":"2021-02-10T04:51:49.499252+0000","last_unstale":"2021-02-10T04:54:00.502197+0000","last_undegraded":"2021-02-10T04:54:00.502197+0000","last_fullsized":"2021-02-10T04:54:00.502197+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:03.228182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.228182+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"102","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.774050+0000","last_change":"2021-02-10T04:53:46.727384+0000","last_active":"2021-02-10T04:53:59.774050+0000","last_peered":"2021-02-10T04:53:59.774050+0000","last_clean":"2021-02-10T04:53:59.774050+0000","last_became_active":"2021-02-10T04:53:46.726771+0000","last_became_peered":"2021-02-10T04:53:46.726771+0000","last_unstale":"2021-02-10T04:53:59.774050+0000","last_undegraded":"2021-02-10T04:53:59.774050+0000","last_fullsized":"2021-02-10T04:53:59.774050+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:33.948348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:33.948348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"1049'39","reported_seq":"1235","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.741836+0000","last_change":"2021-02-10T04:51:47.322426+0000","last_active":"2021-02-10T04:53:59.741836+0000","last_peered":"2021-02-10T04:53:59.741836+0000","last_clean":"2021-02-10T04:53:59.741836+0000","last_became_active":"2021-02-10T04:25:49.002471+0000","last_became_peered":"2021-02-10T04:25:49.002471+0000","last_unstale":"2021-02-10T04:53:59.741836+0000","last_undegraded":"2021-02-10T04:53:59.741836+0000","last_fullsized":"2021-02-10T04:53:59.741836+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'39","last_scrub_stamp":"2021-02-10T04:51:47.322331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.322331+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":51,"num_write":39,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"121","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.772755+0000","last_change":"2021-02-10T04:53:46.714141+0000","last_active":"2021-02-10T04:53:59.772755+0000","last_peered":"2021-02-10T04:53:59.772755+0000","last_clean":"2021-02-10T04:28:45.095539+0000","last_became_active":"2021-02-10T04:53:46.714141+0000","last_became_peered":"2021-02-10T04:53:46.714141+0000","last_unstale":"2021-02-10T04:53:59.772755+0000","last_undegraded":"2021-02-10T04:53:59.772755+0000","last_fullsized":"2021-02-10T04:53:46.700190+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_clean_scrub_stamp":"2021-02-10T04:50:05.424262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"1367","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.744140+0000","last_change":"2021-02-10T04:52:37.324122+0000","last_active":"2021-02-10T04:53:59.744140+0000","last_peered":"2021-02-10T04:53:59.744140+0000","last_clean":"2021-02-10T04:53:59.744140+0000","last_became_active":"2021-02-10T04:51:36.843553+0000","last_became_peered":"2021-02-10T04:51:36.843553+0000","last_unstale":"2021-02-10T04:53:59.744140+0000","last_undegraded":"2021-02-10T04:53:59.744140+0000","last_fullsized":"2021-02-10T04:53:59.744140+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:37.324040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:37.324040+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"1164","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.742026+0000","last_change":"2021-02-10T04:51:57.769992+0000","last_active":"2021-02-10T04:53:59.742026+0000","last_peered":"2021-02-10T04:53:59.742026+0000","last_clean":"2021-02-10T04:53:59.742026+0000","last_became_active":"2021-02-10T04:50:59.587343+0000","last_became_peered":"2021-02-10T04:50:59.587343+0000","last_unstale":"2021-02-10T04:53:59.742026+0000","last_undegraded":"2021-02-10T04:53:59.742026+0000","last_fullsized":"2021-02-10T04:53:59.742026+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"1195","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.745272+0000","last_change":"2021-02-10T04:52:31.403904+0000","last_active":"2021-02-10T04:53:59.745272+0000","last_peered":"2021-02-10T04:53:59.745272+0000","last_clean":"2021-02-10T04:53:59.745272+0000","last_became_active":"2021-02-10T04:50:59.538009+0000","last_became_peered":"2021-02-10T04:50:59.538009+0000","last_unstale":"2021-02-10T04:53:59.745272+0000","last_undegraded":"2021-02-10T04:53:59.745272+0000","last_fullsized":"2021-02-10T04:53:59.745272+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:31.403824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.403824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"1000'490","reported_seq":"3461","reported_epoch":"1196","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:59.774800+0000","last_change":"2021-02-10T04:53:46.711617+0000","last_active":"2021-02-10T04:53:59.774800+0000","last_peered":"2021-02-10T04:53:59.774800+0000","last_clean":"2021-02-10T04:43:50.630768+0000","last_became_active":"2021-02-10T04:53:46.711617+0000","last_became_peered":"2021-02-10T04:53:46.711617+0000","last_unstale":"2021-02-10T04:53:59.774800+0000","last_undegraded":"2021-02-10T04:53:46.702746+0000","last_fullsized":"2021-02-10T04:53:46.702635+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1040,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:47:02.875920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:02.875920+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19371627,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":141,"num_read_kb":148568,"num_write":85,"num_write_kb":31732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":7}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"93","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.742542+0000","last_change":"2021-02-10T04:53:01.972991+0000","last_active":"2021-02-10T04:53:59.742542+0000","last_peered":"2021-02-10T04:53:59.742542+0000","last_clean":"2021-02-10T04:53:59.742542+0000","last_became_active":"2021-02-10T04:51:36.863976+0000","last_became_peered":"2021-02-10T04:51:36.863976+0000","last_unstale":"2021-02-10T04:53:59.742542+0000","last_undegraded":"2021-02-10T04:53:59.742542+0000","last_fullsized":"2021-02-10T04:53:59.742542+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:01.972913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.972913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"1049'37","reported_seq":"875","reported_epoch":"1196","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:00.446725+0000","last_change":"2021-02-10T04:52:32.095888+0000","last_active":"2021-02-10T04:54:00.446725+0000","last_peered":"2021-02-10T04:54:00.446725+0000","last_clean":"2021-02-10T04:52:30.062113+0000","last_became_active":"2021-02-10T04:52:32.095888+0000","last_became_peered":"2021-02-10T04:52:32.095888+0000","last_unstale":"2021-02-10T04:54:00.446725+0000","last_undegraded":"2021-02-10T04:52:32.076227+0000","last_fullsized":"2021-02-10T04:52:32.076110+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:56.431187+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.431187+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":59,"num_read_kb":43,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"94","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.740236+0000","last_change":"2021-02-10T04:52:32.089886+0000","last_active":"2021-02-10T04:53:59.740236+0000","last_peered":"2021-02-10T04:53:59.740236+0000","last_clean":"2021-02-10T04:52:30.064491+0000","last_became_active":"2021-02-10T04:52:32.089886+0000","last_became_peered":"2021-02-10T04:52:32.089886+0000","last_unstale":"2021-02-10T04:53:59.740236+0000","last_undegraded":"2021-02-10T04:53:59.740236+0000","last_fullsized":"2021-02-10T04:52:32.078850+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:04.591695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:04.591695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"986","reported_epoch":"1197","state":"active+clean","last_fresh":"2021-02-10T04:54:06.657979+0000","last_change":"2021-02-10T04:52:25.358146+0000","last_active":"2021-02-10T04:54:06.657979+0000","last_peered":"2021-02-10T04:54:06.657979+0000","last_clean":"2021-02-10T04:54:06.657979+0000","last_became_active":"2021-02-10T04:51:49.496092+0000","last_became_peered":"2021-02-10T04:51:49.496092+0000","last_unstale":"2021-02-10T04:54:06.657979+0000","last_undegraded":"2021-02-10T04:54:06.657979+0000","last_fullsized":"2021-02-10T04:54:06.657979+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":570,"num_read_kb":565,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"1041","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.774381+0000","last_change":"2021-02-10T04:53:46.729138+0000","last_active":"2021-02-10T04:53:59.774381+0000","last_peered":"2021-02-10T04:53:59.774381+0000","last_clean":"2021-02-10T04:53:59.774381+0000","last_became_active":"2021-02-10T04:53:46.728845+0000","last_became_peered":"2021-02-10T04:53:46.728845+0000","last_unstale":"2021-02-10T04:53:59.774381+0000","last_undegraded":"2021-02-10T04:53:59.774381+0000","last_fullsized":"2021-02-10T04:53:59.774381+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:40.066354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:52:40.066354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"1000'620","reported_seq":"4203","reported_epoch":"1196","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:59.741804+0000","last_change":"2021-02-10T04:52:32.082414+0000","last_active":"2021-02-10T04:53:59.741804+0000","last_peered":"2021-02-10T04:53:59.741804+0000","last_clean":"2021-02-10T04:43:51.960482+0000","last_became_active":"2021-02-10T04:52:32.082414+0000","last_became_peered":"2021-02-10T04:52:32.082414+0000","last_unstale":"2021-02-10T04:53:59.741804+0000","last_undegraded":"2021-02-10T04:52:32.068324+0000","last_fullsized":"2021-02-10T04:52:32.068200+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:51:54.444287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:54.444287+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14265195,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":144,"num_read_kb":128482,"num_write":129,"num_write_kb":53443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"151","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.740277+0000","last_change":"2021-02-10T04:52:32.076404+0000","last_active":"2021-02-10T04:53:59.740277+0000","last_peered":"2021-02-10T04:53:59.740277+0000","last_clean":"2021-02-10T04:28:47.053193+0000","last_became_active":"2021-02-10T04:52:32.076404+0000","last_became_peered":"2021-02-10T04:52:32.076404+0000","last_unstale":"2021-02-10T04:53:59.740277+0000","last_undegraded":"2021-02-10T04:53:59.740277+0000","last_fullsized":"2021-02-10T04:52:32.068279+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:14.955544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:47:14.955544+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"58","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:54:00.446438+0000","last_change":"2021-02-10T04:52:32.084350+0000","last_active":"2021-02-10T04:54:00.446438+0000","last_peered":"2021-02-10T04:54:00.446438+0000","last_clean":"2021-02-10T04:25:49.999907+0000","last_became_active":"2021-02-10T04:52:32.084350+0000","last_became_peered":"2021-02-10T04:52:32.084350+0000","last_unstale":"2021-02-10T04:54:00.446438+0000","last_undegraded":"2021-02-10T04:54:00.446438+0000","last_fullsized":"2021-02-10T04:52:32.073004+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:27.908221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:47:27.908221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"1049'55","reported_seq":"831","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:54:00.503370+0000","last_change":"2021-02-10T04:52:30.277009+0000","last_active":"2021-02-10T04:54:00.503370+0000","last_peered":"2021-02-10T04:54:00.503370+0000","last_clean":"2021-02-10T04:54:00.503370+0000","last_became_active":"2021-02-10T04:51:49.501622+0000","last_became_peered":"2021-02-10T04:51:49.501622+0000","last_unstale":"2021-02-10T04:54:00.503370+0000","last_undegraded":"2021-02-10T04:54:00.503370+0000","last_fullsized":"2021-02-10T04:54:00.503370+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'55","last_scrub_stamp":"2021-02-10T04:52:30.276938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.276938+0000","log_size":55,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":98,"num_read_kb":74,"num_write":58,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"1191","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.743967+0000","last_change":"2021-02-10T04:51:59.420149+0000","last_active":"2021-02-10T04:53:59.743967+0000","last_peered":"2021-02-10T04:53:59.743967+0000","last_clean":"2021-02-10T04:53:59.743967+0000","last_became_active":"2021-02-10T04:28:21.125541+0000","last_became_peered":"2021-02-10T04:28:21.125541+0000","last_unstale":"2021-02-10T04:53:59.743967+0000","last_undegraded":"2021-02-10T04:53:59.743967+0000","last_fullsized":"2021-02-10T04:53:59.743967+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.420086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.420086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"1157","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.742476+0000","last_change":"2021-02-10T04:53:12.187225+0000","last_active":"2021-02-10T04:53:59.742476+0000","last_peered":"2021-02-10T04:53:59.742476+0000","last_clean":"2021-02-10T04:53:59.742476+0000","last_became_active":"2021-02-10T04:50:59.555689+0000","last_became_peered":"2021-02-10T04:50:59.555689+0000","last_unstale":"2021-02-10T04:53:59.742476+0000","last_undegraded":"2021-02-10T04:53:59.742476+0000","last_fullsized":"2021-02-10T04:53:59.742476+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.187154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.187154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"1002'2042","reported_seq":"9419","reported_epoch":"1196","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:59.741929+0000","last_change":"2021-02-10T04:52:32.083034+0000","last_active":"2021-02-10T04:53:59.741929+0000","last_peered":"2021-02-10T04:53:59.741929+0000","last_clean":"2021-02-10T04:52:30.068473+0000","last_became_active":"2021-02-10T04:52:32.083034+0000","last_became_peered":"2021-02-10T04:52:32.083034+0000","last_unstale":"2021-02-10T04:53:59.741929+0000","last_undegraded":"2021-02-10T04:52:32.077334+0000","last_fullsized":"2021-02-10T04:52:32.077220+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:46:57.054230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:57.054230+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":85678898,"num_objects":35,"num_object_clones":29,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":35}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"1133","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.743417+0000","last_change":"2021-02-10T04:52:32.094777+0000","last_active":"2021-02-10T04:53:59.743417+0000","last_peered":"2021-02-10T04:53:59.743417+0000","last_clean":"2021-02-10T04:52:30.061448+0000","last_became_active":"2021-02-10T04:52:32.094777+0000","last_became_peered":"2021-02-10T04:52:32.094777+0000","last_unstale":"2021-02-10T04:53:59.743417+0000","last_undegraded":"2021-02-10T04:53:59.743417+0000","last_fullsized":"2021-02-10T04:52:32.069238+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:52.341527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:50:52.341527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"1148'3","reported_seq":"86","reported_epoch":"1196","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:03.020765+0000","last_change":"2021-02-10T04:52:32.089249+0000","last_active":"2021-02-10T04:54:03.020765+0000","last_peered":"2021-02-10T04:54:03.020765+0000","last_clean":"2021-02-10T04:25:51.042416+0000","last_became_active":"2021-02-10T04:52:32.089249+0000","last_became_peered":"2021-02-10T04:52:32.089249+0000","last_unstale":"2021-02-10T04:54:03.020765+0000","last_undegraded":"2021-02-10T04:52:32.073980+0000","last_fullsized":"2021-02-10T04:52:32.073877+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:54.466259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:54.466259+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"1049'54","reported_seq":"880","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:54:00.502402+0000","last_change":"2021-02-10T04:53:01.304693+0000","last_active":"2021-02-10T04:54:00.502402+0000","last_peered":"2021-02-10T04:54:00.502402+0000","last_clean":"2021-02-10T04:54:00.502402+0000","last_became_active":"2021-02-10T04:51:49.498066+0000","last_became_peered":"2021-02-10T04:51:49.498066+0000","last_unstale":"2021-02-10T04:54:00.502402+0000","last_undegraded":"2021-02-10T04:54:00.502402+0000","last_fullsized":"2021-02-10T04:54:00.502402+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:53:01.304626+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.304626+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":74,"num_read_kb":56,"num_write":42,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"447","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:54:00.446561+0000","last_change":"2021-02-10T04:52:32.091144+0000","last_active":"2021-02-10T04:54:00.446561+0000","last_peered":"2021-02-10T04:54:00.446561+0000","last_clean":"2021-02-10T04:52:30.060819+0000","last_became_active":"2021-02-10T04:52:32.091144+0000","last_became_peered":"2021-02-10T04:52:32.091144+0000","last_unstale":"2021-02-10T04:54:00.446561+0000","last_undegraded":"2021-02-10T04:54:00.446561+0000","last_fullsized":"2021-02-10T04:52:32.079969+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:49.447961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:49.447961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"1178","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.740706+0000","last_change":"2021-02-10T04:52:32.088766+0000","last_active":"2021-02-10T04:53:59.740706+0000","last_peered":"2021-02-10T04:53:59.740706+0000","last_clean":"2021-02-10T04:51:27.993073+0000","last_became_active":"2021-02-10T04:52:32.088766+0000","last_became_peered":"2021-02-10T04:52:32.088766+0000","last_unstale":"2021-02-10T04:53:59.740706+0000","last_undegraded":"2021-02-10T04:53:59.740706+0000","last_fullsized":"2021-02-10T04:52:32.071883+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:07.914061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:07.914061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"293'77","reported_seq":"2979","reported_epoch":"1196","state":"active+clean+scrubbing","last_fresh":"2021-02-10T04:53:59.740056+0000","last_change":"2021-02-10T04:53:43.102301+0000","last_active":"2021-02-10T04:53:59.740056+0000","last_peered":"2021-02-10T04:53:59.740056+0000","last_clean":"2021-02-10T04:53:59.740056+0000","last_became_active":"2021-02-10T04:52:26.024812+0000","last_became_peered":"2021-02-10T04:52:26.024812+0000","last_unstale":"2021-02-10T04:53:59.740056+0000","last_undegraded":"2021-02-10T04:53:59.740056+0000","last_fullsized":"2021-02-10T04:53:59.740056+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1886585,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":18587,"num_write":9,"num_write_kb":3315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"1210","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.741866+0000","last_change":"2021-02-10T04:53:07.025068+0000","last_active":"2021-02-10T04:53:59.741866+0000","last_peered":"2021-02-10T04:53:59.741866+0000","last_clean":"2021-02-10T04:53:59.741866+0000","last_became_active":"2021-02-10T04:50:59.589417+0000","last_became_peered":"2021-02-10T04:50:59.589417+0000","last_unstale":"2021-02-10T04:53:59.741866+0000","last_undegraded":"2021-02-10T04:53:59.741866+0000","last_fullsized":"2021-02-10T04:53:59.741866+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.024980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.024980+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"94","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.739514+0000","last_change":"2021-02-10T04:52:32.082945+0000","last_active":"2021-02-10T04:53:59.739514+0000","last_peered":"2021-02-10T04:53:59.739514+0000","last_clean":"2021-02-10T04:52:30.056402+0000","last_became_active":"2021-02-10T04:52:32.082945+0000","last_became_peered":"2021-02-10T04:52:32.082945+0000","last_unstale":"2021-02-10T04:53:59.739514+0000","last_undegraded":"2021-02-10T04:53:59.739514+0000","last_fullsized":"2021-02-10T04:52:32.077773+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.609500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.609500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"1048'39","reported_seq":"1292","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.740316+0000","last_change":"2021-02-10T04:53:38.085663+0000","last_active":"2021-02-10T04:53:59.740316+0000","last_peered":"2021-02-10T04:53:59.740316+0000","last_clean":"2021-02-10T04:53:59.740316+0000","last_became_active":"2021-02-10T04:39:02.560739+0000","last_became_peered":"2021-02-10T04:39:02.560739+0000","last_unstale":"2021-02-10T04:53:59.740316+0000","last_undegraded":"2021-02-10T04:53:59.740316+0000","last_fullsized":"2021-02-10T04:53:59.740316+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'39","last_scrub_stamp":"2021-02-10T04:53:38.085565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.085565+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":76,"num_read_kb":59,"num_write":41,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"1302","reported_epoch":"1196","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:59.741956+0000","last_change":"2021-02-10T04:52:32.084961+0000","last_active":"2021-02-10T04:53:59.741956+0000","last_peered":"2021-02-10T04:53:59.741956+0000","last_clean":"2021-02-10T04:52:30.813009+0000","last_became_active":"2021-02-10T04:52:32.084961+0000","last_became_peered":"2021-02-10T04:52:32.084961+0000","last_unstale":"2021-02-10T04:53:59.741956+0000","last_undegraded":"2021-02-10T04:52:32.070468+0000","last_fullsized":"2021-02-10T04:52:32.070358+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:47:08.784593+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:47:08.784593+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"1159","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.742464+0000","last_change":"2021-02-10T04:52:30.019225+0000","last_active":"2021-02-10T04:53:59.742464+0000","last_peered":"2021-02-10T04:53:59.742464+0000","last_clean":"2021-02-10T04:53:59.742464+0000","last_became_active":"2021-02-10T04:50:59.561436+0000","last_became_peered":"2021-02-10T04:50:59.561436+0000","last_unstale":"2021-02-10T04:53:59.742464+0000","last_undegraded":"2021-02-10T04:53:59.742464+0000","last_fullsized":"2021-02-10T04:53:59.742464+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.019102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.019102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"1001'1528","reported_seq":"6144","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.741935+0000","last_change":"2021-02-10T04:52:26.036366+0000","last_active":"2021-02-10T04:53:59.741935+0000","last_peered":"2021-02-10T04:53:59.741935+0000","last_clean":"2021-02-10T04:53:59.741935+0000","last_became_active":"2021-02-10T04:52:26.034028+0000","last_became_peered":"2021-02-10T04:52:26.034028+0000","last_unstale":"2021-02-10T04:53:59.741935+0000","last_undegraded":"2021-02-10T04:53:59.741935+0000","last_fullsized":"2021-02-10T04:53:59.741935+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46944136,"num_objects":16,"num_object_clones":15,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":358,"num_read_kb":351934,"num_write":286,"num_write_kb":111355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":58968832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.15","version":"1000'446","reported_seq":"2134","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.741523+0000","last_change":"2021-02-10T04:53:11.051124+0000","last_active":"2021-02-10T04:53:59.741523+0000","last_peered":"2021-02-10T04:53:59.741523+0000","last_clean":"2021-02-10T04:53:59.741523+0000","last_became_active":"2021-02-10T04:51:40.510899+0000","last_became_peered":"2021-02-10T04:51:40.510899+0000","last_unstale":"2021-02-10T04:53:59.741523+0000","last_undegraded":"2021-02-10T04:53:59.741523+0000","last_fullsized":"2021-02-10T04:53:59.741523+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:53:11.051031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.051031+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24896285,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":18983408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"1048'40","reported_seq":"87","reported_epoch":"1196","state":"active+clean+remapped","last_fresh":"2021-02-10T04:53:59.741874+0000","last_change":"2021-02-10T04:53:32.953859+0000","last_active":"2021-02-10T04:53:59.741874+0000","last_peered":"2021-02-10T04:53:59.741874+0000","last_clean":"2021-02-10T04:53:59.741874+0000","last_became_active":"2021-02-10T04:53:21.308544+0000","last_became_peered":"2021-02-10T04:53:21.308544+0000","last_unstale":"2021-02-10T04:53:59.741874+0000","last_undegraded":"2021-02-10T04:53:59.741874+0000","last_fullsized":"2021-02-10T04:53:59.741874+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'40","last_scrub_stamp":"2021-02-10T04:53:32.953697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:32.953697+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":79,"num_read_kb":57,"num_write":41,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1206,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1,3],"avail_no_missing":["2","1","3"],"object_location_counts":[{"shards":"1,2,3","objects":13}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"98","reported_epoch":"1196","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:59.740233+0000","last_change":"2021-02-10T04:52:32.094875+0000","last_active":"2021-02-10T04:53:59.740233+0000","last_peered":"2021-02-10T04:53:59.740233+0000","last_clean":"2021-02-10T04:52:30.056787+0000","last_became_active":"2021-02-10T04:52:32.094875+0000","last_became_peered":"2021-02-10T04:52:32.094875+0000","last_unstale":"2021-02-10T04:53:59.740233+0000","last_undegraded":"2021-02-10T04:52:32.083818+0000","last_fullsized":"2021-02-10T04:52:32.083707+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:51:08.639784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.639784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"1128","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.743154+0000","last_change":"2021-02-10T04:51:09.475150+0000","last_active":"2021-02-10T04:53:59.743154+0000","last_peered":"2021-02-10T04:53:59.743154+0000","last_clean":"2021-02-10T04:53:59.743154+0000","last_became_active":"2021-02-10T04:50:59.585917+0000","last_became_peered":"2021-02-10T04:50:59.585917+0000","last_unstale":"2021-02-10T04:53:59.743154+0000","last_undegraded":"2021-02-10T04:53:59.743154+0000","last_fullsized":"2021-02-10T04:53:59.743154+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:09.475063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:09.475063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"98","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.744871+0000","last_change":"2021-02-10T04:53:34.927483+0000","last_active":"2021-02-10T04:53:59.744871+0000","last_peered":"2021-02-10T04:53:59.744871+0000","last_clean":"2021-02-10T04:53:59.744871+0000","last_became_active":"2021-02-10T04:51:36.859707+0000","last_became_peered":"2021-02-10T04:51:36.859707+0000","last_unstale":"2021-02-10T04:53:59.744871+0000","last_undegraded":"2021-02-10T04:53:59.744871+0000","last_fullsized":"2021-02-10T04:53:59.744871+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:34.927424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.927424+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"1000'620","reported_seq":"4483","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.742316+0000","last_change":"2021-02-10T04:53:37.384461+0000","last_active":"2021-02-10T04:53:59.742316+0000","last_peered":"2021-02-10T04:53:59.742316+0000","last_clean":"2021-02-10T04:53:59.742316+0000","last_became_active":"2021-02-10T04:51:40.510114+0000","last_became_peered":"2021-02-10T04:51:40.510114+0000","last_unstale":"2021-02-10T04:53:59.742316+0000","last_undegraded":"2021-02-10T04:53:59.742316+0000","last_fullsized":"2021-02-10T04:53:59.742316+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:53:37.384398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:37.384398+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28530390,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"1063'30","reported_seq":"1172","reported_epoch":"1196","state":"active+clean+remapped","last_fresh":"2021-02-10T04:54:00.503836+0000","last_change":"2021-02-10T04:53:21.312523+0000","last_active":"2021-02-10T04:54:00.503836+0000","last_peered":"2021-02-10T04:54:00.503836+0000","last_clean":"2021-02-10T04:54:00.503836+0000","last_became_active":"2021-02-10T04:53:21.302364+0000","last_became_peered":"2021-02-10T04:53:21.302364+0000","last_unstale":"2021-02-10T04:54:00.503836+0000","last_undegraded":"2021-02-10T04:54:00.503836+0000","last_fullsized":"2021-02-10T04:54:00.503836+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"1063'30","last_scrub_stamp":"2021-02-10T04:52:43.102165+0000","last_deep_scrub":"1063'30","last_deep_scrub_stamp":"2021-02-10T04:47:05.589582+0000","last_clean_scrub_stamp":"2021-02-10T04:52:43.102165+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5,3],"avail_no_missing":["1","3","5"],"object_location_counts":[{"shards":"1,3,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"1184","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.740872+0000","last_change":"2021-02-10T04:52:41.059269+0000","last_active":"2021-02-10T04:53:59.740872+0000","last_peered":"2021-02-10T04:53:59.740872+0000","last_clean":"2021-02-10T04:53:59.740872+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:53:59.740872+0000","last_undegraded":"2021-02-10T04:53:59.740872+0000","last_fullsized":"2021-02-10T04:53:59.740872+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:52:41.059159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.059159+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"1188","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.744923+0000","last_change":"2021-02-10T04:53:12.332890+0000","last_active":"2021-02-10T04:53:59.744923+0000","last_peered":"2021-02-10T04:53:59.744923+0000","last_clean":"2021-02-10T04:53:59.744923+0000","last_became_active":"2021-02-10T04:50:59.540375+0000","last_became_peered":"2021-02-10T04:50:59.540375+0000","last_unstale":"2021-02-10T04:53:59.744923+0000","last_undegraded":"2021-02-10T04:53:59.744923+0000","last_fullsized":"2021-02-10T04:53:59.744923+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.332233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.332233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"100","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.774044+0000","last_change":"2021-02-10T04:53:46.734089+0000","last_active":"2021-02-10T04:53:59.774044+0000","last_peered":"2021-02-10T04:53:59.774044+0000","last_clean":"2021-02-10T04:53:59.774044+0000","last_became_active":"2021-02-10T04:53:46.733191+0000","last_became_peered":"2021-02-10T04:53:46.733191+0000","last_unstale":"2021-02-10T04:53:59.774044+0000","last_undegraded":"2021-02-10T04:53:59.774044+0000","last_fullsized":"2021-02-10T04:53:59.774044+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:45.362537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:45.362537+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"1002'2042","reported_seq":"9170","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.774499+0000","last_change":"2021-02-10T04:53:46.726353+0000","last_active":"2021-02-10T04:53:59.774499+0000","last_peered":"2021-02-10T04:53:59.774499+0000","last_clean":"2021-02-10T04:53:59.774499+0000","last_became_active":"2021-02-10T04:53:46.725828+0000","last_became_peered":"2021-02-10T04:53:46.725828+0000","last_unstale":"2021-02-10T04:53:59.774499+0000","last_undegraded":"2021-02-10T04:53:59.774499+0000","last_fullsized":"2021-02-10T04:53:59.774499+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:52:41.286006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.286006+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46300076,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":545,"num_read_kb":531689,"num_write":399,"num_write_kb":153087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"1049'46","reported_seq":"1352","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.742654+0000","last_change":"2021-02-10T04:52:35.091494+0000","last_active":"2021-02-10T04:53:59.742654+0000","last_peered":"2021-02-10T04:53:59.742654+0000","last_clean":"2021-02-10T04:53:59.742654+0000","last_became_active":"2021-02-10T04:50:59.569915+0000","last_became_peered":"2021-02-10T04:50:59.569915+0000","last_unstale":"2021-02-10T04:53:59.742654+0000","last_undegraded":"2021-02-10T04:53:59.742654+0000","last_fullsized":"2021-02-10T04:53:59.742654+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'46","last_scrub_stamp":"2021-02-10T04:52:35.091374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:35.091374+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":65,"num_write":46,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"84'7","reported_seq":"1212","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.743681+0000","last_change":"2021-02-10T04:51:52.498251+0000","last_active":"2021-02-10T04:53:59.743681+0000","last_peered":"2021-02-10T04:53:59.743681+0000","last_clean":"2021-02-10T04:53:59.743681+0000","last_became_active":"2021-02-10T04:50:59.575924+0000","last_became_peered":"2021-02-10T04:50:59.575924+0000","last_unstale":"2021-02-10T04:53:59.743681+0000","last_undegraded":"2021-02-10T04:53:59.743681+0000","last_fullsized":"2021-02-10T04:53:59.743681+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'7","last_scrub_stamp":"2021-02-10T04:51:52.498158+0000","last_deep_scrub":"84'7","last_deep_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.498158+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"1148'3","reported_seq":"1516","reported_epoch":"1197","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:03.537866+0000","last_change":"2021-02-10T04:52:32.094670+0000","last_active":"2021-02-10T04:54:03.537866+0000","last_peered":"2021-02-10T04:54:03.537866+0000","last_clean":"2021-02-10T04:52:30.059221+0000","last_became_active":"2021-02-10T04:52:32.094670+0000","last_became_peered":"2021-02-10T04:52:32.094670+0000","last_unstale":"2021-02-10T04:54:03.537866+0000","last_undegraded":"2021-02-10T04:52:32.078919+0000","last_fullsized":"2021-02-10T04:52:32.078798+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:47.407557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:47.407557+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"1105","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.774486+0000","last_change":"2021-02-10T04:53:46.727163+0000","last_active":"2021-02-10T04:53:59.774486+0000","last_peered":"2021-02-10T04:53:59.774486+0000","last_clean":"2021-02-10T04:49:56.939276+0000","last_became_active":"2021-02-10T04:53:46.727163+0000","last_became_peered":"2021-02-10T04:53:46.727163+0000","last_unstale":"2021-02-10T04:53:59.774486+0000","last_undegraded":"2021-02-10T04:53:59.774486+0000","last_fullsized":"2021-02-10T04:53:46.702239+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:23.566027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_clean_scrub_stamp":"2021-02-10T04:52:23.566027+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.16","version":"1000'490","reported_seq":"3704","reported_epoch":"1196","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:59.742258+0000","last_change":"2021-02-10T04:52:32.077228+0000","last_active":"2021-02-10T04:53:59.742258+0000","last_peered":"2021-02-10T04:53:59.742258+0000","last_clean":"2021-02-10T04:52:30.062406+0000","last_became_active":"2021-02-10T04:52:32.077228+0000","last_became_peered":"2021-02-10T04:52:32.077228+0000","last_unstale":"2021-02-10T04:53:59.742258+0000","last_undegraded":"2021-02-10T04:52:32.071522+0000","last_fullsized":"2021-02-10T04:52:32.071453+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1013,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:50:03.756248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:50:03.756248+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16154790,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"1112'40","reported_seq":"1256","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.744689+0000","last_change":"2021-02-10T04:53:10.276449+0000","last_active":"2021-02-10T04:53:59.744689+0000","last_peered":"2021-02-10T04:53:59.744689+0000","last_clean":"2021-02-10T04:53:59.744689+0000","last_became_active":"2021-02-10T04:50:59.539529+0000","last_became_peered":"2021-02-10T04:50:59.539529+0000","last_unstale":"2021-02-10T04:53:59.744689+0000","last_undegraded":"2021-02-10T04:53:59.744689+0000","last_fullsized":"2021-02-10T04:53:59.744689+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1112'40","last_scrub_stamp":"2021-02-10T04:53:10.276358+0000","last_deep_scrub":"1112'40","last_deep_scrub_stamp":"2021-02-10T04:51:01.450050+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.276358+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":59,"num_read_kb":44,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"1161","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.740821+0000","last_change":"2021-02-10T04:52:32.080868+0000","last_active":"2021-02-10T04:53:59.740821+0000","last_peered":"2021-02-10T04:53:59.740821+0000","last_clean":"2021-02-10T04:49:56.937464+0000","last_became_active":"2021-02-10T04:52:32.080868+0000","last_became_peered":"2021-02-10T04:52:32.080868+0000","last_unstale":"2021-02-10T04:53:59.740821+0000","last_undegraded":"2021-02-10T04:53:59.740821+0000","last_fullsized":"2021-02-10T04:52:32.068376+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:23.904210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:47:23.904210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"1119'6","reported_seq":"1439","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:54:03.020310+0000","last_change":"2021-02-10T04:52:40.038646+0000","last_active":"2021-02-10T04:54:03.020310+0000","last_peered":"2021-02-10T04:54:03.020310+0000","last_clean":"2021-02-10T04:54:03.020310+0000","last_became_active":"2021-02-10T04:51:36.869733+0000","last_became_peered":"2021-02-10T04:51:36.869733+0000","last_unstale":"2021-02-10T04:54:03.020310+0000","last_undegraded":"2021-02-10T04:54:03.020310+0000","last_fullsized":"2021-02-10T04:54:03.020310+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'6","last_scrub_stamp":"2021-02-10T04:52:40.038554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:40.038554+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"1183","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.742173+0000","last_change":"2021-02-10T04:51:36.849933+0000","last_active":"2021-02-10T04:53:59.742173+0000","last_peered":"2021-02-10T04:53:59.742173+0000","last_clean":"2021-02-10T04:53:59.742173+0000","last_became_active":"2021-02-10T04:51:36.849184+0000","last_became_peered":"2021-02-10T04:51:36.849184+0000","last_unstale":"2021-02-10T04:53:59.742173+0000","last_undegraded":"2021-02-10T04:53:59.742173+0000","last_fullsized":"2021-02-10T04:53:59.742173+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:11.779779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:11.779779+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"1032'89","reported_seq":"791","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.772714+0000","last_change":"2021-02-10T04:53:46.728868+0000","last_active":"2021-02-10T04:53:59.772714+0000","last_peered":"2021-02-10T04:53:59.772714+0000","last_clean":"2021-02-10T04:53:59.772714+0000","last_became_active":"2021-02-10T04:53:46.728356+0000","last_became_peered":"2021-02-10T04:53:46.728356+0000","last_unstale":"2021-02-10T04:53:59.772714+0000","last_undegraded":"2021-02-10T04:53:59.772714+0000","last_fullsized":"2021-02-10T04:53:59.772714+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":5,"last_scrub":"1032'89","last_scrub_stamp":"2021-02-10T04:52:34.089789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:34.089789+0000","log_size":89,"ondisk_log_size":89,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5218574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":30326,"num_write":21,"num_write_kb":12444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3210022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"1163'37","reported_seq":"879","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:54:00.502362+0000","last_change":"2021-02-10T04:53:14.202505+0000","last_active":"2021-02-10T04:54:00.502362+0000","last_peered":"2021-02-10T04:54:00.502362+0000","last_clean":"2021-02-10T04:54:00.502362+0000","last_became_active":"2021-02-10T04:51:49.496294+0000","last_became_peered":"2021-02-10T04:51:49.496294+0000","last_unstale":"2021-02-10T04:54:00.502362+0000","last_undegraded":"2021-02-10T04:54:00.502362+0000","last_fullsized":"2021-02-10T04:54:00.502362+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1163'37","last_scrub_stamp":"2021-02-10T04:53:14.202445+0000","last_deep_scrub":"1163'37","last_deep_scrub_stamp":"2021-02-10T04:53:14.202445+0000","last_clean_scrub_stamp":"2021-02-10T04:53:14.202445+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":33,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"1197","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.741326+0000","last_change":"2021-02-10T04:52:23.095651+0000","last_active":"2021-02-10T04:53:59.741326+0000","last_peered":"2021-02-10T04:53:59.741326+0000","last_clean":"2021-02-10T04:53:59.741326+0000","last_became_active":"2021-02-10T04:50:59.543531+0000","last_became_peered":"2021-02-10T04:50:59.543531+0000","last_unstale":"2021-02-10T04:53:59.741326+0000","last_undegraded":"2021-02-10T04:53:59.741326+0000","last_fullsized":"2021-02-10T04:53:59.741326+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:23.095588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:23.095588+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"1138","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.742088+0000","last_change":"2021-02-10T04:52:32.085370+0000","last_active":"2021-02-10T04:53:59.742088+0000","last_peered":"2021-02-10T04:53:59.742088+0000","last_clean":"2021-02-10T04:52:30.067066+0000","last_became_active":"2021-02-10T04:52:32.085370+0000","last_became_peered":"2021-02-10T04:52:32.085370+0000","last_unstale":"2021-02-10T04:53:59.742088+0000","last_undegraded":"2021-02-10T04:53:59.742088+0000","last_fullsized":"2021-02-10T04:52:32.074435+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:43.663762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:51:43.663762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"58","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.743579+0000","last_change":"2021-02-10T04:52:32.091922+0000","last_active":"2021-02-10T04:53:59.743579+0000","last_peered":"2021-02-10T04:53:59.743579+0000","last_clean":"2021-02-10T04:25:52.015519+0000","last_became_active":"2021-02-10T04:52:32.091922+0000","last_became_peered":"2021-02-10T04:52:32.091922+0000","last_unstale":"2021-02-10T04:53:59.743579+0000","last_undegraded":"2021-02-10T04:53:59.743579+0000","last_fullsized":"2021-02-10T04:52:32.069065+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:57.568541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:57.568541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.10","version":"1000'1071","reported_seq":"4702","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.740627+0000","last_change":"2021-02-10T04:53:35.302879+0000","last_active":"2021-02-10T04:53:59.740627+0000","last_peered":"2021-02-10T04:53:59.740627+0000","last_clean":"2021-02-10T04:53:59.740627+0000","last_became_active":"2021-02-10T04:51:39.508551+0000","last_became_peered":"2021-02-10T04:51:39.508551+0000","last_unstale":"2021-02-10T04:53:59.740627+0000","last_undegraded":"2021-02-10T04:53:59.740627+0000","last_fullsized":"2021-02-10T04:53:59.740627+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:35.302780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.302780+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"1049'31","reported_seq":"1220","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.741107+0000","last_change":"2021-02-10T04:52:39.695340+0000","last_active":"2021-02-10T04:53:59.741107+0000","last_peered":"2021-02-10T04:53:59.741107+0000","last_clean":"2021-02-10T04:53:59.741107+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:53:59.741107+0000","last_undegraded":"2021-02-10T04:53:59.741107+0000","last_fullsized":"2021-02-10T04:53:59.741107+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'31","last_scrub_stamp":"2021-02-10T04:52:39.695237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:39.695237+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":41,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"429","reported_epoch":"1196","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:00.501881+0000","last_change":"2021-02-10T04:52:32.094351+0000","last_active":"2021-02-10T04:54:00.501881+0000","last_peered":"2021-02-10T04:54:00.501881+0000","last_clean":"2021-02-10T04:52:30.061022+0000","last_became_active":"2021-02-10T04:52:32.094351+0000","last_became_peered":"2021-02-10T04:52:32.094351+0000","last_unstale":"2021-02-10T04:54:00.501881+0000","last_undegraded":"2021-02-10T04:52:32.074350+0000","last_fullsized":"2021-02-10T04:52:32.074239+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:47:08.700069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:08.700069+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"851","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:54:00.502452+0000","last_change":"2021-02-10T04:53:35.216489+0000","last_active":"2021-02-10T04:54:00.502452+0000","last_peered":"2021-02-10T04:54:00.502452+0000","last_clean":"2021-02-10T04:54:00.502452+0000","last_became_active":"2021-02-10T04:51:49.501254+0000","last_became_peered":"2021-02-10T04:51:49.501254+0000","last_unstale":"2021-02-10T04:54:00.502452+0000","last_undegraded":"2021-02-10T04:54:00.502452+0000","last_fullsized":"2021-02-10T04:54:00.502452+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.216394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.216394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"1165","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.740970+0000","last_change":"2021-02-10T04:52:32.087218+0000","last_active":"2021-02-10T04:53:59.740970+0000","last_peered":"2021-02-10T04:53:59.740970+0000","last_clean":"2021-02-10T04:52:30.062886+0000","last_became_active":"2021-02-10T04:52:32.087218+0000","last_became_peered":"2021-02-10T04:52:32.087218+0000","last_unstale":"2021-02-10T04:53:59.740970+0000","last_undegraded":"2021-02-10T04:53:59.740970+0000","last_fullsized":"2021-02-10T04:52:32.072167+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:55.814486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.814486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"1002'1657","reported_seq":"4930","reported_epoch":"1196","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:59.773411+0000","last_change":"2021-02-10T04:53:46.722538+0000","last_active":"2021-02-10T04:53:59.773411+0000","last_peered":"2021-02-10T04:53:59.773411+0000","last_clean":"2021-02-10T04:44:25.563579+0000","last_became_active":"2021-02-10T04:53:46.722538+0000","last_became_peered":"2021-02-10T04:53:46.722538+0000","last_unstale":"2021-02-10T04:53:59.773411+0000","last_undegraded":"2021-02-10T04:53:46.700604+0000","last_fullsized":"2021-02-10T04:53:46.700504+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1124,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:46:54.881412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:54.881412+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":42793067,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":424,"num_read_kb":437674,"num_write":319,"num_write_kb":118798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":17936938,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":17}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"1048'20","reported_seq":"1298","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.741057+0000","last_change":"2021-02-10T04:53:04.058324+0000","last_active":"2021-02-10T04:53:59.741057+0000","last_peered":"2021-02-10T04:53:59.741057+0000","last_clean":"2021-02-10T04:53:59.741057+0000","last_became_active":"2021-02-10T04:50:59.544484+0000","last_became_peered":"2021-02-10T04:50:59.544484+0000","last_unstale":"2021-02-10T04:53:59.741057+0000","last_undegraded":"2021-02-10T04:53:59.741057+0000","last_fullsized":"2021-02-10T04:53:59.741057+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'20","last_scrub_stamp":"2021-02-10T04:53:04.058248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:04.058248+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":20,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"1200","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.745110+0000","last_change":"2021-02-10T04:53:00.343307+0000","last_active":"2021-02-10T04:53:59.745110+0000","last_peered":"2021-02-10T04:53:59.745110+0000","last_clean":"2021-02-10T04:53:59.745110+0000","last_became_active":"2021-02-10T04:51:36.844573+0000","last_became_peered":"2021-02-10T04:51:36.844573+0000","last_unstale":"2021-02-10T04:53:59.745110+0000","last_undegraded":"2021-02-10T04:53:59.745110+0000","last_fullsized":"2021-02-10T04:53:59.745110+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:00.343228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:00.343228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"1168","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.742389+0000","last_change":"2021-02-10T04:50:56.099710+0000","last_active":"2021-02-10T04:53:59.742389+0000","last_peered":"2021-02-10T04:53:59.742389+0000","last_clean":"2021-02-10T04:53:59.742389+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:53:59.742389+0000","last_undegraded":"2021-02-10T04:53:59.742389+0000","last_fullsized":"2021-02-10T04:53:59.742389+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:56.099651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:50:56.099651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"516","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.773432+0000","last_change":"2021-02-10T04:53:46.718715+0000","last_active":"2021-02-10T04:53:59.773432+0000","last_peered":"2021-02-10T04:53:59.773432+0000","last_clean":"2021-02-10T04:53:59.773432+0000","last_became_active":"2021-02-10T04:53:46.718266+0000","last_became_peered":"2021-02-10T04:53:46.718266+0000","last_unstale":"2021-02-10T04:53:59.773432+0000","last_undegraded":"2021-02-10T04:53:59.773432+0000","last_fullsized":"2021-02-10T04:53:59.773432+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.876681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.876681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"998'1009","reported_seq":"5035","reported_epoch":"1196","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:59.740857+0000","last_change":"2021-02-10T04:52:32.090443+0000","last_active":"2021-02-10T04:53:59.740857+0000","last_peered":"2021-02-10T04:53:59.740857+0000","last_clean":"2021-02-10T04:52:30.061024+0000","last_became_active":"2021-02-10T04:52:32.090443+0000","last_became_peered":"2021-02-10T04:52:32.090443+0000","last_unstale":"2021-02-10T04:53:59.740857+0000","last_undegraded":"2021-02-10T04:52:32.076916+0000","last_fullsized":"2021-02-10T04:52:32.076790+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1127,"parent":"0.0","parent_split_bits":5,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:13.032506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:13.032506+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49420827,"num_objects":21,"num_object_clones":19,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":36763798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":21}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"1049'37","reported_seq":"1226","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.741588+0000","last_change":"2021-02-10T04:51:44.694779+0000","last_active":"2021-02-10T04:53:59.741588+0000","last_peered":"2021-02-10T04:53:59.741588+0000","last_clean":"2021-02-10T04:53:59.741588+0000","last_became_active":"2021-02-10T04:50:59.549412+0000","last_became_peered":"2021-02-10T04:50:59.549412+0000","last_unstale":"2021-02-10T04:53:59.741588+0000","last_undegraded":"2021-02-10T04:53:59.741588+0000","last_fullsized":"2021-02-10T04:53:59.741588+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:44.694711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.694711+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":64,"num_read_kb":50,"num_write":37,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"1335","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.742187+0000","last_change":"2021-02-10T04:52:33.088525+0000","last_active":"2021-02-10T04:53:59.742187+0000","last_peered":"2021-02-10T04:53:59.742187+0000","last_clean":"2021-02-10T04:53:59.742187+0000","last_became_active":"2021-02-10T04:51:36.870573+0000","last_became_peered":"2021-02-10T04:51:36.870573+0000","last_unstale":"2021-02-10T04:53:59.742187+0000","last_undegraded":"2021-02-10T04:53:59.742187+0000","last_fullsized":"2021-02-10T04:53:59.742187+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:52:33.088428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:33.088428+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"482","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.741109+0000","last_change":"2021-02-10T04:51:51.836723+0000","last_active":"2021-02-10T04:53:59.741109+0000","last_peered":"2021-02-10T04:53:59.741109+0000","last_clean":"2021-02-10T04:53:59.741109+0000","last_became_active":"2021-02-10T04:50:59.564977+0000","last_became_peered":"2021-02-10T04:50:59.564977+0000","last_unstale":"2021-02-10T04:53:59.741109+0000","last_undegraded":"2021-02-10T04:53:59.741109+0000","last_fullsized":"2021-02-10T04:53:59.741109+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.836638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.836638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"1129","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.742022+0000","last_change":"2021-02-10T04:52:32.081484+0000","last_active":"2021-02-10T04:53:59.742022+0000","last_peered":"2021-02-10T04:53:59.742022+0000","last_clean":"2021-02-10T04:52:30.067390+0000","last_became_active":"2021-02-10T04:52:32.081484+0000","last_became_peered":"2021-02-10T04:52:32.081484+0000","last_unstale":"2021-02-10T04:53:59.742022+0000","last_undegraded":"2021-02-10T04:53:59.742022+0000","last_fullsized":"2021-02-10T04:52:32.074112+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:58.514318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_clean_scrub_stamp":"2021-02-10T04:46:58.514318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1d","version":"1001'1165","reported_seq":"4190","reported_epoch":"1196","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:00.501682+0000","last_change":"2021-02-10T04:52:32.087778+0000","last_active":"2021-02-10T04:54:00.501682+0000","last_peered":"2021-02-10T04:54:00.501682+0000","last_clean":"2021-02-10T04:52:30.061103+0000","last_became_active":"2021-02-10T04:52:32.087778+0000","last_became_peered":"2021-02-10T04:52:32.087778+0000","last_unstale":"2021-02-10T04:54:00.501682+0000","last_undegraded":"2021-02-10T04:52:32.074694+0000","last_fullsized":"2021-02-10T04:52:32.074582+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1144,"parent":"0.0","parent_split_bits":5,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29601845,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":287,"num_read_kb":299530,"num_write":200,"num_write_kb":89131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":13}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"1241","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.744135+0000","last_change":"2021-02-10T04:53:01.331566+0000","last_active":"2021-02-10T04:53:59.744135+0000","last_peered":"2021-02-10T04:53:59.744135+0000","last_clean":"2021-02-10T04:53:59.744135+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:53:59.744135+0000","last_undegraded":"2021-02-10T04:53:59.744135+0000","last_fullsized":"2021-02-10T04:53:59.744135+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:01.331466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.331466+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"1098'3","reported_seq":"1515","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:54:03.020439+0000","last_change":"2021-02-10T04:53:34.865668+0000","last_active":"2021-02-10T04:54:03.020439+0000","last_peered":"2021-02-10T04:54:03.020439+0000","last_clean":"2021-02-10T04:54:03.020439+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:54:03.020439+0000","last_undegraded":"2021-02-10T04:54:03.020439+0000","last_fullsized":"2021-02-10T04:54:03.020439+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"1098'3","last_scrub_stamp":"2021-02-10T04:53:34.865579+0000","last_deep_scrub":"1098'3","last_deep_scrub_stamp":"2021-02-10T04:50:15.255137+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.865579+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"1046","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.774141+0000","last_change":"2021-02-10T04:53:46.727603+0000","last_active":"2021-02-10T04:53:59.774141+0000","last_peered":"2021-02-10T04:53:59.774141+0000","last_clean":"2021-02-10T04:53:59.774141+0000","last_became_active":"2021-02-10T04:53:46.727053+0000","last_became_peered":"2021-02-10T04:53:46.727053+0000","last_unstale":"2021-02-10T04:53:59.774141+0000","last_undegraded":"2021-02-10T04:53:59.774141+0000","last_fullsized":"2021-02-10T04:53:59.774141+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:47.493057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.493057+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1c","version":"1002'1090","reported_seq":"5221","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.744236+0000","last_change":"2021-02-10T04:53:41.353991+0000","last_active":"2021-02-10T04:53:59.744236+0000","last_peered":"2021-02-10T04:53:59.744236+0000","last_clean":"2021-02-10T04:53:59.744236+0000","last_became_active":"2021-02-10T04:52:31.088265+0000","last_became_peered":"2021-02-10T04:52:31.088265+0000","last_unstale":"2021-02-10T04:53:59.744236+0000","last_undegraded":"2021-02-10T04:53:59.744236+0000","last_fullsized":"2021-02-10T04:53:59.744236+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:53:41.353840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:41.353840+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"1133","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.741333+0000","last_change":"2021-02-10T04:52:32.079189+0000","last_active":"2021-02-10T04:53:59.741333+0000","last_peered":"2021-02-10T04:53:59.741333+0000","last_clean":"2021-02-10T04:52:30.063079+0000","last_became_active":"2021-02-10T04:52:32.079189+0000","last_became_peered":"2021-02-10T04:52:32.079189+0000","last_unstale":"2021-02-10T04:53:59.741333+0000","last_undegraded":"2021-02-10T04:53:59.741333+0000","last_fullsized":"2021-02-10T04:52:32.071674+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:42.537443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:42.537443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"1117","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.774692+0000","last_change":"2021-02-10T04:53:46.721135+0000","last_active":"2021-02-10T04:53:59.774692+0000","last_peered":"2021-02-10T04:53:59.774692+0000","last_clean":"2021-02-10T04:51:27.994110+0000","last_became_active":"2021-02-10T04:53:46.721135+0000","last_became_peered":"2021-02-10T04:53:46.721135+0000","last_unstale":"2021-02-10T04:53:59.774692+0000","last_undegraded":"2021-02-10T04:53:59.774692+0000","last_fullsized":"2021-02-10T04:53:46.702702+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.938731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.938731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"408","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:54:00.503506+0000","last_change":"2021-02-10T04:53:15.185748+0000","last_active":"2021-02-10T04:54:00.503506+0000","last_peered":"2021-02-10T04:54:00.503506+0000","last_clean":"2021-02-10T04:54:00.503506+0000","last_became_active":"2021-02-10T04:51:49.499348+0000","last_became_peered":"2021-02-10T04:51:49.499348+0000","last_unstale":"2021-02-10T04:54:00.503506+0000","last_undegraded":"2021-02-10T04:54:00.503506+0000","last_fullsized":"2021-02-10T04:54:00.503506+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:15.185686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:15.185686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1f","version":"1004'1093","reported_seq":"6458","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.741696+0000","last_change":"2021-02-10T04:52:31.089952+0000","last_active":"2021-02-10T04:53:59.741696+0000","last_peered":"2021-02-10T04:53:59.741696+0000","last_clean":"2021-02-10T04:53:59.741696+0000","last_became_active":"2021-02-10T04:52:31.088634+0000","last_became_peered":"2021-02-10T04:52:31.088634+0000","last_unstale":"2021-02-10T04:53:59.741696+0000","last_undegraded":"2021-02-10T04:53:59.741696+0000","last_fullsized":"2021-02-10T04:53:59.741696+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39434697,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":279,"num_read_kb":311058,"num_write":186,"num_write_kb":66790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":53995913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"1185","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.741354+0000","last_change":"2021-02-10T04:52:36.083849+0000","last_active":"2021-02-10T04:53:59.741354+0000","last_peered":"2021-02-10T04:53:59.741354+0000","last_clean":"2021-02-10T04:53:59.741354+0000","last_became_active":"2021-02-10T04:25:36.336442+0000","last_became_peered":"2021-02-10T04:25:36.336442+0000","last_unstale":"2021-02-10T04:53:59.741354+0000","last_undegraded":"2021-02-10T04:53:59.741354+0000","last_fullsized":"2021-02-10T04:53:59.741354+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:52:36.083784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:36.083784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"411","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:54:00.446619+0000","last_change":"2021-02-10T04:52:32.084893+0000","last_active":"2021-02-10T04:54:00.446619+0000","last_peered":"2021-02-10T04:54:00.446619+0000","last_clean":"2021-02-10T04:52:30.061372+0000","last_became_active":"2021-02-10T04:52:32.084893+0000","last_became_peered":"2021-02-10T04:52:32.084893+0000","last_unstale":"2021-02-10T04:54:00.446619+0000","last_undegraded":"2021-02-10T04:54:00.446619+0000","last_fullsized":"2021-02-10T04:52:32.073304+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:08.758129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.758129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"1165","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.741383+0000","last_change":"2021-02-10T04:51:44.799512+0000","last_active":"2021-02-10T04:53:59.741383+0000","last_peered":"2021-02-10T04:53:59.741383+0000","last_clean":"2021-02-10T04:53:59.741383+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:53:59.741383+0000","last_undegraded":"2021-02-10T04:53:59.741383+0000","last_fullsized":"2021-02-10T04:53:59.741383+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:44.799429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.799429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1e","version":"1063'654","reported_seq":"4441","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:54:00.503728+0000","last_change":"2021-02-10T04:53:40.135450+0000","last_active":"2021-02-10T04:54:00.503728+0000","last_peered":"2021-02-10T04:54:00.503728+0000","last_clean":"2021-02-10T04:54:00.503728+0000","last_became_active":"2021-02-10T04:52:31.087662+0000","last_became_peered":"2021-02-10T04:52:31.087662+0000","last_unstale":"2021-02-10T04:54:00.503728+0000","last_undegraded":"2021-02-10T04:54:00.503728+0000","last_fullsized":"2021-02-10T04:54:00.503728+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:53:40.135367+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:53:40.135367+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"1148","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.742579+0000","last_change":"2021-02-10T04:51:08.006463+0000","last_active":"2021-02-10T04:53:59.742579+0000","last_peered":"2021-02-10T04:53:59.742579+0000","last_clean":"2021-02-10T04:53:59.742579+0000","last_became_active":"2021-02-10T04:50:59.563657+0000","last_became_peered":"2021-02-10T04:50:59.563657+0000","last_unstale":"2021-02-10T04:53:59.742579+0000","last_undegraded":"2021-02-10T04:53:59.742579+0000","last_fullsized":"2021-02-10T04:53:59.742579+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_clean_scrub_stamp":"2021-02-10T04:51:02.455702+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"1200","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.741188+0000","last_change":"2021-02-10T04:52:33.741801+0000","last_active":"2021-02-10T04:53:59.741188+0000","last_peered":"2021-02-10T04:53:59.741188+0000","last_clean":"2021-02-10T04:53:59.741188+0000","last_became_active":"2021-02-10T04:51:36.860129+0000","last_became_peered":"2021-02-10T04:51:36.860129+0000","last_unstale":"2021-02-10T04:53:59.741188+0000","last_undegraded":"2021-02-10T04:53:59.741188+0000","last_fullsized":"2021-02-10T04:53:59.741188+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:33.741711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:46:58.666264+0000","last_clean_scrub_stamp":"2021-02-10T04:52:33.741711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"1160","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.742976+0000","last_change":"2021-02-10T04:52:25.030419+0000","last_active":"2021-02-10T04:53:59.742976+0000","last_peered":"2021-02-10T04:53:59.742976+0000","last_clean":"2021-02-10T04:53:59.742976+0000","last_became_active":"2021-02-10T04:50:59.539582+0000","last_became_peered":"2021-02-10T04:50:59.539582+0000","last_unstale":"2021-02-10T04:53:59.742976+0000","last_undegraded":"2021-02-10T04:53:59.742976+0000","last_fullsized":"2021-02-10T04:53:59.742976+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:25.030281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.030281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"1003'739","reported_seq":"3945","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.739943+0000","last_change":"2021-02-10T04:53:01.929261+0000","last_active":"2021-02-10T04:53:59.739943+0000","last_peered":"2021-02-10T04:53:59.739943+0000","last_clean":"2021-02-10T04:53:59.739943+0000","last_became_active":"2021-02-10T04:51:40.511505+0000","last_became_peered":"2021-02-10T04:51:40.511505+0000","last_unstale":"2021-02-10T04:53:59.739943+0000","last_undegraded":"2021-02-10T04:53:59.739943+0000","last_fullsized":"2021-02-10T04:53:59.739943+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:53:01.929169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.929169+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22230846,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":18429397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"1112'47","reported_seq":"83","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.740753+0000","last_change":"2021-02-10T04:53:21.334003+0000","last_active":"2021-02-10T04:53:59.740753+0000","last_peered":"2021-02-10T04:53:59.740753+0000","last_clean":"2021-02-10T04:53:59.740753+0000","last_became_active":"2021-02-10T04:53:20.297145+0000","last_became_peered":"2021-02-10T04:53:20.297145+0000","last_unstale":"2021-02-10T04:53:59.740753+0000","last_undegraded":"2021-02-10T04:53:59.740753+0000","last_fullsized":"2021-02-10T04:53:59.740753+0000","mapping_epoch":1166,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"1112'47","last_scrub_stamp":"2021-02-10T04:51:02.860835+0000","last_deep_scrub":"1112'47","last_deep_scrub_stamp":"2021-02-10T04:51:02.860835+0000","last_clean_scrub_stamp":"2021-02-10T04:51:02.860835+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":71,"num_read_kb":55,"num_write":38,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"1037","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.773963+0000","last_change":"2021-02-10T04:53:46.727590+0000","last_active":"2021-02-10T04:53:59.773963+0000","last_peered":"2021-02-10T04:53:59.773963+0000","last_clean":"2021-02-10T04:53:59.773963+0000","last_became_active":"2021-02-10T04:53:46.726584+0000","last_became_peered":"2021-02-10T04:53:46.726584+0000","last_unstale":"2021-02-10T04:53:59.773963+0000","last_undegraded":"2021-02-10T04:53:59.773963+0000","last_fullsized":"2021-02-10T04:53:59.773963+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:26.408326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.408326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"1197","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.744060+0000","last_change":"2021-02-10T04:53:07.324650+0000","last_active":"2021-02-10T04:53:59.744060+0000","last_peered":"2021-02-10T04:53:59.744060+0000","last_clean":"2021-02-10T04:53:59.744060+0000","last_became_active":"2021-02-10T04:50:59.575340+0000","last_became_peered":"2021-02-10T04:50:59.575340+0000","last_unstale":"2021-02-10T04:53:59.744060+0000","last_undegraded":"2021-02-10T04:53:59.744060+0000","last_fullsized":"2021-02-10T04:53:59.744060+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.324564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.324564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"58","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:54:00.446699+0000","last_change":"2021-02-10T04:52:32.091240+0000","last_active":"2021-02-10T04:54:00.446699+0000","last_peered":"2021-02-10T04:54:00.446699+0000","last_clean":"2021-02-10T04:25:52.015519+0000","last_became_active":"2021-02-10T04:52:32.091240+0000","last_became_peered":"2021-02-10T04:52:32.091240+0000","last_unstale":"2021-02-10T04:54:00.446699+0000","last_undegraded":"2021-02-10T04:54:00.446699+0000","last_fullsized":"2021-02-10T04:52:32.077955+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:00.501127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:00.501127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.18","version":"259'74","reported_seq":"59","reported_epoch":"1196","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:53:59.740013+0000","last_change":"2021-02-10T04:52:32.083683+0000","last_active":"2021-02-10T04:53:59.740013+0000","last_peered":"2021-02-10T04:53:59.740013+0000","last_clean":"2021-02-10T04:31:15.165244+0000","last_became_active":"2021-02-10T04:52:32.083683+0000","last_became_peered":"2021-02-10T04:52:32.083683+0000","last_unstale":"2021-02-10T04:53:59.740013+0000","last_undegraded":"2021-02-10T04:52:32.067776+0000","last_fullsized":"2021-02-10T04:52:32.067560+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1010,"parent":"0.0","parent_split_bits":5,"last_scrub":"259'74","last_scrub_stamp":"2021-02-10T04:52:26.597668+0000","last_deep_scrub":"259'74","last_deep_scrub_stamp":"2021-02-10T04:46:49.745104+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.597668+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1218524,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":28228,"num_write":19,"num_write_kb":7022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"1061'20","reported_seq":"1100","reported_epoch":"1196","state":"active+clean+remapped","last_fresh":"2021-02-10T04:54:00.502504+0000","last_change":"2021-02-10T04:53:21.301748+0000","last_active":"2021-02-10T04:54:00.502504+0000","last_peered":"2021-02-10T04:54:00.502504+0000","last_clean":"2021-02-10T04:54:00.502504+0000","last_became_active":"2021-02-10T04:53:21.301122+0000","last_became_peered":"2021-02-10T04:53:21.301122+0000","last_unstale":"2021-02-10T04:54:00.502504+0000","last_undegraded":"2021-02-10T04:54:00.502504+0000","last_fullsized":"2021-02-10T04:54:00.502504+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"1061'20","last_scrub_stamp":"2021-02-10T04:52:31.770907+0000","last_deep_scrub":"1061'20","last_deep_scrub_stamp":"2021-02-10T04:46:55.646261+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.770907+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":17,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,3],"avail_no_missing":["1","3","4"],"object_location_counts":[{"shards":"1,3,4","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"1059","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.774150+0000","last_change":"2021-02-10T04:53:46.724425+0000","last_active":"2021-02-10T04:53:59.774150+0000","last_peered":"2021-02-10T04:53:59.774150+0000","last_clean":"2021-02-10T04:52:30.068110+0000","last_became_active":"2021-02-10T04:53:46.724425+0000","last_became_peered":"2021-02-10T04:53:46.724425+0000","last_unstale":"2021-02-10T04:53:59.774150+0000","last_undegraded":"2021-02-10T04:53:59.774150+0000","last_fullsized":"2021-02-10T04:53:46.707467+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:59.167911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:50:59.167911+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"107","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.772693+0000","last_change":"2021-02-10T04:53:46.718512+0000","last_active":"2021-02-10T04:53:59.772693+0000","last_peered":"2021-02-10T04:53:59.772693+0000","last_clean":"2021-02-10T04:53:59.772693+0000","last_became_active":"2021-02-10T04:53:46.716811+0000","last_became_peered":"2021-02-10T04:53:46.716811+0000","last_unstale":"2021-02-10T04:53:59.772693+0000","last_undegraded":"2021-02-10T04:53:59.772693+0000","last_fullsized":"2021-02-10T04:53:59.772693+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:45.813676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.813676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"1186","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.743517+0000","last_change":"2021-02-10T04:53:38.363899+0000","last_active":"2021-02-10T04:53:59.743517+0000","last_peered":"2021-02-10T04:53:59.743517+0000","last_clean":"2021-02-10T04:53:59.743517+0000","last_became_active":"2021-02-10T04:50:59.539256+0000","last_became_peered":"2021-02-10T04:50:59.539256+0000","last_unstale":"2021-02-10T04:53:59.743517+0000","last_undegraded":"2021-02-10T04:53:59.743517+0000","last_fullsized":"2021-02-10T04:53:59.743517+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:38.363825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:09.433793+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.363825+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1b","version":"1003'1689","reported_seq":"4627","reported_epoch":"1196","state":"active+clean+remapped","last_fresh":"2021-02-10T04:53:59.740754+0000","last_change":"2021-02-10T04:53:10.075976+0000","last_active":"2021-02-10T04:53:59.740754+0000","last_peered":"2021-02-10T04:53:59.740754+0000","last_clean":"2021-02-10T04:53:59.740754+0000","last_became_active":"2021-02-10T04:52:33.087105+0000","last_became_peered":"2021-02-10T04:52:33.087105+0000","last_unstale":"2021-02-10T04:53:59.740754+0000","last_undegraded":"2021-02-10T04:53:59.740754+0000","last_fullsized":"2021-02-10T04:53:59.740754+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:53:10.075879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.075879+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27159495,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3,1],"avail_no_missing":["2","1","3"],"object_location_counts":[{"shards":"1,2,3","objects":12}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"1048'44","reported_seq":"80","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.743254+0000","last_change":"2021-02-10T04:53:32.373404+0000","last_active":"2021-02-10T04:53:59.743254+0000","last_peered":"2021-02-10T04:53:59.743254+0000","last_clean":"2021-02-10T04:53:59.743254+0000","last_became_active":"2021-02-10T04:53:20.304544+0000","last_became_peered":"2021-02-10T04:53:20.304544+0000","last_unstale":"2021-02-10T04:53:59.743254+0000","last_undegraded":"2021-02-10T04:53:59.743254+0000","last_fullsized":"2021-02-10T04:53:59.743254+0000","mapping_epoch":1166,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1167,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'44","last_scrub_stamp":"2021-02-10T04:53:32.373330+0000","last_deep_scrub":"78'32","last_deep_scrub_stamp":"2021-02-10T04:27:09.170428+0000","last_clean_scrub_stamp":"2021-02-10T04:53:32.373330+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":64,"num_read_kb":45,"num_write":34,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"1172","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.742683+0000","last_change":"2021-02-10T04:52:32.087531+0000","last_active":"2021-02-10T04:53:59.742683+0000","last_peered":"2021-02-10T04:53:59.742683+0000","last_clean":"2021-02-10T04:52:30.066743+0000","last_became_active":"2021-02-10T04:52:32.087531+0000","last_became_peered":"2021-02-10T04:52:32.087531+0000","last_unstale":"2021-02-10T04:53:59.742683+0000","last_undegraded":"2021-02-10T04:53:59.742683+0000","last_fullsized":"2021-02-10T04:52:32.076804+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:33.543559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:46:33.543559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"856","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.743321+0000","last_change":"2021-02-10T04:53:04.354508+0000","last_active":"2021-02-10T04:53:59.743321+0000","last_peered":"2021-02-10T04:53:59.743321+0000","last_clean":"2021-02-10T04:53:59.743321+0000","last_became_active":"2021-02-10T04:33:05.169159+0000","last_became_peered":"2021-02-10T04:33:05.169159+0000","last_unstale":"2021-02-10T04:53:59.743321+0000","last_undegraded":"2021-02-10T04:53:59.743321+0000","last_fullsized":"2021-02-10T04:53:59.743321+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:04.354409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:04.354409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"1251","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.740454+0000","last_change":"2021-02-10T04:52:36.777554+0000","last_active":"2021-02-10T04:53:59.740454+0000","last_peered":"2021-02-10T04:53:59.740454+0000","last_clean":"2021-02-10T04:53:59.740454+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:53:59.740454+0000","last_undegraded":"2021-02-10T04:53:59.740454+0000","last_fullsized":"2021-02-10T04:53:59.740454+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:36.777485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:52:36.777485+0000","last_clean_scrub_stamp":"2021-02-10T04:52:36.777485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1a","version":"1001'1528","reported_seq":"5808","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.740161+0000","last_change":"2021-02-10T04:52:31.086406+0000","last_active":"2021-02-10T04:53:59.740161+0000","last_peered":"2021-02-10T04:53:59.740161+0000","last_clean":"2021-02-10T04:53:59.740161+0000","last_became_active":"2021-02-10T04:52:31.085959+0000","last_became_peered":"2021-02-10T04:52:31.085959+0000","last_unstale":"2021-02-10T04:53:59.740161+0000","last_undegraded":"2021-02-10T04:53:59.740161+0000","last_fullsized":"2021-02-10T04:53:59.740161+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46944136,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":358,"num_read_kb":351935,"num_write":286,"num_write_kb":111355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":21311893,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"1049'68","reported_seq":"95","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.740150+0000","last_change":"2021-02-10T04:53:34.122240+0000","last_active":"2021-02-10T04:53:59.740150+0000","last_peered":"2021-02-10T04:53:59.740150+0000","last_clean":"2021-02-10T04:53:59.740150+0000","last_became_active":"2021-02-10T04:53:20.296560+0000","last_became_peered":"2021-02-10T04:53:20.296560+0000","last_unstale":"2021-02-10T04:53:59.740150+0000","last_undegraded":"2021-02-10T04:53:59.740150+0000","last_fullsized":"2021-02-10T04:53:59.740150+0000","mapping_epoch":1166,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1167,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'68","last_scrub_stamp":"2021-02-10T04:53:34.122152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.122152+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":124,"num_read_kb":101,"num_write":69,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"22","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.774935+0000","last_change":"2021-02-10T04:53:46.719798+0000","last_active":"2021-02-10T04:53:59.774935+0000","last_peered":"2021-02-10T04:53:59.774935+0000","last_clean":"2021-02-10T04:25:33.303710+0000","last_became_active":"2021-02-10T04:53:46.719798+0000","last_became_peered":"2021-02-10T04:53:46.719798+0000","last_unstale":"2021-02-10T04:53:59.774935+0000","last_undegraded":"2021-02-10T04:53:59.774935+0000","last_fullsized":"2021-02-10T04:53:46.703571+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:17.919745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:17.919745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"482","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.740600+0000","last_change":"2021-02-10T04:53:07.092032+0000","last_active":"2021-02-10T04:53:59.740600+0000","last_peered":"2021-02-10T04:53:59.740600+0000","last_clean":"2021-02-10T04:53:59.740600+0000","last_became_active":"2021-02-10T04:50:59.572077+0000","last_became_peered":"2021-02-10T04:50:59.572077+0000","last_unstale":"2021-02-10T04:53:59.740600+0000","last_undegraded":"2021-02-10T04:53:59.740600+0000","last_fullsized":"2021-02-10T04:53:59.740600+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.091949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:19.755718+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.091949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"1159","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.742327+0000","last_change":"2021-02-10T04:51:59.838856+0000","last_active":"2021-02-10T04:53:59.742327+0000","last_peered":"2021-02-10T04:53:59.742327+0000","last_clean":"2021-02-10T04:53:59.742327+0000","last_became_active":"2021-02-10T04:50:59.563156+0000","last_became_peered":"2021-02-10T04:50:59.563156+0000","last_unstale":"2021-02-10T04:53:59.742327+0000","last_undegraded":"2021-02-10T04:53:59.742327+0000","last_fullsized":"2021-02-10T04:53:59.742327+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.838790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.838790+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":"892","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.744154+0000","last_change":"2021-02-10T04:52:30.405170+0000","last_active":"2021-02-10T04:53:59.744154+0000","last_peered":"2021-02-10T04:53:59.744154+0000","last_clean":"2021-02-10T04:53:59.744154+0000","last_became_active":"2021-02-10T04:51:36.852058+0000","last_became_peered":"2021-02-10T04:51:36.852058+0000","last_unstale":"2021-02-10T04:53:59.744154+0000","last_undegraded":"2021-02-10T04:53:59.744154+0000","last_fullsized":"2021-02-10T04:53:59.744154+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.405106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:20.635536+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.405106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.20","version":"0'0","reported_seq":"131","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.772489+0000","last_change":"2021-02-10T04:53:46.739888+0000","last_active":"2021-02-10T04:53:59.772489+0000","last_peered":"2021-02-10T04:53:59.772489+0000","last_clean":"2021-02-10T04:53:59.772489+0000","last_became_active":"2021-02-10T04:53:46.739532+0000","last_became_peered":"2021-02-10T04:53:46.739532+0000","last_unstale":"2021-02-10T04:53:59.772489+0000","last_undegraded":"2021-02-10T04:53:59.772489+0000","last_fullsized":"2021-02-10T04:53:59.772489+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:56.449085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:41.375276+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.449085+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":"1124","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.741752+0000","last_change":"2021-02-10T04:53:35.858634+0000","last_active":"2021-02-10T04:53:59.741752+0000","last_peered":"2021-02-10T04:53:59.741752+0000","last_clean":"2021-02-10T04:53:59.741752+0000","last_became_active":"2021-02-10T04:51:36.869785+0000","last_became_peered":"2021-02-10T04:51:36.869785+0000","last_unstale":"2021-02-10T04:53:59.741752+0000","last_undegraded":"2021-02-10T04:53:59.741752+0000","last_fullsized":"2021-02-10T04:53:59.741752+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.858557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.858557+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":"94","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.740331+0000","last_change":"2021-02-10T04:52:32.094666+0000","last_active":"2021-02-10T04:53:59.740331+0000","last_peered":"2021-02-10T04:53:59.740331+0000","last_clean":"2021-02-10T04:52:30.064564+0000","last_became_active":"2021-02-10T04:52:32.094666+0000","last_became_peered":"2021-02-10T04:52:32.094666+0000","last_unstale":"2021-02-10T04:53:59.740331+0000","last_undegraded":"2021-02-10T04:53:59.740331+0000","last_fullsized":"2021-02-10T04:52:32.084052+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:47.481686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:47.481686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":"86","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:54:00.446666+0000","last_change":"2021-02-10T04:52:32.096027+0000","last_active":"2021-02-10T04:54:00.446666+0000","last_peered":"2021-02-10T04:54:00.446666+0000","last_clean":"2021-02-10T04:52:30.061012+0000","last_became_active":"2021-02-10T04:52:32.096027+0000","last_became_peered":"2021-02-10T04:52:32.096027+0000","last_unstale":"2021-02-10T04:54:00.446666+0000","last_undegraded":"2021-02-10T04:54:00.446666+0000","last_fullsized":"2021-02-10T04:52:32.076629+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.706372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.706372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":"1206","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.741040+0000","last_change":"2021-02-10T04:51:57.287452+0000","last_active":"2021-02-10T04:53:59.741040+0000","last_peered":"2021-02-10T04:53:59.741040+0000","last_clean":"2021-02-10T04:53:59.741040+0000","last_became_active":"2021-02-10T04:50:59.566827+0000","last_became_peered":"2021-02-10T04:50:59.566827+0000","last_unstale":"2021-02-10T04:53:59.741040+0000","last_undegraded":"2021-02-10T04:53:59.741040+0000","last_fullsized":"2021-02-10T04:53:59.741040+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.287363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.287363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":"171","reported_epoch":"1196","state":"active+undersized","last_fresh":"2021-02-10T04:53:59.741781+0000","last_change":"2021-02-10T04:52:32.082221+0000","last_active":"2021-02-10T04:53:59.741781+0000","last_peered":"2021-02-10T04:53:59.741781+0000","last_clean":"2021-02-10T04:52:30.068335+0000","last_became_active":"2021-02-10T04:52:32.082221+0000","last_became_peered":"2021-02-10T04:52:32.082221+0000","last_unstale":"2021-02-10T04:53:59.741781+0000","last_undegraded":"2021-02-10T04:53:59.741781+0000","last_fullsized":"2021-02-10T04:52:32.077664+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:10.308282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:10.308282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":"1095","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.742677+0000","last_change":"2021-02-10T04:53:09.121379+0000","last_active":"2021-02-10T04:53:59.742677+0000","last_peered":"2021-02-10T04:53:59.742677+0000","last_clean":"2021-02-10T04:53:59.742677+0000","last_became_active":"2021-02-10T04:50:59.561442+0000","last_became_peered":"2021-02-10T04:50:59.561442+0000","last_unstale":"2021-02-10T04:53:59.742677+0000","last_undegraded":"2021-02-10T04:53:59.742677+0000","last_fullsized":"2021-02-10T04:53:59.742677+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:09.121298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.121298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.21","version":"293'77","reported_seq":"767","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.741451+0000","last_change":"2021-02-10T04:52:28.062923+0000","last_active":"2021-02-10T04:53:59.741451+0000","last_peered":"2021-02-10T04:53:59.741451+0000","last_clean":"2021-02-10T04:53:59.741451+0000","last_became_active":"2021-02-10T04:52:28.062463+0000","last_became_peered":"2021-02-10T04:52:28.062463+0000","last_unstale":"2021-02-10T04:53:59.741451+0000","last_undegraded":"2021-02-10T04:53:59.741451+0000","last_fullsized":"2021-02-10T04:53:59.741451+0000","mapping_epoch":1143,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1144,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1886586,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":15163,"num_write":9,"num_write_kb":3316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":"93","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.739843+0000","last_change":"2021-02-10T04:52:32.124955+0000","last_active":"2021-02-10T04:53:59.739843+0000","last_peered":"2021-02-10T04:53:59.739843+0000","last_clean":"2021-02-10T04:53:59.739843+0000","last_became_active":"2021-02-10T04:51:36.867061+0000","last_became_peered":"2021-02-10T04:51:36.867061+0000","last_unstale":"2021-02-10T04:53:59.739843+0000","last_undegraded":"2021-02-10T04:53:59.739843+0000","last_fullsized":"2021-02-10T04:53:59.739843+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:32.124901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:08.271977+0000","last_clean_scrub_stamp":"2021-02-10T04:52:32.124901+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":"1135","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.741451+0000","last_change":"2021-02-10T04:51:07.182775+0000","last_active":"2021-02-10T04:53:59.741451+0000","last_peered":"2021-02-10T04:53:59.741451+0000","last_clean":"2021-02-10T04:53:59.741451+0000","last_became_active":"2021-02-10T04:28:23.126368+0000","last_became_peered":"2021-02-10T04:28:23.126368+0000","last_unstale":"2021-02-10T04:53:59.741451+0000","last_undegraded":"2021-02-10T04:53:59.741451+0000","last_fullsized":"2021-02-10T04:53:59.741451+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.182678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.182678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.20","version":"1000'1071","reported_seq":"4541","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.739220+0000","last_change":"2021-02-10T04:53:03.915465+0000","last_active":"2021-02-10T04:53:59.739220+0000","last_peered":"2021-02-10T04:53:59.739220+0000","last_clean":"2021-02-10T04:53:59.739220+0000","last_became_active":"2021-02-10T04:52:31.088690+0000","last_became_peered":"2021-02-10T04:52:31.088690+0000","last_unstale":"2021-02-10T04:53:59.739220+0000","last_undegraded":"2021-02-10T04:53:59.739220+0000","last_fullsized":"2021-02-10T04:53:59.739220+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:03.915403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.915403+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16398893,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":"720","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.772624+0000","last_change":"2021-02-10T04:53:46.721599+0000","last_active":"2021-02-10T04:53:59.772624+0000","last_peered":"2021-02-10T04:53:59.772624+0000","last_clean":"2021-02-10T04:53:59.772624+0000","last_became_active":"2021-02-10T04:53:46.721340+0000","last_became_peered":"2021-02-10T04:53:46.721340+0000","last_unstale":"2021-02-10T04:53:59.772624+0000","last_undegraded":"2021-02-10T04:53:59.772624+0000","last_fullsized":"2021-02-10T04:53:59.772624+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:28.092288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:07.767250+0000","last_clean_scrub_stamp":"2021-02-10T04:52:28.092288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.25","version":"53'2","reported_seq":"1152","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.741056+0000","last_change":"2021-02-10T04:53:36.103026+0000","last_active":"2021-02-10T04:53:59.741056+0000","last_peered":"2021-02-10T04:53:59.741056+0000","last_clean":"2021-02-10T04:53:59.741056+0000","last_became_active":"2021-02-10T04:28:23.467770+0000","last_became_peered":"2021-02-10T04:28:23.467770+0000","last_unstale":"2021-02-10T04:53:59.741056+0000","last_undegraded":"2021-02-10T04:53:59.741056+0000","last_fullsized":"2021-02-10T04:53:59.741056+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:53:36.102928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:36.102928+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.23","version":"1002'1657","reported_seq":"5067","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.741217+0000","last_change":"2021-02-10T04:52:31.093874+0000","last_active":"2021-02-10T04:53:59.741217+0000","last_peered":"2021-02-10T04:53:59.741217+0000","last_clean":"2021-02-10T04:53:59.741217+0000","last_became_active":"2021-02-10T04:52:31.092996+0000","last_became_peered":"2021-02-10T04:52:31.092996+0000","last_unstale":"2021-02-10T04:53:59.741217+0000","last_undegraded":"2021-02-10T04:53:59.741217+0000","last_fullsized":"2021-02-10T04:53:59.741217+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21396533,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":212,"num_read_kb":218837,"num_write":159,"num_write_kb":59399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":20479176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":"1242","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.740773+0000","last_change":"2021-02-10T04:53:12.951612+0000","last_active":"2021-02-10T04:53:59.740773+0000","last_peered":"2021-02-10T04:53:59.740773+0000","last_clean":"2021-02-10T04:53:59.740773+0000","last_became_active":"2021-02-10T04:50:59.566548+0000","last_became_peered":"2021-02-10T04:50:59.566548+0000","last_unstale":"2021-02-10T04:53:59.740773+0000","last_undegraded":"2021-02-10T04:53:59.740773+0000","last_fullsized":"2021-02-10T04:53:59.740773+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.951528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.951528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.26","version":"53'1","reported_seq":"34","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.774357+0000","last_change":"2021-02-10T04:53:46.722636+0000","last_active":"2021-02-10T04:53:59.774357+0000","last_peered":"2021-02-10T04:53:59.774357+0000","last_clean":"2021-02-10T04:53:59.774357+0000","last_became_active":"2021-02-10T04:53:46.722227+0000","last_became_peered":"2021-02-10T04:53:46.722227+0000","last_unstale":"2021-02-10T04:53:59.774357+0000","last_undegraded":"2021-02-10T04:53:59.774357+0000","last_fullsized":"2021-02-10T04:53:59.774357+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:53:03.933910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.933910+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":423,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.22","version":"998'1009","reported_seq":"4559","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.740167+0000","last_change":"2021-02-10T04:52:31.086563+0000","last_active":"2021-02-10T04:53:59.740167+0000","last_peered":"2021-02-10T04:53:59.740167+0000","last_clean":"2021-02-10T04:53:59.740167+0000","last_became_active":"2021-02-10T04:52:31.085288+0000","last_became_peered":"2021-02-10T04:52:31.085288+0000","last_unstale":"2021-02-10T04:53:59.740167+0000","last_undegraded":"2021-02-10T04:53:59.740167+0000","last_fullsized":"2021-02-10T04:53:59.740167+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12355207,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":120,"num_read_kb":124702,"num_write":77,"num_write_kb":25052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4145029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":"1124","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.742216+0000","last_change":"2021-02-10T04:53:10.958668+0000","last_active":"2021-02-10T04:53:59.742216+0000","last_peered":"2021-02-10T04:53:59.742216+0000","last_clean":"2021-02-10T04:53:59.742216+0000","last_became_active":"2021-02-10T04:45:49.552026+0000","last_became_peered":"2021-02-10T04:45:49.552026+0000","last_unstale":"2021-02-10T04:53:59.742216+0000","last_undegraded":"2021-02-10T04:53:59.742216+0000","last_fullsized":"2021-02-10T04:53:59.742216+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:10.958579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.958579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":"139","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.774786+0000","last_change":"2021-02-10T04:53:46.725941+0000","last_active":"2021-02-10T04:53:59.774786+0000","last_peered":"2021-02-10T04:53:59.774786+0000","last_clean":"2021-02-10T04:53:59.774786+0000","last_became_active":"2021-02-10T04:53:46.725587+0000","last_became_peered":"2021-02-10T04:53:46.725587+0000","last_unstale":"2021-02-10T04:53:59.774786+0000","last_undegraded":"2021-02-10T04:53:59.774786+0000","last_fullsized":"2021-02-10T04:53:59.774786+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:41.758684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.758684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":"1124","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.740909+0000","last_change":"2021-02-10T04:53:11.139690+0000","last_active":"2021-02-10T04:53:59.740909+0000","last_peered":"2021-02-10T04:53:59.740909+0000","last_clean":"2021-02-10T04:53:59.740909+0000","last_became_active":"2021-02-10T04:50:59.569162+0000","last_became_peered":"2021-02-10T04:50:59.569162+0000","last_unstale":"2021-02-10T04:53:59.740909+0000","last_undegraded":"2021-02-10T04:53:59.740909+0000","last_fullsized":"2021-02-10T04:53:59.740909+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:11.139602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.139602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":"1150","reported_epoch":"1196","state":"active+clean","last_fresh":"2021-02-10T04:53:59.740732+0000","last_change":"2021-02-10T04:52:24.072005+0000","last_active":"2021-02-10T04:53:59.740732+0000","last_peered":"2021-02-10T04:53:59.740732+0000","last_clean":"2021-02-10T04:53:59.740732+0000","last_became_active":"2021-02-10T04:50:59.527945+0000","last_became_peered":"2021-02-10T04:50:59.527945+0000","last_unstale":"2021-02-10T04:53:59.740732+0000","last_undegraded":"2021-02-10T04:53:59.740732+0000","last_fullsized":"2021-02-10T04:53:59.740732+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:24.071915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_clean_scrub_stamp":"2021-02-10T04:52:24.071915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":931687135,"num_objects":374,"num_object_clones":323,"num_object_copies":1122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":164,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":374,"num_whiteouts":10,"num_read":4479,"num_read_kb":4587256,"num_write":3265,"num_write_kb":1255814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":630,"num_bytes_recovered":307448165,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2029686784,"data_stored":2612817805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34722,"ondisk_log_size":34722,"up":94,"acting":95,"num_store_stats":8},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":2368,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":18,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1069,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":5835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":111,"acting":111,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":29,"acting":29,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":570,"num_read_kb":565,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":8,"acting":8,"num_store_stats":7},{"poolid":4,"num_pg":28,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":43,"num_objects_misplaced":32,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":2289,"num_read_kb":1762,"num_write":1306,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":249,"num_bytes_recovered":17362,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2093056,"data_stored":75230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1082,"ondisk_log_size":1082,"up":76,"acting":80,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":84,"acting":84,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":114,"acting":114,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365144,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":189540,"kb_used_data":148444,"kb_used_omap":0,"kb_used_meta":41088,"kb_avail":93555612,"statfs":{"total":95995035648,"available":95800946688,"internally_reserved":0,"allocated":152006656,"data_stored":201926074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42074112},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:27 2021","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":1.2370000000000001,"15min":1.284},"min":{"1min":0.50800000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1220000000000001,"5min":34.299999999999997,"15min":67.277000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.5109999999999999,"15min":1.548},"min":{"1min":0.46300000000000002,"5min":0.438,"15min":0.438},"max":{"1min":2.8690000000000002,"5min":34.652999999999999,"15min":66.933999999999997},"last":0.71399999999999997}]},{"osd":1,"last update":"Wed Feb 10 04:53:36 2021","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.85099999999999998,"15min":0.97299999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.278,"5min":2.254,"15min":10.140000000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.92800000000000005,"15min":0.98999999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.024,"5min":2.5409999999999999,"15min":7.1520000000000001},"last":0.70499999999999996}]},{"osd":2,"last update":"Wed Feb 10 04:53:36 2021","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":1.179,"15min":1.3500000000000001},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.4590000000000001,"5min":11.468,"15min":66.534000000000006},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.84299999999999997,"5min":1.1519999999999999,"15min":1.5249999999999999},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.075,"5min":3.8820000000000001,"15min":66.781999999999996},"last":0.63700000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:53:59 2021","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.2190000000000001,"15min":1.3180000000000001},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.573,"5min":4.5890000000000004,"15min":4.5890000000000004},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.034,"5min":1.306,"15min":1.425},"min":{"1min":0.73199999999999998,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.593,"5min":4.6609999999999996,"15min":4.6609999999999996},"last":0.94599999999999995}]},{"osd":4,"last update":"Wed Feb 10 04:53:36 2021","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.343,"15min":1.47},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.625,"5min":4.694,"15min":10.781000000000001},"last":0.76600000000000001},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.2889999999999999,"15min":1.3700000000000001},"min":{"1min":0.71999999999999997,"5min":0.69099999999999995,"15min":0.65100000000000002},"max":{"1min":1.843,"5min":3.4049999999999998,"15min":10.326000000000001},"last":0.74199999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:53:36 2021","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.4379999999999999,"15min":1.621},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.66700000000000004},"max":{"1min":4.2510000000000003,"5min":4.2510000000000003,"15min":35.283000000000001},"last":0.83199999999999996},{"interface":"front","average":{"1min":1.494,"5min":1.5880000000000001,"15min":1.8799999999999999},"min":{"1min":0.78700000000000003,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":3.9609999999999999,"5min":4.1470000000000002,"15min":35.353999999999999},"last":0.90000000000000002}]},{"osd":6,"last update":"Wed Feb 10 04:53:36 2021","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.593,"15min":1.835},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.8160000000000001,"5min":4.0380000000000003,"15min":16.050000000000001},"last":1.032},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.569,"15min":1.7490000000000001},"min":{"1min":0.878,"5min":0.876,"15min":0.876},"max":{"1min":1.96,"5min":4.2629999999999999,"15min":15.960000000000001},"last":1.002}]}]},{"osd":6,"up_from":43,"seq":184683594076,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":402344,"kb_used_data":341536,"kb_used_omap":0,"kb_used_meta":60800,"kb_avail":93342808,"statfs":{"total":95995035648,"available":95583035392,"internally_reserved":0,"allocated":349732864,"data_stored":442890992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62259200},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:52:12 2021","interfaces":[{"interface":"back","average":{"1min":4.016,"5min":2.7400000000000002,"15min":1.595},"min":{"1min":0.52000000000000002,"5min":0.48099999999999998,"15min":0.44700000000000001},"max":{"1min":61.768000000000001,"5min":137.60599999999999,"15min":137.60599999999999},"last":0.97999999999999998},{"interface":"front","average":{"1min":4.0270000000000001,"5min":2.694,"15min":1.7989999999999999},"min":{"1min":0.504,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":62.683,"5min":137.66,"15min":137.66},"last":0.82099999999999995}]},{"osd":1,"last update":"Wed Feb 10 04:53:14 2021","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.1040000000000001,"15min":1.087},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":13.18,"5min":13.18,"15min":13.18},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.1699999999999999,"15min":1.145},"min":{"1min":0.51800000000000002,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":11.041,"5min":11.041,"15min":11.041},"last":3.6880000000000002}]},{"osd":2,"last update":"Wed Feb 10 04:53:14 2021","interfaces":[{"interface":"back","average":{"1min":8.8919999999999995,"5min":2.7930000000000001,"15min":1.7669999999999999},"min":{"1min":0.60999999999999999,"5min":0.57099999999999995,"15min":0.55600000000000005},"max":{"1min":163.81700000000001,"5min":163.81700000000001,"15min":163.81700000000001},"last":1.4119999999999999},{"interface":"front","average":{"1min":8.9610000000000003,"5min":2.7810000000000001,"15min":1.774},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.59899999999999998},"max":{"1min":164.56800000000001,"5min":164.56800000000001,"15min":164.56800000000001},"last":1.464}]},{"osd":3,"last update":"Wed Feb 10 04:54:01 2021","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.161,"15min":1.1990000000000001},"min":{"1min":0.77600000000000002,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.722,"5min":1.722,"15min":1.722},"last":1.722},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.123,"15min":1.1240000000000001},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.629,"5min":2.363,"15min":2.363},"last":1.629}]},{"osd":4,"last update":"Wed Feb 10 04:53:14 2021","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.341,"15min":1.4399999999999999},"min":{"1min":0.71399999999999997,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":2.2360000000000002,"5min":7.1459999999999999,"15min":7.1459999999999999},"last":1.53},{"interface":"front","average":{"1min":1.147,"5min":1.113,"15min":1.145},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":5.2750000000000004,"5min":11.997999999999999,"15min":11.997999999999999},"last":1.494}]},{"osd":5,"last update":"Wed Feb 10 04:53:14 2021","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.3,"15min":1.6579999999999999},"min":{"1min":0.66800000000000004,"5min":0.61699999999999999,"15min":0.56899999999999995},"max":{"1min":12.045,"5min":12.045,"15min":81.540000000000006},"last":1.379},{"interface":"front","average":{"1min":1.8460000000000001,"5min":1.601,"15min":1.919},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78000000000000003},"max":{"1min":13.401,"5min":13.401,"15min":81.492999999999995},"last":1.7949999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:53:43 2021","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.232,"15min":1.3360000000000001},"min":{"1min":0.80000000000000004,"5min":0.747,"15min":0.747},"max":{"1min":1.7230000000000001,"5min":4.0919999999999996,"15min":5.1820000000000004},"last":1.571},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.5129999999999999,"15min":1.637},"min":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":1.571,"5min":5.4109999999999996,"15min":13.315},"last":2.1440000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575214,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":286000,"kb_used_data":224104,"kb_used_omap":0,"kb_used_meta":61888,"kb_avail":93459152,"statfs":{"total":95995035648,"available":95702171648,"internally_reserved":0,"allocated":229482496,"data_stored":289042522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":63373312},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:48 2021","interfaces":[{"interface":"back","average":{"1min":1.696,"5min":2.6320000000000001,"15min":1.921},"min":{"1min":0.44700000000000001,"5min":0.437,"15min":0.437},"max":{"1min":14.053000000000001,"5min":115.622,"15min":115.622},"last":0.69599999999999995},{"interface":"front","average":{"1min":1.728,"5min":2.585,"15min":1.948},"min":{"1min":0.49199999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":14.109,"5min":115.73099999999999,"15min":115.73099999999999},"last":0.72899999999999998}]},{"osd":2,"last update":"Wed Feb 10 04:54:07 2021","interfaces":[{"interface":"back","average":{"1min":2.3399999999999999,"5min":3.3410000000000002,"15min":2.0350000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":37.975000000000001,"5min":115.59699999999999,"15min":115.59699999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":2.3780000000000001,"5min":3.379,"15min":2.0800000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":37.840000000000003,"5min":115.753,"15min":115.753},"last":0.52200000000000002}]},{"osd":3,"last update":"Wed Feb 10 04:54:03 2021","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.85299999999999998,"15min":0.93000000000000005},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.84699999999999998,"5min":3.077,"15min":3.077},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.96999999999999997,"15min":1.024},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.226,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.70699999999999996}]},{"osd":4,"last update":"Wed Feb 10 04:53:51 2021","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.0289999999999999,"15min":1.1719999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.55200000000000005},"max":{"1min":1.825,"5min":2.9670000000000001,"15min":4.899},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.117,"15min":1.327},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.718,"5min":2.5270000000000001,"15min":29.547999999999998},"last":0.80900000000000005}]},{"osd":5,"last update":"Wed Feb 10 04:53:57 2021","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":1.1020000000000001,"15min":1.198},"min":{"1min":0.628,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.331,"5min":2.7959999999999998,"15min":7.8049999999999997},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.036,"5min":1.236,"15min":1.504},"min":{"1min":0.75800000000000001,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.383,"5min":2.8199999999999998,"15min":28.678000000000001},"last":0.93100000000000005}]},{"osd":6,"last update":"Wed Feb 10 04:54:07 2021","interfaces":[{"interface":"back","average":{"1min":2.2810000000000001,"5min":1.421,"15min":1.571},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":30.997,"5min":30.997,"15min":30.997},"last":0.89100000000000001},{"interface":"front","average":{"1min":2.2599999999999998,"5min":1.4370000000000001,"15min":1.476},"min":{"1min":0.64200000000000002,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":31.138000000000002,"5min":31.138000000000002,"15min":31.138000000000002},"last":1.1559999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:53:23 2021","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.27,"15min":1.524},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.484,"5min":3.274,"15min":28.573},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.3120000000000001,"15min":1.5609999999999999},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.776,"5min":2.298,"15min":29.408999999999999},"last":0.97899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738720,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":399164,"kb_used_data":337140,"kb_used_omap":0,"kb_used_meta":62016,"kb_avail":93345988,"statfs":{"total":95995035648,"available":95586291712,"internally_reserved":0,"allocated":345231360,"data_stored":470653300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":63504384},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 04:51:32 2021","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.81100000000000005,"15min":1.0089999999999999},"min":{"1min":0.47099999999999997,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.0329999999999999,"5min":2.2930000000000001,"15min":28.946999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.874,"5min":0.92300000000000004,"15min":1.0149999999999999},"min":{"1min":0.68000000000000005,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.1819999999999999,"5min":2.0670000000000002,"15min":3.7050000000000001},"last":0.89200000000000002}]},{"osd":2,"last update":"Wed Feb 10 04:52:09 2021","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":2.2389999999999999,"15min":1.613},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0700000000000001,"5min":97.983999999999995,"15min":97.983999999999995},"last":1.056},{"interface":"front","average":{"1min":0.749,"5min":2.153,"15min":1.5169999999999999},"min":{"1min":0.51900000000000002,"5min":0.51300000000000001,"15min":0.47699999999999998},"max":{"1min":1.1319999999999999,"5min":97.844999999999999,"15min":97.844999999999999},"last":1.296}]},{"osd":3,"last update":"Wed Feb 10 04:51:59 2021","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.87,"15min":0.87},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":1.002},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.254,"5min":1.254,"15min":1.254},"last":0.871}]},{"osd":4,"last update":"Wed Feb 10 04:51:32 2021","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":1.119,"15min":1.306},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.3200000000000001,"5min":2.206,"15min":5.1680000000000001},"last":1.1120000000000001},{"interface":"front","average":{"1min":0.96499999999999997,"5min":1.179,"15min":1.252},"min":{"1min":0.64200000000000002,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.607,"5min":3.359,"15min":14.672000000000001},"last":1.0920000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:51:54 2021","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.325,"15min":1.512},"min":{"1min":0.84599999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.714,"5min":3.944,"15min":25.047000000000001},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.413,"15min":1.7090000000000001},"min":{"1min":0.79400000000000004,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":2.1669999999999998,"5min":3.7850000000000001,"15min":28.827999999999999},"last":1.226}]},{"osd":6,"last update":"Wed Feb 10 04:52:23 2021","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.845,"15min":1.8280000000000001},"min":{"1min":1.0109999999999999,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.7889999999999999,"5min":45.012,"15min":45.012},"last":1.587},{"interface":"front","average":{"1min":1.327,"5min":1.891,"15min":1.95},"min":{"1min":0.88800000000000001,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.0979999999999999,"5min":45.225000000000001,"15min":45.225000000000001},"last":1.3620000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:52:28 2021","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.3080000000000001,"15min":1.5069999999999999},"min":{"1min":0.96099999999999997,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.605,"5min":2.8700000000000001,"15min":5.6050000000000004},"last":1.532},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.3959999999999999,"15min":1.5669999999999999},"min":{"1min":1.0129999999999999,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.794,"5min":3.1259999999999999,"15min":13.968999999999999},"last":1.5129999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411692,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":364596,"kb_used_data":317676,"kb_used_omap":0,"kb_used_meta":46912,"kb_avail":93380556,"statfs":{"total":95995035648,"available":95621689344,"internally_reserved":0,"allocated":325300224,"data_stored":374413377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48037888},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:50 2021","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.99099999999999999,"15min":1.718},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":3.698,"5min":3.996,"15min":163.15199999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.91300000000000003,"15min":1.655},"min":{"1min":0.51100000000000001,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.9710000000000001,"5min":2.2349999999999999,"15min":162.959},"last":0.85399999999999998}]},{"osd":1,"last update":"Wed Feb 10 04:53:53 2021","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.874,"15min":1.014},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.044,"5min":3.8159999999999998,"15min":4.9690000000000003},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.73899999999999999,"15min":0.89900000000000002},"min":{"1min":0.47299999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.0009999999999999,"5min":1.506,"15min":5.726},"last":0.66700000000000004}]},{"osd":3,"last update":"Wed Feb 10 04:53:03 2021","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":1.099,"15min":1.143},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.4279999999999999,"5min":4.2380000000000004,"15min":4.2380000000000004},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.05,"5min":1.274,"15min":1.3120000000000001},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.607,"5min":4.54,"15min":4.54},"last":0.96099999999999997}]},{"osd":4,"last update":"Wed Feb 10 04:53:31 2021","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":1.1839999999999999,"15min":1.321},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.742,"5min":4.1310000000000002,"15min":4.3319999999999999},"last":1.0349999999999999},{"interface":"front","average":{"1min":0.90300000000000002,"5min":1.022,"15min":1.121},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.323,"5min":3.411,"15min":4.4489999999999998},"last":1.339}]},{"osd":5,"last update":"Wed Feb 10 04:53:53 2021","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.744,"15min":1.7090000000000001},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":2.4260000000000002,"5min":33.018999999999998,"15min":33.018999999999998},"last":1.1930000000000001},{"interface":"front","average":{"1min":0.88600000000000001,"5min":1.526,"15min":1.476},"min":{"1min":0.65900000000000003,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.2230000000000001,"5min":33.043999999999997,"15min":33.043999999999997},"last":0.97799999999999998}]},{"osd":6,"last update":"Wed Feb 10 04:53:20 2021","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":1.2350000000000001,"15min":1.498},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.46,"5min":2.4609999999999999,"15min":20.007000000000001},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.28,"5min":1.4530000000000001,"15min":1.6830000000000001},"min":{"1min":0.92000000000000004,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.8680000000000001,"5min":4.415,"15min":19.417999999999999},"last":1.4370000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:53:31 2021","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.4770000000000001,"15min":1.6459999999999999},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":1.8839999999999999,"5min":4.6699999999999999,"15min":4.6699999999999999},"last":1.379},{"interface":"front","average":{"1min":1.042,"5min":1.155,"15min":1.3129999999999999},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.4159999999999999,"5min":1.9239999999999999,"15min":4.9189999999999996},"last":1.101}]}]},{"osd":3,"up_from":1111,"seq":4771708665895,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":326708,"kb_used_data":300460,"kb_used_omap":386,"kb_used_meta":25853,"kb_avail":93418444,"statfs":{"total":95995035648,"available":95660486656,"internally_reserved":0,"allocated":307671040,"data_stored":409335718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":396036,"internal_metadata":26473724},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:58 2021","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":0.67200000000000004}]},{"osd":1,"last update":"Wed Feb 10 04:54:03 2021","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65700000000000003,"15min":0.65100000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.8360000000000001,"5min":1.8360000000000001,"15min":1.8360000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.83299999999999996,"15min":0.873},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.641,"5min":2.137,"15min":2.137},"last":0.54600000000000004}]},{"osd":2,"last update":"Wed Feb 10 04:54:03 2021","interfaces":[{"interface":"back","average":{"1min":1.8180000000000001,"5min":1.208,"15min":1.169},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":23.844999999999999,"5min":23.844999999999999,"15min":23.844999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":1.9970000000000001,"5min":1.2669999999999999,"15min":1.169},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":23.783999999999999,"5min":23.783999999999999,"15min":23.783999999999999},"last":0.755}]},{"osd":4,"last update":"Wed Feb 10 04:54:03 2021","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.96499999999999997,"15min":0.96899999999999997},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":2.9430000000000001,"5min":2.9430000000000001,"15min":2.9430000000000001},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.92000000000000004,"5min":1.042,"15min":1.1000000000000001},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.357,"5min":1.8999999999999999,"15min":1.8999999999999999},"last":0.67700000000000005}]},{"osd":5,"last update":"Wed Feb 10 04:54:03 2021","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":2.6379999999999999,"15min":2.7109999999999999},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":2.0979999999999999,"5min":52.353000000000002,"15min":52.353000000000002},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.0660000000000001,"5min":2.4929999999999999,"15min":2.5259999999999998},"min":{"1min":0.71399999999999997,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":3.544,"5min":52.301000000000002,"15min":52.301000000000002},"last":0.96199999999999997}]},{"osd":6,"last update":"Wed Feb 10 04:54:03 2021","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":4.4249999999999998,"15min":5.2930000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.474,"5min":64.376999999999995,"15min":64.376999999999995},"last":1.097},{"interface":"front","average":{"1min":1.228,"5min":4.4640000000000004,"15min":5.2779999999999996},"min":{"1min":0.85599999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":1.8080000000000001,"5min":64.438999999999993,"15min":64.438999999999993},"last":1.0529999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:54:03 2021","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.087,"15min":1.113},"min":{"1min":0.79800000000000004,"5min":0.754,"15min":0.754},"max":{"1min":1.466,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":1.034},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.137,"15min":1.141},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.5649999999999999,"5min":2.044,"15min":2.044},"last":1.159}]}]},{"osd":4,"up_from":31,"seq":133143986532,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":158732,"kb_used_data":119684,"kb_used_omap":0,"kb_used_meta":39040,"kb_avail":93586420,"statfs":{"total":95995035648,"available":95832494080,"internally_reserved":0,"allocated":122556416,"data_stored":155236997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":39976960},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:36 2021","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.0899999999999999,"15min":1.593},"min":{"1min":0.45800000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.7799999999999998,"5min":95.721999999999994,"15min":95.721999999999994},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.67700000000000005,"5min":2.0870000000000002,"15min":1.6000000000000001},"min":{"1min":0.46000000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.002,"5min":95.344999999999999,"15min":95.344999999999999},"last":0.70599999999999996}]},{"osd":1,"last update":"Wed Feb 10 04:53:42 2021","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.84999999999999998,"15min":0.95599999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.1459999999999999,"5min":7.883,"15min":24.295000000000002},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.82099999999999995,"5min":1.04,"15min":1.218},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.5369999999999999,"5min":3.8090000000000002,"15min":24.943000000000001},"last":0.68000000000000005}]},{"osd":2,"last update":"Wed Feb 10 04:53:42 2021","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":2.5990000000000002,"15min":1.677},"min":{"1min":0.624,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.468,"5min":120.39,"15min":120.39},"last":0.88},{"interface":"front","average":{"1min":0.83599999999999997,"5min":2.5449999999999999,"15min":1.6739999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.51900000000000002},"max":{"1min":1.1599999999999999,"5min":120.583,"15min":120.583},"last":0.91500000000000004}]},{"osd":3,"last update":"Wed Feb 10 04:54:01 2021","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.21,"15min":1.2749999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.6339999999999999,"5min":2.1429999999999998,"15min":2.1429999999999998},"last":0.99299999999999999},{"interface":"front","average":{"1min":2.8519999999999999,"5min":1.7490000000000001,"15min":1.603},"min":{"1min":0.65200000000000002,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":26.802,"5min":26.802,"15min":26.802},"last":1.0760000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:53:56 2021","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.28,"15min":1.516},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":2.351,"5min":8.1509999999999998,"15min":35.774000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":1.276,"5min":1.282,"15min":1.4810000000000001},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.58799999999999997},"max":{"1min":4.319,"5min":7.9950000000000001,"15min":35.606000000000002},"last":1.3520000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:54:01 2021","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":3.5539999999999998,"15min":2.3940000000000001},"min":{"1min":0.873,"5min":0.86199999999999999,"15min":0.76600000000000001},"max":{"1min":1.8380000000000001,"5min":201.626,"15min":201.626},"last":1.296},{"interface":"front","average":{"1min":1.125,"5min":3.476,"15min":2.4279999999999999},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.7529999999999999,"5min":201.577,"15min":201.577},"last":1.0409999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:53:29 2021","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.5009999999999999,"15min":1.508},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.2210000000000001,"5min":4.7439999999999998,"15min":6.9199999999999999},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.379,"15min":1.5469999999999999},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":2.198,"5min":2.8010000000000002,"15min":5.149},"last":1.2210000000000001}]}]},{"osd":5,"up_from":37,"seq":158913790305,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":250336,"kb_used_data":205464,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":93494816,"statfs":{"total":95995035648,"available":95738691584,"internally_reserved":0,"allocated":210395136,"data_stored":275310565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:51 2021","interfaces":[{"interface":"back","average":{"1min":5.0590000000000002,"5min":1.714,"15min":1.204},"min":{"1min":0.45700000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":58.624000000000002,"5min":58.624000000000002,"15min":58.624000000000002},"last":0.64600000000000002},{"interface":"front","average":{"1min":5.04,"5min":1.784,"15min":1.3009999999999999},"min":{"1min":0.56599999999999995,"5min":0.46600000000000003,"15min":0.46400000000000002},"max":{"1min":57.880000000000003,"5min":57.880000000000003,"15min":57.880000000000003},"last":1.284}]},{"osd":1,"last update":"Wed Feb 10 04:53:52 2021","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":0.93400000000000005,"15min":0.95399999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":6.391,"5min":6.391,"15min":6.391},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.72899999999999998,"5min":1.1499999999999999,"15min":1.175},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.2569999999999999,"5min":11.818,"15min":11.875999999999999},"last":0.77100000000000002}]},{"osd":2,"last update":"Wed Feb 10 04:53:52 2021","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":1.641,"15min":1.5369999999999999},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.2430000000000001,"5min":50.948999999999998,"15min":50.948999999999998},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.7490000000000001,"15min":1.5760000000000001},"min":{"1min":0.61899999999999999,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":6.7130000000000001,"5min":50.149000000000001,"15min":50.149000000000001},"last":0.747}]},{"osd":3,"last update":"Wed Feb 10 04:54:06 2021","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.2030000000000001,"15min":1.2909999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.357,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.284,"5min":1.2070000000000001,"15min":1.1890000000000001},"min":{"1min":0.73199999999999998,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":6.5129999999999999,"5min":6.5129999999999999,"15min":6.5129999999999999},"last":0.85899999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:53:52 2021","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.329,"15min":1.3320000000000001},"min":{"1min":0.70399999999999996,"5min":0.64500000000000002,"15min":0.57099999999999995},"max":{"1min":2.028,"5min":9.4890000000000008,"15min":9.4890000000000008},"last":0.67500000000000004},{"interface":"front","average":{"1min":1.117,"5min":1.3300000000000001,"15min":1.385},"min":{"1min":0.69099999999999995,"5min":0.68100000000000005,"15min":0.67500000000000004},"max":{"1min":2.3149999999999999,"5min":5.6180000000000003,"15min":5.6180000000000003},"last":0.90000000000000002}]},{"osd":6,"last update":"Wed Feb 10 04:54:06 2021","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.9379999999999999,"15min":3.0150000000000001},"min":{"1min":0.93100000000000005,"5min":0.83899999999999997,"15min":0.65800000000000003},"max":{"1min":2.7829999999999999,"5min":38.014000000000003,"15min":319.51999999999998},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.9019999999999999,"15min":2.9409999999999998},"min":{"1min":0.79300000000000004,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":6.8220000000000001,"5min":37.896999999999998,"15min":319.22300000000001},"last":0.96199999999999997}]},{"osd":7,"last update":"Wed Feb 10 04:53:33 2021","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.5,"15min":1.5229999999999999},"min":{"1min":0.93000000000000005,"5min":0.75600000000000001,"15min":0.752},"max":{"1min":1.4990000000000001,"5min":11.897,"15min":11.897},"last":0.55500000000000005},{"interface":"front","average":{"1min":1.395,"5min":1.5660000000000001,"15min":1.6060000000000001},"min":{"1min":0.94999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.431,"5min":6.0369999999999999,"15min":6.0369999999999999},"last":1.236}]}]}]}} 2021-02-10T04:54:11.251 INFO:tasks.thrashosds.thrasher:Setting 4.5 to [0, 6, 4, 2, 3, 1] 2021-02-10T04:54:11.251 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '4.5', '0', '6', '4', '2', '3', '1'] 2021-02-10T04:54:11.252 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pg-upmap-items 4.5 0 6 4 2 3 1 2021-02-10T04:54:11.467 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:54:12.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:11 smithi164 conmon[29288]: cluster 2021-02-10T04:54:10.467590+0000 mgr.y (mgr.14140 2021-02-10T04:54:12.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:11 smithi164 conmon[29288]: ) 1315 : cluster [DBG] pgmap v2164: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:12.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:11 smithi164 conmon[29288]: cluster 2021-02-10T04:54:10.899649+0000 osd.5 (osd.5) 187 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:54:12.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:11 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:54:12.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:11 smithi164 conmon[29288]: 04:54:10.901683+0000 osd.5 (osd.5) 188 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:54:12.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[29319]: cluster 2021-02-10T04:54:10.467590+0000 mgr.y (mgr.14140) 1315 : cluster [DBG] pgmap v2164: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:12.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[29319]: cluster 2021-02-10T04:54:10.899649+0000 osd.5 (osd.5) 187 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:54:12.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[29319]: cluster 2021-02-10T04:54:10.901683+0000 osd.5 (osd.5) 188 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:54:12.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: cluster 2021-02-10T04:54:12.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 2021-02-10T04:54:12.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 2021 2021-02-10T04:54:12.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: - 2021-02-10T04:54:12.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 02 2021-02-10T04:54:12.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: - 2021-02-10T04:54:12.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 10 2021-02-10T04:54:12.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: T 2021-02-10T04:54:12.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 04 2021-02-10T04:54:12.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: : 2021-02-10T04:54:12.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 54 2021-02-10T04:54:12.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: : 2021-02-10T04:54:12.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 10 2021-02-10T04:54:12.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: . 2021-02-10T04:54:12.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 467590 2021-02-10T04:54:12.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: +0000 2021-02-10T04:54:12.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 2021-02-10T04:54:12.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: mgr.y 2021-02-10T04:54:12.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: ( 2021-02-10T04:54:12.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: mgr 2021-02-10T04:54:12.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: . 2021-02-10T04:54:12.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 14140 2021-02-10T04:54:12.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: ) 2021-02-10T04:54:12.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 1315 2021-02-10T04:54:12.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: : 2021-02-10T04:54:12.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: cluster 2021-02-10T04:54:12.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 2021-02-10T04:54:12.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: [DBG] 2021-02-10T04:54:12.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 2021-02-10T04:54:12.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: pgmap v2164: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:12.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 2021-02-10T04:54:12.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: cluster 2021-02-10T04:54:12.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 2021-02-10T04:54:12.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 2021 2021-02-10T04:54:12.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: - 2021-02-10T04:54:12.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 02 2021-02-10T04:54:12.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: - 2021-02-10T04:54:12.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 10 2021-02-10T04:54:12.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: T 2021-02-10T04:54:12.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 04 2021-02-10T04:54:12.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: : 2021-02-10T04:54:12.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 54 2021-02-10T04:54:12.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: : 2021-02-10T04:54:12.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 10 2021-02-10T04:54:12.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: . 2021-02-10T04:54:12.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 899649 2021-02-10T04:54:12.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: +0000 2021-02-10T04:54:12.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 2021-02-10T04:54:12.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: osd.5 2021-02-10T04:54:12.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: ( 2021-02-10T04:54:12.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: osd 2021-02-10T04:54:12.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: . 2021-02-10T04:54:12.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 5 2021-02-10T04:54:12.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: ) 2021-02-10T04:54:12.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 187 2021-02-10T04:54:12.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: : 2021-02-10T04:54:12.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: cluster 2021-02-10T04:54:12.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 2021-02-10T04:54:12.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: [DBG] 2021-02-10T04:54:12.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 2021-02-10T04:54:12.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: purged_snaps scrub starts 2021-02-10T04:54:12.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 2021-02-10T04:54:12.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: cluster 2021-02-10T04:54:12.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 2021-02-10T04:54:12.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 2021 2021-02-10T04:54:12.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: - 2021-02-10T04:54:12.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 02 2021-02-10T04:54:12.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: - 2021-02-10T04:54:12.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 10 2021-02-10T04:54:12.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: T 2021-02-10T04:54:12.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 04 2021-02-10T04:54:12.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: : 2021-02-10T04:54:12.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: 54 2021-02-10T04:54:12.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: :10.901683+0000 2021-02-10T04:54:12.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:11 smithi085 conmon[35961]: osd.5 (osd.5) 188 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:54:12.359 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:54:12.359 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:54:12.936 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:12 smithi085 conmon[29319]: cluster 2021-02-10T04:54:11.769450+0000 2021-02-10T04:54:12.936 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:12 smithi085 conmon[29319]: mon.a (mon.0) 2557 : cluster [DBG] osdmap e1199: 8 total, 7 up, 8 in 2021-02-10T04:54:12.936 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:12 smithi085 conmon[29319]: cephadm 2021-02-10T04:54:11.915100+0000 mgr.y (mgr.14140) 1316 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:54:12.936 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:12 smithi085 conmon[29319]: cephadm 2021-02-10T04:54:11.915465+0000 mgr.y (mgr.14140) 1317 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:54:12.936 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:12 smithi085 conmon[35961]: cluster 2021-02-10T04:54:11.769450+0000 mon.a (mon 2021-02-10T04:54:12.936 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:12 smithi085 conmon[35961]: .0) 2557 : cluster [DBG] osdmap e1199: 8 total, 7 up, 8 in 2021-02-10T04:54:12.936 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:12 smithi085 conmon[35961]: cephadm 2021-02-10T04:54:11.915100+0000 mgr.y (mgr.14140) 1316 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:54:12.937 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:12 smithi085 conmon[35961]: cephadm 2021-02-10T04:54:11.915465+0000 mgr.y (mgr.14140) 1317 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:54:13.140 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset nodeep-scrub 2021-02-10T04:54:13.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:12 smithi164 conmon[29288]: cluster 2021-02-10T04:54:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:12 smithi164 conmon[29288]: 2021-02-10T04:54:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:12 smithi164 conmon[29288]: 2021 2021-02-10T04:54:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:12 smithi164 conmon[29288]: -02-10T04:54:11.769450 2021-02-10T04:54:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:12 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2557 : cluster [DBG] osdmap e1199: 8 total, 7 up, 8 in 2021-02-10T04:54:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:12 smithi164 conmon[29288]: cephadm 2021-02-10T04:54:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:12 smithi164 conmon[29288]: 2021-02-10T04:54:11.915100+0000 2021-02-10T04:54:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:12 smithi164 conmon[29288]: mgr.y (mgr.14140) 1316 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:54:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:12 smithi164 conmon[29288]: cephadm 2021- 2021-02-10T04:54:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:12 smithi164 conmon[29288]: 02-10T04:54:11.915465+0000 2021-02-10T04:54:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:12 smithi164 conmon[29288]: mgr.y (mgr.14140) 1317 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:54:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:12 smithi164 conmon[29288]: 2021-02-10T04:54:13.425 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:54:14.360 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:54:14.385 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:14 smithi085 conmon[35961]: cluster 2021-02-10T04:54:12. 2021-02-10T04:54:14.385 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:14 smithi085 conmon[35961]: 468239+0000 mgr.y (mgr.14140) 1318 : cluster [DBG] pgmap v2166: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:14.385 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:14 smithi085 conmon[35961]: audit 2021-02-10T04:54:14.385 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:14 smithi085 conmon[35961]: 2021-02-10T04:54:13.058689+0000 mon.a (mon. 2021-02-10T04:54:14.385 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:14 smithi085 conmon[35961]: 0) 2558 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:54:14.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:14 smithi085 conmon[29319]: cluster 2021-02-10T04:54:12.468239+0000 mgr.y (mgr.14140) 1318 : cluster 2021-02-10T04:54:14.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:14 smithi085 conmon[29319]: [DBG] pgmap v2166: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:14.386 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:14 smithi085 conmon[29319]: audit 2021-02-10T04:54:13.058689+0000 mon.a (mon.0) 2558 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:54:14.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:14 smithi164 conmon[29288]: cluster 2021-02-10T04:54:12.468239+0000 mgr.y (mgr.14140) 1318 : cluster 2021-02-10T04:54:14.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:14 smithi164 conmon[29288]: [DBG] pgmap v2166: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:14.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:14 smithi164 conmon[29288]: audit 2021-02-10T04:54:13.058689+0000 mon.a (mon.0) 2558 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:54:14.616 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:54:16.073 INFO:teuthology.orchestra.run.smithi085.stderr:set 4.5 pg_upmap_items mapping to [0->6,4->2,3->1] 2021-02-10T04:54:16.105 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:16 smithi085 conmon[35961]: cluster 2021-02-10T04:54:14.469520+0000 mgr.y (mgr.14140) 1319 : cluster [DBG] pgmap v2167: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:16.105 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:16 smithi085 conmon[35961]: audit 2021-02-10T04:54:15.408397+0000 mon.a (mon.0) 2559 : audit [INF] from='client.? 172.21.15.85:0/2678582595' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.5", "id": [0, 6, 4, 2, 3, 1]}]: dispatch 2021-02-10T04:54:16.105 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:16 smithi085 conmon[29319]: cluster 2021-02-10T04:54:14.469520+0000 mgr.y (mgr.14140) 1319 : cluster [DBG] pgmap v2167: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:16.105 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:16 smithi085 conmon[29319]: audit 2021-02-10T04:54:15.408397+0000 mon.a (mon.0) 2559 : audit [INF] from='client.? 172.21.15.85:0/2678582595' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.5", "id": [0, 6, 4, 2, 3, 1]}]: dispatch 2021-02-10T04:54:16.128 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:54:16.128 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [], 2021-02-10T04:54:16.128 INFO:teuthology.orchestra.run.smithi085.stdout: "complaint_time": 30, 2021-02-10T04:54:16.128 INFO:teuthology.orchestra.run.smithi085.stdout: "num_blocked_ops": 0 2021-02-10T04:54:16.128 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:54:16.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:16 smithi164 conmon[29288]: cluster 2021-02-10T04:54:14.469520+0000 mgr.y (mgr. 2021-02-10T04:54:17.103 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:16 smithi164 conmon[29288]: 14140) 1319 : cluster [DBG] pgmap v2167: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:17.103 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:16 smithi164 conmon[29288]: audit 2021-02-10T04:54:15.408397+0000 mon.a (mon.0) 2021-02-10T04:54:17.103 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:16 smithi164 conmon[29288]: 2559 : audit [INF] from='client.? 172.21.15.85:0/2678582595' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.5", "id": [0, 6, 4, 2, 3, 1]}]: dispatch 2021-02-10T04:54:17.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:17 smithi085 conmon[29319]: audit 2021-02-10T04:54:16.066082+0000 mon.a (mon.0 2021-02-10T04:54:17.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:17 smithi085 conmon[29319]: ) 2560 : audit [INF] from='client.? 172.21.15.85:0/2678582595' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.5", "id": [0, 6, 4, 2, 3, 1]}]': finished 2021-02-10T04:54:17.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:17 smithi085 conmon[29319]: cluster 2021-02-10T04:54:16.066211+0000 mon.a (mon.0) 2561 : cluster [DBG] osdmap e1200: 8 total, 7 up, 8 in 2021-02-10T04:54:17.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:17 smithi085 conmon[35961]: audit 2021-02-10T04:54:16.066082 2021-02-10T04:54:17.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:17 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2560 : audit [INF] from='client.? 172.21.15.85:0/2678582595' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.5", "id": [0, 6, 4, 2, 3, 1]}]': finished 2021-02-10T04:54:17.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:17 smithi085 conmon[35961]: cluster 2021-02-10T04:54: 2021-02-10T04:54:17.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:17 smithi085 conmon[35961]: 16.066211+0000 mon.a (mon.0) 2561 : cluster [DBG] osdmap e1200: 8 total, 7 up, 8 in 2021-02-10T04:54:17.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:17 smithi164 conmon[29288]: audit 2021-02-10T04:54:16.066082+0000 mon.a (mon.0) 2560 : 2021-02-10T04:54:17.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:17 smithi164 conmon[29288]: audit [INF] from='client.? 172.21.15.85:0/2678582595' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.5", "id": [0, 6, 4, 2, 3, 1]}]': finished 2021-02-10T04:54:17.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:17 smithi164 conmon[29288]: cluster 2021-02-10T04:54:16.066211+0000 mon.a (mon.0) 2561 : cluster [DBG] osdmap e1200: 8 total, 7 up, 8 in 2021-02-10T04:54:17.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:17 smithi164 conmon[29288]: 2021-02-10T04:54:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:18 smithi164 conmon[29288]: cluster 2021-02-10T04:54:16.470149+0000 mgr.y (mgr.14140) 1320 : 2021-02-10T04:54:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:18 smithi164 conmon[29288]: cluster [DBG] pgmap v2169: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 763 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:18 smithi164 conmon[29288]: cluster 2021-02-10T04:54:17.072635+0000 mon.a (mon.0) 2562 : cluster [DBG] osdmap e1201: 8 total, 7 up, 8 in 2021-02-10T04:54:18.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:18 smithi085 conmon[35961]: cluster 2021-02-10T04:54:16.470149+0000 mgr.y (mgr.14140) 2021-02-10T04:54:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:18 smithi085 conmon[35961]: 1320 : cluster [DBG] pgmap v2169: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 763 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:18 smithi085 conmon[35961]: cluster 2021-02-10T04:54:17.072635+0000 mon.a (mon.0) 2562 : cluster [DBG] osdmap e1201: 8 total, 7 up, 8 in 2021-02-10T04:54:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:18 smithi085 conmon[29319]: cluster 2021-02-10T04:54:16. 2021-02-10T04:54:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:18 smithi085 conmon[29319]: 470149+0000 mgr.y (mgr.14140) 1320 : cluster [DBG] pgmap v2169: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 763 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:18 smithi085 conmon[29319]: cluster 2021-02-10T04:54:17.072635+0000 mon.a ( 2021-02-10T04:54:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:18 smithi085 conmon[29319]: mon.0) 2562 : cluster [DBG] osdmap e1201: 8 total, 7 up, 8 in 2021-02-10T04:54:18.905 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.2 dump_historic_ops 2021-02-10T04:54:19.184 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:54:19.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:19 smithi085 conmon[29319]: cluster 2021-02-10T04:54:17.516283+0000 osd.2 (osd.2) 2027 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:54:19.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:19 smithi085 conmon[29319]: cluster 2021-02-10T04:54:17.518881+0000 osd.2 (osd.2) 2028 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:54:19.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:19 smithi085 conmon[29319]: cluster 2021-02-10T04:54:18.075324+0000 mon.a (mon.0) 2563 : cluster [DBG] osdmap e1202: 8 total, 7 up, 8 in 2021-02-10T04:54:19.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:19 smithi085 conmon[29319]: cluster 2021-02-10T04:54:18.534668+0000 mon.a (mon.0) 2564 : cluster [DBG] osdmap e1203: 8 total, 7 up, 8 in 2021-02-10T04:54:19.343 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:19 smithi085 conmon[35961]: cluster 2021-02-10T04:54:17.516283+0000 osd.2 (osd.2) 2027 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:54:19.343 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:19 smithi085 conmon[35961]: cluster 2021-02-10T04:54:17.518881+0000 osd.2 (osd.2) 2028 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:54:19.343 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:19 smithi085 conmon[35961]: cluster 2021-02-10T04:54:18.075324+0000 mon.a (mon.0) 2563 : cluster [DBG] osdmap e1202: 8 total, 7 up, 8 in 2021-02-10T04:54:19.343 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:19 smithi085 conmon[35961]: cluster 2021-02-10T04:54:18.534668+0000 mon.a (mon.0) 2564 : cluster [DBG] osdmap e1203: 8 total, 7 up, 8 in 2021-02-10T04:54:19.394 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:54:19 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:54:19.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:19 smithi164 conmon[29288]: cluster 2021-02-10T04:54:19.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:19 smithi164 conmon[29288]: 2021-02-10T04:54:19.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:19 smithi164 conmon[29288]: 2021 2021-02-10T04:54:19.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:19 smithi164 conmon[29288]: - 2021-02-10T04:54:19.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:19 smithi164 conmon[29288]: 02-10T04:54:17.516283 2021-02-10T04:54:19.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:19 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 2027 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:54:19.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:19 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:54:19.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:19 smithi164 conmon[29288]: -10T04:54:17.518881+0000 osd.2 ( 2021-02-10T04:54:19.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:19 smithi164 conmon[29288]: osd.2) 2028 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:54:19.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:19 smithi164 conmon[29288]: cluster 2021-02-10T04:54:19.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:19 smithi164 conmon[29288]: 2021-02-10T04:54:18.075324 2021-02-10T04:54:19.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:19 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2563 : cluster [DBG] 2021-02-10T04:54:19.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:19 smithi164 conmon[29288]: osdmap e1202: 8 total, 7 up, 8 in 2021-02-10T04:54:19.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:19 smithi164 conmon[29288]: cluster 2021-02-10T04:54: 2021-02-10T04:54:19.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:19 smithi164 conmon[29288]: 18.534668+0000 mon.a (mon.0) 2564 2021-02-10T04:54:19.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:19 smithi164 conmon[29288]: : cluster [DBG] osdmap e1203: 8 total, 7 up, 8 in 2021-02-10T04:54:19.664 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:54:19.689 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:54:19.709 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:54:19.727 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:54:19.749 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:54:19.778 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:54:19.809 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:54:19.809 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-02-10T04:54:19.809 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: problem getting command descriptions from osd.0 2021-02-10T04:54:19.809 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:54:19.809 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:54:19.810 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:54:19.810 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:54:19.810 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:54:19.810 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:54:19.810 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:54:20.087 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is unset 2021-02-10T04:54:20.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:20 smithi164 conmon[29288]: cluster 2021-02-10T04:54:18.470895+0000 mgr.y 2021-02-10T04:54:20.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:20 smithi164 conmon[29288]: (mgr.14140) 1321 : cluster [DBG] pgmap v2172: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:20.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:20 smithi164 conmon[29288]: audit 2021-02-10T04:54:19.245056+0000 mgr.y (mgr.14140) 1322 : audit 2021-02-10T04:54:20.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:20 smithi164 conmon[29288]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:54:20.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:20 smithi164 conmon[29288]: audit 2021-02-10T04:54:20.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:20 smithi164 conmon[29288]: 2021-02-10T04:54:19.635310+0000 mon.c (mon.1) 114 : 2021-02-10T04:54:20.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:20 smithi164 conmon[29288]: audit [DBG] from='client.? 172.21.15.85:0/2948217370' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:54:20.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:20 smithi164 conmon[29288]: audit 2021-02-10T04:54:20.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:20 smithi164 conmon[29288]: 2021-02-10T04:54:19.771687+0000 mon.a (mon.0) 2565 : audit [INF] 2021-02-10T04:54:20.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:20 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/424754751' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:54:20.397 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[29319]: cluster 2021-02-10T04:54:18.470895+0000 mgr.y ( 2021-02-10T04:54:20.397 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[29319]: mgr.14140) 1321 : cluster [DBG] pgmap v2172: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:20.397 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[29319]: audit 2021-02-10T04:54:19.245056+0000 mgr.y (mgr.14140) 1322 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:54:20.397 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[29319]: audit 2021-02-10T04:54:19.635310+0000 mon.c (mon.1) 114 : audit [DBG] from='client.? 172.21.15.85:0/2948217370' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:54:20.398 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[29319]: audit 2021-02-10T04:54:19.771687+0000 mon.a (mon.0) 2565 : audit [INF] from='client.? 172.21.15.85:0/424754751' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:54:20.398 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: cluster 2021-02-10T04:54:20.398 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 2021-02-10T04:54:20.398 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 2021 2021-02-10T04:54:20.398 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: - 2021-02-10T04:54:20.398 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 02 2021-02-10T04:54:20.399 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: - 2021-02-10T04:54:20.399 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 10 2021-02-10T04:54:20.399 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: T 2021-02-10T04:54:20.399 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 04 2021-02-10T04:54:20.399 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: : 2021-02-10T04:54:20.399 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 54 2021-02-10T04:54:20.399 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: : 2021-02-10T04:54:20.399 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 18 2021-02-10T04:54:20.399 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: . 2021-02-10T04:54:20.399 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 470895 2021-02-10T04:54:20.399 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: +0000 2021-02-10T04:54:20.399 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 2021-02-10T04:54:20.399 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: mgr.y 2021-02-10T04:54:20.399 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: ( 2021-02-10T04:54:20.400 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: mgr 2021-02-10T04:54:20.400 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: . 2021-02-10T04:54:20.400 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 14140 2021-02-10T04:54:20.400 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: ) 2021-02-10T04:54:20.400 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 1321 2021-02-10T04:54:20.400 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: : 2021-02-10T04:54:20.400 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: cluster 2021-02-10T04:54:20.400 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 2021-02-10T04:54:20.400 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: [DBG] 2021-02-10T04:54:20.400 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 2021-02-10T04:54:20.400 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: pgmap v2172: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:20.400 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 2021-02-10T04:54:20.400 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: audit 2021-02-10T04:54:20.400 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 2021-02-10T04:54:20.401 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 2021 2021-02-10T04:54:20.401 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: - 2021-02-10T04:54:20.401 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 02 2021-02-10T04:54:20.401 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: - 2021-02-10T04:54:20.401 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 10 2021-02-10T04:54:20.401 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: T 2021-02-10T04:54:20.401 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 04 2021-02-10T04:54:20.401 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: : 2021-02-10T04:54:20.401 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 54 2021-02-10T04:54:20.401 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: : 2021-02-10T04:54:20.401 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 19 2021-02-10T04:54:20.401 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: . 2021-02-10T04:54:20.401 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 245056 2021-02-10T04:54:20.401 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: +0000 2021-02-10T04:54:20.402 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 2021-02-10T04:54:20.402 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: mgr.y 2021-02-10T04:54:20.402 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: ( 2021-02-10T04:54:20.402 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: mgr 2021-02-10T04:54:20.402 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: . 2021-02-10T04:54:20.402 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 14140 2021-02-10T04:54:20.402 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: ) 2021-02-10T04:54:20.402 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 1322 2021-02-10T04:54:20.402 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: : 2021-02-10T04:54:20.402 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: audit 2021-02-10T04:54:20.402 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 2021-02-10T04:54:20.402 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: [DBG] 2021-02-10T04:54:20.402 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 2021-02-10T04:54:20.402 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:54:20.403 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 2021-02-10T04:54:20.403 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: audit 2021-02-10T04:54:20.403 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 2021-02-10T04:54:20.403 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 2021 2021-02-10T04:54:20.403 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: - 2021-02-10T04:54:20.403 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 02 2021-02-10T04:54:20.403 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: - 2021-02-10T04:54:20.403 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 10 2021-02-10T04:54:20.403 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: T 2021-02-10T04:54:20.403 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 04 2021-02-10T04:54:20.403 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: : 2021-02-10T04:54:20.403 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 54 2021-02-10T04:54:20.403 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: : 2021-02-10T04:54:20.403 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 19 2021-02-10T04:54:20.404 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: . 2021-02-10T04:54:20.404 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 635310 2021-02-10T04:54:20.404 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: +0000 2021-02-10T04:54:20.404 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 2021-02-10T04:54:20.404 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: mon.c 2021-02-10T04:54:20.404 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: ( 2021-02-10T04:54:20.404 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: mon 2021-02-10T04:54:20.404 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: . 2021-02-10T04:54:20.404 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 1 2021-02-10T04:54:20.405 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: ) 2021-02-10T04:54:20.405 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 114 2021-02-10T04:54:20.405 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: : 2021-02-10T04:54:20.405 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: audit 2021-02-10T04:54:20.405 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 2021-02-10T04:54:20.405 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: [DBG] 2021-02-10T04:54:20.405 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 2021-02-10T04:54:20.405 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/2948217370' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:54:20.405 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 2021-02-10T04:54:20.405 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: audit 2021-02-10T04:54:20.405 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 2021-02-10T04:54:20.405 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 2021 2021-02-10T04:54:20.406 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: - 2021-02-10T04:54:20.406 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 02 2021-02-10T04:54:20.406 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: - 2021-02-10T04:54:20.406 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 10 2021-02-10T04:54:20.406 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: T 2021-02-10T04:54:20.406 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 04 2021-02-10T04:54:20.406 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: : 2021-02-10T04:54:20.406 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 54 2021-02-10T04:54:20.406 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: : 2021-02-10T04:54:20.406 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 19 2021-02-10T04:54:20.406 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: . 2021-02-10T04:54:20.406 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 771687 2021-02-10T04:54:20.406 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: +0000 2021-02-10T04:54:20.406 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 2021-02-10T04:54:20.407 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: mon.a 2021-02-10T04:54:20.407 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: ( 2021-02-10T04:54:20.407 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: mon 2021-02-10T04:54:20.407 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: . 2021-02-10T04:54:20.407 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 0 2021-02-10T04:54:20.407 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: ) 2021-02-10T04:54:20.407 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:20 smithi085 conmon[35961]: 2565 : audit [INF] from='client.? 172.21.15.85:0/424754751' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:54:21.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:21 smithi164 conmon[29288]: cluster 2021-02-10T04:54:20.077791+0000 mon.a (mon.0) 2566 : 2021-02-10T04:54:21.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:21 smithi164 conmon[29288]: cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:54:21.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:21 smithi164 conmon[29288]: audit 2021-02-10T04:54:20.082273+0000 mon.a (mon.0) 2567 : audit [INF] from='client.? 172.21.15.85:0/424754751' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:54:21.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:21 smithi164 conmon[29288]: cluster 2021-02-10T04:54:20.082336+0000 mon.a 2021-02-10T04:54:21.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:21 smithi164 conmon[29288]: (mon.0) 2568 : cluster [DBG] osdmap e1204: 8 total, 7 up, 8 in 2021-02-10T04:54:21.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:21 smithi085 conmon[29319]: cluster 2021-02-10T04:54:20.077791+0000 mon.a (mon.0) 2566 : cluster 2021-02-10T04:54:21.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:21 smithi085 conmon[29319]: [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:54:21.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:21 smithi085 conmon[29319]: audit 2021-02-10T04:54:20.082273+0000 mon.a (mon.0) 2567 : audit [INF] from='client.? 172.21.15.85:0/424754751' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:54:21.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:21 smithi085 conmon[29319]: cluster 2021-02-10T04:54:21.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:21 smithi085 conmon[29319]: 2021-02-10T04:54:20.082336+0000 mon.a (mon. 2021-02-10T04:54:21.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:21 smithi085 conmon[29319]: 0) 2568 : cluster [DBG] osdmap e1204: 8 total, 7 up, 8 in 2021-02-10T04:54:21.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:21 smithi085 conmon[35961]: cluster 2021-02-10T04:54:20.077791 2021-02-10T04:54:21.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:21 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2566 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:54:21.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:21 smithi085 conmon[35961]: audit 2021-02-10T04:54:20.082273+0000 mon.a 2021-02-10T04:54:21.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:21 smithi085 conmon[35961]: (mon.0) 2567 : audit [INF] from='client.? 172.21.15.85:0/424754751' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:54:21.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:21 smithi085 conmon[35961]: cluster 2021-02-10T04:54:21.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:21 smithi085 conmon[35961]: 2021-02-10T04:54:20.082336+0000 2021-02-10T04:54:21.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:21 smithi085 conmon[35961]: mon.a (mon.0) 2568 : cluster [DBG] 2021-02-10T04:54:21.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:21 smithi085 conmon[35961]: osdmap e1204: 8 total, 7 up, 8 in 2021-02-10T04:54:22.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:22 smithi085 conmon[35961]: cluster 2021-02-10T04:54:20.471555+0000 2021-02-10T04:54:22.350 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:22 smithi085 conmon[35961]: mgr.y (mgr.14140) 1323 : cluster [DBG] pgmap v2175: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 116 B/s, 2 objects/s recovering 2021-02-10T04:54:22.350 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:22 smithi085 conmon[29319]: cluster 2021-02-10T04:54:20.471555+0000 mgr.y ( 2021-02-10T04:54:22.351 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:22 smithi085 conmon[29319]: mgr.14140) 1323 : cluster [DBG] pgmap v2175: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 116 B/s, 2 objects/s recovering 2021-02-10T04:54:22.351 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:54:22.351 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:54:22.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:22 smithi164 conmon[29288]: cluster 2021-02-10T04:54:20.471555+0000 mgr.y (mgr. 2021-02-10T04:54:22.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:22 smithi164 conmon[29288]: 14140) 1323 : cluster [DBG] pgmap v2175: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 116 B/s, 2 objects/s recovering 2021-02-10T04:54:22.785 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:54:22.785 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:54:23.773 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.3 dump_ops_in_flight 2021-02-10T04:54:23.866 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2021-02-10T04:54:23.866 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:54:23.866 INFO:tasks.thrashosds.thrasher:Growing pool iscsi 2021-02-10T04:54:23.866 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:54:24.002 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:54:24.102 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:54:24.103 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:24 smithi164 conmon[29288]: cluster 2021-02-10T04:54:22.472370+0000 mgr.y (mgr. 2021-02-10T04:54:24.103 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:24 smithi164 conmon[29288]: 14140) 1324 : cluster [DBG] pgmap v2176: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 947 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 86 B/s, 1 objects/s recovering 2021-02-10T04:54:24.103 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:24 smithi164 conmon[29288]: cluster 2021-02-10T04:54:23.538918+0000 mon.a (mon.0) 2569 : cluster [DBG] osdmap e1205: 8 total, 7 up, 8 in 2021-02-10T04:54:24.104 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:24 smithi085 conmon[29319]: cluster 2021-02-10T04:54:22.472370+0000 mgr.y (mgr.14140) 1324 : 2021-02-10T04:54:24.104 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:24 smithi085 conmon[29319]: cluster [DBG] pgmap v2176: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 947 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 86 B/s, 1 objects/s recovering 2021-02-10T04:54:24.104 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:24 smithi085 conmon[29319]: cluster 2021-02-10T04:54:23.538918+0000 mon.a (mon.0) 2569 : cluster [DBG] osdmap e1205: 8 total, 7 up, 8 in 2021-02-10T04:54:24.104 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:24 smithi085 conmon[35961]: cluster 2021-02-10T04:54:22.472370+0000 mgr.y (mgr. 2021-02-10T04:54:24.104 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:24 smithi085 conmon[35961]: 14140) 1324 : cluster [DBG] pgmap v2176: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 947 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 86 B/s, 1 objects/s recovering 2021-02-10T04:54:24.104 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:24 smithi085 conmon[35961]: cluster 2021-02-10T04:54:23.538918+0000 2021-02-10T04:54:24.104 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:24 smithi085 conmon[35961]: mon.a (mon.0) 2569 : cluster [DBG] osdmap e1205: 8 total, 7 up, 8 in 2021-02-10T04:54:24.352 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:54:24.455 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set noscrub 2021-02-10T04:54:24.580 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:54:24.789 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:54:26.105 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:26 smithi085 conmon[29319]: cluster 2021-02-10T04:54:24.473088+0000 mgr.y (mgr.14140) 1325 : cluster [DBG] pgmap v2178: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 77 B/s, 1 objects/s recovering 2021-02-10T04:54:26.106 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:26 smithi085 conmon[35961]: cluster 2021-02-10T04:54:24.473088+0000 mgr.y (mgr.14140) 1325 : cluster [DBG] pgmap v2178: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 77 B/s, 1 objects/s recovering 2021-02-10T04:54:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:26 smithi164 conmon[29288]: cluster 2021-02-10T04:54:24.473088+0000 mgr.y 2021-02-10T04:54:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:26 smithi164 conmon[29288]: (mgr.14140) 1325 : cluster [DBG] pgmap v2178: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 77 B/s, 1 objects/s recovering 2021-02-10T04:54:27.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:54:27 smithi164 conmon[37263]: debug 2021-02-10T04:54:27.220+0000 7f1c6b9eb700 -1 osd.5 pg_epoch: 1205 pg[1.a( empty local-lis/les=1118/1119 n=0 ec=110/19 lis/c=1118/1118 les/c/f=1119/1119/0 sis=1118) [5,2,6] r=0 lpr=1118 crt=0'0 mlcod 0'0 active+clean] scrubber pg(1.a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 2 (0x5566505d7ce0) 2021-02-10T04:54:28.108 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:28 smithi085 conmon[35961]: cluster 2021-02-10T04:54:26.473812+0000 mgr.y (mgr.14140) 1326 : cluster [DBG] pgmap v2179: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 644 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 58 B/s, 1 objects/s recovering 2021-02-10T04:54:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:28 smithi164 conmon[29288]: cluster 2021-02-10T04:54:26.473812+0000 mgr.y 2021-02-10T04:54:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:28 smithi164 conmon[29288]: (mgr.14140) 1326 : cluster [DBG] pgmap v2179: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 644 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 58 B/s, 1 objects/s recovering 2021-02-10T04:54:28.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:28 smithi085 conmon[29319]: cluster 2021-02-10T04:54:26.473812+0000 mgr.y (mgr.14140) 1326 : cluster [DBG] pgmap v2179: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 644 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 58 B/s, 1 objects/s recovering 2021-02-10T04:54:29.251 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:54:29.255 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:54:29.291 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:54:29.292 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:54:29.549 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:54:29.553 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: cluster 2021-02-10T04:54:29.553 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: 2021-02-10T04:54:29.553 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: 2021 2021-02-10T04:54:29.553 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: - 2021-02-10T04:54:29.553 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: 02 2021-02-10T04:54:29.553 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: - 2021-02-10T04:54:29.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: 10 2021-02-10T04:54:29.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: T 2021-02-10T04:54:29.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: 04 2021-02-10T04:54:29.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: : 2021-02-10T04:54:29.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: 54 2021-02-10T04:54:29.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: : 2021-02-10T04:54:29.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: 28 2021-02-10T04:54:29.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: . 2021-02-10T04:54:29.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: 474662 2021-02-10T04:54:29.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: +0000 2021-02-10T04:54:29.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: 2021-02-10T04:54:29.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: mgr.y 2021-02-10T04:54:29.554 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: ( 2021-02-10T04:54:29.555 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: mgr 2021-02-10T04:54:29.555 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: . 2021-02-10T04:54:29.555 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: 14140 2021-02-10T04:54:29.555 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: ) 2021-02-10T04:54:29.555 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: 1327 2021-02-10T04:54:29.555 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: : 2021-02-10T04:54:29.555 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: cluster 2021-02-10T04:54:29.555 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: 2021-02-10T04:54:29.555 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: [DBG] 2021-02-10T04:54:29.555 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: 2021-02-10T04:54:29.555 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: pgmap v2180: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:29.555 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: 2021-02-10T04:54:29.555 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: cluster 2021-02-10T04:54:29.555 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: 2021-02-10T04:54:29.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: 2021 2021-02-10T04:54:29.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: - 2021-02-10T04:54:29.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: 02 2021-02-10T04:54:29.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: - 2021-02-10T04:54:29.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: 10 2021-02-10T04:54:29.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: T 2021-02-10T04:54:29.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: 04 2021-02-10T04:54:29.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: : 2021-02-10T04:54:29.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: 54 2021-02-10T04:54:29.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: : 2021-02-10T04:54:29.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: 28 2021-02-10T04:54:29.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: . 2021-02-10T04:54:29.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: 756748 2021-02-10T04:54:29.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: +0000 2021-02-10T04:54:29.557 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: 2021-02-10T04:54:29.557 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: osd.6 2021-02-10T04:54:29.557 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: ( 2021-02-10T04:54:29.557 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: osd 2021-02-10T04:54:29.557 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: . 2021-02-10T04:54:29.557 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: 6 2021-02-10T04:54:29.557 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: ) 2021-02-10T04:54:29.557 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: 184 2021-02-10T04:54:29.557 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: : 2021-02-10T04:54:29.557 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: cluster 2021-02-10T04:54:29.557 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: 2021-02-10T04:54:29.557 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: [DBG] 2021-02-10T04:54:29.557 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: 2021-02-10T04:54:29.558 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: purged_snaps scrub starts 2021-02-10T04:54:29.558 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: 2021-02-10T04:54:29.558 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:54:29.558 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: 04:54:28.759859+0000 osd.6 (osd.6) 185 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:54:29.558 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: audit 2021-02-10T04:54:29.231562+0000 mgr.y 2021-02-10T04:54:29.558 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: (mgr.14140) 1328 : audit [DBG] from='client.15945 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:54:29.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:29 smithi085 conmon[29319]: cluster 2021-02-10T04:54:28.474662+0000 2021-02-10T04:54:29.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:29 smithi085 conmon[29319]: mgr.y (mgr.14140) 1327 : cluster [DBG] pgmap v2180: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:29.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:29 smithi085 conmon[29319]: cluster 2021-02-10T04:54:28.756748+0000 osd.6 (osd.6) 184 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:54:29.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:29 smithi085 conmon[29319]: cluster 2021-02-10T04:54:28.759859+0000 osd.6 (osd.6) 185 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:54:29.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:29 smithi085 conmon[29319]: audit 2021-02-10T04:54:29. 2021-02-10T04:54:29.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:29 smithi085 conmon[29319]: 231562+0000 mgr.y (mgr.14140) 1328 : audit [DBG] from='client.15945 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:54:29.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:29 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:54:29.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:29 smithi085 conmon[29319]: :54:29.254966+0000 mgr.y (mgr.14140) 1329 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:54:29.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:29 smithi085 conmon[35961]: cluster 2021-02-10T04:54:28. 2021-02-10T04:54:29.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:29 smithi085 conmon[35961]: 474662+0000 mgr.y (mgr.14140) 1327 : cluster [DBG] pgmap v2180: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:29.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:29 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:54:29.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:29 smithi085 conmon[35961]: 02-10T04:54:28.756748+0000 osd.6 (osd.6) 184 : cluster 2021-02-10T04:54:29.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:29 smithi085 conmon[35961]: [DBG] purged_snaps scrub starts 2021-02-10T04:54:29.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:29 smithi085 conmon[35961]: cluster 2021-02-10T04:54 2021-02-10T04:54:29.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:29 smithi085 conmon[35961]: :28.759859+0000 osd.6 (osd.6) 2021-02-10T04:54:29.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:29 smithi085 conmon[35961]: 185 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:54:29.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:29 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:54:29.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:29 smithi085 conmon[35961]: 04:54:29.231562+0000 mgr.y (mgr 2021-02-10T04:54:29.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:29 smithi085 conmon[35961]: .14140) 1328 : audit [DBG] from='client.15945 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:54:29.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:29 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:54:29.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:29 smithi085 conmon[35961]: -10T04:54:29.254966+0000 2021-02-10T04:54:29.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:29 smithi085 conmon[35961]: mgr.y (mgr.14140) 1329 : audit [DBG] 2021-02-10T04:54:29.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:29 smithi085 conmon[35961]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:54:29.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:54:29.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: 10T04:54:29.254966+0000 mgr.y (mgr 2021-02-10T04:54:29.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:29 smithi164 conmon[29288]: .14140) 1329 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:54:30.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:30 smithi085 conmon[29319]: cluster 2021-02-10T04:54:29.545521+0000 mon.a (mon.0) 2570 : cluster 2021-02-10T04:54:30.866 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:30 smithi085 conmon[29319]: [DBG] osdmap e1206: 8 total, 7 up, 8 in 2021-02-10T04:54:30.866 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:30 smithi085 conmon[35961]: cluster 2021-02-10T04:54:29.545521+0000 mon.a ( 2021-02-10T04:54:30.866 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:30 smithi085 conmon[35961]: mon.0) 2570 : cluster [DBG] osdmap e1206: 8 total, 7 up, 8 in 2021-02-10T04:54:30.867 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.3 dump_blocked_ops 2021-02-10T04:54:30.869 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":2180,"stamp":"2021-02-10T04:54:28.473970+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":931713416,"num_objects":732,"num_object_clones":323,"num_object_copies":2196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":214,"num_objects_misplaced":44,"num_objects_unfound":0,"num_objects_dirty":732,"num_whiteouts":10,"num_read":7377,"num_read_kb":4589622,"num_write":4593,"num_write_kb":1255989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":898,"num_bytes_recovered":307467062,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35869,"ondisk_log_size":35869,"up":516,"acting":521,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":590,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2382828,"kb_used_data":1995180,"kb_used_omap":386,"kb_used_meta":387197,"kb_avail":747578388,"statfs":{"total":767960285184,"available":765520269312,"internally_reserved":0,"allocated":2043064320,"data_stored":2619198850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":396036,"internal_metadata":396489980},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.394300"},"pg_stats":[{"pgid":"6.29","version":"0'0","reported_seq":"1053","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.150850+0000","last_change":"2021-02-10T04:53:46.719266+0000","last_active":"2021-02-10T04:54:20.150850+0000","last_peered":"2021-02-10T04:54:20.150850+0000","last_clean":"2021-02-10T04:52:30.064708+0000","last_became_active":"2021-02-10T04:53:46.719266+0000","last_became_peered":"2021-02-10T04:53:46.719266+0000","last_unstale":"2021-02-10T04:54:20.150850+0000","last_undegraded":"2021-02-10T04:54:20.150850+0000","last_fullsized":"2021-02-10T04:53:46.701117+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:24.121684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:24.121684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":"101","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.084122+0000","last_change":"2021-02-10T04:52:30.133105+0000","last_active":"2021-02-10T04:54:20.084122+0000","last_peered":"2021-02-10T04:54:20.084122+0000","last_clean":"2021-02-10T04:54:20.084122+0000","last_became_active":"2021-02-10T04:51:36.864926+0000","last_became_peered":"2021-02-10T04:51:36.864926+0000","last_unstale":"2021-02-10T04:54:20.084122+0000","last_undegraded":"2021-02-10T04:54:20.084122+0000","last_fullsized":"2021-02-10T04:54:20.084122+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.133043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.133043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"1191'4","reported_seq":"164","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:23.024760+0000","last_change":"2021-02-10T04:53:46.736164+0000","last_active":"2021-02-10T04:54:23.024760+0000","last_peered":"2021-02-10T04:54:23.024760+0000","last_clean":"2021-02-10T04:54:23.024760+0000","last_became_active":"2021-02-10T04:53:46.735711+0000","last_became_peered":"2021-02-10T04:53:46.735711+0000","last_unstale":"2021-02-10T04:54:23.024760+0000","last_undegraded":"2021-02-10T04:54:23.024760+0000","last_fullsized":"2021-02-10T04:54:23.024760+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'3","last_scrub_stamp":"2021-02-10T04:51:01.185820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.185820+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"1132'66","reported_seq":"1292","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.082233+0000","last_change":"2021-02-10T04:51:55.430490+0000","last_active":"2021-02-10T04:54:20.082233+0000","last_peered":"2021-02-10T04:54:20.082233+0000","last_clean":"2021-02-10T04:54:20.082233+0000","last_became_active":"2021-02-10T04:25:49.002058+0000","last_became_peered":"2021-02-10T04:25:49.002058+0000","last_unstale":"2021-02-10T04:54:20.082233+0000","last_undegraded":"2021-02-10T04:54:20.082233+0000","last_fullsized":"2021-02-10T04:54:20.082233+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1132'66","last_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_deep_scrub":"1132'66","last_deep_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.430399+0000","log_size":66,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":95,"num_read_kb":78,"num_write":55,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"110","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.086402+0000","last_change":"2021-02-10T04:53:41.722609+0000","last_active":"2021-02-10T04:54:20.086402+0000","last_peered":"2021-02-10T04:54:20.086402+0000","last_clean":"2021-02-10T04:54:20.086402+0000","last_became_active":"2021-02-10T04:51:36.875000+0000","last_became_peered":"2021-02-10T04:51:36.875000+0000","last_unstale":"2021-02-10T04:54:20.086402+0000","last_undegraded":"2021-02-10T04:54:20.086402+0000","last_fullsized":"2021-02-10T04:54:20.086402+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:53:41.722516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:41.722516+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"1180","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.082658+0000","last_change":"2021-02-10T04:52:32.077502+0000","last_active":"2021-02-10T04:54:20.082658+0000","last_peered":"2021-02-10T04:54:20.082658+0000","last_clean":"2021-02-10T04:52:30.059220+0000","last_became_active":"2021-02-10T04:52:32.077502+0000","last_became_peered":"2021-02-10T04:52:32.077502+0000","last_unstale":"2021-02-10T04:54:20.082658+0000","last_undegraded":"2021-02-10T04:54:20.082658+0000","last_fullsized":"2021-02-10T04:52:32.069476+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:48.408748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:50:48.408748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"1003'1689","reported_seq":"4625","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.083317+0000","last_change":"2021-02-10T04:53:38.170873+0000","last_active":"2021-02-10T04:54:20.083317+0000","last_peered":"2021-02-10T04:54:20.083317+0000","last_clean":"2021-02-10T04:54:20.083317+0000","last_became_active":"2021-02-10T04:52:26.030105+0000","last_became_peered":"2021-02-10T04:52:26.030105+0000","last_unstale":"2021-02-10T04:54:20.083317+0000","last_undegraded":"2021-02-10T04:54:20.083317+0000","last_fullsized":"2021-02-10T04:54:20.083317+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:53:38.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.170802+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":99361981,"num_objects":37,"num_object_clones":33,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"1049'32","reported_seq":"50","reported_epoch":"1204","state":"active+clean+remapped","last_fresh":"2021-02-10T04:54:20.082836+0000","last_change":"2021-02-10T04:53:42.716996+0000","last_active":"2021-02-10T04:54:20.082836+0000","last_peered":"2021-02-10T04:54:20.082836+0000","last_clean":"2021-02-10T04:54:20.082836+0000","last_became_active":"2021-02-10T04:53:42.693156+0000","last_became_peered":"2021-02-10T04:53:42.693156+0000","last_unstale":"2021-02-10T04:54:20.082836+0000","last_undegraded":"2021-02-10T04:54:20.082836+0000","last_fullsized":"2021-02-10T04:54:20.082836+0000","mapping_epoch":1187,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1188,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'32","last_scrub_stamp":"2021-02-10T04:51:51.382040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.382040+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":61,"num_read_kb":44,"num_write":33,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1788,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,5],"avail_no_missing":["1","4","5"],"object_location_counts":[{"shards":"1,4,5","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"857","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.083443+0000","last_change":"2021-02-10T04:53:34.179015+0000","last_active":"2021-02-10T04:54:20.083443+0000","last_peered":"2021-02-10T04:54:20.083443+0000","last_clean":"2021-02-10T04:54:20.083443+0000","last_became_active":"2021-02-10T04:51:49.502576+0000","last_became_peered":"2021-02-10T04:51:49.502576+0000","last_unstale":"2021-02-10T04:54:20.083443+0000","last_undegraded":"2021-02-10T04:54:20.083443+0000","last_fullsized":"2021-02-10T04:54:20.083443+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:34.178949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.178949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"101","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.081709+0000","last_change":"2021-02-10T04:51:53.038893+0000","last_active":"2021-02-10T04:54:20.081709+0000","last_peered":"2021-02-10T04:54:20.081709+0000","last_clean":"2021-02-10T04:54:20.081709+0000","last_became_active":"2021-02-10T04:51:36.855507+0000","last_became_peered":"2021-02-10T04:51:36.855507+0000","last_unstale":"2021-02-10T04:54:20.081709+0000","last_undegraded":"2021-02-10T04:54:20.081709+0000","last_fullsized":"2021-02-10T04:54:20.081709+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:53.038834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:53.038834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"1189","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.081092+0000","last_change":"2021-02-10T04:52:32.089886+0000","last_active":"2021-02-10T04:54:20.081092+0000","last_peered":"2021-02-10T04:54:20.081092+0000","last_clean":"2021-02-10T04:51:27.993389+0000","last_became_active":"2021-02-10T04:52:32.089886+0000","last_became_peered":"2021-02-10T04:52:32.089886+0000","last_unstale":"2021-02-10T04:54:20.081092+0000","last_undegraded":"2021-02-10T04:54:20.081092+0000","last_fullsized":"2021-02-10T04:52:32.083725+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:09.922789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:09.922789+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.8","version":"1168'82","reported_seq":"137","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.085048+0000","last_change":"2021-02-10T04:53:23.326363+0000","last_active":"2021-02-10T04:54:20.085048+0000","last_peered":"2021-02-10T04:54:20.085048+0000","last_clean":"2021-02-10T04:54:20.085048+0000","last_became_active":"2021-02-10T04:51:36.859215+0000","last_became_peered":"2021-02-10T04:51:36.859215+0000","last_unstale":"2021-02-10T04:54:20.085048+0000","last_undegraded":"2021-02-10T04:54:20.085048+0000","last_fullsized":"2021-02-10T04:54:20.085048+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1150'76","last_scrub_stamp":"2021-02-10T04:53:00.964499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:00.964499+0000","log_size":82,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3848803,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"1049'61","reported_seq":"1293","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.085479+0000","last_change":"2021-02-10T04:53:36.836307+0000","last_active":"2021-02-10T04:54:20.085479+0000","last_peered":"2021-02-10T04:54:20.085479+0000","last_clean":"2021-02-10T04:54:20.085479+0000","last_became_active":"2021-02-10T04:25:49.015913+0000","last_became_peered":"2021-02-10T04:25:49.015913+0000","last_unstale":"2021-02-10T04:54:20.085479+0000","last_undegraded":"2021-02-10T04:54:20.085479+0000","last_fullsized":"2021-02-10T04:54:20.085479+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'61","last_scrub_stamp":"2021-02-10T04:53:36.836202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:36.836202+0000","log_size":61,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":104,"num_read_kb":84,"num_write":62,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"1150'5","reported_seq":"146","reported_epoch":"1205","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:23.542295+0000","last_change":"2021-02-10T04:52:32.093814+0000","last_active":"2021-02-10T04:54:23.542295+0000","last_peered":"2021-02-10T04:54:23.542295+0000","last_clean":"2021-02-10T04:52:30.445006+0000","last_became_active":"2021-02-10T04:52:32.093814+0000","last_became_peered":"2021-02-10T04:52:32.093814+0000","last_unstale":"2021-02-10T04:54:23.542295+0000","last_undegraded":"2021-02-10T04:52:32.079897+0000","last_fullsized":"2021-02-10T04:52:32.079762+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'4","last_scrub_stamp":"2021-02-10T04:52:01.121380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.121380+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"102","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.084850+0000","last_change":"2021-02-10T04:52:32.093831+0000","last_active":"2021-02-10T04:54:20.084850+0000","last_peered":"2021-02-10T04:54:20.084850+0000","last_clean":"2021-02-10T04:52:30.057710+0000","last_became_active":"2021-02-10T04:52:32.093831+0000","last_became_peered":"2021-02-10T04:52:32.093831+0000","last_unstale":"2021-02-10T04:54:20.084850+0000","last_undegraded":"2021-02-10T04:54:20.084850+0000","last_fullsized":"2021-02-10T04:52:32.081168+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:52.421793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:52.421793+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"502","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.153060+0000","last_change":"2021-02-10T04:53:46.733236+0000","last_active":"2021-02-10T04:54:20.153060+0000","last_peered":"2021-02-10T04:54:20.153060+0000","last_clean":"2021-02-10T04:54:20.153060+0000","last_became_active":"2021-02-10T04:53:46.732890+0000","last_became_peered":"2021-02-10T04:53:46.732890+0000","last_unstale":"2021-02-10T04:54:20.153060+0000","last_undegraded":"2021-02-10T04:54:20.153060+0000","last_fullsized":"2021-02-10T04:54:20.153060+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:35.170157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:35.170157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"1003'739","reported_seq":"3708","reported_epoch":"1204","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:20.082330+0000","last_change":"2021-02-10T04:52:32.084576+0000","last_active":"2021-02-10T04:54:20.082330+0000","last_peered":"2021-02-10T04:54:20.082330+0000","last_clean":"2021-02-10T04:43:58.648858+0000","last_became_active":"2021-02-10T04:52:32.084576+0000","last_became_peered":"2021-02-10T04:52:32.084576+0000","last_unstale":"2021-02-10T04:54:20.082330+0000","last_undegraded":"2021-02-10T04:52:32.072214+0000","last_fullsized":"2021-02-10T04:52:32.072101+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:52:27.602006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:27.602006+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24059888,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":129,"num_read_kb":109544,"num_write":102,"num_write_kb":46173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"1066'44","reported_seq":"1359","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.083567+0000","last_change":"2021-02-10T04:53:37.108747+0000","last_active":"2021-02-10T04:54:20.083567+0000","last_peered":"2021-02-10T04:54:20.083567+0000","last_clean":"2021-02-10T04:54:20.083567+0000","last_became_active":"2021-02-10T04:53:35.551311+0000","last_became_peered":"2021-02-10T04:53:35.551311+0000","last_unstale":"2021-02-10T04:54:20.083567+0000","last_undegraded":"2021-02-10T04:54:20.083567+0000","last_fullsized":"2021-02-10T04:54:20.083567+0000","mapping_epoch":1118,"log_start":"1066'44","ondisk_log_start":"1066'44","created":55,"last_epoch_clean":1181,"parent":"0.0","parent_split_bits":0,"last_scrub":"1066'44","last_scrub_stamp":"2021-02-10T04:53:37.108677+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:53:37.108677+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1578,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":150,"num_read_kb":114,"num_write":86,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":2312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"1099","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.083198+0000","last_change":"2021-02-10T04:53:39.126356+0000","last_active":"2021-02-10T04:54:20.083198+0000","last_peered":"2021-02-10T04:54:20.083198+0000","last_clean":"2021-02-10T04:54:20.083198+0000","last_became_active":"2021-02-10T04:29:07.145616+0000","last_became_peered":"2021-02-10T04:29:07.145616+0000","last_unstale":"2021-02-10T04:54:20.083198+0000","last_undegraded":"2021-02-10T04:54:20.083198+0000","last_fullsized":"2021-02-10T04:54:20.083198+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:39.126261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:39.126261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"423","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.082835+0000","last_change":"2021-02-10T04:52:32.083894+0000","last_active":"2021-02-10T04:54:20.082835+0000","last_peered":"2021-02-10T04:54:20.082835+0000","last_clean":"2021-02-10T04:52:30.060966+0000","last_became_active":"2021-02-10T04:52:32.083894+0000","last_became_peered":"2021-02-10T04:52:32.083894+0000","last_unstale":"2021-02-10T04:54:20.082835+0000","last_undegraded":"2021-02-10T04:54:20.082835+0000","last_fullsized":"2021-02-10T04:52:32.079256+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:26.311958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.311958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"119","reported_epoch":"1205","state":"active+clean","last_fresh":"2021-02-10T04:54:24.543735+0000","last_change":"2021-02-10T04:54:24.544535+0000","last_active":"2021-02-10T04:54:24.543735+0000","last_peered":"2021-02-10T04:54:24.543735+0000","last_clean":"2021-02-10T04:54:24.543735+0000","last_became_active":"2021-02-10T04:51:36.855744+0000","last_became_peered":"2021-02-10T04:51:36.855744+0000","last_unstale":"2021-02-10T04:54:24.543735+0000","last_undegraded":"2021-02-10T04:54:24.543735+0000","last_fullsized":"2021-02-10T04:54:24.543735+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:02.162851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:13.727241+0000","last_clean_scrub_stamp":"2021-02-10T04:52:02.162851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"1063'654","reported_seq":"4371","reported_epoch":"1204","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:20.086731+0000","last_change":"2021-02-10T04:52:32.093941+0000","last_active":"2021-02-10T04:54:20.086731+0000","last_peered":"2021-02-10T04:54:20.086731+0000","last_clean":"2021-02-10T04:52:30.062769+0000","last_became_active":"2021-02-10T04:52:32.093941+0000","last_became_peered":"2021-02-10T04:52:32.093941+0000","last_unstale":"2021-02-10T04:54:20.086731+0000","last_undegraded":"2021-02-10T04:52:32.085816+0000","last_fullsized":"2021-02-10T04:52:32.085711+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:51:57.121838+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.121838+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24700630,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":171,"num_read_kb":182272,"num_write":111,"num_write_kb":33727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":26349023,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":12}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"1150","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.082318+0000","last_change":"2021-02-10T04:51:50.433056+0000","last_active":"2021-02-10T04:54:20.082318+0000","last_peered":"2021-02-10T04:54:20.082318+0000","last_clean":"2021-02-10T04:54:20.082318+0000","last_became_active":"2021-02-10T04:51:36.843165+0000","last_became_peered":"2021-02-10T04:51:36.843165+0000","last_unstale":"2021-02-10T04:54:20.082318+0000","last_undegraded":"2021-02-10T04:54:20.082318+0000","last_fullsized":"2021-02-10T04:54:20.082318+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.432990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.432990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"1097","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.151083+0000","last_change":"2021-02-10T04:53:46.727888+0000","last_active":"2021-02-10T04:54:20.151083+0000","last_peered":"2021-02-10T04:54:20.151083+0000","last_clean":"2021-02-10T04:54:20.151083+0000","last_became_active":"2021-02-10T04:53:46.727670+0000","last_became_peered":"2021-02-10T04:53:46.727670+0000","last_unstale":"2021-02-10T04:54:20.151083+0000","last_undegraded":"2021-02-10T04:54:20.151083+0000","last_fullsized":"2021-02-10T04:54:20.151083+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:41.041553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.041553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"1187'54","reported_seq":"1280","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.083355+0000","last_change":"2021-02-10T04:53:42.071173+0000","last_active":"2021-02-10T04:54:20.083355+0000","last_peered":"2021-02-10T04:54:20.083355+0000","last_clean":"2021-02-10T04:54:20.083355+0000","last_became_active":"2021-02-10T04:53:39.679752+0000","last_became_peered":"2021-02-10T04:53:39.679752+0000","last_unstale":"2021-02-10T04:54:20.083355+0000","last_undegraded":"2021-02-10T04:54:20.083355+0000","last_fullsized":"2021-02-10T04:54:20.083355+0000","mapping_epoch":1111,"log_start":"1049'39","ondisk_log_start":"1049'39","created":55,"last_epoch_clean":1185,"parent":"0.0","parent_split_bits":0,"last_scrub":"1187'54","last_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_deep_scrub":"1187'54","last_deep_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_clean_scrub_stamp":"2021-02-10T04:53:42.071103+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":114,"num_read_kb":94,"num_write":73,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"842","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.082976+0000","last_change":"2021-02-10T04:51:52.350764+0000","last_active":"2021-02-10T04:54:20.082976+0000","last_peered":"2021-02-10T04:54:20.082976+0000","last_clean":"2021-02-10T04:54:20.082976+0000","last_became_active":"2021-02-10T04:51:49.567784+0000","last_became_peered":"2021-02-10T04:51:49.567784+0000","last_unstale":"2021-02-10T04:54:20.082976+0000","last_undegraded":"2021-02-10T04:54:20.082976+0000","last_fullsized":"2021-02-10T04:54:20.082976+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:52.350706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.350706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"1119'4","reported_seq":"140","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:23.024100+0000","last_change":"2021-02-10T04:52:27.126670+0000","last_active":"2021-02-10T04:54:23.024100+0000","last_peered":"2021-02-10T04:54:23.024100+0000","last_clean":"2021-02-10T04:54:23.024100+0000","last_became_active":"2021-02-10T04:51:36.853647+0000","last_became_peered":"2021-02-10T04:51:36.853647+0000","last_unstale":"2021-02-10T04:54:23.024100+0000","last_undegraded":"2021-02-10T04:54:23.024100+0000","last_fullsized":"2021-02-10T04:54:23.024100+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'4","last_scrub_stamp":"2021-02-10T04:52:27.126599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:27.126599+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"1190","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.083644+0000","last_change":"2021-02-10T04:52:25.745660+0000","last_active":"2021-02-10T04:54:20.083644+0000","last_peered":"2021-02-10T04:54:20.083644+0000","last_clean":"2021-02-10T04:54:20.083644+0000","last_became_active":"2021-02-10T04:51:36.849503+0000","last_became_peered":"2021-02-10T04:51:36.849503+0000","last_unstale":"2021-02-10T04:54:20.083644+0000","last_undegraded":"2021-02-10T04:54:20.083644+0000","last_fullsized":"2021-02-10T04:54:20.083644+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:25.745577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.745577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"1004'1093","reported_seq":"6245","reported_epoch":"1204","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:20.151461+0000","last_change":"2021-02-10T04:52:32.090286+0000","last_active":"2021-02-10T04:54:20.151461+0000","last_peered":"2021-02-10T04:54:20.151461+0000","last_clean":"2021-02-10T04:44:07.998404+0000","last_became_active":"2021-02-10T04:52:32.090286+0000","last_became_peered":"2021-02-10T04:52:32.090286+0000","last_unstale":"2021-02-10T04:54:20.151461+0000","last_undegraded":"2021-02-10T04:52:32.075231+0000","last_fullsized":"2021-02-10T04:52:32.075115+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39434696,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":279,"num_read_kb":311058,"num_write":185,"num_write_kb":66790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":17}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"1149","reported_epoch":"1205","state":"active+clean","last_fresh":"2021-02-10T04:54:27.219902+0000","last_change":"2021-02-10T04:54:27.220611+0000","last_active":"2021-02-10T04:54:27.219902+0000","last_peered":"2021-02-10T04:54:27.219902+0000","last_clean":"2021-02-10T04:54:27.219902+0000","last_became_active":"2021-02-10T04:51:36.841582+0000","last_became_peered":"2021-02-10T04:51:36.841582+0000","last_unstale":"2021-02-10T04:54:27.219902+0000","last_undegraded":"2021-02-10T04:54:27.219902+0000","last_fullsized":"2021-02-10T04:54:27.219902+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:12.389272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:12.389272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"1049'51","reported_seq":"878","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.082653+0000","last_change":"2021-02-10T04:53:27.560299+0000","last_active":"2021-02-10T04:54:20.082653+0000","last_peered":"2021-02-10T04:54:20.082653+0000","last_clean":"2021-02-10T04:54:20.082653+0000","last_became_active":"2021-02-10T04:53:27.559886+0000","last_became_peered":"2021-02-10T04:53:27.559886+0000","last_unstale":"2021-02-10T04:54:20.082653+0000","last_undegraded":"2021-02-10T04:54:20.082653+0000","last_fullsized":"2021-02-10T04:54:20.082653+0000","mapping_epoch":1129,"log_start":"1049'51","ondisk_log_start":"1049'51","created":55,"last_epoch_clean":1173,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'41","last_scrub_stamp":"2021-02-10T04:51:58.353660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:58.353660+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1472,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":139,"num_read_kb":106,"num_write":82,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"865","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.082166+0000","last_change":"2021-02-10T04:53:11.271557+0000","last_active":"2021-02-10T04:54:20.082166+0000","last_peered":"2021-02-10T04:54:20.082166+0000","last_clean":"2021-02-10T04:54:20.082166+0000","last_became_active":"2021-02-10T04:51:49.567883+0000","last_became_peered":"2021-02-10T04:51:49.567883+0000","last_unstale":"2021-02-10T04:54:20.082166+0000","last_undegraded":"2021-02-10T04:54:20.082166+0000","last_fullsized":"2021-02-10T04:54:20.082166+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:11.271459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.271459+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"1148'4","reported_seq":"1526","reported_epoch":"1204","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:23.024087+0000","last_change":"2021-02-10T04:52:32.088315+0000","last_active":"2021-02-10T04:54:23.024087+0000","last_peered":"2021-02-10T04:54:23.024087+0000","last_clean":"2021-02-10T04:52:30.062243+0000","last_became_active":"2021-02-10T04:52:32.088315+0000","last_became_peered":"2021-02-10T04:52:32.088315+0000","last_unstale":"2021-02-10T04:54:23.024087+0000","last_undegraded":"2021-02-10T04:52:32.069861+0000","last_fullsized":"2021-02-10T04:52:32.069742+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:51:47.855283+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.855283+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"1180","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.086360+0000","last_change":"2021-02-10T04:52:32.094688+0000","last_active":"2021-02-10T04:54:20.086360+0000","last_peered":"2021-02-10T04:54:20.086360+0000","last_clean":"2021-02-10T04:52:30.059376+0000","last_became_active":"2021-02-10T04:52:32.094688+0000","last_became_peered":"2021-02-10T04:52:32.094688+0000","last_unstale":"2021-02-10T04:54:20.086360+0000","last_undegraded":"2021-02-10T04:54:20.086360+0000","last_fullsized":"2021-02-10T04:52:32.079768+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:10.176878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:50:10.176878+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"1002'1090","reported_seq":"5398","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.086248+0000","last_change":"2021-02-10T04:52:26.030198+0000","last_active":"2021-02-10T04:54:20.086248+0000","last_peered":"2021-02-10T04:54:20.086248+0000","last_clean":"2021-02-10T04:54:20.086248+0000","last_became_active":"2021-02-10T04:52:26.028182+0000","last_became_peered":"2021-02-10T04:52:26.028182+0000","last_unstale":"2021-02-10T04:54:20.086248+0000","last_undegraded":"2021-02-10T04:54:20.086248+0000","last_fullsized":"2021-02-10T04:54:20.086248+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:51:08.220472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.220472+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24410646,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":245,"num_read_kb":276168,"num_write":175,"num_write_kb":64960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"1049'47","reported_seq":"1275","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.083359+0000","last_change":"2021-02-10T04:53:31.624862+0000","last_active":"2021-02-10T04:54:20.083359+0000","last_peered":"2021-02-10T04:54:20.083359+0000","last_clean":"2021-02-10T04:54:20.083359+0000","last_became_active":"2021-02-10T04:53:31.624318+0000","last_became_peered":"2021-02-10T04:53:31.624318+0000","last_unstale":"2021-02-10T04:54:20.083359+0000","last_undegraded":"2021-02-10T04:54:20.083359+0000","last_fullsized":"2021-02-10T04:54:20.083359+0000","mapping_epoch":55,"log_start":"1049'47","ondisk_log_start":"1049'47","created":55,"last_epoch_clean":1177,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'47","last_scrub_stamp":"2021-02-10T04:53:09.002368+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.002368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1672,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":142,"num_read_kb":108,"num_write":80,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":934,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"133","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.153168+0000","last_change":"2021-02-10T04:53:46.729988+0000","last_active":"2021-02-10T04:54:20.153168+0000","last_peered":"2021-02-10T04:54:20.153168+0000","last_clean":"2021-02-10T04:28:18.347728+0000","last_became_active":"2021-02-10T04:53:46.729988+0000","last_became_peered":"2021-02-10T04:53:46.729988+0000","last_unstale":"2021-02-10T04:54:20.153168+0000","last_undegraded":"2021-02-10T04:54:20.153168+0000","last_fullsized":"2021-02-10T04:53:46.711213+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:08.358300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:50:08.358300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"1181","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.088205+0000","last_change":"2021-02-10T04:51:57.436961+0000","last_active":"2021-02-10T04:54:20.088205+0000","last_peered":"2021-02-10T04:54:20.088205+0000","last_clean":"2021-02-10T04:54:20.088205+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:54:20.088205+0000","last_undegraded":"2021-02-10T04:54:20.088205+0000","last_fullsized":"2021-02-10T04:54:20.088205+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.436862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.436862+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"1136","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.150840+0000","last_change":"2021-02-10T04:53:02.112612+0000","last_active":"2021-02-10T04:54:20.150840+0000","last_peered":"2021-02-10T04:54:20.150840+0000","last_clean":"2021-02-10T04:54:20.150840+0000","last_became_active":"2021-02-10T04:50:59.545501+0000","last_became_peered":"2021-02-10T04:50:59.545501+0000","last_unstale":"2021-02-10T04:54:20.150840+0000","last_undegraded":"2021-02-10T04:54:20.150840+0000","last_fullsized":"2021-02-10T04:54:20.150840+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:02.112533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:02.112533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"1001'1165","reported_seq":"4368","reported_epoch":"1204","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:20.087074+0000","last_change":"2021-02-10T04:52:32.082334+0000","last_active":"2021-02-10T04:54:20.087074+0000","last_peered":"2021-02-10T04:54:20.087074+0000","last_clean":"2021-02-10T04:52:30.068280+0000","last_became_active":"2021-02-10T04:52:32.082334+0000","last_became_peered":"2021-02-10T04:52:32.082334+0000","last_unstale":"2021-02-10T04:54:20.087074+0000","last_undegraded":"2021-02-10T04:52:32.079317+0000","last_fullsized":"2021-02-10T04:52:32.078663+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29601845,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":286,"num_read_kb":299529,"num_write":200,"num_write_kb":89130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":12}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"1113'5","reported_seq":"1422","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:23.024236+0000","last_change":"2021-02-10T04:52:34.047474+0000","last_active":"2021-02-10T04:54:23.024236+0000","last_peered":"2021-02-10T04:54:23.024236+0000","last_clean":"2021-02-10T04:54:23.024236+0000","last_became_active":"2021-02-10T04:50:59.569578+0000","last_became_peered":"2021-02-10T04:50:59.569578+0000","last_unstale":"2021-02-10T04:54:23.024236+0000","last_undegraded":"2021-02-10T04:54:23.024236+0000","last_fullsized":"2021-02-10T04:54:23.024236+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1113'5","last_scrub_stamp":"2021-02-10T04:52:34.047358+0000","last_deep_scrub":"158'3","last_deep_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_clean_scrub_stamp":"2021-02-10T04:52:34.047358+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"1049'54","reported_seq":"1266","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.082679+0000","last_change":"2021-02-10T04:51:48.872937+0000","last_active":"2021-02-10T04:54:20.082679+0000","last_peered":"2021-02-10T04:54:20.082679+0000","last_clean":"2021-02-10T04:54:20.082679+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:54:20.082679+0000","last_undegraded":"2021-02-10T04:54:20.082679+0000","last_fullsized":"2021-02-10T04:54:20.082679+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:51:48.872839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.872839+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":93,"num_read_kb":73,"num_write":54,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"1255","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.086834+0000","last_change":"2021-02-10T04:53:05.984747+0000","last_active":"2021-02-10T04:54:20.086834+0000","last_peered":"2021-02-10T04:54:20.086834+0000","last_clean":"2021-02-10T04:54:20.086834+0000","last_became_active":"2021-02-10T04:51:36.869931+0000","last_became_peered":"2021-02-10T04:51:36.869931+0000","last_unstale":"2021-02-10T04:54:20.086834+0000","last_undegraded":"2021-02-10T04:54:20.086834+0000","last_fullsized":"2021-02-10T04:54:20.086834+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:05.984659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:05.984659+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"1059","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.152298+0000","last_change":"2021-02-10T04:53:46.731709+0000","last_active":"2021-02-10T04:54:20.152298+0000","last_peered":"2021-02-10T04:54:20.152298+0000","last_clean":"2021-02-10T04:54:20.152298+0000","last_became_active":"2021-02-10T04:53:46.731340+0000","last_became_peered":"2021-02-10T04:53:46.731340+0000","last_unstale":"2021-02-10T04:54:20.152298+0000","last_undegraded":"2021-02-10T04:54:20.152298+0000","last_fullsized":"2021-02-10T04:54:20.152298+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:09.005421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.005421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"998'1009","reported_seq":"4695","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.150532+0000","last_change":"2021-02-10T04:52:26.037227+0000","last_active":"2021-02-10T04:54:20.150532+0000","last_peered":"2021-02-10T04:54:20.150532+0000","last_clean":"2021-02-10T04:54:20.150532+0000","last_became_active":"2021-02-10T04:52:26.036723+0000","last_became_peered":"2021-02-10T04:52:26.036723+0000","last_unstale":"2021-02-10T04:54:20.150532+0000","last_undegraded":"2021-02-10T04:54:20.150532+0000","last_fullsized":"2021-02-10T04:54:20.150532+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12355206,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":119,"num_read_kb":124701,"num_write":77,"num_write_kb":25052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4145029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"1075","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.151141+0000","last_change":"2021-02-10T04:53:33.305896+0000","last_active":"2021-02-10T04:54:20.151141+0000","last_peered":"2021-02-10T04:54:20.151141+0000","last_clean":"2021-02-10T04:54:20.151141+0000","last_became_active":"2021-02-10T04:50:59.570854+0000","last_became_peered":"2021-02-10T04:50:59.570854+0000","last_unstale":"2021-02-10T04:54:20.151141+0000","last_undegraded":"2021-02-10T04:54:20.151141+0000","last_fullsized":"2021-02-10T04:54:20.151141+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:33.305827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:33.305827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"1189","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.083046+0000","last_change":"2021-02-10T04:51:50.895471+0000","last_active":"2021-02-10T04:54:20.083046+0000","last_peered":"2021-02-10T04:54:20.083046+0000","last_clean":"2021-02-10T04:54:20.083046+0000","last_became_active":"2021-02-10T04:50:59.527553+0000","last_became_peered":"2021-02-10T04:50:59.527553+0000","last_unstale":"2021-02-10T04:54:20.083046+0000","last_undegraded":"2021-02-10T04:54:20.083046+0000","last_fullsized":"2021-02-10T04:54:20.083046+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.895370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.895370+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"1049'72","reported_seq":"1293","reported_epoch":"1204","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:20.150664+0000","last_change":"2021-02-10T04:52:32.090145+0000","last_active":"2021-02-10T04:54:20.150664+0000","last_peered":"2021-02-10T04:54:20.150664+0000","last_clean":"2021-02-10T04:52:30.063735+0000","last_became_active":"2021-02-10T04:52:32.090145+0000","last_became_peered":"2021-02-10T04:52:32.090145+0000","last_unstale":"2021-02-10T04:54:20.150664+0000","last_undegraded":"2021-02-10T04:52:32.073169+0000","last_fullsized":"2021-02-10T04:52:32.073062+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'72","last_scrub_stamp":"2021-02-10T04:46:30.515026+0000","last_deep_scrub":"152'48","last_deep_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_clean_scrub_stamp":"2021-02-10T04:46:30.515026+0000","log_size":72,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":105,"num_read_kb":83,"num_write":60,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":14}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"451","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.082724+0000","last_change":"2021-02-10T04:54:18.845320+0000","last_active":"2021-02-10T04:54:20.082724+0000","last_peered":"2021-02-10T04:54:20.082724+0000","last_clean":"2021-02-10T04:54:20.082724+0000","last_became_active":"2021-02-10T04:51:49.503801+0000","last_became_peered":"2021-02-10T04:51:49.503801+0000","last_unstale":"2021-02-10T04:54:20.082724+0000","last_undegraded":"2021-02-10T04:54:20.082724+0000","last_fullsized":"2021-02-10T04:54:20.082724+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:51:48.274960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.274960+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"1214","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.082886+0000","last_change":"2021-02-10T04:53:13.099288+0000","last_active":"2021-02-10T04:54:20.082886+0000","last_peered":"2021-02-10T04:54:20.082886+0000","last_clean":"2021-02-10T04:54:20.082886+0000","last_became_active":"2021-02-10T04:51:36.848614+0000","last_became_peered":"2021-02-10T04:51:36.848614+0000","last_unstale":"2021-02-10T04:54:20.082886+0000","last_undegraded":"2021-02-10T04:54:20.082886+0000","last_fullsized":"2021-02-10T04:54:20.082886+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:13.099228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:13.099228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"1002'1657","reported_seq":"5265","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.150749+0000","last_change":"2021-02-10T04:52:26.036806+0000","last_active":"2021-02-10T04:54:20.150749+0000","last_peered":"2021-02-10T04:54:20.150749+0000","last_clean":"2021-02-10T04:54:20.150749+0000","last_became_active":"2021-02-10T04:52:26.032363+0000","last_became_peered":"2021-02-10T04:52:26.032363+0000","last_unstale":"2021-02-10T04:54:20.150749+0000","last_undegraded":"2021-02-10T04:54:20.150749+0000","last_fullsized":"2021-02-10T04:54:20.150749+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21396533,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":212,"num_read_kb":218836,"num_write":159,"num_write_kb":59398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":22326014,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"1133","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.083122+0000","last_change":"2021-02-10T04:51:51.911945+0000","last_active":"2021-02-10T04:54:20.083122+0000","last_peered":"2021-02-10T04:54:20.083122+0000","last_clean":"2021-02-10T04:54:20.083122+0000","last_became_active":"2021-02-10T04:51:36.850015+0000","last_became_peered":"2021-02-10T04:51:36.850015+0000","last_unstale":"2021-02-10T04:54:20.083122+0000","last_undegraded":"2021-02-10T04:54:20.083122+0000","last_fullsized":"2021-02-10T04:54:20.083122+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.911851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.911851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"1253","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.085652+0000","last_change":"2021-02-10T04:52:32.089163+0000","last_active":"2021-02-10T04:54:20.085652+0000","last_peered":"2021-02-10T04:54:20.085652+0000","last_clean":"2021-02-10T04:52:30.062078+0000","last_became_active":"2021-02-10T04:52:32.089163+0000","last_became_peered":"2021-02-10T04:52:32.089163+0000","last_unstale":"2021-02-10T04:54:20.085652+0000","last_undegraded":"2021-02-10T04:54:20.085652+0000","last_fullsized":"2021-02-10T04:52:32.072413+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:58.142526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:58.142526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"1135'58","reported_seq":"1259","reported_epoch":"1204","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:20.084633+0000","last_change":"2021-02-10T04:52:32.074386+0000","last_active":"2021-02-10T04:54:20.084633+0000","last_peered":"2021-02-10T04:54:20.084633+0000","last_clean":"2021-02-10T04:52:01.474364+0000","last_became_active":"2021-02-10T04:52:32.074386+0000","last_became_peered":"2021-02-10T04:52:32.074386+0000","last_unstale":"2021-02-10T04:54:20.084633+0000","last_undegraded":"2021-02-10T04:52:32.071237+0000","last_fullsized":"2021-02-10T04:52:32.070967+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1135'58","last_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_deep_scrub":"1135'58","last_deep_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.483067+0000","log_size":58,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":87,"num_read_kb":68,"num_write":50,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":11}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"131","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.154501+0000","last_change":"2021-02-10T04:53:46.733605+0000","last_active":"2021-02-10T04:54:20.154501+0000","last_peered":"2021-02-10T04:54:20.154501+0000","last_clean":"2021-02-10T04:29:05.137445+0000","last_became_active":"2021-02-10T04:53:46.733605+0000","last_became_peered":"2021-02-10T04:53:46.733605+0000","last_unstale":"2021-02-10T04:54:20.154501+0000","last_undegraded":"2021-02-10T04:54:20.154501+0000","last_fullsized":"2021-02-10T04:53:46.710695+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:50:46.434629+0000","last_deep_scrub":"53'2","last_deep_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_clean_scrub_stamp":"2021-02-10T04:50:46.434629+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"882","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.082244+0000","last_change":"2021-02-10T04:52:38.278980+0000","last_active":"2021-02-10T04:54:20.082244+0000","last_peered":"2021-02-10T04:54:20.082244+0000","last_clean":"2021-02-10T04:54:20.082244+0000","last_became_active":"2021-02-10T04:51:49.496237+0000","last_became_peered":"2021-02-10T04:51:49.496237+0000","last_unstale":"2021-02-10T04:54:20.082244+0000","last_undegraded":"2021-02-10T04:54:20.082244+0000","last_fullsized":"2021-02-10T04:54:20.082244+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:38.278910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:52:38.278910+0000","last_clean_scrub_stamp":"2021-02-10T04:52:38.278910+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"1000'1071","reported_seq":"4755","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.082626+0000","last_change":"2021-02-10T04:53:34.407875+0000","last_active":"2021-02-10T04:54:20.082626+0000","last_peered":"2021-02-10T04:54:20.082626+0000","last_clean":"2021-02-10T04:54:20.082626+0000","last_became_active":"2021-02-10T04:52:26.031050+0000","last_became_peered":"2021-02-10T04:52:26.031050+0000","last_unstale":"2021-02-10T04:54:20.082626+0000","last_undegraded":"2021-02-10T04:54:20.082626+0000","last_fullsized":"2021-02-10T04:54:20.082626+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:34.407784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.407784+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30173843,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"1219","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.086440+0000","last_change":"2021-02-10T04:52:32.081955+0000","last_active":"2021-02-10T04:54:20.086440+0000","last_peered":"2021-02-10T04:54:20.086440+0000","last_clean":"2021-02-10T04:52:30.068208+0000","last_became_active":"2021-02-10T04:52:32.081955+0000","last_became_peered":"2021-02-10T04:52:32.081955+0000","last_unstale":"2021-02-10T04:54:20.086440+0000","last_undegraded":"2021-02-10T04:54:20.086440+0000","last_fullsized":"2021-02-10T04:52:32.077723+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_clean_scrub_stamp":"2021-02-10T04:51:00.195518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"1182","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.088299+0000","last_change":"2021-02-10T04:52:32.097628+0000","last_active":"2021-02-10T04:54:20.088299+0000","last_peered":"2021-02-10T04:54:20.088299+0000","last_clean":"2021-02-10T04:52:30.061275+0000","last_became_active":"2021-02-10T04:52:32.097628+0000","last_became_peered":"2021-02-10T04:52:32.097628+0000","last_unstale":"2021-02-10T04:54:20.088299+0000","last_undegraded":"2021-02-10T04:54:20.088299+0000","last_fullsized":"2021-02-10T04:52:32.086329+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:55.473347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:50:55.473347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"1049'40","reported_seq":"1251","reported_epoch":"1204","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:20.086159+0000","last_change":"2021-02-10T04:52:32.094738+0000","last_active":"2021-02-10T04:54:20.086159+0000","last_peered":"2021-02-10T04:54:20.086159+0000","last_clean":"2021-02-10T04:51:27.993802+0000","last_became_active":"2021-02-10T04:52:32.094738+0000","last_became_peered":"2021-02-10T04:52:32.094738+0000","last_unstale":"2021-02-10T04:54:20.086159+0000","last_undegraded":"2021-02-10T04:52:32.085496+0000","last_fullsized":"2021-02-10T04:52:32.085393+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'40","last_scrub_stamp":"2021-02-10T04:51:47.281929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.281929+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":52,"num_write":40,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":10}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"357","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.153310+0000","last_change":"2021-02-10T04:53:46.731083+0000","last_active":"2021-02-10T04:54:20.153310+0000","last_peered":"2021-02-10T04:54:20.153310+0000","last_clean":"2021-02-10T04:52:30.068091+0000","last_became_active":"2021-02-10T04:53:46.731083+0000","last_became_peered":"2021-02-10T04:53:46.731083+0000","last_unstale":"2021-02-10T04:54:20.153310+0000","last_undegraded":"2021-02-10T04:54:20.153310+0000","last_fullsized":"2021-02-10T04:53:46.711548+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:50.399350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:50.399350+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"1156","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.082925+0000","last_change":"2021-02-10T04:52:32.081196+0000","last_active":"2021-02-10T04:54:20.082925+0000","last_peered":"2021-02-10T04:54:20.082925+0000","last_clean":"2021-02-10T04:49:56.935460+0000","last_became_active":"2021-02-10T04:52:32.081196+0000","last_became_peered":"2021-02-10T04:52:32.081196+0000","last_unstale":"2021-02-10T04:54:20.082925+0000","last_undegraded":"2021-02-10T04:54:20.082925+0000","last_fullsized":"2021-02-10T04:52:32.068851+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"867","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.082909+0000","last_change":"2021-02-10T04:52:31.273037+0000","last_active":"2021-02-10T04:54:20.082909+0000","last_peered":"2021-02-10T04:54:20.082909+0000","last_clean":"2021-02-10T04:54:20.082909+0000","last_became_active":"2021-02-10T04:51:49.501405+0000","last_became_peered":"2021-02-10T04:51:49.501405+0000","last_unstale":"2021-02-10T04:54:20.082909+0000","last_undegraded":"2021-02-10T04:54:20.082909+0000","last_fullsized":"2021-02-10T04:54:20.082909+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:31.272971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:17.598113+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.272971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"1000'446","reported_seq":"1914","reported_epoch":"1204","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:20.150871+0000","last_change":"2021-02-10T04:53:46.723625+0000","last_active":"2021-02-10T04:54:20.150871+0000","last_peered":"2021-02-10T04:54:20.150871+0000","last_clean":"2021-02-10T04:43:52.634592+0000","last_became_active":"2021-02-10T04:53:46.723625+0000","last_became_peered":"2021-02-10T04:53:46.723625+0000","last_unstale":"2021-02-10T04:54:20.150871+0000","last_undegraded":"2021-02-10T04:53:46.707033+0000","last_fullsized":"2021-02-10T04:53:46.706921+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:51:45.964029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.964029+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22255852,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68435,"num_write":72,"num_write_kb":24345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":403693,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":9}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"358","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.083229+0000","last_change":"2021-02-10T04:53:03.228260+0000","last_active":"2021-02-10T04:54:20.083229+0000","last_peered":"2021-02-10T04:54:20.083229+0000","last_clean":"2021-02-10T04:54:20.083229+0000","last_became_active":"2021-02-10T04:51:49.499252+0000","last_became_peered":"2021-02-10T04:51:49.499252+0000","last_unstale":"2021-02-10T04:54:20.083229+0000","last_undegraded":"2021-02-10T04:54:20.083229+0000","last_fullsized":"2021-02-10T04:54:20.083229+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:03.228182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.228182+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"110","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.151886+0000","last_change":"2021-02-10T04:53:46.727384+0000","last_active":"2021-02-10T04:54:20.151886+0000","last_peered":"2021-02-10T04:54:20.151886+0000","last_clean":"2021-02-10T04:54:20.151886+0000","last_became_active":"2021-02-10T04:53:46.726771+0000","last_became_peered":"2021-02-10T04:53:46.726771+0000","last_unstale":"2021-02-10T04:54:20.151886+0000","last_undegraded":"2021-02-10T04:54:20.151886+0000","last_fullsized":"2021-02-10T04:54:20.151886+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:33.948348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:33.948348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"1049'39","reported_seq":"1272","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.086951+0000","last_change":"2021-02-10T04:54:17.118810+0000","last_active":"2021-02-10T04:54:20.086951+0000","last_peered":"2021-02-10T04:54:20.086951+0000","last_clean":"2021-02-10T04:54:20.086951+0000","last_became_active":"2021-02-10T04:54:17.080053+0000","last_became_peered":"2021-02-10T04:54:17.080053+0000","last_unstale":"2021-02-10T04:54:20.086951+0000","last_undegraded":"2021-02-10T04:54:20.086951+0000","last_fullsized":"2021-02-10T04:54:20.086951+0000","mapping_epoch":1200,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1201,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'39","last_scrub_stamp":"2021-02-10T04:51:47.322331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.322331+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":51,"num_write":39,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"129","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.153444+0000","last_change":"2021-02-10T04:53:46.714141+0000","last_active":"2021-02-10T04:54:20.153444+0000","last_peered":"2021-02-10T04:54:20.153444+0000","last_clean":"2021-02-10T04:28:45.095539+0000","last_became_active":"2021-02-10T04:53:46.714141+0000","last_became_peered":"2021-02-10T04:53:46.714141+0000","last_unstale":"2021-02-10T04:54:20.153444+0000","last_undegraded":"2021-02-10T04:54:20.153444+0000","last_fullsized":"2021-02-10T04:53:46.700190+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_clean_scrub_stamp":"2021-02-10T04:50:05.424262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"1375","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.084530+0000","last_change":"2021-02-10T04:52:37.324122+0000","last_active":"2021-02-10T04:54:20.084530+0000","last_peered":"2021-02-10T04:54:20.084530+0000","last_clean":"2021-02-10T04:54:20.084530+0000","last_became_active":"2021-02-10T04:51:36.843553+0000","last_became_peered":"2021-02-10T04:51:36.843553+0000","last_unstale":"2021-02-10T04:54:20.084530+0000","last_undegraded":"2021-02-10T04:54:20.084530+0000","last_fullsized":"2021-02-10T04:54:20.084530+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:37.324040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:37.324040+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"1172","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.150578+0000","last_change":"2021-02-10T04:51:57.769992+0000","last_active":"2021-02-10T04:54:20.150578+0000","last_peered":"2021-02-10T04:54:20.150578+0000","last_clean":"2021-02-10T04:54:20.150578+0000","last_became_active":"2021-02-10T04:50:59.587343+0000","last_became_peered":"2021-02-10T04:50:59.587343+0000","last_unstale":"2021-02-10T04:54:20.150578+0000","last_undegraded":"2021-02-10T04:54:20.150578+0000","last_fullsized":"2021-02-10T04:54:20.150578+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"1203","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.088316+0000","last_change":"2021-02-10T04:52:31.403904+0000","last_active":"2021-02-10T04:54:20.088316+0000","last_peered":"2021-02-10T04:54:20.088316+0000","last_clean":"2021-02-10T04:54:20.088316+0000","last_became_active":"2021-02-10T04:50:59.538009+0000","last_became_peered":"2021-02-10T04:50:59.538009+0000","last_unstale":"2021-02-10T04:54:20.088316+0000","last_undegraded":"2021-02-10T04:54:20.088316+0000","last_fullsized":"2021-02-10T04:54:20.088316+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:31.403824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.403824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"1000'490","reported_seq":"3469","reported_epoch":"1204","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:20.154217+0000","last_change":"2021-02-10T04:53:46.711617+0000","last_active":"2021-02-10T04:54:20.154217+0000","last_peered":"2021-02-10T04:54:20.154217+0000","last_clean":"2021-02-10T04:43:50.630768+0000","last_became_active":"2021-02-10T04:53:46.711617+0000","last_became_peered":"2021-02-10T04:53:46.711617+0000","last_unstale":"2021-02-10T04:54:20.154217+0000","last_undegraded":"2021-02-10T04:53:46.702746+0000","last_fullsized":"2021-02-10T04:53:46.702635+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1040,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:47:02.875920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:02.875920+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19371627,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":141,"num_read_kb":148568,"num_write":85,"num_write_kb":31732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":7}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"101","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.086557+0000","last_change":"2021-02-10T04:53:01.972991+0000","last_active":"2021-02-10T04:54:20.086557+0000","last_peered":"2021-02-10T04:54:20.086557+0000","last_clean":"2021-02-10T04:54:20.086557+0000","last_became_active":"2021-02-10T04:51:36.863976+0000","last_became_peered":"2021-02-10T04:51:36.863976+0000","last_unstale":"2021-02-10T04:54:20.086557+0000","last_undegraded":"2021-02-10T04:54:20.086557+0000","last_fullsized":"2021-02-10T04:54:20.086557+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:01.972913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.972913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"1049'37","reported_seq":"882","reported_epoch":"1204","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:20.082651+0000","last_change":"2021-02-10T04:52:32.095888+0000","last_active":"2021-02-10T04:54:20.082651+0000","last_peered":"2021-02-10T04:54:20.082651+0000","last_clean":"2021-02-10T04:52:30.062113+0000","last_became_active":"2021-02-10T04:52:32.095888+0000","last_became_peered":"2021-02-10T04:52:32.095888+0000","last_unstale":"2021-02-10T04:54:20.082651+0000","last_undegraded":"2021-02-10T04:52:32.076227+0000","last_fullsized":"2021-02-10T04:52:32.076110+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:56.431187+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.431187+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":59,"num_read_kb":43,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"102","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.084280+0000","last_change":"2021-02-10T04:52:32.089886+0000","last_active":"2021-02-10T04:54:20.084280+0000","last_peered":"2021-02-10T04:54:20.084280+0000","last_clean":"2021-02-10T04:52:30.064491+0000","last_became_active":"2021-02-10T04:52:32.089886+0000","last_became_peered":"2021-02-10T04:52:32.089886+0000","last_unstale":"2021-02-10T04:54:20.084280+0000","last_undegraded":"2021-02-10T04:54:20.084280+0000","last_fullsized":"2021-02-10T04:52:32.078850+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:04.591695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:04.591695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"1013","reported_epoch":"1205","state":"active+clean","last_fresh":"2021-02-10T04:54:26.698878+0000","last_change":"2021-02-10T04:52:25.358146+0000","last_active":"2021-02-10T04:54:26.698878+0000","last_peered":"2021-02-10T04:54:26.698878+0000","last_clean":"2021-02-10T04:54:26.698878+0000","last_became_active":"2021-02-10T04:51:49.496092+0000","last_became_peered":"2021-02-10T04:51:49.496092+0000","last_unstale":"2021-02-10T04:54:26.698878+0000","last_undegraded":"2021-02-10T04:54:26.698878+0000","last_fullsized":"2021-02-10T04:54:26.698878+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":590,"num_read_kb":585,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"1049","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.154446+0000","last_change":"2021-02-10T04:53:46.729138+0000","last_active":"2021-02-10T04:54:20.154446+0000","last_peered":"2021-02-10T04:54:20.154446+0000","last_clean":"2021-02-10T04:54:20.154446+0000","last_became_active":"2021-02-10T04:53:46.728845+0000","last_became_peered":"2021-02-10T04:53:46.728845+0000","last_unstale":"2021-02-10T04:54:20.154446+0000","last_undegraded":"2021-02-10T04:54:20.154446+0000","last_fullsized":"2021-02-10T04:54:20.154446+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:40.066354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:52:40.066354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"1000'620","reported_seq":"4210","reported_epoch":"1204","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:20.083134+0000","last_change":"2021-02-10T04:52:32.082414+0000","last_active":"2021-02-10T04:54:20.083134+0000","last_peered":"2021-02-10T04:54:20.083134+0000","last_clean":"2021-02-10T04:43:51.960482+0000","last_became_active":"2021-02-10T04:52:32.082414+0000","last_became_peered":"2021-02-10T04:52:32.082414+0000","last_unstale":"2021-02-10T04:54:20.083134+0000","last_undegraded":"2021-02-10T04:52:32.068324+0000","last_fullsized":"2021-02-10T04:52:32.068200+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:51:54.444287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:54.444287+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14265195,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":144,"num_read_kb":128482,"num_write":129,"num_write_kb":53443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"158","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.082374+0000","last_change":"2021-02-10T04:52:32.076404+0000","last_active":"2021-02-10T04:54:20.082374+0000","last_peered":"2021-02-10T04:54:20.082374+0000","last_clean":"2021-02-10T04:28:47.053193+0000","last_became_active":"2021-02-10T04:52:32.076404+0000","last_became_peered":"2021-02-10T04:52:32.076404+0000","last_unstale":"2021-02-10T04:54:20.082374+0000","last_undegraded":"2021-02-10T04:54:20.082374+0000","last_fullsized":"2021-02-10T04:52:32.068279+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:14.955544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:47:14.955544+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"65","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.083668+0000","last_change":"2021-02-10T04:52:32.084350+0000","last_active":"2021-02-10T04:54:20.083668+0000","last_peered":"2021-02-10T04:54:20.083668+0000","last_clean":"2021-02-10T04:25:49.999907+0000","last_became_active":"2021-02-10T04:52:32.084350+0000","last_became_peered":"2021-02-10T04:52:32.084350+0000","last_unstale":"2021-02-10T04:54:20.083668+0000","last_undegraded":"2021-02-10T04:54:20.083668+0000","last_fullsized":"2021-02-10T04:52:32.073004+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:27.908221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:47:27.908221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"1049'55","reported_seq":"838","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.082552+0000","last_change":"2021-02-10T04:52:30.277009+0000","last_active":"2021-02-10T04:54:20.082552+0000","last_peered":"2021-02-10T04:54:20.082552+0000","last_clean":"2021-02-10T04:54:20.082552+0000","last_became_active":"2021-02-10T04:51:49.501622+0000","last_became_peered":"2021-02-10T04:51:49.501622+0000","last_unstale":"2021-02-10T04:54:20.082552+0000","last_undegraded":"2021-02-10T04:54:20.082552+0000","last_fullsized":"2021-02-10T04:54:20.082552+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'55","last_scrub_stamp":"2021-02-10T04:52:30.276938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.276938+0000","log_size":55,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":98,"num_read_kb":74,"num_write":58,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"1199","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.082815+0000","last_change":"2021-02-10T04:51:59.420149+0000","last_active":"2021-02-10T04:54:20.082815+0000","last_peered":"2021-02-10T04:54:20.082815+0000","last_clean":"2021-02-10T04:54:20.082815+0000","last_became_active":"2021-02-10T04:28:21.125541+0000","last_became_peered":"2021-02-10T04:28:21.125541+0000","last_unstale":"2021-02-10T04:54:20.082815+0000","last_undegraded":"2021-02-10T04:54:20.082815+0000","last_fullsized":"2021-02-10T04:54:20.082815+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.420086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.420086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"1165","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.151194+0000","last_change":"2021-02-10T04:53:12.187225+0000","last_active":"2021-02-10T04:54:20.151194+0000","last_peered":"2021-02-10T04:54:20.151194+0000","last_clean":"2021-02-10T04:54:20.151194+0000","last_became_active":"2021-02-10T04:50:59.555689+0000","last_became_peered":"2021-02-10T04:50:59.555689+0000","last_unstale":"2021-02-10T04:54:20.151194+0000","last_undegraded":"2021-02-10T04:54:20.151194+0000","last_fullsized":"2021-02-10T04:54:20.151194+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.187154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.187154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"1002'2042","reported_seq":"9427","reported_epoch":"1204","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:20.086737+0000","last_change":"2021-02-10T04:52:32.083034+0000","last_active":"2021-02-10T04:54:20.086737+0000","last_peered":"2021-02-10T04:54:20.086737+0000","last_clean":"2021-02-10T04:52:30.068473+0000","last_became_active":"2021-02-10T04:52:32.083034+0000","last_became_peered":"2021-02-10T04:52:32.083034+0000","last_unstale":"2021-02-10T04:54:20.086737+0000","last_undegraded":"2021-02-10T04:52:32.077334+0000","last_fullsized":"2021-02-10T04:52:32.077220+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:46:57.054230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:57.054230+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":85678898,"num_objects":35,"num_object_clones":29,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":35}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"1141","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.082298+0000","last_change":"2021-02-10T04:52:32.094777+0000","last_active":"2021-02-10T04:54:20.082298+0000","last_peered":"2021-02-10T04:54:20.082298+0000","last_clean":"2021-02-10T04:52:30.061448+0000","last_became_active":"2021-02-10T04:52:32.094777+0000","last_became_peered":"2021-02-10T04:52:32.094777+0000","last_unstale":"2021-02-10T04:54:20.082298+0000","last_undegraded":"2021-02-10T04:54:20.082298+0000","last_fullsized":"2021-02-10T04:52:32.069238+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:52.341527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:50:52.341527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"1148'3","reported_seq":"98","reported_epoch":"1204","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:23.024504+0000","last_change":"2021-02-10T04:52:32.089249+0000","last_active":"2021-02-10T04:54:23.024504+0000","last_peered":"2021-02-10T04:54:23.024504+0000","last_clean":"2021-02-10T04:25:51.042416+0000","last_became_active":"2021-02-10T04:52:32.089249+0000","last_became_peered":"2021-02-10T04:52:32.089249+0000","last_unstale":"2021-02-10T04:54:23.024504+0000","last_undegraded":"2021-02-10T04:52:32.073980+0000","last_fullsized":"2021-02-10T04:52:32.073877+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:54.466259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:54.466259+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"1049'54","reported_seq":"887","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.083176+0000","last_change":"2021-02-10T04:53:01.304693+0000","last_active":"2021-02-10T04:54:20.083176+0000","last_peered":"2021-02-10T04:54:20.083176+0000","last_clean":"2021-02-10T04:54:20.083176+0000","last_became_active":"2021-02-10T04:51:49.498066+0000","last_became_peered":"2021-02-10T04:51:49.498066+0000","last_unstale":"2021-02-10T04:54:20.083176+0000","last_undegraded":"2021-02-10T04:54:20.083176+0000","last_fullsized":"2021-02-10T04:54:20.083176+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:53:01.304626+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.304626+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":74,"num_read_kb":56,"num_write":42,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"454","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.082673+0000","last_change":"2021-02-10T04:52:32.091144+0000","last_active":"2021-02-10T04:54:20.082673+0000","last_peered":"2021-02-10T04:54:20.082673+0000","last_clean":"2021-02-10T04:52:30.060819+0000","last_became_active":"2021-02-10T04:52:32.091144+0000","last_became_peered":"2021-02-10T04:52:32.091144+0000","last_unstale":"2021-02-10T04:54:20.082673+0000","last_undegraded":"2021-02-10T04:54:20.082673+0000","last_fullsized":"2021-02-10T04:52:32.079969+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:49.447961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:49.447961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"1186","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.150083+0000","last_change":"2021-02-10T04:52:32.088766+0000","last_active":"2021-02-10T04:54:20.150083+0000","last_peered":"2021-02-10T04:54:20.150083+0000","last_clean":"2021-02-10T04:51:27.993073+0000","last_became_active":"2021-02-10T04:52:32.088766+0000","last_became_peered":"2021-02-10T04:52:32.088766+0000","last_unstale":"2021-02-10T04:54:20.150083+0000","last_undegraded":"2021-02-10T04:54:20.150083+0000","last_fullsized":"2021-02-10T04:52:32.071883+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:07.914061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:07.914061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"293'77","reported_seq":"2986","reported_epoch":"1204","state":"active+clean+scrubbing","last_fresh":"2021-02-10T04:54:20.082345+0000","last_change":"2021-02-10T04:53:43.102301+0000","last_active":"2021-02-10T04:54:20.082345+0000","last_peered":"2021-02-10T04:54:20.082345+0000","last_clean":"2021-02-10T04:54:20.082345+0000","last_became_active":"2021-02-10T04:52:26.024812+0000","last_became_peered":"2021-02-10T04:52:26.024812+0000","last_unstale":"2021-02-10T04:54:20.082345+0000","last_undegraded":"2021-02-10T04:54:20.082345+0000","last_fullsized":"2021-02-10T04:54:20.082345+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1886585,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":18587,"num_write":9,"num_write_kb":3315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"1218","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.086655+0000","last_change":"2021-02-10T04:53:07.025068+0000","last_active":"2021-02-10T04:54:20.086655+0000","last_peered":"2021-02-10T04:54:20.086655+0000","last_clean":"2021-02-10T04:54:20.086655+0000","last_became_active":"2021-02-10T04:50:59.589417+0000","last_became_peered":"2021-02-10T04:50:59.589417+0000","last_unstale":"2021-02-10T04:54:20.086655+0000","last_undegraded":"2021-02-10T04:54:20.086655+0000","last_fullsized":"2021-02-10T04:54:20.086655+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.024980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.024980+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"102","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.084900+0000","last_change":"2021-02-10T04:52:32.082945+0000","last_active":"2021-02-10T04:54:20.084900+0000","last_peered":"2021-02-10T04:54:20.084900+0000","last_clean":"2021-02-10T04:52:30.056402+0000","last_became_active":"2021-02-10T04:52:32.082945+0000","last_became_peered":"2021-02-10T04:52:32.082945+0000","last_unstale":"2021-02-10T04:54:20.084900+0000","last_undegraded":"2021-02-10T04:54:20.084900+0000","last_fullsized":"2021-02-10T04:52:32.077773+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.609500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.609500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"1048'39","reported_seq":"1299","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.082773+0000","last_change":"2021-02-10T04:53:38.085663+0000","last_active":"2021-02-10T04:54:20.082773+0000","last_peered":"2021-02-10T04:54:20.082773+0000","last_clean":"2021-02-10T04:54:20.082773+0000","last_became_active":"2021-02-10T04:39:02.560739+0000","last_became_peered":"2021-02-10T04:39:02.560739+0000","last_unstale":"2021-02-10T04:54:20.082773+0000","last_undegraded":"2021-02-10T04:54:20.082773+0000","last_fullsized":"2021-02-10T04:54:20.082773+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'39","last_scrub_stamp":"2021-02-10T04:53:38.085565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.085565+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":76,"num_read_kb":59,"num_write":41,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"1309","reported_epoch":"1204","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:20.083818+0000","last_change":"2021-02-10T04:52:32.084961+0000","last_active":"2021-02-10T04:54:20.083818+0000","last_peered":"2021-02-10T04:54:20.083818+0000","last_clean":"2021-02-10T04:52:30.813009+0000","last_became_active":"2021-02-10T04:52:32.084961+0000","last_became_peered":"2021-02-10T04:52:32.084961+0000","last_unstale":"2021-02-10T04:54:20.083818+0000","last_undegraded":"2021-02-10T04:52:32.070468+0000","last_fullsized":"2021-02-10T04:52:32.070358+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:47:08.784593+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:47:08.784593+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"1167","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.151045+0000","last_change":"2021-02-10T04:52:30.019225+0000","last_active":"2021-02-10T04:54:20.151045+0000","last_peered":"2021-02-10T04:54:20.151045+0000","last_clean":"2021-02-10T04:54:20.151045+0000","last_became_active":"2021-02-10T04:50:59.561436+0000","last_became_peered":"2021-02-10T04:50:59.561436+0000","last_unstale":"2021-02-10T04:54:20.151045+0000","last_undegraded":"2021-02-10T04:54:20.151045+0000","last_fullsized":"2021-02-10T04:54:20.151045+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.019102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.019102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"1001'1528","reported_seq":"6152","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.150466+0000","last_change":"2021-02-10T04:52:26.036366+0000","last_active":"2021-02-10T04:54:20.150466+0000","last_peered":"2021-02-10T04:54:20.150466+0000","last_clean":"2021-02-10T04:54:20.150466+0000","last_became_active":"2021-02-10T04:52:26.034028+0000","last_became_peered":"2021-02-10T04:52:26.034028+0000","last_unstale":"2021-02-10T04:54:20.150466+0000","last_undegraded":"2021-02-10T04:54:20.150466+0000","last_fullsized":"2021-02-10T04:54:20.150466+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46944136,"num_objects":16,"num_object_clones":15,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":358,"num_read_kb":351934,"num_write":286,"num_write_kb":111355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":58968832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.15","version":"1000'446","reported_seq":"2141","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.083540+0000","last_change":"2021-02-10T04:53:11.051124+0000","last_active":"2021-02-10T04:54:20.083540+0000","last_peered":"2021-02-10T04:54:20.083540+0000","last_clean":"2021-02-10T04:54:20.083540+0000","last_became_active":"2021-02-10T04:51:40.510899+0000","last_became_peered":"2021-02-10T04:51:40.510899+0000","last_unstale":"2021-02-10T04:54:20.083540+0000","last_undegraded":"2021-02-10T04:54:20.083540+0000","last_fullsized":"2021-02-10T04:54:20.083540+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:53:11.051031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.051031+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24896285,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":18983408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"1048'40","reported_seq":"95","reported_epoch":"1204","state":"active+clean+remapped","last_fresh":"2021-02-10T04:54:20.085374+0000","last_change":"2021-02-10T04:53:32.953859+0000","last_active":"2021-02-10T04:54:20.085374+0000","last_peered":"2021-02-10T04:54:20.085374+0000","last_clean":"2021-02-10T04:54:20.085374+0000","last_became_active":"2021-02-10T04:53:21.308544+0000","last_became_peered":"2021-02-10T04:53:21.308544+0000","last_unstale":"2021-02-10T04:54:20.085374+0000","last_undegraded":"2021-02-10T04:54:20.085374+0000","last_fullsized":"2021-02-10T04:54:20.085374+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'40","last_scrub_stamp":"2021-02-10T04:53:32.953697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:32.953697+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":79,"num_read_kb":57,"num_write":41,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1206,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1,3],"avail_no_missing":["2","1","3"],"object_location_counts":[{"shards":"1,2,3","objects":13}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"106","reported_epoch":"1204","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:20.085139+0000","last_change":"2021-02-10T04:52:32.094875+0000","last_active":"2021-02-10T04:54:20.085139+0000","last_peered":"2021-02-10T04:54:20.085139+0000","last_clean":"2021-02-10T04:52:30.056787+0000","last_became_active":"2021-02-10T04:52:32.094875+0000","last_became_peered":"2021-02-10T04:52:32.094875+0000","last_unstale":"2021-02-10T04:54:20.085139+0000","last_undegraded":"2021-02-10T04:52:32.083818+0000","last_fullsized":"2021-02-10T04:52:32.083707+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:51:08.639784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.639784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"1136","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.151520+0000","last_change":"2021-02-10T04:51:09.475150+0000","last_active":"2021-02-10T04:54:20.151520+0000","last_peered":"2021-02-10T04:54:20.151520+0000","last_clean":"2021-02-10T04:54:20.151520+0000","last_became_active":"2021-02-10T04:50:59.585917+0000","last_became_peered":"2021-02-10T04:50:59.585917+0000","last_unstale":"2021-02-10T04:54:20.151520+0000","last_undegraded":"2021-02-10T04:54:20.151520+0000","last_fullsized":"2021-02-10T04:54:20.151520+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:09.475063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:09.475063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"106","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.083405+0000","last_change":"2021-02-10T04:53:34.927483+0000","last_active":"2021-02-10T04:54:20.083405+0000","last_peered":"2021-02-10T04:54:20.083405+0000","last_clean":"2021-02-10T04:54:20.083405+0000","last_became_active":"2021-02-10T04:51:36.859707+0000","last_became_peered":"2021-02-10T04:51:36.859707+0000","last_unstale":"2021-02-10T04:54:20.083405+0000","last_undegraded":"2021-02-10T04:54:20.083405+0000","last_fullsized":"2021-02-10T04:54:20.083405+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:34.927424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.927424+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"1000'620","reported_seq":"4491","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.150908+0000","last_change":"2021-02-10T04:53:37.384461+0000","last_active":"2021-02-10T04:54:20.150908+0000","last_peered":"2021-02-10T04:54:20.150908+0000","last_clean":"2021-02-10T04:54:20.150908+0000","last_became_active":"2021-02-10T04:51:40.510114+0000","last_became_peered":"2021-02-10T04:51:40.510114+0000","last_unstale":"2021-02-10T04:54:20.150908+0000","last_undegraded":"2021-02-10T04:54:20.150908+0000","last_fullsized":"2021-02-10T04:54:20.150908+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:53:37.384398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:37.384398+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28530390,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"1063'30","reported_seq":"1179","reported_epoch":"1204","state":"active+clean+remapped","last_fresh":"2021-02-10T04:54:20.082814+0000","last_change":"2021-02-10T04:53:21.312523+0000","last_active":"2021-02-10T04:54:20.082814+0000","last_peered":"2021-02-10T04:54:20.082814+0000","last_clean":"2021-02-10T04:54:20.082814+0000","last_became_active":"2021-02-10T04:53:21.302364+0000","last_became_peered":"2021-02-10T04:53:21.302364+0000","last_unstale":"2021-02-10T04:54:20.082814+0000","last_undegraded":"2021-02-10T04:54:20.082814+0000","last_fullsized":"2021-02-10T04:54:20.082814+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"1063'30","last_scrub_stamp":"2021-02-10T04:52:43.102165+0000","last_deep_scrub":"1063'30","last_deep_scrub_stamp":"2021-02-10T04:47:05.589582+0000","last_clean_scrub_stamp":"2021-02-10T04:52:43.102165+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5,3],"avail_no_missing":["1","3","5"],"object_location_counts":[{"shards":"1,3,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"1192","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.081539+0000","last_change":"2021-02-10T04:52:41.059269+0000","last_active":"2021-02-10T04:54:20.081539+0000","last_peered":"2021-02-10T04:54:20.081539+0000","last_clean":"2021-02-10T04:54:20.081539+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:54:20.081539+0000","last_undegraded":"2021-02-10T04:54:20.081539+0000","last_fullsized":"2021-02-10T04:54:20.081539+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:52:41.059159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.059159+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"1196","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.086076+0000","last_change":"2021-02-10T04:53:12.332890+0000","last_active":"2021-02-10T04:54:20.086076+0000","last_peered":"2021-02-10T04:54:20.086076+0000","last_clean":"2021-02-10T04:54:20.086076+0000","last_became_active":"2021-02-10T04:50:59.540375+0000","last_became_peered":"2021-02-10T04:50:59.540375+0000","last_unstale":"2021-02-10T04:54:20.086076+0000","last_undegraded":"2021-02-10T04:54:20.086076+0000","last_fullsized":"2021-02-10T04:54:20.086076+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.332233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.332233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"114","reported_epoch":"1205","state":"active+clean","last_fresh":"2021-02-10T04:54:24.816791+0000","last_change":"2021-02-10T04:54:24.817625+0000","last_active":"2021-02-10T04:54:24.816791+0000","last_peered":"2021-02-10T04:54:24.816791+0000","last_clean":"2021-02-10T04:54:24.816791+0000","last_became_active":"2021-02-10T04:53:46.733191+0000","last_became_peered":"2021-02-10T04:53:46.733191+0000","last_unstale":"2021-02-10T04:54:24.816791+0000","last_undegraded":"2021-02-10T04:54:24.816791+0000","last_fullsized":"2021-02-10T04:54:24.816791+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:45.362537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:45.362537+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"1002'2042","reported_seq":"9178","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.153974+0000","last_change":"2021-02-10T04:53:46.726353+0000","last_active":"2021-02-10T04:54:20.153974+0000","last_peered":"2021-02-10T04:54:20.153974+0000","last_clean":"2021-02-10T04:54:20.153974+0000","last_became_active":"2021-02-10T04:53:46.725828+0000","last_became_peered":"2021-02-10T04:53:46.725828+0000","last_unstale":"2021-02-10T04:54:20.153974+0000","last_undegraded":"2021-02-10T04:54:20.153974+0000","last_fullsized":"2021-02-10T04:54:20.153974+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:52:41.286006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.286006+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46300076,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":545,"num_read_kb":531689,"num_write":399,"num_write_kb":153087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"1049'46","reported_seq":"1360","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.151094+0000","last_change":"2021-02-10T04:52:35.091494+0000","last_active":"2021-02-10T04:54:20.151094+0000","last_peered":"2021-02-10T04:54:20.151094+0000","last_clean":"2021-02-10T04:54:20.151094+0000","last_became_active":"2021-02-10T04:50:59.569915+0000","last_became_peered":"2021-02-10T04:50:59.569915+0000","last_unstale":"2021-02-10T04:54:20.151094+0000","last_undegraded":"2021-02-10T04:54:20.151094+0000","last_fullsized":"2021-02-10T04:54:20.151094+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'46","last_scrub_stamp":"2021-02-10T04:52:35.091374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:35.091374+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":65,"num_write":46,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"84'7","reported_seq":"1220","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.082743+0000","last_change":"2021-02-10T04:51:52.498251+0000","last_active":"2021-02-10T04:54:20.082743+0000","last_peered":"2021-02-10T04:54:20.082743+0000","last_clean":"2021-02-10T04:54:20.082743+0000","last_became_active":"2021-02-10T04:50:59.575924+0000","last_became_peered":"2021-02-10T04:50:59.575924+0000","last_unstale":"2021-02-10T04:54:20.082743+0000","last_undegraded":"2021-02-10T04:54:20.082743+0000","last_fullsized":"2021-02-10T04:54:20.082743+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'7","last_scrub_stamp":"2021-02-10T04:51:52.498158+0000","last_deep_scrub":"84'7","last_deep_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.498158+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"1148'3","reported_seq":"1528","reported_epoch":"1205","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:23.541481+0000","last_change":"2021-02-10T04:52:32.094670+0000","last_active":"2021-02-10T04:54:23.541481+0000","last_peered":"2021-02-10T04:54:23.541481+0000","last_clean":"2021-02-10T04:52:30.059221+0000","last_became_active":"2021-02-10T04:52:32.094670+0000","last_became_peered":"2021-02-10T04:52:32.094670+0000","last_unstale":"2021-02-10T04:54:23.541481+0000","last_undegraded":"2021-02-10T04:52:32.078919+0000","last_fullsized":"2021-02-10T04:52:32.078798+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:47.407557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:47.407557+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"1113","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.154104+0000","last_change":"2021-02-10T04:53:46.727163+0000","last_active":"2021-02-10T04:54:20.154104+0000","last_peered":"2021-02-10T04:54:20.154104+0000","last_clean":"2021-02-10T04:49:56.939276+0000","last_became_active":"2021-02-10T04:53:46.727163+0000","last_became_peered":"2021-02-10T04:53:46.727163+0000","last_unstale":"2021-02-10T04:54:20.154104+0000","last_undegraded":"2021-02-10T04:54:20.154104+0000","last_fullsized":"2021-02-10T04:53:46.702239+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:23.566027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_clean_scrub_stamp":"2021-02-10T04:52:23.566027+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.16","version":"1000'490","reported_seq":"3711","reported_epoch":"1204","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:20.083740+0000","last_change":"2021-02-10T04:52:32.077228+0000","last_active":"2021-02-10T04:54:20.083740+0000","last_peered":"2021-02-10T04:54:20.083740+0000","last_clean":"2021-02-10T04:52:30.062406+0000","last_became_active":"2021-02-10T04:52:32.077228+0000","last_became_peered":"2021-02-10T04:52:32.077228+0000","last_unstale":"2021-02-10T04:54:20.083740+0000","last_undegraded":"2021-02-10T04:52:32.071522+0000","last_fullsized":"2021-02-10T04:52:32.071453+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1013,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:50:03.756248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:50:03.756248+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16154790,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"1112'40","reported_seq":"1264","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.083949+0000","last_change":"2021-02-10T04:53:10.276449+0000","last_active":"2021-02-10T04:54:20.083949+0000","last_peered":"2021-02-10T04:54:20.083949+0000","last_clean":"2021-02-10T04:54:20.083949+0000","last_became_active":"2021-02-10T04:50:59.539529+0000","last_became_peered":"2021-02-10T04:50:59.539529+0000","last_unstale":"2021-02-10T04:54:20.083949+0000","last_undegraded":"2021-02-10T04:54:20.083949+0000","last_fullsized":"2021-02-10T04:54:20.083949+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1112'40","last_scrub_stamp":"2021-02-10T04:53:10.276358+0000","last_deep_scrub":"1112'40","last_deep_scrub_stamp":"2021-02-10T04:51:01.450050+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.276358+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":59,"num_read_kb":44,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"1168","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.083115+0000","last_change":"2021-02-10T04:52:32.080868+0000","last_active":"2021-02-10T04:54:20.083115+0000","last_peered":"2021-02-10T04:54:20.083115+0000","last_clean":"2021-02-10T04:49:56.937464+0000","last_became_active":"2021-02-10T04:52:32.080868+0000","last_became_peered":"2021-02-10T04:52:32.080868+0000","last_unstale":"2021-02-10T04:54:20.083115+0000","last_undegraded":"2021-02-10T04:54:20.083115+0000","last_fullsized":"2021-02-10T04:52:32.068376+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:23.904210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:47:23.904210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"1119'6","reported_seq":"1451","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:23.024250+0000","last_change":"2021-02-10T04:52:40.038646+0000","last_active":"2021-02-10T04:54:23.024250+0000","last_peered":"2021-02-10T04:54:23.024250+0000","last_clean":"2021-02-10T04:54:23.024250+0000","last_became_active":"2021-02-10T04:51:36.869733+0000","last_became_peered":"2021-02-10T04:51:36.869733+0000","last_unstale":"2021-02-10T04:54:23.024250+0000","last_undegraded":"2021-02-10T04:54:23.024250+0000","last_fullsized":"2021-02-10T04:54:23.024250+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'6","last_scrub_stamp":"2021-02-10T04:52:40.038554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:40.038554+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"1190","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.086884+0000","last_change":"2021-02-10T04:51:36.849933+0000","last_active":"2021-02-10T04:54:20.086884+0000","last_peered":"2021-02-10T04:54:20.086884+0000","last_clean":"2021-02-10T04:54:20.086884+0000","last_became_active":"2021-02-10T04:51:36.849184+0000","last_became_peered":"2021-02-10T04:51:36.849184+0000","last_unstale":"2021-02-10T04:54:20.086884+0000","last_undegraded":"2021-02-10T04:54:20.086884+0000","last_fullsized":"2021-02-10T04:54:20.086884+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:11.779779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:11.779779+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"1032'89","reported_seq":"799","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.150737+0000","last_change":"2021-02-10T04:53:46.728868+0000","last_active":"2021-02-10T04:54:20.150737+0000","last_peered":"2021-02-10T04:54:20.150737+0000","last_clean":"2021-02-10T04:54:20.150737+0000","last_became_active":"2021-02-10T04:53:46.728356+0000","last_became_peered":"2021-02-10T04:53:46.728356+0000","last_unstale":"2021-02-10T04:54:20.150737+0000","last_undegraded":"2021-02-10T04:54:20.150737+0000","last_fullsized":"2021-02-10T04:54:20.150737+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":5,"last_scrub":"1032'89","last_scrub_stamp":"2021-02-10T04:52:34.089789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:34.089789+0000","log_size":89,"ondisk_log_size":89,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5218574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":30326,"num_write":21,"num_write_kb":12444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3210022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"1163'37","reported_seq":"886","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.083056+0000","last_change":"2021-02-10T04:53:14.202505+0000","last_active":"2021-02-10T04:54:20.083056+0000","last_peered":"2021-02-10T04:54:20.083056+0000","last_clean":"2021-02-10T04:54:20.083056+0000","last_became_active":"2021-02-10T04:51:49.496294+0000","last_became_peered":"2021-02-10T04:51:49.496294+0000","last_unstale":"2021-02-10T04:54:20.083056+0000","last_undegraded":"2021-02-10T04:54:20.083056+0000","last_fullsized":"2021-02-10T04:54:20.083056+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1163'37","last_scrub_stamp":"2021-02-10T04:53:14.202445+0000","last_deep_scrub":"1163'37","last_deep_scrub_stamp":"2021-02-10T04:53:14.202445+0000","last_clean_scrub_stamp":"2021-02-10T04:53:14.202445+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":33,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"1205","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.086103+0000","last_change":"2021-02-10T04:52:23.095651+0000","last_active":"2021-02-10T04:54:20.086103+0000","last_peered":"2021-02-10T04:54:20.086103+0000","last_clean":"2021-02-10T04:54:20.086103+0000","last_became_active":"2021-02-10T04:50:59.543531+0000","last_became_peered":"2021-02-10T04:50:59.543531+0000","last_unstale":"2021-02-10T04:54:20.086103+0000","last_undegraded":"2021-02-10T04:54:20.086103+0000","last_fullsized":"2021-02-10T04:54:20.086103+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:23.095588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:23.095588+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"1146","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.150845+0000","last_change":"2021-02-10T04:52:32.085370+0000","last_active":"2021-02-10T04:54:20.150845+0000","last_peered":"2021-02-10T04:54:20.150845+0000","last_clean":"2021-02-10T04:52:30.067066+0000","last_became_active":"2021-02-10T04:52:32.085370+0000","last_became_peered":"2021-02-10T04:52:32.085370+0000","last_unstale":"2021-02-10T04:54:20.150845+0000","last_undegraded":"2021-02-10T04:54:20.150845+0000","last_fullsized":"2021-02-10T04:52:32.074435+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:43.663762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:51:43.663762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"66","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.082479+0000","last_change":"2021-02-10T04:52:32.091922+0000","last_active":"2021-02-10T04:54:20.082479+0000","last_peered":"2021-02-10T04:54:20.082479+0000","last_clean":"2021-02-10T04:25:52.015519+0000","last_became_active":"2021-02-10T04:52:32.091922+0000","last_became_peered":"2021-02-10T04:52:32.091922+0000","last_unstale":"2021-02-10T04:54:20.082479+0000","last_undegraded":"2021-02-10T04:54:20.082479+0000","last_fullsized":"2021-02-10T04:52:32.069065+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:57.568541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:57.568541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.10","version":"1000'1071","reported_seq":"4710","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.149981+0000","last_change":"2021-02-10T04:53:35.302879+0000","last_active":"2021-02-10T04:54:20.149981+0000","last_peered":"2021-02-10T04:54:20.149981+0000","last_clean":"2021-02-10T04:54:20.149981+0000","last_became_active":"2021-02-10T04:51:39.508551+0000","last_became_peered":"2021-02-10T04:51:39.508551+0000","last_unstale":"2021-02-10T04:54:20.149981+0000","last_undegraded":"2021-02-10T04:54:20.149981+0000","last_fullsized":"2021-02-10T04:54:20.149981+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:35.302780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.302780+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"1049'31","reported_seq":"1227","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.083256+0000","last_change":"2021-02-10T04:52:39.695340+0000","last_active":"2021-02-10T04:54:20.083256+0000","last_peered":"2021-02-10T04:54:20.083256+0000","last_clean":"2021-02-10T04:54:20.083256+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:54:20.083256+0000","last_undegraded":"2021-02-10T04:54:20.083256+0000","last_fullsized":"2021-02-10T04:54:20.083256+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'31","last_scrub_stamp":"2021-02-10T04:52:39.695237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:39.695237+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":41,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"436","reported_epoch":"1204","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:20.083094+0000","last_change":"2021-02-10T04:52:32.094351+0000","last_active":"2021-02-10T04:54:20.083094+0000","last_peered":"2021-02-10T04:54:20.083094+0000","last_clean":"2021-02-10T04:52:30.061022+0000","last_became_active":"2021-02-10T04:52:32.094351+0000","last_became_peered":"2021-02-10T04:52:32.094351+0000","last_unstale":"2021-02-10T04:54:20.083094+0000","last_undegraded":"2021-02-10T04:52:32.074350+0000","last_fullsized":"2021-02-10T04:52:32.074239+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:47:08.700069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:08.700069+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"858","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.083236+0000","last_change":"2021-02-10T04:53:35.216489+0000","last_active":"2021-02-10T04:54:20.083236+0000","last_peered":"2021-02-10T04:54:20.083236+0000","last_clean":"2021-02-10T04:54:20.083236+0000","last_became_active":"2021-02-10T04:51:49.501254+0000","last_became_peered":"2021-02-10T04:51:49.501254+0000","last_unstale":"2021-02-10T04:54:20.083236+0000","last_undegraded":"2021-02-10T04:54:20.083236+0000","last_fullsized":"2021-02-10T04:54:20.083236+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.216394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.216394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"1173","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.150137+0000","last_change":"2021-02-10T04:52:32.087218+0000","last_active":"2021-02-10T04:54:20.150137+0000","last_peered":"2021-02-10T04:54:20.150137+0000","last_clean":"2021-02-10T04:52:30.062886+0000","last_became_active":"2021-02-10T04:52:32.087218+0000","last_became_peered":"2021-02-10T04:52:32.087218+0000","last_unstale":"2021-02-10T04:54:20.150137+0000","last_undegraded":"2021-02-10T04:54:20.150137+0000","last_fullsized":"2021-02-10T04:52:32.072167+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:55.814486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.814486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"1002'1657","reported_seq":"4938","reported_epoch":"1204","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:20.152139+0000","last_change":"2021-02-10T04:53:46.722538+0000","last_active":"2021-02-10T04:54:20.152139+0000","last_peered":"2021-02-10T04:54:20.152139+0000","last_clean":"2021-02-10T04:44:25.563579+0000","last_became_active":"2021-02-10T04:53:46.722538+0000","last_became_peered":"2021-02-10T04:53:46.722538+0000","last_unstale":"2021-02-10T04:54:20.152139+0000","last_undegraded":"2021-02-10T04:53:46.700604+0000","last_fullsized":"2021-02-10T04:53:46.700504+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1124,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:46:54.881412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:54.881412+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":42793067,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":424,"num_read_kb":437674,"num_write":319,"num_write_kb":118798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":17936938,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":17}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"1048'20","reported_seq":"1306","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.150395+0000","last_change":"2021-02-10T04:53:04.058324+0000","last_active":"2021-02-10T04:54:20.150395+0000","last_peered":"2021-02-10T04:54:20.150395+0000","last_clean":"2021-02-10T04:54:20.150395+0000","last_became_active":"2021-02-10T04:50:59.544484+0000","last_became_peered":"2021-02-10T04:50:59.544484+0000","last_unstale":"2021-02-10T04:54:20.150395+0000","last_undegraded":"2021-02-10T04:54:20.150395+0000","last_fullsized":"2021-02-10T04:54:20.150395+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'20","last_scrub_stamp":"2021-02-10T04:53:04.058248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:04.058248+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":20,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"1208","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.087893+0000","last_change":"2021-02-10T04:53:00.343307+0000","last_active":"2021-02-10T04:54:20.087893+0000","last_peered":"2021-02-10T04:54:20.087893+0000","last_clean":"2021-02-10T04:54:20.087893+0000","last_became_active":"2021-02-10T04:51:36.844573+0000","last_became_peered":"2021-02-10T04:51:36.844573+0000","last_unstale":"2021-02-10T04:54:20.087893+0000","last_undegraded":"2021-02-10T04:54:20.087893+0000","last_fullsized":"2021-02-10T04:54:20.087893+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:00.343228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:00.343228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"1176","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.081491+0000","last_change":"2021-02-10T04:50:56.099710+0000","last_active":"2021-02-10T04:54:20.081491+0000","last_peered":"2021-02-10T04:54:20.081491+0000","last_clean":"2021-02-10T04:54:20.081491+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:54:20.081491+0000","last_undegraded":"2021-02-10T04:54:20.081491+0000","last_fullsized":"2021-02-10T04:54:20.081491+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:56.099651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:50:56.099651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"524","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.151357+0000","last_change":"2021-02-10T04:53:46.718715+0000","last_active":"2021-02-10T04:54:20.151357+0000","last_peered":"2021-02-10T04:54:20.151357+0000","last_clean":"2021-02-10T04:54:20.151357+0000","last_became_active":"2021-02-10T04:53:46.718266+0000","last_became_peered":"2021-02-10T04:53:46.718266+0000","last_unstale":"2021-02-10T04:54:20.151357+0000","last_undegraded":"2021-02-10T04:54:20.151357+0000","last_fullsized":"2021-02-10T04:54:20.151357+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.876681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.876681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"998'1009","reported_seq":"5043","reported_epoch":"1204","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:20.084817+0000","last_change":"2021-02-10T04:52:32.090443+0000","last_active":"2021-02-10T04:54:20.084817+0000","last_peered":"2021-02-10T04:54:20.084817+0000","last_clean":"2021-02-10T04:52:30.061024+0000","last_became_active":"2021-02-10T04:52:32.090443+0000","last_became_peered":"2021-02-10T04:52:32.090443+0000","last_unstale":"2021-02-10T04:54:20.084817+0000","last_undegraded":"2021-02-10T04:52:32.076916+0000","last_fullsized":"2021-02-10T04:52:32.076790+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1127,"parent":"0.0","parent_split_bits":5,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:13.032506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:13.032506+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49420827,"num_objects":21,"num_object_clones":19,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":36763798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":21}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"1049'37","reported_seq":"1234","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.150442+0000","last_change":"2021-02-10T04:51:44.694779+0000","last_active":"2021-02-10T04:54:20.150442+0000","last_peered":"2021-02-10T04:54:20.150442+0000","last_clean":"2021-02-10T04:54:20.150442+0000","last_became_active":"2021-02-10T04:50:59.549412+0000","last_became_peered":"2021-02-10T04:50:59.549412+0000","last_unstale":"2021-02-10T04:54:20.150442+0000","last_undegraded":"2021-02-10T04:54:20.150442+0000","last_fullsized":"2021-02-10T04:54:20.150442+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:44.694711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.694711+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":64,"num_read_kb":50,"num_write":37,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"1343","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.086924+0000","last_change":"2021-02-10T04:52:33.088525+0000","last_active":"2021-02-10T04:54:20.086924+0000","last_peered":"2021-02-10T04:54:20.086924+0000","last_clean":"2021-02-10T04:54:20.086924+0000","last_became_active":"2021-02-10T04:51:36.870573+0000","last_became_peered":"2021-02-10T04:51:36.870573+0000","last_unstale":"2021-02-10T04:54:20.086924+0000","last_undegraded":"2021-02-10T04:54:20.086924+0000","last_fullsized":"2021-02-10T04:54:20.086924+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:52:33.088428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:33.088428+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"490","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.150478+0000","last_change":"2021-02-10T04:51:51.836723+0000","last_active":"2021-02-10T04:54:20.150478+0000","last_peered":"2021-02-10T04:54:20.150478+0000","last_clean":"2021-02-10T04:54:20.150478+0000","last_became_active":"2021-02-10T04:50:59.564977+0000","last_became_peered":"2021-02-10T04:50:59.564977+0000","last_unstale":"2021-02-10T04:54:20.150478+0000","last_undegraded":"2021-02-10T04:54:20.150478+0000","last_fullsized":"2021-02-10T04:54:20.150478+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.836638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.836638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"1137","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.150787+0000","last_change":"2021-02-10T04:52:32.081484+0000","last_active":"2021-02-10T04:54:20.150787+0000","last_peered":"2021-02-10T04:54:20.150787+0000","last_clean":"2021-02-10T04:52:30.067390+0000","last_became_active":"2021-02-10T04:52:32.081484+0000","last_became_peered":"2021-02-10T04:52:32.081484+0000","last_unstale":"2021-02-10T04:54:20.150787+0000","last_undegraded":"2021-02-10T04:54:20.150787+0000","last_fullsized":"2021-02-10T04:52:32.074112+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:58.514318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_clean_scrub_stamp":"2021-02-10T04:46:58.514318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1d","version":"1001'1165","reported_seq":"4197","reported_epoch":"1204","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:20.083252+0000","last_change":"2021-02-10T04:52:32.087778+0000","last_active":"2021-02-10T04:54:20.083252+0000","last_peered":"2021-02-10T04:54:20.083252+0000","last_clean":"2021-02-10T04:52:30.061103+0000","last_became_active":"2021-02-10T04:52:32.087778+0000","last_became_peered":"2021-02-10T04:52:32.087778+0000","last_unstale":"2021-02-10T04:54:20.083252+0000","last_undegraded":"2021-02-10T04:52:32.074694+0000","last_fullsized":"2021-02-10T04:52:32.074582+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1144,"parent":"0.0","parent_split_bits":5,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29601845,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":287,"num_read_kb":299530,"num_write":200,"num_write_kb":89131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":13}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"1249","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.082396+0000","last_change":"2021-02-10T04:53:01.331566+0000","last_active":"2021-02-10T04:54:20.082396+0000","last_peered":"2021-02-10T04:54:20.082396+0000","last_clean":"2021-02-10T04:54:20.082396+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:54:20.082396+0000","last_undegraded":"2021-02-10T04:54:20.082396+0000","last_fullsized":"2021-02-10T04:54:20.082396+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:01.331466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.331466+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"1098'3","reported_seq":"1527","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:23.024459+0000","last_change":"2021-02-10T04:53:34.865668+0000","last_active":"2021-02-10T04:54:23.024459+0000","last_peered":"2021-02-10T04:54:23.024459+0000","last_clean":"2021-02-10T04:54:23.024459+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:54:23.024459+0000","last_undegraded":"2021-02-10T04:54:23.024459+0000","last_fullsized":"2021-02-10T04:54:23.024459+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"1098'3","last_scrub_stamp":"2021-02-10T04:53:34.865579+0000","last_deep_scrub":"1098'3","last_deep_scrub_stamp":"2021-02-10T04:50:15.255137+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.865579+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"1054","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.152018+0000","last_change":"2021-02-10T04:53:46.727603+0000","last_active":"2021-02-10T04:54:20.152018+0000","last_peered":"2021-02-10T04:54:20.152018+0000","last_clean":"2021-02-10T04:54:20.152018+0000","last_became_active":"2021-02-10T04:53:46.727053+0000","last_became_peered":"2021-02-10T04:53:46.727053+0000","last_unstale":"2021-02-10T04:54:20.152018+0000","last_undegraded":"2021-02-10T04:54:20.152018+0000","last_fullsized":"2021-02-10T04:54:20.152018+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:47.493057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.493057+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1c","version":"1002'1090","reported_seq":"5229","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.083969+0000","last_change":"2021-02-10T04:53:41.353991+0000","last_active":"2021-02-10T04:54:20.083969+0000","last_peered":"2021-02-10T04:54:20.083969+0000","last_clean":"2021-02-10T04:54:20.083969+0000","last_became_active":"2021-02-10T04:52:31.088265+0000","last_became_peered":"2021-02-10T04:52:31.088265+0000","last_unstale":"2021-02-10T04:54:20.083969+0000","last_undegraded":"2021-02-10T04:54:20.083969+0000","last_fullsized":"2021-02-10T04:54:20.083969+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:53:41.353840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:41.353840+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"1141","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.150312+0000","last_change":"2021-02-10T04:52:32.079189+0000","last_active":"2021-02-10T04:54:20.150312+0000","last_peered":"2021-02-10T04:54:20.150312+0000","last_clean":"2021-02-10T04:52:30.063079+0000","last_became_active":"2021-02-10T04:52:32.079189+0000","last_became_peered":"2021-02-10T04:52:32.079189+0000","last_unstale":"2021-02-10T04:54:20.150312+0000","last_undegraded":"2021-02-10T04:54:20.150312+0000","last_fullsized":"2021-02-10T04:52:32.071674+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:42.537443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:42.537443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"1125","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.154320+0000","last_change":"2021-02-10T04:53:46.721135+0000","last_active":"2021-02-10T04:54:20.154320+0000","last_peered":"2021-02-10T04:54:20.154320+0000","last_clean":"2021-02-10T04:51:27.994110+0000","last_became_active":"2021-02-10T04:53:46.721135+0000","last_became_peered":"2021-02-10T04:53:46.721135+0000","last_unstale":"2021-02-10T04:54:20.154320+0000","last_undegraded":"2021-02-10T04:54:20.154320+0000","last_fullsized":"2021-02-10T04:53:46.702702+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.938731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.938731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"415","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.082650+0000","last_change":"2021-02-10T04:53:15.185748+0000","last_active":"2021-02-10T04:54:20.082650+0000","last_peered":"2021-02-10T04:54:20.082650+0000","last_clean":"2021-02-10T04:54:20.082650+0000","last_became_active":"2021-02-10T04:51:49.499348+0000","last_became_peered":"2021-02-10T04:51:49.499348+0000","last_unstale":"2021-02-10T04:54:20.082650+0000","last_undegraded":"2021-02-10T04:54:20.082650+0000","last_fullsized":"2021-02-10T04:54:20.082650+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:15.185686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:15.185686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1f","version":"1004'1093","reported_seq":"6466","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.086533+0000","last_change":"2021-02-10T04:52:31.089952+0000","last_active":"2021-02-10T04:54:20.086533+0000","last_peered":"2021-02-10T04:54:20.086533+0000","last_clean":"2021-02-10T04:54:20.086533+0000","last_became_active":"2021-02-10T04:52:31.088634+0000","last_became_peered":"2021-02-10T04:52:31.088634+0000","last_unstale":"2021-02-10T04:54:20.086533+0000","last_undegraded":"2021-02-10T04:54:20.086533+0000","last_fullsized":"2021-02-10T04:54:20.086533+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39434697,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":279,"num_read_kb":311058,"num_write":186,"num_write_kb":66790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":53995913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"1193","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.085561+0000","last_change":"2021-02-10T04:52:36.083849+0000","last_active":"2021-02-10T04:54:20.085561+0000","last_peered":"2021-02-10T04:54:20.085561+0000","last_clean":"2021-02-10T04:54:20.085561+0000","last_became_active":"2021-02-10T04:25:36.336442+0000","last_became_peered":"2021-02-10T04:25:36.336442+0000","last_unstale":"2021-02-10T04:54:20.085561+0000","last_undegraded":"2021-02-10T04:54:20.085561+0000","last_fullsized":"2021-02-10T04:54:20.085561+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:52:36.083784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:36.083784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"418","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.083537+0000","last_change":"2021-02-10T04:52:32.084893+0000","last_active":"2021-02-10T04:54:20.083537+0000","last_peered":"2021-02-10T04:54:20.083537+0000","last_clean":"2021-02-10T04:52:30.061372+0000","last_became_active":"2021-02-10T04:52:32.084893+0000","last_became_peered":"2021-02-10T04:52:32.084893+0000","last_unstale":"2021-02-10T04:54:20.083537+0000","last_undegraded":"2021-02-10T04:54:20.083537+0000","last_fullsized":"2021-02-10T04:52:32.073304+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:08.758129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.758129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"1172","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.083475+0000","last_change":"2021-02-10T04:51:44.799512+0000","last_active":"2021-02-10T04:54:20.083475+0000","last_peered":"2021-02-10T04:54:20.083475+0000","last_clean":"2021-02-10T04:54:20.083475+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:54:20.083475+0000","last_undegraded":"2021-02-10T04:54:20.083475+0000","last_fullsized":"2021-02-10T04:54:20.083475+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:44.799429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.799429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1e","version":"1063'654","reported_seq":"4448","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.082863+0000","last_change":"2021-02-10T04:53:40.135450+0000","last_active":"2021-02-10T04:54:20.082863+0000","last_peered":"2021-02-10T04:54:20.082863+0000","last_clean":"2021-02-10T04:54:20.082863+0000","last_became_active":"2021-02-10T04:52:31.087662+0000","last_became_peered":"2021-02-10T04:52:31.087662+0000","last_unstale":"2021-02-10T04:54:20.082863+0000","last_undegraded":"2021-02-10T04:54:20.082863+0000","last_fullsized":"2021-02-10T04:54:20.082863+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:53:40.135367+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:53:40.135367+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"1156","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.151097+0000","last_change":"2021-02-10T04:51:08.006463+0000","last_active":"2021-02-10T04:54:20.151097+0000","last_peered":"2021-02-10T04:54:20.151097+0000","last_clean":"2021-02-10T04:54:20.151097+0000","last_became_active":"2021-02-10T04:50:59.563657+0000","last_became_peered":"2021-02-10T04:50:59.563657+0000","last_unstale":"2021-02-10T04:54:20.151097+0000","last_undegraded":"2021-02-10T04:54:20.151097+0000","last_fullsized":"2021-02-10T04:54:20.151097+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_clean_scrub_stamp":"2021-02-10T04:51:02.455702+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"1207","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.083449+0000","last_change":"2021-02-10T04:52:33.741801+0000","last_active":"2021-02-10T04:54:20.083449+0000","last_peered":"2021-02-10T04:54:20.083449+0000","last_clean":"2021-02-10T04:54:20.083449+0000","last_became_active":"2021-02-10T04:51:36.860129+0000","last_became_peered":"2021-02-10T04:51:36.860129+0000","last_unstale":"2021-02-10T04:54:20.083449+0000","last_undegraded":"2021-02-10T04:54:20.083449+0000","last_fullsized":"2021-02-10T04:54:20.083449+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:33.741711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:46:58.666264+0000","last_clean_scrub_stamp":"2021-02-10T04:52:33.741711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"1168","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.151406+0000","last_change":"2021-02-10T04:52:25.030419+0000","last_active":"2021-02-10T04:54:20.151406+0000","last_peered":"2021-02-10T04:54:20.151406+0000","last_clean":"2021-02-10T04:54:20.151406+0000","last_became_active":"2021-02-10T04:50:59.539582+0000","last_became_peered":"2021-02-10T04:50:59.539582+0000","last_unstale":"2021-02-10T04:54:20.151406+0000","last_undegraded":"2021-02-10T04:54:20.151406+0000","last_fullsized":"2021-02-10T04:54:20.151406+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:25.030281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.030281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"1003'739","reported_seq":"3953","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.081202+0000","last_change":"2021-02-10T04:53:01.929261+0000","last_active":"2021-02-10T04:54:20.081202+0000","last_peered":"2021-02-10T04:54:20.081202+0000","last_clean":"2021-02-10T04:54:20.081202+0000","last_became_active":"2021-02-10T04:51:40.511505+0000","last_became_peered":"2021-02-10T04:51:40.511505+0000","last_unstale":"2021-02-10T04:54:20.081202+0000","last_undegraded":"2021-02-10T04:54:20.081202+0000","last_fullsized":"2021-02-10T04:54:20.081202+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:53:01.929169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.929169+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22230846,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":18429397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"1112'47","reported_seq":"91","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.081361+0000","last_change":"2021-02-10T04:53:21.334003+0000","last_active":"2021-02-10T04:54:20.081361+0000","last_peered":"2021-02-10T04:54:20.081361+0000","last_clean":"2021-02-10T04:54:20.081361+0000","last_became_active":"2021-02-10T04:53:20.297145+0000","last_became_peered":"2021-02-10T04:53:20.297145+0000","last_unstale":"2021-02-10T04:54:20.081361+0000","last_undegraded":"2021-02-10T04:54:20.081361+0000","last_fullsized":"2021-02-10T04:54:20.081361+0000","mapping_epoch":1166,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"1112'47","last_scrub_stamp":"2021-02-10T04:51:02.860835+0000","last_deep_scrub":"1112'47","last_deep_scrub_stamp":"2021-02-10T04:51:02.860835+0000","last_clean_scrub_stamp":"2021-02-10T04:51:02.860835+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":71,"num_read_kb":55,"num_write":38,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"1045","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.151666+0000","last_change":"2021-02-10T04:53:46.727590+0000","last_active":"2021-02-10T04:54:20.151666+0000","last_peered":"2021-02-10T04:54:20.151666+0000","last_clean":"2021-02-10T04:54:20.151666+0000","last_became_active":"2021-02-10T04:53:46.726584+0000","last_became_peered":"2021-02-10T04:53:46.726584+0000","last_unstale":"2021-02-10T04:54:20.151666+0000","last_undegraded":"2021-02-10T04:54:20.151666+0000","last_fullsized":"2021-02-10T04:54:20.151666+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:26.408326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.408326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"1205","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.083671+0000","last_change":"2021-02-10T04:53:07.324650+0000","last_active":"2021-02-10T04:54:20.083671+0000","last_peered":"2021-02-10T04:54:20.083671+0000","last_clean":"2021-02-10T04:54:20.083671+0000","last_became_active":"2021-02-10T04:50:59.575340+0000","last_became_peered":"2021-02-10T04:50:59.575340+0000","last_unstale":"2021-02-10T04:54:20.083671+0000","last_undegraded":"2021-02-10T04:54:20.083671+0000","last_fullsized":"2021-02-10T04:54:20.083671+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.324564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.324564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"65","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.081768+0000","last_change":"2021-02-10T04:52:32.091240+0000","last_active":"2021-02-10T04:54:20.081768+0000","last_peered":"2021-02-10T04:54:20.081768+0000","last_clean":"2021-02-10T04:25:52.015519+0000","last_became_active":"2021-02-10T04:52:32.091240+0000","last_became_peered":"2021-02-10T04:52:32.091240+0000","last_unstale":"2021-02-10T04:54:20.081768+0000","last_undegraded":"2021-02-10T04:54:20.081768+0000","last_fullsized":"2021-02-10T04:52:32.077955+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:00.501127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:00.501127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.18","version":"259'74","reported_seq":"67","reported_epoch":"1204","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:20.081291+0000","last_change":"2021-02-10T04:52:32.083683+0000","last_active":"2021-02-10T04:54:20.081291+0000","last_peered":"2021-02-10T04:54:20.081291+0000","last_clean":"2021-02-10T04:31:15.165244+0000","last_became_active":"2021-02-10T04:52:32.083683+0000","last_became_peered":"2021-02-10T04:52:32.083683+0000","last_unstale":"2021-02-10T04:54:20.081291+0000","last_undegraded":"2021-02-10T04:52:32.067776+0000","last_fullsized":"2021-02-10T04:52:32.067560+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1010,"parent":"0.0","parent_split_bits":5,"last_scrub":"259'74","last_scrub_stamp":"2021-02-10T04:52:26.597668+0000","last_deep_scrub":"259'74","last_deep_scrub_stamp":"2021-02-10T04:46:49.745104+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.597668+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1218524,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":28228,"num_write":19,"num_write_kb":7022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"1061'20","reported_seq":"1107","reported_epoch":"1204","state":"active+clean+remapped","last_fresh":"2021-02-10T04:54:20.083410+0000","last_change":"2021-02-10T04:53:21.301748+0000","last_active":"2021-02-10T04:54:20.083410+0000","last_peered":"2021-02-10T04:54:20.083410+0000","last_clean":"2021-02-10T04:54:20.083410+0000","last_became_active":"2021-02-10T04:53:21.301122+0000","last_became_peered":"2021-02-10T04:53:21.301122+0000","last_unstale":"2021-02-10T04:54:20.083410+0000","last_undegraded":"2021-02-10T04:54:20.083410+0000","last_fullsized":"2021-02-10T04:54:20.083410+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"1061'20","last_scrub_stamp":"2021-02-10T04:52:31.770907+0000","last_deep_scrub":"1061'20","last_deep_scrub_stamp":"2021-02-10T04:46:55.646261+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.770907+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":17,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,3],"avail_no_missing":["1","3","4"],"object_location_counts":[{"shards":"1,3,4","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"1067","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.152790+0000","last_change":"2021-02-10T04:53:46.724425+0000","last_active":"2021-02-10T04:54:20.152790+0000","last_peered":"2021-02-10T04:54:20.152790+0000","last_clean":"2021-02-10T04:52:30.068110+0000","last_became_active":"2021-02-10T04:53:46.724425+0000","last_became_peered":"2021-02-10T04:53:46.724425+0000","last_unstale":"2021-02-10T04:54:20.152790+0000","last_undegraded":"2021-02-10T04:54:20.152790+0000","last_fullsized":"2021-02-10T04:53:46.707467+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:59.167911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:50:59.167911+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"115","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.151100+0000","last_change":"2021-02-10T04:53:46.718512+0000","last_active":"2021-02-10T04:54:20.151100+0000","last_peered":"2021-02-10T04:54:20.151100+0000","last_clean":"2021-02-10T04:54:20.151100+0000","last_became_active":"2021-02-10T04:53:46.716811+0000","last_became_peered":"2021-02-10T04:53:46.716811+0000","last_unstale":"2021-02-10T04:54:20.151100+0000","last_undegraded":"2021-02-10T04:54:20.151100+0000","last_fullsized":"2021-02-10T04:54:20.151100+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:45.813676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.813676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"1194","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.082404+0000","last_change":"2021-02-10T04:53:38.363899+0000","last_active":"2021-02-10T04:54:20.082404+0000","last_peered":"2021-02-10T04:54:20.082404+0000","last_clean":"2021-02-10T04:54:20.082404+0000","last_became_active":"2021-02-10T04:50:59.539256+0000","last_became_peered":"2021-02-10T04:50:59.539256+0000","last_unstale":"2021-02-10T04:54:20.082404+0000","last_undegraded":"2021-02-10T04:54:20.082404+0000","last_fullsized":"2021-02-10T04:54:20.082404+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:38.363825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:09.433793+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.363825+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1b","version":"1003'1689","reported_seq":"4635","reported_epoch":"1204","state":"active+clean+remapped","last_fresh":"2021-02-10T04:54:20.081878+0000","last_change":"2021-02-10T04:53:10.075976+0000","last_active":"2021-02-10T04:54:20.081878+0000","last_peered":"2021-02-10T04:54:20.081878+0000","last_clean":"2021-02-10T04:54:20.081878+0000","last_became_active":"2021-02-10T04:52:33.087105+0000","last_became_peered":"2021-02-10T04:52:33.087105+0000","last_unstale":"2021-02-10T04:54:20.081878+0000","last_undegraded":"2021-02-10T04:54:20.081878+0000","last_fullsized":"2021-02-10T04:54:20.081878+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:53:10.075879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.075879+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27159495,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3,1],"avail_no_missing":["2","1","3"],"object_location_counts":[{"shards":"1,2,3","objects":12}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"1048'44","reported_seq":"88","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.082138+0000","last_change":"2021-02-10T04:53:32.373404+0000","last_active":"2021-02-10T04:54:20.082138+0000","last_peered":"2021-02-10T04:54:20.082138+0000","last_clean":"2021-02-10T04:54:20.082138+0000","last_became_active":"2021-02-10T04:53:20.304544+0000","last_became_peered":"2021-02-10T04:53:20.304544+0000","last_unstale":"2021-02-10T04:54:20.082138+0000","last_undegraded":"2021-02-10T04:54:20.082138+0000","last_fullsized":"2021-02-10T04:54:20.082138+0000","mapping_epoch":1166,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1167,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'44","last_scrub_stamp":"2021-02-10T04:53:32.373330+0000","last_deep_scrub":"78'32","last_deep_scrub_stamp":"2021-02-10T04:27:09.170428+0000","last_clean_scrub_stamp":"2021-02-10T04:53:32.373330+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":64,"num_read_kb":45,"num_write":34,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"1180","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.151360+0000","last_change":"2021-02-10T04:52:32.087531+0000","last_active":"2021-02-10T04:54:20.151360+0000","last_peered":"2021-02-10T04:54:20.151360+0000","last_clean":"2021-02-10T04:52:30.066743+0000","last_became_active":"2021-02-10T04:52:32.087531+0000","last_became_peered":"2021-02-10T04:52:32.087531+0000","last_unstale":"2021-02-10T04:54:20.151360+0000","last_undegraded":"2021-02-10T04:54:20.151360+0000","last_fullsized":"2021-02-10T04:52:32.076804+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:33.543559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:46:33.543559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"864","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.082234+0000","last_change":"2021-02-10T04:53:04.354508+0000","last_active":"2021-02-10T04:54:20.082234+0000","last_peered":"2021-02-10T04:54:20.082234+0000","last_clean":"2021-02-10T04:54:20.082234+0000","last_became_active":"2021-02-10T04:33:05.169159+0000","last_became_peered":"2021-02-10T04:33:05.169159+0000","last_unstale":"2021-02-10T04:54:20.082234+0000","last_undegraded":"2021-02-10T04:54:20.082234+0000","last_fullsized":"2021-02-10T04:54:20.082234+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:04.354409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:04.354409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"1258","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.082548+0000","last_change":"2021-02-10T04:52:36.777554+0000","last_active":"2021-02-10T04:54:20.082548+0000","last_peered":"2021-02-10T04:54:20.082548+0000","last_clean":"2021-02-10T04:54:20.082548+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:54:20.082548+0000","last_undegraded":"2021-02-10T04:54:20.082548+0000","last_fullsized":"2021-02-10T04:54:20.082548+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:36.777485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:52:36.777485+0000","last_clean_scrub_stamp":"2021-02-10T04:52:36.777485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1a","version":"1001'1528","reported_seq":"5815","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.082198+0000","last_change":"2021-02-10T04:52:31.086406+0000","last_active":"2021-02-10T04:54:20.082198+0000","last_peered":"2021-02-10T04:54:20.082198+0000","last_clean":"2021-02-10T04:54:20.082198+0000","last_became_active":"2021-02-10T04:52:31.085959+0000","last_became_peered":"2021-02-10T04:52:31.085959+0000","last_unstale":"2021-02-10T04:54:20.082198+0000","last_undegraded":"2021-02-10T04:54:20.082198+0000","last_fullsized":"2021-02-10T04:54:20.082198+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46944136,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":358,"num_read_kb":351935,"num_write":286,"num_write_kb":111355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":21311893,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"1049'68","reported_seq":"102","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.082435+0000","last_change":"2021-02-10T04:53:34.122240+0000","last_active":"2021-02-10T04:54:20.082435+0000","last_peered":"2021-02-10T04:54:20.082435+0000","last_clean":"2021-02-10T04:54:20.082435+0000","last_became_active":"2021-02-10T04:53:20.296560+0000","last_became_peered":"2021-02-10T04:53:20.296560+0000","last_unstale":"2021-02-10T04:54:20.082435+0000","last_undegraded":"2021-02-10T04:54:20.082435+0000","last_fullsized":"2021-02-10T04:54:20.082435+0000","mapping_epoch":1166,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1167,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'68","last_scrub_stamp":"2021-02-10T04:53:34.122152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.122152+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":124,"num_read_kb":101,"num_write":69,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"30","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.154918+0000","last_change":"2021-02-10T04:53:46.719798+0000","last_active":"2021-02-10T04:54:20.154918+0000","last_peered":"2021-02-10T04:54:20.154918+0000","last_clean":"2021-02-10T04:25:33.303710+0000","last_became_active":"2021-02-10T04:53:46.719798+0000","last_became_peered":"2021-02-10T04:53:46.719798+0000","last_unstale":"2021-02-10T04:54:20.154918+0000","last_undegraded":"2021-02-10T04:54:20.154918+0000","last_fullsized":"2021-02-10T04:53:46.703571+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:17.919745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:17.919745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"490","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.149966+0000","last_change":"2021-02-10T04:53:07.092032+0000","last_active":"2021-02-10T04:54:20.149966+0000","last_peered":"2021-02-10T04:54:20.149966+0000","last_clean":"2021-02-10T04:54:20.149966+0000","last_became_active":"2021-02-10T04:50:59.572077+0000","last_became_peered":"2021-02-10T04:50:59.572077+0000","last_unstale":"2021-02-10T04:54:20.149966+0000","last_undegraded":"2021-02-10T04:54:20.149966+0000","last_fullsized":"2021-02-10T04:54:20.149966+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.091949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:19.755718+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.091949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"1167","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.151008+0000","last_change":"2021-02-10T04:51:59.838856+0000","last_active":"2021-02-10T04:54:20.151008+0000","last_peered":"2021-02-10T04:54:20.151008+0000","last_clean":"2021-02-10T04:54:20.151008+0000","last_became_active":"2021-02-10T04:50:59.563156+0000","last_became_peered":"2021-02-10T04:50:59.563156+0000","last_unstale":"2021-02-10T04:54:20.151008+0000","last_undegraded":"2021-02-10T04:54:20.151008+0000","last_fullsized":"2021-02-10T04:54:20.151008+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.838790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.838790+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":"900","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.083888+0000","last_change":"2021-02-10T04:52:30.405170+0000","last_active":"2021-02-10T04:54:20.083888+0000","last_peered":"2021-02-10T04:54:20.083888+0000","last_clean":"2021-02-10T04:54:20.083888+0000","last_became_active":"2021-02-10T04:51:36.852058+0000","last_became_peered":"2021-02-10T04:51:36.852058+0000","last_unstale":"2021-02-10T04:54:20.083888+0000","last_undegraded":"2021-02-10T04:54:20.083888+0000","last_fullsized":"2021-02-10T04:54:20.083888+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.405106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:20.635536+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.405106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.20","version":"0'0","reported_seq":"139","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.152846+0000","last_change":"2021-02-10T04:53:46.739888+0000","last_active":"2021-02-10T04:54:20.152846+0000","last_peered":"2021-02-10T04:54:20.152846+0000","last_clean":"2021-02-10T04:54:20.152846+0000","last_became_active":"2021-02-10T04:53:46.739532+0000","last_became_peered":"2021-02-10T04:53:46.739532+0000","last_unstale":"2021-02-10T04:54:20.152846+0000","last_undegraded":"2021-02-10T04:54:20.152846+0000","last_fullsized":"2021-02-10T04:54:20.152846+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:56.449085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:41.375276+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.449085+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":"1132","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.086888+0000","last_change":"2021-02-10T04:53:35.858634+0000","last_active":"2021-02-10T04:54:20.086888+0000","last_peered":"2021-02-10T04:54:20.086888+0000","last_clean":"2021-02-10T04:54:20.086888+0000","last_became_active":"2021-02-10T04:51:36.869785+0000","last_became_peered":"2021-02-10T04:51:36.869785+0000","last_unstale":"2021-02-10T04:54:20.086888+0000","last_undegraded":"2021-02-10T04:54:20.086888+0000","last_fullsized":"2021-02-10T04:54:20.086888+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.858557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.858557+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":"102","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.084731+0000","last_change":"2021-02-10T04:52:32.094666+0000","last_active":"2021-02-10T04:54:20.084731+0000","last_peered":"2021-02-10T04:54:20.084731+0000","last_clean":"2021-02-10T04:52:30.064564+0000","last_became_active":"2021-02-10T04:52:32.094666+0000","last_became_peered":"2021-02-10T04:52:32.094666+0000","last_unstale":"2021-02-10T04:54:20.084731+0000","last_undegraded":"2021-02-10T04:54:20.084731+0000","last_fullsized":"2021-02-10T04:52:32.084052+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:47.481686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:47.481686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":"93","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.082511+0000","last_change":"2021-02-10T04:52:32.096027+0000","last_active":"2021-02-10T04:54:20.082511+0000","last_peered":"2021-02-10T04:54:20.082511+0000","last_clean":"2021-02-10T04:52:30.061012+0000","last_became_active":"2021-02-10T04:52:32.096027+0000","last_became_peered":"2021-02-10T04:52:32.096027+0000","last_unstale":"2021-02-10T04:54:20.082511+0000","last_undegraded":"2021-02-10T04:54:20.082511+0000","last_fullsized":"2021-02-10T04:52:32.076629+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.706372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.706372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":"1214","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.085225+0000","last_change":"2021-02-10T04:51:57.287452+0000","last_active":"2021-02-10T04:54:20.085225+0000","last_peered":"2021-02-10T04:54:20.085225+0000","last_clean":"2021-02-10T04:54:20.085225+0000","last_became_active":"2021-02-10T04:50:59.566827+0000","last_became_peered":"2021-02-10T04:50:59.566827+0000","last_unstale":"2021-02-10T04:54:20.085225+0000","last_undegraded":"2021-02-10T04:54:20.085225+0000","last_fullsized":"2021-02-10T04:54:20.085225+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.287363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.287363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":"179","reported_epoch":"1204","state":"active+undersized","last_fresh":"2021-02-10T04:54:20.086584+0000","last_change":"2021-02-10T04:52:32.082221+0000","last_active":"2021-02-10T04:54:20.086584+0000","last_peered":"2021-02-10T04:54:20.086584+0000","last_clean":"2021-02-10T04:52:30.068335+0000","last_became_active":"2021-02-10T04:52:32.082221+0000","last_became_peered":"2021-02-10T04:52:32.082221+0000","last_unstale":"2021-02-10T04:54:20.086584+0000","last_undegraded":"2021-02-10T04:54:20.086584+0000","last_fullsized":"2021-02-10T04:52:32.077664+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:10.308282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:10.308282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":"1103","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.151090+0000","last_change":"2021-02-10T04:53:09.121379+0000","last_active":"2021-02-10T04:54:20.151090+0000","last_peered":"2021-02-10T04:54:20.151090+0000","last_clean":"2021-02-10T04:54:20.151090+0000","last_became_active":"2021-02-10T04:50:59.561442+0000","last_became_peered":"2021-02-10T04:50:59.561442+0000","last_unstale":"2021-02-10T04:54:20.151090+0000","last_undegraded":"2021-02-10T04:54:20.151090+0000","last_fullsized":"2021-02-10T04:54:20.151090+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:09.121298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.121298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.21","version":"293'77","reported_seq":"775","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.150330+0000","last_change":"2021-02-10T04:52:28.062923+0000","last_active":"2021-02-10T04:54:20.150330+0000","last_peered":"2021-02-10T04:54:20.150330+0000","last_clean":"2021-02-10T04:54:20.150330+0000","last_became_active":"2021-02-10T04:52:28.062463+0000","last_became_peered":"2021-02-10T04:52:28.062463+0000","last_unstale":"2021-02-10T04:54:20.150330+0000","last_undegraded":"2021-02-10T04:54:20.150330+0000","last_fullsized":"2021-02-10T04:54:20.150330+0000","mapping_epoch":1143,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1144,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1886586,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":15163,"num_write":9,"num_write_kb":3316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":"101","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.084057+0000","last_change":"2021-02-10T04:52:32.124955+0000","last_active":"2021-02-10T04:54:20.084057+0000","last_peered":"2021-02-10T04:54:20.084057+0000","last_clean":"2021-02-10T04:54:20.084057+0000","last_became_active":"2021-02-10T04:51:36.867061+0000","last_became_peered":"2021-02-10T04:51:36.867061+0000","last_unstale":"2021-02-10T04:54:20.084057+0000","last_undegraded":"2021-02-10T04:54:20.084057+0000","last_fullsized":"2021-02-10T04:54:20.084057+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:32.124901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:08.271977+0000","last_clean_scrub_stamp":"2021-02-10T04:52:32.124901+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":"1143","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.086238+0000","last_change":"2021-02-10T04:51:07.182775+0000","last_active":"2021-02-10T04:54:20.086238+0000","last_peered":"2021-02-10T04:54:20.086238+0000","last_clean":"2021-02-10T04:54:20.086238+0000","last_became_active":"2021-02-10T04:28:23.126368+0000","last_became_peered":"2021-02-10T04:28:23.126368+0000","last_unstale":"2021-02-10T04:54:20.086238+0000","last_undegraded":"2021-02-10T04:54:20.086238+0000","last_fullsized":"2021-02-10T04:54:20.086238+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.182678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.182678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.20","version":"1000'1071","reported_seq":"4549","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.080870+0000","last_change":"2021-02-10T04:53:03.915465+0000","last_active":"2021-02-10T04:54:20.080870+0000","last_peered":"2021-02-10T04:54:20.080870+0000","last_clean":"2021-02-10T04:54:20.080870+0000","last_became_active":"2021-02-10T04:52:31.088690+0000","last_became_peered":"2021-02-10T04:52:31.088690+0000","last_unstale":"2021-02-10T04:54:20.080870+0000","last_undegraded":"2021-02-10T04:54:20.080870+0000","last_fullsized":"2021-02-10T04:54:20.080870+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:03.915403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.915403+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16398893,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":"728","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.150636+0000","last_change":"2021-02-10T04:53:46.721599+0000","last_active":"2021-02-10T04:54:20.150636+0000","last_peered":"2021-02-10T04:54:20.150636+0000","last_clean":"2021-02-10T04:54:20.150636+0000","last_became_active":"2021-02-10T04:53:46.721340+0000","last_became_peered":"2021-02-10T04:53:46.721340+0000","last_unstale":"2021-02-10T04:54:20.150636+0000","last_undegraded":"2021-02-10T04:54:20.150636+0000","last_fullsized":"2021-02-10T04:54:20.150636+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:28.092288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:07.767250+0000","last_clean_scrub_stamp":"2021-02-10T04:52:28.092288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.25","version":"53'2","reported_seq":"1159","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.083152+0000","last_change":"2021-02-10T04:53:36.103026+0000","last_active":"2021-02-10T04:54:20.083152+0000","last_peered":"2021-02-10T04:54:20.083152+0000","last_clean":"2021-02-10T04:54:20.083152+0000","last_became_active":"2021-02-10T04:28:23.467770+0000","last_became_peered":"2021-02-10T04:28:23.467770+0000","last_unstale":"2021-02-10T04:54:20.083152+0000","last_undegraded":"2021-02-10T04:54:20.083152+0000","last_fullsized":"2021-02-10T04:54:20.083152+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:53:36.102928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:36.102928+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.23","version":"1002'1657","reported_seq":"5075","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.085273+0000","last_change":"2021-02-10T04:52:31.093874+0000","last_active":"2021-02-10T04:54:20.085273+0000","last_peered":"2021-02-10T04:54:20.085273+0000","last_clean":"2021-02-10T04:54:20.085273+0000","last_became_active":"2021-02-10T04:52:31.092996+0000","last_became_peered":"2021-02-10T04:52:31.092996+0000","last_unstale":"2021-02-10T04:54:20.085273+0000","last_undegraded":"2021-02-10T04:54:20.085273+0000","last_fullsized":"2021-02-10T04:54:20.085273+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21396533,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":212,"num_read_kb":218837,"num_write":159,"num_write_kb":59399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":20479176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":"1250","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.081453+0000","last_change":"2021-02-10T04:53:12.951612+0000","last_active":"2021-02-10T04:54:20.081453+0000","last_peered":"2021-02-10T04:54:20.081453+0000","last_clean":"2021-02-10T04:54:20.081453+0000","last_became_active":"2021-02-10T04:50:59.566548+0000","last_became_peered":"2021-02-10T04:50:59.566548+0000","last_unstale":"2021-02-10T04:54:20.081453+0000","last_undegraded":"2021-02-10T04:54:20.081453+0000","last_fullsized":"2021-02-10T04:54:20.081453+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.951528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.951528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.26","version":"53'1","reported_seq":"42","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.153960+0000","last_change":"2021-02-10T04:53:46.722636+0000","last_active":"2021-02-10T04:54:20.153960+0000","last_peered":"2021-02-10T04:54:20.153960+0000","last_clean":"2021-02-10T04:54:20.153960+0000","last_became_active":"2021-02-10T04:53:46.722227+0000","last_became_peered":"2021-02-10T04:53:46.722227+0000","last_unstale":"2021-02-10T04:54:20.153960+0000","last_undegraded":"2021-02-10T04:54:20.153960+0000","last_fullsized":"2021-02-10T04:54:20.153960+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:53:03.933910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.933910+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":423,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.22","version":"998'1009","reported_seq":"4566","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.082283+0000","last_change":"2021-02-10T04:52:31.086563+0000","last_active":"2021-02-10T04:54:20.082283+0000","last_peered":"2021-02-10T04:54:20.082283+0000","last_clean":"2021-02-10T04:54:20.082283+0000","last_became_active":"2021-02-10T04:52:31.085288+0000","last_became_peered":"2021-02-10T04:52:31.085288+0000","last_unstale":"2021-02-10T04:54:20.082283+0000","last_undegraded":"2021-02-10T04:54:20.082283+0000","last_fullsized":"2021-02-10T04:54:20.082283+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12355207,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":120,"num_read_kb":124702,"num_write":77,"num_write_kb":25052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4145029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":"1132","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.085190+0000","last_change":"2021-02-10T04:53:10.958668+0000","last_active":"2021-02-10T04:54:20.085190+0000","last_peered":"2021-02-10T04:54:20.085190+0000","last_clean":"2021-02-10T04:54:20.085190+0000","last_became_active":"2021-02-10T04:45:49.552026+0000","last_became_peered":"2021-02-10T04:45:49.552026+0000","last_unstale":"2021-02-10T04:54:20.085190+0000","last_undegraded":"2021-02-10T04:54:20.085190+0000","last_fullsized":"2021-02-10T04:54:20.085190+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:10.958579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.958579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":"147","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.154824+0000","last_change":"2021-02-10T04:53:46.725941+0000","last_active":"2021-02-10T04:54:20.154824+0000","last_peered":"2021-02-10T04:54:20.154824+0000","last_clean":"2021-02-10T04:54:20.154824+0000","last_became_active":"2021-02-10T04:53:46.725587+0000","last_became_peered":"2021-02-10T04:53:46.725587+0000","last_unstale":"2021-02-10T04:54:20.154824+0000","last_undegraded":"2021-02-10T04:54:20.154824+0000","last_fullsized":"2021-02-10T04:54:20.154824+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:41.758684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.758684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":"1132","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.150169+0000","last_change":"2021-02-10T04:53:11.139690+0000","last_active":"2021-02-10T04:54:20.150169+0000","last_peered":"2021-02-10T04:54:20.150169+0000","last_clean":"2021-02-10T04:54:20.150169+0000","last_became_active":"2021-02-10T04:50:59.569162+0000","last_became_peered":"2021-02-10T04:50:59.569162+0000","last_unstale":"2021-02-10T04:54:20.150169+0000","last_undegraded":"2021-02-10T04:54:20.150169+0000","last_fullsized":"2021-02-10T04:54:20.150169+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:11.139602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.139602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":"1158","reported_epoch":"1204","state":"active+clean","last_fresh":"2021-02-10T04:54:20.081298+0000","last_change":"2021-02-10T04:52:24.072005+0000","last_active":"2021-02-10T04:54:20.081298+0000","last_peered":"2021-02-10T04:54:20.081298+0000","last_clean":"2021-02-10T04:54:20.081298+0000","last_became_active":"2021-02-10T04:50:59.527945+0000","last_became_peered":"2021-02-10T04:50:59.527945+0000","last_unstale":"2021-02-10T04:54:20.081298+0000","last_undegraded":"2021-02-10T04:54:20.081298+0000","last_fullsized":"2021-02-10T04:54:20.081298+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:24.071915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_clean_scrub_stamp":"2021-02-10T04:52:24.071915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":931687135,"num_objects":374,"num_object_clones":323,"num_object_copies":1122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":164,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":374,"num_whiteouts":10,"num_read":4479,"num_read_kb":4587256,"num_write":3265,"num_write_kb":1255814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":630,"num_bytes_recovered":307448165,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2029686784,"data_stored":2612817805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34722,"ondisk_log_size":34722,"up":94,"acting":95,"num_store_stats":8},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":2368,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":18,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1069,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":5835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":111,"acting":111,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":29,"acting":29,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":590,"num_read_kb":585,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":8,"acting":8,"num_store_stats":7},{"poolid":4,"num_pg":28,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":43,"num_objects_misplaced":32,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":2289,"num_read_kb":1762,"num_write":1306,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":17828,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2093056,"data_stored":75230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1082,"ondisk_log_size":1082,"up":76,"acting":80,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":84,"acting":84,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":114,"acting":114,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365148,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":189700,"kb_used_data":148540,"kb_used_omap":0,"kb_used_meta":41152,"kb_avail":93555452,"statfs":{"total":95995035648,"available":95800782848,"internally_reserved":0,"allocated":152104960,"data_stored":201981689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42139648},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:27 2021","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":1.2370000000000001,"15min":1.284},"min":{"1min":0.50800000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1220000000000001,"5min":34.299999999999997,"15min":67.277000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.5109999999999999,"15min":1.548},"min":{"1min":0.46300000000000002,"5min":0.438,"15min":0.438},"max":{"1min":2.8690000000000002,"5min":34.652999999999999,"15min":66.933999999999997},"last":0.71399999999999997}]},{"osd":1,"last update":"Wed Feb 10 04:53:36 2021","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.85099999999999998,"15min":0.97299999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.278,"5min":2.254,"15min":10.140000000000001},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.92800000000000005,"15min":0.98999999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.024,"5min":2.5409999999999999,"15min":7.1520000000000001},"last":0.73599999999999999}]},{"osd":2,"last update":"Wed Feb 10 04:53:36 2021","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":1.179,"15min":1.3500000000000001},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.4590000000000001,"5min":11.468,"15min":66.534000000000006},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.84299999999999997,"5min":1.1519999999999999,"15min":1.5249999999999999},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.075,"5min":3.8820000000000001,"15min":66.781999999999996},"last":0.63100000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:53:59 2021","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.2190000000000001,"15min":1.3180000000000001},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.573,"5min":4.5890000000000004,"15min":4.5890000000000004},"last":1.087},{"interface":"front","average":{"1min":1.034,"5min":1.306,"15min":1.425},"min":{"1min":0.73199999999999998,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.593,"5min":4.6609999999999996,"15min":4.6609999999999996},"last":0.84099999999999997}]},{"osd":4,"last update":"Wed Feb 10 04:53:36 2021","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.343,"15min":1.47},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.625,"5min":4.694,"15min":10.781000000000001},"last":0.81200000000000006},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.2889999999999999,"15min":1.3700000000000001},"min":{"1min":0.71999999999999997,"5min":0.69099999999999995,"15min":0.65100000000000002},"max":{"1min":1.843,"5min":3.4049999999999998,"15min":10.326000000000001},"last":1.153}]},{"osd":5,"last update":"Wed Feb 10 04:53:36 2021","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.4379999999999999,"15min":1.621},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.66700000000000004},"max":{"1min":4.2510000000000003,"5min":4.2510000000000003,"15min":35.283000000000001},"last":0.94399999999999995},{"interface":"front","average":{"1min":1.494,"5min":1.5880000000000001,"15min":1.8799999999999999},"min":{"1min":0.78700000000000003,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":3.9609999999999999,"5min":4.1470000000000002,"15min":35.353999999999999},"last":0.91700000000000004}]},{"osd":6,"last update":"Wed Feb 10 04:53:36 2021","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.593,"15min":1.835},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.8160000000000001,"5min":4.0380000000000003,"15min":16.050000000000001},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.569,"15min":1.7490000000000001},"min":{"1min":0.878,"5min":0.876,"15min":0.876},"max":{"1min":1.96,"5min":4.2629999999999999,"15min":15.960000000000001},"last":1.2190000000000001}]}]},{"osd":6,"up_from":43,"seq":184683594080,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":402568,"kb_used_data":341632,"kb_used_omap":0,"kb_used_meta":60928,"kb_avail":93342584,"statfs":{"total":95995035648,"available":95582806016,"internally_reserved":0,"allocated":349831168,"data_stored":442946607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62390272},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:52:12 2021","interfaces":[{"interface":"back","average":{"1min":4.016,"5min":2.7400000000000002,"15min":1.595},"min":{"1min":0.52000000000000002,"5min":0.48099999999999998,"15min":0.44700000000000001},"max":{"1min":61.768000000000001,"5min":137.60599999999999,"15min":137.60599999999999},"last":0.97999999999999998},{"interface":"front","average":{"1min":4.0270000000000001,"5min":2.694,"15min":1.7989999999999999},"min":{"1min":0.504,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":62.683,"5min":137.66,"15min":137.66},"last":0.82099999999999995}]},{"osd":1,"last update":"Wed Feb 10 04:54:16 2021","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":1.05,"15min":1.083},"min":{"1min":0.51800000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.3859999999999999,"5min":13.18,"15min":13.18},"last":0.88},{"interface":"front","average":{"1min":0.91000000000000003,"5min":1.1100000000000001,"15min":1.1180000000000001},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":3.6880000000000002,"5min":11.041,"15min":11.041},"last":0.71499999999999997}]},{"osd":2,"last update":"Wed Feb 10 04:54:16 2021","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":2.7069999999999999,"15min":1.734},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.4119999999999999,"5min":163.81700000000001,"15min":163.81700000000001},"last":0.85799999999999998},{"interface":"front","average":{"1min":1.0620000000000001,"5min":2.722,"15min":1.77},"min":{"1min":0.66200000000000003,"5min":0.60299999999999998,"15min":0.59899999999999998},"max":{"1min":1.633,"5min":164.56800000000001,"15min":164.56800000000001},"last":0.64900000000000002}]},{"osd":3,"last update":"Wed Feb 10 04:54:01 2021","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.161,"15min":1.1990000000000001},"min":{"1min":0.77600000000000002,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.722,"5min":1.722,"15min":1.722},"last":0.75900000000000001},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.123,"15min":1.1240000000000001},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.629,"5min":2.363,"15min":2.363},"last":0.78700000000000003}]},{"osd":4,"last update":"Wed Feb 10 04:54:16 2021","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.274,"15min":1.3999999999999999},"min":{"1min":0.71799999999999997,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.53,"5min":7.1459999999999999,"15min":7.1459999999999999},"last":0.623},{"interface":"front","average":{"1min":0.96199999999999997,"5min":1.0900000000000001,"15min":1.123},"min":{"1min":0.71399999999999997,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.494,"5min":11.997999999999999,"15min":11.997999999999999},"last":0.99299999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:54:16 2021","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.2589999999999999,"15min":1.629},"min":{"1min":0.78800000000000003,"5min":0.61699999999999999,"15min":0.56899999999999995},"max":{"1min":1.829,"5min":12.045,"15min":81.540000000000006},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.522,"15min":1.9019999999999999},"min":{"1min":0.85699999999999998,"5min":0.78700000000000003,"15min":0.78000000000000003},"max":{"1min":1.7949999999999999,"5min":13.401,"15min":81.492999999999995},"last":1.008}]},{"osd":7,"last update":"Wed Feb 10 04:53:43 2021","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.232,"15min":1.3360000000000001},"min":{"1min":0.80000000000000004,"5min":0.747,"15min":0.747},"max":{"1min":1.7230000000000001,"5min":4.0919999999999996,"15min":5.1820000000000004},"last":1.054},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.5129999999999999,"15min":1.637},"min":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":1.571,"5min":5.4109999999999996,"15min":13.315},"last":1.117}]}]},{"osd":1,"up_from":13,"seq":55834575218,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":286160,"kb_used_data":224200,"kb_used_omap":0,"kb_used_meta":61952,"kb_avail":93458992,"statfs":{"total":95995035648,"available":95702007808,"internally_reserved":0,"allocated":229580800,"data_stored":289098137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":63438848},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:48 2021","interfaces":[{"interface":"back","average":{"1min":1.696,"5min":2.6320000000000001,"15min":1.921},"min":{"1min":0.44700000000000001,"5min":0.437,"15min":0.437},"max":{"1min":14.053000000000001,"5min":115.622,"15min":115.622},"last":0.69599999999999995},{"interface":"front","average":{"1min":1.728,"5min":2.585,"15min":1.948},"min":{"1min":0.49199999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":14.109,"5min":115.73099999999999,"15min":115.73099999999999},"last":0.72899999999999998}]},{"osd":2,"last update":"Wed Feb 10 04:54:07 2021","interfaces":[{"interface":"back","average":{"1min":2.3399999999999999,"5min":3.3410000000000002,"15min":2.0350000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":37.975000000000001,"5min":115.59699999999999,"15min":115.59699999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":2.3780000000000001,"5min":3.379,"15min":2.0800000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":37.840000000000003,"5min":115.753,"15min":115.753},"last":0.58799999999999997}]},{"osd":3,"last update":"Wed Feb 10 04:54:03 2021","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.85299999999999998,"15min":0.93000000000000005},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.84699999999999998,"5min":3.077,"15min":3.077},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.96999999999999997,"15min":1.024},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.226,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.68200000000000005}]},{"osd":4,"last update":"Wed Feb 10 04:53:51 2021","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.0289999999999999,"15min":1.1719999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.55200000000000005},"max":{"1min":1.825,"5min":2.9670000000000001,"15min":4.899},"last":1.175},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.117,"15min":1.327},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.718,"5min":2.5270000000000001,"15min":29.547999999999998},"last":0.77400000000000002}]},{"osd":5,"last update":"Wed Feb 10 04:53:57 2021","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":1.1020000000000001,"15min":1.198},"min":{"1min":0.628,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.331,"5min":2.7959999999999998,"15min":7.8049999999999997},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.036,"5min":1.236,"15min":1.504},"min":{"1min":0.75800000000000001,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.383,"5min":2.8199999999999998,"15min":28.678000000000001},"last":1.0720000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:54:07 2021","interfaces":[{"interface":"back","average":{"1min":2.2810000000000001,"5min":1.421,"15min":1.571},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":30.997,"5min":30.997,"15min":30.997},"last":0.97599999999999998},{"interface":"front","average":{"1min":2.2599999999999998,"5min":1.4370000000000001,"15min":1.476},"min":{"1min":0.64200000000000002,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":31.138000000000002,"5min":31.138000000000002,"15min":31.138000000000002},"last":1.1579999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:54:26 2021","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.1890000000000001,"15min":1.4910000000000001},"min":{"1min":0.80200000000000005,"5min":0.628,"15min":0.628},"max":{"1min":1.494,"5min":3.1139999999999999,"15min":28.573},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.151,"5min":1.2649999999999999,"15min":1.54},"min":{"1min":0.86399999999999999,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.7210000000000001,"5min":2.298,"15min":29.408999999999999},"last":0.86399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738720,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":399164,"kb_used_data":337140,"kb_used_omap":0,"kb_used_meta":62016,"kb_avail":93345988,"statfs":{"total":95995035648,"available":95586291712,"internally_reserved":0,"allocated":345231360,"data_stored":470653300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":63504384},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 04:51:32 2021","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.81100000000000005,"15min":1.0089999999999999},"min":{"1min":0.47099999999999997,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.0329999999999999,"5min":2.2930000000000001,"15min":28.946999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.874,"5min":0.92300000000000004,"15min":1.0149999999999999},"min":{"1min":0.68000000000000005,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.1819999999999999,"5min":2.0670000000000002,"15min":3.7050000000000001},"last":0.89200000000000002}]},{"osd":2,"last update":"Wed Feb 10 04:52:09 2021","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":2.2389999999999999,"15min":1.613},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0700000000000001,"5min":97.983999999999995,"15min":97.983999999999995},"last":1.056},{"interface":"front","average":{"1min":0.749,"5min":2.153,"15min":1.5169999999999999},"min":{"1min":0.51900000000000002,"5min":0.51300000000000001,"15min":0.47699999999999998},"max":{"1min":1.1319999999999999,"5min":97.844999999999999,"15min":97.844999999999999},"last":1.296}]},{"osd":3,"last update":"Wed Feb 10 04:51:59 2021","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.87,"15min":0.87},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":1.002},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.254,"5min":1.254,"15min":1.254},"last":0.871}]},{"osd":4,"last update":"Wed Feb 10 04:51:32 2021","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":1.119,"15min":1.306},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.3200000000000001,"5min":2.206,"15min":5.1680000000000001},"last":1.1120000000000001},{"interface":"front","average":{"1min":0.96499999999999997,"5min":1.179,"15min":1.252},"min":{"1min":0.64200000000000002,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.607,"5min":3.359,"15min":14.672000000000001},"last":1.0920000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:51:54 2021","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.325,"15min":1.512},"min":{"1min":0.84599999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.714,"5min":3.944,"15min":25.047000000000001},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.413,"15min":1.7090000000000001},"min":{"1min":0.79400000000000004,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":2.1669999999999998,"5min":3.7850000000000001,"15min":28.827999999999999},"last":1.226}]},{"osd":6,"last update":"Wed Feb 10 04:52:23 2021","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.845,"15min":1.8280000000000001},"min":{"1min":1.0109999999999999,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.7889999999999999,"5min":45.012,"15min":45.012},"last":1.587},{"interface":"front","average":{"1min":1.327,"5min":1.891,"15min":1.95},"min":{"1min":0.88800000000000001,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.0979999999999999,"5min":45.225000000000001,"15min":45.225000000000001},"last":1.3620000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:52:28 2021","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.3080000000000001,"15min":1.5069999999999999},"min":{"1min":0.96099999999999997,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.605,"5min":2.8700000000000001,"15min":5.6050000000000004},"last":1.532},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.3959999999999999,"15min":1.5669999999999999},"min":{"1min":1.0129999999999999,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.794,"5min":3.1259999999999999,"15min":13.968999999999999},"last":1.5129999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411696,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":368928,"kb_used_data":317784,"kb_used_omap":0,"kb_used_meta":51136,"kb_avail":93376224,"statfs":{"total":95995035648,"available":95617253376,"internally_reserved":0,"allocated":325410816,"data_stored":374469458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52363264},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:50 2021","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.99099999999999999,"15min":1.718},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":3.698,"5min":3.996,"15min":163.15199999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.91300000000000003,"15min":1.655},"min":{"1min":0.51100000000000001,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.9710000000000001,"5min":2.2349999999999999,"15min":162.959},"last":0.85399999999999998}]},{"osd":1,"last update":"Wed Feb 10 04:53:53 2021","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.874,"15min":1.014},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.044,"5min":3.8159999999999998,"15min":4.9690000000000003},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.73899999999999999,"15min":0.89900000000000002},"min":{"1min":0.47299999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.0009999999999999,"5min":1.506,"15min":5.726},"last":0.46000000000000002}]},{"osd":3,"last update":"Wed Feb 10 04:54:06 2021","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":1.02,"15min":1.117},"min":{"1min":0.49099999999999999,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.3859999999999999,"5min":4.2380000000000004,"15min":4.2380000000000004},"last":0.56399999999999995},{"interface":"front","average":{"1min":1.024,"5min":1.2130000000000001,"15min":1.2909999999999999},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.589,"5min":4.54,"15min":4.54},"last":0.72799999999999998}]},{"osd":4,"last update":"Wed Feb 10 04:53:31 2021","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":1.1839999999999999,"15min":1.321},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.742,"5min":4.1310000000000002,"15min":4.3319999999999999},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.90300000000000002,"5min":1.022,"15min":1.121},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.323,"5min":3.411,"15min":4.4489999999999998},"last":0.60399999999999998}]},{"osd":5,"last update":"Wed Feb 10 04:53:53 2021","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.744,"15min":1.7090000000000001},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":2.4260000000000002,"5min":33.018999999999998,"15min":33.018999999999998},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.88600000000000001,"5min":1.526,"15min":1.476},"min":{"1min":0.65900000000000003,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.2230000000000001,"5min":33.043999999999997,"15min":33.043999999999997},"last":0.92100000000000004}]},{"osd":6,"last update":"Wed Feb 10 04:54:21 2021","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":1.133,"15min":1.4590000000000001},"min":{"1min":0.76800000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.8180000000000001,"5min":2.0579999999999998,"15min":20.007000000000001},"last":0.747},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.3420000000000001,"15min":1.635},"min":{"1min":0.94599999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.627,"5min":4.415,"15min":19.417999999999999},"last":0.79600000000000004}]},{"osd":7,"last update":"Wed Feb 10 04:53:31 2021","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.4770000000000001,"15min":1.6459999999999999},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":1.8839999999999999,"5min":4.6699999999999999,"15min":4.6699999999999999},"last":1.016},{"interface":"front","average":{"1min":1.042,"5min":1.155,"15min":1.3129999999999999},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.4159999999999999,"5min":1.9239999999999999,"15min":4.9189999999999996},"last":0.90300000000000002}]}]},{"osd":3,"up_from":1111,"seq":4771708665899,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":326932,"kb_used_data":300556,"kb_used_omap":386,"kb_used_meta":25981,"kb_avail":93418220,"statfs":{"total":95995035648,"available":95660257280,"internally_reserved":0,"allocated":307769344,"data_stored":409391333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":396036,"internal_metadata":26604796},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:58 2021","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":0.67200000000000004}]},{"osd":1,"last update":"Wed Feb 10 04:54:03 2021","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65700000000000003,"15min":0.65100000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.8360000000000001,"5min":1.8360000000000001,"15min":1.8360000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.83299999999999996,"15min":0.873},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.641,"5min":2.137,"15min":2.137},"last":1.129}]},{"osd":2,"last update":"Wed Feb 10 04:54:03 2021","interfaces":[{"interface":"back","average":{"1min":1.8180000000000001,"5min":1.208,"15min":1.169},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":23.844999999999999,"5min":23.844999999999999,"15min":23.844999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":1.9970000000000001,"5min":1.2669999999999999,"15min":1.169},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":23.783999999999999,"5min":23.783999999999999,"15min":23.783999999999999},"last":0.82799999999999996}]},{"osd":4,"last update":"Wed Feb 10 04:54:03 2021","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.96499999999999997,"15min":0.96899999999999997},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":2.9430000000000001,"5min":2.9430000000000001,"15min":2.9430000000000001},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.92000000000000004,"5min":1.042,"15min":1.1000000000000001},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.357,"5min":1.8999999999999999,"15min":1.8999999999999999},"last":0.80900000000000005}]},{"osd":5,"last update":"Wed Feb 10 04:54:03 2021","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":2.6379999999999999,"15min":2.7109999999999999},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":2.0979999999999999,"5min":52.353000000000002,"15min":52.353000000000002},"last":1.252},{"interface":"front","average":{"1min":1.0660000000000001,"5min":2.4929999999999999,"15min":2.5259999999999998},"min":{"1min":0.71399999999999997,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":3.544,"5min":52.301000000000002,"15min":52.301000000000002},"last":0.71699999999999997}]},{"osd":6,"last update":"Wed Feb 10 04:54:03 2021","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":4.4249999999999998,"15min":5.2930000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.474,"5min":64.376999999999995,"15min":64.376999999999995},"last":0.86099999999999999},{"interface":"front","average":{"1min":1.228,"5min":4.4640000000000004,"15min":5.2779999999999996},"min":{"1min":0.85599999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":1.8080000000000001,"5min":64.438999999999993,"15min":64.438999999999993},"last":1.1890000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:54:03 2021","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.087,"15min":1.113},"min":{"1min":0.79800000000000004,"5min":0.754,"15min":0.754},"max":{"1min":1.466,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.137,"15min":1.141},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.5649999999999999,"5min":2.044,"15min":2.044},"last":1.292}]}]},{"osd":4,"up_from":31,"seq":133143986536,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":158880,"kb_used_data":119768,"kb_used_omap":0,"kb_used_meta":39104,"kb_avail":93586272,"statfs":{"total":95995035648,"available":95832342528,"internally_reserved":0,"allocated":122642432,"data_stored":155292146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40042496},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:36 2021","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.0899999999999999,"15min":1.593},"min":{"1min":0.45800000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.7799999999999998,"5min":95.721999999999994,"15min":95.721999999999994},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.67700000000000005,"5min":2.0870000000000002,"15min":1.6000000000000001},"min":{"1min":0.46000000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.002,"5min":95.344999999999999,"15min":95.344999999999999},"last":0.70599999999999996}]},{"osd":1,"last update":"Wed Feb 10 04:53:42 2021","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.84999999999999998,"15min":0.95599999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.1459999999999999,"5min":7.883,"15min":24.295000000000002},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.82099999999999995,"5min":1.04,"15min":1.218},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.5369999999999999,"5min":3.8090000000000002,"15min":24.943000000000001},"last":0.75}]},{"osd":2,"last update":"Wed Feb 10 04:53:42 2021","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":2.5990000000000002,"15min":1.677},"min":{"1min":0.624,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.468,"5min":120.39,"15min":120.39},"last":0.87},{"interface":"front","average":{"1min":0.83599999999999997,"5min":2.5449999999999999,"15min":1.6739999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.51900000000000002},"max":{"1min":1.1599999999999999,"5min":120.583,"15min":120.583},"last":0.78400000000000003}]},{"osd":3,"last update":"Wed Feb 10 04:54:01 2021","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.21,"15min":1.2749999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.6339999999999999,"5min":2.1429999999999998,"15min":2.1429999999999998},"last":0.99299999999999999},{"interface":"front","average":{"1min":2.8519999999999999,"5min":1.7490000000000001,"15min":1.603},"min":{"1min":0.65200000000000002,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":26.802,"5min":26.802,"15min":26.802},"last":1.218}]},{"osd":5,"last update":"Wed Feb 10 04:53:56 2021","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.28,"15min":1.516},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":2.351,"5min":8.1509999999999998,"15min":35.774000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":1.276,"5min":1.282,"15min":1.4810000000000001},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.58799999999999997},"max":{"1min":4.319,"5min":7.9950000000000001,"15min":35.606000000000002},"last":0.94399999999999995}]},{"osd":6,"last update":"Wed Feb 10 04:54:01 2021","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":3.5539999999999998,"15min":2.3940000000000001},"min":{"1min":0.873,"5min":0.86199999999999999,"15min":0.76600000000000001},"max":{"1min":1.8380000000000001,"5min":201.626,"15min":201.626},"last":1.327},{"interface":"front","average":{"1min":1.125,"5min":3.476,"15min":2.4279999999999999},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.7529999999999999,"5min":201.577,"15min":201.577},"last":1.0149999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:53:29 2021","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.5009999999999999,"15min":1.508},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.2210000000000001,"5min":4.7439999999999998,"15min":6.9199999999999999},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.379,"15min":1.5469999999999999},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":2.198,"5min":2.8010000000000002,"15min":5.149},"last":1.675}]}]},{"osd":5,"up_from":37,"seq":158913790309,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":250496,"kb_used_data":205560,"kb_used_omap":0,"kb_used_meta":44928,"kb_avail":93494656,"statfs":{"total":95995035648,"available":95738527744,"internally_reserved":0,"allocated":210493440,"data_stored":275366180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":46006272},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:51 2021","interfaces":[{"interface":"back","average":{"1min":5.0590000000000002,"5min":1.714,"15min":1.204},"min":{"1min":0.45700000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":58.624000000000002,"5min":58.624000000000002,"15min":58.624000000000002},"last":0.64600000000000002},{"interface":"front","average":{"1min":5.04,"5min":1.784,"15min":1.3009999999999999},"min":{"1min":0.56599999999999995,"5min":0.46600000000000003,"15min":0.46400000000000002},"max":{"1min":57.880000000000003,"5min":57.880000000000003,"15min":57.880000000000003},"last":1.284}]},{"osd":1,"last update":"Wed Feb 10 04:53:52 2021","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":0.93400000000000005,"15min":0.95399999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":6.391,"5min":6.391,"15min":6.391},"last":1.1279999999999999},{"interface":"front","average":{"1min":0.72899999999999998,"5min":1.1499999999999999,"15min":1.175},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.2569999999999999,"5min":11.818,"15min":11.875999999999999},"last":0.86299999999999999}]},{"osd":2,"last update":"Wed Feb 10 04:53:52 2021","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":1.641,"15min":1.5369999999999999},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.2430000000000001,"5min":50.948999999999998,"15min":50.948999999999998},"last":1.036},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.7490000000000001,"15min":1.5760000000000001},"min":{"1min":0.61899999999999999,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":6.7130000000000001,"5min":50.149000000000001,"15min":50.149000000000001},"last":0.88100000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:54:06 2021","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.2030000000000001,"15min":1.2909999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.357,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":0.96999999999999997},{"interface":"front","average":{"1min":1.284,"5min":1.2070000000000001,"15min":1.1890000000000001},"min":{"1min":0.73199999999999998,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":6.5129999999999999,"5min":6.5129999999999999,"15min":6.5129999999999999},"last":1.2470000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:53:52 2021","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.329,"15min":1.3320000000000001},"min":{"1min":0.70399999999999996,"5min":0.64500000000000002,"15min":0.57099999999999995},"max":{"1min":2.028,"5min":9.4890000000000008,"15min":9.4890000000000008},"last":0.78200000000000003},{"interface":"front","average":{"1min":1.117,"5min":1.3300000000000001,"15min":1.385},"min":{"1min":0.69099999999999995,"5min":0.68100000000000005,"15min":0.67500000000000004},"max":{"1min":2.3149999999999999,"5min":5.6180000000000003,"15min":5.6180000000000003},"last":0.82399999999999995}]},{"osd":6,"last update":"Wed Feb 10 04:54:06 2021","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.9379999999999999,"15min":3.0150000000000001},"min":{"1min":0.93100000000000005,"5min":0.83899999999999997,"15min":0.65800000000000003},"max":{"1min":2.7829999999999999,"5min":38.014000000000003,"15min":319.51999999999998},"last":0.90600000000000003},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.9019999999999999,"15min":2.9409999999999998},"min":{"1min":0.79300000000000004,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":6.8220000000000001,"5min":37.896999999999998,"15min":319.22300000000001},"last":1.0069999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:53:33 2021","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.5,"15min":1.5229999999999999},"min":{"1min":0.93000000000000005,"5min":0.75600000000000001,"15min":0.752},"max":{"1min":1.4990000000000001,"5min":11.897,"15min":11.897},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.395,"5min":1.5660000000000001,"15min":1.6060000000000001},"min":{"1min":0.94999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.431,"5min":6.0369999999999999,"15min":6.0369999999999999},"last":1.099}]}]}]}} 2021-02-10T04:54:30.876 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2021-02-10T04:54:30.876 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pool set iscsi pg_num 13 2021-02-10T04:54:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:30 smithi164 conmon[29288]: cluster 2021-02-10T04:54:29.545521+0000 mon.a (mon.0 2021-02-10T04:54:30.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:30 smithi164 conmon[29288]: ) 2570 : cluster [DBG] osdmap e1206: 8 total, 7 up, 8 in 2021-02-10T04:54:31.185 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:54:31.188 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:54:31.261 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:54:31.286 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:54:31.312 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:54:31.335 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:54:31.362 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:54:31.386 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:54:31.410 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:54:31.411 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-02-10T04:54:31.411 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: problem getting command descriptions from osd.0 2021-02-10T04:54:31.411 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:54:31.411 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:54:31.412 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:54:31.412 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:54:31.412 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:54:31.412 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:54:31.412 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:54:31.588 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is set 2021-02-10T04:54:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:31 smithi085 conmon[29319]: cluster 2021-02-10T04:54:30.475348+0000 mgr.y (mgr.14140) 1330 : cluster [DBG] pgmap v2182: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:31 smithi085 conmon[29319]: audit 2021-02-10T04:54:31.238934+0000 mon.c (mon.1) 115 : audit [DBG] from='client.? 172.21.15.85:0/360855047' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:54:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:31 smithi085 conmon[29319]: audit 2021-02-10T04:54:31.522482+0000 mon.c (mon.1) 116 : audit [INF] from='client.? 172.21.15.85:0/3099160255' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:54:31.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:31 smithi085 conmon[29319]: audit 2021-02-10T04:54:31.522818+0000 mon.a (mon.0) 2571 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:54:31.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:31 smithi085 conmon[35961]: cluster 2021-02-10T04:54:30.475348+0000 mgr.y (mgr.14140) 1330 : cluster 2021-02-10T04:54:31.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:31 smithi085 conmon[35961]: [DBG] pgmap v2182: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:31.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:31 smithi085 conmon[35961]: audit 2021-02-10T04:54:31.238934+0000 2021-02-10T04:54:31.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:31 smithi085 conmon[35961]: mon.c (mon.1) 115 : audit [DBG] from='client.? 172.21.15.85:0/360855047' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:54:31.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:31 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:54:31.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:31 smithi085 conmon[35961]: 54:31.522482+0000 mon.c (mon.1) 116 2021-02-10T04:54:31.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:31 smithi085 conmon[35961]: : audit [INF] from='client.? 172.21.15.85:0/3099160255' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:54:31.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:31 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:54:31.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:31 smithi085 conmon[35961]: T04:54:31.522818+0000 mon.a (mon.0) 2571 : audit 2021-02-10T04:54:31.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:31 smithi085 conmon[35961]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:54:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:31 smithi164 conmon[29288]: cluster 2021-02-10T04:54:30.475348+0000 2021-02-10T04:54:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:31 smithi164 conmon[29288]: mgr.y (mgr.14140) 1330 : cluster [DBG] pgmap v2182: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:31 smithi164 conmon[29288]: audit 2021-02-10T04:54:31.238934+0000 mon.c (mon.1) 115 : audit 2021-02-10T04:54:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:31 smithi164 conmon[29288]: [DBG] from='client.? 172.21.15.85:0/360855047' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:54:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:31 smithi164 conmon[29288]: audit 2021-02-10T04:54:31.522482+0000 mon.c (mon. 2021-02-10T04:54:31.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:31 smithi164 conmon[29288]: 1) 116 : audit [INF] from='client.? 172.21.15.85:0/3099160255' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:54:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:31 smithi164 conmon[29288]: audit 2021-02-10T04:54:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:31 smithi164 conmon[29288]: 2021-02-10T04:54:31.522818+0000 mon.a (mon 2021-02-10T04:54:31.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:31 smithi164 conmon[29288]: .0) 2571 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:54:32.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:32 smithi085 conmon[29319]: cluster 2021-02-10T04:54:31.558317+0000 mon.a (mon 2021-02-10T04:54:32.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:32 smithi085 conmon[29319]: .0) 2572 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:54:32.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:32 smithi085 conmon[29319]: audit 2021-02-10T04:54:31.570266+0000 mon.a (mon.0) 2573 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:54:32.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:32 smithi085 conmon[29319]: cluster 2021-02-10T04:54:31.570349+0000 mon.a (mon.0) 2574 : cluster [DBG] osdmap e1207: 8 total, 7 up, 8 in 2021-02-10T04:54:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:32 smithi085 conmon[35961]: cluster 2021-02-10T04:54:31. 2021-02-10T04:54:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:32 smithi085 conmon[35961]: 558317+0000 mon.a (mon.0) 2572 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:54:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:32 smithi085 conmon[35961]: audit 2021-02-10T04:54:31.570266+0000 mon.a (mon.0) 2573 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:54:32.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:32 smithi085 conmon[35961]: cluster 2021-02-10T04:54:31.570349+0000 mon.a (mon.0) 2574 : cluster [DBG] osdmap e1207: 8 total, 7 up, 8 in 2021-02-10T04:54:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:32 smithi164 conmon[29288]: cluster 2021-02-10T04:54:31.558317+0000 mon.a (mon.0) 2572 : cluster [WRN] 2021-02-10T04:54:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:32 smithi164 conmon[29288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:54:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:32 smithi164 conmon[29288]: audit 2021-02-10T04:54:31.570266+0000 mon.a (mon.0) 2573 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:54:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:32 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:54:32.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:32 smithi164 conmon[29288]: 54:31.570349+0000 mon.a (mon.0) 2574 : cluster [DBG] osdmap e1207: 8 total, 7 up, 8 in 2021-02-10T04:54:33.011 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:54:33.011 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:54:33.889 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:33 smithi085 conmon[29319]: cluster 2021-02-10T04:54:32.476213+0000 mgr.y (mgr.14140) 1331 : cluster [DBG] 2021-02-10T04:54:33.889 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:33 smithi085 conmon[29319]: pgmap v2184: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:33.889 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:33 smithi085 conmon[35961]: cluster 2021-02-10T04:54:32. 2021-02-10T04:54:33.889 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:33 smithi085 conmon[35961]: 476213+0000 mgr.y (mgr.14140) 1331 : cluster [DBG] pgmap v2184: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:33.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:33 smithi164 conmon[29288]: cluster 2021-02-10T04:54:32.476213+0000 mgr.y (mgr.14140) 2021-02-10T04:54:33.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:33 smithi164 conmon[29288]: 1331 : cluster [DBG] pgmap v2184: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:34.816 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:54:34.816 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [], 2021-02-10T04:54:34.816 INFO:teuthology.orchestra.run.smithi085.stdout: "complaint_time": 30, 2021-02-10T04:54:34.816 INFO:teuthology.orchestra.run.smithi085.stdout: "num_blocked_ops": 0 2021-02-10T04:54:34.816 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:54:34.978 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set nodeep-scrub 2021-02-10T04:54:35.012 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:54:35.270 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:54:35.313 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:54:35.577 INFO:teuthology.orchestra.run.smithi085.stderr:set pool 3 pg_num to 13 2021-02-10T04:54:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:35 smithi164 conmon[29288]: cluster 2021-02-10T04:54:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:35 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:54:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:35 smithi164 conmon[29288]: :54:34.477077+0000 mgr.y (mgr.14140) 1332 : cluster [DBG] pgmap v2185: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:35 smithi164 conmon[29288]: audit 2021-02-10T04:54:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:35 smithi164 conmon[29288]: 2021-02-10T04:54:35 2021-02-10T04:54:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:35 smithi164 conmon[29288]: .452864+0000 mon.a (mon.0) 2575 : audit 2021-02-10T04:54:35.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:35 smithi164 conmon[29288]: [INF] from='client.? 172.21.15.85:0/1077566168' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num", "val": "13"}]: dispatch 2021-02-10T04:54:35.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:35 smithi085 conmon[29319]: cluster 2021-02-10T04:54:34.477077+0000 mgr.y (mgr.14140) 1332 : cluster [DBG] pgmap v2185: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:35.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:35 smithi085 conmon[29319]: audit 2021-02-10T04:54:35.452864+0000 mon.a (mon.0) 2575 : audit [INF] from='client.? 172.21.15.85:0/1077566168' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num", "val": "13"}]: dispatch 2021-02-10T04:54:35.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:35 smithi085 conmon[35961]: cluster 2021-02-10T04:54:34.477077+0000 mgr.y (mgr.14140) 1332 : cluster [DBG] pgmap v2185: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:35.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:35 smithi085 conmon[35961]: audit 2021-02-10T04:54:35.452864+0000 mon.a (mon.0) 2575 : audit [INF] from='client.? 172.21.15.85:0/1077566168' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num", "val": "13"}]: dispatch 2021-02-10T04:54:36.569 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.3 dump_historic_ops 2021-02-10T04:54:36.807 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:54:36.826 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:36 smithi085 conmon[29319]: audit 2021-02-10T04:54:35.571536+0000 mon.a ( 2021-02-10T04:54:36.826 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:36 smithi085 conmon[29319]: mon.0) 2576 : audit [INF] from='client.? 172.21.15.85:0/1077566168' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num", "val": "13"}]': finished 2021-02-10T04:54:36.826 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:36 smithi085 conmon[29319]: cluster 2021-02-10T04:54:35.571619+0000 mon.a (mon.0) 2577 : cluster [DBG] osdmap e1208: 8 total, 7 up, 8 in 2021-02-10T04:54:36.827 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:36 smithi085 conmon[29319]: audit 2021-02-10T04:54:36.478635+0000 mon.a (mon.0) 2578 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num_actual", "val": "13"}]: dispatch 2021-02-10T04:54:36.827 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:36 smithi085 conmon[35961]: audit 2021-02-10T04:54:35.571536+0000 mon.a (mon 2021-02-10T04:54:36.827 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:36 smithi085 conmon[35961]: .0) 2576 : audit [INF] from='client.? 172.21.15.85:0/1077566168' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num", "val": "13"}]': finished 2021-02-10T04:54:36.827 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:36 smithi085 conmon[35961]: cluster 2021-02-10T04:54:35.571619+0000 mon.a (mon.0) 2577 : cluster [DBG] osdmap e1208: 8 total, 7 up, 8 in 2021-02-10T04:54:36.827 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:36 smithi085 conmon[35961]: audit 2021-02-10T04:54:36.478635+0000 mon.a (mon.0) 2578 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num_actual", "val": "13"}]: dispatch 2021-02-10T04:54:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:36 smithi164 conmon[29288]: audit 2021-02-10T04:54:35.571536+0000 mon.a (mon.0) 2576 : audit [INF] 2021-02-10T04:54:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:36 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/1077566168' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num", "val": "13"}]': finished 2021-02-10T04:54:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:36 smithi164 conmon[29288]: cluster 2021-02-10T04:54:35.571619+0000 mon.a (mon.0) 2577 : cluster [DBG] osdmap e1208: 8 total, 7 up, 8 in 2021-02-10T04:54:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:36 smithi164 conmon[29288]: audit 2021-02-10T04:54:36.478635+0000 mon.a (mon.0) 2578 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num_actual", "val": "13"}]: dispatch 2021-02-10T04:54:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:37 smithi164 conmon[29288]: cluster 2021-02-10T04:54:36.477713+0000 mgr.y (mgr.14140) 2021-02-10T04:54:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:37 smithi164 conmon[29288]: 1333 : cluster [DBG] pgmap v2187: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 738 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:37 smithi164 conmon[29288]: audit 2021-02-10T04:54:36.584118+0000 mon.a (mon.0) 2579 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num_actual", "val": "13"}]': finished 2021-02-10T04:54:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:37 smithi164 conmon[29288]: cluster 2021-02-10T04:54:36.584180+0000 mon.a ( 2021-02-10T04:54:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:37 smithi164 conmon[29288]: mon.0) 2580 : cluster [DBG] osdmap e1209: 8 total, 7 up, 8 in 2021-02-10T04:54:37.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:37 smithi085 conmon[29319]: cluster 2021-02-10T04:54:36.477713+0000 mgr.y (mgr.14140) 2021-02-10T04:54:37.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:37 smithi085 conmon[29319]: 1333 : cluster [DBG] pgmap v2187: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 738 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:37.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:37 smithi085 conmon[29319]: audit 2021-02-10T04:54:36.584118+0000 mon.a (mon.0) 2579 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num_actual", "val": "13"}]': finished 2021-02-10T04:54:37.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:37 smithi085 conmon[29319]: cluster 2021-02-10T04:54:36.584180+0000 mon.a (mon.0) 2580 : cluster [DBG] osdmap e1209: 8 total, 7 up, 8 in 2021-02-10T04:54:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:37 smithi085 conmon[35961]: cluster 2021-02-10T04:54:36.477713+0000 mgr.y 2021-02-10T04:54:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:37 smithi085 conmon[35961]: (mgr.14140) 1333 : cluster [DBG] pgmap v2187: 194 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 127 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 738 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:37 smithi085 conmon[35961]: audit 2021-02-10T04:54:36.584118+0000 mon.a (mon 2021-02-10T04:54:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:37 smithi085 conmon[35961]: .0) 2579 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num_actual", "val": "13"}]': finished 2021-02-10T04:54:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:37 smithi085 conmon[35961]: cluster 2021-02-10T04:54:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:37 smithi085 conmon[35961]: 2021-02-10T04:54:36.584180+0000 mon.a (mon.0) 2580 2021-02-10T04:54:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:37 smithi085 conmon[35961]: : cluster [DBG] osdmap e1209: 8 total, 7 up, 8 in 2021-02-10T04:54:38.765 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[29319]: cluster 2021-02-10T04:54:37.603028 2021-02-10T04:54:38.765 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2581 : cluster [DBG] osdmap e1210: 8 total, 7 up, 8 in 2021-02-10T04:54:38.765 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[29319]: audit 2021-02-10T04:54:38.252053+0000 2021-02-10T04:54:38.765 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[29319]: mon.a (mon.0) 2582 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:54:38.765 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[29319]: audit 2021-02-10T04:54:38.765 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[29319]: 2021-02-10T04:54:38. 2021-02-10T04:54:38.765 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[29319]: 253101+0000 mon.a (mon.0) 2583 2021-02-10T04:54:38.766 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[29319]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:54:38.766 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[29319]: audit 2021-02-10T04:54:38 2021-02-10T04:54:38.766 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[29319]: .479329+0000 mon.a (mon.0) 2584 2021-02-10T04:54:38.766 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[29319]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num_actual", "val": "13"}]: dispatch 2021-02-10T04:54:38.766 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[29319]: audit 2021-02-10T 2021-02-10T04:54:38.766 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[29319]: 04:54:38.542116+0000 mon.a (mon. 2021-02-10T04:54:38.766 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[29319]: 0) 2585 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num_actual", "val": "13"}]': finished 2021-02-10T04:54:38.766 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:54:38.766 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[29319]: -10T04:54:38.542204+0000 mon.a 2021-02-10T04:54:38.766 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[29319]: (mon.0) 2586 : cluster [DBG] osdmap e1211: 8 total, 7 up, 8 in 2021-02-10T04:54:38.766 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[35961]: cluster 2021-02-10T04:54:37.603028+0000 mon.a ( 2021-02-10T04:54:38.767 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[35961]: mon.0) 2581 : cluster [DBG] osdmap e1210: 8 total, 7 up, 8 in 2021-02-10T04:54:38.767 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[35961]: audit 2021-02-10T04:54:38 2021-02-10T04:54:38.767 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[35961]: .252053+0000 mon.a (mon.0) 2582 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:54:38.767 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:54:38.767 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[35961]: 10T04:54:38.253101+0000 mon.a (mon. 2021-02-10T04:54:38.767 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[35961]: 0) 2583 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:54:38.767 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:54:38.767 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[35961]: -10T04:54:38.479329+0000 mon.a (mon.0 2021-02-10T04:54:38.767 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[35961]: ) 2584 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num_actual", "val": "13"}]: dispatch 2021-02-10T04:54:38.767 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:54:38.767 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[35961]: 04:54:38.542116+0000 mon.a (mon.0) 2585 2021-02-10T04:54:38.767 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[35961]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num_actual", "val": "13"}]': finished 2021-02-10T04:54:38.768 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:54:38.768 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[35961]: 04:54:38.542204+0000 mon.a (mon.0) 2021-02-10T04:54:38.768 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:38 smithi085 conmon[35961]: 2586 : cluster [DBG] osdmap e1211: 8 total, 7 up, 8 in 2021-02-10T04:54:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:38 smithi164 conmon[29288]: cluster 2021-02-10T04:54:37.603028 2021-02-10T04:54:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:38 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2581 : cluster [DBG] osdmap e1210: 8 total, 7 up, 8 in 2021-02-10T04:54:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:38 smithi164 conmon[29288]: audit 2021-02-10T04:54:38.252053+0000 mon.a ( 2021-02-10T04:54:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:38 smithi164 conmon[29288]: mon.0) 2582 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:54:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:38 smithi164 conmon[29288]: audit 2021-02-10T04:54 2021-02-10T04:54:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:38 smithi164 conmon[29288]: :38.253101+0000 mon.a (mon.0) 2583 2021-02-10T04:54:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:38 smithi164 conmon[29288]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:54:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:38 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:54:38.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:38 smithi164 conmon[29288]: 02-10T04:54:38.479329+0000 mon.a ( 2021-02-10T04:54:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:38 smithi164 conmon[29288]: mon.0) 2584 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num_actual", "val": "13"}]: dispatch 2021-02-10T04:54:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:38 smithi164 conmon[29288]: audit 2021-02-10T04:54:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:38 smithi164 conmon[29288]: 2021-02-10T04:54:38. 2021-02-10T04:54:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:38 smithi164 conmon[29288]: 542116+0000 mon.a (mon.0) 2585 : 2021-02-10T04:54:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:38 smithi164 conmon[29288]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num_actual", "val": "13"}]': finished 2021-02-10T04:54:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:38 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:54:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:38 smithi164 conmon[29288]: T04:54:38.542204+0000 mon.a (mon 2021-02-10T04:54:38.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:38 smithi164 conmon[29288]: .0) 2586 : cluster [DBG] osdmap e1211: 8 total, 7 up, 8 in 2021-02-10T04:54:39.638 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:54:39 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:54:39.639 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:39 smithi164 conmon[29288]: cluster 2021-02-10T04:54:38.478511+0000 mgr.y (mgr.14140) 1334 2021-02-10T04:54:39.639 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:39 smithi164 conmon[29288]: : cluster [DBG] pgmap v2190: 204 pgs: 1 active+clean+scrubbing, 2 peering, 7 unknown, 5 active+clean+remapped, 24 active+undersized+degraded, 36 active+undersized, 129 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:39.640 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:39 smithi164 conmon[29288]: cluster 2021-02-10T04:54:38.597687+0000 mon.a (mon.0) 2587 : cluster [WRN] Health check update: Degraded data redundancy: 214/2196 objects degraded (9.745%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2021-02-10T04:54:39.640 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:39 smithi164 conmon[29288]: audit 2021-02-10T04:54:39.264993+0000 mgr.y (mgr.14140) 1335 : audit [DBG] 2021-02-10T04:54:39.640 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:39 smithi164 conmon[29288]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:54:39.640 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:39 smithi164 conmon[29288]: cluster 2021-02-10T04:54:39.640 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:39 smithi164 conmon[29288]: 2021-02-10T04:54:39.543289+0000 mon.a (mon. 2021-02-10T04:54:39.640 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:39 smithi164 conmon[29288]: 0) 2588 : cluster [DBG] osdmap e1212: 8 total, 7 up, 8 in 2021-02-10T04:54:39.880 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:39 smithi085 conmon[29319]: cluster 2021-02-10T04:54:38.478511+0000 mgr.y (mgr 2021-02-10T04:54:39.881 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:39 smithi085 conmon[29319]: .14140) 1334 : cluster [DBG] pgmap v2190: 204 pgs: 1 active+clean+scrubbing, 2 peering, 7 unknown, 5 active+clean+remapped, 24 active+undersized+degraded, 36 active+undersized, 129 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:39.881 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:39 smithi085 conmon[29319]: cluster 2021-02-10T04:54:38.597687+0000 mon.a (mon.0) 2587 : cluster [WRN] Health check update: Degraded data redundancy: 214/2196 objects degraded (9.745%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2021-02-10T04:54:39.881 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:39 smithi085 conmon[29319]: audit 2021-02-10T04:54:39.264993+0000 mgr.y (mgr.14140) 1335 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:54:39.881 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:39 smithi085 conmon[29319]: cluster 2021-02-10T04:54:39.543289+0000 mon.a (mon.0) 2588 : cluster [DBG] osdmap e1212: 8 total, 7 up, 8 in 2021-02-10T04:54:39.881 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:39 smithi085 conmon[35961]: cluster 2021-02-10T04:54:38 2021-02-10T04:54:39.881 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:39 smithi085 conmon[35961]: .478511+0000 mgr.y (mgr.14140) 1334 : cluster [DBG] pgmap v2190: 204 pgs: 1 active+clean+scrubbing, 2 peering, 7 unknown, 5 active+clean+remapped, 24 active+undersized+degraded, 36 active+undersized, 129 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:39.881 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:39 smithi085 conmon[35961]: cluster 2021-02-10T04:54:38. 2021-02-10T04:54:39.881 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:39 smithi085 conmon[35961]: 597687+0000 mon.a (mon.0) 2587 : cluster [WRN] Health check update: Degraded data redundancy: 214/2196 objects degraded (9.745%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2021-02-10T04:54:39.881 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:39 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:54:39.881 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:39 smithi085 conmon[35961]: T04:54:39.264993+0000 mgr.y (mgr.14140 2021-02-10T04:54:39.881 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:39 smithi085 conmon[35961]: ) 1335 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:54:39.882 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:39 smithi085 conmon[35961]: cluster 2021-02-10T04:54:39.882 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:39 smithi085 conmon[35961]: 2021-02-10T04:54 2021-02-10T04:54:39.882 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:39 smithi085 conmon[35961]: :39.543289+0000 mon.a (mon.0) 2588 2021-02-10T04:54:39.882 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:39 smithi085 conmon[35961]: : cluster [DBG] osdmap e1212: 8 total, 7 up, 8 in 2021-02-10T04:54:40.765 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:54:40.773 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:54:40.773 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 20, 2021-02-10T04:54:40.774 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 600, 2021-02-10T04:54:40.774 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [ 2021-02-10T04:54:40.774 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.774 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:3431 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94187755107840 gen 3] snapc 0=[] ondisk+write+known_if_redirected e1112)", 2021-02-10T04:54:40.774 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:02.988562+0000", 2021-02-10T04:54:40.774 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 217.78200006399999, 2021-02-10T04:54:40.774 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9928285949999998, 2021-02-10T04:54:40.774 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:40.774 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:54:40.774 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:54:40.774 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:54:40.774 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:54:40.775 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 3431 2021-02-10T04:54:40.775 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.775 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:40.775 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.775 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:40.775 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988562+0000", 2021-02-10T04:54:40.775 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:40.775 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.775 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.775 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:40.775 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988562+0000", 2021-02-10T04:54:40.775 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.1761000000000001e-05 2021-02-10T04:54:40.776 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.776 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.776 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:40.776 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988583+0000", 2021-02-10T04:54:40.776 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.5929999999999996e-06 2021-02-10T04:54:40.776 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.776 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.776 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:40.776 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988589+0000", 2021-02-10T04:54:40.776 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.8294000000000001e-05 2021-02-10T04:54:40.776 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.776 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.776 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:40.777 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988607+0000", 2021-02-10T04:54:40.777 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.6950000000000003e-06 2021-02-10T04:54:40.777 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.777 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.777 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:40.777 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988614+0000", 2021-02-10T04:54:40.777 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.5459999999999999e-05 2021-02-10T04:54:40.777 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.777 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.777 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:40.777 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988689+0000", 2021-02-10T04:54:40.777 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.0969e-05 2021-02-10T04:54:40.777 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.777 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.778 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for readable", 2021-02-10T04:54:40.778 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988730+0000", 2021-02-10T04:54:40.778 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9912170979999999 2021-02-10T04:54:40.778 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.778 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.778 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:40.778 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.979947+0000", 2021-02-10T04:54:40.778 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00020239699999999999 2021-02-10T04:54:40.778 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.778 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.778 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:54:40.778 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.980150+0000", 2021-02-10T04:54:40.779 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001240328 2021-02-10T04:54:40.779 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.779 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.779 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:40.779 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.981390+0000", 2021-02-10T04:54:40.779 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9928285949999998 2021-02-10T04:54:40.779 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.779 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:40.779 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.779 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.779 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.779 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:3432 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 94187770080640 gen 4] snapc 0=[] ondisk+write+known_if_redirected e1112)", 2021-02-10T04:54:40.779 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:02.988669+0000", 2021-02-10T04:54:40.780 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 217.78189235400001, 2021-02-10T04:54:40.780 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0090249269999996, 2021-02-10T04:54:40.780 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:40.780 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:54:40.780 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:54:40.780 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:54:40.780 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:54:40.780 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 3432 2021-02-10T04:54:40.780 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.780 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:40.780 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.780 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:40.780 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988669+0000", 2021-02-10T04:54:40.780 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:40.781 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.781 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.781 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:40.781 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988669+0000", 2021-02-10T04:54:40.781 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.0630999999999999e-05 2021-02-10T04:54:40.781 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.781 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.781 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:40.781 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988700+0000", 2021-02-10T04:54:40.781 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.3149999999999997e-06 2021-02-10T04:54:40.781 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.781 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.782 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:40.782 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988706+0000", 2021-02-10T04:54:40.782 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0995000000000001e-05 2021-02-10T04:54:40.782 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.782 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.782 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:40.782 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988717+0000", 2021-02-10T04:54:40.782 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.3070000000000002e-06 2021-02-10T04:54:40.782 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.782 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.782 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:40.782 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988722+0000", 2021-02-10T04:54:40.782 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.5676e-05 2021-02-10T04:54:40.783 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.783 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.783 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:40.783 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988778+0000", 2021-02-10T04:54:40.783 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.2759999999999998e-05 2021-02-10T04:54:40.783 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.783 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.783 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for readable", 2021-02-10T04:54:40.783 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988811+0000", 2021-02-10T04:54:40.783 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0076071410000003 2021-02-10T04:54:40.783 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.783 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.783 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:40.784 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.996418+0000", 2021-02-10T04:54:40.784 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000159133 2021-02-10T04:54:40.784 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.784 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.784 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:54:40.784 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.996577+0000", 2021-02-10T04:54:40.784 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001116969 2021-02-10T04:54:40.784 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.784 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.784 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:40.784 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.997694+0000", 2021-02-10T04:54:40.784 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0090249269999996 2021-02-10T04:54:40.784 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.785 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:40.785 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.785 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.785 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.785 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:3435 5.14 5:2b04a3e9:::notify.6:head [watch ping cookie 94187770080640 gen 4] snapc 0=[] ondisk+write+known_if_redirected e1113)", 2021-02-10T04:54:40.785 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:07.988123+0000", 2021-02-10T04:54:40.785 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 212.78243820399999, 2021-02-10T04:54:40.785 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0089928860000000003, 2021-02-10T04:54:40.785 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:40.785 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:54:40.785 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:54:40.785 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:54:40.785 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:54:40.785 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 3435 2021-02-10T04:54:40.786 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.786 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:40.786 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.786 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:40.786 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.988123+0000", 2021-02-10T04:54:40.786 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:40.786 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.786 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.786 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:40.786 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.988123+0000", 2021-02-10T04:54:40.786 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.8739999999999999e-05 2021-02-10T04:54:40.786 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.786 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.787 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:40.787 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.988152+0000", 2021-02-10T04:54:40.787 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.4450000000000001e-06 2021-02-10T04:54:40.787 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.787 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.787 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:40.787 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.988159+0000", 2021-02-10T04:54:40.787 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0798999999999999e-05 2021-02-10T04:54:40.787 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.787 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.787 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:40.787 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.988169+0000", 2021-02-10T04:54:40.787 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.2630000000000003e-06 2021-02-10T04:54:40.788 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.788 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.788 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:40.788 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.988175+0000", 2021-02-10T04:54:40.788 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.1480000000000002e-05 2021-02-10T04:54:40.788 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.788 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.788 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:40.788 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.988226+0000", 2021-02-10T04:54:40.788 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.1520999999999997e-05 2021-02-10T04:54:40.788 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.788 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.788 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for readable", 2021-02-10T04:54:40.789 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.988278+0000", 2021-02-10T04:54:40.789 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0086211429999999995 2021-02-10T04:54:40.789 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.789 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.789 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:40.789 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.996899+0000", 2021-02-10T04:54:40.789 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.0494000000000006e-05 2021-02-10T04:54:40.789 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.789 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.789 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:54:40.789 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.996979+0000", 2021-02-10T04:54:40.789 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00013700100000000001 2021-02-10T04:54:40.789 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.790 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.790 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:40.790 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.997116+0000", 2021-02-10T04:54:40.790 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0089928860000000003 2021-02-10T04:54:40.790 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.790 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:40.790 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.790 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.790 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.790 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:3481 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 94187770080640 gen 5] snapc 0=[] ondisk+write+known_if_redirected e1118)", 2021-02-10T04:54:40.790 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:35.712201+0000", 2021-02-10T04:54:40.790 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 185.058360558, 2021-02-10T04:54:40.791 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.172105363, 2021-02-10T04:54:40.791 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:40.791 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:54:40.791 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:54:40.791 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:54:40.791 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:54:40.791 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 3481 2021-02-10T04:54:40.791 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.791 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:40.791 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.791 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:40.791 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:35.712201+0000", 2021-02-10T04:54:40.791 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:40.792 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.792 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.792 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:40.792 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:35.712201+0000", 2021-02-10T04:54:40.792 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.2008000000000003e-05 2021-02-10T04:54:40.792 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.792 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.792 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:40.792 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:35.712233+0000", 2021-02-10T04:54:40.792 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.0690000000000002e-06 2021-02-10T04:54:40.792 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.792 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.792 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:40.793 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:35.712241+0000", 2021-02-10T04:54:40.793 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2391999999999999e-05 2021-02-10T04:54:40.793 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.793 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.793 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:40.793 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:35.712253+0000", 2021-02-10T04:54:40.793 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.5499999999999997e-06 2021-02-10T04:54:40.793 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.793 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.793 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:40.793 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:35.712261+0000", 2021-02-10T04:54:40.793 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0015999999999997e-05 2021-02-10T04:54:40.793 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.794 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.794 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:40.794 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:35.712311+0000", 2021-02-10T04:54:40.794 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0231000000000001e-05 2021-02-10T04:54:40.794 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.794 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.794 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op must wait for map", 2021-02-10T04:54:40.794 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:35.712321+0000", 2021-02-10T04:54:40.794 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.02825248 2021-02-10T04:54:40.794 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.794 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.794 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:40.794 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:35.740574+0000", 2021-02-10T04:54:40.795 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0251999999999999e-05 2021-02-10T04:54:40.795 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.795 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.795 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for peered", 2021-02-10T04:54:40.795 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:35.740584+0000", 2021-02-10T04:54:40.795 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.129581339 2021-02-10T04:54:40.795 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.795 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.795 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:40.795 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:36.870165+0000", 2021-02-10T04:54:40.795 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00034935599999999999 2021-02-10T04:54:40.795 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.795 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.796 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for degraded object", 2021-02-10T04:54:40.796 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:36.870515+0000", 2021-02-10T04:54:40.796 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.012278183999999999 2021-02-10T04:54:40.796 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.796 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.796 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:40.796 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:36.882793+0000", 2021-02-10T04:54:40.796 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000120188 2021-02-10T04:54:40.796 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.796 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.796 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:54:40.796 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:36.882913+0000", 2021-02-10T04:54:40.796 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0013932980000000001 2021-02-10T04:54:40.796 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.797 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.797 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:40.797 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:36.884306+0000", 2021-02-10T04:54:40.797 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.172105363 2021-02-10T04:54:40.797 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.797 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:40.797 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.797 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.798 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.798 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.14 1121/1119 [PushReplyOp(7:2dcefd69:::smithi08575397-28:333)])", 2021-02-10T04:54:40.798 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:39.199259+0000", 2021-02-10T04:54:40.798 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 181.571302411, 2021-02-10T04:54:40.798 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0030838609999999998, 2021-02-10T04:54:40.798 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:40.798 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:54:40.798 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:40.798 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.798 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:40.798 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.199259+0000", 2021-02-10T04:54:40.798 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:40.798 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.799 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.799 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:40.799 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.199259+0000", 2021-02-10T04:54:40.799 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.2369999999999997e-05 2021-02-10T04:54:40.799 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.799 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.799 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:40.799 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.199292+0000", 2021-02-10T04:54:40.799 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.072e-06 2021-02-10T04:54:40.799 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.799 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.799 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:40.799 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.199300+0000", 2021-02-10T04:54:40.799 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.4684999999999999e-05 2021-02-10T04:54:40.800 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.800 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.800 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:40.800 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.199314+0000", 2021-02-10T04:54:40.800 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.5300000000000002e-06 2021-02-10T04:54:40.800 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.800 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.800 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:40.800 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.199321+0000", 2021-02-10T04:54:40.800 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9565999999999999e-05 2021-02-10T04:54:40.800 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.800 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.800 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:40.801 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.199370+0000", 2021-02-10T04:54:40.801 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0029726380000000001 2021-02-10T04:54:40.801 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.801 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.801 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:40.801 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.202343+0000", 2021-02-10T04:54:40.801 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0030838609999999998 2021-02-10T04:54:40.801 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.801 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:40.801 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.801 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.801 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.802 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.14 1121/1119 [PushReplyOp(7:2dcefd69:::smithi08575397-28:368)])", 2021-02-10T04:54:40.802 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:39.217537+0000", 2021-02-10T04:54:40.802 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 181.55302465599999, 2021-02-10T04:54:40.802 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031144010000000001, 2021-02-10T04:54:40.802 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:40.802 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:54:40.802 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:40.802 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.802 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:40.802 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.217537+0000", 2021-02-10T04:54:40.803 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:40.803 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.803 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.803 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:40.803 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.217537+0000", 2021-02-10T04:54:40.803 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.5524000000000001e-05 2021-02-10T04:54:40.803 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.803 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.803 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:40.803 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.217572+0000", 2021-02-10T04:54:40.803 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1260999999999999e-05 2021-02-10T04:54:40.803 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.803 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.803 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:40.804 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.217584+0000", 2021-02-10T04:54:40.804 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.4684999999999999e-05 2021-02-10T04:54:40.804 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.804 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.804 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:40.804 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.217598+0000", 2021-02-10T04:54:40.804 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.6000000000000007e-06 2021-02-10T04:54:40.804 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.804 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.804 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:40.804 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.217607+0000", 2021-02-10T04:54:40.804 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.8575000000000002e-05 2021-02-10T04:54:40.805 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.805 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.805 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:40.805 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.217656+0000", 2021-02-10T04:54:40.805 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.002995756 2021-02-10T04:54:40.805 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.805 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.805 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:40.805 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.220651+0000", 2021-02-10T04:54:40.805 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031144010000000001 2021-02-10T04:54:40.805 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.805 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:40.805 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.806 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.806 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.806 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52017b30:::smithi08575397-43:20a)])", 2021-02-10T04:54:40.806 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:39.887305+0000", 2021-02-10T04:54:40.806 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 180.88325669899999, 2021-02-10T04:54:40.806 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003468512, 2021-02-10T04:54:40.806 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:40.806 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:54:40.806 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:40.806 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.806 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:40.806 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.887305+0000", 2021-02-10T04:54:40.807 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:40.807 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.807 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.807 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:40.807 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.887305+0000", 2021-02-10T04:54:40.807 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.1676e-05 2021-02-10T04:54:40.807 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.807 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.807 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:40.807 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.887337+0000", 2021-02-10T04:54:40.807 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.9230000000000008e-06 2021-02-10T04:54:40.807 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.807 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.808 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:40.808 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.887345+0000", 2021-02-10T04:54:40.808 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2656999999999999e-05 2021-02-10T04:54:40.808 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.808 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.808 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:40.808 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.887358+0000", 2021-02-10T04:54:40.808 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.0090000000000001e-06 2021-02-10T04:54:40.808 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.808 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.808 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:40.808 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.887365+0000", 2021-02-10T04:54:40.808 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.7717000000000003e-05 2021-02-10T04:54:40.809 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.809 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.809 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:40.809 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.887423+0000", 2021-02-10T04:54:40.809 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00335053 2021-02-10T04:54:40.809 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.809 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.809 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:40.809 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.890773+0000", 2021-02-10T04:54:40.809 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003468512 2021-02-10T04:54:40.809 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.809 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:40.809 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.810 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.810 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.810 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.3 1122/1119 [PushReplyOp(7:c765b4cc:::smithi08575397-30:309)])", 2021-02-10T04:54:40.810 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:39.889361+0000", 2021-02-10T04:54:40.810 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 180.88120098799999, 2021-02-10T04:54:40.810 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0033870189999999998, 2021-02-10T04:54:40.810 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:40.810 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:54:40.810 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:40.810 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.810 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:40.810 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.889361+0000", 2021-02-10T04:54:40.810 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:40.810 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.811 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.811 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:40.811 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.889361+0000", 2021-02-10T04:54:40.811 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.9168999999999999e-05 2021-02-10T04:54:40.811 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.811 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.811 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:40.811 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.889380+0000", 2021-02-10T04:54:40.811 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.5779999999999998e-06 2021-02-10T04:54:40.811 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.811 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.811 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:40.812 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.889385+0000", 2021-02-10T04:54:40.812 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.8790000000000003e-06 2021-02-10T04:54:40.812 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.812 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.812 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:40.812 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.889393+0000", 2021-02-10T04:54:40.812 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.972e-06 2021-02-10T04:54:40.812 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.812 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.812 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:40.812 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.889398+0000", 2021-02-10T04:54:40.812 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4211000000000001e-05 2021-02-10T04:54:40.812 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.813 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.813 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:40.813 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.889442+0000", 2021-02-10T04:54:40.813 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0033052099999999998 2021-02-10T04:54:40.813 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.813 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.813 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:40.813 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.892748+0000", 2021-02-10T04:54:40.813 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0033870189999999998 2021-02-10T04:54:40.813 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.813 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:40.813 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.813 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.813 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.814 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52cf5dd7:::smithi08575397-7:20a)])", 2021-02-10T04:54:40.814 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:39.897118+0000", 2021-02-10T04:54:40.814 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 180.87344307999999, 2021-02-10T04:54:40.814 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0045733759999999997, 2021-02-10T04:54:40.814 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:40.814 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:54:40.814 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:40.814 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.814 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:40.814 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.897118+0000", 2021-02-10T04:54:40.814 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:40.814 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.814 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.815 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:40.815 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.897118+0000", 2021-02-10T04:54:40.815 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.0298000000000001e-05 2021-02-10T04:54:40.815 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.815 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.815 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:40.815 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.897139+0000", 2021-02-10T04:54:40.815 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.7610000000000007e-06 2021-02-10T04:54:40.815 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.815 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.815 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:40.815 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.897147+0000", 2021-02-10T04:54:40.815 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2027e-05 2021-02-10T04:54:40.816 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.816 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.816 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:40.816 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.897159+0000", 2021-02-10T04:54:40.816 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.0929999999999999e-06 2021-02-10T04:54:40.816 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.816 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.816 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:40.816 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.897165+0000", 2021-02-10T04:54:40.816 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000100141 2021-02-10T04:54:40.816 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.816 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.817 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:40.817 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.897265+0000", 2021-02-10T04:54:40.817 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0044270560000000004 2021-02-10T04:54:40.817 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.817 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.817 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:40.817 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.901692+0000", 2021-02-10T04:54:40.817 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0045733759999999997 2021-02-10T04:54:40.817 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.817 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:40.817 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.817 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.817 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.817 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52017b30:::smithi08575397-43:2d9)])", 2021-02-10T04:54:40.818 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:39.946105+0000", 2021-02-10T04:54:40.818 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 180.82445648300001, 2021-02-10T04:54:40.818 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0035851820000000001, 2021-02-10T04:54:40.818 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:40.818 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:54:40.818 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:40.818 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.818 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:40.818 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946105+0000", 2021-02-10T04:54:40.818 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:40.818 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.818 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.818 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:40.819 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946105+0000", 2021-02-10T04:54:40.819 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.7836e-05 2021-02-10T04:54:40.819 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.819 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.819 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:40.819 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946123+0000", 2021-02-10T04:54:40.819 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.9430000000000003e-06 2021-02-10T04:54:40.819 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.819 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.819 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:40.819 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946130+0000", 2021-02-10T04:54:40.819 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.7319999999999993e-06 2021-02-10T04:54:40.820 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.820 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.820 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:40.820 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946140+0000", 2021-02-10T04:54:40.820 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.277e-06 2021-02-10T04:54:40.820 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.820 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.820 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:40.820 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946147+0000", 2021-02-10T04:54:40.820 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.4368000000000004e-05 2021-02-10T04:54:40.820 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.820 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.820 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:40.821 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946211+0000", 2021-02-10T04:54:40.821 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0034790260000000001 2021-02-10T04:54:40.821 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.821 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.821 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:40.821 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.949690+0000", 2021-02-10T04:54:40.821 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0035851820000000001 2021-02-10T04:54:40.821 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.821 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:40.821 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.821 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.821 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.821 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.3 1122/1119 [PushReplyOp(7:c1415b8e:::smithi08575397-38:35f)])", 2021-02-10T04:54:40.821 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:39.946563+0000", 2021-02-10T04:54:40.822 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 180.82399871300001, 2021-02-10T04:54:40.822 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031900539999999999, 2021-02-10T04:54:40.822 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:40.822 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:54:40.822 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:40.822 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.822 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:40.822 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946563+0000", 2021-02-10T04:54:40.822 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:40.822 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.822 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.822 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:40.822 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946563+0000", 2021-02-10T04:54:40.823 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.2059e-05 2021-02-10T04:54:40.823 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.823 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.823 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:40.823 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946585+0000", 2021-02-10T04:54:40.823 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9450000000000001e-06 2021-02-10T04:54:40.823 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.823 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.823 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:40.823 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946590+0000", 2021-02-10T04:54:40.823 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.9759999999999998e-06 2021-02-10T04:54:40.823 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.823 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.824 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:40.824 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946597+0000", 2021-02-10T04:54:40.824 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.9949999999999999e-06 2021-02-10T04:54:40.824 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.824 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.824 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:40.824 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946601+0000", 2021-02-10T04:54:40.824 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4035000000000001e-05 2021-02-10T04:54:40.824 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.824 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.824 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:40.824 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946645+0000", 2021-02-10T04:54:40.824 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031080439999999999 2021-02-10T04:54:40.824 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.825 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.825 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:40.825 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.949753+0000", 2021-02-10T04:54:40.825 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031900539999999999 2021-02-10T04:54:40.825 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.825 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:40.825 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.825 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.825 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.825 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52017b30:::smithi08575397-43:2d9)])", 2021-02-10T04:54:40.825 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:39.951097+0000", 2021-02-10T04:54:40.825 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 180.81946443999999, 2021-02-10T04:54:40.825 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00327655, 2021-02-10T04:54:40.826 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:40.826 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:54:40.826 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:40.826 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.826 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:40.826 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.951097+0000", 2021-02-10T04:54:40.826 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:40.826 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.826 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.826 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:40.826 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.951097+0000", 2021-02-10T04:54:40.826 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.2909e-05 2021-02-10T04:54:40.826 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.827 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.827 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:40.827 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.951120+0000", 2021-02-10T04:54:40.827 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.2708999999999999e-05 2021-02-10T04:54:40.827 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.827 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.827 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:40.827 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.951143+0000", 2021-02-10T04:54:40.827 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.9159999999999996e-06 2021-02-10T04:54:40.827 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.827 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.827 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:40.827 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.951153+0000", 2021-02-10T04:54:40.828 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9230000000000001e-06 2021-02-10T04:54:40.828 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.828 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.828 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:40.828 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.951158+0000", 2021-02-10T04:54:40.828 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.9799000000000003e-05 2021-02-10T04:54:40.828 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.828 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.828 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:40.828 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.951197+0000", 2021-02-10T04:54:40.828 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031762940000000001 2021-02-10T04:54:40.828 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.828 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.828 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:40.829 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.954374+0000", 2021-02-10T04:54:40.829 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00327655 2021-02-10T04:54:40.829 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.829 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:40.829 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.829 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.829 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.829 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:531b0702:::smithi08575397-3:300)])", 2021-02-10T04:54:40.829 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:39.963275+0000", 2021-02-10T04:54:40.829 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 180.80728703700001, 2021-02-10T04:54:40.829 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003308152, 2021-02-10T04:54:40.829 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:40.829 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:54:40.830 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:40.830 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.830 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:40.830 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.963275+0000", 2021-02-10T04:54:40.830 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:40.830 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.830 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.830 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:40.830 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.963275+0000", 2021-02-10T04:54:40.830 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.0713e-05 2021-02-10T04:54:40.830 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.830 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.830 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:40.830 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.963295+0000", 2021-02-10T04:54:40.831 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.6899999999999997e-06 2021-02-10T04:54:40.831 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.831 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.831 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:40.831 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.963301+0000", 2021-02-10T04:54:40.831 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1399e-05 2021-02-10T04:54:40.831 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.831 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.831 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:40.831 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.963312+0000", 2021-02-10T04:54:40.831 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.3340000000000003e-06 2021-02-10T04:54:40.831 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.831 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.832 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:40.832 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.963317+0000", 2021-02-10T04:54:40.832 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.8989000000000002e-05 2021-02-10T04:54:40.832 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.832 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.832 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:40.832 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.963356+0000", 2021-02-10T04:54:40.832 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0032270269999999999 2021-02-10T04:54:40.832 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.832 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.832 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:40.832 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.966583+0000", 2021-02-10T04:54:40.832 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003308152 2021-02-10T04:54:40.832 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.833 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:40.833 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.833 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.833 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.833 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52017b30:::smithi08575397-43:372)])", 2021-02-10T04:54:40.833 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:40.124429+0000", 2021-02-10T04:54:40.833 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 180.646132126, 2021-02-10T04:54:40.833 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0033528249999999998, 2021-02-10T04:54:40.833 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:40.833 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:54:40.833 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:40.833 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.833 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:40.833 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.124429+0000", 2021-02-10T04:54:40.834 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:40.834 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.834 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.834 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:40.834 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.124429+0000", 2021-02-10T04:54:40.834 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.7447000000000001e-05 2021-02-10T04:54:40.834 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.834 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.834 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:40.834 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.124447+0000", 2021-02-10T04:54:40.834 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0030000000000002e-06 2021-02-10T04:54:40.834 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.834 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.835 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:40.835 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.124452+0000", 2021-02-10T04:54:40.835 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.2669999999999995e-06 2021-02-10T04:54:40.835 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.835 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.835 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:40.835 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.124461+0000", 2021-02-10T04:54:40.835 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.2230000000000001e-06 2021-02-10T04:54:40.835 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.835 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.835 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:40.835 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.124465+0000", 2021-02-10T04:54:40.835 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.5660000000000001e-05 2021-02-10T04:54:40.836 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.836 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.836 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:40.836 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.124501+0000", 2021-02-10T04:54:40.836 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003281225 2021-02-10T04:54:40.836 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.836 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.836 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:40.836 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.127782+0000", 2021-02-10T04:54:40.836 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0033528249999999998 2021-02-10T04:54:40.836 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.836 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:40.836 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.837 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.837 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.837 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52017b30:::smithi08575397-43:33a)])", 2021-02-10T04:54:40.837 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:40.158898+0000", 2021-02-10T04:54:40.837 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 180.611663303, 2021-02-10T04:54:40.837 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003157211, 2021-02-10T04:54:40.837 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:40.837 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:54:40.837 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:40.837 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.837 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:40.837 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.158898+0000", 2021-02-10T04:54:40.837 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:40.837 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.838 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.838 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:40.838 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.158898+0000", 2021-02-10T04:54:40.838 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.8318999999999999e-05 2021-02-10T04:54:40.838 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.838 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.838 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:40.838 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.158917+0000", 2021-02-10T04:54:40.838 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.2360000000000004e-06 2021-02-10T04:54:40.838 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.838 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.838 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:40.838 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.158922+0000", 2021-02-10T04:54:40.839 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.7189999999999996e-06 2021-02-10T04:54:40.839 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.839 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.839 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:40.839 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.158932+0000", 2021-02-10T04:54:40.839 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.5459999999999999e-06 2021-02-10T04:54:40.839 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.839 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.839 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:40.839 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.158937+0000", 2021-02-10T04:54:40.839 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.625e-05 2021-02-10T04:54:40.839 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.839 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.840 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:40.840 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.158973+0000", 2021-02-10T04:54:40.840 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003082141 2021-02-10T04:54:40.840 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.840 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.840 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:40.840 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.162055+0000", 2021-02-10T04:54:40.840 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003157211 2021-02-10T04:54:40.840 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.840 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:40.840 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.840 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.840 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.841 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52017b30:::smithi08575397-43:33a)])", 2021-02-10T04:54:40.841 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:40.163313+0000", 2021-02-10T04:54:40.841 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 180.60724873199999, 2021-02-10T04:54:40.841 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0032056989999999998, 2021-02-10T04:54:40.841 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:40.841 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:54:40.841 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:40.841 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.841 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:40.841 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.163313+0000", 2021-02-10T04:54:40.841 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:40.841 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.841 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.841 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:40.842 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.163313+0000", 2021-02-10T04:54:40.842 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.8493999999999999e-05 2021-02-10T04:54:40.842 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.842 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.842 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:40.842 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.163331+0000", 2021-02-10T04:54:40.842 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.1150000000000002e-06 2021-02-10T04:54:40.842 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.842 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.842 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:40.842 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.163336+0000", 2021-02-10T04:54:40.842 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.4429999999999992e-06 2021-02-10T04:54:40.842 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.843 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.843 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:40.843 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.163346+0000", 2021-02-10T04:54:40.843 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.767e-06 2021-02-10T04:54:40.843 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.843 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.843 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:40.843 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.163352+0000", 2021-02-10T04:54:40.843 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.5129000000000002e-05 2021-02-10T04:54:40.843 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.843 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.843 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:40.843 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.163387+0000", 2021-02-10T04:54:40.843 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031317509999999999 2021-02-10T04:54:40.844 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.844 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.844 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:40.844 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.166519+0000", 2021-02-10T04:54:40.844 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0032056989999999998 2021-02-10T04:54:40.844 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.844 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:40.844 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.844 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.844 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.844 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52cf5dd7:::smithi08575397-7:33e)])", 2021-02-10T04:54:40.844 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:40.173135+0000", 2021-02-10T04:54:40.844 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 180.59742663200001, 2021-02-10T04:54:40.845 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003144015, 2021-02-10T04:54:40.845 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:40.845 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:54:40.845 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:40.845 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.845 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:40.845 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.173135+0000", 2021-02-10T04:54:40.845 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:40.845 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.845 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.845 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:40.845 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.173135+0000", 2021-02-10T04:54:40.845 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.7708999999999999e-05 2021-02-10T04:54:40.845 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.846 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.846 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:40.846 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.173153+0000", 2021-02-10T04:54:40.846 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.921e-06 2021-02-10T04:54:40.846 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.846 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.846 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:40.846 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.173158+0000", 2021-02-10T04:54:40.846 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.3769999999999996e-06 2021-02-10T04:54:40.846 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.846 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.846 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:40.846 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.173166+0000", 2021-02-10T04:54:40.847 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.4409999999999999e-06 2021-02-10T04:54:40.847 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.847 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.847 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:40.847 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.173175+0000", 2021-02-10T04:54:40.847 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.6347999999999998e-05 2021-02-10T04:54:40.847 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.847 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.847 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:40.847 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.173212+0000", 2021-02-10T04:54:40.847 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0030672189999999999 2021-02-10T04:54:40.847 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.847 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.848 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:40.848 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.176279+0000", 2021-02-10T04:54:40.848 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003144015 2021-02-10T04:54:40.848 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.848 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:40.848 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.848 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.848 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.848 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52cf5dd7:::smithi08575397-7:33e)])", 2021-02-10T04:54:40.848 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:40.180475+0000", 2021-02-10T04:54:40.848 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 180.590086789, 2021-02-10T04:54:40.848 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031979669999999999, 2021-02-10T04:54:40.848 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:40.848 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:54:40.849 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:40.849 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.849 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:40.849 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.180475+0000", 2021-02-10T04:54:40.849 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:40.849 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.849 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.849 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:40.849 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.180475+0000", 2021-02-10T04:54:40.849 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.9508e-05 2021-02-10T04:54:40.849 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.849 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.849 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:40.850 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.180494+0000", 2021-02-10T04:54:40.850 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.7320000000000003e-06 2021-02-10T04:54:40.850 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.850 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.850 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:40.850 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.180500+0000", 2021-02-10T04:54:40.850 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.4909999999999994e-06 2021-02-10T04:54:40.850 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.850 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.850 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:40.850 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.180510+0000", 2021-02-10T04:54:40.850 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.3980000000000004e-06 2021-02-10T04:54:40.851 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.851 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.851 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:40.851 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.180516+0000", 2021-02-10T04:54:40.851 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.1493999999999999e-05 2021-02-10T04:54:40.851 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.851 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.851 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:40.851 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.180557+0000", 2021-02-10T04:54:40.851 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031153439999999999 2021-02-10T04:54:40.851 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.851 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.851 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:40.851 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.183673+0000", 2021-02-10T04:54:40.852 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031979669999999999 2021-02-10T04:54:40.852 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.852 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:40.852 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.852 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.852 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.852 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52017b30:::smithi08575397-43:349)])", 2021-02-10T04:54:40.852 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:40.214713+0000", 2021-02-10T04:54:40.852 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 180.55584833500001, 2021-02-10T04:54:40.852 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031374720000000001, 2021-02-10T04:54:40.852 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:40.852 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:54:40.853 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:40.853 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.853 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:40.853 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.214713+0000", 2021-02-10T04:54:40.853 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:40.853 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.853 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.853 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:40.853 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.214713+0000", 2021-02-10T04:54:40.853 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.8167000000000001e-05 2021-02-10T04:54:40.853 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.853 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.853 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:40.853 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.214731+0000", 2021-02-10T04:54:40.854 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0479999999999998e-06 2021-02-10T04:54:40.854 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.854 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.854 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:40.854 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.214736+0000", 2021-02-10T04:54:40.854 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.4749999999999999e-06 2021-02-10T04:54:40.854 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.854 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.854 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:40.854 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.214746+0000", 2021-02-10T04:54:40.854 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.7610000000000004e-06 2021-02-10T04:54:40.854 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.855 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.855 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:40.855 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.214752+0000", 2021-02-10T04:54:40.855 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.621e-05 2021-02-10T04:54:40.855 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.855 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.855 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:40.855 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.214798+0000", 2021-02-10T04:54:40.855 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0030528109999999999 2021-02-10T04:54:40.855 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.855 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.855 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:40.855 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.217851+0000", 2021-02-10T04:54:40.855 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031374720000000001 2021-02-10T04:54:40.856 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.856 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:40.856 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.856 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.856 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.856 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDScrubReserve(2.e REQUEST e1187)", 2021-02-10T04:54:40.856 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:53:41.711530+0000", 2021-02-10T04:54:40.856 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 59.059031822000001, 2021-02-10T04:54:40.856 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031529919999999999, 2021-02-10T04:54:40.856 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:54:40.856 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:54:40.856 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:54:40.856 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.857 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:54:40.857 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:41.711530+0000", 2021-02-10T04:54:40.857 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:54:40.857 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.857 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.857 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:54:40.857 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:41.711530+0000", 2021-02-10T04:54:40.857 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.5733e-05 2021-02-10T04:54:40.857 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.857 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.857 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:54:40.857 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:41.711545+0000", 2021-02-10T04:54:40.857 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.1470000000000003e-06 2021-02-10T04:54:40.858 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.858 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.858 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:54:40.858 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:41.711550+0000", 2021-02-10T04:54:40.858 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.8300000000000001e-06 2021-02-10T04:54:40.858 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.858 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.858 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:54:40.858 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:41.711555+0000", 2021-02-10T04:54:40.858 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.4179999999999996e-06 2021-02-10T04:54:40.858 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.858 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.858 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:54:40.858 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:41.711561+0000", 2021-02-10T04:54:40.859 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.8992000000000001e-05 2021-02-10T04:54:40.859 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.859 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.859 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:40.859 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:41.711610+0000", 2021-02-10T04:54:40.859 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.3280000000000006e-06 2021-02-10T04:54:40.859 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.859 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.859 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op must wait for map", 2021-02-10T04:54:40.859 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:41.711618+0000", 2021-02-10T04:54:40.859 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0030070909999999999 2021-02-10T04:54:40.859 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.859 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.860 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:54:40.860 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:41.714625+0000", 2021-02-10T04:54:40.860 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.292e-05 2021-02-10T04:54:40.860 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.860 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.860 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:54:40.860 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:41.714638+0000", 2021-02-10T04:54:40.860 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4533000000000002e-05 2021-02-10T04:54:40.860 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:40.860 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:40.860 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:54:40.860 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:41.714683+0000", 2021-02-10T04:54:40.860 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031529919999999999 2021-02-10T04:54:40.860 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.861 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:40.861 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.861 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:40.861 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:40.861 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:54:40.862 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:54:40.862 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:54:40.862 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:54:40.862 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:54:40.864 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:54:40.886 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:54:40.886 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-02-10T04:54:40.886 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: problem getting command descriptions from osd.0 2021-02-10T04:54:40.886 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:54:40.887 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:54:40.887 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:54:40.887 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:54:40.887 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:54:40.887 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:54:40.887 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:54:41.649 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is set 2021-02-10T04:54:41.881 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2021-02-10T04:54:41.882 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:54:41.882 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd dump -f json-pretty 2021-02-10T04:54:41.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:41 smithi164 conmon[29288]: cluster 2021-02-10T04:54:40.479239+0000 mgr.y (mgr.14140) 2021-02-10T04:54:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:41 smithi164 conmon[29288]: 1336 : cluster [DBG] pgmap v2193: 204 pgs: 1 active+clean+scrubbing, 3 peering, 6 unknown, 5 active+clean+remapped, 24 active+undersized+degraded, 36 active+undersized, 129 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:41 smithi164 conmon[29288]: cluster 2021-02-10T04:54:40.546340+0000 mon.a (mon.0) 2589 : cluster [DBG] osdmap e1213: 8 total, 7 up, 8 in 2021-02-10T04:54:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:41 smithi164 conmon[29288]: audit 2021-02-10T04:54:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:41 smithi164 conmon[29288]: 2021-02-10T04:54:40.738422+0000 mon.b (mon.2) 49 : audit [DBG] from='client.? 172.21.15.85:0/1496343451' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:54:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:41 smithi164 conmon[29288]: audit 2021-02-10T04:54:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:41 smithi164 conmon[29288]: 2021-02-10T04:54:40.954052 2021-02-10T04:54:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:41 smithi164 conmon[29288]: +0000 mon.c (mon.1) 117 : audit [INF] 2021-02-10T04:54:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:41 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/3817571160' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:54:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:41 smithi164 conmon[29288]: audit 2021-02-10T04:54: 2021-02-10T04:54:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:41 smithi164 conmon[29288]: 40.954376+0000 mon.a (mon.0) 2590 : 2021-02-10T04:54:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:41 smithi164 conmon[29288]: audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:54:41.909 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:41 smithi085 conmon[29319]: cluster 2021-02-10T04:54:40.479239+0000 mgr.y (mgr.14140) 1336 : cluster [DBG] pgmap v2193: 204 pgs: 1 active+clean+scrubbing, 3 peering, 6 unknown, 5 active+clean+remapped, 24 active+undersized+degraded, 36 active+undersized, 129 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:41.909 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:41 smithi085 conmon[29319]: cluster 2021-02-10T04:54:40.546340+0000 mon.a (mon.0) 2589 : cluster [DBG] osdmap e1213: 8 total, 7 up, 8 in 2021-02-10T04:54:41.909 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:41 smithi085 conmon[29319]: audit 2021-02-10T04:54:40.738422+0000 mon.b (mon.2) 49 : audit [DBG] from='client.? 172.21.15.85:0/1496343451' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:54:41.909 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:41 smithi085 conmon[29319]: audit 2021-02-10T04:54:40.954052+0000 mon.c (mon.1) 117 : audit [INF] from='client.? 172.21.15.85:0/3817571160' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:54:41.909 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:41 smithi085 conmon[29319]: audit 2021-02-10T04:54:40.954376+0000 mon.a (mon.0) 2590 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:54:41.909 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:41 smithi085 conmon[35961]: cluster 2021-02-10T04:54:40.479239+0000 mgr.y (mgr.14140) 1336 : cluster [DBG] pgmap v2193: 204 pgs: 1 active+clean+scrubbing, 3 peering, 6 unknown, 5 active+clean+remapped, 24 active+undersized+degraded, 36 active+undersized, 129 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:41.909 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:41 smithi085 conmon[35961]: cluster 2021-02-10T04:54:40.546340+0000 mon.a (mon.0) 2589 : cluster [DBG] osdmap e1213: 8 total, 7 up, 8 in 2021-02-10T04:54:41.909 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:41 smithi085 conmon[35961]: audit 2021-02-10T04:54:40.738422+0000 mon.b (mon.2) 49 : audit [DBG] from='client.? 172.21.15.85:0/1496343451' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:54:41.909 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:41 smithi085 conmon[35961]: audit 2021-02-10T04:54:40.954052+0000 mon.c (mon.1) 117 : audit [INF] from='client.? 172.21.15.85:0/3817571160' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:54:41.909 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:41 smithi085 conmon[35961]: audit 2021-02-10T04:54:40.954376+0000 mon.a (mon.0) 2590 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:54:42.192 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:54:42.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:42 smithi085 conmon[29319]: cluster 2021-02-10T04:54:41.638450+0000 mon.a (mon 2021-02-10T04:54:42.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:42 smithi085 conmon[29319]: .0) 2591 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:54:42.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:42 smithi085 conmon[29319]: audit 2021-02-10T04:54:41.642746+0000 mon.a (mon.0) 2592 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:54:42.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:42 smithi085 conmon[29319]: cluster 2021-02-10T04:54:41.642807+0000 mon.a (mon.0) 2593 : cluster [DBG] osdmap e1214: 8 total, 7 up, 8 in 2021-02-10T04:54:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:42 smithi085 conmon[35961]: cluster 2021-02-10T04:54:41.638450+0000 mon.a (mon. 2021-02-10T04:54:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:42 smithi085 conmon[35961]: 0) 2591 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:54:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:42 smithi085 conmon[35961]: audit 2021-02-10T04:54:41 2021-02-10T04:54:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:42 smithi085 conmon[35961]: .642746+0000 mon.a (mon.0) 2592 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:54:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:42 smithi085 conmon[35961]: cluster 2021-02-10T04:54:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:42 smithi085 conmon[35961]: 2021-02-10T04:54:41. 2021-02-10T04:54:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:42 smithi085 conmon[35961]: 642807+0000 mon.a (mon.0) 2593 : cluster [DBG] osdmap e1214: 8 total, 7 up, 8 in 2021-02-10T04:54:42.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:42 smithi085 conmon[35961]: 2021-02-10T04:54:42.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:42 smithi164 conmon[29288]: cluster 2021-02-10T04:54:41.638450+0000 mon.a (mon.0) 2591 : cluster 2021-02-10T04:54:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:42 smithi164 conmon[29288]: [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:54:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:42 smithi164 conmon[29288]: audit 2021-02-10T04:54:41.642746+0000 mon.a (mon.0) 2592 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:54:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:42 smithi164 conmon[29288]: cluster 2021-02-10T04:54:41.642807+0000 mon.a (mon.0) 2593 : 2021-02-10T04:54:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:42 smithi164 conmon[29288]: cluster [DBG] osdmap e1214: 8 total, 7 up, 8 in 2021-02-10T04:54:43.058 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.4 dump_ops_in_flight 2021-02-10T04:54:43.472 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:54:43.472 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:54:43.623 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:54:43.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:43 smithi085 conmon[29319]: cluster 2021-02-10T04:54:42.479973+0000 2021-02-10T04:54:43.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:43 smithi085 conmon[29319]: mgr.y (mgr.14140) 1337 : cluster [DBG] pgmap v2196: 204 pgs: 1 active+clean+scrubbing, 2 peering, 4 unknown, 6 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 130 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:43.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:43 smithi085 conmon[29319]: cluster 2021-02-10T04:54:43.543657+0000 mon.a (mon.0) 2594 : cluster [DBG] osdmap e1215: 8 total, 7 up, 8 in 2021-02-10T04:54:43.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:43 smithi085 conmon[35961]: cluster 2021-02-10T04:54:43.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:43 smithi085 conmon[35961]: 2021-02-10T04:54:42.479973+0000 mgr.y (mgr.14140) 1337 : cluster [DBG] pgmap v2196: 204 pgs: 1 active+clean+scrubbing, 2 peering, 4 unknown, 6 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 130 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:43.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:43 smithi085 conmon[35961]: cluster 2021-02-10T04:54 2021-02-10T04:54:43.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:43 smithi085 conmon[35961]: :43.543657+0000 mon.a (mon.0) 2594 : cluster [DBG] osdmap e1215: 8 total, 7 up, 8 in 2021-02-10T04:54:43.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:43 smithi085 conmon[35961]: 2021-02-10T04:54:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:43 smithi164 conmon[29288]: cluster 2021-02-10T04:54:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:43 smithi164 conmon[29288]: 2021-02-10 2021-02-10T04:54:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:43 smithi164 conmon[29288]: T04:54:42.479973+0000 mgr.y (mgr.14140) 1337 : cluster [DBG] pgmap v2196: 204 pgs: 1 active+clean+scrubbing, 2 peering, 4 unknown, 6 active+clean+remapped, 24 active+undersized+degraded, 37 active+undersized, 130 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:43 smithi164 conmon[29288]: cluster 2021-02-10T04:54:43.543657+0000 mon.a ( 2021-02-10T04:54:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:43 smithi164 conmon[29288]: mon.0) 2594 : cluster [DBG] osdmap e1215: 8 total, 7 up, 8 in 2021-02-10T04:54:45.127 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset noscrub 2021-02-10T04:54:45.172 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:54:45.172 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:54:45.172 INFO:teuthology.orchestra.run.smithi085.stdout: "epoch": 1215, 2021-02-10T04:54:45.172 INFO:teuthology.orchestra.run.smithi085.stdout: "fsid": "42224fce-6b57-11eb-8feb-001a4aab830c", 2021-02-10T04:54:45.173 INFO:teuthology.orchestra.run.smithi085.stdout: "created": "2021-02-10T04:20:57.933279+0000", 2021-02-10T04:54:45.173 INFO:teuthology.orchestra.run.smithi085.stdout: "modified": "2021-02-10T04:54:43.538434+0000", 2021-02-10T04:54:45.173 INFO:teuthology.orchestra.run.smithi085.stdout: "last_up_change": "2021-02-10T04:52:31.054977+0000", 2021-02-10T04:54:45.173 INFO:teuthology.orchestra.run.smithi085.stdout: "last_in_change": "2021-02-10T04:51:35.701776+0000", 2021-02-10T04:54:45.173 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T04:54:45.173 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_num": 5806080, 2021-02-10T04:54:45.173 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_set": [ 2021-02-10T04:54:45.173 INFO:teuthology.orchestra.run.smithi085.stdout: "nodeep-scrub", 2021-02-10T04:54:45.173 INFO:teuthology.orchestra.run.smithi085.stdout: "noscrub", 2021-02-10T04:54:45.173 INFO:teuthology.orchestra.run.smithi085.stdout: "pglog_hardlimit", 2021-02-10T04:54:45.173 INFO:teuthology.orchestra.run.smithi085.stdout: "purged_snapdirs", 2021-02-10T04:54:45.173 INFO:teuthology.orchestra.run.smithi085.stdout: "recovery_deletes", 2021-02-10T04:54:45.173 INFO:teuthology.orchestra.run.smithi085.stdout: "sortbitwise" 2021-02-10T04:54:45.174 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:54:45.174 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_version": 18, 2021-02-10T04:54:45.174 INFO:teuthology.orchestra.run.smithi085.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T04:54:45.174 INFO:teuthology.orchestra.run.smithi085.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T04:54:45.174 INFO:teuthology.orchestra.run.smithi085.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T04:54:45.174 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_snapshot": "", 2021-02-10T04:54:45.174 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_max": 7, 2021-02-10T04:54:45.174 INFO:teuthology.orchestra.run.smithi085.stdout: "max_osd": 8, 2021-02-10T04:54:45.174 INFO:teuthology.orchestra.run.smithi085.stdout: "require_min_compat_client": "luminous", 2021-02-10T04:54:45.174 INFO:teuthology.orchestra.run.smithi085.stdout: "min_compat_client": "luminous", 2021-02-10T04:54:45.175 INFO:teuthology.orchestra.run.smithi085.stdout: "require_osd_release": "quincy", 2021-02-10T04:54:45.175 INFO:teuthology.orchestra.run.smithi085.stdout: "pools": [ 2021-02-10T04:54:45.175 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.175 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 1, 2021-02-10T04:54:45.175 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "device_health_metrics", 2021-02-10T04:54:45.175 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:24:05.258028+0000", 2021-02-10T04:54:45.175 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:54:45.175 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:54:45.175 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:54:45.176 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:54:45.176 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:54:45.176 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:54:45.176 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:54:45.176 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:54:45.176 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:54:45.176 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:54:45.176 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:54:45.176 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:54:45.176 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 11, 2021-02-10T04:54:45.176 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 11, 2021-02-10T04:54:45.176 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 11, 2021-02-10T04:54:45.176 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 11, 2021-02-10T04:54:45.177 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 11, 2021-02-10T04:54:45.177 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:54:45.177 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:54:45.177 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:54:45.177 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:54:45.177 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:54:45.177 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:54:45.177 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:54:45.177 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.177 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "112", 2021-02-10T04:54:45.177 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:54:45.177 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:54:45.177 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "110", 2021-02-10T04:54:45.178 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:54:45.178 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:54:45.178 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:54:45.178 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:54:45.178 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:54:45.178 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:54:45.178 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:54:45.178 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:54:45.178 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:54:45.178 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:54:45.178 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:54:45.178 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:54:45.178 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:54:45.179 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:54:45.179 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:54:45.179 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:54:45.179 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:54:45.179 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:54:45.179 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:54:45.179 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:54:45.179 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:54:45.179 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:54:45.179 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:54:45.179 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.179 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:54:45.180 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:54:45.180 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:54:45.180 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:54:45.180 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:54:45.180 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:54:45.180 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:54:45.180 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:54:45.180 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:54:45.180 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:54:45.180 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:54:45.180 INFO:teuthology.orchestra.run.smithi085.stdout: "options": { 2021-02-10T04:54:45.181 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_min": 1 2021-02-10T04:54:45.181 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.181 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:54:45.181 INFO:teuthology.orchestra.run.smithi085.stdout: "mgr_devicehealth": {} 2021-02-10T04:54:45.181 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.181 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.181 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.181 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 2, 2021-02-10T04:54:45.181 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": ".rgw.root", 2021-02-10T04:54:45.181 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:32.688554+0000", 2021-02-10T04:54:45.181 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:54:45.181 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:54:45.181 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:54:45.182 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:54:45.182 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:54:45.182 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:54:45.182 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:54:45.182 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:54:45.182 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:54:45.182 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:54:45.182 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:54:45.182 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:54:45.182 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 42, 2021-02-10T04:54:45.182 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 42, 2021-02-10T04:54:45.182 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 42, 2021-02-10T04:54:45.183 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 42, 2021-02-10T04:54:45.183 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 42, 2021-02-10T04:54:45.183 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:54:45.183 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:54:45.183 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:54:45.183 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:54:45.183 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:54:45.183 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:54:45.183 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:54:45.183 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.183 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "91", 2021-02-10T04:54:45.183 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:54:45.183 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:54:45.184 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "89", 2021-02-10T04:54:45.184 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:54:45.184 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:54:45.184 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:54:45.184 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:54:45.184 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:54:45.184 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:54:45.184 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:54:45.184 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:54:45.184 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:54:45.184 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:54:45.184 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:54:45.184 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:54:45.185 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:54:45.185 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:54:45.185 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:54:45.185 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:54:45.185 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:54:45.185 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:54:45.185 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:54:45.185 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:54:45.185 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:54:45.185 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:54:45.185 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:54:45.185 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.185 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:54:45.186 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:54:45.186 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:54:45.186 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:54:45.186 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:54:45.186 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:54:45.186 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:54:45.186 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:54:45.186 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:54:45.186 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:54:45.186 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:54:45.186 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:54:45.186 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:54:45.187 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T04:54:45.187 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.187 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.187 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.187 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 3, 2021-02-10T04:54:45.187 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "iscsi", 2021-02-10T04:54:45.187 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:47.658613+0000", 2021-02-10T04:54:45.187 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:54:45.187 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:54:45.187 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:54:45.187 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:54:45.187 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:54:45.187 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:54:45.188 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:54:45.188 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:54:45.188 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:54:45.188 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:54:45.188 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:54:45.188 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:54:45.188 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 13, 2021-02-10T04:54:45.188 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 13, 2021-02-10T04:54:45.188 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 13, 2021-02-10T04:54:45.188 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 13, 2021-02-10T04:54:45.188 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 13, 2021-02-10T04:54:45.188 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:54:45.188 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:54:45.189 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:54:45.189 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:54:45.189 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:54:45.189 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:54:45.189 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:54:45.189 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.189 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "1211", 2021-02-10T04:54:45.189 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:54:45.189 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:54:45.189 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "1209", 2021-02-10T04:54:45.189 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:54:45.189 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:54:45.189 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:54:45.190 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:54:45.190 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:54:45.190 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:54:45.190 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:54:45.190 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:54:45.190 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:54:45.190 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:54:45.190 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:54:45.190 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:54:45.190 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:54:45.190 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:54:45.190 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:54:45.191 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:54:45.191 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:54:45.191 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:54:45.191 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:54:45.191 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:54:45.191 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:54:45.191 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:54:45.191 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:54:45.191 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.191 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:54:45.191 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:54:45.191 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:54:45.191 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:54:45.192 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:54:45.192 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:54:45.192 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:54:45.192 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:54:45.192 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:54:45.192 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:54:45.192 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:54:45.192 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:54:45.192 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:54:45.192 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd": {} 2021-02-10T04:54:45.192 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.192 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.192 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.193 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 4, 2021-02-10T04:54:45.193 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "zone.rgw.log", 2021-02-10T04:54:45.193 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:47.968727+0000", 2021-02-10T04:54:45.193 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:54:45.193 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:54:45.193 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:54:45.193 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:54:45.193 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:54:45.193 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:54:45.193 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:54:45.193 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:54:45.193 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:54:45.193 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:54:45.194 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:54:45.194 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:54:45.194 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 28, 2021-02-10T04:54:45.194 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 22, 2021-02-10T04:54:45.194 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 22, 2021-02-10T04:54:45.194 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 22, 2021-02-10T04:54:45.194 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 28, 2021-02-10T04:54:45.194 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:54:45.194 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "4.1c", 2021-02-10T04:54:45.194 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 1184, 2021-02-10T04:54:45.194 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 1183, 2021-02-10T04:54:45.194 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 1183, 2021-02-10T04:54:45.195 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "1049'39", 2021-02-10T04:54:45.195 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "1049'33" 2021-02-10T04:54:45.195 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.195 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "1184", 2021-02-10T04:54:45.195 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:54:45.195 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "1184", 2021-02-10T04:54:45.195 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "1182", 2021-02-10T04:54:45.195 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:54:45.195 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:54:45.195 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:54:45.195 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:54:45.195 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:54:45.195 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:54:45.196 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:54:45.196 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:54:45.196 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:54:45.196 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:54:45.196 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:54:45.196 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:54:45.196 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:54:45.196 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:54:45.196 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:54:45.196 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:54:45.196 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:54:45.196 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:54:45.197 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:54:45.197 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:54:45.197 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:54:45.197 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:54:45.197 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:54:45.197 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.197 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:54:45.197 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:54:45.197 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:54:45.197 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:54:45.197 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:54:45.198 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:54:45.198 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:54:45.198 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:54:45.198 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:54:45.198 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:54:45.198 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:54:45.198 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:54:45.198 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:54:45.198 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T04:54:45.198 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.198 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.198 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.198 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 5, 2021-02-10T04:54:45.199 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "zone.rgw.control", 2021-02-10T04:54:45.199 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:49.278260+0000", 2021-02-10T04:54:45.199 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:54:45.199 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:54:45.199 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:54:45.199 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:54:45.199 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:54:45.199 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:54:45.199 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:54:45.199 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:54:45.199 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:54:45.199 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:54:45.199 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:54:45.200 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:54:45.200 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 32, 2021-02-10T04:54:45.200 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 32, 2021-02-10T04:54:45.200 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 32, 2021-02-10T04:54:45.200 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 32, 2021-02-10T04:54:45.200 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 32, 2021-02-10T04:54:45.200 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:54:45.200 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:54:45.200 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:54:45.200 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:54:45.201 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:54:45.201 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:54:45.201 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:54:45.201 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.201 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "59", 2021-02-10T04:54:45.201 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:54:45.201 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:54:45.201 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T04:54:45.201 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:54:45.201 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:54:45.201 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:54:45.201 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:54:45.202 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:54:45.202 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:54:45.202 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:54:45.202 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:54:45.202 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:54:45.202 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:54:45.202 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:54:45.202 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:54:45.202 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:54:45.202 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:54:45.202 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:54:45.202 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:54:45.203 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:54:45.203 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:54:45.203 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:54:45.203 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:54:45.203 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:54:45.203 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:54:45.203 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:54:45.203 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.203 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:54:45.203 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:54:45.203 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:54:45.203 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:54:45.203 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:54:45.203 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:54:45.204 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:54:45.204 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:54:45.204 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:54:45.204 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:54:45.204 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:54:45.204 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:54:45.204 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:54:45.204 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T04:54:45.204 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.204 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.204 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.204 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 6, 2021-02-10T04:54:45.204 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "zone.rgw.meta", 2021-02-10T04:54:45.205 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:51.792514+0000", 2021-02-10T04:54:45.205 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:54:45.205 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:54:45.205 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:54:45.205 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:54:45.205 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:54:45.205 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:54:45.205 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:54:45.205 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:54:45.205 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:54:45.205 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:54:45.205 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:54:45.205 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:54:45.206 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 42, 2021-02-10T04:54:45.206 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 42, 2021-02-10T04:54:45.206 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 42, 2021-02-10T04:54:45.206 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 42, 2021-02-10T04:54:45.206 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 42, 2021-02-10T04:54:45.206 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:54:45.206 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:54:45.206 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:54:45.206 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:54:45.206 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:54:45.206 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:54:45.206 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:54:45.207 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.207 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "1046", 2021-02-10T04:54:45.207 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:54:45.207 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:54:45.207 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "1044", 2021-02-10T04:54:45.207 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:54:45.207 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:54:45.207 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:54:45.207 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:54:45.207 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:54:45.207 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:54:45.207 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:54:45.207 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:54:45.207 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:54:45.208 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:54:45.208 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:54:45.208 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:54:45.208 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:54:45.208 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:54:45.208 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:54:45.208 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:54:45.208 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:54:45.208 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:54:45.208 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:54:45.208 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:54:45.208 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:54:45.208 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:54:45.209 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:54:45.209 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.209 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:54:45.209 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:54:45.209 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:54:45.209 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:54:45.209 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:54:45.209 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:54:45.209 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:54:45.209 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:54:45.209 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:54:45.209 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:54:45.209 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:54:45.210 INFO:teuthology.orchestra.run.smithi085.stdout: "options": { 2021-02-10T04:54:45.210 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_bias": 4, 2021-02-10T04:54:45.210 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_min": 8 2021-02-10T04:54:45.210 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.210 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:54:45.210 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T04:54:45.210 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.210 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.210 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.210 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 7, 2021-02-10T04:54:45.210 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "unique_pool_0", 2021-02-10T04:54:45.210 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:28:23.932391+0000", 2021-02-10T04:54:45.211 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 8193, 2021-02-10T04:54:45.211 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2021-02-10T04:54:45.211 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:54:45.211 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:54:45.211 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:54:45.211 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:54:45.211 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:54:45.211 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:54:45.211 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:54:45.211 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:54:45.211 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:54:45.211 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:54:45.211 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 36, 2021-02-10T04:54:45.212 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 36, 2021-02-10T04:54:45.212 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 36, 2021-02-10T04:54:45.212 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 36, 2021-02-10T04:54:45.212 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 36, 2021-02-10T04:54:45.212 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:54:45.212 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:54:45.212 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:54:45.212 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:54:45.212 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:54:45.212 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:54:45.212 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:54:45.212 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.212 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "1143", 2021-02-10T04:54:45.213 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:54:45.213 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:54:45.213 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "1141", 2021-02-10T04:54:45.213 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:54:45.213 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:54:45.213 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 888, 2021-02-10T04:54:45.213 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 994, 2021-02-10T04:54:45.213 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:54:45.213 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:54:45.213 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:54:45.213 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:54:45.213 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:54:45.214 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:54:45.214 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:54:45.214 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:54:45.214 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:54:45.214 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:54:45.214 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:54:45.214 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:54:45.214 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:54:45.214 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:54:45.214 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:54:45.214 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:54:45.214 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:54:45.215 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:54:45.215 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:54:45.215 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.215 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:54:45.215 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:54:45.215 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:54:45.215 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:54:45.215 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:54:45.215 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:54:45.215 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:54:45.215 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:54:45.215 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:54:45.215 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:54:45.216 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:54:45.216 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:54:45.216 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:54:45.216 INFO:teuthology.orchestra.run.smithi085.stdout: "rados": {} 2021-02-10T04:54:45.216 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.216 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.216 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:54:45.216 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T04:54:45.216 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.216 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 0, 2021-02-10T04:54:45.216 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "45d0bb5f-35fa-4596-964f-75abcee53bee", 2021-02-10T04:54:45.216 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 0, 2021-02-10T04:54:45.216 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:54:45.217 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:54:45.217 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:54:45.217 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:54:45.217 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:54:45.217 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 8, 2021-02-10T04:54:45.217 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1146, 2021-02-10T04:54:45.217 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 1147, 2021-02-10T04:54:45.217 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:54:45.217 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:54:45.217 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.217 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.217 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.217 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6801", 2021-02-10T04:54:45.218 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.218 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.218 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.218 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.218 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:54:45.218 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.218 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.218 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.218 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6802", 2021-02-10T04:54:45.218 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.218 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.218 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.218 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.219 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:54:45.219 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.219 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.219 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.219 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6804", 2021-02-10T04:54:45.219 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.219 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.219 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.219 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.219 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:54:45.219 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.219 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.219 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.220 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6803", 2021-02-10T04:54:45.220 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.220 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.220 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.220 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.220 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6801/7", 2021-02-10T04:54:45.220 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6802/7", 2021-02-10T04:54:45.220 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6804/7", 2021-02-10T04:54:45.220 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6803/7", 2021-02-10T04:54:45.220 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:54:45.220 INFO:teuthology.orchestra.run.smithi085.stdout: "exists" 2021-02-10T04:54:45.220 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.220 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.221 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.221 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 1, 2021-02-10T04:54:45.221 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "3421a5ce-872e-4b52-adf7-6aaa595871a4", 2021-02-10T04:54:45.221 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:54:45.221 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:54:45.221 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:54:45.221 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 0.86566162109375, 2021-02-10T04:54:45.221 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:54:45.221 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:54:45.221 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 13, 2021-02-10T04:54:45.221 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1212, 2021-02-10T04:54:45.221 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:54:45.221 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:54:45.222 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:54:45.222 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.222 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.222 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.222 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6805", 2021-02-10T04:54:45.222 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.222 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.222 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.222 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.222 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:54:45.222 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.222 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.223 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.223 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6806", 2021-02-10T04:54:45.223 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.223 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.223 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.223 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.223 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:54:45.223 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.223 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.223 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.223 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6808", 2021-02-10T04:54:45.223 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.223 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.223 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.224 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.224 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:54:45.224 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.224 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.224 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.224 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6807", 2021-02-10T04:54:45.224 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.224 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.224 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.224 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.224 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6805/7", 2021-02-10T04:54:45.224 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6806/7", 2021-02-10T04:54:45.224 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6808/7", 2021-02-10T04:54:45.225 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6807/7", 2021-02-10T04:54:45.225 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:54:45.225 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:54:45.225 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:54:45.225 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.225 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.225 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.225 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 2, 2021-02-10T04:54:45.225 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "96e1952a-1042-4316-9fc9-86813e88b912", 2021-02-10T04:54:45.225 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:54:45.225 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:54:45.225 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:54:45.225 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:54:45.226 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:54:45.226 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:54:45.226 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 18, 2021-02-10T04:54:45.226 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1167, 2021-02-10T04:54:45.226 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:54:45.226 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:54:45.226 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:54:45.226 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.226 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.226 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.226 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6809", 2021-02-10T04:54:45.226 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.226 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.226 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.227 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.227 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:54:45.227 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.227 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.227 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.227 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6810", 2021-02-10T04:54:45.227 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.227 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.227 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.227 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.227 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:54:45.227 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.227 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.228 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.228 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6812", 2021-02-10T04:54:45.228 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.228 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.228 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.228 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.228 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:54:45.228 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.228 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.228 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.228 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6811", 2021-02-10T04:54:45.228 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.228 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.228 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.229 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.229 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6809/7", 2021-02-10T04:54:45.229 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6810/7", 2021-02-10T04:54:45.229 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6812/7", 2021-02-10T04:54:45.229 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6811/7", 2021-02-10T04:54:45.229 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:54:45.229 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:54:45.229 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:54:45.229 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.229 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.229 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.229 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 3, 2021-02-10T04:54:45.229 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "e019f49f-5dd4-4104-9ced-94c710616642", 2021-02-10T04:54:45.230 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:54:45.230 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:54:45.230 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:54:45.230 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:54:45.230 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 25, 2021-02-10T04:54:45.230 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 1094, 2021-02-10T04:54:45.230 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 1111, 2021-02-10T04:54:45.230 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1212, 2021-02-10T04:54:45.230 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 1095, 2021-02-10T04:54:45.230 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:54:45.230 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:54:45.230 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.230 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.231 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.231 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6813", 2021-02-10T04:54:45.231 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.231 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.231 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.231 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.231 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:54:45.231 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.231 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.231 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.231 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6814", 2021-02-10T04:54:45.231 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.231 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.232 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.232 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.232 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:54:45.232 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.232 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.232 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.232 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6816", 2021-02-10T04:54:45.232 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.232 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.232 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.232 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.232 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:54:45.232 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.232 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.233 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.233 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6815", 2021-02-10T04:54:45.233 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.233 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.233 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.233 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.233 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6813/7", 2021-02-10T04:54:45.233 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6814/7", 2021-02-10T04:54:45.233 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6816/7", 2021-02-10T04:54:45.233 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6815/7", 2021-02-10T04:54:45.233 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:54:45.233 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:54:45.233 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:54:45.234 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.234 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.234 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.234 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 4, 2021-02-10T04:54:45.234 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "91c8a80a-37b6-4c30-a32a-647b2e6d801f", 2021-02-10T04:54:45.234 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:54:45.234 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:54:45.234 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:54:45.234 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:54:45.234 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:54:45.234 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:54:45.234 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 31, 2021-02-10T04:54:45.234 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1212, 2021-02-10T04:54:45.235 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:54:45.235 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:54:45.235 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:54:45.235 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.235 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.235 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.235 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6800", 2021-02-10T04:54:45.235 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.235 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.235 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.235 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.235 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:54:45.236 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.236 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.236 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.236 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6801", 2021-02-10T04:54:45.236 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.236 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.236 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.236 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.236 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:54:45.236 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.236 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.236 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.236 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6803", 2021-02-10T04:54:45.237 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.237 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.237 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.237 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.237 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:54:45.237 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.237 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.237 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.237 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6802", 2021-02-10T04:54:45.237 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.237 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.237 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.237 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.238 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6800/7", 2021-02-10T04:54:45.238 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6801/7", 2021-02-10T04:54:45.238 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6803/7", 2021-02-10T04:54:45.238 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6802/7", 2021-02-10T04:54:45.238 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:54:45.238 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:54:45.238 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:54:45.238 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.238 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.238 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.238 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 5, 2021-02-10T04:54:45.238 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e", 2021-02-10T04:54:45.238 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:54:45.239 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:54:45.239 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:54:45.239 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:54:45.239 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:54:45.239 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:54:45.239 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 37, 2021-02-10T04:54:45.239 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1211, 2021-02-10T04:54:45.239 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:54:45.239 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:54:45.239 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:54:45.239 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.239 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.239 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.240 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6804", 2021-02-10T04:54:45.240 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.240 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.240 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.240 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.240 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:54:45.240 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.240 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.240 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.240 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6805", 2021-02-10T04:54:45.240 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.240 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.240 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.240 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.241 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:54:45.241 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.241 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.241 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.241 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6807", 2021-02-10T04:54:45.241 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.241 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.241 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.241 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.241 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:54:45.241 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.241 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.241 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.242 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6806", 2021-02-10T04:54:45.242 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.242 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.242 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.242 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.242 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6804/7", 2021-02-10T04:54:45.242 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6805/7", 2021-02-10T04:54:45.242 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6807/7", 2021-02-10T04:54:45.242 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6806/7", 2021-02-10T04:54:45.242 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:54:45.242 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:54:45.242 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:54:45.242 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.243 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.243 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.243 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 6, 2021-02-10T04:54:45.243 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "adc40430-3eeb-4003-a824-343021243077", 2021-02-10T04:54:45.243 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:54:45.243 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:54:45.243 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:54:45.243 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:54:45.243 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:54:45.243 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:54:45.243 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 43, 2021-02-10T04:54:45.243 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1211, 2021-02-10T04:54:45.243 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:54:45.243 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:54:45.244 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:54:45.244 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.244 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.244 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.244 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6808", 2021-02-10T04:54:45.244 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.244 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.244 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.244 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.244 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:54:45.244 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.244 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.244 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.245 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6809", 2021-02-10T04:54:45.245 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.245 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.245 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.245 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.245 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:54:45.245 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.245 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.245 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.245 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6811", 2021-02-10T04:54:45.245 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.245 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.245 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.246 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.246 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:54:45.246 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.246 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.246 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.246 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6810", 2021-02-10T04:54:45.246 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.246 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.246 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.246 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.246 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6808/7", 2021-02-10T04:54:45.246 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6809/7", 2021-02-10T04:54:45.247 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6811/7", 2021-02-10T04:54:45.247 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6810/7", 2021-02-10T04:54:45.247 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:54:45.247 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:54:45.247 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:54:45.247 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.247 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.247 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.247 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 7, 2021-02-10T04:54:45.247 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37", 2021-02-10T04:54:45.247 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:54:45.247 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:54:45.247 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:54:45.247 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 0.846160888671875, 2021-02-10T04:54:45.248 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:54:45.248 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:54:45.248 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 50, 2021-02-10T04:54:45.248 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1190, 2021-02-10T04:54:45.248 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:54:45.248 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:54:45.248 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:54:45.248 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.248 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.248 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.248 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6812", 2021-02-10T04:54:45.248 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.248 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.249 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.249 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.249 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:54:45.249 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.249 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.249 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.249 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6813", 2021-02-10T04:54:45.249 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.249 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.249 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.249 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.249 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:54:45.249 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.250 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.250 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.250 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6815", 2021-02-10T04:54:45.250 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.250 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.250 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.250 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.250 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:54:45.250 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:54:45.250 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.250 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:54:45.250 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6814", 2021-02-10T04:54:45.251 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:54:45.251 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.251 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.251 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.251 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6812/7", 2021-02-10T04:54:45.251 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6813/7", 2021-02-10T04:54:45.251 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6815/7", 2021-02-10T04:54:45.251 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6814/7", 2021-02-10T04:54:45.251 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:54:45.251 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:54:45.251 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:54:45.251 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.251 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.251 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:54:45.252 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_xinfo": [ 2021-02-10T04:54:45.252 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.252 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 0, 2021-02-10T04:54:45.252 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:54:45.252 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:54:45.252 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:54:45.252 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:54:45.252 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:54:45.252 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:51:38.084702+0000", 2021-02-10T04:54:45.252 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 1147 2021-02-10T04:54:45.252 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.252 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.253 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 1, 2021-02-10T04:54:45.253 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:54:45.253 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:54:45.253 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:54:45.253 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:54:45.253 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:54:45.253 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:53:58.012834+0000", 2021-02-10T04:54:45.253 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:54:45.253 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.253 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.253 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 2, 2021-02-10T04:54:45.253 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:54:45.253 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:54:45.254 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:54:45.254 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:54:45.254 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:54:45.254 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:54:17.518884+0000", 2021-02-10T04:54:45.254 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:54:45.254 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.254 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.254 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 3, 2021-02-10T04:54:45.254 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:54:45.254 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:54:45.254 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:54:45.254 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:54:45.254 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:54:45.255 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:53:58.855054+0000", 2021-02-10T04:54:45.255 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 1095 2021-02-10T04:54:45.255 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.255 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.255 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 4, 2021-02-10T04:54:45.255 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:54:45.255 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:54:45.255 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:54:45.255 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:54:45.255 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:54:45.255 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:53:54.197037+0000", 2021-02-10T04:54:45.255 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:54:45.255 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.256 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.256 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 5, 2021-02-10T04:54:45.256 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:54:45.256 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:54:45.256 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:54:45.256 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:54:45.256 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:54:45.256 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:54:10.901686+0000", 2021-02-10T04:54:45.256 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:54:45.256 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.256 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.256 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 6, 2021-02-10T04:54:45.256 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:54:45.257 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:54:45.257 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:54:45.257 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:54:45.257 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:54:45.257 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:54:28.759862+0000", 2021-02-10T04:54:45.257 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:54:45.257 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.257 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.257 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 7, 2021-02-10T04:54:45.257 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:54:45.257 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:54:45.257 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:54:45.257 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:54:45.258 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:54:45.258 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:53:47.766217+0000", 2021-02-10T04:54:45.258 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:54:45.258 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.258 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:54:45.258 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_upmap": [], 2021-02-10T04:54:45.258 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_upmap_items": [ 2021-02-10T04:54:45.258 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.258 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "2.7", 2021-02-10T04:54:45.258 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:54:45.258 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.258 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:54:45.258 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 4 2021-02-10T04:54:45.259 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.259 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.259 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.259 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.259 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "2.21", 2021-02-10T04:54:45.259 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:54:45.259 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.259 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:54:45.259 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 1 2021-02-10T04:54:45.259 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.259 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.259 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.259 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.260 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.5", 2021-02-10T04:54:45.260 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:54:45.260 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.260 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 4, 2021-02-10T04:54:45.260 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 2 2021-02-10T04:54:45.260 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.260 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.260 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.260 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.260 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.9", 2021-02-10T04:54:45.260 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:54:45.260 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.260 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 1, 2021-02-10T04:54:45.261 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 6 2021-02-10T04:54:45.261 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.261 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.261 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 3, 2021-02-10T04:54:45.261 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 7 2021-02-10T04:54:45.261 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.261 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.261 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.261 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.261 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.10", 2021-02-10T04:54:45.261 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:54:45.261 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.262 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 1, 2021-02-10T04:54:45.262 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 5 2021-02-10T04:54:45.262 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.262 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.262 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.262 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.262 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.6", 2021-02-10T04:54:45.262 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:54:45.262 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.262 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 5, 2021-02-10T04:54:45.262 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 6 2021-02-10T04:54:45.262 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.262 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.262 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.263 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.263 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.8", 2021-02-10T04:54:45.263 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:54:45.263 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.263 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 1, 2021-02-10T04:54:45.263 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 3 2021-02-10T04:54:45.263 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.263 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.263 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.263 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.263 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.10", 2021-02-10T04:54:45.263 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:54:45.263 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.264 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:54:45.264 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 3 2021-02-10T04:54:45.264 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.264 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.264 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.264 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.264 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.18", 2021-02-10T04:54:45.264 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:54:45.264 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.264 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 4, 2021-02-10T04:54:45.264 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 3 2021-02-10T04:54:45.264 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.264 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.265 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.265 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.265 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.1b", 2021-02-10T04:54:45.265 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:54:45.265 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.265 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 0, 2021-02-10T04:54:45.265 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 3 2021-02-10T04:54:45.265 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.265 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.265 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.265 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.265 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "7.6", 2021-02-10T04:54:45.265 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:54:45.266 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.266 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 3, 2021-02-10T04:54:45.266 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 1 2021-02-10T04:54:45.266 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.266 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.266 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.266 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.266 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "7.15", 2021-02-10T04:54:45.266 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:54:45.266 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.266 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:54:45.266 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 4 2021-02-10T04:54:45.266 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.266 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.267 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.267 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:54:45.267 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_temp": [ 2021-02-10T04:54:45.267 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.267 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "3.3", 2021-02-10T04:54:45.267 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T04:54:45.267 INFO:teuthology.orchestra.run.smithi085.stdout: 4, 2021-02-10T04:54:45.267 INFO:teuthology.orchestra.run.smithi085.stdout: 6, 2021-02-10T04:54:45.267 INFO:teuthology.orchestra.run.smithi085.stdout: 3 2021-02-10T04:54:45.267 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.267 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.267 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.268 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "3.6", 2021-02-10T04:54:45.268 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T04:54:45.268 INFO:teuthology.orchestra.run.smithi085.stdout: 1, 2021-02-10T04:54:45.268 INFO:teuthology.orchestra.run.smithi085.stdout: 4, 2021-02-10T04:54:45.268 INFO:teuthology.orchestra.run.smithi085.stdout: 3 2021-02-10T04:54:45.268 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.268 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.268 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.268 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "3.7", 2021-02-10T04:54:45.268 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T04:54:45.269 INFO:teuthology.orchestra.run.smithi085.stdout: 3, 2021-02-10T04:54:45.269 INFO:teuthology.orchestra.run.smithi085.stdout: 7, 2021-02-10T04:54:45.269 INFO:teuthology.orchestra.run.smithi085.stdout: 4 2021-02-10T04:54:45.269 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.269 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.269 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.269 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.b", 2021-02-10T04:54:45.269 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T04:54:45.269 INFO:teuthology.orchestra.run.smithi085.stdout: 1, 2021-02-10T04:54:45.269 INFO:teuthology.orchestra.run.smithi085.stdout: 4, 2021-02-10T04:54:45.269 INFO:teuthology.orchestra.run.smithi085.stdout: 5 2021-02-10T04:54:45.269 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.269 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.270 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.270 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.16", 2021-02-10T04:54:45.270 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T04:54:45.270 INFO:teuthology.orchestra.run.smithi085.stdout: 2, 2021-02-10T04:54:45.270 INFO:teuthology.orchestra.run.smithi085.stdout: 1, 2021-02-10T04:54:45.270 INFO:teuthology.orchestra.run.smithi085.stdout: 3 2021-02-10T04:54:45.270 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.270 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.270 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.270 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.17", 2021-02-10T04:54:45.270 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T04:54:45.270 INFO:teuthology.orchestra.run.smithi085.stdout: 1, 2021-02-10T04:54:45.270 INFO:teuthology.orchestra.run.smithi085.stdout: 5, 2021-02-10T04:54:45.271 INFO:teuthology.orchestra.run.smithi085.stdout: 3 2021-02-10T04:54:45.271 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.271 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.271 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.271 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.1b", 2021-02-10T04:54:45.271 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T04:54:45.271 INFO:teuthology.orchestra.run.smithi085.stdout: 1, 2021-02-10T04:54:45.271 INFO:teuthology.orchestra.run.smithi085.stdout: 4, 2021-02-10T04:54:45.271 INFO:teuthology.orchestra.run.smithi085.stdout: 3 2021-02-10T04:54:45.271 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.271 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.271 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:54:45.271 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "7.1b", 2021-02-10T04:54:45.272 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T04:54:45.272 INFO:teuthology.orchestra.run.smithi085.stdout: 2, 2021-02-10T04:54:45.272 INFO:teuthology.orchestra.run.smithi085.stdout: 3, 2021-02-10T04:54:45.272 INFO:teuthology.orchestra.run.smithi085.stdout: 1 2021-02-10T04:54:45.272 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:54:45.272 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.272 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:54:45.272 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_temp": [], 2021-02-10T04:54:45.272 INFO:teuthology.orchestra.run.smithi085.stdout: "blocklist": { 2021-02-10T04:54:45.272 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/801393150": "2021-02-11T04:21:37.851607+0000", 2021-02-10T04:54:45.272 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/3049671696": "2021-02-11T04:21:37.851607+0000", 2021-02-10T04:54:45.272 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/4056505602": "2021-02-11T04:21:20.700478+0000", 2021-02-10T04:54:45.272 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:6800/7": "2021-02-11T04:21:20.700478+0000", 2021-02-10T04:54:45.273 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/1879026062": "2021-02-11T04:21:20.700478+0000" 2021-02-10T04:54:45.273 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.273 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profiles": { 2021-02-10T04:54:45.273 INFO:teuthology.orchestra.run.smithi085.stdout: "default": { 2021-02-10T04:54:45.273 INFO:teuthology.orchestra.run.smithi085.stdout: "crush-failure-domain": "osd", 2021-02-10T04:54:45.273 INFO:teuthology.orchestra.run.smithi085.stdout: "k": "2", 2021-02-10T04:54:45.273 INFO:teuthology.orchestra.run.smithi085.stdout: "m": "1", 2021-02-10T04:54:45.273 INFO:teuthology.orchestra.run.smithi085.stdout: "plugin": "jerasure", 2021-02-10T04:54:45.273 INFO:teuthology.orchestra.run.smithi085.stdout: "ruleset-failure-domain": "osd", 2021-02-10T04:54:45.273 INFO:teuthology.orchestra.run.smithi085.stdout: "technique": "reed_sol_van" 2021-02-10T04:54:45.273 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.273 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:54:45.273 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps_queue": [], 2021-02-10T04:54:45.273 INFO:teuthology.orchestra.run.smithi085.stdout: "new_removed_snaps": [], 2021-02-10T04:54:45.274 INFO:teuthology.orchestra.run.smithi085.stdout: "new_purged_snaps": [], 2021-02-10T04:54:45.274 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_node_flags": {}, 2021-02-10T04:54:45.274 INFO:teuthology.orchestra.run.smithi085.stdout: "device_class_flags": {}, 2021-02-10T04:54:45.274 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_mode": { 2021-02-10T04:54:45.274 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_mode_enabled": false, 2021-02-10T04:54:45.274 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_bucket_count": 0, 2021-02-10T04:54:45.274 INFO:teuthology.orchestra.run.smithi085.stdout: "degraded_stretch_mode": 0, 2021-02-10T04:54:45.274 INFO:teuthology.orchestra.run.smithi085.stdout: "recovering_stretch_mode": 0, 2021-02-10T04:54:45.274 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_mode_bucket": 0 2021-02-10T04:54:45.274 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:54:45.274 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:54:45.354 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:54:45.473 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:54:45.672 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:54:45.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:45 smithi164 conmon[29288]: cluster 2021-02-10T04:54:45.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:45 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:54:45.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:45 smithi164 conmon[29288]: 54:44.480705+0000 mgr.y (mgr.14140) 1338 : cluster [DBG] pgmap v2198: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 829 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 78 B/s, 0 objects/s recovering 2021-02-10T04:54:45.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:45 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:54:45.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:45 smithi164 conmon[29288]: 02-10T04:54:45.170519+0000 mon.a 2021-02-10T04:54:45.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:45 smithi164 conmon[29288]: (mon.0) 2595 : audit [DBG] 2021-02-10T04:54:45.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:45 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/4053686761' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-02-10T04:54:45.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:45 smithi085 conmon[29319]: cluster 2021-02-10T04:54:44.480705+0000 mgr.y (mgr.14140) 1338 : cluster [DBG] pgmap v2198: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 829 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 78 B/s, 0 objects/s recovering 2021-02-10T04:54:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:45 smithi085 conmon[29319]: audit 2021-02-10T04:54:45.170519+0000 mon.a (mon.0) 2595 : audit [DBG] from='client.? 172.21.15.85:0/4053686761' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-02-10T04:54:45.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:45 smithi085 conmon[35961]: cluster 2021-02-10T04:54:44.480705+0000 mgr.y (mgr 2021-02-10T04:54:45.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:45 smithi085 conmon[35961]: .14140) 1338 : cluster [DBG] pgmap v2198: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 829 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 78 B/s, 0 objects/s recovering 2021-02-10T04:54:45.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:45 smithi085 conmon[35961]: audit 2021-02-10T04:54:45.170519+0000 mon.a (mon.0) 2595 : 2021-02-10T04:54:45.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:45 smithi085 conmon[35961]: audit [DBG] from='client.? 172.21.15.85:0/4053686761' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-02-10T04:54:46.176 INFO:tasks.thrashosds.thrasher:j is {'epoch': 1215, 'fsid': '42224fce-6b57-11eb-8feb-001a4aab830c', 'created': '2021-02-10T04:20:57.933279+0000', 'modified': '2021-02-10T04:54:43.538434+0000', 'last_up_change': '2021-02-10T04:52:31.054977+0000', 'last_in_change': '2021-02-10T04:51:35.701776+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-02-10T04:24:05.258028+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '110', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2021-02-10T04:25:32.688554+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 42, 'pg_placement_num': 42, 'pg_placement_num_target': 42, 'pg_num_target': 42, 'pg_num_pending': 42, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '89', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2021-02-10T04:25:47.658613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 13, 'pg_placement_num': 13, 'pg_placement_num_target': 13, 'pg_num_target': 13, 'pg_num_pending': 13, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1211', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '1209', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'zone.rgw.log', 'create_time': '2021-02-10T04:25:47.968727+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 28, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 28, 'last_pg_merge_meta': {'source_pgid': '4.1c', 'ready_epoch': 1184, 'last_epoch_started': 1183, 'last_epoch_clean': 1183, 'source_version': "1049'39", 'target_version': "1049'33"}, 'last_change': '1184', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '1184', 'last_force_op_resend_preluminous': '1182', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'zone.rgw.control', 'create_time': '2021-02-10T04:25:49.278260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'zone.rgw.meta', 'create_time': '2021-02-10T04:25:51.792514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 42, 'pg_placement_num': 42, 'pg_placement_num_target': 42, 'pg_num_target': 42, 'pg_num_pending': 42, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1046', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '1044', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 8}, 'application_metadata': {'rgw': {}}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2021-02-10T04:28:23.932391+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 36, 'pg_placement_num': 36, 'pg_placement_num_target': 36, 'pg_num_target': 36, 'pg_num_pending': 36, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1143', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '1141', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 888, 'snap_epoch': 994, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '45d0bb5f-35fa-4596-964f-75abcee53bee', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 1146, 'down_at': 1147, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6801', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6802', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6804', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6803', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6801/7', 'cluster_addr': '172.21.15.85:6802/7', 'heartbeat_back_addr': '172.21.15.85:6804/7', 'heartbeat_front_addr': '172.21.15.85:6803/7', 'state': ['exists']}, {'osd': 1, 'uuid': '3421a5ce-872e-4b52-adf7-6aaa595871a4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.86566162109375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 1212, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6805', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6806', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6808', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6807', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6805/7', 'cluster_addr': '172.21.15.85:6806/7', 'heartbeat_back_addr': '172.21.15.85:6808/7', 'heartbeat_front_addr': '172.21.15.85:6807/7', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '96e1952a-1042-4316-9fc9-86813e88b912', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 1167, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6809', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6810', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6812', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6811', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6809/7', 'cluster_addr': '172.21.15.85:6810/7', 'heartbeat_back_addr': '172.21.15.85:6812/7', 'heartbeat_front_addr': '172.21.15.85:6811/7', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'e019f49f-5dd4-4104-9ced-94c710616642', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 25, 'last_clean_end': 1094, 'up_from': 1111, 'up_thru': 1212, 'down_at': 1095, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6813', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6814', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6816', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6815', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6813/7', 'cluster_addr': '172.21.15.85:6814/7', 'heartbeat_back_addr': '172.21.15.85:6816/7', 'heartbeat_front_addr': '172.21.15.85:6815/7', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '91c8a80a-37b6-4c30-a32a-647b2e6d801f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 1212, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6800', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6801', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6803', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6802', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6800/7', 'cluster_addr': '172.21.15.164:6801/7', 'heartbeat_back_addr': '172.21.15.164:6803/7', 'heartbeat_front_addr': '172.21.15.164:6802/7', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 1211, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6804', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6805', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6807', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6806', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6804/7', 'cluster_addr': '172.21.15.164:6805/7', 'heartbeat_back_addr': '172.21.15.164:6807/7', 'heartbeat_front_addr': '172.21.15.164:6806/7', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'adc40430-3eeb-4003-a824-343021243077', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 43, 'up_thru': 1211, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6808', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6809', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6811', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6810', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6808/7', 'cluster_addr': '172.21.15.164:6809/7', 'heartbeat_back_addr': '172.21.15.164:6811/7', 'heartbeat_front_addr': '172.21.15.164:6810/7', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.846160888671875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 1190, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6812', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6813', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6815', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6814', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6812/7', 'cluster_addr': '172.21.15.164:6813/7', 'heartbeat_back_addr': '172.21.15.164:6815/7', 'heartbeat_front_addr': '172.21.15.164:6814/7', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:51:38.084702+0000', 'dead_epoch': 1147}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:53:58.012834+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:54:17.518884+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:53:58.855054+0000', 'dead_epoch': 1095}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:53:54.197037+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:54:10.901686+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:54:28.759862+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:53:47.766217+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.7', 'mappings': [{'from': 7, 'to': 4}]}, {'pgid': '2.21', 'mappings': [{'from': 7, 'to': 1}]}, {'pgid': '4.5', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 6}, {'from': 3, 'to': 7}]}, {'pgid': '4.10', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '5.6', 'mappings': [{'from': 5, 'to': 6}]}, {'pgid': '5.8', 'mappings': [{'from': 1, 'to': 3}]}, {'pgid': '5.10', 'mappings': [{'from': 7, 'to': 3}]}, {'pgid': '5.18', 'mappings': [{'from': 4, 'to': 3}]}, {'pgid': '5.1b', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '7.6', 'mappings': [{'from': 3, 'to': 1}]}, {'pgid': '7.15', 'mappings': [{'from': 7, 'to': 4}]}], 'pg_temp': [{'pgid': '3.3', 'osds': [4, 6, 3]}, {'pgid': '3.6', 'osds': [1, 4, 3]}, {'pgid': '3.7', 'osds': [3, 7, 4]}, {'pgid': '4.b', 'osds': [1, 4, 5]}, {'pgid': '4.16', 'osds': [2, 1, 3]}, {'pgid': '4.17', 'osds': [1, 5, 3]}, {'pgid': '4.1b', 'osds': [1, 4, 3]}, {'pgid': '7.1b', 'osds': [2, 3, 1]}], 'primary_temp': [], 'blocklist': {'172.21.15.85:0/801393150': '2021-02-11T04:21:37.851607+0000', '172.21.15.85:0/3049671696': '2021-02-11T04:21:37.851607+0000', '172.21.15.85:0/4056505602': '2021-02-11T04:21:20.700478+0000', '172.21.15.85:6800/7': '2021-02-11T04:21:20.700478+0000', '172.21.15.85:0/1879026062': '2021-02-11T04:21:20.700478+0000'}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2021-02-10T04:54:46.176 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:54:46.418 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:54:47.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:47 smithi085 conmon[29319]: cluster 2021-02-10T04:54:46.481552 2021-02-10T04:54:49.711 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:47 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 1339 : cluster [DBG] pgmap v2199: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 64 B/s, 0 objects/s recovering 2021-02-10T04:54:49.711 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:47 smithi085 conmon[35961]: cluster 2021-02-10T04:54:46.481552 2021-02-10T04:54:49.712 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:47 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1339 : cluster [DBG] pgmap v2199: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 64 B/s, 0 objects/s recovering 2021-02-10T04:54:49.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:47 smithi164 conmon[29288]: cluster 2021-02-10T04:54:46.481552+0000 2021-02-10T04:54:49.713 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:47 smithi164 conmon[29288]: mgr.y (mgr.14140) 1339 : cluster [DBG] pgmap v2199: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 64 B/s, 0 objects/s recovering 2021-02-10T04:54:49.713 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:54:49 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:54:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:49 smithi164 conmon[29288]: cluster 2021-02-10T04:54:48.453939+0000 2021-02-10T04:54:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:49 smithi164 conmon[29288]: osd.7 (osd.7) 90 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:54:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:49 smithi164 conmon[29288]: cluster 2021-02-10T04:54:48.456115+0000 osd.7 2021-02-10T04:54:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:49 smithi164 conmon[29288]: (osd.7) 91 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:54:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:49 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:54:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:49 smithi164 conmon[29288]: 04:54:48.482418+0000 mgr.y (mgr.14140) 1340 : cluster [DBG] pgmap v2200: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 49 B/s, 0 objects/s recovering 2021-02-10T04:54:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:49 smithi164 conmon[29288]: 2021-02-10T04:54:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:49 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:54:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:49 smithi164 conmon[29288]: T04:54:48.650488+0000 mon.a (mon.0) 2021-02-10T04:54:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:49 smithi164 conmon[29288]: 2596 : cluster 2021-02-10T04:54:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:49 smithi164 conmon[29288]: [WRN] Health check update: Degraded data redundancy: 214/2196 objects degraded (9.745%), 24 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2021-02-10T04:54:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:49 smithi164 conmon[29288]: cluster 2021-02-10T04:54:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:49 smithi164 conmon[29288]: 2021-02-10T04:54:48 2021-02-10T04:54:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:49 smithi164 conmon[29288]: .662667+0000 mon.a (mon.0) 2597 : cluster 2021-02-10T04:54:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:49 smithi164 conmon[29288]: [DBG] osdmap e1216: 8 total, 7 up, 8 in 2021-02-10T04:54:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:49 smithi164 conmon[29288]: audit 2021 2021-02-10T04:54:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:49 smithi164 conmon[29288]: -02-10T04:54:49.274996+0000 mgr.y (mgr.14140) 1341 2021-02-10T04:54:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:49 smithi164 conmon[29288]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:54:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:49 smithi085 conmon[29319]: cluster 2021-02-10T04:54:48.453939+0000 osd.7 (osd.7) 90 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:54:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:49 smithi085 conmon[29319]: cluster 2021-02-10T04:54:48.456115+0000 osd.7 (osd.7) 91 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:54:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:49 smithi085 conmon[29319]: cluster 2021-02-10T04:54:48.482418+0000 mgr.y (mgr.14140) 1340 : cluster [DBG] pgmap v2200: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 49 B/s, 0 objects/s recovering 2021-02-10T04:54:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:49 smithi085 conmon[29319]: cluster 2021-02-10T04:54:48.650488+0000 mon.a (mon.0) 2596 : cluster [WRN] Health check update: Degraded data redundancy: 214/2196 objects degraded (9.745%), 24 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2021-02-10T04:54:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:49 smithi085 conmon[29319]: cluster 2021-02-10T04:54:48.662667+0000 mon.a (mon.0) 2597 : cluster [DBG] osdmap e1216: 8 total, 7 up, 8 in 2021-02-10T04:54:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:49 smithi085 conmon[29319]: audit 2021-02-10T04:54:49.274996+0000 mgr.y (mgr.14140) 1341 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:54:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:49 smithi085 conmon[35961]: cluster 2021-02-10T04:54:48.453939+0000 2021-02-10T04:54:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:49 smithi085 conmon[35961]: osd.7 (osd.7) 90 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:54:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:49 smithi085 conmon[35961]: cluster 2021-02-10T04:54:48.456115+0000 osd.7 (osd.7) 91 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:54:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:49 smithi085 conmon[35961]: cluster 2021-02-10T04:54:48.482418+0000 mgr.y (mgr.14140) 1340 : cluster [DBG] pgmap v2200: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 49 B/s, 0 objects/s recovering 2021-02-10T04:54:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:49 smithi085 conmon[35961]: cluster 2021-02-10T04:54:48.650488+0000 mon.a (mon.0) 2596 : cluster [WRN] Health check update: Degraded data redundancy: 214/2196 objects degraded (9.745%), 24 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2021-02-10T04:54:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:49 smithi085 conmon[35961]: cluster 2021-02-10T04:54:48.662667+0000 mon.a (mon.0) 2597 : cluster [DBG] osdmap e1216: 8 total, 7 up, 8 in 2021-02-10T04:54:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:49 smithi085 conmon[35961]: audit 2021-02-10T04:54:49.274996+0000 mgr.y (mgr.14140) 1341 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:54:49.976 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:54:49.996 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:54:50.018 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:54:50.043 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:54:50.066 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:54:50.090 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:54:50.109 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:54:50.109 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-02-10T04:54:50.109 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: problem getting command descriptions from osd.0 2021-02-10T04:54:50.109 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:54:50.109 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:54:50.109 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:54:50.109 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:54:50.110 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:54:50.110 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:54:50.110 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:54:50.562 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is unset 2021-02-10T04:54:50.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:50 smithi164 conmon[29288]: audit 2021-02-10T04:54:50.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:50 smithi164 conmon[29288]: 2021-02-10T04:54: 2021-02-10T04:54:50.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:50 smithi164 conmon[29288]: 49.951952+0000 mon.a (mon.0) 2598 : audit [DBG] from='client.? 172.21.15.85:0/74986760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:54:50.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:50 smithi164 conmon[29288]: audit 2021-02-10T04:54:50.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:50 smithi164 conmon[29288]: 2021-02-10T04:54:50.165331+0000 2021-02-10T04:54:50.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:50 smithi164 conmon[29288]: mon.c (mon.1) 118 : audit [INF] 2021-02-10T04:54:50.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:50 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/4277766356' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:54:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:50 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:54:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:50 smithi164 conmon[29288]: T04:54:50.165678+0000 mon.a (mon. 2021-02-10T04:54:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:50 smithi164 conmon[29288]: 0) 2599 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:54:50.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:50 smithi085 conmon[29319]: audit 2021-02-10T04:54:49.951952+0000 mon.a ( 2021-02-10T04:54:50.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:50 smithi085 conmon[29319]: mon.0) 2598 : audit [DBG] from='client.? 172.21.15.85:0/74986760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:54:50.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:50 smithi085 conmon[29319]: audit 2021-02-10T04:54:50.165331+0000 mon.c (mon.1) 118 : audit [INF] from='client.? 172.21.15.85:0/4277766356' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:54:50.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:50 smithi085 conmon[29319]: audit 2021-02-10T04:54:50.165678+0000 mon.a (mon.0) 2599 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:54:50.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:50 smithi085 conmon[35961]: audit 2021-02-10T04:54:50.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:50 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:54:50.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:50 smithi085 conmon[35961]: :54:49.951952+0000 mon.a (mon.0) 2598 : audit [DBG] from='client.? 172.21.15.85:0/74986760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:54:50.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:50 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:54:50.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:50 smithi085 conmon[35961]: -10T04:54:50.165331+0000 mon.c (mon. 2021-02-10T04:54:50.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:50 smithi085 conmon[35961]: 1) 118 : audit [INF] from='client.? 172.21.15.85:0/4277766356' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:54:50.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:50 smithi085 conmon[35961]: audit 2021 2021-02-10T04:54:50.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:50 smithi085 conmon[35961]: -02-10T04:54:50.165678+0000 2021-02-10T04:54:50.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:50 smithi085 conmon[35961]: mon.a (mon.0) 2599 : audit [INF] 2021-02-10T04:54:50.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:50 smithi085 conmon[35961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:54:51.156 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:54:51.156 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2021-02-10T04:54:51.156 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 0 2021-02-10T04:54:51.156 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:54:51.299 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:54:51.303 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:54:51.554 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:54:51 smithi164 conmon[37263]: debug 2021-02-10T04:54:51.247+0000 7f1c679e3700 -1 osd.5 pg_epoch: 1217 pg[1.a( empty local-lis/les=1118/1119 n=0 ec=110/19 lis/c=1118/1118 les/c/f=1119/1119/0 sis=1118) [5,2,6] r=0 lpr=1118 crt=0'0 mlcod 0'0 active+clean] scrubber pg(1.a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 2 (0x5566587bab00) 2021-02-10T04:54:51.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:51 smithi164 conmon[29288]: cluster 2021-02-10T04:54:50.483147+0000 mgr.y (mgr.14140) 1342 : cluster [DBG] pgmap v2202: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 48 B/s, 0 objects/s recovering 2021-02-10T04:54:51.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:51 smithi164 conmon[29288]: cluster 2021-02-10T04:54:50.548814+0000 mon.a (mon.0) 2600 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:54:51.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:51 smithi164 conmon[29288]: audit 2021-02-10T04: 2021-02-10T04:54:51.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:51 smithi164 conmon[29288]: 54:50.556113+0000 mon.a (mon. 2021-02-10T04:54:51.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:51 smithi164 conmon[29288]: 0) 2601 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:54:51.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:51 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:54:51.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:51 smithi164 conmon[29288]: 10T04:54:50.556188+0000 mon.a (mon.0) 2602 : cluster [DBG] osdmap e1217: 8 total, 7 up, 8 in 2021-02-10T04:54:51.556 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:51 smithi164 conmon[29288]: audit 2021-02-10T04:54:51.279685 2021-02-10T04:54:51.557 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:51 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1343 : audit [DBG] from='client.15993 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:54:51.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[29319]: cluster 2021-02-10T04:54:51.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:54:51.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[29319]: 04:54:50.483147+0000 mgr.y (mgr.14140) 1342 : cluster [DBG] pgmap v2202: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 48 B/s, 0 objects/s recovering 2021-02-10T04:54:51.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:54:51.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[29319]: -10T04:54:50.548814+0000 mon.a (mon.0 2021-02-10T04:54:51.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[29319]: ) 2600 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:54:51.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:54:51.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[29319]: -10T04:54:50.556113+0000 mon.a ( 2021-02-10T04:54:51.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[29319]: mon.0) 2601 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:54:51.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:54:51.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[29319]: -10T04:54:50.556188+0000 mon.a (mon 2021-02-10T04:54:51.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[29319]: .0) 2602 : cluster [DBG] osdmap e1217: 8 total, 7 up, 8 in 2021-02-10T04:54:51.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:54:51.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[29319]: T04:54:51.279685+0000 mgr.y (mgr. 2021-02-10T04:54:51.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[29319]: 14140) 1343 : audit [DBG] from='client.15993 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:54:51.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[35961]: cluster 2021-02-10T04:54:51.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:54:51.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[35961]: 10T04:54:50.483147+0000 mgr.y (mgr.14140) 1342 : cluster [DBG] pgmap v2202: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%); 48 B/s, 0 objects/s recovering 2021-02-10T04:54:51.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[35961]: cluster 2021-02-10T04:54:51.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:54:51.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[35961]: 54:50.548814+0000 mon.a (mon. 2021-02-10T04:54:51.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[35961]: 0) 2600 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:54:51.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[35961]: audit 2021-02-10T04:54:51.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[35961]: 2021-02-10T04:54:50.556113+0000 2021-02-10T04:54:51.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[35961]: mon.a (mon.0) 2601 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:54:51.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[35961]: cluster 2021-02-10T04:54:51.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[35961]: 2021-02-10T04:54:50. 2021-02-10T04:54:51.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[35961]: 556188+0000 mon.a (mon.0) 2602 : cluster [DBG] osdmap e1217: 8 total, 7 up, 8 in 2021-02-10T04:54:51.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[35961]: 2021-02-10T04:54:51.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:54:51.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[35961]: -10T04:54:51.279685+0000 mgr.y (mgr.14140 2021-02-10T04:54:51.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:51 smithi085 conmon[35961]: ) 1343 : audit [DBG] from='client.15993 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:54:51.942 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:54:51.943 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:54:52.117 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":2203,"stamp":"2021-02-10T04:54:50.553074+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":931713416,"num_objects":732,"num_object_clones":323,"num_object_copies":2196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":214,"num_objects_misplaced":44,"num_objects_unfound":0,"num_objects_dirty":732,"num_whiteouts":10,"num_read":7394,"num_read_kb":4589639,"num_write":4593,"num_write_kb":1255989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":899,"num_bytes_recovered":307467451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35884,"ondisk_log_size":35884,"up":542,"acting":550,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":619,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2388812,"kb_used_data":1996044,"kb_used_omap":386,"kb_used_meta":392317,"kb_avail":747572404,"statfs":{"total":767960285184,"available":765514141696,"internally_reserved":0,"allocated":2043949056,"data_stored":2619710557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":396036,"internal_metadata":401732860},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.011558"},"pg_stats":[{"pgid":"6.29","version":"0'0","reported_seq":"1064","reported_epoch":"1215","state":"active+undersized","last_fresh":"2021-02-10T04:54:43.934423+0000","last_change":"2021-02-10T04:53:46.719266+0000","last_active":"2021-02-10T04:54:43.934423+0000","last_peered":"2021-02-10T04:54:43.934423+0000","last_clean":"2021-02-10T04:52:30.064708+0000","last_became_active":"2021-02-10T04:53:46.719266+0000","last_became_peered":"2021-02-10T04:53:46.719266+0000","last_unstale":"2021-02-10T04:54:43.934423+0000","last_undegraded":"2021-02-10T04:54:43.934423+0000","last_fullsized":"2021-02-10T04:53:46.701117+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:24.121684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:24.121684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":"111","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.641678+0000","last_change":"2021-02-10T04:52:30.133105+0000","last_active":"2021-02-10T04:54:41.641678+0000","last_peered":"2021-02-10T04:54:41.641678+0000","last_clean":"2021-02-10T04:54:41.641678+0000","last_became_active":"2021-02-10T04:51:36.864926+0000","last_became_peered":"2021-02-10T04:51:36.864926+0000","last_unstale":"2021-02-10T04:54:41.641678+0000","last_undegraded":"2021-02-10T04:54:41.641678+0000","last_fullsized":"2021-02-10T04:54:41.641678+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.133043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.133043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"1191'4","reported_seq":"180","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:48.029223+0000","last_change":"2021-02-10T04:53:46.736164+0000","last_active":"2021-02-10T04:54:48.029223+0000","last_peered":"2021-02-10T04:54:48.029223+0000","last_clean":"2021-02-10T04:54:48.029223+0000","last_became_active":"2021-02-10T04:53:46.735711+0000","last_became_peered":"2021-02-10T04:53:46.735711+0000","last_unstale":"2021-02-10T04:54:48.029223+0000","last_undegraded":"2021-02-10T04:54:48.029223+0000","last_fullsized":"2021-02-10T04:54:48.029223+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'3","last_scrub_stamp":"2021-02-10T04:51:01.185820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.185820+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"1132'66","reported_seq":"1302","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.644062+0000","last_change":"2021-02-10T04:51:55.430490+0000","last_active":"2021-02-10T04:54:41.644062+0000","last_peered":"2021-02-10T04:54:41.644062+0000","last_clean":"2021-02-10T04:54:41.644062+0000","last_became_active":"2021-02-10T04:25:49.002058+0000","last_became_peered":"2021-02-10T04:25:49.002058+0000","last_unstale":"2021-02-10T04:54:41.644062+0000","last_undegraded":"2021-02-10T04:54:41.644062+0000","last_fullsized":"2021-02-10T04:54:41.644062+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1132'66","last_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_deep_scrub":"1132'66","last_deep_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.430399+0000","log_size":66,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":95,"num_read_kb":78,"num_write":55,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"120","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.644165+0000","last_change":"2021-02-10T04:53:41.722609+0000","last_active":"2021-02-10T04:54:41.644165+0000","last_peered":"2021-02-10T04:54:41.644165+0000","last_clean":"2021-02-10T04:54:41.644165+0000","last_became_active":"2021-02-10T04:51:36.875000+0000","last_became_peered":"2021-02-10T04:51:36.875000+0000","last_unstale":"2021-02-10T04:54:41.644165+0000","last_undegraded":"2021-02-10T04:54:41.644165+0000","last_fullsized":"2021-02-10T04:54:41.644165+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:53:41.722516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:41.722516+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"1190","reported_epoch":"1214","state":"active+undersized","last_fresh":"2021-02-10T04:54:41.646751+0000","last_change":"2021-02-10T04:52:32.077502+0000","last_active":"2021-02-10T04:54:41.646751+0000","last_peered":"2021-02-10T04:54:41.646751+0000","last_clean":"2021-02-10T04:52:30.059220+0000","last_became_active":"2021-02-10T04:52:32.077502+0000","last_became_peered":"2021-02-10T04:52:32.077502+0000","last_unstale":"2021-02-10T04:54:41.646751+0000","last_undegraded":"2021-02-10T04:54:41.646751+0000","last_fullsized":"2021-02-10T04:52:32.069476+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:48.408748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:50:48.408748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"1003'1689","reported_seq":"4635","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.644719+0000","last_change":"2021-02-10T04:53:38.170873+0000","last_active":"2021-02-10T04:54:41.644719+0000","last_peered":"2021-02-10T04:54:41.644719+0000","last_clean":"2021-02-10T04:54:41.644719+0000","last_became_active":"2021-02-10T04:52:26.030105+0000","last_became_peered":"2021-02-10T04:52:26.030105+0000","last_unstale":"2021-02-10T04:54:41.644719+0000","last_undegraded":"2021-02-10T04:54:41.644719+0000","last_fullsized":"2021-02-10T04:54:41.644719+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:53:38.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.170802+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":99361981,"num_objects":37,"num_object_clones":33,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"62'5","reported_seq":"27","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:46.575124+0000","last_change":"2021-02-10T04:54:39.615784+0000","last_active":"2021-02-10T04:54:46.575124+0000","last_peered":"2021-02-10T04:54:46.575124+0000","last_clean":"2021-02-10T04:54:46.575124+0000","last_became_active":"2021-02-10T04:54:39.560009+0000","last_became_peered":"2021-02-10T04:54:39.560009+0000","last_unstale":"2021-02-10T04:54:46.575124+0000","last_undegraded":"2021-02-10T04:54:46.575124+0000","last_fullsized":"2021-02-10T04:54:46.575124+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":159,"num_read_kb":158,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"1049'32","reported_seq":"60","reported_epoch":"1214","state":"active+clean+remapped","last_fresh":"2021-02-10T04:54:41.644524+0000","last_change":"2021-02-10T04:53:42.716996+0000","last_active":"2021-02-10T04:54:41.644524+0000","last_peered":"2021-02-10T04:54:41.644524+0000","last_clean":"2021-02-10T04:54:41.644524+0000","last_became_active":"2021-02-10T04:53:42.693156+0000","last_became_peered":"2021-02-10T04:53:42.693156+0000","last_unstale":"2021-02-10T04:54:41.644524+0000","last_undegraded":"2021-02-10T04:54:41.644524+0000","last_fullsized":"2021-02-10T04:54:41.644524+0000","mapping_epoch":1187,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1188,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'32","last_scrub_stamp":"2021-02-10T04:51:51.382040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.382040+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":61,"num_read_kb":44,"num_write":33,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1788,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,5],"avail_no_missing":["1","4","5"],"object_location_counts":[{"shards":"1,4,5","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"867","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.645648+0000","last_change":"2021-02-10T04:53:34.179015+0000","last_active":"2021-02-10T04:54:41.645648+0000","last_peered":"2021-02-10T04:54:41.645648+0000","last_clean":"2021-02-10T04:54:41.645648+0000","last_became_active":"2021-02-10T04:51:49.502576+0000","last_became_peered":"2021-02-10T04:51:49.502576+0000","last_unstale":"2021-02-10T04:54:41.645648+0000","last_undegraded":"2021-02-10T04:54:41.645648+0000","last_fullsized":"2021-02-10T04:54:41.645648+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:34.178949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.178949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"111","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.642108+0000","last_change":"2021-02-10T04:51:53.038893+0000","last_active":"2021-02-10T04:54:41.642108+0000","last_peered":"2021-02-10T04:54:41.642108+0000","last_clean":"2021-02-10T04:54:41.642108+0000","last_became_active":"2021-02-10T04:51:36.855507+0000","last_became_peered":"2021-02-10T04:51:36.855507+0000","last_unstale":"2021-02-10T04:54:41.642108+0000","last_undegraded":"2021-02-10T04:54:41.642108+0000","last_fullsized":"2021-02-10T04:54:41.642108+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:53.038834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:53.038834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"1199","reported_epoch":"1214","state":"active+undersized","last_fresh":"2021-02-10T04:54:41.641625+0000","last_change":"2021-02-10T04:52:32.089886+0000","last_active":"2021-02-10T04:54:41.641625+0000","last_peered":"2021-02-10T04:54:41.641625+0000","last_clean":"2021-02-10T04:51:27.993389+0000","last_became_active":"2021-02-10T04:52:32.089886+0000","last_became_peered":"2021-02-10T04:52:32.089886+0000","last_unstale":"2021-02-10T04:54:41.641625+0000","last_undegraded":"2021-02-10T04:54:41.641625+0000","last_fullsized":"2021-02-10T04:52:32.083725+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:09.922789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:09.922789+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.8","version":"1168'82","reported_seq":"147","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.641961+0000","last_change":"2021-02-10T04:53:23.326363+0000","last_active":"2021-02-10T04:54:41.641961+0000","last_peered":"2021-02-10T04:54:41.641961+0000","last_clean":"2021-02-10T04:54:41.641961+0000","last_became_active":"2021-02-10T04:51:36.859215+0000","last_became_peered":"2021-02-10T04:51:36.859215+0000","last_unstale":"2021-02-10T04:54:41.641961+0000","last_undegraded":"2021-02-10T04:54:41.641961+0000","last_fullsized":"2021-02-10T04:54:41.641961+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1150'76","last_scrub_stamp":"2021-02-10T04:53:00.964499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:00.964499+0000","log_size":82,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3848803,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"1049'61","reported_seq":"1304","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.545353+0000","last_change":"2021-02-10T04:53:36.836307+0000","last_active":"2021-02-10T04:54:43.545353+0000","last_peered":"2021-02-10T04:54:43.545353+0000","last_clean":"2021-02-10T04:54:43.545353+0000","last_became_active":"2021-02-10T04:25:49.015913+0000","last_became_peered":"2021-02-10T04:25:49.015913+0000","last_unstale":"2021-02-10T04:54:43.545353+0000","last_undegraded":"2021-02-10T04:54:43.545353+0000","last_fullsized":"2021-02-10T04:54:43.545353+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'61","last_scrub_stamp":"2021-02-10T04:53:36.836202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:36.836202+0000","log_size":61,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":104,"num_read_kb":84,"num_write":62,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"1150'5","reported_seq":"160","reported_epoch":"1215","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:43.550899+0000","last_change":"2021-02-10T04:52:32.093814+0000","last_active":"2021-02-10T04:54:43.550899+0000","last_peered":"2021-02-10T04:54:43.550899+0000","last_clean":"2021-02-10T04:52:30.445006+0000","last_became_active":"2021-02-10T04:52:32.093814+0000","last_became_peered":"2021-02-10T04:52:32.093814+0000","last_unstale":"2021-02-10T04:54:43.550899+0000","last_undegraded":"2021-02-10T04:52:32.079897+0000","last_fullsized":"2021-02-10T04:52:32.079762+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'4","last_scrub_stamp":"2021-02-10T04:52:01.121380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.121380+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"112","reported_epoch":"1214","state":"active+undersized","last_fresh":"2021-02-10T04:54:41.642420+0000","last_change":"2021-02-10T04:52:32.093831+0000","last_active":"2021-02-10T04:54:41.642420+0000","last_peered":"2021-02-10T04:54:41.642420+0000","last_clean":"2021-02-10T04:52:30.057710+0000","last_became_active":"2021-02-10T04:52:32.093831+0000","last_became_peered":"2021-02-10T04:52:32.093831+0000","last_unstale":"2021-02-10T04:54:41.642420+0000","last_undegraded":"2021-02-10T04:54:41.642420+0000","last_fullsized":"2021-02-10T04:52:32.081168+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:52.421793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:52.421793+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"513","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.934542+0000","last_change":"2021-02-10T04:53:46.733236+0000","last_active":"2021-02-10T04:54:43.934542+0000","last_peered":"2021-02-10T04:54:43.934542+0000","last_clean":"2021-02-10T04:54:43.934542+0000","last_became_active":"2021-02-10T04:53:46.732890+0000","last_became_peered":"2021-02-10T04:53:46.732890+0000","last_unstale":"2021-02-10T04:54:43.934542+0000","last_undegraded":"2021-02-10T04:54:43.934542+0000","last_fullsized":"2021-02-10T04:54:43.934542+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:35.170157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:35.170157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"1003'739","reported_seq":"3718","reported_epoch":"1214","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:41.644130+0000","last_change":"2021-02-10T04:52:32.084576+0000","last_active":"2021-02-10T04:54:41.644130+0000","last_peered":"2021-02-10T04:54:41.644130+0000","last_clean":"2021-02-10T04:43:58.648858+0000","last_became_active":"2021-02-10T04:52:32.084576+0000","last_became_peered":"2021-02-10T04:52:32.084576+0000","last_unstale":"2021-02-10T04:54:41.644130+0000","last_undegraded":"2021-02-10T04:52:32.072214+0000","last_fullsized":"2021-02-10T04:52:32.072101+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:52:27.602006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:27.602006+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24059888,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":129,"num_read_kb":109544,"num_write":102,"num_write_kb":46173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"1192","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.544644+0000","last_change":"2021-02-10T04:54:39.559980+0000","last_active":"2021-02-10T04:54:43.544644+0000","last_peered":"2021-02-10T04:54:43.544644+0000","last_clean":"2021-02-10T04:54:43.544644+0000","last_became_active":"2021-02-10T04:54:39.559443+0000","last_became_peered":"2021-02-10T04:54:39.559443+0000","last_unstale":"2021-02-10T04:54:43.544644+0000","last_undegraded":"2021-02-10T04:54:43.544644+0000","last_fullsized":"2021-02-10T04:54:43.544644+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"1066'44","reported_seq":"1369","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.201286+0000","last_change":"2021-02-10T04:53:37.108747+0000","last_active":"2021-02-10T04:54:42.201286+0000","last_peered":"2021-02-10T04:54:42.201286+0000","last_clean":"2021-02-10T04:54:42.201286+0000","last_became_active":"2021-02-10T04:53:35.551311+0000","last_became_peered":"2021-02-10T04:53:35.551311+0000","last_unstale":"2021-02-10T04:54:42.201286+0000","last_undegraded":"2021-02-10T04:54:42.201286+0000","last_fullsized":"2021-02-10T04:54:42.201286+0000","mapping_epoch":1118,"log_start":"1066'44","ondisk_log_start":"1066'44","created":55,"last_epoch_clean":1181,"parent":"0.0","parent_split_bits":0,"last_scrub":"1066'44","last_scrub_stamp":"2021-02-10T04:53:37.108677+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:53:37.108677+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1578,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":150,"num_read_kb":114,"num_write":86,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":2312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"1109","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.201377+0000","last_change":"2021-02-10T04:53:39.126356+0000","last_active":"2021-02-10T04:54:42.201377+0000","last_peered":"2021-02-10T04:54:42.201377+0000","last_clean":"2021-02-10T04:54:42.201377+0000","last_became_active":"2021-02-10T04:29:07.145616+0000","last_became_peered":"2021-02-10T04:29:07.145616+0000","last_unstale":"2021-02-10T04:54:42.201377+0000","last_undegraded":"2021-02-10T04:54:42.201377+0000","last_fullsized":"2021-02-10T04:54:42.201377+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:39.126261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:39.126261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"433","reported_epoch":"1214","state":"active+undersized","last_fresh":"2021-02-10T04:54:41.644842+0000","last_change":"2021-02-10T04:52:32.083894+0000","last_active":"2021-02-10T04:54:41.644842+0000","last_peered":"2021-02-10T04:54:41.644842+0000","last_clean":"2021-02-10T04:52:30.060966+0000","last_became_active":"2021-02-10T04:52:32.083894+0000","last_became_peered":"2021-02-10T04:52:32.083894+0000","last_unstale":"2021-02-10T04:54:41.644842+0000","last_undegraded":"2021-02-10T04:54:41.644842+0000","last_fullsized":"2021-02-10T04:52:32.079256+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:26.311958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.311958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"135","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.644382+0000","last_change":"2021-02-10T04:54:29.567663+0000","last_active":"2021-02-10T04:54:41.644382+0000","last_peered":"2021-02-10T04:54:41.644382+0000","last_clean":"2021-02-10T04:54:41.644382+0000","last_became_active":"2021-02-10T04:51:36.855744+0000","last_became_peered":"2021-02-10T04:51:36.855744+0000","last_unstale":"2021-02-10T04:54:41.644382+0000","last_undegraded":"2021-02-10T04:54:41.644382+0000","last_fullsized":"2021-02-10T04:54:41.644382+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:02.162851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:13.727241+0000","last_clean_scrub_stamp":"2021-02-10T04:52:02.162851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"1063'654","reported_seq":"4381","reported_epoch":"1214","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:41.644026+0000","last_change":"2021-02-10T04:52:32.093941+0000","last_active":"2021-02-10T04:54:41.644026+0000","last_peered":"2021-02-10T04:54:41.644026+0000","last_clean":"2021-02-10T04:52:30.062769+0000","last_became_active":"2021-02-10T04:52:32.093941+0000","last_became_peered":"2021-02-10T04:52:32.093941+0000","last_unstale":"2021-02-10T04:54:41.644026+0000","last_undegraded":"2021-02-10T04:52:32.085816+0000","last_fullsized":"2021-02-10T04:52:32.085711+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:51:57.121838+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.121838+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24700630,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":171,"num_read_kb":182272,"num_write":111,"num_write_kb":33727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":26349023,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":12}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"1160","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.644166+0000","last_change":"2021-02-10T04:51:50.433056+0000","last_active":"2021-02-10T04:54:41.644166+0000","last_peered":"2021-02-10T04:54:41.644166+0000","last_clean":"2021-02-10T04:54:41.644166+0000","last_became_active":"2021-02-10T04:51:36.843165+0000","last_became_peered":"2021-02-10T04:51:36.843165+0000","last_unstale":"2021-02-10T04:54:41.644166+0000","last_undegraded":"2021-02-10T04:54:41.644166+0000","last_fullsized":"2021-02-10T04:54:41.644166+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.432990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.432990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"1172","reported_epoch":"1214","state":"active+undersized","last_fresh":"2021-02-10T04:54:42.328637+0000","last_change":"2021-02-10T04:54:39.557459+0000","last_active":"2021-02-10T04:54:42.328637+0000","last_peered":"2021-02-10T04:54:42.328637+0000","last_clean":"2021-02-10T04:49:56.935460+0000","last_became_active":"2021-02-10T04:54:39.557459+0000","last_became_peered":"2021-02-10T04:54:39.557459+0000","last_unstale":"2021-02-10T04:54:42.328637+0000","last_undegraded":"2021-02-10T04:54:42.328637+0000","last_fullsized":"2021-02-10T04:54:39.551094+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"1108","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.934620+0000","last_change":"2021-02-10T04:53:46.727888+0000","last_active":"2021-02-10T04:54:43.934620+0000","last_peered":"2021-02-10T04:54:43.934620+0000","last_clean":"2021-02-10T04:54:43.934620+0000","last_became_active":"2021-02-10T04:53:46.727670+0000","last_became_peered":"2021-02-10T04:53:46.727670+0000","last_unstale":"2021-02-10T04:54:43.934620+0000","last_undegraded":"2021-02-10T04:54:43.934620+0000","last_fullsized":"2021-02-10T04:54:43.934620+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:41.041553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.041553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"1187'54","reported_seq":"1290","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.200875+0000","last_change":"2021-02-10T04:53:42.071173+0000","last_active":"2021-02-10T04:54:42.200875+0000","last_peered":"2021-02-10T04:54:42.200875+0000","last_clean":"2021-02-10T04:54:42.200875+0000","last_became_active":"2021-02-10T04:53:39.679752+0000","last_became_peered":"2021-02-10T04:53:39.679752+0000","last_unstale":"2021-02-10T04:54:42.200875+0000","last_undegraded":"2021-02-10T04:54:42.200875+0000","last_fullsized":"2021-02-10T04:54:42.200875+0000","mapping_epoch":1111,"log_start":"1049'39","ondisk_log_start":"1049'39","created":55,"last_epoch_clean":1185,"parent":"0.0","parent_split_bits":0,"last_scrub":"1187'54","last_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_deep_scrub":"1187'54","last_deep_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_clean_scrub_stamp":"2021-02-10T04:53:42.071103+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":114,"num_read_kb":94,"num_write":73,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"852","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.644632+0000","last_change":"2021-02-10T04:51:52.350764+0000","last_active":"2021-02-10T04:54:41.644632+0000","last_peered":"2021-02-10T04:54:41.644632+0000","last_clean":"2021-02-10T04:54:41.644632+0000","last_became_active":"2021-02-10T04:51:49.567784+0000","last_became_peered":"2021-02-10T04:51:49.567784+0000","last_unstale":"2021-02-10T04:54:41.644632+0000","last_undegraded":"2021-02-10T04:54:41.644632+0000","last_fullsized":"2021-02-10T04:54:41.644632+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:52.350706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.350706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"1119'4","reported_seq":"154","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:43.028471+0000","last_change":"2021-02-10T04:52:27.126670+0000","last_active":"2021-02-10T04:54:43.028471+0000","last_peered":"2021-02-10T04:54:43.028471+0000","last_clean":"2021-02-10T04:54:43.028471+0000","last_became_active":"2021-02-10T04:51:36.853647+0000","last_became_peered":"2021-02-10T04:51:36.853647+0000","last_unstale":"2021-02-10T04:54:43.028471+0000","last_undegraded":"2021-02-10T04:54:43.028471+0000","last_fullsized":"2021-02-10T04:54:43.028471+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'4","last_scrub_stamp":"2021-02-10T04:52:27.126599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:27.126599+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"1200","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.201594+0000","last_change":"2021-02-10T04:52:25.745660+0000","last_active":"2021-02-10T04:54:42.201594+0000","last_peered":"2021-02-10T04:54:42.201594+0000","last_clean":"2021-02-10T04:54:42.201594+0000","last_became_active":"2021-02-10T04:51:36.849503+0000","last_became_peered":"2021-02-10T04:51:36.849503+0000","last_unstale":"2021-02-10T04:54:42.201594+0000","last_undegraded":"2021-02-10T04:54:42.201594+0000","last_fullsized":"2021-02-10T04:54:42.201594+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:25.745577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.745577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"1004'1093","reported_seq":"6255","reported_epoch":"1214","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:42.329353+0000","last_change":"2021-02-10T04:52:32.090286+0000","last_active":"2021-02-10T04:54:42.329353+0000","last_peered":"2021-02-10T04:54:42.329353+0000","last_clean":"2021-02-10T04:44:07.998404+0000","last_became_active":"2021-02-10T04:52:32.090286+0000","last_became_peered":"2021-02-10T04:52:32.090286+0000","last_unstale":"2021-02-10T04:54:42.329353+0000","last_undegraded":"2021-02-10T04:52:32.075231+0000","last_fullsized":"2021-02-10T04:52:32.075115+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39434696,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":279,"num_read_kb":311058,"num_write":185,"num_write_kb":66790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":17}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"62'5","reported_seq":"13","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.327636+0000","last_change":"2021-02-10T04:54:39.559473+0000","last_active":"2021-02-10T04:54:42.327636+0000","last_peered":"2021-02-10T04:54:42.327636+0000","last_clean":"2021-02-10T04:54:42.327636+0000","last_became_active":"2021-02-10T04:54:39.559048+0000","last_became_peered":"2021-02-10T04:54:39.559048+0000","last_unstale":"2021-02-10T04:54:42.327636+0000","last_undegraded":"2021-02-10T04:54:42.327636+0000","last_fullsized":"2021-02-10T04:54:42.327636+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":149,"num_read_kb":148,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"1159","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.646642+0000","last_change":"2021-02-10T04:54:27.220611+0000","last_active":"2021-02-10T04:54:41.646642+0000","last_peered":"2021-02-10T04:54:41.646642+0000","last_clean":"2021-02-10T04:54:41.646642+0000","last_became_active":"2021-02-10T04:51:36.841582+0000","last_became_peered":"2021-02-10T04:51:36.841582+0000","last_unstale":"2021-02-10T04:54:41.646642+0000","last_undegraded":"2021-02-10T04:54:41.646642+0000","last_fullsized":"2021-02-10T04:54:41.646642+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:12.389272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:12.389272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"1049'51","reported_seq":"888","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.645343+0000","last_change":"2021-02-10T04:53:27.560299+0000","last_active":"2021-02-10T04:54:41.645343+0000","last_peered":"2021-02-10T04:54:41.645343+0000","last_clean":"2021-02-10T04:54:41.645343+0000","last_became_active":"2021-02-10T04:53:27.559886+0000","last_became_peered":"2021-02-10T04:53:27.559886+0000","last_unstale":"2021-02-10T04:54:41.645343+0000","last_undegraded":"2021-02-10T04:54:41.645343+0000","last_fullsized":"2021-02-10T04:54:41.645343+0000","mapping_epoch":1129,"log_start":"1049'51","ondisk_log_start":"1049'51","created":55,"last_epoch_clean":1173,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'41","last_scrub_stamp":"2021-02-10T04:51:58.353660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:58.353660+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1472,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":139,"num_read_kb":106,"num_write":82,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"875","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.643994+0000","last_change":"2021-02-10T04:53:11.271557+0000","last_active":"2021-02-10T04:54:41.643994+0000","last_peered":"2021-02-10T04:54:41.643994+0000","last_clean":"2021-02-10T04:54:41.643994+0000","last_became_active":"2021-02-10T04:51:49.567883+0000","last_became_peered":"2021-02-10T04:51:49.567883+0000","last_unstale":"2021-02-10T04:54:41.643994+0000","last_undegraded":"2021-02-10T04:54:41.643994+0000","last_fullsized":"2021-02-10T04:54:41.643994+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:11.271459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.271459+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"1148'4","reported_seq":"1540","reported_epoch":"1214","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:43.028236+0000","last_change":"2021-02-10T04:52:32.088315+0000","last_active":"2021-02-10T04:54:43.028236+0000","last_peered":"2021-02-10T04:54:43.028236+0000","last_clean":"2021-02-10T04:52:30.062243+0000","last_became_active":"2021-02-10T04:52:32.088315+0000","last_became_peered":"2021-02-10T04:52:32.088315+0000","last_unstale":"2021-02-10T04:54:43.028236+0000","last_undegraded":"2021-02-10T04:52:32.069861+0000","last_fullsized":"2021-02-10T04:52:32.069742+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:51:47.855283+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.855283+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"1190","reported_epoch":"1214","state":"active+undersized","last_fresh":"2021-02-10T04:54:41.653904+0000","last_change":"2021-02-10T04:52:32.094688+0000","last_active":"2021-02-10T04:54:41.653904+0000","last_peered":"2021-02-10T04:54:41.653904+0000","last_clean":"2021-02-10T04:52:30.059376+0000","last_became_active":"2021-02-10T04:52:32.094688+0000","last_became_peered":"2021-02-10T04:52:32.094688+0000","last_unstale":"2021-02-10T04:54:41.653904+0000","last_undegraded":"2021-02-10T04:54:41.653904+0000","last_fullsized":"2021-02-10T04:52:32.079768+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:10.176878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:50:10.176878+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"1002'1090","reported_seq":"5409","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.545408+0000","last_change":"2021-02-10T04:52:26.030198+0000","last_active":"2021-02-10T04:54:43.545408+0000","last_peered":"2021-02-10T04:54:43.545408+0000","last_clean":"2021-02-10T04:54:43.545408+0000","last_became_active":"2021-02-10T04:52:26.028182+0000","last_became_peered":"2021-02-10T04:52:26.028182+0000","last_unstale":"2021-02-10T04:54:43.545408+0000","last_undegraded":"2021-02-10T04:54:43.545408+0000","last_fullsized":"2021-02-10T04:54:43.545408+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:51:08.220472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.220472+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24410646,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":245,"num_read_kb":276168,"num_write":175,"num_write_kb":64960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"1180","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.200427+0000","last_change":"2021-02-10T04:54:39.555379+0000","last_active":"2021-02-10T04:54:42.200427+0000","last_peered":"2021-02-10T04:54:42.200427+0000","last_clean":"2021-02-10T04:54:42.200427+0000","last_became_active":"2021-02-10T04:54:39.554976+0000","last_became_peered":"2021-02-10T04:54:39.554976+0000","last_unstale":"2021-02-10T04:54:42.200427+0000","last_undegraded":"2021-02-10T04:54:42.200427+0000","last_fullsized":"2021-02-10T04:54:42.200427+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"1049'47","reported_seq":"1285","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.201440+0000","last_change":"2021-02-10T04:53:31.624862+0000","last_active":"2021-02-10T04:54:42.201440+0000","last_peered":"2021-02-10T04:54:42.201440+0000","last_clean":"2021-02-10T04:54:42.201440+0000","last_became_active":"2021-02-10T04:53:31.624318+0000","last_became_peered":"2021-02-10T04:53:31.624318+0000","last_unstale":"2021-02-10T04:54:42.201440+0000","last_undegraded":"2021-02-10T04:54:42.201440+0000","last_fullsized":"2021-02-10T04:54:42.201440+0000","mapping_epoch":55,"log_start":"1049'47","ondisk_log_start":"1049'47","created":55,"last_epoch_clean":1177,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'47","last_scrub_stamp":"2021-02-10T04:53:09.002368+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.002368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1672,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":142,"num_read_kb":108,"num_write":80,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":934,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"144","reported_epoch":"1215","state":"active+undersized","last_fresh":"2021-02-10T04:54:43.934633+0000","last_change":"2021-02-10T04:53:46.729988+0000","last_active":"2021-02-10T04:54:43.934633+0000","last_peered":"2021-02-10T04:54:43.934633+0000","last_clean":"2021-02-10T04:28:18.347728+0000","last_became_active":"2021-02-10T04:53:46.729988+0000","last_became_peered":"2021-02-10T04:53:46.729988+0000","last_unstale":"2021-02-10T04:54:43.934633+0000","last_undegraded":"2021-02-10T04:54:43.934633+0000","last_fullsized":"2021-02-10T04:53:46.711213+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:08.358300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:50:08.358300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"1191","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.654165+0000","last_change":"2021-02-10T04:51:57.436961+0000","last_active":"2021-02-10T04:54:41.654165+0000","last_peered":"2021-02-10T04:54:41.654165+0000","last_clean":"2021-02-10T04:54:41.654165+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:54:41.654165+0000","last_undegraded":"2021-02-10T04:54:41.654165+0000","last_fullsized":"2021-02-10T04:54:41.654165+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.436862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.436862+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"1146","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.328904+0000","last_change":"2021-02-10T04:53:02.112612+0000","last_active":"2021-02-10T04:54:42.328904+0000","last_peered":"2021-02-10T04:54:42.328904+0000","last_clean":"2021-02-10T04:54:42.328904+0000","last_became_active":"2021-02-10T04:50:59.545501+0000","last_became_peered":"2021-02-10T04:50:59.545501+0000","last_unstale":"2021-02-10T04:54:42.328904+0000","last_undegraded":"2021-02-10T04:54:42.328904+0000","last_fullsized":"2021-02-10T04:54:42.328904+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:02.112533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:02.112533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"1001'1165","reported_seq":"4379","reported_epoch":"1215","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:43.545914+0000","last_change":"2021-02-10T04:52:32.082334+0000","last_active":"2021-02-10T04:54:43.545914+0000","last_peered":"2021-02-10T04:54:43.545914+0000","last_clean":"2021-02-10T04:52:30.068280+0000","last_became_active":"2021-02-10T04:52:32.082334+0000","last_became_peered":"2021-02-10T04:52:32.082334+0000","last_unstale":"2021-02-10T04:54:43.545914+0000","last_undegraded":"2021-02-10T04:52:32.079317+0000","last_fullsized":"2021-02-10T04:52:32.078663+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29601845,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":286,"num_read_kb":299529,"num_write":200,"num_write_kb":89130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":12}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"14","reported_epoch":"1214","state":"active+clean+remapped","last_fresh":"2021-02-10T04:54:41.645188+0000","last_change":"2021-02-10T04:54:40.559812+0000","last_active":"2021-02-10T04:54:41.645188+0000","last_peered":"2021-02-10T04:54:41.645188+0000","last_clean":"2021-02-10T04:54:41.645188+0000","last_became_active":"2021-02-10T04:54:40.559217+0000","last_became_peered":"2021-02-10T04:54:40.559217+0000","last_unstale":"2021-02-10T04:54:41.645188+0000","last_undegraded":"2021-02-10T04:54:41.645188+0000","last_fullsized":"2021-02-10T04:54:41.645188+0000","mapping_epoch":1212,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1213,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,3],"avail_no_missing":["1","3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"1113'5","reported_seq":"1436","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:43.028179+0000","last_change":"2021-02-10T04:52:34.047474+0000","last_active":"2021-02-10T04:54:43.028179+0000","last_peered":"2021-02-10T04:54:43.028179+0000","last_clean":"2021-02-10T04:54:43.028179+0000","last_became_active":"2021-02-10T04:50:59.569578+0000","last_became_peered":"2021-02-10T04:50:59.569578+0000","last_unstale":"2021-02-10T04:54:43.028179+0000","last_undegraded":"2021-02-10T04:54:43.028179+0000","last_fullsized":"2021-02-10T04:54:43.028179+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1113'5","last_scrub_stamp":"2021-02-10T04:52:34.047358+0000","last_deep_scrub":"158'3","last_deep_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_clean_scrub_stamp":"2021-02-10T04:52:34.047358+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"1049'54","reported_seq":"1276","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.200728+0000","last_change":"2021-02-10T04:51:48.872937+0000","last_active":"2021-02-10T04:54:42.200728+0000","last_peered":"2021-02-10T04:54:42.200728+0000","last_clean":"2021-02-10T04:54:42.200728+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:54:42.200728+0000","last_undegraded":"2021-02-10T04:54:42.200728+0000","last_fullsized":"2021-02-10T04:54:42.200728+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:51:48.872839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.872839+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":93,"num_read_kb":73,"num_write":54,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"1266","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.546486+0000","last_change":"2021-02-10T04:53:05.984747+0000","last_active":"2021-02-10T04:54:43.546486+0000","last_peered":"2021-02-10T04:54:43.546486+0000","last_clean":"2021-02-10T04:54:43.546486+0000","last_became_active":"2021-02-10T04:51:36.869931+0000","last_became_peered":"2021-02-10T04:51:36.869931+0000","last_unstale":"2021-02-10T04:54:43.546486+0000","last_undegraded":"2021-02-10T04:54:43.546486+0000","last_fullsized":"2021-02-10T04:54:43.546486+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:05.984659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:05.984659+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"1070","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.935739+0000","last_change":"2021-02-10T04:53:46.731709+0000","last_active":"2021-02-10T04:54:43.935739+0000","last_peered":"2021-02-10T04:54:43.935739+0000","last_clean":"2021-02-10T04:54:43.935739+0000","last_became_active":"2021-02-10T04:53:46.731340+0000","last_became_peered":"2021-02-10T04:53:46.731340+0000","last_unstale":"2021-02-10T04:54:43.935739+0000","last_undegraded":"2021-02-10T04:54:43.935739+0000","last_fullsized":"2021-02-10T04:54:43.935739+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:09.005421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.005421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"998'1009","reported_seq":"4705","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.327503+0000","last_change":"2021-02-10T04:52:26.037227+0000","last_active":"2021-02-10T04:54:42.327503+0000","last_peered":"2021-02-10T04:54:42.327503+0000","last_clean":"2021-02-10T04:54:42.327503+0000","last_became_active":"2021-02-10T04:52:26.036723+0000","last_became_peered":"2021-02-10T04:52:26.036723+0000","last_unstale":"2021-02-10T04:54:42.327503+0000","last_undegraded":"2021-02-10T04:54:42.327503+0000","last_fullsized":"2021-02-10T04:54:42.327503+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12355206,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":119,"num_read_kb":124701,"num_write":77,"num_write_kb":25052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4145029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"1085","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.329389+0000","last_change":"2021-02-10T04:53:33.305896+0000","last_active":"2021-02-10T04:54:42.329389+0000","last_peered":"2021-02-10T04:54:42.329389+0000","last_clean":"2021-02-10T04:54:42.329389+0000","last_became_active":"2021-02-10T04:50:59.570854+0000","last_became_peered":"2021-02-10T04:50:59.570854+0000","last_unstale":"2021-02-10T04:54:42.329389+0000","last_undegraded":"2021-02-10T04:54:42.329389+0000","last_fullsized":"2021-02-10T04:54:42.329389+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:33.305827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:33.305827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"1176","reported_epoch":"1214","state":"active+clean+remapped","last_fresh":"2021-02-10T04:54:42.329080+0000","last_change":"2021-02-10T04:54:40.965288+0000","last_active":"2021-02-10T04:54:42.329080+0000","last_peered":"2021-02-10T04:54:42.329080+0000","last_clean":"2021-02-10T04:54:42.329080+0000","last_became_active":"2021-02-10T04:54:40.964924+0000","last_became_peered":"2021-02-10T04:54:40.964924+0000","last_unstale":"2021-02-10T04:54:42.329080+0000","last_undegraded":"2021-02-10T04:54:42.329080+0000","last_fullsized":"2021-02-10T04:54:42.329080+0000","mapping_epoch":1212,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1213,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7,4],"avail_no_missing":["3","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"1199","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.201262+0000","last_change":"2021-02-10T04:51:50.895471+0000","last_active":"2021-02-10T04:54:42.201262+0000","last_peered":"2021-02-10T04:54:42.201262+0000","last_clean":"2021-02-10T04:54:42.201262+0000","last_became_active":"2021-02-10T04:50:59.527553+0000","last_became_peered":"2021-02-10T04:50:59.527553+0000","last_unstale":"2021-02-10T04:54:42.201262+0000","last_undegraded":"2021-02-10T04:54:42.201262+0000","last_fullsized":"2021-02-10T04:54:42.201262+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.895370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.895370+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"1049'72","reported_seq":"1303","reported_epoch":"1214","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:42.328129+0000","last_change":"2021-02-10T04:52:32.090145+0000","last_active":"2021-02-10T04:54:42.328129+0000","last_peered":"2021-02-10T04:54:42.328129+0000","last_clean":"2021-02-10T04:52:30.063735+0000","last_became_active":"2021-02-10T04:52:32.090145+0000","last_became_peered":"2021-02-10T04:52:32.090145+0000","last_unstale":"2021-02-10T04:54:42.328129+0000","last_undegraded":"2021-02-10T04:52:32.073169+0000","last_fullsized":"2021-02-10T04:52:32.073062+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'72","last_scrub_stamp":"2021-02-10T04:46:30.515026+0000","last_deep_scrub":"152'48","last_deep_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_clean_scrub_stamp":"2021-02-10T04:46:30.515026+0000","log_size":72,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":105,"num_read_kb":83,"num_write":60,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":14}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"463","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.645562+0000","last_change":"2021-02-10T04:54:29.823653+0000","last_active":"2021-02-10T04:54:41.645562+0000","last_peered":"2021-02-10T04:54:41.645562+0000","last_clean":"2021-02-10T04:54:41.645562+0000","last_became_active":"2021-02-10T04:51:49.503801+0000","last_became_peered":"2021-02-10T04:51:49.503801+0000","last_unstale":"2021-02-10T04:54:41.645562+0000","last_undegraded":"2021-02-10T04:54:41.645562+0000","last_fullsized":"2021-02-10T04:54:41.645562+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:51:48.274960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.274960+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"1224","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.201347+0000","last_change":"2021-02-10T04:53:13.099288+0000","last_active":"2021-02-10T04:54:42.201347+0000","last_peered":"2021-02-10T04:54:42.201347+0000","last_clean":"2021-02-10T04:54:42.201347+0000","last_became_active":"2021-02-10T04:51:36.848614+0000","last_became_peered":"2021-02-10T04:51:36.848614+0000","last_unstale":"2021-02-10T04:54:42.201347+0000","last_undegraded":"2021-02-10T04:54:42.201347+0000","last_fullsized":"2021-02-10T04:54:42.201347+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:13.099228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:13.099228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"1002'1657","reported_seq":"5275","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.327534+0000","last_change":"2021-02-10T04:52:26.036806+0000","last_active":"2021-02-10T04:54:42.327534+0000","last_peered":"2021-02-10T04:54:42.327534+0000","last_clean":"2021-02-10T04:54:42.327534+0000","last_became_active":"2021-02-10T04:52:26.032363+0000","last_became_peered":"2021-02-10T04:52:26.032363+0000","last_unstale":"2021-02-10T04:54:42.327534+0000","last_undegraded":"2021-02-10T04:54:42.327534+0000","last_fullsized":"2021-02-10T04:54:42.327534+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21396533,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":212,"num_read_kb":218836,"num_write":159,"num_write_kb":59398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":22326014,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"1143","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.200843+0000","last_change":"2021-02-10T04:51:51.911945+0000","last_active":"2021-02-10T04:54:42.200843+0000","last_peered":"2021-02-10T04:54:42.200843+0000","last_clean":"2021-02-10T04:54:42.200843+0000","last_became_active":"2021-02-10T04:51:36.850015+0000","last_became_peered":"2021-02-10T04:51:36.850015+0000","last_unstale":"2021-02-10T04:54:42.200843+0000","last_undegraded":"2021-02-10T04:54:42.200843+0000","last_fullsized":"2021-02-10T04:54:42.200843+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.911851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.911851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"62'5","reported_seq":"1048","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.644521+0000","last_change":"2021-02-10T04:54:39.568253+0000","last_active":"2021-02-10T04:54:41.644521+0000","last_peered":"2021-02-10T04:54:41.644521+0000","last_clean":"2021-02-10T04:54:41.644521+0000","last_became_active":"2021-02-10T04:54:39.567642+0000","last_became_peered":"2021-02-10T04:54:39.567642+0000","last_unstale":"2021-02-10T04:54:41.644521+0000","last_undegraded":"2021-02-10T04:54:41.644521+0000","last_fullsized":"2021-02-10T04:54:41.644521+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":98,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":150,"num_read_kb":148,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"1264","reported_epoch":"1215","state":"active+undersized","last_fresh":"2021-02-10T04:54:43.545436+0000","last_change":"2021-02-10T04:52:32.089163+0000","last_active":"2021-02-10T04:54:43.545436+0000","last_peered":"2021-02-10T04:54:43.545436+0000","last_clean":"2021-02-10T04:52:30.062078+0000","last_became_active":"2021-02-10T04:52:32.089163+0000","last_became_peered":"2021-02-10T04:52:32.089163+0000","last_unstale":"2021-02-10T04:54:43.545436+0000","last_undegraded":"2021-02-10T04:54:43.545436+0000","last_fullsized":"2021-02-10T04:52:32.072413+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:58.142526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:58.142526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"1135'58","reported_seq":"1269","reported_epoch":"1214","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:41.652643+0000","last_change":"2021-02-10T04:52:32.074386+0000","last_active":"2021-02-10T04:54:41.652643+0000","last_peered":"2021-02-10T04:54:41.652643+0000","last_clean":"2021-02-10T04:52:01.474364+0000","last_became_active":"2021-02-10T04:52:32.074386+0000","last_became_peered":"2021-02-10T04:52:32.074386+0000","last_unstale":"2021-02-10T04:54:41.652643+0000","last_undegraded":"2021-02-10T04:52:32.071237+0000","last_fullsized":"2021-02-10T04:52:32.070967+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1135'58","last_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_deep_scrub":"1135'58","last_deep_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.483067+0000","log_size":58,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":87,"num_read_kb":68,"num_write":50,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":11}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"142","reported_epoch":"1215","state":"active+undersized","last_fresh":"2021-02-10T04:54:43.936535+0000","last_change":"2021-02-10T04:53:46.733605+0000","last_active":"2021-02-10T04:54:43.936535+0000","last_peered":"2021-02-10T04:54:43.936535+0000","last_clean":"2021-02-10T04:29:05.137445+0000","last_became_active":"2021-02-10T04:53:46.733605+0000","last_became_peered":"2021-02-10T04:53:46.733605+0000","last_unstale":"2021-02-10T04:54:43.936535+0000","last_undegraded":"2021-02-10T04:54:43.936535+0000","last_fullsized":"2021-02-10T04:53:46.710695+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:50:46.434629+0000","last_deep_scrub":"53'2","last_deep_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_clean_scrub_stamp":"2021-02-10T04:50:46.434629+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"892","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.644072+0000","last_change":"2021-02-10T04:52:38.278980+0000","last_active":"2021-02-10T04:54:41.644072+0000","last_peered":"2021-02-10T04:54:41.644072+0000","last_clean":"2021-02-10T04:54:41.644072+0000","last_became_active":"2021-02-10T04:51:49.496237+0000","last_became_peered":"2021-02-10T04:51:49.496237+0000","last_unstale":"2021-02-10T04:54:41.644072+0000","last_undegraded":"2021-02-10T04:54:41.644072+0000","last_fullsized":"2021-02-10T04:54:41.644072+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:38.278910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:52:38.278910+0000","last_clean_scrub_stamp":"2021-02-10T04:52:38.278910+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"1000'1071","reported_seq":"4765","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.644463+0000","last_change":"2021-02-10T04:53:34.407875+0000","last_active":"2021-02-10T04:54:41.644463+0000","last_peered":"2021-02-10T04:54:41.644463+0000","last_clean":"2021-02-10T04:54:41.644463+0000","last_became_active":"2021-02-10T04:52:26.031050+0000","last_became_peered":"2021-02-10T04:52:26.031050+0000","last_unstale":"2021-02-10T04:54:41.644463+0000","last_undegraded":"2021-02-10T04:54:41.644463+0000","last_fullsized":"2021-02-10T04:54:41.644463+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:34.407784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.407784+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30173843,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"1230","reported_epoch":"1215","state":"active+undersized","last_fresh":"2021-02-10T04:54:43.546386+0000","last_change":"2021-02-10T04:52:32.081955+0000","last_active":"2021-02-10T04:54:43.546386+0000","last_peered":"2021-02-10T04:54:43.546386+0000","last_clean":"2021-02-10T04:52:30.068208+0000","last_became_active":"2021-02-10T04:52:32.081955+0000","last_became_peered":"2021-02-10T04:52:32.081955+0000","last_unstale":"2021-02-10T04:54:43.546386+0000","last_undegraded":"2021-02-10T04:54:43.546386+0000","last_fullsized":"2021-02-10T04:52:32.077723+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_clean_scrub_stamp":"2021-02-10T04:51:00.195518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"1192","reported_epoch":"1214","state":"active+undersized","last_fresh":"2021-02-10T04:54:41.654365+0000","last_change":"2021-02-10T04:52:32.097628+0000","last_active":"2021-02-10T04:54:41.654365+0000","last_peered":"2021-02-10T04:54:41.654365+0000","last_clean":"2021-02-10T04:52:30.061275+0000","last_became_active":"2021-02-10T04:52:32.097628+0000","last_became_peered":"2021-02-10T04:52:32.097628+0000","last_unstale":"2021-02-10T04:54:41.654365+0000","last_undegraded":"2021-02-10T04:54:41.654365+0000","last_fullsized":"2021-02-10T04:52:32.086329+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:55.473347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:50:55.473347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"1049'40","reported_seq":"1261","reported_epoch":"1214","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:41.643880+0000","last_change":"2021-02-10T04:52:32.094738+0000","last_active":"2021-02-10T04:54:41.643880+0000","last_peered":"2021-02-10T04:54:41.643880+0000","last_clean":"2021-02-10T04:51:27.993802+0000","last_became_active":"2021-02-10T04:52:32.094738+0000","last_became_peered":"2021-02-10T04:52:32.094738+0000","last_unstale":"2021-02-10T04:54:41.643880+0000","last_undegraded":"2021-02-10T04:52:32.085496+0000","last_fullsized":"2021-02-10T04:52:32.085393+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'40","last_scrub_stamp":"2021-02-10T04:51:47.281929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.281929+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":52,"num_write":40,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":10}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"368","reported_epoch":"1215","state":"active+undersized","last_fresh":"2021-02-10T04:54:43.934773+0000","last_change":"2021-02-10T04:53:46.731083+0000","last_active":"2021-02-10T04:54:43.934773+0000","last_peered":"2021-02-10T04:54:43.934773+0000","last_clean":"2021-02-10T04:52:30.068091+0000","last_became_active":"2021-02-10T04:53:46.731083+0000","last_became_peered":"2021-02-10T04:53:46.731083+0000","last_unstale":"2021-02-10T04:54:43.934773+0000","last_undegraded":"2021-02-10T04:54:43.934773+0000","last_fullsized":"2021-02-10T04:53:46.711548+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:50.399350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:50.399350+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"1172","reported_epoch":"1214","state":"active+undersized","last_fresh":"2021-02-10T04:54:42.201028+0000","last_change":"2021-02-10T04:54:37.618129+0000","last_active":"2021-02-10T04:54:42.201028+0000","last_peered":"2021-02-10T04:54:42.201028+0000","last_clean":"2021-02-10T04:49:56.935460+0000","last_became_active":"2021-02-10T04:54:37.618129+0000","last_became_peered":"2021-02-10T04:54:37.618129+0000","last_unstale":"2021-02-10T04:54:42.201028+0000","last_undegraded":"2021-02-10T04:54:42.201028+0000","last_fullsized":"2021-02-10T04:54:37.607917+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"877","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.644858+0000","last_change":"2021-02-10T04:52:31.273037+0000","last_active":"2021-02-10T04:54:41.644858+0000","last_peered":"2021-02-10T04:54:41.644858+0000","last_clean":"2021-02-10T04:54:41.644858+0000","last_became_active":"2021-02-10T04:51:49.501405+0000","last_became_peered":"2021-02-10T04:51:49.501405+0000","last_unstale":"2021-02-10T04:54:41.644858+0000","last_undegraded":"2021-02-10T04:54:41.644858+0000","last_fullsized":"2021-02-10T04:54:41.644858+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:31.272971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:17.598113+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.272971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"1000'446","reported_seq":"1925","reported_epoch":"1215","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:43.936082+0000","last_change":"2021-02-10T04:53:46.723625+0000","last_active":"2021-02-10T04:54:43.936082+0000","last_peered":"2021-02-10T04:54:43.936082+0000","last_clean":"2021-02-10T04:43:52.634592+0000","last_became_active":"2021-02-10T04:53:46.723625+0000","last_became_peered":"2021-02-10T04:53:46.723625+0000","last_unstale":"2021-02-10T04:54:43.936082+0000","last_undegraded":"2021-02-10T04:53:46.707033+0000","last_fullsized":"2021-02-10T04:53:46.706921+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:51:45.964029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.964029+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22255852,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68435,"num_write":72,"num_write_kb":24345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":403693,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":9}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"368","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.644640+0000","last_change":"2021-02-10T04:53:03.228260+0000","last_active":"2021-02-10T04:54:41.644640+0000","last_peered":"2021-02-10T04:54:41.644640+0000","last_clean":"2021-02-10T04:54:41.644640+0000","last_became_active":"2021-02-10T04:51:49.499252+0000","last_became_peered":"2021-02-10T04:51:49.499252+0000","last_unstale":"2021-02-10T04:54:41.644640+0000","last_undegraded":"2021-02-10T04:54:41.644640+0000","last_fullsized":"2021-02-10T04:54:41.644640+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:03.228182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.228182+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"121","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.936295+0000","last_change":"2021-02-10T04:53:46.727384+0000","last_active":"2021-02-10T04:54:43.936295+0000","last_peered":"2021-02-10T04:54:43.936295+0000","last_clean":"2021-02-10T04:54:43.936295+0000","last_became_active":"2021-02-10T04:53:46.726771+0000","last_became_peered":"2021-02-10T04:53:46.726771+0000","last_unstale":"2021-02-10T04:54:43.936295+0000","last_undegraded":"2021-02-10T04:54:43.936295+0000","last_fullsized":"2021-02-10T04:54:43.936295+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:33.948348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:33.948348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"1049'39","reported_seq":"1283","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.545779+0000","last_change":"2021-02-10T04:54:17.118810+0000","last_active":"2021-02-10T04:54:43.545779+0000","last_peered":"2021-02-10T04:54:43.545779+0000","last_clean":"2021-02-10T04:54:43.545779+0000","last_became_active":"2021-02-10T04:54:17.080053+0000","last_became_peered":"2021-02-10T04:54:17.080053+0000","last_unstale":"2021-02-10T04:54:43.545779+0000","last_undegraded":"2021-02-10T04:54:43.545779+0000","last_fullsized":"2021-02-10T04:54:43.545779+0000","mapping_epoch":1200,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1201,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'39","last_scrub_stamp":"2021-02-10T04:51:47.322331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.322331+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":51,"num_write":39,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"140","reported_epoch":"1215","state":"active+undersized","last_fresh":"2021-02-10T04:54:43.934910+0000","last_change":"2021-02-10T04:53:46.714141+0000","last_active":"2021-02-10T04:54:43.934910+0000","last_peered":"2021-02-10T04:54:43.934910+0000","last_clean":"2021-02-10T04:28:45.095539+0000","last_became_active":"2021-02-10T04:53:46.714141+0000","last_became_peered":"2021-02-10T04:53:46.714141+0000","last_unstale":"2021-02-10T04:54:43.934910+0000","last_undegraded":"2021-02-10T04:54:43.934910+0000","last_fullsized":"2021-02-10T04:53:46.700190+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_clean_scrub_stamp":"2021-02-10T04:50:05.424262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"1385","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.652543+0000","last_change":"2021-02-10T04:52:37.324122+0000","last_active":"2021-02-10T04:54:41.652543+0000","last_peered":"2021-02-10T04:54:41.652543+0000","last_clean":"2021-02-10T04:54:41.652543+0000","last_became_active":"2021-02-10T04:51:36.843553+0000","last_became_peered":"2021-02-10T04:51:36.843553+0000","last_unstale":"2021-02-10T04:54:41.652543+0000","last_undegraded":"2021-02-10T04:54:41.652543+0000","last_fullsized":"2021-02-10T04:54:41.652543+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:37.324040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:37.324040+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"1191","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.328236+0000","last_change":"2021-02-10T04:54:37.612303+0000","last_active":"2021-02-10T04:54:42.328236+0000","last_peered":"2021-02-10T04:54:42.328236+0000","last_clean":"2021-02-10T04:54:42.328236+0000","last_became_active":"2021-02-10T04:54:37.611953+0000","last_became_peered":"2021-02-10T04:54:37.611953+0000","last_unstale":"2021-02-10T04:54:42.328236+0000","last_undegraded":"2021-02-10T04:54:42.328236+0000","last_fullsized":"2021-02-10T04:54:42.328236+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"1213","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.654486+0000","last_change":"2021-02-10T04:52:31.403904+0000","last_active":"2021-02-10T04:54:41.654486+0000","last_peered":"2021-02-10T04:54:41.654486+0000","last_clean":"2021-02-10T04:54:41.654486+0000","last_became_active":"2021-02-10T04:50:59.538009+0000","last_became_peered":"2021-02-10T04:50:59.538009+0000","last_unstale":"2021-02-10T04:54:41.654486+0000","last_undegraded":"2021-02-10T04:54:41.654486+0000","last_fullsized":"2021-02-10T04:54:41.654486+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:31.403824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.403824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"1000'490","reported_seq":"3480","reported_epoch":"1215","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:43.940023+0000","last_change":"2021-02-10T04:53:46.711617+0000","last_active":"2021-02-10T04:54:43.940023+0000","last_peered":"2021-02-10T04:54:43.940023+0000","last_clean":"2021-02-10T04:43:50.630768+0000","last_became_active":"2021-02-10T04:53:46.711617+0000","last_became_peered":"2021-02-10T04:53:46.711617+0000","last_unstale":"2021-02-10T04:54:43.940023+0000","last_undegraded":"2021-02-10T04:53:46.702746+0000","last_fullsized":"2021-02-10T04:53:46.702635+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1040,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:47:02.875920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:02.875920+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19371627,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":141,"num_read_kb":148568,"num_write":85,"num_write_kb":31732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":7}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"111","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.644297+0000","last_change":"2021-02-10T04:53:01.972991+0000","last_active":"2021-02-10T04:54:41.644297+0000","last_peered":"2021-02-10T04:54:41.644297+0000","last_clean":"2021-02-10T04:54:41.644297+0000","last_became_active":"2021-02-10T04:51:36.863976+0000","last_became_peered":"2021-02-10T04:51:36.863976+0000","last_unstale":"2021-02-10T04:54:41.644297+0000","last_undegraded":"2021-02-10T04:54:41.644297+0000","last_fullsized":"2021-02-10T04:54:41.644297+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:01.972913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.972913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"1049'37","reported_seq":"892","reported_epoch":"1214","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:41.645985+0000","last_change":"2021-02-10T04:52:32.095888+0000","last_active":"2021-02-10T04:54:41.645985+0000","last_peered":"2021-02-10T04:54:41.645985+0000","last_clean":"2021-02-10T04:52:30.062113+0000","last_became_active":"2021-02-10T04:52:32.095888+0000","last_became_peered":"2021-02-10T04:52:32.095888+0000","last_unstale":"2021-02-10T04:54:41.645985+0000","last_undegraded":"2021-02-10T04:52:32.076227+0000","last_fullsized":"2021-02-10T04:52:32.076110+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:56.431187+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.431187+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":59,"num_read_kb":43,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"112","reported_epoch":"1214","state":"active+undersized","last_fresh":"2021-02-10T04:54:41.641926+0000","last_change":"2021-02-10T04:52:32.089886+0000","last_active":"2021-02-10T04:54:41.641926+0000","last_peered":"2021-02-10T04:54:41.641926+0000","last_clean":"2021-02-10T04:52:30.064491+0000","last_became_active":"2021-02-10T04:52:32.089886+0000","last_became_peered":"2021-02-10T04:52:32.089886+0000","last_unstale":"2021-02-10T04:54:41.641926+0000","last_undegraded":"2021-02-10T04:54:41.641926+0000","last_fullsized":"2021-02-10T04:52:32.078850+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:04.591695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:04.591695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"1041","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.644112+0000","last_change":"2021-02-10T04:54:37.680913+0000","last_active":"2021-02-10T04:54:41.644112+0000","last_peered":"2021-02-10T04:54:41.644112+0000","last_clean":"2021-02-10T04:54:41.644112+0000","last_became_active":"2021-02-10T04:54:37.680651+0000","last_became_peered":"2021-02-10T04:54:37.680651+0000","last_unstale":"2021-02-10T04:54:41.644112+0000","last_undegraded":"2021-02-10T04:54:41.644112+0000","last_fullsized":"2021-02-10T04:54:41.644112+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1210,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":149,"num_read_kb":148,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"1060","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.936445+0000","last_change":"2021-02-10T04:53:46.729138+0000","last_active":"2021-02-10T04:54:43.936445+0000","last_peered":"2021-02-10T04:54:43.936445+0000","last_clean":"2021-02-10T04:54:43.936445+0000","last_became_active":"2021-02-10T04:53:46.728845+0000","last_became_peered":"2021-02-10T04:53:46.728845+0000","last_unstale":"2021-02-10T04:54:43.936445+0000","last_undegraded":"2021-02-10T04:54:43.936445+0000","last_fullsized":"2021-02-10T04:54:43.936445+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:40.066354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:52:40.066354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"1000'620","reported_seq":"4220","reported_epoch":"1214","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:42.200809+0000","last_change":"2021-02-10T04:52:32.082414+0000","last_active":"2021-02-10T04:54:42.200809+0000","last_peered":"2021-02-10T04:54:42.200809+0000","last_clean":"2021-02-10T04:43:51.960482+0000","last_became_active":"2021-02-10T04:52:32.082414+0000","last_became_peered":"2021-02-10T04:52:32.082414+0000","last_unstale":"2021-02-10T04:54:42.200809+0000","last_undegraded":"2021-02-10T04:52:32.068324+0000","last_fullsized":"2021-02-10T04:52:32.068200+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:51:54.444287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:54.444287+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14265195,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":144,"num_read_kb":128482,"num_write":129,"num_write_kb":53443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"168","reported_epoch":"1214","state":"active+undersized","last_fresh":"2021-02-10T04:54:42.201363+0000","last_change":"2021-02-10T04:52:32.076404+0000","last_active":"2021-02-10T04:54:42.201363+0000","last_peered":"2021-02-10T04:54:42.201363+0000","last_clean":"2021-02-10T04:28:47.053193+0000","last_became_active":"2021-02-10T04:52:32.076404+0000","last_became_peered":"2021-02-10T04:52:32.076404+0000","last_unstale":"2021-02-10T04:54:42.201363+0000","last_undegraded":"2021-02-10T04:54:42.201363+0000","last_fullsized":"2021-02-10T04:52:32.068279+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:14.955544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:47:14.955544+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"1181","reported_epoch":"1214","state":"active+clean+remapped","last_fresh":"2021-02-10T04:54:42.200759+0000","last_change":"2021-02-10T04:54:40.558152+0000","last_active":"2021-02-10T04:54:42.200759+0000","last_peered":"2021-02-10T04:54:42.200759+0000","last_clean":"2021-02-10T04:54:42.200759+0000","last_became_active":"2021-02-10T04:54:40.557329+0000","last_became_peered":"2021-02-10T04:54:40.557329+0000","last_unstale":"2021-02-10T04:54:42.200759+0000","last_undegraded":"2021-02-10T04:54:42.200759+0000","last_fullsized":"2021-02-10T04:54:42.200759+0000","mapping_epoch":1212,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1213,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,3],"avail_no_missing":["4","3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"75","reported_epoch":"1214","state":"active+undersized","last_fresh":"2021-02-10T04:54:41.645863+0000","last_change":"2021-02-10T04:52:32.084350+0000","last_active":"2021-02-10T04:54:41.645863+0000","last_peered":"2021-02-10T04:54:41.645863+0000","last_clean":"2021-02-10T04:25:49.999907+0000","last_became_active":"2021-02-10T04:52:32.084350+0000","last_became_peered":"2021-02-10T04:52:32.084350+0000","last_unstale":"2021-02-10T04:54:41.645863+0000","last_undegraded":"2021-02-10T04:54:41.645863+0000","last_fullsized":"2021-02-10T04:52:32.073004+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:27.908221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:47:27.908221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"1049'55","reported_seq":"848","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.644683+0000","last_change":"2021-02-10T04:52:30.277009+0000","last_active":"2021-02-10T04:54:41.644683+0000","last_peered":"2021-02-10T04:54:41.644683+0000","last_clean":"2021-02-10T04:54:41.644683+0000","last_became_active":"2021-02-10T04:51:49.501622+0000","last_became_peered":"2021-02-10T04:51:49.501622+0000","last_unstale":"2021-02-10T04:54:41.644683+0000","last_undegraded":"2021-02-10T04:54:41.644683+0000","last_fullsized":"2021-02-10T04:54:41.644683+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'55","last_scrub_stamp":"2021-02-10T04:52:30.276938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.276938+0000","log_size":55,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":98,"num_read_kb":74,"num_write":58,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"1209","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.651893+0000","last_change":"2021-02-10T04:51:59.420149+0000","last_active":"2021-02-10T04:54:41.651893+0000","last_peered":"2021-02-10T04:54:41.651893+0000","last_clean":"2021-02-10T04:54:41.651893+0000","last_became_active":"2021-02-10T04:28:21.125541+0000","last_became_peered":"2021-02-10T04:28:21.125541+0000","last_unstale":"2021-02-10T04:54:41.651893+0000","last_undegraded":"2021-02-10T04:54:41.651893+0000","last_fullsized":"2021-02-10T04:54:41.651893+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.420086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.420086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"1175","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.329060+0000","last_change":"2021-02-10T04:53:12.187225+0000","last_active":"2021-02-10T04:54:42.329060+0000","last_peered":"2021-02-10T04:54:42.329060+0000","last_clean":"2021-02-10T04:54:42.329060+0000","last_became_active":"2021-02-10T04:50:59.555689+0000","last_became_peered":"2021-02-10T04:50:59.555689+0000","last_unstale":"2021-02-10T04:54:42.329060+0000","last_undegraded":"2021-02-10T04:54:42.329060+0000","last_fullsized":"2021-02-10T04:54:42.329060+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.187154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.187154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"1002'2042","reported_seq":"9438","reported_epoch":"1215","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:43.546415+0000","last_change":"2021-02-10T04:52:32.083034+0000","last_active":"2021-02-10T04:54:43.546415+0000","last_peered":"2021-02-10T04:54:43.546415+0000","last_clean":"2021-02-10T04:52:30.068473+0000","last_became_active":"2021-02-10T04:52:32.083034+0000","last_became_peered":"2021-02-10T04:52:32.083034+0000","last_unstale":"2021-02-10T04:54:43.546415+0000","last_undegraded":"2021-02-10T04:52:32.077334+0000","last_fullsized":"2021-02-10T04:52:32.077220+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:46:57.054230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:57.054230+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":85678898,"num_objects":35,"num_object_clones":29,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":35}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"1151","reported_epoch":"1214","state":"active+undersized","last_fresh":"2021-02-10T04:54:41.644163+0000","last_change":"2021-02-10T04:52:32.094777+0000","last_active":"2021-02-10T04:54:41.644163+0000","last_peered":"2021-02-10T04:54:41.644163+0000","last_clean":"2021-02-10T04:52:30.061448+0000","last_became_active":"2021-02-10T04:52:32.094777+0000","last_became_peered":"2021-02-10T04:52:32.094777+0000","last_unstale":"2021-02-10T04:54:41.644163+0000","last_undegraded":"2021-02-10T04:54:41.644163+0000","last_fullsized":"2021-02-10T04:52:32.069238+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:52.341527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:50:52.341527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"13","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.653254+0000","last_change":"2021-02-10T04:54:39.558380+0000","last_active":"2021-02-10T04:54:41.653254+0000","last_peered":"2021-02-10T04:54:41.653254+0000","last_clean":"2021-02-10T04:54:41.653254+0000","last_became_active":"2021-02-10T04:54:39.557995+0000","last_became_peered":"2021-02-10T04:54:39.557995+0000","last_unstale":"2021-02-10T04:54:41.653254+0000","last_undegraded":"2021-02-10T04:54:41.653254+0000","last_fullsized":"2021-02-10T04:54:41.653254+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"1148'3","reported_seq":"114","reported_epoch":"1215","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:48.029574+0000","last_change":"2021-02-10T04:52:32.089249+0000","last_active":"2021-02-10T04:54:48.029574+0000","last_peered":"2021-02-10T04:54:48.029574+0000","last_clean":"2021-02-10T04:25:51.042416+0000","last_became_active":"2021-02-10T04:52:32.089249+0000","last_became_peered":"2021-02-10T04:52:32.089249+0000","last_unstale":"2021-02-10T04:54:48.029574+0000","last_undegraded":"2021-02-10T04:52:32.073980+0000","last_fullsized":"2021-02-10T04:52:32.073877+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:54.466259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:54.466259+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"1049'54","reported_seq":"897","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.644861+0000","last_change":"2021-02-10T04:53:01.304693+0000","last_active":"2021-02-10T04:54:41.644861+0000","last_peered":"2021-02-10T04:54:41.644861+0000","last_clean":"2021-02-10T04:54:41.644861+0000","last_became_active":"2021-02-10T04:51:49.498066+0000","last_became_peered":"2021-02-10T04:51:49.498066+0000","last_unstale":"2021-02-10T04:54:41.644861+0000","last_undegraded":"2021-02-10T04:54:41.644861+0000","last_fullsized":"2021-02-10T04:54:41.644861+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:53:01.304626+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.304626+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":74,"num_read_kb":56,"num_write":42,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"464","reported_epoch":"1214","state":"active+undersized","last_fresh":"2021-02-10T04:54:41.644728+0000","last_change":"2021-02-10T04:52:32.091144+0000","last_active":"2021-02-10T04:54:41.644728+0000","last_peered":"2021-02-10T04:54:41.644728+0000","last_clean":"2021-02-10T04:52:30.060819+0000","last_became_active":"2021-02-10T04:52:32.091144+0000","last_became_peered":"2021-02-10T04:52:32.091144+0000","last_unstale":"2021-02-10T04:54:41.644728+0000","last_undegraded":"2021-02-10T04:54:41.644728+0000","last_fullsized":"2021-02-10T04:52:32.079969+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:49.447961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:49.447961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"1196","reported_epoch":"1214","state":"active+undersized","last_fresh":"2021-02-10T04:54:42.327760+0000","last_change":"2021-02-10T04:52:32.088766+0000","last_active":"2021-02-10T04:54:42.327760+0000","last_peered":"2021-02-10T04:54:42.327760+0000","last_clean":"2021-02-10T04:51:27.993073+0000","last_became_active":"2021-02-10T04:52:32.088766+0000","last_became_peered":"2021-02-10T04:52:32.088766+0000","last_unstale":"2021-02-10T04:54:42.327760+0000","last_undegraded":"2021-02-10T04:54:42.327760+0000","last_fullsized":"2021-02-10T04:52:32.071883+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:07.914061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:07.914061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"293'77","reported_seq":"2996","reported_epoch":"1214","state":"active+clean+scrubbing","last_fresh":"2021-02-10T04:54:42.200347+0000","last_change":"2021-02-10T04:53:43.102301+0000","last_active":"2021-02-10T04:54:42.200347+0000","last_peered":"2021-02-10T04:54:42.200347+0000","last_clean":"2021-02-10T04:54:42.200347+0000","last_became_active":"2021-02-10T04:52:26.024812+0000","last_became_peered":"2021-02-10T04:52:26.024812+0000","last_unstale":"2021-02-10T04:54:42.200347+0000","last_undegraded":"2021-02-10T04:54:42.200347+0000","last_fullsized":"2021-02-10T04:54:42.200347+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1886585,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":18587,"num_write":9,"num_write_kb":3315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"1229","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.546359+0000","last_change":"2021-02-10T04:53:07.025068+0000","last_active":"2021-02-10T04:54:43.546359+0000","last_peered":"2021-02-10T04:54:43.546359+0000","last_clean":"2021-02-10T04:54:43.546359+0000","last_became_active":"2021-02-10T04:50:59.589417+0000","last_became_peered":"2021-02-10T04:50:59.589417+0000","last_unstale":"2021-02-10T04:54:43.546359+0000","last_undegraded":"2021-02-10T04:54:43.546359+0000","last_fullsized":"2021-02-10T04:54:43.546359+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.024980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.024980+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"112","reported_epoch":"1214","state":"active+undersized","last_fresh":"2021-02-10T04:54:41.641548+0000","last_change":"2021-02-10T04:52:32.082945+0000","last_active":"2021-02-10T04:54:41.641548+0000","last_peered":"2021-02-10T04:54:41.641548+0000","last_clean":"2021-02-10T04:52:30.056402+0000","last_became_active":"2021-02-10T04:52:32.082945+0000","last_became_peered":"2021-02-10T04:52:32.082945+0000","last_unstale":"2021-02-10T04:54:41.641548+0000","last_undegraded":"2021-02-10T04:54:41.641548+0000","last_fullsized":"2021-02-10T04:52:32.077773+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.609500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.609500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"1048'39","reported_seq":"1309","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.200827+0000","last_change":"2021-02-10T04:53:38.085663+0000","last_active":"2021-02-10T04:54:42.200827+0000","last_peered":"2021-02-10T04:54:42.200827+0000","last_clean":"2021-02-10T04:54:42.200827+0000","last_became_active":"2021-02-10T04:39:02.560739+0000","last_became_peered":"2021-02-10T04:39:02.560739+0000","last_unstale":"2021-02-10T04:54:42.200827+0000","last_undegraded":"2021-02-10T04:54:42.200827+0000","last_fullsized":"2021-02-10T04:54:42.200827+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'39","last_scrub_stamp":"2021-02-10T04:53:38.085565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.085565+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":76,"num_read_kb":59,"num_write":41,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"1319","reported_epoch":"1214","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:42.201790+0000","last_change":"2021-02-10T04:52:32.084961+0000","last_active":"2021-02-10T04:54:42.201790+0000","last_peered":"2021-02-10T04:54:42.201790+0000","last_clean":"2021-02-10T04:52:30.813009+0000","last_became_active":"2021-02-10T04:52:32.084961+0000","last_became_peered":"2021-02-10T04:52:32.084961+0000","last_unstale":"2021-02-10T04:54:42.201790+0000","last_undegraded":"2021-02-10T04:52:32.070468+0000","last_fullsized":"2021-02-10T04:52:32.070358+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:47:08.784593+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:47:08.784593+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"1177","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.328553+0000","last_change":"2021-02-10T04:52:30.019225+0000","last_active":"2021-02-10T04:54:42.328553+0000","last_peered":"2021-02-10T04:54:42.328553+0000","last_clean":"2021-02-10T04:54:42.328553+0000","last_became_active":"2021-02-10T04:50:59.561436+0000","last_became_peered":"2021-02-10T04:50:59.561436+0000","last_unstale":"2021-02-10T04:54:42.328553+0000","last_undegraded":"2021-02-10T04:54:42.328553+0000","last_fullsized":"2021-02-10T04:54:42.328553+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.019102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.019102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"1001'1528","reported_seq":"6162","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.327653+0000","last_change":"2021-02-10T04:52:26.036366+0000","last_active":"2021-02-10T04:54:42.327653+0000","last_peered":"2021-02-10T04:54:42.327653+0000","last_clean":"2021-02-10T04:54:42.327653+0000","last_became_active":"2021-02-10T04:52:26.034028+0000","last_became_peered":"2021-02-10T04:52:26.034028+0000","last_unstale":"2021-02-10T04:54:42.327653+0000","last_undegraded":"2021-02-10T04:54:42.327653+0000","last_fullsized":"2021-02-10T04:54:42.327653+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46944136,"num_objects":16,"num_object_clones":15,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":358,"num_read_kb":351934,"num_write":286,"num_write_kb":111355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":58968832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.15","version":"1000'446","reported_seq":"2151","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.201261+0000","last_change":"2021-02-10T04:53:11.051124+0000","last_active":"2021-02-10T04:54:42.201261+0000","last_peered":"2021-02-10T04:54:42.201261+0000","last_clean":"2021-02-10T04:54:42.201261+0000","last_became_active":"2021-02-10T04:51:40.510899+0000","last_became_peered":"2021-02-10T04:51:40.510899+0000","last_unstale":"2021-02-10T04:54:42.201261+0000","last_undegraded":"2021-02-10T04:54:42.201261+0000","last_fullsized":"2021-02-10T04:54:42.201261+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:53:11.051031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.051031+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24896285,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":18983408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"1048'40","reported_seq":"105","reported_epoch":"1214","state":"active+clean+remapped","last_fresh":"2021-02-10T04:54:41.643609+0000","last_change":"2021-02-10T04:53:32.953859+0000","last_active":"2021-02-10T04:54:41.643609+0000","last_peered":"2021-02-10T04:54:41.643609+0000","last_clean":"2021-02-10T04:54:41.643609+0000","last_became_active":"2021-02-10T04:53:21.308544+0000","last_became_peered":"2021-02-10T04:53:21.308544+0000","last_unstale":"2021-02-10T04:54:41.643609+0000","last_undegraded":"2021-02-10T04:54:41.643609+0000","last_fullsized":"2021-02-10T04:54:41.643609+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'40","last_scrub_stamp":"2021-02-10T04:53:32.953697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:32.953697+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":79,"num_read_kb":57,"num_write":41,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1206,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1,3],"avail_no_missing":["2","1","3"],"object_location_counts":[{"shards":"1,2,3","objects":13}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"116","reported_epoch":"1214","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:41.641823+0000","last_change":"2021-02-10T04:52:32.094875+0000","last_active":"2021-02-10T04:54:41.641823+0000","last_peered":"2021-02-10T04:54:41.641823+0000","last_clean":"2021-02-10T04:52:30.056787+0000","last_became_active":"2021-02-10T04:52:32.094875+0000","last_became_peered":"2021-02-10T04:52:32.094875+0000","last_unstale":"2021-02-10T04:54:41.641823+0000","last_undegraded":"2021-02-10T04:52:32.083818+0000","last_fullsized":"2021-02-10T04:52:32.083707+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:51:08.639784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.639784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"1146","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.329444+0000","last_change":"2021-02-10T04:51:09.475150+0000","last_active":"2021-02-10T04:54:42.329444+0000","last_peered":"2021-02-10T04:54:42.329444+0000","last_clean":"2021-02-10T04:54:42.329444+0000","last_became_active":"2021-02-10T04:50:59.585917+0000","last_became_peered":"2021-02-10T04:50:59.585917+0000","last_unstale":"2021-02-10T04:54:42.329444+0000","last_undegraded":"2021-02-10T04:54:42.329444+0000","last_fullsized":"2021-02-10T04:54:42.329444+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:09.475063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:09.475063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"116","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.643259+0000","last_change":"2021-02-10T04:53:34.927483+0000","last_active":"2021-02-10T04:54:41.643259+0000","last_peered":"2021-02-10T04:54:41.643259+0000","last_clean":"2021-02-10T04:54:41.643259+0000","last_became_active":"2021-02-10T04:51:36.859707+0000","last_became_peered":"2021-02-10T04:51:36.859707+0000","last_unstale":"2021-02-10T04:54:41.643259+0000","last_undegraded":"2021-02-10T04:54:41.643259+0000","last_fullsized":"2021-02-10T04:54:41.643259+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:34.927424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.927424+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"1000'620","reported_seq":"4501","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.328922+0000","last_change":"2021-02-10T04:53:37.384461+0000","last_active":"2021-02-10T04:54:42.328922+0000","last_peered":"2021-02-10T04:54:42.328922+0000","last_clean":"2021-02-10T04:54:42.328922+0000","last_became_active":"2021-02-10T04:51:40.510114+0000","last_became_peered":"2021-02-10T04:51:40.510114+0000","last_unstale":"2021-02-10T04:54:42.328922+0000","last_undegraded":"2021-02-10T04:54:42.328922+0000","last_fullsized":"2021-02-10T04:54:42.328922+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:53:37.384398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:37.384398+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28530390,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"1063'30","reported_seq":"1189","reported_epoch":"1214","state":"active+clean+remapped","last_fresh":"2021-02-10T04:54:41.645640+0000","last_change":"2021-02-10T04:53:21.312523+0000","last_active":"2021-02-10T04:54:41.645640+0000","last_peered":"2021-02-10T04:54:41.645640+0000","last_clean":"2021-02-10T04:54:41.645640+0000","last_became_active":"2021-02-10T04:53:21.302364+0000","last_became_peered":"2021-02-10T04:53:21.302364+0000","last_unstale":"2021-02-10T04:54:41.645640+0000","last_undegraded":"2021-02-10T04:54:41.645640+0000","last_fullsized":"2021-02-10T04:54:41.645640+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"1063'30","last_scrub_stamp":"2021-02-10T04:52:43.102165+0000","last_deep_scrub":"1063'30","last_deep_scrub_stamp":"2021-02-10T04:47:05.589582+0000","last_clean_scrub_stamp":"2021-02-10T04:52:43.102165+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":36,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5,3],"avail_no_missing":["1","3","5"],"object_location_counts":[{"shards":"1,3,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"1203","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.545102+0000","last_change":"2021-02-10T04:52:41.059269+0000","last_active":"2021-02-10T04:54:43.545102+0000","last_peered":"2021-02-10T04:54:43.545102+0000","last_clean":"2021-02-10T04:54:43.545102+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:54:43.545102+0000","last_undegraded":"2021-02-10T04:54:43.545102+0000","last_fullsized":"2021-02-10T04:54:43.545102+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:52:41.059159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.059159+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"1206","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.653364+0000","last_change":"2021-02-10T04:53:12.332890+0000","last_active":"2021-02-10T04:54:41.653364+0000","last_peered":"2021-02-10T04:54:41.653364+0000","last_clean":"2021-02-10T04:54:41.653364+0000","last_became_active":"2021-02-10T04:50:59.540375+0000","last_became_peered":"2021-02-10T04:50:59.540375+0000","last_unstale":"2021-02-10T04:54:41.653364+0000","last_undegraded":"2021-02-10T04:54:41.653364+0000","last_fullsized":"2021-02-10T04:54:41.653364+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.332233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.332233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"127","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.936219+0000","last_change":"2021-02-10T04:54:28.823888+0000","last_active":"2021-02-10T04:54:43.936219+0000","last_peered":"2021-02-10T04:54:43.936219+0000","last_clean":"2021-02-10T04:54:43.936219+0000","last_became_active":"2021-02-10T04:53:46.733191+0000","last_became_peered":"2021-02-10T04:53:46.733191+0000","last_unstale":"2021-02-10T04:54:43.936219+0000","last_undegraded":"2021-02-10T04:54:43.936219+0000","last_fullsized":"2021-02-10T04:54:43.936219+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:45.362537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:45.362537+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"1002'2042","reported_seq":"9189","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.939930+0000","last_change":"2021-02-10T04:53:46.726353+0000","last_active":"2021-02-10T04:54:43.939930+0000","last_peered":"2021-02-10T04:54:43.939930+0000","last_clean":"2021-02-10T04:54:43.939930+0000","last_became_active":"2021-02-10T04:53:46.725828+0000","last_became_peered":"2021-02-10T04:53:46.725828+0000","last_unstale":"2021-02-10T04:54:43.939930+0000","last_undegraded":"2021-02-10T04:54:43.939930+0000","last_fullsized":"2021-02-10T04:54:43.939930+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:52:41.286006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.286006+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46300076,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":545,"num_read_kb":531689,"num_write":399,"num_write_kb":153087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"1049'46","reported_seq":"1370","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.329342+0000","last_change":"2021-02-10T04:52:35.091494+0000","last_active":"2021-02-10T04:54:42.329342+0000","last_peered":"2021-02-10T04:54:42.329342+0000","last_clean":"2021-02-10T04:54:42.329342+0000","last_became_active":"2021-02-10T04:50:59.569915+0000","last_became_peered":"2021-02-10T04:50:59.569915+0000","last_unstale":"2021-02-10T04:54:42.329342+0000","last_undegraded":"2021-02-10T04:54:42.329342+0000","last_fullsized":"2021-02-10T04:54:42.329342+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'46","last_scrub_stamp":"2021-02-10T04:52:35.091374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:35.091374+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":65,"num_write":46,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"84'7","reported_seq":"1230","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.646785+0000","last_change":"2021-02-10T04:51:52.498251+0000","last_active":"2021-02-10T04:54:41.646785+0000","last_peered":"2021-02-10T04:54:41.646785+0000","last_clean":"2021-02-10T04:54:41.646785+0000","last_became_active":"2021-02-10T04:50:59.575924+0000","last_became_peered":"2021-02-10T04:50:59.575924+0000","last_unstale":"2021-02-10T04:54:41.646785+0000","last_undegraded":"2021-02-10T04:54:41.646785+0000","last_fullsized":"2021-02-10T04:54:41.646785+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'7","last_scrub_stamp":"2021-02-10T04:51:52.498158+0000","last_deep_scrub":"84'7","last_deep_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.498158+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"1148'3","reported_seq":"1542","reported_epoch":"1215","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:43.547921+0000","last_change":"2021-02-10T04:52:32.094670+0000","last_active":"2021-02-10T04:54:43.547921+0000","last_peered":"2021-02-10T04:54:43.547921+0000","last_clean":"2021-02-10T04:52:30.059221+0000","last_became_active":"2021-02-10T04:52:32.094670+0000","last_became_peered":"2021-02-10T04:52:32.094670+0000","last_unstale":"2021-02-10T04:54:43.547921+0000","last_undegraded":"2021-02-10T04:52:32.078919+0000","last_fullsized":"2021-02-10T04:52:32.078798+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:47.407557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:47.407557+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"1124","reported_epoch":"1215","state":"active+undersized","last_fresh":"2021-02-10T04:54:43.939932+0000","last_change":"2021-02-10T04:53:46.727163+0000","last_active":"2021-02-10T04:54:43.939932+0000","last_peered":"2021-02-10T04:54:43.939932+0000","last_clean":"2021-02-10T04:49:56.939276+0000","last_became_active":"2021-02-10T04:53:46.727163+0000","last_became_peered":"2021-02-10T04:53:46.727163+0000","last_unstale":"2021-02-10T04:54:43.939932+0000","last_undegraded":"2021-02-10T04:54:43.939932+0000","last_fullsized":"2021-02-10T04:53:46.702239+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:23.566027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_clean_scrub_stamp":"2021-02-10T04:52:23.566027+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.16","version":"1000'490","reported_seq":"3721","reported_epoch":"1214","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:42.201713+0000","last_change":"2021-02-10T04:52:32.077228+0000","last_active":"2021-02-10T04:54:42.201713+0000","last_peered":"2021-02-10T04:54:42.201713+0000","last_clean":"2021-02-10T04:52:30.062406+0000","last_became_active":"2021-02-10T04:52:32.077228+0000","last_became_peered":"2021-02-10T04:52:32.077228+0000","last_unstale":"2021-02-10T04:54:42.201713+0000","last_undegraded":"2021-02-10T04:52:32.071522+0000","last_fullsized":"2021-02-10T04:52:32.071453+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1013,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:50:03.756248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:50:03.756248+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16154790,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"1112'40","reported_seq":"1274","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.653167+0000","last_change":"2021-02-10T04:53:10.276449+0000","last_active":"2021-02-10T04:54:41.653167+0000","last_peered":"2021-02-10T04:54:41.653167+0000","last_clean":"2021-02-10T04:54:41.653167+0000","last_became_active":"2021-02-10T04:50:59.539529+0000","last_became_peered":"2021-02-10T04:50:59.539529+0000","last_unstale":"2021-02-10T04:54:41.653167+0000","last_undegraded":"2021-02-10T04:54:41.653167+0000","last_fullsized":"2021-02-10T04:54:41.653167+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1112'40","last_scrub_stamp":"2021-02-10T04:53:10.276358+0000","last_deep_scrub":"1112'40","last_deep_scrub_stamp":"2021-02-10T04:51:01.450050+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.276358+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":59,"num_read_kb":44,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"1178","reported_epoch":"1214","state":"active+undersized","last_fresh":"2021-02-10T04:54:42.201172+0000","last_change":"2021-02-10T04:52:32.080868+0000","last_active":"2021-02-10T04:54:42.201172+0000","last_peered":"2021-02-10T04:54:42.201172+0000","last_clean":"2021-02-10T04:49:56.937464+0000","last_became_active":"2021-02-10T04:52:32.080868+0000","last_became_peered":"2021-02-10T04:52:32.080868+0000","last_unstale":"2021-02-10T04:54:42.201172+0000","last_undegraded":"2021-02-10T04:54:42.201172+0000","last_fullsized":"2021-02-10T04:52:32.068376+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:23.904210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:47:23.904210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"1119'6","reported_seq":"1465","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:43.028177+0000","last_change":"2021-02-10T04:52:40.038646+0000","last_active":"2021-02-10T04:54:43.028177+0000","last_peered":"2021-02-10T04:54:43.028177+0000","last_clean":"2021-02-10T04:54:43.028177+0000","last_became_active":"2021-02-10T04:51:36.869733+0000","last_became_peered":"2021-02-10T04:51:36.869733+0000","last_unstale":"2021-02-10T04:54:43.028177+0000","last_undegraded":"2021-02-10T04:54:43.028177+0000","last_fullsized":"2021-02-10T04:54:43.028177+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'6","last_scrub_stamp":"2021-02-10T04:52:40.038554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:40.038554+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"1200","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.201956+0000","last_change":"2021-02-10T04:51:36.849933+0000","last_active":"2021-02-10T04:54:42.201956+0000","last_peered":"2021-02-10T04:54:42.201956+0000","last_clean":"2021-02-10T04:54:42.201956+0000","last_became_active":"2021-02-10T04:51:36.849184+0000","last_became_peered":"2021-02-10T04:51:36.849184+0000","last_unstale":"2021-02-10T04:54:42.201956+0000","last_undegraded":"2021-02-10T04:54:42.201956+0000","last_fullsized":"2021-02-10T04:54:42.201956+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:11.779779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:11.779779+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"1032'89","reported_seq":"810","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.934317+0000","last_change":"2021-02-10T04:53:46.728868+0000","last_active":"2021-02-10T04:54:43.934317+0000","last_peered":"2021-02-10T04:54:43.934317+0000","last_clean":"2021-02-10T04:54:43.934317+0000","last_became_active":"2021-02-10T04:53:46.728356+0000","last_became_peered":"2021-02-10T04:53:46.728356+0000","last_unstale":"2021-02-10T04:54:43.934317+0000","last_undegraded":"2021-02-10T04:54:43.934317+0000","last_fullsized":"2021-02-10T04:54:43.934317+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":5,"last_scrub":"1032'89","last_scrub_stamp":"2021-02-10T04:52:34.089789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:34.089789+0000","log_size":89,"ondisk_log_size":89,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5218574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":30326,"num_write":21,"num_write_kb":12444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3210022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"1163'37","reported_seq":"896","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.644772+0000","last_change":"2021-02-10T04:53:14.202505+0000","last_active":"2021-02-10T04:54:41.644772+0000","last_peered":"2021-02-10T04:54:41.644772+0000","last_clean":"2021-02-10T04:54:41.644772+0000","last_became_active":"2021-02-10T04:51:49.496294+0000","last_became_peered":"2021-02-10T04:51:49.496294+0000","last_unstale":"2021-02-10T04:54:41.644772+0000","last_undegraded":"2021-02-10T04:54:41.644772+0000","last_fullsized":"2021-02-10T04:54:41.644772+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1163'37","last_scrub_stamp":"2021-02-10T04:53:14.202445+0000","last_deep_scrub":"1163'37","last_deep_scrub_stamp":"2021-02-10T04:53:14.202445+0000","last_clean_scrub_stamp":"2021-02-10T04:53:14.202445+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":33,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"1216","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.545237+0000","last_change":"2021-02-10T04:52:23.095651+0000","last_active":"2021-02-10T04:54:43.545237+0000","last_peered":"2021-02-10T04:54:43.545237+0000","last_clean":"2021-02-10T04:54:43.545237+0000","last_became_active":"2021-02-10T04:50:59.543531+0000","last_became_peered":"2021-02-10T04:50:59.543531+0000","last_unstale":"2021-02-10T04:54:43.545237+0000","last_undegraded":"2021-02-10T04:54:43.545237+0000","last_fullsized":"2021-02-10T04:54:43.545237+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:23.095588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:23.095588+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"1156","reported_epoch":"1214","state":"active+undersized","last_fresh":"2021-02-10T04:54:42.328356+0000","last_change":"2021-02-10T04:52:32.085370+0000","last_active":"2021-02-10T04:54:42.328356+0000","last_peered":"2021-02-10T04:54:42.328356+0000","last_clean":"2021-02-10T04:52:30.067066+0000","last_became_active":"2021-02-10T04:52:32.085370+0000","last_became_peered":"2021-02-10T04:52:32.085370+0000","last_unstale":"2021-02-10T04:54:42.328356+0000","last_undegraded":"2021-02-10T04:54:42.328356+0000","last_fullsized":"2021-02-10T04:52:32.074435+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:43.663762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:51:43.663762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"76","reported_epoch":"1214","state":"active+undersized","last_fresh":"2021-02-10T04:54:41.644300+0000","last_change":"2021-02-10T04:52:32.091922+0000","last_active":"2021-02-10T04:54:41.644300+0000","last_peered":"2021-02-10T04:54:41.644300+0000","last_clean":"2021-02-10T04:25:52.015519+0000","last_became_active":"2021-02-10T04:52:32.091922+0000","last_became_peered":"2021-02-10T04:52:32.091922+0000","last_unstale":"2021-02-10T04:54:41.644300+0000","last_undegraded":"2021-02-10T04:54:41.644300+0000","last_fullsized":"2021-02-10T04:52:32.069065+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:57.568541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:57.568541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.10","version":"1000'1071","reported_seq":"4720","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.327738+0000","last_change":"2021-02-10T04:53:35.302879+0000","last_active":"2021-02-10T04:54:42.327738+0000","last_peered":"2021-02-10T04:54:42.327738+0000","last_clean":"2021-02-10T04:54:42.327738+0000","last_became_active":"2021-02-10T04:51:39.508551+0000","last_became_peered":"2021-02-10T04:51:39.508551+0000","last_unstale":"2021-02-10T04:54:42.327738+0000","last_undegraded":"2021-02-10T04:54:42.327738+0000","last_fullsized":"2021-02-10T04:54:42.327738+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:35.302780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.302780+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"1049'31","reported_seq":"1237","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.201510+0000","last_change":"2021-02-10T04:52:39.695340+0000","last_active":"2021-02-10T04:54:42.201510+0000","last_peered":"2021-02-10T04:54:42.201510+0000","last_clean":"2021-02-10T04:54:42.201510+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:54:42.201510+0000","last_undegraded":"2021-02-10T04:54:42.201510+0000","last_fullsized":"2021-02-10T04:54:42.201510+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'31","last_scrub_stamp":"2021-02-10T04:52:39.695237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:39.695237+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":41,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"446","reported_epoch":"1214","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:41.645436+0000","last_change":"2021-02-10T04:52:32.094351+0000","last_active":"2021-02-10T04:54:41.645436+0000","last_peered":"2021-02-10T04:54:41.645436+0000","last_clean":"2021-02-10T04:52:30.061022+0000","last_became_active":"2021-02-10T04:52:32.094351+0000","last_became_peered":"2021-02-10T04:52:32.094351+0000","last_unstale":"2021-02-10T04:54:41.645436+0000","last_undegraded":"2021-02-10T04:52:32.074350+0000","last_fullsized":"2021-02-10T04:52:32.074239+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:47:08.700069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:08.700069+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"868","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.644939+0000","last_change":"2021-02-10T04:53:35.216489+0000","last_active":"2021-02-10T04:54:41.644939+0000","last_peered":"2021-02-10T04:54:41.644939+0000","last_clean":"2021-02-10T04:54:41.644939+0000","last_became_active":"2021-02-10T04:51:49.501254+0000","last_became_peered":"2021-02-10T04:51:49.501254+0000","last_unstale":"2021-02-10T04:54:41.644939+0000","last_undegraded":"2021-02-10T04:54:41.644939+0000","last_fullsized":"2021-02-10T04:54:41.644939+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.216394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.216394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"1183","reported_epoch":"1214","state":"active+undersized","last_fresh":"2021-02-10T04:54:42.327830+0000","last_change":"2021-02-10T04:52:32.087218+0000","last_active":"2021-02-10T04:54:42.327830+0000","last_peered":"2021-02-10T04:54:42.327830+0000","last_clean":"2021-02-10T04:52:30.062886+0000","last_became_active":"2021-02-10T04:52:32.087218+0000","last_became_peered":"2021-02-10T04:52:32.087218+0000","last_unstale":"2021-02-10T04:54:42.327830+0000","last_undegraded":"2021-02-10T04:54:42.327830+0000","last_fullsized":"2021-02-10T04:52:32.072167+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:55.814486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.814486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"1002'1657","reported_seq":"4949","reported_epoch":"1215","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:43.935591+0000","last_change":"2021-02-10T04:53:46.722538+0000","last_active":"2021-02-10T04:54:43.935591+0000","last_peered":"2021-02-10T04:54:43.935591+0000","last_clean":"2021-02-10T04:44:25.563579+0000","last_became_active":"2021-02-10T04:53:46.722538+0000","last_became_peered":"2021-02-10T04:53:46.722538+0000","last_unstale":"2021-02-10T04:54:43.935591+0000","last_undegraded":"2021-02-10T04:53:46.700604+0000","last_fullsized":"2021-02-10T04:53:46.700504+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1124,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:46:54.881412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:54.881412+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":42793067,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":424,"num_read_kb":437674,"num_write":319,"num_write_kb":118798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":17936938,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":17}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"1048'20","reported_seq":"1316","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.327991+0000","last_change":"2021-02-10T04:53:04.058324+0000","last_active":"2021-02-10T04:54:42.327991+0000","last_peered":"2021-02-10T04:54:42.327991+0000","last_clean":"2021-02-10T04:54:42.327991+0000","last_became_active":"2021-02-10T04:50:59.544484+0000","last_became_peered":"2021-02-10T04:50:59.544484+0000","last_unstale":"2021-02-10T04:54:42.327991+0000","last_undegraded":"2021-02-10T04:54:42.327991+0000","last_fullsized":"2021-02-10T04:54:42.327991+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'20","last_scrub_stamp":"2021-02-10T04:53:04.058248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:04.058248+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":20,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"1218","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.653758+0000","last_change":"2021-02-10T04:53:00.343307+0000","last_active":"2021-02-10T04:54:41.653758+0000","last_peered":"2021-02-10T04:54:41.653758+0000","last_clean":"2021-02-10T04:54:41.653758+0000","last_became_active":"2021-02-10T04:51:36.844573+0000","last_became_peered":"2021-02-10T04:51:36.844573+0000","last_unstale":"2021-02-10T04:54:41.653758+0000","last_undegraded":"2021-02-10T04:54:41.653758+0000","last_fullsized":"2021-02-10T04:54:41.653758+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:00.343228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:00.343228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"1187","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.545055+0000","last_change":"2021-02-10T04:50:56.099710+0000","last_active":"2021-02-10T04:54:43.545055+0000","last_peered":"2021-02-10T04:54:43.545055+0000","last_clean":"2021-02-10T04:54:43.545055+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:54:43.545055+0000","last_undegraded":"2021-02-10T04:54:43.545055+0000","last_fullsized":"2021-02-10T04:54:43.545055+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:56.099651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:50:56.099651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"535","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.935438+0000","last_change":"2021-02-10T04:53:46.718715+0000","last_active":"2021-02-10T04:54:43.935438+0000","last_peered":"2021-02-10T04:54:43.935438+0000","last_clean":"2021-02-10T04:54:43.935438+0000","last_became_active":"2021-02-10T04:53:46.718266+0000","last_became_peered":"2021-02-10T04:53:46.718266+0000","last_unstale":"2021-02-10T04:54:43.935438+0000","last_undegraded":"2021-02-10T04:54:43.935438+0000","last_fullsized":"2021-02-10T04:54:43.935438+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.876681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.876681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"998'1009","reported_seq":"5054","reported_epoch":"1215","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:43.545023+0000","last_change":"2021-02-10T04:52:32.090443+0000","last_active":"2021-02-10T04:54:43.545023+0000","last_peered":"2021-02-10T04:54:43.545023+0000","last_clean":"2021-02-10T04:52:30.061024+0000","last_became_active":"2021-02-10T04:52:32.090443+0000","last_became_peered":"2021-02-10T04:52:32.090443+0000","last_unstale":"2021-02-10T04:54:43.545023+0000","last_undegraded":"2021-02-10T04:52:32.076916+0000","last_fullsized":"2021-02-10T04:52:32.076790+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1127,"parent":"0.0","parent_split_bits":5,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:13.032506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:13.032506+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49420827,"num_objects":21,"num_object_clones":19,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":36763798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":21}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"1049'37","reported_seq":"1244","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.328195+0000","last_change":"2021-02-10T04:51:44.694779+0000","last_active":"2021-02-10T04:54:42.328195+0000","last_peered":"2021-02-10T04:54:42.328195+0000","last_clean":"2021-02-10T04:54:42.328195+0000","last_became_active":"2021-02-10T04:50:59.549412+0000","last_became_peered":"2021-02-10T04:50:59.549412+0000","last_unstale":"2021-02-10T04:54:42.328195+0000","last_undegraded":"2021-02-10T04:54:42.328195+0000","last_fullsized":"2021-02-10T04:54:42.328195+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:44.694711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.694711+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":64,"num_read_kb":50,"num_write":37,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"1354","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.546608+0000","last_change":"2021-02-10T04:52:33.088525+0000","last_active":"2021-02-10T04:54:43.546608+0000","last_peered":"2021-02-10T04:54:43.546608+0000","last_clean":"2021-02-10T04:54:43.546608+0000","last_became_active":"2021-02-10T04:51:36.870573+0000","last_became_peered":"2021-02-10T04:51:36.870573+0000","last_unstale":"2021-02-10T04:54:43.546608+0000","last_undegraded":"2021-02-10T04:54:43.546608+0000","last_fullsized":"2021-02-10T04:54:43.546608+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:52:33.088428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:33.088428+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"500","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.328005+0000","last_change":"2021-02-10T04:51:51.836723+0000","last_active":"2021-02-10T04:54:42.328005+0000","last_peered":"2021-02-10T04:54:42.328005+0000","last_clean":"2021-02-10T04:54:42.328005+0000","last_became_active":"2021-02-10T04:50:59.564977+0000","last_became_peered":"2021-02-10T04:50:59.564977+0000","last_unstale":"2021-02-10T04:54:42.328005+0000","last_undegraded":"2021-02-10T04:54:42.328005+0000","last_fullsized":"2021-02-10T04:54:42.328005+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.836638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.836638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"1147","reported_epoch":"1214","state":"active+undersized","last_fresh":"2021-02-10T04:54:42.328348+0000","last_change":"2021-02-10T04:52:32.081484+0000","last_active":"2021-02-10T04:54:42.328348+0000","last_peered":"2021-02-10T04:54:42.328348+0000","last_clean":"2021-02-10T04:52:30.067390+0000","last_became_active":"2021-02-10T04:52:32.081484+0000","last_became_peered":"2021-02-10T04:52:32.081484+0000","last_unstale":"2021-02-10T04:54:42.328348+0000","last_undegraded":"2021-02-10T04:54:42.328348+0000","last_fullsized":"2021-02-10T04:52:32.074112+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:58.514318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_clean_scrub_stamp":"2021-02-10T04:46:58.514318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1d","version":"1001'1165","reported_seq":"4207","reported_epoch":"1214","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:41.645972+0000","last_change":"2021-02-10T04:52:32.087778+0000","last_active":"2021-02-10T04:54:41.645972+0000","last_peered":"2021-02-10T04:54:41.645972+0000","last_clean":"2021-02-10T04:52:30.061103+0000","last_became_active":"2021-02-10T04:52:32.087778+0000","last_became_peered":"2021-02-10T04:52:32.087778+0000","last_unstale":"2021-02-10T04:54:41.645972+0000","last_undegraded":"2021-02-10T04:52:32.074694+0000","last_fullsized":"2021-02-10T04:52:32.074582+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1144,"parent":"0.0","parent_split_bits":5,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29601845,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":287,"num_read_kb":299530,"num_write":200,"num_write_kb":89131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":13}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"1259","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.644211+0000","last_change":"2021-02-10T04:53:01.331566+0000","last_active":"2021-02-10T04:54:41.644211+0000","last_peered":"2021-02-10T04:54:41.644211+0000","last_clean":"2021-02-10T04:54:41.644211+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:54:41.644211+0000","last_undegraded":"2021-02-10T04:54:41.644211+0000","last_fullsized":"2021-02-10T04:54:41.644211+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:01.331466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.331466+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"1098'3","reported_seq":"1543","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:48.029277+0000","last_change":"2021-02-10T04:53:34.865668+0000","last_active":"2021-02-10T04:54:48.029277+0000","last_peered":"2021-02-10T04:54:48.029277+0000","last_clean":"2021-02-10T04:54:48.029277+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:54:48.029277+0000","last_undegraded":"2021-02-10T04:54:48.029277+0000","last_fullsized":"2021-02-10T04:54:48.029277+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"1098'3","last_scrub_stamp":"2021-02-10T04:53:34.865579+0000","last_deep_scrub":"1098'3","last_deep_scrub_stamp":"2021-02-10T04:50:15.255137+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.865579+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"1065","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.936416+0000","last_change":"2021-02-10T04:53:46.727603+0000","last_active":"2021-02-10T04:54:43.936416+0000","last_peered":"2021-02-10T04:54:43.936416+0000","last_clean":"2021-02-10T04:54:43.936416+0000","last_became_active":"2021-02-10T04:53:46.727053+0000","last_became_peered":"2021-02-10T04:53:46.727053+0000","last_unstale":"2021-02-10T04:54:43.936416+0000","last_undegraded":"2021-02-10T04:54:43.936416+0000","last_fullsized":"2021-02-10T04:54:43.936416+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:47.493057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.493057+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1c","version":"1002'1090","reported_seq":"5239","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.651934+0000","last_change":"2021-02-10T04:53:41.353991+0000","last_active":"2021-02-10T04:54:41.651934+0000","last_peered":"2021-02-10T04:54:41.651934+0000","last_clean":"2021-02-10T04:54:41.651934+0000","last_became_active":"2021-02-10T04:52:31.088265+0000","last_became_peered":"2021-02-10T04:52:31.088265+0000","last_unstale":"2021-02-10T04:54:41.651934+0000","last_undegraded":"2021-02-10T04:54:41.651934+0000","last_fullsized":"2021-02-10T04:54:41.651934+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:53:41.353840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:41.353840+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"1151","reported_epoch":"1214","state":"active+undersized","last_fresh":"2021-02-10T04:54:42.328058+0000","last_change":"2021-02-10T04:52:32.079189+0000","last_active":"2021-02-10T04:54:42.328058+0000","last_peered":"2021-02-10T04:54:42.328058+0000","last_clean":"2021-02-10T04:52:30.063079+0000","last_became_active":"2021-02-10T04:52:32.079189+0000","last_became_peered":"2021-02-10T04:52:32.079189+0000","last_unstale":"2021-02-10T04:54:42.328058+0000","last_undegraded":"2021-02-10T04:54:42.328058+0000","last_fullsized":"2021-02-10T04:52:32.071674+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:42.537443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:42.537443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"1136","reported_epoch":"1215","state":"active+undersized","last_fresh":"2021-02-10T04:54:43.940043+0000","last_change":"2021-02-10T04:53:46.721135+0000","last_active":"2021-02-10T04:54:43.940043+0000","last_peered":"2021-02-10T04:54:43.940043+0000","last_clean":"2021-02-10T04:51:27.994110+0000","last_became_active":"2021-02-10T04:53:46.721135+0000","last_became_peered":"2021-02-10T04:53:46.721135+0000","last_unstale":"2021-02-10T04:54:43.940043+0000","last_undegraded":"2021-02-10T04:54:43.940043+0000","last_fullsized":"2021-02-10T04:53:46.702702+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.938731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.938731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"425","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.645152+0000","last_change":"2021-02-10T04:53:15.185748+0000","last_active":"2021-02-10T04:54:41.645152+0000","last_peered":"2021-02-10T04:54:41.645152+0000","last_clean":"2021-02-10T04:54:41.645152+0000","last_became_active":"2021-02-10T04:51:49.499348+0000","last_became_peered":"2021-02-10T04:51:49.499348+0000","last_unstale":"2021-02-10T04:54:41.645152+0000","last_undegraded":"2021-02-10T04:54:41.645152+0000","last_fullsized":"2021-02-10T04:54:41.645152+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:15.185686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:15.185686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1f","version":"1004'1093","reported_seq":"6477","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.546173+0000","last_change":"2021-02-10T04:52:31.089952+0000","last_active":"2021-02-10T04:54:43.546173+0000","last_peered":"2021-02-10T04:54:43.546173+0000","last_clean":"2021-02-10T04:54:43.546173+0000","last_became_active":"2021-02-10T04:52:31.088634+0000","last_became_peered":"2021-02-10T04:52:31.088634+0000","last_unstale":"2021-02-10T04:54:43.546173+0000","last_undegraded":"2021-02-10T04:54:43.546173+0000","last_fullsized":"2021-02-10T04:54:43.546173+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39434697,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":279,"num_read_kb":311058,"num_write":186,"num_write_kb":66790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":53995913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"1204","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.545390+0000","last_change":"2021-02-10T04:52:36.083849+0000","last_active":"2021-02-10T04:54:43.545390+0000","last_peered":"2021-02-10T04:54:43.545390+0000","last_clean":"2021-02-10T04:54:43.545390+0000","last_became_active":"2021-02-10T04:25:36.336442+0000","last_became_peered":"2021-02-10T04:25:36.336442+0000","last_unstale":"2021-02-10T04:54:43.545390+0000","last_undegraded":"2021-02-10T04:54:43.545390+0000","last_fullsized":"2021-02-10T04:54:43.545390+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:52:36.083784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:36.083784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"428","reported_epoch":"1214","state":"active+undersized","last_fresh":"2021-02-10T04:54:41.645735+0000","last_change":"2021-02-10T04:52:32.084893+0000","last_active":"2021-02-10T04:54:41.645735+0000","last_peered":"2021-02-10T04:54:41.645735+0000","last_clean":"2021-02-10T04:52:30.061372+0000","last_became_active":"2021-02-10T04:52:32.084893+0000","last_became_peered":"2021-02-10T04:52:32.084893+0000","last_unstale":"2021-02-10T04:54:41.645735+0000","last_undegraded":"2021-02-10T04:54:41.645735+0000","last_fullsized":"2021-02-10T04:52:32.073304+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:08.758129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.758129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"1182","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.201498+0000","last_change":"2021-02-10T04:51:44.799512+0000","last_active":"2021-02-10T04:54:42.201498+0000","last_peered":"2021-02-10T04:54:42.201498+0000","last_clean":"2021-02-10T04:54:42.201498+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:54:42.201498+0000","last_undegraded":"2021-02-10T04:54:42.201498+0000","last_fullsized":"2021-02-10T04:54:42.201498+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:44.799429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.799429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1e","version":"1063'654","reported_seq":"4458","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.645733+0000","last_change":"2021-02-10T04:53:40.135450+0000","last_active":"2021-02-10T04:54:41.645733+0000","last_peered":"2021-02-10T04:54:41.645733+0000","last_clean":"2021-02-10T04:54:41.645733+0000","last_became_active":"2021-02-10T04:52:31.087662+0000","last_became_peered":"2021-02-10T04:52:31.087662+0000","last_unstale":"2021-02-10T04:54:41.645733+0000","last_undegraded":"2021-02-10T04:54:41.645733+0000","last_fullsized":"2021-02-10T04:54:41.645733+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:53:40.135367+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:53:40.135367+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"1166","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.328638+0000","last_change":"2021-02-10T04:51:08.006463+0000","last_active":"2021-02-10T04:54:42.328638+0000","last_peered":"2021-02-10T04:54:42.328638+0000","last_clean":"2021-02-10T04:54:42.328638+0000","last_became_active":"2021-02-10T04:50:59.563657+0000","last_became_peered":"2021-02-10T04:50:59.563657+0000","last_unstale":"2021-02-10T04:54:42.328638+0000","last_undegraded":"2021-02-10T04:54:42.328638+0000","last_fullsized":"2021-02-10T04:54:42.328638+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_clean_scrub_stamp":"2021-02-10T04:51:02.455702+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"1217","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.201052+0000","last_change":"2021-02-10T04:52:33.741801+0000","last_active":"2021-02-10T04:54:42.201052+0000","last_peered":"2021-02-10T04:54:42.201052+0000","last_clean":"2021-02-10T04:54:42.201052+0000","last_became_active":"2021-02-10T04:51:36.860129+0000","last_became_peered":"2021-02-10T04:51:36.860129+0000","last_unstale":"2021-02-10T04:54:42.201052+0000","last_undegraded":"2021-02-10T04:54:42.201052+0000","last_fullsized":"2021-02-10T04:54:42.201052+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:33.741711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:46:58.666264+0000","last_clean_scrub_stamp":"2021-02-10T04:52:33.741711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"1178","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.329313+0000","last_change":"2021-02-10T04:52:25.030419+0000","last_active":"2021-02-10T04:54:42.329313+0000","last_peered":"2021-02-10T04:54:42.329313+0000","last_clean":"2021-02-10T04:54:42.329313+0000","last_became_active":"2021-02-10T04:50:59.539582+0000","last_became_peered":"2021-02-10T04:50:59.539582+0000","last_unstale":"2021-02-10T04:54:42.329313+0000","last_undegraded":"2021-02-10T04:54:42.329313+0000","last_fullsized":"2021-02-10T04:54:42.329313+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:25.030281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.030281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"1003'739","reported_seq":"3964","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.544747+0000","last_change":"2021-02-10T04:53:01.929261+0000","last_active":"2021-02-10T04:54:43.544747+0000","last_peered":"2021-02-10T04:54:43.544747+0000","last_clean":"2021-02-10T04:54:43.544747+0000","last_became_active":"2021-02-10T04:51:40.511505+0000","last_became_peered":"2021-02-10T04:51:40.511505+0000","last_unstale":"2021-02-10T04:54:43.544747+0000","last_undegraded":"2021-02-10T04:54:43.544747+0000","last_fullsized":"2021-02-10T04:54:43.544747+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:53:01.929169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.929169+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22230846,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":18429397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"1112'47","reported_seq":"102","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.544743+0000","last_change":"2021-02-10T04:53:21.334003+0000","last_active":"2021-02-10T04:54:43.544743+0000","last_peered":"2021-02-10T04:54:43.544743+0000","last_clean":"2021-02-10T04:54:43.544743+0000","last_became_active":"2021-02-10T04:53:20.297145+0000","last_became_peered":"2021-02-10T04:53:20.297145+0000","last_unstale":"2021-02-10T04:54:43.544743+0000","last_undegraded":"2021-02-10T04:54:43.544743+0000","last_fullsized":"2021-02-10T04:54:43.544743+0000","mapping_epoch":1166,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"1112'47","last_scrub_stamp":"2021-02-10T04:51:02.860835+0000","last_deep_scrub":"1112'47","last_deep_scrub_stamp":"2021-02-10T04:51:02.860835+0000","last_clean_scrub_stamp":"2021-02-10T04:51:02.860835+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":71,"num_read_kb":55,"num_write":38,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"1056","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.936144+0000","last_change":"2021-02-10T04:53:46.727590+0000","last_active":"2021-02-10T04:54:43.936144+0000","last_peered":"2021-02-10T04:54:43.936144+0000","last_clean":"2021-02-10T04:54:43.936144+0000","last_became_active":"2021-02-10T04:53:46.726584+0000","last_became_peered":"2021-02-10T04:53:46.726584+0000","last_unstale":"2021-02-10T04:54:43.936144+0000","last_undegraded":"2021-02-10T04:54:43.936144+0000","last_fullsized":"2021-02-10T04:54:43.936144+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:26.408326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.408326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"1215","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.652470+0000","last_change":"2021-02-10T04:53:07.324650+0000","last_active":"2021-02-10T04:54:41.652470+0000","last_peered":"2021-02-10T04:54:41.652470+0000","last_clean":"2021-02-10T04:54:41.652470+0000","last_became_active":"2021-02-10T04:50:59.575340+0000","last_became_peered":"2021-02-10T04:50:59.575340+0000","last_unstale":"2021-02-10T04:54:41.652470+0000","last_undegraded":"2021-02-10T04:54:41.652470+0000","last_fullsized":"2021-02-10T04:54:41.652470+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.324564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.324564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"75","reported_epoch":"1214","state":"active+undersized","last_fresh":"2021-02-10T04:54:41.643909+0000","last_change":"2021-02-10T04:52:32.091240+0000","last_active":"2021-02-10T04:54:41.643909+0000","last_peered":"2021-02-10T04:54:41.643909+0000","last_clean":"2021-02-10T04:25:52.015519+0000","last_became_active":"2021-02-10T04:52:32.091240+0000","last_became_peered":"2021-02-10T04:52:32.091240+0000","last_unstale":"2021-02-10T04:54:41.643909+0000","last_undegraded":"2021-02-10T04:54:41.643909+0000","last_fullsized":"2021-02-10T04:52:32.077955+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:00.501127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:00.501127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.18","version":"259'74","reported_seq":"78","reported_epoch":"1215","state":"active+undersized+degraded","last_fresh":"2021-02-10T04:54:43.544583+0000","last_change":"2021-02-10T04:52:32.083683+0000","last_active":"2021-02-10T04:54:43.544583+0000","last_peered":"2021-02-10T04:54:43.544583+0000","last_clean":"2021-02-10T04:31:15.165244+0000","last_became_active":"2021-02-10T04:52:32.083683+0000","last_became_peered":"2021-02-10T04:52:32.083683+0000","last_unstale":"2021-02-10T04:54:43.544583+0000","last_undegraded":"2021-02-10T04:52:32.067776+0000","last_fullsized":"2021-02-10T04:52:32.067560+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1010,"parent":"0.0","parent_split_bits":5,"last_scrub":"259'74","last_scrub_stamp":"2021-02-10T04:52:26.597668+0000","last_deep_scrub":"259'74","last_deep_scrub_stamp":"2021-02-10T04:46:49.745104+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.597668+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1218524,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":28228,"num_write":19,"num_write_kb":7022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"1061'20","reported_seq":"1117","reported_epoch":"1214","state":"active+clean+remapped","last_fresh":"2021-02-10T04:54:41.644809+0000","last_change":"2021-02-10T04:53:21.301748+0000","last_active":"2021-02-10T04:54:41.644809+0000","last_peered":"2021-02-10T04:54:41.644809+0000","last_clean":"2021-02-10T04:54:41.644809+0000","last_became_active":"2021-02-10T04:53:21.301122+0000","last_became_peered":"2021-02-10T04:53:21.301122+0000","last_unstale":"2021-02-10T04:54:41.644809+0000","last_undegraded":"2021-02-10T04:54:41.644809+0000","last_fullsized":"2021-02-10T04:54:41.644809+0000","mapping_epoch":1167,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1168,"parent":"0.0","parent_split_bits":0,"last_scrub":"1061'20","last_scrub_stamp":"2021-02-10T04:52:31.770907+0000","last_deep_scrub":"1061'20","last_deep_scrub_stamp":"2021-02-10T04:46:55.646261+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.770907+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":17,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,3],"avail_no_missing":["1","3","4"],"object_location_counts":[{"shards":"1,3,4","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"1078","reported_epoch":"1215","state":"active+undersized","last_fresh":"2021-02-10T04:54:43.936310+0000","last_change":"2021-02-10T04:53:46.724425+0000","last_active":"2021-02-10T04:54:43.936310+0000","last_peered":"2021-02-10T04:54:43.936310+0000","last_clean":"2021-02-10T04:52:30.068110+0000","last_became_active":"2021-02-10T04:53:46.724425+0000","last_became_peered":"2021-02-10T04:53:46.724425+0000","last_unstale":"2021-02-10T04:54:43.936310+0000","last_undegraded":"2021-02-10T04:54:43.936310+0000","last_fullsized":"2021-02-10T04:53:46.707467+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:59.167911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:50:59.167911+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"126","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.935155+0000","last_change":"2021-02-10T04:53:46.718512+0000","last_active":"2021-02-10T04:54:43.935155+0000","last_peered":"2021-02-10T04:54:43.935155+0000","last_clean":"2021-02-10T04:54:43.935155+0000","last_became_active":"2021-02-10T04:53:46.716811+0000","last_became_peered":"2021-02-10T04:53:46.716811+0000","last_unstale":"2021-02-10T04:54:43.935155+0000","last_undegraded":"2021-02-10T04:54:43.935155+0000","last_fullsized":"2021-02-10T04:54:43.935155+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:45.813676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.813676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"1204","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.644351+0000","last_change":"2021-02-10T04:53:38.363899+0000","last_active":"2021-02-10T04:54:41.644351+0000","last_peered":"2021-02-10T04:54:41.644351+0000","last_clean":"2021-02-10T04:54:41.644351+0000","last_became_active":"2021-02-10T04:50:59.539256+0000","last_became_peered":"2021-02-10T04:50:59.539256+0000","last_unstale":"2021-02-10T04:54:41.644351+0000","last_undegraded":"2021-02-10T04:54:41.644351+0000","last_fullsized":"2021-02-10T04:54:41.644351+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:38.363825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:09.433793+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.363825+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1b","version":"1003'1689","reported_seq":"4645","reported_epoch":"1214","state":"active+clean+remapped","last_fresh":"2021-02-10T04:54:41.642267+0000","last_change":"2021-02-10T04:53:10.075976+0000","last_active":"2021-02-10T04:54:41.642267+0000","last_peered":"2021-02-10T04:54:41.642267+0000","last_clean":"2021-02-10T04:54:41.642267+0000","last_became_active":"2021-02-10T04:52:33.087105+0000","last_became_peered":"2021-02-10T04:52:33.087105+0000","last_unstale":"2021-02-10T04:54:41.642267+0000","last_undegraded":"2021-02-10T04:54:41.642267+0000","last_fullsized":"2021-02-10T04:54:41.642267+0000","mapping_epoch":1148,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1149,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:53:10.075879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.075879+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27159495,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3,1],"avail_no_missing":["2","1","3"],"object_location_counts":[{"shards":"1,2,3","objects":12}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"1048'44","reported_seq":"98","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.644007+0000","last_change":"2021-02-10T04:53:32.373404+0000","last_active":"2021-02-10T04:54:41.644007+0000","last_peered":"2021-02-10T04:54:41.644007+0000","last_clean":"2021-02-10T04:54:41.644007+0000","last_became_active":"2021-02-10T04:53:20.304544+0000","last_became_peered":"2021-02-10T04:53:20.304544+0000","last_unstale":"2021-02-10T04:54:41.644007+0000","last_undegraded":"2021-02-10T04:54:41.644007+0000","last_fullsized":"2021-02-10T04:54:41.644007+0000","mapping_epoch":1166,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1167,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'44","last_scrub_stamp":"2021-02-10T04:53:32.373330+0000","last_deep_scrub":"78'32","last_deep_scrub_stamp":"2021-02-10T04:27:09.170428+0000","last_clean_scrub_stamp":"2021-02-10T04:53:32.373330+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":64,"num_read_kb":45,"num_write":34,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"1190","reported_epoch":"1214","state":"active+undersized","last_fresh":"2021-02-10T04:54:42.329207+0000","last_change":"2021-02-10T04:52:32.087531+0000","last_active":"2021-02-10T04:54:42.329207+0000","last_peered":"2021-02-10T04:54:42.329207+0000","last_clean":"2021-02-10T04:52:30.066743+0000","last_became_active":"2021-02-10T04:52:32.087531+0000","last_became_peered":"2021-02-10T04:52:32.087531+0000","last_unstale":"2021-02-10T04:54:42.329207+0000","last_undegraded":"2021-02-10T04:54:42.329207+0000","last_fullsized":"2021-02-10T04:52:32.076804+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:33.543559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:46:33.543559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"874","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.644066+0000","last_change":"2021-02-10T04:53:04.354508+0000","last_active":"2021-02-10T04:54:41.644066+0000","last_peered":"2021-02-10T04:54:41.644066+0000","last_clean":"2021-02-10T04:54:41.644066+0000","last_became_active":"2021-02-10T04:33:05.169159+0000","last_became_peered":"2021-02-10T04:33:05.169159+0000","last_unstale":"2021-02-10T04:54:41.644066+0000","last_undegraded":"2021-02-10T04:54:41.644066+0000","last_fullsized":"2021-02-10T04:54:41.644066+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:04.354409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:04.354409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"1268","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.200822+0000","last_change":"2021-02-10T04:52:36.777554+0000","last_active":"2021-02-10T04:54:42.200822+0000","last_peered":"2021-02-10T04:54:42.200822+0000","last_clean":"2021-02-10T04:54:42.200822+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:54:42.200822+0000","last_undegraded":"2021-02-10T04:54:42.200822+0000","last_fullsized":"2021-02-10T04:54:42.200822+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:36.777485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:52:36.777485+0000","last_clean_scrub_stamp":"2021-02-10T04:52:36.777485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1a","version":"1001'1528","reported_seq":"5825","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.200581+0000","last_change":"2021-02-10T04:52:31.086406+0000","last_active":"2021-02-10T04:54:42.200581+0000","last_peered":"2021-02-10T04:54:42.200581+0000","last_clean":"2021-02-10T04:54:42.200581+0000","last_became_active":"2021-02-10T04:52:31.085959+0000","last_became_peered":"2021-02-10T04:52:31.085959+0000","last_unstale":"2021-02-10T04:54:42.200581+0000","last_undegraded":"2021-02-10T04:54:42.200581+0000","last_fullsized":"2021-02-10T04:54:42.200581+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46944136,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":358,"num_read_kb":351935,"num_write":286,"num_write_kb":111355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":21311893,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"1049'68","reported_seq":"112","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.200545+0000","last_change":"2021-02-10T04:53:34.122240+0000","last_active":"2021-02-10T04:54:42.200545+0000","last_peered":"2021-02-10T04:54:42.200545+0000","last_clean":"2021-02-10T04:54:42.200545+0000","last_became_active":"2021-02-10T04:53:20.296560+0000","last_became_peered":"2021-02-10T04:53:20.296560+0000","last_unstale":"2021-02-10T04:54:42.200545+0000","last_undegraded":"2021-02-10T04:54:42.200545+0000","last_fullsized":"2021-02-10T04:54:42.200545+0000","mapping_epoch":1166,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1167,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'68","last_scrub_stamp":"2021-02-10T04:53:34.122152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.122152+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":124,"num_read_kb":101,"num_write":69,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"41","reported_epoch":"1215","state":"active+undersized","last_fresh":"2021-02-10T04:54:43.940172+0000","last_change":"2021-02-10T04:53:46.719798+0000","last_active":"2021-02-10T04:54:43.940172+0000","last_peered":"2021-02-10T04:54:43.940172+0000","last_clean":"2021-02-10T04:25:33.303710+0000","last_became_active":"2021-02-10T04:53:46.719798+0000","last_became_peered":"2021-02-10T04:53:46.719798+0000","last_unstale":"2021-02-10T04:54:43.940172+0000","last_undegraded":"2021-02-10T04:54:43.940172+0000","last_fullsized":"2021-02-10T04:53:46.703571+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:17.919745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:17.919745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"500","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.327627+0000","last_change":"2021-02-10T04:53:07.092032+0000","last_active":"2021-02-10T04:54:42.327627+0000","last_peered":"2021-02-10T04:54:42.327627+0000","last_clean":"2021-02-10T04:54:42.327627+0000","last_became_active":"2021-02-10T04:50:59.572077+0000","last_became_peered":"2021-02-10T04:50:59.572077+0000","last_unstale":"2021-02-10T04:54:42.327627+0000","last_undegraded":"2021-02-10T04:54:42.327627+0000","last_fullsized":"2021-02-10T04:54:42.327627+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.091949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:19.755718+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.091949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"1177","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.328511+0000","last_change":"2021-02-10T04:51:59.838856+0000","last_active":"2021-02-10T04:54:42.328511+0000","last_peered":"2021-02-10T04:54:42.328511+0000","last_clean":"2021-02-10T04:54:42.328511+0000","last_became_active":"2021-02-10T04:50:59.563156+0000","last_became_peered":"2021-02-10T04:50:59.563156+0000","last_unstale":"2021-02-10T04:54:42.328511+0000","last_undegraded":"2021-02-10T04:54:42.328511+0000","last_fullsized":"2021-02-10T04:54:42.328511+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.838790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.838790+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":"910","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.652198+0000","last_change":"2021-02-10T04:52:30.405170+0000","last_active":"2021-02-10T04:54:41.652198+0000","last_peered":"2021-02-10T04:54:41.652198+0000","last_clean":"2021-02-10T04:54:41.652198+0000","last_became_active":"2021-02-10T04:51:36.852058+0000","last_became_peered":"2021-02-10T04:51:36.852058+0000","last_unstale":"2021-02-10T04:54:41.652198+0000","last_undegraded":"2021-02-10T04:54:41.652198+0000","last_fullsized":"2021-02-10T04:54:41.652198+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.405106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:20.635536+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.405106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.20","version":"0'0","reported_seq":"150","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.934337+0000","last_change":"2021-02-10T04:53:46.739888+0000","last_active":"2021-02-10T04:54:43.934337+0000","last_peered":"2021-02-10T04:54:43.934337+0000","last_clean":"2021-02-10T04:54:43.934337+0000","last_became_active":"2021-02-10T04:53:46.739532+0000","last_became_peered":"2021-02-10T04:53:46.739532+0000","last_unstale":"2021-02-10T04:54:43.934337+0000","last_undegraded":"2021-02-10T04:54:43.934337+0000","last_fullsized":"2021-02-10T04:54:43.934337+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:56.449085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:41.375276+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.449085+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":"1143","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.545587+0000","last_change":"2021-02-10T04:53:35.858634+0000","last_active":"2021-02-10T04:54:43.545587+0000","last_peered":"2021-02-10T04:54:43.545587+0000","last_clean":"2021-02-10T04:54:43.545587+0000","last_became_active":"2021-02-10T04:51:36.869785+0000","last_became_peered":"2021-02-10T04:51:36.869785+0000","last_unstale":"2021-02-10T04:54:43.545587+0000","last_undegraded":"2021-02-10T04:54:43.545587+0000","last_fullsized":"2021-02-10T04:54:43.545587+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.858557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.858557+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":"112","reported_epoch":"1214","state":"active+undersized","last_fresh":"2021-02-10T04:54:41.642026+0000","last_change":"2021-02-10T04:52:32.094666+0000","last_active":"2021-02-10T04:54:41.642026+0000","last_peered":"2021-02-10T04:54:41.642026+0000","last_clean":"2021-02-10T04:52:30.064564+0000","last_became_active":"2021-02-10T04:52:32.094666+0000","last_became_peered":"2021-02-10T04:52:32.094666+0000","last_unstale":"2021-02-10T04:54:41.642026+0000","last_undegraded":"2021-02-10T04:54:41.642026+0000","last_fullsized":"2021-02-10T04:52:32.084052+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:47.481686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:47.481686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":"103","reported_epoch":"1214","state":"active+undersized","last_fresh":"2021-02-10T04:54:41.645451+0000","last_change":"2021-02-10T04:52:32.096027+0000","last_active":"2021-02-10T04:54:41.645451+0000","last_peered":"2021-02-10T04:54:41.645451+0000","last_clean":"2021-02-10T04:52:30.061012+0000","last_became_active":"2021-02-10T04:52:32.096027+0000","last_became_peered":"2021-02-10T04:52:32.096027+0000","last_unstale":"2021-02-10T04:54:41.645451+0000","last_undegraded":"2021-02-10T04:54:41.645451+0000","last_fullsized":"2021-02-10T04:52:32.076629+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.706372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.706372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":"1225","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.545224+0000","last_change":"2021-02-10T04:51:57.287452+0000","last_active":"2021-02-10T04:54:43.545224+0000","last_peered":"2021-02-10T04:54:43.545224+0000","last_clean":"2021-02-10T04:54:43.545224+0000","last_became_active":"2021-02-10T04:50:59.566827+0000","last_became_peered":"2021-02-10T04:50:59.566827+0000","last_unstale":"2021-02-10T04:54:43.545224+0000","last_undegraded":"2021-02-10T04:54:43.545224+0000","last_fullsized":"2021-02-10T04:54:43.545224+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.287363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.287363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":"190","reported_epoch":"1215","state":"active+undersized","last_fresh":"2021-02-10T04:54:43.546264+0000","last_change":"2021-02-10T04:52:32.082221+0000","last_active":"2021-02-10T04:54:43.546264+0000","last_peered":"2021-02-10T04:54:43.546264+0000","last_clean":"2021-02-10T04:52:30.068335+0000","last_became_active":"2021-02-10T04:52:32.082221+0000","last_became_peered":"2021-02-10T04:52:32.082221+0000","last_unstale":"2021-02-10T04:54:43.546264+0000","last_undegraded":"2021-02-10T04:54:43.546264+0000","last_fullsized":"2021-02-10T04:52:32.077664+0000","mapping_epoch":1147,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:10.308282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:10.308282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":"1113","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.328801+0000","last_change":"2021-02-10T04:53:09.121379+0000","last_active":"2021-02-10T04:54:42.328801+0000","last_peered":"2021-02-10T04:54:42.328801+0000","last_clean":"2021-02-10T04:54:42.328801+0000","last_became_active":"2021-02-10T04:50:59.561442+0000","last_became_peered":"2021-02-10T04:50:59.561442+0000","last_unstale":"2021-02-10T04:54:42.328801+0000","last_undegraded":"2021-02-10T04:54:42.328801+0000","last_fullsized":"2021-02-10T04:54:42.328801+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:09.121298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.121298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.21","version":"293'77","reported_seq":"785","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.328062+0000","last_change":"2021-02-10T04:52:28.062923+0000","last_active":"2021-02-10T04:54:42.328062+0000","last_peered":"2021-02-10T04:54:42.328062+0000","last_clean":"2021-02-10T04:54:42.328062+0000","last_became_active":"2021-02-10T04:52:28.062463+0000","last_became_peered":"2021-02-10T04:52:28.062463+0000","last_unstale":"2021-02-10T04:54:42.328062+0000","last_undegraded":"2021-02-10T04:54:42.328062+0000","last_fullsized":"2021-02-10T04:54:42.328062+0000","mapping_epoch":1143,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1144,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1886586,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":15163,"num_write":9,"num_write_kb":3316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":"111","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.641591+0000","last_change":"2021-02-10T04:52:32.124955+0000","last_active":"2021-02-10T04:54:41.641591+0000","last_peered":"2021-02-10T04:54:41.641591+0000","last_clean":"2021-02-10T04:54:41.641591+0000","last_became_active":"2021-02-10T04:51:36.867061+0000","last_became_peered":"2021-02-10T04:51:36.867061+0000","last_unstale":"2021-02-10T04:54:41.641591+0000","last_undegraded":"2021-02-10T04:54:41.641591+0000","last_fullsized":"2021-02-10T04:54:41.641591+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:32.124901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:08.271977+0000","last_clean_scrub_stamp":"2021-02-10T04:52:32.124901+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":"1154","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.545332+0000","last_change":"2021-02-10T04:51:07.182775+0000","last_active":"2021-02-10T04:54:43.545332+0000","last_peered":"2021-02-10T04:54:43.545332+0000","last_clean":"2021-02-10T04:54:43.545332+0000","last_became_active":"2021-02-10T04:28:23.126368+0000","last_became_peered":"2021-02-10T04:28:23.126368+0000","last_unstale":"2021-02-10T04:54:43.545332+0000","last_undegraded":"2021-02-10T04:54:43.545332+0000","last_fullsized":"2021-02-10T04:54:43.545332+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.182678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.182678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.20","version":"1000'1071","reported_seq":"4559","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:41.641386+0000","last_change":"2021-02-10T04:53:03.915465+0000","last_active":"2021-02-10T04:54:41.641386+0000","last_peered":"2021-02-10T04:54:41.641386+0000","last_clean":"2021-02-10T04:54:41.641386+0000","last_became_active":"2021-02-10T04:52:31.088690+0000","last_became_peered":"2021-02-10T04:52:31.088690+0000","last_unstale":"2021-02-10T04:54:41.641386+0000","last_undegraded":"2021-02-10T04:54:41.641386+0000","last_fullsized":"2021-02-10T04:54:41.641386+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:03.915403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.915403+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16398893,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":"739","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.934216+0000","last_change":"2021-02-10T04:53:46.721599+0000","last_active":"2021-02-10T04:54:43.934216+0000","last_peered":"2021-02-10T04:54:43.934216+0000","last_clean":"2021-02-10T04:54:43.934216+0000","last_became_active":"2021-02-10T04:53:46.721340+0000","last_became_peered":"2021-02-10T04:53:46.721340+0000","last_unstale":"2021-02-10T04:54:43.934216+0000","last_undegraded":"2021-02-10T04:54:43.934216+0000","last_fullsized":"2021-02-10T04:54:43.934216+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:28.092288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:07.767250+0000","last_clean_scrub_stamp":"2021-02-10T04:52:28.092288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.25","version":"53'2","reported_seq":"1169","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.200931+0000","last_change":"2021-02-10T04:53:36.103026+0000","last_active":"2021-02-10T04:54:42.200931+0000","last_peered":"2021-02-10T04:54:42.200931+0000","last_clean":"2021-02-10T04:54:42.200931+0000","last_became_active":"2021-02-10T04:28:23.467770+0000","last_became_peered":"2021-02-10T04:28:23.467770+0000","last_unstale":"2021-02-10T04:54:42.200931+0000","last_undegraded":"2021-02-10T04:54:42.200931+0000","last_fullsized":"2021-02-10T04:54:42.200931+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:53:36.102928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:36.102928+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.23","version":"1002'1657","reported_seq":"5086","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.544988+0000","last_change":"2021-02-10T04:52:31.093874+0000","last_active":"2021-02-10T04:54:43.544988+0000","last_peered":"2021-02-10T04:54:43.544988+0000","last_clean":"2021-02-10T04:54:43.544988+0000","last_became_active":"2021-02-10T04:52:31.092996+0000","last_became_peered":"2021-02-10T04:52:31.092996+0000","last_unstale":"2021-02-10T04:54:43.544988+0000","last_undegraded":"2021-02-10T04:54:43.544988+0000","last_fullsized":"2021-02-10T04:54:43.544988+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21396533,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":212,"num_read_kb":218837,"num_write":159,"num_write_kb":59399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":20479176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":"1261","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.544914+0000","last_change":"2021-02-10T04:53:12.951612+0000","last_active":"2021-02-10T04:54:43.544914+0000","last_peered":"2021-02-10T04:54:43.544914+0000","last_clean":"2021-02-10T04:54:43.544914+0000","last_became_active":"2021-02-10T04:50:59.566548+0000","last_became_peered":"2021-02-10T04:50:59.566548+0000","last_unstale":"2021-02-10T04:54:43.544914+0000","last_undegraded":"2021-02-10T04:54:43.544914+0000","last_fullsized":"2021-02-10T04:54:43.544914+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.951528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.951528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.26","version":"53'1","reported_seq":"53","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.939828+0000","last_change":"2021-02-10T04:53:46.722636+0000","last_active":"2021-02-10T04:54:43.939828+0000","last_peered":"2021-02-10T04:54:43.939828+0000","last_clean":"2021-02-10T04:54:43.939828+0000","last_became_active":"2021-02-10T04:53:46.722227+0000","last_became_peered":"2021-02-10T04:53:46.722227+0000","last_unstale":"2021-02-10T04:54:43.939828+0000","last_undegraded":"2021-02-10T04:54:43.939828+0000","last_fullsized":"2021-02-10T04:54:43.939828+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:53:03.933910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.933910+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":423,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.22","version":"998'1009","reported_seq":"4576","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.200646+0000","last_change":"2021-02-10T04:52:31.086563+0000","last_active":"2021-02-10T04:54:42.200646+0000","last_peered":"2021-02-10T04:54:42.200646+0000","last_clean":"2021-02-10T04:54:42.200646+0000","last_became_active":"2021-02-10T04:52:31.085288+0000","last_became_peered":"2021-02-10T04:52:31.085288+0000","last_unstale":"2021-02-10T04:54:42.200646+0000","last_undegraded":"2021-02-10T04:54:42.200646+0000","last_fullsized":"2021-02-10T04:54:42.200646+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12355207,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":120,"num_read_kb":124702,"num_write":77,"num_write_kb":25052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4145029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":"1143","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.544893+0000","last_change":"2021-02-10T04:53:10.958668+0000","last_active":"2021-02-10T04:54:43.544893+0000","last_peered":"2021-02-10T04:54:43.544893+0000","last_clean":"2021-02-10T04:54:43.544893+0000","last_became_active":"2021-02-10T04:45:49.552026+0000","last_became_peered":"2021-02-10T04:45:49.552026+0000","last_unstale":"2021-02-10T04:54:43.544893+0000","last_undegraded":"2021-02-10T04:54:43.544893+0000","last_fullsized":"2021-02-10T04:54:43.544893+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:10.958579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.958579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":"158","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.940065+0000","last_change":"2021-02-10T04:53:46.725941+0000","last_active":"2021-02-10T04:54:43.940065+0000","last_peered":"2021-02-10T04:54:43.940065+0000","last_clean":"2021-02-10T04:54:43.940065+0000","last_became_active":"2021-02-10T04:53:46.725587+0000","last_became_peered":"2021-02-10T04:53:46.725587+0000","last_unstale":"2021-02-10T04:54:43.940065+0000","last_undegraded":"2021-02-10T04:54:43.940065+0000","last_fullsized":"2021-02-10T04:54:43.940065+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:41.758684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.758684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":"1142","reported_epoch":"1214","state":"active+clean","last_fresh":"2021-02-10T04:54:42.327907+0000","last_change":"2021-02-10T04:53:11.139690+0000","last_active":"2021-02-10T04:54:42.327907+0000","last_peered":"2021-02-10T04:54:42.327907+0000","last_clean":"2021-02-10T04:54:42.327907+0000","last_became_active":"2021-02-10T04:50:59.569162+0000","last_became_peered":"2021-02-10T04:50:59.569162+0000","last_unstale":"2021-02-10T04:54:42.327907+0000","last_undegraded":"2021-02-10T04:54:42.327907+0000","last_fullsized":"2021-02-10T04:54:42.327907+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:11.139602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.139602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":"1169","reported_epoch":"1215","state":"active+clean","last_fresh":"2021-02-10T04:54:43.544914+0000","last_change":"2021-02-10T04:52:24.072005+0000","last_active":"2021-02-10T04:54:43.544914+0000","last_peered":"2021-02-10T04:54:43.544914+0000","last_clean":"2021-02-10T04:54:43.544914+0000","last_became_active":"2021-02-10T04:50:59.527945+0000","last_became_peered":"2021-02-10T04:50:59.527945+0000","last_unstale":"2021-02-10T04:54:43.544914+0000","last_undegraded":"2021-02-10T04:54:43.544914+0000","last_fullsized":"2021-02-10T04:54:43.544914+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:24.071915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_clean_scrub_stamp":"2021-02-10T04:52:24.071915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":931687135,"num_objects":374,"num_object_clones":323,"num_object_copies":1122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":164,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":374,"num_whiteouts":10,"num_read":4479,"num_read_kb":4587256,"num_write":3265,"num_write_kb":1255814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":630,"num_bytes_recovered":307448165,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2029686784,"data_stored":2612817805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34722,"ondisk_log_size":34722,"up":94,"acting":95,"num_store_stats":8},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":2368,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":18,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1069,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":5835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":111,"acting":111,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":29,"acting":29,"num_store_stats":8},{"poolid":3,"num_pg":13,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":607,"num_read_kb":602,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20,"ondisk_log_size":20,"up":34,"acting":37,"num_store_stats":8},{"poolid":4,"num_pg":28,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":43,"num_objects_misplaced":32,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":2289,"num_read_kb":1762,"num_write":1306,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":17828,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2093056,"data_stored":75230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1082,"ondisk_log_size":1082,"up":76,"acting":80,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":84,"acting":84,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":114,"acting":114,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365153,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":189960,"kb_used_data":148672,"kb_used_omap":0,"kb_used_meta":41280,"kb_avail":93555192,"statfs":{"total":95995035648,"available":95800516608,"internally_reserved":0,"allocated":152240128,"data_stored":202059855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42270720},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:27 2021","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":1.2370000000000001,"15min":1.284},"min":{"1min":0.50800000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1220000000000001,"5min":34.299999999999997,"15min":67.277000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.5109999999999999,"15min":1.548},"min":{"1min":0.46300000000000002,"5min":0.438,"15min":0.438},"max":{"1min":2.8690000000000002,"5min":34.652999999999999,"15min":66.933999999999997},"last":0.71399999999999997}]},{"osd":1,"last update":"Wed Feb 10 04:54:39 2021","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.81899999999999995,"15min":0.96799999999999997},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.214,"5min":2.254,"15min":10.140000000000001},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.871,"15min":0.97199999999999998},"min":{"1min":0.52600000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.208,"5min":2.5409999999999999,"15min":7.1520000000000001},"last":0.78800000000000003}]},{"osd":2,"last update":"Wed Feb 10 04:54:39 2021","interfaces":[{"interface":"back","average":{"1min":1.7529999999999999,"5min":1.3080000000000001,"15min":1.3959999999999999},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":16.896999999999998,"5min":16.896999999999998,"15min":66.534000000000006},"last":1.2749999999999999},{"interface":"front","average":{"1min":1.639,"5min":1.2370000000000001,"15min":1.5189999999999999},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":16.792000000000002,"5min":16.792000000000002,"15min":66.781999999999996},"last":4.0620000000000003}]},{"osd":3,"last update":"Wed Feb 10 04:53:59 2021","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.2190000000000001,"15min":1.3180000000000001},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.573,"5min":4.5890000000000004,"15min":4.5890000000000004},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.034,"5min":1.306,"15min":1.425},"min":{"1min":0.73199999999999998,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.593,"5min":4.6609999999999996,"15min":4.6609999999999996},"last":1.0980000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:54:39 2021","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.25,"15min":1.411},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":2.589,"5min":2.9729999999999999,"15min":10.781000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.212,"15min":1.357},"min":{"1min":0.74199999999999999,"5min":0.69699999999999995,"15min":0.68100000000000005},"max":{"1min":1.8069999999999999,"5min":3.4049999999999998,"15min":10.326000000000001},"last":0.74399999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:54:39 2021","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.377,"15min":1.6100000000000001},"min":{"1min":0.747,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.825,"5min":4.2510000000000003,"15min":35.283000000000001},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.294,"5min":1.508,"15min":1.8320000000000001},"min":{"1min":0.83799999999999997,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":4.4020000000000001,"5min":4.4020000000000001,"15min":35.353999999999999},"last":1.3169999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:54:39 2021","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.464,"15min":1.7749999999999999},"min":{"1min":0.78900000000000003,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.823,"5min":4.0380000000000003,"15min":16.050000000000001},"last":1.111},{"interface":"front","average":{"1min":1.179,"5min":1.4410000000000001,"15min":1.724},"min":{"1min":0.89700000000000002,"5min":0.876,"15min":0.876},"max":{"1min":1.52,"5min":3.9159999999999999,"15min":15.960000000000001},"last":1.0660000000000001}]}]},{"osd":6,"up_from":43,"seq":184683594085,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":402892,"kb_used_data":341764,"kb_used_omap":0,"kb_used_meta":61120,"kb_avail":93342260,"statfs":{"total":95995035648,"available":95582474240,"internally_reserved":0,"allocated":349966336,"data_stored":443024773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62586880},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:52:12 2021","interfaces":[{"interface":"back","average":{"1min":4.016,"5min":2.7400000000000002,"15min":1.595},"min":{"1min":0.52000000000000002,"5min":0.48099999999999998,"15min":0.44700000000000001},"max":{"1min":61.768000000000001,"5min":137.60599999999999,"15min":137.60599999999999},"last":0.97999999999999998},{"interface":"front","average":{"1min":4.0270000000000001,"5min":2.694,"15min":1.7989999999999999},"min":{"1min":0.504,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":62.683,"5min":137.66,"15min":137.66},"last":0.82099999999999995}]},{"osd":1,"last update":"Wed Feb 10 04:54:16 2021","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":1.05,"15min":1.083},"min":{"1min":0.51800000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.3859999999999999,"5min":13.18,"15min":13.18},"last":1.1799999999999999},{"interface":"front","average":{"1min":0.91000000000000003,"5min":1.1100000000000001,"15min":1.1180000000000001},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":3.6880000000000002,"5min":11.041,"15min":11.041},"last":0.55900000000000005}]},{"osd":2,"last update":"Wed Feb 10 04:54:16 2021","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":2.7069999999999999,"15min":1.734},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.4119999999999999,"5min":163.81700000000001,"15min":163.81700000000001},"last":1.091},{"interface":"front","average":{"1min":1.0620000000000001,"5min":2.722,"15min":1.77},"min":{"1min":0.66200000000000003,"5min":0.60299999999999998,"15min":0.59899999999999998},"max":{"1min":1.633,"5min":164.56800000000001,"15min":164.56800000000001},"last":1.306}]},{"osd":3,"last update":"Wed Feb 10 04:54:01 2021","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.161,"15min":1.1990000000000001},"min":{"1min":0.77600000000000002,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.722,"5min":1.722,"15min":1.722},"last":1.25},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.123,"15min":1.1240000000000001},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.629,"5min":2.363,"15min":2.363},"last":0.68100000000000005}]},{"osd":4,"last update":"Wed Feb 10 04:54:16 2021","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.274,"15min":1.3999999999999999},"min":{"1min":0.71799999999999997,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.53,"5min":7.1459999999999999,"15min":7.1459999999999999},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.96199999999999997,"5min":1.0900000000000001,"15min":1.123},"min":{"1min":0.71399999999999997,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.494,"5min":11.997999999999999,"15min":11.997999999999999},"last":0.71499999999999997}]},{"osd":5,"last update":"Wed Feb 10 04:54:16 2021","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.2589999999999999,"15min":1.629},"min":{"1min":0.78800000000000003,"5min":0.61699999999999999,"15min":0.56899999999999995},"max":{"1min":1.829,"5min":12.045,"15min":81.540000000000006},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.522,"15min":1.9019999999999999},"min":{"1min":0.85699999999999998,"5min":0.78700000000000003,"15min":0.78000000000000003},"max":{"1min":1.7949999999999999,"5min":13.401,"15min":81.492999999999995},"last":1.4530000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:54:45 2021","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.157,"15min":1.3089999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.637,"5min":4.0919999999999996,"15min":5.1820000000000004},"last":0.83899999999999997},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.4330000000000001,"15min":1.6120000000000001},"min":{"1min":0.91100000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":2.1440000000000001,"5min":5.4109999999999996,"15min":13.315},"last":1.393}]}]},{"osd":1,"up_from":13,"seq":55834575222,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":290500,"kb_used_data":224316,"kb_used_omap":0,"kb_used_meta":66176,"kb_avail":93454652,"statfs":{"total":95995035648,"available":95697563648,"internally_reserved":0,"allocated":229699584,"data_stored":289168823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":67764224},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:48 2021","interfaces":[{"interface":"back","average":{"1min":1.696,"5min":2.6320000000000001,"15min":1.921},"min":{"1min":0.44700000000000001,"5min":0.437,"15min":0.437},"max":{"1min":14.053000000000001,"5min":115.622,"15min":115.622},"last":0.69599999999999995},{"interface":"front","average":{"1min":1.728,"5min":2.585,"15min":1.948},"min":{"1min":0.49199999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":14.109,"5min":115.73099999999999,"15min":115.73099999999999},"last":0.72899999999999998}]},{"osd":2,"last update":"Wed Feb 10 04:54:07 2021","interfaces":[{"interface":"back","average":{"1min":2.3399999999999999,"5min":3.3410000000000002,"15min":2.0350000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":37.975000000000001,"5min":115.59699999999999,"15min":115.59699999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":2.3780000000000001,"5min":3.379,"15min":2.0800000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":37.840000000000003,"5min":115.753,"15min":115.753},"last":1.1279999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:54:03 2021","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.85299999999999998,"15min":0.93000000000000005},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.84699999999999998,"5min":3.077,"15min":3.077},"last":1.6919999999999999},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.96999999999999997,"15min":1.024},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.226,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":1.052}]},{"osd":4,"last update":"Wed Feb 10 04:53:51 2021","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":1.0289999999999999,"15min":1.1719999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.55200000000000005},"max":{"1min":1.825,"5min":2.9670000000000001,"15min":4.899},"last":1.6080000000000001},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.117,"15min":1.327},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.718,"5min":2.5270000000000001,"15min":29.547999999999998},"last":1.173}]},{"osd":5,"last update":"Wed Feb 10 04:53:57 2021","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":1.1020000000000001,"15min":1.198},"min":{"1min":0.628,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.331,"5min":2.7959999999999998,"15min":7.8049999999999997},"last":1.585},{"interface":"front","average":{"1min":1.036,"5min":1.236,"15min":1.504},"min":{"1min":0.75800000000000001,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.383,"5min":2.8199999999999998,"15min":28.678000000000001},"last":1.714}]},{"osd":6,"last update":"Wed Feb 10 04:54:07 2021","interfaces":[{"interface":"back","average":{"1min":2.2810000000000001,"5min":1.421,"15min":1.571},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":30.997,"5min":30.997,"15min":30.997},"last":1.2849999999999999},{"interface":"front","average":{"1min":2.2599999999999998,"5min":1.4370000000000001,"15min":1.476},"min":{"1min":0.64200000000000002,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":31.138000000000002,"5min":31.138000000000002,"15min":31.138000000000002},"last":1.4119999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:54:26 2021","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.1890000000000001,"15min":1.4910000000000001},"min":{"1min":0.80200000000000005,"5min":0.628,"15min":0.628},"max":{"1min":1.494,"5min":3.1139999999999999,"15min":28.573},"last":1.825},{"interface":"front","average":{"1min":1.151,"5min":1.2649999999999999,"15min":1.54},"min":{"1min":0.86399999999999999,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.7210000000000001,"5min":2.298,"15min":29.408999999999999},"last":1.448}]}]},{"osd":0,"up_from":8,"seq":34359738720,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":399164,"kb_used_data":337140,"kb_used_omap":0,"kb_used_meta":62016,"kb_avail":93345988,"statfs":{"total":95995035648,"available":95586291712,"internally_reserved":0,"allocated":345231360,"data_stored":470653300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":63504384},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 04:51:32 2021","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.81100000000000005,"15min":1.0089999999999999},"min":{"1min":0.47099999999999997,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.0329999999999999,"5min":2.2930000000000001,"15min":28.946999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.874,"5min":0.92300000000000004,"15min":1.0149999999999999},"min":{"1min":0.68000000000000005,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.1819999999999999,"5min":2.0670000000000002,"15min":3.7050000000000001},"last":0.89200000000000002}]},{"osd":2,"last update":"Wed Feb 10 04:52:09 2021","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":2.2389999999999999,"15min":1.613},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0700000000000001,"5min":97.983999999999995,"15min":97.983999999999995},"last":1.056},{"interface":"front","average":{"1min":0.749,"5min":2.153,"15min":1.5169999999999999},"min":{"1min":0.51900000000000002,"5min":0.51300000000000001,"15min":0.47699999999999998},"max":{"1min":1.1319999999999999,"5min":97.844999999999999,"15min":97.844999999999999},"last":1.296}]},{"osd":3,"last update":"Wed Feb 10 04:51:59 2021","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.87,"15min":0.87},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":1.002},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.254,"5min":1.254,"15min":1.254},"last":0.871}]},{"osd":4,"last update":"Wed Feb 10 04:51:32 2021","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":1.119,"15min":1.306},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.3200000000000001,"5min":2.206,"15min":5.1680000000000001},"last":1.1120000000000001},{"interface":"front","average":{"1min":0.96499999999999997,"5min":1.179,"15min":1.252},"min":{"1min":0.64200000000000002,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.607,"5min":3.359,"15min":14.672000000000001},"last":1.0920000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:51:54 2021","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.325,"15min":1.512},"min":{"1min":0.84599999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.714,"5min":3.944,"15min":25.047000000000001},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.413,"15min":1.7090000000000001},"min":{"1min":0.79400000000000004,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":2.1669999999999998,"5min":3.7850000000000001,"15min":28.827999999999999},"last":1.226}]},{"osd":6,"last update":"Wed Feb 10 04:52:23 2021","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.845,"15min":1.8280000000000001},"min":{"1min":1.0109999999999999,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.7889999999999999,"5min":45.012,"15min":45.012},"last":1.587},{"interface":"front","average":{"1min":1.327,"5min":1.891,"15min":1.95},"min":{"1min":0.88800000000000001,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.0979999999999999,"5min":45.225000000000001,"15min":45.225000000000001},"last":1.3620000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:52:28 2021","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.3080000000000001,"15min":1.5069999999999999},"min":{"1min":0.96099999999999997,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.605,"5min":2.8700000000000001,"15min":5.6050000000000004},"last":1.532},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.3959999999999999,"15min":1.5669999999999999},"min":{"1min":1.0129999999999999,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.794,"5min":3.1259999999999999,"15min":13.968999999999999},"last":1.5129999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411700,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":369172,"kb_used_data":317900,"kb_used_omap":0,"kb_used_meta":51264,"kb_avail":93375980,"statfs":{"total":95995035648,"available":95617003520,"internally_reserved":0,"allocated":325529600,"data_stored":374540144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52494336},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:50 2021","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.99099999999999999,"15min":1.718},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":3.698,"5min":3.996,"15min":163.15199999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.91300000000000003,"15min":1.655},"min":{"1min":0.51100000000000001,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.9710000000000001,"5min":2.2349999999999999,"15min":162.959},"last":0.85399999999999998}]},{"osd":1,"last update":"Wed Feb 10 04:53:53 2021","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.874,"15min":1.014},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.044,"5min":3.8159999999999998,"15min":4.9690000000000003},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.627,"5min":0.73899999999999999,"15min":0.89900000000000002},"min":{"1min":0.47299999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.0009999999999999,"5min":1.506,"15min":5.726},"last":0.53300000000000003}]},{"osd":3,"last update":"Wed Feb 10 04:54:06 2021","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":1.02,"15min":1.117},"min":{"1min":0.49099999999999999,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.3859999999999999,"5min":4.2380000000000004,"15min":4.2380000000000004},"last":0.58799999999999997},{"interface":"front","average":{"1min":1.024,"5min":1.2130000000000001,"15min":1.2909999999999999},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.589,"5min":4.54,"15min":4.54},"last":0.98999999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:54:35 2021","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":1.073,"15min":1.2909999999999999},"min":{"1min":0.69799999999999995,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.3129999999999999,"5min":4.1310000000000002,"15min":4.3319999999999999},"last":1.099},{"interface":"front","average":{"1min":2.048,"5min":1.2050000000000001,"15min":1.1739999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":24.373999999999999,"5min":24.373999999999999,"15min":24.373999999999999},"last":1.0609999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:53:53 2021","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.744,"15min":1.7090000000000001},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":2.4260000000000002,"5min":33.018999999999998,"15min":33.018999999999998},"last":1.4530000000000001},{"interface":"front","average":{"1min":0.88600000000000001,"5min":1.526,"15min":1.476},"min":{"1min":0.65900000000000003,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.2230000000000001,"5min":33.043999999999997,"15min":33.043999999999997},"last":0.85399999999999998}]},{"osd":6,"last update":"Wed Feb 10 04:54:21 2021","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":1.133,"15min":1.4590000000000001},"min":{"1min":0.76800000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.8180000000000001,"5min":2.0579999999999998,"15min":20.007000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.3420000000000001,"15min":1.635},"min":{"1min":0.94599999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.627,"5min":4.415,"15min":19.417999999999999},"last":1.1739999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:54:35 2021","interfaces":[{"interface":"back","average":{"1min":2.3290000000000002,"5min":1.599,"15min":1.6819999999999999},"min":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"max":{"1min":24.405000000000001,"5min":24.405000000000001,"15min":24.405000000000001},"last":1.258},{"interface":"front","average":{"1min":1.032,"5min":1.089,"15min":1.284},"min":{"1min":0.874,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.3140000000000001,"5min":1.6719999999999999,"15min":4.9189999999999996},"last":1.133}]}]},{"osd":3,"up_from":1111,"seq":4771708665903,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":327248,"kb_used_data":300680,"kb_used_omap":386,"kb_used_meta":26173,"kb_avail":93417904,"statfs":{"total":95995035648,"available":95659933696,"internally_reserved":0,"allocated":307896320,"data_stored":409462797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":396036,"internal_metadata":26801404},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:58 2021","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":0.67200000000000004}]},{"osd":1,"last update":"Wed Feb 10 04:54:03 2021","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65700000000000003,"15min":0.65100000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.8360000000000001,"5min":1.8360000000000001,"15min":1.8360000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.83299999999999996,"15min":0.873},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.641,"5min":2.137,"15min":2.137},"last":0.55800000000000005}]},{"osd":2,"last update":"Wed Feb 10 04:54:03 2021","interfaces":[{"interface":"back","average":{"1min":1.8180000000000001,"5min":1.208,"15min":1.169},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":23.844999999999999,"5min":23.844999999999999,"15min":23.844999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.9970000000000001,"5min":1.2669999999999999,"15min":1.169},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":23.783999999999999,"5min":23.783999999999999,"15min":23.783999999999999},"last":0.61299999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:54:03 2021","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.96499999999999997,"15min":0.96899999999999997},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":2.9430000000000001,"5min":2.9430000000000001,"15min":2.9430000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.92000000000000004,"5min":1.042,"15min":1.1000000000000001},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.357,"5min":1.8999999999999999,"15min":1.8999999999999999},"last":0.90500000000000003}]},{"osd":5,"last update":"Wed Feb 10 04:54:03 2021","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":2.6379999999999999,"15min":2.7109999999999999},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":2.0979999999999999,"5min":52.353000000000002,"15min":52.353000000000002},"last":0.76000000000000001},{"interface":"front","average":{"1min":1.0660000000000001,"5min":2.4929999999999999,"15min":2.5259999999999998},"min":{"1min":0.71399999999999997,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":3.544,"5min":52.301000000000002,"15min":52.301000000000002},"last":0.85899999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:54:03 2021","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":4.4249999999999998,"15min":5.2930000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.474,"5min":64.376999999999995,"15min":64.376999999999995},"last":0.93300000000000005},{"interface":"front","average":{"1min":1.228,"5min":4.4640000000000004,"15min":5.2779999999999996},"min":{"1min":0.85599999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":1.8080000000000001,"5min":64.438999999999993,"15min":64.438999999999993},"last":0.97199999999999998}]},{"osd":7,"last update":"Wed Feb 10 04:54:03 2021","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.087,"15min":1.113},"min":{"1min":0.79800000000000004,"5min":0.754,"15min":0.754},"max":{"1min":1.466,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.137,"15min":1.141},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.5649999999999999,"5min":2.044,"15min":2.044},"last":0.88700000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986540,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":159128,"kb_used_data":119888,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":93586024,"statfs":{"total":95995035648,"available":95832088576,"internally_reserved":0,"allocated":122765312,"data_stored":155363221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:36 2021","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.0899999999999999,"15min":1.593},"min":{"1min":0.45800000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.7799999999999998,"5min":95.721999999999994,"15min":95.721999999999994},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.67700000000000005,"5min":2.0870000000000002,"15min":1.6000000000000001},"min":{"1min":0.46000000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.002,"5min":95.344999999999999,"15min":95.344999999999999},"last":0.70599999999999996}]},{"osd":1,"last update":"Wed Feb 10 04:54:43 2021","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.81999999999999995,"15min":0.95199999999999996},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":3.2650000000000001,"5min":7.883,"15min":24.295000000000002},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.94699999999999995,"15min":1.149},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.276,"5min":3.8090000000000002,"15min":24.943000000000001},"last":0.50700000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:54:43 2021","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":2.6480000000000001,"15min":1.6870000000000001},"min":{"1min":0.53900000000000003,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":11.776,"5min":120.39,"15min":120.39},"last":0.66900000000000004},{"interface":"front","average":{"1min":1.4530000000000001,"5min":2.609,"15min":1.6950000000000001},"min":{"1min":0.56899999999999995,"5min":0.56200000000000006,"15min":0.51900000000000002},"max":{"1min":11.795999999999999,"5min":120.583,"15min":120.583},"last":0.73499999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:54:01 2021","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.21,"15min":1.2749999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.6339999999999999,"5min":2.1429999999999998,"15min":2.1429999999999998},"last":0.99299999999999999},{"interface":"front","average":{"1min":2.8519999999999999,"5min":1.7490000000000001,"15min":1.603},"min":{"1min":0.65200000000000002,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":26.802,"5min":26.802,"15min":26.802},"last":0.87}]},{"osd":5,"last update":"Wed Feb 10 04:53:56 2021","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.28,"15min":1.516},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":2.351,"5min":8.1509999999999998,"15min":35.774000000000001},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.276,"5min":1.282,"15min":1.4810000000000001},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.58799999999999997},"max":{"1min":4.319,"5min":7.9950000000000001,"15min":35.606000000000002},"last":0.77300000000000002}]},{"osd":6,"last update":"Wed Feb 10 04:54:01 2021","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":3.5539999999999998,"15min":2.3940000000000001},"min":{"1min":0.873,"5min":0.86199999999999999,"15min":0.76600000000000001},"max":{"1min":1.8380000000000001,"5min":201.626,"15min":201.626},"last":1.111},{"interface":"front","average":{"1min":1.125,"5min":3.476,"15min":2.4279999999999999},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.7529999999999999,"5min":201.577,"15min":201.577},"last":1.2270000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:54:31 2021","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.472,"15min":1.492},"min":{"1min":0.878,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.5099999999999998,"5min":4.7439999999999998,"15min":5.8920000000000003},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.3,"15min":1.526},"min":{"1min":0.78700000000000003,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.75,"5min":2.2250000000000001,"15min":5.149},"last":0.96399999999999997}]}]},{"osd":5,"up_from":37,"seq":158913790313,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":250748,"kb_used_data":205684,"kb_used_omap":0,"kb_used_meta":45056,"kb_avail":93494404,"statfs":{"total":95995035648,"available":95738269696,"internally_reserved":0,"allocated":210620416,"data_stored":275437644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":46137344},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:51 2021","interfaces":[{"interface":"back","average":{"1min":5.0590000000000002,"5min":1.714,"15min":1.204},"min":{"1min":0.45700000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":58.624000000000002,"5min":58.624000000000002,"15min":58.624000000000002},"last":0.64600000000000002},{"interface":"front","average":{"1min":5.04,"5min":1.784,"15min":1.3009999999999999},"min":{"1min":0.56599999999999995,"5min":0.46600000000000003,"15min":0.46400000000000002},"max":{"1min":57.880000000000003,"5min":57.880000000000003,"15min":57.880000000000003},"last":1.284}]},{"osd":1,"last update":"Wed Feb 10 04:53:52 2021","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":0.93400000000000005,"15min":0.95399999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":6.391,"5min":6.391,"15min":6.391},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.72899999999999998,"5min":1.1499999999999999,"15min":1.175},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.2569999999999999,"5min":11.818,"15min":11.875999999999999},"last":0.70899999999999996}]},{"osd":2,"last update":"Wed Feb 10 04:53:52 2021","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":1.641,"15min":1.5369999999999999},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.2430000000000001,"5min":50.948999999999998,"15min":50.948999999999998},"last":0.995},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.7490000000000001,"15min":1.5760000000000001},"min":{"1min":0.61899999999999999,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":6.7130000000000001,"5min":50.149000000000001,"15min":50.149000000000001},"last":1.0389999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:54:06 2021","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.2030000000000001,"15min":1.2909999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.357,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":1.3360000000000001},{"interface":"front","average":{"1min":1.284,"5min":1.2070000000000001,"15min":1.1890000000000001},"min":{"1min":0.73199999999999998,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":6.5129999999999999,"5min":6.5129999999999999,"15min":6.5129999999999999},"last":1.145}]},{"osd":4,"last update":"Wed Feb 10 04:53:52 2021","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.329,"15min":1.3320000000000001},"min":{"1min":0.70399999999999996,"5min":0.64500000000000002,"15min":0.57099999999999995},"max":{"1min":2.028,"5min":9.4890000000000008,"15min":9.4890000000000008},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.117,"5min":1.3300000000000001,"15min":1.385},"min":{"1min":0.69099999999999995,"5min":0.68100000000000005,"15min":0.67500000000000004},"max":{"1min":2.3149999999999999,"5min":5.6180000000000003,"15min":5.6180000000000003},"last":1.1040000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:54:06 2021","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.9379999999999999,"15min":3.0150000000000001},"min":{"1min":0.93100000000000005,"5min":0.83899999999999997,"15min":0.65800000000000003},"max":{"1min":2.7829999999999999,"5min":38.014000000000003,"15min":319.51999999999998},"last":1.2250000000000001},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.9019999999999999,"15min":2.9409999999999998},"min":{"1min":0.79300000000000004,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":6.8220000000000001,"5min":37.896999999999998,"15min":319.22300000000001},"last":1.25}]},{"osd":7,"last update":"Wed Feb 10 04:54:34 2021","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.452,"15min":1.484},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.5009999999999999,"5min":11.897,"15min":11.897},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.228,"5min":1.5,"15min":1.5640000000000001},"min":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.82999999999999996},"max":{"1min":1.4319999999999999,"5min":6.0369999999999999,"15min":6.0369999999999999},"last":1.736}]}]}]}} 2021-02-10T04:54:52.125 INFO:tasks.thrashosds.thrasher:Setting 5.13 to [2, 3, 5, 0, 4, 1] 2021-02-10T04:54:52.125 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '5.13', '2', '3', '5', '0', '4', '1'] 2021-02-10T04:54:52.125 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pg-upmap-items 5.13 2 3 5 0 4 1 2021-02-10T04:54:52.334 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:54:53.837 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset nodeep-scrub 2021-02-10T04:54:53.862 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:53 smithi085 conmon[29319]: cluster 2021-02-10T04:54:52.484031+0000 mgr.y (mgr.14140) 1344 : cluster [DBG] pgmap v2204: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:53.862 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:53 smithi085 conmon[35961]: cluster 2021-02-10T04:54:52.484031+0000 mgr.y (mgr.14140) 1344 : cluster [DBG] pgmap v2204: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:53 smithi164 conmon[29288]: cluster 2021-02-10T04:54:52.484031 2021-02-10T04:54:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:53 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1344 : cluster [DBG] pgmap v2204: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:53.944 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:54:54.058 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:54:54.165 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:54:54.581 INFO:teuthology.orchestra.run.smithi085.stderr:set 5.13 pg_upmap_items mapping to [2->3,5->0,4->1] 2021-02-10T04:54:54.838 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:54 smithi085 conmon[35961]: audit 2021-02-10T04:54:54.394562 2021-02-10T04:54:54.838 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:54 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2603 : audit [INF] from='client.? 172.21.15.85:0/1371435663' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.13", "id": [2, 3, 5, 0, 4, 1]}]: dispatch 2021-02-10T04:54:54.838 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:54 smithi085 conmon[29319]: audit 2021-02-10T04:54:54.394562 2021-02-10T04:54:54.838 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:54 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2603 : audit [INF] from='client.? 172.21.15.85:0/1371435663' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.13", "id": [2, 3, 5, 0, 4, 1]}]: dispatch 2021-02-10T04:54:54.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:54 smithi164 conmon[29288]: audit 2021-02-10T04:54:54.394562+0000 mon.a 2021-02-10T04:54:54.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:54 smithi164 conmon[29288]: (mon.0) 2603 : audit [INF] from='client.? 172.21.15.85:0/1371435663' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.13", "id": [2, 3, 5, 0, 4, 1]}]: dispatch 2021-02-10T04:54:55.291 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.4 dump_blocked_ops 2021-02-10T04:54:55.564 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:54:55.568 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:55 smithi164 conmon[29288]: cluster 2021-02-10T04:54:54.486303+0000 mgr.y 2021-02-10T04:54:55.569 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:55 smithi164 conmon[29288]: (mgr.14140) 1345 : cluster [DBG] pgmap v2205: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:55.569 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:55 smithi164 conmon[29288]: audit 2021-02-10T04:54:54.572828+0000 mon.a (mon.0) 2604 : audit [INF] from='client.? 172.21.15.85:0/1371435663' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.13", "id": [2, 3, 5, 0, 4, 1]}]': finished 2021-02-10T04:54:55.569 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:55 smithi164 conmon[29288]: cluster 2021-02-10T04:54:54.572874+0000 mon.a (mon.0) 2605 : cluster [DBG] osdmap e1218: 8 total, 7 up, 8 in 2021-02-10T04:54:55.569 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:55 smithi164 conmon[29288]: cluster 2021-02-10T04:54:54.591434+0000 osd.4 (osd.4) 2655 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:54:55.569 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:55 smithi164 conmon[29288]: cluster 2021-02-10T04:54:54.593040+0000 osd.4 (osd.4) 2656 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:54:55.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:55 smithi085 conmon[35961]: cluster 2021-02-10T04:54:54.486303+0000 mgr.y (mgr.14140) 1345 : cluster [DBG] pgmap v2205: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:55.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:55 smithi085 conmon[35961]: audit 2021-02-10T04:54:54.572828+0000 mon.a (mon.0) 2604 : audit [INF] from='client.? 172.21.15.85:0/1371435663' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.13", "id": [2, 3, 5, 0, 4, 1]}]': finished 2021-02-10T04:54:55.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:55 smithi085 conmon[35961]: cluster 2021-02-10T04:54:54.572874+0000 mon.a (mon.0) 2605 : cluster [DBG] osdmap e1218: 8 total, 7 up, 8 in 2021-02-10T04:54:55.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:55 smithi085 conmon[35961]: cluster 2021-02-10T04:54:54.591434+0000 osd.4 (osd.4) 2655 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:54:55.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:55 smithi085 conmon[35961]: cluster 2021-02-10T04:54:54.593040+0000 2021-02-10T04:54:55.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:55 smithi085 conmon[35961]: osd.4 (osd.4) 2656 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:54:55.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:55 smithi085 conmon[29319]: cluster 2021-02-10T04:54:54.486303+0000 mgr.y (mgr.14140) 1345 : cluster [DBG] pgmap v2205: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:55.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:55 smithi085 conmon[29319]: audit 2021-02-10T04:54:54.572828+0000 mon.a (mon.0) 2604 : audit [INF] from='client.? 172.21.15.85:0/1371435663' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.13", "id": [2, 3, 5, 0, 4, 1]}]': finished 2021-02-10T04:54:55.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:55 smithi085 conmon[29319]: cluster 2021-02-10T04:54:54.572874+0000 mon.a (mon.0) 2605 : cluster [DBG] osdmap e1218: 8 total, 7 up, 8 in 2021-02-10T04:54:55.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:55 smithi085 conmon[29319]: cluster 2021-02-10T04:54:54.591434+0000 osd.4 (osd.4) 2655 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:54:55.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:55 smithi085 conmon[29319]: cluster 2021-02-10T04:54:54.593040+0000 osd.4 (osd.4) 2656 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:54:56.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:56 smithi085 conmon[29319]: cluster 2021-02-10T04:54:55.575458+0000 mon.a (mon.0) 2606 : cluster [DBG] osdmap e1219: 8 total, 7 up, 8 in 2021-02-10T04:54:56.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:56 smithi085 conmon[35961]: cluster 2021-02-10T04:54:56.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:56 smithi085 conmon[35961]: 2021-02-10T04:54:56.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:56 smithi085 conmon[35961]: 2021-02-10T04:54:55. 2021-02-10T04:54:56.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:56 smithi085 conmon[35961]: 575458+0000 mon.a (mon.0) 2606 : cluster [DBG] osdmap e1219: 8 total, 7 up, 8 in 2021-02-10T04:54:56.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:56 smithi164 conmon[29288]: cluster 2021-02-10T04:54:55.575458+0000 mon.a ( 2021-02-10T04:54:56.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:56 smithi164 conmon[29288]: mon.0) 2606 : cluster [DBG] osdmap e1219: 8 total, 7 up, 8 in 2021-02-10T04:54:57.366 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:54:57.384 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:54:57.402 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:54:57.421 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:54:57.440 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:54:57.459 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:54:57.477 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:54:57.478 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-02-10T04:54:57.478 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: problem getting command descriptions from osd.0 2021-02-10T04:54:57.478 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:54:57.478 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:54:57.478 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:54:57.478 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:54:57.478 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:54:57.478 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:54:57.478 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:54:57.600 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is unset 2021-02-10T04:54:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:57 smithi164 conmon[29288]: cluster 2021-02-10T04:54:56.487100+0000 mgr.y (mgr.14140) 1346 : cluster [DBG] pgmap v2208: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:57 smithi164 conmon[29288]: audit 2021-02-10T04:54:57.342196+0000 mon.a (mon.0) 2607 : audit [DBG] from='client.? 172.21.15.85:0/1904203700' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:54:57.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:57 smithi164 conmon[29288]: audit 2021-02-10T04:54:57.576638+0000 mon.a (mon.0) 2608 : audit [INF] from='client.? 172.21.15.85:0/1742903310' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:54:57.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:57 smithi085 conmon[29319]: cluster 2021-02-10T04:54:56.487100+0000 mgr.y 2021-02-10T04:54:57.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:57 smithi085 conmon[29319]: (mgr.14140) 1346 : cluster [DBG] pgmap v2208: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:57.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:57 smithi085 conmon[29319]: audit 2021-02-10T04:54:57.342196+0000 mon.a (mon.0) 2607 : audit [DBG] from='client.? 172.21.15.85:0/1904203700' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:54:57.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:57 smithi085 conmon[29319]: audit 2021-02-10T04:54:57.576638+0000 mon.a (mon.0) 2608 : audit [INF] from='client.? 172.21.15.85:0/1742903310' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:54:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:57 smithi085 conmon[35961]: cluster 2021-02-10T04:54:56.487100 2021-02-10T04:54:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:57 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1346 : cluster [DBG] pgmap v2208: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:57 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:54:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:57 smithi085 conmon[35961]: 04:54:57.342196+0000 mon.a (mon.0) 2607 : audit [DBG] from='client.? 172.21.15.85:0/1904203700' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:54:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:57 smithi085 conmon[35961]: audit 2021-02-10T04:54:57.576638+0000 mon.a (mon.0) 2608 : audit [INF] from='client.? 172.21.15.85:0/1742903310' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:54:58.602 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:54:58 smithi164 conmon[37263]: debug 2021-02-10T04:54:58.230+0000 7f1c679e3700 -1 osd.5 pg_epoch: 1220 pg[1.a( empty local-lis/les=1118/1119 n=0 ec=110/19 lis/c=1118/1118 les/c/f=1119/1119/0 sis=1118) [5,2,6] r=0 lpr=1118 crt=0'0 mlcod 0'0 active+clean] scrubber pg(1.a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 2 (0x556655c52f20) 2021-02-10T04:54:58.603 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:58 smithi164 conmon[29288]: cluster 2021-02-10T04:54:57.583208+0000 mon.a 2021-02-10T04:54:58.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:58 smithi164 conmon[29288]: (mon.0) 2609 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:54:58.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:58 smithi164 conmon[29288]: audit 2021-02-10T04:54:57.592217+0000 mon.a (mon.0 2021-02-10T04:54:58.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:58 smithi164 conmon[29288]: ) 2610 : audit [INF] from='client.? 172.21.15.85:0/1742903310' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:54:58.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:58 smithi164 conmon[29288]: cluster 2021-02-10T04:54:58.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:58 smithi164 conmon[29288]: 2021-02-10T04:54:57.592271+0000 mon.a (mon.0) 2611 : cluster 2021-02-10T04:54:58.604 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:58 smithi164 conmon[29288]: [DBG] osdmap e1220: 8 total, 7 up, 8 in 2021-02-10T04:54:58.605 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:58 smithi164 conmon[29288]: cluster 2021-02-10T04:54:58. 2021-02-10T04:54:58.605 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:58 smithi164 conmon[29288]: 549600+0000 mon.a (mon.0) 2612 : cluster [DBG] osdmap e1221: 8 total, 7 up, 8 in 2021-02-10T04:54:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:58 smithi085 conmon[35961]: cluster 2021-02-10T04:54:57.583208+0000 mon.a 2021-02-10T04:54:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:58 smithi085 conmon[35961]: (mon.0) 2609 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:54:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:58 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:54:58.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:58 smithi085 conmon[35961]: :54:57.592217+0000 mon.a (mon.0) 2610 : audit 2021-02-10T04:54:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:58 smithi085 conmon[35961]: [INF] from='client.? 172.21.15.85:0/1742903310' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:54:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:58 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:54:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:58 smithi085 conmon[35961]: 02-10T04:54:57 2021-02-10T04:54:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:58 smithi085 conmon[35961]: .592271+0000 mon.a (mon.0 2021-02-10T04:54:58.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:58 smithi085 conmon[35961]: ) 2611 : cluster [DBG] osdmap e1220: 8 total, 7 up, 8 in 2021-02-10T04:54:58.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:58 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:54:58.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:58 smithi085 conmon[35961]: -02-10T04:54 2021-02-10T04:54:58.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:58 smithi085 conmon[35961]: :58.549600+0000 mon.a (mon.0) 2021-02-10T04:54:58.695 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:58 smithi085 conmon[35961]: 2612 : cluster [DBG] osdmap e1221: 8 total, 7 up, 8 in 2021-02-10T04:54:58.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:58 smithi085 conmon[29319]: cluster 2021-02-10T04:54:57.583208+0000 mon.a ( 2021-02-10T04:54:58.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:58 smithi085 conmon[29319]: mon.0) 2609 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:54:58.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:58 smithi085 conmon[29319]: audit 2021-02-10T04:54:57.592217+0000 mon.a (mon.0) 2610 : audit [INF] from='client.? 172.21.15.85:0/1742903310' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:54:58.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:58 smithi085 conmon[29319]: cluster 2021-02-10T04:54:57.592271+0000 mon.a (mon.0) 2611 : cluster [DBG] osdmap e1220: 8 total, 7 up, 8 in 2021-02-10T04:54:58.695 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:58 smithi085 conmon[29319]: cluster 2021-02-10T04:54:58.549600+0000 mon.a (mon.0) 2612 : cluster [DBG] osdmap e1221: 8 total, 7 up, 8 in 2021-02-10T04:54:58.802 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:54:58.802 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:54:59.618 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:54:59 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:54:59.618 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:59 smithi164 conmon[29288]: cluster 2021-02-10T04:54:58.487851+0000 mgr.y 2021-02-10T04:54:59.618 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:59 smithi164 conmon[29288]: (mgr.14140) 1347 : cluster [DBG] pgmap v2210: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:59.619 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:59 smithi164 conmon[29288]: cluster 2021-02-10T04:54:58.949007+0000 osd.3 (osd.3) 36 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:54:59.619 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:59 smithi164 conmon[29288]: cluster 2021-02-10T04:54:58.952886+0000 osd.3 (osd. 2021-02-10T04:54:59.619 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:59 smithi164 conmon[29288]: 3) 37 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:54:59.619 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:59 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:54:59.619 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:59 smithi164 conmon[29288]: T04:54:59.285302+0000 mgr.y ( 2021-02-10T04:54:59.619 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:54:59 smithi164 conmon[29288]: mgr.14140) 1348 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:54:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:59 smithi085 conmon[29319]: cluster 2021-02-10T04:54:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:59 smithi085 conmon[29319]: 2021-02-10T04: 2021-02-10T04:54:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:59 smithi085 conmon[29319]: 54:58.487851+0000 mgr.y (mgr.14140) 1347 : cluster [DBG] pgmap v2210: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:59 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:54:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:59 smithi085 conmon[29319]: -10T04:54:58.949007+0000 2021-02-10T04:54:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:59 smithi085 conmon[29319]: osd.3 (osd.3) 36 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:54:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:59 smithi085 conmon[29319]: 2021-02-10T04:54:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:59 smithi085 conmon[29319]: cluster 2021-02-10T04:54: 2021-02-10T04:54:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:59 smithi085 conmon[29319]: 58.952886+0000 osd.3 (osd.3) 37 2021-02-10T04:54:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:59 smithi085 conmon[29319]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:54:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:59 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:54:59.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:59 smithi085 conmon[29319]: 10T04:54:59.285302+0000 mgr.y 2021-02-10T04:54:59.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:59 smithi085 conmon[29319]: (mgr.14140) 1348 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:54:59.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:54:59 smithi085 conmon[29319]: 2021-02-10T04:54:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:59 smithi085 conmon[35961]: cluster 2021-02-10T04:54:58.487851 2021-02-10T04:54:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:59 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1347 : cluster [DBG] pgmap v2210: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:54:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:59 smithi085 conmon[35961]: cluster 2021-02-10T04:54:58 2021-02-10T04:54:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:59 smithi085 conmon[35961]: .949007+0000 osd.3 (osd.3) 36 : cluster 2021-02-10T04:54:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:59 smithi085 conmon[35961]: [DBG] purged_snaps scrub starts 2021-02-10T04:54:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:59 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:54:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:59 smithi085 conmon[35961]: :54:58.952886+0000 osd.3 (osd.3 2021-02-10T04:54:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:59 smithi085 conmon[35961]: ) 37 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:54:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:59 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:54:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:59 smithi085 conmon[35961]: 02-10T04:54:59.285302+0000 2021-02-10T04:54:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:59 smithi085 conmon[35961]: mgr.y (mgr.14140) 1348 : audit [DBG] 2021-02-10T04:54:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:54:59 smithi085 conmon[35961]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:55:00.562 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:55:00.562 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:55:00.804 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:55:00.836 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set noscrub 2021-02-10T04:55:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:00 smithi164 conmon[29288]: cluster 2021-02-10T04:54:58.976783+0000 osd.1 (osd.1) 2021-02-10T04:55:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:00 smithi164 conmon[29288]: 504 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:55:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:00 smithi164 conmon[29288]: cluster 2021-02-10T04:54:58.979489+0000 osd.1 (osd.1) 505 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:55:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:00 smithi164 conmon[29288]: cluster 2021-02-10T04:54:59.611191+0000 mon.a (mon.0) 2613 : cluster [DBG] osdmap e1222: 8 total, 7 up, 8 in 2021-02-10T04:55:00.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:00 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:55:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:00 smithi085 conmon[29319]: :54:58.976783+0000 osd.1 (osd.1) 504 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:55:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:00 smithi085 conmon[29319]: cluster 2021-02-10T04:54: 2021-02-10T04:55:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:00 smithi085 conmon[29319]: 58.979489+0000 osd.1 (osd.1) 505 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:55:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:00 smithi085 conmon[29319]: cluster 2021-02-10T04:54:59 2021-02-10T04:55:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:00 smithi085 conmon[29319]: .611191+0000 mon.a (mon.0) 2613 2021-02-10T04:55:00.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:00 smithi085 conmon[29319]: : cluster [DBG] osdmap e1222: 8 total, 7 up, 8 in 2021-02-10T04:55:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:00 smithi085 conmon[35961]: cluster 2021-02-10T04:54:58.976783 2021-02-10T04:55:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:00 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 504 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:55:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:00 smithi085 conmon[35961]: cluster 2021-02-10T04:55:00.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:00 smithi085 conmon[35961]: 2021-02-10T04:54:58. 2021-02-10T04:55:00.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:00 smithi085 conmon[35961]: 979489+0000 osd.1 (osd.1) 2021-02-10T04:55:00.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:00 smithi085 conmon[35961]: 505 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:55:00.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:00 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:55:00.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:00 smithi085 conmon[35961]: -02-10T04:54:59.611191 2021-02-10T04:55:00.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:00 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2613 : cluster 2021-02-10T04:55:00.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:00 smithi085 conmon[35961]: [DBG] osdmap e1222: 8 total, 7 up, 8 in 2021-02-10T04:55:01.056 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:55:01.061 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:55:01.600 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2021-02-10T04:55:01.600 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:55:01.600 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2021-02-10T04:55:01.601 INFO:tasks.cephadm.osd.0:Restarting osd.0 (starting--it wasn't running)... 2021-02-10T04:55:01.601 DEBUG:teuthology.orchestra.run.smithi085:osd.0> sudo journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.0.service 2021-02-10T04:55:01.603 DEBUG:teuthology.orchestra.run.smithi085:> sudo systemctl start ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.0 2021-02-10T04:55:01.624 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:01 smithi085 conmon[29319]: cluster 2021-02-10T04:55:00.488612+0000 mgr.y (mgr.14140) 1349 : cluster [DBG] pgmap v2213: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:55:01.624 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:01 smithi085 conmon[35961]: cluster 2021-02-10T04:55:00. 2021-02-10T04:55:01.625 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:01 smithi085 conmon[35961]: 488612+0000 mgr.y (mgr.14140) 1349 : cluster [DBG] pgmap v2213: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:55:01.672 INFO:journalctl@ceph.osd.0.smithi085.stdout:-- Logs begin at Wed 2021-02-10 04:09:23 UTC. -- 2021-02-10T04:55:01.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:01 smithi164 conmon[29288]: cluster 2021-02-10T04:55:00.488612+0000 mgr.y 2021-02-10T04:55:01.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:01 smithi164 conmon[29288]: (mgr.14140) 1349 : cluster [DBG] pgmap v2213: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:55:02.796 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.4 dump_historic_ops 2021-02-10T04:55:03.027 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:55:03.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:03 smithi164 conmon[29288]: cluster 2021-02-10T04:55:02.489472+0000 mgr.y (mgr.14140) 1350 : cluster [DBG] pgmap v2214: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:55:03.626 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:03 smithi164 conmon[29288]: cluster 2021-02-10T04:55:03.550134+0000 mon.a (mon.0) 2614 : cluster [DBG] osdmap e1223: 8 total, 7 up, 8 in 2021-02-10T04:55:03.627 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:03 smithi085 conmon[35961]: cluster 2021-02-10T04:55:02.489472+0000 2021-02-10T04:55:03.627 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:03 smithi085 conmon[35961]: mgr.y (mgr.14140) 1350 : cluster [DBG] pgmap v2214: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:55:03.627 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:03 smithi085 conmon[35961]: cluster 2021-02-10T04:55:03.550134+0000 mon.a (mon.0) 2614 : cluster [DBG] osdmap e1223: 8 total, 7 up, 8 in 2021-02-10T04:55:03.884 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:03 smithi085 conmon[29319]: cluster 2021-02-10T04:55:02.489472+0000 mgr.y (mgr. 2021-02-10T04:55:03.884 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:03 smithi085 conmon[29319]: 14140) 1350 : cluster [DBG] pgmap v2214: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:55:03.884 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:03 smithi085 conmon[29319]: cluster 2021-02-10T04:55:03.550134+0000 mon.a (mon.0) 2614 : cluster [DBG] osdmap e1223: 8 total, 7 up, 8 in 2021-02-10T04:55:04.358 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:55:04.388 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:55:04.418 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:55:04.436 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:55:04.454 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:55:04.472 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:55:04.490 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:55:04.491 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-02-10T04:55:04.491 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: problem getting command descriptions from osd.0 2021-02-10T04:55:04.491 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:55:04.491 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:55:04.491 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:55:04.491 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:55:04.491 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:55:04.491 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:55:04.491 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:55:04.641 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is set 2021-02-10T04:55:04.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:04 smithi164 conmon[29288]: audit 2021-02-10T04:55:04.336481+0000 mon.c (mon.1 2021-02-10T04:55:04.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:04 smithi164 conmon[29288]: ) 119 : audit [DBG] from='client.? 172.21.15.85:0/3263823994' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:55:04.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:04 smithi164 conmon[29288]: audit 2021-02-10T04:55:04.518272+0000 mon.c (mon.1) 120 : audit [INF] from='client.? 172.21.15.85:0/1833666171' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:55:04.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:04 smithi164 conmon[29288]: audit 2021-02-10T04:55:04.518566+0000 mon.a (mon.0) 2615 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:55:04.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[35961]: audit 2021-02-10T04:55:04.336481 2021-02-10T04:55:04.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[35961]: +0000 mon.c (mon.1) 119 : audit [DBG] from='client.? 172.21.15.85:0/3263823994' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:55:04.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[35961]: audit 2021-02-10T04:55:04.518272+0000 mon.c 2021-02-10T04:55:04.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[35961]: (mon.1) 120 : audit [INF] from='client.? 172.21.15.85:0/1833666171' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:55:04.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:55:04.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[35961]: 02-10T04:55:04.518566+0000 mon.a 2021-02-10T04:55:04.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[35961]: (mon.0) 2615 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:55:04.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[35961]: 2021-02-10T04:55:04.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: audit 2021-02-10T04:55:04.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 2021-02-10T04:55:04.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 2021 2021-02-10T04:55:04.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: - 2021-02-10T04:55:04.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 02 2021-02-10T04:55:04.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: - 2021-02-10T04:55:04.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 10 2021-02-10T04:55:04.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: T 2021-02-10T04:55:04.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 04 2021-02-10T04:55:04.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: : 2021-02-10T04:55:04.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 55 2021-02-10T04:55:04.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: : 2021-02-10T04:55:04.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 04 2021-02-10T04:55:04.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: . 2021-02-10T04:55:04.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 336481 2021-02-10T04:55:04.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: +0000 2021-02-10T04:55:04.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 2021-02-10T04:55:04.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: mon.c 2021-02-10T04:55:04.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: ( 2021-02-10T04:55:04.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: mon 2021-02-10T04:55:04.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: . 2021-02-10T04:55:04.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 1 2021-02-10T04:55:04.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: ) 2021-02-10T04:55:04.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 119 2021-02-10T04:55:04.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: : 2021-02-10T04:55:04.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: audit 2021-02-10T04:55:04.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 2021-02-10T04:55:04.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: [DBG] 2021-02-10T04:55:04.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 2021-02-10T04:55:04.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: from='client.? 172.21.15.85:0/3263823994' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:55:04.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 2021-02-10T04:55:04.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: audit 2021-02-10T04:55:04.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 2021-02-10T04:55:04.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 2021 2021-02-10T04:55:04.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: - 2021-02-10T04:55:04.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 02 2021-02-10T04:55:04.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: - 2021-02-10T04:55:04.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 10 2021-02-10T04:55:04.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: T 2021-02-10T04:55:04.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 04 2021-02-10T04:55:04.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: : 2021-02-10T04:55:04.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 55 2021-02-10T04:55:04.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: : 2021-02-10T04:55:04.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 04 2021-02-10T04:55:04.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: . 2021-02-10T04:55:04.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 518272 2021-02-10T04:55:04.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: +0000 2021-02-10T04:55:04.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 2021-02-10T04:55:04.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: mon.c 2021-02-10T04:55:04.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: ( 2021-02-10T04:55:04.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: mon 2021-02-10T04:55:04.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: . 2021-02-10T04:55:04.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 1 2021-02-10T04:55:04.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: ) 2021-02-10T04:55:04.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 120 2021-02-10T04:55:04.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: : 2021-02-10T04:55:04.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: audit 2021-02-10T04:55:04.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 2021-02-10T04:55:04.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: [INF] 2021-02-10T04:55:04.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 2021-02-10T04:55:04.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: from='client.? 172.21.15.85:0/1833666171' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:55:04.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 2021-02-10T04:55:04.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: audit 2021-02-10T04:55:04.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 2021-02-10T04:55:04.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 2021 2021-02-10T04:55:04.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: - 2021-02-10T04:55:04.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 02 2021-02-10T04:55:04.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: - 2021-02-10T04:55:04.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 10 2021-02-10T04:55:04.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: T 2021-02-10T04:55:04.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 04 2021-02-10T04:55:04.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: : 2021-02-10T04:55:04.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 55 2021-02-10T04:55:04.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: : 2021-02-10T04:55:04.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 04 2021-02-10T04:55:04.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: . 2021-02-10T04:55:04.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 518566 2021-02-10T04:55:04.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: +0000 2021-02-10T04:55:04.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 2021-02-10T04:55:04.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: mon.a 2021-02-10T04:55:04.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: ( 2021-02-10T04:55:04.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: mon 2021-02-10T04:55:04.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: . 2021-02-10T04:55:04.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 0 2021-02-10T04:55:04.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: ) 2021-02-10T04:55:04.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 2615 2021-02-10T04:55:04.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: : 2021-02-10T04:55:04.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: audit 2021-02-10T04:55:04.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 2021-02-10T04:55:04.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: [INF] 2021-02-10T04:55:04.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 2021-02-10T04:55:04.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:55:04.949 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:04 smithi085 conmon[29319]: 2021-02-10T04:55:05.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:05 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:55:05.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:05 smithi164 conmon[29288]: -10T04:55:04.490155+0000 mgr.y (mgr.14140) 1351 : cluster [DBG] pgmap v2216: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:55:05.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:05 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:55:05.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:05 smithi164 conmon[29288]: -10T04:55:04.624838+0000 mon.a (mon.0) 2616 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:55:05.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:05 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:55:05.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:05 smithi164 conmon[29288]: T04:55:04.630174+0000 mon.a (mon.0 2021-02-10T04:55:05.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:05 smithi164 conmon[29288]: ) 2617 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:55:05.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:05 smithi164 conmon[29288]: cluster 2021-02-10T04:55:05.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:05 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:55:05.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:05 smithi164 conmon[29288]: 55:04.630267+0000 mon.a (mon.0 2021-02-10T04:55:05.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:05 smithi164 conmon[29288]: ) 2618 : cluster [DBG] osdmap e1224: 8 total, 7 up, 8 in 2021-02-10T04:55:05.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:05 smithi085 conmon[29319]: cluster 2021-02-10T04:55:04 2021-02-10T04:55:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:05 smithi085 conmon[29319]: .490155+0000 mgr.y (mgr.14140) 1351 : cluster [DBG] pgmap v2216: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:55:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:05 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:55:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:05 smithi085 conmon[29319]: :55:04.624838+0000 mon.a (mon.0) 2616 : cluster [WRN] 2021-02-10T04:55:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:05 smithi085 conmon[29319]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:55:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:05 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:55:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:05 smithi085 conmon[29319]: 02-10T04:55:04.630174+0000 2021-02-10T04:55:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:05 smithi085 conmon[29319]: mon.a (mon.0) 2617 : audit 2021-02-10T04:55:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:05 smithi085 conmon[29319]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:55:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:05 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:55:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:05 smithi085 conmon[29319]: 02- 2021-02-10T04:55:05.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:05 smithi085 conmon[29319]: 10T04 2021-02-10T04:55:05.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:05 smithi085 conmon[29319]: :55:04.630267+0000 mon.a (mon. 2021-02-10T04:55:05.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:05 smithi085 conmon[29319]: 0) 2618 : cluster [DBG] osdmap e1224: 8 total, 7 up, 8 in 2021-02-10T04:55:05.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:05 smithi085 conmon[29319]: 2021-02-10T04:55:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:05 smithi085 conmon[35961]: cluster 2021-02-10T04:55:04.490155+0000 2021-02-10T04:55:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:05 smithi085 conmon[35961]: mgr.y (mgr.14140) 1351 : cluster [DBG] pgmap v2216: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:55:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:05 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:55:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:05 smithi085 conmon[35961]: :55:04.624838+0000 mon.a (mon.0) 2616 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:55:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:05 smithi085 conmon[35961]: audit 2021 2021-02-10T04:55:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:05 smithi085 conmon[35961]: -02-10T04:55:04.630174+0000 2021-02-10T04:55:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:05 smithi085 conmon[35961]: mon.a (mon.0) 2617 : audit [INF] 2021-02-10T04:55:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:05 smithi085 conmon[35961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:55:05.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:05 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:55:05.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:05 smithi085 conmon[35961]: -02-10T04:55:04.630267 2021-02-10T04:55:05.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:05 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2618 : cluster 2021-02-10T04:55:05.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:05 smithi085 conmon[35961]: [DBG] osdmap e1224: 8 total, 7 up, 8 in 2021-02-10T04:55:06.201 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:55:06.202 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:55:07.441 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:07 smithi085 bash[176331]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2021-02-10T04:55:07.441 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:07 smithi085 bash[176331]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2021-02-10T04:55:07.442 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:07 smithi085 bash[176331]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2021-02-10T04:55:07.442 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:07 smithi085 bash[176331]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2021-02-10T04:55:07.442 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:07 smithi085 bash[176331]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2021-02-10T04:55:07.442 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:07 smithi085 bash[176331]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2021-02-10T04:55:07.442 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:07 smithi085 bash[176331]: --> ceph-volume lvm activate successful for osd ID: 0 2021-02-10T04:55:07.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:07 smithi164 conmon[29288]: cluster 2021-02-10T04:55:06.490988 2021-02-10T04:55:07.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:07 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1352 : cluster [DBG] pgmap v2218: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 744 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:55:07.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:07 smithi085 conmon[29319]: cluster 2021-02-10T04:55:06.490988+0000 mgr.y ( 2021-02-10T04:55:07.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:07 smithi085 conmon[29319]: mgr.14140) 1352 : cluster [DBG] pgmap v2218: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 744 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:55:07.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:07 smithi085 conmon[35961]: cluster 2021-02-10T04:55:06.490988+0000 2021-02-10T04:55:07.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:07 smithi085 conmon[35961]: mgr.y (mgr.14140) 1352 : cluster [DBG] pgmap v2218: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 744 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:55:08.202 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:55:08.275 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set nodeep-scrub 2021-02-10T04:55:08.443 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:55:08.446 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.0 dump_ops_in_flight 2021-02-10T04:55:08.619 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:55:08.660 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:08 smithi085 bash[176331]: 330bb351d96a77c87a96463d50838f8ea5fc6291ec64732d103fcdf377e53f96 2021-02-10T04:55:08.660 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:08 smithi085 systemd[1]: Started Ceph osd.0 for 42224fce-6b57-11eb-8feb-001a4aab830c. 2021-02-10T04:55:08.660 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:08 smithi085 conmon[177076]: debug 2021-02-10T04:55:08.450+0000 7f14ce75f700 20 -- 172.21.15.85:0/1840881886 >> [v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0] conn(0x561e56a12c00 msgr2=0x561e5697d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2021-02-10T04:55:08.661 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:08 smithi085 conmon[177076]: debug 2021-02-10T04:55:08.450+0000 7f14ce75f700 20 -- 172.21.15.85:0/1840881886 >> [v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0] conn(0x561e56a12c00 msgr2=0x561e5697d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=96 2021-02-10T04:55:08.661 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:08 smithi085 conmon[177076]: debug 2021-02-10T04:55:08.450+0000 7f14ce75f700 20 --2- 172.21.15.85:0/1840881886 >> [v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0] conn(0x561e56a12c00 0x561e5697d400 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x561e56993d40 tx=0x561e569c4d80).handle_read_frame_preamble_main r=0 2021-02-10T04:55:08.661 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:08 smithi085 conmon[177076]: debug 2021-02-10T04:55:08.450+0000 7f14ce75f700 20 --2- 172.21.15.85:0/1840881886 >> [v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0] conn(0x561e56a12c00 0x561e5697d400 secure :-1 s=THROTTLE_MESSAGE pgs=332 cs=0 l=1 rev1=1 rx=0x561e56993d40 tx=0x561e569c4d80).throttle_message 2021-02-10T04:55:08.661 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:08 smithi085 conmon[177076]: debug 2021-02-10T04:55:08.450+0000 7f14ce75f700 20 --2- 172.21.15.85:0/1840881886 >> [v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0] conn(0x561e56a12c00 0x561e5697d400 secure :-1 s=THROTTLE_BYTES pgs=332 cs=0 l=1 rev1=1 rx=0x561e56993d40 tx=0x561e569c4d80).throttle_bytes 2021-02-10T04:55:08.661 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:08 smithi085 conmon[177076]: debug 2021-02-10T04:55:08.450+0000 7f14ce75f700 20 --2- 172.21.15.85:0/1840881886 >> [v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0] conn(0x561e56a12c00 0x561e5697d400 secure :-1 s=THROTTLE_DISPATCH_QUEUE pgs=332 cs=0 l=1 rev1=1 rx=0x561e56993d40 tx=0x561e569c4d80).throttle_dispatch_queue 2021-02-10T04:55:08.661 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:08 smithi085 conmon[177076]: debug 2021-02-10T04:55:08.450+0000 7f14ce75f700 20 --2- 172.21.15.85:0/1840881886 >> [v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0] conn(0x561e56a12c00 0x561e5697d400 secure :-1 s=THROTTLE_DONE pgs=332 cs=0 l=1 rev1=1 rx=0x561e56993d40 tx=0x561e569c4d80).read_frame_segment seg_idx=0 2021-02-10T04:55:08.661 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:08 smithi085 conmon[177076]: debug 2021-02-10T04:55:08.450+0000 7f14ce75f700 20 --2- 172.21.15.85:0/1840881886 >> [v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0] conn(0x561e56a12c00 0x561e5697d400 secure :-1 s=THROTTLE_DONE pgs=332 cs=0 l=1 rev1=1 rx=0x561e56993d40 tx=0x561e569c4d80).read_frame_segment seg_idx=1 2021-02-10T04:55:08.661 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:08 smithi085 conmon[177076]: debug 2021-02-10T04:55:08.450+0000 7f14ce75f700 20 -- 172.21.15.85:0/1840881886 >> [v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0] conn(0x561e56a12c00 msgr2=0x561e5697d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=208 2021-02-10T04:55:08.661 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:08 smithi085 conmon[177076]: debug 2021-02-10T04:55:08.450+0000 7f14ce75f700 20 --2- 172.21.15.85:0/1840881886 >> [v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0] conn(0x561e56a12c00 0x561e5697d400 secure :-1 s=THROTTLE_DONE pgs=332 cs=0 l=1 rev1=1 rx=0x561e56993d40 tx=0x561e569c4d80).handle_read_frame_segment r=0 2021-02-10T04:55:08.661 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:08 smithi085 conmon[177076]: debug 2021-02-10T04:55:08.450+0000 7f14ce75f700 20 -- 172.21.15.85:0/1840881886 >> [v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0] conn(0x561e56a12c00 msgr2=0x561e5697d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=32 2021-02-10T04:55:08.661 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:08 smithi085 conmon[177076]: debug 2021-02-10T04:55:08.450+0000 7f14ce75f700 20 --2- 172.21.15.85:0/1840881886 >> [v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0] conn(0x561e56a12c00 0x561e5697d400 secure :-1 s=THROTTLE_DONE pgs=332 cs=0 l=1 rev1=1 rx=0x561e56993d40 tx=0x561e569c4d80).handle_read_frame_epilogue_main r=0 2021-02-10T04:55:08.661 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:08 smithi085 conmon[177076]: debug 2021-02-10T04:55:08.450+0000 7f14ce75f700 10 --2- 172.21.15.85:0/1840881886 >> [v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0] conn(0x561e56a12c00 0x561e5697d400 secure :-1 s=THROTTLE_DONE pgs=332 cs=0 l=1 rev1=1 rx=0x561e56993d40 tx=0x561e569c4d80).handle_read_frame_dispatch tag=17 2021-02-10T04:55:08.662 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:08 smithi085 conmon[177076]: debug 2021-02-10T04:55:08.450+0000 7f14ce75f700 20 --2- 172.21.15.85:0/1840881886 >> [v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0] conn(0x561e56a12c00 0x561e5697d400 secure :-1 s=THROTTLE_DONE pgs=332 cs=0 l=1 rev1=1 rx=0x561e56993d40 tx=0x561e569c4d80).handle_message 2021-02-10T04:55:08.662 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:08 smithi085 conmon[177076]: debug 2021-02-10T04:55:08.450+0000 7f14ce75f700 5 --2- 172.21.15.85:0/1840881886 >> [v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0] conn(0x561e56a12c00 0x561e5697d400 secure :-1 s=THROTTLE_DONE pgs=332 cs=0 l=1 rev1=1 rx=0x561e56993d40 tx=0x561e569c4d80).handle_message got 194 + 0 + 0 byte message. envelope type=18 src mon.0 off 0 2021-02-10T04:55:08.662 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:08 smithi085 conmon[177076]: debug 2021-02-10T04:55:08.450+0000 7f14ce75f700 5 --2- 172.21.15.85:0/1840881886 >> [v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0] conn(0x561e56a12c00 0x561e5697d400 secure :-1 s=READ_MESSAGE_COMPLETE pgs=332 cs=0 l=1 rev1=1 rx=0x561e56993d40 tx=0x561e569c4d80).handle_message received message m=0x561e569964e0 seq=4 from=mon.0 type=18 auth_reply(proto 2 0 (0) Success) v1 2021-02-10T04:55:08.662 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:08 smithi085 conmon[177076]: debug 2021-02-10T04:55:08.450+0000 7f14ce75f700 20 -- 172.21.15.85:0/1840881886 queue 0x561e569964e0 prio 196 2021-02-10T04:55:08.662 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:08 smithi085 conmon[177076]: debug 2021-02-10T04:55:08.450+0000 7f14ce75f700 20 --2- 172.21.15.85:0/1840881886 >> [v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0] conn(0x561e56a12c00 0x561e5697d400 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x561e56993d40 tx=0x561e569c4d80).read_frame 2021-02-10T04:55:08.662 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:08 smithi085 conmon[177076]: debug 2021-02-10T04:55:08.450+0000 7f14ce75f700 20 -- 172.21.15.85:0/1840881886 >> [v2:172.21.15.85:3300/0,v1:172.21.15.85:6789/0] conn(0x561e56a12c00 msgr2=0x561e5697d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=96 2021-02-10T04:55:08.662 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:08 smithi085 conmon[177076]: debug 2021-02-10T04:55:08.450+0000 7f14cd75d700 1 -- 172.21.15.85:0/1840881886 <== mon.0 v2:172.21.15.85:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561e569964e0 con 0x561e56a12c00 2021-02-10T04:55:08.662 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:08 smithi085 conmon[177076]: debug 2021-02-10T04:55:08.450+0000 7f14cd75d700 10 -- 172.21.15.85:0/1840881886 dispatch_throttle_release 194 to dispatch throttler 194/104857600 2021-02-10T04:55:08.662 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:08 smithi085 conmon[177076]: debug 2021-02-10T04:55:08.450+0000 7f14cd75d700 20 -- 172.21.15.85:0/1840881886 done calling dispatch on 0x561e569964e0 2021-02-10T04:55:08.738 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:55:09.310 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:08 smithi085 conmon[177076]: debug 2021-02-10T04:55:09.310 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:08 smithi085 conmon[177076]: 2021-02-10T04:55:08.981+0000 7f14d51e8200 -1 Falling back to public interface 2021-02-10T04:55:09.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:55:09 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:55:09.552 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:09 smithi164 conmon[29288]: cluster 2021-02-10T04:55:08.492011+0000 2021-02-10T04:55:09.552 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:09 smithi164 conmon[29288]: mgr.y (mgr.14140) 1353 : cluster [DBG] pgmap v2219: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:55:09.552 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:09 smithi164 conmon[29288]: cluster 2021-02-10T04:55:08.556012+0000 mon.a (mon. 2021-02-10T04:55:09.553 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:09 smithi164 conmon[29288]: 0) 2619 : cluster [DBG] osdmap e1225: 8 total, 7 up, 8 in 2021-02-10T04:55:09.553 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:09 smithi164 conmon[29288]: audit 2021 2021-02-10T04:55:09.553 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:09 smithi164 conmon[29288]: -02-10T04:55:09.294967+0000 mgr.y (mgr.14140) 1354 : 2021-02-10T04:55:09.553 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:09 smithi164 conmon[29288]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:55:09.579 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:55:09.580 INFO:teuthology.orchestra.run.smithi164.stdout: "size": 20, 2021-02-10T04:55:09.580 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 600, 2021-02-10T04:55:09.580 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [ 2021-02-10T04:55:09.580 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.580 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.15 1038/1037 [PushOp(7:adfc1887:::smithi08575397-25:218, version: 778'234, data_included: [2097152~793652], data_size: 793652, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:adfc1887:::smithi08575397-25:218@778'234, size: 2890804, copy_subset: [0~2890804], clone_subset: {}, snapset: 374=[]:{218=[203],27c=[27c],2cb=[2c8,2ae],2ff=[2f5],328=[328],335=[335,332,32f,32a],33e=[33e,33c,33a],343=[343],374=[374,373,372,370,36f,36b,368,366,35f,354,353,34e,34d,34b,349,345,344]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2890804, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:55:09.580 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:45:40.761541+0000", 2021-02-10T04:55:09.580 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 568.81546235799999, 2021-02-10T04:55:09.580 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010475346, 2021-02-10T04:55:09.580 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:55:09.580 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:55:09.581 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:55:09.581 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.581 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:55:09.581 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:40.761541+0000", 2021-02-10T04:55:09.581 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:55:09.581 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.581 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.581 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:55:09.581 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:40.761541+0000", 2021-02-10T04:55:09.581 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00093548000000000004 2021-02-10T04:55:09.581 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.581 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.582 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:55:09.582 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:40.762477+0000", 2021-02-10T04:55:09.582 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.3981e-05 2021-02-10T04:55:09.582 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.582 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.582 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:55:09.582 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:40.762501+0000", 2021-02-10T04:55:09.582 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.7656e-05 2021-02-10T04:55:09.582 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.582 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.582 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:55:09.583 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:40.762538+0000", 2021-02-10T04:55:09.584 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6436999999999999e-05 2021-02-10T04:55:09.584 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.584 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.584 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:55:09.584 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:40.762555+0000", 2021-02-10T04:55:09.584 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0086926 2021-02-10T04:55:09.584 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.584 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.585 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:55:09.585 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:40.771247+0000", 2021-02-10T04:55:09.585 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.9180000000000006e-06 2021-02-10T04:55:09.585 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.585 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.585 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:55:09.585 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:40.771257+0000", 2021-02-10T04:55:09.585 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00075927399999999999 2021-02-10T04:55:09.585 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.585 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.586 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:55:09.586 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:40.772017+0000", 2021-02-10T04:55:09.586 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010475346 2021-02-10T04:55:09.586 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.586 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:55:09.586 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.586 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.586 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.586 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.15 1038/1037 [PushOp(7:adfc1887:::smithi08575397-25:33e, version: 961'409, data_included: [1486901~894536,2801290~154040], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:adfc1887:::smithi08575397-25:33e@961'409, size: 3311276, copy_subset: [0~3311276], clone_subset: {7:adfc1887:::smithi08575397-25:328=[]}, snapset: 374=[]:{218=[203],27c=[27c],2cb=[2c8,2ae],2ff=[2f5],328=[328],335=[335,332,32f,32a],33e=[33e,33c,33a],343=[343],374=[374,373,372,370,36f,36b,368,366,35f,354,353,34e,34d,34b,349,345,344]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2955330, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1486901, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:55:09.586 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:45:41.084420+0000", 2021-02-10T04:55:09.587 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 568.49258331500005, 2021-02-10T04:55:09.587 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010577269, 2021-02-10T04:55:09.587 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:55:09.587 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:55:09.587 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:55:09.587 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.587 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:55:09.587 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.084420+0000", 2021-02-10T04:55:09.587 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:55:09.587 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.587 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.587 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:55:09.588 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.084420+0000", 2021-02-10T04:55:09.588 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000988224 2021-02-10T04:55:09.588 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.588 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.588 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:55:09.588 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.085409+0000", 2021-02-10T04:55:09.588 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4181000000000001e-05 2021-02-10T04:55:09.588 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.588 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.588 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:55:09.588 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.085433+0000", 2021-02-10T04:55:09.588 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.2017000000000002e-05 2021-02-10T04:55:09.588 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.589 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.589 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:55:09.589 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.085465+0000", 2021-02-10T04:55:09.589 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6639000000000001e-05 2021-02-10T04:55:09.589 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.589 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.589 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:55:09.589 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.085481+0000", 2021-02-10T04:55:09.589 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0079138650000000008 2021-02-10T04:55:09.589 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.589 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.589 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:55:09.590 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.093395+0000", 2021-02-10T04:55:09.590 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2374999999999999e-05 2021-02-10T04:55:09.590 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.590 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.590 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:55:09.590 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.093408+0000", 2021-02-10T04:55:09.590 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001589968 2021-02-10T04:55:09.590 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.590 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.590 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:55:09.590 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.094998+0000", 2021-02-10T04:55:09.591 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010577269 2021-02-10T04:55:09.591 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.591 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:55:09.591 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.591 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.591 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.591 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.15 1038/1037 [PushOp(7:adfc1887:::smithi08575397-25:2ff, version: 995'435, data_included: [2154604~68996,2888554~512829], data_size: 581825, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:adfc1887:::smithi08575397-25:2ff@995'435, size: 3496120, copy_subset: [0~3496120], clone_subset: {7:adfc1887:::smithi08575397-25:2cb=[],7:adfc1887:::smithi08575397-25:328=[]}, snapset: 374=[]:{218=[203],27c=[27c],2cb=[2c8,2ae],2ff=[2f5],328=[328],335=[335,332,32f,32a],33e=[33e,33c,33a],343=[343],374=[374,373,372,370,36f,36b,368,366,35f,354,353,34e,34d,34b,349,345,344]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3496120, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2154604, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:55:09.591 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:45:41.177698+0000", 2021-02-10T04:55:09.591 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 568.39930592099995, 2021-02-10T04:55:09.592 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.013292897, 2021-02-10T04:55:09.592 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:55:09.592 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:55:09.592 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:55:09.592 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.592 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:55:09.592 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.177698+0000", 2021-02-10T04:55:09.592 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:55:09.592 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.592 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.592 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:55:09.592 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.177698+0000", 2021-02-10T04:55:09.592 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00050759299999999998 2021-02-10T04:55:09.593 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.593 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.593 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:55:09.593 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.178205+0000", 2021-02-10T04:55:09.593 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.7773999999999999e-05 2021-02-10T04:55:09.593 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.593 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.593 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:55:09.593 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.178233+0000", 2021-02-10T04:55:09.593 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.2228999999999998e-05 2021-02-10T04:55:09.593 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.593 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.594 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:55:09.594 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.178275+0000", 2021-02-10T04:55:09.594 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.9355e-05 2021-02-10T04:55:09.594 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.594 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.594 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:55:09.594 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.178295+0000", 2021-02-10T04:55:09.594 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0086878040000000004 2021-02-10T04:55:09.594 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.594 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.594 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:55:09.595 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.186982+0000", 2021-02-10T04:55:09.595 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.8190000000000003e-06 2021-02-10T04:55:09.595 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.595 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.595 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:55:09.595 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.186992+0000", 2021-02-10T04:55:09.595 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0039983229999999998 2021-02-10T04:55:09.595 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.595 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.595 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:55:09.595 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.190991+0000", 2021-02-10T04:55:09.595 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.013292897 2021-02-10T04:55:09.596 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.596 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:55:09.596 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.596 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.596 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.596 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.15 1038/1037 [PushOp(7:adfc1887:::smithi08575397-25:343, version: 998'441, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:adfc1887:::smithi08575397-25:343@998'441, size: 3529767, copy_subset: [0~3529767], clone_subset: {7:adfc1887:::smithi08575397-25:33e=[]}, snapset: 374=[]:{218=[203],27c=[27c],2cb=[2c8,2ae],2ff=[2f5],328=[328],335=[335,332,32f,32a],33e=[33e,33c,33a],343=[343],374=[374,373,372,370,36f,36b,368,366,35f,354,353,34e,34d,34b,349,345,344]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:55:09.596 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:45:41.284850+0000", 2021-02-10T04:55:09.596 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 568.29215402700004, 2021-02-10T04:55:09.596 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0099639719999999998, 2021-02-10T04:55:09.596 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:55:09.596 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:55:09.597 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:55:09.597 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.597 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:55:09.597 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.284850+0000", 2021-02-10T04:55:09.597 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:55:09.597 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.597 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.597 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:55:09.597 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.284850+0000", 2021-02-10T04:55:09.597 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00104846 2021-02-10T04:55:09.597 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.597 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.598 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:55:09.598 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.285898+0000", 2021-02-10T04:55:09.598 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.8198999999999999e-05 2021-02-10T04:55:09.598 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.598 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.598 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:55:09.598 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.285926+0000", 2021-02-10T04:55:09.598 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.4586999999999998e-05 2021-02-10T04:55:09.598 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.598 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.598 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:55:09.599 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.285971+0000", 2021-02-10T04:55:09.599 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6572e-05 2021-02-10T04:55:09.599 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.599 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.599 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:55:09.599 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.285987+0000", 2021-02-10T04:55:09.599 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0081604750000000004 2021-02-10T04:55:09.599 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.599 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.599 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:55:09.599 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.294148+0000", 2021-02-10T04:55:09.600 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1707e-05 2021-02-10T04:55:09.600 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.600 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.600 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:55:09.600 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.294160+0000", 2021-02-10T04:55:09.600 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00065397199999999995 2021-02-10T04:55:09.600 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.600 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.600 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:55:09.600 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.294814+0000", 2021-02-10T04:55:09.600 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0099639719999999998 2021-02-10T04:55:09.600 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.600 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:55:09.601 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.601 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.601 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.601 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPush(7.15 1038/1037 [PushOp(7:adfc1887:::smithi08575397-25:374, version: 1000'443, data_included: [2051410~13328,2496305~525139,3472104~292961], data_size: 831428, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:adfc1887:::smithi08575397-25:374@1000'443, size: 3765065, copy_subset: [565065~427465,1430299~634439,2496305~525139,3472104~292961], clone_subset: {7:adfc1887:::smithi08575397-25:343=[0~565065,992530~437769,2064738~431567,3021444~450660]}, snapset: 374=[]:{218=[203],27c=[27c],2cb=[2c8,2ae],2ff=[2f5],328=[328],335=[335,332,32f,32a],33e=[33e,33c,33a],343=[343],374=[374,373,372,370,36f,36b,368,366,35f,354,353,34e,34d,34b,349,345,344]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3765065, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2051410, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-02-10T04:55:09.601 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:45:41.320627+0000", 2021-02-10T04:55:09.601 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 568.25637701000005, 2021-02-10T04:55:09.601 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.013224167, 2021-02-10T04:55:09.601 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:55:09.601 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:55:09.601 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:55:09.602 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.602 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:55:09.602 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.320627+0000", 2021-02-10T04:55:09.602 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:55:09.602 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.602 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.602 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:55:09.602 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.320627+0000", 2021-02-10T04:55:09.602 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000743616 2021-02-10T04:55:09.603 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.603 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.603 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:55:09.603 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.321370+0000", 2021-02-10T04:55:09.603 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.4378999999999997e-05 2021-02-10T04:55:09.603 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.603 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.603 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:55:09.603 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.321405+0000", 2021-02-10T04:55:09.603 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0112000000000003e-05 2021-02-10T04:55:09.603 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.603 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.604 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:55:09.604 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.321465+0000", 2021-02-10T04:55:09.604 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2058000000000001e-05 2021-02-10T04:55:09.604 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.604 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.604 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:55:09.604 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.321487+0000", 2021-02-10T04:55:09.604 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0083593250000000008 2021-02-10T04:55:09.604 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.604 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.604 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:55:09.604 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.329846+0000", 2021-02-10T04:55:09.605 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1922999999999999e-05 2021-02-10T04:55:09.605 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.605 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.605 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:55:09.605 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.329858+0000", 2021-02-10T04:55:09.605 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0039927540000000003 2021-02-10T04:55:09.605 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.605 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.605 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:55:09.605 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:45:41.333851+0000", 2021-02-10T04:55:09.605 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.013224167 2021-02-10T04:55:09.605 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.606 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:55:09.606 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.606 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.606 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.606 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPushReply(7.15 1121/1119 [PushReplyOp(7:adfc1887:::smithi08575397-25:33e)])", 2021-02-10T04:55:09.606 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:51:38.576167+0000", 2021-02-10T04:55:09.606 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 211.00083703300001, 2021-02-10T04:55:09.606 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010982059000000001, 2021-02-10T04:55:09.606 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:55:09.606 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T04:55:09.607 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:55:09.607 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.607 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:55:09.607 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.576167+0000", 2021-02-10T04:55:09.607 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:55:09.607 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.607 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.607 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:55:09.607 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.576167+0000", 2021-02-10T04:55:09.607 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.3866999999999999e-05 2021-02-10T04:55:09.607 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.607 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.608 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:55:09.608 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.576200+0000", 2021-02-10T04:55:09.608 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7049999999999999e-06 2021-02-10T04:55:09.608 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.608 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.608 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:55:09.608 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.576208+0000", 2021-02-10T04:55:09.608 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1736000000000001e-05 2021-02-10T04:55:09.608 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.608 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.608 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:55:09.609 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.576220+0000", 2021-02-10T04:55:09.609 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.1890000000000004e-06 2021-02-10T04:55:09.609 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.609 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.609 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:55:09.609 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.576226+0000", 2021-02-10T04:55:09.609 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0097831849999999998 2021-02-10T04:55:09.609 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.609 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.609 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:55:09.609 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.586009+0000", 2021-02-10T04:55:09.610 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001139377 2021-02-10T04:55:09.610 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.610 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.610 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:55:09.610 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.587149+0000", 2021-02-10T04:55:09.610 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010982059000000001 2021-02-10T04:55:09.610 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.610 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:55:09.610 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.610 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.610 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.610 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPushReply(7.15 1121/1119 [PushReplyOp(7:adfc1887:::smithi08575397-25:2cb)])", 2021-02-10T04:55:09.611 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:51:38.601205+0000", 2021-02-10T04:55:09.611 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 210.97579904400001, 2021-02-10T04:55:09.611 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0090249059999999992, 2021-02-10T04:55:09.611 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:55:09.611 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T04:55:09.611 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:55:09.611 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.611 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:55:09.611 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.601205+0000", 2021-02-10T04:55:09.611 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:55:09.611 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.612 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.612 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:55:09.612 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.601205+0000", 2021-02-10T04:55:09.612 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.6381999999999998e-05 2021-02-10T04:55:09.612 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.612 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.612 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:55:09.612 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.601241+0000", 2021-02-10T04:55:09.612 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.4120000000000001e-06 2021-02-10T04:55:09.612 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.612 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.612 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:55:09.613 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.601249+0000", 2021-02-10T04:55:09.613 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3551e-05 2021-02-10T04:55:09.613 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.613 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.613 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:55:09.613 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.601263+0000", 2021-02-10T04:55:09.613 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.3390000000000004e-06 2021-02-10T04:55:09.613 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.614 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.614 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:55:09.614 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.601270+0000", 2021-02-10T04:55:09.614 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0085583159999999998 2021-02-10T04:55:09.614 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.614 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.614 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:55:09.614 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.609829+0000", 2021-02-10T04:55:09.614 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00040090600000000003 2021-02-10T04:55:09.614 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.614 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.615 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:55:09.615 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.610229+0000", 2021-02-10T04:55:09.615 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0090249059999999992 2021-02-10T04:55:09.615 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.615 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:55:09.615 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.615 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.615 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.615 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPushReply(7.15 1121/1119 [PushReplyOp(7:adfc1887:::smithi08575397-25:343)])", 2021-02-10T04:55:09.615 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:51:38.760603+0000", 2021-02-10T04:55:09.615 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 210.81640074699999, 2021-02-10T04:55:09.616 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0094503050000000009, 2021-02-10T04:55:09.616 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:55:09.616 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T04:55:09.616 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:55:09.616 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.616 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:55:09.616 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.760603+0000", 2021-02-10T04:55:09.616 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:55:09.616 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.616 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.616 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:55:09.617 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.760603+0000", 2021-02-10T04:55:09.617 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1231e-05 2021-02-10T04:55:09.617 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.617 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.617 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:55:09.617 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.760634+0000", 2021-02-10T04:55:09.617 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.7470000000000005e-06 2021-02-10T04:55:09.617 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.617 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.617 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:55:09.617 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.760642+0000", 2021-02-10T04:55:09.617 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2286000000000001e-05 2021-02-10T04:55:09.618 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.618 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.618 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:55:09.618 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.760654+0000", 2021-02-10T04:55:09.618 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.4500000000000001e-06 2021-02-10T04:55:09.618 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.618 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.618 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:55:09.618 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.760661+0000", 2021-02-10T04:55:09.618 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0090138119999999995 2021-02-10T04:55:09.619 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.619 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.619 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:55:09.619 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.769674+0000", 2021-02-10T04:55:09.619 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00037877899999999999 2021-02-10T04:55:09.619 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.619 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.619 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:55:09.619 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.770053+0000", 2021-02-10T04:55:09.619 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0094503050000000009 2021-02-10T04:55:09.619 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.620 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:55:09.620 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.620 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.620 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.620 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:1822 7.8 e1150/1118 7:10dac65a:::smithi08575397-39:head v 1150'76, mlcod=0'0)", 2021-02-10T04:55:09.620 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:52:41.692341+0000", 2021-02-10T04:55:09.620 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 147.88466269200001, 2021-02-10T04:55:09.620 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0099972789999999995, 2021-02-10T04:55:09.620 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:55:09.620 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T04:55:09.620 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:55:09.620 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.621 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:55:09.621 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.692341+0000", 2021-02-10T04:55:09.621 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:55:09.621 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.621 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.621 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:55:09.621 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.692341+0000", 2021-02-10T04:55:09.621 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0058453940000000003 2021-02-10T04:55:09.621 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.621 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.621 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:55:09.622 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.698186+0000", 2021-02-10T04:55:09.622 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.2530000000000004e-06 2021-02-10T04:55:09.622 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.622 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.622 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:55:09.622 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.698195+0000", 2021-02-10T04:55:09.622 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.9080000000000001e-06 2021-02-10T04:55:09.622 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.622 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.622 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:55:09.622 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.698203+0000", 2021-02-10T04:55:09.622 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6869999999999997e-06 2021-02-10T04:55:09.623 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.623 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.623 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:55:09.623 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.698210+0000", 2021-02-10T04:55:09.623 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9551e-05 2021-02-10T04:55:09.623 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.623 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.623 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:55:09.623 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.698260+0000", 2021-02-10T04:55:09.623 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.5418999999999998e-05 2021-02-10T04:55:09.624 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.624 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.624 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:55:09.624 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.698285+0000", 2021-02-10T04:55:09.624 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0039851779999999998 2021-02-10T04:55:09.624 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.624 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.624 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T04:55:09.624 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.702270+0000", 2021-02-10T04:55:09.624 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.7889e-05 2021-02-10T04:55:09.624 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.624 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.625 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:55:09.625 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.702338+0000", 2021-02-10T04:55:09.625 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0099972789999999995 2021-02-10T04:55:09.625 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.625 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:55:09.625 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.625 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.625 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.625 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.24373.0:3771 5.e 5:7759931f:::notify.3:head [watch ping cookie 94187770077184 gen 1] snapc 0=[] ondisk+write+known_if_redirected e1205)", 2021-02-10T04:55:09.625 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:54:33.025815+0000", 2021-02-10T04:55:09.626 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 36.551188848000002, 2021-02-10T04:55:09.626 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00029987699999999999, 2021-02-10T04:55:09.626 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:55:09.626 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:55:09.626 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:55:09.626 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.24373", 2021-02-10T04:55:09.626 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:55:09.626 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 3771 2021-02-10T04:55:09.627 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.627 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:55:09.627 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.627 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:55:09.627 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:33.025815+0000", 2021-02-10T04:55:09.627 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:55:09.627 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.627 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.627 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:55:09.627 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:33.025815+0000", 2021-02-10T04:55:09.627 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6990000000000002e-05 2021-02-10T04:55:09.627 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.628 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.628 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:55:09.628 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:33.025832+0000", 2021-02-10T04:55:09.628 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.2319999999999998e-06 2021-02-10T04:55:09.628 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.628 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.628 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:55:09.628 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:33.025838+0000", 2021-02-10T04:55:09.628 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0202e-05 2021-02-10T04:55:09.628 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.628 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.628 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:55:09.629 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:33.025848+0000", 2021-02-10T04:55:09.629 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.5359999999999998e-06 2021-02-10T04:55:09.629 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.629 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.629 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:55:09.629 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:33.025855+0000", 2021-02-10T04:55:09.629 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.8685999999999998e-05 2021-02-10T04:55:09.629 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.629 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.629 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:55:09.629 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:33.025943+0000", 2021-02-10T04:55:09.630 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1975000000000001e-05 2021-02-10T04:55:09.630 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.630 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.630 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:55:09.630 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:33.026015+0000", 2021-02-10T04:55:09.630 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.9256e-05 2021-02-10T04:55:09.630 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.630 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.630 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:55:09.630 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:33.026115+0000", 2021-02-10T04:55:09.630 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00029987699999999999 2021-02-10T04:55:09.630 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.631 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:55:09.631 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.631 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.631 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.631 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.24373.0:3779 5.e 5:7759931f:::notify.3:head [watch ping cookie 94187770077184 gen 1] snapc 0=[] ondisk+write+known_if_redirected e1207)", 2021-02-10T04:55:09.631 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:54:38.026690+0000", 2021-02-10T04:55:09.631 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 31.550313854999999, 2021-02-10T04:55:09.631 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00061227099999999995, 2021-02-10T04:55:09.631 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:55:09.631 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:55:09.631 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:55:09.631 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.24373", 2021-02-10T04:55:09.632 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:55:09.632 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 3779 2021-02-10T04:55:09.632 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.632 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:55:09.632 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.632 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:55:09.632 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:38.026690+0000", 2021-02-10T04:55:09.632 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:55:09.632 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.632 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.632 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:55:09.632 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:38.026690+0000", 2021-02-10T04:55:09.633 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8482999999999999e-05 2021-02-10T04:55:09.633 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.633 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.633 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:55:09.633 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:38.026708+0000", 2021-02-10T04:55:09.633 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2724e-05 2021-02-10T04:55:09.633 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.633 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.633 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:55:09.633 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:38.026721+0000", 2021-02-10T04:55:09.633 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6141e-05 2021-02-10T04:55:09.633 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.634 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.634 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:55:09.634 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:38.026737+0000", 2021-02-10T04:55:09.634 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.1929999999999999e-06 2021-02-10T04:55:09.634 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.634 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.634 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:55:09.634 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:38.026746+0000", 2021-02-10T04:55:09.634 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000310561 2021-02-10T04:55:09.634 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.634 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.635 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:55:09.635 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:38.027057+0000", 2021-02-10T04:55:09.635 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.2785e-05 2021-02-10T04:55:09.635 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.635 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.635 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:55:09.635 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:38.027150+0000", 2021-02-10T04:55:09.635 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00015238400000000001 2021-02-10T04:55:09.635 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.635 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.636 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:55:09.636 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:38.027302+0000", 2021-02-10T04:55:09.636 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00061227099999999995 2021-02-10T04:55:09.636 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.636 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:55:09.636 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.636 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.636 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.636 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.14146.0:58 3.1 3:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected e1210)", 2021-02-10T04:55:09.637 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:54:38.253925+0000", 2021-02-10T04:55:09.637 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 31.323078683999999, 2021-02-10T04:55:09.637 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00022532800000000001, 2021-02-10T04:55:09.637 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:55:09.637 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T04:55:09.637 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:55:09.637 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.14146", 2021-02-10T04:55:09.637 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/1625077306", 2021-02-10T04:55:09.637 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 58 2021-02-10T04:55:09.637 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.637 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:55:09.637 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.638 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:55:09.638 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:38.253925+0000", 2021-02-10T04:55:09.638 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:55:09.638 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.638 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.638 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:55:09.638 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:38.253925+0000", 2021-02-10T04:55:09.638 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.3223999999999998e-05 2021-02-10T04:55:09.638 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.638 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.638 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:55:09.638 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:38.253968+0000", 2021-02-10T04:55:09.639 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.6470000000000004e-06 2021-02-10T04:55:09.639 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.639 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.639 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:55:09.639 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:38.253977+0000", 2021-02-10T04:55:09.639 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5800000000000001e-05 2021-02-10T04:55:09.639 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.639 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.639 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:55:09.639 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:38.253993+0000", 2021-02-10T04:55:09.639 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0002e-05 2021-02-10T04:55:09.639 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.640 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.640 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:55:09.640 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:38.254003+0000", 2021-02-10T04:55:09.640 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.2688000000000001e-05 2021-02-10T04:55:09.640 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.640 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.640 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:55:09.640 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:38.254055+0000", 2021-02-10T04:55:09.640 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.4967000000000001e-05 2021-02-10T04:55:09.640 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.640 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.640 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:55:09.641 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:38.254150+0000", 2021-02-10T04:55:09.641 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00022532800000000001 2021-02-10T04:55:09.641 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.641 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:55:09.641 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.641 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.641 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.641 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDScrubReserve(6.f REQUEST e1217)", 2021-02-10T04:55:09.641 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:54:51.538275+0000", 2021-02-10T04:55:09.641 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 18.038728388999999, 2021-02-10T04:55:09.641 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00014819200000000001, 2021-02-10T04:55:09.641 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:55:09.642 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:55:09.642 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:55:09.642 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.642 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:55:09.642 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:51.538275+0000", 2021-02-10T04:55:09.642 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:55:09.642 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.642 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.642 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:55:09.642 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:51.538275+0000", 2021-02-10T04:55:09.642 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.9940000000000001e-05 2021-02-10T04:55:09.642 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.643 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.643 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:55:09.643 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:51.538305+0000", 2021-02-10T04:55:09.643 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.0869999999999996e-06 2021-02-10T04:55:09.643 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.643 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.643 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:55:09.643 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:51.538314+0000", 2021-02-10T04:55:09.643 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2362e-05 2021-02-10T04:55:09.643 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.643 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.643 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:55:09.644 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:51.538327+0000", 2021-02-10T04:55:09.644 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.5470000000000002e-06 2021-02-10T04:55:09.644 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.644 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.644 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:55:09.644 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:51.538336+0000", 2021-02-10T04:55:09.644 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.8646000000000002e-05 2021-02-10T04:55:09.644 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.644 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.644 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:55:09.644 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:51.538375+0000", 2021-02-10T04:55:09.644 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1606999999999999e-05 2021-02-10T04:55:09.645 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.645 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.645 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:55:09.645 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:51.538386+0000", 2021-02-10T04:55:09.645 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.7002999999999998e-05 2021-02-10T04:55:09.645 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.645 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.645 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:55:09.645 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:51.538423+0000", 2021-02-10T04:55:09.645 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00014819200000000001 2021-02-10T04:55:09.646 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.646 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:55:09.646 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.646 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.646 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.646 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDScrubReserve(6.f RELEASE e1217)", 2021-02-10T04:55:09.646 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:54:51.538692+0000", 2021-02-10T04:55:09.646 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 18.038311371999999, 2021-02-10T04:55:09.647 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00011010699999999999, 2021-02-10T04:55:09.648 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:55:09.648 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:55:09.648 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:55:09.649 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.649 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:55:09.649 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:51.538692+0000", 2021-02-10T04:55:09.649 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:55:09.649 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.649 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.649 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:55:09.649 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:51.538692+0000", 2021-02-10T04:55:09.649 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.7880000000000003e-05 2021-02-10T04:55:09.650 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.650 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.650 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:55:09.650 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:51.538730+0000", 2021-02-10T04:55:09.651 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1550000000000007e-06 2021-02-10T04:55:09.651 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.651 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.651 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:55:09.651 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:51.538738+0000", 2021-02-10T04:55:09.651 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.294e-05 2021-02-10T04:55:09.651 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.651 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.651 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:55:09.651 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:51.538751+0000", 2021-02-10T04:55:09.652 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6930000000000001e-06 2021-02-10T04:55:09.652 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.652 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.652 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:55:09.652 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:51.538758+0000", 2021-02-10T04:55:09.652 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1730000000000003e-05 2021-02-10T04:55:09.652 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.652 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.652 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:55:09.652 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:51.538790+0000", 2021-02-10T04:55:09.652 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1331e-05 2021-02-10T04:55:09.652 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.652 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.653 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:55:09.653 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:51.538801+0000", 2021-02-10T04:55:09.653 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.378e-06 2021-02-10T04:55:09.653 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.653 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.653 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:55:09.653 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:51.538802+0000", 2021-02-10T04:55:09.653 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00011010699999999999 2021-02-10T04:55:09.653 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.653 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:55:09.653 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.653 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.653 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.654 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.24373.0:3803 5.e 5:7759931f:::notify.3:head [watch ping cookie 94187770077184 gen 1] snapc 0=[] ondisk+write+known_if_redirected e1215)", 2021-02-10T04:55:09.654 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:54:53.029824+0000", 2021-02-10T04:55:09.654 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 16.547179602, 2021-02-10T04:55:09.654 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00047421399999999998, 2021-02-10T04:55:09.654 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:55:09.654 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:55:09.654 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:55:09.654 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.24373", 2021-02-10T04:55:09.654 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:55:09.654 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 3803 2021-02-10T04:55:09.654 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.654 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:55:09.654 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.655 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:55:09.655 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.029824+0000", 2021-02-10T04:55:09.655 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:55:09.655 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.655 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.655 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:55:09.655 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.029824+0000", 2021-02-10T04:55:09.655 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.2202000000000001e-05 2021-02-10T04:55:09.655 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.655 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.655 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:55:09.655 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.029856+0000", 2021-02-10T04:55:09.656 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.0059999999999992e-06 2021-02-10T04:55:09.656 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.656 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.656 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:55:09.656 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.029864+0000", 2021-02-10T04:55:09.657 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.3451e-05 2021-02-10T04:55:09.657 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.657 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.657 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:55:09.657 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.029878+0000", 2021-02-10T04:55:09.657 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.2359999999999998e-06 2021-02-10T04:55:09.657 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.657 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.657 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:55:09.657 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.029885+0000", 2021-02-10T04:55:09.657 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00010831 2021-02-10T04:55:09.657 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.657 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.658 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:55:09.658 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.029993+0000", 2021-02-10T04:55:09.658 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000105828 2021-02-10T04:55:09.658 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.658 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.658 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:55:09.658 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.030099+0000", 2021-02-10T04:55:09.658 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00019918099999999999 2021-02-10T04:55:09.658 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.658 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.658 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:55:09.658 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.030298+0000", 2021-02-10T04:55:09.659 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00047421399999999998 2021-02-10T04:55:09.659 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.659 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:55:09.659 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.659 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.659 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.659 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDScrubReserve(6.f REQUEST e1217)", 2021-02-10T04:55:09.659 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:54:53.532713+0000", 2021-02-10T04:55:09.659 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 16.044290758999999, 2021-02-10T04:55:09.659 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00015914799999999999, 2021-02-10T04:55:09.659 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:55:09.659 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:55:09.659 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:55:09.660 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.660 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:55:09.660 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.532713+0000", 2021-02-10T04:55:09.660 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:55:09.660 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.660 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.660 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:55:09.660 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.532713+0000", 2021-02-10T04:55:09.660 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.798e-05 2021-02-10T04:55:09.660 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.660 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.660 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:55:09.661 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.532741+0000", 2021-02-10T04:55:09.661 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.8669999999999996e-06 2021-02-10T04:55:09.661 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.661 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.661 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:55:09.661 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.532748+0000", 2021-02-10T04:55:09.661 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.8640000000000008e-06 2021-02-10T04:55:09.661 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.661 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.661 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:55:09.662 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.532758+0000", 2021-02-10T04:55:09.662 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6220000000000003e-06 2021-02-10T04:55:09.662 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.662 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.662 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:55:09.662 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.532764+0000", 2021-02-10T04:55:09.662 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5518e-05 2021-02-10T04:55:09.662 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.662 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.662 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:55:09.662 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.532810+0000", 2021-02-10T04:55:09.662 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6631000000000001e-05 2021-02-10T04:55:09.663 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.663 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.663 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:55:09.663 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.532826+0000", 2021-02-10T04:55:09.663 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5666e-05 2021-02-10T04:55:09.663 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.663 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.663 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:55:09.663 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.532872+0000", 2021-02-10T04:55:09.663 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00015914799999999999 2021-02-10T04:55:09.663 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.663 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:55:09.663 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.664 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.664 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.664 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDScrubReserve(6.15 REQUEST e1217)", 2021-02-10T04:55:09.664 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:54:53.799915+0000", 2021-02-10T04:55:09.664 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 15.777088796999999, 2021-02-10T04:55:09.664 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000203239, 2021-02-10T04:55:09.664 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:55:09.664 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:55:09.664 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:55:09.664 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.664 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:55:09.664 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.799915+0000", 2021-02-10T04:55:09.664 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:55:09.665 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.665 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.665 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:55:09.665 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.799915+0000", 2021-02-10T04:55:09.665 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.7015999999999998e-05 2021-02-10T04:55:09.665 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.665 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.665 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:55:09.665 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.799962+0000", 2021-02-10T04:55:09.665 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.3720000000000005e-06 2021-02-10T04:55:09.665 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.665 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.665 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:55:09.666 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.799970+0000", 2021-02-10T04:55:09.666 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1464000000000001e-05 2021-02-10T04:55:09.666 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.666 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.666 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:55:09.666 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.799982+0000", 2021-02-10T04:55:09.667 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6959999999999999e-06 2021-02-10T04:55:09.667 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.667 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.667 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:55:09.667 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.799988+0000", 2021-02-10T04:55:09.667 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.2407000000000001e-05 2021-02-10T04:55:09.667 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.667 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.668 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:55:09.668 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.800051+0000", 2021-02-10T04:55:09.668 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4015999999999999e-05 2021-02-10T04:55:09.668 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.668 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.668 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:55:09.668 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.800065+0000", 2021-02-10T04:55:09.668 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.3267999999999999e-05 2021-02-10T04:55:09.668 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.668 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.668 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:55:09.668 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.800118+0000", 2021-02-10T04:55:09.668 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000203239 2021-02-10T04:55:09.669 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.669 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:55:09.669 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.669 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.669 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.669 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDScrubReserve(2.6 REQUEST e1217)", 2021-02-10T04:55:09.669 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:54:53.829832+0000", 2021-02-10T04:55:09.669 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 15.747171163999999, 2021-02-10T04:55:09.669 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00015036300000000001, 2021-02-10T04:55:09.669 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:55:09.669 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:55:09.669 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:55:09.669 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.670 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:55:09.670 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.829832+0000", 2021-02-10T04:55:09.670 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:55:09.670 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.670 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.670 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:55:09.670 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.829832+0000", 2021-02-10T04:55:09.670 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.2566e-05 2021-02-10T04:55:09.670 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.670 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.670 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:55:09.670 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.829865+0000", 2021-02-10T04:55:09.670 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1149999999999994e-06 2021-02-10T04:55:09.671 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.671 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.671 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:55:09.671 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.829873+0000", 2021-02-10T04:55:09.671 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7048e-05 2021-02-10T04:55:09.671 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.671 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.671 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:55:09.671 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.829890+0000", 2021-02-10T04:55:09.671 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1760000000000004e-06 2021-02-10T04:55:09.671 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.671 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.671 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:55:09.672 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.829897+0000", 2021-02-10T04:55:09.672 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.1060999999999999e-05 2021-02-10T04:55:09.672 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.672 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.672 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:55:09.672 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.829938+0000", 2021-02-10T04:55:09.672 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1187e-05 2021-02-10T04:55:09.672 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.672 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.672 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:55:09.672 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.829950+0000", 2021-02-10T04:55:09.672 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.3210000000000002e-05 2021-02-10T04:55:09.672 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.673 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.673 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:55:09.673 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:53.829983+0000", 2021-02-10T04:55:09.673 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00015036300000000001 2021-02-10T04:55:09.673 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.673 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:55:09.673 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.673 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.673 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.673 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDScrubReserve(6.15 REQUEST e1218)", 2021-02-10T04:55:09.673 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:54:54.822278+0000", 2021-02-10T04:55:09.673 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 14.754725895, 2021-02-10T04:55:09.673 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000164387, 2021-02-10T04:55:09.673 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:55:09.674 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:55:09.674 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:55:09.674 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.674 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:55:09.674 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:54.822278+0000", 2021-02-10T04:55:09.674 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:55:09.674 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.674 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.674 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:55:09.674 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:54.822278+0000", 2021-02-10T04:55:09.674 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.0540999999999998e-05 2021-02-10T04:55:09.674 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.675 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.675 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:55:09.675 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:54.822308+0000", 2021-02-10T04:55:09.675 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.5740000000000001e-06 2021-02-10T04:55:09.675 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.675 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.675 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:55:09.675 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:54.822315+0000", 2021-02-10T04:55:09.675 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0197000000000001e-05 2021-02-10T04:55:09.675 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.675 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.675 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:55:09.675 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:54.822325+0000", 2021-02-10T04:55:09.676 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.9990000000000008e-06 2021-02-10T04:55:09.676 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.676 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.676 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:55:09.676 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:54.822333+0000", 2021-02-10T04:55:09.676 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9347e-05 2021-02-10T04:55:09.676 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.676 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.676 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:55:09.676 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:54.822382+0000", 2021-02-10T04:55:09.676 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5233e-05 2021-02-10T04:55:09.676 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.676 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.676 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:55:09.677 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:54.822398+0000", 2021-02-10T04:55:09.677 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.4496000000000002e-05 2021-02-10T04:55:09.677 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.677 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.677 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:55:09.677 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:54.822442+0000", 2021-02-10T04:55:09.677 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000164387 2021-02-10T04:55:09.677 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.678 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:55:09.678 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.678 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.678 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.678 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.24373.0:3827 5.e 5:7759931f:::notify.3:head [watch ping cookie 94187770077184 gen 1] snapc 0=[] ondisk+write+known_if_redirected e1222)", 2021-02-10T04:55:09.678 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:55:08.031783+0000", 2021-02-10T04:55:09.678 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 1.5452208620000001, 2021-02-10T04:55:09.678 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00053447300000000004, 2021-02-10T04:55:09.678 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T04:55:09.678 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T04:55:09.678 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T04:55:09.678 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.24373", 2021-02-10T04:55:09.678 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:55:09.679 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 3827 2021-02-10T04:55:09.679 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.679 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T04:55:09.679 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.679 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T04:55:09.679 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:08.031783+0000", 2021-02-10T04:55:09.679 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T04:55:09.679 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.679 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.679 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T04:55:09.679 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:08.031783+0000", 2021-02-10T04:55:09.679 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.5528999999999998e-05 2021-02-10T04:55:09.679 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.680 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.680 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T04:55:09.680 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:08.031818+0000", 2021-02-10T04:55:09.680 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5238e-05 2021-02-10T04:55:09.680 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.680 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.680 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T04:55:09.680 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:08.031834+0000", 2021-02-10T04:55:09.680 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7907999999999999e-05 2021-02-10T04:55:09.680 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.680 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.680 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T04:55:09.681 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:08.031851+0000", 2021-02-10T04:55:09.681 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0332999999999999e-05 2021-02-10T04:55:09.681 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.681 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.681 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T04:55:09.681 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:08.031862+0000", 2021-02-10T04:55:09.681 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000134988 2021-02-10T04:55:09.681 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.681 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.681 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T04:55:09.682 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:08.031997+0000", 2021-02-10T04:55:09.682 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00013309500000000001 2021-02-10T04:55:09.682 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.682 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.682 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T04:55:09.682 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:08.032130+0000", 2021-02-10T04:55:09.682 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00018738199999999999 2021-02-10T04:55:09.682 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T04:55:09.682 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T04:55:09.682 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T04:55:09.682 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:08.032317+0000", 2021-02-10T04:55:09.682 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00053447300000000004 2021-02-10T04:55:09.683 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.683 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:55:09.683 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.683 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T04:55:09.683 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T04:55:09.683 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:55:09.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:09 smithi085 conmon[29319]: cluster 2021-02-10T04:55:08.492011+0000 mgr.y (mgr.14140) 2021-02-10T04:55:09.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:09 smithi085 conmon[29319]: 1353 : cluster [DBG] pgmap v2219: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:55:09.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:09 smithi085 conmon[29319]: cluster 2021-02-10T04:55:08.556012+0000 mon.a (mon.0) 2619 : cluster [DBG] osdmap e1225: 8 total, 7 up, 8 in 2021-02-10T04:55:09.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:09 smithi085 conmon[29319]: audit 2021-02-10T04:55:09.294967+0000 mgr.y (mgr.14140) 1354 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:55:09.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:09 smithi085 conmon[35961]: cluster 2021-02-10T04:55:08.492011+0000 mgr.y ( 2021-02-10T04:55:09.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:09 smithi085 conmon[35961]: mgr.14140) 1353 : cluster [DBG] pgmap v2219: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:55:09.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:09 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:55:09.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:09 smithi085 conmon[35961]: :55:08.556012+0000 mon.a (mon.0) 2619 : cluster [DBG] osdmap e1225: 8 total, 7 up, 8 in 2021-02-10T04:55:09.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:09 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:55:09.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:09 smithi085 conmon[35961]: :55:09.294967+0000 mgr.y (mgr.14140) 1354 : audit [DBG] 2021-02-10T04:55:09.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:09 smithi085 conmon[35961]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:55:11.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:11 smithi164 conmon[29288]: cluster 2021-02-10T04:55:10.492768+0000 mgr.y (mgr.14140) 1355 : cluster 2021-02-10T04:55:11.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:11 smithi164 conmon[29288]: [DBG] pgmap v2221: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 737 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:55:11.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:11 smithi085 conmon[29319]: cluster 2021-02-10T04:55:10.492768+0000 mgr.y (mgr.14140 2021-02-10T04:55:11.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:11 smithi085 conmon[29319]: ) 1355 : cluster [DBG] pgmap v2221: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 737 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:55:11.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:11 smithi085 conmon[35961]: cluster 2021-02-10T04:55:10.492768 2021-02-10T04:55:11.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:11 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1355 : cluster [DBG] pgmap v2221: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 737 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:55:12.172 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:55:12.208 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:55:12.243 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:55:12.269 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:55:12.296 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:55:12.322 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:55:12.356 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:55:12.357 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-02-10T04:55:12.357 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: problem getting command descriptions from osd.0 2021-02-10T04:55:12.357 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:55:12.357 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:55:12.357 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:55:12.357 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:55:12.357 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:55:12.357 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:55:12.357 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:55:12.386 INFO:teuthology.orchestra.run.smithi085.stderr:no valid command found; 10 closest matches: 2021-02-10T04:55:12.386 INFO:teuthology.orchestra.run.smithi085.stderr:0 2021-02-10T04:55:12.386 INFO:teuthology.orchestra.run.smithi085.stderr:1 2021-02-10T04:55:12.386 INFO:teuthology.orchestra.run.smithi085.stderr:2 2021-02-10T04:55:12.386 INFO:teuthology.orchestra.run.smithi085.stderr:abort 2021-02-10T04:55:12.386 INFO:teuthology.orchestra.run.smithi085.stderr:assert 2021-02-10T04:55:12.386 INFO:teuthology.orchestra.run.smithi085.stderr:bluefs debug_inject_read_zeros 2021-02-10T04:55:12.386 INFO:teuthology.orchestra.run.smithi085.stderr:bluefs files list 2021-02-10T04:55:12.387 INFO:teuthology.orchestra.run.smithi085.stderr:bluefs stats 2021-02-10T04:55:12.387 INFO:teuthology.orchestra.run.smithi085.stderr:bluestore allocator dump block 2021-02-10T04:55:12.387 INFO:teuthology.orchestra.run.smithi085.stderr:bluestore allocator fragmentation block 2021-02-10T04:55:12.387 INFO:teuthology.orchestra.run.smithi085.stderr:admin_socket: invalid command 2021-02-10T04:55:12.400 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[177076]: debug 2021-02-10T04:55:12.355+0000 7f14d51e8200 -1 osd.0 1146 log_to_monitors {default=true} 2021-02-10T04:55:12.589 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is set 2021-02-10T04:55:12.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[29319]: cluster 2021-02-10T04:55:11.623515+0000 osd.5 ( 2021-02-10T04:55:12.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[29319]: osd.5) 189 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:55:12.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[29319]: cluster 2021-02-10T04:55:11.625959+0000 osd.5 (osd.5) 190 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:55:12.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[29319]: audit 2021-02-10T04:55:12.147662+0000 mon.a (mon.0) 2620 : audit [DBG] from='client.? 172.21.15.85:0/3232560424' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:55:12.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[29319]: audit 2021-02-10T04:55:12.406146+0000 mon.c (mon.1) 121 : audit [INF] from='client.? 172.21.15.85:0/852881288' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:55:12.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[29319]: audit 2021-02-10T04:55:12.406501+0000 mon.a (mon.0) 2621 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:55:12.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[29319]: audit 2021 2021-02-10T04:55:12.662 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[29319]: -02-10T04:55:12.477683+0000 2021-02-10T04:55:12.662 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[29319]: mon.b (mon.2) 50 : audit [INF] 2021-02-10T04:55:12.662 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[29319]: from='osd.0 v2:172.21.15.85:6801/7' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-02-10T04:55:12.662 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:55:12.662 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[29319]: -10T04:55:12.478173+0000 2021-02-10T04:55:12.662 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[29319]: mon.a (mon.0) 2622 : audit 2021-02-10T04:55:12.662 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[29319]: [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-02-10T04:55:12.662 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[29319]: audit 2021-02-10T04:55:12.662 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[29319]: 2021-02-10T04:55:12 2021-02-10T04:55:12.662 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[29319]: .479176+0000 mon.b (mon.2) 2021-02-10T04:55:12.662 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[29319]: 51 : audit [INF] from='osd.0 v2:172.21.15.85:6801/7' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:55:12.662 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[29319]: audit 2021-02-10T04:55:12.663 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[29319]: 2021-02-10T04:55:12 2021-02-10T04:55:12.663 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[29319]: .479477+0000 mon.a (mon.0) 2623 2021-02-10T04:55:12.663 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[29319]: : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:55:12.663 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: cluster 2021-02-10T04:55:12.663 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: 2021-02-10T04:55:12.663 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: 2021 2021-02-10T04:55:12.663 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: - 2021-02-10T04:55:12.663 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: 02 2021-02-10T04:55:12.663 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: - 2021-02-10T04:55:12.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: 10 2021-02-10T04:55:12.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: T 2021-02-10T04:55:12.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: 04 2021-02-10T04:55:12.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: : 2021-02-10T04:55:12.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: 55 2021-02-10T04:55:12.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: : 2021-02-10T04:55:12.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: 11 2021-02-10T04:55:12.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: . 2021-02-10T04:55:12.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: 623515 2021-02-10T04:55:12.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: +0000 2021-02-10T04:55:12.664 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: osd.5 (osd.5) 189 : cluster 2021-02-10T04:55:12.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: [DBG] purged_snaps scrub starts 2021-02-10T04:55:12.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: cluster 2021-02-10T04:55:12.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:55:12.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: :55:11.625959+0000 osd.5 (osd.5) 190 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:55:12.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: audit 2021-02-10T04:55:12.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: 2021-02-10T04:55:12.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: 2021 2021-02-10T04:55:12.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: -02-10T04:55:12 2021-02-10T04:55:12.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: .147662+0000 mon.a (mon.0) 2620 : audit [DBG] from='client.? 172.21.15.85:0/3232560424' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:55:12.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:55:12.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: 04:55:12.406146+0000 mon.c (mon.1) 2021-02-10T04:55:12.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: 121 : audit [INF] from='client.? 172.21.15.85:0/852881288' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:55:12.665 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:55:12.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: 02-10T04:55:12.406501 2021-02-10T04:55:12.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2621 : audit [INF] 2021-02-10T04:55:12.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:55:12.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: audit 2021-02-10T04:55 2021-02-10T04:55:12.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: :12.477683+0000 mon.b (mon.2) 2021-02-10T04:55:12.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: 50 : audit [INF] from='osd.0 v2:172.21.15.85:6801/7' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-02-10T04:55:12.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:55:12.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: -10T04:55:12.478173+0000 2021-02-10T04:55:12.666 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: mon.a (mon.0) 2622 : audit [INF] 2021-02-10T04:55:12.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-02-10T04:55:12.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: audit 2021-02-10T04:55:12.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: 2021-02-10T04:55:12.479176+0000 2021-02-10T04:55:12.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: mon.b (mon.2) 51 : audit [INF] 2021-02-10T04:55:12.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: from='osd.0 v2:172.21.15.85:6801/7' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:55:12.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: audit 2021-02-10T04:55: 2021-02-10T04:55:12.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: 12.479477+0000 mon.a (mon.0) 2021-02-10T04:55:12.667 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:12 smithi085 conmon[35961]: 2623 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:55:12.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:12 smithi164 conmon[29288]: cluster 2021-02-10T04:55:11.623515+0000 osd.5 ( 2021-02-10T04:55:12.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:12 smithi164 conmon[29288]: osd.5) 189 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:55:12.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:12 smithi164 conmon[29288]: cluster 2021-02-10T04:55:11.625959+0000 osd.5 (osd.5) 190 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:55:12.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:12 smithi164 conmon[29288]: audit 2021-02-10T04:55:12.147662+0000 mon.a (mon.0) 2620 : audit [DBG] 2021-02-10T04:55:12.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:12 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/3232560424' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:55:12.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:12 smithi164 conmon[29288]: audit 2021 2021-02-10T04:55:12.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:12 smithi164 conmon[29288]: -02-10T04:55:12.406146+0000 mon.c (mon.1) 121 : audit 2021-02-10T04:55:12.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:12 smithi164 conmon[29288]: [INF] from='client.? 172.21.15.85:0/852881288' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:55:12.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:12 smithi164 conmon[29288]: audit 2021-02-10T04:55:12.406501+0000 2021-02-10T04:55:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:12 smithi164 conmon[29288]: mon.a (mon.0) 2621 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:55:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:12 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:55:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:12 smithi164 conmon[29288]: 02-10T04:55:12.477683+0000 mon.b (mon 2021-02-10T04:55:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:12 smithi164 conmon[29288]: .2) 50 : audit [INF] from='osd.0 v2:172.21.15.85:6801/7' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-02-10T04:55:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:12 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:55:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:12 smithi164 conmon[29288]: 10T04:55:12.478173+0000 mon.a (mon.0 2021-02-10T04:55:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:12 smithi164 conmon[29288]: ) 2622 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-02-10T04:55:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:12 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:55:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:12 smithi164 conmon[29288]: 04:55:12.479176+0000 mon.b (mon.2) 2021-02-10T04:55:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:12 smithi164 conmon[29288]: 51 : audit [INF] from='osd.0 v2:172.21.15.85:6801/7' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:55:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:12 smithi164 conmon[29288]: audit 2021-02-10T04: 2021-02-10T04:55:12.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:12 smithi164 conmon[29288]: 55:12.479477+0000 mon.a (mon.0) 2623 : 2021-02-10T04:55:12.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:12 smithi164 conmon[29288]: audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:55:13.412 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.5 dump_ops_in_flight 2021-02-10T04:55:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:13 smithi164 conmon[29288]: cluster 2021-02-10T04:55:12.493718+0000 mgr.y 2021-02-10T04:55:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:13 smithi164 conmon[29288]: (mgr.14140) 1356 : cluster [DBG] pgmap v2222: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:55:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:13 smithi164 conmon[29288]: cluster 2021-02-10T04:55:12.551906+0000 mon.a (mon.0) 2624 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:55:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:13 smithi164 conmon[29288]: audit 2021-02-10T04:55:12.560974+0000 mon.a (mon.0) 2625 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:55:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:13 smithi164 conmon[29288]: cluster 2021-02-10T04:55:12.561066+0000 mon.a (mon.0) 2626 : cluster [DBG] osdmap e1226: 8 total, 7 up, 8 in 2021-02-10T04:55:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:13 smithi164 conmon[29288]: cephadm 2021-02-10T04:55:13.063981+0000 mgr.y (mgr.14140) 1357 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:55:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:13 smithi164 conmon[29288]: audit 2021-02-10T04:55:13.064088+0000 mon.a (mon.0) 2627 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:55:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:13 smithi164 conmon[29288]: cluster 2021-02-10T04:55:13.548442+0000 mon.a (mon.0) 2628 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2021-02-10T04:55:13.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:13 smithi164 conmon[29288]: cluster 2021-02-10T04:55:13.551949+0000 mon.a (mon.0) 2629 : cluster [INF] osd.0 v2:172.21.15.85:6801/7 boot 2021-02-10T04:55:13.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:13 smithi164 conmon[29288]: cluster 2021-02-10T04:55:13.551984+0000 mon.a (mon.0) 2630 : cluster [DBG] osdmap e1227: 8 total, 8 up, 8 in 2021-02-10T04:55:13.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:13 smithi164 conmon[29288]: audit 2021-02-10T04:55:13.552290+0000 mon.a (mon.0) 2631 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-02-10T04:55:13.706 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:55:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[35961]: cluster 2021-02-10T04:55:12.493718+0000 mgr.y 2021-02-10T04:55:13.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[35961]: (mgr.14140) 1356 : cluster [DBG] pgmap v2222: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:55:13.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[35961]: cluster 2021-02-10T04:55: 2021-02-10T04:55:13.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[35961]: 12.551906+0000 mon.a (mon.0) 2624 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:55:13.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:55:13.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[35961]: T04:55:12.560974+0000 mon.a (mon. 2021-02-10T04:55:13.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[35961]: 0) 2625 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:55:13.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:55:13.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[35961]: 02-10T04:55:12.561066+0000 mon.a 2021-02-10T04:55:13.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[35961]: (mon.0) 2626 : cluster [DBG] osdmap e1226: 8 total, 7 up, 8 in 2021-02-10T04:55:13.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[35961]: 2021-02-10T04:55:13.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[35961]: cephadm 2021-02-10T04:55:13.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[35961]: 2021-02-10T04:55:13.063981+0000 mgr.y (mgr 2021-02-10T04:55:13.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[35961]: .14140) 1357 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:55:13.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:55:13.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[35961]: -10T04:55:13.064088+0000 mon.a ( 2021-02-10T04:55:13.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[35961]: mon.0) 2627 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:55:13.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:55:13.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[35961]: 02-10T04:55:13.548442+0000 mon.a 2021-02-10T04:55:13.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[35961]: (mon.0) 2628 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2021-02-10T04:55:13.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[35961]: 2021-02-10T04:55:13.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[35961]: cluster 2021-02-10T04:55:13.551949+0000 2021-02-10T04:55:13.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[35961]: mon.a (mon.0) 2629 : cluster [INF] osd.0 v2:172.21.15.85:6801/7 boot 2021-02-10T04:55:13.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[35961]: cluster 2021-02-10T04:55:13.551984+0000 mon.a (mon 2021-02-10T04:55:13.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[35961]: .0) 2630 : cluster [DBG] osdmap e1227: 8 total, 8 up, 8 in 2021-02-10T04:55:13.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:55:13.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[35961]: T04:55:13.552290+0000 mon.a (mon. 2021-02-10T04:55:13.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[35961]: 0) 2631 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-02-10T04:55:13.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: cluster 2021-02-10T04:55:12.493718+0000 mgr.y (mgr.14140) 1356 2021-02-10T04:55:13.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: : cluster [DBG] pgmap v2222: 204 pgs: 1 active+clean+scrubbing, 8 active+clean+remapped, 24 active+undersized+degraded, 38 active+undersized, 133 active+clean; 889 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 214/2196 objects degraded (9.745%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:55:13.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: cluster 2021-02-10T04:55:12.551906+0000 mon.a (mon.0) 2624 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:55:13.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: audit 2021-02-10T04:55:12.560974+0000 mon.a (mon.0) 2625 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:55:13.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: cluster 2021-02-10T04:55:12.561066+0000 mon.a 2021-02-10T04:55:13.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: (mon.0) 2626 : cluster [DBG] osdmap e1226: 8 total, 7 up, 8 in 2021-02-10T04:55:13.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: cephadm 2021-02-10 2021-02-10T04:55:13.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: T04:55:13.063981+0000 2021-02-10T04:55:13.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: mgr.y (mgr.14140) 1357 : cephadm 2021-02-10T04:55:13.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: [INF] Refreshing smithi164 facts 2021-02-10T04:55:13.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: audit 2021-02-10T04:55:13.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: 2021-02-10T04:55: 2021-02-10T04:55:13.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: 13.064088+0000 mon.a (mon. 2021-02-10T04:55:13.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: 0) 2627 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:55:13.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:55:13.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: 02-10T04:55:13. 2021-02-10T04:55:13.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: 548442+0000 mon.a (mon.0) 2021-02-10T04:55:13.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: 2628 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2021-02-10T04:55:13.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:55:13.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: -02-10T04:55:13 2021-02-10T04:55:13.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: .551949+0000 mon.a (mon. 2021-02-10T04:55:13.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: 0) 2629 : cluster [INF] osd.0 v2:172.21.15.85:6801/7 boot 2021-02-10T04:55:13.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: 2021-02-10T04:55:13.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:55:13.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: -02-10T04:55:13. 2021-02-10T04:55:13.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: 551984+0000 mon.a (mon.0) 2630 : cluster [DBG] 2021-02-10T04:55:13.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: osdmap e1227: 8 total, 8 up, 8 in 2021-02-10T04:55:13.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:55:13.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: :55:13.552290+0000 2021-02-10T04:55:13.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: mon.a (mon.0) 2631 : audit 2021-02-10T04:55:13.948 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:13 smithi085 conmon[29319]: [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-02-10T04:55:14.008 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-02-10T04:55:14.008 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2021-02-10T04:55:14.828 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:55:14.828 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:55:14.870 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:14 smithi164 conmon[29288]: cluster 2021-02-10T04:55:14.566946+0000 mon.a (mon.0) 2632 : cluster [WRN] Health check update: Degraded data redundancy: 146/2196 objects degraded (6.648%), 21 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2021-02-10T04:55:14.871 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:14 smithi164 conmon[29288]: cluster 2021-02-10T04:55:14.576586+0000 mon.a (mon.0) 2633 : cluster [DBG] osdmap e1228: 8 total, 8 up, 8 in 2021-02-10T04:55:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:14 smithi085 conmon[29319]: cluster 2021-02-10T04:55:14.566946+0000 2021-02-10T04:55:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:14 smithi085 conmon[29319]: mon.a (mon.0) 2632 : cluster [WRN] Health check update: Degraded data redundancy: 146/2196 objects degraded (6.648%), 21 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2021-02-10T04:55:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:14 smithi085 conmon[29319]: cluster 2021-02-10T04:55:14.576586+0000 mon.a 2021-02-10T04:55:14.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:14 smithi085 conmon[29319]: (mon.0) 2633 : cluster [DBG] osdmap e1228: 8 total, 8 up, 8 in 2021-02-10T04:55:14.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:14 smithi085 conmon[35961]: cluster 2021-02-10T04:55:14.566946+0000 2021-02-10T04:55:14.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:14 smithi085 conmon[35961]: mon.a (mon.0) 2632 : cluster [WRN] Health check update: Degraded data redundancy: 146/2196 objects degraded (6.648%), 21 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2021-02-10T04:55:14.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:14 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:55:14.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:14 smithi085 conmon[35961]: 55:14.576586+0000 mon.a (mon.0) 2633 : cluster [DBG] osdmap e1228: 8 total, 8 up, 8 in 2021-02-10T04:55:15.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:15 smithi164 conmon[29288]: cluster 2021-02-10T04:55:13.352809+0000 2021-02-10T04:55:15.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:15 smithi164 conmon[29288]: osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:55:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:15 smithi164 conmon[29288]: cluster 2021-02-10T04:55:13.444272+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:55:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:15 smithi164 conmon[29288]: cluster 2021-02-10T04:55:14.494543+0000 mgr.y (mgr.14140) 1358 : cluster [DBG] pgmap v2225: 204 pgs: 6 peering, 1 active+clean+scrubbing, 8 active+clean+remapped, 21 active+undersized+degraded, 35 active+undersized, 133 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 146/2196 objects degraded (6.648%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:55:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:15 smithi164 conmon[29288]: cluster 2021-02-10T04:55:15.579304+0000 mon.a 2021-02-10T04:55:15.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:15 smithi164 conmon[29288]: (mon.0) 2634 : cluster [DBG] osdmap e1229: 8 total, 8 up, 8 in 2021-02-10T04:55:15.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:15 smithi085 conmon[29319]: cluster 2021-02-10T04:55:13.352809+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:55:15.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:15 smithi085 conmon[29319]: cluster 2021-02-10T04:55:13.444272+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:55:15.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:15 smithi085 conmon[29319]: cluster 2021-02-10T04:55:14.494543+0000 mgr.y (mgr.14140) 1358 : cluster [DBG] pgmap v2225: 204 pgs: 6 peering, 1 active+clean+scrubbing, 8 active+clean+remapped, 21 active+undersized+degraded, 35 active+undersized, 133 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 146/2196 objects degraded (6.648%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:55:15.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:15 smithi085 conmon[29319]: cluster 2021-02-10T04:55:15.579304+0000 mon.a (mon.0) 2634 : cluster [DBG] osdmap e1229: 8 total, 8 up, 8 in 2021-02-10T04:55:15.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:15 smithi085 conmon[35961]: cluster 2021-02-10T04:55:13.352809+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:55:15.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:15 smithi085 conmon[35961]: cluster 2021-02-10T04:55:13.444272+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:55:15.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:15 smithi085 conmon[35961]: cluster 2021-02-10T04:55:14.494543+0000 mgr.y (mgr.14140) 1358 : cluster [DBG] pgmap v2225: 204 pgs: 6 peering, 1 active+clean+scrubbing, 8 active+clean+remapped, 21 active+undersized+degraded, 35 active+undersized, 133 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 146/2196 objects degraded (6.648%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:55:15.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:15 smithi085 conmon[35961]: cluster 2021-02-10T04:55:15.579304+0000 mon.a (mon.0) 2634 : cluster [DBG] osdmap e1229: 8 total, 8 up, 8 in 2021-02-10T04:55:16.037 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset noscrub 2021-02-10T04:55:16.289 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:55:16.830 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:55:17.042 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:55:17.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:17 smithi164 conmon[29288]: cluster 2021-02-10T04:55:16.495228+0000 2021-02-10T04:55:17.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:17 smithi164 conmon[29288]: mgr.y (mgr.14140) 1359 : cluster [DBG] pgmap v2228: 204 pgs: 6 peering, 1 active+clean+scrubbing, 8 active+clean+remapped, 17 active+undersized+degraded, 25 active+undersized, 147 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 123/2196 objects degraded (5.601%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:55:17.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:17 smithi085 conmon[29319]: cluster 2021-02-10T04:55:16.495228 2021-02-10T04:55:17.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:17 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 1359 : cluster [DBG] pgmap v2228: 204 pgs: 6 peering, 1 active+clean+scrubbing, 8 active+clean+remapped, 17 active+undersized+degraded, 25 active+undersized, 147 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 123/2196 objects degraded (5.601%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:55:17.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:17 smithi085 conmon[35961]: cluster 2021-02-10T04:55:16.495228 2021-02-10T04:55:17.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:17 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1359 : cluster [DBG] pgmap v2228: 204 pgs: 6 peering, 1 active+clean+scrubbing, 8 active+clean+remapped, 17 active+undersized+degraded, 25 active+undersized, 147 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 123/2196 objects degraded (5.601%); 44/2196 objects misplaced (2.004%) 2021-02-10T04:55:18.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:18 smithi085 conmon[29319]: cluster 2021-02-10T04:55:18.486613+0000 osd.2 (osd.2) 2029 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:55:18.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:18 smithi085 conmon[29319]: cluster 2021-02-10T04:55:18.489176+0000 osd.2 (osd.2) 2030 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:55:18.631 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:18 smithi085 conmon[29319]: audit 2021-02-10T04:55:18.496703+0000 mon.a (mon.0) 2635 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "27"}]: dispatch 2021-02-10T04:55:18.632 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:18 smithi085 conmon[35961]: cluster 2021-02-10T04:55:18.486613 2021-02-10T04:55:18.632 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:18 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 2029 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:55:18.632 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:18 smithi085 conmon[35961]: cluster 2021-02-10T04:55:18.489176+0000 osd.2 (osd.2) 2030 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:55:18.632 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:18 smithi085 conmon[35961]: audit 2021-02-10T04:55:18.496703+0000 mon.a (mon.0) 2635 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "27"}]: dispatch 2021-02-10T04:55:18.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:18 smithi164 conmon[29288]: cluster 2021-02-10T04:55:18.486613+0000 osd.2 2021-02-10T04:55:18.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:18 smithi164 conmon[29288]: (osd.2) 2029 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:55:18.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:18 smithi164 conmon[29288]: cluster 2021-02-10T04:55:18.489176+0000 osd.2 (osd.2) 2030 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:55:18.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:18 smithi164 conmon[29288]: audit 2021-02-10T04:55:18.496703+0000 mon.a (mon.0 2021-02-10T04:55:18.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:18 smithi164 conmon[29288]: ) 2635 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "27"}]: dispatch 2021-02-10T04:55:19.009 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.0 dump_ops_in_flight 2021-02-10T04:55:19.231 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:55:19.642 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:55:19 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:55:19.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:19 smithi164 conmon[29288]: cluster 2021-02-10T04:55:18.495945+0000 mgr.y (mgr.14140) 1360 : cluster [DBG] pgmap v2229: 204 pgs: 1 active+clean+scrubbing, 6 peering, 20 active+clean+wait, 177 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 742 KiB/s, 0 objects/s recovering 2021-02-10T04:55:19.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:19 smithi164 conmon[29288]: cluster 2021-02-10T04:55:18.623947+0000 mon.a (mon.0) 2636 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 123/2196 objects degraded (5.601%), 17 pgs degraded, 40 pgs undersized) 2021-02-10T04:55:19.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:19 smithi164 conmon[29288]: audit 2021 2021-02-10T04:55:19.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:19 smithi164 conmon[29288]: -02-10T04:55:18.634216+0000 mon.a (mon.0) 2637 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "27"}]': finished 2021-02-10T04:55:19.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:19 smithi164 conmon[29288]: cluster 2021-02-10T04:55:19.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:19 smithi164 conmon[29288]: 2021-02-10T04:55:18.634278+0000 mon.a 2021-02-10T04:55:19.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:19 smithi164 conmon[29288]: (mon.0) 2638 : cluster [DBG] osdmap e1230: 8 total, 8 up, 8 in 2021-02-10T04:55:19.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:19 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:55:19.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:19 smithi164 conmon[29288]: 02-10T04:55:19.305011+0000 2021-02-10T04:55:19.643 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:19 smithi164 conmon[29288]: mgr.y (mgr.14140) 1361 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:55:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:19 smithi164 conmon[29288]: 2021-02-10T04:55:19.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:19 smithi085 conmon[29319]: cluster 2021-02-10T04:55:18.495945+0000 mgr.y (mgr.14140) 1360 : cluster 2021-02-10T04:55:19.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:19 smithi085 conmon[29319]: [DBG] pgmap v2229: 204 pgs: 1 active+clean+scrubbing, 6 peering, 20 active+clean+wait, 177 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 742 KiB/s, 0 objects/s recovering 2021-02-10T04:55:19.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:19 smithi085 conmon[29319]: cluster 2021-02-10T04:55:18.623947+0000 mon.a (mon.0) 2636 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 123/2196 objects degraded (5.601%), 17 pgs degraded, 40 pgs undersized) 2021-02-10T04:55:19.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:19 smithi085 conmon[29319]: audit 2021-02-10T04:55:18.634216+0000 mon.a (mon.0) 2637 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "27"}]': finished 2021-02-10T04:55:19.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:19 smithi085 conmon[29319]: cluster 2021-02-10T04:55:18.634278+0000 mon.a (mon.0) 2638 : cluster [DBG] osdmap e1230: 8 total, 8 up, 8 in 2021-02-10T04:55:19.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:19 smithi085 conmon[29319]: audit 2021-02-10T04:55:19.305011+0000 mgr.y (mgr.14140) 1361 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:55:19.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:19 smithi085 conmon[35961]: cluster 2021-02-10T04:55:18.495945+0000 mgr.y 2021-02-10T04:55:19.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:19 smithi085 conmon[35961]: (mgr.14140) 1360 : cluster [DBG] pgmap v2229: 204 pgs: 1 active+clean+scrubbing, 6 peering, 20 active+clean+wait, 177 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 742 KiB/s, 0 objects/s recovering 2021-02-10T04:55:19.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:19 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:55:19.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:19 smithi085 conmon[35961]: 55:18.623947+0000 mon.a (mon.0) 2636 : cluster [INF] 2021-02-10T04:55:19.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:19 smithi085 conmon[35961]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 123/2196 objects degraded (5.601%), 17 pgs degraded, 40 pgs undersized) 2021-02-10T04:55:19.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:19 smithi085 conmon[35961]: audit 2021 2021-02-10T04:55:19.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:19 smithi085 conmon[35961]: -02-10T04:55:18 2021-02-10T04:55:19.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:19 smithi085 conmon[35961]: .634216+0000 mon.a (mon.0 2021-02-10T04:55:19.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:19 smithi085 conmon[35961]: ) 2637 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "27"}]': finished 2021-02-10T04:55:19.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:19 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:55:19.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:19 smithi085 conmon[35961]: 02-10T04:55:18. 2021-02-10T04:55:19.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:19 smithi085 conmon[35961]: 634278+0000 mon.a (mon.0) 2021-02-10T04:55:19.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:19 smithi085 conmon[35961]: 2638 : cluster [DBG] osdmap e1230: 8 total, 8 up, 8 in 2021-02-10T04:55:19.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:19 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:55:19.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:19 smithi085 conmon[35961]: 02-10T04:55:19. 2021-02-10T04:55:19.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:19 smithi085 conmon[35961]: 305011+0000 2021-02-10T04:55:19.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:19 smithi085 conmon[35961]: mgr.y (mgr.14140) 2021-02-10T04:55:19.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:19 smithi085 conmon[35961]: 1361 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:55:20.659 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is unset 2021-02-10T04:55:20.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:20 smithi164 conmon[29288]: cluster 2021-02-10T04:55:19.633348+0000 mon.a (mon 2021-02-10T04:55:20.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:20 smithi164 conmon[29288]: .0) 2639 : cluster [DBG] osdmap e1231: 8 total, 8 up, 8 in 2021-02-10T04:55:20.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:20 smithi164 conmon[29288]: audit 2021-02-10T04:55:20.607468+0000 mon.c (mon.1) 122 : audit [INF] from='client.? 172.21.15.85:0/1874414165' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:55:20.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:20 smithi164 conmon[29288]: audit 2021-02-10T04:55:20.607790+0000 mon.a (mon.0) 2640 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:55:20.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:20 smithi085 conmon[29319]: cluster 2021-02-10T04:55:20.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:20 smithi085 conmon[29319]: 2021-02-10T04:55:20.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:20 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:55:20.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:20 smithi085 conmon[29319]: :55:19.633348+0000 mon.a (mon.0) 2639 : cluster [DBG] osdmap e1231: 8 total, 8 up, 8 in 2021-02-10T04:55:20.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:20 smithi085 conmon[29319]: audit 2021-02-10T04: 2021-02-10T04:55:20.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:20 smithi085 conmon[29319]: 55:20.607468+0000 mon.c (mon.1) 122 : audit [INF] from='client.? 172.21.15.85:0/1874414165' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:55:20.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:20 smithi085 conmon[29319]: audit 2021-02-10T04:55: 2021-02-10T04:55:20.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:20 smithi085 conmon[29319]: 20.607790+0000 mon.a (mon.0) 2640 : audit 2021-02-10T04:55:20.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:20 smithi085 conmon[29319]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:55:20.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:20 smithi085 conmon[35961]: cluster 2021-02-10T04:55:20.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:20 smithi085 conmon[35961]: 2021-02-10T04:55:19.633348 2021-02-10T04:55:20.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:20 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2639 : cluster [DBG] osdmap e1231: 8 total, 8 up, 8 in 2021-02-10T04:55:20.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:20 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:55:20.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:20 smithi085 conmon[35961]: 04:55:20.607468+0000 mon.c (mon.1 2021-02-10T04:55:20.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:20 smithi085 conmon[35961]: ) 122 : audit [INF] from='client.? 172.21.15.85:0/1874414165' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:55:20.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:20 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:55:20.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:20 smithi085 conmon[35961]: 10T04:55:20.607790+0000 mon.a (mon 2021-02-10T04:55:20.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:20 smithi085 conmon[35961]: .0) 2640 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:55:20.999 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:55:20.999 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:55:21.000 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:55:21.000 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:55:21.001 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:55:21.025 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:55:21.047 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:55:21.067 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:55:21.068 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:55:21.068 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:55:21.068 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:55:21.068 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:55:21.068 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:55:21.068 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:55:21.068 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:55:21.068 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:55:21.934 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:21 smithi085 conmon[29319]: cluster 2021-02-10T04:55:20.496426+0000 mgr.y 2021-02-10T04:55:21.934 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:21 smithi085 conmon[29319]: (mgr.14140) 1362 : cluster [DBG] pgmap v2232: 204 pgs: 1 active+clean+scrubbing, 20 active+clean+wait, 183 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 744 KiB/s, 0 objects/s recovering 2021-02-10T04:55:21.934 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:21 smithi085 conmon[29319]: cluster 2021-02-10T04:55:20.640391+0000 mon.a (mon.0) 2641 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:55:21.934 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:21 smithi085 conmon[29319]: audit 2021-02-10T04:55:20.650035+0000 mon.a (mon.0) 2642 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:55:21.934 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:21 smithi085 conmon[29319]: cluster 2021-02-10T04:55:20.650088+0000 mon.a (mon.0) 2643 2021-02-10T04:55:21.935 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:21 smithi085 conmon[29319]: : cluster [DBG] osdmap e1232: 8 total, 8 up, 8 in 2021-02-10T04:55:21.935 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:21 smithi085 conmon[29319]: audit 2021-02-10T04:55:20.897880 2021-02-10T04:55:21.935 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:21 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2644 : audit [DBG] from='client.? 172.21.15.85:0/2983363832' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:55:21.935 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:21 smithi085 conmon[35961]: cluster 2021-02-10T04:55:20.496426 2021-02-10T04:55:21.935 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:21 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1362 : cluster [DBG] pgmap v2232: 204 pgs: 1 active+clean+scrubbing, 20 active+clean+wait, 183 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 744 KiB/s, 0 objects/s recovering 2021-02-10T04:55:21.935 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:21 smithi085 conmon[35961]: cluster 2021-02-10T04:55:20.640391+0000 mon.a (mon.0) 2641 2021-02-10T04:55:21.935 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:21 smithi085 conmon[35961]: : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:55:21.935 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:21 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:55:21.935 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:21 smithi085 conmon[35961]: 10T04:55:20 2021-02-10T04:55:21.935 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:21 smithi085 conmon[35961]: .650035+0000 mon.a (mon.0) 2642 : 2021-02-10T04:55:21.935 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:21 smithi085 conmon[35961]: audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:55:21.935 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:21 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:55:21.936 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:21 smithi085 conmon[35961]: -10T04:55:20.650088+0000 2021-02-10T04:55:21.936 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:21 smithi085 conmon[35961]: mon.a (mon.0) 2643 : cluster [DBG] osdmap e1232: 8 total, 8 up, 8 in 2021-02-10T04:55:21.936 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:21 smithi085 conmon[35961]: 2021-02-10T04:55:21.936 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:21 smithi085 conmon[35961]: audit 2021-02-10T04:55: 2021-02-10T04:55:21.936 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:21 smithi085 conmon[35961]: 20.897880+0000 mon.a (mon.0) 2644 : audit 2021-02-10T04:55:21.936 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:21 smithi085 conmon[35961]: [DBG] from='client.? 172.21.15.85:0/2983363832' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:55:21.947 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:21 smithi164 conmon[29288]: cluster 2021-02-10T04:55:20.496426+0000 mgr.y 2021-02-10T04:55:21.947 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:21 smithi164 conmon[29288]: (mgr.14140) 1362 : cluster [DBG] pgmap v2232: 204 pgs: 1 active+clean+scrubbing, 20 active+clean+wait, 183 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 744 KiB/s, 0 objects/s recovering 2021-02-10T04:55:21.947 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:21 smithi164 conmon[29288]: cluster 2021-02-10T04:55:20.640391+0000 mon.a (mon.0) 2641 : cluster [WRN] 2021-02-10T04:55:21.947 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:21 smithi164 conmon[29288]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:55:21.947 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:21 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:55:21.947 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:21 smithi164 conmon[29288]: T04:55:20.650035+0000 mon.a (mon.0) 2642 : audit 2021-02-10T04:55:21.947 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:21 smithi164 conmon[29288]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:55:21.947 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:21 smithi164 conmon[29288]: cluster 2021-02-10T04:55:21.948 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:21 smithi164 conmon[29288]: 2021-02-10T04:55:20.650088+0000 mon.a (mon 2021-02-10T04:55:21.948 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:21 smithi164 conmon[29288]: .0) 2643 : cluster [DBG] osdmap e1232: 8 total, 8 up, 8 in 2021-02-10T04:55:21.948 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:21 smithi164 conmon[29288]: audit 2021-02-10T04:55:21.948 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:21 smithi164 conmon[29288]: 2021-02-10T04:55:20. 2021-02-10T04:55:21.948 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:21 smithi164 conmon[29288]: 897880+0000 mon.a (mon.0) 2644 : audit 2021-02-10T04:55:21.948 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:21 smithi164 conmon[29288]: [DBG] from='client.? 172.21.15.85:0/2983363832' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:55:22.574 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:55:22.574 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2021-02-10T04:55:22.574 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 0 2021-02-10T04:55:22.575 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:55:22.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:22 smithi164 conmon[29288]: cluster 2021-02-10T04:55:22.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:22 smithi164 conmon[29288]: 2021-02-10T04:55:22.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:22 smithi164 conmon[29288]: 2021 2021-02-10T04:55:22.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:22 smithi164 conmon[29288]: - 2021-02-10T04:55:22.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:22 smithi164 conmon[29288]: 02 2021-02-10T04:55:22.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:22 smithi164 conmon[29288]: - 2021-02-10T04:55:22.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:22 smithi164 conmon[29288]: 10 2021-02-10T04:55:22.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:22 smithi164 conmon[29288]: T 2021-02-10T04:55:22.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:22 smithi164 conmon[29288]: 04 2021-02-10T04:55:22.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:22 smithi164 conmon[29288]: : 2021-02-10T04:55:22.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:22 smithi164 conmon[29288]: 55 2021-02-10T04:55:22.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:22 smithi164 conmon[29288]: : 2021-02-10T04:55:22.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:22 smithi164 conmon[29288]: 20 2021-02-10T04:55:22.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:22 smithi164 conmon[29288]: . 2021-02-10T04:55:22.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:22 smithi164 conmon[29288]: 929326+0000 osd.1 (osd.1) 506 : 2021-02-10T04:55:22.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:22 smithi164 conmon[29288]: cluster [DBG] 2.15 scrub ok 2021-02-10T04:55:22.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:22 smithi164 conmon[29288]: cluster 2021-02-10T04:55:22.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:22 smithi164 conmon[29288]: 2021-02-10T04:55:21.651729+0000 mon.a (mon.0) 2645 : cluster 2021-02-10T04:55:22.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:22 smithi164 conmon[29288]: [DBG] osdmap e1233: 8 total, 8 up, 8 in 2021-02-10T04:55:22.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:22 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:55:22.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:22 smithi164 conmon[29288]: 10T04:55:22.183562+0000 osd.5 (osd.5) 191 : cluster [DBG] 6.d scrub ok 2021-02-10T04:55:22.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:22 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:55:22.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:22 smithi164 conmon[29288]: T04:55:22.331003+0000 osd.0 (osd.0 2021-02-10T04:55:22.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:22 smithi164 conmon[29288]: ) 3 : cluster [DBG] 7.13 scrub ok 2021-02-10T04:55:22.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:22 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:55:22.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:22 smithi164 conmon[29288]: 04:55:22.497583+0000 mon.a (mon.0 2021-02-10T04:55:22.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:22 smithi164 conmon[29288]: ) 2646 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "26"}]: dispatch 2021-02-10T04:55:22.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:22 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:55:22.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:22 smithi164 conmon[29288]: T04:55:22.573614+0000 osd.2 (osd. 2021-02-10T04:55:22.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:22 smithi164 conmon[29288]: 2) 2031 : cluster [DBG] 5.b scrub ok 2021-02-10T04:55:22.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[35961]: cluster 2021-02-10T04:55:20 2021-02-10T04:55:22.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[35961]: .929326+0000 osd.1 (osd.1) 506 : cluster [DBG] 2.15 scrub ok 2021-02-10T04:55:22.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[35961]: cluster 2021-02-10T04:55:22.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[35961]: 2021-02-10T04:55:21.651729+0000 mon.a 2021-02-10T04:55:22.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[35961]: (mon.0) 2645 : cluster [DBG] osdmap e1233: 8 total, 8 up, 8 in 2021-02-10T04:55:22.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[35961]: cluster 2021-02-10T04:55:22.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[35961]: 2021-02-10T04:55:22. 2021-02-10T04:55:22.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[35961]: 183562+0000 osd.5 (osd.5) 191 : cluster 2021-02-10T04:55:22.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[35961]: [DBG] 6.d scrub ok 2021-02-10T04:55:22.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[35961]: cluster 2021-02-10T04:55:22.331003+0000 osd.0 2021-02-10T04:55:22.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[35961]: (osd.0) 3 : cluster [DBG] 7.13 scrub ok 2021-02-10T04:55:22.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:55:22.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[35961]: 02-10T04:55:22.497583+0000 2021-02-10T04:55:22.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[35961]: mon.a (mon.0) 2646 : audit 2021-02-10T04:55:22.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "26"}]: dispatch 2021-02-10T04:55:22.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[35961]: cluster 2021-02-10T04:55:22.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:55:22.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[35961]: T04:55:22.573614+0000 2021-02-10T04:55:22.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[35961]: osd.2 (osd.2) 2031 : cluster 2021-02-10T04:55:22.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[35961]: [DBG] 5.b scrub ok 2021-02-10T04:55:22.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[29319]: cluster 2021-02-10T04:55:22.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:55:22.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[29319]: :55:20.929326+0000 osd.1 (osd.1) 506 : cluster [DBG] 2.15 scrub ok 2021-02-10T04:55:22.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[29319]: cluster 2021-02-10T04:55:22.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[29319]: 2021-02-10T04:55: 2021-02-10T04:55:22.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[29319]: 21.651729+0000 mon.a (mon. 2021-02-10T04:55:22.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[29319]: 0) 2645 : cluster [DBG] 2021-02-10T04:55:22.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[29319]: osdmap e1233: 8 total, 8 up, 8 in 2021-02-10T04:55:22.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:55:22.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[29319]: -10T04:55:22. 2021-02-10T04:55:22.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[29319]: 183562+0000 osd.5 (osd.5) 2021-02-10T04:55:22.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[29319]: 191 : cluster [DBG] 6.d scrub ok 2021-02-10T04:55:22.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:55:22.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[29319]: 02-10T04:55:22.331003+0000 2021-02-10T04:55:22.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[29319]: osd.0 (osd.0) 3 : cluster [DBG] 2021-02-10T04:55:22.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[29319]: 7.13 scrub ok 2021-02-10T04:55:22.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:55:22.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[29319]: T04:55:22.497583 2021-02-10T04:55:22.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2021-02-10T04:55:22.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[29319]: 2646 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "26"}]: dispatch 2021-02-10T04:55:22.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:55:22.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[29319]: 02-10T04:55:22 2021-02-10T04:55:22.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[29319]: .573614+0000 osd.2 (osd. 2021-02-10T04:55:22.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:22 smithi085 conmon[29319]: 2) 2031 : cluster [DBG] 5.b scrub ok 2021-02-10T04:55:23.659 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:23 smithi085 conmon[29319]: cluster 2021-02-10T04:55:22.496853+0000 mgr.y ( 2021-02-10T04:55:23.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:23 smithi085 conmon[29319]: mgr.14140) 1363 : cluster [DBG] pgmap v2235: 203 pgs: 1 active+clean+scrubbing, 20 active+clean+wait, 182 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:55:23.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:23 smithi085 conmon[29319]: audit 2021-02-10T04:55:22.653041+0000 mon.a (mon.0) 2647 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "26"}]': finished 2021-02-10T04:55:23.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:23 smithi085 conmon[29319]: cluster 2021-02-10T04:55:22.653106+0000 mon.a (mon.0) 2648 : cluster [DBG] osdmap e1234: 8 total, 8 up, 8 in 2021-02-10T04:55:23.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:23 smithi085 conmon[29319]: cluster 2021-02-10T04:55:23.557325+0000 mon.a (mon.0) 2649 : cluster [DBG] osdmap e1235: 8 total, 8 up, 8 in 2021-02-10T04:55:23.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:23 smithi085 conmon[29319]: cluster 2021-02-10T04:55:23.586394+0000 osd.2 (osd.2) 2032 : cluster 2021-02-10T04:55:23.660 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:23 smithi085 conmon[29319]: [DBG] 2.21 scrub ok 2021-02-10T04:55:23.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:23 smithi164 conmon[29288]: cluster 2021-02-10T04:55:22.496853+0000 mgr.y (mgr 2021-02-10T04:55:23.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:23 smithi164 conmon[29288]: .14140) 1363 : cluster [DBG] pgmap v2235: 203 pgs: 1 active+clean+scrubbing, 20 active+clean+wait, 182 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:55:23.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:23 smithi164 conmon[29288]: audit 2021-02-10T04:55:22.653041+0000 mon.a (mon.0) 2647 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "26"}]': finished 2021-02-10T04:55:23.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:23 smithi164 conmon[29288]: cluster 2021-02-10T04:55:22.653106+0000 mon.a (mon.0) 2648 : cluster [DBG] osdmap e1234: 8 total, 8 up, 8 in 2021-02-10T04:55:23.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:23 smithi164 conmon[29288]: cluster 2021-02-10T04:55:23.557325+0000 mon.a (mon.0) 2649 : cluster [DBG] osdmap e1235: 8 total, 8 up, 8 in 2021-02-10T04:55:23.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:23 smithi164 conmon[29288]: cluster 2021-02-10T04:55 2021-02-10T04:55:23.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:23 smithi164 conmon[29288]: :23.586394+0000 osd.2 (osd.2) 2032 : cluster [DBG] 2.21 scrub ok 2021-02-10T04:55:23.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:23 smithi085 conmon[35961]: cluster 2021-02-10T04:55 2021-02-10T04:55:23.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:23 smithi085 conmon[35961]: :22.496853+0000 mgr.y (mgr.14140) 1363 : cluster [DBG] pgmap v2235: 203 pgs: 1 active+clean+scrubbing, 20 active+clean+wait, 182 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:55:23.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:23 smithi085 conmon[35961]: audit 2021-02-10T04:55:23.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:23 smithi085 conmon[35961]: 2021- 2021-02-10T04:55:23.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:23 smithi085 conmon[35961]: 02-10T04:55:22.653041+0000 mon.a 2021-02-10T04:55:23.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:23 smithi085 conmon[35961]: (mon.0) 2647 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "26"}]': finished 2021-02-10T04:55:23.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:23 smithi085 conmon[35961]: cluster 2021-02-10T04:55:23.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:23 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:55:23.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:23 smithi085 conmon[35961]: 04:55:22.653106+0000 mon.a (mon.0) 2648 : cluster [DBG] osdmap e1234: 8 total, 8 up, 8 in 2021-02-10T04:55:23.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:23 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:55:23.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:23 smithi085 conmon[35961]: -02-10T04:55:23 2021-02-10T04:55:23.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:23 smithi085 conmon[35961]: .557325+0000 mon.a (mon. 2021-02-10T04:55:23.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:23 smithi085 conmon[35961]: 0) 2649 : cluster [DBG] osdmap e1235: 8 total, 8 up, 8 in 2021-02-10T04:55:23.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:23 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:55:23.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:23 smithi085 conmon[35961]: -10T04:55:23.586394 2021-02-10T04:55:23.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:23 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 2021-02-10T04:55:23.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:23 smithi085 conmon[35961]: 2032 : cluster [DBG] 2.21 scrub ok 2021-02-10T04:55:24.326 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:55:24.327 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [ 2021-02-10T04:55:24.327 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:55:24.327 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:3850 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94187770079488 gen 3] snapc 0=[] ondisk+write+known_if_redirected e1229)", 2021-02-10T04:55:24.327 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:55:18.034772+0000", 2021-02-10T04:55:24.327 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 6.2911410800000001, 2021-02-10T04:55:24.327 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.2911653129999996, 2021-02-10T04:55:24.328 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:55:24.328 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "delayed", 2021-02-10T04:55:24.328 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:55:24.328 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:55:24.328 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:55:24.328 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 3850 2021-02-10T04:55:24.329 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:55:24.329 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:55:24.329 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:55:24.329 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:55:24.329 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:55:18.034772+0000", 2021-02-10T04:55:24.329 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:55:24.330 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:55:24.330 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:55:24.330 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:55:24.330 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:55:18.034772+0000", 2021-02-10T04:55:24.330 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.6140000000000001e-05 2021-02-10T04:55:24.331 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:55:24.331 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:55:24.331 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:55:24.331 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:55:18.034788+0000", 2021-02-10T04:55:24.331 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4569999999999998e-06 2021-02-10T04:55:24.331 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:55:24.331 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:55:24.332 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:55:24.332 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:55:18.034793+0000", 2021-02-10T04:55:24.332 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0343000000000001e-05 2021-02-10T04:55:24.332 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:55:24.332 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:55:24.333 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:55:24.333 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:55:18.034803+0000", 2021-02-10T04:55:24.333 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.0310000000000001e-06 2021-02-10T04:55:24.333 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:55:24.338 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:55:24.338 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:55:24.339 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:55:18.034810+0000", 2021-02-10T04:55:24.339 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.6999e-05 2021-02-10T04:55:24.339 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:55:24.339 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:55:24.339 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:55:24.339 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:55:18.034877+0000", 2021-02-10T04:55:24.339 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.0865000000000003e-05 2021-02-10T04:55:24.339 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:55:24.339 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:55:24.339 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for readable", 2021-02-10T04:55:24.339 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:55:18.034918+0000", 2021-02-10T04:55:24.339 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000145835 2021-02-10T04:55:24.339 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:55:24.340 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:55:24.340 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:55:24.340 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:55:24.340 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:55:24.340 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:3851 5.3 5:c609908c:::notify.5:head [watch ping cookie 94187770079488 gen 3] snapc 0=[] ondisk+write+known_if_redirected e1229)", 2021-02-10T04:55:24.340 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:55:23.034763+0000", 2021-02-10T04:55:24.340 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 1.291149737, 2021-02-10T04:55:24.340 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2912233740000001, 2021-02-10T04:55:24.340 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:55:24.340 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "delayed", 2021-02-10T04:55:24.340 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:55:24.340 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:55:24.340 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:55:24.341 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 3851 2021-02-10T04:55:24.341 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:55:24.341 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:55:24.341 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:55:24.341 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:55:24.341 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:55:23.034763+0000", 2021-02-10T04:55:24.341 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:55:24.341 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:55:24.341 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:55:24.341 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:55:24.341 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:55:23.034763+0000", 2021-02-10T04:55:24.341 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.8384000000000001e-05 2021-02-10T04:55:24.341 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:55:24.341 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:55:24.342 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:55:24.342 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:55:23.034792+0000", 2021-02-10T04:55:24.342 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.5537999999999999e-05 2021-02-10T04:55:24.342 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:55:24.342 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:55:24.342 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:55:24.342 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:55:23.034807+0000", 2021-02-10T04:55:24.342 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.5815999999999999e-05 2021-02-10T04:55:24.342 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:55:24.342 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:55:24.342 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:55:24.342 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:55:23.034823+0000", 2021-02-10T04:55:24.342 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.8399999999999995e-06 2021-02-10T04:55:24.343 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:55:24.343 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:55:24.343 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:55:24.343 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:55:23.034831+0000", 2021-02-10T04:55:24.343 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.3021000000000004e-05 2021-02-10T04:55:24.343 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:55:24.343 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:55:24.343 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:55:24.343 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:55:23.034914+0000", 2021-02-10T04:55:24.343 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.197e-05 2021-02-10T04:55:24.343 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:55:24.343 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:55:24.343 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for readable", 2021-02-10T04:55:24.343 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:55:23.034956+0000", 2021-02-10T04:55:24.344 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00019256899999999999 2021-02-10T04:55:24.344 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:55:24.344 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:55:24.344 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:55:24.344 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:55:24.344 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:55:24.344 INFO:teuthology.orchestra.run.smithi085.stdout: "num_ops": 2 2021-02-10T04:55:24.344 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:55:24.643 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:55:24 smithi164 conmon[37263]: debug 2021-02-10T04:55:24.148+0000 7f1c6b9eb700 -1 osd.5 pg_epoch: 1235 pg[6.a( empty local-lis/les=1227/1228 n=0 ec=59/59 lis/c=1227/1227 les/c/f=1228/1228/0 sis=1227) [5,6,0] r=0 lpr=1227 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x556652dd62c0) 2021-02-10T04:55:24.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:55:24 smithi085 conmon[49106]: debug 2021-02-10T04:55:24.606+0000 7f778ce21700 -1 osd.2 pg_epoch: 1236 pg[5.8( empty local-lis/les=1227/1228 n=0 ec=57/57 lis/c=1227/1227 les/c/f=1228/1228/0 sis=1227) [2,0,3] r=0 lpr=1227 crt=0'0 mlcod 0'0 active+clean] scrubber pg(5.8) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x564a937926e0) 2021-02-10T04:55:24.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:24 smithi085 conmon[29319]: audit 2021-02-10T04:55:24.140336+0000 mon.a (mon 2021-02-10T04:55:24.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:24 smithi085 conmon[29319]: .0) 2650 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:55:24.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:24 smithi085 conmon[29319]: cluster 2021-02-10T04:55:24.573769+0000 mon.a (mon.0) 2651 : cluster [DBG] osdmap e1236: 8 total, 8 up, 8 in 2021-02-10T04:55:24.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:24 smithi085 conmon[35961]: audit 2021-02-10T04:55:24.140336+0000 mon.a ( 2021-02-10T04:55:24.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:24 smithi085 conmon[35961]: mon.0) 2650 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:55:24.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:24 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:55:24.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:24 smithi085 conmon[35961]: :55:24.573769+0000 mon.a (mon.0) 2651 : cluster [DBG] osdmap e1236: 8 total, 8 up, 8 in 2021-02-10T04:55:25.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:24 smithi164 conmon[29288]: audit 2021-02-10T04:55:24.140336 2021-02-10T04:55:25.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:24 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2650 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:55:25.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:24 smithi164 conmon[29288]: cluster 2021-02-10T04:55:24 2021-02-10T04:55:25.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:24 smithi164 conmon[29288]: .573769+0000 mon.a (mon.0) 2651 : cluster [DBG] osdmap e1236: 8 total, 8 up, 8 in 2021-02-10T04:55:25.402 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset nodeep-scrub 2021-02-10T04:55:25.439 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:55:25.554 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.5 dump_blocked_ops 2021-02-10T04:55:25.648 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:55:25.663 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:55:25.890 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:55:26.133 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:55:25 smithi085 conmon[45443]: debug 2021-02-10T04:55:25.969+0000 7f71214ef700 -1 osd.1 pg_epoch: 1237 pg[5.1d( empty local-lis/les=1227/1228 n=0 ec=57/57 lis/c=1227/1227 les/c/f=1228/1228/0 sis=1227) [1,4,0] r=0 lpr=1227 crt=0'0 mlcod 0'0 active+clean] scrubber pg(5.1d) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x55dbcd60adc0) 2021-02-10T04:55:26.133 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[29319]: cluster 2021-02-10T04:55:24.497356+0000 mgr.y (mgr 2021-02-10T04:55:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:26 smithi164 conmon[29288]: cluster 2021-02-10T04:55:24.497356+0000 mgr.y (mgr.14140) 1364 : cluster [DBG] pgmap v2238: 203 pgs: 2 clean+premerge+peered, 1 active+clean+scrubbing, 20 active+clean+wait, 180 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:55:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:26 smithi164 conmon[29288]: audit 2021-02-10T04:55:25.132727+0000 mon.a (mon.0) 2652 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:55:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:26 smithi164 conmon[29288]: cephadm 2021-02-10T04:55:25.133243+0000 mgr.y (mgr.14140) 1365 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:55:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:26 smithi164 conmon[29288]: cluster 2021-02-10T04:55:25.570844+0000 mon.a (mon.0) 2653 : cluster [DBG] osdmap e1237: 8 total, 8 up, 8 in 2021-02-10T04:55:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:26 smithi164 conmon[29288]: audit 2021-02-10T04:55:25.586173+0000 mon.a (mon.0) 2654 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:55:26.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[29319]: .14140) 1364 : cluster [DBG] pgmap v2238: 203 pgs: 2 clean+premerge+peered, 1 active+clean+scrubbing, 20 active+clean+wait, 180 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:55:26.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[29319]: audit 2021-02-10T04:55:25.132727+0000 mon.a (mon.0) 2652 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:55:26.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[29319]: 2021-02-10T04:55:26.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[29319]: cephadm 2021-02-10T 2021-02-10T04:55:26.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[29319]: 04:55:25.133243+0000 mgr.y (mgr.14140) 1365 : cephadm 2021-02-10T04:55:26.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[29319]: [INF] Refreshing smithi085 facts 2021-02-10T04:55:26.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:55:26.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[29319]: 04:55:25.570844+0000 mon.a (mon 2021-02-10T04:55:26.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[29319]: .0) 2653 : cluster [DBG] osdmap e1237: 8 total, 8 up, 8 in 2021-02-10T04:55:26.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[29319]: audit 2021-02-10T04:55:26.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[29319]: 2021-02-10T04:55:25.586173+0000 2021-02-10T04:55:26.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[29319]: mon.a (mon.0) 2654 : audit [INF] 2021-02-10T04:55:26.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:55:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[35961]: cluster 2021-02-10T04:55:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[35961]: 2021-02-10T04:55:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[35961]: 2021 2021-02-10T04:55:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[35961]: -02-10T04:55:24 2021-02-10T04:55:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[35961]: .497356+0000 mgr.y (mgr.14140) 1364 : cluster [DBG] pgmap v2238: 203 pgs: 2 clean+premerge+peered, 1 active+clean+scrubbing, 20 active+clean+wait, 180 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:55:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[35961]: audit 2021 2021-02-10T04:55:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[35961]: -02-10T04:55:25.132727+0000 2021-02-10T04:55:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[35961]: mon.a (mon.0) 2652 : audit [INF] 2021-02-10T04:55:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:55:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[35961]: cephadm 2021-02-10T04: 2021-02-10T04:55:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[35961]: 55:25.133243+0000 mgr.y (mgr.14140 2021-02-10T04:55:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[35961]: ) 1365 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:55:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:55:26.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[35961]: -10T04:55:25.570844+0000 mon.a 2021-02-10T04:55:26.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[35961]: (mon.0) 2653 : cluster [DBG] osdmap e1237: 8 total, 8 up, 8 in 2021-02-10T04:55:26.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:55:26.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[35961]: -10T04:55:25.586173+0000 mon.a 2021-02-10T04:55:26.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:26 smithi085 conmon[35961]: (mon.0) 2654 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:55:27.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:27 smithi164 conmon[29288]: audit 2021-02-10T04:55:26.498767+0000 mon.a (mon 2021-02-10T04:55:27.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:27 smithi164 conmon[29288]: .0) 2655 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "4.9"}]: dispatch 2021-02-10T04:55:27.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:27 smithi164 conmon[29288]: audit 2021-02-10T04:55:26.499018+0000 mon.a (mon.0) 2656 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.9"}]: dispatch 2021-02-10T04:55:27.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:27 smithi085 conmon[29319]: audit 2021-02-10T04:55:26.498767+0000 mon.a (mon. 2021-02-10T04:55:27.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:27 smithi085 conmon[29319]: 0) 2655 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "4.9"}]: dispatch 2021-02-10T04:55:27.431 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:27 smithi085 conmon[29319]: audit 2021-02-10T04:55:26.499018+0000 mon.a (mon.0) 2656 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.9"}]: dispatch 2021-02-10T04:55:27.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:27 smithi085 conmon[35961]: audit 2021-02-10T04:55:26.498767+0000 mon.a ( 2021-02-10T04:55:27.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:27 smithi085 conmon[35961]: mon.0) 2655 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "4.9"}]: dispatch 2021-02-10T04:55:27.431 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:27 smithi085 conmon[35961]: audit 2021-02-10T04:55:26.499018+0000 mon.a (mon.0) 2656 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.9"}]: dispatch 2021-02-10T04:55:28.148 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:28 smithi085 conmon[29319]: cluster 2021-02-10T04:55 2021-02-10T04:55:28.148 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:28 smithi085 conmon[29319]: :26.497822+0000 mgr.y (mgr.14140) 1366 : cluster [DBG] pgmap v2241: 202 pgs: 1 clean+premerge+peered, 1 active+clean+scrubbing, 20 active+clean+wait, 180 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:55:28.148 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:28 smithi085 conmon[29319]: audit 2021-02-10T04:55:27.138706+0000 mon.a (mon.0) 2657 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "4.9"}]': finished 2021-02-10T04:55:28.148 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:28 smithi085 conmon[29319]: audit 2021-02-10T04:55:27.138973+0000 mon.a (mon.0) 2658 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.9"}]': finished 2021-02-10T04:55:28.148 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:28 smithi085 conmon[29319]: cluster 2021-02-10T04:55:27.139048+0000 mon.a ( 2021-02-10T04:55:28.148 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:28 smithi085 conmon[29319]: mon.0) 2659 : cluster [DBG] osdmap e1238: 8 total, 8 up, 8 in 2021-02-10T04:55:28.148 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:28 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:55:28.148 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:28 smithi085 conmon[29319]: 10T04:55:28.140240+0000 mon.a ( 2021-02-10T04:55:28.148 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:28 smithi085 conmon[29319]: mon.0) 2660 : cluster [DBG] 2021-02-10T04:55:28.148 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:28 smithi085 conmon[29319]: osdmap e1239: 8 total, 8 up, 8 in 2021-02-10T04:55:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:28 smithi164 conmon[29288]: cluster 2021-02-10T04:55:26.497822+0000 mgr.y 2021-02-10T04:55:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:28 smithi164 conmon[29288]: (mgr.14140) 1366 : cluster [DBG] pgmap v2241: 202 pgs: 1 clean+premerge+peered, 1 active+clean+scrubbing, 20 active+clean+wait, 180 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:55:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:28 smithi164 conmon[29288]: audit 2021-02-10T04:55:27.138706+0000 mon.a (mon.0) 2657 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "4.9"}]': finished 2021-02-10T04:55:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:28 smithi164 conmon[29288]: audit 2021-02-10T04:55:27.138973+0000 mon.a (mon.0) 2658 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.9"}]': finished 2021-02-10T04:55:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:28 smithi164 conmon[29288]: cluster 2021-02-10T04:55:27.139048+0000 mon.a (mon.0) 2659 : cluster [DBG] osdmap e1238: 8 total, 8 up, 8 in 2021-02-10T04:55:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:28 smithi164 conmon[29288]: cluster 2021-02-10T04:55:28.140240+0000 mon.a (mon.0) 2660 : cluster [DBG] osdmap e1239: 8 total, 8 up, 8 in 2021-02-10T04:55:28.395 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:55:28.414 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:55:28.436 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:55:28.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:28 smithi085 conmon[35961]: cluster 2021-02-10T04:55:26.497822+0000 2021-02-10T04:55:28.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:28 smithi085 conmon[35961]: mgr.y (mgr.14140) 1366 : cluster [DBG] pgmap v2241: 202 pgs: 1 clean+premerge+peered, 1 active+clean+scrubbing, 20 active+clean+wait, 180 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:55:28.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:28 smithi085 conmon[35961]: audit 2021-02-10T04:55:28.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:28 smithi085 conmon[35961]: 2021-02-10T04:55:27.138706+0000 mon.a (mon. 2021-02-10T04:55:28.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:28 smithi085 conmon[35961]: 0) 2657 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "4.9"}]': finished 2021-02-10T04:55:28.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:28 smithi085 conmon[35961]: audit 2021-02-10T04:55:28.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:28 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:55:28.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:28 smithi085 conmon[35961]: 04:55:27.138973+0000 mon.a (mon. 2021-02-10T04:55:28.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:28 smithi085 conmon[35961]: 0) 2658 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.9"}]': finished 2021-02-10T04:55:28.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:28 smithi085 conmon[35961]: cluster 2021-02-10T04:55:28.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:28 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:55:28.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:28 smithi085 conmon[35961]: 04:55:27.139048+0000 mon.a (mon. 2021-02-10T04:55:28.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:28 smithi085 conmon[35961]: 0) 2659 : cluster [DBG] osdmap e1238: 8 total, 8 up, 8 in 2021-02-10T04:55:28.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:28 smithi085 conmon[35961]: cluster 2021-02-10T04:55:28.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:28 smithi085 conmon[35961]: 2021-02-10T04:55:28. 2021-02-10T04:55:28.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:28 smithi085 conmon[35961]: 140240+0000 mon.a (mon.0) 2660 : cluster 2021-02-10T04:55:28.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:28 smithi085 conmon[35961]: [DBG] osdmap e1239: 8 total, 8 up, 8 in 2021-02-10T04:55:28.457 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:55:28.480 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:55:28.503 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:55:28.525 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:55:28.548 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:55:28.549 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:55:28.549 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:55:28.549 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:55:28.549 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:55:28.549 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:55:28.549 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:55:28.550 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:55:28.550 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:55:29.173 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is unset 2021-02-10T04:55:29.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:29 smithi085 conmon[29319]: audit 2021-02-10T04:55:28.370376+0000 mon.c ( 2021-02-10T04:55:29.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:29 smithi085 conmon[29319]: mon.1) 123 : audit [DBG] from='client.? 172.21.15.85:0/4007709560' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:55:29.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:29 smithi085 conmon[29319]: audit 2021-02-10T04:55:28.402390+0000 mon.c (mon.1) 124 : audit [INF] from='client.? 172.21.15.85:0/2596293397' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:55:29.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:29 smithi085 conmon[29319]: audit 2021-02-10T04:55:28.402692+0000 mon.a (mon.0) 2661 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:55:29.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:29 smithi085 conmon[35961]: audit 2021-02-10T04:55:29.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:29 smithi085 conmon[35961]: 2021-02-10T04:55:29.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:29 smithi085 conmon[35961]: 2021 2021-02-10T04:55:29.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:29 smithi085 conmon[35961]: -02-10T04:55: 2021-02-10T04:55:29.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:29 smithi085 conmon[35961]: 28.370376+0000 mon.c (mon.1) 123 : audit [DBG] from='client.? 172.21.15.85:0/4007709560' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:55:29.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:29 smithi085 conmon[35961]: audit 2021-02-10T04:55:29.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:29 smithi085 conmon[35961]: 2021-02-10T04:55:28.402390 2021-02-10T04:55:29.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:29 smithi085 conmon[35961]: +0000 mon.c (mon.1) 124 : audit [INF] 2021-02-10T04:55:29.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:29 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/2596293397' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:55:29.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:29 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:55:29.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:29 smithi085 conmon[35961]: :55:28.402692+0000 mon.a (mon. 2021-02-10T04:55:29.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:29 smithi085 conmon[35961]: 0) 2661 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:55:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:29 smithi164 conmon[29288]: audit 2021-02-10T04:55:28.370376+0000 mon.c 2021-02-10T04:55:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:29 smithi164 conmon[29288]: (mon.1) 123 : audit [DBG] from='client.? 172.21.15.85:0/4007709560' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:55:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:29 smithi164 conmon[29288]: audit 2021-02-10T04:55:28.402390+0000 mon.c (mon.1) 124 : audit [INF] from='client.? 172.21.15.85:0/2596293397' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:55:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:29 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:55:29.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:29 smithi164 conmon[29288]: 10T04:55:28.402692+0000 mon.a (mon.0) 2661 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:55:29.644 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:55:29 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:55:30.171 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2021-02-10T04:55:30.171 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:55:30.171 INFO:tasks.thrashosds.thrasher:inject_pause on 5 2021-02-10T04:55:30.171 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2021-02-10T04:55:30.172 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2021-02-10T04:55:30.172 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.5 config set filestore_inject_stall 3 2021-02-10T04:55:30.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[29319]: cluster 2021-02-10T04:55:28.498279+0000 mgr.y (mgr.14140) 2021-02-10T04:55:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[29319]: 1367 : cluster [DBG] pgmap v2244: 202 pgs: 1 active+clean+scrubbing, 1 clean+premerge+peered, 14 active+clean+wait, 186 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:55:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[29319]: cluster 2021-02-10T04:55:29.154930+0000 mon.a (mon.0) 2662 : cluster 2021-02-10T04:55:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[29319]: [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:55:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:55:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[29319]: 02-10T04:55:29.154955+0000 mon.a (mon.0 2021-02-10T04:55:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[29319]: ) 2663 : cluster [INF] Cluster is now healthy 2021-02-10T04:55:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:55:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[29319]: -10T04:55:29.164291+0000 2021-02-10T04:55:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[29319]: mon.a (mon.0) 2664 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:55:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[29319]: 2021-02-10T04:55:30.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[29319]: cluster 2021-02-10T04:55:30.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[29319]: 2021-02-10T04:55:29.164363+0000 mon.a 2021-02-10T04:55:30.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[29319]: (mon.0) 2665 : cluster [DBG] osdmap e1240: 8 total, 8 up, 8 in 2021-02-10T04:55:30.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:55:30.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[29319]: 02-10T04:55:29.315146+0000 2021-02-10T04:55:30.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[29319]: mgr.y (mgr.14140) 1368 : audit [DBG] 2021-02-10T04:55:30.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[29319]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:55:30.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[29319]: cluster 2021-02-10T04:55 2021-02-10T04:55:30.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[29319]: :29.712620+0000 osd.6 (osd.6) 2021-02-10T04:55:30.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[29319]: 186 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:55:30.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:55:30.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[29319]: 10T04:55:29.715108+0000 osd.6 2021-02-10T04:55:30.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[29319]: (osd.6) 187 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:55:30.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[29319]: 2021-02-10T04:55:30.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[35961]: cluster 2021-02-10T04:55:28.498279+0000 2021-02-10T04:55:30.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[35961]: mgr.y (mgr.14140) 1367 : cluster [DBG] pgmap v2244: 202 pgs: 1 active+clean+scrubbing, 1 clean+premerge+peered, 14 active+clean+wait, 186 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:55:30.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[35961]: cluster 2021-02-10T04:55:30.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[35961]: 2021-02-10T04:55:29.154930+0000 mon.a (mon.0) 2662 : cluster [INF] 2021-02-10T04:55:30.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[35961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:55:30.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:55:30.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[35961]: -02-10T04:55: 2021-02-10T04:55:30.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[35961]: 29.154955+0000 mon.a (mon. 2021-02-10T04:55:30.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[35961]: 0) 2663 : cluster [INF] Cluster is now healthy 2021-02-10T04:55:30.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[35961]: audit 2021 2021-02-10T04:55:30.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[35961]: -02-10T04:55:29.164291+0000 mon.a (mon. 2021-02-10T04:55:30.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[35961]: 0) 2664 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:55:30.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:55:30.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[35961]: T04:55:29.164363+0000 2021-02-10T04:55:30.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[35961]: mon.a (mon.0) 2665 : cluster 2021-02-10T04:55:30.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[35961]: [DBG] osdmap e1240: 8 total, 8 up, 8 in 2021-02-10T04:55:30.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[35961]: audit 2021-02-10T04:55:30.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[35961]: 2021-02-10T04:55:29.315146+0000 2021-02-10T04:55:30.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[35961]: mgr.y (mgr.14140) 1368 : 2021-02-10T04:55:30.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[35961]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:55:30.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:55:30.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[35961]: 04:55:29.712620+0000 osd.6 (osd.6) 186 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:55:30.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[35961]: cluster 2021-02-10T04:55 2021-02-10T04:55:30.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[35961]: :29.715108+0000 osd.6 (osd.6) 187 : cluster 2021-02-10T04:55:30.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:30 smithi085 conmon[35961]: [DBG] purged_snaps scrub ok 2021-02-10T04:55:30.488 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:30 smithi164 conmon[29288]: cluster 2021-02-10T04:55:28.498279+0000 mgr.y ( 2021-02-10T04:55:30.488 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:30 smithi164 conmon[29288]: mgr.14140) 1367 : cluster [DBG] pgmap v2244: 202 pgs: 1 active+clean+scrubbing, 1 clean+premerge+peered, 14 active+clean+wait, 186 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:55:30.488 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:30 smithi164 conmon[29288]: cluster 2021-02-10T04:55:29.154930+0000 mon.a (mon 2021-02-10T04:55:30.488 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:30 smithi164 conmon[29288]: .0) 2662 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:55:30.488 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:30 smithi164 conmon[29288]: cluster 2021-02-10T04:55:30.488 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:30 smithi164 conmon[29288]: 2021-02-10T04:55:29 2021-02-10T04:55:30.488 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:30 smithi164 conmon[29288]: .154955+0000 mon.a (mon.0) 2663 : cluster [INF] 2021-02-10T04:55:30.488 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:30 smithi164 conmon[29288]: Cluster is now healthy 2021-02-10T04:55:30.488 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:30 smithi164 conmon[29288]: audit 2021-02-10T04:55:30.488 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:30 smithi164 conmon[29288]: 2021-02-10T04:55:29.164291+0000 mon.a (mon.0 2021-02-10T04:55:30.489 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:30 smithi164 conmon[29288]: ) 2664 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:55:30.489 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:30 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:55:30.489 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:30 smithi164 conmon[29288]: 55:29.164363+0000 mon.a (mon.0) 2021-02-10T04:55:30.489 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:30 smithi164 conmon[29288]: 2665 : cluster [DBG] osdmap e1240: 8 total, 8 up, 8 in 2021-02-10T04:55:30.489 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:30 smithi164 conmon[29288]: audit 2021-02-10T04: 2021-02-10T04:55:30.489 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:30 smithi164 conmon[29288]: 55:29.315146+0000 mgr.y (mgr.14140) 1368 : audit 2021-02-10T04:55:30.489 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:30 smithi164 conmon[29288]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:55:30.489 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:30 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:55:30.489 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:30 smithi164 conmon[29288]: 55:29.712620+0000 osd.6 ( 2021-02-10T04:55:30.489 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:30 smithi164 conmon[29288]: osd.6) 186 : cluster 2021-02-10T04:55:30.489 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:30 smithi164 conmon[29288]: [DBG] purged_snaps scrub starts 2021-02-10T04:55:30.489 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:30 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:55:30.489 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:30 smithi164 conmon[29288]: -10T04:55:29. 2021-02-10T04:55:30.489 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:30 smithi164 conmon[29288]: 715108+0000 osd.6 (osd.6) 187 : cluster [DBG] 2021-02-10T04:55:30.490 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:30 smithi164 conmon[29288]: purged_snaps scrub ok 2021-02-10T04:55:30.762 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:55:31.238 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:31 smithi164 conmon[29288]: cluster 2021-02-10T04:55:30.169998+0000 mon.a 2021-02-10T04:55:31.239 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:31 smithi164 conmon[29288]: (mon.0) 2666 : cluster [DBG] osdmap e1241: 8 total, 8 up, 8 in 2021-02-10T04:55:31.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:31 smithi085 conmon[35961]: cluster 2021-02-10T04:55:30.169998 2021-02-10T04:55:31.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:31 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2666 : cluster [DBG] osdmap e1241: 8 total, 8 up, 8 in 2021-02-10T04:55:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:31 smithi085 conmon[29319]: cluster 2021-02-10T04:55:30.169998+0000 mon.a ( 2021-02-10T04:55:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:31 smithi085 conmon[29319]: mon.0) 2666 : cluster [DBG] osdmap e1241: 8 total, 8 up, 8 in 2021-02-10T04:55:31.712 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:55:31.929 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:55:31.974 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set noscrub 2021-02-10T04:55:32.196 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:55:32.313 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:55:32.313 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:55:32.404 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:32 smithi085 conmon[35961]: cluster 2021-02-10T04:55:30.498752+0000 2021-02-10T04:55:32.404 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:32 smithi085 conmon[35961]: mgr.y (mgr.14140) 1369 : cluster [DBG] pgmap v2247: 202 pgs: 1 active+clean+scrubbing, 14 active+clean+wait, 187 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 1.2 KiB/s wr, 28 op/s; 183 B/s, 2 objects/s recovering 2021-02-10T04:55:32.404 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:32 smithi085 conmon[29319]: cluster 2021-02-10T04:55:30.498752+0000 2021-02-10T04:55:32.404 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:32 smithi085 conmon[29319]: mgr.y (mgr.14140) 1369 : cluster [DBG] pgmap v2247: 202 pgs: 1 active+clean+scrubbing, 14 active+clean+wait, 187 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 1.2 KiB/s wr, 28 op/s; 183 B/s, 2 objects/s recovering 2021-02-10T04:55:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:32 smithi164 conmon[29288]: cluster 2021-02-10T04:55:30.498752+0000 mgr.y (mgr.14140) 1369 : cluster [DBG] pgmap v2247: 202 pgs: 1 active+clean+scrubbing, 14 active+clean+wait, 187 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 1.2 KiB/s wr, 28 op/s; 183 B/s, 2 objects/s recovering 2021-02-10T04:55:33.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: audit 2021-02-10T04:55:33.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: 2021-02-10T04:55:33.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: 2021 2021-02-10T04:55:33.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: - 2021-02-10T04:55:33.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: 02 2021-02-10T04:55:33.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: - 2021-02-10T04:55:33.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: 10 2021-02-10T04:55:33.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: T 2021-02-10T04:55:33.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: 04 2021-02-10T04:55:33.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: : 2021-02-10T04:55:33.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: 55 2021-02-10T04:55:33.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: : 2021-02-10T04:55:33.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: 32 2021-02-10T04:55:33.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: . 2021-02-10T04:55:33.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: 500220 2021-02-10T04:55:33.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: +0000 2021-02-10T04:55:33.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: 2021-02-10T04:55:33.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: mon.a 2021-02-10T04:55:33.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: ( 2021-02-10T04:55:33.188 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: mon 2021-02-10T04:55:33.188 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: . 2021-02-10T04:55:33.188 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: 0 2021-02-10T04:55:33.188 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: ) 2021-02-10T04:55:33.188 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: 2667 2021-02-10T04:55:33.188 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: : 2021-02-10T04:55:33.188 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: audit 2021-02-10T04:55:33.188 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: 2021-02-10T04:55:33.188 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: [INF] 2021-02-10T04:55:33.188 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: 2021-02-10T04:55:33.188 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "25"}]: dispatch 2021-02-10T04:55:33.188 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[29319]: 2021-02-10T04:55:33.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[35961]: audit 2021-02-10T04:55:32.500220+0000 mon.a (mon.0) 2667 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "25"}]: dispatch 2021-02-10T04:55:33.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: audit 2021-02-10T04:55:33.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: 2021-02-10T04:55:33.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: 2021 2021-02-10T04:55:33.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: - 2021-02-10T04:55:33.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: 02 2021-02-10T04:55:33.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: - 2021-02-10T04:55:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: 10 2021-02-10T04:55:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: T 2021-02-10T04:55:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: 04 2021-02-10T04:55:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: : 2021-02-10T04:55:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: 55 2021-02-10T04:55:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: : 2021-02-10T04:55:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: 32 2021-02-10T04:55:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: . 2021-02-10T04:55:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: 500220 2021-02-10T04:55:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: +0000 2021-02-10T04:55:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: 2021-02-10T04:55:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: mon.a 2021-02-10T04:55:33.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: ( 2021-02-10T04:55:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: mon 2021-02-10T04:55:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: . 2021-02-10T04:55:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: 0 2021-02-10T04:55:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: ) 2021-02-10T04:55:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: 2667 2021-02-10T04:55:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: : 2021-02-10T04:55:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: audit 2021-02-10T04:55:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: 2021-02-10T04:55:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: [INF] 2021-02-10T04:55:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: 2021-02-10T04:55:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "25"}]: dispatch 2021-02-10T04:55:33.646 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:33 smithi164 conmon[29288]: 2021-02-10T04:55:33.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:55:33 smithi085 conmon[49106]: debug 2021-02-10T04:55:33.633+0000 7f7788e19700 -1 osd.2 pg_epoch: 1243 pg[5.8( empty local-lis/les=1227/1228 n=0 ec=57/57 lis/c=1227/1227 les/c/f=1228/1228/0 sis=1227) [2,0,3] r=0 lpr=1227 crt=0'0 mlcod 0'0 active+clean] scrubber pg(5.8) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x564a9fcee160) 2021-02-10T04:55:34.407 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:34 smithi085 conmon[29319]: cluster 2021-02-10T04:55:32.499488+0000 mgr.y (mgr.14140) 1370 : cluster [DBG] pgmap v2248: 202 pgs: 1 active+clean+scrubbing, 14 active+clean+wait, 187 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 954 B/s wr, 21 op/s; 265 B/s, 3 objects/s recovering 2021-02-10T04:55:35.143 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:34 smithi085 conmon[29319]: 2021-02-10T04:55:35.144 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:34 smithi085 conmon[29319]: audit 2021-02-10T04:55:33.178277+0000 mon.a (mon.0) 2668 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "25"}]': finished 2021-02-10T04:55:35.144 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:34 smithi085 conmon[29319]: cluster 2021-02-10T04:55:33.178415+0000 mon.a (mon.0) 2669 : cluster [DBG] osdmap e1242: 8 total, 8 up, 8 in 2021-02-10T04:55:35.144 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:34 smithi085 conmon[29319]: cluster 2021-02-10T04:55:33.559089+0000 mon.a (mon.0) 2670 : cluster [DBG] osdmap e1243: 8 total, 8 up, 8 in 2021-02-10T04:55:35.144 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:34 smithi085 conmon[35961]: cluster 2021-02-10T04:55:32.499488+0000 mgr.y (mgr.14140) 1370 : cluster [DBG] pgmap v2248: 202 pgs: 1 active+clean+scrubbing, 14 active+clean+wait, 187 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 954 B/s wr, 21 op/s; 265 B/s, 3 objects/s recovering 2021-02-10T04:55:35.144 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:34 smithi085 conmon[35961]: audit 2021-02-10T04:55:33.178277+0000 mon.a (mon.0) 2668 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "25"}]': finished 2021-02-10T04:55:35.144 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:34 smithi085 conmon[35961]: cluster 2021-02-10T04:55:33.178415+0000 mon.a (mon.0) 2669 : cluster [DBG] osdmap e1242: 8 total, 8 up, 8 in 2021-02-10T04:55:35.144 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:34 smithi085 conmon[35961]: cluster 2021-02-10T04:55:33.559089+0000 mon.a (mon.0) 2670 : cluster [DBG] osdmap e1243: 8 total, 8 up, 8 in 2021-02-10T04:55:35.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:34 smithi164 conmon[29288]: cluster 2021-02-10T04:55:32.499488+0000 mgr.y (mgr.14140) 2021-02-10T04:55:35.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:34 smithi164 conmon[29288]: 1370 : cluster [DBG] pgmap v2248: 202 pgs: 1 active+clean+scrubbing, 14 active+clean+wait, 187 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 954 B/s wr, 21 op/s; 265 B/s, 3 objects/s recovering 2021-02-10T04:55:35.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:34 smithi164 conmon[29288]: audit 2021-02-10T04:55:33.178277+0000 mon.a (mon.0) 2668 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "25"}]': finished 2021-02-10T04:55:35.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:34 smithi164 conmon[29288]: cluster 2021-02-10T04:55:33.178415+0000 mon.a (mon.0) 2669 : cluster [DBG] osdmap e1242: 8 total, 8 up, 8 in 2021-02-10T04:55:35.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:34 smithi164 conmon[29288]: cluster 2021-02-10T04:55:33.559089+0000 mon.a (mon.0) 2670 : cluster [DBG] osdmap e1243: 8 total, 8 up, 8 in 2021-02-10T04:55:35.146 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:55:35.146 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:55:35.146 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:55:35.146 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:55:35.146 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:55:35.147 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:55:35.147 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:55:35.147 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:55:35.147 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:55:35.147 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:55:35.147 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:55:35.147 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:55:35.147 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:55:35.147 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:55:35.147 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:55:35.147 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:55:35.579 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is set 2021-02-10T04:55:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:35 smithi164 conmon[29288]: cluster 2021-02-10T04:55:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:35 smithi164 conmon[29288]: 2021-02-10T04:55:34. 2021-02-10T04:55:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:35 smithi164 conmon[29288]: 500029+0000 mgr.y (mgr.14140) 1371 : cluster [DBG] pgmap v2251: 202 pgs: 1 active+clean+scrubbing, 201 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 959 B/s rd, 0 op/s; 129 B/s, 1 objects/s recovering 2021-02-10T04:55:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:35 smithi164 conmon[29288]: cluster 2021-02-10T04:55:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:35 smithi164 conmon[29288]: 2021-02-10T04:55:34.567332+0000 mon.a (mon.0) 2671 : cluster 2021-02-10T04:55:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:35 smithi164 conmon[29288]: [DBG] osdmap e1244: 8 total, 8 up, 8 in 2021-02-10T04:55:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:35 smithi164 conmon[29288]: audit 2021-02-10T04:55:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:35 smithi164 conmon[29288]: 2021-02-10T04:55:34.928645+0000 mon.c (mon.1) 125 : 2021-02-10T04:55:35.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:35 smithi164 conmon[29288]: audit [DBG] from='client.? 172.21.15.85:0/4183259889' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:55:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:35 smithi164 conmon[29288]: audit 2021-02-10T04:55:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:35 smithi164 conmon[29288]: 2021-02-10T04:55:35.126376+0000 mon.c (mon.1 2021-02-10T04:55:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:35 smithi164 conmon[29288]: ) 126 : audit [INF] from='client.? 172.21.15.85:0/794617389' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:55:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:35 smithi164 conmon[29288]: audit 2021-02-10T04:55:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:35 smithi164 conmon[29288]: 2021-02-10T04:55: 2021-02-10T04:55:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:35 smithi164 conmon[29288]: 35.126674+0000 mon.a (mon.0) 2672 : 2021-02-10T04:55:35.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:35 smithi164 conmon[29288]: audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:55:35.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[29319]: cluster 2021-02-10T04:55:35.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[29319]: 2021-02-10T04:55:35.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[29319]: 2021-02-10T04:55:34.500029+0000 2021-02-10T04:55:35.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[29319]: mgr.y (mgr.14140) 1371 : cluster [DBG] pgmap v2251: 202 pgs: 1 active+clean+scrubbing, 201 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 959 B/s rd, 0 op/s; 129 B/s, 1 objects/s recovering 2021-02-10T04:55:35.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[29319]: cluster 2021-02-10T04:55:34. 2021-02-10T04:55:35.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[29319]: 567332+0000 mon.a (mon.0) 2671 : cluster 2021-02-10T04:55:35.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[29319]: [DBG] osdmap e1244: 8 total, 8 up, 8 in 2021-02-10T04:55:35.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:55:35.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[29319]: 02-10T04:55:34.928645+0000 mon.c 2021-02-10T04:55:35.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[29319]: (mon.1) 125 : audit [DBG] 2021-02-10T04:55:35.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[29319]: from='client.? 172.21.15.85:0/4183259889' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:55:35.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[29319]: audit 2021-02-10T04:55:35.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[29319]: 2021-02-10T04:55:35 2021-02-10T04:55:35.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[29319]: .126376+0000 mon.c (mon.1 2021-02-10T04:55:35.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[29319]: ) 126 : audit [INF] from='client.? 172.21.15.85:0/794617389' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:55:35.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:55:35.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[29319]: 02-10T04:55:35.126674+0000 2021-02-10T04:55:35.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[29319]: mon.a (mon.0) 2672 : audit 2021-02-10T04:55:35.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[29319]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:55:35.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[35961]: cluster 2021-02-10T04:55:34.500029+0000 mgr.y 2021-02-10T04:55:35.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[35961]: (mgr.14140) 1371 : cluster [DBG] pgmap v2251: 202 pgs: 1 active+clean+scrubbing, 201 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 959 B/s rd, 0 op/s; 129 B/s, 1 objects/s recovering 2021-02-10T04:55:35.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:55:35.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[35961]: 04:55:34.567332+0000 mon.a (mon.0) 2671 : cluster [DBG] osdmap e1244: 8 total, 8 up, 8 in 2021-02-10T04:55:35.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[35961]: audit 2021-02-10T04:55: 2021-02-10T04:55:35.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[35961]: 34.928645+0000 mon.c (mon.1) 125 : audit [DBG] from='client.? 172.21.15.85:0/4183259889' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:55:35.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:55:35.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[35961]: :55:35.126376+0000 mon.c (mon. 2021-02-10T04:55:35.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[35961]: 1) 126 : audit [INF] from='client.? 172.21.15.85:0/794617389' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:55:35.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[35961]: audit 2021 2021-02-10T04:55:35.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[35961]: -02-10T04:55:35 2021-02-10T04:55:35.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[35961]: .126674+0000 mon.a (mon.0) 2672 2021-02-10T04:55:35.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:35 smithi085 conmon[35961]: : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:55:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:36 smithi164 conmon[29288]: cluster 2021-02-10T04:55:35.563090+0000 mon.a (mon.0) 2673 : cluster [WRN] 2021-02-10T04:55:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:36 smithi164 conmon[29288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:55:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:36 smithi164 conmon[29288]: audit 2021-02-10T04:55:35.572229+0000 mon.a (mon.0) 2674 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:55:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:36 smithi164 conmon[29288]: cluster 2021-02-10T04:55: 2021-02-10T04:55:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:36 smithi164 conmon[29288]: 35.572271+0000 mon.a (mon.0) 2675 : cluster [DBG] osdmap e1245: 8 total, 8 up, 8 in 2021-02-10T04:55:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:36 smithi164 conmon[29288]: audit 2021 2021-02-10T04:55:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:36 smithi164 conmon[29288]: -02-10T04:55:36.501330+0000 mon.a (mon.0) 2021-02-10T04:55:36.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:36 smithi164 conmon[29288]: 2676 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "24"}]: dispatch 2021-02-10T04:55:36.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:36 smithi085 conmon[29319]: cluster 2021-02-10T04:55:35.563090+0000 mon.a (mon 2021-02-10T04:55:36.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:36 smithi085 conmon[29319]: .0) 2673 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:55:36.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:36 smithi085 conmon[29319]: audit 2021-02-10T04:55:35.572229+0000 mon.a (mon.0) 2674 : audit 2021-02-10T04:55:36.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:36 smithi085 conmon[29319]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:55:36.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:36 smithi085 conmon[29319]: cluster 2021-02-10T04:55: 2021-02-10T04:55:36.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:36 smithi085 conmon[29319]: 35.572271+0000 mon.a (mon.0) 2675 : cluster 2021-02-10T04:55:36.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:36 smithi085 conmon[29319]: [DBG] osdmap e1245: 8 total, 8 up, 8 in 2021-02-10T04:55:36.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:36 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:55:36.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:36 smithi085 conmon[29319]: T04:55:36.501330+0000 mon.a (mon 2021-02-10T04:55:36.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:36 smithi085 conmon[29319]: .0) 2676 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "24"}]: dispatch 2021-02-10T04:55:36.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:36 smithi085 conmon[35961]: cluster 2021-02-10T04:55:35.563090 2021-02-10T04:55:36.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:36 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2673 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:55:36.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:36 smithi085 conmon[35961]: audit 2021-02-10T04:55:36.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:36 smithi085 conmon[35961]: 2021-02-10T04:55:35.572229+0000 mon.a (mon. 2021-02-10T04:55:36.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:36 smithi085 conmon[35961]: 0) 2674 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:55:36.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:36 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:55:36.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:36 smithi085 conmon[35961]: -02-10T04:55:35.572271+0000 2021-02-10T04:55:36.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:36 smithi085 conmon[35961]: mon.a (mon.0) 2675 : cluster [DBG] 2021-02-10T04:55:36.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:36 smithi085 conmon[35961]: osdmap e1245: 8 total, 8 up, 8 in 2021-02-10T04:55:36.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:36 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:55:36.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:36 smithi085 conmon[35961]: :55:36.501330+0000 mon.a (mon. 2021-02-10T04:55:36.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:36 smithi085 conmon[35961]: 0) 2676 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "24"}]: dispatch 2021-02-10T04:55:37.762 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.5 dump_historic_ops 2021-02-10T04:55:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:37 smithi164 conmon[29288]: cluster 2021-02-10T04:55:36.500496 2021-02-10T04:55:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:37 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1372 : cluster [DBG] pgmap v2254: 201 pgs: 1 active+clean+scrubbing, 200 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:55:37.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:37 smithi164 conmon[29288]: audit 2021-02-10T04:55:36.574068+0000 mon.a (mon 2021-02-10T04:55:37.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:37 smithi164 conmon[29288]: .0) 2677 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "24"}]': finished 2021-02-10T04:55:37.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:37 smithi164 conmon[29288]: cluster 2021-02-10T04:55:37.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:37 smithi164 conmon[29288]: 2021-02-10T04:55:36.574135+0000 2021-02-10T04:55:37.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:37 smithi164 conmon[29288]: mon.a (mon.0) 2678 : cluster [DBG] osdmap e1246: 8 total, 8 up, 8 in 2021-02-10T04:55:37.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:37 smithi085 conmon[29319]: cluster 2021-02-10T04:55:36.500496+0000 mgr.y (mgr.14140) 1372 : cluster [DBG] pgmap v2254: 201 pgs: 1 active+clean+scrubbing, 200 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:55:37.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:37 smithi085 conmon[29319]: audit 2021-02-10T04:55:36.574068+0000 mon.a (mon.0) 2677 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "24"}]': finished 2021-02-10T04:55:37.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:37 smithi085 conmon[29319]: cluster 2021-02-10T04:55:36.574135+0000 mon.a (mon.0) 2678 : cluster [DBG] osdmap e1246: 8 total, 8 up, 8 in 2021-02-10T04:55:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:37 smithi085 conmon[35961]: cluster 2021-02-10T04:55:36. 2021-02-10T04:55:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:37 smithi085 conmon[35961]: 500496+0000 mgr.y (mgr.14140) 1372 : cluster [DBG] pgmap v2254: 201 pgs: 1 active+clean+scrubbing, 200 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:55:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:37 smithi085 conmon[35961]: audit 2021-02-10T04:55:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:37 smithi085 conmon[35961]: 2021-02-10T04:55 2021-02-10T04:55:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:37 smithi085 conmon[35961]: :36.574068+0000 mon.a ( 2021-02-10T04:55:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:37 smithi085 conmon[35961]: mon.0) 2677 : audit [INF] 2021-02-10T04:55:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:37 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "24"}]': finished 2021-02-10T04:55:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:37 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:55:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:37 smithi085 conmon[35961]: 02-10T04:55:36 2021-02-10T04:55:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:37 smithi085 conmon[35961]: .574135+0000 mon.a (mon.0) 2021-02-10T04:55:37.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:37 smithi085 conmon[35961]: 2678 : cluster [DBG] osdmap e1246: 8 total, 8 up, 8 in 2021-02-10T04:55:38.062 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:55:38.504 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:55:38.531 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set nodeep-scrub 2021-02-10T04:55:38.616 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:55:38.616 INFO:teuthology.orchestra.run.smithi164.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2021-02-10T04:55:38.616 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:55:38.764 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:55:38.789 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:55:38.804 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:38 smithi164 conmon[29288]: cluster 2021-02-10T04:55:37.579727+0000 2021-02-10T04:55:38.804 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:38 smithi164 conmon[29288]: mon.a (mon.0) 2679 : cluster [DBG] osdmap e1247: 8 total, 8 up, 8 in 2021-02-10T04:55:38.805 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:38 smithi164 conmon[29288]: audit 2021-02-10T04:55:38.254436+0000 mon.a (mon.0) 2680 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:55:38.805 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:38 smithi164 conmon[29288]: audit 2021-02-10T04:55:38.255496+0000 mon.a (mon.0) 2681 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:55:38.805 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:38 smithi164 conmon[29288]: audit 2021-02-10T04:55:38.410533+0000 mon.a (mon.0) 2682 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 2]}]: dispatch 2021-02-10T04:55:38.805 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:38 smithi164 conmon[29288]: audit 2021-02-10T04:55:38.410727+0000 mon.a (mon.0) 2683 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2021-02-10T04:55:38.805 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:38 smithi164 conmon[29288]: audit 2021-02-10T04:55:38.410881+0000 mon.a (mon.0) 2684 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.28", "id": [7, 2]}]: dispatch 2021-02-10T04:55:38.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:38 smithi085 conmon[35961]: cluster 2021-02-10T04:55:37. 2021-02-10T04:55:38.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:38 smithi085 conmon[35961]: 579727+0000 mon.a (mon.0) 2679 : cluster [DBG] osdmap e1247: 8 total, 8 up, 8 in 2021-02-10T04:55:38.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:38 smithi085 conmon[35961]: audit 2021-02-10T04:55:38.254436+0000 mon.a (mon.0) 2680 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:55:38.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:38 smithi085 conmon[35961]: audit 2021-02-10T04:55:38.255496+0000 mon.a (mon.0) 2681 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:55:38.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:38 smithi085 conmon[35961]: audit 2021-02-10T04:55:38.410533+0000 mon.a (mon.0) 2682 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 2]}]: dispatch 2021-02-10T04:55:38.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:38 smithi085 conmon[35961]: audit 2021-02-10T04:55:38.410727+0000 mon.a (mon.0) 2683 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2021-02-10T04:55:38.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:38 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:55:38.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:38 smithi085 conmon[35961]: 02-10T04:55:38.410881+0000 mon.a (mon.0) 2684 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.28", "id": [7, 2]}]: dispatch 2021-02-10T04:55:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:38 smithi085 conmon[29319]: cluster 2021-02-10T04:55:37 2021-02-10T04:55:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:38 smithi085 conmon[29319]: .579727+0000 mon.a (mon.0) 2679 : cluster [DBG] osdmap e1247: 8 total, 8 up, 8 in 2021-02-10T04:55:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:38 smithi085 conmon[29319]: audit 2021-02-10T04:55:38.254436+0000 mon.a (mon.0) 2680 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:55:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:38 smithi085 conmon[29319]: audit 2021-02-10T04:55:38.255496+0000 mon.a (mon.0) 2681 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:55:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:38 smithi085 conmon[29319]: audit 2021-02-10T04:55:38.410533+0000 mon.a (mon.0) 2682 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 2]}]: dispatch 2021-02-10T04:55:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:38 smithi085 conmon[29319]: audit 2021-02-10T04:55:38.410727+0000 mon.a (mon.0) 2683 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2021-02-10T04:55:38.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:38 smithi085 conmon[29319]: audit 2021-02-10T04:55:38.410881+0000 mon.a (mon.0) 2684 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.28", "id": [7, 2]}]: dispatch 2021-02-10T04:55:39.598 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:55:39 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:55:39.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:39 smithi085 conmon[29319]: cluster 2021-02-10T04:55:38.501059+0000 2021-02-10T04:55:39.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:39 smithi085 conmon[29319]: mgr.y (mgr.14140) 1373 : cluster [DBG] pgmap v2257: 201 pgs: 2 clean+premerge+peered, 1 active+clean+scrubbing, 198 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:55:39.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:39 smithi085 conmon[29319]: audit 2021-02-10T04:55:38.591241+0000 mon.a (mon.0) 2685 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 2]}]': finished 2021-02-10T04:55:39.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:39 smithi085 conmon[29319]: audit 2021-02-10T04:55:38.591438+0000 mon.a (mon.0) 2686 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]': finished 2021-02-10T04:55:39.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:39 smithi085 conmon[29319]: audit 2021-02-10T04:55:38.591590+0000 mon.a (mon.0) 2687 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.28", "id": [7, 2]}]': finished 2021-02-10T04:55:39.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:39 smithi085 conmon[29319]: cluster 2021-02-10T04:55:38.591641+0000 mon.a (mon.0) 2688 : cluster [DBG] osdmap e1248: 8 total, 8 up, 8 in 2021-02-10T04:55:39.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:39 smithi085 conmon[35961]: cluster 2021-02-10T04:55:38.501059+0000 2021-02-10T04:55:39.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:39 smithi085 conmon[35961]: mgr.y (mgr.14140) 1373 : cluster [DBG] pgmap v2257: 201 pgs: 2 clean+premerge+peered, 1 active+clean+scrubbing, 198 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:55:39.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:39 smithi085 conmon[35961]: audit 2021-02-10T04:55:38.591241+0000 mon.a (mon.0) 2685 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 2]}]': finished 2021-02-10T04:55:39.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:39 smithi085 conmon[35961]: audit 2021-02-10T04:55:38.591438 2021-02-10T04:55:39.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:39 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2686 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]': finished 2021-02-10T04:55:39.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:39 smithi085 conmon[35961]: audit 2021-02-10T04:55: 2021-02-10T04:55:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:39 smithi085 conmon[35961]: 38.591590+0000 mon.a (mon.0) 2687 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.28", "id": [7, 2]}]': finished 2021-02-10T04:55:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:39 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:55:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:39 smithi085 conmon[35961]: :55:38.591641+0000 mon.a (mon.0) 2688 2021-02-10T04:55:39.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:39 smithi085 conmon[35961]: : cluster [DBG] osdmap e1248: 8 total, 8 up, 8 in 2021-02-10T04:55:39.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:39 smithi164 conmon[29288]: cluster 2021-02-10T04:55:39.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:39 smithi164 conmon[29288]: 2021-02-10T04:55 2021-02-10T04:55:39.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:39 smithi164 conmon[29288]: :38.501059+0000 mgr.y (mgr.14140) 1373 : cluster 2021-02-10T04:55:39.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:39 smithi164 conmon[29288]: [DBG] pgmap v2257: 201 pgs: 2 clean+premerge+peered, 1 active+clean+scrubbing, 198 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:55:39.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:39 smithi164 conmon[29288]: audit 2021 2021-02-10T04:55:39.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:39 smithi164 conmon[29288]: -02-10T04:55:38.591241+0000 mon.a (mon.0) 2685 : audit [INF] 2021-02-10T04:55:39.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:39 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 2]}]': finished 2021-02-10T04:55:39.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:39 smithi164 conmon[29288]: audit 2021-02-10T04:55:39.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:39 smithi164 conmon[29288]: 2021-02-10T04:55:38.591438+0000 mon.a (mon.0) 2686 2021-02-10T04:55:39.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:39 smithi164 conmon[29288]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]': finished 2021-02-10T04:55:39.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:39 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:55:39.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:39 smithi164 conmon[29288]: 02-10T04:55:38.591590+0000 2021-02-10T04:55:39.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:39 smithi164 conmon[29288]: mon.a (mon.0) 2687 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.28", "id": [7, 2]}]': finished 2021-02-10T04:55:39.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:39 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:55:39.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:39 smithi164 conmon[29288]: -02-10T04:55:38.591641+0000 2021-02-10T04:55:39.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:39 smithi164 conmon[29288]: mon.a (mon.0) 2688 : cluster [DBG] osdmap e1248: 8 total, 8 up, 8 in 2021-02-10T04:55:40.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:40 smithi164 conmon[29288]: audit 2021-02-10T04:55:39.325029+0000 mgr.y (mgr. 2021-02-10T04:55:40.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:40 smithi164 conmon[29288]: 14140) 1374 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:55:40.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:40 smithi164 conmon[29288]: cluster 2021-02-10T04:55:39.589467+0000 mon.a (mon.0) 2689 : cluster [DBG] osdmap e1249: 8 total, 8 up, 8 in 2021-02-10T04:55:40.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:40 smithi164 conmon[29288]: audit 2021-02-10T04:55:40.502389 2021-02-10T04:55:40.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:40 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2690 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "23"}]: dispatch 2021-02-10T04:55:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[29319]: audit 2021-02-10T04:55:39.325029+0000 mgr.y (mgr.14140) 1374 2021-02-10T04:55:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[29319]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:55:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[29319]: cluster 2021-02-10T04:55:39.589467+0000 mon.a (mon.0) 2689 : cluster [DBG] osdmap e1249: 8 total, 8 up, 8 in 2021-02-10T04:55:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[29319]: audit 2021-02-10T04:55:40.502389+0000 mon.a 2021-02-10T04:55:40.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[29319]: (mon.0) 2690 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "23"}]: dispatch 2021-02-10T04:55:40.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[29319]: 2021-02-10T04:55:40.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: audit 2021-02-10T04:55:40.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: 2021-02-10T04:55:40.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: 2021 2021-02-10T04:55:40.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: - 2021-02-10T04:55:40.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: 02 2021-02-10T04:55:40.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: - 2021-02-10T04:55:40.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: 10 2021-02-10T04:55:40.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: T 2021-02-10T04:55:40.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: 04 2021-02-10T04:55:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: : 2021-02-10T04:55:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: 55 2021-02-10T04:55:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: : 2021-02-10T04:55:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: 39 2021-02-10T04:55:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: . 2021-02-10T04:55:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: 325029 2021-02-10T04:55:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: +0000 2021-02-10T04:55:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: 2021-02-10T04:55:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: mgr.y 2021-02-10T04:55:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: ( 2021-02-10T04:55:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: mgr 2021-02-10T04:55:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: . 2021-02-10T04:55:40.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: 14140 2021-02-10T04:55:40.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: ) 2021-02-10T04:55:40.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: 1374 2021-02-10T04:55:40.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: : 2021-02-10T04:55:40.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: audit 2021-02-10T04:55:40.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: 2021-02-10T04:55:40.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: [DBG] 2021-02-10T04:55:40.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: 2021-02-10T04:55:40.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:55:40.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: 2021-02-10T04:55:40.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:55:40.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: :55:39.589467+0000 mon.a (mon.0) 2689 : cluster [DBG] osdmap e1249: 8 total, 8 up, 8 in 2021-02-10T04:55:40.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: audit 2021-02-10T04:55: 2021-02-10T04:55:40.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: 40.502389+0000 mon.a (mon.0) 2690 : audit 2021-02-10T04:55:40.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:40 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "23"}]: dispatch 2021-02-10T04:55:41.827 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:55:41.852 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:55:41.871 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:55:41.888 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:55:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:41 smithi164 conmon[29288]: cluster 2021-02-10T04:55:40.501637+0000 mgr.y 2021-02-10T04:55:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:41 smithi164 conmon[29288]: (mgr.14140) 1375 : cluster [DBG] pgmap v2260: 200 pgs: 1 peering, 1 clean+premerge+peered, 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s; 185 B/s, 5 objects/s recovering 2021-02-10T04:55:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:41 smithi164 conmon[29288]: audit 2021-02-10T04:55:40.603455+0000 mon.a (mon.0) 2691 : 2021-02-10T04:55:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:41 smithi164 conmon[29288]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "23"}]': finished 2021-02-10T04:55:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:41 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:55:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:41 smithi164 conmon[29288]: 10T04:55:40.603504+0000 mon.a (mon.0) 2692 : cluster 2021-02-10T04:55:41.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:41 smithi164 conmon[29288]: [DBG] osdmap e1250: 8 total, 8 up, 8 in 2021-02-10T04:55:41.911 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:55:41.926 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:55:41.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:41 smithi085 conmon[29319]: cluster 2021-02-10T04:55:40.501637+0000 mgr.y (mgr.14140) 2021-02-10T04:55:41.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:41 smithi085 conmon[29319]: 1375 : cluster [DBG] pgmap v2260: 200 pgs: 1 peering, 1 clean+premerge+peered, 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s; 185 B/s, 5 objects/s recovering 2021-02-10T04:55:41.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:41 smithi085 conmon[29319]: audit 2021-02-10T04:55:40.603455+0000 mon.a (mon.0) 2691 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "23"}]': finished 2021-02-10T04:55:41.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:41 smithi085 conmon[29319]: cluster 2021-02-10T04:55:40.603504+0000 mon.a (mon.0) 2692 : cluster [DBG] osdmap e1250: 8 total, 8 up, 8 in 2021-02-10T04:55:41.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:41 smithi085 conmon[35961]: cluster 2021-02-10T04:55:40.501637 2021-02-10T04:55:41.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:41 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1375 : cluster [DBG] pgmap v2260: 200 pgs: 1 peering, 1 clean+premerge+peered, 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s; 185 B/s, 5 objects/s recovering 2021-02-10T04:55:41.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:41 smithi085 conmon[35961]: audit 2021-02-10T04:55:41.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:41 smithi085 conmon[35961]: 2021-02-10T04:55:40.603455+0000 mon.a (mon.0) 2691 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "23"}]': finished 2021-02-10T04:55:41.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:41 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:55:41.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:41 smithi085 conmon[35961]: :55:40.603504+0000 mon.a (mon.0) 2692 2021-02-10T04:55:41.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:41 smithi085 conmon[35961]: : cluster [DBG] osdmap e1250: 8 total, 8 up, 8 in 2021-02-10T04:55:41.944 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:55:41.962 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:55:41.963 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:55:41.963 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:55:41.963 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:55:41.963 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:55:41.963 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:55:41.963 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:55:41.963 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:55:41.963 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:55:42.624 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is set 2021-02-10T04:55:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:42 smithi164 conmon[29288]: cluster 2021-02-10T04:55:41.615853+0000 2021-02-10T04:55:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:42 smithi164 conmon[29288]: mon.a (mon.0) 2693 : cluster [DBG] osdmap e1251: 8 total, 8 up, 8 in 2021-02-10T04:55:42.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:42 smithi164 conmon[29288]: audit 2021-02-10T04:55:41.806830+0000 mon.a (mon.0) 2694 : audit [DBG] from='client.? 172.21.15.85:0/3371983656' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:55:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:42 smithi164 conmon[29288]: audit 2021-02-10T04:55:42.003519+0000 mon.a (mon.0) 2695 2021-02-10T04:55:42.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:42 smithi164 conmon[29288]: : audit [INF] from='client.? 172.21.15.85:0/2632671168' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:55:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:42 smithi085 conmon[29319]: cluster 2021-02-10T04:55:41.615853+0000 2021-02-10T04:55:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:42 smithi085 conmon[29319]: mon.a (mon.0) 2693 : cluster [DBG] osdmap e1251: 8 total, 8 up, 8 in 2021-02-10T04:55:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:42 smithi085 conmon[29319]: audit 2021-02-10T04:55:41.806830+0000 mon.a (mon.0) 2694 : audit [DBG] from='client.? 172.21.15.85:0/3371983656' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:55:42.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:42 smithi085 conmon[29319]: audit 2021-02-10T04:55:42.003519+0000 mon.a (mon.0) 2695 : audit [INF] from='client.? 172.21.15.85:0/2632671168' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:55:42.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:42 smithi085 conmon[35961]: cluster 2021-02-10T04:55:42.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:42 smithi085 conmon[35961]: 2021-02-10T04:55:42.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:42 smithi085 conmon[35961]: 2021 2021-02-10T04:55:42.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:42 smithi085 conmon[35961]: - 2021-02-10T04:55:42.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:42 smithi085 conmon[35961]: 02-10T04:55:41.615853+0000 mon.a (mon 2021-02-10T04:55:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:42 smithi085 conmon[35961]: .0) 2693 : cluster [DBG] osdmap e1251: 8 total, 8 up, 8 in 2021-02-10T04:55:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:42 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:55:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:42 smithi085 conmon[35961]: -10T04:55:41.806830+0000 mon.a (mon.0) 2021-02-10T04:55:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:42 smithi085 conmon[35961]: 2694 : audit [DBG] from='client.? 172.21.15.85:0/3371983656' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:55:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:42 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:55:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:42 smithi085 conmon[35961]: 02-10T04:55:42.003519+0000 mon.a (mon. 2021-02-10T04:55:42.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:42 smithi085 conmon[35961]: 0) 2695 : audit [INF] from='client.? 172.21.15.85:0/2632671168' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:55:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:43 smithi164 conmon[29288]: cluster 2021-02-10T04:55:42.502188+0000 mgr.y (mgr.14140) 1376 : cluster [DBG] pgmap v2263: 200 pgs: 1 active+clean+scrubbing, 1 peering, 1 clean+premerge+peered, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:55:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:43 smithi164 conmon[29288]: cluster 2021-02-10T04:55:42.608599+0000 mon.a (mon.0) 2696 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:55:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:43 smithi164 conmon[29288]: audit 2021-02-10T04:55:42.617261+0000 mon.a (mon.0) 2697 : audit [INF] from='client.? 172.21.15.85:0/2632671168' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:55:43.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:43 smithi164 conmon[29288]: cluster 2021-02-10T04:55:42.617315+0000 mon.a (mon.0) 2698 : cluster [DBG] osdmap e1252: 8 total, 8 up, 8 in 2021-02-10T04:55:43.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:43 smithi164 conmon[29288]: cluster 2021-02-10T04:55:43.560661+0000 mon.a (mon.0) 2699 : cluster [DBG] osdmap e1253: 8 total, 8 up, 8 in 2021-02-10T04:55:43.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[29319]: cluster 2021-02-10T04:55 2021-02-10T04:55:43.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[29319]: :42.502188+0000 mgr.y (mgr.14140) 1376 : cluster [DBG] pgmap v2263: 200 pgs: 1 active+clean+scrubbing, 1 peering, 1 clean+premerge+peered, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:55:43.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:55:43.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[29319]: -02-10T04:55:42.608599 2021-02-10T04:55:43.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2021-02-10T04:55:43.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[29319]: 2696 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:55:43.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:55:43.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[29319]: -10T04:55:42.617261+0000 mon.a (mon. 2021-02-10T04:55:43.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[29319]: 0) 2697 : audit [INF] from='client.? 172.21.15.85:0/2632671168' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:55:43.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:55:43.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[29319]: -10T04:55:42.617315+0000 mon.a ( 2021-02-10T04:55:43.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[29319]: mon.0) 2698 : cluster [DBG] osdmap e1252: 8 total, 8 up, 8 in 2021-02-10T04:55:43.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:55:43.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[29319]: -02-10T04:55:43.560661+0000 mon.a 2021-02-10T04:55:43.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[29319]: (mon.0) 2699 : cluster [DBG] osdmap e1253: 8 total, 8 up, 8 in 2021-02-10T04:55:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[35961]: cluster 2021-02-10T04:55:42 2021-02-10T04:55:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[35961]: .502188+0000 mgr.y (mgr.14140) 1376 : cluster [DBG] pgmap v2263: 200 pgs: 1 active+clean+scrubbing, 1 peering, 1 clean+premerge+peered, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:55:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[35961]: cluster 2021-02-10T04:55: 2021-02-10T04:55:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[35961]: 42.608599+0000 mon.a (mon.0) 2021-02-10T04:55:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[35961]: 2696 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:55:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[35961]: audit 2021-02-10T04:55:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[35961]: 2021-02-10T04:55:42 2021-02-10T04:55:43.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[35961]: .617261+0000 mon.a (mon.0) 2697 : audit 2021-02-10T04:55:43.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[35961]: [INF] from='client.? 172.21.15.85:0/2632671168' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:55:43.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:55:43.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[35961]: 04:55:42.617315+0000 mon.a (mon.0 2021-02-10T04:55:43.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[35961]: ) 2698 : cluster [DBG] osdmap e1252: 8 total, 8 up, 8 in 2021-02-10T04:55:43.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[35961]: cluster 2021-02-10T04:55:43.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[35961]: 2021-02-10T04:55:43.560661 2021-02-10T04:55:43.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2699 : cluster [DBG] 2021-02-10T04:55:43.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:43 smithi085 conmon[35961]: osdmap e1253: 8 total, 8 up, 8 in 2021-02-10T04:55:44.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: audit 2021-02-10T04:55:44.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: 2021-02-10T04:55:44.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: 2021 2021-02-10T04:55:44.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: - 2021-02-10T04:55:44.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: 02 2021-02-10T04:55:44.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: - 2021-02-10T04:55:44.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: 10 2021-02-10T04:55:44.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: T 2021-02-10T04:55:44.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: 04 2021-02-10T04:55:44.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: : 2021-02-10T04:55:44.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: 55 2021-02-10T04:55:44.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: : 2021-02-10T04:55:44.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: 44 2021-02-10T04:55:44.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: . 2021-02-10T04:55:44.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: 503603 2021-02-10T04:55:44.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: +0000 2021-02-10T04:55:44.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: 2021-02-10T04:55:44.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: mon.a 2021-02-10T04:55:44.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: ( 2021-02-10T04:55:44.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: mon 2021-02-10T04:55:44.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: . 2021-02-10T04:55:44.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: 0 2021-02-10T04:55:44.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: ) 2021-02-10T04:55:44.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: 2700 2021-02-10T04:55:44.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: : 2021-02-10T04:55:44.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: audit 2021-02-10T04:55:44.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: 2021-02-10T04:55:44.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: [INF] 2021-02-10T04:55:44.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: 2021-02-10T04:55:44.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "22"}]: dispatch 2021-02-10T04:55:44.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:44 smithi164 conmon[29288]: 2021-02-10T04:55:44.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:44 smithi085 conmon[29319]: audit 2021-02-10T04:55:44.503603+0000 mon.a ( 2021-02-10T04:55:44.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:44 smithi085 conmon[29319]: mon.0) 2700 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "22"}]: dispatch 2021-02-10T04:55:44.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:44 smithi085 conmon[35961]: audit 2021-02-10T04:55:44.503603+0000 2021-02-10T04:55:44.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:44 smithi085 conmon[35961]: mon.a (mon.0) 2700 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "22"}]: dispatch 2021-02-10T04:55:45.265 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:55:45.470 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:55:45.497 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset noscrub 2021-02-10T04:55:45.711 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:55:45.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:45 smithi164 conmon[29288]: cluster 2021-02-10T04:55:45.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:45 smithi164 conmon[29288]: 2021-02-10T04:55:45.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:45 smithi164 conmon[29288]: 2021-02-10T04:55:44. 2021-02-10T04:55:45.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:45 smithi164 conmon[29288]: 502658+0000 mgr.y (mgr.14140) 1377 : cluster [DBG] pgmap v2266: 199 pgs: 1 active+clean+scrubbing, 1 peering, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 1.7 KiB/s wr, 25 op/s; 276 B/s, 2 objects/s recovering 2021-02-10T04:55:45.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:45 smithi164 conmon[29288]: audit 2021-02-10T04:55:44 2021-02-10T04:55:45.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:45 smithi164 conmon[29288]: .618930+0000 mon.a (mon.0) 2701 : audit [INF] 2021-02-10T04:55:45.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:45 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "22"}]': finished 2021-02-10T04:55:45.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:45 smithi164 conmon[29288]: cluster 2021-02-10T04:55: 2021-02-10T04:55:45.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:45 smithi164 conmon[29288]: 44.619173+0000 mon.a (mon.0 2021-02-10T04:55:45.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:45 smithi164 conmon[29288]: ) 2702 : cluster [DBG] osdmap e1254: 8 total, 8 up, 8 in 2021-02-10T04:55:45.901 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:45 smithi085 conmon[35961]: cluster 2021-02-10T04:55:44.502658+0000 mgr.y (mgr.14140) 1377 : cluster [DBG] pgmap v2266: 199 pgs: 1 active+clean+scrubbing, 1 peering, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 1.7 KiB/s wr, 25 op/s; 276 B/s, 2 objects/s recovering 2021-02-10T04:55:45.901 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:45 smithi085 conmon[35961]: audit 2021-02-10T04:55:44.618930+0000 mon.a (mon.0) 2701 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "22"}]': finished 2021-02-10T04:55:45.901 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:45 smithi085 conmon[35961]: cluster 2021-02-10T04:55:44.619173+0000 mon.a (mon.0) 2702 : cluster [DBG] osdmap e1254: 8 total, 8 up, 8 in 2021-02-10T04:55:45.902 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:45 smithi085 conmon[29319]: cluster 2021-02-10T04:55:44.502658+0000 mgr.y (mgr.14140) 1377 : cluster [DBG] pgmap v2266: 199 pgs: 1 active+clean+scrubbing, 1 peering, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 1.7 KiB/s wr, 25 op/s; 276 B/s, 2 objects/s recovering 2021-02-10T04:55:45.902 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:45 smithi085 conmon[29319]: audit 2021-02-10T04:55:44.618930+0000 mon.a (mon.0) 2701 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "22"}]': finished 2021-02-10T04:55:45.902 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:45 smithi085 conmon[29319]: cluster 2021-02-10T04:55:44.619173+0000 mon.a (mon.0) 2702 : cluster [DBG] osdmap e1254: 8 total, 8 up, 8 in 2021-02-10T04:55:46.601 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:55:46.601 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:55:46.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:46 smithi164 conmon[29288]: cluster 2021-02-10T04:55:46.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:46 smithi164 conmon[29288]: 2021-02-10T04:55:46.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:46 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:55:46.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:46 smithi164 conmon[29288]: :55:45.627441+0000 mon.a (mon.0) 2703 : cluster [DBG] osdmap e1255: 8 total, 8 up, 8 in 2021-02-10T04:55:46.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:46 smithi085 conmon[29319]: cluster 2021-02-10T04:55:45.627441+0000 mon.a (mon. 2021-02-10T04:55:46.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:46 smithi085 conmon[29319]: 0) 2703 : cluster [DBG] osdmap e1255: 8 total, 8 up, 8 in 2021-02-10T04:55:46.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:46 smithi085 conmon[35961]: cluster 2021-02-10T04:55:45.627441+0000 mon.a 2021-02-10T04:55:46.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:46 smithi085 conmon[35961]: (mon.0) 2703 : cluster [DBG] osdmap e1255: 8 total, 8 up, 8 in 2021-02-10T04:55:47.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:47 smithi164 conmon[29288]: cluster 2021-02-10T04:55:46.503295 2021-02-10T04:55:47.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:47 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1378 : cluster [DBG] pgmap v2269: 199 pgs: 1 active+clean+scrubbing, 1 peering, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 1.7 KiB/s wr, 25 op/s; 276 B/s, 2 objects/s recovering 2021-02-10T04:55:47.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:47 smithi164 conmon[29288]: cluster 2021-02-10T04:55:46.638115+0000 mon.a (mon.0) 2704 : cluster [DBG] osdmap e1256: 8 total, 8 up, 8 in 2021-02-10T04:55:47.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:47 smithi085 conmon[29319]: cluster 2021-02-10T04:55:46.503295+0000 mgr.y (mgr.14140) 2021-02-10T04:55:47.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:47 smithi085 conmon[29319]: 1378 : cluster [DBG] pgmap v2269: 199 pgs: 1 active+clean+scrubbing, 1 peering, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 1.7 KiB/s wr, 25 op/s; 276 B/s, 2 objects/s recovering 2021-02-10T04:55:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:47 smithi085 conmon[29319]: cluster 2021-02-10T04:55:46.638115+0000 mon.a (mon.0) 2704 : cluster [DBG] osdmap e1256: 8 total, 8 up, 8 in 2021-02-10T04:55:47.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:47 smithi085 conmon[35961]: cluster 2021-02-10T04:55:46.503295+0000 2021-02-10T04:55:47.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:47 smithi085 conmon[35961]: mgr.y (mgr.14140) 1378 : cluster [DBG] pgmap v2269: 199 pgs: 1 active+clean+scrubbing, 1 peering, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 1.7 KiB/s wr, 25 op/s; 276 B/s, 2 objects/s recovering 2021-02-10T04:55:47.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:47 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:55:47.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:47 smithi085 conmon[35961]: -02-10T04:55:46.638115+0000 mon.a (mon.0) 2704 : cluster 2021-02-10T04:55:47.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:47 smithi085 conmon[35961]: [DBG] osdmap e1256: 8 total, 8 up, 8 in 2021-02-10T04:55:48.604 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:55:48.621 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:55:48.645 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:55:48.649 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[29319]: cluster 2021-02-10T04:55:48.671 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:55:48.694 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:55:48.712 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:55:48.729 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:55:48.754 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:55:48.755 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:55:48.755 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:55:48.755 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:55:48.755 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:55:48.755 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:55:48.755 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:55:48.755 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:55:48.755 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:55:48.937 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[29319]: 2021-02-10T04:55:48.937 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[29319]: 2021 2021-02-10T04:55:48.938 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[29319]: - 2021-02-10T04:55:48.938 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[29319]: 02 2021-02-10T04:55:48.938 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[29319]: -10T04:55:47.645484+0000 mon.a (mon. 2021-02-10T04:55:48.938 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[29319]: 0) 2705 : cluster [DBG] osdmap e1257: 8 total, 8 up, 8 in 2021-02-10T04:55:48.938 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[29319]: audit 2021 2021-02-10T04:55:48.938 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[29319]: -02-10T04:55:48.567684+0000 mon.a 2021-02-10T04:55:48.938 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[29319]: (mon.0) 2706 : audit [DBG] from='client.? 172.21.15.85:0/3381607770' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:55:48.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: cluster 2021-02-10T04:55:48.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: 2021-02-10T04:55:48.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: 2021 2021-02-10T04:55:48.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: - 2021-02-10T04:55:48.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: 02 2021-02-10T04:55:48.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: - 2021-02-10T04:55:48.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: 10 2021-02-10T04:55:48.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: T 2021-02-10T04:55:48.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: 04 2021-02-10T04:55:48.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: : 2021-02-10T04:55:48.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: 55 2021-02-10T04:55:48.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: : 2021-02-10T04:55:48.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: 47 2021-02-10T04:55:48.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: . 2021-02-10T04:55:48.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: 645484 2021-02-10T04:55:48.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: +0000 2021-02-10T04:55:48.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: 2021-02-10T04:55:48.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: mon.a 2021-02-10T04:55:48.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: ( 2021-02-10T04:55:48.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: mon 2021-02-10T04:55:48.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: . 2021-02-10T04:55:48.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: 0 2021-02-10T04:55:48.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: ) 2021-02-10T04:55:48.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: 2705 2021-02-10T04:55:48.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: : 2021-02-10T04:55:48.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: cluster 2021-02-10T04:55:48.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: 2021-02-10T04:55:48.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: [DBG] 2021-02-10T04:55:48.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: 2021-02-10T04:55:48.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: osdmap e1257: 8 total, 8 up, 8 in 2021-02-10T04:55:48.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: 2021-02-10T04:55:48.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: audit 2021-02-10T04:55:48.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: 2021-02-10T04:55:48.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: 2021 2021-02-10T04:55:48.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: - 2021-02-10T04:55:48.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: 02 2021-02-10T04:55:48.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: - 2021-02-10T04:55:48.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: 10 2021-02-10T04:55:48.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: T 2021-02-10T04:55:48.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: 04 2021-02-10T04:55:48.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: : 2021-02-10T04:55:48.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: 55 2021-02-10T04:55:48.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: : 2021-02-10T04:55:48.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: 48 2021-02-10T04:55:48.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: . 2021-02-10T04:55:48.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: 567684 2021-02-10T04:55:48.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: +0000 2021-02-10T04:55:48.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: 2021-02-10T04:55:48.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: mon.a 2021-02-10T04:55:48.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: ( 2021-02-10T04:55:48.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: mon 2021-02-10T04:55:48.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: . 2021-02-10T04:55:48.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: 0 2021-02-10T04:55:48.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: ) 2021-02-10T04:55:48.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: 2706 2021-02-10T04:55:48.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: : 2021-02-10T04:55:48.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: audit 2021-02-10T04:55:48.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: 2021-02-10T04:55:48.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: [DBG] 2021-02-10T04:55:48.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: 2021-02-10T04:55:48.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/3381607770' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:55:48.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:48 smithi085 conmon[35961]: 2021-02-10T04:55:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:48 smithi164 conmon[29288]: cluster 2021-02-10T04:55:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:48 smithi164 conmon[29288]: 2021-02-10T04:55:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:48 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:55:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:48 smithi164 conmon[29288]: 55:47.645484+0000 mon.a (mon.0) 2705 : cluster [DBG] osdmap e1257: 8 total, 8 up, 8 in 2021-02-10T04:55:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:48 smithi164 conmon[29288]: 2021-02-10T04:55:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:48 smithi164 conmon[29288]: audit 2021-02-10T04:55:48.567684 2021-02-10T04:55:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:48 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2706 : audit [DBG] from='client.? 172.21.15.85:0/3381607770' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:55:49.644 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:55:49 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:55:49.661 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is unset 2021-02-10T04:55:49.915 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:49 smithi164 conmon[29288]: cluster 2021-02-10T04:55:48.500063+0000 osd.7 (osd.7 2021-02-10T04:55:49.915 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:49 smithi164 conmon[29288]: ) 92 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:55:49.915 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:49 smithi164 conmon[29288]: cluster 2021-02-10T04:55:48.501623+0000 osd.7 (osd.7) 93 : 2021-02-10T04:55:49.915 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:49 smithi164 conmon[29288]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:55:49.915 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:49 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:55:49.915 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:49 smithi164 conmon[29288]: -10T04:55:48.503969 2021-02-10T04:55:49.916 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:49 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1379 2021-02-10T04:55:49.916 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:49 smithi164 conmon[29288]: : cluster [DBG] pgmap v2272: 198 pgs: 1 active+clean+scrubbing, 1 peering, 196 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:55:49.916 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:49 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:55:49.916 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:49 smithi164 conmon[29288]: 02-10T04:55:48. 2021-02-10T04:55:49.916 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:49 smithi164 conmon[29288]: 654013+0000 mon.a (mon.0) 2021-02-10T04:55:49.916 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:49 smithi164 conmon[29288]: 2707 : cluster [DBG] osdmap e1258: 8 total, 8 up, 8 in 2021-02-10T04:55:49.916 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:49 smithi164 conmon[29288]: audit 2021 2021-02-10T04:55:49.916 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:49 smithi164 conmon[29288]: -02-10T04:55:48 2021-02-10T04:55:49.916 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:49 smithi164 conmon[29288]: .776497+0000 mon.c (mon.1 2021-02-10T04:55:49.916 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:49 smithi164 conmon[29288]: ) 127 : audit [INF] from='client.? 172.21.15.85:0/2680956972' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:55:49.916 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:49 smithi164 conmon[29288]: 2021-02-10T04:55:49.916 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:49 smithi164 conmon[29288]: audit 2021-02-10T04:55:49.917 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:49 smithi164 conmon[29288]: 2021-02-10T04:55 2021-02-10T04:55:49.917 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:49 smithi164 conmon[29288]: :48.776856+0000 mon.a ( 2021-02-10T04:55:49.917 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:49 smithi164 conmon[29288]: mon.0) 2708 : audit [INF] 2021-02-10T04:55:49.917 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:49 smithi164 conmon[29288]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:55:49.917 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.6 dump_ops_in_flight 2021-02-10T04:55:49.941 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[49106]: debug 2021-02-10T04:55:49.773+0000 7f778ce21700 -1 osd.2 pg_epoch: 1259 pg[5.8( empty local-lis/les=1227/1228 n=0 ec=57/57 lis/c=1227/1227 les/c/f=1228/1228/0 sis=1227) [2,0,3] r=0 lpr=1227 crt=0'0 mlcod 0'0 active+clean] scrubber pg(5.8) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x564aa6ed7340) 2021-02-10T04:55:49.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[29319]: cluster 2021-02-10T04:55:48.500063+0000 osd.7 ( 2021-02-10T04:55:49.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[29319]: osd.7) 92 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:55:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[29319]: cluster 2021-02-10T04:55:48.501623+0000 osd.7 (osd.7) 2021-02-10T04:55:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[29319]: 93 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:55:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[29319]: cluster 2021-02-10T04:55:48. 2021-02-10T04:55:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[29319]: 503969+0000 mgr.y (mgr.14140) 1379 : cluster 2021-02-10T04:55:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[29319]: [DBG] pgmap v2272: 198 pgs: 1 active+clean+scrubbing, 1 peering, 196 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:55:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[29319]: cluster 2021-02-10T04:55:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[29319]: 2021-02-10T04:55:48.654013+0000 mon.a (mon. 2021-02-10T04:55:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[29319]: 0) 2707 : cluster 2021-02-10T04:55:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[29319]: [DBG] osdmap e1258: 8 total, 8 up, 8 in 2021-02-10T04:55:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[29319]: 2021-02-10T04:55:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[29319]: audit 2021-02-10T04: 2021-02-10T04:55:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[29319]: 55:48.776497+0000 mon.c (mon.1 2021-02-10T04:55:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[29319]: ) 127 : audit [INF] from='client.? 172.21.15.85:0/2680956972' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:55:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[29319]: audit 2021-02-10T04:55:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[29319]: 2021-02-10T04:55:48.776856 2021-02-10T04:55:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2708 : audit 2021-02-10T04:55:49.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[29319]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:55:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[35961]: cluster 2021-02-10T04:55:48.500063+0000 2021-02-10T04:55:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[35961]: osd.7 (osd.7) 92 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:55:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:55:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[35961]: T04:55:48.501623+0000 osd.7 (osd.7) 93 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:55:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[35961]: cluster 2021-02-10T04:55:48.503969+0000 2021-02-10T04:55:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[35961]: mgr.y (mgr.14140) 1379 : cluster [DBG] 2021-02-10T04:55:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[35961]: pgmap v2272: 198 pgs: 1 active+clean+scrubbing, 1 peering, 196 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:55:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:55:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[35961]: 55:48.654013+0000 mon.a (mon.0 2021-02-10T04:55:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[35961]: ) 2707 : cluster [DBG] osdmap e1258: 8 total, 8 up, 8 in 2021-02-10T04:55:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[35961]: audit 2021-02-10T04:55:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:55:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[35961]: 55:48.776497+0000 mon.c (mon.1 2021-02-10T04:55:49.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[35961]: ) 127 : audit [INF] from='client.? 172.21.15.85:0/2680956972' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:55:49.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[35961]: audit 2021-02-10T04:55:49.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[35961]: 2021-02-10T04:55:48.776856 2021-02-10T04:55:49.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2708 : audit 2021-02-10T04:55:49.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:49 smithi085 conmon[35961]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:55:50.198 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:55:50.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:50 smithi085 conmon[29319]: audit 2021-02-10T04:55:49.334977+0000 mgr.y (mgr 2021-02-10T04:55:50.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:50 smithi085 conmon[29319]: .14140) 1380 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:55:50.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:50 smithi085 conmon[29319]: cluster 2021-02-10T04:55:49.648820+0000 mon.a (mon.0) 2709 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:55:50.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:50 smithi085 conmon[29319]: audit 2021-02-10T04:55:49.655525+0000 mon.a 2021-02-10T04:55:50.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:50 smithi085 conmon[29319]: (mon.0) 2710 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:55:50.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:50 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:55:50.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:50 smithi085 conmon[29319]: 04:55:49.655596+0000 mon.a (mon.0) 2021-02-10T04:55:50.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:50 smithi085 conmon[29319]: 2711 : cluster [DBG] osdmap e1259: 8 total, 8 up, 8 in 2021-02-10T04:55:50.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:50 smithi085 conmon[35961]: audit 2021-02-10T04:55:49 2021-02-10T04:55:50.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:50 smithi085 conmon[35961]: .334977+0000 mgr.y (mgr.14140) 1380 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:55:50.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:50 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:55:50.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:50 smithi085 conmon[35961]: -10T04:55:49. 2021-02-10T04:55:50.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:50 smithi085 conmon[35961]: 648820+0000 mon.a (mon.0 2021-02-10T04:55:50.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:50 smithi085 conmon[35961]: ) 2709 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:55:50.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:50 smithi085 conmon[35961]: 2021-02-10T04:55:50.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:50 smithi085 conmon[35961]: audit 2021-02-10T04:55:50.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:50 smithi085 conmon[35961]: 2021- 2021-02-10T04:55:50.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:50 smithi085 conmon[35961]: 02-10T04:55: 2021-02-10T04:55:50.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:50 smithi085 conmon[35961]: 49.655525+0000 mon.a (mon. 2021-02-10T04:55:50.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:50 smithi085 conmon[35961]: 0) 2710 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:55:50.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:50 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:55:50.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:50 smithi085 conmon[35961]: 02-10T04:55:49.655596+0000 2021-02-10T04:55:50.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:50 smithi085 conmon[35961]: mon.a (mon.0) 2711 : cluster 2021-02-10T04:55:50.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:50 smithi085 conmon[35961]: [DBG] osdmap e1259: 8 total, 8 up, 8 in 2021-02-10T04:55:50.814 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2021-02-10T04:55:50.815 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:55:50.815 INFO:tasks.thrashosds.thrasher:Shrinking pool device_health_metrics 2021-02-10T04:55:50.815 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool device_health_metrics by 10 min 8 2021-02-10T04:55:50.815 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:55:50.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:50 smithi164 conmon[29288]: audit 2021-02-10T04:55:50.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:50 smithi164 conmon[29288]: 2021-02-10T04:55:50.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:50 smithi164 conmon[29288]: 2021 2021-02-10T04:55:50.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:50 smithi164 conmon[29288]: - 2021-02-10T04:55:50.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:50 smithi164 conmon[29288]: 02-10T04:55:49.334977+0000 2021-02-10T04:55:50.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:50 smithi164 conmon[29288]: mgr.y (mgr.14140) 1380 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:55:50.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:50 smithi164 conmon[29288]: cluster 2021-02-10T04:55:49.648820+0000 2021-02-10T04:55:50.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:50 smithi164 conmon[29288]: mon.a (mon.0) 2709 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:55:50.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:50 smithi164 conmon[29288]: audit 2021-02-10T04:55:50.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:50 smithi164 conmon[29288]: 2021-02-10T04:55:49. 2021-02-10T04:55:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:50 smithi164 conmon[29288]: 655525+0000 mon.a (mon.0) 2710 : audit [INF] 2021-02-10T04:55:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:50 smithi164 conmon[29288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:55:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:50 smithi164 conmon[29288]: cluster 2021-02-10T04:55:49.655596 2021-02-10T04:55:50.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:50 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2711 : cluster [DBG] osdmap e1259: 8 total, 8 up, 8 in 2021-02-10T04:55:51.071 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:55:51.645 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:55:51.661 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:51 smithi085 conmon[29319]: cluster 2021-02-10T04:55:50.504455+0000 mgr.y (mgr.14140) 1381 : cluster [DBG] pgmap v2275: 198 pgs: 1 active+clean+scrubbing, 1 peering, 196 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:55:51.661 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:51 smithi085 conmon[35961]: cluster 2021-02-10T04:55:50.504455+0000 mgr.y (mgr.14140) 1381 : cluster [DBG] pgmap v2275: 198 pgs: 1 active+clean+scrubbing, 1 peering, 196 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:55:51.887 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:55:52.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:51 smithi164 conmon[29288]: cluster 2021-02-10T04:55:52.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:51 smithi164 conmon[29288]: 2021- 2021-02-10T04:55:52.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:51 smithi164 conmon[29288]: 02-10T04:55:50.504455+0000 mgr.y (mgr.14140) 1381 : cluster [DBG] pgmap v2275: 198 pgs: 1 active+clean+scrubbing, 1 peering, 196 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:55:52.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:51 smithi164 conmon[29288]: 2021-02-10T04:55:52.450 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset nodeep-scrub 2021-02-10T04:55:52.653 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:55:53.809 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:55:53.813 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:55:53.827 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:53 smithi085 conmon[35961]: cluster 2021-02-10T04:55:52. 2021-02-10T04:55:53.827 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:53 smithi085 conmon[35961]: 505121+0000 mgr.y (mgr.14140) 1382 : cluster [DBG] pgmap v2276: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 1.4 KiB/s wr, 21 op/s 2021-02-10T04:55:53.828 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:53 smithi085 conmon[29319]: cluster 2021-02-10T04:55:52.505121+0000 mgr.y (mgr. 2021-02-10T04:55:53.828 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:53 smithi085 conmon[29319]: 14140) 1382 : cluster [DBG] pgmap v2276: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 1.4 KiB/s wr, 21 op/s 2021-02-10T04:55:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:53 smithi164 conmon[29288]: cluster 2021-02-10T04:55:52.505121+0000 mgr.y (mgr.14140) 1382 : cluster [DBG] 2021-02-10T04:55:54.484 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:53 smithi164 conmon[29288]: pgmap v2276: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 1.4 KiB/s wr, 21 op/s 2021-02-10T04:55:54.671 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:54 smithi085 conmon[29319]: audit 2021-02-10T04:55:53.787580+0000 mgr.y (mgr. 2021-02-10T04:55:54.671 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:54 smithi085 conmon[29319]: 14140) 1383 : audit [DBG] from='client.16107 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:55:54.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:54 smithi085 conmon[35961]: audit 2021-02-10T04:55:54.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:54 smithi085 conmon[35961]: 2021-02-10T04:55:53.787580 2021-02-10T04:55:54.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:54 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1383 : audit [DBG] from='client.16107 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:55:55.140 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:55:55.140 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2021-02-10T04:55:55.140 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 0 2021-02-10T04:55:55.140 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:55:55.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:54 smithi164 conmon[29288]: audit 2021-02-10T04:55:53.787580+0000 mgr.y (mgr.14140) 1383 : audit [DBG] from='client.16107 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:55:55.546 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":2276,"stamp":"2021-02-10T04:55:52.504626+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":972357089,"num_objects":748,"num_object_clones":337,"num_object_copies":2244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":748,"num_whiteouts":10,"num_read":7416,"num_read_kb":4561815,"num_write":4555,"num_write_kb":1247750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":887,"num_bytes_recovered":311980434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35364,"ondisk_log_size":35364,"up":594,"acting":594,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":597,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2298756,"kb_used_data":1982212,"kb_used_omap":692,"kb_used_meta":315787,"kb_avail":747662460,"statfs":{"total":767960285184,"available":765606359040,"internally_reserved":0,"allocated":2029785088,"data_stored":2598914233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":709615,"internal_metadata":323365905},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1206,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":84,"num_read_kb":62,"num_write":41,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.868893"},"pg_stats":[{"pgid":"6.29","version":"0'0","reported_seq":"1115","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:49.359498+0000","last_change":"2021-02-10T04:55:14.633086+0000","last_active":"2021-02-10T04:55:49.359498+0000","last_peered":"2021-02-10T04:55:49.359498+0000","last_clean":"2021-02-10T04:55:49.359498+0000","last_became_active":"2021-02-10T04:55:14.632670+0000","last_became_peered":"2021-02-10T04:55:14.632670+0000","last_unstale":"2021-02-10T04:55:49.359498+0000","last_undegraded":"2021-02-10T04:55:49.359498+0000","last_fullsized":"2021-02-10T04:55:49.359498+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:24.121684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:24.121684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":"155","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.651246+0000","last_change":"2021-02-10T04:52:30.133105+0000","last_active":"2021-02-10T04:55:48.651246+0000","last_peered":"2021-02-10T04:55:48.651246+0000","last_clean":"2021-02-10T04:55:48.651246+0000","last_became_active":"2021-02-10T04:51:36.864926+0000","last_became_peered":"2021-02-10T04:51:36.864926+0000","last_unstale":"2021-02-10T04:55:48.651246+0000","last_undegraded":"2021-02-10T04:55:48.651246+0000","last_fullsized":"2021-02-10T04:55:48.651246+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.133043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.133043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"1191'4","reported_seq":"234","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:49.359865+0000","last_change":"2021-02-10T04:53:46.736164+0000","last_active":"2021-02-10T04:55:49.359865+0000","last_peered":"2021-02-10T04:55:49.359865+0000","last_clean":"2021-02-10T04:55:49.359865+0000","last_became_active":"2021-02-10T04:53:46.735711+0000","last_became_peered":"2021-02-10T04:53:46.735711+0000","last_unstale":"2021-02-10T04:55:49.359865+0000","last_undegraded":"2021-02-10T04:55:49.359865+0000","last_fullsized":"2021-02-10T04:55:49.359865+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'3","last_scrub_stamp":"2021-02-10T04:51:01.185820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.185820+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"1132'66","reported_seq":"1362","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.639518+0000","last_change":"2021-02-10T04:55:39.594594+0000","last_active":"2021-02-10T04:55:46.639518+0000","last_peered":"2021-02-10T04:55:46.639518+0000","last_clean":"2021-02-10T04:55:46.639518+0000","last_became_active":"2021-02-10T04:55:39.594275+0000","last_became_peered":"2021-02-10T04:55:39.594275+0000","last_unstale":"2021-02-10T04:55:46.639518+0000","last_undegraded":"2021-02-10T04:55:46.639518+0000","last_fullsized":"2021-02-10T04:55:46.639518+0000","mapping_epoch":55,"log_start":"1132'66","ondisk_log_start":"1132'66","created":55,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":0,"last_scrub":"1132'66","last_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_deep_scrub":"1132'66","last_deep_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.430399+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1616,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":159,"num_read_kb":123,"num_write":89,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"164","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.654270+0000","last_change":"2021-02-10T04:53:41.722609+0000","last_active":"2021-02-10T04:55:48.654270+0000","last_peered":"2021-02-10T04:55:48.654270+0000","last_clean":"2021-02-10T04:55:48.654270+0000","last_became_active":"2021-02-10T04:51:36.875000+0000","last_became_peered":"2021-02-10T04:51:36.875000+0000","last_unstale":"2021-02-10T04:55:48.654270+0000","last_undegraded":"2021-02-10T04:55:48.654270+0000","last_fullsized":"2021-02-10T04:55:48.654270+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:53:41.722516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:41.722516+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"1251","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.640126+0000","last_change":"2021-02-10T04:55:33.326794+0000","last_active":"2021-02-10T04:55:46.640126+0000","last_peered":"2021-02-10T04:55:46.640126+0000","last_clean":"2021-02-10T04:55:46.640126+0000","last_became_active":"2021-02-10T04:55:14.630535+0000","last_became_peered":"2021-02-10T04:55:14.630535+0000","last_unstale":"2021-02-10T04:55:46.640126+0000","last_undegraded":"2021-02-10T04:55:46.640126+0000","last_fullsized":"2021-02-10T04:55:46.640126+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:48.408748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:50:48.408748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"1003'1689","reported_seq":"4676","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.646814+0000","last_change":"2021-02-10T04:53:38.170873+0000","last_active":"2021-02-10T04:55:46.646814+0000","last_peered":"2021-02-10T04:55:46.646814+0000","last_clean":"2021-02-10T04:55:46.646814+0000","last_became_active":"2021-02-10T04:52:26.030105+0000","last_became_peered":"2021-02-10T04:52:26.030105+0000","last_unstale":"2021-02-10T04:55:46.646814+0000","last_undegraded":"2021-02-10T04:55:46.646814+0000","last_fullsized":"2021-02-10T04:55:46.646814+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:53:38.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.170802+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":99361981,"num_objects":37,"num_object_clones":33,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"62'5","reported_seq":"129","reported_epoch":"1257","state":"active+clean","last_fresh":"2021-02-10T04:55:47.648033+0000","last_change":"2021-02-10T04:54:39.615784+0000","last_active":"2021-02-10T04:55:47.648033+0000","last_peered":"2021-02-10T04:55:47.648033+0000","last_clean":"2021-02-10T04:55:47.648033+0000","last_became_active":"2021-02-10T04:54:39.560009+0000","last_became_peered":"2021-02-10T04:54:39.560009+0000","last_unstale":"2021-02-10T04:55:47.648033+0000","last_undegraded":"2021-02-10T04:55:47.648033+0000","last_fullsized":"2021-02-10T04:55:47.648033+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":219,"num_read_kb":218,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"1061'20","reported_seq":"1256","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.653155+0000","last_change":"2021-02-10T04:55:21.669679+0000","last_active":"2021-02-10T04:55:48.653155+0000","last_peered":"2021-02-10T04:55:48.653155+0000","last_clean":"2021-02-10T04:55:48.653155+0000","last_became_active":"2021-02-10T04:55:21.669219+0000","last_became_peered":"2021-02-10T04:55:21.669219+0000","last_unstale":"2021-02-10T04:55:48.653155+0000","last_undegraded":"2021-02-10T04:55:48.653155+0000","last_fullsized":"2021-02-10T04:55:48.653155+0000","mapping_epoch":1228,"log_start":"1061'20","ondisk_log_start":"1061'20","created":55,"last_epoch_clean":1233,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'32","last_scrub_stamp":"2021-02-10T04:51:51.382040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.382040+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1158,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":93,"num_read_kb":69,"num_write":50,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2052,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"908","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.648248+0000","last_change":"2021-02-10T04:53:34.179015+0000","last_active":"2021-02-10T04:55:46.648248+0000","last_peered":"2021-02-10T04:55:46.648248+0000","last_clean":"2021-02-10T04:55:46.648248+0000","last_became_active":"2021-02-10T04:51:49.502576+0000","last_became_peered":"2021-02-10T04:51:49.502576+0000","last_unstale":"2021-02-10T04:55:46.648248+0000","last_undegraded":"2021-02-10T04:55:46.648248+0000","last_fullsized":"2021-02-10T04:55:46.648248+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:34.178949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.178949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"155","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.655036+0000","last_change":"2021-02-10T04:51:53.038893+0000","last_active":"2021-02-10T04:55:48.655036+0000","last_peered":"2021-02-10T04:55:48.655036+0000","last_clean":"2021-02-10T04:55:48.655036+0000","last_became_active":"2021-02-10T04:51:36.855507+0000","last_became_peered":"2021-02-10T04:51:36.855507+0000","last_unstale":"2021-02-10T04:55:48.655036+0000","last_undegraded":"2021-02-10T04:55:48.655036+0000","last_fullsized":"2021-02-10T04:55:48.655036+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:53.038834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:53.038834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"1177","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.651058+0000","last_change":"2021-02-10T04:55:28.097406+0000","last_active":"2021-02-10T04:55:48.651058+0000","last_peered":"2021-02-10T04:55:48.651058+0000","last_clean":"2021-02-10T04:55:48.651058+0000","last_became_active":"2021-02-10T04:55:14.629603+0000","last_became_peered":"2021-02-10T04:55:14.629603+0000","last_unstale":"2021-02-10T04:55:48.651058+0000","last_undegraded":"2021-02-10T04:55:48.651058+0000","last_fullsized":"2021-02-10T04:55:48.651058+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:09.922789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:09.922789+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"1168'82","reported_seq":"191","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.651900+0000","last_change":"2021-02-10T04:53:23.326363+0000","last_active":"2021-02-10T04:55:48.651900+0000","last_peered":"2021-02-10T04:55:48.651900+0000","last_clean":"2021-02-10T04:55:48.651900+0000","last_became_active":"2021-02-10T04:51:36.859215+0000","last_became_peered":"2021-02-10T04:51:36.859215+0000","last_unstale":"2021-02-10T04:55:48.651900+0000","last_undegraded":"2021-02-10T04:55:48.651900+0000","last_fullsized":"2021-02-10T04:55:48.651900+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1150'76","last_scrub_stamp":"2021-02-10T04:53:00.964499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:00.964499+0000","log_size":82,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3848803,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"1112'47","reported_seq":"1364","reported_epoch":"1257","state":"active+clean","last_fresh":"2021-02-10T04:55:47.646079+0000","last_change":"2021-02-10T04:55:25.576927+0000","last_active":"2021-02-10T04:55:47.646079+0000","last_peered":"2021-02-10T04:55:47.646079+0000","last_clean":"2021-02-10T04:55:47.646079+0000","last_became_active":"2021-02-10T04:55:25.576633+0000","last_became_peered":"2021-02-10T04:55:25.576633+0000","last_unstale":"2021-02-10T04:55:47.646079+0000","last_undegraded":"2021-02-10T04:55:47.646079+0000","last_fullsized":"2021-02-10T04:55:47.646079+0000","mapping_epoch":55,"log_start":"1112'47","ondisk_log_start":"1112'47","created":55,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'61","last_scrub_stamp":"2021-02-10T04:53:36.836202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:36.836202+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1466,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":175,"num_read_kb":139,"num_write":100,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"1228'6","reported_seq":"227","reported_epoch":"1259","state":"active+clean","last_fresh":"2021-02-10T04:55:49.655661+0000","last_change":"2021-02-10T04:55:14.605868+0000","last_active":"2021-02-10T04:55:49.655661+0000","last_peered":"2021-02-10T04:55:49.655661+0000","last_clean":"2021-02-10T04:55:49.655661+0000","last_became_active":"2021-02-10T04:55:14.605314+0000","last_became_peered":"2021-02-10T04:55:14.605314+0000","last_unstale":"2021-02-10T04:55:49.655661+0000","last_undegraded":"2021-02-10T04:55:49.655661+0000","last_fullsized":"2021-02-10T04:55:49.655661+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'4","last_scrub_stamp":"2021-02-10T04:52:01.121380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.121380+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"170","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.652291+0000","last_change":"2021-02-10T04:55:22.573715+0000","last_active":"2021-02-10T04:55:48.652291+0000","last_peered":"2021-02-10T04:55:48.652291+0000","last_clean":"2021-02-10T04:55:48.652291+0000","last_became_active":"2021-02-10T04:55:14.636895+0000","last_became_peered":"2021-02-10T04:55:14.636895+0000","last_unstale":"2021-02-10T04:55:48.652291+0000","last_undegraded":"2021-02-10T04:55:48.652291+0000","last_fullsized":"2021-02-10T04:55:48.652291+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:55:22.573621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:55:22.573621+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"555","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:49.359484+0000","last_change":"2021-02-10T04:53:46.733236+0000","last_active":"2021-02-10T04:55:49.359484+0000","last_peered":"2021-02-10T04:55:49.359484+0000","last_clean":"2021-02-10T04:55:49.359484+0000","last_became_active":"2021-02-10T04:53:46.732890+0000","last_became_peered":"2021-02-10T04:53:46.732890+0000","last_unstale":"2021-02-10T04:55:49.359484+0000","last_undegraded":"2021-02-10T04:55:49.359484+0000","last_fullsized":"2021-02-10T04:55:49.359484+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:35.170157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:35.170157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"1003'739","reported_seq":"3856","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.651292+0000","last_change":"2021-02-10T04:55:28.091164+0000","last_active":"2021-02-10T04:55:48.651292+0000","last_peered":"2021-02-10T04:55:48.651292+0000","last_clean":"2021-02-10T04:55:48.651292+0000","last_became_active":"2021-02-10T04:55:14.648601+0000","last_became_peered":"2021-02-10T04:55:14.648601+0000","last_unstale":"2021-02-10T04:55:48.651292+0000","last_undegraded":"2021-02-10T04:55:48.651292+0000","last_fullsized":"2021-02-10T04:55:48.651292+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:52:27.602006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:27.602006+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24059888,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":129,"num_read_kb":109544,"num_write":102,"num_write_kb":46173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"1234","reported_epoch":"1257","state":"active+clean","last_fresh":"2021-02-10T04:55:47.646309+0000","last_change":"2021-02-10T04:54:39.559980+0000","last_active":"2021-02-10T04:55:47.646309+0000","last_peered":"2021-02-10T04:55:47.646309+0000","last_clean":"2021-02-10T04:55:47.646309+0000","last_became_active":"2021-02-10T04:54:39.559443+0000","last_became_peered":"2021-02-10T04:54:39.559443+0000","last_unstale":"2021-02-10T04:55:47.646309+0000","last_undegraded":"2021-02-10T04:55:47.646309+0000","last_fullsized":"2021-02-10T04:55:47.646309+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"1066'44","reported_seq":"1413","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.916251+0000","last_change":"2021-02-10T04:53:37.108747+0000","last_active":"2021-02-10T04:55:48.916251+0000","last_peered":"2021-02-10T04:55:48.916251+0000","last_clean":"2021-02-10T04:55:48.916251+0000","last_became_active":"2021-02-10T04:53:35.551311+0000","last_became_peered":"2021-02-10T04:53:35.551311+0000","last_unstale":"2021-02-10T04:55:48.916251+0000","last_undegraded":"2021-02-10T04:55:48.916251+0000","last_fullsized":"2021-02-10T04:55:48.916251+0000","mapping_epoch":1118,"log_start":"1066'44","ondisk_log_start":"1066'44","created":55,"last_epoch_clean":1181,"parent":"0.0","parent_split_bits":0,"last_scrub":"1066'44","last_scrub_stamp":"2021-02-10T04:53:37.108677+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:53:37.108677+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1578,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":150,"num_read_kb":114,"num_write":86,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":2312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"1153","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.915030+0000","last_change":"2021-02-10T04:53:39.126356+0000","last_active":"2021-02-10T04:55:48.915030+0000","last_peered":"2021-02-10T04:55:48.915030+0000","last_clean":"2021-02-10T04:55:48.915030+0000","last_became_active":"2021-02-10T04:29:07.145616+0000","last_became_peered":"2021-02-10T04:29:07.145616+0000","last_unstale":"2021-02-10T04:55:48.915030+0000","last_undegraded":"2021-02-10T04:55:48.915030+0000","last_fullsized":"2021-02-10T04:55:48.915030+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:39.126261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:39.126261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"483","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.647845+0000","last_change":"2021-02-10T04:55:14.611192+0000","last_active":"2021-02-10T04:55:46.647845+0000","last_peered":"2021-02-10T04:55:46.647845+0000","last_clean":"2021-02-10T04:55:46.647845+0000","last_became_active":"2021-02-10T04:55:14.610821+0000","last_became_peered":"2021-02-10T04:55:14.610821+0000","last_unstale":"2021-02-10T04:55:46.647845+0000","last_undegraded":"2021-02-10T04:55:46.647845+0000","last_fullsized":"2021-02-10T04:55:46.647845+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:26.311958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.311958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"187","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.655402+0000","last_change":"2021-02-10T04:55:03.526061+0000","last_active":"2021-02-10T04:55:48.655402+0000","last_peered":"2021-02-10T04:55:48.655402+0000","last_clean":"2021-02-10T04:55:48.655402+0000","last_became_active":"2021-02-10T04:51:36.855744+0000","last_became_peered":"2021-02-10T04:51:36.855744+0000","last_unstale":"2021-02-10T04:55:48.655402+0000","last_undegraded":"2021-02-10T04:55:48.655402+0000","last_fullsized":"2021-02-10T04:55:48.655402+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:02.162851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:13.727241+0000","last_clean_scrub_stamp":"2021-02-10T04:52:02.162851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"1063'654","reported_seq":"4434","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.654183+0000","last_change":"2021-02-10T04:55:14.637619+0000","last_active":"2021-02-10T04:55:48.654183+0000","last_peered":"2021-02-10T04:55:48.654183+0000","last_clean":"2021-02-10T04:55:48.654183+0000","last_became_active":"2021-02-10T04:55:14.636752+0000","last_became_peered":"2021-02-10T04:55:14.636752+0000","last_unstale":"2021-02-10T04:55:48.654183+0000","last_undegraded":"2021-02-10T04:55:48.654183+0000","last_fullsized":"2021-02-10T04:55:48.654183+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:51:57.121838+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.121838+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24700630,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":171,"num_read_kb":182272,"num_write":111,"num_write_kb":33727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":26349023,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"1202","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.639599+0000","last_change":"2021-02-10T04:51:50.433056+0000","last_active":"2021-02-10T04:55:46.639599+0000","last_peered":"2021-02-10T04:55:46.639599+0000","last_clean":"2021-02-10T04:55:46.639599+0000","last_became_active":"2021-02-10T04:51:36.843165+0000","last_became_peered":"2021-02-10T04:51:36.843165+0000","last_unstale":"2021-02-10T04:55:46.639599+0000","last_undegraded":"2021-02-10T04:55:46.639599+0000","last_fullsized":"2021-02-10T04:55:46.639599+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.432990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.432990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"1222","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.630366+0000","last_change":"2021-02-10T04:55:14.638290+0000","last_active":"2021-02-10T04:55:45.630366+0000","last_peered":"2021-02-10T04:55:45.630366+0000","last_clean":"2021-02-10T04:55:45.630366+0000","last_became_active":"2021-02-10T04:55:14.637927+0000","last_became_peered":"2021-02-10T04:55:14.637927+0000","last_unstale":"2021-02-10T04:55:45.630366+0000","last_undegraded":"2021-02-10T04:55:45.630366+0000","last_fullsized":"2021-02-10T04:55:45.630366+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"1150","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:49.359581+0000","last_change":"2021-02-10T04:53:46.727888+0000","last_active":"2021-02-10T04:55:49.359581+0000","last_peered":"2021-02-10T04:55:49.359581+0000","last_clean":"2021-02-10T04:55:49.359581+0000","last_became_active":"2021-02-10T04:53:46.727670+0000","last_became_peered":"2021-02-10T04:53:46.727670+0000","last_unstale":"2021-02-10T04:55:49.359581+0000","last_undegraded":"2021-02-10T04:55:49.359581+0000","last_fullsized":"2021-02-10T04:55:49.359581+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:41.041553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.041553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"1187'54","reported_seq":"1334","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.915047+0000","last_change":"2021-02-10T04:53:42.071173+0000","last_active":"2021-02-10T04:55:48.915047+0000","last_peered":"2021-02-10T04:55:48.915047+0000","last_clean":"2021-02-10T04:55:48.915047+0000","last_became_active":"2021-02-10T04:53:39.679752+0000","last_became_peered":"2021-02-10T04:53:39.679752+0000","last_unstale":"2021-02-10T04:55:48.915047+0000","last_undegraded":"2021-02-10T04:55:48.915047+0000","last_fullsized":"2021-02-10T04:55:48.915047+0000","mapping_epoch":1111,"log_start":"1049'39","ondisk_log_start":"1049'39","created":55,"last_epoch_clean":1185,"parent":"0.0","parent_split_bits":0,"last_scrub":"1187'54","last_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_deep_scrub":"1187'54","last_deep_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_clean_scrub_stamp":"2021-02-10T04:53:42.071103+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":114,"num_read_kb":94,"num_write":73,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"893","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.639768+0000","last_change":"2021-02-10T04:51:52.350764+0000","last_active":"2021-02-10T04:55:46.639768+0000","last_peered":"2021-02-10T04:55:46.639768+0000","last_clean":"2021-02-10T04:55:46.639768+0000","last_became_active":"2021-02-10T04:51:49.567784+0000","last_became_peered":"2021-02-10T04:51:49.567784+0000","last_unstale":"2021-02-10T04:55:46.639768+0000","last_undegraded":"2021-02-10T04:55:46.639768+0000","last_fullsized":"2021-02-10T04:55:46.639768+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:52.350706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.350706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"1119'4","reported_seq":"211","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.653540+0000","last_change":"2021-02-10T04:52:27.126670+0000","last_active":"2021-02-10T04:55:48.653540+0000","last_peered":"2021-02-10T04:55:48.653540+0000","last_clean":"2021-02-10T04:55:48.653540+0000","last_became_active":"2021-02-10T04:51:36.853647+0000","last_became_peered":"2021-02-10T04:51:36.853647+0000","last_unstale":"2021-02-10T04:55:48.653540+0000","last_undegraded":"2021-02-10T04:55:48.653540+0000","last_fullsized":"2021-02-10T04:55:48.653540+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'4","last_scrub_stamp":"2021-02-10T04:52:27.126599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:27.126599+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"1244","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.915995+0000","last_change":"2021-02-10T04:52:25.745660+0000","last_active":"2021-02-10T04:55:48.915995+0000","last_peered":"2021-02-10T04:55:48.915995+0000","last_clean":"2021-02-10T04:55:48.915995+0000","last_became_active":"2021-02-10T04:51:36.849503+0000","last_became_peered":"2021-02-10T04:51:36.849503+0000","last_unstale":"2021-02-10T04:55:48.915995+0000","last_undegraded":"2021-02-10T04:55:48.915995+0000","last_fullsized":"2021-02-10T04:55:48.915995+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:25.745577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.745577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"1004'1093","reported_seq":"6439","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.653700+0000","last_change":"2021-02-10T04:55:28.097530+0000","last_active":"2021-02-10T04:55:48.653700+0000","last_peered":"2021-02-10T04:55:48.653700+0000","last_clean":"2021-02-10T04:55:48.653700+0000","last_became_active":"2021-02-10T04:55:14.627630+0000","last_became_peered":"2021-02-10T04:55:14.627630+0000","last_unstale":"2021-02-10T04:55:48.653700+0000","last_undegraded":"2021-02-10T04:55:48.653700+0000","last_fullsized":"2021-02-10T04:55:48.653700+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39434696,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":279,"num_read_kb":311058,"num_write":185,"num_write_kb":66790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"62'5","reported_seq":"54","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.627109+0000","last_change":"2021-02-10T04:54:39.559473+0000","last_active":"2021-02-10T04:55:45.627109+0000","last_peered":"2021-02-10T04:55:45.627109+0000","last_clean":"2021-02-10T04:55:45.627109+0000","last_became_active":"2021-02-10T04:54:39.559048+0000","last_became_peered":"2021-02-10T04:54:39.559048+0000","last_unstale":"2021-02-10T04:55:45.627109+0000","last_undegraded":"2021-02-10T04:55:45.627109+0000","last_fullsized":"2021-02-10T04:55:45.627109+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":149,"num_read_kb":148,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"1211","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.639738+0000","last_change":"2021-02-10T04:55:03.230261+0000","last_active":"2021-02-10T04:55:46.639738+0000","last_peered":"2021-02-10T04:55:46.639738+0000","last_clean":"2021-02-10T04:55:46.639738+0000","last_became_active":"2021-02-10T04:51:36.841582+0000","last_became_peered":"2021-02-10T04:51:36.841582+0000","last_unstale":"2021-02-10T04:55:46.639738+0000","last_undegraded":"2021-02-10T04:55:46.639738+0000","last_fullsized":"2021-02-10T04:55:46.639738+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:12.389272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:12.389272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"1049'51","reported_seq":"929","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.647496+0000","last_change":"2021-02-10T04:53:27.560299+0000","last_active":"2021-02-10T04:55:46.647496+0000","last_peered":"2021-02-10T04:55:46.647496+0000","last_clean":"2021-02-10T04:55:46.647496+0000","last_became_active":"2021-02-10T04:53:27.559886+0000","last_became_peered":"2021-02-10T04:53:27.559886+0000","last_unstale":"2021-02-10T04:55:46.647496+0000","last_undegraded":"2021-02-10T04:55:46.647496+0000","last_fullsized":"2021-02-10T04:55:46.647496+0000","mapping_epoch":1129,"log_start":"1049'51","ondisk_log_start":"1049'51","created":55,"last_epoch_clean":1173,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'41","last_scrub_stamp":"2021-02-10T04:51:58.353660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:58.353660+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1472,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":139,"num_read_kb":106,"num_write":82,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"916","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.639556+0000","last_change":"2021-02-10T04:53:11.271557+0000","last_active":"2021-02-10T04:55:46.639556+0000","last_peered":"2021-02-10T04:55:46.639556+0000","last_clean":"2021-02-10T04:55:46.639556+0000","last_became_active":"2021-02-10T04:51:49.567883+0000","last_became_peered":"2021-02-10T04:51:49.567883+0000","last_unstale":"2021-02-10T04:55:46.639556+0000","last_undegraded":"2021-02-10T04:55:46.639556+0000","last_fullsized":"2021-02-10T04:55:46.639556+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:11.271459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.271459+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"1229'5","reported_seq":"1607","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.915943+0000","last_change":"2021-02-10T04:55:14.638050+0000","last_active":"2021-02-10T04:55:48.915943+0000","last_peered":"2021-02-10T04:55:48.915943+0000","last_clean":"2021-02-10T04:55:48.915943+0000","last_became_active":"2021-02-10T04:55:14.637570+0000","last_became_peered":"2021-02-10T04:55:14.637570+0000","last_unstale":"2021-02-10T04:55:48.915943+0000","last_undegraded":"2021-02-10T04:55:48.915943+0000","last_fullsized":"2021-02-10T04:55:48.915943+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:51:47.855283+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.855283+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"1246","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.641559+0000","last_change":"2021-02-10T04:55:22.183630+0000","last_active":"2021-02-10T04:55:46.641559+0000","last_peered":"2021-02-10T04:55:46.641559+0000","last_clean":"2021-02-10T04:55:46.641559+0000","last_became_active":"2021-02-10T04:55:14.656661+0000","last_became_peered":"2021-02-10T04:55:14.656661+0000","last_unstale":"2021-02-10T04:55:46.641559+0000","last_undegraded":"2021-02-10T04:55:46.641559+0000","last_fullsized":"2021-02-10T04:55:46.641559+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:55:22.183566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:55:22.183566+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"1002'1090","reported_seq":"5451","reported_epoch":"1257","state":"active+clean","last_fresh":"2021-02-10T04:55:47.645023+0000","last_change":"2021-02-10T04:52:26.030198+0000","last_active":"2021-02-10T04:55:47.645023+0000","last_peered":"2021-02-10T04:55:47.645023+0000","last_clean":"2021-02-10T04:55:47.645023+0000","last_became_active":"2021-02-10T04:52:26.028182+0000","last_became_peered":"2021-02-10T04:52:26.028182+0000","last_unstale":"2021-02-10T04:55:47.645023+0000","last_undegraded":"2021-02-10T04:55:47.645023+0000","last_fullsized":"2021-02-10T04:55:47.645023+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:51:08.220472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.220472+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24410646,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":245,"num_read_kb":276168,"num_write":175,"num_write_kb":64960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"1224","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.914424+0000","last_change":"2021-02-10T04:54:39.555379+0000","last_active":"2021-02-10T04:55:48.914424+0000","last_peered":"2021-02-10T04:55:48.914424+0000","last_clean":"2021-02-10T04:55:48.914424+0000","last_became_active":"2021-02-10T04:54:39.554976+0000","last_became_peered":"2021-02-10T04:54:39.554976+0000","last_unstale":"2021-02-10T04:55:48.914424+0000","last_undegraded":"2021-02-10T04:55:48.914424+0000","last_fullsized":"2021-02-10T04:55:48.914424+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"1049'47","reported_seq":"1329","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.915846+0000","last_change":"2021-02-10T04:53:31.624862+0000","last_active":"2021-02-10T04:55:48.915846+0000","last_peered":"2021-02-10T04:55:48.915846+0000","last_clean":"2021-02-10T04:55:48.915846+0000","last_became_active":"2021-02-10T04:53:31.624318+0000","last_became_peered":"2021-02-10T04:53:31.624318+0000","last_unstale":"2021-02-10T04:55:48.915846+0000","last_undegraded":"2021-02-10T04:55:48.915846+0000","last_fullsized":"2021-02-10T04:55:48.915846+0000","mapping_epoch":55,"log_start":"1049'47","ondisk_log_start":"1049'47","created":55,"last_epoch_clean":1177,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'47","last_scrub_stamp":"2021-02-10T04:53:09.002368+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.002368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1672,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":142,"num_read_kb":108,"num_write":80,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":934,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"195","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:49.359572+0000","last_change":"2021-02-10T04:55:14.611324+0000","last_active":"2021-02-10T04:55:49.359572+0000","last_peered":"2021-02-10T04:55:49.359572+0000","last_clean":"2021-02-10T04:55:49.359572+0000","last_became_active":"2021-02-10T04:55:14.610986+0000","last_became_peered":"2021-02-10T04:55:14.610986+0000","last_unstale":"2021-02-10T04:55:49.359572+0000","last_undegraded":"2021-02-10T04:55:49.359572+0000","last_fullsized":"2021-02-10T04:55:49.359572+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:08.358300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:50:08.358300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"1233","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.640779+0000","last_change":"2021-02-10T04:51:57.436961+0000","last_active":"2021-02-10T04:55:46.640779+0000","last_peered":"2021-02-10T04:55:46.640779+0000","last_clean":"2021-02-10T04:55:46.640779+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:55:46.640779+0000","last_undegraded":"2021-02-10T04:55:46.640779+0000","last_fullsized":"2021-02-10T04:55:46.640779+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.436862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.436862+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"1187","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.628228+0000","last_change":"2021-02-10T04:53:02.112612+0000","last_active":"2021-02-10T04:55:45.628228+0000","last_peered":"2021-02-10T04:55:45.628228+0000","last_clean":"2021-02-10T04:55:45.628228+0000","last_became_active":"2021-02-10T04:50:59.545501+0000","last_became_peered":"2021-02-10T04:50:59.545501+0000","last_unstale":"2021-02-10T04:55:45.628228+0000","last_undegraded":"2021-02-10T04:55:45.628228+0000","last_fullsized":"2021-02-10T04:55:45.628228+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:02.112533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:02.112533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"1001'1165","reported_seq":"4430","reported_epoch":"1257","state":"active+clean","last_fresh":"2021-02-10T04:55:47.645233+0000","last_change":"2021-02-10T04:55:14.653651+0000","last_active":"2021-02-10T04:55:47.645233+0000","last_peered":"2021-02-10T04:55:47.645233+0000","last_clean":"2021-02-10T04:55:47.645233+0000","last_became_active":"2021-02-10T04:55:14.652235+0000","last_became_peered":"2021-02-10T04:55:14.652235+0000","last_unstale":"2021-02-10T04:55:47.645233+0000","last_undegraded":"2021-02-10T04:55:47.645233+0000","last_fullsized":"2021-02-10T04:55:47.645233+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29601845,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":286,"num_read_kb":299529,"num_write":200,"num_write_kb":89130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"40","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.651551+0000","last_change":"2021-02-10T04:55:15.590113+0000","last_active":"2021-02-10T04:55:48.651551+0000","last_peered":"2021-02-10T04:55:48.651551+0000","last_clean":"2021-02-10T04:55:48.651551+0000","last_became_active":"2021-02-10T04:55:15.589751+0000","last_became_peered":"2021-02-10T04:55:15.589751+0000","last_unstale":"2021-02-10T04:55:48.651551+0000","last_undegraded":"2021-02-10T04:55:48.651551+0000","last_fullsized":"2021-02-10T04:55:48.651551+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"1113'5","reported_seq":"1489","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.627445+0000","last_change":"2021-02-10T04:52:34.047474+0000","last_active":"2021-02-10T04:55:45.627445+0000","last_peered":"2021-02-10T04:55:45.627445+0000","last_clean":"2021-02-10T04:55:45.627445+0000","last_became_active":"2021-02-10T04:50:59.569578+0000","last_became_peered":"2021-02-10T04:50:59.569578+0000","last_unstale":"2021-02-10T04:55:45.627445+0000","last_undegraded":"2021-02-10T04:55:45.627445+0000","last_fullsized":"2021-02-10T04:55:45.627445+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1113'5","last_scrub_stamp":"2021-02-10T04:52:34.047358+0000","last_deep_scrub":"158'3","last_deep_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_clean_scrub_stamp":"2021-02-10T04:52:34.047358+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"1049'54","reported_seq":"1320","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.914813+0000","last_change":"2021-02-10T04:51:48.872937+0000","last_active":"2021-02-10T04:55:48.914813+0000","last_peered":"2021-02-10T04:55:48.914813+0000","last_clean":"2021-02-10T04:55:48.914813+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:55:48.914813+0000","last_undegraded":"2021-02-10T04:55:48.914813+0000","last_fullsized":"2021-02-10T04:55:48.914813+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:51:48.872839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.872839+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":93,"num_read_kb":73,"num_write":54,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"1308","reported_epoch":"1257","state":"active+clean","last_fresh":"2021-02-10T04:55:47.645562+0000","last_change":"2021-02-10T04:53:05.984747+0000","last_active":"2021-02-10T04:55:47.645562+0000","last_peered":"2021-02-10T04:55:47.645562+0000","last_clean":"2021-02-10T04:55:47.645562+0000","last_became_active":"2021-02-10T04:51:36.869931+0000","last_became_peered":"2021-02-10T04:51:36.869931+0000","last_unstale":"2021-02-10T04:55:47.645562+0000","last_undegraded":"2021-02-10T04:55:47.645562+0000","last_fullsized":"2021-02-10T04:55:47.645562+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:05.984659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:05.984659+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"1112","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:49.359704+0000","last_change":"2021-02-10T04:53:46.731709+0000","last_active":"2021-02-10T04:55:49.359704+0000","last_peered":"2021-02-10T04:55:49.359704+0000","last_clean":"2021-02-10T04:55:49.359704+0000","last_became_active":"2021-02-10T04:53:46.731340+0000","last_became_peered":"2021-02-10T04:53:46.731340+0000","last_unstale":"2021-02-10T04:55:49.359704+0000","last_undegraded":"2021-02-10T04:55:49.359704+0000","last_fullsized":"2021-02-10T04:55:49.359704+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:09.005421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.005421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"998'1009","reported_seq":"4746","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.627983+0000","last_change":"2021-02-10T04:52:26.037227+0000","last_active":"2021-02-10T04:55:45.627983+0000","last_peered":"2021-02-10T04:55:45.627983+0000","last_clean":"2021-02-10T04:55:45.627983+0000","last_became_active":"2021-02-10T04:52:26.036723+0000","last_became_peered":"2021-02-10T04:52:26.036723+0000","last_unstale":"2021-02-10T04:55:45.627983+0000","last_undegraded":"2021-02-10T04:55:45.627983+0000","last_fullsized":"2021-02-10T04:55:45.627983+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12355206,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":119,"num_read_kb":124701,"num_write":77,"num_write_kb":25052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4145029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"1126","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.628697+0000","last_change":"2021-02-10T04:53:33.305896+0000","last_active":"2021-02-10T04:55:45.628697+0000","last_peered":"2021-02-10T04:55:45.628697+0000","last_clean":"2021-02-10T04:55:45.628697+0000","last_became_active":"2021-02-10T04:50:59.570854+0000","last_became_peered":"2021-02-10T04:50:59.570854+0000","last_unstale":"2021-02-10T04:55:45.628697+0000","last_undegraded":"2021-02-10T04:55:45.628697+0000","last_fullsized":"2021-02-10T04:55:45.628697+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:33.305827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:33.305827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"1227","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.628353+0000","last_change":"2021-02-10T04:55:15.669988+0000","last_active":"2021-02-10T04:55:45.628353+0000","last_peered":"2021-02-10T04:55:45.628353+0000","last_clean":"2021-02-10T04:55:45.628353+0000","last_became_active":"2021-02-10T04:55:15.669642+0000","last_became_peered":"2021-02-10T04:55:15.669642+0000","last_unstale":"2021-02-10T04:55:45.628353+0000","last_undegraded":"2021-02-10T04:55:45.628353+0000","last_fullsized":"2021-02-10T04:55:45.628353+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"1243","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.914979+0000","last_change":"2021-02-10T04:51:50.895471+0000","last_active":"2021-02-10T04:55:48.914979+0000","last_peered":"2021-02-10T04:55:48.914979+0000","last_clean":"2021-02-10T04:55:48.914979+0000","last_became_active":"2021-02-10T04:50:59.527553+0000","last_became_peered":"2021-02-10T04:50:59.527553+0000","last_unstale":"2021-02-10T04:55:48.914979+0000","last_undegraded":"2021-02-10T04:55:48.914979+0000","last_fullsized":"2021-02-10T04:55:48.914979+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.895370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.895370+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"1049'72","reported_seq":"1353","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.627942+0000","last_change":"2021-02-10T04:55:14.603109+0000","last_active":"2021-02-10T04:55:45.627942+0000","last_peered":"2021-02-10T04:55:45.627942+0000","last_clean":"2021-02-10T04:55:45.627942+0000","last_became_active":"2021-02-10T04:55:14.602669+0000","last_became_peered":"2021-02-10T04:55:14.602669+0000","last_unstale":"2021-02-10T04:55:45.627942+0000","last_undegraded":"2021-02-10T04:55:45.627942+0000","last_fullsized":"2021-02-10T04:55:45.627942+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'72","last_scrub_stamp":"2021-02-10T04:46:30.515026+0000","last_deep_scrub":"152'48","last_deep_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_clean_scrub_stamp":"2021-02-10T04:46:30.515026+0000","log_size":72,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":105,"num_read_kb":83,"num_write":60,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"510","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.647563+0000","last_change":"2021-02-10T04:55:03.834050+0000","last_active":"2021-02-10T04:55:46.647563+0000","last_peered":"2021-02-10T04:55:46.647563+0000","last_clean":"2021-02-10T04:55:46.647563+0000","last_became_active":"2021-02-10T04:51:49.503801+0000","last_became_peered":"2021-02-10T04:51:49.503801+0000","last_unstale":"2021-02-10T04:55:46.647563+0000","last_undegraded":"2021-02-10T04:55:46.647563+0000","last_fullsized":"2021-02-10T04:55:46.647563+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:51:48.274960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.274960+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"1268","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.915061+0000","last_change":"2021-02-10T04:53:13.099288+0000","last_active":"2021-02-10T04:55:48.915061+0000","last_peered":"2021-02-10T04:55:48.915061+0000","last_clean":"2021-02-10T04:55:48.915061+0000","last_became_active":"2021-02-10T04:51:36.848614+0000","last_became_peered":"2021-02-10T04:51:36.848614+0000","last_unstale":"2021-02-10T04:55:48.915061+0000","last_undegraded":"2021-02-10T04:55:48.915061+0000","last_fullsized":"2021-02-10T04:55:48.915061+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:13.099228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:13.099228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"1002'1657","reported_seq":"5316","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.629857+0000","last_change":"2021-02-10T04:52:26.036806+0000","last_active":"2021-02-10T04:55:45.629857+0000","last_peered":"2021-02-10T04:55:45.629857+0000","last_clean":"2021-02-10T04:55:45.629857+0000","last_became_active":"2021-02-10T04:52:26.032363+0000","last_became_peered":"2021-02-10T04:52:26.032363+0000","last_unstale":"2021-02-10T04:55:45.629857+0000","last_undegraded":"2021-02-10T04:55:45.629857+0000","last_fullsized":"2021-02-10T04:55:45.629857+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21396533,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":212,"num_read_kb":218836,"num_write":159,"num_write_kb":59398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":22326014,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"1187","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.915208+0000","last_change":"2021-02-10T04:51:51.911945+0000","last_active":"2021-02-10T04:55:48.915208+0000","last_peered":"2021-02-10T04:55:48.915208+0000","last_clean":"2021-02-10T04:55:48.915208+0000","last_became_active":"2021-02-10T04:51:36.850015+0000","last_became_peered":"2021-02-10T04:51:36.850015+0000","last_unstale":"2021-02-10T04:55:48.915208+0000","last_undegraded":"2021-02-10T04:55:48.915208+0000","last_fullsized":"2021-02-10T04:55:48.915208+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.911851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.911851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"62'5","reported_seq":"1089","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.647172+0000","last_change":"2021-02-10T04:54:39.568253+0000","last_active":"2021-02-10T04:55:46.647172+0000","last_peered":"2021-02-10T04:55:46.647172+0000","last_clean":"2021-02-10T04:55:46.647172+0000","last_became_active":"2021-02-10T04:54:39.567642+0000","last_became_peered":"2021-02-10T04:54:39.567642+0000","last_unstale":"2021-02-10T04:55:46.647172+0000","last_undegraded":"2021-02-10T04:55:46.647172+0000","last_fullsized":"2021-02-10T04:55:46.647172+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":98,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":150,"num_read_kb":148,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"1315","reported_epoch":"1257","state":"active+clean","last_fresh":"2021-02-10T04:55:47.646211+0000","last_change":"2021-02-10T04:55:14.634577+0000","last_active":"2021-02-10T04:55:47.646211+0000","last_peered":"2021-02-10T04:55:47.646211+0000","last_clean":"2021-02-10T04:55:47.646211+0000","last_became_active":"2021-02-10T04:55:14.634259+0000","last_became_peered":"2021-02-10T04:55:14.634259+0000","last_unstale":"2021-02-10T04:55:47.646211+0000","last_undegraded":"2021-02-10T04:55:47.646211+0000","last_fullsized":"2021-02-10T04:55:47.646211+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:58.142526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:58.142526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"1135'58","reported_seq":"1252","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.652876+0000","last_change":"2021-02-10T04:55:28.079376+0000","last_active":"2021-02-10T04:55:48.652876+0000","last_peered":"2021-02-10T04:55:48.652876+0000","last_clean":"2021-02-10T04:55:48.652876+0000","last_became_active":"2021-02-10T04:55:14.654609+0000","last_became_peered":"2021-02-10T04:55:14.654609+0000","last_unstale":"2021-02-10T04:55:48.652876+0000","last_undegraded":"2021-02-10T04:55:48.652876+0000","last_fullsized":"2021-02-10T04:55:48.652876+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1135'58","last_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_deep_scrub":"1135'58","last_deep_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.483067+0000","log_size":58,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":87,"num_read_kb":68,"num_write":50,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"193","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:49.360354+0000","last_change":"2021-02-10T04:55:14.657913+0000","last_active":"2021-02-10T04:55:49.360354+0000","last_peered":"2021-02-10T04:55:49.360354+0000","last_clean":"2021-02-10T04:55:49.360354+0000","last_became_active":"2021-02-10T04:55:14.657153+0000","last_became_peered":"2021-02-10T04:55:14.657153+0000","last_unstale":"2021-02-10T04:55:49.360354+0000","last_undegraded":"2021-02-10T04:55:49.360354+0000","last_fullsized":"2021-02-10T04:55:49.360354+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:50:46.434629+0000","last_deep_scrub":"53'2","last_deep_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_clean_scrub_stamp":"2021-02-10T04:50:46.434629+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"933","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.639670+0000","last_change":"2021-02-10T04:52:38.278980+0000","last_active":"2021-02-10T04:55:46.639670+0000","last_peered":"2021-02-10T04:55:46.639670+0000","last_clean":"2021-02-10T04:55:46.639670+0000","last_became_active":"2021-02-10T04:51:49.496237+0000","last_became_peered":"2021-02-10T04:51:49.496237+0000","last_unstale":"2021-02-10T04:55:46.639670+0000","last_undegraded":"2021-02-10T04:55:46.639670+0000","last_fullsized":"2021-02-10T04:55:46.639670+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:38.278910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:52:38.278910+0000","last_clean_scrub_stamp":"2021-02-10T04:52:38.278910+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"1000'1071","reported_seq":"4807","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.639951+0000","last_change":"2021-02-10T04:53:34.407875+0000","last_active":"2021-02-10T04:55:46.639951+0000","last_peered":"2021-02-10T04:55:46.639951+0000","last_clean":"2021-02-10T04:55:46.639951+0000","last_became_active":"2021-02-10T04:52:26.031050+0000","last_became_peered":"2021-02-10T04:52:26.031050+0000","last_unstale":"2021-02-10T04:55:46.639951+0000","last_undegraded":"2021-02-10T04:55:46.639951+0000","last_fullsized":"2021-02-10T04:55:46.639951+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:34.407784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.407784+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30173843,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"1281","reported_epoch":"1257","state":"active+clean","last_fresh":"2021-02-10T04:55:47.645523+0000","last_change":"2021-02-10T04:55:14.637570+0000","last_active":"2021-02-10T04:55:47.645523+0000","last_peered":"2021-02-10T04:55:47.645523+0000","last_clean":"2021-02-10T04:55:47.645523+0000","last_became_active":"2021-02-10T04:55:14.637203+0000","last_became_peered":"2021-02-10T04:55:14.637203+0000","last_unstale":"2021-02-10T04:55:47.645523+0000","last_undegraded":"2021-02-10T04:55:47.645523+0000","last_fullsized":"2021-02-10T04:55:47.645523+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_clean_scrub_stamp":"2021-02-10T04:51:00.195518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"1243","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.640854+0000","last_change":"2021-02-10T04:55:14.661285+0000","last_active":"2021-02-10T04:55:46.640854+0000","last_peered":"2021-02-10T04:55:46.640854+0000","last_clean":"2021-02-10T04:55:46.640854+0000","last_became_active":"2021-02-10T04:55:14.660902+0000","last_became_peered":"2021-02-10T04:55:14.660902+0000","last_unstale":"2021-02-10T04:55:46.640854+0000","last_undegraded":"2021-02-10T04:55:46.640854+0000","last_fullsized":"2021-02-10T04:55:46.640854+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:55.473347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:50:55.473347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"1049'40","reported_seq":"1291","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.652494+0000","last_change":"2021-02-10T04:55:47.652017+0000","last_active":"2021-02-10T04:55:48.652494+0000","last_peered":"2021-02-10T04:55:48.652494+0000","last_clean":"2021-02-10T04:55:48.652494+0000","last_became_active":"2021-02-10T04:55:47.651592+0000","last_became_peered":"2021-02-10T04:55:47.651592+0000","last_unstale":"2021-02-10T04:55:48.652494+0000","last_undegraded":"2021-02-10T04:55:48.652494+0000","last_fullsized":"2021-02-10T04:55:48.652494+0000","mapping_epoch":1227,"log_start":"1049'40","ondisk_log_start":"1049'40","created":55,"last_epoch_clean":1257,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'40","last_scrub_stamp":"2021-02-10T04:51:47.281929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.281929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1836,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":144,"num_read_kb":108,"num_write":81,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"419","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:49.359666+0000","last_change":"2021-02-10T04:55:14.610765+0000","last_active":"2021-02-10T04:55:49.359666+0000","last_peered":"2021-02-10T04:55:49.359666+0000","last_clean":"2021-02-10T04:55:49.359666+0000","last_became_active":"2021-02-10T04:55:14.610470+0000","last_became_peered":"2021-02-10T04:55:14.610470+0000","last_unstale":"2021-02-10T04:55:49.359666+0000","last_undegraded":"2021-02-10T04:55:49.359666+0000","last_fullsized":"2021-02-10T04:55:49.359666+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:50.399350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:50.399350+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"1145","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.650851+0000","last_change":"2021-02-10T04:55:25.624031+0000","last_active":"2021-02-10T04:55:48.650851+0000","last_peered":"2021-02-10T04:55:48.650851+0000","last_clean":"2021-02-10T04:55:48.650851+0000","last_became_active":"2021-02-10T04:55:14.630129+0000","last_became_peered":"2021-02-10T04:55:14.630129+0000","last_unstale":"2021-02-10T04:55:48.650851+0000","last_undegraded":"2021-02-10T04:55:48.650851+0000","last_fullsized":"2021-02-10T04:55:48.650851+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"918","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.647931+0000","last_change":"2021-02-10T04:52:31.273037+0000","last_active":"2021-02-10T04:55:46.647931+0000","last_peered":"2021-02-10T04:55:46.647931+0000","last_clean":"2021-02-10T04:55:46.647931+0000","last_became_active":"2021-02-10T04:51:49.501405+0000","last_became_peered":"2021-02-10T04:51:49.501405+0000","last_unstale":"2021-02-10T04:55:46.647931+0000","last_undegraded":"2021-02-10T04:55:46.647931+0000","last_fullsized":"2021-02-10T04:55:46.647931+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:31.272971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:17.598113+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.272971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"1000'446","reported_seq":"1976","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:49.360170+0000","last_change":"2021-02-10T04:55:14.658783+0000","last_active":"2021-02-10T04:55:49.360170+0000","last_peered":"2021-02-10T04:55:49.360170+0000","last_clean":"2021-02-10T04:55:49.360170+0000","last_became_active":"2021-02-10T04:55:14.657668+0000","last_became_peered":"2021-02-10T04:55:14.657668+0000","last_unstale":"2021-02-10T04:55:49.360170+0000","last_undegraded":"2021-02-10T04:55:49.360170+0000","last_fullsized":"2021-02-10T04:55:49.360170+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:51:45.964029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.964029+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22255852,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68435,"num_write":72,"num_write_kb":24345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":403693,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"409","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.646904+0000","last_change":"2021-02-10T04:53:03.228260+0000","last_active":"2021-02-10T04:55:46.646904+0000","last_peered":"2021-02-10T04:55:46.646904+0000","last_clean":"2021-02-10T04:55:46.646904+0000","last_became_active":"2021-02-10T04:51:49.499252+0000","last_became_peered":"2021-02-10T04:51:49.499252+0000","last_unstale":"2021-02-10T04:55:46.646904+0000","last_undegraded":"2021-02-10T04:55:46.646904+0000","last_fullsized":"2021-02-10T04:55:46.646904+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:03.228182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.228182+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"163","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:49.360249+0000","last_change":"2021-02-10T04:53:46.727384+0000","last_active":"2021-02-10T04:55:49.360249+0000","last_peered":"2021-02-10T04:55:49.360249+0000","last_clean":"2021-02-10T04:55:49.360249+0000","last_became_active":"2021-02-10T04:53:46.726771+0000","last_became_peered":"2021-02-10T04:53:46.726771+0000","last_unstale":"2021-02-10T04:55:49.360249+0000","last_undegraded":"2021-02-10T04:55:49.360249+0000","last_fullsized":"2021-02-10T04:55:49.360249+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:33.948348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:33.948348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"1049'39","reported_seq":"1325","reported_epoch":"1257","state":"active+clean","last_fresh":"2021-02-10T04:55:47.645122+0000","last_change":"2021-02-10T04:54:17.118810+0000","last_active":"2021-02-10T04:55:47.645122+0000","last_peered":"2021-02-10T04:55:47.645122+0000","last_clean":"2021-02-10T04:55:47.645122+0000","last_became_active":"2021-02-10T04:54:17.080053+0000","last_became_peered":"2021-02-10T04:54:17.080053+0000","last_unstale":"2021-02-10T04:55:47.645122+0000","last_undegraded":"2021-02-10T04:55:47.645122+0000","last_fullsized":"2021-02-10T04:55:47.645122+0000","mapping_epoch":1200,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1201,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'39","last_scrub_stamp":"2021-02-10T04:51:47.322331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.322331+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":51,"num_write":39,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"191","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:49.359762+0000","last_change":"2021-02-10T04:55:14.636888+0000","last_active":"2021-02-10T04:55:49.359762+0000","last_peered":"2021-02-10T04:55:49.359762+0000","last_clean":"2021-02-10T04:55:49.359762+0000","last_became_active":"2021-02-10T04:55:14.634305+0000","last_became_peered":"2021-02-10T04:55:14.634305+0000","last_unstale":"2021-02-10T04:55:49.359762+0000","last_undegraded":"2021-02-10T04:55:49.359762+0000","last_fullsized":"2021-02-10T04:55:49.359762+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_clean_scrub_stamp":"2021-02-10T04:50:05.424262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"1427","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.640294+0000","last_change":"2021-02-10T04:52:37.324122+0000","last_active":"2021-02-10T04:55:46.640294+0000","last_peered":"2021-02-10T04:55:46.640294+0000","last_clean":"2021-02-10T04:55:46.640294+0000","last_became_active":"2021-02-10T04:51:36.843553+0000","last_became_peered":"2021-02-10T04:51:36.843553+0000","last_unstale":"2021-02-10T04:55:46.640294+0000","last_undegraded":"2021-02-10T04:55:46.640294+0000","last_fullsized":"2021-02-10T04:55:46.640294+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:37.324040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:37.324040+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"1232","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.628062+0000","last_change":"2021-02-10T04:54:37.612303+0000","last_active":"2021-02-10T04:55:45.628062+0000","last_peered":"2021-02-10T04:55:45.628062+0000","last_clean":"2021-02-10T04:55:45.628062+0000","last_became_active":"2021-02-10T04:54:37.611953+0000","last_became_peered":"2021-02-10T04:54:37.611953+0000","last_unstale":"2021-02-10T04:55:45.628062+0000","last_undegraded":"2021-02-10T04:55:45.628062+0000","last_fullsized":"2021-02-10T04:55:45.628062+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"1255","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.640935+0000","last_change":"2021-02-10T04:52:31.403904+0000","last_active":"2021-02-10T04:55:46.640935+0000","last_peered":"2021-02-10T04:55:46.640935+0000","last_clean":"2021-02-10T04:55:46.640935+0000","last_became_active":"2021-02-10T04:50:59.538009+0000","last_became_peered":"2021-02-10T04:50:59.538009+0000","last_unstale":"2021-02-10T04:55:46.640935+0000","last_undegraded":"2021-02-10T04:55:46.640935+0000","last_fullsized":"2021-02-10T04:55:46.640935+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:31.403824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.403824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"1000'490","reported_seq":"3725","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.654124+0000","last_change":"2021-02-10T04:55:22.716472+0000","last_active":"2021-02-10T04:55:48.654124+0000","last_peered":"2021-02-10T04:55:48.654124+0000","last_clean":"2021-02-10T04:55:48.654124+0000","last_became_active":"2021-02-10T04:55:14.644596+0000","last_became_peered":"2021-02-10T04:55:14.644596+0000","last_unstale":"2021-02-10T04:55:48.654124+0000","last_undegraded":"2021-02-10T04:55:48.654124+0000","last_fullsized":"2021-02-10T04:55:48.654124+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:47:02.875920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:02.875920+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19371627,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":141,"num_read_kb":148568,"num_write":85,"num_write_kb":31732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"155","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.654404+0000","last_change":"2021-02-10T04:53:01.972991+0000","last_active":"2021-02-10T04:55:48.654404+0000","last_peered":"2021-02-10T04:55:48.654404+0000","last_clean":"2021-02-10T04:55:48.654404+0000","last_became_active":"2021-02-10T04:51:36.863976+0000","last_became_peered":"2021-02-10T04:51:36.863976+0000","last_unstale":"2021-02-10T04:55:48.654404+0000","last_undegraded":"2021-02-10T04:55:48.654404+0000","last_fullsized":"2021-02-10T04:55:48.654404+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:01.972913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.972913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"1063'30","reported_seq":"960","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.647391+0000","last_change":"2021-02-10T04:55:43.569281+0000","last_active":"2021-02-10T04:55:46.647391+0000","last_peered":"2021-02-10T04:55:46.647391+0000","last_clean":"2021-02-10T04:55:46.647391+0000","last_became_active":"2021-02-10T04:55:43.568878+0000","last_became_peered":"2021-02-10T04:55:43.568878+0000","last_unstale":"2021-02-10T04:55:46.647391+0000","last_undegraded":"2021-02-10T04:55:46.647391+0000","last_fullsized":"2021-02-10T04:55:46.647391+0000","mapping_epoch":1227,"log_start":"1063'30","ondisk_log_start":"1063'30","created":55,"last_epoch_clean":1253,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:56.431187+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.431187+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":802,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":101,"num_read_kb":79,"num_write":57,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"165","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.650183+0000","last_change":"2021-02-10T04:55:14.636849+0000","last_active":"2021-02-10T04:55:48.650183+0000","last_peered":"2021-02-10T04:55:48.650183+0000","last_clean":"2021-02-10T04:55:48.650183+0000","last_became_active":"2021-02-10T04:55:14.636237+0000","last_became_peered":"2021-02-10T04:55:14.636237+0000","last_unstale":"2021-02-10T04:55:48.650183+0000","last_undegraded":"2021-02-10T04:55:48.650183+0000","last_fullsized":"2021-02-10T04:55:48.650183+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:04.591695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:04.591695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"1082","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.639793+0000","last_change":"2021-02-10T04:54:37.680913+0000","last_active":"2021-02-10T04:55:46.639793+0000","last_peered":"2021-02-10T04:55:46.639793+0000","last_clean":"2021-02-10T04:55:46.639793+0000","last_became_active":"2021-02-10T04:54:37.680651+0000","last_became_peered":"2021-02-10T04:54:37.680651+0000","last_unstale":"2021-02-10T04:55:46.639793+0000","last_undegraded":"2021-02-10T04:55:46.639793+0000","last_fullsized":"2021-02-10T04:55:46.639793+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1210,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":149,"num_read_kb":148,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"1102","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:49.360262+0000","last_change":"2021-02-10T04:53:46.729138+0000","last_active":"2021-02-10T04:55:49.360262+0000","last_peered":"2021-02-10T04:55:49.360262+0000","last_clean":"2021-02-10T04:55:49.360262+0000","last_became_active":"2021-02-10T04:53:46.728845+0000","last_became_peered":"2021-02-10T04:53:46.728845+0000","last_unstale":"2021-02-10T04:55:49.360262+0000","last_undegraded":"2021-02-10T04:55:49.360262+0000","last_fullsized":"2021-02-10T04:55:49.360262+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:40.066354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:52:40.066354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"1000'620","reported_seq":"4349","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.651694+0000","last_change":"2021-02-10T04:55:28.089086+0000","last_active":"2021-02-10T04:55:48.651694+0000","last_peered":"2021-02-10T04:55:48.651694+0000","last_clean":"2021-02-10T04:55:48.651694+0000","last_became_active":"2021-02-10T04:55:14.640509+0000","last_became_peered":"2021-02-10T04:55:14.640509+0000","last_unstale":"2021-02-10T04:55:48.651694+0000","last_undegraded":"2021-02-10T04:55:48.651694+0000","last_fullsized":"2021-02-10T04:55:48.651694+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:51:54.444287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:54.444287+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14265195,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":144,"num_read_kb":128482,"num_write":129,"num_write_kb":53443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"1138","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.651436+0000","last_change":"2021-02-10T04:55:28.083286+0000","last_active":"2021-02-10T04:55:48.651436+0000","last_peered":"2021-02-10T04:55:48.651436+0000","last_clean":"2021-02-10T04:55:48.651436+0000","last_became_active":"2021-02-10T04:55:14.619846+0000","last_became_peered":"2021-02-10T04:55:14.619846+0000","last_unstale":"2021-02-10T04:55:48.651436+0000","last_undegraded":"2021-02-10T04:55:48.651436+0000","last_fullsized":"2021-02-10T04:55:48.651436+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:14.955544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:47:14.955544+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"1235","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.914949+0000","last_change":"2021-02-10T04:55:15.590025+0000","last_active":"2021-02-10T04:55:48.914949+0000","last_peered":"2021-02-10T04:55:48.914949+0000","last_clean":"2021-02-10T04:55:48.914949+0000","last_became_active":"2021-02-10T04:55:15.589651+0000","last_became_peered":"2021-02-10T04:55:15.589651+0000","last_unstale":"2021-02-10T04:55:48.914949+0000","last_undegraded":"2021-02-10T04:55:48.914949+0000","last_fullsized":"2021-02-10T04:55:48.914949+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"1241","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.651863+0000","last_change":"2021-02-10T04:55:28.090104+0000","last_active":"2021-02-10T04:55:48.651863+0000","last_peered":"2021-02-10T04:55:48.651863+0000","last_clean":"2021-02-10T04:55:48.651863+0000","last_became_active":"2021-02-10T04:55:14.650106+0000","last_became_peered":"2021-02-10T04:55:14.650106+0000","last_unstale":"2021-02-10T04:55:48.651863+0000","last_undegraded":"2021-02-10T04:55:48.651863+0000","last_fullsized":"2021-02-10T04:55:48.651863+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:27.908221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:47:27.908221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"1049'55","reported_seq":"889","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.647552+0000","last_change":"2021-02-10T04:52:30.277009+0000","last_active":"2021-02-10T04:55:46.647552+0000","last_peered":"2021-02-10T04:55:46.647552+0000","last_clean":"2021-02-10T04:55:46.647552+0000","last_became_active":"2021-02-10T04:51:49.501622+0000","last_became_peered":"2021-02-10T04:51:49.501622+0000","last_unstale":"2021-02-10T04:55:46.647552+0000","last_undegraded":"2021-02-10T04:55:46.647552+0000","last_fullsized":"2021-02-10T04:55:46.647552+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'55","last_scrub_stamp":"2021-02-10T04:52:30.276938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.276938+0000","log_size":55,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":98,"num_read_kb":74,"num_write":58,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"1251","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.640283+0000","last_change":"2021-02-10T04:51:59.420149+0000","last_active":"2021-02-10T04:55:46.640283+0000","last_peered":"2021-02-10T04:55:46.640283+0000","last_clean":"2021-02-10T04:55:46.640283+0000","last_became_active":"2021-02-10T04:28:21.125541+0000","last_became_peered":"2021-02-10T04:28:21.125541+0000","last_unstale":"2021-02-10T04:55:46.640283+0000","last_undegraded":"2021-02-10T04:55:46.640283+0000","last_fullsized":"2021-02-10T04:55:46.640283+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.420086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.420086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"1216","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.629004+0000","last_change":"2021-02-10T04:53:12.187225+0000","last_active":"2021-02-10T04:55:45.629004+0000","last_peered":"2021-02-10T04:55:45.629004+0000","last_clean":"2021-02-10T04:55:45.629004+0000","last_became_active":"2021-02-10T04:50:59.555689+0000","last_became_peered":"2021-02-10T04:50:59.555689+0000","last_unstale":"2021-02-10T04:55:45.629004+0000","last_undegraded":"2021-02-10T04:55:45.629004+0000","last_fullsized":"2021-02-10T04:55:45.629004+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.187154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.187154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"1002'2042","reported_seq":"9489","reported_epoch":"1257","state":"active+clean","last_fresh":"2021-02-10T04:55:47.645485+0000","last_change":"2021-02-10T04:55:14.662208+0000","last_active":"2021-02-10T04:55:47.645485+0000","last_peered":"2021-02-10T04:55:47.645485+0000","last_clean":"2021-02-10T04:55:47.645485+0000","last_became_active":"2021-02-10T04:55:14.659601+0000","last_became_peered":"2021-02-10T04:55:14.659601+0000","last_unstale":"2021-02-10T04:55:47.645485+0000","last_undegraded":"2021-02-10T04:55:47.645485+0000","last_fullsized":"2021-02-10T04:55:47.645485+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:46:57.054230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:57.054230+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":85678898,"num_objects":35,"num_object_clones":29,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"1202","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.639826+0000","last_change":"2021-02-10T04:55:14.651301+0000","last_active":"2021-02-10T04:55:46.639826+0000","last_peered":"2021-02-10T04:55:46.639826+0000","last_clean":"2021-02-10T04:55:46.639826+0000","last_became_active":"2021-02-10T04:55:14.650386+0000","last_became_peered":"2021-02-10T04:55:14.650386+0000","last_unstale":"2021-02-10T04:55:46.639826+0000","last_undegraded":"2021-02-10T04:55:46.639826+0000","last_fullsized":"2021-02-10T04:55:46.639826+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:52.341527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:50:52.341527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"55","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.641286+0000","last_change":"2021-02-10T04:54:39.558380+0000","last_active":"2021-02-10T04:55:46.641286+0000","last_peered":"2021-02-10T04:55:46.641286+0000","last_clean":"2021-02-10T04:55:46.641286+0000","last_became_active":"2021-02-10T04:54:39.557995+0000","last_became_peered":"2021-02-10T04:54:39.557995+0000","last_unstale":"2021-02-10T04:55:46.641286+0000","last_undegraded":"2021-02-10T04:55:46.641286+0000","last_fullsized":"2021-02-10T04:55:46.641286+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"1238'4","reported_seq":"1438","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.652608+0000","last_change":"2021-02-10T04:55:28.095426+0000","last_active":"2021-02-10T04:55:48.652608+0000","last_peered":"2021-02-10T04:55:48.652608+0000","last_clean":"2021-02-10T04:55:48.652608+0000","last_became_active":"2021-02-10T04:55:14.645890+0000","last_became_peered":"2021-02-10T04:55:14.645890+0000","last_unstale":"2021-02-10T04:55:48.652608+0000","last_undegraded":"2021-02-10T04:55:48.652608+0000","last_fullsized":"2021-02-10T04:55:48.652608+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:54.466259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:54.466259+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"1049'54","reported_seq":"938","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.646907+0000","last_change":"2021-02-10T04:53:01.304693+0000","last_active":"2021-02-10T04:55:46.646907+0000","last_peered":"2021-02-10T04:55:46.646907+0000","last_clean":"2021-02-10T04:55:46.646907+0000","last_became_active":"2021-02-10T04:51:49.498066+0000","last_became_peered":"2021-02-10T04:51:49.498066+0000","last_unstale":"2021-02-10T04:55:46.646907+0000","last_undegraded":"2021-02-10T04:55:46.646907+0000","last_fullsized":"2021-02-10T04:55:46.646907+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:53:01.304626+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.304626+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":74,"num_read_kb":56,"num_write":42,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"514","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.649551+0000","last_change":"2021-02-10T04:55:14.610552+0000","last_active":"2021-02-10T04:55:46.649551+0000","last_peered":"2021-02-10T04:55:46.649551+0000","last_clean":"2021-02-10T04:55:46.649551+0000","last_became_active":"2021-02-10T04:55:14.609458+0000","last_became_peered":"2021-02-10T04:55:14.609458+0000","last_unstale":"2021-02-10T04:55:46.649551+0000","last_undegraded":"2021-02-10T04:55:46.649551+0000","last_fullsized":"2021-02-10T04:55:46.649551+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:49.447961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:49.447961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"1181","reported_epoch":"1259","state":"active+clean","last_fresh":"2021-02-10T04:55:51.081872+0000","last_change":"2021-02-10T04:55:51.082732+0000","last_active":"2021-02-10T04:55:51.081872+0000","last_peered":"2021-02-10T04:55:51.081872+0000","last_clean":"2021-02-10T04:55:51.081872+0000","last_became_active":"2021-02-10T04:55:14.614581+0000","last_became_peered":"2021-02-10T04:55:14.614581+0000","last_unstale":"2021-02-10T04:55:51.081872+0000","last_undegraded":"2021-02-10T04:55:51.081872+0000","last_fullsized":"2021-02-10T04:55:51.081872+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:07.914061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:07.914061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"293'77","reported_seq":"3040","reported_epoch":"1258","state":"active+clean+scrubbing","last_fresh":"2021-02-10T04:55:48.914534+0000","last_change":"2021-02-10T04:53:43.102301+0000","last_active":"2021-02-10T04:55:48.914534+0000","last_peered":"2021-02-10T04:55:48.914534+0000","last_clean":"2021-02-10T04:55:48.914534+0000","last_became_active":"2021-02-10T04:52:26.024812+0000","last_became_peered":"2021-02-10T04:52:26.024812+0000","last_unstale":"2021-02-10T04:55:48.914534+0000","last_undegraded":"2021-02-10T04:55:48.914534+0000","last_fullsized":"2021-02-10T04:55:48.914534+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1886585,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":18587,"num_write":9,"num_write_kb":3315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"1271","reported_epoch":"1257","state":"active+clean","last_fresh":"2021-02-10T04:55:47.645398+0000","last_change":"2021-02-10T04:53:07.025068+0000","last_active":"2021-02-10T04:55:47.645398+0000","last_peered":"2021-02-10T04:55:47.645398+0000","last_clean":"2021-02-10T04:55:47.645398+0000","last_became_active":"2021-02-10T04:50:59.589417+0000","last_became_peered":"2021-02-10T04:50:59.589417+0000","last_unstale":"2021-02-10T04:55:47.645398+0000","last_undegraded":"2021-02-10T04:55:47.645398+0000","last_fullsized":"2021-02-10T04:55:47.645398+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.024980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.024980+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"179","reported_epoch":"1259","state":"active+clean","last_fresh":"2021-02-10T04:55:50.750255+0000","last_change":"2021-02-10T04:55:50.750967+0000","last_active":"2021-02-10T04:55:50.750255+0000","last_peered":"2021-02-10T04:55:50.750255+0000","last_clean":"2021-02-10T04:55:50.750255+0000","last_became_active":"2021-02-10T04:55:14.605267+0000","last_became_peered":"2021-02-10T04:55:14.605267+0000","last_unstale":"2021-02-10T04:55:50.750255+0000","last_undegraded":"2021-02-10T04:55:50.750255+0000","last_fullsized":"2021-02-10T04:55:50.750255+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.609500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.609500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"1049'68","reported_seq":"1412","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.914811+0000","last_change":"2021-02-10T04:55:35.577794+0000","last_active":"2021-02-10T04:55:48.914811+0000","last_peered":"2021-02-10T04:55:48.914811+0000","last_clean":"2021-02-10T04:55:48.914811+0000","last_became_active":"2021-02-10T04:55:35.577316+0000","last_became_peered":"2021-02-10T04:55:35.577316+0000","last_unstale":"2021-02-10T04:55:48.914811+0000","last_undegraded":"2021-02-10T04:55:48.914811+0000","last_fullsized":"2021-02-10T04:55:48.914811+0000","mapping_epoch":1238,"log_start":"1049'68","ondisk_log_start":"1049'68","created":55,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'39","last_scrub_stamp":"2021-02-10T04:53:38.085565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.085565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1582,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":200,"num_read_kb":160,"num_write":110,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"1372","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.916040+0000","last_change":"2021-02-10T04:55:14.659217+0000","last_active":"2021-02-10T04:55:48.916040+0000","last_peered":"2021-02-10T04:55:48.916040+0000","last_clean":"2021-02-10T04:55:48.916040+0000","last_became_active":"2021-02-10T04:55:14.658879+0000","last_became_peered":"2021-02-10T04:55:14.658879+0000","last_unstale":"2021-02-10T04:55:48.916040+0000","last_undegraded":"2021-02-10T04:55:48.916040+0000","last_fullsized":"2021-02-10T04:55:48.916040+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:47:08.784593+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:47:08.784593+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"1218","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.630292+0000","last_change":"2021-02-10T04:52:30.019225+0000","last_active":"2021-02-10T04:55:45.630292+0000","last_peered":"2021-02-10T04:55:45.630292+0000","last_clean":"2021-02-10T04:55:45.630292+0000","last_became_active":"2021-02-10T04:50:59.561436+0000","last_became_peered":"2021-02-10T04:50:59.561436+0000","last_unstale":"2021-02-10T04:55:45.630292+0000","last_undegraded":"2021-02-10T04:55:45.630292+0000","last_fullsized":"2021-02-10T04:55:45.630292+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.019102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.019102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"1001'1528","reported_seq":"6203","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.627894+0000","last_change":"2021-02-10T04:52:26.036366+0000","last_active":"2021-02-10T04:55:45.627894+0000","last_peered":"2021-02-10T04:55:45.627894+0000","last_clean":"2021-02-10T04:55:45.627894+0000","last_became_active":"2021-02-10T04:52:26.034028+0000","last_became_peered":"2021-02-10T04:52:26.034028+0000","last_unstale":"2021-02-10T04:55:45.627894+0000","last_undegraded":"2021-02-10T04:55:45.627894+0000","last_fullsized":"2021-02-10T04:55:45.627894+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46944136,"num_objects":16,"num_object_clones":15,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":358,"num_read_kb":351934,"num_write":286,"num_write_kb":111355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":58968832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.15","version":"1000'446","reported_seq":"2195","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.915847+0000","last_change":"2021-02-10T04:53:11.051124+0000","last_active":"2021-02-10T04:55:48.915847+0000","last_peered":"2021-02-10T04:55:48.915847+0000","last_clean":"2021-02-10T04:55:48.915847+0000","last_became_active":"2021-02-10T04:51:40.510899+0000","last_became_peered":"2021-02-10T04:51:40.510899+0000","last_unstale":"2021-02-10T04:55:48.915847+0000","last_undegraded":"2021-02-10T04:55:48.915847+0000","last_fullsized":"2021-02-10T04:55:48.915847+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:53:11.051031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.051031+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24896285,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":18983408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"169","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.655498+0000","last_change":"2021-02-10T04:55:14.605585+0000","last_active":"2021-02-10T04:55:48.655498+0000","last_peered":"2021-02-10T04:55:48.655498+0000","last_clean":"2021-02-10T04:55:48.655498+0000","last_became_active":"2021-02-10T04:55:14.604916+0000","last_became_peered":"2021-02-10T04:55:14.604916+0000","last_unstale":"2021-02-10T04:55:48.655498+0000","last_undegraded":"2021-02-10T04:55:48.655498+0000","last_fullsized":"2021-02-10T04:55:48.655498+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:51:08.639784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.639784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"1187","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.629515+0000","last_change":"2021-02-10T04:51:09.475150+0000","last_active":"2021-02-10T04:55:45.629515+0000","last_peered":"2021-02-10T04:55:45.629515+0000","last_clean":"2021-02-10T04:55:45.629515+0000","last_became_active":"2021-02-10T04:50:59.585917+0000","last_became_peered":"2021-02-10T04:50:59.585917+0000","last_unstale":"2021-02-10T04:55:45.629515+0000","last_undegraded":"2021-02-10T04:55:45.629515+0000","last_fullsized":"2021-02-10T04:55:45.629515+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:09.475063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:09.475063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"160","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.653218+0000","last_change":"2021-02-10T04:53:34.927483+0000","last_active":"2021-02-10T04:55:48.653218+0000","last_peered":"2021-02-10T04:55:48.653218+0000","last_clean":"2021-02-10T04:55:48.653218+0000","last_became_active":"2021-02-10T04:51:36.859707+0000","last_became_peered":"2021-02-10T04:51:36.859707+0000","last_unstale":"2021-02-10T04:55:48.653218+0000","last_undegraded":"2021-02-10T04:55:48.653218+0000","last_fullsized":"2021-02-10T04:55:48.653218+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:34.927424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.927424+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"1000'620","reported_seq":"4542","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.628313+0000","last_change":"2021-02-10T04:53:37.384461+0000","last_active":"2021-02-10T04:55:45.628313+0000","last_peered":"2021-02-10T04:55:45.628313+0000","last_clean":"2021-02-10T04:55:45.628313+0000","last_became_active":"2021-02-10T04:51:40.510114+0000","last_became_peered":"2021-02-10T04:51:40.510114+0000","last_unstale":"2021-02-10T04:55:45.628313+0000","last_undegraded":"2021-02-10T04:55:45.628313+0000","last_fullsized":"2021-02-10T04:55:45.628313+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:53:37.384398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:37.384398+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28530390,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"1245","reported_epoch":"1257","state":"active+clean","last_fresh":"2021-02-10T04:55:47.644645+0000","last_change":"2021-02-10T04:52:41.059269+0000","last_active":"2021-02-10T04:55:47.644645+0000","last_peered":"2021-02-10T04:55:47.644645+0000","last_clean":"2021-02-10T04:55:47.644645+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:55:47.644645+0000","last_undegraded":"2021-02-10T04:55:47.644645+0000","last_fullsized":"2021-02-10T04:55:47.644645+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:52:41.059159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.059159+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"1248","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.641842+0000","last_change":"2021-02-10T04:53:12.332890+0000","last_active":"2021-02-10T04:55:46.641842+0000","last_peered":"2021-02-10T04:55:46.641842+0000","last_clean":"2021-02-10T04:55:46.641842+0000","last_became_active":"2021-02-10T04:50:59.540375+0000","last_became_peered":"2021-02-10T04:50:59.540375+0000","last_unstale":"2021-02-10T04:55:46.641842+0000","last_undegraded":"2021-02-10T04:55:46.641842+0000","last_fullsized":"2021-02-10T04:55:46.641842+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.332233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.332233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"185","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:49.360186+0000","last_change":"2021-02-10T04:55:30.910181+0000","last_active":"2021-02-10T04:55:49.360186+0000","last_peered":"2021-02-10T04:55:49.360186+0000","last_clean":"2021-02-10T04:55:49.360186+0000","last_became_active":"2021-02-10T04:53:46.733191+0000","last_became_peered":"2021-02-10T04:53:46.733191+0000","last_unstale":"2021-02-10T04:55:49.360186+0000","last_undegraded":"2021-02-10T04:55:49.360186+0000","last_fullsized":"2021-02-10T04:55:49.360186+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:45.362537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:45.362537+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"1002'2042","reported_seq":"9231","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:49.360495+0000","last_change":"2021-02-10T04:53:46.726353+0000","last_active":"2021-02-10T04:55:49.360495+0000","last_peered":"2021-02-10T04:55:49.360495+0000","last_clean":"2021-02-10T04:55:49.360495+0000","last_became_active":"2021-02-10T04:53:46.725828+0000","last_became_peered":"2021-02-10T04:53:46.725828+0000","last_unstale":"2021-02-10T04:55:49.360495+0000","last_undegraded":"2021-02-10T04:55:49.360495+0000","last_fullsized":"2021-02-10T04:55:49.360495+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:52:41.286006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.286006+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46300076,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":545,"num_read_kb":531689,"num_write":399,"num_write_kb":153087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"1049'46","reported_seq":"1411","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.628689+0000","last_change":"2021-02-10T04:52:35.091494+0000","last_active":"2021-02-10T04:55:45.628689+0000","last_peered":"2021-02-10T04:55:45.628689+0000","last_clean":"2021-02-10T04:55:45.628689+0000","last_became_active":"2021-02-10T04:50:59.569915+0000","last_became_peered":"2021-02-10T04:50:59.569915+0000","last_unstale":"2021-02-10T04:55:45.628689+0000","last_undegraded":"2021-02-10T04:55:45.628689+0000","last_fullsized":"2021-02-10T04:55:45.628689+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'46","last_scrub_stamp":"2021-02-10T04:52:35.091374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:35.091374+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":65,"num_write":46,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"84'7","reported_seq":"1272","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.640209+0000","last_change":"2021-02-10T04:51:52.498251+0000","last_active":"2021-02-10T04:55:46.640209+0000","last_peered":"2021-02-10T04:55:46.640209+0000","last_clean":"2021-02-10T04:55:46.640209+0000","last_became_active":"2021-02-10T04:50:59.575924+0000","last_became_peered":"2021-02-10T04:50:59.575924+0000","last_unstale":"2021-02-10T04:55:46.640209+0000","last_undegraded":"2021-02-10T04:55:46.640209+0000","last_fullsized":"2021-02-10T04:55:46.640209+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'7","last_scrub_stamp":"2021-02-10T04:51:52.498158+0000","last_deep_scrub":"84'7","last_deep_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.498158+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"1229'4","reported_seq":"1606","reported_epoch":"1257","state":"active+clean","last_fresh":"2021-02-10T04:55:47.648588+0000","last_change":"2021-02-10T04:55:14.657278+0000","last_active":"2021-02-10T04:55:47.648588+0000","last_peered":"2021-02-10T04:55:47.648588+0000","last_clean":"2021-02-10T04:55:47.648588+0000","last_became_active":"2021-02-10T04:55:14.656867+0000","last_became_peered":"2021-02-10T04:55:14.656867+0000","last_unstale":"2021-02-10T04:55:47.648588+0000","last_undegraded":"2021-02-10T04:55:47.648588+0000","last_fullsized":"2021-02-10T04:55:47.648588+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:47.407557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:47.407557+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"1172","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.655724+0000","last_change":"2021-02-10T04:55:22.735243+0000","last_active":"2021-02-10T04:55:48.655724+0000","last_peered":"2021-02-10T04:55:48.655724+0000","last_clean":"2021-02-10T04:55:48.655724+0000","last_became_active":"2021-02-10T04:55:14.632398+0000","last_became_peered":"2021-02-10T04:55:14.632398+0000","last_unstale":"2021-02-10T04:55:48.655724+0000","last_undegraded":"2021-02-10T04:55:48.655724+0000","last_fullsized":"2021-02-10T04:55:48.655724+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:23.566027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_clean_scrub_stamp":"2021-02-10T04:52:23.566027+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"1000'490","reported_seq":"3774","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.916022+0000","last_change":"2021-02-10T04:55:14.644527+0000","last_active":"2021-02-10T04:55:48.916022+0000","last_peered":"2021-02-10T04:55:48.916022+0000","last_clean":"2021-02-10T04:55:48.916022+0000","last_became_active":"2021-02-10T04:55:14.643764+0000","last_became_peered":"2021-02-10T04:55:14.643764+0000","last_unstale":"2021-02-10T04:55:48.916022+0000","last_undegraded":"2021-02-10T04:55:48.916022+0000","last_fullsized":"2021-02-10T04:55:48.916022+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:50:03.756248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:50:03.756248+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16154790,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"1112'40","reported_seq":"1316","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.641211+0000","last_change":"2021-02-10T04:53:10.276449+0000","last_active":"2021-02-10T04:55:46.641211+0000","last_peered":"2021-02-10T04:55:46.641211+0000","last_clean":"2021-02-10T04:55:46.641211+0000","last_became_active":"2021-02-10T04:50:59.539529+0000","last_became_peered":"2021-02-10T04:50:59.539529+0000","last_unstale":"2021-02-10T04:55:46.641211+0000","last_undegraded":"2021-02-10T04:55:46.641211+0000","last_fullsized":"2021-02-10T04:55:46.641211+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1112'40","last_scrub_stamp":"2021-02-10T04:53:10.276358+0000","last_deep_scrub":"1112'40","last_deep_scrub_stamp":"2021-02-10T04:51:01.450050+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.276358+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":59,"num_read_kb":44,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"1157","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.653082+0000","last_change":"2021-02-10T04:55:28.088480+0000","last_active":"2021-02-10T04:55:48.653082+0000","last_peered":"2021-02-10T04:55:48.653082+0000","last_clean":"2021-02-10T04:55:48.653082+0000","last_became_active":"2021-02-10T04:55:14.631666+0000","last_became_peered":"2021-02-10T04:55:14.631666+0000","last_unstale":"2021-02-10T04:55:48.653082+0000","last_undegraded":"2021-02-10T04:55:48.653082+0000","last_fullsized":"2021-02-10T04:55:48.653082+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:23.904210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:47:23.904210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"1119'6","reported_seq":"1518","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.628851+0000","last_change":"2021-02-10T04:52:40.038646+0000","last_active":"2021-02-10T04:55:45.628851+0000","last_peered":"2021-02-10T04:55:45.628851+0000","last_clean":"2021-02-10T04:55:45.628851+0000","last_became_active":"2021-02-10T04:51:36.869733+0000","last_became_peered":"2021-02-10T04:51:36.869733+0000","last_unstale":"2021-02-10T04:55:45.628851+0000","last_undegraded":"2021-02-10T04:55:45.628851+0000","last_fullsized":"2021-02-10T04:55:45.628851+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'6","last_scrub_stamp":"2021-02-10T04:52:40.038554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:40.038554+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"1244","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.916231+0000","last_change":"2021-02-10T04:51:36.849933+0000","last_active":"2021-02-10T04:55:48.916231+0000","last_peered":"2021-02-10T04:55:48.916231+0000","last_clean":"2021-02-10T04:55:48.916231+0000","last_became_active":"2021-02-10T04:51:36.849184+0000","last_became_peered":"2021-02-10T04:51:36.849184+0000","last_unstale":"2021-02-10T04:55:48.916231+0000","last_undegraded":"2021-02-10T04:55:48.916231+0000","last_fullsized":"2021-02-10T04:55:48.916231+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:11.779779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:11.779779+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"1032'89","reported_seq":"852","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:49.359482+0000","last_change":"2021-02-10T04:53:46.728868+0000","last_active":"2021-02-10T04:55:49.359482+0000","last_peered":"2021-02-10T04:55:49.359482+0000","last_clean":"2021-02-10T04:55:49.359482+0000","last_became_active":"2021-02-10T04:53:46.728356+0000","last_became_peered":"2021-02-10T04:53:46.728356+0000","last_unstale":"2021-02-10T04:55:49.359482+0000","last_undegraded":"2021-02-10T04:55:49.359482+0000","last_fullsized":"2021-02-10T04:55:49.359482+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":5,"last_scrub":"1032'89","last_scrub_stamp":"2021-02-10T04:52:34.089789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:34.089789+0000","log_size":89,"ondisk_log_size":89,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5218574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":30326,"num_write":21,"num_write_kb":12444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3210022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"1163'37","reported_seq":"937","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.639835+0000","last_change":"2021-02-10T04:53:14.202505+0000","last_active":"2021-02-10T04:55:46.639835+0000","last_peered":"2021-02-10T04:55:46.639835+0000","last_clean":"2021-02-10T04:55:46.639835+0000","last_became_active":"2021-02-10T04:51:49.496294+0000","last_became_peered":"2021-02-10T04:51:49.496294+0000","last_unstale":"2021-02-10T04:55:46.639835+0000","last_undegraded":"2021-02-10T04:55:46.639835+0000","last_fullsized":"2021-02-10T04:55:46.639835+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1163'37","last_scrub_stamp":"2021-02-10T04:53:14.202445+0000","last_deep_scrub":"1163'37","last_deep_scrub_stamp":"2021-02-10T04:53:14.202445+0000","last_clean_scrub_stamp":"2021-02-10T04:53:14.202445+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":33,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"1258","reported_epoch":"1257","state":"active+clean","last_fresh":"2021-02-10T04:55:47.644794+0000","last_change":"2021-02-10T04:52:23.095651+0000","last_active":"2021-02-10T04:55:47.644794+0000","last_peered":"2021-02-10T04:55:47.644794+0000","last_clean":"2021-02-10T04:55:47.644794+0000","last_became_active":"2021-02-10T04:50:59.543531+0000","last_became_peered":"2021-02-10T04:50:59.543531+0000","last_unstale":"2021-02-10T04:55:47.644794+0000","last_undegraded":"2021-02-10T04:55:47.644794+0000","last_fullsized":"2021-02-10T04:55:47.644794+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:23.095588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:23.095588+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"1206","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.630020+0000","last_change":"2021-02-10T04:55:14.636711+0000","last_active":"2021-02-10T04:55:45.630020+0000","last_peered":"2021-02-10T04:55:45.630020+0000","last_clean":"2021-02-10T04:55:45.630020+0000","last_became_active":"2021-02-10T04:55:14.636363+0000","last_became_peered":"2021-02-10T04:55:14.636363+0000","last_unstale":"2021-02-10T04:55:45.630020+0000","last_undegraded":"2021-02-10T04:55:45.630020+0000","last_fullsized":"2021-02-10T04:55:45.630020+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:43.663762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:51:43.663762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"1128","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.650953+0000","last_change":"2021-02-10T04:55:28.101709+0000","last_active":"2021-02-10T04:55:48.650953+0000","last_peered":"2021-02-10T04:55:48.650953+0000","last_clean":"2021-02-10T04:55:48.650953+0000","last_became_active":"2021-02-10T04:55:14.635912+0000","last_became_peered":"2021-02-10T04:55:14.635912+0000","last_unstale":"2021-02-10T04:55:48.650953+0000","last_undegraded":"2021-02-10T04:55:48.650953+0000","last_fullsized":"2021-02-10T04:55:48.650953+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:57.568541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:57.568541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"1000'1071","reported_seq":"4761","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.627275+0000","last_change":"2021-02-10T04:53:35.302879+0000","last_active":"2021-02-10T04:55:45.627275+0000","last_peered":"2021-02-10T04:55:45.627275+0000","last_clean":"2021-02-10T04:55:45.627275+0000","last_became_active":"2021-02-10T04:51:39.508551+0000","last_became_peered":"2021-02-10T04:51:39.508551+0000","last_unstale":"2021-02-10T04:55:45.627275+0000","last_undegraded":"2021-02-10T04:55:45.627275+0000","last_fullsized":"2021-02-10T04:55:45.627275+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:35.302780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.302780+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"1049'31","reported_seq":"1281","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.915087+0000","last_change":"2021-02-10T04:52:39.695340+0000","last_active":"2021-02-10T04:55:48.915087+0000","last_peered":"2021-02-10T04:55:48.915087+0000","last_clean":"2021-02-10T04:55:48.915087+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:55:48.915087+0000","last_undegraded":"2021-02-10T04:55:48.915087+0000","last_fullsized":"2021-02-10T04:55:48.915087+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'31","last_scrub_stamp":"2021-02-10T04:52:39.695237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:39.695237+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":41,"num_write":32,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"501","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.648032+0000","last_change":"2021-02-10T04:55:20.929399+0000","last_active":"2021-02-10T04:55:46.648032+0000","last_peered":"2021-02-10T04:55:46.648032+0000","last_clean":"2021-02-10T04:55:46.648032+0000","last_became_active":"2021-02-10T04:55:14.655543+0000","last_became_peered":"2021-02-10T04:55:14.655543+0000","last_unstale":"2021-02-10T04:55:46.648032+0000","last_undegraded":"2021-02-10T04:55:46.648032+0000","last_fullsized":"2021-02-10T04:55:46.648032+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:55:20.929330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:55:20.929330+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"909","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.646990+0000","last_change":"2021-02-10T04:53:35.216489+0000","last_active":"2021-02-10T04:55:46.646990+0000","last_peered":"2021-02-10T04:55:46.646990+0000","last_clean":"2021-02-10T04:55:46.646990+0000","last_became_active":"2021-02-10T04:51:49.501254+0000","last_became_peered":"2021-02-10T04:51:49.501254+0000","last_unstale":"2021-02-10T04:55:46.646990+0000","last_undegraded":"2021-02-10T04:55:46.646990+0000","last_fullsized":"2021-02-10T04:55:46.646990+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.216394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.216394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"1233","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.628744+0000","last_change":"2021-02-10T04:55:14.633276+0000","last_active":"2021-02-10T04:55:45.628744+0000","last_peered":"2021-02-10T04:55:45.628744+0000","last_clean":"2021-02-10T04:55:45.628744+0000","last_became_active":"2021-02-10T04:55:14.632805+0000","last_became_peered":"2021-02-10T04:55:14.632805+0000","last_unstale":"2021-02-10T04:55:45.628744+0000","last_undegraded":"2021-02-10T04:55:45.628744+0000","last_fullsized":"2021-02-10T04:55:45.628744+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:55.814486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.814486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"1002'1657","reported_seq":"5281","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.652960+0000","last_change":"2021-02-10T04:55:22.728878+0000","last_active":"2021-02-10T04:55:48.652960+0000","last_peered":"2021-02-10T04:55:48.652960+0000","last_clean":"2021-02-10T04:55:48.652960+0000","last_became_active":"2021-02-10T04:55:14.653270+0000","last_became_peered":"2021-02-10T04:55:14.653270+0000","last_unstale":"2021-02-10T04:55:48.652960+0000","last_undegraded":"2021-02-10T04:55:48.652960+0000","last_fullsized":"2021-02-10T04:55:48.652960+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:55:22.331010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:55:22.331010+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":47335497,"num_objects":20,"num_object_clones":18,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"1048'20","reported_seq":"1357","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.627779+0000","last_change":"2021-02-10T04:53:04.058324+0000","last_active":"2021-02-10T04:55:45.627779+0000","last_peered":"2021-02-10T04:55:45.627779+0000","last_clean":"2021-02-10T04:55:45.627779+0000","last_became_active":"2021-02-10T04:50:59.544484+0000","last_became_peered":"2021-02-10T04:50:59.544484+0000","last_unstale":"2021-02-10T04:55:45.627779+0000","last_undegraded":"2021-02-10T04:55:45.627779+0000","last_fullsized":"2021-02-10T04:55:45.627779+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'20","last_scrub_stamp":"2021-02-10T04:53:04.058248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:04.058248+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":20,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"1260","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.641981+0000","last_change":"2021-02-10T04:53:00.343307+0000","last_active":"2021-02-10T04:55:46.641981+0000","last_peered":"2021-02-10T04:55:46.641981+0000","last_clean":"2021-02-10T04:55:46.641981+0000","last_became_active":"2021-02-10T04:51:36.844573+0000","last_became_peered":"2021-02-10T04:51:36.844573+0000","last_unstale":"2021-02-10T04:55:46.641981+0000","last_undegraded":"2021-02-10T04:55:46.641981+0000","last_fullsized":"2021-02-10T04:55:46.641981+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:00.343228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:00.343228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"1229","reported_epoch":"1257","state":"active+clean","last_fresh":"2021-02-10T04:55:47.644566+0000","last_change":"2021-02-10T04:50:56.099710+0000","last_active":"2021-02-10T04:55:47.644566+0000","last_peered":"2021-02-10T04:55:47.644566+0000","last_clean":"2021-02-10T04:55:47.644566+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:55:47.644566+0000","last_undegraded":"2021-02-10T04:55:47.644566+0000","last_fullsized":"2021-02-10T04:55:47.644566+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:56.099651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:50:56.099651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"577","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:49.359964+0000","last_change":"2021-02-10T04:53:46.718715+0000","last_active":"2021-02-10T04:55:49.359964+0000","last_peered":"2021-02-10T04:55:49.359964+0000","last_clean":"2021-02-10T04:55:49.359964+0000","last_became_active":"2021-02-10T04:53:46.718266+0000","last_became_peered":"2021-02-10T04:53:46.718266+0000","last_unstale":"2021-02-10T04:55:49.359964+0000","last_undegraded":"2021-02-10T04:55:49.359964+0000","last_fullsized":"2021-02-10T04:55:49.359964+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.876681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.876681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"998'1009","reported_seq":"5105","reported_epoch":"1257","state":"active+clean","last_fresh":"2021-02-10T04:55:47.645756+0000","last_change":"2021-02-10T04:55:14.635764+0000","last_active":"2021-02-10T04:55:47.645756+0000","last_peered":"2021-02-10T04:55:47.645756+0000","last_clean":"2021-02-10T04:55:47.645756+0000","last_became_active":"2021-02-10T04:55:14.634422+0000","last_became_peered":"2021-02-10T04:55:14.634422+0000","last_unstale":"2021-02-10T04:55:47.645756+0000","last_undegraded":"2021-02-10T04:55:47.645756+0000","last_fullsized":"2021-02-10T04:55:47.645756+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":5,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:13.032506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:13.032506+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49420827,"num_objects":21,"num_object_clones":19,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":36763798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"1049'37","reported_seq":"1285","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.629067+0000","last_change":"2021-02-10T04:51:44.694779+0000","last_active":"2021-02-10T04:55:45.629067+0000","last_peered":"2021-02-10T04:55:45.629067+0000","last_clean":"2021-02-10T04:55:45.629067+0000","last_became_active":"2021-02-10T04:50:59.549412+0000","last_became_peered":"2021-02-10T04:50:59.549412+0000","last_unstale":"2021-02-10T04:55:45.629067+0000","last_undegraded":"2021-02-10T04:55:45.629067+0000","last_fullsized":"2021-02-10T04:55:45.629067+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:44.694711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.694711+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":64,"num_read_kb":50,"num_write":37,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"1396","reported_epoch":"1257","state":"active+clean","last_fresh":"2021-02-10T04:55:47.645955+0000","last_change":"2021-02-10T04:52:33.088525+0000","last_active":"2021-02-10T04:55:47.645955+0000","last_peered":"2021-02-10T04:55:47.645955+0000","last_clean":"2021-02-10T04:55:47.645955+0000","last_became_active":"2021-02-10T04:51:36.870573+0000","last_became_peered":"2021-02-10T04:51:36.870573+0000","last_unstale":"2021-02-10T04:55:47.645955+0000","last_undegraded":"2021-02-10T04:55:47.645955+0000","last_fullsized":"2021-02-10T04:55:47.645955+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:52:33.088428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:33.088428+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"541","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.627860+0000","last_change":"2021-02-10T04:51:51.836723+0000","last_active":"2021-02-10T04:55:45.627860+0000","last_peered":"2021-02-10T04:55:45.627860+0000","last_clean":"2021-02-10T04:55:45.627860+0000","last_became_active":"2021-02-10T04:50:59.564977+0000","last_became_peered":"2021-02-10T04:50:59.564977+0000","last_unstale":"2021-02-10T04:55:45.627860+0000","last_undegraded":"2021-02-10T04:55:45.627860+0000","last_fullsized":"2021-02-10T04:55:45.627860+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.836638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.836638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"1197","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.629942+0000","last_change":"2021-02-10T04:55:14.633046+0000","last_active":"2021-02-10T04:55:45.629942+0000","last_peered":"2021-02-10T04:55:45.629942+0000","last_clean":"2021-02-10T04:55:45.629942+0000","last_became_active":"2021-02-10T04:55:14.632658+0000","last_became_peered":"2021-02-10T04:55:14.632658+0000","last_unstale":"2021-02-10T04:55:45.629942+0000","last_undegraded":"2021-02-10T04:55:45.629942+0000","last_fullsized":"2021-02-10T04:55:45.629942+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:58.514318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_clean_scrub_stamp":"2021-02-10T04:46:58.514318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1d","version":"1001'1165","reported_seq":"4257","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.648125+0000","last_change":"2021-02-10T04:55:14.656236+0000","last_active":"2021-02-10T04:55:46.648125+0000","last_peered":"2021-02-10T04:55:46.648125+0000","last_clean":"2021-02-10T04:55:46.648125+0000","last_became_active":"2021-02-10T04:55:14.655859+0000","last_became_peered":"2021-02-10T04:55:14.655859+0000","last_unstale":"2021-02-10T04:55:46.648125+0000","last_undegraded":"2021-02-10T04:55:46.648125+0000","last_fullsized":"2021-02-10T04:55:46.648125+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":5,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29601845,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":287,"num_read_kb":299530,"num_write":200,"num_write_kb":89131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"1301","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.639668+0000","last_change":"2021-02-10T04:53:01.331566+0000","last_active":"2021-02-10T04:55:46.639668+0000","last_peered":"2021-02-10T04:55:46.639668+0000","last_clean":"2021-02-10T04:55:46.639668+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:55:46.639668+0000","last_undegraded":"2021-02-10T04:55:46.639668+0000","last_fullsized":"2021-02-10T04:55:46.639668+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:01.331466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.331466+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"1098'3","reported_seq":"1597","reported_epoch":"1257","state":"active+clean","last_fresh":"2021-02-10T04:55:48.040110+0000","last_change":"2021-02-10T04:53:34.865668+0000","last_active":"2021-02-10T04:55:48.040110+0000","last_peered":"2021-02-10T04:55:48.040110+0000","last_clean":"2021-02-10T04:55:48.040110+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:55:48.040110+0000","last_undegraded":"2021-02-10T04:55:48.040110+0000","last_fullsized":"2021-02-10T04:55:48.040110+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"1098'3","last_scrub_stamp":"2021-02-10T04:53:34.865579+0000","last_deep_scrub":"1098'3","last_deep_scrub_stamp":"2021-02-10T04:50:15.255137+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.865579+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"1107","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:49.360321+0000","last_change":"2021-02-10T04:53:46.727603+0000","last_active":"2021-02-10T04:55:49.360321+0000","last_peered":"2021-02-10T04:55:49.360321+0000","last_clean":"2021-02-10T04:55:49.360321+0000","last_became_active":"2021-02-10T04:53:46.727053+0000","last_became_peered":"2021-02-10T04:53:46.727053+0000","last_unstale":"2021-02-10T04:55:49.360321+0000","last_undegraded":"2021-02-10T04:55:49.360321+0000","last_fullsized":"2021-02-10T04:55:49.360321+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:47.493057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.493057+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1c","version":"1002'1090","reported_seq":"5281","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.640550+0000","last_change":"2021-02-10T04:53:41.353991+0000","last_active":"2021-02-10T04:55:46.640550+0000","last_peered":"2021-02-10T04:55:46.640550+0000","last_clean":"2021-02-10T04:55:46.640550+0000","last_became_active":"2021-02-10T04:52:31.088265+0000","last_became_peered":"2021-02-10T04:52:31.088265+0000","last_unstale":"2021-02-10T04:55:46.640550+0000","last_undegraded":"2021-02-10T04:55:46.640550+0000","last_fullsized":"2021-02-10T04:55:46.640550+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:53:41.353840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:41.353840+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"1201","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.628953+0000","last_change":"2021-02-10T04:55:14.632817+0000","last_active":"2021-02-10T04:55:45.628953+0000","last_peered":"2021-02-10T04:55:45.628953+0000","last_clean":"2021-02-10T04:55:45.628953+0000","last_became_active":"2021-02-10T04:55:14.632214+0000","last_became_peered":"2021-02-10T04:55:14.632214+0000","last_unstale":"2021-02-10T04:55:45.628953+0000","last_undegraded":"2021-02-10T04:55:45.628953+0000","last_fullsized":"2021-02-10T04:55:45.628953+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:42.537443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:42.537443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"1153","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.651649+0000","last_change":"2021-02-10T04:55:22.725805+0000","last_active":"2021-02-10T04:55:48.651649+0000","last_peered":"2021-02-10T04:55:48.651649+0000","last_clean":"2021-02-10T04:55:48.651649+0000","last_became_active":"2021-02-10T04:55:14.630190+0000","last_became_peered":"2021-02-10T04:55:14.630190+0000","last_unstale":"2021-02-10T04:55:48.651649+0000","last_undegraded":"2021-02-10T04:55:48.651649+0000","last_fullsized":"2021-02-10T04:55:48.651649+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.938731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.938731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"466","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.647891+0000","last_change":"2021-02-10T04:53:15.185748+0000","last_active":"2021-02-10T04:55:46.647891+0000","last_peered":"2021-02-10T04:55:46.647891+0000","last_clean":"2021-02-10T04:55:46.647891+0000","last_became_active":"2021-02-10T04:51:49.499348+0000","last_became_peered":"2021-02-10T04:51:49.499348+0000","last_unstale":"2021-02-10T04:55:46.647891+0000","last_undegraded":"2021-02-10T04:55:46.647891+0000","last_fullsized":"2021-02-10T04:55:46.647891+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:15.185686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:15.185686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1f","version":"1004'1093","reported_seq":"6519","reported_epoch":"1257","state":"active+clean","last_fresh":"2021-02-10T04:55:47.645226+0000","last_change":"2021-02-10T04:52:31.089952+0000","last_active":"2021-02-10T04:55:47.645226+0000","last_peered":"2021-02-10T04:55:47.645226+0000","last_clean":"2021-02-10T04:55:47.645226+0000","last_became_active":"2021-02-10T04:52:31.088634+0000","last_became_peered":"2021-02-10T04:52:31.088634+0000","last_unstale":"2021-02-10T04:55:47.645226+0000","last_undegraded":"2021-02-10T04:55:47.645226+0000","last_fullsized":"2021-02-10T04:55:47.645226+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39434697,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":279,"num_read_kb":311058,"num_write":186,"num_write_kb":66790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":53995913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"1259","reported_epoch":"1257","state":"active+clean","last_fresh":"2021-02-10T04:55:47.646149+0000","last_change":"2021-02-10T04:55:39.651056+0000","last_active":"2021-02-10T04:55:47.646149+0000","last_peered":"2021-02-10T04:55:47.646149+0000","last_clean":"2021-02-10T04:55:47.646149+0000","last_became_active":"2021-02-10T04:55:39.596648+0000","last_became_peered":"2021-02-10T04:55:39.596648+0000","last_unstale":"2021-02-10T04:55:47.646149+0000","last_undegraded":"2021-02-10T04:55:47.646149+0000","last_fullsized":"2021-02-10T04:55:47.646149+0000","mapping_epoch":1248,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:52:36.083784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:36.083784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"488","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.648270+0000","last_change":"2021-02-10T04:55:34.960101+0000","last_active":"2021-02-10T04:55:46.648270+0000","last_peered":"2021-02-10T04:55:46.648270+0000","last_clean":"2021-02-10T04:55:46.648270+0000","last_became_active":"2021-02-10T04:55:14.655649+0000","last_became_peered":"2021-02-10T04:55:14.655649+0000","last_unstale":"2021-02-10T04:55:46.648270+0000","last_undegraded":"2021-02-10T04:55:46.648270+0000","last_fullsized":"2021-02-10T04:55:46.648270+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:08.758129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.758129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"1226","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.915848+0000","last_change":"2021-02-10T04:51:44.799512+0000","last_active":"2021-02-10T04:55:48.915848+0000","last_peered":"2021-02-10T04:55:48.915848+0000","last_clean":"2021-02-10T04:55:48.915848+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:55:48.915848+0000","last_undegraded":"2021-02-10T04:55:48.915848+0000","last_fullsized":"2021-02-10T04:55:48.915848+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:44.799429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.799429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1e","version":"1063'654","reported_seq":"4499","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.649768+0000","last_change":"2021-02-10T04:53:40.135450+0000","last_active":"2021-02-10T04:55:46.649768+0000","last_peered":"2021-02-10T04:55:46.649768+0000","last_clean":"2021-02-10T04:55:46.649768+0000","last_became_active":"2021-02-10T04:52:31.087662+0000","last_became_peered":"2021-02-10T04:52:31.087662+0000","last_unstale":"2021-02-10T04:55:46.649768+0000","last_undegraded":"2021-02-10T04:55:46.649768+0000","last_fullsized":"2021-02-10T04:55:46.649768+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:53:40.135367+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:53:40.135367+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"1207","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.630453+0000","last_change":"2021-02-10T04:51:08.006463+0000","last_active":"2021-02-10T04:55:45.630453+0000","last_peered":"2021-02-10T04:55:45.630453+0000","last_clean":"2021-02-10T04:55:45.630453+0000","last_became_active":"2021-02-10T04:50:59.563657+0000","last_became_peered":"2021-02-10T04:50:59.563657+0000","last_unstale":"2021-02-10T04:55:45.630453+0000","last_undegraded":"2021-02-10T04:55:45.630453+0000","last_fullsized":"2021-02-10T04:55:45.630453+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_clean_scrub_stamp":"2021-02-10T04:51:02.455702+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"1261","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.915173+0000","last_change":"2021-02-10T04:52:33.741801+0000","last_active":"2021-02-10T04:55:48.915173+0000","last_peered":"2021-02-10T04:55:48.915173+0000","last_clean":"2021-02-10T04:55:48.915173+0000","last_became_active":"2021-02-10T04:51:36.860129+0000","last_became_peered":"2021-02-10T04:51:36.860129+0000","last_unstale":"2021-02-10T04:55:48.915173+0000","last_undegraded":"2021-02-10T04:55:48.915173+0000","last_fullsized":"2021-02-10T04:55:48.915173+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:33.741711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:46:58.666264+0000","last_clean_scrub_stamp":"2021-02-10T04:52:33.741711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"1219","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.629256+0000","last_change":"2021-02-10T04:52:25.030419+0000","last_active":"2021-02-10T04:55:45.629256+0000","last_peered":"2021-02-10T04:55:45.629256+0000","last_clean":"2021-02-10T04:55:45.629256+0000","last_became_active":"2021-02-10T04:50:59.539582+0000","last_became_peered":"2021-02-10T04:50:59.539582+0000","last_unstale":"2021-02-10T04:55:45.629256+0000","last_undegraded":"2021-02-10T04:55:45.629256+0000","last_fullsized":"2021-02-10T04:55:45.629256+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:25.030281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.030281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"1003'739","reported_seq":"4006","reported_epoch":"1257","state":"active+clean","last_fresh":"2021-02-10T04:55:47.644197+0000","last_change":"2021-02-10T04:53:01.929261+0000","last_active":"2021-02-10T04:55:47.644197+0000","last_peered":"2021-02-10T04:55:47.644197+0000","last_clean":"2021-02-10T04:55:47.644197+0000","last_became_active":"2021-02-10T04:51:40.511505+0000","last_became_peered":"2021-02-10T04:51:40.511505+0000","last_unstale":"2021-02-10T04:55:47.644197+0000","last_undegraded":"2021-02-10T04:55:47.644197+0000","last_fullsized":"2021-02-10T04:55:47.644197+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:53:01.929169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.929169+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22230846,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":18429397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"1098","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:49.360166+0000","last_change":"2021-02-10T04:53:46.727590+0000","last_active":"2021-02-10T04:55:49.360166+0000","last_peered":"2021-02-10T04:55:49.360166+0000","last_clean":"2021-02-10T04:55:49.360166+0000","last_became_active":"2021-02-10T04:53:46.726584+0000","last_became_peered":"2021-02-10T04:53:46.726584+0000","last_unstale":"2021-02-10T04:55:49.360166+0000","last_undegraded":"2021-02-10T04:55:49.360166+0000","last_fullsized":"2021-02-10T04:55:49.360166+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:26.408326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.408326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"1257","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.640225+0000","last_change":"2021-02-10T04:53:07.324650+0000","last_active":"2021-02-10T04:55:46.640225+0000","last_peered":"2021-02-10T04:55:46.640225+0000","last_clean":"2021-02-10T04:55:46.640225+0000","last_became_active":"2021-02-10T04:50:59.575340+0000","last_became_peered":"2021-02-10T04:50:59.575340+0000","last_unstale":"2021-02-10T04:55:46.640225+0000","last_undegraded":"2021-02-10T04:55:46.640225+0000","last_fullsized":"2021-02-10T04:55:46.640225+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.324564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.324564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"1152","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.650713+0000","last_change":"2021-02-10T04:55:28.098627+0000","last_active":"2021-02-10T04:55:48.650713+0000","last_peered":"2021-02-10T04:55:48.650713+0000","last_clean":"2021-02-10T04:55:48.650713+0000","last_became_active":"2021-02-10T04:55:14.640497+0000","last_became_peered":"2021-02-10T04:55:14.640497+0000","last_unstale":"2021-02-10T04:55:48.650713+0000","last_undegraded":"2021-02-10T04:55:48.650713+0000","last_fullsized":"2021-02-10T04:55:48.650713+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:00.501127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:00.501127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"259'74","reported_seq":"197","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.650861+0000","last_change":"2021-02-10T04:55:28.095458+0000","last_active":"2021-02-10T04:55:48.650861+0000","last_peered":"2021-02-10T04:55:48.650861+0000","last_clean":"2021-02-10T04:55:48.650861+0000","last_became_active":"2021-02-10T04:55:14.624071+0000","last_became_peered":"2021-02-10T04:55:14.624071+0000","last_unstale":"2021-02-10T04:55:48.650861+0000","last_undegraded":"2021-02-10T04:55:48.650861+0000","last_fullsized":"2021-02-10T04:55:48.650861+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":5,"last_scrub":"259'74","last_scrub_stamp":"2021-02-10T04:52:26.597668+0000","last_deep_scrub":"259'74","last_deep_scrub_stamp":"2021-02-10T04:46:49.745104+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.597668+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1218524,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":28228,"num_write":19,"num_write_kb":7022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"20","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.653399+0000","last_change":"2021-02-10T04:55:39.619500+0000","last_active":"2021-02-10T04:55:48.653399+0000","last_peered":"2021-02-10T04:55:48.653399+0000","last_clean":"2021-02-10T04:55:48.653399+0000","last_became_active":"2021-02-10T04:55:39.619062+0000","last_became_peered":"2021-02-10T04:55:39.619062+0000","last_unstale":"2021-02-10T04:55:48.653399+0000","last_undegraded":"2021-02-10T04:55:48.653399+0000","last_fullsized":"2021-02-10T04:55:48.653399+0000","mapping_epoch":1248,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:59.167911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:50:59.167911+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"168","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:49.359665+0000","last_change":"2021-02-10T04:53:46.718512+0000","last_active":"2021-02-10T04:55:49.359665+0000","last_peered":"2021-02-10T04:55:49.359665+0000","last_clean":"2021-02-10T04:55:49.359665+0000","last_became_active":"2021-02-10T04:53:46.716811+0000","last_became_peered":"2021-02-10T04:53:46.716811+0000","last_unstale":"2021-02-10T04:55:49.359665+0000","last_undegraded":"2021-02-10T04:55:49.359665+0000","last_fullsized":"2021-02-10T04:55:49.359665+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:45.813676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.813676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"1246","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.639973+0000","last_change":"2021-02-10T04:53:38.363899+0000","last_active":"2021-02-10T04:55:46.639973+0000","last_peered":"2021-02-10T04:55:46.639973+0000","last_clean":"2021-02-10T04:55:46.639973+0000","last_became_active":"2021-02-10T04:50:59.539256+0000","last_became_peered":"2021-02-10T04:50:59.539256+0000","last_unstale":"2021-02-10T04:55:46.639973+0000","last_undegraded":"2021-02-10T04:55:46.639973+0000","last_fullsized":"2021-02-10T04:55:46.639973+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:38.363825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:09.433793+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.363825+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1b","version":"1003'1689","reported_seq":"4604","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.652793+0000","last_change":"2021-02-10T04:55:15.591337+0000","last_active":"2021-02-10T04:55:48.652793+0000","last_peered":"2021-02-10T04:55:48.652793+0000","last_clean":"2021-02-10T04:55:48.652793+0000","last_became_active":"2021-02-10T04:55:15.590301+0000","last_became_peered":"2021-02-10T04:55:15.590301+0000","last_unstale":"2021-02-10T04:55:48.652793+0000","last_undegraded":"2021-02-10T04:55:48.652793+0000","last_fullsized":"2021-02-10T04:55:48.652793+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:53:10.075879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.075879+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":63260738,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":387,"num_read_kb":409791,"num_write":281,"num_write_kb":110559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":22449943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"1240","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.629150+0000","last_change":"2021-02-10T04:55:14.637968+0000","last_active":"2021-02-10T04:55:45.629150+0000","last_peered":"2021-02-10T04:55:45.629150+0000","last_clean":"2021-02-10T04:55:45.629150+0000","last_became_active":"2021-02-10T04:55:14.636561+0000","last_became_peered":"2021-02-10T04:55:14.636561+0000","last_unstale":"2021-02-10T04:55:45.629150+0000","last_undegraded":"2021-02-10T04:55:45.629150+0000","last_fullsized":"2021-02-10T04:55:45.629150+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:33.543559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:46:33.543559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"916","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.639493+0000","last_change":"2021-02-10T04:53:04.354508+0000","last_active":"2021-02-10T04:55:46.639493+0000","last_peered":"2021-02-10T04:55:46.639493+0000","last_clean":"2021-02-10T04:55:46.639493+0000","last_became_active":"2021-02-10T04:33:05.169159+0000","last_became_peered":"2021-02-10T04:33:05.169159+0000","last_unstale":"2021-02-10T04:55:46.639493+0000","last_undegraded":"2021-02-10T04:55:46.639493+0000","last_fullsized":"2021-02-10T04:55:46.639493+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:04.354409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:04.354409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"1312","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.914787+0000","last_change":"2021-02-10T04:52:36.777554+0000","last_active":"2021-02-10T04:55:48.914787+0000","last_peered":"2021-02-10T04:55:48.914787+0000","last_clean":"2021-02-10T04:55:48.914787+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:55:48.914787+0000","last_undegraded":"2021-02-10T04:55:48.914787+0000","last_fullsized":"2021-02-10T04:55:48.914787+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:36.777485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:52:36.777485+0000","last_clean_scrub_stamp":"2021-02-10T04:52:36.777485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1a","version":"1001'1528","reported_seq":"5869","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.914579+0000","last_change":"2021-02-10T04:52:31.086406+0000","last_active":"2021-02-10T04:55:48.914579+0000","last_peered":"2021-02-10T04:55:48.914579+0000","last_clean":"2021-02-10T04:55:48.914579+0000","last_became_active":"2021-02-10T04:52:31.085959+0000","last_became_peered":"2021-02-10T04:52:31.085959+0000","last_unstale":"2021-02-10T04:55:48.914579+0000","last_undegraded":"2021-02-10T04:55:48.914579+0000","last_fullsized":"2021-02-10T04:55:48.914579+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46944136,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":358,"num_read_kb":351935,"num_write":286,"num_write_kb":111355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":21311893,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"1142","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.653787+0000","last_change":"2021-02-10T04:55:22.726939+0000","last_active":"2021-02-10T04:55:48.653787+0000","last_peered":"2021-02-10T04:55:48.653787+0000","last_clean":"2021-02-10T04:55:48.653787+0000","last_became_active":"2021-02-10T04:55:14.627044+0000","last_became_peered":"2021-02-10T04:55:14.627044+0000","last_unstale":"2021-02-10T04:55:48.653787+0000","last_undegraded":"2021-02-10T04:55:48.653787+0000","last_fullsized":"2021-02-10T04:55:48.653787+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:17.919745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:17.919745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"541","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.627192+0000","last_change":"2021-02-10T04:53:07.092032+0000","last_active":"2021-02-10T04:55:45.627192+0000","last_peered":"2021-02-10T04:55:45.627192+0000","last_clean":"2021-02-10T04:55:45.627192+0000","last_became_active":"2021-02-10T04:50:59.572077+0000","last_became_peered":"2021-02-10T04:50:59.572077+0000","last_unstale":"2021-02-10T04:55:45.627192+0000","last_undegraded":"2021-02-10T04:55:45.627192+0000","last_fullsized":"2021-02-10T04:55:45.627192+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.091949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:19.755718+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.091949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"1218","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.630163+0000","last_change":"2021-02-10T04:51:59.838856+0000","last_active":"2021-02-10T04:55:45.630163+0000","last_peered":"2021-02-10T04:55:45.630163+0000","last_clean":"2021-02-10T04:55:45.630163+0000","last_became_active":"2021-02-10T04:50:59.563156+0000","last_became_peered":"2021-02-10T04:50:59.563156+0000","last_unstale":"2021-02-10T04:55:45.630163+0000","last_undegraded":"2021-02-10T04:55:45.630163+0000","last_fullsized":"2021-02-10T04:55:45.630163+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.838790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.838790+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":"952","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.640464+0000","last_change":"2021-02-10T04:52:30.405170+0000","last_active":"2021-02-10T04:55:46.640464+0000","last_peered":"2021-02-10T04:55:46.640464+0000","last_clean":"2021-02-10T04:55:46.640464+0000","last_became_active":"2021-02-10T04:51:36.852058+0000","last_became_peered":"2021-02-10T04:51:36.852058+0000","last_unstale":"2021-02-10T04:55:46.640464+0000","last_undegraded":"2021-02-10T04:55:46.640464+0000","last_fullsized":"2021-02-10T04:55:46.640464+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.405106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:20.635536+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.405106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.20","version":"0'0","reported_seq":"192","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:49.359393+0000","last_change":"2021-02-10T04:53:46.739888+0000","last_active":"2021-02-10T04:55:49.359393+0000","last_peered":"2021-02-10T04:55:49.359393+0000","last_clean":"2021-02-10T04:55:49.359393+0000","last_became_active":"2021-02-10T04:53:46.739532+0000","last_became_peered":"2021-02-10T04:53:46.739532+0000","last_unstale":"2021-02-10T04:55:49.359393+0000","last_undegraded":"2021-02-10T04:55:49.359393+0000","last_fullsized":"2021-02-10T04:55:49.359393+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:56.449085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:41.375276+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.449085+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":"1185","reported_epoch":"1257","state":"active+clean","last_fresh":"2021-02-10T04:55:47.645074+0000","last_change":"2021-02-10T04:53:35.858634+0000","last_active":"2021-02-10T04:55:47.645074+0000","last_peered":"2021-02-10T04:55:47.645074+0000","last_clean":"2021-02-10T04:55:47.645074+0000","last_became_active":"2021-02-10T04:51:36.869785+0000","last_became_peered":"2021-02-10T04:51:36.869785+0000","last_unstale":"2021-02-10T04:55:47.645074+0000","last_undegraded":"2021-02-10T04:55:47.645074+0000","last_fullsized":"2021-02-10T04:55:47.645074+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.858557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.858557+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":"170","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.650253+0000","last_change":"2021-02-10T04:55:23.586523+0000","last_active":"2021-02-10T04:55:48.650253+0000","last_peered":"2021-02-10T04:55:48.650253+0000","last_clean":"2021-02-10T04:55:48.650253+0000","last_became_active":"2021-02-10T04:55:14.637029+0000","last_became_peered":"2021-02-10T04:55:14.637029+0000","last_unstale":"2021-02-10T04:55:48.650253+0000","last_undegraded":"2021-02-10T04:55:48.650253+0000","last_fullsized":"2021-02-10T04:55:48.650253+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:55:23.586401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:55:23.586401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":"153","reported_epoch":"1256","state":"active+clean","last_fresh":"2021-02-10T04:55:46.647302+0000","last_change":"2021-02-10T04:55:14.635842+0000","last_active":"2021-02-10T04:55:46.647302+0000","last_peered":"2021-02-10T04:55:46.647302+0000","last_clean":"2021-02-10T04:55:46.647302+0000","last_became_active":"2021-02-10T04:55:14.635499+0000","last_became_peered":"2021-02-10T04:55:14.635499+0000","last_unstale":"2021-02-10T04:55:46.647302+0000","last_undegraded":"2021-02-10T04:55:46.647302+0000","last_fullsized":"2021-02-10T04:55:46.647302+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.706372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.706372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":"1267","reported_epoch":"1257","state":"active+clean","last_fresh":"2021-02-10T04:55:47.645875+0000","last_change":"2021-02-10T04:51:57.287452+0000","last_active":"2021-02-10T04:55:47.645875+0000","last_peered":"2021-02-10T04:55:47.645875+0000","last_clean":"2021-02-10T04:55:47.645875+0000","last_became_active":"2021-02-10T04:50:59.566827+0000","last_became_peered":"2021-02-10T04:50:59.566827+0000","last_unstale":"2021-02-10T04:55:47.645875+0000","last_undegraded":"2021-02-10T04:55:47.645875+0000","last_fullsized":"2021-02-10T04:55:47.645875+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.287363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.287363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":"241","reported_epoch":"1257","state":"active+clean","last_fresh":"2021-02-10T04:55:47.645319+0000","last_change":"2021-02-10T04:55:14.659882+0000","last_active":"2021-02-10T04:55:47.645319+0000","last_peered":"2021-02-10T04:55:47.645319+0000","last_clean":"2021-02-10T04:55:47.645319+0000","last_became_active":"2021-02-10T04:55:14.659514+0000","last_became_peered":"2021-02-10T04:55:14.659514+0000","last_unstale":"2021-02-10T04:55:47.645319+0000","last_undegraded":"2021-02-10T04:55:47.645319+0000","last_fullsized":"2021-02-10T04:55:47.645319+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:10.308282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:10.308282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":"1154","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.630530+0000","last_change":"2021-02-10T04:53:09.121379+0000","last_active":"2021-02-10T04:55:45.630530+0000","last_peered":"2021-02-10T04:55:45.630530+0000","last_clean":"2021-02-10T04:55:45.630530+0000","last_became_active":"2021-02-10T04:50:59.561442+0000","last_became_peered":"2021-02-10T04:50:59.561442+0000","last_unstale":"2021-02-10T04:55:45.630530+0000","last_undegraded":"2021-02-10T04:55:45.630530+0000","last_fullsized":"2021-02-10T04:55:45.630530+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:09.121298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.121298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.21","version":"293'77","reported_seq":"826","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.629017+0000","last_change":"2021-02-10T04:52:28.062923+0000","last_active":"2021-02-10T04:55:45.629017+0000","last_peered":"2021-02-10T04:55:45.629017+0000","last_clean":"2021-02-10T04:55:45.629017+0000","last_became_active":"2021-02-10T04:52:28.062463+0000","last_became_peered":"2021-02-10T04:52:28.062463+0000","last_unstale":"2021-02-10T04:55:45.629017+0000","last_undegraded":"2021-02-10T04:55:45.629017+0000","last_fullsized":"2021-02-10T04:55:45.629017+0000","mapping_epoch":1143,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1144,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1886586,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":15163,"num_write":9,"num_write_kb":3316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":"155","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.651163+0000","last_change":"2021-02-10T04:52:32.124955+0000","last_active":"2021-02-10T04:55:48.651163+0000","last_peered":"2021-02-10T04:55:48.651163+0000","last_clean":"2021-02-10T04:55:48.651163+0000","last_became_active":"2021-02-10T04:51:36.867061+0000","last_became_peered":"2021-02-10T04:51:36.867061+0000","last_unstale":"2021-02-10T04:55:48.651163+0000","last_undegraded":"2021-02-10T04:55:48.651163+0000","last_fullsized":"2021-02-10T04:55:48.651163+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:32.124901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:08.271977+0000","last_clean_scrub_stamp":"2021-02-10T04:52:32.124901+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":"1196","reported_epoch":"1257","state":"active+clean","last_fresh":"2021-02-10T04:55:47.644940+0000","last_change":"2021-02-10T04:51:07.182775+0000","last_active":"2021-02-10T04:55:47.644940+0000","last_peered":"2021-02-10T04:55:47.644940+0000","last_clean":"2021-02-10T04:55:47.644940+0000","last_became_active":"2021-02-10T04:28:23.126368+0000","last_became_peered":"2021-02-10T04:28:23.126368+0000","last_unstale":"2021-02-10T04:55:47.644940+0000","last_undegraded":"2021-02-10T04:55:47.644940+0000","last_fullsized":"2021-02-10T04:55:47.644940+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.182678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.182678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.20","version":"1000'1071","reported_seq":"4603","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.649977+0000","last_change":"2021-02-10T04:53:03.915465+0000","last_active":"2021-02-10T04:55:48.649977+0000","last_peered":"2021-02-10T04:55:48.649977+0000","last_clean":"2021-02-10T04:55:48.649977+0000","last_became_active":"2021-02-10T04:52:31.088690+0000","last_became_peered":"2021-02-10T04:52:31.088690+0000","last_unstale":"2021-02-10T04:55:48.649977+0000","last_undegraded":"2021-02-10T04:55:48.649977+0000","last_fullsized":"2021-02-10T04:55:48.649977+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:03.915403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.915403+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16398893,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":"781","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:49.359417+0000","last_change":"2021-02-10T04:53:46.721599+0000","last_active":"2021-02-10T04:55:49.359417+0000","last_peered":"2021-02-10T04:55:49.359417+0000","last_clean":"2021-02-10T04:55:49.359417+0000","last_became_active":"2021-02-10T04:53:46.721340+0000","last_became_peered":"2021-02-10T04:53:46.721340+0000","last_unstale":"2021-02-10T04:55:49.359417+0000","last_undegraded":"2021-02-10T04:55:49.359417+0000","last_fullsized":"2021-02-10T04:55:49.359417+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:28.092288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:07.767250+0000","last_clean_scrub_stamp":"2021-02-10T04:52:28.092288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.25","version":"53'2","reported_seq":"1213","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.915329+0000","last_change":"2021-02-10T04:53:36.103026+0000","last_active":"2021-02-10T04:55:48.915329+0000","last_peered":"2021-02-10T04:55:48.915329+0000","last_clean":"2021-02-10T04:55:48.915329+0000","last_became_active":"2021-02-10T04:28:23.467770+0000","last_became_peered":"2021-02-10T04:28:23.467770+0000","last_unstale":"2021-02-10T04:55:48.915329+0000","last_undegraded":"2021-02-10T04:55:48.915329+0000","last_fullsized":"2021-02-10T04:55:48.915329+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:53:36.102928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:36.102928+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.23","version":"1002'1657","reported_seq":"5128","reported_epoch":"1257","state":"active+clean","last_fresh":"2021-02-10T04:55:47.645889+0000","last_change":"2021-02-10T04:52:31.093874+0000","last_active":"2021-02-10T04:55:47.645889+0000","last_peered":"2021-02-10T04:55:47.645889+0000","last_clean":"2021-02-10T04:55:47.645889+0000","last_became_active":"2021-02-10T04:52:31.092996+0000","last_became_peered":"2021-02-10T04:52:31.092996+0000","last_unstale":"2021-02-10T04:55:47.645889+0000","last_undegraded":"2021-02-10T04:55:47.645889+0000","last_fullsized":"2021-02-10T04:55:47.645889+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21396533,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":212,"num_read_kb":218837,"num_write":159,"num_write_kb":59399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":20479176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":"1303","reported_epoch":"1257","state":"active+clean","last_fresh":"2021-02-10T04:55:47.645673+0000","last_change":"2021-02-10T04:53:12.951612+0000","last_active":"2021-02-10T04:55:47.645673+0000","last_peered":"2021-02-10T04:55:47.645673+0000","last_clean":"2021-02-10T04:55:47.645673+0000","last_became_active":"2021-02-10T04:50:59.566548+0000","last_became_peered":"2021-02-10T04:50:59.566548+0000","last_unstale":"2021-02-10T04:55:47.645673+0000","last_undegraded":"2021-02-10T04:55:47.645673+0000","last_fullsized":"2021-02-10T04:55:47.645673+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.951528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.951528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.26","version":"53'1","reported_seq":"95","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:49.360757+0000","last_change":"2021-02-10T04:53:46.722636+0000","last_active":"2021-02-10T04:55:49.360757+0000","last_peered":"2021-02-10T04:55:49.360757+0000","last_clean":"2021-02-10T04:55:49.360757+0000","last_became_active":"2021-02-10T04:53:46.722227+0000","last_became_peered":"2021-02-10T04:53:46.722227+0000","last_unstale":"2021-02-10T04:55:49.360757+0000","last_undegraded":"2021-02-10T04:55:49.360757+0000","last_fullsized":"2021-02-10T04:55:49.360757+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:53:03.933910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.933910+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":423,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.22","version":"998'1009","reported_seq":"4620","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:48.914630+0000","last_change":"2021-02-10T04:52:31.086563+0000","last_active":"2021-02-10T04:55:48.914630+0000","last_peered":"2021-02-10T04:55:48.914630+0000","last_clean":"2021-02-10T04:55:48.914630+0000","last_became_active":"2021-02-10T04:52:31.085288+0000","last_became_peered":"2021-02-10T04:52:31.085288+0000","last_unstale":"2021-02-10T04:55:48.914630+0000","last_undegraded":"2021-02-10T04:55:48.914630+0000","last_fullsized":"2021-02-10T04:55:48.914630+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12355207,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":120,"num_read_kb":124702,"num_write":77,"num_write_kb":25052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4145029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":"1185","reported_epoch":"1257","state":"active+clean","last_fresh":"2021-02-10T04:55:47.645804+0000","last_change":"2021-02-10T04:53:10.958668+0000","last_active":"2021-02-10T04:55:47.645804+0000","last_peered":"2021-02-10T04:55:47.645804+0000","last_clean":"2021-02-10T04:55:47.645804+0000","last_became_active":"2021-02-10T04:45:49.552026+0000","last_became_peered":"2021-02-10T04:45:49.552026+0000","last_unstale":"2021-02-10T04:55:47.645804+0000","last_undegraded":"2021-02-10T04:55:47.645804+0000","last_fullsized":"2021-02-10T04:55:47.645804+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:10.958579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.958579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":"200","reported_epoch":"1258","state":"active+clean","last_fresh":"2021-02-10T04:55:49.360659+0000","last_change":"2021-02-10T04:53:46.725941+0000","last_active":"2021-02-10T04:55:49.360659+0000","last_peered":"2021-02-10T04:55:49.360659+0000","last_clean":"2021-02-10T04:55:49.360659+0000","last_became_active":"2021-02-10T04:53:46.725587+0000","last_became_peered":"2021-02-10T04:53:46.725587+0000","last_unstale":"2021-02-10T04:55:49.360659+0000","last_undegraded":"2021-02-10T04:55:49.360659+0000","last_fullsized":"2021-02-10T04:55:49.360659+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:41.758684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.758684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":"1192","reported_epoch":"1255","state":"active+clean","last_fresh":"2021-02-10T04:55:45.627523+0000","last_change":"2021-02-10T04:55:39.599193+0000","last_active":"2021-02-10T04:55:45.627523+0000","last_peered":"2021-02-10T04:55:45.627523+0000","last_clean":"2021-02-10T04:55:45.627523+0000","last_became_active":"2021-02-10T04:55:39.598816+0000","last_became_peered":"2021-02-10T04:55:39.598816+0000","last_unstale":"2021-02-10T04:55:45.627523+0000","last_undegraded":"2021-02-10T04:55:45.627523+0000","last_fullsized":"2021-02-10T04:55:45.627523+0000","mapping_epoch":1248,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:11.139602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.139602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":"1211","reported_epoch":"1257","state":"active+clean","last_fresh":"2021-02-10T04:55:47.644434+0000","last_change":"2021-02-10T04:52:24.072005+0000","last_active":"2021-02-10T04:55:47.644434+0000","last_peered":"2021-02-10T04:55:47.644434+0000","last_clean":"2021-02-10T04:55:47.644434+0000","last_became_active":"2021-02-10T04:50:59.527945+0000","last_became_peered":"2021-02-10T04:50:59.527945+0000","last_unstale":"2021-02-10T04:55:47.644434+0000","last_undegraded":"2021-02-10T04:55:47.644434+0000","last_fullsized":"2021-02-10T04:55:47.644434+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:24.071915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_clean_scrub_stamp":"2021-02-10T04:52:24.071915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":972330808,"num_objects":390,"num_object_clones":337,"num_object_copies":1170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":390,"num_whiteouts":10,"num_read":4442,"num_read_kb":4559373,"num_write":3227,"num_write_kb":1247575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":611,"num_bytes_recovered":311961170,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2010624000,"data_stored":2589160348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34722,"ondisk_log_size":34722,"up":108,"acting":108,"num_store_stats":8},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":2368,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":18,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1139,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":5835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":126,"acting":126,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":13,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":667,"num_read_kb":662,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20,"ondisk_log_size":20,"up":39,"acting":39,"num_store_stats":8},{"poolid":4,"num_pg":22,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":2288,"num_read_kb":1761,"num_write":1306,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":17736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":558,"ondisk_log_size":558,"up":66,"acting":66,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35,"ondisk_log_size":35,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":126,"acting":126,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365165,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":191080,"kb_used_data":149152,"kb_used_omap":0,"kb_used_meta":41920,"kb_avail":93554072,"statfs":{"total":95995035648,"available":95799369728,"internally_reserved":0,"allocated":152731648,"data_stored":202353385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42926080},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:27 2021","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":1.2370000000000001,"15min":1.284},"min":{"1min":0.50800000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1220000000000001,"5min":34.299999999999997,"15min":67.277000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.5109999999999999,"15min":1.548},"min":{"1min":0.46300000000000002,"5min":0.438,"15min":0.438},"max":{"1min":2.8690000000000002,"5min":34.652999999999999,"15min":66.933999999999997},"last":0.93200000000000005}]},{"osd":1,"last update":"Wed Feb 10 04:55:42 2021","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.80700000000000005,"15min":0.95899999999999996},"min":{"1min":0.48799999999999999,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.409,"5min":2.254,"15min":10.140000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.85999999999999999,"15min":0.96699999999999997},"min":{"1min":0.63400000000000001,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.647,"5min":1.647,"15min":7.1520000000000001},"last":1.0609999999999999}]},{"osd":2,"last update":"Wed Feb 10 04:55:42 2021","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":1.256,"15min":1.1259999999999999},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.4319999999999999,"5min":16.896999999999998,"15min":16.896999999999998},"last":1.4870000000000001},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.2070000000000001,"15min":1.254},"min":{"1min":0.57999999999999996,"5min":0.496,"15min":0.496},"max":{"1min":4.0620000000000003,"5min":16.792000000000002,"15min":16.792000000000002},"last":0.67400000000000004}]},{"osd":3,"last update":"Wed Feb 10 04:54:59 2021","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":1.1359999999999999,"15min":1.29},"min":{"1min":0.68600000000000005,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.3799999999999999,"5min":4.5890000000000004,"15min":4.5890000000000004},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.2250000000000001,"15min":1.3979999999999999},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.5449999999999999,"5min":4.6609999999999996,"15min":4.6609999999999996},"last":1.8129999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:55:42 2021","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":1.1639999999999999,"15min":1.3899999999999999},"min":{"1min":0.60399999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.399,"5min":2.9729999999999999,"15min":10.781000000000001},"last":0.96999999999999997},{"interface":"front","average":{"1min":1.03,"5min":1.1759999999999999,"15min":1.3460000000000001},"min":{"1min":0.74399999999999999,"5min":0.69699999999999995,"15min":0.69099999999999995},"max":{"1min":1.546,"5min":3.4049999999999998,"15min":10.326000000000001},"last":1.1180000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:55:42 2021","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.349,"15min":1.484},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":4.766,"5min":4.766,"15min":14.972},"last":1.921},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.4450000000000001,"15min":1.6899999999999999},"min":{"1min":0.81000000000000005,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.1549999999999998,"5min":4.4020000000000001,"15min":15.218999999999999},"last":1.1619999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:55:42 2021","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.375,"15min":1.7490000000000001},"min":{"1min":0.92700000000000005,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.5780000000000001,"5min":4.0380000000000003,"15min":16.050000000000001},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.4039999999999999,"15min":1.71},"min":{"1min":0.90300000000000002,"5min":0.876,"15min":0.876},"max":{"1min":1.964,"5min":3.9159999999999999,"15min":15.960000000000001},"last":1.1850000000000001}]}]},{"osd":6,"up_from":43,"seq":184683594097,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":408048,"kb_used_data":342248,"kb_used_omap":0,"kb_used_meta":65792,"kb_avail":93337104,"statfs":{"total":95995035648,"available":95577194496,"internally_reserved":0,"allocated":350461952,"data_stored":443318373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":67371008},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:52:12 2021","interfaces":[{"interface":"back","average":{"1min":4.016,"5min":2.7400000000000002,"15min":1.595},"min":{"1min":0.52000000000000002,"5min":0.48099999999999998,"15min":0.44700000000000001},"max":{"1min":61.768000000000001,"5min":137.60599999999999,"15min":137.60599999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":4.0270000000000001,"5min":2.694,"15min":1.7989999999999999},"min":{"1min":0.504,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":62.683,"5min":137.66,"15min":137.66},"last":1.2110000000000001}]},{"osd":1,"last update":"Wed Feb 10 04:55:20 2021","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":1.0089999999999999,"15min":1.0680000000000001},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.395,"5min":13.18,"15min":13.18},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.77900000000000003,"5min":1.0309999999999999,"15min":1.1020000000000001},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.3440000000000001,"5min":11.041,"15min":11.041},"last":0.91900000000000004}]},{"osd":2,"last update":"Wed Feb 10 04:55:20 2021","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":2.641,"15min":1.712},"min":{"1min":0.59799999999999998,"5min":0.503,"15min":0.503},"max":{"1min":1.865,"5min":163.81700000000001,"15min":163.81700000000001},"last":1.0960000000000001},{"interface":"front","average":{"1min":0.94299999999999995,"5min":2.6720000000000002,"15min":1.742},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.6619999999999999,"5min":164.56800000000001,"15min":164.56800000000001},"last":1.339}]},{"osd":3,"last update":"Wed Feb 10 04:55:07 2021","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":1.117,"15min":1.1850000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.764,"5min":1.764,"15min":1.764},"last":0.95399999999999996},{"interface":"front","average":{"1min":1.014,"5min":1.101,"15min":1.117},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.9430000000000001,"5min":2.363,"15min":2.363},"last":1.1719999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:55:20 2021","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.1850000000000001,"15min":1.369},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.1120000000000001,"5min":7.1459999999999999,"15min":7.1459999999999999},"last":1},{"interface":"front","average":{"1min":0.874,"5min":1.0549999999999999,"15min":1.105},"min":{"1min":0.64000000000000001,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.4770000000000001,"5min":11.997999999999999,"15min":11.997999999999999},"last":0.81499999999999995}]},{"osd":5,"last update":"Wed Feb 10 04:55:20 2021","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.196,"15min":1.595},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.5449999999999999,"5min":12.045,"15min":81.540000000000006},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.246,"5min":1.47,"15min":1.8720000000000001},"min":{"1min":0.875,"5min":0.78700000000000003,"15min":0.78000000000000003},"max":{"1min":2.0910000000000002,"5min":13.401,"15min":81.492999999999995},"last":1.0649999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:55:47 2021","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.161,"15min":1.288},"min":{"1min":0.72899999999999998,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":2.073,"5min":4.0919999999999996,"15min":5.1820000000000004},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.421,"5min":1.4159999999999999,"15min":1.595},"min":{"1min":1.016,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":2.2970000000000002,"5min":5.4109999999999996,"15min":13.315},"last":1.28}]}]},{"osd":1,"up_from":13,"seq":55834575234,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":256792,"kb_used_data":206224,"kb_used_omap":0,"kb_used_meta":50560,"kb_avail":93488360,"statfs":{"total":95995035648,"available":95732080640,"internally_reserved":0,"allocated":211173376,"data_stored":265806564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":51773440},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:48 2021","interfaces":[{"interface":"back","average":{"1min":1.696,"5min":2.6320000000000001,"15min":1.921},"min":{"1min":0.44700000000000001,"5min":0.437,"15min":0.437},"max":{"1min":14.053000000000001,"5min":115.622,"15min":115.622},"last":0.68400000000000005},{"interface":"front","average":{"1min":1.728,"5min":2.585,"15min":1.948},"min":{"1min":0.49199999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":14.109,"5min":115.73099999999999,"15min":115.73099999999999},"last":0.90000000000000002}]},{"osd":2,"last update":"Wed Feb 10 04:55:13 2021","interfaces":[{"interface":"back","average":{"1min":11.944000000000001,"5min":5.5419999999999998,"15min":2.766},"min":{"1min":0.50700000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":162.61199999999999,"5min":162.61199999999999,"15min":162.61199999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":12.112,"5min":5.601,"15min":2.8210000000000002},"min":{"1min":0.498,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":162.762,"5min":162.762,"15min":162.762},"last":0.66700000000000004}]},{"osd":3,"last update":"Wed Feb 10 04:55:03 2021","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.79600000000000004,"15min":0.91100000000000003},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.6919999999999999,"5min":3.077,"15min":3.077},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.91200000000000003,"15min":1.0049999999999999},"min":{"1min":0.54600000000000004,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.052,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.84399999999999997}]},{"osd":4,"last update":"Wed Feb 10 04:54:51 2021","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98299999999999998,"15min":1.149},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.6080000000000001,"5min":2.9670000000000001,"15min":4.899},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.93700000000000006,"5min":1.073,"15min":1.302},"min":{"1min":0.73499999999999999,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.268,"5min":2.5270000000000001,"15min":29.547999999999998},"last":0.98499999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:54:59 2021","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":1.05,"15min":1.1579999999999999},"min":{"1min":0.66800000000000004,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.7949999999999999,"5min":2.7959999999999998,"15min":5.0300000000000002},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.21,"5min":1.1879999999999999,"15min":1.488},"min":{"1min":0.70399999999999996,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.879,"5min":2.8199999999999998,"15min":28.678000000000001},"last":0.81699999999999995}]},{"osd":6,"last update":"Wed Feb 10 04:55:13 2021","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.3660000000000001,"15min":1.538},"min":{"1min":0.68999999999999995,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.4099999999999999,"5min":30.997,"15min":30.997},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.071,"5min":1.3839999999999999,"15min":1.4590000000000001},"min":{"1min":0.80600000000000005,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.4790000000000001,"5min":31.138000000000002,"15min":31.138000000000002},"last":1.2050000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:55:27 2021","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.216,"15min":1.488},"min":{"1min":0.72299999999999998,"5min":0.628,"15min":0.628},"max":{"1min":6.2629999999999999,"5min":6.2629999999999999,"15min":28.573},"last":1.042},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.232,"15min":1.5169999999999999},"min":{"1min":0.89200000000000002,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.895,"5min":1.895,"15min":29.408999999999999},"last":1.127}]}]},{"osd":0,"up_from":1227,"seq":5269924872201,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":348572,"kb_used_data":338452,"kb_used_omap":306,"kb_used_meta":9805,"kb_avail":93396580,"statfs":{"total":95995035648,"available":95638097920,"internally_reserved":0,"allocated":346574848,"data_stored":471449276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":313579,"internal_metadata":10041109},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5189999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.302}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.546}]}]},{"osd":2,"up_from":18,"seq":77309411713,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":353896,"kb_used_data":318432,"kb_used_omap":0,"kb_used_meta":35456,"kb_avail":93391256,"statfs":{"total":95995035648,"available":95632646144,"internally_reserved":0,"allocated":326074368,"data_stored":374848424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36306944},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:50 2021","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.99099999999999999,"15min":1.718},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":3.698,"5min":3.996,"15min":163.15199999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.91300000000000003,"15min":1.655},"min":{"1min":0.51100000000000001,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.9710000000000001,"5min":2.2349999999999999,"15min":162.959},"last":2.2789999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:54:54 2021","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.77200000000000002,"15min":0.98299999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.93200000000000005,"5min":3.8159999999999998,"15min":4.9690000000000003},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.67300000000000004,"15min":0.876},"min":{"1min":0.442,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.93200000000000005,"5min":1.2010000000000001,"15min":5.726},"last":2.3479999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:55:08 2021","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.92200000000000004,"15min":1.0840000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.83799999999999997,"5min":4.2380000000000004,"15min":4.2380000000000004},"last":2.3159999999999998},{"interface":"front","average":{"1min":0.93100000000000005,"5min":1.133,"15min":1.2649999999999999},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.1259999999999999,"5min":4.54,"15min":4.54},"last":2.3879999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:55:36 2021","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.1379999999999999,"15min":1.2849999999999999},"min":{"1min":0.71499999999999997,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":9.2029999999999994,"5min":9.2029999999999994,"15min":9.2029999999999994},"last":2.4630000000000001},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.228,"15min":1.1699999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":3.403,"5min":24.373999999999999,"15min":24.373999999999999},"last":2.4100000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:54:54 2021","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.625,"15min":1.6719999999999999},"min":{"1min":0.66200000000000003,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.605,"5min":33.018999999999998,"15min":33.018999999999998},"last":2.54},{"interface":"front","average":{"1min":0.91000000000000003,"5min":1.448,"15min":1.4399999999999999},"min":{"1min":0.68000000000000005,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.716,"5min":33.043999999999997,"15min":33.043999999999997},"last":2.198}]},{"osd":6,"last update":"Wed Feb 10 04:55:23 2021","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.0629999999999999,"15min":1.419},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":3.1659999999999999,"5min":3.1659999999999999,"15min":20.007000000000001},"last":2.3010000000000002},{"interface":"front","average":{"1min":1.9530000000000001,"5min":1.423,"15min":1.6519999999999999},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":16.135000000000002,"5min":16.135000000000002,"15min":19.417999999999999},"last":2.6640000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:55:36 2021","interfaces":[{"interface":"back","average":{"1min":1.637,"5min":1.637,"15min":1.681},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":9.3460000000000001,"5min":24.405000000000001,"15min":24.405000000000001},"last":2.6059999999999999},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.1040000000000001,"15min":1.278},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":3.6640000000000001,"5min":3.6640000000000001,"15min":4.9189999999999996},"last":2.524}]}]},{"osd":3,"up_from":1111,"seq":4771708665915,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":328280,"kb_used_data":301136,"kb_used_omap":386,"kb_used_meta":26749,"kb_avail":93416872,"statfs":{"total":95995035648,"available":95658876928,"internally_reserved":0,"allocated":308363264,"data_stored":409742359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":396036,"internal_metadata":27391228},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:58 2021","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":0.628}]},{"osd":1,"last update":"Wed Feb 10 04:55:04 2021","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.63500000000000001,"15min":0.64400000000000002},"min":{"1min":0.46999999999999997,"5min":0.44,"15min":0.44},"max":{"1min":0.69399999999999995,"5min":1.8360000000000001,"15min":1.8360000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.79100000000000004,"15min":0.85899999999999999},"min":{"1min":0.46600000000000003,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.534,"5min":2.137,"15min":2.137},"last":0.78500000000000003}]},{"osd":2,"last update":"Wed Feb 10 04:55:04 2021","interfaces":[{"interface":"back","average":{"1min":2.512,"5min":1.48,"15min":1.26},"min":{"1min":0.48699999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":31.100999999999999,"5min":31.100999999999999,"15min":31.100999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":2.637,"5min":1.5700000000000001,"15min":1.27},"min":{"1min":0.53800000000000003,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":31.052,"5min":31.052,"15min":31.052},"last":1.0069999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:55:04 2021","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.93799999999999994,"15min":0.95999999999999996},"min":{"1min":0.66900000000000004,"5min":0.628,"15min":0.628},"max":{"1min":1.502,"5min":2.9430000000000001,"15min":2.9430000000000001},"last":1.2090000000000001},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.997,"15min":1.085},"min":{"1min":0.65300000000000002,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.8580000000000001,"5min":1.8999999999999999,"15min":1.8999999999999999},"last":0.91900000000000004}]},{"osd":5,"last update":"Wed Feb 10 04:55:04 2021","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":2.2839999999999998,"15min":2.593},"min":{"1min":0.76000000000000001,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.462,"5min":52.353000000000002,"15min":52.353000000000002},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.97099999999999997,"5min":2.1789999999999998,"15min":2.4209999999999998},"min":{"1min":0.70399999999999996,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":2.1000000000000001,"5min":52.301000000000002,"15min":52.301000000000002},"last":1.552}]},{"osd":6,"last update":"Wed Feb 10 04:55:04 2021","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":3.4790000000000001,"15min":4.9770000000000003},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.581,"5min":64.376999999999995,"15min":64.376999999999995},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.111,"5min":3.5499999999999998,"15min":4.9729999999999999},"min":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":1.7130000000000001,"5min":64.438999999999993,"15min":64.438999999999993},"last":1.319}]},{"osd":7,"last update":"Wed Feb 10 04:55:04 2021","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.0669999999999999,"15min":1.1060000000000001},"min":{"1min":0.77500000000000002,"5min":0.754,"15min":0.754},"max":{"1min":1.3879999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.115,"15min":1.1339999999999999},"min":{"1min":0.86399999999999999,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.6140000000000001,"5min":2.044,"15min":2.044},"last":1.456}]}]},{"osd":4,"up_from":31,"seq":133143986553,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":160296,"kb_used_data":120416,"kb_used_omap":0,"kb_used_meta":39872,"kb_avail":93584856,"statfs":{"total":95995035648,"available":95830892544,"internally_reserved":0,"allocated":123305984,"data_stored":155671431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40828928},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:36 2021","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.0899999999999999,"15min":1.593},"min":{"1min":0.45800000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.7799999999999998,"5min":95.721999999999994,"15min":95.721999999999994},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.67700000000000005,"5min":2.0870000000000002,"15min":1.6000000000000001},"min":{"1min":0.46000000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.002,"5min":95.344999999999999,"15min":95.344999999999999},"last":1.1279999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:55:46 2021","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.76100000000000001,"15min":0.871},"min":{"1min":0.46500000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.341,"5min":3.5529999999999999,"15min":7.883},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.89700000000000002,"15min":1.0580000000000001},"min":{"1min":0.50700000000000001,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.1850000000000001,"5min":3.8090000000000002,"15min":6.5060000000000002},"last":1.099}]},{"osd":2,"last update":"Wed Feb 10 04:55:46 2021","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":2.6829999999999998,"15min":1.702},"min":{"1min":0.627,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":7.0069999999999997,"5min":120.39,"15min":120.39},"last":0.98199999999999998},{"interface":"front","average":{"1min":1.218,"5min":2.6419999999999999,"15min":1.6990000000000001},"min":{"1min":0.58999999999999997,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":7.1120000000000001,"5min":120.583,"15min":120.583},"last":1.294}]},{"osd":3,"last update":"Wed Feb 10 04:55:02 2021","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":1.139,"15min":1.2509999999999999},"min":{"1min":0.69899999999999995,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.542,"5min":2.1429999999999998,"15min":2.1429999999999998},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.659,"15min":1.573},"min":{"1min":0.81599999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.647,"5min":26.802,"15min":26.802},"last":1.3600000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:54:56 2021","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":1.1799999999999999,"15min":1.494},"min":{"1min":0.59399999999999997,"5min":0.495,"15min":0.495},"max":{"1min":1.722,"5min":8.1509999999999998,"15min":35.774000000000001},"last":1.419},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.226,"15min":1.464},"min":{"1min":0.66700000000000004,"5min":0.60899999999999999,"15min":0.58799999999999997},"max":{"1min":3.1389999999999998,"5min":7.9950000000000001,"15min":35.606000000000002},"last":1.552}]},{"osd":6,"last update":"Wed Feb 10 04:55:02 2021","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.597,"15min":2.3719999999999999},"min":{"1min":0.89400000000000002,"5min":0.86199999999999999,"15min":0.76600000000000001},"max":{"1min":2.6989999999999998,"5min":12.488,"15min":201.626},"last":1.464},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.476,"15min":2.3940000000000001},"min":{"1min":0.75800000000000001,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":2.625,"5min":12.84,"15min":201.577},"last":1.528}]},{"osd":7,"last update":"Wed Feb 10 04:55:32 2021","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.431,"15min":1.4990000000000001},"min":{"1min":0.90300000000000002,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.351,"5min":2.5099999999999998,"15min":5.8920000000000003},"last":1.5840000000000001},{"interface":"front","average":{"1min":1.105,"5min":1.2410000000000001,"15min":1.496},"min":{"1min":0.79100000000000004,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.819,"5min":2.2250000000000001,"15min":5.149},"last":1.633}]}]},{"osd":5,"up_from":37,"seq":158913790325,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":251792,"kb_used_data":206152,"kb_used_omap":0,"kb_used_meta":45632,"kb_avail":93493360,"statfs":{"total":95995035648,"available":95737200640,"internally_reserved":0,"allocated":211099648,"data_stored":275724421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":46727168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:51 2021","interfaces":[{"interface":"back","average":{"1min":5.0590000000000002,"5min":1.714,"15min":1.204},"min":{"1min":0.45700000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":58.624000000000002,"5min":58.624000000000002,"15min":58.624000000000002},"last":0.56299999999999994},{"interface":"front","average":{"1min":5.04,"5min":1.784,"15min":1.3009999999999999},"min":{"1min":0.56599999999999995,"5min":0.46600000000000003,"15min":0.46400000000000002},"max":{"1min":57.880000000000003,"5min":57.880000000000003,"15min":57.880000000000003},"last":0.59299999999999997}]},{"osd":1,"last update":"Wed Feb 10 04:54:54 2021","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.91300000000000003,"15min":0.94499999999999995},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.1279999999999999,"5min":6.391,"15min":6.391},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.69599999999999995,"5min":1.1020000000000001,"15min":1.1479999999999999},"min":{"1min":0.53600000000000003,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.206,"5min":11.818,"15min":11.875999999999999},"last":0.626}]},{"osd":2,"last update":"Wed Feb 10 04:54:54 2021","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":1.5980000000000001,"15min":1.52},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":2.129,"5min":50.948999999999998,"15min":50.948999999999998},"last":47.576999999999998},{"interface":"front","average":{"1min":0.84699999999999998,"5min":1.7050000000000001,"15min":1.5509999999999999},"min":{"1min":0.56200000000000006,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.212,"5min":50.149000000000001,"15min":50.149000000000001},"last":47.944000000000003}]},{"osd":3,"last update":"Wed Feb 10 04:55:09 2021","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.1759999999999999,"15min":1.282},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":4.7000000000000002,"5min":4.7000000000000002,"15min":4.7000000000000002},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.238,"15min":1.2},"min":{"1min":0.83099999999999996,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":5.577,"5min":6.5129999999999999,"15min":6.5129999999999999},"last":0.96099999999999997}]},{"osd":4,"last update":"Wed Feb 10 04:54:54 2021","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":1.272,"15min":1.29},"min":{"1min":0.67500000000000004,"5min":0.64500000000000002,"15min":0.57099999999999995},"max":{"1min":1.274,"5min":9.4890000000000008,"15min":9.4890000000000008},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.2849999999999999,"15min":1.359},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":2.3079999999999998,"5min":5.6180000000000003,"15min":5.6180000000000003},"last":1.0129999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:55:09 2021","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":1.4970000000000001,"15min":2.996},"min":{"1min":0.88200000000000001,"5min":0.83899999999999997,"15min":0.65800000000000003},"max":{"1min":4.9429999999999996,"5min":5.8879999999999999,"15min":319.51999999999998},"last":1.357},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.5029999999999999,"15min":2.931},"min":{"1min":0.95199999999999996,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":5.4660000000000002,"5min":6.8220000000000001,"15min":319.22300000000001},"last":1.0920000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:55:35 2021","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.4870000000000001,"15min":1.486},"min":{"1min":0.90300000000000002,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":5.4160000000000004,"5min":11.897,"15min":11.897},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.675,"5min":1.524,"15min":1.5720000000000001},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.82999999999999996},"max":{"1min":5.0579999999999998,"5min":6.0369999999999999,"15min":6.0369999999999999},"last":1.1619999999999999}]}]}]}} 2021-02-10T04:55:55.555 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 1, currently 11 2021-02-10T04:55:55.730 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:55 smithi085 conmon[29319]: cluster 2021-02-10T04:55:54.505790+0000 mgr.y (mgr.14140) 1384 : cluster [DBG] pgmap v2277: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-02-10T04:55:55.731 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:55 smithi085 conmon[29319]: cluster 2021-02-10T04:55:54.649139+0000 osd.4 (osd.4) 2657 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:55:55.731 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:55 smithi085 conmon[29319]: cluster 2021-02-10T04:55:54.650942+0000 osd.4 (osd.4) 2658 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:55:55.731 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:55 smithi085 conmon[29319]: cluster 2021-02-10T04:55:54.672413+0000 mon.a (mon.0) 2712 : cluster [DBG] osdmap e1260: 8 total, 8 up, 8 in 2021-02-10T04:55:55.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:55 smithi085 conmon[35961]: cluster 2021-02-10T04:55:54.505790+0000 mgr.y (mgr.14140) 1384 : cluster [DBG] pgmap v2277: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-02-10T04:55:55.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:55 smithi085 conmon[35961]: cluster 2021-02-10T04:55:54.649139+0000 osd.4 (osd.4) 2657 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:55:55.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:55 smithi085 conmon[35961]: cluster 2021-02-10T04:55:54.650942+0000 osd.4 (osd.4) 2658 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:55:55.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:55 smithi085 conmon[35961]: cluster 2021-02-10T04:55:54.672413+0000 mon.a (mon.0) 2712 : cluster [DBG] osdmap e1260: 8 total, 8 up, 8 in 2021-02-10T04:55:56.113 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:55:56.138 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:55:56.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:55 smithi164 conmon[29288]: cluster 2021-02-10T04:55:54.505790+0000 mgr.y (mgr.14140) 1384 : cluster 2021-02-10T04:55:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:55 smithi164 conmon[29288]: [DBG] pgmap v2277: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-02-10T04:55:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:55 smithi164 conmon[29288]: cluster 2021-02-10T04:55:54.649139+0000 osd.4 (osd.4) 2657 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:55:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:55 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:55:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:55 smithi164 conmon[29288]: T04:55:54.650942+0000 osd.4 (osd.4) 2658 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:55:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:55 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:55:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:55 smithi164 conmon[29288]: 10T04:55:54.672413+0000 mon.a (mon.0 2021-02-10T04:55:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:55 smithi164 conmon[29288]: ) 2712 : cluster [DBG] osdmap e1260: 8 total, 8 up, 8 in 2021-02-10T04:55:56.163 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:55:56.182 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:55:56.191 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:55:55 smithi085 conmon[49106]: debug 2021-02-10T04:55:55.728+0000 7f7788e19700 -1 osd.2 pg_epoch: 1260 pg[5.8( empty local-lis/les=1227/1228 n=0 ec=57/57 lis/c=1227/1227 les/c/f=1228/1228/0 sis=1227) [2,0,3] r=0 lpr=1227 crt=0'0 mlcod 0'0 active+clean] scrubber pg(5.8) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x564aa70eb340) 2021-02-10T04:55:56.209 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:55:56.235 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:55:56.262 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:55:56.288 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:55:56.289 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:55:56.289 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:55:56.289 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:55:56.289 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:55:56.289 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:55:56.289 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:55:56.289 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:55:56.289 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:55:56.689 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is unset 2021-02-10T04:55:56.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:56 smithi085 conmon[29319]: audit 2021-02-10T04:55: 2021-02-10T04:55:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:56 smithi085 conmon[29319]: 56.093567+0000 mon.c (mon.1) 128 : audit [DBG] from='client.? 172.21.15.85:0/3301895667' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:55:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:56 smithi085 conmon[29319]: audit 2021-02-10T04:55:56.284846+0000 mon.a (mon.0) 2713 : audit [INF] from='client.? 172.21.15.85:0/1567988209' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:55:56.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:56 smithi085 conmon[35961]: audit 2021-02-10T04:55:56.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:56 smithi085 conmon[35961]: 2021-02-10T04:55 2021-02-10T04:55:56.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:56 smithi085 conmon[35961]: :56.093567+0000 mon.c (mon.1) 128 : audit [DBG] from='client.? 172.21.15.85:0/3301895667' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:55:56.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:56 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:55:56.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:56 smithi085 conmon[35961]: -10T04:55:56.284846+0000 mon.a (mon 2021-02-10T04:55:56.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:56 smithi085 conmon[35961]: .0) 2713 : audit [INF] from='client.? 172.21.15.85:0/1567988209' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:55:57.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:56 smithi164 conmon[29288]: audit 2021-02-10T04:55:56.093567+0000 mon.c (mon.1) 128 : audit 2021-02-10T04:55:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:56 smithi164 conmon[29288]: [DBG] from='client.? 172.21.15.85:0/3301895667' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:55:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:56 smithi164 conmon[29288]: audit 2021-02-10T04:55:56.284846+0000 mon.a (mon.0) 2713 : audit [INF] from='client.? 172.21.15.85:0/1567988209' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:55:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:57 smithi085 conmon[35961]: cluster 2021-02-10T04:55:56 2021-02-10T04:55:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:57 smithi085 conmon[35961]: .506395+0000 mgr.y (mgr.14140) 1385 : cluster [DBG] pgmap v2279: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 651 B/s rd, 0 op/s 2021-02-10T04:55:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:57 smithi085 conmon[35961]: cluster 2021-02-10T04:55:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:57 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:55:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:57 smithi085 conmon[35961]: 04:55:56.669524+0000 2021-02-10T04:55:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:57 smithi085 conmon[35961]: mon.a (mon.0) 2714 : cluster 2021-02-10T04:55:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:57 smithi085 conmon[35961]: [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:55:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:57 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:55:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:57 smithi085 conmon[35961]: 55:56.669559+0000 mon.a (mon.0) 2715 : cluster [INF] Cluster is now healthy 2021-02-10T04:55:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:57 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:55:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:57 smithi085 conmon[35961]: 10T04:55:56.676710+0000 mon.a (mon.0) 2021-02-10T04:55:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:57 smithi085 conmon[35961]: 2716 : audit [INF] from='client.? 172.21.15.85:0/1567988209' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:55:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:57 smithi085 conmon[35961]: cluster 2021-02-10T04:55:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:57 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:55:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:57 smithi085 conmon[35961]: T04:55:56.676774+0000 mon.a (mon.0) 2717 2021-02-10T04:55:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:57 smithi085 conmon[35961]: : cluster [DBG] osdmap e1261: 8 total, 8 up, 8 in 2021-02-10T04:55:57.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:57 smithi085 conmon[29319]: cluster 2021-02-10T04:55:56.506395+0000 mgr.y (mgr 2021-02-10T04:55:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:57 smithi085 conmon[29319]: .14140) 1385 : cluster [DBG] pgmap v2279: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 651 B/s rd, 0 op/s 2021-02-10T04:55:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:57 smithi085 conmon[29319]: cluster 2021-02-10T04:55:56.669524+0000 mon.a (mon.0) 2714 2021-02-10T04:55:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:57 smithi085 conmon[29319]: : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:55:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:57 smithi085 conmon[29319]: cluster 2021-02-10T04:55:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:57 smithi085 conmon[29319]: 2021-02-10T04:55:56.669559+0000 2021-02-10T04:55:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:57 smithi085 conmon[29319]: mon.a (mon.0) 2715 : cluster [INF] Cluster is now healthy 2021-02-10T04:55:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:57 smithi085 conmon[29319]: 2021-02-10T04:55:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:57 smithi085 conmon[29319]: audit 2021-02-10T04:55:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:57 smithi085 conmon[29319]: 2021-02-10T04:55:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:57 smithi085 conmon[29319]: 2021-02-10T04:55:56.676710+0000 2021-02-10T04:55:57.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:57 smithi085 conmon[29319]: mon.a (mon.0) 2716 : audit [INF] from='client.? 172.21.15.85:0/1567988209' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:55:57.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:57 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:55:57.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:57 smithi085 conmon[29319]: -02-10T04:55:56.676774+0000 mon.a (mon.0) 2717 : cluster [DBG] osdmap e1261: 8 total, 8 up, 8 in 2021-02-10T04:55:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:57 smithi164 conmon[29288]: cluster 2021-02-10T04:55:56.506395+0000 mgr.y (mgr.14140) 1385 2021-02-10T04:55:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:57 smithi164 conmon[29288]: : cluster [DBG] pgmap v2279: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 651 B/s rd, 0 op/s 2021-02-10T04:55:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:57 smithi164 conmon[29288]: cluster 2021-02-10T04:55:56.669524+0000 mon.a (mon.0) 2714 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:55:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:57 smithi164 conmon[29288]: cluster 2021-02-10T04:55:56.669559+0000 mon.a (mon 2021-02-10T04:55:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:57 smithi164 conmon[29288]: .0) 2715 : cluster [INF] Cluster is now healthy 2021-02-10T04:55:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:57 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:55:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:57 smithi164 conmon[29288]: 04:55:56.676710+0000 mon.a (mon.0) 2716 : audit 2021-02-10T04:55:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:57 smithi164 conmon[29288]: [INF] from='client.? 172.21.15.85:0/1567988209' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:55:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:57 smithi164 conmon[29288]: cluster 2021-02-10T04:55:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:57 smithi164 conmon[29288]: 2021-02-10T04:55:56.676774+0000 mon.a (mon.0) 2021-02-10T04:55:58.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:57 smithi164 conmon[29288]: 2717 : cluster [DBG] osdmap e1261: 8 total, 8 up, 8 in 2021-02-10T04:55:59.018 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.6 dump_blocked_ops 2021-02-10T04:55:59.284 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:55:59.644 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:55:59 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:55:59.900 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:55:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:59 smithi085 conmon[29319]: cluster 2021-02-10T04:55:58.507132+0000 mgr.y (mgr.14140) 2021-02-10T04:55:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:59 smithi085 conmon[29319]: 1386 : cluster [DBG] pgmap v2281: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s 2021-02-10T04:55:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:59 smithi085 conmon[29319]: cluster 2021-02-10T04:55:59.461614+0000 osd.3 (osd.3) 38 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:55:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:59 smithi085 conmon[29319]: 2021-02-10T04:55:59.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:59 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:55:59.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:59 smithi085 conmon[29319]: :55:59.465749+0000 osd.3 (osd.3) 39 : cluster 2021-02-10T04:55:59.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:55:59 smithi085 conmon[29319]: [DBG] purged_snaps scrub ok 2021-02-10T04:55:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:59 smithi085 conmon[35961]: cluster 2021-02-10T04:55:58.507132 2021-02-10T04:55:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:59 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1386 : cluster [DBG] pgmap v2281: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s 2021-02-10T04:55:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:59 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:55:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:59 smithi085 conmon[35961]: -02-10T04:55:59.461614+0000 osd.3 (osd.3) 38 : cluster [DBG] 2021-02-10T04:55:59.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:59 smithi085 conmon[35961]: purged_snaps scrub starts 2021-02-10T04:55:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:59 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:55:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:59 smithi085 conmon[35961]: :55:59.465749+0000 osd.3 (osd.3 2021-02-10T04:55:59.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:55:59 smithi085 conmon[35961]: ) 39 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:55:59.957 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set noscrub 2021-02-10T04:56:00.126 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:56:00.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:59 smithi164 conmon[29288]: cluster 2021-02-10T04:55:58.507132+0000 mgr.y (mgr.14140) 1386 : cluster 2021-02-10T04:56:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:59 smithi164 conmon[29288]: [DBG] pgmap v2281: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s 2021-02-10T04:56:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:59 smithi164 conmon[29288]: cluster 2021-02-10T04:55:59.461614+0000 osd.3 (osd.3) 38 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:56:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:59 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:56:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:55:59 smithi164 conmon[29288]: 02-10T04:55:59.465749+0000 osd.3 (osd.3) 39 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:56:00.199 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:56:00.555 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2021-02-10T04:56:00.556 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:56:00.556 INFO:tasks.thrashosds.thrasher:fixing pg num pool device_health_metrics 2021-02-10T04:56:00.556 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:56:00.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:00 smithi085 conmon[29319]: audit 2021-02-10T04:55:59.344991+0000 mgr.y (mgr.14140) 1387 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:56:00.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:00 smithi085 conmon[29319]: cluster 2021-02-10T04:55:59.626124+0000 osd.1 (osd.1) 507 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:56:00.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:00 smithi085 conmon[29319]: cluster 2021-02-10T04:55:59.628028+0000 osd.1 (osd.1) 508 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:56:00.694 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:00 smithi085 conmon[29319]: cluster 2021-02-10T04:55:59.680526+0000 mon.a (mon.0) 2718 : cluster [DBG] osdmap e1262: 8 total, 8 up, 8 in 2021-02-10T04:56:00.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:00 smithi085 conmon[35961]: audit 2021-02-10T04:55:59.344991+0000 mgr.y (mgr.14140) 1387 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:56:00.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:00 smithi085 conmon[35961]: cluster 2021-02-10T04:55:59.626124+0000 osd.1 (osd.1) 507 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:56:00.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:00 smithi085 conmon[35961]: cluster 2021-02-10T04:55:59.628028+0000 osd.1 (osd.1) 508 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:56:00.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:00 smithi085 conmon[35961]: cluster 2021-02-10T04:55:59.680526+0000 mon.a (mon.0) 2718 : cluster [DBG] osdmap e1262: 8 total, 8 up, 8 in 2021-02-10T04:56:00.776 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:56:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:00 smithi164 conmon[29288]: audit 2021-02-10T04:55:59.344991+0000 mgr.y (mgr.14140 2021-02-10T04:56:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:00 smithi164 conmon[29288]: ) 1387 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:56:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:00 smithi164 conmon[29288]: cluster 2021-02-10T04:55:59.626124+0000 osd.1 (osd.1) 507 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:56:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:00 smithi164 conmon[29288]: cluster 2021-02-10T04:55:59.628028+0000 osd.1 (osd.1) 508 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:56:00.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:00 smithi164 conmon[29288]: cluster 2021-02-10T04:55:59.680526+0000 mon.a (mon.0) 2718 : cluster [DBG] osdmap e1262: 8 total, 8 up, 8 in 2021-02-10T04:56:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:01 smithi164 conmon[29288]: cluster 2021-02-10T04:56:00.507566+0000 mgr.y (mgr.14140) 1388 : 2021-02-10T04:56:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:01 smithi164 conmon[29288]: cluster [DBG] pgmap v2283: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2021-02-10T04:56:02.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:01 smithi085 conmon[29319]: cluster 2021-02-10T04:56:00. 2021-02-10T04:56:02.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:01 smithi085 conmon[29319]: 507566+0000 mgr.y (mgr.14140) 1388 : cluster [DBG] pgmap v2283: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2021-02-10T04:56:02.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:01 smithi085 conmon[35961]: cluster 2021-02-10T04:56:00.507566+0000 mgr.y (mgr.14140) 1388 2021-02-10T04:56:02.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:01 smithi085 conmon[35961]: : cluster [DBG] pgmap v2283: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2021-02-10T04:56:02.692 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:56:02 smithi085 conmon[49106]: debug 2021-02-10T04:56:02.614+0000 7f7788e19700 -1 osd.2 pg_epoch: 1262 pg[5.8( empty local-lis/les=1227/1228 n=0 ec=57/57 lis/c=1227/1227 les/c/f=1228/1228/0 sis=1227) [2,0,3] r=0 lpr=1227 crt=0'0 mlcod 0'0 active+clean] scrubber pg(5.8) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x564aa6f9f080) 2021-02-10T04:56:03.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:03 smithi164 conmon[29288]: cluster 2021-02-10T04:56:02.508123+0000 mgr.y (mgr.14140) 1389 : cluster [DBG] pgmap v2284: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2021-02-10T04:56:03.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:03 smithi164 conmon[29288]: cluster 2021-02-10T04:56:03.572572+0000 mon.a (mon.0) 2719 : cluster [DBG] osdmap e1263: 8 total, 8 up, 8 in 2021-02-10T04:56:04.123 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:03 smithi085 conmon[35961]: cluster 2021-02-10T04:56 2021-02-10T04:56:04.123 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:03 smithi085 conmon[35961]: :02.508123+0000 mgr.y (mgr.14140) 1389 : cluster [DBG] pgmap v2284: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2021-02-10T04:56:04.123 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:03 smithi085 conmon[35961]: cluster 2021-02-10T04:56:04.123 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:03 smithi085 conmon[35961]: 2021-02 2021-02-10T04:56:04.123 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:03 smithi085 conmon[35961]: -10T04:56:03 2021-02-10T04:56:04.123 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:03 smithi085 conmon[35961]: .572572+0000 mon.a (mon. 2021-02-10T04:56:04.123 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:03 smithi085 conmon[35961]: 0) 2719 : cluster [DBG] 2021-02-10T04:56:04.123 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:03 smithi085 conmon[35961]: osdmap e1263: 8 total, 8 up, 8 in 2021-02-10T04:56:04.123 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:03 smithi085 conmon[29319]: cluster 2021-02-10T04:56:02. 2021-02-10T04:56:04.123 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:03 smithi085 conmon[29319]: 508123+0000 mgr.y (mgr.14140) 1389 : cluster [DBG] pgmap v2284: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2021-02-10T04:56:04.123 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:03 smithi085 conmon[29319]: cluster 2021-02-10T04:56:03.572572+0000 mon.a (mon.0) 2719 : cluster [DBG] osdmap e1263: 8 total, 8 up, 8 in 2021-02-10T04:56:04.208 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:56:04.227 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:56:04.244 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:56:04.262 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:56:04.283 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:56:04.302 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:56:04.323 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:56:04.340 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:56:04.341 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:56:04.341 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:56:04.341 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:56:04.342 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:56:04.342 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:56:04.342 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:56:04.342 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:56:04.342 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:56:04.403 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:56:04 smithi085 conmon[177076]: debug 2021-02-10T04:56:04.121+0000 7f14ab657700 -1 osd.0 pg_epoch: 1263 pg[6.0( empty local-lis/les=1227/1228 n=0 ec=59/59 lis/c=1227/1227 les/c/f=1228/1228/0 sis=1227 pruub=13.481251717s) [0,3,2] r=0 lpr=1227 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.0) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 2 (0x561e5b015080) 2021-02-10T04:56:04.416 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:56:04.416 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2021-02-10T04:56:04.416 INFO:teuthology.orchestra.run.smithi164.stdout: "complaint_time": 30, 2021-02-10T04:56:04.417 INFO:teuthology.orchestra.run.smithi164.stdout: "num_blocked_ops": 0 2021-02-10T04:56:04.417 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:56:04.709 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is set 2021-02-10T04:56:04.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:04 smithi085 conmon[35961]: audit 2021-02-10T04:56:04.181289+0000 2021-02-10T04:56:04.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:04 smithi085 conmon[35961]: mon.c (mon.1) 129 : audit [DBG] from='client.? 172.21.15.85:0/427501070' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:56:04.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:04 smithi085 conmon[35961]: audit 2021-02-10T04:56:04.366611+0000 mon.a (mon. 2021-02-10T04:56:04.731 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:04 smithi085 conmon[35961]: 0) 2720 : audit [INF] from='client.? 172.21.15.85:0/1244972568' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:56:04.732 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:04 smithi085 conmon[29319]: audit 2021-02-10T04:56:04.181289+0000 2021-02-10T04:56:04.732 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:04 smithi085 conmon[29319]: mon.c (mon.1) 129 : audit [DBG] from='client.? 172.21.15.85:0/427501070' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:56:04.732 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:04 smithi085 conmon[29319]: audit 2021-02-10T04:56:04.366611+0000 mon.a ( 2021-02-10T04:56:04.732 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:04 smithi085 conmon[29319]: mon.0) 2720 : audit [INF] from='client.? 172.21.15.85:0/1244972568' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:56:05.115 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:56:05.119 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:56:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:04 smithi164 conmon[29288]: audit 2021-02-10T04:56:04.181289+0000 2021-02-10T04:56:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:04 smithi164 conmon[29288]: mon.c (mon.1) 129 : audit [DBG] from='client.? 172.21.15.85:0/427501070' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:56:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:04 smithi164 conmon[29288]: audit 2021-02-10T04:56:04.366611+0000 mon.a (mon.0) 2720 : audit [INF] from='client.? 172.21.15.85:0/1244972568' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:56:06.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:05 smithi164 conmon[29288]: cluster 2021-02-10T04:56:04.508561+0000 mgr.y 2021-02-10T04:56:06.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:05 smithi164 conmon[29288]: (mgr.14140) 1390 : cluster [DBG] pgmap v2286: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2021-02-10T04:56:06.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:05 smithi164 conmon[29288]: cluster 2021-02-10T04:56:04.694933+0000 mon.a (mon.0) 2721 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:56:06.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:05 smithi164 conmon[29288]: 2021-02-10T04:56:06.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:05 smithi164 conmon[29288]: audit 2021-02-10T04:56:04.704465+0000 mon.a (mon.0) 2722 2021-02-10T04:56:06.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:05 smithi164 conmon[29288]: : audit [INF] from='client.? 172.21.15.85:0/1244972568' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:56:06.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:05 smithi164 conmon[29288]: cluster 2021-02-10T04:56 2021-02-10T04:56:06.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:05 smithi164 conmon[29288]: :04.704559+0000 mon.a (mon.0) 2723 : cluster [DBG] osdmap e1264: 8 total, 8 up, 8 in 2021-02-10T04:56:06.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:05 smithi164 conmon[29288]: audit 2021-02-10T04:56:06.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:05 smithi164 conmon[29288]: 2021-02-10T04:56:05.093185+0000 mgr.y ( 2021-02-10T04:56:06.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:05 smithi164 conmon[29288]: mgr.14140) 1391 : audit [DBG] from='client.16134 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:56:06.158 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:05 smithi085 conmon[29319]: cluster 2021-02-10T04:56:04.508561+0000 mgr.y (mgr.14140) 1390 : cluster [DBG] pgmap v2286: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2021-02-10T04:56:06.158 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:05 smithi085 conmon[29319]: cluster 2021-02-10T04:56:04.694933+0000 mon.a (mon.0) 2721 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:56:06.159 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:05 smithi085 conmon[29319]: audit 2021-02-10T04:56:04.704465+0000 mon.a (mon.0) 2722 : audit [INF] from='client.? 172.21.15.85:0/1244972568' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:56:06.159 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:05 smithi085 conmon[29319]: cluster 2021-02-10T04:56:04.704559+0000 mon.a (mon.0) 2723 : cluster [DBG] osdmap e1264: 8 total, 8 up, 8 in 2021-02-10T04:56:06.159 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:05 smithi085 conmon[29319]: audit 2021-02-10T04:56:05.093185+0000 mgr.y (mgr.14140) 1391 : audit [DBG] from='client.16134 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:56:06.159 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:05 smithi085 conmon[35961]: cluster 2021-02-10T04:56:06.160 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:05 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:56:06.160 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:05 smithi085 conmon[35961]: :56:04.508561+0000 mgr.y (mgr.14140) 1390 : cluster [DBG] pgmap v2286: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2021-02-10T04:56:06.160 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:05 smithi085 conmon[35961]: cluster 2021-02-10T04:56:06.160 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:05 smithi085 conmon[35961]: 2021-02-10T04:56:04.694933+0000 mon.a (mon.0) 2021-02-10T04:56:06.160 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:05 smithi085 conmon[35961]: 2721 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:56:06.161 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:05 smithi085 conmon[35961]: audit 2021-02-10T04:56:06.161 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:05 smithi085 conmon[35961]: 2021-02-10T04:56:04.704465+0000 mon.a (mon.0) 2722 : audit [INF] from='client.? 172.21.15.85:0/1244972568' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:56:06.161 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:05 smithi085 conmon[35961]: cluster 2021-02-10T04:56: 2021-02-10T04:56:06.161 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:05 smithi085 conmon[35961]: 04.704559+0000 mon.a (mon.0) 2723 : cluster [DBG] osdmap e1264: 8 total, 8 up, 8 in 2021-02-10T04:56:06.162 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:05 smithi085 conmon[35961]: audit 2021-02-10T04:56: 2021-02-10T04:56:06.162 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:05 smithi085 conmon[35961]: 05.093185+0000 mgr.y (mgr.14140) 1391 : audit [DBG] 2021-02-10T04:56:06.162 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:05 smithi085 conmon[35961]: from='client.16134 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:56:06.466 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":2287,"stamp":"2021-02-10T04:56:04.701372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":972357089,"num_objects":748,"num_object_clones":337,"num_object_copies":2244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":748,"num_whiteouts":10,"num_read":7860,"num_read_kb":4562259,"num_write":4841,"num_write_kb":1247750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":887,"num_bytes_recovered":311980434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35650,"ondisk_log_size":35650,"up":594,"acting":594,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":596,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2304552,"kb_used_data":1982632,"kb_used_omap":692,"kb_used_meta":321163,"kb_avail":747656664,"statfs":{"total":767960285184,"available":765600423936,"internally_reserved":0,"allocated":2030215168,"data_stored":2599154790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":709615,"internal_metadata":328870929},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":137,"num_read_kb":137,"num_write":88,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.194782"},"pg_stats":[{"pgid":"6.29","version":"0'0","reported_seq":"1118","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.763528+0000","last_change":"2021-02-10T04:55:14.633086+0000","last_active":"2021-02-10T04:55:56.763528+0000","last_peered":"2021-02-10T04:55:56.763528+0000","last_clean":"2021-02-10T04:55:56.763528+0000","last_became_active":"2021-02-10T04:55:14.632670+0000","last_became_peered":"2021-02-10T04:55:14.632670+0000","last_unstale":"2021-02-10T04:55:56.763528+0000","last_undegraded":"2021-02-10T04:55:56.763528+0000","last_fullsized":"2021-02-10T04:55:56.763528+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:24.121684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:24.121684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":"158","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.678378+0000","last_change":"2021-02-10T04:52:30.133105+0000","last_active":"2021-02-10T04:55:56.678378+0000","last_peered":"2021-02-10T04:55:56.678378+0000","last_clean":"2021-02-10T04:55:56.678378+0000","last_became_active":"2021-02-10T04:51:36.864926+0000","last_became_peered":"2021-02-10T04:51:36.864926+0000","last_unstale":"2021-02-10T04:55:56.678378+0000","last_undegraded":"2021-02-10T04:55:56.678378+0000","last_fullsized":"2021-02-10T04:55:56.678378+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.133043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.133043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"1191'4","reported_seq":"239","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:58.042203+0000","last_change":"2021-02-10T04:53:46.736164+0000","last_active":"2021-02-10T04:55:58.042203+0000","last_peered":"2021-02-10T04:55:58.042203+0000","last_clean":"2021-02-10T04:55:58.042203+0000","last_became_active":"2021-02-10T04:53:46.735711+0000","last_became_peered":"2021-02-10T04:53:46.735711+0000","last_unstale":"2021-02-10T04:55:58.042203+0000","last_undegraded":"2021-02-10T04:55:58.042203+0000","last_fullsized":"2021-02-10T04:55:58.042203+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'3","last_scrub_stamp":"2021-02-10T04:51:01.185820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.185820+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"1261'84","reported_seq":"1394","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:57.067842+0000","last_change":"2021-02-10T04:55:39.594594+0000","last_active":"2021-02-10T04:55:57.067842+0000","last_peered":"2021-02-10T04:55:57.067842+0000","last_clean":"2021-02-10T04:55:57.067842+0000","last_became_active":"2021-02-10T04:55:39.594275+0000","last_became_peered":"2021-02-10T04:55:39.594275+0000","last_unstale":"2021-02-10T04:55:57.067842+0000","last_undegraded":"2021-02-10T04:55:57.067842+0000","last_fullsized":"2021-02-10T04:55:57.067842+0000","mapping_epoch":55,"log_start":"1132'66","ondisk_log_start":"1132'66","created":55,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":0,"last_scrub":"1132'66","last_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_deep_scrub":"1132'66","last_deep_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.430399+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1616,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":186,"num_read_kb":150,"num_write":107,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"167","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.678252+0000","last_change":"2021-02-10T04:53:41.722609+0000","last_active":"2021-02-10T04:55:56.678252+0000","last_peered":"2021-02-10T04:55:56.678252+0000","last_clean":"2021-02-10T04:55:56.678252+0000","last_became_active":"2021-02-10T04:51:36.875000+0000","last_became_peered":"2021-02-10T04:51:36.875000+0000","last_unstale":"2021-02-10T04:55:56.678252+0000","last_undegraded":"2021-02-10T04:55:56.678252+0000","last_fullsized":"2021-02-10T04:55:56.678252+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:53:41.722516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:41.722516+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"1264","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:56:01.194352+0000","last_change":"2021-02-10T04:56:01.195173+0000","last_active":"2021-02-10T04:56:01.194352+0000","last_peered":"2021-02-10T04:56:01.194352+0000","last_clean":"2021-02-10T04:56:01.194352+0000","last_became_active":"2021-02-10T04:55:14.630535+0000","last_became_peered":"2021-02-10T04:55:14.630535+0000","last_unstale":"2021-02-10T04:56:01.194352+0000","last_undegraded":"2021-02-10T04:56:01.194352+0000","last_fullsized":"2021-02-10T04:56:01.194352+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:48.408748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:50:48.408748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"1003'1689","reported_seq":"4681","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.678126+0000","last_change":"2021-02-10T04:53:38.170873+0000","last_active":"2021-02-10T04:55:56.678126+0000","last_peered":"2021-02-10T04:55:56.678126+0000","last_clean":"2021-02-10T04:55:56.678126+0000","last_became_active":"2021-02-10T04:52:26.030105+0000","last_became_peered":"2021-02-10T04:52:26.030105+0000","last_unstale":"2021-02-10T04:55:56.678126+0000","last_undegraded":"2021-02-10T04:55:56.678126+0000","last_fullsized":"2021-02-10T04:55:56.678126+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:53:38.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.170802+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":99361981,"num_objects":37,"num_object_clones":33,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"62'5","reported_seq":"149","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:56:01.711107+0000","last_change":"2021-02-10T04:54:39.615784+0000","last_active":"2021-02-10T04:56:01.711107+0000","last_peered":"2021-02-10T04:56:01.711107+0000","last_clean":"2021-02-10T04:56:01.711107+0000","last_became_active":"2021-02-10T04:54:39.560009+0000","last_became_peered":"2021-02-10T04:54:39.560009+0000","last_unstale":"2021-02-10T04:56:01.711107+0000","last_undegraded":"2021-02-10T04:56:01.711107+0000","last_fullsized":"2021-02-10T04:56:01.711107+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":234,"num_read_kb":233,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"1261'28","reported_seq":"1271","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:57.107163+0000","last_change":"2021-02-10T04:55:21.669679+0000","last_active":"2021-02-10T04:55:57.107163+0000","last_peered":"2021-02-10T04:55:57.107163+0000","last_clean":"2021-02-10T04:55:57.107163+0000","last_became_active":"2021-02-10T04:55:21.669219+0000","last_became_peered":"2021-02-10T04:55:21.669219+0000","last_unstale":"2021-02-10T04:55:57.107163+0000","last_undegraded":"2021-02-10T04:55:57.107163+0000","last_fullsized":"2021-02-10T04:55:57.107163+0000","mapping_epoch":1228,"log_start":"1061'20","ondisk_log_start":"1061'20","created":55,"last_epoch_clean":1233,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'32","last_scrub_stamp":"2021-02-10T04:51:51.382040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.382040+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1158,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":105,"num_read_kb":81,"num_write":58,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2052,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"913","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.689391+0000","last_change":"2021-02-10T04:53:34.179015+0000","last_active":"2021-02-10T04:55:56.689391+0000","last_peered":"2021-02-10T04:55:56.689391+0000","last_clean":"2021-02-10T04:55:56.689391+0000","last_became_active":"2021-02-10T04:51:49.502576+0000","last_became_peered":"2021-02-10T04:51:49.502576+0000","last_unstale":"2021-02-10T04:55:56.689391+0000","last_undegraded":"2021-02-10T04:55:56.689391+0000","last_fullsized":"2021-02-10T04:55:56.689391+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:34.178949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.178949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"158","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.676101+0000","last_change":"2021-02-10T04:51:53.038893+0000","last_active":"2021-02-10T04:55:56.676101+0000","last_peered":"2021-02-10T04:55:56.676101+0000","last_clean":"2021-02-10T04:55:56.676101+0000","last_became_active":"2021-02-10T04:51:36.855507+0000","last_became_peered":"2021-02-10T04:51:36.855507+0000","last_unstale":"2021-02-10T04:55:56.676101+0000","last_undegraded":"2021-02-10T04:55:56.676101+0000","last_fullsized":"2021-02-10T04:55:56.676101+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:53.038834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:53.038834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"1180","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.703663+0000","last_change":"2021-02-10T04:55:28.097406+0000","last_active":"2021-02-10T04:55:56.703663+0000","last_peered":"2021-02-10T04:55:56.703663+0000","last_clean":"2021-02-10T04:55:56.703663+0000","last_became_active":"2021-02-10T04:55:14.629603+0000","last_became_peered":"2021-02-10T04:55:14.629603+0000","last_unstale":"2021-02-10T04:55:56.703663+0000","last_undegraded":"2021-02-10T04:55:56.703663+0000","last_fullsized":"2021-02-10T04:55:56.703663+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:09.922789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:09.922789+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"1168'82","reported_seq":"194","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.675676+0000","last_change":"2021-02-10T04:53:23.326363+0000","last_active":"2021-02-10T04:55:56.675676+0000","last_peered":"2021-02-10T04:55:56.675676+0000","last_clean":"2021-02-10T04:55:56.675676+0000","last_became_active":"2021-02-10T04:51:36.859215+0000","last_became_peered":"2021-02-10T04:51:36.859215+0000","last_unstale":"2021-02-10T04:55:56.675676+0000","last_undegraded":"2021-02-10T04:55:56.675676+0000","last_fullsized":"2021-02-10T04:55:56.675676+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1150'76","last_scrub_stamp":"2021-02-10T04:53:00.964499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:00.964499+0000","log_size":82,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3848803,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"1261'69","reported_seq":"1402","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:55:59.688092+0000","last_change":"2021-02-10T04:55:25.576927+0000","last_active":"2021-02-10T04:55:59.688092+0000","last_peered":"2021-02-10T04:55:59.688092+0000","last_clean":"2021-02-10T04:55:59.688092+0000","last_became_active":"2021-02-10T04:55:25.576633+0000","last_became_peered":"2021-02-10T04:55:25.576633+0000","last_unstale":"2021-02-10T04:55:59.688092+0000","last_undegraded":"2021-02-10T04:55:59.688092+0000","last_fullsized":"2021-02-10T04:55:59.688092+0000","mapping_epoch":55,"log_start":"1112'47","ondisk_log_start":"1112'47","created":55,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'61","last_scrub_stamp":"2021-02-10T04:53:36.836202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:36.836202+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1466,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":208,"num_read_kb":172,"num_write":122,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"1228'6","reported_seq":"232","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:55:59.682994+0000","last_change":"2021-02-10T04:55:14.605868+0000","last_active":"2021-02-10T04:55:59.682994+0000","last_peered":"2021-02-10T04:55:59.682994+0000","last_clean":"2021-02-10T04:55:59.682994+0000","last_became_active":"2021-02-10T04:55:14.605314+0000","last_became_peered":"2021-02-10T04:55:14.605314+0000","last_unstale":"2021-02-10T04:55:59.682994+0000","last_undegraded":"2021-02-10T04:55:59.682994+0000","last_fullsized":"2021-02-10T04:55:59.682994+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'4","last_scrub_stamp":"2021-02-10T04:52:01.121380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.121380+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"173","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.675802+0000","last_change":"2021-02-10T04:55:22.573715+0000","last_active":"2021-02-10T04:55:56.675802+0000","last_peered":"2021-02-10T04:55:56.675802+0000","last_clean":"2021-02-10T04:55:56.675802+0000","last_became_active":"2021-02-10T04:55:14.636895+0000","last_became_peered":"2021-02-10T04:55:14.636895+0000","last_unstale":"2021-02-10T04:55:56.675802+0000","last_undegraded":"2021-02-10T04:55:56.675802+0000","last_fullsized":"2021-02-10T04:55:56.675802+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:55:22.573621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:55:22.573621+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"558","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.763292+0000","last_change":"2021-02-10T04:53:46.733236+0000","last_active":"2021-02-10T04:55:56.763292+0000","last_peered":"2021-02-10T04:55:56.763292+0000","last_clean":"2021-02-10T04:55:56.763292+0000","last_became_active":"2021-02-10T04:53:46.732890+0000","last_became_peered":"2021-02-10T04:53:46.732890+0000","last_unstale":"2021-02-10T04:55:56.763292+0000","last_undegraded":"2021-02-10T04:55:56.763292+0000","last_fullsized":"2021-02-10T04:55:56.763292+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:35.170157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:35.170157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"1003'739","reported_seq":"3859","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.704269+0000","last_change":"2021-02-10T04:55:28.091164+0000","last_active":"2021-02-10T04:55:56.704269+0000","last_peered":"2021-02-10T04:55:56.704269+0000","last_clean":"2021-02-10T04:55:56.704269+0000","last_became_active":"2021-02-10T04:55:14.648601+0000","last_became_peered":"2021-02-10T04:55:14.648601+0000","last_unstale":"2021-02-10T04:55:56.704269+0000","last_undegraded":"2021-02-10T04:55:56.704269+0000","last_fullsized":"2021-02-10T04:55:56.704269+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:52:27.602006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:27.602006+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24059888,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":129,"num_read_kb":109544,"num_write":102,"num_write_kb":46173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"1239","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:55:59.687737+0000","last_change":"2021-02-10T04:54:39.559980+0000","last_active":"2021-02-10T04:55:59.687737+0000","last_peered":"2021-02-10T04:55:59.687737+0000","last_clean":"2021-02-10T04:55:59.687737+0000","last_became_active":"2021-02-10T04:54:39.559443+0000","last_became_peered":"2021-02-10T04:54:39.559443+0000","last_unstale":"2021-02-10T04:55:59.687737+0000","last_undegraded":"2021-02-10T04:55:59.687737+0000","last_fullsized":"2021-02-10T04:55:59.687737+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"1261'64","reported_seq":"1446","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:57.161966+0000","last_change":"2021-02-10T04:53:37.108747+0000","last_active":"2021-02-10T04:55:57.161966+0000","last_peered":"2021-02-10T04:55:57.161966+0000","last_clean":"2021-02-10T04:55:57.161966+0000","last_became_active":"2021-02-10T04:53:35.551311+0000","last_became_peered":"2021-02-10T04:53:35.551311+0000","last_unstale":"2021-02-10T04:55:57.161966+0000","last_undegraded":"2021-02-10T04:55:57.161966+0000","last_fullsized":"2021-02-10T04:55:57.161966+0000","mapping_epoch":1118,"log_start":"1066'44","ondisk_log_start":"1066'44","created":55,"last_epoch_clean":1181,"parent":"0.0","parent_split_bits":0,"last_scrub":"1066'44","last_scrub_stamp":"2021-02-10T04:53:37.108677+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:53:37.108677+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1578,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":180,"num_read_kb":144,"num_write":106,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":2312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"1156","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.678073+0000","last_change":"2021-02-10T04:53:39.126356+0000","last_active":"2021-02-10T04:55:56.678073+0000","last_peered":"2021-02-10T04:55:56.678073+0000","last_clean":"2021-02-10T04:55:56.678073+0000","last_became_active":"2021-02-10T04:29:07.145616+0000","last_became_peered":"2021-02-10T04:29:07.145616+0000","last_unstale":"2021-02-10T04:55:56.678073+0000","last_undegraded":"2021-02-10T04:55:56.678073+0000","last_fullsized":"2021-02-10T04:55:56.678073+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:39.126261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:39.126261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"488","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.678749+0000","last_change":"2021-02-10T04:55:14.611192+0000","last_active":"2021-02-10T04:55:56.678749+0000","last_peered":"2021-02-10T04:55:56.678749+0000","last_clean":"2021-02-10T04:55:56.678749+0000","last_became_active":"2021-02-10T04:55:14.610821+0000","last_became_peered":"2021-02-10T04:55:14.610821+0000","last_unstale":"2021-02-10T04:55:56.678749+0000","last_undegraded":"2021-02-10T04:55:56.678749+0000","last_fullsized":"2021-02-10T04:55:56.678749+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:26.311958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.311958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"190","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.677899+0000","last_change":"2021-02-10T04:55:03.526061+0000","last_active":"2021-02-10T04:55:56.677899+0000","last_peered":"2021-02-10T04:55:56.677899+0000","last_clean":"2021-02-10T04:55:56.677899+0000","last_became_active":"2021-02-10T04:51:36.855744+0000","last_became_peered":"2021-02-10T04:51:36.855744+0000","last_unstale":"2021-02-10T04:55:56.677899+0000","last_undegraded":"2021-02-10T04:55:56.677899+0000","last_fullsized":"2021-02-10T04:55:56.677899+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:02.162851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:13.727241+0000","last_clean_scrub_stamp":"2021-02-10T04:52:02.162851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"1063'654","reported_seq":"4437","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.678157+0000","last_change":"2021-02-10T04:55:14.637619+0000","last_active":"2021-02-10T04:55:56.678157+0000","last_peered":"2021-02-10T04:55:56.678157+0000","last_clean":"2021-02-10T04:55:56.678157+0000","last_became_active":"2021-02-10T04:55:14.636752+0000","last_became_peered":"2021-02-10T04:55:14.636752+0000","last_unstale":"2021-02-10T04:55:56.678157+0000","last_undegraded":"2021-02-10T04:55:56.678157+0000","last_fullsized":"2021-02-10T04:55:56.678157+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:51:57.121838+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.121838+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24700630,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":171,"num_read_kb":182272,"num_write":111,"num_write_kb":33727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":26349023,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"1207","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.677616+0000","last_change":"2021-02-10T04:51:50.433056+0000","last_active":"2021-02-10T04:55:56.677616+0000","last_peered":"2021-02-10T04:55:56.677616+0000","last_clean":"2021-02-10T04:55:56.677616+0000","last_became_active":"2021-02-10T04:51:36.843165+0000","last_became_peered":"2021-02-10T04:51:36.843165+0000","last_unstale":"2021-02-10T04:55:56.677616+0000","last_undegraded":"2021-02-10T04:55:56.677616+0000","last_fullsized":"2021-02-10T04:55:56.677616+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.432990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.432990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"1228","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.705500+0000","last_change":"2021-02-10T04:55:14.638290+0000","last_active":"2021-02-10T04:55:56.705500+0000","last_peered":"2021-02-10T04:55:56.705500+0000","last_clean":"2021-02-10T04:55:56.705500+0000","last_became_active":"2021-02-10T04:55:14.637927+0000","last_became_peered":"2021-02-10T04:55:14.637927+0000","last_unstale":"2021-02-10T04:55:56.705500+0000","last_undegraded":"2021-02-10T04:55:56.705500+0000","last_fullsized":"2021-02-10T04:55:56.705500+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"1153","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.763833+0000","last_change":"2021-02-10T04:53:46.727888+0000","last_active":"2021-02-10T04:55:56.763833+0000","last_peered":"2021-02-10T04:55:56.763833+0000","last_clean":"2021-02-10T04:55:56.763833+0000","last_became_active":"2021-02-10T04:53:46.727670+0000","last_became_peered":"2021-02-10T04:53:46.727670+0000","last_unstale":"2021-02-10T04:55:56.763833+0000","last_undegraded":"2021-02-10T04:55:56.763833+0000","last_fullsized":"2021-02-10T04:55:56.763833+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:41.041553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.041553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"1261'76","reported_seq":"1370","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:57.071655+0000","last_change":"2021-02-10T04:53:42.071173+0000","last_active":"2021-02-10T04:55:57.071655+0000","last_peered":"2021-02-10T04:55:57.071655+0000","last_clean":"2021-02-10T04:55:57.071655+0000","last_became_active":"2021-02-10T04:53:39.679752+0000","last_became_peered":"2021-02-10T04:53:39.679752+0000","last_unstale":"2021-02-10T04:55:57.071655+0000","last_undegraded":"2021-02-10T04:55:57.071655+0000","last_fullsized":"2021-02-10T04:55:57.071655+0000","mapping_epoch":1111,"log_start":"1049'39","ondisk_log_start":"1049'39","created":55,"last_epoch_clean":1185,"parent":"0.0","parent_split_bits":0,"last_scrub":"1187'54","last_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_deep_scrub":"1187'54","last_deep_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_clean_scrub_stamp":"2021-02-10T04:53:42.071103+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":147,"num_read_kb":127,"num_write":95,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"898","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.678993+0000","last_change":"2021-02-10T04:51:52.350764+0000","last_active":"2021-02-10T04:55:56.678993+0000","last_peered":"2021-02-10T04:55:56.678993+0000","last_clean":"2021-02-10T04:55:56.678993+0000","last_became_active":"2021-02-10T04:51:49.567784+0000","last_became_peered":"2021-02-10T04:51:49.567784+0000","last_unstale":"2021-02-10T04:55:56.678993+0000","last_undegraded":"2021-02-10T04:55:56.678993+0000","last_fullsized":"2021-02-10T04:55:56.678993+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:52.350706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.350706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"1119'4","reported_seq":"216","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:58.042061+0000","last_change":"2021-02-10T04:52:27.126670+0000","last_active":"2021-02-10T04:55:58.042061+0000","last_peered":"2021-02-10T04:55:58.042061+0000","last_clean":"2021-02-10T04:55:58.042061+0000","last_became_active":"2021-02-10T04:51:36.853647+0000","last_became_peered":"2021-02-10T04:51:36.853647+0000","last_unstale":"2021-02-10T04:55:58.042061+0000","last_undegraded":"2021-02-10T04:55:58.042061+0000","last_fullsized":"2021-02-10T04:55:58.042061+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'4","last_scrub_stamp":"2021-02-10T04:52:27.126599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:27.126599+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"1247","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.679491+0000","last_change":"2021-02-10T04:52:25.745660+0000","last_active":"2021-02-10T04:55:56.679491+0000","last_peered":"2021-02-10T04:55:56.679491+0000","last_clean":"2021-02-10T04:55:56.679491+0000","last_became_active":"2021-02-10T04:51:36.849503+0000","last_became_peered":"2021-02-10T04:51:36.849503+0000","last_unstale":"2021-02-10T04:55:56.679491+0000","last_undegraded":"2021-02-10T04:55:56.679491+0000","last_fullsized":"2021-02-10T04:55:56.679491+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:25.745577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.745577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"1004'1093","reported_seq":"6442","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.704532+0000","last_change":"2021-02-10T04:55:28.097530+0000","last_active":"2021-02-10T04:55:56.704532+0000","last_peered":"2021-02-10T04:55:56.704532+0000","last_clean":"2021-02-10T04:55:56.704532+0000","last_became_active":"2021-02-10T04:55:14.627630+0000","last_became_peered":"2021-02-10T04:55:14.627630+0000","last_unstale":"2021-02-10T04:55:56.704532+0000","last_undegraded":"2021-02-10T04:55:56.704532+0000","last_fullsized":"2021-02-10T04:55:56.704532+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39434696,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":279,"num_read_kb":311058,"num_write":185,"num_write_kb":66790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"62'5","reported_seq":"60","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.703563+0000","last_change":"2021-02-10T04:54:39.559473+0000","last_active":"2021-02-10T04:55:56.703563+0000","last_peered":"2021-02-10T04:55:56.703563+0000","last_clean":"2021-02-10T04:55:56.703563+0000","last_became_active":"2021-02-10T04:54:39.559048+0000","last_became_peered":"2021-02-10T04:54:39.559048+0000","last_unstale":"2021-02-10T04:55:56.703563+0000","last_undegraded":"2021-02-10T04:55:56.703563+0000","last_fullsized":"2021-02-10T04:55:56.703563+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":149,"num_read_kb":148,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"1216","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.678421+0000","last_change":"2021-02-10T04:55:03.230261+0000","last_active":"2021-02-10T04:55:56.678421+0000","last_peered":"2021-02-10T04:55:56.678421+0000","last_clean":"2021-02-10T04:55:56.678421+0000","last_became_active":"2021-02-10T04:51:36.841582+0000","last_became_peered":"2021-02-10T04:51:36.841582+0000","last_unstale":"2021-02-10T04:55:56.678421+0000","last_undegraded":"2021-02-10T04:55:56.678421+0000","last_fullsized":"2021-02-10T04:55:56.678421+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:12.389272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:12.389272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"1261'71","reported_seq":"964","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:57.171131+0000","last_change":"2021-02-10T04:53:27.560299+0000","last_active":"2021-02-10T04:55:57.171131+0000","last_peered":"2021-02-10T04:55:57.171131+0000","last_clean":"2021-02-10T04:55:57.171131+0000","last_became_active":"2021-02-10T04:53:27.559886+0000","last_became_peered":"2021-02-10T04:53:27.559886+0000","last_unstale":"2021-02-10T04:55:57.171131+0000","last_undegraded":"2021-02-10T04:55:57.171131+0000","last_fullsized":"2021-02-10T04:55:57.171131+0000","mapping_epoch":1129,"log_start":"1049'51","ondisk_log_start":"1049'51","created":55,"last_epoch_clean":1173,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'41","last_scrub_stamp":"2021-02-10T04:51:58.353660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:58.353660+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1472,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":169,"num_read_kb":136,"num_write":102,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"921","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.677521+0000","last_change":"2021-02-10T04:53:11.271557+0000","last_active":"2021-02-10T04:55:56.677521+0000","last_peered":"2021-02-10T04:55:56.677521+0000","last_clean":"2021-02-10T04:55:56.677521+0000","last_became_active":"2021-02-10T04:51:49.567883+0000","last_became_peered":"2021-02-10T04:51:49.567883+0000","last_unstale":"2021-02-10T04:55:56.677521+0000","last_undegraded":"2021-02-10T04:55:56.677521+0000","last_fullsized":"2021-02-10T04:55:56.677521+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:11.271459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.271459+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"1229'5","reported_seq":"1612","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:58.042155+0000","last_change":"2021-02-10T04:55:14.638050+0000","last_active":"2021-02-10T04:55:58.042155+0000","last_peered":"2021-02-10T04:55:58.042155+0000","last_clean":"2021-02-10T04:55:58.042155+0000","last_became_active":"2021-02-10T04:55:14.637570+0000","last_became_peered":"2021-02-10T04:55:14.637570+0000","last_unstale":"2021-02-10T04:55:58.042155+0000","last_undegraded":"2021-02-10T04:55:58.042155+0000","last_fullsized":"2021-02-10T04:55:58.042155+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:51:47.855283+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.855283+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"1251","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.679518+0000","last_change":"2021-02-10T04:55:22.183630+0000","last_active":"2021-02-10T04:55:56.679518+0000","last_peered":"2021-02-10T04:55:56.679518+0000","last_clean":"2021-02-10T04:55:56.679518+0000","last_became_active":"2021-02-10T04:55:14.656661+0000","last_became_peered":"2021-02-10T04:55:14.656661+0000","last_unstale":"2021-02-10T04:55:56.679518+0000","last_undegraded":"2021-02-10T04:55:56.679518+0000","last_fullsized":"2021-02-10T04:55:56.679518+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:55:22.183566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:55:22.183566+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"1002'1090","reported_seq":"5456","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:55:59.688032+0000","last_change":"2021-02-10T04:52:26.030198+0000","last_active":"2021-02-10T04:55:59.688032+0000","last_peered":"2021-02-10T04:55:59.688032+0000","last_clean":"2021-02-10T04:55:59.688032+0000","last_became_active":"2021-02-10T04:52:26.028182+0000","last_became_peered":"2021-02-10T04:52:26.028182+0000","last_unstale":"2021-02-10T04:55:59.688032+0000","last_undegraded":"2021-02-10T04:55:59.688032+0000","last_fullsized":"2021-02-10T04:55:59.688032+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:51:08.220472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.220472+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24410646,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":245,"num_read_kb":276168,"num_write":175,"num_write_kb":64960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"1227","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.677556+0000","last_change":"2021-02-10T04:54:39.555379+0000","last_active":"2021-02-10T04:55:56.677556+0000","last_peered":"2021-02-10T04:55:56.677556+0000","last_clean":"2021-02-10T04:55:56.677556+0000","last_became_active":"2021-02-10T04:54:39.554976+0000","last_became_peered":"2021-02-10T04:54:39.554976+0000","last_unstale":"2021-02-10T04:55:56.677556+0000","last_undegraded":"2021-02-10T04:55:56.677556+0000","last_fullsized":"2021-02-10T04:55:56.677556+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"1261'65","reported_seq":"1359","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:57.141109+0000","last_change":"2021-02-10T04:53:31.624862+0000","last_active":"2021-02-10T04:55:57.141109+0000","last_peered":"2021-02-10T04:55:57.141109+0000","last_clean":"2021-02-10T04:55:57.141109+0000","last_became_active":"2021-02-10T04:53:31.624318+0000","last_became_peered":"2021-02-10T04:53:31.624318+0000","last_unstale":"2021-02-10T04:55:57.141109+0000","last_undegraded":"2021-02-10T04:55:57.141109+0000","last_fullsized":"2021-02-10T04:55:57.141109+0000","mapping_epoch":55,"log_start":"1049'47","ondisk_log_start":"1049'47","created":55,"last_epoch_clean":1177,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'47","last_scrub_stamp":"2021-02-10T04:53:09.002368+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.002368+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1672,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":169,"num_read_kb":135,"num_write":98,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":934,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"198","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.763360+0000","last_change":"2021-02-10T04:55:14.611324+0000","last_active":"2021-02-10T04:55:56.763360+0000","last_peered":"2021-02-10T04:55:56.763360+0000","last_clean":"2021-02-10T04:55:56.763360+0000","last_became_active":"2021-02-10T04:55:14.610986+0000","last_became_peered":"2021-02-10T04:55:14.610986+0000","last_unstale":"2021-02-10T04:55:56.763360+0000","last_undegraded":"2021-02-10T04:55:56.763360+0000","last_fullsized":"2021-02-10T04:55:56.763360+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:08.358300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:50:08.358300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"1238","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.678377+0000","last_change":"2021-02-10T04:51:57.436961+0000","last_active":"2021-02-10T04:55:56.678377+0000","last_peered":"2021-02-10T04:55:56.678377+0000","last_clean":"2021-02-10T04:55:56.678377+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:55:56.678377+0000","last_undegraded":"2021-02-10T04:55:56.678377+0000","last_fullsized":"2021-02-10T04:55:56.678377+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.436862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.436862+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"1193","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.705317+0000","last_change":"2021-02-10T04:53:02.112612+0000","last_active":"2021-02-10T04:55:56.705317+0000","last_peered":"2021-02-10T04:55:56.705317+0000","last_clean":"2021-02-10T04:55:56.705317+0000","last_became_active":"2021-02-10T04:50:59.545501+0000","last_became_peered":"2021-02-10T04:50:59.545501+0000","last_unstale":"2021-02-10T04:55:56.705317+0000","last_undegraded":"2021-02-10T04:55:56.705317+0000","last_fullsized":"2021-02-10T04:55:56.705317+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:02.112533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:02.112533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"1001'1165","reported_seq":"4435","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:55:59.688501+0000","last_change":"2021-02-10T04:55:14.653651+0000","last_active":"2021-02-10T04:55:59.688501+0000","last_peered":"2021-02-10T04:55:59.688501+0000","last_clean":"2021-02-10T04:55:59.688501+0000","last_became_active":"2021-02-10T04:55:14.652235+0000","last_became_peered":"2021-02-10T04:55:14.652235+0000","last_unstale":"2021-02-10T04:55:59.688501+0000","last_undegraded":"2021-02-10T04:55:59.688501+0000","last_fullsized":"2021-02-10T04:55:59.688501+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29601845,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":286,"num_read_kb":299529,"num_write":200,"num_write_kb":89130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"43","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.705222+0000","last_change":"2021-02-10T04:55:15.590113+0000","last_active":"2021-02-10T04:55:56.705222+0000","last_peered":"2021-02-10T04:55:56.705222+0000","last_clean":"2021-02-10T04:55:56.705222+0000","last_became_active":"2021-02-10T04:55:15.589751+0000","last_became_peered":"2021-02-10T04:55:15.589751+0000","last_unstale":"2021-02-10T04:55:56.705222+0000","last_undegraded":"2021-02-10T04:55:56.705222+0000","last_fullsized":"2021-02-10T04:55:56.705222+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"1113'5","reported_seq":"1498","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:58.042093+0000","last_change":"2021-02-10T04:52:34.047474+0000","last_active":"2021-02-10T04:55:58.042093+0000","last_peered":"2021-02-10T04:55:58.042093+0000","last_clean":"2021-02-10T04:55:58.042093+0000","last_became_active":"2021-02-10T04:50:59.569578+0000","last_became_peered":"2021-02-10T04:50:59.569578+0000","last_unstale":"2021-02-10T04:55:58.042093+0000","last_undegraded":"2021-02-10T04:55:58.042093+0000","last_fullsized":"2021-02-10T04:55:58.042093+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1113'5","last_scrub_stamp":"2021-02-10T04:52:34.047358+0000","last_deep_scrub":"158'3","last_deep_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_clean_scrub_stamp":"2021-02-10T04:52:34.047358+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"1261'68","reported_seq":"1344","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:57.145522+0000","last_change":"2021-02-10T04:51:48.872937+0000","last_active":"2021-02-10T04:55:57.145522+0000","last_peered":"2021-02-10T04:55:57.145522+0000","last_clean":"2021-02-10T04:55:57.145522+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:55:57.145522+0000","last_undegraded":"2021-02-10T04:55:57.145522+0000","last_fullsized":"2021-02-10T04:55:57.145522+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:51:48.872839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.872839+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":114,"num_read_kb":94,"num_write":68,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"1313","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:55:59.689149+0000","last_change":"2021-02-10T04:53:05.984747+0000","last_active":"2021-02-10T04:55:59.689149+0000","last_peered":"2021-02-10T04:55:59.689149+0000","last_clean":"2021-02-10T04:55:59.689149+0000","last_became_active":"2021-02-10T04:51:36.869931+0000","last_became_peered":"2021-02-10T04:51:36.869931+0000","last_unstale":"2021-02-10T04:55:59.689149+0000","last_undegraded":"2021-02-10T04:55:59.689149+0000","last_fullsized":"2021-02-10T04:55:59.689149+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:05.984659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:05.984659+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"1115","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.763894+0000","last_change":"2021-02-10T04:53:46.731709+0000","last_active":"2021-02-10T04:55:56.763894+0000","last_peered":"2021-02-10T04:55:56.763894+0000","last_clean":"2021-02-10T04:55:56.763894+0000","last_became_active":"2021-02-10T04:53:46.731340+0000","last_became_peered":"2021-02-10T04:53:46.731340+0000","last_unstale":"2021-02-10T04:55:56.763894+0000","last_undegraded":"2021-02-10T04:55:56.763894+0000","last_fullsized":"2021-02-10T04:55:56.763894+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:09.005421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.005421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"998'1009","reported_seq":"4752","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.704965+0000","last_change":"2021-02-10T04:52:26.037227+0000","last_active":"2021-02-10T04:55:56.704965+0000","last_peered":"2021-02-10T04:55:56.704965+0000","last_clean":"2021-02-10T04:55:56.704965+0000","last_became_active":"2021-02-10T04:52:26.036723+0000","last_became_peered":"2021-02-10T04:52:26.036723+0000","last_unstale":"2021-02-10T04:55:56.704965+0000","last_undegraded":"2021-02-10T04:55:56.704965+0000","last_fullsized":"2021-02-10T04:55:56.704965+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12355206,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":119,"num_read_kb":124701,"num_write":77,"num_write_kb":25052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4145029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"1132","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.705698+0000","last_change":"2021-02-10T04:53:33.305896+0000","last_active":"2021-02-10T04:55:56.705698+0000","last_peered":"2021-02-10T04:55:56.705698+0000","last_clean":"2021-02-10T04:55:56.705698+0000","last_became_active":"2021-02-10T04:50:59.570854+0000","last_became_peered":"2021-02-10T04:50:59.570854+0000","last_unstale":"2021-02-10T04:55:56.705698+0000","last_undegraded":"2021-02-10T04:55:56.705698+0000","last_fullsized":"2021-02-10T04:55:56.705698+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:33.305827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:33.305827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"1233","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.705833+0000","last_change":"2021-02-10T04:55:15.669988+0000","last_active":"2021-02-10T04:55:56.705833+0000","last_peered":"2021-02-10T04:55:56.705833+0000","last_clean":"2021-02-10T04:55:56.705833+0000","last_became_active":"2021-02-10T04:55:15.669642+0000","last_became_peered":"2021-02-10T04:55:15.669642+0000","last_unstale":"2021-02-10T04:55:56.705833+0000","last_undegraded":"2021-02-10T04:55:56.705833+0000","last_fullsized":"2021-02-10T04:55:56.705833+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"1246","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.678300+0000","last_change":"2021-02-10T04:51:50.895471+0000","last_active":"2021-02-10T04:55:56.678300+0000","last_peered":"2021-02-10T04:55:56.678300+0000","last_clean":"2021-02-10T04:55:56.678300+0000","last_became_active":"2021-02-10T04:50:59.527553+0000","last_became_peered":"2021-02-10T04:50:59.527553+0000","last_unstale":"2021-02-10T04:55:56.678300+0000","last_undegraded":"2021-02-10T04:55:56.678300+0000","last_fullsized":"2021-02-10T04:55:56.678300+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.895370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.895370+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"1261'84","reported_seq":"1377","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:57.096275+0000","last_change":"2021-02-10T04:55:14.603109+0000","last_active":"2021-02-10T04:55:57.096275+0000","last_peered":"2021-02-10T04:55:57.096275+0000","last_clean":"2021-02-10T04:55:57.096275+0000","last_became_active":"2021-02-10T04:55:14.602669+0000","last_became_peered":"2021-02-10T04:55:14.602669+0000","last_unstale":"2021-02-10T04:55:57.096275+0000","last_undegraded":"2021-02-10T04:55:57.096275+0000","last_fullsized":"2021-02-10T04:55:57.096275+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'72","last_scrub_stamp":"2021-02-10T04:46:30.515026+0000","last_deep_scrub":"152'48","last_deep_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_clean_scrub_stamp":"2021-02-10T04:46:30.515026+0000","log_size":84,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":123,"num_read_kb":101,"num_write":72,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"515","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.689388+0000","last_change":"2021-02-10T04:55:03.834050+0000","last_active":"2021-02-10T04:55:56.689388+0000","last_peered":"2021-02-10T04:55:56.689388+0000","last_clean":"2021-02-10T04:55:56.689388+0000","last_became_active":"2021-02-10T04:51:49.503801+0000","last_became_peered":"2021-02-10T04:51:49.503801+0000","last_unstale":"2021-02-10T04:55:56.689388+0000","last_undegraded":"2021-02-10T04:55:56.689388+0000","last_fullsized":"2021-02-10T04:55:56.689388+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:51:48.274960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.274960+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"1271","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.678407+0000","last_change":"2021-02-10T04:53:13.099288+0000","last_active":"2021-02-10T04:55:56.678407+0000","last_peered":"2021-02-10T04:55:56.678407+0000","last_clean":"2021-02-10T04:55:56.678407+0000","last_became_active":"2021-02-10T04:51:36.848614+0000","last_became_peered":"2021-02-10T04:51:36.848614+0000","last_unstale":"2021-02-10T04:55:56.678407+0000","last_undegraded":"2021-02-10T04:55:56.678407+0000","last_fullsized":"2021-02-10T04:55:56.678407+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:13.099228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:13.099228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"1002'1657","reported_seq":"5322","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.705063+0000","last_change":"2021-02-10T04:52:26.036806+0000","last_active":"2021-02-10T04:55:56.705063+0000","last_peered":"2021-02-10T04:55:56.705063+0000","last_clean":"2021-02-10T04:55:56.705063+0000","last_became_active":"2021-02-10T04:52:26.032363+0000","last_became_peered":"2021-02-10T04:52:26.032363+0000","last_unstale":"2021-02-10T04:55:56.705063+0000","last_undegraded":"2021-02-10T04:55:56.705063+0000","last_fullsized":"2021-02-10T04:55:56.705063+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21396533,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":212,"num_read_kb":218836,"num_write":159,"num_write_kb":59398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":22326014,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"1190","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.679045+0000","last_change":"2021-02-10T04:51:51.911945+0000","last_active":"2021-02-10T04:55:56.679045+0000","last_peered":"2021-02-10T04:55:56.679045+0000","last_clean":"2021-02-10T04:55:56.679045+0000","last_became_active":"2021-02-10T04:51:36.850015+0000","last_became_peered":"2021-02-10T04:51:36.850015+0000","last_unstale":"2021-02-10T04:55:56.679045+0000","last_undegraded":"2021-02-10T04:55:56.679045+0000","last_fullsized":"2021-02-10T04:55:56.679045+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.911851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.911851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"62'5","reported_seq":"1094","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.678438+0000","last_change":"2021-02-10T04:54:39.568253+0000","last_active":"2021-02-10T04:55:56.678438+0000","last_peered":"2021-02-10T04:55:56.678438+0000","last_clean":"2021-02-10T04:55:56.678438+0000","last_became_active":"2021-02-10T04:54:39.567642+0000","last_became_peered":"2021-02-10T04:54:39.567642+0000","last_unstale":"2021-02-10T04:55:56.678438+0000","last_undegraded":"2021-02-10T04:55:56.678438+0000","last_fullsized":"2021-02-10T04:55:56.678438+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":98,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":150,"num_read_kb":148,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"1320","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:55:59.688178+0000","last_change":"2021-02-10T04:55:14.634577+0000","last_active":"2021-02-10T04:55:59.688178+0000","last_peered":"2021-02-10T04:55:59.688178+0000","last_clean":"2021-02-10T04:55:59.688178+0000","last_became_active":"2021-02-10T04:55:14.634259+0000","last_became_peered":"2021-02-10T04:55:14.634259+0000","last_unstale":"2021-02-10T04:55:59.688178+0000","last_undegraded":"2021-02-10T04:55:59.688178+0000","last_fullsized":"2021-02-10T04:55:59.688178+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:58.142526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:58.142526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"1261'68","reported_seq":"1270","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:57.114178+0000","last_change":"2021-02-10T04:55:28.079376+0000","last_active":"2021-02-10T04:55:57.114178+0000","last_peered":"2021-02-10T04:55:57.114178+0000","last_clean":"2021-02-10T04:55:57.114178+0000","last_became_active":"2021-02-10T04:55:14.654609+0000","last_became_peered":"2021-02-10T04:55:14.654609+0000","last_unstale":"2021-02-10T04:55:57.114178+0000","last_undegraded":"2021-02-10T04:55:57.114178+0000","last_fullsized":"2021-02-10T04:55:57.114178+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1135'58","last_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_deep_scrub":"1135'58","last_deep_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.483067+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":102,"num_read_kb":83,"num_write":60,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"196","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.765409+0000","last_change":"2021-02-10T04:55:14.657913+0000","last_active":"2021-02-10T04:55:56.765409+0000","last_peered":"2021-02-10T04:55:56.765409+0000","last_clean":"2021-02-10T04:55:56.765409+0000","last_became_active":"2021-02-10T04:55:14.657153+0000","last_became_peered":"2021-02-10T04:55:14.657153+0000","last_unstale":"2021-02-10T04:55:56.765409+0000","last_undegraded":"2021-02-10T04:55:56.765409+0000","last_fullsized":"2021-02-10T04:55:56.765409+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:50:46.434629+0000","last_deep_scrub":"53'2","last_deep_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_clean_scrub_stamp":"2021-02-10T04:50:46.434629+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"938","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.677621+0000","last_change":"2021-02-10T04:52:38.278980+0000","last_active":"2021-02-10T04:55:56.677621+0000","last_peered":"2021-02-10T04:55:56.677621+0000","last_clean":"2021-02-10T04:55:56.677621+0000","last_became_active":"2021-02-10T04:51:49.496237+0000","last_became_peered":"2021-02-10T04:51:49.496237+0000","last_unstale":"2021-02-10T04:55:56.677621+0000","last_undegraded":"2021-02-10T04:55:56.677621+0000","last_fullsized":"2021-02-10T04:55:56.677621+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:38.278910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:52:38.278910+0000","last_clean_scrub_stamp":"2021-02-10T04:52:38.278910+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"1000'1071","reported_seq":"4812","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.678008+0000","last_change":"2021-02-10T04:53:34.407875+0000","last_active":"2021-02-10T04:55:56.678008+0000","last_peered":"2021-02-10T04:55:56.678008+0000","last_clean":"2021-02-10T04:55:56.678008+0000","last_became_active":"2021-02-10T04:52:26.031050+0000","last_became_peered":"2021-02-10T04:52:26.031050+0000","last_unstale":"2021-02-10T04:55:56.678008+0000","last_undegraded":"2021-02-10T04:55:56.678008+0000","last_fullsized":"2021-02-10T04:55:56.678008+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:34.407784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.407784+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30173843,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"1286","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:55:59.688900+0000","last_change":"2021-02-10T04:55:14.637570+0000","last_active":"2021-02-10T04:55:59.688900+0000","last_peered":"2021-02-10T04:55:59.688900+0000","last_clean":"2021-02-10T04:55:59.688900+0000","last_became_active":"2021-02-10T04:55:14.637203+0000","last_became_peered":"2021-02-10T04:55:14.637203+0000","last_unstale":"2021-02-10T04:55:59.688900+0000","last_undegraded":"2021-02-10T04:55:59.688900+0000","last_fullsized":"2021-02-10T04:55:59.688900+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_clean_scrub_stamp":"2021-02-10T04:51:00.195518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"1248","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.678422+0000","last_change":"2021-02-10T04:55:14.661285+0000","last_active":"2021-02-10T04:55:56.678422+0000","last_peered":"2021-02-10T04:55:56.678422+0000","last_clean":"2021-02-10T04:55:56.678422+0000","last_became_active":"2021-02-10T04:55:14.660902+0000","last_became_peered":"2021-02-10T04:55:14.660902+0000","last_unstale":"2021-02-10T04:55:56.678422+0000","last_undegraded":"2021-02-10T04:55:56.678422+0000","last_fullsized":"2021-02-10T04:55:56.678422+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:55.473347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:50:55.473347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"1261'58","reported_seq":"1321","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:57.052557+0000","last_change":"2021-02-10T04:55:47.652017+0000","last_active":"2021-02-10T04:55:57.052557+0000","last_peered":"2021-02-10T04:55:57.052557+0000","last_clean":"2021-02-10T04:55:57.052557+0000","last_became_active":"2021-02-10T04:55:47.651592+0000","last_became_peered":"2021-02-10T04:55:47.651592+0000","last_unstale":"2021-02-10T04:55:57.052557+0000","last_undegraded":"2021-02-10T04:55:57.052557+0000","last_fullsized":"2021-02-10T04:55:57.052557+0000","mapping_epoch":1227,"log_start":"1049'40","ondisk_log_start":"1049'40","created":55,"last_epoch_clean":1257,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'40","last_scrub_stamp":"2021-02-10T04:51:47.281929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.281929+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1836,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":171,"num_read_kb":135,"num_write":99,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"422","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.763409+0000","last_change":"2021-02-10T04:55:14.610765+0000","last_active":"2021-02-10T04:55:56.763409+0000","last_peered":"2021-02-10T04:55:56.763409+0000","last_clean":"2021-02-10T04:55:56.763409+0000","last_became_active":"2021-02-10T04:55:14.610470+0000","last_became_peered":"2021-02-10T04:55:14.610470+0000","last_unstale":"2021-02-10T04:55:56.763409+0000","last_undegraded":"2021-02-10T04:55:56.763409+0000","last_fullsized":"2021-02-10T04:55:56.763409+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:50.399350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:50.399350+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"1148","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.703411+0000","last_change":"2021-02-10T04:55:25.624031+0000","last_active":"2021-02-10T04:55:56.703411+0000","last_peered":"2021-02-10T04:55:56.703411+0000","last_clean":"2021-02-10T04:55:56.703411+0000","last_became_active":"2021-02-10T04:55:14.630129+0000","last_became_peered":"2021-02-10T04:55:14.630129+0000","last_unstale":"2021-02-10T04:55:56.703411+0000","last_undegraded":"2021-02-10T04:55:56.703411+0000","last_fullsized":"2021-02-10T04:55:56.703411+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"923","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.678848+0000","last_change":"2021-02-10T04:52:31.273037+0000","last_active":"2021-02-10T04:55:56.678848+0000","last_peered":"2021-02-10T04:55:56.678848+0000","last_clean":"2021-02-10T04:55:56.678848+0000","last_became_active":"2021-02-10T04:51:49.501405+0000","last_became_peered":"2021-02-10T04:51:49.501405+0000","last_unstale":"2021-02-10T04:55:56.678848+0000","last_undegraded":"2021-02-10T04:55:56.678848+0000","last_fullsized":"2021-02-10T04:55:56.678848+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:31.272971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:17.598113+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.272971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"1000'446","reported_seq":"1979","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.765111+0000","last_change":"2021-02-10T04:55:14.658783+0000","last_active":"2021-02-10T04:55:56.765111+0000","last_peered":"2021-02-10T04:55:56.765111+0000","last_clean":"2021-02-10T04:55:56.765111+0000","last_became_active":"2021-02-10T04:55:14.657668+0000","last_became_peered":"2021-02-10T04:55:14.657668+0000","last_unstale":"2021-02-10T04:55:56.765111+0000","last_undegraded":"2021-02-10T04:55:56.765111+0000","last_fullsized":"2021-02-10T04:55:56.765111+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:51:45.964029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.964029+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22255852,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68435,"num_write":72,"num_write_kb":24345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":403693,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"414","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.678043+0000","last_change":"2021-02-10T04:53:03.228260+0000","last_active":"2021-02-10T04:55:56.678043+0000","last_peered":"2021-02-10T04:55:56.678043+0000","last_clean":"2021-02-10T04:55:56.678043+0000","last_became_active":"2021-02-10T04:51:49.499252+0000","last_became_peered":"2021-02-10T04:51:49.499252+0000","last_unstale":"2021-02-10T04:55:56.678043+0000","last_undegraded":"2021-02-10T04:55:56.678043+0000","last_fullsized":"2021-02-10T04:55:56.678043+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:03.228182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.228182+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"166","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.764742+0000","last_change":"2021-02-10T04:53:46.727384+0000","last_active":"2021-02-10T04:55:56.764742+0000","last_peered":"2021-02-10T04:55:56.764742+0000","last_clean":"2021-02-10T04:55:56.764742+0000","last_became_active":"2021-02-10T04:53:46.726771+0000","last_became_peered":"2021-02-10T04:53:46.726771+0000","last_unstale":"2021-02-10T04:55:56.764742+0000","last_undegraded":"2021-02-10T04:55:56.764742+0000","last_fullsized":"2021-02-10T04:55:56.764742+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:33.948348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:33.948348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"1261'51","reported_seq":"1348","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:55:59.688324+0000","last_change":"2021-02-10T04:54:17.118810+0000","last_active":"2021-02-10T04:55:59.688324+0000","last_peered":"2021-02-10T04:55:59.688324+0000","last_clean":"2021-02-10T04:55:59.688324+0000","last_became_active":"2021-02-10T04:54:17.080053+0000","last_became_peered":"2021-02-10T04:54:17.080053+0000","last_unstale":"2021-02-10T04:55:59.688324+0000","last_undegraded":"2021-02-10T04:55:59.688324+0000","last_fullsized":"2021-02-10T04:55:59.688324+0000","mapping_epoch":1200,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1201,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'39","last_scrub_stamp":"2021-02-10T04:51:47.322331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.322331+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":51,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"194","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.763489+0000","last_change":"2021-02-10T04:55:14.636888+0000","last_active":"2021-02-10T04:55:56.763489+0000","last_peered":"2021-02-10T04:55:56.763489+0000","last_clean":"2021-02-10T04:55:56.763489+0000","last_became_active":"2021-02-10T04:55:14.634305+0000","last_became_peered":"2021-02-10T04:55:14.634305+0000","last_unstale":"2021-02-10T04:55:56.763489+0000","last_undegraded":"2021-02-10T04:55:56.763489+0000","last_fullsized":"2021-02-10T04:55:56.763489+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_clean_scrub_stamp":"2021-02-10T04:50:05.424262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"1432","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.678963+0000","last_change":"2021-02-10T04:52:37.324122+0000","last_active":"2021-02-10T04:55:56.678963+0000","last_peered":"2021-02-10T04:55:56.678963+0000","last_clean":"2021-02-10T04:55:56.678963+0000","last_became_active":"2021-02-10T04:51:36.843553+0000","last_became_peered":"2021-02-10T04:51:36.843553+0000","last_unstale":"2021-02-10T04:55:56.678963+0000","last_undegraded":"2021-02-10T04:55:56.678963+0000","last_fullsized":"2021-02-10T04:55:56.678963+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:37.324040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:37.324040+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"1238","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.705025+0000","last_change":"2021-02-10T04:54:37.612303+0000","last_active":"2021-02-10T04:55:56.705025+0000","last_peered":"2021-02-10T04:55:56.705025+0000","last_clean":"2021-02-10T04:55:56.705025+0000","last_became_active":"2021-02-10T04:54:37.611953+0000","last_became_peered":"2021-02-10T04:54:37.611953+0000","last_unstale":"2021-02-10T04:55:56.705025+0000","last_undegraded":"2021-02-10T04:55:56.705025+0000","last_fullsized":"2021-02-10T04:55:56.705025+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"1260","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.678486+0000","last_change":"2021-02-10T04:52:31.403904+0000","last_active":"2021-02-10T04:55:56.678486+0000","last_peered":"2021-02-10T04:55:56.678486+0000","last_clean":"2021-02-10T04:55:56.678486+0000","last_became_active":"2021-02-10T04:50:59.538009+0000","last_became_peered":"2021-02-10T04:50:59.538009+0000","last_unstale":"2021-02-10T04:55:56.678486+0000","last_undegraded":"2021-02-10T04:55:56.678486+0000","last_fullsized":"2021-02-10T04:55:56.678486+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:31.403824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.403824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"1000'490","reported_seq":"3728","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.705384+0000","last_change":"2021-02-10T04:55:22.716472+0000","last_active":"2021-02-10T04:55:56.705384+0000","last_peered":"2021-02-10T04:55:56.705384+0000","last_clean":"2021-02-10T04:55:56.705384+0000","last_became_active":"2021-02-10T04:55:14.644596+0000","last_became_peered":"2021-02-10T04:55:14.644596+0000","last_unstale":"2021-02-10T04:55:56.705384+0000","last_undegraded":"2021-02-10T04:55:56.705384+0000","last_fullsized":"2021-02-10T04:55:56.705384+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:47:02.875920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:02.875920+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19371627,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":141,"num_read_kb":148568,"num_write":85,"num_write_kb":31732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"158","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.678362+0000","last_change":"2021-02-10T04:53:01.972991+0000","last_active":"2021-02-10T04:55:56.678362+0000","last_peered":"2021-02-10T04:55:56.678362+0000","last_clean":"2021-02-10T04:55:56.678362+0000","last_became_active":"2021-02-10T04:51:36.863976+0000","last_became_peered":"2021-02-10T04:51:36.863976+0000","last_unstale":"2021-02-10T04:55:56.678362+0000","last_undegraded":"2021-02-10T04:55:56.678362+0000","last_fullsized":"2021-02-10T04:55:56.678362+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:01.972913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.972913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"1261'40","reported_seq":"980","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:57.090323+0000","last_change":"2021-02-10T04:55:43.569281+0000","last_active":"2021-02-10T04:55:57.090323+0000","last_peered":"2021-02-10T04:55:57.090323+0000","last_clean":"2021-02-10T04:55:57.090323+0000","last_became_active":"2021-02-10T04:55:43.568878+0000","last_became_peered":"2021-02-10T04:55:43.568878+0000","last_unstale":"2021-02-10T04:55:57.090323+0000","last_undegraded":"2021-02-10T04:55:57.090323+0000","last_fullsized":"2021-02-10T04:55:57.090323+0000","mapping_epoch":1227,"log_start":"1063'30","ondisk_log_start":"1063'30","created":55,"last_epoch_clean":1253,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:56.431187+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.431187+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":802,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":116,"num_read_kb":94,"num_write":67,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"168","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.676537+0000","last_change":"2021-02-10T04:55:14.636849+0000","last_active":"2021-02-10T04:55:56.676537+0000","last_peered":"2021-02-10T04:55:56.676537+0000","last_clean":"2021-02-10T04:55:56.676537+0000","last_became_active":"2021-02-10T04:55:14.636237+0000","last_became_peered":"2021-02-10T04:55:14.636237+0000","last_unstale":"2021-02-10T04:55:56.676537+0000","last_undegraded":"2021-02-10T04:55:56.676537+0000","last_fullsized":"2021-02-10T04:55:56.676537+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:04.591695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:04.591695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"1087","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.677684+0000","last_change":"2021-02-10T04:54:37.680913+0000","last_active":"2021-02-10T04:55:56.677684+0000","last_peered":"2021-02-10T04:55:56.677684+0000","last_clean":"2021-02-10T04:55:56.677684+0000","last_became_active":"2021-02-10T04:54:37.680651+0000","last_became_peered":"2021-02-10T04:54:37.680651+0000","last_unstale":"2021-02-10T04:55:56.677684+0000","last_undegraded":"2021-02-10T04:55:56.677684+0000","last_fullsized":"2021-02-10T04:55:56.677684+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1210,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":149,"num_read_kb":148,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"1105","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.765331+0000","last_change":"2021-02-10T04:53:46.729138+0000","last_active":"2021-02-10T04:55:56.765331+0000","last_peered":"2021-02-10T04:55:56.765331+0000","last_clean":"2021-02-10T04:55:56.765331+0000","last_became_active":"2021-02-10T04:53:46.728845+0000","last_became_peered":"2021-02-10T04:53:46.728845+0000","last_unstale":"2021-02-10T04:55:56.765331+0000","last_undegraded":"2021-02-10T04:55:56.765331+0000","last_fullsized":"2021-02-10T04:55:56.765331+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:40.066354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:52:40.066354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"1000'620","reported_seq":"4352","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.704103+0000","last_change":"2021-02-10T04:55:28.089086+0000","last_active":"2021-02-10T04:55:56.704103+0000","last_peered":"2021-02-10T04:55:56.704103+0000","last_clean":"2021-02-10T04:55:56.704103+0000","last_became_active":"2021-02-10T04:55:14.640509+0000","last_became_peered":"2021-02-10T04:55:14.640509+0000","last_unstale":"2021-02-10T04:55:56.704103+0000","last_undegraded":"2021-02-10T04:55:56.704103+0000","last_fullsized":"2021-02-10T04:55:56.704103+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:51:54.444287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:54.444287+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14265195,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":144,"num_read_kb":128482,"num_write":129,"num_write_kb":53443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"1141","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.703981+0000","last_change":"2021-02-10T04:55:28.083286+0000","last_active":"2021-02-10T04:55:56.703981+0000","last_peered":"2021-02-10T04:55:56.703981+0000","last_clean":"2021-02-10T04:55:56.703981+0000","last_became_active":"2021-02-10T04:55:14.619846+0000","last_became_peered":"2021-02-10T04:55:14.619846+0000","last_unstale":"2021-02-10T04:55:56.703981+0000","last_undegraded":"2021-02-10T04:55:56.703981+0000","last_fullsized":"2021-02-10T04:55:56.703981+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:14.955544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:47:14.955544+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"1238","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.677922+0000","last_change":"2021-02-10T04:55:15.590025+0000","last_active":"2021-02-10T04:55:56.677922+0000","last_peered":"2021-02-10T04:55:56.677922+0000","last_clean":"2021-02-10T04:55:56.677922+0000","last_became_active":"2021-02-10T04:55:15.589651+0000","last_became_peered":"2021-02-10T04:55:15.589651+0000","last_unstale":"2021-02-10T04:55:56.677922+0000","last_undegraded":"2021-02-10T04:55:56.677922+0000","last_fullsized":"2021-02-10T04:55:56.677922+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"1244","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.704100+0000","last_change":"2021-02-10T04:55:28.090104+0000","last_active":"2021-02-10T04:55:56.704100+0000","last_peered":"2021-02-10T04:55:56.704100+0000","last_clean":"2021-02-10T04:55:56.704100+0000","last_became_active":"2021-02-10T04:55:14.650106+0000","last_became_peered":"2021-02-10T04:55:14.650106+0000","last_unstale":"2021-02-10T04:55:56.704100+0000","last_undegraded":"2021-02-10T04:55:56.704100+0000","last_fullsized":"2021-02-10T04:55:56.704100+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:27.908221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:47:27.908221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"1261'67","reported_seq":"912","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:57.126085+0000","last_change":"2021-02-10T04:52:30.277009+0000","last_active":"2021-02-10T04:55:57.126085+0000","last_peered":"2021-02-10T04:55:57.126085+0000","last_clean":"2021-02-10T04:55:57.126085+0000","last_became_active":"2021-02-10T04:51:49.501622+0000","last_became_peered":"2021-02-10T04:51:49.501622+0000","last_unstale":"2021-02-10T04:55:57.126085+0000","last_undegraded":"2021-02-10T04:55:57.126085+0000","last_fullsized":"2021-02-10T04:55:57.126085+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'55","last_scrub_stamp":"2021-02-10T04:52:30.276938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.276938+0000","log_size":67,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":116,"num_read_kb":92,"num_write":70,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"1256","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.678839+0000","last_change":"2021-02-10T04:51:59.420149+0000","last_active":"2021-02-10T04:55:56.678839+0000","last_peered":"2021-02-10T04:55:56.678839+0000","last_clean":"2021-02-10T04:55:56.678839+0000","last_became_active":"2021-02-10T04:28:21.125541+0000","last_became_peered":"2021-02-10T04:28:21.125541+0000","last_unstale":"2021-02-10T04:55:56.678839+0000","last_undegraded":"2021-02-10T04:55:56.678839+0000","last_fullsized":"2021-02-10T04:55:56.678839+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.420086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.420086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"1222","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.705760+0000","last_change":"2021-02-10T04:53:12.187225+0000","last_active":"2021-02-10T04:55:56.705760+0000","last_peered":"2021-02-10T04:55:56.705760+0000","last_clean":"2021-02-10T04:55:56.705760+0000","last_became_active":"2021-02-10T04:50:59.555689+0000","last_became_peered":"2021-02-10T04:50:59.555689+0000","last_unstale":"2021-02-10T04:55:56.705760+0000","last_undegraded":"2021-02-10T04:55:56.705760+0000","last_fullsized":"2021-02-10T04:55:56.705760+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.187154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.187154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"1002'2042","reported_seq":"9494","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:55:59.689107+0000","last_change":"2021-02-10T04:55:14.662208+0000","last_active":"2021-02-10T04:55:59.689107+0000","last_peered":"2021-02-10T04:55:59.689107+0000","last_clean":"2021-02-10T04:55:59.689107+0000","last_became_active":"2021-02-10T04:55:14.659601+0000","last_became_peered":"2021-02-10T04:55:14.659601+0000","last_unstale":"2021-02-10T04:55:59.689107+0000","last_undegraded":"2021-02-10T04:55:59.689107+0000","last_fullsized":"2021-02-10T04:55:59.689107+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:46:57.054230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:57.054230+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":85678898,"num_objects":35,"num_object_clones":29,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"1207","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.678077+0000","last_change":"2021-02-10T04:55:14.651301+0000","last_active":"2021-02-10T04:55:56.678077+0000","last_peered":"2021-02-10T04:55:56.678077+0000","last_clean":"2021-02-10T04:55:56.678077+0000","last_became_active":"2021-02-10T04:55:14.650386+0000","last_became_peered":"2021-02-10T04:55:14.650386+0000","last_unstale":"2021-02-10T04:55:56.678077+0000","last_undegraded":"2021-02-10T04:55:56.678077+0000","last_fullsized":"2021-02-10T04:55:56.678077+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:52.341527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:50:52.341527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"60","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.679258+0000","last_change":"2021-02-10T04:54:39.558380+0000","last_active":"2021-02-10T04:55:56.679258+0000","last_peered":"2021-02-10T04:55:56.679258+0000","last_clean":"2021-02-10T04:55:56.679258+0000","last_became_active":"2021-02-10T04:54:39.557995+0000","last_became_peered":"2021-02-10T04:54:39.557995+0000","last_unstale":"2021-02-10T04:55:56.679258+0000","last_undegraded":"2021-02-10T04:55:56.679258+0000","last_fullsized":"2021-02-10T04:55:56.679258+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"1238'4","reported_seq":"1443","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:58.042053+0000","last_change":"2021-02-10T04:55:28.095426+0000","last_active":"2021-02-10T04:55:58.042053+0000","last_peered":"2021-02-10T04:55:58.042053+0000","last_clean":"2021-02-10T04:55:58.042053+0000","last_became_active":"2021-02-10T04:55:14.645890+0000","last_became_peered":"2021-02-10T04:55:14.645890+0000","last_unstale":"2021-02-10T04:55:58.042053+0000","last_undegraded":"2021-02-10T04:55:58.042053+0000","last_fullsized":"2021-02-10T04:55:58.042053+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:54.466259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:54.466259+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"1261'64","reported_seq":"958","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:57.123514+0000","last_change":"2021-02-10T04:53:01.304693+0000","last_active":"2021-02-10T04:55:57.123514+0000","last_peered":"2021-02-10T04:55:57.123514+0000","last_clean":"2021-02-10T04:55:57.123514+0000","last_became_active":"2021-02-10T04:51:49.498066+0000","last_became_peered":"2021-02-10T04:51:49.498066+0000","last_unstale":"2021-02-10T04:55:57.123514+0000","last_undegraded":"2021-02-10T04:55:57.123514+0000","last_fullsized":"2021-02-10T04:55:57.123514+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:53:01.304626+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.304626+0000","log_size":64,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":89,"num_read_kb":71,"num_write":52,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"519","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.678650+0000","last_change":"2021-02-10T04:55:14.610552+0000","last_active":"2021-02-10T04:55:56.678650+0000","last_peered":"2021-02-10T04:55:56.678650+0000","last_clean":"2021-02-10T04:55:56.678650+0000","last_became_active":"2021-02-10T04:55:14.609458+0000","last_became_peered":"2021-02-10T04:55:14.609458+0000","last_unstale":"2021-02-10T04:55:56.678650+0000","last_undegraded":"2021-02-10T04:55:56.678650+0000","last_fullsized":"2021-02-10T04:55:56.678650+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:49.447961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:49.447961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"1194","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:56:01.105875+0000","last_change":"2021-02-10T04:56:01.106613+0000","last_active":"2021-02-10T04:56:01.105875+0000","last_peered":"2021-02-10T04:56:01.105875+0000","last_clean":"2021-02-10T04:56:01.105875+0000","last_became_active":"2021-02-10T04:55:14.614581+0000","last_became_peered":"2021-02-10T04:55:14.614581+0000","last_unstale":"2021-02-10T04:56:01.105875+0000","last_undegraded":"2021-02-10T04:56:01.105875+0000","last_fullsized":"2021-02-10T04:56:01.105875+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:07.914061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:07.914061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"293'77","reported_seq":"3043","reported_epoch":"1261","state":"active+clean+scrubbing","last_fresh":"2021-02-10T04:55:56.677733+0000","last_change":"2021-02-10T04:53:43.102301+0000","last_active":"2021-02-10T04:55:56.677733+0000","last_peered":"2021-02-10T04:55:56.677733+0000","last_clean":"2021-02-10T04:55:56.677733+0000","last_became_active":"2021-02-10T04:52:26.024812+0000","last_became_peered":"2021-02-10T04:52:26.024812+0000","last_unstale":"2021-02-10T04:55:56.677733+0000","last_undegraded":"2021-02-10T04:55:56.677733+0000","last_fullsized":"2021-02-10T04:55:56.677733+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1886585,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":18587,"num_write":9,"num_write_kb":3315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"1276","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:55:59.689059+0000","last_change":"2021-02-10T04:53:07.025068+0000","last_active":"2021-02-10T04:55:59.689059+0000","last_peered":"2021-02-10T04:55:59.689059+0000","last_clean":"2021-02-10T04:55:59.689059+0000","last_became_active":"2021-02-10T04:50:59.589417+0000","last_became_peered":"2021-02-10T04:50:59.589417+0000","last_unstale":"2021-02-10T04:55:59.689059+0000","last_undegraded":"2021-02-10T04:55:59.689059+0000","last_fullsized":"2021-02-10T04:55:59.689059+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.024980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.024980+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"188","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:58.720112+0000","last_change":"2021-02-10T04:55:58.720112+0000","last_active":"2021-02-10T04:55:58.720112+0000","last_peered":"2021-02-10T04:55:58.720112+0000","last_clean":"2021-02-10T04:55:58.720112+0000","last_became_active":"2021-02-10T04:55:14.605267+0000","last_became_peered":"2021-02-10T04:55:14.605267+0000","last_unstale":"2021-02-10T04:55:58.720112+0000","last_undegraded":"2021-02-10T04:55:58.720112+0000","last_fullsized":"2021-02-10T04:55:58.720112+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.609500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.609500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"1261'86","reported_seq":"1442","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:57.158541+0000","last_change":"2021-02-10T04:55:35.577794+0000","last_active":"2021-02-10T04:55:57.158541+0000","last_peered":"2021-02-10T04:55:57.158541+0000","last_clean":"2021-02-10T04:55:57.158541+0000","last_became_active":"2021-02-10T04:55:35.577316+0000","last_became_peered":"2021-02-10T04:55:35.577316+0000","last_unstale":"2021-02-10T04:55:57.158541+0000","last_undegraded":"2021-02-10T04:55:57.158541+0000","last_fullsized":"2021-02-10T04:55:57.158541+0000","mapping_epoch":1238,"log_start":"1049'68","ondisk_log_start":"1049'68","created":55,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'39","last_scrub_stamp":"2021-02-10T04:53:38.085565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.085565+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1582,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":227,"num_read_kb":187,"num_write":128,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"1375","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.679649+0000","last_change":"2021-02-10T04:55:14.659217+0000","last_active":"2021-02-10T04:55:56.679649+0000","last_peered":"2021-02-10T04:55:56.679649+0000","last_clean":"2021-02-10T04:55:56.679649+0000","last_became_active":"2021-02-10T04:55:14.658879+0000","last_became_peered":"2021-02-10T04:55:14.658879+0000","last_unstale":"2021-02-10T04:55:56.679649+0000","last_undegraded":"2021-02-10T04:55:56.679649+0000","last_fullsized":"2021-02-10T04:55:56.679649+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:47:08.784593+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:47:08.784593+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"1224","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.705423+0000","last_change":"2021-02-10T04:52:30.019225+0000","last_active":"2021-02-10T04:55:56.705423+0000","last_peered":"2021-02-10T04:55:56.705423+0000","last_clean":"2021-02-10T04:55:56.705423+0000","last_became_active":"2021-02-10T04:50:59.561436+0000","last_became_peered":"2021-02-10T04:50:59.561436+0000","last_unstale":"2021-02-10T04:55:56.705423+0000","last_undegraded":"2021-02-10T04:55:56.705423+0000","last_fullsized":"2021-02-10T04:55:56.705423+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.019102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.019102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"1001'1528","reported_seq":"6209","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.704881+0000","last_change":"2021-02-10T04:52:26.036366+0000","last_active":"2021-02-10T04:55:56.704881+0000","last_peered":"2021-02-10T04:55:56.704881+0000","last_clean":"2021-02-10T04:55:56.704881+0000","last_became_active":"2021-02-10T04:52:26.034028+0000","last_became_peered":"2021-02-10T04:52:26.034028+0000","last_unstale":"2021-02-10T04:55:56.704881+0000","last_undegraded":"2021-02-10T04:55:56.704881+0000","last_fullsized":"2021-02-10T04:55:56.704881+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46944136,"num_objects":16,"num_object_clones":15,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":358,"num_read_kb":351934,"num_write":286,"num_write_kb":111355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":58968832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.15","version":"1000'446","reported_seq":"2198","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.679806+0000","last_change":"2021-02-10T04:53:11.051124+0000","last_active":"2021-02-10T04:55:56.679806+0000","last_peered":"2021-02-10T04:55:56.679806+0000","last_clean":"2021-02-10T04:55:56.679806+0000","last_became_active":"2021-02-10T04:51:40.510899+0000","last_became_peered":"2021-02-10T04:51:40.510899+0000","last_unstale":"2021-02-10T04:55:56.679806+0000","last_undegraded":"2021-02-10T04:55:56.679806+0000","last_fullsized":"2021-02-10T04:55:56.679806+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:53:11.051031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.051031+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24896285,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":18983408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"172","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.675599+0000","last_change":"2021-02-10T04:55:14.605585+0000","last_active":"2021-02-10T04:55:56.675599+0000","last_peered":"2021-02-10T04:55:56.675599+0000","last_clean":"2021-02-10T04:55:56.675599+0000","last_became_active":"2021-02-10T04:55:14.604916+0000","last_became_peered":"2021-02-10T04:55:14.604916+0000","last_unstale":"2021-02-10T04:55:56.675599+0000","last_undegraded":"2021-02-10T04:55:56.675599+0000","last_fullsized":"2021-02-10T04:55:56.675599+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:51:08.639784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.639784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"1193","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.706226+0000","last_change":"2021-02-10T04:51:09.475150+0000","last_active":"2021-02-10T04:55:56.706226+0000","last_peered":"2021-02-10T04:55:56.706226+0000","last_clean":"2021-02-10T04:55:56.706226+0000","last_became_active":"2021-02-10T04:50:59.585917+0000","last_became_peered":"2021-02-10T04:50:59.585917+0000","last_unstale":"2021-02-10T04:55:56.706226+0000","last_undegraded":"2021-02-10T04:55:56.706226+0000","last_fullsized":"2021-02-10T04:55:56.706226+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:09.475063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:09.475063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"163","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.677251+0000","last_change":"2021-02-10T04:53:34.927483+0000","last_active":"2021-02-10T04:55:56.677251+0000","last_peered":"2021-02-10T04:55:56.677251+0000","last_clean":"2021-02-10T04:55:56.677251+0000","last_became_active":"2021-02-10T04:51:36.859707+0000","last_became_peered":"2021-02-10T04:51:36.859707+0000","last_unstale":"2021-02-10T04:55:56.677251+0000","last_undegraded":"2021-02-10T04:55:56.677251+0000","last_fullsized":"2021-02-10T04:55:56.677251+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:34.927424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.927424+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"1000'620","reported_seq":"4548","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.705363+0000","last_change":"2021-02-10T04:53:37.384461+0000","last_active":"2021-02-10T04:55:56.705363+0000","last_peered":"2021-02-10T04:55:56.705363+0000","last_clean":"2021-02-10T04:55:56.705363+0000","last_became_active":"2021-02-10T04:51:40.510114+0000","last_became_peered":"2021-02-10T04:51:40.510114+0000","last_unstale":"2021-02-10T04:55:56.705363+0000","last_undegraded":"2021-02-10T04:55:56.705363+0000","last_fullsized":"2021-02-10T04:55:56.705363+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:53:37.384398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:37.384398+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28530390,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"1250","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:55:59.689469+0000","last_change":"2021-02-10T04:52:41.059269+0000","last_active":"2021-02-10T04:55:59.689469+0000","last_peered":"2021-02-10T04:55:59.689469+0000","last_clean":"2021-02-10T04:55:59.689469+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:55:59.689469+0000","last_undegraded":"2021-02-10T04:55:59.689469+0000","last_fullsized":"2021-02-10T04:55:59.689469+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:52:41.059159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.059159+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"1253","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.678962+0000","last_change":"2021-02-10T04:53:12.332890+0000","last_active":"2021-02-10T04:55:56.678962+0000","last_peered":"2021-02-10T04:55:56.678962+0000","last_clean":"2021-02-10T04:55:56.678962+0000","last_became_active":"2021-02-10T04:50:59.540375+0000","last_became_peered":"2021-02-10T04:50:59.540375+0000","last_unstale":"2021-02-10T04:55:56.678962+0000","last_undegraded":"2021-02-10T04:55:56.678962+0000","last_fullsized":"2021-02-10T04:55:56.678962+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.332233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.332233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"200","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:59.725634+0000","last_change":"2021-02-10T04:55:59.725634+0000","last_active":"2021-02-10T04:55:59.725634+0000","last_peered":"2021-02-10T04:55:59.725634+0000","last_clean":"2021-02-10T04:55:59.725634+0000","last_became_active":"2021-02-10T04:53:46.733191+0000","last_became_peered":"2021-02-10T04:53:46.733191+0000","last_unstale":"2021-02-10T04:55:59.725634+0000","last_undegraded":"2021-02-10T04:55:59.725634+0000","last_fullsized":"2021-02-10T04:55:59.725634+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:45.362537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:45.362537+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"1002'2042","reported_seq":"9234","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.765213+0000","last_change":"2021-02-10T04:53:46.726353+0000","last_active":"2021-02-10T04:55:56.765213+0000","last_peered":"2021-02-10T04:55:56.765213+0000","last_clean":"2021-02-10T04:55:56.765213+0000","last_became_active":"2021-02-10T04:53:46.725828+0000","last_became_peered":"2021-02-10T04:53:46.725828+0000","last_unstale":"2021-02-10T04:55:56.765213+0000","last_undegraded":"2021-02-10T04:55:56.765213+0000","last_fullsized":"2021-02-10T04:55:56.765213+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:52:41.286006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.286006+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46300076,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":545,"num_read_kb":531689,"num_write":399,"num_write_kb":153087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"1261'56","reported_seq":"1432","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:57.117075+0000","last_change":"2021-02-10T04:52:35.091494+0000","last_active":"2021-02-10T04:55:57.117075+0000","last_peered":"2021-02-10T04:55:57.117075+0000","last_clean":"2021-02-10T04:55:57.117075+0000","last_became_active":"2021-02-10T04:50:59.569915+0000","last_became_peered":"2021-02-10T04:50:59.569915+0000","last_unstale":"2021-02-10T04:55:57.117075+0000","last_undegraded":"2021-02-10T04:55:57.117075+0000","last_fullsized":"2021-02-10T04:55:57.117075+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'46","last_scrub_stamp":"2021-02-10T04:52:35.091374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:35.091374+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":95,"num_read_kb":80,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"84'7","reported_seq":"1277","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.678687+0000","last_change":"2021-02-10T04:51:52.498251+0000","last_active":"2021-02-10T04:55:56.678687+0000","last_peered":"2021-02-10T04:55:56.678687+0000","last_clean":"2021-02-10T04:55:56.678687+0000","last_became_active":"2021-02-10T04:50:59.575924+0000","last_became_peered":"2021-02-10T04:50:59.575924+0000","last_unstale":"2021-02-10T04:55:56.678687+0000","last_undegraded":"2021-02-10T04:55:56.678687+0000","last_fullsized":"2021-02-10T04:55:56.678687+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'7","last_scrub_stamp":"2021-02-10T04:51:52.498158+0000","last_deep_scrub":"84'7","last_deep_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.498158+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"1229'4","reported_seq":"1614","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:55:59.686141+0000","last_change":"2021-02-10T04:55:14.657278+0000","last_active":"2021-02-10T04:55:59.686141+0000","last_peered":"2021-02-10T04:55:59.686141+0000","last_clean":"2021-02-10T04:55:59.686141+0000","last_became_active":"2021-02-10T04:55:14.656867+0000","last_became_peered":"2021-02-10T04:55:14.656867+0000","last_unstale":"2021-02-10T04:55:59.686141+0000","last_undegraded":"2021-02-10T04:55:59.686141+0000","last_fullsized":"2021-02-10T04:55:59.686141+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:47.407557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:47.407557+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"1175","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.705737+0000","last_change":"2021-02-10T04:55:22.735243+0000","last_active":"2021-02-10T04:55:56.705737+0000","last_peered":"2021-02-10T04:55:56.705737+0000","last_clean":"2021-02-10T04:55:56.705737+0000","last_became_active":"2021-02-10T04:55:14.632398+0000","last_became_peered":"2021-02-10T04:55:14.632398+0000","last_unstale":"2021-02-10T04:55:56.705737+0000","last_undegraded":"2021-02-10T04:55:56.705737+0000","last_fullsized":"2021-02-10T04:55:56.705737+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:23.566027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_clean_scrub_stamp":"2021-02-10T04:52:23.566027+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"1000'490","reported_seq":"3777","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.679526+0000","last_change":"2021-02-10T04:55:14.644527+0000","last_active":"2021-02-10T04:55:56.679526+0000","last_peered":"2021-02-10T04:55:56.679526+0000","last_clean":"2021-02-10T04:55:56.679526+0000","last_became_active":"2021-02-10T04:55:14.643764+0000","last_became_peered":"2021-02-10T04:55:14.643764+0000","last_unstale":"2021-02-10T04:55:56.679526+0000","last_undegraded":"2021-02-10T04:55:56.679526+0000","last_fullsized":"2021-02-10T04:55:56.679526+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:50:03.756248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:50:03.756248+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16154790,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"1261'46","reported_seq":"1330","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:57.120363+0000","last_change":"2021-02-10T04:53:10.276449+0000","last_active":"2021-02-10T04:55:57.120363+0000","last_peered":"2021-02-10T04:55:57.120363+0000","last_clean":"2021-02-10T04:55:57.120363+0000","last_became_active":"2021-02-10T04:50:59.539529+0000","last_became_peered":"2021-02-10T04:50:59.539529+0000","last_unstale":"2021-02-10T04:55:57.120363+0000","last_undegraded":"2021-02-10T04:55:57.120363+0000","last_fullsized":"2021-02-10T04:55:57.120363+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1112'40","last_scrub_stamp":"2021-02-10T04:53:10.276358+0000","last_deep_scrub":"1112'40","last_deep_scrub_stamp":"2021-02-10T04:51:01.450050+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.276358+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":53,"num_write":38,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"1160","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.705094+0000","last_change":"2021-02-10T04:55:28.088480+0000","last_active":"2021-02-10T04:55:56.705094+0000","last_peered":"2021-02-10T04:55:56.705094+0000","last_clean":"2021-02-10T04:55:56.705094+0000","last_became_active":"2021-02-10T04:55:14.631666+0000","last_became_peered":"2021-02-10T04:55:14.631666+0000","last_unstale":"2021-02-10T04:55:56.705094+0000","last_undegraded":"2021-02-10T04:55:56.705094+0000","last_fullsized":"2021-02-10T04:55:56.705094+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:23.904210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:47:23.904210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"1119'6","reported_seq":"1527","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:58.042126+0000","last_change":"2021-02-10T04:52:40.038646+0000","last_active":"2021-02-10T04:55:58.042126+0000","last_peered":"2021-02-10T04:55:58.042126+0000","last_clean":"2021-02-10T04:55:58.042126+0000","last_became_active":"2021-02-10T04:51:36.869733+0000","last_became_peered":"2021-02-10T04:51:36.869733+0000","last_unstale":"2021-02-10T04:55:58.042126+0000","last_undegraded":"2021-02-10T04:55:58.042126+0000","last_fullsized":"2021-02-10T04:55:58.042126+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'6","last_scrub_stamp":"2021-02-10T04:52:40.038554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:40.038554+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"1247","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.679825+0000","last_change":"2021-02-10T04:51:36.849933+0000","last_active":"2021-02-10T04:55:56.679825+0000","last_peered":"2021-02-10T04:55:56.679825+0000","last_clean":"2021-02-10T04:55:56.679825+0000","last_became_active":"2021-02-10T04:51:36.849184+0000","last_became_peered":"2021-02-10T04:51:36.849184+0000","last_unstale":"2021-02-10T04:55:56.679825+0000","last_undegraded":"2021-02-10T04:55:56.679825+0000","last_fullsized":"2021-02-10T04:55:56.679825+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:11.779779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:11.779779+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"1032'89","reported_seq":"855","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.763443+0000","last_change":"2021-02-10T04:53:46.728868+0000","last_active":"2021-02-10T04:55:56.763443+0000","last_peered":"2021-02-10T04:55:56.763443+0000","last_clean":"2021-02-10T04:55:56.763443+0000","last_became_active":"2021-02-10T04:53:46.728356+0000","last_became_peered":"2021-02-10T04:53:46.728356+0000","last_unstale":"2021-02-10T04:55:56.763443+0000","last_undegraded":"2021-02-10T04:55:56.763443+0000","last_fullsized":"2021-02-10T04:55:56.763443+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":5,"last_scrub":"1032'89","last_scrub_stamp":"2021-02-10T04:52:34.089789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:34.089789+0000","log_size":89,"ondisk_log_size":89,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5218574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":30326,"num_write":21,"num_write_kb":12444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3210022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"1261'43","reported_seq":"951","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.996888+0000","last_change":"2021-02-10T04:53:14.202505+0000","last_active":"2021-02-10T04:55:56.996888+0000","last_peered":"2021-02-10T04:55:56.996888+0000","last_clean":"2021-02-10T04:55:56.996888+0000","last_became_active":"2021-02-10T04:51:49.496294+0000","last_became_peered":"2021-02-10T04:51:49.496294+0000","last_unstale":"2021-02-10T04:55:56.996888+0000","last_undegraded":"2021-02-10T04:55:56.996888+0000","last_fullsized":"2021-02-10T04:55:56.996888+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1163'37","last_scrub_stamp":"2021-02-10T04:53:14.202445+0000","last_deep_scrub":"1163'37","last_deep_scrub_stamp":"2021-02-10T04:53:14.202445+0000","last_clean_scrub_stamp":"2021-02-10T04:53:14.202445+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":60,"num_read_kb":42,"num_write":34,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"1263","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:55:59.687813+0000","last_change":"2021-02-10T04:52:23.095651+0000","last_active":"2021-02-10T04:55:59.687813+0000","last_peered":"2021-02-10T04:55:59.687813+0000","last_clean":"2021-02-10T04:55:59.687813+0000","last_became_active":"2021-02-10T04:50:59.543531+0000","last_became_peered":"2021-02-10T04:50:59.543531+0000","last_unstale":"2021-02-10T04:55:59.687813+0000","last_undegraded":"2021-02-10T04:55:59.687813+0000","last_fullsized":"2021-02-10T04:55:59.687813+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:23.095588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:23.095588+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"1212","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.705157+0000","last_change":"2021-02-10T04:55:14.636711+0000","last_active":"2021-02-10T04:55:56.705157+0000","last_peered":"2021-02-10T04:55:56.705157+0000","last_clean":"2021-02-10T04:55:56.705157+0000","last_became_active":"2021-02-10T04:55:14.636363+0000","last_became_peered":"2021-02-10T04:55:14.636363+0000","last_unstale":"2021-02-10T04:55:56.705157+0000","last_undegraded":"2021-02-10T04:55:56.705157+0000","last_fullsized":"2021-02-10T04:55:56.705157+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:43.663762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:51:43.663762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"1131","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.703674+0000","last_change":"2021-02-10T04:55:28.101709+0000","last_active":"2021-02-10T04:55:56.703674+0000","last_peered":"2021-02-10T04:55:56.703674+0000","last_clean":"2021-02-10T04:55:56.703674+0000","last_became_active":"2021-02-10T04:55:14.635912+0000","last_became_peered":"2021-02-10T04:55:14.635912+0000","last_unstale":"2021-02-10T04:55:56.703674+0000","last_undegraded":"2021-02-10T04:55:56.703674+0000","last_fullsized":"2021-02-10T04:55:56.703674+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:57.568541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:57.568541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"1000'1071","reported_seq":"4767","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.703725+0000","last_change":"2021-02-10T04:53:35.302879+0000","last_active":"2021-02-10T04:55:56.703725+0000","last_peered":"2021-02-10T04:55:56.703725+0000","last_clean":"2021-02-10T04:55:56.703725+0000","last_became_active":"2021-02-10T04:51:39.508551+0000","last_became_peered":"2021-02-10T04:51:39.508551+0000","last_unstale":"2021-02-10T04:55:56.703725+0000","last_undegraded":"2021-02-10T04:55:56.703725+0000","last_fullsized":"2021-02-10T04:55:56.703725+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:35.302780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.302780+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"1261'39","reported_seq":"1296","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:57.104198+0000","last_change":"2021-02-10T04:52:39.695340+0000","last_active":"2021-02-10T04:55:57.104198+0000","last_peered":"2021-02-10T04:55:57.104198+0000","last_clean":"2021-02-10T04:55:57.104198+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:55:57.104198+0000","last_undegraded":"2021-02-10T04:55:57.104198+0000","last_fullsized":"2021-02-10T04:55:57.104198+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'31","last_scrub_stamp":"2021-02-10T04:52:39.695237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:39.695237+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":53,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"506","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.678979+0000","last_change":"2021-02-10T04:55:20.929399+0000","last_active":"2021-02-10T04:55:56.678979+0000","last_peered":"2021-02-10T04:55:56.678979+0000","last_clean":"2021-02-10T04:55:56.678979+0000","last_became_active":"2021-02-10T04:55:14.655543+0000","last_became_peered":"2021-02-10T04:55:14.655543+0000","last_unstale":"2021-02-10T04:55:56.678979+0000","last_undegraded":"2021-02-10T04:55:56.678979+0000","last_fullsized":"2021-02-10T04:55:56.678979+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:55:20.929330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:55:20.929330+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"914","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.689867+0000","last_change":"2021-02-10T04:53:35.216489+0000","last_active":"2021-02-10T04:55:56.689867+0000","last_peered":"2021-02-10T04:55:56.689867+0000","last_clean":"2021-02-10T04:55:56.689867+0000","last_became_active":"2021-02-10T04:51:49.501254+0000","last_became_peered":"2021-02-10T04:51:49.501254+0000","last_unstale":"2021-02-10T04:55:56.689867+0000","last_undegraded":"2021-02-10T04:55:56.689867+0000","last_fullsized":"2021-02-10T04:55:56.689867+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.216394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.216394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"1239","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.703699+0000","last_change":"2021-02-10T04:55:14.633276+0000","last_active":"2021-02-10T04:55:56.703699+0000","last_peered":"2021-02-10T04:55:56.703699+0000","last_clean":"2021-02-10T04:55:56.703699+0000","last_became_active":"2021-02-10T04:55:14.632805+0000","last_became_peered":"2021-02-10T04:55:14.632805+0000","last_unstale":"2021-02-10T04:55:56.703699+0000","last_undegraded":"2021-02-10T04:55:56.703699+0000","last_fullsized":"2021-02-10T04:55:56.703699+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:55.814486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.814486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"1002'1657","reported_seq":"5284","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.705021+0000","last_change":"2021-02-10T04:55:22.728878+0000","last_active":"2021-02-10T04:55:56.705021+0000","last_peered":"2021-02-10T04:55:56.705021+0000","last_clean":"2021-02-10T04:55:56.705021+0000","last_became_active":"2021-02-10T04:55:14.653270+0000","last_became_peered":"2021-02-10T04:55:14.653270+0000","last_unstale":"2021-02-10T04:55:56.705021+0000","last_undegraded":"2021-02-10T04:55:56.705021+0000","last_fullsized":"2021-02-10T04:55:56.705021+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:55:22.331010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:55:22.331010+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":47335497,"num_objects":20,"num_object_clones":18,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"1261'24","reported_seq":"1369","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.893183+0000","last_change":"2021-02-10T04:53:04.058324+0000","last_active":"2021-02-10T04:55:56.893183+0000","last_peered":"2021-02-10T04:55:56.893183+0000","last_clean":"2021-02-10T04:55:56.893183+0000","last_became_active":"2021-02-10T04:50:59.544484+0000","last_became_peered":"2021-02-10T04:50:59.544484+0000","last_unstale":"2021-02-10T04:55:56.893183+0000","last_undegraded":"2021-02-10T04:55:56.893183+0000","last_fullsized":"2021-02-10T04:55:56.893183+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'20","last_scrub_stamp":"2021-02-10T04:53:04.058248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:04.058248+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":29,"num_write":24,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"1265","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.679134+0000","last_change":"2021-02-10T04:53:00.343307+0000","last_active":"2021-02-10T04:55:56.679134+0000","last_peered":"2021-02-10T04:55:56.679134+0000","last_clean":"2021-02-10T04:55:56.679134+0000","last_became_active":"2021-02-10T04:51:36.844573+0000","last_became_peered":"2021-02-10T04:51:36.844573+0000","last_unstale":"2021-02-10T04:55:56.679134+0000","last_undegraded":"2021-02-10T04:55:56.679134+0000","last_fullsized":"2021-02-10T04:55:56.679134+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:00.343228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:00.343228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"1234","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:55:59.687363+0000","last_change":"2021-02-10T04:50:56.099710+0000","last_active":"2021-02-10T04:55:59.687363+0000","last_peered":"2021-02-10T04:55:59.687363+0000","last_clean":"2021-02-10T04:55:59.687363+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:55:59.687363+0000","last_undegraded":"2021-02-10T04:55:59.687363+0000","last_fullsized":"2021-02-10T04:55:59.687363+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:56.099651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:50:56.099651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"580","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.763693+0000","last_change":"2021-02-10T04:53:46.718715+0000","last_active":"2021-02-10T04:55:56.763693+0000","last_peered":"2021-02-10T04:55:56.763693+0000","last_clean":"2021-02-10T04:55:56.763693+0000","last_became_active":"2021-02-10T04:53:46.718266+0000","last_became_peered":"2021-02-10T04:53:46.718266+0000","last_unstale":"2021-02-10T04:55:56.763693+0000","last_undegraded":"2021-02-10T04:55:56.763693+0000","last_fullsized":"2021-02-10T04:55:56.763693+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.876681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.876681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"998'1009","reported_seq":"5110","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:55:59.687881+0000","last_change":"2021-02-10T04:55:14.635764+0000","last_active":"2021-02-10T04:55:59.687881+0000","last_peered":"2021-02-10T04:55:59.687881+0000","last_clean":"2021-02-10T04:55:59.687881+0000","last_became_active":"2021-02-10T04:55:14.634422+0000","last_became_peered":"2021-02-10T04:55:14.634422+0000","last_unstale":"2021-02-10T04:55:59.687881+0000","last_undegraded":"2021-02-10T04:55:59.687881+0000","last_fullsized":"2021-02-10T04:55:59.687881+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":5,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:13.032506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:13.032506+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49420827,"num_objects":21,"num_object_clones":19,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":36763798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"1261'45","reported_seq":"1303","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:57.148881+0000","last_change":"2021-02-10T04:51:44.694779+0000","last_active":"2021-02-10T04:55:57.148881+0000","last_peered":"2021-02-10T04:55:57.148881+0000","last_clean":"2021-02-10T04:55:57.148881+0000","last_became_active":"2021-02-10T04:50:59.549412+0000","last_became_peered":"2021-02-10T04:50:59.549412+0000","last_unstale":"2021-02-10T04:55:57.148881+0000","last_undegraded":"2021-02-10T04:55:57.148881+0000","last_fullsized":"2021-02-10T04:55:57.148881+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:44.694711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.694711+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":76,"num_read_kb":62,"num_write":45,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"1401","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:55:59.689302+0000","last_change":"2021-02-10T04:52:33.088525+0000","last_active":"2021-02-10T04:55:59.689302+0000","last_peered":"2021-02-10T04:55:59.689302+0000","last_clean":"2021-02-10T04:55:59.689302+0000","last_became_active":"2021-02-10T04:51:36.870573+0000","last_became_peered":"2021-02-10T04:51:36.870573+0000","last_unstale":"2021-02-10T04:55:59.689302+0000","last_undegraded":"2021-02-10T04:55:59.689302+0000","last_fullsized":"2021-02-10T04:55:59.689302+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:52:33.088428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:33.088428+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"547","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.704079+0000","last_change":"2021-02-10T04:51:51.836723+0000","last_active":"2021-02-10T04:55:56.704079+0000","last_peered":"2021-02-10T04:55:56.704079+0000","last_clean":"2021-02-10T04:55:56.704079+0000","last_became_active":"2021-02-10T04:50:59.564977+0000","last_became_peered":"2021-02-10T04:50:59.564977+0000","last_unstale":"2021-02-10T04:55:56.704079+0000","last_undegraded":"2021-02-10T04:55:56.704079+0000","last_fullsized":"2021-02-10T04:55:56.704079+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.836638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.836638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"1203","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.705115+0000","last_change":"2021-02-10T04:55:14.633046+0000","last_active":"2021-02-10T04:55:56.705115+0000","last_peered":"2021-02-10T04:55:56.705115+0000","last_clean":"2021-02-10T04:55:56.705115+0000","last_became_active":"2021-02-10T04:55:14.632658+0000","last_became_peered":"2021-02-10T04:55:14.632658+0000","last_unstale":"2021-02-10T04:55:56.705115+0000","last_undegraded":"2021-02-10T04:55:56.705115+0000","last_fullsized":"2021-02-10T04:55:56.705115+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:58.514318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_clean_scrub_stamp":"2021-02-10T04:46:58.514318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1d","version":"1001'1165","reported_seq":"4262","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.679009+0000","last_change":"2021-02-10T04:55:14.656236+0000","last_active":"2021-02-10T04:55:56.679009+0000","last_peered":"2021-02-10T04:55:56.679009+0000","last_clean":"2021-02-10T04:55:56.679009+0000","last_became_active":"2021-02-10T04:55:14.655859+0000","last_became_peered":"2021-02-10T04:55:14.655859+0000","last_unstale":"2021-02-10T04:55:56.679009+0000","last_undegraded":"2021-02-10T04:55:56.679009+0000","last_fullsized":"2021-02-10T04:55:56.679009+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":5,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29601845,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":287,"num_read_kb":299530,"num_write":200,"num_write_kb":89131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"1306","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.677698+0000","last_change":"2021-02-10T04:53:01.331566+0000","last_active":"2021-02-10T04:55:56.677698+0000","last_peered":"2021-02-10T04:55:56.677698+0000","last_clean":"2021-02-10T04:55:56.677698+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:55:56.677698+0000","last_undegraded":"2021-02-10T04:55:56.677698+0000","last_fullsized":"2021-02-10T04:55:56.677698+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:01.331466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.331466+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"1098'3","reported_seq":"1605","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:56:03.042621+0000","last_change":"2021-02-10T04:53:34.865668+0000","last_active":"2021-02-10T04:56:03.042621+0000","last_peered":"2021-02-10T04:56:03.042621+0000","last_clean":"2021-02-10T04:56:03.042621+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:56:03.042621+0000","last_undegraded":"2021-02-10T04:56:03.042621+0000","last_fullsized":"2021-02-10T04:56:03.042621+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"1098'3","last_scrub_stamp":"2021-02-10T04:53:34.865579+0000","last_deep_scrub":"1098'3","last_deep_scrub_stamp":"2021-02-10T04:50:15.255137+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.865579+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"1110","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.764891+0000","last_change":"2021-02-10T04:53:46.727603+0000","last_active":"2021-02-10T04:55:56.764891+0000","last_peered":"2021-02-10T04:55:56.764891+0000","last_clean":"2021-02-10T04:55:56.764891+0000","last_became_active":"2021-02-10T04:53:46.727053+0000","last_became_peered":"2021-02-10T04:53:46.727053+0000","last_unstale":"2021-02-10T04:55:56.764891+0000","last_undegraded":"2021-02-10T04:55:56.764891+0000","last_fullsized":"2021-02-10T04:55:56.764891+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:47.493057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.493057+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1c","version":"1002'1090","reported_seq":"5286","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.679455+0000","last_change":"2021-02-10T04:53:41.353991+0000","last_active":"2021-02-10T04:55:56.679455+0000","last_peered":"2021-02-10T04:55:56.679455+0000","last_clean":"2021-02-10T04:55:56.679455+0000","last_became_active":"2021-02-10T04:52:31.088265+0000","last_became_peered":"2021-02-10T04:52:31.088265+0000","last_unstale":"2021-02-10T04:55:56.679455+0000","last_undegraded":"2021-02-10T04:55:56.679455+0000","last_fullsized":"2021-02-10T04:55:56.679455+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:53:41.353840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:41.353840+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"1207","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.703955+0000","last_change":"2021-02-10T04:55:14.632817+0000","last_active":"2021-02-10T04:55:56.703955+0000","last_peered":"2021-02-10T04:55:56.703955+0000","last_clean":"2021-02-10T04:55:56.703955+0000","last_became_active":"2021-02-10T04:55:14.632214+0000","last_became_peered":"2021-02-10T04:55:14.632214+0000","last_unstale":"2021-02-10T04:55:56.703955+0000","last_undegraded":"2021-02-10T04:55:56.703955+0000","last_fullsized":"2021-02-10T04:55:56.703955+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:42.537443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:42.537443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"1156","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.705264+0000","last_change":"2021-02-10T04:55:22.725805+0000","last_active":"2021-02-10T04:55:56.705264+0000","last_peered":"2021-02-10T04:55:56.705264+0000","last_clean":"2021-02-10T04:55:56.705264+0000","last_became_active":"2021-02-10T04:55:14.630190+0000","last_became_peered":"2021-02-10T04:55:14.630190+0000","last_unstale":"2021-02-10T04:55:56.705264+0000","last_undegraded":"2021-02-10T04:55:56.705264+0000","last_fullsized":"2021-02-10T04:55:56.705264+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.938731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.938731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"471","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.678871+0000","last_change":"2021-02-10T04:53:15.185748+0000","last_active":"2021-02-10T04:55:56.678871+0000","last_peered":"2021-02-10T04:55:56.678871+0000","last_clean":"2021-02-10T04:55:56.678871+0000","last_became_active":"2021-02-10T04:51:49.499348+0000","last_became_peered":"2021-02-10T04:51:49.499348+0000","last_unstale":"2021-02-10T04:55:56.678871+0000","last_undegraded":"2021-02-10T04:55:56.678871+0000","last_fullsized":"2021-02-10T04:55:56.678871+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:15.185686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:15.185686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1f","version":"1004'1093","reported_seq":"6524","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:55:59.688960+0000","last_change":"2021-02-10T04:52:31.089952+0000","last_active":"2021-02-10T04:55:59.688960+0000","last_peered":"2021-02-10T04:55:59.688960+0000","last_clean":"2021-02-10T04:55:59.688960+0000","last_became_active":"2021-02-10T04:52:31.088634+0000","last_became_peered":"2021-02-10T04:52:31.088634+0000","last_unstale":"2021-02-10T04:55:59.688960+0000","last_undegraded":"2021-02-10T04:55:59.688960+0000","last_fullsized":"2021-02-10T04:55:59.688960+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39434697,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":279,"num_read_kb":311058,"num_write":186,"num_write_kb":66790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":53995913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"1264","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:55:59.688137+0000","last_change":"2021-02-10T04:55:39.651056+0000","last_active":"2021-02-10T04:55:59.688137+0000","last_peered":"2021-02-10T04:55:59.688137+0000","last_clean":"2021-02-10T04:55:59.688137+0000","last_became_active":"2021-02-10T04:55:39.596648+0000","last_became_peered":"2021-02-10T04:55:39.596648+0000","last_unstale":"2021-02-10T04:55:59.688137+0000","last_undegraded":"2021-02-10T04:55:59.688137+0000","last_fullsized":"2021-02-10T04:55:59.688137+0000","mapping_epoch":1248,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:52:36.083784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:36.083784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"501","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:56:02.021135+0000","last_change":"2021-02-10T04:56:02.021855+0000","last_active":"2021-02-10T04:56:02.021135+0000","last_peered":"2021-02-10T04:56:02.021135+0000","last_clean":"2021-02-10T04:56:02.021135+0000","last_became_active":"2021-02-10T04:55:14.655649+0000","last_became_peered":"2021-02-10T04:55:14.655649+0000","last_unstale":"2021-02-10T04:56:02.021135+0000","last_undegraded":"2021-02-10T04:56:02.021135+0000","last_fullsized":"2021-02-10T04:56:02.021135+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:08.758129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.758129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"1229","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.679381+0000","last_change":"2021-02-10T04:51:44.799512+0000","last_active":"2021-02-10T04:55:56.679381+0000","last_peered":"2021-02-10T04:55:56.679381+0000","last_clean":"2021-02-10T04:55:56.679381+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:55:56.679381+0000","last_undegraded":"2021-02-10T04:55:56.679381+0000","last_fullsized":"2021-02-10T04:55:56.679381+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:44.799429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.799429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1e","version":"1063'654","reported_seq":"4504","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.689645+0000","last_change":"2021-02-10T04:53:40.135450+0000","last_active":"2021-02-10T04:55:56.689645+0000","last_peered":"2021-02-10T04:55:56.689645+0000","last_clean":"2021-02-10T04:55:56.689645+0000","last_became_active":"2021-02-10T04:52:31.087662+0000","last_became_peered":"2021-02-10T04:52:31.087662+0000","last_unstale":"2021-02-10T04:55:56.689645+0000","last_undegraded":"2021-02-10T04:55:56.689645+0000","last_fullsized":"2021-02-10T04:55:56.689645+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:53:40.135367+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:53:40.135367+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"1213","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.705590+0000","last_change":"2021-02-10T04:51:08.006463+0000","last_active":"2021-02-10T04:55:56.705590+0000","last_peered":"2021-02-10T04:55:56.705590+0000","last_clean":"2021-02-10T04:55:56.705590+0000","last_became_active":"2021-02-10T04:50:59.563657+0000","last_became_peered":"2021-02-10T04:50:59.563657+0000","last_unstale":"2021-02-10T04:55:56.705590+0000","last_undegraded":"2021-02-10T04:55:56.705590+0000","last_fullsized":"2021-02-10T04:55:56.705590+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_clean_scrub_stamp":"2021-02-10T04:51:02.455702+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"1264","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.679115+0000","last_change":"2021-02-10T04:52:33.741801+0000","last_active":"2021-02-10T04:55:56.679115+0000","last_peered":"2021-02-10T04:55:56.679115+0000","last_clean":"2021-02-10T04:55:56.679115+0000","last_became_active":"2021-02-10T04:51:36.860129+0000","last_became_peered":"2021-02-10T04:51:36.860129+0000","last_unstale":"2021-02-10T04:55:56.679115+0000","last_undegraded":"2021-02-10T04:55:56.679115+0000","last_fullsized":"2021-02-10T04:55:56.679115+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:33.741711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:46:58.666264+0000","last_clean_scrub_stamp":"2021-02-10T04:52:33.741711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"1225","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.706088+0000","last_change":"2021-02-10T04:52:25.030419+0000","last_active":"2021-02-10T04:55:56.706088+0000","last_peered":"2021-02-10T04:55:56.706088+0000","last_clean":"2021-02-10T04:55:56.706088+0000","last_became_active":"2021-02-10T04:50:59.539582+0000","last_became_peered":"2021-02-10T04:50:59.539582+0000","last_unstale":"2021-02-10T04:55:56.706088+0000","last_undegraded":"2021-02-10T04:55:56.706088+0000","last_fullsized":"2021-02-10T04:55:56.706088+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:25.030281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.030281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"1003'739","reported_seq":"4011","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:55:59.687229+0000","last_change":"2021-02-10T04:53:01.929261+0000","last_active":"2021-02-10T04:55:59.687229+0000","last_peered":"2021-02-10T04:55:59.687229+0000","last_clean":"2021-02-10T04:55:59.687229+0000","last_became_active":"2021-02-10T04:51:40.511505+0000","last_became_peered":"2021-02-10T04:51:40.511505+0000","last_unstale":"2021-02-10T04:55:59.687229+0000","last_undegraded":"2021-02-10T04:55:59.687229+0000","last_fullsized":"2021-02-10T04:55:59.687229+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:53:01.929169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.929169+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22230846,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":18429397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"1101","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.764524+0000","last_change":"2021-02-10T04:53:46.727590+0000","last_active":"2021-02-10T04:55:56.764524+0000","last_peered":"2021-02-10T04:55:56.764524+0000","last_clean":"2021-02-10T04:55:56.764524+0000","last_became_active":"2021-02-10T04:53:46.726584+0000","last_became_peered":"2021-02-10T04:53:46.726584+0000","last_unstale":"2021-02-10T04:55:56.764524+0000","last_undegraded":"2021-02-10T04:55:56.764524+0000","last_fullsized":"2021-02-10T04:55:56.764524+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:26.408326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.408326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"1262","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.678886+0000","last_change":"2021-02-10T04:53:07.324650+0000","last_active":"2021-02-10T04:55:56.678886+0000","last_peered":"2021-02-10T04:55:56.678886+0000","last_clean":"2021-02-10T04:55:56.678886+0000","last_became_active":"2021-02-10T04:50:59.575340+0000","last_became_peered":"2021-02-10T04:50:59.575340+0000","last_unstale":"2021-02-10T04:55:56.678886+0000","last_undegraded":"2021-02-10T04:55:56.678886+0000","last_fullsized":"2021-02-10T04:55:56.678886+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.324564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.324564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"1155","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.703481+0000","last_change":"2021-02-10T04:55:28.098627+0000","last_active":"2021-02-10T04:55:56.703481+0000","last_peered":"2021-02-10T04:55:56.703481+0000","last_clean":"2021-02-10T04:55:56.703481+0000","last_became_active":"2021-02-10T04:55:14.640497+0000","last_became_peered":"2021-02-10T04:55:14.640497+0000","last_unstale":"2021-02-10T04:55:56.703481+0000","last_undegraded":"2021-02-10T04:55:56.703481+0000","last_fullsized":"2021-02-10T04:55:56.703481+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:00.501127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:00.501127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"259'74","reported_seq":"200","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.703602+0000","last_change":"2021-02-10T04:55:28.095458+0000","last_active":"2021-02-10T04:55:56.703602+0000","last_peered":"2021-02-10T04:55:56.703602+0000","last_clean":"2021-02-10T04:55:56.703602+0000","last_became_active":"2021-02-10T04:55:14.624071+0000","last_became_peered":"2021-02-10T04:55:14.624071+0000","last_unstale":"2021-02-10T04:55:56.703602+0000","last_undegraded":"2021-02-10T04:55:56.703602+0000","last_fullsized":"2021-02-10T04:55:56.703602+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":5,"last_scrub":"259'74","last_scrub_stamp":"2021-02-10T04:52:26.597668+0000","last_deep_scrub":"259'74","last_deep_scrub_stamp":"2021-02-10T04:46:49.745104+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.597668+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1218524,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":28228,"num_write":19,"num_write_kb":7022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"23","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.676240+0000","last_change":"2021-02-10T04:55:39.619500+0000","last_active":"2021-02-10T04:55:56.676240+0000","last_peered":"2021-02-10T04:55:56.676240+0000","last_clean":"2021-02-10T04:55:56.676240+0000","last_became_active":"2021-02-10T04:55:39.619062+0000","last_became_peered":"2021-02-10T04:55:39.619062+0000","last_unstale":"2021-02-10T04:55:56.676240+0000","last_undegraded":"2021-02-10T04:55:56.676240+0000","last_fullsized":"2021-02-10T04:55:56.676240+0000","mapping_epoch":1248,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:59.167911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:50:59.167911+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"171","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.763468+0000","last_change":"2021-02-10T04:53:46.718512+0000","last_active":"2021-02-10T04:55:56.763468+0000","last_peered":"2021-02-10T04:55:56.763468+0000","last_clean":"2021-02-10T04:55:56.763468+0000","last_became_active":"2021-02-10T04:53:46.716811+0000","last_became_peered":"2021-02-10T04:53:46.716811+0000","last_unstale":"2021-02-10T04:55:56.763468+0000","last_undegraded":"2021-02-10T04:55:56.763468+0000","last_fullsized":"2021-02-10T04:55:56.763468+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:45.813676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.813676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"1251","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.678219+0000","last_change":"2021-02-10T04:53:38.363899+0000","last_active":"2021-02-10T04:55:56.678219+0000","last_peered":"2021-02-10T04:55:56.678219+0000","last_clean":"2021-02-10T04:55:56.678219+0000","last_became_active":"2021-02-10T04:50:59.539256+0000","last_became_peered":"2021-02-10T04:50:59.539256+0000","last_unstale":"2021-02-10T04:55:56.678219+0000","last_undegraded":"2021-02-10T04:55:56.678219+0000","last_fullsized":"2021-02-10T04:55:56.678219+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:38.363825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:09.433793+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.363825+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1b","version":"1003'1689","reported_seq":"4607","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.704893+0000","last_change":"2021-02-10T04:55:15.591337+0000","last_active":"2021-02-10T04:55:56.704893+0000","last_peered":"2021-02-10T04:55:56.704893+0000","last_clean":"2021-02-10T04:55:56.704893+0000","last_became_active":"2021-02-10T04:55:15.590301+0000","last_became_peered":"2021-02-10T04:55:15.590301+0000","last_unstale":"2021-02-10T04:55:56.704893+0000","last_undegraded":"2021-02-10T04:55:56.704893+0000","last_fullsized":"2021-02-10T04:55:56.704893+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:53:10.075879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.075879+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":63260738,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":387,"num_read_kb":409791,"num_write":281,"num_write_kb":110559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":22449943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"1246","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.706011+0000","last_change":"2021-02-10T04:55:14.637968+0000","last_active":"2021-02-10T04:55:56.706011+0000","last_peered":"2021-02-10T04:55:56.706011+0000","last_clean":"2021-02-10T04:55:56.706011+0000","last_became_active":"2021-02-10T04:55:14.636561+0000","last_became_peered":"2021-02-10T04:55:14.636561+0000","last_unstale":"2021-02-10T04:55:56.706011+0000","last_undegraded":"2021-02-10T04:55:56.706011+0000","last_fullsized":"2021-02-10T04:55:56.706011+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:33.543559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:46:33.543559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"921","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.677992+0000","last_change":"2021-02-10T04:53:04.354508+0000","last_active":"2021-02-10T04:55:56.677992+0000","last_peered":"2021-02-10T04:55:56.677992+0000","last_clean":"2021-02-10T04:55:56.677992+0000","last_became_active":"2021-02-10T04:33:05.169159+0000","last_became_peered":"2021-02-10T04:33:05.169159+0000","last_unstale":"2021-02-10T04:55:56.677992+0000","last_undegraded":"2021-02-10T04:55:56.677992+0000","last_fullsized":"2021-02-10T04:55:56.677992+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:04.354409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:04.354409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"1315","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.678031+0000","last_change":"2021-02-10T04:52:36.777554+0000","last_active":"2021-02-10T04:55:56.678031+0000","last_peered":"2021-02-10T04:55:56.678031+0000","last_clean":"2021-02-10T04:55:56.678031+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:55:56.678031+0000","last_undegraded":"2021-02-10T04:55:56.678031+0000","last_fullsized":"2021-02-10T04:55:56.678031+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:36.777485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:52:36.777485+0000","last_clean_scrub_stamp":"2021-02-10T04:52:36.777485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1a","version":"1001'1528","reported_seq":"5872","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.677701+0000","last_change":"2021-02-10T04:52:31.086406+0000","last_active":"2021-02-10T04:55:56.677701+0000","last_peered":"2021-02-10T04:55:56.677701+0000","last_clean":"2021-02-10T04:55:56.677701+0000","last_became_active":"2021-02-10T04:52:31.085959+0000","last_became_peered":"2021-02-10T04:52:31.085959+0000","last_unstale":"2021-02-10T04:55:56.677701+0000","last_undegraded":"2021-02-10T04:55:56.677701+0000","last_fullsized":"2021-02-10T04:55:56.677701+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46944136,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":358,"num_read_kb":351935,"num_write":286,"num_write_kb":111355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":21311893,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"1145","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.704629+0000","last_change":"2021-02-10T04:55:22.726939+0000","last_active":"2021-02-10T04:55:56.704629+0000","last_peered":"2021-02-10T04:55:56.704629+0000","last_clean":"2021-02-10T04:55:56.704629+0000","last_became_active":"2021-02-10T04:55:14.627044+0000","last_became_peered":"2021-02-10T04:55:14.627044+0000","last_unstale":"2021-02-10T04:55:56.704629+0000","last_undegraded":"2021-02-10T04:55:56.704629+0000","last_fullsized":"2021-02-10T04:55:56.704629+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:17.919745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:17.919745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"547","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.703767+0000","last_change":"2021-02-10T04:53:07.092032+0000","last_active":"2021-02-10T04:55:56.703767+0000","last_peered":"2021-02-10T04:55:56.703767+0000","last_clean":"2021-02-10T04:55:56.703767+0000","last_became_active":"2021-02-10T04:50:59.572077+0000","last_became_peered":"2021-02-10T04:50:59.572077+0000","last_unstale":"2021-02-10T04:55:56.703767+0000","last_undegraded":"2021-02-10T04:55:56.703767+0000","last_fullsized":"2021-02-10T04:55:56.703767+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.091949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:19.755718+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.091949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"1224","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.705285+0000","last_change":"2021-02-10T04:51:59.838856+0000","last_active":"2021-02-10T04:55:56.705285+0000","last_peered":"2021-02-10T04:55:56.705285+0000","last_clean":"2021-02-10T04:55:56.705285+0000","last_became_active":"2021-02-10T04:50:59.563156+0000","last_became_peered":"2021-02-10T04:50:59.563156+0000","last_unstale":"2021-02-10T04:55:56.705285+0000","last_undegraded":"2021-02-10T04:55:56.705285+0000","last_fullsized":"2021-02-10T04:55:56.705285+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.838790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.838790+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":"957","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.679378+0000","last_change":"2021-02-10T04:52:30.405170+0000","last_active":"2021-02-10T04:55:56.679378+0000","last_peered":"2021-02-10T04:55:56.679378+0000","last_clean":"2021-02-10T04:55:56.679378+0000","last_became_active":"2021-02-10T04:51:36.852058+0000","last_became_peered":"2021-02-10T04:51:36.852058+0000","last_unstale":"2021-02-10T04:55:56.679378+0000","last_undegraded":"2021-02-10T04:55:56.679378+0000","last_fullsized":"2021-02-10T04:55:56.679378+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.405106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:20.635536+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.405106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.20","version":"0'0","reported_seq":"195","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.763174+0000","last_change":"2021-02-10T04:53:46.739888+0000","last_active":"2021-02-10T04:55:56.763174+0000","last_peered":"2021-02-10T04:55:56.763174+0000","last_clean":"2021-02-10T04:55:56.763174+0000","last_became_active":"2021-02-10T04:53:46.739532+0000","last_became_peered":"2021-02-10T04:53:46.739532+0000","last_unstale":"2021-02-10T04:55:56.763174+0000","last_undegraded":"2021-02-10T04:55:56.763174+0000","last_fullsized":"2021-02-10T04:55:56.763174+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:56.449085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:41.375276+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.449085+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":"1190","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:55:59.688241+0000","last_change":"2021-02-10T04:53:35.858634+0000","last_active":"2021-02-10T04:55:59.688241+0000","last_peered":"2021-02-10T04:55:59.688241+0000","last_clean":"2021-02-10T04:55:59.688241+0000","last_became_active":"2021-02-10T04:51:36.869785+0000","last_became_peered":"2021-02-10T04:51:36.869785+0000","last_unstale":"2021-02-10T04:55:59.688241+0000","last_undegraded":"2021-02-10T04:55:59.688241+0000","last_fullsized":"2021-02-10T04:55:59.688241+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.858557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.858557+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":"173","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.676617+0000","last_change":"2021-02-10T04:55:23.586523+0000","last_active":"2021-02-10T04:55:56.676617+0000","last_peered":"2021-02-10T04:55:56.676617+0000","last_clean":"2021-02-10T04:55:56.676617+0000","last_became_active":"2021-02-10T04:55:14.637029+0000","last_became_peered":"2021-02-10T04:55:14.637029+0000","last_unstale":"2021-02-10T04:55:56.676617+0000","last_undegraded":"2021-02-10T04:55:56.676617+0000","last_fullsized":"2021-02-10T04:55:56.676617+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:55:23.586401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:55:23.586401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":"158","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.679008+0000","last_change":"2021-02-10T04:55:14.635842+0000","last_active":"2021-02-10T04:55:56.679008+0000","last_peered":"2021-02-10T04:55:56.679008+0000","last_clean":"2021-02-10T04:55:56.679008+0000","last_became_active":"2021-02-10T04:55:14.635499+0000","last_became_peered":"2021-02-10T04:55:14.635499+0000","last_unstale":"2021-02-10T04:55:56.679008+0000","last_undegraded":"2021-02-10T04:55:56.679008+0000","last_fullsized":"2021-02-10T04:55:56.679008+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.706372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.706372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":"1272","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:55:59.687954+0000","last_change":"2021-02-10T04:51:57.287452+0000","last_active":"2021-02-10T04:55:59.687954+0000","last_peered":"2021-02-10T04:55:59.687954+0000","last_clean":"2021-02-10T04:55:59.687954+0000","last_became_active":"2021-02-10T04:50:59.566827+0000","last_became_peered":"2021-02-10T04:50:59.566827+0000","last_unstale":"2021-02-10T04:55:59.687954+0000","last_undegraded":"2021-02-10T04:55:59.687954+0000","last_fullsized":"2021-02-10T04:55:59.687954+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.287363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.287363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":"246","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:55:59.689004+0000","last_change":"2021-02-10T04:55:14.659882+0000","last_active":"2021-02-10T04:55:59.689004+0000","last_peered":"2021-02-10T04:55:59.689004+0000","last_clean":"2021-02-10T04:55:59.689004+0000","last_became_active":"2021-02-10T04:55:14.659514+0000","last_became_peered":"2021-02-10T04:55:14.659514+0000","last_unstale":"2021-02-10T04:55:59.689004+0000","last_undegraded":"2021-02-10T04:55:59.689004+0000","last_fullsized":"2021-02-10T04:55:59.689004+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:10.308282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:10.308282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":"1160","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.705652+0000","last_change":"2021-02-10T04:53:09.121379+0000","last_active":"2021-02-10T04:55:56.705652+0000","last_peered":"2021-02-10T04:55:56.705652+0000","last_clean":"2021-02-10T04:55:56.705652+0000","last_became_active":"2021-02-10T04:50:59.561442+0000","last_became_peered":"2021-02-10T04:50:59.561442+0000","last_unstale":"2021-02-10T04:55:56.705652+0000","last_undegraded":"2021-02-10T04:55:56.705652+0000","last_fullsized":"2021-02-10T04:55:56.705652+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:09.121298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.121298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.21","version":"293'77","reported_seq":"832","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.704033+0000","last_change":"2021-02-10T04:52:28.062923+0000","last_active":"2021-02-10T04:55:56.704033+0000","last_peered":"2021-02-10T04:55:56.704033+0000","last_clean":"2021-02-10T04:55:56.704033+0000","last_became_active":"2021-02-10T04:52:28.062463+0000","last_became_peered":"2021-02-10T04:52:28.062463+0000","last_unstale":"2021-02-10T04:55:56.704033+0000","last_undegraded":"2021-02-10T04:55:56.704033+0000","last_fullsized":"2021-02-10T04:55:56.704033+0000","mapping_epoch":1143,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1144,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1886586,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":15163,"num_write":9,"num_write_kb":3316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":"158","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.675442+0000","last_change":"2021-02-10T04:52:32.124955+0000","last_active":"2021-02-10T04:55:56.675442+0000","last_peered":"2021-02-10T04:55:56.675442+0000","last_clean":"2021-02-10T04:55:56.675442+0000","last_became_active":"2021-02-10T04:51:36.867061+0000","last_became_peered":"2021-02-10T04:51:36.867061+0000","last_unstale":"2021-02-10T04:55:56.675442+0000","last_undegraded":"2021-02-10T04:55:56.675442+0000","last_fullsized":"2021-02-10T04:55:56.675442+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:32.124901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:08.271977+0000","last_clean_scrub_stamp":"2021-02-10T04:52:32.124901+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":"1201","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:55:59.687945+0000","last_change":"2021-02-10T04:51:07.182775+0000","last_active":"2021-02-10T04:55:59.687945+0000","last_peered":"2021-02-10T04:55:59.687945+0000","last_clean":"2021-02-10T04:55:59.687945+0000","last_became_active":"2021-02-10T04:28:23.126368+0000","last_became_peered":"2021-02-10T04:28:23.126368+0000","last_unstale":"2021-02-10T04:55:59.687945+0000","last_undegraded":"2021-02-10T04:55:59.687945+0000","last_fullsized":"2021-02-10T04:55:59.687945+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.182678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.182678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.20","version":"1000'1071","reported_seq":"4606","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.675046+0000","last_change":"2021-02-10T04:53:03.915465+0000","last_active":"2021-02-10T04:55:56.675046+0000","last_peered":"2021-02-10T04:55:56.675046+0000","last_clean":"2021-02-10T04:55:56.675046+0000","last_became_active":"2021-02-10T04:52:31.088690+0000","last_became_peered":"2021-02-10T04:52:31.088690+0000","last_unstale":"2021-02-10T04:55:56.675046+0000","last_undegraded":"2021-02-10T04:55:56.675046+0000","last_fullsized":"2021-02-10T04:55:56.675046+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:03.915403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.915403+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16398893,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":"784","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.763352+0000","last_change":"2021-02-10T04:53:46.721599+0000","last_active":"2021-02-10T04:55:56.763352+0000","last_peered":"2021-02-10T04:55:56.763352+0000","last_clean":"2021-02-10T04:55:56.763352+0000","last_became_active":"2021-02-10T04:53:46.721340+0000","last_became_peered":"2021-02-10T04:53:46.721340+0000","last_unstale":"2021-02-10T04:55:56.763352+0000","last_undegraded":"2021-02-10T04:55:56.763352+0000","last_fullsized":"2021-02-10T04:55:56.763352+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:28.092288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:07.767250+0000","last_clean_scrub_stamp":"2021-02-10T04:52:28.092288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.25","version":"53'2","reported_seq":"1216","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.679085+0000","last_change":"2021-02-10T04:53:36.103026+0000","last_active":"2021-02-10T04:55:56.679085+0000","last_peered":"2021-02-10T04:55:56.679085+0000","last_clean":"2021-02-10T04:55:56.679085+0000","last_became_active":"2021-02-10T04:28:23.467770+0000","last_became_peered":"2021-02-10T04:28:23.467770+0000","last_unstale":"2021-02-10T04:55:56.679085+0000","last_undegraded":"2021-02-10T04:55:56.679085+0000","last_fullsized":"2021-02-10T04:55:56.679085+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:53:36.102928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:36.102928+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.23","version":"1002'1657","reported_seq":"5133","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:55:59.688365+0000","last_change":"2021-02-10T04:52:31.093874+0000","last_active":"2021-02-10T04:55:59.688365+0000","last_peered":"2021-02-10T04:55:59.688365+0000","last_clean":"2021-02-10T04:55:59.688365+0000","last_became_active":"2021-02-10T04:52:31.092996+0000","last_became_peered":"2021-02-10T04:52:31.092996+0000","last_unstale":"2021-02-10T04:55:59.688365+0000","last_undegraded":"2021-02-10T04:55:59.688365+0000","last_fullsized":"2021-02-10T04:55:59.688365+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21396533,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":212,"num_read_kb":218837,"num_write":159,"num_write_kb":59399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":20479176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":"1308","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:55:59.687814+0000","last_change":"2021-02-10T04:53:12.951612+0000","last_active":"2021-02-10T04:55:59.687814+0000","last_peered":"2021-02-10T04:55:59.687814+0000","last_clean":"2021-02-10T04:55:59.687814+0000","last_became_active":"2021-02-10T04:50:59.566548+0000","last_became_peered":"2021-02-10T04:50:59.566548+0000","last_unstale":"2021-02-10T04:55:59.687814+0000","last_undegraded":"2021-02-10T04:55:59.687814+0000","last_fullsized":"2021-02-10T04:55:59.687814+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.951528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.951528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.26","version":"53'1","reported_seq":"98","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.764788+0000","last_change":"2021-02-10T04:53:46.722636+0000","last_active":"2021-02-10T04:55:56.764788+0000","last_peered":"2021-02-10T04:55:56.764788+0000","last_clean":"2021-02-10T04:55:56.764788+0000","last_became_active":"2021-02-10T04:53:46.722227+0000","last_became_peered":"2021-02-10T04:53:46.722227+0000","last_unstale":"2021-02-10T04:55:56.764788+0000","last_undegraded":"2021-02-10T04:55:56.764788+0000","last_fullsized":"2021-02-10T04:55:56.764788+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:53:03.933910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.933910+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":423,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.22","version":"998'1009","reported_seq":"4623","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.677805+0000","last_change":"2021-02-10T04:52:31.086563+0000","last_active":"2021-02-10T04:55:56.677805+0000","last_peered":"2021-02-10T04:55:56.677805+0000","last_clean":"2021-02-10T04:55:56.677805+0000","last_became_active":"2021-02-10T04:52:31.085288+0000","last_became_peered":"2021-02-10T04:52:31.085288+0000","last_unstale":"2021-02-10T04:55:56.677805+0000","last_undegraded":"2021-02-10T04:55:56.677805+0000","last_fullsized":"2021-02-10T04:55:56.677805+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12355207,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":120,"num_read_kb":124702,"num_write":77,"num_write_kb":25052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4145029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":"1190","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:55:59.688322+0000","last_change":"2021-02-10T04:53:10.958668+0000","last_active":"2021-02-10T04:55:59.688322+0000","last_peered":"2021-02-10T04:55:59.688322+0000","last_clean":"2021-02-10T04:55:59.688322+0000","last_became_active":"2021-02-10T04:45:49.552026+0000","last_became_peered":"2021-02-10T04:45:49.552026+0000","last_unstale":"2021-02-10T04:55:59.688322+0000","last_undegraded":"2021-02-10T04:55:59.688322+0000","last_fullsized":"2021-02-10T04:55:59.688322+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:10.958579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.958579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":"203","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.765471+0000","last_change":"2021-02-10T04:53:46.725941+0000","last_active":"2021-02-10T04:55:56.765471+0000","last_peered":"2021-02-10T04:55:56.765471+0000","last_clean":"2021-02-10T04:55:56.765471+0000","last_became_active":"2021-02-10T04:53:46.725587+0000","last_became_peered":"2021-02-10T04:53:46.725587+0000","last_unstale":"2021-02-10T04:55:56.765471+0000","last_undegraded":"2021-02-10T04:55:56.765471+0000","last_fullsized":"2021-02-10T04:55:56.765471+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:41.758684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.758684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":"1198","reported_epoch":"1261","state":"active+clean","last_fresh":"2021-02-10T04:55:56.703890+0000","last_change":"2021-02-10T04:55:39.599193+0000","last_active":"2021-02-10T04:55:56.703890+0000","last_peered":"2021-02-10T04:55:56.703890+0000","last_clean":"2021-02-10T04:55:56.703890+0000","last_became_active":"2021-02-10T04:55:39.598816+0000","last_became_peered":"2021-02-10T04:55:39.598816+0000","last_unstale":"2021-02-10T04:55:56.703890+0000","last_undegraded":"2021-02-10T04:55:56.703890+0000","last_fullsized":"2021-02-10T04:55:56.703890+0000","mapping_epoch":1248,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:11.139602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.139602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":"1216","reported_epoch":"1262","state":"active+clean","last_fresh":"2021-02-10T04:55:59.687267+0000","last_change":"2021-02-10T04:52:24.072005+0000","last_active":"2021-02-10T04:55:59.687267+0000","last_peered":"2021-02-10T04:55:59.687267+0000","last_clean":"2021-02-10T04:55:59.687267+0000","last_became_active":"2021-02-10T04:50:59.527945+0000","last_became_peered":"2021-02-10T04:50:59.527945+0000","last_unstale":"2021-02-10T04:55:59.687267+0000","last_undegraded":"2021-02-10T04:55:59.687267+0000","last_fullsized":"2021-02-10T04:55:59.687267+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:24.071915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_clean_scrub_stamp":"2021-02-10T04:52:24.071915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":972330808,"num_objects":390,"num_object_clones":337,"num_object_copies":1170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":390,"num_whiteouts":10,"num_read":4442,"num_read_kb":4559373,"num_write":3227,"num_write_kb":1247575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":611,"num_bytes_recovered":311961170,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2010624000,"data_stored":2589160348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34722,"ondisk_log_size":34722,"up":108,"acting":108,"num_store_stats":8},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":2368,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":18,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1139,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":5835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":126,"acting":126,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":13,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":682,"num_read_kb":677,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20,"ondisk_log_size":20,"up":39,"acting":39,"num_store_stats":8},{"poolid":4,"num_pg":22,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":2717,"num_read_kb":2190,"num_write":1592,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":17736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":844,"ondisk_log_size":844,"up":66,"acting":66,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35,"ondisk_log_size":35,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":126,"acting":126,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365167,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":191244,"kb_used_data":149188,"kb_used_omap":0,"kb_used_meta":42048,"kb_avail":93553908,"statfs":{"total":95995035648,"available":95799201792,"internally_reserved":0,"allocated":152768512,"data_stored":202373916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43057152},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:27 2021","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":1.2370000000000001,"15min":1.284},"min":{"1min":0.50800000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1220000000000001,"5min":34.299999999999997,"15min":67.277000000000001},"last":1.022},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.5109999999999999,"15min":1.548},"min":{"1min":0.46300000000000002,"5min":0.438,"15min":0.438},"max":{"1min":2.8690000000000002,"5min":34.652999999999999,"15min":66.933999999999997},"last":1.3089999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:55:42 2021","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.80700000000000005,"15min":0.95899999999999996},"min":{"1min":0.48799999999999999,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.409,"5min":2.254,"15min":10.140000000000001},"last":1.274},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.85999999999999999,"15min":0.96699999999999997},"min":{"1min":0.63400000000000001,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.647,"5min":1.647,"15min":7.1520000000000001},"last":1.5680000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:55:42 2021","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":1.256,"15min":1.1259999999999999},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.4319999999999999,"5min":16.896999999999998,"15min":16.896999999999998},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.2070000000000001,"15min":1.254},"min":{"1min":0.57999999999999996,"5min":0.496,"15min":0.496},"max":{"1min":4.0620000000000003,"5min":16.792000000000002,"15min":16.792000000000002},"last":1.7370000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:54:59 2021","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":1.1359999999999999,"15min":1.29},"min":{"1min":0.68600000000000005,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.3799999999999999,"5min":4.5890000000000004,"15min":4.5890000000000004},"last":1.7010000000000001},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.2250000000000001,"15min":1.3979999999999999},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.5449999999999999,"5min":4.6609999999999996,"15min":4.6609999999999996},"last":1.6299999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:55:42 2021","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":1.1639999999999999,"15min":1.3899999999999999},"min":{"1min":0.60399999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.399,"5min":2.9729999999999999,"15min":10.781000000000001},"last":1.9510000000000001},{"interface":"front","average":{"1min":1.03,"5min":1.1759999999999999,"15min":1.3460000000000001},"min":{"1min":0.74399999999999999,"5min":0.69699999999999995,"15min":0.69099999999999995},"max":{"1min":1.546,"5min":3.4049999999999998,"15min":10.326000000000001},"last":1.8280000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:55:42 2021","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.349,"15min":1.484},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":4.766,"5min":4.766,"15min":14.972},"last":2.3319999999999999},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.4450000000000001,"15min":1.6899999999999999},"min":{"1min":0.81000000000000005,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.1549999999999998,"5min":4.4020000000000001,"15min":15.218999999999999},"last":2.625}]},{"osd":6,"last update":"Wed Feb 10 04:55:42 2021","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.375,"15min":1.7490000000000001},"min":{"1min":0.92700000000000005,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.5780000000000001,"5min":4.0380000000000003,"15min":16.050000000000001},"last":2.1909999999999998},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.4039999999999999,"15min":1.71},"min":{"1min":0.90300000000000002,"5min":0.876,"15min":0.876},"max":{"1min":1.964,"5min":3.9159999999999999,"15min":15.960000000000001},"last":1.929}]}]},{"osd":6,"up_from":43,"seq":184683594100,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":408300,"kb_used_data":342308,"kb_used_omap":0,"kb_used_meta":65984,"kb_avail":93336852,"statfs":{"total":95995035648,"available":95576936448,"internally_reserved":0,"allocated":350523392,"data_stored":443352658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":67567616},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:52:12 2021","interfaces":[{"interface":"back","average":{"1min":4.016,"5min":2.7400000000000002,"15min":1.595},"min":{"1min":0.52000000000000002,"5min":0.48099999999999998,"15min":0.44700000000000001},"max":{"1min":61.768000000000001,"5min":137.60599999999999,"15min":137.60599999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":4.0270000000000001,"5min":2.694,"15min":1.7989999999999999},"min":{"1min":0.504,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":62.683,"5min":137.66,"15min":137.66},"last":0.68500000000000005}]},{"osd":1,"last update":"Wed Feb 10 04:55:20 2021","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":1.0089999999999999,"15min":1.0680000000000001},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.395,"5min":13.18,"15min":13.18},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.77900000000000003,"5min":1.0309999999999999,"15min":1.1020000000000001},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.3440000000000001,"5min":11.041,"15min":11.041},"last":0.90500000000000003}]},{"osd":2,"last update":"Wed Feb 10 04:55:20 2021","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":2.641,"15min":1.712},"min":{"1min":0.59799999999999998,"5min":0.503,"15min":0.503},"max":{"1min":1.865,"5min":163.81700000000001,"15min":163.81700000000001},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.94299999999999995,"5min":2.6720000000000002,"15min":1.742},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.6619999999999999,"5min":164.56800000000001,"15min":164.56800000000001},"last":1.036}]},{"osd":3,"last update":"Wed Feb 10 04:55:07 2021","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":1.117,"15min":1.1850000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.764,"5min":1.764,"15min":1.764},"last":1.165},{"interface":"front","average":{"1min":1.014,"5min":1.101,"15min":1.117},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.9430000000000001,"5min":2.363,"15min":2.363},"last":1.139}]},{"osd":4,"last update":"Wed Feb 10 04:55:20 2021","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.1850000000000001,"15min":1.369},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.1120000000000001,"5min":7.1459999999999999,"15min":7.1459999999999999},"last":1.246},{"interface":"front","average":{"1min":0.874,"5min":1.0549999999999999,"15min":1.105},"min":{"1min":0.64000000000000001,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.4770000000000001,"5min":11.997999999999999,"15min":11.997999999999999},"last":0.95399999999999996}]},{"osd":5,"last update":"Wed Feb 10 04:55:20 2021","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.196,"15min":1.595},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.5449999999999999,"5min":12.045,"15min":81.540000000000006},"last":1.073},{"interface":"front","average":{"1min":1.246,"5min":1.47,"15min":1.8720000000000001},"min":{"1min":0.875,"5min":0.78700000000000003,"15min":0.78000000000000003},"max":{"1min":2.0910000000000002,"5min":13.401,"15min":81.492999999999995},"last":1.343}]},{"osd":7,"last update":"Wed Feb 10 04:55:47 2021","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.161,"15min":1.288},"min":{"1min":0.72899999999999998,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":2.073,"5min":4.0919999999999996,"15min":5.1820000000000004},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.421,"5min":1.4159999999999999,"15min":1.595},"min":{"1min":1.016,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":2.2970000000000002,"5min":5.4109999999999996,"15min":13.315},"last":1.278}]}]},{"osd":1,"up_from":13,"seq":55834575237,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":257108,"kb_used_data":206284,"kb_used_omap":0,"kb_used_meta":50816,"kb_avail":93488044,"statfs":{"total":95995035648,"available":95731757056,"internally_reserved":0,"allocated":211234816,"data_stored":265840899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52035584},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:48 2021","interfaces":[{"interface":"back","average":{"1min":1.696,"5min":2.6320000000000001,"15min":1.921},"min":{"1min":0.44700000000000001,"5min":0.437,"15min":0.437},"max":{"1min":14.053000000000001,"5min":115.622,"15min":115.622},"last":0.68700000000000006},{"interface":"front","average":{"1min":1.728,"5min":2.585,"15min":1.948},"min":{"1min":0.49199999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":14.109,"5min":115.73099999999999,"15min":115.73099999999999},"last":0.92700000000000005}]},{"osd":2,"last update":"Wed Feb 10 04:55:13 2021","interfaces":[{"interface":"back","average":{"1min":11.944000000000001,"5min":5.5419999999999998,"15min":2.766},"min":{"1min":0.50700000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":162.61199999999999,"5min":162.61199999999999,"15min":162.61199999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":12.112,"5min":5.601,"15min":2.8210000000000002},"min":{"1min":0.498,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":162.762,"5min":162.762,"15min":162.762},"last":0.64700000000000002}]},{"osd":3,"last update":"Wed Feb 10 04:55:03 2021","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.79600000000000004,"15min":0.91100000000000003},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.6919999999999999,"5min":3.077,"15min":3.077},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.91200000000000003,"15min":1.0049999999999999},"min":{"1min":0.54600000000000004,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.052,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.89100000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:55:56 2021","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.0309999999999999,"15min":1.135},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":6.1479999999999997,"5min":6.1479999999999997,"15min":6.1479999999999997},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.071,"15min":1.2749999999999999},"min":{"1min":0.68400000000000005,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.6879999999999999,"5min":2.5270000000000001,"15min":29.547999999999998},"last":1.385}]},{"osd":5,"last update":"Wed Feb 10 04:54:59 2021","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":1.05,"15min":1.1579999999999999},"min":{"1min":0.66800000000000004,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.7949999999999999,"5min":2.7959999999999998,"15min":5.0300000000000002},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.21,"5min":1.1879999999999999,"15min":1.488},"min":{"1min":0.70399999999999996,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.879,"5min":2.8199999999999998,"15min":28.678000000000001},"last":1.046}]},{"osd":6,"last update":"Wed Feb 10 04:55:13 2021","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.3660000000000001,"15min":1.538},"min":{"1min":0.68999999999999995,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.4099999999999999,"5min":30.997,"15min":30.997},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.071,"5min":1.3839999999999999,"15min":1.4590000000000001},"min":{"1min":0.80600000000000005,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.4790000000000001,"5min":31.138000000000002,"15min":31.138000000000002},"last":1.153}]},{"osd":7,"last update":"Wed Feb 10 04:55:27 2021","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.216,"15min":1.488},"min":{"1min":0.72299999999999998,"5min":0.628,"15min":0.628},"max":{"1min":6.2629999999999999,"5min":6.2629999999999999,"15min":28.573},"last":1.367},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.232,"15min":1.5169999999999999},"min":{"1min":0.89200000000000002,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.895,"5min":1.895,"15min":29.408999999999999},"last":1.492}]}]},{"osd":0,"up_from":1227,"seq":5269924872203,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":348736,"kb_used_data":338488,"kb_used_omap":306,"kb_used_meta":9933,"kb_avail":93396416,"statfs":{"total":95995035648,"available":95637929984,"internally_reserved":0,"allocated":346611712,"data_stored":471469907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":313579,"internal_metadata":10172181},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.395}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79200000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96499999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.54}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1060000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.333}]}]},{"osd":2,"up_from":18,"seq":77309411715,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":354060,"kb_used_data":318468,"kb_used_omap":0,"kb_used_meta":35584,"kb_avail":93391092,"statfs":{"total":95995035648,"available":95632478208,"internally_reserved":0,"allocated":326111232,"data_stored":374869055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36438016},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:50 2021","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.99099999999999999,"15min":1.718},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":3.698,"5min":3.996,"15min":163.15199999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.91300000000000003,"15min":1.655},"min":{"1min":0.51100000000000001,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.9710000000000001,"5min":2.2349999999999999,"15min":162.959},"last":0.71999999999999997}]},{"osd":1,"last update":"Wed Feb 10 04:55:55 2021","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":0.88200000000000001,"15min":1},"min":{"1min":0.48099999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":17.041,"5min":17.041,"15min":17.041},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.8999999999999999,"5min":0.91300000000000003,"15min":0.94899999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":21.172000000000001,"5min":21.172000000000001,"15min":21.172000000000001},"last":0.92100000000000004}]},{"osd":3,"last update":"Wed Feb 10 04:55:08 2021","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.92200000000000004,"15min":1.0840000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.83799999999999997,"5min":4.2380000000000004,"15min":4.2380000000000004},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.93100000000000005,"5min":1.133,"15min":1.2649999999999999},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.1259999999999999,"5min":4.54,"15min":4.54},"last":1.101}]},{"osd":4,"last update":"Wed Feb 10 04:55:36 2021","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.1379999999999999,"15min":1.2849999999999999},"min":{"1min":0.71499999999999997,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":9.2029999999999994,"5min":9.2029999999999994,"15min":9.2029999999999994},"last":1.391},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.228,"15min":1.1699999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":3.403,"5min":24.373999999999999,"15min":24.373999999999999},"last":0.85399999999999998}]},{"osd":5,"last update":"Wed Feb 10 04:55:55 2021","interfaces":[{"interface":"back","average":{"1min":3.0630000000000002,"5min":1.575,"15min":1.7709999999999999},"min":{"1min":0.85399999999999998,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":39.911999999999999,"5min":39.911999999999999,"15min":39.911999999999999},"last":1.48},{"interface":"front","average":{"1min":2.508,"5min":1.284,"15min":1.5109999999999999},"min":{"1min":0.67700000000000005,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":33.539999999999999,"5min":33.539999999999999,"15min":33.539999999999999},"last":1.0249999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:55:23 2021","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.0629999999999999,"15min":1.419},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":3.1659999999999999,"5min":3.1659999999999999,"15min":20.007000000000001},"last":1.1399999999999999},{"interface":"front","average":{"1min":1.9530000000000001,"5min":1.423,"15min":1.6519999999999999},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":16.135000000000002,"5min":16.135000000000002,"15min":19.417999999999999},"last":1.3029999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:55:36 2021","interfaces":[{"interface":"back","average":{"1min":1.637,"5min":1.637,"15min":1.681},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":9.3460000000000001,"5min":24.405000000000001,"15min":24.405000000000001},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.1040000000000001,"15min":1.278},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":3.6640000000000001,"5min":3.6640000000000001,"15min":4.9189999999999996},"last":1.121}]}]},{"osd":3,"up_from":1111,"seq":4771708665918,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":332652,"kb_used_data":301220,"kb_used_omap":386,"kb_used_meta":31037,"kb_avail":93412500,"statfs":{"total":95995035648,"available":95654400000,"internally_reserved":0,"allocated":308449280,"data_stored":409790702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":396036,"internal_metadata":31782140},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:58 2021","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":0.71599999999999997}]},{"osd":1,"last update":"Wed Feb 10 04:55:04 2021","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.63500000000000001,"15min":0.64400000000000002},"min":{"1min":0.46999999999999997,"5min":0.44,"15min":0.44},"max":{"1min":0.69399999999999995,"5min":1.8360000000000001,"15min":1.8360000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.79100000000000004,"15min":0.85899999999999999},"min":{"1min":0.46600000000000003,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.534,"5min":2.137,"15min":2.137},"last":1.093}]},{"osd":2,"last update":"Wed Feb 10 04:55:04 2021","interfaces":[{"interface":"back","average":{"1min":2.512,"5min":1.48,"15min":1.26},"min":{"1min":0.48699999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":31.100999999999999,"5min":31.100999999999999,"15min":31.100999999999999},"last":0.84199999999999997},{"interface":"front","average":{"1min":2.637,"5min":1.5700000000000001,"15min":1.27},"min":{"1min":0.53800000000000003,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":31.052,"5min":31.052,"15min":31.052},"last":0.97899999999999998}]},{"osd":4,"last update":"Wed Feb 10 04:55:04 2021","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.93799999999999994,"15min":0.95999999999999996},"min":{"1min":0.66900000000000004,"5min":0.628,"15min":0.628},"max":{"1min":1.502,"5min":2.9430000000000001,"15min":2.9430000000000001},"last":1.0529999999999999},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.997,"15min":1.085},"min":{"1min":0.65300000000000002,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.8580000000000001,"5min":1.8999999999999999,"15min":1.8999999999999999},"last":1.163}]},{"osd":5,"last update":"Wed Feb 10 04:55:04 2021","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":2.2839999999999998,"15min":2.593},"min":{"1min":0.76000000000000001,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.462,"5min":52.353000000000002,"15min":52.353000000000002},"last":1.2849999999999999},{"interface":"front","average":{"1min":0.97099999999999997,"5min":2.1789999999999998,"15min":2.4209999999999998},"min":{"1min":0.70399999999999996,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":2.1000000000000001,"5min":52.301000000000002,"15min":52.301000000000002},"last":1.028}]},{"osd":6,"last update":"Wed Feb 10 04:55:04 2021","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":3.4790000000000001,"15min":4.9770000000000003},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.581,"5min":64.376999999999995,"15min":64.376999999999995},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.111,"5min":3.5499999999999998,"15min":4.9729999999999999},"min":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":1.7130000000000001,"5min":64.438999999999993,"15min":64.438999999999993},"last":1.3260000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:55:04 2021","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.0669999999999999,"15min":1.1060000000000001},"min":{"1min":0.77500000000000002,"5min":0.754,"15min":0.754},"max":{"1min":1.3879999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.115,"15min":1.1339999999999999},"min":{"1min":0.86399999999999999,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.6140000000000001,"5min":2.044,"15min":2.044},"last":1.2090000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986555,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":160460,"kb_used_data":120452,"kb_used_omap":0,"kb_used_meta":40000,"kb_avail":93584692,"statfs":{"total":95995035648,"available":95830724608,"internally_reserved":0,"allocated":123342848,"data_stored":155692062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40960000},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:36 2021","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.0899999999999999,"15min":1.593},"min":{"1min":0.45800000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.7799999999999998,"5min":95.721999999999994,"15min":95.721999999999994},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":2.0870000000000002,"15min":1.6000000000000001},"min":{"1min":0.46000000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.002,"5min":95.344999999999999,"15min":95.344999999999999},"last":0.78400000000000003}]},{"osd":1,"last update":"Wed Feb 10 04:55:46 2021","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.76100000000000001,"15min":0.871},"min":{"1min":0.46500000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.341,"5min":3.5529999999999999,"15min":7.883},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.89700000000000002,"15min":1.0580000000000001},"min":{"1min":0.50700000000000001,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.1850000000000001,"5min":3.8090000000000002,"15min":6.5060000000000002},"last":1.0209999999999999}]},{"osd":2,"last update":"Wed Feb 10 04:55:46 2021","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":2.6829999999999998,"15min":1.702},"min":{"1min":0.627,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":7.0069999999999997,"5min":120.39,"15min":120.39},"last":1.1359999999999999},{"interface":"front","average":{"1min":1.218,"5min":2.6419999999999999,"15min":1.6990000000000001},"min":{"1min":0.58999999999999997,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":7.1120000000000001,"5min":120.583,"15min":120.583},"last":1.1850000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:55:02 2021","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":1.139,"15min":1.2509999999999999},"min":{"1min":0.69899999999999995,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.542,"5min":2.1429999999999998,"15min":2.1429999999999998},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.659,"15min":1.573},"min":{"1min":0.81599999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.647,"5min":26.802,"15min":26.802},"last":1.2589999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:55:59 2021","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.1080000000000001,"15min":1.478},"min":{"1min":0.624,"5min":0.495,"15min":0.495},"max":{"1min":1.6579999999999999,"5min":2.351,"15min":35.774000000000001},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.325,"5min":1.2090000000000001,"15min":1.47},"min":{"1min":0.78300000000000003,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":3.8410000000000002,"5min":4.319,"15min":35.606000000000002},"last":1.1599999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:55:02 2021","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.597,"15min":2.3719999999999999},"min":{"1min":0.89400000000000002,"5min":0.86199999999999999,"15min":0.76600000000000001},"max":{"1min":2.6989999999999998,"5min":12.488,"15min":201.626},"last":1.373},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.476,"15min":2.3940000000000001},"min":{"1min":0.75800000000000001,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":2.625,"5min":12.84,"15min":201.577},"last":1.306}]},{"osd":7,"last update":"Wed Feb 10 04:55:32 2021","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.431,"15min":1.4990000000000001},"min":{"1min":0.90300000000000002,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.351,"5min":2.5099999999999998,"15min":5.8920000000000003},"last":1.4910000000000001},{"interface":"front","average":{"1min":1.105,"5min":1.2410000000000001,"15min":1.496},"min":{"1min":0.79100000000000004,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.819,"5min":2.2250000000000001,"15min":5.149},"last":1.4159999999999999}]}]},{"osd":5,"up_from":37,"seq":158913790328,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":251992,"kb_used_data":206224,"kb_used_omap":0,"kb_used_meta":45760,"kb_avail":93493160,"statfs":{"total":95995035648,"available":95736995840,"internally_reserved":0,"allocated":211173376,"data_stored":275765591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":46858240},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:51 2021","interfaces":[{"interface":"back","average":{"1min":5.0590000000000002,"5min":1.714,"15min":1.204},"min":{"1min":0.45700000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":58.624000000000002,"5min":58.624000000000002,"15min":58.624000000000002},"last":0.502},{"interface":"front","average":{"1min":5.04,"5min":1.784,"15min":1.3009999999999999},"min":{"1min":0.56599999999999995,"5min":0.46600000000000003,"15min":0.46400000000000002},"max":{"1min":57.880000000000003,"5min":57.880000000000003,"15min":57.880000000000003},"last":0.54300000000000004}]},{"osd":1,"last update":"Wed Feb 10 04:55:58 2021","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":0.93999999999999995,"15min":0.94099999999999995},"min":{"1min":0.50600000000000001,"5min":0.435,"15min":0.435},"max":{"1min":4.242,"5min":6.391,"15min":6.391},"last":0.69999999999999996},{"interface":"front","average":{"1min":1.74,"5min":1.2310000000000001,"15min":1.1950000000000001},"min":{"1min":0.626,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":13.794,"5min":13.794,"15min":13.794},"last":0.79000000000000004}]},{"osd":2,"last update":"Wed Feb 10 04:55:58 2021","interfaces":[{"interface":"back","average":{"1min":7.226,"5min":2.831,"15min":1.7949999999999999},"min":{"1min":0.51400000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":63.270000000000003,"5min":63.270000000000003,"15min":63.270000000000003},"last":1.1779999999999999},{"interface":"front","average":{"1min":7.3579999999999997,"5min":2.9630000000000001,"15min":1.831},"min":{"1min":0.68000000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":63.289000000000001,"5min":63.289000000000001,"15min":63.289000000000001},"last":0.92900000000000005}]},{"osd":3,"last update":"Wed Feb 10 04:55:09 2021","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.1759999999999999,"15min":1.282},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":4.7000000000000002,"5min":4.7000000000000002,"15min":4.7000000000000002},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.238,"15min":1.2},"min":{"1min":0.83099999999999996,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":5.577,"5min":6.5129999999999999,"15min":6.5129999999999999},"last":0.81899999999999995}]},{"osd":4,"last update":"Wed Feb 10 04:55:58 2021","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.304,"15min":1.3},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.57099999999999995},"max":{"1min":5.298,"5min":9.4890000000000008,"15min":9.4890000000000008},"last":0.90700000000000003},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.3400000000000001,"15min":1.3700000000000001},"min":{"1min":0.70899999999999996,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":4.8289999999999997,"5min":5.6180000000000003,"15min":5.6180000000000003},"last":1.0669999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:55:09 2021","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":1.4970000000000001,"15min":2.996},"min":{"1min":0.88200000000000001,"5min":0.83899999999999997,"15min":0.65800000000000003},"max":{"1min":4.9429999999999996,"5min":5.8879999999999999,"15min":319.51999999999998},"last":1.3680000000000001},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.5029999999999999,"15min":2.931},"min":{"1min":0.95199999999999996,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":5.4660000000000002,"5min":6.8220000000000001,"15min":319.22300000000001},"last":1.2190000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:55:35 2021","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.4870000000000001,"15min":1.486},"min":{"1min":0.90300000000000002,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":5.4160000000000004,"5min":11.897,"15min":11.897},"last":1.026},{"interface":"front","average":{"1min":1.675,"5min":1.524,"15min":1.5720000000000001},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.82999999999999996},"max":{"1min":5.0579999999999998,"5min":6.0369999999999999,"15min":6.0369999999999999},"last":1.3089999999999999}]}]}]}} 2021-02-10T04:56:06.474 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pool set device_health_metrics pgp_num 11 2021-02-10T04:56:06.681 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:56:08.030 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:07 smithi085 conmon[29319]: cluster 2021-02-10T04:56:06.509033+0000 mgr.y (mgr.14140 2021-02-10T04:56:08.030 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:07 smithi085 conmon[29319]: ) 1392 : cluster [DBG] pgmap v2288: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-02-10T04:56:08.031 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:07 smithi085 conmon[35961]: cluster 2021-02-10T04:56:06.509033+0000 mgr.y 2021-02-10T04:56:08.031 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:07 smithi085 conmon[35961]: (mgr.14140) 1392 : cluster [DBG] pgmap v2288: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-02-10T04:56:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:07 smithi164 conmon[29288]: cluster 2021-02-10T04:56:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:07 smithi164 conmon[29288]: 2021-02-10T04:56 2021-02-10T04:56:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:07 smithi164 conmon[29288]: :06.509033+0000 mgr.y (mgr.14140) 1392 : cluster [DBG] pgmap v2288: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2021-02-10T04:56:08.210 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:56:08.305 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set nodeep-scrub 2021-02-10T04:56:08.441 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:56:08.519 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:56:09.448 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.6 dump_historic_ops 2021-02-10T04:56:09.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:09 smithi164 conmon[29288]: cluster 2021-02-10T04:56:08.509894+0000 mgr.y ( 2021-02-10T04:56:09.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:09 smithi164 conmon[29288]: mgr.14140) 1393 : cluster [DBG] pgmap v2289: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 18 op/s 2021-02-10T04:56:09.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:09 smithi164 conmon[29288]: cluster 2021-02-10T04:56:08.568771+0000 mon.a (mon.0) 2724 : cluster [DBG] osdmap e1265: 8 total, 8 up, 8 in 2021-02-10T04:56:09.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:09 smithi164 conmon[29288]: audit 2021-02-10T04:56:08.811692+0000 mon.a (mon.0) 2725 : audit [INF] from='client.? 172.21.15.85:0/1645670557' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "11"}]: dispatch 2021-02-10T04:56:09.645 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:56:09 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:56:09.708 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:56:09.714 INFO:teuthology.orchestra.run.smithi085.stderr:set pool 1 pgp_num to 11 2021-02-10T04:56:09.735 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:09 smithi085 conmon[29319]: cluster 2021-02-10T04:56:08.509894+0000 mgr.y (mgr.14140) 1393 : cluster [DBG] pgmap v2289: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 18 op/s 2021-02-10T04:56:09.735 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:09 smithi085 conmon[29319]: cluster 2021-02-10T04:56:08.568771+0000 mon.a (mon.0) 2724 : cluster [DBG] osdmap e1265: 8 total, 8 up, 8 in 2021-02-10T04:56:09.735 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:09 smithi085 conmon[29319]: audit 2021-02-10T04:56:08.811692+0000 mon.a (mon.0) 2725 : audit [INF] from='client.? 172.21.15.85:0/1645670557' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "11"}]: dispatch 2021-02-10T04:56:09.735 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:09 smithi085 conmon[35961]: cluster 2021-02-10T04:56:08.509894+0000 mgr.y (mgr.14140) 1393 : cluster [DBG] pgmap v2289: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 18 op/s 2021-02-10T04:56:09.735 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:09 smithi085 conmon[35961]: cluster 2021-02-10T04:56:08.568771+0000 mon.a (mon.0) 2724 : cluster [DBG] osdmap e1265: 8 total, 8 up, 8 in 2021-02-10T04:56:09.735 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:09 smithi085 conmon[35961]: audit 2021-02-10T04:56:08.811692+0000 mon.a (mon.0) 2725 : audit [INF] from='client.? 172.21.15.85:0/1645670557' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "11"}]: dispatch 2021-02-10T04:56:10.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:10 smithi085 conmon[29319]: audit 2021-02-10T04:56:09.354950+0000 mgr.y (mgr.14140) 2021-02-10T04:56:10.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:10 smithi085 conmon[29319]: 1394 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:56:10.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:10 smithi085 conmon[29319]: audit 2021-02-10T04:56:09.711031+0000 mon.a (mon.0) 2726 : audit [INF] from='client.? 172.21.15.85:0/1645670557' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "11"}]': finished 2021-02-10T04:56:10.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:10 smithi085 conmon[29319]: cluster 2021-02-10T04:56:10.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:10 smithi085 conmon[29319]: 2021-02-10T04:56:09.711100+0000 mon.a (mon.0) 2727 : cluster [DBG] osdmap e1266: 8 total, 8 up, 8 in 2021-02-10T04:56:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:10 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:56:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:10 smithi085 conmon[35961]: 04:56:09.354950+0000 mgr.y (mgr.14140) 1394 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:56:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:10 smithi085 conmon[35961]: audit 2021-02-10T04:56:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:10 smithi085 conmon[35961]: 2021-02-10T04:56:09.711031 2021-02-10T04:56:10.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:10 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2726 : audit [INF] from='client.? 172.21.15.85:0/1645670557' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "11"}]': finished 2021-02-10T04:56:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:10 smithi085 conmon[35961]: 2021-02-10T04:56:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:10 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:56:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:10 smithi085 conmon[35961]: 10T04:56:09.711100+0000 mon.a (mon.0) 2727 : cluster [DBG] 2021-02-10T04:56:10.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:10 smithi085 conmon[35961]: osdmap e1266: 8 total, 8 up, 8 in 2021-02-10T04:56:11.083 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:10 smithi164 conmon[29288]: audit 2021-02-10T04:56:09.354950+0000 mgr.y (mgr.14140) 2021-02-10T04:56:11.084 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:10 smithi164 conmon[29288]: 1394 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:56:11.084 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:10 smithi164 conmon[29288]: audit 2021-02-10T04:56:09.711031+0000 mon.a (mon.0) 2726 : audit [INF] from='client.? 172.21.15.85:0/1645670557' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "11"}]': finished 2021-02-10T04:56:11.084 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:10 smithi164 conmon[29288]: cluster 2021-02-10T04:56:09 2021-02-10T04:56:11.084 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:10 smithi164 conmon[29288]: .711100+0000 mon.a (mon.0) 2727 : cluster [DBG] osdmap e1266: 8 total, 8 up, 8 in 2021-02-10T04:56:11.716 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:56:11.738 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:56:11.757 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:56:11.776 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:56:11.803 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:56:11.828 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:56:11.848 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:56:11.866 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:56:11.868 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:56:11.868 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:56:11.868 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:56:11.868 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:56:11.868 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:56:11.868 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:56:11.868 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:56:11.868 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:56:11.889 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:11 smithi085 conmon[29319]: cluster 2021-02-10T04:56:10.510362+0000 mgr.y (mgr. 2021-02-10T04:56:11.889 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:11 smithi085 conmon[29319]: 14140) 1395 : cluster [DBG] pgmap v2292: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:56:11.889 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:11 smithi085 conmon[29319]: audit 2021-02-10T04:56:11.694044+0000 mon.c (mon.1) 130 : audit [DBG] from='client.? 172.21.15.85:0/3108899286' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:56:11.889 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:11 smithi085 conmon[35961]: cluster 2021-02-10T04:56:10.510362+0000 mgr.y (mgr.14140) 2021-02-10T04:56:11.889 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:11 smithi085 conmon[35961]: 1395 : cluster [DBG] pgmap v2292: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:56:11.889 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:11 smithi085 conmon[35961]: audit 2021-02-10T04:56:11.694044 2021-02-10T04:56:11.890 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:11 smithi085 conmon[35961]: +0000 mon.c (mon.1) 130 : audit [DBG] from='client.? 172.21.15.85:0/3108899286' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:56:11.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:11 smithi164 conmon[29288]: cluster 2021-02-10T04:56:10.510362+0000 mgr.y (mgr. 2021-02-10T04:56:11.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:11 smithi164 conmon[29288]: 14140) 1395 : cluster [DBG] pgmap v2292: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:56:11.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:11 smithi164 conmon[29288]: audit 2021-02-10T04:56:11.694044+0000 mon.c (mon.1) 130 : audit [DBG] from='client.? 172.21.15.85:0/3108899286' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:56:12.733 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is set 2021-02-10T04:56:13.009 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:12 smithi085 conmon[35961]: audit 2021-02-10T04:56:13.009 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:12 smithi085 conmon[35961]: 2021-02 2021-02-10T04:56:13.009 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:12 smithi085 conmon[35961]: -10T04:56:12.049772+0000 mon.c (mon.1) 131 : audit [INF] from='client.? 172.21.15.85:0/4270547972' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:56:13.009 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:12 smithi085 conmon[35961]: 2021-02-10T04:56:13.009 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:12 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:56:13.009 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:12 smithi085 conmon[35961]: 10T04:56:12.050310+0000 mon.a ( 2021-02-10T04:56:13.009 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:12 smithi085 conmon[35961]: mon.0) 2728 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:56:13.009 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:12 smithi085 conmon[29319]: audit 2021-02-10T04:56:12. 2021-02-10T04:56:13.009 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:12 smithi085 conmon[29319]: 049772+0000 mon.c (mon.1) 131 : audit [INF] from='client.? 172.21.15.85:0/4270547972' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:56:13.010 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:12 smithi085 conmon[29319]: audit 2021-02-10T04: 2021-02-10T04:56:13.010 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:12 smithi085 conmon[29319]: 56:12.050310+0000 mon.a (mon.0) 2728 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:56:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:12 smithi164 conmon[29288]: audit 2021-02-10T04:56:12.049772+0000 2021-02-10T04:56:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:12 smithi164 conmon[29288]: mon.c (mon.1) 131 : audit [INF] from='client.? 172.21.15.85:0/4270547972' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:56:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:12 smithi164 conmon[29288]: audit 2021-02-10T04:56:12.050310+0000 mon.a (mon.0) 2728 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:56:13.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[29319]: cluster 2021-02-10T04:56:12.508785+0000 osd.5 (osd. 2021-02-10T04:56:13.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[29319]: 5) 192 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:56:13.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[29319]: cluster 2021-02-10T04:56:12.510688+0000 osd.5 (osd.5) 193 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:56:13.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[29319]: cluster 2021-02-10T04:56:12.510862 2021-02-10T04:56:13.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 1396 : cluster [DBG] 2021-02-10T04:56:13.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[29319]: pgmap v2293: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 655 B/s rd, 0 op/s 2021-02-10T04:56:13.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:56:13.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[29319]: :56:12.714848+0000 mon.a (mon.0 2021-02-10T04:56:13.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[29319]: ) 2729 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:56:13.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[29319]: audit 2021 2021-02-10T04:56:13.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[29319]: -02-10T04:56:12.726406+0000 2021-02-10T04:56:13.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[29319]: mon.a (mon.0) 2730 : audit [INF] 2021-02-10T04:56:13.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[29319]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:56:13.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:56:13.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[29319]: 56:12.726462+0000 mon.a (mon.0) 2021-02-10T04:56:13.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[29319]: 2731 : cluster [DBG] osdmap e1267: 8 total, 8 up, 8 in 2021-02-10T04:56:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[35961]: cluster 2021-02-10T04:56:12 2021-02-10T04:56:13.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[35961]: .508785+0000 osd.5 (osd.5) 192 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:56:13.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[35961]: cluster 2021-02-10T04:56: 2021-02-10T04:56:13.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[35961]: 12.510688+0000 osd.5 (osd.5) 193 : cluster 2021-02-10T04:56:13.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[35961]: [DBG] purged_snaps scrub ok 2021-02-10T04:56:13.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:56:13.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[35961]: T04:56:12.510862+0000 mgr.y (mgr.14140 2021-02-10T04:56:13.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[35961]: ) 1396 : cluster [DBG] pgmap v2293: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 655 B/s rd, 0 op/s 2021-02-10T04:56:13.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:56:13.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[35961]: 04:56:12.714848+0000 mon.a (mon.0 2021-02-10T04:56:13.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[35961]: ) 2729 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:56:13.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[35961]: audit 2021-02-10T04:56 2021-02-10T04:56:13.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[35961]: :12.726406+0000 mon.a (mon.0) 2730 : audit 2021-02-10T04:56:13.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[35961]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:56:13.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[35961]: cluster 2021-02-10T04:56:12. 2021-02-10T04:56:13.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[35961]: 726462+0000 mon.a (mon.0) 2731 : cluster [DBG] 2021-02-10T04:56:13.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:13 smithi085 conmon[35961]: osdmap e1267: 8 total, 8 up, 8 in 2021-02-10T04:56:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:13 smithi164 conmon[29288]: cluster 2021-02-10T04:56:12 2021-02-10T04:56:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:13 smithi164 conmon[29288]: .508785+0000 osd.5 (osd.5) 192 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:56:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:13 smithi164 conmon[29288]: cluster 2021-02-10T04:56:12.510688+0000 osd.5 (osd.5) 193 2021-02-10T04:56:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:13 smithi164 conmon[29288]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:56:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:13 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:56:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:13 smithi164 conmon[29288]: -02-10T04:56:12.510862+0000 mgr.y (mgr.14140) 1396 : 2021-02-10T04:56:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:13 smithi164 conmon[29288]: cluster [DBG] pgmap v2293: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 655 B/s rd, 0 op/s 2021-02-10T04:56:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:13 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:56:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:13 smithi164 conmon[29288]: 56:12.714848+0000 mon.a (mon.0) 2729 2021-02-10T04:56:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:13 smithi164 conmon[29288]: : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:56:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:13 smithi164 conmon[29288]: audit 2021-02-10T04:56:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:13 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:56:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:13 smithi164 conmon[29288]: :56:12.726406+0000 mon.a (mon.0) 2730 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:56:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:13 smithi164 conmon[29288]: cluster 2021-02-10T04:56:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:13 smithi164 conmon[29288]: 2021-02 2021-02-10T04:56:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:13 smithi164 conmon[29288]: -10T04:56:12.726462+0000 2021-02-10T04:56:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:13 smithi164 conmon[29288]: mon.a (mon.0) 2731 : cluster [DBG] 2021-02-10T04:56:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:13 smithi164 conmon[29288]: osdmap e1267: 8 total, 8 up, 8 in 2021-02-10T04:56:14.826 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:56:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:14 smithi164 conmon[29288]: cluster 2021-02-10T04:56:13.462919+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:56:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:14 smithi164 conmon[29288]: cluster 2021-02-10T04:56:13.467199+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:56:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:14 smithi164 conmon[29288]: cluster 2021-02-10T04:56:13.731376+0000 mon.a (mon.0) 2732 : cluster [DBG] osdmap e1268: 8 total, 8 up, 8 in 2021-02-10T04:56:14.963 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:56:14.964 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:56:15.047 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:56:15.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:14 smithi085 conmon[29319]: cluster 2021-02-10T04:56:13.462919+0000 osd.0 (osd.0) 4 : cluster 2021-02-10T04:56:15.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:14 smithi085 conmon[29319]: [DBG] purged_snaps scrub starts 2021-02-10T04:56:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:14 smithi085 conmon[29319]: cluster 2021-02-10T04:56:13.467199+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:56:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:14 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:56:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:14 smithi085 conmon[29319]: -02-10T04:56:13.731376+0000 mon.a (mon.0) 2732 : cluster [DBG] osdmap e1268: 8 total, 8 up, 8 in 2021-02-10T04:56:15.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:14 smithi085 conmon[35961]: cluster 2021-02-10T04:56:13.462919+0000 osd.0 (osd.0) 4 : cluster 2021-02-10T04:56:15.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:14 smithi085 conmon[35961]: [DBG] purged_snaps scrub starts 2021-02-10T04:56:15.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:14 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:56:15.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:14 smithi085 conmon[35961]: 02-10T04:56:13.467199+0000 osd.0 (osd.0) 5 : cluster [DBG] 2021-02-10T04:56:15.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:14 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:56:15.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:14 smithi085 conmon[35961]: cluster 2021-02-10T04:56:15.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:14 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:56:15.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:14 smithi085 conmon[35961]: 04:56:13.731376+0000 mon.a (mon. 2021-02-10T04:56:15.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:14 smithi085 conmon[35961]: 0) 2732 : cluster [DBG] osdmap e1268: 8 total, 8 up, 8 in 2021-02-10T04:56:15.579 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset noscrub 2021-02-10T04:56:15.816 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:56:15.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:15 smithi085 conmon[29319]: cluster 2021-02-10T04:56:14.511546+0000 mgr.y (mgr.14140) 1397 : cluster [DBG] pgmap v2296: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 861 B/s rd, 0 op/s 2021-02-10T04:56:15.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:15 smithi085 conmon[35961]: cluster 2021-02-10T04:56:14.511546+0000 2021-02-10T04:56:15.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:15 smithi085 conmon[35961]: mgr.y (mgr.14140) 1397 : cluster [DBG] pgmap v2296: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 861 B/s rd, 0 op/s 2021-02-10T04:56:16.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:15 smithi164 conmon[29288]: cluster 2021-02-10T04:56:14. 2021-02-10T04:56:16.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:15 smithi164 conmon[29288]: 511546+0000 mgr.y (mgr.14140) 1397 : cluster [DBG] pgmap v2296: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 861 B/s rd, 0 op/s 2021-02-10T04:56:16.358 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2021-02-10T04:56:16.358 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:56:16.359 INFO:tasks.thrashosds.thrasher:fixing pg num pool .rgw.root 2021-02-10T04:56:16.359 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:56:16.581 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:56:17.750 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:56:17.768 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:56:17.786 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:56:17.804 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:56:17.823 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:56:17.922 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:56:17.922 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:56:17.923 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:56:17.923 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:56:17.923 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:56:17.923 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:56:17.923 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:56:17.923 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:56:17.923 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:56:17.923 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:56:17.923 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:56:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:17 smithi164 conmon[29288]: cluster 2021-02-10T04:56:16.512024+0000 mgr.y (mgr 2021-02-10T04:56:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:17 smithi164 conmon[29288]: .14140) 1398 : cluster [DBG] pgmap v2297: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 752 B/s rd, 0 op/s 2021-02-10T04:56:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:17 smithi164 conmon[29288]: audit 2021-02-10T04:56:17.725190+0000 mon.a (mon.0) 2733 : audit [DBG] from='client.? 172.21.15.85:0/1020324601' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:56:18.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:17 smithi085 conmon[35961]: cluster 2021-02-10T04:56:16.512024+0000 mgr.y (mgr. 2021-02-10T04:56:18.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:17 smithi085 conmon[35961]: 14140) 1398 : cluster [DBG] pgmap v2297: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 752 B/s rd, 0 op/s 2021-02-10T04:56:18.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:17 smithi085 conmon[35961]: audit 2021-02-10T04:56:17.725190+0000 mon.a (mon 2021-02-10T04:56:18.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:17 smithi085 conmon[35961]: .0) 2733 : audit [DBG] from='client.? 172.21.15.85:0/1020324601' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:56:18.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:17 smithi085 conmon[29319]: cluster 2021-02-10T04:56:16.512024+0000 mgr.y ( 2021-02-10T04:56:18.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:17 smithi085 conmon[29319]: mgr.14140) 1398 : cluster [DBG] pgmap v2297: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 752 B/s rd, 0 op/s 2021-02-10T04:56:18.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:17 smithi085 conmon[29319]: audit 2021-02-10T04:56:17.725190+0000 mon.a (mon.0) 2733 : audit [DBG] from='client.? 172.21.15.85:0/1020324601' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:56:18.329 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.7 dump_ops_in_flight 2021-02-10T04:56:18.624 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:56:19.366 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:56:19 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:56:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:19 smithi164 conmon[29288]: cluster 2021-02-10T04:56:18.512739+0000 2021-02-10T04:56:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:19 smithi164 conmon[29288]: mgr.y (mgr.14140) 1399 : cluster [DBG] pgmap v2298: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:56:19.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:19 smithi164 conmon[29288]: cluster 2021-02-10T04:56:18.572640+0000 mon.a (mon.0) 2734 : cluster [DBG] osdmap e1269: 8 total, 8 up, 8 in 2021-02-10T04:56:19.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:19 smithi085 conmon[29319]: cluster 2021-02-10T04:56:18.512739+0000 mgr.y (mgr.14140) 1399 : cluster [DBG] pgmap v2298: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:56:19.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:19 smithi085 conmon[29319]: cluster 2021-02-10T04:56:18.572640+0000 mon.a (mon.0) 2734 : cluster [DBG] osdmap e1269: 8 total, 8 up, 8 in 2021-02-10T04:56:19.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:19 smithi085 conmon[35961]: cluster 2021-02-10T04:56:18.512739+0000 mgr.y (mgr.14140) 1399 : cluster [DBG] 2021-02-10T04:56:19.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:19 smithi085 conmon[35961]: pgmap v2298: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:56:19.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:19 smithi085 conmon[35961]: cluster 2021-02-10T04:56:18.572640+0000 mon.a (mon.0) 2734 : cluster [DBG] osdmap e1269: 8 total, 8 up, 8 in 2021-02-10T04:56:20.111 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:56:20.115 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:56:20.755 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is unset 2021-02-10T04:56:20.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:20 smithi164 conmon[29288]: cluster 2021-02-10T04:56:19. 2021-02-10T04:56:20.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:20 smithi164 conmon[29288]: 292675+0000 osd.2 (osd.2) 2033 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:56:20.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:20 smithi164 conmon[29288]: cluster 2021-02-10T04:56:19.295360 2021-02-10T04:56:20.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:20 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 2034 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:56:20.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:20 smithi164 conmon[29288]: audit 2021-02-10T04:56:20.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:20 smithi164 conmon[29288]: 2021-02-10T04:56:19.365023+0000 mgr.y (mgr.14140) 1400 : audit [DBG] 2021-02-10T04:56:20.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:20 smithi164 conmon[29288]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:56:20.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:20 smithi164 conmon[29288]: cluster 2021-02-10T04:56:20.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:20 smithi164 conmon[29288]: 2021-02-10T04:56:19.747724+0000 mon.a (mon.0) 2735 : cluster [DBG] 2021-02-10T04:56:20.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:20 smithi164 conmon[29288]: osdmap e1270: 8 total, 8 up, 8 in 2021-02-10T04:56:20.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:20 smithi164 conmon[29288]: audit 2021-02-10T04:56:20.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:20 smithi164 conmon[29288]: 2021-02-10T04:56:20. 2021-02-10T04:56:20.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:20 smithi164 conmon[29288]: 091139+0000 mgr.y (mgr.14140) 2021-02-10T04:56:20.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:20 smithi164 conmon[29288]: 1401 : audit [DBG] from='client.26068 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:56:20.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:20 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:56:20.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:20 smithi164 conmon[29288]: :56:20.111175+0000 mon.a (mon.0) 2736 : audit [INF] from='client.? 172.21.15.85:0/2063679820' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:56:20.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[35961]: cluster 2021-02-10T04:56:19.292675+0000 2021-02-10T04:56:20.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[35961]: osd.2 (osd.2) 2033 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:56:20.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[35961]: cluster 2021-02-10T04:56: 2021-02-10T04:56:20.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[35961]: 19.295360+0000 osd.2 (osd.2) 2034 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:56:20.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:56:20.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[35961]: 02-10T04:56:19.365023+0000 mgr.y 2021-02-10T04:56:20.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[35961]: (mgr.14140) 1400 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:56:20.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[35961]: cluster 2021-02-10T04:56:20.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[35961]: 2021-02-10T04:56:19.747724+0000 2021-02-10T04:56:20.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[35961]: mon.a (mon.0) 2735 : cluster [DBG] 2021-02-10T04:56:20.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[35961]: osdmap e1270: 8 total, 8 up, 8 in 2021-02-10T04:56:20.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:56:20.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[35961]: 56:20.091139+0000 mgr.y (mgr.14140 2021-02-10T04:56:20.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[35961]: ) 1401 : audit [DBG] from='client.26068 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:56:20.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[35961]: audit 2021 2021-02-10T04:56:20.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[35961]: -02-10T04:56:20.111175+0000 2021-02-10T04:56:20.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[35961]: mon.a (mon.0) 2736 : audit [INF] 2021-02-10T04:56:20.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/2063679820' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:56:20.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[29319]: cluster 2021-02-10T04:56:19. 2021-02-10T04:56:20.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[29319]: 292675+0000 osd.2 (osd.2) 2033 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:56:20.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:56:20.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[29319]: 56:19.295360+0000 osd.2 (osd.2) 2034 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:56:20.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[29319]: 2021-02-10T04:56:20.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[29319]: audit 2021-02-10T04:56 2021-02-10T04:56:20.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[29319]: :19.365023+0000 mgr.y (mgr.14140) 2021-02-10T04:56:20.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[29319]: 1400 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:56:20.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:56:20.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[29319]: -10T04:56:19.747724+0000 2021-02-10T04:56:20.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[29319]: mon.a (mon.0) 2735 : cluster [DBG] 2021-02-10T04:56:20.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[29319]: osdmap e1270: 8 total, 8 up, 8 in 2021-02-10T04:56:20.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[29319]: audit 2021-02-10T04:56 2021-02-10T04:56:20.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[29319]: :20.091139+0000 mgr.y (mgr.14140) 2021-02-10T04:56:20.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[29319]: 1401 : audit [DBG] from='client.26068 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:56:20.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:56:20.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[29319]: -10T04:56:20.111175+0000 2021-02-10T04:56:20.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[29319]: mon.a (mon.0) 2736 : audit [INF] 2021-02-10T04:56:20.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:20 smithi085 conmon[29319]: from='client.? 172.21.15.85:0/2063679820' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:56:21.382 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:56:21.461 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":2300,"stamp":"2021-02-10T04:56:19.745588+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":972357089,"num_objects":748,"num_object_clones":337,"num_object_copies":2244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":748,"num_whiteouts":10,"num_read":7875,"num_read_kb":4562274,"num_write":4841,"num_write_kb":1247750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":887,"num_bytes_recovered":311980434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35650,"ondisk_log_size":35650,"up":594,"acting":594,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":596,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2309388,"kb_used_data":1980340,"kb_used_omap":692,"kb_used_meta":325771,"kb_avail":747651828,"statfs":{"total":767960285184,"available":765595471872,"internally_reserved":0,"allocated":2027868160,"data_stored":2597806573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":709615,"internal_metadata":333589521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.021702"},"pg_stats":[{"pgid":"6.29","version":"0'0","reported_seq":"1124","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781248+0000","last_change":"2021-02-10T04:55:14.633086+0000","last_active":"2021-02-10T04:56:12.781248+0000","last_peered":"2021-02-10T04:56:12.781248+0000","last_clean":"2021-02-10T04:56:12.781248+0000","last_became_active":"2021-02-10T04:55:14.632670+0000","last_became_peered":"2021-02-10T04:55:14.632670+0000","last_unstale":"2021-02-10T04:56:12.781248+0000","last_undegraded":"2021-02-10T04:56:12.781248+0000","last_fullsized":"2021-02-10T04:56:12.781248+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:24.121684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:24.121684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":"164","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.727894+0000","last_change":"2021-02-10T04:52:30.133105+0000","last_active":"2021-02-10T04:56:12.727894+0000","last_peered":"2021-02-10T04:56:12.727894+0000","last_clean":"2021-02-10T04:56:12.727894+0000","last_became_active":"2021-02-10T04:51:36.864926+0000","last_became_peered":"2021-02-10T04:51:36.864926+0000","last_unstale":"2021-02-10T04:56:12.727894+0000","last_undegraded":"2021-02-10T04:56:12.727894+0000","last_fullsized":"2021-02-10T04:56:12.727894+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.133043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.133043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"1191'4","reported_seq":"248","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:13.044603+0000","last_change":"2021-02-10T04:53:46.736164+0000","last_active":"2021-02-10T04:56:13.044603+0000","last_peered":"2021-02-10T04:56:13.044603+0000","last_clean":"2021-02-10T04:56:13.044603+0000","last_became_active":"2021-02-10T04:53:46.735711+0000","last_became_peered":"2021-02-10T04:53:46.735711+0000","last_unstale":"2021-02-10T04:56:13.044603+0000","last_undegraded":"2021-02-10T04:56:13.044603+0000","last_fullsized":"2021-02-10T04:56:13.044603+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'3","last_scrub_stamp":"2021-02-10T04:51:01.185820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.185820+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"1261'84","reported_seq":"1400","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.727351+0000","last_change":"2021-02-10T04:55:39.594594+0000","last_active":"2021-02-10T04:56:12.727351+0000","last_peered":"2021-02-10T04:56:12.727351+0000","last_clean":"2021-02-10T04:56:12.727351+0000","last_became_active":"2021-02-10T04:55:39.594275+0000","last_became_peered":"2021-02-10T04:55:39.594275+0000","last_unstale":"2021-02-10T04:56:12.727351+0000","last_undegraded":"2021-02-10T04:56:12.727351+0000","last_fullsized":"2021-02-10T04:56:12.727351+0000","mapping_epoch":55,"log_start":"1132'66","ondisk_log_start":"1132'66","created":55,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":0,"last_scrub":"1132'66","last_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_deep_scrub":"1132'66","last_deep_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.430399+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1616,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":186,"num_read_kb":150,"num_write":107,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"173","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.730323+0000","last_change":"2021-02-10T04:53:41.722609+0000","last_active":"2021-02-10T04:56:12.730323+0000","last_peered":"2021-02-10T04:56:12.730323+0000","last_clean":"2021-02-10T04:56:12.730323+0000","last_became_active":"2021-02-10T04:51:36.875000+0000","last_became_peered":"2021-02-10T04:51:36.875000+0000","last_unstale":"2021-02-10T04:56:12.730323+0000","last_undegraded":"2021-02-10T04:56:12.730323+0000","last_fullsized":"2021-02-10T04:56:12.730323+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:53:41.722516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:41.722516+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"1270","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.727993+0000","last_change":"2021-02-10T04:56:01.195173+0000","last_active":"2021-02-10T04:56:12.727993+0000","last_peered":"2021-02-10T04:56:12.727993+0000","last_clean":"2021-02-10T04:56:12.727993+0000","last_became_active":"2021-02-10T04:55:14.630535+0000","last_became_peered":"2021-02-10T04:55:14.630535+0000","last_unstale":"2021-02-10T04:56:12.727993+0000","last_undegraded":"2021-02-10T04:56:12.727993+0000","last_fullsized":"2021-02-10T04:56:12.727993+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:48.408748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:50:48.408748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"1003'1689","reported_seq":"4687","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.728444+0000","last_change":"2021-02-10T04:53:38.170873+0000","last_active":"2021-02-10T04:56:12.728444+0000","last_peered":"2021-02-10T04:56:12.728444+0000","last_clean":"2021-02-10T04:56:12.728444+0000","last_became_active":"2021-02-10T04:52:26.030105+0000","last_became_peered":"2021-02-10T04:52:26.030105+0000","last_unstale":"2021-02-10T04:56:12.728444+0000","last_undegraded":"2021-02-10T04:56:12.728444+0000","last_fullsized":"2021-02-10T04:56:12.728444+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:53:38.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.170802+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":99361981,"num_objects":37,"num_object_clones":33,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"62'5","reported_seq":"170","reported_epoch":"1268","state":"active+clean","last_fresh":"2021-02-10T04:56:16.727237+0000","last_change":"2021-02-10T04:54:39.615784+0000","last_active":"2021-02-10T04:56:16.727237+0000","last_peered":"2021-02-10T04:56:16.727237+0000","last_clean":"2021-02-10T04:56:16.727237+0000","last_became_active":"2021-02-10T04:54:39.560009+0000","last_became_peered":"2021-02-10T04:54:39.560009+0000","last_unstale":"2021-02-10T04:56:16.727237+0000","last_undegraded":"2021-02-10T04:56:16.727237+0000","last_fullsized":"2021-02-10T04:56:16.727237+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":249,"num_read_kb":248,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"1261'28","reported_seq":"1277","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.780894+0000","last_change":"2021-02-10T04:55:21.669679+0000","last_active":"2021-02-10T04:56:12.780894+0000","last_peered":"2021-02-10T04:56:12.780894+0000","last_clean":"2021-02-10T04:56:12.780894+0000","last_became_active":"2021-02-10T04:55:21.669219+0000","last_became_peered":"2021-02-10T04:55:21.669219+0000","last_unstale":"2021-02-10T04:56:12.780894+0000","last_undegraded":"2021-02-10T04:56:12.780894+0000","last_fullsized":"2021-02-10T04:56:12.780894+0000","mapping_epoch":1228,"log_start":"1061'20","ondisk_log_start":"1061'20","created":55,"last_epoch_clean":1233,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'32","last_scrub_stamp":"2021-02-10T04:51:51.382040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.382040+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1158,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":105,"num_read_kb":81,"num_write":58,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2052,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"919","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.732299+0000","last_change":"2021-02-10T04:53:34.179015+0000","last_active":"2021-02-10T04:56:12.732299+0000","last_peered":"2021-02-10T04:56:12.732299+0000","last_clean":"2021-02-10T04:56:12.732299+0000","last_became_active":"2021-02-10T04:51:49.502576+0000","last_became_peered":"2021-02-10T04:51:49.502576+0000","last_unstale":"2021-02-10T04:56:12.732299+0000","last_undegraded":"2021-02-10T04:56:12.732299+0000","last_fullsized":"2021-02-10T04:56:12.732299+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:34.178949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.178949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"164","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.728734+0000","last_change":"2021-02-10T04:51:53.038893+0000","last_active":"2021-02-10T04:56:12.728734+0000","last_peered":"2021-02-10T04:56:12.728734+0000","last_clean":"2021-02-10T04:56:12.728734+0000","last_became_active":"2021-02-10T04:51:36.855507+0000","last_became_peered":"2021-02-10T04:51:36.855507+0000","last_unstale":"2021-02-10T04:56:12.728734+0000","last_undegraded":"2021-02-10T04:56:12.728734+0000","last_fullsized":"2021-02-10T04:56:12.728734+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:53.038834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:53.038834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"1186","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.780189+0000","last_change":"2021-02-10T04:55:28.097406+0000","last_active":"2021-02-10T04:56:12.780189+0000","last_peered":"2021-02-10T04:56:12.780189+0000","last_clean":"2021-02-10T04:56:12.780189+0000","last_became_active":"2021-02-10T04:55:14.629603+0000","last_became_peered":"2021-02-10T04:55:14.629603+0000","last_unstale":"2021-02-10T04:56:12.780189+0000","last_undegraded":"2021-02-10T04:56:12.780189+0000","last_fullsized":"2021-02-10T04:56:12.780189+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:09.922789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:09.922789+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"1168'82","reported_seq":"200","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.728005+0000","last_change":"2021-02-10T04:53:23.326363+0000","last_active":"2021-02-10T04:56:12.728005+0000","last_peered":"2021-02-10T04:56:12.728005+0000","last_clean":"2021-02-10T04:56:12.728005+0000","last_became_active":"2021-02-10T04:51:36.859215+0000","last_became_peered":"2021-02-10T04:51:36.859215+0000","last_unstale":"2021-02-10T04:56:12.728005+0000","last_undegraded":"2021-02-10T04:56:12.728005+0000","last_fullsized":"2021-02-10T04:56:12.728005+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1150'76","last_scrub_stamp":"2021-02-10T04:53:00.964499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:00.964499+0000","log_size":82,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3848803,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"1261'69","reported_seq":"1408","reported_epoch":"1268","state":"active+clean","last_fresh":"2021-02-10T04:56:13.734286+0000","last_change":"2021-02-10T04:55:25.576927+0000","last_active":"2021-02-10T04:56:13.734286+0000","last_peered":"2021-02-10T04:56:13.734286+0000","last_clean":"2021-02-10T04:56:13.734286+0000","last_became_active":"2021-02-10T04:55:25.576633+0000","last_became_peered":"2021-02-10T04:55:25.576633+0000","last_unstale":"2021-02-10T04:56:13.734286+0000","last_undegraded":"2021-02-10T04:56:13.734286+0000","last_fullsized":"2021-02-10T04:56:13.734286+0000","mapping_epoch":55,"log_start":"1112'47","ondisk_log_start":"1112'47","created":55,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'61","last_scrub_stamp":"2021-02-10T04:53:36.836202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:36.836202+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1466,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":208,"num_read_kb":172,"num_write":122,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"1228'6","reported_seq":"241","reported_epoch":"1268","state":"active+clean","last_fresh":"2021-02-10T04:56:13.733399+0000","last_change":"2021-02-10T04:55:14.605868+0000","last_active":"2021-02-10T04:56:13.733399+0000","last_peered":"2021-02-10T04:56:13.733399+0000","last_clean":"2021-02-10T04:56:13.733399+0000","last_became_active":"2021-02-10T04:55:14.605314+0000","last_became_peered":"2021-02-10T04:55:14.605314+0000","last_unstale":"2021-02-10T04:56:13.733399+0000","last_undegraded":"2021-02-10T04:56:13.733399+0000","last_fullsized":"2021-02-10T04:56:13.733399+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'4","last_scrub_stamp":"2021-02-10T04:52:01.121380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.121380+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"179","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.728330+0000","last_change":"2021-02-10T04:55:22.573715+0000","last_active":"2021-02-10T04:56:12.728330+0000","last_peered":"2021-02-10T04:56:12.728330+0000","last_clean":"2021-02-10T04:56:12.728330+0000","last_became_active":"2021-02-10T04:55:14.636895+0000","last_became_peered":"2021-02-10T04:55:14.636895+0000","last_unstale":"2021-02-10T04:56:12.728330+0000","last_undegraded":"2021-02-10T04:56:12.728330+0000","last_fullsized":"2021-02-10T04:56:12.728330+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:55:22.573621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:55:22.573621+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"564","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781134+0000","last_change":"2021-02-10T04:53:46.733236+0000","last_active":"2021-02-10T04:56:12.781134+0000","last_peered":"2021-02-10T04:56:12.781134+0000","last_clean":"2021-02-10T04:56:12.781134+0000","last_became_active":"2021-02-10T04:53:46.732890+0000","last_became_peered":"2021-02-10T04:53:46.732890+0000","last_unstale":"2021-02-10T04:56:12.781134+0000","last_undegraded":"2021-02-10T04:56:12.781134+0000","last_fullsized":"2021-02-10T04:56:12.781134+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:35.170157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:35.170157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"1003'739","reported_seq":"3865","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.780282+0000","last_change":"2021-02-10T04:55:28.091164+0000","last_active":"2021-02-10T04:56:12.780282+0000","last_peered":"2021-02-10T04:56:12.780282+0000","last_clean":"2021-02-10T04:56:12.780282+0000","last_became_active":"2021-02-10T04:55:14.648601+0000","last_became_peered":"2021-02-10T04:55:14.648601+0000","last_unstale":"2021-02-10T04:56:12.780282+0000","last_undegraded":"2021-02-10T04:56:12.780282+0000","last_fullsized":"2021-02-10T04:56:12.780282+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:52:27.602006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:27.602006+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24059888,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":129,"num_read_kb":109544,"num_write":102,"num_write_kb":46173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"1245","reported_epoch":"1268","state":"active+clean","last_fresh":"2021-02-10T04:56:13.733737+0000","last_change":"2021-02-10T04:54:39.559980+0000","last_active":"2021-02-10T04:56:13.733737+0000","last_peered":"2021-02-10T04:56:13.733737+0000","last_clean":"2021-02-10T04:56:13.733737+0000","last_became_active":"2021-02-10T04:54:39.559443+0000","last_became_peered":"2021-02-10T04:54:39.559443+0000","last_unstale":"2021-02-10T04:56:13.733737+0000","last_undegraded":"2021-02-10T04:56:13.733737+0000","last_fullsized":"2021-02-10T04:56:13.733737+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"1261'64","reported_seq":"1452","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.725864+0000","last_change":"2021-02-10T04:53:37.108747+0000","last_active":"2021-02-10T04:56:12.725864+0000","last_peered":"2021-02-10T04:56:12.725864+0000","last_clean":"2021-02-10T04:56:12.725864+0000","last_became_active":"2021-02-10T04:53:35.551311+0000","last_became_peered":"2021-02-10T04:53:35.551311+0000","last_unstale":"2021-02-10T04:56:12.725864+0000","last_undegraded":"2021-02-10T04:56:12.725864+0000","last_fullsized":"2021-02-10T04:56:12.725864+0000","mapping_epoch":1118,"log_start":"1066'44","ondisk_log_start":"1066'44","created":55,"last_epoch_clean":1181,"parent":"0.0","parent_split_bits":0,"last_scrub":"1066'44","last_scrub_stamp":"2021-02-10T04:53:37.108677+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:53:37.108677+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1578,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":180,"num_read_kb":144,"num_write":106,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":2312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"1162","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.725383+0000","last_change":"2021-02-10T04:53:39.126356+0000","last_active":"2021-02-10T04:56:12.725383+0000","last_peered":"2021-02-10T04:56:12.725383+0000","last_clean":"2021-02-10T04:56:12.725383+0000","last_became_active":"2021-02-10T04:29:07.145616+0000","last_became_peered":"2021-02-10T04:29:07.145616+0000","last_unstale":"2021-02-10T04:56:12.725383+0000","last_undegraded":"2021-02-10T04:56:12.725383+0000","last_fullsized":"2021-02-10T04:56:12.725383+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:39.126261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:39.126261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"494","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.732817+0000","last_change":"2021-02-10T04:55:14.611192+0000","last_active":"2021-02-10T04:56:12.732817+0000","last_peered":"2021-02-10T04:56:12.732817+0000","last_clean":"2021-02-10T04:56:12.732817+0000","last_became_active":"2021-02-10T04:55:14.610821+0000","last_became_peered":"2021-02-10T04:55:14.610821+0000","last_unstale":"2021-02-10T04:56:12.732817+0000","last_undegraded":"2021-02-10T04:56:12.732817+0000","last_fullsized":"2021-02-10T04:56:12.732817+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:26.311958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.311958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"196","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.730668+0000","last_change":"2021-02-10T04:55:03.526061+0000","last_active":"2021-02-10T04:56:12.730668+0000","last_peered":"2021-02-10T04:56:12.730668+0000","last_clean":"2021-02-10T04:56:12.730668+0000","last_became_active":"2021-02-10T04:51:36.855744+0000","last_became_peered":"2021-02-10T04:51:36.855744+0000","last_unstale":"2021-02-10T04:56:12.730668+0000","last_undegraded":"2021-02-10T04:56:12.730668+0000","last_fullsized":"2021-02-10T04:56:12.730668+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:02.162851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:13.727241+0000","last_clean_scrub_stamp":"2021-02-10T04:52:02.162851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"1063'654","reported_seq":"4443","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.730233+0000","last_change":"2021-02-10T04:55:14.637619+0000","last_active":"2021-02-10T04:56:12.730233+0000","last_peered":"2021-02-10T04:56:12.730233+0000","last_clean":"2021-02-10T04:56:12.730233+0000","last_became_active":"2021-02-10T04:55:14.636752+0000","last_became_peered":"2021-02-10T04:55:14.636752+0000","last_unstale":"2021-02-10T04:56:12.730233+0000","last_undegraded":"2021-02-10T04:56:12.730233+0000","last_fullsized":"2021-02-10T04:56:12.730233+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:51:57.121838+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.121838+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24700630,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":171,"num_read_kb":182272,"num_write":111,"num_write_kb":33727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":26349023,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"1213","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.727349+0000","last_change":"2021-02-10T04:51:50.433056+0000","last_active":"2021-02-10T04:56:12.727349+0000","last_peered":"2021-02-10T04:56:12.727349+0000","last_clean":"2021-02-10T04:56:12.727349+0000","last_became_active":"2021-02-10T04:51:36.843165+0000","last_became_peered":"2021-02-10T04:51:36.843165+0000","last_unstale":"2021-02-10T04:56:12.727349+0000","last_undegraded":"2021-02-10T04:56:12.727349+0000","last_fullsized":"2021-02-10T04:56:12.727349+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.432990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.432990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"1234","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781550+0000","last_change":"2021-02-10T04:55:14.638290+0000","last_active":"2021-02-10T04:56:12.781550+0000","last_peered":"2021-02-10T04:56:12.781550+0000","last_clean":"2021-02-10T04:56:12.781550+0000","last_became_active":"2021-02-10T04:55:14.637927+0000","last_became_peered":"2021-02-10T04:55:14.637927+0000","last_unstale":"2021-02-10T04:56:12.781550+0000","last_undegraded":"2021-02-10T04:56:12.781550+0000","last_fullsized":"2021-02-10T04:56:12.781550+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"1159","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781340+0000","last_change":"2021-02-10T04:53:46.727888+0000","last_active":"2021-02-10T04:56:12.781340+0000","last_peered":"2021-02-10T04:56:12.781340+0000","last_clean":"2021-02-10T04:56:12.781340+0000","last_became_active":"2021-02-10T04:53:46.727670+0000","last_became_peered":"2021-02-10T04:53:46.727670+0000","last_unstale":"2021-02-10T04:56:12.781340+0000","last_undegraded":"2021-02-10T04:56:12.781340+0000","last_fullsized":"2021-02-10T04:56:12.781340+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:41.041553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.041553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"1261'76","reported_seq":"1376","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.725335+0000","last_change":"2021-02-10T04:53:42.071173+0000","last_active":"2021-02-10T04:56:12.725335+0000","last_peered":"2021-02-10T04:56:12.725335+0000","last_clean":"2021-02-10T04:56:12.725335+0000","last_became_active":"2021-02-10T04:53:39.679752+0000","last_became_peered":"2021-02-10T04:53:39.679752+0000","last_unstale":"2021-02-10T04:56:12.725335+0000","last_undegraded":"2021-02-10T04:56:12.725335+0000","last_fullsized":"2021-02-10T04:56:12.725335+0000","mapping_epoch":1111,"log_start":"1049'39","ondisk_log_start":"1049'39","created":55,"last_epoch_clean":1185,"parent":"0.0","parent_split_bits":0,"last_scrub":"1187'54","last_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_deep_scrub":"1187'54","last_deep_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_clean_scrub_stamp":"2021-02-10T04:53:42.071103+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":147,"num_read_kb":127,"num_write":95,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"904","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.729125+0000","last_change":"2021-02-10T04:51:52.350764+0000","last_active":"2021-02-10T04:56:12.729125+0000","last_peered":"2021-02-10T04:56:12.729125+0000","last_clean":"2021-02-10T04:56:12.729125+0000","last_became_active":"2021-02-10T04:51:49.567784+0000","last_became_peered":"2021-02-10T04:51:49.567784+0000","last_unstale":"2021-02-10T04:56:12.729125+0000","last_undegraded":"2021-02-10T04:56:12.729125+0000","last_fullsized":"2021-02-10T04:56:12.729125+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:52.350706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.350706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"1119'4","reported_seq":"225","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:13.081549+0000","last_change":"2021-02-10T04:52:27.126670+0000","last_active":"2021-02-10T04:56:13.081549+0000","last_peered":"2021-02-10T04:56:13.081549+0000","last_clean":"2021-02-10T04:56:13.081549+0000","last_became_active":"2021-02-10T04:51:36.853647+0000","last_became_peered":"2021-02-10T04:51:36.853647+0000","last_unstale":"2021-02-10T04:56:13.081549+0000","last_undegraded":"2021-02-10T04:56:13.081549+0000","last_fullsized":"2021-02-10T04:56:13.081549+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'4","last_scrub_stamp":"2021-02-10T04:52:27.126599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:27.126599+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"1253","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.726832+0000","last_change":"2021-02-10T04:52:25.745660+0000","last_active":"2021-02-10T04:56:12.726832+0000","last_peered":"2021-02-10T04:56:12.726832+0000","last_clean":"2021-02-10T04:56:12.726832+0000","last_became_active":"2021-02-10T04:51:36.849503+0000","last_became_peered":"2021-02-10T04:51:36.849503+0000","last_unstale":"2021-02-10T04:56:12.726832+0000","last_undegraded":"2021-02-10T04:56:12.726832+0000","last_fullsized":"2021-02-10T04:56:12.726832+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:25.745577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.745577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"1004'1093","reported_seq":"6448","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.782302+0000","last_change":"2021-02-10T04:55:28.097530+0000","last_active":"2021-02-10T04:56:12.782302+0000","last_peered":"2021-02-10T04:56:12.782302+0000","last_clean":"2021-02-10T04:56:12.782302+0000","last_became_active":"2021-02-10T04:55:14.627630+0000","last_became_peered":"2021-02-10T04:55:14.627630+0000","last_unstale":"2021-02-10T04:56:12.782302+0000","last_undegraded":"2021-02-10T04:56:12.782302+0000","last_fullsized":"2021-02-10T04:56:12.782302+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39434696,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":279,"num_read_kb":311058,"num_write":185,"num_write_kb":66790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"62'5","reported_seq":"66","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.780135+0000","last_change":"2021-02-10T04:54:39.559473+0000","last_active":"2021-02-10T04:56:12.780135+0000","last_peered":"2021-02-10T04:56:12.780135+0000","last_clean":"2021-02-10T04:56:12.780135+0000","last_became_active":"2021-02-10T04:54:39.559048+0000","last_became_peered":"2021-02-10T04:54:39.559048+0000","last_unstale":"2021-02-10T04:56:12.780135+0000","last_undegraded":"2021-02-10T04:56:12.780135+0000","last_fullsized":"2021-02-10T04:56:12.780135+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":149,"num_read_kb":148,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"1222","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.727475+0000","last_change":"2021-02-10T04:55:03.230261+0000","last_active":"2021-02-10T04:56:12.727475+0000","last_peered":"2021-02-10T04:56:12.727475+0000","last_clean":"2021-02-10T04:56:12.727475+0000","last_became_active":"2021-02-10T04:51:36.841582+0000","last_became_peered":"2021-02-10T04:51:36.841582+0000","last_unstale":"2021-02-10T04:56:12.727475+0000","last_undegraded":"2021-02-10T04:56:12.727475+0000","last_fullsized":"2021-02-10T04:56:12.727475+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:12.389272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:12.389272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"1261'71","reported_seq":"970","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.729536+0000","last_change":"2021-02-10T04:53:27.560299+0000","last_active":"2021-02-10T04:56:12.729536+0000","last_peered":"2021-02-10T04:56:12.729536+0000","last_clean":"2021-02-10T04:56:12.729536+0000","last_became_active":"2021-02-10T04:53:27.559886+0000","last_became_peered":"2021-02-10T04:53:27.559886+0000","last_unstale":"2021-02-10T04:56:12.729536+0000","last_undegraded":"2021-02-10T04:56:12.729536+0000","last_fullsized":"2021-02-10T04:56:12.729536+0000","mapping_epoch":1129,"log_start":"1049'51","ondisk_log_start":"1049'51","created":55,"last_epoch_clean":1173,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'41","last_scrub_stamp":"2021-02-10T04:51:58.353660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:58.353660+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1472,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":169,"num_read_kb":136,"num_write":102,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"927","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.728374+0000","last_change":"2021-02-10T04:53:11.271557+0000","last_active":"2021-02-10T04:56:12.728374+0000","last_peered":"2021-02-10T04:56:12.728374+0000","last_clean":"2021-02-10T04:56:12.728374+0000","last_became_active":"2021-02-10T04:51:49.567883+0000","last_became_peered":"2021-02-10T04:51:49.567883+0000","last_unstale":"2021-02-10T04:56:12.728374+0000","last_undegraded":"2021-02-10T04:56:12.728374+0000","last_fullsized":"2021-02-10T04:56:12.728374+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:11.271459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.271459+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"1229'5","reported_seq":"1621","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:13.044216+0000","last_change":"2021-02-10T04:55:14.638050+0000","last_active":"2021-02-10T04:56:13.044216+0000","last_peered":"2021-02-10T04:56:13.044216+0000","last_clean":"2021-02-10T04:56:13.044216+0000","last_became_active":"2021-02-10T04:55:14.637570+0000","last_became_peered":"2021-02-10T04:55:14.637570+0000","last_unstale":"2021-02-10T04:56:13.044216+0000","last_undegraded":"2021-02-10T04:56:13.044216+0000","last_fullsized":"2021-02-10T04:56:13.044216+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:51:47.855283+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.855283+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"1257","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.728775+0000","last_change":"2021-02-10T04:55:22.183630+0000","last_active":"2021-02-10T04:56:12.728775+0000","last_peered":"2021-02-10T04:56:12.728775+0000","last_clean":"2021-02-10T04:56:12.728775+0000","last_became_active":"2021-02-10T04:55:14.656661+0000","last_became_peered":"2021-02-10T04:55:14.656661+0000","last_unstale":"2021-02-10T04:56:12.728775+0000","last_undegraded":"2021-02-10T04:56:12.728775+0000","last_fullsized":"2021-02-10T04:56:12.728775+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:55:22.183566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:55:22.183566+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"1002'1090","reported_seq":"5462","reported_epoch":"1268","state":"active+clean","last_fresh":"2021-02-10T04:56:13.735802+0000","last_change":"2021-02-10T04:52:26.030198+0000","last_active":"2021-02-10T04:56:13.735802+0000","last_peered":"2021-02-10T04:56:13.735802+0000","last_clean":"2021-02-10T04:56:13.735802+0000","last_became_active":"2021-02-10T04:52:26.028182+0000","last_became_peered":"2021-02-10T04:52:26.028182+0000","last_unstale":"2021-02-10T04:56:13.735802+0000","last_undegraded":"2021-02-10T04:56:13.735802+0000","last_fullsized":"2021-02-10T04:56:13.735802+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:51:08.220472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.220472+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24410646,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":245,"num_read_kb":276168,"num_write":175,"num_write_kb":64960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"1233","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.723670+0000","last_change":"2021-02-10T04:54:39.555379+0000","last_active":"2021-02-10T04:56:12.723670+0000","last_peered":"2021-02-10T04:56:12.723670+0000","last_clean":"2021-02-10T04:56:12.723670+0000","last_became_active":"2021-02-10T04:54:39.554976+0000","last_became_peered":"2021-02-10T04:54:39.554976+0000","last_unstale":"2021-02-10T04:56:12.723670+0000","last_undegraded":"2021-02-10T04:56:12.723670+0000","last_fullsized":"2021-02-10T04:56:12.723670+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"1261'65","reported_seq":"1365","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.726548+0000","last_change":"2021-02-10T04:53:31.624862+0000","last_active":"2021-02-10T04:56:12.726548+0000","last_peered":"2021-02-10T04:56:12.726548+0000","last_clean":"2021-02-10T04:56:12.726548+0000","last_became_active":"2021-02-10T04:53:31.624318+0000","last_became_peered":"2021-02-10T04:53:31.624318+0000","last_unstale":"2021-02-10T04:56:12.726548+0000","last_undegraded":"2021-02-10T04:56:12.726548+0000","last_fullsized":"2021-02-10T04:56:12.726548+0000","mapping_epoch":55,"log_start":"1049'47","ondisk_log_start":"1049'47","created":55,"last_epoch_clean":1177,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'47","last_scrub_stamp":"2021-02-10T04:53:09.002368+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.002368+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1672,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":169,"num_read_kb":135,"num_write":98,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":934,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"204","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781177+0000","last_change":"2021-02-10T04:55:14.611324+0000","last_active":"2021-02-10T04:56:12.781177+0000","last_peered":"2021-02-10T04:56:12.781177+0000","last_clean":"2021-02-10T04:56:12.781177+0000","last_became_active":"2021-02-10T04:55:14.610986+0000","last_became_peered":"2021-02-10T04:55:14.610986+0000","last_unstale":"2021-02-10T04:56:12.781177+0000","last_undegraded":"2021-02-10T04:56:12.781177+0000","last_fullsized":"2021-02-10T04:56:12.781177+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:08.358300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:50:08.358300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"1244","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.728968+0000","last_change":"2021-02-10T04:51:57.436961+0000","last_active":"2021-02-10T04:56:12.728968+0000","last_peered":"2021-02-10T04:56:12.728968+0000","last_clean":"2021-02-10T04:56:12.728968+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:56:12.728968+0000","last_undegraded":"2021-02-10T04:56:12.728968+0000","last_fullsized":"2021-02-10T04:56:12.728968+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.436862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.436862+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"1199","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781825+0000","last_change":"2021-02-10T04:53:02.112612+0000","last_active":"2021-02-10T04:56:12.781825+0000","last_peered":"2021-02-10T04:56:12.781825+0000","last_clean":"2021-02-10T04:56:12.781825+0000","last_became_active":"2021-02-10T04:50:59.545501+0000","last_became_peered":"2021-02-10T04:50:59.545501+0000","last_unstale":"2021-02-10T04:56:12.781825+0000","last_undegraded":"2021-02-10T04:56:12.781825+0000","last_fullsized":"2021-02-10T04:56:12.781825+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:02.112533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:02.112533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"1001'1165","reported_seq":"4441","reported_epoch":"1268","state":"active+clean","last_fresh":"2021-02-10T04:56:13.735988+0000","last_change":"2021-02-10T04:55:14.653651+0000","last_active":"2021-02-10T04:56:13.735988+0000","last_peered":"2021-02-10T04:56:13.735988+0000","last_clean":"2021-02-10T04:56:13.735988+0000","last_became_active":"2021-02-10T04:55:14.652235+0000","last_became_peered":"2021-02-10T04:55:14.652235+0000","last_unstale":"2021-02-10T04:56:13.735988+0000","last_undegraded":"2021-02-10T04:56:13.735988+0000","last_fullsized":"2021-02-10T04:56:13.735988+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29601845,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":286,"num_read_kb":299529,"num_write":200,"num_write_kb":89130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"49","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781015+0000","last_change":"2021-02-10T04:55:15.590113+0000","last_active":"2021-02-10T04:56:12.781015+0000","last_peered":"2021-02-10T04:56:12.781015+0000","last_clean":"2021-02-10T04:56:12.781015+0000","last_became_active":"2021-02-10T04:55:15.589751+0000","last_became_peered":"2021-02-10T04:55:15.589751+0000","last_unstale":"2021-02-10T04:56:12.781015+0000","last_undegraded":"2021-02-10T04:56:12.781015+0000","last_fullsized":"2021-02-10T04:56:12.781015+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"1113'5","reported_seq":"1507","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:13.044412+0000","last_change":"2021-02-10T04:52:34.047474+0000","last_active":"2021-02-10T04:56:13.044412+0000","last_peered":"2021-02-10T04:56:13.044412+0000","last_clean":"2021-02-10T04:56:13.044412+0000","last_became_active":"2021-02-10T04:50:59.569578+0000","last_became_peered":"2021-02-10T04:50:59.569578+0000","last_unstale":"2021-02-10T04:56:13.044412+0000","last_undegraded":"2021-02-10T04:56:13.044412+0000","last_fullsized":"2021-02-10T04:56:13.044412+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1113'5","last_scrub_stamp":"2021-02-10T04:52:34.047358+0000","last_deep_scrub":"158'3","last_deep_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_clean_scrub_stamp":"2021-02-10T04:52:34.047358+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"1261'68","reported_seq":"1350","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.724015+0000","last_change":"2021-02-10T04:51:48.872937+0000","last_active":"2021-02-10T04:56:12.724015+0000","last_peered":"2021-02-10T04:56:12.724015+0000","last_clean":"2021-02-10T04:56:12.724015+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:56:12.724015+0000","last_undegraded":"2021-02-10T04:56:12.724015+0000","last_fullsized":"2021-02-10T04:56:12.724015+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:51:48.872839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.872839+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":114,"num_read_kb":94,"num_write":68,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"1319","reported_epoch":"1268","state":"active+clean","last_fresh":"2021-02-10T04:56:13.736814+0000","last_change":"2021-02-10T04:53:05.984747+0000","last_active":"2021-02-10T04:56:13.736814+0000","last_peered":"2021-02-10T04:56:13.736814+0000","last_clean":"2021-02-10T04:56:13.736814+0000","last_became_active":"2021-02-10T04:51:36.869931+0000","last_became_peered":"2021-02-10T04:51:36.869931+0000","last_unstale":"2021-02-10T04:56:13.736814+0000","last_undegraded":"2021-02-10T04:56:13.736814+0000","last_fullsized":"2021-02-10T04:56:13.736814+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:05.984659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:05.984659+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"1121","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781872+0000","last_change":"2021-02-10T04:53:46.731709+0000","last_active":"2021-02-10T04:56:12.781872+0000","last_peered":"2021-02-10T04:56:12.781872+0000","last_clean":"2021-02-10T04:56:12.781872+0000","last_became_active":"2021-02-10T04:53:46.731340+0000","last_became_peered":"2021-02-10T04:53:46.731340+0000","last_unstale":"2021-02-10T04:56:12.781872+0000","last_undegraded":"2021-02-10T04:56:12.781872+0000","last_fullsized":"2021-02-10T04:56:12.781872+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:09.005421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.005421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"998'1009","reported_seq":"4758","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781437+0000","last_change":"2021-02-10T04:52:26.037227+0000","last_active":"2021-02-10T04:56:12.781437+0000","last_peered":"2021-02-10T04:56:12.781437+0000","last_clean":"2021-02-10T04:56:12.781437+0000","last_became_active":"2021-02-10T04:52:26.036723+0000","last_became_peered":"2021-02-10T04:52:26.036723+0000","last_unstale":"2021-02-10T04:56:12.781437+0000","last_undegraded":"2021-02-10T04:56:12.781437+0000","last_fullsized":"2021-02-10T04:56:12.781437+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12355206,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":119,"num_read_kb":124701,"num_write":77,"num_write_kb":25052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4145029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"1138","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.782064+0000","last_change":"2021-02-10T04:53:33.305896+0000","last_active":"2021-02-10T04:56:12.782064+0000","last_peered":"2021-02-10T04:56:12.782064+0000","last_clean":"2021-02-10T04:56:12.782064+0000","last_became_active":"2021-02-10T04:50:59.570854+0000","last_became_peered":"2021-02-10T04:50:59.570854+0000","last_unstale":"2021-02-10T04:56:12.782064+0000","last_undegraded":"2021-02-10T04:56:12.782064+0000","last_fullsized":"2021-02-10T04:56:12.782064+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:33.305827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:33.305827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"1239","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.782481+0000","last_change":"2021-02-10T04:55:15.669988+0000","last_active":"2021-02-10T04:56:12.782481+0000","last_peered":"2021-02-10T04:56:12.782481+0000","last_clean":"2021-02-10T04:56:12.782481+0000","last_became_active":"2021-02-10T04:55:15.669642+0000","last_became_peered":"2021-02-10T04:55:15.669642+0000","last_unstale":"2021-02-10T04:56:12.782481+0000","last_undegraded":"2021-02-10T04:56:12.782481+0000","last_fullsized":"2021-02-10T04:56:12.782481+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"1252","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.724305+0000","last_change":"2021-02-10T04:51:50.895471+0000","last_active":"2021-02-10T04:56:12.724305+0000","last_peered":"2021-02-10T04:56:12.724305+0000","last_clean":"2021-02-10T04:56:12.724305+0000","last_became_active":"2021-02-10T04:50:59.527553+0000","last_became_peered":"2021-02-10T04:50:59.527553+0000","last_unstale":"2021-02-10T04:56:12.724305+0000","last_undegraded":"2021-02-10T04:56:12.724305+0000","last_fullsized":"2021-02-10T04:56:12.724305+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.895370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.895370+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"1261'84","reported_seq":"1383","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781185+0000","last_change":"2021-02-10T04:55:14.603109+0000","last_active":"2021-02-10T04:56:12.781185+0000","last_peered":"2021-02-10T04:56:12.781185+0000","last_clean":"2021-02-10T04:56:12.781185+0000","last_became_active":"2021-02-10T04:55:14.602669+0000","last_became_peered":"2021-02-10T04:55:14.602669+0000","last_unstale":"2021-02-10T04:56:12.781185+0000","last_undegraded":"2021-02-10T04:56:12.781185+0000","last_fullsized":"2021-02-10T04:56:12.781185+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'72","last_scrub_stamp":"2021-02-10T04:46:30.515026+0000","last_deep_scrub":"152'48","last_deep_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_clean_scrub_stamp":"2021-02-10T04:46:30.515026+0000","log_size":84,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":123,"num_read_kb":101,"num_write":72,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"521","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.728914+0000","last_change":"2021-02-10T04:55:03.834050+0000","last_active":"2021-02-10T04:56:12.728914+0000","last_peered":"2021-02-10T04:56:12.728914+0000","last_clean":"2021-02-10T04:56:12.728914+0000","last_became_active":"2021-02-10T04:51:49.503801+0000","last_became_peered":"2021-02-10T04:51:49.503801+0000","last_unstale":"2021-02-10T04:56:12.728914+0000","last_undegraded":"2021-02-10T04:56:12.728914+0000","last_fullsized":"2021-02-10T04:56:12.728914+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:51:48.274960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.274960+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"1277","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.724419+0000","last_change":"2021-02-10T04:53:13.099288+0000","last_active":"2021-02-10T04:56:12.724419+0000","last_peered":"2021-02-10T04:56:12.724419+0000","last_clean":"2021-02-10T04:56:12.724419+0000","last_became_active":"2021-02-10T04:51:36.848614+0000","last_became_peered":"2021-02-10T04:51:36.848614+0000","last_unstale":"2021-02-10T04:56:12.724419+0000","last_undegraded":"2021-02-10T04:56:12.724419+0000","last_fullsized":"2021-02-10T04:56:12.724419+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:13.099228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:13.099228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"1002'1657","reported_seq":"5328","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781060+0000","last_change":"2021-02-10T04:52:26.036806+0000","last_active":"2021-02-10T04:56:12.781060+0000","last_peered":"2021-02-10T04:56:12.781060+0000","last_clean":"2021-02-10T04:56:12.781060+0000","last_became_active":"2021-02-10T04:52:26.032363+0000","last_became_peered":"2021-02-10T04:52:26.032363+0000","last_unstale":"2021-02-10T04:56:12.781060+0000","last_undegraded":"2021-02-10T04:56:12.781060+0000","last_fullsized":"2021-02-10T04:56:12.781060+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21396533,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":212,"num_read_kb":218836,"num_write":159,"num_write_kb":59398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":22326014,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"1196","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.725244+0000","last_change":"2021-02-10T04:51:51.911945+0000","last_active":"2021-02-10T04:56:12.725244+0000","last_peered":"2021-02-10T04:56:12.725244+0000","last_clean":"2021-02-10T04:56:12.725244+0000","last_became_active":"2021-02-10T04:51:36.850015+0000","last_became_peered":"2021-02-10T04:51:36.850015+0000","last_unstale":"2021-02-10T04:56:12.725244+0000","last_undegraded":"2021-02-10T04:56:12.725244+0000","last_fullsized":"2021-02-10T04:56:12.725244+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.911851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.911851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"62'5","reported_seq":"1100","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.728376+0000","last_change":"2021-02-10T04:54:39.568253+0000","last_active":"2021-02-10T04:56:12.728376+0000","last_peered":"2021-02-10T04:56:12.728376+0000","last_clean":"2021-02-10T04:56:12.728376+0000","last_became_active":"2021-02-10T04:54:39.567642+0000","last_became_peered":"2021-02-10T04:54:39.567642+0000","last_unstale":"2021-02-10T04:56:12.728376+0000","last_undegraded":"2021-02-10T04:56:12.728376+0000","last_fullsized":"2021-02-10T04:56:12.728376+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":98,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":150,"num_read_kb":148,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"1326","reported_epoch":"1268","state":"active+clean","last_fresh":"2021-02-10T04:56:13.734450+0000","last_change":"2021-02-10T04:55:14.634577+0000","last_active":"2021-02-10T04:56:13.734450+0000","last_peered":"2021-02-10T04:56:13.734450+0000","last_clean":"2021-02-10T04:56:13.734450+0000","last_became_active":"2021-02-10T04:55:14.634259+0000","last_became_peered":"2021-02-10T04:55:14.634259+0000","last_unstale":"2021-02-10T04:56:13.734450+0000","last_undegraded":"2021-02-10T04:56:13.734450+0000","last_fullsized":"2021-02-10T04:56:13.734450+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:58.142526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:58.142526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"1261'68","reported_seq":"1276","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.780614+0000","last_change":"2021-02-10T04:55:28.079376+0000","last_active":"2021-02-10T04:56:12.780614+0000","last_peered":"2021-02-10T04:56:12.780614+0000","last_clean":"2021-02-10T04:56:12.780614+0000","last_became_active":"2021-02-10T04:55:14.654609+0000","last_became_peered":"2021-02-10T04:55:14.654609+0000","last_unstale":"2021-02-10T04:56:12.780614+0000","last_undegraded":"2021-02-10T04:56:12.780614+0000","last_fullsized":"2021-02-10T04:56:12.780614+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1135'58","last_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_deep_scrub":"1135'58","last_deep_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.483067+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":102,"num_read_kb":83,"num_write":60,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"202","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.782741+0000","last_change":"2021-02-10T04:55:14.657913+0000","last_active":"2021-02-10T04:56:12.782741+0000","last_peered":"2021-02-10T04:56:12.782741+0000","last_clean":"2021-02-10T04:56:12.782741+0000","last_became_active":"2021-02-10T04:55:14.657153+0000","last_became_peered":"2021-02-10T04:55:14.657153+0000","last_unstale":"2021-02-10T04:56:12.782741+0000","last_undegraded":"2021-02-10T04:56:12.782741+0000","last_fullsized":"2021-02-10T04:56:12.782741+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:50:46.434629+0000","last_deep_scrub":"53'2","last_deep_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_clean_scrub_stamp":"2021-02-10T04:50:46.434629+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"944","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.728517+0000","last_change":"2021-02-10T04:52:38.278980+0000","last_active":"2021-02-10T04:56:12.728517+0000","last_peered":"2021-02-10T04:56:12.728517+0000","last_clean":"2021-02-10T04:56:12.728517+0000","last_became_active":"2021-02-10T04:51:49.496237+0000","last_became_peered":"2021-02-10T04:51:49.496237+0000","last_unstale":"2021-02-10T04:56:12.728517+0000","last_undegraded":"2021-02-10T04:56:12.728517+0000","last_fullsized":"2021-02-10T04:56:12.728517+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:38.278910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:52:38.278910+0000","last_clean_scrub_stamp":"2021-02-10T04:52:38.278910+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"1000'1071","reported_seq":"4818","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.727622+0000","last_change":"2021-02-10T04:53:34.407875+0000","last_active":"2021-02-10T04:56:12.727622+0000","last_peered":"2021-02-10T04:56:12.727622+0000","last_clean":"2021-02-10T04:56:12.727622+0000","last_became_active":"2021-02-10T04:52:26.031050+0000","last_became_peered":"2021-02-10T04:52:26.031050+0000","last_unstale":"2021-02-10T04:56:12.727622+0000","last_undegraded":"2021-02-10T04:56:12.727622+0000","last_fullsized":"2021-02-10T04:56:12.727622+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:34.407784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.407784+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30173843,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"1292","reported_epoch":"1268","state":"active+clean","last_fresh":"2021-02-10T04:56:13.736485+0000","last_change":"2021-02-10T04:55:14.637570+0000","last_active":"2021-02-10T04:56:13.736485+0000","last_peered":"2021-02-10T04:56:13.736485+0000","last_clean":"2021-02-10T04:56:13.736485+0000","last_became_active":"2021-02-10T04:55:14.637203+0000","last_became_peered":"2021-02-10T04:55:14.637203+0000","last_unstale":"2021-02-10T04:56:13.736485+0000","last_undegraded":"2021-02-10T04:56:13.736485+0000","last_fullsized":"2021-02-10T04:56:13.736485+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_clean_scrub_stamp":"2021-02-10T04:51:00.195518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"1254","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.729005+0000","last_change":"2021-02-10T04:55:14.661285+0000","last_active":"2021-02-10T04:56:12.729005+0000","last_peered":"2021-02-10T04:56:12.729005+0000","last_clean":"2021-02-10T04:56:12.729005+0000","last_became_active":"2021-02-10T04:55:14.660902+0000","last_became_peered":"2021-02-10T04:55:14.660902+0000","last_unstale":"2021-02-10T04:56:12.729005+0000","last_undegraded":"2021-02-10T04:56:12.729005+0000","last_fullsized":"2021-02-10T04:56:12.729005+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:55.473347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:50:55.473347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"1261'58","reported_seq":"1327","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781853+0000","last_change":"2021-02-10T04:55:47.652017+0000","last_active":"2021-02-10T04:56:12.781853+0000","last_peered":"2021-02-10T04:56:12.781853+0000","last_clean":"2021-02-10T04:56:12.781853+0000","last_became_active":"2021-02-10T04:55:47.651592+0000","last_became_peered":"2021-02-10T04:55:47.651592+0000","last_unstale":"2021-02-10T04:56:12.781853+0000","last_undegraded":"2021-02-10T04:56:12.781853+0000","last_fullsized":"2021-02-10T04:56:12.781853+0000","mapping_epoch":1227,"log_start":"1049'40","ondisk_log_start":"1049'40","created":55,"last_epoch_clean":1257,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'40","last_scrub_stamp":"2021-02-10T04:51:47.281929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.281929+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1836,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":171,"num_read_kb":135,"num_write":99,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"428","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781221+0000","last_change":"2021-02-10T04:55:14.610765+0000","last_active":"2021-02-10T04:56:12.781221+0000","last_peered":"2021-02-10T04:56:12.781221+0000","last_clean":"2021-02-10T04:56:12.781221+0000","last_became_active":"2021-02-10T04:55:14.610470+0000","last_became_peered":"2021-02-10T04:55:14.610470+0000","last_unstale":"2021-02-10T04:56:12.781221+0000","last_undegraded":"2021-02-10T04:56:12.781221+0000","last_fullsized":"2021-02-10T04:56:12.781221+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:50.399350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:50.399350+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"1154","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.780009+0000","last_change":"2021-02-10T04:55:25.624031+0000","last_active":"2021-02-10T04:56:12.780009+0000","last_peered":"2021-02-10T04:56:12.780009+0000","last_clean":"2021-02-10T04:56:12.780009+0000","last_became_active":"2021-02-10T04:55:14.630129+0000","last_became_peered":"2021-02-10T04:55:14.630129+0000","last_unstale":"2021-02-10T04:56:12.780009+0000","last_undegraded":"2021-02-10T04:56:12.780009+0000","last_fullsized":"2021-02-10T04:56:12.780009+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"929","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.732897+0000","last_change":"2021-02-10T04:52:31.273037+0000","last_active":"2021-02-10T04:56:12.732897+0000","last_peered":"2021-02-10T04:56:12.732897+0000","last_clean":"2021-02-10T04:56:12.732897+0000","last_became_active":"2021-02-10T04:51:49.501405+0000","last_became_peered":"2021-02-10T04:51:49.501405+0000","last_unstale":"2021-02-10T04:56:12.732897+0000","last_undegraded":"2021-02-10T04:56:12.732897+0000","last_fullsized":"2021-02-10T04:56:12.732897+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:31.272971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:17.598113+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.272971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"1000'446","reported_seq":"1985","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.782467+0000","last_change":"2021-02-10T04:55:14.658783+0000","last_active":"2021-02-10T04:56:12.782467+0000","last_peered":"2021-02-10T04:56:12.782467+0000","last_clean":"2021-02-10T04:56:12.782467+0000","last_became_active":"2021-02-10T04:55:14.657668+0000","last_became_peered":"2021-02-10T04:55:14.657668+0000","last_unstale":"2021-02-10T04:56:12.782467+0000","last_undegraded":"2021-02-10T04:56:12.782467+0000","last_fullsized":"2021-02-10T04:56:12.782467+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:51:45.964029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.964029+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22255852,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68435,"num_write":72,"num_write_kb":24345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":403693,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"420","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.728366+0000","last_change":"2021-02-10T04:53:03.228260+0000","last_active":"2021-02-10T04:56:12.728366+0000","last_peered":"2021-02-10T04:56:12.728366+0000","last_clean":"2021-02-10T04:56:12.728366+0000","last_became_active":"2021-02-10T04:51:49.499252+0000","last_became_peered":"2021-02-10T04:51:49.499252+0000","last_unstale":"2021-02-10T04:56:12.728366+0000","last_undegraded":"2021-02-10T04:56:12.728366+0000","last_fullsized":"2021-02-10T04:56:12.728366+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:03.228182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.228182+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"172","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.782601+0000","last_change":"2021-02-10T04:53:46.727384+0000","last_active":"2021-02-10T04:56:12.782601+0000","last_peered":"2021-02-10T04:56:12.782601+0000","last_clean":"2021-02-10T04:56:12.782601+0000","last_became_active":"2021-02-10T04:53:46.726771+0000","last_became_peered":"2021-02-10T04:53:46.726771+0000","last_unstale":"2021-02-10T04:56:12.782601+0000","last_undegraded":"2021-02-10T04:56:12.782601+0000","last_fullsized":"2021-02-10T04:56:12.782601+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:33.948348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:33.948348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"1261'51","reported_seq":"1354","reported_epoch":"1268","state":"active+clean","last_fresh":"2021-02-10T04:56:13.735800+0000","last_change":"2021-02-10T04:54:17.118810+0000","last_active":"2021-02-10T04:56:13.735800+0000","last_peered":"2021-02-10T04:56:13.735800+0000","last_clean":"2021-02-10T04:56:13.735800+0000","last_became_active":"2021-02-10T04:54:17.080053+0000","last_became_peered":"2021-02-10T04:54:17.080053+0000","last_unstale":"2021-02-10T04:56:13.735800+0000","last_undegraded":"2021-02-10T04:56:13.735800+0000","last_fullsized":"2021-02-10T04:56:13.735800+0000","mapping_epoch":1200,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1201,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'39","last_scrub_stamp":"2021-02-10T04:51:47.322331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.322331+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":51,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"200","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781256+0000","last_change":"2021-02-10T04:55:14.636888+0000","last_active":"2021-02-10T04:56:12.781256+0000","last_peered":"2021-02-10T04:56:12.781256+0000","last_clean":"2021-02-10T04:56:12.781256+0000","last_became_active":"2021-02-10T04:55:14.634305+0000","last_became_peered":"2021-02-10T04:55:14.634305+0000","last_unstale":"2021-02-10T04:56:12.781256+0000","last_undegraded":"2021-02-10T04:56:12.781256+0000","last_fullsized":"2021-02-10T04:56:12.781256+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_clean_scrub_stamp":"2021-02-10T04:50:05.424262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"1438","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.728036+0000","last_change":"2021-02-10T04:52:37.324122+0000","last_active":"2021-02-10T04:56:12.728036+0000","last_peered":"2021-02-10T04:56:12.728036+0000","last_clean":"2021-02-10T04:56:12.728036+0000","last_became_active":"2021-02-10T04:51:36.843553+0000","last_became_peered":"2021-02-10T04:51:36.843553+0000","last_unstale":"2021-02-10T04:56:12.728036+0000","last_undegraded":"2021-02-10T04:56:12.728036+0000","last_fullsized":"2021-02-10T04:56:12.728036+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:37.324040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:37.324040+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"1244","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781512+0000","last_change":"2021-02-10T04:54:37.612303+0000","last_active":"2021-02-10T04:56:12.781512+0000","last_peered":"2021-02-10T04:56:12.781512+0000","last_clean":"2021-02-10T04:56:12.781512+0000","last_became_active":"2021-02-10T04:54:37.611953+0000","last_became_peered":"2021-02-10T04:54:37.611953+0000","last_unstale":"2021-02-10T04:56:12.781512+0000","last_undegraded":"2021-02-10T04:56:12.781512+0000","last_fullsized":"2021-02-10T04:56:12.781512+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"1266","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.729012+0000","last_change":"2021-02-10T04:52:31.403904+0000","last_active":"2021-02-10T04:56:12.729012+0000","last_peered":"2021-02-10T04:56:12.729012+0000","last_clean":"2021-02-10T04:56:12.729012+0000","last_became_active":"2021-02-10T04:50:59.538009+0000","last_became_peered":"2021-02-10T04:50:59.538009+0000","last_unstale":"2021-02-10T04:56:12.729012+0000","last_undegraded":"2021-02-10T04:56:12.729012+0000","last_fullsized":"2021-02-10T04:56:12.729012+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:31.403824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.403824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"1000'490","reported_seq":"3734","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781964+0000","last_change":"2021-02-10T04:55:22.716472+0000","last_active":"2021-02-10T04:56:12.781964+0000","last_peered":"2021-02-10T04:56:12.781964+0000","last_clean":"2021-02-10T04:56:12.781964+0000","last_became_active":"2021-02-10T04:55:14.644596+0000","last_became_peered":"2021-02-10T04:55:14.644596+0000","last_unstale":"2021-02-10T04:56:12.781964+0000","last_undegraded":"2021-02-10T04:56:12.781964+0000","last_fullsized":"2021-02-10T04:56:12.781964+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:47:02.875920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:02.875920+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19371627,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":141,"num_read_kb":148568,"num_write":85,"num_write_kb":31732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"164","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.730482+0000","last_change":"2021-02-10T04:53:01.972991+0000","last_active":"2021-02-10T04:56:12.730482+0000","last_peered":"2021-02-10T04:56:12.730482+0000","last_clean":"2021-02-10T04:56:12.730482+0000","last_became_active":"2021-02-10T04:51:36.863976+0000","last_became_peered":"2021-02-10T04:51:36.863976+0000","last_unstale":"2021-02-10T04:56:12.730482+0000","last_undegraded":"2021-02-10T04:56:12.730482+0000","last_fullsized":"2021-02-10T04:56:12.730482+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:01.972913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.972913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"1261'40","reported_seq":"986","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.729449+0000","last_change":"2021-02-10T04:55:43.569281+0000","last_active":"2021-02-10T04:56:12.729449+0000","last_peered":"2021-02-10T04:56:12.729449+0000","last_clean":"2021-02-10T04:56:12.729449+0000","last_became_active":"2021-02-10T04:55:43.568878+0000","last_became_peered":"2021-02-10T04:55:43.568878+0000","last_unstale":"2021-02-10T04:56:12.729449+0000","last_undegraded":"2021-02-10T04:56:12.729449+0000","last_fullsized":"2021-02-10T04:56:12.729449+0000","mapping_epoch":1227,"log_start":"1063'30","ondisk_log_start":"1063'30","created":55,"last_epoch_clean":1253,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:56.431187+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.431187+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":802,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":116,"num_read_kb":94,"num_write":67,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"174","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.728912+0000","last_change":"2021-02-10T04:55:14.636849+0000","last_active":"2021-02-10T04:56:12.728912+0000","last_peered":"2021-02-10T04:56:12.728912+0000","last_clean":"2021-02-10T04:56:12.728912+0000","last_became_active":"2021-02-10T04:55:14.636237+0000","last_became_peered":"2021-02-10T04:55:14.636237+0000","last_unstale":"2021-02-10T04:56:12.728912+0000","last_undegraded":"2021-02-10T04:56:12.728912+0000","last_fullsized":"2021-02-10T04:56:12.728912+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:04.591695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:04.591695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"1093","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.730142+0000","last_change":"2021-02-10T04:54:37.680913+0000","last_active":"2021-02-10T04:56:12.730142+0000","last_peered":"2021-02-10T04:56:12.730142+0000","last_clean":"2021-02-10T04:56:12.730142+0000","last_became_active":"2021-02-10T04:54:37.680651+0000","last_became_peered":"2021-02-10T04:54:37.680651+0000","last_unstale":"2021-02-10T04:56:12.730142+0000","last_undegraded":"2021-02-10T04:56:12.730142+0000","last_fullsized":"2021-02-10T04:56:12.730142+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1210,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":149,"num_read_kb":148,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"1111","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.782705+0000","last_change":"2021-02-10T04:53:46.729138+0000","last_active":"2021-02-10T04:56:12.782705+0000","last_peered":"2021-02-10T04:56:12.782705+0000","last_clean":"2021-02-10T04:56:12.782705+0000","last_became_active":"2021-02-10T04:53:46.728845+0000","last_became_peered":"2021-02-10T04:53:46.728845+0000","last_unstale":"2021-02-10T04:56:12.782705+0000","last_undegraded":"2021-02-10T04:56:12.782705+0000","last_fullsized":"2021-02-10T04:56:12.782705+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:40.066354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:52:40.066354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"1000'620","reported_seq":"4358","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.780622+0000","last_change":"2021-02-10T04:55:28.089086+0000","last_active":"2021-02-10T04:56:12.780622+0000","last_peered":"2021-02-10T04:56:12.780622+0000","last_clean":"2021-02-10T04:56:12.780622+0000","last_became_active":"2021-02-10T04:55:14.640509+0000","last_became_peered":"2021-02-10T04:55:14.640509+0000","last_unstale":"2021-02-10T04:56:12.780622+0000","last_undegraded":"2021-02-10T04:56:12.780622+0000","last_fullsized":"2021-02-10T04:56:12.780622+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:51:54.444287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:54.444287+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14265195,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":144,"num_read_kb":128482,"num_write":129,"num_write_kb":53443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"1147","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.780396+0000","last_change":"2021-02-10T04:55:28.083286+0000","last_active":"2021-02-10T04:56:12.780396+0000","last_peered":"2021-02-10T04:56:12.780396+0000","last_clean":"2021-02-10T04:56:12.780396+0000","last_became_active":"2021-02-10T04:55:14.619846+0000","last_became_peered":"2021-02-10T04:55:14.619846+0000","last_unstale":"2021-02-10T04:56:12.780396+0000","last_undegraded":"2021-02-10T04:56:12.780396+0000","last_fullsized":"2021-02-10T04:56:12.780396+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:14.955544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:47:14.955544+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"1244","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.725222+0000","last_change":"2021-02-10T04:55:15.590025+0000","last_active":"2021-02-10T04:56:12.725222+0000","last_peered":"2021-02-10T04:56:12.725222+0000","last_clean":"2021-02-10T04:56:12.725222+0000","last_became_active":"2021-02-10T04:55:15.589651+0000","last_became_peered":"2021-02-10T04:55:15.589651+0000","last_unstale":"2021-02-10T04:56:12.725222+0000","last_undegraded":"2021-02-10T04:56:12.725222+0000","last_fullsized":"2021-02-10T04:56:12.725222+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"1250","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.780491+0000","last_change":"2021-02-10T04:55:28.090104+0000","last_active":"2021-02-10T04:56:12.780491+0000","last_peered":"2021-02-10T04:56:12.780491+0000","last_clean":"2021-02-10T04:56:12.780491+0000","last_became_active":"2021-02-10T04:55:14.650106+0000","last_became_peered":"2021-02-10T04:55:14.650106+0000","last_unstale":"2021-02-10T04:56:12.780491+0000","last_undegraded":"2021-02-10T04:56:12.780491+0000","last_fullsized":"2021-02-10T04:56:12.780491+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:27.908221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:47:27.908221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"1261'67","reported_seq":"918","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.732613+0000","last_change":"2021-02-10T04:52:30.277009+0000","last_active":"2021-02-10T04:56:12.732613+0000","last_peered":"2021-02-10T04:56:12.732613+0000","last_clean":"2021-02-10T04:56:12.732613+0000","last_became_active":"2021-02-10T04:51:49.501622+0000","last_became_peered":"2021-02-10T04:51:49.501622+0000","last_unstale":"2021-02-10T04:56:12.732613+0000","last_undegraded":"2021-02-10T04:56:12.732613+0000","last_fullsized":"2021-02-10T04:56:12.732613+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'55","last_scrub_stamp":"2021-02-10T04:52:30.276938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.276938+0000","log_size":67,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":116,"num_read_kb":92,"num_write":70,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"1262","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.728157+0000","last_change":"2021-02-10T04:51:59.420149+0000","last_active":"2021-02-10T04:56:12.728157+0000","last_peered":"2021-02-10T04:56:12.728157+0000","last_clean":"2021-02-10T04:56:12.728157+0000","last_became_active":"2021-02-10T04:28:21.125541+0000","last_became_peered":"2021-02-10T04:28:21.125541+0000","last_unstale":"2021-02-10T04:56:12.728157+0000","last_undegraded":"2021-02-10T04:56:12.728157+0000","last_fullsized":"2021-02-10T04:56:12.728157+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.420086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.420086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"1228","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.782259+0000","last_change":"2021-02-10T04:53:12.187225+0000","last_active":"2021-02-10T04:56:12.782259+0000","last_peered":"2021-02-10T04:56:12.782259+0000","last_clean":"2021-02-10T04:56:12.782259+0000","last_became_active":"2021-02-10T04:50:59.555689+0000","last_became_peered":"2021-02-10T04:50:59.555689+0000","last_unstale":"2021-02-10T04:56:12.782259+0000","last_undegraded":"2021-02-10T04:56:12.782259+0000","last_fullsized":"2021-02-10T04:56:12.782259+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.187154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.187154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"1002'2042","reported_seq":"9500","reported_epoch":"1268","state":"active+clean","last_fresh":"2021-02-10T04:56:13.736741+0000","last_change":"2021-02-10T04:55:14.662208+0000","last_active":"2021-02-10T04:56:13.736741+0000","last_peered":"2021-02-10T04:56:13.736741+0000","last_clean":"2021-02-10T04:56:13.736741+0000","last_became_active":"2021-02-10T04:55:14.659601+0000","last_became_peered":"2021-02-10T04:55:14.659601+0000","last_unstale":"2021-02-10T04:56:13.736741+0000","last_undegraded":"2021-02-10T04:56:13.736741+0000","last_fullsized":"2021-02-10T04:56:13.736741+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:46:57.054230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:57.054230+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":85678898,"num_objects":35,"num_object_clones":29,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"1213","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.727613+0000","last_change":"2021-02-10T04:55:14.651301+0000","last_active":"2021-02-10T04:56:12.727613+0000","last_peered":"2021-02-10T04:56:12.727613+0000","last_clean":"2021-02-10T04:56:12.727613+0000","last_became_active":"2021-02-10T04:55:14.650386+0000","last_became_peered":"2021-02-10T04:55:14.650386+0000","last_unstale":"2021-02-10T04:56:12.727613+0000","last_undegraded":"2021-02-10T04:56:12.727613+0000","last_fullsized":"2021-02-10T04:56:12.727613+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:52.341527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:50:52.341527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"66","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.728611+0000","last_change":"2021-02-10T04:54:39.558380+0000","last_active":"2021-02-10T04:56:12.728611+0000","last_peered":"2021-02-10T04:56:12.728611+0000","last_clean":"2021-02-10T04:56:12.728611+0000","last_became_active":"2021-02-10T04:54:39.557995+0000","last_became_peered":"2021-02-10T04:54:39.557995+0000","last_unstale":"2021-02-10T04:56:12.728611+0000","last_undegraded":"2021-02-10T04:56:12.728611+0000","last_fullsized":"2021-02-10T04:56:12.728611+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"1238'4","reported_seq":"1452","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:13.044262+0000","last_change":"2021-02-10T04:55:28.095426+0000","last_active":"2021-02-10T04:56:13.044262+0000","last_peered":"2021-02-10T04:56:13.044262+0000","last_clean":"2021-02-10T04:56:13.044262+0000","last_became_active":"2021-02-10T04:55:14.645890+0000","last_became_peered":"2021-02-10T04:55:14.645890+0000","last_unstale":"2021-02-10T04:56:13.044262+0000","last_undegraded":"2021-02-10T04:56:13.044262+0000","last_fullsized":"2021-02-10T04:56:13.044262+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:54.466259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:54.466259+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"1261'64","reported_seq":"964","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.729347+0000","last_change":"2021-02-10T04:53:01.304693+0000","last_active":"2021-02-10T04:56:12.729347+0000","last_peered":"2021-02-10T04:56:12.729347+0000","last_clean":"2021-02-10T04:56:12.729347+0000","last_became_active":"2021-02-10T04:51:49.498066+0000","last_became_peered":"2021-02-10T04:51:49.498066+0000","last_unstale":"2021-02-10T04:56:12.729347+0000","last_undegraded":"2021-02-10T04:56:12.729347+0000","last_fullsized":"2021-02-10T04:56:12.729347+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:53:01.304626+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.304626+0000","log_size":64,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":89,"num_read_kb":71,"num_write":52,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"525","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.732698+0000","last_change":"2021-02-10T04:55:14.610552+0000","last_active":"2021-02-10T04:56:12.732698+0000","last_peered":"2021-02-10T04:56:12.732698+0000","last_clean":"2021-02-10T04:56:12.732698+0000","last_became_active":"2021-02-10T04:55:14.609458+0000","last_became_peered":"2021-02-10T04:55:14.609458+0000","last_unstale":"2021-02-10T04:56:12.732698+0000","last_undegraded":"2021-02-10T04:56:12.732698+0000","last_fullsized":"2021-02-10T04:56:12.732698+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:49.447961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:49.447961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"1204","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.779928+0000","last_change":"2021-02-10T04:56:05.092900+0000","last_active":"2021-02-10T04:56:12.779928+0000","last_peered":"2021-02-10T04:56:12.779928+0000","last_clean":"2021-02-10T04:56:12.779928+0000","last_became_active":"2021-02-10T04:55:14.614581+0000","last_became_peered":"2021-02-10T04:55:14.614581+0000","last_unstale":"2021-02-10T04:56:12.779928+0000","last_undegraded":"2021-02-10T04:56:12.779928+0000","last_fullsized":"2021-02-10T04:56:12.779928+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:07.914061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:07.914061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"293'77","reported_seq":"3049","reported_epoch":"1267","state":"active+clean+scrubbing","last_fresh":"2021-02-10T04:56:12.723775+0000","last_change":"2021-02-10T04:53:43.102301+0000","last_active":"2021-02-10T04:56:12.723775+0000","last_peered":"2021-02-10T04:56:12.723775+0000","last_clean":"2021-02-10T04:56:12.723775+0000","last_became_active":"2021-02-10T04:52:26.024812+0000","last_became_peered":"2021-02-10T04:52:26.024812+0000","last_unstale":"2021-02-10T04:56:12.723775+0000","last_undegraded":"2021-02-10T04:56:12.723775+0000","last_fullsized":"2021-02-10T04:56:12.723775+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1886585,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":18587,"num_write":9,"num_write_kb":3315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"1282","reported_epoch":"1268","state":"active+clean","last_fresh":"2021-02-10T04:56:13.736650+0000","last_change":"2021-02-10T04:53:07.025068+0000","last_active":"2021-02-10T04:56:13.736650+0000","last_peered":"2021-02-10T04:56:13.736650+0000","last_clean":"2021-02-10T04:56:13.736650+0000","last_became_active":"2021-02-10T04:50:59.589417+0000","last_became_peered":"2021-02-10T04:50:59.589417+0000","last_unstale":"2021-02-10T04:56:13.736650+0000","last_undegraded":"2021-02-10T04:56:13.736650+0000","last_fullsized":"2021-02-10T04:56:13.736650+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.024980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.024980+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"200","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.727717+0000","last_change":"2021-02-10T04:56:04.562387+0000","last_active":"2021-02-10T04:56:12.727717+0000","last_peered":"2021-02-10T04:56:12.727717+0000","last_clean":"2021-02-10T04:56:12.727717+0000","last_became_active":"2021-02-10T04:55:14.605267+0000","last_became_peered":"2021-02-10T04:55:14.605267+0000","last_unstale":"2021-02-10T04:56:12.727717+0000","last_undegraded":"2021-02-10T04:56:12.727717+0000","last_fullsized":"2021-02-10T04:56:12.727717+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.609500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.609500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"1261'86","reported_seq":"1448","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.724105+0000","last_change":"2021-02-10T04:55:35.577794+0000","last_active":"2021-02-10T04:56:12.724105+0000","last_peered":"2021-02-10T04:56:12.724105+0000","last_clean":"2021-02-10T04:56:12.724105+0000","last_became_active":"2021-02-10T04:55:35.577316+0000","last_became_peered":"2021-02-10T04:55:35.577316+0000","last_unstale":"2021-02-10T04:56:12.724105+0000","last_undegraded":"2021-02-10T04:56:12.724105+0000","last_fullsized":"2021-02-10T04:56:12.724105+0000","mapping_epoch":1238,"log_start":"1049'68","ondisk_log_start":"1049'68","created":55,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'39","last_scrub_stamp":"2021-02-10T04:53:38.085565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.085565+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1582,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":227,"num_read_kb":187,"num_write":128,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"1381","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.726150+0000","last_change":"2021-02-10T04:55:14.659217+0000","last_active":"2021-02-10T04:56:12.726150+0000","last_peered":"2021-02-10T04:56:12.726150+0000","last_clean":"2021-02-10T04:56:12.726150+0000","last_became_active":"2021-02-10T04:55:14.658879+0000","last_became_peered":"2021-02-10T04:55:14.658879+0000","last_unstale":"2021-02-10T04:56:12.726150+0000","last_undegraded":"2021-02-10T04:56:12.726150+0000","last_fullsized":"2021-02-10T04:56:12.726150+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:47:08.784593+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:47:08.784593+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"1230","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781481+0000","last_change":"2021-02-10T04:52:30.019225+0000","last_active":"2021-02-10T04:56:12.781481+0000","last_peered":"2021-02-10T04:56:12.781481+0000","last_clean":"2021-02-10T04:56:12.781481+0000","last_became_active":"2021-02-10T04:50:59.561436+0000","last_became_peered":"2021-02-10T04:50:59.561436+0000","last_unstale":"2021-02-10T04:56:12.781481+0000","last_undegraded":"2021-02-10T04:56:12.781481+0000","last_fullsized":"2021-02-10T04:56:12.781481+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.019102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.019102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"1001'1528","reported_seq":"6215","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781358+0000","last_change":"2021-02-10T04:52:26.036366+0000","last_active":"2021-02-10T04:56:12.781358+0000","last_peered":"2021-02-10T04:56:12.781358+0000","last_clean":"2021-02-10T04:56:12.781358+0000","last_became_active":"2021-02-10T04:52:26.034028+0000","last_became_peered":"2021-02-10T04:52:26.034028+0000","last_unstale":"2021-02-10T04:56:12.781358+0000","last_undegraded":"2021-02-10T04:56:12.781358+0000","last_fullsized":"2021-02-10T04:56:12.781358+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46944136,"num_objects":16,"num_object_clones":15,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":358,"num_read_kb":351934,"num_write":286,"num_write_kb":111355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":58968832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.15","version":"1000'446","reported_seq":"2204","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.725832+0000","last_change":"2021-02-10T04:53:11.051124+0000","last_active":"2021-02-10T04:56:12.725832+0000","last_peered":"2021-02-10T04:56:12.725832+0000","last_clean":"2021-02-10T04:56:12.725832+0000","last_became_active":"2021-02-10T04:51:40.510899+0000","last_became_peered":"2021-02-10T04:51:40.510899+0000","last_unstale":"2021-02-10T04:56:12.725832+0000","last_undegraded":"2021-02-10T04:56:12.725832+0000","last_fullsized":"2021-02-10T04:56:12.725832+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:53:11.051031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.051031+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24896285,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":18983408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"178","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.727940+0000","last_change":"2021-02-10T04:55:14.605585+0000","last_active":"2021-02-10T04:56:12.727940+0000","last_peered":"2021-02-10T04:56:12.727940+0000","last_clean":"2021-02-10T04:56:12.727940+0000","last_became_active":"2021-02-10T04:55:14.604916+0000","last_became_peered":"2021-02-10T04:55:14.604916+0000","last_unstale":"2021-02-10T04:56:12.727940+0000","last_undegraded":"2021-02-10T04:56:12.727940+0000","last_fullsized":"2021-02-10T04:56:12.727940+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:51:08.639784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.639784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"1199","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.782792+0000","last_change":"2021-02-10T04:51:09.475150+0000","last_active":"2021-02-10T04:56:12.782792+0000","last_peered":"2021-02-10T04:56:12.782792+0000","last_clean":"2021-02-10T04:56:12.782792+0000","last_became_active":"2021-02-10T04:50:59.585917+0000","last_became_peered":"2021-02-10T04:50:59.585917+0000","last_unstale":"2021-02-10T04:56:12.782792+0000","last_undegraded":"2021-02-10T04:56:12.782792+0000","last_fullsized":"2021-02-10T04:56:12.782792+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:09.475063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:09.475063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"169","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.729533+0000","last_change":"2021-02-10T04:53:34.927483+0000","last_active":"2021-02-10T04:56:12.729533+0000","last_peered":"2021-02-10T04:56:12.729533+0000","last_clean":"2021-02-10T04:56:12.729533+0000","last_became_active":"2021-02-10T04:51:36.859707+0000","last_became_peered":"2021-02-10T04:51:36.859707+0000","last_unstale":"2021-02-10T04:56:12.729533+0000","last_undegraded":"2021-02-10T04:56:12.729533+0000","last_fullsized":"2021-02-10T04:56:12.729533+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:34.927424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.927424+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"1000'620","reported_seq":"4554","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781878+0000","last_change":"2021-02-10T04:53:37.384461+0000","last_active":"2021-02-10T04:56:12.781878+0000","last_peered":"2021-02-10T04:56:12.781878+0000","last_clean":"2021-02-10T04:56:12.781878+0000","last_became_active":"2021-02-10T04:51:40.510114+0000","last_became_peered":"2021-02-10T04:51:40.510114+0000","last_unstale":"2021-02-10T04:56:12.781878+0000","last_undegraded":"2021-02-10T04:56:12.781878+0000","last_fullsized":"2021-02-10T04:56:12.781878+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:53:37.384398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:37.384398+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28530390,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"1256","reported_epoch":"1268","state":"active+clean","last_fresh":"2021-02-10T04:56:13.733402+0000","last_change":"2021-02-10T04:52:41.059269+0000","last_active":"2021-02-10T04:56:13.733402+0000","last_peered":"2021-02-10T04:56:13.733402+0000","last_clean":"2021-02-10T04:56:13.733402+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:56:13.733402+0000","last_undegraded":"2021-02-10T04:56:13.733402+0000","last_fullsized":"2021-02-10T04:56:13.733402+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:52:41.059159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.059159+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"1259","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.728768+0000","last_change":"2021-02-10T04:53:12.332890+0000","last_active":"2021-02-10T04:56:12.728768+0000","last_peered":"2021-02-10T04:56:12.728768+0000","last_clean":"2021-02-10T04:56:12.728768+0000","last_became_active":"2021-02-10T04:50:59.540375+0000","last_became_peered":"2021-02-10T04:50:59.540375+0000","last_unstale":"2021-02-10T04:56:12.728768+0000","last_undegraded":"2021-02-10T04:56:12.728768+0000","last_fullsized":"2021-02-10T04:56:12.728768+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.332233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.332233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"210","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.782621+0000","last_change":"2021-02-10T04:56:04.612627+0000","last_active":"2021-02-10T04:56:12.782621+0000","last_peered":"2021-02-10T04:56:12.782621+0000","last_clean":"2021-02-10T04:56:12.782621+0000","last_became_active":"2021-02-10T04:53:46.733191+0000","last_became_peered":"2021-02-10T04:53:46.733191+0000","last_unstale":"2021-02-10T04:56:12.782621+0000","last_undegraded":"2021-02-10T04:56:12.782621+0000","last_fullsized":"2021-02-10T04:56:12.782621+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:45.362537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:45.362537+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"1002'2042","reported_seq":"9240","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.783103+0000","last_change":"2021-02-10T04:53:46.726353+0000","last_active":"2021-02-10T04:56:12.783103+0000","last_peered":"2021-02-10T04:56:12.783103+0000","last_clean":"2021-02-10T04:56:12.783103+0000","last_became_active":"2021-02-10T04:53:46.725828+0000","last_became_peered":"2021-02-10T04:53:46.725828+0000","last_unstale":"2021-02-10T04:56:12.783103+0000","last_undegraded":"2021-02-10T04:56:12.783103+0000","last_fullsized":"2021-02-10T04:56:12.783103+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:52:41.286006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.286006+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46300076,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":545,"num_read_kb":531689,"num_write":399,"num_write_kb":153087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"1261'56","reported_seq":"1438","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.782036+0000","last_change":"2021-02-10T04:52:35.091494+0000","last_active":"2021-02-10T04:56:12.782036+0000","last_peered":"2021-02-10T04:56:12.782036+0000","last_clean":"2021-02-10T04:56:12.782036+0000","last_became_active":"2021-02-10T04:50:59.569915+0000","last_became_peered":"2021-02-10T04:50:59.569915+0000","last_unstale":"2021-02-10T04:56:12.782036+0000","last_undegraded":"2021-02-10T04:56:12.782036+0000","last_fullsized":"2021-02-10T04:56:12.782036+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'46","last_scrub_stamp":"2021-02-10T04:52:35.091374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:35.091374+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":95,"num_read_kb":80,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"84'7","reported_seq":"1283","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.728090+0000","last_change":"2021-02-10T04:51:52.498251+0000","last_active":"2021-02-10T04:56:12.728090+0000","last_peered":"2021-02-10T04:56:12.728090+0000","last_clean":"2021-02-10T04:56:12.728090+0000","last_became_active":"2021-02-10T04:50:59.575924+0000","last_became_peered":"2021-02-10T04:50:59.575924+0000","last_unstale":"2021-02-10T04:56:12.728090+0000","last_undegraded":"2021-02-10T04:56:12.728090+0000","last_fullsized":"2021-02-10T04:56:12.728090+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'7","last_scrub_stamp":"2021-02-10T04:51:52.498158+0000","last_deep_scrub":"84'7","last_deep_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.498158+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"1229'4","reported_seq":"1623","reported_epoch":"1268","state":"active+clean","last_fresh":"2021-02-10T04:56:13.735224+0000","last_change":"2021-02-10T04:55:14.657278+0000","last_active":"2021-02-10T04:56:13.735224+0000","last_peered":"2021-02-10T04:56:13.735224+0000","last_clean":"2021-02-10T04:56:13.735224+0000","last_became_active":"2021-02-10T04:55:14.656867+0000","last_became_peered":"2021-02-10T04:55:14.656867+0000","last_unstale":"2021-02-10T04:56:13.735224+0000","last_undegraded":"2021-02-10T04:56:13.735224+0000","last_fullsized":"2021-02-10T04:56:13.735224+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:47.407557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:47.407557+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"1181","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.782263+0000","last_change":"2021-02-10T04:55:22.735243+0000","last_active":"2021-02-10T04:56:12.782263+0000","last_peered":"2021-02-10T04:56:12.782263+0000","last_clean":"2021-02-10T04:56:12.782263+0000","last_became_active":"2021-02-10T04:55:14.632398+0000","last_became_peered":"2021-02-10T04:55:14.632398+0000","last_unstale":"2021-02-10T04:56:12.782263+0000","last_undegraded":"2021-02-10T04:56:12.782263+0000","last_fullsized":"2021-02-10T04:56:12.782263+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:23.566027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_clean_scrub_stamp":"2021-02-10T04:52:23.566027+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"1000'490","reported_seq":"3783","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.726849+0000","last_change":"2021-02-10T04:55:14.644527+0000","last_active":"2021-02-10T04:56:12.726849+0000","last_peered":"2021-02-10T04:56:12.726849+0000","last_clean":"2021-02-10T04:56:12.726849+0000","last_became_active":"2021-02-10T04:55:14.643764+0000","last_became_peered":"2021-02-10T04:55:14.643764+0000","last_unstale":"2021-02-10T04:56:12.726849+0000","last_undegraded":"2021-02-10T04:56:12.726849+0000","last_fullsized":"2021-02-10T04:56:12.726849+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:50:03.756248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:50:03.756248+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16154790,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"1261'46","reported_seq":"1336","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.728598+0000","last_change":"2021-02-10T04:53:10.276449+0000","last_active":"2021-02-10T04:56:12.728598+0000","last_peered":"2021-02-10T04:56:12.728598+0000","last_clean":"2021-02-10T04:56:12.728598+0000","last_became_active":"2021-02-10T04:50:59.539529+0000","last_became_peered":"2021-02-10T04:50:59.539529+0000","last_unstale":"2021-02-10T04:56:12.728598+0000","last_undegraded":"2021-02-10T04:56:12.728598+0000","last_fullsized":"2021-02-10T04:56:12.728598+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1112'40","last_scrub_stamp":"2021-02-10T04:53:10.276358+0000","last_deep_scrub":"1112'40","last_deep_scrub_stamp":"2021-02-10T04:51:01.450050+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.276358+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":53,"num_write":38,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"1166","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.780874+0000","last_change":"2021-02-10T04:55:28.088480+0000","last_active":"2021-02-10T04:56:12.780874+0000","last_peered":"2021-02-10T04:56:12.780874+0000","last_clean":"2021-02-10T04:56:12.780874+0000","last_became_active":"2021-02-10T04:55:14.631666+0000","last_became_peered":"2021-02-10T04:55:14.631666+0000","last_unstale":"2021-02-10T04:56:12.780874+0000","last_undegraded":"2021-02-10T04:56:12.780874+0000","last_fullsized":"2021-02-10T04:56:12.780874+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:23.904210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:47:23.904210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"1119'6","reported_seq":"1536","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:13.044370+0000","last_change":"2021-02-10T04:52:40.038646+0000","last_active":"2021-02-10T04:56:13.044370+0000","last_peered":"2021-02-10T04:56:13.044370+0000","last_clean":"2021-02-10T04:56:13.044370+0000","last_became_active":"2021-02-10T04:51:36.869733+0000","last_became_peered":"2021-02-10T04:51:36.869733+0000","last_unstale":"2021-02-10T04:56:13.044370+0000","last_undegraded":"2021-02-10T04:56:13.044370+0000","last_fullsized":"2021-02-10T04:56:13.044370+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'6","last_scrub_stamp":"2021-02-10T04:52:40.038554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:40.038554+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"1253","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.726452+0000","last_change":"2021-02-10T04:51:36.849933+0000","last_active":"2021-02-10T04:56:12.726452+0000","last_peered":"2021-02-10T04:56:12.726452+0000","last_clean":"2021-02-10T04:56:12.726452+0000","last_became_active":"2021-02-10T04:51:36.849184+0000","last_became_peered":"2021-02-10T04:51:36.849184+0000","last_unstale":"2021-02-10T04:56:12.726452+0000","last_undegraded":"2021-02-10T04:56:12.726452+0000","last_fullsized":"2021-02-10T04:56:12.726452+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:11.779779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:11.779779+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"1032'89","reported_seq":"861","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781237+0000","last_change":"2021-02-10T04:53:46.728868+0000","last_active":"2021-02-10T04:56:12.781237+0000","last_peered":"2021-02-10T04:56:12.781237+0000","last_clean":"2021-02-10T04:56:12.781237+0000","last_became_active":"2021-02-10T04:53:46.728356+0000","last_became_peered":"2021-02-10T04:53:46.728356+0000","last_unstale":"2021-02-10T04:56:12.781237+0000","last_undegraded":"2021-02-10T04:56:12.781237+0000","last_fullsized":"2021-02-10T04:56:12.781237+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":5,"last_scrub":"1032'89","last_scrub_stamp":"2021-02-10T04:52:34.089789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:34.089789+0000","log_size":89,"ondisk_log_size":89,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5218574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":30326,"num_write":21,"num_write_kb":12444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3210022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"1261'43","reported_seq":"957","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.729261+0000","last_change":"2021-02-10T04:53:14.202505+0000","last_active":"2021-02-10T04:56:12.729261+0000","last_peered":"2021-02-10T04:56:12.729261+0000","last_clean":"2021-02-10T04:56:12.729261+0000","last_became_active":"2021-02-10T04:51:49.496294+0000","last_became_peered":"2021-02-10T04:51:49.496294+0000","last_unstale":"2021-02-10T04:56:12.729261+0000","last_undegraded":"2021-02-10T04:56:12.729261+0000","last_fullsized":"2021-02-10T04:56:12.729261+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1163'37","last_scrub_stamp":"2021-02-10T04:53:14.202445+0000","last_deep_scrub":"1163'37","last_deep_scrub_stamp":"2021-02-10T04:53:14.202445+0000","last_clean_scrub_stamp":"2021-02-10T04:53:14.202445+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":60,"num_read_kb":42,"num_write":34,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"1269","reported_epoch":"1268","state":"active+clean","last_fresh":"2021-02-10T04:56:13.735568+0000","last_change":"2021-02-10T04:52:23.095651+0000","last_active":"2021-02-10T04:56:13.735568+0000","last_peered":"2021-02-10T04:56:13.735568+0000","last_clean":"2021-02-10T04:56:13.735568+0000","last_became_active":"2021-02-10T04:50:59.543531+0000","last_became_peered":"2021-02-10T04:50:59.543531+0000","last_unstale":"2021-02-10T04:56:13.735568+0000","last_undegraded":"2021-02-10T04:56:13.735568+0000","last_fullsized":"2021-02-10T04:56:13.735568+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:23.095588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:23.095588+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"1218","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781221+0000","last_change":"2021-02-10T04:55:14.636711+0000","last_active":"2021-02-10T04:56:12.781221+0000","last_peered":"2021-02-10T04:56:12.781221+0000","last_clean":"2021-02-10T04:56:12.781221+0000","last_became_active":"2021-02-10T04:55:14.636363+0000","last_became_peered":"2021-02-10T04:55:14.636363+0000","last_unstale":"2021-02-10T04:56:12.781221+0000","last_undegraded":"2021-02-10T04:56:12.781221+0000","last_fullsized":"2021-02-10T04:56:12.781221+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:43.663762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:51:43.663762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"1137","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.780164+0000","last_change":"2021-02-10T04:55:28.101709+0000","last_active":"2021-02-10T04:56:12.780164+0000","last_peered":"2021-02-10T04:56:12.780164+0000","last_clean":"2021-02-10T04:56:12.780164+0000","last_became_active":"2021-02-10T04:55:14.635912+0000","last_became_peered":"2021-02-10T04:55:14.635912+0000","last_unstale":"2021-02-10T04:56:12.780164+0000","last_undegraded":"2021-02-10T04:56:12.780164+0000","last_fullsized":"2021-02-10T04:56:12.780164+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:57.568541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:57.568541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"1000'1071","reported_seq":"4773","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.780210+0000","last_change":"2021-02-10T04:53:35.302879+0000","last_active":"2021-02-10T04:56:12.780210+0000","last_peered":"2021-02-10T04:56:12.780210+0000","last_clean":"2021-02-10T04:56:12.780210+0000","last_became_active":"2021-02-10T04:51:39.508551+0000","last_became_peered":"2021-02-10T04:51:39.508551+0000","last_unstale":"2021-02-10T04:56:12.780210+0000","last_undegraded":"2021-02-10T04:56:12.780210+0000","last_fullsized":"2021-02-10T04:56:12.780210+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:35.302780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.302780+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"1261'39","reported_seq":"1302","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.725471+0000","last_change":"2021-02-10T04:52:39.695340+0000","last_active":"2021-02-10T04:56:12.725471+0000","last_peered":"2021-02-10T04:56:12.725471+0000","last_clean":"2021-02-10T04:56:12.725471+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:56:12.725471+0000","last_undegraded":"2021-02-10T04:56:12.725471+0000","last_fullsized":"2021-02-10T04:56:12.725471+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'31","last_scrub_stamp":"2021-02-10T04:52:39.695237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:39.695237+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":53,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"512","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.732041+0000","last_change":"2021-02-10T04:55:20.929399+0000","last_active":"2021-02-10T04:56:12.732041+0000","last_peered":"2021-02-10T04:56:12.732041+0000","last_clean":"2021-02-10T04:56:12.732041+0000","last_became_active":"2021-02-10T04:55:14.655543+0000","last_became_peered":"2021-02-10T04:55:14.655543+0000","last_unstale":"2021-02-10T04:56:12.732041+0000","last_undegraded":"2021-02-10T04:56:12.732041+0000","last_fullsized":"2021-02-10T04:56:12.732041+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:55:20.929330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:55:20.929330+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"920","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.729424+0000","last_change":"2021-02-10T04:53:35.216489+0000","last_active":"2021-02-10T04:56:12.729424+0000","last_peered":"2021-02-10T04:56:12.729424+0000","last_clean":"2021-02-10T04:56:12.729424+0000","last_became_active":"2021-02-10T04:51:49.501254+0000","last_became_peered":"2021-02-10T04:51:49.501254+0000","last_unstale":"2021-02-10T04:56:12.729424+0000","last_undegraded":"2021-02-10T04:56:12.729424+0000","last_fullsized":"2021-02-10T04:56:12.729424+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.216394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.216394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"1245","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.780549+0000","last_change":"2021-02-10T04:55:14.633276+0000","last_active":"2021-02-10T04:56:12.780549+0000","last_peered":"2021-02-10T04:56:12.780549+0000","last_clean":"2021-02-10T04:56:12.780549+0000","last_became_active":"2021-02-10T04:55:14.632805+0000","last_became_peered":"2021-02-10T04:55:14.632805+0000","last_unstale":"2021-02-10T04:56:12.780549+0000","last_undegraded":"2021-02-10T04:56:12.780549+0000","last_fullsized":"2021-02-10T04:56:12.780549+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:55.814486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.814486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"1002'1657","reported_seq":"5290","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.780724+0000","last_change":"2021-02-10T04:55:22.728878+0000","last_active":"2021-02-10T04:56:12.780724+0000","last_peered":"2021-02-10T04:56:12.780724+0000","last_clean":"2021-02-10T04:56:12.780724+0000","last_became_active":"2021-02-10T04:55:14.653270+0000","last_became_peered":"2021-02-10T04:55:14.653270+0000","last_unstale":"2021-02-10T04:56:12.780724+0000","last_undegraded":"2021-02-10T04:56:12.780724+0000","last_fullsized":"2021-02-10T04:56:12.780724+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:55:22.331010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:55:22.331010+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":47335497,"num_objects":20,"num_object_clones":18,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"1261'24","reported_seq":"1375","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781006+0000","last_change":"2021-02-10T04:53:04.058324+0000","last_active":"2021-02-10T04:56:12.781006+0000","last_peered":"2021-02-10T04:56:12.781006+0000","last_clean":"2021-02-10T04:56:12.781006+0000","last_became_active":"2021-02-10T04:50:59.544484+0000","last_became_peered":"2021-02-10T04:50:59.544484+0000","last_unstale":"2021-02-10T04:56:12.781006+0000","last_undegraded":"2021-02-10T04:56:12.781006+0000","last_fullsized":"2021-02-10T04:56:12.781006+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'20","last_scrub_stamp":"2021-02-10T04:53:04.058248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:04.058248+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":29,"num_write":24,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"1271","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.728870+0000","last_change":"2021-02-10T04:53:00.343307+0000","last_active":"2021-02-10T04:56:12.728870+0000","last_peered":"2021-02-10T04:56:12.728870+0000","last_clean":"2021-02-10T04:56:12.728870+0000","last_became_active":"2021-02-10T04:51:36.844573+0000","last_became_peered":"2021-02-10T04:51:36.844573+0000","last_unstale":"2021-02-10T04:56:12.728870+0000","last_undegraded":"2021-02-10T04:56:12.728870+0000","last_fullsized":"2021-02-10T04:56:12.728870+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:00.343228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:00.343228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"1240","reported_epoch":"1268","state":"active+clean","last_fresh":"2021-02-10T04:56:13.733321+0000","last_change":"2021-02-10T04:50:56.099710+0000","last_active":"2021-02-10T04:56:13.733321+0000","last_peered":"2021-02-10T04:56:13.733321+0000","last_clean":"2021-02-10T04:56:13.733321+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:56:13.733321+0000","last_undegraded":"2021-02-10T04:56:13.733321+0000","last_fullsized":"2021-02-10T04:56:13.733321+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:56.099651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:50:56.099651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"586","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781946+0000","last_change":"2021-02-10T04:53:46.718715+0000","last_active":"2021-02-10T04:56:12.781946+0000","last_peered":"2021-02-10T04:56:12.781946+0000","last_clean":"2021-02-10T04:56:12.781946+0000","last_became_active":"2021-02-10T04:53:46.718266+0000","last_became_peered":"2021-02-10T04:53:46.718266+0000","last_unstale":"2021-02-10T04:56:12.781946+0000","last_undegraded":"2021-02-10T04:56:12.781946+0000","last_fullsized":"2021-02-10T04:56:12.781946+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.876681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.876681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"998'1009","reported_seq":"5116","reported_epoch":"1268","state":"active+clean","last_fresh":"2021-02-10T04:56:13.733905+0000","last_change":"2021-02-10T04:55:14.635764+0000","last_active":"2021-02-10T04:56:13.733905+0000","last_peered":"2021-02-10T04:56:13.733905+0000","last_clean":"2021-02-10T04:56:13.733905+0000","last_became_active":"2021-02-10T04:55:14.634422+0000","last_became_peered":"2021-02-10T04:55:14.634422+0000","last_unstale":"2021-02-10T04:56:13.733905+0000","last_undegraded":"2021-02-10T04:56:13.733905+0000","last_fullsized":"2021-02-10T04:56:13.733905+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":5,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:13.032506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:13.032506+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49420827,"num_objects":21,"num_object_clones":19,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":36763798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"1261'45","reported_seq":"1309","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781023+0000","last_change":"2021-02-10T04:51:44.694779+0000","last_active":"2021-02-10T04:56:12.781023+0000","last_peered":"2021-02-10T04:56:12.781023+0000","last_clean":"2021-02-10T04:56:12.781023+0000","last_became_active":"2021-02-10T04:50:59.549412+0000","last_became_peered":"2021-02-10T04:50:59.549412+0000","last_unstale":"2021-02-10T04:56:12.781023+0000","last_undegraded":"2021-02-10T04:56:12.781023+0000","last_fullsized":"2021-02-10T04:56:12.781023+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:44.694711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.694711+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":76,"num_read_kb":62,"num_write":45,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"1407","reported_epoch":"1268","state":"active+clean","last_fresh":"2021-02-10T04:56:13.736913+0000","last_change":"2021-02-10T04:52:33.088525+0000","last_active":"2021-02-10T04:56:13.736913+0000","last_peered":"2021-02-10T04:56:13.736913+0000","last_clean":"2021-02-10T04:56:13.736913+0000","last_became_active":"2021-02-10T04:51:36.870573+0000","last_became_peered":"2021-02-10T04:51:36.870573+0000","last_unstale":"2021-02-10T04:56:13.736913+0000","last_undegraded":"2021-02-10T04:56:13.736913+0000","last_fullsized":"2021-02-10T04:56:13.736913+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:52:33.088428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:33.088428+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"553","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781084+0000","last_change":"2021-02-10T04:51:51.836723+0000","last_active":"2021-02-10T04:56:12.781084+0000","last_peered":"2021-02-10T04:56:12.781084+0000","last_clean":"2021-02-10T04:56:12.781084+0000","last_became_active":"2021-02-10T04:50:59.564977+0000","last_became_peered":"2021-02-10T04:50:59.564977+0000","last_unstale":"2021-02-10T04:56:12.781084+0000","last_undegraded":"2021-02-10T04:56:12.781084+0000","last_fullsized":"2021-02-10T04:56:12.781084+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.836638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.836638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"1209","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781150+0000","last_change":"2021-02-10T04:55:14.633046+0000","last_active":"2021-02-10T04:56:12.781150+0000","last_peered":"2021-02-10T04:56:12.781150+0000","last_clean":"2021-02-10T04:56:12.781150+0000","last_became_active":"2021-02-10T04:55:14.632658+0000","last_became_peered":"2021-02-10T04:55:14.632658+0000","last_unstale":"2021-02-10T04:56:12.781150+0000","last_undegraded":"2021-02-10T04:56:12.781150+0000","last_fullsized":"2021-02-10T04:56:12.781150+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:58.514318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_clean_scrub_stamp":"2021-02-10T04:46:58.514318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1d","version":"1001'1165","reported_seq":"4268","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.732125+0000","last_change":"2021-02-10T04:55:14.656236+0000","last_active":"2021-02-10T04:56:12.732125+0000","last_peered":"2021-02-10T04:56:12.732125+0000","last_clean":"2021-02-10T04:56:12.732125+0000","last_became_active":"2021-02-10T04:55:14.655859+0000","last_became_peered":"2021-02-10T04:55:14.655859+0000","last_unstale":"2021-02-10T04:56:12.732125+0000","last_undegraded":"2021-02-10T04:56:12.732125+0000","last_fullsized":"2021-02-10T04:56:12.732125+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":5,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29601845,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":287,"num_read_kb":299530,"num_write":200,"num_write_kb":89131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"1312","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.727441+0000","last_change":"2021-02-10T04:53:01.331566+0000","last_active":"2021-02-10T04:56:12.727441+0000","last_peered":"2021-02-10T04:56:12.727441+0000","last_clean":"2021-02-10T04:56:12.727441+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:56:12.727441+0000","last_undegraded":"2021-02-10T04:56:12.727441+0000","last_fullsized":"2021-02-10T04:56:12.727441+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:01.331466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.331466+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"1098'3","reported_seq":"1614","reported_epoch":"1268","state":"active+clean","last_fresh":"2021-02-10T04:56:18.045267+0000","last_change":"2021-02-10T04:53:34.865668+0000","last_active":"2021-02-10T04:56:18.045267+0000","last_peered":"2021-02-10T04:56:18.045267+0000","last_clean":"2021-02-10T04:56:18.045267+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:56:18.045267+0000","last_undegraded":"2021-02-10T04:56:18.045267+0000","last_fullsized":"2021-02-10T04:56:18.045267+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"1098'3","last_scrub_stamp":"2021-02-10T04:53:34.865579+0000","last_deep_scrub":"1098'3","last_deep_scrub_stamp":"2021-02-10T04:50:15.255137+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.865579+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"1116","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.782666+0000","last_change":"2021-02-10T04:53:46.727603+0000","last_active":"2021-02-10T04:56:12.782666+0000","last_peered":"2021-02-10T04:56:12.782666+0000","last_clean":"2021-02-10T04:56:12.782666+0000","last_became_active":"2021-02-10T04:53:46.727053+0000","last_became_peered":"2021-02-10T04:53:46.727053+0000","last_unstale":"2021-02-10T04:56:12.782666+0000","last_undegraded":"2021-02-10T04:56:12.782666+0000","last_fullsized":"2021-02-10T04:56:12.782666+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:47.493057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.493057+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1c","version":"1002'1090","reported_seq":"5292","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.728260+0000","last_change":"2021-02-10T04:53:41.353991+0000","last_active":"2021-02-10T04:56:12.728260+0000","last_peered":"2021-02-10T04:56:12.728260+0000","last_clean":"2021-02-10T04:56:12.728260+0000","last_became_active":"2021-02-10T04:52:31.088265+0000","last_became_peered":"2021-02-10T04:52:31.088265+0000","last_unstale":"2021-02-10T04:56:12.728260+0000","last_undegraded":"2021-02-10T04:56:12.728260+0000","last_fullsized":"2021-02-10T04:56:12.728260+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:53:41.353840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:41.353840+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"1213","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.780744+0000","last_change":"2021-02-10T04:55:14.632817+0000","last_active":"2021-02-10T04:56:12.780744+0000","last_peered":"2021-02-10T04:56:12.780744+0000","last_clean":"2021-02-10T04:56:12.780744+0000","last_became_active":"2021-02-10T04:55:14.632214+0000","last_became_peered":"2021-02-10T04:55:14.632214+0000","last_unstale":"2021-02-10T04:56:12.780744+0000","last_undegraded":"2021-02-10T04:56:12.780744+0000","last_fullsized":"2021-02-10T04:56:12.780744+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:42.537443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:42.537443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"1162","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781838+0000","last_change":"2021-02-10T04:55:22.725805+0000","last_active":"2021-02-10T04:56:12.781838+0000","last_peered":"2021-02-10T04:56:12.781838+0000","last_clean":"2021-02-10T04:56:12.781838+0000","last_became_active":"2021-02-10T04:55:14.630190+0000","last_became_peered":"2021-02-10T04:55:14.630190+0000","last_unstale":"2021-02-10T04:56:12.781838+0000","last_undegraded":"2021-02-10T04:56:12.781838+0000","last_fullsized":"2021-02-10T04:56:12.781838+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.938731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.938731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"477","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.731899+0000","last_change":"2021-02-10T04:53:15.185748+0000","last_active":"2021-02-10T04:56:12.731899+0000","last_peered":"2021-02-10T04:56:12.731899+0000","last_clean":"2021-02-10T04:56:12.731899+0000","last_became_active":"2021-02-10T04:51:49.499348+0000","last_became_peered":"2021-02-10T04:51:49.499348+0000","last_unstale":"2021-02-10T04:56:12.731899+0000","last_undegraded":"2021-02-10T04:56:12.731899+0000","last_fullsized":"2021-02-10T04:56:12.731899+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:15.185686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:15.185686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1f","version":"1004'1093","reported_seq":"6530","reported_epoch":"1268","state":"active+clean","last_fresh":"2021-02-10T04:56:13.736524+0000","last_change":"2021-02-10T04:52:31.089952+0000","last_active":"2021-02-10T04:56:13.736524+0000","last_peered":"2021-02-10T04:56:13.736524+0000","last_clean":"2021-02-10T04:56:13.736524+0000","last_became_active":"2021-02-10T04:52:31.088634+0000","last_became_peered":"2021-02-10T04:52:31.088634+0000","last_unstale":"2021-02-10T04:56:13.736524+0000","last_undegraded":"2021-02-10T04:56:13.736524+0000","last_fullsized":"2021-02-10T04:56:13.736524+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39434697,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":279,"num_read_kb":311058,"num_write":186,"num_write_kb":66790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":53995913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"1270","reported_epoch":"1268","state":"active+clean","last_fresh":"2021-02-10T04:56:13.734374+0000","last_change":"2021-02-10T04:55:39.651056+0000","last_active":"2021-02-10T04:56:13.734374+0000","last_peered":"2021-02-10T04:56:13.734374+0000","last_clean":"2021-02-10T04:56:13.734374+0000","last_became_active":"2021-02-10T04:55:39.596648+0000","last_became_peered":"2021-02-10T04:55:39.596648+0000","last_unstale":"2021-02-10T04:56:13.734374+0000","last_undegraded":"2021-02-10T04:56:13.734374+0000","last_fullsized":"2021-02-10T04:56:13.734374+0000","mapping_epoch":1248,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:52:36.083784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:36.083784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"509","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.732383+0000","last_change":"2021-02-10T04:56:04.044582+0000","last_active":"2021-02-10T04:56:12.732383+0000","last_peered":"2021-02-10T04:56:12.732383+0000","last_clean":"2021-02-10T04:56:12.732383+0000","last_became_active":"2021-02-10T04:55:14.655649+0000","last_became_peered":"2021-02-10T04:55:14.655649+0000","last_unstale":"2021-02-10T04:56:12.732383+0000","last_undegraded":"2021-02-10T04:56:12.732383+0000","last_fullsized":"2021-02-10T04:56:12.732383+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:08.758129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.758129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"1235","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.726625+0000","last_change":"2021-02-10T04:51:44.799512+0000","last_active":"2021-02-10T04:56:12.726625+0000","last_peered":"2021-02-10T04:56:12.726625+0000","last_clean":"2021-02-10T04:56:12.726625+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:56:12.726625+0000","last_undegraded":"2021-02-10T04:56:12.726625+0000","last_fullsized":"2021-02-10T04:56:12.726625+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:44.799429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.799429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1e","version":"1063'654","reported_seq":"4510","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.729187+0000","last_change":"2021-02-10T04:53:40.135450+0000","last_active":"2021-02-10T04:56:12.729187+0000","last_peered":"2021-02-10T04:56:12.729187+0000","last_clean":"2021-02-10T04:56:12.729187+0000","last_became_active":"2021-02-10T04:52:31.087662+0000","last_became_peered":"2021-02-10T04:52:31.087662+0000","last_unstale":"2021-02-10T04:56:12.729187+0000","last_undegraded":"2021-02-10T04:56:12.729187+0000","last_fullsized":"2021-02-10T04:56:12.729187+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:53:40.135367+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:53:40.135367+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"1219","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781637+0000","last_change":"2021-02-10T04:51:08.006463+0000","last_active":"2021-02-10T04:56:12.781637+0000","last_peered":"2021-02-10T04:56:12.781637+0000","last_clean":"2021-02-10T04:56:12.781637+0000","last_became_active":"2021-02-10T04:50:59.563657+0000","last_became_peered":"2021-02-10T04:50:59.563657+0000","last_unstale":"2021-02-10T04:56:12.781637+0000","last_undegraded":"2021-02-10T04:56:12.781637+0000","last_fullsized":"2021-02-10T04:56:12.781637+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_clean_scrub_stamp":"2021-02-10T04:51:02.455702+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"1270","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.725413+0000","last_change":"2021-02-10T04:52:33.741801+0000","last_active":"2021-02-10T04:56:12.725413+0000","last_peered":"2021-02-10T04:56:12.725413+0000","last_clean":"2021-02-10T04:56:12.725413+0000","last_became_active":"2021-02-10T04:51:36.860129+0000","last_became_peered":"2021-02-10T04:51:36.860129+0000","last_unstale":"2021-02-10T04:56:12.725413+0000","last_undegraded":"2021-02-10T04:56:12.725413+0000","last_fullsized":"2021-02-10T04:56:12.725413+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:33.741711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:46:58.666264+0000","last_clean_scrub_stamp":"2021-02-10T04:52:33.741711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"1231","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.782652+0000","last_change":"2021-02-10T04:52:25.030419+0000","last_active":"2021-02-10T04:56:12.782652+0000","last_peered":"2021-02-10T04:56:12.782652+0000","last_clean":"2021-02-10T04:56:12.782652+0000","last_became_active":"2021-02-10T04:50:59.539582+0000","last_became_peered":"2021-02-10T04:50:59.539582+0000","last_unstale":"2021-02-10T04:56:12.782652+0000","last_undegraded":"2021-02-10T04:56:12.782652+0000","last_fullsized":"2021-02-10T04:56:12.782652+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:25.030281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.030281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"1003'739","reported_seq":"4017","reported_epoch":"1268","state":"active+clean","last_fresh":"2021-02-10T04:56:13.733009+0000","last_change":"2021-02-10T04:53:01.929261+0000","last_active":"2021-02-10T04:56:13.733009+0000","last_peered":"2021-02-10T04:56:13.733009+0000","last_clean":"2021-02-10T04:56:13.733009+0000","last_became_active":"2021-02-10T04:51:40.511505+0000","last_became_peered":"2021-02-10T04:51:40.511505+0000","last_unstale":"2021-02-10T04:56:13.733009+0000","last_undegraded":"2021-02-10T04:56:13.733009+0000","last_fullsized":"2021-02-10T04:56:13.733009+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:53:01.929169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.929169+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22230846,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":18429397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"1107","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.782472+0000","last_change":"2021-02-10T04:53:46.727590+0000","last_active":"2021-02-10T04:56:12.782472+0000","last_peered":"2021-02-10T04:56:12.782472+0000","last_clean":"2021-02-10T04:56:12.782472+0000","last_became_active":"2021-02-10T04:53:46.726584+0000","last_became_peered":"2021-02-10T04:53:46.726584+0000","last_unstale":"2021-02-10T04:56:12.782472+0000","last_undegraded":"2021-02-10T04:56:12.782472+0000","last_fullsized":"2021-02-10T04:56:12.782472+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:26.408326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.408326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"1268","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.728019+0000","last_change":"2021-02-10T04:53:07.324650+0000","last_active":"2021-02-10T04:56:12.728019+0000","last_peered":"2021-02-10T04:56:12.728019+0000","last_clean":"2021-02-10T04:56:12.728019+0000","last_became_active":"2021-02-10T04:50:59.575340+0000","last_became_peered":"2021-02-10T04:50:59.575340+0000","last_unstale":"2021-02-10T04:56:12.728019+0000","last_undegraded":"2021-02-10T04:56:12.728019+0000","last_fullsized":"2021-02-10T04:56:12.728019+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.324564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.324564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"1161","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.780048+0000","last_change":"2021-02-10T04:55:28.098627+0000","last_active":"2021-02-10T04:56:12.780048+0000","last_peered":"2021-02-10T04:56:12.780048+0000","last_clean":"2021-02-10T04:56:12.780048+0000","last_became_active":"2021-02-10T04:55:14.640497+0000","last_became_peered":"2021-02-10T04:55:14.640497+0000","last_unstale":"2021-02-10T04:56:12.780048+0000","last_undegraded":"2021-02-10T04:56:12.780048+0000","last_fullsized":"2021-02-10T04:56:12.780048+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:00.501127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:00.501127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"259'74","reported_seq":"206","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.780155+0000","last_change":"2021-02-10T04:55:28.095458+0000","last_active":"2021-02-10T04:56:12.780155+0000","last_peered":"2021-02-10T04:56:12.780155+0000","last_clean":"2021-02-10T04:56:12.780155+0000","last_became_active":"2021-02-10T04:55:14.624071+0000","last_became_peered":"2021-02-10T04:55:14.624071+0000","last_unstale":"2021-02-10T04:56:12.780155+0000","last_undegraded":"2021-02-10T04:56:12.780155+0000","last_fullsized":"2021-02-10T04:56:12.780155+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":5,"last_scrub":"259'74","last_scrub_stamp":"2021-02-10T04:52:26.597668+0000","last_deep_scrub":"259'74","last_deep_scrub_stamp":"2021-02-10T04:46:49.745104+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.597668+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1218524,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":28228,"num_write":19,"num_write_kb":7022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"29","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.729467+0000","last_change":"2021-02-10T04:55:39.619500+0000","last_active":"2021-02-10T04:56:12.729467+0000","last_peered":"2021-02-10T04:56:12.729467+0000","last_clean":"2021-02-10T04:56:12.729467+0000","last_became_active":"2021-02-10T04:55:39.619062+0000","last_became_peered":"2021-02-10T04:55:39.619062+0000","last_unstale":"2021-02-10T04:56:12.729467+0000","last_undegraded":"2021-02-10T04:56:12.729467+0000","last_fullsized":"2021-02-10T04:56:12.729467+0000","mapping_epoch":1248,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:59.167911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:50:59.167911+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"177","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781657+0000","last_change":"2021-02-10T04:53:46.718512+0000","last_active":"2021-02-10T04:56:12.781657+0000","last_peered":"2021-02-10T04:56:12.781657+0000","last_clean":"2021-02-10T04:56:12.781657+0000","last_became_active":"2021-02-10T04:53:46.716811+0000","last_became_peered":"2021-02-10T04:53:46.716811+0000","last_unstale":"2021-02-10T04:56:12.781657+0000","last_undegraded":"2021-02-10T04:56:12.781657+0000","last_fullsized":"2021-02-10T04:56:12.781657+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:45.813676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.813676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"1257","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.727763+0000","last_change":"2021-02-10T04:53:38.363899+0000","last_active":"2021-02-10T04:56:12.727763+0000","last_peered":"2021-02-10T04:56:12.727763+0000","last_clean":"2021-02-10T04:56:12.727763+0000","last_became_active":"2021-02-10T04:50:59.539256+0000","last_became_peered":"2021-02-10T04:50:59.539256+0000","last_unstale":"2021-02-10T04:56:12.727763+0000","last_undegraded":"2021-02-10T04:56:12.727763+0000","last_fullsized":"2021-02-10T04:56:12.727763+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:38.363825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:09.433793+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.363825+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1b","version":"1003'1689","reported_seq":"4613","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.780512+0000","last_change":"2021-02-10T04:55:15.591337+0000","last_active":"2021-02-10T04:56:12.780512+0000","last_peered":"2021-02-10T04:56:12.780512+0000","last_clean":"2021-02-10T04:56:12.780512+0000","last_became_active":"2021-02-10T04:55:15.590301+0000","last_became_peered":"2021-02-10T04:55:15.590301+0000","last_unstale":"2021-02-10T04:56:12.780512+0000","last_undegraded":"2021-02-10T04:56:12.780512+0000","last_fullsized":"2021-02-10T04:56:12.780512+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:53:10.075879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.075879+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":63260738,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":387,"num_read_kb":409791,"num_write":281,"num_write_kb":110559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":22449943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"1252","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.782397+0000","last_change":"2021-02-10T04:55:14.637968+0000","last_active":"2021-02-10T04:56:12.782397+0000","last_peered":"2021-02-10T04:56:12.782397+0000","last_clean":"2021-02-10T04:56:12.782397+0000","last_became_active":"2021-02-10T04:55:14.636561+0000","last_became_peered":"2021-02-10T04:55:14.636561+0000","last_unstale":"2021-02-10T04:56:12.782397+0000","last_undegraded":"2021-02-10T04:56:12.782397+0000","last_fullsized":"2021-02-10T04:56:12.782397+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:33.543559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:46:33.543559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"927","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.727537+0000","last_change":"2021-02-10T04:53:04.354508+0000","last_active":"2021-02-10T04:56:12.727537+0000","last_peered":"2021-02-10T04:56:12.727537+0000","last_clean":"2021-02-10T04:56:12.727537+0000","last_became_active":"2021-02-10T04:33:05.169159+0000","last_became_peered":"2021-02-10T04:33:05.169159+0000","last_unstale":"2021-02-10T04:56:12.727537+0000","last_undegraded":"2021-02-10T04:56:12.727537+0000","last_fullsized":"2021-02-10T04:56:12.727537+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:04.354409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:04.354409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"1321","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.724059+0000","last_change":"2021-02-10T04:52:36.777554+0000","last_active":"2021-02-10T04:56:12.724059+0000","last_peered":"2021-02-10T04:56:12.724059+0000","last_clean":"2021-02-10T04:56:12.724059+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:56:12.724059+0000","last_undegraded":"2021-02-10T04:56:12.724059+0000","last_fullsized":"2021-02-10T04:56:12.724059+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:36.777485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:52:36.777485+0000","last_clean_scrub_stamp":"2021-02-10T04:52:36.777485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1a","version":"1001'1528","reported_seq":"5878","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.724334+0000","last_change":"2021-02-10T04:52:31.086406+0000","last_active":"2021-02-10T04:56:12.724334+0000","last_peered":"2021-02-10T04:56:12.724334+0000","last_clean":"2021-02-10T04:56:12.724334+0000","last_became_active":"2021-02-10T04:52:31.085959+0000","last_became_peered":"2021-02-10T04:52:31.085959+0000","last_unstale":"2021-02-10T04:56:12.724334+0000","last_undegraded":"2021-02-10T04:56:12.724334+0000","last_fullsized":"2021-02-10T04:56:12.724334+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46944136,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":358,"num_read_kb":351935,"num_write":286,"num_write_kb":111355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":21311893,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"1151","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.782393+0000","last_change":"2021-02-10T04:55:22.726939+0000","last_active":"2021-02-10T04:56:12.782393+0000","last_peered":"2021-02-10T04:56:12.782393+0000","last_clean":"2021-02-10T04:56:12.782393+0000","last_became_active":"2021-02-10T04:55:14.627044+0000","last_became_peered":"2021-02-10T04:55:14.627044+0000","last_unstale":"2021-02-10T04:56:12.782393+0000","last_undegraded":"2021-02-10T04:56:12.782393+0000","last_fullsized":"2021-02-10T04:56:12.782393+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:17.919745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:17.919745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"553","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.780140+0000","last_change":"2021-02-10T04:53:07.092032+0000","last_active":"2021-02-10T04:56:12.780140+0000","last_peered":"2021-02-10T04:56:12.780140+0000","last_clean":"2021-02-10T04:56:12.780140+0000","last_became_active":"2021-02-10T04:50:59.572077+0000","last_became_peered":"2021-02-10T04:50:59.572077+0000","last_unstale":"2021-02-10T04:56:12.780140+0000","last_undegraded":"2021-02-10T04:56:12.780140+0000","last_fullsized":"2021-02-10T04:56:12.780140+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.091949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:19.755718+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.091949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"1230","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781358+0000","last_change":"2021-02-10T04:51:59.838856+0000","last_active":"2021-02-10T04:56:12.781358+0000","last_peered":"2021-02-10T04:56:12.781358+0000","last_clean":"2021-02-10T04:56:12.781358+0000","last_became_active":"2021-02-10T04:50:59.563156+0000","last_became_peered":"2021-02-10T04:50:59.563156+0000","last_unstale":"2021-02-10T04:56:12.781358+0000","last_undegraded":"2021-02-10T04:56:12.781358+0000","last_fullsized":"2021-02-10T04:56:12.781358+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.838790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.838790+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":"963","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.728212+0000","last_change":"2021-02-10T04:52:30.405170+0000","last_active":"2021-02-10T04:56:12.728212+0000","last_peered":"2021-02-10T04:56:12.728212+0000","last_clean":"2021-02-10T04:56:12.728212+0000","last_became_active":"2021-02-10T04:51:36.852058+0000","last_became_peered":"2021-02-10T04:51:36.852058+0000","last_unstale":"2021-02-10T04:56:12.728212+0000","last_undegraded":"2021-02-10T04:56:12.728212+0000","last_fullsized":"2021-02-10T04:56:12.728212+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.405106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:20.635536+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.405106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.20","version":"0'0","reported_seq":"201","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.780873+0000","last_change":"2021-02-10T04:53:46.739888+0000","last_active":"2021-02-10T04:56:12.780873+0000","last_peered":"2021-02-10T04:56:12.780873+0000","last_clean":"2021-02-10T04:56:12.780873+0000","last_became_active":"2021-02-10T04:53:46.739532+0000","last_became_peered":"2021-02-10T04:53:46.739532+0000","last_unstale":"2021-02-10T04:56:12.780873+0000","last_undegraded":"2021-02-10T04:56:12.780873+0000","last_fullsized":"2021-02-10T04:56:12.780873+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:56.449085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:41.375276+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.449085+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":"1196","reported_epoch":"1268","state":"active+clean","last_fresh":"2021-02-10T04:56:13.735711+0000","last_change":"2021-02-10T04:53:35.858634+0000","last_active":"2021-02-10T04:56:13.735711+0000","last_peered":"2021-02-10T04:56:13.735711+0000","last_clean":"2021-02-10T04:56:13.735711+0000","last_became_active":"2021-02-10T04:51:36.869785+0000","last_became_peered":"2021-02-10T04:51:36.869785+0000","last_unstale":"2021-02-10T04:56:13.735711+0000","last_undegraded":"2021-02-10T04:56:13.735711+0000","last_fullsized":"2021-02-10T04:56:13.735711+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.858557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.858557+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":"179","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.728989+0000","last_change":"2021-02-10T04:55:23.586523+0000","last_active":"2021-02-10T04:56:12.728989+0000","last_peered":"2021-02-10T04:56:12.728989+0000","last_clean":"2021-02-10T04:56:12.728989+0000","last_became_active":"2021-02-10T04:55:14.637029+0000","last_became_peered":"2021-02-10T04:55:14.637029+0000","last_unstale":"2021-02-10T04:56:12.728989+0000","last_undegraded":"2021-02-10T04:56:12.728989+0000","last_fullsized":"2021-02-10T04:56:12.728989+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:55:23.586401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:55:23.586401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":"164","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.728746+0000","last_change":"2021-02-10T04:55:14.635842+0000","last_active":"2021-02-10T04:56:12.728746+0000","last_peered":"2021-02-10T04:56:12.728746+0000","last_clean":"2021-02-10T04:56:12.728746+0000","last_became_active":"2021-02-10T04:55:14.635499+0000","last_became_peered":"2021-02-10T04:55:14.635499+0000","last_unstale":"2021-02-10T04:56:12.728746+0000","last_undegraded":"2021-02-10T04:56:12.728746+0000","last_fullsized":"2021-02-10T04:56:12.728746+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.706372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.706372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":"1278","reported_epoch":"1268","state":"active+clean","last_fresh":"2021-02-10T04:56:13.734035+0000","last_change":"2021-02-10T04:51:57.287452+0000","last_active":"2021-02-10T04:56:13.734035+0000","last_peered":"2021-02-10T04:56:13.734035+0000","last_clean":"2021-02-10T04:56:13.734035+0000","last_became_active":"2021-02-10T04:50:59.566827+0000","last_became_peered":"2021-02-10T04:50:59.566827+0000","last_unstale":"2021-02-10T04:56:13.734035+0000","last_undegraded":"2021-02-10T04:56:13.734035+0000","last_fullsized":"2021-02-10T04:56:13.734035+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.287363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.287363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":"252","reported_epoch":"1268","state":"active+clean","last_fresh":"2021-02-10T04:56:13.736595+0000","last_change":"2021-02-10T04:55:14.659882+0000","last_active":"2021-02-10T04:56:13.736595+0000","last_peered":"2021-02-10T04:56:13.736595+0000","last_clean":"2021-02-10T04:56:13.736595+0000","last_became_active":"2021-02-10T04:55:14.659514+0000","last_became_peered":"2021-02-10T04:55:14.659514+0000","last_unstale":"2021-02-10T04:56:13.736595+0000","last_undegraded":"2021-02-10T04:56:13.736595+0000","last_fullsized":"2021-02-10T04:56:13.736595+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:10.308282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:10.308282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":"1166","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781703+0000","last_change":"2021-02-10T04:53:09.121379+0000","last_active":"2021-02-10T04:56:12.781703+0000","last_peered":"2021-02-10T04:56:12.781703+0000","last_clean":"2021-02-10T04:56:12.781703+0000","last_became_active":"2021-02-10T04:50:59.561442+0000","last_became_peered":"2021-02-10T04:50:59.561442+0000","last_unstale":"2021-02-10T04:56:12.781703+0000","last_undegraded":"2021-02-10T04:56:12.781703+0000","last_fullsized":"2021-02-10T04:56:12.781703+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:09.121298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.121298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.21","version":"293'77","reported_seq":"838","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.780837+0000","last_change":"2021-02-10T04:52:28.062923+0000","last_active":"2021-02-10T04:56:12.780837+0000","last_peered":"2021-02-10T04:56:12.780837+0000","last_clean":"2021-02-10T04:56:12.780837+0000","last_became_active":"2021-02-10T04:52:28.062463+0000","last_became_peered":"2021-02-10T04:52:28.062463+0000","last_unstale":"2021-02-10T04:56:12.780837+0000","last_undegraded":"2021-02-10T04:56:12.780837+0000","last_fullsized":"2021-02-10T04:56:12.780837+0000","mapping_epoch":1143,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1144,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1886586,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":15163,"num_write":9,"num_write_kb":3316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":"164","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.727774+0000","last_change":"2021-02-10T04:52:32.124955+0000","last_active":"2021-02-10T04:56:12.727774+0000","last_peered":"2021-02-10T04:56:12.727774+0000","last_clean":"2021-02-10T04:56:12.727774+0000","last_became_active":"2021-02-10T04:51:36.867061+0000","last_became_peered":"2021-02-10T04:51:36.867061+0000","last_unstale":"2021-02-10T04:56:12.727774+0000","last_undegraded":"2021-02-10T04:56:12.727774+0000","last_fullsized":"2021-02-10T04:56:12.727774+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:32.124901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:08.271977+0000","last_clean_scrub_stamp":"2021-02-10T04:52:32.124901+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":"1207","reported_epoch":"1268","state":"active+clean","last_fresh":"2021-02-10T04:56:13.735706+0000","last_change":"2021-02-10T04:51:07.182775+0000","last_active":"2021-02-10T04:56:13.735706+0000","last_peered":"2021-02-10T04:56:13.735706+0000","last_clean":"2021-02-10T04:56:13.735706+0000","last_became_active":"2021-02-10T04:28:23.126368+0000","last_became_peered":"2021-02-10T04:28:23.126368+0000","last_unstale":"2021-02-10T04:56:13.735706+0000","last_undegraded":"2021-02-10T04:56:13.735706+0000","last_fullsized":"2021-02-10T04:56:13.735706+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.182678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.182678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.20","version":"1000'1071","reported_seq":"4612","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.727542+0000","last_change":"2021-02-10T04:53:03.915465+0000","last_active":"2021-02-10T04:56:12.727542+0000","last_peered":"2021-02-10T04:56:12.727542+0000","last_clean":"2021-02-10T04:56:12.727542+0000","last_became_active":"2021-02-10T04:52:31.088690+0000","last_became_peered":"2021-02-10T04:52:31.088690+0000","last_unstale":"2021-02-10T04:56:12.727542+0000","last_undegraded":"2021-02-10T04:56:12.727542+0000","last_fullsized":"2021-02-10T04:56:12.727542+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:03.915403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.915403+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16398893,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":"790","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.781160+0000","last_change":"2021-02-10T04:53:46.721599+0000","last_active":"2021-02-10T04:56:12.781160+0000","last_peered":"2021-02-10T04:56:12.781160+0000","last_clean":"2021-02-10T04:56:12.781160+0000","last_became_active":"2021-02-10T04:53:46.721340+0000","last_became_peered":"2021-02-10T04:53:46.721340+0000","last_unstale":"2021-02-10T04:56:12.781160+0000","last_undegraded":"2021-02-10T04:56:12.781160+0000","last_fullsized":"2021-02-10T04:56:12.781160+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:28.092288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:07.767250+0000","last_clean_scrub_stamp":"2021-02-10T04:52:28.092288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.25","version":"53'2","reported_seq":"1222","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.725592+0000","last_change":"2021-02-10T04:53:36.103026+0000","last_active":"2021-02-10T04:56:12.725592+0000","last_peered":"2021-02-10T04:56:12.725592+0000","last_clean":"2021-02-10T04:56:12.725592+0000","last_became_active":"2021-02-10T04:28:23.467770+0000","last_became_peered":"2021-02-10T04:28:23.467770+0000","last_unstale":"2021-02-10T04:56:12.725592+0000","last_undegraded":"2021-02-10T04:56:12.725592+0000","last_fullsized":"2021-02-10T04:56:12.725592+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:53:36.102928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:36.102928+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.23","version":"1002'1657","reported_seq":"5139","reported_epoch":"1268","state":"active+clean","last_fresh":"2021-02-10T04:56:13.734806+0000","last_change":"2021-02-10T04:52:31.093874+0000","last_active":"2021-02-10T04:56:13.734806+0000","last_peered":"2021-02-10T04:56:13.734806+0000","last_clean":"2021-02-10T04:56:13.734806+0000","last_became_active":"2021-02-10T04:52:31.092996+0000","last_became_peered":"2021-02-10T04:52:31.092996+0000","last_unstale":"2021-02-10T04:56:13.734806+0000","last_undegraded":"2021-02-10T04:56:13.734806+0000","last_fullsized":"2021-02-10T04:56:13.734806+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21396533,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":212,"num_read_kb":218837,"num_write":159,"num_write_kb":59399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":20479176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":"1314","reported_epoch":"1268","state":"active+clean","last_fresh":"2021-02-10T04:56:13.733823+0000","last_change":"2021-02-10T04:53:12.951612+0000","last_active":"2021-02-10T04:56:13.733823+0000","last_peered":"2021-02-10T04:56:13.733823+0000","last_clean":"2021-02-10T04:56:13.733823+0000","last_became_active":"2021-02-10T04:50:59.566548+0000","last_became_peered":"2021-02-10T04:50:59.566548+0000","last_unstale":"2021-02-10T04:56:13.733823+0000","last_undegraded":"2021-02-10T04:56:13.733823+0000","last_fullsized":"2021-02-10T04:56:13.733823+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.951528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.951528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.26","version":"53'1","reported_seq":"104","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.782789+0000","last_change":"2021-02-10T04:53:46.722636+0000","last_active":"2021-02-10T04:56:12.782789+0000","last_peered":"2021-02-10T04:56:12.782789+0000","last_clean":"2021-02-10T04:56:12.782789+0000","last_became_active":"2021-02-10T04:53:46.722227+0000","last_became_peered":"2021-02-10T04:53:46.722227+0000","last_unstale":"2021-02-10T04:56:12.782789+0000","last_undegraded":"2021-02-10T04:56:12.782789+0000","last_fullsized":"2021-02-10T04:56:12.782789+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:53:03.933910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.933910+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":423,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.22","version":"998'1009","reported_seq":"4629","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.723851+0000","last_change":"2021-02-10T04:52:31.086563+0000","last_active":"2021-02-10T04:56:12.723851+0000","last_peered":"2021-02-10T04:56:12.723851+0000","last_clean":"2021-02-10T04:56:12.723851+0000","last_became_active":"2021-02-10T04:52:31.085288+0000","last_became_peered":"2021-02-10T04:52:31.085288+0000","last_unstale":"2021-02-10T04:56:12.723851+0000","last_undegraded":"2021-02-10T04:56:12.723851+0000","last_fullsized":"2021-02-10T04:56:12.723851+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12355207,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":120,"num_read_kb":124702,"num_write":77,"num_write_kb":25052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4145029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":"1196","reported_epoch":"1268","state":"active+clean","last_fresh":"2021-02-10T04:56:13.734723+0000","last_change":"2021-02-10T04:53:10.958668+0000","last_active":"2021-02-10T04:56:13.734723+0000","last_peered":"2021-02-10T04:56:13.734723+0000","last_clean":"2021-02-10T04:56:13.734723+0000","last_became_active":"2021-02-10T04:45:49.552026+0000","last_became_peered":"2021-02-10T04:45:49.552026+0000","last_unstale":"2021-02-10T04:56:13.734723+0000","last_undegraded":"2021-02-10T04:56:13.734723+0000","last_fullsized":"2021-02-10T04:56:13.734723+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:10.958579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.958579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":"209","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.783275+0000","last_change":"2021-02-10T04:53:46.725941+0000","last_active":"2021-02-10T04:56:12.783275+0000","last_peered":"2021-02-10T04:56:12.783275+0000","last_clean":"2021-02-10T04:56:12.783275+0000","last_became_active":"2021-02-10T04:53:46.725587+0000","last_became_peered":"2021-02-10T04:53:46.725587+0000","last_unstale":"2021-02-10T04:56:12.783275+0000","last_undegraded":"2021-02-10T04:56:12.783275+0000","last_fullsized":"2021-02-10T04:56:12.783275+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:41.758684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.758684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":"1204","reported_epoch":"1267","state":"active+clean","last_fresh":"2021-02-10T04:56:12.780750+0000","last_change":"2021-02-10T04:55:39.599193+0000","last_active":"2021-02-10T04:56:12.780750+0000","last_peered":"2021-02-10T04:56:12.780750+0000","last_clean":"2021-02-10T04:56:12.780750+0000","last_became_active":"2021-02-10T04:55:39.598816+0000","last_became_peered":"2021-02-10T04:55:39.598816+0000","last_unstale":"2021-02-10T04:56:12.780750+0000","last_undegraded":"2021-02-10T04:56:12.780750+0000","last_fullsized":"2021-02-10T04:56:12.780750+0000","mapping_epoch":1248,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:11.139602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.139602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":"1222","reported_epoch":"1268","state":"active+clean","last_fresh":"2021-02-10T04:56:13.733149+0000","last_change":"2021-02-10T04:52:24.072005+0000","last_active":"2021-02-10T04:56:13.733149+0000","last_peered":"2021-02-10T04:56:13.733149+0000","last_clean":"2021-02-10T04:56:13.733149+0000","last_became_active":"2021-02-10T04:50:59.527945+0000","last_became_peered":"2021-02-10T04:50:59.527945+0000","last_unstale":"2021-02-10T04:56:13.733149+0000","last_undegraded":"2021-02-10T04:56:13.733149+0000","last_fullsized":"2021-02-10T04:56:13.733149+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:24.071915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_clean_scrub_stamp":"2021-02-10T04:52:24.071915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":972330808,"num_objects":390,"num_object_clones":337,"num_object_copies":1170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":390,"num_whiteouts":10,"num_read":4442,"num_read_kb":4559373,"num_write":3227,"num_write_kb":1247575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":611,"num_bytes_recovered":311961170,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2010624000,"data_stored":2589160348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34722,"ondisk_log_size":34722,"up":108,"acting":108,"num_store_stats":8},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":2368,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":18,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1139,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":5835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":126,"acting":126,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":13,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":697,"num_read_kb":692,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20,"ondisk_log_size":20,"up":39,"acting":39,"num_store_stats":8},{"poolid":4,"num_pg":22,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":2717,"num_read_kb":2190,"num_write":1592,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":17736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":844,"ondisk_log_size":844,"up":66,"acting":66,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35,"ondisk_log_size":35,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":126,"acting":126,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365170,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":191392,"kb_used_data":148912,"kb_used_omap":0,"kb_used_meta":42112,"kb_avail":93553760,"statfs":{"total":95995035648,"available":95799050240,"internally_reserved":0,"allocated":152485888,"data_stored":202211450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43122688},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:51:27 2021","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":1.2370000000000001,"15min":1.284},"min":{"1min":0.50800000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1220000000000001,"5min":34.299999999999997,"15min":67.277000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.5109999999999999,"15min":1.548},"min":{"1min":0.46300000000000002,"5min":0.438,"15min":0.438},"max":{"1min":2.8690000000000002,"5min":34.652999999999999,"15min":66.933999999999997},"last":0.81499999999999995}]},{"osd":1,"last update":"Wed Feb 10 04:55:42 2021","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.80700000000000005,"15min":0.95899999999999996},"min":{"1min":0.48799999999999999,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.409,"5min":2.254,"15min":10.140000000000001},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.85999999999999999,"15min":0.96699999999999997},"min":{"1min":0.63400000000000001,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.647,"5min":1.647,"15min":7.1520000000000001},"last":0.64600000000000002}]},{"osd":2,"last update":"Wed Feb 10 04:55:42 2021","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":1.256,"15min":1.1259999999999999},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.4319999999999999,"5min":16.896999999999998,"15min":16.896999999999998},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.2070000000000001,"15min":1.254},"min":{"1min":0.57999999999999996,"5min":0.496,"15min":0.496},"max":{"1min":4.0620000000000003,"5min":16.792000000000002,"15min":16.792000000000002},"last":1.0900000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:56:01 2021","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.0860000000000001,"15min":1.274},"min":{"1min":0.73199999999999998,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.7010000000000001,"5min":4.5890000000000004,"15min":4.5890000000000004},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.238,"5min":1.1759999999999999,"15min":1.3819999999999999},"min":{"1min":0.89800000000000002,"5min":0.629,"15min":0.629},"max":{"1min":1.8129999999999999,"5min":4.6609999999999996,"15min":4.6609999999999996},"last":1.0569999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:55:42 2021","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":1.1639999999999999,"15min":1.3899999999999999},"min":{"1min":0.60399999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.399,"5min":2.9729999999999999,"15min":10.781000000000001},"last":1.216},{"interface":"front","average":{"1min":1.03,"5min":1.1759999999999999,"15min":1.3460000000000001},"min":{"1min":0.74399999999999999,"5min":0.69699999999999995,"15min":0.69099999999999995},"max":{"1min":1.546,"5min":3.4049999999999998,"15min":10.326000000000001},"last":1.1240000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:55:42 2021","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.349,"15min":1.484},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":4.766,"5min":4.766,"15min":14.972},"last":1.181},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.4450000000000001,"15min":1.6899999999999999},"min":{"1min":0.81000000000000005,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.1549999999999998,"5min":4.4020000000000001,"15min":15.218999999999999},"last":1.335}]},{"osd":6,"last update":"Wed Feb 10 04:55:42 2021","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.375,"15min":1.7490000000000001},"min":{"1min":0.92700000000000005,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.5780000000000001,"5min":4.0380000000000003,"15min":16.050000000000001},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.4039999999999999,"15min":1.71},"min":{"1min":0.90300000000000002,"5min":0.876,"15min":0.876},"max":{"1min":1.964,"5min":3.9159999999999999,"15min":15.960000000000001},"last":1.4610000000000001}]}]},{"osd":6,"up_from":43,"seq":184683594103,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":408076,"kb_used_data":342020,"kb_used_omap":0,"kb_used_meta":66048,"kb_avail":93337076,"statfs":{"total":95995035648,"available":95577165824,"internally_reserved":0,"allocated":350228480,"data_stored":443183265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":67633152},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:56:16 2021","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.6599999999999999,"5min":1.6599999999999999,"15min":1.6599999999999999},"last":0.71599999999999997}]},{"osd":1,"last update":"Wed Feb 10 04:55:20 2021","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":1.0089999999999999,"15min":1.0680000000000001},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.395,"5min":13.18,"15min":13.18},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.77900000000000003,"5min":1.0309999999999999,"15min":1.1020000000000001},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.3440000000000001,"5min":11.041,"15min":11.041},"last":0.75}]},{"osd":2,"last update":"Wed Feb 10 04:55:20 2021","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":2.641,"15min":1.712},"min":{"1min":0.59799999999999998,"5min":0.503,"15min":0.503},"max":{"1min":1.865,"5min":163.81700000000001,"15min":163.81700000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.94299999999999995,"5min":2.6720000000000002,"15min":1.742},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.6619999999999999,"5min":164.56800000000001,"15min":164.56800000000001},"last":1.2350000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:56:12 2021","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.1160000000000001,"15min":1.1839999999999999},"min":{"1min":0.60599999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.835,"5min":1.835,"15min":1.835},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.165,"5min":1.109,"15min":1.119},"min":{"1min":0.81399999999999995,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.778,"5min":2.363,"15min":2.363},"last":0.90400000000000003}]},{"osd":4,"last update":"Wed Feb 10 04:55:20 2021","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.1850000000000001,"15min":1.369},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.1120000000000001,"5min":7.1459999999999999,"15min":7.1459999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.874,"5min":1.0549999999999999,"15min":1.105},"min":{"1min":0.64000000000000001,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.4770000000000001,"5min":11.997999999999999,"15min":11.997999999999999},"last":0.69199999999999995}]},{"osd":5,"last update":"Wed Feb 10 04:55:20 2021","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.196,"15min":1.595},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.5449999999999999,"5min":12.045,"15min":81.540000000000006},"last":0.88400000000000001},{"interface":"front","average":{"1min":1.246,"5min":1.47,"15min":1.8720000000000001},"min":{"1min":0.875,"5min":0.78700000000000003,"15min":0.78000000000000003},"max":{"1min":2.0910000000000002,"5min":13.401,"15min":81.492999999999995},"last":1.0740000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:55:47 2021","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.161,"15min":1.288},"min":{"1min":0.72899999999999998,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":2.073,"5min":4.0919999999999996,"15min":5.1820000000000004},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.421,"5min":1.4159999999999999,"15min":1.595},"min":{"1min":1.016,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":2.2970000000000002,"5min":5.4109999999999996,"15min":13.315},"last":1.413}]}]},{"osd":1,"up_from":13,"seq":55834575240,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":261340,"kb_used_data":205996,"kb_used_omap":0,"kb_used_meta":54976,"kb_avail":93483812,"statfs":{"total":95995035648,"available":95727423488,"internally_reserved":0,"allocated":210939904,"data_stored":265671506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":56295424},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:56:14 2021","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.86099999999999999}]},{"osd":2,"last update":"Wed Feb 10 04:56:14 2021","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":4.3220000000000001,"15min":2.6190000000000002},"min":{"1min":0.53300000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.266,"5min":162.61199999999999,"15min":162.61199999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.093,"5min":4.4290000000000003,"15min":2.6949999999999998},"min":{"1min":0.53100000000000003,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":5.8860000000000001,"5min":162.762,"15min":162.762},"last":0.52200000000000002}]},{"osd":3,"last update":"Wed Feb 10 04:56:04 2021","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.77100000000000002,"15min":0.90300000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.387,"5min":3.077,"15min":3.077},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.86699999999999999,"15min":0.98999999999999999},"min":{"1min":0.57099999999999995,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.052,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.74099999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:55:56 2021","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.0309999999999999,"15min":1.135},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":6.1479999999999997,"5min":6.1479999999999997,"15min":6.1479999999999997},"last":0.70199999999999996},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.071,"15min":1.2749999999999999},"min":{"1min":0.68400000000000005,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.6879999999999999,"5min":2.5270000000000001,"15min":29.547999999999998},"last":0.91600000000000004}]},{"osd":5,"last update":"Wed Feb 10 04:56:04 2021","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.0860000000000001,"15min":1.161},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.56599999999999995},"max":{"1min":1.581,"5min":2.7959999999999998,"15min":5.0300000000000002},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.224,"15min":1.4650000000000001},"min":{"1min":0.80900000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":6.032,"5min":6.032,"15min":28.678000000000001},"last":0.89600000000000002}]},{"osd":6,"last update":"Wed Feb 10 04:56:14 2021","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.4039999999999999,"15min":1.512},"min":{"1min":0.80800000000000005,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.5960000000000001,"5min":30.997,"15min":30.997},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.4239999999999999,"15min":1.468},"min":{"1min":1.052,"5min":0.64200000000000002,"15min":0.60499999999999998},"max":{"1min":1.7070000000000001,"5min":31.138000000000002,"15min":31.138000000000002},"last":1.0189999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:55:27 2021","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.216,"15min":1.488},"min":{"1min":0.72299999999999998,"5min":0.628,"15min":0.628},"max":{"1min":6.2629999999999999,"5min":6.2629999999999999,"15min":28.573},"last":1.329},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.232,"15min":1.5169999999999999},"min":{"1min":0.89200000000000002,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.895,"5min":1.895,"15min":29.408999999999999},"last":1.2789999999999999}]}]},{"osd":0,"up_from":1227,"seq":5269924872206,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":348872,"kb_used_data":338200,"kb_used_omap":306,"kb_used_meta":9997,"kb_avail":93396280,"statfs":{"total":95995035648,"available":95637790720,"internally_reserved":0,"allocated":346316800,"data_stored":471300514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":313579,"internal_metadata":10237717},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 04:56:14 2021","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.83699999999999997}]},{"osd":2,"last update":"Wed Feb 10 04:56:14 2021","interfaces":[{"interface":"back","average":{"1min":3.456,"5min":3.456,"15min":3.456},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":59.341999999999999,"5min":59.341999999999999,"15min":59.341999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":3.4660000000000002,"5min":3.4660000000000002,"15min":3.4660000000000002},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":59.405999999999999,"5min":59.405999999999999,"15min":59.405999999999999},"last":0.73699999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:56:14 2021","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.996,"5min":0.996,"15min":0.996},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.669,"5min":1.669,"15min":1.669},"last":1.135}]},{"osd":4,"last update":"Wed Feb 10 04:56:14 2021","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":1.353}]},{"osd":5,"last update":"Wed Feb 10 04:56:14 2021","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.24,"15min":1.24},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.6950000000000001,"5min":1.6950000000000001,"15min":1.6950000000000001},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":1.1839999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:56:14 2021","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.171,"15min":1.171},"min":{"1min":1.018,"5min":1.018,"15min":1.018},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.208,"5min":1.208,"15min":1.208},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.351,"5min":2.351,"15min":2.351},"last":1.278}]},{"osd":7,"last update":"Wed Feb 10 04:56:14 2021","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"min":{"1min":1.02,"5min":1.02,"15min":1.02},"max":{"1min":1.5800000000000001,"5min":1.5800000000000001,"15min":1.5800000000000001},"last":1.02},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":1.643,"5min":1.643,"15min":1.643},"last":1.2050000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411718,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":354196,"kb_used_data":318180,"kb_used_omap":0,"kb_used_meta":35648,"kb_avail":93390956,"statfs":{"total":95995035648,"available":95632338944,"internally_reserved":0,"allocated":325816320,"data_stored":374699662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36503552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:56:14 2021","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.2789999999999999,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":0.82299999999999995}]},{"osd":1,"last update":"Wed Feb 10 04:55:55 2021","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":0.88200000000000001,"15min":1},"min":{"1min":0.48099999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":17.041,"5min":17.041,"15min":17.041},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.8999999999999999,"5min":0.91300000000000003,"15min":0.94899999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":21.172000000000001,"5min":21.172000000000001,"15min":21.172000000000001},"last":0.74399999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:56:14 2021","interfaces":[{"interface":"back","average":{"1min":1.778,"5min":1.044,"15min":1.125},"min":{"1min":0.64700000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":18.904,"5min":18.904,"15min":18.904},"last":0.91600000000000004},{"interface":"front","average":{"1min":2.0219999999999998,"5min":1.2709999999999999,"15min":1.3109999999999999},"min":{"1min":0.86499999999999999,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":19.241,"5min":19.241,"15min":19.241},"last":1.034}]},{"osd":4,"last update":"Wed Feb 10 04:55:36 2021","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.1379999999999999,"15min":1.2849999999999999},"min":{"1min":0.71499999999999997,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":9.2029999999999994,"5min":9.2029999999999994,"15min":9.2029999999999994},"last":1.097},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.228,"15min":1.1699999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":3.403,"5min":24.373999999999999,"15min":24.373999999999999},"last":0.93799999999999994}]},{"osd":5,"last update":"Wed Feb 10 04:55:55 2021","interfaces":[{"interface":"back","average":{"1min":3.0630000000000002,"5min":1.575,"15min":1.7709999999999999},"min":{"1min":0.85399999999999998,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":39.911999999999999,"5min":39.911999999999999,"15min":39.911999999999999},"last":1.1619999999999999},{"interface":"front","average":{"1min":2.508,"5min":1.284,"15min":1.5109999999999999},"min":{"1min":0.67700000000000005,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":33.539999999999999,"5min":33.539999999999999,"15min":33.539999999999999},"last":0.76800000000000002}]},{"osd":6,"last update":"Wed Feb 10 04:55:23 2021","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.0629999999999999,"15min":1.419},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":3.1659999999999999,"5min":3.1659999999999999,"15min":20.007000000000001},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.9530000000000001,"5min":1.423,"15min":1.6519999999999999},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":16.135000000000002,"5min":16.135000000000002,"15min":19.417999999999999},"last":1.2549999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:55:36 2021","interfaces":[{"interface":"back","average":{"1min":1.637,"5min":1.637,"15min":1.681},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":9.3460000000000001,"5min":24.405000000000001,"15min":24.405000000000001},"last":1.3300000000000001},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.1040000000000001,"15min":1.278},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":3.6640000000000001,"5min":3.6640000000000001,"15min":4.9189999999999996},"last":1.018}]}]},{"osd":3,"up_from":1111,"seq":4771708665921,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":332788,"kb_used_data":300932,"kb_used_omap":386,"kb_used_meta":31101,"kb_avail":93412364,"statfs":{"total":95995035648,"available":95654260736,"internally_reserved":0,"allocated":308154368,"data_stored":409621309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":396036,"internal_metadata":31847676},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:56:14 2021","interfaces":[{"interface":"back","average":{"1min":1.98,"5min":1.98,"15min":1.98},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":22.869,"5min":22.869,"15min":22.869},"last":0.63400000000000001},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.6279999999999999,"15min":1.6279999999999999},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":13.587999999999999,"5min":13.587999999999999,"15min":13.587999999999999},"last":0.61299999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:56:06 2021","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.69199999999999995,"15min":0.66300000000000003},"min":{"1min":0.503,"5min":0.44,"15min":0.44},"max":{"1min":3.6629999999999998,"5min":3.6629999999999998,"15min":3.6629999999999998},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.80200000000000005,"15min":0.86299999999999999},"min":{"1min":0.53500000000000003,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":3.7050000000000001,"5min":3.7050000000000001,"15min":3.7050000000000001},"last":0.80800000000000005}]},{"osd":2,"last update":"Wed Feb 10 04:56:06 2021","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":1.413,"15min":1.2370000000000001},"min":{"1min":0.56699999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.173,"5min":31.100999999999999,"15min":31.100999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.569,"15min":1.27},"min":{"1min":0.71399999999999997,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":3.7509999999999999,"5min":31.052,"15min":31.052},"last":1.0900000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:56:06 2021","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":0.95899999999999996,"15min":0.96699999999999997},"min":{"1min":0.77100000000000002,"5min":0.628,"15min":0.628},"max":{"1min":2.6589999999999998,"5min":2.9430000000000001,"15min":2.9430000000000001},"last":1.181},{"interface":"front","average":{"1min":1.0980000000000001,"5min":0.98999999999999999,"15min":1.0820000000000001},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":3.21,"5min":3.21,"15min":3.21},"last":0.93000000000000005}]},{"osd":5,"last update":"Wed Feb 10 04:56:06 2021","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.966,"15min":2.4870000000000001},"min":{"1min":0.75800000000000001,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":3.3250000000000002,"5min":52.353000000000002,"15min":52.353000000000002},"last":1.337},{"interface":"front","average":{"1min":0.996,"5min":1.869,"15min":2.3180000000000001},"min":{"1min":0.83099999999999996,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.552,"5min":52.301000000000002,"15min":52.301000000000002},"last":1.032}]},{"osd":6,"last update":"Wed Feb 10 04:56:06 2021","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":2.5579999999999998,"15min":4.6699999999999999},"min":{"1min":0.871,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.466,"5min":64.376999999999995,"15min":64.376999999999995},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.353,"5min":2.6829999999999998,"15min":4.6840000000000002},"min":{"1min":1.0549999999999999,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":3.3889999999999998,"5min":64.438999999999993,"15min":64.438999999999993},"last":1.4219999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:56:06 2021","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.0629999999999999,"15min":1.105},"min":{"1min":0.94599999999999995,"5min":0.754,"15min":0.754},"max":{"1min":1.349,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":1.3},{"interface":"front","average":{"1min":1.28,"5min":1.1419999999999999,"15min":1.143},"min":{"1min":0.90300000000000002,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.9569999999999999,"5min":2.9569999999999999,"15min":2.9569999999999999},"last":1.274}]}]},{"osd":4,"up_from":31,"seq":133143986558,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":160596,"kb_used_data":120164,"kb_used_omap":0,"kb_used_meta":40064,"kb_avail":93584556,"statfs":{"total":95995035648,"available":95830585344,"internally_reserved":0,"allocated":123047936,"data_stored":155522669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41025536},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:56:16 2021","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":3.637,"5min":3.637,"15min":3.637},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":4.6829999999999998,"5min":4.6829999999999998,"15min":4.6829999999999998},"last":0.78600000000000003}]},{"osd":1,"last update":"Wed Feb 10 04:55:46 2021","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.76100000000000001,"15min":0.871},"min":{"1min":0.46500000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.341,"5min":3.5529999999999999,"15min":7.883},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.89700000000000002,"15min":1.0580000000000001},"min":{"1min":0.50700000000000001,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.1850000000000001,"5min":3.8090000000000002,"15min":6.5060000000000002},"last":0.91600000000000004}]},{"osd":2,"last update":"Wed Feb 10 04:55:46 2021","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":2.6829999999999998,"15min":1.702},"min":{"1min":0.627,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":7.0069999999999997,"5min":120.39,"15min":120.39},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.218,"5min":2.6419999999999999,"15min":1.6990000000000001},"min":{"1min":0.58999999999999997,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":7.1120000000000001,"5min":120.583,"15min":120.583},"last":0.93600000000000005}]},{"osd":3,"last update":"Wed Feb 10 04:56:04 2021","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.1299999999999999,"15min":1.248},"min":{"1min":0.877,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.9530000000000001,"5min":2.1429999999999998,"15min":2.1429999999999998},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.611,"15min":1.5569999999999999},"min":{"1min":0.91900000000000004,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":2.1080000000000001,"5min":26.802,"15min":26.802},"last":1.1240000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:55:59 2021","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.1080000000000001,"15min":1.478},"min":{"1min":0.624,"5min":0.495,"15min":0.495},"max":{"1min":1.6579999999999999,"5min":2.351,"15min":35.774000000000001},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.325,"5min":1.2090000000000001,"15min":1.47},"min":{"1min":0.78300000000000003,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":3.8410000000000002,"5min":4.319,"15min":35.606000000000002},"last":1.236}]},{"osd":6,"last update":"Wed Feb 10 04:56:04 2021","interfaces":[{"interface":"back","average":{"1min":1.615,"5min":1.625,"15min":2.3780000000000001},"min":{"1min":0.98499999999999999,"5min":0.86199999999999999,"15min":0.82599999999999996},"max":{"1min":3.8530000000000002,"5min":12.488,"15min":201.626},"last":1.349},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.504,"15min":2.3820000000000001},"min":{"1min":0.82099999999999995,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":4.7910000000000004,"5min":12.84,"15min":201.577},"last":1.262}]},{"osd":7,"last update":"Wed Feb 10 04:55:32 2021","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.431,"15min":1.4990000000000001},"min":{"1min":0.90300000000000002,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.351,"5min":2.5099999999999998,"15min":5.8920000000000003},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.105,"5min":1.2410000000000001,"15min":1.496},"min":{"1min":0.79100000000000004,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.819,"5min":2.2250000000000001,"15min":5.149},"last":1.159}]}]},{"osd":5,"up_from":37,"seq":158913790331,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":252128,"kb_used_data":205936,"kb_used_omap":0,"kb_used_meta":45824,"kb_avail":93493024,"statfs":{"total":95995035648,"available":95736856576,"internally_reserved":0,"allocated":210878464,"data_stored":275596198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":46923776},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:56:16 2021","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.486,"5min":1.486,"15min":1.486},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.59999999999999998}]},{"osd":1,"last update":"Wed Feb 10 04:55:58 2021","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":0.93999999999999995,"15min":0.94099999999999995},"min":{"1min":0.50600000000000001,"5min":0.435,"15min":0.435},"max":{"1min":4.242,"5min":6.391,"15min":6.391},"last":0.747},{"interface":"front","average":{"1min":1.74,"5min":1.2310000000000001,"15min":1.1950000000000001},"min":{"1min":0.626,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":13.794,"5min":13.794,"15min":13.794},"last":0.622}]},{"osd":2,"last update":"Wed Feb 10 04:55:58 2021","interfaces":[{"interface":"back","average":{"1min":7.226,"5min":2.831,"15min":1.7949999999999999},"min":{"1min":0.51400000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":63.270000000000003,"5min":63.270000000000003,"15min":63.270000000000003},"last":1.147},{"interface":"front","average":{"1min":7.3579999999999997,"5min":2.9630000000000001,"15min":1.831},"min":{"1min":0.68000000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":63.289000000000001,"5min":63.289000000000001,"15min":63.289000000000001},"last":0.86099999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:56:12 2021","interfaces":[{"interface":"back","average":{"1min":1.909,"5min":1.2909999999999999,"15min":1.3200000000000001},"min":{"1min":0.79300000000000004,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":13.755000000000001,"5min":13.755000000000001,"15min":13.755000000000001},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.202,"5min":1.242,"15min":1.2010000000000001},"min":{"1min":0.81899999999999995,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.7809999999999999,"5min":6.5129999999999999,"15min":6.5129999999999999},"last":0.999}]},{"osd":4,"last update":"Wed Feb 10 04:55:58 2021","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.304,"15min":1.3},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.57099999999999995},"max":{"1min":5.298,"5min":9.4890000000000008,"15min":9.4890000000000008},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.3400000000000001,"15min":1.3700000000000001},"min":{"1min":0.70899999999999996,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":4.8289999999999997,"5min":5.6180000000000003,"15min":5.6180000000000003},"last":1.393}]},{"osd":6,"last update":"Wed Feb 10 04:56:12 2021","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.4550000000000001,"15min":2.9689999999999999},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.65800000000000003},"max":{"1min":2.1989999999999998,"5min":4.9429999999999996,"15min":319.51999999999998},"last":1.29},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.4830000000000001,"15min":2.9140000000000001},"min":{"1min":0.82399999999999995,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":2.5659999999999998,"5min":6.8220000000000001,"15min":319.22300000000001},"last":1.4159999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:55:35 2021","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.4870000000000001,"15min":1.486},"min":{"1min":0.90300000000000002,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":5.4160000000000004,"5min":11.897,"15min":11.897},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.675,"5min":1.524,"15min":1.5720000000000001},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.82999999999999996},"max":{"1min":5.0579999999999998,"5min":6.0369999999999999,"15min":6.0369999999999999},"last":1.5009999999999999}]}]}]}} 2021-02-10T04:56:21.470 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pool set .rgw.root pgp_num 42 2021-02-10T04:56:21.610 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:56:21.685 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:56:21.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:21 smithi085 conmon[29319]: cluster 2021-02-10T04:56:20.513238+0000 mgr.y (mgr.14140) 1402 : cluster [DBG] 2021-02-10T04:56:21.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:21 smithi085 conmon[29319]: pgmap v2301: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 754 B/s rd, 0 op/s 2021-02-10T04:56:21.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:21 smithi085 conmon[29319]: cluster 2021-02-10T04:56:20.743202+0000 mon.a (mon.0) 2737 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:56:21.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:21 smithi085 conmon[29319]: audit 2021-02-10T04:56:20.751308+0000 mon.a (mon.0) 2738 : audit [INF] from='client.? 172.21.15.85:0/2063679820' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:56:21.693 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:21 smithi085 conmon[29319]: cluster 2021-02-10T04:56:20.751364+0000 mon.a (mon.0) 2739 : cluster [DBG] osdmap e1271: 8 total, 8 up, 8 in 2021-02-10T04:56:21.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:21 smithi085 conmon[35961]: cluster 2021-02-10T04:56:20. 2021-02-10T04:56:21.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:21 smithi085 conmon[35961]: 513238+0000 mgr.y (mgr.14140) 1402 : cluster [DBG] pgmap v2301: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 754 B/s rd, 0 op/s 2021-02-10T04:56:21.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:21 smithi085 conmon[35961]: cluster 2021-02-10T04:56:21.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:21 smithi085 conmon[35961]: 2021-02-10T04:56:20.743202+0000 mon.a (mon. 2021-02-10T04:56:21.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:21 smithi085 conmon[35961]: 0) 2737 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:56:21.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:21 smithi085 conmon[35961]: audit 2021-02-10T04:56:21.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:21 smithi085 conmon[35961]: 2021-02-10T04:56:20.751308 2021-02-10T04:56:21.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:21 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2738 : audit 2021-02-10T04:56:21.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:21 smithi085 conmon[35961]: [INF] from='client.? 172.21.15.85:0/2063679820' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:56:21.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:21 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:56:21.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:21 smithi085 conmon[35961]: 10T04:56:20.751364+0000 mon.a 2021-02-10T04:56:21.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:21 smithi085 conmon[35961]: (mon.0) 2739 : cluster [DBG] 2021-02-10T04:56:21.694 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:21 smithi085 conmon[35961]: osdmap e1271: 8 total, 8 up, 8 in 2021-02-10T04:56:21.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:21 smithi164 conmon[29288]: cluster 2021-02-10T04:56:20.513238+0000 mgr.y 2021-02-10T04:56:21.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:21 smithi164 conmon[29288]: (mgr.14140) 1402 : cluster [DBG] pgmap v2301: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 754 B/s rd, 0 op/s 2021-02-10T04:56:21.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:21 smithi164 conmon[29288]: cluster 2021-02-10T04:56:20.743202+0000 mon.a (mon.0) 2737 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:56:21.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:21 smithi164 conmon[29288]: audit 2021-02-10T04:56:20.751308+0000 2021-02-10T04:56:21.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:21 smithi164 conmon[29288]: mon.a (mon.0) 2738 : audit [INF] from='client.? 172.21.15.85:0/2063679820' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:56:21.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:21 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:56:21.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:21 smithi164 conmon[29288]: T04:56:20.751364+0000 mon.a (mon.0) 2739 2021-02-10T04:56:21.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:21 smithi164 conmon[29288]: : cluster [DBG] osdmap e1271: 8 total, 8 up, 8 in 2021-02-10T04:56:23.639 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:56:23 smithi085 conmon[49106]: debug 2021-02-10T04:56:23.381+0000 7f778ce21700 -1 osd.2 pg_epoch: 1271 pg[5.8( empty local-lis/les=1227/1228 n=0 ec=57/57 lis/c=1227/1227 les/c/f=1228/1228/0 sis=1227) [2,0,3] r=0 lpr=1227 crt=0'0 mlcod 0'0 active+clean] scrubber pg(5.8) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x564aa604adc0) 2021-02-10T04:56:23.640 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:23 smithi085 conmon[29319]: cluster 2021-02-10T04:56:22.514002+0000 mgr.y 2021-02-10T04:56:23.640 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:23 smithi085 conmon[29319]: (mgr.14140) 1403 : cluster [DBG] pgmap v2303: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:56:23.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:23 smithi085 conmon[35961]: cluster 2021-02-10T04:56:22.514002+0000 mgr.y ( 2021-02-10T04:56:23.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:23 smithi085 conmon[35961]: mgr.14140) 1403 : cluster [DBG] pgmap v2303: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:56:23.735 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset nodeep-scrub 2021-02-10T04:56:23.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:23 smithi164 conmon[29288]: cluster 2021-02-10T04:56:22. 2021-02-10T04:56:23.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:23 smithi164 conmon[29288]: 514002+0000 mgr.y (mgr.14140) 1403 : cluster [DBG] pgmap v2303: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:56:23.991 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:56:24.307 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:56:24.330 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:56:24.349 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:56:24.373 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:56:24.391 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:56:24.410 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:56:24.433 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:56:24.454 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:56:24.455 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:56:24.455 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:56:24.455 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:56:24.456 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:56:24.456 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:56:24.456 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:56:24.456 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:56:24.456 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:56:24.585 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:24 smithi164 conmon[29288]: audit 2021-02-10T04:56:24.285993+0000 mon.a (mon.0) 2740 : audit [DBG] from='client.? 172.21.15.85:0/197203781' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:56:24.585 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:24 smithi164 conmon[29288]: audit 2021-02-10T04:56:24.314968+0000 mon.a (mon.0) 2741 : audit [INF] from='client.? 172.21.15.85:0/2121973233' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "42"}]: dispatch 2021-02-10T04:56:24.592 INFO:teuthology.orchestra.run.smithi085.stderr:set pool 2 pgp_num to 42 2021-02-10T04:56:24.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:24 smithi085 conmon[29319]: audit 2021-02-10T04:56:24. 2021-02-10T04:56:24.618 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:24 smithi085 conmon[29319]: 285993+0000 mon.a (mon.0) 2740 : audit [DBG] from='client.? 172.21.15.85:0/197203781' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:56:24.619 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:24 smithi085 conmon[29319]: audit 2021-02-10T04:56:24.314968+0000 mon.a (mon.0) 2741 : audit [INF] from='client.? 172.21.15.85:0/2121973233' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "42"}]: dispatch 2021-02-10T04:56:24.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:24 smithi085 conmon[35961]: audit 2021-02-10T04:56:24.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:24 smithi085 conmon[35961]: 2021-02-10T04:56:24 2021-02-10T04:56:24.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:24 smithi085 conmon[35961]: .285993+0000 mon.a (mon.0) 2740 : audit [DBG] from='client.? 172.21.15.85:0/197203781' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:56:24.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:24 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:56:24.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:24 smithi085 conmon[35961]: 02-10T04:56:24.314968+0000 mon.a ( 2021-02-10T04:56:24.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:24 smithi085 conmon[35961]: mon.0) 2741 : audit [INF] 2021-02-10T04:56:24.619 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:24 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/2121973233' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "42"}]: dispatch 2021-02-10T04:56:25.090 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:56:25.104 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:56:25.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:25 smithi164 conmon[29288]: cluster 2021-02-10T04:56:24.514669+0000 mgr.y (mgr.14140) 1404 : cluster [DBG] 2021-02-10T04:56:25.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:25 smithi164 conmon[29288]: pgmap v2304: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:56:25.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:25 smithi164 conmon[29288]: audit 2021-02-10T04:56:24.588652+0000 mon.a (mon.0) 2742 : audit [INF] from='client.? 172.21.15.85:0/2121973233' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "42"}]': finished 2021-02-10T04:56:25.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:25 smithi164 conmon[29288]: cluster 2021-02-10T04:56:24.588753+0000 mon.a (mon.0) 2743 : cluster [DBG] osdmap e1272: 8 total, 8 up, 8 in 2021-02-10T04:56:25.914 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:25 smithi085 conmon[29319]: cluster 2021-02-10T04:56:24.514669+0000 mgr.y (mgr.14140) 1404 : cluster [DBG] pgmap v2304: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:56:25.914 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:25 smithi085 conmon[29319]: audit 2021-02-10T04:56:24.588652+0000 mon.a (mon.0) 2742 : audit [INF] from='client.? 172.21.15.85:0/2121973233' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "42"}]': finished 2021-02-10T04:56:25.914 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:25 smithi085 conmon[29319]: cluster 2021-02-10T04:56:24.588753+0000 mon.a (mon.0) 2743 : cluster [DBG] osdmap e1272: 8 total, 8 up, 8 in 2021-02-10T04:56:25.914 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:25 smithi085 conmon[35961]: cluster 2021-02-10T04:56:24.514669+0000 mgr.y (mgr.14140) 1404 : cluster [DBG] pgmap v2304: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:56:25.915 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:25 smithi085 conmon[35961]: audit 2021-02-10T04:56:24.588652+0000 mon.a (mon.0) 2742 : audit [INF] from='client.? 172.21.15.85:0/2121973233' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "42"}]': finished 2021-02-10T04:56:25.915 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:25 smithi085 conmon[35961]: cluster 2021-02-10T04:56:24.588753+0000 mon.a (mon.0) 2743 : cluster [DBG] osdmap e1272: 8 total, 8 up, 8 in 2021-02-10T04:56:26.588 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:56:26 smithi164 conmon[37263]: debug 2021-02-10T04:56:26.170+0000 7f1c6b9eb700 -1 osd.5 pg_epoch: 1272 pg[6.a( empty local-lis/les=1227/1228 n=0 ec=59/59 lis/c=1227/1227 les/c/f=1228/1228/0 sis=1227) [5,6,0] r=0 lpr=1227 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x556650634000) 2021-02-10T04:56:26.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:26 smithi164 conmon[29288]: cephadm 2021-02-10T04:56:25.590581+0000 mgr.y (mgr.14140) 1405 : cephadm 2021-02-10T04:56:26.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:26 smithi164 conmon[29288]: [INF] Refreshing smithi085 facts 2021-02-10T04:56:26.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:26 smithi164 conmon[29288]: audit 2021-02-10T04:56:25.590959+0000 mon.a (mon.0) 2744 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:56:26.922 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:26 smithi085 conmon[35961]: cephadm 2021-02-10T04:56:25.590581 2021-02-10T04:56:26.923 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:26 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1405 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:56:26.923 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:26 smithi085 conmon[35961]: audit 2021-02-10T04:56:25.590959+0000 mon.a (mon.0) 2744 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:56:26.923 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:26 smithi085 conmon[29319]: cephadm 2021-02-10T04:56:25.590581+0000 2021-02-10T04:56:26.923 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:26 smithi085 conmon[29319]: mgr.y (mgr.14140) 1405 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:56:26.923 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:26 smithi085 conmon[29319]: audit 2021-02-10T04:56:25.590959+0000 mon.a (mon.0) 2744 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "osd", "key": "container_image"}]: dispatch 2021-02-10T04:56:27.597 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is unset 2021-02-10T04:56:27.864 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:27 smithi085 conmon[35961]: cluster 2021-02-10T04:56:26.515138+0000 mgr.y (mgr.14140 2021-02-10T04:56:27.864 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:27 smithi085 conmon[35961]: ) 1406 : cluster [DBG] pgmap v2306: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 756 B/s rd, 0 op/s 2021-02-10T04:56:27.864 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:27 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:56:27.864 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:27 smithi085 conmon[35961]: T04:56:27.461141+0000 mon.a (mon.0) 2745 : audit [INF] from='client.? 172.21.15.85:0/187887102' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:56:27.865 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:27 smithi085 conmon[29319]: cluster 2021-02-10T04:56:26.515138+0000 2021-02-10T04:56:27.865 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:27 smithi085 conmon[29319]: mgr.y (mgr.14140) 1406 : cluster [DBG] pgmap v2306: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 756 B/s rd, 0 op/s 2021-02-10T04:56:27.865 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:27 smithi085 conmon[29319]: audit 2021-02-10T04:56:27.461141+0000 mon.a (mon.0) 2745 : audit [INF] from='client.? 172.21.15.85:0/187887102' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:56:27.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:27 smithi164 conmon[29288]: cluster 2021-02-10T04:56:26. 2021-02-10T04:56:27.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:27 smithi164 conmon[29288]: 515138+0000 mgr.y (mgr.14140) 1406 : cluster [DBG] pgmap v2306: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 756 B/s rd, 0 op/s 2021-02-10T04:56:27.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:27 smithi164 conmon[29288]: audit 2021-02-10T04:56:27. 2021-02-10T04:56:27.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:27 smithi164 conmon[29288]: 461141+0000 mon.a (mon.0) 2745 : audit [INF] from='client.? 172.21.15.85:0/187887102' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:56:27.934 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.7 dump_blocked_ops 2021-02-10T04:56:27.936 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:56:28.155 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:56:28.217 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:56:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:28 smithi164 conmon[29288]: cluster 2021-02-10T04:56:27.587765+0000 mon.a (mon.0) 2746 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:56:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:28 smithi164 conmon[29288]: cluster 2021-02-10T04:56:27.587791+0000 mon.a (mon.0) 2747 : cluster [INF] Cluster is now healthy 2021-02-10T04:56:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:28 smithi164 conmon[29288]: audit 2021-02-10T04:56:27.591973+0000 mon.a (mon.0) 2748 : audit [INF] from='client.? 172.21.15.85:0/187887102' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:56:28.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:28 smithi164 conmon[29288]: cluster 2021-02-10T04:56:27.592185+0000 mon.a (mon.0) 2749 : cluster [DBG] osdmap e1273: 8 total, 8 up, 8 in 2021-02-10T04:56:28.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:28 smithi085 conmon[29319]: cluster 2021-02-10T04:56:27.587765+0000 mon.a (mon.0) 2746 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:56:28.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:28 smithi085 conmon[29319]: cluster 2021-02-10T04:56:27.587791+0000 mon.a (mon.0) 2747 : cluster [INF] Cluster is now healthy 2021-02-10T04:56:28.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:28 smithi085 conmon[29319]: audit 2021-02-10T04:56:27.591973+0000 mon.a (mon.0) 2748 : audit [INF] from='client.? 172.21.15.85:0/187887102' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:56:28.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:28 smithi085 conmon[29319]: cluster 2021-02-10T04:56:27.592185+0000 mon.a (mon.0) 2749 : cluster [DBG] osdmap e1273: 8 total, 8 up, 8 in 2021-02-10T04:56:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:28 smithi085 conmon[35961]: cluster 2021-02-10T04:56:27.587765+0000 2021-02-10T04:56:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:28 smithi085 conmon[35961]: mon.a (mon.0) 2746 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:56:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:28 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:56:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:28 smithi085 conmon[35961]: :56:27.587791+0000 mon.a (mon.0) 2747 : cluster [INF] Cluster is now healthy 2021-02-10T04:56:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:28 smithi085 conmon[35961]: audit 2021-02-10T04:56:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:28 smithi085 conmon[35961]: 2021-02-10T04:56 2021-02-10T04:56:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:28 smithi085 conmon[35961]: :27.591973+0000 mon.a (mon.0) 2748 2021-02-10T04:56:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:28 smithi085 conmon[35961]: : audit [INF] from='client.? 172.21.15.85:0/187887102' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:56:28.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:28 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:56:28.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:28 smithi085 conmon[35961]: -02-10T04:56:27.592185+0000 2021-02-10T04:56:28.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:28 smithi085 conmon[35961]: mon.a (mon.0) 2749 : cluster [DBG] osdmap e1273: 8 total, 8 up, 8 in 2021-02-10T04:56:28.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:28 smithi085 conmon[35961]: 2021-02-10T04:56:29.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:56:29 smithi164 conmon[37263]: debug 2021-02-10T04:56:29.130+0000 7f1c6b9eb700 -1 osd.5 pg_epoch: 1273 pg[6.a( empty local-lis/les=1227/1228 n=0 ec=59/59 lis/c=1227/1227 les/c/f=1228/1228/0 sis=1227) [5,6,0] r=0 lpr=1227 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x556658e63ce0) 2021-02-10T04:56:29.394 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:56:29 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:56:29.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:29 smithi085 conmon[29319]: cluster 2021-02-10T04:56:28.515622+0000 mgr.y (mgr 2021-02-10T04:56:29.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:29 smithi085 conmon[29319]: .14140) 1407 : cluster [DBG] pgmap v2308: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:56:29.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:29 smithi085 conmon[35961]: cluster 2021-02-10T04:56:28.515622+0000 2021-02-10T04:56:29.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:29 smithi085 conmon[35961]: mgr.y (mgr.14140) 1407 : cluster [DBG] pgmap v2308: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:56:30.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:29 smithi164 conmon[29288]: cluster 2021-02-10T04:56:28.515622+0000 2021-02-10T04:56:30.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:29 smithi164 conmon[29288]: mgr.y (mgr.14140) 1407 : cluster [DBG] pgmap v2308: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:56:30.290 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:56:30.315 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:56:30.334 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:56:30.356 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:56:30.380 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:56:30.403 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:56:30.425 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:56:30.447 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:56:30.449 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:56:30.449 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:56:30.449 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:56:30.449 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:56:30.449 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:56:30.449 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:56:30.449 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:56:30.449 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:56:30.688 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set noscrub 2021-02-10T04:56:30.719 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:30 smithi085 conmon[29319]: audit 2021-02-10T04:56:29.375245+0000 mgr.y ( 2021-02-10T04:56:30.719 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:30 smithi085 conmon[29319]: mgr.14140) 1408 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:56:30.719 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:30 smithi085 conmon[29319]: audit 2021-02-10T04:56:30.271657+0000 mon.c (mon.1) 132 : audit [DBG] from='client.? 172.21.15.85:0/763783706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:56:30.719 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:30 smithi085 conmon[35961]: audit 2021-02-10T04:56:30.719 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:30 smithi085 conmon[35961]: 2021-02-10T04:56:30.719 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:30 smithi085 conmon[35961]: 2021 2021-02-10T04:56:30.719 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:30 smithi085 conmon[35961]: - 2021-02-10T04:56:30.719 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:30 smithi085 conmon[35961]: 02 2021-02-10T04:56:30.719 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:30 smithi085 conmon[35961]: - 2021-02-10T04:56:30.719 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:30 smithi085 conmon[35961]: 10 2021-02-10T04:56:30.720 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:30 smithi085 conmon[35961]: T 2021-02-10T04:56:30.720 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:30 smithi085 conmon[35961]: 04 2021-02-10T04:56:30.720 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:30 smithi085 conmon[35961]: : 2021-02-10T04:56:30.720 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:30 smithi085 conmon[35961]: 56:29.375245+0000 mgr.y (mgr.14140) 1408 : 2021-02-10T04:56:30.720 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:30 smithi085 conmon[35961]: audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:56:30.720 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:30 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:56:30.720 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:30 smithi085 conmon[35961]: 10T04:56:30.271657+0000 mon.c (mon.1) 132 : audit [DBG] from='client.? 172.21.15.85:0/763783706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:56:30.932 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:56:31.134 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:30 smithi164 conmon[29288]: audit 2021-02-10T04:56:31.135 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:30 smithi164 conmon[29288]: 2021-02-10T04:56:31.135 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:30 smithi164 conmon[29288]: 2021 2021-02-10T04:56:31.135 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:30 smithi164 conmon[29288]: - 2021-02-10T04:56:31.135 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:30 smithi164 conmon[29288]: 02 2021-02-10T04:56:31.135 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:30 smithi164 conmon[29288]: - 2021-02-10T04:56:31.135 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:30 smithi164 conmon[29288]: 10 2021-02-10T04:56:31.135 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:30 smithi164 conmon[29288]: T 2021-02-10T04:56:31.135 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:30 smithi164 conmon[29288]: 04 2021-02-10T04:56:31.135 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:30 smithi164 conmon[29288]: : 2021-02-10T04:56:31.135 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:30 smithi164 conmon[29288]: 56 2021-02-10T04:56:31.135 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:30 smithi164 conmon[29288]: : 2021-02-10T04:56:31.135 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:30 smithi164 conmon[29288]: 29 2021-02-10T04:56:31.136 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:30 smithi164 conmon[29288]: . 2021-02-10T04:56:31.136 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:30 smithi164 conmon[29288]: 375245 2021-02-10T04:56:31.136 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:30 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1408 : audit 2021-02-10T04:56:31.136 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:30 smithi164 conmon[29288]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:56:31.136 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:30 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:56:31.136 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:30 smithi164 conmon[29288]: 02-10T04:56:30. 2021-02-10T04:56:31.136 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:30 smithi164 conmon[29288]: 271657+0000 mon.c (mon.1) 2021-02-10T04:56:31.136 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:30 smithi164 conmon[29288]: 132 : audit [DBG] from='client.? 172.21.15.85:0/763783706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:56:31.189 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2021-02-10T04:56:31.189 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:56:31.189 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd dump -f json-pretty 2021-02-10T04:56:31.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:56:31 smithi164 conmon[37263]: debug 2021-02-10T04:56:31.133+0000 7f1c6b9eb700 -1 osd.5 pg_epoch: 1274 pg[6.a( empty local-lis/les=1227/1228 n=0 ec=59/59 lis/c=1227/1227 les/c/f=1228/1228/0 sis=1227) [5,6,0] r=0 lpr=1227 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x5566571c1340) 2021-02-10T04:56:31.433 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:56:31.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:31 smithi085 conmon[29319]: cluster 2021-02-10T04:56:30.516154+0000 mgr.y (mgr.14140) 1409 : cluster [DBG] pgmap v2309: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:56:31.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:31 smithi085 conmon[29319]: cluster 2021-02-10T04:56:30.643521+0000 osd.6 (osd.6) 188 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:56:31.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:31 smithi085 conmon[29319]: cluster 2021-02-10T04:56:30.647437+0000 osd.6 (osd.6) 189 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:56:31.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:31 smithi085 conmon[29319]: cluster 2021-02-10T04:56:30.665380+0000 mon.a (mon.0) 2750 : cluster [DBG] osdmap e1274: 8 total, 8 up, 8 in 2021-02-10T04:56:31.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:31 smithi085 conmon[35961]: cluster 2021-02-10T04:56:30.516154+0000 mgr.y (mgr.14140) 1409 : cluster [DBG] pgmap v2309: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:56:31.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:31 smithi085 conmon[35961]: cluster 2021-02-10T04:56:30.643521+0000 osd.6 (osd.6) 188 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:56:31.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:31 smithi085 conmon[35961]: cluster 2021-02-10T04:56:30.647437+0000 osd.6 (osd.6) 189 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:56:31.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:31 smithi085 conmon[35961]: cluster 2021-02-10T04:56:30.665380+0000 mon.a (mon.0) 2750 : cluster [DBG] osdmap e1274: 8 total, 8 up, 8 in 2021-02-10T04:56:32.073 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:31 smithi164 conmon[29288]: cluster 2021-02-10T04:56:30.516154+0000 mgr.y (mgr.14140) 1409 : cluster 2021-02-10T04:56:32.073 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:31 smithi164 conmon[29288]: [DBG] pgmap v2309: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:56:32.073 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:31 smithi164 conmon[29288]: cluster 2021-02-10T04:56:30.643521+0000 osd.6 (osd.6) 188 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:56:32.074 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:31 smithi164 conmon[29288]: cluster 2021-02-10T04:56:30.647437+0000 2021-02-10T04:56:32.074 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:31 smithi164 conmon[29288]: osd.6 (osd.6) 189 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:56:32.074 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:31 smithi164 conmon[29288]: cluster 2021-02-10T04:56:30 2021-02-10T04:56:32.074 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:31 smithi164 conmon[29288]: .665380+0000 mon.a (mon.0) 2750 : cluster [DBG] osdmap e1274: 8 total, 8 up, 8 in 2021-02-10T04:56:32.441 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:56:32 smithi085 conmon[45443]: debug 2021-02-10T04:56:32.199+0000 7f71254f7700 -1 osd.1 pg_epoch: 1274 pg[5.1d( empty local-lis/les=1227/1228 n=0 ec=57/57 lis/c=1227/1227 les/c/f=1228/1228/0 sis=1227) [1,4,0] r=0 lpr=1227 crt=0'0 mlcod 0'0 active+clean] scrubber pg(5.1d) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x55dbcdc08b00) 2021-02-10T04:56:33.307 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:56:33.546 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:56:33.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:33 smithi085 conmon[35961]: cluster 2021-02-10T04:56:32.516636+0000 mgr.y (mgr.14140) 1410 : cluster 2021-02-10T04:56:33.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:33 smithi085 conmon[35961]: [DBG] pgmap v2311: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 645 B/s rd, 0 op/s 2021-02-10T04:56:33.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:33 smithi085 conmon[35961]: cluster 2021-02-10T04:56:33.578206+0000 mon.a (mon.0) 2751 : cluster [DBG] osdmap e1275: 8 total, 8 up, 8 in 2021-02-10T04:56:33.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:33 smithi085 conmon[29319]: cluster 2021-02-10T04:56:32.516636+0000 mgr.y (mgr. 2021-02-10T04:56:33.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:33 smithi085 conmon[29319]: 14140) 1410 : cluster [DBG] pgmap v2311: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 645 B/s rd, 0 op/s 2021-02-10T04:56:33.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:33 smithi085 conmon[29319]: cluster 2021-02-10T04:56:33.578206+0000 mon.a (mon.0) 2751 : cluster [DBG] osdmap e1275: 8 total, 8 up, 8 in 2021-02-10T04:56:34.038 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:33 smithi164 conmon[29288]: cluster 2021-02-10T04:56:32.516636+0000 mgr.y (mgr.14140) 1410 2021-02-10T04:56:34.038 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:33 smithi164 conmon[29288]: : cluster [DBG] pgmap v2311: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 645 B/s rd, 0 op/s 2021-02-10T04:56:34.038 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:33 smithi164 conmon[29288]: cluster 2021-02-10T04:56:33.578206+0000 mon.a (mon.0) 2751 : cluster [DBG] osdmap e1275: 8 total, 8 up, 8 in 2021-02-10T04:56:34.257 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:56:34.258 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:56:34.258 INFO:teuthology.orchestra.run.smithi085.stdout: "epoch": 1275, 2021-02-10T04:56:34.258 INFO:teuthology.orchestra.run.smithi085.stdout: "fsid": "42224fce-6b57-11eb-8feb-001a4aab830c", 2021-02-10T04:56:34.258 INFO:teuthology.orchestra.run.smithi085.stdout: "created": "2021-02-10T04:20:57.933279+0000", 2021-02-10T04:56:34.258 INFO:teuthology.orchestra.run.smithi085.stdout: "modified": "2021-02-10T04:56:33.572989+0000", 2021-02-10T04:56:34.258 INFO:teuthology.orchestra.run.smithi085.stdout: "last_up_change": "2021-02-10T04:55:13.547923+0000", 2021-02-10T04:56:34.258 INFO:teuthology.orchestra.run.smithi085.stdout: "last_in_change": "2021-02-10T04:51:35.701776+0000", 2021-02-10T04:56:34.258 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T04:56:34.259 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_num": 5799936, 2021-02-10T04:56:34.259 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_set": [ 2021-02-10T04:56:34.259 INFO:teuthology.orchestra.run.smithi085.stdout: "pglog_hardlimit", 2021-02-10T04:56:34.259 INFO:teuthology.orchestra.run.smithi085.stdout: "purged_snapdirs", 2021-02-10T04:56:34.259 INFO:teuthology.orchestra.run.smithi085.stdout: "recovery_deletes", 2021-02-10T04:56:34.259 INFO:teuthology.orchestra.run.smithi085.stdout: "sortbitwise" 2021-02-10T04:56:34.259 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:56:34.259 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_version": 18, 2021-02-10T04:56:34.259 INFO:teuthology.orchestra.run.smithi085.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T04:56:34.260 INFO:teuthology.orchestra.run.smithi085.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T04:56:34.260 INFO:teuthology.orchestra.run.smithi085.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T04:56:34.260 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_snapshot": "", 2021-02-10T04:56:34.260 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_max": 7, 2021-02-10T04:56:34.260 INFO:teuthology.orchestra.run.smithi085.stdout: "max_osd": 8, 2021-02-10T04:56:34.260 INFO:teuthology.orchestra.run.smithi085.stdout: "require_min_compat_client": "luminous", 2021-02-10T04:56:34.260 INFO:teuthology.orchestra.run.smithi085.stdout: "min_compat_client": "luminous", 2021-02-10T04:56:34.260 INFO:teuthology.orchestra.run.smithi085.stdout: "require_osd_release": "quincy", 2021-02-10T04:56:34.260 INFO:teuthology.orchestra.run.smithi085.stdout: "pools": [ 2021-02-10T04:56:34.261 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.261 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 1, 2021-02-10T04:56:34.261 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "device_health_metrics", 2021-02-10T04:56:34.261 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:24:05.258028+0000", 2021-02-10T04:56:34.261 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:56:34.261 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:56:34.261 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:56:34.261 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:56:34.261 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:56:34.262 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:56:34.262 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:56:34.262 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:56:34.262 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:56:34.262 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:56:34.262 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:56:34.262 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:56:34.262 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 11, 2021-02-10T04:56:34.262 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 11, 2021-02-10T04:56:34.263 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 11, 2021-02-10T04:56:34.263 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 11, 2021-02-10T04:56:34.263 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 11, 2021-02-10T04:56:34.263 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:56:34.263 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:56:34.263 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:56:34.263 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:56:34.263 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:56:34.264 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:56:34.264 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:56:34.264 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.264 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "1266", 2021-02-10T04:56:34.264 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:56:34.264 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:56:34.264 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "110", 2021-02-10T04:56:34.264 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:56:34.264 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:56:34.265 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:56:34.265 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:56:34.265 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:56:34.265 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:56:34.265 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:56:34.265 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:56:34.265 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:56:34.265 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:56:34.265 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:56:34.266 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:56:34.266 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:56:34.266 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:56:34.266 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:56:34.266 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:56:34.266 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:56:34.266 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:56:34.266 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:56:34.266 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:56:34.267 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:56:34.267 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:56:34.267 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:56:34.267 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.267 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:56:34.267 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:56:34.267 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:56:34.267 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:56:34.267 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:56:34.268 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:56:34.268 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:56:34.268 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:56:34.268 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:56:34.268 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:56:34.268 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:56:34.268 INFO:teuthology.orchestra.run.smithi085.stdout: "options": { 2021-02-10T04:56:34.268 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_min": 1 2021-02-10T04:56:34.269 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.269 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:56:34.269 INFO:teuthology.orchestra.run.smithi085.stdout: "mgr_devicehealth": {} 2021-02-10T04:56:34.269 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.269 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.269 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.269 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 2, 2021-02-10T04:56:34.270 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": ".rgw.root", 2021-02-10T04:56:34.270 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:32.688554+0000", 2021-02-10T04:56:34.270 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:56:34.270 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:56:34.270 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:56:34.270 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:56:34.270 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:56:34.270 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:56:34.271 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:56:34.271 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:56:34.271 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:56:34.271 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:56:34.271 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:56:34.271 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:56:34.272 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 42, 2021-02-10T04:56:34.272 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 42, 2021-02-10T04:56:34.272 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 42, 2021-02-10T04:56:34.272 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 42, 2021-02-10T04:56:34.272 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 42, 2021-02-10T04:56:34.272 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:56:34.272 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:56:34.272 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:56:34.273 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:56:34.273 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:56:34.273 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:56:34.273 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:56:34.273 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.273 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "1272", 2021-02-10T04:56:34.273 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:56:34.273 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:56:34.273 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "89", 2021-02-10T04:56:34.274 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:56:34.274 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:56:34.274 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:56:34.274 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:56:34.274 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:56:34.274 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:56:34.274 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:56:34.274 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:56:34.274 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:56:34.275 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:56:34.275 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:56:34.275 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:56:34.275 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:56:34.275 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:56:34.275 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:56:34.275 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:56:34.275 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:56:34.275 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:56:34.276 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:56:34.276 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:56:34.276 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:56:34.276 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:56:34.276 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:56:34.276 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.276 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:56:34.276 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:56:34.276 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:56:34.277 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:56:34.277 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:56:34.277 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:56:34.277 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:56:34.277 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:56:34.277 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:56:34.277 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:56:34.277 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:56:34.277 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:56:34.278 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:56:34.278 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T04:56:34.278 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.278 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.278 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.278 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 3, 2021-02-10T04:56:34.278 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "iscsi", 2021-02-10T04:56:34.279 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:47.658613+0000", 2021-02-10T04:56:34.279 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:56:34.279 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:56:34.279 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:56:34.279 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:56:34.279 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:56:34.279 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:56:34.280 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:56:34.280 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:56:34.280 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:56:34.281 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:56:34.281 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:56:34.281 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:56:34.281 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 13, 2021-02-10T04:56:34.281 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 13, 2021-02-10T04:56:34.281 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 13, 2021-02-10T04:56:34.282 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 13, 2021-02-10T04:56:34.282 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 13, 2021-02-10T04:56:34.282 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:56:34.282 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:56:34.282 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:56:34.282 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:56:34.282 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:56:34.282 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:56:34.282 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:56:34.282 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.282 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "1211", 2021-02-10T04:56:34.283 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:56:34.283 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:56:34.283 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "1209", 2021-02-10T04:56:34.283 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:56:34.283 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:56:34.283 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:56:34.283 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:56:34.283 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:56:34.283 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:56:34.283 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:56:34.283 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:56:34.283 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:56:34.283 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:56:34.284 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:56:34.284 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:56:34.284 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:56:34.284 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:56:34.284 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:56:34.284 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:56:34.284 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:56:34.284 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:56:34.284 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:56:34.284 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:56:34.284 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:56:34.284 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:56:34.285 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:56:34.285 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.285 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:56:34.285 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:56:34.285 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:56:34.285 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:56:34.285 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:56:34.285 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:56:34.285 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:56:34.285 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:56:34.285 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:56:34.285 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:56:34.286 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:56:34.286 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:56:34.286 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:56:34.286 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd": {} 2021-02-10T04:56:34.286 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.286 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.286 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.286 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 4, 2021-02-10T04:56:34.286 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "zone.rgw.log", 2021-02-10T04:56:34.286 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:47.968727+0000", 2021-02-10T04:56:34.286 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:56:34.287 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:56:34.287 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:56:34.287 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:56:34.287 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:56:34.287 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:56:34.287 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:56:34.287 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:56:34.287 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:56:34.287 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:56:34.287 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:56:34.287 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:56:34.287 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 22, 2021-02-10T04:56:34.287 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 22, 2021-02-10T04:56:34.288 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 22, 2021-02-10T04:56:34.288 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 22, 2021-02-10T04:56:34.288 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 22, 2021-02-10T04:56:34.288 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:56:34.288 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "4.16", 2021-02-10T04:56:34.288 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 1256, 2021-02-10T04:56:34.288 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 1255, 2021-02-10T04:56:34.288 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 1255, 2021-02-10T04:56:34.288 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "1048'40", 2021-02-10T04:56:34.288 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "1049'40" 2021-02-10T04:56:34.288 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.288 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "1256", 2021-02-10T04:56:34.288 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:56:34.289 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "1256", 2021-02-10T04:56:34.289 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "1254", 2021-02-10T04:56:34.289 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:56:34.289 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:56:34.289 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:56:34.289 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:56:34.289 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:56:34.289 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:56:34.289 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:56:34.289 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:56:34.289 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:56:34.289 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:56:34.290 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:56:34.290 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:56:34.290 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:56:34.290 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:56:34.290 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:56:34.290 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:56:34.290 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:56:34.290 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:56:34.290 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:56:34.290 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:56:34.290 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:56:34.290 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:56:34.290 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:56:34.291 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.291 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:56:34.291 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:56:34.291 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:56:34.291 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:56:34.291 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:56:34.291 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:56:34.291 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:56:34.291 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:56:34.291 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:56:34.291 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:56:34.291 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:56:34.292 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:56:34.292 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:56:34.292 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T04:56:34.292 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.292 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.292 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.292 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 5, 2021-02-10T04:56:34.292 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "zone.rgw.control", 2021-02-10T04:56:34.292 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:49.278260+0000", 2021-02-10T04:56:34.292 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:56:34.292 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:56:34.292 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:56:34.292 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:56:34.293 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:56:34.293 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:56:34.293 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:56:34.293 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:56:34.293 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:56:34.293 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:56:34.293 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:56:34.293 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:56:34.293 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 32, 2021-02-10T04:56:34.293 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 32, 2021-02-10T04:56:34.293 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 32, 2021-02-10T04:56:34.293 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 32, 2021-02-10T04:56:34.294 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 32, 2021-02-10T04:56:34.294 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:56:34.294 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:56:34.294 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:56:34.294 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:56:34.294 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:56:34.294 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:56:34.294 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:56:34.294 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.294 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "59", 2021-02-10T04:56:34.294 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:56:34.294 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:56:34.294 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T04:56:34.295 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:56:34.295 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:56:34.295 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:56:34.295 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:56:34.295 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:56:34.295 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:56:34.295 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:56:34.295 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:56:34.295 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:56:34.295 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:56:34.295 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:56:34.296 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:56:34.296 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:56:34.296 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:56:34.296 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:56:34.296 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:56:34.296 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:56:34.296 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:56:34.296 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:56:34.296 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:56:34.296 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:56:34.296 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:56:34.296 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:56:34.296 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.296 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:56:34.297 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:56:34.297 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:56:34.297 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:56:34.297 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:56:34.297 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:56:34.297 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:56:34.297 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:56:34.297 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:56:34.297 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:56:34.297 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:56:34.297 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:56:34.297 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:56:34.298 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T04:56:34.298 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.298 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.298 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.298 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 6, 2021-02-10T04:56:34.298 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "zone.rgw.meta", 2021-02-10T04:56:34.298 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:51.792514+0000", 2021-02-10T04:56:34.298 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:56:34.298 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:56:34.298 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:56:34.298 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:56:34.298 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:56:34.298 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:56:34.299 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:56:34.299 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:56:34.299 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:56:34.299 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:56:34.299 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:56:34.299 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:56:34.299 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 42, 2021-02-10T04:56:34.299 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 42, 2021-02-10T04:56:34.299 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 42, 2021-02-10T04:56:34.299 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 42, 2021-02-10T04:56:34.299 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 42, 2021-02-10T04:56:34.300 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:56:34.300 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:56:34.300 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:56:34.300 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:56:34.300 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:56:34.300 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:56:34.300 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:56:34.300 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.300 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "1046", 2021-02-10T04:56:34.300 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:56:34.300 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:56:34.300 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "1044", 2021-02-10T04:56:34.300 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:56:34.300 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:56:34.301 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:56:34.301 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:56:34.301 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:56:34.301 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:56:34.301 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:56:34.301 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:56:34.301 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:56:34.301 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:56:34.301 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:56:34.301 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:56:34.301 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:56:34.301 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:56:34.302 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:56:34.302 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:56:34.302 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:56:34.302 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:56:34.302 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:56:34.302 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:56:34.302 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:56:34.302 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:56:34.302 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:56:34.302 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.302 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:56:34.302 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:56:34.303 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:56:34.303 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:56:34.303 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:56:34.303 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:56:34.303 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:56:34.303 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:56:34.303 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:56:34.303 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:56:34.303 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:56:34.303 INFO:teuthology.orchestra.run.smithi085.stdout: "options": { 2021-02-10T04:56:34.303 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_bias": 4, 2021-02-10T04:56:34.303 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_min": 8 2021-02-10T04:56:34.303 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.304 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:56:34.304 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T04:56:34.304 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.304 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.304 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.304 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 7, 2021-02-10T04:56:34.304 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "unique_pool_0", 2021-02-10T04:56:34.304 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:28:23.932391+0000", 2021-02-10T04:56:34.304 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 8193, 2021-02-10T04:56:34.304 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2021-02-10T04:56:34.304 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:56:34.304 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:56:34.304 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:56:34.305 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:56:34.305 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:56:34.305 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:56:34.305 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:56:34.305 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:56:34.305 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:56:34.305 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:56:34.305 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 36, 2021-02-10T04:56:34.305 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 36, 2021-02-10T04:56:34.305 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 36, 2021-02-10T04:56:34.305 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 36, 2021-02-10T04:56:34.305 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 36, 2021-02-10T04:56:34.305 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:56:34.305 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:56:34.306 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:56:34.306 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:56:34.306 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:56:34.306 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:56:34.306 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:56:34.306 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.306 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "1143", 2021-02-10T04:56:34.306 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:56:34.306 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:56:34.306 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "1141", 2021-02-10T04:56:34.306 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:56:34.306 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:56:34.306 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 888, 2021-02-10T04:56:34.307 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 994, 2021-02-10T04:56:34.307 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:56:34.307 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:56:34.307 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:56:34.307 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:56:34.307 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:56:34.307 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:56:34.307 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:56:34.307 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:56:34.307 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:56:34.307 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:56:34.307 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:56:34.307 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:56:34.307 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:56:34.308 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:56:34.308 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:56:34.308 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:56:34.308 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:56:34.308 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:56:34.308 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:56:34.308 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.308 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:56:34.308 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:56:34.308 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:56:34.308 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:56:34.308 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:56:34.308 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:56:34.309 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:56:34.309 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:56:34.309 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:56:34.309 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:56:34.309 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:56:34.309 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:56:34.309 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:56:34.309 INFO:teuthology.orchestra.run.smithi085.stdout: "rados": {} 2021-02-10T04:56:34.309 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.309 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.309 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:56:34.309 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T04:56:34.309 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.310 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 0, 2021-02-10T04:56:34.310 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "45d0bb5f-35fa-4596-964f-75abcee53bee", 2021-02-10T04:56:34.310 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:56:34.310 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:56:34.310 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:56:34.310 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:56:34.310 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 8, 2021-02-10T04:56:34.310 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 1146, 2021-02-10T04:56:34.310 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 1227, 2021-02-10T04:56:34.310 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1256, 2021-02-10T04:56:34.310 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 1147, 2021-02-10T04:56:34.310 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:56:34.310 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:56:34.311 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.311 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.311 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.311 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6801", 2021-02-10T04:56:34.311 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.311 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.311 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.311 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.311 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:56:34.311 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.311 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.311 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.311 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6802", 2021-02-10T04:56:34.312 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.312 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.312 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.312 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.312 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:56:34.312 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.312 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.312 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.312 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6804", 2021-02-10T04:56:34.312 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.312 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.312 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.312 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.312 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:56:34.313 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.313 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.313 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.313 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6803", 2021-02-10T04:56:34.313 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.313 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.313 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.313 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.313 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6801/7", 2021-02-10T04:56:34.313 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6802/7", 2021-02-10T04:56:34.313 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6804/7", 2021-02-10T04:56:34.313 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6803/7", 2021-02-10T04:56:34.313 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:56:34.314 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:56:34.314 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:56:34.314 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.314 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.314 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.314 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 1, 2021-02-10T04:56:34.314 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "3421a5ce-872e-4b52-adf7-6aaa595871a4", 2021-02-10T04:56:34.314 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:56:34.314 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:56:34.314 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:56:34.314 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 0.86566162109375, 2021-02-10T04:56:34.314 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:56:34.314 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:56:34.315 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 13, 2021-02-10T04:56:34.315 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1252, 2021-02-10T04:56:34.315 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:56:34.315 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:56:34.315 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:56:34.315 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.315 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.315 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.315 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6805", 2021-02-10T04:56:34.315 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.315 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.315 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.316 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.316 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:56:34.316 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.316 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.316 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.316 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6806", 2021-02-10T04:56:34.316 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.316 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.316 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.316 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.316 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:56:34.316 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.316 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.317 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.317 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6808", 2021-02-10T04:56:34.317 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.317 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.317 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.317 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.317 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:56:34.317 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.317 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.317 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.317 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6807", 2021-02-10T04:56:34.317 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.317 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.318 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.318 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.318 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6805/7", 2021-02-10T04:56:34.318 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6806/7", 2021-02-10T04:56:34.318 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6808/7", 2021-02-10T04:56:34.318 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6807/7", 2021-02-10T04:56:34.318 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:56:34.318 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:56:34.318 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:56:34.318 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.318 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.319 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.319 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 2, 2021-02-10T04:56:34.319 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "96e1952a-1042-4316-9fc9-86813e88b912", 2021-02-10T04:56:34.319 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:56:34.319 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:56:34.319 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:56:34.319 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:56:34.319 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:56:34.319 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:56:34.319 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 18, 2021-02-10T04:56:34.319 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1248, 2021-02-10T04:56:34.320 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:56:34.320 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:56:34.320 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:56:34.320 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.320 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.320 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.320 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6809", 2021-02-10T04:56:34.320 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.320 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.320 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.320 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.320 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:56:34.321 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.321 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.321 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.321 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6810", 2021-02-10T04:56:34.321 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.321 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.321 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.321 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.321 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:56:34.321 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.322 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.322 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.322 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6812", 2021-02-10T04:56:34.322 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.322 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.322 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.322 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.322 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:56:34.322 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.322 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.322 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.322 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6811", 2021-02-10T04:56:34.323 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.323 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.323 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.323 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.323 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6809/7", 2021-02-10T04:56:34.323 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6810/7", 2021-02-10T04:56:34.323 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6812/7", 2021-02-10T04:56:34.323 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6811/7", 2021-02-10T04:56:34.323 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:56:34.323 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:56:34.323 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:56:34.323 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.323 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.324 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.324 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 3, 2021-02-10T04:56:34.324 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "e019f49f-5dd4-4104-9ced-94c710616642", 2021-02-10T04:56:34.324 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:56:34.324 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:56:34.324 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:56:34.324 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:56:34.324 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 25, 2021-02-10T04:56:34.324 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 1094, 2021-02-10T04:56:34.324 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 1111, 2021-02-10T04:56:34.324 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1248, 2021-02-10T04:56:34.324 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 1095, 2021-02-10T04:56:34.324 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:56:34.325 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:56:34.325 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.325 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.325 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.325 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6813", 2021-02-10T04:56:34.325 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.325 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.325 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.325 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.325 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:56:34.325 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.325 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.325 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.325 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6814", 2021-02-10T04:56:34.326 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.326 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.326 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.326 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.326 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:56:34.326 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.326 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.326 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.326 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6816", 2021-02-10T04:56:34.326 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.326 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.326 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.327 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.327 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:56:34.327 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.327 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.327 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.327 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6815", 2021-02-10T04:56:34.327 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.327 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.327 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.327 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.327 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6813/7", 2021-02-10T04:56:34.327 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6814/7", 2021-02-10T04:56:34.327 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6816/7", 2021-02-10T04:56:34.328 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6815/7", 2021-02-10T04:56:34.328 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:56:34.328 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:56:34.328 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:56:34.328 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.328 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.328 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.328 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 4, 2021-02-10T04:56:34.328 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "91c8a80a-37b6-4c30-a32a-647b2e6d801f", 2021-02-10T04:56:34.328 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:56:34.328 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:56:34.328 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:56:34.328 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:56:34.329 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:56:34.329 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:56:34.329 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 31, 2021-02-10T04:56:34.329 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1244, 2021-02-10T04:56:34.329 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:56:34.329 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:56:34.329 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:56:34.329 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.329 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.329 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.329 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6800", 2021-02-10T04:56:34.329 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.329 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.330 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.330 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.330 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:56:34.330 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.330 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.330 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.330 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6801", 2021-02-10T04:56:34.330 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.330 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.330 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.330 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.330 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:56:34.331 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.331 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.331 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.331 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6803", 2021-02-10T04:56:34.331 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.331 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.331 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.331 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.331 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:56:34.331 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.331 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.331 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.331 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6802", 2021-02-10T04:56:34.332 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.332 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.332 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.332 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.332 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6800/7", 2021-02-10T04:56:34.332 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6801/7", 2021-02-10T04:56:34.332 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6803/7", 2021-02-10T04:56:34.332 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6802/7", 2021-02-10T04:56:34.332 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:56:34.332 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:56:34.332 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:56:34.332 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.332 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.332 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.333 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 5, 2021-02-10T04:56:34.333 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e", 2021-02-10T04:56:34.333 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:56:34.333 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:56:34.333 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:56:34.333 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:56:34.333 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:56:34.333 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:56:34.333 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 37, 2021-02-10T04:56:34.333 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1248, 2021-02-10T04:56:34.333 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:56:34.333 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:56:34.333 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:56:34.334 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.334 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.334 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.334 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6804", 2021-02-10T04:56:34.334 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.334 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.334 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.334 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.334 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:56:34.334 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.334 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.334 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.334 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6805", 2021-02-10T04:56:34.335 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.335 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.335 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.335 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.335 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:56:34.335 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.335 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.335 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.335 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6807", 2021-02-10T04:56:34.335 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.335 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.335 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.335 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.336 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:56:34.336 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.336 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.336 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.336 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6806", 2021-02-10T04:56:34.336 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.336 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.336 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.336 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.336 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6804/7", 2021-02-10T04:56:34.336 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6805/7", 2021-02-10T04:56:34.336 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6807/7", 2021-02-10T04:56:34.336 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6806/7", 2021-02-10T04:56:34.337 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:56:34.337 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:56:34.337 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:56:34.337 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.337 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.337 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.337 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 6, 2021-02-10T04:56:34.337 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "adc40430-3eeb-4003-a824-343021243077", 2021-02-10T04:56:34.337 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:56:34.337 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:56:34.337 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:56:34.337 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:56:34.337 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:56:34.337 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:56:34.338 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 43, 2021-02-10T04:56:34.338 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1248, 2021-02-10T04:56:34.338 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:56:34.338 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:56:34.338 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:56:34.338 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.338 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.338 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.338 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6808", 2021-02-10T04:56:34.338 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.338 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.338 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.339 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.339 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:56:34.339 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.339 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.339 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.339 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6809", 2021-02-10T04:56:34.339 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.339 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.339 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.339 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.339 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:56:34.339 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.339 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.339 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.340 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6811", 2021-02-10T04:56:34.340 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.340 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.340 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.340 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.340 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:56:34.340 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.340 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.340 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.340 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6810", 2021-02-10T04:56:34.340 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.340 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.340 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.341 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.341 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6808/7", 2021-02-10T04:56:34.341 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6809/7", 2021-02-10T04:56:34.341 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6811/7", 2021-02-10T04:56:34.341 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6810/7", 2021-02-10T04:56:34.341 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:56:34.341 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:56:34.341 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:56:34.341 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.341 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.341 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.341 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 7, 2021-02-10T04:56:34.341 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37", 2021-02-10T04:56:34.341 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:56:34.342 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:56:34.342 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:56:34.342 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 0.846160888671875, 2021-02-10T04:56:34.342 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:56:34.342 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:56:34.342 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 50, 2021-02-10T04:56:34.342 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1227, 2021-02-10T04:56:34.342 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:56:34.342 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:56:34.342 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:56:34.342 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.342 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.342 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.343 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6812", 2021-02-10T04:56:34.343 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.343 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.343 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.343 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.343 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:56:34.343 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.343 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.343 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.343 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6813", 2021-02-10T04:56:34.343 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.343 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.343 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.344 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.344 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:56:34.344 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.344 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.344 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.344 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6815", 2021-02-10T04:56:34.344 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.344 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.344 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.344 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.344 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:56:34.344 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:56:34.345 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.345 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:56:34.345 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6814", 2021-02-10T04:56:34.345 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:56:34.345 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.345 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.345 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.345 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6812/7", 2021-02-10T04:56:34.345 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6813/7", 2021-02-10T04:56:34.345 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6815/7", 2021-02-10T04:56:34.345 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6814/7", 2021-02-10T04:56:34.345 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:56:34.346 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:56:34.346 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:56:34.346 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.346 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.346 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:56:34.346 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_xinfo": [ 2021-02-10T04:56:34.346 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.346 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 0, 2021-02-10T04:56:34.346 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:56:34.346 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:56:34.346 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:56:34.346 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:56:34.346 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:56:34.347 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:56:13.467204+0000", 2021-02-10T04:56:34.347 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 1147 2021-02-10T04:56:34.347 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.347 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.347 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 1, 2021-02-10T04:56:34.347 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:56:34.347 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:56:34.347 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:56:34.347 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:56:34.347 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:56:34.347 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:55:59.628031+0000", 2021-02-10T04:56:34.347 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:56:34.347 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.347 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.348 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 2, 2021-02-10T04:56:34.348 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:56:34.348 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:56:34.348 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:56:34.348 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:56:34.348 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:56:34.348 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:56:19.295363+0000", 2021-02-10T04:56:34.348 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:56:34.348 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.348 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.348 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 3, 2021-02-10T04:56:34.348 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:56:34.349 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:56:34.349 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:56:34.349 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:56:34.349 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:56:34.349 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:55:59.465751+0000", 2021-02-10T04:56:34.349 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 1095 2021-02-10T04:56:34.349 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.349 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.349 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 4, 2021-02-10T04:56:34.349 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:56:34.349 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:56:34.349 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:56:34.349 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:56:34.350 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:56:34.350 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:55:54.650945+0000", 2021-02-10T04:56:34.350 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:56:34.350 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.350 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.350 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 5, 2021-02-10T04:56:34.350 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:56:34.350 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:56:34.350 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:56:34.350 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:56:34.350 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:56:34.350 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:56:12.510690+0000", 2021-02-10T04:56:34.351 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:56:34.351 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.351 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.351 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 6, 2021-02-10T04:56:34.351 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:56:34.351 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:56:34.351 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:56:34.351 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:56:34.351 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:56:34.351 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:56:30.647440+0000", 2021-02-10T04:56:34.352 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:56:34.352 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.352 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.352 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 7, 2021-02-10T04:56:34.352 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:56:34.352 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:56:34.352 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:56:34.352 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:56:34.352 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:56:34.352 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:55:48.501625+0000", 2021-02-10T04:56:34.352 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:56:34.352 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.352 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:56:34.353 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_upmap": [], 2021-02-10T04:56:34.353 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_upmap_items": [ 2021-02-10T04:56:34.353 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.353 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "2.7", 2021-02-10T04:56:34.353 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:56:34.353 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.353 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:56:34.353 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 4 2021-02-10T04:56:34.353 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.353 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.353 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.353 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.353 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "2.1a", 2021-02-10T04:56:34.354 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:56:34.354 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.354 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:56:34.354 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 2 2021-02-10T04:56:34.354 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.354 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.354 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.354 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.354 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "2.1d", 2021-02-10T04:56:34.354 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:56:34.354 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.354 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:56:34.354 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 2 2021-02-10T04:56:34.355 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.355 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.355 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.355 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.355 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "2.21", 2021-02-10T04:56:34.355 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:56:34.355 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.355 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:56:34.355 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 1 2021-02-10T04:56:34.355 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.355 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.355 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.355 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.356 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "2.28", 2021-02-10T04:56:34.356 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:56:34.356 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.356 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:56:34.356 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 2 2021-02-10T04:56:34.356 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.356 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.356 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.356 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.356 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.5", 2021-02-10T04:56:34.356 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:56:34.356 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.356 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 4, 2021-02-10T04:56:34.356 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 2 2021-02-10T04:56:34.357 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.357 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.357 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.357 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.357 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.10", 2021-02-10T04:56:34.357 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:56:34.357 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.357 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 1, 2021-02-10T04:56:34.357 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 5 2021-02-10T04:56:34.357 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.357 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.357 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.357 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.358 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.6", 2021-02-10T04:56:34.358 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:56:34.358 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.358 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 5, 2021-02-10T04:56:34.358 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 6 2021-02-10T04:56:34.358 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.358 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.358 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.358 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.358 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.8", 2021-02-10T04:56:34.358 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:56:34.358 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.359 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 1, 2021-02-10T04:56:34.359 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 3 2021-02-10T04:56:34.359 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.359 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.359 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.359 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.359 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.10", 2021-02-10T04:56:34.359 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:56:34.359 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.359 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:56:34.359 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 3 2021-02-10T04:56:34.359 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.359 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.359 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.360 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.360 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.18", 2021-02-10T04:56:34.360 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:56:34.360 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.360 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 4, 2021-02-10T04:56:34.360 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 3 2021-02-10T04:56:34.360 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.360 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.360 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.360 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.360 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.1b", 2021-02-10T04:56:34.360 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:56:34.360 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.361 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 0, 2021-02-10T04:56:34.361 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 3 2021-02-10T04:56:34.361 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.361 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.361 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.361 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.361 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "7.6", 2021-02-10T04:56:34.361 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:56:34.361 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.361 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 3, 2021-02-10T04:56:34.361 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 1 2021-02-10T04:56:34.361 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.361 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.361 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.362 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.362 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "7.15", 2021-02-10T04:56:34.362 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:56:34.362 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:56:34.362 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:56:34.362 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 4 2021-02-10T04:56:34.362 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.362 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:56:34.362 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.362 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:56:34.362 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_temp": [], 2021-02-10T04:56:34.362 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_temp": [], 2021-02-10T04:56:34.362 INFO:teuthology.orchestra.run.smithi085.stdout: "blocklist": { 2021-02-10T04:56:34.363 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/801393150": "2021-02-11T04:21:37.851607+0000", 2021-02-10T04:56:34.363 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/3049671696": "2021-02-11T04:21:37.851607+0000", 2021-02-10T04:56:34.363 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/4056505602": "2021-02-11T04:21:20.700478+0000", 2021-02-10T04:56:34.363 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:6800/7": "2021-02-11T04:21:20.700478+0000", 2021-02-10T04:56:34.363 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/1879026062": "2021-02-11T04:21:20.700478+0000" 2021-02-10T04:56:34.363 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.363 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profiles": { 2021-02-10T04:56:34.363 INFO:teuthology.orchestra.run.smithi085.stdout: "default": { 2021-02-10T04:56:34.363 INFO:teuthology.orchestra.run.smithi085.stdout: "crush-failure-domain": "osd", 2021-02-10T04:56:34.363 INFO:teuthology.orchestra.run.smithi085.stdout: "k": "2", 2021-02-10T04:56:34.363 INFO:teuthology.orchestra.run.smithi085.stdout: "m": "1", 2021-02-10T04:56:34.363 INFO:teuthology.orchestra.run.smithi085.stdout: "plugin": "jerasure", 2021-02-10T04:56:34.363 INFO:teuthology.orchestra.run.smithi085.stdout: "ruleset-failure-domain": "osd", 2021-02-10T04:56:34.363 INFO:teuthology.orchestra.run.smithi085.stdout: "technique": "reed_sol_van" 2021-02-10T04:56:34.364 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.364 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:56:34.364 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps_queue": [], 2021-02-10T04:56:34.364 INFO:teuthology.orchestra.run.smithi085.stdout: "new_removed_snaps": [], 2021-02-10T04:56:34.364 INFO:teuthology.orchestra.run.smithi085.stdout: "new_purged_snaps": [], 2021-02-10T04:56:34.364 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_node_flags": {}, 2021-02-10T04:56:34.364 INFO:teuthology.orchestra.run.smithi085.stdout: "device_class_flags": {}, 2021-02-10T04:56:34.364 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_mode": { 2021-02-10T04:56:34.364 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_mode_enabled": false, 2021-02-10T04:56:34.364 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_bucket_count": 0, 2021-02-10T04:56:34.364 INFO:teuthology.orchestra.run.smithi085.stdout: "degraded_stretch_mode": 0, 2021-02-10T04:56:34.364 INFO:teuthology.orchestra.run.smithi085.stdout: "recovering_stretch_mode": 0, 2021-02-10T04:56:34.365 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_mode_bucket": 0 2021-02-10T04:56:34.365 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:56:34.365 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:56:34.680 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is set 2021-02-10T04:56:34.809 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:56:34.810 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2021-02-10T04:56:34.810 INFO:teuthology.orchestra.run.smithi164.stdout: "complaint_time": 30, 2021-02-10T04:56:34.810 INFO:teuthology.orchestra.run.smithi164.stdout: "num_blocked_ops": 0 2021-02-10T04:56:34.810 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:56:34.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:34 smithi085 conmon[29319]: audit 2021-02-10T04:56:34.255187+0000 mon.a (mon.0) 2752 : audit [DBG] from='client.? 172.21.15.85:0/1072071771' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-02-10T04:56:34.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:34 smithi085 conmon[29319]: audit 2021-02-10T04:56:34.287723+0000 mon.a (mon.0) 2753 : audit [INF] from='client.? 172.21.15.85:0/1729935414' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:56:34.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:34 smithi085 conmon[35961]: audit 2021-02-10T04:56:34.255187+0000 mon.a (mon.0) 2752 : audit [DBG] from='client.? 172.21.15.85:0/1072071771' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-02-10T04:56:34.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:34 smithi085 conmon[35961]: audit 2021-02-10T04:56:34.287723+0000 mon.a (mon.0) 2753 : audit [INF] from='client.? 172.21.15.85:0/1729935414' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:56:35.037 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:34 smithi164 conmon[29288]: audit 2021-02-10T04:56:34.255187+0000 mon.a ( 2021-02-10T04:56:35.037 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:34 smithi164 conmon[29288]: mon.0) 2752 : audit [DBG] from='client.? 172.21.15.85:0/1072071771' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-02-10T04:56:35.037 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:34 smithi164 conmon[29288]: audit 2021-02-10T04:56:34.287723+0000 mon.a (mon.0) 2753 : audit [INF] from='client.? 172.21.15.85:0/1729935414' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:56:35.795 INFO:tasks.thrashosds.thrasher:j is {'epoch': 1275, 'fsid': '42224fce-6b57-11eb-8feb-001a4aab830c', 'created': '2021-02-10T04:20:57.933279+0000', 'modified': '2021-02-10T04:56:33.572989+0000', 'last_up_change': '2021-02-10T04:55:13.547923+0000', 'last_in_change': '2021-02-10T04:51:35.701776+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-02-10T04:24:05.258028+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1266', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '110', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2021-02-10T04:25:32.688554+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 42, 'pg_placement_num': 42, 'pg_placement_num_target': 42, 'pg_num_target': 42, 'pg_num_pending': 42, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1272', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '89', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2021-02-10T04:25:47.658613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 13, 'pg_placement_num': 13, 'pg_placement_num_target': 13, 'pg_num_target': 13, 'pg_num_pending': 13, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1211', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '1209', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'zone.rgw.log', 'create_time': '2021-02-10T04:25:47.968727+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '4.16', 'ready_epoch': 1256, 'last_epoch_started': 1255, 'last_epoch_clean': 1255, 'source_version': "1048'40", 'target_version': "1049'40"}, 'last_change': '1256', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '1256', 'last_force_op_resend_preluminous': '1254', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'zone.rgw.control', 'create_time': '2021-02-10T04:25:49.278260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'zone.rgw.meta', 'create_time': '2021-02-10T04:25:51.792514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 42, 'pg_placement_num': 42, 'pg_placement_num_target': 42, 'pg_num_target': 42, 'pg_num_pending': 42, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1046', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '1044', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 8}, 'application_metadata': {'rgw': {}}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2021-02-10T04:28:23.932391+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 36, 'pg_placement_num': 36, 'pg_placement_num_target': 36, 'pg_num_target': 36, 'pg_num_pending': 36, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1143', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '1141', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 888, 'snap_epoch': 994, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '45d0bb5f-35fa-4596-964f-75abcee53bee', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 8, 'last_clean_end': 1146, 'up_from': 1227, 'up_thru': 1256, 'down_at': 1147, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6801', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6802', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6804', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6803', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6801/7', 'cluster_addr': '172.21.15.85:6802/7', 'heartbeat_back_addr': '172.21.15.85:6804/7', 'heartbeat_front_addr': '172.21.15.85:6803/7', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '3421a5ce-872e-4b52-adf7-6aaa595871a4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.86566162109375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 1252, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6805', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6806', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6808', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6807', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6805/7', 'cluster_addr': '172.21.15.85:6806/7', 'heartbeat_back_addr': '172.21.15.85:6808/7', 'heartbeat_front_addr': '172.21.15.85:6807/7', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '96e1952a-1042-4316-9fc9-86813e88b912', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 1248, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6809', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6810', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6812', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6811', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6809/7', 'cluster_addr': '172.21.15.85:6810/7', 'heartbeat_back_addr': '172.21.15.85:6812/7', 'heartbeat_front_addr': '172.21.15.85:6811/7', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'e019f49f-5dd4-4104-9ced-94c710616642', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 25, 'last_clean_end': 1094, 'up_from': 1111, 'up_thru': 1248, 'down_at': 1095, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6813', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6814', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6816', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6815', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6813/7', 'cluster_addr': '172.21.15.85:6814/7', 'heartbeat_back_addr': '172.21.15.85:6816/7', 'heartbeat_front_addr': '172.21.15.85:6815/7', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '91c8a80a-37b6-4c30-a32a-647b2e6d801f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 1244, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6800', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6801', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6803', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6802', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6800/7', 'cluster_addr': '172.21.15.164:6801/7', 'heartbeat_back_addr': '172.21.15.164:6803/7', 'heartbeat_front_addr': '172.21.15.164:6802/7', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 1248, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6804', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6805', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6807', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6806', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6804/7', 'cluster_addr': '172.21.15.164:6805/7', 'heartbeat_back_addr': '172.21.15.164:6807/7', 'heartbeat_front_addr': '172.21.15.164:6806/7', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'adc40430-3eeb-4003-a824-343021243077', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 43, 'up_thru': 1248, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6808', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6809', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6811', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6810', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6808/7', 'cluster_addr': '172.21.15.164:6809/7', 'heartbeat_back_addr': '172.21.15.164:6811/7', 'heartbeat_front_addr': '172.21.15.164:6810/7', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.846160888671875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 1227, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6812', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6813', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6815', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6814', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6812/7', 'cluster_addr': '172.21.15.164:6813/7', 'heartbeat_back_addr': '172.21.15.164:6815/7', 'heartbeat_front_addr': '172.21.15.164:6814/7', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:56:13.467204+0000', 'dead_epoch': 1147}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:55:59.628031+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:56:19.295363+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:55:59.465751+0000', 'dead_epoch': 1095}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:55:54.650945+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:56:12.510690+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:56:30.647440+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:55:48.501625+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.7', 'mappings': [{'from': 7, 'to': 4}]}, {'pgid': '2.1a', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '2.1d', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '2.21', 'mappings': [{'from': 7, 'to': 1}]}, {'pgid': '2.28', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.5', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '4.10', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '5.6', 'mappings': [{'from': 5, 'to': 6}]}, {'pgid': '5.8', 'mappings': [{'from': 1, 'to': 3}]}, {'pgid': '5.10', 'mappings': [{'from': 7, 'to': 3}]}, {'pgid': '5.18', 'mappings': [{'from': 4, 'to': 3}]}, {'pgid': '5.1b', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '7.6', 'mappings': [{'from': 3, 'to': 1}]}, {'pgid': '7.15', 'mappings': [{'from': 7, 'to': 4}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.85:0/801393150': '2021-02-11T04:21:37.851607+0000', '172.21.15.85:0/3049671696': '2021-02-11T04:21:37.851607+0000', '172.21.15.85:0/4056505602': '2021-02-11T04:21:20.700478+0000', '172.21.15.85:6800/7': '2021-02-11T04:21:20.700478+0000', '172.21.15.85:0/1879026062': '2021-02-11T04:21:20.700478+0000'}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2021-02-10T04:56:35.796 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:56:35.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:35 smithi085 conmon[29319]: cluster 2021-02-10T04:56:34.517123 2021-02-10T04:56:35.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:35 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 1411 : cluster [DBG] pgmap v2313: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-02-10T04:56:35.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:35 smithi085 conmon[29319]: cluster 2021-02-10T04:56:34.666529+0000 mon.a (mon.0) 2754 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:56:35.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:35 smithi085 conmon[29319]: audit 2021-02-10T04:56:34.673879+0000 mon.a (mon.0) 2755 : audit [INF] from='client.? 172.21.15.85:0/1729935414' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:56:35.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:35 smithi085 conmon[29319]: cluster 2021-02-10T04:56:34.673937+0000 mon.a (mon. 2021-02-10T04:56:35.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:35 smithi085 conmon[29319]: 0) 2756 : cluster [DBG] osdmap e1276: 8 total, 8 up, 8 in 2021-02-10T04:56:35.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:35 smithi085 conmon[35961]: cluster 2021-02-10T04:56:34.517123 2021-02-10T04:56:35.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:35 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1411 : cluster [DBG] pgmap v2313: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-02-10T04:56:35.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:35 smithi085 conmon[35961]: cluster 2021-02-10T04:56:35.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:35 smithi085 conmon[35961]: 2021-02-10T04:56:34.666529+0000 mon.a (mon.0) 2754 : cluster 2021-02-10T04:56:35.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:35 smithi085 conmon[35961]: [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:56:35.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:35 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:56:35.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:35 smithi085 conmon[35961]: 56:34.673879+0000 mon.a (mon.0) 2021-02-10T04:56:35.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:35 smithi085 conmon[35961]: 2755 : audit [INF] from='client.? 172.21.15.85:0/1729935414' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:56:35.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:35 smithi085 conmon[35961]: cluster 2021-02-10T04:56:35.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:35 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:56:35.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:35 smithi085 conmon[35961]: 56:34.673937+0000 mon.a (mon.0 2021-02-10T04:56:35.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:35 smithi085 conmon[35961]: ) 2756 : cluster [DBG] osdmap e1276: 8 total, 8 up, 8 in 2021-02-10T04:56:36.067 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:56:36.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:35 smithi164 conmon[29288]: cluster 2021-02-10T04:56:34.517123+0000 mgr.y (mgr.14140) 1411 : cluster 2021-02-10T04:56:36.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:35 smithi164 conmon[29288]: [DBG] pgmap v2313: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-02-10T04:56:36.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:35 smithi164 conmon[29288]: cluster 2021-02-10T04:56:34.666529+0000 mon.a (mon.0) 2754 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:56:36.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:35 smithi164 conmon[29288]: audit 2021-02-10T04:56:34.673879+0000 mon.a (mon.0) 2755 : audit [INF] from='client.? 172.21.15.85:0/1729935414' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:56:36.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:35 smithi164 conmon[29288]: cluster 2021-02-10T04:56:34.673937+0000 mon.a (mon.0) 2756 : cluster [DBG] osdmap e1276: 8 total, 8 up, 8 in 2021-02-10T04:56:37.014 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:56:37.035 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:56:37.053 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:56:37.071 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:56:37.094 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:56:37.118 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:56:37.136 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:56:37.155 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:56:37.157 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:56:37.157 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:56:37.157 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:56:37.157 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:56:37.157 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:56:37.157 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:56:37.157 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:56:37.157 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:56:37.825 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.7 dump_historic_ops 2021-02-10T04:56:37.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:37 smithi085 conmon[29319]: cluster 2021-02-10T04:56:36.517632+0000 mgr.y (mgr.14140) 1412 : cluster 2021-02-10T04:56:37.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:37 smithi085 conmon[29319]: [DBG] pgmap v2315: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:56:37.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:37 smithi085 conmon[29319]: audit 2021-02-10T04:56:36.991193+0000 mon.c (mon.1) 133 : audit [DBG] from='client.? 172.21.15.85:0/4213571443' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:56:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:37 smithi085 conmon[35961]: cluster 2021-02-10T04:56:36.517632+0000 mgr.y ( 2021-02-10T04:56:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:37 smithi085 conmon[35961]: mgr.14140) 1412 : cluster [DBG] pgmap v2315: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:56:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:37 smithi085 conmon[35961]: audit 2021-02-10T04:56:36.991193+0000 mon.c (mon. 2021-02-10T04:56:37.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:37 smithi085 conmon[35961]: 1) 133 : audit [DBG] from='client.? 172.21.15.85:0/4213571443' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:56:38.085 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:56:38.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:37 smithi164 conmon[29288]: cluster 2021-02-10T04:56:38.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:37 smithi164 conmon[29288]: 2021-02-10T04:56:36. 2021-02-10T04:56:38.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:37 smithi164 conmon[29288]: 517632+0000 mgr.y (mgr.14140) 1412 : cluster [DBG] pgmap v2315: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:56:38.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:37 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:56:38.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:37 smithi164 conmon[29288]: 02-10T04:56:36.991193+0000 mon.c (mon.1) 133 : audit [DBG] from='client.? 172.21.15.85:0/4213571443' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:56:38.601 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set nodeep-scrub 2021-02-10T04:56:38.831 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:56:38.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:38 smithi085 conmon[29319]: audit 2021-02-10T04:56:38.256782 2021-02-10T04:56:38.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:38 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2757 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:56:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:38 smithi085 conmon[29319]: audit 2021-02-10T04:56:38.257897+0000 mon.a (mon.0) 2758 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:56:38.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:38 smithi085 conmon[29319]: cluster 2021-02-10T04:56:38.580153+0000 mon.a (mon.0) 2759 : cluster [DBG] osdmap e1277: 8 total, 8 up, 8 in 2021-02-10T04:56:38.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:38 smithi085 conmon[35961]: audit 2021-02-10T04:56:38.256782+0000 2021-02-10T04:56:38.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:38 smithi085 conmon[35961]: mon.a (mon.0) 2757 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:56:38.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:38 smithi085 conmon[35961]: audit 2021-02-10T04:56:38.257897+0000 mon.a (mon.0) 2758 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:56:38.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:38 smithi085 conmon[35961]: cluster 2021-02-10T04:56:38.580153+0000 mon.a (mon.0) 2759 : cluster [DBG] osdmap e1277: 8 total, 8 up, 8 in 2021-02-10T04:56:39.072 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:56:39.075 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:56:39.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:38 smithi164 conmon[29288]: audit 2021-02-10T04:56:38.256782+0000 mon.a (mon.0) 2757 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:56:39.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:38 smithi164 conmon[29288]: audit 2021-02-10T04:56:38.257897+0000 mon.a (mon.0) 2758 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:56:39.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:38 smithi164 conmon[29288]: cluster 2021-02-10T04:56:38.580153+0000 mon.a (mon.0) 2759 : cluster [DBG] osdmap e1277: 8 total, 8 up, 8 in 2021-02-10T04:56:39.644 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:56:39 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:56:39.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:39 smithi085 conmon[35961]: cluster 2021-02-10T04:56: 2021-02-10T04:56:39.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:39 smithi085 conmon[35961]: 38.518274+0000 mgr.y (mgr.14140) 1413 : cluster [DBG] pgmap v2316: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:56:39.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:39 smithi085 conmon[35961]: audit 2021-02-10T04:56:39.051757+0000 mgr.y (mgr.14140) 1414 : audit [DBG] from='client.26119 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:56:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:39 smithi085 conmon[29319]: cluster 2021-02-10T04:56: 2021-02-10T04:56:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:39 smithi085 conmon[29319]: 38.518274+0000 mgr.y (mgr.14140) 1413 : cluster [DBG] pgmap v2316: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:56:39.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:39 smithi085 conmon[29319]: audit 2021-02-10T04:56:39.051757+0000 mgr.y (mgr.14140) 1414 : audit [DBG] from='client.26119 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:56:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:39 smithi164 conmon[29288]: cluster 2021-02-10T04:56:38.518274+0000 mgr.y ( 2021-02-10T04:56:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:39 smithi164 conmon[29288]: mgr.14140) 1413 : cluster [DBG] pgmap v2316: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:56:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:39 smithi164 conmon[29288]: audit 2021-02-10T04:56:39.051757+0000 mgr.y (mgr.14140) 1414 : audit [DBG] from='client.26119 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:56:40.329 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":2317,"stamp":"2021-02-10T04:56:38.577759+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":972357089,"num_objects":748,"num_object_clones":337,"num_object_copies":2244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":748,"num_whiteouts":10,"num_read":7895,"num_read_kb":4562294,"num_write":4841,"num_write_kb":1247750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":887,"num_bytes_recovered":311980434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35650,"ondisk_log_size":35650,"up":594,"acting":594,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":596,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2312592,"kb_used_data":1981072,"kb_used_omap":692,"kb_used_meta":330763,"kb_avail":747648624,"statfs":{"total":767960285184,"available":765592190976,"internally_reserved":0,"allocated":2028617728,"data_stored":2598226524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":709615,"internal_metadata":338701329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.061473"},"pg_stats":[{"pgid":"6.29","version":"0'0","reported_seq":"1132","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.579053+0000","last_change":"2021-02-10T04:55:14.633086+0000","last_active":"2021-02-10T04:56:33.579053+0000","last_peered":"2021-02-10T04:56:33.579053+0000","last_clean":"2021-02-10T04:56:33.579053+0000","last_became_active":"2021-02-10T04:55:14.632670+0000","last_became_peered":"2021-02-10T04:55:14.632670+0000","last_unstale":"2021-02-10T04:56:33.579053+0000","last_undegraded":"2021-02-10T04:56:33.579053+0000","last_fullsized":"2021-02-10T04:56:33.579053+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:24.121684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:24.121684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":"172","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.577122+0000","last_change":"2021-02-10T04:52:30.133105+0000","last_active":"2021-02-10T04:56:33.577122+0000","last_peered":"2021-02-10T04:56:33.577122+0000","last_clean":"2021-02-10T04:56:33.577122+0000","last_became_active":"2021-02-10T04:51:36.864926+0000","last_became_peered":"2021-02-10T04:51:36.864926+0000","last_unstale":"2021-02-10T04:56:33.577122+0000","last_undegraded":"2021-02-10T04:56:33.577122+0000","last_fullsized":"2021-02-10T04:56:33.577122+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.133043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.133043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"1191'4","reported_seq":"260","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.579674+0000","last_change":"2021-02-10T04:53:46.736164+0000","last_active":"2021-02-10T04:56:33.579674+0000","last_peered":"2021-02-10T04:56:33.579674+0000","last_clean":"2021-02-10T04:56:33.579674+0000","last_became_active":"2021-02-10T04:53:46.735711+0000","last_became_peered":"2021-02-10T04:53:46.735711+0000","last_unstale":"2021-02-10T04:56:33.579674+0000","last_undegraded":"2021-02-10T04:56:33.579674+0000","last_fullsized":"2021-02-10T04:56:33.579674+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'3","last_scrub_stamp":"2021-02-10T04:51:01.185820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.185820+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"1261'84","reported_seq":"1408","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.577498+0000","last_change":"2021-02-10T04:55:39.594594+0000","last_active":"2021-02-10T04:56:33.577498+0000","last_peered":"2021-02-10T04:56:33.577498+0000","last_clean":"2021-02-10T04:56:33.577498+0000","last_became_active":"2021-02-10T04:55:39.594275+0000","last_became_peered":"2021-02-10T04:55:39.594275+0000","last_unstale":"2021-02-10T04:56:33.577498+0000","last_undegraded":"2021-02-10T04:56:33.577498+0000","last_fullsized":"2021-02-10T04:56:33.577498+0000","mapping_epoch":55,"log_start":"1132'66","ondisk_log_start":"1132'66","created":55,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":0,"last_scrub":"1132'66","last_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_deep_scrub":"1132'66","last_deep_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.430399+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1616,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":186,"num_read_kb":150,"num_write":107,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"181","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.578656+0000","last_change":"2021-02-10T04:53:41.722609+0000","last_active":"2021-02-10T04:56:33.578656+0000","last_peered":"2021-02-10T04:56:33.578656+0000","last_clean":"2021-02-10T04:56:33.578656+0000","last_became_active":"2021-02-10T04:51:36.875000+0000","last_became_peered":"2021-02-10T04:51:36.875000+0000","last_unstale":"2021-02-10T04:56:33.578656+0000","last_undegraded":"2021-02-10T04:56:33.578656+0000","last_fullsized":"2021-02-10T04:56:33.578656+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:53:41.722516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:41.722516+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"1292","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.580124+0000","last_change":"2021-02-10T04:56:33.137887+0000","last_active":"2021-02-10T04:56:33.580124+0000","last_peered":"2021-02-10T04:56:33.580124+0000","last_clean":"2021-02-10T04:56:33.580124+0000","last_became_active":"2021-02-10T04:55:14.630535+0000","last_became_peered":"2021-02-10T04:55:14.630535+0000","last_unstale":"2021-02-10T04:56:33.580124+0000","last_undegraded":"2021-02-10T04:56:33.580124+0000","last_fullsized":"2021-02-10T04:56:33.580124+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:48.408748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:50:48.408748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"1003'1689","reported_seq":"4695","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.034521+0000","last_change":"2021-02-10T04:53:38.170873+0000","last_active":"2021-02-10T04:56:34.034521+0000","last_peered":"2021-02-10T04:56:34.034521+0000","last_clean":"2021-02-10T04:56:34.034521+0000","last_became_active":"2021-02-10T04:52:26.030105+0000","last_became_peered":"2021-02-10T04:52:26.030105+0000","last_unstale":"2021-02-10T04:56:34.034521+0000","last_undegraded":"2021-02-10T04:56:34.034521+0000","last_fullsized":"2021-02-10T04:56:34.034521+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:53:38.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.170802+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":99361981,"num_objects":37,"num_object_clones":33,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"62'5","reported_seq":"198","reported_epoch":"1276","state":"active+clean","last_fresh":"2021-02-10T04:56:36.747163+0000","last_change":"2021-02-10T04:54:39.615784+0000","last_active":"2021-02-10T04:56:36.747163+0000","last_peered":"2021-02-10T04:56:36.747163+0000","last_clean":"2021-02-10T04:56:36.747163+0000","last_became_active":"2021-02-10T04:54:39.560009+0000","last_became_peered":"2021-02-10T04:54:39.560009+0000","last_unstale":"2021-02-10T04:56:36.747163+0000","last_undegraded":"2021-02-10T04:56:36.747163+0000","last_fullsized":"2021-02-10T04:56:36.747163+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":269,"num_read_kb":268,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"1261'28","reported_seq":"1285","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.035660+0000","last_change":"2021-02-10T04:55:21.669679+0000","last_active":"2021-02-10T04:56:34.035660+0000","last_peered":"2021-02-10T04:56:34.035660+0000","last_clean":"2021-02-10T04:56:34.035660+0000","last_became_active":"2021-02-10T04:55:21.669219+0000","last_became_peered":"2021-02-10T04:55:21.669219+0000","last_unstale":"2021-02-10T04:56:34.035660+0000","last_undegraded":"2021-02-10T04:56:34.035660+0000","last_fullsized":"2021-02-10T04:56:34.035660+0000","mapping_epoch":1228,"log_start":"1061'20","ondisk_log_start":"1061'20","created":55,"last_epoch_clean":1233,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'32","last_scrub_stamp":"2021-02-10T04:51:51.382040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.382040+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1158,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":105,"num_read_kb":81,"num_write":58,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2052,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"927","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.035159+0000","last_change":"2021-02-10T04:53:34.179015+0000","last_active":"2021-02-10T04:56:34.035159+0000","last_peered":"2021-02-10T04:56:34.035159+0000","last_clean":"2021-02-10T04:56:34.035159+0000","last_became_active":"2021-02-10T04:51:49.502576+0000","last_became_peered":"2021-02-10T04:51:49.502576+0000","last_unstale":"2021-02-10T04:56:34.035159+0000","last_undegraded":"2021-02-10T04:56:34.035159+0000","last_fullsized":"2021-02-10T04:56:34.035159+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:34.178949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.178949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"172","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.578139+0000","last_change":"2021-02-10T04:51:53.038893+0000","last_active":"2021-02-10T04:56:33.578139+0000","last_peered":"2021-02-10T04:56:33.578139+0000","last_clean":"2021-02-10T04:56:33.578139+0000","last_became_active":"2021-02-10T04:51:36.855507+0000","last_became_peered":"2021-02-10T04:51:36.855507+0000","last_unstale":"2021-02-10T04:56:33.578139+0000","last_undegraded":"2021-02-10T04:56:33.578139+0000","last_fullsized":"2021-02-10T04:56:33.578139+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:53.038834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:53.038834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"1194","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.033748+0000","last_change":"2021-02-10T04:55:28.097406+0000","last_active":"2021-02-10T04:56:34.033748+0000","last_peered":"2021-02-10T04:56:34.033748+0000","last_clean":"2021-02-10T04:56:34.033748+0000","last_became_active":"2021-02-10T04:55:14.629603+0000","last_became_peered":"2021-02-10T04:55:14.629603+0000","last_unstale":"2021-02-10T04:56:34.033748+0000","last_undegraded":"2021-02-10T04:56:34.033748+0000","last_fullsized":"2021-02-10T04:56:34.033748+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:09.922789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:09.922789+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"1168'82","reported_seq":"208","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.577276+0000","last_change":"2021-02-10T04:53:23.326363+0000","last_active":"2021-02-10T04:56:33.577276+0000","last_peered":"2021-02-10T04:56:33.577276+0000","last_clean":"2021-02-10T04:56:33.577276+0000","last_became_active":"2021-02-10T04:51:36.859215+0000","last_became_peered":"2021-02-10T04:51:36.859215+0000","last_unstale":"2021-02-10T04:56:33.577276+0000","last_undegraded":"2021-02-10T04:56:33.577276+0000","last_fullsized":"2021-02-10T04:56:33.577276+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1150'76","last_scrub_stamp":"2021-02-10T04:53:00.964499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:00.964499+0000","log_size":82,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3848803,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"1261'69","reported_seq":"1414","reported_epoch":"1274","state":"active+clean","last_fresh":"2021-02-10T04:56:30.668656+0000","last_change":"2021-02-10T04:55:25.576927+0000","last_active":"2021-02-10T04:56:30.668656+0000","last_peered":"2021-02-10T04:56:30.668656+0000","last_clean":"2021-02-10T04:56:30.668656+0000","last_became_active":"2021-02-10T04:55:25.576633+0000","last_became_peered":"2021-02-10T04:55:25.576633+0000","last_unstale":"2021-02-10T04:56:30.668656+0000","last_undegraded":"2021-02-10T04:56:30.668656+0000","last_fullsized":"2021-02-10T04:56:30.668656+0000","mapping_epoch":55,"log_start":"1112'47","ondisk_log_start":"1112'47","created":55,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'61","last_scrub_stamp":"2021-02-10T04:53:36.836202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:36.836202+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1466,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":208,"num_read_kb":172,"num_write":122,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"1228'6","reported_seq":"253","reported_epoch":"1276","state":"active+clean","last_fresh":"2021-02-10T04:56:34.676686+0000","last_change":"2021-02-10T04:55:14.605868+0000","last_active":"2021-02-10T04:56:34.676686+0000","last_peered":"2021-02-10T04:56:34.676686+0000","last_clean":"2021-02-10T04:56:34.676686+0000","last_became_active":"2021-02-10T04:55:14.605314+0000","last_became_peered":"2021-02-10T04:55:14.605314+0000","last_unstale":"2021-02-10T04:56:34.676686+0000","last_undegraded":"2021-02-10T04:56:34.676686+0000","last_fullsized":"2021-02-10T04:56:34.676686+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'4","last_scrub_stamp":"2021-02-10T04:52:01.121380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.121380+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"187","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.577736+0000","last_change":"2021-02-10T04:55:22.573715+0000","last_active":"2021-02-10T04:56:33.577736+0000","last_peered":"2021-02-10T04:56:33.577736+0000","last_clean":"2021-02-10T04:56:33.577736+0000","last_became_active":"2021-02-10T04:55:14.636895+0000","last_became_peered":"2021-02-10T04:55:14.636895+0000","last_unstale":"2021-02-10T04:56:33.577736+0000","last_undegraded":"2021-02-10T04:56:33.577736+0000","last_fullsized":"2021-02-10T04:56:33.577736+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:55:22.573621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:55:22.573621+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"572","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.578560+0000","last_change":"2021-02-10T04:53:46.733236+0000","last_active":"2021-02-10T04:56:33.578560+0000","last_peered":"2021-02-10T04:56:33.578560+0000","last_clean":"2021-02-10T04:56:33.578560+0000","last_became_active":"2021-02-10T04:53:46.732890+0000","last_became_peered":"2021-02-10T04:53:46.732890+0000","last_unstale":"2021-02-10T04:56:33.578560+0000","last_undegraded":"2021-02-10T04:56:33.578560+0000","last_fullsized":"2021-02-10T04:56:33.578560+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:35.170157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:35.170157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"1003'739","reported_seq":"3873","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.033782+0000","last_change":"2021-02-10T04:55:28.091164+0000","last_active":"2021-02-10T04:56:34.033782+0000","last_peered":"2021-02-10T04:56:34.033782+0000","last_clean":"2021-02-10T04:56:34.033782+0000","last_became_active":"2021-02-10T04:55:14.648601+0000","last_became_peered":"2021-02-10T04:55:14.648601+0000","last_unstale":"2021-02-10T04:56:34.033782+0000","last_undegraded":"2021-02-10T04:56:34.033782+0000","last_fullsized":"2021-02-10T04:56:34.033782+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:52:27.602006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:27.602006+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24059888,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":129,"num_read_kb":109544,"num_write":102,"num_write_kb":46173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"1251","reported_epoch":"1274","state":"active+clean","last_fresh":"2021-02-10T04:56:30.668121+0000","last_change":"2021-02-10T04:54:39.559980+0000","last_active":"2021-02-10T04:56:30.668121+0000","last_peered":"2021-02-10T04:56:30.668121+0000","last_clean":"2021-02-10T04:56:30.668121+0000","last_became_active":"2021-02-10T04:54:39.559443+0000","last_became_peered":"2021-02-10T04:54:39.559443+0000","last_unstale":"2021-02-10T04:56:30.668121+0000","last_undegraded":"2021-02-10T04:56:30.668121+0000","last_fullsized":"2021-02-10T04:56:30.668121+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"1261'64","reported_seq":"1460","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.038183+0000","last_change":"2021-02-10T04:53:37.108747+0000","last_active":"2021-02-10T04:56:34.038183+0000","last_peered":"2021-02-10T04:56:34.038183+0000","last_clean":"2021-02-10T04:56:34.038183+0000","last_became_active":"2021-02-10T04:53:35.551311+0000","last_became_peered":"2021-02-10T04:53:35.551311+0000","last_unstale":"2021-02-10T04:56:34.038183+0000","last_undegraded":"2021-02-10T04:56:34.038183+0000","last_fullsized":"2021-02-10T04:56:34.038183+0000","mapping_epoch":1118,"log_start":"1066'44","ondisk_log_start":"1066'44","created":55,"last_epoch_clean":1181,"parent":"0.0","parent_split_bits":0,"last_scrub":"1066'44","last_scrub_stamp":"2021-02-10T04:53:37.108677+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:53:37.108677+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1578,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":180,"num_read_kb":144,"num_write":106,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":2312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"1170","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.037071+0000","last_change":"2021-02-10T04:53:39.126356+0000","last_active":"2021-02-10T04:56:34.037071+0000","last_peered":"2021-02-10T04:56:34.037071+0000","last_clean":"2021-02-10T04:56:34.037071+0000","last_became_active":"2021-02-10T04:29:07.145616+0000","last_became_peered":"2021-02-10T04:29:07.145616+0000","last_unstale":"2021-02-10T04:56:34.037071+0000","last_undegraded":"2021-02-10T04:56:34.037071+0000","last_fullsized":"2021-02-10T04:56:34.037071+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:39.126261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:39.126261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"502","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.035003+0000","last_change":"2021-02-10T04:55:14.611192+0000","last_active":"2021-02-10T04:56:34.035003+0000","last_peered":"2021-02-10T04:56:34.035003+0000","last_clean":"2021-02-10T04:56:34.035003+0000","last_became_active":"2021-02-10T04:55:14.610821+0000","last_became_peered":"2021-02-10T04:55:14.610821+0000","last_unstale":"2021-02-10T04:56:34.035003+0000","last_undegraded":"2021-02-10T04:56:34.035003+0000","last_fullsized":"2021-02-10T04:56:34.035003+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:26.311958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.311958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"204","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.578702+0000","last_change":"2021-02-10T04:55:03.526061+0000","last_active":"2021-02-10T04:56:33.578702+0000","last_peered":"2021-02-10T04:56:33.578702+0000","last_clean":"2021-02-10T04:56:33.578702+0000","last_became_active":"2021-02-10T04:51:36.855744+0000","last_became_peered":"2021-02-10T04:51:36.855744+0000","last_unstale":"2021-02-10T04:56:33.578702+0000","last_undegraded":"2021-02-10T04:56:33.578702+0000","last_fullsized":"2021-02-10T04:56:33.578702+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:02.162851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:13.727241+0000","last_clean_scrub_stamp":"2021-02-10T04:52:02.162851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"1063'654","reported_seq":"4451","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.578561+0000","last_change":"2021-02-10T04:55:14.637619+0000","last_active":"2021-02-10T04:56:33.578561+0000","last_peered":"2021-02-10T04:56:33.578561+0000","last_clean":"2021-02-10T04:56:33.578561+0000","last_became_active":"2021-02-10T04:55:14.636752+0000","last_became_peered":"2021-02-10T04:55:14.636752+0000","last_unstale":"2021-02-10T04:56:33.578561+0000","last_undegraded":"2021-02-10T04:56:33.578561+0000","last_fullsized":"2021-02-10T04:56:33.578561+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:51:57.121838+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.121838+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24700630,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":171,"num_read_kb":182272,"num_write":111,"num_write_kb":33727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":26349023,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"1221","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.577582+0000","last_change":"2021-02-10T04:51:50.433056+0000","last_active":"2021-02-10T04:56:33.577582+0000","last_peered":"2021-02-10T04:56:33.577582+0000","last_clean":"2021-02-10T04:56:33.577582+0000","last_became_active":"2021-02-10T04:51:36.843165+0000","last_became_peered":"2021-02-10T04:51:36.843165+0000","last_unstale":"2021-02-10T04:56:33.577582+0000","last_undegraded":"2021-02-10T04:56:33.577582+0000","last_fullsized":"2021-02-10T04:56:33.577582+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.432990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.432990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"1242","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.583720+0000","last_change":"2021-02-10T04:55:14.638290+0000","last_active":"2021-02-10T04:56:33.583720+0000","last_peered":"2021-02-10T04:56:33.583720+0000","last_clean":"2021-02-10T04:56:33.583720+0000","last_became_active":"2021-02-10T04:55:14.637927+0000","last_became_peered":"2021-02-10T04:55:14.637927+0000","last_unstale":"2021-02-10T04:56:33.583720+0000","last_undegraded":"2021-02-10T04:56:33.583720+0000","last_fullsized":"2021-02-10T04:56:33.583720+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"1167","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.579186+0000","last_change":"2021-02-10T04:53:46.727888+0000","last_active":"2021-02-10T04:56:33.579186+0000","last_peered":"2021-02-10T04:56:33.579186+0000","last_clean":"2021-02-10T04:56:33.579186+0000","last_became_active":"2021-02-10T04:53:46.727670+0000","last_became_peered":"2021-02-10T04:53:46.727670+0000","last_unstale":"2021-02-10T04:56:33.579186+0000","last_undegraded":"2021-02-10T04:56:33.579186+0000","last_fullsized":"2021-02-10T04:56:33.579186+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:41.041553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.041553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"1261'76","reported_seq":"1384","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.037526+0000","last_change":"2021-02-10T04:53:42.071173+0000","last_active":"2021-02-10T04:56:34.037526+0000","last_peered":"2021-02-10T04:56:34.037526+0000","last_clean":"2021-02-10T04:56:34.037526+0000","last_became_active":"2021-02-10T04:53:39.679752+0000","last_became_peered":"2021-02-10T04:53:39.679752+0000","last_unstale":"2021-02-10T04:56:34.037526+0000","last_undegraded":"2021-02-10T04:56:34.037526+0000","last_fullsized":"2021-02-10T04:56:34.037526+0000","mapping_epoch":1111,"log_start":"1049'39","ondisk_log_start":"1049'39","created":55,"last_epoch_clean":1185,"parent":"0.0","parent_split_bits":0,"last_scrub":"1187'54","last_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_deep_scrub":"1187'54","last_deep_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_clean_scrub_stamp":"2021-02-10T04:53:42.071103+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":147,"num_read_kb":127,"num_write":95,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"912","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.034324+0000","last_change":"2021-02-10T04:51:52.350764+0000","last_active":"2021-02-10T04:56:34.034324+0000","last_peered":"2021-02-10T04:56:34.034324+0000","last_clean":"2021-02-10T04:56:34.034324+0000","last_became_active":"2021-02-10T04:51:49.567784+0000","last_became_peered":"2021-02-10T04:51:49.567784+0000","last_unstale":"2021-02-10T04:56:34.034324+0000","last_undegraded":"2021-02-10T04:56:34.034324+0000","last_fullsized":"2021-02-10T04:56:34.034324+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:52.350706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.350706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"1119'4","reported_seq":"237","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.578086+0000","last_change":"2021-02-10T04:52:27.126670+0000","last_active":"2021-02-10T04:56:33.578086+0000","last_peered":"2021-02-10T04:56:33.578086+0000","last_clean":"2021-02-10T04:56:33.578086+0000","last_became_active":"2021-02-10T04:51:36.853647+0000","last_became_peered":"2021-02-10T04:51:36.853647+0000","last_unstale":"2021-02-10T04:56:33.578086+0000","last_undegraded":"2021-02-10T04:56:33.578086+0000","last_fullsized":"2021-02-10T04:56:33.578086+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'4","last_scrub_stamp":"2021-02-10T04:52:27.126599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:27.126599+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"1261","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.038611+0000","last_change":"2021-02-10T04:52:25.745660+0000","last_active":"2021-02-10T04:56:34.038611+0000","last_peered":"2021-02-10T04:56:34.038611+0000","last_clean":"2021-02-10T04:56:34.038611+0000","last_became_active":"2021-02-10T04:51:36.849503+0000","last_became_peered":"2021-02-10T04:51:36.849503+0000","last_unstale":"2021-02-10T04:56:34.038611+0000","last_undegraded":"2021-02-10T04:56:34.038611+0000","last_fullsized":"2021-02-10T04:56:34.038611+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:25.745577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.745577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"1004'1093","reported_seq":"6456","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.035596+0000","last_change":"2021-02-10T04:55:28.097530+0000","last_active":"2021-02-10T04:56:34.035596+0000","last_peered":"2021-02-10T04:56:34.035596+0000","last_clean":"2021-02-10T04:56:34.035596+0000","last_became_active":"2021-02-10T04:55:14.627630+0000","last_became_peered":"2021-02-10T04:55:14.627630+0000","last_unstale":"2021-02-10T04:56:34.035596+0000","last_undegraded":"2021-02-10T04:56:34.035596+0000","last_fullsized":"2021-02-10T04:56:34.035596+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39434696,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":279,"num_read_kb":311058,"num_write":185,"num_write_kb":66790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"62'5","reported_seq":"74","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.579973+0000","last_change":"2021-02-10T04:54:39.559473+0000","last_active":"2021-02-10T04:56:33.579973+0000","last_peered":"2021-02-10T04:56:33.579973+0000","last_clean":"2021-02-10T04:56:33.579973+0000","last_became_active":"2021-02-10T04:54:39.559048+0000","last_became_peered":"2021-02-10T04:54:39.559048+0000","last_unstale":"2021-02-10T04:56:33.579973+0000","last_undegraded":"2021-02-10T04:56:33.579973+0000","last_fullsized":"2021-02-10T04:56:33.579973+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":149,"num_read_kb":148,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"1230","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.579919+0000","last_change":"2021-02-10T04:55:03.230261+0000","last_active":"2021-02-10T04:56:33.579919+0000","last_peered":"2021-02-10T04:56:33.579919+0000","last_clean":"2021-02-10T04:56:33.579919+0000","last_became_active":"2021-02-10T04:51:36.841582+0000","last_became_peered":"2021-02-10T04:51:36.841582+0000","last_unstale":"2021-02-10T04:56:33.579919+0000","last_undegraded":"2021-02-10T04:56:33.579919+0000","last_fullsized":"2021-02-10T04:56:33.579919+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:12.389272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:12.389272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"1261'71","reported_seq":"978","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.036651+0000","last_change":"2021-02-10T04:53:27.560299+0000","last_active":"2021-02-10T04:56:34.036651+0000","last_peered":"2021-02-10T04:56:34.036651+0000","last_clean":"2021-02-10T04:56:34.036651+0000","last_became_active":"2021-02-10T04:53:27.559886+0000","last_became_peered":"2021-02-10T04:53:27.559886+0000","last_unstale":"2021-02-10T04:56:34.036651+0000","last_undegraded":"2021-02-10T04:56:34.036651+0000","last_fullsized":"2021-02-10T04:56:34.036651+0000","mapping_epoch":1129,"log_start":"1049'51","ondisk_log_start":"1049'51","created":55,"last_epoch_clean":1173,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'41","last_scrub_stamp":"2021-02-10T04:51:58.353660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:58.353660+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1472,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":169,"num_read_kb":136,"num_write":102,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"935","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.033610+0000","last_change":"2021-02-10T04:53:11.271557+0000","last_active":"2021-02-10T04:56:34.033610+0000","last_peered":"2021-02-10T04:56:34.033610+0000","last_clean":"2021-02-10T04:56:34.033610+0000","last_became_active":"2021-02-10T04:51:49.567883+0000","last_became_peered":"2021-02-10T04:51:49.567883+0000","last_unstale":"2021-02-10T04:56:34.033610+0000","last_undegraded":"2021-02-10T04:56:34.033610+0000","last_fullsized":"2021-02-10T04:56:34.033610+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:11.271459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.271459+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"1229'5","reported_seq":"1633","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.038546+0000","last_change":"2021-02-10T04:55:14.638050+0000","last_active":"2021-02-10T04:56:34.038546+0000","last_peered":"2021-02-10T04:56:34.038546+0000","last_clean":"2021-02-10T04:56:34.038546+0000","last_became_active":"2021-02-10T04:55:14.637570+0000","last_became_peered":"2021-02-10T04:55:14.637570+0000","last_unstale":"2021-02-10T04:56:34.038546+0000","last_undegraded":"2021-02-10T04:56:34.038546+0000","last_fullsized":"2021-02-10T04:56:34.038546+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:51:47.855283+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.855283+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"1265","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.579875+0000","last_change":"2021-02-10T04:55:22.183630+0000","last_active":"2021-02-10T04:56:33.579875+0000","last_peered":"2021-02-10T04:56:33.579875+0000","last_clean":"2021-02-10T04:56:33.579875+0000","last_became_active":"2021-02-10T04:55:14.656661+0000","last_became_peered":"2021-02-10T04:55:14.656661+0000","last_unstale":"2021-02-10T04:56:33.579875+0000","last_undegraded":"2021-02-10T04:56:33.579875+0000","last_fullsized":"2021-02-10T04:56:33.579875+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:55:22.183566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:55:22.183566+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"1002'1090","reported_seq":"5468","reported_epoch":"1274","state":"active+clean","last_fresh":"2021-02-10T04:56:30.669117+0000","last_change":"2021-02-10T04:52:26.030198+0000","last_active":"2021-02-10T04:56:30.669117+0000","last_peered":"2021-02-10T04:56:30.669117+0000","last_clean":"2021-02-10T04:56:30.669117+0000","last_became_active":"2021-02-10T04:52:26.028182+0000","last_became_peered":"2021-02-10T04:52:26.028182+0000","last_unstale":"2021-02-10T04:56:30.669117+0000","last_undegraded":"2021-02-10T04:56:30.669117+0000","last_fullsized":"2021-02-10T04:56:30.669117+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:51:08.220472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.220472+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24410646,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":245,"num_read_kb":276168,"num_write":175,"num_write_kb":64960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"1241","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.035766+0000","last_change":"2021-02-10T04:54:39.555379+0000","last_active":"2021-02-10T04:56:34.035766+0000","last_peered":"2021-02-10T04:56:34.035766+0000","last_clean":"2021-02-10T04:56:34.035766+0000","last_became_active":"2021-02-10T04:54:39.554976+0000","last_became_peered":"2021-02-10T04:54:39.554976+0000","last_unstale":"2021-02-10T04:56:34.035766+0000","last_undegraded":"2021-02-10T04:56:34.035766+0000","last_fullsized":"2021-02-10T04:56:34.035766+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"1261'65","reported_seq":"1373","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.038354+0000","last_change":"2021-02-10T04:53:31.624862+0000","last_active":"2021-02-10T04:56:34.038354+0000","last_peered":"2021-02-10T04:56:34.038354+0000","last_clean":"2021-02-10T04:56:34.038354+0000","last_became_active":"2021-02-10T04:53:31.624318+0000","last_became_peered":"2021-02-10T04:53:31.624318+0000","last_unstale":"2021-02-10T04:56:34.038354+0000","last_undegraded":"2021-02-10T04:56:34.038354+0000","last_fullsized":"2021-02-10T04:56:34.038354+0000","mapping_epoch":55,"log_start":"1049'47","ondisk_log_start":"1049'47","created":55,"last_epoch_clean":1177,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'47","last_scrub_stamp":"2021-02-10T04:53:09.002368+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.002368+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1672,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":169,"num_read_kb":135,"num_write":98,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":934,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"212","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.578640+0000","last_change":"2021-02-10T04:55:14.611324+0000","last_active":"2021-02-10T04:56:33.578640+0000","last_peered":"2021-02-10T04:56:33.578640+0000","last_clean":"2021-02-10T04:56:33.578640+0000","last_became_active":"2021-02-10T04:55:14.610986+0000","last_became_peered":"2021-02-10T04:55:14.610986+0000","last_unstale":"2021-02-10T04:56:33.578640+0000","last_undegraded":"2021-02-10T04:56:33.578640+0000","last_fullsized":"2021-02-10T04:56:33.578640+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:08.358300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:50:08.358300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"1252","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.580343+0000","last_change":"2021-02-10T04:51:57.436961+0000","last_active":"2021-02-10T04:56:33.580343+0000","last_peered":"2021-02-10T04:56:33.580343+0000","last_clean":"2021-02-10T04:56:33.580343+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:56:33.580343+0000","last_undegraded":"2021-02-10T04:56:33.580343+0000","last_fullsized":"2021-02-10T04:56:33.580343+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.436862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.436862+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"1207","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.582053+0000","last_change":"2021-02-10T04:53:02.112612+0000","last_active":"2021-02-10T04:56:33.582053+0000","last_peered":"2021-02-10T04:56:33.582053+0000","last_clean":"2021-02-10T04:56:33.582053+0000","last_became_active":"2021-02-10T04:50:59.545501+0000","last_became_peered":"2021-02-10T04:50:59.545501+0000","last_unstale":"2021-02-10T04:56:33.582053+0000","last_undegraded":"2021-02-10T04:56:33.582053+0000","last_fullsized":"2021-02-10T04:56:33.582053+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:02.112533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:02.112533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"1001'1165","reported_seq":"4447","reported_epoch":"1274","state":"active+clean","last_fresh":"2021-02-10T04:56:30.668544+0000","last_change":"2021-02-10T04:55:14.653651+0000","last_active":"2021-02-10T04:56:30.668544+0000","last_peered":"2021-02-10T04:56:30.668544+0000","last_clean":"2021-02-10T04:56:30.668544+0000","last_became_active":"2021-02-10T04:55:14.652235+0000","last_became_peered":"2021-02-10T04:55:14.652235+0000","last_unstale":"2021-02-10T04:56:30.668544+0000","last_undegraded":"2021-02-10T04:56:30.668544+0000","last_fullsized":"2021-02-10T04:56:30.668544+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29601845,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":286,"num_read_kb":299529,"num_write":200,"num_write_kb":89130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"57","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.034807+0000","last_change":"2021-02-10T04:55:15.590113+0000","last_active":"2021-02-10T04:56:34.034807+0000","last_peered":"2021-02-10T04:56:34.034807+0000","last_clean":"2021-02-10T04:56:34.034807+0000","last_became_active":"2021-02-10T04:55:15.589751+0000","last_became_peered":"2021-02-10T04:55:15.589751+0000","last_unstale":"2021-02-10T04:56:34.034807+0000","last_undegraded":"2021-02-10T04:56:34.034807+0000","last_fullsized":"2021-02-10T04:56:34.034807+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"1113'5","reported_seq":"1519","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.580324+0000","last_change":"2021-02-10T04:52:34.047474+0000","last_active":"2021-02-10T04:56:33.580324+0000","last_peered":"2021-02-10T04:56:33.580324+0000","last_clean":"2021-02-10T04:56:33.580324+0000","last_became_active":"2021-02-10T04:50:59.569578+0000","last_became_peered":"2021-02-10T04:50:59.569578+0000","last_unstale":"2021-02-10T04:56:33.580324+0000","last_undegraded":"2021-02-10T04:56:33.580324+0000","last_fullsized":"2021-02-10T04:56:33.580324+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1113'5","last_scrub_stamp":"2021-02-10T04:52:34.047358+0000","last_deep_scrub":"158'3","last_deep_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_clean_scrub_stamp":"2021-02-10T04:52:34.047358+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"1261'68","reported_seq":"1358","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.036101+0000","last_change":"2021-02-10T04:51:48.872937+0000","last_active":"2021-02-10T04:56:34.036101+0000","last_peered":"2021-02-10T04:56:34.036101+0000","last_clean":"2021-02-10T04:56:34.036101+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:56:34.036101+0000","last_undegraded":"2021-02-10T04:56:34.036101+0000","last_fullsized":"2021-02-10T04:56:34.036101+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:51:48.872839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.872839+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":114,"num_read_kb":94,"num_write":68,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"1325","reported_epoch":"1274","state":"active+clean","last_fresh":"2021-02-10T04:56:30.669760+0000","last_change":"2021-02-10T04:53:05.984747+0000","last_active":"2021-02-10T04:56:30.669760+0000","last_peered":"2021-02-10T04:56:30.669760+0000","last_clean":"2021-02-10T04:56:30.669760+0000","last_became_active":"2021-02-10T04:51:36.869931+0000","last_became_peered":"2021-02-10T04:51:36.869931+0000","last_unstale":"2021-02-10T04:56:30.669760+0000","last_undegraded":"2021-02-10T04:56:30.669760+0000","last_fullsized":"2021-02-10T04:56:30.669760+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:05.984659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:05.984659+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"1129","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.580445+0000","last_change":"2021-02-10T04:53:46.731709+0000","last_active":"2021-02-10T04:56:33.580445+0000","last_peered":"2021-02-10T04:56:33.580445+0000","last_clean":"2021-02-10T04:56:33.580445+0000","last_became_active":"2021-02-10T04:53:46.731340+0000","last_became_peered":"2021-02-10T04:53:46.731340+0000","last_unstale":"2021-02-10T04:56:33.580445+0000","last_undegraded":"2021-02-10T04:56:33.580445+0000","last_fullsized":"2021-02-10T04:56:33.580445+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:09.005421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.005421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"998'1009","reported_seq":"4766","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.581457+0000","last_change":"2021-02-10T04:52:26.037227+0000","last_active":"2021-02-10T04:56:33.581457+0000","last_peered":"2021-02-10T04:56:33.581457+0000","last_clean":"2021-02-10T04:56:33.581457+0000","last_became_active":"2021-02-10T04:52:26.036723+0000","last_became_peered":"2021-02-10T04:52:26.036723+0000","last_unstale":"2021-02-10T04:56:33.581457+0000","last_undegraded":"2021-02-10T04:56:33.581457+0000","last_fullsized":"2021-02-10T04:56:33.581457+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12355206,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":119,"num_read_kb":124701,"num_write":77,"num_write_kb":25052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4145029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"1146","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.582536+0000","last_change":"2021-02-10T04:53:33.305896+0000","last_active":"2021-02-10T04:56:33.582536+0000","last_peered":"2021-02-10T04:56:33.582536+0000","last_clean":"2021-02-10T04:56:33.582536+0000","last_became_active":"2021-02-10T04:50:59.570854+0000","last_became_peered":"2021-02-10T04:50:59.570854+0000","last_unstale":"2021-02-10T04:56:33.582536+0000","last_undegraded":"2021-02-10T04:56:33.582536+0000","last_fullsized":"2021-02-10T04:56:33.582536+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:33.305827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:33.305827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"1247","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.584188+0000","last_change":"2021-02-10T04:55:15.669988+0000","last_active":"2021-02-10T04:56:33.584188+0000","last_peered":"2021-02-10T04:56:33.584188+0000","last_clean":"2021-02-10T04:56:33.584188+0000","last_became_active":"2021-02-10T04:55:15.669642+0000","last_became_peered":"2021-02-10T04:55:15.669642+0000","last_unstale":"2021-02-10T04:56:33.584188+0000","last_undegraded":"2021-02-10T04:56:33.584188+0000","last_fullsized":"2021-02-10T04:56:33.584188+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"1260","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.036375+0000","last_change":"2021-02-10T04:51:50.895471+0000","last_active":"2021-02-10T04:56:34.036375+0000","last_peered":"2021-02-10T04:56:34.036375+0000","last_clean":"2021-02-10T04:56:34.036375+0000","last_became_active":"2021-02-10T04:50:59.527553+0000","last_became_peered":"2021-02-10T04:50:59.527553+0000","last_unstale":"2021-02-10T04:56:34.036375+0000","last_undegraded":"2021-02-10T04:56:34.036375+0000","last_fullsized":"2021-02-10T04:56:34.036375+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.895370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.895370+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"1261'84","reported_seq":"1391","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.580830+0000","last_change":"2021-02-10T04:55:14.603109+0000","last_active":"2021-02-10T04:56:33.580830+0000","last_peered":"2021-02-10T04:56:33.580830+0000","last_clean":"2021-02-10T04:56:33.580830+0000","last_became_active":"2021-02-10T04:55:14.602669+0000","last_became_peered":"2021-02-10T04:55:14.602669+0000","last_unstale":"2021-02-10T04:56:33.580830+0000","last_undegraded":"2021-02-10T04:56:33.580830+0000","last_fullsized":"2021-02-10T04:56:33.580830+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'72","last_scrub_stamp":"2021-02-10T04:46:30.515026+0000","last_deep_scrub":"152'48","last_deep_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_clean_scrub_stamp":"2021-02-10T04:46:30.515026+0000","log_size":84,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":123,"num_read_kb":101,"num_write":72,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"529","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.034773+0000","last_change":"2021-02-10T04:55:03.834050+0000","last_active":"2021-02-10T04:56:34.034773+0000","last_peered":"2021-02-10T04:56:34.034773+0000","last_clean":"2021-02-10T04:56:34.034773+0000","last_became_active":"2021-02-10T04:51:49.503801+0000","last_became_peered":"2021-02-10T04:51:49.503801+0000","last_unstale":"2021-02-10T04:56:34.034773+0000","last_undegraded":"2021-02-10T04:56:34.034773+0000","last_fullsized":"2021-02-10T04:56:34.034773+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:51:48.274960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.274960+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"1285","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.036681+0000","last_change":"2021-02-10T04:53:13.099288+0000","last_active":"2021-02-10T04:56:34.036681+0000","last_peered":"2021-02-10T04:56:34.036681+0000","last_clean":"2021-02-10T04:56:34.036681+0000","last_became_active":"2021-02-10T04:51:36.848614+0000","last_became_peered":"2021-02-10T04:51:36.848614+0000","last_unstale":"2021-02-10T04:56:34.036681+0000","last_undegraded":"2021-02-10T04:56:34.036681+0000","last_fullsized":"2021-02-10T04:56:34.036681+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:13.099228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:13.099228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"1002'1657","reported_seq":"5336","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.583178+0000","last_change":"2021-02-10T04:52:26.036806+0000","last_active":"2021-02-10T04:56:33.583178+0000","last_peered":"2021-02-10T04:56:33.583178+0000","last_clean":"2021-02-10T04:56:33.583178+0000","last_became_active":"2021-02-10T04:52:26.032363+0000","last_became_peered":"2021-02-10T04:52:26.032363+0000","last_unstale":"2021-02-10T04:56:33.583178+0000","last_undegraded":"2021-02-10T04:56:33.583178+0000","last_fullsized":"2021-02-10T04:56:33.583178+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21396533,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":212,"num_read_kb":218836,"num_write":159,"num_write_kb":59398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":22326014,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"1204","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.037773+0000","last_change":"2021-02-10T04:51:51.911945+0000","last_active":"2021-02-10T04:56:34.037773+0000","last_peered":"2021-02-10T04:56:34.037773+0000","last_clean":"2021-02-10T04:56:34.037773+0000","last_became_active":"2021-02-10T04:51:36.850015+0000","last_became_peered":"2021-02-10T04:51:36.850015+0000","last_unstale":"2021-02-10T04:56:34.037773+0000","last_undegraded":"2021-02-10T04:56:34.037773+0000","last_fullsized":"2021-02-10T04:56:34.037773+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.911851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.911851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"62'5","reported_seq":"1108","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.034484+0000","last_change":"2021-02-10T04:54:39.568253+0000","last_active":"2021-02-10T04:56:34.034484+0000","last_peered":"2021-02-10T04:56:34.034484+0000","last_clean":"2021-02-10T04:56:34.034484+0000","last_became_active":"2021-02-10T04:54:39.567642+0000","last_became_peered":"2021-02-10T04:54:39.567642+0000","last_unstale":"2021-02-10T04:56:34.034484+0000","last_undegraded":"2021-02-10T04:56:34.034484+0000","last_fullsized":"2021-02-10T04:56:34.034484+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":98,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":150,"num_read_kb":148,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"1332","reported_epoch":"1274","state":"active+clean","last_fresh":"2021-02-10T04:56:30.668829+0000","last_change":"2021-02-10T04:55:14.634577+0000","last_active":"2021-02-10T04:56:30.668829+0000","last_peered":"2021-02-10T04:56:30.668829+0000","last_clean":"2021-02-10T04:56:30.668829+0000","last_became_active":"2021-02-10T04:55:14.634259+0000","last_became_peered":"2021-02-10T04:55:14.634259+0000","last_unstale":"2021-02-10T04:56:30.668829+0000","last_undegraded":"2021-02-10T04:56:30.668829+0000","last_fullsized":"2021-02-10T04:56:30.668829+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:58.142526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:58.142526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"1261'68","reported_seq":"1284","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.035355+0000","last_change":"2021-02-10T04:55:28.079376+0000","last_active":"2021-02-10T04:56:34.035355+0000","last_peered":"2021-02-10T04:56:34.035355+0000","last_clean":"2021-02-10T04:56:34.035355+0000","last_became_active":"2021-02-10T04:55:14.654609+0000","last_became_peered":"2021-02-10T04:55:14.654609+0000","last_unstale":"2021-02-10T04:56:34.035355+0000","last_undegraded":"2021-02-10T04:56:34.035355+0000","last_fullsized":"2021-02-10T04:56:34.035355+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1135'58","last_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_deep_scrub":"1135'58","last_deep_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.483067+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":102,"num_read_kb":83,"num_write":60,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"210","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.581325+0000","last_change":"2021-02-10T04:55:14.657913+0000","last_active":"2021-02-10T04:56:33.581325+0000","last_peered":"2021-02-10T04:56:33.581325+0000","last_clean":"2021-02-10T04:56:33.581325+0000","last_became_active":"2021-02-10T04:55:14.657153+0000","last_became_peered":"2021-02-10T04:55:14.657153+0000","last_unstale":"2021-02-10T04:56:33.581325+0000","last_undegraded":"2021-02-10T04:56:33.581325+0000","last_fullsized":"2021-02-10T04:56:33.581325+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:50:46.434629+0000","last_deep_scrub":"53'2","last_deep_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_clean_scrub_stamp":"2021-02-10T04:50:46.434629+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"952","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.033749+0000","last_change":"2021-02-10T04:52:38.278980+0000","last_active":"2021-02-10T04:56:34.033749+0000","last_peered":"2021-02-10T04:56:34.033749+0000","last_clean":"2021-02-10T04:56:34.033749+0000","last_became_active":"2021-02-10T04:51:49.496237+0000","last_became_peered":"2021-02-10T04:51:49.496237+0000","last_unstale":"2021-02-10T04:56:34.033749+0000","last_undegraded":"2021-02-10T04:56:34.033749+0000","last_fullsized":"2021-02-10T04:56:34.033749+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:38.278910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:52:38.278910+0000","last_clean_scrub_stamp":"2021-02-10T04:52:38.278910+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"1000'1071","reported_seq":"4826","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.577932+0000","last_change":"2021-02-10T04:53:34.407875+0000","last_active":"2021-02-10T04:56:33.577932+0000","last_peered":"2021-02-10T04:56:33.577932+0000","last_clean":"2021-02-10T04:56:33.577932+0000","last_became_active":"2021-02-10T04:52:26.031050+0000","last_became_peered":"2021-02-10T04:52:26.031050+0000","last_unstale":"2021-02-10T04:56:33.577932+0000","last_undegraded":"2021-02-10T04:56:33.577932+0000","last_fullsized":"2021-02-10T04:56:33.577932+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:34.407784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.407784+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30173843,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"1298","reported_epoch":"1274","state":"active+clean","last_fresh":"2021-02-10T04:56:30.669199+0000","last_change":"2021-02-10T04:55:14.637570+0000","last_active":"2021-02-10T04:56:30.669199+0000","last_peered":"2021-02-10T04:56:30.669199+0000","last_clean":"2021-02-10T04:56:30.669199+0000","last_became_active":"2021-02-10T04:55:14.637203+0000","last_became_peered":"2021-02-10T04:55:14.637203+0000","last_unstale":"2021-02-10T04:56:30.669199+0000","last_undegraded":"2021-02-10T04:56:30.669199+0000","last_fullsized":"2021-02-10T04:56:30.669199+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_clean_scrub_stamp":"2021-02-10T04:51:00.195518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"1262","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.580414+0000","last_change":"2021-02-10T04:55:14.661285+0000","last_active":"2021-02-10T04:56:33.580414+0000","last_peered":"2021-02-10T04:56:33.580414+0000","last_clean":"2021-02-10T04:56:33.580414+0000","last_became_active":"2021-02-10T04:55:14.660902+0000","last_became_peered":"2021-02-10T04:55:14.660902+0000","last_unstale":"2021-02-10T04:56:33.580414+0000","last_undegraded":"2021-02-10T04:56:33.580414+0000","last_fullsized":"2021-02-10T04:56:33.580414+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:55.473347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:50:55.473347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"1261'58","reported_seq":"1335","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.035279+0000","last_change":"2021-02-10T04:55:47.652017+0000","last_active":"2021-02-10T04:56:34.035279+0000","last_peered":"2021-02-10T04:56:34.035279+0000","last_clean":"2021-02-10T04:56:34.035279+0000","last_became_active":"2021-02-10T04:55:47.651592+0000","last_became_peered":"2021-02-10T04:55:47.651592+0000","last_unstale":"2021-02-10T04:56:34.035279+0000","last_undegraded":"2021-02-10T04:56:34.035279+0000","last_fullsized":"2021-02-10T04:56:34.035279+0000","mapping_epoch":1227,"log_start":"1049'40","ondisk_log_start":"1049'40","created":55,"last_epoch_clean":1257,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'40","last_scrub_stamp":"2021-02-10T04:51:47.281929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.281929+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1836,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":171,"num_read_kb":135,"num_write":99,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"436","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.578716+0000","last_change":"2021-02-10T04:55:14.610765+0000","last_active":"2021-02-10T04:56:33.578716+0000","last_peered":"2021-02-10T04:56:33.578716+0000","last_clean":"2021-02-10T04:56:33.578716+0000","last_became_active":"2021-02-10T04:55:14.610470+0000","last_became_peered":"2021-02-10T04:55:14.610470+0000","last_unstale":"2021-02-10T04:56:33.578716+0000","last_undegraded":"2021-02-10T04:56:33.578716+0000","last_fullsized":"2021-02-10T04:56:33.578716+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:50.399350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:50.399350+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"1162","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.033606+0000","last_change":"2021-02-10T04:55:25.624031+0000","last_active":"2021-02-10T04:56:34.033606+0000","last_peered":"2021-02-10T04:56:34.033606+0000","last_clean":"2021-02-10T04:56:34.033606+0000","last_became_active":"2021-02-10T04:55:14.630129+0000","last_became_peered":"2021-02-10T04:55:14.630129+0000","last_unstale":"2021-02-10T04:56:34.033606+0000","last_undegraded":"2021-02-10T04:56:34.033606+0000","last_fullsized":"2021-02-10T04:56:34.033606+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"937","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.036537+0000","last_change":"2021-02-10T04:52:31.273037+0000","last_active":"2021-02-10T04:56:34.036537+0000","last_peered":"2021-02-10T04:56:34.036537+0000","last_clean":"2021-02-10T04:56:34.036537+0000","last_became_active":"2021-02-10T04:51:49.501405+0000","last_became_peered":"2021-02-10T04:51:49.501405+0000","last_unstale":"2021-02-10T04:56:34.036537+0000","last_undegraded":"2021-02-10T04:56:34.036537+0000","last_fullsized":"2021-02-10T04:56:34.036537+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:31.272971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:17.598113+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.272971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"1000'446","reported_seq":"1993","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.581164+0000","last_change":"2021-02-10T04:55:14.658783+0000","last_active":"2021-02-10T04:56:33.581164+0000","last_peered":"2021-02-10T04:56:33.581164+0000","last_clean":"2021-02-10T04:56:33.581164+0000","last_became_active":"2021-02-10T04:55:14.657668+0000","last_became_peered":"2021-02-10T04:55:14.657668+0000","last_unstale":"2021-02-10T04:56:33.581164+0000","last_undegraded":"2021-02-10T04:56:33.581164+0000","last_fullsized":"2021-02-10T04:56:33.581164+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:51:45.964029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.964029+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22255852,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68435,"num_write":72,"num_write_kb":24345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":403693,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"428","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.034425+0000","last_change":"2021-02-10T04:53:03.228260+0000","last_active":"2021-02-10T04:56:34.034425+0000","last_peered":"2021-02-10T04:56:34.034425+0000","last_clean":"2021-02-10T04:56:34.034425+0000","last_became_active":"2021-02-10T04:51:49.499252+0000","last_became_peered":"2021-02-10T04:51:49.499252+0000","last_unstale":"2021-02-10T04:56:34.034425+0000","last_undegraded":"2021-02-10T04:56:34.034425+0000","last_fullsized":"2021-02-10T04:56:34.034425+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:03.228182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.228182+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"180","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.581019+0000","last_change":"2021-02-10T04:53:46.727384+0000","last_active":"2021-02-10T04:56:33.581019+0000","last_peered":"2021-02-10T04:56:33.581019+0000","last_clean":"2021-02-10T04:56:33.581019+0000","last_became_active":"2021-02-10T04:53:46.726771+0000","last_became_peered":"2021-02-10T04:53:46.726771+0000","last_unstale":"2021-02-10T04:56:33.581019+0000","last_undegraded":"2021-02-10T04:56:33.581019+0000","last_fullsized":"2021-02-10T04:56:33.581019+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:33.948348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:33.948348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"1261'51","reported_seq":"1360","reported_epoch":"1274","state":"active+clean","last_fresh":"2021-02-10T04:56:30.668748+0000","last_change":"2021-02-10T04:54:17.118810+0000","last_active":"2021-02-10T04:56:30.668748+0000","last_peered":"2021-02-10T04:56:30.668748+0000","last_clean":"2021-02-10T04:56:30.668748+0000","last_became_active":"2021-02-10T04:54:17.080053+0000","last_became_peered":"2021-02-10T04:54:17.080053+0000","last_unstale":"2021-02-10T04:56:30.668748+0000","last_undegraded":"2021-02-10T04:56:30.668748+0000","last_fullsized":"2021-02-10T04:56:30.668748+0000","mapping_epoch":1200,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1201,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'39","last_scrub_stamp":"2021-02-10T04:51:47.322331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.322331+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":51,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"208","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.578803+0000","last_change":"2021-02-10T04:55:14.636888+0000","last_active":"2021-02-10T04:56:33.578803+0000","last_peered":"2021-02-10T04:56:33.578803+0000","last_clean":"2021-02-10T04:56:33.578803+0000","last_became_active":"2021-02-10T04:55:14.634305+0000","last_became_peered":"2021-02-10T04:55:14.634305+0000","last_unstale":"2021-02-10T04:56:33.578803+0000","last_undegraded":"2021-02-10T04:56:33.578803+0000","last_fullsized":"2021-02-10T04:56:33.578803+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_clean_scrub_stamp":"2021-02-10T04:50:05.424262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"1446","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.580724+0000","last_change":"2021-02-10T04:52:37.324122+0000","last_active":"2021-02-10T04:56:33.580724+0000","last_peered":"2021-02-10T04:56:33.580724+0000","last_clean":"2021-02-10T04:56:33.580724+0000","last_became_active":"2021-02-10T04:51:36.843553+0000","last_became_peered":"2021-02-10T04:51:36.843553+0000","last_unstale":"2021-02-10T04:56:33.580724+0000","last_undegraded":"2021-02-10T04:56:33.580724+0000","last_fullsized":"2021-02-10T04:56:33.580724+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:37.324040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:37.324040+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"1252","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.581545+0000","last_change":"2021-02-10T04:54:37.612303+0000","last_active":"2021-02-10T04:56:33.581545+0000","last_peered":"2021-02-10T04:56:33.581545+0000","last_clean":"2021-02-10T04:56:33.581545+0000","last_became_active":"2021-02-10T04:54:37.611953+0000","last_became_peered":"2021-02-10T04:54:37.611953+0000","last_unstale":"2021-02-10T04:56:33.581545+0000","last_undegraded":"2021-02-10T04:56:33.581545+0000","last_fullsized":"2021-02-10T04:56:33.581545+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"1274","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.580493+0000","last_change":"2021-02-10T04:52:31.403904+0000","last_active":"2021-02-10T04:56:33.580493+0000","last_peered":"2021-02-10T04:56:33.580493+0000","last_clean":"2021-02-10T04:56:33.580493+0000","last_became_active":"2021-02-10T04:50:59.538009+0000","last_became_peered":"2021-02-10T04:50:59.538009+0000","last_unstale":"2021-02-10T04:56:33.580493+0000","last_undegraded":"2021-02-10T04:56:33.580493+0000","last_fullsized":"2021-02-10T04:56:33.580493+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:31.403824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.403824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"1000'490","reported_seq":"3742","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.035436+0000","last_change":"2021-02-10T04:55:22.716472+0000","last_active":"2021-02-10T04:56:34.035436+0000","last_peered":"2021-02-10T04:56:34.035436+0000","last_clean":"2021-02-10T04:56:34.035436+0000","last_became_active":"2021-02-10T04:55:14.644596+0000","last_became_peered":"2021-02-10T04:55:14.644596+0000","last_unstale":"2021-02-10T04:56:34.035436+0000","last_undegraded":"2021-02-10T04:56:34.035436+0000","last_fullsized":"2021-02-10T04:56:34.035436+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:47:02.875920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:02.875920+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19371627,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":141,"num_read_kb":148568,"num_write":85,"num_write_kb":31732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"172","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.578754+0000","last_change":"2021-02-10T04:53:01.972991+0000","last_active":"2021-02-10T04:56:33.578754+0000","last_peered":"2021-02-10T04:56:33.578754+0000","last_clean":"2021-02-10T04:56:33.578754+0000","last_became_active":"2021-02-10T04:51:36.863976+0000","last_became_peered":"2021-02-10T04:51:36.863976+0000","last_unstale":"2021-02-10T04:56:33.578754+0000","last_undegraded":"2021-02-10T04:56:33.578754+0000","last_fullsized":"2021-02-10T04:56:33.578754+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:01.972913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.972913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"1261'40","reported_seq":"994","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.034886+0000","last_change":"2021-02-10T04:55:43.569281+0000","last_active":"2021-02-10T04:56:34.034886+0000","last_peered":"2021-02-10T04:56:34.034886+0000","last_clean":"2021-02-10T04:56:34.034886+0000","last_became_active":"2021-02-10T04:55:43.568878+0000","last_became_peered":"2021-02-10T04:55:43.568878+0000","last_unstale":"2021-02-10T04:56:34.034886+0000","last_undegraded":"2021-02-10T04:56:34.034886+0000","last_fullsized":"2021-02-10T04:56:34.034886+0000","mapping_epoch":1227,"log_start":"1063'30","ondisk_log_start":"1063'30","created":55,"last_epoch_clean":1253,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:56.431187+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.431187+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":802,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":116,"num_read_kb":94,"num_write":67,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"182","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.577446+0000","last_change":"2021-02-10T04:55:14.636849+0000","last_active":"2021-02-10T04:56:33.577446+0000","last_peered":"2021-02-10T04:56:33.577446+0000","last_clean":"2021-02-10T04:56:33.577446+0000","last_became_active":"2021-02-10T04:55:14.636237+0000","last_became_peered":"2021-02-10T04:55:14.636237+0000","last_unstale":"2021-02-10T04:56:33.577446+0000","last_undegraded":"2021-02-10T04:56:33.577446+0000","last_fullsized":"2021-02-10T04:56:33.577446+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:04.591695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:04.591695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"1101","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.034021+0000","last_change":"2021-02-10T04:54:37.680913+0000","last_active":"2021-02-10T04:56:34.034021+0000","last_peered":"2021-02-10T04:56:34.034021+0000","last_clean":"2021-02-10T04:56:34.034021+0000","last_became_active":"2021-02-10T04:54:37.680651+0000","last_became_peered":"2021-02-10T04:54:37.680651+0000","last_unstale":"2021-02-10T04:56:34.034021+0000","last_undegraded":"2021-02-10T04:56:34.034021+0000","last_fullsized":"2021-02-10T04:56:34.034021+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1210,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":149,"num_read_kb":148,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"1119","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.581282+0000","last_change":"2021-02-10T04:53:46.729138+0000","last_active":"2021-02-10T04:56:33.581282+0000","last_peered":"2021-02-10T04:56:33.581282+0000","last_clean":"2021-02-10T04:56:33.581282+0000","last_became_active":"2021-02-10T04:53:46.728845+0000","last_became_peered":"2021-02-10T04:53:46.728845+0000","last_unstale":"2021-02-10T04:56:33.581282+0000","last_undegraded":"2021-02-10T04:56:33.581282+0000","last_fullsized":"2021-02-10T04:56:33.581282+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:40.066354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:52:40.066354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"1000'620","reported_seq":"4366","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.034911+0000","last_change":"2021-02-10T04:55:28.089086+0000","last_active":"2021-02-10T04:56:34.034911+0000","last_peered":"2021-02-10T04:56:34.034911+0000","last_clean":"2021-02-10T04:56:34.034911+0000","last_became_active":"2021-02-10T04:55:14.640509+0000","last_became_peered":"2021-02-10T04:55:14.640509+0000","last_unstale":"2021-02-10T04:56:34.034911+0000","last_undegraded":"2021-02-10T04:56:34.034911+0000","last_fullsized":"2021-02-10T04:56:34.034911+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:51:54.444287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:54.444287+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14265195,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":144,"num_read_kb":128482,"num_write":129,"num_write_kb":53443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"1155","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.034767+0000","last_change":"2021-02-10T04:55:28.083286+0000","last_active":"2021-02-10T04:56:34.034767+0000","last_peered":"2021-02-10T04:56:34.034767+0000","last_clean":"2021-02-10T04:56:34.034767+0000","last_became_active":"2021-02-10T04:55:14.619846+0000","last_became_peered":"2021-02-10T04:55:14.619846+0000","last_unstale":"2021-02-10T04:56:34.034767+0000","last_undegraded":"2021-02-10T04:56:34.034767+0000","last_fullsized":"2021-02-10T04:56:34.034767+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:14.955544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:47:14.955544+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"1252","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.036935+0000","last_change":"2021-02-10T04:55:15.590025+0000","last_active":"2021-02-10T04:56:34.036935+0000","last_peered":"2021-02-10T04:56:34.036935+0000","last_clean":"2021-02-10T04:56:34.036935+0000","last_became_active":"2021-02-10T04:55:15.589651+0000","last_became_peered":"2021-02-10T04:55:15.589651+0000","last_unstale":"2021-02-10T04:56:34.036935+0000","last_undegraded":"2021-02-10T04:56:34.036935+0000","last_fullsized":"2021-02-10T04:56:34.036935+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"1258","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.035073+0000","last_change":"2021-02-10T04:55:28.090104+0000","last_active":"2021-02-10T04:56:34.035073+0000","last_peered":"2021-02-10T04:56:34.035073+0000","last_clean":"2021-02-10T04:56:34.035073+0000","last_became_active":"2021-02-10T04:55:14.650106+0000","last_became_peered":"2021-02-10T04:55:14.650106+0000","last_unstale":"2021-02-10T04:56:34.035073+0000","last_undegraded":"2021-02-10T04:56:34.035073+0000","last_fullsized":"2021-02-10T04:56:34.035073+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:27.908221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:47:27.908221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"1261'67","reported_seq":"926","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.034788+0000","last_change":"2021-02-10T04:52:30.277009+0000","last_active":"2021-02-10T04:56:34.034788+0000","last_peered":"2021-02-10T04:56:34.034788+0000","last_clean":"2021-02-10T04:56:34.034788+0000","last_became_active":"2021-02-10T04:51:49.501622+0000","last_became_peered":"2021-02-10T04:51:49.501622+0000","last_unstale":"2021-02-10T04:56:34.034788+0000","last_undegraded":"2021-02-10T04:56:34.034788+0000","last_fullsized":"2021-02-10T04:56:34.034788+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'55","last_scrub_stamp":"2021-02-10T04:52:30.276938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.276938+0000","log_size":67,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":116,"num_read_kb":92,"num_write":70,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"1270","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.580291+0000","last_change":"2021-02-10T04:51:59.420149+0000","last_active":"2021-02-10T04:56:33.580291+0000","last_peered":"2021-02-10T04:56:33.580291+0000","last_clean":"2021-02-10T04:56:33.580291+0000","last_became_active":"2021-02-10T04:28:21.125541+0000","last_became_peered":"2021-02-10T04:28:21.125541+0000","last_unstale":"2021-02-10T04:56:33.580291+0000","last_undegraded":"2021-02-10T04:56:33.580291+0000","last_fullsized":"2021-02-10T04:56:33.580291+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.420086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.420086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"1236","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.584014+0000","last_change":"2021-02-10T04:53:12.187225+0000","last_active":"2021-02-10T04:56:33.584014+0000","last_peered":"2021-02-10T04:56:33.584014+0000","last_clean":"2021-02-10T04:56:33.584014+0000","last_became_active":"2021-02-10T04:50:59.555689+0000","last_became_peered":"2021-02-10T04:50:59.555689+0000","last_unstale":"2021-02-10T04:56:33.584014+0000","last_undegraded":"2021-02-10T04:56:33.584014+0000","last_fullsized":"2021-02-10T04:56:33.584014+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.187154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.187154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"1002'2042","reported_seq":"9506","reported_epoch":"1274","state":"active+clean","last_fresh":"2021-02-10T04:56:30.669184+0000","last_change":"2021-02-10T04:55:14.662208+0000","last_active":"2021-02-10T04:56:30.669184+0000","last_peered":"2021-02-10T04:56:30.669184+0000","last_clean":"2021-02-10T04:56:30.669184+0000","last_became_active":"2021-02-10T04:55:14.659601+0000","last_became_peered":"2021-02-10T04:55:14.659601+0000","last_unstale":"2021-02-10T04:56:30.669184+0000","last_undegraded":"2021-02-10T04:56:30.669184+0000","last_fullsized":"2021-02-10T04:56:30.669184+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:46:57.054230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:57.054230+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":85678898,"num_objects":35,"num_object_clones":29,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"1221","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.578229+0000","last_change":"2021-02-10T04:55:14.651301+0000","last_active":"2021-02-10T04:56:33.578229+0000","last_peered":"2021-02-10T04:56:33.578229+0000","last_clean":"2021-02-10T04:56:33.578229+0000","last_became_active":"2021-02-10T04:55:14.650386+0000","last_became_peered":"2021-02-10T04:55:14.650386+0000","last_unstale":"2021-02-10T04:56:33.578229+0000","last_undegraded":"2021-02-10T04:56:33.578229+0000","last_fullsized":"2021-02-10T04:56:33.578229+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:52.341527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:50:52.341527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"74","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.579575+0000","last_change":"2021-02-10T04:54:39.558380+0000","last_active":"2021-02-10T04:56:33.579575+0000","last_peered":"2021-02-10T04:56:33.579575+0000","last_clean":"2021-02-10T04:56:33.579575+0000","last_became_active":"2021-02-10T04:54:39.557995+0000","last_became_peered":"2021-02-10T04:54:39.557995+0000","last_unstale":"2021-02-10T04:56:33.579575+0000","last_undegraded":"2021-02-10T04:56:33.579575+0000","last_fullsized":"2021-02-10T04:56:33.579575+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"1238'4","reported_seq":"1464","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.035039+0000","last_change":"2021-02-10T04:55:28.095426+0000","last_active":"2021-02-10T04:56:34.035039+0000","last_peered":"2021-02-10T04:56:34.035039+0000","last_clean":"2021-02-10T04:56:34.035039+0000","last_became_active":"2021-02-10T04:55:14.645890+0000","last_became_peered":"2021-02-10T04:55:14.645890+0000","last_unstale":"2021-02-10T04:56:34.035039+0000","last_undegraded":"2021-02-10T04:56:34.035039+0000","last_fullsized":"2021-02-10T04:56:34.035039+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:54.466259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:54.466259+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"1261'64","reported_seq":"972","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.034443+0000","last_change":"2021-02-10T04:53:01.304693+0000","last_active":"2021-02-10T04:56:34.034443+0000","last_peered":"2021-02-10T04:56:34.034443+0000","last_clean":"2021-02-10T04:56:34.034443+0000","last_became_active":"2021-02-10T04:51:49.498066+0000","last_became_peered":"2021-02-10T04:51:49.498066+0000","last_unstale":"2021-02-10T04:56:34.034443+0000","last_undegraded":"2021-02-10T04:56:34.034443+0000","last_fullsized":"2021-02-10T04:56:34.034443+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:53:01.304626+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.304626+0000","log_size":64,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":89,"num_read_kb":71,"num_write":52,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"533","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.034870+0000","last_change":"2021-02-10T04:55:14.610552+0000","last_active":"2021-02-10T04:56:34.034870+0000","last_peered":"2021-02-10T04:56:34.034870+0000","last_clean":"2021-02-10T04:56:34.034870+0000","last_became_active":"2021-02-10T04:55:14.609458+0000","last_became_peered":"2021-02-10T04:55:14.609458+0000","last_unstale":"2021-02-10T04:56:34.034870+0000","last_undegraded":"2021-02-10T04:56:34.034870+0000","last_fullsized":"2021-02-10T04:56:34.034870+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:49.447961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:49.447961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"1216","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.033248+0000","last_change":"2021-02-10T04:56:25.909251+0000","last_active":"2021-02-10T04:56:34.033248+0000","last_peered":"2021-02-10T04:56:34.033248+0000","last_clean":"2021-02-10T04:56:34.033248+0000","last_became_active":"2021-02-10T04:55:14.614581+0000","last_became_peered":"2021-02-10T04:55:14.614581+0000","last_unstale":"2021-02-10T04:56:34.033248+0000","last_undegraded":"2021-02-10T04:56:34.033248+0000","last_fullsized":"2021-02-10T04:56:34.033248+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:07.914061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:07.914061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"293'77","reported_seq":"3057","reported_epoch":"1275","state":"active+clean+scrubbing","last_fresh":"2021-02-10T04:56:34.035593+0000","last_change":"2021-02-10T04:53:43.102301+0000","last_active":"2021-02-10T04:56:34.035593+0000","last_peered":"2021-02-10T04:56:34.035593+0000","last_clean":"2021-02-10T04:56:34.035593+0000","last_became_active":"2021-02-10T04:52:26.024812+0000","last_became_peered":"2021-02-10T04:52:26.024812+0000","last_unstale":"2021-02-10T04:56:34.035593+0000","last_undegraded":"2021-02-10T04:56:34.035593+0000","last_fullsized":"2021-02-10T04:56:34.035593+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1886585,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":18587,"num_write":9,"num_write_kb":3315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"1288","reported_epoch":"1274","state":"active+clean","last_fresh":"2021-02-10T04:56:30.669093+0000","last_change":"2021-02-10T04:53:07.025068+0000","last_active":"2021-02-10T04:56:30.669093+0000","last_peered":"2021-02-10T04:56:30.669093+0000","last_clean":"2021-02-10T04:56:30.669093+0000","last_became_active":"2021-02-10T04:50:59.589417+0000","last_became_peered":"2021-02-10T04:50:59.589417+0000","last_unstale":"2021-02-10T04:56:30.669093+0000","last_undegraded":"2021-02-10T04:56:30.669093+0000","last_fullsized":"2021-02-10T04:56:30.669093+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.024980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.024980+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"214","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.577025+0000","last_change":"2021-02-10T04:56:32.324158+0000","last_active":"2021-02-10T04:56:33.577025+0000","last_peered":"2021-02-10T04:56:33.577025+0000","last_clean":"2021-02-10T04:56:33.577025+0000","last_became_active":"2021-02-10T04:55:14.605267+0000","last_became_peered":"2021-02-10T04:55:14.605267+0000","last_unstale":"2021-02-10T04:56:33.577025+0000","last_undegraded":"2021-02-10T04:56:33.577025+0000","last_fullsized":"2021-02-10T04:56:33.577025+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.609500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.609500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"1261'86","reported_seq":"1456","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.036188+0000","last_change":"2021-02-10T04:55:35.577794+0000","last_active":"2021-02-10T04:56:34.036188+0000","last_peered":"2021-02-10T04:56:34.036188+0000","last_clean":"2021-02-10T04:56:34.036188+0000","last_became_active":"2021-02-10T04:55:35.577316+0000","last_became_peered":"2021-02-10T04:55:35.577316+0000","last_unstale":"2021-02-10T04:56:34.036188+0000","last_undegraded":"2021-02-10T04:56:34.036188+0000","last_fullsized":"2021-02-10T04:56:34.036188+0000","mapping_epoch":1238,"log_start":"1049'68","ondisk_log_start":"1049'68","created":55,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'39","last_scrub_stamp":"2021-02-10T04:53:38.085565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.085565+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1582,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":227,"num_read_kb":187,"num_write":128,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"1389","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.038703+0000","last_change":"2021-02-10T04:55:14.659217+0000","last_active":"2021-02-10T04:56:34.038703+0000","last_peered":"2021-02-10T04:56:34.038703+0000","last_clean":"2021-02-10T04:56:34.038703+0000","last_became_active":"2021-02-10T04:55:14.658879+0000","last_became_peered":"2021-02-10T04:55:14.658879+0000","last_unstale":"2021-02-10T04:56:34.038703+0000","last_undegraded":"2021-02-10T04:56:34.038703+0000","last_fullsized":"2021-02-10T04:56:34.038703+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:47:08.784593+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:47:08.784593+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"1238","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.583632+0000","last_change":"2021-02-10T04:52:30.019225+0000","last_active":"2021-02-10T04:56:33.583632+0000","last_peered":"2021-02-10T04:56:33.583632+0000","last_clean":"2021-02-10T04:56:33.583632+0000","last_became_active":"2021-02-10T04:50:59.561436+0000","last_became_peered":"2021-02-10T04:50:59.561436+0000","last_unstale":"2021-02-10T04:56:33.583632+0000","last_undegraded":"2021-02-10T04:56:33.583632+0000","last_fullsized":"2021-02-10T04:56:33.583632+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.019102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.019102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"1001'1528","reported_seq":"6223","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.581387+0000","last_change":"2021-02-10T04:52:26.036366+0000","last_active":"2021-02-10T04:56:33.581387+0000","last_peered":"2021-02-10T04:56:33.581387+0000","last_clean":"2021-02-10T04:56:33.581387+0000","last_became_active":"2021-02-10T04:52:26.034028+0000","last_became_peered":"2021-02-10T04:52:26.034028+0000","last_unstale":"2021-02-10T04:56:33.581387+0000","last_undegraded":"2021-02-10T04:56:33.581387+0000","last_fullsized":"2021-02-10T04:56:33.581387+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46944136,"num_objects":16,"num_object_clones":15,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":358,"num_read_kb":351934,"num_write":286,"num_write_kb":111355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":58968832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.15","version":"1000'446","reported_seq":"2212","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.038129+0000","last_change":"2021-02-10T04:53:11.051124+0000","last_active":"2021-02-10T04:56:34.038129+0000","last_peered":"2021-02-10T04:56:34.038129+0000","last_clean":"2021-02-10T04:56:34.038129+0000","last_became_active":"2021-02-10T04:51:40.510899+0000","last_became_peered":"2021-02-10T04:51:40.510899+0000","last_unstale":"2021-02-10T04:56:34.038129+0000","last_undegraded":"2021-02-10T04:56:34.038129+0000","last_fullsized":"2021-02-10T04:56:34.038129+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:53:11.051031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.051031+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24896285,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":18983408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"186","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.577253+0000","last_change":"2021-02-10T04:55:14.605585+0000","last_active":"2021-02-10T04:56:33.577253+0000","last_peered":"2021-02-10T04:56:33.577253+0000","last_clean":"2021-02-10T04:56:33.577253+0000","last_became_active":"2021-02-10T04:55:14.604916+0000","last_became_peered":"2021-02-10T04:55:14.604916+0000","last_unstale":"2021-02-10T04:56:33.577253+0000","last_undegraded":"2021-02-10T04:56:33.577253+0000","last_fullsized":"2021-02-10T04:56:33.577253+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:51:08.639784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.639784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"1207","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.584454+0000","last_change":"2021-02-10T04:51:09.475150+0000","last_active":"2021-02-10T04:56:33.584454+0000","last_peered":"2021-02-10T04:56:33.584454+0000","last_clean":"2021-02-10T04:56:33.584454+0000","last_became_active":"2021-02-10T04:50:59.585917+0000","last_became_peered":"2021-02-10T04:50:59.585917+0000","last_unstale":"2021-02-10T04:56:33.584454+0000","last_undegraded":"2021-02-10T04:56:33.584454+0000","last_fullsized":"2021-02-10T04:56:33.584454+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:09.475063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:09.475063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"177","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.578189+0000","last_change":"2021-02-10T04:53:34.927483+0000","last_active":"2021-02-10T04:56:33.578189+0000","last_peered":"2021-02-10T04:56:33.578189+0000","last_clean":"2021-02-10T04:56:33.578189+0000","last_became_active":"2021-02-10T04:51:36.859707+0000","last_became_peered":"2021-02-10T04:51:36.859707+0000","last_unstale":"2021-02-10T04:56:33.578189+0000","last_undegraded":"2021-02-10T04:56:33.578189+0000","last_fullsized":"2021-02-10T04:56:33.578189+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:34.927424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.927424+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"1000'620","reported_seq":"4562","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.582131+0000","last_change":"2021-02-10T04:53:37.384461+0000","last_active":"2021-02-10T04:56:33.582131+0000","last_peered":"2021-02-10T04:56:33.582131+0000","last_clean":"2021-02-10T04:56:33.582131+0000","last_became_active":"2021-02-10T04:51:40.510114+0000","last_became_peered":"2021-02-10T04:51:40.510114+0000","last_unstale":"2021-02-10T04:56:33.582131+0000","last_undegraded":"2021-02-10T04:56:33.582131+0000","last_fullsized":"2021-02-10T04:56:33.582131+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:53:37.384398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:37.384398+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28530390,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"1262","reported_epoch":"1274","state":"active+clean","last_fresh":"2021-02-10T04:56:30.669831+0000","last_change":"2021-02-10T04:52:41.059269+0000","last_active":"2021-02-10T04:56:30.669831+0000","last_peered":"2021-02-10T04:56:30.669831+0000","last_clean":"2021-02-10T04:56:30.669831+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:56:30.669831+0000","last_undegraded":"2021-02-10T04:56:30.669831+0000","last_fullsized":"2021-02-10T04:56:30.669831+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:52:41.059159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.059159+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"1267","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.579651+0000","last_change":"2021-02-10T04:53:12.332890+0000","last_active":"2021-02-10T04:56:33.579651+0000","last_peered":"2021-02-10T04:56:33.579651+0000","last_clean":"2021-02-10T04:56:33.579651+0000","last_became_active":"2021-02-10T04:50:59.540375+0000","last_became_peered":"2021-02-10T04:50:59.540375+0000","last_unstale":"2021-02-10T04:56:33.579651+0000","last_undegraded":"2021-02-10T04:56:33.579651+0000","last_fullsized":"2021-02-10T04:56:33.579651+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.332233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.332233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"226","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.714951+0000","last_change":"2021-02-10T04:56:34.714951+0000","last_active":"2021-02-10T04:56:34.714951+0000","last_peered":"2021-02-10T04:56:34.714951+0000","last_clean":"2021-02-10T04:56:34.714951+0000","last_became_active":"2021-02-10T04:53:46.733191+0000","last_became_peered":"2021-02-10T04:53:46.733191+0000","last_unstale":"2021-02-10T04:56:34.714951+0000","last_undegraded":"2021-02-10T04:56:34.714951+0000","last_fullsized":"2021-02-10T04:56:34.714951+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:45.362537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:45.362537+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"1002'2042","reported_seq":"9248","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.581770+0000","last_change":"2021-02-10T04:53:46.726353+0000","last_active":"2021-02-10T04:56:33.581770+0000","last_peered":"2021-02-10T04:56:33.581770+0000","last_clean":"2021-02-10T04:56:33.581770+0000","last_became_active":"2021-02-10T04:53:46.725828+0000","last_became_peered":"2021-02-10T04:53:46.725828+0000","last_unstale":"2021-02-10T04:56:33.581770+0000","last_undegraded":"2021-02-10T04:56:33.581770+0000","last_fullsized":"2021-02-10T04:56:33.581770+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:52:41.286006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.286006+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46300076,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":545,"num_read_kb":531689,"num_write":399,"num_write_kb":153087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"1261'56","reported_seq":"1446","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.582462+0000","last_change":"2021-02-10T04:52:35.091494+0000","last_active":"2021-02-10T04:56:33.582462+0000","last_peered":"2021-02-10T04:56:33.582462+0000","last_clean":"2021-02-10T04:56:33.582462+0000","last_became_active":"2021-02-10T04:50:59.569915+0000","last_became_peered":"2021-02-10T04:50:59.569915+0000","last_unstale":"2021-02-10T04:56:33.582462+0000","last_undegraded":"2021-02-10T04:56:33.582462+0000","last_fullsized":"2021-02-10T04:56:33.582462+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'46","last_scrub_stamp":"2021-02-10T04:52:35.091374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:35.091374+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":95,"num_read_kb":80,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"84'7","reported_seq":"1291","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.580210+0000","last_change":"2021-02-10T04:51:52.498251+0000","last_active":"2021-02-10T04:56:33.580210+0000","last_peered":"2021-02-10T04:56:33.580210+0000","last_clean":"2021-02-10T04:56:33.580210+0000","last_became_active":"2021-02-10T04:50:59.575924+0000","last_became_peered":"2021-02-10T04:50:59.575924+0000","last_unstale":"2021-02-10T04:56:33.580210+0000","last_undegraded":"2021-02-10T04:56:33.580210+0000","last_fullsized":"2021-02-10T04:56:33.580210+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'7","last_scrub_stamp":"2021-02-10T04:51:52.498158+0000","last_deep_scrub":"84'7","last_deep_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.498158+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"1229'4","reported_seq":"1635","reported_epoch":"1276","state":"active+clean","last_fresh":"2021-02-10T04:56:34.678113+0000","last_change":"2021-02-10T04:55:14.657278+0000","last_active":"2021-02-10T04:56:34.678113+0000","last_peered":"2021-02-10T04:56:34.678113+0000","last_clean":"2021-02-10T04:56:34.678113+0000","last_became_active":"2021-02-10T04:55:14.656867+0000","last_became_peered":"2021-02-10T04:55:14.656867+0000","last_unstale":"2021-02-10T04:56:34.678113+0000","last_undegraded":"2021-02-10T04:56:34.678113+0000","last_fullsized":"2021-02-10T04:56:34.678113+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:47.407557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:47.407557+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"1189","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.035767+0000","last_change":"2021-02-10T04:55:22.735243+0000","last_active":"2021-02-10T04:56:34.035767+0000","last_peered":"2021-02-10T04:56:34.035767+0000","last_clean":"2021-02-10T04:56:34.035767+0000","last_became_active":"2021-02-10T04:55:14.632398+0000","last_became_peered":"2021-02-10T04:55:14.632398+0000","last_unstale":"2021-02-10T04:56:34.035767+0000","last_undegraded":"2021-02-10T04:56:34.035767+0000","last_fullsized":"2021-02-10T04:56:34.035767+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:23.566027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_clean_scrub_stamp":"2021-02-10T04:52:23.566027+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"1000'490","reported_seq":"3791","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.038684+0000","last_change":"2021-02-10T04:55:14.644527+0000","last_active":"2021-02-10T04:56:34.038684+0000","last_peered":"2021-02-10T04:56:34.038684+0000","last_clean":"2021-02-10T04:56:34.038684+0000","last_became_active":"2021-02-10T04:55:14.643764+0000","last_became_peered":"2021-02-10T04:55:14.643764+0000","last_unstale":"2021-02-10T04:56:34.038684+0000","last_undegraded":"2021-02-10T04:56:34.038684+0000","last_fullsized":"2021-02-10T04:56:34.038684+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:50:03.756248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:50:03.756248+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16154790,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"1261'46","reported_seq":"1344","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.579496+0000","last_change":"2021-02-10T04:53:10.276449+0000","last_active":"2021-02-10T04:56:33.579496+0000","last_peered":"2021-02-10T04:56:33.579496+0000","last_clean":"2021-02-10T04:56:33.579496+0000","last_became_active":"2021-02-10T04:50:59.539529+0000","last_became_peered":"2021-02-10T04:50:59.539529+0000","last_unstale":"2021-02-10T04:56:33.579496+0000","last_undegraded":"2021-02-10T04:56:33.579496+0000","last_fullsized":"2021-02-10T04:56:33.579496+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1112'40","last_scrub_stamp":"2021-02-10T04:53:10.276358+0000","last_deep_scrub":"1112'40","last_deep_scrub_stamp":"2021-02-10T04:51:01.450050+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.276358+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":53,"num_write":38,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"1174","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.035608+0000","last_change":"2021-02-10T04:55:28.088480+0000","last_active":"2021-02-10T04:56:34.035608+0000","last_peered":"2021-02-10T04:56:34.035608+0000","last_clean":"2021-02-10T04:56:34.035608+0000","last_became_active":"2021-02-10T04:55:14.631666+0000","last_became_peered":"2021-02-10T04:55:14.631666+0000","last_unstale":"2021-02-10T04:56:34.035608+0000","last_undegraded":"2021-02-10T04:56:34.035608+0000","last_fullsized":"2021-02-10T04:56:34.035608+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:23.904210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:47:23.904210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"1119'6","reported_seq":"1548","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.582737+0000","last_change":"2021-02-10T04:52:40.038646+0000","last_active":"2021-02-10T04:56:33.582737+0000","last_peered":"2021-02-10T04:56:33.582737+0000","last_clean":"2021-02-10T04:56:33.582737+0000","last_became_active":"2021-02-10T04:51:36.869733+0000","last_became_peered":"2021-02-10T04:51:36.869733+0000","last_unstale":"2021-02-10T04:56:33.582737+0000","last_undegraded":"2021-02-10T04:56:33.582737+0000","last_fullsized":"2021-02-10T04:56:33.582737+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'6","last_scrub_stamp":"2021-02-10T04:52:40.038554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:40.038554+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"1261","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.038896+0000","last_change":"2021-02-10T04:51:36.849933+0000","last_active":"2021-02-10T04:56:34.038896+0000","last_peered":"2021-02-10T04:56:34.038896+0000","last_clean":"2021-02-10T04:56:34.038896+0000","last_became_active":"2021-02-10T04:51:36.849184+0000","last_became_peered":"2021-02-10T04:51:36.849184+0000","last_unstale":"2021-02-10T04:56:34.038896+0000","last_undegraded":"2021-02-10T04:56:34.038896+0000","last_fullsized":"2021-02-10T04:56:34.038896+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:11.779779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:11.779779+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"1032'89","reported_seq":"869","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.578969+0000","last_change":"2021-02-10T04:53:46.728868+0000","last_active":"2021-02-10T04:56:33.578969+0000","last_peered":"2021-02-10T04:56:33.578969+0000","last_clean":"2021-02-10T04:56:33.578969+0000","last_became_active":"2021-02-10T04:53:46.728356+0000","last_became_peered":"2021-02-10T04:53:46.728356+0000","last_unstale":"2021-02-10T04:56:33.578969+0000","last_undegraded":"2021-02-10T04:56:33.578969+0000","last_fullsized":"2021-02-10T04:56:33.578969+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":5,"last_scrub":"1032'89","last_scrub_stamp":"2021-02-10T04:52:34.089789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:34.089789+0000","log_size":89,"ondisk_log_size":89,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5218574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":30326,"num_write":21,"num_write_kb":12444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3210022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"1261'43","reported_seq":"965","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.034435+0000","last_change":"2021-02-10T04:53:14.202505+0000","last_active":"2021-02-10T04:56:34.034435+0000","last_peered":"2021-02-10T04:56:34.034435+0000","last_clean":"2021-02-10T04:56:34.034435+0000","last_became_active":"2021-02-10T04:51:49.496294+0000","last_became_peered":"2021-02-10T04:51:49.496294+0000","last_unstale":"2021-02-10T04:56:34.034435+0000","last_undegraded":"2021-02-10T04:56:34.034435+0000","last_fullsized":"2021-02-10T04:56:34.034435+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1163'37","last_scrub_stamp":"2021-02-10T04:53:14.202445+0000","last_deep_scrub":"1163'37","last_deep_scrub_stamp":"2021-02-10T04:53:14.202445+0000","last_clean_scrub_stamp":"2021-02-10T04:53:14.202445+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":60,"num_read_kb":42,"num_write":34,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"1275","reported_epoch":"1274","state":"active+clean","last_fresh":"2021-02-10T04:56:30.668145+0000","last_change":"2021-02-10T04:52:23.095651+0000","last_active":"2021-02-10T04:56:30.668145+0000","last_peered":"2021-02-10T04:56:30.668145+0000","last_clean":"2021-02-10T04:56:30.668145+0000","last_became_active":"2021-02-10T04:50:59.543531+0000","last_became_peered":"2021-02-10T04:50:59.543531+0000","last_unstale":"2021-02-10T04:56:30.668145+0000","last_undegraded":"2021-02-10T04:56:30.668145+0000","last_fullsized":"2021-02-10T04:56:30.668145+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:23.095588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:23.095588+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"1226","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.583347+0000","last_change":"2021-02-10T04:55:14.636711+0000","last_active":"2021-02-10T04:56:33.583347+0000","last_peered":"2021-02-10T04:56:33.583347+0000","last_clean":"2021-02-10T04:56:33.583347+0000","last_became_active":"2021-02-10T04:55:14.636363+0000","last_became_peered":"2021-02-10T04:55:14.636363+0000","last_unstale":"2021-02-10T04:56:33.583347+0000","last_undegraded":"2021-02-10T04:56:33.583347+0000","last_fullsized":"2021-02-10T04:56:33.583347+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:43.663762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:51:43.663762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"1145","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.034316+0000","last_change":"2021-02-10T04:55:28.101709+0000","last_active":"2021-02-10T04:56:34.034316+0000","last_peered":"2021-02-10T04:56:34.034316+0000","last_clean":"2021-02-10T04:56:34.034316+0000","last_became_active":"2021-02-10T04:55:14.635912+0000","last_became_peered":"2021-02-10T04:55:14.635912+0000","last_unstale":"2021-02-10T04:56:34.034316+0000","last_undegraded":"2021-02-10T04:56:34.034316+0000","last_fullsized":"2021-02-10T04:56:34.034316+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:57.568541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:57.568541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"1000'1071","reported_seq":"4781","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.580139+0000","last_change":"2021-02-10T04:53:35.302879+0000","last_active":"2021-02-10T04:56:33.580139+0000","last_peered":"2021-02-10T04:56:33.580139+0000","last_clean":"2021-02-10T04:56:33.580139+0000","last_became_active":"2021-02-10T04:51:39.508551+0000","last_became_peered":"2021-02-10T04:51:39.508551+0000","last_unstale":"2021-02-10T04:56:33.580139+0000","last_undegraded":"2021-02-10T04:56:33.580139+0000","last_fullsized":"2021-02-10T04:56:33.580139+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:35.302780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.302780+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"1261'39","reported_seq":"1310","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.037155+0000","last_change":"2021-02-10T04:52:39.695340+0000","last_active":"2021-02-10T04:56:34.037155+0000","last_peered":"2021-02-10T04:56:34.037155+0000","last_clean":"2021-02-10T04:56:34.037155+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:56:34.037155+0000","last_undegraded":"2021-02-10T04:56:34.037155+0000","last_fullsized":"2021-02-10T04:56:34.037155+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'31","last_scrub_stamp":"2021-02-10T04:52:39.695237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:39.695237+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":53,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"520","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.034908+0000","last_change":"2021-02-10T04:55:20.929399+0000","last_active":"2021-02-10T04:56:34.034908+0000","last_peered":"2021-02-10T04:56:34.034908+0000","last_clean":"2021-02-10T04:56:34.034908+0000","last_became_active":"2021-02-10T04:55:14.655543+0000","last_became_peered":"2021-02-10T04:55:14.655543+0000","last_unstale":"2021-02-10T04:56:34.034908+0000","last_undegraded":"2021-02-10T04:56:34.034908+0000","last_fullsized":"2021-02-10T04:56:34.034908+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:55:20.929330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:55:20.929330+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"928","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.034528+0000","last_change":"2021-02-10T04:53:35.216489+0000","last_active":"2021-02-10T04:56:34.034528+0000","last_peered":"2021-02-10T04:56:34.034528+0000","last_clean":"2021-02-10T04:56:34.034528+0000","last_became_active":"2021-02-10T04:51:49.501254+0000","last_became_peered":"2021-02-10T04:51:49.501254+0000","last_unstale":"2021-02-10T04:56:34.034528+0000","last_undegraded":"2021-02-10T04:56:34.034528+0000","last_fullsized":"2021-02-10T04:56:34.034528+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.216394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.216394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"1253","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.581070+0000","last_change":"2021-02-10T04:55:14.633276+0000","last_active":"2021-02-10T04:56:33.581070+0000","last_peered":"2021-02-10T04:56:33.581070+0000","last_clean":"2021-02-10T04:56:33.581070+0000","last_became_active":"2021-02-10T04:55:14.632805+0000","last_became_peered":"2021-02-10T04:55:14.632805+0000","last_unstale":"2021-02-10T04:56:33.581070+0000","last_undegraded":"2021-02-10T04:56:33.581070+0000","last_fullsized":"2021-02-10T04:56:33.581070+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:55.814486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.814486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"1002'1657","reported_seq":"5298","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.035458+0000","last_change":"2021-02-10T04:55:22.728878+0000","last_active":"2021-02-10T04:56:34.035458+0000","last_peered":"2021-02-10T04:56:34.035458+0000","last_clean":"2021-02-10T04:56:34.035458+0000","last_became_active":"2021-02-10T04:55:14.653270+0000","last_became_peered":"2021-02-10T04:55:14.653270+0000","last_unstale":"2021-02-10T04:56:34.035458+0000","last_undegraded":"2021-02-10T04:56:34.035458+0000","last_fullsized":"2021-02-10T04:56:34.035458+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:55:22.331010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:55:22.331010+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":47335497,"num_objects":20,"num_object_clones":18,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"1261'24","reported_seq":"1383","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.580662+0000","last_change":"2021-02-10T04:53:04.058324+0000","last_active":"2021-02-10T04:56:33.580662+0000","last_peered":"2021-02-10T04:56:33.580662+0000","last_clean":"2021-02-10T04:56:33.580662+0000","last_became_active":"2021-02-10T04:50:59.544484+0000","last_became_peered":"2021-02-10T04:50:59.544484+0000","last_unstale":"2021-02-10T04:56:33.580662+0000","last_undegraded":"2021-02-10T04:56:33.580662+0000","last_fullsized":"2021-02-10T04:56:33.580662+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'20","last_scrub_stamp":"2021-02-10T04:53:04.058248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:04.058248+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":29,"num_write":24,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"1279","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.579838+0000","last_change":"2021-02-10T04:53:00.343307+0000","last_active":"2021-02-10T04:56:33.579838+0000","last_peered":"2021-02-10T04:56:33.579838+0000","last_clean":"2021-02-10T04:56:33.579838+0000","last_became_active":"2021-02-10T04:51:36.844573+0000","last_became_peered":"2021-02-10T04:51:36.844573+0000","last_unstale":"2021-02-10T04:56:33.579838+0000","last_undegraded":"2021-02-10T04:56:33.579838+0000","last_fullsized":"2021-02-10T04:56:33.579838+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:00.343228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:00.343228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"1246","reported_epoch":"1274","state":"active+clean","last_fresh":"2021-02-10T04:56:30.669797+0000","last_change":"2021-02-10T04:50:56.099710+0000","last_active":"2021-02-10T04:56:30.669797+0000","last_peered":"2021-02-10T04:56:30.669797+0000","last_clean":"2021-02-10T04:56:30.669797+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:56:30.669797+0000","last_undegraded":"2021-02-10T04:56:30.669797+0000","last_fullsized":"2021-02-10T04:56:30.669797+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:56.099651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:50:56.099651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"594","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.580213+0000","last_change":"2021-02-10T04:53:46.718715+0000","last_active":"2021-02-10T04:56:33.580213+0000","last_peered":"2021-02-10T04:56:33.580213+0000","last_clean":"2021-02-10T04:56:33.580213+0000","last_became_active":"2021-02-10T04:53:46.718266+0000","last_became_peered":"2021-02-10T04:53:46.718266+0000","last_unstale":"2021-02-10T04:56:33.580213+0000","last_undegraded":"2021-02-10T04:56:33.580213+0000","last_fullsized":"2021-02-10T04:56:33.580213+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.876681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.876681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"998'1009","reported_seq":"5122","reported_epoch":"1274","state":"active+clean","last_fresh":"2021-02-10T04:56:30.668277+0000","last_change":"2021-02-10T04:55:14.635764+0000","last_active":"2021-02-10T04:56:30.668277+0000","last_peered":"2021-02-10T04:56:30.668277+0000","last_clean":"2021-02-10T04:56:30.668277+0000","last_became_active":"2021-02-10T04:55:14.634422+0000","last_became_peered":"2021-02-10T04:55:14.634422+0000","last_unstale":"2021-02-10T04:56:30.668277+0000","last_undegraded":"2021-02-10T04:56:30.668277+0000","last_fullsized":"2021-02-10T04:56:30.668277+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":5,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:13.032506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:13.032506+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49420827,"num_objects":21,"num_object_clones":19,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":36763798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"1261'45","reported_seq":"1317","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.581340+0000","last_change":"2021-02-10T04:51:44.694779+0000","last_active":"2021-02-10T04:56:33.581340+0000","last_peered":"2021-02-10T04:56:33.581340+0000","last_clean":"2021-02-10T04:56:33.581340+0000","last_became_active":"2021-02-10T04:50:59.549412+0000","last_became_peered":"2021-02-10T04:50:59.549412+0000","last_unstale":"2021-02-10T04:56:33.581340+0000","last_undegraded":"2021-02-10T04:56:33.581340+0000","last_fullsized":"2021-02-10T04:56:33.581340+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:44.694711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.694711+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":76,"num_read_kb":62,"num_write":45,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"1413","reported_epoch":"1274","state":"active+clean","last_fresh":"2021-02-10T04:56:30.669857+0000","last_change":"2021-02-10T04:52:33.088525+0000","last_active":"2021-02-10T04:56:30.669857+0000","last_peered":"2021-02-10T04:56:30.669857+0000","last_clean":"2021-02-10T04:56:30.669857+0000","last_became_active":"2021-02-10T04:51:36.870573+0000","last_became_peered":"2021-02-10T04:51:36.870573+0000","last_unstale":"2021-02-10T04:56:30.669857+0000","last_undegraded":"2021-02-10T04:56:30.669857+0000","last_fullsized":"2021-02-10T04:56:30.669857+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:52:33.088428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:33.088428+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"561","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.580744+0000","last_change":"2021-02-10T04:51:51.836723+0000","last_active":"2021-02-10T04:56:33.580744+0000","last_peered":"2021-02-10T04:56:33.580744+0000","last_clean":"2021-02-10T04:56:33.580744+0000","last_became_active":"2021-02-10T04:50:59.564977+0000","last_became_peered":"2021-02-10T04:50:59.564977+0000","last_unstale":"2021-02-10T04:56:33.580744+0000","last_undegraded":"2021-02-10T04:56:33.580744+0000","last_fullsized":"2021-02-10T04:56:33.580744+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.836638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.836638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"1217","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.583273+0000","last_change":"2021-02-10T04:55:14.633046+0000","last_active":"2021-02-10T04:56:33.583273+0000","last_peered":"2021-02-10T04:56:33.583273+0000","last_clean":"2021-02-10T04:56:33.583273+0000","last_became_active":"2021-02-10T04:55:14.632658+0000","last_became_peered":"2021-02-10T04:55:14.632658+0000","last_unstale":"2021-02-10T04:56:33.583273+0000","last_undegraded":"2021-02-10T04:56:33.583273+0000","last_fullsized":"2021-02-10T04:56:33.583273+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:58.514318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_clean_scrub_stamp":"2021-02-10T04:46:58.514318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1d","version":"1001'1165","reported_seq":"4276","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.034997+0000","last_change":"2021-02-10T04:55:14.656236+0000","last_active":"2021-02-10T04:56:34.034997+0000","last_peered":"2021-02-10T04:56:34.034997+0000","last_clean":"2021-02-10T04:56:34.034997+0000","last_became_active":"2021-02-10T04:55:14.655859+0000","last_became_peered":"2021-02-10T04:55:14.655859+0000","last_unstale":"2021-02-10T04:56:34.034997+0000","last_undegraded":"2021-02-10T04:56:34.034997+0000","last_fullsized":"2021-02-10T04:56:34.034997+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":5,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29601845,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":287,"num_read_kb":299530,"num_write":200,"num_write_kb":89131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"1320","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.577659+0000","last_change":"2021-02-10T04:53:01.331566+0000","last_active":"2021-02-10T04:56:33.577659+0000","last_peered":"2021-02-10T04:56:33.577659+0000","last_clean":"2021-02-10T04:56:33.577659+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:56:33.577659+0000","last_undegraded":"2021-02-10T04:56:33.577659+0000","last_fullsized":"2021-02-10T04:56:33.577659+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:01.331466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.331466+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"1098'3","reported_seq":"1623","reported_epoch":"1274","state":"active+clean","last_fresh":"2021-02-10T04:56:33.048377+0000","last_change":"2021-02-10T04:53:34.865668+0000","last_active":"2021-02-10T04:56:33.048377+0000","last_peered":"2021-02-10T04:56:33.048377+0000","last_clean":"2021-02-10T04:56:33.048377+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:56:33.048377+0000","last_undegraded":"2021-02-10T04:56:33.048377+0000","last_fullsized":"2021-02-10T04:56:33.048377+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"1098'3","last_scrub_stamp":"2021-02-10T04:53:34.865579+0000","last_deep_scrub":"1098'3","last_deep_scrub_stamp":"2021-02-10T04:50:15.255137+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.865579+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"1124","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.581100+0000","last_change":"2021-02-10T04:53:46.727603+0000","last_active":"2021-02-10T04:56:33.581100+0000","last_peered":"2021-02-10T04:56:33.581100+0000","last_clean":"2021-02-10T04:56:33.581100+0000","last_became_active":"2021-02-10T04:53:46.727053+0000","last_became_peered":"2021-02-10T04:53:46.727053+0000","last_unstale":"2021-02-10T04:56:33.581100+0000","last_undegraded":"2021-02-10T04:56:33.581100+0000","last_fullsized":"2021-02-10T04:56:33.581100+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:47.493057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.493057+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1c","version":"1002'1090","reported_seq":"5300","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.578878+0000","last_change":"2021-02-10T04:53:41.353991+0000","last_active":"2021-02-10T04:56:33.578878+0000","last_peered":"2021-02-10T04:56:33.578878+0000","last_clean":"2021-02-10T04:56:33.578878+0000","last_became_active":"2021-02-10T04:52:31.088265+0000","last_became_peered":"2021-02-10T04:52:31.088265+0000","last_unstale":"2021-02-10T04:56:33.578878+0000","last_undegraded":"2021-02-10T04:56:33.578878+0000","last_fullsized":"2021-02-10T04:56:33.578878+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:53:41.353840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:41.353840+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"1221","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.581206+0000","last_change":"2021-02-10T04:55:14.632817+0000","last_active":"2021-02-10T04:56:33.581206+0000","last_peered":"2021-02-10T04:56:33.581206+0000","last_clean":"2021-02-10T04:56:33.581206+0000","last_became_active":"2021-02-10T04:55:14.632214+0000","last_became_peered":"2021-02-10T04:55:14.632214+0000","last_unstale":"2021-02-10T04:56:33.581206+0000","last_undegraded":"2021-02-10T04:56:33.581206+0000","last_fullsized":"2021-02-10T04:56:33.581206+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:42.537443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:42.537443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"1170","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.034805+0000","last_change":"2021-02-10T04:55:22.725805+0000","last_active":"2021-02-10T04:56:34.034805+0000","last_peered":"2021-02-10T04:56:34.034805+0000","last_clean":"2021-02-10T04:56:34.034805+0000","last_became_active":"2021-02-10T04:55:14.630190+0000","last_became_peered":"2021-02-10T04:55:14.630190+0000","last_unstale":"2021-02-10T04:56:34.034805+0000","last_undegraded":"2021-02-10T04:56:34.034805+0000","last_fullsized":"2021-02-10T04:56:34.034805+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.938731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.938731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"485","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.034758+0000","last_change":"2021-02-10T04:53:15.185748+0000","last_active":"2021-02-10T04:56:34.034758+0000","last_peered":"2021-02-10T04:56:34.034758+0000","last_clean":"2021-02-10T04:56:34.034758+0000","last_became_active":"2021-02-10T04:51:49.499348+0000","last_became_peered":"2021-02-10T04:51:49.499348+0000","last_unstale":"2021-02-10T04:56:34.034758+0000","last_undegraded":"2021-02-10T04:56:34.034758+0000","last_fullsized":"2021-02-10T04:56:34.034758+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:15.185686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:15.185686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1f","version":"1004'1093","reported_seq":"6536","reported_epoch":"1274","state":"active+clean","last_fresh":"2021-02-10T04:56:30.668908+0000","last_change":"2021-02-10T04:52:31.089952+0000","last_active":"2021-02-10T04:56:30.668908+0000","last_peered":"2021-02-10T04:56:30.668908+0000","last_clean":"2021-02-10T04:56:30.668908+0000","last_became_active":"2021-02-10T04:52:31.088634+0000","last_became_peered":"2021-02-10T04:52:31.088634+0000","last_unstale":"2021-02-10T04:56:30.668908+0000","last_undegraded":"2021-02-10T04:56:30.668908+0000","last_fullsized":"2021-02-10T04:56:30.668908+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39434697,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":279,"num_read_kb":311058,"num_write":186,"num_write_kb":66790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":53995913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"1276","reported_epoch":"1274","state":"active+clean","last_fresh":"2021-02-10T04:56:30.668749+0000","last_change":"2021-02-10T04:55:39.651056+0000","last_active":"2021-02-10T04:56:30.668749+0000","last_peered":"2021-02-10T04:56:30.668749+0000","last_clean":"2021-02-10T04:56:30.668749+0000","last_became_active":"2021-02-10T04:55:39.596648+0000","last_became_peered":"2021-02-10T04:55:39.596648+0000","last_unstale":"2021-02-10T04:56:30.668749+0000","last_undegraded":"2021-02-10T04:56:30.668749+0000","last_fullsized":"2021-02-10T04:56:30.668749+0000","mapping_epoch":1248,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:52:36.083784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:36.083784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"525","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.035156+0000","last_change":"2021-02-10T04:56:33.223926+0000","last_active":"2021-02-10T04:56:34.035156+0000","last_peered":"2021-02-10T04:56:34.035156+0000","last_clean":"2021-02-10T04:56:34.035156+0000","last_became_active":"2021-02-10T04:55:14.655649+0000","last_became_peered":"2021-02-10T04:55:14.655649+0000","last_unstale":"2021-02-10T04:56:34.035156+0000","last_undegraded":"2021-02-10T04:56:34.035156+0000","last_fullsized":"2021-02-10T04:56:34.035156+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:08.758129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.758129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"1243","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.038422+0000","last_change":"2021-02-10T04:51:44.799512+0000","last_active":"2021-02-10T04:56:34.038422+0000","last_peered":"2021-02-10T04:56:34.038422+0000","last_clean":"2021-02-10T04:56:34.038422+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:56:34.038422+0000","last_undegraded":"2021-02-10T04:56:34.038422+0000","last_fullsized":"2021-02-10T04:56:34.038422+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:44.799429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.799429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1e","version":"1063'654","reported_seq":"4518","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.035158+0000","last_change":"2021-02-10T04:53:40.135450+0000","last_active":"2021-02-10T04:56:34.035158+0000","last_peered":"2021-02-10T04:56:34.035158+0000","last_clean":"2021-02-10T04:56:34.035158+0000","last_became_active":"2021-02-10T04:52:31.087662+0000","last_became_peered":"2021-02-10T04:52:31.087662+0000","last_unstale":"2021-02-10T04:56:34.035158+0000","last_undegraded":"2021-02-10T04:56:34.035158+0000","last_fullsized":"2021-02-10T04:56:34.035158+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:53:40.135367+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:53:40.135367+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"1227","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.583799+0000","last_change":"2021-02-10T04:51:08.006463+0000","last_active":"2021-02-10T04:56:33.583799+0000","last_peered":"2021-02-10T04:56:33.583799+0000","last_clean":"2021-02-10T04:56:33.583799+0000","last_became_active":"2021-02-10T04:50:59.563657+0000","last_became_peered":"2021-02-10T04:50:59.563657+0000","last_unstale":"2021-02-10T04:56:33.583799+0000","last_undegraded":"2021-02-10T04:56:33.583799+0000","last_fullsized":"2021-02-10T04:56:33.583799+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_clean_scrub_stamp":"2021-02-10T04:51:02.455702+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"1278","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.037670+0000","last_change":"2021-02-10T04:52:33.741801+0000","last_active":"2021-02-10T04:56:34.037670+0000","last_peered":"2021-02-10T04:56:34.037670+0000","last_clean":"2021-02-10T04:56:34.037670+0000","last_became_active":"2021-02-10T04:51:36.860129+0000","last_became_peered":"2021-02-10T04:51:36.860129+0000","last_unstale":"2021-02-10T04:56:34.037670+0000","last_undegraded":"2021-02-10T04:56:34.037670+0000","last_fullsized":"2021-02-10T04:56:34.037670+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:33.741711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:46:58.666264+0000","last_clean_scrub_stamp":"2021-02-10T04:52:33.741711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"1239","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.584298+0000","last_change":"2021-02-10T04:52:25.030419+0000","last_active":"2021-02-10T04:56:33.584298+0000","last_peered":"2021-02-10T04:56:33.584298+0000","last_clean":"2021-02-10T04:56:33.584298+0000","last_became_active":"2021-02-10T04:50:59.539582+0000","last_became_peered":"2021-02-10T04:50:59.539582+0000","last_unstale":"2021-02-10T04:56:33.584298+0000","last_undegraded":"2021-02-10T04:56:33.584298+0000","last_fullsized":"2021-02-10T04:56:33.584298+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:25.030281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.030281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"1003'739","reported_seq":"4023","reported_epoch":"1274","state":"active+clean","last_fresh":"2021-02-10T04:56:30.668025+0000","last_change":"2021-02-10T04:53:01.929261+0000","last_active":"2021-02-10T04:56:30.668025+0000","last_peered":"2021-02-10T04:56:30.668025+0000","last_clean":"2021-02-10T04:56:30.668025+0000","last_became_active":"2021-02-10T04:51:40.511505+0000","last_became_peered":"2021-02-10T04:51:40.511505+0000","last_unstale":"2021-02-10T04:56:30.668025+0000","last_undegraded":"2021-02-10T04:56:30.668025+0000","last_fullsized":"2021-02-10T04:56:30.668025+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:53:01.929169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.929169+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22230846,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":18429397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"1115","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.580870+0000","last_change":"2021-02-10T04:53:46.727590+0000","last_active":"2021-02-10T04:56:33.580870+0000","last_peered":"2021-02-10T04:56:33.580870+0000","last_clean":"2021-02-10T04:56:33.580870+0000","last_became_active":"2021-02-10T04:53:46.726584+0000","last_became_peered":"2021-02-10T04:53:46.726584+0000","last_unstale":"2021-02-10T04:56:33.580870+0000","last_undegraded":"2021-02-10T04:56:33.580870+0000","last_fullsized":"2021-02-10T04:56:33.580870+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:26.408326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.408326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"1276","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.580647+0000","last_change":"2021-02-10T04:53:07.324650+0000","last_active":"2021-02-10T04:56:33.580647+0000","last_peered":"2021-02-10T04:56:33.580647+0000","last_clean":"2021-02-10T04:56:33.580647+0000","last_became_active":"2021-02-10T04:50:59.575340+0000","last_became_peered":"2021-02-10T04:50:59.575340+0000","last_unstale":"2021-02-10T04:56:33.580647+0000","last_undegraded":"2021-02-10T04:56:33.580647+0000","last_fullsized":"2021-02-10T04:56:33.580647+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.324564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.324564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"1169","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.034083+0000","last_change":"2021-02-10T04:55:28.098627+0000","last_active":"2021-02-10T04:56:34.034083+0000","last_peered":"2021-02-10T04:56:34.034083+0000","last_clean":"2021-02-10T04:56:34.034083+0000","last_became_active":"2021-02-10T04:55:14.640497+0000","last_became_peered":"2021-02-10T04:55:14.640497+0000","last_unstale":"2021-02-10T04:56:34.034083+0000","last_undegraded":"2021-02-10T04:56:34.034083+0000","last_fullsized":"2021-02-10T04:56:34.034083+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:00.501127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:00.501127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"259'74","reported_seq":"214","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.034225+0000","last_change":"2021-02-10T04:55:28.095458+0000","last_active":"2021-02-10T04:56:34.034225+0000","last_peered":"2021-02-10T04:56:34.034225+0000","last_clean":"2021-02-10T04:56:34.034225+0000","last_became_active":"2021-02-10T04:55:14.624071+0000","last_became_peered":"2021-02-10T04:55:14.624071+0000","last_unstale":"2021-02-10T04:56:34.034225+0000","last_undegraded":"2021-02-10T04:56:34.034225+0000","last_fullsized":"2021-02-10T04:56:34.034225+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":5,"last_scrub":"259'74","last_scrub_stamp":"2021-02-10T04:52:26.597668+0000","last_deep_scrub":"259'74","last_deep_scrub_stamp":"2021-02-10T04:46:49.745104+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.597668+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1218524,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":28228,"num_write":19,"num_write_kb":7022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"37","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.577777+0000","last_change":"2021-02-10T04:55:39.619500+0000","last_active":"2021-02-10T04:56:33.577777+0000","last_peered":"2021-02-10T04:56:33.577777+0000","last_clean":"2021-02-10T04:56:33.577777+0000","last_became_active":"2021-02-10T04:55:39.619062+0000","last_became_peered":"2021-02-10T04:55:39.619062+0000","last_unstale":"2021-02-10T04:56:33.577777+0000","last_undegraded":"2021-02-10T04:56:33.577777+0000","last_fullsized":"2021-02-10T04:56:33.577777+0000","mapping_epoch":1248,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:59.167911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:50:59.167911+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"185","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.579927+0000","last_change":"2021-02-10T04:53:46.718512+0000","last_active":"2021-02-10T04:56:33.579927+0000","last_peered":"2021-02-10T04:56:33.579927+0000","last_clean":"2021-02-10T04:56:33.579927+0000","last_became_active":"2021-02-10T04:53:46.716811+0000","last_became_peered":"2021-02-10T04:53:46.716811+0000","last_unstale":"2021-02-10T04:56:33.579927+0000","last_undegraded":"2021-02-10T04:56:33.579927+0000","last_fullsized":"2021-02-10T04:56:33.579927+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:45.813676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.813676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"1265","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.578342+0000","last_change":"2021-02-10T04:53:38.363899+0000","last_active":"2021-02-10T04:56:33.578342+0000","last_peered":"2021-02-10T04:56:33.578342+0000","last_clean":"2021-02-10T04:56:33.578342+0000","last_became_active":"2021-02-10T04:50:59.539256+0000","last_became_peered":"2021-02-10T04:50:59.539256+0000","last_unstale":"2021-02-10T04:56:33.578342+0000","last_undegraded":"2021-02-10T04:56:33.578342+0000","last_fullsized":"2021-02-10T04:56:33.578342+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:38.363825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:09.433793+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.363825+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1b","version":"1003'1689","reported_seq":"4621","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.035254+0000","last_change":"2021-02-10T04:55:15.591337+0000","last_active":"2021-02-10T04:56:34.035254+0000","last_peered":"2021-02-10T04:56:34.035254+0000","last_clean":"2021-02-10T04:56:34.035254+0000","last_became_active":"2021-02-10T04:55:15.590301+0000","last_became_peered":"2021-02-10T04:55:15.590301+0000","last_unstale":"2021-02-10T04:56:34.035254+0000","last_undegraded":"2021-02-10T04:56:34.035254+0000","last_fullsized":"2021-02-10T04:56:34.035254+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:53:10.075879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.075879+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":63260738,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":387,"num_read_kb":409791,"num_write":281,"num_write_kb":110559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":22449943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"1260","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.584150+0000","last_change":"2021-02-10T04:55:14.637968+0000","last_active":"2021-02-10T04:56:33.584150+0000","last_peered":"2021-02-10T04:56:33.584150+0000","last_clean":"2021-02-10T04:56:33.584150+0000","last_became_active":"2021-02-10T04:55:14.636561+0000","last_became_peered":"2021-02-10T04:55:14.636561+0000","last_unstale":"2021-02-10T04:56:33.584150+0000","last_undegraded":"2021-02-10T04:56:33.584150+0000","last_fullsized":"2021-02-10T04:56:33.584150+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:33.543559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:46:33.543559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"935","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.578158+0000","last_change":"2021-02-10T04:53:04.354508+0000","last_active":"2021-02-10T04:56:33.578158+0000","last_peered":"2021-02-10T04:56:33.578158+0000","last_clean":"2021-02-10T04:56:33.578158+0000","last_became_active":"2021-02-10T04:33:05.169159+0000","last_became_peered":"2021-02-10T04:33:05.169159+0000","last_unstale":"2021-02-10T04:56:33.578158+0000","last_undegraded":"2021-02-10T04:56:33.578158+0000","last_fullsized":"2021-02-10T04:56:33.578158+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:04.354409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:04.354409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"1329","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.036337+0000","last_change":"2021-02-10T04:52:36.777554+0000","last_active":"2021-02-10T04:56:34.036337+0000","last_peered":"2021-02-10T04:56:34.036337+0000","last_clean":"2021-02-10T04:56:34.036337+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:56:34.036337+0000","last_undegraded":"2021-02-10T04:56:34.036337+0000","last_fullsized":"2021-02-10T04:56:34.036337+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:36.777485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:52:36.777485+0000","last_clean_scrub_stamp":"2021-02-10T04:52:36.777485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1a","version":"1001'1528","reported_seq":"5886","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.036026+0000","last_change":"2021-02-10T04:52:31.086406+0000","last_active":"2021-02-10T04:56:34.036026+0000","last_peered":"2021-02-10T04:56:34.036026+0000","last_clean":"2021-02-10T04:56:34.036026+0000","last_became_active":"2021-02-10T04:52:31.085959+0000","last_became_peered":"2021-02-10T04:52:31.085959+0000","last_unstale":"2021-02-10T04:56:34.036026+0000","last_undegraded":"2021-02-10T04:56:34.036026+0000","last_fullsized":"2021-02-10T04:56:34.036026+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46944136,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":358,"num_read_kb":351935,"num_write":286,"num_write_kb":111355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":21311893,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"1159","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.035707+0000","last_change":"2021-02-10T04:55:22.726939+0000","last_active":"2021-02-10T04:56:34.035707+0000","last_peered":"2021-02-10T04:56:34.035707+0000","last_clean":"2021-02-10T04:56:34.035707+0000","last_became_active":"2021-02-10T04:55:14.627044+0000","last_became_peered":"2021-02-10T04:55:14.627044+0000","last_unstale":"2021-02-10T04:56:34.035707+0000","last_undegraded":"2021-02-10T04:56:34.035707+0000","last_fullsized":"2021-02-10T04:56:34.035707+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:17.919745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:17.919745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"561","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.580056+0000","last_change":"2021-02-10T04:53:07.092032+0000","last_active":"2021-02-10T04:56:33.580056+0000","last_peered":"2021-02-10T04:56:33.580056+0000","last_clean":"2021-02-10T04:56:33.580056+0000","last_became_active":"2021-02-10T04:50:59.572077+0000","last_became_peered":"2021-02-10T04:50:59.572077+0000","last_unstale":"2021-02-10T04:56:33.580056+0000","last_undegraded":"2021-02-10T04:56:33.580056+0000","last_fullsized":"2021-02-10T04:56:33.580056+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.091949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:19.755718+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.091949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"1238","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.583497+0000","last_change":"2021-02-10T04:51:59.838856+0000","last_active":"2021-02-10T04:56:33.583497+0000","last_peered":"2021-02-10T04:56:33.583497+0000","last_clean":"2021-02-10T04:56:33.583497+0000","last_became_active":"2021-02-10T04:50:59.563156+0000","last_became_peered":"2021-02-10T04:50:59.563156+0000","last_unstale":"2021-02-10T04:56:33.583497+0000","last_undegraded":"2021-02-10T04:56:33.583497+0000","last_fullsized":"2021-02-10T04:56:33.583497+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.838790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.838790+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":"971","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.578800+0000","last_change":"2021-02-10T04:52:30.405170+0000","last_active":"2021-02-10T04:56:33.578800+0000","last_peered":"2021-02-10T04:56:33.578800+0000","last_clean":"2021-02-10T04:56:33.578800+0000","last_became_active":"2021-02-10T04:51:36.852058+0000","last_became_peered":"2021-02-10T04:51:36.852058+0000","last_unstale":"2021-02-10T04:56:33.578800+0000","last_undegraded":"2021-02-10T04:56:33.578800+0000","last_fullsized":"2021-02-10T04:56:33.578800+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.405106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:20.635536+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.405106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.20","version":"0'0","reported_seq":"209","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.578349+0000","last_change":"2021-02-10T04:53:46.739888+0000","last_active":"2021-02-10T04:56:33.578349+0000","last_peered":"2021-02-10T04:56:33.578349+0000","last_clean":"2021-02-10T04:56:33.578349+0000","last_became_active":"2021-02-10T04:53:46.739532+0000","last_became_peered":"2021-02-10T04:53:46.739532+0000","last_unstale":"2021-02-10T04:56:33.578349+0000","last_undegraded":"2021-02-10T04:56:33.578349+0000","last_fullsized":"2021-02-10T04:56:33.578349+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:56.449085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:41.375276+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.449085+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":"1202","reported_epoch":"1274","state":"active+clean","last_fresh":"2021-02-10T04:56:30.668309+0000","last_change":"2021-02-10T04:53:35.858634+0000","last_active":"2021-02-10T04:56:30.668309+0000","last_peered":"2021-02-10T04:56:30.668309+0000","last_clean":"2021-02-10T04:56:30.668309+0000","last_became_active":"2021-02-10T04:51:36.869785+0000","last_became_peered":"2021-02-10T04:51:36.869785+0000","last_unstale":"2021-02-10T04:56:30.668309+0000","last_undegraded":"2021-02-10T04:56:30.668309+0000","last_fullsized":"2021-02-10T04:56:30.668309+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.858557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.858557+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":"187","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.577444+0000","last_change":"2021-02-10T04:55:23.586523+0000","last_active":"2021-02-10T04:56:33.577444+0000","last_peered":"2021-02-10T04:56:33.577444+0000","last_clean":"2021-02-10T04:56:33.577444+0000","last_became_active":"2021-02-10T04:55:14.637029+0000","last_became_peered":"2021-02-10T04:55:14.637029+0000","last_unstale":"2021-02-10T04:56:33.577444+0000","last_undegraded":"2021-02-10T04:56:33.577444+0000","last_fullsized":"2021-02-10T04:56:33.577444+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:55:23.586401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:55:23.586401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":"172","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.034600+0000","last_change":"2021-02-10T04:55:14.635842+0000","last_active":"2021-02-10T04:56:34.034600+0000","last_peered":"2021-02-10T04:56:34.034600+0000","last_clean":"2021-02-10T04:56:34.034600+0000","last_became_active":"2021-02-10T04:55:14.635499+0000","last_became_peered":"2021-02-10T04:55:14.635499+0000","last_unstale":"2021-02-10T04:56:34.034600+0000","last_undegraded":"2021-02-10T04:56:34.034600+0000","last_fullsized":"2021-02-10T04:56:34.034600+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.706372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.706372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":"1284","reported_epoch":"1274","state":"active+clean","last_fresh":"2021-02-10T04:56:30.668409+0000","last_change":"2021-02-10T04:51:57.287452+0000","last_active":"2021-02-10T04:56:30.668409+0000","last_peered":"2021-02-10T04:56:30.668409+0000","last_clean":"2021-02-10T04:56:30.668409+0000","last_became_active":"2021-02-10T04:50:59.566827+0000","last_became_peered":"2021-02-10T04:50:59.566827+0000","last_unstale":"2021-02-10T04:56:30.668409+0000","last_undegraded":"2021-02-10T04:56:30.668409+0000","last_fullsized":"2021-02-10T04:56:30.668409+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.287363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.287363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":"258","reported_epoch":"1274","state":"active+clean","last_fresh":"2021-02-10T04:56:30.669011+0000","last_change":"2021-02-10T04:55:14.659882+0000","last_active":"2021-02-10T04:56:30.669011+0000","last_peered":"2021-02-10T04:56:30.669011+0000","last_clean":"2021-02-10T04:56:30.669011+0000","last_became_active":"2021-02-10T04:55:14.659514+0000","last_became_peered":"2021-02-10T04:55:14.659514+0000","last_unstale":"2021-02-10T04:56:30.669011+0000","last_undegraded":"2021-02-10T04:56:30.669011+0000","last_fullsized":"2021-02-10T04:56:30.669011+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:10.308282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:10.308282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":"1174","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.583872+0000","last_change":"2021-02-10T04:53:09.121379+0000","last_active":"2021-02-10T04:56:33.583872+0000","last_peered":"2021-02-10T04:56:33.583872+0000","last_clean":"2021-02-10T04:56:33.583872+0000","last_became_active":"2021-02-10T04:50:59.561442+0000","last_became_peered":"2021-02-10T04:50:59.561442+0000","last_unstale":"2021-02-10T04:56:33.583872+0000","last_undegraded":"2021-02-10T04:56:33.583872+0000","last_fullsized":"2021-02-10T04:56:33.583872+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:09.121298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.121298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.21","version":"293'77","reported_seq":"846","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.581252+0000","last_change":"2021-02-10T04:52:28.062923+0000","last_active":"2021-02-10T04:56:33.581252+0000","last_peered":"2021-02-10T04:56:33.581252+0000","last_clean":"2021-02-10T04:56:33.581252+0000","last_became_active":"2021-02-10T04:52:28.062463+0000","last_became_peered":"2021-02-10T04:52:28.062463+0000","last_unstale":"2021-02-10T04:56:33.581252+0000","last_undegraded":"2021-02-10T04:56:33.581252+0000","last_fullsized":"2021-02-10T04:56:33.581252+0000","mapping_epoch":1143,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1144,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1886586,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":15163,"num_write":9,"num_write_kb":3316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":"172","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.577123+0000","last_change":"2021-02-10T04:52:32.124955+0000","last_active":"2021-02-10T04:56:33.577123+0000","last_peered":"2021-02-10T04:56:33.577123+0000","last_clean":"2021-02-10T04:56:33.577123+0000","last_became_active":"2021-02-10T04:51:36.867061+0000","last_became_peered":"2021-02-10T04:51:36.867061+0000","last_unstale":"2021-02-10T04:56:33.577123+0000","last_undegraded":"2021-02-10T04:56:33.577123+0000","last_fullsized":"2021-02-10T04:56:33.577123+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:32.124901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:08.271977+0000","last_clean_scrub_stamp":"2021-02-10T04:52:32.124901+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":"1213","reported_epoch":"1274","state":"active+clean","last_fresh":"2021-02-10T04:56:30.669030+0000","last_change":"2021-02-10T04:51:07.182775+0000","last_active":"2021-02-10T04:56:30.669030+0000","last_peered":"2021-02-10T04:56:30.669030+0000","last_clean":"2021-02-10T04:56:30.669030+0000","last_became_active":"2021-02-10T04:28:23.126368+0000","last_became_peered":"2021-02-10T04:28:23.126368+0000","last_unstale":"2021-02-10T04:56:30.669030+0000","last_undegraded":"2021-02-10T04:56:30.669030+0000","last_fullsized":"2021-02-10T04:56:30.669030+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.182678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.182678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.20","version":"1000'1071","reported_seq":"4620","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.576849+0000","last_change":"2021-02-10T04:53:03.915465+0000","last_active":"2021-02-10T04:56:33.576849+0000","last_peered":"2021-02-10T04:56:33.576849+0000","last_clean":"2021-02-10T04:56:33.576849+0000","last_became_active":"2021-02-10T04:52:31.088690+0000","last_became_peered":"2021-02-10T04:52:31.088690+0000","last_unstale":"2021-02-10T04:56:33.576849+0000","last_undegraded":"2021-02-10T04:56:33.576849+0000","last_fullsized":"2021-02-10T04:56:33.576849+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:03.915403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.915403+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16398893,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":"798","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.578882+0000","last_change":"2021-02-10T04:53:46.721599+0000","last_active":"2021-02-10T04:56:33.578882+0000","last_peered":"2021-02-10T04:56:33.578882+0000","last_clean":"2021-02-10T04:56:33.578882+0000","last_became_active":"2021-02-10T04:53:46.721340+0000","last_became_peered":"2021-02-10T04:53:46.721340+0000","last_unstale":"2021-02-10T04:56:33.578882+0000","last_undegraded":"2021-02-10T04:56:33.578882+0000","last_fullsized":"2021-02-10T04:56:33.578882+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:28.092288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:07.767250+0000","last_clean_scrub_stamp":"2021-02-10T04:52:28.092288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.25","version":"53'2","reported_seq":"1230","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.037824+0000","last_change":"2021-02-10T04:53:36.103026+0000","last_active":"2021-02-10T04:56:34.037824+0000","last_peered":"2021-02-10T04:56:34.037824+0000","last_clean":"2021-02-10T04:56:34.037824+0000","last_became_active":"2021-02-10T04:28:23.467770+0000","last_became_peered":"2021-02-10T04:28:23.467770+0000","last_unstale":"2021-02-10T04:56:34.037824+0000","last_undegraded":"2021-02-10T04:56:34.037824+0000","last_fullsized":"2021-02-10T04:56:34.037824+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:53:36.102928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:36.102928+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.23","version":"1002'1657","reported_seq":"5145","reported_epoch":"1274","state":"active+clean","last_fresh":"2021-02-10T04:56:30.668421+0000","last_change":"2021-02-10T04:52:31.093874+0000","last_active":"2021-02-10T04:56:30.668421+0000","last_peered":"2021-02-10T04:56:30.668421+0000","last_clean":"2021-02-10T04:56:30.668421+0000","last_became_active":"2021-02-10T04:52:31.092996+0000","last_became_peered":"2021-02-10T04:52:31.092996+0000","last_unstale":"2021-02-10T04:56:30.668421+0000","last_undegraded":"2021-02-10T04:56:30.668421+0000","last_fullsized":"2021-02-10T04:56:30.668421+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21396533,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":212,"num_read_kb":218837,"num_write":159,"num_write_kb":59399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":20479176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":"1320","reported_epoch":"1274","state":"active+clean","last_fresh":"2021-02-10T04:56:30.668186+0000","last_change":"2021-02-10T04:53:12.951612+0000","last_active":"2021-02-10T04:56:30.668186+0000","last_peered":"2021-02-10T04:56:30.668186+0000","last_clean":"2021-02-10T04:56:30.668186+0000","last_became_active":"2021-02-10T04:50:59.566548+0000","last_became_peered":"2021-02-10T04:50:59.566548+0000","last_unstale":"2021-02-10T04:56:30.668186+0000","last_undegraded":"2021-02-10T04:56:30.668186+0000","last_fullsized":"2021-02-10T04:56:30.668186+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.951528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.951528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.26","version":"53'1","reported_seq":"112","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.581447+0000","last_change":"2021-02-10T04:53:46.722636+0000","last_active":"2021-02-10T04:56:33.581447+0000","last_peered":"2021-02-10T04:56:33.581447+0000","last_clean":"2021-02-10T04:56:33.581447+0000","last_became_active":"2021-02-10T04:53:46.722227+0000","last_became_peered":"2021-02-10T04:53:46.722227+0000","last_unstale":"2021-02-10T04:56:33.581447+0000","last_undegraded":"2021-02-10T04:56:33.581447+0000","last_fullsized":"2021-02-10T04:56:33.581447+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:53:03.933910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.933910+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":423,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.22","version":"998'1009","reported_seq":"4637","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:34.036117+0000","last_change":"2021-02-10T04:52:31.086563+0000","last_active":"2021-02-10T04:56:34.036117+0000","last_peered":"2021-02-10T04:56:34.036117+0000","last_clean":"2021-02-10T04:56:34.036117+0000","last_became_active":"2021-02-10T04:52:31.085288+0000","last_became_peered":"2021-02-10T04:52:31.085288+0000","last_unstale":"2021-02-10T04:56:34.036117+0000","last_undegraded":"2021-02-10T04:56:34.036117+0000","last_fullsized":"2021-02-10T04:56:34.036117+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12355207,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":120,"num_read_kb":124702,"num_write":77,"num_write_kb":25052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4145029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":"1202","reported_epoch":"1274","state":"active+clean","last_fresh":"2021-02-10T04:56:30.668341+0000","last_change":"2021-02-10T04:53:10.958668+0000","last_active":"2021-02-10T04:56:30.668341+0000","last_peered":"2021-02-10T04:56:30.668341+0000","last_clean":"2021-02-10T04:56:30.668341+0000","last_became_active":"2021-02-10T04:45:49.552026+0000","last_became_peered":"2021-02-10T04:45:49.552026+0000","last_unstale":"2021-02-10T04:56:30.668341+0000","last_undegraded":"2021-02-10T04:56:30.668341+0000","last_fullsized":"2021-02-10T04:56:30.668341+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:10.958579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.958579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":"217","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.581915+0000","last_change":"2021-02-10T04:53:46.725941+0000","last_active":"2021-02-10T04:56:33.581915+0000","last_peered":"2021-02-10T04:56:33.581915+0000","last_clean":"2021-02-10T04:56:33.581915+0000","last_became_active":"2021-02-10T04:53:46.725587+0000","last_became_peered":"2021-02-10T04:53:46.725587+0000","last_unstale":"2021-02-10T04:56:33.581915+0000","last_undegraded":"2021-02-10T04:56:33.581915+0000","last_fullsized":"2021-02-10T04:56:33.581915+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:41.758684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.758684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":"1212","reported_epoch":"1275","state":"active+clean","last_fresh":"2021-02-10T04:56:33.580405+0000","last_change":"2021-02-10T04:55:39.599193+0000","last_active":"2021-02-10T04:56:33.580405+0000","last_peered":"2021-02-10T04:56:33.580405+0000","last_clean":"2021-02-10T04:56:33.580405+0000","last_became_active":"2021-02-10T04:55:39.598816+0000","last_became_peered":"2021-02-10T04:55:39.598816+0000","last_unstale":"2021-02-10T04:56:33.580405+0000","last_undegraded":"2021-02-10T04:56:33.580405+0000","last_fullsized":"2021-02-10T04:56:33.580405+0000","mapping_epoch":1248,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:11.139602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.139602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":"1228","reported_epoch":"1274","state":"active+clean","last_fresh":"2021-02-10T04:56:30.669206+0000","last_change":"2021-02-10T04:52:24.072005+0000","last_active":"2021-02-10T04:56:30.669206+0000","last_peered":"2021-02-10T04:56:30.669206+0000","last_clean":"2021-02-10T04:56:30.669206+0000","last_became_active":"2021-02-10T04:50:59.527945+0000","last_became_peered":"2021-02-10T04:50:59.527945+0000","last_unstale":"2021-02-10T04:56:30.669206+0000","last_undegraded":"2021-02-10T04:56:30.669206+0000","last_fullsized":"2021-02-10T04:56:30.669206+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:24.071915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_clean_scrub_stamp":"2021-02-10T04:52:24.071915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":972330808,"num_objects":390,"num_object_clones":337,"num_object_copies":1170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":390,"num_whiteouts":10,"num_read":4442,"num_read_kb":4559373,"num_write":3227,"num_write_kb":1247575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":611,"num_bytes_recovered":311961170,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2010624000,"data_stored":2589160348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34722,"ondisk_log_size":34722,"up":108,"acting":108,"num_store_stats":8},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":2368,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":18,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1139,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":5835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":126,"acting":126,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":13,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":717,"num_read_kb":712,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20,"ondisk_log_size":20,"up":39,"acting":39,"num_store_stats":8},{"poolid":4,"num_pg":22,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":2717,"num_read_kb":2190,"num_write":1592,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":17736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":844,"ondisk_log_size":844,"up":66,"acting":66,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35,"ondisk_log_size":35,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":126,"acting":126,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365174,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":191244,"kb_used_data":148996,"kb_used_omap":0,"kb_used_meta":42240,"kb_avail":93553908,"statfs":{"total":95995035648,"available":95799201792,"internally_reserved":0,"allocated":152571904,"data_stored":202259677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43253760},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:56:17 2021","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":4.4340000000000002,"5min":4.4340000000000002,"15min":4.4340000000000002},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.506,"5min":1.506,"15min":1.506},"last":0.71599999999999997}]},{"osd":1,"last update":"Wed Feb 10 04:55:42 2021","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.80700000000000005,"15min":0.95899999999999996},"min":{"1min":0.48799999999999999,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.409,"5min":2.254,"15min":10.140000000000001},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.85999999999999999,"15min":0.96699999999999997},"min":{"1min":0.63400000000000001,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.647,"5min":1.647,"15min":7.1520000000000001},"last":1.024}]},{"osd":2,"last update":"Wed Feb 10 04:55:42 2021","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":1.256,"15min":1.1259999999999999},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.4319999999999999,"5min":16.896999999999998,"15min":16.896999999999998},"last":0.63400000000000001},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.2070000000000001,"15min":1.254},"min":{"1min":0.57999999999999996,"5min":0.496,"15min":0.496},"max":{"1min":4.0620000000000003,"5min":16.792000000000002,"15min":16.792000000000002},"last":0.67300000000000004}]},{"osd":3,"last update":"Wed Feb 10 04:56:01 2021","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.0860000000000001,"15min":1.274},"min":{"1min":0.73199999999999998,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.7010000000000001,"5min":4.5890000000000004,"15min":4.5890000000000004},"last":0.93799999999999994},{"interface":"front","average":{"1min":1.238,"5min":1.1759999999999999,"15min":1.3819999999999999},"min":{"1min":0.89800000000000002,"5min":0.629,"15min":0.629},"max":{"1min":1.8129999999999999,"5min":4.6609999999999996,"15min":4.6609999999999996},"last":0.95999999999999996}]},{"osd":4,"last update":"Wed Feb 10 04:55:42 2021","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":1.1639999999999999,"15min":1.3899999999999999},"min":{"1min":0.60399999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.399,"5min":2.9729999999999999,"15min":10.781000000000001},"last":0.85699999999999998},{"interface":"front","average":{"1min":1.03,"5min":1.1759999999999999,"15min":1.3460000000000001},"min":{"1min":0.74399999999999999,"5min":0.69699999999999995,"15min":0.69099999999999995},"max":{"1min":1.546,"5min":3.4049999999999998,"15min":10.326000000000001},"last":1.046}]},{"osd":5,"last update":"Wed Feb 10 04:55:42 2021","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.349,"15min":1.484},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":4.766,"5min":4.766,"15min":14.972},"last":0.879},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.4450000000000001,"15min":1.6899999999999999},"min":{"1min":0.81000000000000005,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.1549999999999998,"5min":4.4020000000000001,"15min":15.218999999999999},"last":1.113}]},{"osd":6,"last update":"Wed Feb 10 04:55:42 2021","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.375,"15min":1.7490000000000001},"min":{"1min":0.92700000000000005,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.5780000000000001,"5min":4.0380000000000003,"15min":16.050000000000001},"last":1.1990000000000001},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.4039999999999999,"15min":1.71},"min":{"1min":0.90300000000000002,"5min":0.876,"15min":0.876},"max":{"1min":1.964,"5min":3.9159999999999999,"15min":15.960000000000001},"last":1.157}]}]},{"osd":6,"up_from":43,"seq":184683594106,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":408212,"kb_used_data":342092,"kb_used_omap":0,"kb_used_meta":66112,"kb_avail":93336940,"statfs":{"total":95995035648,"available":95577026560,"internally_reserved":0,"allocated":350302208,"data_stored":443224665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":67698688},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:56:16 2021","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":1.4259999999999999},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.6599999999999999,"5min":1.6599999999999999,"15min":1.6599999999999999},"last":0.623}]},{"osd":1,"last update":"Wed Feb 10 04:56:21 2021","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.996,"15min":1.0680000000000001},"min":{"1min":0.57999999999999996,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.7529999999999999,"5min":13.18,"15min":13.18},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.98899999999999999,"15min":1.091},"min":{"1min":0.54200000000000004,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.514,"5min":11.041,"15min":11.041},"last":1.032}]},{"osd":2,"last update":"Wed Feb 10 04:56:21 2021","interfaces":[{"interface":"back","average":{"1min":4.7039999999999997,"5min":3.2999999999999998,"15min":1.9430000000000001},"min":{"1min":0.59499999999999997,"5min":0.503,"15min":0.503},"max":{"1min":69.653000000000006,"5min":163.81700000000001,"15min":163.81700000000001},"last":1.1459999999999999},{"interface":"front","average":{"1min":4.7809999999999997,"5min":3.3809999999999998,"15min":1.9730000000000001},"min":{"1min":0.55600000000000005,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":69.561000000000007,"5min":164.56800000000001,"15min":164.56800000000001},"last":1.0009999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:56:12 2021","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.1160000000000001,"15min":1.1839999999999999},"min":{"1min":0.60599999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.835,"5min":1.835,"15min":1.835},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.165,"5min":1.109,"15min":1.119},"min":{"1min":0.81399999999999995,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.778,"5min":2.363,"15min":2.363},"last":1.6519999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:56:21 2021","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.141,"15min":1.3500000000000001},"min":{"1min":0.622,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.968,"5min":2.661,"15min":7.1459999999999999},"last":1.532},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.98599999999999999,"15min":1.091},"min":{"1min":0.57399999999999995,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.958,"5min":5.2750000000000004,"15min":11.997999999999999},"last":0.76800000000000002}]},{"osd":5,"last update":"Wed Feb 10 04:56:21 2021","interfaces":[{"interface":"back","average":{"1min":2.327,"5min":1.419,"15min":1.6519999999999999},"min":{"1min":0.84299999999999997,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":22.663,"5min":22.663,"15min":81.540000000000006},"last":0.92600000000000005},{"interface":"front","average":{"1min":2.617,"5min":1.706,"15min":1.9399999999999999},"min":{"1min":0.89100000000000001,"5min":0.78700000000000003,"15min":0.78000000000000003},"max":{"1min":22.933,"5min":22.933,"15min":81.492999999999995},"last":1.3260000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:55:47 2021","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.161,"15min":1.288},"min":{"1min":0.72899999999999998,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":2.073,"5min":4.0919999999999996,"15min":5.1820000000000004},"last":1.073},{"interface":"front","average":{"1min":1.421,"5min":1.4159999999999999,"15min":1.595},"min":{"1min":1.016,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":2.2970000000000002,"5min":5.4109999999999996,"15min":13.315},"last":1.26}]}]},{"osd":1,"up_from":13,"seq":55834575244,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":261204,"kb_used_data":206092,"kb_used_omap":0,"kb_used_meta":55104,"kb_avail":93483948,"statfs":{"total":95995035648,"available":95727562752,"internally_reserved":0,"allocated":211038208,"data_stored":265726560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":56426496},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:56:14 2021","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.54500000000000004}]},{"osd":2,"last update":"Wed Feb 10 04:56:14 2021","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":4.3220000000000001,"15min":2.6190000000000002},"min":{"1min":0.53300000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.266,"5min":162.61199999999999,"15min":162.61199999999999},"last":101.06100000000001},{"interface":"front","average":{"1min":1.093,"5min":4.4290000000000003,"15min":2.6949999999999998},"min":{"1min":0.53100000000000003,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":5.8860000000000001,"5min":162.762,"15min":162.762},"last":101.66500000000001}]},{"osd":3,"last update":"Wed Feb 10 04:56:04 2021","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.77100000000000002,"15min":0.90300000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.387,"5min":3.077,"15min":3.077},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.86699999999999999,"15min":0.98999999999999999},"min":{"1min":0.57099999999999995,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.052,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.46800000000000003}]},{"osd":4,"last update":"Wed Feb 10 04:55:56 2021","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.0309999999999999,"15min":1.135},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":6.1479999999999997,"5min":6.1479999999999997,"15min":6.1479999999999997},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.071,"15min":1.2749999999999999},"min":{"1min":0.68400000000000005,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.6879999999999999,"5min":2.5270000000000001,"15min":29.547999999999998},"last":0.83499999999999996}]},{"osd":5,"last update":"Wed Feb 10 04:56:04 2021","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.0860000000000001,"15min":1.161},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.56599999999999995},"max":{"1min":1.581,"5min":2.7959999999999998,"15min":5.0300000000000002},"last":0.63100000000000001},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.224,"15min":1.4650000000000001},"min":{"1min":0.80900000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":6.032,"5min":6.032,"15min":28.678000000000001},"last":0.81599999999999995}]},{"osd":6,"last update":"Wed Feb 10 04:56:14 2021","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.4039999999999999,"15min":1.512},"min":{"1min":0.80800000000000005,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.5960000000000001,"5min":30.997,"15min":30.997},"last":0.91500000000000004},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.4239999999999999,"15min":1.468},"min":{"1min":1.052,"5min":0.64200000000000002,"15min":0.60499999999999998},"max":{"1min":1.7070000000000001,"5min":31.138000000000002,"15min":31.138000000000002},"last":0.93500000000000005}]},{"osd":7,"last update":"Wed Feb 10 04:56:29 2021","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.2150000000000001,"15min":1.4570000000000001},"min":{"1min":0.90600000000000003,"5min":0.628,"15min":0.628},"max":{"1min":1.7689999999999999,"5min":6.2629999999999999,"15min":28.573},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.23,"5min":1.23,"15min":1.4830000000000001},"min":{"1min":0.94499999999999995,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.516,"5min":1.895,"15min":29.408999999999999},"last":0.98199999999999998}]}]},{"osd":0,"up_from":1227,"seq":5269924872210,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":348736,"kb_used_data":338296,"kb_used_omap":306,"kb_used_meta":10125,"kb_avail":93396416,"statfs":{"total":95995035648,"available":95637929984,"internally_reserved":0,"allocated":346415104,"data_stored":471355568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":313579,"internal_metadata":10368789},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 04:56:14 2021","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.65200000000000002}]},{"osd":2,"last update":"Wed Feb 10 04:56:14 2021","interfaces":[{"interface":"back","average":{"1min":3.456,"5min":3.456,"15min":3.456},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":59.341999999999999,"5min":59.341999999999999,"15min":59.341999999999999},"last":0.92900000000000005},{"interface":"front","average":{"1min":3.4660000000000002,"5min":3.4660000000000002,"15min":3.4660000000000002},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":59.405999999999999,"5min":59.405999999999999,"15min":59.405999999999999},"last":0.80000000000000004}]},{"osd":3,"last update":"Wed Feb 10 04:56:14 2021","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.996,"5min":0.996,"15min":0.996},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.669,"5min":1.669,"15min":1.669},"last":0.84599999999999997}]},{"osd":4,"last update":"Wed Feb 10 04:56:14 2021","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":1.0349999999999999},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":1.0669999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:56:14 2021","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.24,"15min":1.24},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.6950000000000001,"5min":1.6950000000000001,"15min":1.6950000000000001},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":1.0069999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:56:14 2021","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.171,"15min":1.171},"min":{"1min":1.018,"5min":1.018,"15min":1.018},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":1.125},{"interface":"front","average":{"1min":1.208,"5min":1.208,"15min":1.208},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.351,"5min":2.351,"15min":2.351},"last":1.1020000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:56:14 2021","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"min":{"1min":1.02,"5min":1.02,"15min":1.02},"max":{"1min":1.5800000000000001,"5min":1.5800000000000001,"15min":1.5800000000000001},"last":1.244},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":1.643,"5min":1.643,"15min":1.643},"last":1.204}]}]},{"osd":2,"up_from":18,"seq":77309411722,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":353996,"kb_used_data":318276,"kb_used_omap":0,"kb_used_meta":35712,"kb_avail":93391156,"statfs":{"total":95995035648,"available":95632543744,"internally_reserved":0,"allocated":325914624,"data_stored":374754716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36569088},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:56:14 2021","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.2789999999999999,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":0.78200000000000003}]},{"osd":1,"last update":"Wed Feb 10 04:55:55 2021","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":0.88200000000000001,"15min":1},"min":{"1min":0.48099999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":17.041,"5min":17.041,"15min":17.041},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.8999999999999999,"5min":0.91300000000000003,"15min":0.94899999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":21.172000000000001,"5min":21.172000000000001,"15min":21.172000000000001},"last":0.85799999999999998}]},{"osd":3,"last update":"Wed Feb 10 04:56:14 2021","interfaces":[{"interface":"back","average":{"1min":1.778,"5min":1.044,"15min":1.125},"min":{"1min":0.64700000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":18.904,"5min":18.904,"15min":18.904},"last":1.218},{"interface":"front","average":{"1min":2.0219999999999998,"5min":1.2709999999999999,"15min":1.3109999999999999},"min":{"1min":0.86499999999999999,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":19.241,"5min":19.241,"15min":19.241},"last":1.4810000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:55:36 2021","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.1379999999999999,"15min":1.2849999999999999},"min":{"1min":0.71499999999999997,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":9.2029999999999994,"5min":9.2029999999999994,"15min":9.2029999999999994},"last":1.6100000000000001},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.228,"15min":1.1699999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":3.403,"5min":24.373999999999999,"15min":24.373999999999999},"last":1.0569999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:55:55 2021","interfaces":[{"interface":"back","average":{"1min":3.0630000000000002,"5min":1.575,"15min":1.7709999999999999},"min":{"1min":0.85399999999999998,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":39.911999999999999,"5min":39.911999999999999,"15min":39.911999999999999},"last":1.748},{"interface":"front","average":{"1min":2.508,"5min":1.284,"15min":1.5109999999999999},"min":{"1min":0.67700000000000005,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":33.539999999999999,"5min":33.539999999999999,"15min":33.539999999999999},"last":1.0029999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:56:26 2021","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.0649999999999999,"15min":1.395},"min":{"1min":0.72899999999999998,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":2.3010000000000002,"5min":3.1659999999999999,"15min":20.007000000000001},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.47,"5min":1.464,"15min":1.6419999999999999},"min":{"1min":1.1240000000000001,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":2.6640000000000001,"5min":16.135000000000002,"15min":19.417999999999999},"last":1.8999999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:55:36 2021","interfaces":[{"interface":"back","average":{"1min":1.637,"5min":1.637,"15min":1.681},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":9.3460000000000001,"5min":24.405000000000001,"15min":24.405000000000001},"last":2.0499999999999998},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.1040000000000001,"15min":1.278},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":3.6640000000000001,"5min":3.6640000000000001,"15min":4.9189999999999996},"last":1.1279999999999999}]}]},{"osd":3,"up_from":1111,"seq":4771708665925,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":332652,"kb_used_data":301028,"kb_used_omap":386,"kb_used_meta":31229,"kb_avail":93412500,"statfs":{"total":95995035648,"available":95654400000,"internally_reserved":0,"allocated":308252672,"data_stored":409676363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":396036,"internal_metadata":31978748},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:56:14 2021","interfaces":[{"interface":"back","average":{"1min":1.98,"5min":1.98,"15min":1.98},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":22.869,"5min":22.869,"15min":22.869},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.6279999999999999,"15min":1.6279999999999999},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":13.587999999999999,"5min":13.587999999999999,"15min":13.587999999999999},"last":0.80700000000000005}]},{"osd":1,"last update":"Wed Feb 10 04:56:06 2021","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.69199999999999995,"15min":0.66300000000000003},"min":{"1min":0.503,"5min":0.44,"15min":0.44},"max":{"1min":3.6629999999999998,"5min":3.6629999999999998,"15min":3.6629999999999998},"last":1.0429999999999999},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.80200000000000005,"15min":0.86299999999999999},"min":{"1min":0.53500000000000003,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":3.7050000000000001,"5min":3.7050000000000001,"15min":3.7050000000000001},"last":1.022}]},{"osd":2,"last update":"Wed Feb 10 04:56:06 2021","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":1.413,"15min":1.2370000000000001},"min":{"1min":0.56699999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.173,"5min":31.100999999999999,"15min":31.100999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.569,"15min":1.27},"min":{"1min":0.71399999999999997,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":3.7509999999999999,"5min":31.052,"15min":31.052},"last":1.1419999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:56:06 2021","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":0.95899999999999996,"15min":0.96699999999999997},"min":{"1min":0.77100000000000002,"5min":0.628,"15min":0.628},"max":{"1min":2.6589999999999998,"5min":2.9430000000000001,"15min":2.9430000000000001},"last":1.284},{"interface":"front","average":{"1min":1.0980000000000001,"5min":0.98999999999999999,"15min":1.0820000000000001},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":3.21,"5min":3.21,"15min":3.21},"last":1.329}]},{"osd":5,"last update":"Wed Feb 10 04:56:06 2021","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.966,"15min":2.4870000000000001},"min":{"1min":0.75800000000000001,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":3.3250000000000002,"5min":52.353000000000002,"15min":52.353000000000002},"last":1.5569999999999999},{"interface":"front","average":{"1min":0.996,"5min":1.869,"15min":2.3180000000000001},"min":{"1min":0.83099999999999996,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.552,"5min":52.301000000000002,"15min":52.301000000000002},"last":1.0720000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:56:06 2021","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":2.5579999999999998,"15min":4.6699999999999999},"min":{"1min":0.871,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.466,"5min":64.376999999999995,"15min":64.376999999999995},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.353,"5min":2.6829999999999998,"15min":4.6840000000000002},"min":{"1min":1.0549999999999999,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":3.3889999999999998,"5min":64.438999999999993,"15min":64.438999999999993},"last":1.385}]},{"osd":7,"last update":"Wed Feb 10 04:56:06 2021","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.0629999999999999,"15min":1.105},"min":{"1min":0.94599999999999995,"5min":0.754,"15min":0.754},"max":{"1min":1.349,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.28,"5min":1.1419999999999999,"15min":1.143},"min":{"1min":0.90300000000000002,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.9569999999999999,"5min":2.9569999999999999,"15min":2.9569999999999999},"last":1.2230000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986562,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":160460,"kb_used_data":120260,"kb_used_omap":0,"kb_used_meta":40192,"kb_avail":93584692,"statfs":{"total":95995035648,"available":95830724608,"internally_reserved":0,"allocated":123146240,"data_stored":155577723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41156608},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:56:16 2021","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":3.637,"5min":3.637,"15min":3.637},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":4.6829999999999998,"5min":4.6829999999999998,"15min":4.6829999999999998},"last":0.89400000000000002}]},{"osd":1,"last update":"Wed Feb 10 04:55:46 2021","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.76100000000000001,"15min":0.871},"min":{"1min":0.46500000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.341,"5min":3.5529999999999999,"15min":7.883},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.89700000000000002,"15min":1.0580000000000001},"min":{"1min":0.50700000000000001,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.1850000000000001,"5min":3.8090000000000002,"15min":6.5060000000000002},"last":1.0580000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:55:46 2021","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":2.6829999999999998,"15min":1.702},"min":{"1min":0.627,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":7.0069999999999997,"5min":120.39,"15min":120.39},"last":1.415},{"interface":"front","average":{"1min":1.218,"5min":2.6419999999999999,"15min":1.6990000000000001},"min":{"1min":0.58999999999999997,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":7.1120000000000001,"5min":120.583,"15min":120.583},"last":1.236}]},{"osd":3,"last update":"Wed Feb 10 04:56:04 2021","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.1299999999999999,"15min":1.248},"min":{"1min":0.877,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.9530000000000001,"5min":2.1429999999999998,"15min":2.1429999999999998},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.611,"15min":1.5569999999999999},"min":{"1min":0.91900000000000004,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":2.1080000000000001,"5min":26.802,"15min":26.802},"last":1.298}]},{"osd":5,"last update":"Wed Feb 10 04:55:59 2021","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.1080000000000001,"15min":1.478},"min":{"1min":0.624,"5min":0.495,"15min":0.495},"max":{"1min":1.6579999999999999,"5min":2.351,"15min":35.774000000000001},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.325,"5min":1.2090000000000001,"15min":1.47},"min":{"1min":0.78300000000000003,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":3.8410000000000002,"5min":4.319,"15min":35.606000000000002},"last":0.97399999999999998}]},{"osd":6,"last update":"Wed Feb 10 04:56:04 2021","interfaces":[{"interface":"back","average":{"1min":1.615,"5min":1.625,"15min":2.3780000000000001},"min":{"1min":0.98499999999999999,"5min":0.86199999999999999,"15min":0.82599999999999996},"max":{"1min":3.8530000000000002,"5min":12.488,"15min":201.626},"last":1.528},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.504,"15min":2.3820000000000001},"min":{"1min":0.82099999999999995,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":4.7910000000000004,"5min":12.84,"15min":201.577},"last":1.359}]},{"osd":7,"last update":"Wed Feb 10 04:56:35 2021","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.486,"15min":1.508},"min":{"1min":1.105,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":4.6200000000000001,"5min":4.6200000000000001,"15min":5.8920000000000003},"last":1.609},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.252,"15min":1.4670000000000001},"min":{"1min":1.0029999999999999,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.738,"5min":2.198,"15min":5.149},"last":1.46}]}]},{"osd":5,"up_from":37,"seq":158913790335,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":256088,"kb_used_data":206032,"kb_used_omap":0,"kb_used_meta":50048,"kb_avail":93489064,"statfs":{"total":95995035648,"available":95732801536,"internally_reserved":0,"allocated":210976768,"data_stored":275651252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":51249152},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:56:16 2021","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.486,"5min":1.486,"15min":1.486},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.57999999999999996}]},{"osd":1,"last update":"Wed Feb 10 04:55:58 2021","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":0.93999999999999995,"15min":0.94099999999999995},"min":{"1min":0.50600000000000001,"5min":0.435,"15min":0.435},"max":{"1min":4.242,"5min":6.391,"15min":6.391},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.74,"5min":1.2310000000000001,"15min":1.1950000000000001},"min":{"1min":0.626,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":13.794,"5min":13.794,"15min":13.794},"last":0.63100000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:55:58 2021","interfaces":[{"interface":"back","average":{"1min":7.226,"5min":2.831,"15min":1.7949999999999999},"min":{"1min":0.51400000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":63.270000000000003,"5min":63.270000000000003,"15min":63.270000000000003},"last":1.069},{"interface":"front","average":{"1min":7.3579999999999997,"5min":2.9630000000000001,"15min":1.831},"min":{"1min":0.68000000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":63.289000000000001,"5min":63.289000000000001,"15min":63.289000000000001},"last":0.76700000000000002}]},{"osd":3,"last update":"Wed Feb 10 04:56:12 2021","interfaces":[{"interface":"back","average":{"1min":1.909,"5min":1.2909999999999999,"15min":1.3200000000000001},"min":{"1min":0.79300000000000004,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":13.755000000000001,"5min":13.755000000000001,"15min":13.755000000000001},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.202,"5min":1.242,"15min":1.2010000000000001},"min":{"1min":0.81899999999999995,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.7809999999999999,"5min":6.5129999999999999,"15min":6.5129999999999999},"last":1.147}]},{"osd":4,"last update":"Wed Feb 10 04:55:58 2021","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.304,"15min":1.3},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.57099999999999995},"max":{"1min":5.298,"5min":9.4890000000000008,"15min":9.4890000000000008},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.3400000000000001,"15min":1.3700000000000001},"min":{"1min":0.70899999999999996,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":4.8289999999999997,"5min":5.6180000000000003,"15min":5.6180000000000003},"last":1.28}]},{"osd":6,"last update":"Wed Feb 10 04:56:12 2021","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.4550000000000001,"15min":2.9689999999999999},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.65800000000000003},"max":{"1min":2.1989999999999998,"5min":4.9429999999999996,"15min":319.51999999999998},"last":1.504},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.4830000000000001,"15min":2.9140000000000001},"min":{"1min":0.82399999999999995,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":2.5659999999999998,"5min":6.8220000000000001,"15min":319.22300000000001},"last":1.373}]},{"osd":7,"last update":"Wed Feb 10 04:55:35 2021","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.4870000000000001,"15min":1.486},"min":{"1min":0.90300000000000002,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":5.4160000000000004,"5min":11.897,"15min":11.897},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.675,"5min":1.524,"15min":1.5720000000000001},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.82999999999999996},"max":{"1min":5.0579999999999998,"5min":6.0369999999999999,"15min":6.0369999999999999},"last":1.401}]}]}]}} 2021-02-10T04:56:40.337 INFO:tasks.thrashosds.thrasher:Setting 2.23 to [3, 0, 6] 2021-02-10T04:56:40.337 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.23', '3', '0', '6'] 2021-02-10T04:56:40.338 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pg-upmap 2.23 3 0 6 2021-02-10T04:56:40.497 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:56:40.570 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:56:40.741 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:56:40.838 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:40 smithi085 conmon[29319]: audit 2021-02-10T04:56:39.386050+0000 mgr.y (mgr.14140) 1415 : audit [DBG] 2021-02-10T04:56:40.838 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:40 smithi085 conmon[29319]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:56:40.839 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:40 smithi085 conmon[35961]: audit 2021-02-10T04:56:39 2021-02-10T04:56:40.839 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:40 smithi085 conmon[35961]: .386050+0000 mgr.y (mgr.14140) 1415 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:56:41.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:40 smithi164 conmon[29288]: audit 2021-02-10T04:56:39.386050+0000 mgr.y 2021-02-10T04:56:41.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:40 smithi164 conmon[29288]: (mgr.14140) 1415 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:56:41.697 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is set 2021-02-10T04:56:41.925 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:41 smithi085 conmon[29319]: cluster 2021-02-10T04:56:40.518842+0000 mgr.y (mgr.14140) 1416 : cluster [DBG] pgmap v2318: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 737 B/s rd, 0 op/s 2021-02-10T04:56:41.925 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:41 smithi085 conmon[29319]: audit 2021-02-10T04:56:41.204217+0000 mon.a (mon.0) 2760 : audit [INF] from='client.? 172.21.15.85:0/2381244857' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:56:41.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:41 smithi085 conmon[35961]: cluster 2021-02-10T04:56:40.518842 2021-02-10T04:56:41.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:41 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1416 : cluster [DBG] pgmap v2318: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 737 B/s rd, 0 op/s 2021-02-10T04:56:41.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:41 smithi085 conmon[35961]: audit 2021-02-10T04:56:41.204217+0000 mon.a (mon 2021-02-10T04:56:41.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:41 smithi085 conmon[35961]: .0) 2760 : audit [INF] 2021-02-10T04:56:41.925 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:41 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/2381244857' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:56:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:41 smithi164 conmon[29288]: cluster 2021-02-10T04:56:40.518842+0000 mgr.y 2021-02-10T04:56:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:41 smithi164 conmon[29288]: (mgr.14140) 1416 : cluster [DBG] pgmap v2318: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 737 B/s rd, 0 op/s 2021-02-10T04:56:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:41 smithi164 conmon[29288]: audit 2021-02-10T04:56:41.204217+0000 mon.a (mon.0) 2760 : audit [INF] 2021-02-10T04:56:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:41 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/2381244857' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:56:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:42 smithi085 conmon[29319]: cluster 2021-02-10T04:56:41.682478+0000 2021-02-10T04:56:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:42 smithi085 conmon[29319]: mon.a (mon.0) 2761 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:56:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:42 smithi085 conmon[29319]: audit 2021-02-10T04:56:41.692868+0000 mon.a (mon.0) 2762 : audit [INF] from='client.? 172.21.15.85:0/2381244857' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:56:42.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:42 smithi085 conmon[29319]: cluster 2021-02-10T04:56:41.692926+0000 mon.a (mon.0) 2763 : cluster [DBG] osdmap e1278: 8 total, 8 up, 8 in 2021-02-10T04:56:42.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:42 smithi085 conmon[35961]: cluster 2021-02-10T04:56:42.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:42 smithi085 conmon[35961]: 2021-02-10T04:56:42.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:42 smithi085 conmon[35961]: 2021-02-10T04:56:41.682478 2021-02-10T04:56:42.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:42 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2761 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:56:42.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:42 smithi085 conmon[35961]: audit 2021 2021-02-10T04:56:42.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:42 smithi085 conmon[35961]: -02-10T04:56:41.692868+0000 2021-02-10T04:56:42.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:42 smithi085 conmon[35961]: mon.a (mon.0) 2762 : audit [INF] from='client.? 172.21.15.85:0/2381244857' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:56:42.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:42 smithi085 conmon[35961]: cluster 2021-02-10T04:56:42.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:42 smithi085 conmon[35961]: 2021-02-10T04:56:41.692926 2021-02-10T04:56:42.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:42 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2763 : cluster [DBG] 2021-02-10T04:56:42.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:42 smithi085 conmon[35961]: osdmap e1278: 8 total, 8 up, 8 in 2021-02-10T04:56:43.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:42 smithi164 conmon[29288]: cluster 2021-02-10T04:56:41.682478+0000 mon.a ( 2021-02-10T04:56:43.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:42 smithi164 conmon[29288]: mon.0) 2761 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:56:43.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:42 smithi164 conmon[29288]: audit 2021-02-10T04: 2021-02-10T04:56:43.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:42 smithi164 conmon[29288]: 56:41.692868+0000 mon.a (mon.0) 2762 : audit [INF] from='client.? 172.21.15.85:0/2381244857' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:56:43.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:42 smithi164 conmon[29288]: cluster 2021-02-10T04:56:43.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:42 smithi164 conmon[29288]: 2021-02-10T04:56:43.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:42 smithi164 conmon[29288]: 2021 2021-02-10T04:56:43.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:42 smithi164 conmon[29288]: -02-10T04:56: 2021-02-10T04:56:43.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:42 smithi164 conmon[29288]: 41.692926+0000 mon.a (mon.0) 2763 : cluster [DBG] osdmap e1278: 8 total, 8 up, 8 in 2021-02-10T04:56:44.027 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:43 smithi085 conmon[29319]: cluster 2021-02-10T04:56:42.519417+0000 mgr.y (mgr. 2021-02-10T04:56:44.027 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:43 smithi085 conmon[29319]: 14140) 1417 : cluster [DBG] pgmap v2320: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 652 B/s rd, 0 op/s 2021-02-10T04:56:44.027 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:43 smithi085 conmon[29319]: cluster 2021-02-10T04:56:43.580826+0000 2021-02-10T04:56:44.028 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:43 smithi085 conmon[29319]: mon.a (mon.0) 2764 : cluster [DBG] osdmap e1279: 8 total, 8 up, 8 in 2021-02-10T04:56:44.028 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:43 smithi085 conmon[35961]: cluster 2021-02-10T04:56:42. 2021-02-10T04:56:44.028 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:43 smithi085 conmon[35961]: 519417+0000 mgr.y (mgr.14140) 1417 : cluster [DBG] pgmap v2320: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 652 B/s rd, 0 op/s 2021-02-10T04:56:44.028 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:43 smithi085 conmon[35961]: cluster 2021-02-10T04:56:44.028 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:43 smithi085 conmon[35961]: 2021-02-10T04:56:43.580826 2021-02-10T04:56:44.028 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:43 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2764 : cluster [DBG] 2021-02-10T04:56:44.028 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:43 smithi085 conmon[35961]: osdmap e1279: 8 total, 8 up, 8 in 2021-02-10T04:56:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:43 smithi164 conmon[29288]: cluster 2021-02-10T04:56:42.519417+0000 mgr.y (mgr.14140) 1417 : cluster 2021-02-10T04:56:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:43 smithi164 conmon[29288]: [DBG] pgmap v2320: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 652 B/s rd, 0 op/s 2021-02-10T04:56:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:43 smithi164 conmon[29288]: cluster 2021-02-10T04:56:43.580826+0000 mon.a (mon.0) 2764 : cluster [DBG] osdmap e1279: 8 total, 8 up, 8 in 2021-02-10T04:56:44.336 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:56:44.353 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:56:44.371 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:56:44.390 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:56:44.409 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:56:44.427 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:56:44.446 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:56:44.464 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:56:44.465 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:56:44.465 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:56:44.465 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:56:44.465 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:56:44.465 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:56:44.465 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:56:44.466 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:56:44.466 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:56:44.706 INFO:teuthology.orchestra.run.smithi085.stderr:set 2.23 pg_upmap mapping to [3,0,6] 2021-02-10T04:56:44.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:44 smithi085 conmon[35961]: audit 2021-02-10T04:56:44.313885+0000 mon.a 2021-02-10T04:56:44.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:44 smithi085 conmon[35961]: (mon.0) 2765 : audit [DBG] from='client.? 172.21.15.85:0/1983392617' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:56:44.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:44 smithi085 conmon[35961]: audit 2021-02-10T04:56:44.553264+0000 2021-02-10T04:56:44.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:44 smithi085 conmon[35961]: mon.a (mon.0) 2766 : audit [INF] from='client.? 172.21.15.85:0/1645106523' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.23", "id": [3, 0, 6]}]: dispatch 2021-02-10T04:56:44.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:44 smithi085 conmon[29319]: audit 2021-02-10T04:56:44.313885+0000 mon.a (mon 2021-02-10T04:56:44.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:44 smithi085 conmon[29319]: .0) 2765 : audit [DBG] from='client.? 172.21.15.85:0/1983392617' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:56:44.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:44 smithi085 conmon[29319]: audit 2021-02-10T04:56:44.553264+0000 mon.a (mon.0) 2766 : audit [INF] from='client.? 172.21.15.85:0/1645106523' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.23", "id": [3, 0, 6]}]: dispatch 2021-02-10T04:56:45.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:44 smithi164 conmon[29288]: audit 2021-02-10T04:56:44.313885+0000 mon.a (mon. 2021-02-10T04:56:45.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:44 smithi164 conmon[29288]: 0) 2765 : audit [DBG] from='client.? 172.21.15.85:0/1983392617' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:56:45.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:44 smithi164 conmon[29288]: audit 2021-02-10T04:56:44.553264+0000 mon.a (mon.0) 2766 : audit [INF] from='client.? 172.21.15.85:0/1645106523' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.23", "id": [3, 0, 6]}]: dispatch 2021-02-10T04:56:45.486 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset noscrub 2021-02-10T04:56:45.719 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:56:45.996 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:45 smithi085 conmon[29319]: cluster 2021-02-10T04:56: 2021-02-10T04:56:45.996 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:45 smithi085 conmon[29319]: 44.519952+0000 mgr.y (mgr.14140) 1418 : cluster [DBG] pgmap v2322: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:56:45.996 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:45 smithi085 conmon[29319]: audit 2021-02-10T04:56:44.699874+0000 mon.a (mon.0) 2767 : audit [INF] from='client.? 172.21.15.85:0/1645106523' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.23", "id": [3, 0, 6]}]': finished 2021-02-10T04:56:45.996 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:45 smithi085 conmon[29319]: cluster 2021-02-10T04:56:44.699934+0000 mon.a (mon.0) 2768 : cluster [DBG] osdmap e1280: 8 total, 8 up, 8 in 2021-02-10T04:56:45.997 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:45 smithi085 conmon[35961]: cluster 2021-02-10T04:56:44.519952 2021-02-10T04:56:45.997 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:45 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1418 : cluster [DBG] pgmap v2322: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:56:45.997 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:45 smithi085 conmon[35961]: audit 2021-02-10T04:56:44.699874+0000 mon.a (mon.0) 2767 : audit [INF] from='client.? 172.21.15.85:0/1645106523' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.23", "id": [3, 0, 6]}]': finished 2021-02-10T04:56:45.997 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:45 smithi085 conmon[35961]: cluster 2021-02-10T04:56:44.699934+0000 mon.a (mon.0) 2768 : cluster [DBG] osdmap e1280: 8 total, 8 up, 8 in 2021-02-10T04:56:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:45 smithi164 conmon[29288]: cluster 2021-02-10T04:56:44.519952+0000 mgr.y (mgr.14140) 1418 2021-02-10T04:56:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:45 smithi164 conmon[29288]: : cluster [DBG] pgmap v2322: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:56:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:45 smithi164 conmon[29288]: audit 2021-02-10T04:56:44.699874+0000 mon.a (mon.0) 2767 : audit [INF] from='client.? 172.21.15.85:0/1645106523' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.23", "id": [3, 0, 6]}]': finished 2021-02-10T04:56:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:45 smithi164 conmon[29288]: cluster 2021-02-10T04:56:44.699934+0000 mon.a ( 2021-02-10T04:56:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:45 smithi164 conmon[29288]: mon.0) 2768 : cluster [DBG] osdmap e1280: 8 total, 8 up, 8 in 2021-02-10T04:56:46.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:46 smithi085 conmon[29319]: cluster 2021-02-10T04:56:45.701977+0000 mon.a (mon.0) 2769 : cluster [DBG] osdmap e1281: 8 total, 8 up, 8 in 2021-02-10T04:56:46.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:46 smithi085 conmon[35961]: cluster 2021-02-10T04:56:45.701977+0000 mon.a (mon.0) 2769 : cluster [DBG] osdmap e1281: 8 total, 8 up, 8 in 2021-02-10T04:56:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:46 smithi164 conmon[29288]: cluster 2021-02-10T04:56:45.701977+0000 mon.a (mon.0) 2769 : cluster [DBG] 2021-02-10T04:56:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:46 smithi164 conmon[29288]: osdmap e1281: 8 total, 8 up, 8 in 2021-02-10T04:56:48.013 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:47 smithi164 conmon[29288]: cluster 2021-02-10T04:56:46.520382+0000 mgr.y (mgr 2021-02-10T04:56:48.014 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:47 smithi164 conmon[29288]: .14140) 1419 : cluster [DBG] pgmap v2325: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:56:48.016 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:56:48.041 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:47 smithi085 conmon[35961]: cluster 2021-02-10T04:56:46.520382+0000 mgr.y 2021-02-10T04:56:48.041 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:47 smithi085 conmon[35961]: (mgr.14140) 1419 : cluster [DBG] pgmap v2325: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:56:48.041 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:47 smithi085 conmon[29319]: cluster 2021-02-10T04:56:46.520382+0000 mgr.y ( 2021-02-10T04:56:48.041 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:47 smithi085 conmon[29319]: mgr.14140) 1419 : cluster [DBG] pgmap v2325: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-02-10T04:56:48.271 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:56:48.750 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is unset 2021-02-10T04:56:48.766 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:48 smithi085 conmon[29319]: audit 2021-02-10T04:56:48.109021+0000 mon.c (mon.1) 134 2021-02-10T04:56:48.767 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:48 smithi085 conmon[29319]: : audit [INF] from='client.? 172.21.15.85:0/2896657134' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:56:48.767 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:48 smithi085 conmon[29319]: audit 2021-02-10T04:56:48.109340+0000 mon.a (mon.0) 2770 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:56:48.767 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:48 smithi085 conmon[35961]: audit 2021-02-10T04:56:48.767 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:48 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:56:48.767 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:48 smithi085 conmon[35961]: 56:48.109021+0000 mon.c (mon.1) 134 : audit [INF] from='client.? 172.21.15.85:0/2896657134' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:56:48.767 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:48 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:56:48.767 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:48 smithi085 conmon[35961]: 56:48.109340+0000 mon.a (mon.0) 2770 : 2021-02-10T04:56:48.767 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:48 smithi085 conmon[35961]: audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:56:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:48 smithi164 conmon[29288]: audit 2021-02-10T04:56:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:48 smithi164 conmon[29288]: 2021-02-10T04:56:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:48 smithi164 conmon[29288]: 2021 2021-02-10T04:56:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:48 smithi164 conmon[29288]: - 2021-02-10T04:56:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:48 smithi164 conmon[29288]: 02-10T04:56:48.109021+0000 2021-02-10T04:56:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:48 smithi164 conmon[29288]: mon.c (mon.1) 134 : audit [INF] from='client.? 172.21.15.85:0/2896657134' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:56:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:48 smithi164 conmon[29288]: audit 2021-02-10T04:56: 2021-02-10T04:56:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:48 smithi164 conmon[29288]: 48.109340+0000 mon.a (mon.0) 2770 : audit 2021-02-10T04:56:48.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:48 smithi164 conmon[29288]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:56:49.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:56:49 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:56:49.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:49 smithi085 conmon[29319]: cluster 2021-02-10T04:56:48.521078+0000 mgr.y (mgr.14140 2021-02-10T04:56:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:49 smithi085 conmon[29319]: ) 1420 : cluster [DBG] pgmap v2326: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2021-02-10T04:56:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:49 smithi085 conmon[29319]: cluster 2021-02-10T04:56:48.583765+0000 osd.7 (osd.7) 94 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:56:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:49 smithi085 conmon[29319]: cluster 2021-02-10T04:56:48.585692+0000 osd.7 (osd.7) 95 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:56:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:49 smithi085 conmon[29319]: cluster 2021-02-10T04:56:48.727126+0000 mon.a (mon.0) 2771 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:56:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:49 smithi085 conmon[29319]: audit 2021-02-10T04:56:48.740415+0000 mon.a (mon.0) 2772 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:56:49.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:49 smithi085 conmon[29319]: cluster 2021-02-10T04:56:48.740483+0000 mon.a (mon.0) 2773 : cluster [DBG] osdmap e1282: 8 total, 8 up, 8 in 2021-02-10T04:56:49.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:49 smithi085 conmon[35961]: cluster 2021-02-10T04:56:48.521078+0000 2021-02-10T04:56:49.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:49 smithi085 conmon[35961]: mgr.y (mgr.14140) 1420 : cluster [DBG] pgmap v2326: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2021-02-10T04:56:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:49 smithi085 conmon[35961]: cluster 2021-02-10T04:56:48.583765+0000 osd.7 (osd.7) 94 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:56:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:49 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:56:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:49 smithi085 conmon[35961]: 04:56:48.585692+0000 osd.7 (osd.7) 95 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:56:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:49 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:56:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:49 smithi085 conmon[35961]: 04:56:48.727126+0000 mon.a (mon.0) 2771 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:56:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:49 smithi085 conmon[35961]: audit 2021-02-10T04:56 2021-02-10T04:56:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:49 smithi085 conmon[35961]: :48.740415+0000 mon.a (mon.0) 2772 2021-02-10T04:56:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:49 smithi085 conmon[35961]: : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:56:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:49 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:56:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:49 smithi085 conmon[35961]: -02-10T04:56:48.740483+0000 2021-02-10T04:56:49.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:49 smithi085 conmon[35961]: mon.a (mon.0) 2773 : cluster [DBG] osdmap e1282: 8 total, 8 up, 8 in 2021-02-10T04:56:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:49 smithi164 conmon[29288]: cluster 2021-02-10T04:56:48.521078 2021-02-10T04:56:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:49 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1420 : cluster [DBG] pgmap v2326: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2021-02-10T04:56:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:49 smithi164 conmon[29288]: cluster 2021-02-10T04:56: 2021-02-10T04:56:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:49 smithi164 conmon[29288]: 48.583765+0000 osd.7 (osd.7) 94 : cluster [DBG] 2021-02-10T04:56:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:49 smithi164 conmon[29288]: purged_snaps scrub starts 2021-02-10T04:56:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:49 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:56:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:49 smithi164 conmon[29288]: -10T04:56:48.585692 2021-02-10T04:56:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:49 smithi164 conmon[29288]: +0000 osd.7 (osd.7) 95 2021-02-10T04:56:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:49 smithi164 conmon[29288]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:56:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:49 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:56:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:49 smithi164 conmon[29288]: 02-10T04:56:48.727126+0000 mon.a (mon. 2021-02-10T04:56:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:49 smithi164 conmon[29288]: 0) 2771 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:56:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:49 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:56:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:49 smithi164 conmon[29288]: 02-10T04:56:48.740415+0000 mon.a ( 2021-02-10T04:56:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:49 smithi164 conmon[29288]: mon.0) 2772 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:56:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:49 smithi164 conmon[29288]: cluster 2021-02-10T04:56:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:49 smithi164 conmon[29288]: 2021-02-10T04:56:48.740483+0000 2021-02-10T04:56:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:49 smithi164 conmon[29288]: mon.a (mon.0) 2773 : cluster [DBG] osdmap e1282: 8 total, 8 up, 8 in 2021-02-10T04:56:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:49 smithi164 conmon[29288]: 2021-02-10T04:56:50.807 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:56:50.830 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:56:50.848 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:56:50.866 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:56:50.885 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:56:50.906 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:56:50.928 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:56:50.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:50 smithi085 conmon[35961]: audit 2021-02-10T04:56:49.396019+0000 mgr.y (mgr.14140) 1421 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:56:50.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:50 smithi085 conmon[35961]: audit 2021-02-10T04:56:50.299057+0000 mon.a (mon.0) 2774 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:56:50.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:50 smithi085 conmon[29319]: audit 2021-02-10T04:56:49.396019+0000 mgr.y (mgr.14140) 1421 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:56:50.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:50 smithi085 conmon[29319]: audit 2021-02-10T04:56:50.299057+0000 mon.a (mon.0) 2774 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:56:50.948 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:56:50.949 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:56:50.949 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:56:50.949 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:56:50.950 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:56:50.950 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:56:50.950 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:56:50.950 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:56:50.950 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:56:50.954 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:56:50.954 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:56:51.041 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2021-02-10T04:56:51.041 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:56:51.042 INFO:tasks.thrashosds.thrasher:inject_pause on 7 2021-02-10T04:56:51.042 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2021-02-10T04:56:51.042 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2021-02-10T04:56:51.042 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.7 config set filestore_inject_stall 3 2021-02-10T04:56:51.092 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:50 smithi164 conmon[29288]: audit 2021-02-10T04:56:49.396019 2021-02-10T04:56:51.093 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:50 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1421 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:56:51.093 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:50 smithi164 conmon[29288]: audit 2021-02-10T04:56:50.299057+0000 mon.a (mon.0) 2774 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:56:51.367 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:56:51 smithi085 conmon[45443]: debug 2021-02-10T04:56:51.364+0000 7f71254f7700 -1 osd.1 pg_epoch: 1282 pg[5.1d( empty local-lis/les=1227/1228 n=0 ec=57/57 lis/c=1227/1227 les/c/f=1228/1228/0 sis=1227) [1,4,0] r=0 lpr=1227 crt=0'0 mlcod 0'0 active+clean] scrubber pg(5.1d) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x55dbcdb33080) 2021-02-10T04:56:51.754 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:56:52.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:51 smithi164 conmon[29288]: cluster 2021-02-10T04:56:52.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:51 smithi164 conmon[29288]: 2021-02- 2021-02-10T04:56:52.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:51 smithi164 conmon[29288]: 10T04:56:50.521681+0000 mgr.y (mgr.14140) 1422 : cluster [DBG] pgmap v2328: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:56:52.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:51 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:56:52.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:51 smithi164 conmon[29288]: 02-10T04:56:50.788255+0000 mon.a (mon. 2021-02-10T04:56:52.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:51 smithi164 conmon[29288]: 0) 2775 : audit [DBG] from='client.? 172.21.15.85:0/2236163676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:56:52.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:51 smithi164 conmon[29288]: audit 2021-02-10T04:56:52.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:51 smithi164 conmon[29288]: 2021-02- 2021-02-10T04:56:52.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:51 smithi164 conmon[29288]: 10T04:56:50.909783+0000 mon.a (mon 2021-02-10T04:56:52.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:51 smithi164 conmon[29288]: .0) 2776 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:56:52.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:51 smithi164 conmon[29288]: cephadm 2021-02-10T04:56:52.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:51 smithi164 conmon[29288]: 2021-02-10T04:56:50.910414+0000 2021-02-10T04:56:52.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:51 smithi164 conmon[29288]: mgr.y (mgr.14140) 1423 : cephadm [INF] 2021-02-10T04:56:52.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:51 smithi164 conmon[29288]: Refreshing smithi164 facts 2021-02-10T04:56:52.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:51 smithi085 conmon[29319]: cluster 2021-02-10T04:56:50.521681+0000 mgr.y (mgr. 2021-02-10T04:56:52.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:51 smithi085 conmon[29319]: 14140) 1422 : cluster [DBG] pgmap v2328: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:56:52.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:51 smithi085 conmon[29319]: audit 2021-02-10T04:56:50.788255+0000 mon.a (mon.0) 2775 : audit [DBG] from='client.? 172.21.15.85:0/2236163676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:56:52.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:51 smithi085 conmon[29319]: audit 2021-02-10T04:56:50.909783+0000 mon.a (mon.0) 2776 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:56:52.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:51 smithi085 conmon[29319]: cephadm 2021-02-10T04:56:50.910414+0000 mgr.y (mgr.14140) 1423 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:56:52.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:51 smithi085 conmon[35961]: cluster 2021-02-10T04:56: 2021-02-10T04:56:52.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:51 smithi085 conmon[35961]: 50.521681+0000 mgr.y (mgr.14140) 1422 : cluster [DBG] pgmap v2328: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:56:52.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:51 smithi085 conmon[35961]: audit 2021-02-10T04:56:52.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:51 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:56:52.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:51 smithi085 conmon[35961]: :56:50.788255+0000 mon.a 2021-02-10T04:56:52.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:51 smithi085 conmon[35961]: (mon.0) 2775 : audit 2021-02-10T04:56:52.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:51 smithi085 conmon[35961]: [DBG] from='client.? 172.21.15.85:0/2236163676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:56:52.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:51 smithi085 conmon[35961]: audit 2021-02-10T04:56:50.909783+0000 mon.a (mon. 2021-02-10T04:56:52.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:51 smithi085 conmon[35961]: 0) 2776 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:56:52.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:51 smithi085 conmon[35961]: cephadm 2021-02-10T04:56:52.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:51 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:56:52.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:51 smithi085 conmon[35961]: 10T04:56:50.910414+0000 mgr.y 2021-02-10T04:56:52.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:51 smithi085 conmon[35961]: (mgr.14140) 1423 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:56:52.224 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset nodeep-scrub 2021-02-10T04:56:52.439 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:56:53.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:53 smithi164 conmon[29288]: audit 2021-02-10T04:56:51.997447+0000 mon.a (mon.0) 2777 2021-02-10T04:56:53.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:53 smithi164 conmon[29288]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:56:53.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:53 smithi085 conmon[29319]: audit 2021-02-10T04:56:51.997447+0000 mon.a (mon.0) 2021-02-10T04:56:53.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:53 smithi085 conmon[29319]: 2777 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:56:53.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:53 smithi085 conmon[35961]: audit 2021-02-10T04:56:51.997447+0000 2021-02-10T04:56:53.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:53 smithi085 conmon[35961]: mon.a (mon.0) 2777 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:56:53.777 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:56:54.024 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:56:54.069 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:54 smithi085 conmon[29319]: cluster 2021-02-10T04:56:52.522246+0000 2021-02-10T04:56:54.069 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:54 smithi085 conmon[29319]: mgr.y (mgr.14140) 1424 : cluster [DBG] pgmap v2329: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 654 B/s rd, 0 op/s 2021-02-10T04:56:54.069 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:54 smithi085 conmon[29319]: cluster 2021-02-10T04:56:53.584219+0000 mon.a (mon.0) 2778 : cluster [DBG] osdmap e1283: 8 total, 8 up, 8 in 2021-02-10T04:56:54.070 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:54 smithi085 conmon[35961]: cluster 2021-02-10T04:56:52.522246 2021-02-10T04:56:54.070 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:54 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1424 : cluster [DBG] pgmap v2329: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 654 B/s rd, 0 op/s 2021-02-10T04:56:54.070 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:54 smithi085 conmon[35961]: cluster 2021-02-10T04:56:53.584219+0000 mon.a (mon.0) 2778 : cluster [DBG] osdmap e1283: 8 total, 8 up, 8 in 2021-02-10T04:56:54.135 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.0 dump_ops_in_flight 2021-02-10T04:56:54.266 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:54 smithi164 conmon[29288]: cluster 2021-02-10T04:56:52.522246+0000 mgr.y (mgr.14140) 2021-02-10T04:56:54.266 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:54 smithi164 conmon[29288]: 1424 : cluster [DBG] pgmap v2329: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 654 B/s rd, 0 op/s 2021-02-10T04:56:54.266 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:54 smithi164 conmon[29288]: cluster 2021-02-10T04:56:53.584219+0000 mon.a (mon.0) 2778 : cluster [DBG] osdmap e1283: 8 total, 8 up, 8 in 2021-02-10T04:56:54.384 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:56:55.019 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is unset 2021-02-10T04:56:55.151 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:55 smithi085 conmon[29319]: audit 2021-02-10T04:56:54.521412+0000 mon.c (mon.1) 135 : audit [INF] from='client.? 172.21.15.85:0/3518199064' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:56:55.151 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:55 smithi085 conmon[29319]: audit 2021-02-10T04:56:54.521732+0000 mon.a (mon.0) 2779 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:56:55.151 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:55 smithi085 conmon[35961]: audit 2021-02-10T04:56:54.521412+0000 mon.c (mon.1) 135 : audit [INF] from='client.? 172.21.15.85:0/3518199064' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:56:55.152 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:55 smithi085 conmon[35961]: audit 2021-02-10T04:56:54.521732+0000 mon.a (mon.0) 2779 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:56:55.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:55 smithi164 conmon[29288]: audit 2021-02-10T04:56:54.521412+0000 mon.c (mon. 2021-02-10T04:56:55.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:55 smithi164 conmon[29288]: 1) 135 : audit [INF] from='client.? 172.21.15.85:0/3518199064' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:56:55.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:55 smithi164 conmon[29288]: audit 2021-02-10T04:56:54.521732+0000 mon.a (mon.0) 2779 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:56:56.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:56 smithi085 conmon[29319]: cluster 2021-02-10T04:56:54.522704+0000 mgr.y (mgr.14140 2021-02-10T04:56:56.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:56 smithi085 conmon[29319]: ) 1425 : cluster [DBG] pgmap v2331: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:56:56.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:56 smithi085 conmon[29319]: cluster 2021-02-10T04:56:55.006618+0000 mon.a (mon.0) 2780 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:56:56.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:56 smithi085 conmon[29319]: cluster 2021-02-10T04:56:55.006644+0000 mon.a (mon.0) 2781 : cluster [INF] Cluster is now healthy 2021-02-10T04:56:56.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:56 smithi085 conmon[29319]: audit 2021-02-10T04:56:55.014597+0000 mon.a (mon.0) 2782 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:56:56.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:56 smithi085 conmon[29319]: cluster 2021-02-10T04:56:55.014698+0000 mon.a (mon.0) 2783 : cluster 2021-02-10T04:56:56.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:56 smithi085 conmon[29319]: [DBG] osdmap e1284: 8 total, 8 up, 8 in 2021-02-10T04:56:56.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:56 smithi085 conmon[29319]: cluster 2021-02-10T04:56:55.018630+0000 osd.4 (osd.4) 2659 : cluster [DBG] 2021-02-10T04:56:56.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:56 smithi085 conmon[29319]: purged_snaps scrub starts 2021-02-10T04:56:56.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:56 smithi085 conmon[29319]: cluster 2021-02-10T04:56:56.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:56 smithi085 conmon[29319]: 2021-02-10T04:56:55.019831+0000 2021-02-10T04:56:56.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:56 smithi085 conmon[29319]: osd.4 (osd.4) 2660 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:56:56.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:56 smithi085 conmon[29319]: 2021-02-10T04:56:56.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:56 smithi085 conmon[35961]: cluster 2021-02-10T04:56:54.522704+0000 mgr.y (mgr.14140) 2021-02-10T04:56:56.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:56 smithi085 conmon[35961]: 1425 : cluster [DBG] pgmap v2331: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:56:56.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:56 smithi085 conmon[35961]: cluster 2021-02-10T04:56:55.006618+0000 mon.a (mon.0) 2780 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:56:56.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:56 smithi085 conmon[35961]: cluster 2021-02-10T04:56:55.006644+0000 mon.a (mon.0 2021-02-10T04:56:56.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:56 smithi085 conmon[35961]: ) 2781 : cluster [INF] Cluster is now healthy 2021-02-10T04:56:56.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:56 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:56:56.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:56 smithi085 conmon[35961]: 10T04:56:55.014597+0000 mon.a (mon 2021-02-10T04:56:56.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:56 smithi085 conmon[35961]: .0) 2782 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:56:56.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:56 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:56:56.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:56 smithi085 conmon[35961]: 02-10T04:56:55.014698+0000 2021-02-10T04:56:56.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:56 smithi085 conmon[35961]: mon.a (mon.0) 2783 : cluster [DBG] osdmap e1284: 8 total, 8 up, 8 in 2021-02-10T04:56:56.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:56 smithi085 conmon[35961]: 2021-02-10T04:56:56.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:56 smithi085 conmon[35961]: cluster 2021-02-10T04:56:56.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:56 smithi085 conmon[35961]: 2021-02-10T04:56:55.018630+0000 osd.4 (osd.4) 2659 : cluster 2021-02-10T04:56:56.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:56 smithi085 conmon[35961]: [DBG] purged_snaps scrub starts 2021-02-10T04:56:56.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:56 smithi085 conmon[35961]: cluster 2021-02-10T04:56:55.019831+0000 osd.4 (osd.4) 2660 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:56:56.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:56 smithi164 conmon[29288]: cluster 2021-02-10T04:56:54.522704 2021-02-10T04:56:56.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:56 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1425 : cluster [DBG] pgmap v2331: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:56:56.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:56 smithi164 conmon[29288]: cluster 2021-02-10T04:56:55.006618+0000 mon.a (mon.0 2021-02-10T04:56:56.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:56 smithi164 conmon[29288]: ) 2780 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:56:56.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:56 smithi164 conmon[29288]: cluster 2021-02-10T04:56:56.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:56 smithi164 conmon[29288]: 2021-02-10T04:56:55.006644+0000 mon.a (mon. 2021-02-10T04:56:56.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:56 smithi164 conmon[29288]: 0) 2781 : cluster [INF] Cluster is now healthy 2021-02-10T04:56:56.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:56 smithi164 conmon[29288]: audit 2021 2021-02-10T04:56:56.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:56 smithi164 conmon[29288]: -02-10T04:56:55.014597+0000 2021-02-10T04:56:56.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:56 smithi164 conmon[29288]: mon.a (mon.0) 2782 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:56:56.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:56 smithi164 conmon[29288]: cluster 2021-02-10T04:56:56.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:56 smithi164 conmon[29288]: 2021-02-10T04:56: 2021-02-10T04:56:56.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:56 smithi164 conmon[29288]: 55.014698+0000 mon.a (mon 2021-02-10T04:56:56.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:56 smithi164 conmon[29288]: .0) 2783 : cluster [DBG] 2021-02-10T04:56:56.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:56 smithi164 conmon[29288]: osdmap e1284: 8 total, 8 up, 8 in 2021-02-10T04:56:56.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:56 smithi164 conmon[29288]: cluster 2021-02-10T04:56:55 2021-02-10T04:56:56.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:56 smithi164 conmon[29288]: .018630+0000 osd.4 (osd.4) 2659 : cluster 2021-02-10T04:56:56.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:56 smithi164 conmon[29288]: [DBG] purged_snaps scrub starts 2021-02-10T04:56:56.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:56 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:56:56.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:56 smithi164 conmon[29288]: 56:55.019831+0000 osd.4 (osd.4) 2021-02-10T04:56:56.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:56 smithi164 conmon[29288]: 2660 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:56:57.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:57 smithi164 conmon[29288]: cluster 2021-02-10T04:56:56.014838+0000 mon.a ( 2021-02-10T04:56:57.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:57 smithi164 conmon[29288]: mon.0) 2784 : cluster [DBG] osdmap e1285: 8 total, 8 up, 8 in 2021-02-10T04:56:57.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:57 smithi085 conmon[29319]: cluster 2021-02-10T04:56:56.014838 2021-02-10T04:56:57.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:57 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2784 : cluster [DBG] osdmap e1285: 8 total, 8 up, 8 in 2021-02-10T04:56:57.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:57 smithi085 conmon[35961]: cluster 2021-02-10T04:56:56.014838+0000 mon.a 2021-02-10T04:56:57.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:57 smithi085 conmon[35961]: (mon.0) 2784 : cluster [DBG] osdmap e1285: 8 total, 8 up, 8 in 2021-02-10T04:56:57.815 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:56:57.821 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:56:57.821 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [], 2021-02-10T04:56:57.822 INFO:teuthology.orchestra.run.smithi085.stdout: "num_ops": 0 2021-02-10T04:56:57.822 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:56:57.825 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:56:57.825 INFO:teuthology.orchestra.run.smithi164.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2021-02-10T04:56:57.825 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:56:57.833 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:56:57.851 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:56:57.870 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:56:57.894 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:56:57.914 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:56:57.933 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:56:57.952 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:56:57.953 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:56:57.953 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:56:57.953 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:56:57.953 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:56:57.953 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:56:57.953 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:56:57.953 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:56:57.954 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:56:58.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:58 smithi164 conmon[29288]: cluster 2021-02-10T04:56:56 2021-02-10T04:56:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:58 smithi164 conmon[29288]: .523133+0000 mgr.y (mgr.14140) 1426 : cluster [DBG] pgmap v2334: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:56:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:56:58 smithi164 conmon[29288]: audit 2021-02-10T04:56:57.790673+0000 mon.a (mon.0) 2785 : audit [DBG] from='client.? 172.21.15.85:0/1970425799' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:56:58.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:58 smithi085 conmon[29319]: cluster 2021-02-10T04:56:56.523133+0000 mgr.y (mgr.14140) 1426 : cluster [DBG] pgmap v2334: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:56:58.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:56:58 smithi085 conmon[29319]: audit 2021-02-10T04:56:57.790673+0000 mon.a (mon.0) 2785 : audit [DBG] from='client.? 172.21.15.85:0/1970425799' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:56:58.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:58 smithi085 conmon[35961]: cluster 2021-02-10T04:56:56.523133+0000 mgr.y (mgr.14140) 1426 : cluster [DBG] pgmap v2334: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:56:58.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:56:58 smithi085 conmon[35961]: audit 2021-02-10T04:56:57.790673+0000 mon.a (mon.0) 2785 : audit [DBG] from='client.? 172.21.15.85:0/1970425799' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:56:58.402 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set noscrub 2021-02-10T04:56:58.618 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:56:59.156 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.0 dump_blocked_ops 2021-02-10T04:56:59.376 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:56:59.893 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:56:59 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:57:00.027 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:00 smithi085 conmon[35961]: cluster 2021-02-10T04:56:58 2021-02-10T04:57:00.027 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:00 smithi085 conmon[35961]: .523878+0000 mgr.y (mgr.14140) 1427 : cluster [DBG] pgmap v2335: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2021-02-10T04:57:00.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:00 smithi164 conmon[29288]: cluster 2021-02-10T04:56:58.523878+0000 mgr.y (mgr.14140 2021-02-10T04:57:00.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:00 smithi164 conmon[29288]: ) 1427 : cluster [DBG] pgmap v2335: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2021-02-10T04:57:00.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:00 smithi085 conmon[29319]: cluster 2021-02-10T04:56:58.523878+0000 mgr.y 2021-02-10T04:57:00.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:00 smithi085 conmon[29319]: (mgr.14140) 1427 : cluster [DBG] pgmap v2335: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2021-02-10T04:57:00.442 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:57:00 smithi085 conmon[49106]: debug 2021-02-10T04:57:00.175+0000 7f7788e19700 -1 osd.2 pg_epoch: 1286 pg[5.8( empty local-lis/les=1227/1228 n=0 ec=57/57 lis/c=1227/1227 les/c/f=1228/1228/0 sis=1227) [2,0,3] r=0 lpr=1227 crt=0'0 mlcod 0'0 active+clean] scrubber pg(5.8) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x564aa6fe98c0) 2021-02-10T04:57:01.032 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:01 smithi164 conmon[29288]: audit 2021-02-10T04:56:59.406017+0000 mgr.y ( 2021-02-10T04:57:01.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:01 smithi164 conmon[29288]: mgr.14140) 1428 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:57:01.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:01 smithi164 conmon[29288]: cluster 2021-02-10T04:56:59.569758+0000 osd.3 (osd.3) 40 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:57:01.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:01 smithi164 conmon[29288]: cluster 2021-02-10T04:56:59.573134+0000 osd.3 (osd.3) 41 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:57:01.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:01 smithi164 conmon[29288]: cluster 2021-02-10T04:57:00.022937+0000 mon.a (mon.0) 2786 : cluster [DBG] osdmap e1286: 8 total, 8 up, 8 in 2021-02-10T04:57:01.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:01 smithi164 conmon[29288]: cluster 2021-02-10T04:57:00.439734+0000 osd.1 (osd.1) 509 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:57:01.033 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:01 smithi164 conmon[29288]: cluster 2021-02-10T04:57:00.442561+0000 osd.1 (osd.1) 510 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:57:01.217 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:57:01.426 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[29319]: audit 2021-02-10T04:56:59.406017+0000 2021-02-10T04:57:01.426 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[29319]: mgr.y (mgr.14140) 1428 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:57:01.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[29319]: cluster 2021-02-10T04:56:59 2021-02-10T04:57:01.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[29319]: .569758+0000 osd.3 (osd.3) 40 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:57:01.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[29319]: cluster 2021-02-10T04:57:01.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[29319]: 2021-02-10T04:56:59. 2021-02-10T04:57:01.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[29319]: 573134+0000 osd.3 (osd.3) 41 : cluster [DBG] 2021-02-10T04:57:01.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[29319]: purged_snaps scrub ok 2021-02-10T04:57:01.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:57:01.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[29319]: -10T04:57: 2021-02-10T04:57:01.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[29319]: 00.022937+0000 mon.a ( 2021-02-10T04:57:01.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[29319]: mon.0) 2786 : cluster [DBG] osdmap e1286: 8 total, 8 up, 8 in 2021-02-10T04:57:01.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[29319]: 2021-02-10T04:57:01.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:57:01.427 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[29319]: T04:57:00.439734+0000 osd.1 2021-02-10T04:57:01.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[29319]: (osd.1) 509 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:57:01.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:57:01.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[29319]: -10T04:57:00.442561+0000 osd.1 (osd.1) 2021-02-10T04:57:01.428 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[29319]: 510 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:57:01.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[35961]: audit 2021-02-10T04:56:59. 2021-02-10T04:57:01.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[35961]: 406017+0000 mgr.y (mgr.14140) 1428 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:57:01.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[35961]: cluster 2021-02-10T04:57:01.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[35961]: 2021-02-10T04:56:59.569758+0000 osd.3 (osd.3) 40 : cluster 2021-02-10T04:57:01.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[35961]: [DBG] purged_snaps scrub starts 2021-02-10T04:57:01.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:57:01.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[35961]: -10T04: 2021-02-10T04:57:01.428 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[35961]: 56:59.573134+0000 osd.3 (osd.3) 41 : cluster 2021-02-10T04:57:01.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[35961]: [DBG] purged_snaps scrub ok 2021-02-10T04:57:01.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:57:01.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[35961]: 04:57:00.022937+0000 mon.a (mon.0 2021-02-10T04:57:01.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[35961]: ) 2786 : cluster [DBG] osdmap e1286: 8 total, 8 up, 8 in 2021-02-10T04:57:01.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:57:01.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[35961]: 10T04:57:00.439734+0000 2021-02-10T04:57:01.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[35961]: osd.1 (osd.1) 509 : cluster [DBG] 2021-02-10T04:57:01.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[35961]: purged_snaps scrub starts 2021-02-10T04:57:01.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[35961]: cluster 2021-02-10T04:57:01.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[35961]: 2021-02-10T04:57:00.442561+0000 osd.1 2021-02-10T04:57:01.429 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:01 smithi085 conmon[35961]: (osd.1) 510 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:57:01.462 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:01.955 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:57:01.956 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [], 2021-02-10T04:57:01.956 INFO:teuthology.orchestra.run.smithi085.stdout: "complaint_time": 30, 2021-02-10T04:57:01.956 INFO:teuthology.orchestra.run.smithi085.stdout: "num_blocked_ops": 0 2021-02-10T04:57:01.956 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:57:02.096 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is set 2021-02-10T04:57:02.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:02 smithi164 conmon[29288]: cluster 2021-02-10T04:57:00.524382+0000 2021-02-10T04:57:02.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:02 smithi164 conmon[29288]: mgr.y (mgr.14140) 1429 : cluster [DBG] pgmap v2337: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:02.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:02 smithi164 conmon[29288]: cluster 2021-02-10T04:57:01.032833+0000 mon.a (mon.0) 2787 : cluster [DBG] osdmap e1287: 8 total, 8 up, 8 in 2021-02-10T04:57:02.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:02 smithi164 conmon[29288]: audit 2021-02-10T04:57:01.925453+0000 mon.a (mon.0) 2788 : 2021-02-10T04:57:02.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:02 smithi164 conmon[29288]: audit [INF] from='client.? 172.21.15.85:0/3408516774' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:57:02.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:02 smithi085 conmon[29319]: cluster 2021-02-10T04:57:00.524382+0000 mgr.y (mgr.14140) 1429 : cluster [DBG] pgmap v2337: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:02.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:02 smithi085 conmon[29319]: cluster 2021-02-10T04:57:01.032833+0000 mon.a (mon.0) 2787 : cluster [DBG] osdmap e1287: 8 total, 8 up, 8 in 2021-02-10T04:57:02.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:02 smithi085 conmon[29319]: audit 2021-02-10T04:57:01.925453+0000 mon.a (mon.0) 2788 : audit [INF] from='client.? 172.21.15.85:0/3408516774' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:57:02.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:02 smithi085 conmon[35961]: cluster 2021-02-10T04:57:00.524382+0000 mgr.y (mgr.14140) 1429 : cluster [DBG] pgmap v2337: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:02.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:02 smithi085 conmon[35961]: cluster 2021-02-10T04:57:01.032833+0000 mon.a (mon.0) 2787 : cluster [DBG] osdmap e1287: 8 total, 8 up, 8 in 2021-02-10T04:57:02.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:02 smithi085 conmon[35961]: audit 2021-02-10T04:57:01.925453+0000 mon.a (mon.0) 2788 : audit [INF] from='client.? 172.21.15.85:0/3408516774' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:57:03.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:03 smithi085 conmon[29319]: cluster 2021-02-10T04:57:02.030451+0000 2021-02-10T04:57:03.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:03 smithi085 conmon[29319]: mon.a (mon.0) 2789 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:57:03.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:03 smithi085 conmon[29319]: audit 2021-02-10T04:57:02.038655+0000 mon.a (mon.0) 2790 : audit [INF] from='client.? 172.21.15.85:0/3408516774' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:57:03.384 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:03 smithi085 conmon[29319]: cluster 2021-02-10T04:57:02.038697+0000 mon.a (mon.0) 2791 : cluster [DBG] osdmap e1288: 8 total, 8 up, 8 in 2021-02-10T04:57:03.384 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:03 smithi085 conmon[35961]: cluster 2021-02-10T04:57: 2021-02-10T04:57:03.384 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:03 smithi085 conmon[35961]: 02.030451+0000 mon.a (mon.0) 2789 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:57:03.384 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:03 smithi085 conmon[35961]: audit 2021 2021-02-10T04:57:03.384 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:03 smithi085 conmon[35961]: -02-10T04:57: 2021-02-10T04:57:03.384 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:03 smithi085 conmon[35961]: 02.038655+0000 mon.a (mon.0 2021-02-10T04:57:03.385 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:03 smithi085 conmon[35961]: ) 2790 : audit [INF] from='client.? 172.21.15.85:0/3408516774' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:57:03.385 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:03 smithi085 conmon[35961]: 2021-02-10T04:57:03.385 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:03 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:57:03.385 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:03 smithi085 conmon[35961]: :57:02.038697+0000 mon.a ( 2021-02-10T04:57:03.385 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:03 smithi085 conmon[35961]: mon.0) 2791 : cluster [DBG] osdmap e1288: 8 total, 8 up, 8 in 2021-02-10T04:57:03.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:03 smithi164 conmon[29288]: cluster 2021-02-10T04:57:02.030451+0000 mon.a 2021-02-10T04:57:03.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:03 smithi164 conmon[29288]: (mon.0) 2789 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:57:03.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:03 smithi164 conmon[29288]: audit 2021-02-10T04:57:02.038655+0000 mon.a (mon.0) 2790 : audit 2021-02-10T04:57:03.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:03 smithi164 conmon[29288]: [INF] from='client.? 172.21.15.85:0/3408516774' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:57:03.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:03 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:57:03.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:03 smithi164 conmon[29288]: 02-10T04:57:02.038697+0000 mon.a ( 2021-02-10T04:57:03.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:03 smithi164 conmon[29288]: mon.0) 2791 : cluster [DBG] 2021-02-10T04:57:03.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:03 smithi164 conmon[29288]: osdmap e1288: 8 total, 8 up, 8 in 2021-02-10T04:57:03.545 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.0 dump_historic_ops 2021-02-10T04:57:03.754 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:04.311 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:04 smithi085 conmon[29319]: cluster 2021-02-10T04:57:02.525159+0000 mgr.y 2021-02-10T04:57:04.311 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:04 smithi085 conmon[29319]: (mgr.14140) 1430 : cluster [DBG] pgmap v2340: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:04.311 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:04 smithi085 conmon[29319]: cluster 2021-02-10T04:57:03.587456+0000 mon.a (mon.0) 2792 : cluster [DBG] osdmap e1289: 8 total, 8 up, 8 in 2021-02-10T04:57:04.311 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:04 smithi085 conmon[35961]: cluster 2021-02-10T04:57:02.525159+0000 mgr.y ( 2021-02-10T04:57:04.312 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:04 smithi085 conmon[35961]: mgr.14140) 1430 : cluster [DBG] pgmap v2340: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:04.312 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:04 smithi085 conmon[35961]: cluster 2021-02-10T04:57:03.587456+0000 mon.a (mon.0) 2792 : cluster [DBG] osdmap e1289: 8 total, 8 up, 8 in 2021-02-10T04:57:04.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:04 smithi164 conmon[29288]: cluster 2021-02-10T04:57:02.525159+0000 mgr.y (mgr.14140 2021-02-10T04:57:04.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:04 smithi164 conmon[29288]: ) 1430 : cluster [DBG] pgmap v2340: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:04.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:04 smithi164 conmon[29288]: cluster 2021-02-10T04:57:03.587456+0000 mon.a (mon.0) 2792 : cluster [DBG] osdmap e1289: 8 total, 8 up, 8 in 2021-02-10T04:57:05.027 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:57:05.047 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:57:05.066 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:57:05.085 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:57:05.110 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:57:05.128 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:57:05.147 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:57:05.165 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:57:05.166 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:57:05.166 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:57:05.166 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:57:05.167 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:57:05.167 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:57:05.167 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:57:05.167 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:57:05.167 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:57:05.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:05 smithi085 conmon[29319]: audit 2021-02-10T04:57:04.999620+0000 mon.c (mon.1) 136 : audit [DBG] from='client.? 172.21.15.85:0/2113449241' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:57:05.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:05 smithi085 conmon[35961]: audit 2021-02-10T04:57:04.999620+0000 2021-02-10T04:57:05.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:05 smithi085 conmon[35961]: mon.c (mon.1) 136 : audit [DBG] from='client.? 172.21.15.85:0/2113449241' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:57:05.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:05 smithi164 conmon[29288]: audit 2021-02-10T04:57:04.999620+0000 mon.c (mon.1) 136 : audit [DBG] from='client.? 172.21.15.85:0/2113449241' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:57:05.721 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2021-02-10T04:57:05.721 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 0 1 2021-02-10T04:57:05.927 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:06.051 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:06 smithi085 conmon[35961]: cluster 2021-02-10T04:57:04.525629+0000 mgr.y (mgr. 2021-02-10T04:57:06.051 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:06 smithi085 conmon[35961]: 14140) 1431 : cluster [DBG] pgmap v2342: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:57:06.236 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set nodeep-scrub 2021-02-10T04:57:06.306 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:06 smithi085 conmon[29319]: cluster 2021-02-10T04:57:04.525629+0000 mgr.y ( 2021-02-10T04:57:06.306 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:06 smithi085 conmon[29319]: mgr.14140) 1431 : cluster [DBG] pgmap v2342: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:57:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:06 smithi164 conmon[29288]: cluster 2021-02-10T04:57:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:06 smithi164 conmon[29288]: 2021-02 2021-02-10T04:57:06.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:06 smithi164 conmon[29288]: -10T04:57:04.525629+0000 mgr.y (mgr.14140) 1431 : cluster [DBG] pgmap v2342: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:57:06.475 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:06.855 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:57:06.855 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:57:08.246 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:08 smithi085 conmon[29319]: cluster 2021-02-10T04:57:06.526113+0000 mgr.y (mgr.14140) 1432 : cluster [DBG] pgmap v2343: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:08.247 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:08 smithi085 conmon[35961]: cluster 2021-02-10T04:57:06.526113+0000 mgr.y (mgr.14140) 1432 : cluster [DBG] pgmap v2343: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:08.266 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight 2021-02-10T04:57:08.307 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:57:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:08 smithi164 conmon[29288]: cluster 2021-02-10T04:57:06.526113+0000 mgr.y (mgr.14140) 2021-02-10T04:57:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:08 smithi164 conmon[29288]: 1432 : cluster [DBG] pgmap v2343: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:08.504 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:08.554 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:09.894 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:57:09 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:57:10.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:10 smithi085 conmon[29319]: cluster 2021-02-10T04:57:08.526831+0000 mgr.y (mgr.14140) 1433 : cluster [DBG] pgmap v2344: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:57:10.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:10 smithi085 conmon[35961]: cluster 2021-02-10T04:57:08.526831 2021-02-10T04:57:10.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:10 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1433 : cluster [DBG] pgmap v2344: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:57:10.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:10 smithi164 conmon[29288]: cluster 2021-02-10T04:57:08.526831+0000 mgr.y (mgr.14140) 1433 2021-02-10T04:57:10.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:10 smithi164 conmon[29288]: : cluster [DBG] pgmap v2344: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:57:11.069 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.0 to 1 (10000) 2021-02-10T04:57:11.365 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:11 smithi085 conmon[29319]: audit 2021-02-10T04:57:09.416035+0000 2021-02-10T04:57:11.365 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:11 smithi085 conmon[29319]: mgr.y (mgr.14140) 1434 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:57:11.366 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:11 smithi085 conmon[29319]: audit 2021-02-10T04:57:10.123913+0000 mon.c (mon.1) 137 : audit [INF] from='client.? 172.21.15.85:0/2906418592' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T04:57:11.366 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:11 smithi085 conmon[29319]: 2021-02-10T04:57:11.366 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:11 smithi085 conmon[29319]: audit 2021-02-10T04:57 2021-02-10T04:57:11.366 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:11 smithi085 conmon[29319]: :10.124249+0000 mon.a (mon.0) 2793 : audit [INF] 2021-02-10T04:57:11.366 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:11 smithi085 conmon[29319]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T04:57:11.366 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:11 smithi085 conmon[35961]: audit 2021-02-10T04:57:09.416035 2021-02-10T04:57:11.366 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:11 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1434 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:57:11.366 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:11 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:57:11.366 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:11 smithi085 conmon[35961]: T04:57:10.123913+0000 mon.c (mon. 2021-02-10T04:57:11.367 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:11 smithi085 conmon[35961]: 1) 137 : audit [INF] from='client.? 172.21.15.85:0/2906418592' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T04:57:11.367 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:11 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:57:11.367 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:11 smithi085 conmon[35961]: -10T04:57:10.124249+0000 mon.a (mon 2021-02-10T04:57:11.367 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:11 smithi085 conmon[35961]: .0) 2793 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T04:57:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:11 smithi164 conmon[29288]: audit 2021-02-10T04:57:09. 2021-02-10T04:57:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:11 smithi164 conmon[29288]: 416035+0000 mgr.y (mgr.14140) 1434 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:57:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:11 smithi164 conmon[29288]: audit 2021-02-10T04:57:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:11 smithi164 conmon[29288]: 2021-02-10T04:57:10.123913 2021-02-10T04:57:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:11 smithi164 conmon[29288]: +0000 mon.c (mon.1) 137 : 2021-02-10T04:57:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:11 smithi164 conmon[29288]: audit [INF] from='client.? 172.21.15.85:0/2906418592' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T04:57:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:11 smithi164 conmon[29288]: audit 2021 2021-02-10T04:57:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:11 smithi164 conmon[29288]: -02-10T04:57:10.124249+0000 mon.a 2021-02-10T04:57:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:11 smithi164 conmon[29288]: (mon.0) 2793 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T04:57:11.815 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:57:11.816 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:57:12.098 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:12 smithi085 conmon[29319]: cluster 2021-02-10T04:57:10.527326+0000 mgr.y (mgr 2021-02-10T04:57:12.099 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:12 smithi085 conmon[29319]: .14140) 1435 : cluster [DBG] pgmap v2345: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:57:12.099 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:12 smithi085 conmon[29319]: audit 2021-02-10T04:57:11.064454+0000 mon.a (mon.0) 2794 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2021-02-10T04:57:12.099 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:12 smithi085 conmon[29319]: cluster 2021-02-10T04:57:11.064609+0000 mon.a (mon.0) 2795 : cluster [DBG] osdmap e1290: 8 total, 8 up, 8 in 2021-02-10T04:57:12.099 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:12 smithi085 conmon[29319]: audit 2021-02-10T04:57:11.705555+0000 mon.c (mon.1) 138 : audit [INF] from='client.? 172.21.15.85:0/341792265' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:57:12.099 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:12 smithi085 conmon[29319]: audit 2021-02-10T04:57:11.705851+0000 mon.a (mon.0) 2796 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:57:12.100 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:12 smithi085 conmon[35961]: cluster 2021-02-10T04:57:10.527326+0000 mgr.y (mgr.14140) 2021-02-10T04:57:12.100 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:12 smithi085 conmon[35961]: 1435 : cluster [DBG] pgmap v2345: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:57:12.100 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:12 smithi085 conmon[35961]: audit 2021-02-10T04:57:11.064454+0000 mon.a 2021-02-10T04:57:12.100 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:12 smithi085 conmon[35961]: (mon.0) 2794 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2021-02-10T04:57:12.100 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:12 smithi085 conmon[35961]: cluster 2021-02-10T04:57:11. 2021-02-10T04:57:12.101 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:12 smithi085 conmon[35961]: 064609+0000 mon.a (mon.0) 2795 : cluster [DBG] 2021-02-10T04:57:12.101 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:12 smithi085 conmon[35961]: osdmap e1290: 8 total, 8 up, 8 in 2021-02-10T04:57:12.101 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:12 smithi085 conmon[35961]: audit 2021-02-10T04:57 2021-02-10T04:57:12.101 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:12 smithi085 conmon[35961]: :11.705555+0000 mon.c (mon.1) 2021-02-10T04:57:12.101 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:12 smithi085 conmon[35961]: 138 : audit [INF] from='client.? 172.21.15.85:0/341792265' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:57:12.101 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:12 smithi085 conmon[35961]: audit 2021-02-10T04:57:12.102 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:12 smithi085 conmon[35961]: 2021-02-10T04:57: 2021-02-10T04:57:12.102 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:12 smithi085 conmon[35961]: 11.705851+0000 mon.a (mon.0) 2796 2021-02-10T04:57:12.102 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:12 smithi085 conmon[35961]: : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:57:12.379 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 1 1 2021-02-10T04:57:12.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:12 smithi164 conmon[29288]: cluster 2021-02-10T04:57:10.527326+0000 mgr.y ( 2021-02-10T04:57:12.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:12 smithi164 conmon[29288]: mgr.14140) 1435 : cluster [DBG] pgmap v2345: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:57:12.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:12 smithi164 conmon[29288]: audit 2021-02-10T04:57:11.064454+0000 mon.a (mon.0) 2794 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2021-02-10T04:57:12.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:12 smithi164 conmon[29288]: cluster 2021-02-10T04:57:11.064609 2021-02-10T04:57:12.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:12 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2795 : cluster [DBG] osdmap e1290: 8 total, 8 up, 8 in 2021-02-10T04:57:12.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:12 smithi164 conmon[29288]: 2021-02-10T04:57:12.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:12 smithi164 conmon[29288]: audit 2021-02-10T04:57:11 2021-02-10T04:57:12.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:12 smithi164 conmon[29288]: .705555+0000 mon.c (mon.1) 138 : audit 2021-02-10T04:57:12.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:12 smithi164 conmon[29288]: [INF] from='client.? 172.21.15.85:0/341792265' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:57:12.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:12 smithi164 conmon[29288]: audit 2021-02-10T04: 2021-02-10T04:57:12.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:12 smithi164 conmon[29288]: 57:11.705851+0000 mon.a (mon.0) 2021-02-10T04:57:12.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:12 smithi164 conmon[29288]: 2796 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:57:12.397 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is set 2021-02-10T04:57:12.597 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:13.389 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:13 smithi085 conmon[35961]: cluster 2021-02-10T04:57:12.062313 2021-02-10T04:57:13.389 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:13 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2797 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:57:13.389 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:13 smithi085 conmon[35961]: audit 2021-02-10T04:57:12.072151+0000 mon.a (mon.0) 2798 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:57:13.389 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:13 smithi085 conmon[35961]: cluster 2021-02-10T04:57:12.072263+0000 mon.a (mon.0) 2799 : cluster [DBG] osdmap e1291: 8 total, 8 up, 8 in 2021-02-10T04:57:13.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:13 smithi085 conmon[29319]: cluster 2021-02-10T04:57:12.062313+0000 mon.a (mon.0) 2797 2021-02-10T04:57:13.389 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:13 smithi085 conmon[29319]: : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:57:13.390 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:13 smithi085 conmon[29319]: audit 2021-02-10T04:57:12.072151+0000 mon.a (mon.0) 2798 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:57:13.390 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:13 smithi085 conmon[29319]: cluster 2021-02-10T04:57:12.072263+0000 mon.a (mon.0) 2799 : cluster [DBG] osdmap e1291: 8 total, 8 up, 8 in 2021-02-10T04:57:13.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:13 smithi164 conmon[29288]: cluster 2021-02-10T04:57:12.062313+0000 mon.a (mon.0) 2797 : cluster 2021-02-10T04:57:13.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:13 smithi164 conmon[29288]: [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:57:13.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:13 smithi164 conmon[29288]: audit 2021-02-10T04:57:12.072151+0000 mon.a (mon.0) 2798 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:57:13.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:13 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:57:13.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:13 smithi164 conmon[29288]: 10T04:57:12.072263+0000 mon.a (mon.0) 2799 : cluster [DBG] osdmap e1291: 8 total, 8 up, 8 in 2021-02-10T04:57:13.415 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.1 dump_blocked_ops 2021-02-10T04:57:13.644 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:57:13.667 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:57:13.691 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:57:13.712 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:57:13.734 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:57:13.758 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:57:13.763 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:13.778 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:57:13.797 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:57:13.797 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:57:13.797 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:57:13.798 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:57:13.798 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:57:13.798 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:57:13.798 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:57:13.798 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:57:13.798 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:57:14.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:14 smithi164 conmon[29288]: cluster 2021-02-10T04:57:12.524095+0000 osd.5 (osd 2021-02-10T04:57:14.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:14 smithi164 conmon[29288]: .5) 194 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:57:14.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:14 smithi164 conmon[29288]: cluster 2021-02-10T04:57:12.526819+0000 osd.5 (osd.5) 195 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:57:14.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:14 smithi164 conmon[29288]: cluster 2021-02-10T04:57:12.527834+0000 mgr.y (mgr.14140 2021-02-10T04:57:14.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:14 smithi164 conmon[29288]: ) 1436 : cluster [DBG] pgmap v2348: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:57:14.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:14 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:57:14.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:14 smithi164 conmon[29288]: -10T04:57:13.075276+0000 mon.a (mon 2021-02-10T04:57:14.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:14 smithi164 conmon[29288]: .0) 2800 : cluster [DBG] osdmap e1292: 8 total, 8 up, 8 in 2021-02-10T04:57:14.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:14 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:57:14.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:14 smithi164 conmon[29288]: T04:57:13.590948+0000 mon.a (mon. 2021-02-10T04:57:14.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:14 smithi164 conmon[29288]: 0) 2801 : cluster [DBG] osdmap e1293: 8 total, 8 up, 8 in 2021-02-10T04:57:14.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:14 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:57:14.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:14 smithi164 conmon[29288]: -10T04:57:13.621042+0000 mon.c ( 2021-02-10T04:57:14.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:14 smithi164 conmon[29288]: mon.1) 139 : audit [DBG] from='client.? 172.21.15.85:0/2727129082' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:57:14.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:14 smithi085 conmon[29319]: cluster 2021-02-10T04:57:12.524095+0000 osd.5 (osd.5) 194 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:57:14.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:14 smithi085 conmon[29319]: cluster 2021-02-10T04:57:12.526819+0000 osd.5 (osd.5) 195 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:57:14.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:14 smithi085 conmon[29319]: cluster 2021-02-10T04:57:12.527834+0000 mgr.y (mgr.14140) 1436 : cluster [DBG] pgmap v2348: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:57:14.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:14 smithi085 conmon[29319]: cluster 2021-02-10T04:57:13.075276+0000 mon.a (mon.0) 2800 : cluster [DBG] osdmap e1292: 8 total, 8 up, 8 in 2021-02-10T04:57:14.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:14 smithi085 conmon[29319]: cluster 2021-02-10T04:57:14.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:14 smithi085 conmon[29319]: 2021-02-10T04:57:13.590948+0000 mon.a (mon.0) 2801 : cluster [DBG] osdmap e1293: 8 total, 8 up, 8 in 2021-02-10T04:57:14.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:14 smithi085 conmon[29319]: audit 2021-02-10T04:57:13.621042+0000 mon.c (mon.1) 139 : audit [DBG] from='client.? 172.21.15.85:0/2727129082' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:57:14.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:14 smithi085 conmon[35961]: cluster 2021-02-10T04:57:12.524095+0000 2021-02-10T04:57:14.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:14 smithi085 conmon[35961]: osd.5 (osd.5) 194 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:57:14.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:14 smithi085 conmon[35961]: cluster 2021-02-10T04:57:12.526819+0000 2021-02-10T04:57:14.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:14 smithi085 conmon[35961]: osd.5 (osd.5) 195 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:57:14.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:14 smithi085 conmon[35961]: cluster 2021-02-10T04:57:12.527834+0000 mgr.y ( 2021-02-10T04:57:14.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:14 smithi085 conmon[35961]: mgr.14140) 1436 : cluster [DBG] pgmap v2348: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:57:14.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:14 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:57:14.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:14 smithi085 conmon[35961]: 02-10T04:57:13.075276+0000 mon.a (mon.0) 2800 : cluster [DBG] osdmap e1292: 8 total, 8 up, 8 in 2021-02-10T04:57:14.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:14 smithi085 conmon[35961]: cluster 2021-02-10T04:57:13. 2021-02-10T04:57:14.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:14 smithi085 conmon[35961]: 590948+0000 mon.a (mon.0) 2801 : cluster [DBG] osdmap e1293: 8 total, 8 up, 8 in 2021-02-10T04:57:14.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:14 smithi085 conmon[35961]: audit 2021-02-10T04:57 2021-02-10T04:57:14.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:14 smithi085 conmon[35961]: :13.621042+0000 mon.c (mon.1) 139 : audit [DBG] from='client.? 172.21.15.85:0/2727129082' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:57:15.439 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset noscrub 2021-02-10T04:57:15.667 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:16.294 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:16 smithi085 conmon[29319]: cluster 2021-02-10T04:57:14.455442+0000 osd.0 (osd. 2021-02-10T04:57:16.294 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:16 smithi085 conmon[29319]: 0) 6 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:57:16.294 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:16 smithi085 conmon[29319]: cluster 2021-02-10T04:57:14.458362+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:57:16.294 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:16 smithi085 conmon[29319]: cluster 2021-02-10T04:57:14.528308+0000 mgr.y (mgr.14140) 1437 : cluster [DBG] pgmap v2351: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:57:16.294 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:16 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:57:16.294 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:16 smithi085 conmon[29319]: :57:15. 2021-02-10T04:57:16.295 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:16 smithi085 conmon[29319]: 082181+0000 mon.a (mon.0) 2802 : cluster [DBG] osdmap e1294: 8 total, 8 up, 8 in 2021-02-10T04:57:16.295 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:16 smithi085 conmon[35961]: cluster 2021-02-10T04:57 2021-02-10T04:57:16.295 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:16 smithi085 conmon[35961]: :14.455442+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:57:16.295 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:16 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:57:16.295 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:16 smithi085 conmon[35961]: -10T04:57:14.458362+0000 osd.0 ( 2021-02-10T04:57:16.295 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:16 smithi085 conmon[35961]: osd.0) 7 : cluster [DBG] 2021-02-10T04:57:16.295 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:16 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:57:16.295 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:16 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:57:16.295 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:16 smithi085 conmon[35961]: 02-10T04:57:14.528308+0000 mgr.y (mgr. 2021-02-10T04:57:16.295 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:16 smithi085 conmon[35961]: 14140) 1437 : cluster [DBG] pgmap v2351: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:57:16.295 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:16 smithi085 conmon[35961]: cluster 2021-02-10T04:57:16.296 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:16 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:57:16.296 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:16 smithi085 conmon[35961]: 10T04:57:15.082181+0000 2021-02-10T04:57:16.296 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:16 smithi085 conmon[35961]: mon.a (mon.0) 2802 : cluster 2021-02-10T04:57:16.296 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:16 smithi085 conmon[35961]: [DBG] osdmap e1294: 8 total, 8 up, 8 in 2021-02-10T04:57:16.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:16 smithi164 conmon[29288]: cluster 2021-02-10T04:57:16.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:16 smithi164 conmon[29288]: 2021-02-10T04:57:16.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:16 smithi164 conmon[29288]: 2021 2021-02-10T04:57:16.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:16 smithi164 conmon[29288]: - 2021-02-10T04:57:16.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:16 smithi164 conmon[29288]: 02 2021-02-10T04:57:16.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:16 smithi164 conmon[29288]: - 2021-02-10T04:57:16.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:16 smithi164 conmon[29288]: 10T04:57:14.455442+0000 2021-02-10T04:57:16.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:16 smithi164 conmon[29288]: osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:57:16.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:16 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:57:16.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:16 smithi164 conmon[29288]: 02-10T04:57:14.458362+0000 osd.0 (osd.0) 2021-02-10T04:57:16.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:16 smithi164 conmon[29288]: 7 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:57:16.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:16 smithi164 conmon[29288]: cluster 2021-02-10T04:57:16.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:16 smithi164 conmon[29288]: 2021-02-10T04:57 2021-02-10T04:57:16.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:16 smithi164 conmon[29288]: :14.528308+0000 mgr.y (mgr 2021-02-10T04:57:16.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:16 smithi164 conmon[29288]: .14140) 1437 : cluster [DBG] 2021-02-10T04:57:16.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:16 smithi164 conmon[29288]: pgmap v2351: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:57:16.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:16 smithi164 conmon[29288]: cluster 2021-02-10T04:57:16.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:16 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:57:16.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:16 smithi164 conmon[29288]: 57:15.082181+0000 mon.a 2021-02-10T04:57:16.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:16 smithi164 conmon[29288]: (mon.0) 2802 : cluster 2021-02-10T04:57:16.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:16 smithi164 conmon[29288]: [DBG] osdmap e1294: 8 total, 8 up, 8 in 2021-02-10T04:57:16.814 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:57:16.815 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [], 2021-02-10T04:57:16.815 INFO:teuthology.orchestra.run.smithi085.stdout: "complaint_time": 30, 2021-02-10T04:57:16.815 INFO:teuthology.orchestra.run.smithi085.stdout: "num_blocked_ops": 0 2021-02-10T04:57:16.815 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:57:16.986 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:57:17.099 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.1 to 1 (10000) 2021-02-10T04:57:17.118 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:17 smithi085 conmon[29319]: audit 2021-02-10T04:57:16.692486+0000 mon.a (mon.0) 2803 : audit [INF] from='client.? 172.21.15.85:0/1784522241' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-02-10T04:57:17.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:17 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:57:17.119 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:17 smithi085 conmon[35961]: 57:16.692486+0000 mon.a (mon.0) 2803 : audit [INF] from='client.? 172.21.15.85:0/1784522241' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-02-10T04:57:17.305 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:17.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:17 smithi164 conmon[29288]: audit 2021-02-10T04:57:16.692486+0000 mon.a (mon.0) 2803 : 2021-02-10T04:57:17.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:17 smithi164 conmon[29288]: audit [INF] from='client.? 172.21.15.85:0/1784522241' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-02-10T04:57:18.137 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.1 dump_historic_ops 2021-02-10T04:57:18.390 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:18 smithi164 conmon[29288]: cluster 2021-02-10T04:57:16.528858 2021-02-10T04:57:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:18 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1438 : cluster [DBG] pgmap v2353: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:57:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:18 smithi164 conmon[29288]: audit 2021-02-10T04:57:17.095445+0000 2021-02-10T04:57:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:18 smithi164 conmon[29288]: mon.a (mon.0) 2804 : audit [INF] from='client.? 172.21.15.85:0/1784522241' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-02-10T04:57:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:18 smithi164 conmon[29288]: cluster 2021-02-10T04:57: 2021-02-10T04:57:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:18 smithi164 conmon[29288]: 17.095512+0000 mon.a (mon.0) 2805 : cluster [DBG] osdmap e1295: 8 total, 8 up, 8 in 2021-02-10T04:57:18.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:18 smithi164 conmon[29288]: 2021-02-10T04:57:18.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:18 smithi085 conmon[35961]: cluster 2021-02-10T04:57:16.528858+0000 mgr.y 2021-02-10T04:57:18.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:18 smithi085 conmon[35961]: (mgr.14140) 1438 : cluster [DBG] pgmap v2353: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:57:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:18 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:57:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:18 smithi085 conmon[35961]: 02-10T04:57:17.095445+0000 mon.a (mon.0) 2804 : audit 2021-02-10T04:57:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:18 smithi085 conmon[35961]: [INF] from='client.? 172.21.15.85:0/1784522241' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-02-10T04:57:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:18 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:57:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:18 smithi085 conmon[35961]: 57:17.095512+0000 mon.a (mon.0) 2805 : cluster 2021-02-10T04:57:18.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:18 smithi085 conmon[35961]: [DBG] osdmap e1295: 8 total, 8 up, 8 in 2021-02-10T04:57:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:18 smithi085 conmon[29319]: cluster 2021-02-10T04:57:16.528858+0000 mgr.y (mgr.14140) 1438 : 2021-02-10T04:57:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:18 smithi085 conmon[29319]: cluster [DBG] pgmap v2353: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-02-10T04:57:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:18 smithi085 conmon[29319]: audit 2021-02-10T04:57:17.095445+0000 mon.a (mon.0) 2804 : audit [INF] from='client.? 172.21.15.85:0/1784522241' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-02-10T04:57:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:18 smithi085 conmon[29319]: cluster 2021-02-10T04:57:17.095512+0000 mon.a (mon.0) 2805 : cluster [DBG] osdmap e1295: 8 total, 8 up, 8 in 2021-02-10T04:57:18.865 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 3 1 2021-02-10T04:57:19.092 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:19.894 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:57:19 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:57:20.101 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is unset 2021-02-10T04:57:20.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:20 smithi085 conmon[35961]: cluster 2021-02-10T04:57:20.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:20 smithi085 conmon[35961]: 2021-02-10T04:57:20.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:20 smithi085 conmon[35961]: 2021 2021-02-10T04:57:20.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:20 smithi085 conmon[35961]: -02-10T04:57:18 2021-02-10T04:57:20.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:20 smithi085 conmon[35961]: .529493+0000 mgr.y (mgr.14140) 1439 : cluster [DBG] pgmap v2355: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2021-02-10T04:57:20.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:20 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:57:20.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:20 smithi085 conmon[35961]: 02-10T04:57:19.957577+0000 mon.c ( 2021-02-10T04:57:20.374 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:20 smithi085 conmon[35961]: mon.1) 140 : audit [INF] 2021-02-10T04:57:20.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:20 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/2261476010' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:57:20.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:20 smithi085 conmon[35961]: audit 2021 2021-02-10T04:57:20.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:20 smithi085 conmon[35961]: -02-10T04:57:19.957887+0000 2021-02-10T04:57:20.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:20 smithi085 conmon[35961]: mon.a (mon.0) 2806 : audit 2021-02-10T04:57:20.375 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:20 smithi085 conmon[35961]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:57:20.375 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:20 smithi085 conmon[29319]: cluster 2021-02-10T04:57:18.529493+0000 mgr.y (mgr.14140) 1439 : cluster [DBG] pgmap v2355: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2021-02-10T04:57:20.376 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:20 smithi085 conmon[29319]: audit 2021-02-10T04:57:19.957577+0000 mon.c (mon.1) 140 : audit [INF] from='client.? 172.21.15.85:0/2261476010' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:57:20.376 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:20 smithi085 conmon[29319]: audit 2021-02-10T04:57:19.957887+0000 mon.a (mon.0) 2806 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:57:20.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:20 smithi164 conmon[29288]: cluster 2021-02-10T04:57:18.529493+0000 mgr.y (mgr 2021-02-10T04:57:20.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:20 smithi164 conmon[29288]: .14140) 1439 : cluster [DBG] pgmap v2355: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2021-02-10T04:57:20.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:20 smithi164 conmon[29288]: audit 2021-02-10T04:57:19.957577+0000 mon.c (mon.1) 140 : audit [INF] 2021-02-10T04:57:20.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:20 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/2261476010' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:57:20.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:20 smithi164 conmon[29288]: audit 2021-02-10T04: 2021-02-10T04:57:20.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:20 smithi164 conmon[29288]: 57:19.957887+0000 mon.a (mon.0) 2806 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:57:21.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:21 smithi085 conmon[29319]: audit 2021-02-10T04:57:19.425976+0000 mgr.y (mgr. 2021-02-10T04:57:21.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:21 smithi085 conmon[29319]: 14140) 1440 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:57:21.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:21 smithi085 conmon[29319]: cluster 2021-02-10T04:57:20.027524+0000 osd.2 (osd.2) 2035 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:57:21.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:21 smithi085 conmon[29319]: cluster 2021-02-10T04:57:20.029216+0000 osd.2 (osd.2) 2036 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:57:21.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:21 smithi085 conmon[29319]: cluster 2021-02-10T04:57:20.090550+0000 mon.a (mon.0) 2807 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:57:21.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:21 smithi085 conmon[29319]: audit 2021-02-10T04:57:20.095149+0000 mon.a (mon.0) 2808 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:57:21.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:21 smithi085 conmon[29319]: cluster 2021-02-10T04:57:20.095229+0000 mon.a (mon.0) 2809 : cluster [DBG] osdmap e1296: 8 total, 8 up, 8 in 2021-02-10T04:57:21.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:21 smithi085 conmon[35961]: audit 2021-02-10T04:57:19.425976 2021-02-10T04:57:21.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:21 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1440 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:57:21.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:21 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:57:21.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:21 smithi085 conmon[35961]: T04:57:20.027524+0000 osd.2 (osd.2) 2035 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:57:21.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:21 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:57:21.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:21 smithi085 conmon[35961]: -10T04:57:20.029216+0000 osd.2 ( 2021-02-10T04:57:21.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:21 smithi085 conmon[35961]: osd.2) 2036 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:57:21.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:21 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:57:21.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:21 smithi085 conmon[35961]: -10T04:57:20.090550+0000 mon.a ( 2021-02-10T04:57:21.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:21 smithi085 conmon[35961]: mon.0) 2807 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:57:21.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:21 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:57:21.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:21 smithi085 conmon[35961]: 10T04:57:20.095149+0000 mon.a (mon.0) 2808 2021-02-10T04:57:21.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:21 smithi085 conmon[35961]: : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:57:21.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:21 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:57:21.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:21 smithi085 conmon[35961]: -02-10T04:57:20.095229+0000 mon.a (mon.0) 2809 : cluster [DBG] osdmap e1296: 8 total, 8 up, 8 in 2021-02-10T04:57:21.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:21 smithi164 conmon[29288]: audit 2021-02-10T04:57:19.425976+0000 2021-02-10T04:57:21.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:21 smithi164 conmon[29288]: mgr.y (mgr.14140) 1440 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:57:21.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:21 smithi164 conmon[29288]: cluster 2021-02-10T04:57:20.027524 2021-02-10T04:57:21.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:21 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 2035 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:57:21.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:21 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:57:21.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:21 smithi164 conmon[29288]: -10T04:57:20.029216 2021-02-10T04:57:21.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:21 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 2036 2021-02-10T04:57:21.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:21 smithi164 conmon[29288]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:57:21.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:21 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:57:21.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:21 smithi164 conmon[29288]: 10T04:57:20.090550+0000 mon.a (mon.0) 2807 : cluster [WRN] 2021-02-10T04:57:21.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:21 smithi164 conmon[29288]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:57:21.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:21 smithi164 conmon[29288]: audit 2021-02-10T04:57 2021-02-10T04:57:21.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:21 smithi164 conmon[29288]: :20.095149+0000 mon.a (mon.0) 2808 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:57:21.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:21 smithi164 conmon[29288]: cluster 2021-02-10T04:57:20.095229 2021-02-10T04:57:21.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:21 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2809 : cluster [DBG] osdmap e1296: 8 total, 8 up, 8 in 2021-02-10T04:57:21.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:57:21 smithi085 conmon[49106]: debug 2021-02-10T04:57:21.390+0000 7f778ce21700 -1 osd.2 pg_epoch: 1296 pg[5.8( empty local-lis/les=1227/1228 n=0 ec=57/57 lis/c=1227/1227 les/c/f=1228/1228/0 sis=1227) [2,0,3] r=0 lpr=1227 crt=0'0 mlcod 0'0 active+clean] scrubber pg(5.8) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x564aa6f5fce0) 2021-02-10T04:57:21.691 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:57:21 smithi085 conmon[45443]: debug 2021-02-10T04:57:21.399+0000 7f71214ef700 -1 osd.1 pg_epoch: 1296 pg[5.1d( empty local-lis/les=1227/1228 n=0 ec=57/57 lis/c=1227/1227 les/c/f=1228/1228/0 sis=1227) [1,4,0] r=0 lpr=1227 crt=0'0 mlcod 0'0 active+clean] scrubber pg(5.1d) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x55dbce779ce0) 2021-02-10T04:57:21.965 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:57:21.984 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:57:22.014 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:57:22.037 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:57:22.049 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:57:22.049 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:57:22.060 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:57:22.080 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:57:22.100 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:57:22.133 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:57:22.134 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:57:22.134 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:57:22.134 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:57:22.134 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:57:22.134 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:57:22.134 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:57:22.134 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:57:22.134 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:57:22.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:22 smithi164 conmon[29288]: cluster 2021-02-10T04:57:20.529907+0000 mgr.y ( 2021-02-10T04:57:22.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:22 smithi164 conmon[29288]: mgr.14140) 1441 : cluster [DBG] pgmap v2357: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:22.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:22 smithi164 conmon[29288]: audit 2021-02-10T04:57:21.941533+0000 mon.a (mon.0) 2810 : audit [DBG] from='client.? 172.21.15.85:0/180905388' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:57:22.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:22 smithi085 conmon[29319]: cluster 2021-02-10T04:57:22.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:22 smithi085 conmon[29319]: 2021-02-10T04 2021-02-10T04:57:22.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:22 smithi085 conmon[29319]: :57:20.529907+0000 mgr.y (mgr.14140) 1441 : cluster [DBG] pgmap v2357: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:22 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:57:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:22 smithi085 conmon[29319]: 02-10T04:57:21.941533+0000 mon.a (mon 2021-02-10T04:57:22.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:22 smithi085 conmon[29319]: .0) 2810 : audit [DBG] from='client.? 172.21.15.85:0/180905388' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:57:22.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:22 smithi085 conmon[35961]: cluster 2021-02-10T04:57:20.529907+0000 mgr.y (mgr.14140) 1441 : cluster [DBG] pgmap v2357: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:22.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:22 smithi085 conmon[35961]: audit 2021-02-10T04:57:21.941533+0000 mon.a (mon.0) 2810 : audit [DBG] from='client.? 172.21.15.85:0/180905388' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:57:23.404 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.2 dump_ops_in_flight 2021-02-10T04:57:23.466 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset nodeep-scrub 2021-02-10T04:57:23.637 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:23.701 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:24 smithi164 conmon[29288]: cluster 2021-02-10T04:57:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:24 smithi164 conmon[29288]: 2021-02 2021-02-10T04:57:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:24 smithi164 conmon[29288]: -10T04:57:22.530455+0000 mgr.y (mgr.14140) 1442 : cluster [DBG] pgmap v2358: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 687 B/s rd, 0 op/s 2021-02-10T04:57:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:24 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:57:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:24 smithi164 conmon[29288]: 02-10T04:57:23.596628+0000 mon.a (mon.0) 2811 : cluster [DBG] osdmap e1297: 8 total, 8 up, 8 in 2021-02-10T04:57:24.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:24 smithi085 conmon[29319]: cluster 2021-02-10T04:57:22.530455+0000 mgr.y (mgr. 2021-02-10T04:57:24.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:24 smithi085 conmon[29319]: 14140) 1442 : cluster [DBG] pgmap v2358: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 687 B/s rd, 0 op/s 2021-02-10T04:57:24.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:24 smithi085 conmon[29319]: cluster 2021-02-10T04:57:23.596628+0000 mon.a (mon.0) 2811 : cluster [DBG] osdmap e1297: 8 total, 8 up, 8 in 2021-02-10T04:57:24.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:24 smithi085 conmon[35961]: cluster 2021-02-10T04:57:22.530455+0000 mgr.y (mgr.14140 2021-02-10T04:57:24.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:24 smithi085 conmon[35961]: ) 1442 : cluster [DBG] pgmap v2358: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 687 B/s rd, 0 op/s 2021-02-10T04:57:24.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:24 smithi085 conmon[35961]: cluster 2021-02-10T04:57:23.596628+0000 mon.a (mon.0) 2811 : cluster [DBG] osdmap e1297: 8 total, 8 up, 8 in 2021-02-10T04:57:25.113 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.3 to 1 (10000) 2021-02-10T04:57:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:25 smithi164 conmon[29288]: audit 2021-02-10T04:57:24.703066+0000 mon.a 2021-02-10T04:57:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:25 smithi164 conmon[29288]: (mon.0) 2812 : audit [INF] from='client.? 172.21.15.85:0/1546079801' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-02-10T04:57:25.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:25 smithi085 conmon[29319]: audit 2021-02-10T04:57:24.703066+0000 mon.a (mon.0) 2812 : audit [INF] from='client.? 172.21.15.85:0/1546079801' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-02-10T04:57:25.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:25 smithi085 conmon[35961]: audit 2021-02-10T04:57:25.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:25 smithi085 conmon[35961]: 2021-02-10T04:57:25.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:25 smithi085 conmon[35961]: 2021-02-10T04:57:24. 2021-02-10T04:57:25.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:25 smithi085 conmon[35961]: 703066+0000 mon.a (mon.0) 2812 : audit [INF] from='client.? 172.21.15.85:0/1546079801' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-02-10T04:57:26.079 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:57:26.298 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:26 smithi164 conmon[29288]: cluster 2021-02-10T04:57:24.530907+0000 mgr.y (mgr 2021-02-10T04:57:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:26 smithi164 conmon[29288]: .14140) 1443 : cluster [DBG] pgmap v2360: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:57:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:26 smithi164 conmon[29288]: audit 2021-02-10T04:57:25.109831+0000 mon.a (mon.0) 2813 : audit [INF] from='client.? 172.21.15.85:0/1546079801' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2021-02-10T04:57:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:26 smithi164 conmon[29288]: cluster 2021-02-10T04:57:25.109894+0000 mon.a (mon.0) 2814 : cluster [DBG] osdmap e1298: 8 total, 8 up, 8 in 2021-02-10T04:57:26.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:26 smithi085 conmon[29319]: cluster 2021-02-10T04:57:24.530907+0000 2021-02-10T04:57:26.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:26 smithi085 conmon[29319]: mgr.y (mgr.14140) 1443 : cluster [DBG] pgmap v2360: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:57:26.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:26 smithi085 conmon[29319]: audit 2021-02-10T04:57:25.109831+0000 mon.a (mon.0) 2813 : audit [INF] from='client.? 172.21.15.85:0/1546079801' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2021-02-10T04:57:26.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:26 smithi085 conmon[29319]: cluster 2021-02-10T04:57:25.109894+0000 mon.a (mon.0) 2814 : cluster [DBG] osdmap e1298: 8 total, 8 up, 8 in 2021-02-10T04:57:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:26 smithi085 conmon[35961]: cluster 2021-02-10T04:57:24.530907 2021-02-10T04:57:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:26 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1443 : cluster [DBG] pgmap v2360: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-02-10T04:57:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:26 smithi085 conmon[35961]: audit 2021-02-10T04:57:25.109831+0000 mon.a (mon 2021-02-10T04:57:26.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:26 smithi085 conmon[35961]: .0) 2813 : audit [INF] from='client.? 172.21.15.85:0/1546079801' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2021-02-10T04:57:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:26 smithi085 conmon[35961]: cluster 2021-02-10T04:57: 2021-02-10T04:57:26.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:26 smithi085 conmon[35961]: 25.109894+0000 mon.a (mon.0) 2814 : cluster [DBG] osdmap e1298: 8 total, 8 up, 8 in 2021-02-10T04:57:26.644 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 4 1 2021-02-10T04:57:26.898 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:28.029 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:57:28.029 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:57:28.129 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is unset 2021-02-10T04:57:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:28 smithi164 conmon[29288]: cluster 2021-02-10T04:57:26.531478+0000 mgr.y (mgr.14140 2021-02-10T04:57:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:28 smithi164 conmon[29288]: ) 1444 : cluster [DBG] pgmap v2362: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 795 B/s rd, 0 op/s 2021-02-10T04:57:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:28 smithi164 conmon[29288]: audit 2021-02-10T04:57:28.021378+0000 mon.c (mon.1) 141 : audit [INF] from='client.? 172.21.15.85:0/3632504425' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:57:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:28 smithi164 conmon[29288]: audit 2021-02-10T04:57:28.021629+0000 mon.a (mon.0) 2815 : audit 2021-02-10T04:57:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:28 smithi164 conmon[29288]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:57:28.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:28 smithi085 conmon[35961]: cluster 2021-02-10T04:57:26.531478+0000 mgr.y (mgr.14140) 1444 : cluster [DBG] pgmap v2362: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 795 B/s rd, 0 op/s 2021-02-10T04:57:28.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:28 smithi085 conmon[35961]: audit 2021-02-10T04:57:28.021378+0000 mon.c (mon.1) 141 : audit [INF] 2021-02-10T04:57:28.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:28 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/3632504425' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:57:28.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:28 smithi085 conmon[35961]: audit 2021-02-10T04:57: 2021-02-10T04:57:28.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:28 smithi085 conmon[35961]: 28.021629+0000 mon.a (mon.0) 2815 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:57:28.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:28 smithi085 conmon[29319]: cluster 2021-02-10T04:57:26.531478+0000 mgr.y (mgr.14140) 1444 : cluster [DBG] pgmap v2362: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 795 B/s rd, 0 op/s 2021-02-10T04:57:28.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:28 smithi085 conmon[29319]: audit 2021-02-10T04:57:28.021378+0000 mon.c (mon.1) 141 : audit [INF] from='client.? 172.21.15.85:0/3632504425' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:57:28.440 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:28 smithi085 conmon[29319]: audit 2021-02-10T04:57:28.021629+0000 mon.a (mon.0) 2815 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:57:28.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:57:28 smithi085 conmon[49106]: debug 2021-02-10T04:57:28.438+0000 7f778ce21700 -1 osd.2 pg_epoch: 1299 pg[5.8( empty local-lis/les=1227/1228 n=0 ec=57/57 lis/c=1227/1227 les/c/f=1228/1228/0 sis=1227) [2,0,3] r=0 lpr=1227 crt=0'0 mlcod 0'0 active+clean] scrubber pg(5.8) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x564aa5de9340) 2021-02-10T04:57:29.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:29 smithi164 conmon[29288]: cluster 2021-02-10T04:57:29.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:29 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:57:29.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:29 smithi164 conmon[29288]: 57:28.116387+0000 mon.a (mon.0) 2816 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:57:29.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:29 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:57:29.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:29 smithi164 conmon[29288]: 10T04:57:28.116413+0000 mon.a (mon.0) 2817 : cluster [INF] Cluster is now healthy 2021-02-10T04:57:29.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:29 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:57:29.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:29 smithi164 conmon[29288]: 04:57:28.119763+0000 mon.a 2021-02-10T04:57:29.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:29 smithi164 conmon[29288]: (mon.0) 2818 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:57:29.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:29 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:57:29.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:29 smithi164 conmon[29288]: 02-10T04:57:28.119815+0000 2021-02-10T04:57:29.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:29 smithi164 conmon[29288]: mon.a (mon.0) 2819 : cluster [DBG] 2021-02-10T04:57:29.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:29 smithi164 conmon[29288]: osdmap e1299: 8 total, 8 up, 8 in 2021-02-10T04:57:29.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:29 smithi085 conmon[29319]: cluster 2021-02-10T04:57:28.116387 2021-02-10T04:57:29.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:29 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2816 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:57:29.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:29 smithi085 conmon[29319]: cluster 2021-02-10T04:57:28.116413+0000 mon.a (mon.0) 2817 : cluster [INF] Cluster is now healthy 2021-02-10T04:57:29.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:29 smithi085 conmon[29319]: audit 2021-02-10T04:57:28.119763+0000 mon.a (mon.0) 2818 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:57:29.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:29 smithi085 conmon[29319]: cluster 2021-02-10T04:57:28.119815+0000 mon.a ( 2021-02-10T04:57:29.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:29 smithi085 conmon[29319]: mon.0) 2819 : cluster [DBG] osdmap e1299: 8 total, 8 up, 8 in 2021-02-10T04:57:29.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:29 smithi085 conmon[35961]: cluster 2021-02-10T04:57:28.116387+0000 2021-02-10T04:57:29.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:29 smithi085 conmon[35961]: mon.a (mon.0) 2816 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:57:29.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:29 smithi085 conmon[35961]: cluster 2021-02-10T04:57:29.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:29 smithi085 conmon[35961]: 2021-02-10T04:57:28.116413+0000 mon.a (mon 2021-02-10T04:57:29.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:29 smithi085 conmon[35961]: .0) 2817 : cluster [INF] Cluster is now healthy 2021-02-10T04:57:29.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:29 smithi085 conmon[35961]: audit 2021-02-10T04:57:29.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:29 smithi085 conmon[35961]: 2021-02-10T04:57:28.119763+0000 mon.a (mon 2021-02-10T04:57:29.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:29 smithi085 conmon[35961]: .0) 2818 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:57:29.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:29 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:57:29.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:29 smithi085 conmon[35961]: -02-10T04:57:28.119815+0000 2021-02-10T04:57:29.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:29 smithi085 conmon[35961]: mon.a (mon.0) 2819 : cluster [DBG] 2021-02-10T04:57:29.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:29 smithi085 conmon[35961]: osdmap e1299: 8 total, 8 up, 8 in 2021-02-10T04:57:29.894 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:57:29 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:57:30.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:30 smithi085 conmon[35961]: cluster 2021-02-10T04:57:28.531805+0000 mgr.y (mgr. 2021-02-10T04:57:30.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:30 smithi085 conmon[35961]: 14140) 1445 : cluster [DBG] pgmap v2364: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2021-02-10T04:57:30.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:30 smithi085 conmon[29319]: cluster 2021-02-10T04:57:28.531805+0000 mgr.y (mgr.14140) 2021-02-10T04:57:30.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:30 smithi085 conmon[29319]: 1445 : cluster [DBG] pgmap v2364: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2021-02-10T04:57:30.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:57:29 smithi164 conmon[37263]: debug 2021-02-10T04:57:30.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:57:29 smithi164 conmon[37263]: 2021-02-10T04:57:29.997+0000 7f1c679e3700 -1 osd.5 pg_epoch: 1299 pg[6.a( empty local-lis/les=1227/1228 n=0 ec=59/59 lis/c=1227/1227 les/c/f=1228/1228/0 sis=1227) [5,6,0] r=0 lpr=1227 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x5566505eab00) 2021-02-10T04:57:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:30 smithi164 conmon[29288]: cluster 2021-02-10T04:57:28. 2021-02-10T04:57:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:30 smithi164 conmon[29288]: 531805+0000 mgr.y (mgr.14140) 1445 : cluster [DBG] pgmap v2364: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2021-02-10T04:57:30.427 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.2 dump_blocked_ops 2021-02-10T04:57:30.680 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:31.214 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:57:31.235 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:57:31.252 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:57:31.261 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:31 smithi085 conmon[29319]: audit 2021-02-10T04:57:29.436030+0000 mgr.y (mgr.14140) 1446 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:57:31.261 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:31 smithi085 conmon[29319]: cluster 2021-02-10T04:57:30.950555+0000 osd.6 (osd.6) 190 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:57:31.261 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:31 smithi085 conmon[29319]: cluster 2021-02-10T04:57:30.953565+0000 osd.6 (osd.6) 191 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:57:31.261 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:31 smithi085 conmon[35961]: audit 2021-02-10T04:57:29.436030+0000 2021-02-10T04:57:31.261 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:31 smithi085 conmon[35961]: mgr.y (mgr.14140) 1446 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:57:31.262 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:31 smithi085 conmon[35961]: cluster 2021-02-10T04:57:31.262 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:31 smithi085 conmon[35961]: 2021-02-10T04:57: 2021-02-10T04:57:31.262 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:31 smithi085 conmon[35961]: 30.950555+0000 osd.6 (osd.6) 190 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:57:31.262 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:31 smithi085 conmon[35961]: cluster 2021-02-10T04:57:31.262 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:31 smithi085 conmon[35961]: 2021-02-10T04:57:31.262 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:31 smithi085 conmon[35961]: 2021 2021-02-10T04:57:31.262 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:31 smithi085 conmon[35961]: - 2021-02-10T04:57:31.262 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:31 smithi085 conmon[35961]: 02 2021-02-10T04:57:31.262 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:31 smithi085 conmon[35961]: - 2021-02-10T04:57:31.262 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:31 smithi085 conmon[35961]: 10 2021-02-10T04:57:31.262 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:31 smithi085 conmon[35961]: T 2021-02-10T04:57:31.262 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:31 smithi085 conmon[35961]: 04 2021-02-10T04:57:31.263 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:31 smithi085 conmon[35961]: : 2021-02-10T04:57:31.263 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:31 smithi085 conmon[35961]: 57 2021-02-10T04:57:31.263 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:31 smithi085 conmon[35961]: : 2021-02-10T04:57:31.263 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:31 smithi085 conmon[35961]: 30 2021-02-10T04:57:31.263 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:31 smithi085 conmon[35961]: . 2021-02-10T04:57:31.263 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:31 smithi085 conmon[35961]: 953565 2021-02-10T04:57:31.263 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:31 smithi085 conmon[35961]: +0000 2021-02-10T04:57:31.263 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:31 smithi085 conmon[35961]: 2021-02-10T04:57:31.263 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:31 smithi085 conmon[35961]: osd.6 2021-02-10T04:57:31.263 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:31 smithi085 conmon[35961]: ( 2021-02-10T04:57:31.263 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:31 smithi085 conmon[35961]: osd 2021-02-10T04:57:31.263 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:31 smithi085 conmon[35961]: . 2021-02-10T04:57:31.263 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:31 smithi085 conmon[35961]: 6) 191 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:57:31.278 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:57:31.325 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:57:31.345 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:57:31.364 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:57:31.386 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:57:31.387 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:57:31.387 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:57:31.387 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:57:31.387 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:57:31.387 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:57:31.387 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:57:31.388 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:57:31.388 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:57:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:31 smithi164 conmon[29288]: audit 2021-02-10T04:57:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:31 smithi164 conmon[29288]: 2021-02-10T04:57:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:31 smithi164 conmon[29288]: 2021 2021-02-10T04:57:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:31 smithi164 conmon[29288]: - 2021-02-10T04:57:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:31 smithi164 conmon[29288]: 02-10T04:57:29. 2021-02-10T04:57:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:31 smithi164 conmon[29288]: 436030+0000 mgr.y (mgr.14140) 1446 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:57:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:31 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:57:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:31 smithi164 conmon[29288]: 02-10T04:57:30.950555+0000 osd.6 2021-02-10T04:57:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:31 smithi164 conmon[29288]: (osd.6) 190 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:57:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:31 smithi164 conmon[29288]: cluster 2021-02-10T04:57:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:31 smithi164 conmon[29288]: 2021-02-10T04:57:30.953565+0000 2021-02-10T04:57:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:31 smithi164 conmon[29288]: osd.6 (osd.6) 191 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:57:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:31 smithi164 conmon[29288]: 2021-02-10T04:57:32.147 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.4 to 1 (10000) 2021-02-10T04:57:32.269 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set noscrub 2021-02-10T04:57:32.294 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:32 smithi085 conmon[29319]: cluster 2021-02-10T04:57:30.532345+0000 mgr.y ( 2021-02-10T04:57:32.294 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:32 smithi085 conmon[29319]: mgr.14140) 1447 : cluster [DBG] pgmap v2365: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2021-02-10T04:57:32.294 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:32 smithi085 conmon[29319]: cluster 2021-02-10T04:57:31.140685+0000 mon.a (mon.0) 2820 : cluster [DBG] osdmap e1300: 8 total, 8 up, 8 in 2021-02-10T04:57:32.295 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:32 smithi085 conmon[29319]: audit 2021-02-10T04:57:31.183040+0000 mon.c (mon.1) 142 : audit [DBG] from='client.? 172.21.15.85:0/2166458311' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:57:32.295 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:32 smithi085 conmon[29319]: audit 2021-02-10T04:57:31.469394+0000 mon.c (mon.1) 143 : audit [INF] from='client.? 172.21.15.85:0/1701783337' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-02-10T04:57:32.295 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:32 smithi085 conmon[29319]: audit 2021-02-10T04:57:31.469795+0000 mon.a (mon.0) 2821 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-02-10T04:57:32.295 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:32 smithi085 conmon[35961]: cluster 2021-02-10T04:57:30.532345+0000 mgr.y (mgr 2021-02-10T04:57:32.295 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:32 smithi085 conmon[35961]: .14140) 1447 : cluster [DBG] pgmap v2365: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2021-02-10T04:57:32.295 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:32 smithi085 conmon[35961]: cluster 2021-02-10T04:57:31.140685+0000 mon.a (mon.0) 2820 : cluster [DBG] osdmap e1300: 8 total, 8 up, 8 in 2021-02-10T04:57:32.295 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:32 smithi085 conmon[35961]: audit 2021-02-10T04:57:31.183040+0000 mon.c ( 2021-02-10T04:57:32.295 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:32 smithi085 conmon[35961]: mon.1) 142 : audit [DBG] from='client.? 172.21.15.85:0/2166458311' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:57:32.295 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:32 smithi085 conmon[35961]: audit 2021-02-10T04:57: 2021-02-10T04:57:32.295 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:32 smithi085 conmon[35961]: 31.469394+0000 mon.c (mon.1) 143 : audit [INF] 2021-02-10T04:57:32.296 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:32 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/1701783337' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-02-10T04:57:32.296 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:32 smithi085 conmon[35961]: audit 2021-02-10T04:57:32.296 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:32 smithi085 conmon[35961]: 2021-02-10T04:57:31.469795+0000 mon.a 2021-02-10T04:57:32.296 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:32 smithi085 conmon[35961]: (mon.0) 2821 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-02-10T04:57:32.492 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:32 smithi164 conmon[29288]: cluster 2021-02-10T04:57:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:32 smithi164 conmon[29288]: 2021-02-10T04:57:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:32 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:57:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:32 smithi164 conmon[29288]: :57:30.532345+0000 mgr.y (mgr.14140) 1447 : cluster [DBG] pgmap v2365: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2021-02-10T04:57:32.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:32 smithi164 conmon[29288]: cluster 2021-02-10T04:57:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:32 smithi164 conmon[29288]: 2021-02-10T04:57:31.140685 2021-02-10T04:57:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:32 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2820 : cluster [DBG] 2021-02-10T04:57:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:32 smithi164 conmon[29288]: osdmap e1300: 8 total, 8 up, 8 in 2021-02-10T04:57:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:32 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:57:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:32 smithi164 conmon[29288]: 02-10T04:57:31.183040+0000 mon.c 2021-02-10T04:57:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:32 smithi164 conmon[29288]: (mon.1) 142 : audit [DBG] from='client.? 172.21.15.85:0/2166458311' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:57:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:32 smithi164 conmon[29288]: audit 2021-02-10T04:57:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:32 smithi164 conmon[29288]: 2021-02-10T04:57:31.469394+0000 mon.c 2021-02-10T04:57:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:32 smithi164 conmon[29288]: (mon.1) 143 : audit [INF] from='client.? 172.21.15.85:0/1701783337' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-02-10T04:57:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:32 smithi164 conmon[29288]: 2021-02-10T04:57:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:32 smithi164 conmon[29288]: audit 2021-02-10T04:57:31 2021-02-10T04:57:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:32 smithi164 conmon[29288]: .469795+0000 mon.a (mon.0) 2821 : audit 2021-02-10T04:57:32.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:32 smithi164 conmon[29288]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-02-10T04:57:33.366 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:33 smithi085 conmon[29319]: audit 2021-02-10T04:57:32.142796 2021-02-10T04:57:33.366 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:33 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2822 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-02-10T04:57:33.366 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:33 smithi085 conmon[29319]: cluster 2021-02-10T04:57:32.142970+0000 mon.a (mon.0) 2823 : cluster [DBG] osdmap e1301: 8 total, 8 up, 8 in 2021-02-10T04:57:33.366 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:33 smithi085 conmon[35961]: audit 2021-02-10T04:57 2021-02-10T04:57:33.366 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:33 smithi085 conmon[35961]: :32.142796+0000 mon.a (mon.0) 2822 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-02-10T04:57:33.366 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:33 smithi085 conmon[35961]: cluster 2021-02-10T04:57:33.366 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:33 smithi085 conmon[35961]: 2021-02 2021-02-10T04:57:33.366 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:33 smithi085 conmon[35961]: -10T04:57:32. 2021-02-10T04:57:33.367 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:33 smithi085 conmon[35961]: 142970+0000 mon.a ( 2021-02-10T04:57:33.367 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:33 smithi085 conmon[35961]: mon.0 2021-02-10T04:57:33.367 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:33 smithi085 conmon[35961]: ) 2823 : cluster [DBG] osdmap e1301: 8 total, 8 up, 8 in 2021-02-10T04:57:33.367 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:33 smithi085 conmon[35961]: 2021-02-10T04:57:33.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:33 smithi164 conmon[29288]: audit 2021-02-10T04:57:32.142796+0000 mon.a ( 2021-02-10T04:57:33.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:33 smithi164 conmon[29288]: mon.0) 2822 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-02-10T04:57:33.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:33 smithi164 conmon[29288]: cluster 2021-02-10T04:57:32.142970+0000 mon.a (mon.0) 2823 : cluster [DBG] 2021-02-10T04:57:33.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:33 smithi164 conmon[29288]: osdmap e1301: 8 total, 8 up, 8 in 2021-02-10T04:57:34.020 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 5 1 2021-02-10T04:57:34.231 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:34 smithi085 conmon[35961]: cluster 2021-02-10T04:57:32.532860+0000 mgr.y (mgr.14140) 1448 : cluster [DBG] pgmap v2368: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:34.232 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:34 smithi085 conmon[29319]: cluster 2021-02-10T04:57:32.532860+0000 mgr.y (mgr.14140) 1448 : cluster [DBG] pgmap v2368: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:34.271 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:34.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:34 smithi164 conmon[29288]: cluster 2021-02-10T04:57:32.532860+0000 mgr.y (mgr.14140) 1448 : cluster [DBG] 2021-02-10T04:57:34.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:34 smithi164 conmon[29288]: pgmap v2368: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:35.035 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:57:35.035 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [], 2021-02-10T04:57:35.035 INFO:teuthology.orchestra.run.smithi085.stdout: "complaint_time": 30, 2021-02-10T04:57:35.035 INFO:teuthology.orchestra.run.smithi085.stdout: "num_blocked_ops": 0 2021-02-10T04:57:35.035 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:57:35.103 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:57:35.322 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:35.957 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.2 dump_historic_ops 2021-02-10T04:57:36.201 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:36.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:36 smithi085 conmon[29319]: cluster 2021-02-10T04:57:34.533410+0000 mgr.y (mgr.14140) 1449 : cluster [DBG] pgmap v2369: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 798 B/s rd, 0 op/s 2021-02-10T04:57:36.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:36 smithi085 conmon[35961]: cluster 2021-02-10T04:57:34.533410+0000 mgr.y (mgr.14140) 1449 : cluster [DBG] pgmap v2369: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 798 B/s rd, 0 op/s 2021-02-10T04:57:36.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:36 smithi164 conmon[29288]: cluster 2021-02-10T04:57:34.533410+0000 mgr.y (mgr.14140) 1449 2021-02-10T04:57:36.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:36 smithi164 conmon[29288]: : cluster [DBG] pgmap v2369: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 798 B/s rd, 0 op/s 2021-02-10T04:57:37.169 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is set 2021-02-10T04:57:37.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:57:37 smithi164 conmon[37263]: debug 2021-02-10T04:57:37.069+0000 7f1c6b9eb700 -1 osd.5 pg_epoch: 1301 pg[6.a( empty local-lis/les=1227/1228 n=0 ec=59/59 lis/c=1227/1227 les/c/f=1228/1228/0 sis=1227) [5,6,0] r=0 lpr=1227 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x5566532d8c60) 2021-02-10T04:57:37.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:37 smithi164 conmon[29288]: audit 2021-02-10T04:57:37.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:37 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:57:37.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:37 smithi164 conmon[29288]: :57:36.654817+0000 mon.a (mon.0) 2824 : audit [INF] from='client.? 172.21.15.85:0/9045651' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:57:37.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:37 smithi085 conmon[29319]: audit 2021-02-10T 2021-02-10T04:57:37.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:37 smithi085 conmon[29319]: 04:57:36.654817+0000 mon.a (mon.0) 2824 : audit [INF] from='client.? 172.21.15.85:0/9045651' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:57:37.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:37 smithi085 conmon[35961]: audit 2021-02-10T04:57: 2021-02-10T04:57:37.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:37 smithi085 conmon[35961]: 36.654817+0000 mon.a (mon.0) 2824 : audit [INF] from='client.? 172.21.15.85:0/9045651' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:57:38.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:38 smithi085 conmon[29319]: cluster 2021-02-10T04:57:36.533981+0000 mgr.y (mgr.14140) 1450 2021-02-10T04:57:38.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:38 smithi085 conmon[29319]: : cluster [DBG] pgmap v2370: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:57:38.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:38 smithi085 conmon[29319]: cluster 2021-02-10T04:57:37.157114+0000 mon.a (mon.0) 2825 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:57:38.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:38 smithi085 conmon[29319]: audit 2021-02-10T04:57:37.164352+0000 mon.a (mon.0) 2826 : audit [INF] from='client.? 172.21.15.85:0/9045651' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:57:38.433 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:38 smithi085 conmon[29319]: cluster 2021-02-10T04:57:37.164424+0000 mon.a (mon.0) 2827 : cluster [DBG] osdmap e1302: 8 total, 8 up, 8 in 2021-02-10T04:57:38.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:38 smithi085 conmon[35961]: cluster 2021-02-10T04:57:36.533981+0000 2021-02-10T04:57:38.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:38 smithi085 conmon[35961]: mgr.y (mgr.14140) 1450 : cluster [DBG] pgmap v2370: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:57:38.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:38 smithi085 conmon[35961]: cluster 2021-02-10T04:57:38.433 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:38 smithi085 conmon[35961]: 2021-02-10T04:57:37.157114+0000 mon.a (mon.0) 2825 : cluster 2021-02-10T04:57:38.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:38 smithi085 conmon[35961]: [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:57:38.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:38 smithi085 conmon[35961]: audit 2021-02-10T04:57:38.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:38 smithi085 conmon[35961]: 2021-02-10T04:57:37.164352+0000 mon.a (mon 2021-02-10T04:57:38.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:38 smithi085 conmon[35961]: .0) 2826 : audit [INF] from='client.? 172.21.15.85:0/9045651' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:57:38.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:38 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:57:38.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:38 smithi085 conmon[35961]: 57:37.164424+0000 mon.a (mon.0) 2827 : 2021-02-10T04:57:38.434 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:38 smithi085 conmon[35961]: cluster [DBG] osdmap e1302: 8 total, 8 up, 8 in 2021-02-10T04:57:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:38 smithi164 conmon[29288]: cluster 2021-02-10T04:57:36.533981+0000 mgr.y ( 2021-02-10T04:57:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:38 smithi164 conmon[29288]: mgr.14140) 1450 : cluster [DBG] pgmap v2370: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-02-10T04:57:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:38 smithi164 conmon[29288]: cluster 2021-02-10T04:57:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:38 smithi164 conmon[29288]: 2021-02-10T04:57:37.157114+0000 mon.a ( 2021-02-10T04:57:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:38 smithi164 conmon[29288]: mon.0) 2825 : cluster [WRN] 2021-02-10T04:57:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:38 smithi164 conmon[29288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:57:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:38 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:57:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:38 smithi164 conmon[29288]: 02-10T04:57:37.164352 2021-02-10T04:57:38.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:38 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2826 2021-02-10T04:57:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:38 smithi164 conmon[29288]: : audit [INF] from='client.? 172.21.15.85:0/9045651' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:57:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:38 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:57:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:38 smithi164 conmon[29288]: -02-10T04:57:37.164424+0000 2021-02-10T04:57:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:38 smithi164 conmon[29288]: mon.a (mon.0) 2827 : cluster 2021-02-10T04:57:38.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:38 smithi164 conmon[29288]: [DBG] osdmap e1302: 8 total, 8 up, 8 in 2021-02-10T04:57:39.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:39 smithi085 conmon[29319]: audit 2021-02-10T04:57:38.259874+0000 mon.a (mon. 2021-02-10T04:57:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:39 smithi085 conmon[29319]: 0) 2828 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:57:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:39 smithi085 conmon[29319]: audit 2021-02-10T04:57:38.260158+0000 mon.a (mon.0) 2829 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:57:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:39 smithi085 conmon[29319]: cluster 2021-02-10T04:57:38.600624+0000 mon.a (mon.0) 2830 : cluster [DBG] osdmap e1303: 8 total, 8 up, 8 in 2021-02-10T04:57:39.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:39 smithi085 conmon[35961]: audit 2021-02-10T04:57:38.259874 2021-02-10T04:57:39.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:39 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2828 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:57:39.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:39 smithi085 conmon[35961]: audit 2021-02-10T04:57:39.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:39 smithi085 conmon[35961]: 2021-02-10T04:57:38.260158+0000 mon.a (mon.0) 2829 : 2021-02-10T04:57:39.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:39 smithi085 conmon[35961]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:57:39.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:39 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:57:39.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:39 smithi085 conmon[35961]: 04:57:38.600624+0000 mon.a (mon. 2021-02-10T04:57:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:39 smithi085 conmon[35961]: 0) 2830 : cluster [DBG] osdmap e1303: 8 total, 8 up, 8 in 2021-02-10T04:57:39.446 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:39 smithi164 conmon[29288]: audit 2021-02-10T04:57:38.259874+0000 mon.a 2021-02-10T04:57:39.446 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:39 smithi164 conmon[29288]: (mon.0) 2828 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:57:39.447 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:39 smithi164 conmon[29288]: audit 2021-02-10T04:57:39.447 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:39 smithi164 conmon[29288]: 2021-02-10T04:57:38.260158+0000 mon.a (mon.0) 2829 : audit [INF] 2021-02-10T04:57:39.447 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:39 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:57:39.447 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:39 smithi164 conmon[29288]: cluster 2021-02-10T04:57:39.447 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:39 smithi164 conmon[29288]: 2021-02-10T04:57:38.600624+0000 2021-02-10T04:57:39.447 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:39 smithi164 conmon[29288]: mon.a (mon.0) 2830 : cluster [DBG] osdmap e1303: 8 total, 8 up, 8 in 2021-02-10T04:57:39.447 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:39 smithi164 conmon[29288]: 2021-02-10T04:57:39.684 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:57:39.705 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:57:39.724 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:57:39.744 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:57:39.771 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:57:39.791 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:57:39.825 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:57:39.858 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:57:39.859 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:57:39.859 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:57:39.859 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:57:39.859 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:57:39.859 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:57:39.859 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:57:39.859 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:57:39.860 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:57:39.893 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:57:39 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:57:39.911 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:57:39.912 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:57:40.174 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.5 to 1 (10000) 2021-02-10T04:57:40.435 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set nodeep-scrub 2021-02-10T04:57:40.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:40 smithi085 conmon[29319]: cluster 2021-02-10T04:57:38.534603+0000 mgr.y (mgr.14140) 1451 : cluster [DBG] pgmap v2372: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-02-10T04:57:40.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:40 smithi085 conmon[29319]: audit 2021-02-10T04:57:39.660882+0000 mon.a (mon.0) 2831 : audit [DBG] from='client.? 172.21.15.85:0/2557312168' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:57:40.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:40 smithi085 conmon[29319]: audit 2021-02-10T04:57:39.856870+0000 mon.a (mon.0) 2832 : audit [INF] from='client.? 172.21.15.85:0/1964803368' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-02-10T04:57:40.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:40 smithi085 conmon[35961]: cluster 2021-02-10T04:57:38.534603+0000 2021-02-10T04:57:40.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:40 smithi085 conmon[35961]: mgr.y (mgr.14140) 1451 : cluster [DBG] pgmap v2372: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-02-10T04:57:40.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:40 smithi085 conmon[35961]: audit 2021-02-10T04:57:39.660882+0000 mon.a ( 2021-02-10T04:57:40.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:40 smithi085 conmon[35961]: mon.0) 2831 : audit [DBG] from='client.? 172.21.15.85:0/2557312168' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:57:40.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:40 smithi085 conmon[35961]: audit 2021-02-10T04:57:39.856870 2021-02-10T04:57:40.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:40 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2832 : audit [INF] from='client.? 172.21.15.85:0/1964803368' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-02-10T04:57:40.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:40 smithi085 conmon[35961]: 2021-02-10T04:57:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:40 smithi164 conmon[29288]: cluster 2021-02-10T04:57:38.534603+0000 2021-02-10T04:57:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:40 smithi164 conmon[29288]: mgr.y (mgr.14140) 1451 : cluster [DBG] pgmap v2372: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-02-10T04:57:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:40 smithi164 conmon[29288]: audit 2021-02-10T04:57:39.660882+0000 mon.a (mon.0) 2831 2021-02-10T04:57:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:40 smithi164 conmon[29288]: : audit [DBG] from='client.? 172.21.15.85:0/2557312168' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:57:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:40 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:57:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:40 smithi164 conmon[29288]: T04:57:39.856870+0000 mon.a (mon.0) 2832 : audit [INF] from='client.? 172.21.15.85:0/1964803368' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-02-10T04:57:40.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:40 smithi164 conmon[29288]: 2021-02-10T04:57:40.652 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:41.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:41 smithi085 conmon[29319]: audit 2021-02-10T04:57:39.446031+0000 mgr.y (mgr.14140) 2021-02-10T04:57:41.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:41 smithi085 conmon[29319]: 1452 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:57:41.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:41 smithi085 conmon[29319]: audit 2021-02-10T04:57:40.169946+0000 mon.a (mon.0) 2833 : audit [INF] from='client.? 172.21.15.85:0/1964803368' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-02-10T04:57:41.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:41 smithi085 conmon[29319]: cluster 2021-02-10T04:57:40.170013+0000 mon.a (mon.0) 2834 : cluster [DBG] osdmap e1304: 8 total, 8 up, 8 in 2021-02-10T04:57:41.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:41 smithi085 conmon[35961]: audit 2021-02-10T04:57:39.446031+0000 mgr.y (mgr 2021-02-10T04:57:41.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:41 smithi085 conmon[35961]: .14140) 1452 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:57:41.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:41 smithi085 conmon[35961]: audit 2021-02-10T04:57:40.169946+0000 mon.a (mon.0) 2833 : audit [INF] 2021-02-10T04:57:41.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:41 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/1964803368' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-02-10T04:57:41.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:41 smithi085 conmon[35961]: cluster 2021-02-10T04:57:40.170013 2021-02-10T04:57:41.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:41 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2834 : cluster [DBG] osdmap e1304: 8 total, 8 up, 8 in 2021-02-10T04:57:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:41 smithi164 conmon[29288]: audit 2021-02-10T04:57:39.446031 2021-02-10T04:57:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:41 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1452 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:57:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:41 smithi164 conmon[29288]: audit 2021-02-10T04:57:40.169946+0000 mon.a (mon.0) 2833 : audit [INF] from='client.? 172.21.15.85:0/1964803368' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-02-10T04:57:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:41 smithi164 conmon[29288]: cluster 2021-02-10T04:57:40.170013+0000 mon.a (mon.0) 2834 : cluster [DBG] osdmap e1304: 8 total, 8 up, 8 in 2021-02-10T04:57:41.942 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.3 dump_ops_in_flight 2021-02-10T04:57:42.048 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 6 1 2021-02-10T04:57:42.176 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:42.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:42 smithi085 conmon[35961]: cluster 2021-02-10T04:57:40. 2021-02-10T04:57:42.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:42 smithi085 conmon[35961]: 535128+0000 mgr.y (mgr.14140) 1453 : cluster [DBG] pgmap v2375: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:42.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:42 smithi085 conmon[29319]: cluster 2021-02-10T04:57:40.535128+0000 mgr.y (mgr. 2021-02-10T04:57:42.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:42 smithi085 conmon[29319]: 14140) 1453 : cluster [DBG] pgmap v2375: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:42.296 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:42.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:42 smithi164 conmon[29288]: cluster 2021-02-10T04:57:40.535128 2021-02-10T04:57:42.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:42 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1453 : cluster [DBG] pgmap v2375: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:43.603 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:57:43.823 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:44.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:44 smithi085 conmon[29319]: cluster 2021-02-10T04:57:42.535927+0000 mgr.y ( 2021-02-10T04:57:44.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:44 smithi085 conmon[29319]: mgr.14140) 1454 : cluster [DBG] pgmap v2376: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:44.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:44 smithi085 conmon[29319]: cluster 2021-02-10T04:57:43.603580+0000 mon.a (mon.0) 2835 : cluster [DBG] osdmap e1305: 8 total, 8 up, 8 in 2021-02-10T04:57:44.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:44 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:57:44.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:44 smithi085 conmon[35961]: 57:42.535927+0000 mgr.y (mgr.14140) 1454 : cluster [DBG] pgmap v2376: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:44.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:44 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:57:44.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:44 smithi085 conmon[35961]: 57:43.603580+0000 mon.a (mon.0) 2835 2021-02-10T04:57:44.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:44 smithi085 conmon[35961]: : cluster [DBG] osdmap e1305: 8 total, 8 up, 8 in 2021-02-10T04:57:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:44 smithi164 conmon[29288]: cluster 2021-02-10T04:57:42.535927+0000 mgr.y 2021-02-10T04:57:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:44 smithi164 conmon[29288]: (mgr.14140) 1454 : cluster [DBG] pgmap v2376: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:44.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:44 smithi164 conmon[29288]: cluster 2021-02-10T04:57:43.603580+0000 mon.a (mon.0) 2835 : cluster [DBG] osdmap e1305: 8 total, 8 up, 8 in 2021-02-10T04:57:46.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:46 smithi085 conmon[29319]: cluster 2021-02-10T04:57:44.536393+0000 mgr.y (mgr.14140) 1455 : cluster [DBG] pgmap v2378: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:46 smithi085 conmon[35961]: cluster 2021-02-10T04:57:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:46 smithi085 conmon[35961]: 2021-02-10T04:57:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:46 smithi085 conmon[35961]: 2021 2021-02-10T04:57:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:46 smithi085 conmon[35961]: - 2021-02-10T04:57:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:46 smithi085 conmon[35961]: 02 2021-02-10T04:57:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:46 smithi085 conmon[35961]: - 2021-02-10T04:57:46.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:46 smithi085 conmon[35961]: 10 2021-02-10T04:57:46.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:46 smithi085 conmon[35961]: T 2021-02-10T04:57:46.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:46 smithi085 conmon[35961]: 04 2021-02-10T04:57:46.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:46 smithi085 conmon[35961]: : 2021-02-10T04:57:46.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:46 smithi085 conmon[35961]: 57 2021-02-10T04:57:46.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:46 smithi085 conmon[35961]: : 2021-02-10T04:57:46.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:46 smithi085 conmon[35961]: 44 2021-02-10T04:57:46.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:46 smithi085 conmon[35961]: . 2021-02-10T04:57:46.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:46 smithi085 conmon[35961]: 536393 2021-02-10T04:57:46.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:46 smithi085 conmon[35961]: +0000 2021-02-10T04:57:46.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:46 smithi085 conmon[35961]: mgr.y (mgr.14140) 1455 : cluster [DBG] 2021-02-10T04:57:46.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:46 smithi085 conmon[35961]: pgmap v2378: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:46 smithi164 conmon[29288]: cluster 2021-02-10T04:57:44.536393+0000 mgr.y (mgr 2021-02-10T04:57:46.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:46 smithi164 conmon[29288]: .14140) 1455 : cluster [DBG] pgmap v2378: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:46.658 INFO:teuthology.orchestra.run.smithi085.stdout:ERROR: (22) Invalid argument 2021-02-10T04:57:46.658 INFO:teuthology.orchestra.run.smithi085.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:57:47.195 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is set 2021-02-10T04:57:47.195 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.6 to 1 (10000) 2021-02-10T04:57:47.250 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:47 smithi085 conmon[29319]: audit 2021-02-10T04:57:46.491854+0000 2021-02-10T04:57:47.250 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:47 smithi085 conmon[29319]: mon.a (mon.0) 2836 : audit [INF] from='client.? 172.21.15.85:0/1709163604' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T04:57:47.251 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:47 smithi085 conmon[29319]: audit 2021-02-10T04:57:46.546506+0000 mon.c (mon.1) 144 : audit [INF] from='client.? 172.21.15.85:0/1243896139' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:57:47.251 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:47 smithi085 conmon[29319]: audit 2021-02-10T04:57:46.546796+0000 mon.a (mon.0) 2837 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:57:47.251 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:47 smithi085 conmon[35961]: audit 2021-02-10T04:57:46.491854 2021-02-10T04:57:47.251 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:47 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2836 : audit [INF] from='client.? 172.21.15.85:0/1709163604' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T04:57:47.251 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:47 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:57:47.252 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:47 smithi085 conmon[35961]: 04:57:46.546506+0000 mon.c (mon.1) 144 : audit [INF] from='client.? 172.21.15.85:0/1243896139' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:57:47.252 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:47 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:57:47.252 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:47 smithi085 conmon[35961]: 10T04:57:46.546796+0000 mon.a (mon 2021-02-10T04:57:47.252 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:47 smithi085 conmon[35961]: .0) 2837 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:57:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:47 smithi164 conmon[29288]: audit 2021-02-10T04:57:46.491854+0000 2021-02-10T04:57:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:47 smithi164 conmon[29288]: mon.a (mon.0) 2836 : audit [INF] from='client.? 172.21.15.85:0/1709163604' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T04:57:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:47 smithi164 conmon[29288]: audit 2021-02-10T04:57:46.546506+0000 2021-02-10T04:57:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:47 smithi164 conmon[29288]: mon.c (mon.1) 144 : audit [INF] from='client.? 172.21.15.85:0/1243896139' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:57:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:47 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:57:47.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:47 smithi164 conmon[29288]: 04:57:46.546796+0000 mon.a (mon.0) 2837 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:57:48.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[35961]: cluster 2021-02-10T04:57:46.536874 2021-02-10T04:57:48.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1456 : cluster [DBG] pgmap v2379: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 645 B/s rd, 0 op/s 2021-02-10T04:57:48.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[35961]: cluster 2021-02-10T04:57:48.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[35961]: 2021-02-10T04:57:47.184636+0000 mon.a ( 2021-02-10T04:57:48.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[35961]: mon.0) 2838 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:57:48.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[35961]: 2021-02-10T04:57:48.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[35961]: audit 2021-02-10T04:57 2021-02-10T04:57:48.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[35961]: :47.189180+0000 mon.a (mon.0 2021-02-10T04:57:48.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[35961]: ) 2839 : audit [INF] from='client.? 172.21.15.85:0/1709163604' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2021-02-10T04:57:48.463 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[35961]: audit 2021 2021-02-10T04:57:48.464 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[35961]: -02-10T04:57:47. 2021-02-10T04:57:48.464 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[35961]: 189254+0000 mon.a (mon.0) 2840 : 2021-02-10T04:57:48.464 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[35961]: audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:57:48.464 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:57:48.464 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[35961]: T04:57:47.189324+0000 2021-02-10T04:57:48.464 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[35961]: mon.a (mon.0) 2841 : cluster 2021-02-10T04:57:48.464 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[35961]: [DBG] osdmap e1306: 8 total, 8 up, 8 in 2021-02-10T04:57:48.464 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[29319]: cluster 2021-02-10T04:57:46.536874 2021-02-10T04:57:48.464 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 1456 : cluster [DBG] pgmap v2379: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 645 B/s rd, 0 op/s 2021-02-10T04:57:48.464 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[29319]: cluster 2021-02-10T04:57:48.464 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[29319]: 2021-02-10T04:57:47.184636+0000 mon.a (mon. 2021-02-10T04:57:48.464 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[29319]: 0) 2838 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:57:48.465 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[29319]: audit 2021-02-10T 2021-02-10T04:57:48.465 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[29319]: 04:57:47.189180+0000 mon.a (mon.0 2021-02-10T04:57:48.465 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[29319]: ) 2839 : audit [INF] from='client.? 172.21.15.85:0/1709163604' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2021-02-10T04:57:48.465 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:57:48.465 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[29319]: -10T04:57:47.189254+0000 2021-02-10T04:57:48.465 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[29319]: mon.a (mon.0) 2840 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:57:48.465 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[29319]: 2021-02-10T04:57:48.465 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:57:48.465 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[29319]: -02-10T04:57 2021-02-10T04:57:48.465 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[29319]: :47.189324+0000 mon.a ( 2021-02-10T04:57:48.465 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[29319]: mon.0) 2841 : cluster [DBG] 2021-02-10T04:57:48.465 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:48 smithi085 conmon[29319]: osdmap e1306: 8 total, 8 up, 8 in 2021-02-10T04:57:48.466 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 7 1 2021-02-10T04:57:48.524 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.3 dump_blocked_ops 2021-02-10T04:57:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:48 smithi164 conmon[29288]: cluster 2021-02-10T04:57:46.536874+0000 mgr.y 2021-02-10T04:57:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:48 smithi164 conmon[29288]: (mgr.14140) 1456 : cluster [DBG] pgmap v2379: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 645 B/s rd, 0 op/s 2021-02-10T04:57:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:48 smithi164 conmon[29288]: cluster 2021-02-10T04:57:48.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:48 smithi164 conmon[29288]: 2021-02-10T04:57:47.184636+0000 mon.a (mon.0 2021-02-10T04:57:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:48 smithi164 conmon[29288]: ) 2838 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:57:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:48 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:57:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:48 smithi164 conmon[29288]: 10T04:57:47.189180+0000 mon.a (mon.0) 2839 : audit [INF] from='client.? 172.21.15.85:0/1709163604' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2021-02-10T04:57:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:48 smithi164 conmon[29288]: audit 2021-02-10T04:57:47.189254+0000 mon.a (mon. 2021-02-10T04:57:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:48 smithi164 conmon[29288]: 0) 2840 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:57:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:48 smithi164 conmon[29288]: cluster 2021-02-10T04:57:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:48 smithi164 conmon[29288]: 2021-02-10T04:57:47.189324+0000 mon.a (mon.0) 2841 : cluster [DBG] 2021-02-10T04:57:48.645 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:48 smithi164 conmon[29288]: osdmap e1306: 8 total, 8 up, 8 in 2021-02-10T04:57:48.764 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:48.818 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:49.018 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:57:49.038 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:57:49.061 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:57:49.080 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:57:49.099 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:57:49.118 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:57:49.136 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:57:49.161 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:57:49.162 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:57:49.163 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:57:49.163 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:57:49.163 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:57:49.163 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:57:49.163 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:57:49.163 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:57:49.163 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:57:49.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:49 smithi085 conmon[29319]: cluster 2021-02-10T04:57:48.537524+0000 mgr.y (mgr.14140) 1457 : cluster [DBG] pgmap v2381: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:57:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:49 smithi085 conmon[29319]: cluster 2021-02-10T04:57:48.608033+0000 mon.a (mon.0) 2842 : cluster [DBG] osdmap e1307: 8 total, 8 up, 8 in 2021-02-10T04:57:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:49 smithi085 conmon[29319]: cluster 2021-02-10T04:57:48.724723+0000 osd.7 (osd.7) 96 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:57:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:49 smithi085 conmon[29319]: cluster 2021-02-10T04:57:48.726191+0000 osd.7 (osd.7) 97 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:57:49.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:49 smithi085 conmon[29319]: audit 2021-02-10T04:57:48.997879+0000 mon.c (mon.1) 145 : audit [DBG] from='client.? 172.21.15.85:0/420036895' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:57:49.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:49 smithi085 conmon[35961]: cluster 2021-02-10T04:57:48.537524+0000 mgr.y (mgr.14140) 1457 : cluster [DBG] pgmap v2381: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:57:49.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:49 smithi085 conmon[35961]: cluster 2021-02-10T04:57:48.608033+0000 mon.a (mon.0) 2842 : cluster [DBG] osdmap e1307: 8 total, 8 up, 8 in 2021-02-10T04:57:49.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:49 smithi085 conmon[35961]: cluster 2021-02-10T04:57:48.724723+0000 osd.7 (osd.7) 96 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:57:49.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:49 smithi085 conmon[35961]: cluster 2021-02-10T04:57:48.726191+0000 osd.7 (osd.7) 97 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:57:49.693 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:49 smithi085 conmon[35961]: audit 2021-02-10T04:57:48.997879+0000 mon.c (mon.1) 145 : audit [DBG] from='client.? 172.21.15.85:0/420036895' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:57:49.893 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:57:49 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:57:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:49 smithi164 conmon[29288]: cluster 2021-02-10T04:57:48.537524 2021-02-10T04:57:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:49 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1457 : cluster [DBG] pgmap v2381: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:57:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:49 smithi164 conmon[29288]: cluster 2021-02-10T04:57:48.608033+0000 mon.a (mon.0) 2842 : cluster [DBG] osdmap e1307: 8 total, 8 up, 8 in 2021-02-10T04:57:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:49 smithi164 conmon[29288]: cluster 2021-02-10T04:57:48.724723+0000 osd.7 (osd.7) 96 : cluster 2021-02-10T04:57:49.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:49 smithi164 conmon[29288]: [DBG] purged_snaps scrub starts 2021-02-10T04:57:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:49 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:57:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:49 smithi164 conmon[29288]: -02-10T04:57:48.726191+0000 osd.7 (osd.7) 97 : cluster [DBG] 2021-02-10T04:57:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:49 smithi164 conmon[29288]: purged_snaps scrub ok 2021-02-10T04:57:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:49 smithi164 conmon[29288]: audit 2021-02-10T04:57:48. 2021-02-10T04:57:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:49 smithi164 conmon[29288]: 997879+0000 mon.c (mon.1) 145 : audit [DBG] from='client.? 172.21.15.85:0/420036895' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:57:50.444 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset noscrub 2021-02-10T04:57:50.647 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:50.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:50 smithi164 conmon[29288]: audit 2021-02-10T04:57:49.455971+0000 mgr.y (mgr.14140) 1458 : audit [DBG] 2021-02-10T04:57:50.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:50 smithi164 conmon[29288]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:57:50.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:50 smithi164 conmon[29288]: cluster 2021-02-10T04:57:49.616345+0000 mon.a (mon.0) 2843 : cluster [DBG] osdmap e1308: 8 total, 8 up, 8 in 2021-02-10T04:57:50.927 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:50 smithi085 conmon[29319]: audit 2021-02-10T04:57:49.455971+0000 mgr.y (mgr 2021-02-10T04:57:50.927 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:50 smithi085 conmon[29319]: .14140) 1458 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:57:50.927 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:50 smithi085 conmon[29319]: cluster 2021-02-10T04:57:49.616345+0000 mon.a (mon.0) 2843 : cluster [DBG] osdmap e1308: 8 total, 8 up, 8 in 2021-02-10T04:57:50.927 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:50 smithi085 conmon[35961]: audit 2021-02-10T04:57 2021-02-10T04:57:50.927 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:50 smithi085 conmon[35961]: :49.455971+0000 mgr.y (mgr.14140) 1458 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:57:50.927 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:50 smithi085 conmon[35961]: cluster 2021-02-10T04:57:49.616345+0000 mon.a (mon.0) 2843 : cluster [DBG] osdmap e1308: 8 total, 8 up, 8 in 2021-02-10T04:57:51.717 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:51 smithi085 conmon[29319]: cluster 2021-02-10T04:57:50.538023+0000 mgr.y (mgr.14140) 2021-02-10T04:57:51.717 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:51 smithi085 conmon[29319]: 1459 : cluster [DBG] pgmap v2384: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:51.717 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:51 smithi085 conmon[35961]: cluster 2021-02-10T04:57:50.538023+0000 mgr.y 2021-02-10T04:57:51.717 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:51 smithi085 conmon[35961]: (mgr.14140) 1459 : cluster [DBG] pgmap v2384: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:51.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:51 smithi164 conmon[29288]: cluster 2021-02-10T04:57:52.285 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:51 smithi164 conmon[29288]: 2021-02- 2021-02-10T04:57:52.285 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:51 smithi164 conmon[29288]: 10T04:57:50.538023+0000 mgr.y (mgr.14140) 1459 : cluster [DBG] pgmap v2384: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:52.574 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:57:52.575 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [], 2021-02-10T04:57:52.575 INFO:teuthology.orchestra.run.smithi085.stdout: "complaint_time": 30, 2021-02-10T04:57:52.575 INFO:teuthology.orchestra.run.smithi085.stdout: "num_blocked_ops": 0 2021-02-10T04:57:52.575 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:57:52.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:52 smithi164 conmon[29288]: cephadm 2021-02-10T04:57:52.001855+0000 mgr.y (mgr.14140) 2021-02-10T04:57:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:52 smithi164 conmon[29288]: 1460 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:57:52.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:52 smithi164 conmon[29288]: cephadm 2021-02-10T04:57:52.002357+0000 mgr.y (mgr.14140) 1461 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:57:52.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:52 smithi085 conmon[35961]: cephadm 2021-02-10T04:57:52.001855+0000 mgr.y (mgr.14140) 1460 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:57:52.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:52 smithi085 conmon[35961]: cephadm 2021-02-10T04:57:52.002357+0000 mgr.y (mgr.14140) 1461 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:57:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:52 smithi085 conmon[29319]: cephadm 2021-02-10T04:57:52.001855+0000 mgr.y (mgr.14140) 1460 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:57:52.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:52 smithi085 conmon[29319]: cephadm 2021-02-10T04:57:52.002357+0000 mgr.y (mgr.14140) 1461 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:57:53.452 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:57:53.613 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.7 to 1 (10000) 2021-02-10T04:57:53.628 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[35961]: cluster 2021-02-10T04:57:53.628 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[35961]: 2021-02-10T04:57: 2021-02-10T04:57:53.628 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[35961]: 52.538656+0000 mgr.y (mgr.14140) 1462 : cluster [DBG] pgmap v2385: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:53.628 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:57:53.628 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[35961]: 02-10T04:57:52.687950+0000 mon.c ( 2021-02-10T04:57:53.628 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[35961]: mon.1) 146 : audit [INF] from='client.? 172.21.15.85:0/1260358574' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T04:57:53.629 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:57:53.629 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[35961]: -10T04:57:52.688355+0000 mon.a (mon. 2021-02-10T04:57:53.629 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[35961]: 0) 2844 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T04:57:53.629 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:57:53.629 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[35961]: -10T04:57:53.604518+0000 mon.a 2021-02-10T04:57:53.629 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[35961]: (mon.0) 2845 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-02-10T04:57:53.629 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:57:53.629 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[35961]: -02-10T04:57:53.604596+0000 mon.a 2021-02-10T04:57:53.629 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[35961]: (mon.0) 2846 : cluster [DBG] osdmap e1309: 8 total, 8 up, 8 in 2021-02-10T04:57:53.677 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:53.849 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.3 dump_historic_ops 2021-02-10T04:57:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:53 smithi164 conmon[29288]: cluster 2021-02-10T04:57:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:53 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:57:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:53 smithi164 conmon[29288]: :57:52.538656+0000 mgr.y (mgr.14140) 1462 : cluster [DBG] pgmap v2385: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:53 smithi164 conmon[29288]: audit 2021-02-10T04:57:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:53 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:57:53.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:53 smithi164 conmon[29288]: 57:52.687950+0000 mon.c (mon.1 2021-02-10T04:57:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:53 smithi164 conmon[29288]: ) 146 : audit [INF] from='client.? 172.21.15.85:0/1260358574' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T04:57:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:53 smithi164 conmon[29288]: audit 2021-02-10T04:57:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:53 smithi164 conmon[29288]: 2021-02-10T04:57:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:53 smithi164 conmon[29288]: 2021-02-10T04:57 2021-02-10T04:57:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:53 smithi164 conmon[29288]: :52.688355+0000 mon.a (mon.0) 2844 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T04:57:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:53 smithi164 conmon[29288]: audit 2021-02-10T04:57:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:53 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:57:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:53 smithi164 conmon[29288]: 04:57:53.604518+0000 mon.a (mon 2021-02-10T04:57:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:53 smithi164 conmon[29288]: .0) 2845 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-02-10T04:57:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:53 smithi164 conmon[29288]: cluster 2021-02-10T04:57:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:53 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:57:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:53 smithi164 conmon[29288]: :57:53.604596+0000 2021-02-10T04:57:53.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:53 smithi164 conmon[29288]: mon.a (mon.0) 2846 : cluster 2021-02-10T04:57:53.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:53 smithi164 conmon[29288]: [DBG] osdmap e1309: 8 total, 8 up, 8 in 2021-02-10T04:57:53.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[29319]: cluster 2021-02-10T04:57:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[29319]: 2021-02-10T04:57 2021-02-10T04:57:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[29319]: :52.538656+0000 mgr.y (mgr.14140) 1462 : cluster [DBG] pgmap v2385: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[29319]: audit 2021-02-10T04:57:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[29319]: 2021-02-10T04:57:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[29319]: 2021 2021-02-10T04:57:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[29319]: - 2021-02-10T04:57:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[29319]: 02 2021-02-10T04:57:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[29319]: -10T04:57:52. 2021-02-10T04:57:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[29319]: 687950+0000 mon.c (mon.1) 146 : audit [INF] from='client.? 172.21.15.85:0/1260358574' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T04:57:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:57:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[29319]: :57:52.688355+0000 mon.a (mon.0) 2844 : audit [INF] 2021-02-10T04:57:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[29319]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T04:57:53.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[29319]: audit 2021-02-10T04:57:53.604518+0000 2021-02-10T04:57:53.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[29319]: mon.a (mon.0) 2845 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-02-10T04:57:53.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:57:53.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[29319]: 02-10T04:57:53.604596+0000 mon.a 2021-02-10T04:57:53.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:53 smithi085 conmon[29319]: (mon.0) 2846 : cluster [DBG] osdmap e1309: 8 total, 8 up, 8 in 2021-02-10T04:57:54.094 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:54.681 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 2 1 2021-02-10T04:57:54.925 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:55.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:54 smithi164 conmon[29288]: audit 2021-02-10T04:57:53.843446+0000 mon.a (mon.0) 2847 : audit 2021-02-10T04:57:55.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:54 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:57:55.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:54 smithi085 conmon[29319]: audit 2021-02-10T04:57:53.843446+0000 mon.a (mon.0) 2847 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:57:55.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:54 smithi085 conmon[35961]: audit 2021-02-10T04:57:55.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:54 smithi085 conmon[35961]: 2021-02-10T04:57:55.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:54 smithi085 conmon[35961]: 2021-02-10T04:57:53. 2021-02-10T04:57:55.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:54 smithi085 conmon[35961]: 843446+0000 mon.a (mon.0) 2847 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:57:55.862 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is unset 2021-02-10T04:57:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:55 smithi164 conmon[29288]: cluster 2021-02-10T04:57:54.539082+0000 2021-02-10T04:57:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:55 smithi164 conmon[29288]: mgr.y (mgr.14140) 1463 : cluster [DBG] pgmap v2387: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:55 smithi164 conmon[29288]: audit 2021-02-10T04:57:55.556772+0000 mon.a (mon.0) 2848 : audit [INF] 2021-02-10T04:57:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:55 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/311185289' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:57:56.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:55 smithi085 conmon[29319]: cluster 2021-02-10T04:57:54.539082+0000 mgr.y (mgr.14140) 1463 2021-02-10T04:57:56.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:55 smithi085 conmon[29319]: : cluster [DBG] pgmap v2387: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:56.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:55 smithi085 conmon[29319]: audit 2021-02-10T04:57:55.556772+0000 mon.a (mon.0) 2848 : audit [INF] from='client.? 172.21.15.85:0/311185289' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:57:56.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:55 smithi085 conmon[35961]: cluster 2021-02-10T04:57:56.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:55 smithi085 conmon[35961]: 2021-02-10T04:57:56.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:55 smithi085 conmon[35961]: 2021 2021-02-10T04:57:56.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:55 smithi085 conmon[35961]: - 2021-02-10T04:57:56.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:55 smithi085 conmon[35961]: 02 2021-02-10T04:57:56.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:55 smithi085 conmon[35961]: - 2021-02-10T04:57:56.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:55 smithi085 conmon[35961]: 10T04:57:54.539082+0000 2021-02-10T04:57:56.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:55 smithi085 conmon[35961]: mgr.y (mgr.14140) 1463 : cluster [DBG] pgmap v2387: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:57:56.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:55 smithi085 conmon[35961]: audit 2021-02-10T04:57 2021-02-10T04:57:56.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:55 smithi085 conmon[35961]: :55.556772+0000 mon.a (mon.0) 2848 : audit 2021-02-10T04:57:56.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:55 smithi085 conmon[35961]: [INF] from='client.? 172.21.15.85:0/311185289' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:57:56.662 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[49106]: debug 2021-02-10T04:57:56.408+0000 7f778ce21700 -1 osd.2 pg_epoch: 1310 pg[5.8( empty local-lis/les=1227/1228 n=0 ec=57/57 lis/c=1227/1227 les/c/f=1228/1228/0 sis=1227) [2,0,3] r=0 lpr=1227 crt=0'0 mlcod 0'0 active+clean] scrubber pg(5.8) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x564aa30b5ce0) 2021-02-10T04:57:56.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[35961]: cluster 2021-02-10T04:57:55.808326 2021-02-10T04:57:56.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2661 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:57:56.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[35961]: cluster 2021-02-10T04:57:56.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[35961]: 2021-02-10T04:57:55.809972+0000 osd.4 (osd.4) 2662 2021-02-10T04:57:56.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[35961]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:57:56.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:57:56.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[35961]: T04:57:55.846328+0000 mon.a (mon.0) 2849 : 2021-02-10T04:57:56.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[35961]: cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:57:56.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[35961]: audit 2021-02-10T04:57 2021-02-10T04:57:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[35961]: :55.855383+0000 mon.a (mon.0) 2021-02-10T04:57:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[35961]: 2850 : audit [INF] from='client.? 172.21.15.85:0/311185289' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:57:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:57:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[35961]: :57:55.855467+0000 mon.a (mon.0 2021-02-10T04:57:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[35961]: ) 2851 : cluster [DBG] osdmap e1310: 8 total, 8 up, 8 in 2021-02-10T04:57:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[29319]: cluster 2021-02-10T04:57:55.808326+0000 2021-02-10T04:57:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[29319]: osd.4 (osd.4) 2661 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:57:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[29319]: cluster 2021-02-10T04:57:55.809972+0000 2021-02-10T04:57:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[29319]: osd.4 (osd.4) 2662 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:57:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:57:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[29319]: 02-10T04:57:55.846328+0000 2021-02-10T04:57:56.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[29319]: mon.a (mon.0) 2849 : 2021-02-10T04:57:56.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[29319]: cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:57:56.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[29319]: audit 2021-02-10T04:57:56.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[29319]: 2021-02- 2021-02-10T04:57:56.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[29319]: 10T04:57:55. 2021-02-10T04:57:56.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[29319]: 855383+0000 mon.a (mon.0) 2021-02-10T04:57:56.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[29319]: 2850 : audit [INF] from='client.? 172.21.15.85:0/311185289' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:57:56.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[29319]: cluster 2021-02-10T04:57:56.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[29319]: 2021-02-10T04:57 2021-02-10T04:57:56.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[29319]: :55.855467+0000 mon.a (mon 2021-02-10T04:57:56.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[29319]: .0) 2851 : cluster [DBG] 2021-02-10T04:57:56.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:56 smithi085 conmon[29319]: osdmap e1310: 8 total, 8 up, 8 in 2021-02-10T04:57:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:56 smithi164 conmon[29288]: cluster 2021-02-10T04:57:55.808326+0000 osd.4 (osd. 2021-02-10T04:57:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:56 smithi164 conmon[29288]: 4) 2661 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:57:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:56 smithi164 conmon[29288]: cluster 2021-02-10T04:57:55.809972+0000 osd.4 (osd.4) 2662 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:57:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:56 smithi164 conmon[29288]: cluster 2021-02-10T04:57:55.846328+0000 mon.a (mon.0 2021-02-10T04:57:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:56 smithi164 conmon[29288]: ) 2849 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:57:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:56 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:57:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:56 smithi164 conmon[29288]: 02-10T04:57:55.855383+0000 mon.a ( 2021-02-10T04:57:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:56 smithi164 conmon[29288]: mon.0) 2850 : audit [INF] from='client.? 172.21.15.85:0/311185289' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:57:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:56 smithi164 conmon[29288]: cluster 2021-02-10T04:57:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:56 smithi164 conmon[29288]: 2021-02-10T04:57:55.855467 2021-02-10T04:57:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:56 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2851 : cluster [DBG] 2021-02-10T04:57:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:56 smithi164 conmon[29288]: osdmap e1310: 8 total, 8 up, 8 in 2021-02-10T04:57:57.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:57:57 smithi085 conmon[49106]: debug 2021-02-10T04:57:57.362+0000 7f778ce21700 -1 osd.2 pg_epoch: 1310 pg[5.8( empty local-lis/les=1227/1228 n=0 ec=57/57 lis/c=1227/1227 les/c/f=1228/1228/0 sis=1227) [2,0,3] r=0 lpr=1227 crt=0'0 mlcod 0'0 active+clean] scrubber pg(5.8) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x564aa65ba420) 2021-02-10T04:57:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:57 smithi164 conmon[29288]: cluster 2021-02-10T04:57:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:57 smithi164 conmon[29288]: 2021-02-10T04:57:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:57 smithi164 conmon[29288]: 2021 2021-02-10T04:57:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:57 smithi164 conmon[29288]: - 2021-02-10T04:57:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:57 smithi164 conmon[29288]: 02-10T04:57:56. 2021-02-10T04:57:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:57 smithi164 conmon[29288]: 539484+0000 mgr.y (mgr.14140) 1464 : cluster [DBG] pgmap v2389: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2021-02-10T04:57:58.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:57 smithi085 conmon[35961]: cluster 2021-02-10T04:57:56.539484+0000 mgr.y (mgr.14140) 1464 : cluster [DBG] pgmap v2389: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2021-02-10T04:57:58.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:57 smithi085 conmon[29319]: cluster 2021-02-10T04:57:56.539484+0000 mgr.y (mgr.14140) 1464 : cluster [DBG] pgmap v2389: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2021-02-10T04:57:59.044 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:57:59.061 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:57:59.061 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 20, 2021-02-10T04:57:59.061 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 600, 2021-02-10T04:57:59.061 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [ 2021-02-10T04:57:59.061 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.061 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:3431 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94187755107840 gen 3] snapc 0=[] ondisk+write+known_if_redirected e1112)", 2021-02-10T04:57:59.061 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:02.988562+0000", 2021-02-10T04:57:59.061 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 416.06968104999999, 2021-02-10T04:57:59.062 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9928285949999998, 2021-02-10T04:57:59.062 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:57:59.062 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:57:59.062 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:57:59.062 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:57:59.062 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:57:59.062 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 3431 2021-02-10T04:57:59.062 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.062 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:57:59.062 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.062 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:57:59.062 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988562+0000", 2021-02-10T04:57:59.062 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:57:59.063 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.063 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.063 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:57:59.063 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988562+0000", 2021-02-10T04:57:59.063 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.1761000000000001e-05 2021-02-10T04:57:59.063 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.063 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.063 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:57:59.063 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988583+0000", 2021-02-10T04:57:59.063 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.5929999999999996e-06 2021-02-10T04:57:59.063 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.063 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.063 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:57:59.064 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988589+0000", 2021-02-10T04:57:59.064 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.8294000000000001e-05 2021-02-10T04:57:59.064 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.064 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.064 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:57:59.064 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988607+0000", 2021-02-10T04:57:59.064 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.6950000000000003e-06 2021-02-10T04:57:59.064 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.064 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.064 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:57:59.064 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988614+0000", 2021-02-10T04:57:59.064 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.5459999999999999e-05 2021-02-10T04:57:59.064 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.064 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.065 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:57:59.065 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988689+0000", 2021-02-10T04:57:59.065 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.0969e-05 2021-02-10T04:57:59.065 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.065 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.065 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for readable", 2021-02-10T04:57:59.065 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988730+0000", 2021-02-10T04:57:59.065 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9912170979999999 2021-02-10T04:57:59.065 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.065 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.065 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:57:59.065 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.979947+0000", 2021-02-10T04:57:59.065 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00020239699999999999 2021-02-10T04:57:59.066 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.066 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.066 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:57:59.066 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.980150+0000", 2021-02-10T04:57:59.066 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001240328 2021-02-10T04:57:59.066 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.066 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.066 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:57:59.066 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.981390+0000", 2021-02-10T04:57:59.066 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9928285949999998 2021-02-10T04:57:59.066 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.066 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:57:59.066 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.067 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.067 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.067 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:3432 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 94187770080640 gen 4] snapc 0=[] ondisk+write+known_if_redirected e1112)", 2021-02-10T04:57:59.067 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:02.988669+0000", 2021-02-10T04:57:59.067 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 416.06957333999998, 2021-02-10T04:57:59.067 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0090249269999996, 2021-02-10T04:57:59.067 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:57:59.067 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:57:59.067 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:57:59.067 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:57:59.067 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:57:59.067 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 3432 2021-02-10T04:57:59.067 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.068 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:57:59.068 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.068 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:57:59.068 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988669+0000", 2021-02-10T04:57:59.068 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:57:59.068 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.068 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.068 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:57:59.068 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988669+0000", 2021-02-10T04:57:59.068 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.0630999999999999e-05 2021-02-10T04:57:59.068 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.068 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.068 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:57:59.069 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988700+0000", 2021-02-10T04:57:59.069 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.3149999999999997e-06 2021-02-10T04:57:59.069 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.069 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.069 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:57:59.069 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988706+0000", 2021-02-10T04:57:59.069 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0995000000000001e-05 2021-02-10T04:57:59.069 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.069 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.069 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:57:59.069 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988717+0000", 2021-02-10T04:57:59.069 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.3070000000000002e-06 2021-02-10T04:57:59.069 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.070 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.070 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:57:59.070 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988722+0000", 2021-02-10T04:57:59.070 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.5676e-05 2021-02-10T04:57:59.070 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.070 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.070 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:57:59.070 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988778+0000", 2021-02-10T04:57:59.070 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.2759999999999998e-05 2021-02-10T04:57:59.070 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.070 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.070 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for readable", 2021-02-10T04:57:59.070 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:02.988811+0000", 2021-02-10T04:57:59.071 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0076071410000003 2021-02-10T04:57:59.071 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.071 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.071 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:57:59.071 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.996418+0000", 2021-02-10T04:57:59.071 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000159133 2021-02-10T04:57:59.071 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.071 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.071 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:57:59.071 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.996577+0000", 2021-02-10T04:57:59.071 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001116969 2021-02-10T04:57:59.071 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.071 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.071 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:57:59.072 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.997694+0000", 2021-02-10T04:57:59.072 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0090249269999996 2021-02-10T04:57:59.072 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.072 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:57:59.072 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.072 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.072 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.072 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:3435 5.14 5:2b04a3e9:::notify.6:head [watch ping cookie 94187770080640 gen 4] snapc 0=[] ondisk+write+known_if_redirected e1113)", 2021-02-10T04:57:59.072 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:07.988123+0000", 2021-02-10T04:57:59.072 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 411.07011919000001, 2021-02-10T04:57:59.072 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0089928860000000003, 2021-02-10T04:57:59.072 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:57:59.073 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:57:59.073 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:57:59.073 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:57:59.073 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:57:59.073 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 3435 2021-02-10T04:57:59.073 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.073 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:57:59.073 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.073 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:57:59.073 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.988123+0000", 2021-02-10T04:57:59.073 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:57:59.073 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.073 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.073 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:57:59.074 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.988123+0000", 2021-02-10T04:57:59.074 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.8739999999999999e-05 2021-02-10T04:57:59.074 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.074 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.074 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:57:59.074 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.988152+0000", 2021-02-10T04:57:59.074 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.4450000000000001e-06 2021-02-10T04:57:59.074 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.074 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.074 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:57:59.074 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.988159+0000", 2021-02-10T04:57:59.074 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0798999999999999e-05 2021-02-10T04:57:59.074 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.075 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.075 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:57:59.075 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.988169+0000", 2021-02-10T04:57:59.075 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.2630000000000003e-06 2021-02-10T04:57:59.075 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.075 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.075 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:57:59.075 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.988175+0000", 2021-02-10T04:57:59.075 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.1480000000000002e-05 2021-02-10T04:57:59.075 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.075 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.075 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:57:59.075 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.988226+0000", 2021-02-10T04:57:59.076 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.1520999999999997e-05 2021-02-10T04:57:59.076 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.076 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.076 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for readable", 2021-02-10T04:57:59.076 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.988278+0000", 2021-02-10T04:57:59.076 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0086211429999999995 2021-02-10T04:57:59.076 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.076 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.076 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:57:59.076 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.996899+0000", 2021-02-10T04:57:59.076 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.0494000000000006e-05 2021-02-10T04:57:59.076 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.076 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.077 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:57:59.077 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.996979+0000", 2021-02-10T04:57:59.077 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00013700100000000001 2021-02-10T04:57:59.077 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.077 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.077 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:57:59.077 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:07.997116+0000", 2021-02-10T04:57:59.077 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0089928860000000003 2021-02-10T04:57:59.077 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.077 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:57:59.077 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.077 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.077 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.078 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:3481 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 94187770080640 gen 5] snapc 0=[] ondisk+write+known_if_redirected e1118)", 2021-02-10T04:57:59.078 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:35.712201+0000", 2021-02-10T04:57:59.078 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 383.346041544, 2021-02-10T04:57:59.078 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.172105363, 2021-02-10T04:57:59.078 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:57:59.078 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:57:59.078 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T04:57:59.078 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T04:57:59.078 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T04:57:59.078 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 3481 2021-02-10T04:57:59.078 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.078 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:57:59.078 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.079 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:57:59.079 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:35.712201+0000", 2021-02-10T04:57:59.079 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:57:59.079 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.079 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.079 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:57:59.079 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:35.712201+0000", 2021-02-10T04:57:59.079 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.2008000000000003e-05 2021-02-10T04:57:59.079 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.079 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.079 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:57:59.079 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:35.712233+0000", 2021-02-10T04:57:59.079 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.0690000000000002e-06 2021-02-10T04:57:59.080 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.080 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.080 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:57:59.080 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:35.712241+0000", 2021-02-10T04:57:59.080 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2391999999999999e-05 2021-02-10T04:57:59.080 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.080 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.080 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:57:59.080 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:35.712253+0000", 2021-02-10T04:57:59.080 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.5499999999999997e-06 2021-02-10T04:57:59.080 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.080 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.080 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:57:59.080 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:35.712261+0000", 2021-02-10T04:57:59.081 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0015999999999997e-05 2021-02-10T04:57:59.081 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.081 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.081 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:57:59.081 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:35.712311+0000", 2021-02-10T04:57:59.081 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0231000000000001e-05 2021-02-10T04:57:59.081 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.081 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.081 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op must wait for map", 2021-02-10T04:57:59.081 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:35.712321+0000", 2021-02-10T04:57:59.081 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.02825248 2021-02-10T04:57:59.081 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.081 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.081 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:57:59.082 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:35.740574+0000", 2021-02-10T04:57:59.082 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0251999999999999e-05 2021-02-10T04:57:59.082 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.082 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.082 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for peered", 2021-02-10T04:57:59.082 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:35.740584+0000", 2021-02-10T04:57:59.082 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.129581339 2021-02-10T04:57:59.082 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.082 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.082 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:57:59.082 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:36.870165+0000", 2021-02-10T04:57:59.082 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00034935599999999999 2021-02-10T04:57:59.082 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.083 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.083 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting for degraded object", 2021-02-10T04:57:59.083 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:36.870515+0000", 2021-02-10T04:57:59.083 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.012278183999999999 2021-02-10T04:57:59.083 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.083 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.083 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:57:59.083 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:36.882793+0000", 2021-02-10T04:57:59.083 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000120188 2021-02-10T04:57:59.083 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.083 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.083 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:57:59.083 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:36.882913+0000", 2021-02-10T04:57:59.084 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0013932980000000001 2021-02-10T04:57:59.084 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.084 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.084 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:57:59.084 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:36.884306+0000", 2021-02-10T04:57:59.084 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.172105363 2021-02-10T04:57:59.084 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.084 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:57:59.084 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.084 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.084 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.084 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.14 1121/1119 [PushReplyOp(7:2dcefd69:::smithi08575397-28:333)])", 2021-02-10T04:57:59.084 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:39.199259+0000", 2021-02-10T04:57:59.084 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 379.85898339699997, 2021-02-10T04:57:59.085 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0030838609999999998, 2021-02-10T04:57:59.085 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:57:59.085 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:57:59.085 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:57:59.085 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.085 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:57:59.085 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.199259+0000", 2021-02-10T04:57:59.085 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:57:59.085 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.085 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.085 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:57:59.085 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.199259+0000", 2021-02-10T04:57:59.085 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.2369999999999997e-05 2021-02-10T04:57:59.085 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.086 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.086 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:57:59.086 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.199292+0000", 2021-02-10T04:57:59.086 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.072e-06 2021-02-10T04:57:59.086 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.086 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.086 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:57:59.086 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.199300+0000", 2021-02-10T04:57:59.086 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.4684999999999999e-05 2021-02-10T04:57:59.086 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.086 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.086 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:57:59.086 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.199314+0000", 2021-02-10T04:57:59.087 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.5300000000000002e-06 2021-02-10T04:57:59.087 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.087 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.087 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:57:59.087 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.199321+0000", 2021-02-10T04:57:59.087 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9565999999999999e-05 2021-02-10T04:57:59.087 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.087 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.087 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:57:59.087 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.199370+0000", 2021-02-10T04:57:59.087 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0029726380000000001 2021-02-10T04:57:59.087 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.087 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.087 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:57:59.088 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.202343+0000", 2021-02-10T04:57:59.088 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0030838609999999998 2021-02-10T04:57:59.088 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.088 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:57:59.088 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.088 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.088 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.088 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.14 1121/1119 [PushReplyOp(7:2dcefd69:::smithi08575397-28:368)])", 2021-02-10T04:57:59.088 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:39.217537+0000", 2021-02-10T04:57:59.088 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 379.84070564199999, 2021-02-10T04:57:59.088 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031144010000000001, 2021-02-10T04:57:59.088 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:57:59.088 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:57:59.089 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:57:59.089 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.089 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:57:59.089 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.217537+0000", 2021-02-10T04:57:59.089 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:57:59.089 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.089 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.089 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:57:59.089 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.217537+0000", 2021-02-10T04:57:59.089 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.5524000000000001e-05 2021-02-10T04:57:59.089 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.089 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.089 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:57:59.089 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.217572+0000", 2021-02-10T04:57:59.090 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1260999999999999e-05 2021-02-10T04:57:59.090 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.090 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.090 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:57:59.090 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.217584+0000", 2021-02-10T04:57:59.090 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.4684999999999999e-05 2021-02-10T04:57:59.090 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.090 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.090 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:57:59.090 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.217598+0000", 2021-02-10T04:57:59.090 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.6000000000000007e-06 2021-02-10T04:57:59.090 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.091 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.091 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:57:59.091 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.217607+0000", 2021-02-10T04:57:59.091 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.8575000000000002e-05 2021-02-10T04:57:59.091 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.091 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.091 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:57:59.091 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.217656+0000", 2021-02-10T04:57:59.091 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.002995756 2021-02-10T04:57:59.091 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.091 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.091 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:57:59.091 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.220651+0000", 2021-02-10T04:57:59.092 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031144010000000001 2021-02-10T04:57:59.092 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.092 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:57:59.092 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.092 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.092 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.092 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52017b30:::smithi08575397-43:20a)])", 2021-02-10T04:57:59.092 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:39.887305+0000", 2021-02-10T04:57:59.092 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 379.17093768500001, 2021-02-10T04:57:59.092 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003468512, 2021-02-10T04:57:59.092 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:57:59.092 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:57:59.093 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:57:59.093 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.093 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:57:59.093 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.887305+0000", 2021-02-10T04:57:59.093 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:57:59.093 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.093 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.093 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:57:59.093 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.887305+0000", 2021-02-10T04:57:59.093 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.1676e-05 2021-02-10T04:57:59.093 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.093 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.093 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:57:59.094 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.887337+0000", 2021-02-10T04:57:59.094 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.9230000000000008e-06 2021-02-10T04:57:59.094 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.094 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.094 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:57:59.094 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.887345+0000", 2021-02-10T04:57:59.094 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2656999999999999e-05 2021-02-10T04:57:59.094 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.094 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.094 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:57:59.094 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.887358+0000", 2021-02-10T04:57:59.094 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.0090000000000001e-06 2021-02-10T04:57:59.095 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.095 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.095 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:57:59.095 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.887365+0000", 2021-02-10T04:57:59.095 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.7717000000000003e-05 2021-02-10T04:57:59.095 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.095 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.095 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:57:59.095 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.887423+0000", 2021-02-10T04:57:59.095 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00335053 2021-02-10T04:57:59.095 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.095 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.095 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:57:59.096 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.890773+0000", 2021-02-10T04:57:59.096 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003468512 2021-02-10T04:57:59.096 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.096 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:57:59.096 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.096 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.096 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.096 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.3 1122/1119 [PushReplyOp(7:c765b4cc:::smithi08575397-30:309)])", 2021-02-10T04:57:59.096 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:39.889361+0000", 2021-02-10T04:57:59.096 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 379.16888197399999, 2021-02-10T04:57:59.096 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0033870189999999998, 2021-02-10T04:57:59.096 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:57:59.096 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:57:59.097 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:57:59.097 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.097 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:57:59.097 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.889361+0000", 2021-02-10T04:57:59.097 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:57:59.097 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.097 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.097 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:57:59.097 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.889361+0000", 2021-02-10T04:57:59.097 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.9168999999999999e-05 2021-02-10T04:57:59.097 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.097 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.098 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:57:59.098 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.889380+0000", 2021-02-10T04:57:59.098 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.5779999999999998e-06 2021-02-10T04:57:59.098 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.098 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.098 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:57:59.098 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.889385+0000", 2021-02-10T04:57:59.098 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.8790000000000003e-06 2021-02-10T04:57:59.098 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.098 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.098 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:57:59.098 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.889393+0000", 2021-02-10T04:57:59.098 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.972e-06 2021-02-10T04:57:59.099 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.099 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.099 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:57:59.099 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.889398+0000", 2021-02-10T04:57:59.099 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4211000000000001e-05 2021-02-10T04:57:59.099 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.099 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.099 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:57:59.099 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.889442+0000", 2021-02-10T04:57:59.099 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0033052099999999998 2021-02-10T04:57:59.099 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.099 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.099 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:57:59.100 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.892748+0000", 2021-02-10T04:57:59.100 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0033870189999999998 2021-02-10T04:57:59.100 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.100 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:57:59.100 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.100 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.100 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.100 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52cf5dd7:::smithi08575397-7:20a)])", 2021-02-10T04:57:59.100 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:39.897118+0000", 2021-02-10T04:57:59.100 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 379.16112406600001, 2021-02-10T04:57:59.100 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0045733759999999997, 2021-02-10T04:57:59.100 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:57:59.100 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:57:59.101 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:57:59.101 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.101 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:57:59.101 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.897118+0000", 2021-02-10T04:57:59.101 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:57:59.101 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.101 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.101 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:57:59.101 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.897118+0000", 2021-02-10T04:57:59.101 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.0298000000000001e-05 2021-02-10T04:57:59.101 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.101 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.101 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:57:59.101 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.897139+0000", 2021-02-10T04:57:59.102 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.7610000000000007e-06 2021-02-10T04:57:59.102 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.102 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.102 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:57:59.102 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.897147+0000", 2021-02-10T04:57:59.102 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2027e-05 2021-02-10T04:57:59.102 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.102 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.102 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:57:59.102 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.897159+0000", 2021-02-10T04:57:59.102 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.0929999999999999e-06 2021-02-10T04:57:59.102 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.102 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.103 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:57:59.103 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.897165+0000", 2021-02-10T04:57:59.103 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000100141 2021-02-10T04:57:59.103 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.103 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.103 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:57:59.103 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.897265+0000", 2021-02-10T04:57:59.103 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0044270560000000004 2021-02-10T04:57:59.103 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.103 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.103 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:57:59.103 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.901692+0000", 2021-02-10T04:57:59.103 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0045733759999999997 2021-02-10T04:57:59.103 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.104 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:57:59.104 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.104 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.104 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.104 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52017b30:::smithi08575397-43:2d9)])", 2021-02-10T04:57:59.104 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:39.946105+0000", 2021-02-10T04:57:59.104 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 379.112137469, 2021-02-10T04:57:59.104 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0035851820000000001, 2021-02-10T04:57:59.104 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:57:59.104 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:57:59.104 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:57:59.104 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.104 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:57:59.104 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946105+0000", 2021-02-10T04:57:59.105 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:57:59.105 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.105 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.105 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:57:59.105 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946105+0000", 2021-02-10T04:57:59.105 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.7836e-05 2021-02-10T04:57:59.105 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.105 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.105 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:57:59.105 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946123+0000", 2021-02-10T04:57:59.105 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.9430000000000003e-06 2021-02-10T04:57:59.105 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.106 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.106 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:57:59.106 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946130+0000", 2021-02-10T04:57:59.106 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.7319999999999993e-06 2021-02-10T04:57:59.106 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.106 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.106 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:57:59.106 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946140+0000", 2021-02-10T04:57:59.106 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.277e-06 2021-02-10T04:57:59.106 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.106 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.106 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:57:59.106 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946147+0000", 2021-02-10T04:57:59.107 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.4368000000000004e-05 2021-02-10T04:57:59.107 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.107 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.107 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:57:59.107 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946211+0000", 2021-02-10T04:57:59.107 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0034790260000000001 2021-02-10T04:57:59.107 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.107 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.107 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:57:59.107 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.949690+0000", 2021-02-10T04:57:59.107 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0035851820000000001 2021-02-10T04:57:59.107 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.107 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:57:59.108 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.108 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.108 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.108 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.3 1122/1119 [PushReplyOp(7:c1415b8e:::smithi08575397-38:35f)])", 2021-02-10T04:57:59.108 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:39.946563+0000", 2021-02-10T04:57:59.108 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 379.11167969899998, 2021-02-10T04:57:59.108 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031900539999999999, 2021-02-10T04:57:59.108 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:57:59.108 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:57:59.108 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:57:59.108 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.108 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:57:59.108 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946563+0000", 2021-02-10T04:57:59.108 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:57:59.109 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.109 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.109 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:57:59.109 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946563+0000", 2021-02-10T04:57:59.109 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.2059e-05 2021-02-10T04:57:59.109 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.109 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.109 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:57:59.109 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946585+0000", 2021-02-10T04:57:59.109 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9450000000000001e-06 2021-02-10T04:57:59.109 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.109 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.109 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:57:59.109 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946590+0000", 2021-02-10T04:57:59.110 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.9759999999999998e-06 2021-02-10T04:57:59.110 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.110 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.110 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:57:59.110 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946597+0000", 2021-02-10T04:57:59.110 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.9949999999999999e-06 2021-02-10T04:57:59.110 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.110 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.110 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:57:59.110 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946601+0000", 2021-02-10T04:57:59.110 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4035000000000001e-05 2021-02-10T04:57:59.110 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.110 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.111 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:57:59.111 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.946645+0000", 2021-02-10T04:57:59.111 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031080439999999999 2021-02-10T04:57:59.111 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.111 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.111 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:57:59.111 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.949753+0000", 2021-02-10T04:57:59.111 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031900539999999999 2021-02-10T04:57:59.111 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.111 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:57:59.111 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.111 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.111 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.111 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52017b30:::smithi08575397-43:2d9)])", 2021-02-10T04:57:59.112 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:39.951097+0000", 2021-02-10T04:57:59.112 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 379.10714542599999, 2021-02-10T04:57:59.112 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00327655, 2021-02-10T04:57:59.112 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:57:59.112 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:57:59.112 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:57:59.112 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.112 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:57:59.112 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.951097+0000", 2021-02-10T04:57:59.112 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:57:59.112 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.112 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.112 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:57:59.113 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.951097+0000", 2021-02-10T04:57:59.113 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.2909e-05 2021-02-10T04:57:59.113 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.113 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.113 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:57:59.113 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.951120+0000", 2021-02-10T04:57:59.113 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.2708999999999999e-05 2021-02-10T04:57:59.113 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.113 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.113 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:57:59.113 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.951143+0000", 2021-02-10T04:57:59.113 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.9159999999999996e-06 2021-02-10T04:57:59.113 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.114 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.114 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:57:59.114 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.951153+0000", 2021-02-10T04:57:59.114 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9230000000000001e-06 2021-02-10T04:57:59.114 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.114 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.114 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:57:59.114 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.951158+0000", 2021-02-10T04:57:59.114 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.9799000000000003e-05 2021-02-10T04:57:59.114 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.114 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.114 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:57:59.114 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.951197+0000", 2021-02-10T04:57:59.114 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031762940000000001 2021-02-10T04:57:59.115 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.115 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.115 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:57:59.115 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.954374+0000", 2021-02-10T04:57:59.115 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00327655 2021-02-10T04:57:59.115 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.115 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:57:59.115 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.115 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.115 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.115 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:531b0702:::smithi08575397-3:300)])", 2021-02-10T04:57:59.115 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:39.963275+0000", 2021-02-10T04:57:59.115 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 379.09496802299998, 2021-02-10T04:57:59.116 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003308152, 2021-02-10T04:57:59.116 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:57:59.116 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:57:59.116 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:57:59.116 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.116 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:57:59.116 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.963275+0000", 2021-02-10T04:57:59.116 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:57:59.116 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.116 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.116 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:57:59.116 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.963275+0000", 2021-02-10T04:57:59.116 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.0713e-05 2021-02-10T04:57:59.117 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.117 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.117 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:57:59.117 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.963295+0000", 2021-02-10T04:57:59.117 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.6899999999999997e-06 2021-02-10T04:57:59.117 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.117 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.117 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:57:59.117 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.963301+0000", 2021-02-10T04:57:59.117 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1399e-05 2021-02-10T04:57:59.117 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.117 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.117 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:57:59.117 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.963312+0000", 2021-02-10T04:57:59.118 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.3340000000000003e-06 2021-02-10T04:57:59.118 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.118 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.118 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:57:59.118 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.963317+0000", 2021-02-10T04:57:59.118 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.8989000000000002e-05 2021-02-10T04:57:59.118 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.118 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.118 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:57:59.118 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.963356+0000", 2021-02-10T04:57:59.118 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0032270269999999999 2021-02-10T04:57:59.118 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.118 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.119 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:57:59.119 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:39.966583+0000", 2021-02-10T04:57:59.119 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003308152 2021-02-10T04:57:59.119 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.119 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:57:59.119 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.119 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.119 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.119 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52017b30:::smithi08575397-43:372)])", 2021-02-10T04:57:59.119 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:40.124429+0000", 2021-02-10T04:57:59.119 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 378.933813112, 2021-02-10T04:57:59.119 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0033528249999999998, 2021-02-10T04:57:59.119 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:57:59.120 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:57:59.120 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:57:59.120 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.120 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:57:59.120 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.124429+0000", 2021-02-10T04:57:59.120 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:57:59.120 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.120 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.120 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:57:59.120 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.124429+0000", 2021-02-10T04:57:59.120 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.7447000000000001e-05 2021-02-10T04:57:59.120 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.120 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.121 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:57:59.121 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.124447+0000", 2021-02-10T04:57:59.121 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0030000000000002e-06 2021-02-10T04:57:59.121 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.121 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.121 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:57:59.121 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.124452+0000", 2021-02-10T04:57:59.121 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.2669999999999995e-06 2021-02-10T04:57:59.121 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.121 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.121 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:57:59.121 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.124461+0000", 2021-02-10T04:57:59.121 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.2230000000000001e-06 2021-02-10T04:57:59.122 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.122 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.122 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:57:59.122 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.124465+0000", 2021-02-10T04:57:59.122 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.5660000000000001e-05 2021-02-10T04:57:59.122 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.122 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.122 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:57:59.122 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.124501+0000", 2021-02-10T04:57:59.122 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003281225 2021-02-10T04:57:59.122 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.122 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.122 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:57:59.123 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.127782+0000", 2021-02-10T04:57:59.123 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0033528249999999998 2021-02-10T04:57:59.123 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.123 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:57:59.123 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.123 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.123 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.123 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52017b30:::smithi08575397-43:33a)])", 2021-02-10T04:57:59.123 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:40.158898+0000", 2021-02-10T04:57:59.123 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 378.899344289, 2021-02-10T04:57:59.123 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003157211, 2021-02-10T04:57:59.123 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:57:59.123 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:57:59.123 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:57:59.124 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.124 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:57:59.124 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.158898+0000", 2021-02-10T04:57:59.124 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:57:59.124 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.124 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.124 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:57:59.124 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.158898+0000", 2021-02-10T04:57:59.124 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.8318999999999999e-05 2021-02-10T04:57:59.124 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.124 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.124 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:57:59.124 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.158917+0000", 2021-02-10T04:57:59.125 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.2360000000000004e-06 2021-02-10T04:57:59.125 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.125 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.125 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:57:59.125 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.158922+0000", 2021-02-10T04:57:59.125 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.7189999999999996e-06 2021-02-10T04:57:59.125 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.125 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.125 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:57:59.125 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.158932+0000", 2021-02-10T04:57:59.125 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.5459999999999999e-06 2021-02-10T04:57:59.125 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.125 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.126 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:57:59.126 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.158937+0000", 2021-02-10T04:57:59.126 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.625e-05 2021-02-10T04:57:59.126 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.126 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.126 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:57:59.126 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.158973+0000", 2021-02-10T04:57:59.126 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003082141 2021-02-10T04:57:59.126 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.126 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.126 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:57:59.126 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.162055+0000", 2021-02-10T04:57:59.126 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003157211 2021-02-10T04:57:59.127 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.127 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:57:59.127 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.127 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.127 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.127 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52017b30:::smithi08575397-43:33a)])", 2021-02-10T04:57:59.127 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:40.163313+0000", 2021-02-10T04:57:59.127 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 378.89492971800001, 2021-02-10T04:57:59.127 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0032056989999999998, 2021-02-10T04:57:59.127 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:57:59.127 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:57:59.127 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:57:59.128 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.128 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:57:59.128 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.163313+0000", 2021-02-10T04:57:59.128 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:57:59.128 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.128 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.128 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:57:59.128 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.163313+0000", 2021-02-10T04:57:59.128 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.8493999999999999e-05 2021-02-10T04:57:59.128 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.128 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.128 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:57:59.128 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.163331+0000", 2021-02-10T04:57:59.129 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.1150000000000002e-06 2021-02-10T04:57:59.129 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.129 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.129 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:57:59.129 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.163336+0000", 2021-02-10T04:57:59.129 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.4429999999999992e-06 2021-02-10T04:57:59.129 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.129 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.129 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:57:59.129 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.163346+0000", 2021-02-10T04:57:59.129 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.767e-06 2021-02-10T04:57:59.129 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.129 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.129 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:57:59.130 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.163352+0000", 2021-02-10T04:57:59.130 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.5129000000000002e-05 2021-02-10T04:57:59.130 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.130 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.130 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:57:59.130 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.163387+0000", 2021-02-10T04:57:59.130 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031317509999999999 2021-02-10T04:57:59.130 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.130 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.130 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:57:59.130 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.166519+0000", 2021-02-10T04:57:59.130 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0032056989999999998 2021-02-10T04:57:59.130 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.131 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:57:59.131 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.131 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.131 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.131 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52cf5dd7:::smithi08575397-7:33e)])", 2021-02-10T04:57:59.131 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:40.173135+0000", 2021-02-10T04:57:59.131 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 378.88510761800001, 2021-02-10T04:57:59.131 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003144015, 2021-02-10T04:57:59.131 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:57:59.131 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:57:59.131 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:57:59.131 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.131 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:57:59.132 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.173135+0000", 2021-02-10T04:57:59.132 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:57:59.132 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.132 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.132 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:57:59.132 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.173135+0000", 2021-02-10T04:57:59.132 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.7708999999999999e-05 2021-02-10T04:57:59.132 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.132 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.132 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:57:59.132 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.173153+0000", 2021-02-10T04:57:59.132 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.921e-06 2021-02-10T04:57:59.132 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.132 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.133 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:57:59.133 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.173158+0000", 2021-02-10T04:57:59.133 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.3769999999999996e-06 2021-02-10T04:57:59.133 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.133 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.133 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:57:59.133 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.173166+0000", 2021-02-10T04:57:59.133 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.4409999999999999e-06 2021-02-10T04:57:59.133 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.133 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.133 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:57:59.133 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.173175+0000", 2021-02-10T04:57:59.133 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.6347999999999998e-05 2021-02-10T04:57:59.133 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.134 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.134 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:57:59.134 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.173212+0000", 2021-02-10T04:57:59.134 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0030672189999999999 2021-02-10T04:57:59.134 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.134 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.134 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:57:59.134 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.176279+0000", 2021-02-10T04:57:59.134 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.003144015 2021-02-10T04:57:59.134 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.134 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:57:59.135 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.135 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.135 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.135 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52cf5dd7:::smithi08575397-7:33e)])", 2021-02-10T04:57:59.135 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:40.180475+0000", 2021-02-10T04:57:59.135 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 378.877767775, 2021-02-10T04:57:59.135 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031979669999999999, 2021-02-10T04:57:59.135 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:57:59.135 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:57:59.135 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:57:59.136 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.136 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:57:59.136 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.180475+0000", 2021-02-10T04:57:59.136 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:57:59.136 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.136 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.136 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:57:59.136 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.180475+0000", 2021-02-10T04:57:59.136 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.9508e-05 2021-02-10T04:57:59.136 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.136 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.136 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:57:59.136 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.180494+0000", 2021-02-10T04:57:59.137 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.7320000000000003e-06 2021-02-10T04:57:59.137 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.137 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.137 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:57:59.137 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.180500+0000", 2021-02-10T04:57:59.137 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.4909999999999994e-06 2021-02-10T04:57:59.137 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.137 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.137 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:57:59.137 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.180510+0000", 2021-02-10T04:57:59.137 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.3980000000000004e-06 2021-02-10T04:57:59.137 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.137 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.138 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:57:59.138 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.180516+0000", 2021-02-10T04:57:59.138 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.1493999999999999e-05 2021-02-10T04:57:59.138 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.138 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.138 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:57:59.138 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.180557+0000", 2021-02-10T04:57:59.138 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031153439999999999 2021-02-10T04:57:59.138 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.138 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.138 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:57:59.138 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.183673+0000", 2021-02-10T04:57:59.138 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031979669999999999 2021-02-10T04:57:59.139 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.139 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:57:59.139 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.139 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.139 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.139 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(7.a 1122/1119 [PushReplyOp(7:52017b30:::smithi08575397-43:349)])", 2021-02-10T04:57:59.139 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:51:40.214713+0000", 2021-02-10T04:57:59.139 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 378.84352932100001, 2021-02-10T04:57:59.139 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031374720000000001, 2021-02-10T04:57:59.139 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:57:59.139 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T04:57:59.139 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:57:59.139 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.140 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:57:59.140 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.214713+0000", 2021-02-10T04:57:59.140 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:57:59.140 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.140 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.140 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:57:59.140 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.214713+0000", 2021-02-10T04:57:59.140 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.8167000000000001e-05 2021-02-10T04:57:59.140 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.140 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.140 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:57:59.140 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.214731+0000", 2021-02-10T04:57:59.140 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0479999999999998e-06 2021-02-10T04:57:59.140 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.141 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.141 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:57:59.141 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.214736+0000", 2021-02-10T04:57:59.141 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.4749999999999999e-06 2021-02-10T04:57:59.141 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.141 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.141 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:57:59.141 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.214746+0000", 2021-02-10T04:57:59.141 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.7610000000000004e-06 2021-02-10T04:57:59.141 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.141 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.141 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:57:59.141 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.214752+0000", 2021-02-10T04:57:59.142 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.621e-05 2021-02-10T04:57:59.142 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.142 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.142 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:57:59.142 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.214798+0000", 2021-02-10T04:57:59.142 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0030528109999999999 2021-02-10T04:57:59.142 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.142 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.142 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:57:59.142 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:51:40.217851+0000", 2021-02-10T04:57:59.142 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031374720000000001 2021-02-10T04:57:59.142 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.142 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:57:59.143 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.143 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.143 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.143 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDScrubReserve(2.e REQUEST e1187)", 2021-02-10T04:57:59.143 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:53:41.711530+0000", 2021-02-10T04:57:59.143 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 257.34671280800001, 2021-02-10T04:57:59.143 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031529919999999999, 2021-02-10T04:57:59.143 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T04:57:59.143 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T04:57:59.143 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T04:57:59.143 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.143 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T04:57:59.143 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:41.711530+0000", 2021-02-10T04:57:59.144 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T04:57:59.144 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.144 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.144 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T04:57:59.144 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:41.711530+0000", 2021-02-10T04:57:59.144 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.5733e-05 2021-02-10T04:57:59.144 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.144 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.144 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T04:57:59.144 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:41.711545+0000", 2021-02-10T04:57:59.144 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.1470000000000003e-06 2021-02-10T04:57:59.144 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.144 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.144 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T04:57:59.145 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:41.711550+0000", 2021-02-10T04:57:59.145 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.8300000000000001e-06 2021-02-10T04:57:59.145 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.145 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.145 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T04:57:59.145 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:41.711555+0000", 2021-02-10T04:57:59.145 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.4179999999999996e-06 2021-02-10T04:57:59.145 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.145 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.145 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T04:57:59.145 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:41.711561+0000", 2021-02-10T04:57:59.145 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.8992000000000001e-05 2021-02-10T04:57:59.145 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.146 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.146 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:57:59.146 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:41.711610+0000", 2021-02-10T04:57:59.146 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.3280000000000006e-06 2021-02-10T04:57:59.146 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.146 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.146 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op must wait for map", 2021-02-10T04:57:59.146 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:41.711618+0000", 2021-02-10T04:57:59.146 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0030070909999999999 2021-02-10T04:57:59.146 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.146 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.146 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T04:57:59.146 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:41.714625+0000", 2021-02-10T04:57:59.146 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.292e-05 2021-02-10T04:57:59.147 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.147 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.147 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T04:57:59.147 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:41.714638+0000", 2021-02-10T04:57:59.147 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4533000000000002e-05 2021-02-10T04:57:59.147 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:57:59.147 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:57:59.147 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T04:57:59.147 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:53:41.714683+0000", 2021-02-10T04:57:59.147 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031529919999999999 2021-02-10T04:57:59.147 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.147 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:57:59.147 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.147 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:57:59.148 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:57:59.148 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:57:59.148 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:57:59.148 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:57:59.149 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:57:59.149 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:57:59.155 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:57:59.175 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:57:59.196 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:57:59.197 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:57:59.198 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:57:59.198 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:57:59.198 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:57:59.198 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:57:59.198 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:57:59.198 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:57:59.198 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:57:59.444 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset nodeep-scrub 2021-02-10T04:57:59.712 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:57:59.885 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.2 to 1 (10000) 2021-02-10T04:57:59.894 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:57:59 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:57:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:59 smithi164 conmon[29288]: cluster 2021-02-10T04:57:58.540146 2021-02-10T04:57:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:59 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1465 : cluster [DBG] pgmap v2390: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:57:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:59 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:57:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:59 smithi164 conmon[29288]: 02-10T04: 2021-02-10T04:57:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:59 smithi164 conmon[29288]: 57:58.606668+0000 mon.a (mon.0) 2852 : cluster [DBG] 2021-02-10T04:57:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:59 smithi164 conmon[29288]: osdmap e1311: 8 total, 8 up, 8 in 2021-02-10T04:57:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:59 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:57:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:59 smithi164 conmon[29288]: 02-10T04:57:59.019027+0000 mon.a ( 2021-02-10T04:57:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:59 smithi164 conmon[29288]: mon.0) 2853 : audit [DBG] 2021-02-10T04:57:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:59 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/3746919878' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:57:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:59 smithi164 conmon[29288]: audit 2021-02-10T04:57:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:59 smithi164 conmon[29288]: 2021-02-10T04:57:59.239350+0000 2021-02-10T04:57:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:59 smithi164 conmon[29288]: mon.c (mon.1) 147 : audit [INF] 2021-02-10T04:57:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:59 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/3902923721' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-02-10T04:57:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:59 smithi164 conmon[29288]: audit 2021-02-10T04:57:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:59 smithi164 conmon[29288]: 2021-02-10T04:57:59.239748+0000 mon.a 2021-02-10T04:57:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:57:59 smithi164 conmon[29288]: (mon.0) 2854 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-02-10T04:57:59.908 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:59 smithi085 conmon[29319]: cluster 2021-02-10T04:57:58.540146+0000 mgr.y 2021-02-10T04:57:59.908 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:59 smithi085 conmon[29319]: (mgr.14140) 1465 : cluster [DBG] pgmap v2390: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:57:59.908 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:59 smithi085 conmon[29319]: cluster 2021-02-10T04:57:58.606668+0000 mon.a (mon.0) 2852 : cluster [DBG] osdmap e1311: 8 total, 8 up, 8 in 2021-02-10T04:57:59.908 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:59 smithi085 conmon[29319]: audit 2021-02-10T04:57:59.019027+0000 mon.a (mon.0) 2853 : audit [DBG] from='client.? 172.21.15.85:0/3746919878' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:57:59.908 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:59 smithi085 conmon[29319]: audit 2021-02-10T04:57:59.239350+0000 mon.c (mon.1) 147 : audit [INF] from='client.? 172.21.15.85:0/3902923721' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-02-10T04:57:59.908 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:57:59 smithi085 conmon[29319]: audit 2021-02-10T04:57:59.239748+0000 mon.a (mon.0) 2854 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-02-10T04:57:59.908 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:59 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:57:59.908 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:59 smithi085 conmon[35961]: 57:58.540146+0000 mgr.y (mgr.14140) 1465 : cluster [DBG] pgmap v2390: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:57:59.909 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:59 smithi085 conmon[35961]: cluster 2021-02-10T04:57:58.606668+0000 mon.a (mon.0) 2852 : cluster [DBG] osdmap e1311: 8 total, 8 up, 8 in 2021-02-10T04:57:59.909 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:59 smithi085 conmon[35961]: audit 2021-02-10T04:57:59.019027+0000 mon.a (mon.0) 2853 : audit [DBG] from='client.? 172.21.15.85:0/3746919878' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:57:59.909 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:59 smithi085 conmon[35961]: audit 2021-02-10T04:57:59.239350+0000 mon.c (mon.1) 147 : audit [INF] from='client.? 172.21.15.85:0/3902923721' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-02-10T04:57:59.909 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:57:59 smithi085 conmon[35961]: audit 2021-02-10T04:57:59.239748+0000 mon.a (mon.0) 2854 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-02-10T04:58:00.879 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:00 smithi085 conmon[35961]: audit 2021-02-10T04:57:59.465981 2021-02-10T04:58:00.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:00 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1466 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:58:00.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:00 smithi085 conmon[35961]: cluster 2021-02-10T04:57:59.673239+0000 osd.3 (osd.3) 42 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:58:00.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:00 smithi085 conmon[35961]: cluster 2021-02-10T04:57:59.677235+0000 osd.3 (osd.3) 43 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:58:00.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:00 smithi085 conmon[35961]: audit 2021-02-10T04:57:59.876638+0000 mon.a (mon.0) 2855 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2021-02-10T04:58:00.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:00 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:58:00.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:00 smithi085 conmon[35961]: 02-10T04:57:59.876701+0000 mon.a (mon.0 2021-02-10T04:58:00.880 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:00 smithi085 conmon[35961]: ) 2856 : cluster [DBG] osdmap e1312: 8 total, 8 up, 8 in 2021-02-10T04:58:00.880 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:00 smithi085 conmon[29319]: audit 2021-02-10T04:57:59.465981 2021-02-10T04:58:00.880 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:00 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 1466 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:58:00.880 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:00 smithi085 conmon[29319]: cluster 2021-02-10T04:57:59.673239+0000 osd.3 (osd.3) 42 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:58:00.880 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:00 smithi085 conmon[29319]: cluster 2021-02-10T04:57:59.677235+0000 osd.3 (osd.3) 43 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:58:00.880 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:00 smithi085 conmon[29319]: audit 2021-02-10T04:57:59.876638+0000 mon.a (mon.0) 2855 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2021-02-10T04:58:00.880 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:00 smithi085 conmon[29319]: cluster 2021-02-10T04:57:59.876701+0000 mon.a (mon.0) 2856 : cluster [DBG] osdmap e1312: 8 total, 8 up, 8 in 2021-02-10T04:58:01.093 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.4 dump_ops_in_flight 2021-02-10T04:58:01.129 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2021-02-10T04:58:01.130 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:58:01.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:00 smithi164 conmon[29288]: audit 2021-02-10T04:57:59.465981+0000 2021-02-10T04:58:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:00 smithi164 conmon[29288]: mgr.y (mgr.14140) 1466 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:58:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:00 smithi164 conmon[29288]: cluster 2021-02-10T04:57:59 2021-02-10T04:58:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:00 smithi164 conmon[29288]: .673239+0000 osd.3 (osd.3) 42 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:58:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:00 smithi164 conmon[29288]: cluster 2021-02-10T04:58:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:00 smithi164 conmon[29288]: 2021-02-10T04:57:59. 2021-02-10T04:58:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:00 smithi164 conmon[29288]: 677235+0000 osd.3 (osd.3) 43 : cluster 2021-02-10T04:58:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:00 smithi164 conmon[29288]: [DBG] purged_snaps scrub ok 2021-02-10T04:58:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:00 smithi164 conmon[29288]: audit 2021-02-10T04:58:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:00 smithi164 conmon[29288]: 2021-02-10T04:57:59. 2021-02-10T04:58:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:00 smithi164 conmon[29288]: 876638+0000 mon.a (mon.0) 2855 : audit 2021-02-10T04:58:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:00 smithi164 conmon[29288]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2021-02-10T04:58:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:00 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:58:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:00 smithi164 conmon[29288]: 02-10T04:57:59.876701+0000 2021-02-10T04:58:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:00 smithi164 conmon[29288]: mon.a (mon.0) 2856 : cluster [DBG] 2021-02-10T04:58:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:00 smithi164 conmon[29288]: osdmap e1312: 8 total, 8 up, 8 in 2021-02-10T04:58:01.408 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:58:01.820 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:58:01.881 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:01 smithi164 conmon[29288]: cluster 2021-02-10T04:58:00.540556+0000 2021-02-10T04:58:01.881 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:01 smithi164 conmon[29288]: mgr.y (mgr.14140) 1467 : cluster [DBG] pgmap v2393: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:58:01.881 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:01 smithi164 conmon[29288]: cluster 2021-02-10T04:58:01.247094+0000 osd.1 (osd.1) 511 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:58:01.881 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:01 smithi164 conmon[29288]: cluster 2021-02-10T04:58:01.249357+0000 osd.1 (osd.1) 512 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:58:02.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:01 smithi085 conmon[29319]: cluster 2021-02-10T04:58:00.540556+0000 mgr.y (mgr.14140) 1467 : cluster [DBG] pgmap v2393: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:58:02.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:01 smithi085 conmon[29319]: cluster 2021-02-10T04:58:01.247094+0000 osd.1 (osd.1) 511 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:58:02.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:01 smithi085 conmon[29319]: cluster 2021-02-10T04:58:01.249357+0000 osd.1 (osd.1) 512 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:58:02.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:01 smithi085 conmon[35961]: cluster 2021-02-10T04:58:00.540556+0000 mgr.y (mgr.14140) 1467 : cluster [DBG] pgmap v2393: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:58:02.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:01 smithi085 conmon[35961]: cluster 2021-02-10T04:58:01.247094+0000 osd.1 (osd.1) 511 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:58:02.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:01 smithi085 conmon[35961]: cluster 2021-02-10T04:58:01.249357+0000 osd.1 (osd.1) 512 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:58:02.567 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:58:02 smithi085 conmon[49106]: debug 2021-02-10T04:58:02.567 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:58:02 smithi085 conmon[49106]: 2021-02-10T04:58:02.412+0000 7f778ce21700 -1 osd.2 pg_epoch: 1313 pg[5.8( empty local-lis/les=1227/1228 n=0 ec=57/57 lis/c=1227/1227 les/c/f=1228/1228/0 sis=1227) [2,0,3] r=0 lpr=1227 crt=0'0 mlcod 0'0 active+clean] scrubber pg(5.8) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x564aa5a42000) 2021-02-10T04:58:02.568 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:58:02 smithi085 conmon[49106]: 2021-02-10T04:58:02.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:02 smithi085 conmon[35961]: cluster 2021-02-10T04:58:01.890720+0000 mon.a (mon. 2021-02-10T04:58:02.895 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:02 smithi085 conmon[35961]: 0) 2857 : cluster [DBG] osdmap e1313: 8 total, 8 up, 8 in 2021-02-10T04:58:03.142 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:58:03.167 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:02 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:58:03.167 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:02 smithi085 conmon[29319]: 04:58:01.890720+0000 mon.a (mon.0) 2857 : cluster [DBG] osdmap e1313: 8 total, 8 up, 8 in 2021-02-10T04:58:03.341 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:58:03.393 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:58:03 smithi164 conmon[37263]: debug 2021-02-10T04:58:03.128+0000 7f1c6b9eb700 -1 osd.5 pg_epoch: 1313 pg[6.a( empty local-lis/les=1227/1228 n=0 ec=59/59 lis/c=1227/1227 les/c/f=1228/1228/0 sis=1227) [5,6,0] r=0 lpr=1227 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x5566566b8b00) 2021-02-10T04:58:03.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:02 smithi164 conmon[29288]: cluster 2021-02-10T04:58:01.890720+0000 mon.a ( 2021-02-10T04:58:03.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:02 smithi164 conmon[29288]: mon.0) 2857 : cluster [DBG] osdmap e1313: 8 total, 8 up, 8 in 2021-02-10T04:58:04.044 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:03 smithi085 conmon[29319]: cluster 2021-02-10T04:58:02.541066+0000 mgr.y (mgr.14140) 1468 : cluster [DBG] pgmap v2395: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:58:04.045 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:03 smithi085 conmon[35961]: cluster 2021-02-10T04:58:02.541066+0000 mgr.y (mgr.14140) 1468 : cluster [DBG] pgmap v2395: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:58:04.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:03 smithi164 conmon[29288]: cluster 2021-02-10T04:58:04.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:03 smithi164 conmon[29288]: 2021-02-10T04:58:04.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:03 smithi164 conmon[29288]: 2021-02-10T04:58:02.541066+0000 2021-02-10T04:58:04.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:03 smithi164 conmon[29288]: mgr.y (mgr.14140) 1468 : cluster [DBG] pgmap v2395: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:58:04.410 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:58:04.413 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:58:04.908 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is unset 2021-02-10T04:58:05.166 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:04 smithi164 conmon[29288]: audit 2021-02-10T04:58:05.167 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:04 smithi164 conmon[29288]: 2021-02-10T04:58:05.167 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:04 smithi164 conmon[29288]: 2021 2021-02-10T04:58:05.167 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:04 smithi164 conmon[29288]: - 2021-02-10T04:58:05.167 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:04 smithi164 conmon[29288]: 02 2021-02-10T04:58:05.167 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:04 smithi164 conmon[29288]: - 2021-02-10T04:58:05.167 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:04 smithi164 conmon[29288]: 10 2021-02-10T04:58:05.167 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:04 smithi164 conmon[29288]: T 2021-02-10T04:58:05.167 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:04 smithi164 conmon[29288]: 04 2021-02-10T04:58:05.168 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:04 smithi164 conmon[29288]: : 2021-02-10T04:58:05.168 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:04 smithi164 conmon[29288]: 58 2021-02-10T04:58:05.168 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:04 smithi164 conmon[29288]: :04.143056+0000 2021-02-10T04:58:05.168 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:04 smithi164 conmon[29288]: mon.a (mon.0) 2858 : audit [INF] from='client.? 172.21.15.85:0/1199555575' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:58:05.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:04 smithi085 conmon[29319]: audit 2021-02-10T04:58:04.143056+0000 mon.a (mon.0) 2858 : 2021-02-10T04:58:05.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:04 smithi085 conmon[29319]: audit [INF] from='client.? 172.21.15.85:0/1199555575' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:58:05.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:04 smithi085 conmon[35961]: audit 2021-02-10T04:58:05.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:04 smithi085 conmon[35961]: 2021-02-10T04:58: 2021-02-10T04:58:05.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:04 smithi085 conmon[35961]: 04.143056+0000 mon.a (mon.0) 2858 : audit [INF] from='client.? 172.21.15.85:0/1199555575' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:58:05.643 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:58:05 smithi164 conmon[37263]: debug 2021-02-10T04:58:05.164+0000 7f1c679e3700 -1 osd.5 pg_epoch: 1314 pg[6.a( empty local-lis/les=1227/1228 n=0 ec=59/59 lis/c=1227/1227 les/c/f=1228/1228/0 sis=1227) [5,6,0] r=0 lpr=1227 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x5566437b6160) 2021-02-10T04:58:05.750 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[49106]: debug 2021-02-10T04:58:05.491+0000 7f7788e19700 -1 osd.2 pg_epoch: 1314 pg[5.8( empty local-lis/les=1227/1228 n=0 ec=57/57 lis/c=1227/1227 les/c/f=1228/1228/0 sis=1227) [2,0,3] r=0 lpr=1227 crt=0'0 mlcod 0'0 active+clean] scrubber pg(5.8) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x564aa5ae4580) 2021-02-10T04:58:06.027 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[29319]: audit 2021-02-10T04:58:04.387294+0000 mgr.y (mgr.14140) 2021-02-10T04:58:06.027 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[29319]: 1469 : audit [DBG] from='client.16404 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:58:06.027 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[29319]: cluster 2021-02-10T04:58:04.541675+0000 mgr.y (mgr.14140) 1470 : cluster [DBG] pgmap v2396: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:58:06.027 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:58:06.027 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[29319]: -02-10T04:58:04.895541+0000 mon.a (mon.0) 2859 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:58:06.027 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:58:06.027 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[29319]: -02-10T04:58:04.895573 2021-02-10T04:58:06.028 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2860 : cluster [INF] 2021-02-10T04:58:06.028 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[29319]: Cluster is now healthy 2021-02-10T04:58:06.028 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[29319]: audit 2021-02-10T04:58 2021-02-10T04:58:06.028 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[29319]: :04.904379+0000 mon.a (mon.0) 2861 2021-02-10T04:58:06.028 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[29319]: : audit [INF] from='client.? 172.21.15.85:0/1199555575' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:58:06.028 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:58:06.028 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[29319]: 04:58:04.904451+0000 mon.a ( 2021-02-10T04:58:06.028 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[29319]: mon.0) 2862 : cluster [DBG] 2021-02-10T04:58:06.028 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[29319]: osdmap e1314: 8 total, 8 up, 8 in 2021-02-10T04:58:06.028 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[35961]: audit 2021-02-10T04:58:04.387294 2021-02-10T04:58:06.028 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1469 : audit [DBG] from='client.16404 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:58:06.029 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[35961]: cluster 2021-02-10T04:58:06.029 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[35961]: 2021-02-10T04:58:04.541675+0000 mgr.y (mgr.14140) 1470 : cluster [DBG] 2021-02-10T04:58:06.029 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[35961]: pgmap v2396: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:58:06.029 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:58:06.029 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[35961]: -10T04:58:04.895541+0000 mon.a (mon.0) 2859 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:58:06.029 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[35961]: cluster 2021-02-10T04:58:04. 2021-02-10T04:58:06.029 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[35961]: 895573+0000 mon.a (mon.0) 2860 : cluster [INF] Cluster is now healthy 2021-02-10T04:58:06.029 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[35961]: audit 2021-02-10T04:58 2021-02-10T04:58:06.029 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[35961]: :04.904379+0000 mon.a (mon.0) 2861 : audit [INF] 2021-02-10T04:58:06.029 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/1199555575' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:58:06.029 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[35961]: cluster 2021-02-10T04:58:06.029 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[35961]: 2021-02-10T04:58:04.904451+0000 mon.a (mon. 2021-02-10T04:58:06.030 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:05 smithi085 conmon[35961]: 0) 2862 : cluster [DBG] osdmap e1314: 8 total, 8 up, 8 in 2021-02-10T04:58:06.030 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":2395,"stamp":"2021-02-10T04:58:02.540690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":972357089,"num_objects":748,"num_object_clones":337,"num_object_copies":2244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":748,"num_whiteouts":10,"num_read":7975,"num_read_kb":4562374,"num_write":4841,"num_write_kb":1247750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":887,"num_bytes_recovered":311980434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35650,"ondisk_log_size":35650,"up":594,"acting":594,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":596,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2310752,"kb_used_data":1984480,"kb_used_omap":692,"kb_used_meta":325515,"kb_avail":747650464,"statfs":{"total":767960285184,"available":765594075136,"internally_reserved":0,"allocated":2032107520,"data_stored":2600178972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":709615,"internal_metadata":333327377},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001467"},"pg_stats":[{"pgid":"6.29","version":"0'0","reported_seq":"1168","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.699416+0000","last_change":"2021-02-10T04:55:14.633086+0000","last_active":"2021-02-10T04:57:58.699416+0000","last_peered":"2021-02-10T04:57:58.699416+0000","last_clean":"2021-02-10T04:57:58.699416+0000","last_became_active":"2021-02-10T04:55:14.632670+0000","last_became_peered":"2021-02-10T04:55:14.632670+0000","last_unstale":"2021-02-10T04:57:58.699416+0000","last_undegraded":"2021-02-10T04:57:58.699416+0000","last_fullsized":"2021-02-10T04:57:58.699416+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:24.121684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:24.121684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":"208","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.605870+0000","last_change":"2021-02-10T04:52:30.133105+0000","last_active":"2021-02-10T04:57:58.605870+0000","last_peered":"2021-02-10T04:57:58.605870+0000","last_clean":"2021-02-10T04:57:58.605870+0000","last_became_active":"2021-02-10T04:51:36.864926+0000","last_became_peered":"2021-02-10T04:51:36.864926+0000","last_unstale":"2021-02-10T04:57:58.605870+0000","last_undegraded":"2021-02-10T04:57:58.605870+0000","last_fullsized":"2021-02-10T04:57:58.605870+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.133043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.133043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"1191'4","reported_seq":"313","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.699989+0000","last_change":"2021-02-10T04:53:46.736164+0000","last_active":"2021-02-10T04:57:58.699989+0000","last_peered":"2021-02-10T04:57:58.699989+0000","last_clean":"2021-02-10T04:57:58.699989+0000","last_became_active":"2021-02-10T04:53:46.735711+0000","last_became_peered":"2021-02-10T04:53:46.735711+0000","last_unstale":"2021-02-10T04:57:58.699989+0000","last_undegraded":"2021-02-10T04:57:58.699989+0000","last_fullsized":"2021-02-10T04:57:58.699989+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'3","last_scrub_stamp":"2021-02-10T04:51:01.185820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.185820+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"1261'84","reported_seq":"1442","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.606277+0000","last_change":"2021-02-10T04:55:39.594594+0000","last_active":"2021-02-10T04:57:53.606277+0000","last_peered":"2021-02-10T04:57:53.606277+0000","last_clean":"2021-02-10T04:57:53.606277+0000","last_became_active":"2021-02-10T04:55:39.594275+0000","last_became_peered":"2021-02-10T04:55:39.594275+0000","last_unstale":"2021-02-10T04:57:53.606277+0000","last_undegraded":"2021-02-10T04:57:53.606277+0000","last_fullsized":"2021-02-10T04:57:53.606277+0000","mapping_epoch":55,"log_start":"1132'66","ondisk_log_start":"1132'66","created":55,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":0,"last_scrub":"1132'66","last_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_deep_scrub":"1132'66","last_deep_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.430399+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1616,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":186,"num_read_kb":150,"num_write":107,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"217","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.608167+0000","last_change":"2021-02-10T04:53:41.722609+0000","last_active":"2021-02-10T04:57:58.608167+0000","last_peered":"2021-02-10T04:57:58.608167+0000","last_clean":"2021-02-10T04:57:58.608167+0000","last_became_active":"2021-02-10T04:51:36.875000+0000","last_became_peered":"2021-02-10T04:51:36.875000+0000","last_unstale":"2021-02-10T04:57:58.608167+0000","last_undegraded":"2021-02-10T04:57:58.608167+0000","last_fullsized":"2021-02-10T04:57:58.608167+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:53:41.722516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:41.722516+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"1342","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:57.105837+0000","last_change":"2021-02-10T04:57:57.106707+0000","last_active":"2021-02-10T04:57:57.105837+0000","last_peered":"2021-02-10T04:57:57.105837+0000","last_clean":"2021-02-10T04:57:57.105837+0000","last_became_active":"2021-02-10T04:55:14.630535+0000","last_became_peered":"2021-02-10T04:55:14.630535+0000","last_unstale":"2021-02-10T04:57:57.105837+0000","last_undegraded":"2021-02-10T04:57:57.105837+0000","last_fullsized":"2021-02-10T04:57:57.105837+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:48.408748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:50:48.408748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"1003'1689","reported_seq":"4729","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.690860+0000","last_change":"2021-02-10T04:53:38.170873+0000","last_active":"2021-02-10T04:57:53.690860+0000","last_peered":"2021-02-10T04:57:53.690860+0000","last_clean":"2021-02-10T04:57:53.690860+0000","last_became_active":"2021-02-10T04:52:26.030105+0000","last_became_peered":"2021-02-10T04:52:26.030105+0000","last_unstale":"2021-02-10T04:57:53.690860+0000","last_undegraded":"2021-02-10T04:57:53.690860+0000","last_fullsized":"2021-02-10T04:57:53.690860+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:53:38.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.170802+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":99361981,"num_objects":37,"num_object_clones":33,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"62'5","reported_seq":"312","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:56.831151+0000","last_change":"2021-02-10T04:54:39.615784+0000","last_active":"2021-02-10T04:57:56.831151+0000","last_peered":"2021-02-10T04:57:56.831151+0000","last_clean":"2021-02-10T04:57:56.831151+0000","last_became_active":"2021-02-10T04:54:39.560009+0000","last_became_peered":"2021-02-10T04:54:39.560009+0000","last_unstale":"2021-02-10T04:57:56.831151+0000","last_undegraded":"2021-02-10T04:57:56.831151+0000","last_fullsized":"2021-02-10T04:57:56.831151+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":349,"num_read_kb":348,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"1261'28","reported_seq":"1322","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.875612+0000","last_change":"2021-02-10T04:55:21.669679+0000","last_active":"2021-02-10T04:57:59.875612+0000","last_peered":"2021-02-10T04:57:59.875612+0000","last_clean":"2021-02-10T04:57:59.875612+0000","last_became_active":"2021-02-10T04:55:21.669219+0000","last_became_peered":"2021-02-10T04:55:21.669219+0000","last_unstale":"2021-02-10T04:57:59.875612+0000","last_undegraded":"2021-02-10T04:57:59.875612+0000","last_fullsized":"2021-02-10T04:57:59.875612+0000","mapping_epoch":1228,"log_start":"1061'20","ondisk_log_start":"1061'20","created":55,"last_epoch_clean":1233,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'32","last_scrub_stamp":"2021-02-10T04:51:51.382040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.382040+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1158,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":105,"num_read_kb":81,"num_write":58,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2052,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"961","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.691131+0000","last_change":"2021-02-10T04:53:34.179015+0000","last_active":"2021-02-10T04:57:53.691131+0000","last_peered":"2021-02-10T04:57:53.691131+0000","last_clean":"2021-02-10T04:57:53.691131+0000","last_became_active":"2021-02-10T04:51:49.502576+0000","last_became_peered":"2021-02-10T04:51:49.502576+0000","last_unstale":"2021-02-10T04:57:53.691131+0000","last_undegraded":"2021-02-10T04:57:53.691131+0000","last_fullsized":"2021-02-10T04:57:53.691131+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:34.178949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.178949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"208","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.608846+0000","last_change":"2021-02-10T04:51:53.038893+0000","last_active":"2021-02-10T04:57:58.608846+0000","last_peered":"2021-02-10T04:57:58.608846+0000","last_clean":"2021-02-10T04:57:58.608846+0000","last_became_active":"2021-02-10T04:51:36.855507+0000","last_became_peered":"2021-02-10T04:51:36.855507+0000","last_unstale":"2021-02-10T04:57:58.608846+0000","last_undegraded":"2021-02-10T04:57:58.608846+0000","last_fullsized":"2021-02-10T04:57:58.608846+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:53.038834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:53.038834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"1231","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.874944+0000","last_change":"2021-02-10T04:55:28.097406+0000","last_active":"2021-02-10T04:57:59.874944+0000","last_peered":"2021-02-10T04:57:59.874944+0000","last_clean":"2021-02-10T04:57:59.874944+0000","last_became_active":"2021-02-10T04:55:14.629603+0000","last_became_peered":"2021-02-10T04:55:14.629603+0000","last_unstale":"2021-02-10T04:57:59.874944+0000","last_undegraded":"2021-02-10T04:57:59.874944+0000","last_fullsized":"2021-02-10T04:57:59.874944+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:09.922789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:09.922789+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"1168'82","reported_seq":"244","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.606095+0000","last_change":"2021-02-10T04:53:23.326363+0000","last_active":"2021-02-10T04:57:58.606095+0000","last_peered":"2021-02-10T04:57:58.606095+0000","last_clean":"2021-02-10T04:57:58.606095+0000","last_became_active":"2021-02-10T04:51:36.859215+0000","last_became_peered":"2021-02-10T04:51:36.859215+0000","last_unstale":"2021-02-10T04:57:58.606095+0000","last_undegraded":"2021-02-10T04:57:58.606095+0000","last_fullsized":"2021-02-10T04:57:58.606095+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1150'76","last_scrub_stamp":"2021-02-10T04:53:00.964499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:00.964499+0000","log_size":82,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3848803,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"1261'69","reported_seq":"1450","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:55.860822+0000","last_change":"2021-02-10T04:55:25.576927+0000","last_active":"2021-02-10T04:57:55.860822+0000","last_peered":"2021-02-10T04:57:55.860822+0000","last_clean":"2021-02-10T04:57:55.860822+0000","last_became_active":"2021-02-10T04:55:25.576633+0000","last_became_peered":"2021-02-10T04:55:25.576633+0000","last_unstale":"2021-02-10T04:57:55.860822+0000","last_undegraded":"2021-02-10T04:57:55.860822+0000","last_fullsized":"2021-02-10T04:57:55.860822+0000","mapping_epoch":55,"log_start":"1112'47","ondisk_log_start":"1112'47","created":55,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'61","last_scrub_stamp":"2021-02-10T04:53:36.836202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:36.836202+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1466,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":208,"num_read_kb":172,"num_write":122,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"1228'6","reported_seq":"306","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.880893+0000","last_change":"2021-02-10T04:55:14.605868+0000","last_active":"2021-02-10T04:57:59.880893+0000","last_peered":"2021-02-10T04:57:59.880893+0000","last_clean":"2021-02-10T04:57:59.880893+0000","last_became_active":"2021-02-10T04:55:14.605314+0000","last_became_peered":"2021-02-10T04:55:14.605314+0000","last_unstale":"2021-02-10T04:57:59.880893+0000","last_undegraded":"2021-02-10T04:57:59.880893+0000","last_fullsized":"2021-02-10T04:57:59.880893+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'4","last_scrub_stamp":"2021-02-10T04:52:01.121380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.121380+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"223","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.606510+0000","last_change":"2021-02-10T04:55:22.573715+0000","last_active":"2021-02-10T04:57:58.606510+0000","last_peered":"2021-02-10T04:57:58.606510+0000","last_clean":"2021-02-10T04:57:58.606510+0000","last_became_active":"2021-02-10T04:55:14.636895+0000","last_became_peered":"2021-02-10T04:55:14.636895+0000","last_unstale":"2021-02-10T04:57:58.606510+0000","last_undegraded":"2021-02-10T04:57:58.606510+0000","last_fullsized":"2021-02-10T04:57:58.606510+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:55:22.573621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:55:22.573621+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"608","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.699355+0000","last_change":"2021-02-10T04:53:46.733236+0000","last_active":"2021-02-10T04:57:58.699355+0000","last_peered":"2021-02-10T04:57:58.699355+0000","last_clean":"2021-02-10T04:57:58.699355+0000","last_became_active":"2021-02-10T04:53:46.732890+0000","last_became_peered":"2021-02-10T04:53:46.732890+0000","last_unstale":"2021-02-10T04:57:58.699355+0000","last_undegraded":"2021-02-10T04:57:58.699355+0000","last_fullsized":"2021-02-10T04:57:58.699355+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:35.170157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:35.170157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"1003'739","reported_seq":"3910","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.875051+0000","last_change":"2021-02-10T04:55:28.091164+0000","last_active":"2021-02-10T04:57:59.875051+0000","last_peered":"2021-02-10T04:57:59.875051+0000","last_clean":"2021-02-10T04:57:59.875051+0000","last_became_active":"2021-02-10T04:55:14.648601+0000","last_became_peered":"2021-02-10T04:55:14.648601+0000","last_unstale":"2021-02-10T04:57:59.875051+0000","last_undegraded":"2021-02-10T04:57:59.875051+0000","last_fullsized":"2021-02-10T04:57:59.875051+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:52:27.602006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:27.602006+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24059888,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":129,"num_read_kb":109544,"num_write":102,"num_write_kb":46173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"1287","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:55.859265+0000","last_change":"2021-02-10T04:54:39.559980+0000","last_active":"2021-02-10T04:57:55.859265+0000","last_peered":"2021-02-10T04:57:55.859265+0000","last_clean":"2021-02-10T04:57:55.859265+0000","last_became_active":"2021-02-10T04:54:39.559443+0000","last_became_peered":"2021-02-10T04:54:39.559443+0000","last_unstale":"2021-02-10T04:57:55.859265+0000","last_undegraded":"2021-02-10T04:57:55.859265+0000","last_fullsized":"2021-02-10T04:57:55.859265+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"1261'64","reported_seq":"1497","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.205577+0000","last_change":"2021-02-10T04:53:37.108747+0000","last_active":"2021-02-10T04:58:00.205577+0000","last_peered":"2021-02-10T04:58:00.205577+0000","last_clean":"2021-02-10T04:58:00.205577+0000","last_became_active":"2021-02-10T04:53:35.551311+0000","last_became_peered":"2021-02-10T04:53:35.551311+0000","last_unstale":"2021-02-10T04:58:00.205577+0000","last_undegraded":"2021-02-10T04:58:00.205577+0000","last_fullsized":"2021-02-10T04:58:00.205577+0000","mapping_epoch":1118,"log_start":"1066'44","ondisk_log_start":"1066'44","created":55,"last_epoch_clean":1181,"parent":"0.0","parent_split_bits":0,"last_scrub":"1066'44","last_scrub_stamp":"2021-02-10T04:53:37.108677+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:53:37.108677+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1578,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":180,"num_read_kb":144,"num_write":106,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":2312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"1207","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.204220+0000","last_change":"2021-02-10T04:53:39.126356+0000","last_active":"2021-02-10T04:58:00.204220+0000","last_peered":"2021-02-10T04:58:00.204220+0000","last_clean":"2021-02-10T04:58:00.204220+0000","last_became_active":"2021-02-10T04:29:07.145616+0000","last_became_peered":"2021-02-10T04:29:07.145616+0000","last_unstale":"2021-02-10T04:58:00.204220+0000","last_undegraded":"2021-02-10T04:58:00.204220+0000","last_fullsized":"2021-02-10T04:58:00.204220+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:39.126261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:39.126261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"536","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.690519+0000","last_change":"2021-02-10T04:55:14.611192+0000","last_active":"2021-02-10T04:57:53.690519+0000","last_peered":"2021-02-10T04:57:53.690519+0000","last_clean":"2021-02-10T04:57:53.690519+0000","last_became_active":"2021-02-10T04:55:14.610821+0000","last_became_peered":"2021-02-10T04:55:14.610821+0000","last_unstale":"2021-02-10T04:57:53.690519+0000","last_undegraded":"2021-02-10T04:57:53.690519+0000","last_fullsized":"2021-02-10T04:57:53.690519+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:26.311958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.311958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"240","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.607894+0000","last_change":"2021-02-10T04:55:03.526061+0000","last_active":"2021-02-10T04:57:58.607894+0000","last_peered":"2021-02-10T04:57:58.607894+0000","last_clean":"2021-02-10T04:57:58.607894+0000","last_became_active":"2021-02-10T04:51:36.855744+0000","last_became_peered":"2021-02-10T04:51:36.855744+0000","last_unstale":"2021-02-10T04:57:58.607894+0000","last_undegraded":"2021-02-10T04:57:58.607894+0000","last_fullsized":"2021-02-10T04:57:58.607894+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:02.162851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:13.727241+0000","last_clean_scrub_stamp":"2021-02-10T04:52:02.162851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"1063'654","reported_seq":"4487","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.608085+0000","last_change":"2021-02-10T04:55:14.637619+0000","last_active":"2021-02-10T04:57:58.608085+0000","last_peered":"2021-02-10T04:57:58.608085+0000","last_clean":"2021-02-10T04:57:58.608085+0000","last_became_active":"2021-02-10T04:55:14.636752+0000","last_became_peered":"2021-02-10T04:55:14.636752+0000","last_unstale":"2021-02-10T04:57:58.608085+0000","last_undegraded":"2021-02-10T04:57:58.608085+0000","last_fullsized":"2021-02-10T04:57:58.608085+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:51:57.121838+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.121838+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24700630,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":171,"num_read_kb":182272,"num_write":111,"num_write_kb":33727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":26349023,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"1255","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.606303+0000","last_change":"2021-02-10T04:51:50.433056+0000","last_active":"2021-02-10T04:57:53.606303+0000","last_peered":"2021-02-10T04:57:53.606303+0000","last_clean":"2021-02-10T04:57:53.606303+0000","last_became_active":"2021-02-10T04:51:36.843165+0000","last_became_peered":"2021-02-10T04:51:36.843165+0000","last_unstale":"2021-02-10T04:57:53.606303+0000","last_undegraded":"2021-02-10T04:57:53.606303+0000","last_fullsized":"2021-02-10T04:57:53.606303+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.432990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.432990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"1279","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.203739+0000","last_change":"2021-02-10T04:55:14.638290+0000","last_active":"2021-02-10T04:58:00.203739+0000","last_peered":"2021-02-10T04:58:00.203739+0000","last_clean":"2021-02-10T04:58:00.203739+0000","last_became_active":"2021-02-10T04:55:14.637927+0000","last_became_peered":"2021-02-10T04:55:14.637927+0000","last_unstale":"2021-02-10T04:58:00.203739+0000","last_undegraded":"2021-02-10T04:58:00.203739+0000","last_fullsized":"2021-02-10T04:58:00.203739+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"1203","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.699569+0000","last_change":"2021-02-10T04:53:46.727888+0000","last_active":"2021-02-10T04:57:58.699569+0000","last_peered":"2021-02-10T04:57:58.699569+0000","last_clean":"2021-02-10T04:57:58.699569+0000","last_became_active":"2021-02-10T04:53:46.727670+0000","last_became_peered":"2021-02-10T04:53:46.727670+0000","last_unstale":"2021-02-10T04:57:58.699569+0000","last_undegraded":"2021-02-10T04:57:58.699569+0000","last_fullsized":"2021-02-10T04:57:58.699569+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:41.041553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.041553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"1261'76","reported_seq":"1421","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.203781+0000","last_change":"2021-02-10T04:53:42.071173+0000","last_active":"2021-02-10T04:58:00.203781+0000","last_peered":"2021-02-10T04:58:00.203781+0000","last_clean":"2021-02-10T04:58:00.203781+0000","last_became_active":"2021-02-10T04:53:39.679752+0000","last_became_peered":"2021-02-10T04:53:39.679752+0000","last_unstale":"2021-02-10T04:58:00.203781+0000","last_undegraded":"2021-02-10T04:58:00.203781+0000","last_fullsized":"2021-02-10T04:58:00.203781+0000","mapping_epoch":1111,"log_start":"1049'39","ondisk_log_start":"1049'39","created":55,"last_epoch_clean":1185,"parent":"0.0","parent_split_bits":0,"last_scrub":"1187'54","last_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_deep_scrub":"1187'54","last_deep_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_clean_scrub_stamp":"2021-02-10T04:53:42.071103+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":147,"num_read_kb":127,"num_write":95,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"946","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.690181+0000","last_change":"2021-02-10T04:51:52.350764+0000","last_active":"2021-02-10T04:57:53.690181+0000","last_peered":"2021-02-10T04:57:53.690181+0000","last_clean":"2021-02-10T04:57:53.690181+0000","last_became_active":"2021-02-10T04:51:49.567784+0000","last_became_peered":"2021-02-10T04:51:49.567784+0000","last_unstale":"2021-02-10T04:57:53.690181+0000","last_undegraded":"2021-02-10T04:57:53.690181+0000","last_fullsized":"2021-02-10T04:57:53.690181+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:52.350706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.350706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"1119'4","reported_seq":"290","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.607207+0000","last_change":"2021-02-10T04:52:27.126670+0000","last_active":"2021-02-10T04:57:58.607207+0000","last_peered":"2021-02-10T04:57:58.607207+0000","last_clean":"2021-02-10T04:57:58.607207+0000","last_became_active":"2021-02-10T04:51:36.853647+0000","last_became_peered":"2021-02-10T04:51:36.853647+0000","last_unstale":"2021-02-10T04:57:58.607207+0000","last_undegraded":"2021-02-10T04:57:58.607207+0000","last_fullsized":"2021-02-10T04:57:58.607207+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'4","last_scrub_stamp":"2021-02-10T04:52:27.126599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:27.126599+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"1298","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.204946+0000","last_change":"2021-02-10T04:52:25.745660+0000","last_active":"2021-02-10T04:58:00.204946+0000","last_peered":"2021-02-10T04:58:00.204946+0000","last_clean":"2021-02-10T04:58:00.204946+0000","last_became_active":"2021-02-10T04:51:36.849503+0000","last_became_peered":"2021-02-10T04:51:36.849503+0000","last_unstale":"2021-02-10T04:58:00.204946+0000","last_undegraded":"2021-02-10T04:58:00.204946+0000","last_fullsized":"2021-02-10T04:58:00.204946+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:25.745577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.745577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"1004'1093","reported_seq":"6493","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.875919+0000","last_change":"2021-02-10T04:55:28.097530+0000","last_active":"2021-02-10T04:57:59.875919+0000","last_peered":"2021-02-10T04:57:59.875919+0000","last_clean":"2021-02-10T04:57:59.875919+0000","last_became_active":"2021-02-10T04:55:14.627630+0000","last_became_peered":"2021-02-10T04:55:14.627630+0000","last_unstale":"2021-02-10T04:57:59.875919+0000","last_undegraded":"2021-02-10T04:57:59.875919+0000","last_fullsized":"2021-02-10T04:57:59.875919+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39434696,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":279,"num_read_kb":311058,"num_write":185,"num_write_kb":66790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"62'5","reported_seq":"111","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.202667+0000","last_change":"2021-02-10T04:54:39.559473+0000","last_active":"2021-02-10T04:58:00.202667+0000","last_peered":"2021-02-10T04:58:00.202667+0000","last_clean":"2021-02-10T04:58:00.202667+0000","last_became_active":"2021-02-10T04:54:39.559048+0000","last_became_peered":"2021-02-10T04:54:39.559048+0000","last_unstale":"2021-02-10T04:58:00.202667+0000","last_undegraded":"2021-02-10T04:58:00.202667+0000","last_fullsized":"2021-02-10T04:58:00.202667+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":149,"num_read_kb":148,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"1264","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.606776+0000","last_change":"2021-02-10T04:55:03.230261+0000","last_active":"2021-02-10T04:57:53.606776+0000","last_peered":"2021-02-10T04:57:53.606776+0000","last_clean":"2021-02-10T04:57:53.606776+0000","last_became_active":"2021-02-10T04:51:36.841582+0000","last_became_peered":"2021-02-10T04:51:36.841582+0000","last_unstale":"2021-02-10T04:57:53.606776+0000","last_undegraded":"2021-02-10T04:57:53.606776+0000","last_fullsized":"2021-02-10T04:57:53.606776+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:12.389272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:12.389272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"1261'71","reported_seq":"1012","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.691146+0000","last_change":"2021-02-10T04:53:27.560299+0000","last_active":"2021-02-10T04:57:53.691146+0000","last_peered":"2021-02-10T04:57:53.691146+0000","last_clean":"2021-02-10T04:57:53.691146+0000","last_became_active":"2021-02-10T04:53:27.559886+0000","last_became_peered":"2021-02-10T04:53:27.559886+0000","last_unstale":"2021-02-10T04:57:53.691146+0000","last_undegraded":"2021-02-10T04:57:53.691146+0000","last_fullsized":"2021-02-10T04:57:53.691146+0000","mapping_epoch":1129,"log_start":"1049'51","ondisk_log_start":"1049'51","created":55,"last_epoch_clean":1173,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'41","last_scrub_stamp":"2021-02-10T04:51:58.353660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:58.353660+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1472,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":169,"num_read_kb":136,"num_write":102,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"969","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.689433+0000","last_change":"2021-02-10T04:53:11.271557+0000","last_active":"2021-02-10T04:57:53.689433+0000","last_peered":"2021-02-10T04:57:53.689433+0000","last_clean":"2021-02-10T04:57:53.689433+0000","last_became_active":"2021-02-10T04:51:49.567883+0000","last_became_peered":"2021-02-10T04:51:49.567883+0000","last_unstale":"2021-02-10T04:57:53.689433+0000","last_undegraded":"2021-02-10T04:57:53.689433+0000","last_fullsized":"2021-02-10T04:57:53.689433+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:11.271459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.271459+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"1229'5","reported_seq":"1687","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.205175+0000","last_change":"2021-02-10T04:55:14.638050+0000","last_active":"2021-02-10T04:58:00.205175+0000","last_peered":"2021-02-10T04:58:00.205175+0000","last_clean":"2021-02-10T04:58:00.205175+0000","last_became_active":"2021-02-10T04:55:14.637570+0000","last_became_peered":"2021-02-10T04:55:14.637570+0000","last_unstale":"2021-02-10T04:58:00.205175+0000","last_undegraded":"2021-02-10T04:58:00.205175+0000","last_fullsized":"2021-02-10T04:58:00.205175+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:51:47.855283+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.855283+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"1299","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.609155+0000","last_change":"2021-02-10T04:55:22.183630+0000","last_active":"2021-02-10T04:57:53.609155+0000","last_peered":"2021-02-10T04:57:53.609155+0000","last_clean":"2021-02-10T04:57:53.609155+0000","last_became_active":"2021-02-10T04:55:14.656661+0000","last_became_peered":"2021-02-10T04:55:14.656661+0000","last_unstale":"2021-02-10T04:57:53.609155+0000","last_undegraded":"2021-02-10T04:57:53.609155+0000","last_fullsized":"2021-02-10T04:57:53.609155+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:55:22.183566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:55:22.183566+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"1002'1090","reported_seq":"5504","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:55.861352+0000","last_change":"2021-02-10T04:52:26.030198+0000","last_active":"2021-02-10T04:57:55.861352+0000","last_peered":"2021-02-10T04:57:55.861352+0000","last_clean":"2021-02-10T04:57:55.861352+0000","last_became_active":"2021-02-10T04:52:26.028182+0000","last_became_peered":"2021-02-10T04:52:26.028182+0000","last_unstale":"2021-02-10T04:57:55.861352+0000","last_undegraded":"2021-02-10T04:57:55.861352+0000","last_fullsized":"2021-02-10T04:57:55.861352+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:51:08.220472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.220472+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24410646,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":245,"num_read_kb":276168,"num_write":175,"num_write_kb":64960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"1278","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.202896+0000","last_change":"2021-02-10T04:54:39.555379+0000","last_active":"2021-02-10T04:58:00.202896+0000","last_peered":"2021-02-10T04:58:00.202896+0000","last_clean":"2021-02-10T04:58:00.202896+0000","last_became_active":"2021-02-10T04:54:39.554976+0000","last_became_peered":"2021-02-10T04:54:39.554976+0000","last_unstale":"2021-02-10T04:58:00.202896+0000","last_undegraded":"2021-02-10T04:58:00.202896+0000","last_fullsized":"2021-02-10T04:58:00.202896+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"1261'65","reported_seq":"1410","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.204685+0000","last_change":"2021-02-10T04:53:31.624862+0000","last_active":"2021-02-10T04:58:00.204685+0000","last_peered":"2021-02-10T04:58:00.204685+0000","last_clean":"2021-02-10T04:58:00.204685+0000","last_became_active":"2021-02-10T04:53:31.624318+0000","last_became_peered":"2021-02-10T04:53:31.624318+0000","last_unstale":"2021-02-10T04:58:00.204685+0000","last_undegraded":"2021-02-10T04:58:00.204685+0000","last_fullsized":"2021-02-10T04:58:00.204685+0000","mapping_epoch":55,"log_start":"1049'47","ondisk_log_start":"1049'47","created":55,"last_epoch_clean":1177,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'47","last_scrub_stamp":"2021-02-10T04:53:09.002368+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.002368+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1672,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":169,"num_read_kb":135,"num_write":98,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":934,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"248","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.699439+0000","last_change":"2021-02-10T04:55:14.611324+0000","last_active":"2021-02-10T04:57:58.699439+0000","last_peered":"2021-02-10T04:57:58.699439+0000","last_clean":"2021-02-10T04:57:58.699439+0000","last_became_active":"2021-02-10T04:55:14.610986+0000","last_became_peered":"2021-02-10T04:55:14.610986+0000","last_unstale":"2021-02-10T04:57:58.699439+0000","last_undegraded":"2021-02-10T04:57:58.699439+0000","last_fullsized":"2021-02-10T04:57:58.699439+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:08.358300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:50:08.358300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"1286","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.609324+0000","last_change":"2021-02-10T04:51:57.436961+0000","last_active":"2021-02-10T04:57:53.609324+0000","last_peered":"2021-02-10T04:57:53.609324+0000","last_clean":"2021-02-10T04:57:53.609324+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:57:53.609324+0000","last_undegraded":"2021-02-10T04:57:53.609324+0000","last_fullsized":"2021-02-10T04:57:53.609324+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.436862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.436862+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"1244","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.203597+0000","last_change":"2021-02-10T04:53:02.112612+0000","last_active":"2021-02-10T04:58:00.203597+0000","last_peered":"2021-02-10T04:58:00.203597+0000","last_clean":"2021-02-10T04:58:00.203597+0000","last_became_active":"2021-02-10T04:50:59.545501+0000","last_became_peered":"2021-02-10T04:50:59.545501+0000","last_unstale":"2021-02-10T04:58:00.203597+0000","last_undegraded":"2021-02-10T04:58:00.203597+0000","last_fullsized":"2021-02-10T04:58:00.203597+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:02.112533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:02.112533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"1001'1165","reported_seq":"4483","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:55.863100+0000","last_change":"2021-02-10T04:55:14.653651+0000","last_active":"2021-02-10T04:57:55.863100+0000","last_peered":"2021-02-10T04:57:55.863100+0000","last_clean":"2021-02-10T04:57:55.863100+0000","last_became_active":"2021-02-10T04:55:14.652235+0000","last_became_peered":"2021-02-10T04:55:14.652235+0000","last_unstale":"2021-02-10T04:57:55.863100+0000","last_undegraded":"2021-02-10T04:57:55.863100+0000","last_fullsized":"2021-02-10T04:57:55.863100+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29601845,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":286,"num_read_kb":299529,"num_write":200,"num_write_kb":89130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"94","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.875236+0000","last_change":"2021-02-10T04:55:15.590113+0000","last_active":"2021-02-10T04:57:59.875236+0000","last_peered":"2021-02-10T04:57:59.875236+0000","last_clean":"2021-02-10T04:57:59.875236+0000","last_became_active":"2021-02-10T04:55:15.589751+0000","last_became_peered":"2021-02-10T04:55:15.589751+0000","last_unstale":"2021-02-10T04:57:59.875236+0000","last_undegraded":"2021-02-10T04:57:59.875236+0000","last_fullsized":"2021-02-10T04:57:59.875236+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"1113'5","reported_seq":"1573","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.202884+0000","last_change":"2021-02-10T04:52:34.047474+0000","last_active":"2021-02-10T04:58:00.202884+0000","last_peered":"2021-02-10T04:58:00.202884+0000","last_clean":"2021-02-10T04:58:00.202884+0000","last_became_active":"2021-02-10T04:50:59.569578+0000","last_became_peered":"2021-02-10T04:50:59.569578+0000","last_unstale":"2021-02-10T04:58:00.202884+0000","last_undegraded":"2021-02-10T04:58:00.202884+0000","last_fullsized":"2021-02-10T04:58:00.202884+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1113'5","last_scrub_stamp":"2021-02-10T04:52:34.047358+0000","last_deep_scrub":"158'3","last_deep_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_clean_scrub_stamp":"2021-02-10T04:52:34.047358+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"1261'68","reported_seq":"1395","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.203186+0000","last_change":"2021-02-10T04:51:48.872937+0000","last_active":"2021-02-10T04:58:00.203186+0000","last_peered":"2021-02-10T04:58:00.203186+0000","last_clean":"2021-02-10T04:58:00.203186+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:58:00.203186+0000","last_undegraded":"2021-02-10T04:58:00.203186+0000","last_fullsized":"2021-02-10T04:58:00.203186+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:51:48.872839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.872839+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":114,"num_read_kb":94,"num_write":68,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"1361","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:55.862707+0000","last_change":"2021-02-10T04:53:05.984747+0000","last_active":"2021-02-10T04:57:55.862707+0000","last_peered":"2021-02-10T04:57:55.862707+0000","last_clean":"2021-02-10T04:57:55.862707+0000","last_became_active":"2021-02-10T04:51:36.869931+0000","last_became_peered":"2021-02-10T04:51:36.869931+0000","last_unstale":"2021-02-10T04:57:55.862707+0000","last_undegraded":"2021-02-10T04:57:55.862707+0000","last_fullsized":"2021-02-10T04:57:55.862707+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:05.984659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:05.984659+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"1165","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.699518+0000","last_change":"2021-02-10T04:53:46.731709+0000","last_active":"2021-02-10T04:57:58.699518+0000","last_peered":"2021-02-10T04:57:58.699518+0000","last_clean":"2021-02-10T04:57:58.699518+0000","last_became_active":"2021-02-10T04:53:46.731340+0000","last_became_peered":"2021-02-10T04:53:46.731340+0000","last_unstale":"2021-02-10T04:57:58.699518+0000","last_undegraded":"2021-02-10T04:57:58.699518+0000","last_fullsized":"2021-02-10T04:57:58.699518+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:09.005421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.005421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"998'1009","reported_seq":"4803","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.203801+0000","last_change":"2021-02-10T04:52:26.037227+0000","last_active":"2021-02-10T04:58:00.203801+0000","last_peered":"2021-02-10T04:58:00.203801+0000","last_clean":"2021-02-10T04:58:00.203801+0000","last_became_active":"2021-02-10T04:52:26.036723+0000","last_became_peered":"2021-02-10T04:52:26.036723+0000","last_unstale":"2021-02-10T04:58:00.203801+0000","last_undegraded":"2021-02-10T04:58:00.203801+0000","last_fullsized":"2021-02-10T04:58:00.203801+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12355206,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":119,"num_read_kb":124701,"num_write":77,"num_write_kb":25052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4145029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"1183","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.203995+0000","last_change":"2021-02-10T04:53:33.305896+0000","last_active":"2021-02-10T04:58:00.203995+0000","last_peered":"2021-02-10T04:58:00.203995+0000","last_clean":"2021-02-10T04:58:00.203995+0000","last_became_active":"2021-02-10T04:50:59.570854+0000","last_became_peered":"2021-02-10T04:50:59.570854+0000","last_unstale":"2021-02-10T04:58:00.203995+0000","last_undegraded":"2021-02-10T04:58:00.203995+0000","last_fullsized":"2021-02-10T04:58:00.203995+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:33.305827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:33.305827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"1284","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.204132+0000","last_change":"2021-02-10T04:55:15.669988+0000","last_active":"2021-02-10T04:58:00.204132+0000","last_peered":"2021-02-10T04:58:00.204132+0000","last_clean":"2021-02-10T04:58:00.204132+0000","last_became_active":"2021-02-10T04:55:15.669642+0000","last_became_peered":"2021-02-10T04:55:15.669642+0000","last_unstale":"2021-02-10T04:58:00.204132+0000","last_undegraded":"2021-02-10T04:58:00.204132+0000","last_fullsized":"2021-02-10T04:58:00.204132+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"1297","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.203489+0000","last_change":"2021-02-10T04:51:50.895471+0000","last_active":"2021-02-10T04:58:00.203489+0000","last_peered":"2021-02-10T04:58:00.203489+0000","last_clean":"2021-02-10T04:58:00.203489+0000","last_became_active":"2021-02-10T04:50:59.527553+0000","last_became_peered":"2021-02-10T04:50:59.527553+0000","last_unstale":"2021-02-10T04:58:00.203489+0000","last_undegraded":"2021-02-10T04:58:00.203489+0000","last_fullsized":"2021-02-10T04:58:00.203489+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.895370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.895370+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"1261'84","reported_seq":"1428","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.203235+0000","last_change":"2021-02-10T04:55:14.603109+0000","last_active":"2021-02-10T04:58:00.203235+0000","last_peered":"2021-02-10T04:58:00.203235+0000","last_clean":"2021-02-10T04:58:00.203235+0000","last_became_active":"2021-02-10T04:55:14.602669+0000","last_became_peered":"2021-02-10T04:55:14.602669+0000","last_unstale":"2021-02-10T04:58:00.203235+0000","last_undegraded":"2021-02-10T04:58:00.203235+0000","last_fullsized":"2021-02-10T04:58:00.203235+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'72","last_scrub_stamp":"2021-02-10T04:46:30.515026+0000","last_deep_scrub":"152'48","last_deep_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_clean_scrub_stamp":"2021-02-10T04:46:30.515026+0000","log_size":84,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":123,"num_read_kb":101,"num_write":72,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"563","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.690922+0000","last_change":"2021-02-10T04:55:03.834050+0000","last_active":"2021-02-10T04:57:53.690922+0000","last_peered":"2021-02-10T04:57:53.690922+0000","last_clean":"2021-02-10T04:57:53.690922+0000","last_became_active":"2021-02-10T04:51:49.503801+0000","last_became_peered":"2021-02-10T04:51:49.503801+0000","last_unstale":"2021-02-10T04:57:53.690922+0000","last_undegraded":"2021-02-10T04:57:53.690922+0000","last_fullsized":"2021-02-10T04:57:53.690922+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:51:48.274960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.274960+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"1322","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.204813+0000","last_change":"2021-02-10T04:53:13.099288+0000","last_active":"2021-02-10T04:58:00.204813+0000","last_peered":"2021-02-10T04:58:00.204813+0000","last_clean":"2021-02-10T04:58:00.204813+0000","last_became_active":"2021-02-10T04:51:36.848614+0000","last_became_peered":"2021-02-10T04:51:36.848614+0000","last_unstale":"2021-02-10T04:58:00.204813+0000","last_undegraded":"2021-02-10T04:58:00.204813+0000","last_fullsized":"2021-02-10T04:58:00.204813+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:13.099228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:13.099228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"1002'1657","reported_seq":"5373","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.203447+0000","last_change":"2021-02-10T04:52:26.036806+0000","last_active":"2021-02-10T04:58:00.203447+0000","last_peered":"2021-02-10T04:58:00.203447+0000","last_clean":"2021-02-10T04:58:00.203447+0000","last_became_active":"2021-02-10T04:52:26.032363+0000","last_became_peered":"2021-02-10T04:52:26.032363+0000","last_unstale":"2021-02-10T04:58:00.203447+0000","last_undegraded":"2021-02-10T04:58:00.203447+0000","last_fullsized":"2021-02-10T04:58:00.203447+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21396533,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":212,"num_read_kb":218836,"num_write":159,"num_write_kb":59398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":22326014,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"1241","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.204975+0000","last_change":"2021-02-10T04:51:51.911945+0000","last_active":"2021-02-10T04:58:00.204975+0000","last_peered":"2021-02-10T04:58:00.204975+0000","last_clean":"2021-02-10T04:58:00.204975+0000","last_became_active":"2021-02-10T04:51:36.850015+0000","last_became_peered":"2021-02-10T04:51:36.850015+0000","last_unstale":"2021-02-10T04:58:00.204975+0000","last_undegraded":"2021-02-10T04:58:00.204975+0000","last_fullsized":"2021-02-10T04:58:00.204975+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.911851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.911851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"62'5","reported_seq":"1142","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.689895+0000","last_change":"2021-02-10T04:54:39.568253+0000","last_active":"2021-02-10T04:57:53.689895+0000","last_peered":"2021-02-10T04:57:53.689895+0000","last_clean":"2021-02-10T04:57:53.689895+0000","last_became_active":"2021-02-10T04:54:39.567642+0000","last_became_peered":"2021-02-10T04:54:39.567642+0000","last_unstale":"2021-02-10T04:57:53.689895+0000","last_undegraded":"2021-02-10T04:57:53.689895+0000","last_fullsized":"2021-02-10T04:57:53.689895+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":98,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":150,"num_read_kb":148,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"1368","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:55.860901+0000","last_change":"2021-02-10T04:55:14.634577+0000","last_active":"2021-02-10T04:57:55.860901+0000","last_peered":"2021-02-10T04:57:55.860901+0000","last_clean":"2021-02-10T04:57:55.860901+0000","last_became_active":"2021-02-10T04:55:14.634259+0000","last_became_peered":"2021-02-10T04:55:14.634259+0000","last_unstale":"2021-02-10T04:57:55.860901+0000","last_undegraded":"2021-02-10T04:57:55.860901+0000","last_fullsized":"2021-02-10T04:57:55.860901+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:58.142526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:58.142526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"1261'68","reported_seq":"1321","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.875437+0000","last_change":"2021-02-10T04:55:28.079376+0000","last_active":"2021-02-10T04:57:59.875437+0000","last_peered":"2021-02-10T04:57:59.875437+0000","last_clean":"2021-02-10T04:57:59.875437+0000","last_became_active":"2021-02-10T04:55:14.654609+0000","last_became_peered":"2021-02-10T04:55:14.654609+0000","last_unstale":"2021-02-10T04:57:59.875437+0000","last_undegraded":"2021-02-10T04:57:59.875437+0000","last_fullsized":"2021-02-10T04:57:59.875437+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1135'58","last_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_deep_scrub":"1135'58","last_deep_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.483067+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":102,"num_read_kb":83,"num_write":60,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"246","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.700245+0000","last_change":"2021-02-10T04:55:14.657913+0000","last_active":"2021-02-10T04:57:58.700245+0000","last_peered":"2021-02-10T04:57:58.700245+0000","last_clean":"2021-02-10T04:57:58.700245+0000","last_became_active":"2021-02-10T04:55:14.657153+0000","last_became_peered":"2021-02-10T04:55:14.657153+0000","last_unstale":"2021-02-10T04:57:58.700245+0000","last_undegraded":"2021-02-10T04:57:58.700245+0000","last_fullsized":"2021-02-10T04:57:58.700245+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:50:46.434629+0000","last_deep_scrub":"53'2","last_deep_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_clean_scrub_stamp":"2021-02-10T04:50:46.434629+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"986","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.689557+0000","last_change":"2021-02-10T04:52:38.278980+0000","last_active":"2021-02-10T04:57:53.689557+0000","last_peered":"2021-02-10T04:57:53.689557+0000","last_clean":"2021-02-10T04:57:53.689557+0000","last_became_active":"2021-02-10T04:51:49.496237+0000","last_became_peered":"2021-02-10T04:51:49.496237+0000","last_unstale":"2021-02-10T04:57:53.689557+0000","last_undegraded":"2021-02-10T04:57:53.689557+0000","last_fullsized":"2021-02-10T04:57:53.689557+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:38.278910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:52:38.278910+0000","last_clean_scrub_stamp":"2021-02-10T04:52:38.278910+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"1000'1071","reported_seq":"4860","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.606602+0000","last_change":"2021-02-10T04:53:34.407875+0000","last_active":"2021-02-10T04:57:53.606602+0000","last_peered":"2021-02-10T04:57:53.606602+0000","last_clean":"2021-02-10T04:57:53.606602+0000","last_became_active":"2021-02-10T04:52:26.031050+0000","last_became_peered":"2021-02-10T04:52:26.031050+0000","last_unstale":"2021-02-10T04:57:53.606602+0000","last_undegraded":"2021-02-10T04:57:53.606602+0000","last_fullsized":"2021-02-10T04:57:53.606602+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:34.407784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.407784+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30173843,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"1334","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:55.862266+0000","last_change":"2021-02-10T04:55:14.637570+0000","last_active":"2021-02-10T04:57:55.862266+0000","last_peered":"2021-02-10T04:57:55.862266+0000","last_clean":"2021-02-10T04:57:55.862266+0000","last_became_active":"2021-02-10T04:55:14.637203+0000","last_became_peered":"2021-02-10T04:55:14.637203+0000","last_unstale":"2021-02-10T04:57:55.862266+0000","last_undegraded":"2021-02-10T04:57:55.862266+0000","last_fullsized":"2021-02-10T04:57:55.862266+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_clean_scrub_stamp":"2021-02-10T04:51:00.195518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"1296","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.609396+0000","last_change":"2021-02-10T04:55:14.661285+0000","last_active":"2021-02-10T04:57:53.609396+0000","last_peered":"2021-02-10T04:57:53.609396+0000","last_clean":"2021-02-10T04:57:53.609396+0000","last_became_active":"2021-02-10T04:55:14.660902+0000","last_became_peered":"2021-02-10T04:55:14.660902+0000","last_unstale":"2021-02-10T04:57:53.609396+0000","last_undegraded":"2021-02-10T04:57:53.609396+0000","last_fullsized":"2021-02-10T04:57:53.609396+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:55.473347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:50:55.473347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"1261'58","reported_seq":"1372","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.875588+0000","last_change":"2021-02-10T04:55:47.652017+0000","last_active":"2021-02-10T04:57:59.875588+0000","last_peered":"2021-02-10T04:57:59.875588+0000","last_clean":"2021-02-10T04:57:59.875588+0000","last_became_active":"2021-02-10T04:55:47.651592+0000","last_became_peered":"2021-02-10T04:55:47.651592+0000","last_unstale":"2021-02-10T04:57:59.875588+0000","last_undegraded":"2021-02-10T04:57:59.875588+0000","last_fullsized":"2021-02-10T04:57:59.875588+0000","mapping_epoch":1227,"log_start":"1049'40","ondisk_log_start":"1049'40","created":55,"last_epoch_clean":1257,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'40","last_scrub_stamp":"2021-02-10T04:51:47.281929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.281929+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1836,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":171,"num_read_kb":135,"num_write":99,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"472","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.699442+0000","last_change":"2021-02-10T04:55:14.610765+0000","last_active":"2021-02-10T04:57:58.699442+0000","last_peered":"2021-02-10T04:57:58.699442+0000","last_clean":"2021-02-10T04:57:58.699442+0000","last_became_active":"2021-02-10T04:55:14.610470+0000","last_became_peered":"2021-02-10T04:55:14.610470+0000","last_unstale":"2021-02-10T04:57:58.699442+0000","last_undegraded":"2021-02-10T04:57:58.699442+0000","last_fullsized":"2021-02-10T04:57:58.699442+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:50.399350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:50.399350+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"1199","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.874830+0000","last_change":"2021-02-10T04:55:25.624031+0000","last_active":"2021-02-10T04:57:59.874830+0000","last_peered":"2021-02-10T04:57:59.874830+0000","last_clean":"2021-02-10T04:57:59.874830+0000","last_became_active":"2021-02-10T04:55:14.630129+0000","last_became_peered":"2021-02-10T04:55:14.630129+0000","last_unstale":"2021-02-10T04:57:59.874830+0000","last_undegraded":"2021-02-10T04:57:59.874830+0000","last_fullsized":"2021-02-10T04:57:59.874830+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"971","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.690532+0000","last_change":"2021-02-10T04:52:31.273037+0000","last_active":"2021-02-10T04:57:53.690532+0000","last_peered":"2021-02-10T04:57:53.690532+0000","last_clean":"2021-02-10T04:57:53.690532+0000","last_became_active":"2021-02-10T04:51:49.501405+0000","last_became_peered":"2021-02-10T04:51:49.501405+0000","last_unstale":"2021-02-10T04:57:53.690532+0000","last_undegraded":"2021-02-10T04:57:53.690532+0000","last_fullsized":"2021-02-10T04:57:53.690532+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:31.272971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:17.598113+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.272971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"1000'446","reported_seq":"2029","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.699975+0000","last_change":"2021-02-10T04:55:14.658783+0000","last_active":"2021-02-10T04:57:58.699975+0000","last_peered":"2021-02-10T04:57:58.699975+0000","last_clean":"2021-02-10T04:57:58.699975+0000","last_became_active":"2021-02-10T04:55:14.657668+0000","last_became_peered":"2021-02-10T04:55:14.657668+0000","last_unstale":"2021-02-10T04:57:58.699975+0000","last_undegraded":"2021-02-10T04:57:58.699975+0000","last_fullsized":"2021-02-10T04:57:58.699975+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:51:45.964029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.964029+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22255852,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68435,"num_write":72,"num_write_kb":24345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":403693,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"462","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.690775+0000","last_change":"2021-02-10T04:53:03.228260+0000","last_active":"2021-02-10T04:57:53.690775+0000","last_peered":"2021-02-10T04:57:53.690775+0000","last_clean":"2021-02-10T04:57:53.690775+0000","last_became_active":"2021-02-10T04:51:49.499252+0000","last_became_peered":"2021-02-10T04:51:49.499252+0000","last_unstale":"2021-02-10T04:57:53.690775+0000","last_undegraded":"2021-02-10T04:57:53.690775+0000","last_fullsized":"2021-02-10T04:57:53.690775+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:03.228182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.228182+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"216","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.699948+0000","last_change":"2021-02-10T04:53:46.727384+0000","last_active":"2021-02-10T04:57:58.699948+0000","last_peered":"2021-02-10T04:57:58.699948+0000","last_clean":"2021-02-10T04:57:58.699948+0000","last_became_active":"2021-02-10T04:53:46.726771+0000","last_became_peered":"2021-02-10T04:53:46.726771+0000","last_unstale":"2021-02-10T04:57:58.699948+0000","last_undegraded":"2021-02-10T04:57:58.699948+0000","last_fullsized":"2021-02-10T04:57:58.699948+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:33.948348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:33.948348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"1261'51","reported_seq":"1396","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:55.861479+0000","last_change":"2021-02-10T04:54:17.118810+0000","last_active":"2021-02-10T04:57:55.861479+0000","last_peered":"2021-02-10T04:57:55.861479+0000","last_clean":"2021-02-10T04:57:55.861479+0000","last_became_active":"2021-02-10T04:54:17.080053+0000","last_became_peered":"2021-02-10T04:54:17.080053+0000","last_unstale":"2021-02-10T04:57:55.861479+0000","last_undegraded":"2021-02-10T04:57:55.861479+0000","last_fullsized":"2021-02-10T04:57:55.861479+0000","mapping_epoch":1200,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1201,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'39","last_scrub_stamp":"2021-02-10T04:51:47.322331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.322331+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":51,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"244","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.699534+0000","last_change":"2021-02-10T04:55:14.636888+0000","last_active":"2021-02-10T04:57:58.699534+0000","last_peered":"2021-02-10T04:57:58.699534+0000","last_clean":"2021-02-10T04:57:58.699534+0000","last_became_active":"2021-02-10T04:55:14.634305+0000","last_became_peered":"2021-02-10T04:55:14.634305+0000","last_unstale":"2021-02-10T04:57:58.699534+0000","last_undegraded":"2021-02-10T04:57:58.699534+0000","last_fullsized":"2021-02-10T04:57:58.699534+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_clean_scrub_stamp":"2021-02-10T04:50:05.424262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"1480","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.608343+0000","last_change":"2021-02-10T04:52:37.324122+0000","last_active":"2021-02-10T04:57:53.608343+0000","last_peered":"2021-02-10T04:57:53.608343+0000","last_clean":"2021-02-10T04:57:53.608343+0000","last_became_active":"2021-02-10T04:51:36.843553+0000","last_became_peered":"2021-02-10T04:51:36.843553+0000","last_unstale":"2021-02-10T04:57:53.608343+0000","last_undegraded":"2021-02-10T04:57:53.608343+0000","last_fullsized":"2021-02-10T04:57:53.608343+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:37.324040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:37.324040+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"1289","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.203610+0000","last_change":"2021-02-10T04:54:37.612303+0000","last_active":"2021-02-10T04:58:00.203610+0000","last_peered":"2021-02-10T04:58:00.203610+0000","last_clean":"2021-02-10T04:58:00.203610+0000","last_became_active":"2021-02-10T04:54:37.611953+0000","last_became_peered":"2021-02-10T04:54:37.611953+0000","last_unstale":"2021-02-10T04:58:00.203610+0000","last_undegraded":"2021-02-10T04:58:00.203610+0000","last_fullsized":"2021-02-10T04:58:00.203610+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"1308","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.609477+0000","last_change":"2021-02-10T04:52:31.403904+0000","last_active":"2021-02-10T04:57:53.609477+0000","last_peered":"2021-02-10T04:57:53.609477+0000","last_clean":"2021-02-10T04:57:53.609477+0000","last_became_active":"2021-02-10T04:50:59.538009+0000","last_became_peered":"2021-02-10T04:50:59.538009+0000","last_unstale":"2021-02-10T04:57:53.609477+0000","last_undegraded":"2021-02-10T04:57:53.609477+0000","last_fullsized":"2021-02-10T04:57:53.609477+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:31.403824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.403824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"1000'490","reported_seq":"3779","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.875679+0000","last_change":"2021-02-10T04:55:22.716472+0000","last_active":"2021-02-10T04:57:59.875679+0000","last_peered":"2021-02-10T04:57:59.875679+0000","last_clean":"2021-02-10T04:57:59.875679+0000","last_became_active":"2021-02-10T04:55:14.644596+0000","last_became_peered":"2021-02-10T04:55:14.644596+0000","last_unstale":"2021-02-10T04:57:59.875679+0000","last_undegraded":"2021-02-10T04:57:59.875679+0000","last_fullsized":"2021-02-10T04:57:59.875679+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:47:02.875920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:02.875920+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19371627,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":141,"num_read_kb":148568,"num_write":85,"num_write_kb":31732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"208","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.608290+0000","last_change":"2021-02-10T04:53:01.972991+0000","last_active":"2021-02-10T04:57:58.608290+0000","last_peered":"2021-02-10T04:57:58.608290+0000","last_clean":"2021-02-10T04:57:58.608290+0000","last_became_active":"2021-02-10T04:51:36.863976+0000","last_became_peered":"2021-02-10T04:51:36.863976+0000","last_unstale":"2021-02-10T04:57:58.608290+0000","last_undegraded":"2021-02-10T04:57:58.608290+0000","last_fullsized":"2021-02-10T04:57:58.608290+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:01.972913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.972913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"1261'40","reported_seq":"1028","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.691063+0000","last_change":"2021-02-10T04:55:43.569281+0000","last_active":"2021-02-10T04:57:53.691063+0000","last_peered":"2021-02-10T04:57:53.691063+0000","last_clean":"2021-02-10T04:57:53.691063+0000","last_became_active":"2021-02-10T04:55:43.568878+0000","last_became_peered":"2021-02-10T04:55:43.568878+0000","last_unstale":"2021-02-10T04:57:53.691063+0000","last_undegraded":"2021-02-10T04:57:53.691063+0000","last_fullsized":"2021-02-10T04:57:53.691063+0000","mapping_epoch":1227,"log_start":"1063'30","ondisk_log_start":"1063'30","created":55,"last_epoch_clean":1253,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:56.431187+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.431187+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":802,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":116,"num_read_kb":94,"num_write":67,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"218","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.606137+0000","last_change":"2021-02-10T04:55:14.636849+0000","last_active":"2021-02-10T04:57:58.606137+0000","last_peered":"2021-02-10T04:57:58.606137+0000","last_clean":"2021-02-10T04:57:58.606137+0000","last_became_active":"2021-02-10T04:55:14.636237+0000","last_became_peered":"2021-02-10T04:55:14.636237+0000","last_unstale":"2021-02-10T04:57:58.606137+0000","last_undegraded":"2021-02-10T04:57:58.606137+0000","last_fullsized":"2021-02-10T04:57:58.606137+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:04.591695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:04.591695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"1135","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.689796+0000","last_change":"2021-02-10T04:54:37.680913+0000","last_active":"2021-02-10T04:57:53.689796+0000","last_peered":"2021-02-10T04:57:53.689796+0000","last_clean":"2021-02-10T04:57:53.689796+0000","last_became_active":"2021-02-10T04:54:37.680651+0000","last_became_peered":"2021-02-10T04:54:37.680651+0000","last_unstale":"2021-02-10T04:57:53.689796+0000","last_undegraded":"2021-02-10T04:57:53.689796+0000","last_fullsized":"2021-02-10T04:57:53.689796+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1210,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":149,"num_read_kb":148,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"1155","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.700158+0000","last_change":"2021-02-10T04:53:46.729138+0000","last_active":"2021-02-10T04:57:58.700158+0000","last_peered":"2021-02-10T04:57:58.700158+0000","last_clean":"2021-02-10T04:57:58.700158+0000","last_became_active":"2021-02-10T04:53:46.728845+0000","last_became_peered":"2021-02-10T04:53:46.728845+0000","last_unstale":"2021-02-10T04:57:58.700158+0000","last_undegraded":"2021-02-10T04:57:58.700158+0000","last_fullsized":"2021-02-10T04:57:58.700158+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:40.066354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:52:40.066354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"1000'620","reported_seq":"4403","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.875387+0000","last_change":"2021-02-10T04:55:28.089086+0000","last_active":"2021-02-10T04:57:59.875387+0000","last_peered":"2021-02-10T04:57:59.875387+0000","last_clean":"2021-02-10T04:57:59.875387+0000","last_became_active":"2021-02-10T04:55:14.640509+0000","last_became_peered":"2021-02-10T04:55:14.640509+0000","last_unstale":"2021-02-10T04:57:59.875387+0000","last_undegraded":"2021-02-10T04:57:59.875387+0000","last_fullsized":"2021-02-10T04:57:59.875387+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:51:54.444287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:54.444287+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14265195,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":144,"num_read_kb":128482,"num_write":129,"num_write_kb":53443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"1192","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.875191+0000","last_change":"2021-02-10T04:55:28.083286+0000","last_active":"2021-02-10T04:57:59.875191+0000","last_peered":"2021-02-10T04:57:59.875191+0000","last_clean":"2021-02-10T04:57:59.875191+0000","last_became_active":"2021-02-10T04:55:14.619846+0000","last_became_peered":"2021-02-10T04:55:14.619846+0000","last_unstale":"2021-02-10T04:57:59.875191+0000","last_undegraded":"2021-02-10T04:57:59.875191+0000","last_fullsized":"2021-02-10T04:57:59.875191+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:14.955544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:47:14.955544+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"1289","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.204147+0000","last_change":"2021-02-10T04:55:15.590025+0000","last_active":"2021-02-10T04:58:00.204147+0000","last_peered":"2021-02-10T04:58:00.204147+0000","last_clean":"2021-02-10T04:58:00.204147+0000","last_became_active":"2021-02-10T04:55:15.589651+0000","last_became_peered":"2021-02-10T04:55:15.589651+0000","last_unstale":"2021-02-10T04:58:00.204147+0000","last_undegraded":"2021-02-10T04:58:00.204147+0000","last_fullsized":"2021-02-10T04:58:00.204147+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"1295","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.875170+0000","last_change":"2021-02-10T04:55:28.090104+0000","last_active":"2021-02-10T04:57:59.875170+0000","last_peered":"2021-02-10T04:57:59.875170+0000","last_clean":"2021-02-10T04:57:59.875170+0000","last_became_active":"2021-02-10T04:55:14.650106+0000","last_became_peered":"2021-02-10T04:55:14.650106+0000","last_unstale":"2021-02-10T04:57:59.875170+0000","last_undegraded":"2021-02-10T04:57:59.875170+0000","last_fullsized":"2021-02-10T04:57:59.875170+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:27.908221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:47:27.908221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"1261'67","reported_seq":"960","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.690392+0000","last_change":"2021-02-10T04:52:30.277009+0000","last_active":"2021-02-10T04:57:53.690392+0000","last_peered":"2021-02-10T04:57:53.690392+0000","last_clean":"2021-02-10T04:57:53.690392+0000","last_became_active":"2021-02-10T04:51:49.501622+0000","last_became_peered":"2021-02-10T04:51:49.501622+0000","last_unstale":"2021-02-10T04:57:53.690392+0000","last_undegraded":"2021-02-10T04:57:53.690392+0000","last_fullsized":"2021-02-10T04:57:53.690392+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'55","last_scrub_stamp":"2021-02-10T04:52:30.276938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.276938+0000","log_size":67,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":116,"num_read_kb":92,"num_write":70,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"1304","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.607020+0000","last_change":"2021-02-10T04:51:59.420149+0000","last_active":"2021-02-10T04:57:53.607020+0000","last_peered":"2021-02-10T04:57:53.607020+0000","last_clean":"2021-02-10T04:57:53.607020+0000","last_became_active":"2021-02-10T04:28:21.125541+0000","last_became_peered":"2021-02-10T04:28:21.125541+0000","last_unstale":"2021-02-10T04:57:53.607020+0000","last_undegraded":"2021-02-10T04:57:53.607020+0000","last_fullsized":"2021-02-10T04:57:53.607020+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.420086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.420086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"1273","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.204210+0000","last_change":"2021-02-10T04:53:12.187225+0000","last_active":"2021-02-10T04:58:00.204210+0000","last_peered":"2021-02-10T04:58:00.204210+0000","last_clean":"2021-02-10T04:58:00.204210+0000","last_became_active":"2021-02-10T04:50:59.555689+0000","last_became_peered":"2021-02-10T04:50:59.555689+0000","last_unstale":"2021-02-10T04:58:00.204210+0000","last_undegraded":"2021-02-10T04:58:00.204210+0000","last_fullsized":"2021-02-10T04:58:00.204210+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.187154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.187154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"1002'2042","reported_seq":"9542","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:55.862643+0000","last_change":"2021-02-10T04:55:14.662208+0000","last_active":"2021-02-10T04:57:55.862643+0000","last_peered":"2021-02-10T04:57:55.862643+0000","last_clean":"2021-02-10T04:57:55.862643+0000","last_became_active":"2021-02-10T04:55:14.659601+0000","last_became_peered":"2021-02-10T04:55:14.659601+0000","last_unstale":"2021-02-10T04:57:55.862643+0000","last_undegraded":"2021-02-10T04:57:55.862643+0000","last_fullsized":"2021-02-10T04:57:55.862643+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:46:57.054230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:57.054230+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":85678898,"num_objects":35,"num_object_clones":29,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"1255","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.606520+0000","last_change":"2021-02-10T04:55:14.651301+0000","last_active":"2021-02-10T04:57:53.606520+0000","last_peered":"2021-02-10T04:57:53.606520+0000","last_clean":"2021-02-10T04:57:53.606520+0000","last_became_active":"2021-02-10T04:55:14.650386+0000","last_became_peered":"2021-02-10T04:55:14.650386+0000","last_unstale":"2021-02-10T04:57:53.606520+0000","last_undegraded":"2021-02-10T04:57:53.606520+0000","last_fullsized":"2021-02-10T04:57:53.606520+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:52.341527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:50:52.341527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"108","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.608919+0000","last_change":"2021-02-10T04:54:39.558380+0000","last_active":"2021-02-10T04:57:53.608919+0000","last_peered":"2021-02-10T04:57:53.608919+0000","last_clean":"2021-02-10T04:57:53.608919+0000","last_became_active":"2021-02-10T04:54:39.557995+0000","last_became_peered":"2021-02-10T04:54:39.557995+0000","last_unstale":"2021-02-10T04:57:53.608919+0000","last_undegraded":"2021-02-10T04:57:53.608919+0000","last_fullsized":"2021-02-10T04:57:53.608919+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"1238'4","reported_seq":"1518","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.875276+0000","last_change":"2021-02-10T04:55:28.095426+0000","last_active":"2021-02-10T04:57:59.875276+0000","last_peered":"2021-02-10T04:57:59.875276+0000","last_clean":"2021-02-10T04:57:59.875276+0000","last_became_active":"2021-02-10T04:55:14.645890+0000","last_became_peered":"2021-02-10T04:55:14.645890+0000","last_unstale":"2021-02-10T04:57:59.875276+0000","last_undegraded":"2021-02-10T04:57:59.875276+0000","last_fullsized":"2021-02-10T04:57:59.875276+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:54.466259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:54.466259+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"1261'64","reported_seq":"1006","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.690412+0000","last_change":"2021-02-10T04:53:01.304693+0000","last_active":"2021-02-10T04:57:53.690412+0000","last_peered":"2021-02-10T04:57:53.690412+0000","last_clean":"2021-02-10T04:57:53.690412+0000","last_became_active":"2021-02-10T04:51:49.498066+0000","last_became_peered":"2021-02-10T04:51:49.498066+0000","last_unstale":"2021-02-10T04:57:53.690412+0000","last_undegraded":"2021-02-10T04:57:53.690412+0000","last_fullsized":"2021-02-10T04:57:53.690412+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:53:01.304626+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.304626+0000","log_size":64,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":89,"num_read_kb":71,"num_write":52,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"567","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.690438+0000","last_change":"2021-02-10T04:55:14.610552+0000","last_active":"2021-02-10T04:57:53.690438+0000","last_peered":"2021-02-10T04:57:53.690438+0000","last_clean":"2021-02-10T04:57:53.690438+0000","last_became_active":"2021-02-10T04:55:14.609458+0000","last_became_peered":"2021-02-10T04:55:14.609458+0000","last_unstale":"2021-02-10T04:57:53.690438+0000","last_undegraded":"2021-02-10T04:57:53.690438+0000","last_fullsized":"2021-02-10T04:57:53.690438+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:49.447961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:49.447961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"1289","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.874705+0000","last_change":"2021-02-10T04:57:58.775453+0000","last_active":"2021-02-10T04:57:59.874705+0000","last_peered":"2021-02-10T04:57:59.874705+0000","last_clean":"2021-02-10T04:57:59.874705+0000","last_became_active":"2021-02-10T04:55:14.614581+0000","last_became_peered":"2021-02-10T04:55:14.614581+0000","last_unstale":"2021-02-10T04:57:59.874705+0000","last_undegraded":"2021-02-10T04:57:59.874705+0000","last_fullsized":"2021-02-10T04:57:59.874705+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:07.914061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:07.914061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"293'77","reported_seq":"3094","reported_epoch":"1312","state":"active+clean+scrubbing","last_fresh":"2021-02-10T04:58:00.202980+0000","last_change":"2021-02-10T04:53:43.102301+0000","last_active":"2021-02-10T04:58:00.202980+0000","last_peered":"2021-02-10T04:58:00.202980+0000","last_clean":"2021-02-10T04:58:00.202980+0000","last_became_active":"2021-02-10T04:52:26.024812+0000","last_became_peered":"2021-02-10T04:52:26.024812+0000","last_unstale":"2021-02-10T04:58:00.202980+0000","last_undegraded":"2021-02-10T04:58:00.202980+0000","last_fullsized":"2021-02-10T04:58:00.202980+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1886585,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":18587,"num_write":9,"num_write_kb":3315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"1324","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:55.862569+0000","last_change":"2021-02-10T04:53:07.025068+0000","last_active":"2021-02-10T04:57:55.862569+0000","last_peered":"2021-02-10T04:57:55.862569+0000","last_clean":"2021-02-10T04:57:55.862569+0000","last_became_active":"2021-02-10T04:50:59.589417+0000","last_became_peered":"2021-02-10T04:50:59.589417+0000","last_unstale":"2021-02-10T04:57:55.862569+0000","last_undegraded":"2021-02-10T04:57:55.862569+0000","last_fullsized":"2021-02-10T04:57:55.862569+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.024980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.024980+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"278","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.605643+0000","last_change":"2021-02-10T04:57:58.324510+0000","last_active":"2021-02-10T04:57:58.605643+0000","last_peered":"2021-02-10T04:57:58.605643+0000","last_clean":"2021-02-10T04:57:58.605643+0000","last_became_active":"2021-02-10T04:55:14.605267+0000","last_became_peered":"2021-02-10T04:55:14.605267+0000","last_unstale":"2021-02-10T04:57:58.605643+0000","last_undegraded":"2021-02-10T04:57:58.605643+0000","last_fullsized":"2021-02-10T04:57:58.605643+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.609500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.609500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"1261'86","reported_seq":"1493","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.203276+0000","last_change":"2021-02-10T04:55:35.577794+0000","last_active":"2021-02-10T04:58:00.203276+0000","last_peered":"2021-02-10T04:58:00.203276+0000","last_clean":"2021-02-10T04:58:00.203276+0000","last_became_active":"2021-02-10T04:55:35.577316+0000","last_became_peered":"2021-02-10T04:55:35.577316+0000","last_unstale":"2021-02-10T04:58:00.203276+0000","last_undegraded":"2021-02-10T04:58:00.203276+0000","last_fullsized":"2021-02-10T04:58:00.203276+0000","mapping_epoch":1238,"log_start":"1049'68","ondisk_log_start":"1049'68","created":55,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'39","last_scrub_stamp":"2021-02-10T04:53:38.085565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.085565+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1582,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":227,"num_read_kb":187,"num_write":128,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"1426","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.204809+0000","last_change":"2021-02-10T04:55:14.659217+0000","last_active":"2021-02-10T04:58:00.204809+0000","last_peered":"2021-02-10T04:58:00.204809+0000","last_clean":"2021-02-10T04:58:00.204809+0000","last_became_active":"2021-02-10T04:55:14.658879+0000","last_became_peered":"2021-02-10T04:55:14.658879+0000","last_unstale":"2021-02-10T04:58:00.204809+0000","last_undegraded":"2021-02-10T04:58:00.204809+0000","last_fullsized":"2021-02-10T04:58:00.204809+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:47:08.784593+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:47:08.784593+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"1275","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.203835+0000","last_change":"2021-02-10T04:52:30.019225+0000","last_active":"2021-02-10T04:58:00.203835+0000","last_peered":"2021-02-10T04:58:00.203835+0000","last_clean":"2021-02-10T04:58:00.203835+0000","last_became_active":"2021-02-10T04:50:59.561436+0000","last_became_peered":"2021-02-10T04:50:59.561436+0000","last_unstale":"2021-02-10T04:58:00.203835+0000","last_undegraded":"2021-02-10T04:58:00.203835+0000","last_fullsized":"2021-02-10T04:58:00.203835+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.019102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.019102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"1001'1528","reported_seq":"6260","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.203512+0000","last_change":"2021-02-10T04:52:26.036366+0000","last_active":"2021-02-10T04:58:00.203512+0000","last_peered":"2021-02-10T04:58:00.203512+0000","last_clean":"2021-02-10T04:58:00.203512+0000","last_became_active":"2021-02-10T04:52:26.034028+0000","last_became_peered":"2021-02-10T04:52:26.034028+0000","last_unstale":"2021-02-10T04:58:00.203512+0000","last_undegraded":"2021-02-10T04:58:00.203512+0000","last_fullsized":"2021-02-10T04:58:00.203512+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46944136,"num_objects":16,"num_object_clones":15,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":358,"num_read_kb":351934,"num_write":286,"num_write_kb":111355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":58968832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.15","version":"1000'446","reported_seq":"2249","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.205491+0000","last_change":"2021-02-10T04:53:11.051124+0000","last_active":"2021-02-10T04:58:00.205491+0000","last_peered":"2021-02-10T04:58:00.205491+0000","last_clean":"2021-02-10T04:58:00.205491+0000","last_became_active":"2021-02-10T04:51:40.510899+0000","last_became_peered":"2021-02-10T04:51:40.510899+0000","last_unstale":"2021-02-10T04:58:00.205491+0000","last_undegraded":"2021-02-10T04:58:00.205491+0000","last_fullsized":"2021-02-10T04:58:00.205491+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:53:11.051031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.051031+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24896285,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":18983408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"222","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.606008+0000","last_change":"2021-02-10T04:55:14.605585+0000","last_active":"2021-02-10T04:57:58.606008+0000","last_peered":"2021-02-10T04:57:58.606008+0000","last_clean":"2021-02-10T04:57:58.606008+0000","last_became_active":"2021-02-10T04:55:14.604916+0000","last_became_peered":"2021-02-10T04:55:14.604916+0000","last_unstale":"2021-02-10T04:57:58.606008+0000","last_undegraded":"2021-02-10T04:57:58.606008+0000","last_fullsized":"2021-02-10T04:57:58.606008+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:51:08.639784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.639784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"1244","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.204407+0000","last_change":"2021-02-10T04:51:09.475150+0000","last_active":"2021-02-10T04:58:00.204407+0000","last_peered":"2021-02-10T04:58:00.204407+0000","last_clean":"2021-02-10T04:58:00.204407+0000","last_became_active":"2021-02-10T04:50:59.585917+0000","last_became_peered":"2021-02-10T04:50:59.585917+0000","last_unstale":"2021-02-10T04:58:00.204407+0000","last_undegraded":"2021-02-10T04:58:00.204407+0000","last_fullsized":"2021-02-10T04:58:00.204407+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:09.475063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:09.475063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"213","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.608126+0000","last_change":"2021-02-10T04:53:34.927483+0000","last_active":"2021-02-10T04:57:58.608126+0000","last_peered":"2021-02-10T04:57:58.608126+0000","last_clean":"2021-02-10T04:57:58.608126+0000","last_became_active":"2021-02-10T04:51:36.859707+0000","last_became_peered":"2021-02-10T04:51:36.859707+0000","last_unstale":"2021-02-10T04:57:58.608126+0000","last_undegraded":"2021-02-10T04:57:58.608126+0000","last_fullsized":"2021-02-10T04:57:58.608126+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:34.927424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.927424+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"1000'620","reported_seq":"4599","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.203681+0000","last_change":"2021-02-10T04:53:37.384461+0000","last_active":"2021-02-10T04:58:00.203681+0000","last_peered":"2021-02-10T04:58:00.203681+0000","last_clean":"2021-02-10T04:58:00.203681+0000","last_became_active":"2021-02-10T04:51:40.510114+0000","last_became_peered":"2021-02-10T04:51:40.510114+0000","last_unstale":"2021-02-10T04:58:00.203681+0000","last_undegraded":"2021-02-10T04:58:00.203681+0000","last_fullsized":"2021-02-10T04:58:00.203681+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:53:37.384398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:37.384398+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28530390,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"1298","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:55.859470+0000","last_change":"2021-02-10T04:52:41.059269+0000","last_active":"2021-02-10T04:57:55.859470+0000","last_peered":"2021-02-10T04:57:55.859470+0000","last_clean":"2021-02-10T04:57:55.859470+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:57:55.859470+0000","last_undegraded":"2021-02-10T04:57:55.859470+0000","last_fullsized":"2021-02-10T04:57:55.859470+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:52:41.059159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.059159+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"1301","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.609218+0000","last_change":"2021-02-10T04:53:12.332890+0000","last_active":"2021-02-10T04:57:53.609218+0000","last_peered":"2021-02-10T04:57:53.609218+0000","last_clean":"2021-02-10T04:57:53.609218+0000","last_became_active":"2021-02-10T04:50:59.540375+0000","last_became_peered":"2021-02-10T04:50:59.540375+0000","last_unstale":"2021-02-10T04:57:53.609218+0000","last_undegraded":"2021-02-10T04:57:53.609218+0000","last_fullsized":"2021-02-10T04:57:53.609218+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.332233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.332233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"282","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.700049+0000","last_change":"2021-02-10T04:57:58.275470+0000","last_active":"2021-02-10T04:57:58.700049+0000","last_peered":"2021-02-10T04:57:58.700049+0000","last_clean":"2021-02-10T04:57:58.700049+0000","last_became_active":"2021-02-10T04:53:46.733191+0000","last_became_peered":"2021-02-10T04:53:46.733191+0000","last_unstale":"2021-02-10T04:57:58.700049+0000","last_undegraded":"2021-02-10T04:57:58.700049+0000","last_fullsized":"2021-02-10T04:57:58.700049+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:45.362537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:45.362537+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"1002'2042","reported_seq":"9284","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.700077+0000","last_change":"2021-02-10T04:53:46.726353+0000","last_active":"2021-02-10T04:57:58.700077+0000","last_peered":"2021-02-10T04:57:58.700077+0000","last_clean":"2021-02-10T04:57:58.700077+0000","last_became_active":"2021-02-10T04:53:46.725828+0000","last_became_peered":"2021-02-10T04:53:46.725828+0000","last_unstale":"2021-02-10T04:57:58.700077+0000","last_undegraded":"2021-02-10T04:57:58.700077+0000","last_fullsized":"2021-02-10T04:57:58.700077+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:52:41.286006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.286006+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46300076,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":545,"num_read_kb":531689,"num_write":399,"num_write_kb":153087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"1261'56","reported_seq":"1483","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.203945+0000","last_change":"2021-02-10T04:52:35.091494+0000","last_active":"2021-02-10T04:58:00.203945+0000","last_peered":"2021-02-10T04:58:00.203945+0000","last_clean":"2021-02-10T04:58:00.203945+0000","last_became_active":"2021-02-10T04:50:59.569915+0000","last_became_peered":"2021-02-10T04:50:59.569915+0000","last_unstale":"2021-02-10T04:58:00.203945+0000","last_undegraded":"2021-02-10T04:58:00.203945+0000","last_fullsized":"2021-02-10T04:58:00.203945+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'46","last_scrub_stamp":"2021-02-10T04:52:35.091374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:35.091374+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":95,"num_read_kb":80,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"84'7","reported_seq":"1325","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.606955+0000","last_change":"2021-02-10T04:51:52.498251+0000","last_active":"2021-02-10T04:57:53.606955+0000","last_peered":"2021-02-10T04:57:53.606955+0000","last_clean":"2021-02-10T04:57:53.606955+0000","last_became_active":"2021-02-10T04:50:59.575924+0000","last_became_peered":"2021-02-10T04:50:59.575924+0000","last_unstale":"2021-02-10T04:57:53.606955+0000","last_undegraded":"2021-02-10T04:57:53.606955+0000","last_fullsized":"2021-02-10T04:57:53.606955+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'7","last_scrub_stamp":"2021-02-10T04:51:52.498158+0000","last_deep_scrub":"84'7","last_deep_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.498158+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"1229'4","reported_seq":"1685","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:55.858772+0000","last_change":"2021-02-10T04:55:14.657278+0000","last_active":"2021-02-10T04:57:55.858772+0000","last_peered":"2021-02-10T04:57:55.858772+0000","last_clean":"2021-02-10T04:57:55.858772+0000","last_became_active":"2021-02-10T04:55:14.656867+0000","last_became_peered":"2021-02-10T04:55:14.656867+0000","last_unstale":"2021-02-10T04:57:55.858772+0000","last_undegraded":"2021-02-10T04:57:55.858772+0000","last_fullsized":"2021-02-10T04:57:55.858772+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:47.407557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:47.407557+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"1226","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.875961+0000","last_change":"2021-02-10T04:55:22.735243+0000","last_active":"2021-02-10T04:57:59.875961+0000","last_peered":"2021-02-10T04:57:59.875961+0000","last_clean":"2021-02-10T04:57:59.875961+0000","last_became_active":"2021-02-10T04:55:14.632398+0000","last_became_peered":"2021-02-10T04:55:14.632398+0000","last_unstale":"2021-02-10T04:57:59.875961+0000","last_undegraded":"2021-02-10T04:57:59.875961+0000","last_fullsized":"2021-02-10T04:57:59.875961+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:23.566027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_clean_scrub_stamp":"2021-02-10T04:52:23.566027+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"1000'490","reported_seq":"3828","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.205037+0000","last_change":"2021-02-10T04:55:14.644527+0000","last_active":"2021-02-10T04:58:00.205037+0000","last_peered":"2021-02-10T04:58:00.205037+0000","last_clean":"2021-02-10T04:58:00.205037+0000","last_became_active":"2021-02-10T04:55:14.643764+0000","last_became_peered":"2021-02-10T04:55:14.643764+0000","last_unstale":"2021-02-10T04:58:00.205037+0000","last_undegraded":"2021-02-10T04:58:00.205037+0000","last_fullsized":"2021-02-10T04:58:00.205037+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:50:03.756248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:50:03.756248+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16154790,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"1261'46","reported_seq":"1378","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.608852+0000","last_change":"2021-02-10T04:53:10.276449+0000","last_active":"2021-02-10T04:57:53.608852+0000","last_peered":"2021-02-10T04:57:53.608852+0000","last_clean":"2021-02-10T04:57:53.608852+0000","last_became_active":"2021-02-10T04:50:59.539529+0000","last_became_peered":"2021-02-10T04:50:59.539529+0000","last_unstale":"2021-02-10T04:57:53.608852+0000","last_undegraded":"2021-02-10T04:57:53.608852+0000","last_fullsized":"2021-02-10T04:57:53.608852+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1112'40","last_scrub_stamp":"2021-02-10T04:53:10.276358+0000","last_deep_scrub":"1112'40","last_deep_scrub_stamp":"2021-02-10T04:51:01.450050+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.276358+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":53,"num_write":38,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"1211","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.875618+0000","last_change":"2021-02-10T04:55:28.088480+0000","last_active":"2021-02-10T04:57:59.875618+0000","last_peered":"2021-02-10T04:57:59.875618+0000","last_clean":"2021-02-10T04:57:59.875618+0000","last_became_active":"2021-02-10T04:55:14.631666+0000","last_became_peered":"2021-02-10T04:55:14.631666+0000","last_unstale":"2021-02-10T04:57:59.875618+0000","last_undegraded":"2021-02-10T04:57:59.875618+0000","last_fullsized":"2021-02-10T04:57:59.875618+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:23.904210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:47:23.904210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"1119'6","reported_seq":"1602","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.204120+0000","last_change":"2021-02-10T04:52:40.038646+0000","last_active":"2021-02-10T04:58:00.204120+0000","last_peered":"2021-02-10T04:58:00.204120+0000","last_clean":"2021-02-10T04:58:00.204120+0000","last_became_active":"2021-02-10T04:51:36.869733+0000","last_became_peered":"2021-02-10T04:51:36.869733+0000","last_unstale":"2021-02-10T04:58:00.204120+0000","last_undegraded":"2021-02-10T04:58:00.204120+0000","last_fullsized":"2021-02-10T04:58:00.204120+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'6","last_scrub_stamp":"2021-02-10T04:52:40.038554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:40.038554+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"1298","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.205012+0000","last_change":"2021-02-10T04:51:36.849933+0000","last_active":"2021-02-10T04:58:00.205012+0000","last_peered":"2021-02-10T04:58:00.205012+0000","last_clean":"2021-02-10T04:58:00.205012+0000","last_became_active":"2021-02-10T04:51:36.849184+0000","last_became_peered":"2021-02-10T04:51:36.849184+0000","last_unstale":"2021-02-10T04:58:00.205012+0000","last_undegraded":"2021-02-10T04:58:00.205012+0000","last_fullsized":"2021-02-10T04:58:00.205012+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:11.779779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:11.779779+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"1032'89","reported_seq":"905","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.699342+0000","last_change":"2021-02-10T04:53:46.728868+0000","last_active":"2021-02-10T04:57:58.699342+0000","last_peered":"2021-02-10T04:57:58.699342+0000","last_clean":"2021-02-10T04:57:58.699342+0000","last_became_active":"2021-02-10T04:53:46.728356+0000","last_became_peered":"2021-02-10T04:53:46.728356+0000","last_unstale":"2021-02-10T04:57:58.699342+0000","last_undegraded":"2021-02-10T04:57:58.699342+0000","last_fullsized":"2021-02-10T04:57:58.699342+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":5,"last_scrub":"1032'89","last_scrub_stamp":"2021-02-10T04:52:34.089789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:34.089789+0000","log_size":89,"ondisk_log_size":89,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5218574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":30326,"num_write":21,"num_write_kb":12444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3210022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"1261'43","reported_seq":"999","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.690323+0000","last_change":"2021-02-10T04:53:14.202505+0000","last_active":"2021-02-10T04:57:53.690323+0000","last_peered":"2021-02-10T04:57:53.690323+0000","last_clean":"2021-02-10T04:57:53.690323+0000","last_became_active":"2021-02-10T04:51:49.496294+0000","last_became_peered":"2021-02-10T04:51:49.496294+0000","last_unstale":"2021-02-10T04:57:53.690323+0000","last_undegraded":"2021-02-10T04:57:53.690323+0000","last_fullsized":"2021-02-10T04:57:53.690323+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1163'37","last_scrub_stamp":"2021-02-10T04:53:14.202445+0000","last_deep_scrub":"1163'37","last_deep_scrub_stamp":"2021-02-10T04:53:14.202445+0000","last_clean_scrub_stamp":"2021-02-10T04:53:14.202445+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":60,"num_read_kb":42,"num_write":34,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"1311","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:55.861136+0000","last_change":"2021-02-10T04:52:23.095651+0000","last_active":"2021-02-10T04:57:55.861136+0000","last_peered":"2021-02-10T04:57:55.861136+0000","last_clean":"2021-02-10T04:57:55.861136+0000","last_became_active":"2021-02-10T04:50:59.543531+0000","last_became_peered":"2021-02-10T04:50:59.543531+0000","last_unstale":"2021-02-10T04:57:55.861136+0000","last_undegraded":"2021-02-10T04:57:55.861136+0000","last_fullsized":"2021-02-10T04:57:55.861136+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:23.095588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:23.095588+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"1263","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.203553+0000","last_change":"2021-02-10T04:55:14.636711+0000","last_active":"2021-02-10T04:58:00.203553+0000","last_peered":"2021-02-10T04:58:00.203553+0000","last_clean":"2021-02-10T04:58:00.203553+0000","last_became_active":"2021-02-10T04:55:14.636363+0000","last_became_peered":"2021-02-10T04:55:14.636363+0000","last_unstale":"2021-02-10T04:58:00.203553+0000","last_undegraded":"2021-02-10T04:58:00.203553+0000","last_fullsized":"2021-02-10T04:58:00.203553+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:43.663762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:51:43.663762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"1182","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.874961+0000","last_change":"2021-02-10T04:55:28.101709+0000","last_active":"2021-02-10T04:57:59.874961+0000","last_peered":"2021-02-10T04:57:59.874961+0000","last_clean":"2021-02-10T04:57:59.874961+0000","last_became_active":"2021-02-10T04:55:14.635912+0000","last_became_peered":"2021-02-10T04:55:14.635912+0000","last_unstale":"2021-02-10T04:57:59.874961+0000","last_undegraded":"2021-02-10T04:57:59.874961+0000","last_fullsized":"2021-02-10T04:57:59.874961+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:57.568541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:57.568541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"1000'1071","reported_seq":"4818","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.202801+0000","last_change":"2021-02-10T04:53:35.302879+0000","last_active":"2021-02-10T04:58:00.202801+0000","last_peered":"2021-02-10T04:58:00.202801+0000","last_clean":"2021-02-10T04:58:00.202801+0000","last_became_active":"2021-02-10T04:51:39.508551+0000","last_became_peered":"2021-02-10T04:51:39.508551+0000","last_unstale":"2021-02-10T04:58:00.202801+0000","last_undegraded":"2021-02-10T04:58:00.202801+0000","last_fullsized":"2021-02-10T04:58:00.202801+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:35.302780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.302780+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"1261'39","reported_seq":"1347","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.204249+0000","last_change":"2021-02-10T04:52:39.695340+0000","last_active":"2021-02-10T04:58:00.204249+0000","last_peered":"2021-02-10T04:58:00.204249+0000","last_clean":"2021-02-10T04:58:00.204249+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:58:00.204249+0000","last_undegraded":"2021-02-10T04:58:00.204249+0000","last_fullsized":"2021-02-10T04:58:00.204249+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'31","last_scrub_stamp":"2021-02-10T04:52:39.695237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:39.695237+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":53,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"554","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.690888+0000","last_change":"2021-02-10T04:55:20.929399+0000","last_active":"2021-02-10T04:57:53.690888+0000","last_peered":"2021-02-10T04:57:53.690888+0000","last_clean":"2021-02-10T04:57:53.690888+0000","last_became_active":"2021-02-10T04:55:14.655543+0000","last_became_peered":"2021-02-10T04:55:14.655543+0000","last_unstale":"2021-02-10T04:57:53.690888+0000","last_undegraded":"2021-02-10T04:57:53.690888+0000","last_fullsized":"2021-02-10T04:57:53.690888+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:55:20.929330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:55:20.929330+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"962","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.690494+0000","last_change":"2021-02-10T04:53:35.216489+0000","last_active":"2021-02-10T04:57:53.690494+0000","last_peered":"2021-02-10T04:57:53.690494+0000","last_clean":"2021-02-10T04:57:53.690494+0000","last_became_active":"2021-02-10T04:51:49.501254+0000","last_became_peered":"2021-02-10T04:51:49.501254+0000","last_unstale":"2021-02-10T04:57:53.690494+0000","last_undegraded":"2021-02-10T04:57:53.690494+0000","last_fullsized":"2021-02-10T04:57:53.690494+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.216394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.216394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"1290","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.203235+0000","last_change":"2021-02-10T04:55:14.633276+0000","last_active":"2021-02-10T04:58:00.203235+0000","last_peered":"2021-02-10T04:58:00.203235+0000","last_clean":"2021-02-10T04:58:00.203235+0000","last_became_active":"2021-02-10T04:55:14.632805+0000","last_became_peered":"2021-02-10T04:55:14.632805+0000","last_unstale":"2021-02-10T04:58:00.203235+0000","last_undegraded":"2021-02-10T04:58:00.203235+0000","last_fullsized":"2021-02-10T04:58:00.203235+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:55.814486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.814486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"1002'1657","reported_seq":"5335","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.875456+0000","last_change":"2021-02-10T04:55:22.728878+0000","last_active":"2021-02-10T04:57:59.875456+0000","last_peered":"2021-02-10T04:57:59.875456+0000","last_clean":"2021-02-10T04:57:59.875456+0000","last_became_active":"2021-02-10T04:55:14.653270+0000","last_became_peered":"2021-02-10T04:55:14.653270+0000","last_unstale":"2021-02-10T04:57:59.875456+0000","last_undegraded":"2021-02-10T04:57:59.875456+0000","last_fullsized":"2021-02-10T04:57:59.875456+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:55:22.331010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:55:22.331010+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":47335497,"num_objects":20,"num_object_clones":18,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"1261'24","reported_seq":"1420","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.203068+0000","last_change":"2021-02-10T04:53:04.058324+0000","last_active":"2021-02-10T04:58:00.203068+0000","last_peered":"2021-02-10T04:58:00.203068+0000","last_clean":"2021-02-10T04:58:00.203068+0000","last_became_active":"2021-02-10T04:50:59.544484+0000","last_became_peered":"2021-02-10T04:50:59.544484+0000","last_unstale":"2021-02-10T04:58:00.203068+0000","last_undegraded":"2021-02-10T04:58:00.203068+0000","last_fullsized":"2021-02-10T04:58:00.203068+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'20","last_scrub_stamp":"2021-02-10T04:53:04.058248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:04.058248+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":29,"num_write":24,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"1313","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.609208+0000","last_change":"2021-02-10T04:53:00.343307+0000","last_active":"2021-02-10T04:57:53.609208+0000","last_peered":"2021-02-10T04:57:53.609208+0000","last_clean":"2021-02-10T04:57:53.609208+0000","last_became_active":"2021-02-10T04:51:36.844573+0000","last_became_peered":"2021-02-10T04:51:36.844573+0000","last_unstale":"2021-02-10T04:57:53.609208+0000","last_undegraded":"2021-02-10T04:57:53.609208+0000","last_fullsized":"2021-02-10T04:57:53.609208+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:00.343228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:00.343228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"1282","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:55.859400+0000","last_change":"2021-02-10T04:50:56.099710+0000","last_active":"2021-02-10T04:57:55.859400+0000","last_peered":"2021-02-10T04:57:55.859400+0000","last_clean":"2021-02-10T04:57:55.859400+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:57:55.859400+0000","last_undegraded":"2021-02-10T04:57:55.859400+0000","last_fullsized":"2021-02-10T04:57:55.859400+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:56.099651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:50:56.099651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"630","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.699795+0000","last_change":"2021-02-10T04:53:46.718715+0000","last_active":"2021-02-10T04:57:58.699795+0000","last_peered":"2021-02-10T04:57:58.699795+0000","last_clean":"2021-02-10T04:57:58.699795+0000","last_became_active":"2021-02-10T04:53:46.718266+0000","last_became_peered":"2021-02-10T04:53:46.718266+0000","last_unstale":"2021-02-10T04:57:58.699795+0000","last_undegraded":"2021-02-10T04:57:58.699795+0000","last_fullsized":"2021-02-10T04:57:58.699795+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.876681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.876681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"998'1009","reported_seq":"5158","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:55.860462+0000","last_change":"2021-02-10T04:55:14.635764+0000","last_active":"2021-02-10T04:57:55.860462+0000","last_peered":"2021-02-10T04:57:55.860462+0000","last_clean":"2021-02-10T04:57:55.860462+0000","last_became_active":"2021-02-10T04:55:14.634422+0000","last_became_peered":"2021-02-10T04:55:14.634422+0000","last_unstale":"2021-02-10T04:57:55.860462+0000","last_undegraded":"2021-02-10T04:57:55.860462+0000","last_fullsized":"2021-02-10T04:57:55.860462+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":5,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:13.032506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:13.032506+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49420827,"num_objects":21,"num_object_clones":19,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":36763798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"1261'45","reported_seq":"1354","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.203700+0000","last_change":"2021-02-10T04:51:44.694779+0000","last_active":"2021-02-10T04:58:00.203700+0000","last_peered":"2021-02-10T04:58:00.203700+0000","last_clean":"2021-02-10T04:58:00.203700+0000","last_became_active":"2021-02-10T04:50:59.549412+0000","last_became_peered":"2021-02-10T04:50:59.549412+0000","last_unstale":"2021-02-10T04:58:00.203700+0000","last_undegraded":"2021-02-10T04:58:00.203700+0000","last_fullsized":"2021-02-10T04:58:00.203700+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:44.694711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.694711+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":76,"num_read_kb":62,"num_write":45,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"1449","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:55.862847+0000","last_change":"2021-02-10T04:52:33.088525+0000","last_active":"2021-02-10T04:57:55.862847+0000","last_peered":"2021-02-10T04:57:55.862847+0000","last_clean":"2021-02-10T04:57:55.862847+0000","last_became_active":"2021-02-10T04:51:36.870573+0000","last_became_peered":"2021-02-10T04:51:36.870573+0000","last_unstale":"2021-02-10T04:57:55.862847+0000","last_undegraded":"2021-02-10T04:57:55.862847+0000","last_fullsized":"2021-02-10T04:57:55.862847+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:52:33.088428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:33.088428+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"598","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.203063+0000","last_change":"2021-02-10T04:51:51.836723+0000","last_active":"2021-02-10T04:58:00.203063+0000","last_peered":"2021-02-10T04:58:00.203063+0000","last_clean":"2021-02-10T04:58:00.203063+0000","last_became_active":"2021-02-10T04:50:59.564977+0000","last_became_peered":"2021-02-10T04:50:59.564977+0000","last_unstale":"2021-02-10T04:58:00.203063+0000","last_undegraded":"2021-02-10T04:58:00.203063+0000","last_fullsized":"2021-02-10T04:58:00.203063+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.836638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.836638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"1254","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.203461+0000","last_change":"2021-02-10T04:55:14.633046+0000","last_active":"2021-02-10T04:58:00.203461+0000","last_peered":"2021-02-10T04:58:00.203461+0000","last_clean":"2021-02-10T04:58:00.203461+0000","last_became_active":"2021-02-10T04:55:14.632658+0000","last_became_peered":"2021-02-10T04:55:14.632658+0000","last_unstale":"2021-02-10T04:58:00.203461+0000","last_undegraded":"2021-02-10T04:58:00.203461+0000","last_fullsized":"2021-02-10T04:58:00.203461+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:58.514318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_clean_scrub_stamp":"2021-02-10T04:46:58.514318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1d","version":"1001'1165","reported_seq":"4310","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.690975+0000","last_change":"2021-02-10T04:55:14.656236+0000","last_active":"2021-02-10T04:57:53.690975+0000","last_peered":"2021-02-10T04:57:53.690975+0000","last_clean":"2021-02-10T04:57:53.690975+0000","last_became_active":"2021-02-10T04:55:14.655859+0000","last_became_peered":"2021-02-10T04:55:14.655859+0000","last_unstale":"2021-02-10T04:57:53.690975+0000","last_undegraded":"2021-02-10T04:57:53.690975+0000","last_fullsized":"2021-02-10T04:57:53.690975+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":5,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29601845,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":287,"num_read_kb":299530,"num_write":200,"num_write_kb":89131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"1354","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.606370+0000","last_change":"2021-02-10T04:53:01.331566+0000","last_active":"2021-02-10T04:57:53.606370+0000","last_peered":"2021-02-10T04:57:53.606370+0000","last_clean":"2021-02-10T04:57:53.606370+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:57:53.606370+0000","last_undegraded":"2021-02-10T04:57:53.606370+0000","last_fullsized":"2021-02-10T04:57:53.606370+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:01.331466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.331466+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"1098'3","reported_seq":"1676","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:58.063452+0000","last_change":"2021-02-10T04:53:34.865668+0000","last_active":"2021-02-10T04:57:58.063452+0000","last_peered":"2021-02-10T04:57:58.063452+0000","last_clean":"2021-02-10T04:57:58.063452+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:57:58.063452+0000","last_undegraded":"2021-02-10T04:57:58.063452+0000","last_fullsized":"2021-02-10T04:57:58.063452+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"1098'3","last_scrub_stamp":"2021-02-10T04:53:34.865579+0000","last_deep_scrub":"1098'3","last_deep_scrub_stamp":"2021-02-10T04:50:15.255137+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.865579+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"1160","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.700022+0000","last_change":"2021-02-10T04:53:46.727603+0000","last_active":"2021-02-10T04:57:58.700022+0000","last_peered":"2021-02-10T04:57:58.700022+0000","last_clean":"2021-02-10T04:57:58.700022+0000","last_became_active":"2021-02-10T04:53:46.727053+0000","last_became_peered":"2021-02-10T04:53:46.727053+0000","last_unstale":"2021-02-10T04:57:58.700022+0000","last_undegraded":"2021-02-10T04:57:58.700022+0000","last_fullsized":"2021-02-10T04:57:58.700022+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:47.493057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.493057+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1c","version":"1002'1090","reported_seq":"5334","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.607556+0000","last_change":"2021-02-10T04:53:41.353991+0000","last_active":"2021-02-10T04:57:53.607556+0000","last_peered":"2021-02-10T04:57:53.607556+0000","last_clean":"2021-02-10T04:57:53.607556+0000","last_became_active":"2021-02-10T04:52:31.088265+0000","last_became_peered":"2021-02-10T04:52:31.088265+0000","last_unstale":"2021-02-10T04:57:53.607556+0000","last_undegraded":"2021-02-10T04:57:53.607556+0000","last_fullsized":"2021-02-10T04:57:53.607556+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:53:41.353840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:41.353840+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"1258","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.203461+0000","last_change":"2021-02-10T04:55:14.632817+0000","last_active":"2021-02-10T04:58:00.203461+0000","last_peered":"2021-02-10T04:58:00.203461+0000","last_clean":"2021-02-10T04:58:00.203461+0000","last_became_active":"2021-02-10T04:55:14.632214+0000","last_became_peered":"2021-02-10T04:55:14.632214+0000","last_unstale":"2021-02-10T04:58:00.203461+0000","last_undegraded":"2021-02-10T04:58:00.203461+0000","last_fullsized":"2021-02-10T04:58:00.203461+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:42.537443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:42.537443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"1207","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.875576+0000","last_change":"2021-02-10T04:55:22.725805+0000","last_active":"2021-02-10T04:57:59.875576+0000","last_peered":"2021-02-10T04:57:59.875576+0000","last_clean":"2021-02-10T04:57:59.875576+0000","last_became_active":"2021-02-10T04:55:14.630190+0000","last_became_peered":"2021-02-10T04:55:14.630190+0000","last_unstale":"2021-02-10T04:57:59.875576+0000","last_undegraded":"2021-02-10T04:57:59.875576+0000","last_fullsized":"2021-02-10T04:57:59.875576+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.938731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.938731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"519","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.690745+0000","last_change":"2021-02-10T04:53:15.185748+0000","last_active":"2021-02-10T04:57:53.690745+0000","last_peered":"2021-02-10T04:57:53.690745+0000","last_clean":"2021-02-10T04:57:53.690745+0000","last_became_active":"2021-02-10T04:51:49.499348+0000","last_became_peered":"2021-02-10T04:51:49.499348+0000","last_unstale":"2021-02-10T04:57:53.690745+0000","last_undegraded":"2021-02-10T04:57:53.690745+0000","last_fullsized":"2021-02-10T04:57:53.690745+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:15.185686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:15.185686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1f","version":"1004'1093","reported_seq":"6572","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:55.862384+0000","last_change":"2021-02-10T04:52:31.089952+0000","last_active":"2021-02-10T04:57:55.862384+0000","last_peered":"2021-02-10T04:57:55.862384+0000","last_clean":"2021-02-10T04:57:55.862384+0000","last_became_active":"2021-02-10T04:52:31.088634+0000","last_became_peered":"2021-02-10T04:52:31.088634+0000","last_unstale":"2021-02-10T04:57:55.862384+0000","last_undegraded":"2021-02-10T04:57:55.862384+0000","last_fullsized":"2021-02-10T04:57:55.862384+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39434697,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":279,"num_read_kb":311058,"num_write":186,"num_write_kb":66790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":53995913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"1312","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:55.860887+0000","last_change":"2021-02-10T04:55:39.651056+0000","last_active":"2021-02-10T04:57:55.860887+0000","last_peered":"2021-02-10T04:57:55.860887+0000","last_clean":"2021-02-10T04:57:55.860887+0000","last_became_active":"2021-02-10T04:55:39.596648+0000","last_became_peered":"2021-02-10T04:55:39.596648+0000","last_unstale":"2021-02-10T04:57:55.860887+0000","last_undegraded":"2021-02-10T04:57:55.860887+0000","last_fullsized":"2021-02-10T04:57:55.860887+0000","mapping_epoch":1248,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:52:36.083784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:36.083784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"585","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:57.580016+0000","last_change":"2021-02-10T04:57:57.580830+0000","last_active":"2021-02-10T04:57:57.580016+0000","last_peered":"2021-02-10T04:57:57.580016+0000","last_clean":"2021-02-10T04:57:57.580016+0000","last_became_active":"2021-02-10T04:55:14.655649+0000","last_became_peered":"2021-02-10T04:55:14.655649+0000","last_unstale":"2021-02-10T04:57:57.580016+0000","last_undegraded":"2021-02-10T04:57:57.580016+0000","last_fullsized":"2021-02-10T04:57:57.580016+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:08.758129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.758129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"1280","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.204785+0000","last_change":"2021-02-10T04:51:44.799512+0000","last_active":"2021-02-10T04:58:00.204785+0000","last_peered":"2021-02-10T04:58:00.204785+0000","last_clean":"2021-02-10T04:58:00.204785+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:58:00.204785+0000","last_undegraded":"2021-02-10T04:58:00.204785+0000","last_fullsized":"2021-02-10T04:58:00.204785+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:44.799429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.799429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1e","version":"1063'654","reported_seq":"4552","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.691177+0000","last_change":"2021-02-10T04:53:40.135450+0000","last_active":"2021-02-10T04:57:53.691177+0000","last_peered":"2021-02-10T04:57:53.691177+0000","last_clean":"2021-02-10T04:57:53.691177+0000","last_became_active":"2021-02-10T04:52:31.087662+0000","last_became_peered":"2021-02-10T04:52:31.087662+0000","last_unstale":"2021-02-10T04:57:53.691177+0000","last_undegraded":"2021-02-10T04:57:53.691177+0000","last_fullsized":"2021-02-10T04:57:53.691177+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:53:40.135367+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:53:40.135367+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"1264","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.203837+0000","last_change":"2021-02-10T04:51:08.006463+0000","last_active":"2021-02-10T04:58:00.203837+0000","last_peered":"2021-02-10T04:58:00.203837+0000","last_clean":"2021-02-10T04:58:00.203837+0000","last_became_active":"2021-02-10T04:50:59.563657+0000","last_became_peered":"2021-02-10T04:50:59.563657+0000","last_unstale":"2021-02-10T04:58:00.203837+0000","last_undegraded":"2021-02-10T04:58:00.203837+0000","last_fullsized":"2021-02-10T04:58:00.203837+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_clean_scrub_stamp":"2021-02-10T04:51:02.455702+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"1315","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.203942+0000","last_change":"2021-02-10T04:52:33.741801+0000","last_active":"2021-02-10T04:58:00.203942+0000","last_peered":"2021-02-10T04:58:00.203942+0000","last_clean":"2021-02-10T04:58:00.203942+0000","last_became_active":"2021-02-10T04:51:36.860129+0000","last_became_peered":"2021-02-10T04:51:36.860129+0000","last_unstale":"2021-02-10T04:58:00.203942+0000","last_undegraded":"2021-02-10T04:58:00.203942+0000","last_fullsized":"2021-02-10T04:58:00.203942+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:33.741711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:46:58.666264+0000","last_clean_scrub_stamp":"2021-02-10T04:52:33.741711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"1276","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.204310+0000","last_change":"2021-02-10T04:52:25.030419+0000","last_active":"2021-02-10T04:58:00.204310+0000","last_peered":"2021-02-10T04:58:00.204310+0000","last_clean":"2021-02-10T04:58:00.204310+0000","last_became_active":"2021-02-10T04:50:59.539582+0000","last_became_peered":"2021-02-10T04:50:59.539582+0000","last_unstale":"2021-02-10T04:58:00.204310+0000","last_undegraded":"2021-02-10T04:58:00.204310+0000","last_fullsized":"2021-02-10T04:58:00.204310+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:25.030281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.030281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"1003'739","reported_seq":"4059","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:55.859073+0000","last_change":"2021-02-10T04:53:01.929261+0000","last_active":"2021-02-10T04:57:55.859073+0000","last_peered":"2021-02-10T04:57:55.859073+0000","last_clean":"2021-02-10T04:57:55.859073+0000","last_became_active":"2021-02-10T04:51:40.511505+0000","last_became_peered":"2021-02-10T04:51:40.511505+0000","last_unstale":"2021-02-10T04:57:55.859073+0000","last_undegraded":"2021-02-10T04:57:55.859073+0000","last_fullsized":"2021-02-10T04:57:55.859073+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:53:01.929169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.929169+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22230846,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":18429397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"1151","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.699665+0000","last_change":"2021-02-10T04:53:46.727590+0000","last_active":"2021-02-10T04:57:58.699665+0000","last_peered":"2021-02-10T04:57:58.699665+0000","last_clean":"2021-02-10T04:57:58.699665+0000","last_became_active":"2021-02-10T04:53:46.726584+0000","last_became_peered":"2021-02-10T04:53:46.726584+0000","last_unstale":"2021-02-10T04:57:58.699665+0000","last_undegraded":"2021-02-10T04:57:58.699665+0000","last_fullsized":"2021-02-10T04:57:58.699665+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:26.408326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.408326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"1310","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.608266+0000","last_change":"2021-02-10T04:53:07.324650+0000","last_active":"2021-02-10T04:57:53.608266+0000","last_peered":"2021-02-10T04:57:53.608266+0000","last_clean":"2021-02-10T04:57:53.608266+0000","last_became_active":"2021-02-10T04:50:59.575340+0000","last_became_peered":"2021-02-10T04:50:59.575340+0000","last_unstale":"2021-02-10T04:57:53.608266+0000","last_undegraded":"2021-02-10T04:57:53.608266+0000","last_fullsized":"2021-02-10T04:57:53.608266+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.324564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.324564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"1206","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.874844+0000","last_change":"2021-02-10T04:55:28.098627+0000","last_active":"2021-02-10T04:57:59.874844+0000","last_peered":"2021-02-10T04:57:59.874844+0000","last_clean":"2021-02-10T04:57:59.874844+0000","last_became_active":"2021-02-10T04:55:14.640497+0000","last_became_peered":"2021-02-10T04:55:14.640497+0000","last_unstale":"2021-02-10T04:57:59.874844+0000","last_undegraded":"2021-02-10T04:57:59.874844+0000","last_fullsized":"2021-02-10T04:57:59.874844+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:00.501127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:00.501127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"259'74","reported_seq":"251","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.874968+0000","last_change":"2021-02-10T04:55:28.095458+0000","last_active":"2021-02-10T04:57:59.874968+0000","last_peered":"2021-02-10T04:57:59.874968+0000","last_clean":"2021-02-10T04:57:59.874968+0000","last_became_active":"2021-02-10T04:55:14.624071+0000","last_became_peered":"2021-02-10T04:55:14.624071+0000","last_unstale":"2021-02-10T04:57:59.874968+0000","last_undegraded":"2021-02-10T04:57:59.874968+0000","last_fullsized":"2021-02-10T04:57:59.874968+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":5,"last_scrub":"259'74","last_scrub_stamp":"2021-02-10T04:52:26.597668+0000","last_deep_scrub":"259'74","last_deep_scrub_stamp":"2021-02-10T04:46:49.745104+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.597668+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1218524,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":28228,"num_write":19,"num_write_kb":7022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"73","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.607692+0000","last_change":"2021-02-10T04:55:39.619500+0000","last_active":"2021-02-10T04:57:58.607692+0000","last_peered":"2021-02-10T04:57:58.607692+0000","last_clean":"2021-02-10T04:57:58.607692+0000","last_became_active":"2021-02-10T04:55:39.619062+0000","last_became_peered":"2021-02-10T04:55:39.619062+0000","last_unstale":"2021-02-10T04:57:58.607692+0000","last_undegraded":"2021-02-10T04:57:58.607692+0000","last_fullsized":"2021-02-10T04:57:58.607692+0000","mapping_epoch":1248,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:59.167911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:50:59.167911+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"221","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.699428+0000","last_change":"2021-02-10T04:53:46.718512+0000","last_active":"2021-02-10T04:57:58.699428+0000","last_peered":"2021-02-10T04:57:58.699428+0000","last_clean":"2021-02-10T04:57:58.699428+0000","last_became_active":"2021-02-10T04:53:46.716811+0000","last_became_peered":"2021-02-10T04:53:46.716811+0000","last_unstale":"2021-02-10T04:57:58.699428+0000","last_undegraded":"2021-02-10T04:57:58.699428+0000","last_fullsized":"2021-02-10T04:57:58.699428+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:45.813676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.813676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"1299","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.606628+0000","last_change":"2021-02-10T04:53:38.363899+0000","last_active":"2021-02-10T04:57:53.606628+0000","last_peered":"2021-02-10T04:57:53.606628+0000","last_clean":"2021-02-10T04:57:53.606628+0000","last_became_active":"2021-02-10T04:50:59.539256+0000","last_became_peered":"2021-02-10T04:50:59.539256+0000","last_unstale":"2021-02-10T04:57:53.606628+0000","last_undegraded":"2021-02-10T04:57:53.606628+0000","last_fullsized":"2021-02-10T04:57:53.606628+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:38.363825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:09.433793+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.363825+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1b","version":"1003'1689","reported_seq":"4658","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.875363+0000","last_change":"2021-02-10T04:55:15.591337+0000","last_active":"2021-02-10T04:57:59.875363+0000","last_peered":"2021-02-10T04:57:59.875363+0000","last_clean":"2021-02-10T04:57:59.875363+0000","last_became_active":"2021-02-10T04:55:15.590301+0000","last_became_peered":"2021-02-10T04:55:15.590301+0000","last_unstale":"2021-02-10T04:57:59.875363+0000","last_undegraded":"2021-02-10T04:57:59.875363+0000","last_fullsized":"2021-02-10T04:57:59.875363+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:53:10.075879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.075879+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":63260738,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":387,"num_read_kb":409791,"num_write":281,"num_write_kb":110559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":22449943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"1297","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.204283+0000","last_change":"2021-02-10T04:55:14.637968+0000","last_active":"2021-02-10T04:58:00.204283+0000","last_peered":"2021-02-10T04:58:00.204283+0000","last_clean":"2021-02-10T04:58:00.204283+0000","last_became_active":"2021-02-10T04:55:14.636561+0000","last_became_peered":"2021-02-10T04:55:14.636561+0000","last_unstale":"2021-02-10T04:58:00.204283+0000","last_undegraded":"2021-02-10T04:58:00.204283+0000","last_fullsized":"2021-02-10T04:58:00.204283+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:33.543559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:46:33.543559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"969","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.606450+0000","last_change":"2021-02-10T04:53:04.354508+0000","last_active":"2021-02-10T04:57:53.606450+0000","last_peered":"2021-02-10T04:57:53.606450+0000","last_clean":"2021-02-10T04:57:53.606450+0000","last_became_active":"2021-02-10T04:33:05.169159+0000","last_became_peered":"2021-02-10T04:33:05.169159+0000","last_unstale":"2021-02-10T04:57:53.606450+0000","last_undegraded":"2021-02-10T04:57:53.606450+0000","last_fullsized":"2021-02-10T04:57:53.606450+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:04.354409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:04.354409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"1366","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.204625+0000","last_change":"2021-02-10T04:52:36.777554+0000","last_active":"2021-02-10T04:58:00.204625+0000","last_peered":"2021-02-10T04:58:00.204625+0000","last_clean":"2021-02-10T04:58:00.204625+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:58:00.204625+0000","last_undegraded":"2021-02-10T04:58:00.204625+0000","last_fullsized":"2021-02-10T04:58:00.204625+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:36.777485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:52:36.777485+0000","last_clean_scrub_stamp":"2021-02-10T04:52:36.777485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1a","version":"1001'1528","reported_seq":"5923","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.204450+0000","last_change":"2021-02-10T04:52:31.086406+0000","last_active":"2021-02-10T04:58:00.204450+0000","last_peered":"2021-02-10T04:58:00.204450+0000","last_clean":"2021-02-10T04:58:00.204450+0000","last_became_active":"2021-02-10T04:52:31.085959+0000","last_became_peered":"2021-02-10T04:52:31.085959+0000","last_unstale":"2021-02-10T04:58:00.204450+0000","last_undegraded":"2021-02-10T04:58:00.204450+0000","last_fullsized":"2021-02-10T04:58:00.204450+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46944136,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":358,"num_read_kb":351935,"num_write":286,"num_write_kb":111355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":21311893,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"1196","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.876010+0000","last_change":"2021-02-10T04:55:22.726939+0000","last_active":"2021-02-10T04:57:59.876010+0000","last_peered":"2021-02-10T04:57:59.876010+0000","last_clean":"2021-02-10T04:57:59.876010+0000","last_became_active":"2021-02-10T04:55:14.627044+0000","last_became_peered":"2021-02-10T04:55:14.627044+0000","last_unstale":"2021-02-10T04:57:59.876010+0000","last_undegraded":"2021-02-10T04:57:59.876010+0000","last_fullsized":"2021-02-10T04:57:59.876010+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:17.919745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:17.919745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"598","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.202759+0000","last_change":"2021-02-10T04:53:07.092032+0000","last_active":"2021-02-10T04:58:00.202759+0000","last_peered":"2021-02-10T04:58:00.202759+0000","last_clean":"2021-02-10T04:58:00.202759+0000","last_became_active":"2021-02-10T04:50:59.572077+0000","last_became_peered":"2021-02-10T04:50:59.572077+0000","last_unstale":"2021-02-10T04:58:00.202759+0000","last_undegraded":"2021-02-10T04:58:00.202759+0000","last_fullsized":"2021-02-10T04:58:00.202759+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.091949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:19.755718+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.091949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"1275","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.203639+0000","last_change":"2021-02-10T04:51:59.838856+0000","last_active":"2021-02-10T04:58:00.203639+0000","last_peered":"2021-02-10T04:58:00.203639+0000","last_clean":"2021-02-10T04:58:00.203639+0000","last_became_active":"2021-02-10T04:50:59.563156+0000","last_became_peered":"2021-02-10T04:50:59.563156+0000","last_unstale":"2021-02-10T04:58:00.203639+0000","last_undegraded":"2021-02-10T04:58:00.203639+0000","last_fullsized":"2021-02-10T04:58:00.203639+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.838790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.838790+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":"1005","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.607471+0000","last_change":"2021-02-10T04:52:30.405170+0000","last_active":"2021-02-10T04:57:53.607471+0000","last_peered":"2021-02-10T04:57:53.607471+0000","last_clean":"2021-02-10T04:57:53.607471+0000","last_became_active":"2021-02-10T04:51:36.852058+0000","last_became_peered":"2021-02-10T04:51:36.852058+0000","last_unstale":"2021-02-10T04:57:53.607471+0000","last_undegraded":"2021-02-10T04:57:53.607471+0000","last_fullsized":"2021-02-10T04:57:53.607471+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.405106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:20.635536+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.405106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.20","version":"0'0","reported_seq":"245","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.699255+0000","last_change":"2021-02-10T04:53:46.739888+0000","last_active":"2021-02-10T04:57:58.699255+0000","last_peered":"2021-02-10T04:57:58.699255+0000","last_clean":"2021-02-10T04:57:58.699255+0000","last_became_active":"2021-02-10T04:53:46.739532+0000","last_became_peered":"2021-02-10T04:53:46.739532+0000","last_unstale":"2021-02-10T04:57:58.699255+0000","last_undegraded":"2021-02-10T04:57:58.699255+0000","last_fullsized":"2021-02-10T04:57:58.699255+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:56.449085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:41.375276+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.449085+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":"1238","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:55.861391+0000","last_change":"2021-02-10T04:53:35.858634+0000","last_active":"2021-02-10T04:57:55.861391+0000","last_peered":"2021-02-10T04:57:55.861391+0000","last_clean":"2021-02-10T04:57:55.861391+0000","last_became_active":"2021-02-10T04:51:36.869785+0000","last_became_peered":"2021-02-10T04:51:36.869785+0000","last_unstale":"2021-02-10T04:57:55.861391+0000","last_undegraded":"2021-02-10T04:57:55.861391+0000","last_fullsized":"2021-02-10T04:57:55.861391+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.858557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.858557+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":"223","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.606198+0000","last_change":"2021-02-10T04:55:23.586523+0000","last_active":"2021-02-10T04:57:58.606198+0000","last_peered":"2021-02-10T04:57:58.606198+0000","last_clean":"2021-02-10T04:57:58.606198+0000","last_became_active":"2021-02-10T04:55:14.637029+0000","last_became_peered":"2021-02-10T04:55:14.637029+0000","last_unstale":"2021-02-10T04:57:58.606198+0000","last_undegraded":"2021-02-10T04:57:58.606198+0000","last_fullsized":"2021-02-10T04:57:58.606198+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:55:23.586401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:55:23.586401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":"206","reported_epoch":"1309","state":"active+clean","last_fresh":"2021-02-10T04:57:53.690757+0000","last_change":"2021-02-10T04:55:14.635842+0000","last_active":"2021-02-10T04:57:53.690757+0000","last_peered":"2021-02-10T04:57:53.690757+0000","last_clean":"2021-02-10T04:57:53.690757+0000","last_became_active":"2021-02-10T04:55:14.635499+0000","last_became_peered":"2021-02-10T04:55:14.635499+0000","last_unstale":"2021-02-10T04:57:53.690757+0000","last_undegraded":"2021-02-10T04:57:53.690757+0000","last_fullsized":"2021-02-10T04:57:53.690757+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.706372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.706372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":"1320","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:55.860618+0000","last_change":"2021-02-10T04:51:57.287452+0000","last_active":"2021-02-10T04:57:55.860618+0000","last_peered":"2021-02-10T04:57:55.860618+0000","last_clean":"2021-02-10T04:57:55.860618+0000","last_became_active":"2021-02-10T04:50:59.566827+0000","last_became_peered":"2021-02-10T04:50:59.566827+0000","last_unstale":"2021-02-10T04:57:55.860618+0000","last_undegraded":"2021-02-10T04:57:55.860618+0000","last_fullsized":"2021-02-10T04:57:55.860618+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.287363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.287363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":"294","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:55.862476+0000","last_change":"2021-02-10T04:55:14.659882+0000","last_active":"2021-02-10T04:57:55.862476+0000","last_peered":"2021-02-10T04:57:55.862476+0000","last_clean":"2021-02-10T04:57:55.862476+0000","last_became_active":"2021-02-10T04:55:14.659514+0000","last_became_peered":"2021-02-10T04:55:14.659514+0000","last_unstale":"2021-02-10T04:57:55.862476+0000","last_undegraded":"2021-02-10T04:57:55.862476+0000","last_fullsized":"2021-02-10T04:57:55.862476+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:10.308282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:10.308282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":"1220","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.203908+0000","last_change":"2021-02-10T04:56:45.790734+0000","last_active":"2021-02-10T04:58:00.203908+0000","last_peered":"2021-02-10T04:58:00.203908+0000","last_clean":"2021-02-10T04:58:00.203908+0000","last_became_active":"2021-02-10T04:56:45.790365+0000","last_became_peered":"2021-02-10T04:56:45.790365+0000","last_unstale":"2021-02-10T04:58:00.203908+0000","last_undegraded":"2021-02-10T04:58:00.203908+0000","last_fullsized":"2021-02-10T04:58:00.203908+0000","mapping_epoch":1280,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1281,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:09.121298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.121298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.21","version":"293'77","reported_seq":"883","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.203596+0000","last_change":"2021-02-10T04:52:28.062923+0000","last_active":"2021-02-10T04:58:00.203596+0000","last_peered":"2021-02-10T04:58:00.203596+0000","last_clean":"2021-02-10T04:58:00.203596+0000","last_became_active":"2021-02-10T04:52:28.062463+0000","last_became_peered":"2021-02-10T04:52:28.062463+0000","last_unstale":"2021-02-10T04:58:00.203596+0000","last_undegraded":"2021-02-10T04:58:00.203596+0000","last_fullsized":"2021-02-10T04:58:00.203596+0000","mapping_epoch":1143,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1144,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1886586,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":15163,"num_write":9,"num_write_kb":3316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":"208","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.605790+0000","last_change":"2021-02-10T04:52:32.124955+0000","last_active":"2021-02-10T04:57:58.605790+0000","last_peered":"2021-02-10T04:57:58.605790+0000","last_clean":"2021-02-10T04:57:58.605790+0000","last_became_active":"2021-02-10T04:51:36.867061+0000","last_became_peered":"2021-02-10T04:51:36.867061+0000","last_unstale":"2021-02-10T04:57:58.605790+0000","last_undegraded":"2021-02-10T04:57:58.605790+0000","last_fullsized":"2021-02-10T04:57:58.605790+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:32.124901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:08.271977+0000","last_clean_scrub_stamp":"2021-02-10T04:52:32.124901+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":"1249","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:55.861272+0000","last_change":"2021-02-10T04:51:07.182775+0000","last_active":"2021-02-10T04:57:55.861272+0000","last_peered":"2021-02-10T04:57:55.861272+0000","last_clean":"2021-02-10T04:57:55.861272+0000","last_became_active":"2021-02-10T04:28:23.126368+0000","last_became_peered":"2021-02-10T04:28:23.126368+0000","last_unstale":"2021-02-10T04:57:55.861272+0000","last_undegraded":"2021-02-10T04:57:55.861272+0000","last_fullsized":"2021-02-10T04:57:55.861272+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.182678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.182678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.20","version":"1000'1071","reported_seq":"4656","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.605476+0000","last_change":"2021-02-10T04:53:03.915465+0000","last_active":"2021-02-10T04:57:58.605476+0000","last_peered":"2021-02-10T04:57:58.605476+0000","last_clean":"2021-02-10T04:57:58.605476+0000","last_became_active":"2021-02-10T04:52:31.088690+0000","last_became_peered":"2021-02-10T04:52:31.088690+0000","last_unstale":"2021-02-10T04:57:58.605476+0000","last_undegraded":"2021-02-10T04:57:58.605476+0000","last_fullsized":"2021-02-10T04:57:58.605476+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:03.915403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.915403+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16398893,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":"834","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.699324+0000","last_change":"2021-02-10T04:53:46.721599+0000","last_active":"2021-02-10T04:57:58.699324+0000","last_peered":"2021-02-10T04:57:58.699324+0000","last_clean":"2021-02-10T04:57:58.699324+0000","last_became_active":"2021-02-10T04:53:46.721340+0000","last_became_peered":"2021-02-10T04:53:46.721340+0000","last_unstale":"2021-02-10T04:57:58.699324+0000","last_undegraded":"2021-02-10T04:57:58.699324+0000","last_fullsized":"2021-02-10T04:57:58.699324+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:28.092288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:07.767250+0000","last_clean_scrub_stamp":"2021-02-10T04:52:28.092288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.25","version":"53'2","reported_seq":"1267","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.205063+0000","last_change":"2021-02-10T04:53:36.103026+0000","last_active":"2021-02-10T04:58:00.205063+0000","last_peered":"2021-02-10T04:58:00.205063+0000","last_clean":"2021-02-10T04:58:00.205063+0000","last_became_active":"2021-02-10T04:28:23.467770+0000","last_became_peered":"2021-02-10T04:28:23.467770+0000","last_unstale":"2021-02-10T04:58:00.205063+0000","last_undegraded":"2021-02-10T04:58:00.205063+0000","last_fullsized":"2021-02-10T04:58:00.205063+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:53:36.102928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:36.102928+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.23","version":"1002'1657","reported_seq":"5181","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:55.859864+0000","last_change":"2021-02-10T04:52:31.093874+0000","last_active":"2021-02-10T04:57:55.859864+0000","last_peered":"2021-02-10T04:57:55.859864+0000","last_clean":"2021-02-10T04:57:55.859864+0000","last_became_active":"2021-02-10T04:52:31.092996+0000","last_became_peered":"2021-02-10T04:52:31.092996+0000","last_unstale":"2021-02-10T04:57:55.859864+0000","last_undegraded":"2021-02-10T04:57:55.859864+0000","last_fullsized":"2021-02-10T04:57:55.859864+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21396533,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":212,"num_read_kb":218837,"num_write":159,"num_write_kb":59399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":20479176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":"1356","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:55.859382+0000","last_change":"2021-02-10T04:53:12.951612+0000","last_active":"2021-02-10T04:57:55.859382+0000","last_peered":"2021-02-10T04:57:55.859382+0000","last_clean":"2021-02-10T04:57:55.859382+0000","last_became_active":"2021-02-10T04:50:59.566548+0000","last_became_peered":"2021-02-10T04:50:59.566548+0000","last_unstale":"2021-02-10T04:57:55.859382+0000","last_undegraded":"2021-02-10T04:57:55.859382+0000","last_fullsized":"2021-02-10T04:57:55.859382+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.951528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.951528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.26","version":"53'1","reported_seq":"148","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.700275+0000","last_change":"2021-02-10T04:53:46.722636+0000","last_active":"2021-02-10T04:57:58.700275+0000","last_peered":"2021-02-10T04:57:58.700275+0000","last_clean":"2021-02-10T04:57:58.700275+0000","last_became_active":"2021-02-10T04:53:46.722227+0000","last_became_peered":"2021-02-10T04:53:46.722227+0000","last_unstale":"2021-02-10T04:57:58.700275+0000","last_undegraded":"2021-02-10T04:57:58.700275+0000","last_fullsized":"2021-02-10T04:57:58.700275+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:53:03.933910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.933910+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":423,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.22","version":"998'1009","reported_seq":"4674","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.204497+0000","last_change":"2021-02-10T04:52:31.086563+0000","last_active":"2021-02-10T04:58:00.204497+0000","last_peered":"2021-02-10T04:58:00.204497+0000","last_clean":"2021-02-10T04:58:00.204497+0000","last_became_active":"2021-02-10T04:52:31.085288+0000","last_became_peered":"2021-02-10T04:52:31.085288+0000","last_unstale":"2021-02-10T04:58:00.204497+0000","last_undegraded":"2021-02-10T04:58:00.204497+0000","last_fullsized":"2021-02-10T04:58:00.204497+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12355207,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":120,"num_read_kb":124702,"num_write":77,"num_write_kb":25052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4145029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":"1238","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:55.859780+0000","last_change":"2021-02-10T04:53:10.958668+0000","last_active":"2021-02-10T04:57:55.859780+0000","last_peered":"2021-02-10T04:57:55.859780+0000","last_clean":"2021-02-10T04:57:55.859780+0000","last_became_active":"2021-02-10T04:45:49.552026+0000","last_became_peered":"2021-02-10T04:45:49.552026+0000","last_unstale":"2021-02-10T04:57:55.859780+0000","last_undegraded":"2021-02-10T04:57:55.859780+0000","last_fullsized":"2021-02-10T04:57:55.859780+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:10.958579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.958579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":"253","reported_epoch":"1311","state":"active+clean","last_fresh":"2021-02-10T04:57:58.700234+0000","last_change":"2021-02-10T04:53:46.725941+0000","last_active":"2021-02-10T04:57:58.700234+0000","last_peered":"2021-02-10T04:57:58.700234+0000","last_clean":"2021-02-10T04:57:58.700234+0000","last_became_active":"2021-02-10T04:53:46.725587+0000","last_became_peered":"2021-02-10T04:53:46.725587+0000","last_unstale":"2021-02-10T04:57:58.700234+0000","last_undegraded":"2021-02-10T04:57:58.700234+0000","last_fullsized":"2021-02-10T04:57:58.700234+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:41.758684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.758684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":"1249","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:58:00.202918+0000","last_change":"2021-02-10T04:55:39.599193+0000","last_active":"2021-02-10T04:58:00.202918+0000","last_peered":"2021-02-10T04:58:00.202918+0000","last_clean":"2021-02-10T04:58:00.202918+0000","last_became_active":"2021-02-10T04:55:39.598816+0000","last_became_peered":"2021-02-10T04:55:39.598816+0000","last_unstale":"2021-02-10T04:58:00.202918+0000","last_undegraded":"2021-02-10T04:58:00.202918+0000","last_fullsized":"2021-02-10T04:58:00.202918+0000","mapping_epoch":1248,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:11.139602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.139602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":"1264","reported_epoch":"1310","state":"active+clean","last_fresh":"2021-02-10T04:57:55.859092+0000","last_change":"2021-02-10T04:52:24.072005+0000","last_active":"2021-02-10T04:57:55.859092+0000","last_peered":"2021-02-10T04:57:55.859092+0000","last_clean":"2021-02-10T04:57:55.859092+0000","last_became_active":"2021-02-10T04:50:59.527945+0000","last_became_peered":"2021-02-10T04:50:59.527945+0000","last_unstale":"2021-02-10T04:57:55.859092+0000","last_undegraded":"2021-02-10T04:57:55.859092+0000","last_fullsized":"2021-02-10T04:57:55.859092+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:24.071915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_clean_scrub_stamp":"2021-02-10T04:52:24.071915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":972330808,"num_objects":390,"num_object_clones":337,"num_object_copies":1170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":390,"num_whiteouts":10,"num_read":4442,"num_read_kb":4559373,"num_write":3227,"num_write_kb":1247575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":611,"num_bytes_recovered":311961170,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2010624000,"data_stored":2589160348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34722,"ondisk_log_size":34722,"up":108,"acting":108,"num_store_stats":8},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":2368,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":18,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1139,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":5835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":126,"acting":126,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":13,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":797,"num_read_kb":792,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20,"ondisk_log_size":20,"up":39,"acting":39,"num_store_stats":8},{"poolid":4,"num_pg":22,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":2717,"num_read_kb":2190,"num_write":1592,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":17736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":844,"ondisk_log_size":844,"up":66,"acting":66,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35,"ondisk_log_size":35,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":126,"acting":126,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365191,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":196168,"kb_used_data":149440,"kb_used_omap":0,"kb_used_meta":46720,"kb_avail":93548984,"statfs":{"total":95995035648,"available":95794159616,"internally_reserved":0,"allocated":153026560,"data_stored":202514020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":47841280},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:57:20 2021","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.89500000000000002,"15min":0.92800000000000005},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.98999999999999999,"5min":4.4340000000000002,"15min":4.4340000000000002},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.89900000000000002}]},{"osd":1,"last update":"Wed Feb 10 04:57:47 2021","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.86199999999999999,"15min":0.91500000000000004},"min":{"1min":0.60499999999999998,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.3200000000000001,"5min":4.6760000000000002,"15min":5.1349999999999998},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.90800000000000003,"15min":0.96799999999999997},"min":{"1min":0.54400000000000004,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.796,"5min":4.4619999999999997,"15min":4.4619999999999997},"last":0.63100000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:57:47 2021","interfaces":[{"interface":"back","average":{"1min":7.0999999999999996,"5min":2.351,"15min":1.5489999999999999},"min":{"1min":0.54600000000000004,"5min":0.499,"15min":0.499},"max":{"1min":94.391999999999996,"5min":94.391999999999996,"15min":94.391999999999996},"last":0.76600000000000001},{"interface":"front","average":{"1min":6.9870000000000001,"5min":2.2930000000000001,"15min":1.575},"min":{"1min":0.58799999999999997,"5min":0.496,"15min":0.496},"max":{"1min":94.269999999999996,"5min":94.269999999999996,"15min":94.269999999999996},"last":0.65500000000000003}]},{"osd":3,"last update":"Wed Feb 10 04:57:01 2021","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.0549999999999999,"15min":1.2629999999999999},"min":{"1min":0.65300000000000002,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":4.5599999999999996,"5min":4.5599999999999996,"15min":4.5890000000000004},"last":0.95099999999999996},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.149,"15min":1.373},"min":{"1min":0.90800000000000003,"5min":0.629,"15min":0.629},"max":{"1min":4.6130000000000004,"5min":4.6130000000000004,"15min":4.6609999999999996},"last":1.0109999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:57:47 2021","interfaces":[{"interface":"back","average":{"1min":4.7930000000000001,"5min":1.78,"15min":1.5449999999999999},"min":{"1min":0.64800000000000002,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":72.662999999999997,"5min":72.662999999999997,"15min":72.662999999999997},"last":1.0609999999999999},{"interface":"front","average":{"1min":5.0499999999999998,"5min":1.921,"15min":1.5429999999999999},"min":{"1min":0.85499999999999998,"5min":0.71999999999999997,"15min":0.69099999999999995},"max":{"1min":73.564999999999998,"5min":73.564999999999998,"15min":73.564999999999998},"last":1.145}]},{"osd":5,"last update":"Wed Feb 10 04:57:47 2021","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.4079999999999999,"15min":1.486},"min":{"1min":1.032,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.258,"5min":4.766,"15min":14.972},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.171,"5min":1.298,"15min":1.5700000000000001},"min":{"1min":0.84899999999999998,"5min":0.78700000000000003,"15min":0.72699999999999998},"max":{"1min":2.1110000000000002,"5min":4.4020000000000001,"15min":15.218999999999999},"last":1.1819999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:57:47 2021","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.2649999999999999,"15min":1.631},"min":{"1min":0.78200000000000003,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.617,"5min":2.617,"15min":16.050000000000001},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.401,"5min":1.3560000000000001,"15min":1.6279999999999999},"min":{"1min":0.92700000000000005,"5min":0.878,"15min":0.876},"max":{"1min":1.9410000000000001,"5min":4.7889999999999997,"15min":15.960000000000001},"last":1.224}]}]},{"osd":6,"up_from":43,"seq":184683594123,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":392632,"kb_used_data":342512,"kb_used_omap":0,"kb_used_meta":50112,"kb_avail":93352520,"statfs":{"total":95995035648,"available":95592980480,"internally_reserved":0,"allocated":350732288,"data_stored":443465230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":51314688},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:57:18 2021","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.88100000000000001,"15min":0.88800000000000001},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.6040000000000001,"5min":1.6599999999999999,"15min":1.6599999999999999},"last":0.71199999999999997}]},{"osd":1,"last update":"Wed Feb 10 04:57:25 2021","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.999,"15min":1.0289999999999999},"min":{"1min":0.55400000000000005,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.7330000000000001,"5min":13.18,"15min":13.18},"last":1.5109999999999999},{"interface":"front","average":{"1min":0.88,"5min":0.97799999999999998,"15min":1.075},"min":{"1min":0.59199999999999997,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.3740000000000001,"5min":11.041,"15min":11.041},"last":6.5270000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:57:25 2021","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":3.2759999999999998,"15min":1.9279999999999999},"min":{"1min":0.60399999999999998,"5min":0.503,"15min":0.503},"max":{"1min":1.6459999999999999,"5min":163.81700000000001,"15min":163.81700000000001},"last":0.84499999999999997},{"interface":"front","average":{"1min":1.1499999999999999,"5min":3.379,"15min":1.974},"min":{"1min":0.78200000000000003,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.8460000000000001,"5min":164.56800000000001,"15min":164.56800000000001},"last":6.4870000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.1060000000000001,"15min":1.181},"min":{"1min":0.69499999999999995,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.944,"5min":1.944,"15min":1.944},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.1180000000000001,"15min":1.1220000000000001},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.766,"5min":2.363,"15min":2.363},"last":6.6130000000000004}]},{"osd":4,"last update":"Wed Feb 10 04:57:25 2021","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.135,"15min":1.333},"min":{"1min":0.77700000000000002,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.016,"5min":2.2360000000000002,"15min":7.1459999999999999},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.98399999999999999,"15min":1.0840000000000001},"min":{"1min":0.66100000000000003,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.5649999999999999,"5min":5.2750000000000004,"15min":11.997999999999999},"last":1.101}]},{"osd":5,"last update":"Wed Feb 10 04:57:25 2021","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.4239999999999999,"15min":1.6519999999999999},"min":{"1min":0.79100000000000004,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":4.3449999999999998,"5min":22.663,"15min":81.540000000000006},"last":1.4930000000000001},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.6990000000000001,"15min":1.9359999999999999},"min":{"1min":0.99399999999999999,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":2.048,"5min":22.933,"15min":81.492999999999995},"last":1.1579999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:57:49 2021","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.1279999999999999,"15min":1.2669999999999999},"min":{"1min":0.72599999999999998,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.677,"5min":2.073,"15min":5.1820000000000004},"last":1.645},{"interface":"front","average":{"1min":1.401,"5min":1.3640000000000001,"15min":1.5409999999999999},"min":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":2.1459999999999999,"5min":2.2970000000000002,"15min":5.4109999999999996},"last":1.373}]}]},{"osd":1,"up_from":13,"seq":55834575260,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":261932,"kb_used_data":206500,"kb_used_omap":0,"kb_used_meta":55424,"kb_avail":93483220,"statfs":{"total":95995035648,"available":95726817280,"internally_reserved":0,"allocated":211456000,"data_stored":265960298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":56754176},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:57:18 2021","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.69799999999999995,"15min":0.68899999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":2.3439999999999999,"5min":2.3439999999999999,"15min":2.3439999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.996,"5min":0.93899999999999995,"15min":0.92900000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":2.7589999999999999,"5min":2.7589999999999999,"15min":2.7589999999999999},"last":0.66300000000000003}]},{"osd":2,"last update":"Wed Feb 10 04:57:18 2021","interfaces":[{"interface":"back","average":{"1min":11.41,"5min":6.0540000000000003,"15min":3.2999999999999998},"min":{"1min":0.60199999999999998,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":101.06100000000001,"5min":162.61199999999999,"15min":162.61199999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":11.478999999999999,"5min":6.1760000000000002,"15min":3.3900000000000001},"min":{"1min":0.52200000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":101.66500000000001,"5min":162.762,"15min":162.762},"last":0.64400000000000002}]},{"osd":3,"last update":"Wed Feb 10 04:57:07 2021","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.76400000000000001,"15min":0.90000000000000002},"min":{"1min":0.66900000000000004,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.2030000000000001,"5min":1.6919999999999999,"15min":3.077},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.82999999999999996,"15min":0.97799999999999998},"min":{"1min":0.46800000000000003,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.3580000000000001,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.626}]},{"osd":4,"last update":"Wed Feb 10 04:57:01 2021","interfaces":[{"interface":"back","average":{"1min":5.4379999999999997,"5min":1.891,"15min":1.415},"min":{"1min":0.64900000000000002,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":100.071,"5min":100.071,"15min":100.071},"last":0.748},{"interface":"front","average":{"1min":5.5810000000000004,"5min":1.9470000000000001,"15min":1.5509999999999999},"min":{"1min":0.75800000000000001,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":100.43899999999999,"5min":100.43899999999999,"15min":100.43899999999999},"last":0.55700000000000005}]},{"osd":5,"last update":"Wed Feb 10 04:57:07 2021","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.0800000000000001,"15min":1.163},"min":{"1min":0.63100000000000001,"5min":0.59799999999999998,"15min":0.56599999999999995},"max":{"1min":2.738,"5min":2.738,"15min":5.0300000000000002},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.147,"15min":1.4379999999999999},"min":{"1min":0.77000000000000002,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.462,"5min":6.032,"15min":28.678000000000001},"last":0.84499999999999997}]},{"osd":6,"last update":"Wed Feb 10 04:57:18 2021","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.379,"15min":1.4930000000000001},"min":{"1min":0.78700000000000003,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":3.302,"5min":30.997,"15min":30.997},"last":0.69099999999999995},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.4510000000000001,"15min":1.478},"min":{"1min":0.90700000000000003,"5min":0.64200000000000002,"15min":0.60499999999999998},"max":{"1min":3.1549999999999998,"5min":31.138000000000002,"15min":31.138000000000002},"last":0.92200000000000004}]},{"osd":7,"last update":"Wed Feb 10 04:57:29 2021","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.2170000000000001,"15min":1.452},"min":{"1min":0.87,"5min":0.628,"15min":0.628},"max":{"1min":2.9889999999999999,"5min":6.2629999999999999,"15min":28.573},"last":1.026},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.226,"15min":1.4710000000000001},"min":{"1min":0.98199999999999998,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":3.4129999999999998,"5min":3.4129999999999998,"15min":29.408999999999999},"last":0.83299999999999996}]}]},{"osd":0,"up_from":1227,"seq":5269924872227,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":349552,"kb_used_data":338728,"kb_used_omap":306,"kb_used_meta":10509,"kb_avail":93395600,"statfs":{"total":95995035648,"available":95637094400,"internally_reserved":0,"allocated":346857472,"data_stored":471603084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":313579,"internal_metadata":10762005},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 04:57:15 2021","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.64600000000000002,"15min":0.64000000000000001},"min":{"1min":0.52900000000000003,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.73099999999999998,"15min":0.72299999999999998},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":2.4279999999999999,"5min":2.4279999999999999,"15min":2.4279999999999999},"last":0.83299999999999996}]},{"osd":2,"last update":"Wed Feb 10 04:57:15 2021","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":2.9620000000000002,"15min":3.2909999999999999},"min":{"1min":0.71899999999999997,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.1520000000000001,"5min":59.341999999999999,"15min":59.341999999999999},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.01,"5min":2.9740000000000002,"15min":3.302},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.2360000000000002,"5min":59.405999999999999,"15min":59.405999999999999},"last":0.98799999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:57:15 2021","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.76600000000000001,"15min":0.751},"min":{"1min":0.59099999999999997,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.181,"5min":2.181,"15min":2.181},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.002,"15min":0.998},"min":{"1min":0.76400000000000001,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.99,"5min":1.99,"15min":1.99},"last":1.161}]},{"osd":4,"last update":"Wed Feb 10 04:57:15 2021","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.95299999999999996,"15min":0.94899999999999995},"min":{"1min":0.64800000000000002,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.6280000000000001,"5min":2.6280000000000001,"15min":2.6280000000000001},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.125,"15min":1.1120000000000001},"min":{"1min":0.70699999999999996,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.0529999999999999,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":1.2549999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:57:15 2021","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.2490000000000001,"15min":1.2430000000000001},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":2.5459999999999998,"5min":2.5459999999999998,"15min":2.5459999999999998},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.194,"5min":1.1459999999999999,"15min":1.1379999999999999},"min":{"1min":0.89600000000000002,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.4540000000000002,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":1.179}]},{"osd":6,"last update":"Wed Feb 10 04:57:15 2021","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.1890000000000001,"15min":1.177},"min":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"max":{"1min":1.9390000000000001,"5min":1.9390000000000001,"15min":1.9390000000000001},"last":1.2889999999999999},{"interface":"front","average":{"1min":1.26,"5min":1.218,"15min":1.2110000000000001},"min":{"1min":0.90000000000000002,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.5129999999999999,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":1.2370000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:57:15 2021","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.2629999999999999,"15min":1.2569999999999999},"min":{"1min":1.0660000000000001,"5min":1.02,"15min":1.02},"max":{"1min":2.0710000000000002,"5min":2.0710000000000002,"15min":2.0710000000000002},"last":1.601},{"interface":"front","average":{"1min":1.258,"5min":1.2150000000000001,"15min":1.208},"min":{"1min":0.95999999999999996,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":2.335,"5min":2.335,"15min":2.335},"last":1.6339999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411739,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":354812,"kb_used_data":318708,"kb_used_omap":0,"kb_used_meta":36096,"kb_avail":93390340,"statfs":{"total":95995035648,"available":95631708160,"internally_reserved":0,"allocated":326356992,"data_stored":375002232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36962304},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:57:18 2021","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.67200000000000004,"15min":0.67000000000000004},"min":{"1min":0.50800000000000001,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.94799999999999995,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.93799999999999994,"15min":0.93999999999999995},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.9950000000000001,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":0.97399999999999998}]},{"osd":1,"last update":"Wed Feb 10 04:57:58 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.83699999999999997,"15min":0.95599999999999996},"min":{"1min":0.47599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.26,"5min":17.041,"15min":17.041},"last":0.86699999999999999},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.014,"15min":0.93300000000000005},"min":{"1min":0.61199999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":2.1019999999999999,"5min":21.172000000000001,"15min":21.172000000000001},"last":1.0700000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:57:18 2021","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.0129999999999999,"15min":1.115},"min":{"1min":0.58399999999999996,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.29,"5min":18.904,"15min":18.904},"last":1.054},{"interface":"front","average":{"1min":1.286,"5min":1.262,"15min":1.3080000000000001},"min":{"1min":0.84799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":2.419,"5min":19.241,"15min":19.241},"last":1.117}]},{"osd":4,"last update":"Wed Feb 10 04:57:37 2021","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.1970000000000001,"15min":1.278},"min":{"1min":0.68799999999999994,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.7470000000000001,"5min":9.2029999999999994,"15min":9.2029999999999994},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.2989999999999999,"15min":1.1919999999999999},"min":{"1min":0.76300000000000001,"5min":0.503,"15min":0.503},"max":{"1min":2.2109999999999999,"5min":24.373999999999999,"15min":24.373999999999999},"last":1.319}]},{"osd":5,"last update":"Wed Feb 10 04:57:58 2021","interfaces":[{"interface":"back","average":{"1min":1.76,"5min":1.675,"15min":1.7490000000000001},"min":{"1min":0.89500000000000002,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":7.9050000000000002,"5min":39.911999999999999,"15min":39.911999999999999},"last":1.268},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.3660000000000001,"15min":1.488},"min":{"1min":0.69399999999999995,"5min":0.65900000000000003,"15min":0.57599999999999996},"max":{"1min":7.9969999999999999,"5min":33.539999999999999,"15min":33.539999999999999},"last":1.0980000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:57:31 2021","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":1.0409999999999999,"15min":1.375},"min":{"1min":0.84899999999999998,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.7290000000000001,"5min":3.1659999999999999,"15min":20.007000000000001},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.4530000000000001,"15min":1.6220000000000001},"min":{"1min":1.069,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.8999999999999999,"5min":16.135000000000002,"15min":19.417999999999999},"last":1.405}]},{"osd":7,"last update":"Wed Feb 10 04:57:37 2021","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.637,"15min":1.641},"min":{"1min":1.117,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.748,"5min":24.405000000000001,"15min":24.405000000000001},"last":1.345},{"interface":"front","average":{"1min":1.284,"5min":1.196,"15min":1.2789999999999999},"min":{"1min":0.95899999999999996,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.4740000000000002,"5min":3.6640000000000001,"15min":4.9189999999999996},"last":1.252}]}]},{"osd":3,"up_from":1111,"seq":4771708665942,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":333468,"kb_used_data":301460,"kb_used_omap":386,"kb_used_meta":31613,"kb_avail":93411684,"statfs":{"total":95995035648,"available":95653564416,"internally_reserved":0,"allocated":308695040,"data_stored":409923879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":396036,"internal_metadata":32371964},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:57:18 2021","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":1.724,"15min":1.8939999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.004,"5min":22.869,"15min":22.869},"last":0.626},{"interface":"front","average":{"1min":0.72099999999999997,"5min":1.446,"15min":1.5669999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.98099999999999998,"5min":13.587999999999999,"15min":13.587999999999999},"last":0.64400000000000002}]},{"osd":1,"last update":"Wed Feb 10 04:57:09 2021","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.73899999999999999,"15min":0.67900000000000005},"min":{"1min":0.54100000000000004,"5min":0.44,"15min":0.44},"max":{"1min":1.304,"5min":3.6629999999999998,"15min":3.6629999999999998},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.879,"5min":0.79900000000000004,"15min":0.86199999999999999},"min":{"1min":0.60399999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":2.2450000000000001,"5min":3.7050000000000001,"15min":3.7050000000000001},"last":0.78000000000000003}]},{"osd":2,"last update":"Wed Feb 10 04:57:09 2021","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":1.337,"15min":1.212},"min":{"1min":0.53600000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0940000000000001,"5min":31.100999999999999,"15min":31.100999999999999},"last":0.754},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.587,"15min":1.276},"min":{"1min":0.92000000000000004,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":2.4390000000000001,"5min":31.052,"15min":31.052},"last":0.94599999999999995}]},{"osd":4,"last update":"Wed Feb 10 04:57:09 2021","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1,"15min":0.97999999999999998},"min":{"1min":0.78600000000000003,"5min":0.628,"15min":0.628},"max":{"1min":1.734,"5min":2.9430000000000001,"15min":2.9430000000000001},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.194,"5min":1.0029999999999999,"15min":1.087},"min":{"1min":0.81299999999999994,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":2.6739999999999999,"5min":3.21,"15min":3.21},"last":1.1659999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:57:09 2021","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.6930000000000001,"15min":2.3959999999999999},"min":{"1min":0.80900000000000005,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":2.7919999999999998,"5min":52.353000000000002,"15min":52.353000000000002},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.571,"15min":2.2189999999999999},"min":{"1min":0.78600000000000003,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.6140000000000001,"5min":52.301000000000002,"15min":52.301000000000002},"last":0.84799999999999998}]},{"osd":6,"last update":"Wed Feb 10 04:57:09 2021","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.6459999999999999,"15min":4.3659999999999997},"min":{"1min":0.86099999999999999,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.571,"5min":51.914999999999999,"15min":64.376999999999995},"last":0.92900000000000005},{"interface":"front","average":{"1min":1.429,"5min":1.8320000000000001,"15min":4.4000000000000004},"min":{"1min":1.091,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":2,"5min":51.896000000000001,"15min":64.438999999999993},"last":1.079}]},{"osd":7,"last update":"Wed Feb 10 04:57:09 2021","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.075,"15min":1.109},"min":{"1min":0.95299999999999996,"5min":0.77500000000000002,"15min":0.754},"max":{"1min":1.508,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":1.004},{"interface":"front","average":{"1min":1.353,"5min":1.1839999999999999,"15min":1.157},"min":{"1min":1.071,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.9059999999999999,"5min":2.9569999999999999,"15min":2.9569999999999999},"last":1.194}]}]},{"osd":4,"up_from":31,"seq":133143986579,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":165372,"kb_used_data":120692,"kb_used_omap":0,"kb_used_meta":44672,"kb_avail":93579780,"statfs":{"total":95995035648,"available":95825694720,"internally_reserved":0,"allocated":123588608,"data_stored":155825239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45744128},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:57:16 2021","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.85799999999999998,"15min":0.86499999999999999},"min":{"1min":0.53700000000000003,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.5449999999999999,"5min":3.637,"15min":3.637},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.878,"15min":0.89000000000000001},"min":{"1min":0.61399999999999999,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.2949999999999999,"5min":4.6829999999999998,"15min":4.6829999999999998},"last":0.50900000000000001}]},{"osd":1,"last update":"Wed Feb 10 04:57:48 2021","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.745,"15min":0.86099999999999999},"min":{"1min":0.58899999999999997,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.8120000000000001,"5min":3.2650000000000001,"15min":7.883},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.0369999999999999,"5min":0.89500000000000002,"15min":1.0589999999999999},"min":{"1min":0.68999999999999995,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":6.5060000000000002},"last":0.81100000000000005}]},{"osd":2,"last update":"Wed Feb 10 04:57:48 2021","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.1910000000000001,"15min":1.702},"min":{"1min":0.77600000000000002,"5min":0.53900000000000003,"15min":0.51000000000000001},"max":{"1min":1.889,"5min":11.776,"15min":120.39},"last":0.70499999999999996},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.129,"15min":1.6779999999999999},"min":{"1min":0.82499999999999996,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.4019999999999999,"5min":11.795999999999999,"15min":120.583},"last":0.99299999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:57:05 2021","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.115,"15min":1.2430000000000001},"min":{"1min":0.89900000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.998,"5min":1.998,"15min":2.1429999999999998},"last":0.91200000000000003},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.5600000000000001,"15min":1.54},"min":{"1min":0.82399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":2.1030000000000002,"5min":26.802,"15min":26.802},"last":1.0209999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:57:02 2021","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.0780000000000001,"15min":1.458},"min":{"1min":0.85399999999999998,"5min":0.495,"15min":0.495},"max":{"1min":1.6220000000000001,"5min":2.351,"15min":35.774000000000001},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.1890000000000001,"15min":1.454},"min":{"1min":0.72799999999999998,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.7030000000000001,"5min":4.319,"15min":35.606000000000002},"last":1.1479999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:57:05 2021","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.575,"15min":2.3690000000000002},"min":{"1min":1.0289999999999999,"5min":0.86199999999999999,"15min":0.82599999999999996},"max":{"1min":2.3639999999999999,"5min":12.488,"15min":201.626},"last":1.131},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.464,"15min":2.3679999999999999},"min":{"1min":0.95399999999999996,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":2.5489999999999999,"5min":12.84,"15min":201.577},"last":1.3300000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:57:37 2021","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.4470000000000001,"15min":1.508},"min":{"1min":1.111,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.2170000000000001,"5min":4.6200000000000001,"15min":5.8920000000000003},"last":1.244},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.2609999999999999,"15min":1.4590000000000001},"min":{"1min":0.99399999999999999,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":3.423,"5min":3.423,"15min":5.149},"last":1.2210000000000001}]}]},{"osd":5,"up_from":37,"seq":158913790351,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":256816,"kb_used_data":206440,"kb_used_omap":0,"kb_used_meta":50368,"kb_avail":93488336,"statfs":{"total":95995035648,"available":95732056064,"internally_reserved":0,"allocated":211394560,"data_stored":275884990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":51576832},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:57:18 2021","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.1799999999999999,"5min":1.486,"15min":1.486},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.76400000000000001,"15min":0.753},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.51600000000000001}]},{"osd":1,"last update":"Wed Feb 10 04:56:59 2021","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.90300000000000002,"15min":0.90900000000000003},"min":{"1min":0.57899999999999996,"5min":0.435,"15min":0.435},"max":{"1min":1.754,"5min":6.391,"15min":6.391},"last":0.995},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.21,"15min":1.1539999999999999},"min":{"1min":0.56000000000000005,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":12.260999999999999,"5min":13.794,"15min":13.794},"last":0.70299999999999996}]},{"osd":2,"last update":"Wed Feb 10 04:56:59 2021","interfaces":[{"interface":"back","average":{"1min":7.3460000000000001,"5min":4.0110000000000001,"15min":2.1669999999999998},"min":{"1min":0.58699999999999997,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":116.438,"5min":116.438,"15min":116.438},"last":1.0960000000000001},{"interface":"front","average":{"1min":7.3730000000000002,"5min":4.1550000000000002,"15min":2.2269999999999999},"min":{"1min":0.70299999999999996,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":116.96899999999999,"5min":116.96899999999999,"15min":116.96899999999999},"last":1.194}]},{"osd":3,"last update":"Wed Feb 10 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.3300000000000001,"15min":1.333},"min":{"1min":0.81100000000000005,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":12.428000000000001,"5min":13.755000000000001,"15min":13.755000000000001},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.234,"15min":1.198},"min":{"1min":0.79700000000000004,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.95,"5min":6.5129999999999999,"15min":6.5129999999999999},"last":1.319}]},{"osd":4,"last update":"Wed Feb 10 04:56:59 2021","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.1819999999999999,"15min":1.2689999999999999},"min":{"1min":0.67000000000000004,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.478,"5min":9.4890000000000008,"15min":9.4890000000000008},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.125,"5min":1.222,"15min":1.357},"min":{"1min":0.77100000000000002,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.8080000000000001,"5min":4.8289999999999997,"15min":5.6180000000000003},"last":1.292}]},{"osd":6,"last update":"Wed Feb 10 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.397,"15min":2.9590000000000001},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.2149999999999999,"5min":4.9429999999999996,"15min":319.51999999999998},"last":1.6200000000000001},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.448,"15min":2.895},"min":{"1min":0.83499999999999996,"5min":0.71399999999999997,"15min":0.68799999999999994},"max":{"1min":2.7290000000000001,"5min":6.8220000000000001,"15min":319.22300000000001},"last":1.448}]},{"osd":7,"last update":"Wed Feb 10 04:57:41 2021","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.244,"15min":1.4550000000000001},"min":{"1min":0.754,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.8009999999999999,"5min":5.4160000000000004,"15min":11.897},"last":1.254},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.4099999999999999,"15min":1.548},"min":{"1min":1.0189999999999999,"5min":0.91000000000000003,"15min":0.82999999999999996},"max":{"1min":2.4369999999999998,"5min":5.0579999999999998,"15min":6.0369999999999999},"last":1.532}]}]}]}} 2021-02-10T04:58:06.040 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:58:06.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:05 smithi164 conmon[29288]: audit 2021-02-10T04:58:04.387294+0000 mgr.y (mgr.14140 2021-02-10T04:58:06.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:05 smithi164 conmon[29288]: ) 1469 : audit [DBG] from='client.16404 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:58:06.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:05 smithi164 conmon[29288]: cluster 2021-02-10T04:58:04.541675+0000 mgr.y (mgr.14140) 1470 : cluster [DBG] pgmap v2396: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:58:06.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:05 smithi164 conmon[29288]: cluster 2021-02-10T04:58:04.895541+0000 mon.a (mon.0) 2859 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:58:06.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:05 smithi164 conmon[29288]: cluster 2021-02-10T04:58:04.895573+0000 mon.a (mon.0) 2860 : cluster [INF] Cluster is now healthy 2021-02-10T04:58:06.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:05 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:58:06.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:05 smithi164 conmon[29288]: :58:04.904379+0000 mon.a (mon.0) 2861 : audit [INF] from='client.? 172.21.15.85:0/1199555575' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:58:06.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:05 smithi164 conmon[29288]: cluster 2021-02-10T04:58:04.904451+0000 2021-02-10T04:58:06.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:05 smithi164 conmon[29288]: mon.a (mon.0) 2862 : cluster [DBG] osdmap e1314: 8 total, 8 up, 8 in 2021-02-10T04:58:06.276 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:58:06.790 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:58:06.814 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:58:06.835 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:58:06.856 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:58:06.876 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:58:06.899 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:58:06.922 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:58:06.943 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:58:06.944 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:58:06.944 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:58:06.944 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:58:06.945 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:58:06.945 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:58:06.945 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:58:06.945 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:58:06.945 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:58:07.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:06 smithi085 conmon[35961]: audit 2021-02-10T04:58:06.768748+0000 mon.c (mon.1 2021-02-10T04:58:07.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:06 smithi085 conmon[35961]: ) 148 : audit [DBG] from='client.? 172.21.15.85:0/1007986408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:58:07.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:06 smithi085 conmon[29319]: audit 2021-02-10T04:58:06.768748+0000 mon.c (mon. 2021-02-10T04:58:07.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:06 smithi085 conmon[29319]: 1) 148 : audit [DBG] from='client.? 172.21.15.85:0/1007986408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:58:07.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:06 smithi164 conmon[29288]: audit 2021-02-10T04:58:07.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:06 smithi164 conmon[29288]: 2021-02-10T04:58: 2021-02-10T04:58:07.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:06 smithi164 conmon[29288]: 06.768748+0000 mon.c (mon.1) 148 : audit [DBG] from='client.? 172.21.15.85:0/1007986408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:58:08.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:07 smithi085 conmon[29319]: cluster 2021-02-10T04:58:06.542148+0000 mgr.y (mgr.14140) 1471 : cluster [DBG] pgmap v2398: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-02-10T04:58:08.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:07 smithi085 conmon[35961]: cluster 2021-02-10T04:58:06.542148+0000 mgr.y (mgr.14140) 1471 : cluster [DBG] pgmap v2398: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-02-10T04:58:08.304 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set noscrub 2021-02-10T04:58:08.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:07 smithi164 conmon[29288]: cluster 2021-02-10T04:58:06.542148+0000 mgr.y (mgr.14140) 2021-02-10T04:58:08.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:07 smithi164 conmon[29288]: 1471 : cluster [DBG] pgmap v2398: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2021-02-10T04:58:08.536 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:58:09.001 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:58:09.004 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:58:09.611 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:09 smithi164 conmon[29288]: cluster 2021-02-10T04:58:08.542994+0000 mgr.y (mgr.14140) 1472 : cluster [DBG] 2021-02-10T04:58:09.611 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:09 smithi164 conmon[29288]: pgmap v2399: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:58:09.611 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:09 smithi164 conmon[29288]: cluster 2021-02-10T04:58:08.609741+0000 mon.a (mon.0) 2863 : cluster [DBG] osdmap e1315: 8 total, 8 up, 8 in 2021-02-10T04:58:09.611 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:09 smithi164 conmon[29288]: audit 2021-02-10T04:58:08.981523+0000 mgr.y (mgr.14140) 1473 : audit [DBG] from='client.26311 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:58:09.612 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:58:09 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:58:09.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:09 smithi085 conmon[29319]: cluster 2021-02-10T04:58:08.542994+0000 mgr.y (mgr.14140 2021-02-10T04:58:09.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:09 smithi085 conmon[29319]: ) 1472 : cluster [DBG] pgmap v2399: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:58:09.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:09 smithi085 conmon[29319]: cluster 2021-02-10T04:58:08.609741+0000 mon.a (mon.0) 2863 : cluster [DBG] osdmap e1315: 8 total, 8 up, 8 in 2021-02-10T04:58:09.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:09 smithi085 conmon[29319]: audit 2021-02-10T04:58:08.981523+0000 mgr.y (mgr.14140) 1473 : audit [DBG] from='client.26311 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:58:09.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:09 smithi085 conmon[35961]: cluster 2021-02-10T04:58:08. 2021-02-10T04:58:09.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:09 smithi085 conmon[35961]: 542994+0000 mgr.y (mgr.14140) 1472 : cluster [DBG] pgmap v2399: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:58:09.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:09 smithi085 conmon[35961]: cluster 2021-02-10T04:58:08.609741+0000 mon.a (mon.0) 2863 : cluster [DBG] osdmap e1315: 8 total, 8 up, 8 in 2021-02-10T04:58:09.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:09 smithi085 conmon[35961]: audit 2021-02-10T04:58:08.981523+0000 mgr.y (mgr.14140) 1473 : audit [DBG] from='client.26311 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:58:10.336 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:58:10.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:09 smithi164 conmon[29288]: 2021-02-10T04:58:10.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:58:10 smithi164 conmon[37263]: debug 2021-02-10T04:58:10.135+0000 7f1c679e3700 -1 osd.5 pg_epoch: 1315 pg[6.a( empty local-lis/les=1227/1228 n=0 ec=59/59 lis/c=1227/1227 les/c/f=1228/1228/0 sis=1227) [5,6,0] r=0 lpr=1227 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x55664fde78c0) 2021-02-10T04:58:10.473 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":2400,"stamp":"2021-02-10T04:58:08.607817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":972357089,"num_objects":748,"num_object_clones":337,"num_object_copies":2244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":748,"num_whiteouts":10,"num_read":7985,"num_read_kb":4562384,"num_write":4841,"num_write_kb":1247750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":887,"num_bytes_recovered":311980434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35650,"ondisk_log_size":35650,"up":594,"acting":594,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":596,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2311196,"kb_used_data":1984732,"kb_used_omap":692,"kb_used_meta":325707,"kb_avail":747650020,"statfs":{"total":767960285184,"available":765593620480,"internally_reserved":0,"allocated":2032365568,"data_stored":2600323606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":709615,"internal_metadata":333523985},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.719449"},"pg_stats":[{"pgid":"6.29","version":"0'0","reported_seq":"1170","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.278251+0000","last_change":"2021-02-10T04:55:14.633086+0000","last_active":"2021-02-10T04:58:02.278251+0000","last_peered":"2021-02-10T04:58:02.278251+0000","last_clean":"2021-02-10T04:58:02.278251+0000","last_became_active":"2021-02-10T04:55:14.632670+0000","last_became_peered":"2021-02-10T04:55:14.632670+0000","last_unstale":"2021-02-10T04:58:02.278251+0000","last_undegraded":"2021-02-10T04:58:02.278251+0000","last_fullsized":"2021-02-10T04:58:02.278251+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:24.121684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:24.121684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":"210","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.892189+0000","last_change":"2021-02-10T04:52:30.133105+0000","last_active":"2021-02-10T04:58:01.892189+0000","last_peered":"2021-02-10T04:58:01.892189+0000","last_clean":"2021-02-10T04:58:01.892189+0000","last_became_active":"2021-02-10T04:51:36.864926+0000","last_became_peered":"2021-02-10T04:51:36.864926+0000","last_unstale":"2021-02-10T04:58:01.892189+0000","last_undegraded":"2021-02-10T04:58:01.892189+0000","last_fullsized":"2021-02-10T04:58:01.892189+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.133043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.133043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"1191'4","reported_seq":"316","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:03.064399+0000","last_change":"2021-02-10T04:53:46.736164+0000","last_active":"2021-02-10T04:58:03.064399+0000","last_peered":"2021-02-10T04:58:03.064399+0000","last_clean":"2021-02-10T04:58:03.064399+0000","last_became_active":"2021-02-10T04:53:46.735711+0000","last_became_peered":"2021-02-10T04:53:46.735711+0000","last_unstale":"2021-02-10T04:58:03.064399+0000","last_undegraded":"2021-02-10T04:58:03.064399+0000","last_fullsized":"2021-02-10T04:58:03.064399+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'3","last_scrub_stamp":"2021-02-10T04:51:01.185820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.185820+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"1261'84","reported_seq":"1446","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.888143+0000","last_change":"2021-02-10T04:55:39.594594+0000","last_active":"2021-02-10T04:58:01.888143+0000","last_peered":"2021-02-10T04:58:01.888143+0000","last_clean":"2021-02-10T04:58:01.888143+0000","last_became_active":"2021-02-10T04:55:39.594275+0000","last_became_peered":"2021-02-10T04:55:39.594275+0000","last_unstale":"2021-02-10T04:58:01.888143+0000","last_undegraded":"2021-02-10T04:58:01.888143+0000","last_fullsized":"2021-02-10T04:58:01.888143+0000","mapping_epoch":55,"log_start":"1132'66","ondisk_log_start":"1132'66","created":55,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":0,"last_scrub":"1132'66","last_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_deep_scrub":"1132'66","last_deep_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.430399+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1616,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":186,"num_read_kb":150,"num_write":107,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"219","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.894539+0000","last_change":"2021-02-10T04:53:41.722609+0000","last_active":"2021-02-10T04:58:01.894539+0000","last_peered":"2021-02-10T04:58:01.894539+0000","last_clean":"2021-02-10T04:58:01.894539+0000","last_became_active":"2021-02-10T04:51:36.875000+0000","last_became_peered":"2021-02-10T04:51:36.875000+0000","last_unstale":"2021-02-10T04:58:01.894539+0000","last_undegraded":"2021-02-10T04:58:01.894539+0000","last_fullsized":"2021-02-10T04:58:01.894539+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:53:41.722516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:41.722516+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"1356","reported_epoch":"1314","state":"active+clean","last_fresh":"2021-02-10T04:58:06.189009+0000","last_change":"2021-02-10T04:58:06.189844+0000","last_active":"2021-02-10T04:58:06.189009+0000","last_peered":"2021-02-10T04:58:06.189009+0000","last_clean":"2021-02-10T04:58:06.189009+0000","last_became_active":"2021-02-10T04:55:14.630535+0000","last_became_peered":"2021-02-10T04:55:14.630535+0000","last_unstale":"2021-02-10T04:58:06.189009+0000","last_undegraded":"2021-02-10T04:58:06.189009+0000","last_fullsized":"2021-02-10T04:58:06.189009+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:48.408748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:50:48.408748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"1003'1689","reported_seq":"4733","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.414069+0000","last_change":"2021-02-10T04:53:38.170873+0000","last_active":"2021-02-10T04:58:02.414069+0000","last_peered":"2021-02-10T04:58:02.414069+0000","last_clean":"2021-02-10T04:58:02.414069+0000","last_became_active":"2021-02-10T04:52:26.030105+0000","last_became_peered":"2021-02-10T04:52:26.030105+0000","last_unstale":"2021-02-10T04:58:02.414069+0000","last_undegraded":"2021-02-10T04:58:02.414069+0000","last_fullsized":"2021-02-10T04:58:02.414069+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:53:38.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.170802+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":99361981,"num_objects":37,"num_object_clones":33,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"62'5","reported_seq":"326","reported_epoch":"1314","state":"active+clean","last_fresh":"2021-02-10T04:58:06.841056+0000","last_change":"2021-02-10T04:54:39.615784+0000","last_active":"2021-02-10T04:58:06.841056+0000","last_peered":"2021-02-10T04:58:06.841056+0000","last_clean":"2021-02-10T04:58:06.841056+0000","last_became_active":"2021-02-10T04:54:39.560009+0000","last_became_peered":"2021-02-10T04:54:39.560009+0000","last_unstale":"2021-02-10T04:58:06.841056+0000","last_undegraded":"2021-02-10T04:58:06.841056+0000","last_fullsized":"2021-02-10T04:58:06.841056+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":359,"num_read_kb":358,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"1261'28","reported_seq":"1323","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.413202+0000","last_change":"2021-02-10T04:55:21.669679+0000","last_active":"2021-02-10T04:58:02.413202+0000","last_peered":"2021-02-10T04:58:02.413202+0000","last_clean":"2021-02-10T04:58:02.413202+0000","last_became_active":"2021-02-10T04:55:21.669219+0000","last_became_peered":"2021-02-10T04:55:21.669219+0000","last_unstale":"2021-02-10T04:58:02.413202+0000","last_undegraded":"2021-02-10T04:58:02.413202+0000","last_fullsized":"2021-02-10T04:58:02.413202+0000","mapping_epoch":1228,"log_start":"1061'20","ondisk_log_start":"1061'20","created":55,"last_epoch_clean":1233,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'32","last_scrub_stamp":"2021-02-10T04:51:51.382040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.382040+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1158,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":105,"num_read_kb":81,"num_write":58,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2052,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"965","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.415149+0000","last_change":"2021-02-10T04:53:34.179015+0000","last_active":"2021-02-10T04:58:02.415149+0000","last_peered":"2021-02-10T04:58:02.415149+0000","last_clean":"2021-02-10T04:58:02.415149+0000","last_became_active":"2021-02-10T04:51:49.502576+0000","last_became_peered":"2021-02-10T04:51:49.502576+0000","last_unstale":"2021-02-10T04:58:02.415149+0000","last_undegraded":"2021-02-10T04:58:02.415149+0000","last_fullsized":"2021-02-10T04:58:02.415149+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:34.178949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.178949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"210","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.892458+0000","last_change":"2021-02-10T04:51:53.038893+0000","last_active":"2021-02-10T04:58:01.892458+0000","last_peered":"2021-02-10T04:58:01.892458+0000","last_clean":"2021-02-10T04:58:01.892458+0000","last_became_active":"2021-02-10T04:51:36.855507+0000","last_became_peered":"2021-02-10T04:51:36.855507+0000","last_unstale":"2021-02-10T04:58:01.892458+0000","last_undegraded":"2021-02-10T04:58:01.892458+0000","last_fullsized":"2021-02-10T04:58:01.892458+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:53.038834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:53.038834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"1232","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.412337+0000","last_change":"2021-02-10T04:55:28.097406+0000","last_active":"2021-02-10T04:58:02.412337+0000","last_peered":"2021-02-10T04:58:02.412337+0000","last_clean":"2021-02-10T04:58:02.412337+0000","last_became_active":"2021-02-10T04:55:14.629603+0000","last_became_peered":"2021-02-10T04:55:14.629603+0000","last_unstale":"2021-02-10T04:58:02.412337+0000","last_undegraded":"2021-02-10T04:58:02.412337+0000","last_fullsized":"2021-02-10T04:58:02.412337+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:09.922789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:09.922789+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"1168'82","reported_seq":"246","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.892329+0000","last_change":"2021-02-10T04:53:23.326363+0000","last_active":"2021-02-10T04:58:01.892329+0000","last_peered":"2021-02-10T04:58:01.892329+0000","last_clean":"2021-02-10T04:58:01.892329+0000","last_became_active":"2021-02-10T04:51:36.859215+0000","last_became_peered":"2021-02-10T04:51:36.859215+0000","last_unstale":"2021-02-10T04:58:01.892329+0000","last_undegraded":"2021-02-10T04:58:01.892329+0000","last_fullsized":"2021-02-10T04:58:01.892329+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1150'76","last_scrub_stamp":"2021-02-10T04:53:00.964499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:00.964499+0000","log_size":82,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3848803,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"1261'69","reported_seq":"1452","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.874486+0000","last_change":"2021-02-10T04:55:25.576927+0000","last_active":"2021-02-10T04:57:59.874486+0000","last_peered":"2021-02-10T04:57:59.874486+0000","last_clean":"2021-02-10T04:57:59.874486+0000","last_became_active":"2021-02-10T04:55:25.576633+0000","last_became_peered":"2021-02-10T04:55:25.576633+0000","last_unstale":"2021-02-10T04:57:59.874486+0000","last_undegraded":"2021-02-10T04:57:59.874486+0000","last_fullsized":"2021-02-10T04:57:59.874486+0000","mapping_epoch":55,"log_start":"1112'47","ondisk_log_start":"1112'47","created":55,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'61","last_scrub_stamp":"2021-02-10T04:53:36.836202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:36.836202+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1466,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":208,"num_read_kb":172,"num_write":122,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"1228'6","reported_seq":"309","reported_epoch":"1314","state":"active+clean","last_fresh":"2021-02-10T04:58:04.906466+0000","last_change":"2021-02-10T04:55:14.605868+0000","last_active":"2021-02-10T04:58:04.906466+0000","last_peered":"2021-02-10T04:58:04.906466+0000","last_clean":"2021-02-10T04:58:04.906466+0000","last_became_active":"2021-02-10T04:55:14.605314+0000","last_became_peered":"2021-02-10T04:55:14.605314+0000","last_unstale":"2021-02-10T04:58:04.906466+0000","last_undegraded":"2021-02-10T04:58:04.906466+0000","last_fullsized":"2021-02-10T04:58:04.906466+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'4","last_scrub_stamp":"2021-02-10T04:52:01.121380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.121380+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"225","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.894809+0000","last_change":"2021-02-10T04:55:22.573715+0000","last_active":"2021-02-10T04:58:01.894809+0000","last_peered":"2021-02-10T04:58:01.894809+0000","last_clean":"2021-02-10T04:58:01.894809+0000","last_became_active":"2021-02-10T04:55:14.636895+0000","last_became_peered":"2021-02-10T04:55:14.636895+0000","last_unstale":"2021-02-10T04:58:01.894809+0000","last_undegraded":"2021-02-10T04:58:01.894809+0000","last_fullsized":"2021-02-10T04:58:01.894809+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:55:22.573621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:55:22.573621+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"610","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.278219+0000","last_change":"2021-02-10T04:53:46.733236+0000","last_active":"2021-02-10T04:58:02.278219+0000","last_peered":"2021-02-10T04:58:02.278219+0000","last_clean":"2021-02-10T04:58:02.278219+0000","last_became_active":"2021-02-10T04:53:46.732890+0000","last_became_peered":"2021-02-10T04:53:46.732890+0000","last_unstale":"2021-02-10T04:58:02.278219+0000","last_undegraded":"2021-02-10T04:58:02.278219+0000","last_fullsized":"2021-02-10T04:58:02.278219+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:35.170157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:35.170157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"1003'739","reported_seq":"3911","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.412371+0000","last_change":"2021-02-10T04:55:28.091164+0000","last_active":"2021-02-10T04:58:02.412371+0000","last_peered":"2021-02-10T04:58:02.412371+0000","last_clean":"2021-02-10T04:58:02.412371+0000","last_became_active":"2021-02-10T04:55:14.648601+0000","last_became_peered":"2021-02-10T04:55:14.648601+0000","last_unstale":"2021-02-10T04:58:02.412371+0000","last_undegraded":"2021-02-10T04:58:02.412371+0000","last_fullsized":"2021-02-10T04:58:02.412371+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:52:27.602006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:27.602006+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24059888,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":129,"num_read_kb":109544,"num_write":102,"num_write_kb":46173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"1289","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.873798+0000","last_change":"2021-02-10T04:54:39.559980+0000","last_active":"2021-02-10T04:57:59.873798+0000","last_peered":"2021-02-10T04:57:59.873798+0000","last_clean":"2021-02-10T04:57:59.873798+0000","last_became_active":"2021-02-10T04:54:39.559443+0000","last_became_peered":"2021-02-10T04:54:39.559443+0000","last_unstale":"2021-02-10T04:57:59.873798+0000","last_undegraded":"2021-02-10T04:57:59.873798+0000","last_fullsized":"2021-02-10T04:57:59.873798+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"1261'64","reported_seq":"1498","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.888958+0000","last_change":"2021-02-10T04:53:37.108747+0000","last_active":"2021-02-10T04:58:01.888958+0000","last_peered":"2021-02-10T04:58:01.888958+0000","last_clean":"2021-02-10T04:58:01.888958+0000","last_became_active":"2021-02-10T04:53:35.551311+0000","last_became_peered":"2021-02-10T04:53:35.551311+0000","last_unstale":"2021-02-10T04:58:01.888958+0000","last_undegraded":"2021-02-10T04:58:01.888958+0000","last_fullsized":"2021-02-10T04:58:01.888958+0000","mapping_epoch":1118,"log_start":"1066'44","ondisk_log_start":"1066'44","created":55,"last_epoch_clean":1181,"parent":"0.0","parent_split_bits":0,"last_scrub":"1066'44","last_scrub_stamp":"2021-02-10T04:53:37.108677+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:53:37.108677+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1578,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":180,"num_read_kb":144,"num_write":106,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":2312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"1208","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.888858+0000","last_change":"2021-02-10T04:53:39.126356+0000","last_active":"2021-02-10T04:58:01.888858+0000","last_peered":"2021-02-10T04:58:01.888858+0000","last_clean":"2021-02-10T04:58:01.888858+0000","last_became_active":"2021-02-10T04:29:07.145616+0000","last_became_peered":"2021-02-10T04:29:07.145616+0000","last_unstale":"2021-02-10T04:58:01.888858+0000","last_undegraded":"2021-02-10T04:58:01.888858+0000","last_fullsized":"2021-02-10T04:58:01.888858+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:39.126261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:39.126261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"540","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.415084+0000","last_change":"2021-02-10T04:55:14.611192+0000","last_active":"2021-02-10T04:58:02.415084+0000","last_peered":"2021-02-10T04:58:02.415084+0000","last_clean":"2021-02-10T04:58:02.415084+0000","last_became_active":"2021-02-10T04:55:14.610821+0000","last_became_peered":"2021-02-10T04:55:14.610821+0000","last_unstale":"2021-02-10T04:58:02.415084+0000","last_undegraded":"2021-02-10T04:58:02.415084+0000","last_fullsized":"2021-02-10T04:58:02.415084+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:26.311958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.311958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"242","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.894758+0000","last_change":"2021-02-10T04:55:03.526061+0000","last_active":"2021-02-10T04:58:01.894758+0000","last_peered":"2021-02-10T04:58:01.894758+0000","last_clean":"2021-02-10T04:58:01.894758+0000","last_became_active":"2021-02-10T04:51:36.855744+0000","last_became_peered":"2021-02-10T04:51:36.855744+0000","last_unstale":"2021-02-10T04:58:01.894758+0000","last_undegraded":"2021-02-10T04:58:01.894758+0000","last_fullsized":"2021-02-10T04:58:01.894758+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:02.162851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:13.727241+0000","last_clean_scrub_stamp":"2021-02-10T04:52:02.162851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"1063'654","reported_seq":"4489","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.894451+0000","last_change":"2021-02-10T04:55:14.637619+0000","last_active":"2021-02-10T04:58:01.894451+0000","last_peered":"2021-02-10T04:58:01.894451+0000","last_clean":"2021-02-10T04:58:01.894451+0000","last_became_active":"2021-02-10T04:55:14.636752+0000","last_became_peered":"2021-02-10T04:55:14.636752+0000","last_unstale":"2021-02-10T04:58:01.894451+0000","last_undegraded":"2021-02-10T04:58:01.894451+0000","last_fullsized":"2021-02-10T04:58:01.894451+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:51:57.121838+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.121838+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24700630,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":171,"num_read_kb":182272,"num_write":111,"num_write_kb":33727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":26349023,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"1259","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.889114+0000","last_change":"2021-02-10T04:51:50.433056+0000","last_active":"2021-02-10T04:58:01.889114+0000","last_peered":"2021-02-10T04:58:01.889114+0000","last_clean":"2021-02-10T04:58:01.889114+0000","last_became_active":"2021-02-10T04:51:36.843165+0000","last_became_peered":"2021-02-10T04:51:36.843165+0000","last_unstale":"2021-02-10T04:58:01.889114+0000","last_undegraded":"2021-02-10T04:58:01.889114+0000","last_fullsized":"2021-02-10T04:58:01.889114+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.432990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.432990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"1280","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.891093+0000","last_change":"2021-02-10T04:55:14.638290+0000","last_active":"2021-02-10T04:58:01.891093+0000","last_peered":"2021-02-10T04:58:01.891093+0000","last_clean":"2021-02-10T04:58:01.891093+0000","last_became_active":"2021-02-10T04:55:14.637927+0000","last_became_peered":"2021-02-10T04:55:14.637927+0000","last_unstale":"2021-02-10T04:58:01.891093+0000","last_undegraded":"2021-02-10T04:58:01.891093+0000","last_fullsized":"2021-02-10T04:58:01.891093+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"1205","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.278340+0000","last_change":"2021-02-10T04:53:46.727888+0000","last_active":"2021-02-10T04:58:02.278340+0000","last_peered":"2021-02-10T04:58:02.278340+0000","last_clean":"2021-02-10T04:58:02.278340+0000","last_became_active":"2021-02-10T04:53:46.727670+0000","last_became_peered":"2021-02-10T04:53:46.727670+0000","last_unstale":"2021-02-10T04:58:02.278340+0000","last_undegraded":"2021-02-10T04:58:02.278340+0000","last_fullsized":"2021-02-10T04:58:02.278340+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:41.041553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.041553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"1261'76","reported_seq":"1422","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.888755+0000","last_change":"2021-02-10T04:53:42.071173+0000","last_active":"2021-02-10T04:58:01.888755+0000","last_peered":"2021-02-10T04:58:01.888755+0000","last_clean":"2021-02-10T04:58:01.888755+0000","last_became_active":"2021-02-10T04:53:39.679752+0000","last_became_peered":"2021-02-10T04:53:39.679752+0000","last_unstale":"2021-02-10T04:58:01.888755+0000","last_undegraded":"2021-02-10T04:58:01.888755+0000","last_fullsized":"2021-02-10T04:58:01.888755+0000","mapping_epoch":1111,"log_start":"1049'39","ondisk_log_start":"1049'39","created":55,"last_epoch_clean":1185,"parent":"0.0","parent_split_bits":0,"last_scrub":"1187'54","last_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_deep_scrub":"1187'54","last_deep_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_clean_scrub_stamp":"2021-02-10T04:53:42.071103+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":147,"num_read_kb":127,"num_write":95,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"950","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.414639+0000","last_change":"2021-02-10T04:51:52.350764+0000","last_active":"2021-02-10T04:58:02.414639+0000","last_peered":"2021-02-10T04:58:02.414639+0000","last_clean":"2021-02-10T04:58:02.414639+0000","last_became_active":"2021-02-10T04:51:49.567784+0000","last_became_peered":"2021-02-10T04:51:49.567784+0000","last_unstale":"2021-02-10T04:58:02.414639+0000","last_undegraded":"2021-02-10T04:58:02.414639+0000","last_fullsized":"2021-02-10T04:58:02.414639+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:52.350706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.350706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"1119'4","reported_seq":"293","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:03.064083+0000","last_change":"2021-02-10T04:52:27.126670+0000","last_active":"2021-02-10T04:58:03.064083+0000","last_peered":"2021-02-10T04:58:03.064083+0000","last_clean":"2021-02-10T04:58:03.064083+0000","last_became_active":"2021-02-10T04:51:36.853647+0000","last_became_peered":"2021-02-10T04:51:36.853647+0000","last_unstale":"2021-02-10T04:58:03.064083+0000","last_undegraded":"2021-02-10T04:58:03.064083+0000","last_fullsized":"2021-02-10T04:58:03.064083+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'4","last_scrub_stamp":"2021-02-10T04:52:27.126599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:27.126599+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"1299","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.889255+0000","last_change":"2021-02-10T04:52:25.745660+0000","last_active":"2021-02-10T04:58:01.889255+0000","last_peered":"2021-02-10T04:58:01.889255+0000","last_clean":"2021-02-10T04:58:01.889255+0000","last_became_active":"2021-02-10T04:51:36.849503+0000","last_became_peered":"2021-02-10T04:51:36.849503+0000","last_unstale":"2021-02-10T04:58:01.889255+0000","last_undegraded":"2021-02-10T04:58:01.889255+0000","last_fullsized":"2021-02-10T04:58:01.889255+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:25.745577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.745577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"1004'1093","reported_seq":"6494","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.413132+0000","last_change":"2021-02-10T04:55:28.097530+0000","last_active":"2021-02-10T04:58:02.413132+0000","last_peered":"2021-02-10T04:58:02.413132+0000","last_clean":"2021-02-10T04:58:02.413132+0000","last_became_active":"2021-02-10T04:55:14.627630+0000","last_became_peered":"2021-02-10T04:55:14.627630+0000","last_unstale":"2021-02-10T04:58:02.413132+0000","last_undegraded":"2021-02-10T04:58:02.413132+0000","last_fullsized":"2021-02-10T04:58:02.413132+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39434696,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":279,"num_read_kb":311058,"num_write":185,"num_write_kb":66790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"62'5","reported_seq":"112","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.889521+0000","last_change":"2021-02-10T04:54:39.559473+0000","last_active":"2021-02-10T04:58:01.889521+0000","last_peered":"2021-02-10T04:58:01.889521+0000","last_clean":"2021-02-10T04:58:01.889521+0000","last_became_active":"2021-02-10T04:54:39.559048+0000","last_became_peered":"2021-02-10T04:54:39.559048+0000","last_unstale":"2021-02-10T04:58:01.889521+0000","last_undegraded":"2021-02-10T04:58:01.889521+0000","last_fullsized":"2021-02-10T04:58:01.889521+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":149,"num_read_kb":148,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"1268","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.890901+0000","last_change":"2021-02-10T04:55:03.230261+0000","last_active":"2021-02-10T04:58:01.890901+0000","last_peered":"2021-02-10T04:58:01.890901+0000","last_clean":"2021-02-10T04:58:01.890901+0000","last_became_active":"2021-02-10T04:51:36.841582+0000","last_became_peered":"2021-02-10T04:51:36.841582+0000","last_unstale":"2021-02-10T04:58:01.890901+0000","last_undegraded":"2021-02-10T04:58:01.890901+0000","last_fullsized":"2021-02-10T04:58:01.890901+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:12.389272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:12.389272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"1261'71","reported_seq":"1016","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.415701+0000","last_change":"2021-02-10T04:53:27.560299+0000","last_active":"2021-02-10T04:58:02.415701+0000","last_peered":"2021-02-10T04:58:02.415701+0000","last_clean":"2021-02-10T04:58:02.415701+0000","last_became_active":"2021-02-10T04:53:27.559886+0000","last_became_peered":"2021-02-10T04:53:27.559886+0000","last_unstale":"2021-02-10T04:58:02.415701+0000","last_undegraded":"2021-02-10T04:58:02.415701+0000","last_fullsized":"2021-02-10T04:58:02.415701+0000","mapping_epoch":1129,"log_start":"1049'51","ondisk_log_start":"1049'51","created":55,"last_epoch_clean":1173,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'41","last_scrub_stamp":"2021-02-10T04:51:58.353660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:58.353660+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1472,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":169,"num_read_kb":136,"num_write":102,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"973","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.414680+0000","last_change":"2021-02-10T04:53:11.271557+0000","last_active":"2021-02-10T04:58:02.414680+0000","last_peered":"2021-02-10T04:58:02.414680+0000","last_clean":"2021-02-10T04:58:02.414680+0000","last_became_active":"2021-02-10T04:51:49.567883+0000","last_became_peered":"2021-02-10T04:51:49.567883+0000","last_unstale":"2021-02-10T04:58:02.414680+0000","last_undegraded":"2021-02-10T04:58:02.414680+0000","last_fullsized":"2021-02-10T04:58:02.414680+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:11.271459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.271459+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"1229'5","reported_seq":"1689","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:03.064406+0000","last_change":"2021-02-10T04:55:14.638050+0000","last_active":"2021-02-10T04:58:03.064406+0000","last_peered":"2021-02-10T04:58:03.064406+0000","last_clean":"2021-02-10T04:58:03.064406+0000","last_became_active":"2021-02-10T04:55:14.637570+0000","last_became_peered":"2021-02-10T04:55:14.637570+0000","last_unstale":"2021-02-10T04:58:03.064406+0000","last_undegraded":"2021-02-10T04:58:03.064406+0000","last_fullsized":"2021-02-10T04:58:03.064406+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:51:47.855283+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.855283+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"1303","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.892308+0000","last_change":"2021-02-10T04:55:22.183630+0000","last_active":"2021-02-10T04:58:01.892308+0000","last_peered":"2021-02-10T04:58:01.892308+0000","last_clean":"2021-02-10T04:58:01.892308+0000","last_became_active":"2021-02-10T04:55:14.656661+0000","last_became_peered":"2021-02-10T04:55:14.656661+0000","last_unstale":"2021-02-10T04:58:01.892308+0000","last_undegraded":"2021-02-10T04:58:01.892308+0000","last_fullsized":"2021-02-10T04:58:01.892308+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:55:22.183566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:55:22.183566+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"1002'1090","reported_seq":"5506","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.874644+0000","last_change":"2021-02-10T04:52:26.030198+0000","last_active":"2021-02-10T04:57:59.874644+0000","last_peered":"2021-02-10T04:57:59.874644+0000","last_clean":"2021-02-10T04:57:59.874644+0000","last_became_active":"2021-02-10T04:52:26.028182+0000","last_became_peered":"2021-02-10T04:52:26.028182+0000","last_unstale":"2021-02-10T04:57:59.874644+0000","last_undegraded":"2021-02-10T04:57:59.874644+0000","last_fullsized":"2021-02-10T04:57:59.874644+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:51:08.220472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.220472+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24410646,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":245,"num_read_kb":276168,"num_write":175,"num_write_kb":64960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"1279","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.888027+0000","last_change":"2021-02-10T04:54:39.555379+0000","last_active":"2021-02-10T04:58:01.888027+0000","last_peered":"2021-02-10T04:58:01.888027+0000","last_clean":"2021-02-10T04:58:01.888027+0000","last_became_active":"2021-02-10T04:54:39.554976+0000","last_became_peered":"2021-02-10T04:54:39.554976+0000","last_unstale":"2021-02-10T04:58:01.888027+0000","last_undegraded":"2021-02-10T04:58:01.888027+0000","last_fullsized":"2021-02-10T04:58:01.888027+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"1261'65","reported_seq":"1411","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.889012+0000","last_change":"2021-02-10T04:53:31.624862+0000","last_active":"2021-02-10T04:58:01.889012+0000","last_peered":"2021-02-10T04:58:01.889012+0000","last_clean":"2021-02-10T04:58:01.889012+0000","last_became_active":"2021-02-10T04:53:31.624318+0000","last_became_peered":"2021-02-10T04:53:31.624318+0000","last_unstale":"2021-02-10T04:58:01.889012+0000","last_undegraded":"2021-02-10T04:58:01.889012+0000","last_fullsized":"2021-02-10T04:58:01.889012+0000","mapping_epoch":55,"log_start":"1049'47","ondisk_log_start":"1049'47","created":55,"last_epoch_clean":1177,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'47","last_scrub_stamp":"2021-02-10T04:53:09.002368+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.002368+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1672,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":169,"num_read_kb":135,"num_write":98,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":934,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"250","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.278256+0000","last_change":"2021-02-10T04:55:14.611324+0000","last_active":"2021-02-10T04:58:02.278256+0000","last_peered":"2021-02-10T04:58:02.278256+0000","last_clean":"2021-02-10T04:58:02.278256+0000","last_became_active":"2021-02-10T04:55:14.610986+0000","last_became_peered":"2021-02-10T04:55:14.610986+0000","last_unstale":"2021-02-10T04:58:02.278256+0000","last_undegraded":"2021-02-10T04:58:02.278256+0000","last_fullsized":"2021-02-10T04:58:02.278256+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:08.358300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:50:08.358300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"1290","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.892881+0000","last_change":"2021-02-10T04:51:57.436961+0000","last_active":"2021-02-10T04:58:01.892881+0000","last_peered":"2021-02-10T04:58:01.892881+0000","last_clean":"2021-02-10T04:58:01.892881+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:58:01.892881+0000","last_undegraded":"2021-02-10T04:58:01.892881+0000","last_fullsized":"2021-02-10T04:58:01.892881+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.436862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.436862+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"1245","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.892505+0000","last_change":"2021-02-10T04:53:02.112612+0000","last_active":"2021-02-10T04:58:01.892505+0000","last_peered":"2021-02-10T04:58:01.892505+0000","last_clean":"2021-02-10T04:58:01.892505+0000","last_became_active":"2021-02-10T04:50:59.545501+0000","last_became_peered":"2021-02-10T04:50:59.545501+0000","last_unstale":"2021-02-10T04:58:01.892505+0000","last_undegraded":"2021-02-10T04:58:01.892505+0000","last_fullsized":"2021-02-10T04:58:01.892505+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:02.112533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:02.112533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"1001'1165","reported_seq":"4485","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.875099+0000","last_change":"2021-02-10T04:55:14.653651+0000","last_active":"2021-02-10T04:57:59.875099+0000","last_peered":"2021-02-10T04:57:59.875099+0000","last_clean":"2021-02-10T04:57:59.875099+0000","last_became_active":"2021-02-10T04:55:14.652235+0000","last_became_peered":"2021-02-10T04:55:14.652235+0000","last_unstale":"2021-02-10T04:57:59.875099+0000","last_undegraded":"2021-02-10T04:57:59.875099+0000","last_fullsized":"2021-02-10T04:57:59.875099+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29601845,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":286,"num_read_kb":299529,"num_write":200,"num_write_kb":89130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"95","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.412781+0000","last_change":"2021-02-10T04:55:15.590113+0000","last_active":"2021-02-10T04:58:02.412781+0000","last_peered":"2021-02-10T04:58:02.412781+0000","last_clean":"2021-02-10T04:58:02.412781+0000","last_became_active":"2021-02-10T04:55:15.589751+0000","last_became_peered":"2021-02-10T04:55:15.589751+0000","last_unstale":"2021-02-10T04:58:02.412781+0000","last_undegraded":"2021-02-10T04:58:02.412781+0000","last_fullsized":"2021-02-10T04:58:02.412781+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"1113'5","reported_seq":"1575","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:03.064485+0000","last_change":"2021-02-10T04:52:34.047474+0000","last_active":"2021-02-10T04:58:03.064485+0000","last_peered":"2021-02-10T04:58:03.064485+0000","last_clean":"2021-02-10T04:58:03.064485+0000","last_became_active":"2021-02-10T04:50:59.569578+0000","last_became_peered":"2021-02-10T04:50:59.569578+0000","last_unstale":"2021-02-10T04:58:03.064485+0000","last_undegraded":"2021-02-10T04:58:03.064485+0000","last_fullsized":"2021-02-10T04:58:03.064485+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1113'5","last_scrub_stamp":"2021-02-10T04:52:34.047358+0000","last_deep_scrub":"158'3","last_deep_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_clean_scrub_stamp":"2021-02-10T04:52:34.047358+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"1261'68","reported_seq":"1396","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.888380+0000","last_change":"2021-02-10T04:51:48.872937+0000","last_active":"2021-02-10T04:58:01.888380+0000","last_peered":"2021-02-10T04:58:01.888380+0000","last_clean":"2021-02-10T04:58:01.888380+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:58:01.888380+0000","last_undegraded":"2021-02-10T04:58:01.888380+0000","last_fullsized":"2021-02-10T04:58:01.888380+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:51:48.872839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.872839+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":114,"num_read_kb":94,"num_write":68,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"1363","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.875536+0000","last_change":"2021-02-10T04:53:05.984747+0000","last_active":"2021-02-10T04:57:59.875536+0000","last_peered":"2021-02-10T04:57:59.875536+0000","last_clean":"2021-02-10T04:57:59.875536+0000","last_became_active":"2021-02-10T04:51:36.869931+0000","last_became_peered":"2021-02-10T04:51:36.869931+0000","last_unstale":"2021-02-10T04:57:59.875536+0000","last_undegraded":"2021-02-10T04:57:59.875536+0000","last_fullsized":"2021-02-10T04:57:59.875536+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:05.984659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:05.984659+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"1167","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.278685+0000","last_change":"2021-02-10T04:53:46.731709+0000","last_active":"2021-02-10T04:58:02.278685+0000","last_peered":"2021-02-10T04:58:02.278685+0000","last_clean":"2021-02-10T04:58:02.278685+0000","last_became_active":"2021-02-10T04:53:46.731340+0000","last_became_peered":"2021-02-10T04:53:46.731340+0000","last_unstale":"2021-02-10T04:58:02.278685+0000","last_undegraded":"2021-02-10T04:58:02.278685+0000","last_fullsized":"2021-02-10T04:58:02.278685+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:09.005421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.005421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"998'1009","reported_seq":"4804","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.890107+0000","last_change":"2021-02-10T04:52:26.037227+0000","last_active":"2021-02-10T04:58:01.890107+0000","last_peered":"2021-02-10T04:58:01.890107+0000","last_clean":"2021-02-10T04:58:01.890107+0000","last_became_active":"2021-02-10T04:52:26.036723+0000","last_became_peered":"2021-02-10T04:52:26.036723+0000","last_unstale":"2021-02-10T04:58:01.890107+0000","last_undegraded":"2021-02-10T04:58:01.890107+0000","last_fullsized":"2021-02-10T04:58:01.890107+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12355206,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":119,"num_read_kb":124701,"num_write":77,"num_write_kb":25052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4145029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"1184","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.893004+0000","last_change":"2021-02-10T04:53:33.305896+0000","last_active":"2021-02-10T04:58:01.893004+0000","last_peered":"2021-02-10T04:58:01.893004+0000","last_clean":"2021-02-10T04:58:01.893004+0000","last_became_active":"2021-02-10T04:50:59.570854+0000","last_became_peered":"2021-02-10T04:50:59.570854+0000","last_unstale":"2021-02-10T04:58:01.893004+0000","last_undegraded":"2021-02-10T04:58:01.893004+0000","last_fullsized":"2021-02-10T04:58:01.893004+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:33.305827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:33.305827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"1285","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.892743+0000","last_change":"2021-02-10T04:55:15.669988+0000","last_active":"2021-02-10T04:58:01.892743+0000","last_peered":"2021-02-10T04:58:01.892743+0000","last_clean":"2021-02-10T04:58:01.892743+0000","last_became_active":"2021-02-10T04:55:15.669642+0000","last_became_peered":"2021-02-10T04:55:15.669642+0000","last_unstale":"2021-02-10T04:58:01.892743+0000","last_undegraded":"2021-02-10T04:58:01.892743+0000","last_fullsized":"2021-02-10T04:58:01.892743+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"1298","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.888534+0000","last_change":"2021-02-10T04:51:50.895471+0000","last_active":"2021-02-10T04:58:01.888534+0000","last_peered":"2021-02-10T04:58:01.888534+0000","last_clean":"2021-02-10T04:58:01.888534+0000","last_became_active":"2021-02-10T04:50:59.527553+0000","last_became_peered":"2021-02-10T04:50:59.527553+0000","last_unstale":"2021-02-10T04:58:01.888534+0000","last_undegraded":"2021-02-10T04:58:01.888534+0000","last_fullsized":"2021-02-10T04:58:01.888534+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.895370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.895370+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"1261'84","reported_seq":"1429","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.890059+0000","last_change":"2021-02-10T04:55:14.603109+0000","last_active":"2021-02-10T04:58:01.890059+0000","last_peered":"2021-02-10T04:58:01.890059+0000","last_clean":"2021-02-10T04:58:01.890059+0000","last_became_active":"2021-02-10T04:55:14.602669+0000","last_became_peered":"2021-02-10T04:55:14.602669+0000","last_unstale":"2021-02-10T04:58:01.890059+0000","last_undegraded":"2021-02-10T04:58:01.890059+0000","last_fullsized":"2021-02-10T04:58:01.890059+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'72","last_scrub_stamp":"2021-02-10T04:46:30.515026+0000","last_deep_scrub":"152'48","last_deep_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_clean_scrub_stamp":"2021-02-10T04:46:30.515026+0000","log_size":84,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":123,"num_read_kb":101,"num_write":72,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"567","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.415369+0000","last_change":"2021-02-10T04:55:03.834050+0000","last_active":"2021-02-10T04:58:02.415369+0000","last_peered":"2021-02-10T04:58:02.415369+0000","last_clean":"2021-02-10T04:58:02.415369+0000","last_became_active":"2021-02-10T04:51:49.503801+0000","last_became_peered":"2021-02-10T04:51:49.503801+0000","last_unstale":"2021-02-10T04:58:02.415369+0000","last_undegraded":"2021-02-10T04:58:02.415369+0000","last_fullsized":"2021-02-10T04:58:02.415369+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:51:48.274960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.274960+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"1323","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.890056+0000","last_change":"2021-02-10T04:53:13.099288+0000","last_active":"2021-02-10T04:58:01.890056+0000","last_peered":"2021-02-10T04:58:01.890056+0000","last_clean":"2021-02-10T04:58:01.890056+0000","last_became_active":"2021-02-10T04:51:36.848614+0000","last_became_peered":"2021-02-10T04:51:36.848614+0000","last_unstale":"2021-02-10T04:58:01.890056+0000","last_undegraded":"2021-02-10T04:58:01.890056+0000","last_fullsized":"2021-02-10T04:58:01.890056+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:13.099228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:13.099228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"1002'1657","reported_seq":"5374","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.890545+0000","last_change":"2021-02-10T04:52:26.036806+0000","last_active":"2021-02-10T04:58:01.890545+0000","last_peered":"2021-02-10T04:58:01.890545+0000","last_clean":"2021-02-10T04:58:01.890545+0000","last_became_active":"2021-02-10T04:52:26.032363+0000","last_became_peered":"2021-02-10T04:52:26.032363+0000","last_unstale":"2021-02-10T04:58:01.890545+0000","last_undegraded":"2021-02-10T04:58:01.890545+0000","last_fullsized":"2021-02-10T04:58:01.890545+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21396533,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":212,"num_read_kb":218836,"num_write":159,"num_write_kb":59398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":22326014,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"1242","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.888605+0000","last_change":"2021-02-10T04:51:51.911945+0000","last_active":"2021-02-10T04:58:01.888605+0000","last_peered":"2021-02-10T04:58:01.888605+0000","last_clean":"2021-02-10T04:58:01.888605+0000","last_became_active":"2021-02-10T04:51:36.850015+0000","last_became_peered":"2021-02-10T04:51:36.850015+0000","last_unstale":"2021-02-10T04:58:01.888605+0000","last_undegraded":"2021-02-10T04:58:01.888605+0000","last_fullsized":"2021-02-10T04:58:01.888605+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.911851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.911851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"62'5","reported_seq":"1146","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.414867+0000","last_change":"2021-02-10T04:54:39.568253+0000","last_active":"2021-02-10T04:58:02.414867+0000","last_peered":"2021-02-10T04:58:02.414867+0000","last_clean":"2021-02-10T04:58:02.414867+0000","last_became_active":"2021-02-10T04:54:39.567642+0000","last_became_peered":"2021-02-10T04:54:39.567642+0000","last_unstale":"2021-02-10T04:58:02.414867+0000","last_undegraded":"2021-02-10T04:58:02.414867+0000","last_fullsized":"2021-02-10T04:58:02.414867+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":98,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":150,"num_read_kb":148,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"1370","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.874714+0000","last_change":"2021-02-10T04:55:14.634577+0000","last_active":"2021-02-10T04:57:59.874714+0000","last_peered":"2021-02-10T04:57:59.874714+0000","last_clean":"2021-02-10T04:57:59.874714+0000","last_became_active":"2021-02-10T04:55:14.634259+0000","last_became_peered":"2021-02-10T04:55:14.634259+0000","last_unstale":"2021-02-10T04:57:59.874714+0000","last_undegraded":"2021-02-10T04:57:59.874714+0000","last_fullsized":"2021-02-10T04:57:59.874714+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:58.142526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:58.142526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"1261'68","reported_seq":"1322","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.412850+0000","last_change":"2021-02-10T04:55:28.079376+0000","last_active":"2021-02-10T04:58:02.412850+0000","last_peered":"2021-02-10T04:58:02.412850+0000","last_clean":"2021-02-10T04:58:02.412850+0000","last_became_active":"2021-02-10T04:55:14.654609+0000","last_became_peered":"2021-02-10T04:55:14.654609+0000","last_unstale":"2021-02-10T04:58:02.412850+0000","last_undegraded":"2021-02-10T04:58:02.412850+0000","last_fullsized":"2021-02-10T04:58:02.412850+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1135'58","last_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_deep_scrub":"1135'58","last_deep_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.483067+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":102,"num_read_kb":83,"num_write":60,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"248","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.279100+0000","last_change":"2021-02-10T04:55:14.657913+0000","last_active":"2021-02-10T04:58:02.279100+0000","last_peered":"2021-02-10T04:58:02.279100+0000","last_clean":"2021-02-10T04:58:02.279100+0000","last_became_active":"2021-02-10T04:55:14.657153+0000","last_became_peered":"2021-02-10T04:55:14.657153+0000","last_unstale":"2021-02-10T04:58:02.279100+0000","last_undegraded":"2021-02-10T04:58:02.279100+0000","last_fullsized":"2021-02-10T04:58:02.279100+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:50:46.434629+0000","last_deep_scrub":"53'2","last_deep_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_clean_scrub_stamp":"2021-02-10T04:50:46.434629+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"990","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.414626+0000","last_change":"2021-02-10T04:52:38.278980+0000","last_active":"2021-02-10T04:58:02.414626+0000","last_peered":"2021-02-10T04:58:02.414626+0000","last_clean":"2021-02-10T04:58:02.414626+0000","last_became_active":"2021-02-10T04:51:49.496237+0000","last_became_peered":"2021-02-10T04:51:49.496237+0000","last_unstale":"2021-02-10T04:58:02.414626+0000","last_undegraded":"2021-02-10T04:58:02.414626+0000","last_fullsized":"2021-02-10T04:58:02.414626+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:38.278910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:52:38.278910+0000","last_clean_scrub_stamp":"2021-02-10T04:52:38.278910+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"1000'1071","reported_seq":"4864","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.890729+0000","last_change":"2021-02-10T04:53:34.407875+0000","last_active":"2021-02-10T04:58:01.890729+0000","last_peered":"2021-02-10T04:58:01.890729+0000","last_clean":"2021-02-10T04:58:01.890729+0000","last_became_active":"2021-02-10T04:52:26.031050+0000","last_became_peered":"2021-02-10T04:52:26.031050+0000","last_unstale":"2021-02-10T04:58:01.890729+0000","last_undegraded":"2021-02-10T04:58:01.890729+0000","last_fullsized":"2021-02-10T04:58:01.890729+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:34.407784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.407784+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30173843,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"1336","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.874903+0000","last_change":"2021-02-10T04:55:14.637570+0000","last_active":"2021-02-10T04:57:59.874903+0000","last_peered":"2021-02-10T04:57:59.874903+0000","last_clean":"2021-02-10T04:57:59.874903+0000","last_became_active":"2021-02-10T04:55:14.637203+0000","last_became_peered":"2021-02-10T04:55:14.637203+0000","last_unstale":"2021-02-10T04:57:59.874903+0000","last_undegraded":"2021-02-10T04:57:59.874903+0000","last_fullsized":"2021-02-10T04:57:59.874903+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_clean_scrub_stamp":"2021-02-10T04:51:00.195518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"1300","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.892955+0000","last_change":"2021-02-10T04:55:14.661285+0000","last_active":"2021-02-10T04:58:01.892955+0000","last_peered":"2021-02-10T04:58:01.892955+0000","last_clean":"2021-02-10T04:58:01.892955+0000","last_became_active":"2021-02-10T04:55:14.660902+0000","last_became_peered":"2021-02-10T04:55:14.660902+0000","last_unstale":"2021-02-10T04:58:01.892955+0000","last_undegraded":"2021-02-10T04:58:01.892955+0000","last_fullsized":"2021-02-10T04:58:01.892955+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:55.473347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:50:55.473347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"1261'58","reported_seq":"1373","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.412844+0000","last_change":"2021-02-10T04:55:47.652017+0000","last_active":"2021-02-10T04:58:02.412844+0000","last_peered":"2021-02-10T04:58:02.412844+0000","last_clean":"2021-02-10T04:58:02.412844+0000","last_became_active":"2021-02-10T04:55:47.651592+0000","last_became_peered":"2021-02-10T04:55:47.651592+0000","last_unstale":"2021-02-10T04:58:02.412844+0000","last_undegraded":"2021-02-10T04:58:02.412844+0000","last_fullsized":"2021-02-10T04:58:02.412844+0000","mapping_epoch":1227,"log_start":"1049'40","ondisk_log_start":"1049'40","created":55,"last_epoch_clean":1257,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'40","last_scrub_stamp":"2021-02-10T04:51:47.281929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.281929+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1836,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":171,"num_read_kb":135,"num_write":99,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"474","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.278297+0000","last_change":"2021-02-10T04:55:14.610765+0000","last_active":"2021-02-10T04:58:02.278297+0000","last_peered":"2021-02-10T04:58:02.278297+0000","last_clean":"2021-02-10T04:58:02.278297+0000","last_became_active":"2021-02-10T04:55:14.610470+0000","last_became_peered":"2021-02-10T04:55:14.610470+0000","last_unstale":"2021-02-10T04:58:02.278297+0000","last_undegraded":"2021-02-10T04:58:02.278297+0000","last_fullsized":"2021-02-10T04:58:02.278297+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:50.399350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:50.399350+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"1200","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.412207+0000","last_change":"2021-02-10T04:55:25.624031+0000","last_active":"2021-02-10T04:58:02.412207+0000","last_peered":"2021-02-10T04:58:02.412207+0000","last_clean":"2021-02-10T04:58:02.412207+0000","last_became_active":"2021-02-10T04:55:14.630129+0000","last_became_peered":"2021-02-10T04:55:14.630129+0000","last_unstale":"2021-02-10T04:58:02.412207+0000","last_undegraded":"2021-02-10T04:58:02.412207+0000","last_fullsized":"2021-02-10T04:58:02.412207+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"975","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.415143+0000","last_change":"2021-02-10T04:52:31.273037+0000","last_active":"2021-02-10T04:58:02.415143+0000","last_peered":"2021-02-10T04:58:02.415143+0000","last_clean":"2021-02-10T04:58:02.415143+0000","last_became_active":"2021-02-10T04:51:49.501405+0000","last_became_peered":"2021-02-10T04:51:49.501405+0000","last_unstale":"2021-02-10T04:58:02.415143+0000","last_undegraded":"2021-02-10T04:58:02.415143+0000","last_fullsized":"2021-02-10T04:58:02.415143+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:31.272971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:17.598113+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.272971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"1000'446","reported_seq":"2031","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.278927+0000","last_change":"2021-02-10T04:55:14.658783+0000","last_active":"2021-02-10T04:58:02.278927+0000","last_peered":"2021-02-10T04:58:02.278927+0000","last_clean":"2021-02-10T04:58:02.278927+0000","last_became_active":"2021-02-10T04:55:14.657668+0000","last_became_peered":"2021-02-10T04:55:14.657668+0000","last_unstale":"2021-02-10T04:58:02.278927+0000","last_undegraded":"2021-02-10T04:58:02.278927+0000","last_fullsized":"2021-02-10T04:58:02.278927+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:51:45.964029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.964029+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22255852,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68435,"num_write":72,"num_write_kb":24345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":403693,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"466","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.414790+0000","last_change":"2021-02-10T04:53:03.228260+0000","last_active":"2021-02-10T04:58:02.414790+0000","last_peered":"2021-02-10T04:58:02.414790+0000","last_clean":"2021-02-10T04:58:02.414790+0000","last_became_active":"2021-02-10T04:51:49.499252+0000","last_became_peered":"2021-02-10T04:51:49.499252+0000","last_unstale":"2021-02-10T04:58:02.414790+0000","last_undegraded":"2021-02-10T04:58:02.414790+0000","last_fullsized":"2021-02-10T04:58:02.414790+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:03.228182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.228182+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"218","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.280065+0000","last_change":"2021-02-10T04:53:46.727384+0000","last_active":"2021-02-10T04:58:02.280065+0000","last_peered":"2021-02-10T04:58:02.280065+0000","last_clean":"2021-02-10T04:58:02.280065+0000","last_became_active":"2021-02-10T04:53:46.726771+0000","last_became_peered":"2021-02-10T04:53:46.726771+0000","last_unstale":"2021-02-10T04:58:02.280065+0000","last_undegraded":"2021-02-10T04:58:02.280065+0000","last_fullsized":"2021-02-10T04:58:02.280065+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:33.948348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:33.948348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"1261'51","reported_seq":"1398","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.874683+0000","last_change":"2021-02-10T04:54:17.118810+0000","last_active":"2021-02-10T04:57:59.874683+0000","last_peered":"2021-02-10T04:57:59.874683+0000","last_clean":"2021-02-10T04:57:59.874683+0000","last_became_active":"2021-02-10T04:54:17.080053+0000","last_became_peered":"2021-02-10T04:54:17.080053+0000","last_unstale":"2021-02-10T04:57:59.874683+0000","last_undegraded":"2021-02-10T04:57:59.874683+0000","last_fullsized":"2021-02-10T04:57:59.874683+0000","mapping_epoch":1200,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1201,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'39","last_scrub_stamp":"2021-02-10T04:51:47.322331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.322331+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":51,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"246","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.278341+0000","last_change":"2021-02-10T04:55:14.636888+0000","last_active":"2021-02-10T04:58:02.278341+0000","last_peered":"2021-02-10T04:58:02.278341+0000","last_clean":"2021-02-10T04:58:02.278341+0000","last_became_active":"2021-02-10T04:55:14.634305+0000","last_became_peered":"2021-02-10T04:55:14.634305+0000","last_unstale":"2021-02-10T04:58:02.278341+0000","last_undegraded":"2021-02-10T04:58:02.278341+0000","last_fullsized":"2021-02-10T04:58:02.278341+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_clean_scrub_stamp":"2021-02-10T04:50:05.424262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"1484","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.891442+0000","last_change":"2021-02-10T04:52:37.324122+0000","last_active":"2021-02-10T04:58:01.891442+0000","last_peered":"2021-02-10T04:58:01.891442+0000","last_clean":"2021-02-10T04:58:01.891442+0000","last_became_active":"2021-02-10T04:51:36.843553+0000","last_became_peered":"2021-02-10T04:51:36.843553+0000","last_unstale":"2021-02-10T04:58:01.891442+0000","last_undegraded":"2021-02-10T04:58:01.891442+0000","last_fullsized":"2021-02-10T04:58:01.891442+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:37.324040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:37.324040+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"1290","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.890178+0000","last_change":"2021-02-10T04:54:37.612303+0000","last_active":"2021-02-10T04:58:01.890178+0000","last_peered":"2021-02-10T04:58:01.890178+0000","last_clean":"2021-02-10T04:58:01.890178+0000","last_became_active":"2021-02-10T04:54:37.611953+0000","last_became_peered":"2021-02-10T04:54:37.611953+0000","last_unstale":"2021-02-10T04:58:01.890178+0000","last_undegraded":"2021-02-10T04:58:01.890178+0000","last_fullsized":"2021-02-10T04:58:01.890178+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"1312","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.893035+0000","last_change":"2021-02-10T04:52:31.403904+0000","last_active":"2021-02-10T04:58:01.893035+0000","last_peered":"2021-02-10T04:58:01.893035+0000","last_clean":"2021-02-10T04:58:01.893035+0000","last_became_active":"2021-02-10T04:50:59.538009+0000","last_became_peered":"2021-02-10T04:50:59.538009+0000","last_unstale":"2021-02-10T04:58:01.893035+0000","last_undegraded":"2021-02-10T04:58:01.893035+0000","last_fullsized":"2021-02-10T04:58:01.893035+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:31.403824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.403824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"1000'490","reported_seq":"3780","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.412929+0000","last_change":"2021-02-10T04:55:22.716472+0000","last_active":"2021-02-10T04:58:02.412929+0000","last_peered":"2021-02-10T04:58:02.412929+0000","last_clean":"2021-02-10T04:58:02.412929+0000","last_became_active":"2021-02-10T04:55:14.644596+0000","last_became_peered":"2021-02-10T04:55:14.644596+0000","last_unstale":"2021-02-10T04:58:02.412929+0000","last_undegraded":"2021-02-10T04:58:02.412929+0000","last_fullsized":"2021-02-10T04:58:02.412929+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:47:02.875920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:02.875920+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19371627,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":141,"num_read_kb":148568,"num_write":85,"num_write_kb":31732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"210","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.894678+0000","last_change":"2021-02-10T04:53:01.972991+0000","last_active":"2021-02-10T04:58:01.894678+0000","last_peered":"2021-02-10T04:58:01.894678+0000","last_clean":"2021-02-10T04:58:01.894678+0000","last_became_active":"2021-02-10T04:51:36.863976+0000","last_became_peered":"2021-02-10T04:51:36.863976+0000","last_unstale":"2021-02-10T04:58:01.894678+0000","last_undegraded":"2021-02-10T04:58:01.894678+0000","last_fullsized":"2021-02-10T04:58:01.894678+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:01.972913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.972913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"1261'40","reported_seq":"1032","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.415652+0000","last_change":"2021-02-10T04:55:43.569281+0000","last_active":"2021-02-10T04:58:02.415652+0000","last_peered":"2021-02-10T04:58:02.415652+0000","last_clean":"2021-02-10T04:58:02.415652+0000","last_became_active":"2021-02-10T04:55:43.568878+0000","last_became_peered":"2021-02-10T04:55:43.568878+0000","last_unstale":"2021-02-10T04:58:02.415652+0000","last_undegraded":"2021-02-10T04:58:02.415652+0000","last_fullsized":"2021-02-10T04:58:02.415652+0000","mapping_epoch":1227,"log_start":"1063'30","ondisk_log_start":"1063'30","created":55,"last_epoch_clean":1253,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:56.431187+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.431187+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":802,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":116,"num_read_kb":94,"num_write":67,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"220","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.892236+0000","last_change":"2021-02-10T04:55:14.636849+0000","last_active":"2021-02-10T04:58:01.892236+0000","last_peered":"2021-02-10T04:58:01.892236+0000","last_clean":"2021-02-10T04:58:01.892236+0000","last_became_active":"2021-02-10T04:55:14.636237+0000","last_became_peered":"2021-02-10T04:55:14.636237+0000","last_unstale":"2021-02-10T04:58:01.892236+0000","last_undegraded":"2021-02-10T04:58:01.892236+0000","last_fullsized":"2021-02-10T04:58:01.892236+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:04.591695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:04.591695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"1139","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.415057+0000","last_change":"2021-02-10T04:54:37.680913+0000","last_active":"2021-02-10T04:58:02.415057+0000","last_peered":"2021-02-10T04:58:02.415057+0000","last_clean":"2021-02-10T04:58:02.415057+0000","last_became_active":"2021-02-10T04:54:37.680651+0000","last_became_peered":"2021-02-10T04:54:37.680651+0000","last_unstale":"2021-02-10T04:58:02.415057+0000","last_undegraded":"2021-02-10T04:58:02.415057+0000","last_fullsized":"2021-02-10T04:58:02.415057+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1210,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":149,"num_read_kb":148,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"1157","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.279054+0000","last_change":"2021-02-10T04:53:46.729138+0000","last_active":"2021-02-10T04:58:02.279054+0000","last_peered":"2021-02-10T04:58:02.279054+0000","last_clean":"2021-02-10T04:58:02.279054+0000","last_became_active":"2021-02-10T04:53:46.728845+0000","last_became_peered":"2021-02-10T04:53:46.728845+0000","last_unstale":"2021-02-10T04:58:02.279054+0000","last_undegraded":"2021-02-10T04:58:02.279054+0000","last_fullsized":"2021-02-10T04:58:02.279054+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:40.066354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:52:40.066354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"1000'620","reported_seq":"4404","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.412533+0000","last_change":"2021-02-10T04:55:28.089086+0000","last_active":"2021-02-10T04:58:02.412533+0000","last_peered":"2021-02-10T04:58:02.412533+0000","last_clean":"2021-02-10T04:58:02.412533+0000","last_became_active":"2021-02-10T04:55:14.640509+0000","last_became_peered":"2021-02-10T04:55:14.640509+0000","last_unstale":"2021-02-10T04:58:02.412533+0000","last_undegraded":"2021-02-10T04:58:02.412533+0000","last_fullsized":"2021-02-10T04:58:02.412533+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:51:54.444287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:54.444287+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14265195,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":144,"num_read_kb":128482,"num_write":129,"num_write_kb":53443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"1193","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.412409+0000","last_change":"2021-02-10T04:55:28.083286+0000","last_active":"2021-02-10T04:58:02.412409+0000","last_peered":"2021-02-10T04:58:02.412409+0000","last_clean":"2021-02-10T04:58:02.412409+0000","last_became_active":"2021-02-10T04:55:14.619846+0000","last_became_peered":"2021-02-10T04:55:14.619846+0000","last_unstale":"2021-02-10T04:58:02.412409+0000","last_undegraded":"2021-02-10T04:58:02.412409+0000","last_fullsized":"2021-02-10T04:58:02.412409+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:14.955544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:47:14.955544+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"1290","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.888672+0000","last_change":"2021-02-10T04:55:15.590025+0000","last_active":"2021-02-10T04:58:01.888672+0000","last_peered":"2021-02-10T04:58:01.888672+0000","last_clean":"2021-02-10T04:58:01.888672+0000","last_became_active":"2021-02-10T04:55:15.589651+0000","last_became_peered":"2021-02-10T04:55:15.589651+0000","last_unstale":"2021-02-10T04:58:01.888672+0000","last_undegraded":"2021-02-10T04:58:01.888672+0000","last_fullsized":"2021-02-10T04:58:01.888672+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"1296","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.412437+0000","last_change":"2021-02-10T04:55:28.090104+0000","last_active":"2021-02-10T04:58:02.412437+0000","last_peered":"2021-02-10T04:58:02.412437+0000","last_clean":"2021-02-10T04:58:02.412437+0000","last_became_active":"2021-02-10T04:55:14.650106+0000","last_became_peered":"2021-02-10T04:55:14.650106+0000","last_unstale":"2021-02-10T04:58:02.412437+0000","last_undegraded":"2021-02-10T04:58:02.412437+0000","last_fullsized":"2021-02-10T04:58:02.412437+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:27.908221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:47:27.908221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"1261'67","reported_seq":"964","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.414887+0000","last_change":"2021-02-10T04:52:30.277009+0000","last_active":"2021-02-10T04:58:02.414887+0000","last_peered":"2021-02-10T04:58:02.414887+0000","last_clean":"2021-02-10T04:58:02.414887+0000","last_became_active":"2021-02-10T04:51:49.501622+0000","last_became_peered":"2021-02-10T04:51:49.501622+0000","last_unstale":"2021-02-10T04:58:02.414887+0000","last_undegraded":"2021-02-10T04:58:02.414887+0000","last_fullsized":"2021-02-10T04:58:02.414887+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'55","last_scrub_stamp":"2021-02-10T04:52:30.276938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.276938+0000","log_size":67,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":116,"num_read_kb":92,"num_write":70,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"1308","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.891102+0000","last_change":"2021-02-10T04:51:59.420149+0000","last_active":"2021-02-10T04:58:01.891102+0000","last_peered":"2021-02-10T04:58:01.891102+0000","last_clean":"2021-02-10T04:58:01.891102+0000","last_became_active":"2021-02-10T04:28:21.125541+0000","last_became_peered":"2021-02-10T04:28:21.125541+0000","last_unstale":"2021-02-10T04:58:01.891102+0000","last_undegraded":"2021-02-10T04:58:01.891102+0000","last_fullsized":"2021-02-10T04:58:01.891102+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.420086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.420086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"1274","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.892803+0000","last_change":"2021-02-10T04:53:12.187225+0000","last_active":"2021-02-10T04:58:01.892803+0000","last_peered":"2021-02-10T04:58:01.892803+0000","last_clean":"2021-02-10T04:58:01.892803+0000","last_became_active":"2021-02-10T04:50:59.555689+0000","last_became_peered":"2021-02-10T04:50:59.555689+0000","last_unstale":"2021-02-10T04:58:01.892803+0000","last_undegraded":"2021-02-10T04:58:01.892803+0000","last_fullsized":"2021-02-10T04:58:01.892803+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.187154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.187154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"1002'2042","reported_seq":"9544","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.875491+0000","last_change":"2021-02-10T04:55:14.662208+0000","last_active":"2021-02-10T04:57:59.875491+0000","last_peered":"2021-02-10T04:57:59.875491+0000","last_clean":"2021-02-10T04:57:59.875491+0000","last_became_active":"2021-02-10T04:55:14.659601+0000","last_became_peered":"2021-02-10T04:55:14.659601+0000","last_unstale":"2021-02-10T04:57:59.875491+0000","last_undegraded":"2021-02-10T04:57:59.875491+0000","last_fullsized":"2021-02-10T04:57:59.875491+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:46:57.054230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:57.054230+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":85678898,"num_objects":35,"num_object_clones":29,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"1259","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.890899+0000","last_change":"2021-02-10T04:55:14.651301+0000","last_active":"2021-02-10T04:58:01.890899+0000","last_peered":"2021-02-10T04:58:01.890899+0000","last_clean":"2021-02-10T04:58:01.890899+0000","last_became_active":"2021-02-10T04:55:14.650386+0000","last_became_peered":"2021-02-10T04:55:14.650386+0000","last_unstale":"2021-02-10T04:58:01.890899+0000","last_undegraded":"2021-02-10T04:58:01.890899+0000","last_fullsized":"2021-02-10T04:58:01.890899+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:52.341527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:50:52.341527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"112","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.892018+0000","last_change":"2021-02-10T04:54:39.558380+0000","last_active":"2021-02-10T04:58:01.892018+0000","last_peered":"2021-02-10T04:58:01.892018+0000","last_clean":"2021-02-10T04:58:01.892018+0000","last_became_active":"2021-02-10T04:54:39.557995+0000","last_became_peered":"2021-02-10T04:54:39.557995+0000","last_unstale":"2021-02-10T04:58:01.892018+0000","last_undegraded":"2021-02-10T04:58:01.892018+0000","last_fullsized":"2021-02-10T04:58:01.892018+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"1238'4","reported_seq":"1520","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:03.064193+0000","last_change":"2021-02-10T04:55:28.095426+0000","last_active":"2021-02-10T04:58:03.064193+0000","last_peered":"2021-02-10T04:58:03.064193+0000","last_clean":"2021-02-10T04:58:03.064193+0000","last_became_active":"2021-02-10T04:55:14.645890+0000","last_became_peered":"2021-02-10T04:55:14.645890+0000","last_unstale":"2021-02-10T04:58:03.064193+0000","last_undegraded":"2021-02-10T04:58:03.064193+0000","last_fullsized":"2021-02-10T04:58:03.064193+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:54.466259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:54.466259+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"1261'64","reported_seq":"1010","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.414729+0000","last_change":"2021-02-10T04:53:01.304693+0000","last_active":"2021-02-10T04:58:02.414729+0000","last_peered":"2021-02-10T04:58:02.414729+0000","last_clean":"2021-02-10T04:58:02.414729+0000","last_became_active":"2021-02-10T04:51:49.498066+0000","last_became_peered":"2021-02-10T04:51:49.498066+0000","last_unstale":"2021-02-10T04:58:02.414729+0000","last_undegraded":"2021-02-10T04:58:02.414729+0000","last_fullsized":"2021-02-10T04:58:02.414729+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:53:01.304626+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.304626+0000","log_size":64,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":89,"num_read_kb":71,"num_write":52,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"571","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.414964+0000","last_change":"2021-02-10T04:55:14.610552+0000","last_active":"2021-02-10T04:58:02.414964+0000","last_peered":"2021-02-10T04:58:02.414964+0000","last_clean":"2021-02-10T04:58:02.414964+0000","last_became_active":"2021-02-10T04:55:14.609458+0000","last_became_peered":"2021-02-10T04:55:14.609458+0000","last_unstale":"2021-02-10T04:58:02.414964+0000","last_undegraded":"2021-02-10T04:58:02.414964+0000","last_fullsized":"2021-02-10T04:58:02.414964+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:49.447961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:49.447961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"1294","reported_epoch":"1314","state":"active+clean","last_fresh":"2021-02-10T04:58:06.677155+0000","last_change":"2021-02-10T04:58:06.677869+0000","last_active":"2021-02-10T04:58:06.677155+0000","last_peered":"2021-02-10T04:58:06.677155+0000","last_clean":"2021-02-10T04:58:06.677155+0000","last_became_active":"2021-02-10T04:55:14.614581+0000","last_became_peered":"2021-02-10T04:55:14.614581+0000","last_unstale":"2021-02-10T04:58:06.677155+0000","last_undegraded":"2021-02-10T04:58:06.677155+0000","last_fullsized":"2021-02-10T04:58:06.677155+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:07.914061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:07.914061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"293'77","reported_seq":"3095","reported_epoch":"1313","state":"active+clean+scrubbing","last_fresh":"2021-02-10T04:58:01.888140+0000","last_change":"2021-02-10T04:53:43.102301+0000","last_active":"2021-02-10T04:58:01.888140+0000","last_peered":"2021-02-10T04:58:01.888140+0000","last_clean":"2021-02-10T04:58:01.888140+0000","last_became_active":"2021-02-10T04:52:26.024812+0000","last_became_peered":"2021-02-10T04:52:26.024812+0000","last_unstale":"2021-02-10T04:58:01.888140+0000","last_undegraded":"2021-02-10T04:58:01.888140+0000","last_fullsized":"2021-02-10T04:58:01.888140+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1886585,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":18587,"num_write":9,"num_write_kb":3315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"1326","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.875475+0000","last_change":"2021-02-10T04:53:07.025068+0000","last_active":"2021-02-10T04:57:59.875475+0000","last_peered":"2021-02-10T04:57:59.875475+0000","last_clean":"2021-02-10T04:57:59.875475+0000","last_became_active":"2021-02-10T04:50:59.589417+0000","last_became_peered":"2021-02-10T04:50:59.589417+0000","last_unstale":"2021-02-10T04:57:59.875475+0000","last_undegraded":"2021-02-10T04:57:59.875475+0000","last_fullsized":"2021-02-10T04:57:59.875475+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.024980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.024980+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"286","reported_epoch":"1314","state":"active+clean","last_fresh":"2021-02-10T04:58:05.490920+0000","last_change":"2021-02-10T04:58:05.491629+0000","last_active":"2021-02-10T04:58:05.490920+0000","last_peered":"2021-02-10T04:58:05.490920+0000","last_clean":"2021-02-10T04:58:05.490920+0000","last_became_active":"2021-02-10T04:55:14.605267+0000","last_became_peered":"2021-02-10T04:55:14.605267+0000","last_unstale":"2021-02-10T04:58:05.490920+0000","last_undegraded":"2021-02-10T04:58:05.490920+0000","last_fullsized":"2021-02-10T04:58:05.490920+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.609500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.609500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"1261'86","reported_seq":"1494","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.888424+0000","last_change":"2021-02-10T04:55:35.577794+0000","last_active":"2021-02-10T04:58:01.888424+0000","last_peered":"2021-02-10T04:58:01.888424+0000","last_clean":"2021-02-10T04:58:01.888424+0000","last_became_active":"2021-02-10T04:55:35.577316+0000","last_became_peered":"2021-02-10T04:55:35.577316+0000","last_unstale":"2021-02-10T04:58:01.888424+0000","last_undegraded":"2021-02-10T04:58:01.888424+0000","last_fullsized":"2021-02-10T04:58:01.888424+0000","mapping_epoch":1238,"log_start":"1049'68","ondisk_log_start":"1049'68","created":55,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'39","last_scrub_stamp":"2021-02-10T04:53:38.085565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.085565+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1582,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":227,"num_read_kb":187,"num_write":128,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"1427","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.889529+0000","last_change":"2021-02-10T04:55:14.659217+0000","last_active":"2021-02-10T04:58:01.889529+0000","last_peered":"2021-02-10T04:58:01.889529+0000","last_clean":"2021-02-10T04:58:01.889529+0000","last_became_active":"2021-02-10T04:55:14.658879+0000","last_became_peered":"2021-02-10T04:55:14.658879+0000","last_unstale":"2021-02-10T04:58:01.889529+0000","last_undegraded":"2021-02-10T04:58:01.889529+0000","last_fullsized":"2021-02-10T04:58:01.889529+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:47:08.784593+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:47:08.784593+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"1276","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.891021+0000","last_change":"2021-02-10T04:52:30.019225+0000","last_active":"2021-02-10T04:58:01.891021+0000","last_peered":"2021-02-10T04:58:01.891021+0000","last_clean":"2021-02-10T04:58:01.891021+0000","last_became_active":"2021-02-10T04:50:59.561436+0000","last_became_peered":"2021-02-10T04:50:59.561436+0000","last_unstale":"2021-02-10T04:58:01.891021+0000","last_undegraded":"2021-02-10T04:58:01.891021+0000","last_fullsized":"2021-02-10T04:58:01.891021+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.019102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.019102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"1001'1528","reported_seq":"6261","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.890025+0000","last_change":"2021-02-10T04:52:26.036366+0000","last_active":"2021-02-10T04:58:01.890025+0000","last_peered":"2021-02-10T04:58:01.890025+0000","last_clean":"2021-02-10T04:58:01.890025+0000","last_became_active":"2021-02-10T04:52:26.034028+0000","last_became_peered":"2021-02-10T04:52:26.034028+0000","last_unstale":"2021-02-10T04:58:01.890025+0000","last_undegraded":"2021-02-10T04:58:01.890025+0000","last_fullsized":"2021-02-10T04:58:01.890025+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46944136,"num_objects":16,"num_object_clones":15,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":358,"num_read_kb":351934,"num_write":286,"num_write_kb":111355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":58968832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.15","version":"1000'446","reported_seq":"2250","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.888952+0000","last_change":"2021-02-10T04:53:11.051124+0000","last_active":"2021-02-10T04:58:01.888952+0000","last_peered":"2021-02-10T04:58:01.888952+0000","last_clean":"2021-02-10T04:58:01.888952+0000","last_became_active":"2021-02-10T04:51:40.510899+0000","last_became_peered":"2021-02-10T04:51:40.510899+0000","last_unstale":"2021-02-10T04:58:01.888952+0000","last_undegraded":"2021-02-10T04:58:01.888952+0000","last_fullsized":"2021-02-10T04:58:01.888952+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:53:11.051031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.051031+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24896285,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":18983408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"224","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.892277+0000","last_change":"2021-02-10T04:55:14.605585+0000","last_active":"2021-02-10T04:58:01.892277+0000","last_peered":"2021-02-10T04:58:01.892277+0000","last_clean":"2021-02-10T04:58:01.892277+0000","last_became_active":"2021-02-10T04:55:14.604916+0000","last_became_peered":"2021-02-10T04:55:14.604916+0000","last_unstale":"2021-02-10T04:58:01.892277+0000","last_undegraded":"2021-02-10T04:58:01.892277+0000","last_fullsized":"2021-02-10T04:58:01.892277+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:51:08.639784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.639784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"1245","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.893079+0000","last_change":"2021-02-10T04:51:09.475150+0000","last_active":"2021-02-10T04:58:01.893079+0000","last_peered":"2021-02-10T04:58:01.893079+0000","last_clean":"2021-02-10T04:58:01.893079+0000","last_became_active":"2021-02-10T04:50:59.585917+0000","last_became_peered":"2021-02-10T04:50:59.585917+0000","last_unstale":"2021-02-10T04:58:01.893079+0000","last_undegraded":"2021-02-10T04:58:01.893079+0000","last_fullsized":"2021-02-10T04:58:01.893079+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:09.475063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:09.475063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"215","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.893484+0000","last_change":"2021-02-10T04:53:34.927483+0000","last_active":"2021-02-10T04:58:01.893484+0000","last_peered":"2021-02-10T04:58:01.893484+0000","last_clean":"2021-02-10T04:58:01.893484+0000","last_became_active":"2021-02-10T04:51:36.859707+0000","last_became_peered":"2021-02-10T04:51:36.859707+0000","last_unstale":"2021-02-10T04:58:01.893484+0000","last_undegraded":"2021-02-10T04:58:01.893484+0000","last_fullsized":"2021-02-10T04:58:01.893484+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:34.927424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.927424+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"1000'620","reported_seq":"4600","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.892563+0000","last_change":"2021-02-10T04:53:37.384461+0000","last_active":"2021-02-10T04:58:01.892563+0000","last_peered":"2021-02-10T04:58:01.892563+0000","last_clean":"2021-02-10T04:58:01.892563+0000","last_became_active":"2021-02-10T04:51:40.510114+0000","last_became_peered":"2021-02-10T04:51:40.510114+0000","last_unstale":"2021-02-10T04:58:01.892563+0000","last_undegraded":"2021-02-10T04:58:01.892563+0000","last_fullsized":"2021-02-10T04:58:01.892563+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:53:37.384398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:37.384398+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28530390,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"1300","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.874242+0000","last_change":"2021-02-10T04:52:41.059269+0000","last_active":"2021-02-10T04:57:59.874242+0000","last_peered":"2021-02-10T04:57:59.874242+0000","last_clean":"2021-02-10T04:57:59.874242+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:57:59.874242+0000","last_undegraded":"2021-02-10T04:57:59.874242+0000","last_fullsized":"2021-02-10T04:57:59.874242+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:52:41.059159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.059159+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"1305","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.892554+0000","last_change":"2021-02-10T04:53:12.332890+0000","last_active":"2021-02-10T04:58:01.892554+0000","last_peered":"2021-02-10T04:58:01.892554+0000","last_clean":"2021-02-10T04:58:01.892554+0000","last_became_active":"2021-02-10T04:50:59.540375+0000","last_became_peered":"2021-02-10T04:50:59.540375+0000","last_unstale":"2021-02-10T04:58:01.892554+0000","last_undegraded":"2021-02-10T04:58:01.892554+0000","last_fullsized":"2021-02-10T04:58:01.892554+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.332233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.332233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"288","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:03.260180+0000","last_change":"2021-02-10T04:58:03.260180+0000","last_active":"2021-02-10T04:58:03.260180+0000","last_peered":"2021-02-10T04:58:03.260180+0000","last_clean":"2021-02-10T04:58:03.260180+0000","last_became_active":"2021-02-10T04:53:46.733191+0000","last_became_peered":"2021-02-10T04:53:46.733191+0000","last_unstale":"2021-02-10T04:58:03.260180+0000","last_undegraded":"2021-02-10T04:58:03.260180+0000","last_fullsized":"2021-02-10T04:58:03.260180+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:45.362537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:45.362537+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"1002'2042","reported_seq":"9286","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.279551+0000","last_change":"2021-02-10T04:53:46.726353+0000","last_active":"2021-02-10T04:58:02.279551+0000","last_peered":"2021-02-10T04:58:02.279551+0000","last_clean":"2021-02-10T04:58:02.279551+0000","last_became_active":"2021-02-10T04:53:46.725828+0000","last_became_peered":"2021-02-10T04:53:46.725828+0000","last_unstale":"2021-02-10T04:58:02.279551+0000","last_undegraded":"2021-02-10T04:58:02.279551+0000","last_fullsized":"2021-02-10T04:58:02.279551+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:52:41.286006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.286006+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46300076,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":545,"num_read_kb":531689,"num_write":399,"num_write_kb":153087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"1261'56","reported_seq":"1484","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.892918+0000","last_change":"2021-02-10T04:52:35.091494+0000","last_active":"2021-02-10T04:58:01.892918+0000","last_peered":"2021-02-10T04:58:01.892918+0000","last_clean":"2021-02-10T04:58:01.892918+0000","last_became_active":"2021-02-10T04:50:59.569915+0000","last_became_peered":"2021-02-10T04:50:59.569915+0000","last_unstale":"2021-02-10T04:58:01.892918+0000","last_undegraded":"2021-02-10T04:58:01.892918+0000","last_fullsized":"2021-02-10T04:58:01.892918+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'46","last_scrub_stamp":"2021-02-10T04:52:35.091374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:35.091374+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":95,"num_read_kb":80,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"84'7","reported_seq":"1329","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.891061+0000","last_change":"2021-02-10T04:51:52.498251+0000","last_active":"2021-02-10T04:58:01.891061+0000","last_peered":"2021-02-10T04:58:01.891061+0000","last_clean":"2021-02-10T04:58:01.891061+0000","last_became_active":"2021-02-10T04:50:59.575924+0000","last_became_peered":"2021-02-10T04:50:59.575924+0000","last_unstale":"2021-02-10T04:58:01.891061+0000","last_undegraded":"2021-02-10T04:58:01.891061+0000","last_fullsized":"2021-02-10T04:58:01.891061+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'7","last_scrub_stamp":"2021-02-10T04:51:52.498158+0000","last_deep_scrub":"84'7","last_deep_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.498158+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"1229'4","reported_seq":"1691","reported_epoch":"1314","state":"active+clean","last_fresh":"2021-02-10T04:58:04.907834+0000","last_change":"2021-02-10T04:55:14.657278+0000","last_active":"2021-02-10T04:58:04.907834+0000","last_peered":"2021-02-10T04:58:04.907834+0000","last_clean":"2021-02-10T04:58:04.907834+0000","last_became_active":"2021-02-10T04:55:14.656867+0000","last_became_peered":"2021-02-10T04:55:14.656867+0000","last_unstale":"2021-02-10T04:58:04.907834+0000","last_undegraded":"2021-02-10T04:58:04.907834+0000","last_fullsized":"2021-02-10T04:58:04.907834+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:47.407557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:47.407557+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"1227","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.413338+0000","last_change":"2021-02-10T04:55:22.735243+0000","last_active":"2021-02-10T04:58:02.413338+0000","last_peered":"2021-02-10T04:58:02.413338+0000","last_clean":"2021-02-10T04:58:02.413338+0000","last_became_active":"2021-02-10T04:55:14.632398+0000","last_became_peered":"2021-02-10T04:55:14.632398+0000","last_unstale":"2021-02-10T04:58:02.413338+0000","last_undegraded":"2021-02-10T04:58:02.413338+0000","last_fullsized":"2021-02-10T04:58:02.413338+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:23.566027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_clean_scrub_stamp":"2021-02-10T04:52:23.566027+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"1000'490","reported_seq":"3829","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.889297+0000","last_change":"2021-02-10T04:55:14.644527+0000","last_active":"2021-02-10T04:58:01.889297+0000","last_peered":"2021-02-10T04:58:01.889297+0000","last_clean":"2021-02-10T04:58:01.889297+0000","last_became_active":"2021-02-10T04:55:14.643764+0000","last_became_peered":"2021-02-10T04:55:14.643764+0000","last_unstale":"2021-02-10T04:58:01.889297+0000","last_undegraded":"2021-02-10T04:58:01.889297+0000","last_fullsized":"2021-02-10T04:58:01.889297+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:50:03.756248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:50:03.756248+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16154790,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"1261'46","reported_seq":"1382","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.891936+0000","last_change":"2021-02-10T04:53:10.276449+0000","last_active":"2021-02-10T04:58:01.891936+0000","last_peered":"2021-02-10T04:58:01.891936+0000","last_clean":"2021-02-10T04:58:01.891936+0000","last_became_active":"2021-02-10T04:50:59.539529+0000","last_became_peered":"2021-02-10T04:50:59.539529+0000","last_unstale":"2021-02-10T04:58:01.891936+0000","last_undegraded":"2021-02-10T04:58:01.891936+0000","last_fullsized":"2021-02-10T04:58:01.891936+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1112'40","last_scrub_stamp":"2021-02-10T04:53:10.276358+0000","last_deep_scrub":"1112'40","last_deep_scrub_stamp":"2021-02-10T04:51:01.450050+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.276358+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":53,"num_write":38,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"1212","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.413103+0000","last_change":"2021-02-10T04:55:28.088480+0000","last_active":"2021-02-10T04:58:02.413103+0000","last_peered":"2021-02-10T04:58:02.413103+0000","last_clean":"2021-02-10T04:58:02.413103+0000","last_became_active":"2021-02-10T04:55:14.631666+0000","last_became_peered":"2021-02-10T04:55:14.631666+0000","last_unstale":"2021-02-10T04:58:02.413103+0000","last_undegraded":"2021-02-10T04:58:02.413103+0000","last_fullsized":"2021-02-10T04:58:02.413103+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:23.904210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:47:23.904210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"1119'6","reported_seq":"1604","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:03.064434+0000","last_change":"2021-02-10T04:52:40.038646+0000","last_active":"2021-02-10T04:58:03.064434+0000","last_peered":"2021-02-10T04:58:03.064434+0000","last_clean":"2021-02-10T04:58:03.064434+0000","last_became_active":"2021-02-10T04:51:36.869733+0000","last_became_peered":"2021-02-10T04:51:36.869733+0000","last_unstale":"2021-02-10T04:58:03.064434+0000","last_undegraded":"2021-02-10T04:58:03.064434+0000","last_fullsized":"2021-02-10T04:58:03.064434+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'6","last_scrub_stamp":"2021-02-10T04:52:40.038554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:40.038554+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"1299","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.889811+0000","last_change":"2021-02-10T04:51:36.849933+0000","last_active":"2021-02-10T04:58:01.889811+0000","last_peered":"2021-02-10T04:58:01.889811+0000","last_clean":"2021-02-10T04:58:01.889811+0000","last_became_active":"2021-02-10T04:51:36.849184+0000","last_became_peered":"2021-02-10T04:51:36.849184+0000","last_unstale":"2021-02-10T04:58:01.889811+0000","last_undegraded":"2021-02-10T04:58:01.889811+0000","last_fullsized":"2021-02-10T04:58:01.889811+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:11.779779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:11.779779+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"1032'89","reported_seq":"907","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.278233+0000","last_change":"2021-02-10T04:53:46.728868+0000","last_active":"2021-02-10T04:58:02.278233+0000","last_peered":"2021-02-10T04:58:02.278233+0000","last_clean":"2021-02-10T04:58:02.278233+0000","last_became_active":"2021-02-10T04:53:46.728356+0000","last_became_peered":"2021-02-10T04:53:46.728356+0000","last_unstale":"2021-02-10T04:58:02.278233+0000","last_undegraded":"2021-02-10T04:58:02.278233+0000","last_fullsized":"2021-02-10T04:58:02.278233+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":5,"last_scrub":"1032'89","last_scrub_stamp":"2021-02-10T04:52:34.089789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:34.089789+0000","log_size":89,"ondisk_log_size":89,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5218574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":30326,"num_write":21,"num_write_kb":12444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3210022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"1261'43","reported_seq":"1003","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.414663+0000","last_change":"2021-02-10T04:53:14.202505+0000","last_active":"2021-02-10T04:58:02.414663+0000","last_peered":"2021-02-10T04:58:02.414663+0000","last_clean":"2021-02-10T04:58:02.414663+0000","last_became_active":"2021-02-10T04:51:49.496294+0000","last_became_peered":"2021-02-10T04:51:49.496294+0000","last_unstale":"2021-02-10T04:58:02.414663+0000","last_undegraded":"2021-02-10T04:58:02.414663+0000","last_fullsized":"2021-02-10T04:58:02.414663+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1163'37","last_scrub_stamp":"2021-02-10T04:53:14.202445+0000","last_deep_scrub":"1163'37","last_deep_scrub_stamp":"2021-02-10T04:53:14.202445+0000","last_clean_scrub_stamp":"2021-02-10T04:53:14.202445+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":60,"num_read_kb":42,"num_write":34,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"1313","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.874369+0000","last_change":"2021-02-10T04:52:23.095651+0000","last_active":"2021-02-10T04:57:59.874369+0000","last_peered":"2021-02-10T04:57:59.874369+0000","last_clean":"2021-02-10T04:57:59.874369+0000","last_became_active":"2021-02-10T04:50:59.543531+0000","last_became_peered":"2021-02-10T04:50:59.543531+0000","last_unstale":"2021-02-10T04:57:59.874369+0000","last_undegraded":"2021-02-10T04:57:59.874369+0000","last_fullsized":"2021-02-10T04:57:59.874369+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:23.095588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:23.095588+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"1264","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.890722+0000","last_change":"2021-02-10T04:55:14.636711+0000","last_active":"2021-02-10T04:58:01.890722+0000","last_peered":"2021-02-10T04:58:01.890722+0000","last_clean":"2021-02-10T04:58:01.890722+0000","last_became_active":"2021-02-10T04:55:14.636363+0000","last_became_peered":"2021-02-10T04:55:14.636363+0000","last_unstale":"2021-02-10T04:58:01.890722+0000","last_undegraded":"2021-02-10T04:58:01.890722+0000","last_fullsized":"2021-02-10T04:58:01.890722+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:43.663762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:51:43.663762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"1183","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.412619+0000","last_change":"2021-02-10T04:55:28.101709+0000","last_active":"2021-02-10T04:58:02.412619+0000","last_peered":"2021-02-10T04:58:02.412619+0000","last_clean":"2021-02-10T04:58:02.412619+0000","last_became_active":"2021-02-10T04:55:14.635912+0000","last_became_peered":"2021-02-10T04:55:14.635912+0000","last_unstale":"2021-02-10T04:58:02.412619+0000","last_undegraded":"2021-02-10T04:58:02.412619+0000","last_fullsized":"2021-02-10T04:58:02.412619+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:57.568541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:57.568541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"1000'1071","reported_seq":"4819","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.889631+0000","last_change":"2021-02-10T04:53:35.302879+0000","last_active":"2021-02-10T04:58:01.889631+0000","last_peered":"2021-02-10T04:58:01.889631+0000","last_clean":"2021-02-10T04:58:01.889631+0000","last_became_active":"2021-02-10T04:51:39.508551+0000","last_became_peered":"2021-02-10T04:51:39.508551+0000","last_unstale":"2021-02-10T04:58:01.889631+0000","last_undegraded":"2021-02-10T04:58:01.889631+0000","last_fullsized":"2021-02-10T04:58:01.889631+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:35.302780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.302780+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"1261'39","reported_seq":"1348","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.889007+0000","last_change":"2021-02-10T04:52:39.695340+0000","last_active":"2021-02-10T04:58:01.889007+0000","last_peered":"2021-02-10T04:58:01.889007+0000","last_clean":"2021-02-10T04:58:01.889007+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:58:01.889007+0000","last_undegraded":"2021-02-10T04:58:01.889007+0000","last_fullsized":"2021-02-10T04:58:01.889007+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'31","last_scrub_stamp":"2021-02-10T04:52:39.695237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:39.695237+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":53,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"558","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.415075+0000","last_change":"2021-02-10T04:55:20.929399+0000","last_active":"2021-02-10T04:58:02.415075+0000","last_peered":"2021-02-10T04:58:02.415075+0000","last_clean":"2021-02-10T04:58:02.415075+0000","last_became_active":"2021-02-10T04:55:14.655543+0000","last_became_peered":"2021-02-10T04:55:14.655543+0000","last_unstale":"2021-02-10T04:58:02.415075+0000","last_undegraded":"2021-02-10T04:58:02.415075+0000","last_fullsized":"2021-02-10T04:58:02.415075+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:55:20.929330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:55:20.929330+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"966","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.414755+0000","last_change":"2021-02-10T04:53:35.216489+0000","last_active":"2021-02-10T04:58:02.414755+0000","last_peered":"2021-02-10T04:58:02.414755+0000","last_clean":"2021-02-10T04:58:02.414755+0000","last_became_active":"2021-02-10T04:51:49.501254+0000","last_became_peered":"2021-02-10T04:51:49.501254+0000","last_unstale":"2021-02-10T04:58:02.414755+0000","last_undegraded":"2021-02-10T04:58:02.414755+0000","last_fullsized":"2021-02-10T04:58:02.414755+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.216394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.216394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"1291","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.891648+0000","last_change":"2021-02-10T04:55:14.633276+0000","last_active":"2021-02-10T04:58:01.891648+0000","last_peered":"2021-02-10T04:58:01.891648+0000","last_clean":"2021-02-10T04:58:01.891648+0000","last_became_active":"2021-02-10T04:55:14.632805+0000","last_became_peered":"2021-02-10T04:55:14.632805+0000","last_unstale":"2021-02-10T04:58:01.891648+0000","last_undegraded":"2021-02-10T04:58:01.891648+0000","last_fullsized":"2021-02-10T04:58:01.891648+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:55.814486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.814486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"1002'1657","reported_seq":"5336","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.412955+0000","last_change":"2021-02-10T04:55:22.728878+0000","last_active":"2021-02-10T04:58:02.412955+0000","last_peered":"2021-02-10T04:58:02.412955+0000","last_clean":"2021-02-10T04:58:02.412955+0000","last_became_active":"2021-02-10T04:55:14.653270+0000","last_became_peered":"2021-02-10T04:55:14.653270+0000","last_unstale":"2021-02-10T04:58:02.412955+0000","last_undegraded":"2021-02-10T04:58:02.412955+0000","last_fullsized":"2021-02-10T04:58:02.412955+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:55:22.331010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:55:22.331010+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":47335497,"num_objects":20,"num_object_clones":18,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"1261'24","reported_seq":"1421","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.889968+0000","last_change":"2021-02-10T04:53:04.058324+0000","last_active":"2021-02-10T04:58:01.889968+0000","last_peered":"2021-02-10T04:58:01.889968+0000","last_clean":"2021-02-10T04:58:01.889968+0000","last_became_active":"2021-02-10T04:50:59.544484+0000","last_became_peered":"2021-02-10T04:50:59.544484+0000","last_unstale":"2021-02-10T04:58:01.889968+0000","last_undegraded":"2021-02-10T04:58:01.889968+0000","last_fullsized":"2021-02-10T04:58:01.889968+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'20","last_scrub_stamp":"2021-02-10T04:53:04.058248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:04.058248+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":29,"num_write":24,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"1317","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.892655+0000","last_change":"2021-02-10T04:53:00.343307+0000","last_active":"2021-02-10T04:58:01.892655+0000","last_peered":"2021-02-10T04:58:01.892655+0000","last_clean":"2021-02-10T04:58:01.892655+0000","last_became_active":"2021-02-10T04:51:36.844573+0000","last_became_peered":"2021-02-10T04:51:36.844573+0000","last_unstale":"2021-02-10T04:58:01.892655+0000","last_undegraded":"2021-02-10T04:58:01.892655+0000","last_fullsized":"2021-02-10T04:58:01.892655+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:00.343228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:00.343228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"1284","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.874150+0000","last_change":"2021-02-10T04:50:56.099710+0000","last_active":"2021-02-10T04:57:59.874150+0000","last_peered":"2021-02-10T04:57:59.874150+0000","last_clean":"2021-02-10T04:57:59.874150+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:57:59.874150+0000","last_undegraded":"2021-02-10T04:57:59.874150+0000","last_fullsized":"2021-02-10T04:57:59.874150+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:56.099651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:50:56.099651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"632","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.278583+0000","last_change":"2021-02-10T04:53:46.718715+0000","last_active":"2021-02-10T04:58:02.278583+0000","last_peered":"2021-02-10T04:58:02.278583+0000","last_clean":"2021-02-10T04:58:02.278583+0000","last_became_active":"2021-02-10T04:53:46.718266+0000","last_became_peered":"2021-02-10T04:53:46.718266+0000","last_unstale":"2021-02-10T04:58:02.278583+0000","last_undegraded":"2021-02-10T04:58:02.278583+0000","last_fullsized":"2021-02-10T04:58:02.278583+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.876681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.876681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"998'1009","reported_seq":"5160","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.874205+0000","last_change":"2021-02-10T04:55:14.635764+0000","last_active":"2021-02-10T04:57:59.874205+0000","last_peered":"2021-02-10T04:57:59.874205+0000","last_clean":"2021-02-10T04:57:59.874205+0000","last_became_active":"2021-02-10T04:55:14.634422+0000","last_became_peered":"2021-02-10T04:55:14.634422+0000","last_unstale":"2021-02-10T04:57:59.874205+0000","last_undegraded":"2021-02-10T04:57:59.874205+0000","last_fullsized":"2021-02-10T04:57:59.874205+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":5,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:13.032506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:13.032506+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49420827,"num_objects":21,"num_object_clones":19,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":36763798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"1261'45","reported_seq":"1355","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.892089+0000","last_change":"2021-02-10T04:51:44.694779+0000","last_active":"2021-02-10T04:58:01.892089+0000","last_peered":"2021-02-10T04:58:01.892089+0000","last_clean":"2021-02-10T04:58:01.892089+0000","last_became_active":"2021-02-10T04:50:59.549412+0000","last_became_peered":"2021-02-10T04:50:59.549412+0000","last_unstale":"2021-02-10T04:58:01.892089+0000","last_undegraded":"2021-02-10T04:58:01.892089+0000","last_fullsized":"2021-02-10T04:58:01.892089+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:44.694711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.694711+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":76,"num_read_kb":62,"num_write":45,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"1451","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.875607+0000","last_change":"2021-02-10T04:52:33.088525+0000","last_active":"2021-02-10T04:57:59.875607+0000","last_peered":"2021-02-10T04:57:59.875607+0000","last_clean":"2021-02-10T04:57:59.875607+0000","last_became_active":"2021-02-10T04:51:36.870573+0000","last_became_peered":"2021-02-10T04:51:36.870573+0000","last_unstale":"2021-02-10T04:57:59.875607+0000","last_undegraded":"2021-02-10T04:57:59.875607+0000","last_fullsized":"2021-02-10T04:57:59.875607+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:52:33.088428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:33.088428+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"599","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.889965+0000","last_change":"2021-02-10T04:51:51.836723+0000","last_active":"2021-02-10T04:58:01.889965+0000","last_peered":"2021-02-10T04:58:01.889965+0000","last_clean":"2021-02-10T04:58:01.889965+0000","last_became_active":"2021-02-10T04:50:59.564977+0000","last_became_peered":"2021-02-10T04:50:59.564977+0000","last_unstale":"2021-02-10T04:58:01.889965+0000","last_undegraded":"2021-02-10T04:58:01.889965+0000","last_fullsized":"2021-02-10T04:58:01.889965+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.836638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.836638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"1255","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.890640+0000","last_change":"2021-02-10T04:55:14.633046+0000","last_active":"2021-02-10T04:58:01.890640+0000","last_peered":"2021-02-10T04:58:01.890640+0000","last_clean":"2021-02-10T04:58:01.890640+0000","last_became_active":"2021-02-10T04:55:14.632658+0000","last_became_peered":"2021-02-10T04:55:14.632658+0000","last_unstale":"2021-02-10T04:58:01.890640+0000","last_undegraded":"2021-02-10T04:58:01.890640+0000","last_fullsized":"2021-02-10T04:58:01.890640+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:58.514318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_clean_scrub_stamp":"2021-02-10T04:46:58.514318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1d","version":"1001'1165","reported_seq":"4314","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.415079+0000","last_change":"2021-02-10T04:55:14.656236+0000","last_active":"2021-02-10T04:58:02.415079+0000","last_peered":"2021-02-10T04:58:02.415079+0000","last_clean":"2021-02-10T04:58:02.415079+0000","last_became_active":"2021-02-10T04:55:14.655859+0000","last_became_peered":"2021-02-10T04:55:14.655859+0000","last_unstale":"2021-02-10T04:58:02.415079+0000","last_undegraded":"2021-02-10T04:58:02.415079+0000","last_fullsized":"2021-02-10T04:58:02.415079+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":5,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29601845,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":287,"num_read_kb":299530,"num_write":200,"num_write_kb":89131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"1358","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.890580+0000","last_change":"2021-02-10T04:53:01.331566+0000","last_active":"2021-02-10T04:58:01.890580+0000","last_peered":"2021-02-10T04:58:01.890580+0000","last_clean":"2021-02-10T04:58:01.890580+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:58:01.890580+0000","last_undegraded":"2021-02-10T04:58:01.890580+0000","last_fullsized":"2021-02-10T04:58:01.890580+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:01.331466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.331466+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"1098'3","reported_seq":"1679","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.889595+0000","last_change":"2021-02-10T04:53:34.865668+0000","last_active":"2021-02-10T04:58:01.889595+0000","last_peered":"2021-02-10T04:58:01.889595+0000","last_clean":"2021-02-10T04:58:01.889595+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:58:01.889595+0000","last_undegraded":"2021-02-10T04:58:01.889595+0000","last_fullsized":"2021-02-10T04:58:01.889595+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"1098'3","last_scrub_stamp":"2021-02-10T04:53:34.865579+0000","last_deep_scrub":"1098'3","last_deep_scrub_stamp":"2021-02-10T04:50:15.255137+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.865579+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"1162","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.278900+0000","last_change":"2021-02-10T04:53:46.727603+0000","last_active":"2021-02-10T04:58:02.278900+0000","last_peered":"2021-02-10T04:58:02.278900+0000","last_clean":"2021-02-10T04:58:02.278900+0000","last_became_active":"2021-02-10T04:53:46.727053+0000","last_became_peered":"2021-02-10T04:53:46.727053+0000","last_unstale":"2021-02-10T04:58:02.278900+0000","last_undegraded":"2021-02-10T04:58:02.278900+0000","last_fullsized":"2021-02-10T04:58:02.278900+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:47.493057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.493057+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1c","version":"1002'1090","reported_seq":"5338","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.891934+0000","last_change":"2021-02-10T04:53:41.353991+0000","last_active":"2021-02-10T04:58:01.891934+0000","last_peered":"2021-02-10T04:58:01.891934+0000","last_clean":"2021-02-10T04:58:01.891934+0000","last_became_active":"2021-02-10T04:52:31.088265+0000","last_became_peered":"2021-02-10T04:52:31.088265+0000","last_unstale":"2021-02-10T04:58:01.891934+0000","last_undegraded":"2021-02-10T04:58:01.891934+0000","last_fullsized":"2021-02-10T04:58:01.891934+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:53:41.353840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:41.353840+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"1259","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.891872+0000","last_change":"2021-02-10T04:55:14.632817+0000","last_active":"2021-02-10T04:58:01.891872+0000","last_peered":"2021-02-10T04:58:01.891872+0000","last_clean":"2021-02-10T04:58:01.891872+0000","last_became_active":"2021-02-10T04:55:14.632214+0000","last_became_peered":"2021-02-10T04:55:14.632214+0000","last_unstale":"2021-02-10T04:58:01.891872+0000","last_undegraded":"2021-02-10T04:58:01.891872+0000","last_fullsized":"2021-02-10T04:58:01.891872+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:42.537443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:42.537443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"1208","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.413446+0000","last_change":"2021-02-10T04:55:22.725805+0000","last_active":"2021-02-10T04:58:02.413446+0000","last_peered":"2021-02-10T04:58:02.413446+0000","last_clean":"2021-02-10T04:58:02.413446+0000","last_became_active":"2021-02-10T04:55:14.630190+0000","last_became_peered":"2021-02-10T04:55:14.630190+0000","last_unstale":"2021-02-10T04:58:02.413446+0000","last_undegraded":"2021-02-10T04:58:02.413446+0000","last_fullsized":"2021-02-10T04:58:02.413446+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.938731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.938731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"523","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.414986+0000","last_change":"2021-02-10T04:53:15.185748+0000","last_active":"2021-02-10T04:58:02.414986+0000","last_peered":"2021-02-10T04:58:02.414986+0000","last_clean":"2021-02-10T04:58:02.414986+0000","last_became_active":"2021-02-10T04:51:49.499348+0000","last_became_peered":"2021-02-10T04:51:49.499348+0000","last_unstale":"2021-02-10T04:58:02.414986+0000","last_undegraded":"2021-02-10T04:58:02.414986+0000","last_fullsized":"2021-02-10T04:58:02.414986+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:15.185686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:15.185686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1f","version":"1004'1093","reported_seq":"6574","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.874315+0000","last_change":"2021-02-10T04:52:31.089952+0000","last_active":"2021-02-10T04:57:59.874315+0000","last_peered":"2021-02-10T04:57:59.874315+0000","last_clean":"2021-02-10T04:57:59.874315+0000","last_became_active":"2021-02-10T04:52:31.088634+0000","last_became_peered":"2021-02-10T04:52:31.088634+0000","last_unstale":"2021-02-10T04:57:59.874315+0000","last_undegraded":"2021-02-10T04:57:59.874315+0000","last_fullsized":"2021-02-10T04:57:59.874315+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39434697,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":279,"num_read_kb":311058,"num_write":186,"num_write_kb":66790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":53995913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"1314","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.874636+0000","last_change":"2021-02-10T04:55:39.651056+0000","last_active":"2021-02-10T04:57:59.874636+0000","last_peered":"2021-02-10T04:57:59.874636+0000","last_clean":"2021-02-10T04:57:59.874636+0000","last_became_active":"2021-02-10T04:55:39.596648+0000","last_became_peered":"2021-02-10T04:55:39.596648+0000","last_unstale":"2021-02-10T04:57:59.874636+0000","last_undegraded":"2021-02-10T04:57:59.874636+0000","last_fullsized":"2021-02-10T04:57:59.874636+0000","mapping_epoch":1248,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:52:36.083784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:36.083784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"593","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.574259+0000","last_change":"2021-02-10T04:58:02.574259+0000","last_active":"2021-02-10T04:58:02.574259+0000","last_peered":"2021-02-10T04:58:02.574259+0000","last_clean":"2021-02-10T04:58:02.574259+0000","last_became_active":"2021-02-10T04:55:14.655649+0000","last_became_peered":"2021-02-10T04:55:14.655649+0000","last_unstale":"2021-02-10T04:58:02.574259+0000","last_undegraded":"2021-02-10T04:58:02.574259+0000","last_fullsized":"2021-02-10T04:58:02.574259+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:08.758129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.758129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"1281","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.889068+0000","last_change":"2021-02-10T04:51:44.799512+0000","last_active":"2021-02-10T04:58:01.889068+0000","last_peered":"2021-02-10T04:58:01.889068+0000","last_clean":"2021-02-10T04:58:01.889068+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:58:01.889068+0000","last_undegraded":"2021-02-10T04:58:01.889068+0000","last_fullsized":"2021-02-10T04:58:01.889068+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:44.799429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.799429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1e","version":"1063'654","reported_seq":"4556","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.414105+0000","last_change":"2021-02-10T04:53:40.135450+0000","last_active":"2021-02-10T04:58:02.414105+0000","last_peered":"2021-02-10T04:58:02.414105+0000","last_clean":"2021-02-10T04:58:02.414105+0000","last_became_active":"2021-02-10T04:52:31.087662+0000","last_became_peered":"2021-02-10T04:52:31.087662+0000","last_unstale":"2021-02-10T04:58:02.414105+0000","last_undegraded":"2021-02-10T04:58:02.414105+0000","last_fullsized":"2021-02-10T04:58:02.414105+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:53:40.135367+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:53:40.135367+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"1265","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.891178+0000","last_change":"2021-02-10T04:51:08.006463+0000","last_active":"2021-02-10T04:58:01.891178+0000","last_peered":"2021-02-10T04:58:01.891178+0000","last_clean":"2021-02-10T04:58:01.891178+0000","last_became_active":"2021-02-10T04:50:59.563657+0000","last_became_peered":"2021-02-10T04:50:59.563657+0000","last_unstale":"2021-02-10T04:58:01.891178+0000","last_undegraded":"2021-02-10T04:58:01.891178+0000","last_fullsized":"2021-02-10T04:58:01.891178+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_clean_scrub_stamp":"2021-02-10T04:51:02.455702+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"1316","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.888890+0000","last_change":"2021-02-10T04:52:33.741801+0000","last_active":"2021-02-10T04:58:01.888890+0000","last_peered":"2021-02-10T04:58:01.888890+0000","last_clean":"2021-02-10T04:58:01.888890+0000","last_became_active":"2021-02-10T04:51:36.860129+0000","last_became_peered":"2021-02-10T04:51:36.860129+0000","last_unstale":"2021-02-10T04:58:01.888890+0000","last_undegraded":"2021-02-10T04:58:01.888890+0000","last_fullsized":"2021-02-10T04:58:01.888890+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:33.741711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:46:58.666264+0000","last_clean_scrub_stamp":"2021-02-10T04:52:33.741711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"1277","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.892907+0000","last_change":"2021-02-10T04:52:25.030419+0000","last_active":"2021-02-10T04:58:01.892907+0000","last_peered":"2021-02-10T04:58:01.892907+0000","last_clean":"2021-02-10T04:58:01.892907+0000","last_became_active":"2021-02-10T04:50:59.539582+0000","last_became_peered":"2021-02-10T04:50:59.539582+0000","last_unstale":"2021-02-10T04:58:01.892907+0000","last_undegraded":"2021-02-10T04:58:01.892907+0000","last_fullsized":"2021-02-10T04:58:01.892907+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:25.030281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.030281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"1003'739","reported_seq":"4061","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.873610+0000","last_change":"2021-02-10T04:53:01.929261+0000","last_active":"2021-02-10T04:57:59.873610+0000","last_peered":"2021-02-10T04:57:59.873610+0000","last_clean":"2021-02-10T04:57:59.873610+0000","last_became_active":"2021-02-10T04:51:40.511505+0000","last_became_peered":"2021-02-10T04:51:40.511505+0000","last_unstale":"2021-02-10T04:57:59.873610+0000","last_undegraded":"2021-02-10T04:57:59.873610+0000","last_fullsized":"2021-02-10T04:57:59.873610+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:53:01.929169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.929169+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22230846,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":18429397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"1153","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.278678+0000","last_change":"2021-02-10T04:53:46.727590+0000","last_active":"2021-02-10T04:58:02.278678+0000","last_peered":"2021-02-10T04:58:02.278678+0000","last_clean":"2021-02-10T04:58:02.278678+0000","last_became_active":"2021-02-10T04:53:46.726584+0000","last_became_peered":"2021-02-10T04:53:46.726584+0000","last_unstale":"2021-02-10T04:58:02.278678+0000","last_undegraded":"2021-02-10T04:58:02.278678+0000","last_fullsized":"2021-02-10T04:58:02.278678+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:26.408326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.408326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"1314","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.891369+0000","last_change":"2021-02-10T04:53:07.324650+0000","last_active":"2021-02-10T04:58:01.891369+0000","last_peered":"2021-02-10T04:58:01.891369+0000","last_clean":"2021-02-10T04:58:01.891369+0000","last_became_active":"2021-02-10T04:50:59.575340+0000","last_became_peered":"2021-02-10T04:50:59.575340+0000","last_unstale":"2021-02-10T04:58:01.891369+0000","last_undegraded":"2021-02-10T04:58:01.891369+0000","last_fullsized":"2021-02-10T04:58:01.891369+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.324564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.324564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"1207","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.412207+0000","last_change":"2021-02-10T04:55:28.098627+0000","last_active":"2021-02-10T04:58:02.412207+0000","last_peered":"2021-02-10T04:58:02.412207+0000","last_clean":"2021-02-10T04:58:02.412207+0000","last_became_active":"2021-02-10T04:55:14.640497+0000","last_became_peered":"2021-02-10T04:55:14.640497+0000","last_unstale":"2021-02-10T04:58:02.412207+0000","last_undegraded":"2021-02-10T04:58:02.412207+0000","last_fullsized":"2021-02-10T04:58:02.412207+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:00.501127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:00.501127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"259'74","reported_seq":"252","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.412399+0000","last_change":"2021-02-10T04:55:28.095458+0000","last_active":"2021-02-10T04:58:02.412399+0000","last_peered":"2021-02-10T04:58:02.412399+0000","last_clean":"2021-02-10T04:58:02.412399+0000","last_became_active":"2021-02-10T04:55:14.624071+0000","last_became_peered":"2021-02-10T04:55:14.624071+0000","last_unstale":"2021-02-10T04:58:02.412399+0000","last_undegraded":"2021-02-10T04:58:02.412399+0000","last_fullsized":"2021-02-10T04:58:02.412399+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":5,"last_scrub":"259'74","last_scrub_stamp":"2021-02-10T04:52:26.597668+0000","last_deep_scrub":"259'74","last_deep_scrub_stamp":"2021-02-10T04:46:49.745104+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.597668+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1218524,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":28228,"num_write":19,"num_write_kb":7022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"75","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.893673+0000","last_change":"2021-02-10T04:55:39.619500+0000","last_active":"2021-02-10T04:58:01.893673+0000","last_peered":"2021-02-10T04:58:01.893673+0000","last_clean":"2021-02-10T04:58:01.893673+0000","last_became_active":"2021-02-10T04:55:39.619062+0000","last_became_peered":"2021-02-10T04:55:39.619062+0000","last_unstale":"2021-02-10T04:58:01.893673+0000","last_undegraded":"2021-02-10T04:58:01.893673+0000","last_fullsized":"2021-02-10T04:58:01.893673+0000","mapping_epoch":1248,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:59.167911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:50:59.167911+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"223","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.278347+0000","last_change":"2021-02-10T04:53:46.718512+0000","last_active":"2021-02-10T04:58:02.278347+0000","last_peered":"2021-02-10T04:58:02.278347+0000","last_clean":"2021-02-10T04:58:02.278347+0000","last_became_active":"2021-02-10T04:53:46.716811+0000","last_became_peered":"2021-02-10T04:53:46.716811+0000","last_unstale":"2021-02-10T04:58:02.278347+0000","last_undegraded":"2021-02-10T04:58:02.278347+0000","last_fullsized":"2021-02-10T04:58:02.278347+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:45.813676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.813676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"1303","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.891047+0000","last_change":"2021-02-10T04:53:38.363899+0000","last_active":"2021-02-10T04:58:01.891047+0000","last_peered":"2021-02-10T04:58:01.891047+0000","last_clean":"2021-02-10T04:58:01.891047+0000","last_became_active":"2021-02-10T04:50:59.539256+0000","last_became_peered":"2021-02-10T04:50:59.539256+0000","last_unstale":"2021-02-10T04:58:01.891047+0000","last_undegraded":"2021-02-10T04:58:01.891047+0000","last_fullsized":"2021-02-10T04:58:01.891047+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:38.363825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:09.433793+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.363825+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1b","version":"1003'1689","reported_seq":"4659","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.412740+0000","last_change":"2021-02-10T04:55:15.591337+0000","last_active":"2021-02-10T04:58:02.412740+0000","last_peered":"2021-02-10T04:58:02.412740+0000","last_clean":"2021-02-10T04:58:02.412740+0000","last_became_active":"2021-02-10T04:55:15.590301+0000","last_became_peered":"2021-02-10T04:55:15.590301+0000","last_unstale":"2021-02-10T04:58:02.412740+0000","last_undegraded":"2021-02-10T04:58:02.412740+0000","last_fullsized":"2021-02-10T04:58:02.412740+0000","mapping_epoch":1228,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:53:10.075879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.075879+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":63260738,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":387,"num_read_kb":409791,"num_write":281,"num_write_kb":110559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":22449943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"1298","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.893071+0000","last_change":"2021-02-10T04:55:14.637968+0000","last_active":"2021-02-10T04:58:01.893071+0000","last_peered":"2021-02-10T04:58:01.893071+0000","last_clean":"2021-02-10T04:58:01.893071+0000","last_became_active":"2021-02-10T04:55:14.636561+0000","last_became_peered":"2021-02-10T04:55:14.636561+0000","last_unstale":"2021-02-10T04:58:01.893071+0000","last_undegraded":"2021-02-10T04:58:01.893071+0000","last_fullsized":"2021-02-10T04:58:01.893071+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:33.543559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:46:33.543559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"973","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.890820+0000","last_change":"2021-02-10T04:53:04.354508+0000","last_active":"2021-02-10T04:58:01.890820+0000","last_peered":"2021-02-10T04:58:01.890820+0000","last_clean":"2021-02-10T04:58:01.890820+0000","last_became_active":"2021-02-10T04:33:05.169159+0000","last_became_peered":"2021-02-10T04:33:05.169159+0000","last_unstale":"2021-02-10T04:58:01.890820+0000","last_undegraded":"2021-02-10T04:58:01.890820+0000","last_fullsized":"2021-02-10T04:58:01.890820+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:04.354409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:04.354409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"1367","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.889931+0000","last_change":"2021-02-10T04:52:36.777554+0000","last_active":"2021-02-10T04:58:01.889931+0000","last_peered":"2021-02-10T04:58:01.889931+0000","last_clean":"2021-02-10T04:58:01.889931+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:58:01.889931+0000","last_undegraded":"2021-02-10T04:58:01.889931+0000","last_fullsized":"2021-02-10T04:58:01.889931+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:36.777485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:52:36.777485+0000","last_clean_scrub_stamp":"2021-02-10T04:52:36.777485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1a","version":"1001'1528","reported_seq":"5924","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.888704+0000","last_change":"2021-02-10T04:52:31.086406+0000","last_active":"2021-02-10T04:58:01.888704+0000","last_peered":"2021-02-10T04:58:01.888704+0000","last_clean":"2021-02-10T04:58:01.888704+0000","last_became_active":"2021-02-10T04:52:31.085959+0000","last_became_peered":"2021-02-10T04:52:31.085959+0000","last_unstale":"2021-02-10T04:58:01.888704+0000","last_undegraded":"2021-02-10T04:58:01.888704+0000","last_fullsized":"2021-02-10T04:58:01.888704+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46944136,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":358,"num_read_kb":351935,"num_write":286,"num_write_kb":111355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":21311893,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"1197","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.413229+0000","last_change":"2021-02-10T04:55:22.726939+0000","last_active":"2021-02-10T04:58:02.413229+0000","last_peered":"2021-02-10T04:58:02.413229+0000","last_clean":"2021-02-10T04:58:02.413229+0000","last_became_active":"2021-02-10T04:55:14.627044+0000","last_became_peered":"2021-02-10T04:55:14.627044+0000","last_unstale":"2021-02-10T04:58:02.413229+0000","last_undegraded":"2021-02-10T04:58:02.413229+0000","last_fullsized":"2021-02-10T04:58:02.413229+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:17.919745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:17.919745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"599","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.889519+0000","last_change":"2021-02-10T04:53:07.092032+0000","last_active":"2021-02-10T04:58:01.889519+0000","last_peered":"2021-02-10T04:58:01.889519+0000","last_clean":"2021-02-10T04:58:01.889519+0000","last_became_active":"2021-02-10T04:50:59.572077+0000","last_became_peered":"2021-02-10T04:50:59.572077+0000","last_unstale":"2021-02-10T04:58:01.889519+0000","last_undegraded":"2021-02-10T04:58:01.889519+0000","last_fullsized":"2021-02-10T04:58:01.889519+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.091949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:19.755718+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.091949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"1276","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.890899+0000","last_change":"2021-02-10T04:51:59.838856+0000","last_active":"2021-02-10T04:58:01.890899+0000","last_peered":"2021-02-10T04:58:01.890899+0000","last_clean":"2021-02-10T04:58:01.890899+0000","last_became_active":"2021-02-10T04:50:59.563156+0000","last_became_peered":"2021-02-10T04:50:59.563156+0000","last_unstale":"2021-02-10T04:58:01.890899+0000","last_undegraded":"2021-02-10T04:58:01.890899+0000","last_fullsized":"2021-02-10T04:58:01.890899+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.838790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.838790+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":"1009","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.891860+0000","last_change":"2021-02-10T04:52:30.405170+0000","last_active":"2021-02-10T04:58:01.891860+0000","last_peered":"2021-02-10T04:58:01.891860+0000","last_clean":"2021-02-10T04:58:01.891860+0000","last_became_active":"2021-02-10T04:51:36.852058+0000","last_became_peered":"2021-02-10T04:51:36.852058+0000","last_unstale":"2021-02-10T04:58:01.891860+0000","last_undegraded":"2021-02-10T04:58:01.891860+0000","last_fullsized":"2021-02-10T04:58:01.891860+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.405106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:20.635536+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.405106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.20","version":"0'0","reported_seq":"247","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.278085+0000","last_change":"2021-02-10T04:53:46.739888+0000","last_active":"2021-02-10T04:58:02.278085+0000","last_peered":"2021-02-10T04:58:02.278085+0000","last_clean":"2021-02-10T04:58:02.278085+0000","last_became_active":"2021-02-10T04:53:46.739532+0000","last_became_peered":"2021-02-10T04:53:46.739532+0000","last_unstale":"2021-02-10T04:58:02.278085+0000","last_undegraded":"2021-02-10T04:58:02.278085+0000","last_fullsized":"2021-02-10T04:58:02.278085+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:56.449085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:41.375276+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.449085+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":"1240","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.874363+0000","last_change":"2021-02-10T04:53:35.858634+0000","last_active":"2021-02-10T04:57:59.874363+0000","last_peered":"2021-02-10T04:57:59.874363+0000","last_clean":"2021-02-10T04:57:59.874363+0000","last_became_active":"2021-02-10T04:51:36.869785+0000","last_became_peered":"2021-02-10T04:51:36.869785+0000","last_unstale":"2021-02-10T04:57:59.874363+0000","last_undegraded":"2021-02-10T04:57:59.874363+0000","last_fullsized":"2021-02-10T04:57:59.874363+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.858557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.858557+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":"225","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.892312+0000","last_change":"2021-02-10T04:55:23.586523+0000","last_active":"2021-02-10T04:58:01.892312+0000","last_peered":"2021-02-10T04:58:01.892312+0000","last_clean":"2021-02-10T04:58:01.892312+0000","last_became_active":"2021-02-10T04:55:14.637029+0000","last_became_peered":"2021-02-10T04:55:14.637029+0000","last_unstale":"2021-02-10T04:58:01.892312+0000","last_undegraded":"2021-02-10T04:58:01.892312+0000","last_fullsized":"2021-02-10T04:58:01.892312+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:55:23.586401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:55:23.586401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":"210","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.415210+0000","last_change":"2021-02-10T04:55:14.635842+0000","last_active":"2021-02-10T04:58:02.415210+0000","last_peered":"2021-02-10T04:58:02.415210+0000","last_clean":"2021-02-10T04:58:02.415210+0000","last_became_active":"2021-02-10T04:55:14.635499+0000","last_became_peered":"2021-02-10T04:55:14.635499+0000","last_unstale":"2021-02-10T04:58:02.415210+0000","last_undegraded":"2021-02-10T04:58:02.415210+0000","last_fullsized":"2021-02-10T04:58:02.415210+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.706372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.706372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":"1322","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.874343+0000","last_change":"2021-02-10T04:51:57.287452+0000","last_active":"2021-02-10T04:57:59.874343+0000","last_peered":"2021-02-10T04:57:59.874343+0000","last_clean":"2021-02-10T04:57:59.874343+0000","last_became_active":"2021-02-10T04:50:59.566827+0000","last_became_peered":"2021-02-10T04:50:59.566827+0000","last_unstale":"2021-02-10T04:57:59.874343+0000","last_undegraded":"2021-02-10T04:57:59.874343+0000","last_fullsized":"2021-02-10T04:57:59.874343+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.287363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.287363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":"296","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.874748+0000","last_change":"2021-02-10T04:55:14.659882+0000","last_active":"2021-02-10T04:57:59.874748+0000","last_peered":"2021-02-10T04:57:59.874748+0000","last_clean":"2021-02-10T04:57:59.874748+0000","last_became_active":"2021-02-10T04:55:14.659514+0000","last_became_peered":"2021-02-10T04:55:14.659514+0000","last_unstale":"2021-02-10T04:57:59.874748+0000","last_undegraded":"2021-02-10T04:57:59.874748+0000","last_fullsized":"2021-02-10T04:57:59.874748+0000","mapping_epoch":1227,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:10.308282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:10.308282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":"1221","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.891251+0000","last_change":"2021-02-10T04:56:45.790734+0000","last_active":"2021-02-10T04:58:01.891251+0000","last_peered":"2021-02-10T04:58:01.891251+0000","last_clean":"2021-02-10T04:58:01.891251+0000","last_became_active":"2021-02-10T04:56:45.790365+0000","last_became_peered":"2021-02-10T04:56:45.790365+0000","last_unstale":"2021-02-10T04:58:01.891251+0000","last_undegraded":"2021-02-10T04:58:01.891251+0000","last_fullsized":"2021-02-10T04:58:01.891251+0000","mapping_epoch":1280,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1281,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:09.121298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.121298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.21","version":"293'77","reported_seq":"884","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.891949+0000","last_change":"2021-02-10T04:52:28.062923+0000","last_active":"2021-02-10T04:58:01.891949+0000","last_peered":"2021-02-10T04:58:01.891949+0000","last_clean":"2021-02-10T04:58:01.891949+0000","last_became_active":"2021-02-10T04:52:28.062463+0000","last_became_peered":"2021-02-10T04:52:28.062463+0000","last_unstale":"2021-02-10T04:58:01.891949+0000","last_undegraded":"2021-02-10T04:58:01.891949+0000","last_fullsized":"2021-02-10T04:58:01.891949+0000","mapping_epoch":1143,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1144,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1886586,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":15163,"num_write":9,"num_write_kb":3316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":"210","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.892178+0000","last_change":"2021-02-10T04:52:32.124955+0000","last_active":"2021-02-10T04:58:01.892178+0000","last_peered":"2021-02-10T04:58:01.892178+0000","last_clean":"2021-02-10T04:58:01.892178+0000","last_became_active":"2021-02-10T04:51:36.867061+0000","last_became_peered":"2021-02-10T04:51:36.867061+0000","last_unstale":"2021-02-10T04:58:01.892178+0000","last_undegraded":"2021-02-10T04:58:01.892178+0000","last_fullsized":"2021-02-10T04:58:01.892178+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:32.124901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:08.271977+0000","last_clean_scrub_stamp":"2021-02-10T04:52:32.124901+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":"1251","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.874544+0000","last_change":"2021-02-10T04:51:07.182775+0000","last_active":"2021-02-10T04:57:59.874544+0000","last_peered":"2021-02-10T04:57:59.874544+0000","last_clean":"2021-02-10T04:57:59.874544+0000","last_became_active":"2021-02-10T04:28:23.126368+0000","last_became_peered":"2021-02-10T04:28:23.126368+0000","last_unstale":"2021-02-10T04:57:59.874544+0000","last_undegraded":"2021-02-10T04:57:59.874544+0000","last_fullsized":"2021-02-10T04:57:59.874544+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.182678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.182678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.20","version":"1000'1071","reported_seq":"4658","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.891926+0000","last_change":"2021-02-10T04:53:03.915465+0000","last_active":"2021-02-10T04:58:01.891926+0000","last_peered":"2021-02-10T04:58:01.891926+0000","last_clean":"2021-02-10T04:58:01.891926+0000","last_became_active":"2021-02-10T04:52:31.088690+0000","last_became_peered":"2021-02-10T04:52:31.088690+0000","last_unstale":"2021-02-10T04:58:01.891926+0000","last_undegraded":"2021-02-10T04:58:01.891926+0000","last_fullsized":"2021-02-10T04:58:01.891926+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:03.915403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.915403+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16398893,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":"836","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.278155+0000","last_change":"2021-02-10T04:53:46.721599+0000","last_active":"2021-02-10T04:58:02.278155+0000","last_peered":"2021-02-10T04:58:02.278155+0000","last_clean":"2021-02-10T04:58:02.278155+0000","last_became_active":"2021-02-10T04:53:46.721340+0000","last_became_peered":"2021-02-10T04:53:46.721340+0000","last_unstale":"2021-02-10T04:58:02.278155+0000","last_undegraded":"2021-02-10T04:58:02.278155+0000","last_fullsized":"2021-02-10T04:58:02.278155+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:28.092288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:07.767250+0000","last_clean_scrub_stamp":"2021-02-10T04:52:28.092288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.25","version":"53'2","reported_seq":"1268","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.888661+0000","last_change":"2021-02-10T04:53:36.103026+0000","last_active":"2021-02-10T04:58:01.888661+0000","last_peered":"2021-02-10T04:58:01.888661+0000","last_clean":"2021-02-10T04:58:01.888661+0000","last_became_active":"2021-02-10T04:28:23.467770+0000","last_became_peered":"2021-02-10T04:28:23.467770+0000","last_unstale":"2021-02-10T04:58:01.888661+0000","last_undegraded":"2021-02-10T04:58:01.888661+0000","last_fullsized":"2021-02-10T04:58:01.888661+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:53:36.102928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:36.102928+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.23","version":"1002'1657","reported_seq":"5183","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.874600+0000","last_change":"2021-02-10T04:52:31.093874+0000","last_active":"2021-02-10T04:57:59.874600+0000","last_peered":"2021-02-10T04:57:59.874600+0000","last_clean":"2021-02-10T04:57:59.874600+0000","last_became_active":"2021-02-10T04:52:31.092996+0000","last_became_peered":"2021-02-10T04:52:31.092996+0000","last_unstale":"2021-02-10T04:57:59.874600+0000","last_undegraded":"2021-02-10T04:57:59.874600+0000","last_fullsized":"2021-02-10T04:57:59.874600+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21396533,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":212,"num_read_kb":218837,"num_write":159,"num_write_kb":59399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":20479176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":"1358","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.873890+0000","last_change":"2021-02-10T04:53:12.951612+0000","last_active":"2021-02-10T04:57:59.873890+0000","last_peered":"2021-02-10T04:57:59.873890+0000","last_clean":"2021-02-10T04:57:59.873890+0000","last_became_active":"2021-02-10T04:50:59.566548+0000","last_became_peered":"2021-02-10T04:50:59.566548+0000","last_unstale":"2021-02-10T04:57:59.873890+0000","last_undegraded":"2021-02-10T04:57:59.873890+0000","last_fullsized":"2021-02-10T04:57:59.873890+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.951528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.951528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.26","version":"53'1","reported_seq":"150","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.279215+0000","last_change":"2021-02-10T04:53:46.722636+0000","last_active":"2021-02-10T04:58:02.279215+0000","last_peered":"2021-02-10T04:58:02.279215+0000","last_clean":"2021-02-10T04:58:02.279215+0000","last_became_active":"2021-02-10T04:53:46.722227+0000","last_became_peered":"2021-02-10T04:53:46.722227+0000","last_unstale":"2021-02-10T04:58:02.279215+0000","last_undegraded":"2021-02-10T04:58:02.279215+0000","last_fullsized":"2021-02-10T04:58:02.279215+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:53:03.933910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.933910+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":423,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.22","version":"998'1009","reported_seq":"4675","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.889813+0000","last_change":"2021-02-10T04:52:31.086563+0000","last_active":"2021-02-10T04:58:01.889813+0000","last_peered":"2021-02-10T04:58:01.889813+0000","last_clean":"2021-02-10T04:58:01.889813+0000","last_became_active":"2021-02-10T04:52:31.085288+0000","last_became_peered":"2021-02-10T04:52:31.085288+0000","last_unstale":"2021-02-10T04:58:01.889813+0000","last_undegraded":"2021-02-10T04:58:01.889813+0000","last_fullsized":"2021-02-10T04:58:01.889813+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12355207,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":120,"num_read_kb":124702,"num_write":77,"num_write_kb":25052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4145029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":"1240","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.874326+0000","last_change":"2021-02-10T04:53:10.958668+0000","last_active":"2021-02-10T04:57:59.874326+0000","last_peered":"2021-02-10T04:57:59.874326+0000","last_clean":"2021-02-10T04:57:59.874326+0000","last_became_active":"2021-02-10T04:45:49.552026+0000","last_became_peered":"2021-02-10T04:45:49.552026+0000","last_unstale":"2021-02-10T04:57:59.874326+0000","last_undegraded":"2021-02-10T04:57:59.874326+0000","last_fullsized":"2021-02-10T04:57:59.874326+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:10.958579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.958579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":"255","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:02.279813+0000","last_change":"2021-02-10T04:53:46.725941+0000","last_active":"2021-02-10T04:58:02.279813+0000","last_peered":"2021-02-10T04:58:02.279813+0000","last_clean":"2021-02-10T04:58:02.279813+0000","last_became_active":"2021-02-10T04:53:46.725587+0000","last_became_peered":"2021-02-10T04:53:46.725587+0000","last_unstale":"2021-02-10T04:58:02.279813+0000","last_undegraded":"2021-02-10T04:58:02.279813+0000","last_fullsized":"2021-02-10T04:58:02.279813+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:41.758684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.758684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":"1250","reported_epoch":"1313","state":"active+clean","last_fresh":"2021-02-10T04:58:01.889738+0000","last_change":"2021-02-10T04:55:39.599193+0000","last_active":"2021-02-10T04:58:01.889738+0000","last_peered":"2021-02-10T04:58:01.889738+0000","last_clean":"2021-02-10T04:58:01.889738+0000","last_became_active":"2021-02-10T04:55:39.598816+0000","last_became_peered":"2021-02-10T04:55:39.598816+0000","last_unstale":"2021-02-10T04:58:01.889738+0000","last_undegraded":"2021-02-10T04:58:01.889738+0000","last_fullsized":"2021-02-10T04:58:01.889738+0000","mapping_epoch":1248,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:11.139602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.139602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":"1266","reported_epoch":"1312","state":"active+clean","last_fresh":"2021-02-10T04:57:59.874656+0000","last_change":"2021-02-10T04:52:24.072005+0000","last_active":"2021-02-10T04:57:59.874656+0000","last_peered":"2021-02-10T04:57:59.874656+0000","last_clean":"2021-02-10T04:57:59.874656+0000","last_became_active":"2021-02-10T04:50:59.527945+0000","last_became_peered":"2021-02-10T04:50:59.527945+0000","last_unstale":"2021-02-10T04:57:59.874656+0000","last_undegraded":"2021-02-10T04:57:59.874656+0000","last_fullsized":"2021-02-10T04:57:59.874656+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:24.071915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_clean_scrub_stamp":"2021-02-10T04:52:24.071915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":972330808,"num_objects":390,"num_object_clones":337,"num_object_copies":1170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":390,"num_whiteouts":10,"num_read":4442,"num_read_kb":4559373,"num_write":3227,"num_write_kb":1247575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":611,"num_bytes_recovered":311961170,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2010624000,"data_stored":2589160348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34722,"ondisk_log_size":34722,"up":108,"acting":108,"num_store_stats":8},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":2368,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":18,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1139,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":5835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":126,"acting":126,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":13,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":807,"num_read_kb":802,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20,"ondisk_log_size":20,"up":39,"acting":39,"num_store_stats":8},{"poolid":4,"num_pg":22,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":2717,"num_read_kb":2190,"num_write":1592,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":17736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":844,"ondisk_log_size":844,"up":66,"acting":66,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35,"ondisk_log_size":35,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":126,"acting":126,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365192,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":196192,"kb_used_data":149464,"kb_used_omap":0,"kb_used_meta":46720,"kb_avail":93548960,"statfs":{"total":95995035648,"available":95794135040,"internally_reserved":0,"allocated":153051136,"data_stored":202527790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":47841280},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:57:20 2021","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.89500000000000002,"15min":0.92800000000000005},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.98999999999999999,"5min":4.4340000000000002,"15min":4.4340000000000002},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":1.1990000000000001}]},{"osd":1,"last update":"Wed Feb 10 04:57:47 2021","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.86199999999999999,"15min":0.91500000000000004},"min":{"1min":0.60499999999999998,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.3200000000000001,"5min":4.6760000000000002,"15min":5.1349999999999998},"last":1.083},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.90800000000000003,"15min":0.96799999999999997},"min":{"1min":0.54400000000000004,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.796,"5min":4.4619999999999997,"15min":4.4619999999999997},"last":1.3200000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:57:47 2021","interfaces":[{"interface":"back","average":{"1min":7.0999999999999996,"5min":2.351,"15min":1.5489999999999999},"min":{"1min":0.54600000000000004,"5min":0.499,"15min":0.499},"max":{"1min":94.391999999999996,"5min":94.391999999999996,"15min":94.391999999999996},"last":1.4319999999999999},{"interface":"front","average":{"1min":6.9870000000000001,"5min":2.2930000000000001,"15min":1.575},"min":{"1min":0.58799999999999997,"5min":0.496,"15min":0.496},"max":{"1min":94.269999999999996,"5min":94.269999999999996,"15min":94.269999999999996},"last":0.77500000000000002}]},{"osd":3,"last update":"Wed Feb 10 04:58:03 2021","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.0569999999999999,"15min":1.244},"min":{"1min":0.73899999999999999,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.4510000000000001,"5min":4.5599999999999996,"15min":4.5890000000000004},"last":1.6000000000000001},{"interface":"front","average":{"1min":1.252,"5min":1.1910000000000001,"15min":1.357},"min":{"1min":0.78200000000000003,"5min":0.629,"15min":0.629},"max":{"1min":2.0499999999999998,"5min":4.6130000000000004,"15min":4.6609999999999996},"last":1.54}]},{"osd":4,"last update":"Wed Feb 10 04:57:47 2021","interfaces":[{"interface":"back","average":{"1min":4.7930000000000001,"5min":1.78,"15min":1.5449999999999999},"min":{"1min":0.64800000000000002,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":72.662999999999997,"5min":72.662999999999997,"15min":72.662999999999997},"last":0.90500000000000003},{"interface":"front","average":{"1min":5.0499999999999998,"5min":1.921,"15min":1.5429999999999999},"min":{"1min":0.85499999999999998,"5min":0.71999999999999997,"15min":0.69099999999999995},"max":{"1min":73.564999999999998,"5min":73.564999999999998,"15min":73.564999999999998},"last":1.736}]},{"osd":5,"last update":"Wed Feb 10 04:57:47 2021","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.4079999999999999,"15min":1.486},"min":{"1min":1.032,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.258,"5min":4.766,"15min":14.972},"last":3.125},{"interface":"front","average":{"1min":1.171,"5min":1.298,"15min":1.5700000000000001},"min":{"1min":0.84899999999999998,"5min":0.78700000000000003,"15min":0.72699999999999998},"max":{"1min":2.1110000000000002,"5min":4.4020000000000001,"15min":15.218999999999999},"last":2.0289999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:57:47 2021","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.2649999999999999,"15min":1.631},"min":{"1min":0.78200000000000003,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.617,"5min":2.617,"15min":16.050000000000001},"last":3.1680000000000001},{"interface":"front","average":{"1min":1.401,"5min":1.3560000000000001,"15min":1.6279999999999999},"min":{"1min":0.92700000000000005,"5min":0.878,"15min":0.876},"max":{"1min":1.9410000000000001,"5min":4.7889999999999997,"15min":15.960000000000001},"last":2.2120000000000002}]}]},{"osd":6,"up_from":43,"seq":184683594124,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":392668,"kb_used_data":342548,"kb_used_omap":0,"kb_used_meta":50112,"kb_avail":93352484,"statfs":{"total":95995035648,"available":95592943616,"internally_reserved":0,"allocated":350769152,"data_stored":443485918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":51314688},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:57:18 2021","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.88100000000000001,"15min":0.88800000000000001},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.6040000000000001,"5min":1.6599999999999999,"15min":1.6599999999999999},"last":1.4179999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:57:25 2021","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.999,"15min":1.0289999999999999},"min":{"1min":0.55400000000000005,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.7330000000000001,"5min":13.18,"15min":13.18},"last":1.28},{"interface":"front","average":{"1min":0.88,"5min":0.97799999999999998,"15min":1.075},"min":{"1min":0.59199999999999997,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.3740000000000001,"5min":11.041,"15min":11.041},"last":1.042}]},{"osd":2,"last update":"Wed Feb 10 04:57:25 2021","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":3.2759999999999998,"15min":1.9279999999999999},"min":{"1min":0.60399999999999998,"5min":0.503,"15min":0.503},"max":{"1min":1.6459999999999999,"5min":163.81700000000001,"15min":163.81700000000001},"last":1.175},{"interface":"front","average":{"1min":1.1499999999999999,"5min":3.379,"15min":1.974},"min":{"1min":0.78200000000000003,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.8460000000000001,"5min":164.56800000000001,"15min":164.56800000000001},"last":1.4930000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.1060000000000001,"15min":1.181},"min":{"1min":0.69499999999999995,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.944,"5min":1.944,"15min":1.944},"last":1.556},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.1180000000000001,"15min":1.1220000000000001},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.766,"5min":2.363,"15min":2.363},"last":1.3169999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:57:25 2021","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.135,"15min":1.333},"min":{"1min":0.77700000000000002,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.016,"5min":2.2360000000000002,"15min":7.1459999999999999},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.98399999999999999,"15min":1.0840000000000001},"min":{"1min":0.66100000000000003,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.5649999999999999,"5min":5.2750000000000004,"15min":11.997999999999999},"last":1.3580000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:57:25 2021","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.4239999999999999,"15min":1.6519999999999999},"min":{"1min":0.79100000000000004,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":4.3449999999999998,"5min":22.663,"15min":81.540000000000006},"last":0.877},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.6990000000000001,"15min":1.9359999999999999},"min":{"1min":0.99399999999999999,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":2.048,"5min":22.933,"15min":81.492999999999995},"last":1.6160000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:57:49 2021","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.1279999999999999,"15min":1.2669999999999999},"min":{"1min":0.72599999999999998,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.677,"5min":2.073,"15min":5.1820000000000004},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.401,"5min":1.3640000000000001,"15min":1.5409999999999999},"min":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":2.1459999999999999,"5min":2.2970000000000002,"15min":5.4109999999999996},"last":1.7470000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575262,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":262044,"kb_used_data":206548,"kb_used_omap":0,"kb_used_meta":55488,"kb_avail":93483108,"statfs":{"total":95995035648,"available":95726702592,"internally_reserved":0,"allocated":211505152,"data_stored":265987846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":56819712},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:57:18 2021","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.69799999999999995,"15min":0.68899999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":2.3439999999999999,"5min":2.3439999999999999,"15min":2.3439999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.996,"5min":0.93899999999999995,"15min":0.92900000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":2.7589999999999999,"5min":2.7589999999999999,"15min":2.7589999999999999},"last":1.046}]},{"osd":2,"last update":"Wed Feb 10 04:57:18 2021","interfaces":[{"interface":"back","average":{"1min":11.41,"5min":6.0540000000000003,"15min":3.2999999999999998},"min":{"1min":0.60199999999999998,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":101.06100000000001,"5min":162.61199999999999,"15min":162.61199999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":11.478999999999999,"5min":6.1760000000000002,"15min":3.3900000000000001},"min":{"1min":0.52200000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":101.66500000000001,"5min":162.762,"15min":162.762},"last":0.66200000000000003}]},{"osd":3,"last update":"Wed Feb 10 04:58:07 2021","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":0.81799999999999995,"15min":0.90400000000000003},"min":{"1min":0.65500000000000003,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":2.8919999999999999,"5min":2.8919999999999999,"15min":3.077},"last":0.746},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.84299999999999997,"15min":0.97399999999999998},"min":{"1min":0.58999999999999997,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":3.1890000000000001,"5min":3.1890000000000001,"15min":3.1890000000000001},"last":0.95399999999999996}]},{"osd":4,"last update":"Wed Feb 10 04:58:03 2021","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.917,"15min":1.415},"min":{"1min":0.63100000000000001,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":3.101,"5min":100.071,"15min":100.071},"last":0.81200000000000006},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.9370000000000001,"15min":1.5349999999999999},"min":{"1min":0.55700000000000005,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":3.5390000000000001,"5min":100.43899999999999,"15min":100.43899999999999},"last":1.1699999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:58:07 2021","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.1240000000000001,"15min":1.159},"min":{"1min":0.96199999999999997,"5min":0.59799999999999998,"15min":0.56599999999999995},"max":{"1min":3.0219999999999998,"5min":3.0219999999999998,"15min":5.0300000000000002},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.294,"5min":1.179,"15min":1.4219999999999999},"min":{"1min":0.84499999999999997,"5min":0.70399999999999996,"15min":0.68600000000000005},"max":{"1min":3.2410000000000001,"5min":6.032,"15min":28.678000000000001},"last":0.91600000000000004}]},{"osd":6,"last update":"Wed Feb 10 04:57:18 2021","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.379,"15min":1.4930000000000001},"min":{"1min":0.78700000000000003,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":3.302,"5min":30.997,"15min":30.997},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.4510000000000001,"15min":1.478},"min":{"1min":0.90700000000000003,"5min":0.64200000000000002,"15min":0.60499999999999998},"max":{"1min":3.1549999999999998,"5min":31.138000000000002,"15min":31.138000000000002},"last":1.2549999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:57:29 2021","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.2170000000000001,"15min":1.452},"min":{"1min":0.87,"5min":0.628,"15min":0.628},"max":{"1min":2.9889999999999999,"5min":6.2629999999999999,"15min":28.573},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.226,"15min":1.4710000000000001},"min":{"1min":0.98199999999999998,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":3.4129999999999998,"5min":3.4129999999999998,"15min":29.408999999999999},"last":1.341}]}]},{"osd":0,"up_from":1227,"seq":5269924872228,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":349576,"kb_used_data":338752,"kb_used_omap":306,"kb_used_meta":10509,"kb_avail":93395576,"statfs":{"total":95995035648,"available":95637069824,"internally_reserved":0,"allocated":346882048,"data_stored":471616854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":313579,"internal_metadata":10762005},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 04:57:15 2021","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.64600000000000002,"15min":0.64000000000000001},"min":{"1min":0.52900000000000003,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.622},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.73099999999999998,"15min":0.72299999999999998},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":2.4279999999999999,"5min":2.4279999999999999,"15min":2.4279999999999999},"last":0.85499999999999998}]},{"osd":2,"last update":"Wed Feb 10 04:57:15 2021","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":2.9620000000000002,"15min":3.2909999999999999},"min":{"1min":0.71899999999999997,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.1520000000000001,"5min":59.341999999999999,"15min":59.341999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":1.01,"5min":2.9740000000000002,"15min":3.302},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.2360000000000002,"5min":59.405999999999999,"15min":59.405999999999999},"last":0.77500000000000002}]},{"osd":3,"last update":"Wed Feb 10 04:57:15 2021","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.76600000000000001,"15min":0.751},"min":{"1min":0.59099999999999997,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.181,"5min":2.181,"15min":2.181},"last":0.64400000000000002},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.002,"15min":0.998},"min":{"1min":0.76400000000000001,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.99,"5min":1.99,"15min":1.99},"last":0.96899999999999997}]},{"osd":4,"last update":"Wed Feb 10 04:57:15 2021","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.95299999999999996,"15min":0.94899999999999995},"min":{"1min":0.64800000000000002,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.6280000000000001,"5min":2.6280000000000001,"15min":2.6280000000000001},"last":0.752},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.125,"15min":1.1120000000000001},"min":{"1min":0.70699999999999996,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.0529999999999999,"5min":2.0529999999999999,"15min":2.0529999999999999},"last":1.089}]},{"osd":5,"last update":"Wed Feb 10 04:57:15 2021","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.2490000000000001,"15min":1.2430000000000001},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":2.5459999999999998,"5min":2.5459999999999998,"15min":2.5459999999999998},"last":1.232},{"interface":"front","average":{"1min":1.194,"5min":1.1459999999999999,"15min":1.1379999999999999},"min":{"1min":0.89600000000000002,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.4540000000000002,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":1.2070000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:57:15 2021","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.1890000000000001,"15min":1.177},"min":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"max":{"1min":1.9390000000000001,"5min":1.9390000000000001,"15min":1.9390000000000001},"last":1.6579999999999999},{"interface":"front","average":{"1min":1.26,"5min":1.218,"15min":1.2110000000000001},"min":{"1min":0.90000000000000002,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.5129999999999999,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":0.83699999999999997}]},{"osd":7,"last update":"Wed Feb 10 04:57:15 2021","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.2629999999999999,"15min":1.2569999999999999},"min":{"1min":1.0660000000000001,"5min":1.02,"15min":1.02},"max":{"1min":2.0710000000000002,"5min":2.0710000000000002,"15min":2.0710000000000002},"last":1.28},{"interface":"front","average":{"1min":1.258,"5min":1.2150000000000001,"15min":1.208},"min":{"1min":0.95999999999999996,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":2.335,"5min":2.335,"15min":2.335},"last":1.028}]}]},{"osd":2,"up_from":18,"seq":77309411740,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":354900,"kb_used_data":318732,"kb_used_omap":0,"kb_used_meta":36160,"kb_avail":93390252,"statfs":{"total":95995035648,"available":95631618048,"internally_reserved":0,"allocated":326381568,"data_stored":375016002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37027840},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:57:18 2021","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.67200000000000004,"15min":0.67000000000000004},"min":{"1min":0.50800000000000001,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.94799999999999995,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.87},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.93799999999999994,"15min":0.93999999999999995},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.9950000000000001,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":1.5229999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:57:58 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.83699999999999997,"15min":0.95599999999999996},"min":{"1min":0.47599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.26,"5min":17.041,"15min":17.041},"last":0.69299999999999995},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.014,"15min":0.93300000000000005},"min":{"1min":0.61199999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":2.1019999999999999,"5min":21.172000000000001,"15min":21.172000000000001},"last":1.1499999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:57:18 2021","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.0129999999999999,"15min":1.115},"min":{"1min":0.58399999999999996,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.29,"5min":18.904,"15min":18.904},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.286,"5min":1.262,"15min":1.3080000000000001},"min":{"1min":0.84799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":2.419,"5min":19.241,"15min":19.241},"last":1.117}]},{"osd":4,"last update":"Wed Feb 10 04:57:37 2021","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.1970000000000001,"15min":1.278},"min":{"1min":0.68799999999999994,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.7470000000000001,"5min":9.2029999999999994,"15min":9.2029999999999994},"last":0.78100000000000003},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.2989999999999999,"15min":1.1919999999999999},"min":{"1min":0.76300000000000001,"5min":0.503,"15min":0.503},"max":{"1min":2.2109999999999999,"5min":24.373999999999999,"15min":24.373999999999999},"last":1.3320000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:57:58 2021","interfaces":[{"interface":"back","average":{"1min":1.76,"5min":1.675,"15min":1.7490000000000001},"min":{"1min":0.89500000000000002,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":7.9050000000000002,"5min":39.911999999999999,"15min":39.911999999999999},"last":1.21},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.3660000000000001,"15min":1.488},"min":{"1min":0.69399999999999995,"5min":0.65900000000000003,"15min":0.57599999999999996},"max":{"1min":7.9969999999999999,"5min":33.539999999999999,"15min":33.539999999999999},"last":1.077}]},{"osd":6,"last update":"Wed Feb 10 04:57:31 2021","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":1.0409999999999999,"15min":1.375},"min":{"1min":0.84899999999999998,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.7290000000000001,"5min":3.1659999999999999,"15min":20.007000000000001},"last":1.226},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.4530000000000001,"15min":1.6220000000000001},"min":{"1min":1.069,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.8999999999999999,"5min":16.135000000000002,"15min":19.417999999999999},"last":1.2709999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:57:37 2021","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.637,"15min":1.641},"min":{"1min":1.117,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.748,"5min":24.405000000000001,"15min":24.405000000000001},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.284,"5min":1.196,"15min":1.2789999999999999},"min":{"1min":0.95899999999999996,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.4740000000000002,"5min":3.6640000000000001,"15min":4.9189999999999996},"last":1.651}]}]},{"osd":3,"up_from":1111,"seq":4771708665943,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":333492,"kb_used_data":301484,"kb_used_omap":386,"kb_used_meta":31613,"kb_avail":93411660,"statfs":{"total":95995035648,"available":95653539840,"internally_reserved":0,"allocated":308719616,"data_stored":409937649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":396036,"internal_metadata":32371964},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:57:18 2021","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":1.724,"15min":1.8939999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.004,"5min":22.869,"15min":22.869},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.72099999999999997,"5min":1.446,"15min":1.5669999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.98099999999999998,"5min":13.587999999999999,"15min":13.587999999999999},"last":0.64500000000000002}]},{"osd":1,"last update":"Wed Feb 10 04:57:09 2021","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.73899999999999999,"15min":0.67900000000000005},"min":{"1min":0.54100000000000004,"5min":0.44,"15min":0.44},"max":{"1min":1.304,"5min":3.6629999999999998,"15min":3.6629999999999998},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.879,"5min":0.79900000000000004,"15min":0.86199999999999999},"min":{"1min":0.60399999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":2.2450000000000001,"5min":3.7050000000000001,"15min":3.7050000000000001},"last":0.84399999999999997}]},{"osd":2,"last update":"Wed Feb 10 04:57:09 2021","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":1.337,"15min":1.212},"min":{"1min":0.53600000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0940000000000001,"5min":31.100999999999999,"15min":31.100999999999999},"last":0.622},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.587,"15min":1.276},"min":{"1min":0.92000000000000004,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":2.4390000000000001,"5min":31.052,"15min":31.052},"last":1.024}]},{"osd":4,"last update":"Wed Feb 10 04:57:09 2021","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1,"15min":0.97999999999999998},"min":{"1min":0.78600000000000003,"5min":0.628,"15min":0.628},"max":{"1min":1.734,"5min":2.9430000000000001,"15min":2.9430000000000001},"last":1.1399999999999999},{"interface":"front","average":{"1min":1.194,"5min":1.0029999999999999,"15min":1.087},"min":{"1min":0.81299999999999994,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":2.6739999999999999,"5min":3.21,"15min":3.21},"last":1.095}]},{"osd":5,"last update":"Wed Feb 10 04:57:09 2021","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.6930000000000001,"15min":2.3959999999999999},"min":{"1min":0.80900000000000005,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":2.7919999999999998,"5min":52.353000000000002,"15min":52.353000000000002},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.571,"15min":2.2189999999999999},"min":{"1min":0.78600000000000003,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.6140000000000001,"5min":52.301000000000002,"15min":52.301000000000002},"last":1.0680000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:57:09 2021","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.6459999999999999,"15min":4.3659999999999997},"min":{"1min":0.86099999999999999,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.571,"5min":51.914999999999999,"15min":64.376999999999995},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.429,"5min":1.8320000000000001,"15min":4.4000000000000004},"min":{"1min":1.091,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":2,"5min":51.896000000000001,"15min":64.438999999999993},"last":1.5669999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:57:09 2021","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.075,"15min":1.109},"min":{"1min":0.95299999999999996,"5min":0.77500000000000002,"15min":0.754},"max":{"1min":1.508,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.999},{"interface":"front","average":{"1min":1.353,"5min":1.1839999999999999,"15min":1.157},"min":{"1min":1.071,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.9059999999999999,"5min":2.9569999999999999,"15min":2.9569999999999999},"last":1.234}]}]},{"osd":4,"up_from":31,"seq":133143986580,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":165396,"kb_used_data":120716,"kb_used_omap":0,"kb_used_meta":44672,"kb_avail":93579756,"statfs":{"total":95995035648,"available":95825670144,"internally_reserved":0,"allocated":123613184,"data_stored":155839009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45744128},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:57:16 2021","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.85799999999999998,"15min":0.86499999999999999},"min":{"1min":0.53700000000000003,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.5449999999999999,"5min":3.637,"15min":3.637},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.878,"15min":0.89000000000000001},"min":{"1min":0.61399999999999999,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.2949999999999999,"5min":4.6829999999999998,"15min":4.6829999999999998},"last":0.85099999999999998}]},{"osd":1,"last update":"Wed Feb 10 04:57:48 2021","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.745,"15min":0.86099999999999999},"min":{"1min":0.58899999999999997,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.8120000000000001,"5min":3.2650000000000001,"15min":7.883},"last":1.5409999999999999},{"interface":"front","average":{"1min":1.0369999999999999,"5min":0.89500000000000002,"15min":1.0589999999999999},"min":{"1min":0.68999999999999995,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":6.5060000000000002},"last":0.93100000000000005}]},{"osd":2,"last update":"Wed Feb 10 04:57:48 2021","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.1910000000000001,"15min":1.702},"min":{"1min":0.77600000000000002,"5min":0.53900000000000003,"15min":0.51000000000000001},"max":{"1min":1.889,"5min":11.776,"15min":120.39},"last":1.397},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.129,"15min":1.6779999999999999},"min":{"1min":0.82499999999999996,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.4019999999999999,"5min":11.795999999999999,"15min":120.583},"last":1.4339999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:58:05 2021","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.107,"15min":1.2290000000000001},"min":{"1min":0.69599999999999995,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.581,"5min":1.998,"15min":2.1429999999999998},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.171,"5min":1.534,"15min":1.516},"min":{"1min":0.89800000000000002,"5min":0.65200000000000002,"15min":0.57399999999999995},"max":{"1min":1.6479999999999999,"5min":26.802,"15min":26.802},"last":1.0469999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:58:05 2021","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.046,"15min":1.431},"min":{"1min":0.628,"5min":0.495,"15min":0.495},"max":{"1min":1.7010000000000001,"5min":2.351,"15min":35.774000000000001},"last":0.97099999999999997},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.1890000000000001,"15min":1.431},"min":{"1min":0.66900000000000004,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.6990000000000001,"5min":4.319,"15min":35.606000000000002},"last":1.655}]},{"osd":6,"last update":"Wed Feb 10 04:58:05 2021","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.464,"15min":2.3679999999999999},"min":{"1min":1.1120000000000001,"5min":0.873,"15min":0.86199999999999999},"max":{"1min":2.073,"5min":3.8530000000000002,"15min":201.626},"last":1.502},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.3049999999999999,"15min":2.351},"min":{"1min":0.96899999999999997,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.8480000000000001,"5min":4.7910000000000004,"15min":201.577},"last":1.079}]},{"osd":7,"last update":"Wed Feb 10 04:57:37 2021","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.4470000000000001,"15min":1.508},"min":{"1min":1.111,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.2170000000000001,"5min":4.6200000000000001,"15min":5.8920000000000003},"last":1.284},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.2609999999999999,"15min":1.4590000000000001},"min":{"1min":0.99399999999999999,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":3.423,"5min":3.423,"15min":5.149},"last":1.5669999999999999}]}]},{"osd":5,"up_from":37,"seq":158913790353,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":256928,"kb_used_data":206488,"kb_used_omap":0,"kb_used_meta":50432,"kb_avail":93488224,"statfs":{"total":95995035648,"available":95731941376,"internally_reserved":0,"allocated":211443712,"data_stored":275912538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":51642368},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:57:18 2021","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.1799999999999999,"5min":1.486,"15min":1.486},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.76400000000000001,"15min":0.753},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.55700000000000005}]},{"osd":1,"last update":"Wed Feb 10 04:58:04 2021","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.90900000000000003,"15min":0.90300000000000002},"min":{"1min":0.54500000000000004,"5min":0.435,"15min":0.435},"max":{"1min":1.3069999999999999,"5min":6.391,"15min":6.391},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.83099999999999996,"5min":1.0620000000000001,"15min":1.147},"min":{"1min":0.54200000000000004,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.653,"5min":13.794,"15min":13.794},"last":0.81799999999999995}]},{"osd":2,"last update":"Wed Feb 10 04:58:04 2021","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":3.4500000000000002,"15min":2.1640000000000001},"min":{"1min":0.61899999999999999,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.478,"5min":116.438,"15min":116.438},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.004,"5min":3.569,"15min":2.21},"min":{"1min":0.61199999999999999,"5min":0.56200000000000006,"15min":0.55200000000000005},"max":{"1min":1.5609999999999999,"5min":116.96899999999999,"15min":116.96899999999999},"last":0.76400000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.3300000000000001,"15min":1.333},"min":{"1min":0.81100000000000005,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":12.428000000000001,"5min":13.755000000000001,"15min":13.755000000000001},"last":1.7010000000000001},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.234,"15min":1.198},"min":{"1min":0.79700000000000004,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.95,"5min":6.5129999999999999,"15min":6.5129999999999999},"last":1.5289999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:58:04 2021","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.0740000000000001,"15min":1.26},"min":{"1min":0.66600000000000004,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.865,"5min":5.298,"15min":9.4890000000000008},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.23,"5min":1.2110000000000001,"15min":1.343},"min":{"1min":0.80400000000000005,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":2.238,"5min":4.8289999999999997,"15min":5.6180000000000003},"last":1.4830000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.397,"15min":2.9590000000000001},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.2149999999999999,"5min":4.9429999999999996,"15min":319.51999999999998},"last":1.371},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.448,"15min":2.895},"min":{"1min":0.83499999999999996,"5min":0.71399999999999997,"15min":0.68799999999999994},"max":{"1min":2.7290000000000001,"5min":6.8220000000000001,"15min":319.22300000000001},"last":1.3460000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:57:41 2021","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.244,"15min":1.4550000000000001},"min":{"1min":0.754,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.8009999999999999,"5min":5.4160000000000004,"15min":11.897},"last":1.083},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.4099999999999999,"15min":1.548},"min":{"1min":1.0189999999999999,"5min":0.91000000000000003,"15min":0.82999999999999996},"max":{"1min":2.4369999999999998,"5min":5.0579999999999998,"15min":6.0369999999999999},"last":1.593}]}]}]}} 2021-02-10T04:58:10.481 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2021-02-10T04:58:10.481 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:58:10.481 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [1, 2, 3, 4, 5, 6, 7, 0] 2021-02-10T04:58:10.482 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2021-02-10T04:58:10.482 DEBUG:teuthology.orchestra.run.smithi085:> sudo systemctl stop ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.0 2021-02-10T04:58:10.559 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:58:10.614 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:10 smithi085 systemd[1]: Stopping Ceph osd.0 for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T04:58:10.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:10 smithi085 conmon[29319]: audit 2021-02-10T04:58:09.476033+0000 mgr.y ( 2021-02-10T04:58:10.615 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:10 smithi085 conmon[29319]: mgr.14140) 1474 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:58:10.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:10 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:58:10.615 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:10 smithi085 conmon[35961]: :58:09.476033+0000 mgr.y (mgr.14140) 1474 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:58:10.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:10 smithi164 conmon[29288]: audit 2021-02-10T04:58:09.476033+0000 mgr.y (mgr.14140) 2021-02-10T04:58:10.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:10 smithi164 conmon[29288]: 1474 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:58:10.942 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:10 smithi085 conmon[177076]: debug 2021-02-10T04:58:10.639+0000 7f14cc75b700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T04:58:10.942 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:10 smithi085 conmon[177076]: debug 2021-02-10T04:58:10.639+0000 7f14cc75b700 -1 osd.0 1315 *** Got signal Terminated *** 2021-02-10T04:58:10.942 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:10 smithi085 conmon[177076]: debug 2021-02-10T04:58:10.639+0000 7f14cc75b700 -1 osd.0 1315 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-02-10T04:58:11.638 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is set 2021-02-10T04:58:11.758 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T04:58:10.543628+0000 mgr.y ( 2021-02-10T04:58:11.759 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: mgr.14140) 1475 : cluster [DBG] pgmap v2401: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:58:11.759 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T04:58:10.869070+0000 mon.a (mon.0) 2864 : cluster [DBG] osd.0 reported immediately failed by osd.2 2021-02-10T04:58:11.759 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T04:58:10.869112+0000 mon.a (mon.0) 2865 : cluster [INF] osd.0 failed (root=default,host=smithi085) (connection refused reported by osd.2) 2021-02-10T04:58:11.759 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T04:58:10.869309+0000 mon.a (mon.0) 2866 : cluster [DBG] osd.0 reported immediately failed by osd.2 2021-02-10T04:58:11.759 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T04:58:10.869453+0000 mon.a (mon.0) 2867 : cluster [DBG] osd.0 reported immediately failed by osd.1 2021-02-10T04:58:11.759 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T04:58:10.869663+0000 mon.a (mon.0) 2868 : cluster [DBG] osd.0 reported immediately failed by osd.7 2021-02-10T04:58:11.759 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T04:58:10.869824+0000 mon.a (mon.0) 2869 : cluster [DBG] osd.0 reported immediately failed by osd.3 2021-02-10T04:58:11.759 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T04:58:10.869984+0000 mon.a (mon.0) 2870 : cluster [DBG] osd.0 reported immediately failed by osd.7 2021-02-10T04:58:11.759 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T04:58:10.870077+0000 mon.a (mon.0) 2871 : cluster [DBG] osd.0 reported immediately failed by osd.2 2021-02-10T04:58:11.759 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T04:58:10.870176+0000 mon.a (mon.0) 2872 : cluster [DBG] osd.0 reported immediately failed by osd.3 2021-02-10T04:58:11.759 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T04:58:10.870279+0000 mon.a (mon.0) 2873 : cluster [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:58:11.759 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T04:58:10.870398+0000 mon.a 2021-02-10T04:58:11.759 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: (mon.0) 2874 : cluster [DBG] osd.0 reported immediately failed by osd.3 2021-02-10T04:58:11.760 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T04:58:10.870492+0000 mon.a (mon.0) 2875 : cluster [DBG] osd.0 reported immediately failed by osd.6 2021-02-10T04:58:11.760 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T04:58:10.870617+0000 mon.a (mon.0) 2876 : cluster 2021-02-10T04:58:11.760 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: [DBG] osd.0 reported immediately failed by osd.1 2021-02-10T04:58:11.760 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T04:58:10.870720+0000 mon.a (mon.0) 2877 : cluster 2021-02-10T04:58:11.760 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:58:11.760 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:58:11.760 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: 10T04:58:10.870838+0000 mon.a (mon.0) 2878 : cluster [DBG] 2021-02-10T04:58:11.760 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: osd.0 reported immediately failed by osd.1 2021-02-10T04:58:11.760 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:58:11.760 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: 10T04:58:10.870937+0000 mon.a (mon.0) 2879 : cluster 2021-02-10T04:58:11.760 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: [DBG] osd.0 reported immediately failed by osd.5 2021-02-10T04:58:11.760 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T04:58: 2021-02-10T04:58:11.760 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: 10.871066+0000 mon.a (mon.0) 2880 : cluster 2021-02-10T04:58:11.761 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: [DBG] osd.0 reported immediately failed by osd.6 2021-02-10T04:58:11.761 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T04:58:10 2021-02-10T04:58:11.761 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: .871180+0000 mon.a (mon.0) 2881 : cluster 2021-02-10T04:58:11.761 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: [DBG] osd.0 reported immediately failed by osd.7 2021-02-10T04:58:11.761 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T04:58:11.761 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: 2021-02-10T04:58:10.871297+0000 mon.a ( 2021-02-10T04:58:11.761 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: mon.0) 2882 : cluster [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:58:11.761 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:58:11.761 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: 04:58:10.871416+0000 mon.a (mon.0) 2021-02-10T04:58:11.761 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: 2883 : cluster [DBG] osd.0 reported immediately failed by osd.5 2021-02-10T04:58:11.761 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T04:58:10.871531+0000 2021-02-10T04:58:11.761 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: mon.a (mon.0) 2884 : cluster [DBG] osd.0 reported immediately failed by osd.6 2021-02-10T04:58:11.761 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:58:11.762 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: 02-10T04:58:10.871658+0000 mon.a (mon 2021-02-10T04:58:11.762 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: .0) 2885 : cluster [DBG] osd.0 reported immediately failed by osd.5 2021-02-10T04:58:11.762 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:58:11.762 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: 10T04:58:11.270089+0000 mon.a (mon. 2021-02-10T04:58:11.762 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: 0) 2886 : cluster [DBG] osd.0 reported immediately failed by osd.2 2021-02-10T04:58:11.762 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T04 2021-02-10T04:58:11.762 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: :58:11.270227+0000 mon.a (mon.0) 2021-02-10T04:58:11.762 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: 2887 : cluster [DBG] osd.0 reported immediately failed by osd.2 2021-02-10T04:58:11.762 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T04:58: 2021-02-10T04:58:11.762 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: 11.270407+0000 mon.a (mon.0) 2888 : 2021-02-10T04:58:11.762 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster [DBG] osd.0 reported immediately failed by osd.1 2021-02-10T04:58:11.762 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T04:58 2021-02-10T04:58:11.762 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: :11.270556+0000 mon.a (mon.0) 2889 2021-02-10T04:58:11.763 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: : cluster [DBG] osd.0 reported immediately failed by osd.1 2021-02-10T04:58:11.763 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T04:58 2021-02-10T04:58:11.763 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: :11.270730+0000 mon.a (mon.0) 2890 : cluster 2021-02-10T04:58:11.763 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: [DBG] osd.0 reported immediately failed by osd.1 2021-02-10T04:58:11.763 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T04:58:11 2021-02-10T04:58:11.763 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: .270865+0000 mon.a (mon.0) 2891 : cluster 2021-02-10T04:58:11.763 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: [DBG] osd.0 reported immediately failed by osd.7 2021-02-10T04:58:11.763 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T04:58: 2021-02-10T04:58:11.763 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: 11.271002+0000 mon.a (mon.0) 2892 : 2021-02-10T04:58:11.763 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster [DBG] osd.0 reported immediately failed by osd.2 2021-02-10T04:58:11.763 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T04:58:11 2021-02-10T04:58:11.763 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: .271116+0000 mon.a (mon.0) 2893 : cluster 2021-02-10T04:58:11.763 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: [DBG] osd.0 reported immediately failed by osd.3 2021-02-10T04:58:11.763 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T04:58:11.271269 2021-02-10T04:58:11.764 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2894 : cluster [DBG] osd.0 reported immediately failed by osd.5 2021-02-10T04:58:11.764 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: 2021-02-10T04:58:11.764 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T04:58:11.764 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: 2021-02-10T04:58:11.271415+0000 mon.a (mon. 2021-02-10T04:58:11.764 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: 0) 2895 : cluster [DBG] osd.0 reported immediately failed by osd.3 2021-02-10T04:58:11.764 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:58:11.764 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: 10T04:58:11.271544+0000 mon.a (mon 2021-02-10T04:58:11.764 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: .0) 2896 : cluster [DBG] osd.0 reported immediately failed by osd.6 2021-02-10T04:58:11.764 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T04:58 2021-02-10T04:58:11.764 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: :11.271700+0000 mon.a (mon.0) 2897 2021-02-10T04:58:11.764 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: : cluster [DBG] osd.0 reported immediately failed by osd.3 2021-02-10T04:58:11.764 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:58:11.764 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: 04:58:11.271804+0000 mon.a (mon.0) 2021-02-10T04:58:11.765 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: 2898 : cluster [DBG] osd.0 reported immediately failed by osd.7 2021-02-10T04:58:11.765 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:58:11.765 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: T04:58:11.271896+0000 mon.a (mon.0 2021-02-10T04:58:11.765 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: ) 2899 : cluster [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:58:11.765 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:58:11.765 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: T04:58:11.271987+0000 mon.a (mon 2021-02-10T04:58:11.765 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: .0) 2900 : cluster [DBG] osd.0 reported immediately failed by osd.5 2021-02-10T04:58:11.765 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:58:11.765 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: -10T04:58:11.272086+0000 2021-02-10T04:58:11.765 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: mon.a (mon.0) 2901 : cluster [DBG] osd.0 reported immediately failed by osd.6 2021-02-10T04:58:11.765 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: 2021-02-10T04:58:11.765 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02-10T04:58: 2021-02-10T04:58:11.765 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: 11.272176+0000 mon.a (mon.0 2021-02-10T04:58:11.765 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: ) 2902 : cluster [DBG] osd.0 reported immediately failed by osd.7 2021-02-10T04:58:11.766 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:58:11.766 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: -10T04:58:11.272270+0000 mon.a 2021-02-10T04:58:11.766 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: (mon.0) 2903 : cluster [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:58:11.766 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:58:11.766 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: 02-10T04:58:11.272362+0000 mon.a (mon. 2021-02-10T04:58:11.766 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: 0) 2904 : cluster [DBG] osd.0 reported immediately failed by osd.5 2021-02-10T04:58:11.766 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:58:11.766 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: -10T04:58:11.272457+0000 mon.a 2021-02-10T04:58:11.766 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: (mon.0) 2905 : cluster [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:58:11.766 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:58:11.766 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: 02-10T04:58:11.272553+0000 2021-02-10T04:58:11.766 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: mon.a (mon.0) 2906 : cluster [DBG] 2021-02-10T04:58:11.767 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: osd.0 reported immediately failed by osd.6 2021-02-10T04:58:11.767 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: audit 2021-02-10T04:58:11.767 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: 2021-02-10T04:58:11.487112+0000 mon.a (mon 2021-02-10T04:58:11.767 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[29319]: .0) 2907 : audit [INF] from='client.? 172.21.15.85:0/206669841' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:58:11.767 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02-10T04:58:10.543628 2021-02-10T04:58:11.767 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1475 : cluster [DBG] pgmap v2401: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:58:11.767 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02-10T04:58:10.869070+0000 mon.a (mon.0) 2864 : cluster [DBG] osd.0 reported immediately failed by osd.2 2021-02-10T04:58:11.767 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02-10T04:58:10.869112+0000 mon.a (mon.0) 2865 : cluster [INF] osd.0 failed (root=default,host=smithi085) (connection refused reported by osd.2) 2021-02-10T04:58:11.767 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02-10T04:58:10.869309+0000 mon.a (mon.0) 2866 : cluster [DBG] osd.0 reported immediately failed by osd.2 2021-02-10T04:58:11.767 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02-10T04:58:10.869453+0000 mon.a (mon.0) 2867 : cluster [DBG] osd.0 reported immediately failed by osd.1 2021-02-10T04:58:11.767 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02-10T04:58:10.869663+0000 mon.a (mon.0) 2868 : cluster [DBG] osd.0 reported immediately failed by osd.7 2021-02-10T04:58:11.767 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02-10T04:58:10.869824+0000 mon.a (mon.0) 2869 : cluster [DBG] osd.0 reported immediately failed by osd.3 2021-02-10T04:58:11.767 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02-10T04:58:10.869984+0000 mon.a (mon.0) 2870 : cluster [DBG] osd.0 reported immediately failed by osd.7 2021-02-10T04:58:11.768 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02-10T04:58:10.870077+0000 mon.a (mon.0) 2871 : cluster [DBG] osd.0 reported immediately failed by osd.2 2021-02-10T04:58:11.768 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02-10T04:58:10 2021-02-10T04:58:11.768 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: .870176+0000 mon.a (mon.0) 2872 : cluster [DBG] osd.0 reported immediately failed by osd.3 2021-02-10T04:58:11.768 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:58:11.768 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: -02-10T04:58:10.870279+0000 mon.a 2021-02-10T04:58:11.768 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: (mon.0) 2873 : cluster [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:58:11.768 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02-10T04:58:11.768 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: 2021-02-10T04:58:10.870398+0000 2021-02-10T04:58:11.768 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: mon.a (mon.0) 2874 : cluster [DBG] osd.0 reported immediately failed by osd.3 2021-02-10T04:58:11.768 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:58:11.768 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: -10T04:58:10.870492+0000 2021-02-10T04:58:11.768 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: mon.a (mon.0) 2875 : cluster [DBG] osd.0 reported immediately failed by osd.6 2021-02-10T04:58:11.768 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:58:11.769 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: -02-10T04:58:10. 2021-02-10T04:58:11.769 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: 870617+0000 mon.a (mon.0) 2876 : cluster [DBG] 2021-02-10T04:58:11.769 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: osd.0 reported immediately failed by osd.1 2021-02-10T04:58:11.769 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02-10T04:58: 2021-02-10T04:58:11.769 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: 10.870720+0000 mon.a (mon.0) 2877 : cluster 2021-02-10T04:58:11.769 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:58:11.769 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:58:11.769 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: 58:10.870838+0000 mon.a (mon.0) 2878 : 2021-02-10T04:58:11.769 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster [DBG] osd.0 reported immediately failed by osd.1 2021-02-10T04:58:11.769 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:58:11.769 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: -10T04:58:10.870937+0000 mon.a (mon. 2021-02-10T04:58:11.769 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: 0) 2879 : cluster [DBG] osd.0 reported immediately failed by osd.5 2021-02-10T04:58:11.769 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:58:11.770 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: -02-10T04:58: 2021-02-10T04:58:11.770 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: 10.871066+0000 mon.a (mon. 2021-02-10T04:58:11.770 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: 0) 2880 : cluster [DBG] 2021-02-10T04:58:11.770 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: osd.0 reported immediately failed by osd.6 2021-02-10T04:58:11.770 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02-10T04:58:11.770 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:58:11.770 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: 58:10.871180+0000 mon.a (mon.0) 2021-02-10T04:58:11.770 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: 2881 : cluster [DBG] osd.0 reported immediately failed by osd.7 2021-02-10T04:58:11.770 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:58:11.770 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: -02-10T04:58:10.871297 2021-02-10T04:58:11.770 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2882 : cluster [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:58:11.770 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: 2021-02-10T04:58:11.770 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02-10T04:58:11.771 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: 2021-02-10T04:58:10.871416+0000 mon.a ( 2021-02-10T04:58:11.771 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: mon.0) 2883 : cluster [DBG] osd.0 reported immediately failed by osd.5 2021-02-10T04:58:11.771 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:58:11.771 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: -02-10T04:58:10.871531+0000 mon.a 2021-02-10T04:58:11.771 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: (mon.0) 2884 : cluster [DBG] osd.0 reported immediately failed by osd.6 2021-02-10T04:58:11.771 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:58:11.771 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: -02-10T04:58:10.871658+0000 2021-02-10T04:58:11.771 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: mon.a (mon.0) 2885 : cluster [DBG] osd.0 reported immediately failed by osd.5 2021-02-10T04:58:11.771 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02-10T04:58:11.270089+0000 mon.a (mon.0) 2021-02-10T04:58:11.771 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: 2886 : cluster [DBG] osd.0 reported immediately failed by osd.2 2021-02-10T04:58:11.771 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02-10T04:58:11.270227+0000 mon.a (mon.0) 2887 : cluster [DBG] osd.0 reported immediately failed by osd.2 2021-02-10T04:58:11.771 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:58:11.771 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: -02-10T04:58:11.270407+0000 mon.a (mon 2021-02-10T04:58:11.771 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: .0) 2888 : cluster [DBG] osd.0 reported immediately failed by osd.1 2021-02-10T04:58:11.772 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: 2021-02-10T04:58:11.772 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02-10T04:58:11.772 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: 2021-02-10T04:58 2021-02-10T04:58:11.772 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: :11.270556+0000 mon.a (mon 2021-02-10T04:58:11.772 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: .0) 2889 : cluster [DBG] 2021-02-10T04:58:11.772 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: osd.0 reported immediately failed by osd.1 2021-02-10T04:58:11.772 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:58:11.772 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: -02-10T04:58:11.270730+0000 2021-02-10T04:58:11.772 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: mon.a (mon.0) 2890 : cluster [DBG] osd.0 reported immediately failed by osd.1 2021-02-10T04:58:11.772 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:58:11.772 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: -02-10T04:58:11 2021-02-10T04:58:11.772 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: .270865+0000 mon.a (mon. 2021-02-10T04:58:11.772 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: 0) 2891 : cluster [DBG] osd.0 reported immediately failed by osd.7 2021-02-10T04:58:11.772 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: 2021-02-10T04:58:11.773 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:58:11.773 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: -02-10T04:58: 2021-02-10T04:58:11.773 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: 11.271002+0000 mon.a (mon 2021-02-10T04:58:11.773 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: .0) 2892 : cluster [DBG] 2021-02-10T04:58:11.773 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: osd.0 reported immediately failed by osd.2 2021-02-10T04:58:11.773 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02-10T04:58:11.773 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:58:11.773 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: 58:11.271116+0000 mon.a ( 2021-02-10T04:58:11.773 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: mon.0) 2893 : cluster [DBG] osd.0 reported immediately failed by osd.3 2021-02-10T04:58:11.773 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02-10T04:58:11.773 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: 2021-02-10T04:58:11.271269+0000 mon.a (mon.0) 2894 : cluster [DBG] osd.0 reported immediately failed by osd.5 2021-02-10T04:58:11.773 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02-10T04:58:11.271415+0000 mon.a (mon.0) 2895 : cluster [DBG] osd.0 reported immediately failed by osd.3 2021-02-10T04:58:11.773 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:58:11.774 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: 58:11.271544+0000 mon.a ( 2021-02-10T04:58:11.774 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: mon.0) 2896 : cluster 2021-02-10T04:58:11.774 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: [DBG] osd.0 reported immediately failed by osd.6 2021-02-10T04:58:11.774 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:58:11.774 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: -02-10T04:58:11.271700+0000 mon.a 2021-02-10T04:58:11.774 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: (mon.0) 2897 : cluster [DBG] osd.0 reported immediately failed by osd.3 2021-02-10T04:58:11.774 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02-10T04:58:11.774 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: 2021-02-10T04:58:11.271804+0000 2021-02-10T04:58:11.774 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: mon.a (mon.0) 2898 : cluster [DBG] osd.0 reported immediately failed by osd.7 2021-02-10T04:58:11.774 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: 2021-02-10T04:58:11.774 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:58:11.774 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: -10T04:58:11.271896+0000 mon.a (mon 2021-02-10T04:58:11.775 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: .0) 2899 : cluster [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:58:11.775 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:58:11.775 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: 02-10T04:58:11.271987+0000 2021-02-10T04:58:11.775 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: mon.a (mon.0) 2900 : cluster 2021-02-10T04:58:11.775 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: [DBG] osd.0 reported immediately failed by osd.5 2021-02-10T04:58:11.775 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:58:11.775 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: -10T04:58:11.272086+0000 mon.a 2021-02-10T04:58:11.775 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: (mon.0) 2901 : cluster [DBG] osd.0 reported immediately failed by osd.6 2021-02-10T04:58:11.775 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:58:11.775 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: -10T04:58:11.272176+0000 mon.a (mon.0) 2902 2021-02-10T04:58:11.775 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: : cluster [DBG] osd.0 reported immediately failed by osd.7 2021-02-10T04:58:11.775 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02-10T04:58:11. 2021-02-10T04:58:11.775 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: 272270+0000 mon.a (mon.0) 2903 : cluster [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:58:11.775 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:58:11.776 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: 58:11.272362+0000 mon.a (mon.0) 2904 : cluster 2021-02-10T04:58:11.776 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: [DBG] osd.0 reported immediately failed by osd.5 2021-02-10T04:58:11.776 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02-10T04:58:11.272457+0000 mon.a 2021-02-10T04:58:11.776 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: (mon.0) 2905 : cluster [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:58:11.776 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:58:11.776 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: T04:58:11.272553+0000 mon.a (mon.0) 2906 2021-02-10T04:58:11.776 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: : cluster [DBG] osd.0 reported immediately failed by osd.6 2021-02-10T04:58:11.776 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: audit 2021-02-10T04:58: 2021-02-10T04:58:11.776 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: 11.487112+0000 mon.a (mon.0) 2907 2021-02-10T04:58:11.776 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:11 smithi085 conmon[35961]: : audit [INF] from='client.? 172.21.15.85:0/206669841' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:58:11.776 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:11 smithi085 podman[200829]: 330bb351d96a77c87a96463d50838f8ea5fc6291ec64732d103fcdf377e53f96 2021-02-10T04:58:11.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02-10T04:58:10. 2021-02-10T04:58:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 543628+0000 mgr.y (mgr.14140) 1475 : cluster [DBG] pgmap v2401: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-02-10T04:58:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:58:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 02-10T04:58:10.869070+0000 mon.a (mon.0) 2864 : cluster [DBG] 2021-02-10T04:58:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: osd.0 reported immediately failed by osd.2 2021-02-10T04:58:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02-10T04:58:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 2021-02-10T04:58:10.869112+0000 mon.a (mon.0) 2865 : 2021-02-10T04:58:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster [INF] osd.0 failed (root=default,host=smithi085) (connection refused reported by osd.2) 2021-02-10T04:58:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:58:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: :58:10.869309+0000 mon.a (mon.0) 2866 2021-02-10T04:58:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: : cluster [DBG] osd.0 reported immediately failed by osd.2 2021-02-10T04:58:11.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02-10T04:58:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 2021-02-10T04:58:10.869453+0000 mon.a (mon.0) 2867 2021-02-10T04:58:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: : cluster [DBG] osd.0 reported immediately failed by osd.1 2021-02-10T04:58:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02-10T04:58:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 2021-02-10T04:58:10.869663+0000 mon.a (mon.0 2021-02-10T04:58:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: ) 2868 : cluster [DBG] osd.0 reported immediately failed by osd.7 2021-02-10T04:58:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:58:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 02-10T04:58:10.869824+0000 2021-02-10T04:58:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: mon.a (mon.0) 2869 : 2021-02-10T04:58:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster [DBG] osd.0 reported immediately failed by osd.3 2021-02-10T04:58:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:58:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 02-10T04:58:10.869984+0000 mon.a (mon 2021-02-10T04:58:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: .0) 2870 : cluster [DBG] osd.0 reported immediately failed by osd.7 2021-02-10T04:58:11.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:58:11.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: -10T04:58:10.870077+0000 mon.a 2021-02-10T04:58:11.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: (mon.0) 2871 : cluster [DBG] osd.0 reported immediately failed by osd.2 2021-02-10T04:58:11.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:58:11.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 02-10T04:58:10.870176+0000 2021-02-10T04:58:11.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: mon.a (mon.0) 2872 : cluster [DBG] 2021-02-10T04:58:11.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: osd.0 reported immediately failed by osd.3 2021-02-10T04:58:11.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:58:11.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: -02-10T04:58:10.870279+0000 2021-02-10T04:58:11.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: mon.a (mon.0) 2873 2021-02-10T04:58:11.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: : cluster [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:58:11.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:58:11.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 10T04:58:10.870398+0000 mon.a (mon.0) 2874 2021-02-10T04:58:11.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: : cluster [DBG] osd.0 reported immediately failed by osd.3 2021-02-10T04:58:11.897 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:58:11.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 58:10.870492+0000 mon.a (mon.0) 2875 : cluster 2021-02-10T04:58:11.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: [DBG] osd.0 reported immediately failed by osd.6 2021-02-10T04:58:11.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02-10T04:58:10.870617+0000 2021-02-10T04:58:11.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: mon.a (mon.0) 2876 : cluster [DBG] osd.0 reported immediately failed by osd.1 2021-02-10T04:58:11.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02-10T04:58:10.870720+0000 mon.a ( 2021-02-10T04:58:11.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: mon.0) 2877 : cluster [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:58:11.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02-10T04:58:10.870838+0000 mon.a (mon 2021-02-10T04:58:11.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: .0) 2878 : cluster [DBG] osd.0 reported immediately failed by osd.1 2021-02-10T04:58:11.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02-10T04:58:10.870937+0000 mon.a ( 2021-02-10T04:58:11.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: mon.0) 2879 : cluster [DBG] osd.0 reported immediately failed by osd.5 2021-02-10T04:58:11.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 2021-02-10T04:58:11.898 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:58:11.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 10T04:58:10.871066+0000 mon.a (mon.0) 2880 : cluster [DBG] osd.0 reported immediately failed by osd.6 2021-02-10T04:58:11.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02-10T04:58:10.871180+0000 2021-02-10T04:58:11.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: mon.a (mon.0) 2881 : cluster [DBG] osd.0 reported immediately failed by osd.7 2021-02-10T04:58:11.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02-10T04:58:10 2021-02-10T04:58:11.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: .871297+0000 mon.a (mon.0) 2882 : cluster 2021-02-10T04:58:11.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:58:11.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:58:11.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 58:10.871416+0000 mon.a (mon.0) 2021-02-10T04:58:11.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 2883 : cluster [DBG] osd.0 reported immediately failed by osd.5 2021-02-10T04:58:11.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:58:11.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: -10T04:58:10.871531+0000 mon.a 2021-02-10T04:58:11.899 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: (mon.0) 2884 : cluster [DBG] osd.0 reported immediately failed by osd.6 2021-02-10T04:58:11.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:58:11.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: -02-10T04:58:10 2021-02-10T04:58:11.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: .871658+0000 mon.a (mon. 2021-02-10T04:58:11.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 0) 2885 : cluster [DBG] osd.0 reported immediately failed by osd.5 2021-02-10T04:58:11.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:58:11.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: -02-10T04:58:11.270089+0000 mon.a 2021-02-10T04:58:11.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: (mon.0) 2886 : cluster [DBG] osd.0 reported immediately failed by osd.2 2021-02-10T04:58:11.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02-10T04:58:11.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 2021-02-10T04:58:11 2021-02-10T04:58:11.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: .270227+0000 mon.a (mon.0) 2887 : cluster 2021-02-10T04:58:11.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: [DBG] osd.0 reported immediately failed by osd.2 2021-02-10T04:58:11.900 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02-10T04:58:11.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 2021-02- 2021-02-10T04:58:11.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 10T04:58:11.270407+0000 mon.a 2021-02-10T04:58:11.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: (mon.0) 2888 : cluster [DBG] osd.0 reported immediately failed by osd.1 2021-02-10T04:58:11.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02-10T04:58:11.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 2021-02-10T04:58:11.270556 2021-02-10T04:58:11.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2889 : cluster [DBG] 2021-02-10T04:58:11.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: osd.0 reported immediately failed by osd.1 2021-02-10T04:58:11.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02-10T04:58:11.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 2021-02-10T04:58:11.270730+0000 mon.a 2021-02-10T04:58:11.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: (mon.0) 2890 : cluster 2021-02-10T04:58:11.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: [DBG] osd.0 reported immediately failed by osd.1 2021-02-10T04:58:11.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:58:11.901 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 04:58:11.270865+0000 mon.a (mon. 2021-02-10T04:58:11.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 0) 2891 : cluster [DBG] osd.0 reported immediately failed by osd.7 2021-02-10T04:58:11.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:58:11.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 04:58:11.271002+0000 mon.a 2021-02-10T04:58:11.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: (mon.0) 2892 : cluster 2021-02-10T04:58:11.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: [DBG] osd.0 reported immediately failed by osd.2 2021-02-10T04:58:11.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:58:11.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 58:11.271116+0000 mon.a ( 2021-02-10T04:58:11.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: mon.0) 2893 : cluster 2021-02-10T04:58:11.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: [DBG] osd.0 reported immediately failed by osd.3 2021-02-10T04:58:11.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02-10T04:58:11. 2021-02-10T04:58:11.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 271269+0000 mon.a (mon.0) 2894 : cluster [DBG] osd.0 reported immediately failed by osd.5 2021-02-10T04:58:11.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 2021-02-10T04:58:11.902 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02-10T04:58:11.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 2021-02-10T04:58:11.271415+0000 mon.a (mon.0 2021-02-10T04:58:11.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: ) 2895 : cluster [DBG] osd.0 reported immediately failed by osd.3 2021-02-10T04:58:11.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:58:11.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 58:11.271544+0000 mon.a (mon.0 2021-02-10T04:58:11.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: ) 2896 : cluster [DBG] osd.0 reported immediately failed by osd.6 2021-02-10T04:58:11.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:58:11.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: :58:11.271700+0000 mon.a (mon.0) 2021-02-10T04:58:11.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 2897 : cluster [DBG] osd.0 reported immediately failed by osd.3 2021-02-10T04:58:11.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:58:11.903 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: -02-10T04:58:11 2021-02-10T04:58:11.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: .271804+0000 mon.a (mon. 2021-02-10T04:58:11.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 0) 2898 : cluster [DBG] 2021-02-10T04:58:11.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: osd.0 reported immediately failed by osd.7 2021-02-10T04:58:11.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:58:11.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: -10T04:58:11.271896+0000 2021-02-10T04:58:11.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: mon.a (mon.0) 2899 2021-02-10T04:58:11.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: : cluster [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:58:11.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02-10T04:58 2021-02-10T04:58:11.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: :11.271987+0000 mon.a (mon.0) 2900 : 2021-02-10T04:58:11.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster [DBG] osd.0 reported immediately failed by osd.5 2021-02-10T04:58:11.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02-10T04:58 2021-02-10T04:58:11.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: :11.272086+0000 mon.a (mon.0) 2901 : cluster 2021-02-10T04:58:11.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: [DBG] osd.0 reported immediately failed by osd.6 2021-02-10T04:58:11.904 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:58:11.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: :58:11.272176+0000 2021-02-10T04:58:11.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: mon.a (mon.0) 2902 : cluster 2021-02-10T04:58:11.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: [DBG] osd.0 reported immediately failed by osd.7 2021-02-10T04:58:11.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02-10T04:58:11.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:58:11.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 58:11.272270+0000 mon.a (mon 2021-02-10T04:58:11.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: .0) 2903 : cluster [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:58:11.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:58:11.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: -02-10T04:58:11.272362 2021-02-10T04:58:11.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2904 : cluster [DBG] 2021-02-10T04:58:11.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: osd.0 reported immediately failed by osd.5 2021-02-10T04:58:11.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02-10T04:58:11.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 2021-02-10T04:58:11.272457+0000 mon.a (mon 2021-02-10T04:58:11.905 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: .0) 2905 : cluster [DBG] osd.0 reported immediately failed by osd.4 2021-02-10T04:58:11.906 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:58:11.906 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 04:58:11.272553+0000 mon.a (mon. 2021-02-10T04:58:11.906 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: 0) 2906 : cluster [DBG] osd.0 reported immediately failed by osd.6 2021-02-10T04:58:11.906 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:58:11.906 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: T04:58:11.487112+0000 mon.a (mon 2021-02-10T04:58:11.906 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:11 smithi164 conmon[29288]: .0) 2907 : audit [INF] from='client.? 172.21.15.85:0/206669841' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:58:12.353 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:58:12.353 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2021-02-10T04:58:12.353 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 0 2021-02-10T04:58:12.353 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:58:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:12 smithi164 conmon[29288]: cluster 2021-02-10T04:58:11.613865+0000 mon.a (mon.0) 2908 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:58:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:12 smithi164 conmon[29288]: cluster 2021-02-10T04:58:11.613909+0000 mon.a (mon.0) 2909 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2021-02-10T04:58:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:12 smithi164 conmon[29288]: audit 2021-02-10T04:58:11.623613+0000 mon.a (mon.0) 2910 : audit [INF] from='client.? 172.21.15.85:0/206669841' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:58:12.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:12 smithi164 conmon[29288]: cluster 2021-02-10T04:58:11.623662+0000 mon.a (mon.0) 2911 : cluster [DBG] osdmap e1316: 8 total, 7 up, 8 in 2021-02-10T04:58:12.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:12 smithi085 conmon[29319]: cluster 2021-02-10T04:58:11.613865+0000 mon.a (mon. 2021-02-10T04:58:12.691 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:12 smithi085 conmon[29319]: 0) 2908 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:58:12.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:12 smithi085 conmon[29319]: cluster 2021-02-10T04:58:11.613909+0000 mon.a (mon.0) 2909 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2021-02-10T04:58:12.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:12 smithi085 conmon[29319]: audit 2021-02-10T04:58:11.623613+0000 mon.a (mon.0) 2910 : audit [INF] from='client.? 172.21.15.85:0/206669841' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:58:12.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:12 smithi085 conmon[29319]: cluster 2021-02-10T04:58:11.623662+0000 mon.a (mon.0) 2911 : cluster [DBG] osdmap e1316: 8 total, 7 up, 8 in 2021-02-10T04:58:12.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:12 smithi085 conmon[35961]: cluster 2021-02-10T04:58:11.613865+0000 2021-02-10T04:58:12.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:12 smithi085 conmon[35961]: mon.a (mon.0) 2908 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:58:12.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:12 smithi085 conmon[35961]: cluster 2021-02-10T04:58:11.613909+0000 mon.a (mon.0) 2909 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2021-02-10T04:58:12.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:12 smithi085 conmon[35961]: audit 2021-02-10T04:58:11.623613+0000 mon.a (mon.0) 2910 : audit [INF] from='client.? 172.21.15.85:0/206669841' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:58:12.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:12 smithi085 conmon[35961]: cluster 2021-02-10T04:58:11.623662+0000 mon.a (mon.0) 2911 : cluster [DBG] osdmap e1316: 8 total, 7 up, 8 in 2021-02-10T04:58:13.878 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:58:13.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:13 smithi164 conmon[29288]: cluster 2021-02-10T04:58:12.544208+0000 mgr.y (mgr.14140) 1476 2021-02-10T04:58:13.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:13 smithi164 conmon[29288]: : cluster [DBG] pgmap v2403: 198 pgs: 1 active+clean+scrubbing, 19 peering, 17 stale+active+clean, 161 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 669 B/s rd, 0 op/s 2021-02-10T04:58:13.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:13 smithi164 conmon[29288]: cluster 2021-02-10T04:58:12.620890+0000 mon.a (mon.0) 2912 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2021-02-10T04:58:13.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:13 smithi164 conmon[29288]: cluster 2021-02-10T04:58:12.630888+0000 mon.a (mon.0) 2913 : cluster [DBG] osdmap e1317: 8 total, 7 up, 8 in 2021-02-10T04:58:13.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:13 smithi164 conmon[29288]: cluster 2021-02-10T04:58:12.852147+0000 osd.5 (osd.5) 196 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:58:13.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:13 smithi164 conmon[29288]: cluster 2021-02-10T04:58:12.854400+0000 osd.5 (osd.5) 197 : cluster 2021-02-10T04:58:13.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:13 smithi164 conmon[29288]: [DBG] purged_snaps scrub ok 2021-02-10T04:58:13.900 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:58:13.917 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:58:13.935 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:58:13.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:13 smithi085 conmon[29319]: cluster 2021-02-10T04:58:12.544208+0000 mgr.y (mgr. 2021-02-10T04:58:13.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:13 smithi085 conmon[29319]: 14140) 1476 : cluster [DBG] pgmap v2403: 198 pgs: 1 active+clean+scrubbing, 19 peering, 17 stale+active+clean, 161 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 669 B/s rd, 0 op/s 2021-02-10T04:58:13.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:13 smithi085 conmon[29319]: cluster 2021-02-10T04:58:12.620890+0000 mon.a (mon.0) 2912 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2021-02-10T04:58:13.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:13 smithi085 conmon[29319]: cluster 2021-02-10T04:58:12.630888+0000 mon.a (mon.0) 2913 : cluster [DBG] osdmap e1317: 8 total, 7 up, 8 in 2021-02-10T04:58:13.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:13 smithi085 conmon[29319]: cluster 2021-02-10T04:58:12.852147+0000 osd.5 (osd.5) 196 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:58:13.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:13 smithi085 conmon[29319]: cluster 2021-02-10T04:58:12.854400+0000 osd.5 (osd.5) 197 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:58:13.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:13 smithi085 conmon[35961]: cluster 2021-02-10T04:58:12.544208+0000 mgr.y ( 2021-02-10T04:58:13.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:13 smithi085 conmon[35961]: mgr.14140) 1476 : cluster [DBG] pgmap v2403: 198 pgs: 1 active+clean+scrubbing, 19 peering, 17 stale+active+clean, 161 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 669 B/s rd, 0 op/s 2021-02-10T04:58:13.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:13 smithi085 conmon[35961]: cluster 2021-02-10T04:58:12.620890+0000 mon.a (mon.0 2021-02-10T04:58:13.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:13 smithi085 conmon[35961]: ) 2912 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2021-02-10T04:58:13.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:13 smithi085 conmon[35961]: cluster 2021-02-10T04:58:12 2021-02-10T04:58:13.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:13 smithi085 conmon[35961]: .630888+0000 mon.a (mon.0) 2913 : cluster [DBG] osdmap e1317: 8 total, 7 up, 8 in 2021-02-10T04:58:13.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:13 smithi085 conmon[35961]: cluster 2021-02-10T04:58:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:13 smithi085 conmon[35961]: 2021-02-10T04:58:12.852147+0000 osd.5 (osd.5) 196 : cluster [DBG] 2021-02-10T04:58:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:13 smithi085 conmon[35961]: purged_snaps scrub starts 2021-02-10T04:58:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:13 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:58:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:13 smithi085 conmon[35961]: 04:58:12.854400+0000 osd.5 (osd.5) 197 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:58:13.960 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:58:13.983 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:58:14.008 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:58:14.009 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-02-10T04:58:14.009 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: problem getting command descriptions from osd.0 2021-02-10T04:58:14.009 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:58:14.009 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:58:14.009 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:58:14.009 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:58:14.009 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:58:14.009 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:58:14.009 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:58:14.639 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set nodeep-scrub 2021-02-10T04:58:14.892 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:58:14.893 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:14 smithi164 conmon[29288]: cluster 2021-02-10T04:58:13.632815 2021-02-10T04:58:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:14 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2914 : cluster [DBG] osdmap e1318: 8 total, 7 up, 8 in 2021-02-10T04:58:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:14 smithi164 conmon[29288]: audit 2021-02-10T04:58:13.858841+0000 mon.a (mon.0) 2915 2021-02-10T04:58:14.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:14 smithi164 conmon[29288]: : audit [DBG] from='client.? 172.21.15.85:0/2885481405' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:58:14.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:14 smithi085 conmon[35961]: cluster 2021-02-10T04:58:13.632815+0000 mon.a (mon.0) 2914 : cluster [DBG] osdmap e1318: 8 total, 7 up, 8 in 2021-02-10T04:58:14.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:14 smithi085 conmon[35961]: audit 2021-02-10T04:58:13.858841+0000 mon.a (mon.0) 2915 : audit [DBG] from='client.? 172.21.15.85:0/2885481405' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:58:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:14 smithi085 conmon[29319]: cluster 2021-02-10T04:58:13.632815+0000 mon.a (mon.0) 2914 : cluster [DBG] osdmap e1318: 8 total, 7 up, 8 in 2021-02-10T04:58:14.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:14 smithi085 conmon[29319]: audit 2021-02-10T04:58:13.858841+0000 mon.a (mon.0) 2915 : audit [DBG] from='client.? 172.21.15.85:0/2885481405' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:58:14.980 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:58:14.980 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:58:15.819 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.4 dump_blocked_ops 2021-02-10T04:58:15.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:15 smithi164 conmon[29288]: cluster 2021-02-10T04:58:14.544733+0000 mgr.y (mgr.14140) 1477 : 2021-02-10T04:58:15.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:15 smithi164 conmon[29288]: cluster [DBG] pgmap v2406: 198 pgs: 12 active+undersized+degraded, 15 active+undersized, 1 active+clean+scrubbing, 19 peering, 8 stale+active+clean, 143 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 517 B/s rd, 0 op/s; 132/2244 objects degraded (5.882%) 2021-02-10T04:58:15.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:15 smithi164 conmon[29288]: cluster 2021-02-10T04:58:14.629389+0000 mon.a (mon.0) 2916 : cluster [WRN] Health check failed: Degraded data redundancy: 132/2244 objects degraded (5.882%), 12 pgs degraded (PG_DEGRADED) 2021-02-10T04:58:15.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:15 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:58:15.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:15 smithi085 conmon[29319]: -10T04:58:14.544733+0000 mgr.y (mgr.14140) 1477 : cluster [DBG] pgmap v2406: 198 pgs: 12 active+undersized+degraded, 15 active+undersized, 1 active+clean+scrubbing, 19 peering, 8 stale+active+clean, 143 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 517 B/s rd, 0 op/s; 132/2244 objects degraded (5.882%) 2021-02-10T04:58:15.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:15 smithi085 conmon[29319]: cluster 2021-02-10T04:58:14.629389+0000 mon.a (mon.0) 2916 : cluster [WRN] Health check failed: Degraded data redundancy: 132/2244 objects degraded (5.882%), 12 pgs degraded (PG_DEGRADED) 2021-02-10T04:58:15.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:15 smithi085 conmon[35961]: cluster 2021-02-10T04:58: 2021-02-10T04:58:15.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:15 smithi085 conmon[35961]: 14.544733+0000 mgr.y (mgr.14140) 1477 : cluster [DBG] pgmap v2406: 198 pgs: 12 active+undersized+degraded, 15 active+undersized, 1 active+clean+scrubbing, 19 peering, 8 stale+active+clean, 143 active+clean; 927 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 517 B/s rd, 0 op/s; 132/2244 objects degraded (5.882%) 2021-02-10T04:58:15.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:15 smithi085 conmon[35961]: cluster 2021-02-10T04:58:14.629389+0000 mon.a (mon.0) 2916 : cluster [WRN] Health check failed: Degraded data redundancy: 132/2244 objects degraded (5.882%), 12 pgs degraded (PG_DEGRADED) 2021-02-10T04:58:16.100 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:58:16.982 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:58:17.238 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:58:17.639 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:17 smithi085 conmon[35961]: cluster 2021-02-10T04:58:16 2021-02-10T04:58:17.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:17 smithi085 conmon[35961]: .545394+0000 mgr.y (mgr.14140) 1478 : cluster [DBG] pgmap v2407: 198 pgs: 20 active+undersized+degraded, 29 active+undersized, 1 active+clean+scrubbing, 19 peering, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 4.5 MiB/s rd, 1.3 MiB/s wr, 12 op/s; 214/2196 objects degraded (9.745%); 0 B/s, 3 objects/s recovering 2021-02-10T04:58:17.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:17 smithi085 conmon[35961]: audit 2021-02-10T04:58:17.180898+0000 mon.a (mon.0) 2917 : audit [INF] from='client.? 172.21.15.85:0/3628481267' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:58:17.651 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is set 2021-02-10T04:58:17.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:17 smithi164 conmon[29288]: cluster 2021-02-10T04:58:16.545394+0000 2021-02-10T04:58:17.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:17 smithi164 conmon[29288]: mgr.y (mgr.14140) 1478 : cluster [DBG] pgmap v2407: 198 pgs: 20 active+undersized+degraded, 29 active+undersized, 1 active+clean+scrubbing, 19 peering, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 4.5 MiB/s rd, 1.3 MiB/s wr, 12 op/s; 214/2196 objects degraded (9.745%); 0 B/s, 3 objects/s recovering 2021-02-10T04:58:17.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:17 smithi164 conmon[29288]: audit 2021-02-10T04:58:17.180898+0000 mon.a (mon.0) 2917 : audit [INF] from='client.? 172.21.15.85:0/3628481267' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:58:17.922 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:17 smithi085 conmon[29319]: cluster 2021-02-10T04:58 2021-02-10T04:58:17.922 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:17 smithi085 conmon[29319]: :16.545394+0000 mgr.y (mgr.14140) 1478 : cluster [DBG] pgmap v2407: 198 pgs: 20 active+undersized+degraded, 29 active+undersized, 1 active+clean+scrubbing, 19 peering, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 4.5 MiB/s rd, 1.3 MiB/s wr, 12 op/s; 214/2196 objects degraded (9.745%); 0 B/s, 3 objects/s recovering 2021-02-10T04:58:17.923 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:17 smithi085 conmon[29319]: audit 2021-02-10T04:58:17.180898+0000 mon.a (mon.0) 2917 : audit [INF] from='client.? 172.21.15.85:0/3628481267' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:58:18.480 DEBUG:teuthology.orchestra.run.smithi085:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.0.service' 2021-02-10T04:58:18.565 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-02-10T04:58:18.566 INFO:tasks.cephadm.osd.0:Stopped osd.0 2021-02-10T04:58:18.566 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.0 2021-02-10T04:58:18.566 INFO:tasks.thrashosds.thrasher:remote for osd 0 is ubuntu@smithi085.front.sepia.ceph.com 2021-02-10T04:58:18.566 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -n osd.0 --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op list-pgs 2021-02-10T04:58:18.691 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:18 smithi085 conmon[35961]: cluster 2021-02-10T04:58:17.637818+0000 mon.a (mon.0) 2918 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:58:18.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:18 smithi085 conmon[35961]: audit 2021-02-10T04:58:17.645450+0000 mon.a (mon.0) 2919 : audit [INF] from='client.? 172.21.15.85:0/3628481267' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:58:18.692 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:18 smithi085 conmon[35961]: cluster 2021-02-10T04:58:17.645592+0000 mon.a (mon.0) 2920 : cluster [DBG] osdmap e1319: 8 total, 7 up, 8 in 2021-02-10T04:58:18.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:18 smithi085 conmon[29319]: cluster 2021-02-10T04:58:17.637818+0000 mon.a (mon.0) 2918 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:58:18.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:18 smithi085 conmon[29319]: audit 2021-02-10T04:58:17.645450+0000 mon.a (mon.0) 2919 : audit [INF] from='client.? 172.21.15.85:0/3628481267' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:58:18.692 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:18 smithi085 conmon[29319]: cluster 2021-02-10T04:58:17.645592+0000 mon.a (mon.0) 2920 : cluster [DBG] osdmap e1319: 8 total, 7 up, 8 in 2021-02-10T04:58:18.840 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/osd.0/config 2021-02-10T04:58:18.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:18 smithi164 conmon[29288]: cluster 2021-02-10T04:58:17.637818+0000 mon.a (mon.0) 2918 2021-02-10T04:58:18.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:18 smithi164 conmon[29288]: : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:58:18.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:18 smithi164 conmon[29288]: audit 2021-02-10T04:58:17.645450+0000 mon.a (mon.0) 2919 : audit [INF] from='client.? 172.21.15.85:0/3628481267' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:58:18.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:18 smithi164 conmon[29288]: cluster 2021-02-10T04:58:17.645592+0000 mon.a (mon.0) 2920 : cluster [DBG] osdmap e1319: 8 total, 7 up, 8 in 2021-02-10T04:58:19.894 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:58:19 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:58:19.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:19 smithi164 conmon[29288]: cluster 2021-02-10T04:58:18.546089+0000 mgr.y ( 2021-02-10T04:58:19.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:19 smithi164 conmon[29288]: mgr.14140) 1479 : cluster [DBG] pgmap v2409: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 4.5 MiB/s rd, 1.3 MiB/s wr, 13 op/s; 258/2196 objects degraded (11.749%); 0 B/s, 3 objects/s recovering 2021-02-10T04:58:19.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:19 smithi164 conmon[29288]: cluster 2021-02-10T04:58:18.643264+0000 mon.a (mon.0) 2921 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 6 pgs peering) 2021-02-10T04:58:19.910 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:19 smithi085 conmon[29319]: cluster 2021-02-10T04:58:18.546089+0000 2021-02-10T04:58:19.910 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:19 smithi085 conmon[29319]: mgr.y (mgr.14140) 1479 : cluster [DBG] pgmap v2409: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 4.5 MiB/s rd, 1.3 MiB/s wr, 13 op/s; 258/2196 objects degraded (11.749%); 0 B/s, 3 objects/s recovering 2021-02-10T04:58:19.910 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:19 smithi085 conmon[29319]: cluster 2021-02-10T04:58:18.643264+0000 mon.a (mon.0) 2921 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 6 pgs peering) 2021-02-10T04:58:19.910 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:19 smithi085 conmon[35961]: cluster 2021-02-10T04:58:18.546089+0000 2021-02-10T04:58:19.910 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:19 smithi085 conmon[35961]: mgr.y (mgr.14140) 1479 : cluster [DBG] pgmap v2409: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 4.5 MiB/s rd, 1.3 MiB/s wr, 13 op/s; 258/2196 objects degraded (11.749%); 0 B/s, 3 objects/s recovering 2021-02-10T04:58:19.910 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:19 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:58:19.910 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:19 smithi085 conmon[35961]: 04:58:18.643264+0000 mon.a (mon.0) 2921 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 6 pgs peering) 2021-02-10T04:58:20.445 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset noscrub 2021-02-10T04:58:20.667 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:58:20.810 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:20 smithi164 conmon[29288]: audit 2021-02-10T04:58:19.486004+0000 2021-02-10T04:58:20.811 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:20 smithi164 conmon[29288]: mgr.y (mgr.14140) 1480 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:58:20.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:20 smithi085 conmon[29319]: audit 2021-02-10T04:58:19.486004+0000 mgr.y (mgr. 2021-02-10T04:58:20.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:20 smithi085 conmon[29319]: 14140) 1480 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:58:20.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:20 smithi085 conmon[35961]: audit 2021-02-10T04:58:19.486004+0000 mgr.y (mgr.14140) 1480 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:58:21.322 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:58:21.322 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:58:21.685 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:58:21.702 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:58:21.719 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:58:21.737 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:58:21.756 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:58:21.774 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:58:21.789 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:21 smithi085 conmon[35961]: cluster 2021-02-10T04:58:20.038907+0000 osd.2 (osd.2) 2037 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:58:21.789 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:21 smithi085 conmon[35961]: cluster 2021-02-10T04:58:20.041755+0000 osd.2 (osd.2) 2038 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:58:21.789 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:21 smithi085 conmon[35961]: cluster 2021-02-10T04:58:20.546716+0000 mgr.y (mgr.14140) 1481 : cluster [DBG] pgmap v2410: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 1.0 MiB/s wr, 10 op/s; 258/2196 objects degraded (11.749%); 0 B/s, 2 objects/s recovering 2021-02-10T04:58:21.790 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:21 smithi085 conmon[35961]: cluster 2021-02-10T04:58:20.649640+0000 mon.a (mon.0) 2922 : cluster [DBG] osdmap e1320: 8 total, 7 up, 8 in 2021-02-10T04:58:21.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:21 smithi085 conmon[29319]: cluster 2021-02-10T04:58:20.038907+0000 osd.2 (osd.2) 2037 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:58:21.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:21 smithi085 conmon[29319]: cluster 2021-02-10T04:58:20.041755+0000 osd.2 (osd.2) 2038 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:58:21.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:21 smithi085 conmon[29319]: cluster 2021-02-10T04:58:20.546716+0000 mgr.y (mgr.14140) 1481 : cluster [DBG] pgmap v2410: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 1.0 MiB/s wr, 10 op/s; 258/2196 objects degraded (11.749%); 0 B/s, 2 objects/s recovering 2021-02-10T04:58:21.790 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:21 smithi085 conmon[29319]: cluster 2021-02-10T04:58:20.649640+0000 mon.a (mon.0) 2922 : cluster [DBG] osdmap e1320: 8 total, 7 up, 8 in 2021-02-10T04:58:21.794 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:58:21.794 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-02-10T04:58:21.794 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: problem getting command descriptions from osd.0 2021-02-10T04:58:21.794 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:58:21.794 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:58:21.794 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:58:21.794 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:58:21.794 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:58:21.794 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:58:21.794 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:58:22.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:21 smithi164 conmon[29288]: cluster 2021-02-10T04:58:20.038907+0000 osd.2 (osd.2) 2037 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:58:22.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:21 smithi164 conmon[29288]: cluster 2021-02-10T04:58:20.041755+0000 osd.2 (osd.2) 2038 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:58:22.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:21 smithi164 conmon[29288]: cluster 2021-02-10T04:58:20.546716+0000 mgr.y (mgr.14140) 1481 : cluster [DBG] pgmap v2410: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 1.0 MiB/s wr, 10 op/s; 258/2196 objects degraded (11.749%); 0 B/s, 2 objects/s recovering 2021-02-10T04:58:22.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:21 smithi164 conmon[29288]: cluster 2021-02-10T04:58:20.649640+0000 mon.a (mon.0) 2922 : cluster [DBG] osdmap e1320: 8 total, 7 up, 8 in 2021-02-10T04:58:22.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:22 smithi085 conmon[29319]: audit 2021-02-10T04:58:21.666300+0000 2021-02-10T04:58:22.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:22 smithi085 conmon[29319]: mon.c (mon.1) 149 : audit [DBG] from='client.? 172.21.15.85:0/2964452948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:58:22.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:22 smithi085 conmon[35961]: audit 2021-02-10T04:58:21.666300+0000 mon.c (mon 2021-02-10T04:58:22.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:22 smithi085 conmon[35961]: .1) 149 : audit [DBG] from='client.? 172.21.15.85:0/2964452948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:58:23.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:22 smithi164 conmon[29288]: audit 2021-02-10T04:58:21.666300+0000 mon.c (mon.1) 149 : audit [DBG] from='client.? 172.21.15.85:0/2964452948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:58:23.149 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:58:23.149 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:58:23.502 INFO:teuthology.orchestra.run.smithi085.stdout:7.f 2021-02-10T04:58:23.503 INFO:teuthology.orchestra.run.smithi085.stdout:7.e 2021-02-10T04:58:23.503 INFO:teuthology.orchestra.run.smithi085.stdout:7.d 2021-02-10T04:58:23.503 INFO:teuthology.orchestra.run.smithi085.stdout:7.5 2021-02-10T04:58:23.503 INFO:teuthology.orchestra.run.smithi085.stdout:7.1f 2021-02-10T04:58:23.503 INFO:teuthology.orchestra.run.smithi085.stdout:7.1d 2021-02-10T04:58:23.503 INFO:teuthology.orchestra.run.smithi085.stdout:7.18 2021-02-10T04:58:23.504 INFO:teuthology.orchestra.run.smithi085.stdout:7.1e 2021-02-10T04:58:23.504 INFO:teuthology.orchestra.run.smithi085.stdout:7.16 2021-02-10T04:58:23.504 INFO:teuthology.orchestra.run.smithi085.stdout:7.13 2021-02-10T04:58:23.504 INFO:teuthology.orchestra.run.smithi085.stdout:7.4 2021-02-10T04:58:23.504 INFO:teuthology.orchestra.run.smithi085.stdout:6.9 2021-02-10T04:58:23.504 INFO:teuthology.orchestra.run.smithi085.stdout:6.26 2021-02-10T04:58:23.505 INFO:teuthology.orchestra.run.smithi085.stdout:6.18 2021-02-10T04:58:23.505 INFO:teuthology.orchestra.run.smithi085.stdout:6.16 2021-02-10T04:58:23.505 INFO:teuthology.orchestra.run.smithi085.stdout:6.13 2021-02-10T04:58:23.505 INFO:teuthology.orchestra.run.smithi085.stdout:6.11 2021-02-10T04:58:23.505 INFO:teuthology.orchestra.run.smithi085.stdout:7.7 2021-02-10T04:58:23.505 INFO:teuthology.orchestra.run.smithi085.stdout:6.10 2021-02-10T04:58:23.506 INFO:teuthology.orchestra.run.smithi085.stdout:7.9 2021-02-10T04:58:23.506 INFO:teuthology.orchestra.run.smithi085.stdout:6.27 2021-02-10T04:58:23.506 INFO:teuthology.orchestra.run.smithi085.stdout:5.e 2021-02-10T04:58:23.506 INFO:teuthology.orchestra.run.smithi085.stdout:5.c 2021-02-10T04:58:23.506 INFO:teuthology.orchestra.run.smithi085.stdout:7.6 2021-02-10T04:58:23.506 INFO:teuthology.orchestra.run.smithi085.stdout:5.7 2021-02-10T04:58:23.507 INFO:teuthology.orchestra.run.smithi085.stdout:2.4 2021-02-10T04:58:23.507 INFO:teuthology.orchestra.run.smithi085.stdout:2.21 2021-02-10T04:58:23.507 INFO:teuthology.orchestra.run.smithi085.stdout:7.12 2021-02-10T04:58:23.507 INFO:teuthology.orchestra.run.smithi085.stdout:2.1d 2021-02-10T04:58:23.507 INFO:teuthology.orchestra.run.smithi085.stdout:2.10 2021-02-10T04:58:23.507 INFO:teuthology.orchestra.run.smithi085.stdout:6.d 2021-02-10T04:58:23.508 INFO:teuthology.orchestra.run.smithi085.stdout:2.15 2021-02-10T04:58:23.508 INFO:teuthology.orchestra.run.smithi085.stdout:2.1 2021-02-10T04:58:23.508 INFO:teuthology.orchestra.run.smithi085.stdout:6.29 2021-02-10T04:58:23.508 INFO:teuthology.orchestra.run.smithi085.stdout:2.1f 2021-02-10T04:58:23.508 INFO:teuthology.orchestra.run.smithi085.stdout:1.1 2021-02-10T04:58:23.508 INFO:teuthology.orchestra.run.smithi085.stdout:2.1e 2021-02-10T04:58:23.509 INFO:teuthology.orchestra.run.smithi085.stdout:1.0 2021-02-10T04:58:23.509 INFO:teuthology.orchestra.run.smithi085.stdout:2.23 2021-02-10T04:58:23.509 INFO:teuthology.orchestra.run.smithi085.stdout:7.1b 2021-02-10T04:58:23.509 INFO:teuthology.orchestra.run.smithi085.stdout:2.13 2021-02-10T04:58:23.509 INFO:teuthology.orchestra.run.smithi085.stdout:2.0 2021-02-10T04:58:23.510 INFO:teuthology.orchestra.run.smithi085.stdout:5.b 2021-02-10T04:58:23.510 INFO:teuthology.orchestra.run.smithi085.stdout:5.5 2021-02-10T04:58:23.510 INFO:teuthology.orchestra.run.smithi085.stdout:2.8 2021-02-10T04:58:23.510 INFO:teuthology.orchestra.run.smithi085.stdout:2.c 2021-02-10T04:58:23.510 INFO:teuthology.orchestra.run.smithi085.stdout:4.7 2021-02-10T04:58:23.510 INFO:teuthology.orchestra.run.smithi085.stdout:1.6 2021-02-10T04:58:23.511 INFO:teuthology.orchestra.run.smithi085.stdout:2.f 2021-02-10T04:58:23.511 INFO:teuthology.orchestra.run.smithi085.stdout:5.8 2021-02-10T04:58:23.511 INFO:teuthology.orchestra.run.smithi085.stdout:3.1 2021-02-10T04:58:23.511 INFO:teuthology.orchestra.run.smithi085.stdout:3.3 2021-02-10T04:58:23.512 INFO:teuthology.orchestra.run.smithi085.stdout:3.6 2021-02-10T04:58:23.512 INFO:teuthology.orchestra.run.smithi085.stdout:2.5 2021-02-10T04:58:23.512 INFO:teuthology.orchestra.run.smithi085.stdout:3.b 2021-02-10T04:58:23.512 INFO:teuthology.orchestra.run.smithi085.stdout:2.19 2021-02-10T04:58:23.512 INFO:teuthology.orchestra.run.smithi085.stdout:3.7 2021-02-10T04:58:23.512 INFO:teuthology.orchestra.run.smithi085.stdout:1.2 2021-02-10T04:58:23.513 INFO:teuthology.orchestra.run.smithi085.stdout:4.0 2021-02-10T04:58:23.513 INFO:teuthology.orchestra.run.smithi085.stdout:4.3 2021-02-10T04:58:23.513 INFO:teuthology.orchestra.run.smithi085.stdout:4.6 2021-02-10T04:58:23.513 INFO:teuthology.orchestra.run.smithi085.stdout:4.b 2021-02-10T04:58:23.513 INFO:teuthology.orchestra.run.smithi085.stdout:6.0 2021-02-10T04:58:23.514 INFO:teuthology.orchestra.run.smithi085.stdout:5.13 2021-02-10T04:58:23.514 INFO:teuthology.orchestra.run.smithi085.stdout:5.1e 2021-02-10T04:58:23.514 INFO:teuthology.orchestra.run.smithi085.stdout:5.15 2021-02-10T04:58:23.514 INFO:teuthology.orchestra.run.smithi085.stdout:5.3 2021-02-10T04:58:23.514 INFO:teuthology.orchestra.run.smithi085.stdout:6.a 2021-02-10T04:58:23.514 INFO:teuthology.orchestra.run.smithi085.stdout:5.1d 2021-02-10T04:58:23.515 INFO:teuthology.orchestra.run.smithi085.stdout:5.2 2021-02-10T04:58:23.657 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:23 smithi085 conmon[29319]: cluster 2021-02-10T04:58:22.547312+0000 2021-02-10T04:58:23.658 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:23 smithi085 conmon[29319]: mgr.y (mgr.14140) 1482 : cluster [DBG] pgmap v2412: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 1.0 MiB/s wr, 9 op/s; 258/2196 objects degraded (11.749%); 0 B/s, 2 objects/s recovering 2021-02-10T04:58:23.658 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:23 smithi085 conmon[29319]: cluster 2021-02-10T04:58:23.609842+0000 mon.a (mon.0) 2923 : cluster [WRN] Health check update: Degraded data redundancy: 258/2196 objects degraded (11.749%), 26 pgs degraded (PG_DEGRADED) 2021-02-10T04:58:23.658 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:23 smithi085 conmon[29319]: cluster 2021-02-10T04:58:23.615642+0000 mon.a (mon.0) 2924 : cluster [DBG] osdmap e1321: 8 total, 7 up, 8 in 2021-02-10T04:58:23.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:23 smithi085 conmon[35961]: cluster 2021-02-10T04:58:22.547312+0000 2021-02-10T04:58:23.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:23 smithi085 conmon[35961]: mgr.y (mgr.14140) 1482 : cluster [DBG] pgmap v2412: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 1.0 MiB/s wr, 9 op/s; 258/2196 objects degraded (11.749%); 0 B/s, 2 objects/s recovering 2021-02-10T04:58:23.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:23 smithi085 conmon[35961]: cluster 2021-02-10T04:58:23.609842 2021-02-10T04:58:23.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:23 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2923 : cluster [WRN] Health check update: Degraded data redundancy: 258/2196 objects degraded (11.749%), 26 pgs degraded (PG_DEGRADED) 2021-02-10T04:58:23.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:23 smithi085 conmon[35961]: cluster 2021-02-10T04:58:23.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:23 smithi085 conmon[35961]: 2021-02-10T04:58:23.615642+0000 2021-02-10T04:58:23.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:23 smithi085 conmon[35961]: mon.a (mon.0) 2924 : cluster [DBG] osdmap e1321: 8 total, 7 up, 8 in 2021-02-10T04:58:23.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:23 smithi085 conmon[35961]: 2021-02-10T04:58:24.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:23 smithi164 conmon[29288]: cluster 2021-02-10T04:58:22.547312+0000 mgr.y 2021-02-10T04:58:24.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:23 smithi164 conmon[29288]: (mgr.14140) 1482 : cluster [DBG] pgmap v2412: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.4 MiB/s rd, 1.0 MiB/s wr, 9 op/s; 258/2196 objects degraded (11.749%); 0 B/s, 2 objects/s recovering 2021-02-10T04:58:24.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:23 smithi164 conmon[29288]: cluster 2021-02-10T04:58:23.609842+0000 mon.a (mon.0 2021-02-10T04:58:24.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:23 smithi164 conmon[29288]: ) 2923 : cluster [WRN] Health check update: Degraded data redundancy: 258/2196 objects degraded (11.749%), 26 pgs degraded (PG_DEGRADED) 2021-02-10T04:58:24.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:23 smithi164 conmon[29288]: cluster 2021-02-10T04:58: 2021-02-10T04:58:24.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:23 smithi164 conmon[29288]: 23.615642+0000 mon.a (mon.0) 2924 : cluster [DBG] osdmap e1321: 8 total, 7 up, 8 in 2021-02-10T04:58:24.684 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is unset 2021-02-10T04:58:24.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:24 smithi085 conmon[35961]: audit 2021-02-10T04:58:23.771789+0000 2021-02-10T04:58:24.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:24 smithi085 conmon[35961]: mon.c (mon.1) 150 : audit [INF] from='client.? 172.21.15.85:0/4125888381' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:58:24.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:24 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:58:24.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:24 smithi085 conmon[35961]: 10T04:58:23.772098+0000 mon.a (mon.0) 2925 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:58:24.939 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:24 smithi085 conmon[29319]: audit 2021-02-10T04:58:23.771789+0000 mon.c ( 2021-02-10T04:58:24.939 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:24 smithi085 conmon[29319]: mon.1) 150 : audit [INF] from='client.? 172.21.15.85:0/4125888381' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:58:24.939 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:24 smithi085 conmon[29319]: audit 2021-02-10T04:58:23.772098+0000 mon.a (mon.0) 2925 : audit [INF] 2021-02-10T04:58:24.939 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:24 smithi085 conmon[29319]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:58:25.099 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:24 smithi164 conmon[29288]: audit 2021-02-10T04:58:23.771789+0000 mon.c (mon.1) 150 : 2021-02-10T04:58:25.100 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:24 smithi164 conmon[29288]: audit [INF] from='client.? 172.21.15.85:0/4125888381' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:58:25.100 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:24 smithi164 conmon[29288]: audit 2021-02-10T04:58:23.772098+0000 mon.a (mon.0) 2925 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:58:25.118 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.4 dump_historic_ops 2021-02-10T04:58:25.150 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:58:25.245 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -n osd.0 --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op export --pgid 7.13 --file /var/log/ceph/exp.7.13.0 2021-02-10T04:58:25.343 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:58:25.402 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:58:25.475 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/osd.0/config 2021-02-10T04:58:25.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:25 smithi164 conmon[29288]: cluster 2021-02-10T04:58:24.547843 2021-02-10T04:58:25.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:25 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1483 : cluster [DBG] pgmap v2414: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:25.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:25 smithi164 conmon[29288]: cluster 2021-02-10T04:58:24.656255+0000 mon.a (mon.0) 2926 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:58:25.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:25 smithi164 conmon[29288]: audit 2021-02-10T04:58:24.668251+0000 mon.a (mon.0) 2927 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:58:25.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:25 smithi164 conmon[29288]: cluster 2021-02-10T04:58:24.668335+0000 mon.a (mon.0) 2928 : cluster [DBG] osdmap e1322: 8 total, 7 up, 8 in 2021-02-10T04:58:25.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:25 smithi085 conmon[29319]: cluster 2021-02-10T04:58:24.547843+0000 mgr.y (mgr.14140) 1483 : cluster [DBG] pgmap v2414: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:25.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:25 smithi085 conmon[29319]: cluster 2021-02-10T04:58:24.656255+0000 mon.a (mon.0) 2926 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:58:25.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:25 smithi085 conmon[29319]: audit 2021-02-10T04:58:24.668251+0000 mon.a (mon.0) 2927 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:58:25.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:25 smithi085 conmon[29319]: cluster 2021-02-10T04:58:24.668335+0000 mon.a (mon.0) 2928 : cluster [DBG] osdmap e1322: 8 total, 7 up, 8 in 2021-02-10T04:58:25.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:25 smithi085 conmon[35961]: cluster 2021-02-10T04:58:24.547843+0000 mgr.y (mgr.14140) 1483 : cluster [DBG] pgmap v2414: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:25.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:25 smithi085 conmon[35961]: cluster 2021-02-10T04:58:24.656255+0000 mon.a (mon.0) 2926 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:58:25.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:25 smithi085 conmon[35961]: audit 2021-02-10T04:58:24.668251+0000 mon.a (mon.0) 2927 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:58:25.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:25 smithi085 conmon[35961]: cluster 2021-02-10T04:58:24.668335+0000 mon.a (mon.0) 2928 : cluster [DBG] osdmap e1322: 8 total, 7 up, 8 in 2021-02-10T04:58:27.456 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset nodeep-scrub 2021-02-10T04:58:27.668 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:27 smithi085 conmon[29319]: cluster 2021-02-10T04:58: 2021-02-10T04:58:27.668 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:27 smithi085 conmon[29319]: 26.548394+0000 mgr.y (mgr.14140) 1484 : cluster [DBG] pgmap v2416: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:27.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:27 smithi085 conmon[35961]: cluster 2021-02-10T04:58:27.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:27 smithi085 conmon[35961]: 2021-02-10T04:58:27.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:27 smithi085 conmon[35961]: 2021 2021-02-10T04:58:27.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:27 smithi085 conmon[35961]: - 2021-02-10T04:58:27.668 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:27 smithi085 conmon[35961]: 02-10T04:58:26. 2021-02-10T04:58:27.669 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:27 smithi085 conmon[35961]: 548394+0000 mgr.y (mgr.14140) 1484 : cluster [DBG] pgmap v2416: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:27.715 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:58:28.051 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:58:28.070 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:58:28.087 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:58:28.109 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:58:28.133 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:58:28.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:27 smithi164 conmon[29288]: cluster 2021-02-10T04:58:26.548394 2021-02-10T04:58:28.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:27 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1484 : cluster [DBG] pgmap v2416: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:28.157 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:58:28.183 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:58:28.183 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-02-10T04:58:28.183 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: problem getting command descriptions from osd.0 2021-02-10T04:58:28.183 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:58:28.183 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:58:28.183 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:58:28.183 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:58:28.183 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:58:28.183 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:58:28.184 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:58:28.535 INFO:teuthology.orchestra.run.smithi085.stderr:Exporting 7.13 info 7.13( v 1002'1657 (0'0,1002'1657] local-lis/les=1227/1228 n=20 ec=1007/93 lis/c=1227/1227 les/c/f=1228/1228/0 sis=1227 pruub=0.397587001s) 2021-02-10T04:58:28.541 INFO:teuthology.orchestra.run.smithi085.stderr:Read #7:cd29c741:::smithi08575397-46:2a2# 2021-02-10T04:58:28.545 INFO:teuthology.orchestra.run.smithi085.stderr:Read #7:cd29c741:::smithi08575397-46:2b6# 2021-02-10T04:58:28.551 INFO:teuthology.orchestra.run.smithi085.stderr:Read #7:cd29c741:::smithi08575397-46:304# 2021-02-10T04:58:28.554 INFO:teuthology.orchestra.run.smithi085.stderr:Read #7:cd29c741:::smithi08575397-46:309# 2021-02-10T04:58:28.559 INFO:teuthology.orchestra.run.smithi085.stderr:Read #7:cd29c741:::smithi08575397-46:33e# 2021-02-10T04:58:28.564 INFO:teuthology.orchestra.run.smithi085.stderr:Read #7:cd29c741:::smithi08575397-46:343# 2021-02-10T04:58:28.567 INFO:teuthology.orchestra.run.smithi085.stderr:Read #7:cd29c741:::smithi08575397-46:364# 2021-02-10T04:58:28.572 INFO:teuthology.orchestra.run.smithi085.stderr:Read #7:cd29c741:::smithi08575397-46:head# 2021-02-10T04:58:28.575 INFO:teuthology.orchestra.run.smithi085.stderr:Read #7:cf4c8c3b:::smithi08575397-6:203# 2021-02-10T04:58:28.580 INFO:teuthology.orchestra.run.smithi085.stderr:Read #7:cf4c8c3b:::smithi08575397-6:27c# 2021-02-10T04:58:28.584 INFO:teuthology.orchestra.run.smithi085.stderr:Read #7:cf4c8c3b:::smithi08575397-6:2bf# 2021-02-10T04:58:28.588 INFO:teuthology.orchestra.run.smithi085.stderr:Read #7:cf4c8c3b:::smithi08575397-6:2cb# 2021-02-10T04:58:28.592 INFO:teuthology.orchestra.run.smithi085.stderr:Read #7:cf4c8c3b:::smithi08575397-6:2f1# 2021-02-10T04:58:28.597 INFO:teuthology.orchestra.run.smithi085.stderr:Read #7:cf4c8c3b:::smithi08575397-6:308# 2021-02-10T04:58:28.601 INFO:teuthology.orchestra.run.smithi085.stderr:Read #7:cf4c8c3b:::smithi08575397-6:32c# 2021-02-10T04:58:28.607 INFO:teuthology.orchestra.run.smithi085.stderr:Read #7:cf4c8c3b:::smithi08575397-6:335# 2021-02-10T04:58:28.609 INFO:teuthology.orchestra.run.smithi085.stderr:Read #7:cf4c8c3b:::smithi08575397-6:351# 2021-02-10T04:58:28.614 INFO:teuthology.orchestra.run.smithi085.stderr:Read #7:cf4c8c3b:::smithi08575397-6:364# 2021-02-10T04:58:28.620 INFO:teuthology.orchestra.run.smithi085.stderr:Read #7:cf4c8c3b:::smithi08575397-6:372# 2021-02-10T04:58:28.632 INFO:teuthology.orchestra.run.smithi085.stderr:Read #7:cf4c8c3b:::smithi08575397-6:head# 2021-02-10T04:58:28.632 INFO:teuthology.orchestra.run.smithi085.stderr:Export successful 2021-02-10T04:58:29.044 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:28 smithi085 conmon[29319]: audit 2021-02-10T04:58:28.026741+0000 mon.a (mon.0) 2929 : audit [DBG] from='client.? 172.21.15.85:0/605171713' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:58:29.044 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:28 smithi085 conmon[29319]: cluster 2021-02-10T04:58:28.621853+0000 mon.a (mon.0) 2930 : cluster [DBG] osdmap e1323: 8 total, 7 up, 8 in 2021-02-10T04:58:29.044 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:28 smithi085 conmon[35961]: audit 2021-02-10T04:58:28.026741+0000 mon.a (mon.0) 2929 : audit [DBG] from='client.? 172.21.15.85:0/605171713' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:58:29.044 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:28 smithi085 conmon[35961]: cluster 2021-02-10T04:58:28.621853+0000 mon.a (mon.0) 2930 : cluster [DBG] osdmap e1323: 8 total, 7 up, 8 in 2021-02-10T04:58:29.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:28 smithi164 conmon[29288]: audit 2021-02-10T04:58:28.026741+0000 mon.a (mon.0) 2929 : audit 2021-02-10T04:58:29.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:28 smithi164 conmon[29288]: [DBG] from='client.? 172.21.15.85:0/605171713' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:58:29.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:28 smithi164 conmon[29288]: cluster 2021-02-10T04:58:28.621853+0000 mon.a (mon.0) 2930 : cluster [DBG] osdmap e1323: 8 total, 7 up, 8 in 2021-02-10T04:58:29.644 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:58:29 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:58:30.127 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:29 smithi085 conmon[35961]: cluster 2021-02-10T04:58:28.549135+0000 2021-02-10T04:58:30.128 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:29 smithi085 conmon[35961]: mgr.y (mgr.14140) 1485 : cluster [DBG] pgmap v2417: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:30.128 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:29 smithi085 conmon[29319]: cluster 2021-02-10T04:58:28.549135+0000 mgr.y (mgr.14140) 1485 : cluster 2021-02-10T04:58:30.128 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:29 smithi085 conmon[29319]: [DBG] pgmap v2417: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:30.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:29 smithi164 conmon[29288]: cluster 2021-02-10T04:58:28 2021-02-10T04:58:30.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:29 smithi164 conmon[29288]: .549135+0000 mgr.y (mgr.14140) 1485 : cluster [DBG] pgmap v2417: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:30.157 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:58:30.157 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:58:30.636 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -n osd.0 --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --force --op remove --pgid 7.13 2021-02-10T04:58:30.767 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:58:30.767 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:58:30.884 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/osd.0/config 2021-02-10T04:58:30.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:30 smithi085 conmon[29319]: audit 2021-02-10T04:58: 2021-02-10T04:58:30.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:30 smithi085 conmon[29319]: 29.492663+0000 mgr.y (mgr.14140) 1486 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:58:30.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:30 smithi085 conmon[35961]: audit 2021-02-10T04:58:29. 2021-02-10T04:58:30.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:30 smithi085 conmon[35961]: 492663+0000 mgr.y (mgr.14140) 1486 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:58:31.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:30 smithi164 conmon[29288]: audit 2021-02-10T04:58:29.492663+0000 2021-02-10T04:58:31.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:30 smithi164 conmon[29288]: mgr.y (mgr.14140) 1486 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:58:31.749 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is unset 2021-02-10T04:58:31.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:31 smithi085 conmon[29319]: cluster 2021-02-10T04:58:30.549746+0000 mgr.y (mgr.14140) 1487 : cluster [DBG] pgmap v2419: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 738 B/s rd, 0 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:31.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:31 smithi085 conmon[29319]: audit 2021-02-10T04:58:31.719188+0000 mon.a (mon.0) 2931 : audit [INF] from='client.? 172.21.15.85:0/3086509122' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:58:31.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:31 smithi085 conmon[35961]: cluster 2021-02-10T04:58:30.549746+0000 mgr.y (mgr.14140) 1487 : cluster [DBG] pgmap v2419: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 738 B/s rd, 0 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:31.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:31 smithi085 conmon[35961]: audit 2021-02-10T04:58:31.719188+0000 mon.a (mon.0) 2931 : audit [INF] from='client.? 172.21.15.85:0/3086509122' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:58:32.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:31 smithi164 conmon[29288]: cluster 2021-02-10T04:58:30.549746+0000 mgr.y (mgr.14140 2021-02-10T04:58:32.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:31 smithi164 conmon[29288]: ) 1487 : cluster [DBG] pgmap v2419: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 738 B/s rd, 0 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:32.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:31 smithi164 conmon[29288]: audit 2021-02-10T04:58:31.719188+0000 mon.a (mon.0) 2931 : audit [INF] from='client.? 172.21.15.85:0/3086509122' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:58:32.767 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:58:32.802 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:32 smithi085 conmon[35961]: cluster 2021-02-10T04:58:31.238731 2021-02-10T04:58:32.802 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:32 smithi085 conmon[35961]: +0000 osd.6 (osd.6) 192 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:58:32.802 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:32 smithi085 conmon[35961]: cluster 2021-02-10T04:58:31.242529+0000 2021-02-10T04:58:32.802 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:32 smithi085 conmon[35961]: osd.6 (osd.6) 193 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:58:32.802 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:32 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:58:32.802 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:32 smithi085 conmon[35961]: -10T04:58:31.734251+0000 mon.a (mon 2021-02-10T04:58:32.803 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:32 smithi085 conmon[35961]: .0) 2932 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:58:32.803 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:32 smithi085 conmon[35961]: audit 2021 2021-02-10T04:58:32.803 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:32 smithi085 conmon[35961]: -02-10T04:58:31.741443+0000 2021-02-10T04:58:32.803 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:32 smithi085 conmon[35961]: mon.a (mon.0) 2933 : audit [INF] 2021-02-10T04:58:32.803 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:32 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/3086509122' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:58:32.803 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:32 smithi085 conmon[35961]: cluster 2021-02-10T04:58 2021-02-10T04:58:32.803 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:32 smithi085 conmon[35961]: :31.741670+0000 mon.a (mon.0) 2934 : 2021-02-10T04:58:32.803 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:32 smithi085 conmon[35961]: cluster [DBG] osdmap e1324: 8 total, 7 up, 8 in 2021-02-10T04:58:32.803 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:32 smithi085 conmon[29319]: cluster 2021-02-10T04:58:31.238731+0000 osd.6 (osd 2021-02-10T04:58:32.803 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:32 smithi085 conmon[29319]: .6) 192 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:58:32.803 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:32 smithi085 conmon[29319]: cluster 2021-02-10T04:58:31.242529+0000 osd.6 (osd.6) 193 : cluster [DBG] 2021-02-10T04:58:32.803 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:32 smithi085 conmon[29319]: purged_snaps scrub ok 2021-02-10T04:58:32.804 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:32 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:58:32.804 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:32 smithi085 conmon[29319]: -10T04:58:31.734251+0000 mon.a (mon.0) 2932 2021-02-10T04:58:32.804 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:32 smithi085 conmon[29319]: : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:58:32.804 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:32 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:58:32.804 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:32 smithi085 conmon[29319]: -10T04:58:31.741443+0000 mon.a 2021-02-10T04:58:32.804 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:32 smithi085 conmon[29319]: (mon.0) 2933 : audit [INF] from='client.? 172.21.15.85:0/3086509122' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:58:32.804 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:32 smithi085 conmon[29319]: 2021-02-10T04:58:32.804 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:32 smithi085 conmon[29319]: cluster 2021-02-10T04:58 2021-02-10T04:58:32.804 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:32 smithi085 conmon[29319]: :31.741670+0000 mon.a (mon.0) 2021-02-10T04:58:32.804 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:32 smithi085 conmon[29319]: 2934 : cluster [DBG] osdmap e1324: 8 total, 7 up, 8 in 2021-02-10T04:58:33.008 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:58:33.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:32 smithi164 conmon[29288]: cluster 2021-02-10T04:58:31.238731+0000 osd.6 (osd.6 2021-02-10T04:58:33.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:32 smithi164 conmon[29288]: ) 192 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:58:33.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:32 smithi164 conmon[29288]: cluster 2021-02-10T04:58:31.242529+0000 osd.6 (osd.6) 193 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:58:33.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:32 smithi164 conmon[29288]: cluster 2021-02-10T04:58:31.734251+0000 mon.a (mon.0) 2932 : 2021-02-10T04:58:33.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:32 smithi164 conmon[29288]: cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:58:33.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:32 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:58:33.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:32 smithi164 conmon[29288]: T04:58:31.741443+0000 mon.a (mon 2021-02-10T04:58:33.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:32 smithi164 conmon[29288]: .0) 2933 : audit [INF] from='client.? 172.21.15.85:0/3086509122' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:58:33.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:32 smithi164 conmon[29288]: cluster 2021-02-10T04:58:33.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:32 smithi164 conmon[29288]: 2021-02-10T04:58:31.741670 2021-02-10T04:58:33.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:32 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2934 : cluster [DBG] 2021-02-10T04:58:33.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:32 smithi164 conmon[29288]: osdmap e1324: 8 total, 7 up, 8 in 2021-02-10T04:58:33.682 INFO:teuthology.orchestra.run.smithi085.stdout: marking collection for removal 2021-02-10T04:58:33.683 INFO:teuthology.orchestra.run.smithi085.stdout:setting '_remove' omap key 2021-02-10T04:58:33.705 INFO:teuthology.orchestra.run.smithi085.stdout:finish_remove_pgs 7.13_head removing 7.13 2021-02-10T04:58:33.714 INFO:teuthology.orchestra.run.smithi085.stdout:Remove successful 2021-02-10T04:58:33.779 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.5 dump_ops_in_flight 2021-02-10T04:58:34.032 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:58:34.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:33 smithi164 conmon[29288]: cluster 2021-02-10T04:58:32.550380 2021-02-10T04:58:34.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:33 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1488 : cluster [DBG] pgmap v2421: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 649 B/s rd, 0 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:34.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:33 smithi164 conmon[29288]: cluster 2021-02-10T04:58:33.618134+0000 mon.a (mon.0) 2935 : cluster [DBG] osdmap e1325: 8 total, 7 up, 8 in 2021-02-10T04:58:34.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:33 smithi085 conmon[29319]: cluster 2021-02-10T04:58:32.550380+0000 mgr.y (mgr.14140) 1488 : cluster [DBG] pgmap v2421: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 649 B/s rd, 0 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:34.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:33 smithi085 conmon[29319]: cluster 2021-02-10T04:58:33.618134+0000 mon.a (mon.0) 2935 : cluster [DBG] osdmap e1325: 8 total, 7 up, 8 in 2021-02-10T04:58:34.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:33 smithi085 conmon[35961]: cluster 2021-02-10T04:58:32.550380+0000 mgr.y ( 2021-02-10T04:58:34.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:33 smithi085 conmon[35961]: mgr.14140) 1488 : cluster [DBG] pgmap v2421: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 649 B/s rd, 0 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:34.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:33 smithi085 conmon[35961]: cluster 2021-02-10T04:58:33.618134+0000 2021-02-10T04:58:34.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:33 smithi085 conmon[35961]: mon.a (mon.0) 2935 : cluster [DBG] osdmap e1325: 8 total, 7 up, 8 in 2021-02-10T04:58:35.017 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set noscrub 2021-02-10T04:58:35.225 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell -n osd.0 --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.7.13.0 2021-02-10T04:58:35.262 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:58:35.466 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/osd.0/config 2021-02-10T04:58:35.664 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:58:35.686 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:58:35.707 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:58:35.729 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:58:35.751 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:58:35.770 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:58:35.793 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:58:35.793 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-02-10T04:58:35.793 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: problem getting command descriptions from osd.0 2021-02-10T04:58:35.793 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:58:35.793 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:58:35.794 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:58:35.794 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:58:35.794 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:58:35.794 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:58:35.794 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:58:35.954 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:35 smithi085 conmon[29319]: cluster 2021-02-10T04:58:34.550890+0000 mgr.y (mgr.14140) 1489 : cluster [DBG] pgmap v2423: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:35.954 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:35 smithi085 conmon[29319]: audit 2021-02-10T04:58:35.636659+0000 mon.a (mon.0) 2936 : audit [DBG] from='client.? 172.21.15.85:0/854148966' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:58:35.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:35 smithi085 conmon[35961]: cluster 2021-02-10T04:58:34.550890 2021-02-10T04:58:35.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:35 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1489 : cluster [DBG] pgmap v2423: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:35.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:35 smithi085 conmon[35961]: audit 2021-02-10T04:58:35.636659 2021-02-10T04:58:35.955 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:35 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2936 : audit [DBG] from='client.? 172.21.15.85:0/854148966' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:58:36.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:35 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:58:36.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:35 smithi164 conmon[29288]: 58:34.550890+0000 mgr.y (mgr.14140) 1489 : cluster [DBG] pgmap v2423: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:36.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:35 smithi164 conmon[29288]: audit 2021-02-10T04:58:35.636659+0000 mon.a (mon.0) 2936 : audit [DBG] from='client.? 172.21.15.85:0/854148966' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:58:36.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:58:36 smithi164 conmon[37263]: debug 2021-02-10T04:58:36.233+0000 7f1c6b9eb700 -1 osd.5 pg_epoch: 1325 pg[1.a( empty local-lis/les=1118/1119 n=0 ec=110/19 lis/c=1118/1118 les/c/f=1119/1119/0 sis=1118) [5,2,6] r=0 lpr=1118 crt=0'0 mlcod 0'0 active+clean] scrubber pg(1.a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 2 (0x5566532d54a0) 2021-02-10T04:58:37.345 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:58:37.345 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:58:38.120 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:37 smithi085 conmon[29319]: cluster 2021-02-10T04:58:36.551637+0000 2021-02-10T04:58:38.121 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:37 smithi085 conmon[29319]: mgr.y (mgr.14140) 1490 : cluster [DBG] pgmap v2424: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 645 B/s rd, 0 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:38.121 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:37 smithi085 conmon[35961]: cluster 2021-02-10T04:58:36 2021-02-10T04:58:38.121 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:37 smithi085 conmon[35961]: .551637+0000 mgr.y (mgr.14140) 1490 : cluster [DBG] pgmap v2424: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 645 B/s rd, 0 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:38.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:37 smithi164 conmon[29288]: cluster 2021-02-10T04:58:36.551637+0000 mgr.y (mgr.14140) 1490 2021-02-10T04:58:38.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:37 smithi164 conmon[29288]: : cluster [DBG] pgmap v2424: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 645 B/s rd, 0 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:38.749 INFO:teuthology.orchestra.run.smithi085.stdout:get_pg_num_history pg_num_history pg_num_history(e1315 pg_nums {1={110=11},2={89=42},3={55=3,1209=13},4={55=32,1172=31,1176=30,1180=29,1184=28,1232=27,1236=26,1244=25,1248=24,1252=23,1256=22},5={57=32},6={1044=42},7={1007=26,1141=36}} deleted_pools ) 2021-02-10T04:58:38.749 INFO:teuthology.orchestra.run.smithi085.stdout:Importing pgid 7.13 2021-02-10T04:58:38.770 INFO:teuthology.orchestra.run.smithi085.stdout:Write #7:cd29c741:::smithi08575397-46:2a2# 2021-02-10T04:58:38.774 INFO:teuthology.orchestra.run.smithi085.stdout:Write #7:cd29c741:::smithi08575397-46:2b6# 2021-02-10T04:58:38.780 INFO:teuthology.orchestra.run.smithi085.stdout:Write #7:cd29c741:::smithi08575397-46:304# 2021-02-10T04:58:38.782 INFO:teuthology.orchestra.run.smithi085.stdout:Write #7:cd29c741:::smithi08575397-46:309# 2021-02-10T04:58:38.787 INFO:teuthology.orchestra.run.smithi085.stdout:Write #7:cd29c741:::smithi08575397-46:33e# 2021-02-10T04:58:38.790 INFO:teuthology.orchestra.run.smithi085.stdout:Write #7:cd29c741:::smithi08575397-46:343# 2021-02-10T04:58:38.793 INFO:teuthology.orchestra.run.smithi085.stdout:Write #7:cd29c741:::smithi08575397-46:364# 2021-02-10T04:58:38.798 INFO:teuthology.orchestra.run.smithi085.stdout:Write #7:cd29c741:::smithi08575397-46:head# 2021-02-10T04:58:38.798 INFO:teuthology.orchestra.run.smithi085.stdout:snapset 369=[]:{2a2=[27c],2b6=[2ae],304=[300,2f5,2f0,2ee,2ed,2eb,2ea,2d9,2c8],309=[308],33e=[33e,33c],343=[343],364=[35f,354,353,34e,34d,34b,349,345,344]} 2021-02-10T04:58:38.807 INFO:teuthology.orchestra.run.smithi085.stdout:Write #7:cf4c8c3b:::smithi08575397-6:203# 2021-02-10T04:58:38.812 INFO:teuthology.orchestra.run.smithi085.stdout:Write #7:cf4c8c3b:::smithi08575397-6:27c# 2021-02-10T04:58:38.814 INFO:teuthology.orchestra.run.smithi085.stdout:Write #7:cf4c8c3b:::smithi08575397-6:2bf# 2021-02-10T04:58:38.817 INFO:teuthology.orchestra.run.smithi085.stdout:Write #7:cf4c8c3b:::smithi08575397-6:2cb# 2021-02-10T04:58:38.820 INFO:teuthology.orchestra.run.smithi085.stdout:Write #7:cf4c8c3b:::smithi08575397-6:2f1# 2021-02-10T04:58:38.822 INFO:teuthology.orchestra.run.smithi085.stdout:Write #7:cf4c8c3b:::smithi08575397-6:308# 2021-02-10T04:58:38.827 INFO:teuthology.orchestra.run.smithi085.stdout:Write #7:cf4c8c3b:::smithi08575397-6:32c# 2021-02-10T04:58:38.832 INFO:teuthology.orchestra.run.smithi085.stdout:Write #7:cf4c8c3b:::smithi08575397-6:335# 2021-02-10T04:58:38.837 INFO:teuthology.orchestra.run.smithi085.stdout:Write #7:cf4c8c3b:::smithi08575397-6:351# 2021-02-10T04:58:38.840 INFO:teuthology.orchestra.run.smithi085.stdout:Write #7:cf4c8c3b:::smithi08575397-6:364# 2021-02-10T04:58:38.842 INFO:teuthology.orchestra.run.smithi085.stdout:Write #7:cf4c8c3b:::smithi08575397-6:372# 2021-02-10T04:58:38.845 INFO:teuthology.orchestra.run.smithi085.stdout:Write #7:cf4c8c3b:::smithi08575397-6:head# 2021-02-10T04:58:38.845 INFO:teuthology.orchestra.run.smithi085.stdout:snapset 372=[]:{203=[203],27c=[27c],2bf=[2ae],2cb=[2c8],2f1=[2f0,2ee,2ed,2eb,2ea],308=[308,300,2f5],32c=[32a,328,31a],335=[335,332,32f],351=[34e,34d,34b,349,345,344,343,33e,33c,33a],364=[35f,354,353],372=[372,370,36f,36b,368,366]} 2021-02-10T04:58:38.846 INFO:teuthology.orchestra.run.smithi085.stdout:write_pg epoch 1315 info 7.13( v 1002'1657 (0'0,1002'1657] local-lis/les=1227/1228 n=20 ec=1007/93 lis/c=1227/1227 les/c/f=1228/1228/0 sis=1227 pruub=0.397587001s) 2021-02-10T04:58:38.848 INFO:teuthology.orchestra.run.smithi085.stdout:Import successful 2021-02-10T04:58:39.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:38 smithi164 conmon[29288]: audit 2021-02-10T04:58:38.262426+0000 mon.a (mon.0) 2937 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:58:39.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:38 smithi164 conmon[29288]: audit 2021-02-10T04:58:38.262639+0000 mon.a (mon.0) 2938 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:58:39.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:38 smithi085 conmon[29319]: audit 2021-02-10T04:58:38.262426+0000 2021-02-10T04:58:39.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:38 smithi085 conmon[29319]: mon.a (mon.0) 2937 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:58:39.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:38 smithi085 conmon[29319]: audit 2021-02-10T04:58:38.262639+0000 mon.a (mon.0) 2938 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:58:39.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:38 smithi085 conmon[35961]: audit 2021-02-10T04:58:38.262426+0000 2021-02-10T04:58:39.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:38 smithi085 conmon[35961]: mon.a (mon.0) 2937 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:58:39.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:38 smithi085 conmon[35961]: audit 2021-02-10T04:58:38.262639+0000 mon.a 2021-02-10T04:58:39.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:38 smithi085 conmon[35961]: (mon.0) 2938 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:58:39.281 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:58:39.281 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2021-02-10T04:58:39.281 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 0 2021-02-10T04:58:39.281 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:58:39.347 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:58:39.565 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:58:39.644 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:58:39 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:58:39.777 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is set 2021-02-10T04:58:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:39 smithi164 conmon[29288]: cluster 2021-02-10T04:58:38.552377+0000 2021-02-10T04:58:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:39 smithi164 conmon[29288]: mgr.y (mgr.14140) 1491 : cluster [DBG] pgmap v2425: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:39 smithi164 conmon[29288]: audit 2021-02-10T04:58:38.789353+0000 mon.c (mon.1) 151 : audit [INF] from='client.? 172.21.15.85:0/3424359915' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:58:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:39 smithi164 conmon[29288]: audit 2021-02-10T04:58:38.789764+0000 mon.a (mon.0 2021-02-10T04:58:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:39 smithi164 conmon[29288]: ) 2939 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:58:40.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:39 smithi085 conmon[29319]: cluster 2021-02-10T04:58:38.552377+0000 mgr.y (mgr.14140) 1491 : cluster [DBG] pgmap v2425: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:40.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:39 smithi085 conmon[29319]: audit 2021-02-10T04:58:38.789353+0000 mon.c (mon.1) 151 : audit [INF] from='client.? 172.21.15.85:0/3424359915' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:58:40.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:39 smithi085 conmon[29319]: audit 2021-02-10T04:58:38.789764+0000 mon.a (mon.0) 2939 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:58:40.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:39 smithi085 conmon[35961]: cluster 2021-02-10T04:58:38.552377+0000 mgr.y (mgr.14140) 1491 : cluster [DBG] pgmap v2425: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:40.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:39 smithi085 conmon[35961]: audit 2021-02-10T04:58:38.789353+0000 mon.c (mon.1) 151 : audit [INF] from='client.? 172.21.15.85:0/3424359915' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:58:40.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:39 smithi085 conmon[35961]: audit 2021-02-10T04:58:38.789764+0000 mon.a (mon.0) 2939 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:58:40.760 DEBUG:teuthology.orchestra.run.smithi085:> sudo rm -f /var/log/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/exp.7.13.0 2021-02-10T04:58:41.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:40 smithi085 conmon[35961]: audit 2021-02-10T04:58:41.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:40 smithi085 conmon[35961]: 2021-02-10T04:58 2021-02-10T04:58:41.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:40 smithi085 conmon[35961]: :39.503025+0000 mgr.y (mgr.14140) 1492 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:58:41.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:40 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:58:41.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:40 smithi085 conmon[35961]: 10T04:58:39.752143+0000 mon.a (mon. 2021-02-10T04:58:41.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:40 smithi085 conmon[35961]: 0) 2940 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:58:41.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:40 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:58:41.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:40 smithi085 conmon[35961]: T04:58:39.763453+0000 mon.a (mon.0 2021-02-10T04:58:41.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:40 smithi085 conmon[35961]: ) 2941 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:58:41.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:40 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:58:41.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:40 smithi085 conmon[35961]: 58:39.763537+0000 mon.a (mon.0) 2021-02-10T04:58:41.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:40 smithi085 conmon[35961]: 2942 : cluster [DBG] osdmap e1326: 8 total, 7 up, 8 in 2021-02-10T04:58:41.026 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:40 smithi085 conmon[29319]: audit 2021-02-10T04:58:39.503025 2021-02-10T04:58:41.027 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:40 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 1492 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:58:41.027 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:40 smithi085 conmon[29319]: cluster 2021-02-10T04:58:39.752143+0000 mon.a (mon.0) 2940 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:58:41.027 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:40 smithi085 conmon[29319]: audit 2021-02-10T04:58:39.763453+0000 mon.a (mon.0) 2941 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:58:41.027 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:40 smithi085 conmon[29319]: cluster 2021-02-10T04:58:39.763537 2021-02-10T04:58:41.027 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:40 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2942 : cluster [DBG] osdmap e1326: 8 total, 7 up, 8 in 2021-02-10T04:58:41.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:40 smithi164 conmon[29288]: audit 2021-02-10T04:58:39.503025+0000 2021-02-10T04:58:41.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:40 smithi164 conmon[29288]: mgr.y (mgr.14140) 1492 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:58:41.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:40 smithi164 conmon[29288]: cluster 2021-02-10T04:58:39.752143+0000 mon.a (mon.0) 2940 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:58:41.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:40 smithi164 conmon[29288]: audit 2021-02-10T04:58:39.763453+0000 mon.a (mon.0) 2941 : audit [INF] 2021-02-10T04:58:41.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:40 smithi164 conmon[29288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:58:41.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:40 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:58:41.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:40 smithi164 conmon[29288]: 58:39.763537+0000 mon.a (mon.0) 2942 : cluster [DBG] osdmap e1326: 8 total, 7 up, 8 in 2021-02-10T04:58:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:41 smithi164 conmon[29288]: cluster 2021-02-10T04:58:40.552964+0000 2021-02-10T04:58:42.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:41 smithi164 conmon[29288]: mgr.y (mgr.14140) 1493 : cluster [DBG] pgmap v2427: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:42.173 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:41 smithi085 conmon[35961]: cluster 2021-02-10T04:58:40.552964 2021-02-10T04:58:42.173 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:41 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1493 : cluster [DBG] pgmap v2427: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:42.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:41 smithi085 conmon[29319]: cluster 2021-02-10T04:58:40.552964+0000 mgr.y (mgr.14140) 1493 : cluster 2021-02-10T04:58:42.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:41 smithi085 conmon[29319]: [DBG] pgmap v2427: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:42.647 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:58:42.665 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:58:42.682 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:58:42.704 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:58:42.727 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:58:42.745 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:58:42.763 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:58:42.763 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-02-10T04:58:42.764 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: problem getting command descriptions from osd.0 2021-02-10T04:58:42.764 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:58:42.764 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:58:42.764 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:58:42.764 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:58:42.764 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:58:42.764 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:58:42.764 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:58:43.099 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set nodeep-scrub 2021-02-10T04:58:43.123 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:42 smithi085 conmon[29319]: audit 2021-02-10T04:58:42.628267+0000 mon.c 2021-02-10T04:58:43.124 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:42 smithi085 conmon[29319]: (mon.1) 152 : audit [DBG] from='client.? 172.21.15.85:0/2847827962' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:58:43.124 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:42 smithi085 conmon[35961]: audit 2021-02-10T04:58:42.628267+0000 2021-02-10T04:58:43.124 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:42 smithi085 conmon[35961]: mon.c (mon.1) 152 : audit [DBG] from='client.? 172.21.15.85:0/2847827962' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:58:43.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:42 smithi164 conmon[29288]: audit 2021-02-10T04:58:42.628267 2021-02-10T04:58:43.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:42 smithi164 conmon[29288]: +0000 mon.c (mon.1) 152 : audit [DBG] from='client.? 172.21.15.85:0/2847827962' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:58:43.314 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:58:43.516 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:58:43.516 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:58:44.144 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:43 smithi085 conmon[29319]: cluster 2021-02-10T04:58:42.553611+0000 mgr.y (mgr.14140) 1494 : cluster [DBG] pgmap v2428: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 572 B/s rd, 0 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:44.144 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:43 smithi085 conmon[29319]: cluster 2021-02-10T04:58:43.622225+0000 mon.a (mon.0) 2943 : cluster [DBG] osdmap e1327: 8 total, 7 up, 8 in 2021-02-10T04:58:44.144 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:43 smithi085 conmon[35961]: cluster 2021-02-10T04:58:42. 2021-02-10T04:58:44.144 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:43 smithi085 conmon[35961]: 553611+0000 mgr.y (mgr.14140) 1494 : cluster [DBG] pgmap v2428: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 572 B/s rd, 0 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:44.144 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:43 smithi085 conmon[35961]: cluster 2021-02-10T04:58:44.144 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:43 smithi085 conmon[35961]: 2021-02-10T04:58:43.622225+0000 mon.a (mon.0) 2943 : cluster [DBG] osdmap e1327: 8 total, 7 up, 8 in 2021-02-10T04:58:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:43 smithi164 conmon[29288]: cluster 2021-02-10T04:58:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:43 smithi164 conmon[29288]: 2021-02-10T04:58:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:43 smithi164 conmon[29288]: 2021 2021-02-10T04:58:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:43 smithi164 conmon[29288]: - 2021-02-10T04:58:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:43 smithi164 conmon[29288]: 02 2021-02-10T04:58:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:43 smithi164 conmon[29288]: - 2021-02-10T04:58:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:43 smithi164 conmon[29288]: 10 2021-02-10T04:58:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:43 smithi164 conmon[29288]: T 2021-02-10T04:58:44.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:43 smithi164 conmon[29288]: 04 2021-02-10T04:58:44.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:43 smithi164 conmon[29288]: : 2021-02-10T04:58:44.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:43 smithi164 conmon[29288]: 58 2021-02-10T04:58:44.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:43 smithi164 conmon[29288]: : 2021-02-10T04:58:44.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:43 smithi164 conmon[29288]: 42 2021-02-10T04:58:44.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:43 smithi164 conmon[29288]: . 2021-02-10T04:58:44.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:43 smithi164 conmon[29288]: 553611 2021-02-10T04:58:44.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:43 smithi164 conmon[29288]: +0000 2021-02-10T04:58:44.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:43 smithi164 conmon[29288]: 2021-02-10T04:58:44.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:43 smithi164 conmon[29288]: mgr.y 2021-02-10T04:58:44.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:43 smithi164 conmon[29288]: ( 2021-02-10T04:58:44.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:43 smithi164 conmon[29288]: mgr 2021-02-10T04:58:44.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:43 smithi164 conmon[29288]: . 2021-02-10T04:58:44.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:43 smithi164 conmon[29288]: 14140) 1494 : cluster [DBG] pgmap v2428: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 572 B/s rd, 0 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:44.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:43 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:58:44.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:43 smithi164 conmon[29288]: 02-10T04:58:43.622225+0000 mon.a ( 2021-02-10T04:58:44.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:43 smithi164 conmon[29288]: mon.0) 2943 : cluster [DBG] osdmap e1327: 8 total, 7 up, 8 in 2021-02-10T04:58:44.323 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.5 dump_blocked_ops 2021-02-10T04:58:44.676 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:58:45.518 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:58:45.734 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:58:45.792 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2021-02-10T04:58:45.792 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:58:45.793 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2021-02-10T04:58:45.793 INFO:tasks.cephadm.osd.0:Restarting osd.0 (starting--it wasn't running)... 2021-02-10T04:58:45.793 DEBUG:teuthology.orchestra.run.smithi085:osd.0> sudo journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.0.service 2021-02-10T04:58:45.794 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is set 2021-02-10T04:58:45.796 DEBUG:teuthology.orchestra.run.smithi085:> sudo systemctl start ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.0 2021-02-10T04:58:45.815 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:45 smithi085 conmon[29319]: cluster 2021-02-10T04:58:44.554272+0000 mgr.y (mgr.14140 2021-02-10T04:58:45.815 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:45 smithi085 conmon[29319]: ) 1495 : cluster [DBG] pgmap v2430: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:45.815 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:45 smithi085 conmon[29319]: audit 2021-02-10T04:58:45.344563+0000 mon.a (mon.0) 2944 : audit [INF] from='client.? 172.21.15.85:0/3366354685' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:58:45.815 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:45 smithi085 conmon[35961]: cluster 2021-02-10T04:58:44.554272 2021-02-10T04:58:45.816 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:45 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1495 : cluster [DBG] pgmap v2430: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:45.816 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:45 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:58:45.816 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:45 smithi085 conmon[35961]: -10T04:58:45.344563+0000 mon.a (mon.0) 2944 : audit 2021-02-10T04:58:45.816 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:45 smithi085 conmon[35961]: [INF] from='client.? 172.21.15.85:0/3366354685' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:58:45.822 INFO:journalctl@ceph.osd.0.smithi085.stdout:-- Logs begin at Wed 2021-02-10 04:09:23 UTC. -- 2021-02-10T04:58:46.102 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:45 smithi085 systemd[1]: Starting Ceph osd.0 for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T04:58:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:45 smithi164 conmon[29288]: cluster 2021-02-10T04:58:44.554272+0000 2021-02-10T04:58:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:45 smithi164 conmon[29288]: mgr.y (mgr.14140) 1495 : cluster [DBG] pgmap v2430: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:45 smithi164 conmon[29288]: audit 2021-02-10T04:58:45.344563+0000 mon.a (mon.0) 2944 : audit [INF] from='client.? 172.21.15.85:0/3366354685' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:58:46.910 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:46 smithi085 conmon[29319]: cluster 2021-02-10T04:58:45.772317+0000 mon.a (mon.0) 2945 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:58:46.911 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:46 smithi085 conmon[29319]: audit 2021-02-10T04:58:45.786711+0000 mon.a (mon.0) 2946 : audit [INF] from='client.? 172.21.15.85:0/3366354685' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:58:46.911 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:46 smithi085 conmon[29319]: cluster 2021-02-10T04:58:46.911 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:46 smithi085 conmon[29319]: 2021-02-10T04:58:45.786858+0000 mon.a (mon.0) 2947 : cluster [DBG] osdmap e1328: 8 total, 7 up, 8 in 2021-02-10T04:58:46.911 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:46 smithi085 conmon[35961]: cluster 2021-02-10T04:58:45.772317+0000 mon.a (mon 2021-02-10T04:58:46.911 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:46 smithi085 conmon[35961]: .0) 2945 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:58:46.911 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:46 smithi085 conmon[35961]: audit 2021-02-10T04:58:45.786711+0000 mon.a (mon.0) 2946 : audit [INF] from='client.? 172.21.15.85:0/3366354685' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:58:46.911 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:46 smithi085 conmon[35961]: cluster 2021-02-10T04:58:45.786858+0000 mon.a (mon.0) 2947 : cluster [DBG] osdmap e1328: 8 total, 7 up, 8 in 2021-02-10T04:58:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:46 smithi164 conmon[29288]: cluster 2021-02-10T04:58:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:46 smithi164 conmon[29288]: 2021-02-10T04:58:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:46 smithi164 conmon[29288]: 2021 2021-02-10T04:58:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:46 smithi164 conmon[29288]: -02-10T04:58: 2021-02-10T04:58:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:46 smithi164 conmon[29288]: 45.772317+0000 mon.a (mon.0) 2945 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:58:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:46 smithi164 conmon[29288]: audit 2021 2021-02-10T04:58:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:46 smithi164 conmon[29288]: -02-10T04:58:45.786711+0000 2021-02-10T04:58:47.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:46 smithi164 conmon[29288]: mon.a (mon.0) 2946 : audit [INF] from='client.? 172.21.15.85:0/3366354685' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:58:47.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:46 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:58:47.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:46 smithi164 conmon[29288]: 04:58:45.786858+0000 mon.a (mon.0) 2947 : 2021-02-10T04:58:47.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:46 smithi164 conmon[29288]: cluster [DBG] osdmap e1328: 8 total, 7 up, 8 in 2021-02-10T04:58:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:47 smithi164 conmon[29288]: cluster 2021-02-10T04:58:46.554801+0000 mgr.y (mgr.14140 2021-02-10T04:58:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:47 smithi164 conmon[29288]: ) 1496 : cluster [DBG] pgmap v2432: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 753 B/s rd, 0 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:48.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:47 smithi085 conmon[29319]: cluster 2021-02-10T04:58:46.554801+0000 mgr.y (mgr.14140) 2021-02-10T04:58:48.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:47 smithi085 conmon[29319]: 1496 : cluster [DBG] pgmap v2432: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 753 B/s rd, 0 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:47 smithi085 conmon[35961]: cluster 2021-02-10T04:58:46.554801+0000 mgr.y (mgr.14140) 1496 : cluster [DBG] pgmap v2432: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 753 B/s rd, 0 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:48.219 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:58:48.241 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:58:48.262 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:58:48.280 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:58:48.301 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:58:48.320 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:58:48.339 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:58:48.339 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-02-10T04:58:48.339 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: problem getting command descriptions from osd.0 2021-02-10T04:58:48.339 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:58:48.339 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:58:48.339 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:58:48.340 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:58:48.340 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:58:48.340 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:58:48.340 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:58:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:48 smithi164 conmon[29288]: audit 2021-02 2021-02-10T04:58:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:48 smithi164 conmon[29288]: -10T04:58:48.193301+0000 mon.c (mon.1) 153 : audit [DBG] from='client.? 172.21.15.85:0/3339155974' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:58:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:48 smithi164 conmon[29288]: cluster 2021-02-10T04:58:48.622759+0000 mon.a ( 2021-02-10T04:58:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:48 smithi164 conmon[29288]: mon.0) 2948 : cluster [DBG] osdmap e1329: 8 total, 7 up, 8 in 2021-02-10T04:58:49.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:48 smithi085 conmon[29319]: audit 2021-02-10T04:58:48.193301+0000 mon.c (mon.1) 153 : audit [DBG] from='client.? 172.21.15.85:0/3339155974' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:58:49.228 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:48 smithi085 conmon[29319]: cluster 2021-02-10T04:58:48.622759+0000 mon.a (mon.0) 2948 : cluster [DBG] osdmap e1329: 8 total, 7 up, 8 in 2021-02-10T04:58:49.228 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:48 smithi085 conmon[35961]: audit 2021-02-10T04:58:48.193301+0000 mon.c (mon.1) 153 : audit [DBG] from='client.? 172.21.15.85:0/3339155974' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:58:49.228 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:48 smithi085 conmon[35961]: cluster 2021-02-10T04:58:48.622759+0000 mon.a (mon.0) 2948 : cluster [DBG] osdmap e1329: 8 total, 7 up, 8 in 2021-02-10T04:58:49.307 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:58:49.307 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:58:49.505 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset noscrub 2021-02-10T04:58:49.554 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:49 smithi085 bash[206868]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2021-02-10T04:58:49.554 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:49 smithi085 bash[206868]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2021-02-10T04:58:49.555 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:49 smithi085 bash[206868]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2021-02-10T04:58:49.555 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:49 smithi085 bash[206868]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2021-02-10T04:58:49.555 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:49 smithi085 bash[206868]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2021-02-10T04:58:49.555 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:49 smithi085 bash[206868]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2021-02-10T04:58:49.555 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:49 smithi085 bash[206868]: --> ceph-volume lvm activate successful for osd ID: 0 2021-02-10T04:58:49.736 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:58:49.803 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:58:49 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:58:49.803 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:49 smithi164 conmon[29288]: cluster 2021-02-10T04:58:49.807 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:49 smithi085 conmon[29319]: cluster 2021-02-10T04:58:48.555522+0000 mgr.y (mgr 2021-02-10T04:58:49.807 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:49 smithi085 conmon[29319]: .14140) 1497 : cluster [DBG] pgmap v2433: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:49.807 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:49 smithi085 conmon[29319]: cluster 2021-02-10T04:58:48.786772+0000 osd.7 (osd.7) 98 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:58:49.807 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:49 smithi085 conmon[29319]: cluster 2021-02-10T04:58:48.788973+0000 osd.7 (osd.7) 99 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:58:49.808 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:49 smithi085 conmon[35961]: cluster 2021-02-10T04:58:48.555522+0000 mgr.y (mgr.14140) 1497 : cluster [DBG] pgmap v2433: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:49.808 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:49 smithi085 conmon[35961]: cluster 2021-02-10T04:58:48.786772+0000 osd.7 (osd.7) 98 : cluster 2021-02-10T04:58:49.808 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:49 smithi085 conmon[35961]: [DBG] purged_snaps scrub starts 2021-02-10T04:58:49.808 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:49 smithi085 conmon[35961]: cluster 2021-02-10T04:58:49.808 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:49 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T04:58:49.808 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:49 smithi085 conmon[35961]: 04:58:48.788973+0000 osd.7 (osd 2021-02-10T04:58:49.808 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:49 smithi085 conmon[35961]: .7) 99 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:58:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:49 smithi164 conmon[29288]: 2021-02-10T04:58: 2021-02-10T04:58:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:49 smithi164 conmon[29288]: 48.555522+0000 mgr.y (mgr.14140) 1497 : cluster [DBG] pgmap v2433: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:49 smithi164 conmon[29288]: cluster 2021-02-10T04:58:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:49 smithi164 conmon[29288]: 2021-02-10T04:58:48.786772+0000 osd.7 2021-02-10T04:58:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:49 smithi164 conmon[29288]: (osd.7) 98 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:58:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:49 smithi164 conmon[29288]: cluster 2021-02-10T04:58:50.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:49 smithi164 conmon[29288]: 2021-02- 2021-02-10T04:58:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:49 smithi164 conmon[29288]: 10T04:58:48.788973+0000 2021-02-10T04:58:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:49 smithi164 conmon[29288]: osd.7 (osd.7) 99 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:58:50.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:49 smithi164 conmon[29288]: 2021-02-10T04:58:51.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:50 smithi164 conmon[29288]: audit 2021-02-10T04:58:49.513002+0000 mgr.y 2021-02-10T04:58:51.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:50 smithi164 conmon[29288]: (mgr.14140) 1498 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:58:51.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:50 smithi164 conmon[29288]: cluster 2021-02-10T04:58:51.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:50 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:58:51.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:50 smithi164 conmon[29288]: 58:49.798640+0000 mon.a (mon.0) 2021-02-10T04:58:51.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:50 smithi164 conmon[29288]: 2949 : cluster [DBG] osdmap e1330: 8 total, 7 up, 8 in 2021-02-10T04:58:51.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:50 smithi085 conmon[29319]: audit 2021-02-10T04:58:49.513002+0000 mgr.y (mgr.14140 2021-02-10T04:58:51.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:50 smithi085 conmon[29319]: ) 1498 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:58:51.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:50 smithi085 conmon[29319]: cluster 2021-02-10T04:58:49.798640+0000 mon.a (mon.0) 2949 : cluster [DBG] osdmap e1330: 8 total, 7 up, 8 in 2021-02-10T04:58:51.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:50 smithi085 conmon[35961]: audit 2021-02-10T04:58:49.513002+0000 mgr.y (mgr 2021-02-10T04:58:51.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:50 smithi085 conmon[35961]: .14140) 1498 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:58:51.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:50 smithi085 conmon[35961]: cluster 2021-02-10T04:58:49.798640+0000 mon.a (mon.0) 2949 : cluster [DBG] osdmap e1330: 8 total, 7 up, 8 in 2021-02-10T04:58:51.308 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:58:51.534 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.0 dump_ops_in_flight 2021-02-10T04:58:51.566 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:58:51.692 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:51 smithi085 bash[206868]: 45efca45f6704f069274172006007b59e5dc24b241978723ce7baff2bdc0c00b 2021-02-10T04:58:51.692 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:51 smithi085 systemd[1]: Started Ceph osd.0 for 42224fce-6b57-11eb-8feb-001a4aab830c. 2021-02-10T04:58:51.692 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:51 smithi085 conmon[207560]: debug 2021-02-10T04:58:51.692 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:51 smithi085 conmon[207560]: 2021-02-10T04:58:51.540+0000 7f26b5ddc700 20 -- 172.21.15.85:0/528677346 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x563eeeae2c00 msgr2=0x563eeea4d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2021-02-10T04:58:51.692 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:51 smithi085 conmon[207560]: debug 2021-02-10T04:58:51.540+0000 7f26b5ddc700 20 -- 172.21.15.85:0/528677346 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x563eeeae2c00 msgr2=0x563eeea4d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=96 2021-02-10T04:58:51.692 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:51 smithi085 conmon[207560]: debug 2021-02-10T04:58:51.540+0000 7f26b5ddc700 20 --2- 172.21.15.85:0/528677346 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x563eeeae2c00 0x563eeea4d400 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x563eeea63d40 tx=0x563eeea94d80).handle_read_frame_preamble_main r=0 2021-02-10T04:58:51.692 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:51 smithi085 conmon[207560]: debug 2021-02-10T04:58:51.540+0000 7f26b5ddc700 20 --2- 172.21.15.85:0/528677346 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x563eeeae2c00 0x563eeea4d400 secure :-1 s=THROTTLE_MESSAGE pgs=360 cs=0 l=1 rev1=1 rx=0x563eeea63d40 tx=0x563eeea94d80).throttle_message 2021-02-10T04:58:51.692 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:51 smithi085 conmon[207560]: debug 2021-02-10T04:58:51.540+0000 7f26b5ddc700 20 --2- 172.21.15.85:0/528677346 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x563eeeae2c00 0x563eeea4d400 secure :-1 s=THROTTLE_BYTES pgs=360 cs=0 l=1 rev1=1 rx=0x563eeea63d40 tx=0x563eeea94d80).throttle_bytes 2021-02-10T04:58:51.692 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:51 smithi085 conmon[207560]: 2021-02-10T04:58:51.692 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:51 smithi085 conmon[207560]: debug 2021-02-10T04:58:51.540+0000 7f26b5ddc700 20 --2- 172.21.15.85:0/528677346 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x563eeeae2c00 0x563eeea4d400 secure :-1 s=THROTTLE_DISPATCH_QUEUE pgs=360 cs=0 l=1 rev1=1 rx=0x563eeea63d40 tx=0x563eeea94d80).throttle_dispatch_queue 2021-02-10T04:58:51.693 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:51 smithi085 conmon[207560]: debug 2021-02-10T04:58:51.540+0000 7f26b5ddc700 20 --2- 172.21.15.85:0/528677346 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x563eeeae2c00 0x563eeea4d400 secure :-1 s=THROTTLE_DONE pgs=360 cs=0 l=1 rev1=1 rx=0x563eeea63d40 tx=0x563eeea94d80).read_frame_segment seg_idx=0 2021-02-10T04:58:51.693 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:51 smithi085 conmon[207560]: debug 2021-02-10T04:58:51.693 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:51 smithi085 conmon[207560]: 2021-02-10T04:58:51.540+0000 7f26b5ddc700 20 --2- 172.21.15.85:0/528677346 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x563eeeae2c00 0x563eeea4d400 secure :-1 s=THROTTLE_DONE pgs=360 cs=0 l=1 rev1=1 rx=0x563eeea63d40 tx=0x563eeea94d80).read_frame_segment seg_idx=1 2021-02-10T04:58:51.693 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:51 smithi085 conmon[207560]: debug 2021-02-10T04:58:51.540+0000 7f26b5ddc700 20 -- 172.21.15.85:0/528677346 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x563eeeae2c00 msgr2=0x563eeea4d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=208 2021-02-10T04:58:51.693 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:51 smithi085 conmon[207560]: debug 2021-02-10T04:58:51.540+0000 7f26b5ddc700 20 --2- 172.21.15.85:0/528677346 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x563eeeae2c00 0x563eeea4d400 secure :-1 s=THROTTLE_DONE pgs=360 cs=0 l=1 rev1=1 rx=0x563eeea63d40 tx=0x563eeea94d80).handle_read_frame_segment r=0 2021-02-10T04:58:51.693 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:51 smithi085 conmon[207560]: debug 2021-02-10T04:58:51.540+0000 7f26b5ddc700 20 -- 172.21.15.85:0/528677346 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x563eeeae2c00 msgr2=0x563eeea4d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=32 2021-02-10T04:58:51.693 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:51 smithi085 conmon[207560]: debug 2021-02-10T04:58:51.540+0000 7f26b5ddc700 20 --2- 172.21.15.85:0/528677346 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x563eeeae2c00 0x563eeea4d400 secure :-1 s=THROTTLE_DONE pgs=360 cs=0 l=1 rev1=1 rx=0x563eeea63d40 tx=0x563eeea94d80).handle_read_frame_epilogue_main r=0 2021-02-10T04:58:51.693 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:51 smithi085 conmon[207560]: debug 2021-02-10T04:58:51.540+0000 7f26b5ddc700 10 --2- 172.21.15.85:0/528677346 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x563eeeae2c00 0x563eeea4d400 secure :-1 s=THROTTLE_DONE pgs=360 cs=0 l=1 rev1=1 rx=0x563eeea63d40 tx=0x563eeea94d80).handle_read_frame_dispatch tag=17 2021-02-10T04:58:51.693 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:51 smithi085 conmon[207560]: 2021-02-10T04:58:51.693 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:51 smithi085 conmon[207560]: debug 2021-02-10T04:58:51.540+0000 7f26b5ddc700 20 --2- 172.21.15.85:0/528677346 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x563eeeae2c00 0x563eeea4d400 secure :-1 s=THROTTLE_DONE pgs=360 cs=0 l=1 rev1=1 rx=0x563eeea63d40 tx=0x563eeea94d80).handle_message 2021-02-10T04:58:51.693 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:51 smithi085 conmon[207560]: debug 2021-02-10T04:58:51.540+0000 7f26b5ddc700 5 --2- 172.21.15.85:0/528677346 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x563eeeae2c00 0x563eeea4d400 secure :-1 s=THROTTLE_DONE pgs=360 cs=0 l=1 rev1=1 rx=0x563eeea63d40 tx=0x563eeea94d80).handle_message got 194 + 0 + 0 byte message. envelope type=18 src mon.1 off 0 2021-02-10T04:58:51.693 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:51 smithi085 conmon[207560]: debug 2021-02-10T04:58:51.540+0000 7f26b5ddc700 5 --2- 172.21.15.85:0/528677346 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x563eeeae2c00 0x563eeea4d400 secure :-1 s=READ_MESSAGE_COMPLETE pgs=360 cs=0 l=1 rev1=1 rx=0x563eeea63d40 tx=0x563eeea94d80).handle_message received message m=0x563eeea664e0 seq=4 from=mon.1 type=18 auth_reply(proto 2 0 (0) Success) v1 2021-02-10T04:58:51.694 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:51 smithi085 conmon[207560]: debug 2021-02-10T04:58:51.541+0000 7f26b5ddc700 20 -- 172.21.15.85:0/528677346 queue 0x563eeea664e0 prio 196 2021-02-10T04:58:51.694 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:51 smithi085 conmon[207560]: debug 2021-02-10T04:58:51.541+0000 7f26b5ddc700 20 --2- 172.21.15.85:0/528677346 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x563eeeae2c00 0x563eeea4d400 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x563eeea63d40 tx=0x563eeea94d80).read_frame 2021-02-10T04:58:51.694 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:51 smithi085 conmon[207560]: debug 2021-02-10T04:58:51.541+0000 7f26b5ddc700 20 -- 172.21.15.85:0/528677346 >> [v2:172.21.15.85:3301/0,v1:172.21.15.85:6790/0] conn(0x563eeeae2c00 msgr2=0x563eeea4d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=96 2021-02-10T04:58:51.694 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:51 smithi085 conmon[207560]: debug 2021-02-10T04:58:51.541+0000 7f26b4dda700 1 -- 172.21.15.85:0/528677346 <== mon.1 v2:172.21.15.85:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563eeea664e0 con 0x563eeeae2c00 2021-02-10T04:58:51.694 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:51 smithi085 conmon[207560]: debug 2021-02-10T04:58:51.541+0000 7f26b4dda700 10 -- 172.21.15.85:0/528677346 dispatch_throttle_release 194 to dispatch throttler 194/104857600 2021-02-10T04:58:51.694 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:51 smithi085 conmon[207560]: debug 2021-02-10T04:58:51.541+0000 7f26b4dda700 20 -- 172.21.15.85:0/528677346 done calling dispatch on 0x563eeea664e0 2021-02-10T04:58:51.694 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:51 smithi085 conmon[207560]: 2021-02-10T04:58:51.779 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:58:51.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:51 smithi085 conmon[29319]: cluster 2021-02-10T04:58:50.556167+0000 2021-02-10T04:58:51.950 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:51 smithi085 conmon[29319]: mgr.y (mgr.14140) 1499 : cluster [DBG] pgmap v2436: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:51.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:51 smithi085 conmon[35961]: cluster 2021-02-10T04:58:50.556167+0000 mgr.y (mgr.14140) 1499 : cluster [DBG] pgmap v2436: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:52.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:51 smithi164 conmon[29288]: cluster 2021-02-10T04:58:50.556167 2021-02-10T04:58:52.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:51 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1499 : cluster [DBG] pgmap v2436: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:52.299 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:52 smithi085 conmon[207560]: debug 2021-02-10T04:58:52.062+0000 7f26bc865200 -1 Falling back to public interface 2021-02-10T04:58:53.227 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:58:53.228 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2021-02-10T04:58:53.228 INFO:teuthology.orchestra.run.smithi164.stdout: "complaint_time": 30, 2021-02-10T04:58:53.228 INFO:teuthology.orchestra.run.smithi164.stdout: "num_blocked_ops": 0 2021-02-10T04:58:53.228 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:58:53.634 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is unset 2021-02-10T04:58:53.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:53 smithi085 conmon[29319]: cluster 2021-02-10T04:58:52.556801+0000 mgr.y (mgr. 2021-02-10T04:58:53.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:53 smithi085 conmon[29319]: 14140) 1500 : cluster [DBG] pgmap v2437: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 756 B/s rd, 0 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:53.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:53 smithi085 conmon[29319]: audit 2021-02-10T04:58:53.416257+0000 mon.c (mon.1) 154 : audit [INF] from='client.? 172.21.15.85:0/846649463' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:58:53.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:53 smithi085 conmon[29319]: audit 2021-02-10T04:58:53.416586+0000 mon.a (mon.0) 2950 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:58:53.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:53 smithi085 conmon[29319]: cluster 2021-02-10T04:58:53.620534+0000 mon.a (mon.0) 2951 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:58:53.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:53 smithi085 conmon[29319]: audit 2021-02-10T04:58:53.625022+0000 mon.a (mon.0) 2952 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:58:53.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:53 smithi085 conmon[29319]: cluster 2021-02-10T04:58:53.625101+0000 mon.a (mon.0) 2953 : cluster [DBG] osdmap e1331: 8 total, 7 up, 8 in 2021-02-10T04:58:53.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:53 smithi085 conmon[35961]: cluster 2021-02-10T04:58:53.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:53 smithi085 conmon[35961]: 2021-02-10T04:58: 2021-02-10T04:58:53.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:53 smithi085 conmon[35961]: 52.556801+0000 mgr.y (mgr.14140) 1500 : cluster [DBG] pgmap v2437: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 756 B/s rd, 0 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:53.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:53 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:58:53.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:53 smithi085 conmon[35961]: 02-10T04:58:53.416257 2021-02-10T04:58:53.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:53 smithi085 conmon[35961]: +0000 mon.c (mon.1) 154 : audit [INF] from='client.? 172.21.15.85:0/846649463' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:58:53.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:53 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:58:53.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:53 smithi085 conmon[35961]: 10T04:58:53.416586+0000 mon.a (mon.0) 2021-02-10T04:58:53.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:53 smithi085 conmon[35961]: 2950 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:58:53.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:53 smithi085 conmon[35961]: cluster 2021-02-10T04:58:53 2021-02-10T04:58:53.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:53 smithi085 conmon[35961]: .620534+0000 mon.a (mon.0) 2951 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:58:53.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:53 smithi085 conmon[35961]: audit 2021-02-10T04:58:53.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:53 smithi085 conmon[35961]: 2021-02-10 2021-02-10T04:58:53.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:53 smithi085 conmon[35961]: T04:58:53.625022+0000 mon.a (mon.0) 2952 : audit [INF] 2021-02-10T04:58:53.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:53 smithi085 conmon[35961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:58:53.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:53 smithi085 conmon[35961]: cluster 2021-02-10T04:58:53.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:53 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:58:53.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:53 smithi085 conmon[35961]: 10T04:58:53.625101+0000 mon.a (mon.0) 2953 : cluster [DBG] osdmap e1331: 8 total, 7 up, 8 in 2021-02-10T04:58:53.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:53 smithi085 conmon[35961]: 2021-02-10T04:58:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:53 smithi164 conmon[29288]: cluster 2021-02-10T04:58:52.556801+0000 mgr.y ( 2021-02-10T04:58:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:53 smithi164 conmon[29288]: mgr.14140) 1500 : cluster [DBG] pgmap v2437: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 756 B/s rd, 0 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:53 smithi164 conmon[29288]: audit 2021-02-10T04:58:53.416257+0000 mon.c (mon.1) 154 : audit [INF] from='client.? 172.21.15.85:0/846649463' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:58:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:53 smithi164 conmon[29288]: audit 2021-02-10T04:58:53.416586+0000 mon.a (mon.0) 2950 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:58:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:53 smithi164 conmon[29288]: cluster 2021-02-10T04:58:53.620534+0000 mon.a (mon.0) 2951 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:58:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:53 smithi164 conmon[29288]: audit 2021-02-10T04:58:53.625022+0000 mon.a (mon.0) 2952 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:58:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:53 smithi164 conmon[29288]: cluster 2021-02-10T04:58:53.625101+0000 mon.a ( 2021-02-10T04:58:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:53 smithi164 conmon[29288]: mon.0) 2953 : cluster [DBG] osdmap e1331: 8 total, 7 up, 8 in 2021-02-10T04:58:55.077 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:54 smithi085 conmon[29319]: cephadm 2021-02-10T04:58:53.849530+0000 mgr.y (mgr.14140) 1501 : cephadm 2021-02-10T04:58:55.077 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:54 smithi085 conmon[29319]: [INF] Refreshing smithi085 facts 2021-02-10T04:58:55.077 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:54 smithi085 conmon[29319]: cephadm 2021-02-10T04:58:53.849907+0000 mgr.y (mgr.14140) 1502 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:58:55.077 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:54 smithi085 conmon[29319]: audit 2021-02-10T04:58:54.670421+0000 mon.a (mon.0) 2954 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:58:55.078 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:54 smithi085 conmon[35961]: cephadm 2021-02-10T04:58:53.849530+0000 2021-02-10T04:58:55.078 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:54 smithi085 conmon[35961]: mgr.y (mgr.14140) 1501 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:58:55.078 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:54 smithi085 conmon[35961]: cephadm 2021-02-10T04 2021-02-10T04:58:55.079 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:54 smithi085 conmon[35961]: :58:53.849907+0000 mgr.y (mgr.14140) 1502 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:58:55.079 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:54 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:58:55.079 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:54 smithi085 conmon[35961]: 02-10T04:58:54.670421+0000 mon.a ( 2021-02-10T04:58:55.079 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:54 smithi085 conmon[35961]: mon.0) 2954 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:58:55.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:54 smithi164 conmon[29288]: cephadm 2021-02-10T04:58:53.849530+0000 mgr.y (mgr. 2021-02-10T04:58:55.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:54 smithi164 conmon[29288]: 14140) 1501 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:58:55.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:54 smithi164 conmon[29288]: cephadm 2021-02-10T04:58:53.849907+0000 mgr.y (mgr.14140) 1502 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T04:58:55.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:54 smithi164 conmon[29288]: audit 2021-02-10T04:58:54.670421+0000 mon.a (mon.0) 2954 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T04:58:55.333 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[207560]: debug 2021-02-10T04:58:55.155+0000 7f26bc865200 -1 osd.0 1315 log_to_monitors {default=true} 2021-02-10T04:58:55.634 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:58:55.648 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:58:55.648 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [], 2021-02-10T04:58:55.648 INFO:teuthology.orchestra.run.smithi085.stdout: "num_ops": 0 2021-02-10T04:58:55.648 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:58:55.653 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:58:55.674 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:58:55.693 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:58:55.718 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:58:55.739 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:58:55.759 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:58:55.759 INFO:teuthology.orchestra.run.smithi085.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-02-10T04:58:55.760 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: problem getting command descriptions from osd.0 2021-02-10T04:58:55.760 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:58:55.760 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:58:55.760 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:58:55.760 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:58:55.760 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:58:55.760 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:58:55.761 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:58:55.934 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[29319]: cluster 2021-02-10T04:58:55.934 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[29319]: 2021-02-10T04:58:55.934 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[29319]: 2021 2021-02-10T04:58:55.935 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[29319]: - 2021-02-10T04:58:55.935 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[29319]: 02 2021-02-10T04:58:55.935 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[29319]: -10T04:58:54.557325 2021-02-10T04:58:55.935 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 1503 : cluster [DBG] pgmap v2439: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:55.935 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:58:55.936 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[29319]: 02-10T04:58:55.160527+0000 mon.b (mon. 2021-02-10T04:58:55.936 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[29319]: 2) 52 : audit [INF] from='osd.0 v2:172.21.15.85:6801/7' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-02-10T04:58:55.936 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:58:55.936 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[29319]: 02-10T04:58:55.160924+0000 mon.a ( 2021-02-10T04:58:55.936 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[29319]: mon.0) 2955 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-02-10T04:58:55.936 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:58:55.936 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[29319]: 02-10T04:58:55.161851+0000 mon.b ( 2021-02-10T04:58:55.937 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[29319]: mon.2) 53 : audit [INF] from='osd.0 v2:172.21.15.85:6801/7' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:58:55.937 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:58:55.937 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[29319]: 10T04:58:55.162220+0000 mon.a (mon 2021-02-10T04:58:55.937 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[29319]: .0) 2956 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:58:55.937 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:58:55.937 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[29319]: 02-10T04:58:55.614112+0000 mon.c 2021-02-10T04:58:55.938 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[29319]: (mon.1) 155 : audit [DBG] from='client.? 172.21.15.85:0/2769047239' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:58:55.938 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[35961]: cluster 2021-02-10T04:58:55.938 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:58:55.938 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[35961]: :58:54.557325+0000 mgr.y (mgr.14140) 1503 : cluster [DBG] pgmap v2439: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:55.938 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[35961]: audit 2021 2021-02-10T04:58:55.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[35961]: -02-10T04:58:55.160527+0000 2021-02-10T04:58:55.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[35961]: mon.b (mon.2) 52 : audit [INF] 2021-02-10T04:58:55.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[35961]: from='osd.0 v2:172.21.15.85:6801/7' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-02-10T04:58:55.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:58:55.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[35961]: 58:55.160924+0000 mon.a (mon.0 2021-02-10T04:58:55.939 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[35961]: ) 2955 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-02-10T04:58:55.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:58:55.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[35961]: 02-10T04:58:55.161851+0000 mon.b 2021-02-10T04:58:55.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[35961]: (mon.2) 53 : audit [INF] from='osd.0 v2:172.21.15.85:6801/7' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:58:55.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:58:55.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[35961]: 02-10T04:58:55.162220+0000 2021-02-10T04:58:55.940 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[35961]: mon.a (mon.0) 2956 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:58:55.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:58:55.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[35961]: 10T04:58:55.614112+0000 mon.c (mon. 2021-02-10T04:58:55.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:55 smithi085 conmon[35961]: 1) 155 : audit [DBG] from='client.? 172.21.15.85:0/2769047239' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:58:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:55 smithi164 conmon[29288]: cluster 2021-02-10T04:58:54.557325+0000 mgr.y ( 2021-02-10T04:58:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:55 smithi164 conmon[29288]: mgr.14140) 1503 : cluster [DBG] pgmap v2439: 198 pgs: 1 active+clean+scrubbing, 26 active+undersized+degraded, 42 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 258/2196 objects degraded (11.749%) 2021-02-10T04:58:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:55 smithi164 conmon[29288]: audit 2021-02-10T04:58:55.160527+0000 mon.b (mon 2021-02-10T04:58:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:55 smithi164 conmon[29288]: .2) 52 : audit [INF] from='osd.0 v2:172.21.15.85:6801/7' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-02-10T04:58:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:55 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:58:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:55 smithi164 conmon[29288]: T04:58:55.160924+0000 mon.a (mon.0 2021-02-10T04:58:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:55 smithi164 conmon[29288]: ) 2955 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-02-10T04:58:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:55 smithi164 conmon[29288]: audit 2021-02-10T04:58:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:55 smithi164 conmon[29288]: 2021-02-10T04:58:55.161851 2021-02-10T04:58:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:55 smithi164 conmon[29288]: +0000 mon.b (mon.2) 53 : audit [INF] from='osd.0 v2:172.21.15.85:6801/7' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:58:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:55 smithi164 conmon[29288]: 2021-02-10T04:58:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:55 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:58:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:55 smithi164 conmon[29288]: :58:55.162220+0000 mon.a (mon.0 2021-02-10T04:58:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:55 smithi164 conmon[29288]: ) 2956 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi085", "root=default"]}]: dispatch 2021-02-10T04:58:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:55 smithi164 conmon[29288]: audit 2021-02-10T04:58:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:55 smithi164 conmon[29288]: 2021-02-10T04:58:55.614112+0000 2021-02-10T04:58:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:55 smithi164 conmon[29288]: mon.c (mon.1) 155 : audit [DBG] from='client.? 172.21.15.85:0/2769047239' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:58:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:55 smithi164 conmon[29288]: 2021-02-10T04:58:56.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[29319]: cluster 2021-02-10T04:58:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[29319]: 2021-02-10T04:58 2021-02-10T04:58:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[29319]: :55.818222+0000 mon.a (mon.0) 2957 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2021-02-10T04:58:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:58:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[29319]: 02-10T04:58:55.828883+0000 2021-02-10T04:58:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[29319]: mon.a (mon.0) 2958 : cluster [INF] 2021-02-10T04:58:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[29319]: osd.0 v2:172.21.15.85:6801/7 boot 2021-02-10T04:58:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T04:58:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[29319]: -10T04:58:55. 2021-02-10T04:58:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[29319]: 828914+0000 mon.a (mon.0) 2021-02-10T04:58:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[29319]: 2959 : cluster [DBG] osdmap e1332: 8 total, 8 up, 8 in 2021-02-10T04:58:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[29319]: audit 2021 2021-02-10T04:58:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[29319]: -02-10T04: 2021-02-10T04:58:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[29319]: 58:55.831406+0000 mon.a (mon.0 2021-02-10T04:58:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[29319]: ) 2960 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-02-10T04:58:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:58:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[29319]: -02-10T04:58: 2021-02-10T04:58:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[29319]: 55.999885+0000 osd.4 (osd.4) 2663 2021-02-10T04:58:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[29319]: : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:58:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:58:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[29319]: 02-10T04:58:56.001556+0000 2021-02-10T04:58:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[29319]: osd.4 (osd.4) 2664 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:58:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[29319]: cluster 2021-02-10T04:58: 2021-02-10T04:58:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[29319]: 56.823405+0000 mon.a (mon.0) 2961 : cluster [DBG] osdmap e1333: 8 total, 8 up, 8 in 2021-02-10T04:58:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[29319]: 2021-02-10T04:58:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: cluster 2021-02-10T04:58:55.818222 2021-02-10T04:58:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2957 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2021-02-10T04:58:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:58:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: -02-10T04:58:55.828883+0000 mon.a (mon.0) 2958 : cluster [INF] 2021-02-10T04:58:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: osd.0 v2:172.21.15.85:6801/7 boot 2021-02-10T04:58:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:58:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: 04:58:55.828914+0000 mon.a (mon.0 2021-02-10T04:58:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: ) 2959 : cluster [DBG] osdmap e1332: 8 total, 8 up, 8 in 2021-02-10T04:58:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: audit 2021-02-10T04:58:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: 2021-02- 2021-02-10T04:58:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: 10T04:58:55.831406+0000 mon.a (mon. 2021-02-10T04:58:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: 0) 2960 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-02-10T04:58:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:58:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: 02-10T04:58:55.999885+0000 osd.4 (osd. 2021-02-10T04:58:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: 4) 2663 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:58:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: cluster 2021-02-10T04:58:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: 2021 2021-02-10T04:58:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: - 2021-02-10T04:58:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: 02 2021-02-10T04:58:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: - 2021-02-10T04:58:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: 10 2021-02-10T04:58:56.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: T 2021-02-10T04:58:56.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: 04 2021-02-10T04:58:56.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: : 2021-02-10T04:58:56.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: 58 2021-02-10T04:58:56.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: : 2021-02-10T04:58:56.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: 56 2021-02-10T04:58:56.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: . 2021-02-10T04:58:56.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: 001556 2021-02-10T04:58:56.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: +0000 2021-02-10T04:58:56.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: 2021-02-10T04:58:56.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: osd.4 (osd.4) 2664 : cluster 2021-02-10T04:58:56.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: [DBG] purged_snaps scrub ok 2021-02-10T04:58:56.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: cluster 2021-02-10T04:58:56.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: 2021-02-10T04:58:56.823405+0000 mon.a ( 2021-02-10T04:58:56.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:56 smithi085 conmon[35961]: mon.0) 2961 : cluster [DBG] osdmap e1333: 8 total, 8 up, 8 in 2021-02-10T04:58:57.079 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset nodeep-scrub 2021-02-10T04:58:57.127 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-02-10T04:58:57.128 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-02-10T04:58:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:56 smithi164 conmon[29288]: cluster 2021-02-10T04:58:55.818222+0000 mon.a (mon.0 2021-02-10T04:58:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:56 smithi164 conmon[29288]: ) 2957 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2021-02-10T04:58:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:56 smithi164 conmon[29288]: cluster 2021-02-10T04:58:55.828883+0000 mon.a (mon.0) 2958 : cluster [INF] osd.0 v2:172.21.15.85:6801/7 boot 2021-02-10T04:58:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:56 smithi164 conmon[29288]: cluster 2021-02-10T04:58:55.828914+0000 mon.a (mon 2021-02-10T04:58:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:56 smithi164 conmon[29288]: .0) 2959 : cluster [DBG] osdmap e1332: 8 total, 8 up, 8 in 2021-02-10T04:58:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:56 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:58:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:56 smithi164 conmon[29288]: 04:58:55.831406+0000 mon.a (mon. 2021-02-10T04:58:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:56 smithi164 conmon[29288]: 0) 2960 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-02-10T04:58:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:56 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:58:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:56 smithi164 conmon[29288]: 02-10T04:58:55.999885+0000 osd.4 (osd 2021-02-10T04:58:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:56 smithi164 conmon[29288]: .4) 2663 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:58:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:56 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:58:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:56 smithi164 conmon[29288]: -10T04:58:56.001556+0000 osd.4 (osd.4 2021-02-10T04:58:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:56 smithi164 conmon[29288]: ) 2664 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:58:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:56 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:58:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:56 smithi164 conmon[29288]: 04:58:56.823405+0000 mon.a (mon.0) 2021-02-10T04:58:57.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:56 smithi164 conmon[29288]: 2961 : cluster [DBG] osdmap e1333: 8 total, 8 up, 8 in 2021-02-10T04:58:57.299 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:58:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:57 smithi164 conmon[29288]: cluster 2021-02-10T04:58:56.557874+0000 mgr.y (mgr.14140) 2021-02-10T04:58:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:57 smithi164 conmon[29288]: 1504 : cluster [DBG] pgmap v2441: 198 pgs: 8 peering, 1 active+clean+scrubbing, 23 active+undersized+degraded, 37 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 757 B/s rd, 0 op/s; 244/2196 objects degraded (11.111%) 2021-02-10T04:58:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:57 smithi164 conmon[29288]: cluster 2021-02-10T04:58:56.835862+0000 mon.a (mon.0) 2962 : cluster [WRN] Health check update: Degraded data redundancy: 244/2196 objects degraded (11.111%), 23 pgs degraded (PG_DEGRADED) 2021-02-10T04:58:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:57 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:58:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:57 smithi164 conmon[29288]: T04:58:57.207520+0000 osd.5 (osd.5) 198 : cluster [DBG] 5.15 scrub ok 2021-02-10T04:58:58.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:57 smithi085 conmon[29319]: cluster 2021-02-10T04:58:56.557874+0000 mgr.y (mgr.14140) 1504 : cluster [DBG] pgmap v2441: 198 pgs: 8 peering, 1 active+clean+scrubbing, 23 active+undersized+degraded, 37 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 757 B/s rd, 0 op/s; 244/2196 objects degraded (11.111%) 2021-02-10T04:58:58.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:57 smithi085 conmon[29319]: cluster 2021-02-10T04:58:56.835862+0000 mon.a (mon.0) 2962 : cluster [WRN] Health check update: Degraded data redundancy: 244/2196 objects degraded (11.111%), 23 pgs degraded (PG_DEGRADED) 2021-02-10T04:58:58.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:57 smithi085 conmon[29319]: cluster 2021-02-10T04:58:57.207520+0000 osd.5 (osd.5) 198 : cluster [DBG] 5.15 scrub ok 2021-02-10T04:58:58.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:57 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:58:58.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:57 smithi085 conmon[35961]: :58:56.557874+0000 mgr.y (mgr.14140) 1504 : cluster [DBG] pgmap v2441: 198 pgs: 8 peering, 1 active+clean+scrubbing, 23 active+undersized+degraded, 37 active+undersized, 129 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 757 B/s rd, 0 op/s; 244/2196 objects degraded (11.111%) 2021-02-10T04:58:58.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:57 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:58:58.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:57 smithi085 conmon[35961]: 02-10T04:58:56.835862+0000 mon.a (mon.0) 2962 : cluster [WRN] Health check update: Degraded data redundancy: 244/2196 objects degraded (11.111%), 23 pgs degraded (PG_DEGRADED) 2021-02-10T04:58:58.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:57 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:58:58.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:57 smithi085 conmon[35961]: T04:58:57.207520+0000 osd.5 (osd. 2021-02-10T04:58:58.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:57 smithi085 conmon[35961]: 5) 198 : cluster [DBG] 5.15 scrub ok 2021-02-10T04:58:59.063 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:58 smithi085 conmon[29319]: cluster 2021-02-10T04:58: 2021-02-10T04:58:59.063 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:58 smithi085 conmon[29319]: 56.121338+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:58:59.063 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:58 smithi085 conmon[29319]: cluster 2021-02-10T04:58:56.125496+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:58:59.063 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:58 smithi085 conmon[29319]: cluster 2021-02-10T04:58:57.081138+0000 osd.0 (osd.0) 3 : cluster [DBG] 6.9 scrub ok 2021-02-10T04:58:59.064 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:58 smithi085 conmon[35961]: cluster 2021-02-10T04:58:56. 2021-02-10T04:58:59.064 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:58 smithi085 conmon[35961]: 121338+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:58:59.064 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:58 smithi085 conmon[35961]: cluster 2021-02-10T04:58:56 2021-02-10T04:58:59.064 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:58 smithi085 conmon[35961]: .125496+0000 osd.0 (osd.0 2021-02-10T04:58:59.064 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:58 smithi085 conmon[35961]: ) 2 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:58:59.064 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:58 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:58:59.064 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:58 smithi085 conmon[35961]: -02-10T04:58:57. 2021-02-10T04:58:59.064 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:58 smithi085 conmon[35961]: 081138+0000 osd.0 (osd.0) 3 : cluster [DBG] 6.9 scrub ok 2021-02-10T04:58:59.129 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:58:59.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:58 smithi164 conmon[29288]: cluster 2021-02-10T04:58:56.121338+0000 osd.0 (osd. 2021-02-10T04:58:59.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:58 smithi164 conmon[29288]: 0) 1 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:58:59.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:58 smithi164 conmon[29288]: cluster 2021-02-10T04:58:56.125496+0000 osd.0 (osd. 2021-02-10T04:58:59.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:58 smithi164 conmon[29288]: 0) 2 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:58:59.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:58 smithi164 conmon[29288]: cluster 2021-02-10T04:58:59.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:58 smithi164 conmon[29288]: 2021-02-10T04:58:57.081138+0000 osd.0 (osd.0) 2021-02-10T04:58:59.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:58 smithi164 conmon[29288]: 3 : cluster [DBG] 6.9 scrub ok 2021-02-10T04:58:59.358 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:58:59.720 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.5 dump_historic_ops 2021-02-10T04:58:59.854 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is unset 2021-02-10T04:58:59.855 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:58:59 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:59:00.125 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:59:00.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:59 smithi164 conmon[29288]: cluster 2021-02-10T04:58:58.251964+0000 osd.5 (osd.5) 199 : 2021-02-10T04:59:00.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:59 smithi164 conmon[29288]: cluster [DBG] 5.7 scrub ok 2021-02-10T04:59:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:59 smithi164 conmon[29288]: cluster 2021-02-10T04:58:58.558623+0000 mgr.y (mgr.14140) 1505 : cluster [DBG] pgmap v2443: 198 pgs: 1 active+clean+scrubbing, 2 activating, 8 peering, 15 active+undersized+degraded, 20 active+undersized, 152 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 194/2196 objects degraded (8.834%) 2021-02-10T04:59:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:59 smithi164 conmon[29288]: audit 2021-02-10T04:58:59.520160+0000 mon.a (mon. 2021-02-10T04:59:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:58:59 smithi164 conmon[29288]: 0) 2963 : audit [INF] from='client.? 172.21.15.85:0/4043852611' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:59:00.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:59 smithi085 conmon[29319]: cluster 2021-02-10T04:58:58.251964+0000 osd.5 (osd.5) 199 : 2021-02-10T04:59:00.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:59 smithi085 conmon[29319]: cluster [DBG] 5.7 scrub ok 2021-02-10T04:59:00.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:59 smithi085 conmon[29319]: cluster 2021-02-10T04:58:58.558623+0000 mgr.y (mgr.14140) 1505 : cluster [DBG] pgmap v2443: 198 pgs: 1 active+clean+scrubbing, 2 activating, 8 peering, 15 active+undersized+degraded, 20 active+undersized, 152 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 194/2196 objects degraded (8.834%) 2021-02-10T04:59:00.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:58:59 smithi085 conmon[29319]: audit 2021-02-10T04:58:59.520160+0000 mon.a (mon.0) 2963 : audit [INF] from='client.? 172.21.15.85:0/4043852611' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:59:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:59 smithi085 conmon[35961]: cluster 2021-02-10T04:58:58.251964+0000 osd.5 2021-02-10T04:59:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:59 smithi085 conmon[35961]: (osd.5) 199 : cluster [DBG] 5.7 scrub ok 2021-02-10T04:59:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:59 smithi085 conmon[35961]: cluster 2021-02-10T04:58:58.558623+0000 mgr.y (mgr 2021-02-10T04:59:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:59 smithi085 conmon[35961]: .14140) 1505 : cluster [DBG] pgmap v2443: 198 pgs: 1 active+clean+scrubbing, 2 activating, 8 peering, 15 active+undersized+degraded, 20 active+undersized, 152 active+clean; 889 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 194/2196 objects degraded (8.834%) 2021-02-10T04:59:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:59 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:59:00.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:58:59 smithi085 conmon[35961]: 04:58:59.520160+0000 mon.a (mon.0) 2963 : audit [INF] from='client.? 172.21.15.85:0/4043852611' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:59:00.862 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:00 smithi164 conmon[29288]: audit 2021-02-10T04:58:59.522956+0000 2021-02-10T04:59:00.862 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:00 smithi164 conmon[29288]: mgr.y (mgr.14140) 1506 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:59:00.862 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:00 smithi164 conmon[29288]: cluster 2021-02-10T04:58:59.845068+0000 mon.a (mon.0) 2964 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:59:00.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:00 smithi164 conmon[29288]: audit 2021-02-10T04:58:59.849286+0000 mon.a (mon.0) 2965 : audit [INF] from='client.? 172.21.15.85:0/4043852611' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:59:00.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:00 smithi164 conmon[29288]: cluster 2021-02-10T04:58:59.849403+0000 mon.a (mon.0) 2966 : cluster [DBG] osdmap e1334: 8 total, 8 up, 8 in 2021-02-10T04:59:00.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:00 smithi164 conmon[29288]: cluster 2021-02-10T04:59:00.023403+0000 osd.3 (osd.3) 44 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:59:00.863 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:00 smithi164 conmon[29288]: cluster 2021-02-10T04:59:00.027873+0000 osd.3 (osd.3) 45 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:59:01.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[29319]: audit 2021-02-10T04:58:59.522956+0000 mgr.y ( 2021-02-10T04:59:01.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[29319]: mgr.14140) 1506 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:59:01.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[29319]: cluster 2021-02-10T04:58:59.845068+0000 mon.a (mon.0) 2964 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:59:01.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[29319]: audit 2021-02-10T04:58:59.849286+0000 mon.a (mon.0) 2965 : audit [INF] from='client.? 172.21.15.85:0/4043852611' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:59:01.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[29319]: cluster 2021-02-10T04:58:59.849403+0000 mon.a (mon.0) 2966 : cluster [DBG] osdmap e1334: 8 total, 8 up, 8 in 2021-02-10T04:59:01.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[29319]: cluster 2021-02-10T04:59:00.023403+0000 osd.3 (osd.3) 44 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:59:01.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[29319]: cluster 2021-02-10T04:59:00.027873+0000 osd.3 (osd.3) 45 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:59:01.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: audit 2021-02-10T04:59:01.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 2021-02-10T04:59:01.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 2021 2021-02-10T04:59:01.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: - 2021-02-10T04:59:01.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 02 2021-02-10T04:59:01.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: - 2021-02-10T04:59:01.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 10 2021-02-10T04:59:01.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: T 2021-02-10T04:59:01.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 04 2021-02-10T04:59:01.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: : 2021-02-10T04:59:01.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 58 2021-02-10T04:59:01.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: : 2021-02-10T04:59:01.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 59 2021-02-10T04:59:01.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: . 2021-02-10T04:59:01.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 522956 2021-02-10T04:59:01.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: +0000 2021-02-10T04:59:01.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 2021-02-10T04:59:01.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: mgr.y 2021-02-10T04:59:01.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: ( 2021-02-10T04:59:01.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: mgr 2021-02-10T04:59:01.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: . 2021-02-10T04:59:01.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 14140 2021-02-10T04:59:01.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: ) 2021-02-10T04:59:01.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 1506 2021-02-10T04:59:01.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: : 2021-02-10T04:59:01.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: audit 2021-02-10T04:59:01.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 2021-02-10T04:59:01.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: [DBG] 2021-02-10T04:59:01.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 2021-02-10T04:59:01.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:59:01.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 2021-02-10T04:59:01.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: cluster 2021-02-10T04:59:01.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 2021-02-10T04:59:01.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 2021 2021-02-10T04:59:01.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: - 2021-02-10T04:59:01.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 02 2021-02-10T04:59:01.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: - 2021-02-10T04:59:01.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 10 2021-02-10T04:59:01.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: T 2021-02-10T04:59:01.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 04 2021-02-10T04:59:01.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: : 2021-02-10T04:59:01.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 58 2021-02-10T04:59:01.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: : 2021-02-10T04:59:01.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 59 2021-02-10T04:59:01.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: . 2021-02-10T04:59:01.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 845068 2021-02-10T04:59:01.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: +0000 2021-02-10T04:59:01.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 2021-02-10T04:59:01.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: mon.a 2021-02-10T04:59:01.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: ( 2021-02-10T04:59:01.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: mon 2021-02-10T04:59:01.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: . 2021-02-10T04:59:01.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 0 2021-02-10T04:59:01.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: ) 2021-02-10T04:59:01.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 2964 2021-02-10T04:59:01.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: : 2021-02-10T04:59:01.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: cluster 2021-02-10T04:59:01.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 2021-02-10T04:59:01.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: [INF] 2021-02-10T04:59:01.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 2021-02-10T04:59:01.198 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:59:01.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 2021-02-10T04:59:01.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: audit 2021-02-10T04:59:01.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 2021-02-10T04:58: 2021-02-10T04:59:01.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 59.849286+0000 mon.a (mon.0) 2965 : audit [INF] from='client.? 172.21.15.85:0/4043852611' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:59:01.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:59:01.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 04:58:59.849403+0000 mon.a (mon. 2021-02-10T04:59:01.199 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 0) 2966 : cluster [DBG] osdmap e1334: 8 total, 8 up, 8 in 2021-02-10T04:59:01.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:59:01.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: -02-10T04:59:00.023403+0000 2021-02-10T04:59:01.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: osd.3 (osd.3) 44 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:59:01.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 2021-02-10T04:59:01.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: cluster 2021-02-10T04:59:01.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 2021-02-10T04:59:00.027873+0000 osd.3 (osd. 2021-02-10T04:59:01.200 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:00 smithi085 conmon[35961]: 3) 45 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:59:01.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:01 smithi085 conmon[29319]: cluster 2021-02-10T04:59:00.559175+0000 mgr.y (mgr.14140) 1507 : cluster [DBG] pgmap v2445: 198 pgs: 23 active+clean+wait, 1 active+clean+scrubbing, 2 activating, 8 peering, 164 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:59:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:01 smithi085 conmon[29319]: cluster 2021-02-10T04:59:00.853757+0000 mon.a (mon.0) 2967 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 194/2196 objects degraded (8.834%), 15 pgs degraded) 2021-02-10T04:59:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:01 smithi085 conmon[29319]: cluster 2021-02-10T04:59:00.853787+0000 mon.a (mon.0) 2968 : cluster [INF] Cluster is now healthy 2021-02-10T04:59:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:01 smithi085 conmon[29319]: cluster 2021-02-10T04:59:00.862377+0000 mon.a (mon.0) 2969 : cluster [DBG] osdmap e1335: 8 total, 8 up, 8 in 2021-02-10T04:59:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:01 smithi085 conmon[29319]: cluster 2021-02-10T04:59:01.694090+0000 osd.1 (osd.1) 513 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:59:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:01 smithi085 conmon[29319]: cluster 2021-02-10T04:59:01.695876+0000 osd.1 (osd.1) 514 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:59:01.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:01 smithi085 conmon[35961]: cluster 2021-02-10T04:59:00.559175+0000 2021-02-10T04:59:01.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:01 smithi085 conmon[35961]: mgr.y (mgr.14140) 1507 : cluster [DBG] pgmap v2445: 198 pgs: 23 active+clean+wait, 1 active+clean+scrubbing, 2 activating, 8 peering, 164 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:59:01.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:01 smithi085 conmon[35961]: cluster 2021-02-10T04:59:00.853757+0000 mon.a (mon.0) 2967 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 194/2196 objects degraded (8.834%), 15 pgs degraded) 2021-02-10T04:59:01.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:01 smithi085 conmon[35961]: cluster 2021-02-10T04:59:00.853787+0000 mon.a (mon.0) 2968 : cluster [INF] Cluster is now healthy 2021-02-10T04:59:01.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:01 smithi085 conmon[35961]: cluster 2021-02-10T04:59:00.862377+0000 2021-02-10T04:59:01.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:01 smithi085 conmon[35961]: mon.a (mon.0) 2969 : cluster [DBG] osdmap e1335: 8 total, 8 up, 8 in 2021-02-10T04:59:01.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:01 smithi085 conmon[35961]: cluster 2021-02-10T04:59: 2021-02-10T04:59:01.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:01 smithi085 conmon[35961]: 01.694090+0000 osd.1 (osd.1) 513 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:59:01.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:01 smithi085 conmon[35961]: 2021-02-10T04:59:01.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:01 smithi085 conmon[35961]: cluster 2021-02-10T04:59:01.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:01 smithi085 conmon[35961]: 2021-02-10T04:59:01.695876+0000 osd.1 (osd 2021-02-10T04:59:01.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:01 smithi085 conmon[35961]: .1) 514 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:59:01.981 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:59:02.003 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:59:02.025 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:59:02.047 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:59:02.068 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:59:02.091 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:59:02.113 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:59:02.135 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:59:02.136 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:59:02.137 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:59:02.137 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:59:02.137 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:59:02.137 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:59:02.137 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:59:02.137 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:59:02.137 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:59:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:01 smithi164 conmon[29288]: cluster 2021-02-10T04:59:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:01 smithi164 conmon[29288]: 2021-02-10T04:59:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:01 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:59:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:01 smithi164 conmon[29288]: :59:00.559175+0000 mgr.y (mgr.14140) 1507 : cluster [DBG] pgmap v2445: 198 pgs: 23 active+clean+wait, 1 active+clean+scrubbing, 2 activating, 8 peering, 164 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:59:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:01 smithi164 conmon[29288]: cluster 2021-02-10T04:59:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:01 smithi164 conmon[29288]: 2021-02-10T04:59: 2021-02-10T04:59:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:01 smithi164 conmon[29288]: 00.853757+0000 mon.a (mon.0) 2967 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 194/2196 objects degraded (8.834%), 15 pgs degraded) 2021-02-10T04:59:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:01 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:59:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:01 smithi164 conmon[29288]: :59:00.853787+0000 mon.a (mon.0) 2968 : cluster [INF] 2021-02-10T04:59:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:01 smithi164 conmon[29288]: Cluster is now healthy 2021-02-10T04:59:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:01 smithi164 conmon[29288]: cluster 2021-02-10T04:59:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:01 smithi164 conmon[29288]: 2021-02-10T04:59:00.862377+0000 mon.a ( 2021-02-10T04:59:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:01 smithi164 conmon[29288]: mon.0) 2969 : cluster [DBG] osdmap e1335: 8 total, 8 up, 8 in 2021-02-10T04:59:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:01 smithi164 conmon[29288]: cluster 2021-02-10T04:59:01 2021-02-10T04:59:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:01 smithi164 conmon[29288]: .694090+0000 osd.1 (osd.1) 513 : cluster [DBG] 2021-02-10T04:59:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:01 smithi164 conmon[29288]: purged_snaps scrub starts 2021-02-10T04:59:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:01 smithi164 conmon[29288]: cluster 2021-02-10T04:59:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:01 smithi164 conmon[29288]: 2021-02-10T04:59:01.695876+0000 osd.1 (osd.1) 514 : 2021-02-10T04:59:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:01 smithi164 conmon[29288]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:59:02.164 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2021-02-10T04:59:02.164 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:59:02.164 INFO:tasks.thrashosds.thrasher:Growing pool zone.rgw.log 2021-02-10T04:59:02.164 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:59:02.394 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:59:02.442 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 04:59:02 smithi085 conmon[45443]: debug 2021-02-10T04:59:02.383+0000 7f71214ef700 -1 osd.1 pg_epoch: 1336 pg[5.1d( empty local-lis/les=1332/1333 n=0 ec=57/57 lis/c=1332/1332 les/c/f=1333/1333/0 sis=1332) [1,4,0] r=0 lpr=1332 crt=0'0 mlcod 0'0 active+clean] scrubber pg(5.1d) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x55dbcdd2a9a0) 2021-02-10T04:59:02.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:59:02 smithi164 conmon[37263]: debug 2021-02-10T04:59:02.274+0000 7f1c679e3700 -1 osd.5 pg_epoch: 1336 pg[6.a( empty local-lis/les=1332/1333 n=0 ec=59/59 lis/c=1332/1332 les/c/f=1333/1333/0 sis=1332) [5,6,0] r=0 lpr=1332 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x5566532d58c0) 2021-02-10T04:59:03.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:02 smithi164 conmon[29288]: cluster 2021-02-10T04:59:01. 2021-02-10T04:59:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:02 smithi164 conmon[29288]: 867534+0000 mon.a (mon.0) 2970 : cluster [DBG] osdmap e1336: 8 total, 8 up, 8 in 2021-02-10T04:59:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:02 smithi164 conmon[29288]: audit 2021-02-10T04:59:01.958740+0000 mon.c ( 2021-02-10T04:59:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:02 smithi164 conmon[29288]: mon.1) 156 : audit [DBG] from='client.? 172.21.15.85:0/3630543153' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:59:03.155 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:02 smithi085 conmon[29319]: cluster 2021-02-10T04:59:01.867534+0000 mon.a (mon.0 2021-02-10T04:59:03.155 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:02 smithi085 conmon[29319]: ) 2970 : cluster [DBG] osdmap e1336: 8 total, 8 up, 8 in 2021-02-10T04:59:03.155 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:02 smithi085 conmon[29319]: audit 2021-02-10T04:59:01.958740+0000 mon.c (mon.1) 156 : audit [DBG] from='client.? 172.21.15.85:0/3630543153' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:59:03.155 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:02 smithi085 conmon[35961]: cluster 2021-02-10T04:59:01.867534+0000 2021-02-10T04:59:03.156 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:02 smithi085 conmon[35961]: mon.a (mon.0) 2970 : cluster [DBG] osdmap e1336: 8 total, 8 up, 8 in 2021-02-10T04:59:03.156 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:02 smithi085 conmon[35961]: audit 2021-02-10T04:59:03.156 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:02 smithi085 conmon[35961]: 2021-02-10T04:59:01.958740+0000 mon.c (mon.1) 2021-02-10T04:59:03.156 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:02 smithi085 conmon[35961]: 156 : audit [DBG] from='client.? 172.21.15.85:0/3630543153' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:59:03.354 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set noscrub 2021-02-10T04:59:03.583 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:59:03.872 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:03 smithi085 conmon[35961]: cluster 2021-02-10T04:59:02 2021-02-10T04:59:03.873 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:03 smithi085 conmon[35961]: .559660+0000 mgr.y (mgr.14140) 1508 : cluster [DBG] pgmap v2448: 198 pgs: 1 active+clean+scrubbing, 23 active+clean+wait, 174 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:59:03.873 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:03 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:59:03.873 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:03 smithi085 conmon[35961]: -02-10T04:59:03.627829+0000 mon.a 2021-02-10T04:59:04.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:03 smithi164 conmon[29288]: cluster 2021-02-10T04:59:02.559660+0000 mgr.y (mgr. 2021-02-10T04:59:04.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:03 smithi164 conmon[29288]: 14140) 1508 : cluster [DBG] pgmap v2448: 198 pgs: 1 active+clean+scrubbing, 23 active+clean+wait, 174 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:59:04.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:03 smithi164 conmon[29288]: cluster 2021-02-10T04:59:03.627829+0000 mon.a (mon.0) 2971 : 2021-02-10T04:59:04.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:03 smithi164 conmon[29288]: cluster [DBG] osdmap e1337: 8 total, 8 up, 8 in 2021-02-10T04:59:04.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:03 smithi085 conmon[35961]: (mon.0) 2971 : cluster [DBG] osdmap e1337: 8 total, 8 up, 8 in 2021-02-10T04:59:04.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:03 smithi085 conmon[29319]: cluster 2021-02-10T04:59:02.559660 2021-02-10T04:59:04.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:03 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 1508 : cluster [DBG] pgmap v2448: 198 pgs: 1 active+clean+scrubbing, 23 active+clean+wait, 174 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:59:04.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:03 smithi085 conmon[29319]: cluster 2021-02-10T04:59:03.627829+0000 mon.a (mon.0) 2971 : cluster [DBG] osdmap e1337: 8 total, 8 up, 8 in 2021-02-10T04:59:05.093 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:59:05.115 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:59:05.119 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:59:05.313 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:59:05.875 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:05 smithi085 conmon[29319]: cluster 2021-02-10T04:59:04.560167+0000 mgr.y (mgr.14140) 1509 : cluster [DBG] pgmap v2450: 198 pgs: 1 active+clean+scrubbing, 23 active+clean+wait, 174 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:59:05.875 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:05 smithi085 conmon[29319]: audit 2021-02-10T04:59:05.091148+0000 mgr.y (mgr.14140) 1510 : audit [DBG] from='client.16527 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:59:05.875 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:05 smithi085 conmon[35961]: cluster 2021-02-10T04:59:04.560167+0000 mgr.y (mgr.14140) 1509 : cluster [DBG] pgmap v2450: 198 pgs: 1 active+clean+scrubbing, 23 active+clean+wait, 174 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:59:05.875 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:05 smithi085 conmon[35961]: audit 2021-02-10T04:59:05.091148+0000 mgr.y (mgr.14140) 1510 : audit [DBG] from='client.16527 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:59:06.139 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":2450,"stamp":"2021-02-10T04:59:04.559810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":936255846,"num_objects":735,"num_object_clones":326,"num_object_copies":2205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":735,"num_whiteouts":10,"num_read":7650,"num_read_kb":4152645,"num_write":4560,"num_write_kb":1137191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":851,"num_bytes_recovered":289530491,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35657,"ondisk_log_size":35657,"up":594,"acting":594,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":596,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2321004,"kb_used_data":2004908,"kb_used_omap":711,"kb_used_meta":315320,"kb_avail":747640212,"statfs":{"total":767960285184,"available":765583577088,"internally_reserved":0,"allocated":2053025792,"data_stored":2610280563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":728429,"internal_metadata":322888339},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4542430,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":-419,"num_read_kb":-437669,"num_write":-319,"num_write_kb":-118798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-55,"num_bytes_recovered":-17936938,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.713257"},"pg_stats":[{"pgid":"6.29","version":"0'0","reported_seq":"1205","reported_epoch":"1333","state":"active+clean","last_fresh":"2021-02-10T04:58:56.857247+0000","last_change":"2021-02-10T04:58:56.856970+0000","last_active":"2021-02-10T04:58:56.857247+0000","last_peered":"2021-02-10T04:58:56.857247+0000","last_clean":"2021-02-10T04:58:56.857247+0000","last_became_active":"2021-02-10T04:58:56.856652+0000","last_became_peered":"2021-02-10T04:58:56.856652+0000","last_unstale":"2021-02-10T04:58:56.857247+0000","last_undegraded":"2021-02-10T04:58:56.857247+0000","last_fullsized":"2021-02-10T04:58:56.857247+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:24.121684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:24.121684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":"231","reported_epoch":"1334","state":"active+clean","last_fresh":"2021-02-10T04:58:59.848450+0000","last_change":"2021-02-10T04:52:30.133105+0000","last_active":"2021-02-10T04:58:59.848450+0000","last_peered":"2021-02-10T04:58:59.848450+0000","last_clean":"2021-02-10T04:58:59.848450+0000","last_became_active":"2021-02-10T04:51:36.864926+0000","last_became_peered":"2021-02-10T04:51:36.864926+0000","last_unstale":"2021-02-10T04:58:59.848450+0000","last_undegraded":"2021-02-10T04:58:59.848450+0000","last_fullsized":"2021-02-10T04:58:59.848450+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.133043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.133043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"1191'4","reported_seq":"347","reported_epoch":"1333","state":"active+clean","last_fresh":"2021-02-10T04:58:58.074611+0000","last_change":"2021-02-10T04:53:46.736164+0000","last_active":"2021-02-10T04:58:58.074611+0000","last_peered":"2021-02-10T04:58:58.074611+0000","last_clean":"2021-02-10T04:58:58.074611+0000","last_became_active":"2021-02-10T04:53:46.735711+0000","last_became_peered":"2021-02-10T04:53:46.735711+0000","last_unstale":"2021-02-10T04:58:58.074611+0000","last_undegraded":"2021-02-10T04:58:58.074611+0000","last_fullsized":"2021-02-10T04:58:58.074611+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'3","last_scrub_stamp":"2021-02-10T04:51:01.185820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.185820+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"1261'84","reported_seq":"1468","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.864094+0000","last_change":"2021-02-10T04:55:39.594594+0000","last_active":"2021-02-10T04:59:00.864094+0000","last_peered":"2021-02-10T04:59:00.864094+0000","last_clean":"2021-02-10T04:59:00.864094+0000","last_became_active":"2021-02-10T04:55:39.594275+0000","last_became_peered":"2021-02-10T04:55:39.594275+0000","last_unstale":"2021-02-10T04:59:00.864094+0000","last_undegraded":"2021-02-10T04:59:00.864094+0000","last_fullsized":"2021-02-10T04:59:00.864094+0000","mapping_epoch":55,"log_start":"1132'66","ondisk_log_start":"1132'66","created":55,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":0,"last_scrub":"1132'66","last_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_deep_scrub":"1132'66","last_deep_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.430399+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1616,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":186,"num_read_kb":150,"num_write":107,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"240","reported_epoch":"1334","state":"active+clean","last_fresh":"2021-02-10T04:58:59.850343+0000","last_change":"2021-02-10T04:53:41.722609+0000","last_active":"2021-02-10T04:58:59.850343+0000","last_peered":"2021-02-10T04:58:59.850343+0000","last_clean":"2021-02-10T04:58:59.850343+0000","last_became_active":"2021-02-10T04:51:36.875000+0000","last_became_peered":"2021-02-10T04:51:36.875000+0000","last_unstale":"2021-02-10T04:58:59.850343+0000","last_undegraded":"2021-02-10T04:58:59.850343+0000","last_fullsized":"2021-02-10T04:58:59.850343+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:53:41.722516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:41.722516+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"1399","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:02.273789+0000","last_change":"2021-02-10T04:59:02.274760+0000","last_active":"2021-02-10T04:59:02.273789+0000","last_peered":"2021-02-10T04:59:02.273789+0000","last_clean":"2021-02-10T04:59:02.273789+0000","last_became_active":"2021-02-10T04:58:56.875749+0000","last_became_peered":"2021-02-10T04:58:56.875749+0000","last_unstale":"2021-02-10T04:59:02.273789+0000","last_undegraded":"2021-02-10T04:59:02.273789+0000","last_fullsized":"2021-02-10T04:59:02.273789+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:48.408748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:50:48.408748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"1003'1689","reported_seq":"4755","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.861618+0000","last_change":"2021-02-10T04:53:38.170873+0000","last_active":"2021-02-10T04:59:00.861618+0000","last_peered":"2021-02-10T04:59:00.861618+0000","last_clean":"2021-02-10T04:59:00.861618+0000","last_became_active":"2021-02-10T04:52:26.030105+0000","last_became_peered":"2021-02-10T04:52:26.030105+0000","last_unstale":"2021-02-10T04:59:00.861618+0000","last_undegraded":"2021-02-10T04:59:00.861618+0000","last_fullsized":"2021-02-10T04:59:00.861618+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:53:38.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.170802+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":99361981,"num_objects":37,"num_object_clones":33,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"62'5","reported_seq":"403","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:01.897296+0000","last_change":"2021-02-10T04:54:39.615784+0000","last_active":"2021-02-10T04:59:01.897296+0000","last_peered":"2021-02-10T04:59:01.897296+0000","last_clean":"2021-02-10T04:59:01.897296+0000","last_became_active":"2021-02-10T04:54:39.560009+0000","last_became_peered":"2021-02-10T04:54:39.560009+0000","last_unstale":"2021-02-10T04:59:01.897296+0000","last_undegraded":"2021-02-10T04:59:01.897296+0000","last_fullsized":"2021-02-10T04:59:01.897296+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":414,"num_read_kb":413,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"1261'28","reported_seq":"1326","reported_epoch":"1333","state":"active+clean+wait","last_fresh":"2021-02-10T04:58:56.969563+0000","last_change":"2021-02-10T04:58:56.961239+0000","last_active":"2021-02-10T04:58:56.969563+0000","last_peered":"2021-02-10T04:58:56.969563+0000","last_clean":"2021-02-10T04:58:56.969563+0000","last_became_active":"2021-02-10T04:58:56.960018+0000","last_became_peered":"2021-02-10T04:58:56.960018+0000","last_unstale":"2021-02-10T04:58:56.969563+0000","last_undegraded":"2021-02-10T04:58:56.969563+0000","last_fullsized":"2021-02-10T04:58:56.969563+0000","mapping_epoch":1332,"log_start":"1061'20","ondisk_log_start":"1061'20","created":55,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'32","last_scrub_stamp":"2021-02-10T04:51:51.382040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.382040+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1158,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":104,"num_read_kb":80,"num_write":58,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2052,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"987","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.864669+0000","last_change":"2021-02-10T04:53:34.179015+0000","last_active":"2021-02-10T04:59:00.864669+0000","last_peered":"2021-02-10T04:59:00.864669+0000","last_clean":"2021-02-10T04:59:00.864669+0000","last_became_active":"2021-02-10T04:51:49.502576+0000","last_became_peered":"2021-02-10T04:51:49.502576+0000","last_unstale":"2021-02-10T04:59:00.864669+0000","last_undegraded":"2021-02-10T04:59:00.864669+0000","last_fullsized":"2021-02-10T04:59:00.864669+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:34.178949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.178949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"231","reported_epoch":"1334","state":"active+clean","last_fresh":"2021-02-10T04:58:59.849486+0000","last_change":"2021-02-10T04:51:53.038893+0000","last_active":"2021-02-10T04:58:59.849486+0000","last_peered":"2021-02-10T04:58:59.849486+0000","last_clean":"2021-02-10T04:58:59.849486+0000","last_became_active":"2021-02-10T04:51:36.855507+0000","last_became_peered":"2021-02-10T04:51:36.855507+0000","last_unstale":"2021-02-10T04:58:59.849486+0000","last_undegraded":"2021-02-10T04:58:59.849486+0000","last_fullsized":"2021-02-10T04:58:59.849486+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:53.038834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:53.038834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"1248","reported_epoch":"1333","state":"active+clean+wait","last_fresh":"2021-02-10T04:58:57.081276+0000","last_change":"2021-02-10T04:58:57.081222+0000","last_active":"2021-02-10T04:58:57.081276+0000","last_peered":"2021-02-10T04:58:57.081276+0000","last_clean":"2021-02-10T04:58:57.081276+0000","last_became_active":"2021-02-10T04:58:56.865590+0000","last_became_peered":"2021-02-10T04:58:56.865590+0000","last_unstale":"2021-02-10T04:58:57.081276+0000","last_undegraded":"2021-02-10T04:58:57.081276+0000","last_fullsized":"2021-02-10T04:58:57.081276+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:58:57.081143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:58:57.081143+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"1168'82","reported_seq":"267","reported_epoch":"1334","state":"active+clean","last_fresh":"2021-02-10T04:58:59.848724+0000","last_change":"2021-02-10T04:53:23.326363+0000","last_active":"2021-02-10T04:58:59.848724+0000","last_peered":"2021-02-10T04:58:59.848724+0000","last_clean":"2021-02-10T04:58:59.848724+0000","last_became_active":"2021-02-10T04:51:36.859215+0000","last_became_peered":"2021-02-10T04:51:36.859215+0000","last_unstale":"2021-02-10T04:58:59.848724+0000","last_undegraded":"2021-02-10T04:58:59.848724+0000","last_fullsized":"2021-02-10T04:58:59.848724+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1150'76","last_scrub_stamp":"2021-02-10T04:53:00.964499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:00.964499+0000","log_size":82,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3848803,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"1261'69","reported_seq":"1476","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:01.872251+0000","last_change":"2021-02-10T04:55:25.576927+0000","last_active":"2021-02-10T04:59:01.872251+0000","last_peered":"2021-02-10T04:59:01.872251+0000","last_clean":"2021-02-10T04:59:01.872251+0000","last_became_active":"2021-02-10T04:55:25.576633+0000","last_became_peered":"2021-02-10T04:55:25.576633+0000","last_unstale":"2021-02-10T04:59:01.872251+0000","last_undegraded":"2021-02-10T04:59:01.872251+0000","last_fullsized":"2021-02-10T04:59:01.872251+0000","mapping_epoch":55,"log_start":"1112'47","ondisk_log_start":"1112'47","created":55,"last_epoch_clean":1237,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'61","last_scrub_stamp":"2021-02-10T04:53:36.836202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:36.836202+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1466,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":208,"num_read_kb":172,"num_write":122,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"1334'8","reported_seq":"357","reported_epoch":"1334","state":"active+clean","last_fresh":"2021-02-10T04:59:00.520258+0000","last_change":"2021-02-10T04:58:56.880162+0000","last_active":"2021-02-10T04:59:00.520258+0000","last_peered":"2021-02-10T04:59:00.520258+0000","last_clean":"2021-02-10T04:59:00.520258+0000","last_became_active":"2021-02-10T04:58:56.879760+0000","last_became_peered":"2021-02-10T04:58:56.879760+0000","last_unstale":"2021-02-10T04:59:00.520258+0000","last_undegraded":"2021-02-10T04:59:00.520258+0000","last_fullsized":"2021-02-10T04:59:00.520258+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'4","last_scrub_stamp":"2021-02-10T04:52:01.121380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.121380+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"261","reported_epoch":"1334","state":"active+clean","last_fresh":"2021-02-10T04:58:59.850870+0000","last_change":"2021-02-10T04:58:56.964109+0000","last_active":"2021-02-10T04:58:59.850870+0000","last_peered":"2021-02-10T04:58:59.850870+0000","last_clean":"2021-02-10T04:58:59.850870+0000","last_became_active":"2021-02-10T04:58:56.963768+0000","last_became_peered":"2021-02-10T04:58:56.963768+0000","last_unstale":"2021-02-10T04:58:59.850870+0000","last_undegraded":"2021-02-10T04:58:59.850870+0000","last_fullsized":"2021-02-10T04:58:59.850870+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:55:22.573621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:55:22.573621+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"630","reported_epoch":"1333","state":"active+clean","last_fresh":"2021-02-10T04:58:56.828333+0000","last_change":"2021-02-10T04:53:46.733236+0000","last_active":"2021-02-10T04:58:56.828333+0000","last_peered":"2021-02-10T04:58:56.828333+0000","last_clean":"2021-02-10T04:58:56.828333+0000","last_became_active":"2021-02-10T04:53:46.732890+0000","last_became_peered":"2021-02-10T04:53:46.732890+0000","last_unstale":"2021-02-10T04:58:56.828333+0000","last_undegraded":"2021-02-10T04:58:56.828333+0000","last_fullsized":"2021-02-10T04:58:56.828333+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:35.170157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:35.170157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"1003'739","reported_seq":"3922","reported_epoch":"1333","state":"active+clean+wait","last_fresh":"2021-02-10T04:58:56.962072+0000","last_change":"2021-02-10T04:58:56.959493+0000","last_active":"2021-02-10T04:58:56.962072+0000","last_peered":"2021-02-10T04:58:56.962072+0000","last_clean":"2021-02-10T04:58:56.962072+0000","last_became_active":"2021-02-10T04:58:56.956888+0000","last_became_peered":"2021-02-10T04:58:56.956888+0000","last_unstale":"2021-02-10T04:58:56.962072+0000","last_undegraded":"2021-02-10T04:58:56.962072+0000","last_fullsized":"2021-02-10T04:58:56.962072+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:52:27.602006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:27.602006+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24059888,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":129,"num_read_kb":109544,"num_write":102,"num_write_kb":46173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"1313","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:01.871698+0000","last_change":"2021-02-10T04:54:39.559980+0000","last_active":"2021-02-10T04:59:01.871698+0000","last_peered":"2021-02-10T04:59:01.871698+0000","last_clean":"2021-02-10T04:59:01.871698+0000","last_became_active":"2021-02-10T04:54:39.559443+0000","last_became_peered":"2021-02-10T04:54:39.559443+0000","last_unstale":"2021-02-10T04:59:01.871698+0000","last_undegraded":"2021-02-10T04:59:01.871698+0000","last_fullsized":"2021-02-10T04:59:01.871698+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"1261'64","reported_seq":"1520","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.863796+0000","last_change":"2021-02-10T04:53:37.108747+0000","last_active":"2021-02-10T04:59:00.863796+0000","last_peered":"2021-02-10T04:59:00.863796+0000","last_clean":"2021-02-10T04:59:00.863796+0000","last_became_active":"2021-02-10T04:53:35.551311+0000","last_became_peered":"2021-02-10T04:53:35.551311+0000","last_unstale":"2021-02-10T04:59:00.863796+0000","last_undegraded":"2021-02-10T04:59:00.863796+0000","last_fullsized":"2021-02-10T04:59:00.863796+0000","mapping_epoch":1118,"log_start":"1066'44","ondisk_log_start":"1066'44","created":55,"last_epoch_clean":1181,"parent":"0.0","parent_split_bits":0,"last_scrub":"1066'44","last_scrub_stamp":"2021-02-10T04:53:37.108677+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:53:37.108677+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1578,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":180,"num_read_kb":144,"num_write":106,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":2312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"1230","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.864069+0000","last_change":"2021-02-10T04:53:39.126356+0000","last_active":"2021-02-10T04:59:00.864069+0000","last_peered":"2021-02-10T04:59:00.864069+0000","last_clean":"2021-02-10T04:59:00.864069+0000","last_became_active":"2021-02-10T04:29:07.145616+0000","last_became_peered":"2021-02-10T04:29:07.145616+0000","last_unstale":"2021-02-10T04:59:00.864069+0000","last_undegraded":"2021-02-10T04:59:00.864069+0000","last_fullsized":"2021-02-10T04:59:00.864069+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:39.126261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:39.126261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"577","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.863102+0000","last_change":"2021-02-10T04:58:56.868713+0000","last_active":"2021-02-10T04:59:00.863102+0000","last_peered":"2021-02-10T04:59:00.863102+0000","last_clean":"2021-02-10T04:59:00.863102+0000","last_became_active":"2021-02-10T04:58:56.868368+0000","last_became_peered":"2021-02-10T04:58:56.868368+0000","last_unstale":"2021-02-10T04:59:00.863102+0000","last_undegraded":"2021-02-10T04:59:00.863102+0000","last_fullsized":"2021-02-10T04:59:00.863102+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:26.311958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.311958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"277","reported_epoch":"1334","state":"active+clean","last_fresh":"2021-02-10T04:58:59.850797+0000","last_change":"2021-02-10T04:58:39.423097+0000","last_active":"2021-02-10T04:58:59.850797+0000","last_peered":"2021-02-10T04:58:59.850797+0000","last_clean":"2021-02-10T04:58:59.850797+0000","last_became_active":"2021-02-10T04:51:36.855744+0000","last_became_peered":"2021-02-10T04:51:36.855744+0000","last_unstale":"2021-02-10T04:58:59.850797+0000","last_undegraded":"2021-02-10T04:58:59.850797+0000","last_fullsized":"2021-02-10T04:58:59.850797+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:02.162851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:13.727241+0000","last_clean_scrub_stamp":"2021-02-10T04:52:02.162851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"1063'654","reported_seq":"4525","reported_epoch":"1334","state":"active+clean","last_fresh":"2021-02-10T04:58:59.850252+0000","last_change":"2021-02-10T04:58:56.966357+0000","last_active":"2021-02-10T04:58:59.850252+0000","last_peered":"2021-02-10T04:58:59.850252+0000","last_clean":"2021-02-10T04:58:59.850252+0000","last_became_active":"2021-02-10T04:58:56.965471+0000","last_became_peered":"2021-02-10T04:58:56.965471+0000","last_unstale":"2021-02-10T04:58:59.850252+0000","last_undegraded":"2021-02-10T04:58:59.850252+0000","last_fullsized":"2021-02-10T04:58:59.850252+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:51:57.121838+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.121838+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24700630,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":171,"num_read_kb":182272,"num_write":111,"num_write_kb":33727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":26349023,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"1281","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.864189+0000","last_change":"2021-02-10T04:51:50.433056+0000","last_active":"2021-02-10T04:59:00.864189+0000","last_peered":"2021-02-10T04:59:00.864189+0000","last_clean":"2021-02-10T04:59:00.864189+0000","last_became_active":"2021-02-10T04:51:36.843165+0000","last_became_peered":"2021-02-10T04:51:36.843165+0000","last_unstale":"2021-02-10T04:59:00.864189+0000","last_undegraded":"2021-02-10T04:59:00.864189+0000","last_fullsized":"2021-02-10T04:59:00.864189+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.432990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.432990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"1317","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.573641+0000","last_change":"2021-02-10T04:58:56.965752+0000","last_active":"2021-02-10T04:59:01.573641+0000","last_peered":"2021-02-10T04:59:01.573641+0000","last_clean":"2021-02-10T04:59:01.573641+0000","last_became_active":"2021-02-10T04:58:56.963934+0000","last_became_peered":"2021-02-10T04:58:56.963934+0000","last_unstale":"2021-02-10T04:59:01.573641+0000","last_undegraded":"2021-02-10T04:59:01.573641+0000","last_fullsized":"2021-02-10T04:59:01.573641+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"1225","reported_epoch":"1333","state":"active+clean","last_fresh":"2021-02-10T04:58:56.830812+0000","last_change":"2021-02-10T04:53:46.727888+0000","last_active":"2021-02-10T04:58:56.830812+0000","last_peered":"2021-02-10T04:58:56.830812+0000","last_clean":"2021-02-10T04:58:56.830812+0000","last_became_active":"2021-02-10T04:53:46.727670+0000","last_became_peered":"2021-02-10T04:53:46.727670+0000","last_unstale":"2021-02-10T04:58:56.830812+0000","last_undegraded":"2021-02-10T04:58:56.830812+0000","last_fullsized":"2021-02-10T04:58:56.830812+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:41.041553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.041553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"1261'76","reported_seq":"1444","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.862480+0000","last_change":"2021-02-10T04:53:42.071173+0000","last_active":"2021-02-10T04:59:00.862480+0000","last_peered":"2021-02-10T04:59:00.862480+0000","last_clean":"2021-02-10T04:59:00.862480+0000","last_became_active":"2021-02-10T04:53:39.679752+0000","last_became_peered":"2021-02-10T04:53:39.679752+0000","last_unstale":"2021-02-10T04:59:00.862480+0000","last_undegraded":"2021-02-10T04:59:00.862480+0000","last_fullsized":"2021-02-10T04:59:00.862480+0000","mapping_epoch":1111,"log_start":"1049'39","ondisk_log_start":"1049'39","created":55,"last_epoch_clean":1185,"parent":"0.0","parent_split_bits":0,"last_scrub":"1187'54","last_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_deep_scrub":"1187'54","last_deep_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_clean_scrub_stamp":"2021-02-10T04:53:42.071103+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":147,"num_read_kb":127,"num_write":95,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"972","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.862302+0000","last_change":"2021-02-10T04:51:52.350764+0000","last_active":"2021-02-10T04:59:00.862302+0000","last_peered":"2021-02-10T04:59:00.862302+0000","last_clean":"2021-02-10T04:59:00.862302+0000","last_became_active":"2021-02-10T04:51:49.567784+0000","last_became_peered":"2021-02-10T04:51:49.567784+0000","last_unstale":"2021-02-10T04:59:00.862302+0000","last_undegraded":"2021-02-10T04:59:00.862302+0000","last_fullsized":"2021-02-10T04:59:00.862302+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:52.350706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.350706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"1119'4","reported_seq":"325","reported_epoch":"1334","state":"active+clean","last_fresh":"2021-02-10T04:58:59.849630+0000","last_change":"2021-02-10T04:52:27.126670+0000","last_active":"2021-02-10T04:58:59.849630+0000","last_peered":"2021-02-10T04:58:59.849630+0000","last_clean":"2021-02-10T04:58:59.849630+0000","last_became_active":"2021-02-10T04:51:36.853647+0000","last_became_peered":"2021-02-10T04:51:36.853647+0000","last_unstale":"2021-02-10T04:58:59.849630+0000","last_undegraded":"2021-02-10T04:58:59.849630+0000","last_fullsized":"2021-02-10T04:58:59.849630+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'4","last_scrub_stamp":"2021-02-10T04:52:27.126599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:27.126599+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"1321","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.863120+0000","last_change":"2021-02-10T04:52:25.745660+0000","last_active":"2021-02-10T04:59:00.863120+0000","last_peered":"2021-02-10T04:59:00.863120+0000","last_clean":"2021-02-10T04:59:00.863120+0000","last_became_active":"2021-02-10T04:51:36.849503+0000","last_became_peered":"2021-02-10T04:51:36.849503+0000","last_unstale":"2021-02-10T04:59:00.863120+0000","last_undegraded":"2021-02-10T04:59:00.863120+0000","last_fullsized":"2021-02-10T04:59:00.863120+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:25.745577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.745577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"1004'1093","reported_seq":"6505","reported_epoch":"1333","state":"active+clean+wait","last_fresh":"2021-02-10T04:58:56.978129+0000","last_change":"2021-02-10T04:58:56.877548+0000","last_active":"2021-02-10T04:58:56.978129+0000","last_peered":"2021-02-10T04:58:56.978129+0000","last_clean":"2021-02-10T04:58:56.978129+0000","last_became_active":"2021-02-10T04:58:56.875893+0000","last_became_peered":"2021-02-10T04:58:56.875893+0000","last_unstale":"2021-02-10T04:58:56.978129+0000","last_undegraded":"2021-02-10T04:58:56.978129+0000","last_fullsized":"2021-02-10T04:58:56.978129+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39434696,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":279,"num_read_kb":311058,"num_write":185,"num_write_kb":66790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"62'5","reported_seq":"134","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.572420+0000","last_change":"2021-02-10T04:54:39.559473+0000","last_active":"2021-02-10T04:59:01.572420+0000","last_peered":"2021-02-10T04:59:01.572420+0000","last_clean":"2021-02-10T04:59:01.572420+0000","last_became_active":"2021-02-10T04:54:39.559048+0000","last_became_peered":"2021-02-10T04:54:39.559048+0000","last_unstale":"2021-02-10T04:59:01.572420+0000","last_undegraded":"2021-02-10T04:59:01.572420+0000","last_fullsized":"2021-02-10T04:59:01.572420+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":149,"num_read_kb":148,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"1306","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.864171+0000","last_change":"2021-02-10T04:58:56.245075+0000","last_active":"2021-02-10T04:59:00.864171+0000","last_peered":"2021-02-10T04:59:00.864171+0000","last_clean":"2021-02-10T04:59:00.864171+0000","last_became_active":"2021-02-10T04:51:36.841582+0000","last_became_peered":"2021-02-10T04:51:36.841582+0000","last_unstale":"2021-02-10T04:59:00.864171+0000","last_undegraded":"2021-02-10T04:59:00.864171+0000","last_fullsized":"2021-02-10T04:59:00.864171+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:12.389272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:12.389272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"1261'71","reported_seq":"1038","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.863883+0000","last_change":"2021-02-10T04:53:27.560299+0000","last_active":"2021-02-10T04:59:00.863883+0000","last_peered":"2021-02-10T04:59:00.863883+0000","last_clean":"2021-02-10T04:59:00.863883+0000","last_became_active":"2021-02-10T04:53:27.559886+0000","last_became_peered":"2021-02-10T04:53:27.559886+0000","last_unstale":"2021-02-10T04:59:00.863883+0000","last_undegraded":"2021-02-10T04:59:00.863883+0000","last_fullsized":"2021-02-10T04:59:00.863883+0000","mapping_epoch":1129,"log_start":"1049'51","ondisk_log_start":"1049'51","created":55,"last_epoch_clean":1173,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'41","last_scrub_stamp":"2021-02-10T04:51:58.353660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:58.353660+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1472,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":169,"num_read_kb":136,"num_write":102,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"995","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.862829+0000","last_change":"2021-02-10T04:53:11.271557+0000","last_active":"2021-02-10T04:59:00.862829+0000","last_peered":"2021-02-10T04:59:00.862829+0000","last_clean":"2021-02-10T04:59:00.862829+0000","last_became_active":"2021-02-10T04:51:49.567883+0000","last_became_peered":"2021-02-10T04:51:49.567883+0000","last_unstale":"2021-02-10T04:59:00.862829+0000","last_undegraded":"2021-02-10T04:59:00.862829+0000","last_fullsized":"2021-02-10T04:59:00.862829+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:11.271459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.271459+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"1333'7","reported_seq":"1738","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.863047+0000","last_change":"2021-02-10T04:58:56.875943+0000","last_active":"2021-02-10T04:59:00.863047+0000","last_peered":"2021-02-10T04:59:00.863047+0000","last_clean":"2021-02-10T04:59:00.863047+0000","last_became_active":"2021-02-10T04:58:56.875512+0000","last_became_peered":"2021-02-10T04:58:56.875512+0000","last_unstale":"2021-02-10T04:59:00.863047+0000","last_undegraded":"2021-02-10T04:59:00.863047+0000","last_fullsized":"2021-02-10T04:59:00.863047+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:51:47.855283+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.855283+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"1340","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.872432+0000","last_change":"2021-02-10T04:58:56.854797+0000","last_active":"2021-02-10T04:59:00.872432+0000","last_peered":"2021-02-10T04:59:00.872432+0000","last_clean":"2021-02-10T04:59:00.872432+0000","last_became_active":"2021-02-10T04:58:56.854440+0000","last_became_peered":"2021-02-10T04:58:56.854440+0000","last_unstale":"2021-02-10T04:59:00.872432+0000","last_undegraded":"2021-02-10T04:59:00.872432+0000","last_fullsized":"2021-02-10T04:59:00.872432+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:55:22.183566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:55:22.183566+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"1002'1090","reported_seq":"5530","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:01.873865+0000","last_change":"2021-02-10T04:52:26.030198+0000","last_active":"2021-02-10T04:59:01.873865+0000","last_peered":"2021-02-10T04:59:01.873865+0000","last_clean":"2021-02-10T04:59:01.873865+0000","last_became_active":"2021-02-10T04:52:26.028182+0000","last_became_peered":"2021-02-10T04:52:26.028182+0000","last_unstale":"2021-02-10T04:59:01.873865+0000","last_undegraded":"2021-02-10T04:59:01.873865+0000","last_fullsized":"2021-02-10T04:59:01.873865+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:51:08.220472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.220472+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24410646,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":245,"num_read_kb":276168,"num_write":175,"num_write_kb":64960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"1301","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.860932+0000","last_change":"2021-02-10T04:54:39.555379+0000","last_active":"2021-02-10T04:59:00.860932+0000","last_peered":"2021-02-10T04:59:00.860932+0000","last_clean":"2021-02-10T04:59:00.860932+0000","last_became_active":"2021-02-10T04:54:39.554976+0000","last_became_peered":"2021-02-10T04:54:39.554976+0000","last_unstale":"2021-02-10T04:59:00.860932+0000","last_undegraded":"2021-02-10T04:59:00.860932+0000","last_fullsized":"2021-02-10T04:59:00.860932+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"1261'65","reported_seq":"1433","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.862823+0000","last_change":"2021-02-10T04:53:31.624862+0000","last_active":"2021-02-10T04:59:00.862823+0000","last_peered":"2021-02-10T04:59:00.862823+0000","last_clean":"2021-02-10T04:59:00.862823+0000","last_became_active":"2021-02-10T04:53:31.624318+0000","last_became_peered":"2021-02-10T04:53:31.624318+0000","last_unstale":"2021-02-10T04:59:00.862823+0000","last_undegraded":"2021-02-10T04:59:00.862823+0000","last_fullsized":"2021-02-10T04:59:00.862823+0000","mapping_epoch":55,"log_start":"1049'47","ondisk_log_start":"1049'47","created":55,"last_epoch_clean":1177,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'47","last_scrub_stamp":"2021-02-10T04:53:09.002368+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.002368+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1672,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":169,"num_read_kb":135,"num_write":98,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":934,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"285","reported_epoch":"1333","state":"active+clean","last_fresh":"2021-02-10T04:58:56.842841+0000","last_change":"2021-02-10T04:58:56.842228+0000","last_active":"2021-02-10T04:58:56.842841+0000","last_peered":"2021-02-10T04:58:56.842841+0000","last_clean":"2021-02-10T04:58:56.842841+0000","last_became_active":"2021-02-10T04:58:56.841107+0000","last_became_peered":"2021-02-10T04:58:56.841107+0000","last_unstale":"2021-02-10T04:58:56.842841+0000","last_undegraded":"2021-02-10T04:58:56.842841+0000","last_fullsized":"2021-02-10T04:58:56.842841+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:08.358300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:50:08.358300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"1312","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.873427+0000","last_change":"2021-02-10T04:51:57.436961+0000","last_active":"2021-02-10T04:59:00.873427+0000","last_peered":"2021-02-10T04:59:00.873427+0000","last_clean":"2021-02-10T04:59:00.873427+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:59:00.873427+0000","last_undegraded":"2021-02-10T04:59:00.873427+0000","last_fullsized":"2021-02-10T04:59:00.873427+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.436862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.436862+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"1267","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.574608+0000","last_change":"2021-02-10T04:53:02.112612+0000","last_active":"2021-02-10T04:59:01.574608+0000","last_peered":"2021-02-10T04:59:01.574608+0000","last_clean":"2021-02-10T04:59:01.574608+0000","last_became_active":"2021-02-10T04:50:59.545501+0000","last_became_peered":"2021-02-10T04:50:59.545501+0000","last_unstale":"2021-02-10T04:59:01.574608+0000","last_undegraded":"2021-02-10T04:59:01.574608+0000","last_fullsized":"2021-02-10T04:59:01.574608+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:02.112533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:02.112533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"1001'1165","reported_seq":"4524","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:01.874380+0000","last_change":"2021-02-10T04:58:56.967168+0000","last_active":"2021-02-10T04:59:01.874380+0000","last_peered":"2021-02-10T04:59:01.874380+0000","last_clean":"2021-02-10T04:59:01.874380+0000","last_became_active":"2021-02-10T04:58:56.965483+0000","last_became_peered":"2021-02-10T04:58:56.965483+0000","last_unstale":"2021-02-10T04:59:01.874380+0000","last_undegraded":"2021-02-10T04:59:01.874380+0000","last_fullsized":"2021-02-10T04:59:01.874380+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29601845,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":286,"num_read_kb":299529,"num_write":200,"num_write_kb":89130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"107","reported_epoch":"1333","state":"active+clean+wait","last_fresh":"2021-02-10T04:58:56.963696+0000","last_change":"2021-02-10T04:58:56.961957+0000","last_active":"2021-02-10T04:58:56.963696+0000","last_peered":"2021-02-10T04:58:56.963696+0000","last_clean":"2021-02-10T04:58:56.963696+0000","last_became_active":"2021-02-10T04:58:56.956556+0000","last_became_peered":"2021-02-10T04:58:56.956556+0000","last_unstale":"2021-02-10T04:58:56.963696+0000","last_undegraded":"2021-02-10T04:58:56.963696+0000","last_fullsized":"2021-02-10T04:58:56.963696+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"1113'5","reported_seq":"1608","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.572671+0000","last_change":"2021-02-10T04:52:34.047474+0000","last_active":"2021-02-10T04:59:01.572671+0000","last_peered":"2021-02-10T04:59:01.572671+0000","last_clean":"2021-02-10T04:59:01.572671+0000","last_became_active":"2021-02-10T04:50:59.569578+0000","last_became_peered":"2021-02-10T04:50:59.569578+0000","last_unstale":"2021-02-10T04:59:01.572671+0000","last_undegraded":"2021-02-10T04:59:01.572671+0000","last_fullsized":"2021-02-10T04:59:01.572671+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1113'5","last_scrub_stamp":"2021-02-10T04:52:34.047358+0000","last_deep_scrub":"158'3","last_deep_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_clean_scrub_stamp":"2021-02-10T04:52:34.047358+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"1261'68","reported_seq":"1418","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.862409+0000","last_change":"2021-02-10T04:51:48.872937+0000","last_active":"2021-02-10T04:59:00.862409+0000","last_peered":"2021-02-10T04:59:00.862409+0000","last_clean":"2021-02-10T04:59:00.862409+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:59:00.862409+0000","last_undegraded":"2021-02-10T04:59:00.862409+0000","last_fullsized":"2021-02-10T04:59:00.862409+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:51:48.872839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.872839+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":114,"num_read_kb":94,"num_write":68,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"1387","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:01.875395+0000","last_change":"2021-02-10T04:53:05.984747+0000","last_active":"2021-02-10T04:59:01.875395+0000","last_peered":"2021-02-10T04:59:01.875395+0000","last_clean":"2021-02-10T04:59:01.875395+0000","last_became_active":"2021-02-10T04:51:36.869931+0000","last_became_peered":"2021-02-10T04:51:36.869931+0000","last_unstale":"2021-02-10T04:59:01.875395+0000","last_undegraded":"2021-02-10T04:59:01.875395+0000","last_fullsized":"2021-02-10T04:59:01.875395+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:05.984659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:05.984659+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"1187","reported_epoch":"1333","state":"active+clean","last_fresh":"2021-02-10T04:58:56.825519+0000","last_change":"2021-02-10T04:53:46.731709+0000","last_active":"2021-02-10T04:58:56.825519+0000","last_peered":"2021-02-10T04:58:56.825519+0000","last_clean":"2021-02-10T04:58:56.825519+0000","last_became_active":"2021-02-10T04:53:46.731340+0000","last_became_peered":"2021-02-10T04:53:46.731340+0000","last_unstale":"2021-02-10T04:58:56.825519+0000","last_undegraded":"2021-02-10T04:58:56.825519+0000","last_fullsized":"2021-02-10T04:58:56.825519+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:09.005421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.005421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"998'1009","reported_seq":"4826","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.574183+0000","last_change":"2021-02-10T04:52:26.037227+0000","last_active":"2021-02-10T04:59:01.574183+0000","last_peered":"2021-02-10T04:59:01.574183+0000","last_clean":"2021-02-10T04:59:01.574183+0000","last_became_active":"2021-02-10T04:52:26.036723+0000","last_became_peered":"2021-02-10T04:52:26.036723+0000","last_unstale":"2021-02-10T04:59:01.574183+0000","last_undegraded":"2021-02-10T04:59:01.574183+0000","last_fullsized":"2021-02-10T04:59:01.574183+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12355206,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":119,"num_read_kb":124701,"num_write":77,"num_write_kb":25052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4145029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"1206","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.574812+0000","last_change":"2021-02-10T04:53:33.305896+0000","last_active":"2021-02-10T04:59:01.574812+0000","last_peered":"2021-02-10T04:59:01.574812+0000","last_clean":"2021-02-10T04:59:01.574812+0000","last_became_active":"2021-02-10T04:50:59.570854+0000","last_became_peered":"2021-02-10T04:50:59.570854+0000","last_unstale":"2021-02-10T04:59:01.574812+0000","last_undegraded":"2021-02-10T04:59:01.574812+0000","last_fullsized":"2021-02-10T04:59:01.574812+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:33.305827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:33.305827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"1322","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.574727+0000","last_change":"2021-02-10T04:58:56.979110+0000","last_active":"2021-02-10T04:59:01.574727+0000","last_peered":"2021-02-10T04:59:01.574727+0000","last_clean":"2021-02-10T04:59:01.574727+0000","last_became_active":"2021-02-10T04:58:56.978745+0000","last_became_peered":"2021-02-10T04:58:56.978745+0000","last_unstale":"2021-02-10T04:59:01.574727+0000","last_undegraded":"2021-02-10T04:59:01.574727+0000","last_fullsized":"2021-02-10T04:59:01.574727+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"1320","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.862686+0000","last_change":"2021-02-10T04:51:50.895471+0000","last_active":"2021-02-10T04:59:00.862686+0000","last_peered":"2021-02-10T04:59:00.862686+0000","last_clean":"2021-02-10T04:59:00.862686+0000","last_became_active":"2021-02-10T04:50:59.527553+0000","last_became_peered":"2021-02-10T04:50:59.527553+0000","last_unstale":"2021-02-10T04:59:00.862686+0000","last_undegraded":"2021-02-10T04:59:00.862686+0000","last_fullsized":"2021-02-10T04:59:00.862686+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.895370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.895370+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"1261'84","reported_seq":"1466","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.573074+0000","last_change":"2021-02-10T04:58:56.880156+0000","last_active":"2021-02-10T04:59:01.573074+0000","last_peered":"2021-02-10T04:59:01.573074+0000","last_clean":"2021-02-10T04:59:01.573074+0000","last_became_active":"2021-02-10T04:58:56.879837+0000","last_became_peered":"2021-02-10T04:58:56.879837+0000","last_unstale":"2021-02-10T04:59:01.573074+0000","last_undegraded":"2021-02-10T04:59:01.573074+0000","last_fullsized":"2021-02-10T04:59:01.573074+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'72","last_scrub_stamp":"2021-02-10T04:46:30.515026+0000","last_deep_scrub":"152'48","last_deep_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_clean_scrub_stamp":"2021-02-10T04:46:30.515026+0000","log_size":84,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":123,"num_read_kb":101,"num_write":72,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"601","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.865265+0000","last_change":"2021-02-10T04:58:38.400978+0000","last_active":"2021-02-10T04:59:00.865265+0000","last_peered":"2021-02-10T04:59:00.865265+0000","last_clean":"2021-02-10T04:59:00.865265+0000","last_became_active":"2021-02-10T04:51:49.503801+0000","last_became_peered":"2021-02-10T04:51:49.503801+0000","last_unstale":"2021-02-10T04:59:00.865265+0000","last_undegraded":"2021-02-10T04:59:00.865265+0000","last_fullsized":"2021-02-10T04:59:00.865265+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:51:48.274960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.274960+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"1345","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.862195+0000","last_change":"2021-02-10T04:53:13.099288+0000","last_active":"2021-02-10T04:59:00.862195+0000","last_peered":"2021-02-10T04:59:00.862195+0000","last_clean":"2021-02-10T04:59:00.862195+0000","last_became_active":"2021-02-10T04:51:36.848614+0000","last_became_peered":"2021-02-10T04:51:36.848614+0000","last_unstale":"2021-02-10T04:59:00.862195+0000","last_undegraded":"2021-02-10T04:59:00.862195+0000","last_fullsized":"2021-02-10T04:59:00.862195+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:13.099228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:13.099228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"1002'1657","reported_seq":"5396","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.573139+0000","last_change":"2021-02-10T04:52:26.036806+0000","last_active":"2021-02-10T04:59:01.573139+0000","last_peered":"2021-02-10T04:59:01.573139+0000","last_clean":"2021-02-10T04:59:01.573139+0000","last_became_active":"2021-02-10T04:52:26.032363+0000","last_became_peered":"2021-02-10T04:52:26.032363+0000","last_unstale":"2021-02-10T04:59:01.573139+0000","last_undegraded":"2021-02-10T04:59:01.573139+0000","last_fullsized":"2021-02-10T04:59:01.573139+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21396533,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":212,"num_read_kb":218836,"num_write":159,"num_write_kb":59398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":22326014,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"1264","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.863312+0000","last_change":"2021-02-10T04:51:51.911945+0000","last_active":"2021-02-10T04:59:00.863312+0000","last_peered":"2021-02-10T04:59:00.863312+0000","last_clean":"2021-02-10T04:59:00.863312+0000","last_became_active":"2021-02-10T04:51:36.850015+0000","last_became_peered":"2021-02-10T04:51:36.850015+0000","last_unstale":"2021-02-10T04:59:00.863312+0000","last_undegraded":"2021-02-10T04:59:00.863312+0000","last_fullsized":"2021-02-10T04:59:00.863312+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.911851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.911851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"62'5","reported_seq":"1168","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.862618+0000","last_change":"2021-02-10T04:54:39.568253+0000","last_active":"2021-02-10T04:59:00.862618+0000","last_peered":"2021-02-10T04:59:00.862618+0000","last_clean":"2021-02-10T04:59:00.862618+0000","last_became_active":"2021-02-10T04:54:39.567642+0000","last_became_peered":"2021-02-10T04:54:39.567642+0000","last_unstale":"2021-02-10T04:59:00.862618+0000","last_undegraded":"2021-02-10T04:59:00.862618+0000","last_fullsized":"2021-02-10T04:59:00.862618+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":98,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":150,"num_read_kb":148,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"1409","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:01.872427+0000","last_change":"2021-02-10T04:58:56.957964+0000","last_active":"2021-02-10T04:59:01.872427+0000","last_peered":"2021-02-10T04:59:01.872427+0000","last_clean":"2021-02-10T04:59:01.872427+0000","last_became_active":"2021-02-10T04:58:56.957624+0000","last_became_peered":"2021-02-10T04:58:56.957624+0000","last_unstale":"2021-02-10T04:59:01.872427+0000","last_undegraded":"2021-02-10T04:59:01.872427+0000","last_fullsized":"2021-02-10T04:59:01.872427+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:58.142526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:58.142526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"1261'68","reported_seq":"1325","reported_epoch":"1333","state":"active+clean+wait","last_fresh":"2021-02-10T04:58:56.969533+0000","last_change":"2021-02-10T04:58:56.958881+0000","last_active":"2021-02-10T04:58:56.969533+0000","last_peered":"2021-02-10T04:58:56.969533+0000","last_clean":"2021-02-10T04:58:56.969533+0000","last_became_active":"2021-02-10T04:58:56.954019+0000","last_became_peered":"2021-02-10T04:58:56.954019+0000","last_unstale":"2021-02-10T04:58:56.969533+0000","last_undegraded":"2021-02-10T04:58:56.969533+0000","last_fullsized":"2021-02-10T04:58:56.969533+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1135'58","last_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_deep_scrub":"1135'58","last_deep_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.483067+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":101,"num_read_kb":82,"num_write":60,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"283","reported_epoch":"1333","state":"active+clean","last_fresh":"2021-02-10T04:58:56.850705+0000","last_change":"2021-02-10T04:58:56.850446+0000","last_active":"2021-02-10T04:58:56.850705+0000","last_peered":"2021-02-10T04:58:56.850705+0000","last_clean":"2021-02-10T04:58:56.850705+0000","last_became_active":"2021-02-10T04:58:56.849868+0000","last_became_peered":"2021-02-10T04:58:56.849868+0000","last_unstale":"2021-02-10T04:58:56.850705+0000","last_undegraded":"2021-02-10T04:58:56.850705+0000","last_fullsized":"2021-02-10T04:58:56.850705+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:50:46.434629+0000","last_deep_scrub":"53'2","last_deep_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_clean_scrub_stamp":"2021-02-10T04:50:46.434629+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"1012","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.862954+0000","last_change":"2021-02-10T04:52:38.278980+0000","last_active":"2021-02-10T04:59:00.862954+0000","last_peered":"2021-02-10T04:59:00.862954+0000","last_clean":"2021-02-10T04:59:00.862954+0000","last_became_active":"2021-02-10T04:51:49.496237+0000","last_became_peered":"2021-02-10T04:51:49.496237+0000","last_unstale":"2021-02-10T04:59:00.862954+0000","last_undegraded":"2021-02-10T04:59:00.862954+0000","last_fullsized":"2021-02-10T04:59:00.862954+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:38.278910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:52:38.278910+0000","last_clean_scrub_stamp":"2021-02-10T04:52:38.278910+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"1000'1071","reported_seq":"4886","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.864608+0000","last_change":"2021-02-10T04:53:34.407875+0000","last_active":"2021-02-10T04:59:00.864608+0000","last_peered":"2021-02-10T04:59:00.864608+0000","last_clean":"2021-02-10T04:59:00.864608+0000","last_became_active":"2021-02-10T04:52:26.031050+0000","last_became_peered":"2021-02-10T04:52:26.031050+0000","last_unstale":"2021-02-10T04:59:00.864608+0000","last_undegraded":"2021-02-10T04:59:00.864608+0000","last_fullsized":"2021-02-10T04:59:00.864608+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:34.407784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.407784+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30173843,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"1375","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:01.874949+0000","last_change":"2021-02-10T04:58:56.867738+0000","last_active":"2021-02-10T04:59:01.874949+0000","last_peered":"2021-02-10T04:59:01.874949+0000","last_clean":"2021-02-10T04:59:01.874949+0000","last_became_active":"2021-02-10T04:58:56.867381+0000","last_became_peered":"2021-02-10T04:58:56.867381+0000","last_unstale":"2021-02-10T04:59:01.874949+0000","last_undegraded":"2021-02-10T04:59:01.874949+0000","last_fullsized":"2021-02-10T04:59:01.874949+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_clean_scrub_stamp":"2021-02-10T04:51:00.195518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"1342","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.873503+0000","last_change":"2021-02-10T04:58:58.252055+0000","last_active":"2021-02-10T04:59:00.873503+0000","last_peered":"2021-02-10T04:59:00.873503+0000","last_clean":"2021-02-10T04:59:00.873503+0000","last_became_active":"2021-02-10T04:58:56.854414+0000","last_became_peered":"2021-02-10T04:58:56.854414+0000","last_unstale":"2021-02-10T04:59:00.873503+0000","last_undegraded":"2021-02-10T04:59:00.873503+0000","last_fullsized":"2021-02-10T04:59:00.873503+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:58:58.251969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:58:58.251969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"1261'58","reported_seq":"1376","reported_epoch":"1333","state":"active+clean+wait","last_fresh":"2021-02-10T04:58:56.963723+0000","last_change":"2021-02-10T04:58:56.963026+0000","last_active":"2021-02-10T04:58:56.963723+0000","last_peered":"2021-02-10T04:58:56.963723+0000","last_clean":"2021-02-10T04:58:56.963723+0000","last_became_active":"2021-02-10T04:58:56.958817+0000","last_became_peered":"2021-02-10T04:58:56.958817+0000","last_unstale":"2021-02-10T04:58:56.963723+0000","last_undegraded":"2021-02-10T04:58:56.963723+0000","last_fullsized":"2021-02-10T04:58:56.963723+0000","mapping_epoch":1332,"log_start":"1049'40","ondisk_log_start":"1049'40","created":55,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'40","last_scrub_stamp":"2021-02-10T04:51:47.281929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.281929+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1836,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":170,"num_read_kb":134,"num_write":99,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"509","reported_epoch":"1333","state":"active+clean","last_fresh":"2021-02-10T04:58:56.842823+0000","last_change":"2021-02-10T04:58:56.841926+0000","last_active":"2021-02-10T04:58:56.842823+0000","last_peered":"2021-02-10T04:58:56.842823+0000","last_clean":"2021-02-10T04:58:56.842823+0000","last_became_active":"2021-02-10T04:58:56.840576+0000","last_became_peered":"2021-02-10T04:58:56.840576+0000","last_unstale":"2021-02-10T04:58:56.842823+0000","last_undegraded":"2021-02-10T04:58:56.842823+0000","last_fullsized":"2021-02-10T04:58:56.842823+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:50.399350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:50.399350+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"1211","reported_epoch":"1333","state":"active+clean+wait","last_fresh":"2021-02-10T04:58:56.952672+0000","last_change":"2021-02-10T04:58:56.875857+0000","last_active":"2021-02-10T04:58:56.952672+0000","last_peered":"2021-02-10T04:58:56.952672+0000","last_clean":"2021-02-10T04:58:56.952672+0000","last_became_active":"2021-02-10T04:58:56.875409+0000","last_became_peered":"2021-02-10T04:58:56.875409+0000","last_unstale":"2021-02-10T04:58:56.952672+0000","last_undegraded":"2021-02-10T04:58:56.952672+0000","last_fullsized":"2021-02-10T04:58:56.952672+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"997","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.863173+0000","last_change":"2021-02-10T04:52:31.273037+0000","last_active":"2021-02-10T04:59:00.863173+0000","last_peered":"2021-02-10T04:59:00.863173+0000","last_clean":"2021-02-10T04:59:00.863173+0000","last_became_active":"2021-02-10T04:51:49.501405+0000","last_became_peered":"2021-02-10T04:51:49.501405+0000","last_unstale":"2021-02-10T04:59:00.863173+0000","last_undegraded":"2021-02-10T04:59:00.863173+0000","last_fullsized":"2021-02-10T04:59:00.863173+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:31.272971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:17.598113+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.272971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"1000'446","reported_seq":"2066","reported_epoch":"1333","state":"active+clean","last_fresh":"2021-02-10T04:58:56.846894+0000","last_change":"2021-02-10T04:58:56.846600+0000","last_active":"2021-02-10T04:58:56.846894+0000","last_peered":"2021-02-10T04:58:56.846894+0000","last_clean":"2021-02-10T04:58:56.846894+0000","last_became_active":"2021-02-10T04:58:56.846008+0000","last_became_peered":"2021-02-10T04:58:56.846008+0000","last_unstale":"2021-02-10T04:58:56.846894+0000","last_undegraded":"2021-02-10T04:58:56.846894+0000","last_fullsized":"2021-02-10T04:58:56.846894+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:51:45.964029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.964029+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22255852,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68435,"num_write":72,"num_write_kb":24345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":403693,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"488","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.861527+0000","last_change":"2021-02-10T04:53:03.228260+0000","last_active":"2021-02-10T04:59:00.861527+0000","last_peered":"2021-02-10T04:59:00.861527+0000","last_clean":"2021-02-10T04:59:00.861527+0000","last_became_active":"2021-02-10T04:51:49.499252+0000","last_became_peered":"2021-02-10T04:51:49.499252+0000","last_unstale":"2021-02-10T04:59:00.861527+0000","last_undegraded":"2021-02-10T04:59:00.861527+0000","last_fullsized":"2021-02-10T04:59:00.861527+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:03.228182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.228182+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"238","reported_epoch":"1333","state":"active+clean","last_fresh":"2021-02-10T04:58:56.827704+0000","last_change":"2021-02-10T04:53:46.727384+0000","last_active":"2021-02-10T04:58:56.827704+0000","last_peered":"2021-02-10T04:58:56.827704+0000","last_clean":"2021-02-10T04:58:56.827704+0000","last_became_active":"2021-02-10T04:53:46.726771+0000","last_became_peered":"2021-02-10T04:53:46.726771+0000","last_unstale":"2021-02-10T04:58:56.827704+0000","last_undegraded":"2021-02-10T04:58:56.827704+0000","last_fullsized":"2021-02-10T04:58:56.827704+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:33.948348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:33.948348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"1261'51","reported_seq":"1422","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:01.874175+0000","last_change":"2021-02-10T04:54:17.118810+0000","last_active":"2021-02-10T04:59:01.874175+0000","last_peered":"2021-02-10T04:59:01.874175+0000","last_clean":"2021-02-10T04:59:01.874175+0000","last_became_active":"2021-02-10T04:54:17.080053+0000","last_became_peered":"2021-02-10T04:54:17.080053+0000","last_unstale":"2021-02-10T04:59:01.874175+0000","last_undegraded":"2021-02-10T04:59:01.874175+0000","last_fullsized":"2021-02-10T04:59:01.874175+0000","mapping_epoch":1200,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1201,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'39","last_scrub_stamp":"2021-02-10T04:51:47.322331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.322331+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":51,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"281","reported_epoch":"1333","state":"active+clean","last_fresh":"2021-02-10T04:58:56.842857+0000","last_change":"2021-02-10T04:58:56.842526+0000","last_active":"2021-02-10T04:58:56.842857+0000","last_peered":"2021-02-10T04:58:56.842857+0000","last_clean":"2021-02-10T04:58:56.842857+0000","last_became_active":"2021-02-10T04:58:56.841692+0000","last_became_peered":"2021-02-10T04:58:56.841692+0000","last_unstale":"2021-02-10T04:58:56.842857+0000","last_undegraded":"2021-02-10T04:58:56.842857+0000","last_fullsized":"2021-02-10T04:58:56.842857+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_clean_scrub_stamp":"2021-02-10T04:50:05.424262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"1506","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.871036+0000","last_change":"2021-02-10T04:52:37.324122+0000","last_active":"2021-02-10T04:59:00.871036+0000","last_peered":"2021-02-10T04:59:00.871036+0000","last_clean":"2021-02-10T04:59:00.871036+0000","last_became_active":"2021-02-10T04:51:36.843553+0000","last_became_peered":"2021-02-10T04:51:36.843553+0000","last_unstale":"2021-02-10T04:59:00.871036+0000","last_undegraded":"2021-02-10T04:59:00.871036+0000","last_fullsized":"2021-02-10T04:59:00.871036+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:37.324040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:37.324040+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"1312","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.574257+0000","last_change":"2021-02-10T04:54:37.612303+0000","last_active":"2021-02-10T04:59:01.574257+0000","last_peered":"2021-02-10T04:59:01.574257+0000","last_clean":"2021-02-10T04:59:01.574257+0000","last_became_active":"2021-02-10T04:54:37.611953+0000","last_became_peered":"2021-02-10T04:54:37.611953+0000","last_unstale":"2021-02-10T04:59:01.574257+0000","last_undegraded":"2021-02-10T04:59:01.574257+0000","last_fullsized":"2021-02-10T04:59:01.574257+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"1334","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.873585+0000","last_change":"2021-02-10T04:52:31.403904+0000","last_active":"2021-02-10T04:59:00.873585+0000","last_peered":"2021-02-10T04:59:00.873585+0000","last_clean":"2021-02-10T04:59:00.873585+0000","last_became_active":"2021-02-10T04:50:59.538009+0000","last_became_peered":"2021-02-10T04:50:59.538009+0000","last_unstale":"2021-02-10T04:59:00.873585+0000","last_undegraded":"2021-02-10T04:59:00.873585+0000","last_fullsized":"2021-02-10T04:59:00.873585+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:31.403824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.403824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"1000'490","reported_seq":"3791","reported_epoch":"1333","state":"active+clean+wait","last_fresh":"2021-02-10T04:58:56.968952+0000","last_change":"2021-02-10T04:58:56.963530+0000","last_active":"2021-02-10T04:58:56.968952+0000","last_peered":"2021-02-10T04:58:56.968952+0000","last_clean":"2021-02-10T04:58:56.968952+0000","last_became_active":"2021-02-10T04:58:56.962979+0000","last_became_peered":"2021-02-10T04:58:56.962979+0000","last_unstale":"2021-02-10T04:58:56.968952+0000","last_undegraded":"2021-02-10T04:58:56.968952+0000","last_fullsized":"2021-02-10T04:58:56.968952+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:47:02.875920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:02.875920+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19371627,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":141,"num_read_kb":148568,"num_write":85,"num_write_kb":31732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"231","reported_epoch":"1334","state":"active+clean","last_fresh":"2021-02-10T04:58:59.850477+0000","last_change":"2021-02-10T04:53:01.972991+0000","last_active":"2021-02-10T04:58:59.850477+0000","last_peered":"2021-02-10T04:58:59.850477+0000","last_clean":"2021-02-10T04:58:59.850477+0000","last_became_active":"2021-02-10T04:51:36.863976+0000","last_became_peered":"2021-02-10T04:51:36.863976+0000","last_unstale":"2021-02-10T04:58:59.850477+0000","last_undegraded":"2021-02-10T04:58:59.850477+0000","last_fullsized":"2021-02-10T04:58:59.850477+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:01.972913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.972913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"1261'40","reported_seq":"1069","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.863800+0000","last_change":"2021-02-10T04:58:56.979335+0000","last_active":"2021-02-10T04:59:00.863800+0000","last_peered":"2021-02-10T04:59:00.863800+0000","last_clean":"2021-02-10T04:59:00.863800+0000","last_became_active":"2021-02-10T04:58:56.978871+0000","last_became_peered":"2021-02-10T04:58:56.978871+0000","last_unstale":"2021-02-10T04:59:00.863800+0000","last_undegraded":"2021-02-10T04:59:00.863800+0000","last_fullsized":"2021-02-10T04:59:00.863800+0000","mapping_epoch":1332,"log_start":"1063'30","ondisk_log_start":"1063'30","created":55,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:56.431187+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.431187+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":802,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":116,"num_read_kb":94,"num_write":67,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"256","reported_epoch":"1334","state":"active+clean","last_fresh":"2021-02-10T04:58:59.848379+0000","last_change":"2021-02-10T04:58:56.879863+0000","last_active":"2021-02-10T04:58:59.848379+0000","last_peered":"2021-02-10T04:58:59.848379+0000","last_clean":"2021-02-10T04:58:59.848379+0000","last_became_active":"2021-02-10T04:58:56.879496+0000","last_became_peered":"2021-02-10T04:58:56.879496+0000","last_unstale":"2021-02-10T04:58:59.848379+0000","last_undegraded":"2021-02-10T04:58:59.848379+0000","last_fullsized":"2021-02-10T04:58:59.848379+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:04.591695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:04.591695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"1161","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.865097+0000","last_change":"2021-02-10T04:54:37.680913+0000","last_active":"2021-02-10T04:59:00.865097+0000","last_peered":"2021-02-10T04:59:00.865097+0000","last_clean":"2021-02-10T04:59:00.865097+0000","last_became_active":"2021-02-10T04:54:37.680651+0000","last_became_peered":"2021-02-10T04:54:37.680651+0000","last_unstale":"2021-02-10T04:59:00.865097+0000","last_undegraded":"2021-02-10T04:59:00.865097+0000","last_fullsized":"2021-02-10T04:59:00.865097+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1210,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":149,"num_read_kb":148,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"1177","reported_epoch":"1333","state":"active+clean","last_fresh":"2021-02-10T04:58:56.825909+0000","last_change":"2021-02-10T04:53:46.729138+0000","last_active":"2021-02-10T04:58:56.825909+0000","last_peered":"2021-02-10T04:58:56.825909+0000","last_clean":"2021-02-10T04:58:56.825909+0000","last_became_active":"2021-02-10T04:53:46.728845+0000","last_became_peered":"2021-02-10T04:53:46.728845+0000","last_unstale":"2021-02-10T04:58:56.825909+0000","last_undegraded":"2021-02-10T04:58:56.825909+0000","last_fullsized":"2021-02-10T04:58:56.825909+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:40.066354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:52:40.066354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"1000'620","reported_seq":"4415","reported_epoch":"1333","state":"active+clean+wait","last_fresh":"2021-02-10T04:58:56.956191+0000","last_change":"2021-02-10T04:58:56.953602+0000","last_active":"2021-02-10T04:58:56.956191+0000","last_peered":"2021-02-10T04:58:56.956191+0000","last_clean":"2021-02-10T04:58:56.956191+0000","last_became_active":"2021-02-10T04:58:56.878577+0000","last_became_peered":"2021-02-10T04:58:56.878577+0000","last_unstale":"2021-02-10T04:58:56.956191+0000","last_undegraded":"2021-02-10T04:58:56.956191+0000","last_fullsized":"2021-02-10T04:58:56.956191+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:51:54.444287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:54.444287+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14265195,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":144,"num_read_kb":128482,"num_write":129,"num_write_kb":53443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"1204","reported_epoch":"1333","state":"active+clean+wait","last_fresh":"2021-02-10T04:58:56.874421+0000","last_change":"2021-02-10T04:58:56.845035+0000","last_active":"2021-02-10T04:58:56.874421+0000","last_peered":"2021-02-10T04:58:56.874421+0000","last_clean":"2021-02-10T04:58:56.874421+0000","last_became_active":"2021-02-10T04:58:56.842941+0000","last_became_peered":"2021-02-10T04:58:56.842941+0000","last_unstale":"2021-02-10T04:58:56.874421+0000","last_undegraded":"2021-02-10T04:58:56.874421+0000","last_fullsized":"2021-02-10T04:58:56.874421+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:14.955544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:47:14.955544+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"1327","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.862324+0000","last_change":"2021-02-10T04:58:56.964034+0000","last_active":"2021-02-10T04:59:00.862324+0000","last_peered":"2021-02-10T04:59:00.862324+0000","last_clean":"2021-02-10T04:59:00.862324+0000","last_became_active":"2021-02-10T04:58:56.963677+0000","last_became_peered":"2021-02-10T04:58:56.963677+0000","last_unstale":"2021-02-10T04:59:00.862324+0000","last_undegraded":"2021-02-10T04:59:00.862324+0000","last_fullsized":"2021-02-10T04:59:00.862324+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"1307","reported_epoch":"1333","state":"active+clean+wait","last_fresh":"2021-02-10T04:58:56.962037+0000","last_change":"2021-02-10T04:58:56.958744+0000","last_active":"2021-02-10T04:58:56.962037+0000","last_peered":"2021-02-10T04:58:56.962037+0000","last_clean":"2021-02-10T04:58:56.962037+0000","last_became_active":"2021-02-10T04:58:56.957166+0000","last_became_peered":"2021-02-10T04:58:56.957166+0000","last_unstale":"2021-02-10T04:58:56.962037+0000","last_undegraded":"2021-02-10T04:58:56.962037+0000","last_fullsized":"2021-02-10T04:58:56.962037+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:27.908221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:47:27.908221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"1261'67","reported_seq":"986","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.862915+0000","last_change":"2021-02-10T04:52:30.277009+0000","last_active":"2021-02-10T04:59:00.862915+0000","last_peered":"2021-02-10T04:59:00.862915+0000","last_clean":"2021-02-10T04:59:00.862915+0000","last_became_active":"2021-02-10T04:51:49.501622+0000","last_became_peered":"2021-02-10T04:51:49.501622+0000","last_unstale":"2021-02-10T04:59:00.862915+0000","last_undegraded":"2021-02-10T04:59:00.862915+0000","last_fullsized":"2021-02-10T04:59:00.862915+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'55","last_scrub_stamp":"2021-02-10T04:52:30.276938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.276938+0000","log_size":67,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":116,"num_read_kb":92,"num_write":70,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"1330","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.864598+0000","last_change":"2021-02-10T04:51:59.420149+0000","last_active":"2021-02-10T04:59:00.864598+0000","last_peered":"2021-02-10T04:59:00.864598+0000","last_clean":"2021-02-10T04:59:00.864598+0000","last_became_active":"2021-02-10T04:28:21.125541+0000","last_became_peered":"2021-02-10T04:28:21.125541+0000","last_unstale":"2021-02-10T04:59:00.864598+0000","last_undegraded":"2021-02-10T04:59:00.864598+0000","last_fullsized":"2021-02-10T04:59:00.864598+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.420086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.420086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"1296","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.574788+0000","last_change":"2021-02-10T04:53:12.187225+0000","last_active":"2021-02-10T04:59:01.574788+0000","last_peered":"2021-02-10T04:59:01.574788+0000","last_clean":"2021-02-10T04:59:01.574788+0000","last_became_active":"2021-02-10T04:50:59.555689+0000","last_became_peered":"2021-02-10T04:50:59.555689+0000","last_unstale":"2021-02-10T04:59:01.574788+0000","last_undegraded":"2021-02-10T04:59:01.574788+0000","last_fullsized":"2021-02-10T04:59:01.574788+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.187154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.187154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"1002'2042","reported_seq":"9583","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:01.875320+0000","last_change":"2021-02-10T04:58:56.982754+0000","last_active":"2021-02-10T04:59:01.875320+0000","last_peered":"2021-02-10T04:59:01.875320+0000","last_clean":"2021-02-10T04:59:01.875320+0000","last_became_active":"2021-02-10T04:58:56.980189+0000","last_became_peered":"2021-02-10T04:58:56.980189+0000","last_unstale":"2021-02-10T04:59:01.875320+0000","last_undegraded":"2021-02-10T04:59:01.875320+0000","last_fullsized":"2021-02-10T04:59:01.875320+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:46:57.054230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:57.054230+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":85678898,"num_objects":35,"num_object_clones":29,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"1296","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.864840+0000","last_change":"2021-02-10T04:58:56.856707+0000","last_active":"2021-02-10T04:59:00.864840+0000","last_peered":"2021-02-10T04:59:00.864840+0000","last_clean":"2021-02-10T04:59:00.864840+0000","last_became_active":"2021-02-10T04:58:56.856323+0000","last_became_peered":"2021-02-10T04:58:56.856323+0000","last_unstale":"2021-02-10T04:59:00.864840+0000","last_undegraded":"2021-02-10T04:59:00.864840+0000","last_fullsized":"2021-02-10T04:59:00.864840+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:52.341527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:50:52.341527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"134","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.872103+0000","last_change":"2021-02-10T04:54:39.558380+0000","last_active":"2021-02-10T04:59:00.872103+0000","last_peered":"2021-02-10T04:59:00.872103+0000","last_clean":"2021-02-10T04:59:00.872103+0000","last_became_active":"2021-02-10T04:54:39.557995+0000","last_became_peered":"2021-02-10T04:54:39.557995+0000","last_unstale":"2021-02-10T04:59:00.872103+0000","last_undegraded":"2021-02-10T04:59:00.872103+0000","last_fullsized":"2021-02-10T04:59:00.872103+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"1317'5","reported_seq":"1484","reported_epoch":"1333","state":"active+clean+wait","last_fresh":"2021-02-10T04:58:56.969548+0000","last_change":"2021-02-10T04:58:56.959516+0000","last_active":"2021-02-10T04:58:56.969548+0000","last_peered":"2021-02-10T04:58:56.969548+0000","last_clean":"2021-02-10T04:58:56.969548+0000","last_became_active":"2021-02-10T04:58:56.956377+0000","last_became_peered":"2021-02-10T04:58:56.956377+0000","last_unstale":"2021-02-10T04:58:56.969548+0000","last_undegraded":"2021-02-10T04:58:56.969548+0000","last_fullsized":"2021-02-10T04:58:56.969548+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:54.466259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:54.466259+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"1261'64","reported_seq":"1032","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.862492+0000","last_change":"2021-02-10T04:53:01.304693+0000","last_active":"2021-02-10T04:59:00.862492+0000","last_peered":"2021-02-10T04:59:00.862492+0000","last_clean":"2021-02-10T04:59:00.862492+0000","last_became_active":"2021-02-10T04:51:49.498066+0000","last_became_peered":"2021-02-10T04:51:49.498066+0000","last_unstale":"2021-02-10T04:59:00.862492+0000","last_undegraded":"2021-02-10T04:59:00.862492+0000","last_fullsized":"2021-02-10T04:59:00.862492+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:53:01.304626+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.304626+0000","log_size":64,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":89,"num_read_kb":71,"num_write":52,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"608","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.862987+0000","last_change":"2021-02-10T04:58:56.868826+0000","last_active":"2021-02-10T04:59:00.862987+0000","last_peered":"2021-02-10T04:59:00.862987+0000","last_clean":"2021-02-10T04:59:00.862987+0000","last_became_active":"2021-02-10T04:58:56.868475+0000","last_became_peered":"2021-02-10T04:58:56.868475+0000","last_unstale":"2021-02-10T04:59:00.862987+0000","last_undegraded":"2021-02-10T04:59:00.862987+0000","last_fullsized":"2021-02-10T04:59:00.862987+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:49.447961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:49.447961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"1301","reported_epoch":"1334","state":"active+clean+wait","last_fresh":"2021-02-10T04:59:00.162877+0000","last_change":"2021-02-10T04:59:00.163845+0000","last_active":"2021-02-10T04:59:00.162877+0000","last_peered":"2021-02-10T04:59:00.162877+0000","last_clean":"2021-02-10T04:59:00.162877+0000","last_became_active":"2021-02-10T04:58:56.874543+0000","last_became_peered":"2021-02-10T04:58:56.874543+0000","last_unstale":"2021-02-10T04:59:00.162877+0000","last_undegraded":"2021-02-10T04:59:00.162877+0000","last_fullsized":"2021-02-10T04:59:00.162877+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:07.914061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:07.914061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"293'77","reported_seq":"3117","reported_epoch":"1335","state":"active+clean+scrubbing","last_fresh":"2021-02-10T04:59:00.861100+0000","last_change":"2021-02-10T04:53:43.102301+0000","last_active":"2021-02-10T04:59:00.861100+0000","last_peered":"2021-02-10T04:59:00.861100+0000","last_clean":"2021-02-10T04:59:00.861100+0000","last_became_active":"2021-02-10T04:52:26.024812+0000","last_became_peered":"2021-02-10T04:52:26.024812+0000","last_unstale":"2021-02-10T04:59:00.861100+0000","last_undegraded":"2021-02-10T04:59:00.861100+0000","last_fullsized":"2021-02-10T04:59:00.861100+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1886585,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":18587,"num_write":9,"num_write_kb":3315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"1350","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:01.875239+0000","last_change":"2021-02-10T04:53:07.025068+0000","last_active":"2021-02-10T04:59:01.875239+0000","last_peered":"2021-02-10T04:59:01.875239+0000","last_clean":"2021-02-10T04:59:01.875239+0000","last_became_active":"2021-02-10T04:50:59.589417+0000","last_became_peered":"2021-02-10T04:50:59.589417+0000","last_unstale":"2021-02-10T04:59:01.875239+0000","last_undegraded":"2021-02-10T04:59:01.875239+0000","last_fullsized":"2021-02-10T04:59:01.875239+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.024980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.024980+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"326","reported_epoch":"1334","state":"active+clean","last_fresh":"2021-02-10T04:58:59.848227+0000","last_change":"2021-02-10T04:58:56.851037+0000","last_active":"2021-02-10T04:58:59.848227+0000","last_peered":"2021-02-10T04:58:59.848227+0000","last_clean":"2021-02-10T04:58:59.848227+0000","last_became_active":"2021-02-10T04:58:56.850742+0000","last_became_peered":"2021-02-10T04:58:56.850742+0000","last_unstale":"2021-02-10T04:58:59.848227+0000","last_undegraded":"2021-02-10T04:58:59.848227+0000","last_fullsized":"2021-02-10T04:58:59.848227+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.609500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.609500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"1261'86","reported_seq":"1516","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.862501+0000","last_change":"2021-02-10T04:55:35.577794+0000","last_active":"2021-02-10T04:59:00.862501+0000","last_peered":"2021-02-10T04:59:00.862501+0000","last_clean":"2021-02-10T04:59:00.862501+0000","last_became_active":"2021-02-10T04:55:35.577316+0000","last_became_peered":"2021-02-10T04:55:35.577316+0000","last_unstale":"2021-02-10T04:59:00.862501+0000","last_undegraded":"2021-02-10T04:59:00.862501+0000","last_fullsized":"2021-02-10T04:59:00.862501+0000","mapping_epoch":1238,"log_start":"1049'68","ondisk_log_start":"1049'68","created":55,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'39","last_scrub_stamp":"2021-02-10T04:53:38.085565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.085565+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1582,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":227,"num_read_kb":187,"num_write":128,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"1464","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.862922+0000","last_change":"2021-02-10T04:58:56.844400+0000","last_active":"2021-02-10T04:59:00.862922+0000","last_peered":"2021-02-10T04:59:00.862922+0000","last_clean":"2021-02-10T04:59:00.862922+0000","last_became_active":"2021-02-10T04:58:56.843963+0000","last_became_peered":"2021-02-10T04:58:56.843963+0000","last_unstale":"2021-02-10T04:59:00.862922+0000","last_undegraded":"2021-02-10T04:59:00.862922+0000","last_fullsized":"2021-02-10T04:59:00.862922+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:47:08.784593+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:47:08.784593+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"1298","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.573561+0000","last_change":"2021-02-10T04:52:30.019225+0000","last_active":"2021-02-10T04:59:01.573561+0000","last_peered":"2021-02-10T04:59:01.573561+0000","last_clean":"2021-02-10T04:59:01.573561+0000","last_became_active":"2021-02-10T04:50:59.561436+0000","last_became_peered":"2021-02-10T04:50:59.561436+0000","last_unstale":"2021-02-10T04:59:01.573561+0000","last_undegraded":"2021-02-10T04:59:01.573561+0000","last_fullsized":"2021-02-10T04:59:01.573561+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.019102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.019102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"1001'1528","reported_seq":"6283","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.574096+0000","last_change":"2021-02-10T04:52:26.036366+0000","last_active":"2021-02-10T04:59:01.574096+0000","last_peered":"2021-02-10T04:59:01.574096+0000","last_clean":"2021-02-10T04:59:01.574096+0000","last_became_active":"2021-02-10T04:52:26.034028+0000","last_became_peered":"2021-02-10T04:52:26.034028+0000","last_unstale":"2021-02-10T04:59:01.574096+0000","last_undegraded":"2021-02-10T04:59:01.574096+0000","last_fullsized":"2021-02-10T04:59:01.574096+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46944136,"num_objects":16,"num_object_clones":15,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":358,"num_read_kb":351934,"num_write":286,"num_write_kb":111355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":58968832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.15","version":"1000'446","reported_seq":"2272","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.863706+0000","last_change":"2021-02-10T04:53:11.051124+0000","last_active":"2021-02-10T04:59:00.863706+0000","last_peered":"2021-02-10T04:59:00.863706+0000","last_clean":"2021-02-10T04:59:00.863706+0000","last_became_active":"2021-02-10T04:51:40.510899+0000","last_became_peered":"2021-02-10T04:51:40.510899+0000","last_unstale":"2021-02-10T04:59:00.863706+0000","last_undegraded":"2021-02-10T04:59:00.863706+0000","last_fullsized":"2021-02-10T04:59:00.863706+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:53:11.051031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.051031+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24896285,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":18983408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"260","reported_epoch":"1334","state":"active+clean","last_fresh":"2021-02-10T04:58:59.848624+0000","last_change":"2021-02-10T04:58:56.851004+0000","last_active":"2021-02-10T04:58:59.848624+0000","last_peered":"2021-02-10T04:58:59.848624+0000","last_clean":"2021-02-10T04:58:59.848624+0000","last_became_active":"2021-02-10T04:58:56.850650+0000","last_became_peered":"2021-02-10T04:58:56.850650+0000","last_unstale":"2021-02-10T04:58:59.848624+0000","last_undegraded":"2021-02-10T04:58:59.848624+0000","last_fullsized":"2021-02-10T04:58:59.848624+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:51:08.639784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.639784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"1267","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.575016+0000","last_change":"2021-02-10T04:51:09.475150+0000","last_active":"2021-02-10T04:59:01.575016+0000","last_peered":"2021-02-10T04:59:01.575016+0000","last_clean":"2021-02-10T04:59:01.575016+0000","last_became_active":"2021-02-10T04:50:59.585917+0000","last_became_peered":"2021-02-10T04:50:59.585917+0000","last_unstale":"2021-02-10T04:59:01.575016+0000","last_undegraded":"2021-02-10T04:59:01.575016+0000","last_fullsized":"2021-02-10T04:59:01.575016+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:09.475063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:09.475063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"236","reported_epoch":"1334","state":"active+clean","last_fresh":"2021-02-10T04:58:59.849295+0000","last_change":"2021-02-10T04:53:34.927483+0000","last_active":"2021-02-10T04:58:59.849295+0000","last_peered":"2021-02-10T04:58:59.849295+0000","last_clean":"2021-02-10T04:58:59.849295+0000","last_became_active":"2021-02-10T04:51:36.859707+0000","last_became_peered":"2021-02-10T04:51:36.859707+0000","last_unstale":"2021-02-10T04:58:59.849295+0000","last_undegraded":"2021-02-10T04:58:59.849295+0000","last_fullsized":"2021-02-10T04:58:59.849295+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:34.927424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.927424+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.14","version":"1000'620","reported_seq":"4622","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.574600+0000","last_change":"2021-02-10T04:53:37.384461+0000","last_active":"2021-02-10T04:59:01.574600+0000","last_peered":"2021-02-10T04:59:01.574600+0000","last_clean":"2021-02-10T04:59:01.574600+0000","last_became_active":"2021-02-10T04:51:40.510114+0000","last_became_peered":"2021-02-10T04:51:40.510114+0000","last_unstale":"2021-02-10T04:59:01.574600+0000","last_undegraded":"2021-02-10T04:59:01.574600+0000","last_fullsized":"2021-02-10T04:59:01.574600+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:53:37.384398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:37.384398+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28530390,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"1324","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:01.871356+0000","last_change":"2021-02-10T04:52:41.059269+0000","last_active":"2021-02-10T04:59:01.871356+0000","last_peered":"2021-02-10T04:59:01.871356+0000","last_clean":"2021-02-10T04:59:01.871356+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:59:01.871356+0000","last_undegraded":"2021-02-10T04:59:01.871356+0000","last_fullsized":"2021-02-10T04:59:01.871356+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:52:41.059159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.059159+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"1327","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.872750+0000","last_change":"2021-02-10T04:53:12.332890+0000","last_active":"2021-02-10T04:59:00.872750+0000","last_peered":"2021-02-10T04:59:00.872750+0000","last_clean":"2021-02-10T04:59:00.872750+0000","last_became_active":"2021-02-10T04:50:59.540375+0000","last_became_peered":"2021-02-10T04:50:59.540375+0000","last_unstale":"2021-02-10T04:59:00.872750+0000","last_undegraded":"2021-02-10T04:59:00.872750+0000","last_fullsized":"2021-02-10T04:59:00.872750+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.332233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.332233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"322","reported_epoch":"1333","state":"active+clean","last_fresh":"2021-02-10T04:58:56.825761+0000","last_change":"2021-02-10T04:58:56.576305+0000","last_active":"2021-02-10T04:58:56.825761+0000","last_peered":"2021-02-10T04:58:56.825761+0000","last_clean":"2021-02-10T04:58:56.825761+0000","last_became_active":"2021-02-10T04:53:46.733191+0000","last_became_peered":"2021-02-10T04:53:46.733191+0000","last_unstale":"2021-02-10T04:58:56.825761+0000","last_undegraded":"2021-02-10T04:58:56.825761+0000","last_fullsized":"2021-02-10T04:58:56.825761+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:45.362537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:45.362537+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"1002'2042","reported_seq":"9306","reported_epoch":"1333","state":"active+clean","last_fresh":"2021-02-10T04:58:56.825712+0000","last_change":"2021-02-10T04:53:46.726353+0000","last_active":"2021-02-10T04:58:56.825712+0000","last_peered":"2021-02-10T04:58:56.825712+0000","last_clean":"2021-02-10T04:58:56.825712+0000","last_became_active":"2021-02-10T04:53:46.725828+0000","last_became_peered":"2021-02-10T04:53:46.725828+0000","last_unstale":"2021-02-10T04:58:56.825712+0000","last_undegraded":"2021-02-10T04:58:56.825712+0000","last_fullsized":"2021-02-10T04:58:56.825712+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:52:41.286006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.286006+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46300076,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":545,"num_read_kb":531689,"num_write":399,"num_write_kb":153087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"1261'56","reported_seq":"1506","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.574759+0000","last_change":"2021-02-10T04:52:35.091494+0000","last_active":"2021-02-10T04:59:01.574759+0000","last_peered":"2021-02-10T04:59:01.574759+0000","last_clean":"2021-02-10T04:59:01.574759+0000","last_became_active":"2021-02-10T04:50:59.569915+0000","last_became_peered":"2021-02-10T04:50:59.569915+0000","last_unstale":"2021-02-10T04:59:01.574759+0000","last_undegraded":"2021-02-10T04:59:01.574759+0000","last_fullsized":"2021-02-10T04:59:01.574759+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'46","last_scrub_stamp":"2021-02-10T04:52:35.091374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:35.091374+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":95,"num_read_kb":80,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"84'7","reported_seq":"1351","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.864475+0000","last_change":"2021-02-10T04:51:52.498251+0000","last_active":"2021-02-10T04:59:00.864475+0000","last_peered":"2021-02-10T04:59:00.864475+0000","last_clean":"2021-02-10T04:59:00.864475+0000","last_became_active":"2021-02-10T04:50:59.575924+0000","last_became_peered":"2021-02-10T04:50:59.575924+0000","last_unstale":"2021-02-10T04:59:00.864475+0000","last_undegraded":"2021-02-10T04:59:00.864475+0000","last_fullsized":"2021-02-10T04:59:00.864475+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'7","last_scrub_stamp":"2021-02-10T04:51:52.498158+0000","last_deep_scrub":"84'7","last_deep_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.498158+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"1333'6","reported_seq":"1745","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:01.868866+0000","last_change":"2021-02-10T04:58:57.207618+0000","last_active":"2021-02-10T04:59:01.868866+0000","last_peered":"2021-02-10T04:59:01.868866+0000","last_clean":"2021-02-10T04:59:01.868866+0000","last_became_active":"2021-02-10T04:58:56.849778+0000","last_became_peered":"2021-02-10T04:58:56.849778+0000","last_unstale":"2021-02-10T04:59:01.868866+0000","last_undegraded":"2021-02-10T04:59:01.868866+0000","last_fullsized":"2021-02-10T04:59:01.868866+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1317'5","last_scrub_stamp":"2021-02-10T04:58:57.207527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:58:57.207527+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"1238","reported_epoch":"1333","state":"active+clean+wait","last_fresh":"2021-02-10T04:58:56.968980+0000","last_change":"2021-02-10T04:58:56.965746+0000","last_active":"2021-02-10T04:58:56.968980+0000","last_peered":"2021-02-10T04:58:56.968980+0000","last_clean":"2021-02-10T04:58:56.968980+0000","last_became_active":"2021-02-10T04:58:56.965228+0000","last_became_peered":"2021-02-10T04:58:56.965228+0000","last_unstale":"2021-02-10T04:58:56.968980+0000","last_undegraded":"2021-02-10T04:58:56.968980+0000","last_fullsized":"2021-02-10T04:58:56.968980+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:23.566027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_clean_scrub_stamp":"2021-02-10T04:52:23.566027+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"1000'490","reported_seq":"3866","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.863198+0000","last_change":"2021-02-10T04:58:56.966763+0000","last_active":"2021-02-10T04:59:00.863198+0000","last_peered":"2021-02-10T04:59:00.863198+0000","last_clean":"2021-02-10T04:59:00.863198+0000","last_became_active":"2021-02-10T04:58:56.965659+0000","last_became_peered":"2021-02-10T04:58:56.965659+0000","last_unstale":"2021-02-10T04:59:00.863198+0000","last_undegraded":"2021-02-10T04:59:00.863198+0000","last_fullsized":"2021-02-10T04:59:00.863198+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:50:03.756248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:50:03.756248+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16154790,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"1261'46","reported_seq":"1404","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.872021+0000","last_change":"2021-02-10T04:53:10.276449+0000","last_active":"2021-02-10T04:59:00.872021+0000","last_peered":"2021-02-10T04:59:00.872021+0000","last_clean":"2021-02-10T04:59:00.872021+0000","last_became_active":"2021-02-10T04:50:59.539529+0000","last_became_peered":"2021-02-10T04:50:59.539529+0000","last_unstale":"2021-02-10T04:59:00.872021+0000","last_undegraded":"2021-02-10T04:59:00.872021+0000","last_fullsized":"2021-02-10T04:59:00.872021+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1112'40","last_scrub_stamp":"2021-02-10T04:53:10.276358+0000","last_deep_scrub":"1112'40","last_deep_scrub_stamp":"2021-02-10T04:51:01.450050+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.276358+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":53,"num_write":38,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"1223","reported_epoch":"1333","state":"active+clean+wait","last_fresh":"2021-02-10T04:58:56.972218+0000","last_change":"2021-02-10T04:58:56.971548+0000","last_active":"2021-02-10T04:58:56.972218+0000","last_peered":"2021-02-10T04:58:56.972218+0000","last_clean":"2021-02-10T04:58:56.972218+0000","last_became_active":"2021-02-10T04:58:56.970118+0000","last_became_peered":"2021-02-10T04:58:56.970118+0000","last_unstale":"2021-02-10T04:58:56.972218+0000","last_undegraded":"2021-02-10T04:58:56.972218+0000","last_fullsized":"2021-02-10T04:58:56.972218+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:23.904210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:47:23.904210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"1119'6","reported_seq":"1637","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.574904+0000","last_change":"2021-02-10T04:52:40.038646+0000","last_active":"2021-02-10T04:59:01.574904+0000","last_peered":"2021-02-10T04:59:01.574904+0000","last_clean":"2021-02-10T04:59:01.574904+0000","last_became_active":"2021-02-10T04:51:36.869733+0000","last_became_peered":"2021-02-10T04:51:36.869733+0000","last_unstale":"2021-02-10T04:59:01.574904+0000","last_undegraded":"2021-02-10T04:59:01.574904+0000","last_fullsized":"2021-02-10T04:59:01.574904+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'6","last_scrub_stamp":"2021-02-10T04:52:40.038554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:40.038554+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"1321","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.863133+0000","last_change":"2021-02-10T04:51:36.849933+0000","last_active":"2021-02-10T04:59:00.863133+0000","last_peered":"2021-02-10T04:59:00.863133+0000","last_clean":"2021-02-10T04:59:00.863133+0000","last_became_active":"2021-02-10T04:51:36.849184+0000","last_became_peered":"2021-02-10T04:51:36.849184+0000","last_unstale":"2021-02-10T04:59:00.863133+0000","last_undegraded":"2021-02-10T04:59:00.863133+0000","last_fullsized":"2021-02-10T04:59:00.863133+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:11.779779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:11.779779+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"1032'89","reported_seq":"927","reported_epoch":"1333","state":"active+clean","last_fresh":"2021-02-10T04:58:56.830279+0000","last_change":"2021-02-10T04:53:46.728868+0000","last_active":"2021-02-10T04:58:56.830279+0000","last_peered":"2021-02-10T04:58:56.830279+0000","last_clean":"2021-02-10T04:58:56.830279+0000","last_became_active":"2021-02-10T04:53:46.728356+0000","last_became_peered":"2021-02-10T04:53:46.728356+0000","last_unstale":"2021-02-10T04:58:56.830279+0000","last_undegraded":"2021-02-10T04:58:56.830279+0000","last_fullsized":"2021-02-10T04:58:56.830279+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":5,"last_scrub":"1032'89","last_scrub_stamp":"2021-02-10T04:52:34.089789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:34.089789+0000","log_size":89,"ondisk_log_size":89,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5218574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":30326,"num_write":21,"num_write_kb":12444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3210022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"1261'43","reported_seq":"1025","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.862445+0000","last_change":"2021-02-10T04:53:14.202505+0000","last_active":"2021-02-10T04:59:00.862445+0000","last_peered":"2021-02-10T04:59:00.862445+0000","last_clean":"2021-02-10T04:59:00.862445+0000","last_became_active":"2021-02-10T04:51:49.496294+0000","last_became_peered":"2021-02-10T04:51:49.496294+0000","last_unstale":"2021-02-10T04:59:00.862445+0000","last_undegraded":"2021-02-10T04:59:00.862445+0000","last_fullsized":"2021-02-10T04:59:00.862445+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1163'37","last_scrub_stamp":"2021-02-10T04:53:14.202445+0000","last_deep_scrub":"1163'37","last_deep_scrub_stamp":"2021-02-10T04:53:14.202445+0000","last_clean_scrub_stamp":"2021-02-10T04:53:14.202445+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":60,"num_read_kb":42,"num_write":34,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"1337","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:01.873598+0000","last_change":"2021-02-10T04:52:23.095651+0000","last_active":"2021-02-10T04:59:01.873598+0000","last_peered":"2021-02-10T04:59:01.873598+0000","last_clean":"2021-02-10T04:59:01.873598+0000","last_became_active":"2021-02-10T04:50:59.543531+0000","last_became_peered":"2021-02-10T04:50:59.543531+0000","last_unstale":"2021-02-10T04:59:01.873598+0000","last_undegraded":"2021-02-10T04:59:01.873598+0000","last_fullsized":"2021-02-10T04:59:01.873598+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:23.095588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:23.095588+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"1301","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.573293+0000","last_change":"2021-02-10T04:58:56.965982+0000","last_active":"2021-02-10T04:59:01.573293+0000","last_peered":"2021-02-10T04:59:01.573293+0000","last_clean":"2021-02-10T04:59:01.573293+0000","last_became_active":"2021-02-10T04:58:56.964476+0000","last_became_peered":"2021-02-10T04:58:56.964476+0000","last_unstale":"2021-02-10T04:59:01.573293+0000","last_undegraded":"2021-02-10T04:59:01.573293+0000","last_fullsized":"2021-02-10T04:59:01.573293+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:43.663762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:51:43.663762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"1194","reported_epoch":"1333","state":"active+clean+wait","last_fresh":"2021-02-10T04:58:56.878617+0000","last_change":"2021-02-10T04:58:56.878092+0000","last_active":"2021-02-10T04:58:56.878617+0000","last_peered":"2021-02-10T04:58:56.878617+0000","last_clean":"2021-02-10T04:58:56.878617+0000","last_became_active":"2021-02-10T04:58:56.877585+0000","last_became_peered":"2021-02-10T04:58:56.877585+0000","last_unstale":"2021-02-10T04:58:56.878617+0000","last_undegraded":"2021-02-10T04:58:56.878617+0000","last_fullsized":"2021-02-10T04:58:56.878617+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:57.568541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:57.568541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"1000'1071","reported_seq":"4841","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.572727+0000","last_change":"2021-02-10T04:53:35.302879+0000","last_active":"2021-02-10T04:59:01.572727+0000","last_peered":"2021-02-10T04:59:01.572727+0000","last_clean":"2021-02-10T04:59:01.572727+0000","last_became_active":"2021-02-10T04:51:39.508551+0000","last_became_peered":"2021-02-10T04:51:39.508551+0000","last_unstale":"2021-02-10T04:59:01.572727+0000","last_undegraded":"2021-02-10T04:59:01.572727+0000","last_fullsized":"2021-02-10T04:59:01.572727+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:35.302780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.302780+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"1261'39","reported_seq":"1370","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.863932+0000","last_change":"2021-02-10T04:52:39.695340+0000","last_active":"2021-02-10T04:59:00.863932+0000","last_peered":"2021-02-10T04:59:00.863932+0000","last_clean":"2021-02-10T04:59:00.863932+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:59:00.863932+0000","last_undegraded":"2021-02-10T04:59:00.863932+0000","last_fullsized":"2021-02-10T04:59:00.863932+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'31","last_scrub_stamp":"2021-02-10T04:52:39.695237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:39.695237+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":53,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"595","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.864441+0000","last_change":"2021-02-10T04:58:56.964767+0000","last_active":"2021-02-10T04:59:00.864441+0000","last_peered":"2021-02-10T04:59:00.864441+0000","last_clean":"2021-02-10T04:59:00.864441+0000","last_became_active":"2021-02-10T04:58:56.964387+0000","last_became_peered":"2021-02-10T04:58:56.964387+0000","last_unstale":"2021-02-10T04:59:00.864441+0000","last_undegraded":"2021-02-10T04:59:00.864441+0000","last_fullsized":"2021-02-10T04:59:00.864441+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:55:20.929330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:55:20.929330+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"988","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.862532+0000","last_change":"2021-02-10T04:53:35.216489+0000","last_active":"2021-02-10T04:59:00.862532+0000","last_peered":"2021-02-10T04:59:00.862532+0000","last_clean":"2021-02-10T04:59:00.862532+0000","last_became_active":"2021-02-10T04:51:49.501254+0000","last_became_peered":"2021-02-10T04:51:49.501254+0000","last_unstale":"2021-02-10T04:59:00.862532+0000","last_undegraded":"2021-02-10T04:59:00.862532+0000","last_fullsized":"2021-02-10T04:59:00.862532+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.216394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.216394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"1328","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.572655+0000","last_change":"2021-02-10T04:58:56.879817+0000","last_active":"2021-02-10T04:59:01.572655+0000","last_peered":"2021-02-10T04:59:01.572655+0000","last_clean":"2021-02-10T04:59:01.572655+0000","last_became_active":"2021-02-10T04:58:56.879372+0000","last_became_peered":"2021-02-10T04:58:56.879372+0000","last_unstale":"2021-02-10T04:59:01.572655+0000","last_undegraded":"2021-02-10T04:59:01.572655+0000","last_fullsized":"2021-02-10T04:59:01.572655+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:55.814486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.814486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"1002'1657","reported_seq":"5351","reported_epoch":"1333","state":"active+clean+wait","last_fresh":"2021-02-10T04:58:56.971991+0000","last_change":"2021-02-10T04:58:56.970595+0000","last_active":"2021-02-10T04:58:56.971991+0000","last_peered":"2021-02-10T04:58:56.971991+0000","last_clean":"2021-02-10T04:58:56.971991+0000","last_became_active":"2021-02-10T04:58:56.961771+0000","last_became_peered":"2021-02-10T04:58:56.961771+0000","last_unstale":"2021-02-10T04:58:56.971991+0000","last_undegraded":"2021-02-10T04:58:56.971991+0000","last_fullsized":"2021-02-10T04:58:56.971991+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:55:22.331010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:55:22.331010+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":47335497,"num_objects":20,"num_object_clones":18,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"1261'24","reported_seq":"1443","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.572921+0000","last_change":"2021-02-10T04:53:04.058324+0000","last_active":"2021-02-10T04:59:01.572921+0000","last_peered":"2021-02-10T04:59:01.572921+0000","last_clean":"2021-02-10T04:59:01.572921+0000","last_became_active":"2021-02-10T04:50:59.544484+0000","last_became_peered":"2021-02-10T04:50:59.544484+0000","last_unstale":"2021-02-10T04:59:01.572921+0000","last_undegraded":"2021-02-10T04:59:01.572921+0000","last_fullsized":"2021-02-10T04:59:01.572921+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'20","last_scrub_stamp":"2021-02-10T04:53:04.058248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:04.058248+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":29,"num_write":24,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"1339","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.872937+0000","last_change":"2021-02-10T04:53:00.343307+0000","last_active":"2021-02-10T04:59:00.872937+0000","last_peered":"2021-02-10T04:59:00.872937+0000","last_clean":"2021-02-10T04:59:00.872937+0000","last_became_active":"2021-02-10T04:51:36.844573+0000","last_became_peered":"2021-02-10T04:51:36.844573+0000","last_unstale":"2021-02-10T04:59:00.872937+0000","last_undegraded":"2021-02-10T04:59:00.872937+0000","last_fullsized":"2021-02-10T04:59:00.872937+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:00.343228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:00.343228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"1308","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:01.871272+0000","last_change":"2021-02-10T04:50:56.099710+0000","last_active":"2021-02-10T04:59:01.871272+0000","last_peered":"2021-02-10T04:59:01.871272+0000","last_clean":"2021-02-10T04:59:01.871272+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:59:01.871272+0000","last_undegraded":"2021-02-10T04:59:01.871272+0000","last_fullsized":"2021-02-10T04:59:01.871272+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:56.099651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:50:56.099651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"652","reported_epoch":"1333","state":"active+clean","last_fresh":"2021-02-10T04:58:56.825336+0000","last_change":"2021-02-10T04:53:46.718715+0000","last_active":"2021-02-10T04:58:56.825336+0000","last_peered":"2021-02-10T04:58:56.825336+0000","last_clean":"2021-02-10T04:58:56.825336+0000","last_became_active":"2021-02-10T04:53:46.718266+0000","last_became_peered":"2021-02-10T04:53:46.718266+0000","last_unstale":"2021-02-10T04:58:56.825336+0000","last_undegraded":"2021-02-10T04:58:56.825336+0000","last_fullsized":"2021-02-10T04:58:56.825336+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.876681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.876681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"998'1009","reported_seq":"5199","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:01.871864+0000","last_change":"2021-02-10T04:58:56.958942+0000","last_active":"2021-02-10T04:59:01.871864+0000","last_peered":"2021-02-10T04:59:01.871864+0000","last_clean":"2021-02-10T04:59:01.871864+0000","last_became_active":"2021-02-10T04:58:56.957778+0000","last_became_peered":"2021-02-10T04:58:56.957778+0000","last_unstale":"2021-02-10T04:59:01.871864+0000","last_undegraded":"2021-02-10T04:59:01.871864+0000","last_fullsized":"2021-02-10T04:59:01.871864+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":5,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:13.032506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:13.032506+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49420827,"num_objects":21,"num_object_clones":19,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":36763798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"1261'45","reported_seq":"1377","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.572917+0000","last_change":"2021-02-10T04:51:44.694779+0000","last_active":"2021-02-10T04:59:01.572917+0000","last_peered":"2021-02-10T04:59:01.572917+0000","last_clean":"2021-02-10T04:59:01.572917+0000","last_became_active":"2021-02-10T04:50:59.549412+0000","last_became_peered":"2021-02-10T04:50:59.549412+0000","last_unstale":"2021-02-10T04:59:01.572917+0000","last_undegraded":"2021-02-10T04:59:01.572917+0000","last_fullsized":"2021-02-10T04:59:01.572917+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:44.694711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.694711+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":76,"num_read_kb":62,"num_write":45,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"1475","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:01.875563+0000","last_change":"2021-02-10T04:52:33.088525+0000","last_active":"2021-02-10T04:59:01.875563+0000","last_peered":"2021-02-10T04:59:01.875563+0000","last_clean":"2021-02-10T04:59:01.875563+0000","last_became_active":"2021-02-10T04:51:36.870573+0000","last_became_peered":"2021-02-10T04:51:36.870573+0000","last_unstale":"2021-02-10T04:59:01.875563+0000","last_undegraded":"2021-02-10T04:59:01.875563+0000","last_fullsized":"2021-02-10T04:59:01.875563+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:52:33.088428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:33.088428+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"621","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.572977+0000","last_change":"2021-02-10T04:51:51.836723+0000","last_active":"2021-02-10T04:59:01.572977+0000","last_peered":"2021-02-10T04:59:01.572977+0000","last_clean":"2021-02-10T04:59:01.572977+0000","last_became_active":"2021-02-10T04:50:59.564977+0000","last_became_peered":"2021-02-10T04:50:59.564977+0000","last_unstale":"2021-02-10T04:59:01.572977+0000","last_undegraded":"2021-02-10T04:59:01.572977+0000","last_fullsized":"2021-02-10T04:59:01.572977+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.836638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.836638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"1292","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.573221+0000","last_change":"2021-02-10T04:58:56.966197+0000","last_active":"2021-02-10T04:59:01.573221+0000","last_peered":"2021-02-10T04:59:01.573221+0000","last_clean":"2021-02-10T04:59:01.573221+0000","last_became_active":"2021-02-10T04:58:56.964968+0000","last_became_peered":"2021-02-10T04:58:56.964968+0000","last_unstale":"2021-02-10T04:59:01.573221+0000","last_undegraded":"2021-02-10T04:59:01.573221+0000","last_fullsized":"2021-02-10T04:59:01.573221+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:58.514318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_clean_scrub_stamp":"2021-02-10T04:46:58.514318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1d","version":"1001'1165","reported_seq":"4351","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.864524+0000","last_change":"2021-02-10T04:58:56.966207+0000","last_active":"2021-02-10T04:59:00.864524+0000","last_peered":"2021-02-10T04:59:00.864524+0000","last_clean":"2021-02-10T04:59:00.864524+0000","last_became_active":"2021-02-10T04:58:56.965303+0000","last_became_peered":"2021-02-10T04:58:56.965303+0000","last_unstale":"2021-02-10T04:59:00.864524+0000","last_undegraded":"2021-02-10T04:59:00.864524+0000","last_fullsized":"2021-02-10T04:59:00.864524+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":5,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29601845,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":287,"num_read_kb":299530,"num_write":200,"num_write_kb":89131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"1380","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.864322+0000","last_change":"2021-02-10T04:53:01.331566+0000","last_active":"2021-02-10T04:59:00.864322+0000","last_peered":"2021-02-10T04:59:00.864322+0000","last_clean":"2021-02-10T04:59:00.864322+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:59:00.864322+0000","last_undegraded":"2021-02-10T04:59:00.864322+0000","last_fullsized":"2021-02-10T04:59:00.864322+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:01.331466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.331466+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"1098'3","reported_seq":"1715","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:03.076668+0000","last_change":"2021-02-10T04:53:34.865668+0000","last_active":"2021-02-10T04:59:03.076668+0000","last_peered":"2021-02-10T04:59:03.076668+0000","last_clean":"2021-02-10T04:59:03.076668+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:59:03.076668+0000","last_undegraded":"2021-02-10T04:59:03.076668+0000","last_fullsized":"2021-02-10T04:59:03.076668+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"1098'3","last_scrub_stamp":"2021-02-10T04:53:34.865579+0000","last_deep_scrub":"1098'3","last_deep_scrub_stamp":"2021-02-10T04:50:15.255137+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.865579+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"1182","reported_epoch":"1333","state":"active+clean","last_fresh":"2021-02-10T04:58:56.827874+0000","last_change":"2021-02-10T04:53:46.727603+0000","last_active":"2021-02-10T04:58:56.827874+0000","last_peered":"2021-02-10T04:58:56.827874+0000","last_clean":"2021-02-10T04:58:56.827874+0000","last_became_active":"2021-02-10T04:53:46.727053+0000","last_became_peered":"2021-02-10T04:53:46.727053+0000","last_unstale":"2021-02-10T04:58:56.827874+0000","last_undegraded":"2021-02-10T04:58:56.827874+0000","last_fullsized":"2021-02-10T04:58:56.827874+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:47.493057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.493057+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1c","version":"1002'1090","reported_seq":"5360","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.871274+0000","last_change":"2021-02-10T04:53:41.353991+0000","last_active":"2021-02-10T04:59:00.871274+0000","last_peered":"2021-02-10T04:59:00.871274+0000","last_clean":"2021-02-10T04:59:00.871274+0000","last_became_active":"2021-02-10T04:52:31.088265+0000","last_became_peered":"2021-02-10T04:52:31.088265+0000","last_unstale":"2021-02-10T04:59:00.871274+0000","last_undegraded":"2021-02-10T04:59:00.871274+0000","last_fullsized":"2021-02-10T04:59:00.871274+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:53:41.353840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:41.353840+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"1296","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.572775+0000","last_change":"2021-02-10T04:58:56.879938+0000","last_active":"2021-02-10T04:59:01.572775+0000","last_peered":"2021-02-10T04:59:01.572775+0000","last_clean":"2021-02-10T04:59:01.572775+0000","last_became_active":"2021-02-10T04:58:56.879661+0000","last_became_peered":"2021-02-10T04:58:56.879661+0000","last_unstale":"2021-02-10T04:59:01.572775+0000","last_undegraded":"2021-02-10T04:59:01.572775+0000","last_fullsized":"2021-02-10T04:59:01.572775+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:42.537443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:42.537443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"1219","reported_epoch":"1333","state":"active+clean+wait","last_fresh":"2021-02-10T04:58:56.962271+0000","last_change":"2021-02-10T04:58:56.872154+0000","last_active":"2021-02-10T04:58:56.962271+0000","last_peered":"2021-02-10T04:58:56.962271+0000","last_clean":"2021-02-10T04:58:56.962271+0000","last_became_active":"2021-02-10T04:58:56.869951+0000","last_became_peered":"2021-02-10T04:58:56.869951+0000","last_unstale":"2021-02-10T04:58:56.962271+0000","last_undegraded":"2021-02-10T04:58:56.962271+0000","last_fullsized":"2021-02-10T04:58:56.962271+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.938731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.938731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"545","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.864302+0000","last_change":"2021-02-10T04:53:15.185748+0000","last_active":"2021-02-10T04:59:00.864302+0000","last_peered":"2021-02-10T04:59:00.864302+0000","last_clean":"2021-02-10T04:59:00.864302+0000","last_became_active":"2021-02-10T04:51:49.499348+0000","last_became_peered":"2021-02-10T04:51:49.499348+0000","last_unstale":"2021-02-10T04:59:00.864302+0000","last_undegraded":"2021-02-10T04:59:00.864302+0000","last_fullsized":"2021-02-10T04:59:00.864302+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:15.185686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:15.185686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1f","version":"1004'1093","reported_seq":"6598","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:01.875060+0000","last_change":"2021-02-10T04:52:31.089952+0000","last_active":"2021-02-10T04:59:01.875060+0000","last_peered":"2021-02-10T04:59:01.875060+0000","last_clean":"2021-02-10T04:59:01.875060+0000","last_became_active":"2021-02-10T04:52:31.088634+0000","last_became_peered":"2021-02-10T04:52:31.088634+0000","last_unstale":"2021-02-10T04:59:01.875060+0000","last_undegraded":"2021-02-10T04:59:01.875060+0000","last_fullsized":"2021-02-10T04:59:01.875060+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39434697,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":279,"num_read_kb":311058,"num_write":186,"num_write_kb":66790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":53995913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"1338","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:01.872334+0000","last_change":"2021-02-10T04:55:39.651056+0000","last_active":"2021-02-10T04:59:01.872334+0000","last_peered":"2021-02-10T04:59:01.872334+0000","last_clean":"2021-02-10T04:59:01.872334+0000","last_became_active":"2021-02-10T04:55:39.596648+0000","last_became_peered":"2021-02-10T04:55:39.596648+0000","last_unstale":"2021-02-10T04:59:01.872334+0000","last_undegraded":"2021-02-10T04:59:01.872334+0000","last_fullsized":"2021-02-10T04:59:01.872334+0000","mapping_epoch":1248,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:52:36.083784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:36.083784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"638","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:02.383789+0000","last_change":"2021-02-10T04:59:02.384471+0000","last_active":"2021-02-10T04:59:02.383789+0000","last_peered":"2021-02-10T04:59:02.383789+0000","last_clean":"2021-02-10T04:59:02.383789+0000","last_became_active":"2021-02-10T04:58:56.965808+0000","last_became_peered":"2021-02-10T04:58:56.965808+0000","last_unstale":"2021-02-10T04:59:02.383789+0000","last_undegraded":"2021-02-10T04:59:02.383789+0000","last_fullsized":"2021-02-10T04:59:02.383789+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:08.758129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.758129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"1303","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.862895+0000","last_change":"2021-02-10T04:51:44.799512+0000","last_active":"2021-02-10T04:59:00.862895+0000","last_peered":"2021-02-10T04:59:00.862895+0000","last_clean":"2021-02-10T04:59:00.862895+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:59:00.862895+0000","last_undegraded":"2021-02-10T04:59:00.862895+0000","last_fullsized":"2021-02-10T04:59:00.862895+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:44.799429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.799429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1e","version":"1063'654","reported_seq":"4578","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.865124+0000","last_change":"2021-02-10T04:53:40.135450+0000","last_active":"2021-02-10T04:59:00.865124+0000","last_peered":"2021-02-10T04:59:00.865124+0000","last_clean":"2021-02-10T04:59:00.865124+0000","last_became_active":"2021-02-10T04:52:31.087662+0000","last_became_peered":"2021-02-10T04:52:31.087662+0000","last_unstale":"2021-02-10T04:59:00.865124+0000","last_undegraded":"2021-02-10T04:59:00.865124+0000","last_fullsized":"2021-02-10T04:59:00.865124+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:53:40.135367+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:53:40.135367+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"1287","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.573721+0000","last_change":"2021-02-10T04:51:08.006463+0000","last_active":"2021-02-10T04:59:01.573721+0000","last_peered":"2021-02-10T04:59:01.573721+0000","last_clean":"2021-02-10T04:59:01.573721+0000","last_became_active":"2021-02-10T04:50:59.563657+0000","last_became_peered":"2021-02-10T04:50:59.563657+0000","last_unstale":"2021-02-10T04:59:01.573721+0000","last_undegraded":"2021-02-10T04:59:01.573721+0000","last_fullsized":"2021-02-10T04:59:01.573721+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_clean_scrub_stamp":"2021-02-10T04:51:02.455702+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"1338","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.863177+0000","last_change":"2021-02-10T04:52:33.741801+0000","last_active":"2021-02-10T04:59:00.863177+0000","last_peered":"2021-02-10T04:59:00.863177+0000","last_clean":"2021-02-10T04:59:00.863177+0000","last_became_active":"2021-02-10T04:51:36.860129+0000","last_became_peered":"2021-02-10T04:51:36.860129+0000","last_unstale":"2021-02-10T04:59:00.863177+0000","last_undegraded":"2021-02-10T04:59:00.863177+0000","last_fullsized":"2021-02-10T04:59:00.863177+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:33.741711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:46:58.666264+0000","last_clean_scrub_stamp":"2021-02-10T04:52:33.741711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"1299","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.574891+0000","last_change":"2021-02-10T04:52:25.030419+0000","last_active":"2021-02-10T04:59:01.574891+0000","last_peered":"2021-02-10T04:59:01.574891+0000","last_clean":"2021-02-10T04:59:01.574891+0000","last_became_active":"2021-02-10T04:50:59.539582+0000","last_became_peered":"2021-02-10T04:50:59.539582+0000","last_unstale":"2021-02-10T04:59:01.574891+0000","last_undegraded":"2021-02-10T04:59:01.574891+0000","last_fullsized":"2021-02-10T04:59:01.574891+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:25.030281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.030281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"1003'739","reported_seq":"4085","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:01.870946+0000","last_change":"2021-02-10T04:53:01.929261+0000","last_active":"2021-02-10T04:59:01.870946+0000","last_peered":"2021-02-10T04:59:01.870946+0000","last_clean":"2021-02-10T04:59:01.870946+0000","last_became_active":"2021-02-10T04:51:40.511505+0000","last_became_peered":"2021-02-10T04:51:40.511505+0000","last_unstale":"2021-02-10T04:59:01.870946+0000","last_undegraded":"2021-02-10T04:59:01.870946+0000","last_fullsized":"2021-02-10T04:59:01.870946+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:53:01.929169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.929169+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22230846,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":18429397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"1173","reported_epoch":"1333","state":"active+clean","last_fresh":"2021-02-10T04:58:56.827501+0000","last_change":"2021-02-10T04:53:46.727590+0000","last_active":"2021-02-10T04:58:56.827501+0000","last_peered":"2021-02-10T04:58:56.827501+0000","last_clean":"2021-02-10T04:58:56.827501+0000","last_became_active":"2021-02-10T04:53:46.726584+0000","last_became_peered":"2021-02-10T04:53:46.726584+0000","last_unstale":"2021-02-10T04:58:56.827501+0000","last_undegraded":"2021-02-10T04:58:56.827501+0000","last_fullsized":"2021-02-10T04:58:56.827501+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:26.408326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.408326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"1336","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.870954+0000","last_change":"2021-02-10T04:53:07.324650+0000","last_active":"2021-02-10T04:59:00.870954+0000","last_peered":"2021-02-10T04:59:00.870954+0000","last_clean":"2021-02-10T04:59:00.870954+0000","last_became_active":"2021-02-10T04:50:59.575340+0000","last_became_peered":"2021-02-10T04:50:59.575340+0000","last_unstale":"2021-02-10T04:59:00.870954+0000","last_undegraded":"2021-02-10T04:59:00.870954+0000","last_fullsized":"2021-02-10T04:59:00.870954+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.324564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.324564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"1218","reported_epoch":"1333","state":"active+clean+wait","last_fresh":"2021-02-10T04:58:56.956207+0000","last_change":"2021-02-10T04:58:56.954145+0000","last_active":"2021-02-10T04:58:56.956207+0000","last_peered":"2021-02-10T04:58:56.956207+0000","last_clean":"2021-02-10T04:58:56.956207+0000","last_became_active":"2021-02-10T04:58:56.953706+0000","last_became_peered":"2021-02-10T04:58:56.953706+0000","last_unstale":"2021-02-10T04:58:56.956207+0000","last_undegraded":"2021-02-10T04:58:56.956207+0000","last_fullsized":"2021-02-10T04:58:56.956207+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:00.501127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:00.501127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"259'74","reported_seq":"263","reported_epoch":"1333","state":"active+clean+wait","last_fresh":"2021-02-10T04:58:56.873407+0000","last_change":"2021-02-10T04:58:56.871779+0000","last_active":"2021-02-10T04:58:56.873407+0000","last_peered":"2021-02-10T04:58:56.873407+0000","last_clean":"2021-02-10T04:58:56.873407+0000","last_became_active":"2021-02-10T04:58:56.870024+0000","last_became_peered":"2021-02-10T04:58:56.870024+0000","last_unstale":"2021-02-10T04:58:56.873407+0000","last_undegraded":"2021-02-10T04:58:56.873407+0000","last_fullsized":"2021-02-10T04:58:56.873407+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":5,"last_scrub":"259'74","last_scrub_stamp":"2021-02-10T04:52:26.597668+0000","last_deep_scrub":"259'74","last_deep_scrub_stamp":"2021-02-10T04:46:49.745104+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.597668+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1218524,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":28228,"num_write":19,"num_write_kb":7022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"115","reported_epoch":"1334","state":"active+clean","last_fresh":"2021-02-10T04:58:59.849481+0000","last_change":"2021-02-10T04:58:59.426612+0000","last_active":"2021-02-10T04:58:59.849481+0000","last_peered":"2021-02-10T04:58:59.849481+0000","last_clean":"2021-02-10T04:58:59.849481+0000","last_became_active":"2021-02-10T04:58:56.850901+0000","last_became_peered":"2021-02-10T04:58:56.850901+0000","last_unstale":"2021-02-10T04:58:59.849481+0000","last_undegraded":"2021-02-10T04:58:59.849481+0000","last_fullsized":"2021-02-10T04:58:59.849481+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:59.167911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:50:59.167911+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"243","reported_epoch":"1333","state":"active+clean","last_fresh":"2021-02-10T04:58:56.825022+0000","last_change":"2021-02-10T04:53:46.718512+0000","last_active":"2021-02-10T04:58:56.825022+0000","last_peered":"2021-02-10T04:58:56.825022+0000","last_clean":"2021-02-10T04:58:56.825022+0000","last_became_active":"2021-02-10T04:53:46.716811+0000","last_became_peered":"2021-02-10T04:53:46.716811+0000","last_unstale":"2021-02-10T04:58:56.825022+0000","last_undegraded":"2021-02-10T04:58:56.825022+0000","last_fullsized":"2021-02-10T04:58:56.825022+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:45.813676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.813676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"1325","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.864972+0000","last_change":"2021-02-10T04:53:38.363899+0000","last_active":"2021-02-10T04:59:00.864972+0000","last_peered":"2021-02-10T04:59:00.864972+0000","last_clean":"2021-02-10T04:59:00.864972+0000","last_became_active":"2021-02-10T04:50:59.539256+0000","last_became_peered":"2021-02-10T04:50:59.539256+0000","last_unstale":"2021-02-10T04:59:00.864972+0000","last_undegraded":"2021-02-10T04:59:00.864972+0000","last_fullsized":"2021-02-10T04:59:00.864972+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:38.363825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:09.433793+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.363825+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1b","version":"1003'1689","reported_seq":"4671","reported_epoch":"1333","state":"active+clean+wait","last_fresh":"2021-02-10T04:58:56.972245+0000","last_change":"2021-02-10T04:58:56.971785+0000","last_active":"2021-02-10T04:58:56.972245+0000","last_peered":"2021-02-10T04:58:56.972245+0000","last_clean":"2021-02-10T04:58:56.972245+0000","last_became_active":"2021-02-10T04:58:56.970049+0000","last_became_peered":"2021-02-10T04:58:56.970049+0000","last_unstale":"2021-02-10T04:58:56.972245+0000","last_undegraded":"2021-02-10T04:58:56.972245+0000","last_fullsized":"2021-02-10T04:58:56.972245+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:53:10.075879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.075879+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27159495,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"1335","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.574983+0000","last_change":"2021-02-10T04:58:56.966215+0000","last_active":"2021-02-10T04:59:01.574983+0000","last_peered":"2021-02-10T04:59:01.574983+0000","last_clean":"2021-02-10T04:59:01.574983+0000","last_became_active":"2021-02-10T04:58:56.965975+0000","last_became_peered":"2021-02-10T04:58:56.965975+0000","last_unstale":"2021-02-10T04:59:01.574983+0000","last_undegraded":"2021-02-10T04:59:01.574983+0000","last_fullsized":"2021-02-10T04:59:01.574983+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:33.543559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:46:33.543559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"995","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.864765+0000","last_change":"2021-02-10T04:53:04.354508+0000","last_active":"2021-02-10T04:59:00.864765+0000","last_peered":"2021-02-10T04:59:00.864765+0000","last_clean":"2021-02-10T04:59:00.864765+0000","last_became_active":"2021-02-10T04:33:05.169159+0000","last_became_peered":"2021-02-10T04:33:05.169159+0000","last_unstale":"2021-02-10T04:59:00.864765+0000","last_undegraded":"2021-02-10T04:59:00.864765+0000","last_fullsized":"2021-02-10T04:59:00.864765+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:04.354409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:04.354409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"1389","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.861818+0000","last_change":"2021-02-10T04:52:36.777554+0000","last_active":"2021-02-10T04:59:00.861818+0000","last_peered":"2021-02-10T04:59:00.861818+0000","last_clean":"2021-02-10T04:59:00.861818+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:59:00.861818+0000","last_undegraded":"2021-02-10T04:59:00.861818+0000","last_fullsized":"2021-02-10T04:59:00.861818+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:36.777485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:52:36.777485+0000","last_clean_scrub_stamp":"2021-02-10T04:52:36.777485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1a","version":"1001'1528","reported_seq":"5946","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.861487+0000","last_change":"2021-02-10T04:52:31.086406+0000","last_active":"2021-02-10T04:59:00.861487+0000","last_peered":"2021-02-10T04:59:00.861487+0000","last_clean":"2021-02-10T04:59:00.861487+0000","last_became_active":"2021-02-10T04:52:31.085959+0000","last_became_peered":"2021-02-10T04:52:31.085959+0000","last_unstale":"2021-02-10T04:59:00.861487+0000","last_undegraded":"2021-02-10T04:59:00.861487+0000","last_fullsized":"2021-02-10T04:59:00.861487+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46944136,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":358,"num_read_kb":351935,"num_write":286,"num_write_kb":111355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":21311893,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"1208","reported_epoch":"1333","state":"active+clean+wait","last_fresh":"2021-02-10T04:58:56.856032+0000","last_change":"2021-02-10T04:58:56.855666+0000","last_active":"2021-02-10T04:58:56.856032+0000","last_peered":"2021-02-10T04:58:56.856032+0000","last_clean":"2021-02-10T04:58:56.856032+0000","last_became_active":"2021-02-10T04:58:56.855338+0000","last_became_peered":"2021-02-10T04:58:56.855338+0000","last_unstale":"2021-02-10T04:58:56.856032+0000","last_undegraded":"2021-02-10T04:58:56.856032+0000","last_fullsized":"2021-02-10T04:58:56.856032+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:17.919745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:17.919745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"621","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.572487+0000","last_change":"2021-02-10T04:53:07.092032+0000","last_active":"2021-02-10T04:59:01.572487+0000","last_peered":"2021-02-10T04:59:01.572487+0000","last_clean":"2021-02-10T04:59:01.572487+0000","last_became_active":"2021-02-10T04:50:59.572077+0000","last_became_peered":"2021-02-10T04:50:59.572077+0000","last_unstale":"2021-02-10T04:59:01.572487+0000","last_undegraded":"2021-02-10T04:59:01.572487+0000","last_fullsized":"2021-02-10T04:59:01.572487+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.091949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:19.755718+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.091949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"1298","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.573438+0000","last_change":"2021-02-10T04:51:59.838856+0000","last_active":"2021-02-10T04:59:01.573438+0000","last_peered":"2021-02-10T04:59:01.573438+0000","last_clean":"2021-02-10T04:59:01.573438+0000","last_became_active":"2021-02-10T04:50:59.563156+0000","last_became_peered":"2021-02-10T04:50:59.563156+0000","last_unstale":"2021-02-10T04:59:01.573438+0000","last_undegraded":"2021-02-10T04:59:01.573438+0000","last_fullsized":"2021-02-10T04:59:01.573438+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.838790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.838790+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":"1031","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.871190+0000","last_change":"2021-02-10T04:52:30.405170+0000","last_active":"2021-02-10T04:59:00.871190+0000","last_peered":"2021-02-10T04:59:00.871190+0000","last_clean":"2021-02-10T04:59:00.871190+0000","last_became_active":"2021-02-10T04:51:36.852058+0000","last_became_peered":"2021-02-10T04:51:36.852058+0000","last_unstale":"2021-02-10T04:59:00.871190+0000","last_undegraded":"2021-02-10T04:59:00.871190+0000","last_fullsized":"2021-02-10T04:59:00.871190+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.405106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:20.635536+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.405106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.20","version":"0'0","reported_seq":"267","reported_epoch":"1333","state":"active+clean","last_fresh":"2021-02-10T04:58:56.826019+0000","last_change":"2021-02-10T04:53:46.739888+0000","last_active":"2021-02-10T04:58:56.826019+0000","last_peered":"2021-02-10T04:58:56.826019+0000","last_clean":"2021-02-10T04:58:56.826019+0000","last_became_active":"2021-02-10T04:53:46.739532+0000","last_became_peered":"2021-02-10T04:53:46.739532+0000","last_unstale":"2021-02-10T04:58:56.826019+0000","last_undegraded":"2021-02-10T04:58:56.826019+0000","last_fullsized":"2021-02-10T04:58:56.826019+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:56.449085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:41.375276+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.449085+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":"1264","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:01.874085+0000","last_change":"2021-02-10T04:53:35.858634+0000","last_active":"2021-02-10T04:59:01.874085+0000","last_peered":"2021-02-10T04:59:01.874085+0000","last_clean":"2021-02-10T04:59:01.874085+0000","last_became_active":"2021-02-10T04:51:36.869785+0000","last_became_peered":"2021-02-10T04:51:36.869785+0000","last_unstale":"2021-02-10T04:59:01.874085+0000","last_undegraded":"2021-02-10T04:59:01.874085+0000","last_fullsized":"2021-02-10T04:59:01.874085+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.858557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.858557+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":"261","reported_epoch":"1334","state":"active+clean","last_fresh":"2021-02-10T04:58:59.848449+0000","last_change":"2021-02-10T04:58:56.879602+0000","last_active":"2021-02-10T04:58:59.848449+0000","last_peered":"2021-02-10T04:58:59.848449+0000","last_clean":"2021-02-10T04:58:59.848449+0000","last_became_active":"2021-02-10T04:58:56.879153+0000","last_became_peered":"2021-02-10T04:58:56.879153+0000","last_unstale":"2021-02-10T04:58:59.848449+0000","last_undegraded":"2021-02-10T04:58:59.848449+0000","last_fullsized":"2021-02-10T04:58:59.848449+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:55:23.586401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:55:23.586401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":"247","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.863321+0000","last_change":"2021-02-10T04:58:56.966218+0000","last_active":"2021-02-10T04:59:00.863321+0000","last_peered":"2021-02-10T04:59:00.863321+0000","last_clean":"2021-02-10T04:59:00.863321+0000","last_became_active":"2021-02-10T04:58:56.965861+0000","last_became_peered":"2021-02-10T04:58:56.965861+0000","last_unstale":"2021-02-10T04:59:00.863321+0000","last_undegraded":"2021-02-10T04:59:00.863321+0000","last_fullsized":"2021-02-10T04:59:00.863321+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.706372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.706372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":"1346","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:01.872011+0000","last_change":"2021-02-10T04:51:57.287452+0000","last_active":"2021-02-10T04:59:01.872011+0000","last_peered":"2021-02-10T04:59:01.872011+0000","last_clean":"2021-02-10T04:59:01.872011+0000","last_became_active":"2021-02-10T04:50:59.566827+0000","last_became_peered":"2021-02-10T04:50:59.566827+0000","last_unstale":"2021-02-10T04:59:01.872011+0000","last_undegraded":"2021-02-10T04:59:01.872011+0000","last_fullsized":"2021-02-10T04:59:01.872011+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.287363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.287363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":"335","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:01.875151+0000","last_change":"2021-02-10T04:58:56.867715+0000","last_active":"2021-02-10T04:59:01.875151+0000","last_peered":"2021-02-10T04:59:01.875151+0000","last_clean":"2021-02-10T04:59:01.875151+0000","last_became_active":"2021-02-10T04:58:56.867324+0000","last_became_peered":"2021-02-10T04:58:56.867324+0000","last_unstale":"2021-02-10T04:59:01.875151+0000","last_undegraded":"2021-02-10T04:59:01.875151+0000","last_fullsized":"2021-02-10T04:59:01.875151+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:10.308282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:10.308282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":"1258","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.573794+0000","last_change":"2021-02-10T04:58:56.966618+0000","last_active":"2021-02-10T04:59:01.573794+0000","last_peered":"2021-02-10T04:59:01.573794+0000","last_clean":"2021-02-10T04:59:01.573794+0000","last_became_active":"2021-02-10T04:58:56.965458+0000","last_became_peered":"2021-02-10T04:58:56.965458+0000","last_unstale":"2021-02-10T04:59:01.573794+0000","last_undegraded":"2021-02-10T04:59:01.573794+0000","last_fullsized":"2021-02-10T04:59:01.573794+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:09.121298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.121298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.21","version":"293'77","reported_seq":"906","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.572825+0000","last_change":"2021-02-10T04:52:28.062923+0000","last_active":"2021-02-10T04:59:01.572825+0000","last_peered":"2021-02-10T04:59:01.572825+0000","last_clean":"2021-02-10T04:59:01.572825+0000","last_became_active":"2021-02-10T04:52:28.062463+0000","last_became_peered":"2021-02-10T04:52:28.062463+0000","last_unstale":"2021-02-10T04:59:01.572825+0000","last_undegraded":"2021-02-10T04:59:01.572825+0000","last_fullsized":"2021-02-10T04:59:01.572825+0000","mapping_epoch":1143,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1144,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1886586,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":15163,"num_write":9,"num_write_kb":3316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":"231","reported_epoch":"1334","state":"active+clean","last_fresh":"2021-02-10T04:58:59.848373+0000","last_change":"2021-02-10T04:52:32.124955+0000","last_active":"2021-02-10T04:58:59.848373+0000","last_peered":"2021-02-10T04:58:59.848373+0000","last_clean":"2021-02-10T04:58:59.848373+0000","last_became_active":"2021-02-10T04:51:36.867061+0000","last_became_peered":"2021-02-10T04:51:36.867061+0000","last_unstale":"2021-02-10T04:58:59.848373+0000","last_undegraded":"2021-02-10T04:58:59.848373+0000","last_fullsized":"2021-02-10T04:58:59.848373+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:32.124901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:08.271977+0000","last_clean_scrub_stamp":"2021-02-10T04:52:32.124901+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":"1275","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:01.873766+0000","last_change":"2021-02-10T04:51:07.182775+0000","last_active":"2021-02-10T04:59:01.873766+0000","last_peered":"2021-02-10T04:59:01.873766+0000","last_clean":"2021-02-10T04:59:01.873766+0000","last_became_active":"2021-02-10T04:28:23.126368+0000","last_became_peered":"2021-02-10T04:28:23.126368+0000","last_unstale":"2021-02-10T04:59:01.873766+0000","last_undegraded":"2021-02-10T04:59:01.873766+0000","last_fullsized":"2021-02-10T04:59:01.873766+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.182678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.182678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.20","version":"1000'1071","reported_seq":"4679","reported_epoch":"1334","state":"active+clean","last_fresh":"2021-02-10T04:58:59.847919+0000","last_change":"2021-02-10T04:53:03.915465+0000","last_active":"2021-02-10T04:58:59.847919+0000","last_peered":"2021-02-10T04:58:59.847919+0000","last_clean":"2021-02-10T04:58:59.847919+0000","last_became_active":"2021-02-10T04:52:31.088690+0000","last_became_peered":"2021-02-10T04:52:31.088690+0000","last_unstale":"2021-02-10T04:58:59.847919+0000","last_undegraded":"2021-02-10T04:58:59.847919+0000","last_fullsized":"2021-02-10T04:58:59.847919+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:03.915403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.915403+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16398893,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":"856","reported_epoch":"1333","state":"active+clean","last_fresh":"2021-02-10T04:58:56.829323+0000","last_change":"2021-02-10T04:53:46.721599+0000","last_active":"2021-02-10T04:58:56.829323+0000","last_peered":"2021-02-10T04:58:56.829323+0000","last_clean":"2021-02-10T04:58:56.829323+0000","last_became_active":"2021-02-10T04:53:46.721340+0000","last_became_peered":"2021-02-10T04:53:46.721340+0000","last_unstale":"2021-02-10T04:58:56.829323+0000","last_undegraded":"2021-02-10T04:58:56.829323+0000","last_fullsized":"2021-02-10T04:58:56.829323+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:28.092288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:07.767250+0000","last_clean_scrub_stamp":"2021-02-10T04:52:28.092288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.25","version":"53'2","reported_seq":"1290","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.863326+0000","last_change":"2021-02-10T04:53:36.103026+0000","last_active":"2021-02-10T04:59:00.863326+0000","last_peered":"2021-02-10T04:59:00.863326+0000","last_clean":"2021-02-10T04:59:00.863326+0000","last_became_active":"2021-02-10T04:28:23.467770+0000","last_became_peered":"2021-02-10T04:28:23.467770+0000","last_unstale":"2021-02-10T04:59:00.863326+0000","last_undegraded":"2021-02-10T04:59:00.863326+0000","last_fullsized":"2021-02-10T04:59:00.863326+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:53:36.102928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:36.102928+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.23","version":"1002'1657","reported_seq":"5207","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:01.872813+0000","last_change":"2021-02-10T04:52:31.093874+0000","last_active":"2021-02-10T04:59:01.872813+0000","last_peered":"2021-02-10T04:59:01.872813+0000","last_clean":"2021-02-10T04:59:01.872813+0000","last_became_active":"2021-02-10T04:52:31.092996+0000","last_became_peered":"2021-02-10T04:52:31.092996+0000","last_unstale":"2021-02-10T04:59:01.872813+0000","last_undegraded":"2021-02-10T04:59:01.872813+0000","last_fullsized":"2021-02-10T04:59:01.872813+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21396533,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":212,"num_read_kb":218837,"num_write":159,"num_write_kb":59399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":20479176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":"1382","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:01.871783+0000","last_change":"2021-02-10T04:53:12.951612+0000","last_active":"2021-02-10T04:59:01.871783+0000","last_peered":"2021-02-10T04:59:01.871783+0000","last_clean":"2021-02-10T04:59:01.871783+0000","last_became_active":"2021-02-10T04:50:59.566548+0000","last_became_peered":"2021-02-10T04:50:59.566548+0000","last_unstale":"2021-02-10T04:59:01.871783+0000","last_undegraded":"2021-02-10T04:59:01.871783+0000","last_fullsized":"2021-02-10T04:59:01.871783+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.951528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.951528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.26","version":"53'1","reported_seq":"170","reported_epoch":"1333","state":"active+clean","last_fresh":"2021-02-10T04:58:56.825538+0000","last_change":"2021-02-10T04:53:46.722636+0000","last_active":"2021-02-10T04:58:56.825538+0000","last_peered":"2021-02-10T04:58:56.825538+0000","last_clean":"2021-02-10T04:58:56.825538+0000","last_became_active":"2021-02-10T04:53:46.722227+0000","last_became_peered":"2021-02-10T04:53:46.722227+0000","last_unstale":"2021-02-10T04:58:56.825538+0000","last_undegraded":"2021-02-10T04:58:56.825538+0000","last_fullsized":"2021-02-10T04:58:56.825538+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:53:03.933910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.933910+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":423,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.22","version":"998'1009","reported_seq":"4697","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:00.861576+0000","last_change":"2021-02-10T04:52:31.086563+0000","last_active":"2021-02-10T04:59:00.861576+0000","last_peered":"2021-02-10T04:59:00.861576+0000","last_clean":"2021-02-10T04:59:00.861576+0000","last_became_active":"2021-02-10T04:52:31.085288+0000","last_became_peered":"2021-02-10T04:52:31.085288+0000","last_unstale":"2021-02-10T04:59:00.861576+0000","last_undegraded":"2021-02-10T04:59:00.861576+0000","last_fullsized":"2021-02-10T04:59:00.861576+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12355207,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":120,"num_read_kb":124702,"num_write":77,"num_write_kb":25052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4145029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":"1264","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:01.872730+0000","last_change":"2021-02-10T04:53:10.958668+0000","last_active":"2021-02-10T04:59:01.872730+0000","last_peered":"2021-02-10T04:59:01.872730+0000","last_clean":"2021-02-10T04:59:01.872730+0000","last_became_active":"2021-02-10T04:45:49.552026+0000","last_became_peered":"2021-02-10T04:45:49.552026+0000","last_unstale":"2021-02-10T04:59:01.872730+0000","last_undegraded":"2021-02-10T04:59:01.872730+0000","last_fullsized":"2021-02-10T04:59:01.872730+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:10.958579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.958579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":"275","reported_epoch":"1333","state":"active+clean","last_fresh":"2021-02-10T04:58:56.826407+0000","last_change":"2021-02-10T04:53:46.725941+0000","last_active":"2021-02-10T04:58:56.826407+0000","last_peered":"2021-02-10T04:58:56.826407+0000","last_clean":"2021-02-10T04:58:56.826407+0000","last_became_active":"2021-02-10T04:53:46.725587+0000","last_became_peered":"2021-02-10T04:53:46.725587+0000","last_unstale":"2021-02-10T04:58:56.826407+0000","last_undegraded":"2021-02-10T04:58:56.826407+0000","last_fullsized":"2021-02-10T04:58:56.826407+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:41.758684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.758684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":"1272","reported_epoch":"1335","state":"active+clean","last_fresh":"2021-02-10T04:59:01.572758+0000","last_change":"2021-02-10T04:55:39.599193+0000","last_active":"2021-02-10T04:59:01.572758+0000","last_peered":"2021-02-10T04:59:01.572758+0000","last_clean":"2021-02-10T04:59:01.572758+0000","last_became_active":"2021-02-10T04:55:39.598816+0000","last_became_peered":"2021-02-10T04:55:39.598816+0000","last_unstale":"2021-02-10T04:59:01.572758+0000","last_undegraded":"2021-02-10T04:59:01.572758+0000","last_fullsized":"2021-02-10T04:59:01.572758+0000","mapping_epoch":1248,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:11.139602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.139602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":"1290","reported_epoch":"1336","state":"active+clean","last_fresh":"2021-02-10T04:59:01.871086+0000","last_change":"2021-02-10T04:52:24.072005+0000","last_active":"2021-02-10T04:59:01.871086+0000","last_peered":"2021-02-10T04:59:01.871086+0000","last_clean":"2021-02-10T04:59:01.871086+0000","last_became_active":"2021-02-10T04:50:59.527945+0000","last_became_peered":"2021-02-10T04:50:59.527945+0000","last_unstale":"2021-02-10T04:59:01.871086+0000","last_undegraded":"2021-02-10T04:59:01.871086+0000","last_fullsized":"2021-02-10T04:59:01.871086+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:24.071915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_clean_scrub_stamp":"2021-02-10T04:52:24.071915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":936229565,"num_objects":377,"num_object_clones":326,"num_object_copies":1131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":377,"num_whiteouts":10,"num_read":4055,"num_read_kb":4149582,"num_write":2946,"num_write_kb":1137016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":575,"num_bytes_recovered":289511227,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2029219840,"data_stored":2597935849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1639,"internal_metadata":0},"log_size":34722,"ondisk_log_size":34722,"up":108,"acting":108,"num_store_stats":8},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":2368,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":18,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1139,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":5835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31,"ondisk_log_size":31,"up":126,"acting":126,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":13,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":862,"num_read_kb":857,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20,"ondisk_log_size":20,"up":39,"acting":39,"num_store_stats":8},{"poolid":4,"num_pg":22,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":2714,"num_read_kb":2187,"num_write":1592,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":17736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":844,"ondisk_log_size":844,"up":66,"acting":66,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40,"ondisk_log_size":40,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":126,"acting":126,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365203,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":196804,"kb_used_data":149692,"kb_used_omap":0,"kb_used_meta":47104,"kb_avail":93548348,"statfs":{"total":95995035648,"available":95793508352,"internally_reserved":0,"allocated":153284608,"data_stored":202661702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48234496},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:57:20 2021","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.89500000000000002,"15min":0.92800000000000005},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.98999999999999999,"5min":4.4340000000000002,"15min":4.4340000000000002},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.73399999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:58:48 2021","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.02,"15min":0.96099999999999997},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":15.253,"5min":15.253,"15min":15.253},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.669,"5min":1.105,"15min":1.022},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.45500000000000002},"max":{"1min":18.713000000000001,"5min":18.713000000000001,"15min":18.713000000000001},"last":0.76200000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:58:48 2021","interfaces":[{"interface":"back","average":{"1min":3.3239999999999998,"5min":2.8490000000000002,"15min":1.7070000000000001},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":40.536999999999999,"5min":94.391999999999996,"15min":94.391999999999996},"last":0.91500000000000004},{"interface":"front","average":{"1min":3.2959999999999998,"5min":2.7839999999999998,"15min":1.7110000000000001},"min":{"1min":0.54300000000000004,"5min":0.496,"15min":0.496},"max":{"1min":40.429000000000002,"5min":94.269999999999996,"15min":94.269999999999996},"last":1.0529999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:58:03 2021","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.0569999999999999,"15min":1.244},"min":{"1min":0.73899999999999999,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.4510000000000001,"5min":4.5599999999999996,"15min":4.5890000000000004},"last":1.083},{"interface":"front","average":{"1min":1.252,"5min":1.1910000000000001,"15min":1.357},"min":{"1min":0.78200000000000003,"5min":0.629,"15min":0.629},"max":{"1min":2.0499999999999998,"5min":4.6130000000000004,"15min":4.6609999999999996},"last":1.02}]},{"osd":4,"last update":"Wed Feb 10 04:58:48 2021","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.841,"15min":1.5429999999999999},"min":{"1min":0.67000000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":9.9659999999999993,"5min":72.662999999999997,"15min":72.662999999999997},"last":1.1850000000000001},{"interface":"front","average":{"1min":2.1949999999999998,"5min":2.1309999999999998,"15min":1.6100000000000001},"min":{"1min":0.76500000000000001,"5min":0.73999999999999999,"15min":0.69099999999999995},"max":{"1min":20.542999999999999,"5min":73.564999999999998,"15min":73.564999999999998},"last":1.1679999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:58:48 2021","interfaces":[{"interface":"back","average":{"1min":1.7929999999999999,"5min":1.4910000000000001,"15min":1.5209999999999999},"min":{"1min":0.74099999999999999,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":11.627000000000001,"5min":11.627000000000001,"15min":14.972},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.8919999999999999,"5min":1.377,"15min":1.595},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.72699999999999998},"max":{"1min":13.172000000000001,"5min":13.172000000000001,"15min":15.218999999999999},"last":1.72}]},{"osd":6,"last update":"Wed Feb 10 04:58:48 2021","interfaces":[{"interface":"back","average":{"1min":1.7769999999999999,"5min":1.3660000000000001,"15min":1.6180000000000001},"min":{"1min":0.84399999999999997,"5min":0.78200000000000003,"15min":0.72699999999999998},"max":{"1min":10.286,"5min":10.286,"15min":16.050000000000001},"last":1.3080000000000001},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.5389999999999999,"15min":1.6579999999999999},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.876},"max":{"1min":17.460000000000001,"5min":17.460000000000001,"15min":17.460000000000001},"last":1.252}]}]},{"osd":6,"up_from":43,"seq":184683594136,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":393328,"kb_used_data":342824,"kb_used_omap":0,"kb_used_meta":50496,"kb_avail":93351824,"statfs":{"total":95995035648,"available":95592267776,"internally_reserved":0,"allocated":351051776,"data_stored":443647370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":51707904},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:57:18 2021","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.88100000000000001,"15min":0.88800000000000001},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.6040000000000001,"5min":1.6599999999999999,"15min":1.6599999999999999},"last":0.97199999999999998}]},{"osd":1,"last update":"Wed Feb 10 04:58:28 2021","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.91600000000000004,"15min":1.0249999999999999},"min":{"1min":0.51800000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.8169999999999999,"5min":1.8169999999999999,"15min":13.18},"last":0.80700000000000005},{"interface":"front","average":{"1min":1.1479999999999999,"5min":0.92700000000000005,"15min":1.077},"min":{"1min":0.61099999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":6.5270000000000001,"5min":6.5270000000000001,"15min":11.041},"last":0.92400000000000004}]},{"osd":2,"last update":"Wed Feb 10 04:58:28 2021","interfaces":[{"interface":"back","average":{"1min":3.2959999999999998,"5min":2.157,"15min":2.0630000000000002},"min":{"1min":0.56299999999999994,"5min":0.503,"15min":0.503},"max":{"1min":45.514000000000003,"5min":69.653000000000006,"15min":163.81700000000001},"last":1.3260000000000001},{"interface":"front","average":{"1min":3.7450000000000001,"5min":2.3359999999999999,"15min":2.1360000000000001},"min":{"1min":0.75700000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":45.347999999999999,"5min":69.561000000000007,"15min":164.56800000000001},"last":1.0660000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:58:15 2021","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.1259999999999999,"15min":1.1779999999999999},"min":{"1min":0.79600000000000004,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.5720000000000001,"5min":1.944,"15min":1.944},"last":1.129},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.171,"15min":1.147},"min":{"1min":0.73799999999999999,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":6.6130000000000004,"5min":6.6130000000000004,"15min":6.6130000000000004},"last":1.0309999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:58:28 2021","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.0980000000000001,"15min":1.3049999999999999},"min":{"1min":0.66300000000000003,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.615,"5min":2.1120000000000001,"15min":7.1459999999999999},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.0149999999999999,"5min":0.95799999999999996,"15min":1.0860000000000001},"min":{"1min":0.63900000000000001,"5min":0.57399999999999995,"15min":0.56000000000000005},"max":{"1min":1.5009999999999999,"5min":1.958,"15min":11.997999999999999},"last":0.87}]},{"osd":5,"last update":"Wed Feb 10 04:58:28 2021","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.3480000000000001,"15min":1.6419999999999999},"min":{"1min":0.69699999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.7629999999999999,"5min":22.663,"15min":81.540000000000006},"last":1.5009999999999999},{"interface":"front","average":{"1min":1.353,"5min":1.6000000000000001,"15min":1.9239999999999999},"min":{"1min":0.91400000000000003,"5min":0.85699999999999998,"15min":0.78700000000000003},"max":{"1min":1.9259999999999999,"5min":22.933,"15min":81.492999999999995},"last":1.524}]},{"osd":7,"last update":"Wed Feb 10 04:58:50 2021","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.121,"15min":1.258},"min":{"1min":0.76100000000000001,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.645,"5min":2.073,"15min":5.1820000000000004},"last":1.254},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.3939999999999999,"15min":1.5169999999999999},"min":{"1min":0.98299999999999998,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":2.0659999999999998,"5min":2.2970000000000002,"15min":5.4109999999999996},"last":1.393}]}]},{"osd":1,"up_from":13,"seq":55834575273,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":262756,"kb_used_data":206812,"kb_used_omap":0,"kb_used_meta":55936,"kb_avail":93482396,"statfs":{"total":95995035648,"available":95725973504,"internally_reserved":0,"allocated":211775488,"data_stored":266142438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":57278464},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:57:18 2021","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.69799999999999995,"15min":0.68899999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":2.3439999999999999,"5min":2.3439999999999999,"15min":2.3439999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.996,"5min":0.93899999999999995,"15min":0.92900000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":2.7589999999999999,"5min":2.7589999999999999,"15min":2.7589999999999999},"last":0.72799999999999998}]},{"osd":2,"last update":"Wed Feb 10 04:58:22 2021","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":5.4390000000000001,"15min":3.2810000000000001},"min":{"1min":0.54200000000000004,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.3540000000000001,"5min":162.61199999999999,"15min":162.61199999999999},"last":1.3460000000000001},{"interface":"front","average":{"1min":0.83699999999999997,"5min":5.5789999999999997,"15min":3.371},"min":{"1min":0.57699999999999996,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.504,"5min":162.762,"15min":162.762},"last":1.7669999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:58:07 2021","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":0.81799999999999995,"15min":0.90400000000000003},"min":{"1min":0.65500000000000003,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":2.8919999999999999,"5min":2.8919999999999999,"15min":3.077},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.84299999999999997,"15min":0.97399999999999998},"min":{"1min":0.58999999999999997,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":3.1890000000000001,"5min":3.1890000000000001,"15min":3.1890000000000001},"last":1.4159999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:58:03 2021","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.917,"15min":1.415},"min":{"1min":0.63100000000000001,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":3.101,"5min":100.071,"15min":100.071},"last":1.837},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.9370000000000001,"15min":1.5349999999999999},"min":{"1min":0.55700000000000005,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":3.5390000000000001,"5min":100.43899999999999,"15min":100.43899999999999},"last":1.6990000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:58:07 2021","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.1240000000000001,"15min":1.159},"min":{"1min":0.96199999999999997,"5min":0.59799999999999998,"15min":0.56599999999999995},"max":{"1min":3.0219999999999998,"5min":3.0219999999999998,"15min":5.0300000000000002},"last":1.5249999999999999},{"interface":"front","average":{"1min":1.294,"5min":1.179,"15min":1.4219999999999999},"min":{"1min":0.84499999999999997,"5min":0.70399999999999996,"15min":0.68600000000000005},"max":{"1min":3.2410000000000001,"5min":6.032,"15min":28.678000000000001},"last":1.9319999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:58:22 2021","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.3799999999999999,"15min":1.4670000000000001},"min":{"1min":0.69099999999999995,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.5669999999999999,"5min":30.997,"15min":30.997},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.254,"5min":1.46,"15min":1.468},"min":{"1min":0.91200000000000003,"5min":0.64200000000000002,"15min":0.60499999999999998},"max":{"1min":1.8460000000000001,"5min":31.138000000000002,"15min":31.138000000000002},"last":1.893}]},{"osd":7,"last update":"Wed Feb 10 04:58:30 2021","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.2410000000000001,"15min":1.347},"min":{"1min":0.85099999999999998,"5min":0.72299999999999998,"15min":0.628},"max":{"1min":1.663,"5min":6.2629999999999999,"15min":6.2629999999999999},"last":2.052},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.2230000000000001,"15min":1.353},"min":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.78700000000000003},"max":{"1min":1.522,"5min":3.4129999999999998,"15min":3.754},"last":1.5029999999999999}]}]},{"osd":0,"up_from":1332,"seq":5720896438274,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":367208,"kb_used_data":357152,"kb_used_omap":324,"kb_used_meta":9723,"kb_avail":93377944,"statfs":{"total":95995035648,"available":95619014656,"internally_reserved":0,"allocated":365723648,"data_stored":480533127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":332393,"internal_metadata":9956759},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.121}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.365}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2150000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411751,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":359556,"kb_used_data":318972,"kb_used_omap":0,"kb_used_meta":40576,"kb_avail":93385596,"statfs":{"total":95995035648,"available":95626850304,"internally_reserved":0,"allocated":326627328,"data_stored":375156774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41549824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:57:18 2021","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.67200000000000004,"15min":0.67000000000000004},"min":{"1min":0.50800000000000001,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.94799999999999995,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.93799999999999994,"15min":0.93999999999999995},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.9950000000000001,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":0.72799999999999998}]},{"osd":1,"last update":"Wed Feb 10 04:57:58 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.83699999999999997,"15min":0.95599999999999996},"min":{"1min":0.47599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.26,"5min":17.041,"15min":17.041},"last":0.84499999999999997},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.014,"15min":0.93300000000000005},"min":{"1min":0.61199999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":2.1019999999999999,"5min":21.172000000000001,"15min":21.172000000000001},"last":0.59399999999999997}]},{"osd":3,"last update":"Wed Feb 10 04:58:21 2021","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":1.018,"15min":1.0940000000000001},"min":{"1min":0.53100000000000003,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.631,"5min":18.904,"15min":18.904},"last":1.242},{"interface":"front","average":{"1min":1.073,"5min":1.2669999999999999,"15min":1.2909999999999999},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.742,"5min":19.241,"15min":19.241},"last":1.577}]},{"osd":4,"last update":"Wed Feb 10 04:58:38 2021","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.2110000000000001,"15min":1.248},"min":{"1min":0.621,"5min":0.621,"15min":0.61299999999999999},"max":{"1min":1.851,"5min":9.2029999999999994,"15min":9.2029999999999994},"last":1.4670000000000001},{"interface":"front","average":{"1min":1.018,"5min":1.3220000000000001,"15min":1.1739999999999999},"min":{"1min":0.58299999999999996,"5min":0.503,"15min":0.503},"max":{"1min":1.716,"5min":24.373999999999999,"15min":24.373999999999999},"last":0.77800000000000002}]},{"osd":5,"last update":"Wed Feb 10 04:57:58 2021","interfaces":[{"interface":"back","average":{"1min":1.76,"5min":1.675,"15min":1.7490000000000001},"min":{"1min":0.89500000000000002,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":7.9050000000000002,"5min":39.911999999999999,"15min":39.911999999999999},"last":1.353},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.3660000000000001,"15min":1.488},"min":{"1min":0.69399999999999995,"5min":0.65900000000000003,"15min":0.57599999999999996},"max":{"1min":7.9969999999999999,"5min":33.539999999999999,"15min":33.539999999999999},"last":0.88400000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:58:33 2021","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0620000000000001,"15min":1.341},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.6280000000000001,"5min":3.1659999999999999,"15min":20.007000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.298,"5min":1.4570000000000001,"15min":1.5900000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":2.6579999999999999,"5min":16.135000000000002,"15min":19.417999999999999},"last":1.6910000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:58:38 2021","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.645,"15min":1.609},"min":{"1min":0.84999999999999998,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":2.1389999999999998,"5min":24.405000000000001,"15min":24.405000000000001},"last":1.8009999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.216,"15min":1.2549999999999999},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.829,"5min":3.6640000000000001,"15min":4.9189999999999996},"last":1.0720000000000001}]}]},{"osd":3,"up_from":1111,"seq":4771708665954,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":317744,"kb_used_data":301736,"kb_used_omap":386,"kb_used_meta":15613,"kb_avail":93427408,"statfs":{"total":95995035648,"available":95669665792,"internally_reserved":0,"allocated":308977664,"data_stored":410085331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":396036,"internal_metadata":15987964},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:57:18 2021","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":1.724,"15min":1.8939999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.004,"5min":22.869,"15min":22.869},"last":0.98299999999999998},{"interface":"front","average":{"1min":0.72099999999999997,"5min":1.446,"15min":1.5669999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.98099999999999998,"5min":13.587999999999999,"15min":13.587999999999999},"last":0.72399999999999998}]},{"osd":1,"last update":"Wed Feb 10 04:58:11 2021","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.77100000000000002,"15min":0.69099999999999995},"min":{"1min":0.52000000000000002,"5min":0.44,"15min":0.44},"max":{"1min":2.984,"5min":3.6629999999999998,"15min":3.6629999999999998},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.79900000000000004,"15min":0.85299999999999998},"min":{"1min":0.51900000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.107,"5min":3.7050000000000001,"15min":3.7050000000000001},"last":0.60399999999999998}]},{"osd":2,"last update":"Wed Feb 10 04:58:11 2021","interfaces":[{"interface":"back","average":{"1min":1.9770000000000001,"5min":1.577,"15min":1.2669999999999999},"min":{"1min":0.56799999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":22.731000000000002,"5min":31.100999999999999,"15min":31.100999999999999},"last":0.93000000000000005},{"interface":"front","average":{"1min":2.089,"5min":1.8100000000000001,"15min":1.341},"min":{"1min":0.72699999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":22.687000000000001,"5min":31.052,"15min":31.052},"last":0.96399999999999997}]},{"osd":4,"last update":"Wed Feb 10 04:58:11 2021","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.0109999999999999,"15min":0.98199999999999998},"min":{"1min":0.76800000000000002,"5min":0.628,"15min":0.628},"max":{"1min":1.353,"5min":2.9430000000000001,"15min":2.9430000000000001},"last":1.2250000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.0620000000000001,"15min":1.091},"min":{"1min":0.60399999999999998,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":5.6369999999999996,"5min":5.6369999999999996,"15min":5.6369999999999996},"last":1.097}]},{"osd":5,"last update":"Wed Feb 10 04:58:11 2021","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.1719999999999999,"15min":2.3029999999999999},"min":{"1min":0.71799999999999997,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":5.7489999999999997,"5min":5.7489999999999997,"15min":52.353000000000002},"last":1.345},{"interface":"front","average":{"1min":1.03,"5min":1.022,"15min":2.1179999999999999},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.69599999999999995},"max":{"1min":1.7929999999999999,"5min":3.544,"15min":52.301000000000002},"last":0.81599999999999995}]},{"osd":6,"last update":"Wed Feb 10 04:58:11 2021","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.077,"15min":4.056},"min":{"1min":0.747,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.4399999999999999,"5min":1.581,"15min":64.376999999999995},"last":1.069},{"interface":"front","average":{"1min":1.171,"5min":1.258,"15min":4.0990000000000002},"min":{"1min":0.94199999999999995,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":1.5669999999999999,"5min":3.3889999999999998,"15min":64.438999999999993},"last":1.653}]},{"osd":7,"last update":"Wed Feb 10 04:58:11 2021","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.095,"15min":1.109},"min":{"1min":0.874,"5min":0.77500000000000002,"15min":0.754},"max":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.141,"5min":1.1839999999999999,"15min":1.157},"min":{"1min":0.91300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.5840000000000001,"5min":2.9569999999999999,"15min":2.9569999999999999},"last":1.472}]}]},{"osd":4,"up_from":31,"seq":133143986591,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":166032,"kb_used_data":120968,"kb_used_omap":0,"kb_used_meta":45056,"kb_avail":93579120,"statfs":{"total":95995035648,"available":95825018880,"internally_reserved":0,"allocated":123871232,"data_stored":155986691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":46137344},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:57:16 2021","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.85799999999999998,"15min":0.86499999999999999},"min":{"1min":0.53700000000000003,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.5449999999999999,"5min":3.637,"15min":3.637},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.878,"15min":0.89000000000000001},"min":{"1min":0.61399999999999999,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.2949999999999999,"5min":4.6829999999999998,"15min":4.6829999999999998},"last":1.081}]},{"osd":1,"last update":"Wed Feb 10 04:58:50 2021","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76200000000000001,"15min":0.85299999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.5409999999999999,"5min":3.2650000000000001,"15min":7.883},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.89400000000000002,"15min":1.0129999999999999},"min":{"1min":0.47999999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.46,"5min":1.6779999999999999,"15min":4.3449999999999998},"last":0.96099999999999997}]},{"osd":2,"last update":"Wed Feb 10 04:58:50 2021","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":1.177,"15min":1.6819999999999999},"min":{"1min":0.60299999999999998,"5min":0.53900000000000003,"15min":0.51000000000000001},"max":{"1min":1.397,"5min":11.776,"15min":120.39},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.83599999999999997,"5min":1.129,"15min":1.6599999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.4339999999999999,"5min":11.795999999999999,"15min":120.583},"last":0.82499999999999996}]},{"osd":3,"last update":"Wed Feb 10 04:58:05 2021","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.107,"15min":1.2290000000000001},"min":{"1min":0.69599999999999995,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.581,"5min":1.998,"15min":2.1429999999999998},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.171,"5min":1.534,"15min":1.516},"min":{"1min":0.89800000000000002,"5min":0.65200000000000002,"15min":0.57399999999999995},"max":{"1min":1.6479999999999999,"5min":26.802,"15min":26.802},"last":1.454}]},{"osd":5,"last update":"Wed Feb 10 04:58:05 2021","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.046,"15min":1.431},"min":{"1min":0.628,"5min":0.495,"15min":0.495},"max":{"1min":1.7010000000000001,"5min":2.351,"15min":35.774000000000001},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.1890000000000001,"15min":1.431},"min":{"1min":0.66900000000000004,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.6990000000000001,"5min":4.319,"15min":35.606000000000002},"last":1.1879999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:58:05 2021","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.464,"15min":2.3679999999999999},"min":{"1min":1.1120000000000001,"5min":0.873,"15min":0.86199999999999999},"max":{"1min":2.073,"5min":3.8530000000000002,"15min":201.626},"last":1.7609999999999999},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.3049999999999999,"15min":2.351},"min":{"1min":0.96899999999999997,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.8480000000000001,"5min":4.7910000000000004,"15min":201.577},"last":1.427}]},{"osd":7,"last update":"Wed Feb 10 04:58:37 2021","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.4419999999999999,"15min":1.498},"min":{"1min":0.94499999999999995,"5min":0.878,"15min":0.67100000000000004},"max":{"1min":1.9239999999999999,"5min":4.6200000000000001,"15min":5.8920000000000003},"last":1.6299999999999999},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.25,"15min":1.4339999999999999},"min":{"1min":0.80100000000000005,"5min":0.78700000000000003,"15min":0.63200000000000001},"max":{"1min":1.5669999999999999,"5min":3.423,"15min":5.149},"last":1.046}]}]},{"osd":5,"up_from":37,"seq":158913790364,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":257576,"kb_used_data":206752,"kb_used_omap":0,"kb_used_meta":50816,"kb_avail":93487576,"statfs":{"total":95995035648,"available":95731277824,"internally_reserved":0,"allocated":211714048,"data_stored":276067130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52035584},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:57:18 2021","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.1799999999999999,"5min":1.486,"15min":1.486},"last":0.753},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.76400000000000001,"15min":0.753},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":1.1379999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:58:04 2021","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.90900000000000003,"15min":0.90300000000000002},"min":{"1min":0.54500000000000004,"5min":0.435,"15min":0.435},"max":{"1min":1.3069999999999999,"5min":6.391,"15min":6.391},"last":0.98799999999999999},{"interface":"front","average":{"1min":0.83099999999999996,"5min":1.0620000000000001,"15min":1.147},"min":{"1min":0.54200000000000004,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.653,"5min":13.794,"15min":13.794},"last":0.94299999999999995}]},{"osd":2,"last update":"Wed Feb 10 04:58:04 2021","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":3.4500000000000002,"15min":2.1640000000000001},"min":{"1min":0.61899999999999999,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.478,"5min":116.438,"15min":116.438},"last":0.65100000000000002},{"interface":"front","average":{"1min":1.004,"5min":3.569,"15min":2.21},"min":{"1min":0.61199999999999999,"5min":0.56200000000000006,"15min":0.55200000000000005},"max":{"1min":1.5609999999999999,"5min":116.96899999999999,"15min":116.96899999999999},"last":1.109}]},{"osd":3,"last update":"Wed Feb 10 04:58:14 2021","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.3580000000000001,"15min":1.323},"min":{"1min":0.73799999999999999,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":2.1619999999999999,"5min":13.755000000000001,"15min":13.755000000000001},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.071,"5min":1.2070000000000001,"15min":1.1910000000000001},"min":{"1min":0.81999999999999995,"5min":0.73199999999999998,"15min":0.64700000000000002},"max":{"1min":1.5289999999999999,"5min":6.5129999999999999,"15min":6.5129999999999999},"last":1.2370000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:58:04 2021","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.0740000000000001,"15min":1.26},"min":{"1min":0.66600000000000004,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.865,"5min":5.298,"15min":9.4890000000000008},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.23,"5min":1.2110000000000001,"15min":1.343},"min":{"1min":0.80400000000000005,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":2.238,"5min":4.8289999999999997,"15min":5.6180000000000003},"last":1.0309999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:58:14 2021","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.3560000000000001,"15min":2.9359999999999999},"min":{"1min":0.88600000000000001,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.294,"5min":4.9429999999999996,"15min":319.51999999999998},"last":1.417},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.3420000000000001,"15min":2.8759999999999999},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.68799999999999994},"max":{"1min":1.8380000000000001,"5min":6.8220000000000001,"15min":319.22300000000001},"last":1.502}]},{"osd":7,"last update":"Wed Feb 10 04:58:42 2021","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.24,"15min":1.4430000000000001},"min":{"1min":0.82299999999999995,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.4020000000000001,"5min":5.4160000000000004,"15min":11.897},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4119999999999999,"15min":1.5309999999999999},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":3.246,"5min":5.0579999999999998,"15min":6.0369999999999999},"last":1.1859999999999999}]}]}]}} 2021-02-10T04:59:06.148 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2021-02-10T04:59:06.148 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pool set zone.rgw.log pg_num 32 2021-02-10T04:59:06.149 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:05 smithi164 conmon[29288]: cluster 2021-02-10T04:59:04.560167+0000 2021-02-10T04:59:06.149 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:05 smithi164 conmon[29288]: mgr.y (mgr.14140) 1509 : cluster [DBG] pgmap v2450: 198 pgs: 1 active+clean+scrubbing, 23 active+clean+wait, 174 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:59:06.149 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:05 smithi164 conmon[29288]: audit 2021-02-10T04:59:05.091148+0000 mgr.y (mgr.14140) 1510 : audit [DBG] from='client.16527 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:59:06.373 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:59:07.886 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is set 2021-02-10T04:59:08.029 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:59:08.049 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:59:08.068 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:59:08.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:07 smithi085 conmon[29319]: cluster 2021-02-10T04:59:08.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:07 smithi085 conmon[29319]: 2021-02-10T04:59: 2021-02-10T04:59:08.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:07 smithi085 conmon[29319]: 06.560733+0000 mgr.y (mgr.14140) 1511 : cluster [DBG] pgmap v2451: 198 pgs: 1 active+clean+scrubbing, 23 active+clean+wait, 174 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:59:08.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:07 smithi085 conmon[29319]: audit 2021 2021-02-10T04:59:08.083 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:07 smithi085 conmon[29319]: -02-10T04:59:07.374633+0000 mon.c ( 2021-02-10T04:59:08.083 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:07 smithi085 conmon[29319]: mon.1) 157 : audit [INF] from='client.? 172.21.15.85:0/2925794816' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:59:08.083 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:07 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:59:08.083 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:07 smithi085 conmon[29319]: -10T04:59:07.375010+0000 mon.a (mon.0 2021-02-10T04:59:08.083 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:07 smithi085 conmon[29319]: ) 2972 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:59:08.083 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:07 smithi085 conmon[35961]: cluster 2021-02-10T04:59:06.560733+0000 2021-02-10T04:59:08.083 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:07 smithi085 conmon[35961]: mgr.y (mgr.14140) 1511 : cluster [DBG] pgmap v2451: 198 pgs: 1 active+clean+scrubbing, 23 active+clean+wait, 174 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:59:08.083 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:07 smithi085 conmon[35961]: audit 2021-02-10T04:59:07.374633+0000 mon.c (mon.1) 157 : audit [INF] from='client.? 172.21.15.85:0/2925794816' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:59:08.083 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:07 smithi085 conmon[35961]: audit 2021-02-10T04:59:07.375010+0000 mon.a (mon.0) 2972 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:59:08.089 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:59:08.112 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:59:08.134 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:59:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:07 smithi164 conmon[29288]: cluster 2021-02-10T04:59:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:07 smithi164 conmon[29288]: 2021-02-10T04:59:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:07 smithi164 conmon[29288]: 2021 2021-02-10T04:59:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:07 smithi164 conmon[29288]: - 2021-02-10T04:59:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:07 smithi164 conmon[29288]: 02 2021-02-10T04:59:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:07 smithi164 conmon[29288]: - 2021-02-10T04:59:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:07 smithi164 conmon[29288]: 10T04:59:06.560733 2021-02-10T04:59:08.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:07 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1511 : cluster [DBG] pgmap v2451: 198 pgs: 1 active+clean+scrubbing, 23 active+clean+wait, 174 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:59:08.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:07 smithi164 conmon[29288]: audit 2021 2021-02-10T04:59:08.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:07 smithi164 conmon[29288]: -02-10T04:59:07. 2021-02-10T04:59:08.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:07 smithi164 conmon[29288]: 374633+0000 mon.c (mon.1 2021-02-10T04:59:08.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:07 smithi164 conmon[29288]: ) 157 : audit [INF] from='client.? 172.21.15.85:0/2925794816' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:59:08.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:07 smithi164 conmon[29288]: 2021-02-10T04:59:08.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:07 smithi164 conmon[29288]: audit 2021 2021-02-10T04:59:08.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:07 smithi164 conmon[29288]: -02-10T04:59 2021-02-10T04:59:08.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:07 smithi164 conmon[29288]: :07.375010+0000 mon.a ( 2021-02-10T04:59:08.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:07 smithi164 conmon[29288]: mon.0) 2972 : audit [INF] 2021-02-10T04:59:08.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:07 smithi164 conmon[29288]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:59:08.152 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:59:08.171 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:59:08.172 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:59:08.172 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:59:08.172 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:59:08.173 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:59:08.173 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:59:08.173 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:59:08.173 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:59:08.173 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:59:09.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:08 smithi164 conmon[29288]: cluster 2021-02-10T04:59:07.874042+0000 mon.a (mon.0) 2973 : cluster 2021-02-10T04:59:09.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:08 smithi164 conmon[29288]: [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:59:09.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:08 smithi164 conmon[29288]: audit 2021-02-10T04:59:07.877827+0000 mon.a (mon.0) 2974 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:59:09.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:08 smithi164 conmon[29288]: cluster 2021-02-10T04:59:07.877904+0000 mon.a (mon.0) 2975 : cluster [DBG] osdmap e1338: 8 total, 8 up, 8 in 2021-02-10T04:59:09.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:08 smithi164 conmon[29288]: audit 2021-02-10T04:59:08.009765+0000 mon.a (mon.0) 2976 : audit [DBG] from='client.? 172.21.15.85:0/319133314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:59:09.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:08 smithi164 conmon[29288]: cluster 2021-02-10T04:59:08.629698+0000 mon.a (mon.0) 2977 : cluster [DBG] osdmap e1339: 8 total, 8 up, 8 in 2021-02-10T04:59:09.185 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:08 smithi085 conmon[29319]: cluster 2021-02-10T04:59:07.874042+0000 2021-02-10T04:59:09.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:08 smithi085 conmon[29319]: mon.a (mon.0) 2973 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:59:09.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:08 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T04:59:09.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:08 smithi085 conmon[29319]: T04:59:07.877827+0000 mon.a (mon.0) 2974 : audit 2021-02-10T04:59:09.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:08 smithi085 conmon[29319]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:59:09.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:08 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:59:09.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:08 smithi085 conmon[29319]: 59:07.877904+0000 mon.a (mon. 2021-02-10T04:59:09.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:08 smithi085 conmon[29319]: 0) 2975 : cluster [DBG] osdmap e1338: 8 total, 8 up, 8 in 2021-02-10T04:59:09.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:08 smithi085 conmon[29319]: audit 2021-02-10T 2021-02-10T04:59:09.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:08 smithi085 conmon[29319]: 04:59:08.009765+0000 mon.a (mon.0) 2976 2021-02-10T04:59:09.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:08 smithi085 conmon[29319]: : audit [DBG] from='client.? 172.21.15.85:0/319133314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:59:09.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:08 smithi085 conmon[29319]: cluster 2021-02-10T04:59:08.629698 2021-02-10T04:59:09.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:08 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2977 : cluster [DBG] osdmap e1339: 8 total, 8 up, 8 in 2021-02-10T04:59:09.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:08 smithi085 conmon[35961]: cluster 2021-02-10T04:59:07.874042+0000 mon.a 2021-02-10T04:59:09.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:08 smithi085 conmon[35961]: (mon.0) 2973 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:59:09.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:08 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:59:09.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:08 smithi085 conmon[35961]: 59:07.877827+0000 mon.a 2021-02-10T04:59:09.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:08 smithi085 conmon[35961]: (mon.0) 2974 : audit 2021-02-10T04:59:09.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:08 smithi085 conmon[35961]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:59:09.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:08 smithi085 conmon[35961]: cluster 2021-02-10T04:59:07.877904+0000 mon.a (mon.0) 2975 : cluster [DBG] osdmap e1338: 8 total, 8 up, 8 in 2021-02-10T04:59:09.187 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:08 smithi085 conmon[35961]: audit 2021-02-10T04:59:08.009765+0000 mon.a (mon.0) 2976 : audit [DBG] from='client.? 172.21.15.85:0/319133314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:59:09.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:08 smithi085 conmon[35961]: cluster 2021-02-10T04:59:08.629698+0000 mon.a (mon.0) 2977 : cluster [DBG] osdmap e1339: 8 total, 8 up, 8 in 2021-02-10T04:59:09.208 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:59:09.209 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:59:09.534 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:59:09 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:59:09.890 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:09 smithi085 conmon[35961]: cluster 2021-02-10T04:59:08.561215+0000 mgr.y (mgr.14140) 2021-02-10T04:59:09.890 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:09 smithi085 conmon[35961]: 1512 : cluster [DBG] pgmap v2453: 198 pgs: 1 active+clean+scrubbing, 23 active+clean+wait, 174 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-02-10T04:59:10.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:09 smithi164 conmon[29288]: cluster 2021-02-10T04:59:10.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:09 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:59:10.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:09 smithi164 conmon[29288]: 59:08.561215+0000 mgr.y (mgr.14140) 1512 : cluster [DBG] pgmap v2453: 198 pgs: 1 active+clean+scrubbing, 23 active+clean+wait, 174 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-02-10T04:59:10.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:09 smithi085 conmon[29319]: cluster 2021-02-10T04:59:08.561215+0000 mgr.y (mgr.14140) 1512 : cluster [DBG] 2021-02-10T04:59:10.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:09 smithi085 conmon[29319]: pgmap v2453: 198 pgs: 1 active+clean+scrubbing, 23 active+clean+wait, 174 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2021-02-10T04:59:10.909 INFO:teuthology.orchestra.run.smithi085.stderr:set pool 4 pg_num to 32 2021-02-10T04:59:11.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[35961]: audit 2021-02-10T04:59:09.533011+0000 mgr.y 2021-02-10T04:59:11.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[35961]: (mgr.14140) 1513 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:59:11.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[35961]: audit 2021-02-10T04:59:10.479325+0000 mon.a (mon.0) 2978 : audit 2021-02-10T04:59:11.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[35961]: [INF] from='client.? 172.21.15.85:0/3666227693' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num", "val": "32"}]: dispatch 2021-02-10T04:59:11.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[29319]: audit 2021-02-10T04:59:11.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[29319]: 2021-02-10T04:59:11.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[29319]: 2021 2021-02-10T04:59:11.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[29319]: - 2021-02-10T04:59:11.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[29319]: 02 2021-02-10T04:59:11.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[29319]: - 2021-02-10T04:59:11.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[29319]: 10 2021-02-10T04:59:11.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[29319]: T 2021-02-10T04:59:11.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[29319]: 04 2021-02-10T04:59:11.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[29319]: : 2021-02-10T04:59:11.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[29319]: 59 2021-02-10T04:59:11.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[29319]: : 2021-02-10T04:59:11.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[29319]: 09 2021-02-10T04:59:11.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[29319]: . 2021-02-10T04:59:11.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[29319]: 533011 2021-02-10T04:59:11.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[29319]: +0000 2021-02-10T04:59:11.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[29319]: 2021-02-10T04:59:11.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[29319]: mgr.y 2021-02-10T04:59:11.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[29319]: ( 2021-02-10T04:59:11.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[29319]: mgr 2021-02-10T04:59:11.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[29319]: . 2021-02-10T04:59:11.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[29319]: 14140 2021-02-10T04:59:11.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[29319]: ) 2021-02-10T04:59:11.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[29319]: 1513 2021-02-10T04:59:11.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[29319]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:59:11.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:59:11.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[29319]: 02-10T04:59:10.479325+0000 mon.a 2021-02-10T04:59:11.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:10 smithi085 conmon[29319]: (mon.0) 2978 : audit [INF] from='client.? 172.21.15.85:0/3666227693' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num", "val": "32"}]: dispatch 2021-02-10T04:59:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:10 smithi164 conmon[29288]: audit 2021-02-10T04:59:09.533011+0000 mgr.y 2021-02-10T04:59:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:10 smithi164 conmon[29288]: (mgr.14140) 1513 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:59:11.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:10 smithi164 conmon[29288]: audit 2021-02-10T04:59:10.479325+0000 mon.a (mon.0) 2978 : audit [INF] from='client.? 172.21.15.85:0/3666227693' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num", "val": "32"}]: dispatch 2021-02-10T04:59:11.848 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:59:11.881 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set nodeep-scrub 2021-02-10T04:59:12.097 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:59:12.151 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:59:12.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:11 smithi085 conmon[35961]: cluster 2021-02-10T04:59:10.561758+0000 mgr.y (mgr.14140) 1514 : cluster [DBG] pgmap v2455: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2021-02-10T04:59:12.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:11 smithi085 conmon[35961]: audit 2021-02-10T04:59:10.905055+0000 mon.a (mon.0) 2979 : audit [INF] from='client.? 172.21.15.85:0/3666227693' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num", "val": "32"}]': finished 2021-02-10T04:59:12.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:11 smithi085 conmon[35961]: cluster 2021-02-10T04:59:10.905122+0000 mon.a (mon.0) 2980 : cluster [DBG] osdmap e1340: 8 total, 8 up, 8 in 2021-02-10T04:59:12.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:11 smithi085 conmon[29319]: cluster 2021-02-10T04:59:10.561758+0000 mgr.y (mgr.14140) 1514 : cluster [DBG] pgmap v2455: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2021-02-10T04:59:12.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:11 smithi085 conmon[29319]: audit 2021-02-10T04:59:10.905055+0000 mon.a (mon.0) 2979 : audit [INF] from='client.? 172.21.15.85:0/3666227693' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num", "val": "32"}]': finished 2021-02-10T04:59:12.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:11 smithi085 conmon[29319]: cluster 2021-02-10T04:59:10.905122+0000 mon.a (mon.0) 2980 : cluster [DBG] osdmap e1340: 8 total, 8 up, 8 in 2021-02-10T04:59:12.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:11 smithi164 conmon[29288]: cluster 2021-02-10T04:59:10.561758+0000 mgr.y (mgr.14140) 1514 2021-02-10T04:59:12.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:11 smithi164 conmon[29288]: : cluster [DBG] pgmap v2455: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2021-02-10T04:59:12.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:11 smithi164 conmon[29288]: audit 2021-02-10T04:59:10.905055+0000 mon.a (mon.0) 2979 : audit [INF] from='client.? 172.21.15.85:0/3666227693' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num", "val": "32"}]': finished 2021-02-10T04:59:12.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:11 smithi164 conmon[29288]: cluster 2021-02-10T04:59:10.905122+0000 mon.a (mon.0) 2980 : cluster [DBG] osdmap e1340: 8 total, 8 up, 8 in 2021-02-10T04:59:12.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:12 smithi085 conmon[29319]: audit 2021-02-10T04:59:12.563555+0000 2021-02-10T04:59:12.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:12 smithi085 conmon[29319]: mon.a (mon.0) 2981 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "32"}]: dispatch 2021-02-10T04:59:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:12 smithi085 conmon[35961]: audit 2021-02-10T04:59:12.563555+0000 2021-02-10T04:59:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:12 smithi085 conmon[35961]: mon.a (mon.0) 2981 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "32"}]: dispatch 2021-02-10T04:59:13.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:12 smithi164 conmon[29288]: audit 2021-02-10T04:59:12.563555+0000 mon.a (mon. 2021-02-10T04:59:13.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:12 smithi164 conmon[29288]: 0) 2981 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "32"}]: dispatch 2021-02-10T04:59:14.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[29319]: cluster 2021-02-10T04:59:12.562334+0000 mgr.y (mgr.14140) 1515 : cluster 2021-02-10T04:59:14.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[29319]: [DBG] pgmap v2457: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:59:14.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[29319]: audit 2021-02-10T04:59:12.906713+0000 mon.a (mon.0) 2982 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "32"}]': finished 2021-02-10T04:59:14.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[29319]: cluster 2021-02-10T04:59:14.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[29319]: 2021-02-10T04:59:12.906767+0000 mon.a (mon.0) 2983 : 2021-02-10T04:59:14.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[29319]: cluster [DBG] osdmap e1341: 8 total, 8 up, 8 in 2021-02-10T04:59:14.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:59:14.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[29319]: 02-10T04:59:12. 2021-02-10T04:59:14.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[29319]: 929980+0000 osd.5 (osd.5) 200 : 2021-02-10T04:59:14.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[29319]: cluster [DBG] purged_snaps scrub starts 2021-02-10T04:59:14.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:59:14.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[29319]: 02-10T04:59:12.932200 2021-02-10T04:59:14.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[29319]: +0000 osd.5 (osd.5) 201 : cluster 2021-02-10T04:59:14.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[29319]: [DBG] purged_snaps scrub ok 2021-02-10T04:59:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[35961]: cluster 2021-02-10T04:59:12. 2021-02-10T04:59:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[35961]: 562334+0000 mgr.y (mgr.14140) 1515 : cluster [DBG] pgmap v2457: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:59:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:59:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[35961]: 04:59:12.906713+0000 mon.a (mon.0 2021-02-10T04:59:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[35961]: ) 2982 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "32"}]': finished 2021-02-10T04:59:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:59:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[35961]: 02-10T04:59:12.906767 2021-02-10T04:59:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2983 : cluster [DBG] osdmap e1341: 8 total, 8 up, 8 in 2021-02-10T04:59:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[35961]: 2021-02-10T04:59:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[35961]: cluster 2021-02-10T04:59:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[35961]: 2021-02-10T04:59:12 2021-02-10T04:59:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[35961]: .929980+0000 osd.5 (osd.5) 200 2021-02-10T04:59:14.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[35961]: : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:59:14.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[35961]: cluster 2021-02-10T04:59:14.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:59:14.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[35961]: 59:12.932200+0000 osd.5 (osd 2021-02-10T04:59:14.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:13 smithi085 conmon[35961]: .5) 201 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:59:14.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:13 smithi164 conmon[29288]: cluster 2021-02-10T04:59:14.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:13 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:59:14.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:13 smithi164 conmon[29288]: 59:12.562334+0000 mgr.y (mgr.14140) 1515 : cluster [DBG] pgmap v2457: 198 pgs: 1 active+clean+scrubbing, 197 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T04:59:14.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:13 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:59:14.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:13 smithi164 conmon[29288]: T04:59:12.906713+0000 mon.a (mon.0 2021-02-10T04:59:14.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:13 smithi164 conmon[29288]: ) 2982 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pg_num_actual", "val": "32"}]': finished 2021-02-10T04:59:14.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:13 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:59:14.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:13 smithi164 conmon[29288]: :59:12.906767+0000 mon.a (mon.0) 2983 : cluster [DBG] osdmap e1341: 8 total, 8 up, 8 in 2021-02-10T04:59:14.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:13 smithi164 conmon[29288]: cluster 2021-02-10T04:59:14.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:13 smithi164 conmon[29288]: 2021-02-10T04:59 2021-02-10T04:59:14.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:13 smithi164 conmon[29288]: :12.929980+0000 osd.5 (osd.5) 200 : cluster 2021-02-10T04:59:14.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:13 smithi164 conmon[29288]: [DBG] purged_snaps scrub starts 2021-02-10T04:59:14.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:13 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:59:14.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:13 smithi164 conmon[29288]: 02-10T04:59:12.932200+0000 osd.5 2021-02-10T04:59:14.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:13 smithi164 conmon[29288]: (osd.5) 201 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:59:14.690 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:59:14.708 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:59:14.726 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:59:14.747 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:59:14.767 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:59:14.785 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:59:14.805 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:59:14.820 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:59:14.821 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:59:14.822 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:59:14.822 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:59:14.822 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:59:14.822 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:59:14.822 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:59:14.822 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:59:14.822 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:59:15.040 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[29319]: cluster 2021-02-10T04:59:15.040 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[29319]: 2021-02-10T04:59:15.040 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[29319]: 2021 2021-02-10T04:59:15.040 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[29319]: - 2021-02-10T04:59:15.040 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[29319]: 02 2021-02-10T04:59:15.040 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[29319]: - 2021-02-10T04:59:15.041 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[29319]: 10 2021-02-10T04:59:15.041 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[29319]: T 2021-02-10T04:59:15.041 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[29319]: 04 2021-02-10T04:59:15.041 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[29319]: : 2021-02-10T04:59:15.041 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[29319]: 59 2021-02-10T04:59:15.041 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[29319]: : 2021-02-10T04:59:15.041 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[29319]: 13.915517+0000 mon.a (mon. 2021-02-10T04:59:15.041 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[29319]: 0) 2984 : cluster [DBG] osdmap e1342: 8 total, 8 up, 8 in 2021-02-10T04:59:15.041 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[29319]: audit 2021-02 2021-02-10T04:59:15.041 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[29319]: -10T04:59:14.563778+0000 mon.a (mon. 2021-02-10T04:59:15.041 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[29319]: 0) 2985 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pgp_num_actual", "val": "32"}]: dispatch 2021-02-10T04:59:15.041 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[29319]: audit 2021-02-10T04:59:15.041 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[29319]: 2021-02-10T04:59:14. 2021-02-10T04:59:15.042 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[29319]: 671514+0000 mon.c (mon.1) 158 : audit [DBG] from='client.? 172.21.15.85:0/722552664' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:59:15.042 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[29319]: 2021-02-10T04:59:15.042 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[35961]: cluster 2021-02-10T04:59:15.042 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[35961]: 2021-02-10T04:59:15.042 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[35961]: 2021 2021-02-10T04:59:15.042 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[35961]: - 2021-02-10T04:59:15.042 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[35961]: 02 2021-02-10T04:59:15.042 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[35961]: - 2021-02-10T04:59:15.042 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[35961]: 10 2021-02-10T04:59:15.042 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[35961]: T 2021-02-10T04:59:15.042 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[35961]: 04 2021-02-10T04:59:15.043 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[35961]: : 2021-02-10T04:59:15.043 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[35961]: 59 2021-02-10T04:59:15.043 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[35961]: : 2021-02-10T04:59:15.043 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[35961]: 13 2021-02-10T04:59:15.043 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[35961]: . 2021-02-10T04:59:15.043 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[35961]: 915517 2021-02-10T04:59:15.043 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[35961]: +0000 2021-02-10T04:59:15.043 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[35961]: mon.a (mon.0) 2984 2021-02-10T04:59:15.043 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[35961]: : cluster [DBG] osdmap e1342: 8 total, 8 up, 8 in 2021-02-10T04:59:15.043 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:59:15.043 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[35961]: 02-10T04:59:14.563778+0000 mon.a 2021-02-10T04:59:15.043 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[35961]: (mon.0) 2985 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pgp_num_actual", "val": "32"}]: dispatch 2021-02-10T04:59:15.043 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[35961]: audit 2021-02-10T04:59:15.044 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[35961]: 2021-02-10T04:59:14.671514 2021-02-10T04:59:15.044 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[35961]: +0000 mon.c (mon.1) 158 : audit 2021-02-10T04:59:15.044 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:14 smithi085 conmon[35961]: [DBG] from='client.? 172.21.15.85:0/722552664' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:59:15.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:14 smithi164 conmon[29288]: cluster 2021-02-10T04:59:15.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:14 smithi164 conmon[29288]: 2021-02-10T04:59:15.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:14 smithi164 conmon[29288]: 2021 2021-02-10T04:59:15.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:14 smithi164 conmon[29288]: - 2021-02-10T04:59:15.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:14 smithi164 conmon[29288]: 02 2021-02-10T04:59:15.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:14 smithi164 conmon[29288]: - 2021-02-10T04:59:15.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:14 smithi164 conmon[29288]: 10 2021-02-10T04:59:15.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:14 smithi164 conmon[29288]: T04:59:13.915517 2021-02-10T04:59:15.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:14 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2984 : cluster [DBG] osdmap e1342: 8 total, 8 up, 8 in 2021-02-10T04:59:15.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:14 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:59:15.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:14 smithi164 conmon[29288]: 04:59:14.563778+0000 mon.a (mon.0) 2985 : 2021-02-10T04:59:15.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:14 smithi164 conmon[29288]: audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pgp_num_actual", "val": "32"}]: dispatch 2021-02-10T04:59:15.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:14 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:59:15.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:14 smithi164 conmon[29288]: :59:14.671514+0000 mon.c (mon.1 2021-02-10T04:59:15.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:14 smithi164 conmon[29288]: ) 158 : audit [DBG] from='client.? 172.21.15.85:0/722552664' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:59:15.945 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is set 2021-02-10T04:59:15.964 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:15 smithi085 conmon[35961]: cluster 2021-02-10T04:59:14.562811 2021-02-10T04:59:15.965 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:15 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1516 : cluster [DBG] pgmap v2460: 208 pgs: 3 peering, 9 unknown, 1 active+clean+scrubbing, 195 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:59:15.965 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:15 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:59:15.965 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:15 smithi085 conmon[35961]: 10T04:59:14.916967+0000 mon.a (mon.0) 2986 : audit [INF] 2021-02-10T04:59:15.965 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:15 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pgp_num_actual", "val": "32"}]': finished 2021-02-10T04:59:15.965 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:15 smithi085 conmon[35961]: cluster 2021-02-10T04:59:14. 2021-02-10T04:59:15.965 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:15 smithi085 conmon[35961]: 917025+0000 mon.a (mon.0) 2987 : cluster 2021-02-10T04:59:15.965 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:15 smithi085 conmon[35961]: [DBG] osdmap e1343: 8 total, 8 up, 8 in 2021-02-10T04:59:15.965 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:15 smithi085 conmon[35961]: audit 2021-02-10T04:59:15 2021-02-10T04:59:15.965 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:15 smithi085 conmon[35961]: .124730+0000 mon.a (mon.0) 2988 : audit [INF] 2021-02-10T04:59:15.965 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:15 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/2338558721' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:59:15.965 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:15 smithi085 conmon[29319]: cluster 2021-02-10T04:59:15.966 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:15 smithi085 conmon[29319]: 2021-02-10T04:59:15.966 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:15 smithi085 conmon[29319]: 2021 2021-02-10T04:59:15.966 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:15 smithi085 conmon[29319]: - 2021-02-10T04:59:15.966 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:15 smithi085 conmon[29319]: 02 2021-02-10T04:59:15.966 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:15 smithi085 conmon[29319]: - 2021-02-10T04:59:15.966 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:15 smithi085 conmon[29319]: 10 2021-02-10T04:59:15.966 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:15 smithi085 conmon[29319]: T04:59:14.562811+0000 mgr.y 2021-02-10T04:59:15.966 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:15 smithi085 conmon[29319]: (mgr.14140) 1516 : cluster [DBG] pgmap v2460: 208 pgs: 3 peering, 9 unknown, 1 active+clean+scrubbing, 195 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:59:15.966 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:15 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:59:15.966 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:15 smithi085 conmon[29319]: 02-10T04:59:14.916967+0000 mon.a 2021-02-10T04:59:15.966 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:15 smithi085 conmon[29319]: (mon.0) 2986 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pgp_num_actual", "val": "32"}]': finished 2021-02-10T04:59:15.966 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:15 smithi085 conmon[29319]: cluster 2021-02-10T04:59:15.966 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:15 smithi085 conmon[29319]: 2021-02-10T04:59:14.917025 2021-02-10T04:59:15.967 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:15 smithi085 conmon[29319]: +0000 mon.a (mon.0) 2987 : cluster [DBG] 2021-02-10T04:59:15.967 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:15 smithi085 conmon[29319]: osdmap e1343: 8 total, 8 up, 8 in 2021-02-10T04:59:15.967 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:15 smithi085 conmon[29319]: audit 2021-02-10T04:59 2021-02-10T04:59:15.967 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:15 smithi085 conmon[29319]: :15.124730+0000 mon.a (mon.0) 2988 2021-02-10T04:59:15.967 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:15 smithi085 conmon[29319]: : audit [INF] from='client.? 172.21.15.85:0/2338558721' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:59:16.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:15 smithi164 conmon[29288]: cluster 2021-02-10T04:59:14.562811+0000 2021-02-10T04:59:16.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:15 smithi164 conmon[29288]: mgr.y (mgr.14140) 1516 : cluster [DBG] pgmap v2460: 208 pgs: 3 peering, 9 unknown, 1 active+clean+scrubbing, 195 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:59:16.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:15 smithi164 conmon[29288]: audit 2021-02-10T04:59:14.916967+0000 mon.a (mon.0) 2986 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pgp_num_actual", "val": "32"}]': finished 2021-02-10T04:59:16.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:15 smithi164 conmon[29288]: cluster 2021-02-10T04:59:14.917025+0000 2021-02-10T04:59:16.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:15 smithi164 conmon[29288]: mon.a (mon.0) 2987 : cluster [DBG] osdmap e1343: 8 total, 8 up, 8 in 2021-02-10T04:59:16.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:15 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:59:16.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:15 smithi164 conmon[29288]: 10T04:59:15.124730+0000 mon.a (mon.0 2021-02-10T04:59:16.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:15 smithi164 conmon[29288]: ) 2988 : audit [INF] from='client.? 172.21.15.85:0/2338558721' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:59:16.894 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2021-02-10T04:59:16.894 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:59:16.894 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd dump -f json-pretty 2021-02-10T04:59:17.128 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:59:17.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:16 smithi085 conmon[29319]: cluster 2021-02-10T04:59:15.925031+0000 mon.a (mon.0) 2989 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:59:17.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:16 smithi085 conmon[29319]: audit 2021-02-10T04:59:15.930667+0000 mon.a (mon.0) 2990 : audit [INF] from='client.? 172.21.15.85:0/2338558721' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:59:17.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:16 smithi085 conmon[29319]: cluster 2021-02-10T04:59:15.930960+0000 mon.a (mon.0) 2991 : cluster [DBG] osdmap e1344: 8 total, 8 up, 8 in 2021-02-10T04:59:17.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:16 smithi085 conmon[35961]: cluster 2021-02-10T04:59:15.925031+0000 2021-02-10T04:59:17.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:16 smithi085 conmon[35961]: mon.a (mon.0) 2989 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:59:17.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:16 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:59:17.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:16 smithi085 conmon[35961]: T04:59:15.930667+0000 mon.a (mon.0) 2990 : audit [INF] from='client.? 172.21.15.85:0/2338558721' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:59:17.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:16 smithi085 conmon[35961]: 2021-02-10T04:59:17.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:16 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:59:17.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:16 smithi085 conmon[35961]: 02-10T04:59:15.930960+0000 mon.a (mon.0) 2021-02-10T04:59:17.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:16 smithi085 conmon[35961]: 2991 : cluster [DBG] osdmap e1344: 8 total, 8 up, 8 in 2021-02-10T04:59:17.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:16 smithi164 conmon[29288]: cluster 2021-02-10T04:59:17.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:16 smithi164 conmon[29288]: 2021-02-10T04:59:17.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:16 smithi164 conmon[29288]: 2021 2021-02-10T04:59:17.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:16 smithi164 conmon[29288]: - 2021-02-10T04:59:17.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:16 smithi164 conmon[29288]: 02 2021-02-10T04:59:17.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:16 smithi164 conmon[29288]: - 2021-02-10T04:59:17.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:16 smithi164 conmon[29288]: 10 2021-02-10T04:59:17.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:16 smithi164 conmon[29288]: T04:59:15.925031+0000 mon.a 2021-02-10T04:59:17.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:16 smithi164 conmon[29288]: (mon.0) 2989 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:59:17.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:16 smithi164 conmon[29288]: audit 2021-02-10T04:59:17.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:16 smithi164 conmon[29288]: 2021-02-10T04:59:15.930667+0000 mon.a ( 2021-02-10T04:59:17.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:16 smithi164 conmon[29288]: mon.0) 2990 : audit [INF] from='client.? 172.21.15.85:0/2338558721' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:59:17.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:16 smithi164 conmon[29288]: cluster 2021-02-10T04:59:17.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:16 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:59:17.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:16 smithi164 conmon[29288]: 59:15.930960+0000 mon.a (mon.0) 2991 : 2021-02-10T04:59:17.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:16 smithi164 conmon[29288]: cluster [DBG] osdmap e1344: 8 total, 8 up, 8 in 2021-02-10T04:59:17.533 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.6 dump_ops_in_flight 2021-02-10T04:59:17.730 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:59:17.947 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: cluster 2021-02-10T04:59:16.563256+0000 mgr.y ( 2021-02-10T04:59:17.950 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: cluster 2021-02-10T04:59:16. 2021-02-10T04:59:17.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: 563256+0000 mgr.y (mgr.14140) 1517 : cluster [DBG] pgmap v2463: 208 pgs: 5 peering, 7 unknown, 1 active+clean+scrubbing, 195 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:59:17.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: cluster 2021-02-10T04:59:16.947402+0000 osd.3 (osd.3) 46 : cluster [DBG] 4.19 starting backfill to osd.0 from (0'0,0'0] MAX to 1261'86 2021-02-10T04:59:17.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: cluster 2021-02-10T04:59:16.947683+0000 osd.3 (osd.3) 47 : cluster [DBG] 4.19 starting backfill to osd.2 from (0'0,0'0] MAX to 1261'86 2021-02-10T04:59:17.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: cluster 2021-02-10T04:59:16.947826+0000 osd.3 (osd.3) 48 : cluster [DBG] 4.1c starting backfill to osd.1 from (0'0,0'0] MAX to 1261'76 2021-02-10T04:59:17.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: cluster 2021-02-10T04:59:16.948026+0000 osd.3 (osd.3) 49 : cluster [DBG] 4.1c starting backfill to osd.2 from (0'0,0'0] MAX to 1261'76 2021-02-10T04:59:17.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: cluster 2021-02-10T04:59:16.948596+0000 mon.a (mon.0) 2992 : cluster [DBG] osdmap e1345: 8 total, 8 up, 8 in 2021-02-10T04:59:17.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: cluster 2021-02-10T04:59:16.952278+0000 osd.1 (osd.1) 515 : cluster [DBG] 4.1e starting backfill to osd.0 from (0'0,0'0] MAX to 1261'65 2021-02-10T04:59:17.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: cluster 2021-02-10T04:59:16.952343+0000 osd.1 (osd.1) 516 : cluster [DBG] 4.1d starting backfill to osd.3 from (0'0,0'0] MAX to 1261'64 2021-02-10T04:59:17.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: cluster 2021-02-10T04:59:16.952396+0000 osd.1 (osd.1) 517 : cluster [DBG] 4.1e starting backfill to osd.3 from (0'0,0'0] MAX to 1261'65 2021-02-10T04:59:17.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: cluster 2021-02-10T04:59:16.952639+0000 osd.1 (osd.1) 518 : cluster [DBG] 4.1e starting backfill to osd.7 from (0'0,0'0] MAX to 1261'65 2021-02-10T04:59:17.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: cluster 2021-02-10T04:59:16.954246+0000 osd.0 (osd.0) 4 : cluster [DBG] 4.16 starting backfill to osd.3 from (0'0,0'0] MAX to 1261'58 2021-02-10T04:59:17.951 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: cluster 2021-02-10T04:59:16.954378+0000 osd.4 (osd.4) 2665 : cluster [DBG] 4.1b starting backfill to osd.3 from (0'0,0'0] MAX to 1261'28 2021-02-10T04:59:17.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: cluster 2021-02-10T04:59:16.954425+0000 osd.0 (osd.0) 5 : cluster [DBG] 4.16 starting backfill to osd.7 from (0'0,0'0] MAX to 1261'58 2021-02-10T04:59:17.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: cluster 2021-02-10T04:59:16.955555+0000 osd.0 (osd.0) 6 : cluster [DBG] 4.17 starting backfill to osd.3 from (0'0,0'0] MAX to 1261'40 2021-02-10T04:59:17.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:59:17.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: 59:16.955717+0000 osd.0 (osd.0) 7 : cluster [DBG] 4.17 starting backfill to osd.7 from (0'0,0'0] MAX to 1261'40 2021-02-10T04:59:17.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:59:17.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: 02-10T04:59:16.958617+0000 osd.1 2021-02-10T04:59:17.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: (osd.1) 519 : cluster [DBG] 4.1f starting backfill to osd.5 from (0'0,0'0] MAX to 1261'71 2021-02-10T04:59:17.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:59:17.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: 04:59:16.958813+0000 osd.1 (osd.1) 520 : cluster 2021-02-10T04:59:17.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: [DBG] 4.1f starting backfill to osd.6 from (0'0,0'0] MAX to 1261'71 2021-02-10T04:59:17.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:59:17.952 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: 10T04:59:16.959368+0000 osd.5 (osd.5) 202 : cluster [DBG] 4.18 starting backfill to osd.1 from (0'0,0'0] MAX to 1261'84 2021-02-10T04:59:17.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:59:17.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: 59:16.959568+0000 osd.5 (osd.5) 203 : cluster [DBG] 2021-02-10T04:59:17.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: 4.18 starting backfill to osd.4 from (0'0,0'0] MAX to 1261'84 2021-02-10T04:59:17.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:59:17.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: 04:59:16.962188+0000 osd.1 (osd.1) 521 : cluster [DBG] 4.1a starting backfill to osd.0 from (0'0,0'0] MAX to 1261'69 2021-02-10T04:59:17.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: cluster 2021-02-10T04:59:16.962379 2021-02-10T04:59:17.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 522 : cluster [DBG] 4.1a starting backfill to osd.3 from (0'0,0'0] MAX to 1261'69 2021-02-10T04:59:17.953 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: cluster 2021-02-10T04:59 2021-02-10T04:59:17.954 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[35961]: :16.962544+0000 osd.1 (osd.1) 523 : cluster [DBG] 4.1a starting backfill to osd.4 from (0'0,0'0] MAX to 1261'69 2021-02-10T04:59:17.961 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:59:18.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: cluster 2021-02-10T04:59:18.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:59:18.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: 04:59:16.563256+0000 mgr.y (mgr.14140) 1517 : cluster [DBG] pgmap v2463: 208 pgs: 5 peering, 7 unknown, 1 active+clean+scrubbing, 195 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:59:18.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:59:18.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: -02-10T04:59:16.947402+0000 2021-02-10T04:59:18.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: osd.3 (osd.3) 46 : cluster [DBG] 2021-02-10T04:59:18.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: 4.19 starting backfill to osd.0 from (0'0,0'0] MAX to 1261'86 2021-02-10T04:59:18.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:59:18.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: 02-10T04:59:16.947683+0000 2021-02-10T04:59:18.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: osd.3 (osd.3) 47 : cluster [DBG] 2021-02-10T04:59:18.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: 4.19 starting backfill to osd.2 from (0'0,0'0] MAX to 1261'86 2021-02-10T04:59:18.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:59:18.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: 04:59:16.947826+0000 osd.3 (osd 2021-02-10T04:59:18.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: .3) 48 : cluster [DBG] 4.1c starting backfill to osd.1 from (0'0,0'0] MAX to 1261'76 2021-02-10T04:59:18.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:59:18.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: -02-10T04:59:16.948026+0000 osd.3 (osd.3 2021-02-10T04:59:18.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: ) 49 : cluster [DBG] 4.1c starting backfill to osd.2 from (0'0,0'0] MAX to 1261'76 2021-02-10T04:59:18.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: cluster 2021-02-10T04:59:18.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: 2021-02-10T04:59:16.948596+0000 mon.a (mon.0) 2992 : cluster 2021-02-10T04:59:18.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: [DBG] osdmap e1345: 8 total, 8 up, 8 in 2021-02-10T04:59:18.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:59:18.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: 10T04:59:16.952278+0000 osd.1 (osd.1) 515 : cluster [DBG] 4.1e starting backfill to osd.0 from (0'0,0'0] MAX to 1261'65 2021-02-10T04:59:18.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: cluster 2021-02-10T04:59:16.952343+0000 osd.1 2021-02-10T04:59:18.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: (osd.1) 516 : cluster [DBG] 4.1d starting backfill to osd.3 from (0'0,0'0] MAX to 1261'64 2021-02-10T04:59:18.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: cluster 2021-02-10T04:59:16.952396+0000 osd.1 (osd 2021-02-10T04:59:18.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: .1) 517 : cluster [DBG] 4.1e starting backfill to osd.3 from (0'0,0'0] MAX to 1261'65 2021-02-10T04:59:18.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: cluster 2021-02-10T04:59:16 2021-02-10T04:59:18.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: .952639+0000 osd.1 (osd.1) 518 : cluster [DBG] 4.1e starting backfill to osd.7 from (0'0,0'0] MAX to 1261'65 2021-02-10T04:59:18.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:59:18.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: 04:59:16.954246+0000 osd.0 (osd.0 2021-02-10T04:59:18.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: ) 4 : cluster [DBG] 4.16 starting backfill to osd.3 from (0'0,0'0] MAX to 1261'58 2021-02-10T04:59:18.399 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:59:18.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: -10T04:59:16.954378+0000 osd.4 2021-02-10T04:59:18.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: (osd.4) 2665 : cluster [DBG] 4.1b starting backfill to osd.3 from (0'0,0'0] MAX to 1261'28 2021-02-10T04:59:18.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:59:18.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: -02-10T04:59:16.954425+0000 2021-02-10T04:59:18.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: osd.0 (osd.0) 5 : cluster [DBG] 4.16 starting backfill to osd.7 from (0'0,0'0] MAX to 1261'58 2021-02-10T04:59:18.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: 2021-02-10T04:59:18.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: cluster 2021-02-10T04:59:18.400 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: 2021-02-10T04:59:16.955555+0000 osd.0 (osd. 2021-02-10T04:59:18.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: 0) 6 : cluster [DBG] 4.17 starting backfill to osd.3 from (0'0,0'0] MAX to 1261'40 2021-02-10T04:59:18.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:59:18.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: -10T04:59:16.955717+0000 osd.0 2021-02-10T04:59:18.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: (osd.0) 7 : cluster [DBG] 4.17 starting backfill to osd.7 from (0'0,0'0] MAX to 1261'40 2021-02-10T04:59:18.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: 2021-02-10T04:59:18.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:59:18.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: 02-10T04:59:16.958617+0000 osd.1 (osd.1) 2021-02-10T04:59:18.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: 519 : cluster [DBG] 4.1f starting backfill to osd.5 from (0'0,0'0] MAX to 1261'71 2021-02-10T04:59:18.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:59:18.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: 10T04:59:16.958813+0000 osd.1 (osd 2021-02-10T04:59:18.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: .1) 520 : cluster [DBG] 4.1f starting backfill to osd.6 from (0'0,0'0] MAX to 1261'71 2021-02-10T04:59:18.401 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:59:18.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: 04:59:16.959368+0000 osd.5 (osd. 2021-02-10T04:59:18.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: 5) 202 : cluster [DBG] 4.18 starting backfill to osd.1 from (0'0,0'0] MAX to 1261'84 2021-02-10T04:59:18.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:59:18.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: -10T04:59:16.959568+0000 osd.5 (osd.5) 2021-02-10T04:59:18.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: 203 : cluster [DBG] 4.18 starting backfill to osd.4 from (0'0,0'0] MAX to 1261'84 2021-02-10T04:59:18.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: cluster 2021-02-10T04:59:18.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: 2021-02-10T04:59:16.962188+0000 osd.1 2021-02-10T04:59:18.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: (osd.1) 521 : cluster [DBG] 4.1a starting backfill to osd.0 from (0'0,0'0] MAX to 1261'69 2021-02-10T04:59:18.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:59:18.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: :59:16.962379+0000 osd.1 (osd.1) 522 : cluster [DBG] 2021-02-10T04:59:18.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: 4.1a starting backfill to osd.3 from (0'0,0'0] MAX to 1261'69 2021-02-10T04:59:18.402 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:59:18.403 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:17 smithi164 conmon[29288]: -10T04:59:16.962544+0000 osd.1 (osd.1) 523 : cluster [DBG] 4.1a starting backfill to osd.4 from (0'0,0'0] MAX to 1261'69 2021-02-10T04:59:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: mgr.14140) 1517 : cluster [DBG] pgmap v2463: 208 pgs: 5 peering, 7 unknown, 1 active+clean+scrubbing, 195 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2021-02-10T04:59:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: cluster 2021-02-10T04:59:16.947402+0000 osd.3 (osd.3) 46 : cluster [DBG] 4.19 starting backfill to osd.0 from (0'0,0'0] MAX to 1261'86 2021-02-10T04:59:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: cluster 2021-02-10T04:59:16.947683+0000 osd.3 (osd.3) 47 : cluster [DBG] 4.19 starting backfill to osd.2 from (0'0,0'0] MAX to 1261'86 2021-02-10T04:59:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: cluster 2021-02-10T04:59:16.947826+0000 osd.3 (osd.3) 48 : cluster [DBG] 4.1c starting backfill to osd.1 from (0'0,0'0] MAX to 1261'76 2021-02-10T04:59:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: cluster 2021-02-10T04:59:16.948026+0000 osd.3 (osd.3) 49 : cluster [DBG] 4.1c starting backfill to osd.2 from (0'0,0'0] MAX to 1261'76 2021-02-10T04:59:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: cluster 2021-02-10T04:59:16.948596+0000 mon.a (mon.0) 2992 : cluster [DBG] osdmap e1345: 8 total, 8 up, 8 in 2021-02-10T04:59:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: cluster 2021-02-10T04:59:16.952278+0000 osd.1 (osd.1) 515 : cluster [DBG] 4.1e starting backfill to osd.0 from (0'0,0'0] MAX to 1261'65 2021-02-10T04:59:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: cluster 2021-02-10T04:59:16.952343+0000 osd.1 (osd.1) 516 : cluster [DBG] 4.1d starting backfill to osd.3 from (0'0,0'0] MAX to 1261'64 2021-02-10T04:59:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: cluster 2021-02-10T04:59:16.952396+0000 osd.1 (osd.1) 517 : cluster [DBG] 4.1e starting backfill to osd.3 from (0'0,0'0] MAX to 1261'65 2021-02-10T04:59:18.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: cluster 2021-02-10T04:59:16.952639+0000 osd.1 (osd.1) 518 : cluster [DBG] 4.1e starting backfill to osd.7 from (0'0,0'0] MAX to 1261'65 2021-02-10T04:59:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: cluster 2021-02-10T04:59:16.954246+0000 osd.0 (osd.0) 4 : cluster [DBG] 4.16 starting backfill to osd.3 from (0'0,0'0] MAX to 1261'58 2021-02-10T04:59:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: cluster 2021-02-10T04:59:16.954378+0000 osd.4 (osd.4) 2665 : cluster [DBG] 4.1b starting backfill to osd.3 from (0'0,0'0] MAX to 1261'28 2021-02-10T04:59:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: cluster 2021-02-10T04:59:16.954425+0000 osd.0 (osd.0) 5 : cluster [DBG] 4.16 starting backfill to osd.7 from (0'0,0'0] MAX to 1261'58 2021-02-10T04:59:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: cluster 2021-02-10T04:59:16.955555+0000 osd.0 (osd.0) 6 : cluster [DBG] 4.17 starting backfill to osd.3 from (0'0,0'0] MAX to 1261'40 2021-02-10T04:59:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: cluster 2021-02-10T04:59:16.955717+0000 osd.0 (osd.0) 7 : cluster [DBG] 4.17 starting backfill to osd.7 from (0'0,0'0] MAX to 1261'40 2021-02-10T04:59:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: cluster 2021-02-10T04:59:16.958617+0000 osd.1 ( 2021-02-10T04:59:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: osd.1) 519 : cluster [DBG] 4.1f starting backfill to osd.5 from (0'0,0'0] MAX to 1261'71 2021-02-10T04:59:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: cluster 2021-02-10T04:59:16.958813+0000 osd.1 (osd.1) 520 : cluster [DBG] 4.1f starting backfill to osd.6 from (0'0,0'0] MAX to 1261'71 2021-02-10T04:59:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: cluster 2021-02-10T04:59:16.959368+0000 osd.5 (osd.5) 202 : cluster [DBG] 4.18 starting backfill to osd.1 from (0'0,0'0] MAX to 1261'84 2021-02-10T04:59:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: cluster 2021-02-10T04:59:16.959568+0000 osd.5 (osd.5 2021-02-10T04:59:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: ) 203 : cluster [DBG] 4.18 starting backfill to osd.4 from (0'0,0'0] MAX to 1261'84 2021-02-10T04:59:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: cluster 2021-02-10T04:59:16 2021-02-10T04:59:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: .962188+0000 osd.1 (osd.1) 521 : cluster 2021-02-10T04:59:18.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: [DBG] 4.1a starting backfill to osd.0 from (0'0,0'0] MAX to 1261'69 2021-02-10T04:59:18.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: cluster 2021-02-10T04:59:18.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: 2021-02-10T04:59:16.962379 2021-02-10T04:59:18.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: +0000 osd.1 (osd.1) 522 : cluster 2021-02-10T04:59:18.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: [DBG] 4.1a starting backfill to osd.3 from (0'0,0'0] MAX to 1261'69 2021-02-10T04:59:18.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:59:18.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: T04:59:16.962544+0000 osd.1 ( 2021-02-10T04:59:18.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: osd.1) 523 : cluster [DBG] 2021-02-10T04:59:18.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:17 smithi085 conmon[29319]: 4.1a starting backfill to osd.4 from (0'0,0'0] MAX to 1261'69 2021-02-10T04:59:18.659 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset noscrub 2021-02-10T04:59:18.902 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:59:19.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:18 smithi085 conmon[29319]: cluster 2021-02-10T04:59:17. 2021-02-10T04:59:19.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:18 smithi085 conmon[29319]: 955006+0000 mon.a (mon.0) 2993 : cluster [DBG] osdmap e1346: 8 total, 8 up, 8 in 2021-02-10T04:59:19.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:18 smithi085 conmon[29319]: cluster 2021-02-10T04:59:18.633039+0000 mon.a (mon.0) 2994 : cluster [DBG] osdmap e1347: 8 total, 8 up, 8 in 2021-02-10T04:59:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:18 smithi085 conmon[35961]: cluster 2021-02-10T04:59:17.955006 2021-02-10T04:59:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:18 smithi085 conmon[35961]: +0000 mon.a (mon.0) 2993 : cluster [DBG] osdmap e1346: 8 total, 8 up, 8 in 2021-02-10T04:59:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:18 smithi085 conmon[35961]: cluster 2021-02-10T04:59:18.633039+0000 mon.a (mon.0) 2994 : cluster [DBG] osdmap e1347: 8 total, 8 up, 8 in 2021-02-10T04:59:19.258 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:59:19.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:18 smithi164 conmon[29288]: cluster 2021-02-10T04:59:17.955006+0000 mon.a (mon.0) 2993 2021-02-10T04:59:19.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:18 smithi164 conmon[29288]: : cluster [DBG] osdmap e1346: 8 total, 8 up, 8 in 2021-02-10T04:59:19.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:18 smithi164 conmon[29288]: cluster 2021-02-10T04:59:18.633039+0000 mon.a (mon.0) 2994 : cluster [DBG] osdmap e1347: 8 total, 8 up, 8 in 2021-02-10T04:59:19.606 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:59:19.606 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T04:59:19.606 INFO:teuthology.orchestra.run.smithi085.stdout: "epoch": 1347, 2021-02-10T04:59:19.606 INFO:teuthology.orchestra.run.smithi085.stdout: "fsid": "42224fce-6b57-11eb-8feb-001a4aab830c", 2021-02-10T04:59:19.606 INFO:teuthology.orchestra.run.smithi085.stdout: "created": "2021-02-10T04:20:57.933279+0000", 2021-02-10T04:59:19.606 INFO:teuthology.orchestra.run.smithi085.stdout: "modified": "2021-02-10T04:59:18.628529+0000", 2021-02-10T04:59:19.607 INFO:teuthology.orchestra.run.smithi085.stdout: "last_up_change": "2021-02-10T04:58:55.817746+0000", 2021-02-10T04:59:19.607 INFO:teuthology.orchestra.run.smithi085.stdout: "last_in_change": "2021-02-10T04:51:35.701776+0000", 2021-02-10T04:59:19.607 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T04:59:19.607 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_num": 5806080, 2021-02-10T04:59:19.607 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_set": [ 2021-02-10T04:59:19.607 INFO:teuthology.orchestra.run.smithi085.stdout: "nodeep-scrub", 2021-02-10T04:59:19.607 INFO:teuthology.orchestra.run.smithi085.stdout: "noscrub", 2021-02-10T04:59:19.607 INFO:teuthology.orchestra.run.smithi085.stdout: "pglog_hardlimit", 2021-02-10T04:59:19.607 INFO:teuthology.orchestra.run.smithi085.stdout: "purged_snapdirs", 2021-02-10T04:59:19.607 INFO:teuthology.orchestra.run.smithi085.stdout: "recovery_deletes", 2021-02-10T04:59:19.607 INFO:teuthology.orchestra.run.smithi085.stdout: "sortbitwise" 2021-02-10T04:59:19.607 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:59:19.607 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_version": 18, 2021-02-10T04:59:19.608 INFO:teuthology.orchestra.run.smithi085.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T04:59:19.608 INFO:teuthology.orchestra.run.smithi085.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T04:59:19.608 INFO:teuthology.orchestra.run.smithi085.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T04:59:19.608 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_snapshot": "", 2021-02-10T04:59:19.608 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_max": 7, 2021-02-10T04:59:19.608 INFO:teuthology.orchestra.run.smithi085.stdout: "max_osd": 8, 2021-02-10T04:59:19.608 INFO:teuthology.orchestra.run.smithi085.stdout: "require_min_compat_client": "luminous", 2021-02-10T04:59:19.608 INFO:teuthology.orchestra.run.smithi085.stdout: "min_compat_client": "luminous", 2021-02-10T04:59:19.608 INFO:teuthology.orchestra.run.smithi085.stdout: "require_osd_release": "quincy", 2021-02-10T04:59:19.608 INFO:teuthology.orchestra.run.smithi085.stdout: "pools": [ 2021-02-10T04:59:19.608 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.608 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 1, 2021-02-10T04:59:19.608 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "device_health_metrics", 2021-02-10T04:59:19.609 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:24:05.258028+0000", 2021-02-10T04:59:19.609 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:59:19.609 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:59:19.609 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:59:19.609 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:59:19.609 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:59:19.609 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:59:19.609 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:59:19.609 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:59:19.609 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:59:19.609 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:59:19.609 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:59:19.609 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:59:19.610 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 11, 2021-02-10T04:59:19.610 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 11, 2021-02-10T04:59:19.610 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 11, 2021-02-10T04:59:19.610 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 11, 2021-02-10T04:59:19.610 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 11, 2021-02-10T04:59:19.610 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:59:19.610 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:59:19.610 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:59:19.610 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:59:19.610 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:59:19.610 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:59:19.610 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:59:19.610 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.611 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "1266", 2021-02-10T04:59:19.611 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:59:19.611 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:59:19.611 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "110", 2021-02-10T04:59:19.611 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:59:19.611 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:59:19.611 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:59:19.611 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:59:19.611 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:59:19.611 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:59:19.611 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:59:19.611 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:59:19.611 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:59:19.612 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:59:19.612 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:59:19.612 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:59:19.612 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:59:19.612 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:59:19.612 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:59:19.612 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:59:19.612 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:59:19.612 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:59:19.612 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:59:19.612 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:59:19.612 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:59:19.612 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:59:19.613 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:59:19.613 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.613 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:59:19.613 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:59:19.613 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:59:19.613 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:59:19.613 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:59:19.613 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:59:19.613 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:59:19.613 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:59:19.613 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:59:19.614 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:59:19.614 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:59:19.614 INFO:teuthology.orchestra.run.smithi085.stdout: "options": { 2021-02-10T04:59:19.614 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_min": 1 2021-02-10T04:59:19.614 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.614 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:59:19.614 INFO:teuthology.orchestra.run.smithi085.stdout: "mgr_devicehealth": {} 2021-02-10T04:59:19.614 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.614 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.614 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.614 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 2, 2021-02-10T04:59:19.615 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": ".rgw.root", 2021-02-10T04:59:19.615 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:32.688554+0000", 2021-02-10T04:59:19.615 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:59:19.615 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:59:19.615 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:59:19.615 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:59:19.615 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:59:19.615 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:59:19.615 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:59:19.615 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:59:19.615 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:59:19.615 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:59:19.615 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:59:19.616 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:59:19.616 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 42, 2021-02-10T04:59:19.616 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 42, 2021-02-10T04:59:19.616 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 42, 2021-02-10T04:59:19.616 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 42, 2021-02-10T04:59:19.616 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 42, 2021-02-10T04:59:19.616 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:59:19.616 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:59:19.617 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:59:19.617 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:59:19.617 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:59:19.617 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:59:19.617 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:59:19.617 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.617 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "1272", 2021-02-10T04:59:19.617 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:59:19.617 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:59:19.617 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "89", 2021-02-10T04:59:19.617 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:59:19.617 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:59:19.618 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:59:19.618 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:59:19.618 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:59:19.618 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:59:19.618 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:59:19.618 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:59:19.618 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:59:19.618 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:59:19.618 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:59:19.618 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:59:19.618 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:59:19.619 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:59:19.619 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:59:19.619 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:59:19.619 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:59:19.619 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:59:19.619 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:59:19.619 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:59:19.619 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:59:19.619 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:59:19.619 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:59:19.619 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.619 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:59:19.619 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:59:19.619 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:59:19.619 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:59:19.620 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:59:19.620 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:59:19.620 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:59:19.620 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:59:19.620 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:59:19.620 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:59:19.620 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:59:19.620 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:59:19.620 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:59:19.620 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T04:59:19.620 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.620 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.620 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.620 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 3, 2021-02-10T04:59:19.621 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "iscsi", 2021-02-10T04:59:19.621 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:47.658613+0000", 2021-02-10T04:59:19.621 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:59:19.621 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:59:19.621 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:59:19.621 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:59:19.621 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:59:19.621 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:59:19.621 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:59:19.621 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:59:19.621 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:59:19.621 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:59:19.621 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:59:19.622 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:59:19.622 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 13, 2021-02-10T04:59:19.622 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 13, 2021-02-10T04:59:19.622 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 13, 2021-02-10T04:59:19.622 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 13, 2021-02-10T04:59:19.622 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 13, 2021-02-10T04:59:19.622 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:59:19.622 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:59:19.622 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:59:19.622 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:59:19.622 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:59:19.622 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:59:19.622 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:59:19.623 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.623 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "1211", 2021-02-10T04:59:19.623 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:59:19.623 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:59:19.623 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "1209", 2021-02-10T04:59:19.623 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:59:19.623 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:59:19.623 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:59:19.623 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:59:19.623 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:59:19.623 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:59:19.623 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:59:19.623 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:59:19.624 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:59:19.624 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:59:19.624 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:59:19.624 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:59:19.624 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:59:19.624 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:59:19.624 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:59:19.624 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:59:19.624 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:59:19.624 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:59:19.624 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:59:19.624 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:59:19.624 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:59:19.624 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:59:19.625 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:59:19.625 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.625 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:59:19.625 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:59:19.625 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:59:19.625 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:59:19.625 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:59:19.625 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:59:19.625 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:59:19.625 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:59:19.625 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:59:19.625 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:59:19.625 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:59:19.626 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:59:19.626 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:59:19.626 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd": {} 2021-02-10T04:59:19.626 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.626 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.626 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.626 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 4, 2021-02-10T04:59:19.626 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "zone.rgw.log", 2021-02-10T04:59:19.626 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:47.968727+0000", 2021-02-10T04:59:19.626 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:59:19.626 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:59:19.627 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:59:19.627 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:59:19.627 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:59:19.627 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:59:19.627 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:59:19.627 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:59:19.627 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:59:19.627 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:59:19.627 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:59:19.627 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:59:19.627 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 32, 2021-02-10T04:59:19.627 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 32, 2021-02-10T04:59:19.627 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 32, 2021-02-10T04:59:19.628 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 32, 2021-02-10T04:59:19.628 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 32, 2021-02-10T04:59:19.628 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:59:19.628 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "4.16", 2021-02-10T04:59:19.628 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 1256, 2021-02-10T04:59:19.628 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 1255, 2021-02-10T04:59:19.628 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 1255, 2021-02-10T04:59:19.628 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "1048'40", 2021-02-10T04:59:19.628 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "1049'40" 2021-02-10T04:59:19.628 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.628 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "1343", 2021-02-10T04:59:19.628 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:59:19.629 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "1256", 2021-02-10T04:59:19.629 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "1341", 2021-02-10T04:59:19.629 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:59:19.629 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:59:19.629 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:59:19.629 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:59:19.629 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:59:19.629 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:59:19.629 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:59:19.629 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:59:19.629 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:59:19.629 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:59:19.629 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:59:19.630 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:59:19.630 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:59:19.630 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:59:19.630 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:59:19.630 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:59:19.630 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:59:19.630 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:59:19.630 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:59:19.630 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:59:19.630 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:59:19.630 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:59:19.630 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:59:19.630 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.631 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:59:19.631 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:59:19.631 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:59:19.631 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:59:19.631 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:59:19.631 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:59:19.631 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:59:19.631 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:59:19.631 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:59:19.631 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:59:19.631 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:59:19.631 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:59:19.631 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:59:19.632 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T04:59:19.632 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.632 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.632 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.632 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 5, 2021-02-10T04:59:19.632 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "zone.rgw.control", 2021-02-10T04:59:19.632 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:49.278260+0000", 2021-02-10T04:59:19.632 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:59:19.632 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:59:19.632 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:59:19.632 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:59:19.632 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:59:19.632 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:59:19.632 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:59:19.633 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:59:19.633 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:59:19.633 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:59:19.633 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:59:19.633 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:59:19.633 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 32, 2021-02-10T04:59:19.633 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 32, 2021-02-10T04:59:19.633 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 32, 2021-02-10T04:59:19.633 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 32, 2021-02-10T04:59:19.633 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 32, 2021-02-10T04:59:19.633 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:59:19.633 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:59:19.633 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:59:19.634 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:59:19.634 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:59:19.634 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:59:19.634 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:59:19.634 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.634 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "59", 2021-02-10T04:59:19.634 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:59:19.634 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:59:19.634 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T04:59:19.634 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:59:19.634 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:59:19.634 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:59:19.634 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:59:19.635 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:59:19.635 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:59:19.635 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:59:19.635 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:59:19.635 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:59:19.635 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:59:19.635 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:59:19.635 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:59:19.635 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:59:19.635 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:59:19.635 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:59:19.635 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:59:19.635 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:59:19.636 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:59:19.636 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:59:19.636 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:59:19.636 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:59:19.636 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:59:19.636 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:59:19.636 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.636 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:59:19.636 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:59:19.636 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:59:19.636 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:59:19.636 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:59:19.637 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:59:19.637 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:59:19.637 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:59:19.637 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:59:19.637 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:59:19.637 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:59:19.637 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:59:19.637 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:59:19.637 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T04:59:19.637 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.637 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.637 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.637 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 6, 2021-02-10T04:59:19.638 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "zone.rgw.meta", 2021-02-10T04:59:19.638 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:51.792514+0000", 2021-02-10T04:59:19.638 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T04:59:19.638 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T04:59:19.638 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:59:19.638 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:59:19.638 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:59:19.638 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:59:19.638 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:59:19.638 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:59:19.638 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:59:19.638 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:59:19.639 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:59:19.639 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:59:19.639 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 42, 2021-02-10T04:59:19.639 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 42, 2021-02-10T04:59:19.639 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 42, 2021-02-10T04:59:19.639 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 42, 2021-02-10T04:59:19.639 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 42, 2021-02-10T04:59:19.639 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:59:19.639 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:59:19.639 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:59:19.639 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:59:19.639 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:59:19.639 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:59:19.640 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:59:19.640 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.640 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "1046", 2021-02-10T04:59:19.640 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:59:19.640 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:59:19.640 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "1044", 2021-02-10T04:59:19.640 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:59:19.640 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:59:19.640 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T04:59:19.640 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T04:59:19.640 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:59:19.640 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:59:19.640 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:59:19.641 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:59:19.641 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:59:19.641 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:59:19.641 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:59:19.641 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:59:19.641 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:59:19.641 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:59:19.641 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:59:19.641 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:59:19.641 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:59:19.641 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:59:19.641 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:59:19.642 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:59:19.642 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:59:19.642 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:59:19.642 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:59:19.642 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.642 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:59:19.642 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:59:19.642 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:59:19.642 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:59:19.642 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:59:19.642 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:59:19.642 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:59:19.643 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:59:19.643 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:59:19.643 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:59:19.643 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:59:19.643 INFO:teuthology.orchestra.run.smithi085.stdout: "options": { 2021-02-10T04:59:19.643 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_bias": 4, 2021-02-10T04:59:19.643 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_min": 8 2021-02-10T04:59:19.643 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.643 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:59:19.643 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T04:59:19.643 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.643 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.644 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.644 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 7, 2021-02-10T04:59:19.644 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "unique_pool_0", 2021-02-10T04:59:19.644 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:28:23.932391+0000", 2021-02-10T04:59:19.644 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 8193, 2021-02-10T04:59:19.644 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2021-02-10T04:59:19.644 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T04:59:19.644 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T04:59:19.644 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T04:59:19.645 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T04:59:19.645 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T04:59:19.645 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T04:59:19.645 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T04:59:19.645 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T04:59:19.645 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T04:59:19.645 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T04:59:19.645 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 36, 2021-02-10T04:59:19.645 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 36, 2021-02-10T04:59:19.645 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 36, 2021-02-10T04:59:19.646 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 36, 2021-02-10T04:59:19.646 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 36, 2021-02-10T04:59:19.646 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T04:59:19.646 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T04:59:19.646 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T04:59:19.646 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T04:59:19.646 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T04:59:19.646 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T04:59:19.646 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T04:59:19.646 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.646 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "1143", 2021-02-10T04:59:19.647 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T04:59:19.647 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T04:59:19.647 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "1141", 2021-02-10T04:59:19.647 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T04:59:19.647 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T04:59:19.647 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 888, 2021-02-10T04:59:19.647 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 994, 2021-02-10T04:59:19.647 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T04:59:19.647 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T04:59:19.647 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T04:59:19.647 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T04:59:19.648 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T04:59:19.648 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T04:59:19.648 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T04:59:19.648 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T04:59:19.648 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T04:59:19.648 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T04:59:19.648 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T04:59:19.648 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T04:59:19.648 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T04:59:19.648 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T04:59:19.648 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T04:59:19.649 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T04:59:19.649 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T04:59:19.649 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T04:59:19.649 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T04:59:19.649 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.649 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T04:59:19.649 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T04:59:19.649 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T04:59:19.649 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T04:59:19.649 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T04:59:19.649 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T04:59:19.649 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T04:59:19.649 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T04:59:19.650 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T04:59:19.650 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T04:59:19.650 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T04:59:19.650 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T04:59:19.650 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T04:59:19.650 INFO:teuthology.orchestra.run.smithi085.stdout: "rados": {} 2021-02-10T04:59:19.650 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.650 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.650 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:59:19.651 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T04:59:19.651 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.651 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 0, 2021-02-10T04:59:19.651 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "45d0bb5f-35fa-4596-964f-75abcee53bee", 2021-02-10T04:59:19.651 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:59:19.651 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:59:19.651 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:59:19.651 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:59:19.651 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 1227, 2021-02-10T04:59:19.651 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 1315, 2021-02-10T04:59:19.651 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 1332, 2021-02-10T04:59:19.651 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1345, 2021-02-10T04:59:19.651 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 1316, 2021-02-10T04:59:19.651 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:59:19.652 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:59:19.652 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.652 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.652 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.652 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6801", 2021-02-10T04:59:19.652 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.652 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.652 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.652 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.652 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:59:19.653 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.653 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.653 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.653 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6802", 2021-02-10T04:59:19.653 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.653 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.653 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.653 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.653 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:59:19.653 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.653 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.653 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.653 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6804", 2021-02-10T04:59:19.653 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.654 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.654 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.654 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.654 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:59:19.654 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.654 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.654 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.654 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6803", 2021-02-10T04:59:19.654 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.654 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.654 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.654 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.655 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6801/7", 2021-02-10T04:59:19.655 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6802/7", 2021-02-10T04:59:19.655 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6804/7", 2021-02-10T04:59:19.655 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6803/7", 2021-02-10T04:59:19.655 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:59:19.655 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:59:19.655 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:59:19.655 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.655 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.655 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.655 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 1, 2021-02-10T04:59:19.655 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "3421a5ce-872e-4b52-adf7-6aaa595871a4", 2021-02-10T04:59:19.655 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:59:19.656 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:59:19.656 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:59:19.656 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 0.86566162109375, 2021-02-10T04:59:19.656 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:59:19.656 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:59:19.656 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 13, 2021-02-10T04:59:19.656 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1345, 2021-02-10T04:59:19.656 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:59:19.656 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:59:19.656 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:59:19.656 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.656 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.656 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.656 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6805", 2021-02-10T04:59:19.656 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.657 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.657 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.657 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.657 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:59:19.657 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.657 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.657 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.657 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6806", 2021-02-10T04:59:19.657 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.657 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.657 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.657 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.657 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:59:19.657 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.658 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.658 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.658 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6808", 2021-02-10T04:59:19.658 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.658 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.658 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.658 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.658 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:59:19.658 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.658 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.658 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.658 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6807", 2021-02-10T04:59:19.659 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.659 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.659 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.659 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.659 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6805/7", 2021-02-10T04:59:19.659 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6806/7", 2021-02-10T04:59:19.659 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6808/7", 2021-02-10T04:59:19.659 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6807/7", 2021-02-10T04:59:19.659 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:59:19.659 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:59:19.659 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:59:19.659 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.659 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.659 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.660 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 2, 2021-02-10T04:59:19.660 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "96e1952a-1042-4316-9fc9-86813e88b912", 2021-02-10T04:59:19.660 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:59:19.660 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:59:19.660 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:59:19.660 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:59:19.660 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:59:19.660 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:59:19.660 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 18, 2021-02-10T04:59:19.660 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1343, 2021-02-10T04:59:19.660 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:59:19.660 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:59:19.660 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:59:19.660 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.661 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.661 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.661 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6809", 2021-02-10T04:59:19.661 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.661 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.661 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.661 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.661 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:59:19.661 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.661 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.661 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.661 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6810", 2021-02-10T04:59:19.662 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.662 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.662 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.662 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.662 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:59:19.662 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.662 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.662 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.662 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6812", 2021-02-10T04:59:19.662 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.662 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.662 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.662 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.663 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:59:19.663 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.663 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.663 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.663 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6811", 2021-02-10T04:59:19.663 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.663 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.663 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.663 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.663 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6809/7", 2021-02-10T04:59:19.663 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6810/7", 2021-02-10T04:59:19.663 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6812/7", 2021-02-10T04:59:19.663 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6811/7", 2021-02-10T04:59:19.663 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:59:19.664 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:59:19.664 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:59:19.664 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.664 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.664 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.664 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 3, 2021-02-10T04:59:19.664 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "e019f49f-5dd4-4104-9ced-94c710616642", 2021-02-10T04:59:19.664 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:59:19.664 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:59:19.664 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:59:19.664 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:59:19.664 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 25, 2021-02-10T04:59:19.665 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 1094, 2021-02-10T04:59:19.665 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 1111, 2021-02-10T04:59:19.665 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1346, 2021-02-10T04:59:19.665 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 1095, 2021-02-10T04:59:19.665 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:59:19.665 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:59:19.665 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.665 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.665 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.665 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6813", 2021-02-10T04:59:19.665 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.665 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.666 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.666 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.666 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:59:19.666 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.666 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.666 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.666 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6814", 2021-02-10T04:59:19.666 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.666 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.666 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.666 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.666 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:59:19.666 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.667 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.667 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.667 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6816", 2021-02-10T04:59:19.667 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.667 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.667 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.667 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.667 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:59:19.667 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.667 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.667 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.667 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6815", 2021-02-10T04:59:19.667 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.667 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.668 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.668 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.668 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6813/7", 2021-02-10T04:59:19.668 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6814/7", 2021-02-10T04:59:19.668 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6816/7", 2021-02-10T04:59:19.668 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6815/7", 2021-02-10T04:59:19.668 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:59:19.668 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:59:19.668 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:59:19.668 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.668 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.668 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.669 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 4, 2021-02-10T04:59:19.669 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "91c8a80a-37b6-4c30-a32a-647b2e6d801f", 2021-02-10T04:59:19.669 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:59:19.669 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:59:19.669 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:59:19.669 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:59:19.669 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:59:19.669 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:59:19.669 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 31, 2021-02-10T04:59:19.669 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1344, 2021-02-10T04:59:19.669 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:59:19.669 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:59:19.669 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:59:19.669 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.670 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.670 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.670 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6800", 2021-02-10T04:59:19.670 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.670 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.670 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.670 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.670 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:59:19.670 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.670 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.670 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.670 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6801", 2021-02-10T04:59:19.671 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.671 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.671 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.671 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.671 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:59:19.671 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.671 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.671 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.671 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6803", 2021-02-10T04:59:19.671 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.671 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.671 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.671 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.671 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:59:19.672 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.672 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.672 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.672 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6802", 2021-02-10T04:59:19.672 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.672 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.672 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.672 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.672 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6800/7", 2021-02-10T04:59:19.672 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6801/7", 2021-02-10T04:59:19.672 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6803/7", 2021-02-10T04:59:19.672 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6802/7", 2021-02-10T04:59:19.672 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:59:19.673 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:59:19.673 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:59:19.673 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.673 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.673 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.673 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 5, 2021-02-10T04:59:19.673 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e", 2021-02-10T04:59:19.673 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:59:19.673 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:59:19.673 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:59:19.673 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:59:19.673 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:59:19.673 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:59:19.673 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 37, 2021-02-10T04:59:19.673 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1344, 2021-02-10T04:59:19.674 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:59:19.674 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:59:19.674 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:59:19.674 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.674 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.674 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.674 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6804", 2021-02-10T04:59:19.674 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.674 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.674 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.674 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.674 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:59:19.674 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.674 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.674 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.675 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6805", 2021-02-10T04:59:19.675 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.675 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.675 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.675 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.675 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:59:19.675 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.675 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.675 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.675 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6807", 2021-02-10T04:59:19.675 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.675 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.675 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.675 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.676 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:59:19.676 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.676 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.676 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.676 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6806", 2021-02-10T04:59:19.676 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.676 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.676 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.676 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.676 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6804/7", 2021-02-10T04:59:19.676 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6805/7", 2021-02-10T04:59:19.676 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6807/7", 2021-02-10T04:59:19.676 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6806/7", 2021-02-10T04:59:19.676 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:59:19.677 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:59:19.677 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:59:19.677 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.677 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.677 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.677 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 6, 2021-02-10T04:59:19.677 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "adc40430-3eeb-4003-a824-343021243077", 2021-02-10T04:59:19.677 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:59:19.677 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:59:19.677 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:59:19.677 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T04:59:19.677 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:59:19.677 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:59:19.677 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 43, 2021-02-10T04:59:19.678 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1343, 2021-02-10T04:59:19.678 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:59:19.678 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:59:19.678 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:59:19.678 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.678 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.678 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.678 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6808", 2021-02-10T04:59:19.678 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.678 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.678 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.679 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.679 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:59:19.679 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.679 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.679 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.679 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6809", 2021-02-10T04:59:19.679 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.679 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.679 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.679 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.679 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:59:19.679 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.680 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.680 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.680 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6811", 2021-02-10T04:59:19.680 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.680 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.680 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.680 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.680 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:59:19.680 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.680 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.680 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.680 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6810", 2021-02-10T04:59:19.680 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.680 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.681 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.681 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.681 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6808/7", 2021-02-10T04:59:19.681 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6809/7", 2021-02-10T04:59:19.681 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6811/7", 2021-02-10T04:59:19.681 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6810/7", 2021-02-10T04:59:19.681 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:59:19.681 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:59:19.681 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:59:19.681 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.681 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.681 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.681 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 7, 2021-02-10T04:59:19.682 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37", 2021-02-10T04:59:19.682 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T04:59:19.682 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T04:59:19.682 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T04:59:19.682 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 0.846160888671875, 2021-02-10T04:59:19.682 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T04:59:19.682 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T04:59:19.682 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 50, 2021-02-10T04:59:19.682 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1332, 2021-02-10T04:59:19.682 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T04:59:19.682 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T04:59:19.682 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T04:59:19.682 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.683 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.683 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.683 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6812", 2021-02-10T04:59:19.683 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.683 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.683 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.683 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.683 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T04:59:19.683 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.683 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.683 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.683 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6813", 2021-02-10T04:59:19.683 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.684 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.684 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.684 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.684 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T04:59:19.684 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.684 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.684 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.684 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6815", 2021-02-10T04:59:19.684 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.684 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.684 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.684 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.684 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T04:59:19.684 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T04:59:19.685 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.685 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T04:59:19.685 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6814", 2021-02-10T04:59:19.685 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T04:59:19.685 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.685 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.685 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.685 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6812/7", 2021-02-10T04:59:19.685 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6813/7", 2021-02-10T04:59:19.685 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6815/7", 2021-02-10T04:59:19.685 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6814/7", 2021-02-10T04:59:19.685 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T04:59:19.685 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T04:59:19.686 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T04:59:19.686 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.686 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.686 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:59:19.686 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_xinfo": [ 2021-02-10T04:59:19.686 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.686 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 0, 2021-02-10T04:59:19.686 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:59:19.686 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:59:19.686 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:59:19.686 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:59:19.686 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:59:19.686 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:58:56.125499+0000", 2021-02-10T04:59:19.687 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 1316 2021-02-10T04:59:19.687 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.687 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.687 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 1, 2021-02-10T04:59:19.687 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:59:19.687 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:59:19.687 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:59:19.687 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:59:19.687 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:59:19.687 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:59:01.695880+0000", 2021-02-10T04:59:19.687 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:59:19.687 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.687 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.688 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 2, 2021-02-10T04:59:19.688 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:59:19.688 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:59:19.688 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:59:19.688 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:59:19.688 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:59:19.688 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:58:20.041758+0000", 2021-02-10T04:59:19.688 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:59:19.688 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.688 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.688 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 3, 2021-02-10T04:59:19.688 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:59:19.688 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:59:19.689 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:59:19.689 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:59:19.689 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:59:19.689 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:59:00.027876+0000", 2021-02-10T04:59:19.689 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 1095 2021-02-10T04:59:19.689 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.689 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.689 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 4, 2021-02-10T04:59:19.689 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:59:19.689 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:59:19.689 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:59:19.689 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:59:19.689 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:59:19.689 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:58:56.001560+0000", 2021-02-10T04:59:19.690 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:59:19.690 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.690 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.690 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 5, 2021-02-10T04:59:19.690 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:59:19.690 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:59:19.690 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:59:19.690 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:59:19.690 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:59:19.690 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:59:12.932203+0000", 2021-02-10T04:59:19.690 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:59:19.690 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.690 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.691 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 6, 2021-02-10T04:59:19.691 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:59:19.691 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:59:19.691 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:59:19.691 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:59:19.691 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:59:19.691 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:58:31.242532+0000", 2021-02-10T04:59:19.691 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:59:19.691 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.691 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.691 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 7, 2021-02-10T04:59:19.691 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T04:59:19.692 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T04:59:19.692 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T04:59:19.692 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T04:59:19.692 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T04:59:19.692 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:58:48.788976+0000", 2021-02-10T04:59:19.692 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T04:59:19.692 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.692 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:59:19.692 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_upmap": [ 2021-02-10T04:59:19.692 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.692 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "2.23", 2021-02-10T04:59:19.692 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T04:59:19.693 INFO:teuthology.orchestra.run.smithi085.stdout: 3, 2021-02-10T04:59:19.693 INFO:teuthology.orchestra.run.smithi085.stdout: 0, 2021-02-10T04:59:19.693 INFO:teuthology.orchestra.run.smithi085.stdout: 6 2021-02-10T04:59:19.693 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.693 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.693 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:59:19.693 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_upmap_items": [ 2021-02-10T04:59:19.693 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.693 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "2.7", 2021-02-10T04:59:19.693 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:59:19.693 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.693 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:59:19.693 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 4 2021-02-10T04:59:19.694 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.694 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.694 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.694 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.694 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "2.1a", 2021-02-10T04:59:19.694 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:59:19.694 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.694 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:59:19.694 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 2 2021-02-10T04:59:19.694 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.694 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.694 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.694 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.695 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "2.1d", 2021-02-10T04:59:19.695 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:59:19.695 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.695 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:59:19.695 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 2 2021-02-10T04:59:19.695 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.695 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.695 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.695 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.695 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "2.21", 2021-02-10T04:59:19.695 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:59:19.695 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.696 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:59:19.696 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 1 2021-02-10T04:59:19.696 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.696 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.696 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.696 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.696 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "2.28", 2021-02-10T04:59:19.696 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:59:19.696 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.696 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:59:19.696 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 2 2021-02-10T04:59:19.696 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.696 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.697 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.697 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.697 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.5", 2021-02-10T04:59:19.697 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:59:19.697 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.697 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 4, 2021-02-10T04:59:19.697 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 2 2021-02-10T04:59:19.697 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.697 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.697 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.697 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.697 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.10", 2021-02-10T04:59:19.697 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:59:19.698 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.698 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 1, 2021-02-10T04:59:19.698 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 5 2021-02-10T04:59:19.698 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.698 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.698 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.698 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.698 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.6", 2021-02-10T04:59:19.698 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:59:19.698 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.698 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 5, 2021-02-10T04:59:19.698 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 6 2021-02-10T04:59:19.699 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.699 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.699 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.699 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.699 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.8", 2021-02-10T04:59:19.699 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:59:19.699 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.699 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 1, 2021-02-10T04:59:19.699 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 3 2021-02-10T04:59:19.699 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.699 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.699 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.699 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.699 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.10", 2021-02-10T04:59:19.700 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:59:19.700 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.700 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:59:19.700 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 3 2021-02-10T04:59:19.700 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.700 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.700 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.700 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.700 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.18", 2021-02-10T04:59:19.700 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:59:19.700 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.700 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 4, 2021-02-10T04:59:19.700 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 3 2021-02-10T04:59:19.701 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.701 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.701 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.701 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.701 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.1b", 2021-02-10T04:59:19.701 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:59:19.701 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.701 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 0, 2021-02-10T04:59:19.701 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 3 2021-02-10T04:59:19.701 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.701 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.701 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.701 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.702 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "7.6", 2021-02-10T04:59:19.702 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:59:19.702 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.702 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 3, 2021-02-10T04:59:19.702 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 1 2021-02-10T04:59:19.702 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.702 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.702 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.702 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.702 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "7.15", 2021-02-10T04:59:19.702 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T04:59:19.702 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.703 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T04:59:19.703 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 4 2021-02-10T04:59:19.703 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.703 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.703 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.703 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:59:19.703 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_temp": [ 2021-02-10T04:59:19.703 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.703 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.16", 2021-02-10T04:59:19.703 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T04:59:19.703 INFO:teuthology.orchestra.run.smithi085.stdout: 0, 2021-02-10T04:59:19.703 INFO:teuthology.orchestra.run.smithi085.stdout: 1, 2021-02-10T04:59:19.703 INFO:teuthology.orchestra.run.smithi085.stdout: 2 2021-02-10T04:59:19.704 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.704 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.704 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.704 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.18", 2021-02-10T04:59:19.704 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T04:59:19.704 INFO:teuthology.orchestra.run.smithi085.stdout: 5, 2021-02-10T04:59:19.704 INFO:teuthology.orchestra.run.smithi085.stdout: 6, 2021-02-10T04:59:19.704 INFO:teuthology.orchestra.run.smithi085.stdout: 7 2021-02-10T04:59:19.704 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.704 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.704 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.704 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.19", 2021-02-10T04:59:19.704 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T04:59:19.705 INFO:teuthology.orchestra.run.smithi085.stdout: 3, 2021-02-10T04:59:19.705 INFO:teuthology.orchestra.run.smithi085.stdout: 1, 2021-02-10T04:59:19.705 INFO:teuthology.orchestra.run.smithi085.stdout: 4 2021-02-10T04:59:19.705 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.705 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.705 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.705 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.1a", 2021-02-10T04:59:19.705 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T04:59:19.705 INFO:teuthology.orchestra.run.smithi085.stdout: 1, 2021-02-10T04:59:19.705 INFO:teuthology.orchestra.run.smithi085.stdout: 6, 2021-02-10T04:59:19.705 INFO:teuthology.orchestra.run.smithi085.stdout: 7 2021-02-10T04:59:19.706 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.706 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.706 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.706 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.1b", 2021-02-10T04:59:19.706 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T04:59:19.706 INFO:teuthology.orchestra.run.smithi085.stdout: 4, 2021-02-10T04:59:19.706 INFO:teuthology.orchestra.run.smithi085.stdout: 1, 2021-02-10T04:59:19.706 INFO:teuthology.orchestra.run.smithi085.stdout: 0 2021-02-10T04:59:19.706 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.706 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.706 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.706 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.1c", 2021-02-10T04:59:19.706 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T04:59:19.707 INFO:teuthology.orchestra.run.smithi085.stdout: 3, 2021-02-10T04:59:19.707 INFO:teuthology.orchestra.run.smithi085.stdout: 4, 2021-02-10T04:59:19.707 INFO:teuthology.orchestra.run.smithi085.stdout: 6 2021-02-10T04:59:19.707 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.707 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.707 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.707 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.1e", 2021-02-10T04:59:19.707 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T04:59:19.707 INFO:teuthology.orchestra.run.smithi085.stdout: 1, 2021-02-10T04:59:19.707 INFO:teuthology.orchestra.run.smithi085.stdout: 4, 2021-02-10T04:59:19.707 INFO:teuthology.orchestra.run.smithi085.stdout: 6 2021-02-10T04:59:19.707 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.707 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.708 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T04:59:19.708 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.1f", 2021-02-10T04:59:19.708 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T04:59:19.708 INFO:teuthology.orchestra.run.smithi085.stdout: 1, 2021-02-10T04:59:19.708 INFO:teuthology.orchestra.run.smithi085.stdout: 3, 2021-02-10T04:59:19.708 INFO:teuthology.orchestra.run.smithi085.stdout: 4 2021-02-10T04:59:19.708 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T04:59:19.708 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.708 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T04:59:19.708 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_temp": [], 2021-02-10T04:59:19.708 INFO:teuthology.orchestra.run.smithi085.stdout: "blocklist": { 2021-02-10T04:59:19.708 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/801393150": "2021-02-11T04:21:37.851607+0000", 2021-02-10T04:59:19.709 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/3049671696": "2021-02-11T04:21:37.851607+0000", 2021-02-10T04:59:19.709 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/4056505602": "2021-02-11T04:21:20.700478+0000", 2021-02-10T04:59:19.709 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:6800/7": "2021-02-11T04:21:20.700478+0000", 2021-02-10T04:59:19.709 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/1879026062": "2021-02-11T04:21:20.700478+0000" 2021-02-10T04:59:19.709 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.709 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profiles": { 2021-02-10T04:59:19.709 INFO:teuthology.orchestra.run.smithi085.stdout: "default": { 2021-02-10T04:59:19.709 INFO:teuthology.orchestra.run.smithi085.stdout: "crush-failure-domain": "osd", 2021-02-10T04:59:19.709 INFO:teuthology.orchestra.run.smithi085.stdout: "k": "2", 2021-02-10T04:59:19.709 INFO:teuthology.orchestra.run.smithi085.stdout: "m": "1", 2021-02-10T04:59:19.709 INFO:teuthology.orchestra.run.smithi085.stdout: "plugin": "jerasure", 2021-02-10T04:59:19.709 INFO:teuthology.orchestra.run.smithi085.stdout: "ruleset-failure-domain": "osd", 2021-02-10T04:59:19.709 INFO:teuthology.orchestra.run.smithi085.stdout: "technique": "reed_sol_van" 2021-02-10T04:59:19.710 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.710 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T04:59:19.710 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps_queue": [], 2021-02-10T04:59:19.710 INFO:teuthology.orchestra.run.smithi085.stdout: "new_removed_snaps": [], 2021-02-10T04:59:19.710 INFO:teuthology.orchestra.run.smithi085.stdout: "new_purged_snaps": [], 2021-02-10T04:59:19.710 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_node_flags": {}, 2021-02-10T04:59:19.710 INFO:teuthology.orchestra.run.smithi085.stdout: "device_class_flags": {}, 2021-02-10T04:59:19.710 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_mode": { 2021-02-10T04:59:19.710 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_mode_enabled": false, 2021-02-10T04:59:19.710 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_bucket_count": 0, 2021-02-10T04:59:19.710 INFO:teuthology.orchestra.run.smithi085.stdout: "degraded_stretch_mode": 0, 2021-02-10T04:59:19.710 INFO:teuthology.orchestra.run.smithi085.stdout: "recovering_stretch_mode": 0, 2021-02-10T04:59:19.710 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_mode_bucket": 0 2021-02-10T04:59:19.711 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T04:59:19.711 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T04:59:19.894 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:59:19 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:59:20.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:19 smithi085 conmon[29319]: cluster 2021-02-10T04:59:18.563813+0000 mgr.y (mgr.14140) 1518 : cluster [DBG] pgmap v2466: 208 pgs: 6 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 1 remapped+peering, 1 peering, 2 unknown, 197 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 18 KiB/s rd, 1.2 KiB/s wr, 35 op/s; 157/2151 objects misplaced (7.299%) 2021-02-10T04:59:20.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:19 smithi085 conmon[29319]: cluster 2021-02-10T04:59:18.951467+0000 mon.a (mon.0) 2995 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2021-02-10T04:59:20.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:19 smithi085 conmon[29319]: cluster 2021-02-10T04:59:18.951512+0000 mon.a (mon.0) 2996 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-02-10T04:59:20.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:19 smithi085 conmon[29319]: audit 2021-02-10T04:59:19.604283+0000 mon.c (mon.1) 159 : audit [DBG] from='client.? 172.21.15.85:0/767181371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-02-10T04:59:20.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:19 smithi085 conmon[35961]: cluster 2021-02-10T04:59:18.563813+0000 mgr.y (mgr.14140) 1518 : cluster [DBG] pgmap v2466: 208 pgs: 6 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 1 remapped+peering, 1 peering, 2 unknown, 197 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 18 KiB/s rd, 1.2 KiB/s wr, 35 op/s; 157/2151 objects misplaced (7.299%) 2021-02-10T04:59:20.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:19 smithi085 conmon[35961]: cluster 2021-02-10T04:59:18.951467+0000 mon.a (mon.0) 2995 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2021-02-10T04:59:20.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:19 smithi085 conmon[35961]: cluster 2021-02-10T04:59:18.951512+0000 mon.a (mon.0) 2996 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-02-10T04:59:20.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:19 smithi085 conmon[35961]: audit 2021-02-10T04:59:19.604283+0000 mon.c (mon.1) 159 : audit [DBG] from='client.? 172.21.15.85:0/767181371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-02-10T04:59:20.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:19 smithi164 conmon[29288]: cluster 2021-02-10T04:59:20.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:19 smithi164 conmon[29288]: 2021-02-10T04:59:18.563813 2021-02-10T04:59:20.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:19 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1518 : cluster [DBG] pgmap v2466: 208 pgs: 6 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 1 remapped+peering, 1 peering, 2 unknown, 197 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 18 KiB/s rd, 1.2 KiB/s wr, 35 op/s; 157/2151 objects misplaced (7.299%) 2021-02-10T04:59:20.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:19 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:59:20.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:19 smithi164 conmon[29288]: -10T04:59:18.951467+0000 mon.a (mon.0) 2995 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2021-02-10T04:59:20.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:19 smithi164 conmon[29288]: cluster 2021-02-10T04:59:20.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:19 smithi164 conmon[29288]: 2021-02-10T04:59:18.951512+0000 2021-02-10T04:59:20.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:19 smithi164 conmon[29288]: mon.a (mon.0) 2996 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-02-10T04:59:20.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:19 smithi164 conmon[29288]: audit 2021-02-10T04:59:20.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:19 smithi164 conmon[29288]: 2021-02-10T04:59:19.604283+0000 2021-02-10T04:59:20.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:19 smithi164 conmon[29288]: mon.c (mon.1) 159 : audit [DBG] from='client.? 172.21.15.85:0/767181371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-02-10T04:59:21.100 INFO:tasks.thrashosds.thrasher:j is {'epoch': 1347, 'fsid': '42224fce-6b57-11eb-8feb-001a4aab830c', 'created': '2021-02-10T04:20:57.933279+0000', 'modified': '2021-02-10T04:59:18.628529+0000', 'last_up_change': '2021-02-10T04:58:55.817746+0000', 'last_in_change': '2021-02-10T04:51:35.701776+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-02-10T04:24:05.258028+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1266', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '110', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2021-02-10T04:25:32.688554+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 42, 'pg_placement_num': 42, 'pg_placement_num_target': 42, 'pg_num_target': 42, 'pg_num_pending': 42, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1272', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '89', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2021-02-10T04:25:47.658613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 13, 'pg_placement_num': 13, 'pg_placement_num_target': 13, 'pg_num_target': 13, 'pg_num_pending': 13, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1211', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '1209', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'zone.rgw.log', 'create_time': '2021-02-10T04:25:47.968727+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '4.16', 'ready_epoch': 1256, 'last_epoch_started': 1255, 'last_epoch_clean': 1255, 'source_version': "1048'40", 'target_version': "1049'40"}, 'last_change': '1343', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '1256', 'last_force_op_resend_preluminous': '1341', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'zone.rgw.control', 'create_time': '2021-02-10T04:25:49.278260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'zone.rgw.meta', 'create_time': '2021-02-10T04:25:51.792514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 42, 'pg_placement_num': 42, 'pg_placement_num_target': 42, 'pg_num_target': 42, 'pg_num_pending': 42, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1046', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '1044', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 8}, 'application_metadata': {'rgw': {}}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2021-02-10T04:28:23.932391+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 36, 'pg_placement_num': 36, 'pg_placement_num_target': 36, 'pg_num_target': 36, 'pg_num_pending': 36, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1143', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '1141', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 888, 'snap_epoch': 994, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '45d0bb5f-35fa-4596-964f-75abcee53bee', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 1227, 'last_clean_end': 1315, 'up_from': 1332, 'up_thru': 1345, 'down_at': 1316, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6801', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6802', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6804', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6803', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6801/7', 'cluster_addr': '172.21.15.85:6802/7', 'heartbeat_back_addr': '172.21.15.85:6804/7', 'heartbeat_front_addr': '172.21.15.85:6803/7', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '3421a5ce-872e-4b52-adf7-6aaa595871a4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.86566162109375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 1345, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6805', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6806', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6808', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6807', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6805/7', 'cluster_addr': '172.21.15.85:6806/7', 'heartbeat_back_addr': '172.21.15.85:6808/7', 'heartbeat_front_addr': '172.21.15.85:6807/7', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '96e1952a-1042-4316-9fc9-86813e88b912', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 1343, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6809', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6810', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6812', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6811', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6809/7', 'cluster_addr': '172.21.15.85:6810/7', 'heartbeat_back_addr': '172.21.15.85:6812/7', 'heartbeat_front_addr': '172.21.15.85:6811/7', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'e019f49f-5dd4-4104-9ced-94c710616642', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 25, 'last_clean_end': 1094, 'up_from': 1111, 'up_thru': 1346, 'down_at': 1095, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6813', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6814', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6816', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6815', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6813/7', 'cluster_addr': '172.21.15.85:6814/7', 'heartbeat_back_addr': '172.21.15.85:6816/7', 'heartbeat_front_addr': '172.21.15.85:6815/7', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '91c8a80a-37b6-4c30-a32a-647b2e6d801f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 1344, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6800', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6801', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6803', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6802', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6800/7', 'cluster_addr': '172.21.15.164:6801/7', 'heartbeat_back_addr': '172.21.15.164:6803/7', 'heartbeat_front_addr': '172.21.15.164:6802/7', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 1344, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6804', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6805', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6807', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6806', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6804/7', 'cluster_addr': '172.21.15.164:6805/7', 'heartbeat_back_addr': '172.21.15.164:6807/7', 'heartbeat_front_addr': '172.21.15.164:6806/7', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'adc40430-3eeb-4003-a824-343021243077', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 43, 'up_thru': 1343, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6808', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6809', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6811', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6810', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6808/7', 'cluster_addr': '172.21.15.164:6809/7', 'heartbeat_back_addr': '172.21.15.164:6811/7', 'heartbeat_front_addr': '172.21.15.164:6810/7', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.846160888671875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 1332, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6812', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6813', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6815', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6814', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6812/7', 'cluster_addr': '172.21.15.164:6813/7', 'heartbeat_back_addr': '172.21.15.164:6815/7', 'heartbeat_front_addr': '172.21.15.164:6814/7', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:58:56.125499+0000', 'dead_epoch': 1316}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:59:01.695880+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:58:20.041758+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:59:00.027876+0000', 'dead_epoch': 1095}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:58:56.001560+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:59:12.932203+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:58:31.242532+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:58:48.788976+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.23', 'osds': [3, 0, 6]}], 'pg_upmap_items': [{'pgid': '2.7', 'mappings': [{'from': 7, 'to': 4}]}, {'pgid': '2.1a', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '2.1d', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '2.21', 'mappings': [{'from': 7, 'to': 1}]}, {'pgid': '2.28', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.5', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '4.10', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '5.6', 'mappings': [{'from': 5, 'to': 6}]}, {'pgid': '5.8', 'mappings': [{'from': 1, 'to': 3}]}, {'pgid': '5.10', 'mappings': [{'from': 7, 'to': 3}]}, {'pgid': '5.18', 'mappings': [{'from': 4, 'to': 3}]}, {'pgid': '5.1b', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '7.6', 'mappings': [{'from': 3, 'to': 1}]}, {'pgid': '7.15', 'mappings': [{'from': 7, 'to': 4}]}], 'pg_temp': [{'pgid': '4.16', 'osds': [0, 1, 2]}, {'pgid': '4.18', 'osds': [5, 6, 7]}, {'pgid': '4.19', 'osds': [3, 1, 4]}, {'pgid': '4.1a', 'osds': [1, 6, 7]}, {'pgid': '4.1b', 'osds': [4, 1, 0]}, {'pgid': '4.1c', 'osds': [3, 4, 6]}, {'pgid': '4.1e', 'osds': [1, 4, 6]}, {'pgid': '4.1f', 'osds': [1, 3, 4]}], 'primary_temp': [], 'blocklist': {'172.21.15.85:0/801393150': '2021-02-11T04:21:37.851607+0000', '172.21.15.85:0/3049671696': '2021-02-11T04:21:37.851607+0000', '172.21.15.85:0/4056505602': '2021-02-11T04:21:20.700478+0000', '172.21.15.85:6800/7': '2021-02-11T04:21:20.700478+0000', '172.21.15.85:0/1879026062': '2021-02-11T04:21:20.700478+0000'}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2021-02-10T04:59:21.101 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T04:59:21.239 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:20 smithi085 conmon[29319]: audit 2021-02-10T04:59:21.239 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:20 smithi085 conmon[29319]: 2021-02-10T04:59:21.239 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:20 smithi085 conmon[29319]: 2021 2021-02-10T04:59:21.239 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:20 smithi085 conmon[29319]: - 2021-02-10T04:59:21.240 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:20 smithi085 conmon[29319]: 02 2021-02-10T04:59:21.240 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:20 smithi085 conmon[29319]: - 2021-02-10T04:59:21.240 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:20 smithi085 conmon[29319]: 10T04:59:19.543748 2021-02-10T04:59:21.240 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:20 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 1519 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:59:21.240 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:20 smithi085 conmon[35961]: audit 2021-02-10T04:59:19.543748+0000 2021-02-10T04:59:21.240 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:20 smithi085 conmon[35961]: mgr.y (mgr.14140) 1519 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:59:21.345 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:20 smithi164 conmon[29288]: audit 2021-02-10T04:59:19.543748+0000 mgr.y ( 2021-02-10T04:59:21.345 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:20 smithi164 conmon[29288]: mgr.14140) 1519 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:59:21.354 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:59:21.763 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:59:21.781 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:59:21.799 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:59:21.818 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:59:21.839 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:59:21.860 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:59:21.885 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:59:21.903 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:59:21.905 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:59:21.905 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:59:21.905 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:59:21.905 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:59:21.906 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:59:21.906 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:59:21.906 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:59:21.906 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:59:22.097 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:21 smithi085 conmon[29319]: cluster 2021-02-10T04:59:22.097 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:21 smithi085 conmon[29319]: 2021-02-10 2021-02-10T04:59:22.097 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:21 smithi085 conmon[29319]: T04:59:20.564407+0000 mgr.y (mgr.14140) 1520 : cluster [DBG] pgmap v2468: 208 pgs: 7 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 1 remapped+peering, 2 unknown, 197 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 1.9 KiB/s wr, 48 op/s; 181/2151 objects misplaced (8.415%); 197 B/s, 1 objects/s recovering 2021-02-10T04:59:22.097 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:21 smithi085 conmon[29319]: cluster 2021-02-10T04:59:20.882073+0000 osd.2 ( 2021-02-10T04:59:22.097 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:21 smithi085 conmon[29319]: osd.2) 2039 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:59:22.097 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:21 smithi085 conmon[29319]: cluster 2021-02-10T04:59:22.097 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:21 smithi085 conmon[29319]: 2021-02-10T04:59:20. 2021-02-10T04:59:22.098 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:21 smithi085 conmon[29319]: 883564+0000 osd.2 (osd.2) 2040 : 2021-02-10T04:59:22.098 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:21 smithi085 conmon[29319]: cluster [DBG] purged_snaps scrub ok 2021-02-10T04:59:22.098 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:21 smithi085 conmon[29319]: cluster 2021-02-10T04:59:22.098 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:21 smithi085 conmon[29319]: 2021-02-10T04:59:22.098 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:21 smithi085 conmon[29319]: 2021 2021-02-10T04:59:22.098 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:21 smithi085 conmon[29319]: -02-10T04:59: 2021-02-10T04:59:22.098 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:21 smithi085 conmon[29319]: 20.973372+0000 mon.a (mon.0) 2997 : cluster [DBG] osdmap e1348: 8 total, 8 up, 8 in 2021-02-10T04:59:22.098 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:21 smithi085 conmon[29319]: audit 2021-02-10T04:59:22.098 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:21 smithi085 conmon[29319]: 2021-02-10T04:59:21.745545+0000 2021-02-10T04:59:22.098 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:21 smithi085 conmon[29319]: mon.a (mon.0) 2998 : audit 2021-02-10T04:59:22.098 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:21 smithi085 conmon[29319]: [DBG] from='client.? 172.21.15.85:0/4192192166' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:59:22.098 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:21 smithi085 conmon[35961]: cluster 2021-02-10T04:59:20.564407+0000 2021-02-10T04:59:22.099 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:21 smithi085 conmon[35961]: mgr.y (mgr.14140) 1520 : cluster [DBG] pgmap v2468: 208 pgs: 7 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 1 remapped+peering, 2 unknown, 197 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 1.9 KiB/s wr, 48 op/s; 181/2151 objects misplaced (8.415%); 197 B/s, 1 objects/s recovering 2021-02-10T04:59:22.099 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:21 smithi085 conmon[35961]: cluster 2021-02-10T04:59:20.882073+0000 osd.2 (osd.2) 2039 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:59:22.099 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:21 smithi085 conmon[35961]: cluster 2021-02-10T04:59:20.883564+0000 osd.2 (osd.2) 2040 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:59:22.099 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:21 smithi085 conmon[35961]: cluster 2021-02-10T04:59:20.973372+0000 mon.a (mon.0) 2997 : cluster [DBG] osdmap e1348: 8 total, 8 up, 8 in 2021-02-10T04:59:22.099 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:21 smithi085 conmon[35961]: audit 2021-02-10T04:59:21.745545+0000 mon.a (mon.0) 2998 : audit [DBG] from='client.? 172.21.15.85:0/4192192166' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:59:22.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:21 smithi164 conmon[29288]: cluster 2021-02-10T04:59:22.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:21 smithi164 conmon[29288]: 2021-02-10T04:59:20.564407+0000 2021-02-10T04:59:22.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:21 smithi164 conmon[29288]: mgr.y (mgr.14140) 1520 : cluster [DBG] pgmap v2468: 208 pgs: 7 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 1 remapped+peering, 2 unknown, 197 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 1.9 KiB/s wr, 48 op/s; 181/2151 objects misplaced (8.415%); 197 B/s, 1 objects/s recovering 2021-02-10T04:59:22.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:21 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:59:22.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:21 smithi164 conmon[29288]: -02-10T04:59:20.882073+0000 osd.2 ( 2021-02-10T04:59:22.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:21 smithi164 conmon[29288]: osd.2) 2039 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:59:22.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:21 smithi164 conmon[29288]: cluster 2021-02-10T04:59:22.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:21 smithi164 conmon[29288]: 2021-02-10T04:59:20.883564+0000 2021-02-10T04:59:22.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:21 smithi164 conmon[29288]: osd.2 (osd.2) 2040 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:59:22.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:21 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:59:22.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:21 smithi164 conmon[29288]: 10T04:59:20.973372+0000 mon.a (mon.0 2021-02-10T04:59:22.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:21 smithi164 conmon[29288]: ) 2997 : cluster [DBG] osdmap e1348: 8 total, 8 up, 8 in 2021-02-10T04:59:22.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:21 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:59:22.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:21 smithi164 conmon[29288]: T04:59:21.745545+0000 mon.a (mon.0 2021-02-10T04:59:22.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:21 smithi164 conmon[29288]: ) 2998 : audit [DBG] from='client.? 172.21.15.85:0/4192192166' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:59:22.991 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is unset 2021-02-10T04:59:23.015 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:22 smithi085 conmon[29319]: audit 2021-02-10T04:59:22.303445+0000 mon.c (mon.1) 160 : audit [INF] from='client.? 172.21.15.85:0/3875386410' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:59:23.016 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:22 smithi085 conmon[29319]: audit 2021-02-10T04:59:22.303755+0000 mon.a (mon.0) 2999 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:59:23.016 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:22 smithi085 conmon[35961]: audit 2021-02-10T04:59:22.303445+0000 mon.c (mon.1) 160 : audit [INF] from='client.? 172.21.15.85:0/3875386410' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:59:23.016 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:22 smithi085 conmon[35961]: audit 2021-02-10T04:59:22.303755+0000 mon.a (mon.0) 2999 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:59:23.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:22 smithi164 conmon[29288]: audit 2021-02-10T04:59:22.303445+0000 2021-02-10T04:59:23.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:22 smithi164 conmon[29288]: mon.c (mon.1) 160 : audit [INF] from='client.? 172.21.15.85:0/3875386410' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:59:23.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:22 smithi164 conmon[29288]: audit 2021-02-10T04:59:23.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:22 smithi164 conmon[29288]: 2021 2021-02-10T04:59:23.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:22 smithi164 conmon[29288]: - 2021-02-10T04:59:23.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:22 smithi164 conmon[29288]: 02 2021-02-10T04:59:23.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:22 smithi164 conmon[29288]: - 2021-02-10T04:59:23.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:22 smithi164 conmon[29288]: 10 2021-02-10T04:59:23.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:22 smithi164 conmon[29288]: T04:59:22.303755 2021-02-10T04:59:23.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:22 smithi164 conmon[29288]: +0000 mon.a (mon.0) 2999 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:59:24.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:23 smithi085 conmon[29319]: cluster 2021-02-10T04:59:22.564995+0000 mgr.y (mgr.14140) 1521 : cluster 2021-02-10T04:59:24.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:23 smithi085 conmon[29319]: [DBG] pgmap v2470: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 87 KiB/s rd, 6.1 KiB/s wr, 169 op/s; 188/2205 objects misplaced (8.526%); 1010 B/s, 16 objects/s recovering 2021-02-10T04:59:24.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:23 smithi085 conmon[29319]: cluster 2021-02-10T04:59:22.972144+0000 mon.a (mon.0) 3000 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2021-02-10T04:59:24.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:23 smithi085 conmon[29319]: cluster 2021-02-10T04:59:22.972511+0000 mon.a (mon.0) 3001 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:59:24.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:23 smithi085 conmon[29319]: audit 2021-02-10T04:59:22.986934+0000 mon.a (mon.0) 3002 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:59:24.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:23 smithi085 conmon[29319]: cluster 2021-02-10T04:59:22.986986+0000 mon.a (mon.0) 3003 : cluster [DBG] osdmap e1349: 8 total, 8 up, 8 in 2021-02-10T04:59:24.279 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:23 smithi085 conmon[35961]: cluster 2021-02-10T04:59:22.564995+0000 mgr.y (mgr 2021-02-10T04:59:24.279 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:23 smithi085 conmon[35961]: .14140) 1521 : cluster [DBG] pgmap v2470: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 87 KiB/s rd, 6.1 KiB/s wr, 169 op/s; 188/2205 objects misplaced (8.526%); 1010 B/s, 16 objects/s recovering 2021-02-10T04:59:24.279 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:23 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:59:24.279 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:23 smithi085 conmon[35961]: 04:59:22.972144+0000 mon.a (mon.0) 3000 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2021-02-10T04:59:24.280 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:23 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:59:24.280 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:23 smithi085 conmon[35961]: 02-10T04:59:22.972511+0000 2021-02-10T04:59:24.280 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:23 smithi085 conmon[35961]: mon.a (mon.0) 3001 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:59:24.280 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:23 smithi085 conmon[35961]: 2021-02-10T04:59:24.280 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:23 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:59:24.280 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:23 smithi085 conmon[35961]: 02-10T04:59:22.986934+0000 mon.a (mon. 2021-02-10T04:59:24.280 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:23 smithi085 conmon[35961]: 0) 3002 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:59:24.280 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:23 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:59:24.280 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:23 smithi085 conmon[35961]: T04:59:22.986986+0000 mon.a ( 2021-02-10T04:59:24.280 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:23 smithi085 conmon[35961]: mon.0) 3003 : cluster [DBG] osdmap e1349: 8 total, 8 up, 8 in 2021-02-10T04:59:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:23 smithi164 conmon[29288]: cluster 2021-02-10T04:59:22.564995+0000 mgr.y (mgr.14140) 1521 : cluster [DBG] pgmap v2470: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 87 KiB/s rd, 6.1 KiB/s wr, 169 op/s; 188/2205 objects misplaced (8.526%); 1010 B/s, 16 objects/s recovering 2021-02-10T04:59:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:23 smithi164 conmon[29288]: cluster 2021-02-10T04:59:22.972144+0000 mon.a (mon.0) 3000 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2021-02-10T04:59:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:23 smithi164 conmon[29288]: cluster 2021-02-10T04:59:22.972511+0000 mon.a (mon.0) 2021-02-10T04:59:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:23 smithi164 conmon[29288]: 3001 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:59:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:23 smithi164 conmon[29288]: audit 2021-02-10T04:59:22.986934+0000 mon.a (mon.0) 3002 2021-02-10T04:59:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:23 smithi164 conmon[29288]: : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:59:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:23 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:59:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:23 smithi164 conmon[29288]: :59:22.986986+0000 mon.a (mon.0) 3003 : cluster [DBG] osdmap e1349: 8 total, 8 up, 8 in 2021-02-10T04:59:24.808 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:59:24.812 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T04:59:24.932 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:59:25.164 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:59:25.820 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":2472,"stamp":"2021-02-10T04:59:24.565132+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":936255846,"num_objects":735,"num_object_clones":326,"num_object_copies":2205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":188,"num_objects_unfound":0,"num_objects_dirty":735,"num_whiteouts":10,"num_read":7667,"num_read_kb":4152662,"num_write":4560,"num_write_kb":1137191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":866,"num_bytes_recovered":289531493,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35847,"ondisk_log_size":35847,"up":624,"acting":624,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":644,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2324728,"kb_used_data":2006136,"kb_used_omap":711,"kb_used_meta":317816,"kb_avail":747636488,"statfs":{"total":767960285184,"available":765579763712,"internally_reserved":0,"allocated":2054283264,"data_stored":2611004611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":728429,"internal_metadata":325444243},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1190,"num_objects":18,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":31,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":153,"num_read_kb":124,"num_write":87,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":2474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001727"},"pg_stats":[{"pgid":"6.29","version":"0'0","reported_seq":"1218","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.964182+0000","last_change":"2021-02-10T04:58:56.856970+0000","last_active":"2021-02-10T04:59:17.964182+0000","last_peered":"2021-02-10T04:59:17.964182+0000","last_clean":"2021-02-10T04:59:17.964182+0000","last_became_active":"2021-02-10T04:58:56.856652+0000","last_became_peered":"2021-02-10T04:58:56.856652+0000","last_unstale":"2021-02-10T04:59:17.964182+0000","last_undegraded":"2021-02-10T04:59:17.964182+0000","last_fullsized":"2021-02-10T04:59:17.964182+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:24.121684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:24.121684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":"244","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.632337+0000","last_change":"2021-02-10T04:52:30.133105+0000","last_active":"2021-02-10T04:59:18.632337+0000","last_peered":"2021-02-10T04:59:18.632337+0000","last_clean":"2021-02-10T04:59:18.632337+0000","last_became_active":"2021-02-10T04:51:36.864926+0000","last_became_peered":"2021-02-10T04:51:36.864926+0000","last_unstale":"2021-02-10T04:59:18.632337+0000","last_undegraded":"2021-02-10T04:59:18.632337+0000","last_fullsized":"2021-02-10T04:59:18.632337+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.133043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.133043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"1191'4","reported_seq":"364","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:18.078210+0000","last_change":"2021-02-10T04:53:46.736164+0000","last_active":"2021-02-10T04:59:18.078210+0000","last_peered":"2021-02-10T04:59:18.078210+0000","last_clean":"2021-02-10T04:59:18.078210+0000","last_became_active":"2021-02-10T04:53:46.735711+0000","last_became_peered":"2021-02-10T04:53:46.735711+0000","last_unstale":"2021-02-10T04:59:18.078210+0000","last_undegraded":"2021-02-10T04:59:18.078210+0000","last_fullsized":"2021-02-10T04:59:18.078210+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'3","last_scrub_stamp":"2021-02-10T04:51:01.185820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.185820+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"1261'84","reported_seq":"1491","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.972971+0000","last_change":"2021-02-10T04:59:13.928336+0000","last_active":"2021-02-10T04:59:20.972971+0000","last_peered":"2021-02-10T04:59:20.972971+0000","last_clean":"2021-02-10T04:59:20.972971+0000","last_became_active":"2021-02-10T04:59:13.927863+0000","last_became_peered":"2021-02-10T04:59:13.927863+0000","last_unstale":"2021-02-10T04:59:20.972971+0000","last_undegraded":"2021-02-10T04:59:20.972971+0000","last_fullsized":"2021-02-10T04:59:20.972971+0000","mapping_epoch":55,"log_start":"1132'66","ondisk_log_start":"1132'66","created":55,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":0,"last_scrub":"1132'66","last_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_deep_scrub":"1132'66","last_deep_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.430399+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":808,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":93,"num_read_kb":75,"num_write":53,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":519,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"253","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.633380+0000","last_change":"2021-02-10T04:53:41.722609+0000","last_active":"2021-02-10T04:59:18.633380+0000","last_peered":"2021-02-10T04:59:18.633380+0000","last_clean":"2021-02-10T04:59:18.633380+0000","last_became_active":"2021-02-10T04:51:36.875000+0000","last_became_peered":"2021-02-10T04:51:36.875000+0000","last_unstale":"2021-02-10T04:59:18.633380+0000","last_undegraded":"2021-02-10T04:59:18.633380+0000","last_fullsized":"2021-02-10T04:59:18.633380+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:53:41.722516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:41.722516+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"1416","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.976096+0000","last_change":"2021-02-10T04:59:07.257616+0000","last_active":"2021-02-10T04:59:20.976096+0000","last_peered":"2021-02-10T04:59:20.976096+0000","last_clean":"2021-02-10T04:59:20.976096+0000","last_became_active":"2021-02-10T04:58:56.875749+0000","last_became_peered":"2021-02-10T04:58:56.875749+0000","last_unstale":"2021-02-10T04:59:20.976096+0000","last_undegraded":"2021-02-10T04:59:20.976096+0000","last_fullsized":"2021-02-10T04:59:20.976096+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:48.408748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:50:48.408748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"1003'1689","reported_seq":"4767","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.637655+0000","last_change":"2021-02-10T04:53:38.170873+0000","last_active":"2021-02-10T04:59:18.637655+0000","last_peered":"2021-02-10T04:59:18.637655+0000","last_clean":"2021-02-10T04:59:18.637655+0000","last_became_active":"2021-02-10T04:52:26.030105+0000","last_became_peered":"2021-02-10T04:52:26.030105+0000","last_unstale":"2021-02-10T04:59:18.637655+0000","last_undegraded":"2021-02-10T04:59:18.637655+0000","last_fullsized":"2021-02-10T04:59:18.637655+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:53:38.170802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.170802+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":99361981,"num_objects":37,"num_object_clones":33,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"62'5","reported_seq":"436","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:22.922115+0000","last_change":"2021-02-10T04:54:39.615784+0000","last_active":"2021-02-10T04:59:22.922115+0000","last_peered":"2021-02-10T04:59:22.922115+0000","last_clean":"2021-02-10T04:59:22.922115+0000","last_became_active":"2021-02-10T04:54:39.560009+0000","last_became_peered":"2021-02-10T04:54:39.560009+0000","last_unstale":"2021-02-10T04:59:22.922115+0000","last_undegraded":"2021-02-10T04:59:22.922115+0000","last_fullsized":"2021-02-10T04:59:22.922115+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":434,"num_read_kb":433,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"1261'28","reported_seq":"1350","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.961534+0000","last_change":"2021-02-10T04:59:13.926028+0000","last_active":"2021-02-10T04:59:17.961534+0000","last_peered":"2021-02-10T04:59:17.961534+0000","last_clean":"2021-02-10T04:59:17.961534+0000","last_became_active":"2021-02-10T04:59:13.925333+0000","last_became_peered":"2021-02-10T04:59:13.925333+0000","last_unstale":"2021-02-10T04:59:17.961534+0000","last_undegraded":"2021-02-10T04:59:17.961534+0000","last_fullsized":"2021-02-10T04:59:17.961534+0000","mapping_epoch":1332,"log_start":"1061'20","ondisk_log_start":"1061'20","created":55,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'32","last_scrub_stamp":"2021-02-10T04:51:51.382040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.382040+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":579,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":29,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1026,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"999","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.638354+0000","last_change":"2021-02-10T04:53:34.179015+0000","last_active":"2021-02-10T04:59:18.638354+0000","last_peered":"2021-02-10T04:59:18.638354+0000","last_clean":"2021-02-10T04:59:18.638354+0000","last_became_active":"2021-02-10T04:51:49.502576+0000","last_became_peered":"2021-02-10T04:51:49.502576+0000","last_unstale":"2021-02-10T04:59:18.638354+0000","last_undegraded":"2021-02-10T04:59:18.638354+0000","last_fullsized":"2021-02-10T04:59:18.638354+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:34.178949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.178949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"244","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.636057+0000","last_change":"2021-02-10T04:51:53.038893+0000","last_active":"2021-02-10T04:59:18.636057+0000","last_peered":"2021-02-10T04:59:18.636057+0000","last_clean":"2021-02-10T04:59:18.636057+0000","last_became_active":"2021-02-10T04:51:36.855507+0000","last_became_peered":"2021-02-10T04:51:36.855507+0000","last_unstale":"2021-02-10T04:59:18.636057+0000","last_undegraded":"2021-02-10T04:59:18.636057+0000","last_fullsized":"2021-02-10T04:59:18.636057+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:53.038834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:53.038834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"1262","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.960440+0000","last_change":"2021-02-10T04:59:08.662159+0000","last_active":"2021-02-10T04:59:17.960440+0000","last_peered":"2021-02-10T04:59:17.960440+0000","last_clean":"2021-02-10T04:59:17.960440+0000","last_became_active":"2021-02-10T04:58:56.865590+0000","last_became_peered":"2021-02-10T04:58:56.865590+0000","last_unstale":"2021-02-10T04:59:17.960440+0000","last_undegraded":"2021-02-10T04:59:17.960440+0000","last_fullsized":"2021-02-10T04:59:17.960440+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:58:57.081143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:58:57.081143+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"1168'82","reported_seq":"280","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.632444+0000","last_change":"2021-02-10T04:53:23.326363+0000","last_active":"2021-02-10T04:59:18.632444+0000","last_peered":"2021-02-10T04:59:18.632444+0000","last_clean":"2021-02-10T04:59:18.632444+0000","last_became_active":"2021-02-10T04:51:36.859215+0000","last_became_peered":"2021-02-10T04:51:36.859215+0000","last_unstale":"2021-02-10T04:59:18.632444+0000","last_undegraded":"2021-02-10T04:59:18.632444+0000","last_fullsized":"2021-02-10T04:59:18.632444+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1150'76","last_scrub_stamp":"2021-02-10T04:53:00.964499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:00.964499+0000","log_size":82,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3848803,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"1261'69","reported_seq":"1498","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.974188+0000","last_change":"2021-02-10T04:59:13.926442+0000","last_active":"2021-02-10T04:59:20.974188+0000","last_peered":"2021-02-10T04:59:20.974188+0000","last_clean":"2021-02-10T04:59:20.974188+0000","last_became_active":"2021-02-10T04:59:13.926047+0000","last_became_peered":"2021-02-10T04:59:13.926047+0000","last_unstale":"2021-02-10T04:59:20.974188+0000","last_undegraded":"2021-02-10T04:59:20.974188+0000","last_fullsized":"2021-02-10T04:59:20.974188+0000","mapping_epoch":55,"log_start":"1112'47","ondisk_log_start":"1112'47","created":55,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'61","last_scrub_stamp":"2021-02-10T04:53:36.836202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:36.836202+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":733,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":104,"num_read_kb":86,"num_write":61,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":367,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"1334'8","reported_seq":"374","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:20.522921+0000","last_change":"2021-02-10T04:58:56.880162+0000","last_active":"2021-02-10T04:59:20.522921+0000","last_peered":"2021-02-10T04:59:20.522921+0000","last_clean":"2021-02-10T04:59:20.522921+0000","last_became_active":"2021-02-10T04:58:56.879760+0000","last_became_peered":"2021-02-10T04:58:56.879760+0000","last_unstale":"2021-02-10T04:59:20.522921+0000","last_undegraded":"2021-02-10T04:59:20.522921+0000","last_fullsized":"2021-02-10T04:59:20.522921+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'4","last_scrub_stamp":"2021-02-10T04:52:01.121380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.121380+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"274","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.632519+0000","last_change":"2021-02-10T04:58:56.964109+0000","last_active":"2021-02-10T04:59:18.632519+0000","last_peered":"2021-02-10T04:59:18.632519+0000","last_clean":"2021-02-10T04:59:18.632519+0000","last_became_active":"2021-02-10T04:58:56.963768+0000","last_became_peered":"2021-02-10T04:58:56.963768+0000","last_unstale":"2021-02-10T04:59:18.632519+0000","last_undegraded":"2021-02-10T04:59:18.632519+0000","last_fullsized":"2021-02-10T04:59:18.632519+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:55:22.573621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:55:22.573621+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"643","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.963351+0000","last_change":"2021-02-10T04:53:46.733236+0000","last_active":"2021-02-10T04:59:17.963351+0000","last_peered":"2021-02-10T04:59:17.963351+0000","last_clean":"2021-02-10T04:59:17.963351+0000","last_became_active":"2021-02-10T04:53:46.732890+0000","last_became_peered":"2021-02-10T04:53:46.732890+0000","last_unstale":"2021-02-10T04:59:17.963351+0000","last_undegraded":"2021-02-10T04:59:17.963351+0000","last_fullsized":"2021-02-10T04:59:17.963351+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:35.170157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:35.170157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"1003'739","reported_seq":"3936","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.960532+0000","last_change":"2021-02-10T04:59:08.649874+0000","last_active":"2021-02-10T04:59:17.960532+0000","last_peered":"2021-02-10T04:59:17.960532+0000","last_clean":"2021-02-10T04:59:17.960532+0000","last_became_active":"2021-02-10T04:58:56.956888+0000","last_became_peered":"2021-02-10T04:58:56.956888+0000","last_unstale":"2021-02-10T04:59:17.960532+0000","last_undegraded":"2021-02-10T04:59:17.960532+0000","last_fullsized":"2021-02-10T04:59:17.960532+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:52:27.602006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:27.602006+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24059888,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":129,"num_read_kb":109544,"num_write":102,"num_write_kb":46173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"1325","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.973743+0000","last_change":"2021-02-10T04:54:39.559980+0000","last_active":"2021-02-10T04:59:20.973743+0000","last_peered":"2021-02-10T04:59:20.973743+0000","last_clean":"2021-02-10T04:59:20.973743+0000","last_became_active":"2021-02-10T04:54:39.559443+0000","last_became_peered":"2021-02-10T04:54:39.559443+0000","last_unstale":"2021-02-10T04:59:20.973743+0000","last_undegraded":"2021-02-10T04:59:20.973743+0000","last_fullsized":"2021-02-10T04:59:20.973743+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"1261'64","reported_seq":"1542","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.637001+0000","last_change":"2021-02-10T04:59:13.932815+0000","last_active":"2021-02-10T04:59:18.637001+0000","last_peered":"2021-02-10T04:59:18.637001+0000","last_clean":"2021-02-10T04:59:18.637001+0000","last_became_active":"2021-02-10T04:59:13.931863+0000","last_became_peered":"2021-02-10T04:59:13.931863+0000","last_unstale":"2021-02-10T04:59:18.637001+0000","last_undegraded":"2021-02-10T04:59:18.637001+0000","last_fullsized":"2021-02-10T04:59:18.637001+0000","mapping_epoch":1118,"log_start":"1066'44","ondisk_log_start":"1066'44","created":55,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":0,"last_scrub":"1066'44","last_scrub_stamp":"2021-02-10T04:53:37.108677+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:53:37.108677+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":789,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":90,"num_read_kb":72,"num_write":53,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"1242","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.636008+0000","last_change":"2021-02-10T04:53:39.126356+0000","last_active":"2021-02-10T04:59:18.636008+0000","last_peered":"2021-02-10T04:59:18.636008+0000","last_clean":"2021-02-10T04:59:18.636008+0000","last_became_active":"2021-02-10T04:29:07.145616+0000","last_became_peered":"2021-02-10T04:29:07.145616+0000","last_unstale":"2021-02-10T04:59:18.636008+0000","last_undegraded":"2021-02-10T04:59:18.636008+0000","last_fullsized":"2021-02-10T04:59:18.636008+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:39.126261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:39.126261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"589","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.637672+0000","last_change":"2021-02-10T04:58:56.868713+0000","last_active":"2021-02-10T04:59:18.637672+0000","last_peered":"2021-02-10T04:59:18.637672+0000","last_clean":"2021-02-10T04:59:18.637672+0000","last_became_active":"2021-02-10T04:58:56.868368+0000","last_became_peered":"2021-02-10T04:58:56.868368+0000","last_unstale":"2021-02-10T04:59:18.637672+0000","last_undegraded":"2021-02-10T04:59:18.637672+0000","last_fullsized":"2021-02-10T04:59:18.637672+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:26.311958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.311958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"290","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.633794+0000","last_change":"2021-02-10T04:58:39.423097+0000","last_active":"2021-02-10T04:59:18.633794+0000","last_peered":"2021-02-10T04:59:18.633794+0000","last_clean":"2021-02-10T04:59:18.633794+0000","last_became_active":"2021-02-10T04:51:36.855744+0000","last_became_peered":"2021-02-10T04:51:36.855744+0000","last_unstale":"2021-02-10T04:59:18.633794+0000","last_undegraded":"2021-02-10T04:59:18.633794+0000","last_fullsized":"2021-02-10T04:59:18.633794+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:02.162851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:13.727241+0000","last_clean_scrub_stamp":"2021-02-10T04:52:02.162851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"1063'654","reported_seq":"4538","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.633294+0000","last_change":"2021-02-10T04:58:56.966357+0000","last_active":"2021-02-10T04:59:18.633294+0000","last_peered":"2021-02-10T04:59:18.633294+0000","last_clean":"2021-02-10T04:59:18.633294+0000","last_became_active":"2021-02-10T04:58:56.965471+0000","last_became_peered":"2021-02-10T04:58:56.965471+0000","last_unstale":"2021-02-10T04:59:18.633294+0000","last_undegraded":"2021-02-10T04:59:18.633294+0000","last_fullsized":"2021-02-10T04:59:18.633294+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:51:57.121838+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.121838+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24700630,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":171,"num_read_kb":182272,"num_write":111,"num_write_kb":33727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":26349023,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"1294","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.973053+0000","last_change":"2021-02-10T04:51:50.433056+0000","last_active":"2021-02-10T04:59:20.973053+0000","last_peered":"2021-02-10T04:59:20.973053+0000","last_clean":"2021-02-10T04:59:20.973053+0000","last_became_active":"2021-02-10T04:51:36.843165+0000","last_became_peered":"2021-02-10T04:51:36.843165+0000","last_unstale":"2021-02-10T04:59:20.973053+0000","last_undegraded":"2021-02-10T04:59:20.973053+0000","last_fullsized":"2021-02-10T04:59:20.973053+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.432990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.432990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"1329","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.635035+0000","last_change":"2021-02-10T04:58:56.965752+0000","last_active":"2021-02-10T04:59:18.635035+0000","last_peered":"2021-02-10T04:59:18.635035+0000","last_clean":"2021-02-10T04:59:18.635035+0000","last_became_active":"2021-02-10T04:58:56.963934+0000","last_became_peered":"2021-02-10T04:58:56.963934+0000","last_unstale":"2021-02-10T04:59:18.635035+0000","last_undegraded":"2021-02-10T04:59:18.635035+0000","last_fullsized":"2021-02-10T04:59:18.635035+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"1238","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.964761+0000","last_change":"2021-02-10T04:53:46.727888+0000","last_active":"2021-02-10T04:59:17.964761+0000","last_peered":"2021-02-10T04:59:17.964761+0000","last_clean":"2021-02-10T04:59:17.964761+0000","last_became_active":"2021-02-10T04:53:46.727670+0000","last_became_peered":"2021-02-10T04:53:46.727670+0000","last_unstale":"2021-02-10T04:59:17.964761+0000","last_undegraded":"2021-02-10T04:59:17.964761+0000","last_fullsized":"2021-02-10T04:59:17.964761+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:41.041553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.041553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"1261'76","reported_seq":"1466","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.635585+0000","last_change":"2021-02-10T04:59:13.929785+0000","last_active":"2021-02-10T04:59:18.635585+0000","last_peered":"2021-02-10T04:59:18.635585+0000","last_clean":"2021-02-10T04:59:18.635585+0000","last_became_active":"2021-02-10T04:59:13.929387+0000","last_became_peered":"2021-02-10T04:59:13.929387+0000","last_unstale":"2021-02-10T04:59:18.635585+0000","last_undegraded":"2021-02-10T04:59:18.635585+0000","last_fullsized":"2021-02-10T04:59:18.635585+0000","mapping_epoch":1111,"log_start":"1049'39","ondisk_log_start":"1049'39","created":55,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":0,"last_scrub":"1187'54","last_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_deep_scrub":"1187'54","last_deep_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_clean_scrub_stamp":"2021-02-10T04:53:42.071103+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":289,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":73,"num_read_kb":63,"num_write":47,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"984","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.638441+0000","last_change":"2021-02-10T04:51:52.350764+0000","last_active":"2021-02-10T04:59:18.638441+0000","last_peered":"2021-02-10T04:59:18.638441+0000","last_clean":"2021-02-10T04:59:18.638441+0000","last_became_active":"2021-02-10T04:51:49.567784+0000","last_became_peered":"2021-02-10T04:51:49.567784+0000","last_unstale":"2021-02-10T04:59:18.638441+0000","last_undegraded":"2021-02-10T04:59:18.638441+0000","last_fullsized":"2021-02-10T04:59:18.638441+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:52.350706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.350706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"1119'4","reported_seq":"342","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.633868+0000","last_change":"2021-02-10T04:52:27.126670+0000","last_active":"2021-02-10T04:59:18.633868+0000","last_peered":"2021-02-10T04:59:18.633868+0000","last_clean":"2021-02-10T04:59:18.633868+0000","last_became_active":"2021-02-10T04:51:36.853647+0000","last_became_peered":"2021-02-10T04:51:36.853647+0000","last_unstale":"2021-02-10T04:59:18.633868+0000","last_undegraded":"2021-02-10T04:59:18.633868+0000","last_fullsized":"2021-02-10T04:59:18.633868+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'4","last_scrub_stamp":"2021-02-10T04:52:27.126599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:27.126599+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"1333","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.637087+0000","last_change":"2021-02-10T04:52:25.745660+0000","last_active":"2021-02-10T04:59:18.637087+0000","last_peered":"2021-02-10T04:59:18.637087+0000","last_clean":"2021-02-10T04:59:18.637087+0000","last_became_active":"2021-02-10T04:51:36.849503+0000","last_became_peered":"2021-02-10T04:51:36.849503+0000","last_unstale":"2021-02-10T04:59:18.637087+0000","last_undegraded":"2021-02-10T04:59:18.637087+0000","last_fullsized":"2021-02-10T04:59:18.637087+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:25.745577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.745577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"1004'1093","reported_seq":"6519","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.963155+0000","last_change":"2021-02-10T04:59:08.653144+0000","last_active":"2021-02-10T04:59:17.963155+0000","last_peered":"2021-02-10T04:59:17.963155+0000","last_clean":"2021-02-10T04:59:17.963155+0000","last_became_active":"2021-02-10T04:58:56.875893+0000","last_became_peered":"2021-02-10T04:58:56.875893+0000","last_unstale":"2021-02-10T04:59:17.963155+0000","last_undegraded":"2021-02-10T04:59:17.963155+0000","last_fullsized":"2021-02-10T04:59:17.963155+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39434696,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":279,"num_read_kb":311058,"num_write":185,"num_write_kb":66790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"62'5","reported_seq":"146","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.632979+0000","last_change":"2021-02-10T04:54:39.559473+0000","last_active":"2021-02-10T04:59:18.632979+0000","last_peered":"2021-02-10T04:59:18.632979+0000","last_clean":"2021-02-10T04:59:18.632979+0000","last_became_active":"2021-02-10T04:54:39.559048+0000","last_became_peered":"2021-02-10T04:54:39.559048+0000","last_unstale":"2021-02-10T04:59:18.632979+0000","last_undegraded":"2021-02-10T04:59:18.632979+0000","last_fullsized":"2021-02-10T04:59:18.632979+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":149,"num_read_kb":148,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"1319","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.975953+0000","last_change":"2021-02-10T04:58:56.245075+0000","last_active":"2021-02-10T04:59:20.975953+0000","last_peered":"2021-02-10T04:59:20.975953+0000","last_clean":"2021-02-10T04:59:20.975953+0000","last_became_active":"2021-02-10T04:51:36.841582+0000","last_became_peered":"2021-02-10T04:51:36.841582+0000","last_unstale":"2021-02-10T04:59:20.975953+0000","last_undegraded":"2021-02-10T04:59:20.975953+0000","last_fullsized":"2021-02-10T04:59:20.975953+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:12.389272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:12.389272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"1261'71","reported_seq":"1060","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.638893+0000","last_change":"2021-02-10T04:59:13.930947+0000","last_active":"2021-02-10T04:59:18.638893+0000","last_peered":"2021-02-10T04:59:18.638893+0000","last_clean":"2021-02-10T04:59:18.638893+0000","last_became_active":"2021-02-10T04:59:13.930554+0000","last_became_peered":"2021-02-10T04:59:13.930554+0000","last_unstale":"2021-02-10T04:59:18.638893+0000","last_undegraded":"2021-02-10T04:59:18.638893+0000","last_fullsized":"2021-02-10T04:59:18.638893+0000","mapping_epoch":1129,"log_start":"1049'51","ondisk_log_start":"1049'51","created":55,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'41","last_scrub_stamp":"2021-02-10T04:51:58.353660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:58.353660+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":68,"num_write":51,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":525,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"1007","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.637284+0000","last_change":"2021-02-10T04:53:11.271557+0000","last_active":"2021-02-10T04:59:18.637284+0000","last_peered":"2021-02-10T04:59:18.637284+0000","last_clean":"2021-02-10T04:59:18.637284+0000","last_became_active":"2021-02-10T04:51:49.567883+0000","last_became_peered":"2021-02-10T04:51:49.567883+0000","last_unstale":"2021-02-10T04:59:18.637284+0000","last_undegraded":"2021-02-10T04:59:18.637284+0000","last_fullsized":"2021-02-10T04:59:18.637284+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:11.271459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.271459+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"1333'7","reported_seq":"1754","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.636989+0000","last_change":"2021-02-10T04:58:56.875943+0000","last_active":"2021-02-10T04:59:18.636989+0000","last_peered":"2021-02-10T04:59:18.636989+0000","last_clean":"2021-02-10T04:59:18.636989+0000","last_became_active":"2021-02-10T04:58:56.875512+0000","last_became_peered":"2021-02-10T04:58:56.875512+0000","last_unstale":"2021-02-10T04:59:18.636989+0000","last_undegraded":"2021-02-10T04:59:18.636989+0000","last_fullsized":"2021-02-10T04:59:18.636989+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:51:47.855283+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.855283+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"1353","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.977510+0000","last_change":"2021-02-10T04:58:56.854797+0000","last_active":"2021-02-10T04:59:20.977510+0000","last_peered":"2021-02-10T04:59:20.977510+0000","last_clean":"2021-02-10T04:59:20.977510+0000","last_became_active":"2021-02-10T04:58:56.854440+0000","last_became_peered":"2021-02-10T04:58:56.854440+0000","last_unstale":"2021-02-10T04:59:20.977510+0000","last_undegraded":"2021-02-10T04:59:20.977510+0000","last_fullsized":"2021-02-10T04:59:20.977510+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:55:22.183566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:55:22.183566+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"1002'1090","reported_seq":"5542","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.974506+0000","last_change":"2021-02-10T04:52:26.030198+0000","last_active":"2021-02-10T04:59:20.974506+0000","last_peered":"2021-02-10T04:59:20.974506+0000","last_clean":"2021-02-10T04:59:20.974506+0000","last_became_active":"2021-02-10T04:52:26.028182+0000","last_became_peered":"2021-02-10T04:52:26.028182+0000","last_unstale":"2021-02-10T04:59:20.974506+0000","last_undegraded":"2021-02-10T04:59:20.974506+0000","last_fullsized":"2021-02-10T04:59:20.974506+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:51:08.220472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.220472+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24410646,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":245,"num_read_kb":276168,"num_write":175,"num_write_kb":64960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"1313","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.633550+0000","last_change":"2021-02-10T04:54:39.555379+0000","last_active":"2021-02-10T04:59:18.633550+0000","last_peered":"2021-02-10T04:59:18.633550+0000","last_clean":"2021-02-10T04:59:18.633550+0000","last_became_active":"2021-02-10T04:54:39.554976+0000","last_became_peered":"2021-02-10T04:54:39.554976+0000","last_unstale":"2021-02-10T04:59:18.633550+0000","last_undegraded":"2021-02-10T04:59:18.633550+0000","last_fullsized":"2021-02-10T04:59:18.633550+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"1261'65","reported_seq":"1455","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.636752+0000","last_change":"2021-02-10T04:59:13.932372+0000","last_active":"2021-02-10T04:59:18.636752+0000","last_peered":"2021-02-10T04:59:18.636752+0000","last_clean":"2021-02-10T04:59:18.636752+0000","last_became_active":"2021-02-10T04:59:13.931978+0000","last_became_peered":"2021-02-10T04:59:13.931978+0000","last_unstale":"2021-02-10T04:59:18.636752+0000","last_undegraded":"2021-02-10T04:59:18.636752+0000","last_fullsized":"2021-02-10T04:59:18.636752+0000","mapping_epoch":55,"log_start":"1049'47","ondisk_log_start":"1049'47","created":55,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'47","last_scrub_stamp":"2021-02-10T04:53:09.002368+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.002368+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":836,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":84,"num_read_kb":67,"num_write":49,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":467,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"298","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.963436+0000","last_change":"2021-02-10T04:58:56.842228+0000","last_active":"2021-02-10T04:59:17.963436+0000","last_peered":"2021-02-10T04:59:17.963436+0000","last_clean":"2021-02-10T04:59:17.963436+0000","last_became_active":"2021-02-10T04:58:56.841107+0000","last_became_peered":"2021-02-10T04:58:56.841107+0000","last_unstale":"2021-02-10T04:59:17.963436+0000","last_undegraded":"2021-02-10T04:59:17.963436+0000","last_fullsized":"2021-02-10T04:59:17.963436+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:08.358300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:50:08.358300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"1325","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.977112+0000","last_change":"2021-02-10T04:51:57.436961+0000","last_active":"2021-02-10T04:59:20.977112+0000","last_peered":"2021-02-10T04:59:20.977112+0000","last_clean":"2021-02-10T04:59:20.977112+0000","last_became_active":"2021-02-10T04:25:51.031375+0000","last_became_peered":"2021-02-10T04:25:51.031375+0000","last_unstale":"2021-02-10T04:59:20.977112+0000","last_undegraded":"2021-02-10T04:59:20.977112+0000","last_fullsized":"2021-02-10T04:59:20.977112+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.436862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.436862+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"1279","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.634972+0000","last_change":"2021-02-10T04:53:02.112612+0000","last_active":"2021-02-10T04:59:18.634972+0000","last_peered":"2021-02-10T04:59:18.634972+0000","last_clean":"2021-02-10T04:59:18.634972+0000","last_became_active":"2021-02-10T04:50:59.545501+0000","last_became_peered":"2021-02-10T04:50:59.545501+0000","last_unstale":"2021-02-10T04:59:18.634972+0000","last_undegraded":"2021-02-10T04:59:18.634972+0000","last_fullsized":"2021-02-10T04:59:18.634972+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:02.112533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:02.112533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"1001'1165","reported_seq":"4536","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.974779+0000","last_change":"2021-02-10T04:58:56.967168+0000","last_active":"2021-02-10T04:59:20.974779+0000","last_peered":"2021-02-10T04:59:20.974779+0000","last_clean":"2021-02-10T04:59:20.974779+0000","last_became_active":"2021-02-10T04:58:56.965483+0000","last_became_peered":"2021-02-10T04:58:56.965483+0000","last_unstale":"2021-02-10T04:59:20.974779+0000","last_undegraded":"2021-02-10T04:59:20.974779+0000","last_fullsized":"2021-02-10T04:59:20.974779+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29601845,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":286,"num_read_kb":299529,"num_write":200,"num_write_kb":89130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"121","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.962375+0000","last_change":"2021-02-10T04:59:08.658203+0000","last_active":"2021-02-10T04:59:17.962375+0000","last_peered":"2021-02-10T04:59:17.962375+0000","last_clean":"2021-02-10T04:59:17.962375+0000","last_became_active":"2021-02-10T04:58:56.956556+0000","last_became_peered":"2021-02-10T04:58:56.956556+0000","last_unstale":"2021-02-10T04:59:17.962375+0000","last_undegraded":"2021-02-10T04:59:17.962375+0000","last_fullsized":"2021-02-10T04:59:17.962375+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"1113'5","reported_seq":"1624","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.633245+0000","last_change":"2021-02-10T04:52:34.047474+0000","last_active":"2021-02-10T04:59:18.633245+0000","last_peered":"2021-02-10T04:59:18.633245+0000","last_clean":"2021-02-10T04:59:18.633245+0000","last_became_active":"2021-02-10T04:50:59.569578+0000","last_became_peered":"2021-02-10T04:50:59.569578+0000","last_unstale":"2021-02-10T04:59:18.633245+0000","last_undegraded":"2021-02-10T04:59:18.633245+0000","last_fullsized":"2021-02-10T04:59:18.633245+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1113'5","last_scrub_stamp":"2021-02-10T04:52:34.047358+0000","last_deep_scrub":"158'3","last_deep_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_clean_scrub_stamp":"2021-02-10T04:52:34.047358+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"1261'68","reported_seq":"1430","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.634145+0000","last_change":"2021-02-10T04:51:48.872937+0000","last_active":"2021-02-10T04:59:18.634145+0000","last_peered":"2021-02-10T04:59:18.634145+0000","last_clean":"2021-02-10T04:59:18.634145+0000","last_became_active":"2021-02-10T04:25:49.002523+0000","last_became_peered":"2021-02-10T04:25:49.002523+0000","last_unstale":"2021-02-10T04:59:18.634145+0000","last_undegraded":"2021-02-10T04:59:18.634145+0000","last_fullsized":"2021-02-10T04:59:18.634145+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:51:48.872839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.872839+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":114,"num_read_kb":94,"num_write":68,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"1399","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.975886+0000","last_change":"2021-02-10T04:53:05.984747+0000","last_active":"2021-02-10T04:59:20.975886+0000","last_peered":"2021-02-10T04:59:20.975886+0000","last_clean":"2021-02-10T04:59:20.975886+0000","last_became_active":"2021-02-10T04:51:36.869931+0000","last_became_peered":"2021-02-10T04:51:36.869931+0000","last_unstale":"2021-02-10T04:59:20.975886+0000","last_undegraded":"2021-02-10T04:59:20.975886+0000","last_fullsized":"2021-02-10T04:59:20.975886+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:05.984659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:05.984659+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"1200","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.964347+0000","last_change":"2021-02-10T04:53:46.731709+0000","last_active":"2021-02-10T04:59:17.964347+0000","last_peered":"2021-02-10T04:59:17.964347+0000","last_clean":"2021-02-10T04:59:17.964347+0000","last_became_active":"2021-02-10T04:53:46.731340+0000","last_became_peered":"2021-02-10T04:53:46.731340+0000","last_unstale":"2021-02-10T04:59:17.964347+0000","last_undegraded":"2021-02-10T04:59:17.964347+0000","last_fullsized":"2021-02-10T04:59:17.964347+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:09.005421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.005421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"998'1009","reported_seq":"4838","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.633465+0000","last_change":"2021-02-10T04:52:26.037227+0000","last_active":"2021-02-10T04:59:18.633465+0000","last_peered":"2021-02-10T04:59:18.633465+0000","last_clean":"2021-02-10T04:59:18.633465+0000","last_became_active":"2021-02-10T04:52:26.036723+0000","last_became_peered":"2021-02-10T04:52:26.036723+0000","last_unstale":"2021-02-10T04:59:18.633465+0000","last_undegraded":"2021-02-10T04:59:18.633465+0000","last_fullsized":"2021-02-10T04:59:18.633465+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12355206,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":119,"num_read_kb":124701,"num_write":77,"num_write_kb":25052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4145029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"1218","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.639116+0000","last_change":"2021-02-10T04:53:33.305896+0000","last_active":"2021-02-10T04:59:18.639116+0000","last_peered":"2021-02-10T04:59:18.639116+0000","last_clean":"2021-02-10T04:59:18.639116+0000","last_became_active":"2021-02-10T04:50:59.570854+0000","last_became_peered":"2021-02-10T04:50:59.570854+0000","last_unstale":"2021-02-10T04:59:18.639116+0000","last_undegraded":"2021-02-10T04:59:18.639116+0000","last_fullsized":"2021-02-10T04:59:18.639116+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:33.305827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:33.305827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"1334","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.635156+0000","last_change":"2021-02-10T04:58:56.979110+0000","last_active":"2021-02-10T04:59:18.635156+0000","last_peered":"2021-02-10T04:59:18.635156+0000","last_clean":"2021-02-10T04:59:18.635156+0000","last_became_active":"2021-02-10T04:58:56.978745+0000","last_became_peered":"2021-02-10T04:58:56.978745+0000","last_unstale":"2021-02-10T04:59:18.635156+0000","last_undegraded":"2021-02-10T04:59:18.635156+0000","last_fullsized":"2021-02-10T04:59:18.635156+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"1332","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.634305+0000","last_change":"2021-02-10T04:51:50.895471+0000","last_active":"2021-02-10T04:59:18.634305+0000","last_peered":"2021-02-10T04:59:18.634305+0000","last_clean":"2021-02-10T04:59:18.634305+0000","last_became_active":"2021-02-10T04:50:59.527553+0000","last_became_peered":"2021-02-10T04:50:59.527553+0000","last_unstale":"2021-02-10T04:59:18.634305+0000","last_undegraded":"2021-02-10T04:59:18.634305+0000","last_fullsized":"2021-02-10T04:59:18.634305+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.895370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.895370+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"1261'84","reported_seq":"1478","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.633611+0000","last_change":"2021-02-10T04:58:56.880156+0000","last_active":"2021-02-10T04:59:18.633611+0000","last_peered":"2021-02-10T04:59:18.633611+0000","last_clean":"2021-02-10T04:59:18.633611+0000","last_became_active":"2021-02-10T04:58:56.879837+0000","last_became_peered":"2021-02-10T04:58:56.879837+0000","last_unstale":"2021-02-10T04:59:18.633611+0000","last_undegraded":"2021-02-10T04:59:18.633611+0000","last_fullsized":"2021-02-10T04:59:18.633611+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'72","last_scrub_stamp":"2021-02-10T04:46:30.515026+0000","last_deep_scrub":"152'48","last_deep_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_clean_scrub_stamp":"2021-02-10T04:46:30.515026+0000","log_size":84,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":123,"num_read_kb":101,"num_write":72,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"613","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.638217+0000","last_change":"2021-02-10T04:58:38.400978+0000","last_active":"2021-02-10T04:59:18.638217+0000","last_peered":"2021-02-10T04:59:18.638217+0000","last_clean":"2021-02-10T04:59:18.638217+0000","last_became_active":"2021-02-10T04:51:49.503801+0000","last_became_peered":"2021-02-10T04:51:49.503801+0000","last_unstale":"2021-02-10T04:59:18.638217+0000","last_undegraded":"2021-02-10T04:59:18.638217+0000","last_fullsized":"2021-02-10T04:59:18.638217+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:51:48.274960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.274960+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"1357","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.634632+0000","last_change":"2021-02-10T04:53:13.099288+0000","last_active":"2021-02-10T04:59:18.634632+0000","last_peered":"2021-02-10T04:59:18.634632+0000","last_clean":"2021-02-10T04:59:18.634632+0000","last_became_active":"2021-02-10T04:51:36.848614+0000","last_became_peered":"2021-02-10T04:51:36.848614+0000","last_unstale":"2021-02-10T04:59:18.634632+0000","last_undegraded":"2021-02-10T04:59:18.634632+0000","last_fullsized":"2021-02-10T04:59:18.634632+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:13.099228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:13.099228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"1002'1657","reported_seq":"5408","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.634741+0000","last_change":"2021-02-10T04:52:26.036806+0000","last_active":"2021-02-10T04:59:18.634741+0000","last_peered":"2021-02-10T04:59:18.634741+0000","last_clean":"2021-02-10T04:59:18.634741+0000","last_became_active":"2021-02-10T04:52:26.032363+0000","last_became_peered":"2021-02-10T04:52:26.032363+0000","last_unstale":"2021-02-10T04:59:18.634741+0000","last_undegraded":"2021-02-10T04:59:18.634741+0000","last_fullsized":"2021-02-10T04:59:18.634741+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21396533,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":212,"num_read_kb":218836,"num_write":159,"num_write_kb":59398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":22326014,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"1276","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.636513+0000","last_change":"2021-02-10T04:51:51.911945+0000","last_active":"2021-02-10T04:59:18.636513+0000","last_peered":"2021-02-10T04:59:18.636513+0000","last_clean":"2021-02-10T04:59:18.636513+0000","last_became_active":"2021-02-10T04:51:36.850015+0000","last_became_peered":"2021-02-10T04:51:36.850015+0000","last_unstale":"2021-02-10T04:59:18.636513+0000","last_undegraded":"2021-02-10T04:59:18.636513+0000","last_fullsized":"2021-02-10T04:59:18.636513+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.911851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.911851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"62'5","reported_seq":"1180","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.637379+0000","last_change":"2021-02-10T04:54:39.568253+0000","last_active":"2021-02-10T04:59:18.637379+0000","last_peered":"2021-02-10T04:59:18.637379+0000","last_clean":"2021-02-10T04:59:18.637379+0000","last_became_active":"2021-02-10T04:54:39.567642+0000","last_became_peered":"2021-02-10T04:54:39.567642+0000","last_unstale":"2021-02-10T04:59:18.637379+0000","last_undegraded":"2021-02-10T04:59:18.637379+0000","last_fullsized":"2021-02-10T04:59:18.637379+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":98,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":150,"num_read_kb":148,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"1421","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.974346+0000","last_change":"2021-02-10T04:58:56.957964+0000","last_active":"2021-02-10T04:59:20.974346+0000","last_peered":"2021-02-10T04:59:20.974346+0000","last_clean":"2021-02-10T04:59:20.974346+0000","last_became_active":"2021-02-10T04:58:56.957624+0000","last_became_peered":"2021-02-10T04:58:56.957624+0000","last_unstale":"2021-02-10T04:59:20.974346+0000","last_undegraded":"2021-02-10T04:59:20.974346+0000","last_fullsized":"2021-02-10T04:59:20.974346+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:58.142526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:58.142526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"1261'68","reported_seq":"1339","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.961537+0000","last_change":"2021-02-10T04:59:08.646165+0000","last_active":"2021-02-10T04:59:17.961537+0000","last_peered":"2021-02-10T04:59:17.961537+0000","last_clean":"2021-02-10T04:59:17.961537+0000","last_became_active":"2021-02-10T04:58:56.954019+0000","last_became_peered":"2021-02-10T04:58:56.954019+0000","last_unstale":"2021-02-10T04:59:17.961537+0000","last_undegraded":"2021-02-10T04:59:17.961537+0000","last_fullsized":"2021-02-10T04:59:17.961537+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1135'58","last_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_deep_scrub":"1135'58","last_deep_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.483067+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":101,"num_read_kb":82,"num_write":60,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"302","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.964553+0000","last_change":"2021-02-10T04:59:06.591009+0000","last_active":"2021-02-10T04:59:17.964553+0000","last_peered":"2021-02-10T04:59:17.964553+0000","last_clean":"2021-02-10T04:59:17.964553+0000","last_became_active":"2021-02-10T04:58:56.849868+0000","last_became_peered":"2021-02-10T04:58:56.849868+0000","last_unstale":"2021-02-10T04:59:17.964553+0000","last_undegraded":"2021-02-10T04:59:17.964553+0000","last_fullsized":"2021-02-10T04:59:17.964553+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:50:46.434629+0000","last_deep_scrub":"53'2","last_deep_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_clean_scrub_stamp":"2021-02-10T04:50:46.434629+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"1024","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.637431+0000","last_change":"2021-02-10T04:52:38.278980+0000","last_active":"2021-02-10T04:59:18.637431+0000","last_peered":"2021-02-10T04:59:18.637431+0000","last_clean":"2021-02-10T04:59:18.637431+0000","last_became_active":"2021-02-10T04:51:49.496237+0000","last_became_peered":"2021-02-10T04:51:49.496237+0000","last_unstale":"2021-02-10T04:59:18.637431+0000","last_undegraded":"2021-02-10T04:59:18.637431+0000","last_fullsized":"2021-02-10T04:59:18.637431+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:38.278910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:52:38.278910+0000","last_clean_scrub_stamp":"2021-02-10T04:52:38.278910+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"1000'1071","reported_seq":"4899","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.973395+0000","last_change":"2021-02-10T04:53:34.407875+0000","last_active":"2021-02-10T04:59:20.973395+0000","last_peered":"2021-02-10T04:59:20.973395+0000","last_clean":"2021-02-10T04:59:20.973395+0000","last_became_active":"2021-02-10T04:52:26.031050+0000","last_became_peered":"2021-02-10T04:52:26.031050+0000","last_unstale":"2021-02-10T04:59:20.973395+0000","last_undegraded":"2021-02-10T04:59:20.973395+0000","last_fullsized":"2021-02-10T04:59:20.973395+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:34.407784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.407784+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30173843,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"1387","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.975702+0000","last_change":"2021-02-10T04:58:56.867738+0000","last_active":"2021-02-10T04:59:20.975702+0000","last_peered":"2021-02-10T04:59:20.975702+0000","last_clean":"2021-02-10T04:59:20.975702+0000","last_became_active":"2021-02-10T04:58:56.867381+0000","last_became_peered":"2021-02-10T04:58:56.867381+0000","last_unstale":"2021-02-10T04:59:20.975702+0000","last_undegraded":"2021-02-10T04:59:20.975702+0000","last_fullsized":"2021-02-10T04:59:20.975702+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_clean_scrub_stamp":"2021-02-10T04:51:00.195518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"1355","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.977138+0000","last_change":"2021-02-10T04:58:58.252055+0000","last_active":"2021-02-10T04:59:20.977138+0000","last_peered":"2021-02-10T04:59:20.977138+0000","last_clean":"2021-02-10T04:59:20.977138+0000","last_became_active":"2021-02-10T04:58:56.854414+0000","last_became_peered":"2021-02-10T04:58:56.854414+0000","last_unstale":"2021-02-10T04:59:20.977138+0000","last_undegraded":"2021-02-10T04:59:20.977138+0000","last_fullsized":"2021-02-10T04:59:20.977138+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:58:58.251969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:58:58.251969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"1261'58","reported_seq":"1400","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.962291+0000","last_change":"2021-02-10T04:59:13.927049+0000","last_active":"2021-02-10T04:59:17.962291+0000","last_peered":"2021-02-10T04:59:17.962291+0000","last_clean":"2021-02-10T04:59:17.962291+0000","last_became_active":"2021-02-10T04:59:13.926626+0000","last_became_peered":"2021-02-10T04:59:13.926626+0000","last_unstale":"2021-02-10T04:59:17.962291+0000","last_undegraded":"2021-02-10T04:59:17.962291+0000","last_fullsized":"2021-02-10T04:59:17.962291+0000","mapping_epoch":1332,"log_start":"1049'40","ondisk_log_start":"1049'40","created":55,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'40","last_scrub_stamp":"2021-02-10T04:51:47.281929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.281929+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":918,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":85,"num_read_kb":67,"num_write":49,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"522","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.963482+0000","last_change":"2021-02-10T04:58:56.841926+0000","last_active":"2021-02-10T04:59:17.963482+0000","last_peered":"2021-02-10T04:59:17.963482+0000","last_clean":"2021-02-10T04:59:17.963482+0000","last_became_active":"2021-02-10T04:58:56.840576+0000","last_became_peered":"2021-02-10T04:58:56.840576+0000","last_unstale":"2021-02-10T04:59:17.963482+0000","last_undegraded":"2021-02-10T04:59:17.963482+0000","last_fullsized":"2021-02-10T04:59:17.963482+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:50.399350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:50.399350+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"1225","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.960242+0000","last_change":"2021-02-10T04:59:08.647800+0000","last_active":"2021-02-10T04:59:17.960242+0000","last_peered":"2021-02-10T04:59:17.960242+0000","last_clean":"2021-02-10T04:59:17.960242+0000","last_became_active":"2021-02-10T04:58:56.875409+0000","last_became_peered":"2021-02-10T04:58:56.875409+0000","last_unstale":"2021-02-10T04:59:17.960242+0000","last_undegraded":"2021-02-10T04:59:17.960242+0000","last_fullsized":"2021-02-10T04:59:17.960242+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"1009","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.637780+0000","last_change":"2021-02-10T04:52:31.273037+0000","last_active":"2021-02-10T04:59:18.637780+0000","last_peered":"2021-02-10T04:59:18.637780+0000","last_clean":"2021-02-10T04:59:18.637780+0000","last_became_active":"2021-02-10T04:51:49.501405+0000","last_became_peered":"2021-02-10T04:51:49.501405+0000","last_unstale":"2021-02-10T04:59:18.637780+0000","last_undegraded":"2021-02-10T04:59:18.637780+0000","last_fullsized":"2021-02-10T04:59:18.637780+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:31.272971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:17.598113+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.272971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"1000'446","reported_seq":"2079","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.964258+0000","last_change":"2021-02-10T04:58:56.846600+0000","last_active":"2021-02-10T04:59:17.964258+0000","last_peered":"2021-02-10T04:59:17.964258+0000","last_clean":"2021-02-10T04:59:17.964258+0000","last_became_active":"2021-02-10T04:58:56.846008+0000","last_became_peered":"2021-02-10T04:58:56.846008+0000","last_unstale":"2021-02-10T04:59:17.964258+0000","last_undegraded":"2021-02-10T04:59:17.964258+0000","last_fullsized":"2021-02-10T04:59:17.964258+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:51:45.964029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.964029+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22255852,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68435,"num_write":72,"num_write_kb":24345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":403693,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"500","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.637584+0000","last_change":"2021-02-10T04:53:03.228260+0000","last_active":"2021-02-10T04:59:18.637584+0000","last_peered":"2021-02-10T04:59:18.637584+0000","last_clean":"2021-02-10T04:59:18.637584+0000","last_became_active":"2021-02-10T04:51:49.499252+0000","last_became_peered":"2021-02-10T04:51:49.499252+0000","last_unstale":"2021-02-10T04:59:18.637584+0000","last_undegraded":"2021-02-10T04:59:18.637584+0000","last_fullsized":"2021-02-10T04:59:18.637584+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:03.228182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.228182+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"251","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.964662+0000","last_change":"2021-02-10T04:53:46.727384+0000","last_active":"2021-02-10T04:59:17.964662+0000","last_peered":"2021-02-10T04:59:17.964662+0000","last_clean":"2021-02-10T04:59:17.964662+0000","last_became_active":"2021-02-10T04:53:46.726771+0000","last_became_peered":"2021-02-10T04:53:46.726771+0000","last_unstale":"2021-02-10T04:59:17.964662+0000","last_undegraded":"2021-02-10T04:59:17.964662+0000","last_fullsized":"2021-02-10T04:59:17.964662+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:33.948348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:33.948348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"1261'51","reported_seq":"1434","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.974666+0000","last_change":"2021-02-10T04:54:17.118810+0000","last_active":"2021-02-10T04:59:20.974666+0000","last_peered":"2021-02-10T04:59:20.974666+0000","last_clean":"2021-02-10T04:59:20.974666+0000","last_became_active":"2021-02-10T04:54:17.080053+0000","last_became_peered":"2021-02-10T04:54:17.080053+0000","last_unstale":"2021-02-10T04:59:20.974666+0000","last_undegraded":"2021-02-10T04:59:20.974666+0000","last_fullsized":"2021-02-10T04:59:20.974666+0000","mapping_epoch":1200,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1201,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'39","last_scrub_stamp":"2021-02-10T04:51:47.322331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.322331+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":51,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"294","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.963538+0000","last_change":"2021-02-10T04:58:56.842526+0000","last_active":"2021-02-10T04:59:17.963538+0000","last_peered":"2021-02-10T04:59:17.963538+0000","last_clean":"2021-02-10T04:59:17.963538+0000","last_became_active":"2021-02-10T04:58:56.841692+0000","last_became_peered":"2021-02-10T04:58:56.841692+0000","last_unstale":"2021-02-10T04:59:17.963538+0000","last_undegraded":"2021-02-10T04:59:17.963538+0000","last_fullsized":"2021-02-10T04:59:17.963538+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_clean_scrub_stamp":"2021-02-10T04:50:05.424262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"1519","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.976243+0000","last_change":"2021-02-10T04:52:37.324122+0000","last_active":"2021-02-10T04:59:20.976243+0000","last_peered":"2021-02-10T04:59:20.976243+0000","last_clean":"2021-02-10T04:59:20.976243+0000","last_became_active":"2021-02-10T04:51:36.843553+0000","last_became_peered":"2021-02-10T04:51:36.843553+0000","last_unstale":"2021-02-10T04:59:20.976243+0000","last_undegraded":"2021-02-10T04:59:20.976243+0000","last_fullsized":"2021-02-10T04:59:20.976243+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:37.324040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:37.324040+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"1324","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.634210+0000","last_change":"2021-02-10T04:54:37.612303+0000","last_active":"2021-02-10T04:59:18.634210+0000","last_peered":"2021-02-10T04:59:18.634210+0000","last_clean":"2021-02-10T04:59:18.634210+0000","last_became_active":"2021-02-10T04:54:37.611953+0000","last_became_peered":"2021-02-10T04:54:37.611953+0000","last_unstale":"2021-02-10T04:59:18.634210+0000","last_undegraded":"2021-02-10T04:59:18.634210+0000","last_fullsized":"2021-02-10T04:59:18.634210+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"1347","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.977178+0000","last_change":"2021-02-10T04:52:31.403904+0000","last_active":"2021-02-10T04:59:20.977178+0000","last_peered":"2021-02-10T04:59:20.977178+0000","last_clean":"2021-02-10T04:59:20.977178+0000","last_became_active":"2021-02-10T04:50:59.538009+0000","last_became_peered":"2021-02-10T04:50:59.538009+0000","last_unstale":"2021-02-10T04:59:20.977178+0000","last_undegraded":"2021-02-10T04:59:20.977178+0000","last_fullsized":"2021-02-10T04:59:20.977178+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:31.403824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.403824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"1000'490","reported_seq":"3805","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.962769+0000","last_change":"2021-02-10T04:59:08.651746+0000","last_active":"2021-02-10T04:59:17.962769+0000","last_peered":"2021-02-10T04:59:17.962769+0000","last_clean":"2021-02-10T04:59:17.962769+0000","last_became_active":"2021-02-10T04:58:56.962979+0000","last_became_peered":"2021-02-10T04:58:56.962979+0000","last_unstale":"2021-02-10T04:59:17.962769+0000","last_undegraded":"2021-02-10T04:59:17.962769+0000","last_fullsized":"2021-02-10T04:59:17.962769+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:47:02.875920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:02.875920+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19371627,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":141,"num_read_kb":148568,"num_write":85,"num_write_kb":31732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"244","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.633506+0000","last_change":"2021-02-10T04:53:01.972991+0000","last_active":"2021-02-10T04:59:18.633506+0000","last_peered":"2021-02-10T04:59:18.633506+0000","last_clean":"2021-02-10T04:59:18.633506+0000","last_became_active":"2021-02-10T04:51:36.863976+0000","last_became_peered":"2021-02-10T04:51:36.863976+0000","last_unstale":"2021-02-10T04:59:18.633506+0000","last_undegraded":"2021-02-10T04:59:18.633506+0000","last_fullsized":"2021-02-10T04:59:18.633506+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:01.972913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.972913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"1261'40","reported_seq":"1091","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.638841+0000","last_change":"2021-02-10T04:59:13.926580+0000","last_active":"2021-02-10T04:59:18.638841+0000","last_peered":"2021-02-10T04:59:18.638841+0000","last_clean":"2021-02-10T04:59:18.638841+0000","last_became_active":"2021-02-10T04:59:13.926146+0000","last_became_peered":"2021-02-10T04:59:13.926146+0000","last_unstale":"2021-02-10T04:59:18.638841+0000","last_undegraded":"2021-02-10T04:59:18.638841+0000","last_fullsized":"2021-02-10T04:59:18.638841+0000","mapping_epoch":1332,"log_start":"1063'30","ondisk_log_start":"1063'30","created":55,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:56.431187+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.431187+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":401,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":58,"num_read_kb":47,"num_write":33,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"269","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.634752+0000","last_change":"2021-02-10T04:58:56.879863+0000","last_active":"2021-02-10T04:59:18.634752+0000","last_peered":"2021-02-10T04:59:18.634752+0000","last_clean":"2021-02-10T04:59:18.634752+0000","last_became_active":"2021-02-10T04:58:56.879496+0000","last_became_peered":"2021-02-10T04:58:56.879496+0000","last_unstale":"2021-02-10T04:59:18.634752+0000","last_undegraded":"2021-02-10T04:59:18.634752+0000","last_fullsized":"2021-02-10T04:59:18.634752+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:04.591695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:04.591695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"1173","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.637499+0000","last_change":"2021-02-10T04:54:37.680913+0000","last_active":"2021-02-10T04:59:18.637499+0000","last_peered":"2021-02-10T04:59:18.637499+0000","last_clean":"2021-02-10T04:59:18.637499+0000","last_became_active":"2021-02-10T04:54:37.680651+0000","last_became_peered":"2021-02-10T04:54:37.680651+0000","last_unstale":"2021-02-10T04:59:18.637499+0000","last_undegraded":"2021-02-10T04:59:18.637499+0000","last_fullsized":"2021-02-10T04:59:18.637499+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1210,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":149,"num_read_kb":148,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"1190","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.964454+0000","last_change":"2021-02-10T04:53:46.729138+0000","last_active":"2021-02-10T04:59:17.964454+0000","last_peered":"2021-02-10T04:59:17.964454+0000","last_clean":"2021-02-10T04:59:17.964454+0000","last_became_active":"2021-02-10T04:53:46.728845+0000","last_became_peered":"2021-02-10T04:53:46.728845+0000","last_unstale":"2021-02-10T04:59:17.964454+0000","last_undegraded":"2021-02-10T04:59:17.964454+0000","last_fullsized":"2021-02-10T04:59:17.964454+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:40.066354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:52:40.066354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"1000'620","reported_seq":"4429","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.962315+0000","last_change":"2021-02-10T04:59:08.651277+0000","last_active":"2021-02-10T04:59:17.962315+0000","last_peered":"2021-02-10T04:59:17.962315+0000","last_clean":"2021-02-10T04:59:17.962315+0000","last_became_active":"2021-02-10T04:58:56.878577+0000","last_became_peered":"2021-02-10T04:58:56.878577+0000","last_unstale":"2021-02-10T04:59:17.962315+0000","last_undegraded":"2021-02-10T04:59:17.962315+0000","last_fullsized":"2021-02-10T04:59:17.962315+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:51:54.444287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:54.444287+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14265195,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":144,"num_read_kb":128482,"num_write":129,"num_write_kb":53443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"1218","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.960069+0000","last_change":"2021-02-10T04:59:08.653244+0000","last_active":"2021-02-10T04:59:17.960069+0000","last_peered":"2021-02-10T04:59:17.960069+0000","last_clean":"2021-02-10T04:59:17.960069+0000","last_became_active":"2021-02-10T04:58:56.842941+0000","last_became_peered":"2021-02-10T04:58:56.842941+0000","last_unstale":"2021-02-10T04:59:17.960069+0000","last_undegraded":"2021-02-10T04:59:17.960069+0000","last_fullsized":"2021-02-10T04:59:17.960069+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:14.955544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:47:14.955544+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"1339","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.635014+0000","last_change":"2021-02-10T04:58:56.964034+0000","last_active":"2021-02-10T04:59:18.635014+0000","last_peered":"2021-02-10T04:59:18.635014+0000","last_clean":"2021-02-10T04:59:18.635014+0000","last_became_active":"2021-02-10T04:58:56.963677+0000","last_became_peered":"2021-02-10T04:58:56.963677+0000","last_unstale":"2021-02-10T04:59:18.635014+0000","last_undegraded":"2021-02-10T04:59:18.635014+0000","last_fullsized":"2021-02-10T04:59:18.635014+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"1321","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.962353+0000","last_change":"2021-02-10T04:59:08.656087+0000","last_active":"2021-02-10T04:59:17.962353+0000","last_peered":"2021-02-10T04:59:17.962353+0000","last_clean":"2021-02-10T04:59:17.962353+0000","last_became_active":"2021-02-10T04:58:56.957166+0000","last_became_peered":"2021-02-10T04:58:56.957166+0000","last_unstale":"2021-02-10T04:59:17.962353+0000","last_undegraded":"2021-02-10T04:59:17.962353+0000","last_fullsized":"2021-02-10T04:59:17.962353+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:27.908221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:47:27.908221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"1261'67","reported_seq":"998","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.637531+0000","last_change":"2021-02-10T04:52:30.277009+0000","last_active":"2021-02-10T04:59:18.637531+0000","last_peered":"2021-02-10T04:59:18.637531+0000","last_clean":"2021-02-10T04:59:18.637531+0000","last_became_active":"2021-02-10T04:51:49.501622+0000","last_became_peered":"2021-02-10T04:51:49.501622+0000","last_unstale":"2021-02-10T04:59:18.637531+0000","last_undegraded":"2021-02-10T04:59:18.637531+0000","last_fullsized":"2021-02-10T04:59:18.637531+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'55","last_scrub_stamp":"2021-02-10T04:52:30.276938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.276938+0000","log_size":67,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":116,"num_read_kb":92,"num_write":70,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"1343","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.976248+0000","last_change":"2021-02-10T04:51:59.420149+0000","last_active":"2021-02-10T04:59:20.976248+0000","last_peered":"2021-02-10T04:59:20.976248+0000","last_clean":"2021-02-10T04:59:20.976248+0000","last_became_active":"2021-02-10T04:28:21.125541+0000","last_became_peered":"2021-02-10T04:28:21.125541+0000","last_unstale":"2021-02-10T04:59:20.976248+0000","last_undegraded":"2021-02-10T04:59:20.976248+0000","last_fullsized":"2021-02-10T04:59:20.976248+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.420086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.420086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"1308","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.634066+0000","last_change":"2021-02-10T04:53:12.187225+0000","last_active":"2021-02-10T04:59:18.634066+0000","last_peered":"2021-02-10T04:59:18.634066+0000","last_clean":"2021-02-10T04:59:18.634066+0000","last_became_active":"2021-02-10T04:50:59.555689+0000","last_became_peered":"2021-02-10T04:50:59.555689+0000","last_unstale":"2021-02-10T04:59:18.634066+0000","last_undegraded":"2021-02-10T04:59:18.634066+0000","last_fullsized":"2021-02-10T04:59:18.634066+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.187154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.187154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"1002'2042","reported_seq":"9595","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.975810+0000","last_change":"2021-02-10T04:58:56.982754+0000","last_active":"2021-02-10T04:59:20.975810+0000","last_peered":"2021-02-10T04:59:20.975810+0000","last_clean":"2021-02-10T04:59:20.975810+0000","last_became_active":"2021-02-10T04:58:56.980189+0000","last_became_peered":"2021-02-10T04:58:56.980189+0000","last_unstale":"2021-02-10T04:59:20.975810+0000","last_undegraded":"2021-02-10T04:59:20.975810+0000","last_fullsized":"2021-02-10T04:59:20.975810+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:46:57.054230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:57.054230+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":85678898,"num_objects":35,"num_object_clones":29,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"1309","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.975838+0000","last_change":"2021-02-10T04:58:56.856707+0000","last_active":"2021-02-10T04:59:20.975838+0000","last_peered":"2021-02-10T04:59:20.975838+0000","last_clean":"2021-02-10T04:59:20.975838+0000","last_became_active":"2021-02-10T04:58:56.856323+0000","last_became_peered":"2021-02-10T04:58:56.856323+0000","last_unstale":"2021-02-10T04:59:20.975838+0000","last_undegraded":"2021-02-10T04:59:20.975838+0000","last_fullsized":"2021-02-10T04:59:20.975838+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:52.341527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:50:52.341527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"147","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.977294+0000","last_change":"2021-02-10T04:54:39.558380+0000","last_active":"2021-02-10T04:59:20.977294+0000","last_peered":"2021-02-10T04:59:20.977294+0000","last_clean":"2021-02-10T04:59:20.977294+0000","last_became_active":"2021-02-10T04:54:39.557995+0000","last_became_peered":"2021-02-10T04:54:39.557995+0000","last_unstale":"2021-02-10T04:59:20.977294+0000","last_undegraded":"2021-02-10T04:59:20.977294+0000","last_fullsized":"2021-02-10T04:59:20.977294+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"1338'6","reported_seq":"1503","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:18.078092+0000","last_change":"2021-02-10T04:59:08.663963+0000","last_active":"2021-02-10T04:59:18.078092+0000","last_peered":"2021-02-10T04:59:18.078092+0000","last_clean":"2021-02-10T04:59:18.078092+0000","last_became_active":"2021-02-10T04:58:56.956377+0000","last_became_peered":"2021-02-10T04:58:56.956377+0000","last_unstale":"2021-02-10T04:59:18.078092+0000","last_undegraded":"2021-02-10T04:59:18.078092+0000","last_fullsized":"2021-02-10T04:59:18.078092+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:54.466259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:54.466259+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"1261'64","reported_seq":"1044","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.638690+0000","last_change":"2021-02-10T04:53:01.304693+0000","last_active":"2021-02-10T04:59:18.638690+0000","last_peered":"2021-02-10T04:59:18.638690+0000","last_clean":"2021-02-10T04:59:18.638690+0000","last_became_active":"2021-02-10T04:51:49.498066+0000","last_became_peered":"2021-02-10T04:51:49.498066+0000","last_unstale":"2021-02-10T04:59:18.638690+0000","last_undegraded":"2021-02-10T04:59:18.638690+0000","last_fullsized":"2021-02-10T04:59:18.638690+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:53:01.304626+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.304626+0000","log_size":64,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":89,"num_read_kb":71,"num_write":52,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"620","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.637585+0000","last_change":"2021-02-10T04:58:56.868826+0000","last_active":"2021-02-10T04:59:18.637585+0000","last_peered":"2021-02-10T04:59:18.637585+0000","last_clean":"2021-02-10T04:59:18.637585+0000","last_became_active":"2021-02-10T04:58:56.868475+0000","last_became_peered":"2021-02-10T04:58:56.868475+0000","last_unstale":"2021-02-10T04:59:18.637585+0000","last_undegraded":"2021-02-10T04:59:18.637585+0000","last_fullsized":"2021-02-10T04:59:18.637585+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:49.447961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:49.447961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"1321","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.958895+0000","last_change":"2021-02-10T04:59:08.658201+0000","last_active":"2021-02-10T04:59:17.958895+0000","last_peered":"2021-02-10T04:59:17.958895+0000","last_clean":"2021-02-10T04:59:17.958895+0000","last_became_active":"2021-02-10T04:58:56.874543+0000","last_became_peered":"2021-02-10T04:58:56.874543+0000","last_unstale":"2021-02-10T04:59:17.958895+0000","last_undegraded":"2021-02-10T04:59:17.958895+0000","last_fullsized":"2021-02-10T04:59:17.958895+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:07.914061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:07.914061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"293'77","reported_seq":"3129","reported_epoch":"1347","state":"active+clean+scrubbing","last_fresh":"2021-02-10T04:59:18.633925+0000","last_change":"2021-02-10T04:53:43.102301+0000","last_active":"2021-02-10T04:59:18.633925+0000","last_peered":"2021-02-10T04:59:18.633925+0000","last_clean":"2021-02-10T04:59:18.633925+0000","last_became_active":"2021-02-10T04:52:26.024812+0000","last_became_peered":"2021-02-10T04:52:26.024812+0000","last_unstale":"2021-02-10T04:59:18.633925+0000","last_undegraded":"2021-02-10T04:59:18.633925+0000","last_fullsized":"2021-02-10T04:59:18.633925+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1886585,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":18587,"num_write":9,"num_write_kb":3315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"1362","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.975730+0000","last_change":"2021-02-10T04:53:07.025068+0000","last_active":"2021-02-10T04:59:20.975730+0000","last_peered":"2021-02-10T04:59:20.975730+0000","last_clean":"2021-02-10T04:59:20.975730+0000","last_became_active":"2021-02-10T04:50:59.589417+0000","last_became_peered":"2021-02-10T04:50:59.589417+0000","last_unstale":"2021-02-10T04:59:20.975730+0000","last_undegraded":"2021-02-10T04:59:20.975730+0000","last_fullsized":"2021-02-10T04:59:20.975730+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.024980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.024980+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"339","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.632237+0000","last_change":"2021-02-10T04:58:56.851037+0000","last_active":"2021-02-10T04:59:18.632237+0000","last_peered":"2021-02-10T04:59:18.632237+0000","last_clean":"2021-02-10T04:59:18.632237+0000","last_became_active":"2021-02-10T04:58:56.850742+0000","last_became_peered":"2021-02-10T04:58:56.850742+0000","last_unstale":"2021-02-10T04:59:18.632237+0000","last_undegraded":"2021-02-10T04:59:18.632237+0000","last_fullsized":"2021-02-10T04:59:18.632237+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.609500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.609500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"1261'86","reported_seq":"1538","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.634188+0000","last_change":"2021-02-10T04:59:13.931273+0000","last_active":"2021-02-10T04:59:18.634188+0000","last_peered":"2021-02-10T04:59:18.634188+0000","last_clean":"2021-02-10T04:59:18.634188+0000","last_became_active":"2021-02-10T04:59:13.930845+0000","last_became_peered":"2021-02-10T04:59:13.930845+0000","last_unstale":"2021-02-10T04:59:18.634188+0000","last_undegraded":"2021-02-10T04:59:18.634188+0000","last_fullsized":"2021-02-10T04:59:18.634188+0000","mapping_epoch":1238,"log_start":"1049'68","ondisk_log_start":"1049'68","created":55,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'39","last_scrub_stamp":"2021-02-10T04:53:38.085565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.085565+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":791,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":113,"num_read_kb":93,"num_write":64,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1582,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"1476","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.637006+0000","last_change":"2021-02-10T04:58:56.844400+0000","last_active":"2021-02-10T04:59:18.637006+0000","last_peered":"2021-02-10T04:59:18.637006+0000","last_clean":"2021-02-10T04:59:18.637006+0000","last_became_active":"2021-02-10T04:58:56.843963+0000","last_became_peered":"2021-02-10T04:58:56.843963+0000","last_unstale":"2021-02-10T04:59:18.637006+0000","last_undegraded":"2021-02-10T04:59:18.637006+0000","last_fullsized":"2021-02-10T04:59:18.637006+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:47:08.784593+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:47:08.784593+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"1310","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.634987+0000","last_change":"2021-02-10T04:52:30.019225+0000","last_active":"2021-02-10T04:59:18.634987+0000","last_peered":"2021-02-10T04:59:18.634987+0000","last_clean":"2021-02-10T04:59:18.634987+0000","last_became_active":"2021-02-10T04:50:59.561436+0000","last_became_peered":"2021-02-10T04:50:59.561436+0000","last_unstale":"2021-02-10T04:59:18.634987+0000","last_undegraded":"2021-02-10T04:59:18.634987+0000","last_fullsized":"2021-02-10T04:59:18.634987+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.019102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.019102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"1001'1528","reported_seq":"6295","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.633433+0000","last_change":"2021-02-10T04:52:26.036366+0000","last_active":"2021-02-10T04:59:18.633433+0000","last_peered":"2021-02-10T04:59:18.633433+0000","last_clean":"2021-02-10T04:59:18.633433+0000","last_became_active":"2021-02-10T04:52:26.034028+0000","last_became_peered":"2021-02-10T04:52:26.034028+0000","last_unstale":"2021-02-10T04:59:18.633433+0000","last_undegraded":"2021-02-10T04:59:18.633433+0000","last_fullsized":"2021-02-10T04:59:18.633433+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46944136,"num_objects":16,"num_object_clones":15,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":358,"num_read_kb":351934,"num_write":286,"num_write_kb":111355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":58968832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"1261'58","reported_seq":"1412","reported_epoch":"1346","state":"active+remapped+backfill_toofull","last_fresh":"2021-02-10T04:59:17.962561+0000","last_change":"2021-02-10T04:59:16.972680+0000","last_active":"2021-02-10T04:59:17.962561+0000","last_peered":"2021-02-10T04:59:17.962561+0000","last_clean":"2021-02-10T04:59:13.930979+0000","last_became_active":"2021-02-10T04:59:16.970832+0000","last_became_peered":"2021-02-10T04:59:16.970832+0000","last_unstale":"2021-02-10T04:59:17.962561+0000","last_undegraded":"2021-02-10T04:59:17.962561+0000","last_fullsized":"2021-02-10T04:59:17.962561+0000","mapping_epoch":1344,"log_start":"1049'40","ondisk_log_start":"1049'40","created":1341,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":5,"last_scrub":"1049'40","last_scrub_stamp":"2021-02-10T04:51:47.281929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.281929+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":918,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":24,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":85,"num_read_kb":67,"num_write":50,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,1,2],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":12}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"1000'446","reported_seq":"2284","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.636995+0000","last_change":"2021-02-10T04:53:11.051124+0000","last_active":"2021-02-10T04:59:18.636995+0000","last_peered":"2021-02-10T04:59:18.636995+0000","last_clean":"2021-02-10T04:59:18.636995+0000","last_became_active":"2021-02-10T04:51:40.510899+0000","last_became_peered":"2021-02-10T04:51:40.510899+0000","last_unstale":"2021-02-10T04:59:18.636995+0000","last_undegraded":"2021-02-10T04:59:18.636995+0000","last_fullsized":"2021-02-10T04:59:18.636995+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:53:11.051031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.051031+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24896285,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":18983408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"273","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.634027+0000","last_change":"2021-02-10T04:58:56.851004+0000","last_active":"2021-02-10T04:59:18.634027+0000","last_peered":"2021-02-10T04:59:18.634027+0000","last_clean":"2021-02-10T04:59:18.634027+0000","last_became_active":"2021-02-10T04:58:56.850650+0000","last_became_peered":"2021-02-10T04:58:56.850650+0000","last_unstale":"2021-02-10T04:59:18.634027+0000","last_undegraded":"2021-02-10T04:59:18.634027+0000","last_fullsized":"2021-02-10T04:59:18.634027+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:51:08.639784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.639784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"1279","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.635384+0000","last_change":"2021-02-10T04:51:09.475150+0000","last_active":"2021-02-10T04:59:18.635384+0000","last_peered":"2021-02-10T04:59:18.635384+0000","last_clean":"2021-02-10T04:59:18.635384+0000","last_became_active":"2021-02-10T04:50:59.585917+0000","last_became_peered":"2021-02-10T04:50:59.585917+0000","last_unstale":"2021-02-10T04:59:18.635384+0000","last_undegraded":"2021-02-10T04:59:18.635384+0000","last_fullsized":"2021-02-10T04:59:18.635384+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:09.475063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:09.475063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"249","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.635661+0000","last_change":"2021-02-10T04:53:34.927483+0000","last_active":"2021-02-10T04:59:18.635661+0000","last_peered":"2021-02-10T04:59:18.635661+0000","last_clean":"2021-02-10T04:59:18.635661+0000","last_became_active":"2021-02-10T04:51:36.859707+0000","last_became_peered":"2021-02-10T04:51:36.859707+0000","last_unstale":"2021-02-10T04:59:18.635661+0000","last_undegraded":"2021-02-10T04:59:18.635661+0000","last_fullsized":"2021-02-10T04:59:18.635661+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:34.927424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.927424+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"1261'40","reported_seq":"1018","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.642825+0000","last_change":"2021-02-10T04:59:18.642444+0000","last_active":"2021-02-10T04:59:18.642825+0000","last_peered":"2021-02-10T04:59:18.642825+0000","last_clean":"2021-02-10T04:59:18.642825+0000","last_became_active":"2021-02-10T04:59:18.641966+0000","last_became_peered":"2021-02-10T04:59:18.641966+0000","last_unstale":"2021-02-10T04:59:18.642825+0000","last_undegraded":"2021-02-10T04:59:18.642825+0000","last_fullsized":"2021-02-10T04:59:18.642825+0000","mapping_epoch":1346,"log_start":"1063'30","ondisk_log_start":"1063'30","created":1341,"last_epoch_clean":1347,"parent":"0.0","parent_split_bits":5,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:56.431187+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.431187+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":401,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":58,"num_read_kb":47,"num_write":34,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.14","version":"1000'620","reported_seq":"4634","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.635322+0000","last_change":"2021-02-10T04:53:37.384461+0000","last_active":"2021-02-10T04:59:18.635322+0000","last_peered":"2021-02-10T04:59:18.635322+0000","last_clean":"2021-02-10T04:59:18.635322+0000","last_became_active":"2021-02-10T04:51:40.510114+0000","last_became_peered":"2021-02-10T04:51:40.510114+0000","last_unstale":"2021-02-10T04:59:18.635322+0000","last_undegraded":"2021-02-10T04:59:18.635322+0000","last_fullsized":"2021-02-10T04:59:18.635322+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:53:37.384398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:37.384398+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28530390,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"1336","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.973517+0000","last_change":"2021-02-10T04:52:41.059269+0000","last_active":"2021-02-10T04:59:20.973517+0000","last_peered":"2021-02-10T04:59:20.973517+0000","last_clean":"2021-02-10T04:59:20.973517+0000","last_became_active":"2021-02-10T04:25:34.333561+0000","last_became_peered":"2021-02-10T04:25:34.333561+0000","last_unstale":"2021-02-10T04:59:20.973517+0000","last_undegraded":"2021-02-10T04:59:20.973517+0000","last_fullsized":"2021-02-10T04:59:20.973517+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:52:41.059159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.059159+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"1340","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.976839+0000","last_change":"2021-02-10T04:53:12.332890+0000","last_active":"2021-02-10T04:59:20.976839+0000","last_peered":"2021-02-10T04:59:20.976839+0000","last_clean":"2021-02-10T04:59:20.976839+0000","last_became_active":"2021-02-10T04:50:59.540375+0000","last_became_peered":"2021-02-10T04:50:59.540375+0000","last_unstale":"2021-02-10T04:59:20.976839+0000","last_undegraded":"2021-02-10T04:59:20.976839+0000","last_fullsized":"2021-02-10T04:59:20.976839+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.332233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.332233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"335","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.964325+0000","last_change":"2021-02-10T04:58:56.576305+0000","last_active":"2021-02-10T04:59:17.964325+0000","last_peered":"2021-02-10T04:59:17.964325+0000","last_clean":"2021-02-10T04:59:17.964325+0000","last_became_active":"2021-02-10T04:53:46.733191+0000","last_became_peered":"2021-02-10T04:53:46.733191+0000","last_unstale":"2021-02-10T04:59:17.964325+0000","last_undegraded":"2021-02-10T04:59:17.964325+0000","last_fullsized":"2021-02-10T04:59:17.964325+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:45.362537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:45.362537+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"1002'2042","reported_seq":"9319","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.965259+0000","last_change":"2021-02-10T04:53:46.726353+0000","last_active":"2021-02-10T04:59:17.965259+0000","last_peered":"2021-02-10T04:59:17.965259+0000","last_clean":"2021-02-10T04:59:17.965259+0000","last_became_active":"2021-02-10T04:53:46.725828+0000","last_became_peered":"2021-02-10T04:53:46.725828+0000","last_unstale":"2021-02-10T04:59:17.965259+0000","last_undegraded":"2021-02-10T04:59:17.965259+0000","last_fullsized":"2021-02-10T04:59:17.965259+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:52:41.286006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.286006+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46300076,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":545,"num_read_kb":531689,"num_write":399,"num_write_kb":153087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"1261'56","reported_seq":"1518","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.639119+0000","last_change":"2021-02-10T04:52:35.091494+0000","last_active":"2021-02-10T04:59:18.639119+0000","last_peered":"2021-02-10T04:59:18.639119+0000","last_clean":"2021-02-10T04:59:18.639119+0000","last_became_active":"2021-02-10T04:50:59.569915+0000","last_became_peered":"2021-02-10T04:50:59.569915+0000","last_unstale":"2021-02-10T04:59:18.639119+0000","last_undegraded":"2021-02-10T04:59:18.639119+0000","last_fullsized":"2021-02-10T04:59:18.639119+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'46","last_scrub_stamp":"2021-02-10T04:52:35.091374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:35.091374+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":95,"num_read_kb":80,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"84'7","reported_seq":"1364","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.976175+0000","last_change":"2021-02-10T04:51:52.498251+0000","last_active":"2021-02-10T04:59:20.976175+0000","last_peered":"2021-02-10T04:59:20.976175+0000","last_clean":"2021-02-10T04:59:20.976175+0000","last_became_active":"2021-02-10T04:50:59.575924+0000","last_became_peered":"2021-02-10T04:50:59.575924+0000","last_unstale":"2021-02-10T04:59:20.976175+0000","last_undegraded":"2021-02-10T04:59:20.976175+0000","last_fullsized":"2021-02-10T04:59:20.976175+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'7","last_scrub_stamp":"2021-02-10T04:51:52.498158+0000","last_deep_scrub":"84'7","last_deep_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.498158+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"1333'6","reported_seq":"1761","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.977389+0000","last_change":"2021-02-10T04:58:57.207618+0000","last_active":"2021-02-10T04:59:20.977389+0000","last_peered":"2021-02-10T04:59:20.977389+0000","last_clean":"2021-02-10T04:59:20.977389+0000","last_became_active":"2021-02-10T04:58:56.849778+0000","last_became_peered":"2021-02-10T04:58:56.849778+0000","last_unstale":"2021-02-10T04:59:20.977389+0000","last_undegraded":"2021-02-10T04:59:20.977389+0000","last_fullsized":"2021-02-10T04:59:20.977389+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1317'5","last_scrub_stamp":"2021-02-10T04:58:57.207527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:58:57.207527+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"1252","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.963686+0000","last_change":"2021-02-10T04:59:08.653632+0000","last_active":"2021-02-10T04:59:17.963686+0000","last_peered":"2021-02-10T04:59:17.963686+0000","last_clean":"2021-02-10T04:59:17.963686+0000","last_became_active":"2021-02-10T04:58:56.965228+0000","last_became_peered":"2021-02-10T04:58:56.965228+0000","last_unstale":"2021-02-10T04:59:17.963686+0000","last_undegraded":"2021-02-10T04:59:17.963686+0000","last_fullsized":"2021-02-10T04:59:17.963686+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:23.566027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_clean_scrub_stamp":"2021-02-10T04:52:23.566027+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"1000'490","reported_seq":"3878","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.637171+0000","last_change":"2021-02-10T04:58:56.966763+0000","last_active":"2021-02-10T04:59:18.637171+0000","last_peered":"2021-02-10T04:59:18.637171+0000","last_clean":"2021-02-10T04:59:18.637171+0000","last_became_active":"2021-02-10T04:58:56.965659+0000","last_became_peered":"2021-02-10T04:58:56.965659+0000","last_unstale":"2021-02-10T04:59:18.637171+0000","last_undegraded":"2021-02-10T04:59:18.637171+0000","last_fullsized":"2021-02-10T04:59:18.637171+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:50:03.756248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:50:03.756248+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16154790,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"1261'46","reported_seq":"1417","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.977247+0000","last_change":"2021-02-10T04:53:10.276449+0000","last_active":"2021-02-10T04:59:20.977247+0000","last_peered":"2021-02-10T04:59:20.977247+0000","last_clean":"2021-02-10T04:59:20.977247+0000","last_became_active":"2021-02-10T04:50:59.539529+0000","last_became_peered":"2021-02-10T04:50:59.539529+0000","last_unstale":"2021-02-10T04:59:20.977247+0000","last_undegraded":"2021-02-10T04:59:20.977247+0000","last_fullsized":"2021-02-10T04:59:20.977247+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1112'40","last_scrub_stamp":"2021-02-10T04:53:10.276358+0000","last_deep_scrub":"1112'40","last_deep_scrub_stamp":"2021-02-10T04:51:01.450050+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.276358+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":53,"num_write":38,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"1237","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.962054+0000","last_change":"2021-02-10T04:59:08.645662+0000","last_active":"2021-02-10T04:59:17.962054+0000","last_peered":"2021-02-10T04:59:17.962054+0000","last_clean":"2021-02-10T04:59:17.962054+0000","last_became_active":"2021-02-10T04:58:56.970118+0000","last_became_peered":"2021-02-10T04:58:56.970118+0000","last_unstale":"2021-02-10T04:59:17.962054+0000","last_undegraded":"2021-02-10T04:59:17.962054+0000","last_fullsized":"2021-02-10T04:59:17.962054+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:23.904210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:47:23.904210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"1119'6","reported_seq":"1653","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.639259+0000","last_change":"2021-02-10T04:52:40.038646+0000","last_active":"2021-02-10T04:59:18.639259+0000","last_peered":"2021-02-10T04:59:18.639259+0000","last_clean":"2021-02-10T04:59:18.639259+0000","last_became_active":"2021-02-10T04:51:36.869733+0000","last_became_peered":"2021-02-10T04:51:36.869733+0000","last_unstale":"2021-02-10T04:59:18.639259+0000","last_undegraded":"2021-02-10T04:59:18.639259+0000","last_fullsized":"2021-02-10T04:59:18.639259+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'6","last_scrub_stamp":"2021-02-10T04:52:40.038554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:40.038554+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"1333","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.637394+0000","last_change":"2021-02-10T04:51:36.849933+0000","last_active":"2021-02-10T04:59:18.637394+0000","last_peered":"2021-02-10T04:59:18.637394+0000","last_clean":"2021-02-10T04:59:18.637394+0000","last_became_active":"2021-02-10T04:51:36.849184+0000","last_became_peered":"2021-02-10T04:51:36.849184+0000","last_unstale":"2021-02-10T04:59:18.637394+0000","last_undegraded":"2021-02-10T04:59:18.637394+0000","last_fullsized":"2021-02-10T04:59:18.637394+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:11.779779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:11.779779+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"1032'89","reported_seq":"940","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.964051+0000","last_change":"2021-02-10T04:53:46.728868+0000","last_active":"2021-02-10T04:59:17.964051+0000","last_peered":"2021-02-10T04:59:17.964051+0000","last_clean":"2021-02-10T04:59:17.964051+0000","last_became_active":"2021-02-10T04:53:46.728356+0000","last_became_peered":"2021-02-10T04:53:46.728356+0000","last_unstale":"2021-02-10T04:59:17.964051+0000","last_undegraded":"2021-02-10T04:59:17.964051+0000","last_fullsized":"2021-02-10T04:59:17.964051+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":5,"last_scrub":"1032'89","last_scrub_stamp":"2021-02-10T04:52:34.089789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:34.089789+0000","log_size":89,"ondisk_log_size":89,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5218574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":30326,"num_write":21,"num_write_kb":12444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3210022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"1261'43","reported_seq":"1037","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.638604+0000","last_change":"2021-02-10T04:53:14.202505+0000","last_active":"2021-02-10T04:59:18.638604+0000","last_peered":"2021-02-10T04:59:18.638604+0000","last_clean":"2021-02-10T04:59:18.638604+0000","last_became_active":"2021-02-10T04:51:49.496294+0000","last_became_peered":"2021-02-10T04:51:49.496294+0000","last_unstale":"2021-02-10T04:59:18.638604+0000","last_undegraded":"2021-02-10T04:59:18.638604+0000","last_fullsized":"2021-02-10T04:59:18.638604+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"1163'37","last_scrub_stamp":"2021-02-10T04:53:14.202445+0000","last_deep_scrub":"1163'37","last_deep_scrub_stamp":"2021-02-10T04:53:14.202445+0000","last_clean_scrub_stamp":"2021-02-10T04:53:14.202445+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":60,"num_read_kb":42,"num_write":34,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"1349","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.974389+0000","last_change":"2021-02-10T04:52:23.095651+0000","last_active":"2021-02-10T04:59:20.974389+0000","last_peered":"2021-02-10T04:59:20.974389+0000","last_clean":"2021-02-10T04:59:20.974389+0000","last_became_active":"2021-02-10T04:50:59.543531+0000","last_became_peered":"2021-02-10T04:50:59.543531+0000","last_unstale":"2021-02-10T04:59:20.974389+0000","last_undegraded":"2021-02-10T04:59:20.974389+0000","last_fullsized":"2021-02-10T04:59:20.974389+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:23.095588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:23.095588+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"1313","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.634837+0000","last_change":"2021-02-10T04:58:56.965982+0000","last_active":"2021-02-10T04:59:18.634837+0000","last_peered":"2021-02-10T04:59:18.634837+0000","last_clean":"2021-02-10T04:59:18.634837+0000","last_became_active":"2021-02-10T04:58:56.964476+0000","last_became_peered":"2021-02-10T04:58:56.964476+0000","last_unstale":"2021-02-10T04:59:18.634837+0000","last_undegraded":"2021-02-10T04:59:18.634837+0000","last_fullsized":"2021-02-10T04:59:18.634837+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:43.663762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:51:43.663762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"1208","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.959519+0000","last_change":"2021-02-10T04:59:08.655593+0000","last_active":"2021-02-10T04:59:17.959519+0000","last_peered":"2021-02-10T04:59:17.959519+0000","last_clean":"2021-02-10T04:59:17.959519+0000","last_became_active":"2021-02-10T04:58:56.877585+0000","last_became_peered":"2021-02-10T04:58:56.877585+0000","last_unstale":"2021-02-10T04:59:17.959519+0000","last_undegraded":"2021-02-10T04:59:17.959519+0000","last_fullsized":"2021-02-10T04:59:17.959519+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:57.568541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:57.568541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"1000'1071","reported_seq":"4853","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.633151+0000","last_change":"2021-02-10T04:53:35.302879+0000","last_active":"2021-02-10T04:59:18.633151+0000","last_peered":"2021-02-10T04:59:18.633151+0000","last_clean":"2021-02-10T04:59:18.633151+0000","last_became_active":"2021-02-10T04:51:39.508551+0000","last_became_peered":"2021-02-10T04:51:39.508551+0000","last_unstale":"2021-02-10T04:59:18.633151+0000","last_undegraded":"2021-02-10T04:59:18.633151+0000","last_fullsized":"2021-02-10T04:59:18.633151+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:35.302780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.302780+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"1261'39","reported_seq":"1382","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.636088+0000","last_change":"2021-02-10T04:52:39.695340+0000","last_active":"2021-02-10T04:59:18.636088+0000","last_peered":"2021-02-10T04:59:18.636088+0000","last_clean":"2021-02-10T04:59:18.636088+0000","last_became_active":"2021-02-10T04:25:49.011275+0000","last_became_peered":"2021-02-10T04:25:49.011275+0000","last_unstale":"2021-02-10T04:59:18.636088+0000","last_undegraded":"2021-02-10T04:59:18.636088+0000","last_fullsized":"2021-02-10T04:59:18.636088+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'31","last_scrub_stamp":"2021-02-10T04:52:39.695237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:39.695237+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":53,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"607","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.638134+0000","last_change":"2021-02-10T04:58:56.964767+0000","last_active":"2021-02-10T04:59:18.638134+0000","last_peered":"2021-02-10T04:59:18.638134+0000","last_clean":"2021-02-10T04:59:18.638134+0000","last_became_active":"2021-02-10T04:58:56.964387+0000","last_became_peered":"2021-02-10T04:58:56.964387+0000","last_unstale":"2021-02-10T04:59:18.638134+0000","last_undegraded":"2021-02-10T04:59:18.638134+0000","last_fullsized":"2021-02-10T04:59:18.638134+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T04:55:20.929330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:55:20.929330+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"1000","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.638760+0000","last_change":"2021-02-10T04:53:35.216489+0000","last_active":"2021-02-10T04:59:18.638760+0000","last_peered":"2021-02-10T04:59:18.638760+0000","last_clean":"2021-02-10T04:59:18.638760+0000","last_became_active":"2021-02-10T04:51:49.501254+0000","last_became_peered":"2021-02-10T04:51:49.501254+0000","last_unstale":"2021-02-10T04:59:18.638760+0000","last_undegraded":"2021-02-10T04:59:18.638760+0000","last_fullsized":"2021-02-10T04:59:18.638760+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.216394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.216394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"1340","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.633557+0000","last_change":"2021-02-10T04:58:56.879817+0000","last_active":"2021-02-10T04:59:18.633557+0000","last_peered":"2021-02-10T04:59:18.633557+0000","last_clean":"2021-02-10T04:59:18.633557+0000","last_became_active":"2021-02-10T04:58:56.879372+0000","last_became_peered":"2021-02-10T04:58:56.879372+0000","last_unstale":"2021-02-10T04:59:18.633557+0000","last_undegraded":"2021-02-10T04:59:18.633557+0000","last_fullsized":"2021-02-10T04:59:18.633557+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:55.814486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.814486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"1002'1657","reported_seq":"5365","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.961873+0000","last_change":"2021-02-10T04:59:08.654625+0000","last_active":"2021-02-10T04:59:17.961873+0000","last_peered":"2021-02-10T04:59:17.961873+0000","last_clean":"2021-02-10T04:59:17.961873+0000","last_became_active":"2021-02-10T04:58:56.961771+0000","last_became_peered":"2021-02-10T04:58:56.961771+0000","last_unstale":"2021-02-10T04:59:17.961873+0000","last_undegraded":"2021-02-10T04:59:17.961873+0000","last_fullsized":"2021-02-10T04:59:17.961873+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:55:22.331010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:55:22.331010+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":47335497,"num_objects":20,"num_object_clones":18,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"1261'24","reported_seq":"1455","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.633507+0000","last_change":"2021-02-10T04:53:04.058324+0000","last_active":"2021-02-10T04:59:18.633507+0000","last_peered":"2021-02-10T04:59:18.633507+0000","last_clean":"2021-02-10T04:59:18.633507+0000","last_became_active":"2021-02-10T04:50:59.544484+0000","last_became_peered":"2021-02-10T04:50:59.544484+0000","last_unstale":"2021-02-10T04:59:18.633507+0000","last_undegraded":"2021-02-10T04:59:18.633507+0000","last_fullsized":"2021-02-10T04:59:18.633507+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'20","last_scrub_stamp":"2021-02-10T04:53:04.058248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:04.058248+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":29,"num_write":24,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"1352","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.977003+0000","last_change":"2021-02-10T04:53:00.343307+0000","last_active":"2021-02-10T04:59:20.977003+0000","last_peered":"2021-02-10T04:59:20.977003+0000","last_clean":"2021-02-10T04:59:20.977003+0000","last_became_active":"2021-02-10T04:51:36.844573+0000","last_became_peered":"2021-02-10T04:51:36.844573+0000","last_unstale":"2021-02-10T04:59:20.977003+0000","last_undegraded":"2021-02-10T04:59:20.977003+0000","last_fullsized":"2021-02-10T04:59:20.977003+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:00.343228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:00.343228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"1320","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.973475+0000","last_change":"2021-02-10T04:50:56.099710+0000","last_active":"2021-02-10T04:59:20.973475+0000","last_peered":"2021-02-10T04:59:20.973475+0000","last_clean":"2021-02-10T04:59:20.973475+0000","last_became_active":"2021-02-10T04:25:51.015243+0000","last_became_peered":"2021-02-10T04:25:51.015243+0000","last_unstale":"2021-02-10T04:59:20.973475+0000","last_undegraded":"2021-02-10T04:59:20.973475+0000","last_fullsized":"2021-02-10T04:59:20.973475+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:56.099651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T04:50:56.099651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"665","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.964343+0000","last_change":"2021-02-10T04:53:46.718715+0000","last_active":"2021-02-10T04:59:17.964343+0000","last_peered":"2021-02-10T04:59:17.964343+0000","last_clean":"2021-02-10T04:59:17.964343+0000","last_became_active":"2021-02-10T04:53:46.718266+0000","last_became_peered":"2021-02-10T04:53:46.718266+0000","last_unstale":"2021-02-10T04:59:17.964343+0000","last_undegraded":"2021-02-10T04:59:17.964343+0000","last_fullsized":"2021-02-10T04:59:17.964343+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.876681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.876681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"998'1009","reported_seq":"5211","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.973830+0000","last_change":"2021-02-10T04:58:56.958942+0000","last_active":"2021-02-10T04:59:20.973830+0000","last_peered":"2021-02-10T04:59:20.973830+0000","last_clean":"2021-02-10T04:59:20.973830+0000","last_became_active":"2021-02-10T04:58:56.957778+0000","last_became_peered":"2021-02-10T04:58:56.957778+0000","last_unstale":"2021-02-10T04:59:20.973830+0000","last_undegraded":"2021-02-10T04:59:20.973830+0000","last_fullsized":"2021-02-10T04:59:20.973830+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":5,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:13.032506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:13.032506+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49420827,"num_objects":21,"num_object_clones":19,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":36763798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"1261'45","reported_seq":"1389","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.634066+0000","last_change":"2021-02-10T04:51:44.694779+0000","last_active":"2021-02-10T04:59:18.634066+0000","last_peered":"2021-02-10T04:59:18.634066+0000","last_clean":"2021-02-10T04:59:18.634066+0000","last_became_active":"2021-02-10T04:50:59.549412+0000","last_became_peered":"2021-02-10T04:50:59.549412+0000","last_unstale":"2021-02-10T04:59:18.634066+0000","last_undegraded":"2021-02-10T04:59:18.634066+0000","last_fullsized":"2021-02-10T04:59:18.634066+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:44.694711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.694711+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":76,"num_read_kb":62,"num_write":45,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"1487","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.976042+0000","last_change":"2021-02-10T04:52:33.088525+0000","last_active":"2021-02-10T04:59:20.976042+0000","last_peered":"2021-02-10T04:59:20.976042+0000","last_clean":"2021-02-10T04:59:20.976042+0000","last_became_active":"2021-02-10T04:51:36.870573+0000","last_became_peered":"2021-02-10T04:51:36.870573+0000","last_unstale":"2021-02-10T04:59:20.976042+0000","last_undegraded":"2021-02-10T04:59:20.976042+0000","last_fullsized":"2021-02-10T04:59:20.976042+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:52:33.088428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:33.088428+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"633","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.633600+0000","last_change":"2021-02-10T04:51:51.836723+0000","last_active":"2021-02-10T04:59:18.633600+0000","last_peered":"2021-02-10T04:59:18.633600+0000","last_clean":"2021-02-10T04:59:18.633600+0000","last_became_active":"2021-02-10T04:50:59.564977+0000","last_became_peered":"2021-02-10T04:50:59.564977+0000","last_unstale":"2021-02-10T04:59:18.633600+0000","last_undegraded":"2021-02-10T04:59:18.633600+0000","last_fullsized":"2021-02-10T04:59:18.633600+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.836638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.836638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"1304","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.634724+0000","last_change":"2021-02-10T04:58:56.966197+0000","last_active":"2021-02-10T04:59:18.634724+0000","last_peered":"2021-02-10T04:59:18.634724+0000","last_clean":"2021-02-10T04:59:18.634724+0000","last_became_active":"2021-02-10T04:58:56.964968+0000","last_became_peered":"2021-02-10T04:58:56.964968+0000","last_unstale":"2021-02-10T04:59:18.634724+0000","last_undegraded":"2021-02-10T04:59:18.634724+0000","last_fullsized":"2021-02-10T04:59:18.634724+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:58.514318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_clean_scrub_stamp":"2021-02-10T04:46:58.514318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"1261'65","reported_seq":"1375","reported_epoch":"1347","state":"active+remapped+backfill_toofull","last_fresh":"2021-02-10T04:59:18.637788+0000","last_change":"2021-02-10T04:59:16.977791+0000","last_active":"2021-02-10T04:59:18.637788+0000","last_peered":"2021-02-10T04:59:18.637788+0000","last_clean":"2021-02-10T04:55:57.139329+0000","last_became_active":"2021-02-10T04:59:16.972842+0000","last_became_peered":"2021-02-10T04:59:16.972842+0000","last_unstale":"2021-02-10T04:59:18.637788+0000","last_undegraded":"2021-02-10T04:59:18.637788+0000","last_fullsized":"2021-02-10T04:59:18.637788+0000","mapping_epoch":1344,"log_start":"1049'47","ondisk_log_start":"1049'47","created":1341,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":5,"last_scrub":"1049'47","last_scrub_stamp":"2021-02-10T04:53:09.002368+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.002368+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":836,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":84,"num_read_kb":67,"num_write":49,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":467,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[1,4,6],"avail_no_missing":["1","4","6"],"object_location_counts":[{"shards":"1,4,6","objects":11}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"1001'1165","reported_seq":"4363","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.638217+0000","last_change":"2021-02-10T04:58:56.966207+0000","last_active":"2021-02-10T04:59:18.638217+0000","last_peered":"2021-02-10T04:59:18.638217+0000","last_clean":"2021-02-10T04:59:18.638217+0000","last_became_active":"2021-02-10T04:58:56.965303+0000","last_became_peered":"2021-02-10T04:58:56.965303+0000","last_unstale":"2021-02-10T04:59:18.638217+0000","last_undegraded":"2021-02-10T04:59:18.638217+0000","last_fullsized":"2021-02-10T04:59:18.638217+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":5,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29601845,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":287,"num_read_kb":299530,"num_write":200,"num_write_kb":89131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"1393","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.973128+0000","last_change":"2021-02-10T04:53:01.331566+0000","last_active":"2021-02-10T04:59:20.973128+0000","last_peered":"2021-02-10T04:59:20.973128+0000","last_clean":"2021-02-10T04:59:20.973128+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T04:59:20.973128+0000","last_undegraded":"2021-02-10T04:59:20.973128+0000","last_fullsized":"2021-02-10T04:59:20.973128+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:01.331466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.331466+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"1098'3","reported_seq":"1731","reported_epoch":"1349","state":"active+clean","last_fresh":"2021-02-10T04:59:22.985130+0000","last_change":"2021-02-10T04:53:34.865668+0000","last_active":"2021-02-10T04:59:22.985130+0000","last_peered":"2021-02-10T04:59:22.985130+0000","last_clean":"2021-02-10T04:59:22.985130+0000","last_became_active":"2021-02-10T04:25:51.016653+0000","last_became_peered":"2021-02-10T04:25:51.016653+0000","last_unstale":"2021-02-10T04:59:22.985130+0000","last_undegraded":"2021-02-10T04:59:22.985130+0000","last_fullsized":"2021-02-10T04:59:22.985130+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"1098'3","last_scrub_stamp":"2021-02-10T04:53:34.865579+0000","last_deep_scrub":"1098'3","last_deep_scrub_stamp":"2021-02-10T04:50:15.255137+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.865579+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"1195","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.964814+0000","last_change":"2021-02-10T04:53:46.727603+0000","last_active":"2021-02-10T04:59:17.964814+0000","last_peered":"2021-02-10T04:59:17.964814+0000","last_clean":"2021-02-10T04:59:17.964814+0000","last_became_active":"2021-02-10T04:53:46.727053+0000","last_became_peered":"2021-02-10T04:53:46.727053+0000","last_unstale":"2021-02-10T04:59:17.964814+0000","last_undegraded":"2021-02-10T04:59:17.964814+0000","last_fullsized":"2021-02-10T04:59:17.964814+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:47.493057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.493057+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"1261'71","reported_seq":"1070","reported_epoch":"1347","state":"active+remapped+backfill_toofull","last_fresh":"2021-02-10T04:59:18.638688+0000","last_change":"2021-02-10T04:59:16.974340+0000","last_active":"2021-02-10T04:59:18.638688+0000","last_peered":"2021-02-10T04:59:18.638688+0000","last_clean":"2021-02-10T04:59:13.937559+0000","last_became_active":"2021-02-10T04:59:16.973179+0000","last_became_peered":"2021-02-10T04:59:16.973179+0000","last_unstale":"2021-02-10T04:59:18.638688+0000","last_undegraded":"2021-02-10T04:59:18.638688+0000","last_fullsized":"2021-02-10T04:59:18.638688+0000","mapping_epoch":1344,"log_start":"1049'51","ondisk_log_start":"1049'51","created":1341,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":5,"last_scrub":"1049'41","last_scrub_stamp":"2021-02-10T04:51:58.353660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:58.353660+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":85,"num_read_kb":68,"num_write":51,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":525,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[1,3,4],"avail_no_missing":["1","3","4"],"object_location_counts":[{"shards":"1,3,4","objects":11}],"blocked_by":[],"up_primary":6,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"1002'1090","reported_seq":"5373","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.976706+0000","last_change":"2021-02-10T04:53:41.353991+0000","last_active":"2021-02-10T04:59:20.976706+0000","last_peered":"2021-02-10T04:59:20.976706+0000","last_clean":"2021-02-10T04:59:20.976706+0000","last_became_active":"2021-02-10T04:52:31.088265+0000","last_became_peered":"2021-02-10T04:52:31.088265+0000","last_unstale":"2021-02-10T04:59:20.976706+0000","last_undegraded":"2021-02-10T04:59:20.976706+0000","last_fullsized":"2021-02-10T04:59:20.976706+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:53:41.353840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:41.353840+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"1308","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.633783+0000","last_change":"2021-02-10T04:58:56.879938+0000","last_active":"2021-02-10T04:59:18.633783+0000","last_peered":"2021-02-10T04:59:18.633783+0000","last_clean":"2021-02-10T04:59:18.633783+0000","last_became_active":"2021-02-10T04:58:56.879661+0000","last_became_peered":"2021-02-10T04:58:56.879661+0000","last_unstale":"2021-02-10T04:59:18.633783+0000","last_undegraded":"2021-02-10T04:59:18.633783+0000","last_fullsized":"2021-02-10T04:59:18.633783+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:42.537443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:42.537443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"1233","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.962197+0000","last_change":"2021-02-10T04:59:08.666315+0000","last_active":"2021-02-10T04:59:17.962197+0000","last_peered":"2021-02-10T04:59:17.962197+0000","last_clean":"2021-02-10T04:59:17.962197+0000","last_became_active":"2021-02-10T04:58:56.869951+0000","last_became_peered":"2021-02-10T04:58:56.869951+0000","last_unstale":"2021-02-10T04:59:17.962197+0000","last_undegraded":"2021-02-10T04:59:17.962197+0000","last_fullsized":"2021-02-10T04:59:17.962197+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.938731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.938731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"557","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.638001+0000","last_change":"2021-02-10T04:53:15.185748+0000","last_active":"2021-02-10T04:59:18.638001+0000","last_peered":"2021-02-10T04:59:18.638001+0000","last_clean":"2021-02-10T04:59:18.638001+0000","last_became_active":"2021-02-10T04:51:49.499348+0000","last_became_peered":"2021-02-10T04:51:49.499348+0000","last_unstale":"2021-02-10T04:59:18.638001+0000","last_undegraded":"2021-02-10T04:59:18.638001+0000","last_fullsized":"2021-02-10T04:59:18.638001+0000","mapping_epoch":1129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:15.185686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:15.185686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"1261'76","reported_seq":"1385","reported_epoch":"1347","state":"active+remapped+backfill_toofull","last_fresh":"2021-02-10T04:59:18.635084+0000","last_change":"2021-02-10T04:59:16.971417+0000","last_active":"2021-02-10T04:59:18.635084+0000","last_peered":"2021-02-10T04:59:18.635084+0000","last_clean":"2021-02-10T04:55:57.070023+0000","last_became_active":"2021-02-10T04:59:16.967773+0000","last_became_peered":"2021-02-10T04:59:16.967773+0000","last_unstale":"2021-02-10T04:59:18.635084+0000","last_undegraded":"2021-02-10T04:59:18.635084+0000","last_fullsized":"2021-02-10T04:59:18.635084+0000","mapping_epoch":1344,"log_start":"1049'39","ondisk_log_start":"1049'39","created":1341,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":5,"last_scrub":"1187'54","last_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_deep_scrub":"1187'54","last_deep_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_clean_scrub_stamp":"2021-02-10T04:53:42.071103+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":289,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":16,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":73,"num_read_kb":63,"num_write":48,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[3,4,6],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":8}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"1004'1093","reported_seq":"6610","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.975560+0000","last_change":"2021-02-10T04:52:31.089952+0000","last_active":"2021-02-10T04:59:20.975560+0000","last_peered":"2021-02-10T04:59:20.975560+0000","last_clean":"2021-02-10T04:59:20.975560+0000","last_became_active":"2021-02-10T04:52:31.088634+0000","last_became_peered":"2021-02-10T04:52:31.088634+0000","last_unstale":"2021-02-10T04:59:20.975560+0000","last_undegraded":"2021-02-10T04:59:20.975560+0000","last_fullsized":"2021-02-10T04:59:20.975560+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39434697,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":279,"num_read_kb":311058,"num_write":186,"num_write_kb":66790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":53995913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"1350","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.974302+0000","last_change":"2021-02-10T04:55:39.651056+0000","last_active":"2021-02-10T04:59:20.974302+0000","last_peered":"2021-02-10T04:59:20.974302+0000","last_clean":"2021-02-10T04:59:20.974302+0000","last_became_active":"2021-02-10T04:55:39.596648+0000","last_became_peered":"2021-02-10T04:55:39.596648+0000","last_unstale":"2021-02-10T04:59:20.974302+0000","last_undegraded":"2021-02-10T04:59:20.974302+0000","last_fullsized":"2021-02-10T04:59:20.974302+0000","mapping_epoch":1248,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:52:36.083784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:36.083784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"654","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.638424+0000","last_change":"2021-02-10T04:59:07.422972+0000","last_active":"2021-02-10T04:59:18.638424+0000","last_peered":"2021-02-10T04:59:18.638424+0000","last_clean":"2021-02-10T04:59:18.638424+0000","last_became_active":"2021-02-10T04:58:56.965808+0000","last_became_peered":"2021-02-10T04:58:56.965808+0000","last_unstale":"2021-02-10T04:59:18.638424+0000","last_undegraded":"2021-02-10T04:59:18.638424+0000","last_fullsized":"2021-02-10T04:59:18.638424+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:08.758129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.758129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"1315","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.636841+0000","last_change":"2021-02-10T04:51:44.799512+0000","last_active":"2021-02-10T04:59:18.636841+0000","last_peered":"2021-02-10T04:59:18.636841+0000","last_clean":"2021-02-10T04:59:18.636841+0000","last_became_active":"2021-02-10T04:25:53.149887+0000","last_became_peered":"2021-02-10T04:25:53.149887+0000","last_unstale":"2021-02-10T04:59:18.636841+0000","last_undegraded":"2021-02-10T04:59:18.636841+0000","last_fullsized":"2021-02-10T04:59:18.636841+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:44.799429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.799429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"1261'64","reported_seq":"1488","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.642399+0000","last_change":"2021-02-10T04:59:18.642028+0000","last_active":"2021-02-10T04:59:18.642399+0000","last_peered":"2021-02-10T04:59:18.642399+0000","last_clean":"2021-02-10T04:59:18.642399+0000","last_became_active":"2021-02-10T04:59:18.641628+0000","last_became_peered":"2021-02-10T04:59:18.641628+0000","last_unstale":"2021-02-10T04:59:18.642399+0000","last_undegraded":"2021-02-10T04:59:18.642399+0000","last_fullsized":"2021-02-10T04:59:18.642399+0000","mapping_epoch":1346,"log_start":"1066'44","ondisk_log_start":"1066'44","created":1341,"last_epoch_clean":1347,"parent":"0.0","parent_split_bits":5,"last_scrub":"1066'44","last_scrub_stamp":"2021-02-10T04:53:37.108677+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:53:37.108677+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":789,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":90,"num_read_kb":72,"num_write":53,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":2000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"1063'654","reported_seq":"4590","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.638465+0000","last_change":"2021-02-10T04:53:40.135450+0000","last_active":"2021-02-10T04:59:18.638465+0000","last_peered":"2021-02-10T04:59:18.638465+0000","last_clean":"2021-02-10T04:59:18.638465+0000","last_became_active":"2021-02-10T04:52:31.087662+0000","last_became_peered":"2021-02-10T04:52:31.087662+0000","last_unstale":"2021-02-10T04:59:18.638465+0000","last_undegraded":"2021-02-10T04:59:18.638465+0000","last_fullsized":"2021-02-10T04:59:18.638465+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:53:40.135367+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:53:40.135367+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"1299","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.635109+0000","last_change":"2021-02-10T04:51:08.006463+0000","last_active":"2021-02-10T04:59:18.635109+0000","last_peered":"2021-02-10T04:59:18.635109+0000","last_clean":"2021-02-10T04:59:18.635109+0000","last_became_active":"2021-02-10T04:50:59.563657+0000","last_became_peered":"2021-02-10T04:50:59.563657+0000","last_unstale":"2021-02-10T04:59:18.635109+0000","last_undegraded":"2021-02-10T04:59:18.635109+0000","last_fullsized":"2021-02-10T04:59:18.635109+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_clean_scrub_stamp":"2021-02-10T04:51:02.455702+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"1350","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.635711+0000","last_change":"2021-02-10T04:52:33.741801+0000","last_active":"2021-02-10T04:59:18.635711+0000","last_peered":"2021-02-10T04:59:18.635711+0000","last_clean":"2021-02-10T04:59:18.635711+0000","last_became_active":"2021-02-10T04:51:36.860129+0000","last_became_peered":"2021-02-10T04:51:36.860129+0000","last_unstale":"2021-02-10T04:59:18.635711+0000","last_undegraded":"2021-02-10T04:59:18.635711+0000","last_fullsized":"2021-02-10T04:59:18.635711+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:33.741711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:46:58.666264+0000","last_clean_scrub_stamp":"2021-02-10T04:52:33.741711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"1311","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.635219+0000","last_change":"2021-02-10T04:52:25.030419+0000","last_active":"2021-02-10T04:59:18.635219+0000","last_peered":"2021-02-10T04:59:18.635219+0000","last_clean":"2021-02-10T04:59:18.635219+0000","last_became_active":"2021-02-10T04:50:59.539582+0000","last_became_peered":"2021-02-10T04:50:59.539582+0000","last_unstale":"2021-02-10T04:59:18.635219+0000","last_undegraded":"2021-02-10T04:59:18.635219+0000","last_fullsized":"2021-02-10T04:59:18.635219+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:25.030281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.030281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"1261'69","reported_seq":"1417","reported_epoch":"1347","state":"active+remapped+backfill_toofull","last_fresh":"2021-02-10T04:59:18.637243+0000","last_change":"2021-02-10T04:59:16.979308+0000","last_active":"2021-02-10T04:59:18.637243+0000","last_peered":"2021-02-10T04:59:18.637243+0000","last_clean":"2021-02-10T04:55:57.166865+0000","last_became_active":"2021-02-10T04:59:16.974764+0000","last_became_peered":"2021-02-10T04:59:16.974764+0000","last_unstale":"2021-02-10T04:59:18.637243+0000","last_undegraded":"2021-02-10T04:59:18.637243+0000","last_fullsized":"2021-02-10T04:59:18.637243+0000","mapping_epoch":1344,"log_start":"1112'47","ondisk_log_start":"1112'47","created":1341,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":5,"last_scrub":"1049'61","last_scrub_stamp":"2021-02-10T04:53:36.836202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:36.836202+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":733,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":36,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":104,"num_read_kb":86,"num_write":61,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":367,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[1,6,7],"avail_no_missing":["1","6","7"],"object_location_counts":[{"shards":"1,6,7","objects":12}],"blocked_by":[],"up_primary":4,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.19","version":"1003'739","reported_seq":"4097","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.973304+0000","last_change":"2021-02-10T04:53:01.929261+0000","last_active":"2021-02-10T04:59:20.973304+0000","last_peered":"2021-02-10T04:59:20.973304+0000","last_clean":"2021-02-10T04:59:20.973304+0000","last_became_active":"2021-02-10T04:51:40.511505+0000","last_became_peered":"2021-02-10T04:51:40.511505+0000","last_unstale":"2021-02-10T04:59:20.973304+0000","last_undegraded":"2021-02-10T04:59:20.973304+0000","last_fullsized":"2021-02-10T04:59:20.973304+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:53:01.929169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.929169+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22230846,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":18429397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"1186","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.964473+0000","last_change":"2021-02-10T04:53:46.727590+0000","last_active":"2021-02-10T04:59:17.964473+0000","last_peered":"2021-02-10T04:59:17.964473+0000","last_clean":"2021-02-10T04:59:17.964473+0000","last_became_active":"2021-02-10T04:53:46.726584+0000","last_became_peered":"2021-02-10T04:53:46.726584+0000","last_unstale":"2021-02-10T04:59:17.964473+0000","last_undegraded":"2021-02-10T04:59:17.964473+0000","last_fullsized":"2021-02-10T04:59:17.964473+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:26.408326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.408326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"1349","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.976171+0000","last_change":"2021-02-10T04:53:07.324650+0000","last_active":"2021-02-10T04:59:20.976171+0000","last_peered":"2021-02-10T04:59:20.976171+0000","last_clean":"2021-02-10T04:59:20.976171+0000","last_became_active":"2021-02-10T04:50:59.575340+0000","last_became_peered":"2021-02-10T04:50:59.575340+0000","last_unstale":"2021-02-10T04:59:20.976171+0000","last_undegraded":"2021-02-10T04:59:20.976171+0000","last_fullsized":"2021-02-10T04:59:20.976171+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.324564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.324564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"1232","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.959287+0000","last_change":"2021-02-10T04:59:08.651086+0000","last_active":"2021-02-10T04:59:17.959287+0000","last_peered":"2021-02-10T04:59:17.959287+0000","last_clean":"2021-02-10T04:59:17.959287+0000","last_became_active":"2021-02-10T04:58:56.953706+0000","last_became_peered":"2021-02-10T04:58:56.953706+0000","last_unstale":"2021-02-10T04:59:17.959287+0000","last_undegraded":"2021-02-10T04:59:17.959287+0000","last_fullsized":"2021-02-10T04:59:17.959287+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:00.501127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:00.501127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"1261'28","reported_seq":"1287","reported_epoch":"1347","state":"active+remapped+backfill_toofull","last_fresh":"2021-02-10T04:59:18.634085+0000","last_change":"2021-02-10T04:59:16.968346+0000","last_active":"2021-02-10T04:59:18.634085+0000","last_peered":"2021-02-10T04:59:18.634085+0000","last_clean":"2021-02-10T04:55:57.106046+0000","last_became_active":"2021-02-10T04:59:16.966181+0000","last_became_peered":"2021-02-10T04:59:16.966181+0000","last_unstale":"2021-02-10T04:59:18.634085+0000","last_undegraded":"2021-02-10T04:59:18.634085+0000","last_fullsized":"2021-02-10T04:59:18.634085+0000","mapping_epoch":1344,"log_start":"1061'20","ondisk_log_start":"1061'20","created":1341,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":5,"last_scrub":"1049'32","last_scrub_stamp":"2021-02-10T04:51:51.382040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.382040+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":579,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":29,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1026,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,1,0],"avail_no_missing":["4","0","1"],"object_location_counts":[{"shards":"0,1,4","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"259'74","reported_seq":"277","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.959438+0000","last_change":"2021-02-10T04:59:08.650175+0000","last_active":"2021-02-10T04:59:17.959438+0000","last_peered":"2021-02-10T04:59:17.959438+0000","last_clean":"2021-02-10T04:59:17.959438+0000","last_became_active":"2021-02-10T04:58:56.870024+0000","last_became_peered":"2021-02-10T04:58:56.870024+0000","last_unstale":"2021-02-10T04:59:17.959438+0000","last_undegraded":"2021-02-10T04:59:17.959438+0000","last_fullsized":"2021-02-10T04:59:17.959438+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":5,"last_scrub":"259'74","last_scrub_stamp":"2021-02-10T04:52:26.597668+0000","last_deep_scrub":"259'74","last_deep_scrub_stamp":"2021-02-10T04:46:49.745104+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.597668+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1218524,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":28228,"num_write":19,"num_write_kb":7022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"132","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.632633+0000","last_change":"2021-02-10T04:59:06.400383+0000","last_active":"2021-02-10T04:59:18.632633+0000","last_peered":"2021-02-10T04:59:18.632633+0000","last_clean":"2021-02-10T04:59:18.632633+0000","last_became_active":"2021-02-10T04:58:56.850901+0000","last_became_peered":"2021-02-10T04:58:56.850901+0000","last_unstale":"2021-02-10T04:59:18.632633+0000","last_undegraded":"2021-02-10T04:59:18.632633+0000","last_fullsized":"2021-02-10T04:59:18.632633+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:59.167911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:50:59.167911+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"256","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.964182+0000","last_change":"2021-02-10T04:53:46.718512+0000","last_active":"2021-02-10T04:59:17.964182+0000","last_peered":"2021-02-10T04:59:17.964182+0000","last_clean":"2021-02-10T04:59:17.964182+0000","last_became_active":"2021-02-10T04:53:46.716811+0000","last_became_peered":"2021-02-10T04:53:46.716811+0000","last_unstale":"2021-02-10T04:59:17.964182+0000","last_undegraded":"2021-02-10T04:59:17.964182+0000","last_fullsized":"2021-02-10T04:59:17.964182+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:45.813676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.813676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"1338","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.975998+0000","last_change":"2021-02-10T04:53:38.363899+0000","last_active":"2021-02-10T04:59:20.975998+0000","last_peered":"2021-02-10T04:59:20.975998+0000","last_clean":"2021-02-10T04:59:20.975998+0000","last_became_active":"2021-02-10T04:50:59.539256+0000","last_became_peered":"2021-02-10T04:50:59.539256+0000","last_unstale":"2021-02-10T04:59:20.975998+0000","last_undegraded":"2021-02-10T04:59:20.975998+0000","last_fullsized":"2021-02-10T04:59:20.975998+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:38.363825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:09.433793+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.363825+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"1261'84","reported_seq":"1501","reported_epoch":"1348","state":"active+remapped+backfill_toofull","last_fresh":"2021-02-10T04:59:20.972775+0000","last_change":"2021-02-10T04:59:16.970178+0000","last_active":"2021-02-10T04:59:20.972775+0000","last_peered":"2021-02-10T04:59:20.972775+0000","last_clean":"2021-02-10T04:59:13.933368+0000","last_became_active":"2021-02-10T04:59:16.964278+0000","last_became_peered":"2021-02-10T04:59:16.964278+0000","last_unstale":"2021-02-10T04:59:20.972775+0000","last_undegraded":"2021-02-10T04:59:20.972775+0000","last_fullsized":"2021-02-10T04:59:20.972775+0000","mapping_epoch":1344,"log_start":"1132'66","ondisk_log_start":"1132'66","created":1341,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":5,"last_scrub":"1132'66","last_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_deep_scrub":"1132'66","last_deep_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.430399+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":808,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":24,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":93,"num_read_kb":75,"num_write":54,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":519,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[5,6,7],"avail_no_missing":["5","6","7"],"object_location_counts":[{"shards":"5,6,7","objects":12}],"blocked_by":[],"up_primary":4,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1b","version":"1003'1689","reported_seq":"4685","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.961748+0000","last_change":"2021-02-10T04:59:08.659989+0000","last_active":"2021-02-10T04:59:17.961748+0000","last_peered":"2021-02-10T04:59:17.961748+0000","last_clean":"2021-02-10T04:59:17.961748+0000","last_became_active":"2021-02-10T04:58:56.970049+0000","last_became_peered":"2021-02-10T04:58:56.970049+0000","last_unstale":"2021-02-10T04:59:17.961748+0000","last_undegraded":"2021-02-10T04:59:17.961748+0000","last_fullsized":"2021-02-10T04:59:17.961748+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:53:10.075879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.075879+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27159495,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"1347","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.634719+0000","last_change":"2021-02-10T04:58:56.966215+0000","last_active":"2021-02-10T04:59:18.634719+0000","last_peered":"2021-02-10T04:59:18.634719+0000","last_clean":"2021-02-10T04:59:18.634719+0000","last_became_active":"2021-02-10T04:58:56.965975+0000","last_became_peered":"2021-02-10T04:58:56.965975+0000","last_unstale":"2021-02-10T04:59:18.634719+0000","last_undegraded":"2021-02-10T04:59:18.634719+0000","last_fullsized":"2021-02-10T04:59:18.634719+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:33.543559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:46:33.543559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"1008","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.975837+0000","last_change":"2021-02-10T04:53:04.354508+0000","last_active":"2021-02-10T04:59:20.975837+0000","last_peered":"2021-02-10T04:59:20.975837+0000","last_clean":"2021-02-10T04:59:20.975837+0000","last_became_active":"2021-02-10T04:33:05.169159+0000","last_became_peered":"2021-02-10T04:33:05.169159+0000","last_unstale":"2021-02-10T04:59:20.975837+0000","last_undegraded":"2021-02-10T04:59:20.975837+0000","last_fullsized":"2021-02-10T04:59:20.975837+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:04.354409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:04.354409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"1401","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.634397+0000","last_change":"2021-02-10T04:52:36.777554+0000","last_active":"2021-02-10T04:59:18.634397+0000","last_peered":"2021-02-10T04:59:18.634397+0000","last_clean":"2021-02-10T04:59:18.634397+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:59:18.634397+0000","last_undegraded":"2021-02-10T04:59:18.634397+0000","last_fullsized":"2021-02-10T04:59:18.634397+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:36.777485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:52:36.777485+0000","last_clean_scrub_stamp":"2021-02-10T04:52:36.777485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"1261'86","reported_seq":"1459","reported_epoch":"1347","state":"active+remapped+backfill_toofull","last_fresh":"2021-02-10T04:59:18.633148+0000","last_change":"2021-02-10T04:59:16.969345+0000","last_active":"2021-02-10T04:59:18.633148+0000","last_peered":"2021-02-10T04:59:18.633148+0000","last_clean":"2021-02-10T04:55:57.156886+0000","last_became_active":"2021-02-10T04:59:16.967624+0000","last_became_peered":"2021-02-10T04:59:16.967624+0000","last_unstale":"2021-02-10T04:59:18.633148+0000","last_undegraded":"2021-02-10T04:59:18.633148+0000","last_fullsized":"2021-02-10T04:59:18.633148+0000","mapping_epoch":1344,"log_start":"1049'68","ondisk_log_start":"1049'68","created":1341,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":5,"last_scrub":"1048'39","last_scrub_stamp":"2021-02-10T04:53:38.085565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.085565+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":791,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":26,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":113,"num_read_kb":93,"num_write":64,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1582,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,1,4],"avail_no_missing":["3","1","4"],"object_location_counts":[{"shards":"1,3,4","objects":13}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"1001'1528","reported_seq":"5958","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.634166+0000","last_change":"2021-02-10T04:52:31.086406+0000","last_active":"2021-02-10T04:59:18.634166+0000","last_peered":"2021-02-10T04:59:18.634166+0000","last_clean":"2021-02-10T04:59:18.634166+0000","last_became_active":"2021-02-10T04:52:31.085959+0000","last_became_peered":"2021-02-10T04:52:31.085959+0000","last_unstale":"2021-02-10T04:59:18.634166+0000","last_undegraded":"2021-02-10T04:59:18.634166+0000","last_fullsized":"2021-02-10T04:59:18.634166+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46944136,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":358,"num_read_kb":351935,"num_write":286,"num_write_kb":111355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":21311893,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"1222","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.963238+0000","last_change":"2021-02-10T04:59:08.656402+0000","last_active":"2021-02-10T04:59:17.963238+0000","last_peered":"2021-02-10T04:59:17.963238+0000","last_clean":"2021-02-10T04:59:17.963238+0000","last_became_active":"2021-02-10T04:58:56.855338+0000","last_became_peered":"2021-02-10T04:58:56.855338+0000","last_unstale":"2021-02-10T04:59:17.963238+0000","last_undegraded":"2021-02-10T04:59:17.963238+0000","last_fullsized":"2021-02-10T04:59:17.963238+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:17.919745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:17.919745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"633","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.633059+0000","last_change":"2021-02-10T04:53:07.092032+0000","last_active":"2021-02-10T04:59:18.633059+0000","last_peered":"2021-02-10T04:59:18.633059+0000","last_clean":"2021-02-10T04:59:18.633059+0000","last_became_active":"2021-02-10T04:50:59.572077+0000","last_became_peered":"2021-02-10T04:50:59.572077+0000","last_unstale":"2021-02-10T04:59:18.633059+0000","last_undegraded":"2021-02-10T04:59:18.633059+0000","last_fullsized":"2021-02-10T04:59:18.633059+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.091949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:19.755718+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.091949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"1310","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.634947+0000","last_change":"2021-02-10T04:51:59.838856+0000","last_active":"2021-02-10T04:59:18.634947+0000","last_peered":"2021-02-10T04:59:18.634947+0000","last_clean":"2021-02-10T04:59:18.634947+0000","last_became_active":"2021-02-10T04:50:59.563156+0000","last_became_peered":"2021-02-10T04:50:59.563156+0000","last_unstale":"2021-02-10T04:59:18.634947+0000","last_undegraded":"2021-02-10T04:59:18.634947+0000","last_fullsized":"2021-02-10T04:59:18.634947+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.838790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.838790+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":"1044","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.976626+0000","last_change":"2021-02-10T04:52:30.405170+0000","last_active":"2021-02-10T04:59:20.976626+0000","last_peered":"2021-02-10T04:59:20.976626+0000","last_clean":"2021-02-10T04:59:20.976626+0000","last_became_active":"2021-02-10T04:51:36.852058+0000","last_became_peered":"2021-02-10T04:51:36.852058+0000","last_unstale":"2021-02-10T04:59:20.976626+0000","last_undegraded":"2021-02-10T04:59:20.976626+0000","last_fullsized":"2021-02-10T04:59:20.976626+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.405106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:20.635536+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.405106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.20","version":"0'0","reported_seq":"280","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.963078+0000","last_change":"2021-02-10T04:53:46.739888+0000","last_active":"2021-02-10T04:59:17.963078+0000","last_peered":"2021-02-10T04:59:17.963078+0000","last_clean":"2021-02-10T04:59:17.963078+0000","last_became_active":"2021-02-10T04:53:46.739532+0000","last_became_peered":"2021-02-10T04:53:46.739532+0000","last_unstale":"2021-02-10T04:59:17.963078+0000","last_undegraded":"2021-02-10T04:59:17.963078+0000","last_fullsized":"2021-02-10T04:59:17.963078+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:56.449085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:41.375276+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.449085+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":"1276","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.974621+0000","last_change":"2021-02-10T04:53:35.858634+0000","last_active":"2021-02-10T04:59:20.974621+0000","last_peered":"2021-02-10T04:59:20.974621+0000","last_clean":"2021-02-10T04:59:20.974621+0000","last_became_active":"2021-02-10T04:51:36.869785+0000","last_became_peered":"2021-02-10T04:51:36.869785+0000","last_unstale":"2021-02-10T04:59:20.974621+0000","last_undegraded":"2021-02-10T04:59:20.974621+0000","last_fullsized":"2021-02-10T04:59:20.974621+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.858557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.858557+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":"274","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.634825+0000","last_change":"2021-02-10T04:58:56.879602+0000","last_active":"2021-02-10T04:59:18.634825+0000","last_peered":"2021-02-10T04:59:18.634825+0000","last_clean":"2021-02-10T04:59:18.634825+0000","last_became_active":"2021-02-10T04:58:56.879153+0000","last_became_peered":"2021-02-10T04:58:56.879153+0000","last_unstale":"2021-02-10T04:59:18.634825+0000","last_undegraded":"2021-02-10T04:59:18.634825+0000","last_fullsized":"2021-02-10T04:59:18.634825+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:55:23.586401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:55:23.586401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":"259","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.637994+0000","last_change":"2021-02-10T04:58:56.966218+0000","last_active":"2021-02-10T04:59:18.637994+0000","last_peered":"2021-02-10T04:59:18.637994+0000","last_clean":"2021-02-10T04:59:18.637994+0000","last_became_active":"2021-02-10T04:58:56.965861+0000","last_became_peered":"2021-02-10T04:58:56.965861+0000","last_unstale":"2021-02-10T04:59:18.637994+0000","last_undegraded":"2021-02-10T04:59:18.637994+0000","last_fullsized":"2021-02-10T04:59:18.637994+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.706372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.706372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":"1358","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.974267+0000","last_change":"2021-02-10T04:51:57.287452+0000","last_active":"2021-02-10T04:59:20.974267+0000","last_peered":"2021-02-10T04:59:20.974267+0000","last_clean":"2021-02-10T04:59:20.974267+0000","last_became_active":"2021-02-10T04:50:59.566827+0000","last_became_peered":"2021-02-10T04:50:59.566827+0000","last_unstale":"2021-02-10T04:59:20.974267+0000","last_undegraded":"2021-02-10T04:59:20.974267+0000","last_fullsized":"2021-02-10T04:59:20.974267+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.287363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.287363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":"347","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.975644+0000","last_change":"2021-02-10T04:58:56.867715+0000","last_active":"2021-02-10T04:59:20.975644+0000","last_peered":"2021-02-10T04:59:20.975644+0000","last_clean":"2021-02-10T04:59:20.975644+0000","last_became_active":"2021-02-10T04:58:56.867324+0000","last_became_peered":"2021-02-10T04:58:56.867324+0000","last_unstale":"2021-02-10T04:59:20.975644+0000","last_undegraded":"2021-02-10T04:59:20.975644+0000","last_fullsized":"2021-02-10T04:59:20.975644+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:10.308282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:10.308282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":"1270","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.635138+0000","last_change":"2021-02-10T04:58:56.966618+0000","last_active":"2021-02-10T04:59:18.635138+0000","last_peered":"2021-02-10T04:59:18.635138+0000","last_clean":"2021-02-10T04:59:18.635138+0000","last_became_active":"2021-02-10T04:58:56.965458+0000","last_became_peered":"2021-02-10T04:58:56.965458+0000","last_unstale":"2021-02-10T04:59:18.635138+0000","last_undegraded":"2021-02-10T04:59:18.635138+0000","last_fullsized":"2021-02-10T04:59:18.635138+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:09.121298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.121298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.21","version":"293'77","reported_seq":"918","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.634577+0000","last_change":"2021-02-10T04:52:28.062923+0000","last_active":"2021-02-10T04:59:18.634577+0000","last_peered":"2021-02-10T04:59:18.634577+0000","last_clean":"2021-02-10T04:59:18.634577+0000","last_became_active":"2021-02-10T04:52:28.062463+0000","last_became_peered":"2021-02-10T04:52:28.062463+0000","last_unstale":"2021-02-10T04:59:18.634577+0000","last_undegraded":"2021-02-10T04:59:18.634577+0000","last_fullsized":"2021-02-10T04:59:18.634577+0000","mapping_epoch":1143,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1144,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1886586,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":15163,"num_write":9,"num_write_kb":3316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":"244","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.632320+0000","last_change":"2021-02-10T04:52:32.124955+0000","last_active":"2021-02-10T04:59:18.632320+0000","last_peered":"2021-02-10T04:59:18.632320+0000","last_clean":"2021-02-10T04:59:18.632320+0000","last_became_active":"2021-02-10T04:51:36.867061+0000","last_became_peered":"2021-02-10T04:51:36.867061+0000","last_unstale":"2021-02-10T04:59:18.632320+0000","last_undegraded":"2021-02-10T04:59:18.632320+0000","last_fullsized":"2021-02-10T04:59:18.632320+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:32.124901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:08.271977+0000","last_clean_scrub_stamp":"2021-02-10T04:52:32.124901+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":"1287","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.974462+0000","last_change":"2021-02-10T04:51:07.182775+0000","last_active":"2021-02-10T04:59:20.974462+0000","last_peered":"2021-02-10T04:59:20.974462+0000","last_clean":"2021-02-10T04:59:20.974462+0000","last_became_active":"2021-02-10T04:28:23.126368+0000","last_became_peered":"2021-02-10T04:28:23.126368+0000","last_unstale":"2021-02-10T04:59:20.974462+0000","last_undegraded":"2021-02-10T04:59:20.974462+0000","last_fullsized":"2021-02-10T04:59:20.974462+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.182678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.182678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.20","version":"1000'1071","reported_seq":"4692","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.632107+0000","last_change":"2021-02-10T04:53:03.915465+0000","last_active":"2021-02-10T04:59:18.632107+0000","last_peered":"2021-02-10T04:59:18.632107+0000","last_clean":"2021-02-10T04:59:18.632107+0000","last_became_active":"2021-02-10T04:52:31.088690+0000","last_became_peered":"2021-02-10T04:52:31.088690+0000","last_unstale":"2021-02-10T04:59:18.632107+0000","last_undegraded":"2021-02-10T04:59:18.632107+0000","last_fullsized":"2021-02-10T04:59:18.632107+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:03.915403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.915403+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16398893,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":"869","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.962753+0000","last_change":"2021-02-10T04:53:46.721599+0000","last_active":"2021-02-10T04:59:17.962753+0000","last_peered":"2021-02-10T04:59:17.962753+0000","last_clean":"2021-02-10T04:59:17.962753+0000","last_became_active":"2021-02-10T04:53:46.721340+0000","last_became_peered":"2021-02-10T04:53:46.721340+0000","last_unstale":"2021-02-10T04:59:17.962753+0000","last_undegraded":"2021-02-10T04:59:17.962753+0000","last_fullsized":"2021-02-10T04:59:17.962753+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:28.092288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:07.767250+0000","last_clean_scrub_stamp":"2021-02-10T04:52:28.092288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.25","version":"53'2","reported_seq":"1302","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.636601+0000","last_change":"2021-02-10T04:53:36.103026+0000","last_active":"2021-02-10T04:59:18.636601+0000","last_peered":"2021-02-10T04:59:18.636601+0000","last_clean":"2021-02-10T04:59:18.636601+0000","last_became_active":"2021-02-10T04:28:23.467770+0000","last_became_peered":"2021-02-10T04:28:23.467770+0000","last_unstale":"2021-02-10T04:59:18.636601+0000","last_undegraded":"2021-02-10T04:59:18.636601+0000","last_fullsized":"2021-02-10T04:59:18.636601+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:53:36.102928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:36.102928+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.23","version":"1002'1657","reported_seq":"5219","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.974067+0000","last_change":"2021-02-10T04:52:31.093874+0000","last_active":"2021-02-10T04:59:20.974067+0000","last_peered":"2021-02-10T04:59:20.974067+0000","last_clean":"2021-02-10T04:59:20.974067+0000","last_became_active":"2021-02-10T04:52:31.092996+0000","last_became_peered":"2021-02-10T04:52:31.092996+0000","last_unstale":"2021-02-10T04:59:20.974067+0000","last_undegraded":"2021-02-10T04:59:20.974067+0000","last_fullsized":"2021-02-10T04:59:20.974067+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21396533,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":212,"num_read_kb":218837,"num_write":159,"num_write_kb":59399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":20479176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":"1394","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.973883+0000","last_change":"2021-02-10T04:53:12.951612+0000","last_active":"2021-02-10T04:59:20.973883+0000","last_peered":"2021-02-10T04:59:20.973883+0000","last_clean":"2021-02-10T04:59:20.973883+0000","last_became_active":"2021-02-10T04:50:59.566548+0000","last_became_peered":"2021-02-10T04:50:59.566548+0000","last_unstale":"2021-02-10T04:59:20.973883+0000","last_undegraded":"2021-02-10T04:59:20.973883+0000","last_fullsized":"2021-02-10T04:59:20.973883+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.951528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.951528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.26","version":"53'1","reported_seq":"183","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.964637+0000","last_change":"2021-02-10T04:53:46.722636+0000","last_active":"2021-02-10T04:59:17.964637+0000","last_peered":"2021-02-10T04:59:17.964637+0000","last_clean":"2021-02-10T04:59:17.964637+0000","last_became_active":"2021-02-10T04:53:46.722227+0000","last_became_peered":"2021-02-10T04:53:46.722227+0000","last_unstale":"2021-02-10T04:59:17.964637+0000","last_undegraded":"2021-02-10T04:59:17.964637+0000","last_fullsized":"2021-02-10T04:59:17.964637+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:53:03.933910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.933910+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":423,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.22","version":"998'1009","reported_seq":"4709","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.634252+0000","last_change":"2021-02-10T04:52:31.086563+0000","last_active":"2021-02-10T04:59:18.634252+0000","last_peered":"2021-02-10T04:59:18.634252+0000","last_clean":"2021-02-10T04:59:18.634252+0000","last_became_active":"2021-02-10T04:52:31.085288+0000","last_became_peered":"2021-02-10T04:52:31.085288+0000","last_unstale":"2021-02-10T04:59:18.634252+0000","last_undegraded":"2021-02-10T04:59:18.634252+0000","last_fullsized":"2021-02-10T04:59:18.634252+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12355207,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":120,"num_read_kb":124702,"num_write":77,"num_write_kb":25052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4145029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":"1276","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.973986+0000","last_change":"2021-02-10T04:53:10.958668+0000","last_active":"2021-02-10T04:59:20.973986+0000","last_peered":"2021-02-10T04:59:20.973986+0000","last_clean":"2021-02-10T04:59:20.973986+0000","last_became_active":"2021-02-10T04:45:49.552026+0000","last_became_peered":"2021-02-10T04:45:49.552026+0000","last_unstale":"2021-02-10T04:59:20.973986+0000","last_undegraded":"2021-02-10T04:59:20.973986+0000","last_fullsized":"2021-02-10T04:59:20.973986+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:10.958579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.958579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":"288","reported_epoch":"1346","state":"active+clean","last_fresh":"2021-02-10T04:59:17.965495+0000","last_change":"2021-02-10T04:53:46.725941+0000","last_active":"2021-02-10T04:59:17.965495+0000","last_peered":"2021-02-10T04:59:17.965495+0000","last_clean":"2021-02-10T04:59:17.965495+0000","last_became_active":"2021-02-10T04:53:46.725587+0000","last_became_peered":"2021-02-10T04:53:46.725587+0000","last_unstale":"2021-02-10T04:59:17.965495+0000","last_undegraded":"2021-02-10T04:59:17.965495+0000","last_fullsized":"2021-02-10T04:59:17.965495+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:41.758684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.758684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":"1284","reported_epoch":"1347","state":"active+clean","last_fresh":"2021-02-10T04:59:18.633249+0000","last_change":"2021-02-10T04:55:39.599193+0000","last_active":"2021-02-10T04:59:18.633249+0000","last_peered":"2021-02-10T04:59:18.633249+0000","last_clean":"2021-02-10T04:59:18.633249+0000","last_became_active":"2021-02-10T04:55:39.598816+0000","last_became_peered":"2021-02-10T04:55:39.598816+0000","last_unstale":"2021-02-10T04:59:18.633249+0000","last_undegraded":"2021-02-10T04:59:18.633249+0000","last_fullsized":"2021-02-10T04:59:18.633249+0000","mapping_epoch":1248,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:11.139602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.139602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":"1302","reported_epoch":"1348","state":"active+clean","last_fresh":"2021-02-10T04:59:20.973378+0000","last_change":"2021-02-10T04:52:24.072005+0000","last_active":"2021-02-10T04:59:20.973378+0000","last_peered":"2021-02-10T04:59:20.973378+0000","last_clean":"2021-02-10T04:59:20.973378+0000","last_became_active":"2021-02-10T04:50:59.527945+0000","last_became_peered":"2021-02-10T04:50:59.527945+0000","last_unstale":"2021-02-10T04:59:20.973378+0000","last_undegraded":"2021-02-10T04:59:20.973378+0000","last_fullsized":"2021-02-10T04:59:20.973378+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:24.071915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_clean_scrub_stamp":"2021-02-10T04:52:24.071915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":936229565,"num_objects":377,"num_object_clones":326,"num_object_copies":1131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":377,"num_whiteouts":10,"num_read":4055,"num_read_kb":4149582,"num_write":2946,"num_write_kb":1137016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":575,"num_bytes_recovered":289511227,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2029219840,"data_stored":2597935849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1639,"internal_metadata":0},"log_size":34722,"ondisk_log_size":34722,"up":108,"acting":108,"num_store_stats":8},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":2368,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":18,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1139,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":5835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31,"ondisk_log_size":31,"up":126,"acting":126,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":13,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":882,"num_read_kb":877,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20,"ondisk_log_size":20,"up":39,"acting":39,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":188,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":2711,"num_read_kb":2184,"num_write":1592,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":279,"num_bytes_recovered":18738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1033,"ondisk_log_size":1033,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":126,"acting":126,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365207,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":197232,"kb_used_data":149864,"kb_used_omap":0,"kb_used_meta":47360,"kb_avail":93547920,"statfs":{"total":95995035648,"available":95793070080,"internally_reserved":0,"allocated":153460736,"data_stored":202760776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48496640},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:57:20 2021","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.89500000000000002,"15min":0.92800000000000005},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.98999999999999999,"5min":4.4340000000000002,"15min":4.4340000000000002},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.78000000000000003}]},{"osd":1,"last update":"Wed Feb 10 04:58:48 2021","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.02,"15min":0.96099999999999997},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":15.253,"5min":15.253,"15min":15.253},"last":0.60599999999999998},{"interface":"front","average":{"1min":1.669,"5min":1.105,"15min":1.022},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.45500000000000002},"max":{"1min":18.713000000000001,"5min":18.713000000000001,"15min":18.713000000000001},"last":1.165}]},{"osd":2,"last update":"Wed Feb 10 04:58:48 2021","interfaces":[{"interface":"back","average":{"1min":3.3239999999999998,"5min":2.8490000000000002,"15min":1.7070000000000001},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":40.536999999999999,"5min":94.391999999999996,"15min":94.391999999999996},"last":1.278},{"interface":"front","average":{"1min":3.2959999999999998,"5min":2.7839999999999998,"15min":1.7110000000000001},"min":{"1min":0.54300000000000004,"5min":0.496,"15min":0.496},"max":{"1min":40.429000000000002,"5min":94.269999999999996,"15min":94.269999999999996},"last":0.84099999999999997}]},{"osd":3,"last update":"Wed Feb 10 04:59:03 2021","interfaces":[{"interface":"back","average":{"1min":1.657,"5min":1.2050000000000001,"15min":1.264},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.58199999999999996},"max":{"1min":15.148,"5min":15.148,"15min":15.148},"last":1.028},{"interface":"front","average":{"1min":1.891,"5min":1.363,"15min":1.3839999999999999},"min":{"1min":0.71899999999999997,"5min":0.629,"15min":0.629},"max":{"1min":18.611000000000001,"5min":18.611000000000001,"15min":18.611000000000001},"last":1.375}]},{"osd":4,"last update":"Wed Feb 10 04:58:48 2021","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.841,"15min":1.5429999999999999},"min":{"1min":0.67000000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":9.9659999999999993,"5min":72.662999999999997,"15min":72.662999999999997},"last":0.98599999999999999},{"interface":"front","average":{"1min":2.1949999999999998,"5min":2.1309999999999998,"15min":1.6100000000000001},"min":{"1min":0.76500000000000001,"5min":0.73999999999999999,"15min":0.69099999999999995},"max":{"1min":20.542999999999999,"5min":73.564999999999998,"15min":73.564999999999998},"last":0.90000000000000002}]},{"osd":5,"last update":"Wed Feb 10 04:58:48 2021","interfaces":[{"interface":"back","average":{"1min":1.7929999999999999,"5min":1.4910000000000001,"15min":1.5209999999999999},"min":{"1min":0.74099999999999999,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":11.627000000000001,"5min":11.627000000000001,"15min":14.972},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.8919999999999999,"5min":1.377,"15min":1.595},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.72699999999999998},"max":{"1min":13.172000000000001,"5min":13.172000000000001,"15min":15.218999999999999},"last":1.123}]},{"osd":6,"last update":"Wed Feb 10 04:58:48 2021","interfaces":[{"interface":"back","average":{"1min":1.7769999999999999,"5min":1.3660000000000001,"15min":1.6180000000000001},"min":{"1min":0.84399999999999997,"5min":0.78200000000000003,"15min":0.72699999999999998},"max":{"1min":10.286,"5min":10.286,"15min":16.050000000000001},"last":1.2989999999999999},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.5389999999999999,"15min":1.6579999999999999},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.876},"max":{"1min":17.460000000000001,"5min":17.460000000000001,"15min":17.460000000000001},"last":1.145}]}]},{"osd":6,"up_from":43,"seq":184683594140,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":393804,"kb_used_data":342980,"kb_used_omap":0,"kb_used_meta":50816,"kb_avail":93351348,"statfs":{"total":95995035648,"available":95591780352,"internally_reserved":0,"allocated":351211520,"data_stored":443739904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52035584},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:57:18 2021","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.88100000000000001,"15min":0.88800000000000001},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.6040000000000001,"5min":1.6599999999999999,"15min":1.6599999999999999},"last":0.85799999999999998}]},{"osd":1,"last update":"Wed Feb 10 04:58:28 2021","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.91600000000000004,"15min":1.0249999999999999},"min":{"1min":0.51800000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.8169999999999999,"5min":1.8169999999999999,"15min":13.18},"last":0.61499999999999999},{"interface":"front","average":{"1min":1.1479999999999999,"5min":0.92700000000000005,"15min":1.077},"min":{"1min":0.61099999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":6.5270000000000001,"5min":6.5270000000000001,"15min":11.041},"last":1.1759999999999999}]},{"osd":2,"last update":"Wed Feb 10 04:58:28 2021","interfaces":[{"interface":"back","average":{"1min":3.2959999999999998,"5min":2.157,"15min":2.0630000000000002},"min":{"1min":0.56299999999999994,"5min":0.503,"15min":0.503},"max":{"1min":45.514000000000003,"5min":69.653000000000006,"15min":163.81700000000001},"last":0.83799999999999997},{"interface":"front","average":{"1min":3.7450000000000001,"5min":2.3359999999999999,"15min":2.1360000000000001},"min":{"1min":0.75700000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":45.347999999999999,"5min":69.561000000000007,"15min":164.56800000000001},"last":0.97799999999999998}]},{"osd":3,"last update":"Wed Feb 10 04:59:16 2021","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.1559999999999999,"15min":1.1779999999999999},"min":{"1min":0.85099999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.9039999999999999,"5min":1.944,"15min":1.944},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.2070000000000001,"15min":1.151},"min":{"1min":0.81699999999999995,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.9370000000000001,"5min":6.6130000000000004,"15min":6.6130000000000004},"last":1.0649999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:58:28 2021","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.0980000000000001,"15min":1.3049999999999999},"min":{"1min":0.66300000000000003,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.615,"5min":2.1120000000000001,"15min":7.1459999999999999},"last":0.95099999999999996},{"interface":"front","average":{"1min":1.0149999999999999,"5min":0.95799999999999996,"15min":1.0860000000000001},"min":{"1min":0.63900000000000001,"5min":0.57399999999999995,"15min":0.56000000000000005},"max":{"1min":1.5009999999999999,"5min":1.958,"15min":11.997999999999999},"last":1.0189999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:58:28 2021","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.3480000000000001,"15min":1.6419999999999999},"min":{"1min":0.69699999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.7629999999999999,"5min":22.663,"15min":81.540000000000006},"last":1.4510000000000001},{"interface":"front","average":{"1min":1.353,"5min":1.6000000000000001,"15min":1.9239999999999999},"min":{"1min":0.91400000000000003,"5min":0.85699999999999998,"15min":0.78700000000000003},"max":{"1min":1.9259999999999999,"5min":22.933,"15min":81.492999999999995},"last":1.228}]},{"osd":7,"last update":"Wed Feb 10 04:58:50 2021","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.121,"15min":1.258},"min":{"1min":0.76100000000000001,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.645,"5min":2.073,"15min":5.1820000000000004},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.3939999999999999,"15min":1.5169999999999999},"min":{"1min":0.98299999999999998,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":2.0659999999999998,"5min":2.2970000000000002,"15min":5.4109999999999996},"last":1.2889999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575277,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":263348,"kb_used_data":206956,"kb_used_omap":0,"kb_used_meta":56384,"kb_avail":93481804,"statfs":{"total":95995035648,"available":95725367296,"internally_reserved":0,"allocated":211922944,"data_stored":266227848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":57737216},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:57:18 2021","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.69799999999999995,"15min":0.68899999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":2.3439999999999999,"5min":2.3439999999999999,"15min":2.3439999999999999},"last":0.622},{"interface":"front","average":{"1min":0.996,"5min":0.93899999999999995,"15min":0.92900000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":2.7589999999999999,"5min":2.7589999999999999,"15min":2.7589999999999999},"last":0.67400000000000004}]},{"osd":2,"last update":"Wed Feb 10 04:58:22 2021","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":5.4390000000000001,"15min":3.2810000000000001},"min":{"1min":0.54200000000000004,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.3540000000000001,"5min":162.61199999999999,"15min":162.61199999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.83699999999999997,"5min":5.5789999999999997,"15min":3.371},"min":{"1min":0.57699999999999996,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.504,"5min":162.762,"15min":162.762},"last":0.88600000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:59:08 2021","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.83399999999999996,"15min":0.88500000000000001},"min":{"1min":0.495,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.0629999999999999,"5min":2.8919999999999999,"15min":3.077},"last":1.1859999999999999},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.84699999999999998,"15min":0.95599999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42199999999999999},"max":{"1min":1.4159999999999999,"5min":3.1890000000000001,"15min":3.1890000000000001},"last":0.84499999999999997}]},{"osd":4,"last update":"Wed Feb 10 04:59:05 2021","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":1.919,"15min":1.393},"min":{"1min":0.58899999999999997,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.837,"5min":100.071,"15min":100.071},"last":1.284},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.9370000000000001,"15min":1.4239999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.53500000000000003},"max":{"1min":1.6990000000000001,"5min":100.43899999999999,"15min":100.43899999999999},"last":1.056}]},{"osd":5,"last update":"Wed Feb 10 04:59:08 2021","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.1579999999999999,"15min":1.1599999999999999},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.603,"5min":3.0219999999999998,"15min":5.0300000000000002},"last":1.351},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.202,"15min":1.3120000000000001},"min":{"1min":0.79900000000000004,"5min":0.70399999999999996,"15min":0.68600000000000005},"max":{"1min":1.9319999999999999,"5min":6.032,"15min":6.032},"last":1.071}]},{"osd":6,"last update":"Wed Feb 10 04:58:22 2021","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.3799999999999999,"15min":1.4670000000000001},"min":{"1min":0.69099999999999995,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.5669999999999999,"5min":30.997,"15min":30.997},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.254,"5min":1.46,"15min":1.468},"min":{"1min":0.91200000000000003,"5min":0.64200000000000002,"15min":0.60499999999999998},"max":{"1min":1.8460000000000001,"5min":31.138000000000002,"15min":31.138000000000002},"last":1.411}]},{"osd":7,"last update":"Wed Feb 10 04:58:30 2021","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.2410000000000001,"15min":1.347},"min":{"1min":0.85099999999999998,"5min":0.72299999999999998,"15min":0.628},"max":{"1min":1.663,"5min":6.2629999999999999,"15min":6.2629999999999999},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.2230000000000001,"15min":1.353},"min":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.78700000000000003},"max":{"1min":1.522,"5min":3.4129999999999998,"15min":3.754},"last":1.488}]}]},{"osd":0,"up_from":1332,"seq":5720896438278,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":367608,"kb_used_data":357296,"kb_used_omap":324,"kb_used_meta":9979,"kb_avail":93377544,"statfs":{"total":95995035648,"available":95618605056,"internally_reserved":0,"allocated":365871104,"data_stored":480618051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":332393,"internal_metadata":10218903},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.413}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2490000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8029999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2649999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411755,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":359944,"kb_used_data":319104,"kb_used_omap":0,"kb_used_meta":40832,"kb_avail":93385208,"statfs":{"total":95995035648,"available":95626452992,"internally_reserved":0,"allocated":326762496,"data_stored":375247986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41811968},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:57:18 2021","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.67200000000000004,"15min":0.67000000000000004},"min":{"1min":0.50800000000000001,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.94799999999999995,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.93799999999999994,"15min":0.93999999999999995},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.9950000000000001,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":0.53400000000000003}]},{"osd":1,"last update":"Wed Feb 10 04:59:03 2021","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.82699999999999996,"15min":0.92000000000000004},"min":{"1min":0.41999999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.84499999999999997,"5min":17.041,"15min":17.041},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.69899999999999995,"5min":1.0289999999999999,"15min":0.92200000000000004},"min":{"1min":0.46300000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.825,"5min":21.172000000000001,"15min":21.172000000000001},"last":0.83599999999999997}]},{"osd":3,"last update":"Wed Feb 10 04:58:21 2021","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":1.018,"15min":1.0940000000000001},"min":{"1min":0.53100000000000003,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.631,"5min":18.904,"15min":18.904},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.073,"5min":1.2669999999999999,"15min":1.2909999999999999},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.742,"5min":19.241,"15min":19.241},"last":0.93400000000000005}]},{"osd":4,"last update":"Wed Feb 10 04:58:38 2021","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.2110000000000001,"15min":1.248},"min":{"1min":0.621,"5min":0.621,"15min":0.61299999999999999},"max":{"1min":1.851,"5min":9.2029999999999994,"15min":9.2029999999999994},"last":0.86099999999999999},{"interface":"front","average":{"1min":1.018,"5min":1.3220000000000001,"15min":1.1739999999999999},"min":{"1min":0.58299999999999996,"5min":0.503,"15min":0.503},"max":{"1min":1.716,"5min":24.373999999999999,"15min":24.373999999999999},"last":1.4139999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:59:03 2021","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.665,"15min":1.7090000000000001},"min":{"1min":0.68300000000000005,"5min":0.66200000000000003,"15min":0.63500000000000001},"max":{"1min":1.6599999999999999,"5min":39.911999999999999,"15min":39.911999999999999},"last":0.999},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.379,"15min":1.4410000000000001},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.57599999999999996},"max":{"1min":1.3480000000000001,"5min":33.539999999999999,"15min":33.539999999999999},"last":1.1040000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:58:33 2021","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0620000000000001,"15min":1.341},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":2.6280000000000001,"5min":3.1659999999999999,"15min":20.007000000000001},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.298,"5min":1.4570000000000001,"15min":1.5900000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":2.6579999999999999,"5min":16.135000000000002,"15min":19.417999999999999},"last":1.0629999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:58:38 2021","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.645,"15min":1.609},"min":{"1min":0.84999999999999998,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":2.1389999999999998,"5min":24.405000000000001,"15min":24.405000000000001},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.216,"15min":1.2549999999999999},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.829,"5min":3.6640000000000001,"15min":4.9189999999999996},"last":1.2509999999999999}]}]},{"osd":3,"up_from":1111,"seq":4771708665958,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318312,"kb_used_data":301920,"kb_used_omap":386,"kb_used_meta":15997,"kb_avail":93426840,"statfs":{"total":95995035648,"available":95669084160,"internally_reserved":0,"allocated":309166080,"data_stored":410178653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":396036,"internal_metadata":16381180},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:57:18 2021","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":1.724,"15min":1.8939999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.004,"5min":22.869,"15min":22.869},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.72099999999999997,"5min":1.446,"15min":1.5669999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.98099999999999998,"5min":13.587999999999999,"15min":13.587999999999999},"last":0.73399999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:59:12 2021","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.82699999999999996,"15min":0.71099999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44},"max":{"1min":6.1980000000000004,"5min":6.1980000000000004,"15min":6.1980000000000004},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.1859999999999999,"5min":0.89200000000000002,"15min":0.873},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.45600000000000002},"max":{"1min":7.7709999999999999,"5min":7.7709999999999999,"15min":7.7709999999999999},"last":0.67700000000000005}]},{"osd":2,"last update":"Wed Feb 10 04:59:12 2021","interfaces":[{"interface":"back","average":{"1min":4.5419999999999998,"5min":2.1219999999999999,"15min":1.4930000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":81.138000000000005,"5min":81.138000000000005,"15min":81.138000000000005},"last":0.70499999999999996},{"interface":"front","average":{"1min":4.5590000000000002,"5min":2.3220000000000001,"15min":1.5700000000000001},"min":{"1min":0.54400000000000004,"5min":0.53800000000000003,"15min":0.50800000000000001},"max":{"1min":81.099999999999994,"5min":81.099999999999994,"15min":81.099999999999994},"last":1.014}]},{"osd":4,"last update":"Wed Feb 10 04:59:12 2021","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":1.0009999999999999,"15min":0.97899999999999998},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.5,"5min":2.6589999999999998,"15min":2.9430000000000001},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.327,"5min":1.143,"15min":1.1040000000000001},"min":{"1min":0.63900000000000001,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":7.484,"5min":7.484,"15min":7.484},"last":1.0780000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:59:12 2021","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.2649999999999999,"15min":2.2170000000000001},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":7.6379999999999999,"5min":7.6379999999999999,"15min":52.353000000000002},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.0249999999999999,"15min":2.02},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":4.8730000000000002,"5min":4.8730000000000002,"15min":52.301000000000002},"last":1.486}]},{"osd":6,"last update":"Wed Feb 10 04:59:12 2021","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.1180000000000001,"15min":3.7570000000000001},"min":{"1min":0.68500000000000005,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":5.258,"5min":5.258,"15min":64.376999999999995},"last":1.377},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.2909999999999999,"15min":3.8130000000000002},"min":{"1min":0.88400000000000001,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":64.438999999999993},"last":1.5780000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:59:12 2021","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.1319999999999999,"15min":1.115},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.754},"max":{"1min":5.3819999999999997,"5min":5.3819999999999997,"15min":5.3819999999999997},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.145,"5min":1.1899999999999999,"15min":1.157},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.79900000000000004},"max":{"1min":1.6859999999999999,"5min":2.9569999999999999,"15min":2.9569999999999999},"last":1.268}]}]},{"osd":4,"up_from":31,"seq":133143986595,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":166572,"kb_used_data":121124,"kb_used_omap":0,"kb_used_meta":45440,"kb_avail":93578580,"statfs":{"total":95995035648,"available":95824465920,"internally_reserved":0,"allocated":124030976,"data_stored":156079011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":46530560},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:57:16 2021","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.85799999999999998,"15min":0.86499999999999999},"min":{"1min":0.53700000000000003,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.5449999999999999,"5min":3.637,"15min":3.637},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.878,"15min":0.89000000000000001},"min":{"1min":0.61399999999999999,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.2949999999999999,"5min":4.6829999999999998,"15min":4.6829999999999998},"last":1.1220000000000001}]},{"osd":1,"last update":"Wed Feb 10 04:58:50 2021","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.76200000000000001,"15min":0.85299999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.5409999999999999,"5min":3.2650000000000001,"15min":7.883},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.89400000000000002,"15min":1.0129999999999999},"min":{"1min":0.47999999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.46,"5min":1.6779999999999999,"15min":4.3449999999999998},"last":0.68000000000000005}]},{"osd":2,"last update":"Wed Feb 10 04:58:50 2021","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":1.177,"15min":1.6819999999999999},"min":{"1min":0.60299999999999998,"5min":0.53900000000000003,"15min":0.51000000000000001},"max":{"1min":1.397,"5min":11.776,"15min":120.39},"last":1.4990000000000001},{"interface":"front","average":{"1min":0.83599999999999997,"5min":1.129,"15min":1.6599999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.4339999999999999,"5min":11.795999999999999,"15min":120.583},"last":0.85099999999999998}]},{"osd":3,"last update":"Wed Feb 10 04:59:06 2021","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.1299999999999999,"15min":1.216},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":2.3100000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.6200000000000001},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.2030000000000001,"15min":1.494},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.57399999999999995},"max":{"1min":2.548,"5min":2.548,"15min":26.802},"last":1.778}]},{"osd":5,"last update":"Wed Feb 10 04:59:06 2021","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.04,"15min":1.401},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.495},"max":{"1min":2.5499999999999998,"5min":2.5499999999999998,"15min":35.774000000000001},"last":1.101},{"interface":"front","average":{"1min":1.024,"5min":1.1379999999999999,"15min":1.419},"min":{"1min":0.621,"5min":0.621,"15min":0.60899999999999999},"max":{"1min":2.3170000000000002,"5min":3.8410000000000002,"15min":35.606000000000002},"last":1.2470000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:59:06 2021","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.4830000000000001,"15min":2.3570000000000002},"min":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"max":{"1min":2.6819999999999999,"5min":3.8530000000000002,"15min":201.626},"last":1.903},{"interface":"front","average":{"1min":1.2,"5min":1.3200000000000001,"15min":2.319},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.68100000000000005},"max":{"1min":2.4100000000000001,"5min":4.7910000000000004,"15min":201.577},"last":1.532}]},{"osd":7,"last update":"Wed Feb 10 04:58:37 2021","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.4419999999999999,"15min":1.498},"min":{"1min":0.94499999999999995,"5min":0.878,"15min":0.67100000000000004},"max":{"1min":1.9239999999999999,"5min":4.6200000000000001,"15min":5.8920000000000003},"last":2.0190000000000001},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.25,"15min":1.4339999999999999},"min":{"1min":0.80100000000000005,"5min":0.78700000000000003,"15min":0.63200000000000001},"max":{"1min":1.5669999999999999,"5min":3.423,"15min":5.149},"last":1.7549999999999999}]}]},{"osd":5,"up_from":37,"seq":158913790368,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":257908,"kb_used_data":206892,"kb_used_omap":0,"kb_used_meta":51008,"kb_avail":93487244,"statfs":{"total":95995035648,"available":95730937856,"internally_reserved":0,"allocated":211857408,"data_stored":276152382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52232192},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:57:18 2021","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.1799999999999999,"5min":1.486,"15min":1.486},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.76400000000000001,"15min":0.753},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":1.2769999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:59:06 2021","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.85299999999999998,"15min":0.90400000000000003},"min":{"1min":0.438,"5min":0.435,"15min":0.435},"max":{"1min":1.9790000000000001,"5min":4.242,"15min":6.391},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.82299999999999995,"5min":1.0800000000000001,"15min":1.1339999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.41299999999999998},"max":{"1min":1.3480000000000001,"5min":13.794,"15min":13.794},"last":1.1479999999999999}]},{"osd":2,"last update":"Wed Feb 10 04:59:06 2021","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":3.4569999999999999,"15min":2.1440000000000001},"min":{"1min":0.504,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.748,"5min":116.438,"15min":116.438},"last":68.697000000000003},{"interface":"front","average":{"1min":0.92500000000000004,"5min":3.5009999999999999,"15min":2.206},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.8759999999999999,"5min":116.96899999999999,"15min":116.96899999999999},"last":68.483999999999995}]},{"osd":3,"last update":"Wed Feb 10 04:59:15 2021","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.367,"15min":1.302},"min":{"1min":0.57499999999999996,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.4750000000000001,"5min":13.755000000000001,"15min":13.755000000000001},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.073,"5min":1.1639999999999999,"15min":1.1839999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.64700000000000002},"max":{"1min":2.331,"5min":5.577,"15min":6.5129999999999999},"last":0.88400000000000001}]},{"osd":4,"last update":"Wed Feb 10 04:59:06 2021","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":1.054,"15min":1.246},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.841,"5min":5.298,"15min":9.4890000000000008},"last":1.508},{"interface":"front","average":{"1min":1.016,"5min":1.1899999999999999,"15min":1.319},"min":{"1min":0.61499999999999999,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.327,"5min":4.8289999999999997,"15min":5.6180000000000003},"last":1.3620000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:59:15 2021","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.3440000000000001,"15min":1.7370000000000001},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.7450000000000001,"5min":4.9429999999999996,"15min":38.014000000000003},"last":1.246},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.302,"15min":1.6859999999999999},"min":{"1min":0.80600000000000005,"5min":0.71499999999999997,"15min":0.68799999999999994},"max":{"1min":2.2080000000000002,"5min":5.4660000000000002,"15min":37.896999999999998},"last":1.1659999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:58:42 2021","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.24,"15min":1.4430000000000001},"min":{"1min":0.82299999999999995,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.4020000000000001,"5min":5.4160000000000004,"15min":11.897},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4119999999999999,"15min":1.5309999999999999},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":3.246,"5min":5.0579999999999998,"15min":6.0369999999999999},"last":1.625}]}]}]}} 2021-02-10T04:59:25.829 INFO:tasks.thrashosds.thrasher:Setting 7.5 to [1, 5, 4, 6, 0, 2] 2021-02-10T04:59:25.829 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '7.5', '1', '5', '4', '6', '0', '2'] 2021-02-10T04:59:25.829 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pg-upmap-items 7.5 1 5 4 6 0 2 2021-02-10T04:59:26.052 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:59:26.127 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset nodeep-scrub 2021-02-10T04:59:26.149 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:25 smithi085 conmon[29319]: cluster 2021-02-10T04:59:24.565598+0000 mgr.y (mgr.14140) 1522 : cluster [DBG] pgmap v2472: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 1.5 KiB/s wr, 39 op/s; 188/2205 objects misplaced (8.526%); 412 B/s, 6 objects/s recovering 2021-02-10T04:59:26.149 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:25 smithi085 conmon[29319]: audit 2021-02-10T04:59:24.787684+0000 mgr.y (mgr.14140) 1523 : audit [DBG] from='client.26476 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:59:26.150 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:25 smithi085 conmon[35961]: cluster 2021-02-10T04:59:24.565598+0000 mgr.y (mgr.14140) 1522 : cluster [DBG] pgmap v2472: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 1.5 KiB/s wr, 39 op/s; 188/2205 objects misplaced (8.526%); 412 B/s, 6 objects/s recovering 2021-02-10T04:59:26.150 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:25 smithi085 conmon[35961]: audit 2021-02-10T04:59:24.787684+0000 mgr.y (mgr.14140) 1523 : audit [DBG] from='client.26476 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:59:26.358 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:59:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:25 smithi164 conmon[29288]: cluster 2021-02-10T04:59:24.565598+0000 mgr.y (mgr.14140) 1522 : 2021-02-10T04:59:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:25 smithi164 conmon[29288]: cluster [DBG] pgmap v2472: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 1.5 KiB/s wr, 39 op/s; 188/2205 objects misplaced (8.526%); 412 B/s, 6 objects/s recovering 2021-02-10T04:59:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:25 smithi164 conmon[29288]: audit 2021-02-10T04:59:24.787684+0000 mgr.y (mgr.14140) 1523 : audit [DBG] from='client.26476 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T04:59:28.069 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:27 smithi085 conmon[29319]: cluster 2021-02-10T04:59:26.566139+0000 mgr.y (mgr.14140 2021-02-10T04:59:28.069 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:27 smithi085 conmon[29319]: ) 1524 : cluster [DBG] pgmap v2473: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 1.1 KiB/s wr, 30 op/s; 188/2205 objects misplaced (8.526%); 311 B/s, 5 objects/s recovering 2021-02-10T04:59:28.069 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:27 smithi085 conmon[35961]: cluster 2021-02-10T04:59:26.566139+0000 mgr.y (mgr.14140 2021-02-10T04:59:28.069 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:27 smithi085 conmon[35961]: ) 1524 : cluster [DBG] pgmap v2473: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 1.1 KiB/s wr, 30 op/s; 188/2205 objects misplaced (8.526%); 311 B/s, 5 objects/s recovering 2021-02-10T04:59:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:27 smithi164 conmon[29288]: cluster 2021-02-10T04:59:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:27 smithi164 conmon[29288]: 2021-02-10T04:59:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:27 smithi164 conmon[29288]: 2021 2021-02-10T04:59:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:27 smithi164 conmon[29288]: -02-10 2021-02-10T04:59:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:27 smithi164 conmon[29288]: T04:59:26.566139+0000 mgr.y (mgr.14140) 1524 : cluster [DBG] pgmap v2473: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 1.1 KiB/s wr, 30 op/s; 188/2205 objects misplaced (8.526%); 311 B/s, 5 objects/s recovering 2021-02-10T04:59:29.023 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:59:29.040 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:59:29.057 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:59:29.079 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:59:29.094 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:28 smithi085 conmon[35961]: cluster 2021-02-10T04:59:28.630943+0000 mon.a (mon.0) 3004 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-02-10T04:59:29.094 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:28 smithi085 conmon[29319]: cluster 2021-02-10T04:59:28.630943+0000 mon.a (mon.0) 3004 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-02-10T04:59:29.104 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:59:29.129 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:59:29.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:28 smithi164 conmon[29288]: cluster 2021-02-10T04:59:28.630943 2021-02-10T04:59:29.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:28 smithi164 conmon[29288]: +0000 mon.a (mon.0) 3004 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-02-10T04:59:29.151 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:59:29.170 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:59:29.171 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:59:29.171 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:59:29.171 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:59:29.171 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:59:29.172 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:59:29.172 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:59:29.172 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:59:29.172 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:59:29.345 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:59:29 smithi085 conmon[207560]: debug 2021-02-10T04:59:29.342+0000 7f2695cda700 -1 osd.0 pg_epoch: 1349 pg[6.0( empty local-lis/les=1332/1333 n=0 ec=59/59 lis/c=1332/1332 les/c/f=1333/1333/0 sis=1332 pruub=11.782563210s) [0,3,2] r=0 lpr=1332 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.0) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 2 (0x563eefa0e9a0) 2021-02-10T04:59:29.353 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T04:59:29.353 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2021-02-10T04:59:29.353 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 0 2021-02-10T04:59:29.353 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T04:59:29.554 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:59:29 smithi164 conmon[37263]: debug 2021-02-10T04:59:29.165+0000 7f1c679e3700 -1 osd.5 pg_epoch: 1349 pg[6.a( empty local-lis/les=1332/1333 n=0 ec=59/59 lis/c=1332/1332 les/c/f=1333/1333/0 sis=1332) [5,6,0] r=0 lpr=1332 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x55665012e6e0) 2021-02-10T04:59:29.555 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:59:29 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:59:30.002 INFO:teuthology.orchestra.run.smithi085.stderr:set 7.5 pg_upmap_items mapping to [1->5,4->6,0->2] 2021-02-10T04:59:30.254 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:29 smithi085 conmon[29319]: cluster 2021-02-10T04:59:28.566837+0000 mgr.y (mgr.14140 2021-02-10T04:59:30.254 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:29 smithi085 conmon[29319]: ) 1525 : cluster [DBG] pgmap v2474: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 1.1 KiB/s wr, 30 op/s; 188/2205 objects misplaced (8.526%); 309 B/s, 5 objects/s recovering 2021-02-10T04:59:30.254 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:29 smithi085 conmon[29319]: audit 2021-02-10T04:59:29.004418+0000 mon.a ( 2021-02-10T04:59:30.254 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:29 smithi085 conmon[29319]: mon.0) 3005 : audit [DBG] from='client.? 172.21.15.85:0/1248180402' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:59:30.254 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:29 smithi085 conmon[29319]: audit 2021-02-10T04:59:29 2021-02-10T04:59:30.254 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:29 smithi085 conmon[29319]: .395162+0000 mon.c (mon.1) 161 : 2021-02-10T04:59:30.254 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:29 smithi085 conmon[29319]: audit [INF] from='client.? 172.21.15.85:0/2547339659' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.5", "id": [1, 5, 4, 6, 0, 2]}]: dispatch 2021-02-10T04:59:30.255 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:29 smithi085 conmon[29319]: audit 2021-02-10T04 2021-02-10T04:59:30.255 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:29 smithi085 conmon[29319]: :59:29.395507+0000 mon.a (mon. 2021-02-10T04:59:30.255 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:29 smithi085 conmon[29319]: 0) 3006 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.5", "id": [1, 5, 4, 6, 0, 2]}]: dispatch 2021-02-10T04:59:30.255 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:29 smithi085 conmon[35961]: cluster 2021-02-10T04:59:28.566837+0000 2021-02-10T04:59:30.255 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:29 smithi085 conmon[35961]: mgr.y (mgr.14140) 1525 : cluster [DBG] pgmap v2474: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 1.1 KiB/s wr, 30 op/s; 188/2205 objects misplaced (8.526%); 309 B/s, 5 objects/s recovering 2021-02-10T04:59:30.255 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:29 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T04:59:30.255 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:29 smithi085 conmon[35961]: T04:59:29.004418+0000 mon.a (mon.0) 3005 : audit [DBG] from='client.? 172.21.15.85:0/1248180402' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:59:30.255 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:29 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T04:59:30.255 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:29 smithi085 conmon[35961]: 04:59:29.395162+0000 mon.c (mon.1) 2021-02-10T04:59:30.255 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:29 smithi085 conmon[35961]: 161 : audit [INF] from='client.? 172.21.15.85:0/2547339659' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.5", "id": [1, 5, 4, 6, 0, 2]}]: dispatch 2021-02-10T04:59:30.255 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:29 smithi085 conmon[35961]: audit 2021 2021-02-10T04:59:30.255 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:29 smithi085 conmon[35961]: -02-10T04:59:29.395507+0000 2021-02-10T04:59:30.256 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:29 smithi085 conmon[35961]: mon.a (mon.0) 3006 : audit [INF] 2021-02-10T04:59:30.256 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:29 smithi085 conmon[35961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.5", "id": [1, 5, 4, 6, 0, 2]}]: dispatch 2021-02-10T04:59:30.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 04:59:30 smithi164 conmon[37263]: debug 2021-02-10T04:59:30.139+0000 7f1c6b9eb700 -1 osd.5 pg_epoch: 1350 pg[6.a( empty local-lis/les=1332/1333 n=0 ec=59/59 lis/c=1332/1332 les/c/f=1333/1333/0 sis=1332) [5,6,0] r=0 lpr=1332 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.a) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x556650329340) 2021-02-10T04:59:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:29 smithi164 conmon[29288]: cluster 2021-02-10T04:59:28.566837+0000 mgr.y 2021-02-10T04:59:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:29 smithi164 conmon[29288]: (mgr.14140) 1525 : cluster [DBG] pgmap v2474: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 1.1 KiB/s wr, 30 op/s; 188/2205 objects misplaced (8.526%); 309 B/s, 5 objects/s recovering 2021-02-10T04:59:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:29 smithi164 conmon[29288]: audit 2021-02-10T04:59:29.004418+0000 mon.a (mon.0) 3005 : audit [DBG] from='client.? 172.21.15.85:0/1248180402' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:59:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:29 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:59:30.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:29 smithi164 conmon[29288]: 02-10T04:59:29.395162+0000 mon.c (mon.1) 161 : audit [INF] from='client.? 172.21.15.85:0/2547339659' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.5", "id": [1, 5, 4, 6, 0, 2]}]: dispatch 2021-02-10T04:59:30.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:29 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:59:30.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:29 smithi164 conmon[29288]: 02-10T04:59:29.395507+0000 mon.a (mon.0) 3006 : audit 2021-02-10T04:59:30.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:29 smithi164 conmon[29288]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.5", "id": [1, 5, 4, 6, 0, 2]}]: dispatch 2021-02-10T04:59:30.549 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 04:59:30 smithi085 conmon[207560]: debug 2021-02-10T04:59:30.306+0000 7f2695cda700 -1 osd.0 pg_epoch: 1350 pg[6.0( empty local-lis/les=1332/1333 n=0 ec=59/59 lis/c=1332/1332 les/c/f=1333/1333/0 sis=1332 pruub=11.782563210s) [0,3,2] r=0 lpr=1332 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.0) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 2 (0x563eefa36420) 2021-02-10T04:59:31.008 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is unset 2021-02-10T04:59:31.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[29319]: audit 2021-02-10T04:59:29.553984+0000 mgr.y 2021-02-10T04:59:31.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[29319]: (mgr.14140) 1526 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:59:31.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[29319]: audit 2021-02-10T04:59:29.994894+0000 mon.a (mon.0) 3007 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "7.5", "id": [1, 5, 4, 6, 0, 2]}]': finished 2021-02-10T04:59:31.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[29319]: cluster 2021-02-10T04:59:29.994971 2021-02-10T04:59:31.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[29319]: +0000 mon.a (mon.0) 3008 : cluster [DBG] osdmap e1350: 8 total, 8 up, 8 in 2021-02-10T04:59:31.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[29319]: 2021-02-10T04:59:31.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[29319]: audit 2021-02-10T04:59 2021-02-10T04:59:31.187 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[29319]: :30.359694+0000 mon.c (mon.1) 162 : audit [INF] 2021-02-10T04:59:31.188 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[29319]: from='client.? 172.21.15.85:0/2949415910' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:59:31.188 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[29319]: audit 2021-02-10T04:59:31.188 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[29319]: 2021-02-10T04:59:30.359940+0000 mon.a (mon.0) 3009 2021-02-10T04:59:31.188 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[29319]: : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:59:31.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[35961]: audit 2021-02-10T04:59:31.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[35961]: 2021-02-10T04:59:31.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[35961]: 2021 2021-02-10T04:59:31.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[35961]: - 2021-02-10T04:59:31.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[35961]: 02 2021-02-10T04:59:31.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[35961]: - 2021-02-10T04:59:31.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[35961]: 10 2021-02-10T04:59:31.188 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[35961]: T 2021-02-10T04:59:31.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[35961]: 04 2021-02-10T04:59:31.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[35961]: :59:29.553984+0000 mgr.y ( 2021-02-10T04:59:31.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[35961]: mgr.14140) 1526 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:59:31.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:59:31.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[35961]: 59:29.994894+0000 mon.a (mon 2021-02-10T04:59:31.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[35961]: .0) 3007 : audit [INF] 2021-02-10T04:59:31.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[35961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "7.5", "id": [1, 5, 4, 6, 0, 2]}]': finished 2021-02-10T04:59:31.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[35961]: cluster 2021-02-10T04:59:29 2021-02-10T04:59:31.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[35961]: .994971+0000 mon.a (mon.0) 3008 : cluster [DBG] 2021-02-10T04:59:31.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[35961]: osdmap e1350: 8 total, 8 up, 8 in 2021-02-10T04:59:31.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[35961]: audit 2021-02-10T04:59: 2021-02-10T04:59:31.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[35961]: 30.359694+0000 mon.c (mon.1) 162 : 2021-02-10T04:59:31.189 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[35961]: audit [INF] from='client.? 172.21.15.85:0/2949415910' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:59:31.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[35961]: audit 2021-02-10T04:59 2021-02-10T04:59:31.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[35961]: :30.359940+0000 mon.a (mon.0) 3009 : audit 2021-02-10T04:59:31.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:31 smithi085 conmon[35961]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:59:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:31 smithi164 conmon[29288]: audit 2021-02-10T04:59:29.553984+0000 mgr.y (mgr. 2021-02-10T04:59:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:31 smithi164 conmon[29288]: 14140) 1526 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:59:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:31 smithi164 conmon[29288]: audit 2021-02-10T04:59:29.994894+0000 mon.a (mon.0) 3007 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "7.5", "id": [1, 5, 4, 6, 0, 2]}]': finished 2021-02-10T04:59:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:31 smithi164 conmon[29288]: cluster 2021-02-10T04:59:29.994971+0000 mon.a (mon.0) 3008 : cluster [DBG] osdmap e1350: 8 total, 8 up, 8 in 2021-02-10T04:59:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:31 smithi164 conmon[29288]: audit 2021-02-10T04:59:30.359694+0000 mon.c (mon.1) 162 : audit [INF] from='client.? 172.21.15.85:0/2949415910' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:59:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:31 smithi164 conmon[29288]: audit 2021-02-10T04:59:30.359940+0000 mon.a (mon.0) 3009 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:59:32.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:32 smithi164 conmon[29288]: cluster 2021-02-10T04:59:30.567432+0000 mgr.y 2021-02-10T04:59:32.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:32 smithi164 conmon[29288]: (mgr.14140) 1527 : cluster [DBG] pgmap v2476: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 188/2205 objects misplaced (8.526%) 2021-02-10T04:59:32.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:32 smithi164 conmon[29288]: cluster 2021-02-10T04:59:30.993383+0000 mon.a (mon.0) 3010 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:59:32.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:32 smithi164 conmon[29288]: audit 2021- 2021-02-10T04:59:32.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:32 smithi164 conmon[29288]: 02-10T04:59:31.003813+0000 mon.a ( 2021-02-10T04:59:32.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:32 smithi164 conmon[29288]: mon.0) 3011 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:59:32.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:32 smithi164 conmon[29288]: cluster 2021-02-10T04:59:32.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:32 smithi164 conmon[29288]: 2021-02-10T04:59 2021-02-10T04:59:32.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:32 smithi164 conmon[29288]: :31.004104+0000 mon.a (mon.0) 2021-02-10T04:59:32.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:32 smithi164 conmon[29288]: 3012 : cluster [DBG] osdmap e1351: 8 total, 8 up, 8 in 2021-02-10T04:59:32.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:32 smithi085 conmon[29319]: cluster 2021-02-10T04:59:30.567432+0000 mgr.y (mgr.14140) 1527 : cluster 2021-02-10T04:59:32.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:32 smithi085 conmon[29319]: [DBG] pgmap v2476: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 188/2205 objects misplaced (8.526%) 2021-02-10T04:59:32.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:32 smithi085 conmon[29319]: cluster 2021-02-10T04:59:30.993383+0000 mon.a (mon.0) 3010 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:59:32.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:32 smithi085 conmon[29319]: audit 2021-02-10T04:59:32.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:32 smithi085 conmon[29319]: 2021-02-10T04:59:31.003813+0000 mon.a (mon.0) 3011 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:59:32.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:32 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:59:32.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:32 smithi085 conmon[29319]: 02-10T04:59:31.004104+0000 mon.a (mon.0) 3012 : cluster 2021-02-10T04:59:32.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:32 smithi085 conmon[29319]: [DBG] osdmap e1351: 8 total, 8 up, 8 in 2021-02-10T04:59:32.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:32 smithi085 conmon[35961]: cluster 2021-02-10T04:59:32.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:32 smithi085 conmon[35961]: 2021-02-10T04:59:30. 2021-02-10T04:59:32.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:32 smithi085 conmon[35961]: 567432+0000 mgr.y (mgr.14140) 1527 : cluster [DBG] pgmap v2476: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 188/2205 objects misplaced (8.526%) 2021-02-10T04:59:32.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:32 smithi085 conmon[35961]: cluster 2021-02-10T04:59:32.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:32 smithi085 conmon[35961]: 2021-02-10T04:59:30.993383 2021-02-10T04:59:32.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:32 smithi085 conmon[35961]: +0000 mon.a (mon.0) 3010 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:59:32.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:32 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:59:32.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:32 smithi085 conmon[35961]: 02-10T04:59:31.003813+0000 mon.a ( 2021-02-10T04:59:32.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:32 smithi085 conmon[35961]: mon.0) 3011 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:59:32.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:32 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:59:32.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:32 smithi085 conmon[35961]: :59:31.004104+0000 mon.a (mon.0) 3012 : cluster 2021-02-10T04:59:32.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:32 smithi085 conmon[35961]: [DBG] osdmap e1351: 8 total, 8 up, 8 in 2021-02-10T04:59:32.589 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:59:32.808 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:59:33.369 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:33 smithi085 conmon[35961]: cluster 2021-02-10T04:59:31.286708+0000 2021-02-10T04:59:33.369 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:33 smithi085 conmon[35961]: osd.6 (osd.6) 194 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:59:33.369 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:33 smithi085 conmon[35961]: cluster 2021-02-10T04:59:31.289420 2021-02-10T04:59:33.369 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:33 smithi085 conmon[35961]: +0000 osd.6 (osd.6) 195 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:59:33.369 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:33 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:59:33.369 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:33 smithi085 conmon[35961]: 10T04:59:32.008302+0000 mon.a ( 2021-02-10T04:59:33.369 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:33 smithi085 conmon[35961]: mon.0) 3013 : cluster [DBG] osdmap e1352: 8 total, 8 up, 8 in 2021-02-10T04:59:33.369 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:33 smithi085 conmon[29319]: cluster 2021-02-10T04:59:31.286708+0000 2021-02-10T04:59:33.370 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:33 smithi085 conmon[29319]: osd.6 (osd.6) 194 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:59:33.370 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:33 smithi085 conmon[29319]: cluster 2021-02-10T04:59:31.289420+0000 osd.6 (osd.6) 195 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:59:33.370 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:33 smithi085 conmon[29319]: cluster 2021-02-10T04:59:32.008302+0000 mon.a (mon.0) 3013 : cluster [DBG] osdmap e1352: 8 total, 8 up, 8 in 2021-02-10T04:59:33.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:33 smithi164 conmon[29288]: cluster 2021-02-10T04:59:31.286708+0000 osd.6 2021-02-10T04:59:33.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:33 smithi164 conmon[29288]: (osd.6) 194 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:59:33.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:33 smithi164 conmon[29288]: cluster 2021-02-10T04:59:31.289420+0000 osd.6 (osd.6) 195 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:59:33.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:33 smithi164 conmon[29288]: cluster 2021-02-10T04:59:32.008302+0000 2021-02-10T04:59:33.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:33 smithi164 conmon[29288]: mon.a (mon.0) 3013 : cluster [DBG] osdmap e1352: 8 total, 8 up, 8 in 2021-02-10T04:59:33.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:33 smithi164 conmon[29288]: 2021-02-10T04:59:33.757 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set noscrub 2021-02-10T04:59:33.969 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:59:34.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:34 smithi085 conmon[29319]: cluster 2021-02-10T04:59:32.567984+0000 mgr.y (mgr.14140) 1528 : cluster [DBG] pgmap v2479: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 188/2205 objects misplaced (8.526%) 2021-02-10T04:59:34.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:34 smithi085 conmon[29319]: cluster 2021-02-10T04:59:33.638799+0000 mon.a (mon.0) 3014 : cluster [DBG] osdmap e1353: 8 total, 8 up, 8 in 2021-02-10T04:59:34.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:34 smithi085 conmon[35961]: cluster 2021-02-10T04:59:32.567984+0000 mgr.y (mgr.14140) 1528 : cluster [DBG] pgmap v2479: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 188/2205 objects misplaced (8.526%) 2021-02-10T04:59:34.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:34 smithi085 conmon[35961]: cluster 2021-02-10T04:59:33.638799+0000 mon.a (mon.0) 3014 : cluster [DBG] osdmap e1353: 8 total, 8 up, 8 in 2021-02-10T04:59:34.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:34 smithi164 conmon[29288]: cluster 2021-02-10T04:59:32.567984+0000 mgr.y (mgr 2021-02-10T04:59:34.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:34 smithi164 conmon[29288]: .14140) 1528 : cluster [DBG] pgmap v2479: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 188/2205 objects misplaced (8.526%) 2021-02-10T04:59:34.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:34 smithi164 conmon[29288]: cluster 2021-02-10T04:59:33.638799+0000 mon.a (mon.0) 3014 : cluster [DBG] osdmap e1353: 8 total, 8 up, 8 in 2021-02-10T04:59:34.805 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:59:34.834 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:59:34.852 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:59:34.875 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:59:34.896 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:59:34.915 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:59:34.933 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:59:34.955 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:59:34.956 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:59:34.956 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:59:34.956 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:59:34.956 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:59:34.956 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:59:34.957 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:59:34.957 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:59:34.957 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:59:35.019 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:59:34 smithi164 conmon[44544]: debug 2021-02-10T04:59:34.683+0000 7f64ab094700 -1 osd.7 pg_epoch: 1353 pg[2.5( v 53'2 (0'0,53'2] local-lis/les=1332/1333 n=0 ec=48/48 lis/c=1332/1332 les/c/f=1333/1333/0 sis=1332) [7,0,4] r=0 lpr=1332 crt=53'2 lcod 53'1 mlcod 0'0 active+clean] scrubber pg(2.5) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x55caba1bb760) 2021-02-10T04:59:35.020 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:35 smithi164 conmon[29288]: audit 2021-02-10T04:59:34.786069+0000 mon.c (mon.1) 2021-02-10T04:59:35.020 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:35 smithi164 conmon[29288]: 163 : audit [DBG] from='client.? 172.21.15.85:0/2683803972' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:59:35.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:35 smithi085 conmon[29319]: audit 2021-02-10T04:59:34.786069+0000 mon.c 2021-02-10T04:59:35.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:35 smithi085 conmon[29319]: (mon.1) 163 : audit [DBG] from='client.? 172.21.15.85:0/2683803972' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:59:35.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:35 smithi085 conmon[35961]: audit 2021-02-10T04:59:34.786069+0000 mon.c (mon. 2021-02-10T04:59:35.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:35 smithi085 conmon[35961]: 1) 163 : audit [DBG] from='client.? 172.21.15.85:0/2683803972' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:59:35.852 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2021-02-10T04:59:35.853 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:59:35.853 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.164484 2021-02-10T04:59:35.853 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd primary-affinity 1 0.16448375783200897 2021-02-10T04:59:36.076 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:59:36.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[35961]: cluster 2021-02-10T04:59:34. 2021-02-10T04:59:36.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[35961]: 568519+0000 mgr.y (mgr.14140) 1529 : cluster [DBG] pgmap v2481: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 188/2205 objects misplaced (8.526%) 2021-02-10T04:59:36.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: cluster 2021-02-10T04:59:36.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: 2021-02-10T04:59:36.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: 2021 2021-02-10T04:59:36.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: - 2021-02-10T04:59:36.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: 02 2021-02-10T04:59:36.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: - 2021-02-10T04:59:36.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: 10 2021-02-10T04:59:36.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: T 2021-02-10T04:59:36.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: 04 2021-02-10T04:59:36.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: : 2021-02-10T04:59:36.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: 59 2021-02-10T04:59:36.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: : 2021-02-10T04:59:36.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: 34 2021-02-10T04:59:36.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: . 2021-02-10T04:59:36.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: 568519 2021-02-10T04:59:36.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: +0000 2021-02-10T04:59:36.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: 2021-02-10T04:59:36.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: mgr.y 2021-02-10T04:59:36.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: ( 2021-02-10T04:59:36.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: mgr 2021-02-10T04:59:36.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: . 2021-02-10T04:59:36.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: 14140 2021-02-10T04:59:36.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: ) 2021-02-10T04:59:36.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: 1529 2021-02-10T04:59:36.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: : 2021-02-10T04:59:36.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: cluster 2021-02-10T04:59:36.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: 2021-02-10T04:59:36.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: [DBG] 2021-02-10T04:59:36.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: 2021-02-10T04:59:36.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: pgmap v2481: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 188/2205 objects misplaced (8.526%) 2021-02-10T04:59:36.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:36 smithi085 conmon[29319]: 2021-02-10T04:59:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:36 smithi164 conmon[29288]: cluster 2021-02-10T04:59:34.568519+0000 mgr.y ( 2021-02-10T04:59:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:36 smithi164 conmon[29288]: mgr.14140) 1529 : cluster [DBG] pgmap v2481: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 188/2205 objects misplaced (8.526%) 2021-02-10T04:59:37.032 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is set 2021-02-10T04:59:37.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:37 smithi164 conmon[29288]: audit 2021-02-10T04:59:37.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:37 smithi164 conmon[29288]: 2021-02-10T04:59:37.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:37 smithi164 conmon[29288]: 2021 2021-02-10T04:59:37.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:37 smithi164 conmon[29288]: - 2021-02-10T04:59:37.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:37 smithi164 conmon[29288]: 02 2021-02-10T04:59:37.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:37 smithi164 conmon[29288]: -10T04:59:36.751742+0000 2021-02-10T04:59:37.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:37 smithi164 conmon[29288]: mon.a (mon.0) 3015 : audit [INF] from='client.? 172.21.15.85:0/2369372324' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:59:37.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:37 smithi085 conmon[29319]: audit 2021-02-10T04:59:36.751742+0000 mon.a (mon.0) 3015 : audit [INF] from='client.? 172.21.15.85:0/2369372324' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:59:37.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:37 smithi085 conmon[35961]: audit 2021-02-10T04:59:36.751742+0000 2021-02-10T04:59:37.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:37 smithi085 conmon[35961]: mon.a (mon.0) 3015 : audit [INF] from='client.? 172.21.15.85:0/2369372324' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T04:59:38.088 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:59:38.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:38 smithi085 conmon[29319]: cluster 2021-02-10T04:59:36.569085 2021-02-10T04:59:38.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:38 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 1530 : cluster [DBG] pgmap v2482: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 188/2205 objects misplaced (8.526%) 2021-02-10T04:59:38.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:38 smithi085 conmon[29319]: cluster 2021-02-10T04:59:38.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:38 smithi085 conmon[29319]: 2021-02-10T04:59:37. 2021-02-10T04:59:38.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:38 smithi085 conmon[29319]: 020600+0000 mon.a (mon.0 2021-02-10T04:59:38.278 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:38 smithi085 conmon[29319]: ) 3016 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:59:38.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:38 smithi085 conmon[29319]: 2021-02-10T04:59:38.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:38 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:59:38.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:38 smithi085 conmon[29319]: 02-10T04:59:37.028136+0000 2021-02-10T04:59:38.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:38 smithi085 conmon[29319]: mon.a (mon.0) 3017 : audit [INF] 2021-02-10T04:59:38.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:38 smithi085 conmon[29319]: from='client.? 172.21.15.85:0/2369372324' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:59:38.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:38 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T04:59:38.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:38 smithi085 conmon[29319]: T04:59:37.028196+0000 mon.a ( 2021-02-10T04:59:38.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:38 smithi085 conmon[29319]: mon.0) 3018 : cluster [DBG] 2021-02-10T04:59:38.279 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:38 smithi085 conmon[29319]: osdmap e1354: 8 total, 8 up, 8 in 2021-02-10T04:59:38.279 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:38 smithi085 conmon[35961]: cluster 2021-02-10T04:59:36.569085+0000 2021-02-10T04:59:38.279 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:38 smithi085 conmon[35961]: mgr.y (mgr.14140) 1530 : cluster [DBG] pgmap v2482: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 188/2205 objects misplaced (8.526%) 2021-02-10T04:59:38.279 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:38 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:59:38.280 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:38 smithi085 conmon[35961]: T04:59:37.020600+0000 mon.a (mon.0) 3016 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:59:38.280 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:38 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:59:38.280 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:38 smithi085 conmon[35961]: 02-10T04:59:37.028136+0000 mon.a ( 2021-02-10T04:59:38.280 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:38 smithi085 conmon[35961]: mon.0) 3017 : audit [INF] from='client.? 172.21.15.85:0/2369372324' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:59:38.280 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:38 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:59:38.280 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:38 smithi085 conmon[35961]: 02-10T04:59:37.028196+0000 2021-02-10T04:59:38.280 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:38 smithi085 conmon[35961]: mon.a (mon.0) 3018 : 2021-02-10T04:59:38.280 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:38 smithi085 conmon[35961]: cluster [DBG] osdmap e1354: 8 total, 8 up, 8 in 2021-02-10T04:59:38.318 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:59:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:38 smithi164 conmon[29288]: cluster 2021-02-10T04:59:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:38 smithi164 conmon[29288]: 2021-02-10T04:59:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:38 smithi164 conmon[29288]: 2021 2021-02-10T04:59:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:38 smithi164 conmon[29288]: - 2021-02-10T04:59:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:38 smithi164 conmon[29288]: 02-10T04:59:36.569085 2021-02-10T04:59:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:38 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1530 : cluster [DBG] pgmap v2482: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 188/2205 objects misplaced (8.526%) 2021-02-10T04:59:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:38 smithi164 conmon[29288]: cluster 2021-02-10T04:59:37 2021-02-10T04:59:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:38 smithi164 conmon[29288]: .020600+0000 mon.a (mon.0) 3016 : cluster [WRN] 2021-02-10T04:59:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:38 smithi164 conmon[29288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:59:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:38 smithi164 conmon[29288]: audit 2021-02-10T04:59:37. 2021-02-10T04:59:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:38 smithi164 conmon[29288]: 028136+0000 mon.a (mon.0) 3017 : audit [INF] 2021-02-10T04:59:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:38 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/2369372324' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T04:59:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:38 smithi164 conmon[29288]: cluster 2021-02-10T04:59:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:38 smithi164 conmon[29288]: 2021-02-10T04:59:37.028196+0000 mon.a (mon.0) 3018 2021-02-10T04:59:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:38 smithi164 conmon[29288]: : cluster [DBG] osdmap e1354: 8 total, 8 up, 8 in 2021-02-10T04:59:39.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:39 smithi085 conmon[29319]: audit 2021-02-10T04:59:38.264923+0000 mon.a (mon.0) 3019 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:59:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:39 smithi085 conmon[29319]: audit 2021-02-10T04:59:38.266224+0000 mon.a (mon.0) 3020 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:59:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:39 smithi085 conmon[29319]: cluster 2021-02-10T04:59:38.639669+0000 mon.a (mon.0) 3021 : cluster [DBG] osdmap e1355: 8 total, 8 up, 8 in 2021-02-10T04:59:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:39 smithi085 conmon[29319]: audit 2021-02-10T04:59:38.641557+0000 mon.c (mon.1) 164 : audit [INF] from='client.? 172.21.15.85:0/361187404' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.16448375783200897}]: dispatch 2021-02-10T04:59:39.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:39 smithi085 conmon[29319]: audit 2021-02-10T04:59:38.646939+0000 mon.a (mon.0) 3022 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.16448375783200897}]: dispatch 2021-02-10T04:59:39.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:39 smithi085 conmon[35961]: audit 2021-02-10T04:59:38.264923+0000 mon.a (mon.0) 3019 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:59:39.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:39 smithi085 conmon[35961]: audit 2021-02-10T04:59:38.266224+0000 mon.a (mon.0) 3020 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:59:39.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:39 smithi085 conmon[35961]: cluster 2021-02-10T04:59:38.639669+0000 mon.a (mon.0) 3021 : cluster [DBG] osdmap e1355: 8 total, 8 up, 8 in 2021-02-10T04:59:39.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:39 smithi085 conmon[35961]: audit 2021-02-10T04:59:38.641557+0000 mon.c (mon.1) 164 : audit [INF] from='client.? 172.21.15.85:0/361187404' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.16448375783200897}]: dispatch 2021-02-10T04:59:39.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:39 smithi085 conmon[35961]: audit 2021-02-10T04:59:38.646939+0000 mon.a (mon.0) 3022 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.16448375783200897}]: dispatch 2021-02-10T04:59:39.228 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.6 dump_blocked_ops 2021-02-10T04:59:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:39 smithi164 conmon[29288]: audit 2021-02-10T04:59:38.264923+0000 mon.a (mon.0) 3019 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T04:59:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:39 smithi164 conmon[29288]: audit 2021-02-10T04:59:38.266224+0000 mon.a (mon.0) 3020 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T04:59:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:39 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:59:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:39 smithi164 conmon[29288]: 02-10T04:59:38.639669+0000 mon.a (mon.0) 3021 : cluster [DBG] osdmap e1355: 8 total, 8 up, 8 in 2021-02-10T04:59:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:39 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:59:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:39 smithi164 conmon[29288]: T04:59:38.641557+0000 mon.c (mon.1) 164 : audit [INF] from='client.? 172.21.15.85:0/361187404' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.16448375783200897}]: dispatch 2021-02-10T04:59:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:39 smithi164 conmon[29288]: 2021-02-10T04:59:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:39 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:59:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:39 smithi164 conmon[29288]: :59:38.646939+0000 mon.a (mon.0 2021-02-10T04:59:39.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:39 smithi164 conmon[29288]: ) 3022 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.16448375783200897}]: dispatch 2021-02-10T04:59:39.542 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T04:59:39.658 INFO:teuthology.orchestra.run.smithi085.stderr:set osd.1 primary-affinity to 0.164484 (8107792) 2021-02-10T04:59:39.833 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:59:39 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:59:40.073 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set nodeep-scrub 2021-02-10T04:59:40.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:40 smithi164 conmon[29288]: cluster 2021-02-10T04:59:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:40 smithi164 conmon[29288]: 2021-02-10T04:59 2021-02-10T04:59:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:40 smithi164 conmon[29288]: :38.569733+0000 mgr.y (mgr.14140) 1531 : cluster [DBG] pgmap v2484: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 188/2205 objects misplaced (8.526%) 2021-02-10T04:59:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:40 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T04:59:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:40 smithi164 conmon[29288]: 10T04:59:39.646211+0000 mon.a (mon.0 2021-02-10T04:59:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:40 smithi164 conmon[29288]: ) 3023 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.16448375783200897}]': finished 2021-02-10T04:59:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:40 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:59:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:40 smithi164 conmon[29288]: 10T04:59:39.646291+0000 mon.a 2021-02-10T04:59:40.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:40 smithi164 conmon[29288]: (mon.0) 3024 : cluster [DBG] osdmap e1356: 8 total, 8 up, 8 in 2021-02-10T04:59:40.303 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:59:40.438 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:40 smithi085 conmon[29319]: cluster 2021-02-10T04:59:38.569733+0000 mgr.y (mgr.14140) 1531 : 2021-02-10T04:59:40.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:40 smithi085 conmon[29319]: cluster [DBG] pgmap v2484: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 188/2205 objects misplaced (8.526%) 2021-02-10T04:59:40.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:40 smithi085 conmon[29319]: audit 2021-02-10T04:59:39.646211+0000 mon.a (mon.0) 3023 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.16448375783200897}]': finished 2021-02-10T04:59:40.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:40 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:59:40.439 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:40 smithi085 conmon[29319]: 10T04:59:39.646291+0000 mon.a (mon.0) 3024 : cluster [DBG] osdmap e1356: 8 total, 8 up, 8 in 2021-02-10T04:59:40.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:40 smithi085 conmon[35961]: cluster 2021-02-10T04:59:38.569733+0000 mgr.y (mgr. 2021-02-10T04:59:40.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:40 smithi085 conmon[35961]: 14140) 1531 : cluster [DBG] pgmap v2484: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 188/2205 objects misplaced (8.526%) 2021-02-10T04:59:40.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:40 smithi085 conmon[35961]: audit 2021-02-10T04:59: 2021-02-10T04:59:40.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:40 smithi085 conmon[35961]: 39.646211+0000 mon.a (mon.0) 3023 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.16448375783200897}]': finished 2021-02-10T04:59:40.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:40 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:59:40.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:40 smithi085 conmon[35961]: 04:59:39.646291+0000 mon.a (mon. 2021-02-10T04:59:40.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:40 smithi085 conmon[35961]: 0) 3024 : cluster [DBG] osdmap e1356: 8 total, 8 up, 8 in 2021-02-10T04:59:41.030 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:59:41.052 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:59:41.072 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:59:41.090 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:59:41.110 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:59:41.134 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:59:41.153 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:59:41.172 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:59:41.173 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:59:41.173 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:59:41.173 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:59:41.173 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:59:41.173 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:59:41.173 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:59:41.174 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:59:41.174 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:59:41.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:41 smithi164 conmon[29288]: audit 2021-02-10T04:59:41.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:41 smithi164 conmon[29288]: 2021-02-10T04:59:41.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:41 smithi164 conmon[29288]: 2021 2021-02-10T04:59:41.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:41 smithi164 conmon[29288]: - 2021-02-10T04:59:41.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:41 smithi164 conmon[29288]: 02 2021-02-10T04:59:41.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:41 smithi164 conmon[29288]: - 2021-02-10T04:59:41.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:41 smithi164 conmon[29288]: 10 2021-02-10T04:59:41.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:41 smithi164 conmon[29288]: T 2021-02-10T04:59:41.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:41 smithi164 conmon[29288]: 04 2021-02-10T04:59:41.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:41 smithi164 conmon[29288]: : 2021-02-10T04:59:41.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:41 smithi164 conmon[29288]: 59 2021-02-10T04:59:41.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:41 smithi164 conmon[29288]: :39.564046+0000 mgr.y (mgr.14140 2021-02-10T04:59:41.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:41 smithi164 conmon[29288]: ) 1532 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:59:41.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:41 smithi164 conmon[29288]: cluster 2021-02-10T04:59:40. 2021-02-10T04:59:41.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:41 smithi164 conmon[29288]: 648156+0000 mon.a (mon.0) 3025 : cluster [DBG] 2021-02-10T04:59:41.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:41 smithi164 conmon[29288]: osdmap e1357: 8 total, 8 up, 8 in 2021-02-10T04:59:41.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:41 smithi164 conmon[29288]: audit 2021-02-10T04:59: 2021-02-10T04:59:41.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:41 smithi164 conmon[29288]: 41.007346+0000 mon.a (mon.0) 3026 : audit [DBG] 2021-02-10T04:59:41.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:41 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/1566509016' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:59:41.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:41 smithi085 conmon[29319]: audit 2021-02-10T04:59:39.564046+0000 mgr.y 2021-02-10T04:59:41.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:41 smithi085 conmon[29319]: (mgr.14140) 1532 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:59:41.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:41 smithi085 conmon[29319]: cluster 2021-02-10T04:59:40.648156+0000 mon.a (mon.0) 3025 : cluster [DBG] osdmap e1357: 8 total, 8 up, 8 in 2021-02-10T04:59:41.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:41 smithi085 conmon[29319]: audit 2021-02-10T04:59:41.007346+0000 mon.a (mon.0) 3026 : audit [DBG] from='client.? 172.21.15.85:0/1566509016' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:59:41.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:41 smithi085 conmon[35961]: audit 2021-02-10T04:59:39. 2021-02-10T04:59:41.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:41 smithi085 conmon[35961]: 564046+0000 mgr.y (mgr.14140) 1532 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:59:41.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:41 smithi085 conmon[35961]: cluster 2021-02-10T04:59:40.648156+0000 mon.a (mon.0) 3025 : cluster [DBG] osdmap e1357: 8 total, 8 up, 8 in 2021-02-10T04:59:41.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:41 smithi085 conmon[35961]: audit 2021-02-10T04:59:41.007346+0000 mon.a (mon.0) 3026 : audit [DBG] from='client.? 172.21.15.85:0/1566509016' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:59:42.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:42 smithi085 conmon[29319]: cluster 2021-02-10T04:59:40.570295+0000 mgr.y (mgr 2021-02-10T04:59:42.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:42 smithi085 conmon[29319]: .14140) 1533 : cluster [DBG] pgmap v2487: 208 pgs: 2 peering, 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 197 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 188/2205 objects misplaced (8.526%) 2021-02-10T04:59:42.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:42 smithi085 conmon[29319]: cluster 2021-02-10T04:59:41.031953+0000 mon.a (mon.0) 3027 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-02-10T04:59:42.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:42 smithi085 conmon[35961]: cluster 2021-02-10T04:59:40.570295+0000 2021-02-10T04:59:42.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:42 smithi085 conmon[35961]: mgr.y (mgr.14140) 1533 : cluster [DBG] pgmap v2487: 208 pgs: 2 peering, 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 197 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 188/2205 objects misplaced (8.526%) 2021-02-10T04:59:42.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:42 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T04:59:42.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:42 smithi085 conmon[35961]: 04:59:41.031953+0000 mon.a (mon.0) 3027 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-02-10T04:59:42.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:42 smithi085 conmon[35961]: 2021-02-10T04:59:42.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:42 smithi164 conmon[29288]: cluster 2021-02-10T04:59:40.570295+0000 2021-02-10T04:59:42.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:42 smithi164 conmon[29288]: mgr.y (mgr.14140) 1533 : cluster [DBG] pgmap v2487: 208 pgs: 2 peering, 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 197 active+clean; 893 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 188/2205 objects misplaced (8.526%) 2021-02-10T04:59:42.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:42 smithi164 conmon[29288]: cluster 2021-02-10T04:59:41.031953+0000 mon.a (mon.0) 3027 : cluster [WRN] 2021-02-10T04:59:42.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:42 smithi164 conmon[29288]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-02-10T04:59:43.058 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is set 2021-02-10T04:59:43.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:43 smithi164 conmon[29288]: audit 2021-02-10T04:59:42.953866 2021-02-10T04:59:43.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:43 smithi164 conmon[29288]: +0000 mon.c (mon.1) 165 : audit [INF] from='client.? 172.21.15.85:0/1257976917' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:59:43.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:43 smithi164 conmon[29288]: audit 2021-02-10T04:59:42 2021-02-10T04:59:43.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:43 smithi164 conmon[29288]: .954159+0000 mon.a (mon.0) 3028 : audit [INF] 2021-02-10T04:59:43.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:43 smithi164 conmon[29288]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:59:43.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:43 smithi085 conmon[29319]: audit 2021-02-10T04:59:42.953866+0000 mon.c (mon.1) 165 : audit [INF] from='client.? 172.21.15.85:0/1257976917' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:59:43.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:43 smithi085 conmon[29319]: audit 2021-02-10T04:59:42.954159+0000 mon.a (mon.0) 3028 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:59:43.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:43 smithi085 conmon[35961]: audit 2021-02-10T04:59:42.953866+0000 mon.c (mon.1) 165 : audit [INF] from='client.? 172.21.15.85:0/1257976917' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:59:43.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:43 smithi085 conmon[35961]: audit 2021-02-10T04:59:42.954159+0000 mon.a (mon.0) 3028 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:59:44.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:44 smithi085 conmon[29319]: cluster 2021-02-10T04:59:42.570994+0000 mgr.y (mgr.14140) 2021-02-10T04:59:44.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:44 smithi085 conmon[29319]: 1534 : cluster [DBG] pgmap v2489: 208 pgs: 1 active+clean+scrubbing, 2 peering, 8 active+remapped+backfill_toofull, 197 active+clean; 916 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.9 MiB/s wr, 20 op/s; 188/2244 objects misplaced (8.378%); 4.5 MiB/s, 6 objects/s recovering 2021-02-10T04:59:44.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:44 smithi085 conmon[29319]: cluster 2021-02-10T04:59:43.040675+0000 mon.a (mon.0) 3029 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:59:44.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:44 smithi085 conmon[29319]: audit 2021-02-10T04:59:43.051300+0000 mon.a (mon.0) 3030 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:59:44.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:44 smithi085 conmon[29319]: cluster 2021-02-10T04:59:43.051369+0000 mon.a (mon.0) 3031 : cluster [DBG] osdmap e1358: 8 total, 8 up, 8 in 2021-02-10T04:59:44.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:44 smithi085 conmon[29319]: cluster 2021-02-10T04:59:43.639865+0000 mon.a (mon.0) 3032 : cluster [DBG] osdmap e1359: 8 total, 8 up, 8 in 2021-02-10T04:59:44.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:44 smithi085 conmon[35961]: cluster 2021-02-10T04:59:42. 2021-02-10T04:59:44.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:44 smithi085 conmon[35961]: 570994+0000 mgr.y (mgr.14140) 1534 : cluster [DBG] pgmap v2489: 208 pgs: 1 active+clean+scrubbing, 2 peering, 8 active+remapped+backfill_toofull, 197 active+clean; 916 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.9 MiB/s wr, 20 op/s; 188/2244 objects misplaced (8.378%); 4.5 MiB/s, 6 objects/s recovering 2021-02-10T04:59:44.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:44 smithi085 conmon[35961]: cluster 2021-02-10T04:59:43.040675+0000 mon.a 2021-02-10T04:59:44.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:44 smithi085 conmon[35961]: (mon.0) 3029 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:59:44.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:44 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:59:44.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:44 smithi085 conmon[35961]: 02-10T04:59:43.051300+0000 mon.a 2021-02-10T04:59:44.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:44 smithi085 conmon[35961]: (mon.0) 3030 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:59:44.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:44 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:59:44.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:44 smithi085 conmon[35961]: 02-10T04:59:43.051369+0000 2021-02-10T04:59:44.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:44 smithi085 conmon[35961]: mon.a (mon.0) 3031 : cluster [DBG] 2021-02-10T04:59:44.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:44 smithi085 conmon[35961]: osdmap e1358: 8 total, 8 up, 8 in 2021-02-10T04:59:44.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:44 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:59:44.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:44 smithi085 conmon[35961]: -10T04:59:43.639865+0000 mon.a 2021-02-10T04:59:44.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:44 smithi085 conmon[35961]: (mon.0) 3032 : cluster [DBG] osdmap e1359: 8 total, 8 up, 8 in 2021-02-10T04:59:44.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:44 smithi164 conmon[29288]: cluster 2021-02-10T04:59:42.570994+0000 mgr.y (mgr.14140) 1534 : 2021-02-10T04:59:44.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:44 smithi164 conmon[29288]: cluster [DBG] pgmap v2489: 208 pgs: 1 active+clean+scrubbing, 2 peering, 8 active+remapped+backfill_toofull, 197 active+clean; 916 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.9 MiB/s wr, 20 op/s; 188/2244 objects misplaced (8.378%); 4.5 MiB/s, 6 objects/s recovering 2021-02-10T04:59:44.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:44 smithi164 conmon[29288]: cluster 2021-02-10T04:59:43.040675+0000 mon.a (mon.0) 3029 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:59:44.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:44 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T04:59:44.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:44 smithi164 conmon[29288]: 04:59:43.051300+0000 mon.a (mon.0) 3030 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T04:59:44.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:44 smithi164 conmon[29288]: cluster 2021-02-10T04:59:43.051369+0000 mon.a 2021-02-10T04:59:44.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:44 smithi164 conmon[29288]: (mon.0) 3031 : cluster [DBG] osdmap e1358: 8 total, 8 up, 8 in 2021-02-10T04:59:44.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:44 smithi164 conmon[29288]: cluster 2021-02-10T04:59:43. 2021-02-10T04:59:44.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:44 smithi164 conmon[29288]: 639865+0000 mon.a (mon.0) 3032 : cluster 2021-02-10T04:59:44.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:44 smithi164 conmon[29288]: [DBG] osdmap e1359: 8 total, 8 up, 8 in 2021-02-10T04:59:44.416 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:59:44.627 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:59:45.441 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2021-02-10T04:59:45.442 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:59:45.442 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 3, 4, 5, 6, 7, 2] 2021-02-10T04:59:45.442 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd out 6 2021-02-10T04:59:45.678 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:59:45.874 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset noscrub 2021-02-10T04:59:46.094 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:59:46.173 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:46 smithi085 conmon[35961]: cluster 2021-02-10T04:59: 2021-02-10T04:59:46.173 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:46 smithi085 conmon[35961]: 44.571532+0000 mgr.y (mgr.14140) 1535 : cluster [DBG] pgmap v2492: 208 pgs: 1 active+clean+scrubbing, 2 peering, 8 active+remapped+backfill_toofull, 197 active+clean; 916 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 6.7 MiB/s wr, 22 op/s; 188/2244 objects misplaced (8.378%); 5.1 MiB/s, 7 objects/s recovering 2021-02-10T04:59:46.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:46 smithi085 conmon[29319]: cluster 2021-02-10T04:59:44.571532+0000 2021-02-10T04:59:46.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:46 smithi085 conmon[29319]: mgr.y (mgr.14140) 1535 : cluster [DBG] pgmap v2492: 208 pgs: 1 active+clean+scrubbing, 2 peering, 8 active+remapped+backfill_toofull, 197 active+clean; 916 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 6.7 MiB/s wr, 22 op/s; 188/2244 objects misplaced (8.378%); 5.1 MiB/s, 7 objects/s recovering 2021-02-10T04:59:46.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:46 smithi164 conmon[29288]: cluster 2021-02-10T04:59:44.571532+0000 mgr.y ( 2021-02-10T04:59:46.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:46 smithi164 conmon[29288]: mgr.14140) 1535 : cluster [DBG] pgmap v2492: 208 pgs: 1 active+clean+scrubbing, 2 peering, 8 active+remapped+backfill_toofull, 197 active+clean; 916 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 6.7 MiB/s wr, 22 op/s; 188/2244 objects misplaced (8.378%); 5.1 MiB/s, 7 objects/s recovering 2021-02-10T04:59:46.984 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:59:47.013 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:59:47.035 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:59:47.062 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:59:47.083 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:59:47.102 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:59:47.124 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:59:47.148 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:59:47.149 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T04:59:47.149 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T04:59:47.149 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T04:59:47.149 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T04:59:47.149 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T04:59:47.149 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T04:59:47.149 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T04:59:47.150 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T04:59:47.174 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:47 smithi085 conmon[29319]: audit 2021-02-10T04:59:47.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:47 smithi085 conmon[29319]: 2021- 2021-02-10T04:59:47.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:47 smithi085 conmon[29319]: 02-10T04:59:46.959971+0000 mon.c (mon.1) 166 : audit 2021-02-10T04:59:47.175 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:47 smithi085 conmon[29319]: [DBG] from='client.? 172.21.15.85:0/703360410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:59:47.175 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:47 smithi085 conmon[35961]: audit 2021-02-10T04: 2021-02-10T04:59:47.175 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:47 smithi085 conmon[35961]: 59:46.959971+0000 mon.c (mon.1) 166 : audit [DBG] from='client.? 172.21.15.85:0/703360410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:59:47.393 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:47 smithi164 conmon[29288]: audit 2021-02-10T04:59:47.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:47 smithi164 conmon[29288]: 2021-02-10T04:59:47.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:47 smithi164 conmon[29288]: 2021-02-10T04 2021-02-10T04:59:47.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:47 smithi164 conmon[29288]: :59:46.959971+0000 mon.c (mon.1) 166 : audit [DBG] from='client.? 172.21.15.85:0/703360410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:59:48.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:48 smithi085 conmon[29319]: cluster 2021-02-10T04:59:46.572318+0000 2021-02-10T04:59:48.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:48 smithi085 conmon[29319]: mgr.y (mgr.14140) 1536 : cluster [DBG] pgmap v2493: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 916 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.5 MiB/s wr, 18 op/s; 188/2244 objects misplaced (8.378%); 4.2 MiB/s, 5 objects/s recovering 2021-02-10T04:59:48.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:48 smithi085 conmon[29319]: cluster 2021-02-10T04:59:47.047000+0000 mon.a (mon.0) 3033 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2021-02-10T04:59:48.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:48 smithi085 conmon[29319]: cluster 2021-02-10T04:59:47.056482+0000 mon.a (mon.0) 3034 : cluster [DBG] osdmap e1360: 8 total, 8 up, 8 in 2021-02-10T04:59:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:48 smithi085 conmon[35961]: cluster 2021-02-10T04:59:46.572318+0000 2021-02-10T04:59:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:48 smithi085 conmon[35961]: mgr.y (mgr.14140) 1536 : cluster [DBG] pgmap v2493: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 916 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.5 MiB/s wr, 18 op/s; 188/2244 objects misplaced (8.378%); 4.2 MiB/s, 5 objects/s recovering 2021-02-10T04:59:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:48 smithi085 conmon[35961]: cluster 2021-02-10T04:59:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:48 smithi085 conmon[35961]: 2021-02-10T04:59:47.047000 2021-02-10T04:59:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:48 smithi085 conmon[35961]: +0000 mon.a (mon.0) 3033 : cluster [INF] 2021-02-10T04:59:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:48 smithi085 conmon[35961]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2021-02-10T04:59:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:48 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:59:48.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:48 smithi085 conmon[35961]: -10T04:59:47.056482+0000 mon.a 2021-02-10T04:59:48.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:48 smithi085 conmon[35961]: (mon.0) 3034 : cluster [DBG] osdmap e1360: 8 total, 8 up, 8 in 2021-02-10T04:59:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:48 smithi164 conmon[29288]: cluster 2021-02-10T04:59:46.572318+0000 mgr.y (mgr 2021-02-10T04:59:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:48 smithi164 conmon[29288]: .14140) 1536 : cluster [DBG] pgmap v2493: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 916 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.5 MiB/s wr, 18 op/s; 188/2244 objects misplaced (8.378%); 4.2 MiB/s, 5 objects/s recovering 2021-02-10T04:59:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:48 smithi164 conmon[29288]: cluster 2021-02-10T04:59:47.047000+0000 mon.a (mon.0) 3033 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2021-02-10T04:59:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:48 smithi164 conmon[29288]: cluster 2021-02-10T04:59:47.056482+0000 mon.a (mon.0) 3034 : cluster 2021-02-10T04:59:48.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:48 smithi164 conmon[29288]: [DBG] osdmap e1360: 8 total, 8 up, 8 in 2021-02-10T04:59:49.132 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:49 smithi085 conmon[29319]: cluster 2021-02-10T04:59:49.133 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:49 smithi085 conmon[29319]: 2021-02-10T04:59:48.059069+0000 2021-02-10T04:59:49.133 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:49 smithi085 conmon[29319]: mon.a (mon.0) 3035 : cluster [DBG] osdmap e1361: 8 total, 8 up, 8 in 2021-02-10T04:59:49.133 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:49 smithi085 conmon[35961]: cluster 2021-02-10T04:59:48 2021-02-10T04:59:49.133 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:49 smithi085 conmon[35961]: .059069+0000 mon.a (mon.0) 3035 : cluster [DBG] osdmap e1361: 8 total, 8 up, 8 in 2021-02-10T04:59:49.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:49 smithi164 conmon[29288]: cluster 2021-02-10T04:59:49.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:49 smithi164 conmon[29288]: 2021-02-10T04:59:49.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:49 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:59:49.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:49 smithi164 conmon[29288]: 59:48.059069+0000 mon.a (mon.0) 3035 : cluster [DBG] osdmap e1361: 8 total, 8 up, 8 in 2021-02-10T04:59:49.894 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:59:49 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:59:49.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:49 smithi085 conmon[29319]: debug 2021-02-10T04:59:49.537+0000 7f9e105d6700 -1 mon.a@0(leader).osd e1362 definitely_dead 0 2021-02-10T04:59:50.091 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is unset 2021-02-10T04:59:50.091 INFO:teuthology.orchestra.run.smithi085.stderr:marked out osd.6. 2021-02-10T04:59:50.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:50 smithi164 conmon[29288]: cluster 2021-02-10T04:59:48.572939+0000 2021-02-10T04:59:50.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:50 smithi164 conmon[29288]: mgr.y (mgr.14140) 1537 : cluster [DBG] pgmap v2496: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 916 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 188/2244 objects misplaced (8.378%) 2021-02-10T04:59:50.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:50 smithi164 conmon[29288]: cluster 2021-02-10T04:59:49.076560+0000 mon.a (mon.0) 3036 : cluster [DBG] osdmap e1362: 8 total, 8 up, 8 in 2021-02-10T04:59:50.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:50 smithi164 conmon[29288]: audit 2021-02-10T04:59:49.532378+0000 mon.c 2021-02-10T04:59:50.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:50 smithi164 conmon[29288]: (mon.1) 167 : audit [INF] from='client.? 172.21.15.85:0/941353797' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:59:50.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:50 smithi164 conmon[29288]: audit 2021-02-10T04 2021-02-10T04:59:50.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:50 smithi164 conmon[29288]: :59:49.532675+0000 mon.a (mon.0) 3037 : 2021-02-10T04:59:50.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:50 smithi164 conmon[29288]: audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:59:50.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:50 smithi164 conmon[29288]: audit 2021 2021-02-10T04:59:50.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:50 smithi164 conmon[29288]: -02-10T04:59:49.538343+0000 mon.c 2021-02-10T04:59:50.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:50 smithi164 conmon[29288]: (mon.1) 168 : audit [INF] from='client.? 172.21.15.85:0/2990533027' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2021-02-10T04:59:50.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:50 smithi164 conmon[29288]: audit 2021-02-10T04:59:50.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:50 smithi164 conmon[29288]: 2021-02-10T04:59:49.538576+0000 2021-02-10T04:59:50.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:50 smithi164 conmon[29288]: mon.a (mon.0) 3038 : audit 2021-02-10T04:59:50.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:50 smithi164 conmon[29288]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2021-02-10T04:59:50.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:50 smithi164 conmon[29288]: cluster 2021-02-10T04:59:50.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:50 smithi164 conmon[29288]: 2021-02-10T04:59: 2021-02-10T04:59:50.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:50 smithi164 conmon[29288]: 49.538653+0000 mon.a (mon. 2021-02-10T04:59:50.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:50 smithi164 conmon[29288]: 0) 3039 : cluster [INF] Client client.admin marked osd.6 out, while it was still marked up 2021-02-10T04:59:50.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:50 smithi164 conmon[29288]: 2021-02-10T04:59:50.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[29319]: cluster 2021-02-10T04:59:48.572939+0000 mgr.y (mgr.14140) 1537 : cluster [DBG] 2021-02-10T04:59:50.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[29319]: pgmap v2496: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 916 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 188/2244 objects misplaced (8.378%) 2021-02-10T04:59:50.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[29319]: cluster 2021-02-10T04:59:49.076560+0000 mon.a (mon.0) 3036 : cluster [DBG] osdmap e1362: 8 total, 8 up, 8 in 2021-02-10T04:59:50.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[29319]: audit 2021-02-10T04:59 2021-02-10T04:59:50.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[29319]: :49.532378+0000 mon.c (mon.1) 167 : audit [INF] from='client.? 172.21.15.85:0/941353797' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:59:50.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[29319]: audit 2021-02-10T04:59:50.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[29319]: 2021-02-10T04:59:49.532675+0000 mon.a (mon 2021-02-10T04:59:50.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[29319]: .0) 3037 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:59:50.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[29319]: audit 2021-02- 2021-02-10T04:59:50.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[29319]: 10T04:59:49.538343+0000 mon.c 2021-02-10T04:59:50.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[29319]: (mon.1) 168 : audit [INF] from='client.? 172.21.15.85:0/2990533027' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2021-02-10T04:59:50.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[29319]: 2021-02-10T04:59:50.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[29319]: audit 2021-02-10T04:59:50.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[29319]: 2021-02-10T04:59:49.538576+0000 mon.a (mon.0) 2021-02-10T04:59:50.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[29319]: 3038 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2021-02-10T04:59:50.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:59:50.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[29319]: -02-10T04:59:49.538653+0000 mon.a (mon.0) 3039 : cluster [INF] Client client.admin marked osd.6 out, while it was still marked up 2021-02-10T04:59:50.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[35961]: cluster 2021-02-10T04:59: 2021-02-10T04:59:50.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[35961]: 48.572939+0000 mgr.y (mgr.14140) 1537 : cluster [DBG] pgmap v2496: 208 pgs: 1 active+clean+scrubbing, 8 active+remapped+backfill_toofull, 199 active+clean; 916 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 188/2244 objects misplaced (8.378%) 2021-02-10T04:59:50.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[35961]: cluster 2021-02-10T04:59:49.076560+0000 2021-02-10T04:59:50.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[35961]: mon.a (mon.0) 3036 : cluster [DBG] osdmap e1362: 8 total, 8 up, 8 in 2021-02-10T04:59:50.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[35961]: audit 2021-02-10T04:59:49.532378+0000 2021-02-10T04:59:50.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[35961]: mon.c (mon.1) 167 : audit [INF] from='client.? 172.21.15.85:0/941353797' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:59:50.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[35961]: audit 2021-02-10T04:59:49.532675+0000 2021-02-10T04:59:50.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[35961]: mon.a (mon.0) 3037 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T04:59:50.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[35961]: audit 2021 2021-02-10T04:59:50.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[35961]: -02-10T04:59:49.538343+0000 2021-02-10T04:59:50.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[35961]: mon.c (mon.1) 168 : audit [INF] 2021-02-10T04:59:50.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/2990533027' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2021-02-10T04:59:50.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[35961]: audit 2021-02-10T04 2021-02-10T04:59:50.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[35961]: :59:49.538576+0000 mon.a (mon.0) 2021-02-10T04:59:50.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[35961]: 3038 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2021-02-10T04:59:50.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[35961]: cluster 2021-02-10T04:59:49.538653 2021-02-10T04:59:50.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[35961]: +0000 mon.a (mon.0) 3039 : cluster [INF] 2021-02-10T04:59:50.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:50 smithi085 conmon[35961]: Client client.admin marked osd.6 out, while it was still marked up 2021-02-10T04:59:50.921 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T04:59:51.167 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:59:51.300 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:51 smithi085 conmon[29319]: audit 2021-02-10T04:59:49.575034+0000 mgr.y (mgr.14140) 1538 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:59:51.301 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:51 smithi085 conmon[29319]: cluster 2021-02-10T04:59:49.652933+0000 osd.7 (osd.7) 100 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:59:51.301 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:51 smithi085 conmon[29319]: cluster 2021-02-10T04:59:49.655398+0000 osd.7 (osd.7) 101 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:59:51.301 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:51 smithi085 conmon[29319]: cluster 2021-02-10T04:59:50.065398+0000 mon.a (mon.0) 3040 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:59:51.301 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:51 smithi085 conmon[29319]: audit 2021-02-10T04:59:50.079141+0000 mon.a (mon.0) 3041 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:59:51.301 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:51 smithi085 conmon[29319]: audit 2021-02-10T04:59:50.079217+0000 mon.a (mon.0) 3042 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2021-02-10T04:59:51.301 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:51 smithi085 conmon[29319]: cluster 2021-02-10T04:59:50.079269+0000 mon.a (mon.0) 3043 : cluster [DBG] osdmap e1363: 8 total, 8 up, 7 in 2021-02-10T04:59:51.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:51 smithi085 conmon[35961]: audit 2021-02-10T04:59:49.575034+0000 2021-02-10T04:59:51.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:51 smithi085 conmon[35961]: mgr.y (mgr.14140) 1538 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:59:51.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:51 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:59:51.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:51 smithi085 conmon[35961]: -02-10T04:59:49.652933+0000 osd.7 (osd.7) 100 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:59:51.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:51 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:59:51.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:51 smithi085 conmon[35961]: -02-10T04:59:49.655398+0000 2021-02-10T04:59:51.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:51 smithi085 conmon[35961]: osd.7 (osd.7) 101 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:59:51.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:51 smithi085 conmon[35961]: 2021-02-10T04:59:51.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:51 smithi085 conmon[35961]: cluster 2021-02-10T04:59 2021-02-10T04:59:51.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:51 smithi085 conmon[35961]: :50.065398+0000 mon.a (mon.0) 3040 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:59:51.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:51 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:59:51.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:51 smithi085 conmon[35961]: 10T04:59:50.079141+0000 mon.a (mon.0 2021-02-10T04:59:51.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:51 smithi085 conmon[35961]: ) 3041 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:59:51.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:51 smithi085 conmon[35961]: audit 2021-02-10T04:59:51.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:51 smithi085 conmon[35961]: 2021-02-10T04:59:50.079217 2021-02-10T04:59:51.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:51 smithi085 conmon[35961]: +0000 mon.a (mon.0) 3042 : audit [INF] 2021-02-10T04:59:51.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:51 smithi085 conmon[35961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2021-02-10T04:59:51.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:51 smithi085 conmon[35961]: cluster 2021-02-10T04:59:51.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:51 smithi085 conmon[35961]: 2021-02-10T04:59:50.079269 2021-02-10T04:59:51.303 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:51 smithi085 conmon[35961]: +0000 mon.a (mon.0) 3043 : cluster 2021-02-10T04:59:51.303 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:51 smithi085 conmon[35961]: [DBG] osdmap e1363: 8 total, 8 up, 7 in 2021-02-10T04:59:51.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:51 smithi164 conmon[29288]: audit 2021-02-10T04:59:49.575034+0000 mgr.y (mgr. 2021-02-10T04:59:51.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:51 smithi164 conmon[29288]: 14140) 1538 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T04:59:51.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:51 smithi164 conmon[29288]: cluster 2021-02-10T04:59:49.652933+0000 osd.7 (osd.7) 100 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:59:51.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:51 smithi164 conmon[29288]: cluster 2021-02-10T04:59:49.655398+0000 osd.7 (osd.7) 101 : cluster [DBG] 2021-02-10T04:59:51.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:51 smithi164 conmon[29288]: purged_snaps scrub ok 2021-02-10T04:59:51.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:51 smithi164 conmon[29288]: cluster 2021-02-10T04:59:50.065398 2021-02-10T04:59:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:51 smithi164 conmon[29288]: +0000 mon.a (mon.0) 3040 : cluster [WRN] 2021-02-10T04:59:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:51 smithi164 conmon[29288]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T04:59:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:51 smithi164 conmon[29288]: audit 2021-02-10T04:59:50. 2021-02-10T04:59:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:51 smithi164 conmon[29288]: 079141+0000 mon.a (mon.0) 3041 : audit 2021-02-10T04:59:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:51 smithi164 conmon[29288]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T04:59:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:51 smithi164 conmon[29288]: audit 2021-02-10T04:59:50 2021-02-10T04:59:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:51 smithi164 conmon[29288]: .079217+0000 mon.a (mon.0) 3042 : audit 2021-02-10T04:59:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:51 smithi164 conmon[29288]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2021-02-10T04:59:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:51 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:59:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:51 smithi164 conmon[29288]: 59:50.079269+0000 mon.a (mon.0) 2021-02-10T04:59:51.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:51 smithi164 conmon[29288]: 3043 : cluster [DBG] osdmap e1363: 8 total, 8 up, 7 in 2021-02-10T04:59:52.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:52 smithi164 conmon[29288]: cluster 2021-02-10T04:59:52.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:52 smithi164 conmon[29288]: 2021-02-10T04:59 2021-02-10T04:59:52.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:52 smithi164 conmon[29288]: :50.573530+0000 mgr.y (mgr.14140) 1539 : cluster [DBG] pgmap v2499: 208 pgs: 18 peering, 1 active+clean+scrubbing, 7 active+remapped+backfill_toofull, 182 active+clean; 857 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 217 MiB/s rd, 61 MiB/s wr, 393 op/s; 166/2169 objects misplaced (7.653%) 2021-02-10T04:59:52.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:52 smithi164 conmon[29288]: cluster 2021-02-10T04:59:52.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:52 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:59:52.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:52 smithi164 conmon[29288]: 59:51.071054+0000 mon.a (mon.0) 3044 2021-02-10T04:59:52.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:52 smithi164 conmon[29288]: : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2021-02-10T04:59:52.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:52 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:59:52.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:52 smithi164 conmon[29288]: -02-10T04:59:51.071078+0000 2021-02-10T04:59:52.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:52 smithi164 conmon[29288]: mon.a (mon.0) 3045 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-02-10T04:59:52.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:52 smithi164 conmon[29288]: cluster 2021-02-10T04:59:52.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:52 smithi164 conmon[29288]: 2021-02-10T04:59:51.078957 2021-02-10T04:59:52.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:52 smithi164 conmon[29288]: +0000 mon.a (mon.0) 3046 : cluster [DBG] osdmap e1364: 8 total, 8 up, 7 in 2021-02-10T04:59:52.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:52 smithi164 conmon[29288]: 2021-02-10T04:59:52.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:52 smithi085 conmon[29319]: cluster 2021-02-10T04:59:50.573530+0000 2021-02-10T04:59:52.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:52 smithi085 conmon[29319]: mgr.y (mgr.14140) 1539 : cluster [DBG] pgmap v2499: 208 pgs: 18 peering, 1 active+clean+scrubbing, 7 active+remapped+backfill_toofull, 182 active+clean; 857 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 217 MiB/s rd, 61 MiB/s wr, 393 op/s; 166/2169 objects misplaced (7.653%) 2021-02-10T04:59:52.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:52 smithi085 conmon[29319]: cluster 2021-02-10T04:59:51.071054+0000 mon.a (mon.0) 3044 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2021-02-10T04:59:52.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:52 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:59:52.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:52 smithi085 conmon[29319]: 59:51.071078+0000 mon.a (mon.0) 3045 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-02-10T04:59:52.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:52 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:59:52.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:52 smithi085 conmon[29319]: 59:51.078957+0000 mon.a (mon.0) 2021-02-10T04:59:52.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:52 smithi085 conmon[29319]: 3046 : cluster [DBG] osdmap e1364: 8 total, 8 up, 7 in 2021-02-10T04:59:52.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:52 smithi085 conmon[35961]: cluster 2021-02-10T04:59:50 2021-02-10T04:59:52.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:52 smithi085 conmon[35961]: .573530+0000 mgr.y (mgr.14140) 1539 : cluster [DBG] pgmap v2499: 208 pgs: 18 peering, 1 active+clean+scrubbing, 7 active+remapped+backfill_toofull, 182 active+clean; 857 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 217 MiB/s rd, 61 MiB/s wr, 393 op/s; 166/2169 objects misplaced (7.653%) 2021-02-10T04:59:52.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:52 smithi085 conmon[35961]: cluster 2021-02-10T04:59:52.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:52 smithi085 conmon[35961]: 2021-02-10T04:59:51 2021-02-10T04:59:52.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:52 smithi085 conmon[35961]: .071054+0000 mon.a (mon.0) 3044 : 2021-02-10T04:59:52.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:52 smithi085 conmon[35961]: cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2021-02-10T04:59:52.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:52 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:59:52.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:52 smithi085 conmon[35961]: -10T04:59:51.071078+0000 mon.a (mon.0) 3045 : 2021-02-10T04:59:52.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:52 smithi085 conmon[35961]: cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-02-10T04:59:52.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:52 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:59:52.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:52 smithi085 conmon[35961]: 10T04:59:51.078957+0000 mon.a (mon.0) 3046 : cluster [DBG] osdmap e1364: 8 total, 8 up, 7 in 2021-02-10T04:59:52.520 INFO:tasks.rados.rados.0.smithi085.stdout:328: finishing copy_from to smithi08575397-33 2021-02-10T04:59:52.521 INFO:tasks.rados.rados.0.smithi085.stdout:update_object_version oid 33 v 110 (ObjNum 126 snap 23 seq_num 126) dirty exists 2021-02-10T04:59:52.521 INFO:tasks.rados.rados.0.smithi085.stdout:328: finishing copy_from racing read to smithi08575397-33 2021-02-10T04:59:52.521 INFO:tasks.rados.rados.0.smithi085.stdout:328: done (0 left) 2021-02-10T04:59:52.770 INFO:tasks.rados.rados.0.smithi085.stderr:0 errors. 2021-02-10T04:59:52.770 INFO:tasks.rados.rados.0.smithi085.stderr: 2021-02-10T04:59:53.234 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd dump --format=json 2021-02-10T04:59:53.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:53 smithi164 conmon[29288]: cluster 2021-02-10T04:59:53.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:53 smithi164 conmon[29288]: 2021-02-10T04:59:53.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:53 smithi164 conmon[29288]: 2021-02-10T04:59:51 2021-02-10T04:59:53.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:53 smithi164 conmon[29288]: .749562+0000 osd.7 (osd.7) 102 : cluster [DBG] 6.15 scrub ok 2021-02-10T04:59:53.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:53 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:59:53.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:53 smithi164 conmon[29288]: :59:52.079347+0000 osd.4 (osd.4) 2666 : 2021-02-10T04:59:53.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:53 smithi164 conmon[29288]: cluster [DBG] 4.18 starting backfill to osd.3 from (0'0,0'0] MAX to 1261'84 2021-02-10T04:59:53.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:53 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:59:53.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:53 smithi164 conmon[29288]: 04:59:52.081264+0000 osd.1 (osd.1 2021-02-10T04:59:53.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:53 smithi164 conmon[29288]: ) 524 : cluster [DBG] 4.a starting backfill to osd.5 from (0'0,0'0] MAX to 1261'69 2021-02-10T04:59:53.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:53 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:59:53.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:53 smithi164 conmon[29288]: 10T04:59:52.081794+0000 mon.a (mon 2021-02-10T04:59:53.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:53 smithi164 conmon[29288]: .0) 3047 : cluster [DBG] osdmap e1365: 8 total, 8 up, 7 in 2021-02-10T04:59:53.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:53 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:59:53.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:53 smithi164 conmon[29288]: 02-10T04:59:52.084098+0000 osd.4 2021-02-10T04:59:53.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:53 smithi164 conmon[29288]: (osd.4) 2667 : cluster [DBG] 4.c starting backfill to osd.1 from (0'0,0'0] MAX to 1261'76 2021-02-10T04:59:53.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:53 smithi164 conmon[29288]: cluster 2021 2021-02-10T04:59:53.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:53 smithi164 conmon[29288]: -02-10T04:59:52.084397+0000 2021-02-10T04:59:53.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:53 smithi164 conmon[29288]: osd.4 (osd.4) 2668 : cluster [DBG] 4.e starting backfill to osd.3 from (0'0,0'0] MAX to 1261'65 2021-02-10T04:59:53.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:53 smithi164 conmon[29288]: 2021-02-10T04:59:53.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:53 smithi164 conmon[29288]: cluster 2021-02-10T04:59:53.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:53 smithi164 conmon[29288]: 2021-02-10T04:59:52.086227+0000 osd.5 (osd 2021-02-10T04:59:53.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:53 smithi164 conmon[29288]: .5) 204 : cluster [DBG] 4.8 starting backfill to osd.4 from (0'0,0'0] MAX to 1261'84 2021-02-10T04:59:53.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:53 smithi164 conmon[29288]: cluster 2021- 2021-02-10T04:59:53.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:53 smithi164 conmon[29288]: 02-10T04:59:52.091563+0000 osd.5 (osd 2021-02-10T04:59:53.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:53 smithi164 conmon[29288]: .5) 205 : cluster [DBG] 4.1f starting backfill to osd.0 from (0'0,0'0] MAX to 1261'71 2021-02-10T04:59:53.441 INFO:journalctl@ceph.rgw.realm.zone.a.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[54650]: debug 2021-02-10T04:59:53.157+0000 7ffb94c42700 -1 RGWWatcher::handle_error cookie 94187770079488 err (110) Connection timed out 2021-02-10T04:59:53.442 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:59:53.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[29319]: cluster 2021-02-10T04:59:51.749562+0000 2021-02-10T04:59:53.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[29319]: osd.7 (osd.7) 102 : cluster [DBG] 6.15 scrub ok 2021-02-10T04:59:53.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[29319]: cluster 2021-02-10T04:59:52.079347+0000 osd.4 (osd.4) 2666 : cluster [DBG] 4.18 starting backfill to osd.3 from (0'0,0'0] MAX to 1261'84 2021-02-10T04:59:53.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[29319]: cluster 2021-02-10T04:59:52.081264+0000 osd.1 (osd.1) 524 : cluster [DBG] 4.a starting backfill to osd.5 from (0'0,0'0] MAX to 1261'69 2021-02-10T04:59:53.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T04:59:53.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[29319]: 04:59:52.081794+0000 mon.a (mon.0) 3047 : cluster [DBG] osdmap e1365: 8 total, 8 up, 7 in 2021-02-10T04:59:53.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:59:53.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[29319]: 59:52.084098+0000 osd.4 (osd.4) 2021-02-10T04:59:53.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[29319]: 2667 : cluster [DBG] 4.c starting backfill to osd.1 from (0'0,0'0] MAX to 1261'76 2021-02-10T04:59:53.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[29319]: cluster 2021-02-10T04:59:53.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[29319]: 2021-02-10T04:59:52.084397+0000 2021-02-10T04:59:53.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[29319]: osd.4 (osd.4) 2668 : cluster [DBG] 2021-02-10T04:59:53.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[29319]: 4.e starting backfill to osd.3 from (0'0,0'0] MAX to 1261'65 2021-02-10T04:59:53.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[29319]: cluster 2021-02-10T04: 2021-02-10T04:59:53.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[29319]: 59:52.086227+0000 osd.5 (osd.5) 2021-02-10T04:59:53.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[29319]: 204 : cluster [DBG] 4.8 starting backfill to osd.4 from (0'0,0'0] MAX to 1261'84 2021-02-10T04:59:53.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:59:53.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[29319]: 02-10T04:59:52.091563+0000 2021-02-10T04:59:53.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[29319]: osd.5 (osd.5) 205 : cluster [DBG] 4.1f starting backfill to osd.0 from (0'0,0'0] MAX to 1261'71 2021-02-10T04:59:53.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[29319]: 2021-02-10T04:59:53.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[35961]: cluster 2021-02-10T04:59:51.749562+0000 2021-02-10T04:59:53.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[35961]: osd.7 (osd.7) 102 : cluster [DBG] 6.15 scrub ok 2021-02-10T04:59:53.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[35961]: cluster 2021-02-10T04:59:52.079347+0000 osd.4 (osd.4) 2666 2021-02-10T04:59:53.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[35961]: : cluster [DBG] 4.18 starting backfill to osd.3 from (0'0,0'0] MAX to 1261'84 2021-02-10T04:59:53.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:59:53.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[35961]: 10T04:59:52.081264+0000 osd.1 (osd.1) 524 : 2021-02-10T04:59:53.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[35961]: cluster [DBG] 4.a starting backfill to osd.5 from (0'0,0'0] MAX to 1261'69 2021-02-10T04:59:53.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[35961]: cluster 2021-02-10T04:59:52.081794+0000 mon.a ( 2021-02-10T04:59:53.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[35961]: mon.0) 3047 : cluster [DBG] osdmap e1365: 8 total, 8 up, 7 in 2021-02-10T04:59:53.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[35961]: cluster 2021 2021-02-10T04:59:53.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[35961]: -02-10T04:59:52.084098+0000 2021-02-10T04:59:53.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[35961]: osd.4 (osd.4) 2667 : cluster [DBG] 4.c starting backfill to osd.1 from (0'0,0'0] MAX to 1261'76 2021-02-10T04:59:53.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:59:53.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[35961]: 02-10T04:59:52.084397+0000 osd.4 2021-02-10T04:59:53.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[35961]: (osd.4) 2668 : cluster [DBG] 4.e starting backfill to osd.3 from (0'0,0'0] MAX to 1261'65 2021-02-10T04:59:53.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:59:53.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[35961]: 02-10T04:59:52.086227+0000 osd.5 ( 2021-02-10T04:59:53.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[35961]: osd.5) 204 : cluster [DBG] 4.8 starting backfill to osd.4 from (0'0,0'0] MAX to 1261'84 2021-02-10T04:59:53.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:59:53.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[35961]: 02-10T04:59:52.091563+0000 osd.5 ( 2021-02-10T04:59:53.446 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:53 smithi085 conmon[35961]: osd.5) 205 : cluster [DBG] 4.1f starting backfill to osd.0 from (0'0,0'0] MAX to 1261'71 2021-02-10T04:59:53.717 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset nodeep-scrub 2021-02-10T04:59:53.926 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:59:54.060 INFO:teuthology.orchestra.run.smithi164.stdout:ERROR: (22) Invalid argument 2021-02-10T04:59:54.060 INFO:teuthology.orchestra.run.smithi164.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-02-10T04:59:54.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:54 smithi085 conmon[29319]: cluster 2021-02-10T04:59:52.574230+0000 mgr.y (mgr.14140) 1540 : cluster [DBG] 2021-02-10T04:59:54.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:54 smithi085 conmon[29319]: pgmap v2502: 208 pgs: 1 active+recovering+undersized+remapped, 6 remapped+peering, 2 active+recovering, 18 peering, 4 active+remapped+backfill_toofull, 177 active+clean; 857 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 217 MiB/s rd, 61 MiB/s wr, 391 op/s; 108/2166 objects misplaced (4.986%); 0 B/s, 6 objects/s recovering 2021-02-10T04:59:54.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:54 smithi085 conmon[29319]: cluster 2021-02-10T04:59:53.084847+0000 mon.a (mon.0) 3048 : cluster [DBG] osdmap e1366: 8 total, 8 up, 7 in 2021-02-10T04:59:54.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:54 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T04:59:54.381 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:54 smithi085 conmon[29319]: 10T04:59:53.644913+0000 mon.a (mon.0) 3049 : cluster [DBG] osdmap e1367: 8 total, 8 up, 7 in 2021-02-10T04:59:54.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:54 smithi085 conmon[35961]: cluster 2021-02-10T04:59: 2021-02-10T04:59:54.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:54 smithi085 conmon[35961]: 52.574230+0000 mgr.y (mgr.14140) 1540 : cluster [DBG] pgmap v2502: 208 pgs: 1 active+recovering+undersized+remapped, 6 remapped+peering, 2 active+recovering, 18 peering, 4 active+remapped+backfill_toofull, 177 active+clean; 857 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 217 MiB/s rd, 61 MiB/s wr, 391 op/s; 108/2166 objects misplaced (4.986%); 0 B/s, 6 objects/s recovering 2021-02-10T04:59:54.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:54 smithi085 conmon[35961]: cluster 2021-02-10T04:59:54.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:54 smithi085 conmon[35961]: 2021-02-10T04:59:53.084847+0000 mon.a (mon.0) 3048 : cluster [DBG] osdmap e1366: 8 total, 8 up, 7 in 2021-02-10T04:59:54.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:54 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T04:59:54.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:54 smithi085 conmon[35961]: T04:59:53.644913+0000 mon.a (mon 2021-02-10T04:59:54.382 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:54 smithi085 conmon[35961]: .0) 3049 : cluster [DBG] osdmap e1367: 8 total, 8 up, 7 in 2021-02-10T04:59:54.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:54 smithi164 conmon[29288]: cluster 2021-02-10T04:59:54.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:54 smithi164 conmon[29288]: 2021-02-10T04: 2021-02-10T04:59:54.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:54 smithi164 conmon[29288]: 59:52.574230+0000 mgr.y (mgr.14140) 1540 : cluster [DBG] pgmap v2502: 208 pgs: 1 active+recovering+undersized+remapped, 6 remapped+peering, 2 active+recovering, 18 peering, 4 active+remapped+backfill_toofull, 177 active+clean; 857 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 217 MiB/s rd, 61 MiB/s wr, 391 op/s; 108/2166 objects misplaced (4.986%); 0 B/s, 6 objects/s recovering 2021-02-10T04:59:54.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:54 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:59:54.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:54 smithi164 conmon[29288]: 10T04:59:53.084847+0000 mon.a (mon. 2021-02-10T04:59:54.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:54 smithi164 conmon[29288]: 0) 3048 : cluster [DBG] osdmap e1366: 8 total, 8 up, 7 in 2021-02-10T04:59:54.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:54 smithi164 conmon[29288]: cluster 2021-02-10T04:59:54.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:54 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T04:59:54.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:54 smithi164 conmon[29288]: 04:59:53.644913+0000 mon.a (mon.0) 3049 : 2021-02-10T04:59:54.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:54 smithi164 conmon[29288]: cluster [DBG] osdmap e1367: 8 total, 8 up, 7 in 2021-02-10T04:59:55.862 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T04:59:55.880 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T04:59:55.899 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T04:59:55.917 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T04:59:55.937 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T04:59:55.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:55 smithi085 conmon[35961]: cluster 2021-02-10T04:59:54.574698+0000 mgr.y (mgr.14140) 1541 : cluster [DBG] pgmap v2505: 208 pgs: 2 active+clean+remapped, 2 active+recovering+undersized+remapped, 6 remapped+peering, 2 active+recovering, 20 peering, 4 active+remapped+backfill_toofull, 172 active+clean; 857 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 111/2166 objects misplaced (5.125%); 13 MiB/s, 57 objects/s recovering 2021-02-10T04:59:55.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:55 smithi085 conmon[35961]: cluster 2021-02-10T04:59:54.655089+0000 mon.a (mon.0) 3050 : cluster [DBG] osdmap e1368: 8 total, 8 up, 7 in 2021-02-10T04:59:55.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:55 smithi085 conmon[35961]: cephadm 2021-02-10T04:59:54.675135+0000 mgr.y (mgr.14140) 1542 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:59:55.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:55 smithi085 conmon[35961]: audit 2021-02-10T04:59:54.675636+0000 mon.a (mon.0) 3051 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:59:55.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:55 smithi085 conmon[29319]: cluster 2021-02-10T04:59:54.574698+0000 mgr.y (mgr.14140) 1541 : cluster [DBG] pgmap v2505: 208 pgs: 2 active+clean+remapped, 2 active+recovering+undersized+remapped, 6 remapped+peering, 2 active+recovering, 20 peering, 4 active+remapped+backfill_toofull, 172 active+clean; 857 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 111/2166 objects misplaced (5.125%); 13 MiB/s, 57 objects/s recovering 2021-02-10T04:59:55.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:55 smithi085 conmon[29319]: cluster 2021-02-10T04:59:54.655089+0000 mon.a (mon.0) 3050 : cluster [DBG] osdmap e1368: 8 total, 8 up, 7 in 2021-02-10T04:59:55.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:55 smithi085 conmon[29319]: cephadm 2021-02-10T04:59:54.675135+0000 mgr.y (mgr.14140) 1542 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:59:55.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:55 smithi085 conmon[29319]: audit 2021- 2021-02-10T04:59:55.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:55 smithi085 conmon[29319]: 02-10T04:59:54.675636+0000 mon.a (mon.0) 3051 : audit [DBG] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:59:55.962 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T04:59:55.984 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T04:59:56.006 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T04:59:56.008 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T04:59:56.008 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T04:59:56.008 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T04:59:56.008 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T04:59:56.008 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T04:59:56.008 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T04:59:56.008 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T04:59:56.008 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T04:59:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:55 smithi164 conmon[29288]: cluster 2021-02-10T04:59 2021-02-10T04:59:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:55 smithi164 conmon[29288]: :54.574698+0000 mgr.y (mgr.14140) 1541 : cluster [DBG] pgmap v2505: 208 pgs: 2 active+clean+remapped, 2 active+recovering+undersized+remapped, 6 remapped+peering, 2 active+recovering, 20 peering, 4 active+remapped+backfill_toofull, 172 active+clean; 857 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 111/2166 objects misplaced (5.125%); 13 MiB/s, 57 objects/s recovering 2021-02-10T04:59:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:55 smithi164 conmon[29288]: cluster 2021-02-10T04:59:54.655089+0000 mon.a (mon.0) 3050 : cluster [DBG] osdmap e1368: 8 total, 8 up, 7 in 2021-02-10T04:59:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:55 smithi164 conmon[29288]: cephadm 2021-02-10T04:59:54.675135+0000 mgr.y (mgr.14140) 1542 : cephadm [INF] Refreshing smithi085 facts 2021-02-10T04:59:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:55 smithi164 conmon[29288]: audit 2021-02-10T04:59:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:55 smithi164 conmon[29288]: 2021-02-10T04:59:54.675636+0000 2021-02-10T04:59:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:55 smithi164 conmon[29288]: mon.a (mon.0) 3051 : audit [DBG] 2021-02-10T04:59:56.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:55 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "container_image"}]: dispatch 2021-02-10T04:59:56.145 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:59:55 smithi164 conmon[44544]: debug 2021-02-10T04:59:56.145 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 04:59:55 smithi164 conmon[44544]: 2021-02-10T04:59:55.726+0000 7f64ab094700 -1 osd.7 pg_epoch: 1369 pg[2.5( v 53'2 (0'0,53'2] local-lis/les=1332/1333 n=0 ec=48/48 lis/c=1332/1332 les/c/f=1333/1333/0 sis=1332) [7,0,4] r=0 lpr=1332 crt=53'2 lcod 53'1 mlcod 0'0 active+clean] scrubber pg(2.5) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x55cab7208840) 2021-02-10T04:59:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[29319]: cluster 2021-02-10T04:59:55.163987+0000 2021-02-10T04:59:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[29319]: osd.2 (osd.2) 2041 : cluster [DBG] 2.10 scrub ok 2021-02-10T04:59:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[29319]: cluster 2021-02-10T04:59:55.660631+0000 mon.a (mon.0) 3052 : cluster [DBG] 2021-02-10T04:59:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[29319]: osdmap e1369: 8 total, 8 up, 7 in 2021-02-10T04:59:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[29319]: audit 2021-02-10T 2021-02-10T04:59:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[29319]: 04:59:55.844153+0000 mon.a (mon.0) 3053 : audit [DBG] from='client.? 172.21.15.85:0/1893944317' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:59:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[29319]: 2021-02-10T04:59:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[29319]: cluster 2021-02-10T04:59:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[29319]: 2021-02-10T04:59:56.057285+0000 osd.0 ( 2021-02-10T04:59:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[29319]: osd.0) 8 : cluster [DBG] 6.0 scrub ok 2021-02-10T04:59:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:59:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[29319]: 02-10T04:59:56.125713+0000 2021-02-10T04:59:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[29319]: osd.4 (osd.4) 2669 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:59:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[29319]: 2021-02-10T04:59:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[29319]: cluster 2021-02-10T04:59:56. 2021-02-10T04:59:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[29319]: 127227+0000 osd.4 (osd.4) 2670 : cluster 2021-02-10T04:59:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[29319]: [DBG] purged_snaps scrub found 2 strays 2021-02-10T04:59:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[29319]: cluster 2021-02-10T04:59 2021-02-10T04:59:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[29319]: :56.274655+0000 osd.4 (osd.4 2021-02-10T04:59:56.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[29319]: ) 2671 : cluster [DBG] 2.f scrub ok 2021-02-10T04:59:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: cluster 2021-02-10T04:59:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: 2021-02-10T04:59:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: 2021 2021-02-10T04:59:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: - 2021-02-10T04:59:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: 02 2021-02-10T04:59:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: - 2021-02-10T04:59:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: 10 2021-02-10T04:59:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: T 2021-02-10T04:59:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: 04 2021-02-10T04:59:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: : 2021-02-10T04:59:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: 59 2021-02-10T04:59:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: : 2021-02-10T04:59:56.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: 55 2021-02-10T04:59:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: . 2021-02-10T04:59:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: 163987 2021-02-10T04:59:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: +0000 2021-02-10T04:59:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: 2021-02-10T04:59:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: osd.2 2021-02-10T04:59:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: ( 2021-02-10T04:59:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: osd 2021-02-10T04:59:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: . 2021-02-10T04:59:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: 2 2021-02-10T04:59:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: ) 2021-02-10T04:59:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: 2041 2021-02-10T04:59:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: : cluster [DBG] 2.10 scrub ok 2021-02-10T04:59:56.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: cluster 2021-02-10T04:59:56.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: 2021-02-10T04:59: 2021-02-10T04:59:56.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: 55.660631+0000 mon.a (mon 2021-02-10T04:59:56.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: .0) 3052 : cluster [DBG] 2021-02-10T04:59:56.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: osdmap e1369: 8 total, 8 up, 7 in 2021-02-10T04:59:56.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: audit 2021-02-10T04:59:56.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: 2021-02-10T04 2021-02-10T04:59:56.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: :59:55.844153+0000 2021-02-10T04:59:56.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: mon.a (mon.0) 3053 : 2021-02-10T04:59:56.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: audit [DBG] from='client.? 172.21.15.85:0/1893944317' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:59:56.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: cluster 2021-02-10T04:59:56.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: 2021-02-10T04: 2021-02-10T04:59:56.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: 59:56.057285+0000 osd.0 2021-02-10T04:59:56.946 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: (osd.0) 8 : cluster 2021-02-10T04:59:56.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: [DBG] 6.0 scrub ok 2021-02-10T04:59:56.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:59:56.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: 02-10T04:59:56.125713+0000 2021-02-10T04:59:56.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: osd.4 (osd.4) 2669 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:59:56.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:59:56.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: -10T04:59:56.127227+0000 osd.4 (osd 2021-02-10T04:59:56.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: .4) 2670 : cluster [DBG] purged_snaps scrub found 2 strays 2021-02-10T04:59:56.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: 2021-02-10T04:59:56.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:59:56.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: 10T04:59:56. 2021-02-10T04:59:56.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: 274655+0000 osd.4 (osd.4) 2021-02-10T04:59:56.947 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:56 smithi085 conmon[35961]: 2671 : cluster [DBG] 2.f scrub ok 2021-02-10T04:59:57.026 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 2] out_osds: [6] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2021-02-10T04:59:57.026 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-02-10T04:59:57.026 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd dump -f json-pretty 2021-02-10T04:59:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:56 smithi164 conmon[29288]: cluster 2021-02-10T04:59:55.163987+0000 osd.2 ( 2021-02-10T04:59:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:56 smithi164 conmon[29288]: osd.2) 2041 : cluster [DBG] 2.10 scrub ok 2021-02-10T04:59:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:56 smithi164 conmon[29288]: cluster 2021-02-10T04:59:55.660631+0000 mon.a (mon.0) 3052 : cluster [DBG] osdmap e1369: 8 total, 8 up, 7 in 2021-02-10T04:59:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:56 smithi164 conmon[29288]: audit 2021-02-10T04:59:55.844153+0000 mon.a (mon.0 2021-02-10T04:59:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:56 smithi164 conmon[29288]: ) 3053 : audit [DBG] from='client.? 172.21.15.85:0/1893944317' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T04:59:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:56 smithi164 conmon[29288]: cluster 2021-02-10T04:59:56 2021-02-10T04:59:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:56 smithi164 conmon[29288]: .057285+0000 osd.0 (osd.0) 8 : cluster [DBG] 6.0 scrub ok 2021-02-10T04:59:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:56 smithi164 conmon[29288]: cluster 2021-02-10T04:59:56.125713+0000 osd.4 (osd.4) 2669 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:59:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:56 smithi164 conmon[29288]: cluster 2021-02-10T04:59:56.127227+0000 osd.4 (osd.4) 2670 : cluster [DBG] purged_snaps scrub found 2 strays 2021-02-10T04:59:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:56 smithi164 conmon[29288]: cluster 2021-02-10T04:59:56.274655+0000 osd.4 (osd.4) 2671 : cluster [DBG] 2.f scrub ok 2021-02-10T04:59:57.259 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:59:57.747 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T04:59:57.748 INFO:teuthology.orchestra.run.smithi085.stdout:{"epoch":1371,"fsid":"42224fce-6b57-11eb-8feb-001a4aab830c","created":"2021-02-10T04:20:57.933279+0000","modified":"2021-02-10T04:59:57.661214+0000","last_up_change":"2021-02-10T04:58:55.817746+0000","last_in_change":"2021-02-10T04:59:50.064782+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-10T04:24:05.258028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"1266","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"110","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2021-02-10T04:25:32.688554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":42,"pg_placement_num":42,"pg_placement_num_target":42,"pg_num_target":42,"pg_num_pending":42,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"1272","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"89","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"iscsi","create_time":"2021-02-10T04:25:47.658613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":13,"pg_placement_num":13,"pg_placement_num_target":13,"pg_num_target":13,"pg_num_pending":13,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"1211","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"1209","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"zone.rgw.log","create_time":"2021-02-10T04:25:47.968727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"4.16","ready_epoch":1256,"last_epoch_started":1255,"last_epoch_clean":1255,"source_version":"1048'40","target_version":"1049'40"},"last_change":"1343","last_force_op_resend":"0","last_force_op_resend_prenautilus":"1256","last_force_op_resend_preluminous":"1341","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"zone.rgw.control","create_time":"2021-02-10T04:25:49.278260+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"zone.rgw.meta","create_time":"2021-02-10T04:25:51.792514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":42,"pg_placement_num":42,"pg_placement_num_target":42,"pg_num_target":42,"pg_num_pending":42,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"1046","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"1044","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"unique_pool_0","create_time":"2021-02-10T04:28:23.932391+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":36,"pg_placement_num":36,"pg_placement_num_target":36,"pg_num_target":36,"pg_num_pending":36,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"1143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"1141","auid":0,"snap_mode":"selfmanaged","snap_seq":888,"snap_epoch":994,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"45d0bb5f-35fa-4596-964f-75abcee53bee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":1227,"last_clean_end":1315,"up_from":1332,"up_thru":1366,"down_at":1316,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6801","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6802","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6804","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6803","nonce":7}]},"public_addr":"172.21.15.85:6801/7","cluster_addr":"172.21.15.85:6802/7","heartbeat_back_addr":"172.21.15.85:6804/7","heartbeat_front_addr":"172.21.15.85:6803/7","state":["exists","up"]},{"osd":1,"uuid":"3421a5ce-872e-4b52-adf7-6aaa595871a4","up":1,"in":1,"weight":1,"primary_affinity":0.1644744873046875,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1367,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6805","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6806","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6808","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6807","nonce":7}]},"public_addr":"172.21.15.85:6805/7","cluster_addr":"172.21.15.85:6806/7","heartbeat_back_addr":"172.21.15.85:6808/7","heartbeat_front_addr":"172.21.15.85:6807/7","state":["exists","up"]},{"osd":2,"uuid":"96e1952a-1042-4316-9fc9-86813e88b912","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":1368,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6809","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6810","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6812","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6811","nonce":7}]},"public_addr":"172.21.15.85:6809/7","cluster_addr":"172.21.15.85:6810/7","heartbeat_back_addr":"172.21.15.85:6812/7","heartbeat_front_addr":"172.21.15.85:6811/7","state":["exists","up"]},{"osd":3,"uuid":"e019f49f-5dd4-4104-9ced-94c710616642","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":1094,"up_from":1111,"up_thru":1366,"down_at":1095,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6813","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6814","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6816","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.85:6815","nonce":7}]},"public_addr":"172.21.15.85:6813/7","cluster_addr":"172.21.15.85:6814/7","heartbeat_back_addr":"172.21.15.85:6816/7","heartbeat_front_addr":"172.21.15.85:6815/7","state":["exists","up"]},{"osd":4,"uuid":"91c8a80a-37b6-4c30-a32a-647b2e6d801f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":1370,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6801","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6803","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":7}]},"public_addr":"172.21.15.164:6800/7","cluster_addr":"172.21.15.164:6801/7","heartbeat_back_addr":"172.21.15.164:6803/7","heartbeat_front_addr":"172.21.15.164:6802/7","state":["exists","up"]},{"osd":5,"uuid":"cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":1366,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6805","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":7}]},"public_addr":"172.21.15.164:6804/7","cluster_addr":"172.21.15.164:6805/7","heartbeat_back_addr":"172.21.15.164:6807/7","heartbeat_front_addr":"172.21.15.164:6806/7","state":["exists","up"]},{"osd":6,"uuid":"adc40430-3eeb-4003-a824-343021243077","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":1360,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6809","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6811","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":7}]},"public_addr":"172.21.15.164:6808/7","cluster_addr":"172.21.15.164:6809/7","heartbeat_back_addr":"172.21.15.164:6811/7","heartbeat_front_addr":"172.21.15.164:6810/7","state":["exists","up"]},{"osd":7,"uuid":"a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37","up":1,"in":1,"weight":1,"primary_affinity":0.846160888671875,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":1367,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":7}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6813","nonce":7}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":7}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":7}]},"public_addr":"172.21.15.164:6812/7","cluster_addr":"172.21.15.164:6813/7","heartbeat_back_addr":"172.21.15.164:6815/7","heartbeat_front_addr":"172.21.15.164:6814/7","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:59:56.261166+0000","dead_epoch":1316},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:59:01.695880+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:59:20.883571+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:59:00.027876+0000","dead_epoch":1095},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:59:56.313080+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:59:12.932203+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":65536,"last_purged_snaps_scrub":"2021-02-10T04:59:31.289424+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-10T04:59:49.655402+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.23","osds":[3,0,6]}],"pg_upmap_items":[{"pgid":"2.7","mappings":[{"from":7,"to":4}]},{"pgid":"2.1a","mappings":[{"from":7,"to":2}]},{"pgid":"2.1d","mappings":[{"from":7,"to":2}]},{"pgid":"2.21","mappings":[{"from":7,"to":1}]},{"pgid":"2.28","mappings":[{"from":7,"to":2}]},{"pgid":"4.5","mappings":[{"from":4,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":5}]},{"pgid":"5.8","mappings":[{"from":1,"to":3}]},{"pgid":"5.10","mappings":[{"from":7,"to":3}]},{"pgid":"5.18","mappings":[{"from":4,"to":3}]},{"pgid":"5.1b","mappings":[{"from":0,"to":3}]},{"pgid":"7.5","mappings":[{"from":0,"to":2}]},{"pgid":"7.6","mappings":[{"from":3,"to":1}]},{"pgid":"7.15","mappings":[{"from":7,"to":4}]}],"pg_temp":[{"pgid":"4.c","osds":[4,3,6]},{"pgid":"4.e","osds":[4,1,6]},{"pgid":"4.16","osds":[0,1,2]},{"pgid":"4.18","osds":[4,1,6]},{"pgid":"4.19","osds":[3,1,4]},{"pgid":"4.1b","osds":[4,1,0]},{"pgid":"4.1e","osds":[1,4,6]}],"primary_temp":[],"blocklist":{"172.21.15.85:0/801393150":"2021-02-11T04:21:37.851607+0000","172.21.15.85:0/3049671696":"2021-02-11T04:21:37.851607+0000","172.21.15.85:0/4056505602":"2021-02-11T04:21:20.700478+0000","172.21.15.85:6800/7":"2021-02-11T04:21:20.700478+0000","172.21.15.85:0/1879026062":"2021-02-11T04:21:20.700478+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-10T04:59:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[35961]: cluster 2021-02-10T04:59:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[35961]: 2021-02-10T04:59:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[35961]: 2021-02-10T04:59:56. 2021-02-10T04:59:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[35961]: 256768+0000 osd.0 (osd.0) 9 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:59:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:59:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[35961]: 02-10T04:59:56.261162+0000 2021-02-10T04:59:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[35961]: osd.0 (osd.0) 10 : cluster [DBG] 2021-02-10T04:59:57.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T04:59:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[35961]: cluster 2021-02-10T04 2021-02-10T04:59:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[35961]: :59:56.575185+0000 mgr.y (mgr. 2021-02-10T04:59:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[35961]: 14140) 1543 : cluster [DBG] pgmap v2508: 208 pgs: 2 active+clean+remapped, 2 active+recovering+undersized+remapped, 5 remapped+peering, 2 peering, 4 active+remapped+backfill_toofull, 193 active+clean; 857 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 94/2166 objects misplaced (4.340%); 62 MiB/s, 145 objects/s recovering 2021-02-10T04:59:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:59:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[35961]: 10T04:59:56.656604+0000 mon.a ( 2021-02-10T04:59:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[35961]: mon.0) 3054 : cluster [WRN] Health check update: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-02-10T04:59:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[35961]: cluster 2021-02-10T04:59:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[35961]: 2021-02-10T04:59:56 2021-02-10T04:59:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[35961]: .665919+0000 mon.a (mon.0) 3055 2021-02-10T04:59:57.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[35961]: : cluster [DBG] osdmap e1370: 8 total, 8 up, 7 in 2021-02-10T04:59:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[29319]: cluster 2021-02-10T04:59:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:59:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[29319]: 04:59:56.256768+0000 osd.0 (osd.0) 9 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:59:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[29319]: cluster 2021-02-10T04:59:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T04:59:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[29319]: 04:59:56.261162+0000 osd.0 (osd.0) 10 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:59:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[29319]: 2021-02-10T04:59:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:59:57.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[29319]: 02-10T04:59:56.575185+0000 2021-02-10T04:59:57.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[29319]: mgr.y (mgr. 2021-02-10T04:59:57.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[29319]: 14140 2021-02-10T04:59:57.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[29319]: ) 2021-02-10T04:59:57.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[29319]: 1543 2021-02-10T04:59:57.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[29319]: : 2021-02-10T04:59:57.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[29319]: cluster [DBG] 2021-02-10T04:59:57.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[29319]: pgmap v2508: 208 pgs: 2 active+clean+remapped, 2 active+recovering+undersized+remapped, 5 remapped+peering, 2 peering, 4 active+remapped+backfill_toofull, 193 active+clean; 857 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 94/2166 objects misplaced (4.340%); 62 MiB/s, 145 objects/s recovering 2021-02-10T04:59:57.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[29319]: cluster 2021- 2021-02-10T04:59:57.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[29319]: 02-10T04:59:56.656604 2021-02-10T04:59:57.945 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[29319]: +0000 mon.a (mon.0) 3054 : cluster 2021-02-10T04:59:57.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[29319]: [WRN] Health check update: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-02-10T04:59:57.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[29319]: cluster 2021 2021-02-10T04:59:57.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[29319]: -02-10T04:59:56 2021-02-10T04:59:57.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[29319]: .665919+0000 mon.a (mon.0 2021-02-10T04:59:57.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[29319]: ) 3055 : cluster 2021-02-10T04:59:57.946 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:57 smithi085 conmon[29319]: [DBG] osdmap e1370: 8 total, 8 up, 7 in 2021-02-10T04:59:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:57 smithi164 conmon[29288]: cluster 2021-02-10T04:59:56.256768+0000 osd.0 (osd 2021-02-10T04:59:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:57 smithi164 conmon[29288]: .0) 9 : cluster [DBG] purged_snaps scrub starts 2021-02-10T04:59:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:57 smithi164 conmon[29288]: cluster 2021-02-10T04:59:56.261162+0000 osd.0 (osd.0) 10 : cluster [DBG] purged_snaps scrub ok 2021-02-10T04:59:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:57 smithi164 conmon[29288]: cluster 2021-02-10T04:59:56.575185+0000 mgr.y (mgr.14140 2021-02-10T04:59:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:57 smithi164 conmon[29288]: ) 1543 : cluster [DBG] pgmap v2508: 208 pgs: 2 active+clean+remapped, 2 active+recovering+undersized+remapped, 5 remapped+peering, 2 peering, 4 active+remapped+backfill_toofull, 193 active+clean; 857 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 94/2166 objects misplaced (4.340%); 62 MiB/s, 145 objects/s recovering 2021-02-10T04:59:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:57 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:59:58.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:57 smithi164 conmon[29288]: 59:56.656604+0000 mon.a (mon.0) 2021-02-10T04:59:58.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:57 smithi164 conmon[29288]: 3054 : cluster [WRN] Health check update: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-02-10T04:59:58.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:57 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T04:59:58.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:57 smithi164 conmon[29288]: 04:59:56.665919+0000 mon.a (mon.0 2021-02-10T04:59:58.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:57 smithi164 conmon[29288]: ) 3055 : cluster [DBG] osdmap e1370: 8 total, 8 up, 7 in 2021-02-10T04:59:58.688 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is unset 2021-02-10T04:59:58.850 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-02-10T04:59:58.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:58 smithi085 conmon[29319]: cluster 2021-02-10T04:59:57.159900+0000 osd.2 (osd.2) 2042 : cluster [DBG] 2.1d scrub ok 2021-02-10T04:59:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:58 smithi085 conmon[29319]: cluster 2021-02-10T04:59:57.666024+0000 mon.a (mon.0) 3056 : cluster [DBG] osdmap e1371: 8 total, 8 up, 7 in 2021-02-10T04:59:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:58 smithi085 conmon[29319]: audit 2021-02-10T04:59:57.745884+0000 mon.c (mon.1) 169 : audit [DBG] from='client.? 172.21.15.85:0/122221145' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-02-10T04:59:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:58 smithi085 conmon[29319]: audit 2021-02-10T04:59:57.773227+0000 mon.c (mon.1) 170 : audit [INF] from='client.? 172.21.15.85:0/2442000065' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:59:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:58 smithi085 conmon[29319]: audit 2021-02-10T04:59:57.773558+0000 mon.a (mon.0) 3057 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:59:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:58 smithi085 conmon[29319]: cluster 2021-02-10T04:59:58.274898+0000 osd.4 (osd.4) 2672 : cluster [DBG] 7.16 scrub ok 2021-02-10T04:59:58.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:58 smithi085 conmon[29319]: cluster 2021-02-10T04:59:58.640567+0000 mon.a (mon.0) 3058 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-02-10T04:59:58.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:58 smithi085 conmon[35961]: cluster 2021-02-10T04:59:58.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:58 smithi085 conmon[35961]: 2021-02-10T04:59 2021-02-10T04:59:58.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:58 smithi085 conmon[35961]: :57.159900+0000 osd.2 (osd.2) 2042 : cluster [DBG] 2.1d scrub ok 2021-02-10T04:59:58.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:58 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:59:58.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:58 smithi085 conmon[35961]: 10T04:59:57.666024+0000 mon.a (mon. 2021-02-10T04:59:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:58 smithi085 conmon[35961]: 0) 3056 : cluster [DBG] osdmap e1371: 8 total, 8 up, 7 in 2021-02-10T04:59:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:58 smithi085 conmon[35961]: audit 2021-02 2021-02-10T04:59:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:58 smithi085 conmon[35961]: -10T04:59:57.745884+0000 mon.c (mon.1 2021-02-10T04:59:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:58 smithi085 conmon[35961]: ) 169 : audit [DBG] from='client.? 172.21.15.85:0/122221145' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-02-10T04:59:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:58 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T04:59:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:58 smithi085 conmon[35961]: 10T04:59:57.773227+0000 mon.c (mon. 2021-02-10T04:59:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:58 smithi085 conmon[35961]: 1) 170 : audit [INF] from='client.? 172.21.15.85:0/2442000065' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:59:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:58 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:59:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:58 smithi085 conmon[35961]: 02-10T04:59:57.773558+0000 mon.a 2021-02-10T04:59:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:58 smithi085 conmon[35961]: (mon.0) 3057 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:59:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:58 smithi085 conmon[35961]: cluster 2021- 2021-02-10T04:59:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:58 smithi085 conmon[35961]: 02-10T04:59:58.274898+0000 2021-02-10T04:59:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:58 smithi085 conmon[35961]: osd.4 (osd.4) 2672 : cluster [DBG] 7.16 scrub ok 2021-02-10T04:59:58.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:58 smithi085 conmon[35961]: cluster 2021-02-10T04:59:58.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:58 smithi085 conmon[35961]: 2021-02-10T04:59:58.640567+0000 2021-02-10T04:59:58.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:58 smithi085 conmon[35961]: mon.a (mon.0) 3058 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-02-10T04:59:59.055 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T04:59:59.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:58 smithi164 conmon[29288]: cluster 2021-02-10T04:59:57.159900+0000 osd.2 (osd. 2021-02-10T04:59:59.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:58 smithi164 conmon[29288]: 2) 2042 : cluster [DBG] 2.1d scrub ok 2021-02-10T04:59:59.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:58 smithi164 conmon[29288]: cluster 2021-02-10T04:59:57.666024+0000 mon.a (mon.0) 3056 : cluster [DBG] osdmap e1371: 8 total, 8 up, 7 in 2021-02-10T04:59:59.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:58 smithi164 conmon[29288]: audit 2021-02-10T04:59:57.745884+0000 mon.c (mon.1) 2021-02-10T04:59:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:58 smithi164 conmon[29288]: 169 : audit [DBG] from='client.? 172.21.15.85:0/122221145' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-02-10T04:59:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:58 smithi164 conmon[29288]: audit 2021-02-10T04:59:57.773227 2021-02-10T04:59:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:58 smithi164 conmon[29288]: +0000 mon.c (mon.1) 170 : audit [INF] from='client.? 172.21.15.85:0/2442000065' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:59:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:58 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T04:59:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:58 smithi164 conmon[29288]: T04:59:57.773558+0000 mon.a (mon. 2021-02-10T04:59:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:58 smithi164 conmon[29288]: 0) 3057 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T04:59:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:58 smithi164 conmon[29288]: cluster 2021-02-10T04 2021-02-10T04:59:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:58 smithi164 conmon[29288]: :59:58.274898+0000 osd.4 (osd.4 2021-02-10T04:59:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:58 smithi164 conmon[29288]: ) 2672 : cluster [DBG] 7.16 scrub ok 2021-02-10T04:59:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:58 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:59:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:58 smithi164 conmon[29288]: -10T04:59:58.640567+0000 2021-02-10T04:59:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:58 smithi164 conmon[29288]: mon.a (mon.0) 3058 : cluster 2021-02-10T04:59:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:58 smithi164 conmon[29288]: [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-02-10T04:59:59.811 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:59 smithi085 conmon[29319]: cluster 2021-02-10T04:59:58.575720+0000 mgr.y (mgr.14140) 2021-02-10T04:59:59.811 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:59 smithi085 conmon[29319]: 1544 : cluster [DBG] pgmap v2511: 208 pgs: 1 active+clean+snaptrim, 1 peering, 7 active+remapped+backfill_toofull, 199 active+clean; 859 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 517 KiB/s wr, 1 op/s; 120/2169 objects misplaced (5.533%); 60 MiB/s, 136 objects/s recovering 2021-02-10T04:59:59.811 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:59 smithi085 conmon[29319]: cluster 2021-02-10T04:59:58.577480+0000 osd.1 (osd.1) 525 : cluster [DBG] 7.b scrub ok 2021-02-10T04:59:59.811 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:59 smithi085 conmon[29319]: cluster 2021-02-10T04:59:58.670549+0000 mon.a (mon.0) 3059 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2021-02-10T04:59:59.811 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:59 smithi085 conmon[29319]: cluster 2021-02-10T04:59:58.671208+0000 mon.a (mon.0) 3060 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:59:59.811 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:59 smithi085 conmon[29319]: audit 2021-02-10T04:59:58.683742+0000 mon.a (mon.0) 3061 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:59:59.811 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 04:59:59 smithi085 conmon[29319]: cluster 2021-02-10T04:59:58.683799+0000 mon.a (mon.0) 3062 : cluster [DBG] osdmap e1372: 8 total, 8 up, 7 in 2021-02-10T04:59:59.812 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:59 smithi085 conmon[35961]: cluster 2021-02-10T04:59:58.575720+0000 mgr.y 2021-02-10T04:59:59.812 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:59 smithi085 conmon[35961]: (mgr.14140) 1544 : cluster [DBG] pgmap v2511: 208 pgs: 1 active+clean+snaptrim, 1 peering, 7 active+remapped+backfill_toofull, 199 active+clean; 859 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 517 KiB/s wr, 1 op/s; 120/2169 objects misplaced (5.533%); 60 MiB/s, 136 objects/s recovering 2021-02-10T04:59:59.812 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:59 smithi085 conmon[35961]: cluster 2021-02-10T04:59: 2021-02-10T04:59:59.812 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:59 smithi085 conmon[35961]: 58.577480+0000 osd.1 (osd.1) 525 : cluster [DBG] 7.b scrub ok 2021-02-10T04:59:59.812 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:59 smithi085 conmon[35961]: cluster 2021-02-10T04: 2021-02-10T04:59:59.812 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:59 smithi085 conmon[35961]: 59:58.670549+0000 mon.a (mon.0) 2021-02-10T04:59:59.812 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:59 smithi085 conmon[35961]: 3059 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2021-02-10T04:59:59.812 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:59 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T04:59:59.812 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:59 smithi085 conmon[35961]: 10T04:59:58.671208 2021-02-10T04:59:59.812 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:59 smithi085 conmon[35961]: +0000 mon.a (mon.0) 3060 2021-02-10T04:59:59.812 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:59 smithi085 conmon[35961]: : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:59:59.812 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:59 smithi085 conmon[35961]: audit 2021- 2021-02-10T04:59:59.813 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:59 smithi085 conmon[35961]: 02-10T04:59:58.683742+0000 2021-02-10T04:59:59.813 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:59 smithi085 conmon[35961]: mon.a (mon.0) 3061 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:59:59.813 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:59 smithi085 conmon[35961]: 2021-02-10T04:59:59.813 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:59 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T04:59:59.813 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:59 smithi085 conmon[35961]: -10T04:59:58.683799+0000 mon.a (mon 2021-02-10T04:59:59.813 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 04:59:59 smithi085 conmon[35961]: .0) 3062 : cluster [DBG] osdmap e1372: 8 total, 8 up, 7 in 2021-02-10T04:59:59.894 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 04:59:59 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T04:59:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:59 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T04:59:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:59 smithi164 conmon[29288]: 10T04:59:58.575720+0000 mgr.y (mgr.14140) 1544 : cluster [DBG] pgmap v2511: 208 pgs: 1 active+clean+snaptrim, 1 peering, 7 active+remapped+backfill_toofull, 199 active+clean; 859 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 517 KiB/s wr, 1 op/s; 120/2169 objects misplaced (5.533%); 60 MiB/s, 136 objects/s recovering 2021-02-10T04:59:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:59 smithi164 conmon[29288]: cluster 2021-02-10T04:59:58.577480+0000 osd.1 2021-02-10T04:59:59.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:59 smithi164 conmon[29288]: (osd.1) 525 : cluster [DBG] 7.b scrub ok 2021-02-10T04:59:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:59 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T04:59:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:59 smithi164 conmon[29288]: T04:59:58.670549+0000 mon.a (mon. 2021-02-10T04:59:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:59 smithi164 conmon[29288]: 0) 3059 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2021-02-10T04:59:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:59 smithi164 conmon[29288]: cluster 2021-02-10T04: 2021-02-10T04:59:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:59 smithi164 conmon[29288]: 59:58.671208+0000 mon.a (mon.0) 3060 2021-02-10T04:59:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:59 smithi164 conmon[29288]: : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T04:59:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:59 smithi164 conmon[29288]: audit 2021-02-10T04:59 2021-02-10T04:59:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:59 smithi164 conmon[29288]: :58.683742+0000 mon.a (mon 2021-02-10T04:59:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:59 smithi164 conmon[29288]: .0) 3061 : audit [INF] 2021-02-10T04:59:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:59 smithi164 conmon[29288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T04:59:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:59 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T04:59:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:59 smithi164 conmon[29288]: -10T04:59:58. 2021-02-10T04:59:59.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:59 smithi164 conmon[29288]: 683799+0000 mon.a (mon.0 2021-02-10T04:59:59.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 04:59:59 smithi164 conmon[29288]: ) 3062 : cluster [DBG] osdmap e1372: 8 total, 8 up, 7 in 2021-02-10T04:59:59.979 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T05:00:00.200 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:00:01.016 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[29319]: cluster 2021-02-10T04:59:59.241022+0000 osd.2 (osd.2) 2043 : cluster [DBG] 5.8 scrub ok 2021-02-10T05:00:01.016 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[29319]: audit 2021-02-10T04:59:59.585017+0000 mgr.y (mgr.14140) 1545 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:00:01.016 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[29319]: cluster 2021-02-10T05:00:00.000111+0000 mon.a (mon.0) 3063 : cluster [WRN] Health detail: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull 2021-02-10T05:00:01.017 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[29319]: cluster 2021-02-10T05:00:00.000152+0000 mon.a (mon.0) 3064 : cluster [WRN] [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull 2021-02-10T05:00:01.017 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[29319]: cluster 2021-02-10T05:00:00.000172+0000 mon.a (mon.0) 3065 : cluster [WRN] pg 4.c is active+remapped+backfill_toofull, acting [4,3,6] 2021-02-10T05:00:01.017 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[29319]: cluster 2021-02-10T05:00:00.000194+0000 mon.a (mon.0) 3066 : cluster [WRN] pg 4.e is active+remapped+backfill_toofull, acting [4,1,6] 2021-02-10T05:00:01.017 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[29319]: cluster 2021-02-10T05:00:00.000212+0000 mon.a (mon.0) 3067 : cluster [WRN] pg 4.16 is active+remapped+backfill_toofull, acting [0,1,2] 2021-02-10T05:00:01.017 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[29319]: cluster 2021-02-10T05:00:00.000228+0000 mon.a (mon.0) 3068 : cluster [WRN] pg 4.18 is active+remapped+backfill_toofull, acting [4,1,6] 2021-02-10T05:00:01.017 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[29319]: cluster 2021-02-10T05:00:00.000243+0000 mon.a (mon.0) 3069 : cluster [WRN] pg 4.19 is active+remapped+backfill_toofull, acting [3,1,4] 2021-02-10T05:00:01.017 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[29319]: cluster 2021-02-10T05:00:00.000260+0000 mon.a (mon.0) 3070 : cluster [WRN] pg 4.1b is active+remapped+backfill_toofull, acting [4,1,0] 2021-02-10T05:00:01.017 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[29319]: cluster 2021-02-10T05: 2021-02-10T05:00:01.017 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[29319]: 00:00.000278+0000 mon.a (mon.0) 3071 : cluster [WRN] pg 4.1e is active+remapped+backfill_toofull, acting [1,4,6] 2021-02-10T05:00:01.017 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[29319]: cluster 2021-02-10T05:00:00.081911+0000 osd.0 (osd.0) 11 : cluster [DBG] 2.1f scrub ok 2021-02-10T05:00:01.017 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[29319]: cluster 2021-02-10T05:00:00.194382+0000 osd.3 (osd.3) 50 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:00:01.017 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[29319]: cluster 2021-02-10T05:00:00.199355+0000 osd.3 (osd.3) 51 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:00:01.017 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[29319]: cluster 2021-02-10T05:00:00.267583+0000 osd.4 (osd.4) 2673 : cluster [DBG] 5.11 scrub ok 2021-02-10T05:00:01.018 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: cluster 2021-02-10T04:59:59.241022 2021-02-10T05:00:01.018 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 2043 : cluster [DBG] 5.8 scrub ok 2021-02-10T05:00:01.018 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: audit 2021-02-10T05:00:01.018 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: 2021-02-10T04:59:59.585017+0000 mgr.y (mgr.14140) 1545 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:00:01.018 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: cluster 2021-02-10T05:00:01.018 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: 2021-02-10T05:00: 2021-02-10T05:00:01.018 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: 00.000111+0000 mon.a (mon.0) 3063 : 2021-02-10T05:00:01.019 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: cluster [WRN] Health detail: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull 2021-02-10T05:00:01.019 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: cluster 2021-02-10T05:00:00 2021-02-10T05:00:01.019 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: .000152+0000 mon.a (mon.0) 3064 : cluster 2021-02-10T05:00:01.019 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: [WRN] [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull 2021-02-10T05:00:01.019 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: cluster 2021-02-10T05:00:00 2021-02-10T05:00:01.019 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: .000172+0000 mon.a (mon.0) 3065 : cluster 2021-02-10T05:00:01.019 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: [WRN] pg 4.c is active+remapped+backfill_toofull, acting [4,3,6] 2021-02-10T05:00:01.019 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: cluster 2021-02-10T05:00 2021-02-10T05:00:01.019 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: :00.000194+0000 mon.a (mon.0) 3066 : cluster 2021-02-10T05:00:01.019 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: [WRN] pg 4.e is active+remapped+backfill_toofull, acting [4,1,6] 2021-02-10T05:00:01.019 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: cluster 2021-02-10T05:00:01.019 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: 2021-02-10T05:00:00.000212+0000 mon.a ( 2021-02-10T05:00:01.019 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: mon.0) 3067 : cluster [WRN] pg 4.16 is active+remapped+backfill_toofull, acting [0,1,2] 2021-02-10T05:00:01.020 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T05:00:01.020 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: T05:00:00.000228+0000 mon.a (mon.0) 2021-02-10T05:00:01.020 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: 3068 : cluster [WRN] pg 4.18 is active+remapped+backfill_toofull, acting [4,1,6] 2021-02-10T05:00:01.020 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T05:00:01.020 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: -10T05:00:00.000243+0000 2021-02-10T05:00:01.020 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: mon.a (mon.0) 3069 : cluster [WRN] pg 4.19 is active+remapped+backfill_toofull, acting [3,1,4] 2021-02-10T05:00:01.020 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: cluster 2021 2021-02-10T05:00:01.020 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: -02-10T05:00:00.000260 2021-02-10T05:00:01.020 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: +0000 mon.a (mon.0) 3070 : cluster 2021-02-10T05:00:01.020 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: [WRN] pg 4.1b is active+remapped+backfill_toofull, acting [4,1,0] 2021-02-10T05:00:01.020 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T05:00:01.020 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: 05:00:00.000278+0000 mon.a ( 2021-02-10T05:00:01.021 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: mon.0) 3071 : cluster [WRN] 2021-02-10T05:00:01.021 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: pg 4.1e is active+remapped+backfill_toofull, acting [1,4,6] 2021-02-10T05:00:01.021 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: cluster 2021 2021-02-10T05:00:01.021 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: -02-10T05:00: 2021-02-10T05:00:01.021 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: 00.081911+0000 osd.0 (osd 2021-02-10T05:00:01.021 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: .0) 11 : cluster [DBG] 2021-02-10T05:00:01.021 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: 2.1f scrub ok 2021-02-10T05:00:01.021 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: cluster 2021-02-10T05:00:01.021 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: 2021-02-10T05: 2021-02-10T05:00:01.021 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: 00:00.194382+0000 osd.3 ( 2021-02-10T05:00:01.021 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: osd.3) 50 : cluster [DBG] 2021-02-10T05:00:01.021 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: purged_snaps scrub starts 2021-02-10T05:00:01.021 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: cluster 2021 2021-02-10T05:00:01.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: -02-10T05:00 2021-02-10T05:00:01.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: :00.199355+0000 osd.3 (osd. 2021-02-10T05:00:01.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: 3) 51 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:00:01.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T05:00:01.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: -10T05:00:00.267583+0000 osd.4 ( 2021-02-10T05:00:01.022 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:00 smithi085 conmon[35961]: osd.4) 2673 : cluster [DBG] 5.11 scrub ok 2021-02-10T05:00:01.046 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T05:00:01.046 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T05:00:01.046 INFO:teuthology.orchestra.run.smithi085.stdout: "epoch": 1373, 2021-02-10T05:00:01.046 INFO:teuthology.orchestra.run.smithi085.stdout: "fsid": "42224fce-6b57-11eb-8feb-001a4aab830c", 2021-02-10T05:00:01.046 INFO:teuthology.orchestra.run.smithi085.stdout: "created": "2021-02-10T04:20:57.933279+0000", 2021-02-10T05:00:01.046 INFO:teuthology.orchestra.run.smithi085.stdout: "modified": "2021-02-10T05:00:00.681249+0000", 2021-02-10T05:00:01.046 INFO:teuthology.orchestra.run.smithi085.stdout: "last_up_change": "2021-02-10T04:58:55.817746+0000", 2021-02-10T05:00:01.046 INFO:teuthology.orchestra.run.smithi085.stdout: "last_in_change": "2021-02-10T04:59:50.064782+0000", 2021-02-10T05:00:01.046 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T05:00:01.047 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_num": 5799936, 2021-02-10T05:00:01.047 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_set": [ 2021-02-10T05:00:01.047 INFO:teuthology.orchestra.run.smithi085.stdout: "pglog_hardlimit", 2021-02-10T05:00:01.047 INFO:teuthology.orchestra.run.smithi085.stdout: "purged_snapdirs", 2021-02-10T05:00:01.047 INFO:teuthology.orchestra.run.smithi085.stdout: "recovery_deletes", 2021-02-10T05:00:01.047 INFO:teuthology.orchestra.run.smithi085.stdout: "sortbitwise" 2021-02-10T05:00:01.047 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T05:00:01.047 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_version": 18, 2021-02-10T05:00:01.047 INFO:teuthology.orchestra.run.smithi085.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T05:00:01.047 INFO:teuthology.orchestra.run.smithi085.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T05:00:01.047 INFO:teuthology.orchestra.run.smithi085.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T05:00:01.047 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_snapshot": "", 2021-02-10T05:00:01.048 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_max": 7, 2021-02-10T05:00:01.048 INFO:teuthology.orchestra.run.smithi085.stdout: "max_osd": 8, 2021-02-10T05:00:01.048 INFO:teuthology.orchestra.run.smithi085.stdout: "require_min_compat_client": "luminous", 2021-02-10T05:00:01.048 INFO:teuthology.orchestra.run.smithi085.stdout: "min_compat_client": "luminous", 2021-02-10T05:00:01.048 INFO:teuthology.orchestra.run.smithi085.stdout: "require_osd_release": "quincy", 2021-02-10T05:00:01.048 INFO:teuthology.orchestra.run.smithi085.stdout: "pools": [ 2021-02-10T05:00:01.048 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.048 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 1, 2021-02-10T05:00:01.048 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "device_health_metrics", 2021-02-10T05:00:01.048 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:24:05.258028+0000", 2021-02-10T05:00:01.048 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T05:00:01.048 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T05:00:01.048 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T05:00:01.049 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T05:00:01.049 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T05:00:01.049 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T05:00:01.049 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T05:00:01.049 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T05:00:01.049 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T05:00:01.049 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T05:00:01.049 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T05:00:01.049 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T05:00:01.049 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 11, 2021-02-10T05:00:01.049 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 11, 2021-02-10T05:00:01.049 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 11, 2021-02-10T05:00:01.049 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 11, 2021-02-10T05:00:01.049 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 11, 2021-02-10T05:00:01.050 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T05:00:01.050 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T05:00:01.050 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T05:00:01.050 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T05:00:01.050 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T05:00:01.050 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T05:00:01.050 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T05:00:01.050 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.050 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "1266", 2021-02-10T05:00:01.050 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T05:00:01.050 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T05:00:01.050 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "110", 2021-02-10T05:00:01.051 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T05:00:01.051 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T05:00:01.051 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T05:00:01.051 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T05:00:01.051 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T05:00:01.051 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T05:00:01.051 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T05:00:01.051 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T05:00:01.051 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T05:00:01.051 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T05:00:01.051 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T05:00:01.051 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T05:00:01.051 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T05:00:01.052 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T05:00:01.052 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T05:00:01.052 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T05:00:01.052 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T05:00:01.052 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T05:00:01.052 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T05:00:01.052 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T05:00:01.052 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T05:00:01.052 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T05:00:01.052 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T05:00:01.052 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.052 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T05:00:01.052 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T05:00:01.053 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T05:00:01.053 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T05:00:01.053 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T05:00:01.053 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T05:00:01.053 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T05:00:01.053 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T05:00:01.053 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T05:00:01.053 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T05:00:01.053 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T05:00:01.053 INFO:teuthology.orchestra.run.smithi085.stdout: "options": { 2021-02-10T05:00:01.053 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_min": 1 2021-02-10T05:00:01.053 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.054 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T05:00:01.054 INFO:teuthology.orchestra.run.smithi085.stdout: "mgr_devicehealth": {} 2021-02-10T05:00:01.054 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.054 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.054 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.054 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 2, 2021-02-10T05:00:01.054 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": ".rgw.root", 2021-02-10T05:00:01.054 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:32.688554+0000", 2021-02-10T05:00:01.054 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T05:00:01.054 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T05:00:01.054 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T05:00:01.054 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T05:00:01.054 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T05:00:01.055 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T05:00:01.055 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T05:00:01.055 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T05:00:01.055 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T05:00:01.055 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T05:00:01.055 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T05:00:01.055 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T05:00:01.055 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 42, 2021-02-10T05:00:01.055 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 42, 2021-02-10T05:00:01.055 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 42, 2021-02-10T05:00:01.055 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 42, 2021-02-10T05:00:01.055 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 42, 2021-02-10T05:00:01.056 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T05:00:01.056 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T05:00:01.056 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T05:00:01.056 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T05:00:01.056 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T05:00:01.056 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T05:00:01.056 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T05:00:01.056 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.056 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "1272", 2021-02-10T05:00:01.056 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T05:00:01.056 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T05:00:01.056 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "89", 2021-02-10T05:00:01.057 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T05:00:01.057 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T05:00:01.057 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T05:00:01.057 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T05:00:01.057 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T05:00:01.057 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T05:00:01.057 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T05:00:01.057 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T05:00:01.057 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T05:00:01.057 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T05:00:01.057 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T05:00:01.057 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T05:00:01.058 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T05:00:01.058 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T05:00:01.058 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T05:00:01.058 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T05:00:01.058 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T05:00:01.058 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T05:00:01.058 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T05:00:01.058 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T05:00:01.058 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T05:00:01.058 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T05:00:01.058 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T05:00:01.058 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.059 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T05:00:01.059 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T05:00:01.059 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T05:00:01.059 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T05:00:01.059 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T05:00:01.059 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T05:00:01.059 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T05:00:01.059 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T05:00:01.059 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T05:00:01.059 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T05:00:01.059 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T05:00:01.059 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T05:00:01.060 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T05:00:01.060 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T05:00:01.060 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.060 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.060 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.060 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 3, 2021-02-10T05:00:01.060 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "iscsi", 2021-02-10T05:00:01.060 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:47.658613+0000", 2021-02-10T05:00:01.060 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T05:00:01.060 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T05:00:01.060 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T05:00:01.060 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T05:00:01.061 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T05:00:01.061 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T05:00:01.061 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T05:00:01.061 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T05:00:01.061 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T05:00:01.061 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T05:00:01.061 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T05:00:01.061 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T05:00:01.061 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 13, 2021-02-10T05:00:01.061 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 13, 2021-02-10T05:00:01.061 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 13, 2021-02-10T05:00:01.061 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 13, 2021-02-10T05:00:01.061 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 13, 2021-02-10T05:00:01.062 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T05:00:01.062 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T05:00:01.062 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T05:00:01.062 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T05:00:01.062 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T05:00:01.062 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T05:00:01.062 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T05:00:01.062 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.062 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "1211", 2021-02-10T05:00:01.062 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T05:00:01.062 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T05:00:01.062 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "1209", 2021-02-10T05:00:01.063 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T05:00:01.063 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T05:00:01.063 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T05:00:01.063 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T05:00:01.063 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T05:00:01.063 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T05:00:01.063 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T05:00:01.063 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T05:00:01.063 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T05:00:01.063 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T05:00:01.063 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T05:00:01.063 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T05:00:01.064 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T05:00:01.064 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T05:00:01.064 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T05:00:01.064 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T05:00:01.064 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T05:00:01.064 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T05:00:01.064 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T05:00:01.064 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T05:00:01.064 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T05:00:01.064 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T05:00:01.064 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T05:00:01.064 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.064 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T05:00:01.065 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T05:00:01.065 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T05:00:01.065 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T05:00:01.065 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T05:00:01.065 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T05:00:01.065 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T05:00:01.065 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T05:00:01.065 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T05:00:01.065 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T05:00:01.065 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T05:00:01.065 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T05:00:01.065 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T05:00:01.066 INFO:teuthology.orchestra.run.smithi085.stdout: "rbd": {} 2021-02-10T05:00:01.066 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.066 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.066 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.066 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 4, 2021-02-10T05:00:01.066 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "zone.rgw.log", 2021-02-10T05:00:01.066 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:47.968727+0000", 2021-02-10T05:00:01.066 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T05:00:01.066 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T05:00:01.066 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T05:00:01.066 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T05:00:01.066 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T05:00:01.066 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T05:00:01.067 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T05:00:01.067 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T05:00:01.067 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T05:00:01.067 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T05:00:01.067 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T05:00:01.067 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T05:00:01.067 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 32, 2021-02-10T05:00:01.067 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 32, 2021-02-10T05:00:01.067 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 32, 2021-02-10T05:00:01.067 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 32, 2021-02-10T05:00:01.067 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 32, 2021-02-10T05:00:01.067 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T05:00:01.067 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "4.16", 2021-02-10T05:00:01.068 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 1256, 2021-02-10T05:00:01.068 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 1255, 2021-02-10T05:00:01.068 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 1255, 2021-02-10T05:00:01.068 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "1048'40", 2021-02-10T05:00:01.068 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "1049'40" 2021-02-10T05:00:01.068 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.068 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "1343", 2021-02-10T05:00:01.068 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T05:00:01.068 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "1256", 2021-02-10T05:00:01.068 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "1341", 2021-02-10T05:00:01.068 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T05:00:01.068 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T05:00:01.068 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T05:00:01.069 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T05:00:01.069 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T05:00:01.069 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T05:00:01.069 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T05:00:01.069 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T05:00:01.069 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T05:00:01.069 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T05:00:01.069 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T05:00:01.069 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T05:00:01.069 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T05:00:01.069 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T05:00:01.069 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T05:00:01.070 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T05:00:01.070 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T05:00:01.070 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T05:00:01.070 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T05:00:01.070 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T05:00:01.070 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T05:00:01.070 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T05:00:01.070 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T05:00:01.070 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.070 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T05:00:01.070 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T05:00:01.070 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T05:00:01.070 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T05:00:01.071 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T05:00:01.071 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T05:00:01.071 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T05:00:01.071 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T05:00:01.071 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T05:00:01.071 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T05:00:01.071 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T05:00:01.071 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T05:00:01.071 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T05:00:01.071 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T05:00:01.071 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.071 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.071 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.072 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 5, 2021-02-10T05:00:01.072 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "zone.rgw.control", 2021-02-10T05:00:01.072 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:49.278260+0000", 2021-02-10T05:00:01.072 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T05:00:01.072 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T05:00:01.072 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T05:00:01.072 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T05:00:01.072 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T05:00:01.072 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T05:00:01.072 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T05:00:01.072 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T05:00:01.072 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T05:00:01.073 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T05:00:01.073 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T05:00:01.073 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T05:00:01.073 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 32, 2021-02-10T05:00:01.073 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 32, 2021-02-10T05:00:01.073 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 32, 2021-02-10T05:00:01.073 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 32, 2021-02-10T05:00:01.073 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 32, 2021-02-10T05:00:01.073 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T05:00:01.073 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T05:00:01.073 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T05:00:01.073 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T05:00:01.073 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T05:00:01.074 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T05:00:01.074 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T05:00:01.074 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.074 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "59", 2021-02-10T05:00:01.074 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T05:00:01.074 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T05:00:01.074 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T05:00:01.074 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T05:00:01.074 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T05:00:01.074 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T05:00:01.074 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T05:00:01.074 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T05:00:01.075 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T05:00:01.075 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T05:00:01.075 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T05:00:01.075 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T05:00:01.075 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T05:00:01.075 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T05:00:01.075 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T05:00:01.075 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T05:00:01.075 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T05:00:01.075 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T05:00:01.075 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T05:00:01.075 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T05:00:01.075 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T05:00:01.075 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T05:00:01.075 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T05:00:01.076 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T05:00:01.076 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T05:00:01.076 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T05:00:01.076 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.076 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T05:00:01.076 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T05:00:01.076 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T05:00:01.076 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T05:00:01.076 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T05:00:01.076 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T05:00:01.076 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T05:00:01.076 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T05:00:01.076 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T05:00:01.076 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T05:00:01.077 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T05:00:01.077 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T05:00:01.077 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T05:00:01.077 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T05:00:01.077 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.077 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.077 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.077 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 6, 2021-02-10T05:00:01.077 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "zone.rgw.meta", 2021-02-10T05:00:01.077 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:25:51.792514+0000", 2021-02-10T05:00:01.077 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 1, 2021-02-10T05:00:01.077 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool", 2021-02-10T05:00:01.077 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T05:00:01.077 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T05:00:01.078 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T05:00:01.078 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T05:00:01.078 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T05:00:01.078 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T05:00:01.078 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T05:00:01.078 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T05:00:01.078 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T05:00:01.078 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T05:00:01.078 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 42, 2021-02-10T05:00:01.078 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 42, 2021-02-10T05:00:01.078 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 42, 2021-02-10T05:00:01.078 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 42, 2021-02-10T05:00:01.078 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 42, 2021-02-10T05:00:01.078 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T05:00:01.079 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T05:00:01.079 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T05:00:01.079 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T05:00:01.079 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T05:00:01.079 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T05:00:01.079 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T05:00:01.079 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.079 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "1046", 2021-02-10T05:00:01.079 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T05:00:01.079 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T05:00:01.079 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "1044", 2021-02-10T05:00:01.079 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T05:00:01.079 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T05:00:01.080 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 0, 2021-02-10T05:00:01.080 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 0, 2021-02-10T05:00:01.080 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T05:00:01.080 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T05:00:01.080 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T05:00:01.080 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T05:00:01.080 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T05:00:01.080 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T05:00:01.080 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T05:00:01.080 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T05:00:01.080 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T05:00:01.080 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T05:00:01.080 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T05:00:01.081 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T05:00:01.081 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T05:00:01.081 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T05:00:01.081 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T05:00:01.081 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T05:00:01.081 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T05:00:01.081 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T05:00:01.081 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T05:00:01.081 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.081 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T05:00:01.081 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T05:00:01.081 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T05:00:01.082 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T05:00:01.082 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T05:00:01.082 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T05:00:01.082 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T05:00:01.082 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T05:00:01.082 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T05:00:01.082 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T05:00:01.082 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T05:00:01.082 INFO:teuthology.orchestra.run.smithi085.stdout: "options": { 2021-02-10T05:00:01.082 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_bias": 4, 2021-02-10T05:00:01.082 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_min": 8 2021-02-10T05:00:01.082 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.082 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T05:00:01.083 INFO:teuthology.orchestra.run.smithi085.stdout: "rgw": {} 2021-02-10T05:00:01.083 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.083 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.083 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.083 INFO:teuthology.orchestra.run.smithi085.stdout: "pool": 7, 2021-02-10T05:00:01.083 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_name": "unique_pool_0", 2021-02-10T05:00:01.083 INFO:teuthology.orchestra.run.smithi085.stdout: "create_time": "2021-02-10T04:28:23.932391+0000", 2021-02-10T05:00:01.083 INFO:teuthology.orchestra.run.smithi085.stdout: "flags": 8193, 2021-02-10T05:00:01.083 INFO:teuthology.orchestra.run.smithi085.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2021-02-10T05:00:01.083 INFO:teuthology.orchestra.run.smithi085.stdout: "type": 1, 2021-02-10T05:00:01.083 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 3, 2021-02-10T05:00:01.083 INFO:teuthology.orchestra.run.smithi085.stdout: "min_size": 2, 2021-02-10T05:00:01.084 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_rule": 0, 2021-02-10T05:00:01.084 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_count": 0, 2021-02-10T05:00:01.084 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_target": 0, 2021-02-10T05:00:01.084 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_barrier": 0, 2021-02-10T05:00:01.084 INFO:teuthology.orchestra.run.smithi085.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-02-10T05:00:01.084 INFO:teuthology.orchestra.run.smithi085.stdout: "object_hash": 2, 2021-02-10T05:00:01.084 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_autoscale_mode": "off", 2021-02-10T05:00:01.084 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num": 36, 2021-02-10T05:00:01.084 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num": 36, 2021-02-10T05:00:01.084 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_placement_num_target": 36, 2021-02-10T05:00:01.084 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_target": 36, 2021-02-10T05:00:01.084 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_num_pending": 36, 2021-02-10T05:00:01.084 INFO:teuthology.orchestra.run.smithi085.stdout: "last_pg_merge_meta": { 2021-02-10T05:00:01.085 INFO:teuthology.orchestra.run.smithi085.stdout: "source_pgid": "0.0", 2021-02-10T05:00:01.085 INFO:teuthology.orchestra.run.smithi085.stdout: "ready_epoch": 0, 2021-02-10T05:00:01.085 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_started": 0, 2021-02-10T05:00:01.085 INFO:teuthology.orchestra.run.smithi085.stdout: "last_epoch_clean": 0, 2021-02-10T05:00:01.085 INFO:teuthology.orchestra.run.smithi085.stdout: "source_version": "0'0", 2021-02-10T05:00:01.085 INFO:teuthology.orchestra.run.smithi085.stdout: "target_version": "0'0" 2021-02-10T05:00:01.085 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.085 INFO:teuthology.orchestra.run.smithi085.stdout: "last_change": "1143", 2021-02-10T05:00:01.085 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend": "0", 2021-02-10T05:00:01.085 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T05:00:01.085 INFO:teuthology.orchestra.run.smithi085.stdout: "last_force_op_resend_preluminous": "1141", 2021-02-10T05:00:01.085 INFO:teuthology.orchestra.run.smithi085.stdout: "auid": 0, 2021-02-10T05:00:01.086 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_mode": "selfmanaged", 2021-02-10T05:00:01.086 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_seq": 888, 2021-02-10T05:00:01.086 INFO:teuthology.orchestra.run.smithi085.stdout: "snap_epoch": 994, 2021-02-10T05:00:01.086 INFO:teuthology.orchestra.run.smithi085.stdout: "pool_snaps": [], 2021-02-10T05:00:01.086 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps": "[]", 2021-02-10T05:00:01.086 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_bytes": 0, 2021-02-10T05:00:01.086 INFO:teuthology.orchestra.run.smithi085.stdout: "quota_max_objects": 0, 2021-02-10T05:00:01.086 INFO:teuthology.orchestra.run.smithi085.stdout: "tiers": [], 2021-02-10T05:00:01.086 INFO:teuthology.orchestra.run.smithi085.stdout: "tier_of": -1, 2021-02-10T05:00:01.086 INFO:teuthology.orchestra.run.smithi085.stdout: "read_tier": -1, 2021-02-10T05:00:01.086 INFO:teuthology.orchestra.run.smithi085.stdout: "write_tier": -1, 2021-02-10T05:00:01.086 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_mode": "none", 2021-02-10T05:00:01.086 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_bytes": 0, 2021-02-10T05:00:01.087 INFO:teuthology.orchestra.run.smithi085.stdout: "target_max_objects": 0, 2021-02-10T05:00:01.087 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T05:00:01.087 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T05:00:01.087 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T05:00:01.087 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_flush_age": 0, 2021-02-10T05:00:01.087 INFO:teuthology.orchestra.run.smithi085.stdout: "cache_min_evict_age": 0, 2021-02-10T05:00:01.087 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profile": "", 2021-02-10T05:00:01.087 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_params": { 2021-02-10T05:00:01.087 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "none" 2021-02-10T05:00:01.087 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.087 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_period": 0, 2021-02-10T05:00:01.087 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_count": 0, 2021-02-10T05:00:01.087 INFO:teuthology.orchestra.run.smithi085.stdout: "use_gmt_hitset": true, 2021-02-10T05:00:01.088 INFO:teuthology.orchestra.run.smithi085.stdout: "min_read_recency_for_promote": 0, 2021-02-10T05:00:01.088 INFO:teuthology.orchestra.run.smithi085.stdout: "min_write_recency_for_promote": 0, 2021-02-10T05:00:01.088 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T05:00:01.088 INFO:teuthology.orchestra.run.smithi085.stdout: "hit_set_search_last_n": 0, 2021-02-10T05:00:01.088 INFO:teuthology.orchestra.run.smithi085.stdout: "grade_table": [], 2021-02-10T05:00:01.088 INFO:teuthology.orchestra.run.smithi085.stdout: "stripe_width": 0, 2021-02-10T05:00:01.088 INFO:teuthology.orchestra.run.smithi085.stdout: "expected_num_objects": 0, 2021-02-10T05:00:01.088 INFO:teuthology.orchestra.run.smithi085.stdout: "fast_read": false, 2021-02-10T05:00:01.088 INFO:teuthology.orchestra.run.smithi085.stdout: "options": {}, 2021-02-10T05:00:01.088 INFO:teuthology.orchestra.run.smithi085.stdout: "application_metadata": { 2021-02-10T05:00:01.088 INFO:teuthology.orchestra.run.smithi085.stdout: "rados": {} 2021-02-10T05:00:01.088 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.088 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.089 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T05:00:01.089 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T05:00:01.089 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.089 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 0, 2021-02-10T05:00:01.089 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "45d0bb5f-35fa-4596-964f-75abcee53bee", 2021-02-10T05:00:01.089 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T05:00:01.089 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T05:00:01.089 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T05:00:01.089 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T05:00:01.089 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 1227, 2021-02-10T05:00:01.089 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 1315, 2021-02-10T05:00:01.089 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 1332, 2021-02-10T05:00:01.089 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1366, 2021-02-10T05:00:01.090 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 1316, 2021-02-10T05:00:01.090 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T05:00:01.090 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T05:00:01.090 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.090 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.090 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.090 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6801", 2021-02-10T05:00:01.090 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.090 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.090 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.090 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.090 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T05:00:01.091 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.091 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.091 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.091 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6802", 2021-02-10T05:00:01.091 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.091 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.091 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.091 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.091 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T05:00:01.091 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.091 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.091 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.091 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6804", 2021-02-10T05:00:01.092 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.092 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.092 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.092 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.092 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T05:00:01.092 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.092 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.092 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.092 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6803", 2021-02-10T05:00:01.092 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.092 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.092 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.092 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.093 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6801/7", 2021-02-10T05:00:01.093 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6802/7", 2021-02-10T05:00:01.093 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6804/7", 2021-02-10T05:00:01.093 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6803/7", 2021-02-10T05:00:01.093 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T05:00:01.093 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T05:00:01.093 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T05:00:01.093 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.093 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.093 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.093 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 1, 2021-02-10T05:00:01.093 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "3421a5ce-872e-4b52-adf7-6aaa595871a4", 2021-02-10T05:00:01.094 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T05:00:01.094 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T05:00:01.094 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T05:00:01.094 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 0.1644744873046875, 2021-02-10T05:00:01.094 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T05:00:01.094 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T05:00:01.094 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 13, 2021-02-10T05:00:01.094 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1367, 2021-02-10T05:00:01.094 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T05:00:01.094 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T05:00:01.094 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T05:00:01.095 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.095 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.095 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.095 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6805", 2021-02-10T05:00:01.095 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.095 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.095 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.095 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.095 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T05:00:01.095 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.095 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.095 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.095 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6806", 2021-02-10T05:00:01.096 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.096 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.096 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.096 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.096 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T05:00:01.096 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.096 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.096 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.096 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6808", 2021-02-10T05:00:01.096 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.096 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.096 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.096 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.097 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T05:00:01.097 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.097 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.097 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.097 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6807", 2021-02-10T05:00:01.097 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.097 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.097 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.097 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.097 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6805/7", 2021-02-10T05:00:01.097 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6806/7", 2021-02-10T05:00:01.097 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6808/7", 2021-02-10T05:00:01.097 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6807/7", 2021-02-10T05:00:01.098 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T05:00:01.098 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T05:00:01.098 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T05:00:01.098 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.098 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.098 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.098 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 2, 2021-02-10T05:00:01.098 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "96e1952a-1042-4316-9fc9-86813e88b912", 2021-02-10T05:00:01.098 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T05:00:01.098 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T05:00:01.098 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T05:00:01.098 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T05:00:01.099 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T05:00:01.099 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T05:00:01.099 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 18, 2021-02-10T05:00:01.099 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1368, 2021-02-10T05:00:01.099 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T05:00:01.099 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T05:00:01.099 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T05:00:01.099 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.099 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.099 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.099 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6809", 2021-02-10T05:00:01.099 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.100 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.100 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.100 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.100 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T05:00:01.100 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.100 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.100 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.100 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6810", 2021-02-10T05:00:01.100 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.100 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.100 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.100 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.101 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T05:00:01.101 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.101 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.101 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.101 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6812", 2021-02-10T05:00:01.101 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.101 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.101 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.101 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.101 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T05:00:01.101 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.101 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.101 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.102 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6811", 2021-02-10T05:00:01.102 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.102 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.102 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.102 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.102 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6809/7", 2021-02-10T05:00:01.102 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6810/7", 2021-02-10T05:00:01.102 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6812/7", 2021-02-10T05:00:01.102 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6811/7", 2021-02-10T05:00:01.102 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T05:00:01.102 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T05:00:01.102 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T05:00:01.103 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.103 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.103 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.103 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 3, 2021-02-10T05:00:01.103 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "e019f49f-5dd4-4104-9ced-94c710616642", 2021-02-10T05:00:01.103 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T05:00:01.103 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T05:00:01.103 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T05:00:01.103 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T05:00:01.103 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 25, 2021-02-10T05:00:01.103 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 1094, 2021-02-10T05:00:01.103 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 1111, 2021-02-10T05:00:01.104 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1366, 2021-02-10T05:00:01.104 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 1095, 2021-02-10T05:00:01.104 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T05:00:01.104 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T05:00:01.104 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.104 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.104 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.104 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6813", 2021-02-10T05:00:01.104 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.104 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.104 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.104 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.105 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T05:00:01.105 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.105 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.105 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.105 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6814", 2021-02-10T05:00:01.105 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.105 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.105 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.105 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.105 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T05:00:01.105 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.105 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.105 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.106 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6816", 2021-02-10T05:00:01.106 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.106 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.106 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.106 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.106 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T05:00:01.106 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.106 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.106 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.106 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.85:6815", 2021-02-10T05:00:01.106 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.106 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.107 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.107 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.107 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.85:6813/7", 2021-02-10T05:00:01.107 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.85:6814/7", 2021-02-10T05:00:01.107 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.85:6816/7", 2021-02-10T05:00:01.107 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.85:6815/7", 2021-02-10T05:00:01.107 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T05:00:01.107 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T05:00:01.107 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T05:00:01.107 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.107 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.107 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.108 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 4, 2021-02-10T05:00:01.108 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "91c8a80a-37b6-4c30-a32a-647b2e6d801f", 2021-02-10T05:00:01.108 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T05:00:01.108 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T05:00:01.108 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T05:00:01.108 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T05:00:01.108 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T05:00:01.108 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T05:00:01.108 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 31, 2021-02-10T05:00:01.108 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1370, 2021-02-10T05:00:01.108 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T05:00:01.108 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T05:00:01.108 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T05:00:01.109 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.109 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.109 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.109 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6800", 2021-02-10T05:00:01.109 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.109 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.109 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.109 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.109 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T05:00:01.109 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.109 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.109 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.109 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6801", 2021-02-10T05:00:01.110 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.110 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.110 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.110 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.110 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T05:00:01.110 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.110 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.110 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.110 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6803", 2021-02-10T05:00:01.110 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.110 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.111 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.111 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.111 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T05:00:01.111 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.111 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.111 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.111 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6802", 2021-02-10T05:00:01.111 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.111 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.111 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.111 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.111 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6800/7", 2021-02-10T05:00:01.111 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6801/7", 2021-02-10T05:00:01.112 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6803/7", 2021-02-10T05:00:01.112 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6802/7", 2021-02-10T05:00:01.112 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T05:00:01.112 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T05:00:01.112 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T05:00:01.112 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.112 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.112 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.112 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 5, 2021-02-10T05:00:01.112 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e", 2021-02-10T05:00:01.112 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T05:00:01.112 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T05:00:01.112 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T05:00:01.113 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T05:00:01.113 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T05:00:01.113 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T05:00:01.113 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 37, 2021-02-10T05:00:01.113 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1366, 2021-02-10T05:00:01.113 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T05:00:01.113 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T05:00:01.113 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T05:00:01.113 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.113 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.113 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.113 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6804", 2021-02-10T05:00:01.113 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.114 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.114 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.114 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.114 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T05:00:01.114 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.114 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.114 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.114 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6805", 2021-02-10T05:00:01.114 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.114 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.114 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.114 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.115 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T05:00:01.115 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.115 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.115 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.115 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6807", 2021-02-10T05:00:01.115 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.115 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.115 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.115 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.115 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T05:00:01.115 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.115 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.116 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.116 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6806", 2021-02-10T05:00:01.116 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.116 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.116 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.116 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.116 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6804/7", 2021-02-10T05:00:01.116 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6805/7", 2021-02-10T05:00:01.116 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6807/7", 2021-02-10T05:00:01.116 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6806/7", 2021-02-10T05:00:01.116 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T05:00:01.116 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T05:00:01.117 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T05:00:01.117 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.117 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.117 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.117 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 6, 2021-02-10T05:00:01.117 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "adc40430-3eeb-4003-a824-343021243077", 2021-02-10T05:00:01.117 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T05:00:01.117 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 0, 2021-02-10T05:00:01.117 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 0, 2021-02-10T05:00:01.117 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 1, 2021-02-10T05:00:01.117 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T05:00:01.117 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T05:00:01.117 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 43, 2021-02-10T05:00:01.118 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1360, 2021-02-10T05:00:01.118 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T05:00:01.118 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T05:00:01.118 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T05:00:01.118 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.118 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.118 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.118 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6808", 2021-02-10T05:00:01.118 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.118 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.118 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.118 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.119 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T05:00:01.119 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.119 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.119 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.119 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6809", 2021-02-10T05:00:01.119 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.119 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.119 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.119 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.119 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T05:00:01.119 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.119 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.119 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.120 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6811", 2021-02-10T05:00:01.120 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.120 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.120 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.120 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.120 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T05:00:01.120 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.120 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.120 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.120 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6810", 2021-02-10T05:00:01.120 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.120 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.121 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.121 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.121 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6808/7", 2021-02-10T05:00:01.121 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6809/7", 2021-02-10T05:00:01.121 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6811/7", 2021-02-10T05:00:01.121 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6810/7", 2021-02-10T05:00:01.121 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T05:00:01.121 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T05:00:01.121 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T05:00:01.121 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.121 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.121 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.122 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 7, 2021-02-10T05:00:01.122 INFO:teuthology.orchestra.run.smithi085.stdout: "uuid": "a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37", 2021-02-10T05:00:01.122 INFO:teuthology.orchestra.run.smithi085.stdout: "up": 1, 2021-02-10T05:00:01.122 INFO:teuthology.orchestra.run.smithi085.stdout: "in": 1, 2021-02-10T05:00:01.122 INFO:teuthology.orchestra.run.smithi085.stdout: "weight": 1, 2021-02-10T05:00:01.122 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_affinity": 0.846160888671875, 2021-02-10T05:00:01.122 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_begin": 0, 2021-02-10T05:00:01.122 INFO:teuthology.orchestra.run.smithi085.stdout: "last_clean_end": 0, 2021-02-10T05:00:01.122 INFO:teuthology.orchestra.run.smithi085.stdout: "up_from": 50, 2021-02-10T05:00:01.122 INFO:teuthology.orchestra.run.smithi085.stdout: "up_thru": 1367, 2021-02-10T05:00:01.122 INFO:teuthology.orchestra.run.smithi085.stdout: "down_at": 0, 2021-02-10T05:00:01.122 INFO:teuthology.orchestra.run.smithi085.stdout: "lost_at": 0, 2021-02-10T05:00:01.122 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addrs": { 2021-02-10T05:00:01.123 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.123 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.123 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.123 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6812", 2021-02-10T05:00:01.123 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.123 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.123 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.123 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.123 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addrs": { 2021-02-10T05:00:01.123 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.123 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.123 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.124 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6813", 2021-02-10T05:00:01.124 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.124 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.124 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.124 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.124 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addrs": { 2021-02-10T05:00:01.124 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.124 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.124 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.124 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6815", 2021-02-10T05:00:01.124 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.124 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.124 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.125 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.125 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addrs": { 2021-02-10T05:00:01.125 INFO:teuthology.orchestra.run.smithi085.stdout: "addrvec": [ 2021-02-10T05:00:01.125 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.125 INFO:teuthology.orchestra.run.smithi085.stdout: "type": "v2", 2021-02-10T05:00:01.125 INFO:teuthology.orchestra.run.smithi085.stdout: "addr": "172.21.15.164:6814", 2021-02-10T05:00:01.125 INFO:teuthology.orchestra.run.smithi085.stdout: "nonce": 7 2021-02-10T05:00:01.125 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.125 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.125 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.125 INFO:teuthology.orchestra.run.smithi085.stdout: "public_addr": "172.21.15.164:6812/7", 2021-02-10T05:00:01.125 INFO:teuthology.orchestra.run.smithi085.stdout: "cluster_addr": "172.21.15.164:6813/7", 2021-02-10T05:00:01.126 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_back_addr": "172.21.15.164:6815/7", 2021-02-10T05:00:01.126 INFO:teuthology.orchestra.run.smithi085.stdout: "heartbeat_front_addr": "172.21.15.164:6814/7", 2021-02-10T05:00:01.126 INFO:teuthology.orchestra.run.smithi085.stdout: "state": [ 2021-02-10T05:00:01.126 INFO:teuthology.orchestra.run.smithi085.stdout: "exists", 2021-02-10T05:00:01.126 INFO:teuthology.orchestra.run.smithi085.stdout: "up" 2021-02-10T05:00:01.126 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.126 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.126 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T05:00:01.126 INFO:teuthology.orchestra.run.smithi085.stdout: "osd_xinfo": [ 2021-02-10T05:00:01.126 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.126 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 0, 2021-02-10T05:00:01.126 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T05:00:01.127 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T05:00:01.127 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T05:00:01.127 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T05:00:01.127 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T05:00:01.127 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:59:56.261166+0000", 2021-02-10T05:00:01.127 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 1316 2021-02-10T05:00:01.127 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.127 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.127 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 1, 2021-02-10T05:00:01.127 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T05:00:01.127 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T05:00:01.127 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T05:00:01.127 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T05:00:01.128 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T05:00:01.128 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:59:01.695880+0000", 2021-02-10T05:00:01.128 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T05:00:01.128 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.128 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.128 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 2, 2021-02-10T05:00:01.128 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T05:00:01.128 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T05:00:01.128 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T05:00:01.128 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T05:00:01.128 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T05:00:01.128 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:59:20.883571+0000", 2021-02-10T05:00:01.129 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T05:00:01.129 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.129 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.129 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 3, 2021-02-10T05:00:01.129 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T05:00:01.129 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T05:00:01.129 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T05:00:01.129 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T05:00:01.129 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T05:00:01.129 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T05:00:00.199359+0000", 2021-02-10T05:00:01.129 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 1095 2021-02-10T05:00:01.129 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.129 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.130 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 4, 2021-02-10T05:00:01.130 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T05:00:01.130 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T05:00:01.130 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T05:00:01.130 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T05:00:01.130 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T05:00:01.130 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:59:56.313080+0000", 2021-02-10T05:00:01.130 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T05:00:01.130 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.130 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.130 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 5, 2021-02-10T05:00:01.130 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T05:00:01.131 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T05:00:01.131 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T05:00:01.131 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T05:00:01.131 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T05:00:01.131 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:59:12.932203+0000", 2021-02-10T05:00:01.131 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T05:00:01.131 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.131 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.131 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 6, 2021-02-10T05:00:01.131 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T05:00:01.131 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T05:00:01.131 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T05:00:01.131 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T05:00:01.132 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 65536, 2021-02-10T05:00:01.132 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:59:31.289424+0000", 2021-02-10T05:00:01.132 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T05:00:01.132 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.132 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.132 INFO:teuthology.orchestra.run.smithi085.stdout: "osd": 7, 2021-02-10T05:00:01.132 INFO:teuthology.orchestra.run.smithi085.stdout: "down_stamp": "0.000000", 2021-02-10T05:00:01.132 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_probability": 0, 2021-02-10T05:00:01.132 INFO:teuthology.orchestra.run.smithi085.stdout: "laggy_interval": 0, 2021-02-10T05:00:01.132 INFO:teuthology.orchestra.run.smithi085.stdout: "features": 4540138303579357183, 2021-02-10T05:00:01.132 INFO:teuthology.orchestra.run.smithi085.stdout: "old_weight": 0, 2021-02-10T05:00:01.132 INFO:teuthology.orchestra.run.smithi085.stdout: "last_purged_snaps_scrub": "2021-02-10T04:59:49.655402+0000", 2021-02-10T05:00:01.132 INFO:teuthology.orchestra.run.smithi085.stdout: "dead_epoch": 0 2021-02-10T05:00:01.132 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.133 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T05:00:01.133 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_upmap": [ 2021-02-10T05:00:01.133 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.133 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "2.23", 2021-02-10T05:00:01.133 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T05:00:01.133 INFO:teuthology.orchestra.run.smithi085.stdout: 3, 2021-02-10T05:00:01.133 INFO:teuthology.orchestra.run.smithi085.stdout: 0, 2021-02-10T05:00:01.133 INFO:teuthology.orchestra.run.smithi085.stdout: 6 2021-02-10T05:00:01.133 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.133 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.133 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T05:00:01.133 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_upmap_items": [ 2021-02-10T05:00:01.134 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.134 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "2.7", 2021-02-10T05:00:01.134 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T05:00:01.134 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.134 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T05:00:01.134 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 4 2021-02-10T05:00:01.134 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.134 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.134 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.134 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.134 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "2.1a", 2021-02-10T05:00:01.134 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T05:00:01.135 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.135 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T05:00:01.135 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 2 2021-02-10T05:00:01.135 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.135 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.135 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.135 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.135 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "2.1d", 2021-02-10T05:00:01.135 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T05:00:01.135 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.135 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T05:00:01.135 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 2 2021-02-10T05:00:01.135 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.136 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.136 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.136 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.136 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "2.21", 2021-02-10T05:00:01.136 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T05:00:01.136 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.136 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T05:00:01.136 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 1 2021-02-10T05:00:01.136 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.136 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.136 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.136 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.137 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "2.28", 2021-02-10T05:00:01.137 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T05:00:01.137 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.137 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T05:00:01.137 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 2 2021-02-10T05:00:01.137 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.137 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.137 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.137 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.137 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.5", 2021-02-10T05:00:01.137 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T05:00:01.137 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.137 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 4, 2021-02-10T05:00:01.137 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 2 2021-02-10T05:00:01.138 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.138 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.138 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.138 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.138 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.10", 2021-02-10T05:00:01.138 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T05:00:01.138 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.138 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 1, 2021-02-10T05:00:01.138 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 5 2021-02-10T05:00:01.138 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.138 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.138 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.139 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.139 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.8", 2021-02-10T05:00:01.139 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T05:00:01.139 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.139 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 1, 2021-02-10T05:00:01.139 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 3 2021-02-10T05:00:01.139 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.139 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.139 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.139 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.139 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.10", 2021-02-10T05:00:01.139 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T05:00:01.139 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.140 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T05:00:01.140 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 3 2021-02-10T05:00:01.140 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.140 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.140 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.140 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.140 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.18", 2021-02-10T05:00:01.140 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T05:00:01.140 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.140 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 4, 2021-02-10T05:00:01.140 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 3 2021-02-10T05:00:01.140 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.140 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.140 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.141 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.141 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "5.1b", 2021-02-10T05:00:01.141 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T05:00:01.141 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.141 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 0, 2021-02-10T05:00:01.141 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 3 2021-02-10T05:00:01.141 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.141 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.141 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.141 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.141 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "7.5", 2021-02-10T05:00:01.141 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T05:00:01.141 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.141 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 0, 2021-02-10T05:00:01.142 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 2 2021-02-10T05:00:01.142 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.142 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.142 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.142 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.142 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "7.6", 2021-02-10T05:00:01.142 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T05:00:01.142 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.142 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 3, 2021-02-10T05:00:01.142 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 1 2021-02-10T05:00:01.142 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.142 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.142 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.143 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.143 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "7.15", 2021-02-10T05:00:01.143 INFO:teuthology.orchestra.run.smithi085.stdout: "mappings": [ 2021-02-10T05:00:01.143 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.143 INFO:teuthology.orchestra.run.smithi085.stdout: "from": 7, 2021-02-10T05:00:01.143 INFO:teuthology.orchestra.run.smithi085.stdout: "to": 4 2021-02-10T05:00:01.143 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.143 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.143 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.143 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T05:00:01.143 INFO:teuthology.orchestra.run.smithi085.stdout: "pg_temp": [ 2021-02-10T05:00:01.143 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.143 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.c", 2021-02-10T05:00:01.144 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T05:00:01.144 INFO:teuthology.orchestra.run.smithi085.stdout: 4, 2021-02-10T05:00:01.144 INFO:teuthology.orchestra.run.smithi085.stdout: 3, 2021-02-10T05:00:01.144 INFO:teuthology.orchestra.run.smithi085.stdout: 6 2021-02-10T05:00:01.144 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.144 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.144 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.144 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.e", 2021-02-10T05:00:01.144 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T05:00:01.144 INFO:teuthology.orchestra.run.smithi085.stdout: 4, 2021-02-10T05:00:01.144 INFO:teuthology.orchestra.run.smithi085.stdout: 1, 2021-02-10T05:00:01.144 INFO:teuthology.orchestra.run.smithi085.stdout: 6 2021-02-10T05:00:01.144 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.145 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.145 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.145 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.16", 2021-02-10T05:00:01.145 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T05:00:01.145 INFO:teuthology.orchestra.run.smithi085.stdout: 0, 2021-02-10T05:00:01.145 INFO:teuthology.orchestra.run.smithi085.stdout: 1, 2021-02-10T05:00:01.145 INFO:teuthology.orchestra.run.smithi085.stdout: 2 2021-02-10T05:00:01.145 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.145 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.145 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.145 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.18", 2021-02-10T05:00:01.145 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T05:00:01.146 INFO:teuthology.orchestra.run.smithi085.stdout: 4, 2021-02-10T05:00:01.146 INFO:teuthology.orchestra.run.smithi085.stdout: 1, 2021-02-10T05:00:01.146 INFO:teuthology.orchestra.run.smithi085.stdout: 6 2021-02-10T05:00:01.146 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.146 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.146 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.146 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.19", 2021-02-10T05:00:01.146 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T05:00:01.146 INFO:teuthology.orchestra.run.smithi085.stdout: 3, 2021-02-10T05:00:01.146 INFO:teuthology.orchestra.run.smithi085.stdout: 1, 2021-02-10T05:00:01.146 INFO:teuthology.orchestra.run.smithi085.stdout: 4 2021-02-10T05:00:01.146 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.146 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.147 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.147 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.1b", 2021-02-10T05:00:01.147 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T05:00:01.147 INFO:teuthology.orchestra.run.smithi085.stdout: 4, 2021-02-10T05:00:01.147 INFO:teuthology.orchestra.run.smithi085.stdout: 1, 2021-02-10T05:00:01.147 INFO:teuthology.orchestra.run.smithi085.stdout: 0 2021-02-10T05:00:01.147 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.147 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.147 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:00:01.147 INFO:teuthology.orchestra.run.smithi085.stdout: "pgid": "4.1e", 2021-02-10T05:00:01.147 INFO:teuthology.orchestra.run.smithi085.stdout: "osds": [ 2021-02-10T05:00:01.147 INFO:teuthology.orchestra.run.smithi085.stdout: 1, 2021-02-10T05:00:01.147 INFO:teuthology.orchestra.run.smithi085.stdout: 4, 2021-02-10T05:00:01.148 INFO:teuthology.orchestra.run.smithi085.stdout: 6 2021-02-10T05:00:01.148 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:00:01.148 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.148 INFO:teuthology.orchestra.run.smithi085.stdout: ], 2021-02-10T05:00:01.148 INFO:teuthology.orchestra.run.smithi085.stdout: "primary_temp": [], 2021-02-10T05:00:01.148 INFO:teuthology.orchestra.run.smithi085.stdout: "blocklist": { 2021-02-10T05:00:01.148 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/801393150": "2021-02-11T04:21:37.851607+0000", 2021-02-10T05:00:01.148 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/3049671696": "2021-02-11T04:21:37.851607+0000", 2021-02-10T05:00:01.148 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/4056505602": "2021-02-11T04:21:20.700478+0000", 2021-02-10T05:00:01.148 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:6800/7": "2021-02-11T04:21:20.700478+0000", 2021-02-10T05:00:01.148 INFO:teuthology.orchestra.run.smithi085.stdout: "172.21.15.85:0/1879026062": "2021-02-11T04:21:20.700478+0000" 2021-02-10T05:00:01.148 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.148 INFO:teuthology.orchestra.run.smithi085.stdout: "erasure_code_profiles": { 2021-02-10T05:00:01.149 INFO:teuthology.orchestra.run.smithi085.stdout: "default": { 2021-02-10T05:00:01.149 INFO:teuthology.orchestra.run.smithi085.stdout: "crush-failure-domain": "osd", 2021-02-10T05:00:01.149 INFO:teuthology.orchestra.run.smithi085.stdout: "k": "2", 2021-02-10T05:00:01.149 INFO:teuthology.orchestra.run.smithi085.stdout: "m": "1", 2021-02-10T05:00:01.149 INFO:teuthology.orchestra.run.smithi085.stdout: "plugin": "jerasure", 2021-02-10T05:00:01.149 INFO:teuthology.orchestra.run.smithi085.stdout: "ruleset-failure-domain": "osd", 2021-02-10T05:00:01.149 INFO:teuthology.orchestra.run.smithi085.stdout: "technique": "reed_sol_van" 2021-02-10T05:00:01.149 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.149 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:00:01.149 INFO:teuthology.orchestra.run.smithi085.stdout: "removed_snaps_queue": [], 2021-02-10T05:00:01.149 INFO:teuthology.orchestra.run.smithi085.stdout: "new_removed_snaps": [], 2021-02-10T05:00:01.149 INFO:teuthology.orchestra.run.smithi085.stdout: "new_purged_snaps": [], 2021-02-10T05:00:01.149 INFO:teuthology.orchestra.run.smithi085.stdout: "crush_node_flags": {}, 2021-02-10T05:00:01.150 INFO:teuthology.orchestra.run.smithi085.stdout: "device_class_flags": {}, 2021-02-10T05:00:01.150 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_mode": { 2021-02-10T05:00:01.150 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_mode_enabled": false, 2021-02-10T05:00:01.150 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_bucket_count": 0, 2021-02-10T05:00:01.150 INFO:teuthology.orchestra.run.smithi085.stdout: "degraded_stretch_mode": 0, 2021-02-10T05:00:01.150 INFO:teuthology.orchestra.run.smithi085.stdout: "recovering_stretch_mode": 0, 2021-02-10T05:00:01.150 INFO:teuthology.orchestra.run.smithi085.stdout: "stretch_mode_bucket": 0 2021-02-10T05:00:01.150 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:00:01.150 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T05:00:01.151 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: cluster 2021-02-10T05:00:01.151 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: 2021-02-10T05:00:01.151 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: 2021 2021-02-10T05:00:01.151 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: - 2021-02-10T05:00:01.151 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: 02 2021-02-10T05:00:01.151 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: - 2021-02-10T05:00:01.151 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: 10 2021-02-10T05:00:01.151 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: T 2021-02-10T05:00:01.151 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: 04 2021-02-10T05:00:01.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: : 2021-02-10T05:00:01.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: 59 2021-02-10T05:00:01.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: :59.241022+0000 osd.2 ( 2021-02-10T05:00:01.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: osd.2) 2043 : cluster [DBG] 5.8 scrub ok 2021-02-10T05:00:01.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: audit 2021-02-10T05:00:01.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: 2021-02-10T04:59:59.585017+0000 mgr.y 2021-02-10T05:00:01.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: (mgr.14140) 1545 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:00:01.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: cluster 2021-02-10T05:00: 2021-02-10T05:00:01.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: 00.000111+0000 mon.a (mon.0) 3063 : cluster 2021-02-10T05:00:01.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: [WRN] Health detail: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull 2021-02-10T05:00:01.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: cluster 2021-02-10T05:00:00 2021-02-10T05:00:01.152 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: .000152+0000 mon.a (mon.0) 3064 : cluster 2021-02-10T05:00:01.153 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: [WRN] [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull 2021-02-10T05:00:01.153 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T05:00:01.153 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: 05:00:00.000172+0000 mon.a (mon.0) 3065 : 2021-02-10T05:00:01.153 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: cluster [WRN] pg 4.c is active+remapped+backfill_toofull, acting [4,3,6] 2021-02-10T05:00:01.153 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: cluster 2021- 2021-02-10T05:00:01.153 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: 02-10T05:00:00.000194+0000 mon.a (mon.0) 3066 : cluster [WRN] pg 4.e is active+remapped+backfill_toofull, acting [4,1,6] 2021-02-10T05:00:01.153 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: cluster 2021-02-10T05:00:00.000212 2021-02-10T05:00:01.153 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: +0000 mon.a (mon.0) 3067 : cluster [WRN] pg 4.16 is active+remapped+backfill_toofull, acting [0,1,2] 2021-02-10T05:00:01.153 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: cluster 2021-02-10T05:00:00.000228+0000 2021-02-10T05:00:01.153 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: mon.a (mon.0) 3068 : cluster [WRN] pg 4.18 is active+remapped+backfill_toofull, acting [4,1,6] 2021-02-10T05:00:01.153 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: cluster 2021-02-10T05:00:01.153 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: 2021-02-10T05:00:00.000243+0000 mon.a (mon. 2021-02-10T05:00:01.154 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: 0) 3069 : cluster [WRN] pg 4.19 is active+remapped+backfill_toofull, acting [3,1,4] 2021-02-10T05:00:01.154 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T05:00:01.154 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: 05:00:00.000260+0000 mon.a (mon.0 2021-02-10T05:00:01.154 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: ) 3070 : cluster [WRN] pg 4.1b is active+remapped+backfill_toofull, acting [4,1,0] 2021-02-10T05:00:01.154 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T05:00:01.154 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: T05:00:00.000278+0000 mon.a (mon.0 2021-02-10T05:00:01.154 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: ) 3071 : cluster [WRN] pg 4.1e is active+remapped+backfill_toofull, acting [1,4,6] 2021-02-10T05:00:01.154 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T05:00:01.154 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: T05:00:00.081911+0000 osd.0 (osd.0 2021-02-10T05:00:01.154 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: ) 11 : cluster [DBG] 2.1f scrub ok 2021-02-10T05:00:01.154 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: cluster 2021-02-10T05:00:00.194382+0000 2021-02-10T05:00:01.154 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: osd.3 (osd.3) 50 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:00:01.155 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T05:00:01.155 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: -10T05:00:00.199355+0000 osd.3 ( 2021-02-10T05:00:01.155 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: osd.3) 51 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:00:01.155 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: cluster 2021-02-10T05:00 2021-02-10T05:00:01.155 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: :00.267583+0000 osd.4 (osd.4) 2673 : cluster [DBG] 2021-02-10T05:00:01.155 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:00 smithi164 conmon[29288]: 5.11 scrub ok 2021-02-10T05:00:01.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:01 smithi085 conmon[35961]: cluster 2021-02-10T05:00:00.576348+0000 2021-02-10T05:00:01.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:01 smithi085 conmon[35961]: mgr.y (mgr.14140) 1546 : cluster [DBG] pgmap v2513: 208 pgs: 1 active+clean+snaptrim, 1 peering, 7 active+remapped+backfill_toofull, 199 active+clean; 859 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 832 B/s rd, 421 KiB/s wr, 1 op/s; 120/2169 objects misplaced (5.533%); 14 MiB/s, 32 objects/s recovering 2021-02-10T05:00:01.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:01 smithi085 conmon[35961]: cluster 2021-02-10T05:00:00.695041+0000 mon.a (mon.0) 3072 : cluster [DBG] osdmap e1373: 8 total, 8 up, 7 in 2021-02-10T05:00:01.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:01 smithi085 conmon[35961]: audit 2021-02-10T05:00:01.044314+0000 mon.a (mon.0) 3073 : audit [DBG] from='client.? 172.21.15.85:0/2558462305' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-02-10T05:00:01.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:01 smithi085 conmon[35961]: cluster 2021-02-10T05:00:01.225477+0000 osd.4 (osd.4) 2674 : cluster [DBG] 5.1d scrub ok 2021-02-10T05:00:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:01 smithi085 conmon[29319]: cluster 2021-02-10T05:00:00.576348+0000 2021-02-10T05:00:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:01 smithi085 conmon[29319]: mgr.y (mgr.14140) 1546 : cluster [DBG] pgmap v2513: 208 pgs: 1 active+clean+snaptrim, 1 peering, 7 active+remapped+backfill_toofull, 199 active+clean; 859 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 832 B/s rd, 421 KiB/s wr, 1 op/s; 120/2169 objects misplaced (5.533%); 14 MiB/s, 32 objects/s recovering 2021-02-10T05:00:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:01 smithi085 conmon[29319]: cluster 2021-02-10T05:00:00.695041+0000 mon.a (mon.0) 3072 : cluster [DBG] osdmap e1373: 8 total, 8 up, 7 in 2021-02-10T05:00:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:01 smithi085 conmon[29319]: audit 2021-02-10T05:00:01.044314+0000 mon.a (mon.0) 3073 : audit [DBG] from='client.? 172.21.15.85:0/2558462305' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-02-10T05:00:01.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:01 smithi085 conmon[29319]: cluster 2021-02-10T05:00:01.225477+0000 osd.4 (osd.4) 2674 : cluster [DBG] 5.1d scrub ok 2021-02-10T05:00:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:01 smithi164 conmon[29288]: cluster 2021-02-10T05:00:00.576348+0000 2021-02-10T05:00:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:01 smithi164 conmon[29288]: mgr.y (mgr.14140) 1546 : cluster [DBG] pgmap v2513: 208 pgs: 1 active+clean+snaptrim, 1 peering, 7 active+remapped+backfill_toofull, 199 active+clean; 859 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 832 B/s rd, 421 KiB/s wr, 1 op/s; 120/2169 objects misplaced (5.533%); 14 MiB/s, 32 objects/s recovering 2021-02-10T05:00:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:01 smithi164 conmon[29288]: cluster 2021-02-10T05:00:00.695041+0000 mon.a (mon.0) 3072 : cluster [DBG] 2021-02-10T05:00:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:01 smithi164 conmon[29288]: osdmap e1373: 8 total, 8 up, 7 in 2021-02-10T05:00:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:01 smithi164 conmon[29288]: audit 2021-02-10T05:00: 2021-02-10T05:00:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:01 smithi164 conmon[29288]: 01.044314+0000 mon.a (mon.0) 3073 : audit [DBG] from='client.? 172.21.15.85:0/2558462305' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-02-10T05:00:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:01 smithi164 conmon[29288]: cluster 2021 2021-02-10T05:00:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:01 smithi164 conmon[29288]: -02-10T05:00:01.225477+0000 osd.4 (osd. 2021-02-10T05:00:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:01 smithi164 conmon[29288]: 4) 2674 : cluster [DBG] 5.1d scrub ok 2021-02-10T05:00:02.182 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.6 dump_historic_ops 2021-02-10T05:00:02.310 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set noscrub 2021-02-10T05:00:02.504 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T05:00:02.521 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:00:03.019 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:02 smithi085 conmon[29319]: cluster 2021-02-10T05:00:01.697936+0000 2021-02-10T05:00:03.019 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:02 smithi085 conmon[29319]: osd.7 (osd.7) 103 : cluster [DBG] 2.5 scrub ok 2021-02-10T05:00:03.019 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:02 smithi085 conmon[35961]: cluster 2021-02-10T05:00:01.697936+0000 2021-02-10T05:00:03.019 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:02 smithi085 conmon[35961]: osd.7 (osd.7) 103 : cluster [DBG] 2.5 scrub ok 2021-02-10T05:00:03.112 INFO:tasks.thrashosds.thrasher:j is {'epoch': 1373, 'fsid': '42224fce-6b57-11eb-8feb-001a4aab830c', 'created': '2021-02-10T04:20:57.933279+0000', 'modified': '2021-02-10T05:00:00.681249+0000', 'last_up_change': '2021-02-10T04:58:55.817746+0000', 'last_in_change': '2021-02-10T04:59:50.064782+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-02-10T04:24:05.258028+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1266', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '110', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2021-02-10T04:25:32.688554+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 42, 'pg_placement_num': 42, 'pg_placement_num_target': 42, 'pg_num_target': 42, 'pg_num_pending': 42, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1272', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '89', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2021-02-10T04:25:47.658613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 13, 'pg_placement_num': 13, 'pg_placement_num_target': 13, 'pg_num_target': 13, 'pg_num_pending': 13, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1211', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '1209', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'zone.rgw.log', 'create_time': '2021-02-10T04:25:47.968727+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '4.16', 'ready_epoch': 1256, 'last_epoch_started': 1255, 'last_epoch_clean': 1255, 'source_version': "1048'40", 'target_version': "1049'40"}, 'last_change': '1343', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '1256', 'last_force_op_resend_preluminous': '1341', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'zone.rgw.control', 'create_time': '2021-02-10T04:25:49.278260+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'zone.rgw.meta', 'create_time': '2021-02-10T04:25:51.792514+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 42, 'pg_placement_num': 42, 'pg_placement_num_target': 42, 'pg_num_target': 42, 'pg_num_pending': 42, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1046', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '1044', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 8}, 'application_metadata': {'rgw': {}}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2021-02-10T04:28:23.932391+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 36, 'pg_placement_num': 36, 'pg_placement_num_target': 36, 'pg_num_target': 36, 'pg_num_pending': 36, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1143', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '1141', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 888, 'snap_epoch': 994, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '45d0bb5f-35fa-4596-964f-75abcee53bee', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 1227, 'last_clean_end': 1315, 'up_from': 1332, 'up_thru': 1366, 'down_at': 1316, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6801', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6802', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6804', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6803', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6801/7', 'cluster_addr': '172.21.15.85:6802/7', 'heartbeat_back_addr': '172.21.15.85:6804/7', 'heartbeat_front_addr': '172.21.15.85:6803/7', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '3421a5ce-872e-4b52-adf7-6aaa595871a4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.1644744873046875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 1367, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6805', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6806', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6808', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6807', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6805/7', 'cluster_addr': '172.21.15.85:6806/7', 'heartbeat_back_addr': '172.21.15.85:6808/7', 'heartbeat_front_addr': '172.21.15.85:6807/7', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '96e1952a-1042-4316-9fc9-86813e88b912', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 1368, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6809', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6810', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6812', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6811', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6809/7', 'cluster_addr': '172.21.15.85:6810/7', 'heartbeat_back_addr': '172.21.15.85:6812/7', 'heartbeat_front_addr': '172.21.15.85:6811/7', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'e019f49f-5dd4-4104-9ced-94c710616642', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 25, 'last_clean_end': 1094, 'up_from': 1111, 'up_thru': 1366, 'down_at': 1095, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6813', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6814', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6816', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.85:6815', 'nonce': 7}]}, 'public_addr': '172.21.15.85:6813/7', 'cluster_addr': '172.21.15.85:6814/7', 'heartbeat_back_addr': '172.21.15.85:6816/7', 'heartbeat_front_addr': '172.21.15.85:6815/7', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '91c8a80a-37b6-4c30-a32a-647b2e6d801f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 1370, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6800', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6801', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6803', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6802', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6800/7', 'cluster_addr': '172.21.15.164:6801/7', 'heartbeat_back_addr': '172.21.15.164:6803/7', 'heartbeat_front_addr': '172.21.15.164:6802/7', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'cc5f0ad8-f6dd-43ef-86f4-967bd67c7b9e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 1366, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6804', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6805', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6807', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6806', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6804/7', 'cluster_addr': '172.21.15.164:6805/7', 'heartbeat_back_addr': '172.21.15.164:6807/7', 'heartbeat_front_addr': '172.21.15.164:6806/7', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'adc40430-3eeb-4003-a824-343021243077', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 43, 'up_thru': 1360, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6808', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6809', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6811', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6810', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6808/7', 'cluster_addr': '172.21.15.164:6809/7', 'heartbeat_back_addr': '172.21.15.164:6811/7', 'heartbeat_front_addr': '172.21.15.164:6810/7', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'a7a056ff-f7f2-4e3e-a1dd-e25ba2fd9a37', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.846160888671875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 1367, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6812', 'nonce': 7}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6813', 'nonce': 7}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6815', 'nonce': 7}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6814', 'nonce': 7}]}, 'public_addr': '172.21.15.164:6812/7', 'cluster_addr': '172.21.15.164:6813/7', 'heartbeat_back_addr': '172.21.15.164:6815/7', 'heartbeat_front_addr': '172.21.15.164:6814/7', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:59:56.261166+0000', 'dead_epoch': 1316}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:59:01.695880+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:59:20.883571+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T05:00:00.199359+0000', 'dead_epoch': 1095}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:59:56.313080+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:59:12.932203+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 65536, 'last_purged_snaps_scrub': '2021-02-10T04:59:31.289424+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-02-10T04:59:49.655402+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.23', 'osds': [3, 0, 6]}], 'pg_upmap_items': [{'pgid': '2.7', 'mappings': [{'from': 7, 'to': 4}]}, {'pgid': '2.1a', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '2.1d', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '2.21', 'mappings': [{'from': 7, 'to': 1}]}, {'pgid': '2.28', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.5', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '4.10', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '5.8', 'mappings': [{'from': 1, 'to': 3}]}, {'pgid': '5.10', 'mappings': [{'from': 7, 'to': 3}]}, {'pgid': '5.18', 'mappings': [{'from': 4, 'to': 3}]}, {'pgid': '5.1b', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '7.5', 'mappings': [{'from': 0, 'to': 2}]}, {'pgid': '7.6', 'mappings': [{'from': 3, 'to': 1}]}, {'pgid': '7.15', 'mappings': [{'from': 7, 'to': 4}]}], 'pg_temp': [{'pgid': '4.c', 'osds': [4, 3, 6]}, {'pgid': '4.e', 'osds': [4, 1, 6]}, {'pgid': '4.16', 'osds': [0, 1, 2]}, {'pgid': '4.18', 'osds': [4, 1, 6]}, {'pgid': '4.19', 'osds': [3, 1, 4]}, {'pgid': '4.1b', 'osds': [4, 1, 0]}, {'pgid': '4.1e', 'osds': [1, 4, 6]}], 'primary_temp': [], 'blocklist': {'172.21.15.85:0/801393150': '2021-02-11T04:21:37.851607+0000', '172.21.15.85:0/3049671696': '2021-02-11T04:21:37.851607+0000', '172.21.15.85:0/4056505602': '2021-02-11T04:21:20.700478+0000', '172.21.15.85:6800/7': '2021-02-11T04:21:20.700478+0000', '172.21.15.85:0/1879026062': '2021-02-11T04:21:20.700478+0000'}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2021-02-10T05:00:03.112 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph pg dump --format=json 2021-02-10T05:00:03.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:02 smithi164 conmon[29288]: cluster 2021-02-10T05:00:03.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:02 smithi164 conmon[29288]: 2021-02-10T05:00:01.697936 2021-02-10T05:00:03.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:02 smithi164 conmon[29288]: +0000 osd.7 (osd.7) 103 : cluster [DBG] 2.5 scrub ok 2021-02-10T05:00:03.346 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:00:03.699 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T05:00:03.717 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T05:00:03.749 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T05:00:03.767 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T05:00:03.787 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T05:00:03.805 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T05:00:03.826 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T05:00:03.844 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T05:00:03.846 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-02-10T05:00:03.846 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-02-10T05:00:03.846 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-02-10T05:00:03.846 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-02-10T05:00:03.846 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-02-10T05:00:03.846 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-02-10T05:00:03.846 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-02-10T05:00:03.846 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-02-10T05:00:03.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:03 smithi085 conmon[29319]: cluster 2021-02-10T05:00:02.279700+0000 osd.2 (osd.2) 2044 : cluster [DBG] 4.5 deep-scrub ok 2021-02-10T05:00:03.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:03 smithi085 conmon[29319]: cluster 2021-02-10T05:00:02.546198+0000 osd.1 (osd.1) 526 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:00:03.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:03 smithi085 conmon[29319]: cluster 2021-02-10T05:00:02.549101+0000 osd.1 (osd.1) 527 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:00:03.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:03 smithi085 conmon[29319]: cluster 2021-02-10T05:00:02.577092+0000 mgr.y (mgr.14140) 1547 : cluster [DBG] pgmap v2515: 208 pgs: 7 active+remapped+backfill_toofull, 201 active+clean; 856 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 693 B/s rd, 350 KiB/s wr, 1 op/s; 120/2166 objects misplaced (5.540%); 10 MiB/s, 17 objects/s recovering 2021-02-10T05:00:03.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:03 smithi085 conmon[29319]: cluster 2021-02-10T05:00:02.625491+0000 osd.1 (osd.1) 528 : cluster [DBG] 2.15 scrub ok 2021-02-10T05:00:03.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:03 smithi085 conmon[29319]: cluster 2021-02-10T05:00:02.659612+0000 osd.7 (osd.7) 104 : cluster [DBG] 5.9 scrub ok 2021-02-10T05:00:03.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:03 smithi085 conmon[29319]: cluster 2021-02-10T05:00:02.702991+0000 mon.a (mon.0) 3074 : cluster [DBG] osdmap e1374: 8 total, 8 up, 7 in 2021-02-10T05:00:03.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:03 smithi085 conmon[29319]: cluster 2021-02-10T05:00:02.811980+0000 osd.3 (osd.3) 52 : cluster [DBG] 4.0 scrub ok 2021-02-10T05:00:03.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:03 smithi085 conmon[29319]: cluster 2021-02-10T05:00:03.232866+0000 osd.4 (osd.4) 2675 : cluster [DBG] 6.17 scrub ok 2021-02-10T05:00:03.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:03 smithi085 conmon[29319]: cluster 2021-02-10T05:00:03.641953+0000 mon.a (mon.0) 3075 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-02-10T05:00:03.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:03 smithi085 conmon[29319]: cluster 2021-02-10T05:00:03.649465+0000 mon.a (mon.0) 3076 : cluster [DBG] osdmap e1375: 8 total, 8 up, 7 in 2021-02-10T05:00:03.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:03 smithi085 conmon[29319]: audit 2021-02-10T05:00:03.680922+0000 mon.a (mon.0) 3077 : audit [DBG] from='client.? 172.21.15.85:0/1005037121' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T05:00:03.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:03 smithi085 conmon[35961]: cluster 2021-02-10T05:00:02.279700+0000 osd.2 (osd.2) 2044 : cluster [DBG] 4.5 deep-scrub ok 2021-02-10T05:00:03.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:03 smithi085 conmon[35961]: cluster 2021-02-10T05:00:02.546198+0000 osd.1 (osd.1) 526 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:00:03.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:03 smithi085 conmon[35961]: cluster 2021-02-10T05:00:02.549101+0000 osd.1 (osd.1) 527 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:00:03.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:03 smithi085 conmon[35961]: cluster 2021-02-10T05:00:02.577092+0000 mgr.y (mgr.14140) 1547 : cluster [DBG] pgmap v2515: 208 pgs: 7 active+remapped+backfill_toofull, 201 active+clean; 856 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 693 B/s rd, 350 KiB/s wr, 1 op/s; 120/2166 objects misplaced (5.540%); 10 MiB/s, 17 objects/s recovering 2021-02-10T05:00:03.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:03 smithi085 conmon[35961]: cluster 2021-02-10T05:00:02.625491+0000 osd.1 (osd.1) 528 : cluster [DBG] 2.15 scrub ok 2021-02-10T05:00:03.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:03 smithi085 conmon[35961]: cluster 2021-02-10T05:00:02.659612+0000 osd.7 (osd.7) 104 : cluster [DBG] 5.9 scrub ok 2021-02-10T05:00:03.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:03 smithi085 conmon[35961]: cluster 2021-02-10T05:00:02.702991+0000 mon.a (mon.0) 3074 : cluster [DBG] osdmap e1374: 8 total, 8 up, 7 in 2021-02-10T05:00:03.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:03 smithi085 conmon[35961]: cluster 2021-02-10T05:00:02.811980+0000 2021-02-10T05:00:03.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:03 smithi085 conmon[35961]: osd.3 (osd.3) 52 : cluster [DBG] 4.0 scrub ok 2021-02-10T05:00:03.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:03 smithi085 conmon[35961]: cluster 2021-02-10T05:00:03.232866+0000 osd.4 (osd.4) 2675 : cluster [DBG] 6.17 scrub ok 2021-02-10T05:00:03.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:03 smithi085 conmon[35961]: cluster 2021-02-10T05:00:03.641953+0000 mon.a (mon.0) 3075 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-02-10T05:00:03.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:03 smithi085 conmon[35961]: cluster 2021-02-10T05:00:03.649465+0000 mon.a (mon.0) 3076 : cluster [DBG] osdmap e1375: 8 total, 8 up, 7 in 2021-02-10T05:00:03.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:03 smithi085 conmon[35961]: audit 2021-02-10T05:00:03.680922+0000 mon.a (mon.0) 3077 : audit [DBG] from='client.? 172.21.15.85:0/1005037121' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T05:00:04.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:03 smithi164 conmon[29288]: cluster 2021-02-10T05:00:02.279700+0000 osd.2 (osd 2021-02-10T05:00:04.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:03 smithi164 conmon[29288]: .2) 2044 : cluster [DBG] 4.5 deep-scrub ok 2021-02-10T05:00:04.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:03 smithi164 conmon[29288]: cluster 2021-02-10T05:00:02.546198+0000 osd.1 (osd.1) 526 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:00:04.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:03 smithi164 conmon[29288]: cluster 2021-02-10T05:00:02.549101+0000 osd.1 (osd.1) 527 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:00:04.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:03 smithi164 conmon[29288]: cluster 2021-02-10T05:00:02.577092+0000 mgr.y (mgr.14140) 1547 : cluster [DBG] pgmap v2515: 208 pgs: 7 active+remapped+backfill_toofull, 201 active+clean; 856 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 693 B/s rd, 350 KiB/s wr, 1 op/s; 120/2166 objects misplaced (5.540%); 10 MiB/s, 17 objects/s recovering 2021-02-10T05:00:04.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:03 smithi164 conmon[29288]: cluster 2021-02-10T05:00:02.625491+0000 osd.1 (osd.1) 528 : cluster [DBG] 2021-02-10T05:00:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:03 smithi164 conmon[29288]: 2.15 scrub ok 2021-02-10T05:00:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:03 smithi164 conmon[29288]: cluster 2021-02-10T05:00:02.659612+0000 osd.7 (osd 2021-02-10T05:00:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:03 smithi164 conmon[29288]: .7) 104 : cluster [DBG] 5.9 scrub ok 2021-02-10T05:00:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:03 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T05:00:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:03 smithi164 conmon[29288]: T05:00:02.702991+0000 mon.a (mon 2021-02-10T05:00:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:03 smithi164 conmon[29288]: .0) 3074 : cluster [DBG] osdmap e1374: 8 total, 8 up, 7 in 2021-02-10T05:00:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:03 smithi164 conmon[29288]: cluster 2021 2021-02-10T05:00:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:03 smithi164 conmon[29288]: -02-10T05:00:02.811980+0000 osd.3 2021-02-10T05:00:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:03 smithi164 conmon[29288]: (osd.3) 52 : cluster [DBG] 4.0 scrub ok 2021-02-10T05:00:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:03 smithi164 conmon[29288]: 2021-02-10T05:00:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:03 smithi164 conmon[29288]: cluster 2021-02-10T05: 2021-02-10T05:00:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:03 smithi164 conmon[29288]: 00:03.232866+0000 osd.4 (osd.4) 2021-02-10T05:00:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:03 smithi164 conmon[29288]: 2675 : cluster [DBG] 6.17 scrub ok 2021-02-10T05:00:04.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:03 smithi164 conmon[29288]: cluster 2021- 2021-02-10T05:00:04.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:03 smithi164 conmon[29288]: 02-10T05:00:03.641953+0000 2021-02-10T05:00:04.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:03 smithi164 conmon[29288]: mon.a (mon.0) 3075 : cluster [WRN] 2021-02-10T05:00:04.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:03 smithi164 conmon[29288]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-02-10T05:00:04.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:03 smithi164 conmon[29288]: cluster 2021-02-10T05:00: 2021-02-10T05:00:04.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:03 smithi164 conmon[29288]: 03.649465+0000 mon.a (mon.0) 3076 : cluster 2021-02-10T05:00:04.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:03 smithi164 conmon[29288]: [DBG] osdmap e1375: 8 total, 8 up, 7 in 2021-02-10T05:00:04.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:03 smithi164 conmon[29288]: audit 2021 2021-02-10T05:00:04.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:03 smithi164 conmon[29288]: -02-10T05:00:03.680922+0000 2021-02-10T05:00:04.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:03 smithi164 conmon[29288]: mon.a (mon.0) 3077 : audit [DBG] 2021-02-10T05:00:04.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:03 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/1005037121' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T05:00:04.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:04 smithi085 conmon[29319]: cluster 2021-02-10T05:00:03.253156 2021-02-10T05:00:04.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:04 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 2045 : cluster [DBG] 6.f scrub ok 2021-02-10T05:00:04.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:04 smithi085 conmon[29319]: cluster 2021-02-10T05:00:03.678662+0000 osd.1 (osd.1) 529 : cluster [DBG] 6.26 scrub ok 2021-02-10T05:00:04.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:04 smithi085 conmon[35961]: cluster 2021-02-10T05:00:03.253156+0000 2021-02-10T05:00:04.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:04 smithi085 conmon[35961]: osd.2 (osd.2) 2045 : cluster [DBG] 6.f scrub ok 2021-02-10T05:00:04.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:04 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T05:00:04.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:04 smithi085 conmon[35961]: -10T05:00:03.678662+0000 osd.1 (osd.1) 529 2021-02-10T05:00:04.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:04 smithi085 conmon[35961]: : cluster [DBG] 6.26 scrub ok 2021-02-10T05:00:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:04 smithi164 conmon[29288]: cluster 2021-02-10T05:00:03.253156+0000 osd.2 (osd.2) 2045 : cluster 2021-02-10T05:00:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:04 smithi164 conmon[29288]: [DBG] 6.f scrub ok 2021-02-10T05:00:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:04 smithi164 conmon[29288]: cluster 2021-02-10T05:00:03.678662+0000 osd.1 (osd.1) 529 : cluster [DBG] 6.26 scrub ok 2021-02-10T05:00:05.519 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T05:00:05.523 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T05:00:06.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:05 smithi164 conmon[29288]: cluster 2021-02-10T05:00:06.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:05 smithi164 conmon[29288]: 2021-02-10T05:00:06.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:05 smithi164 conmon[29288]: 2021 2021-02-10T05:00:06.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:05 smithi164 conmon[29288]: - 2021-02-10T05:00:06.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:05 smithi164 conmon[29288]: 02 2021-02-10T05:00:06.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:05 smithi164 conmon[29288]: - 2021-02-10T05:00:06.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:05 smithi164 conmon[29288]: 10 2021-02-10T05:00:06.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:05 smithi164 conmon[29288]: T 2021-02-10T05:00:06.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:05 smithi164 conmon[29288]: 05 2021-02-10T05:00:06.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:05 smithi164 conmon[29288]: : 2021-02-10T05:00:06.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:05 smithi164 conmon[29288]: 00:04.256059+0000 osd.2 (osd. 2021-02-10T05:00:06.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:05 smithi164 conmon[29288]: 2) 2046 : cluster [DBG] 5.a scrub ok 2021-02-10T05:00:06.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:05 smithi164 conmon[29288]: cluster 2021-02-10T05:00 2021-02-10T05:00:06.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:05 smithi164 conmon[29288]: :04.577470+0000 mgr.y (mgr.14140) 1548 : cluster [DBG] 2021-02-10T05:00:06.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:05 smithi164 conmon[29288]: pgmap v2518: 208 pgs: 7 active+remapped+backfill_toofull, 201 active+clean; 856 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 869 B/s rd, 0 op/s; 120/2166 objects misplaced (5.540%) 2021-02-10T05:00:06.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:05 smithi164 conmon[29288]: cluster 2021-02-10T05:00:06.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:05 smithi164 conmon[29288]: 2021-02-10T05:00:05.122847+0000 osd.5 (osd.5) 206 : cluster [DBG] 6.a scrub ok 2021-02-10T05:00:06.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:05 smithi085 conmon[29319]: cluster 2021-02-10T05:00:04.256059+0000 2021-02-10T05:00:06.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:05 smithi085 conmon[29319]: osd.2 (osd.2) 2046 : cluster [DBG] 5.a scrub ok 2021-02-10T05:00:06.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:05 smithi085 conmon[29319]: cluster 2021-02-10T05:00:04.577470+0000 mgr.y (mgr.14140) 1548 : cluster [DBG] pgmap v2518: 208 pgs: 7 active+remapped+backfill_toofull, 201 active+clean; 856 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 869 B/s rd, 0 op/s; 120/2166 objects misplaced (5.540%) 2021-02-10T05:00:06.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:05 smithi085 conmon[29319]: cluster 2021-02-10T05:00:05.122847+0000 osd.5 (osd.5) 206 : cluster [DBG] 6.a scrub ok 2021-02-10T05:00:06.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:05 smithi085 conmon[35961]: cluster 2021-02-10T05:00:04.256059+0000 osd.2 (osd.2 2021-02-10T05:00:06.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:05 smithi085 conmon[35961]: ) 2046 : cluster [DBG] 5.a scrub ok 2021-02-10T05:00:06.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:05 smithi085 conmon[35961]: cluster 2021-02-10T05:00:04.577470+0000 mgr.y (mgr.14140) 1548 2021-02-10T05:00:06.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:05 smithi085 conmon[35961]: : cluster [DBG] pgmap v2518: 208 pgs: 7 active+remapped+backfill_toofull, 201 active+clean; 856 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 869 B/s rd, 0 op/s; 120/2166 objects misplaced (5.540%) 2021-02-10T05:00:06.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:05 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T05:00:06.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:05 smithi085 conmon[35961]: -10T05:00:05.122847+0000 osd.5 (osd.5) 206 : 2021-02-10T05:00:06.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:05 smithi085 conmon[35961]: cluster [DBG] 6.a scrub ok 2021-02-10T05:00:06.842 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[35961]: cluster 2021-02-10T05:00: 2021-02-10T05:00:06.842 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[35961]: 05.286876+0000 osd.2 (osd.2) 2047 : cluster [DBG] 2.c scrub ok 2021-02-10T05:00:06.842 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[35961]: audit 2021-02-10T05:00:06.842 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[35961]: 2021-02-10T05:00: 2021-02-10T05:00:06.842 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[35961]: 05.497718+0000 mgr.y (mgr. 2021-02-10T05:00:06.842 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[35961]: 14140) 1549 : audit [DBG] from='client.16683 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T05:00:06.842 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[35961]: 2021-02-10T05:00:06.842 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[35961]: cluster 2021- 2021-02-10T05:00:06.843 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[35961]: 02-10T05:00:05.604037+0000 2021-02-10T05:00:06.843 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[35961]: osd.7 (osd.7) 105 : cluster 2021-02-10T05:00:06.843 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[35961]: [DBG] 5.1a scrub ok 2021-02-10T05:00:06.843 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[35961]: cluster 2021- 2021-02-10T05:00:06.843 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[35961]: 02-10T05:00:05.680310+0000 2021-02-10T05:00:06.843 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[35961]: osd.1 (osd.1) 530 : 2021-02-10T05:00:06.843 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[35961]: cluster [DBG] 4.7 scrub ok 2021-02-10T05:00:06.843 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[35961]: cluster 2021- 2021-02-10T05:00:06.843 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[35961]: 02-10T05:00:06.153621+0000 2021-02-10T05:00:06.843 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[35961]: osd.4 (osd.4) 2676 : 2021-02-10T05:00:06.843 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[35961]: cluster [DBG] 7.22 deep-scrub ok 2021-02-10T05:00:06.843 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[29319]: cluster 2021-02-10T05:00:05.286876+0000 osd.2 (osd.2) 2021-02-10T05:00:06.844 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[29319]: 2047 : cluster [DBG] 2.c scrub ok 2021-02-10T05:00:06.844 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[29319]: audit 2021-02-10T05:00:05.497718+0000 mgr.y (mgr.14140) 1549 : audit 2021-02-10T05:00:06.844 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[29319]: [DBG] from='client.16683 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T05:00:06.844 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[29319]: cluster 2021- 2021-02-10T05:00:06.844 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[29319]: 02-10T05:00 2021-02-10T05:00:06.844 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[29319]: :05.604037+0000 osd.7 (osd.7) 2021-02-10T05:00:06.844 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[29319]: 105 : cluster [DBG] 5.1a scrub ok 2021-02-10T05:00:06.844 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T05:00:06.844 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[29319]: 10T05:00:05.680310+0000 osd.1 (osd. 2021-02-10T05:00:06.844 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[29319]: 1) 530 : cluster [DBG] 4.7 scrub ok 2021-02-10T05:00:06.844 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[29319]: cluster 2021-02-10T05:00:06.844 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[29319]: 2021-02-10T05:00:06. 2021-02-10T05:00:06.844 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[29319]: 153621+0000 osd.4 (osd.4) 2676 : cluster 2021-02-10T05:00:06.845 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:06 smithi085 conmon[29319]: [DBG] 7.22 deep-scrub ok 2021-02-10T05:00:06.857 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":2518,"stamp":"2021-02-10T05:00:04.577221+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":897771510,"num_objects":722,"num_object_clones":315,"num_object_copies":2166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":120,"num_objects_unfound":0,"num_objects_dirty":722,"num_whiteouts":10,"num_read":8492,"num_read_kb":5225309,"num_write":5329,"num_write_kb":1422352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1611,"num_bytes_recovered":575201585,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35858,"ondisk_log_size":35858,"up":624,"acting":624,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":634,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2277312,"kb_used_data":1979400,"kb_used_omap":711,"kb_used_meta":297144,"kb_avail":653938752,"statfs":{"total":671965249536,"available":669633282048,"internally_reserved":0,"allocated":2026905600,"data_stored":2537984347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":728429,"internal_metadata":304276115},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3042034,"num_objects":-1,"num_object_clones":-1,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.890114"},"pg_stats":[{"pgid":"6.29","version":"0'0","reported_seq":"1243","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.779349+0000","last_change":"2021-02-10T04:58:56.856970+0000","last_active":"2021-02-10T04:59:57.779349+0000","last_peered":"2021-02-10T04:59:57.779349+0000","last_clean":"2021-02-10T04:59:57.779349+0000","last_became_active":"2021-02-10T04:58:56.856652+0000","last_became_peered":"2021-02-10T04:58:56.856652+0000","last_unstale":"2021-02-10T04:59:57.779349+0000","last_undegraded":"2021-02-10T04:59:57.779349+0000","last_fullsized":"2021-02-10T04:59:57.779349+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:24.121684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:24.121684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":"278","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.688973+0000","last_change":"2021-02-10T04:59:51.125621+0000","last_active":"2021-02-10T04:59:58.688973+0000","last_peered":"2021-02-10T04:59:58.688973+0000","last_clean":"2021-02-10T04:59:58.688973+0000","last_became_active":"2021-02-10T04:59:51.125212+0000","last_became_peered":"2021-02-10T04:59:51.125212+0000","last_unstale":"2021-02-10T04:59:58.688973+0000","last_undegraded":"2021-02-10T04:59:58.688973+0000","last_fullsized":"2021-02-10T04:59:58.688973+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.133043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.133043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"1365'5","reported_seq":"410","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:58.087336+0000","last_change":"2021-02-10T04:59:51.188665+0000","last_active":"2021-02-10T04:59:58.087336+0000","last_peered":"2021-02-10T04:59:58.087336+0000","last_clean":"2021-02-10T04:59:58.087336+0000","last_became_active":"2021-02-10T04:59:51.116307+0000","last_became_peered":"2021-02-10T04:59:51.116307+0000","last_unstale":"2021-02-10T04:59:58.087336+0000","last_undegraded":"2021-02-10T04:59:58.087336+0000","last_fullsized":"2021-02-10T04:59:58.087336+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'3","last_scrub_stamp":"2021-02-10T04:51:01.185820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.185820+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"1261'84","reported_seq":"1563","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.701900+0000","last_change":"2021-02-10T04:59:53.658456+0000","last_active":"2021-02-10T05:00:00.701900+0000","last_peered":"2021-02-10T05:00:00.701900+0000","last_clean":"2021-02-10T05:00:00.701900+0000","last_became_active":"2021-02-10T04:59:53.657939+0000","last_became_peered":"2021-02-10T04:59:53.657939+0000","last_unstale":"2021-02-10T05:00:00.701900+0000","last_undegraded":"2021-02-10T05:00:00.701900+0000","last_fullsized":"2021-02-10T05:00:00.701900+0000","mapping_epoch":1366,"log_start":"1132'66","ondisk_log_start":"1132'66","created":55,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"1132'66","last_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_deep_scrub":"1132'66","last_deep_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.430399+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":808,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":93,"num_read_kb":75,"num_write":53,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"278","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.690228+0000","last_change":"2021-02-10T04:53:41.722609+0000","last_active":"2021-02-10T04:59:58.690228+0000","last_peered":"2021-02-10T04:59:58.690228+0000","last_clean":"2021-02-10T04:59:58.690228+0000","last_became_active":"2021-02-10T04:51:36.875000+0000","last_became_peered":"2021-02-10T04:51:36.875000+0000","last_unstale":"2021-02-10T04:59:58.690228+0000","last_undegraded":"2021-02-10T04:59:58.690228+0000","last_fullsized":"2021-02-10T04:59:58.690228+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:53:41.722516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:41.722516+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"1460","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.707467+0000","last_change":"2021-02-10T04:59:51.117362+0000","last_active":"2021-02-10T05:00:00.707467+0000","last_peered":"2021-02-10T05:00:00.707467+0000","last_clean":"2021-02-10T05:00:00.707467+0000","last_became_active":"2021-02-10T04:59:51.116877+0000","last_became_peered":"2021-02-10T04:59:51.116877+0000","last_unstale":"2021-02-10T05:00:00.707467+0000","last_undegraded":"2021-02-10T05:00:00.707467+0000","last_fullsized":"2021-02-10T05:00:00.707467+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:48.408748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:50:48.408748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"1003'1689","reported_seq":"5046","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:59.694219+0000","last_change":"2021-02-10T04:59:58.577583+0000","last_active":"2021-02-10T04:59:59.694219+0000","last_peered":"2021-02-10T04:59:59.694219+0000","last_clean":"2021-02-10T04:59:59.694219+0000","last_became_active":"2021-02-10T04:59:54.659936+0000","last_became_peered":"2021-02-10T04:59:54.659936+0000","last_unstale":"2021-02-10T04:59:59.694219+0000","last_undegraded":"2021-02-10T04:59:59.694219+0000","last_fullsized":"2021-02-10T04:59:59.694219+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:59:58.577489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:59:58.577489+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":99361981,"num_objects":37,"num_object_clones":33,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":76107542,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"62'5","reported_seq":"513","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:02.127511+0000","last_change":"2021-02-10T04:59:51.183782+0000","last_active":"2021-02-10T05:00:02.127511+0000","last_peered":"2021-02-10T05:00:02.127511+0000","last_clean":"2021-02-10T05:00:02.127511+0000","last_became_active":"2021-02-10T04:59:51.113091+0000","last_became_peered":"2021-02-10T04:59:51.113091+0000","last_unstale":"2021-02-10T05:00:02.127511+0000","last_undegraded":"2021-02-10T05:00:02.127511+0000","last_fullsized":"2021-02-10T05:00:02.127511+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":472,"num_read_kb":471,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"1261'28","reported_seq":"1375","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.780109+0000","last_change":"2021-02-10T04:59:13.926028+0000","last_active":"2021-02-10T04:59:57.780109+0000","last_peered":"2021-02-10T04:59:57.780109+0000","last_clean":"2021-02-10T04:59:57.780109+0000","last_became_active":"2021-02-10T04:59:13.925333+0000","last_became_peered":"2021-02-10T04:59:13.925333+0000","last_unstale":"2021-02-10T04:59:57.780109+0000","last_undegraded":"2021-02-10T04:59:57.780109+0000","last_fullsized":"2021-02-10T04:59:57.780109+0000","mapping_epoch":1332,"log_start":"1061'20","ondisk_log_start":"1061'20","created":55,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'32","last_scrub_stamp":"2021-02-10T04:51:51.382040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.382040+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":579,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":29,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1026,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"823","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.688903+0000","last_change":"2021-02-10T04:59:40.662477+0000","last_active":"2021-02-10T04:59:58.688903+0000","last_peered":"2021-02-10T04:59:58.688903+0000","last_clean":"2021-02-10T04:59:58.688903+0000","last_became_active":"2021-02-10T04:59:40.661863+0000","last_became_peered":"2021-02-10T04:59:40.661863+0000","last_unstale":"2021-02-10T04:59:58.688903+0000","last_undegraded":"2021-02-10T04:59:58.688903+0000","last_fullsized":"2021-02-10T04:59:58.688903+0000","mapping_epoch":1356,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:34.178949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.178949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"269","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.689302+0000","last_change":"2021-02-10T04:51:53.038893+0000","last_active":"2021-02-10T04:59:58.689302+0000","last_peered":"2021-02-10T04:59:58.689302+0000","last_clean":"2021-02-10T04:59:58.689302+0000","last_became_active":"2021-02-10T04:51:36.855507+0000","last_became_peered":"2021-02-10T04:51:36.855507+0000","last_unstale":"2021-02-10T04:59:58.689302+0000","last_undegraded":"2021-02-10T04:59:58.689302+0000","last_fullsized":"2021-02-10T04:59:58.689302+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:53.038834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:53.038834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"1287","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.778892+0000","last_change":"2021-02-10T04:59:08.662159+0000","last_active":"2021-02-10T04:59:57.778892+0000","last_peered":"2021-02-10T04:59:57.778892+0000","last_clean":"2021-02-10T04:59:57.778892+0000","last_became_active":"2021-02-10T04:58:56.865590+0000","last_became_peered":"2021-02-10T04:58:56.865590+0000","last_unstale":"2021-02-10T04:59:57.778892+0000","last_undegraded":"2021-02-10T04:59:57.778892+0000","last_fullsized":"2021-02-10T04:59:57.778892+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:58:57.081143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:58:57.081143+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"1168'82","reported_seq":"305","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.689076+0000","last_change":"2021-02-10T04:53:23.326363+0000","last_active":"2021-02-10T04:59:58.689076+0000","last_peered":"2021-02-10T04:59:58.689076+0000","last_clean":"2021-02-10T04:59:58.689076+0000","last_became_active":"2021-02-10T04:51:36.859215+0000","last_became_peered":"2021-02-10T04:51:36.859215+0000","last_unstale":"2021-02-10T04:59:58.689076+0000","last_undegraded":"2021-02-10T04:59:58.689076+0000","last_fullsized":"2021-02-10T04:59:58.689076+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1150'76","last_scrub_stamp":"2021-02-10T04:53:00.964499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:00.964499+0000","log_size":82,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3848803,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"1261'69","reported_seq":"1460","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.707285+0000","last_change":"2021-02-10T04:59:53.656353+0000","last_active":"2021-02-10T05:00:00.707285+0000","last_peered":"2021-02-10T05:00:00.707285+0000","last_clean":"2021-02-10T05:00:00.707285+0000","last_became_active":"2021-02-10T04:59:53.655770+0000","last_became_peered":"2021-02-10T04:59:53.655770+0000","last_unstale":"2021-02-10T05:00:00.707285+0000","last_undegraded":"2021-02-10T05:00:00.707285+0000","last_fullsized":"2021-02-10T05:00:00.707285+0000","mapping_epoch":1366,"log_start":"1112'47","ondisk_log_start":"1112'47","created":55,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'61","last_scrub_stamp":"2021-02-10T04:53:36.836202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:36.836202+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":733,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":103,"num_read_kb":85,"num_write":61,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1099,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"1334'8","reported_seq":"407","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T05:00:00.529970+0000","last_change":"2021-02-10T04:58:56.880162+0000","last_active":"2021-02-10T05:00:00.529970+0000","last_peered":"2021-02-10T05:00:00.529970+0000","last_clean":"2021-02-10T05:00:00.529970+0000","last_became_active":"2021-02-10T04:58:56.879760+0000","last_became_peered":"2021-02-10T04:58:56.879760+0000","last_unstale":"2021-02-10T05:00:00.529970+0000","last_undegraded":"2021-02-10T05:00:00.529970+0000","last_fullsized":"2021-02-10T05:00:00.529970+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'4","last_scrub_stamp":"2021-02-10T04:52:01.121380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.121380+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"299","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.689434+0000","last_change":"2021-02-10T04:58:56.964109+0000","last_active":"2021-02-10T04:59:58.689434+0000","last_peered":"2021-02-10T04:59:58.689434+0000","last_clean":"2021-02-10T04:59:58.689434+0000","last_became_active":"2021-02-10T04:58:56.963768+0000","last_became_peered":"2021-02-10T04:58:56.963768+0000","last_unstale":"2021-02-10T04:59:58.689434+0000","last_undegraded":"2021-02-10T04:59:58.689434+0000","last_fullsized":"2021-02-10T04:59:58.689434+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:55:22.573621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:55:22.573621+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"668","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.779582+0000","last_change":"2021-02-10T04:53:46.733236+0000","last_active":"2021-02-10T04:59:57.779582+0000","last_peered":"2021-02-10T04:59:57.779582+0000","last_clean":"2021-02-10T04:59:57.779582+0000","last_became_active":"2021-02-10T04:53:46.732890+0000","last_became_peered":"2021-02-10T04:53:46.732890+0000","last_unstale":"2021-02-10T04:59:57.779582+0000","last_undegraded":"2021-02-10T04:59:57.779582+0000","last_fullsized":"2021-02-10T04:59:57.779582+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:35.170157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:35.170157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"1003'739","reported_seq":"3961","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.778988+0000","last_change":"2021-02-10T04:59:08.649874+0000","last_active":"2021-02-10T04:59:57.778988+0000","last_peered":"2021-02-10T04:59:57.778988+0000","last_clean":"2021-02-10T04:59:57.778988+0000","last_became_active":"2021-02-10T04:58:56.956888+0000","last_became_peered":"2021-02-10T04:58:56.956888+0000","last_unstale":"2021-02-10T04:59:57.778988+0000","last_undegraded":"2021-02-10T04:59:57.778988+0000","last_fullsized":"2021-02-10T04:59:57.778988+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:52:27.602006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:27.602006+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24059888,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":129,"num_read_kb":109544,"num_write":102,"num_write_kb":46173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"1200","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.687223+0000","last_change":"2021-02-10T04:59:51.125208+0000","last_active":"2021-02-10T04:59:58.687223+0000","last_peered":"2021-02-10T04:59:58.687223+0000","last_clean":"2021-02-10T04:59:58.687223+0000","last_became_active":"2021-02-10T04:59:51.124785+0000","last_became_peered":"2021-02-10T04:59:51.124785+0000","last_unstale":"2021-02-10T04:59:58.687223+0000","last_undegraded":"2021-02-10T04:59:58.687223+0000","last_fullsized":"2021-02-10T04:59:58.687223+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"1261'64","reported_seq":"1567","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.689059+0000","last_change":"2021-02-10T04:59:13.932815+0000","last_active":"2021-02-10T04:59:58.689059+0000","last_peered":"2021-02-10T04:59:58.689059+0000","last_clean":"2021-02-10T04:59:58.689059+0000","last_became_active":"2021-02-10T04:59:13.931863+0000","last_became_peered":"2021-02-10T04:59:13.931863+0000","last_unstale":"2021-02-10T04:59:58.689059+0000","last_undegraded":"2021-02-10T04:59:58.689059+0000","last_fullsized":"2021-02-10T04:59:58.689059+0000","mapping_epoch":1118,"log_start":"1066'44","ondisk_log_start":"1066'44","created":55,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":0,"last_scrub":"1066'44","last_scrub_stamp":"2021-02-10T04:53:37.108677+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:53:37.108677+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":789,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":90,"num_read_kb":72,"num_write":53,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"1267","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.687999+0000","last_change":"2021-02-10T04:53:39.126356+0000","last_active":"2021-02-10T04:59:58.687999+0000","last_peered":"2021-02-10T04:59:58.687999+0000","last_clean":"2021-02-10T04:59:58.687999+0000","last_became_active":"2021-02-10T04:29:07.145616+0000","last_became_peered":"2021-02-10T04:29:07.145616+0000","last_unstale":"2021-02-10T04:59:58.687999+0000","last_undegraded":"2021-02-10T04:59:58.687999+0000","last_fullsized":"2021-02-10T04:59:58.687999+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:39.126261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:39.126261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"803","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.687942+0000","last_change":"2021-02-10T04:59:40.660733+0000","last_active":"2021-02-10T04:59:58.687942+0000","last_peered":"2021-02-10T04:59:58.687942+0000","last_clean":"2021-02-10T04:59:58.687942+0000","last_became_active":"2021-02-10T04:59:40.660383+0000","last_became_peered":"2021-02-10T04:59:40.660383+0000","last_unstale":"2021-02-10T04:59:58.687942+0000","last_undegraded":"2021-02-10T04:59:58.687942+0000","last_fullsized":"2021-02-10T04:59:58.687942+0000","mapping_epoch":1356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:26.311958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.311958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"315","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.690131+0000","last_change":"2021-02-10T04:58:39.423097+0000","last_active":"2021-02-10T04:59:58.690131+0000","last_peered":"2021-02-10T04:59:58.690131+0000","last_clean":"2021-02-10T04:59:58.690131+0000","last_became_active":"2021-02-10T04:51:36.855744+0000","last_became_peered":"2021-02-10T04:51:36.855744+0000","last_unstale":"2021-02-10T04:59:58.690131+0000","last_undegraded":"2021-02-10T04:59:58.690131+0000","last_fullsized":"2021-02-10T04:59:58.690131+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:02.162851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:13.727241+0000","last_clean_scrub_stamp":"2021-02-10T04:52:02.162851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"1063'654","reported_seq":"4563","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.690181+0000","last_change":"2021-02-10T04:58:56.966357+0000","last_active":"2021-02-10T04:59:58.690181+0000","last_peered":"2021-02-10T04:59:58.690181+0000","last_clean":"2021-02-10T04:59:58.690181+0000","last_became_active":"2021-02-10T04:58:56.965471+0000","last_became_peered":"2021-02-10T04:58:56.965471+0000","last_unstale":"2021-02-10T04:59:58.690181+0000","last_undegraded":"2021-02-10T04:59:58.690181+0000","last_fullsized":"2021-02-10T04:59:58.690181+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:51:57.121838+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.121838+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24700630,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":171,"num_read_kb":182272,"num_write":111,"num_write_kb":33727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":26349023,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"1319","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.701989+0000","last_change":"2021-02-10T04:51:50.433056+0000","last_active":"2021-02-10T05:00:00.701989+0000","last_peered":"2021-02-10T05:00:00.701989+0000","last_clean":"2021-02-10T05:00:00.701989+0000","last_became_active":"2021-02-10T04:51:36.843165+0000","last_became_peered":"2021-02-10T04:51:36.843165+0000","last_unstale":"2021-02-10T05:00:00.701989+0000","last_undegraded":"2021-02-10T05:00:00.701989+0000","last_fullsized":"2021-02-10T05:00:00.701989+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.432990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.432990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"1354","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.683120+0000","last_change":"2021-02-10T04:58:56.965752+0000","last_active":"2021-02-10T04:59:58.683120+0000","last_peered":"2021-02-10T04:59:58.683120+0000","last_clean":"2021-02-10T04:59:58.683120+0000","last_became_active":"2021-02-10T04:58:56.963934+0000","last_became_peered":"2021-02-10T04:58:56.963934+0000","last_unstale":"2021-02-10T04:59:58.683120+0000","last_undegraded":"2021-02-10T04:59:58.683120+0000","last_fullsized":"2021-02-10T04:59:58.683120+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"1263","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.779424+0000","last_change":"2021-02-10T04:53:46.727888+0000","last_active":"2021-02-10T04:59:57.779424+0000","last_peered":"2021-02-10T04:59:57.779424+0000","last_clean":"2021-02-10T04:59:57.779424+0000","last_became_active":"2021-02-10T04:53:46.727670+0000","last_became_peered":"2021-02-10T04:53:46.727670+0000","last_unstale":"2021-02-10T04:59:57.779424+0000","last_undegraded":"2021-02-10T04:59:57.779424+0000","last_fullsized":"2021-02-10T04:59:57.779424+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:41.041553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.041553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"1261'76","reported_seq":"1503","reported_epoch":"1372","state":"active+remapped+backfill_toofull","last_fresh":"2021-02-10T04:59:58.688141+0000","last_change":"2021-02-10T04:59:52.122307+0000","last_active":"2021-02-10T04:59:58.688141+0000","last_peered":"2021-02-10T04:59:58.688141+0000","last_clean":"2021-02-10T04:59:49.068531+0000","last_became_active":"2021-02-10T04:59:52.104080+0000","last_became_peered":"2021-02-10T04:59:52.104080+0000","last_unstale":"2021-02-10T04:59:58.688141+0000","last_undegraded":"2021-02-10T04:59:58.688141+0000","last_fullsized":"2021-02-10T04:59:58.688141+0000","mapping_epoch":1364,"log_start":"1049'39","ondisk_log_start":"1049'39","created":55,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":0,"last_scrub":"1187'54","last_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_deep_scrub":"1187'54","last_deep_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_clean_scrub_stamp":"2021-02-10T04:53:42.071103+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":289,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":73,"num_read_kb":63,"num_write":47,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,6],"avail_no_missing":["4","3","6"],"object_location_counts":[{"shards":"3,4,6","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"804","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.683235+0000","last_change":"2021-02-10T04:59:40.658734+0000","last_active":"2021-02-10T04:59:58.683235+0000","last_peered":"2021-02-10T04:59:58.683235+0000","last_clean":"2021-02-10T04:59:58.683235+0000","last_became_active":"2021-02-10T04:59:40.658301+0000","last_became_peered":"2021-02-10T04:59:40.658301+0000","last_unstale":"2021-02-10T04:59:58.683235+0000","last_undegraded":"2021-02-10T04:59:58.683235+0000","last_fullsized":"2021-02-10T04:59:58.683235+0000","mapping_epoch":1356,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:52.350706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.350706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"1119'4","reported_seq":"375","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.689906+0000","last_change":"2021-02-10T04:52:27.126670+0000","last_active":"2021-02-10T04:59:58.689906+0000","last_peered":"2021-02-10T04:59:58.689906+0000","last_clean":"2021-02-10T04:59:58.689906+0000","last_became_active":"2021-02-10T04:51:36.853647+0000","last_became_peered":"2021-02-10T04:51:36.853647+0000","last_unstale":"2021-02-10T04:59:58.689906+0000","last_undegraded":"2021-02-10T04:59:58.689906+0000","last_fullsized":"2021-02-10T04:59:58.689906+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'4","last_scrub_stamp":"2021-02-10T04:52:27.126599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:27.126599+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"1358","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.689208+0000","last_change":"2021-02-10T04:52:25.745660+0000","last_active":"2021-02-10T04:59:58.689208+0000","last_peered":"2021-02-10T04:59:58.689208+0000","last_clean":"2021-02-10T04:59:58.689208+0000","last_became_active":"2021-02-10T04:51:36.849503+0000","last_became_peered":"2021-02-10T04:51:36.849503+0000","last_unstale":"2021-02-10T04:59:58.689208+0000","last_undegraded":"2021-02-10T04:59:58.689208+0000","last_fullsized":"2021-02-10T04:59:58.689208+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:25.745577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.745577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"1004'1093","reported_seq":"6544","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.780845+0000","last_change":"2021-02-10T04:59:08.653144+0000","last_active":"2021-02-10T04:59:57.780845+0000","last_peered":"2021-02-10T04:59:57.780845+0000","last_clean":"2021-02-10T04:59:57.780845+0000","last_became_active":"2021-02-10T04:58:56.875893+0000","last_became_peered":"2021-02-10T04:58:56.875893+0000","last_unstale":"2021-02-10T04:59:57.780845+0000","last_undegraded":"2021-02-10T04:59:57.780845+0000","last_fullsized":"2021-02-10T04:59:57.780845+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39434696,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":279,"num_read_kb":311058,"num_write":185,"num_write_kb":66790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"62'5","reported_seq":"171","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.681810+0000","last_change":"2021-02-10T04:54:39.559473+0000","last_active":"2021-02-10T04:59:58.681810+0000","last_peered":"2021-02-10T04:59:58.681810+0000","last_clean":"2021-02-10T04:59:58.681810+0000","last_became_active":"2021-02-10T04:54:39.559048+0000","last_became_peered":"2021-02-10T04:54:39.559048+0000","last_unstale":"2021-02-10T04:59:58.681810+0000","last_undegraded":"2021-02-10T04:59:58.681810+0000","last_fullsized":"2021-02-10T04:59:58.681810+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":149,"num_read_kb":148,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"1353","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.707362+0000","last_change":"2021-02-10T04:59:51.119011+0000","last_active":"2021-02-10T05:00:00.707362+0000","last_peered":"2021-02-10T05:00:00.707362+0000","last_clean":"2021-02-10T05:00:00.707362+0000","last_became_active":"2021-02-10T04:59:51.118611+0000","last_became_peered":"2021-02-10T04:59:51.118611+0000","last_unstale":"2021-02-10T05:00:00.707362+0000","last_undegraded":"2021-02-10T05:00:00.707362+0000","last_fullsized":"2021-02-10T05:00:00.707362+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:12.389272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:12.389272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"1261'71","reported_seq":"1085","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:59.694065+0000","last_change":"2021-02-10T04:59:13.930947+0000","last_active":"2021-02-10T04:59:59.694065+0000","last_peered":"2021-02-10T04:59:59.694065+0000","last_clean":"2021-02-10T04:59:59.694065+0000","last_became_active":"2021-02-10T04:59:13.930554+0000","last_became_peered":"2021-02-10T04:59:13.930554+0000","last_unstale":"2021-02-10T04:59:59.694065+0000","last_undegraded":"2021-02-10T04:59:59.694065+0000","last_fullsized":"2021-02-10T04:59:59.694065+0000","mapping_epoch":1129,"log_start":"1049'51","ondisk_log_start":"1049'51","created":55,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'41","last_scrub_stamp":"2021-02-10T04:51:58.353660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:58.353660+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":68,"num_write":51,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":525,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"804","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.779561+0000","last_change":"2021-02-10T04:59:40.660482+0000","last_active":"2021-02-10T04:59:57.779561+0000","last_peered":"2021-02-10T04:59:57.779561+0000","last_clean":"2021-02-10T04:59:57.779561+0000","last_became_active":"2021-02-10T04:59:40.660098+0000","last_became_peered":"2021-02-10T04:59:40.660098+0000","last_unstale":"2021-02-10T04:59:57.779561+0000","last_undegraded":"2021-02-10T04:59:57.779561+0000","last_fullsized":"2021-02-10T04:59:57.779561+0000","mapping_epoch":1356,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:11.271459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.271459+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"1333'7","reported_seq":"1787","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.689128+0000","last_change":"2021-02-10T04:58:56.875943+0000","last_active":"2021-02-10T04:59:58.689128+0000","last_peered":"2021-02-10T04:59:58.689128+0000","last_clean":"2021-02-10T04:59:58.689128+0000","last_became_active":"2021-02-10T04:58:56.875512+0000","last_became_peered":"2021-02-10T04:58:56.875512+0000","last_unstale":"2021-02-10T04:59:58.689128+0000","last_undegraded":"2021-02-10T04:59:58.689128+0000","last_fullsized":"2021-02-10T04:59:58.689128+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:51:47.855283+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.855283+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"1378","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.706563+0000","last_change":"2021-02-10T04:58:56.854797+0000","last_active":"2021-02-10T05:00:00.706563+0000","last_peered":"2021-02-10T05:00:00.706563+0000","last_clean":"2021-02-10T05:00:00.706563+0000","last_became_active":"2021-02-10T04:58:56.854440+0000","last_became_peered":"2021-02-10T04:58:56.854440+0000","last_unstale":"2021-02-10T05:00:00.706563+0000","last_undegraded":"2021-02-10T05:00:00.706563+0000","last_fullsized":"2021-02-10T05:00:00.706563+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:55:22.183566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:55:22.183566+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"1002'1090","reported_seq":"5373","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.684336+0000","last_change":"2021-02-10T04:59:53.666790+0000","last_active":"2021-02-10T04:59:58.684336+0000","last_peered":"2021-02-10T04:59:58.684336+0000","last_clean":"2021-02-10T04:59:58.684336+0000","last_became_active":"2021-02-10T04:59:53.664286+0000","last_became_peered":"2021-02-10T04:59:53.664286+0000","last_unstale":"2021-02-10T04:59:58.684336+0000","last_undegraded":"2021-02-10T04:59:58.684336+0000","last_fullsized":"2021-02-10T04:59:58.684336+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:51:08.220472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.220472+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24410646,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":245,"num_read_kb":276168,"num_write":175,"num_write_kb":64960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":36920818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"1338","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.687032+0000","last_change":"2021-02-10T04:54:39.555379+0000","last_active":"2021-02-10T04:59:58.687032+0000","last_peered":"2021-02-10T04:59:58.687032+0000","last_clean":"2021-02-10T04:59:58.687032+0000","last_became_active":"2021-02-10T04:54:39.554976+0000","last_became_peered":"2021-02-10T04:54:39.554976+0000","last_unstale":"2021-02-10T04:59:58.687032+0000","last_undegraded":"2021-02-10T04:59:58.687032+0000","last_fullsized":"2021-02-10T04:59:58.687032+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"1261'65","reported_seq":"1497","reported_epoch":"1372","state":"active+remapped+backfill_toofull","last_fresh":"2021-02-10T04:59:58.688745+0000","last_change":"2021-02-10T04:59:52.160041+0000","last_active":"2021-02-10T04:59:58.688745+0000","last_peered":"2021-02-10T04:59:58.688745+0000","last_clean":"2021-02-10T04:59:49.068996+0000","last_became_active":"2021-02-10T04:59:52.099405+0000","last_became_peered":"2021-02-10T04:59:52.099405+0000","last_unstale":"2021-02-10T04:59:58.688745+0000","last_undegraded":"2021-02-10T04:59:58.688745+0000","last_fullsized":"2021-02-10T04:59:58.688745+0000","mapping_epoch":1364,"log_start":"1049'47","ondisk_log_start":"1049'47","created":55,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'47","last_scrub_stamp":"2021-02-10T04:53:09.002368+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.002368+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":836,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":84,"num_read_kb":67,"num_write":49,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":467,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,6],"avail_no_missing":["4","1","6"],"object_location_counts":[{"shards":"1,4,6","objects":11}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"323","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.779657+0000","last_change":"2021-02-10T04:58:56.842228+0000","last_active":"2021-02-10T04:59:57.779657+0000","last_peered":"2021-02-10T04:59:57.779657+0000","last_clean":"2021-02-10T04:59:57.779657+0000","last_became_active":"2021-02-10T04:58:56.841107+0000","last_became_peered":"2021-02-10T04:58:56.841107+0000","last_unstale":"2021-02-10T04:59:57.779657+0000","last_undegraded":"2021-02-10T04:59:57.779657+0000","last_fullsized":"2021-02-10T04:59:57.779657+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:08.358300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:50:08.358300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"1359","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.705710+0000","last_change":"2021-02-10T04:59:51.118440+0000","last_active":"2021-02-10T05:00:00.705710+0000","last_peered":"2021-02-10T05:00:00.705710+0000","last_clean":"2021-02-10T05:00:00.705710+0000","last_became_active":"2021-02-10T04:59:51.118081+0000","last_became_peered":"2021-02-10T04:59:51.118081+0000","last_unstale":"2021-02-10T05:00:00.705710+0000","last_undegraded":"2021-02-10T05:00:00.705710+0000","last_fullsized":"2021-02-10T05:00:00.705710+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.436862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.436862+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"1313","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.683974+0000","last_change":"2021-02-10T04:59:51.116502+0000","last_active":"2021-02-10T04:59:58.683974+0000","last_peered":"2021-02-10T04:59:58.683974+0000","last_clean":"2021-02-10T04:59:58.683974+0000","last_became_active":"2021-02-10T04:59:51.116133+0000","last_became_peered":"2021-02-10T04:59:51.116133+0000","last_unstale":"2021-02-10T04:59:58.683974+0000","last_undegraded":"2021-02-10T04:59:58.683974+0000","last_fullsized":"2021-02-10T04:59:58.683974+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:02.112533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:02.112533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"1001'1165","reported_seq":"4359","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.780283+0000","last_change":"2021-02-10T04:59:53.660114+0000","last_active":"2021-02-10T04:59:57.780283+0000","last_peered":"2021-02-10T04:59:57.780283+0000","last_clean":"2021-02-10T04:59:57.780283+0000","last_became_active":"2021-02-10T04:59:53.658358+0000","last_became_peered":"2021-02-10T04:59:53.658358+0000","last_unstale":"2021-02-10T04:59:57.780283+0000","last_undegraded":"2021-02-10T04:59:57.780283+0000","last_fullsized":"2021-02-10T04:59:57.780283+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29601845,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":286,"num_read_kb":299529,"num_write":200,"num_write_kb":89130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":50371563,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"146","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.780256+0000","last_change":"2021-02-10T04:59:08.658203+0000","last_active":"2021-02-10T04:59:57.780256+0000","last_peered":"2021-02-10T04:59:57.780256+0000","last_clean":"2021-02-10T04:59:57.780256+0000","last_became_active":"2021-02-10T04:58:56.956556+0000","last_became_peered":"2021-02-10T04:58:56.956556+0000","last_unstale":"2021-02-10T04:59:57.780256+0000","last_undegraded":"2021-02-10T04:59:57.780256+0000","last_fullsized":"2021-02-10T04:59:57.780256+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"1113'5","reported_seq":"1657","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.682007+0000","last_change":"2021-02-10T04:52:34.047474+0000","last_active":"2021-02-10T04:59:58.682007+0000","last_peered":"2021-02-10T04:59:58.682007+0000","last_clean":"2021-02-10T04:59:58.682007+0000","last_became_active":"2021-02-10T04:50:59.569578+0000","last_became_peered":"2021-02-10T04:50:59.569578+0000","last_unstale":"2021-02-10T04:59:58.682007+0000","last_undegraded":"2021-02-10T04:59:58.682007+0000","last_fullsized":"2021-02-10T04:59:58.682007+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1113'5","last_scrub_stamp":"2021-02-10T04:52:34.047358+0000","last_deep_scrub":"158'3","last_deep_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_clean_scrub_stamp":"2021-02-10T04:52:34.047358+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"1261'68","reported_seq":"1494","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.687530+0000","last_change":"2021-02-10T04:59:51.162320+0000","last_active":"2021-02-10T04:59:58.687530+0000","last_peered":"2021-02-10T04:59:58.687530+0000","last_clean":"2021-02-10T04:59:58.687530+0000","last_became_active":"2021-02-10T04:59:51.111783+0000","last_became_peered":"2021-02-10T04:59:51.111783+0000","last_unstale":"2021-02-10T04:59:58.687530+0000","last_undegraded":"2021-02-10T04:59:58.687530+0000","last_fullsized":"2021-02-10T04:59:58.687530+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:51:48.872839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.872839+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":114,"num_read_kb":94,"num_write":68,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"1176","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.688537+0000","last_change":"2021-02-10T04:59:51.126437+0000","last_active":"2021-02-10T04:59:58.688537+0000","last_peered":"2021-02-10T04:59:58.688537+0000","last_clean":"2021-02-10T04:59:58.688537+0000","last_became_active":"2021-02-10T04:59:51.125256+0000","last_became_peered":"2021-02-10T04:59:51.125256+0000","last_unstale":"2021-02-10T04:59:58.688537+0000","last_undegraded":"2021-02-10T04:59:58.688537+0000","last_fullsized":"2021-02-10T04:59:58.688537+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:05.984659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:05.984659+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"1234","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.779925+0000","last_change":"2021-02-10T04:59:51.130527+0000","last_active":"2021-02-10T04:59:57.779925+0000","last_peered":"2021-02-10T04:59:57.779925+0000","last_clean":"2021-02-10T04:59:57.779925+0000","last_became_active":"2021-02-10T04:59:51.130127+0000","last_became_peered":"2021-02-10T04:59:51.130127+0000","last_unstale":"2021-02-10T04:59:57.779925+0000","last_undegraded":"2021-02-10T04:59:57.779925+0000","last_fullsized":"2021-02-10T04:59:57.779925+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:09.005421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.005421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"998'1009","reported_seq":"4863","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.683079+0000","last_change":"2021-02-10T04:52:26.037227+0000","last_active":"2021-02-10T04:59:58.683079+0000","last_peered":"2021-02-10T04:59:58.683079+0000","last_clean":"2021-02-10T04:59:58.683079+0000","last_became_active":"2021-02-10T04:52:26.036723+0000","last_became_peered":"2021-02-10T04:52:26.036723+0000","last_unstale":"2021-02-10T04:59:58.683079+0000","last_undegraded":"2021-02-10T04:59:58.683079+0000","last_fullsized":"2021-02-10T04:59:58.683079+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12355206,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":119,"num_read_kb":124701,"num_write":77,"num_write_kb":25052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4145029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"1243","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.684507+0000","last_change":"2021-02-10T04:53:33.305896+0000","last_active":"2021-02-10T04:59:58.684507+0000","last_peered":"2021-02-10T04:59:58.684507+0000","last_clean":"2021-02-10T04:59:58.684507+0000","last_became_active":"2021-02-10T04:50:59.570854+0000","last_became_peered":"2021-02-10T04:50:59.570854+0000","last_unstale":"2021-02-10T04:59:58.684507+0000","last_undegraded":"2021-02-10T04:59:58.684507+0000","last_fullsized":"2021-02-10T04:59:58.684507+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:33.305827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:33.305827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"1359","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.684183+0000","last_change":"2021-02-10T04:58:56.979110+0000","last_active":"2021-02-10T04:59:58.684183+0000","last_peered":"2021-02-10T04:59:58.684183+0000","last_clean":"2021-02-10T04:59:58.684183+0000","last_became_active":"2021-02-10T04:58:56.978745+0000","last_became_peered":"2021-02-10T04:58:56.978745+0000","last_unstale":"2021-02-10T04:59:58.684183+0000","last_undegraded":"2021-02-10T04:59:58.684183+0000","last_fullsized":"2021-02-10T04:59:58.684183+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"1357","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.687873+0000","last_change":"2021-02-10T04:51:50.895471+0000","last_active":"2021-02-10T04:59:58.687873+0000","last_peered":"2021-02-10T04:59:58.687873+0000","last_clean":"2021-02-10T04:59:58.687873+0000","last_became_active":"2021-02-10T04:50:59.527553+0000","last_became_peered":"2021-02-10T04:50:59.527553+0000","last_unstale":"2021-02-10T04:59:58.687873+0000","last_undegraded":"2021-02-10T04:59:58.687873+0000","last_fullsized":"2021-02-10T04:59:58.687873+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.895370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.895370+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"1261'84","reported_seq":"1503","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.682417+0000","last_change":"2021-02-10T04:58:56.880156+0000","last_active":"2021-02-10T04:59:58.682417+0000","last_peered":"2021-02-10T04:59:58.682417+0000","last_clean":"2021-02-10T04:59:58.682417+0000","last_became_active":"2021-02-10T04:58:56.879837+0000","last_became_peered":"2021-02-10T04:58:56.879837+0000","last_unstale":"2021-02-10T04:59:58.682417+0000","last_undegraded":"2021-02-10T04:59:58.682417+0000","last_fullsized":"2021-02-10T04:59:58.682417+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'72","last_scrub_stamp":"2021-02-10T04:46:30.515026+0000","last_deep_scrub":"152'48","last_deep_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_clean_scrub_stamp":"2021-02-10T04:46:30.515026+0000","log_size":84,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":123,"num_read_kb":101,"num_write":72,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"38","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.689047+0000","last_change":"2021-02-10T04:59:51.275465+0000","last_active":"2021-02-10T04:59:58.689047+0000","last_peered":"2021-02-10T04:59:58.689047+0000","last_clean":"2021-02-10T04:59:58.689047+0000","last_became_active":"2021-02-10T04:59:51.125942+0000","last_became_peered":"2021-02-10T04:59:51.125942+0000","last_unstale":"2021-02-10T04:59:58.689047+0000","last_undegraded":"2021-02-10T04:59:58.689047+0000","last_fullsized":"2021-02-10T04:59:58.689047+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:51:48.274960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.274960+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":423,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":846,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"1382","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.687940+0000","last_change":"2021-02-10T04:53:13.099288+0000","last_active":"2021-02-10T04:59:58.687940+0000","last_peered":"2021-02-10T04:59:58.687940+0000","last_clean":"2021-02-10T04:59:58.687940+0000","last_became_active":"2021-02-10T04:51:36.848614+0000","last_became_peered":"2021-02-10T04:51:36.848614+0000","last_unstale":"2021-02-10T04:59:58.687940+0000","last_undegraded":"2021-02-10T04:59:58.687940+0000","last_fullsized":"2021-02-10T04:59:58.687940+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:13.099228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:13.099228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"1002'1657","reported_seq":"5433","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.682604+0000","last_change":"2021-02-10T04:52:26.036806+0000","last_active":"2021-02-10T04:59:58.682604+0000","last_peered":"2021-02-10T04:59:58.682604+0000","last_clean":"2021-02-10T04:59:58.682604+0000","last_became_active":"2021-02-10T04:52:26.032363+0000","last_became_peered":"2021-02-10T04:52:26.032363+0000","last_unstale":"2021-02-10T04:59:58.682604+0000","last_undegraded":"2021-02-10T04:59:58.682604+0000","last_fullsized":"2021-02-10T04:59:58.682604+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21396533,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":212,"num_read_kb":218836,"num_write":159,"num_write_kb":59398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":22326014,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"1310","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.688399+0000","last_change":"2021-02-10T04:59:51.112802+0000","last_active":"2021-02-10T04:59:58.688399+0000","last_peered":"2021-02-10T04:59:58.688399+0000","last_clean":"2021-02-10T04:59:58.688399+0000","last_became_active":"2021-02-10T04:59:51.112487+0000","last_became_peered":"2021-02-10T04:59:51.112487+0000","last_unstale":"2021-02-10T04:59:58.688399+0000","last_undegraded":"2021-02-10T04:59:58.688399+0000","last_fullsized":"2021-02-10T04:59:58.688399+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.911851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.911851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"62'5","reported_seq":"62","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.689629+0000","last_change":"2021-02-10T04:59:40.662441+0000","last_active":"2021-02-10T04:59:58.689629+0000","last_peered":"2021-02-10T04:59:58.689629+0000","last_clean":"2021-02-10T04:59:58.689629+0000","last_became_active":"2021-02-10T04:59:40.661336+0000","last_became_peered":"2021-02-10T04:59:40.661336+0000","last_unstale":"2021-02-10T04:59:58.689629+0000","last_undegraded":"2021-02-10T04:59:58.689629+0000","last_fullsized":"2021-02-10T04:59:58.689629+0000","mapping_epoch":1356,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1357,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":98,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":98,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"1416","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.779983+0000","last_change":"2021-02-10T04:59:51.117511+0000","last_active":"2021-02-10T04:59:57.779983+0000","last_peered":"2021-02-10T04:59:57.779983+0000","last_clean":"2021-02-10T04:59:57.779983+0000","last_became_active":"2021-02-10T04:59:51.117148+0000","last_became_peered":"2021-02-10T04:59:51.117148+0000","last_unstale":"2021-02-10T04:59:57.779983+0000","last_undegraded":"2021-02-10T04:59:57.779983+0000","last_fullsized":"2021-02-10T04:59:57.779983+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:58.142526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:58.142526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"1261'68","reported_seq":"1364","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.780215+0000","last_change":"2021-02-10T04:59:08.646165+0000","last_active":"2021-02-10T04:59:57.780215+0000","last_peered":"2021-02-10T04:59:57.780215+0000","last_clean":"2021-02-10T04:59:57.780215+0000","last_became_active":"2021-02-10T04:58:56.954019+0000","last_became_peered":"2021-02-10T04:58:56.954019+0000","last_unstale":"2021-02-10T04:59:57.780215+0000","last_undegraded":"2021-02-10T04:59:57.780215+0000","last_fullsized":"2021-02-10T04:59:57.780215+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1135'58","last_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_deep_scrub":"1135'58","last_deep_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.483067+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":101,"num_read_kb":82,"num_write":60,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"339","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.780531+0000","last_change":"2021-02-10T04:59:55.726805+0000","last_active":"2021-02-10T04:59:57.780531+0000","last_peered":"2021-02-10T04:59:57.780531+0000","last_clean":"2021-02-10T04:59:57.780531+0000","last_became_active":"2021-02-10T04:58:56.849868+0000","last_became_peered":"2021-02-10T04:58:56.849868+0000","last_unstale":"2021-02-10T04:59:57.780531+0000","last_undegraded":"2021-02-10T04:59:57.780531+0000","last_fullsized":"2021-02-10T04:59:57.780531+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:50:46.434629+0000","last_deep_scrub":"53'2","last_deep_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_clean_scrub_stamp":"2021-02-10T04:50:46.434629+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"832","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.688841+0000","last_change":"2021-02-10T04:59:51.143488+0000","last_active":"2021-02-10T04:59:58.688841+0000","last_peered":"2021-02-10T04:59:58.688841+0000","last_clean":"2021-02-10T04:59:58.688841+0000","last_became_active":"2021-02-10T04:59:51.143081+0000","last_became_peered":"2021-02-10T04:59:51.143081+0000","last_unstale":"2021-02-10T04:59:58.688841+0000","last_undegraded":"2021-02-10T04:59:58.688841+0000","last_fullsized":"2021-02-10T04:59:58.688841+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:38.278910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:52:38.278910+0000","last_clean_scrub_stamp":"2021-02-10T04:52:38.278910+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.0","version":"1000'1071","reported_seq":"4924","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.702168+0000","last_change":"2021-02-10T04:53:34.407875+0000","last_active":"2021-02-10T05:00:00.702168+0000","last_peered":"2021-02-10T05:00:00.702168+0000","last_clean":"2021-02-10T05:00:00.702168+0000","last_became_active":"2021-02-10T04:52:26.031050+0000","last_became_peered":"2021-02-10T04:52:26.031050+0000","last_unstale":"2021-02-10T05:00:00.702168+0000","last_undegraded":"2021-02-10T05:00:00.702168+0000","last_fullsized":"2021-02-10T05:00:00.702168+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:34.407784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.407784+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30173843,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"1382","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.780397+0000","last_change":"2021-02-10T04:59:51.120656+0000","last_active":"2021-02-10T04:59:57.780397+0000","last_peered":"2021-02-10T04:59:57.780397+0000","last_clean":"2021-02-10T04:59:57.780397+0000","last_became_active":"2021-02-10T04:59:51.120335+0000","last_became_peered":"2021-02-10T04:59:51.120335+0000","last_unstale":"2021-02-10T04:59:57.780397+0000","last_undegraded":"2021-02-10T04:59:57.780397+0000","last_fullsized":"2021-02-10T04:59:57.780397+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_clean_scrub_stamp":"2021-02-10T04:51:00.195518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"1380","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.705781+0000","last_change":"2021-02-10T04:58:58.252055+0000","last_active":"2021-02-10T05:00:00.705781+0000","last_peered":"2021-02-10T05:00:00.705781+0000","last_clean":"2021-02-10T05:00:00.705781+0000","last_became_active":"2021-02-10T04:58:56.854414+0000","last_became_peered":"2021-02-10T04:58:56.854414+0000","last_unstale":"2021-02-10T05:00:00.705781+0000","last_undegraded":"2021-02-10T05:00:00.705781+0000","last_fullsized":"2021-02-10T05:00:00.705781+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:58:58.251969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:58:58.251969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"1261'58","reported_seq":"1425","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.780196+0000","last_change":"2021-02-10T04:59:13.927049+0000","last_active":"2021-02-10T04:59:57.780196+0000","last_peered":"2021-02-10T04:59:57.780196+0000","last_clean":"2021-02-10T04:59:57.780196+0000","last_became_active":"2021-02-10T04:59:13.926626+0000","last_became_peered":"2021-02-10T04:59:13.926626+0000","last_unstale":"2021-02-10T04:59:57.780196+0000","last_undegraded":"2021-02-10T04:59:57.780196+0000","last_fullsized":"2021-02-10T04:59:57.780196+0000","mapping_epoch":1332,"log_start":"1049'40","ondisk_log_start":"1049'40","created":55,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'40","last_scrub_stamp":"2021-02-10T04:51:47.281929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.281929+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":918,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":85,"num_read_kb":67,"num_write":49,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"547","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.779837+0000","last_change":"2021-02-10T04:58:56.841926+0000","last_active":"2021-02-10T04:59:57.779837+0000","last_peered":"2021-02-10T04:59:57.779837+0000","last_clean":"2021-02-10T04:59:57.779837+0000","last_became_active":"2021-02-10T04:58:56.840576+0000","last_became_peered":"2021-02-10T04:58:56.840576+0000","last_unstale":"2021-02-10T04:59:57.779837+0000","last_undegraded":"2021-02-10T04:59:57.779837+0000","last_fullsized":"2021-02-10T04:59:57.779837+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:50.399350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:50.399350+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"1250","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.778680+0000","last_change":"2021-02-10T04:59:08.647800+0000","last_active":"2021-02-10T04:59:57.778680+0000","last_peered":"2021-02-10T04:59:57.778680+0000","last_clean":"2021-02-10T04:59:57.778680+0000","last_became_active":"2021-02-10T04:58:56.875409+0000","last_became_peered":"2021-02-10T04:58:56.875409+0000","last_unstale":"2021-02-10T04:59:57.778680+0000","last_undegraded":"2021-02-10T04:59:57.778680+0000","last_fullsized":"2021-02-10T04:59:57.778680+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"829","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.706900+0000","last_change":"2021-02-10T04:59:40.661283+0000","last_active":"2021-02-10T05:00:00.706900+0000","last_peered":"2021-02-10T05:00:00.706900+0000","last_clean":"2021-02-10T05:00:00.706900+0000","last_became_active":"2021-02-10T04:59:40.660922+0000","last_became_peered":"2021-02-10T04:59:40.660922+0000","last_unstale":"2021-02-10T05:00:00.706900+0000","last_undegraded":"2021-02-10T05:00:00.706900+0000","last_fullsized":"2021-02-10T05:00:00.706900+0000","mapping_epoch":1356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:31.272971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:17.598113+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.272971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.5","version":"1000'446","reported_seq":"2104","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.780250+0000","last_change":"2021-02-10T04:58:56.846600+0000","last_active":"2021-02-10T04:59:57.780250+0000","last_peered":"2021-02-10T04:59:57.780250+0000","last_clean":"2021-02-10T04:59:57.780250+0000","last_became_active":"2021-02-10T04:58:56.846008+0000","last_became_peered":"2021-02-10T04:58:56.846008+0000","last_unstale":"2021-02-10T04:59:57.780250+0000","last_undegraded":"2021-02-10T04:59:57.780250+0000","last_fullsized":"2021-02-10T04:59:57.780250+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:51:45.964029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.964029+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22255852,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68435,"num_write":72,"num_write_kb":24345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":403693,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"884","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.687810+0000","last_change":"2021-02-10T04:59:51.123025+0000","last_active":"2021-02-10T04:59:58.687810+0000","last_peered":"2021-02-10T04:59:58.687810+0000","last_clean":"2021-02-10T04:59:58.687810+0000","last_became_active":"2021-02-10T04:59:51.122151+0000","last_became_peered":"2021-02-10T04:59:51.122151+0000","last_unstale":"2021-02-10T04:59:58.687810+0000","last_undegraded":"2021-02-10T04:59:58.687810+0000","last_fullsized":"2021-02-10T04:59:58.687810+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:03.228182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.228182+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"276","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.780544+0000","last_change":"2021-02-10T04:53:46.727384+0000","last_active":"2021-02-10T04:59:57.780544+0000","last_peered":"2021-02-10T04:59:57.780544+0000","last_clean":"2021-02-10T04:59:57.780544+0000","last_became_active":"2021-02-10T04:53:46.726771+0000","last_became_peered":"2021-02-10T04:53:46.726771+0000","last_unstale":"2021-02-10T04:59:57.780544+0000","last_undegraded":"2021-02-10T04:59:57.780544+0000","last_fullsized":"2021-02-10T04:59:57.780544+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:33.948348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:33.948348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"1261'51","reported_seq":"1385","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.690058+0000","last_change":"2021-02-10T04:59:51.674648+0000","last_active":"2021-02-10T04:59:58.690058+0000","last_peered":"2021-02-10T04:59:58.690058+0000","last_clean":"2021-02-10T04:59:58.690058+0000","last_became_active":"2021-02-10T04:59:51.118282+0000","last_became_peered":"2021-02-10T04:59:51.118282+0000","last_unstale":"2021-02-10T04:59:58.690058+0000","last_undegraded":"2021-02-10T04:59:58.690058+0000","last_fullsized":"2021-02-10T04:59:58.690058+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'39","last_scrub_stamp":"2021-02-10T04:51:47.322331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.322331+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":68,"num_write":51,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":932,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"328","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.779747+0000","last_change":"2021-02-10T04:59:51.117366+0000","last_active":"2021-02-10T04:59:57.779747+0000","last_peered":"2021-02-10T04:59:57.779747+0000","last_clean":"2021-02-10T04:59:57.779747+0000","last_became_active":"2021-02-10T04:59:51.116190+0000","last_became_peered":"2021-02-10T04:59:51.116190+0000","last_unstale":"2021-02-10T04:59:57.779747+0000","last_undegraded":"2021-02-10T04:59:57.779747+0000","last_fullsized":"2021-02-10T04:59:57.779747+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_clean_scrub_stamp":"2021-02-10T04:50:05.424262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"1544","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.706101+0000","last_change":"2021-02-10T04:52:37.324122+0000","last_active":"2021-02-10T05:00:00.706101+0000","last_peered":"2021-02-10T05:00:00.706101+0000","last_clean":"2021-02-10T05:00:00.706101+0000","last_became_active":"2021-02-10T04:51:36.843553+0000","last_became_peered":"2021-02-10T04:51:36.843553+0000","last_unstale":"2021-02-10T05:00:00.706101+0000","last_undegraded":"2021-02-10T05:00:00.706101+0000","last_fullsized":"2021-02-10T05:00:00.706101+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:37.324040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:37.324040+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"1358","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.683157+0000","last_change":"2021-02-10T04:59:51.114576+0000","last_active":"2021-02-10T04:59:58.683157+0000","last_peered":"2021-02-10T04:59:58.683157+0000","last_clean":"2021-02-10T04:59:58.683157+0000","last_became_active":"2021-02-10T04:59:51.114177+0000","last_became_peered":"2021-02-10T04:59:51.114177+0000","last_unstale":"2021-02-10T04:59:58.683157+0000","last_undegraded":"2021-02-10T04:59:58.683157+0000","last_fullsized":"2021-02-10T04:59:58.683157+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"1372","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.705859+0000","last_change":"2021-02-10T04:52:31.403904+0000","last_active":"2021-02-10T05:00:00.705859+0000","last_peered":"2021-02-10T05:00:00.705859+0000","last_clean":"2021-02-10T05:00:00.705859+0000","last_became_active":"2021-02-10T04:50:59.538009+0000","last_became_peered":"2021-02-10T04:50:59.538009+0000","last_unstale":"2021-02-10T05:00:00.705859+0000","last_undegraded":"2021-02-10T05:00:00.705859+0000","last_fullsized":"2021-02-10T05:00:00.705859+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:31.403824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.403824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"1000'490","reported_seq":"3830","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.780453+0000","last_change":"2021-02-10T04:59:08.651746+0000","last_active":"2021-02-10T04:59:57.780453+0000","last_peered":"2021-02-10T04:59:57.780453+0000","last_clean":"2021-02-10T04:59:57.780453+0000","last_became_active":"2021-02-10T04:58:56.962979+0000","last_became_peered":"2021-02-10T04:58:56.962979+0000","last_unstale":"2021-02-10T04:59:57.780453+0000","last_undegraded":"2021-02-10T04:59:57.780453+0000","last_fullsized":"2021-02-10T04:59:57.780453+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:47:02.875920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:02.875920+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19371627,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":141,"num_read_kb":148568,"num_write":85,"num_write_kb":31732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"278","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.690299+0000","last_change":"2021-02-10T04:59:51.117341+0000","last_active":"2021-02-10T04:59:58.690299+0000","last_peered":"2021-02-10T04:59:58.690299+0000","last_clean":"2021-02-10T04:59:58.690299+0000","last_became_active":"2021-02-10T04:59:51.116854+0000","last_became_peered":"2021-02-10T04:59:51.116854+0000","last_unstale":"2021-02-10T04:59:58.690299+0000","last_undegraded":"2021-02-10T04:59:58.690299+0000","last_fullsized":"2021-02-10T04:59:58.690299+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:01.972913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.972913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"1261'40","reported_seq":"1116","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:59.694058+0000","last_change":"2021-02-10T04:59:13.926580+0000","last_active":"2021-02-10T04:59:59.694058+0000","last_peered":"2021-02-10T04:59:59.694058+0000","last_clean":"2021-02-10T04:59:59.694058+0000","last_became_active":"2021-02-10T04:59:13.926146+0000","last_became_peered":"2021-02-10T04:59:13.926146+0000","last_unstale":"2021-02-10T04:59:59.694058+0000","last_undegraded":"2021-02-10T04:59:59.694058+0000","last_fullsized":"2021-02-10T04:59:59.694058+0000","mapping_epoch":1332,"log_start":"1063'30","ondisk_log_start":"1063'30","created":55,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:56.431187+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.431187+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":401,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":58,"num_read_kb":47,"num_write":33,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"294","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.688985+0000","last_change":"2021-02-10T04:58:56.879863+0000","last_active":"2021-02-10T04:59:58.688985+0000","last_peered":"2021-02-10T04:59:58.688985+0000","last_clean":"2021-02-10T04:59:58.688985+0000","last_became_active":"2021-02-10T04:58:56.879496+0000","last_became_peered":"2021-02-10T04:58:56.879496+0000","last_unstale":"2021-02-10T04:59:58.688985+0000","last_undegraded":"2021-02-10T04:59:58.688985+0000","last_fullsized":"2021-02-10T04:59:58.688985+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:04.591695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:04.591695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"71","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.688857+0000","last_change":"2021-02-10T04:59:51.143398+0000","last_active":"2021-02-10T04:59:58.688857+0000","last_peered":"2021-02-10T04:59:58.688857+0000","last_clean":"2021-02-10T04:59:58.688857+0000","last_became_active":"2021-02-10T04:59:51.142973+0000","last_became_peered":"2021-02-10T04:59:51.142973+0000","last_unstale":"2021-02-10T04:59:58.688857+0000","last_undegraded":"2021-02-10T04:59:58.688857+0000","last_fullsized":"2021-02-10T04:59:58.688857+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":97,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"1224","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.780440+0000","last_change":"2021-02-10T04:59:51.130891+0000","last_active":"2021-02-10T04:59:57.780440+0000","last_peered":"2021-02-10T04:59:57.780440+0000","last_clean":"2021-02-10T04:59:57.780440+0000","last_became_active":"2021-02-10T04:59:51.130334+0000","last_became_peered":"2021-02-10T04:59:51.130334+0000","last_unstale":"2021-02-10T04:59:57.780440+0000","last_undegraded":"2021-02-10T04:59:57.780440+0000","last_fullsized":"2021-02-10T04:59:57.780440+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:40.066354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:52:40.066354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"1000'620","reported_seq":"4454","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.780276+0000","last_change":"2021-02-10T04:59:08.651277+0000","last_active":"2021-02-10T04:59:57.780276+0000","last_peered":"2021-02-10T04:59:57.780276+0000","last_clean":"2021-02-10T04:59:57.780276+0000","last_became_active":"2021-02-10T04:58:56.878577+0000","last_became_peered":"2021-02-10T04:58:56.878577+0000","last_unstale":"2021-02-10T04:59:57.780276+0000","last_undegraded":"2021-02-10T04:59:57.780276+0000","last_fullsized":"2021-02-10T04:59:57.780276+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:51:54.444287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:54.444287+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14265195,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":144,"num_read_kb":128482,"num_write":129,"num_write_kb":53443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"1243","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.779797+0000","last_change":"2021-02-10T04:59:08.653244+0000","last_active":"2021-02-10T04:59:57.779797+0000","last_peered":"2021-02-10T04:59:57.779797+0000","last_clean":"2021-02-10T04:59:57.779797+0000","last_became_active":"2021-02-10T04:58:56.842941+0000","last_became_peered":"2021-02-10T04:58:56.842941+0000","last_unstale":"2021-02-10T04:59:57.779797+0000","last_undegraded":"2021-02-10T04:59:57.779797+0000","last_fullsized":"2021-02-10T04:59:57.779797+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:14.955544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:47:14.955544+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"1373","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.687888+0000","last_change":"2021-02-10T04:59:51.123202+0000","last_active":"2021-02-10T04:59:58.687888+0000","last_peered":"2021-02-10T04:59:58.687888+0000","last_clean":"2021-02-10T04:59:58.687888+0000","last_became_active":"2021-02-10T04:59:51.122639+0000","last_became_peered":"2021-02-10T04:59:51.122639+0000","last_unstale":"2021-02-10T04:59:58.687888+0000","last_undegraded":"2021-02-10T04:59:58.687888+0000","last_fullsized":"2021-02-10T04:59:58.687888+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"1346","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.780145+0000","last_change":"2021-02-10T04:59:08.656087+0000","last_active":"2021-02-10T04:59:57.780145+0000","last_peered":"2021-02-10T04:59:57.780145+0000","last_clean":"2021-02-10T04:59:57.780145+0000","last_became_active":"2021-02-10T04:58:56.957166+0000","last_became_peered":"2021-02-10T04:58:56.957166+0000","last_unstale":"2021-02-10T04:59:57.780145+0000","last_undegraded":"2021-02-10T04:59:57.780145+0000","last_fullsized":"2021-02-10T04:59:57.780145+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:27.908221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:47:27.908221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"1261'67","reported_seq":"937","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.689767+0000","last_change":"2021-02-10T04:59:40.661729+0000","last_active":"2021-02-10T04:59:58.689767+0000","last_peered":"2021-02-10T04:59:58.689767+0000","last_clean":"2021-02-10T04:59:58.689767+0000","last_became_active":"2021-02-10T04:59:40.661402+0000","last_became_peered":"2021-02-10T04:59:40.661402+0000","last_unstale":"2021-02-10T04:59:58.689767+0000","last_undegraded":"2021-02-10T04:59:58.689767+0000","last_fullsized":"2021-02-10T04:59:58.689767+0000","mapping_epoch":1356,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1357,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'55","last_scrub_stamp":"2021-02-10T04:52:30.276938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.276938+0000","log_size":67,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":115,"num_read_kb":91,"num_write":70,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"1377","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.707555+0000","last_change":"2021-02-10T04:59:51.119728+0000","last_active":"2021-02-10T05:00:00.707555+0000","last_peered":"2021-02-10T05:00:00.707555+0000","last_clean":"2021-02-10T05:00:00.707555+0000","last_became_active":"2021-02-10T04:59:51.119372+0000","last_became_peered":"2021-02-10T04:59:51.119372+0000","last_unstale":"2021-02-10T05:00:00.707555+0000","last_undegraded":"2021-02-10T05:00:00.707555+0000","last_fullsized":"2021-02-10T05:00:00.707555+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.420086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.420086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"1333","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.683733+0000","last_change":"2021-02-10T04:53:12.187225+0000","last_active":"2021-02-10T04:59:58.683733+0000","last_peered":"2021-02-10T04:59:58.683733+0000","last_clean":"2021-02-10T04:59:58.683733+0000","last_became_active":"2021-02-10T04:50:59.555689+0000","last_became_peered":"2021-02-10T04:50:59.555689+0000","last_unstale":"2021-02-10T04:59:58.683733+0000","last_undegraded":"2021-02-10T04:59:58.683733+0000","last_fullsized":"2021-02-10T04:59:58.683733+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.187154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.187154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"1002'2042","reported_seq":"9525","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.780900+0000","last_change":"2021-02-10T04:59:53.663243+0000","last_active":"2021-02-10T04:59:57.780900+0000","last_peered":"2021-02-10T04:59:57.780900+0000","last_clean":"2021-02-10T04:59:57.780900+0000","last_became_active":"2021-02-10T04:59:53.660539+0000","last_became_peered":"2021-02-10T04:59:53.660539+0000","last_unstale":"2021-02-10T04:59:57.780900+0000","last_undegraded":"2021-02-10T04:59:57.780900+0000","last_fullsized":"2021-02-10T04:59:57.780900+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:46:57.054230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:57.054230+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46300076,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":545,"num_read_kb":531689,"num_write":399,"num_write_kb":153087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":66804807,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"1334","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.703215+0000","last_change":"2021-02-10T04:58:56.856707+0000","last_active":"2021-02-10T05:00:00.703215+0000","last_peered":"2021-02-10T05:00:00.703215+0000","last_clean":"2021-02-10T05:00:00.703215+0000","last_became_active":"2021-02-10T04:58:56.856323+0000","last_became_peered":"2021-02-10T04:58:56.856323+0000","last_unstale":"2021-02-10T05:00:00.703215+0000","last_undegraded":"2021-02-10T05:00:00.703215+0000","last_fullsized":"2021-02-10T05:00:00.703215+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:52.341527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:50:52.341527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"172","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.706304+0000","last_change":"2021-02-10T04:54:39.558380+0000","last_active":"2021-02-10T05:00:00.706304+0000","last_peered":"2021-02-10T05:00:00.706304+0000","last_clean":"2021-02-10T05:00:00.706304+0000","last_became_active":"2021-02-10T04:54:39.557995+0000","last_became_peered":"2021-02-10T04:54:39.557995+0000","last_unstale":"2021-02-10T05:00:00.706304+0000","last_undegraded":"2021-02-10T05:00:00.706304+0000","last_fullsized":"2021-02-10T05:00:00.706304+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"1366'9","reported_seq":"1551","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:58.087175+0000","last_change":"2021-02-10T04:59:51.169990+0000","last_active":"2021-02-10T04:59:58.087175+0000","last_peered":"2021-02-10T04:59:58.087175+0000","last_clean":"2021-02-10T04:59:58.087175+0000","last_became_active":"2021-02-10T04:59:51.117900+0000","last_became_peered":"2021-02-10T04:59:51.117900+0000","last_unstale":"2021-02-10T04:59:58.087175+0000","last_undegraded":"2021-02-10T04:59:58.087175+0000","last_fullsized":"2021-02-10T04:59:58.087175+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:54.466259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:54.466259+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"1261'64","reported_seq":"984","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.707711+0000","last_change":"2021-02-10T04:59:40.660545+0000","last_active":"2021-02-10T05:00:00.707711+0000","last_peered":"2021-02-10T05:00:00.707711+0000","last_clean":"2021-02-10T05:00:00.707711+0000","last_became_active":"2021-02-10T04:59:40.660072+0000","last_became_peered":"2021-02-10T04:59:40.660072+0000","last_unstale":"2021-02-10T05:00:00.707711+0000","last_undegraded":"2021-02-10T05:00:00.707711+0000","last_fullsized":"2021-02-10T05:00:00.707711+0000","mapping_epoch":1356,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1357,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:53:01.304626+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.304626+0000","log_size":64,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":88,"num_read_kb":70,"num_write":52,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"889","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.780125+0000","last_change":"2021-02-10T04:59:40.663233+0000","last_active":"2021-02-10T04:59:57.780125+0000","last_peered":"2021-02-10T04:59:57.780125+0000","last_clean":"2021-02-10T04:59:57.780125+0000","last_became_active":"2021-02-10T04:59:40.662896+0000","last_became_peered":"2021-02-10T04:59:40.662896+0000","last_unstale":"2021-02-10T04:59:57.780125+0000","last_undegraded":"2021-02-10T04:59:57.780125+0000","last_fullsized":"2021-02-10T04:59:57.780125+0000","mapping_epoch":1356,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:49.447961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:49.447961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"1361","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.778604+0000","last_change":"2021-02-10T04:59:56.057363+0000","last_active":"2021-02-10T04:59:57.778604+0000","last_peered":"2021-02-10T04:59:57.778604+0000","last_clean":"2021-02-10T04:59:57.778604+0000","last_became_active":"2021-02-10T04:58:56.874543+0000","last_became_peered":"2021-02-10T04:58:56.874543+0000","last_unstale":"2021-02-10T04:59:57.778604+0000","last_undegraded":"2021-02-10T04:59:57.778604+0000","last_fullsized":"2021-02-10T04:59:57.778604+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:59:56.057291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:59:56.057291+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"1371'83","reported_seq":"3187","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T05:00:00.403522+0000","last_change":"2021-02-10T05:00:00.403522+0000","last_active":"2021-02-10T05:00:00.403522+0000","last_peered":"2021-02-10T05:00:00.403522+0000","last_clean":"2021-02-10T05:00:00.403522+0000","last_became_active":"2021-02-10T04:59:51.124908+0000","last_became_peered":"2021-02-10T04:59:51.124908+0000","last_unstale":"2021-02-10T05:00:00.403522+0000","last_undegraded":"2021-02-10T05:00:00.403522+0000","last_fullsized":"2021-02-10T05:00:00.403522+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":83,"ondisk_log_size":83,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":962233,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":18587,"num_write":10,"num_write_kb":5384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3042034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"1132","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.684401+0000","last_change":"2021-02-10T04:59:51.122465+0000","last_active":"2021-02-10T04:59:58.684401+0000","last_peered":"2021-02-10T04:59:58.684401+0000","last_clean":"2021-02-10T04:59:58.684401+0000","last_became_active":"2021-02-10T04:59:51.122127+0000","last_became_peered":"2021-02-10T04:59:51.122127+0000","last_unstale":"2021-02-10T04:59:58.684401+0000","last_undegraded":"2021-02-10T04:59:58.684401+0000","last_fullsized":"2021-02-10T04:59:58.684401+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.024980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.024980+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"369","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:59.241166+0000","last_change":"2021-02-10T04:59:59.241100+0000","last_active":"2021-02-10T04:59:59.241166+0000","last_peered":"2021-02-10T04:59:59.241166+0000","last_clean":"2021-02-10T04:59:59.241166+0000","last_became_active":"2021-02-10T04:58:56.850742+0000","last_became_peered":"2021-02-10T04:58:56.850742+0000","last_unstale":"2021-02-10T04:59:59.241166+0000","last_undegraded":"2021-02-10T04:59:59.241166+0000","last_fullsized":"2021-02-10T04:59:59.241166+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:59:59.241025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:59:59.241025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"1261'86","reported_seq":"1563","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.687605+0000","last_change":"2021-02-10T04:59:13.931273+0000","last_active":"2021-02-10T04:59:58.687605+0000","last_peered":"2021-02-10T04:59:58.687605+0000","last_clean":"2021-02-10T04:59:58.687605+0000","last_became_active":"2021-02-10T04:59:13.930845+0000","last_became_peered":"2021-02-10T04:59:13.930845+0000","last_unstale":"2021-02-10T04:59:58.687605+0000","last_undegraded":"2021-02-10T04:59:58.687605+0000","last_fullsized":"2021-02-10T04:59:58.687605+0000","mapping_epoch":1238,"log_start":"1049'68","ondisk_log_start":"1049'68","created":55,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'39","last_scrub_stamp":"2021-02-10T04:53:38.085565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.085565+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":791,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":113,"num_read_kb":93,"num_write":64,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1582,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"1508","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.689599+0000","last_change":"2021-02-10T04:59:56.274760+0000","last_active":"2021-02-10T04:59:58.689599+0000","last_peered":"2021-02-10T04:59:58.689599+0000","last_clean":"2021-02-10T04:59:58.689599+0000","last_became_active":"2021-02-10T04:58:56.843963+0000","last_became_peered":"2021-02-10T04:58:56.843963+0000","last_unstale":"2021-02-10T04:59:58.689599+0000","last_undegraded":"2021-02-10T04:59:58.689599+0000","last_fullsized":"2021-02-10T04:59:58.689599+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:59:56.274662+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:59:56.274662+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"1335","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.683048+0000","last_change":"2021-02-10T04:52:30.019225+0000","last_active":"2021-02-10T04:59:58.683048+0000","last_peered":"2021-02-10T04:59:58.683048+0000","last_clean":"2021-02-10T04:59:58.683048+0000","last_became_active":"2021-02-10T04:50:59.561436+0000","last_became_peered":"2021-02-10T04:50:59.561436+0000","last_unstale":"2021-02-10T04:59:58.683048+0000","last_undegraded":"2021-02-10T04:59:58.683048+0000","last_fullsized":"2021-02-10T04:59:58.683048+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.019102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.019102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"1001'1528","reported_seq":"6320","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.682998+0000","last_change":"2021-02-10T04:52:26.036366+0000","last_active":"2021-02-10T04:59:58.682998+0000","last_peered":"2021-02-10T04:59:58.682998+0000","last_clean":"2021-02-10T04:59:58.682998+0000","last_became_active":"2021-02-10T04:52:26.034028+0000","last_became_peered":"2021-02-10T04:52:26.034028+0000","last_unstale":"2021-02-10T04:59:58.682998+0000","last_undegraded":"2021-02-10T04:59:58.682998+0000","last_fullsized":"2021-02-10T04:59:58.682998+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46944136,"num_objects":16,"num_object_clones":15,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":358,"num_read_kb":351934,"num_write":286,"num_write_kb":111355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":58968832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"1261'58","reported_seq":"1441","reported_epoch":"1371","state":"active+remapped+backfill_toofull","last_fresh":"2021-02-10T04:59:57.780428+0000","last_change":"2021-02-10T04:59:46.973622+0000","last_active":"2021-02-10T04:59:57.780428+0000","last_peered":"2021-02-10T04:59:57.780428+0000","last_clean":"2021-02-10T04:59:13.930979+0000","last_became_active":"2021-02-10T04:59:16.970832+0000","last_became_peered":"2021-02-10T04:59:16.970832+0000","last_unstale":"2021-02-10T04:59:57.780428+0000","last_undegraded":"2021-02-10T04:59:57.780428+0000","last_fullsized":"2021-02-10T04:59:57.780428+0000","mapping_epoch":1344,"log_start":"1049'40","ondisk_log_start":"1049'40","created":1341,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":5,"last_scrub":"1049'40","last_scrub_stamp":"2021-02-10T04:51:47.281929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.281929+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":918,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":24,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":85,"num_read_kb":67,"num_write":50,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,1,2],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":12}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"1000'446","reported_seq":"2309","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.688978+0000","last_change":"2021-02-10T04:53:11.051124+0000","last_active":"2021-02-10T04:59:58.688978+0000","last_peered":"2021-02-10T04:59:58.688978+0000","last_clean":"2021-02-10T04:59:58.688978+0000","last_became_active":"2021-02-10T04:51:40.510899+0000","last_became_peered":"2021-02-10T04:51:40.510899+0000","last_unstale":"2021-02-10T04:59:58.688978+0000","last_undegraded":"2021-02-10T04:59:58.688978+0000","last_fullsized":"2021-02-10T04:59:58.688978+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:53:11.051031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.051031+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24896285,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":18983408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"303","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.689045+0000","last_change":"2021-02-10T04:59:55.164058+0000","last_active":"2021-02-10T04:59:58.689045+0000","last_peered":"2021-02-10T04:59:58.689045+0000","last_clean":"2021-02-10T04:59:58.689045+0000","last_became_active":"2021-02-10T04:58:56.850650+0000","last_became_peered":"2021-02-10T04:58:56.850650+0000","last_unstale":"2021-02-10T04:59:58.689045+0000","last_undegraded":"2021-02-10T04:59:58.689045+0000","last_fullsized":"2021-02-10T04:59:58.689045+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:59:55.163992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:59:55.163992+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"1304","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.684443+0000","last_change":"2021-02-10T04:51:09.475150+0000","last_active":"2021-02-10T04:59:58.684443+0000","last_peered":"2021-02-10T04:59:58.684443+0000","last_clean":"2021-02-10T04:59:58.684443+0000","last_became_active":"2021-02-10T04:50:59.585917+0000","last_became_peered":"2021-02-10T04:50:59.585917+0000","last_unstale":"2021-02-10T04:59:58.684443+0000","last_undegraded":"2021-02-10T04:59:58.684443+0000","last_fullsized":"2021-02-10T04:59:58.684443+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:09.475063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:09.475063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"274","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.689811+0000","last_change":"2021-02-10T04:53:34.927483+0000","last_active":"2021-02-10T04:59:58.689811+0000","last_peered":"2021-02-10T04:59:58.689811+0000","last_clean":"2021-02-10T04:59:58.689811+0000","last_became_active":"2021-02-10T04:51:36.859707+0000","last_became_peered":"2021-02-10T04:51:36.859707+0000","last_unstale":"2021-02-10T04:59:58.689811+0000","last_undegraded":"2021-02-10T04:59:58.689811+0000","last_fullsized":"2021-02-10T04:59:58.689811+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:34.927424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.927424+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"1261'40","reported_seq":"1043","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.684097+0000","last_change":"2021-02-10T04:59:18.642444+0000","last_active":"2021-02-10T04:59:58.684097+0000","last_peered":"2021-02-10T04:59:58.684097+0000","last_clean":"2021-02-10T04:59:58.684097+0000","last_became_active":"2021-02-10T04:59:18.641966+0000","last_became_peered":"2021-02-10T04:59:18.641966+0000","last_unstale":"2021-02-10T04:59:58.684097+0000","last_undegraded":"2021-02-10T04:59:58.684097+0000","last_fullsized":"2021-02-10T04:59:58.684097+0000","mapping_epoch":1346,"log_start":"1063'30","ondisk_log_start":"1063'30","created":1341,"last_epoch_clean":1347,"parent":"0.0","parent_split_bits":5,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:56.431187+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.431187+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":401,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":58,"num_read_kb":47,"num_write":34,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.14","version":"1000'620","reported_seq":"4659","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.684062+0000","last_change":"2021-02-10T04:53:37.384461+0000","last_active":"2021-02-10T04:59:58.684062+0000","last_peered":"2021-02-10T04:59:58.684062+0000","last_clean":"2021-02-10T04:59:58.684062+0000","last_became_active":"2021-02-10T04:51:40.510114+0000","last_became_peered":"2021-02-10T04:51:40.510114+0000","last_unstale":"2021-02-10T04:59:58.684062+0000","last_undegraded":"2021-02-10T04:59:58.684062+0000","last_fullsized":"2021-02-10T04:59:58.684062+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:53:37.384398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:37.384398+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28530390,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"37","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.687692+0000","last_change":"2021-02-10T04:59:51.205792+0000","last_active":"2021-02-10T04:59:58.687692+0000","last_peered":"2021-02-10T04:59:58.687692+0000","last_clean":"2021-02-10T04:59:58.687692+0000","last_became_active":"2021-02-10T04:59:51.122736+0000","last_became_peered":"2021-02-10T04:59:51.122736+0000","last_unstale":"2021-02-10T04:59:58.687692+0000","last_undegraded":"2021-02-10T04:59:58.687692+0000","last_fullsized":"2021-02-10T04:59:58.687692+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:52:41.059159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.059159+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"1365","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.706957+0000","last_change":"2021-02-10T04:53:12.332890+0000","last_active":"2021-02-10T05:00:00.706957+0000","last_peered":"2021-02-10T05:00:00.706957+0000","last_clean":"2021-02-10T05:00:00.706957+0000","last_became_active":"2021-02-10T04:50:59.540375+0000","last_became_peered":"2021-02-10T04:50:59.540375+0000","last_unstale":"2021-02-10T05:00:00.706957+0000","last_undegraded":"2021-02-10T05:00:00.706957+0000","last_fullsized":"2021-02-10T05:00:00.706957+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.332233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.332233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"374","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.780250+0000","last_change":"2021-02-10T04:59:51.749664+0000","last_active":"2021-02-10T04:59:57.780250+0000","last_peered":"2021-02-10T04:59:57.780250+0000","last_clean":"2021-02-10T04:59:57.780250+0000","last_became_active":"2021-02-10T04:59:51.099036+0000","last_became_peered":"2021-02-10T04:59:51.099036+0000","last_unstale":"2021-02-10T04:59:57.780250+0000","last_undegraded":"2021-02-10T04:59:57.780250+0000","last_fullsized":"2021-02-10T04:59:57.780250+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:59:51.749568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:59:51.749568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"1002'2042","reported_seq":"9344","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.780886+0000","last_change":"2021-02-10T04:53:46.726353+0000","last_active":"2021-02-10T04:59:57.780886+0000","last_peered":"2021-02-10T04:59:57.780886+0000","last_clean":"2021-02-10T04:59:57.780886+0000","last_became_active":"2021-02-10T04:53:46.725828+0000","last_became_peered":"2021-02-10T04:53:46.725828+0000","last_unstale":"2021-02-10T04:59:57.780886+0000","last_undegraded":"2021-02-10T04:59:57.780886+0000","last_fullsized":"2021-02-10T04:59:57.780886+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:52:41.286006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.286006+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46300076,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":545,"num_read_kb":531689,"num_write":399,"num_write_kb":153087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"1261'56","reported_seq":"1543","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.684425+0000","last_change":"2021-02-10T04:52:35.091494+0000","last_active":"2021-02-10T04:59:58.684425+0000","last_peered":"2021-02-10T04:59:58.684425+0000","last_clean":"2021-02-10T04:59:58.684425+0000","last_became_active":"2021-02-10T04:50:59.569915+0000","last_became_peered":"2021-02-10T04:50:59.569915+0000","last_unstale":"2021-02-10T04:59:58.684425+0000","last_undegraded":"2021-02-10T04:59:58.684425+0000","last_fullsized":"2021-02-10T04:59:58.684425+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'46","last_scrub_stamp":"2021-02-10T04:52:35.091374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:35.091374+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":95,"num_read_kb":80,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"84'7","reported_seq":"1389","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.707513+0000","last_change":"2021-02-10T04:51:52.498251+0000","last_active":"2021-02-10T05:00:00.707513+0000","last_peered":"2021-02-10T05:00:00.707513+0000","last_clean":"2021-02-10T05:00:00.707513+0000","last_became_active":"2021-02-10T04:50:59.575924+0000","last_became_peered":"2021-02-10T04:50:59.575924+0000","last_unstale":"2021-02-10T05:00:00.707513+0000","last_undegraded":"2021-02-10T05:00:00.707513+0000","last_fullsized":"2021-02-10T05:00:00.707513+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'7","last_scrub_stamp":"2021-02-10T04:51:52.498158+0000","last_deep_scrub":"84'7","last_deep_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.498158+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"1333'6","reported_seq":"1794","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.706433+0000","last_change":"2021-02-10T04:58:57.207618+0000","last_active":"2021-02-10T05:00:00.706433+0000","last_peered":"2021-02-10T05:00:00.706433+0000","last_clean":"2021-02-10T05:00:00.706433+0000","last_became_active":"2021-02-10T04:58:56.849778+0000","last_became_peered":"2021-02-10T04:58:56.849778+0000","last_unstale":"2021-02-10T05:00:00.706433+0000","last_undegraded":"2021-02-10T05:00:00.706433+0000","last_fullsized":"2021-02-10T05:00:00.706433+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1317'5","last_scrub_stamp":"2021-02-10T04:58:57.207527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:58:57.207527+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"1277","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.780690+0000","last_change":"2021-02-10T04:59:08.653632+0000","last_active":"2021-02-10T04:59:57.780690+0000","last_peered":"2021-02-10T04:59:57.780690+0000","last_clean":"2021-02-10T04:59:57.780690+0000","last_became_active":"2021-02-10T04:58:56.965228+0000","last_became_peered":"2021-02-10T04:58:56.965228+0000","last_unstale":"2021-02-10T04:59:57.780690+0000","last_undegraded":"2021-02-10T04:59:57.780690+0000","last_fullsized":"2021-02-10T04:59:57.780690+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:23.566027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_clean_scrub_stamp":"2021-02-10T04:52:23.566027+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"1000'490","reported_seq":"3908","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.689274+0000","last_change":"2021-02-10T04:59:58.274981+0000","last_active":"2021-02-10T04:59:58.689274+0000","last_peered":"2021-02-10T04:59:58.689274+0000","last_clean":"2021-02-10T04:59:58.689274+0000","last_became_active":"2021-02-10T04:58:56.965659+0000","last_became_peered":"2021-02-10T04:58:56.965659+0000","last_unstale":"2021-02-10T04:59:58.689274+0000","last_undegraded":"2021-02-10T04:59:58.689274+0000","last_fullsized":"2021-02-10T04:59:58.689274+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:59:58.274905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:59:58.274905+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16154790,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"1261'46","reported_seq":"1442","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.706231+0000","last_change":"2021-02-10T04:53:10.276449+0000","last_active":"2021-02-10T05:00:00.706231+0000","last_peered":"2021-02-10T05:00:00.706231+0000","last_clean":"2021-02-10T05:00:00.706231+0000","last_became_active":"2021-02-10T04:50:59.539529+0000","last_became_peered":"2021-02-10T04:50:59.539529+0000","last_unstale":"2021-02-10T05:00:00.706231+0000","last_undegraded":"2021-02-10T05:00:00.706231+0000","last_fullsized":"2021-02-10T05:00:00.706231+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1112'40","last_scrub_stamp":"2021-02-10T04:53:10.276358+0000","last_deep_scrub":"1112'40","last_deep_scrub_stamp":"2021-02-10T04:51:01.450050+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.276358+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":53,"num_write":38,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"1262","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.780773+0000","last_change":"2021-02-10T04:59:08.645662+0000","last_active":"2021-02-10T04:59:57.780773+0000","last_peered":"2021-02-10T04:59:57.780773+0000","last_clean":"2021-02-10T04:59:57.780773+0000","last_became_active":"2021-02-10T04:58:56.970118+0000","last_became_peered":"2021-02-10T04:58:56.970118+0000","last_unstale":"2021-02-10T04:59:57.780773+0000","last_undegraded":"2021-02-10T04:59:57.780773+0000","last_fullsized":"2021-02-10T04:59:57.780773+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:23.904210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:47:23.904210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"1119'6","reported_seq":"1686","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.684729+0000","last_change":"2021-02-10T04:52:40.038646+0000","last_active":"2021-02-10T04:59:58.684729+0000","last_peered":"2021-02-10T04:59:58.684729+0000","last_clean":"2021-02-10T04:59:58.684729+0000","last_became_active":"2021-02-10T04:51:36.869733+0000","last_became_peered":"2021-02-10T04:51:36.869733+0000","last_unstale":"2021-02-10T04:59:58.684729+0000","last_undegraded":"2021-02-10T04:59:58.684729+0000","last_fullsized":"2021-02-10T04:59:58.684729+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'6","last_scrub_stamp":"2021-02-10T04:52:40.038554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:40.038554+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"1358","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.689783+0000","last_change":"2021-02-10T04:51:36.849933+0000","last_active":"2021-02-10T04:59:58.689783+0000","last_peered":"2021-02-10T04:59:58.689783+0000","last_clean":"2021-02-10T04:59:58.689783+0000","last_became_active":"2021-02-10T04:51:36.849184+0000","last_became_peered":"2021-02-10T04:51:36.849184+0000","last_unstale":"2021-02-10T04:59:58.689783+0000","last_undegraded":"2021-02-10T04:59:58.689783+0000","last_fullsized":"2021-02-10T04:59:58.689783+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:11.779779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:11.779779+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"1032'89","reported_seq":"988","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.779322+0000","last_change":"2021-02-10T04:59:51.521944+0000","last_active":"2021-02-10T04:59:57.779322+0000","last_peered":"2021-02-10T04:59:57.779322+0000","last_clean":"2021-02-10T04:59:57.779322+0000","last_became_active":"2021-02-10T04:59:51.116954+0000","last_became_peered":"2021-02-10T04:59:51.116954+0000","last_unstale":"2021-02-10T04:59:57.779322+0000","last_undegraded":"2021-02-10T04:59:57.779322+0000","last_fullsized":"2021-02-10T04:59:57.779322+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":5,"last_scrub":"1032'89","last_scrub_stamp":"2021-02-10T04:52:34.089789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:34.089789+0000","log_size":89,"ondisk_log_size":89,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5218574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":30326,"num_write":21,"num_write_kb":12444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":9159733,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"1261'43","reported_seq":"991","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.689159+0000","last_change":"2021-02-10T04:59:51.202467+0000","last_active":"2021-02-10T04:59:58.689159+0000","last_peered":"2021-02-10T04:59:58.689159+0000","last_clean":"2021-02-10T04:59:58.689159+0000","last_became_active":"2021-02-10T04:59:51.144449+0000","last_became_peered":"2021-02-10T04:59:51.144449+0000","last_unstale":"2021-02-10T04:59:58.689159+0000","last_undegraded":"2021-02-10T04:59:58.689159+0000","last_fullsized":"2021-02-10T04:59:58.689159+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"1163'37","last_scrub_stamp":"2021-02-10T04:53:14.202445+0000","last_deep_scrub":"1163'37","last_deep_scrub_stamp":"2021-02-10T04:53:14.202445+0000","last_clean_scrub_stamp":"2021-02-10T04:53:14.202445+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":59,"num_read_kb":41,"num_write":34,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"1119","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.684593+0000","last_change":"2021-02-10T04:59:51.125357+0000","last_active":"2021-02-10T04:59:58.684593+0000","last_peered":"2021-02-10T04:59:58.684593+0000","last_clean":"2021-02-10T04:59:58.684593+0000","last_became_active":"2021-02-10T04:59:51.124981+0000","last_became_peered":"2021-02-10T04:59:51.124981+0000","last_unstale":"2021-02-10T04:59:58.684593+0000","last_undegraded":"2021-02-10T04:59:58.684593+0000","last_fullsized":"2021-02-10T04:59:58.684593+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:23.095588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:23.095588+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"1338","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.682765+0000","last_change":"2021-02-10T04:58:56.965982+0000","last_active":"2021-02-10T04:59:58.682765+0000","last_peered":"2021-02-10T04:59:58.682765+0000","last_clean":"2021-02-10T04:59:58.682765+0000","last_became_active":"2021-02-10T04:58:56.964476+0000","last_became_peered":"2021-02-10T04:58:56.964476+0000","last_unstale":"2021-02-10T04:59:58.682765+0000","last_undegraded":"2021-02-10T04:59:58.682765+0000","last_fullsized":"2021-02-10T04:59:58.682765+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:43.663762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:51:43.663762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"1233","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.779112+0000","last_change":"2021-02-10T04:59:08.655593+0000","last_active":"2021-02-10T04:59:57.779112+0000","last_peered":"2021-02-10T04:59:57.779112+0000","last_clean":"2021-02-10T04:59:57.779112+0000","last_became_active":"2021-02-10T04:58:56.877585+0000","last_became_peered":"2021-02-10T04:58:56.877585+0000","last_unstale":"2021-02-10T04:59:57.779112+0000","last_undegraded":"2021-02-10T04:59:57.779112+0000","last_fullsized":"2021-02-10T04:59:57.779112+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:57.568541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:57.568541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"1000'1071","reported_seq":"4878","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.681915+0000","last_change":"2021-02-10T04:53:35.302879+0000","last_active":"2021-02-10T04:59:58.681915+0000","last_peered":"2021-02-10T04:59:58.681915+0000","last_clean":"2021-02-10T04:59:58.681915+0000","last_became_active":"2021-02-10T04:51:39.508551+0000","last_became_peered":"2021-02-10T04:51:39.508551+0000","last_unstale":"2021-02-10T04:59:58.681915+0000","last_undegraded":"2021-02-10T04:59:58.681915+0000","last_fullsized":"2021-02-10T04:59:58.681915+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:35.302780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.302780+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"1261'39","reported_seq":"1436","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.688086+0000","last_change":"2021-02-10T04:59:51.220684+0000","last_active":"2021-02-10T04:59:58.688086+0000","last_peered":"2021-02-10T04:59:58.688086+0000","last_clean":"2021-02-10T04:59:58.688086+0000","last_became_active":"2021-02-10T04:59:51.122966+0000","last_became_peered":"2021-02-10T04:59:51.122966+0000","last_unstale":"2021-02-10T04:59:58.688086+0000","last_undegraded":"2021-02-10T04:59:58.688086+0000","last_fullsized":"2021-02-10T04:59:58.688086+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'31","last_scrub_stamp":"2021-02-10T04:52:39.695237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:39.695237+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":53,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"637","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:02.625565+0000","last_change":"2021-02-10T05:00:02.625565+0000","last_active":"2021-02-10T05:00:02.625565+0000","last_peered":"2021-02-10T05:00:02.625565+0000","last_clean":"2021-02-10T05:00:02.625565+0000","last_became_active":"2021-02-10T04:58:56.964387+0000","last_became_peered":"2021-02-10T04:58:56.964387+0000","last_unstale":"2021-02-10T05:00:02.625565+0000","last_undegraded":"2021-02-10T05:00:02.625565+0000","last_fullsized":"2021-02-10T05:00:02.625565+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T05:00:02.625497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T05:00:02.625497+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"825","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.707674+0000","last_change":"2021-02-10T04:59:40.662976+0000","last_active":"2021-02-10T05:00:00.707674+0000","last_peered":"2021-02-10T05:00:00.707674+0000","last_clean":"2021-02-10T05:00:00.707674+0000","last_became_active":"2021-02-10T04:59:40.662450+0000","last_became_peered":"2021-02-10T04:59:40.662450+0000","last_unstale":"2021-02-10T05:00:00.707674+0000","last_undegraded":"2021-02-10T05:00:00.707674+0000","last_fullsized":"2021-02-10T05:00:00.707674+0000","mapping_epoch":1356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.216394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.216394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"1365","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.683579+0000","last_change":"2021-02-10T04:58:56.879817+0000","last_active":"2021-02-10T04:59:58.683579+0000","last_peered":"2021-02-10T04:59:58.683579+0000","last_clean":"2021-02-10T04:59:58.683579+0000","last_became_active":"2021-02-10T04:58:56.879372+0000","last_became_peered":"2021-02-10T04:58:56.879372+0000","last_unstale":"2021-02-10T04:59:58.683579+0000","last_undegraded":"2021-02-10T04:59:58.683579+0000","last_fullsized":"2021-02-10T04:59:58.683579+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:55.814486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.814486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"1002'1657","reported_seq":"5390","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.780696+0000","last_change":"2021-02-10T04:59:08.654625+0000","last_active":"2021-02-10T04:59:57.780696+0000","last_peered":"2021-02-10T04:59:57.780696+0000","last_clean":"2021-02-10T04:59:57.780696+0000","last_became_active":"2021-02-10T04:58:56.961771+0000","last_became_peered":"2021-02-10T04:58:56.961771+0000","last_unstale":"2021-02-10T04:59:57.780696+0000","last_undegraded":"2021-02-10T04:59:57.780696+0000","last_fullsized":"2021-02-10T04:59:57.780696+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:55:22.331010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:55:22.331010+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":47335497,"num_objects":20,"num_object_clones":18,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"1261'24","reported_seq":"1511","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.682292+0000","last_change":"2021-02-10T04:59:51.179096+0000","last_active":"2021-02-10T04:59:58.682292+0000","last_peered":"2021-02-10T04:59:58.682292+0000","last_clean":"2021-02-10T04:59:58.682292+0000","last_became_active":"2021-02-10T04:59:51.114167+0000","last_became_peered":"2021-02-10T04:59:51.114167+0000","last_unstale":"2021-02-10T04:59:58.682292+0000","last_undegraded":"2021-02-10T04:59:58.682292+0000","last_fullsized":"2021-02-10T04:59:58.682292+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'20","last_scrub_stamp":"2021-02-10T04:53:04.058248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:04.058248+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":29,"num_write":24,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"1386","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.707183+0000","last_change":"2021-02-10T04:59:51.115470+0000","last_active":"2021-02-10T05:00:00.707183+0000","last_peered":"2021-02-10T05:00:00.707183+0000","last_clean":"2021-02-10T05:00:00.707183+0000","last_became_active":"2021-02-10T04:59:51.114568+0000","last_became_peered":"2021-02-10T04:59:51.114568+0000","last_unstale":"2021-02-10T05:00:00.707183+0000","last_undegraded":"2021-02-10T05:00:00.707183+0000","last_fullsized":"2021-02-10T05:00:00.707183+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:00.343228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:00.343228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"26","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T05:00:00.267755+0000","last_change":"2021-02-10T05:00:00.267680+0000","last_active":"2021-02-10T05:00:00.267755+0000","last_peered":"2021-02-10T05:00:00.267755+0000","last_clean":"2021-02-10T05:00:00.267755+0000","last_became_active":"2021-02-10T04:59:51.103624+0000","last_became_peered":"2021-02-10T04:59:51.103624+0000","last_unstale":"2021-02-10T05:00:00.267755+0000","last_undegraded":"2021-02-10T05:00:00.267755+0000","last_fullsized":"2021-02-10T05:00:00.267755+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T05:00:00.267589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T05:00:00.267589+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"690","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.780015+0000","last_change":"2021-02-10T04:53:46.718715+0000","last_active":"2021-02-10T04:59:57.780015+0000","last_peered":"2021-02-10T04:59:57.780015+0000","last_clean":"2021-02-10T04:59:57.780015+0000","last_became_active":"2021-02-10T04:53:46.718266+0000","last_became_peered":"2021-02-10T04:53:46.718266+0000","last_unstale":"2021-02-10T04:59:57.780015+0000","last_undegraded":"2021-02-10T04:59:57.780015+0000","last_fullsized":"2021-02-10T04:59:57.780015+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.876681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.876681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"998'1009","reported_seq":"4661","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.779897+0000","last_change":"2021-02-10T04:59:53.667158+0000","last_active":"2021-02-10T04:59:57.779897+0000","last_peered":"2021-02-10T04:59:57.779897+0000","last_clean":"2021-02-10T04:59:57.779897+0000","last_became_active":"2021-02-10T04:59:53.665843+0000","last_became_peered":"2021-02-10T04:59:53.665843+0000","last_unstale":"2021-02-10T04:59:57.779897+0000","last_undegraded":"2021-02-10T04:59:57.779897+0000","last_fullsized":"2021-02-10T04:59:57.779897+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":5,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:13.032506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:13.032506+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24710414,"num_objects":11,"num_object_clones":10,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":239,"num_read_kb":249403,"num_write":155,"num_write_kb":50104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":45053856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"1261'45","reported_seq":"1443","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.683860+0000","last_change":"2021-02-10T04:59:51.177904+0000","last_active":"2021-02-10T04:59:58.683860+0000","last_peered":"2021-02-10T04:59:58.683860+0000","last_clean":"2021-02-10T04:59:58.683860+0000","last_became_active":"2021-02-10T04:59:51.119425+0000","last_became_peered":"2021-02-10T04:59:51.119425+0000","last_unstale":"2021-02-10T04:59:58.683860+0000","last_undegraded":"2021-02-10T04:59:58.683860+0000","last_fullsized":"2021-02-10T04:59:58.683860+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:44.694711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.694711+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":76,"num_read_kb":62,"num_write":45,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"34","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.705562+0000","last_change":"2021-02-10T04:59:51.182394+0000","last_active":"2021-02-10T05:00:00.705562+0000","last_peered":"2021-02-10T05:00:00.705562+0000","last_clean":"2021-02-10T05:00:00.705562+0000","last_became_active":"2021-02-10T04:59:51.120643+0000","last_became_peered":"2021-02-10T04:59:51.120643+0000","last_unstale":"2021-02-10T05:00:00.705562+0000","last_undegraded":"2021-02-10T05:00:00.705562+0000","last_fullsized":"2021-02-10T05:00:00.705562+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:52:33.088428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:33.088428+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"667","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.682370+0000","last_change":"2021-02-10T04:59:51.110291+0000","last_active":"2021-02-10T04:59:58.682370+0000","last_peered":"2021-02-10T04:59:58.682370+0000","last_clean":"2021-02-10T04:59:58.682370+0000","last_became_active":"2021-02-10T04:59:51.109910+0000","last_became_peered":"2021-02-10T04:59:51.109910+0000","last_unstale":"2021-02-10T04:59:58.682370+0000","last_undegraded":"2021-02-10T04:59:58.682370+0000","last_fullsized":"2021-02-10T04:59:58.682370+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.836638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.836638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"1338","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.682691+0000","last_change":"2021-02-10T04:59:51.130995+0000","last_active":"2021-02-10T04:59:58.682691+0000","last_peered":"2021-02-10T04:59:58.682691+0000","last_clean":"2021-02-10T04:59:58.682691+0000","last_became_active":"2021-02-10T04:59:51.130617+0000","last_became_peered":"2021-02-10T04:59:51.130617+0000","last_unstale":"2021-02-10T04:59:58.682691+0000","last_undegraded":"2021-02-10T04:59:58.682691+0000","last_fullsized":"2021-02-10T04:59:58.682691+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:58.514318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_clean_scrub_stamp":"2021-02-10T04:46:58.514318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"1261'65","reported_seq":"1404","reported_epoch":"1372","state":"active+remapped+backfill_toofull","last_fresh":"2021-02-10T04:59:59.693500+0000","last_change":"2021-02-10T04:59:46.979060+0000","last_active":"2021-02-10T04:59:59.693500+0000","last_peered":"2021-02-10T04:59:59.693500+0000","last_clean":"2021-02-10T04:55:57.139329+0000","last_became_active":"2021-02-10T04:59:16.972842+0000","last_became_peered":"2021-02-10T04:59:16.972842+0000","last_unstale":"2021-02-10T04:59:59.693500+0000","last_undegraded":"2021-02-10T04:59:59.693500+0000","last_fullsized":"2021-02-10T04:59:59.693500+0000","mapping_epoch":1344,"log_start":"1049'47","ondisk_log_start":"1049'47","created":1341,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":5,"last_scrub":"1049'47","last_scrub_stamp":"2021-02-10T04:53:09.002368+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.002368+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":836,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":84,"num_read_kb":67,"num_write":49,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":467,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[1,4,6],"avail_no_missing":["1","4","6"],"object_location_counts":[{"shards":"1,4,6","objects":11}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"1001'1165","reported_seq":"4405","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:59.693527+0000","last_change":"2021-02-10T04:59:53.658749+0000","last_active":"2021-02-10T04:59:59.693527+0000","last_peered":"2021-02-10T04:59:59.693527+0000","last_clean":"2021-02-10T04:59:59.693527+0000","last_became_active":"2021-02-10T04:59:53.658313+0000","last_became_peered":"2021-02-10T04:59:53.658313+0000","last_unstale":"2021-02-10T04:59:59.693527+0000","last_undegraded":"2021-02-10T04:59:59.693527+0000","last_fullsized":"2021-02-10T04:59:59.693527+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":5,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29601845,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":287,"num_read_kb":299530,"num_write":200,"num_write_kb":89131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"1418","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.702022+0000","last_change":"2021-02-10T04:53:01.331566+0000","last_active":"2021-02-10T05:00:00.702022+0000","last_peered":"2021-02-10T05:00:00.702022+0000","last_clean":"2021-02-10T05:00:00.702022+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T05:00:00.702022+0000","last_undegraded":"2021-02-10T05:00:00.702022+0000","last_fullsized":"2021-02-10T05:00:00.702022+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:01.331466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.331466+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"1366'4","reported_seq":"1387","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.689736+0000","last_change":"2021-02-10T04:59:51.169693+0000","last_active":"2021-02-10T04:59:58.689736+0000","last_peered":"2021-02-10T04:59:58.689736+0000","last_clean":"2021-02-10T04:59:58.689736+0000","last_became_active":"2021-02-10T04:59:51.112805+0000","last_became_peered":"2021-02-10T04:59:51.112805+0000","last_unstale":"2021-02-10T04:59:58.689736+0000","last_undegraded":"2021-02-10T04:59:58.689736+0000","last_fullsized":"2021-02-10T04:59:58.689736+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"1098'3","last_scrub_stamp":"2021-02-10T04:53:34.865579+0000","last_deep_scrub":"1098'3","last_deep_scrub_stamp":"2021-02-10T04:50:15.255137+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.865579+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"1220","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.780596+0000","last_change":"2021-02-10T04:53:46.727603+0000","last_active":"2021-02-10T04:59:57.780596+0000","last_peered":"2021-02-10T04:59:57.780596+0000","last_clean":"2021-02-10T04:59:57.780596+0000","last_became_active":"2021-02-10T04:53:46.727053+0000","last_became_peered":"2021-02-10T04:53:46.727053+0000","last_unstale":"2021-02-10T04:59:57.780596+0000","last_undegraded":"2021-02-10T04:59:57.780596+0000","last_fullsized":"2021-02-10T04:59:57.780596+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:47.493057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.493057+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"1261'71","reported_seq":"1179","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.705668+0000","last_change":"2021-02-10T04:59:53.657664+0000","last_active":"2021-02-10T05:00:00.705668+0000","last_peered":"2021-02-10T05:00:00.705668+0000","last_clean":"2021-02-10T05:00:00.705668+0000","last_became_active":"2021-02-10T04:59:53.657339+0000","last_became_peered":"2021-02-10T04:59:53.657339+0000","last_unstale":"2021-02-10T05:00:00.705668+0000","last_undegraded":"2021-02-10T05:00:00.705668+0000","last_fullsized":"2021-02-10T05:00:00.705668+0000","mapping_epoch":1366,"log_start":"1049'51","ondisk_log_start":"1049'51","created":1341,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":5,"last_scrub":"1049'41","last_scrub_stamp":"2021-02-10T04:51:58.353660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:58.353660+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":85,"num_read_kb":68,"num_write":51,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":2625,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1c","version":"1002'1090","reported_seq":"5398","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.706643+0000","last_change":"2021-02-10T04:53:41.353991+0000","last_active":"2021-02-10T05:00:00.706643+0000","last_peered":"2021-02-10T05:00:00.706643+0000","last_clean":"2021-02-10T05:00:00.706643+0000","last_became_active":"2021-02-10T04:52:31.088265+0000","last_became_peered":"2021-02-10T04:52:31.088265+0000","last_unstale":"2021-02-10T05:00:00.706643+0000","last_undegraded":"2021-02-10T05:00:00.706643+0000","last_fullsized":"2021-02-10T05:00:00.706643+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:53:41.353840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:41.353840+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"1342","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.683734+0000","last_change":"2021-02-10T04:59:51.130114+0000","last_active":"2021-02-10T04:59:58.683734+0000","last_peered":"2021-02-10T04:59:58.683734+0000","last_clean":"2021-02-10T04:59:58.683734+0000","last_became_active":"2021-02-10T04:59:51.129732+0000","last_became_peered":"2021-02-10T04:59:51.129732+0000","last_unstale":"2021-02-10T04:59:58.683734+0000","last_undegraded":"2021-02-10T04:59:58.683734+0000","last_fullsized":"2021-02-10T04:59:58.683734+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:42.537443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:42.537443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"1258","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.780133+0000","last_change":"2021-02-10T04:59:08.666315+0000","last_active":"2021-02-10T04:59:57.780133+0000","last_peered":"2021-02-10T04:59:57.780133+0000","last_clean":"2021-02-10T04:59:57.780133+0000","last_became_active":"2021-02-10T04:58:56.869951+0000","last_became_peered":"2021-02-10T04:58:56.869951+0000","last_unstale":"2021-02-10T04:59:57.780133+0000","last_undegraded":"2021-02-10T04:59:57.780133+0000","last_fullsized":"2021-02-10T04:59:57.780133+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.938731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.938731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"810","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.706639+0000","last_change":"2021-02-10T04:59:40.659174+0000","last_active":"2021-02-10T05:00:00.706639+0000","last_peered":"2021-02-10T05:00:00.706639+0000","last_clean":"2021-02-10T05:00:00.706639+0000","last_became_active":"2021-02-10T04:59:40.658818+0000","last_became_peered":"2021-02-10T04:59:40.658818+0000","last_unstale":"2021-02-10T05:00:00.706639+0000","last_undegraded":"2021-02-10T05:00:00.706639+0000","last_fullsized":"2021-02-10T05:00:00.706639+0000","mapping_epoch":1356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:15.185686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:15.185686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"1261'76","reported_seq":"1443","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.689585+0000","last_change":"2021-02-10T04:59:48.064635+0000","last_active":"2021-02-10T04:59:58.689585+0000","last_peered":"2021-02-10T04:59:58.689585+0000","last_clean":"2021-02-10T04:59:58.689585+0000","last_became_active":"2021-02-10T04:59:48.064161+0000","last_became_peered":"2021-02-10T04:59:48.064161+0000","last_unstale":"2021-02-10T04:59:58.689585+0000","last_undegraded":"2021-02-10T04:59:58.689585+0000","last_fullsized":"2021-02-10T04:59:58.689585+0000","mapping_epoch":1360,"log_start":"1049'39","ondisk_log_start":"1049'39","created":1341,"last_epoch_clean":1361,"parent":"0.0","parent_split_bits":5,"last_scrub":"1187'54","last_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_deep_scrub":"1187'54","last_deep_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_clean_scrub_stamp":"2021-02-10T04:53:42.071103+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":289,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":73,"num_read_kb":63,"num_write":48,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1f","version":"1004'1093","reported_seq":"6590","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.688515+0000","last_change":"2021-02-10T04:59:57.679787+0000","last_active":"2021-02-10T04:59:58.688515+0000","last_peered":"2021-02-10T04:59:58.688515+0000","last_clean":"2021-02-10T04:59:58.688515+0000","last_became_active":"2021-02-10T04:59:57.678313+0000","last_became_peered":"2021-02-10T04:59:57.678313+0000","last_unstale":"2021-02-10T04:59:58.688515+0000","last_undegraded":"2021-02-10T04:59:58.688515+0000","last_fullsized":"2021-02-10T04:59:58.688515+0000","mapping_epoch":1370,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1371,"parent":"0.0","parent_split_bits":5,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39434697,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":279,"num_read_kb":311058,"num_write":186,"num_write_kb":66790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":53995913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"37","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.687588+0000","last_change":"2021-02-10T04:59:51.137374+0000","last_active":"2021-02-10T04:59:58.687588+0000","last_peered":"2021-02-10T04:59:58.687588+0000","last_clean":"2021-02-10T04:59:58.687588+0000","last_became_active":"2021-02-10T04:59:51.118991+0000","last_became_peered":"2021-02-10T04:59:51.118991+0000","last_unstale":"2021-02-10T04:59:58.687588+0000","last_undegraded":"2021-02-10T04:59:58.687588+0000","last_fullsized":"2021-02-10T04:59:58.687588+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:52:36.083784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:36.083784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"813","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:01.225550+0000","last_change":"2021-02-10T05:00:01.225550+0000","last_active":"2021-02-10T05:00:01.225550+0000","last_peered":"2021-02-10T05:00:01.225550+0000","last_clean":"2021-02-10T05:00:01.225550+0000","last_became_active":"2021-02-10T04:59:40.661806+0000","last_became_peered":"2021-02-10T04:59:40.661806+0000","last_unstale":"2021-02-10T05:00:01.225550+0000","last_undegraded":"2021-02-10T05:00:01.225550+0000","last_fullsized":"2021-02-10T05:00:01.225550+0000","mapping_epoch":1356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T05:00:01.225483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T05:00:01.225483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"1349","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.688967+0000","last_change":"2021-02-10T04:59:51.126018+0000","last_active":"2021-02-10T04:59:58.688967+0000","last_peered":"2021-02-10T04:59:58.688967+0000","last_clean":"2021-02-10T04:59:58.688967+0000","last_became_active":"2021-02-10T04:59:51.125671+0000","last_became_peered":"2021-02-10T04:59:51.125671+0000","last_unstale":"2021-02-10T04:59:58.688967+0000","last_undegraded":"2021-02-10T04:59:58.688967+0000","last_fullsized":"2021-02-10T04:59:58.688967+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:44.799429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.799429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"1261'64","reported_seq":"1513","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.683666+0000","last_change":"2021-02-10T04:59:18.642028+0000","last_active":"2021-02-10T04:59:58.683666+0000","last_peered":"2021-02-10T04:59:58.683666+0000","last_clean":"2021-02-10T04:59:58.683666+0000","last_became_active":"2021-02-10T04:59:18.641628+0000","last_became_peered":"2021-02-10T04:59:18.641628+0000","last_unstale":"2021-02-10T04:59:58.683666+0000","last_undegraded":"2021-02-10T04:59:58.683666+0000","last_fullsized":"2021-02-10T04:59:58.683666+0000","mapping_epoch":1346,"log_start":"1066'44","ondisk_log_start":"1066'44","created":1341,"last_epoch_clean":1347,"parent":"0.0","parent_split_bits":5,"last_scrub":"1066'44","last_scrub_stamp":"2021-02-10T04:53:37.108677+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:53:37.108677+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":789,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":90,"num_read_kb":72,"num_write":53,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":2000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"1063'654","reported_seq":"4401","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.683618+0000","last_change":"2021-02-10T04:59:40.665470+0000","last_active":"2021-02-10T04:59:58.683618+0000","last_peered":"2021-02-10T04:59:58.683618+0000","last_clean":"2021-02-10T04:59:58.683618+0000","last_became_active":"2021-02-10T04:59:40.665009+0000","last_became_peered":"2021-02-10T04:59:40.665009+0000","last_unstale":"2021-02-10T04:59:58.683618+0000","last_undegraded":"2021-02-10T04:59:58.683618+0000","last_fullsized":"2021-02-10T04:59:58.683618+0000","mapping_epoch":1356,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1357,"parent":"0.0","parent_split_bits":5,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:53:40.135367+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:53:40.135367+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24700631,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":171,"num_read_kb":182272,"num_write":112,"num_write_kb":33727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":26349024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"1333","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.683193+0000","last_change":"2021-02-10T04:59:51.120413+0000","last_active":"2021-02-10T04:59:58.683193+0000","last_peered":"2021-02-10T04:59:58.683193+0000","last_clean":"2021-02-10T04:59:58.683193+0000","last_became_active":"2021-02-10T04:59:51.120031+0000","last_became_peered":"2021-02-10T04:59:51.120031+0000","last_unstale":"2021-02-10T04:59:58.683193+0000","last_undegraded":"2021-02-10T04:59:58.683193+0000","last_fullsized":"2021-02-10T04:59:58.683193+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_clean_scrub_stamp":"2021-02-10T04:51:02.455702+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"1375","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.688104+0000","last_change":"2021-02-10T04:52:33.741801+0000","last_active":"2021-02-10T04:59:58.688104+0000","last_peered":"2021-02-10T04:59:58.688104+0000","last_clean":"2021-02-10T04:59:58.688104+0000","last_became_active":"2021-02-10T04:51:36.860129+0000","last_became_peered":"2021-02-10T04:51:36.860129+0000","last_unstale":"2021-02-10T04:59:58.688104+0000","last_undegraded":"2021-02-10T04:59:58.688104+0000","last_fullsized":"2021-02-10T04:59:58.688104+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:33.741711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:46:58.666264+0000","last_clean_scrub_stamp":"2021-02-10T04:52:33.741711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"1345","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.684272+0000","last_change":"2021-02-10T04:59:51.111726+0000","last_active":"2021-02-10T04:59:58.684272+0000","last_peered":"2021-02-10T04:59:58.684272+0000","last_clean":"2021-02-10T04:59:58.684272+0000","last_became_active":"2021-02-10T04:59:51.111316+0000","last_became_peered":"2021-02-10T04:59:51.111316+0000","last_unstale":"2021-02-10T04:59:58.684272+0000","last_undegraded":"2021-02-10T04:59:58.684272+0000","last_fullsized":"2021-02-10T04:59:58.684272+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:25.030281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.030281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"1261'69","reported_seq":"1494","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.687201+0000","last_change":"2021-02-10T04:59:48.070593+0000","last_active":"2021-02-10T04:59:58.687201+0000","last_peered":"2021-02-10T04:59:58.687201+0000","last_clean":"2021-02-10T04:59:58.687201+0000","last_became_active":"2021-02-10T04:59:48.069866+0000","last_became_peered":"2021-02-10T04:59:48.069866+0000","last_unstale":"2021-02-10T04:59:58.687201+0000","last_undegraded":"2021-02-10T04:59:58.687201+0000","last_fullsized":"2021-02-10T04:59:58.687201+0000","mapping_epoch":1360,"log_start":"1112'47","ondisk_log_start":"1112'47","created":1341,"last_epoch_clean":1361,"parent":"0.0","parent_split_bits":5,"last_scrub":"1049'61","last_scrub_stamp":"2021-02-10T04:53:36.836202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:36.836202+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":733,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":104,"num_read_kb":86,"num_write":61,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1101,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.19","version":"1003'739","reported_seq":"3747","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.779426+0000","last_change":"2021-02-10T04:59:54.663625+0000","last_active":"2021-02-10T04:59:57.779426+0000","last_peered":"2021-02-10T04:59:57.779426+0000","last_clean":"2021-02-10T04:59:57.779426+0000","last_became_active":"2021-02-10T04:59:54.662899+0000","last_became_peered":"2021-02-10T04:59:54.662899+0000","last_unstale":"2021-02-10T04:59:57.779426+0000","last_undegraded":"2021-02-10T04:59:57.779426+0000","last_fullsized":"2021-02-10T04:59:57.779426+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:53:01.929169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.929169+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24059889,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":130,"num_read_kb":109544,"num_write":102,"num_write_kb":46174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":18429397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"1211","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.780374+0000","last_change":"2021-02-10T04:53:46.727590+0000","last_active":"2021-02-10T04:59:57.780374+0000","last_peered":"2021-02-10T04:59:57.780374+0000","last_clean":"2021-02-10T04:59:57.780374+0000","last_became_active":"2021-02-10T04:53:46.726584+0000","last_became_peered":"2021-02-10T04:53:46.726584+0000","last_unstale":"2021-02-10T04:59:57.780374+0000","last_undegraded":"2021-02-10T04:59:57.780374+0000","last_fullsized":"2021-02-10T04:59:57.780374+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:26.408326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.408326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"1374","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.706028+0000","last_change":"2021-02-10T04:53:07.324650+0000","last_active":"2021-02-10T05:00:00.706028+0000","last_peered":"2021-02-10T05:00:00.706028+0000","last_clean":"2021-02-10T05:00:00.706028+0000","last_became_active":"2021-02-10T04:50:59.575340+0000","last_became_peered":"2021-02-10T04:50:59.575340+0000","last_unstale":"2021-02-10T05:00:00.706028+0000","last_undegraded":"2021-02-10T05:00:00.706028+0000","last_fullsized":"2021-02-10T05:00:00.706028+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.324564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.324564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"1257","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.778878+0000","last_change":"2021-02-10T04:59:08.651086+0000","last_active":"2021-02-10T04:59:57.778878+0000","last_peered":"2021-02-10T04:59:57.778878+0000","last_clean":"2021-02-10T04:59:57.778878+0000","last_became_active":"2021-02-10T04:58:56.953706+0000","last_became_peered":"2021-02-10T04:58:56.953706+0000","last_unstale":"2021-02-10T04:59:57.778878+0000","last_undegraded":"2021-02-10T04:59:57.778878+0000","last_fullsized":"2021-02-10T04:59:57.778878+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:00.501127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:00.501127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"1261'28","reported_seq":"1316","reported_epoch":"1372","state":"active+remapped+backfill_toofull","last_fresh":"2021-02-10T04:59:58.687356+0000","last_change":"2021-02-10T04:59:46.969555+0000","last_active":"2021-02-10T04:59:58.687356+0000","last_peered":"2021-02-10T04:59:58.687356+0000","last_clean":"2021-02-10T04:55:57.106046+0000","last_became_active":"2021-02-10T04:59:16.966181+0000","last_became_peered":"2021-02-10T04:59:16.966181+0000","last_unstale":"2021-02-10T04:59:58.687356+0000","last_undegraded":"2021-02-10T04:59:58.687356+0000","last_fullsized":"2021-02-10T04:59:58.687356+0000","mapping_epoch":1344,"log_start":"1061'20","ondisk_log_start":"1061'20","created":1341,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":5,"last_scrub":"1049'32","last_scrub_stamp":"2021-02-10T04:51:51.382040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.382040+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":579,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":29,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1026,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,1,0],"avail_no_missing":["4","0","1"],"object_location_counts":[{"shards":"0,1,4","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"259'74","reported_seq":"311","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.779015+0000","last_change":"2021-02-10T04:59:51.110714+0000","last_active":"2021-02-10T04:59:57.779015+0000","last_peered":"2021-02-10T04:59:57.779015+0000","last_clean":"2021-02-10T04:59:57.779015+0000","last_became_active":"2021-02-10T04:59:51.110275+0000","last_became_peered":"2021-02-10T04:59:51.110275+0000","last_unstale":"2021-02-10T04:59:57.779015+0000","last_undegraded":"2021-02-10T04:59:57.779015+0000","last_fullsized":"2021-02-10T04:59:57.779015+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":5,"last_scrub":"259'74","last_scrub_stamp":"2021-02-10T04:52:26.597668+0000","last_deep_scrub":"259'74","last_deep_scrub_stamp":"2021-02-10T04:46:49.745104+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.597668+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1218524,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":28228,"num_write":19,"num_write_kb":7022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"185","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.689644+0000","last_change":"2021-02-10T04:59:57.159970+0000","last_active":"2021-02-10T04:59:58.689644+0000","last_peered":"2021-02-10T04:59:58.689644+0000","last_clean":"2021-02-10T04:59:58.689644+0000","last_became_active":"2021-02-10T04:59:51.139457+0000","last_became_peered":"2021-02-10T04:59:51.139457+0000","last_unstale":"2021-02-10T04:59:58.689644+0000","last_undegraded":"2021-02-10T04:59:58.689644+0000","last_fullsized":"2021-02-10T04:59:58.689644+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:59:57.159904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:59:57.159904+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"281","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.779859+0000","last_change":"2021-02-10T04:53:46.718512+0000","last_active":"2021-02-10T04:59:57.779859+0000","last_peered":"2021-02-10T04:59:57.779859+0000","last_clean":"2021-02-10T04:59:57.779859+0000","last_became_active":"2021-02-10T04:53:46.716811+0000","last_became_peered":"2021-02-10T04:53:46.716811+0000","last_unstale":"2021-02-10T04:59:57.779859+0000","last_undegraded":"2021-02-10T04:59:57.779859+0000","last_fullsized":"2021-02-10T04:59:57.779859+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:45.813676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.813676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"1363","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.703337+0000","last_change":"2021-02-10T04:53:38.363899+0000","last_active":"2021-02-10T05:00:00.703337+0000","last_peered":"2021-02-10T05:00:00.703337+0000","last_clean":"2021-02-10T05:00:00.703337+0000","last_became_active":"2021-02-10T04:50:59.539256+0000","last_became_peered":"2021-02-10T04:50:59.539256+0000","last_unstale":"2021-02-10T05:00:00.703337+0000","last_undegraded":"2021-02-10T05:00:00.703337+0000","last_fullsized":"2021-02-10T05:00:00.703337+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:38.363825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:09.433793+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.363825+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"1261'84","reported_seq":"1582","reported_epoch":"1372","state":"active+remapped+backfill_toofull","last_fresh":"2021-02-10T04:59:58.687000+0000","last_change":"2021-02-10T04:59:52.107625+0000","last_active":"2021-02-10T04:59:58.687000+0000","last_peered":"2021-02-10T04:59:58.687000+0000","last_clean":"2021-02-10T04:59:49.079006+0000","last_became_active":"2021-02-10T04:59:52.099182+0000","last_became_peered":"2021-02-10T04:59:52.099182+0000","last_unstale":"2021-02-10T04:59:58.687000+0000","last_undegraded":"2021-02-10T04:59:58.687000+0000","last_fullsized":"2021-02-10T04:59:58.687000+0000","mapping_epoch":1364,"log_start":"1132'66","ondisk_log_start":"1132'66","created":1341,"last_epoch_clean":1362,"parent":"0.0","parent_split_bits":5,"last_scrub":"1132'66","last_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_deep_scrub":"1132'66","last_deep_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.430399+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":808,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":93,"num_read_kb":75,"num_write":54,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1557,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,6],"avail_no_missing":["4","1","5","6","7"],"object_location_counts":[{"shards":"1,4,6","objects":12}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"1003'1689","reported_seq":"4710","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.780454+0000","last_change":"2021-02-10T04:59:08.659989+0000","last_active":"2021-02-10T04:59:57.780454+0000","last_peered":"2021-02-10T04:59:57.780454+0000","last_clean":"2021-02-10T04:59:57.780454+0000","last_became_active":"2021-02-10T04:58:56.970049+0000","last_became_peered":"2021-02-10T04:58:56.970049+0000","last_unstale":"2021-02-10T04:59:57.780454+0000","last_undegraded":"2021-02-10T04:59:57.780454+0000","last_fullsized":"2021-02-10T04:59:57.780454+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:53:10.075879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.075879+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27159495,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"1372","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.684086+0000","last_change":"2021-02-10T04:58:56.966215+0000","last_active":"2021-02-10T04:59:58.684086+0000","last_peered":"2021-02-10T04:59:58.684086+0000","last_clean":"2021-02-10T04:59:58.684086+0000","last_became_active":"2021-02-10T04:58:56.965975+0000","last_became_peered":"2021-02-10T04:58:56.965975+0000","last_unstale":"2021-02-10T04:59:58.684086+0000","last_undegraded":"2021-02-10T04:59:58.684086+0000","last_fullsized":"2021-02-10T04:59:58.684086+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:33.543559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T04:46:33.543559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"1033","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.703135+0000","last_change":"2021-02-10T04:53:04.354508+0000","last_active":"2021-02-10T05:00:00.703135+0000","last_peered":"2021-02-10T05:00:00.703135+0000","last_clean":"2021-02-10T05:00:00.703135+0000","last_became_active":"2021-02-10T04:33:05.169159+0000","last_became_peered":"2021-02-10T04:33:05.169159+0000","last_unstale":"2021-02-10T05:00:00.703135+0000","last_undegraded":"2021-02-10T05:00:00.703135+0000","last_fullsized":"2021-02-10T05:00:00.703135+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:04.354409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:04.354409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"1426","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.687506+0000","last_change":"2021-02-10T04:52:36.777554+0000","last_active":"2021-02-10T04:59:58.687506+0000","last_peered":"2021-02-10T04:59:58.687506+0000","last_clean":"2021-02-10T04:59:58.687506+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T04:59:58.687506+0000","last_undegraded":"2021-02-10T04:59:58.687506+0000","last_fullsized":"2021-02-10T04:59:58.687506+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:36.777485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:52:36.777485+0000","last_clean_scrub_stamp":"2021-02-10T04:52:36.777485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"1261'86","reported_seq":"1488","reported_epoch":"1372","state":"active+remapped+backfill_toofull","last_fresh":"2021-02-10T04:59:58.683341+0000","last_change":"2021-02-10T04:59:46.970284+0000","last_active":"2021-02-10T04:59:58.683341+0000","last_peered":"2021-02-10T04:59:58.683341+0000","last_clean":"2021-02-10T04:55:57.156886+0000","last_became_active":"2021-02-10T04:59:16.967624+0000","last_became_peered":"2021-02-10T04:59:16.967624+0000","last_unstale":"2021-02-10T04:59:58.683341+0000","last_undegraded":"2021-02-10T04:59:58.683341+0000","last_fullsized":"2021-02-10T04:59:58.683341+0000","mapping_epoch":1344,"log_start":"1049'68","ondisk_log_start":"1049'68","created":1341,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":5,"last_scrub":"1048'39","last_scrub_stamp":"2021-02-10T04:53:38.085565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.085565+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":791,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":26,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":113,"num_read_kb":93,"num_write":64,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1582,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,1,4],"avail_no_missing":["3","1","4"],"object_location_counts":[{"shards":"1,3,4","objects":13}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"1001'1528","reported_seq":"6000","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.687367+0000","last_change":"2021-02-10T04:59:53.654638+0000","last_active":"2021-02-10T04:59:58.687367+0000","last_peered":"2021-02-10T04:59:58.687367+0000","last_clean":"2021-02-10T04:59:58.687367+0000","last_became_active":"2021-02-10T04:59:53.654185+0000","last_became_peered":"2021-02-10T04:59:53.654185+0000","last_unstale":"2021-02-10T04:59:58.687367+0000","last_undegraded":"2021-02-10T04:59:58.687367+0000","last_fullsized":"2021-02-10T04:59:58.687367+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":5,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46944136,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":358,"num_read_kb":351935,"num_write":286,"num_write_kb":111355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":21311893,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"1252","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T05:00:00.081989+0000","last_change":"2021-02-10T05:00:00.081989+0000","last_active":"2021-02-10T05:00:00.081989+0000","last_peered":"2021-02-10T05:00:00.081989+0000","last_clean":"2021-02-10T05:00:00.081989+0000","last_became_active":"2021-02-10T04:58:56.855338+0000","last_became_peered":"2021-02-10T04:58:56.855338+0000","last_unstale":"2021-02-10T05:00:00.081989+0000","last_undegraded":"2021-02-10T05:00:00.081989+0000","last_fullsized":"2021-02-10T05:00:00.081989+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T05:00:00.081916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T05:00:00.081916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"667","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.681895+0000","last_change":"2021-02-10T04:59:51.127111+0000","last_active":"2021-02-10T04:59:58.681895+0000","last_peered":"2021-02-10T04:59:58.681895+0000","last_clean":"2021-02-10T04:59:58.681895+0000","last_became_active":"2021-02-10T04:59:51.126766+0000","last_became_peered":"2021-02-10T04:59:51.126766+0000","last_unstale":"2021-02-10T04:59:58.681895+0000","last_undegraded":"2021-02-10T04:59:58.681895+0000","last_fullsized":"2021-02-10T04:59:58.681895+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.091949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:19.755718+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.091949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"1344","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.682917+0000","last_change":"2021-02-10T04:59:51.116613+0000","last_active":"2021-02-10T04:59:58.682917+0000","last_peered":"2021-02-10T04:59:58.682917+0000","last_clean":"2021-02-10T04:59:58.682917+0000","last_became_active":"2021-02-10T04:59:51.116241+0000","last_became_peered":"2021-02-10T04:59:51.116241+0000","last_unstale":"2021-02-10T04:59:58.682917+0000","last_undegraded":"2021-02-10T04:59:58.682917+0000","last_fullsized":"2021-02-10T04:59:58.682917+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.838790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.838790+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":"1069","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.706565+0000","last_change":"2021-02-10T04:52:30.405170+0000","last_active":"2021-02-10T05:00:00.706565+0000","last_peered":"2021-02-10T05:00:00.706565+0000","last_clean":"2021-02-10T05:00:00.706565+0000","last_became_active":"2021-02-10T04:51:36.852058+0000","last_became_peered":"2021-02-10T04:51:36.852058+0000","last_unstale":"2021-02-10T05:00:00.706565+0000","last_undegraded":"2021-02-10T05:00:00.706565+0000","last_fullsized":"2021-02-10T05:00:00.706565+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.405106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:20.635536+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.405106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.20","version":"0'0","reported_seq":"314","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.779377+0000","last_change":"2021-02-10T04:59:51.118148+0000","last_active":"2021-02-10T04:59:57.779377+0000","last_peered":"2021-02-10T04:59:57.779377+0000","last_clean":"2021-02-10T04:59:57.779377+0000","last_became_active":"2021-02-10T04:59:51.117759+0000","last_became_peered":"2021-02-10T04:59:51.117759+0000","last_unstale":"2021-02-10T04:59:57.779377+0000","last_undegraded":"2021-02-10T04:59:57.779377+0000","last_fullsized":"2021-02-10T04:59:57.779377+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:56.449085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:41.375276+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.449085+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":"1053","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.690141+0000","last_change":"2021-02-10T04:59:51.134402+0000","last_active":"2021-02-10T04:59:58.690141+0000","last_peered":"2021-02-10T04:59:58.690141+0000","last_clean":"2021-02-10T04:59:58.690141+0000","last_became_active":"2021-02-10T04:59:51.133981+0000","last_became_peered":"2021-02-10T04:59:51.133981+0000","last_unstale":"2021-02-10T04:59:58.690141+0000","last_undegraded":"2021-02-10T04:59:58.690141+0000","last_fullsized":"2021-02-10T04:59:58.690141+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.858557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.858557+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":"299","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.689062+0000","last_change":"2021-02-10T04:58:56.879602+0000","last_active":"2021-02-10T04:59:58.689062+0000","last_peered":"2021-02-10T04:59:58.689062+0000","last_clean":"2021-02-10T04:59:58.689062+0000","last_became_active":"2021-02-10T04:58:56.879153+0000","last_became_peered":"2021-02-10T04:58:56.879153+0000","last_unstale":"2021-02-10T04:59:58.689062+0000","last_undegraded":"2021-02-10T04:59:58.689062+0000","last_fullsized":"2021-02-10T04:59:58.689062+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:55:23.586401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:55:23.586401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":"284","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:59.693800+0000","last_change":"2021-02-10T04:58:56.966218+0000","last_active":"2021-02-10T04:59:59.693800+0000","last_peered":"2021-02-10T04:59:59.693800+0000","last_clean":"2021-02-10T04:59:59.693800+0000","last_became_active":"2021-02-10T04:58:56.965861+0000","last_became_peered":"2021-02-10T04:58:56.965861+0000","last_unstale":"2021-02-10T04:59:59.693800+0000","last_undegraded":"2021-02-10T04:59:59.693800+0000","last_fullsized":"2021-02-10T04:59:59.693800+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.706372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.706372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":"1128","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.687798+0000","last_change":"2021-02-10T04:59:51.127003+0000","last_active":"2021-02-10T04:59:58.687798+0000","last_peered":"2021-02-10T04:59:58.687798+0000","last_clean":"2021-02-10T04:59:58.687798+0000","last_became_active":"2021-02-10T04:59:51.126491+0000","last_became_peered":"2021-02-10T04:59:51.126491+0000","last_unstale":"2021-02-10T04:59:58.687798+0000","last_undegraded":"2021-02-10T04:59:58.687798+0000","last_fullsized":"2021-02-10T04:59:58.687798+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.287363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.287363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":"342","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.780968+0000","last_change":"2021-02-10T04:59:51.117057+0000","last_active":"2021-02-10T04:59:57.780968+0000","last_peered":"2021-02-10T04:59:57.780968+0000","last_clean":"2021-02-10T04:59:57.780968+0000","last_became_active":"2021-02-10T04:59:51.116609+0000","last_became_peered":"2021-02-10T04:59:51.116609+0000","last_unstale":"2021-02-10T04:59:57.780968+0000","last_undegraded":"2021-02-10T04:59:57.780968+0000","last_fullsized":"2021-02-10T04:59:57.780968+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:10.308282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:10.308282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":"1304","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.683267+0000","last_change":"2021-02-10T04:59:51.133004+0000","last_active":"2021-02-10T04:59:58.683267+0000","last_peered":"2021-02-10T04:59:58.683267+0000","last_clean":"2021-02-10T04:59:58.683267+0000","last_became_active":"2021-02-10T04:59:51.132662+0000","last_became_peered":"2021-02-10T04:59:51.132662+0000","last_unstale":"2021-02-10T04:59:58.683267+0000","last_undegraded":"2021-02-10T04:59:58.683267+0000","last_fullsized":"2021-02-10T04:59:58.683267+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:09.121298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.121298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.21","version":"293'77","reported_seq":"943","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.683790+0000","last_change":"2021-02-10T04:52:28.062923+0000","last_active":"2021-02-10T04:59:58.683790+0000","last_peered":"2021-02-10T04:59:58.683790+0000","last_clean":"2021-02-10T04:59:58.683790+0000","last_became_active":"2021-02-10T04:52:28.062463+0000","last_became_peered":"2021-02-10T04:52:28.062463+0000","last_unstale":"2021-02-10T04:59:58.683790+0000","last_undegraded":"2021-02-10T04:59:58.683790+0000","last_fullsized":"2021-02-10T04:59:58.683790+0000","mapping_epoch":1143,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1144,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1886586,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":15163,"num_write":9,"num_write_kb":3316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":"269","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.688916+0000","last_change":"2021-02-10T04:52:32.124955+0000","last_active":"2021-02-10T04:59:58.688916+0000","last_peered":"2021-02-10T04:59:58.688916+0000","last_clean":"2021-02-10T04:59:58.688916+0000","last_became_active":"2021-02-10T04:51:36.867061+0000","last_became_peered":"2021-02-10T04:51:36.867061+0000","last_unstale":"2021-02-10T04:59:58.688916+0000","last_undegraded":"2021-02-10T04:59:58.688916+0000","last_fullsized":"2021-02-10T04:59:58.688916+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:32.124901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:08.271977+0000","last_clean_scrub_stamp":"2021-02-10T04:52:32.124901+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":"92","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.780353+0000","last_change":"2021-02-10T04:59:51.115844+0000","last_active":"2021-02-10T04:59:57.780353+0000","last_peered":"2021-02-10T04:59:57.780353+0000","last_clean":"2021-02-10T04:59:57.780353+0000","last_became_active":"2021-02-10T04:59:51.115454+0000","last_became_peered":"2021-02-10T04:59:51.115454+0000","last_unstale":"2021-02-10T04:59:57.780353+0000","last_undegraded":"2021-02-10T04:59:57.780353+0000","last_fullsized":"2021-02-10T04:59:57.780353+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.182678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.182678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.20","version":"1000'1071","reported_seq":"4820","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.688559+0000","last_change":"2021-02-10T04:59:55.666986+0000","last_active":"2021-02-10T04:59:58.688559+0000","last_peered":"2021-02-10T04:59:58.688559+0000","last_clean":"2021-02-10T04:59:58.688559+0000","last_became_active":"2021-02-10T04:59:55.666246+0000","last_became_peered":"2021-02-10T04:59:55.666246+0000","last_unstale":"2021-02-10T04:59:58.688559+0000","last_undegraded":"2021-02-10T04:59:58.688559+0000","last_fullsized":"2021-02-10T04:59:58.688559+0000","mapping_epoch":1368,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1369,"parent":"0.0","parent_split_bits":6,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:03.915403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.915403+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16398893,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":11822743,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":"894","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.779267+0000","last_change":"2021-02-10T04:53:46.721599+0000","last_active":"2021-02-10T04:59:57.779267+0000","last_peered":"2021-02-10T04:59:57.779267+0000","last_clean":"2021-02-10T04:59:57.779267+0000","last_became_active":"2021-02-10T04:53:46.721340+0000","last_became_peered":"2021-02-10T04:53:46.721340+0000","last_unstale":"2021-02-10T04:59:57.779267+0000","last_undegraded":"2021-02-10T04:59:57.779267+0000","last_fullsized":"2021-02-10T04:59:57.779267+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:28.092288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:07.767250+0000","last_clean_scrub_stamp":"2021-02-10T04:52:28.092288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.25","version":"53'2","reported_seq":"1340","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.688486+0000","last_change":"2021-02-10T04:59:51.233212+0000","last_active":"2021-02-10T04:59:58.688486+0000","last_peered":"2021-02-10T04:59:58.688486+0000","last_clean":"2021-02-10T04:59:58.688486+0000","last_became_active":"2021-02-10T04:59:51.122097+0000","last_became_peered":"2021-02-10T04:59:51.122097+0000","last_unstale":"2021-02-10T04:59:58.688486+0000","last_undegraded":"2021-02-10T04:59:58.688486+0000","last_fullsized":"2021-02-10T04:59:58.688486+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:53:36.102928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:36.102928+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.23","version":"1002'1657","reported_seq":"5101","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:59.694094+0000","last_change":"2021-02-10T04:59:53.661728+0000","last_active":"2021-02-10T04:59:59.694094+0000","last_peered":"2021-02-10T04:59:59.694094+0000","last_clean":"2021-02-10T04:59:59.694094+0000","last_became_active":"2021-02-10T04:59:53.658988+0000","last_became_peered":"2021-02-10T04:59:53.658988+0000","last_unstale":"2021-02-10T04:59:59.694094+0000","last_undegraded":"2021-02-10T04:59:59.694094+0000","last_fullsized":"2021-02-10T04:59:59.694094+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":6,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21396533,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":212,"num_read_kb":218837,"num_write":159,"num_write_kb":59399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":20479176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":"1164","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.682799+0000","last_change":"2021-02-10T04:59:51.127951+0000","last_active":"2021-02-10T04:59:58.682799+0000","last_peered":"2021-02-10T04:59:58.682799+0000","last_clean":"2021-02-10T04:59:58.682799+0000","last_became_active":"2021-02-10T04:59:51.127586+0000","last_became_peered":"2021-02-10T04:59:51.127586+0000","last_unstale":"2021-02-10T04:59:58.682799+0000","last_undegraded":"2021-02-10T04:59:58.682799+0000","last_fullsized":"2021-02-10T04:59:58.682799+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.951528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.951528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.26","version":"53'1","reported_seq":"217","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.780581+0000","last_change":"2021-02-10T04:59:51.108138+0000","last_active":"2021-02-10T04:59:57.780581+0000","last_peered":"2021-02-10T04:59:57.780581+0000","last_clean":"2021-02-10T04:59:57.780581+0000","last_became_active":"2021-02-10T04:59:51.107693+0000","last_became_peered":"2021-02-10T04:59:51.107693+0000","last_unstale":"2021-02-10T04:59:57.780581+0000","last_undegraded":"2021-02-10T04:59:57.780581+0000","last_fullsized":"2021-02-10T04:59:57.780581+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:53:03.933910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.933910+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":423,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.22","version":"998'1009","reported_seq":"4734","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.687328+0000","last_change":"2021-02-10T04:52:31.086563+0000","last_active":"2021-02-10T04:59:58.687328+0000","last_peered":"2021-02-10T04:59:58.687328+0000","last_clean":"2021-02-10T04:59:58.687328+0000","last_became_active":"2021-02-10T04:52:31.085288+0000","last_became_peered":"2021-02-10T04:52:31.085288+0000","last_unstale":"2021-02-10T04:59:58.687328+0000","last_undegraded":"2021-02-10T04:59:58.687328+0000","last_fullsized":"2021-02-10T04:59:58.687328+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12355207,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":120,"num_read_kb":124702,"num_write":77,"num_write_kb":25052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4145029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":"976","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.687642+0000","last_change":"2021-02-10T04:59:51.125119+0000","last_active":"2021-02-10T04:59:58.687642+0000","last_peered":"2021-02-10T04:59:58.687642+0000","last_clean":"2021-02-10T04:59:58.687642+0000","last_became_active":"2021-02-10T04:59:51.124340+0000","last_became_peered":"2021-02-10T04:59:51.124340+0000","last_unstale":"2021-02-10T04:59:58.687642+0000","last_undegraded":"2021-02-10T04:59:58.687642+0000","last_fullsized":"2021-02-10T04:59:58.687642+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:10.958579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.958579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":"313","reported_epoch":"1371","state":"active+clean","last_fresh":"2021-02-10T04:59:57.780963+0000","last_change":"2021-02-10T04:53:46.725941+0000","last_active":"2021-02-10T04:59:57.780963+0000","last_peered":"2021-02-10T04:59:57.780963+0000","last_clean":"2021-02-10T04:59:57.780963+0000","last_became_active":"2021-02-10T04:53:46.725587+0000","last_became_peered":"2021-02-10T04:53:46.725587+0000","last_unstale":"2021-02-10T04:59:57.780963+0000","last_undegraded":"2021-02-10T04:59:57.780963+0000","last_fullsized":"2021-02-10T04:59:57.780963+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:41.758684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.758684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":"1309","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.682038+0000","last_change":"2021-02-10T04:55:39.599193+0000","last_active":"2021-02-10T04:59:58.682038+0000","last_peered":"2021-02-10T04:59:58.682038+0000","last_clean":"2021-02-10T04:59:58.682038+0000","last_became_active":"2021-02-10T04:55:39.598816+0000","last_became_peered":"2021-02-10T04:55:39.598816+0000","last_unstale":"2021-02-10T04:59:58.682038+0000","last_undegraded":"2021-02-10T04:59:58.682038+0000","last_fullsized":"2021-02-10T04:59:58.682038+0000","mapping_epoch":1248,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:11.139602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.139602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":"1072","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:58.683632+0000","last_change":"2021-02-10T04:59:51.121337+0000","last_active":"2021-02-10T04:59:58.683632+0000","last_peered":"2021-02-10T04:59:58.683632+0000","last_clean":"2021-02-10T04:59:58.683632+0000","last_became_active":"2021-02-10T04:59:51.118780+0000","last_became_peered":"2021-02-10T04:59:51.118780+0000","last_unstale":"2021-02-10T04:59:58.683632+0000","last_undegraded":"2021-02-10T04:59:58.683632+0000","last_fullsized":"2021-02-10T04:59:58.683632+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:24.071915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_clean_scrub_stamp":"2021-02-10T04:52:24.071915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":897745652,"num_objects":365,"num_object_clones":315,"num_object_copies":1095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":365,"num_whiteouts":10,"num_read":5141,"num_read_kb":5222490,"num_write":3715,"num_write_kb":1422177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1173,"num_bytes_recovered":575169527,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2167754752,"data_stored":2724507644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1639,"internal_metadata":0},"log_size":34728,"ondisk_log_size":34728,"up":108,"acting":108,"num_store_stats":8},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":1945,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":18,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":2031,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":6889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31,"ondisk_log_size":31,"up":126,"acting":126,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":13,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":630,"num_read_kb":625,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":973,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20,"ondisk_log_size":20,"up":39,"acting":39,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":120,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":2706,"num_read_kb":2179,"num_write":1592,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":419,"num_bytes_recovered":29054,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1033,"ondisk_log_size":1033,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":126,"acting":126,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365215,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":250540,"kb_used_data":197732,"kb_used_omap":0,"kb_used_meta":52800,"kb_avail":93494612,"statfs":{"total":95995035648,"available":95738482688,"internally_reserved":0,"allocated":202477568,"data_stored":257924771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54067200},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:59:55 2021","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.494,"5min":1.494,"15min":1.494},"last":0.872},{"interface":"front","average":{"1min":0.995,"5min":0.995,"15min":0.995},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":2.02,"5min":2.02,"15min":2.02},"last":1.206}]},{"osd":1,"last update":"Wed Feb 10 04:59:48 2021","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":1.054,"15min":0.94599999999999995},"min":{"1min":0.53400000000000003,"5min":0.443,"15min":0.443},"max":{"1min":1.78,"5min":15.253,"15min":15.253},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.89700000000000002,"5min":1.1279999999999999,"15min":1.016},"min":{"1min":0.59599999999999997,"5min":0.47499999999999998,"15min":0.45500000000000002},"max":{"1min":1.74,"5min":18.713000000000001,"15min":18.713000000000001},"last":0.96099999999999997}]},{"osd":2,"last update":"Wed Feb 10 04:59:48 2021","interfaces":[{"interface":"back","average":{"1min":3.2949999999999999,"5min":3.1579999999999999,"15min":1.853},"min":{"1min":0.67600000000000005,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":49.811999999999998,"5min":94.391999999999996,"15min":94.391999999999996},"last":1.1499999999999999},{"interface":"front","average":{"1min":3.286,"5min":3.113,"15min":1.8500000000000001},"min":{"1min":0.55900000000000005,"5min":0.54300000000000004,"15min":0.496},"max":{"1min":49.668999999999997,"5min":94.269999999999996,"15min":94.269999999999996},"last":1.26}]},{"osd":3,"last update":"Wed Feb 10 04:59:03 2021","interfaces":[{"interface":"back","average":{"1min":1.657,"5min":1.2050000000000001,"15min":1.264},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.58199999999999996},"max":{"1min":15.148,"5min":15.148,"15min":15.148},"last":1.347},{"interface":"front","average":{"1min":1.891,"5min":1.363,"15min":1.3839999999999999},"min":{"1min":0.71899999999999997,"5min":0.629,"15min":0.629},"max":{"1min":18.611000000000001,"5min":18.611000000000001,"15min":18.611000000000001},"last":1.2969999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:59:48 2021","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.883,"15min":1.528},"min":{"1min":0.68600000000000005,"5min":0.60399999999999998,"15min":0.59899999999999998},"max":{"1min":2.206,"5min":72.662999999999997,"15min":72.662999999999997},"last":1.532},{"interface":"front","average":{"1min":1.2709999999999999,"5min":2.1659999999999999,"15min":1.595},"min":{"1min":0.81499999999999995,"5min":0.73999999999999999,"15min":0.69099999999999995},"max":{"1min":2.3900000000000001,"5min":73.564999999999998,"15min":73.564999999999998},"last":1.4650000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:59:48 2021","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.5449999999999999,"15min":1.5109999999999999},"min":{"1min":0.88800000000000001,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.2909999999999999,"5min":11.627000000000001,"15min":14.972},"last":1.4279999999999999},{"interface":"front","average":{"1min":1.544,"5min":1.427,"15min":1.581},"min":{"1min":0.79200000000000004,"5min":0.76600000000000001,"15min":0.72699999999999998},"max":{"1min":2.5880000000000001,"5min":13.172000000000001,"15min":15.218999999999999},"last":1.3999999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:59:48 2021","interfaces":[{"interface":"back","average":{"1min":1.5309999999999999,"5min":1.4370000000000001,"15min":1.5960000000000001},"min":{"1min":0.85999999999999999,"5min":0.78200000000000003,"15min":0.72699999999999998},"max":{"1min":2.0139999999999998,"5min":10.286,"15min":16.050000000000001},"last":1.671},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.6000000000000001,"15min":1.635},"min":{"1min":0.96699999999999997,"5min":0.88100000000000001,"15min":0.876},"max":{"1min":2.5049999999999999,"5min":17.460000000000001,"15min":17.460000000000001},"last":1.645}]}]},{"osd":6,"up_from":0,"seq":184683594148,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834575285,"num_pgs":102,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":347108,"kb_used_data":280668,"kb_used_omap":0,"kb_used_meta":66432,"kb_avail":93398044,"statfs":{"total":95995035648,"available":95639597056,"internally_reserved":0,"allocated":287404032,"data_stored":354806114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":68026368},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 05:00:01 2021","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.238,"5min":1.238,"15min":1.238},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.64400000000000002}]},{"osd":2,"last update":"Wed Feb 10 04:59:24 2021","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":5.3099999999999996,"15min":3.1949999999999998},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.42399999999999999},"max":{"1min":20.527000000000001,"5min":162.61199999999999,"15min":162.61199999999999},"last":1.028},{"interface":"front","average":{"1min":1.7130000000000001,"5min":5.4459999999999997,"15min":3.286},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":20.483000000000001,"5min":162.762,"15min":162.762},"last":0.71699999999999997}]},{"osd":3,"last update":"Wed Feb 10 04:59:08 2021","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.83399999999999996,"15min":0.88500000000000001},"min":{"1min":0.495,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.0629999999999999,"5min":2.8919999999999999,"15min":3.077},"last":1.115},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.84699999999999998,"15min":0.95599999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42199999999999999},"max":{"1min":1.4159999999999999,"5min":3.1890000000000001,"15min":3.1890000000000001},"last":1.3919999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:59:05 2021","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":1.919,"15min":1.393},"min":{"1min":0.58899999999999997,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.837,"5min":100.071,"15min":100.071},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.9370000000000001,"15min":1.4239999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.53500000000000003},"max":{"1min":1.6990000000000001,"5min":100.43899999999999,"15min":100.43899999999999},"last":1.784}]},{"osd":5,"last update":"Wed Feb 10 04:59:08 2021","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.1579999999999999,"15min":1.1599999999999999},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.603,"5min":3.0219999999999998,"15min":5.0300000000000002},"last":1.421},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.202,"15min":1.3120000000000001},"min":{"1min":0.79900000000000004,"5min":0.70399999999999996,"15min":0.68600000000000005},"max":{"1min":1.9319999999999999,"5min":6.032,"15min":6.032},"last":0.94099999999999995}]},{"osd":6,"last update":"Wed Feb 10 04:59:24 2021","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.159,"15min":1.333},"min":{"1min":0.83599999999999997,"5min":0.68999999999999995,"15min":0.59599999999999997},"max":{"1min":1.599,"5min":3.302,"15min":30.997},"last":1.6699999999999999},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.2609999999999999,"15min":1.3460000000000001},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.60499999999999998},"max":{"1min":1.893,"5min":3.1549999999999998,"15min":31.138000000000002},"last":1.49}]},{"osd":7,"last update":"Wed Feb 10 04:59:30 2021","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.2709999999999999,"15min":1.331},"min":{"1min":0.89500000000000002,"5min":0.72299999999999998,"15min":0.628},"max":{"1min":2.052,"5min":6.2629999999999999,"15min":6.2629999999999999},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.244,"15min":1.3300000000000001},"min":{"1min":0.92000000000000004,"5min":0.83299999999999996,"15min":0.78700000000000003},"max":{"1min":1.647,"5min":3.4129999999999998,"15min":3.754},"last":1.5529999999999999}]}]},{"osd":0,"up_from":1332,"seq":5720896438286,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":325108,"kb_used_data":310060,"kb_used_omap":324,"kb_used_meta":14715,"kb_avail":93420044,"statfs":{"total":95995035648,"available":95662125056,"internally_reserved":0,"allocated":317501440,"data_stored":410927004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":332393,"internal_metadata":15068567},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 04:59:58 2021","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.62},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.753}]},{"osd":2,"last update":"Wed Feb 10 04:59:58 2021","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.76,"5min":1.76,"15min":1.76},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.339,"5min":1.339,"15min":1.339},"last":0.88800000000000001}]},{"osd":3,"last update":"Wed Feb 10 04:59:58 2021","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.78100000000000003}]},{"osd":4,"last update":"Wed Feb 10 04:59:58 2021","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":2.6389999999999998,"5min":2.6389999999999998,"15min":2.6389999999999998},"last":0.81100000000000005}]},{"osd":5,"last update":"Wed Feb 10 04:59:58 2021","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.323,"15min":1.323},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":2.5139999999999998,"5min":2.5139999999999998,"15min":2.5139999999999998},"last":1.014},{"interface":"front","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":2.2709999999999999,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.83499999999999996}]},{"osd":6,"last update":"Wed Feb 10 04:59:58 2021","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.282,"15min":1.282},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":3.0329999999999999,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":1.042},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"min":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"max":{"1min":2.1840000000000002,"5min":2.1840000000000002,"15min":2.1840000000000002},"last":1.0680000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:59:58 2021","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"min":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"max":{"1min":2.9729999999999999,"5min":2.9729999999999999,"15min":2.9729999999999999},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"min":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"max":{"1min":2.2970000000000002,"5min":2.2970000000000002,"15min":2.2970000000000002},"last":1.171}]}]},{"osd":2,"up_from":18,"seq":77309411763,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":477252,"kb_used_data":430204,"kb_used_omap":0,"kb_used_meta":47040,"kb_avail":93267900,"statfs":{"total":95995035648,"available":95506329600,"internally_reserved":0,"allocated":440528896,"data_stored":508506684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48168960},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:59:57 2021","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.76100000000000001}]},{"osd":1,"last update":"Wed Feb 10 04:59:03 2021","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.82699999999999996,"15min":0.92000000000000004},"min":{"1min":0.41999999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.84499999999999997,"5min":17.041,"15min":17.041},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.69899999999999995,"5min":1.0289999999999999,"15min":0.92200000000000004},"min":{"1min":0.46300000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.825,"5min":21.172000000000001,"15min":21.172000000000001},"last":0.89800000000000002}]},{"osd":3,"last update":"Wed Feb 10 04:59:22 2021","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":1.022,"15min":1.069},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.3460000000000001,"5min":18.904,"15min":18.904},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.002,"5min":1.262,"15min":1.2689999999999999},"min":{"1min":0.71299999999999997,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.6000000000000001,"5min":19.241,"15min":19.241},"last":1.169}]},{"osd":4,"last update":"Wed Feb 10 04:59:39 2021","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.2410000000000001,"15min":1.228},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.677,"5min":9.2029999999999994,"15min":9.2029999999999994},"last":1.4670000000000001},{"interface":"front","average":{"1min":1.01,"5min":1.115,"15min":1.161},"min":{"1min":0.66500000000000004,"5min":0.503,"15min":0.503},"max":{"1min":1.518,"5min":3.403,"15min":24.373999999999999},"last":1.091}]},{"osd":5,"last update":"Wed Feb 10 04:59:03 2021","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.665,"15min":1.7090000000000001},"min":{"1min":0.68300000000000005,"5min":0.66200000000000003,"15min":0.63500000000000001},"max":{"1min":1.6599999999999999,"5min":39.911999999999999,"15min":39.911999999999999},"last":1.2929999999999999},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.379,"15min":1.4410000000000001},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.57599999999999996},"max":{"1min":1.3480000000000001,"5min":33.539999999999999,"15min":33.539999999999999},"last":0.86499999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:59:36 2021","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.071,"15min":1.3009999999999999},"min":{"1min":0.70799999999999996,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.677,"5min":3.1659999999999999,"15min":20.007000000000001},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.4770000000000001,"15min":1.556},"min":{"1min":0.65800000000000003,"5min":0.622,"15min":0.622},"max":{"1min":1.768,"5min":16.135000000000002,"15min":19.417999999999999},"last":1.577}]},{"osd":7,"last update":"Wed Feb 10 04:59:39 2021","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.4370000000000001,"15min":1.5840000000000001},"min":{"1min":0.86699999999999999,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.8260000000000001,"5min":9.3460000000000001,"15min":24.405000000000001},"last":1.393},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.234,"15min":1.232},"min":{"1min":0.871,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.478,"5min":3.6640000000000001,"15min":4.9189999999999996},"last":1.1970000000000001}]}]},{"osd":3,"up_from":1111,"seq":4771708665966,"num_pgs":102,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":368356,"kb_used_data":350684,"kb_used_omap":386,"kb_used_meta":17277,"kb_avail":93376796,"statfs":{"total":95995035648,"available":95617839104,"internally_reserved":0,"allocated":359100416,"data_stored":471846720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":396036,"internal_metadata":17691900},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:59:56 2021","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.556,"5min":1.556,"15min":1.556},"last":1.196}]},{"osd":1,"last update":"Wed Feb 10 04:59:12 2021","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.82699999999999996,"15min":0.71099999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44},"max":{"1min":6.1980000000000004,"5min":6.1980000000000004,"15min":6.1980000000000004},"last":0.55700000000000005},{"interface":"front","average":{"1min":1.1859999999999999,"5min":0.89200000000000002,"15min":0.873},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.45600000000000002},"max":{"1min":7.7709999999999999,"5min":7.7709999999999999,"15min":7.7709999999999999},"last":1.0580000000000001}]},{"osd":2,"last update":"Wed Feb 10 04:59:12 2021","interfaces":[{"interface":"back","average":{"1min":4.5419999999999998,"5min":2.1219999999999999,"15min":1.4930000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":81.138000000000005,"5min":81.138000000000005,"15min":81.138000000000005},"last":3.621},{"interface":"front","average":{"1min":4.5590000000000002,"5min":2.3220000000000001,"15min":1.5700000000000001},"min":{"1min":0.54400000000000004,"5min":0.53800000000000003,"15min":0.50800000000000001},"max":{"1min":81.099999999999994,"5min":81.099999999999994,"15min":81.099999999999994},"last":0.83199999999999996}]},{"osd":4,"last update":"Wed Feb 10 04:59:12 2021","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":1.0009999999999999,"15min":0.97899999999999998},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.5,"5min":2.6589999999999998,"15min":2.9430000000000001},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.327,"5min":1.143,"15min":1.1040000000000001},"min":{"1min":0.63900000000000001,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":7.484,"5min":7.484,"15min":7.484},"last":1.532}]},{"osd":5,"last update":"Wed Feb 10 04:59:12 2021","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.2649999999999999,"15min":2.2170000000000001},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":7.6379999999999999,"5min":7.6379999999999999,"15min":52.353000000000002},"last":1.645},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.0249999999999999,"15min":2.02},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":4.8730000000000002,"5min":4.8730000000000002,"15min":52.301000000000002},"last":0.91500000000000004}]},{"osd":6,"last update":"Wed Feb 10 04:59:12 2021","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.1180000000000001,"15min":3.7570000000000001},"min":{"1min":0.68500000000000005,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":5.258,"5min":5.258,"15min":64.376999999999995},"last":1.5},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.2909999999999999,"15min":3.8130000000000002},"min":{"1min":0.88400000000000001,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":64.438999999999993},"last":1.5780000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:59:12 2021","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.1319999999999999,"15min":1.115},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.754},"max":{"1min":5.3819999999999997,"5min":5.3819999999999997,"15min":5.3819999999999997},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.145,"5min":1.1899999999999999,"15min":1.157},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.79900000000000004},"max":{"1min":1.6859999999999999,"5min":2.9569999999999999,"15min":2.9569999999999999},"last":1.4650000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986603,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":166612,"kb_used_data":120588,"kb_used_omap":0,"kb_used_meta":46016,"kb_avail":93578540,"statfs":{"total":95995035648,"available":95824424960,"internally_reserved":0,"allocated":123482112,"data_stored":155339097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":47120384},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:59:57 2021","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":2.1619999999999999,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":2.4609999999999999,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":0.95199999999999996}]},{"osd":1,"last update":"Wed Feb 10 04:59:51 2021","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.80600000000000005,"15min":0.85099999999999998},"min":{"1min":0.59899999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":2.9630000000000001,"5min":2.9630000000000001,"15min":7.883},"last":0.78700000000000003},{"interface":"front","average":{"1min":1.226,"5min":0.98599999999999999,"15min":1.024},"min":{"1min":0.65800000000000003,"5min":0.47999999999999998,"15min":0.45900000000000002},"max":{"1min":3.5619999999999998,"5min":3.5619999999999998,"15min":4.3449999999999998},"last":0.82199999999999995}]},{"osd":2,"last update":"Wed Feb 10 04:59:51 2021","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":1.1930000000000001,"15min":1.6899999999999999},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.51000000000000001},"max":{"1min":4.1070000000000002,"5min":7.0069999999999997,"15min":120.39},"last":11.425000000000001},{"interface":"front","average":{"1min":1.228,"5min":1.0840000000000001,"15min":1.659},"min":{"1min":0.70399999999999996,"5min":0.504,"15min":0.504},"max":{"1min":4.1989999999999998,"5min":7.1120000000000001,"15min":120.583},"last":11.407}]},{"osd":3,"last update":"Wed Feb 10 04:59:06 2021","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.1299999999999999,"15min":1.216},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":2.3100000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.157},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.2030000000000001,"15min":1.494},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.57399999999999995},"max":{"1min":2.548,"5min":2.548,"15min":26.802},"last":1.2669999999999999}]},{"osd":5,"last update":"Wed Feb 10 04:59:06 2021","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.04,"15min":1.401},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.495},"max":{"1min":2.5499999999999998,"5min":2.5499999999999998,"15min":35.774000000000001},"last":1.704},{"interface":"front","average":{"1min":1.024,"5min":1.1379999999999999,"15min":1.419},"min":{"1min":0.621,"5min":0.621,"15min":0.60899999999999999},"max":{"1min":2.3170000000000002,"5min":3.8410000000000002,"15min":35.606000000000002},"last":1.893}]},{"osd":6,"last update":"Wed Feb 10 04:59:06 2021","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.4830000000000001,"15min":2.3570000000000002},"min":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"max":{"1min":2.6819999999999999,"5min":3.8530000000000002,"15min":201.626},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.2,"5min":1.3200000000000001,"15min":2.319},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.68100000000000005},"max":{"1min":2.4100000000000001,"5min":4.7910000000000004,"15min":201.577},"last":1.2909999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:59:39 2021","interfaces":[{"interface":"back","average":{"1min":1.784,"5min":1.5309999999999999,"15min":1.496},"min":{"1min":1.1599999999999999,"5min":0.90300000000000002,"15min":0.67100000000000004},"max":{"1min":3.3420000000000001,"5min":4.6200000000000001,"15min":4.7439999999999998},"last":1.383},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.3169999999999999,"15min":1.4219999999999999},"min":{"1min":0.81100000000000005,"5min":0.79100000000000004,"15min":0.63200000000000001},"max":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"last":1.2}]}]},{"osd":5,"up_from":37,"seq":158913790376,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":342336,"kb_used_data":289464,"kb_used_omap":0,"kb_used_meta":52864,"kb_avail":93402816,"statfs":{"total":95995035648,"available":95644483584,"internally_reserved":0,"allocated":296411136,"data_stored":378633957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54132736},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:59:56 2021","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.597,"5min":2.597,"15min":2.597},"last":1.24}]},{"osd":1,"last update":"Wed Feb 10 04:59:06 2021","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.85299999999999998,"15min":0.90400000000000003},"min":{"1min":0.438,"5min":0.435,"15min":0.435},"max":{"1min":1.9790000000000001,"5min":4.242,"15min":6.391},"last":1.611},{"interface":"front","average":{"1min":0.82299999999999995,"5min":1.0800000000000001,"15min":1.1339999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.41299999999999998},"max":{"1min":1.3480000000000001,"5min":13.794,"15min":13.794},"last":1.117}]},{"osd":2,"last update":"Wed Feb 10 04:59:06 2021","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":3.4569999999999999,"15min":2.1440000000000001},"min":{"1min":0.504,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.748,"5min":116.438,"15min":116.438},"last":3.5649999999999999},{"interface":"front","average":{"1min":0.92500000000000004,"5min":3.5009999999999999,"15min":2.206},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.8759999999999999,"5min":116.96899999999999,"15min":116.96899999999999},"last":3.0649999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:59:15 2021","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.367,"15min":1.302},"min":{"1min":0.57499999999999996,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.4750000000000001,"5min":13.755000000000001,"15min":13.755000000000001},"last":1.274},{"interface":"front","average":{"1min":1.073,"5min":1.1639999999999999,"15min":1.1839999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.64700000000000002},"max":{"1min":2.331,"5min":5.577,"15min":6.5129999999999999},"last":1.6899999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:59:06 2021","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":1.054,"15min":1.246},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.841,"5min":5.298,"15min":9.4890000000000008},"last":1.522},{"interface":"front","average":{"1min":1.016,"5min":1.1899999999999999,"15min":1.319},"min":{"1min":0.61499999999999999,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.327,"5min":4.8289999999999997,"15min":5.6180000000000003},"last":1.4239999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:59:15 2021","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.3440000000000001,"15min":1.7370000000000001},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.7450000000000001,"5min":4.9429999999999996,"15min":38.014000000000003},"last":1.573},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.302,"15min":1.6859999999999999},"min":{"1min":0.80600000000000005,"5min":0.71499999999999997,"15min":0.68799999999999994},"max":{"1min":2.2080000000000002,"5min":5.4660000000000002,"15min":37.896999999999998},"last":1.8080000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:59:44 2021","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":1.3049999999999999,"15min":1.4399999999999999},"min":{"1min":1.048,"5min":0.754,"15min":0.55500000000000005},"max":{"1min":2.8919999999999999,"5min":5.4160000000000004,"15min":11.897},"last":1.667},{"interface":"front","average":{"1min":1.472,"5min":1.4610000000000001,"15min":1.5229999999999999},"min":{"1min":0.89700000000000002,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":2.1080000000000001,"5min":5.0579999999999998,"15min":6.0369999999999999},"last":1.8400000000000001}]}]}]}} 2021-02-10T05:00:06.866 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2021-02-10T05:00:06.866 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2021-02-10T05:00:07.107 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:00:07.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:06 smithi164 conmon[29288]: cluster 2021-02-10T05:00:05.286876+0000 osd.2 (osd.2 2021-02-10T05:00:07.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:06 smithi164 conmon[29288]: ) 2047 : cluster [DBG] 2.c scrub ok 2021-02-10T05:00:07.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:06 smithi164 conmon[29288]: audit 2021-02-10T05:00:05.497718+0000 mgr.y (mgr.14140) 1549 : audit [DBG] from='client.16683 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T05:00:07.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:06 smithi164 conmon[29288]: cluster 2021-02-10T05:00:05.604037+0000 osd.7 (osd 2021-02-10T05:00:07.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:06 smithi164 conmon[29288]: .7) 105 : cluster [DBG] 5.1a scrub ok 2021-02-10T05:00:07.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:06 smithi164 conmon[29288]: cluster 2021-02-10T05:00:07.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:06 smithi164 conmon[29288]: 2021-02-10T05:00: 2021-02-10T05:00:07.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:06 smithi164 conmon[29288]: 05.680310+0000 osd.1 (osd 2021-02-10T05:00:07.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:06 smithi164 conmon[29288]: .1) 530 : cluster [DBG] 2021-02-10T05:00:07.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:06 smithi164 conmon[29288]: 4.7 scrub ok 2021-02-10T05:00:07.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:06 smithi164 conmon[29288]: cluster 2021-02-10T05:00:07.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:06 smithi164 conmon[29288]: 2021-02-10T05:00: 2021-02-10T05:00:07.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:06 smithi164 conmon[29288]: 06.153621+0000 osd.4 (osd. 2021-02-10T05:00:07.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:06 smithi164 conmon[29288]: 4) 2676 : cluster [DBG] 2021-02-10T05:00:07.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:06 smithi164 conmon[29288]: 7.22 deep-scrub ok 2021-02-10T05:00:07.722 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is set 2021-02-10T05:00:07.887 INFO:teuthology.orchestra.run.smithi085.stdout: 2021-02-10T05:00:07.891 INFO:teuthology.orchestra.run.smithi085.stderr:dumped all 2021-02-10T05:00:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:07 smithi164 conmon[29288]: cluster 2021-02-10T05:00:06.290948+0000 2021-02-10T05:00:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:07 smithi164 conmon[29288]: osd.2 (osd.2) 2048 : cluster [DBG] 5.d scrub ok 2021-02-10T05:00:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:07 smithi164 conmon[29288]: cluster 2021-02-10T05:00:06.578069+0000 mgr.y (mgr.14140) 1550 : cluster [DBG] pgmap v2519: 208 pgs: 7 active+remapped+backfill_toofull, 201 active+clean; 856 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 120/2166 objects misplaced (5.540%) 2021-02-10T05:00:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:07 smithi164 conmon[29288]: cluster 2021-02-10T05:00:06.841592+0000 osd.3 (osd.3) 53 : cluster [DBG] 2.1e scrub ok 2021-02-10T05:00:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:07 smithi164 conmon[29288]: cluster 2021-02-10T05:00:07.026791+0000 2021-02-10T05:00:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:07 smithi164 conmon[29288]: osd.0 (osd.0) 12 : cluster [DBG] 5.5 scrub ok 2021-02-10T05:00:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:07 smithi164 conmon[29288]: audit 2021-02-10T05:00:07.424072+0000 mon.a (mon 2021-02-10T05:00:08.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:07 smithi164 conmon[29288]: .0) 3078 : audit [INF] from='client.? 172.21.15.85:0/1175869320' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T05:00:08.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:07 smithi085 conmon[29319]: cluster 2021-02-10T05:00:06.290948+0000 osd.2 (osd.2) 2048 : cluster [DBG] 5.d scrub ok 2021-02-10T05:00:08.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:07 smithi085 conmon[29319]: cluster 2021-02-10T05:00:06.578069+0000 mgr.y (mgr.14140) 1550 : cluster [DBG] pgmap v2519: 208 pgs: 7 active+remapped+backfill_toofull, 201 active+clean; 856 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 120/2166 objects misplaced (5.540%) 2021-02-10T05:00:08.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:07 smithi085 conmon[29319]: cluster 2021-02-10T05:00:06.841592+0000 osd.3 (osd.3) 53 : cluster [DBG] 2.1e scrub ok 2021-02-10T05:00:08.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:07 smithi085 conmon[29319]: cluster 2021-02-10T05:00:07.026791+0000 osd.0 (osd.0) 12 : cluster [DBG] 5.5 scrub ok 2021-02-10T05:00:08.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:07 smithi085 conmon[29319]: audit 2021-02-10T05:00:07.424072+0000 mon.a (mon.0) 2021-02-10T05:00:08.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:07 smithi085 conmon[29319]: 3078 : audit [INF] from='client.? 172.21.15.85:0/1175869320' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T05:00:08.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:07 smithi085 conmon[35961]: cluster 2021-02-10T05:00:06.290948+0000 osd.2 2021-02-10T05:00:08.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:07 smithi085 conmon[35961]: (osd.2) 2048 : cluster [DBG] 5.d scrub ok 2021-02-10T05:00:08.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:07 smithi085 conmon[35961]: cluster 2021-02-10T05:00:06.578069+0000 mgr.y (mgr.14140 2021-02-10T05:00:08.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:07 smithi085 conmon[35961]: ) 1550 : cluster [DBG] pgmap v2519: 208 pgs: 7 active+remapped+backfill_toofull, 201 active+clean; 856 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 120/2166 objects misplaced (5.540%) 2021-02-10T05:00:08.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:07 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T05:00:08.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:07 smithi085 conmon[35961]: 05:00:06.841592+0000 osd.3 (osd.3) 53 : cluster [DBG] 2.1e scrub ok 2021-02-10T05:00:08.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:07 smithi085 conmon[35961]: 2021-02-10T05:00:08.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:07 smithi085 conmon[35961]: cluster 2021-02-10T05:00:07. 2021-02-10T05:00:08.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:07 smithi085 conmon[35961]: 026791+0000 osd.0 (osd.0) 12 : cluster 2021-02-10T05:00:08.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:07 smithi085 conmon[35961]: [DBG] 5.5 scrub ok 2021-02-10T05:00:08.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:07 smithi085 conmon[35961]: audit 2021-02-10T05:00:07. 2021-02-10T05:00:08.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:07 smithi085 conmon[35961]: 424072+0000 mon.a (mon.0) 3078 : audit 2021-02-10T05:00:08.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:07 smithi085 conmon[35961]: [INF] from='client.? 172.21.15.85:0/1175869320' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-02-10T05:00:08.551 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-02-10T05:00:08.788 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:00:08.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:08 smithi085 conmon[29319]: cluster 2021-02-10T05:00:07.711243+0000 mon.a (mon 2021-02-10T05:00:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:08 smithi085 conmon[29319]: .0) 3079 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T05:00:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:08 smithi085 conmon[29319]: audit 2021-02-10T05:00:07.715091+0000 mon.a (mon.0) 3080 : audit [INF] from='client.? 172.21.15.85:0/1175869320' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T05:00:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:08 smithi085 conmon[29319]: cluster 2021-02-10T05:00:07.715340+0000 mon.a (mon.0) 3081 : cluster [DBG] osdmap e1376: 8 total, 8 up, 7 in 2021-02-10T05:00:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:08 smithi085 conmon[29319]: audit 2021-02-10T05:00:07.863440+0000 mgr.y (mgr.14140) 1551 : audit [DBG] from='client.26590 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T05:00:08.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:08 smithi085 conmon[29319]: cluster 2021-02-10T05:00:08.649890+0000 mon.a (mon.0) 3082 : cluster [DBG] osdmap e1377: 8 total, 8 up, 7 in 2021-02-10T05:00:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:08 smithi085 conmon[35961]: cluster 2021-02-10T05:00:07.711243+0000 2021-02-10T05:00:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:08 smithi085 conmon[35961]: mon.a (mon.0) 3079 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T05:00:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:08 smithi085 conmon[35961]: audit 2021-02-10T05:00:07.715091+0000 mon.a (mon.0) 3080 : audit [INF] from='client.? 172.21.15.85:0/1175869320' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T05:00:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:08 smithi085 conmon[35961]: cluster 2021-02-10T05:00:07.715340+0000 mon.a (mon.0) 3081 : cluster [DBG] osdmap e1376: 8 total, 8 up, 7 in 2021-02-10T05:00:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:08 smithi085 conmon[35961]: audit 2021-02-10T05:00:07.863440+0000 mgr.y (mgr.14140) 1551 : audit [DBG] from='client.26590 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T05:00:08.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:08 smithi085 conmon[35961]: cluster 2021-02-10T05:00:08.649890+0000 mon.a (mon.0) 3082 : cluster [DBG] osdmap e1377: 8 total, 8 up, 7 in 2021-02-10T05:00:09.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:08 smithi164 conmon[29288]: cluster 2021-02-10T05:00:07.711243+0000 mon.a 2021-02-10T05:00:09.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:08 smithi164 conmon[29288]: (mon.0) 3079 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T05:00:09.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:08 smithi164 conmon[29288]: audit 2021-02-10T05:00:07.715091+0000 mon.a (mon.0) 3080 : audit [INF] from='client.? 172.21.15.85:0/1175869320' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-02-10T05:00:09.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:08 smithi164 conmon[29288]: cluster 2021-02-10T05:00:07.715340+0000 mon.a (mon.0) 3081 : cluster [DBG] osdmap e1376: 8 total, 8 up, 7 in 2021-02-10T05:00:09.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:08 smithi164 conmon[29288]: audit 2021-02-10T05:00:07.863440+0000 mgr.y 2021-02-10T05:00:09.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:08 smithi164 conmon[29288]: (mgr.14140) 1551 : audit [DBG] from='client.26590 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-02-10T05:00:09.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:08 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T05:00:09.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:08 smithi164 conmon[29288]: 05:00:08.649890+0000 mon.a (mon.0) 3082 : cluster [DBG] 2021-02-10T05:00:09.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:08 smithi164 conmon[29288]: osdmap e1377: 8 total, 8 up, 7 in 2021-02-10T05:00:09.855 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:09 smithi085 conmon[29319]: cluster 2021-02-10T05:00:08.578648 2021-02-10T05:00:09.855 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:09 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 1552 : cluster [DBG] pgmap v2521: 208 pgs: 7 active+remapped+backfill_toofull, 201 active+clean; 844 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 120/2157 objects misplaced (5.563%) 2021-02-10T05:00:09.855 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:09 smithi085 conmon[35961]: cluster 2021-02-10T05:00:08. 2021-02-10T05:00:09.856 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:09 smithi085 conmon[35961]: 578648+0000 mgr.y (mgr.14140) 1552 : cluster [DBG] pgmap v2521: 208 pgs: 7 active+remapped+backfill_toofull, 201 active+clean; 844 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 120/2157 objects misplaced (5.563%) 2021-02-10T05:00:09.894 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 05:00:09 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T05:00:09.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:09 smithi164 conmon[29288]: cluster 2021-02-10T05:00:08.578648+0000 mgr.y ( 2021-02-10T05:00:09.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:09 smithi164 conmon[29288]: mgr.14140) 1552 : cluster [DBG] pgmap v2521: 208 pgs: 7 active+remapped+backfill_toofull, 201 active+clean; 844 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 120/2157 objects misplaced (5.563%) 2021-02-10T05:00:09.952 INFO:teuthology.orchestra.run.smithi085.stdout:{"pg_ready":true,"pg_map":{"version":2520,"stamp":"2021-02-10T05:00:07.712637+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":885416303,"num_objects":717,"num_object_clones":310,"num_object_copies":2151,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":120,"num_objects_unfound":0,"num_objects_dirty":717,"num_whiteouts":9,"num_read":8372,"num_read_kb":5100607,"num_write":5252,"num_write_kb":1397300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1599,"num_bytes_recovered":571056556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35867,"ondisk_log_size":35867,"up":624,"acting":624,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":633,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2281832,"kb_used_data":1979568,"kb_used_omap":711,"kb_used_meta":301496,"kb_avail":653934232,"statfs":{"total":671965249536,"available":669628653568,"internally_reserved":0,"allocated":2027077632,"data_stored":2538084021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":728429,"internal_metadata":308732563},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15397241,"num_objects":-6,"num_object_clones":-6,"num_object_copies":-18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6,"num_whiteouts":-1,"num_read":-115,"num_read_kb":-124697,"num_write":-77,"num_write_kb":-25052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-12,"num_bytes_recovered":-4145029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.020704"},"pg_stats":[{"pgid":"6.29","version":"0'0","reported_seq":"1246","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.805370+0000","last_change":"2021-02-10T04:58:56.856970+0000","last_active":"2021-02-10T05:00:02.805370+0000","last_peered":"2021-02-10T05:00:02.805370+0000","last_clean":"2021-02-10T05:00:02.805370+0000","last_became_active":"2021-02-10T04:58:56.856652+0000","last_became_peered":"2021-02-10T04:58:56.856652+0000","last_unstale":"2021-02-10T05:00:02.805370+0000","last_undegraded":"2021-02-10T05:00:02.805370+0000","last_fullsized":"2021-02-10T05:00:02.805370+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:24.121684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:24.121684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":"280","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705003+0000","last_change":"2021-02-10T04:59:51.125621+0000","last_active":"2021-02-10T05:00:02.705003+0000","last_peered":"2021-02-10T05:00:02.705003+0000","last_clean":"2021-02-10T05:00:02.705003+0000","last_became_active":"2021-02-10T04:59:51.125212+0000","last_became_peered":"2021-02-10T04:59:51.125212+0000","last_unstale":"2021-02-10T05:00:02.705003+0000","last_undegraded":"2021-02-10T05:00:02.705003+0000","last_fullsized":"2021-02-10T05:00:02.705003+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.133043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.133043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"1365'5","reported_seq":"419","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:03.088378+0000","last_change":"2021-02-10T05:00:02.659739+0000","last_active":"2021-02-10T05:00:03.088378+0000","last_peered":"2021-02-10T05:00:03.088378+0000","last_clean":"2021-02-10T05:00:03.088378+0000","last_became_active":"2021-02-10T04:59:51.116307+0000","last_became_peered":"2021-02-10T04:59:51.116307+0000","last_unstale":"2021-02-10T05:00:03.088378+0000","last_undegraded":"2021-02-10T05:00:03.088378+0000","last_fullsized":"2021-02-10T05:00:03.088378+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"1365'5","last_scrub_stamp":"2021-02-10T05:00:02.659621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T05:00:02.659621+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"1261'84","reported_seq":"1563","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.701900+0000","last_change":"2021-02-10T04:59:53.658456+0000","last_active":"2021-02-10T05:00:00.701900+0000","last_peered":"2021-02-10T05:00:00.701900+0000","last_clean":"2021-02-10T05:00:00.701900+0000","last_became_active":"2021-02-10T04:59:53.657939+0000","last_became_peered":"2021-02-10T04:59:53.657939+0000","last_unstale":"2021-02-10T05:00:00.701900+0000","last_undegraded":"2021-02-10T05:00:00.701900+0000","last_fullsized":"2021-02-10T05:00:00.701900+0000","mapping_epoch":1366,"log_start":"1132'66","ondisk_log_start":"1132'66","created":55,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"1132'66","last_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_deep_scrub":"1132'66","last_deep_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.430399+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":808,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":93,"num_read_kb":75,"num_write":53,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"53'2","reported_seq":"280","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705692+0000","last_change":"2021-02-10T04:53:41.722609+0000","last_active":"2021-02-10T05:00:02.705692+0000","last_peered":"2021-02-10T05:00:02.705692+0000","last_clean":"2021-02-10T05:00:02.705692+0000","last_became_active":"2021-02-10T04:51:36.875000+0000","last_became_peered":"2021-02-10T04:51:36.875000+0000","last_unstale":"2021-02-10T05:00:02.705692+0000","last_undegraded":"2021-02-10T05:00:02.705692+0000","last_fullsized":"2021-02-10T05:00:02.705692+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:53:41.722516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:41.722516+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":391,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"1460","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.707467+0000","last_change":"2021-02-10T04:59:51.117362+0000","last_active":"2021-02-10T05:00:00.707467+0000","last_peered":"2021-02-10T05:00:00.707467+0000","last_clean":"2021-02-10T05:00:00.707467+0000","last_became_active":"2021-02-10T04:59:51.116877+0000","last_became_peered":"2021-02-10T04:59:51.116877+0000","last_unstale":"2021-02-10T05:00:00.707467+0000","last_undegraded":"2021-02-10T05:00:00.707467+0000","last_fullsized":"2021-02-10T05:00:00.707467+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:48.408748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.006716+0000","last_clean_scrub_stamp":"2021-02-10T04:50:48.408748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"1003'1689","reported_seq":"5046","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:59.694219+0000","last_change":"2021-02-10T04:59:58.577583+0000","last_active":"2021-02-10T04:59:59.694219+0000","last_peered":"2021-02-10T04:59:59.694219+0000","last_clean":"2021-02-10T04:59:59.694219+0000","last_became_active":"2021-02-10T04:59:54.659936+0000","last_became_peered":"2021-02-10T04:59:54.659936+0000","last_unstale":"2021-02-10T04:59:59.694219+0000","last_undegraded":"2021-02-10T04:59:59.694219+0000","last_fullsized":"2021-02-10T04:59:59.694219+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:59:58.577489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:59:58.577489+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":99361981,"num_objects":37,"num_object_clones":33,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":76107542,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"62'5","reported_seq":"513","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:02.127511+0000","last_change":"2021-02-10T04:59:51.183782+0000","last_active":"2021-02-10T05:00:02.127511+0000","last_peered":"2021-02-10T05:00:02.127511+0000","last_clean":"2021-02-10T05:00:02.127511+0000","last_became_active":"2021-02-10T04:59:51.113091+0000","last_became_peered":"2021-02-10T04:59:51.113091+0000","last_unstale":"2021-02-10T05:00:02.127511+0000","last_undegraded":"2021-02-10T05:00:02.127511+0000","last_fullsized":"2021-02-10T05:00:02.127511+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":472,"num_read_kb":471,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"1261'28","reported_seq":"1378","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.805836+0000","last_change":"2021-02-10T04:59:13.926028+0000","last_active":"2021-02-10T05:00:02.805836+0000","last_peered":"2021-02-10T05:00:02.805836+0000","last_clean":"2021-02-10T05:00:02.805836+0000","last_became_active":"2021-02-10T04:59:13.925333+0000","last_became_peered":"2021-02-10T04:59:13.925333+0000","last_unstale":"2021-02-10T05:00:02.805836+0000","last_undegraded":"2021-02-10T05:00:02.805836+0000","last_fullsized":"2021-02-10T05:00:02.805836+0000","mapping_epoch":1332,"log_start":"1061'20","ondisk_log_start":"1061'20","created":55,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'32","last_scrub_stamp":"2021-02-10T04:51:51.382040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.382040+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":579,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":29,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1026,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"825","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705937+0000","last_change":"2021-02-10T04:59:40.662477+0000","last_active":"2021-02-10T05:00:02.705937+0000","last_peered":"2021-02-10T05:00:02.705937+0000","last_clean":"2021-02-10T05:00:02.705937+0000","last_became_active":"2021-02-10T04:59:40.661863+0000","last_became_peered":"2021-02-10T04:59:40.661863+0000","last_unstale":"2021-02-10T05:00:02.705937+0000","last_undegraded":"2021-02-10T05:00:02.705937+0000","last_fullsized":"2021-02-10T05:00:02.705937+0000","mapping_epoch":1356,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:34.178949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.178949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"276","reported_epoch":"1375","state":"active+clean","last_fresh":"2021-02-10T05:00:04.256127+0000","last_change":"2021-02-10T05:00:04.256127+0000","last_active":"2021-02-10T05:00:04.256127+0000","last_peered":"2021-02-10T05:00:04.256127+0000","last_clean":"2021-02-10T05:00:04.256127+0000","last_became_active":"2021-02-10T04:51:36.855507+0000","last_became_peered":"2021-02-10T04:51:36.855507+0000","last_unstale":"2021-02-10T05:00:04.256127+0000","last_undegraded":"2021-02-10T05:00:04.256127+0000","last_fullsized":"2021-02-10T05:00:04.256127+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T05:00:04.256062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T05:00:04.256062+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"1290","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.805149+0000","last_change":"2021-02-10T04:59:08.662159+0000","last_active":"2021-02-10T05:00:02.805149+0000","last_peered":"2021-02-10T05:00:02.805149+0000","last_clean":"2021-02-10T05:00:02.805149+0000","last_became_active":"2021-02-10T04:58:56.865590+0000","last_became_peered":"2021-02-10T04:58:56.865590+0000","last_unstale":"2021-02-10T05:00:02.805149+0000","last_undegraded":"2021-02-10T05:00:02.805149+0000","last_fullsized":"2021-02-10T05:00:02.805149+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:58:57.081143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:58:57.081143+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"1168'82","reported_seq":"307","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705111+0000","last_change":"2021-02-10T04:53:23.326363+0000","last_active":"2021-02-10T05:00:02.705111+0000","last_peered":"2021-02-10T05:00:02.705111+0000","last_clean":"2021-02-10T05:00:02.705111+0000","last_became_active":"2021-02-10T04:51:36.859215+0000","last_became_peered":"2021-02-10T04:51:36.859215+0000","last_unstale":"2021-02-10T05:00:02.705111+0000","last_undegraded":"2021-02-10T05:00:02.705111+0000","last_fullsized":"2021-02-10T05:00:02.705111+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1150'76","last_scrub_stamp":"2021-02-10T04:53:00.964499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:00.964499+0000","log_size":82,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3848803,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"1261'69","reported_seq":"1460","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.707285+0000","last_change":"2021-02-10T04:59:53.656353+0000","last_active":"2021-02-10T05:00:00.707285+0000","last_peered":"2021-02-10T05:00:00.707285+0000","last_clean":"2021-02-10T05:00:00.707285+0000","last_became_active":"2021-02-10T04:59:53.655770+0000","last_became_peered":"2021-02-10T04:59:53.655770+0000","last_unstale":"2021-02-10T05:00:00.707285+0000","last_undegraded":"2021-02-10T05:00:00.707285+0000","last_fullsized":"2021-02-10T05:00:00.707285+0000","mapping_epoch":1366,"log_start":"1112'47","ondisk_log_start":"1112'47","created":55,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'61","last_scrub_stamp":"2021-02-10T04:53:36.836202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:36.836202+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":733,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":103,"num_read_kb":85,"num_write":61,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1099,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"1334'8","reported_seq":"415","reported_epoch":"1375","state":"active+clean","last_fresh":"2021-02-10T05:00:05.286987+0000","last_change":"2021-02-10T05:00:05.286942+0000","last_active":"2021-02-10T05:00:05.286987+0000","last_peered":"2021-02-10T05:00:05.286987+0000","last_clean":"2021-02-10T05:00:05.286987+0000","last_became_active":"2021-02-10T04:58:56.879760+0000","last_became_peered":"2021-02-10T04:58:56.879760+0000","last_unstale":"2021-02-10T05:00:05.286987+0000","last_undegraded":"2021-02-10T05:00:05.286987+0000","last_fullsized":"2021-02-10T05:00:05.286987+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1334'8","last_scrub_stamp":"2021-02-10T05:00:05.286880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T05:00:05.286880+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"301","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.704944+0000","last_change":"2021-02-10T04:58:56.964109+0000","last_active":"2021-02-10T05:00:02.704944+0000","last_peered":"2021-02-10T05:00:02.704944+0000","last_clean":"2021-02-10T05:00:02.704944+0000","last_became_active":"2021-02-10T04:58:56.963768+0000","last_became_peered":"2021-02-10T04:58:56.963768+0000","last_unstale":"2021-02-10T05:00:02.704944+0000","last_undegraded":"2021-02-10T05:00:02.704944+0000","last_fullsized":"2021-02-10T05:00:02.704944+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:55:22.573621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:55:22.573621+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"671","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.804835+0000","last_change":"2021-02-10T04:53:46.733236+0000","last_active":"2021-02-10T05:00:02.804835+0000","last_peered":"2021-02-10T05:00:02.804835+0000","last_clean":"2021-02-10T05:00:02.804835+0000","last_became_active":"2021-02-10T04:53:46.732890+0000","last_became_peered":"2021-02-10T04:53:46.732890+0000","last_unstale":"2021-02-10T05:00:02.804835+0000","last_undegraded":"2021-02-10T05:00:02.804835+0000","last_fullsized":"2021-02-10T05:00:02.804835+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:35.170157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:35.170157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"1003'739","reported_seq":"3964","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.805274+0000","last_change":"2021-02-10T04:59:08.649874+0000","last_active":"2021-02-10T05:00:02.805274+0000","last_peered":"2021-02-10T05:00:02.805274+0000","last_clean":"2021-02-10T05:00:02.805274+0000","last_became_active":"2021-02-10T04:58:56.956888+0000","last_became_peered":"2021-02-10T04:58:56.956888+0000","last_unstale":"2021-02-10T05:00:02.805274+0000","last_undegraded":"2021-02-10T05:00:02.805274+0000","last_fullsized":"2021-02-10T05:00:02.805274+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:52:27.602006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:27.602006+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24059888,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":129,"num_read_kb":109544,"num_write":102,"num_write_kb":46173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"1202","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705221+0000","last_change":"2021-02-10T04:59:51.125208+0000","last_active":"2021-02-10T05:00:02.705221+0000","last_peered":"2021-02-10T05:00:02.705221+0000","last_clean":"2021-02-10T05:00:02.705221+0000","last_became_active":"2021-02-10T04:59:51.124785+0000","last_became_peered":"2021-02-10T04:59:51.124785+0000","last_unstale":"2021-02-10T05:00:02.705221+0000","last_undegraded":"2021-02-10T05:00:02.705221+0000","last_fullsized":"2021-02-10T05:00:02.705221+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"1261'64","reported_seq":"1569","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.706486+0000","last_change":"2021-02-10T04:59:13.932815+0000","last_active":"2021-02-10T05:00:02.706486+0000","last_peered":"2021-02-10T05:00:02.706486+0000","last_clean":"2021-02-10T05:00:02.706486+0000","last_became_active":"2021-02-10T04:59:13.931863+0000","last_became_peered":"2021-02-10T04:59:13.931863+0000","last_unstale":"2021-02-10T05:00:02.706486+0000","last_undegraded":"2021-02-10T05:00:02.706486+0000","last_fullsized":"2021-02-10T05:00:02.706486+0000","mapping_epoch":1118,"log_start":"1066'44","ondisk_log_start":"1066'44","created":55,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":0,"last_scrub":"1066'44","last_scrub_stamp":"2021-02-10T04:53:37.108677+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:53:37.108677+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":789,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":90,"num_read_kb":72,"num_write":53,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"1269","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705612+0000","last_change":"2021-02-10T04:53:39.126356+0000","last_active":"2021-02-10T05:00:02.705612+0000","last_peered":"2021-02-10T05:00:02.705612+0000","last_clean":"2021-02-10T05:00:02.705612+0000","last_became_active":"2021-02-10T04:29:07.145616+0000","last_became_peered":"2021-02-10T04:29:07.145616+0000","last_unstale":"2021-02-10T05:00:02.705612+0000","last_undegraded":"2021-02-10T05:00:02.705612+0000","last_fullsized":"2021-02-10T05:00:02.705612+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:39.126261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:39.126261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"805","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705351+0000","last_change":"2021-02-10T04:59:40.660733+0000","last_active":"2021-02-10T05:00:02.705351+0000","last_peered":"2021-02-10T05:00:02.705351+0000","last_clean":"2021-02-10T05:00:02.705351+0000","last_became_active":"2021-02-10T04:59:40.660383+0000","last_became_peered":"2021-02-10T04:59:40.660383+0000","last_unstale":"2021-02-10T05:00:02.705351+0000","last_undegraded":"2021-02-10T05:00:02.705351+0000","last_fullsized":"2021-02-10T05:00:02.705351+0000","mapping_epoch":1356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:26.311958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.311958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"322","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:03.253366+0000","last_change":"2021-02-10T05:00:03.253273+0000","last_active":"2021-02-10T05:00:03.253366+0000","last_peered":"2021-02-10T05:00:03.253366+0000","last_clean":"2021-02-10T05:00:03.253366+0000","last_became_active":"2021-02-10T04:51:36.855744+0000","last_became_peered":"2021-02-10T04:51:36.855744+0000","last_unstale":"2021-02-10T05:00:03.253366+0000","last_undegraded":"2021-02-10T05:00:03.253366+0000","last_fullsized":"2021-02-10T05:00:03.253366+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T05:00:03.253161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:13.727241+0000","last_clean_scrub_stamp":"2021-02-10T05:00:03.253161+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"1063'654","reported_seq":"4565","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705652+0000","last_change":"2021-02-10T04:58:56.966357+0000","last_active":"2021-02-10T05:00:02.705652+0000","last_peered":"2021-02-10T05:00:02.705652+0000","last_clean":"2021-02-10T05:00:02.705652+0000","last_became_active":"2021-02-10T04:58:56.965471+0000","last_became_peered":"2021-02-10T04:58:56.965471+0000","last_unstale":"2021-02-10T05:00:02.705652+0000","last_undegraded":"2021-02-10T05:00:02.705652+0000","last_fullsized":"2021-02-10T05:00:02.705652+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:51:57.121838+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.121838+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24700630,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":171,"num_read_kb":182272,"num_write":111,"num_write_kb":33727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":26349023,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"1319","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.701989+0000","last_change":"2021-02-10T04:51:50.433056+0000","last_active":"2021-02-10T05:00:00.701989+0000","last_peered":"2021-02-10T05:00:00.701989+0000","last_clean":"2021-02-10T05:00:00.701989+0000","last_became_active":"2021-02-10T04:51:36.843165+0000","last_became_peered":"2021-02-10T04:51:36.843165+0000","last_unstale":"2021-02-10T05:00:00.701989+0000","last_undegraded":"2021-02-10T05:00:00.701989+0000","last_fullsized":"2021-02-10T05:00:00.701989+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.432990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.432990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"1356","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.704060+0000","last_change":"2021-02-10T04:58:56.965752+0000","last_active":"2021-02-10T05:00:02.704060+0000","last_peered":"2021-02-10T05:00:02.704060+0000","last_clean":"2021-02-10T05:00:02.704060+0000","last_became_active":"2021-02-10T04:58:56.963934+0000","last_became_peered":"2021-02-10T04:58:56.963934+0000","last_unstale":"2021-02-10T05:00:02.704060+0000","last_undegraded":"2021-02-10T05:00:02.704060+0000","last_fullsized":"2021-02-10T05:00:02.704060+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"1266","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.804828+0000","last_change":"2021-02-10T04:53:46.727888+0000","last_active":"2021-02-10T05:00:02.804828+0000","last_peered":"2021-02-10T05:00:02.804828+0000","last_clean":"2021-02-10T05:00:02.804828+0000","last_became_active":"2021-02-10T04:53:46.727670+0000","last_became_peered":"2021-02-10T04:53:46.727670+0000","last_unstale":"2021-02-10T05:00:02.804828+0000","last_undegraded":"2021-02-10T05:00:02.804828+0000","last_fullsized":"2021-02-10T05:00:02.804828+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:41.041553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.041553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"1261'76","reported_seq":"1505","reported_epoch":"1374","state":"active+remapped+backfill_toofull","last_fresh":"2021-02-10T05:00:02.705799+0000","last_change":"2021-02-10T04:59:52.122307+0000","last_active":"2021-02-10T05:00:02.705799+0000","last_peered":"2021-02-10T05:00:02.705799+0000","last_clean":"2021-02-10T04:59:49.068531+0000","last_became_active":"2021-02-10T04:59:52.104080+0000","last_became_peered":"2021-02-10T04:59:52.104080+0000","last_unstale":"2021-02-10T05:00:02.705799+0000","last_undegraded":"2021-02-10T05:00:02.705799+0000","last_fullsized":"2021-02-10T05:00:02.705799+0000","mapping_epoch":1364,"log_start":"1049'39","ondisk_log_start":"1049'39","created":55,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":0,"last_scrub":"1187'54","last_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_deep_scrub":"1187'54","last_deep_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_clean_scrub_stamp":"2021-02-10T04:53:42.071103+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":289,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":73,"num_read_kb":63,"num_write":47,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,6],"avail_no_missing":["4","3","6"],"object_location_counts":[{"shards":"3,4,6","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"806","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.702935+0000","last_change":"2021-02-10T04:59:40.658734+0000","last_active":"2021-02-10T05:00:02.702935+0000","last_peered":"2021-02-10T05:00:02.702935+0000","last_clean":"2021-02-10T05:00:02.702935+0000","last_became_active":"2021-02-10T04:59:40.658301+0000","last_became_peered":"2021-02-10T04:59:40.658301+0000","last_unstale":"2021-02-10T05:00:02.702935+0000","last_undegraded":"2021-02-10T05:00:02.702935+0000","last_fullsized":"2021-02-10T05:00:02.702935+0000","mapping_epoch":1356,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:52.350706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.350706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"1119'4","reported_seq":"378","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:03.088189+0000","last_change":"2021-02-10T04:52:27.126670+0000","last_active":"2021-02-10T05:00:03.088189+0000","last_peered":"2021-02-10T05:00:03.088189+0000","last_clean":"2021-02-10T05:00:03.088189+0000","last_became_active":"2021-02-10T04:51:36.853647+0000","last_became_peered":"2021-02-10T04:51:36.853647+0000","last_unstale":"2021-02-10T05:00:03.088189+0000","last_undegraded":"2021-02-10T05:00:03.088189+0000","last_fullsized":"2021-02-10T05:00:03.088189+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'4","last_scrub_stamp":"2021-02-10T04:52:27.126599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:27.126599+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"1360","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.706304+0000","last_change":"2021-02-10T04:52:25.745660+0000","last_active":"2021-02-10T05:00:02.706304+0000","last_peered":"2021-02-10T05:00:02.706304+0000","last_clean":"2021-02-10T05:00:02.706304+0000","last_became_active":"2021-02-10T04:51:36.849503+0000","last_became_peered":"2021-02-10T04:51:36.849503+0000","last_unstale":"2021-02-10T05:00:02.706304+0000","last_undegraded":"2021-02-10T05:00:02.706304+0000","last_fullsized":"2021-02-10T05:00:02.706304+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:25.745577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.745577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"1004'1093","reported_seq":"6547","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.806756+0000","last_change":"2021-02-10T04:59:08.653144+0000","last_active":"2021-02-10T05:00:02.806756+0000","last_peered":"2021-02-10T05:00:02.806756+0000","last_clean":"2021-02-10T05:00:02.806756+0000","last_became_active":"2021-02-10T04:58:56.875893+0000","last_became_peered":"2021-02-10T04:58:56.875893+0000","last_unstale":"2021-02-10T05:00:02.806756+0000","last_undegraded":"2021-02-10T05:00:02.806756+0000","last_fullsized":"2021-02-10T05:00:02.806756+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39434696,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":279,"num_read_kb":311058,"num_write":185,"num_write_kb":66790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"62'5","reported_seq":"173","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.702082+0000","last_change":"2021-02-10T04:54:39.559473+0000","last_active":"2021-02-10T05:00:02.702082+0000","last_peered":"2021-02-10T05:00:02.702082+0000","last_clean":"2021-02-10T05:00:02.702082+0000","last_became_active":"2021-02-10T04:54:39.559048+0000","last_became_peered":"2021-02-10T04:54:39.559048+0000","last_unstale":"2021-02-10T05:00:02.702082+0000","last_undegraded":"2021-02-10T05:00:02.702082+0000","last_fullsized":"2021-02-10T05:00:02.702082+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":149,"num_read_kb":148,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"1353","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.707362+0000","last_change":"2021-02-10T04:59:51.119011+0000","last_active":"2021-02-10T05:00:00.707362+0000","last_peered":"2021-02-10T05:00:00.707362+0000","last_clean":"2021-02-10T05:00:00.707362+0000","last_became_active":"2021-02-10T04:59:51.118611+0000","last_became_peered":"2021-02-10T04:59:51.118611+0000","last_unstale":"2021-02-10T05:00:00.707362+0000","last_undegraded":"2021-02-10T05:00:00.707362+0000","last_fullsized":"2021-02-10T05:00:00.707362+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:12.389272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:12.389272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"1261'71","reported_seq":"1085","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:59.694065+0000","last_change":"2021-02-10T04:59:13.930947+0000","last_active":"2021-02-10T04:59:59.694065+0000","last_peered":"2021-02-10T04:59:59.694065+0000","last_clean":"2021-02-10T04:59:59.694065+0000","last_became_active":"2021-02-10T04:59:13.930554+0000","last_became_peered":"2021-02-10T04:59:13.930554+0000","last_unstale":"2021-02-10T04:59:59.694065+0000","last_undegraded":"2021-02-10T04:59:59.694065+0000","last_fullsized":"2021-02-10T04:59:59.694065+0000","mapping_epoch":1129,"log_start":"1049'51","ondisk_log_start":"1049'51","created":55,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'41","last_scrub_stamp":"2021-02-10T04:51:58.353660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:58.353660+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":68,"num_write":51,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":525,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"807","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.805745+0000","last_change":"2021-02-10T04:59:40.660482+0000","last_active":"2021-02-10T05:00:02.805745+0000","last_peered":"2021-02-10T05:00:02.805745+0000","last_clean":"2021-02-10T05:00:02.805745+0000","last_became_active":"2021-02-10T04:59:40.660098+0000","last_became_peered":"2021-02-10T04:59:40.660098+0000","last_unstale":"2021-02-10T05:00:02.805745+0000","last_undegraded":"2021-02-10T05:00:02.805745+0000","last_fullsized":"2021-02-10T05:00:02.805745+0000","mapping_epoch":1356,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:11.271459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:39.128781+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.271459+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"1333'7","reported_seq":"1790","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:03.088217+0000","last_change":"2021-02-10T04:58:56.875943+0000","last_active":"2021-02-10T05:00:03.088217+0000","last_peered":"2021-02-10T05:00:03.088217+0000","last_clean":"2021-02-10T05:00:03.088217+0000","last_became_active":"2021-02-10T04:58:56.875512+0000","last_became_peered":"2021-02-10T04:58:56.875512+0000","last_unstale":"2021-02-10T05:00:03.088217+0000","last_undegraded":"2021-02-10T05:00:03.088217+0000","last_fullsized":"2021-02-10T05:00:03.088217+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'3","last_scrub_stamp":"2021-02-10T04:51:47.855283+0000","last_deep_scrub":"78'3","last_deep_scrub_stamp":"2021-02-10T04:27:13.268412+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.855283+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"1378","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.706563+0000","last_change":"2021-02-10T04:58:56.854797+0000","last_active":"2021-02-10T05:00:00.706563+0000","last_peered":"2021-02-10T05:00:00.706563+0000","last_clean":"2021-02-10T05:00:00.706563+0000","last_became_active":"2021-02-10T04:58:56.854440+0000","last_became_peered":"2021-02-10T04:58:56.854440+0000","last_unstale":"2021-02-10T05:00:00.706563+0000","last_undegraded":"2021-02-10T05:00:00.706563+0000","last_fullsized":"2021-02-10T05:00:00.706563+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:55:22.183566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:24.209460+0000","last_clean_scrub_stamp":"2021-02-10T04:55:22.183566+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"1002'1090","reported_seq":"5375","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.704336+0000","last_change":"2021-02-10T04:59:53.666790+0000","last_active":"2021-02-10T05:00:02.704336+0000","last_peered":"2021-02-10T05:00:02.704336+0000","last_clean":"2021-02-10T05:00:02.704336+0000","last_became_active":"2021-02-10T04:59:53.664286+0000","last_became_peered":"2021-02-10T04:59:53.664286+0000","last_unstale":"2021-02-10T05:00:02.704336+0000","last_undegraded":"2021-02-10T05:00:02.704336+0000","last_fullsized":"2021-02-10T05:00:02.704336+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:51:08.220472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:08.220472+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24410646,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":245,"num_read_kb":276168,"num_write":175,"num_write_kb":64960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":36920818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"1340","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.704243+0000","last_change":"2021-02-10T04:54:39.555379+0000","last_active":"2021-02-10T05:00:02.704243+0000","last_peered":"2021-02-10T05:00:02.704243+0000","last_clean":"2021-02-10T05:00:02.704243+0000","last_became_active":"2021-02-10T04:54:39.554976+0000","last_became_peered":"2021-02-10T04:54:39.554976+0000","last_unstale":"2021-02-10T05:00:02.704243+0000","last_undegraded":"2021-02-10T05:00:02.704243+0000","last_fullsized":"2021-02-10T05:00:02.704243+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"1261'65","reported_seq":"1499","reported_epoch":"1374","state":"active+remapped+backfill_toofull","last_fresh":"2021-02-10T05:00:02.705750+0000","last_change":"2021-02-10T04:59:52.160041+0000","last_active":"2021-02-10T05:00:02.705750+0000","last_peered":"2021-02-10T05:00:02.705750+0000","last_clean":"2021-02-10T04:59:49.068996+0000","last_became_active":"2021-02-10T04:59:52.099405+0000","last_became_peered":"2021-02-10T04:59:52.099405+0000","last_unstale":"2021-02-10T05:00:02.705750+0000","last_undegraded":"2021-02-10T05:00:02.705750+0000","last_fullsized":"2021-02-10T05:00:02.705750+0000","mapping_epoch":1364,"log_start":"1049'47","ondisk_log_start":"1049'47","created":55,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'47","last_scrub_stamp":"2021-02-10T04:53:09.002368+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.002368+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":836,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":84,"num_read_kb":67,"num_write":49,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":467,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,6],"avail_no_missing":["4","1","6"],"object_location_counts":[{"shards":"1,4,6","objects":11}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"326","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.805700+0000","last_change":"2021-02-10T04:58:56.842228+0000","last_active":"2021-02-10T05:00:02.805700+0000","last_peered":"2021-02-10T05:00:02.805700+0000","last_clean":"2021-02-10T05:00:02.805700+0000","last_became_active":"2021-02-10T04:58:56.841107+0000","last_became_peered":"2021-02-10T04:58:56.841107+0000","last_unstale":"2021-02-10T05:00:02.805700+0000","last_undegraded":"2021-02-10T05:00:02.805700+0000","last_fullsized":"2021-02-10T05:00:02.805700+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:08.358300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:50:08.358300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"1359","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.705710+0000","last_change":"2021-02-10T04:59:51.118440+0000","last_active":"2021-02-10T05:00:00.705710+0000","last_peered":"2021-02-10T05:00:00.705710+0000","last_clean":"2021-02-10T05:00:00.705710+0000","last_became_active":"2021-02-10T04:59:51.118081+0000","last_became_peered":"2021-02-10T04:59:51.118081+0000","last_unstale":"2021-02-10T05:00:00.705710+0000","last_undegraded":"2021-02-10T05:00:00.705710+0000","last_fullsized":"2021-02-10T05:00:00.705710+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.436862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.436862+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"1315","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.703948+0000","last_change":"2021-02-10T04:59:51.116502+0000","last_active":"2021-02-10T05:00:02.703948+0000","last_peered":"2021-02-10T05:00:02.703948+0000","last_clean":"2021-02-10T05:00:02.703948+0000","last_became_active":"2021-02-10T04:59:51.116133+0000","last_became_peered":"2021-02-10T04:59:51.116133+0000","last_unstale":"2021-02-10T05:00:02.703948+0000","last_undegraded":"2021-02-10T05:00:02.703948+0000","last_fullsized":"2021-02-10T05:00:02.703948+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:02.112533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:02.112533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"1001'1165","reported_seq":"4362","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.806129+0000","last_change":"2021-02-10T04:59:53.660114+0000","last_active":"2021-02-10T05:00:02.806129+0000","last_peered":"2021-02-10T05:00:02.806129+0000","last_clean":"2021-02-10T05:00:02.806129+0000","last_became_active":"2021-02-10T04:59:53.658358+0000","last_became_peered":"2021-02-10T04:59:53.658358+0000","last_unstale":"2021-02-10T05:00:02.806129+0000","last_undegraded":"2021-02-10T05:00:02.806129+0000","last_fullsized":"2021-02-10T05:00:02.806129+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29601845,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":286,"num_read_kb":299529,"num_write":200,"num_write_kb":89130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":50371563,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"149","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.806045+0000","last_change":"2021-02-10T04:59:08.658203+0000","last_active":"2021-02-10T05:00:02.806045+0000","last_peered":"2021-02-10T05:00:02.806045+0000","last_clean":"2021-02-10T05:00:02.806045+0000","last_became_active":"2021-02-10T04:58:56.956556+0000","last_became_peered":"2021-02-10T04:58:56.956556+0000","last_unstale":"2021-02-10T05:00:02.806045+0000","last_undegraded":"2021-02-10T05:00:02.806045+0000","last_fullsized":"2021-02-10T05:00:02.806045+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"1113'5","reported_seq":"1660","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:03.088699+0000","last_change":"2021-02-10T04:52:34.047474+0000","last_active":"2021-02-10T05:00:03.088699+0000","last_peered":"2021-02-10T05:00:03.088699+0000","last_clean":"2021-02-10T05:00:03.088699+0000","last_became_active":"2021-02-10T04:50:59.569578+0000","last_became_peered":"2021-02-10T04:50:59.569578+0000","last_unstale":"2021-02-10T05:00:03.088699+0000","last_undegraded":"2021-02-10T05:00:03.088699+0000","last_fullsized":"2021-02-10T05:00:03.088699+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1113'5","last_scrub_stamp":"2021-02-10T04:52:34.047358+0000","last_deep_scrub":"158'3","last_deep_scrub_stamp":"2021-02-10T04:29:34.388912+0000","last_clean_scrub_stamp":"2021-02-10T04:52:34.047358+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"1261'68","reported_seq":"1496","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705929+0000","last_change":"2021-02-10T04:59:51.162320+0000","last_active":"2021-02-10T05:00:02.705929+0000","last_peered":"2021-02-10T05:00:02.705929+0000","last_clean":"2021-02-10T05:00:02.705929+0000","last_became_active":"2021-02-10T04:59:51.111783+0000","last_became_peered":"2021-02-10T04:59:51.111783+0000","last_unstale":"2021-02-10T05:00:02.705929+0000","last_undegraded":"2021-02-10T05:00:02.705929+0000","last_fullsized":"2021-02-10T05:00:02.705929+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:51:48.872839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.872839+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":114,"num_read_kb":94,"num_write":68,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"1178","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705515+0000","last_change":"2021-02-10T04:59:51.126437+0000","last_active":"2021-02-10T05:00:02.705515+0000","last_peered":"2021-02-10T05:00:02.705515+0000","last_clean":"2021-02-10T05:00:02.705515+0000","last_became_active":"2021-02-10T04:59:51.125256+0000","last_became_peered":"2021-02-10T04:59:51.125256+0000","last_unstale":"2021-02-10T05:00:02.705515+0000","last_undegraded":"2021-02-10T05:00:02.705515+0000","last_fullsized":"2021-02-10T05:00:02.705515+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:05.984659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:05.984659+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"1237","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.806118+0000","last_change":"2021-02-10T04:59:51.130527+0000","last_active":"2021-02-10T05:00:02.806118+0000","last_peered":"2021-02-10T05:00:02.806118+0000","last_clean":"2021-02-10T05:00:02.806118+0000","last_became_active":"2021-02-10T04:59:51.130127+0000","last_became_peered":"2021-02-10T04:59:51.130127+0000","last_unstale":"2021-02-10T05:00:02.806118+0000","last_undegraded":"2021-02-10T05:00:02.806118+0000","last_fullsized":"2021-02-10T05:00:02.806118+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:09.005421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.005421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"998'1009","reported_seq":"4865","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.702784+0000","last_change":"2021-02-10T04:52:26.037227+0000","last_active":"2021-02-10T05:00:02.702784+0000","last_peered":"2021-02-10T05:00:02.702784+0000","last_clean":"2021-02-10T05:00:02.702784+0000","last_became_active":"2021-02-10T04:52:26.036723+0000","last_became_peered":"2021-02-10T04:52:26.036723+0000","last_unstale":"2021-02-10T05:00:02.702784+0000","last_undegraded":"2021-02-10T05:00:02.702784+0000","last_fullsized":"2021-02-10T05:00:02.702784+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T04:47:20.802735+0000","last_clean_scrub_stamp":"2021-02-10T04:47:20.802735+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12355206,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":119,"num_read_kb":124701,"num_write":77,"num_write_kb":25052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4145029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"1245","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.704503+0000","last_change":"2021-02-10T04:53:33.305896+0000","last_active":"2021-02-10T05:00:02.704503+0000","last_peered":"2021-02-10T05:00:02.704503+0000","last_clean":"2021-02-10T05:00:02.704503+0000","last_became_active":"2021-02-10T04:50:59.570854+0000","last_became_peered":"2021-02-10T04:50:59.570854+0000","last_unstale":"2021-02-10T05:00:02.704503+0000","last_undegraded":"2021-02-10T05:00:02.704503+0000","last_fullsized":"2021-02-10T05:00:02.704503+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:33.305827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:33.305827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"1361","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705906+0000","last_change":"2021-02-10T04:58:56.979110+0000","last_active":"2021-02-10T05:00:02.705906+0000","last_peered":"2021-02-10T05:00:02.705906+0000","last_clean":"2021-02-10T05:00:02.705906+0000","last_became_active":"2021-02-10T04:58:56.978745+0000","last_became_peered":"2021-02-10T04:58:56.978745+0000","last_unstale":"2021-02-10T05:00:02.705906+0000","last_undegraded":"2021-02-10T05:00:02.705906+0000","last_fullsized":"2021-02-10T05:00:02.705906+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"1359","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.706196+0000","last_change":"2021-02-10T04:51:50.895471+0000","last_active":"2021-02-10T05:00:02.706196+0000","last_peered":"2021-02-10T05:00:02.706196+0000","last_clean":"2021-02-10T05:00:02.706196+0000","last_became_active":"2021-02-10T04:50:59.527553+0000","last_became_peered":"2021-02-10T04:50:59.527553+0000","last_unstale":"2021-02-10T05:00:02.706196+0000","last_undegraded":"2021-02-10T05:00:02.706196+0000","last_fullsized":"2021-02-10T05:00:02.706196+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:50.895370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:50.895370+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"1261'84","reported_seq":"1510","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.812156+0000","last_change":"2021-02-10T05:00:02.812081+0000","last_active":"2021-02-10T05:00:02.812156+0000","last_peered":"2021-02-10T05:00:02.812156+0000","last_clean":"2021-02-10T05:00:02.812156+0000","last_became_active":"2021-02-10T04:58:56.879837+0000","last_became_peered":"2021-02-10T04:58:56.879837+0000","last_unstale":"2021-02-10T05:00:02.812156+0000","last_undegraded":"2021-02-10T05:00:02.812156+0000","last_fullsized":"2021-02-10T05:00:02.812156+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1261'84","last_scrub_stamp":"2021-02-10T05:00:02.811984+0000","last_deep_scrub":"152'48","last_deep_scrub_stamp":"2021-02-10T04:29:28.398429+0000","last_clean_scrub_stamp":"2021-02-10T05:00:02.811984+0000","log_size":84,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":123,"num_read_kb":101,"num_write":72,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"53'1","reported_seq":"40","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.706098+0000","last_change":"2021-02-10T04:59:51.275465+0000","last_active":"2021-02-10T05:00:02.706098+0000","last_peered":"2021-02-10T05:00:02.706098+0000","last_clean":"2021-02-10T05:00:02.706098+0000","last_became_active":"2021-02-10T04:59:51.125942+0000","last_became_peered":"2021-02-10T04:59:51.125942+0000","last_unstale":"2021-02-10T05:00:02.706098+0000","last_undegraded":"2021-02-10T05:00:02.706098+0000","last_fullsized":"2021-02-10T05:00:02.706098+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:51:48.274960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:48.274960+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":423,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":846,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"1384","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.706783+0000","last_change":"2021-02-10T04:53:13.099288+0000","last_active":"2021-02-10T05:00:02.706783+0000","last_peered":"2021-02-10T05:00:02.706783+0000","last_clean":"2021-02-10T05:00:02.706783+0000","last_became_active":"2021-02-10T04:51:36.848614+0000","last_became_peered":"2021-02-10T04:51:36.848614+0000","last_unstale":"2021-02-10T05:00:02.706783+0000","last_undegraded":"2021-02-10T05:00:02.706783+0000","last_fullsized":"2021-02-10T05:00:02.706783+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:13.099228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:13.099228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"1002'1657","reported_seq":"5435","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.703558+0000","last_change":"2021-02-10T04:52:26.036806+0000","last_active":"2021-02-10T05:00:02.703558+0000","last_peered":"2021-02-10T05:00:02.703558+0000","last_clean":"2021-02-10T05:00:02.703558+0000","last_became_active":"2021-02-10T04:52:26.032363+0000","last_became_peered":"2021-02-10T04:52:26.032363+0000","last_unstale":"2021-02-10T05:00:02.703558+0000","last_undegraded":"2021-02-10T05:00:02.703558+0000","last_fullsized":"2021-02-10T05:00:02.703558+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21396533,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":212,"num_read_kb":218836,"num_write":159,"num_write_kb":59398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":22326014,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"1312","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705430+0000","last_change":"2021-02-10T04:59:51.112802+0000","last_active":"2021-02-10T05:00:02.705430+0000","last_peered":"2021-02-10T05:00:02.705430+0000","last_clean":"2021-02-10T05:00:02.705430+0000","last_became_active":"2021-02-10T04:59:51.112487+0000","last_became_peered":"2021-02-10T04:59:51.112487+0000","last_unstale":"2021-02-10T05:00:02.705430+0000","last_undegraded":"2021-02-10T05:00:02.705430+0000","last_fullsized":"2021-02-10T05:00:02.705430+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.911851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.911851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"62'5","reported_seq":"64","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705276+0000","last_change":"2021-02-10T04:59:40.662441+0000","last_active":"2021-02-10T05:00:02.705276+0000","last_peered":"2021-02-10T05:00:02.705276+0000","last_clean":"2021-02-10T05:00:02.705276+0000","last_became_active":"2021-02-10T04:59:40.661336+0000","last_became_peered":"2021-02-10T04:59:40.661336+0000","last_unstale":"2021-02-10T05:00:02.705276+0000","last_undegraded":"2021-02-10T05:00:02.705276+0000","last_fullsized":"2021-02-10T05:00:02.705276+0000","mapping_epoch":1356,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1357,"parent":"0.0","parent_split_bits":4,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":98,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":98,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"1419","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.805836+0000","last_change":"2021-02-10T04:59:51.117511+0000","last_active":"2021-02-10T05:00:02.805836+0000","last_peered":"2021-02-10T05:00:02.805836+0000","last_clean":"2021-02-10T05:00:02.805836+0000","last_became_active":"2021-02-10T04:59:51.117148+0000","last_became_peered":"2021-02-10T04:59:51.117148+0000","last_unstale":"2021-02-10T05:00:02.805836+0000","last_undegraded":"2021-02-10T05:00:02.805836+0000","last_fullsized":"2021-02-10T05:00:02.805836+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:58.142526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:58.142526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"1261'68","reported_seq":"1367","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.805934+0000","last_change":"2021-02-10T04:59:08.646165+0000","last_active":"2021-02-10T05:00:02.805934+0000","last_peered":"2021-02-10T05:00:02.805934+0000","last_clean":"2021-02-10T05:00:02.805934+0000","last_became_active":"2021-02-10T04:58:56.954019+0000","last_became_peered":"2021-02-10T04:58:56.954019+0000","last_unstale":"2021-02-10T05:00:02.805934+0000","last_undegraded":"2021-02-10T05:00:02.805934+0000","last_fullsized":"2021-02-10T05:00:02.805934+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1135'58","last_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_deep_scrub":"1135'58","last_deep_scrub_stamp":"2021-02-10T04:52:01.483067+0000","last_clean_scrub_stamp":"2021-02-10T04:52:01.483067+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":101,"num_read_kb":82,"num_write":60,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"53'2","reported_seq":"347","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.806943+0000","last_change":"2021-02-10T05:00:01.698057+0000","last_active":"2021-02-10T05:00:02.806943+0000","last_peered":"2021-02-10T05:00:02.806943+0000","last_clean":"2021-02-10T05:00:02.806943+0000","last_became_active":"2021-02-10T04:58:56.849868+0000","last_became_peered":"2021-02-10T04:58:56.849868+0000","last_unstale":"2021-02-10T05:00:02.806943+0000","last_undegraded":"2021-02-10T05:00:02.806943+0000","last_fullsized":"2021-02-10T05:00:02.806943+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T05:00:01.697946+0000","last_deep_scrub":"53'2","last_deep_scrub_stamp":"2021-02-10T04:29:35.711557+0000","last_clean_scrub_stamp":"2021-02-10T05:00:01.697946+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"834","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.704227+0000","last_change":"2021-02-10T04:59:51.143488+0000","last_active":"2021-02-10T05:00:02.704227+0000","last_peered":"2021-02-10T05:00:02.704227+0000","last_clean":"2021-02-10T05:00:02.704227+0000","last_became_active":"2021-02-10T04:59:51.143081+0000","last_became_peered":"2021-02-10T04:59:51.143081+0000","last_unstale":"2021-02-10T05:00:02.704227+0000","last_undegraded":"2021-02-10T05:00:02.704227+0000","last_fullsized":"2021-02-10T05:00:02.704227+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:38.278910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:52:38.278910+0000","last_clean_scrub_stamp":"2021-02-10T04:52:38.278910+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.0","version":"1000'1071","reported_seq":"4924","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.702168+0000","last_change":"2021-02-10T04:53:34.407875+0000","last_active":"2021-02-10T05:00:00.702168+0000","last_peered":"2021-02-10T05:00:00.702168+0000","last_clean":"2021-02-10T05:00:00.702168+0000","last_became_active":"2021-02-10T04:52:26.031050+0000","last_became_peered":"2021-02-10T04:52:26.031050+0000","last_unstale":"2021-02-10T05:00:00.702168+0000","last_undegraded":"2021-02-10T05:00:00.702168+0000","last_fullsized":"2021-02-10T05:00:00.702168+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:34.407784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.407784+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30173843,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"1385","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.806158+0000","last_change":"2021-02-10T04:59:51.120656+0000","last_active":"2021-02-10T05:00:02.806158+0000","last_peered":"2021-02-10T05:00:02.806158+0000","last_clean":"2021-02-10T05:00:02.806158+0000","last_became_active":"2021-02-10T04:59:51.120335+0000","last_became_peered":"2021-02-10T04:59:51.120335+0000","last_unstale":"2021-02-10T05:00:02.806158+0000","last_undegraded":"2021-02-10T05:00:02.806158+0000","last_fullsized":"2021-02-10T05:00:02.806158+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:00.195518+0000","last_clean_scrub_stamp":"2021-02-10T04:51:00.195518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"1380","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.705781+0000","last_change":"2021-02-10T04:58:58.252055+0000","last_active":"2021-02-10T05:00:00.705781+0000","last_peered":"2021-02-10T05:00:00.705781+0000","last_clean":"2021-02-10T05:00:00.705781+0000","last_became_active":"2021-02-10T04:58:56.854414+0000","last_became_peered":"2021-02-10T04:58:56.854414+0000","last_unstale":"2021-02-10T05:00:00.705781+0000","last_undegraded":"2021-02-10T05:00:00.705781+0000","last_fullsized":"2021-02-10T05:00:00.705781+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:58:58.251969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:25.216260+0000","last_clean_scrub_stamp":"2021-02-10T04:58:58.251969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"1261'58","reported_seq":"1428","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.805893+0000","last_change":"2021-02-10T04:59:13.927049+0000","last_active":"2021-02-10T05:00:02.805893+0000","last_peered":"2021-02-10T05:00:02.805893+0000","last_clean":"2021-02-10T05:00:02.805893+0000","last_became_active":"2021-02-10T04:59:13.926626+0000","last_became_peered":"2021-02-10T04:59:13.926626+0000","last_unstale":"2021-02-10T05:00:02.805893+0000","last_undegraded":"2021-02-10T05:00:02.805893+0000","last_fullsized":"2021-02-10T05:00:02.805893+0000","mapping_epoch":1332,"log_start":"1049'40","ondisk_log_start":"1049'40","created":55,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'40","last_scrub_stamp":"2021-02-10T04:51:47.281929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.281929+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":918,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":85,"num_read_kb":67,"num_write":49,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"550","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.805805+0000","last_change":"2021-02-10T04:58:56.841926+0000","last_active":"2021-02-10T05:00:02.805805+0000","last_peered":"2021-02-10T05:00:02.805805+0000","last_clean":"2021-02-10T05:00:02.805805+0000","last_became_active":"2021-02-10T04:58:56.840576+0000","last_became_peered":"2021-02-10T04:58:56.840576+0000","last_unstale":"2021-02-10T05:00:02.805805+0000","last_undegraded":"2021-02-10T05:00:02.805805+0000","last_fullsized":"2021-02-10T05:00:02.805805+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:50.399350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:50.399350+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"1253","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.805303+0000","last_change":"2021-02-10T04:59:08.647800+0000","last_active":"2021-02-10T05:00:02.805303+0000","last_peered":"2021-02-10T05:00:02.805303+0000","last_clean":"2021-02-10T05:00:02.805303+0000","last_became_active":"2021-02-10T04:58:56.875409+0000","last_became_peered":"2021-02-10T04:58:56.875409+0000","last_unstale":"2021-02-10T05:00:02.805303+0000","last_undegraded":"2021-02-10T05:00:02.805303+0000","last_fullsized":"2021-02-10T05:00:02.805303+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"829","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.706900+0000","last_change":"2021-02-10T04:59:40.661283+0000","last_active":"2021-02-10T05:00:00.706900+0000","last_peered":"2021-02-10T05:00:00.706900+0000","last_clean":"2021-02-10T05:00:00.706900+0000","last_became_active":"2021-02-10T04:59:40.660922+0000","last_became_peered":"2021-02-10T04:59:40.660922+0000","last_unstale":"2021-02-10T05:00:00.706900+0000","last_undegraded":"2021-02-10T05:00:00.706900+0000","last_fullsized":"2021-02-10T05:00:00.706900+0000","mapping_epoch":1356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:31.272971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:17.598113+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.272971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.5","version":"1000'446","reported_seq":"2107","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.806163+0000","last_change":"2021-02-10T04:58:56.846600+0000","last_active":"2021-02-10T05:00:02.806163+0000","last_peered":"2021-02-10T05:00:02.806163+0000","last_clean":"2021-02-10T05:00:02.806163+0000","last_became_active":"2021-02-10T04:58:56.846008+0000","last_became_peered":"2021-02-10T04:58:56.846008+0000","last_unstale":"2021-02-10T05:00:02.806163+0000","last_undegraded":"2021-02-10T05:00:02.806163+0000","last_fullsized":"2021-02-10T05:00:02.806163+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:51:45.964029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.964029+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22255852,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68435,"num_write":72,"num_write_kb":24345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":403693,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"886","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705475+0000","last_change":"2021-02-10T04:59:51.123025+0000","last_active":"2021-02-10T05:00:02.705475+0000","last_peered":"2021-02-10T05:00:02.705475+0000","last_clean":"2021-02-10T05:00:02.705475+0000","last_became_active":"2021-02-10T04:59:51.122151+0000","last_became_peered":"2021-02-10T04:59:51.122151+0000","last_unstale":"2021-02-10T05:00:02.705475+0000","last_undegraded":"2021-02-10T05:00:02.705475+0000","last_fullsized":"2021-02-10T05:00:02.705475+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:03.228182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.228182+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"279","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.805075+0000","last_change":"2021-02-10T04:53:46.727384+0000","last_active":"2021-02-10T05:00:02.805075+0000","last_peered":"2021-02-10T05:00:02.805075+0000","last_clean":"2021-02-10T05:00:02.805075+0000","last_became_active":"2021-02-10T04:53:46.726771+0000","last_became_peered":"2021-02-10T04:53:46.726771+0000","last_unstale":"2021-02-10T05:00:02.805075+0000","last_undegraded":"2021-02-10T05:00:02.805075+0000","last_fullsized":"2021-02-10T05:00:02.805075+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:33.948348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:33.948348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"1373'60","reported_seq":"1401","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705645+0000","last_change":"2021-02-10T05:00:02.279766+0000","last_active":"2021-02-10T05:00:02.705645+0000","last_peered":"2021-02-10T05:00:02.705645+0000","last_clean":"2021-02-10T05:00:02.705645+0000","last_became_active":"2021-02-10T04:59:51.118282+0000","last_became_peered":"2021-02-10T04:59:51.118282+0000","last_unstale":"2021-02-10T05:00:02.705645+0000","last_undegraded":"2021-02-10T05:00:02.705645+0000","last_fullsized":"2021-02-10T05:00:02.705645+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"1373'60","last_scrub_stamp":"2021-02-10T05:00:02.279704+0000","last_deep_scrub":"1373'60","last_deep_scrub_stamp":"2021-02-10T05:00:02.279704+0000","last_clean_scrub_stamp":"2021-02-10T05:00:02.279704+0000","log_size":60,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":68,"num_write":51,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":932,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"331","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.805904+0000","last_change":"2021-02-10T04:59:51.117366+0000","last_active":"2021-02-10T05:00:02.805904+0000","last_peered":"2021-02-10T05:00:02.805904+0000","last_clean":"2021-02-10T05:00:02.805904+0000","last_became_active":"2021-02-10T04:59:51.116190+0000","last_became_peered":"2021-02-10T04:59:51.116190+0000","last_unstale":"2021-02-10T05:00:02.805904+0000","last_undegraded":"2021-02-10T05:00:02.805904+0000","last_fullsized":"2021-02-10T05:00:02.805904+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:05.424262+0000","last_clean_scrub_stamp":"2021-02-10T04:50:05.424262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"1544","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.706101+0000","last_change":"2021-02-10T04:52:37.324122+0000","last_active":"2021-02-10T05:00:00.706101+0000","last_peered":"2021-02-10T05:00:00.706101+0000","last_clean":"2021-02-10T05:00:00.706101+0000","last_became_active":"2021-02-10T04:51:36.843553+0000","last_became_peered":"2021-02-10T04:51:36.843553+0000","last_unstale":"2021-02-10T05:00:00.706101+0000","last_undegraded":"2021-02-10T05:00:00.706101+0000","last_fullsized":"2021-02-10T05:00:00.706101+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:37.324040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:37.324040+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"1360","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.702859+0000","last_change":"2021-02-10T04:59:51.114576+0000","last_active":"2021-02-10T05:00:02.702859+0000","last_peered":"2021-02-10T05:00:02.702859+0000","last_clean":"2021-02-10T05:00:02.702859+0000","last_became_active":"2021-02-10T04:59:51.114177+0000","last_became_peered":"2021-02-10T04:59:51.114177+0000","last_unstale":"2021-02-10T05:00:02.702859+0000","last_undegraded":"2021-02-10T05:00:02.702859+0000","last_fullsized":"2021-02-10T05:00:02.702859+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.769916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.769916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"1372","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.705859+0000","last_change":"2021-02-10T04:52:31.403904+0000","last_active":"2021-02-10T05:00:00.705859+0000","last_peered":"2021-02-10T05:00:00.705859+0000","last_clean":"2021-02-10T05:00:00.705859+0000","last_became_active":"2021-02-10T04:50:59.538009+0000","last_became_peered":"2021-02-10T04:50:59.538009+0000","last_unstale":"2021-02-10T05:00:00.705859+0000","last_undegraded":"2021-02-10T05:00:00.705859+0000","last_fullsized":"2021-02-10T05:00:00.705859+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:31.403824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:31.403824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"1000'490","reported_seq":"3833","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.806418+0000","last_change":"2021-02-10T04:59:08.651746+0000","last_active":"2021-02-10T05:00:02.806418+0000","last_peered":"2021-02-10T05:00:02.806418+0000","last_clean":"2021-02-10T05:00:02.806418+0000","last_became_active":"2021-02-10T04:58:56.962979+0000","last_became_peered":"2021-02-10T04:58:56.962979+0000","last_unstale":"2021-02-10T05:00:02.806418+0000","last_undegraded":"2021-02-10T05:00:02.806418+0000","last_fullsized":"2021-02-10T05:00:02.806418+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:47:02.875920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:02.875920+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19371627,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":141,"num_read_kb":148568,"num_write":85,"num_write_kb":31732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"280","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705777+0000","last_change":"2021-02-10T04:59:51.117341+0000","last_active":"2021-02-10T05:00:02.705777+0000","last_peered":"2021-02-10T05:00:02.705777+0000","last_clean":"2021-02-10T05:00:02.705777+0000","last_became_active":"2021-02-10T04:59:51.116854+0000","last_became_peered":"2021-02-10T04:59:51.116854+0000","last_unstale":"2021-02-10T05:00:02.705777+0000","last_undegraded":"2021-02-10T05:00:02.705777+0000","last_fullsized":"2021-02-10T05:00:02.705777+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:01.972913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:16.564638+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.972913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"1261'40","reported_seq":"1116","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:59.694058+0000","last_change":"2021-02-10T04:59:13.926580+0000","last_active":"2021-02-10T04:59:59.694058+0000","last_peered":"2021-02-10T04:59:59.694058+0000","last_clean":"2021-02-10T04:59:59.694058+0000","last_became_active":"2021-02-10T04:59:13.926146+0000","last_became_peered":"2021-02-10T04:59:13.926146+0000","last_unstale":"2021-02-10T04:59:59.694058+0000","last_undegraded":"2021-02-10T04:59:59.694058+0000","last_fullsized":"2021-02-10T04:59:59.694058+0000","mapping_epoch":1332,"log_start":"1063'30","ondisk_log_start":"1063'30","created":55,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:56.431187+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.431187+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":401,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":58,"num_read_kb":47,"num_write":33,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"296","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.704340+0000","last_change":"2021-02-10T04:58:56.879863+0000","last_active":"2021-02-10T05:00:02.704340+0000","last_peered":"2021-02-10T05:00:02.704340+0000","last_clean":"2021-02-10T05:00:02.704340+0000","last_became_active":"2021-02-10T04:58:56.879496+0000","last_became_peered":"2021-02-10T04:58:56.879496+0000","last_unstale":"2021-02-10T05:00:02.704340+0000","last_undegraded":"2021-02-10T05:00:02.704340+0000","last_fullsized":"2021-02-10T05:00:02.704340+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:04.591695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:47:04.591695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"62'5","reported_seq":"73","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.704808+0000","last_change":"2021-02-10T04:59:51.143398+0000","last_active":"2021-02-10T05:00:02.704808+0000","last_peered":"2021-02-10T05:00:02.704808+0000","last_clean":"2021-02-10T05:00:02.704808+0000","last_became_active":"2021-02-10T04:59:51.142973+0000","last_became_peered":"2021-02-10T04:59:51.142973+0000","last_unstale":"2021-02-10T05:00:02.704808+0000","last_undegraded":"2021-02-10T05:00:02.704808+0000","last_fullsized":"2021-02-10T05:00:02.704808+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-02-10T04:52:25.358033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.358033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":97,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"1227","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.806854+0000","last_change":"2021-02-10T04:59:51.130891+0000","last_active":"2021-02-10T05:00:02.806854+0000","last_peered":"2021-02-10T05:00:02.806854+0000","last_clean":"2021-02-10T05:00:02.806854+0000","last_became_active":"2021-02-10T04:59:51.130334+0000","last_became_peered":"2021-02-10T04:59:51.130334+0000","last_unstale":"2021-02-10T05:00:02.806854+0000","last_undegraded":"2021-02-10T05:00:02.806854+0000","last_fullsized":"2021-02-10T05:00:02.806854+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:40.066354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:52:40.066354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"1000'620","reported_seq":"4457","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.805759+0000","last_change":"2021-02-10T04:59:08.651277+0000","last_active":"2021-02-10T05:00:02.805759+0000","last_peered":"2021-02-10T05:00:02.805759+0000","last_clean":"2021-02-10T05:00:02.805759+0000","last_became_active":"2021-02-10T04:58:56.878577+0000","last_became_peered":"2021-02-10T04:58:56.878577+0000","last_unstale":"2021-02-10T05:00:02.805759+0000","last_undegraded":"2021-02-10T05:00:02.805759+0000","last_fullsized":"2021-02-10T05:00:02.805759+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:51:54.444287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:54.444287+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14265195,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":144,"num_read_kb":128482,"num_write":129,"num_write_kb":53443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"1246","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.805601+0000","last_change":"2021-02-10T04:59:08.653244+0000","last_active":"2021-02-10T05:00:02.805601+0000","last_peered":"2021-02-10T05:00:02.805601+0000","last_clean":"2021-02-10T05:00:02.805601+0000","last_became_active":"2021-02-10T04:58:56.842941+0000","last_became_peered":"2021-02-10T04:58:56.842941+0000","last_unstale":"2021-02-10T05:00:02.805601+0000","last_undegraded":"2021-02-10T05:00:02.805601+0000","last_fullsized":"2021-02-10T05:00:02.805601+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:14.955544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:47:14.955544+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"1375","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705534+0000","last_change":"2021-02-10T04:59:51.123202+0000","last_active":"2021-02-10T05:00:02.705534+0000","last_peered":"2021-02-10T05:00:02.705534+0000","last_clean":"2021-02-10T05:00:02.705534+0000","last_became_active":"2021-02-10T04:59:51.122639+0000","last_became_peered":"2021-02-10T04:59:51.122639+0000","last_unstale":"2021-02-10T05:00:02.705534+0000","last_undegraded":"2021-02-10T05:00:02.705534+0000","last_fullsized":"2021-02-10T05:00:02.705534+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"1349","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.805985+0000","last_change":"2021-02-10T04:59:08.656087+0000","last_active":"2021-02-10T05:00:02.805985+0000","last_peered":"2021-02-10T05:00:02.805985+0000","last_clean":"2021-02-10T05:00:02.805985+0000","last_became_active":"2021-02-10T04:58:56.957166+0000","last_became_peered":"2021-02-10T04:58:56.957166+0000","last_unstale":"2021-02-10T05:00:02.805985+0000","last_undegraded":"2021-02-10T05:00:02.805985+0000","last_fullsized":"2021-02-10T05:00:02.805985+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:27.908221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:31.650033+0000","last_clean_scrub_stamp":"2021-02-10T04:47:27.908221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"1261'67","reported_seq":"939","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705643+0000","last_change":"2021-02-10T04:59:40.661729+0000","last_active":"2021-02-10T05:00:02.705643+0000","last_peered":"2021-02-10T05:00:02.705643+0000","last_clean":"2021-02-10T05:00:02.705643+0000","last_became_active":"2021-02-10T04:59:40.661402+0000","last_became_peered":"2021-02-10T04:59:40.661402+0000","last_unstale":"2021-02-10T05:00:02.705643+0000","last_undegraded":"2021-02-10T05:00:02.705643+0000","last_fullsized":"2021-02-10T05:00:02.705643+0000","mapping_epoch":1356,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1357,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'55","last_scrub_stamp":"2021-02-10T04:52:30.276938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.276938+0000","log_size":67,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":115,"num_read_kb":91,"num_write":70,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"1377","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.707555+0000","last_change":"2021-02-10T04:59:51.119728+0000","last_active":"2021-02-10T05:00:00.707555+0000","last_peered":"2021-02-10T05:00:00.707555+0000","last_clean":"2021-02-10T05:00:00.707555+0000","last_became_active":"2021-02-10T04:59:51.119372+0000","last_became_peered":"2021-02-10T04:59:51.119372+0000","last_unstale":"2021-02-10T05:00:00.707555+0000","last_undegraded":"2021-02-10T05:00:00.707555+0000","last_fullsized":"2021-02-10T05:00:00.707555+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.420086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.420086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"1335","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705165+0000","last_change":"2021-02-10T04:53:12.187225+0000","last_active":"2021-02-10T05:00:02.705165+0000","last_peered":"2021-02-10T05:00:02.705165+0000","last_clean":"2021-02-10T05:00:02.705165+0000","last_became_active":"2021-02-10T04:50:59.555689+0000","last_became_peered":"2021-02-10T04:50:59.555689+0000","last_unstale":"2021-02-10T05:00:02.705165+0000","last_undegraded":"2021-02-10T05:00:02.705165+0000","last_fullsized":"2021-02-10T05:00:02.705165+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.187154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.187154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"1002'2042","reported_seq":"9528","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.806986+0000","last_change":"2021-02-10T04:59:53.663243+0000","last_active":"2021-02-10T05:00:02.806986+0000","last_peered":"2021-02-10T05:00:02.806986+0000","last_clean":"2021-02-10T05:00:02.806986+0000","last_became_active":"2021-02-10T04:59:53.660539+0000","last_became_peered":"2021-02-10T04:59:53.660539+0000","last_unstale":"2021-02-10T05:00:02.806986+0000","last_undegraded":"2021-02-10T05:00:02.806986+0000","last_fullsized":"2021-02-10T05:00:02.806986+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:46:57.054230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:46:57.054230+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46300076,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":545,"num_read_kb":531689,"num_write":399,"num_write_kb":153087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":66804807,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"1334","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.703215+0000","last_change":"2021-02-10T04:58:56.856707+0000","last_active":"2021-02-10T05:00:00.703215+0000","last_peered":"2021-02-10T05:00:00.703215+0000","last_clean":"2021-02-10T05:00:00.703215+0000","last_became_active":"2021-02-10T04:58:56.856323+0000","last_became_peered":"2021-02-10T04:58:56.856323+0000","last_unstale":"2021-02-10T05:00:00.703215+0000","last_undegraded":"2021-02-10T05:00:00.703215+0000","last_fullsized":"2021-02-10T05:00:00.703215+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:52.341527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:50:52.341527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"172","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.706304+0000","last_change":"2021-02-10T04:54:39.558380+0000","last_active":"2021-02-10T05:00:00.706304+0000","last_peered":"2021-02-10T05:00:00.706304+0000","last_clean":"2021-02-10T05:00:00.706304+0000","last_became_active":"2021-02-10T04:54:39.557995+0000","last_became_peered":"2021-02-10T04:54:39.557995+0000","last_unstale":"2021-02-10T05:00:00.706304+0000","last_undegraded":"2021-02-10T05:00:00.706304+0000","last_fullsized":"2021-02-10T05:00:00.706304+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":1209,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:10.907529+0000","last_clean_scrub_stamp":"2021-02-10T04:47:10.907529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"1366'9","reported_seq":"1555","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:03.088282+0000","last_change":"2021-02-10T04:59:51.169990+0000","last_active":"2021-02-10T05:00:03.088282+0000","last_peered":"2021-02-10T05:00:03.088282+0000","last_clean":"2021-02-10T05:00:03.088282+0000","last_became_active":"2021-02-10T04:59:51.117900+0000","last_became_peered":"2021-02-10T04:59:51.117900+0000","last_unstale":"2021-02-10T05:00:03.088282+0000","last_undegraded":"2021-02-10T05:00:03.088282+0000","last_fullsized":"2021-02-10T05:00:03.088282+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2021-02-10T04:50:54.466259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:50:54.466259+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"1261'64","reported_seq":"984","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.707711+0000","last_change":"2021-02-10T04:59:40.660545+0000","last_active":"2021-02-10T05:00:00.707711+0000","last_peered":"2021-02-10T05:00:00.707711+0000","last_clean":"2021-02-10T05:00:00.707711+0000","last_became_active":"2021-02-10T04:59:40.660072+0000","last_became_peered":"2021-02-10T04:59:40.660072+0000","last_unstale":"2021-02-10T05:00:00.707711+0000","last_undegraded":"2021-02-10T05:00:00.707711+0000","last_fullsized":"2021-02-10T05:00:00.707711+0000","mapping_epoch":1356,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1357,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'54","last_scrub_stamp":"2021-02-10T04:53:01.304626+0000","last_deep_scrub":"79'34","last_deep_scrub_stamp":"2021-02-10T04:27:18.990088+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.304626+0000","log_size":64,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":88,"num_read_kb":70,"num_write":52,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"892","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.805503+0000","last_change":"2021-02-10T04:59:40.663233+0000","last_active":"2021-02-10T05:00:02.805503+0000","last_peered":"2021-02-10T05:00:02.805503+0000","last_clean":"2021-02-10T05:00:02.805503+0000","last_became_active":"2021-02-10T04:59:40.662896+0000","last_became_peered":"2021-02-10T04:59:40.662896+0000","last_unstale":"2021-02-10T05:00:02.805503+0000","last_undegraded":"2021-02-10T05:00:02.805503+0000","last_fullsized":"2021-02-10T05:00:02.805503+0000","mapping_epoch":1356,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:49.447961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:50:49.447961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"1364","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.804773+0000","last_change":"2021-02-10T04:59:56.057363+0000","last_active":"2021-02-10T05:00:02.804773+0000","last_peered":"2021-02-10T05:00:02.804773+0000","last_clean":"2021-02-10T05:00:02.804773+0000","last_became_active":"2021-02-10T04:58:56.874543+0000","last_became_peered":"2021-02-10T04:58:56.874543+0000","last_unstale":"2021-02-10T05:00:02.804773+0000","last_undegraded":"2021-02-10T05:00:02.804773+0000","last_fullsized":"2021-02-10T05:00:02.804773+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:59:56.057291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:59:56.057291+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"1371'83","reported_seq":"3189","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.706002+0000","last_change":"2021-02-10T05:00:00.403522+0000","last_active":"2021-02-10T05:00:02.706002+0000","last_peered":"2021-02-10T05:00:02.706002+0000","last_clean":"2021-02-10T05:00:02.706002+0000","last_became_active":"2021-02-10T04:59:51.124908+0000","last_became_peered":"2021-02-10T04:59:51.124908+0000","last_unstale":"2021-02-10T05:00:02.706002+0000","last_undegraded":"2021-02-10T05:00:02.706002+0000","last_fullsized":"2021-02-10T05:00:02.706002+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":83,"ondisk_log_size":83,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":962233,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":18587,"num_write":10,"num_write_kb":5384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3042034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"1134","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.706376+0000","last_change":"2021-02-10T04:59:51.122465+0000","last_active":"2021-02-10T05:00:02.706376+0000","last_peered":"2021-02-10T05:00:02.706376+0000","last_clean":"2021-02-10T05:00:02.706376+0000","last_became_active":"2021-02-10T04:59:51.122127+0000","last_became_peered":"2021-02-10T04:59:51.122127+0000","last_unstale":"2021-02-10T05:00:02.706376+0000","last_undegraded":"2021-02-10T05:00:02.706376+0000","last_fullsized":"2021-02-10T05:00:02.706376+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.024980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:24:06.001969+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.024980+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"371","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.704736+0000","last_change":"2021-02-10T04:59:59.241100+0000","last_active":"2021-02-10T05:00:02.704736+0000","last_peered":"2021-02-10T05:00:02.704736+0000","last_clean":"2021-02-10T05:00:02.704736+0000","last_became_active":"2021-02-10T04:58:56.850742+0000","last_became_peered":"2021-02-10T04:58:56.850742+0000","last_unstale":"2021-02-10T05:00:02.704736+0000","last_undegraded":"2021-02-10T05:00:02.704736+0000","last_fullsized":"2021-02-10T05:00:02.704736+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:59:59.241025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:59:59.241025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"1261'86","reported_seq":"1565","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.706043+0000","last_change":"2021-02-10T04:59:13.931273+0000","last_active":"2021-02-10T05:00:02.706043+0000","last_peered":"2021-02-10T05:00:02.706043+0000","last_clean":"2021-02-10T05:00:02.706043+0000","last_became_active":"2021-02-10T04:59:13.930845+0000","last_became_peered":"2021-02-10T04:59:13.930845+0000","last_unstale":"2021-02-10T05:00:02.706043+0000","last_undegraded":"2021-02-10T05:00:02.706043+0000","last_fullsized":"2021-02-10T05:00:02.706043+0000","mapping_epoch":1238,"log_start":"1049'68","ondisk_log_start":"1049'68","created":55,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'39","last_scrub_stamp":"2021-02-10T04:53:38.085565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.085565+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":791,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":113,"num_read_kb":93,"num_write":64,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1582,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"69'2","reported_seq":"1510","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.706925+0000","last_change":"2021-02-10T04:59:56.274760+0000","last_active":"2021-02-10T05:00:02.706925+0000","last_peered":"2021-02-10T05:00:02.706925+0000","last_clean":"2021-02-10T05:00:02.706925+0000","last_became_active":"2021-02-10T04:58:56.843963+0000","last_became_peered":"2021-02-10T04:58:56.843963+0000","last_unstale":"2021-02-10T05:00:02.706925+0000","last_undegraded":"2021-02-10T05:00:02.706925+0000","last_fullsized":"2021-02-10T05:00:02.706925+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2021-02-10T04:59:56.274662+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2021-02-10T04:26:34.066149+0000","last_clean_scrub_stamp":"2021-02-10T04:59:56.274662+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"1337","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.703987+0000","last_change":"2021-02-10T04:52:30.019225+0000","last_active":"2021-02-10T05:00:02.703987+0000","last_peered":"2021-02-10T05:00:02.703987+0000","last_clean":"2021-02-10T05:00:02.703987+0000","last_became_active":"2021-02-10T04:50:59.561436+0000","last_became_peered":"2021-02-10T04:50:59.561436+0000","last_unstale":"2021-02-10T05:00:02.703987+0000","last_undegraded":"2021-02-10T05:00:02.703987+0000","last_fullsized":"2021-02-10T05:00:02.703987+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.019102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.019102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"1001'1528","reported_seq":"6322","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.702697+0000","last_change":"2021-02-10T04:52:26.036366+0000","last_active":"2021-02-10T05:00:02.702697+0000","last_peered":"2021-02-10T05:00:02.702697+0000","last_clean":"2021-02-10T05:00:02.702697+0000","last_became_active":"2021-02-10T04:52:26.034028+0000","last_became_peered":"2021-02-10T04:52:26.034028+0000","last_unstale":"2021-02-10T05:00:02.702697+0000","last_undegraded":"2021-02-10T05:00:02.702697+0000","last_fullsized":"2021-02-10T05:00:02.702697+0000","mapping_epoch":1123,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":1142,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46944136,"num_objects":16,"num_object_clones":15,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":358,"num_read_kb":351934,"num_write":286,"num_write_kb":111355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":58968832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"1261'58","reported_seq":"1444","reported_epoch":"1374","state":"active+remapped+backfill_toofull","last_fresh":"2021-02-10T05:00:02.806435+0000","last_change":"2021-02-10T04:59:46.973622+0000","last_active":"2021-02-10T05:00:02.806435+0000","last_peered":"2021-02-10T05:00:02.806435+0000","last_clean":"2021-02-10T04:59:13.930979+0000","last_became_active":"2021-02-10T04:59:16.970832+0000","last_became_peered":"2021-02-10T04:59:16.970832+0000","last_unstale":"2021-02-10T05:00:02.806435+0000","last_undegraded":"2021-02-10T05:00:02.806435+0000","last_fullsized":"2021-02-10T05:00:02.806435+0000","mapping_epoch":1344,"log_start":"1049'40","ondisk_log_start":"1049'40","created":1341,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":5,"last_scrub":"1049'40","last_scrub_stamp":"2021-02-10T04:51:47.281929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.281929+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":918,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":24,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":85,"num_read_kb":67,"num_write":50,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,1,2],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":12}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"1000'446","reported_seq":"2311","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.706020+0000","last_change":"2021-02-10T04:53:11.051124+0000","last_active":"2021-02-10T05:00:02.706020+0000","last_peered":"2021-02-10T05:00:02.706020+0000","last_clean":"2021-02-10T05:00:02.706020+0000","last_became_active":"2021-02-10T04:51:40.510899+0000","last_became_peered":"2021-02-10T04:51:40.510899+0000","last_unstale":"2021-02-10T05:00:02.706020+0000","last_undegraded":"2021-02-10T05:00:02.706020+0000","last_fullsized":"2021-02-10T05:00:02.706020+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'446","last_scrub_stamp":"2021-02-10T04:53:11.051031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.051031+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24896285,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":18983408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.10","version":"50'1","reported_seq":"305","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705098+0000","last_change":"2021-02-10T04:59:55.164058+0000","last_active":"2021-02-10T05:00:02.705098+0000","last_peered":"2021-02-10T05:00:02.705098+0000","last_clean":"2021-02-10T05:00:02.705098+0000","last_became_active":"2021-02-10T04:58:56.850650+0000","last_became_peered":"2021-02-10T04:58:56.850650+0000","last_unstale":"2021-02-10T05:00:02.705098+0000","last_undegraded":"2021-02-10T05:00:02.705098+0000","last_fullsized":"2021-02-10T05:00:02.705098+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:59:55.163992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:59:55.163992+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":247,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"1306","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.708009+0000","last_change":"2021-02-10T04:51:09.475150+0000","last_active":"2021-02-10T05:00:02.708009+0000","last_peered":"2021-02-10T05:00:02.708009+0000","last_clean":"2021-02-10T05:00:02.708009+0000","last_became_active":"2021-02-10T04:50:59.585917+0000","last_became_peered":"2021-02-10T04:50:59.585917+0000","last_unstale":"2021-02-10T05:00:02.708009+0000","last_undegraded":"2021-02-10T05:00:02.708009+0000","last_fullsized":"2021-02-10T05:00:02.708009+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:09.475063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:09.475063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"276","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705699+0000","last_change":"2021-02-10T04:53:34.927483+0000","last_active":"2021-02-10T05:00:02.705699+0000","last_peered":"2021-02-10T05:00:02.705699+0000","last_clean":"2021-02-10T05:00:02.705699+0000","last_became_active":"2021-02-10T04:51:36.859707+0000","last_became_peered":"2021-02-10T04:51:36.859707+0000","last_unstale":"2021-02-10T05:00:02.705699+0000","last_undegraded":"2021-02-10T05:00:02.705699+0000","last_fullsized":"2021-02-10T05:00:02.705699+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:34.927424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.927424+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"1261'40","reported_seq":"1045","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705827+0000","last_change":"2021-02-10T04:59:18.642444+0000","last_active":"2021-02-10T05:00:02.705827+0000","last_peered":"2021-02-10T05:00:02.705827+0000","last_clean":"2021-02-10T05:00:02.705827+0000","last_became_active":"2021-02-10T04:59:18.641966+0000","last_became_peered":"2021-02-10T04:59:18.641966+0000","last_unstale":"2021-02-10T05:00:02.705827+0000","last_undegraded":"2021-02-10T05:00:02.705827+0000","last_fullsized":"2021-02-10T05:00:02.705827+0000","mapping_epoch":1346,"log_start":"1063'30","ondisk_log_start":"1063'30","created":1341,"last_epoch_clean":1347,"parent":"0.0","parent_split_bits":5,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:56.431187+0000","last_deep_scrub":"82'29","last_deep_scrub_stamp":"2021-02-10T04:27:38.176754+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.431187+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":401,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":58,"num_read_kb":47,"num_write":34,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.14","version":"1000'620","reported_seq":"4661","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.704002+0000","last_change":"2021-02-10T04:53:37.384461+0000","last_active":"2021-02-10T05:00:02.704002+0000","last_peered":"2021-02-10T05:00:02.704002+0000","last_clean":"2021-02-10T05:00:02.704002+0000","last_became_active":"2021-02-10T04:51:40.510114+0000","last_became_peered":"2021-02-10T04:51:40.510114+0000","last_unstale":"2021-02-10T05:00:02.704002+0000","last_undegraded":"2021-02-10T05:00:02.704002+0000","last_fullsized":"2021-02-10T05:00:02.704002+0000","mapping_epoch":1122,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1123,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'620","last_scrub_stamp":"2021-02-10T04:53:37.384398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:37.384398+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28530390,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"53'1","reported_seq":"39","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.706108+0000","last_change":"2021-02-10T04:59:51.205792+0000","last_active":"2021-02-10T05:00:02.706108+0000","last_peered":"2021-02-10T05:00:02.706108+0000","last_clean":"2021-02-10T05:00:02.706108+0000","last_became_active":"2021-02-10T04:59:51.122736+0000","last_became_peered":"2021-02-10T04:59:51.122736+0000","last_unstale":"2021-02-10T05:00:02.706108+0000","last_undegraded":"2021-02-10T05:00:02.706108+0000","last_fullsized":"2021-02-10T05:00:02.706108+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:52:41.059159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.059159+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"1365","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.706957+0000","last_change":"2021-02-10T04:53:12.332890+0000","last_active":"2021-02-10T05:00:00.706957+0000","last_peered":"2021-02-10T05:00:00.706957+0000","last_clean":"2021-02-10T05:00:00.706957+0000","last_became_active":"2021-02-10T04:50:59.540375+0000","last_became_peered":"2021-02-10T04:50:59.540375+0000","last_unstale":"2021-02-10T05:00:00.706957+0000","last_undegraded":"2021-02-10T05:00:00.706957+0000","last_fullsized":"2021-02-10T05:00:00.706957+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.332233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.332233+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"377","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.806938+0000","last_change":"2021-02-10T04:59:51.749664+0000","last_active":"2021-02-10T05:00:02.806938+0000","last_peered":"2021-02-10T05:00:02.806938+0000","last_clean":"2021-02-10T05:00:02.806938+0000","last_became_active":"2021-02-10T04:59:51.099036+0000","last_became_peered":"2021-02-10T04:59:51.099036+0000","last_unstale":"2021-02-10T05:00:02.806938+0000","last_undegraded":"2021-02-10T05:00:02.806938+0000","last_fullsized":"2021-02-10T05:00:02.806938+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:59:51.749568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:59:51.749568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"1002'2042","reported_seq":"9347","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.805207+0000","last_change":"2021-02-10T04:53:46.726353+0000","last_active":"2021-02-10T05:00:02.805207+0000","last_peered":"2021-02-10T05:00:02.805207+0000","last_clean":"2021-02-10T05:00:02.805207+0000","last_became_active":"2021-02-10T04:53:46.725828+0000","last_became_peered":"2021-02-10T04:53:46.725828+0000","last_unstale":"2021-02-10T05:00:02.805207+0000","last_undegraded":"2021-02-10T05:00:02.805207+0000","last_fullsized":"2021-02-10T05:00:02.805207+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'2042","last_scrub_stamp":"2021-02-10T04:52:41.286006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.286006+0000","log_size":2042,"ondisk_log_size":2042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46300076,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":545,"num_read_kb":531689,"num_write":399,"num_write_kb":153087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"1261'56","reported_seq":"1545","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.704426+0000","last_change":"2021-02-10T04:52:35.091494+0000","last_active":"2021-02-10T05:00:02.704426+0000","last_peered":"2021-02-10T05:00:02.704426+0000","last_clean":"2021-02-10T05:00:02.704426+0000","last_became_active":"2021-02-10T04:50:59.569915+0000","last_became_peered":"2021-02-10T04:50:59.569915+0000","last_unstale":"2021-02-10T05:00:02.704426+0000","last_undegraded":"2021-02-10T05:00:02.704426+0000","last_fullsized":"2021-02-10T05:00:02.704426+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'46","last_scrub_stamp":"2021-02-10T04:52:35.091374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:35.091374+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":95,"num_read_kb":80,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"84'7","reported_seq":"1389","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.707513+0000","last_change":"2021-02-10T04:51:52.498251+0000","last_active":"2021-02-10T05:00:00.707513+0000","last_peered":"2021-02-10T05:00:00.707513+0000","last_clean":"2021-02-10T05:00:00.707513+0000","last_became_active":"2021-02-10T04:50:59.575924+0000","last_became_peered":"2021-02-10T04:50:59.575924+0000","last_unstale":"2021-02-10T05:00:00.707513+0000","last_undegraded":"2021-02-10T05:00:00.707513+0000","last_fullsized":"2021-02-10T05:00:00.707513+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'7","last_scrub_stamp":"2021-02-10T04:51:52.498158+0000","last_deep_scrub":"84'7","last_deep_scrub_stamp":"2021-02-10T04:27:54.402855+0000","last_clean_scrub_stamp":"2021-02-10T04:51:52.498158+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"1333'6","reported_seq":"1794","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.706433+0000","last_change":"2021-02-10T04:58:57.207618+0000","last_active":"2021-02-10T05:00:00.706433+0000","last_peered":"2021-02-10T05:00:00.706433+0000","last_clean":"2021-02-10T05:00:00.706433+0000","last_became_active":"2021-02-10T04:58:56.849778+0000","last_became_peered":"2021-02-10T04:58:56.849778+0000","last_unstale":"2021-02-10T05:00:00.706433+0000","last_undegraded":"2021-02-10T05:00:00.706433+0000","last_fullsized":"2021-02-10T05:00:00.706433+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"1317'5","last_scrub_stamp":"2021-02-10T04:58:57.207527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:58:57.207527+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"1280","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.806785+0000","last_change":"2021-02-10T04:59:08.653632+0000","last_active":"2021-02-10T05:00:02.806785+0000","last_peered":"2021-02-10T05:00:02.806785+0000","last_clean":"2021-02-10T05:00:02.806785+0000","last_became_active":"2021-02-10T04:58:56.965228+0000","last_became_peered":"2021-02-10T04:58:56.965228+0000","last_unstale":"2021-02-10T05:00:02.806785+0000","last_undegraded":"2021-02-10T05:00:02.806785+0000","last_fullsized":"2021-02-10T05:00:02.806785+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:23.566027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.697677+0000","last_clean_scrub_stamp":"2021-02-10T04:52:23.566027+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"1000'490","reported_seq":"3910","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.706389+0000","last_change":"2021-02-10T04:59:58.274981+0000","last_active":"2021-02-10T05:00:02.706389+0000","last_peered":"2021-02-10T05:00:02.706389+0000","last_clean":"2021-02-10T05:00:02.706389+0000","last_became_active":"2021-02-10T04:58:56.965659+0000","last_became_peered":"2021-02-10T04:58:56.965659+0000","last_unstale":"2021-02-10T05:00:02.706389+0000","last_undegraded":"2021-02-10T05:00:02.706389+0000","last_fullsized":"2021-02-10T05:00:02.706389+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'490","last_scrub_stamp":"2021-02-10T04:59:58.274905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:59:58.274905+0000","log_size":490,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16154790,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"1261'46","reported_seq":"1442","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.706231+0000","last_change":"2021-02-10T04:53:10.276449+0000","last_active":"2021-02-10T05:00:00.706231+0000","last_peered":"2021-02-10T05:00:00.706231+0000","last_clean":"2021-02-10T05:00:00.706231+0000","last_became_active":"2021-02-10T04:50:59.539529+0000","last_became_peered":"2021-02-10T04:50:59.539529+0000","last_unstale":"2021-02-10T05:00:00.706231+0000","last_undegraded":"2021-02-10T05:00:00.706231+0000","last_fullsized":"2021-02-10T05:00:00.706231+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"1112'40","last_scrub_stamp":"2021-02-10T04:53:10.276358+0000","last_deep_scrub":"1112'40","last_deep_scrub_stamp":"2021-02-10T04:51:01.450050+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.276358+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":53,"num_write":38,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"1265","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.806619+0000","last_change":"2021-02-10T04:59:08.645662+0000","last_active":"2021-02-10T05:00:02.806619+0000","last_peered":"2021-02-10T05:00:02.806619+0000","last_clean":"2021-02-10T05:00:02.806619+0000","last_became_active":"2021-02-10T04:58:56.970118+0000","last_became_peered":"2021-02-10T04:58:56.970118+0000","last_unstale":"2021-02-10T05:00:02.806619+0000","last_undegraded":"2021-02-10T05:00:02.806619+0000","last_fullsized":"2021-02-10T05:00:02.806619+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:23.904210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:39.616381+0000","last_clean_scrub_stamp":"2021-02-10T04:47:23.904210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"1119'6","reported_seq":"1689","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:03.088663+0000","last_change":"2021-02-10T04:52:40.038646+0000","last_active":"2021-02-10T05:00:03.088663+0000","last_peered":"2021-02-10T05:00:03.088663+0000","last_clean":"2021-02-10T05:00:03.088663+0000","last_became_active":"2021-02-10T04:51:36.869733+0000","last_became_peered":"2021-02-10T04:51:36.869733+0000","last_unstale":"2021-02-10T05:00:03.088663+0000","last_undegraded":"2021-02-10T05:00:03.088663+0000","last_fullsized":"2021-02-10T05:00:03.088663+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'6","last_scrub_stamp":"2021-02-10T04:52:40.038554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:52:40.038554+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"1365","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:03.233010+0000","last_change":"2021-02-10T05:00:03.232943+0000","last_active":"2021-02-10T05:00:03.233010+0000","last_peered":"2021-02-10T05:00:03.233010+0000","last_clean":"2021-02-10T05:00:03.233010+0000","last_became_active":"2021-02-10T04:51:36.849184+0000","last_became_peered":"2021-02-10T04:51:36.849184+0000","last_unstale":"2021-02-10T05:00:03.233010+0000","last_undegraded":"2021-02-10T05:00:03.233010+0000","last_fullsized":"2021-02-10T05:00:03.233010+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T05:00:03.232871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T05:00:03.232871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"1032'89","reported_seq":"991","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.805285+0000","last_change":"2021-02-10T04:59:51.521944+0000","last_active":"2021-02-10T05:00:02.805285+0000","last_peered":"2021-02-10T05:00:02.805285+0000","last_clean":"2021-02-10T05:00:02.805285+0000","last_became_active":"2021-02-10T04:59:51.116954+0000","last_became_peered":"2021-02-10T04:59:51.116954+0000","last_unstale":"2021-02-10T05:00:02.805285+0000","last_undegraded":"2021-02-10T05:00:02.805285+0000","last_fullsized":"2021-02-10T05:00:02.805285+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":5,"last_scrub":"1032'89","last_scrub_stamp":"2021-02-10T04:52:34.089789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:52:34.089789+0000","log_size":89,"ondisk_log_size":89,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5218574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":30326,"num_write":21,"num_write_kb":12444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":9159733,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"1261'43","reported_seq":"993","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.704890+0000","last_change":"2021-02-10T04:59:51.202467+0000","last_active":"2021-02-10T05:00:02.704890+0000","last_peered":"2021-02-10T05:00:02.704890+0000","last_clean":"2021-02-10T05:00:02.704890+0000","last_became_active":"2021-02-10T04:59:51.144449+0000","last_became_peered":"2021-02-10T04:59:51.144449+0000","last_unstale":"2021-02-10T05:00:02.704890+0000","last_undegraded":"2021-02-10T05:00:02.704890+0000","last_fullsized":"2021-02-10T05:00:02.704890+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"1163'37","last_scrub_stamp":"2021-02-10T04:53:14.202445+0000","last_deep_scrub":"1163'37","last_deep_scrub_stamp":"2021-02-10T04:53:14.202445+0000","last_clean_scrub_stamp":"2021-02-10T04:53:14.202445+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":59,"num_read_kb":41,"num_write":34,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"1121","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.704587+0000","last_change":"2021-02-10T04:59:51.125357+0000","last_active":"2021-02-10T05:00:02.704587+0000","last_peered":"2021-02-10T05:00:02.704587+0000","last_clean":"2021-02-10T05:00:02.704587+0000","last_became_active":"2021-02-10T04:59:51.124981+0000","last_became_peered":"2021-02-10T04:59:51.124981+0000","last_unstale":"2021-02-10T05:00:02.704587+0000","last_undegraded":"2021-02-10T05:00:02.704587+0000","last_fullsized":"2021-02-10T05:00:02.704587+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:23.095588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:23.095588+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"1340","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.703664+0000","last_change":"2021-02-10T04:58:56.965982+0000","last_active":"2021-02-10T05:00:02.703664+0000","last_peered":"2021-02-10T05:00:02.703664+0000","last_clean":"2021-02-10T05:00:02.703664+0000","last_became_active":"2021-02-10T04:58:56.964476+0000","last_became_peered":"2021-02-10T04:58:56.964476+0000","last_unstale":"2021-02-10T05:00:02.703664+0000","last_undegraded":"2021-02-10T05:00:02.703664+0000","last_fullsized":"2021-02-10T05:00:02.703664+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:43.663762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:05.297647+0000","last_clean_scrub_stamp":"2021-02-10T04:51:43.663762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"1236","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.805178+0000","last_change":"2021-02-10T04:59:08.655593+0000","last_active":"2021-02-10T05:00:02.805178+0000","last_peered":"2021-02-10T05:00:02.805178+0000","last_clean":"2021-02-10T05:00:02.805178+0000","last_became_active":"2021-02-10T04:58:56.877585+0000","last_became_peered":"2021-02-10T04:58:56.877585+0000","last_unstale":"2021-02-10T05:00:02.805178+0000","last_undegraded":"2021-02-10T05:00:02.805178+0000","last_fullsized":"2021-02-10T05:00:02.805178+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:57.568541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:57.568541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"1000'1071","reported_seq":"4880","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.702197+0000","last_change":"2021-02-10T04:53:35.302879+0000","last_active":"2021-02-10T05:00:02.702197+0000","last_peered":"2021-02-10T05:00:02.702197+0000","last_clean":"2021-02-10T05:00:02.702197+0000","last_became_active":"2021-02-10T04:51:39.508551+0000","last_became_peered":"2021-02-10T04:51:39.508551+0000","last_unstale":"2021-02-10T05:00:02.702197+0000","last_undegraded":"2021-02-10T05:00:02.702197+0000","last_fullsized":"2021-02-10T05:00:02.702197+0000","mapping_epoch":1121,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1122,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:35.302780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.302780+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"1261'39","reported_seq":"1438","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705657+0000","last_change":"2021-02-10T04:59:51.220684+0000","last_active":"2021-02-10T05:00:02.705657+0000","last_peered":"2021-02-10T05:00:02.705657+0000","last_clean":"2021-02-10T05:00:02.705657+0000","last_became_active":"2021-02-10T04:59:51.122966+0000","last_became_peered":"2021-02-10T04:59:51.122966+0000","last_unstale":"2021-02-10T05:00:02.705657+0000","last_undegraded":"2021-02-10T05:00:02.705657+0000","last_fullsized":"2021-02-10T05:00:02.705657+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'31","last_scrub_stamp":"2021-02-10T04:52:39.695237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:52:39.695237+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":53,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"51'2","reported_seq":"637","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:02.625565+0000","last_change":"2021-02-10T05:00:02.625565+0000","last_active":"2021-02-10T05:00:02.625565+0000","last_peered":"2021-02-10T05:00:02.625565+0000","last_clean":"2021-02-10T05:00:02.625565+0000","last_became_active":"2021-02-10T04:58:56.964387+0000","last_became_peered":"2021-02-10T04:58:56.964387+0000","last_unstale":"2021-02-10T05:00:02.625565+0000","last_undegraded":"2021-02-10T05:00:02.625565+0000","last_fullsized":"2021-02-10T05:00:02.625565+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'2","last_scrub_stamp":"2021-02-10T05:00:02.625497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T05:00:02.625497+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"825","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.707674+0000","last_change":"2021-02-10T04:59:40.662976+0000","last_active":"2021-02-10T05:00:00.707674+0000","last_peered":"2021-02-10T05:00:00.707674+0000","last_clean":"2021-02-10T05:00:00.707674+0000","last_became_active":"2021-02-10T04:59:40.662450+0000","last_became_peered":"2021-02-10T04:59:40.662450+0000","last_unstale":"2021-02-10T05:00:00.707674+0000","last_undegraded":"2021-02-10T05:00:00.707674+0000","last_fullsized":"2021-02-10T05:00:00.707674+0000","mapping_epoch":1356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.216394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:33.029912+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.216394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"1367","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.702557+0000","last_change":"2021-02-10T04:58:56.879817+0000","last_active":"2021-02-10T05:00:02.702557+0000","last_peered":"2021-02-10T05:00:02.702557+0000","last_clean":"2021-02-10T05:00:02.702557+0000","last_became_active":"2021-02-10T04:58:56.879372+0000","last_became_peered":"2021-02-10T04:58:56.879372+0000","last_unstale":"2021-02-10T05:00:02.702557+0000","last_undegraded":"2021-02-10T05:00:02.702557+0000","last_fullsized":"2021-02-10T05:00:02.702557+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:55.814486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.814486+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"1002'1657","reported_seq":"5393","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.806540+0000","last_change":"2021-02-10T04:59:08.654625+0000","last_active":"2021-02-10T05:00:02.806540+0000","last_peered":"2021-02-10T05:00:02.806540+0000","last_clean":"2021-02-10T05:00:02.806540+0000","last_became_active":"2021-02-10T04:58:56.961771+0000","last_became_peered":"2021-02-10T04:58:56.961771+0000","last_unstale":"2021-02-10T05:00:02.806540+0000","last_undegraded":"2021-02-10T05:00:02.806540+0000","last_fullsized":"2021-02-10T05:00:02.806540+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:55:22.331010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:55:22.331010+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":47335497,"num_objects":20,"num_object_clones":18,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"1261'24","reported_seq":"1513","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.702483+0000","last_change":"2021-02-10T04:59:51.179096+0000","last_active":"2021-02-10T05:00:02.702483+0000","last_peered":"2021-02-10T05:00:02.702483+0000","last_clean":"2021-02-10T05:00:02.702483+0000","last_became_active":"2021-02-10T04:59:51.114167+0000","last_became_peered":"2021-02-10T04:59:51.114167+0000","last_unstale":"2021-02-10T05:00:02.702483+0000","last_undegraded":"2021-02-10T05:00:02.702483+0000","last_fullsized":"2021-02-10T05:00:02.702483+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"1048'20","last_scrub_stamp":"2021-02-10T04:53:04.058248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:04.058248+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":29,"num_write":24,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"1386","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.707183+0000","last_change":"2021-02-10T04:59:51.115470+0000","last_active":"2021-02-10T05:00:00.707183+0000","last_peered":"2021-02-10T05:00:00.707183+0000","last_clean":"2021-02-10T05:00:00.707183+0000","last_became_active":"2021-02-10T04:59:51.114568+0000","last_became_peered":"2021-02-10T04:59:51.114568+0000","last_unstale":"2021-02-10T05:00:00.707183+0000","last_undegraded":"2021-02-10T05:00:00.707183+0000","last_fullsized":"2021-02-10T05:00:00.707183+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:00.343228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:00.343228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"28","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705137+0000","last_change":"2021-02-10T05:00:00.267680+0000","last_active":"2021-02-10T05:00:02.705137+0000","last_peered":"2021-02-10T05:00:02.705137+0000","last_clean":"2021-02-10T05:00:02.705137+0000","last_became_active":"2021-02-10T04:59:51.103624+0000","last_became_peered":"2021-02-10T04:59:51.103624+0000","last_unstale":"2021-02-10T05:00:02.705137+0000","last_undegraded":"2021-02-10T05:00:02.705137+0000","last_fullsized":"2021-02-10T05:00:02.705137+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T05:00:00.267589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.077777+0000","last_clean_scrub_stamp":"2021-02-10T05:00:00.267589+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"693","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.804924+0000","last_change":"2021-02-10T04:53:46.718715+0000","last_active":"2021-02-10T05:00:02.804924+0000","last_peered":"2021-02-10T05:00:02.804924+0000","last_clean":"2021-02-10T05:00:02.804924+0000","last_became_active":"2021-02-10T04:53:46.718266+0000","last_became_peered":"2021-02-10T04:53:46.718266+0000","last_unstale":"2021-02-10T05:00:02.804924+0000","last_undegraded":"2021-02-10T05:00:02.804924+0000","last_fullsized":"2021-02-10T05:00:02.804924+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.876681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:19.134212+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.876681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.12","version":"998'1009","reported_seq":"4664","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.805737+0000","last_change":"2021-02-10T04:59:53.667158+0000","last_active":"2021-02-10T05:00:02.805737+0000","last_peered":"2021-02-10T05:00:02.805737+0000","last_clean":"2021-02-10T05:00:02.805737+0000","last_became_active":"2021-02-10T04:59:53.665843+0000","last_became_peered":"2021-02-10T04:59:53.665843+0000","last_unstale":"2021-02-10T05:00:02.805737+0000","last_undegraded":"2021-02-10T05:00:02.805737+0000","last_fullsized":"2021-02-10T05:00:02.805737+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":5,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T04:47:13.032506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:13.032506+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24710414,"num_objects":11,"num_object_clones":10,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":239,"num_read_kb":249403,"num_write":155,"num_write_kb":50104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":45053856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"1261'45","reported_seq":"1445","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.703058+0000","last_change":"2021-02-10T04:59:51.177904+0000","last_active":"2021-02-10T05:00:02.703058+0000","last_peered":"2021-02-10T05:00:02.703058+0000","last_clean":"2021-02-10T05:00:02.703058+0000","last_became_active":"2021-02-10T04:59:51.119425+0000","last_became_peered":"2021-02-10T04:59:51.119425+0000","last_unstale":"2021-02-10T05:00:02.703058+0000","last_undegraded":"2021-02-10T05:00:02.703058+0000","last_fullsized":"2021-02-10T05:00:02.703058+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"1049'37","last_scrub_stamp":"2021-02-10T04:51:44.694711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.694711+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":76,"num_read_kb":62,"num_write":45,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"50'1","reported_seq":"34","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.705562+0000","last_change":"2021-02-10T04:59:51.182394+0000","last_active":"2021-02-10T05:00:00.705562+0000","last_peered":"2021-02-10T05:00:00.705562+0000","last_clean":"2021-02-10T05:00:00.705562+0000","last_became_active":"2021-02-10T04:59:51.120643+0000","last_became_peered":"2021-02-10T04:59:51.120643+0000","last_unstale":"2021-02-10T05:00:00.705562+0000","last_undegraded":"2021-02-10T05:00:00.705562+0000","last_fullsized":"2021-02-10T05:00:00.705562+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"50'1","last_scrub_stamp":"2021-02-10T04:52:33.088428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:33.088428+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"669","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.702489+0000","last_change":"2021-02-10T04:59:51.110291+0000","last_active":"2021-02-10T05:00:02.702489+0000","last_peered":"2021-02-10T05:00:02.702489+0000","last_clean":"2021-02-10T05:00:02.702489+0000","last_became_active":"2021-02-10T04:59:51.109910+0000","last_became_peered":"2021-02-10T04:59:51.109910+0000","last_unstale":"2021-02-10T05:00:02.702489+0000","last_undegraded":"2021-02-10T05:00:02.702489+0000","last_fullsized":"2021-02-10T05:00:02.702489+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:51.836638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.836638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"1340","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.703594+0000","last_change":"2021-02-10T04:59:51.130995+0000","last_active":"2021-02-10T05:00:02.703594+0000","last_peered":"2021-02-10T05:00:02.703594+0000","last_clean":"2021-02-10T05:00:02.703594+0000","last_became_active":"2021-02-10T04:59:51.130617+0000","last_became_peered":"2021-02-10T04:59:51.130617+0000","last_unstale":"2021-02-10T05:00:02.703594+0000","last_undegraded":"2021-02-10T05:00:02.703594+0000","last_fullsized":"2021-02-10T05:00:02.703594+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:58.514318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.299942+0000","last_clean_scrub_stamp":"2021-02-10T04:46:58.514318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"1261'65","reported_seq":"1404","reported_epoch":"1372","state":"active+remapped+backfill_toofull","last_fresh":"2021-02-10T04:59:59.693500+0000","last_change":"2021-02-10T04:59:46.979060+0000","last_active":"2021-02-10T04:59:59.693500+0000","last_peered":"2021-02-10T04:59:59.693500+0000","last_clean":"2021-02-10T04:55:57.139329+0000","last_became_active":"2021-02-10T04:59:16.972842+0000","last_became_peered":"2021-02-10T04:59:16.972842+0000","last_unstale":"2021-02-10T04:59:59.693500+0000","last_undegraded":"2021-02-10T04:59:59.693500+0000","last_fullsized":"2021-02-10T04:59:59.693500+0000","mapping_epoch":1344,"log_start":"1049'47","ondisk_log_start":"1049'47","created":1341,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":5,"last_scrub":"1049'47","last_scrub_stamp":"2021-02-10T04:53:09.002368+0000","last_deep_scrub":"86'31","last_deep_scrub_stamp":"2021-02-10T04:28:11.693616+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.002368+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":836,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":84,"num_read_kb":67,"num_write":49,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":467,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[1,4,6],"avail_no_missing":["1","4","6"],"object_location_counts":[{"shards":"1,4,6","objects":11}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"1001'1165","reported_seq":"4405","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:59.693527+0000","last_change":"2021-02-10T04:59:53.658749+0000","last_active":"2021-02-10T04:59:59.693527+0000","last_peered":"2021-02-10T04:59:59.693527+0000","last_clean":"2021-02-10T04:59:59.693527+0000","last_became_active":"2021-02-10T04:59:53.658313+0000","last_became_peered":"2021-02-10T04:59:53.658313+0000","last_unstale":"2021-02-10T04:59:59.693527+0000","last_undegraded":"2021-02-10T04:59:59.693527+0000","last_fullsized":"2021-02-10T04:59:59.693527+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":5,"last_scrub":"1001'1165","last_scrub_stamp":"2021-02-10T04:51:03.221254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:03.221254+0000","log_size":1165,"ondisk_log_size":1165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29601845,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":287,"num_read_kb":299530,"num_write":200,"num_write_kb":89131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"1418","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.702022+0000","last_change":"2021-02-10T04:53:01.331566+0000","last_active":"2021-02-10T05:00:00.702022+0000","last_peered":"2021-02-10T05:00:00.702022+0000","last_clean":"2021-02-10T05:00:00.702022+0000","last_became_active":"2021-02-10T04:25:36.338136+0000","last_became_peered":"2021-02-10T04:25:36.338136+0000","last_unstale":"2021-02-10T05:00:00.702022+0000","last_undegraded":"2021-02-10T05:00:00.702022+0000","last_fullsized":"2021-02-10T05:00:00.702022+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:01.331466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.331466+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"1366'4","reported_seq":"1390","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:03.088293+0000","last_change":"2021-02-10T04:59:51.169693+0000","last_active":"2021-02-10T05:00:03.088293+0000","last_peered":"2021-02-10T05:00:03.088293+0000","last_clean":"2021-02-10T05:00:03.088293+0000","last_became_active":"2021-02-10T04:59:51.112805+0000","last_became_peered":"2021-02-10T04:59:51.112805+0000","last_unstale":"2021-02-10T05:00:03.088293+0000","last_undegraded":"2021-02-10T05:00:03.088293+0000","last_fullsized":"2021-02-10T05:00:03.088293+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"1098'3","last_scrub_stamp":"2021-02-10T04:53:34.865579+0000","last_deep_scrub":"1098'3","last_deep_scrub_stamp":"2021-02-10T04:50:15.255137+0000","last_clean_scrub_stamp":"2021-02-10T04:53:34.865579+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"1223","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.804905+0000","last_change":"2021-02-10T04:53:46.727603+0000","last_active":"2021-02-10T05:00:02.804905+0000","last_peered":"2021-02-10T05:00:02.804905+0000","last_clean":"2021-02-10T05:00:02.804905+0000","last_became_active":"2021-02-10T04:53:46.727053+0000","last_became_peered":"2021-02-10T04:53:46.727053+0000","last_unstale":"2021-02-10T05:00:02.804905+0000","last_undegraded":"2021-02-10T05:00:02.804905+0000","last_fullsized":"2021-02-10T05:00:02.804905+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:47.493057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:47.493057+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"1261'71","reported_seq":"1179","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.705668+0000","last_change":"2021-02-10T04:59:53.657664+0000","last_active":"2021-02-10T05:00:00.705668+0000","last_peered":"2021-02-10T05:00:00.705668+0000","last_clean":"2021-02-10T05:00:00.705668+0000","last_became_active":"2021-02-10T04:59:53.657339+0000","last_became_peered":"2021-02-10T04:59:53.657339+0000","last_unstale":"2021-02-10T05:00:00.705668+0000","last_undegraded":"2021-02-10T05:00:00.705668+0000","last_fullsized":"2021-02-10T05:00:00.705668+0000","mapping_epoch":1366,"log_start":"1049'51","ondisk_log_start":"1049'51","created":1341,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":5,"last_scrub":"1049'41","last_scrub_stamp":"2021-02-10T04:51:58.353660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:58.353660+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":85,"num_read_kb":68,"num_write":51,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":2625,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1c","version":"1002'1090","reported_seq":"5398","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.706643+0000","last_change":"2021-02-10T04:53:41.353991+0000","last_active":"2021-02-10T05:00:00.706643+0000","last_peered":"2021-02-10T05:00:00.706643+0000","last_clean":"2021-02-10T05:00:00.706643+0000","last_became_active":"2021-02-10T04:52:31.088265+0000","last_became_peered":"2021-02-10T04:52:31.088265+0000","last_unstale":"2021-02-10T05:00:00.706643+0000","last_undegraded":"2021-02-10T05:00:00.706643+0000","last_fullsized":"2021-02-10T05:00:00.706643+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":5,"last_scrub":"1002'1090","last_scrub_stamp":"2021-02-10T04:53:41.353840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:41.353840+0000","log_size":1090,"ondisk_log_size":1090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"1344","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.702890+0000","last_change":"2021-02-10T04:59:51.130114+0000","last_active":"2021-02-10T05:00:02.702890+0000","last_peered":"2021-02-10T05:00:02.702890+0000","last_clean":"2021-02-10T05:00:02.702890+0000","last_became_active":"2021-02-10T04:59:51.129732+0000","last_became_peered":"2021-02-10T04:59:51.129732+0000","last_unstale":"2021-02-10T05:00:02.702890+0000","last_undegraded":"2021-02-10T05:00:02.702890+0000","last_fullsized":"2021-02-10T05:00:02.702890+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:46:42.537443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:46:42.537443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"1261","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.805765+0000","last_change":"2021-02-10T04:59:08.666315+0000","last_active":"2021-02-10T05:00:02.805765+0000","last_peered":"2021-02-10T05:00:02.805765+0000","last_clean":"2021-02-10T05:00:02.805765+0000","last_became_active":"2021-02-10T04:58:56.869951+0000","last_became_peered":"2021-02-10T04:58:56.869951+0000","last_unstale":"2021-02-10T05:00:02.805765+0000","last_undegraded":"2021-02-10T05:00:02.805765+0000","last_fullsized":"2021-02-10T05:00:02.805765+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.938731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.938731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"810","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.706639+0000","last_change":"2021-02-10T04:59:40.659174+0000","last_active":"2021-02-10T05:00:00.706639+0000","last_peered":"2021-02-10T05:00:00.706639+0000","last_clean":"2021-02-10T05:00:00.706639+0000","last_became_active":"2021-02-10T04:59:40.658818+0000","last_became_peered":"2021-02-10T04:59:40.658818+0000","last_unstale":"2021-02-10T05:00:00.706639+0000","last_undegraded":"2021-02-10T05:00:00.706639+0000","last_fullsized":"2021-02-10T05:00:00.706639+0000","mapping_epoch":1356,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:15.185686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:15.185686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"1261'76","reported_seq":"1445","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705269+0000","last_change":"2021-02-10T04:59:48.064635+0000","last_active":"2021-02-10T05:00:02.705269+0000","last_peered":"2021-02-10T05:00:02.705269+0000","last_clean":"2021-02-10T05:00:02.705269+0000","last_became_active":"2021-02-10T04:59:48.064161+0000","last_became_peered":"2021-02-10T04:59:48.064161+0000","last_unstale":"2021-02-10T05:00:02.705269+0000","last_undegraded":"2021-02-10T05:00:02.705269+0000","last_fullsized":"2021-02-10T05:00:02.705269+0000","mapping_epoch":1360,"log_start":"1049'39","ondisk_log_start":"1049'39","created":1341,"last_epoch_clean":1361,"parent":"0.0","parent_split_bits":5,"last_scrub":"1187'54","last_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_deep_scrub":"1187'54","last_deep_scrub_stamp":"2021-02-10T04:53:42.071103+0000","last_clean_scrub_stamp":"2021-02-10T04:53:42.071103+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":289,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":73,"num_read_kb":63,"num_write":48,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1f","version":"1004'1093","reported_seq":"6592","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705434+0000","last_change":"2021-02-10T04:59:57.679787+0000","last_active":"2021-02-10T05:00:02.705434+0000","last_peered":"2021-02-10T05:00:02.705434+0000","last_clean":"2021-02-10T05:00:02.705434+0000","last_became_active":"2021-02-10T04:59:57.678313+0000","last_became_peered":"2021-02-10T04:59:57.678313+0000","last_unstale":"2021-02-10T05:00:02.705434+0000","last_undegraded":"2021-02-10T05:00:02.705434+0000","last_fullsized":"2021-02-10T05:00:02.705434+0000","mapping_epoch":1370,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1371,"parent":"0.0","parent_split_bits":5,"last_scrub":"1004'1093","last_scrub_stamp":"2021-02-10T04:51:49.332995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:49.332995+0000","log_size":1093,"ondisk_log_size":1093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":39434697,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":279,"num_read_kb":311058,"num_write":186,"num_write_kb":66790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":53995913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"51'1","reported_seq":"39","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.706468+0000","last_change":"2021-02-10T04:59:51.137374+0000","last_active":"2021-02-10T05:00:02.706468+0000","last_peered":"2021-02-10T05:00:02.706468+0000","last_clean":"2021-02-10T05:00:02.706468+0000","last_became_active":"2021-02-10T04:59:51.118991+0000","last_became_peered":"2021-02-10T04:59:51.118991+0000","last_unstale":"2021-02-10T05:00:02.706468+0000","last_undegraded":"2021-02-10T05:00:02.706468+0000","last_fullsized":"2021-02-10T05:00:02.706468+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'1","last_scrub_stamp":"2021-02-10T04:52:36.083784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:36.083784+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"815","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705830+0000","last_change":"2021-02-10T05:00:01.225550+0000","last_active":"2021-02-10T05:00:02.705830+0000","last_peered":"2021-02-10T05:00:02.705830+0000","last_clean":"2021-02-10T05:00:02.705830+0000","last_became_active":"2021-02-10T04:59:40.661806+0000","last_became_peered":"2021-02-10T04:59:40.661806+0000","last_unstale":"2021-02-10T05:00:02.705830+0000","last_undegraded":"2021-02-10T05:00:02.705830+0000","last_fullsized":"2021-02-10T05:00:02.705830+0000","mapping_epoch":1356,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T05:00:01.225483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T05:00:01.225483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"1351","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.706003+0000","last_change":"2021-02-10T04:59:51.126018+0000","last_active":"2021-02-10T05:00:02.706003+0000","last_peered":"2021-02-10T05:00:02.706003+0000","last_clean":"2021-02-10T05:00:02.706003+0000","last_became_active":"2021-02-10T04:59:51.125671+0000","last_became_peered":"2021-02-10T04:59:51.125671+0000","last_unstale":"2021-02-10T05:00:02.706003+0000","last_undegraded":"2021-02-10T05:00:02.706003+0000","last_fullsized":"2021-02-10T05:00:02.706003+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:44.799429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:44.799429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"1261'64","reported_seq":"1515","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.704358+0000","last_change":"2021-02-10T04:59:18.642028+0000","last_active":"2021-02-10T05:00:02.704358+0000","last_peered":"2021-02-10T05:00:02.704358+0000","last_clean":"2021-02-10T05:00:02.704358+0000","last_became_active":"2021-02-10T04:59:18.641628+0000","last_became_peered":"2021-02-10T04:59:18.641628+0000","last_unstale":"2021-02-10T05:00:02.704358+0000","last_undegraded":"2021-02-10T05:00:02.704358+0000","last_fullsized":"2021-02-10T05:00:02.704358+0000","mapping_epoch":1346,"log_start":"1066'44","ondisk_log_start":"1066'44","created":1341,"last_epoch_clean":1347,"parent":"0.0","parent_split_bits":5,"last_scrub":"1066'44","last_scrub_stamp":"2021-02-10T04:53:37.108677+0000","last_deep_scrub":"113'33","last_deep_scrub_stamp":"2021-02-10T04:28:49.514984+0000","last_clean_scrub_stamp":"2021-02-10T04:53:37.108677+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":789,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":90,"num_read_kb":72,"num_write":53,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":2000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"1063'654","reported_seq":"4403","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705081+0000","last_change":"2021-02-10T04:59:40.665470+0000","last_active":"2021-02-10T05:00:02.705081+0000","last_peered":"2021-02-10T05:00:02.705081+0000","last_clean":"2021-02-10T05:00:02.705081+0000","last_became_active":"2021-02-10T04:59:40.665009+0000","last_became_peered":"2021-02-10T04:59:40.665009+0000","last_unstale":"2021-02-10T05:00:02.705081+0000","last_undegraded":"2021-02-10T05:00:02.705081+0000","last_fullsized":"2021-02-10T05:00:02.705081+0000","mapping_epoch":1356,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1357,"parent":"0.0","parent_split_bits":5,"last_scrub":"1063'654","last_scrub_stamp":"2021-02-10T04:53:40.135367+0000","last_deep_scrub":"1063'654","last_deep_scrub_stamp":"2021-02-10T04:47:06.861811+0000","last_clean_scrub_stamp":"2021-02-10T04:53:40.135367+0000","log_size":654,"ondisk_log_size":654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24700631,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":171,"num_read_kb":182272,"num_write":112,"num_write_kb":33727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":26349024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":"1335","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.704207+0000","last_change":"2021-02-10T04:59:51.120413+0000","last_active":"2021-02-10T05:00:02.704207+0000","last_peered":"2021-02-10T05:00:02.704207+0000","last_clean":"2021-02-10T05:00:02.704207+0000","last_became_active":"2021-02-10T04:59:51.120031+0000","last_became_peered":"2021-02-10T04:59:51.120031+0000","last_unstale":"2021-02-10T05:00:02.704207+0000","last_undegraded":"2021-02-10T05:00:02.704207+0000","last_fullsized":"2021-02-10T05:00:02.704207+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:02.455702+0000","last_clean_scrub_stamp":"2021-02-10T04:51:02.455702+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"1377","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.706075+0000","last_change":"2021-02-10T04:52:33.741801+0000","last_active":"2021-02-10T05:00:02.706075+0000","last_peered":"2021-02-10T05:00:02.706075+0000","last_clean":"2021-02-10T05:00:02.706075+0000","last_became_active":"2021-02-10T04:51:36.860129+0000","last_became_peered":"2021-02-10T04:51:36.860129+0000","last_unstale":"2021-02-10T05:00:02.706075+0000","last_undegraded":"2021-02-10T05:00:02.706075+0000","last_fullsized":"2021-02-10T05:00:02.706075+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:33.741711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:46:58.666264+0000","last_clean_scrub_stamp":"2021-02-10T04:52:33.741711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"1347","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.706196+0000","last_change":"2021-02-10T04:59:51.111726+0000","last_active":"2021-02-10T05:00:02.706196+0000","last_peered":"2021-02-10T05:00:02.706196+0000","last_clean":"2021-02-10T05:00:02.706196+0000","last_became_active":"2021-02-10T04:59:51.111316+0000","last_became_peered":"2021-02-10T04:59:51.111316+0000","last_unstale":"2021-02-10T05:00:02.706196+0000","last_undegraded":"2021-02-10T05:00:02.706196+0000","last_fullsized":"2021-02-10T05:00:02.706196+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:25.030281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:25.030281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"1261'69","reported_seq":"1496","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705135+0000","last_change":"2021-02-10T04:59:48.070593+0000","last_active":"2021-02-10T05:00:02.705135+0000","last_peered":"2021-02-10T05:00:02.705135+0000","last_clean":"2021-02-10T05:00:02.705135+0000","last_became_active":"2021-02-10T04:59:48.069866+0000","last_became_peered":"2021-02-10T04:59:48.069866+0000","last_unstale":"2021-02-10T05:00:02.705135+0000","last_undegraded":"2021-02-10T05:00:02.705135+0000","last_fullsized":"2021-02-10T05:00:02.705135+0000","mapping_epoch":1360,"log_start":"1112'47","ondisk_log_start":"1112'47","created":1341,"last_epoch_clean":1361,"parent":"0.0","parent_split_bits":5,"last_scrub":"1049'61","last_scrub_stamp":"2021-02-10T04:53:36.836202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:36.836202+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":733,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":104,"num_read_kb":86,"num_write":61,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1101,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.19","version":"1003'739","reported_seq":"3750","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.805493+0000","last_change":"2021-02-10T04:59:54.663625+0000","last_active":"2021-02-10T05:00:02.805493+0000","last_peered":"2021-02-10T05:00:02.805493+0000","last_clean":"2021-02-10T05:00:02.805493+0000","last_became_active":"2021-02-10T04:59:54.662899+0000","last_became_peered":"2021-02-10T04:59:54.662899+0000","last_unstale":"2021-02-10T05:00:02.805493+0000","last_undegraded":"2021-02-10T05:00:02.805493+0000","last_fullsized":"2021-02-10T05:00:02.805493+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'739","last_scrub_stamp":"2021-02-10T04:53:01.929169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:01.929169+0000","log_size":739,"ondisk_log_size":739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24059889,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":130,"num_read_kb":109544,"num_write":102,"num_write_kb":46174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":18429397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":"1214","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.805212+0000","last_change":"2021-02-10T04:53:46.727590+0000","last_active":"2021-02-10T05:00:02.805212+0000","last_peered":"2021-02-10T05:00:02.805212+0000","last_clean":"2021-02-10T05:00:02.805212+0000","last_became_active":"2021-02-10T04:53:46.726584+0000","last_became_peered":"2021-02-10T04:53:46.726584+0000","last_unstale":"2021-02-10T05:00:02.805212+0000","last_undegraded":"2021-02-10T05:00:02.805212+0000","last_fullsized":"2021-02-10T05:00:02.805212+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:26.408326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.408326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"1374","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.706028+0000","last_change":"2021-02-10T04:53:07.324650+0000","last_active":"2021-02-10T05:00:00.706028+0000","last_peered":"2021-02-10T05:00:00.706028+0000","last_clean":"2021-02-10T05:00:00.706028+0000","last_became_active":"2021-02-10T04:50:59.575340+0000","last_became_peered":"2021-02-10T04:50:59.575340+0000","last_unstale":"2021-02-10T05:00:00.706028+0000","last_undegraded":"2021-02-10T05:00:00.706028+0000","last_fullsized":"2021-02-10T05:00:00.706028+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.324564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.324564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"1260","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.805050+0000","last_change":"2021-02-10T04:59:08.651086+0000","last_active":"2021-02-10T05:00:02.805050+0000","last_peered":"2021-02-10T05:00:02.805050+0000","last_clean":"2021-02-10T05:00:02.805050+0000","last_became_active":"2021-02-10T04:58:56.953706+0000","last_became_peered":"2021-02-10T04:58:56.953706+0000","last_unstale":"2021-02-10T05:00:02.805050+0000","last_undegraded":"2021-02-10T05:00:02.805050+0000","last_fullsized":"2021-02-10T05:00:02.805050+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:00.501127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:52:00.501127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"1261'28","reported_seq":"1318","reported_epoch":"1374","state":"active+remapped+backfill_toofull","last_fresh":"2021-02-10T05:00:02.705185+0000","last_change":"2021-02-10T04:59:46.969555+0000","last_active":"2021-02-10T05:00:02.705185+0000","last_peered":"2021-02-10T05:00:02.705185+0000","last_clean":"2021-02-10T04:55:57.106046+0000","last_became_active":"2021-02-10T04:59:16.966181+0000","last_became_peered":"2021-02-10T04:59:16.966181+0000","last_unstale":"2021-02-10T05:00:02.705185+0000","last_undegraded":"2021-02-10T05:00:02.705185+0000","last_fullsized":"2021-02-10T05:00:02.705185+0000","mapping_epoch":1344,"log_start":"1061'20","ondisk_log_start":"1061'20","created":1341,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":5,"last_scrub":"1049'32","last_scrub_stamp":"2021-02-10T04:51:51.382040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:51:51.382040+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":579,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":29,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1026,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,1,0],"avail_no_missing":["4","0","1"],"object_location_counts":[{"shards":"0,1,4","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"259'74","reported_seq":"314","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.805133+0000","last_change":"2021-02-10T04:59:51.110714+0000","last_active":"2021-02-10T05:00:02.805133+0000","last_peered":"2021-02-10T05:00:02.805133+0000","last_clean":"2021-02-10T05:00:02.805133+0000","last_became_active":"2021-02-10T04:59:51.110275+0000","last_became_peered":"2021-02-10T04:59:51.110275+0000","last_unstale":"2021-02-10T05:00:02.805133+0000","last_undegraded":"2021-02-10T05:00:02.805133+0000","last_fullsized":"2021-02-10T05:00:02.805133+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":1007,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":5,"last_scrub":"259'74","last_scrub_stamp":"2021-02-10T04:52:26.597668+0000","last_deep_scrub":"259'74","last_deep_scrub_stamp":"2021-02-10T04:46:49.745104+0000","last_clean_scrub_stamp":"2021-02-10T04:52:26.597668+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1218524,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":28228,"num_write":19,"num_write_kb":7022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":"187","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705374+0000","last_change":"2021-02-10T04:59:57.159970+0000","last_active":"2021-02-10T05:00:02.705374+0000","last_peered":"2021-02-10T05:00:02.705374+0000","last_clean":"2021-02-10T05:00:02.705374+0000","last_became_active":"2021-02-10T04:59:51.139457+0000","last_became_peered":"2021-02-10T04:59:51.139457+0000","last_unstale":"2021-02-10T05:00:02.705374+0000","last_undegraded":"2021-02-10T05:00:02.705374+0000","last_fullsized":"2021-02-10T05:00:02.705374+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:59:57.159904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:50.297650+0000","last_clean_scrub_stamp":"2021-02-10T04:59:57.159904+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"284","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.805070+0000","last_change":"2021-02-10T04:53:46.718512+0000","last_active":"2021-02-10T05:00:02.805070+0000","last_peered":"2021-02-10T05:00:02.805070+0000","last_clean":"2021-02-10T05:00:02.805070+0000","last_became_active":"2021-02-10T04:53:46.716811+0000","last_became_peered":"2021-02-10T04:53:46.716811+0000","last_unstale":"2021-02-10T05:00:02.805070+0000","last_undegraded":"2021-02-10T05:00:02.805070+0000","last_fullsized":"2021-02-10T05:00:02.805070+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:45.813676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:51:45.813676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"1363","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.703337+0000","last_change":"2021-02-10T04:53:38.363899+0000","last_active":"2021-02-10T05:00:00.703337+0000","last_peered":"2021-02-10T05:00:00.703337+0000","last_clean":"2021-02-10T05:00:00.703337+0000","last_became_active":"2021-02-10T04:50:59.539256+0000","last_became_peered":"2021-02-10T04:50:59.539256+0000","last_unstale":"2021-02-10T05:00:00.703337+0000","last_undegraded":"2021-02-10T05:00:00.703337+0000","last_fullsized":"2021-02-10T05:00:00.703337+0000","mapping_epoch":1111,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:38.363825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:51:09.433793+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.363825+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"1261'84","reported_seq":"1584","reported_epoch":"1374","state":"active+remapped+backfill_toofull","last_fresh":"2021-02-10T05:00:02.704410+0000","last_change":"2021-02-10T04:59:52.107625+0000","last_active":"2021-02-10T05:00:02.704410+0000","last_peered":"2021-02-10T05:00:02.704410+0000","last_clean":"2021-02-10T04:59:49.079006+0000","last_became_active":"2021-02-10T04:59:52.099182+0000","last_became_peered":"2021-02-10T04:59:52.099182+0000","last_unstale":"2021-02-10T05:00:02.704410+0000","last_undegraded":"2021-02-10T05:00:02.704410+0000","last_fullsized":"2021-02-10T05:00:02.704410+0000","mapping_epoch":1364,"log_start":"1132'66","ondisk_log_start":"1132'66","created":1341,"last_epoch_clean":1362,"parent":"0.0","parent_split_bits":5,"last_scrub":"1132'66","last_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_deep_scrub":"1132'66","last_deep_scrub_stamp":"2021-02-10T04:51:55.430399+0000","last_clean_scrub_stamp":"2021-02-10T04:51:55.430399+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":808,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":93,"num_read_kb":75,"num_write":54,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1557,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,6],"avail_no_missing":["4","1","5","6","7"],"object_location_counts":[{"shards":"1,4,6","objects":12}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"1003'1689","reported_seq":"4713","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.806413+0000","last_change":"2021-02-10T04:59:08.659989+0000","last_active":"2021-02-10T05:00:02.806413+0000","last_peered":"2021-02-10T05:00:02.806413+0000","last_clean":"2021-02-10T05:00:02.806413+0000","last_became_active":"2021-02-10T04:58:56.970049+0000","last_became_peered":"2021-02-10T04:58:56.970049+0000","last_unstale":"2021-02-10T05:00:02.806413+0000","last_undegraded":"2021-02-10T05:00:02.806413+0000","last_fullsized":"2021-02-10T05:00:02.806413+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'1689","last_scrub_stamp":"2021-02-10T04:53:10.075879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.075879+0000","log_size":1689,"ondisk_log_size":1689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27159495,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":"1379","reported_epoch":"1375","state":"active+clean","last_fresh":"2021-02-10T05:00:06.842011+0000","last_change":"2021-02-10T05:00:06.842011+0000","last_active":"2021-02-10T05:00:06.842011+0000","last_peered":"2021-02-10T05:00:06.842011+0000","last_clean":"2021-02-10T05:00:06.842011+0000","last_became_active":"2021-02-10T04:58:56.965975+0000","last_became_peered":"2021-02-10T04:58:56.965975+0000","last_unstale":"2021-02-10T05:00:06.842011+0000","last_undegraded":"2021-02-10T05:00:06.842011+0000","last_fullsized":"2021-02-10T05:00:06.842011+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T05:00:06.841599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:27:13.332224+0000","last_clean_scrub_stamp":"2021-02-10T05:00:06.841599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"1033","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.703135+0000","last_change":"2021-02-10T04:53:04.354508+0000","last_active":"2021-02-10T05:00:00.703135+0000","last_peered":"2021-02-10T05:00:00.703135+0000","last_clean":"2021-02-10T05:00:00.703135+0000","last_became_active":"2021-02-10T04:33:05.169159+0000","last_became_peered":"2021-02-10T04:33:05.169159+0000","last_unstale":"2021-02-10T05:00:00.703135+0000","last_undegraded":"2021-02-10T05:00:00.703135+0000","last_fullsized":"2021-02-10T05:00:00.703135+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:04.354409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:49.999907+0000","last_clean_scrub_stamp":"2021-02-10T04:53:04.354409+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"1428","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.706377+0000","last_change":"2021-02-10T04:52:36.777554+0000","last_active":"2021-02-10T05:00:02.706377+0000","last_peered":"2021-02-10T05:00:02.706377+0000","last_clean":"2021-02-10T05:00:02.706377+0000","last_became_active":"2021-02-10T04:25:53.043900+0000","last_became_peered":"2021-02-10T04:25:53.043900+0000","last_unstale":"2021-02-10T05:00:02.706377+0000","last_undegraded":"2021-02-10T05:00:02.706377+0000","last_fullsized":"2021-02-10T05:00:02.706377+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:36.777485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:52:36.777485+0000","last_clean_scrub_stamp":"2021-02-10T04:52:36.777485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"1261'86","reported_seq":"1490","reported_epoch":"1374","state":"active+remapped+backfill_toofull","last_fresh":"2021-02-10T05:00:02.702190+0000","last_change":"2021-02-10T04:59:46.970284+0000","last_active":"2021-02-10T05:00:02.702190+0000","last_peered":"2021-02-10T05:00:02.702190+0000","last_clean":"2021-02-10T04:55:57.156886+0000","last_became_active":"2021-02-10T04:59:16.967624+0000","last_became_peered":"2021-02-10T04:59:16.967624+0000","last_unstale":"2021-02-10T05:00:02.702190+0000","last_undegraded":"2021-02-10T05:00:02.702190+0000","last_fullsized":"2021-02-10T05:00:02.702190+0000","mapping_epoch":1344,"log_start":"1049'68","ondisk_log_start":"1049'68","created":1341,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":5,"last_scrub":"1048'39","last_scrub_stamp":"2021-02-10T04:53:38.085565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:47.970100+0000","last_clean_scrub_stamp":"2021-02-10T04:53:38.085565+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":791,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":26,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":113,"num_read_kb":93,"num_write":64,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1582,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,1,4],"avail_no_missing":["3","1","4"],"object_location_counts":[{"shards":"1,3,4","objects":13}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"1001'1528","reported_seq":"6002","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.706270+0000","last_change":"2021-02-10T04:59:53.654638+0000","last_active":"2021-02-10T05:00:02.706270+0000","last_peered":"2021-02-10T05:00:02.706270+0000","last_clean":"2021-02-10T05:00:02.706270+0000","last_became_active":"2021-02-10T04:59:53.654185+0000","last_became_peered":"2021-02-10T04:59:53.654185+0000","last_unstale":"2021-02-10T05:00:02.706270+0000","last_undegraded":"2021-02-10T05:00:02.706270+0000","last_fullsized":"2021-02-10T05:00:02.706270+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":5,"last_scrub":"1001'1528","last_scrub_stamp":"2021-02-10T04:51:01.466315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:51:01.466315+0000","log_size":1528,"ondisk_log_size":1528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46944136,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":358,"num_read_kb":351935,"num_write":286,"num_write_kb":111355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":21311893,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":"1255","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.806915+0000","last_change":"2021-02-10T05:00:00.081989+0000","last_active":"2021-02-10T05:00:02.806915+0000","last_peered":"2021-02-10T05:00:02.806915+0000","last_clean":"2021-02-10T05:00:02.806915+0000","last_became_active":"2021-02-10T04:58:56.855338+0000","last_became_peered":"2021-02-10T04:58:56.855338+0000","last_unstale":"2021-02-10T05:00:02.806915+0000","last_undegraded":"2021-02-10T05:00:02.806915+0000","last_fullsized":"2021-02-10T05:00:02.806915+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T05:00:00.081916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T05:00:00.081916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"669","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.702154+0000","last_change":"2021-02-10T04:59:51.127111+0000","last_active":"2021-02-10T05:00:02.702154+0000","last_peered":"2021-02-10T05:00:02.702154+0000","last_clean":"2021-02-10T05:00:02.702154+0000","last_became_active":"2021-02-10T04:59:51.126766+0000","last_became_peered":"2021-02-10T04:59:51.126766+0000","last_unstale":"2021-02-10T05:00:02.702154+0000","last_undegraded":"2021-02-10T05:00:02.702154+0000","last_fullsized":"2021-02-10T05:00:02.702154+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:07.091949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:19.755718+0000","last_clean_scrub_stamp":"2021-02-10T04:53:07.091949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"1346","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.704117+0000","last_change":"2021-02-10T04:59:51.116613+0000","last_active":"2021-02-10T05:00:02.704117+0000","last_peered":"2021-02-10T05:00:02.704117+0000","last_clean":"2021-02-10T05:00:02.704117+0000","last_became_active":"2021-02-10T04:59:51.116241+0000","last_became_peered":"2021-02-10T04:59:51.116241+0000","last_unstale":"2021-02-10T05:00:02.704117+0000","last_undegraded":"2021-02-10T05:00:02.704117+0000","last_fullsized":"2021-02-10T05:00:02.704117+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:59.838790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:51:59.838790+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":"1069","reported_epoch":"1373","state":"active+clean","last_fresh":"2021-02-10T05:00:00.706565+0000","last_change":"2021-02-10T04:52:30.405170+0000","last_active":"2021-02-10T05:00:00.706565+0000","last_peered":"2021-02-10T05:00:00.706565+0000","last_clean":"2021-02-10T05:00:00.706565+0000","last_became_active":"2021-02-10T04:51:36.852058+0000","last_became_peered":"2021-02-10T04:51:36.852058+0000","last_unstale":"2021-02-10T05:00:00.706565+0000","last_undegraded":"2021-02-10T05:00:00.706565+0000","last_fullsized":"2021-02-10T05:00:00.706565+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:30.405106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:47:20.635536+0000","last_clean_scrub_stamp":"2021-02-10T04:52:30.405106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.20","version":"0'0","reported_seq":"317","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.805231+0000","last_change":"2021-02-10T04:59:51.118148+0000","last_active":"2021-02-10T05:00:02.805231+0000","last_peered":"2021-02-10T05:00:02.805231+0000","last_clean":"2021-02-10T05:00:02.805231+0000","last_became_active":"2021-02-10T04:59:51.117759+0000","last_became_peered":"2021-02-10T04:59:51.117759+0000","last_unstale":"2021-02-10T05:00:02.805231+0000","last_undegraded":"2021-02-10T05:00:02.805231+0000","last_fullsized":"2021-02-10T05:00:02.805231+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:56.449085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:41.375276+0000","last_clean_scrub_stamp":"2021-02-10T04:51:56.449085+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":"1055","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705705+0000","last_change":"2021-02-10T04:59:51.134402+0000","last_active":"2021-02-10T05:00:02.705705+0000","last_peered":"2021-02-10T05:00:02.705705+0000","last_clean":"2021-02-10T05:00:02.705705+0000","last_became_active":"2021-02-10T04:59:51.133981+0000","last_became_peered":"2021-02-10T04:59:51.133981+0000","last_unstale":"2021-02-10T05:00:02.705705+0000","last_undegraded":"2021-02-10T05:00:02.705705+0000","last_fullsized":"2021-02-10T05:00:02.705705+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:35.858557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:44:45.683902+0000","last_clean_scrub_stamp":"2021-02-10T04:53:35.858557+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":"301","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.704409+0000","last_change":"2021-02-10T04:58:56.879602+0000","last_active":"2021-02-10T05:00:02.704409+0000","last_peered":"2021-02-10T05:00:02.704409+0000","last_clean":"2021-02-10T05:00:02.704409+0000","last_became_active":"2021-02-10T04:58:56.879153+0000","last_became_peered":"2021-02-10T04:58:56.879153+0000","last_unstale":"2021-02-10T05:00:02.704409+0000","last_undegraded":"2021-02-10T05:00:02.704409+0000","last_fullsized":"2021-02-10T05:00:02.704409+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:55:23.586401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:55:23.586401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":"284","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:59.693800+0000","last_change":"2021-02-10T04:58:56.966218+0000","last_active":"2021-02-10T04:59:59.693800+0000","last_peered":"2021-02-10T04:59:59.693800+0000","last_clean":"2021-02-10T04:59:59.693800+0000","last_became_active":"2021-02-10T04:58:56.965861+0000","last_became_peered":"2021-02-10T04:58:56.965861+0000","last_unstale":"2021-02-10T04:59:59.693800+0000","last_undegraded":"2021-02-10T04:59:59.693800+0000","last_fullsized":"2021-02-10T04:59:59.693800+0000","mapping_epoch":1332,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:47:28.706372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:47:28.706372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":"1130","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.706654+0000","last_change":"2021-02-10T04:59:51.127003+0000","last_active":"2021-02-10T05:00:02.706654+0000","last_peered":"2021-02-10T05:00:02.706654+0000","last_clean":"2021-02-10T05:00:02.706654+0000","last_became_active":"2021-02-10T04:59:51.126491+0000","last_became_peered":"2021-02-10T04:59:51.126491+0000","last_unstale":"2021-02-10T05:00:02.706654+0000","last_undegraded":"2021-02-10T05:00:02.706654+0000","last_fullsized":"2021-02-10T05:00:02.706654+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:57.287363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:57.287363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":"345","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.806999+0000","last_change":"2021-02-10T04:59:51.117057+0000","last_active":"2021-02-10T05:00:02.806999+0000","last_peered":"2021-02-10T05:00:02.806999+0000","last_clean":"2021-02-10T05:00:02.806999+0000","last_became_active":"2021-02-10T04:59:51.116609+0000","last_became_peered":"2021-02-10T04:59:51.116609+0000","last_unstale":"2021-02-10T05:00:02.806999+0000","last_undegraded":"2021-02-10T05:00:02.806999+0000","last_fullsized":"2021-02-10T05:00:02.806999+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:50:10.308282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:50:10.308282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":"1306","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.704198+0000","last_change":"2021-02-10T04:59:51.133004+0000","last_active":"2021-02-10T05:00:02.704198+0000","last_peered":"2021-02-10T05:00:02.704198+0000","last_clean":"2021-02-10T05:00:02.704198+0000","last_became_active":"2021-02-10T04:59:51.132662+0000","last_became_peered":"2021-02-10T04:59:51.132662+0000","last_unstale":"2021-02-10T05:00:02.704198+0000","last_undegraded":"2021-02-10T05:00:02.704198+0000","last_fullsized":"2021-02-10T05:00:02.704198+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:09.121298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:09.121298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.21","version":"293'77","reported_seq":"945","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.702952+0000","last_change":"2021-02-10T04:52:28.062923+0000","last_active":"2021-02-10T05:00:02.702952+0000","last_peered":"2021-02-10T05:00:02.702952+0000","last_clean":"2021-02-10T05:00:02.702952+0000","last_became_active":"2021-02-10T04:52:28.062463+0000","last_became_peered":"2021-02-10T04:52:28.062463+0000","last_unstale":"2021-02-10T05:00:02.702952+0000","last_undegraded":"2021-02-10T05:00:02.702952+0000","last_fullsized":"2021-02-10T05:00:02.702952+0000","mapping_epoch":1143,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1144,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:28:24.128217+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1886586,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":15163,"num_write":9,"num_write_kb":3316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":"271","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.704875+0000","last_change":"2021-02-10T04:52:32.124955+0000","last_active":"2021-02-10T05:00:02.704875+0000","last_peered":"2021-02-10T05:00:02.704875+0000","last_clean":"2021-02-10T05:00:02.704875+0000","last_became_active":"2021-02-10T04:51:36.867061+0000","last_became_peered":"2021-02-10T04:51:36.867061+0000","last_unstale":"2021-02-10T05:00:02.704875+0000","last_undegraded":"2021-02-10T05:00:02.704875+0000","last_fullsized":"2021-02-10T05:00:02.704875+0000","mapping_epoch":1118,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1119,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:32.124901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:08.271977+0000","last_clean_scrub_stamp":"2021-02-10T04:52:32.124901+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":"95","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.806143+0000","last_change":"2021-02-10T04:59:51.115844+0000","last_active":"2021-02-10T05:00:02.806143+0000","last_peered":"2021-02-10T05:00:02.806143+0000","last_clean":"2021-02-10T05:00:02.806143+0000","last_became_active":"2021-02-10T04:59:51.115454+0000","last_became_peered":"2021-02-10T04:59:51.115454+0000","last_unstale":"2021-02-10T05:00:02.806143+0000","last_undegraded":"2021-02-10T05:00:02.806143+0000","last_fullsized":"2021-02-10T05:00:02.806143+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:51:07.182678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:51:07.182678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.20","version":"1000'1071","reported_seq":"4822","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.703630+0000","last_change":"2021-02-10T04:59:55.666986+0000","last_active":"2021-02-10T05:00:02.703630+0000","last_peered":"2021-02-10T05:00:02.703630+0000","last_clean":"2021-02-10T05:00:02.703630+0000","last_became_active":"2021-02-10T04:59:55.666246+0000","last_became_peered":"2021-02-10T04:59:55.666246+0000","last_unstale":"2021-02-10T05:00:02.703630+0000","last_undegraded":"2021-02-10T05:00:02.703630+0000","last_fullsized":"2021-02-10T05:00:02.703630+0000","mapping_epoch":1368,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1369,"parent":"0.0","parent_split_bits":6,"last_scrub":"1000'1071","last_scrub_stamp":"2021-02-10T04:53:03.915403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.915403+0000","log_size":1071,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16398893,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":11822743,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":"897","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.804989+0000","last_change":"2021-02-10T04:53:46.721599+0000","last_active":"2021-02-10T05:00:02.804989+0000","last_peered":"2021-02-10T05:00:02.804989+0000","last_clean":"2021-02-10T05:00:02.804989+0000","last_became_active":"2021-02-10T04:53:46.721340+0000","last_became_peered":"2021-02-10T04:53:46.721340+0000","last_unstale":"2021-02-10T05:00:02.804989+0000","last_undegraded":"2021-02-10T05:00:02.804989+0000","last_fullsized":"2021-02-10T05:00:02.804989+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:28.092288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:50:07.767250+0000","last_clean_scrub_stamp":"2021-02-10T04:52:28.092288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.25","version":"53'2","reported_seq":"1342","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705518+0000","last_change":"2021-02-10T04:59:51.233212+0000","last_active":"2021-02-10T05:00:02.705518+0000","last_peered":"2021-02-10T05:00:02.705518+0000","last_clean":"2021-02-10T05:00:02.705518+0000","last_became_active":"2021-02-10T04:59:51.122097+0000","last_became_peered":"2021-02-10T04:59:51.122097+0000","last_unstale":"2021-02-10T05:00:02.705518+0000","last_undegraded":"2021-02-10T05:00:02.705518+0000","last_fullsized":"2021-02-10T05:00:02.705518+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'2","last_scrub_stamp":"2021-02-10T04:53:36.102928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:36.102928+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.23","version":"1002'1657","reported_seq":"5101","reported_epoch":"1372","state":"active+clean","last_fresh":"2021-02-10T04:59:59.694094+0000","last_change":"2021-02-10T04:59:53.661728+0000","last_active":"2021-02-10T04:59:59.694094+0000","last_peered":"2021-02-10T04:59:59.694094+0000","last_clean":"2021-02-10T04:59:59.694094+0000","last_became_active":"2021-02-10T04:59:53.658988+0000","last_became_peered":"2021-02-10T04:59:53.658988+0000","last_unstale":"2021-02-10T04:59:59.694094+0000","last_undegraded":"2021-02-10T04:59:59.694094+0000","last_fullsized":"2021-02-10T04:59:59.694094+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":6,"last_scrub":"1002'1657","last_scrub_stamp":"2021-02-10T04:47:22.855742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:24.128217+0000","last_clean_scrub_stamp":"2021-02-10T04:47:22.855742+0000","log_size":1657,"ondisk_log_size":1657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21396533,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":212,"num_read_kb":218837,"num_write":159,"num_write_kb":59399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":20479176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":"1166","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.702474+0000","last_change":"2021-02-10T04:59:51.127951+0000","last_active":"2021-02-10T05:00:02.702474+0000","last_peered":"2021-02-10T05:00:02.702474+0000","last_clean":"2021-02-10T05:00:02.702474+0000","last_became_active":"2021-02-10T04:59:51.127586+0000","last_became_peered":"2021-02-10T04:59:51.127586+0000","last_unstale":"2021-02-10T05:00:02.702474+0000","last_undegraded":"2021-02-10T05:00:02.702474+0000","last_fullsized":"2021-02-10T05:00:02.702474+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:12.951528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:12.951528+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.26","version":"53'1","reported_seq":"220","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.806365+0000","last_change":"2021-02-10T04:59:51.108138+0000","last_active":"2021-02-10T05:00:02.806365+0000","last_peered":"2021-02-10T05:00:02.806365+0000","last_clean":"2021-02-10T05:00:02.806365+0000","last_became_active":"2021-02-10T04:59:51.107693+0000","last_became_peered":"2021-02-10T04:59:51.107693+0000","last_unstale":"2021-02-10T05:00:02.806365+0000","last_undegraded":"2021-02-10T05:00:02.806365+0000","last_fullsized":"2021-02-10T05:00:02.806365+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'1","last_scrub_stamp":"2021-02-10T04:53:03.933910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:53:03.933910+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":423,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.22","version":"998'1009","reported_seq":"4741","reported_epoch":"1375","state":"active+clean","last_fresh":"2021-02-10T05:00:06.153707+0000","last_change":"2021-02-10T05:00:06.153707+0000","last_active":"2021-02-10T05:00:06.153707+0000","last_peered":"2021-02-10T05:00:06.153707+0000","last_clean":"2021-02-10T05:00:06.153707+0000","last_became_active":"2021-02-10T04:52:31.085288+0000","last_became_peered":"2021-02-10T04:52:31.085288+0000","last_unstale":"2021-02-10T05:00:06.153707+0000","last_undegraded":"2021-02-10T05:00:06.153707+0000","last_fullsized":"2021-02-10T05:00:06.153707+0000","mapping_epoch":1146,"log_start":"0'0","ondisk_log_start":"0'0","created":1141,"last_epoch_clean":1147,"parent":"0.0","parent_split_bits":6,"last_scrub":"998'1009","last_scrub_stamp":"2021-02-10T05:00:06.153628+0000","last_deep_scrub":"998'1009","last_deep_scrub_stamp":"2021-02-10T05:00:06.153628+0000","last_clean_scrub_stamp":"2021-02-10T05:00:06.153628+0000","log_size":1009,"ondisk_log_size":1009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":"978","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.705446+0000","last_change":"2021-02-10T04:59:51.125119+0000","last_active":"2021-02-10T05:00:02.705446+0000","last_peered":"2021-02-10T05:00:02.705446+0000","last_clean":"2021-02-10T05:00:02.705446+0000","last_became_active":"2021-02-10T04:59:51.124340+0000","last_became_peered":"2021-02-10T04:59:51.124340+0000","last_unstale":"2021-02-10T05:00:02.705446+0000","last_undegraded":"2021-02-10T05:00:02.705446+0000","last_fullsized":"2021-02-10T05:00:02.705446+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":1044,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:10.958579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:52.015519+0000","last_clean_scrub_stamp":"2021-02-10T04:53:10.958579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":"316","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.804955+0000","last_change":"2021-02-10T04:53:46.725941+0000","last_active":"2021-02-10T05:00:02.804955+0000","last_peered":"2021-02-10T05:00:02.804955+0000","last_clean":"2021-02-10T05:00:02.804955+0000","last_became_active":"2021-02-10T04:53:46.725587+0000","last_became_peered":"2021-02-10T04:53:46.725587+0000","last_unstale":"2021-02-10T05:00:02.804955+0000","last_undegraded":"2021-02-10T05:00:02.804955+0000","last_fullsized":"2021-02-10T05:00:02.804955+0000","mapping_epoch":1190,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1191,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:41.758684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:25:33.303710+0000","last_clean_scrub_stamp":"2021-02-10T04:52:41.758684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":"1311","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.702305+0000","last_change":"2021-02-10T04:55:39.599193+0000","last_active":"2021-02-10T05:00:02.702305+0000","last_peered":"2021-02-10T05:00:02.702305+0000","last_clean":"2021-02-10T05:00:02.702305+0000","last_became_active":"2021-02-10T04:55:39.598816+0000","last_became_peered":"2021-02-10T04:55:39.598816+0000","last_unstale":"2021-02-10T05:00:02.702305+0000","last_undegraded":"2021-02-10T05:00:02.702305+0000","last_fullsized":"2021-02-10T05:00:02.702305+0000","mapping_epoch":1248,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1249,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:53:11.139602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:13.486341+0000","last_clean_scrub_stamp":"2021-02-10T04:53:11.139602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":"1074","reported_epoch":"1374","state":"active+clean","last_fresh":"2021-02-10T05:00:02.702686+0000","last_change":"2021-02-10T04:59:51.121337+0000","last_active":"2021-02-10T05:00:02.702686+0000","last_peered":"2021-02-10T05:00:02.702686+0000","last_clean":"2021-02-10T05:00:02.702686+0000","last_became_active":"2021-02-10T04:59:51.118780+0000","last_became_peered":"2021-02-10T04:59:51.118780+0000","last_unstale":"2021-02-10T05:00:02.702686+0000","last_undegraded":"2021-02-10T05:00:02.702686+0000","last_fullsized":"2021-02-10T05:00:02.702686+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10T04:52:24.071915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10T04:28:45.220258+0000","last_clean_scrub_stamp":"2021-02-10T04:52:24.071915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":885390445,"num_objects":360,"num_object_clones":310,"num_object_copies":1080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":360,"num_whiteouts":9,"num_read":5021,"num_read_kb":5097788,"num_write":3638,"num_write_kb":1397125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1161,"num_bytes_recovered":571024498,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2167754752,"data_stored":2724507644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1639,"internal_metadata":0},"log_size":34728,"ondisk_log_size":34728,"up":108,"acting":108,"num_store_stats":8},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":1945,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":18,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":2031,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":6889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31,"ondisk_log_size":31,"up":126,"acting":126,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":13,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":630,"num_read_kb":625,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":973,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20,"ondisk_log_size":20,"up":39,"acting":39,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":120,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":2706,"num_read_kb":2179,"num_write":1592,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":419,"num_bytes_recovered":29054,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1042,"ondisk_log_size":1042,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":126,"acting":126,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748365216,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":250652,"kb_used_data":197780,"kb_used_omap":0,"kb_used_meta":52864,"kb_avail":93494500,"statfs":{"total":95995035648,"available":95738368000,"internally_reserved":0,"allocated":202526720,"data_stored":257953235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54132736},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:59:55 2021","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.494,"5min":1.494,"15min":1.494},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.995,"5min":0.995,"15min":0.995},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":2.02,"5min":2.02,"15min":2.02},"last":0.69799999999999995}]},{"osd":1,"last update":"Wed Feb 10 04:59:48 2021","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":1.054,"15min":0.94599999999999995},"min":{"1min":0.53400000000000003,"5min":0.443,"15min":0.443},"max":{"1min":1.78,"5min":15.253,"15min":15.253},"last":0.872},{"interface":"front","average":{"1min":0.89700000000000002,"5min":1.1279999999999999,"15min":1.016},"min":{"1min":0.59599999999999997,"5min":0.47499999999999998,"15min":0.45500000000000002},"max":{"1min":1.74,"5min":18.713000000000001,"15min":18.713000000000001},"last":0.77800000000000002}]},{"osd":2,"last update":"Wed Feb 10 04:59:48 2021","interfaces":[{"interface":"back","average":{"1min":3.2949999999999999,"5min":3.1579999999999999,"15min":1.853},"min":{"1min":0.67600000000000005,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":49.811999999999998,"5min":94.391999999999996,"15min":94.391999999999996},"last":1.05},{"interface":"front","average":{"1min":3.286,"5min":3.113,"15min":1.8500000000000001},"min":{"1min":0.55900000000000005,"5min":0.54300000000000004,"15min":0.496},"max":{"1min":49.668999999999997,"5min":94.269999999999996,"15min":94.269999999999996},"last":1.103}]},{"osd":3,"last update":"Wed Feb 10 04:59:03 2021","interfaces":[{"interface":"back","average":{"1min":1.657,"5min":1.2050000000000001,"15min":1.264},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.58199999999999996},"max":{"1min":15.148,"5min":15.148,"15min":15.148},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.891,"5min":1.363,"15min":1.3839999999999999},"min":{"1min":0.71899999999999997,"5min":0.629,"15min":0.629},"max":{"1min":18.611000000000001,"5min":18.611000000000001,"15min":18.611000000000001},"last":1.1639999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:59:48 2021","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.883,"15min":1.528},"min":{"1min":0.68600000000000005,"5min":0.60399999999999998,"15min":0.59899999999999998},"max":{"1min":2.206,"5min":72.662999999999997,"15min":72.662999999999997},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.2709999999999999,"5min":2.1659999999999999,"15min":1.595},"min":{"1min":0.81499999999999995,"5min":0.73999999999999999,"15min":0.69099999999999995},"max":{"1min":2.3900000000000001,"5min":73.564999999999998,"15min":73.564999999999998},"last":1.077}]},{"osd":5,"last update":"Wed Feb 10 04:59:48 2021","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.5449999999999999,"15min":1.5109999999999999},"min":{"1min":0.88800000000000001,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.2909999999999999,"5min":11.627000000000001,"15min":14.972},"last":1.5449999999999999},{"interface":"front","average":{"1min":1.544,"5min":1.427,"15min":1.581},"min":{"1min":0.79200000000000004,"5min":0.76600000000000001,"15min":0.72699999999999998},"max":{"1min":2.5880000000000001,"5min":13.172000000000001,"15min":15.218999999999999},"last":1.409}]},{"osd":6,"last update":"Wed Feb 10 04:59:48 2021","interfaces":[{"interface":"back","average":{"1min":1.5309999999999999,"5min":1.4370000000000001,"15min":1.5960000000000001},"min":{"1min":0.85999999999999999,"5min":0.78200000000000003,"15min":0.72699999999999998},"max":{"1min":2.0139999999999998,"5min":10.286,"15min":16.050000000000001},"last":1.321},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.6000000000000001,"15min":1.635},"min":{"1min":0.96699999999999997,"5min":0.88100000000000001,"15min":0.876},"max":{"1min":2.5049999999999999,"5min":17.460000000000001,"15min":17.460000000000001},"last":1.1910000000000001}]}]},{"osd":6,"up_from":0,"seq":184683594148,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834575285,"num_pgs":102,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":347108,"kb_used_data":280668,"kb_used_omap":0,"kb_used_meta":66432,"kb_avail":93398044,"statfs":{"total":95995035648,"available":95639597056,"internally_reserved":0,"allocated":287404032,"data_stored":354806114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":68026368},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 05:00:01 2021","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.238,"5min":1.238,"15min":1.238},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.64400000000000002}]},{"osd":2,"last update":"Wed Feb 10 04:59:24 2021","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":5.3099999999999996,"15min":3.1949999999999998},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.42399999999999999},"max":{"1min":20.527000000000001,"5min":162.61199999999999,"15min":162.61199999999999},"last":1.028},{"interface":"front","average":{"1min":1.7130000000000001,"5min":5.4459999999999997,"15min":3.286},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":20.483000000000001,"5min":162.762,"15min":162.762},"last":0.71699999999999997}]},{"osd":3,"last update":"Wed Feb 10 04:59:08 2021","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.83399999999999996,"15min":0.88500000000000001},"min":{"1min":0.495,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.0629999999999999,"5min":2.8919999999999999,"15min":3.077},"last":1.115},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.84699999999999998,"15min":0.95599999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42199999999999999},"max":{"1min":1.4159999999999999,"5min":3.1890000000000001,"15min":3.1890000000000001},"last":1.3919999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:59:05 2021","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":1.919,"15min":1.393},"min":{"1min":0.58899999999999997,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.837,"5min":100.071,"15min":100.071},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.9370000000000001,"15min":1.4239999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.53500000000000003},"max":{"1min":1.6990000000000001,"5min":100.43899999999999,"15min":100.43899999999999},"last":1.784}]},{"osd":5,"last update":"Wed Feb 10 04:59:08 2021","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.1579999999999999,"15min":1.1599999999999999},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.603,"5min":3.0219999999999998,"15min":5.0300000000000002},"last":1.421},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.202,"15min":1.3120000000000001},"min":{"1min":0.79900000000000004,"5min":0.70399999999999996,"15min":0.68600000000000005},"max":{"1min":1.9319999999999999,"5min":6.032,"15min":6.032},"last":0.94099999999999995}]},{"osd":6,"last update":"Wed Feb 10 04:59:24 2021","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.159,"15min":1.333},"min":{"1min":0.83599999999999997,"5min":0.68999999999999995,"15min":0.59599999999999997},"max":{"1min":1.599,"5min":3.302,"15min":30.997},"last":1.6699999999999999},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.2609999999999999,"15min":1.3460000000000001},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.60499999999999998},"max":{"1min":1.893,"5min":3.1549999999999998,"15min":31.138000000000002},"last":1.49}]},{"osd":7,"last update":"Wed Feb 10 04:59:30 2021","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.2709999999999999,"15min":1.331},"min":{"1min":0.89500000000000002,"5min":0.72299999999999998,"15min":0.628},"max":{"1min":2.052,"5min":6.2629999999999999,"15min":6.2629999999999999},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.244,"15min":1.3300000000000001},"min":{"1min":0.92000000000000004,"5min":0.83299999999999996,"15min":0.78700000000000003},"max":{"1min":1.647,"5min":3.4129999999999998,"15min":3.754},"last":1.5529999999999999}]}]},{"osd":0,"up_from":1332,"seq":5720896438287,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":325208,"kb_used_data":310096,"kb_used_omap":324,"kb_used_meta":14779,"kb_avail":93419944,"statfs":{"total":95995035648,"available":95662022656,"internally_reserved":0,"allocated":317538304,"data_stored":410948377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":332393,"internal_metadata":15134103},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 04:59:58 2021","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.55600000000000005}]},{"osd":2,"last update":"Wed Feb 10 04:59:58 2021","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.76,"5min":1.76,"15min":1.76},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.339,"5min":1.339,"15min":1.339},"last":0.80200000000000005}]},{"osd":3,"last update":"Wed Feb 10 04:59:58 2021","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.59699999999999998}]},{"osd":4,"last update":"Wed Feb 10 04:59:58 2021","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.751},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":2.6389999999999998,"5min":2.6389999999999998,"15min":2.6389999999999998},"last":0.95899999999999996}]},{"osd":5,"last update":"Wed Feb 10 04:59:58 2021","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.323,"15min":1.323},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":2.5139999999999998,"5min":2.5139999999999998,"15min":2.5139999999999998},"last":1.073},{"interface":"front","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":2.2709999999999999,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":1.2330000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:59:58 2021","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.282,"15min":1.282},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":3.0329999999999999,"5min":3.0329999999999999,"15min":3.0329999999999999},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"min":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"max":{"1min":2.1840000000000002,"5min":2.1840000000000002,"15min":2.1840000000000002},"last":0.91500000000000004}]},{"osd":7,"last update":"Wed Feb 10 04:59:58 2021","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"min":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"max":{"1min":2.9729999999999999,"5min":2.9729999999999999,"15min":2.9729999999999999},"last":1.258},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"min":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"max":{"1min":2.2970000000000002,"5min":2.2970000000000002,"15min":2.2970000000000002},"last":1.1739999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411764,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":477288,"kb_used_data":430240,"kb_used_omap":0,"kb_used_meta":47040,"kb_avail":93267864,"statfs":{"total":95995035648,"available":95506292736,"internally_reserved":0,"allocated":440565760,"data_stored":508528057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48168960},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:59:57 2021","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.496},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.52800000000000002}]},{"osd":1,"last update":"Wed Feb 10 05:00:05 2021","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.89400000000000002,"15min":0.92000000000000004},"min":{"1min":0.60699999999999998,"5min":0.41999999999999998,"15min":0.41599999999999998},"max":{"1min":2.5510000000000002,"5min":17.041,"15min":17.041},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.91100000000000003,"5min":1.0900000000000001,"15min":0.90000000000000002},"min":{"1min":0.55300000000000005,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":2.4660000000000002,"5min":21.172000000000001,"15min":21.172000000000001},"last":0.78800000000000003}]},{"osd":3,"last update":"Wed Feb 10 04:59:22 2021","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":1.022,"15min":1.069},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.3460000000000001,"5min":18.904,"15min":18.904},"last":0.92500000000000004},{"interface":"front","average":{"1min":1.002,"5min":1.262,"15min":1.2689999999999999},"min":{"1min":0.71299999999999997,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.6000000000000001,"5min":19.241,"15min":19.241},"last":1.0309999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:59:39 2021","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.2410000000000001,"15min":1.228},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.677,"5min":9.2029999999999994,"15min":9.2029999999999994},"last":1.357},{"interface":"front","average":{"1min":1.01,"5min":1.115,"15min":1.161},"min":{"1min":0.66500000000000004,"5min":0.503,"15min":0.503},"max":{"1min":1.518,"5min":3.403,"15min":24.373999999999999},"last":1.1759999999999999}]},{"osd":5,"last update":"Wed Feb 10 05:00:05 2021","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.7210000000000001,"15min":1.7},"min":{"1min":0.83599999999999997,"5min":0.68300000000000005,"15min":0.63500000000000001},"max":{"1min":2.024,"5min":39.911999999999999,"15min":39.911999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.97799999999999998,"5min":1.3919999999999999,"15min":1.411},"min":{"1min":0.69899999999999995,"5min":0.64300000000000002,"15min":0.57599999999999996},"max":{"1min":1.377,"5min":33.539999999999999,"15min":33.539999999999999},"last":0.81399999999999995}]},{"osd":6,"last update":"Wed Feb 10 04:59:36 2021","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.071,"15min":1.3009999999999999},"min":{"1min":0.70799999999999996,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.677,"5min":3.1659999999999999,"15min":20.007000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.4770000000000001,"15min":1.556},"min":{"1min":0.65800000000000003,"5min":0.622,"15min":0.622},"max":{"1min":1.768,"5min":16.135000000000002,"15min":19.417999999999999},"last":1.266}]},{"osd":7,"last update":"Wed Feb 10 04:59:39 2021","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.4370000000000001,"15min":1.5840000000000001},"min":{"1min":0.86699999999999999,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.8260000000000001,"5min":9.3460000000000001,"15min":24.405000000000001},"last":1.153},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.234,"15min":1.232},"min":{"1min":0.871,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.478,"5min":3.6640000000000001,"15min":4.9189999999999996},"last":1.0629999999999999}]}]},{"osd":3,"up_from":1111,"seq":4771708665967,"num_pgs":102,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":368444,"kb_used_data":350708,"kb_used_omap":386,"kb_used_meta":17341,"kb_avail":93376708,"statfs":{"total":95995035648,"available":95617748992,"internally_reserved":0,"allocated":359124992,"data_stored":471860952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":396036,"internal_metadata":17757436},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:59:56 2021","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.556,"5min":1.556,"15min":1.556},"last":0.96099999999999997}]},{"osd":1,"last update":"Wed Feb 10 04:59:12 2021","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.82699999999999996,"15min":0.71099999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44},"max":{"1min":6.1980000000000004,"5min":6.1980000000000004,"15min":6.1980000000000004},"last":1.224},{"interface":"front","average":{"1min":1.1859999999999999,"5min":0.89200000000000002,"15min":0.873},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.45600000000000002},"max":{"1min":7.7709999999999999,"5min":7.7709999999999999,"15min":7.7709999999999999},"last":1.079}]},{"osd":2,"last update":"Wed Feb 10 04:59:12 2021","interfaces":[{"interface":"back","average":{"1min":4.5419999999999998,"5min":2.1219999999999999,"15min":1.4930000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":81.138000000000005,"5min":81.138000000000005,"15min":81.138000000000005},"last":0.99099999999999999},{"interface":"front","average":{"1min":4.5590000000000002,"5min":2.3220000000000001,"15min":1.5700000000000001},"min":{"1min":0.54400000000000004,"5min":0.53800000000000003,"15min":0.50800000000000001},"max":{"1min":81.099999999999994,"5min":81.099999999999994,"15min":81.099999999999994},"last":1.2949999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:59:12 2021","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":1.0009999999999999,"15min":0.97899999999999998},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.5,"5min":2.6589999999999998,"15min":2.9430000000000001},"last":1.6890000000000001},{"interface":"front","average":{"1min":1.327,"5min":1.143,"15min":1.1040000000000001},"min":{"1min":0.63900000000000001,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":7.484,"5min":7.484,"15min":7.484},"last":1.367}]},{"osd":5,"last update":"Wed Feb 10 04:59:12 2021","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.2649999999999999,"15min":2.2170000000000001},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":7.6379999999999999,"5min":7.6379999999999999,"15min":52.353000000000002},"last":1.548},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.0249999999999999,"15min":2.02},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":4.8730000000000002,"5min":4.8730000000000002,"15min":52.301000000000002},"last":1.5940000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:59:12 2021","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.1180000000000001,"15min":3.7570000000000001},"min":{"1min":0.68500000000000005,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":5.258,"5min":5.258,"15min":64.376999999999995},"last":1.712},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.2909999999999999,"15min":3.8130000000000002},"min":{"1min":0.88400000000000001,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":5.1399999999999997,"5min":5.1399999999999997,"15min":64.438999999999993},"last":1.782}]},{"osd":7,"last update":"Wed Feb 10 04:59:12 2021","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.1319999999999999,"15min":1.115},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.754},"max":{"1min":5.3819999999999997,"5min":5.3819999999999997,"15min":5.3819999999999997},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.145,"5min":1.1899999999999999,"15min":1.157},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.79900000000000004},"max":{"1min":1.6859999999999999,"5min":2.9569999999999999,"15min":2.9569999999999999},"last":1.5649999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986604,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":170796,"kb_used_data":120612,"kb_used_omap":0,"kb_used_meta":50176,"kb_avail":93574356,"statfs":{"total":95995035648,"available":95820140544,"internally_reserved":0,"allocated":123506688,"data_stored":155353329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":51380224},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:59:57 2021","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":2.1619999999999999,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":2.4609999999999999,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":1.1439999999999999}]},{"osd":1,"last update":"Wed Feb 10 04:59:51 2021","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.80600000000000005,"15min":0.85099999999999998},"min":{"1min":0.59899999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":2.9630000000000001,"5min":2.9630000000000001,"15min":7.883},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.226,"5min":0.98599999999999999,"15min":1.024},"min":{"1min":0.65800000000000003,"5min":0.47999999999999998,"15min":0.45900000000000002},"max":{"1min":3.5619999999999998,"5min":3.5619999999999998,"15min":4.3449999999999998},"last":1.3779999999999999}]},{"osd":2,"last update":"Wed Feb 10 04:59:51 2021","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":1.1930000000000001,"15min":1.6899999999999999},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.51000000000000001},"max":{"1min":4.1070000000000002,"5min":7.0069999999999997,"15min":120.39},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.228,"5min":1.0840000000000001,"15min":1.659},"min":{"1min":0.70399999999999996,"5min":0.504,"15min":0.504},"max":{"1min":4.1989999999999998,"5min":7.1120000000000001,"15min":120.583},"last":1.085}]},{"osd":3,"last update":"Wed Feb 10 04:59:06 2021","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.1299999999999999,"15min":1.216},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":2.3100000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.264},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.2030000000000001,"15min":1.494},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.57399999999999995},"max":{"1min":2.548,"5min":2.548,"15min":26.802},"last":1.4870000000000001}]},{"osd":5,"last update":"Wed Feb 10 04:59:06 2021","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.04,"15min":1.401},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.495},"max":{"1min":2.5499999999999998,"5min":2.5499999999999998,"15min":35.774000000000001},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.024,"5min":1.1379999999999999,"15min":1.419},"min":{"1min":0.621,"5min":0.621,"15min":0.60899999999999999},"max":{"1min":2.3170000000000002,"5min":3.8410000000000002,"15min":35.606000000000002},"last":1.2290000000000001}]},{"osd":6,"last update":"Wed Feb 10 04:59:06 2021","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.4830000000000001,"15min":2.3570000000000002},"min":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"max":{"1min":2.6819999999999999,"5min":3.8530000000000002,"15min":201.626},"last":1.5940000000000001},{"interface":"front","average":{"1min":1.2,"5min":1.3200000000000001,"15min":2.319},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.68100000000000005},"max":{"1min":2.4100000000000001,"5min":4.7910000000000004,"15min":201.577},"last":1.2889999999999999}]},{"osd":7,"last update":"Wed Feb 10 04:59:39 2021","interfaces":[{"interface":"back","average":{"1min":1.784,"5min":1.5309999999999999,"15min":1.496},"min":{"1min":1.1599999999999999,"5min":0.90300000000000002,"15min":0.67100000000000004},"max":{"1min":3.3420000000000001,"5min":4.6200000000000001,"15min":4.7439999999999998},"last":1.7050000000000001},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.3169999999999999,"15min":1.4219999999999999},"min":{"1min":0.81100000000000005,"5min":0.79100000000000004,"15min":0.63200000000000001},"max":{"1min":5.9299999999999997,"5min":5.9299999999999997,"15min":5.9299999999999997},"last":1.351}]}]},{"osd":5,"up_from":37,"seq":158913790376,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":342336,"kb_used_data":289464,"kb_used_omap":0,"kb_used_meta":52864,"kb_avail":93402816,"statfs":{"total":95995035648,"available":95644483584,"internally_reserved":0,"allocated":296411136,"data_stored":378633957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54132736},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 04:59:56 2021","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.597,"5min":2.597,"15min":2.597},"last":1.24}]},{"osd":1,"last update":"Wed Feb 10 04:59:06 2021","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.85299999999999998,"15min":0.90400000000000003},"min":{"1min":0.438,"5min":0.435,"15min":0.435},"max":{"1min":1.9790000000000001,"5min":4.242,"15min":6.391},"last":1.611},{"interface":"front","average":{"1min":0.82299999999999995,"5min":1.0800000000000001,"15min":1.1339999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.41299999999999998},"max":{"1min":1.3480000000000001,"5min":13.794,"15min":13.794},"last":1.117}]},{"osd":2,"last update":"Wed Feb 10 04:59:06 2021","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":3.4569999999999999,"15min":2.1440000000000001},"min":{"1min":0.504,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.748,"5min":116.438,"15min":116.438},"last":3.5649999999999999},{"interface":"front","average":{"1min":0.92500000000000004,"5min":3.5009999999999999,"15min":2.206},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.8759999999999999,"5min":116.96899999999999,"15min":116.96899999999999},"last":3.0649999999999999}]},{"osd":3,"last update":"Wed Feb 10 04:59:15 2021","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.367,"15min":1.302},"min":{"1min":0.57499999999999996,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.4750000000000001,"5min":13.755000000000001,"15min":13.755000000000001},"last":1.274},{"interface":"front","average":{"1min":1.073,"5min":1.1639999999999999,"15min":1.1839999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.64700000000000002},"max":{"1min":2.331,"5min":5.577,"15min":6.5129999999999999},"last":1.6899999999999999}]},{"osd":4,"last update":"Wed Feb 10 04:59:06 2021","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":1.054,"15min":1.246},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.841,"5min":5.298,"15min":9.4890000000000008},"last":1.522},{"interface":"front","average":{"1min":1.016,"5min":1.1899999999999999,"15min":1.319},"min":{"1min":0.61499999999999999,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.327,"5min":4.8289999999999997,"15min":5.6180000000000003},"last":1.4239999999999999}]},{"osd":6,"last update":"Wed Feb 10 04:59:15 2021","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.3440000000000001,"15min":1.7370000000000001},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.7450000000000001,"5min":4.9429999999999996,"15min":38.014000000000003},"last":1.573},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.302,"15min":1.6859999999999999},"min":{"1min":0.80600000000000005,"5min":0.71499999999999997,"15min":0.68799999999999994},"max":{"1min":2.2080000000000002,"5min":5.4660000000000002,"15min":37.896999999999998},"last":1.8080000000000001}]},{"osd":7,"last update":"Wed Feb 10 04:59:44 2021","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":1.3049999999999999,"15min":1.4399999999999999},"min":{"1min":1.048,"5min":0.754,"15min":0.55500000000000005},"max":{"1min":2.8919999999999999,"5min":5.4160000000000004,"15min":11.897},"last":1.667},{"interface":"front","average":{"1min":1.472,"5min":1.4610000000000001,"15min":1.5229999999999999},"min":{"1min":0.89700000000000002,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":2.1080000000000001,"5min":5.0579999999999998,"15min":6.0369999999999999},"last":1.8400000000000001}]}]}]}} 2021-02-10T05:00:09.960 INFO:tasks.thrashosds.thrasher:Setting 2.5 to [1, 4, 6, 3, 7, 5] 2021-02-10T05:00:09.960 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.5', '1', '4', '6', '3', '7', '5'] 2021-02-10T05:00:09.960 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pg-upmap-items 2.5 1 4 6 3 7 5 2021-02-10T05:00:10.205 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:00:11.140 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:10 smithi085 conmon[29319]: audit 2021-02-10T05:00:09.594968+0000 mgr.y (mgr.14140) 1553 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:00:11.141 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:10 smithi085 conmon[35961]: audit 2021-02-10T05:00:09.594968+0000 mgr.y (mgr.14140) 1553 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:00:11.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:10 smithi164 conmon[29288]: audit 2021-02-10T05:00:09.594968+0000 mgr.y 2021-02-10T05:00:11.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:10 smithi164 conmon[29288]: (mgr.14140) 1553 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:00:11.671 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd set nodeep-scrub 2021-02-10T05:00:11.761 INFO:teuthology.orchestra.run.smithi085.stderr:pool 'unique_pool_0' removed 2021-02-10T05:00:11.906 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:00:11.957 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:11 smithi085 conmon[29319]: cluster 2021-02-10T05:00:10. 2021-02-10T05:00:11.958 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:11 smithi085 conmon[29319]: 579182+0000 mgr.y (mgr.14140) 1554 : cluster [DBG] pgmap v2523: 208 pgs: 7 active+remapped+backfill_toofull, 201 active+clean; 844 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 120/2157 objects misplaced (5.563%) 2021-02-10T05:00:11.958 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:11 smithi085 conmon[29319]: audit 2021-02-10T05:00:11.236398+0000 mon.c (mon.1) 171 : audit [INF] from='client.? 172.21.15.85:0/2323471136' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2021-02-10T05:00:11.958 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:11 smithi085 conmon[29319]: audit 2021-02-10T05:00:11.236731+0000 mon.a (mon.0) 3083 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2021-02-10T05:00:11.958 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:11 smithi085 conmon[35961]: cluster 2021-02-10T05:00:10.579182 2021-02-10T05:00:11.958 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:11 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1554 : cluster [DBG] pgmap v2523: 208 pgs: 7 active+remapped+backfill_toofull, 201 active+clean; 844 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 120/2157 objects misplaced (5.563%) 2021-02-10T05:00:11.958 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:11 smithi085 conmon[35961]: audit 2021-02-10T05:00:11.958 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:11 smithi085 conmon[35961]: 2021-02-10T05:00:11.236398+0000 mon.c (mon.1) 171 : audit [INF] 2021-02-10T05:00:11.958 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:11 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/2323471136' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2021-02-10T05:00:11.958 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:11 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T05:00:11.958 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:11 smithi085 conmon[35961]: T05:00:11.236731+0000 mon.a (mon 2021-02-10T05:00:11.959 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:11 smithi085 conmon[35961]: .0) 3083 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2021-02-10T05:00:12.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:11 smithi164 conmon[29288]: cluster 2021-02-10T05:00:10.579182+0000 mgr.y (mgr.14140) 1554 : cluster [DBG] pgmap v2523: 208 pgs: 7 active+remapped+backfill_toofull, 201 active+clean; 844 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 120/2157 objects misplaced (5.563%) 2021-02-10T05:00:12.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:11 smithi164 conmon[29288]: audit 2021-02-10T05:00:11.236398+0000 2021-02-10T05:00:12.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:11 smithi164 conmon[29288]: mon.c (mon.1) 171 : audit [INF] from='client.? 172.21.15.85:0/2323471136' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2021-02-10T05:00:12.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:11 smithi164 conmon[29288]: audit 2021-02-10T05:00:11.236731+0000 mon.a (mon.0) 3083 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2021-02-10T05:00:12.878 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2021-02-10T05:00:12.885 INFO:tasks.thrashosds:joining thrashosds 2021-02-10T05:00:12.924 INFO:teuthology.orchestra.run.smithi085.stdout:osd.0: {} 2021-02-10T05:00:12.941 INFO:teuthology.orchestra.run.smithi085.stdout:osd.1: {} 2021-02-10T05:00:12.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:12 smithi085 conmon[29319]: audit 2021-02-10T05:00:11.752295+0000 mon.a ( 2021-02-10T05:00:12.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:12 smithi085 conmon[29319]: mon.0) 3084 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2021-02-10T05:00:12.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:12 smithi085 conmon[29319]: cluster 2021-02-10T05:00:11.752396+0000 mon.a (mon.0) 3085 : cluster [DBG] osdmap e1378: 8 total, 8 up, 7 in 2021-02-10T05:00:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:12 smithi085 conmon[35961]: audit 2021-02-10T05:00 2021-02-10T05:00:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:12 smithi085 conmon[35961]: :11.752295+0000 mon.a (mon.0) 3084 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2021-02-10T05:00:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:12 smithi085 conmon[35961]: cluster 2021 2021-02-10T05:00:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:12 smithi085 conmon[35961]: -02-10T05:00: 2021-02-10T05:00:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:12 smithi085 conmon[35961]: 11.752396+0000 mon.a (mon 2021-02-10T05:00:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:12 smithi085 conmon[35961]: .0) 3085 : cluster [DBG] 2021-02-10T05:00:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:12 smithi085 conmon[35961]: osdmap e1378: 8 total, 8 up, 7 in 2021-02-10T05:00:12.960 INFO:teuthology.orchestra.run.smithi085.stdout:osd.2: {} 2021-02-10T05:00:12.978 INFO:teuthology.orchestra.run.smithi085.stdout:osd.3: {} 2021-02-10T05:00:12.998 INFO:teuthology.orchestra.run.smithi085.stdout:osd.4: {} 2021-02-10T05:00:13.017 INFO:teuthology.orchestra.run.smithi085.stdout:osd.5: {} 2021-02-10T05:00:13.036 INFO:teuthology.orchestra.run.smithi085.stdout:osd.6: {} 2021-02-10T05:00:13.055 INFO:teuthology.orchestra.run.smithi085.stdout:osd.7: {} 2021-02-10T05:00:13.056 INFO:teuthology.orchestra.run.smithi085.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-02-10T05:00:13.056 INFO:teuthology.orchestra.run.smithi085.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-02-10T05:00:13.056 INFO:teuthology.orchestra.run.smithi085.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-02-10T05:00:13.056 INFO:teuthology.orchestra.run.smithi085.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-02-10T05:00:13.056 INFO:teuthology.orchestra.run.smithi085.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-02-10T05:00:13.056 INFO:teuthology.orchestra.run.smithi085.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-02-10T05:00:13.056 INFO:teuthology.orchestra.run.smithi085.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-02-10T05:00:13.056 INFO:teuthology.orchestra.run.smithi085.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-02-10T05:00:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:12 smithi164 conmon[29288]: audit 2021-02-10T05:00:11.752295+0000 mon.a (mon.0) 3084 : 2021-02-10T05:00:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:12 smithi164 conmon[29288]: audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2021-02-10T05:00:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:12 smithi164 conmon[29288]: cluster 2021-02-10T05:00:11.752396+0000 mon.a (mon.0) 3085 2021-02-10T05:00:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:12 smithi164 conmon[29288]: : 2021-02-10T05:00:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:12 smithi164 conmon[29288]: cluster [DBG] osdmap e1378: 8 total, 8 up, 7 in 2021-02-10T05:00:13.655 INFO:teuthology.orchestra.run.smithi085.stderr:set 2.5 pg_upmap_items mapping to [1->4,6->3,7->5] 2021-02-10T05:00:13.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:13 smithi085 conmon[29319]: cluster 2021-02-10T05:00:12.579726+0000 mgr.y (mgr.14140) 1555 : cluster [DBG] pgmap v2525: 172 pgs: 7 active+remapped+backfill_toofull, 165 active+clean; 25 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 120/1077 objects misplaced (11.142%) 2021-02-10T05:00:13.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:13 smithi085 conmon[29319]: audit 2021-02-10T05:00:12.905780+0000 mon.a (mon.0) 3086 : audit [DBG] from='client.? 172.21.15.85:0/814831805' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T05:00:13.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:13 smithi085 conmon[29319]: audit 2021-02-10T05:00:13.414334+0000 mon.c (mon.1) 172 : audit [INF] from='client.? 172.21.15.85:0/323771513' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.5", "id": [1, 4, 6, 3, 7, 5]}]: dispatch 2021-02-10T05:00:13.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:13 smithi085 conmon[29319]: audit 2021-02-10T05:00:13.414713+0000 mon.a (mon.0) 3087 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.5", "id": [1, 4, 6, 3, 7, 5]}]: dispatch 2021-02-10T05:00:13.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:13 smithi085 conmon[29319]: audit 2021-02-10T05:00:13.650690+0000 mon.a (mon.0) 3088 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.5", "id": [1, 4, 6, 3, 7, 5]}]': finished 2021-02-10T05:00:13.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:13 smithi085 conmon[29319]: cluster 2021-02-10T05:00:13.650759+0000 mon.a (mon.0) 3089 : cluster [DBG] osdmap e1379: 8 total, 8 up, 7 in 2021-02-10T05:00:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:13 smithi085 conmon[35961]: cluster 2021-02-10T05:00:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:13 smithi085 conmon[35961]: 2021-02-10T05:00:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:13 smithi085 conmon[35961]: 2021-02-10T05:00:12.579726 2021-02-10T05:00:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:13 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1555 : cluster [DBG] pgmap v2525: 172 pgs: 7 active+remapped+backfill_toofull, 165 active+clean; 25 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 120/1077 objects misplaced (11.142%) 2021-02-10T05:00:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:13 smithi085 conmon[35961]: audit 2021-02-10T05 2021-02-10T05:00:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:13 smithi085 conmon[35961]: :00:12.905780+0000 mon.a (mon.0) 3086 : audit 2021-02-10T05:00:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:13 smithi085 conmon[35961]: [DBG] from='client.? 172.21.15.85:0/814831805' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T05:00:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:13 smithi085 conmon[35961]: audit 2021-02-10T05:00:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:13 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T05:00:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:13 smithi085 conmon[35961]: 05:00:13.414334+0000 mon.c (mon. 2021-02-10T05:00:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:13 smithi085 conmon[35961]: 1) 172 : audit [INF] from='client.? 172.21.15.85:0/323771513' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.5", "id": [1, 4, 6, 3, 7, 5]}]: dispatch 2021-02-10T05:00:13.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:13 smithi085 conmon[35961]: audit 2021-02-10T05:00:13.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:13 smithi085 conmon[35961]: 2021-02-10T05:00:13 2021-02-10T05:00:13.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:13 smithi085 conmon[35961]: .414713+0000 mon.a (mon.0) 3087 : audit 2021-02-10T05:00:13.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:13 smithi085 conmon[35961]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.5", "id": [1, 4, 6, 3, 7, 5]}]: dispatch 2021-02-10T05:00:13.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:13 smithi085 conmon[35961]: audit 2021-02-10 2021-02-10T05:00:13.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:13 smithi085 conmon[35961]: T05:00:13.650690+0000 mon.a (mon 2021-02-10T05:00:13.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:13 smithi085 conmon[35961]: .0) 3088 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.5", "id": [1, 4, 6, 3, 7, 5]}]': finished 2021-02-10T05:00:13.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:13 smithi085 conmon[35961]: cluster 2021 2021-02-10T05:00:13.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:13 smithi085 conmon[35961]: -02-10T05:00:13.650759+0000 2021-02-10T05:00:13.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:13 smithi085 conmon[35961]: mon.a (mon.0) 3089 : cluster [DBG] 2021-02-10T05:00:13.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:13 smithi085 conmon[35961]: osdmap e1379: 8 total, 8 up, 7 in 2021-02-10T05:00:14.018 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T05:00:14.018 INFO:teuthology.orchestra.run.smithi164.stdout: "size": 20, 2021-02-10T05:00:14.018 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 600, 2021-02-10T05:00:14.018 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [ 2021-02-10T05:00:14.018 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.018 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPushReply(7.19 1121/1119 [PushReplyOp(7:9fd44245:::smithi08575397-20:320)])", 2021-02-10T05:00:14.018 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:51:38.637056+0000", 2021-02-10T05:00:14.018 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 515.37778484800003, 2021-02-10T05:00:14.018 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0097061760000000004, 2021-02-10T05:00:14.019 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:00:14.019 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T05:00:14.019 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:00:14.019 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.019 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:00:14.019 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.637056+0000", 2021-02-10T05:00:14.019 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:00:14.019 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.019 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.019 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:00:14.019 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.637056+0000", 2021-02-10T05:00:14.019 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7246999999999999e-05 2021-02-10T05:00:14.019 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.020 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.020 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:00:14.020 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.637073+0000", 2021-02-10T05:00:14.020 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.5610000000000004e-06 2021-02-10T05:00:14.020 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.020 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.020 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:00:14.020 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.637080+0000", 2021-02-10T05:00:14.020 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2588e-05 2021-02-10T05:00:14.020 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.020 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.020 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:00:14.021 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.637092+0000", 2021-02-10T05:00:14.021 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.2509999999999997e-06 2021-02-10T05:00:14.021 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.021 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.021 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:00:14.021 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.637100+0000", 2021-02-10T05:00:14.021 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.009392503 2021-02-10T05:00:14.021 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.021 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.021 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:00:14.021 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.646492+0000", 2021-02-10T05:00:14.021 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00027002599999999998 2021-02-10T05:00:14.021 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.022 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.022 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:00:14.022 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.646762+0000", 2021-02-10T05:00:14.022 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0097061760000000004 2021-02-10T05:00:14.022 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.022 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:00:14.022 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.022 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.022 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.022 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPushReply(7.19 1121/1119 [PushReplyOp(7:9fd44245:::smithi08575397-20:32b)])", 2021-02-10T05:00:14.022 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:51:38.673650+0000", 2021-02-10T05:00:14.022 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 515.34119075299998, 2021-02-10T05:00:14.022 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0095316570000000007, 2021-02-10T05:00:14.023 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:00:14.023 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T05:00:14.023 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:00:14.023 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.023 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:00:14.023 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.673650+0000", 2021-02-10T05:00:14.023 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:00:14.023 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.023 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.023 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:00:14.023 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.673650+0000", 2021-02-10T05:00:14.023 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.0898000000000001e-05 2021-02-10T05:00:14.023 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.023 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.024 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:00:14.024 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.673681+0000", 2021-02-10T05:00:14.024 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1110000000000008e-06 2021-02-10T05:00:14.024 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.024 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.024 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:00:14.024 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.673689+0000", 2021-02-10T05:00:14.024 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1749999999999999e-05 2021-02-10T05:00:14.024 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.024 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.024 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:00:14.024 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.673701+0000", 2021-02-10T05:00:14.024 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.4670000000000001e-06 2021-02-10T05:00:14.025 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.025 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.025 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:00:14.025 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.673707+0000", 2021-02-10T05:00:14.025 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0083215470000000003 2021-02-10T05:00:14.025 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.025 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.025 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:00:14.025 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.682029+0000", 2021-02-10T05:00:14.025 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0011528840000000001 2021-02-10T05:00:14.025 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.025 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.026 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:00:14.026 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.683182+0000", 2021-02-10T05:00:14.026 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0095316570000000007 2021-02-10T05:00:14.026 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.026 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:00:14.026 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.026 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.026 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.026 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPushReply(7.19 1121/1119 [PushReplyOp(7:9fd44245:::smithi08575397-20:372)])", 2021-02-10T05:00:14.026 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:51:38.815832+0000", 2021-02-10T05:00:14.026 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 515.19900930599999, 2021-02-10T05:00:14.026 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0097133849999999997, 2021-02-10T05:00:14.026 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:00:14.027 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T05:00:14.027 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:00:14.027 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.027 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:00:14.027 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.815832+0000", 2021-02-10T05:00:14.027 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:00:14.027 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.027 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.027 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:00:14.027 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.815832+0000", 2021-02-10T05:00:14.027 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.5043000000000001e-05 2021-02-10T05:00:14.027 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.027 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.027 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:00:14.028 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.815857+0000", 2021-02-10T05:00:14.028 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.8670000000000002e-06 2021-02-10T05:00:14.028 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.028 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.028 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:00:14.028 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.815862+0000", 2021-02-10T05:00:14.028 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.5229999999999998e-06 2021-02-10T05:00:14.028 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.028 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.028 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:00:14.028 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.815869+0000", 2021-02-10T05:00:14.028 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.3270000000000002e-06 2021-02-10T05:00:14.028 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.029 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.029 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:00:14.029 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.815873+0000", 2021-02-10T05:00:14.029 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0086872519999999995 2021-02-10T05:00:14.029 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.029 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.029 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:00:14.029 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.824561+0000", 2021-02-10T05:00:14.029 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00098437299999999998 2021-02-10T05:00:14.029 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.029 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.029 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:00:14.029 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.825545+0000", 2021-02-10T05:00:14.029 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0097133849999999997 2021-02-10T05:00:14.030 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.030 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:00:14.030 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.030 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.030 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.030 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPushReply(7.19 1121/1119 [PushReplyOp(7:9fd44245:::smithi08575397-20:372)])", 2021-02-10T05:00:14.030 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:51:38.828580+0000", 2021-02-10T05:00:14.030 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 515.18626138100001, 2021-02-10T05:00:14.030 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0096339019999999997, 2021-02-10T05:00:14.030 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:00:14.030 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T05:00:14.030 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:00:14.030 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.031 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:00:14.031 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.828580+0000", 2021-02-10T05:00:14.031 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:00:14.031 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.031 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.031 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:00:14.031 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.828580+0000", 2021-02-10T05:00:14.031 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.3241000000000002e-05 2021-02-10T05:00:14.031 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.031 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.031 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:00:14.031 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.828613+0000", 2021-02-10T05:00:14.031 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1236999999999999e-05 2021-02-10T05:00:14.031 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.032 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.032 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:00:14.032 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.828624+0000", 2021-02-10T05:00:14.032 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4205999999999999e-05 2021-02-10T05:00:14.032 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.032 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.032 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:00:14.032 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.828638+0000", 2021-02-10T05:00:14.032 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.0210000000000005e-06 2021-02-10T05:00:14.032 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.032 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.032 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:00:14.032 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.828647+0000", 2021-02-10T05:00:14.033 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0078234389999999997 2021-02-10T05:00:14.033 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.033 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.033 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:00:14.033 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.836471+0000", 2021-02-10T05:00:14.033 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0017427580000000001 2021-02-10T05:00:14.033 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.033 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.033 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:00:14.033 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:38.838213+0000", 2021-02-10T05:00:14.033 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0096339019999999997 2021-02-10T05:00:14.033 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.033 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:00:14.034 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.034 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.034 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.034 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPushReply(7.12 1124/1119 [PushReplyOp(7:484ca2f5:::smithi08575397-23:320)])", 2021-02-10T05:00:14.034 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:51:42.011621+0000", 2021-02-10T05:00:14.034 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 512.00322022800003, 2021-02-10T05:00:14.034 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0098031100000000003, 2021-02-10T05:00:14.034 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:00:14.034 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T05:00:14.034 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:00:14.034 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.034 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:00:14.035 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.011621+0000", 2021-02-10T05:00:14.035 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:00:14.035 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.035 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.035 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:00:14.035 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.011621+0000", 2021-02-10T05:00:14.035 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.7418000000000001e-05 2021-02-10T05:00:14.035 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.035 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.035 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:00:14.035 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.011648+0000", 2021-02-10T05:00:14.035 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.592e-06 2021-02-10T05:00:14.035 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.036 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.036 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:00:14.036 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.011654+0000", 2021-02-10T05:00:14.036 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.6429999999999995e-06 2021-02-10T05:00:14.036 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.036 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.036 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:00:14.036 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.011661+0000", 2021-02-10T05:00:14.036 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.4660000000000001e-06 2021-02-10T05:00:14.036 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.036 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.036 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:00:14.037 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.011666+0000", 2021-02-10T05:00:14.037 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0088022929999999992 2021-02-10T05:00:14.037 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.037 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.037 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:00:14.037 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.020468+0000", 2021-02-10T05:00:14.037 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00095569800000000005 2021-02-10T05:00:14.037 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.037 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.037 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:00:14.037 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.021424+0000", 2021-02-10T05:00:14.037 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0098031100000000003 2021-02-10T05:00:14.038 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.038 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:00:14.038 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.038 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.038 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.038 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPushReply(7.12 1124/1119 [PushReplyOp(7:484ca2f5:::smithi08575397-23:320)])", 2021-02-10T05:00:14.038 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:51:42.034551+0000", 2021-02-10T05:00:14.038 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 511.980289605, 2021-02-10T05:00:14.038 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0099585999999999997, 2021-02-10T05:00:14.038 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:00:14.038 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T05:00:14.038 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:00:14.039 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.039 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:00:14.039 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.034551+0000", 2021-02-10T05:00:14.039 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:00:14.039 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.039 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.039 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:00:14.039 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.034551+0000", 2021-02-10T05:00:14.039 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7431e-05 2021-02-10T05:00:14.039 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.039 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.039 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:00:14.040 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.034569+0000", 2021-02-10T05:00:14.040 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.3749999999999999e-06 2021-02-10T05:00:14.040 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.040 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.040 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:00:14.040 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.034575+0000", 2021-02-10T05:00:14.040 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.8359999999999992e-06 2021-02-10T05:00:14.040 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.040 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.040 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:00:14.040 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.034583+0000", 2021-02-10T05:00:14.040 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.4320000000000001e-06 2021-02-10T05:00:14.040 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.041 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.041 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:00:14.041 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.034587+0000", 2021-02-10T05:00:14.041 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0096822220000000007 2021-02-10T05:00:14.041 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.041 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.041 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:00:14.041 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.044270+0000", 2021-02-10T05:00:14.041 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000240304 2021-02-10T05:00:14.041 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.041 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.041 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:00:14.041 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.044510+0000", 2021-02-10T05:00:14.041 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0099585999999999997 2021-02-10T05:00:14.042 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.042 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:00:14.042 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.042 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.042 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.042 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPushReply(7.12 1124/1119 [PushReplyOp(7:4e1089df:::smithi08575397-36:32b)])", 2021-02-10T05:00:14.042 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:51:42.070322+0000", 2021-02-10T05:00:14.042 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 511.944518759, 2021-02-10T05:00:14.042 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010101593000000001, 2021-02-10T05:00:14.042 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:00:14.042 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T05:00:14.042 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:00:14.043 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.043 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:00:14.043 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.070322+0000", 2021-02-10T05:00:14.043 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:00:14.043 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.043 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.043 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:00:14.043 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.070322+0000", 2021-02-10T05:00:14.043 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.0620999999999997e-05 2021-02-10T05:00:14.043 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.043 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.043 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:00:14.044 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.070353+0000", 2021-02-10T05:00:14.044 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.8589999999999999e-06 2021-02-10T05:00:14.044 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.044 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.044 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:00:14.044 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.070360+0000", 2021-02-10T05:00:14.044 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4623999999999999e-05 2021-02-10T05:00:14.044 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.044 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.044 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:00:14.044 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.070374+0000", 2021-02-10T05:00:14.044 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.5140000000000004e-06 2021-02-10T05:00:14.044 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.045 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.045 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:00:14.045 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.070382+0000", 2021-02-10T05:00:14.045 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0098706009999999997 2021-02-10T05:00:14.045 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.045 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.045 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:00:14.045 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.080252+0000", 2021-02-10T05:00:14.045 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00017137399999999999 2021-02-10T05:00:14.045 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.045 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.045 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:00:14.046 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.080424+0000", 2021-02-10T05:00:14.046 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010101593000000001 2021-02-10T05:00:14.046 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.046 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:00:14.046 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.046 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.046 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.046 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPushReply(7.12 1124/1119 [PushReplyOp(7:484ca2f5:::smithi08575397-23:28f)])", 2021-02-10T05:00:14.046 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:51:42.297230+0000", 2021-02-10T05:00:14.046 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 511.71761142700001, 2021-02-10T05:00:14.046 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0098073039999999993, 2021-02-10T05:00:14.046 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:00:14.046 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T05:00:14.047 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:00:14.047 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.047 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:00:14.047 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.297230+0000", 2021-02-10T05:00:14.047 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:00:14.047 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.047 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.047 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:00:14.047 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.297230+0000", 2021-02-10T05:00:14.047 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7042999999999999e-05 2021-02-10T05:00:14.047 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.047 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.048 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:00:14.048 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.297247+0000", 2021-02-10T05:00:14.048 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.8250000000000004e-06 2021-02-10T05:00:14.048 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.048 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.048 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:00:14.048 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.297251+0000", 2021-02-10T05:00:14.048 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.4570000000000005e-06 2021-02-10T05:00:14.048 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.048 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.048 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:00:14.048 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.297260+0000", 2021-02-10T05:00:14.048 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.707e-06 2021-02-10T05:00:14.049 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.049 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.049 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:00:14.049 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.297265+0000", 2021-02-10T05:00:14.049 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0088711150000000006 2021-02-10T05:00:14.049 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.049 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.049 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:00:14.049 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.306136+0000", 2021-02-10T05:00:14.049 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00090115699999999998 2021-02-10T05:00:14.049 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.049 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.050 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:00:14.050 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.307037+0000", 2021-02-10T05:00:14.050 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0098073039999999993 2021-02-10T05:00:14.050 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.050 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:00:14.050 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.050 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.050 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.050 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPushReply(7.12 1124/1119 [PushReplyOp(7:484ca2f5:::smithi08575397-23:28f)])", 2021-02-10T05:00:14.050 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:51:42.309120+0000", 2021-02-10T05:00:14.050 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 511.70572058499999, 2021-02-10T05:00:14.051 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0095591960000000007, 2021-02-10T05:00:14.051 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:00:14.051 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T05:00:14.051 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:00:14.051 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.051 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:00:14.051 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.309120+0000", 2021-02-10T05:00:14.051 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:00:14.051 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.051 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.051 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:00:14.051 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.309120+0000", 2021-02-10T05:00:14.051 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1809e-05 2021-02-10T05:00:14.052 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.052 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.052 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:00:14.052 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.309152+0000", 2021-02-10T05:00:14.052 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.9999999999999996e-06 2021-02-10T05:00:14.052 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.052 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.052 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:00:14.052 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.309160+0000", 2021-02-10T05:00:14.052 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5373e-05 2021-02-10T05:00:14.052 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.052 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.053 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:00:14.053 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.309176+0000", 2021-02-10T05:00:14.053 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.8600000000000004e-06 2021-02-10T05:00:14.053 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.053 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.053 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:00:14.053 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.309182+0000", 2021-02-10T05:00:14.053 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0088539199999999995 2021-02-10T05:00:14.053 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.053 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.053 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:00:14.053 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.318036+0000", 2021-02-10T05:00:14.053 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00064323400000000001 2021-02-10T05:00:14.053 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.054 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.054 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:00:14.054 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.318680+0000", 2021-02-10T05:00:14.054 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0095591960000000007 2021-02-10T05:00:14.054 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.054 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:00:14.054 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.054 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.054 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.054 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPushReply(7.12 1124/1119 [PushReplyOp(7:484ca2f5:::smithi08575397-23:28f)])", 2021-02-10T05:00:14.054 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:51:42.320232+0000", 2021-02-10T05:00:14.054 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 511.694608453, 2021-02-10T05:00:14.054 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010015698, 2021-02-10T05:00:14.055 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:00:14.055 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T05:00:14.055 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:00:14.055 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.055 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:00:14.055 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.320232+0000", 2021-02-10T05:00:14.055 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:00:14.055 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.055 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.055 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:00:14.055 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.320232+0000", 2021-02-10T05:00:14.055 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.4786999999999998e-05 2021-02-10T05:00:14.055 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.056 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.056 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:00:14.056 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.320257+0000", 2021-02-10T05:00:14.056 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.0049999999999998e-06 2021-02-10T05:00:14.056 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.056 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.056 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:00:14.056 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.320264+0000", 2021-02-10T05:00:14.056 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0848e-05 2021-02-10T05:00:14.056 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.056 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.056 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:00:14.056 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.320275+0000", 2021-02-10T05:00:14.056 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.131e-06 2021-02-10T05:00:14.057 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.057 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.057 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:00:14.057 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.320282+0000", 2021-02-10T05:00:14.057 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0096587119999999999 2021-02-10T05:00:14.057 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.057 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.057 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:00:14.057 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.329941+0000", 2021-02-10T05:00:14.057 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00030721500000000001 2021-02-10T05:00:14.057 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.057 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.057 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:00:14.058 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.330248+0000", 2021-02-10T05:00:14.058 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010015698 2021-02-10T05:00:14.058 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.058 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:00:14.058 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.058 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.058 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.058 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPushReply(7.12 1124/1119 [PushReplyOp(7:4e1089df:::smithi08575397-36:2ae)])", 2021-02-10T05:00:14.058 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:51:42.451427+0000", 2021-02-10T05:00:14.058 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 511.563414362, 2021-02-10T05:00:14.058 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010379849, 2021-02-10T05:00:14.058 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:00:14.058 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T05:00:14.059 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:00:14.059 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.059 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:00:14.059 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.451427+0000", 2021-02-10T05:00:14.059 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:00:14.059 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.059 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.059 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:00:14.059 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.451427+0000", 2021-02-10T05:00:14.059 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8593e-05 2021-02-10T05:00:14.059 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.059 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.059 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:00:14.059 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.451445+0000", 2021-02-10T05:00:14.060 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.2650000000000002e-06 2021-02-10T05:00:14.060 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.060 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.060 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:00:14.060 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.451454+0000", 2021-02-10T05:00:14.060 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6314999999999999e-05 2021-02-10T05:00:14.060 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.060 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.060 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:00:14.060 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.451471+0000", 2021-02-10T05:00:14.060 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.4740000000000002e-06 2021-02-10T05:00:14.060 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.060 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.061 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:00:14.061 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.451477+0000", 2021-02-10T05:00:14.061 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010011664999999999 2021-02-10T05:00:14.061 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.061 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.061 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:00:14.061 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.461489+0000", 2021-02-10T05:00:14.061 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00031753700000000001 2021-02-10T05:00:14.061 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.061 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.061 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:00:14.061 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.461806+0000", 2021-02-10T05:00:14.062 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010379849 2021-02-10T05:00:14.062 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.062 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:00:14.062 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.062 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.062 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.062 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPushReply(7.12 1125/1119 [PushReplyOp(7:4e1089df:::smithi08575397-36:358)])", 2021-02-10T05:00:14.062 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:51:42.831902+0000", 2021-02-10T05:00:14.062 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 511.18293892999998, 2021-02-10T05:00:14.062 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010245440999999999, 2021-02-10T05:00:14.062 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:00:14.062 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T05:00:14.062 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:00:14.063 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.063 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:00:14.063 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.831902+0000", 2021-02-10T05:00:14.063 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:00:14.063 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.063 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.063 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:00:14.063 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.831902+0000", 2021-02-10T05:00:14.063 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.1631e-05 2021-02-10T05:00:14.063 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.063 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.063 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:00:14.063 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.831924+0000", 2021-02-10T05:00:14.063 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9699999999999998e-06 2021-02-10T05:00:14.064 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.064 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.064 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:00:14.064 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.831929+0000", 2021-02-10T05:00:14.064 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.4689999999999997e-06 2021-02-10T05:00:14.064 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.064 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.064 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:00:14.064 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.831937+0000", 2021-02-10T05:00:14.064 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.1009999999999997e-06 2021-02-10T05:00:14.064 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.064 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.065 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:00:14.065 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.831943+0000", 2021-02-10T05:00:14.065 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010021138000000001 2021-02-10T05:00:14.065 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.065 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.065 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:00:14.065 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.841964+0000", 2021-02-10T05:00:14.065 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00018313199999999999 2021-02-10T05:00:14.065 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.065 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.065 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:00:14.065 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:42.842147+0000", 2021-02-10T05:00:14.065 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.010245440999999999 2021-02-10T05:00:14.066 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.066 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:00:14.066 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.066 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.066 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.066 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(osd.2.18:9 7.7 7:e3fec724:::smithi08575397-41:head [assert-version v158,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected e1119)", 2021-02-10T05:00:14.066 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:52:41.683638+0000", 2021-02-10T05:00:14.066 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 452.33120251899999, 2021-02-10T05:00:14.066 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0036160649999999999, 2021-02-10T05:00:14.066 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:00:14.066 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T05:00:14.066 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:00:14.067 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.067 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:00:14.067 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.683638+0000", 2021-02-10T05:00:14.067 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:00:14.067 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.067 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.067 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:00:14.067 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.683638+0000", 2021-02-10T05:00:14.067 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.0250000000000003e-05 2021-02-10T05:00:14.067 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.067 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.067 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:00:14.067 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.683679+0000", 2021-02-10T05:00:14.068 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.3920000000000002e-06 2021-02-10T05:00:14.068 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.068 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.068 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:00:14.068 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.683684+0000", 2021-02-10T05:00:14.068 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0539e-05 2021-02-10T05:00:14.068 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.068 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.068 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:00:14.068 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.683695+0000", 2021-02-10T05:00:14.068 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.5140000000000001e-06 2021-02-10T05:00:14.068 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.068 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.068 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:00:14.069 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.683700+0000", 2021-02-10T05:00:14.069 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.8287000000000003e-05 2021-02-10T05:00:14.069 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.069 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.069 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:00:14.069 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.683798+0000", 2021-02-10T05:00:14.069 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.2349999999999996e-05 2021-02-10T05:00:14.069 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.069 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.069 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T05:00:14.069 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.683881+0000", 2021-02-10T05:00:14.069 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0033737329999999999 2021-02-10T05:00:14.069 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.069 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.070 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:00:14.070 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.687254+0000", 2021-02-10T05:00:14.070 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0036160649999999999 2021-02-10T05:00:14.070 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.070 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:00:14.070 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.070 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.070 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.070 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDPGPushReply(4.5 1201/1200 [PushReplyOp(4:a2ace8b3:::data_log.0:head)])", 2021-02-10T05:00:14.070 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:54:17.086793+0000", 2021-02-10T05:00:14.070 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 356.92804812000003, 2021-02-10T05:00:14.070 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0062416980000000004, 2021-02-10T05:00:14.071 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:00:14.071 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T05:00:14.071 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:00:14.071 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.071 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:00:14.071 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:17.086793+0000", 2021-02-10T05:00:14.071 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:00:14.071 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.071 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.071 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:00:14.071 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:17.086793+0000", 2021-02-10T05:00:14.071 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1062000000000002e-05 2021-02-10T05:00:14.071 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.072 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.072 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:00:14.072 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:17.086824+0000", 2021-02-10T05:00:14.072 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.9160000000000001e-06 2021-02-10T05:00:14.072 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.072 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.072 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:00:14.072 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:17.086832+0000", 2021-02-10T05:00:14.072 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1676e-05 2021-02-10T05:00:14.072 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.072 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.072 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:00:14.072 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:17.086843+0000", 2021-02-10T05:00:14.073 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.1960000000000002e-06 2021-02-10T05:00:14.073 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.073 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.073 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:00:14.073 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:17.086851+0000", 2021-02-10T05:00:14.073 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.005942134 2021-02-10T05:00:14.073 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.073 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.073 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:00:14.073 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:17.092793+0000", 2021-02-10T05:00:14.073 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000241714 2021-02-10T05:00:14.073 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.073 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.074 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:00:14.074 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:54:17.093035+0000", 2021-02-10T05:00:14.074 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0062416980000000004 2021-02-10T05:00:14.074 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.074 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:00:14.074 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.074 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.074 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.074 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.24382.0:1796 4.a 4:55b0279d:reshard::reshard.0000000009:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e1260)", 2021-02-10T05:00:14.074 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:55:55.369727+0000", 2021-02-10T05:00:14.074 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 258.64511379300001, 2021-02-10T05:00:14.074 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0014305419999999999, 2021-02-10T05:00:14.074 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:00:14.075 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T05:00:14.075 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T05:00:14.075 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.24382", 2021-02-10T05:00:14.075 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/3915233377", 2021-02-10T05:00:14.075 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 1796 2021-02-10T05:00:14.075 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.075 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:00:14.075 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.075 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:00:14.075 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.369727+0000", 2021-02-10T05:00:14.075 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:00:14.075 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.075 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.076 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:00:14.076 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.369727+0000", 2021-02-10T05:00:14.076 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.5062999999999999e-05 2021-02-10T05:00:14.076 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.076 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.076 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:00:14.076 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.369772+0000", 2021-02-10T05:00:14.076 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.4659999999999999e-06 2021-02-10T05:00:14.076 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.076 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.076 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:00:14.076 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.369781+0000", 2021-02-10T05:00:14.077 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4781e-05 2021-02-10T05:00:14.077 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.077 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.077 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:00:14.077 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.369795+0000", 2021-02-10T05:00:14.077 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.6820000000000003e-06 2021-02-10T05:00:14.077 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.077 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.077 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:00:14.077 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.369803+0000", 2021-02-10T05:00:14.077 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.6399000000000001e-05 2021-02-10T05:00:14.077 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.077 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.078 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:00:14.078 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.369850+0000", 2021-02-10T05:00:14.078 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000125582 2021-02-10T05:00:14.078 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.078 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.078 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T05:00:14.078 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.369975+0000", 2021-02-10T05:00:14.078 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000186823 2021-02-10T05:00:14.078 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.078 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.078 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 1,7", 2021-02-10T05:00:14.078 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.370162+0000", 2021-02-10T05:00:14.079 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00042619100000000003 2021-02-10T05:00:14.079 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.079 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.079 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T05:00:14.079 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.370588+0000", 2021-02-10T05:00:14.079 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00038333300000000002 2021-02-10T05:00:14.079 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.079 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.079 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T05:00:14.079 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.370971+0000", 2021-02-10T05:00:14.079 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.9279999999999999e-05 2021-02-10T05:00:14.079 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.080 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.080 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T05:00:14.080 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.371061+0000", 2021-02-10T05:00:14.080 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6626000000000003e-05 2021-02-10T05:00:14.080 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.080 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.080 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T05:00:14.080 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.371117+0000", 2021-02-10T05:00:14.080 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.0315999999999998e-05 2021-02-10T05:00:14.080 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.080 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.080 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:00:14.080 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.371158+0000", 2021-02-10T05:00:14.081 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0014305419999999999 2021-02-10T05:00:14.081 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.081 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:00:14.081 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.081 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.081 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.081 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.24382.0:1798 4.a 4:55b0279d:reshard::reshard.0000000009:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e1260)", 2021-02-10T05:00:14.081 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:55:55.371648+0000", 2021-02-10T05:00:14.081 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 258.64319339899998, 2021-02-10T05:00:14.081 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0017401700000000001, 2021-02-10T05:00:14.081 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:00:14.081 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T05:00:14.082 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T05:00:14.082 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.24382", 2021-02-10T05:00:14.082 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/3915233377", 2021-02-10T05:00:14.082 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 1798 2021-02-10T05:00:14.082 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.082 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:00:14.082 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.082 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:00:14.082 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.371648+0000", 2021-02-10T05:00:14.082 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:00:14.082 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.082 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.083 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:00:14.083 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.371648+0000", 2021-02-10T05:00:14.083 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.6873000000000003e-05 2021-02-10T05:00:14.083 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.083 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.083 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:00:14.083 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.371694+0000", 2021-02-10T05:00:14.083 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.7229999999999998e-06 2021-02-10T05:00:14.083 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.083 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.083 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:00:14.083 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.371701+0000", 2021-02-10T05:00:14.084 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.294e-05 2021-02-10T05:00:14.084 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.084 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.084 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:00:14.084 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.371714+0000", 2021-02-10T05:00:14.084 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1076e-05 2021-02-10T05:00:14.084 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.084 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.084 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:00:14.084 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.371725+0000", 2021-02-10T05:00:14.084 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00038639000000000002 2021-02-10T05:00:14.084 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.084 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.085 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:00:14.085 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.372112+0000", 2021-02-10T05:00:14.085 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.8333999999999999e-05 2021-02-10T05:00:14.085 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.085 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.085 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T05:00:14.085 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.372190+0000", 2021-02-10T05:00:14.085 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00016950799999999999 2021-02-10T05:00:14.085 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.085 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.085 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 1,7", 2021-02-10T05:00:14.085 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.372359+0000", 2021-02-10T05:00:14.085 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00037974299999999999 2021-02-10T05:00:14.086 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.086 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.086 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T05:00:14.086 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.372739+0000", 2021-02-10T05:00:14.086 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00052858500000000004 2021-02-10T05:00:14.086 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.086 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.086 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T05:00:14.086 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.373268+0000", 2021-02-10T05:00:14.086 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.7107999999999998e-05 2021-02-10T05:00:14.086 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.086 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.087 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T05:00:14.087 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.373305+0000", 2021-02-10T05:00:14.087 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9131000000000001e-05 2021-02-10T05:00:14.087 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.087 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.087 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T05:00:14.087 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.373354+0000", 2021-02-10T05:00:14.087 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.3759e-05 2021-02-10T05:00:14.087 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.087 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.087 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:00:14.088 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.373388+0000", 2021-02-10T05:00:14.088 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0017401700000000001 2021-02-10T05:00:14.088 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.088 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:00:14.088 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.088 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.088 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.088 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.24382.0:1807 4.5 4:a7f90313:reshard::reshard.0000000012:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e1260)", 2021-02-10T05:00:14.088 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:55:55.382100+0000", 2021-02-10T05:00:14.088 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 258.63274088399999, 2021-02-10T05:00:14.088 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001372882, 2021-02-10T05:00:14.088 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:00:14.089 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T05:00:14.089 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T05:00:14.089 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.24382", 2021-02-10T05:00:14.089 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/3915233377", 2021-02-10T05:00:14.089 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 1807 2021-02-10T05:00:14.089 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.089 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:00:14.089 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.089 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:00:14.089 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.382100+0000", 2021-02-10T05:00:14.089 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:00:14.089 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.089 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.090 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:00:14.090 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.382100+0000", 2021-02-10T05:00:14.090 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.1854000000000003e-05 2021-02-10T05:00:14.090 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.090 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.090 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:00:14.090 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.382142+0000", 2021-02-10T05:00:14.090 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.6639999999999997e-06 2021-02-10T05:00:14.090 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.090 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.091 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:00:14.091 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.382147+0000", 2021-02-10T05:00:14.091 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.3749999999999997e-06 2021-02-10T05:00:14.091 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.091 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.091 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:00:14.091 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.382154+0000", 2021-02-10T05:00:14.091 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7790000000000007e-06 2021-02-10T05:00:14.091 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.091 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.091 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:00:14.091 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.382164+0000", 2021-02-10T05:00:14.091 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.9691000000000003e-05 2021-02-10T05:00:14.091 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.092 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.092 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:00:14.092 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.382203+0000", 2021-02-10T05:00:14.092 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.0742999999999996e-05 2021-02-10T05:00:14.092 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.092 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.092 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T05:00:14.092 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.382284+0000", 2021-02-10T05:00:14.092 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00017197800000000001 2021-02-10T05:00:14.092 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.092 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.092 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 2,7", 2021-02-10T05:00:14.092 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.382456+0000", 2021-02-10T05:00:14.093 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00048776400000000001 2021-02-10T05:00:14.093 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.093 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.093 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T05:00:14.093 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.382944+0000", 2021-02-10T05:00:14.093 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00019884500000000001 2021-02-10T05:00:14.093 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.093 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.093 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T05:00:14.093 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.383143+0000", 2021-02-10T05:00:14.093 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000248385 2021-02-10T05:00:14.093 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.093 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.094 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T05:00:14.094 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.383391+0000", 2021-02-10T05:00:14.094 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0380000000000001e-05 2021-02-10T05:00:14.094 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.094 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.094 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T05:00:14.094 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.383452+0000", 2021-02-10T05:00:14.094 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.1424e-05 2021-02-10T05:00:14.094 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.094 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.094 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:00:14.094 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.383473+0000", 2021-02-10T05:00:14.094 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001372882 2021-02-10T05:00:14.095 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.095 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:00:14.095 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.095 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.095 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.095 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.24382.0:1816 4.a 4:5c7e8a6d:reshard::reshard.0000000015:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e1260)", 2021-02-10T05:00:14.095 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:55:55.391951+0000", 2021-02-10T05:00:14.095 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 258.62289039699999, 2021-02-10T05:00:14.095 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0014484490000000001, 2021-02-10T05:00:14.095 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:00:14.095 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T05:00:14.095 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T05:00:14.096 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.24382", 2021-02-10T05:00:14.096 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/3915233377", 2021-02-10T05:00:14.096 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 1816 2021-02-10T05:00:14.096 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.096 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:00:14.096 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.096 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:00:14.096 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.391951+0000", 2021-02-10T05:00:14.096 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:00:14.096 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.096 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.096 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:00:14.096 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.391951+0000", 2021-02-10T05:00:14.096 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.2986999999999999e-05 2021-02-10T05:00:14.097 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.097 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.097 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:00:14.097 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.391974+0000", 2021-02-10T05:00:14.097 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.143e-06 2021-02-10T05:00:14.097 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.097 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.097 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:00:14.097 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.391978+0000", 2021-02-10T05:00:14.097 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.8719999999999996e-06 2021-02-10T05:00:14.097 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.097 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.097 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:00:14.098 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.391985+0000", 2021-02-10T05:00:14.098 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.9979999999999997e-06 2021-02-10T05:00:14.098 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.098 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.098 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:00:14.098 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.391989+0000", 2021-02-10T05:00:14.098 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00038064700000000002 2021-02-10T05:00:14.098 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.098 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.098 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:00:14.098 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.392369+0000", 2021-02-10T05:00:14.099 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.6422999999999999e-05 2021-02-10T05:00:14.099 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.099 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.099 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T05:00:14.099 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.392416+0000", 2021-02-10T05:00:14.099 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.8925000000000006e-05 2021-02-10T05:00:14.099 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.099 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.099 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "waiting for subops from 1,7", 2021-02-10T05:00:14.099 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.392515+0000", 2021-02-10T05:00:14.099 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00035246499999999999 2021-02-10T05:00:14.099 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.099 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.100 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op_commit", 2021-02-10T05:00:14.100 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.392867+0000", 2021-02-10T05:00:14.100 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00029486200000000001 2021-02-10T05:00:14.100 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.100 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.100 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T05:00:14.100 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.393162+0000", 2021-02-10T05:00:14.100 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00015411799999999999 2021-02-10T05:00:14.100 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.100 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.100 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "sub_op_commit_rec", 2021-02-10T05:00:14.100 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.393316+0000", 2021-02-10T05:00:14.100 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.3789999999999998e-05 2021-02-10T05:00:14.100 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.101 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.101 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T05:00:14.101 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.393370+0000", 2021-02-10T05:00:14.101 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.9218999999999999e-05 2021-02-10T05:00:14.101 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.101 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.101 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:00:14.101 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:55.393399+0000", 2021-02-10T05:00:14.101 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0014484490000000001 2021-02-10T05:00:14.101 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.101 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:00:14.101 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.101 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.102 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.102 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.24373.0:4259 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94187770079488 gen 5] snapc 0=[] ondisk+write+known_if_redirected e1317)", 2021-02-10T05:00:14.102 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:58:13.066341+0000", 2021-02-10T05:00:14.102 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 120.94849972900001, 2021-02-10T05:00:14.102 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0017203800000000001, 2021-02-10T05:00:14.102 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:00:14.102 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T05:00:14.102 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T05:00:14.102 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.24373", 2021-02-10T05:00:14.102 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T05:00:14.102 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 4259 2021-02-10T05:00:14.102 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.103 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:00:14.103 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.103 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:00:14.103 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:58:13.066341+0000", 2021-02-10T05:00:14.103 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:00:14.103 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.103 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.103 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:00:14.103 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:58:13.066341+0000", 2021-02-10T05:00:14.103 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.9862e-05 2021-02-10T05:00:14.103 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.103 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.103 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:00:14.104 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:58:13.066371+0000", 2021-02-10T05:00:14.104 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6279999999999999e-06 2021-02-10T05:00:14.104 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.104 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.104 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:00:14.104 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:58:13.066378+0000", 2021-02-10T05:00:14.104 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2364999999999999e-05 2021-02-10T05:00:14.104 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.104 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.104 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:00:14.104 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:58:13.066390+0000", 2021-02-10T05:00:14.105 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.7440000000000001e-06 2021-02-10T05:00:14.105 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.105 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.105 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:00:14.105 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:58:13.066397+0000", 2021-02-10T05:00:14.105 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.0065999999999998e-05 2021-02-10T05:00:14.105 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.105 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.105 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:00:14.105 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:58:13.066447+0000", 2021-02-10T05:00:14.105 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00014150100000000001 2021-02-10T05:00:14.105 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.105 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.105 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T05:00:14.106 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:58:13.066588+0000", 2021-02-10T05:00:14.106 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001473214 2021-02-10T05:00:14.106 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.106 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.106 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:00:14.106 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:58:13.068062+0000", 2021-02-10T05:00:14.106 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0017203800000000001 2021-02-10T05:00:14.106 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.106 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:00:14.106 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.106 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.106 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.107 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.24373.0:4343 5.1f 5:f95f44c2:::notify.0:head [watch ping cookie 94187755108992] snapc 0=[] ondisk+write+known_if_redirected e1333)", 2021-02-10T05:00:14.107 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:59:03.074970+0000", 2021-02-10T05:00:14.107 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 70.939871331999996, 2021-02-10T05:00:14.107 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0017204550000000001, 2021-02-10T05:00:14.107 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:00:14.107 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T05:00:14.107 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T05:00:14.107 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.24373", 2021-02-10T05:00:14.107 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T05:00:14.107 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 4343 2021-02-10T05:00:14.107 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.107 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:00:14.107 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.108 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:00:14.108 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:59:03.074970+0000", 2021-02-10T05:00:14.108 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:00:14.108 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.108 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.108 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:00:14.108 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:59:03.074970+0000", 2021-02-10T05:00:14.108 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.7701000000000001e-05 2021-02-10T05:00:14.108 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.108 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.108 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:00:14.108 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:59:03.074997+0000", 2021-02-10T05:00:14.108 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.8540000000000002e-06 2021-02-10T05:00:14.109 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.109 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.109 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:00:14.109 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:59:03.075003+0000", 2021-02-10T05:00:14.109 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0967e-05 2021-02-10T05:00:14.109 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.109 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.109 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:00:14.109 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:59:03.075014+0000", 2021-02-10T05:00:14.109 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.5309999999999998e-06 2021-02-10T05:00:14.109 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.109 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.109 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:00:14.109 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:59:03.075021+0000", 2021-02-10T05:00:14.110 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0015240049999999999 2021-02-10T05:00:14.110 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.110 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.110 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:00:14.110 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:59:03.076545+0000", 2021-02-10T05:00:14.110 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.6161000000000005e-05 2021-02-10T05:00:14.110 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.110 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.110 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T05:00:14.110 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:59:03.076621+0000", 2021-02-10T05:00:14.110 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.9235999999999999e-05 2021-02-10T05:00:14.110 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:00:14.111 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:00:14.111 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:00:14.111 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:59:03.076690+0000", 2021-02-10T05:00:14.111 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0017204550000000001 2021-02-10T05:00:14.111 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.111 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:00:14.111 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.111 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:00:14.111 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:00:14.111 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T05:00:14.112 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:13 smithi164 conmon[29288]: cluster 2021-02-10T05:00:12.579726+0000 mgr.y (mgr.14140) 1555 : cluster [DBG] pgmap v2525: 172 pgs: 7 active+remapped+backfill_toofull, 165 active+clean; 25 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 120/1077 objects misplaced (11.142%) 2021-02-10T05:00:14.112 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:13 smithi164 conmon[29288]: audit 2021-02-10T05:00:12.905780+0000 mon.a (mon.0) 3086 : audit [DBG] from='client.? 172.21.15.85:0/814831805' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-02-10T05:00:14.112 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:13 smithi164 conmon[29288]: audit 2021-02-10T05:00:13.414334+0000 mon.c (mon.1) 172 : audit [INF] from='client.? 172.21.15.85:0/323771513' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.5", "id": [1, 4, 6, 3, 7, 5]}]: dispatch 2021-02-10T05:00:14.112 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:13 smithi164 conmon[29288]: audit 2021-02-10T05:00:13.414713+0000 mon.a (mon.0) 3087 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.5", "id": [1, 4, 6, 3, 7, 5]}]: dispatch 2021-02-10T05:00:14.112 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:13 smithi164 conmon[29288]: audit 2021-02-10T05:00:13.650690+0000 mon.a (mon.0) 3088 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.5", "id": [1, 4, 6, 3, 7, 5]}]': finished 2021-02-10T05:00:14.112 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:13 smithi164 conmon[29288]: cluster 2021-02-10T05:00:13.650759+0000 mon.a (mon.0) 3089 : cluster [DBG] osdmap e1379: 8 total, 8 up, 7 in 2021-02-10T05:00:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:14 smithi164 conmon[29288]: cluster 2021-02-10T05:00:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:14 smithi164 conmon[29288]: 2021-02-10T05 2021-02-10T05:00:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:14 smithi164 conmon[29288]: :00:13.455458+0000 osd.5 (osd.5) 207 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:00:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:14 smithi164 conmon[29288]: cluster 2021- 2021-02-10T05:00:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:14 smithi164 conmon[29288]: 02-10T05:00:13.456862+0000 osd.5 ( 2021-02-10T05:00:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:14 smithi164 conmon[29288]: osd.5) 208 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:00:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:14 smithi164 conmon[29288]: cluster 2021- 2021-02-10T05:00:14.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:14 smithi164 conmon[29288]: 02-10T05:00:14.659213+0000 mon.a ( 2021-02-10T05:00:14.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:14 smithi164 conmon[29288]: mon.0) 3090 : cluster [DBG] osdmap e1380: 8 total, 8 up, 7 in 2021-02-10T05:00:15.024 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:14 smithi085 conmon[35961]: cluster 2021-02-10T05:00:13.455458+0000 osd.5 (osd. 2021-02-10T05:00:15.024 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:14 smithi085 conmon[35961]: 5) 207 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:00:15.024 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:14 smithi085 conmon[35961]: cluster 2021-02-10T05:00:13.456862+0000 2021-02-10T05:00:15.024 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:14 smithi085 conmon[35961]: osd.5 (osd.5) 208 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:00:15.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:14 smithi085 conmon[35961]: cluster 2021-02-10T05:00:15.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:14 smithi085 conmon[35961]: 2021-02-10T05:00:14.659213+0000 mon.a (mon.0) 3090 : cluster 2021-02-10T05:00:15.025 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:14 smithi085 conmon[35961]: [DBG] osdmap e1380: 8 total, 8 up, 7 in 2021-02-10T05:00:15.025 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:14 smithi085 conmon[29319]: cluster 2021-02-10T05:00:13.455458+0000 2021-02-10T05:00:15.025 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:14 smithi085 conmon[29319]: osd.5 (osd.5) 207 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:00:15.025 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:14 smithi085 conmon[29319]: cluster 2021 2021-02-10T05:00:15.025 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:14 smithi085 conmon[29319]: -02-10T05:00:13.456862+0000 osd.5 (osd.5) 2021-02-10T05:00:15.025 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:14 smithi085 conmon[29319]: 208 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:00:15.025 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:14 smithi085 conmon[29319]: cluster 2021-02-10T05:00:15.025 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:14 smithi085 conmon[29319]: 2021-02-10T05:00:14.659213+0000 2021-02-10T05:00:15.025 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:14 smithi085 conmon[29319]: mon.a (mon.0) 3090 : cluster [DBG] 2021-02-10T05:00:15.025 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:14 smithi085 conmon[29319]: osdmap e1380: 8 total, 8 up, 7 in 2021-02-10T05:00:15.773 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is set 2021-02-10T05:00:16.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:15 smithi164 conmon[29288]: cluster 2021-02-10T05:00:14.580092 2021-02-10T05:00:16.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:15 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1556 : cluster [DBG] pgmap v2527: 172 pgs: 7 active+remapped+backfill_toofull, 165 active+clean; 25 KiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 120/1077 objects misplaced (11.142%) 2021-02-10T05:00:16.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:15 smithi164 conmon[29288]: audit 2021-02-10T05:00:15.564123+0000 2021-02-10T05:00:16.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:15 smithi164 conmon[29288]: mon.c (mon.1) 173 : audit [INF] from='client.? 172.21.15.85:0/816772170' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T05:00:16.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:15 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T05:00:16.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:15 smithi164 conmon[29288]: T05:00:15.564405+0000 mon.a ( 2021-02-10T05:00:16.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:15 smithi164 conmon[29288]: mon.0) 3091 : audit [INF] 2021-02-10T05:00:16.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:15 smithi164 conmon[29288]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T05:00:16.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:15 smithi085 conmon[29319]: cluster 2021-02-10T05:00:14.580092+0000 mgr.y (mgr.14140 2021-02-10T05:00:16.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:15 smithi085 conmon[29319]: ) 1556 : cluster [DBG] pgmap v2527: 172 pgs: 7 active+remapped+backfill_toofull, 165 active+clean; 25 KiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 120/1077 objects misplaced (11.142%) 2021-02-10T05:00:16.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:15 smithi085 conmon[29319]: audit 2021-02-10T05:00:15.564123+0000 mon.c (mon.1) 173 : audit [INF] from='client.? 172.21.15.85:0/816772170' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T05:00:16.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:15 smithi085 conmon[29319]: audit 2021-02-10T05:00:15.564405+0000 mon.a (mon.0) 3091 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T05:00:16.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:15 smithi085 conmon[35961]: cluster 2021-02-10T05:00: 2021-02-10T05:00:16.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:15 smithi085 conmon[35961]: 14.580092+0000 mgr.y (mgr.14140) 1556 : cluster [DBG] pgmap v2527: 172 pgs: 7 active+remapped+backfill_toofull, 165 active+clean; 25 KiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 120/1077 objects misplaced (11.142%) 2021-02-10T05:00:16.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:15 smithi085 conmon[35961]: audit 2021-02-10T05:00:15.564123+0000 mon.c (mon.1) 173 : audit [INF] from='client.? 172.21.15.85:0/816772170' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T05:00:16.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:15 smithi085 conmon[35961]: audit 2021-02-10T05:00:15.564405+0000 mon.a (mon.0) 3091 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-02-10T05:00:16.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:16 smithi085 conmon[29319]: cluster 2021-02-10T05:00:15.760103+0000 2021-02-10T05:00:16.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:16 smithi085 conmon[29319]: mon.a (mon.0) 3092 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T05:00:16.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:16 smithi085 conmon[29319]: audit 2021-02-10T05:00:15.768114+0000 mon.a (mon.0) 3093 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T05:00:16.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:16 smithi085 conmon[29319]: cluster 2021-02-10T05:00:15.768183+0000 mon.a (mon.0) 3094 : cluster [DBG] osdmap e1381: 8 total, 8 up, 7 in 2021-02-10T05:00:16.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:16 smithi085 conmon[35961]: cluster 2021-02-10T05:00:15.760103+0000 mon.a (mon.0) 2021-02-10T05:00:16.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:16 smithi085 conmon[35961]: 3092 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T05:00:16.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:16 smithi085 conmon[35961]: audit 2021-02-10T05:00:15 2021-02-10T05:00:16.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:16 smithi085 conmon[35961]: .768114+0000 mon.a (mon.0) 3093 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T05:00:16.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:16 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T05:00:16.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:16 smithi085 conmon[35961]: -10T05:00:15.768183+0000 mon.a ( 2021-02-10T05:00:16.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:16 smithi085 conmon[35961]: mon.0) 3094 : cluster [DBG] osdmap e1381: 8 total, 8 up, 7 in 2021-02-10T05:00:17.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:16 smithi164 conmon[29288]: cluster 2021-02-10T05:00:15.760103+0000 mon.a ( 2021-02-10T05:00:17.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:16 smithi164 conmon[29288]: mon.0) 3092 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T05:00:17.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:16 smithi164 conmon[29288]: audit 2021-02-10T05:00:15.768114+0000 mon.a (mon.0) 3093 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-02-10T05:00:17.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:16 smithi164 conmon[29288]: cluster 2021-02-10T05:00:15.768183+0000 mon.a (mon. 2021-02-10T05:00:17.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:16 smithi164 conmon[29288]: 0) 3094 : cluster [DBG] osdmap e1381: 8 total, 8 up, 7 in 2021-02-10T05:00:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:17 smithi164 conmon[29288]: cluster 2021-02-10T05:00:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:17 smithi164 conmon[29288]: 2021-02-10T05: 2021-02-10T05:00:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:17 smithi164 conmon[29288]: 00:16.580664+0000 mgr.y (mgr.14140) 1557 : cluster [DBG] pgmap v2530: 172 pgs: 7 active+remapped+backfill_toofull, 165 active+clean; 25 KiB data, 794 MiB used, 625 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 120/1077 objects misplaced (11.142%) 2021-02-10T05:00:18.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:17 smithi085 conmon[29319]: cluster 2021-02-10T05:00:16.580664+0000 mgr.y (mgr. 2021-02-10T05:00:18.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:17 smithi085 conmon[29319]: 14140) 1557 : cluster [DBG] pgmap v2530: 172 pgs: 7 active+remapped+backfill_toofull, 165 active+clean; 25 KiB data, 794 MiB used, 625 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 120/1077 objects misplaced (11.142%) 2021-02-10T05:00:18.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:17 smithi085 conmon[35961]: cluster 2021-02-10T05:00:16.580664 2021-02-10T05:00:18.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:17 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1557 : cluster [DBG] pgmap v2530: 172 pgs: 7 active+remapped+backfill_toofull, 165 active+clean; 25 KiB data, 794 MiB used, 625 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 120/1077 objects misplaced (11.142%) 2021-02-10T05:00:18.702 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset noscrub 2021-02-10T05:00:18.920 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:00:19.127 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:18 smithi085 conmon[29319]: cluster 2021-02-10T05:00:17.775119+0000 mon.a (mon. 2021-02-10T05:00:19.127 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:18 smithi085 conmon[29319]: 0) 3095 : cluster [DBG] osdmap e1382: 8 total, 8 up, 7 in 2021-02-10T05:00:19.127 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:18 smithi085 conmon[29319]: cluster 2021-02-10T05:00:18.651794+0000 mon.a (mon.0) 3096 : cluster [DBG] osdmap e1383: 8 total, 8 up, 7 in 2021-02-10T05:00:19.127 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:18 smithi085 conmon[35961]: cluster 2021-02-10T05:00:17. 2021-02-10T05:00:19.127 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:18 smithi085 conmon[35961]: 775119+0000 mon.a (mon.0) 3095 : cluster [DBG] osdmap e1382: 8 total, 8 up, 7 in 2021-02-10T05:00:19.127 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:18 smithi085 conmon[35961]: cluster 2021-02-10T05:00: 2021-02-10T05:00:19.127 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:18 smithi085 conmon[35961]: 18.651794+0000 mon.a (mon.0) 3096 : cluster [DBG] osdmap e1383: 8 total, 8 up, 7 in 2021-02-10T05:00:19.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:18 smithi164 conmon[29288]: cluster 2021-02-10T05:00:17.775119+0000 mon.a 2021-02-10T05:00:19.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:18 smithi164 conmon[29288]: (mon.0) 3095 : cluster [DBG] osdmap e1382: 8 total, 8 up, 7 in 2021-02-10T05:00:19.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:18 smithi164 conmon[29288]: cluster 2021-02-10T05:00:18.651794+0000 mon.a (mon.0) 3096 : cluster [DBG] 2021-02-10T05:00:19.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:18 smithi164 conmon[29288]: osdmap e1383: 8 total, 8 up, 7 in 2021-02-10T05:00:19.848 INFO:tasks.thrashosds.thrasher:inning osd 2021-02-10T05:00:19.848 INFO:tasks.thrashosds.thrasher:Adding osd 6 2021-02-10T05:00:19.848 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd in 6 2021-02-10T05:00:19.893 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 05:00:19 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T05:00:19.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:19 smithi164 conmon[29288]: cluster 2021-02-10T05:00:18.581249+0000 2021-02-10T05:00:19.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:19 smithi164 conmon[29288]: mgr.y (mgr.14140) 1558 : cluster [DBG] pgmap v2532: 172 pgs: 1 active+clean+remapped, 6 active+remapped+backfill_toofull, 165 active+clean; 25 KiB data, 323 MiB used, 626 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 94/1077 objects misplaced (8.728%); 180 B/s, 3 objects/s recovering 2021-02-10T05:00:19.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:19 smithi164 conmon[29288]: cluster 2021-02-10T05:00:18.773148+0000 mon.a (mon.0) 2021-02-10T05:00:19.894 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:19 smithi164 conmon[29288]: 3097 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-02-10T05:00:20.056 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:00:20.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:19 smithi085 conmon[29319]: cluster 2021-02-10T05:00:18.581249+0000 mgr.y (mgr.14140) 1558 : cluster [DBG] pgmap v2532: 172 pgs: 1 active+clean+remapped, 6 active+remapped+backfill_toofull, 165 active+clean; 25 KiB data, 323 MiB used, 626 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 94/1077 objects misplaced (8.728%); 180 B/s, 3 objects/s recovering 2021-02-10T05:00:20.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:19 smithi085 conmon[29319]: cluster 2021-02-10T05:00:18.773148+0000 mon.a (mon.0) 3097 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-02-10T05:00:20.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:19 smithi085 conmon[35961]: cluster 2021-02-10T05:00:18.581249+0000 mgr.y (mgr.14140) 1558 : cluster [DBG] pgmap v2532: 172 pgs: 1 active+clean+remapped, 6 active+remapped+backfill_toofull, 165 active+clean; 25 KiB data, 323 MiB used, 626 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 94/1077 objects misplaced (8.728%); 180 B/s, 3 objects/s recovering 2021-02-10T05:00:20.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:19 smithi085 conmon[35961]: cluster 2021-02-10T05:00:18.773148+0000 mon.a (mon.0) 3097 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-02-10T05:00:20.919 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:20 smithi085 conmon[29319]: audit 2021-02-10T05:00:19.604998+0000 2021-02-10T05:00:20.919 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:20 smithi085 conmon[29319]: mgr.y (mgr.14140) 1559 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:00:20.919 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:20 smithi085 conmon[35961]: audit 2021-02-10T05:00:19.604998 2021-02-10T05:00:20.920 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:20 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1559 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:00:21.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:20 smithi164 conmon[29288]: audit 2021-02-10T05:00:19.604998+0000 mgr.y 2021-02-10T05:00:21.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:20 smithi164 conmon[29288]: (mgr.14140) 1559 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:00:21.809 INFO:teuthology.orchestra.run.smithi085.stderr:noscrub is unset 2021-02-10T05:00:22.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:21 smithi164 conmon[29288]: cluster 2021-02-10T05:00:20.581797+0000 2021-02-10T05:00:22.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:21 smithi164 conmon[29288]: mgr.y (mgr.14140) 1560 : cluster [DBG] pgmap v2534: 172 pgs: 1 active+clean+remapped, 5 active+remapped+backfill_toofull, 166 active+clean; 25 KiB data, 323 MiB used, 626 GiB / 626 GiB avail; 864 B/s rd, 0 op/s; 70/1077 objects misplaced (6.500%); 353 B/s, 4 objects/s recovering 2021-02-10T05:00:22.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:21 smithi164 conmon[29288]: audit 2021- 2021-02-10T05:00:22.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:21 smithi164 conmon[29288]: 02-10T05:00:21.317958+0000 mon.c (mon.1) 174 : audit [INF] from='client.? 172.21.15.85:0/2752336648' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T05:00:22.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:21 smithi164 conmon[29288]: 2021-02-10T05:00:22.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:21 smithi164 conmon[29288]: audit 2021- 2021-02-10T05:00:22.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:21 smithi164 conmon[29288]: 02-10T05:00:21 2021-02-10T05:00:22.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:21 smithi164 conmon[29288]: .318275+0000 mon.a (mon.0) 3098 : 2021-02-10T05:00:22.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:21 smithi164 conmon[29288]: audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T05:00:22.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:21 smithi085 conmon[29319]: cluster 2021-02-10T05:00:22.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:21 smithi085 conmon[29319]: 2021-02-10T05:00:20. 2021-02-10T05:00:22.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:21 smithi085 conmon[29319]: 581797+0000 mgr.y (mgr.14140) 1560 : cluster [DBG] pgmap v2534: 172 pgs: 1 active+clean+remapped, 5 active+remapped+backfill_toofull, 166 active+clean; 25 KiB data, 323 MiB used, 626 GiB / 626 GiB avail; 864 B/s rd, 0 op/s; 70/1077 objects misplaced (6.500%); 353 B/s, 4 objects/s recovering 2021-02-10T05:00:22.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:21 smithi085 conmon[29319]: audit 2021-02-10T05:00:22.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:21 smithi085 conmon[29319]: 2021-02-10T05:00:21.317958+0000 mon.c (mon.1) 174 : audit [INF] from='client.? 172.21.15.85:0/2752336648' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T05:00:22.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:21 smithi085 conmon[29319]: audit 2021-02-10 2021-02-10T05:00:22.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:21 smithi085 conmon[29319]: T05:00:21.318275+0000 mon.a (mon.0) 3098 : audit 2021-02-10T05:00:22.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:21 smithi085 conmon[29319]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T05:00:22.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:21 smithi085 conmon[35961]: cluster 2021-02-10T05:00:20.581797+0000 2021-02-10T05:00:22.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:21 smithi085 conmon[35961]: mgr.y (mgr.14140) 1560 : cluster [DBG] pgmap v2534: 172 pgs: 1 active+clean+remapped, 5 active+remapped+backfill_toofull, 166 active+clean; 25 KiB data, 323 MiB used, 626 GiB / 626 GiB avail; 864 B/s rd, 0 op/s; 70/1077 objects misplaced (6.500%); 353 B/s, 4 objects/s recovering 2021-02-10T05:00:22.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:21 smithi085 conmon[35961]: audit 2021-02-10T05:00:21.317958+0000 mon.c (mon.1) 174 : audit [INF] 2021-02-10T05:00:22.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:21 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/2752336648' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T05:00:22.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:21 smithi085 conmon[35961]: audit 2021-02-10T05:00:21.318275+0000 2021-02-10T05:00:22.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:21 smithi085 conmon[35961]: mon.a (mon.0) 3098 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-02-10T05:00:22.685 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd unset nodeep-scrub 2021-02-10T05:00:22.904 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:00:22.933 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.7 dump_ops_in_flight 2021-02-10T05:00:22.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:22 smithi085 conmon[35961]: cluster 2021-02-10T05:00:21.358000 2021-02-10T05:00:22.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:22 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 2049 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:00:22.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:22 smithi085 conmon[35961]: cluster 2021-02-10T05:00:21.358625+0000 osd.2 (osd.2) 2050 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:00:22.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:22 smithi085 conmon[35961]: cluster 2021-02-10T05:00:21.792653+0000 mon.a (mon.0) 3099 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T05:00:22.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:22 smithi085 conmon[35961]: audit 2021-02-10T05:00:21.804307+0000 mon.a (mon.0) 3100 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T05:00:22.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:22 smithi085 conmon[35961]: cluster 2021-02-10T05:00:21.804366+0000 mon.a (mon.0) 3101 : cluster [DBG] osdmap e1384: 8 total, 8 up, 7 in 2021-02-10T05:00:22.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:22 smithi085 conmon[35961]: cluster 2021-02-10T05:00:22.170870+0000 osd.5 (osd.5) 209 : cluster [DBG] 1.1 scrub ok 2021-02-10T05:00:22.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:22 smithi085 conmon[29319]: cluster 2021-02-10T05:00:21. 2021-02-10T05:00:22.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:22 smithi085 conmon[29319]: 358000+0000 osd.2 (osd.2) 2049 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:00:22.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:22 smithi085 conmon[29319]: cluster 2021-02-10T05:00:21.358625+0000 osd.2 (osd.2) 2050 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:00:22.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:22 smithi085 conmon[29319]: cluster 2021-02-10T05:00:21.792653+0000 mon.a (mon.0) 3099 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T05:00:22.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:22 smithi085 conmon[29319]: audit 2021-02-10T05:00:21.804307+0000 mon.a (mon.0) 3100 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T05:00:22.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:22 smithi085 conmon[29319]: cluster 2021-02-10T05:00:21.804366+0000 mon.a (mon.0) 3101 : cluster [DBG] osdmap e1384: 8 total, 8 up, 7 in 2021-02-10T05:00:22.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:22 smithi085 conmon[29319]: cluster 2021-02-10T05:00:22.170870+0000 osd.5 (osd.5) 209 : cluster [DBG] 1.1 scrub ok 2021-02-10T05:00:23.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:22 smithi164 conmon[29288]: cluster 2021-02-10T05:00:21.358000+0000 osd.2 ( 2021-02-10T05:00:23.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:22 smithi164 conmon[29288]: osd.2) 2049 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:00:23.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:22 smithi164 conmon[29288]: cluster 2021-02-10T05:00:21.358625+0000 osd.2 ( 2021-02-10T05:00:23.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:22 smithi164 conmon[29288]: osd.2) 2050 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:00:23.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:22 smithi164 conmon[29288]: cluster 2021-02-10T05:00 2021-02-10T05:00:23.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:22 smithi164 conmon[29288]: :21.792653+0000 mon.a (mon.0) 3099 2021-02-10T05:00:23.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:22 smithi164 conmon[29288]: : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-02-10T05:00:23.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:22 smithi164 conmon[29288]: audit 2021-02-10T05:00:23.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:22 smithi164 conmon[29288]: 2021-02-10T05:00:21.804307+0000 2021-02-10T05:00:23.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:22 smithi164 conmon[29288]: mon.a (mon.0) 3100 : audit [INF] 2021-02-10T05:00:23.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:22 smithi164 conmon[29288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-02-10T05:00:23.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:22 smithi164 conmon[29288]: cluster 2021-02-10T05:00 2021-02-10T05:00:23.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:22 smithi164 conmon[29288]: :21.804366+0000 mon.a (mon.0) 2021-02-10T05:00:23.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:22 smithi164 conmon[29288]: 3101 : cluster [DBG] osdmap e1384: 8 total, 8 up, 7 in 2021-02-10T05:00:23.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:22 smithi164 conmon[29288]: cluster 2021- 2021-02-10T05:00:23.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:22 smithi164 conmon[29288]: 02-10T05:00:22.170870+0000 osd.5 2021-02-10T05:00:23.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:22 smithi164 conmon[29288]: (osd.5) 209 : cluster [DBG] 1.1 scrub ok 2021-02-10T05:00:23.312 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T05:00:23.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[29319]: debug 2021-02-10T05:00:23.235+0000 7f9e105d6700 -1 mon.a@0(leader).osd e1385 definitely_dead 0 2021-02-10T05:00:23.833 INFO:teuthology.orchestra.run.smithi085.stderr:marked in osd.6. 2021-02-10T05:00:24.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:23 smithi164 conmon[29288]: cluster 2021-02-10T05:00:24.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:23 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T05:00:24.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:23 smithi164 conmon[29288]: 05:00:22.582509+0000 mgr.y (mgr.14140) 1561 : cluster [DBG] pgmap v2536: 172 pgs: 5 active+remapped+backfill_toofull, 167 active+clean; 25 KiB data, 327 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 70/1077 objects misplaced (6.500%); 349 B/s, 4 objects/s recovering 2021-02-10T05:00:24.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:23 smithi164 conmon[29288]: cluster 2021-02-10T05:00:24.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:23 smithi164 conmon[29288]: 2021-02 2021-02-10T05:00:24.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:23 smithi164 conmon[29288]: -10T05:00:22.600685+0000 osd.7 (osd.7) 2021-02-10T05:00:24.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:23 smithi164 conmon[29288]: 106 : cluster [DBG] 6.1c scrub ok 2021-02-10T05:00:24.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:23 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T05:00:24.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:23 smithi164 conmon[29288]: -10T05:00:22.812648+0000 mon.a (mon.0) 3102 2021-02-10T05:00:24.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:23 smithi164 conmon[29288]: : cluster [DBG] osdmap e1385: 8 total, 8 up, 7 in 2021-02-10T05:00:24.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:23 smithi164 conmon[29288]: audit 2021-02-10T05:00:24.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:23 smithi164 conmon[29288]: 2021-02-10T05:00:23.235904+0000 2021-02-10T05:00:24.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:23 smithi164 conmon[29288]: mon.c (mon.1) 175 : audit [DBG] 2021-02-10T05:00:24.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:23 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/2664795903' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2021-02-10T05:00:24.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:23 smithi164 conmon[29288]: audit 2021-02-10T05: 2021-02-10T05:00:24.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:23 smithi164 conmon[29288]: 00:23.236246+0000 mon.a (mon.0) 3103 : audit [DBG] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2021-02-10T05:00:24.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:23 smithi164 conmon[29288]: cluster 2021-02-10T05:00:24.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:23 smithi164 conmon[29288]: 2021-02-10T05:00:23.307925+0000 2021-02-10T05:00:24.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:23 smithi164 conmon[29288]: osd.4 (osd.4) 2677 : cluster [DBG] 2021-02-10T05:00:24.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:23 smithi164 conmon[29288]: 1.5 scrub ok 2021-02-10T05:00:24.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[29319]: cluster 2021-02-10T05:00:22.582509+0000 mgr.y (mgr.14140 2021-02-10T05:00:24.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[29319]: ) 1561 : cluster [DBG] pgmap v2536: 172 pgs: 5 active+remapped+backfill_toofull, 167 active+clean; 25 KiB data, 327 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 70/1077 objects misplaced (6.500%); 349 B/s, 4 objects/s recovering 2021-02-10T05:00:24.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[29319]: cluster 2021-02-10T05:00:22.600685+0000 osd.7 (osd.7) 106 : cluster [DBG] 6.1c scrub ok 2021-02-10T05:00:24.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[29319]: cluster 2021-02-10T05:00:22.812648+0000 mon.a 2021-02-10T05:00:24.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[29319]: (mon.0) 3102 : cluster 2021-02-10T05:00:24.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[29319]: [DBG] osdmap e1385: 8 total, 8 up, 7 in 2021-02-10T05:00:24.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[29319]: audit 2021-02-10T05:00:23.235904+0000 mon.c (mon.1 2021-02-10T05:00:24.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[29319]: ) 175 : audit [DBG] from='client.? 172.21.15.85:0/2664795903' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2021-02-10T05:00:24.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[29319]: audit 2021-02-10T05:00:23. 2021-02-10T05:00:24.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[29319]: 236246+0000 mon.a (mon.0) 3103 : audit [DBG] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2021-02-10T05:00:24.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[29319]: cluster 2021-02-10T05:00:23.307925 2021-02-10T05:00:24.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2677 : cluster [DBG] 2021-02-10T05:00:24.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[29319]: 1.5 scrub ok 2021-02-10T05:00:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[35961]: cluster 2021-02-10T05:00:22.582509+0000 2021-02-10T05:00:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[35961]: mgr.y (mgr.14140) 1561 : cluster [DBG] pgmap v2536: 172 pgs: 5 active+remapped+backfill_toofull, 167 active+clean; 25 KiB data, 327 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 70/1077 objects misplaced (6.500%); 349 B/s, 4 objects/s recovering 2021-02-10T05:00:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[35961]: cluster 2021-02-10T05:00:22.600685+0000 osd.7 (osd.7) 106 : 2021-02-10T05:00:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[35961]: cluster [DBG] 6.1c scrub ok 2021-02-10T05:00:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T05:00:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[35961]: 05:00:22.812648+0000 mon.a (mon.0) 3102 : cluster 2021-02-10T05:00:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[35961]: [DBG] osdmap e1385: 8 total, 8 up, 7 in 2021-02-10T05:00:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[35961]: audit 2021 2021-02-10T05:00:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[35961]: -02-10T05:00:23.235904+0000 2021-02-10T05:00:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[35961]: mon.c (mon.1) 175 : audit [DBG] from='client.? 172.21.15.85:0/2664795903' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2021-02-10T05:00:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[35961]: 2021-02-10T05:00:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[35961]: audit 2021-02-10T05:00:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[35961]: 2021-02-10T05:00:23.236246+0000 mon.a (mon 2021-02-10T05:00:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[35961]: .0) 3103 : audit [DBG] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2021-02-10T05:00:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T05:00:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[35961]: -10T05:00:23.307925+0000 osd.4 2021-02-10T05:00:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:23 smithi085 conmon[35961]: (osd.4) 2677 : cluster [DBG] 1.5 scrub ok 2021-02-10T05:00:24.875 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:24 smithi085 conmon[29319]: cluster 2021-02-10T05:00:23.773916+0000 osd.1 (osd.1 2021-02-10T05:00:24.876 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:24 smithi085 conmon[29319]: ) 531 : cluster [DBG] 4.f scrub ok 2021-02-10T05:00:24.876 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:24 smithi085 conmon[29319]: audit 2021-02-10T05:00:23.813408+0000 mon.a (mon.0) 3104 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2021-02-10T05:00:24.876 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:24 smithi085 conmon[29319]: cluster 2021-02-10T05:00:23.813470+0000 mon.a (mon.0) 3105 : cluster 2021-02-10T05:00:24.876 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:24 smithi085 conmon[29319]: [DBG] osdmap e1386: 8 total, 8 up, 8 in 2021-02-10T05:00:24.876 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:24 smithi085 conmon[29319]: cluster 2021-02-10T05:00:24 2021-02-10T05:00:24.876 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:24 smithi085 conmon[29319]: .185741+0000 osd.5 (osd.5) 210 : cluster [DBG] 2021-02-10T05:00:24.876 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:24 smithi085 conmon[29319]: 1.8 scrub ok 2021-02-10T05:00:24.876 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:24 smithi085 conmon[29319]: cluster 2021-02-10T05:00:24.876 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:24 smithi085 conmon[29319]: 2021-02-10T05:00:24.815288+0000 mon.a (mon.0 2021-02-10T05:00:24.876 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:24 smithi085 conmon[29319]: ) 3106 : cluster [DBG] osdmap e1387: 8 total, 8 up, 8 in 2021-02-10T05:00:24.876 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:24 smithi085 conmon[35961]: cluster 2021-02-10T05:00:23.773916+0000 osd.1 2021-02-10T05:00:24.877 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:24 smithi085 conmon[35961]: (osd.1) 531 : cluster [DBG] 4.f scrub ok 2021-02-10T05:00:24.877 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:24 smithi085 conmon[35961]: audit 2021-02-10T05:00:23.813408+0000 mon.a (mon.0) 3104 : audit 2021-02-10T05:00:24.877 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:24 smithi085 conmon[35961]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2021-02-10T05:00:24.877 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:24 smithi085 conmon[35961]: cluster 2021-02-10T05:00:23.813470+0000 mon.a (mon.0 2021-02-10T05:00:24.877 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:24 smithi085 conmon[35961]: ) 3105 : cluster [DBG] osdmap e1386: 8 total, 8 up, 8 in 2021-02-10T05:00:24.877 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:24 smithi085 conmon[35961]: cluster 2021-02-10T05:00:24 2021-02-10T05:00:24.877 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:24 smithi085 conmon[35961]: .185741+0000 osd.5 (osd.5) 210 : cluster [DBG] 1.8 scrub ok 2021-02-10T05:00:24.877 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:24 smithi085 conmon[35961]: cluster 2021 2021-02-10T05:00:24.877 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:24 smithi085 conmon[35961]: -02-10T05:00:24.815288+0000 mon.a (mon.0) 3106 : cluster [DBG] osdmap e1387: 8 total, 8 up, 8 in 2021-02-10T05:00:24.900 INFO:tasks.thrashosds.thrasher:Added osd 6 2021-02-10T05:00:24.900 INFO:tasks.thrashosds.thrasher:fixing pg num pool zone.rgw.meta 2021-02-10T05:00:24.901 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pool set zone.rgw.meta pgp_num 42 2021-02-10T05:00:25.133 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:00:25.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:24 smithi164 conmon[29288]: cluster 2021-02-10T05:00:23.773916+0000 osd.1 (osd.1 2021-02-10T05:00:25.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:24 smithi164 conmon[29288]: ) 531 : cluster [DBG] 4.f scrub ok 2021-02-10T05:00:25.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:24 smithi164 conmon[29288]: audit 2021-02-10T05:00:23.813408+0000 mon.a (mon.0) 3104 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2021-02-10T05:00:25.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:24 smithi164 conmon[29288]: cluster 2021-02-10T05:00:23.813470+0000 mon.a (mon.0) 3105 : cluster [DBG] osdmap e1386: 8 total, 8 up, 8 in 2021-02-10T05:00:25.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:24 smithi164 conmon[29288]: cluster 2021-02-10T05:00:24.185741+0000 osd.5 (osd.5) 210 : cluster [DBG] 1.8 scrub ok 2021-02-10T05:00:25.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:24 smithi164 conmon[29288]: cluster 2021-02-10T05:00: 2021-02-10T05:00:25.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:24 smithi164 conmon[29288]: 24.815288+0000 mon.a (mon.0) 3106 : cluster 2021-02-10T05:00:25.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:24 smithi164 conmon[29288]: [DBG] osdmap e1387: 8 total, 8 up, 8 in 2021-02-10T05:00:25.826 INFO:teuthology.orchestra.run.smithi085.stderr:nodeep-scrub is unset 2021-02-10T05:00:26.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:25 smithi164 conmon[29288]: cluster 2021-02-10T05:00:24.583002+0000 mgr.y (mgr 2021-02-10T05:00:26.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:25 smithi164 conmon[29288]: .14140) 1562 : cluster [DBG] pgmap v2539: 172 pgs: 23 peering, 5 active+remapped+backfill_toofull, 144 active+clean; 23 KiB data, 382 MiB used, 715 GiB / 715 GiB avail; 70/1008 objects misplaced (6.944%) 2021-02-10T05:00:26.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:25 smithi164 conmon[29288]: audit 2021-02-10T05:00:25.515774+0000 mon.a (mon.0) 3107 : audit [INF] from='client.? 172.21.15.85:0/1165669597' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T05:00:26.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:25 smithi164 conmon[29288]: cluster 2021-02-10T05:00:25.812619+0000 mon.a (mon.0) 3108 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T05:00:26.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:25 smithi164 conmon[29288]: audit 2021-02-10T05:00:25.819238+0000 mon.a (mon.0) 2021-02-10T05:00:26.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:25 smithi164 conmon[29288]: 3109 : audit [INF] from='client.? 172.21.15.85:0/1165669597' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T05:00:26.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:25 smithi164 conmon[29288]: cluster 2021-02-10T05:00: 2021-02-10T05:00:26.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:25 smithi164 conmon[29288]: 25.819482+0000 mon.a (mon.0) 3110 : cluster 2021-02-10T05:00:26.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:25 smithi164 conmon[29288]: [DBG] osdmap e1388: 8 total, 8 up, 8 in 2021-02-10T05:00:26.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:25 smithi085 conmon[29319]: cluster 2021-02-10T05:00:24.583002+0000 mgr.y (mgr.14140) 1562 : cluster [DBG] pgmap v2539: 172 pgs: 23 peering, 5 active+remapped+backfill_toofull, 144 active+clean; 23 KiB data, 382 MiB used, 715 GiB / 715 GiB avail; 70/1008 objects misplaced (6.944%) 2021-02-10T05:00:26.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:25 smithi085 conmon[29319]: audit 2021-02-10T05:00:25.515774+0000 mon.a (mon.0) 3107 : audit [INF] from='client.? 172.21.15.85:0/1165669597' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T05:00:26.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:25 smithi085 conmon[29319]: cluster 2021-02-10T05:00:25.812619+0000 mon.a (mon.0) 3108 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T05:00:26.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:25 smithi085 conmon[29319]: audit 2021-02-10T05:00:25.819238+0000 mon.a (mon.0) 3109 : audit [INF] from='client.? 172.21.15.85:0/1165669597' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T05:00:26.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:25 smithi085 conmon[29319]: cluster 2021-02-10T05:00:25.819482+0000 mon.a (mon.0) 3110 : cluster [DBG] osdmap e1388: 8 total, 8 up, 8 in 2021-02-10T05:00:26.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:25 smithi085 conmon[35961]: cluster 2021-02-10T05:00:24.583002+0000 mgr.y ( 2021-02-10T05:00:26.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:25 smithi085 conmon[35961]: mgr.14140) 1562 : cluster [DBG] pgmap v2539: 172 pgs: 23 peering, 5 active+remapped+backfill_toofull, 144 active+clean; 23 KiB data, 382 MiB used, 715 GiB / 715 GiB avail; 70/1008 objects misplaced (6.944%) 2021-02-10T05:00:26.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:25 smithi085 conmon[35961]: audit 2021-02-10T05:00:25.515774 2021-02-10T05:00:26.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:25 smithi085 conmon[35961]: +0000 2021-02-10T05:00:26.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:25 smithi085 conmon[35961]: 2021-02-10T05:00:26.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:25 smithi085 conmon[35961]: mon.a 2021-02-10T05:00:26.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:25 smithi085 conmon[35961]: ( 2021-02-10T05:00:26.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:25 smithi085 conmon[35961]: mon.0) 3107 : 2021-02-10T05:00:26.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:25 smithi085 conmon[35961]: audit [INF] from='client.? 172.21.15.85:0/1165669597' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-02-10T05:00:26.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:25 smithi085 conmon[35961]: cluster 2021-02-10T05:00:26.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:25 smithi085 conmon[35961]: 2021-02-10T05: 2021-02-10T05:00:26.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:25 smithi085 conmon[35961]: 00:25.812619+0000 mon.a (mon.0 2021-02-10T05:00:26.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:25 smithi085 conmon[35961]: ) 3108 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-02-10T05:00:26.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:25 smithi085 conmon[35961]: audit 2021-02 2021-02-10T05:00:26.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:25 smithi085 conmon[35961]: -10T05:00:25.819238+0000 mon.a ( 2021-02-10T05:00:26.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:25 smithi085 conmon[35961]: mon.0) 3109 : audit [INF] from='client.? 172.21.15.85:0/1165669597' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-02-10T05:00:26.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:25 smithi085 conmon[35961]: cluster 2021-02-10T05:00:26.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:25 smithi085 conmon[35961]: 2021-02-10T05:00:25.819482+0000 mon.a 2021-02-10T05:00:26.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:25 smithi085 conmon[35961]: (mon.0) 3110 : cluster [DBG] osdmap e1388: 8 total, 8 up, 8 in 2021-02-10T05:00:26.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:25 smithi085 conmon[35961]: 2021-02-10T05:00:26.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:26 smithi085 conmon[29319]: cluster 2021-02-10T05:00:26.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:26 smithi085 conmon[29319]: 2021-02-10T05:00 2021-02-10T05:00:26.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:26 smithi085 conmon[29319]: :25.818717+0000 osd.1 (osd.1) 532 : cluster [DBG] 4.a starting backfill to osd.6 from (0'0,0'0] MAX to 1261'69 2021-02-10T05:00:26.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:26 smithi085 conmon[29319]: cluster 2021-02-10T05:00:25 2021-02-10T05:00:26.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:26 smithi085 conmon[29319]: .819474+0000 osd.5 (osd.5) 211 : cluster [DBG] 2021-02-10T05:00:26.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:26 smithi085 conmon[29319]: 4.8 starting backfill to osd.6 from (0'0,0'0] MAX to 1261'84 2021-02-10T05:00:26.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:26 smithi085 conmon[29319]: cluster 2021-02-10T05:00: 2021-02-10T05:00:26.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:26 smithi085 conmon[29319]: 25.821663+0000 osd.0 (osd.0) 13 : 2021-02-10T05:00:26.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:26 smithi085 conmon[29319]: cluster [DBG] 4.1f starting backfill to osd.6 from (0'0,0'0] MAX to 1261'71 2021-02-10T05:00:26.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:26 smithi085 conmon[35961]: cluster 2021-02-10T05:00:26.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:26 smithi085 conmon[35961]: 2021-02-10T05:00:26.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:26 smithi085 conmon[35961]: 2021 2021-02-10T05:00:26.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:26 smithi085 conmon[35961]: -02-10T05:00:25 2021-02-10T05:00:26.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:26 smithi085 conmon[35961]: .818717+0000 osd.1 (osd.1) 532 : cluster [DBG] 4.a starting backfill to osd.6 from (0'0,0'0] MAX to 1261'69 2021-02-10T05:00:26.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:26 smithi085 conmon[35961]: cluster 2021 2021-02-10T05:00:26.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:26 smithi085 conmon[35961]: -02-10 2021-02-10T05:00:26.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:26 smithi085 conmon[35961]: T05:00:25.819474+0000 2021-02-10T05:00:26.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:26 smithi085 conmon[35961]: osd.5 (osd.5) 211 : cluster [DBG] 4.8 starting backfill to osd.6 from (0'0,0'0] MAX to 1261'84 2021-02-10T05:00:26.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:26 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T05:00:26.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:26 smithi085 conmon[35961]: -10T05:00:25.821663+0000 osd.0 (osd. 2021-02-10T05:00:26.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:26 smithi085 conmon[35961]: 0) 13 : cluster [DBG] 4.1f starting backfill to osd.6 from (0'0,0'0] MAX to 1261'71 2021-02-10T05:00:27.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:26 smithi164 conmon[29288]: cluster 2021-02-10T05:00:27.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:26 smithi164 conmon[29288]: 2021-02-10T05:00:27.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:26 smithi164 conmon[29288]: 2021 2021-02-10T05:00:27.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:26 smithi164 conmon[29288]: - 2021-02-10T05:00:27.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:26 smithi164 conmon[29288]: 02 2021-02-10T05:00:27.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:26 smithi164 conmon[29288]: - 2021-02-10T05:00:27.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:26 smithi164 conmon[29288]: 10 2021-02-10T05:00:27.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:26 smithi164 conmon[29288]: T 2021-02-10T05:00:27.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:26 smithi164 conmon[29288]: 05 2021-02-10T05:00:27.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:26 smithi164 conmon[29288]: : 2021-02-10T05:00:27.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:26 smithi164 conmon[29288]: 00:25.818717+0000 osd.1 (osd 2021-02-10T05:00:27.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:26 smithi164 conmon[29288]: .1) 532 : cluster [DBG] 4.a starting backfill to osd.6 from (0'0,0'0] MAX to 1261'69 2021-02-10T05:00:27.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:26 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T05:00:27.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:26 smithi164 conmon[29288]: 10T05:00:25.819474+0000 osd.5 ( 2021-02-10T05:00:27.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:26 smithi164 conmon[29288]: osd.5) 211 : cluster [DBG] 4.8 starting backfill to osd.6 from (0'0,0'0] MAX to 1261'84 2021-02-10T05:00:27.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:26 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T05:00:27.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:26 smithi164 conmon[29288]: 10T05:00:25.821663+0000 osd.0 (osd. 2021-02-10T05:00:27.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:26 smithi164 conmon[29288]: 0) 13 : cluster [DBG] 4.1f starting backfill to osd.6 from (0'0,0'0] MAX to 1261'71 2021-02-10T05:00:27.855 INFO:teuthology.orchestra.run.smithi085.stderr:set pool 6 pgp_num to 42 2021-02-10T05:00:28.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: cluster 2021-02-10T05:00:28.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: 2021-02-10T05:00:28.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: 2021 2021-02-10T05:00:28.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: - 2021-02-10T05:00:28.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: 02 2021-02-10T05:00:28.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: - 2021-02-10T05:00:28.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: 10 2021-02-10T05:00:28.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: T 2021-02-10T05:00:28.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: 05 2021-02-10T05:00:28.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: : 2021-02-10T05:00:28.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: 00 2021-02-10T05:00:28.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: :26.174073+0000 osd.0 (osd.0) 14 : 2021-02-10T05:00:28.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: cluster [DBG] 2.13 deep-scrub ok 2021-02-10T05:00:28.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: cluster 2021-02-10T05:00:28.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: 2021-02-10T05:00:26.278138+0000 osd.2 2021-02-10T05:00:28.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: (osd.2) 2051 : cluster 2021-02-10T05:00:28.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: [DBG] 6.28 scrub ok 2021-02-10T05:00:28.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: cluster 2021 2021-02-10T05:00:28.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: -02-10T05:00: 2021-02-10T05:00:28.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: 26.583436+0000 mgr.y (mgr. 2021-02-10T05:00:28.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: 14140) 1563 : cluster [DBG] pgmap v2542: 172 pgs: 1 remapped+peering, 23 peering, 5 active+remapped+backfill_toofull, 143 active+clean; 23 KiB data, 382 MiB used, 715 GiB / 715 GiB avail; 70/1008 objects misplaced (6.944%) 2021-02-10T05:00:28.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: 2021-02-10T05:00:28.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T05:00:28.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: T05:00:26.598005+0000 osd.7 (osd. 2021-02-10T05:00:28.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: 7) 107 : cluster [DBG] 2.1c scrub ok 2021-02-10T05:00:28.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T05:00:28.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: -10T05:00:26.843508+0000 mon.a ( 2021-02-10T05:00:28.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: mon.0) 3111 : cluster [DBG] osdmap e1389: 8 total, 8 up, 8 in 2021-02-10T05:00:28.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: cluster 2021- 2021-02-10T05:00:28.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: 02-10T05:00:26.909020+0000 osd.6 ( 2021-02-10T05:00:28.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: osd.6) 196 : cluster [DBG] 6.27 scrub ok 2021-02-10T05:00:28.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: cluster 2021- 2021-02-10T05:00:28.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: 02-10T05:00:27.180805+0000 osd.5 2021-02-10T05:00:28.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: (osd.5) 212 : cluster [DBG] 1.a scrub ok 2021-02-10T05:00:28.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: cluster 2021 2021-02-10T05:00:28.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: -02-10T05:00:27.252954+0000 osd.4 ( 2021-02-10T05:00:28.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: osd.4) 2678 : cluster [DBG] 5.e deep-scrub ok 2021-02-10T05:00:28.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: audit 2021- 2021-02-10T05:00:28.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: 02-10T05:00:27.707552+0000 2021-02-10T05:00:28.148 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:27 smithi164 conmon[29288]: mon.a (mon.0) 3112 : audit [INF] from='client.? 172.21.15.85:0/2576324429' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pgp_num", "val": "42"}]: dispatch 2021-02-10T05:00:28.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[29319]: cluster 2021-02-10T05:00:26.174073+0000 osd.0 (osd.0 2021-02-10T05:00:28.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[29319]: ) 14 : cluster [DBG] 2.13 deep-scrub ok 2021-02-10T05:00:28.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[29319]: cluster 2021-02-10T05:00:26.278138+0000 osd.2 (osd.2) 2051 : cluster [DBG] 6.28 scrub ok 2021-02-10T05:00:28.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[29319]: cluster 2021-02-10T05:00:26.583436+0000 mgr.y (mgr.14140) 1563 : cluster [DBG] pgmap v2542: 172 pgs: 1 remapped+peering, 23 peering, 5 active+remapped+backfill_toofull, 143 active+clean; 23 KiB data, 382 MiB used, 715 GiB / 715 GiB avail; 70/1008 objects misplaced (6.944%) 2021-02-10T05:00:28.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[29319]: cluster 2021-02-10T05:00:26.598005+0000 osd.7 (osd.7) 107 : cluster [DBG] 2.1c scrub ok 2021-02-10T05:00:28.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[29319]: cluster 2021-02-10T05:00:26.843508+0000 mon.a (mon.0) 3111 : cluster [DBG] osdmap e1389: 8 total, 8 up, 8 in 2021-02-10T05:00:28.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[29319]: cluster 2021-02-10T05:00:26.909020+0000 osd.6 (osd.6) 196 : cluster [DBG] 6.27 scrub ok 2021-02-10T05:00:28.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[29319]: cluster 2021-02-10T05:00:27 2021-02-10T05:00:28.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[29319]: .180805+0000 osd.5 (osd.5) 212 : cluster [DBG] 1.a scrub ok 2021-02-10T05:00:28.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[29319]: cluster 2021-02-10T05:00:27.252954+0000 osd.4 (osd.4) 2021-02-10T05:00:28.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[29319]: 2678 : cluster [DBG] 5.e deep-scrub ok 2021-02-10T05:00:28.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[29319]: audit 2021-02-10T05:00 2021-02-10T05:00:28.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[29319]: :27.707552+0000 mon.a (mon.0) 3112 2021-02-10T05:00:28.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[29319]: : audit [INF] from='client.? 172.21.15.85:0/2576324429' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pgp_num", "val": "42"}]: dispatch 2021-02-10T05:00:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: cluster 2021-02-10T05:00:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: 2021-02-10T05 2021-02-10T05:00:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: :00:26.174073+0000 osd.0 (osd.0) 14 : cluster [DBG] 2.13 deep-scrub ok 2021-02-10T05:00:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: cluster 2021-02-10T05:00:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: 2021-02-10T05:00:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: 2021 2021-02-10T05:00:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: - 2021-02-10T05:00:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: 02 2021-02-10T05:00:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: -10T05:00:26 2021-02-10T05:00:28.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: .278138+0000 osd.2 (osd.2) 2051 : cluster [DBG] 6.28 scrub ok 2021-02-10T05:00:28.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: 2021-02-10T05:00:28.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: cluster 2021- 2021-02-10T05:00:28.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: 02-10T05:00:26 2021-02-10T05:00:28.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: .583436+0000 mgr.y (mgr. 2021-02-10T05:00:28.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: 14140) 1563 : cluster [DBG] 2021-02-10T05:00:28.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: pgmap v2542: 172 pgs: 1 remapped+peering, 23 peering, 5 active+remapped+backfill_toofull, 143 active+clean; 23 KiB data, 382 MiB used, 715 GiB / 715 GiB avail; 70/1008 objects misplaced (6.944%) 2021-02-10T05:00:28.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: cluster 2021-02-10T05:00:28.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: 2021- 2021-02-10T05:00:28.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: 02-10T05:00: 2021-02-10T05:00:28.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: 26.598005+0000 osd.7 (osd 2021-02-10T05:00:28.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: .7) 107 : cluster [DBG] 2021-02-10T05:00:28.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: 2.1c scrub ok 2021-02-10T05:00:28.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: cluster 2021-02-10T05:00:28.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: 2021-02-10T05 2021-02-10T05:00:28.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: :00:26.843508+0000 2021-02-10T05:00:28.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: mon.a (mon.0) 3111 2021-02-10T05:00:28.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: : cluster [DBG] osdmap e1389: 8 total, 8 up, 8 in 2021-02-10T05:00:28.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: cluster 2021 2021-02-10T05:00:28.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: -02-10T05:00: 2021-02-10T05:00:28.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: 26.909020+0000 osd.6 (osd 2021-02-10T05:00:28.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: .6) 196 : cluster [DBG] 2021-02-10T05:00:28.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: 6.27 scrub ok 2021-02-10T05:00:28.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: cluster 2021-02-10T05:00:28.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T05:00:28.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: 05:00:27.180805+0000 2021-02-10T05:00:28.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: osd.5 (osd.5) 212 2021-02-10T05:00:28.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: : cluster [DBG] 1.a scrub ok 2021-02-10T05:00:28.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: cluster 2021 2021-02-10T05:00:28.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: -02-10T05:00: 2021-02-10T05:00:28.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: 27.252954+0000 osd.4 (osd.4) 2021-02-10T05:00:28.196 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: 2678 : cluster [DBG] 5.e deep-scrub ok 2021-02-10T05:00:28.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: audit 2021 2021-02-10T05:00:28.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: -02-10T05:00: 2021-02-10T05:00:28.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: 27.707552+0000 mon.a (mon 2021-02-10T05:00:28.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: .0) 3112 : audit [INF] 2021-02-10T05:00:28.197 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:27 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/2576324429' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pgp_num", "val": "42"}]: dispatch 2021-02-10T05:00:28.697 INFO:tasks.thrashosds.thrasher:fixing pg num pool iscsi 2021-02-10T05:00:28.698 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pool set iscsi pgp_num 13 2021-02-10T05:00:28.917 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:00:28.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:28 smithi085 conmon[29319]: cluster 2021-02-10T05:00:27.276822+0000 osd.2 (osd.2) 2052 : 2021-02-10T05:00:28.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:28 smithi085 conmon[29319]: cluster [DBG] 4.4 scrub ok 2021-02-10T05:00:28.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:28 smithi085 conmon[29319]: audit 2021-02-10T05:00:27.845073+0000 mon.a (mon.0) 3113 : audit [INF] from='client.? 172.21.15.85:0/2576324429' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pgp_num", "val": "42"}]': finished 2021-02-10T05:00:28.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:28 smithi085 conmon[29319]: cluster 2021-02-10T05:00:27.845274+0000 mon.a (mon.0) 3114 : cluster [DBG] osdmap e1390: 8 total, 8 up, 8 in 2021-02-10T05:00:28.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:28 smithi085 conmon[29319]: cluster 2021-02-10T05:00:27.867649+0000 osd.3 (osd.3) 54 : cluster [DBG] 6.13 scrub ok 2021-02-10T05:00:28.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:28 smithi085 conmon[29319]: audit 2021-02-10T05:00:28.113871+0000 mon.a (mon.0) 3115 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T05:00:28.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:28 smithi085 conmon[29319]: cluster 2021-02-10T05:00:28.650746+0000 mon.a (mon.0) 3116 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-02-10T05:00:28.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:28 smithi085 conmon[29319]: cluster 2021-02-10T05:00:28.656164+0000 mon.a (mon.0 2021-02-10T05:00:28.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:28 smithi085 conmon[29319]: ) 3117 : cluster [DBG] osdmap e1391: 8 total, 8 up, 8 in 2021-02-10T05:00:28.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:28 smithi085 conmon[35961]: cluster 2021-02-10T05:00:28.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:28 smithi085 conmon[35961]: 2021-02-10T05:00: 2021-02-10T05:00:28.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:28 smithi085 conmon[35961]: 27.276822+0000 osd.2 (osd.2) 2052 : cluster [DBG] 4.4 scrub ok 2021-02-10T05:00:28.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:28 smithi085 conmon[35961]: audit 2021-02-10T05 2021-02-10T05:00:28.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:28 smithi085 conmon[35961]: :00:27.845073+0000 mon.a (mon.0 2021-02-10T05:00:28.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:28 smithi085 conmon[35961]: ) 3113 : audit [INF] from='client.? 172.21.15.85:0/2576324429' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pgp_num", "val": "42"}]': finished 2021-02-10T05:00:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:28 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T05:00:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:28 smithi085 conmon[35961]: -10T05:00:27.845274+0000 mon.a ( 2021-02-10T05:00:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:28 smithi085 conmon[35961]: mon.0) 3114 : cluster [DBG] osdmap e1390: 8 total, 8 up, 8 in 2021-02-10T05:00:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:28 smithi085 conmon[35961]: cluster 2021- 2021-02-10T05:00:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:28 smithi085 conmon[35961]: 02-10T05:00:27.867649+0000 2021-02-10T05:00:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:28 smithi085 conmon[35961]: osd.3 (osd.3) 54 : cluster [DBG] 6.13 scrub ok 2021-02-10T05:00:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:28 smithi085 conmon[35961]: 2021-02-10T05:00:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:28 smithi085 conmon[35961]: audit 2021-02 2021-02-10T05:00:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:28 smithi085 conmon[35961]: -10T05:00:28.113871+0000 mon.a (mon.0) 3115 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T05:00:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:28 smithi085 conmon[35961]: 2021-02-10T05:00:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:28 smithi085 conmon[35961]: cluster 2021-02-10T05: 2021-02-10T05:00:28.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:28 smithi085 conmon[35961]: 00:28.650746+0000 mon.a (mon.0) 3116 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-02-10T05:00:28.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:28 smithi085 conmon[35961]: cluster 2021-02-10T05:00:28.656164 2021-02-10T05:00:28.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:28 smithi085 conmon[35961]: +0000 mon.a (mon.0) 3117 : cluster [DBG] 2021-02-10T05:00:28.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:28 smithi085 conmon[35961]: osdmap e1391: 8 total, 8 up, 8 in 2021-02-10T05:00:29.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:28 smithi164 conmon[29288]: cluster 2021-02-10T05:00:29.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:28 smithi164 conmon[29288]: 2021-02-10T05:00:29.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:28 smithi164 conmon[29288]: 2021 2021-02-10T05:00:29.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:28 smithi164 conmon[29288]: - 2021-02-10T05:00:29.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:28 smithi164 conmon[29288]: 02-10T05:00 2021-02-10T05:00:29.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:28 smithi164 conmon[29288]: :27.276822+0000 osd.2 (osd.2) 2052 : cluster [DBG] 4.4 scrub ok 2021-02-10T05:00:29.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:28 smithi164 conmon[29288]: audit 2021-02-10T05:00:29.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:28 smithi164 conmon[29288]: 2021-02-10T05:00:27.845073+0000 mon.a 2021-02-10T05:00:29.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:28 smithi164 conmon[29288]: (mon.0) 3113 : audit [INF] from='client.? 172.21.15.85:0/2576324429' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.meta", "var": "pgp_num", "val": "42"}]': finished 2021-02-10T05:00:29.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:28 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T05:00:29.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:28 smithi164 conmon[29288]: -10T05:00:27.845274+0000 mon.a (mon. 2021-02-10T05:00:29.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:28 smithi164 conmon[29288]: 0) 3114 : cluster [DBG] osdmap e1390: 8 total, 8 up, 8 in 2021-02-10T05:00:29.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:28 smithi164 conmon[29288]: cluster 2021-02-10T05:00:29.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:28 smithi164 conmon[29288]: 2021-02-10T05:00:27.867649 2021-02-10T05:00:29.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:28 smithi164 conmon[29288]: +0000 osd.3 (osd.3) 54 : cluster [DBG] 2021-02-10T05:00:29.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:28 smithi164 conmon[29288]: 6.13 scrub ok 2021-02-10T05:00:29.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:28 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T05:00:29.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:28 smithi164 conmon[29288]: T05:00:28.113871+0000 mon.a (mon 2021-02-10T05:00:29.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:28 smithi164 conmon[29288]: .0) 3115 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T05:00:29.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:28 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T05:00:29.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:28 smithi164 conmon[29288]: -10T05:00:28.650746+0000 mon.a (mon 2021-02-10T05:00:29.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:28 smithi164 conmon[29288]: .0) 3116 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-02-10T05:00:29.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:28 smithi164 conmon[29288]: cluster 2021-02-10T05:00:29.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:28 smithi164 conmon[29288]: 2021-02-10T05:00:28. 2021-02-10T05:00:29.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:28 smithi164 conmon[29288]: 656164+0000 mon.a (mon.0) 3117 : cluster 2021-02-10T05:00:29.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:28 smithi164 conmon[29288]: [DBG] osdmap e1391: 8 total, 8 up, 8 in 2021-02-10T05:00:29.894 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 05:00:29 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T05:00:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:30 smithi164 conmon[29288]: cluster 2021-02-10T05:00:28 2021-02-10T05:00:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:30 smithi164 conmon[29288]: .310260+0000 osd.2 (osd.2) 2053 : cluster [DBG] 6.20 scrub ok 2021-02-10T05:00:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:30 smithi164 conmon[29288]: cluster 2021-02-10T05:00:28. 2021-02-10T05:00:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:30 smithi164 conmon[29288]: 583931+0000 mgr.y (mgr.14140) 1564 : cluster [DBG] pgmap v2545: 172 pgs: 1 remapped+peering, 23 peering, 2 active+remapped+backfill_toofull, 146 active+clean; 23 KiB data, 386 MiB used, 715 GiB / 715 GiB avail; 40/1005 objects misplaced (3.980%); 1.2 KiB/s, 15 objects/s recovering 2021-02-10T05:00:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:30 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T05:00:30.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:30 smithi164 conmon[29288]: T05:00:29.249279+0000 osd.4 (osd.4) 2679 : cluster [DBG] 4.1 scrub ok 2021-02-10T05:00:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:30 smithi085 conmon[29319]: cluster 2021-02-10T05:00:28.310260+0000 osd.2 (osd.2) 2053 : cluster [DBG] 6.20 scrub ok 2021-02-10T05:00:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:30 smithi085 conmon[29319]: cluster 2021-02-10T05:00:28.583931+0000 mgr.y (mgr.14140) 1564 : cluster [DBG] pgmap v2545: 172 pgs: 1 remapped+peering, 23 peering, 2 active+remapped+backfill_toofull, 146 active+clean; 23 KiB data, 386 MiB used, 715 GiB / 715 GiB avail; 40/1005 objects misplaced (3.980%); 1.2 KiB/s, 15 objects/s recovering 2021-02-10T05:00:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:30 smithi085 conmon[29319]: cluster 2021-02-10T05:00:29.249279+0000 osd.4 (osd.4) 2679 : cluster [DBG] 4.1 scrub ok 2021-02-10T05:00:30.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:30 smithi085 conmon[35961]: cluster 2021-02-10T05:00:28.310260+0000 osd.2 (osd.2) 2053 : cluster [DBG] 6.20 scrub ok 2021-02-10T05:00:30.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:30 smithi085 conmon[35961]: cluster 2021-02-10T05:00:28.583931+0000 mgr.y (mgr.14140) 1564 : cluster [DBG] pgmap v2545: 172 pgs: 1 remapped+peering, 23 peering, 2 active+remapped+backfill_toofull, 146 active+clean; 23 KiB data, 386 MiB used, 715 GiB / 715 GiB avail; 40/1005 objects misplaced (3.980%); 1.2 KiB/s, 15 objects/s recovering 2021-02-10T05:00:30.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:30 smithi085 conmon[35961]: cluster 2021-02-10T05:00:29.249279+0000 osd.4 (osd.4) 2679 : cluster [DBG] 4.1 scrub ok 2021-02-10T05:00:31.128 INFO:teuthology.orchestra.run.smithi085.stderr:set pool 3 pgp_num to 13 2021-02-10T05:00:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:31 smithi164 conmon[29288]: audit 2021-02-10T05:00:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:31 smithi164 conmon[29288]: 2021-02-10T05:00:29. 2021-02-10T05:00:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:31 smithi164 conmon[29288]: 614998+0000 mgr.y (mgr.14140) 1565 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:00:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:31 smithi164 conmon[29288]: cluster 2021-02-10T05:00:29.792689+0000 osd.6 ( 2021-02-10T05:00:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:31 smithi164 conmon[29288]: osd.6) 197 : cluster [DBG] 1.6 scrub ok 2021-02-10T05:00:31.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:31 smithi164 conmon[29288]: cluster 2021- 2021-02-10T05:00:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:31 smithi164 conmon[29288]: 02-10T05:00:30.222743+0000 2021-02-10T05:00:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:31 smithi164 conmon[29288]: osd.4 (osd.4) 2680 : cluster [DBG] 2021-02-10T05:00:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:31 smithi164 conmon[29288]: 6.1e scrub ok 2021-02-10T05:00:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:31 smithi164 conmon[29288]: audit 2021-02-10T05:00: 2021-02-10T05:00:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:31 smithi164 conmon[29288]: 30.977487+0000 mon.c (mon.1) 176 : 2021-02-10T05:00:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:31 smithi164 conmon[29288]: audit [INF] from='client.? 172.21.15.85:0/3501407188' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "13"}]: dispatch 2021-02-10T05:00:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:31 smithi164 conmon[29288]: audit 2021-02-10T05:00:30.977807 2021-02-10T05:00:31.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:31 smithi164 conmon[29288]: +0000 mon.a (mon.0) 3118 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "13"}]: dispatch 2021-02-10T05:00:31.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:31 smithi085 conmon[35961]: audit 2021-02-10T05:00:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:31 smithi085 conmon[35961]: 2021-02-10T05:00 2021-02-10T05:00:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:31 smithi085 conmon[35961]: :29.614998+0000 mgr.y (mgr.14140) 1565 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:00:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:31 smithi085 conmon[35961]: cluster 2021-02-10T05:00 2021-02-10T05:00:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:31 smithi085 conmon[35961]: :29.792689+0000 osd.6 (osd.6) 197 2021-02-10T05:00:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:31 smithi085 conmon[35961]: : cluster [DBG] 1.6 scrub ok 2021-02-10T05:00:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:31 smithi085 conmon[35961]: cluster 2021-02-10T05:00:30.222743 2021-02-10T05:00:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:31 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2680 : cluster [DBG] 6.1e scrub ok 2021-02-10T05:00:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:31 smithi085 conmon[35961]: audit 2021-02 2021-02-10T05:00:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:31 smithi085 conmon[35961]: -10T05:00:30.977487+0000 2021-02-10T05:00:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:31 smithi085 conmon[35961]: mon.c (mon.1) 176 : audit [INF] 2021-02-10T05:00:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:31 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/3501407188' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "13"}]: dispatch 2021-02-10T05:00:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:31 smithi085 conmon[35961]: audit 2021-02-10T05:00:31.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:31 smithi085 conmon[35961]: 2021-02-10T05:00:30.977807+0000 mon.a (mon.0) 3118 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "13"}]: dispatch 2021-02-10T05:00:31.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:31 smithi085 conmon[35961]: 2021-02-10T05:00:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:31 smithi085 conmon[29319]: audit 2021-02-10T05:00:29.614998 2021-02-10T05:00:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:31 smithi085 conmon[29319]: +0000 mgr.y (mgr.14140) 1565 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:00:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:31 smithi085 conmon[29319]: cluster 2021-02-10T05:00:29.792689+0000 osd.6 (osd.6) 197 : cluster [DBG] 1.6 scrub ok 2021-02-10T05:00:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:31 smithi085 conmon[29319]: cluster 2021-02-10T05:00:30.222743+0000 osd.4 (osd.4) 2680 : cluster [DBG] 6.1e scrub ok 2021-02-10T05:00:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:31 smithi085 conmon[29319]: audit 2021-02-10T05:00:30.977487+0000 mon.c (mon.1) 176 : audit [INF] from='client.? 172.21.15.85:0/3501407188' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "13"}]: dispatch 2021-02-10T05:00:31.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:31 smithi085 conmon[29319]: audit 2021-02-10T05:00:30.977807+0000 mon.a (mon.0) 3118 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "13"}]: dispatch 2021-02-10T05:00:31.999 INFO:tasks.thrashosds.thrasher:fixing pg num pool zone.rgw.log 2021-02-10T05:00:32.000 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd pool set zone.rgw.log pgp_num 32 2021-02-10T05:00:32.215 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:00:32.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:32 smithi164 conmon[29288]: cluster 2021-02-10T05:00:30.367192+0000 2021-02-10T05:00:32.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:32 smithi164 conmon[29288]: osd.2 (osd.2) 2054 : cluster [DBG] 5.6 deep-scrub ok 2021-02-10T05:00:32.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:32 smithi164 conmon[29288]: cluster 2021-02-10T05:00:30.584651+0000 mgr.y (mgr.14140) 1566 2021-02-10T05:00:32.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:32 smithi164 conmon[29288]: : cluster [DBG] pgmap v2547: 172 pgs: 2 active+remapped+backfill_toofull, 170 active+clean; 26 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 9.4 KiB/s rd, 1.7 KiB/s wr, 20 op/s; 40/1077 objects misplaced (3.714%); 2.3 KiB/s, 28 objects/s recovering 2021-02-10T05:00:32.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:32 smithi164 conmon[29288]: audit 2021-02-10T05:00:32.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:32 smithi164 conmon[29288]: 2021-02-10T05:00:31.122656+0000 mon.a (mon.0) 3119 : audit 2021-02-10T05:00:32.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:32 smithi164 conmon[29288]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "13"}]': finished 2021-02-10T05:00:32.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:32 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T05:00:32.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:32 smithi164 conmon[29288]: -10T05:00:31.122811+0000 mon.a ( 2021-02-10T05:00:32.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:32 smithi164 conmon[29288]: mon.0) 3120 : cluster [DBG] osdmap e1392: 8 total, 8 up, 8 in 2021-02-10T05:00:32.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:32 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T05:00:32.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:32 smithi164 conmon[29288]: -10T05:00:31.226302+0000 osd.5 (osd. 2021-02-10T05:00:32.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:32 smithi164 conmon[29288]: 5) 213 : cluster [DBG] 5.f scrub ok 2021-02-10T05:00:32.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:32 smithi164 conmon[29288]: cluster 2021 2021-02-10T05:00:32.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:32 smithi164 conmon[29288]: -02-10T05:00:31.270141+0000 2021-02-10T05:00:32.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:32 smithi164 conmon[29288]: osd.4 (osd.4) 2681 : cluster [DBG] 5.1 scrub ok 2021-02-10T05:00:32.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:32 smithi164 conmon[29288]: 2021-02-10T05:00:32.408 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[35961]: cluster 2021-02-10T05:00: 2021-02-10T05:00:32.408 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[35961]: 30.367192+0000 osd.2 (osd.2) 2054 : cluster [DBG] 5.6 deep-scrub ok 2021-02-10T05:00:32.408 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[35961]: cluster 2021-02-10T05:00:32.409 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[35961]: 2021-02- 2021-02-10T05:00:32.409 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[35961]: 10T05:00:30.584651 2021-02-10T05:00:32.409 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1566 2021-02-10T05:00:32.409 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[35961]: : cluster [DBG] pgmap v2547: 172 pgs: 2 active+remapped+backfill_toofull, 170 active+clean; 26 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 9.4 KiB/s rd, 1.7 KiB/s wr, 20 op/s; 40/1077 objects misplaced (3.714%); 2.3 KiB/s, 28 objects/s recovering 2021-02-10T05:00:32.409 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[35961]: audit 2021- 2021-02-10T05:00:32.409 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[35961]: 02-10T05:00:31. 2021-02-10T05:00:32.409 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[35961]: 122656+0000 mon.a (mon.0) 3119 : audit 2021-02-10T05:00:32.409 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[35961]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "13"}]': finished 2021-02-10T05:00:32.409 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[35961]: cluster 2021-02-10T05:00:31. 2021-02-10T05:00:32.409 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[35961]: 122811+0000 mon.a (mon.0) 3120 : cluster [DBG] osdmap e1392: 8 total, 8 up, 8 in 2021-02-10T05:00:32.409 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T05:00:32.409 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[35961]: 05:00:31.226302+0000 osd.5 (osd.5) 213 : cluster 2021-02-10T05:00:32.410 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[35961]: [DBG] 5.f scrub ok 2021-02-10T05:00:32.410 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[35961]: cluster 2021-02-10T05:00:32.410 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[35961]: 2021-02-10T05:00:31.270141+0000 osd.4 (osd 2021-02-10T05:00:32.410 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[35961]: .4) 2681 : cluster [DBG] 5.1 scrub ok 2021-02-10T05:00:32.410 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[29319]: cluster 2021-02-10T05:00:30 2021-02-10T05:00:32.410 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[29319]: .367192+0000 osd.2 (osd.2) 2054 : cluster [DBG] 5.6 deep-scrub ok 2021-02-10T05:00:32.410 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[29319]: cluster 2021-02-10T05:00:30.584651+0000 mgr.y 2021-02-10T05:00:32.410 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[29319]: (mgr.14140) 1566 : cluster [DBG] pgmap v2547: 172 pgs: 2 active+remapped+backfill_toofull, 170 active+clean; 26 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 9.4 KiB/s rd, 1.7 KiB/s wr, 20 op/s; 40/1077 objects misplaced (3.714%); 2.3 KiB/s, 28 objects/s recovering 2021-02-10T05:00:32.411 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[29319]: audit 2021-02-10T05:00:31 2021-02-10T05:00:32.411 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[29319]: .122656+0000 mon.a (mon.0) 3119 : audit 2021-02-10T05:00:32.411 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[29319]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "13"}]': finished 2021-02-10T05:00:32.411 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[29319]: cluster 2021 2021-02-10T05:00:32.411 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[29319]: -02-10T05:00 2021-02-10T05:00:32.411 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[29319]: :31.122811+0000 mon.a (mon.0) 2021-02-10T05:00:32.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[29319]: 3120 : cluster [DBG] osdmap e1392: 8 total, 8 up, 8 in 2021-02-10T05:00:32.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[29319]: cluster 2021-02-10T05:00:32.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[29319]: 2021-02-10T05:00:31.226302 2021-02-10T05:00:32.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[29319]: +0000 osd.5 (osd.5) 213 : cluster 2021-02-10T05:00:32.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[29319]: [DBG] 5.f scrub ok 2021-02-10T05:00:32.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[29319]: cluster 2021-02-10T05:00:32.412 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[29319]: 2021-02-10T05:00:31.270141 2021-02-10T05:00:32.413 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2681 : cluster 2021-02-10T05:00:32.413 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:32 smithi085 conmon[29319]: [DBG] 5.1 scrub ok 2021-02-10T05:00:33.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:33 smithi164 conmon[29288]: cluster 2021-02-10T05:00 2021-02-10T05:00:33.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:33 smithi164 conmon[29288]: :31.627286+0000 osd.7 (osd.7) 108 : cluster [DBG] 6.8 scrub ok 2021-02-10T05:00:33.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:33 smithi164 conmon[29288]: cluster 2021-02-10T05:00:32.052655+0000 2021-02-10T05:00:33.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:33 smithi164 conmon[29288]: osd.6 (osd.6) 198 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:00:33.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:33 smithi164 conmon[29288]: cluster 2021-02-10T05 2021-02-10T05:00:33.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:33 smithi164 conmon[29288]: :00:32.055034+0000 osd.6 (osd.6) 199 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:00:33.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:33 smithi164 conmon[29288]: cluster 2021-02-10T05:00:33.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:33 smithi164 conmon[29288]: 2021-02-10T05: 2021-02-10T05:00:33.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:33 smithi164 conmon[29288]: 00:32.127011+0000 mon.a (mon.0) 3121 : cluster [DBG] osdmap e1393: 8 total, 8 up, 8 in 2021-02-10T05:00:33.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:33 smithi164 conmon[29288]: cluster 2021- 2021-02-10T05:00:33.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:33 smithi164 conmon[29288]: 02-10T05:00:32.284497+0000 2021-02-10T05:00:33.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:33 smithi164 conmon[29288]: osd.4 (osd.4) 2682 : cluster [DBG] 6.e scrub ok 2021-02-10T05:00:33.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:33 smithi164 conmon[29288]: 2021-02-10T05:00:33.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:33 smithi085 conmon[29319]: cluster 2021-02-10T05:00:31.627286+0000 osd.7 (osd 2021-02-10T05:00:33.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:33 smithi085 conmon[29319]: .7) 108 : cluster [DBG] 6.8 scrub ok 2021-02-10T05:00:33.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:33 smithi085 conmon[29319]: cluster 2021-02-10T05:00:32.052655+0000 osd.6 (osd.6) 198 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:00:33.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:33 smithi085 conmon[29319]: cluster 2021-02-10T05:00:32.055034+0000 osd.6 (osd.6) 199 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:00:33.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:33 smithi085 conmon[29319]: cluster 2021-02-10T05:00:32.127011+0000 mon.a (mon.0) 3121 : cluster [DBG] osdmap e1393: 8 total, 8 up, 8 in 2021-02-10T05:00:33.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:33 smithi085 conmon[29319]: cluster 2021-02-10T05:00: 2021-02-10T05:00:33.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:33 smithi085 conmon[29319]: 32.284497+0000 osd.4 (osd.4) 2682 : cluster [DBG] 6.e scrub ok 2021-02-10T05:00:33.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:33 smithi085 conmon[35961]: cluster 2021-02-10T05:00:31.627286 2021-02-10T05:00:33.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:33 smithi085 conmon[35961]: +0000 osd.7 (osd.7) 108 : cluster [DBG] 6.8 scrub ok 2021-02-10T05:00:33.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:33 smithi085 conmon[35961]: cluster 2021-02-10T05:00:32.052655+0000 osd.6 (osd.6) 198 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:00:33.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:33 smithi085 conmon[35961]: cluster 2021-02-10T05:00:32.055034+0000 osd.6 (osd.6) 199 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:00:33.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:33 smithi085 conmon[35961]: cluster 2021-02-10T05:00:32.127011+0000 mon.a (mon.0) 3121 : cluster [DBG] osdmap e1393: 8 total, 8 up, 8 in 2021-02-10T05:00:33.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:33 smithi085 conmon[35961]: cluster 2021-02-10T05:00:32.284497+0000 osd.4 (osd. 2021-02-10T05:00:33.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:33 smithi085 conmon[35961]: 4) 2682 : cluster [DBG] 6.e scrub ok 2021-02-10T05:00:34.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:34 smithi164 conmon[29288]: cluster 2021-02-10T05:00:32.585126+0000 mgr.y (mgr.14140 2021-02-10T05:00:34.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:34 smithi164 conmon[29288]: ) 1567 : cluster [DBG] pgmap v2550: 172 pgs: 2 active+remapped+backfill_toofull, 170 active+clean; 26 KiB data, 389 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 1.7 KiB/s wr, 20 op/s; 40/1077 objects misplaced (3.714%); 1.4 KiB/s, 17 objects/s recovering 2021-02-10T05:00:34.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:34 smithi164 conmon[29288]: cluster 2021-02-10T05:00:32.641569+0000 osd.7 (osd.7) 109 : cluster [DBG] 6.21 deep-scrub ok 2021-02-10T05:00:34.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:34 smithi164 conmon[29288]: cluster 2021-02-10T05:00:32.869716+0000 osd.3 (osd.3) 55 : cluster [DBG] 2.19 scrub ok 2021-02-10T05:00:34.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:34 smithi164 conmon[29288]: cluster 2021-02-10T05:00:33.203219+0000 osd.5 (osd.5) 214 : cluster [DBG] 2.2 scrub ok 2021-02-10T05:00:34.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:34 smithi164 conmon[29288]: cluster 2021-02-10T05:00:33.259380+0000 osd.4 (osd.4) 2683 : cluster [DBG] 5.1c deep-scrub ok 2021-02-10T05:00:34.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:34 smithi164 conmon[29288]: cluster 2021-02-10T05:00:33.311104+0000 osd.2 (osd.2) 2055 : cluster [DBG] 6.14 scrub ok 2021-02-10T05:00:34.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:34 smithi164 conmon[29288]: cluster 2021-02-10T05:00:33.652684+0000 mon.a (mon.0) 3122 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-02-10T05:00:34.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:34 smithi085 conmon[29319]: cluster 2021-02-10T05:00:32.585126+0000 mgr.y (mgr.14140) 1567 : cluster [DBG] pgmap v2550: 172 pgs: 2 active+remapped+backfill_toofull, 170 active+clean; 26 KiB data, 389 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 1.7 KiB/s wr, 20 op/s; 40/1077 objects misplaced (3.714%); 1.4 KiB/s, 17 objects/s recovering 2021-02-10T05:00:34.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:34 smithi085 conmon[29319]: cluster 2021-02-10T05:00:32.641569+0000 osd.7 (osd.7) 109 : cluster [DBG] 6.21 deep-scrub ok 2021-02-10T05:00:34.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:34 smithi085 conmon[29319]: cluster 2021-02-10T05:00:32.869716+0000 osd.3 (osd.3) 55 : cluster [DBG] 2.19 scrub ok 2021-02-10T05:00:34.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:34 smithi085 conmon[29319]: cluster 2021-02-10T05:00:33.203219+0000 osd.5 (osd.5) 214 : cluster [DBG] 2.2 scrub ok 2021-02-10T05:00:34.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:34 smithi085 conmon[29319]: cluster 2021-02-10T05:00:33.259380+0000 osd.4 (osd.4) 2683 : cluster [DBG] 5.1c deep-scrub ok 2021-02-10T05:00:34.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:34 smithi085 conmon[29319]: cluster 2021-02-10T05:00:33.311104+0000 osd.2 (osd.2) 2055 : cluster [DBG] 6.14 scrub ok 2021-02-10T05:00:34.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:34 smithi085 conmon[29319]: cluster 2021-02-10T05:00:33.652684+0000 mon.a (mon.0) 3122 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-02-10T05:00:34.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:34 smithi085 conmon[35961]: cluster 2021-02-10T05:00:32.585126+0000 mgr.y (mgr.14140) 1567 : cluster [DBG] pgmap v2550: 172 pgs: 2 active+remapped+backfill_toofull, 170 active+clean; 26 KiB data, 389 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 1.7 KiB/s wr, 20 op/s; 40/1077 objects misplaced (3.714%); 1.4 KiB/s, 17 objects/s recovering 2021-02-10T05:00:34.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:34 smithi085 conmon[35961]: cluster 2021-02-10T05:00:32.641569+0000 osd.7 (osd.7) 109 : cluster [DBG] 6.21 deep-scrub ok 2021-02-10T05:00:34.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:34 smithi085 conmon[35961]: cluster 2021-02-10T05:00:32.869716+0000 osd.3 (osd.3) 55 : cluster [DBG] 2.19 scrub ok 2021-02-10T05:00:34.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:34 smithi085 conmon[35961]: cluster 2021-02-10T05:00:33.203219+0000 osd.5 (osd.5) 214 : cluster [DBG] 2.2 scrub ok 2021-02-10T05:00:34.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:34 smithi085 conmon[35961]: cluster 2021-02-10T05:00:33.259380+0000 osd.4 (osd.4) 2683 : cluster [DBG] 5.1c deep-scrub ok 2021-02-10T05:00:34.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:34 smithi085 conmon[35961]: cluster 2021-02-10T05:00:33.311104+0000 osd.2 (osd.2) 2055 : cluster [DBG] 6.14 scrub ok 2021-02-10T05:00:34.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:34 smithi085 conmon[35961]: cluster 2021-02-10T05:00:33.652684+0000 mon.a (mon.0) 3122 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-02-10T05:00:35.151 INFO:teuthology.orchestra.run.smithi085.stderr:set pool 4 pgp_num to 32 2021-02-10T05:00:35.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:35 smithi164 conmon[29288]: cluster 2021-02-10T05:00:35.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:35 smithi164 conmon[29288]: 2021- 2021-02-10T05:00:35.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:35 smithi164 conmon[29288]: 02-10T05:00:33.657427+0000 osd.7 (osd.7) 110 : cluster [DBG] 2.27 scrub ok 2021-02-10T05:00:35.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:35 smithi164 conmon[29288]: cluster 2021-02-10T05:00:35.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:35 smithi164 conmon[29288]: 2021-02-10T05:00:33.781401+0000 2021-02-10T05:00:35.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:35 smithi164 conmon[29288]: osd.6 (osd.6) 200 : cluster [DBG] 2021-02-10T05:00:35.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:35 smithi164 conmon[29288]: 5.2 scrub ok 2021-02-10T05:00:35.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:35 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T05:00:35.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:35 smithi164 conmon[29288]: T05:00:33.837651+0000 osd.3 (osd 2021-02-10T05:00:35.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:35 smithi164 conmon[29288]: .3) 56 : cluster [DBG] 2.1b scrub ok 2021-02-10T05:00:35.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:35 smithi164 conmon[29288]: audit 2021-02-10T05:00:35.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:35 smithi164 conmon[29288]: 2021-02-10T05:00:34.154075 2021-02-10T05:00:35.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:35 smithi164 conmon[29288]: +0000 mon.a (mon.0) 3123 : audit [INF] 2021-02-10T05:00:35.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:35 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/1553817307' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pgp_num", "val": "32"}]: dispatch 2021-02-10T05:00:35.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:35 smithi164 conmon[29288]: cluster 2021-02-10T05:00: 2021-02-10T05:00:35.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:35 smithi164 conmon[29288]: 34.355696+0000 osd.2 (osd.2) 2056 : 2021-02-10T05:00:35.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:35 smithi164 conmon[29288]: cluster [DBG] 2.e scrub ok 2021-02-10T05:00:35.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:35 smithi085 conmon[35961]: cluster 2021-02-10T05:00:33.657427+0000 osd.7 (osd.7) 110 : cluster 2021-02-10T05:00:35.441 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:35 smithi085 conmon[35961]: [DBG] 2.27 scrub ok 2021-02-10T05:00:35.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:35 smithi085 conmon[35961]: cluster 2021-02-10T05:00:33.781401+0000 osd.6 (osd.6) 200 : cluster [DBG] 5.2 scrub ok 2021-02-10T05:00:35.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:35 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T05:00:35.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:35 smithi085 conmon[35961]: 05:00:33.837651+0000 osd.3 (osd. 2021-02-10T05:00:35.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:35 smithi085 conmon[35961]: 3) 56 : cluster 2021-02-10T05:00:35.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:35 smithi085 conmon[35961]: [DBG] 2.1b scrub ok 2021-02-10T05:00:35.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:35 smithi085 conmon[35961]: audit 2021- 2021-02-10T05:00:35.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:35 smithi085 conmon[35961]: 02-10T05:00:34.154075+0000 mon.a (mon. 2021-02-10T05:00:35.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:35 smithi085 conmon[35961]: 0) 3123 : audit [INF] from='client.? 172.21.15.85:0/1553817307' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pgp_num", "val": "32"}]: dispatch 2021-02-10T05:00:35.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:35 smithi085 conmon[35961]: cluster 2021-02-10T05:00:34.355696+0000 2021-02-10T05:00:35.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:35 smithi085 conmon[35961]: osd.2 (osd.2) 2056 : cluster [DBG] 2.e scrub ok 2021-02-10T05:00:35.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:35 smithi085 conmon[29319]: cluster 2021-02-10T05:00:33.657427+0000 osd.7 ( 2021-02-10T05:00:35.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:35 smithi085 conmon[29319]: osd.7) 110 : cluster [DBG] 2.27 scrub ok 2021-02-10T05:00:35.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:35 smithi085 conmon[29319]: cluster 2021-02-10T05:00:33.781401+0000 osd.6 ( 2021-02-10T05:00:35.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:35 smithi085 conmon[29319]: osd.6) 200 : cluster [DBG] 5.2 scrub ok 2021-02-10T05:00:35.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:35 smithi085 conmon[29319]: cluster 2021- 2021-02-10T05:00:35.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:35 smithi085 conmon[29319]: 02-10T05:00:33.837651+0000 osd.3 ( 2021-02-10T05:00:35.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:35 smithi085 conmon[29319]: osd.3) 56 : cluster [DBG] 2.1b scrub ok 2021-02-10T05:00:35.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:35 smithi085 conmon[29319]: audit 2021-02-10T05:00:35.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:35 smithi085 conmon[29319]: 2021-02-10T05:00: 2021-02-10T05:00:35.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:35 smithi085 conmon[29319]: 34.154075+0000 mon.a (mon.0) 3123 2021-02-10T05:00:35.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:35 smithi085 conmon[29319]: : audit [INF] from='client.? 172.21.15.85:0/1553817307' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pgp_num", "val": "32"}]: dispatch 2021-02-10T05:00:35.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:35 smithi085 conmon[29319]: cluster 2021-02-10T05:00: 2021-02-10T05:00:35.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:35 smithi085 conmon[29319]: 34.355696+0000 osd.2 (osd.2) 2056 : cluster 2021-02-10T05:00:35.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:35 smithi085 conmon[29319]: [DBG] 2.e scrub ok 2021-02-10T05:00:36.120 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2021-02-10T05:00:36.328 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:00:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:36 smithi164 conmon[29288]: cluster 2021-02-10T05:00:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:36 smithi164 conmon[29288]: 2021-02-10T05:00:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:36 smithi164 conmon[29288]: 2021 2021-02-10T05:00:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:36 smithi164 conmon[29288]: - 2021-02-10T05:00:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:36 smithi164 conmon[29288]: 02 2021-02-10T05:00:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:36 smithi164 conmon[29288]: - 2021-02-10T05:00:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:36 smithi164 conmon[29288]: 10 2021-02-10T05:00:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:36 smithi164 conmon[29288]: T05:00:34.585680+0000 2021-02-10T05:00:36.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:36 smithi164 conmon[29288]: mgr.y (mgr.14140) 1568 : cluster [DBG] pgmap v2551: 172 pgs: 2 active+remapped+backfill_toofull, 170 active+clean; 26 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 2.2 KiB/s wr, 38 op/s; 40/1077 objects misplaced (3.714%); 1.0 KiB/s, 12 objects/s recovering 2021-02-10T05:00:36.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:36 smithi164 conmon[29288]: cluster 2021-02-10T05:00:34.808615 2021-02-10T05:00:36.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:36 smithi164 conmon[29288]: +0000 osd.3 (osd.3) 57 : cluster [DBG] 5.17 scrub ok 2021-02-10T05:00:36.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:36 smithi164 conmon[29288]: audit 2021-02 2021-02-10T05:00:36.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:36 smithi164 conmon[29288]: -10T05:00:35.143687+0000 mon.a (mon 2021-02-10T05:00:36.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:36 smithi164 conmon[29288]: .0) 3124 : audit [INF] from='client.? 172.21.15.85:0/1553817307' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pgp_num", "val": "32"}]': finished 2021-02-10T05:00:36.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:36 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T05:00:36.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:36 smithi164 conmon[29288]: T05:00:35.143739+0000 mon.a (mon.0) 2021-02-10T05:00:36.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:36 smithi164 conmon[29288]: 3125 : cluster [DBG] osdmap e1394: 8 total, 8 up, 8 in 2021-02-10T05:00:36.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:36 smithi164 conmon[29288]: cluster 2021-02-10T05 2021-02-10T05:00:36.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:36 smithi164 conmon[29288]: :00:35.148955+0000 osd.0 (osd.0 2021-02-10T05:00:36.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:36 smithi164 conmon[29288]: ) 15 : cluster [DBG] 5.1e scrub ok 2021-02-10T05:00:36.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:36 smithi164 conmon[29288]: cluster 2021-02-10T05 2021-02-10T05:00:36.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:36 smithi164 conmon[29288]: :00:35.330170+0000 osd.2 (osd.2) 2057 2021-02-10T05:00:36.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:36 smithi164 conmon[29288]: : cluster [DBG] 5.b scrub ok 2021-02-10T05:00:36.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:36 smithi085 conmon[29319]: cluster 2021-02-10T05:00:34.585680+0000 2021-02-10T05:00:36.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:36 smithi085 conmon[29319]: mgr.y (mgr.14140) 1568 : cluster [DBG] pgmap v2551: 172 pgs: 2 active+remapped+backfill_toofull, 170 active+clean; 26 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 2.2 KiB/s wr, 38 op/s; 40/1077 objects misplaced (3.714%); 1.0 KiB/s, 12 objects/s recovering 2021-02-10T05:00:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:36 smithi085 conmon[29319]: cluster 2021-02-10T05:00:34.808615+0000 osd.3 (osd.3) 57 : cluster [DBG] 5.17 scrub ok 2021-02-10T05:00:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:36 smithi085 conmon[29319]: audit 2021-02-10T05:00:35.143687+0000 mon.a (mon.0) 3124 : audit [INF] from='client.? 172.21.15.85:0/1553817307' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pgp_num", "val": "32"}]': finished 2021-02-10T05:00:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:36 smithi085 conmon[29319]: cluster 2021-02-10T05:00:35.143739+0000 mon.a (mon.0) 3125 : cluster [DBG] osdmap e1394: 8 total, 8 up, 8 in 2021-02-10T05:00:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:36 smithi085 conmon[29319]: cluster 2021-02-10T05:00:35.148955+0000 osd.0 (osd.0) 15 : cluster [DBG] 5.1e scrub ok 2021-02-10T05:00:36.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:36 smithi085 conmon[29319]: cluster 2021-02-10T05:00:35.330170+0000 osd.2 (osd.2) 2057 : cluster [DBG] 5.b scrub ok 2021-02-10T05:00:36.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:36 smithi085 conmon[35961]: cluster 2021-02-10T05:00:34.585680+0000 2021-02-10T05:00:36.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:36 smithi085 conmon[35961]: mgr.y (mgr.14140) 1568 : cluster [DBG] pgmap v2551: 172 pgs: 2 active+remapped+backfill_toofull, 170 active+clean; 26 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 2.2 KiB/s wr, 38 op/s; 40/1077 objects misplaced (3.714%); 1.0 KiB/s, 12 objects/s recovering 2021-02-10T05:00:36.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:36 smithi085 conmon[35961]: cluster 2021-02-10T05:00:34.808615+0000 osd.3 (osd.3) 57 : cluster [DBG] 5.17 scrub ok 2021-02-10T05:00:36.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:36 smithi085 conmon[35961]: audit 2021-02-10T05:00:35. 2021-02-10T05:00:36.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:36 smithi085 conmon[35961]: 143687+0000 mon.a (mon.0) 3124 : audit [INF] 2021-02-10T05:00:36.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:36 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/1553817307' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "zone.rgw.log", "var": "pgp_num", "val": "32"}]': finished 2021-02-10T05:00:36.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:36 smithi085 conmon[35961]: cluster 2021-02-10T05:00:36.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:36 smithi085 conmon[35961]: 2021-02-10T05:00:35.143739+0000 2021-02-10T05:00:36.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:36 smithi085 conmon[35961]: mon.a (mon.0) 3125 : cluster [DBG] 2021-02-10T05:00:36.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:36 smithi085 conmon[35961]: osdmap e1394: 8 total, 8 up, 8 in 2021-02-10T05:00:36.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:36 smithi085 conmon[35961]: cluster 2021-02-10T05:00:36.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:36 smithi085 conmon[35961]: 2021-02-10T05:00:35.148955 2021-02-10T05:00:36.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:36 smithi085 conmon[35961]: +0000 osd.0 (osd.0) 15 : cluster [DBG] 2021-02-10T05:00:36.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:36 smithi085 conmon[35961]: 5.1e scrub ok 2021-02-10T05:00:36.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:36 smithi085 conmon[35961]: cluster 2021 2021-02-10T05:00:36.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:36 smithi085 conmon[35961]: -02-10T05:00:35.330170+0000 2021-02-10T05:00:36.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:36 smithi085 conmon[35961]: osd.2 (osd.2) 2057 : cluster [DBG] 2021-02-10T05:00:36.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:36 smithi085 conmon[35961]: 5.b scrub ok 2021-02-10T05:00:37.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:37 smithi085 conmon[29319]: cluster 2021-02-10T05:00:35.639375+0000 osd.7 (osd.7) 111 : cluster [DBG] 6.29 scrub ok 2021-02-10T05:00:37.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:37 smithi085 conmon[29319]: cluster 2021-02-10T05:00:36.219074+0000 osd.5 (osd.5) 215 : cluster [DBG] 2.12 deep-scrub ok 2021-02-10T05:00:37.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:37 smithi085 conmon[29319]: cluster 2021-02-10T05:00:36.303650+0000 osd.4 (osd.4) 2684 : cluster [DBG] 5.c scrub ok 2021-02-10T05:00:37.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:37 smithi085 conmon[29319]: cluster 2021-02-10T05:00:36.345726+0000 osd.2 (osd.2) 2058 : cluster [DBG] 2021-02-10T05:00:37.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:37 smithi085 conmon[29319]: 2.21 scrub ok 2021-02-10T05:00:37.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:37 smithi085 conmon[35961]: cluster 2021-02-10T05:00:37.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:37 smithi085 conmon[35961]: 2021-02-10T05: 2021-02-10T05:00:37.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:37 smithi085 conmon[35961]: 00:35.639375+0000 osd.7 (osd.7) 111 : cluster [DBG] 6.29 scrub ok 2021-02-10T05:00:37.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:37 smithi085 conmon[35961]: cluster 2021-02-10T05:00:36 2021-02-10T05:00:37.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:37 smithi085 conmon[35961]: .219074+0000 osd.5 (osd.5) 215 : cluster [DBG] 2.12 deep-scrub ok 2021-02-10T05:00:37.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:37 smithi085 conmon[35961]: cluster 2021-02-10T05:00:36 2021-02-10T05:00:37.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:37 smithi085 conmon[35961]: .303650+0000 osd.4 (osd.4) 2684 : cluster 2021-02-10T05:00:37.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:37 smithi085 conmon[35961]: [DBG] 5.c scrub ok 2021-02-10T05:00:37.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:37 smithi085 conmon[35961]: cluster 2021-02-10T05:00:37.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:37 smithi085 conmon[35961]: 2021-02-10T05:00:36.345726+0000 2021-02-10T05:00:37.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:37 smithi085 conmon[35961]: osd.2 (osd.2) 2058 : cluster [DBG] 2021-02-10T05:00:37.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:37 smithi085 conmon[35961]: 2.21 scrub ok 2021-02-10T05:00:37.505 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:37 smithi164 conmon[29288]: cluster 2021-02-10T05:00:35.639375+0000 2021-02-10T05:00:37.505 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:37 smithi164 conmon[29288]: osd.7 (osd.7) 111 : cluster [DBG] 6.29 scrub ok 2021-02-10T05:00:37.505 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:37 smithi164 conmon[29288]: cluster 2021-02-10T05:00:36.219074+0000 osd.5 (osd.5) 215 : cluster [DBG] 2.12 deep-scrub ok 2021-02-10T05:00:37.505 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:37 smithi164 conmon[29288]: cluster 2021-02-10T05:00:36.303650+0000 osd.4 (osd.4) 2684 2021-02-10T05:00:37.505 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:37 smithi164 conmon[29288]: : cluster [DBG] 5.c scrub ok 2021-02-10T05:00:37.505 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:37 smithi164 conmon[29288]: cluster 2021-02-10T05:00: 2021-02-10T05:00:37.505 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:37 smithi164 conmon[29288]: 36.345726+0000 osd.2 (osd.2) 2058 : cluster [DBG] 2.21 scrub ok 2021-02-10T05:00:38.435 INFO:teuthology.orchestra.run.smithi085.stdout:mon.a: {} 2021-02-10T05:00:38.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:38 smithi085 conmon[29319]: cluster 2021-02-10T05:00: 2021-02-10T05:00:38.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:38 smithi085 conmon[29319]: 36.586457+0000 mgr.y (mgr.14140) 1569 : cluster [DBG] pgmap v2553: 172 pgs: 2 active+remapped+backfill_toofull, 170 active+clean; 26 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 40/1077 objects misplaced (3.714%) 2021-02-10T05:00:38.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:38 smithi085 conmon[29319]: audit 2021-02-10T05:00:38.117947+0000 mon.a (mon.0) 3126 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T05:00:38.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:38 smithi085 conmon[35961]: cluster 2021-02-10T05:00:36.586457+0000 mgr.y (mgr. 2021-02-10T05:00:38.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:38 smithi085 conmon[35961]: 14140) 1569 : cluster [DBG] pgmap v2553: 172 pgs: 2 active+remapped+backfill_toofull, 170 active+clean; 26 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 40/1077 objects misplaced (3.714%) 2021-02-10T05:00:38.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:38 smithi085 conmon[35961]: audit 2021-02-10T05:00:38.117947+0000 mon.a (mon.0) 3126 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T05:00:38.444 INFO:teuthology.orchestra.run.smithi085.stdout:mon.c: {} 2021-02-10T05:00:38.555 INFO:teuthology.orchestra.run.smithi085.stdout:mon.b: {} 2021-02-10T05:00:38.556 INFO:teuthology.orchestra.run.smithi085.stderr:mon.a: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2021-02-10T05:00:38.556 INFO:teuthology.orchestra.run.smithi085.stderr:mon.c: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2021-02-10T05:00:38.556 INFO:teuthology.orchestra.run.smithi085.stderr:mon.b: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2021-02-10T05:00:38.619 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:38 smithi164 conmon[29288]: cluster 2021-02-10T05:00:38.619 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:38 smithi164 conmon[29288]: 2021-02-10T05: 2021-02-10T05:00:38.620 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:38 smithi164 conmon[29288]: 00:36.586457+0000 mgr.y (mgr.14140) 1569 : cluster [DBG] pgmap v2553: 172 pgs: 2 active+remapped+backfill_toofull, 170 active+clean; 26 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 40/1077 objects misplaced (3.714%) 2021-02-10T05:00:38.620 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:38 smithi164 conmon[29288]: audit 2021-02-10T05:00:38.117947+0000 2021-02-10T05:00:38.620 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:38 smithi164 conmon[29288]: mon.a (mon.0) 3126 : audit [INF] 2021-02-10T05:00:38.620 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:38 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T05:00:39.323 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T05:00:39.323 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2021-02-10T05:00:39.323 INFO:teuthology.orchestra.run.smithi164.stdout: "num_ops": 0 2021-02-10T05:00:39.324 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T05:00:39.350 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:39 smithi164 conmon[29288]: cluster 2021-02-10T05:00:39.350 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:39 smithi164 conmon[29288]: 2021-02-10T05:00:37.351092+0000 osd.4 (osd.4) 2685 : cluster [DBG] 4.13 deep-scrub ok 2021-02-10T05:00:39.350 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:39 smithi164 conmon[29288]: audit 2021-02-10T05:00:38.267776+0000 mon.a (mon.0) 3127 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T05:00:39.350 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:39 smithi164 conmon[29288]: audit 2021-02-10T05:00:38.269499+0000 mon.a (mon.0) 3128 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T05:00:39.350 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:39 smithi164 conmon[29288]: cluster 2021-02-10T05:00:38.269553+0000 osd.5 (osd.5) 216 : cluster [DBG] 6.4 scrub ok 2021-02-10T05:00:39.350 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:39 smithi164 conmon[29288]: audit 2021-02-10T05:00:38.424128+0000 mon.c (mon.1) 177 : audit [DBG] from='client.? 172.21.15.85:0/1429822776' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-02-10T05:00:39.350 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:39 smithi164 conmon[29288]: audit 2021-02-10T05:00:38.603702+0000 mon.a (mon.0) 3129 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2021-02-10T05:00:39.351 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:39 smithi164 conmon[29288]: audit 2021-02-10T05:00:38.603941+0000 mon.a (mon.0) 3130 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2021-02-10T05:00:39.351 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:39 smithi164 conmon[29288]: audit 2021-02-10T05:00:38.659683+0000 mon.a (mon.0) 3131 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2021-02-10T05:00:39.351 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:39 smithi164 conmon[29288]: audit 2021-02-10T05:00:38.659943+0000 mon.a (mon.0) 3132 : audit 2021-02-10T05:00:39.351 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:39 smithi164 conmon[29288]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2021-02-10T05:00:39.351 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:39 smithi164 conmon[29288]: cluster 2021-02-10T05:00:38.659980+0000 mon.a (mon.0) 3133 : cluster [DBG] osdmap e1395: 8 total, 8 up, 8 in 2021-02-10T05:00:39.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[29319]: cluster 2021-02-10T05:00:37.351092+0000 osd.4 (osd.4) 2685 : 2021-02-10T05:00:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[29319]: cluster [DBG] 4.13 deep-scrub ok 2021-02-10T05:00:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[29319]: audit 2021-02-10T05:00:38.267776+0000 mon.a (mon.0) 3127 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T05:00:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[29319]: audit 2021-02-10T05:00:38.269499+0000 mon.a (mon.0) 3128 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T05:00:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[29319]: cluster 2021-02-10T05:00:38.269553+0000 osd.5 (osd.5) 216 : cluster [DBG] 6.4 scrub ok 2021-02-10T05:00:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[29319]: audit 2021-02-10T05:00:38.424128+0000 mon.c (mon.1) 177 : audit [DBG] from='client.? 172.21.15.85:0/1429822776' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-02-10T05:00:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[29319]: audit 2021-02-10T05:00:38.603702+0000 mon.a (mon.0) 3129 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2021-02-10T05:00:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[29319]: audit 2021-02-10T05:00:38.603941+0000 mon.a (mon.0) 3130 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2021-02-10T05:00:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[29319]: audit 2021-02-10T05:00:38.659683+0000 mon.a (mon.0) 3131 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2021-02-10T05:00:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[29319]: audit 2021-02-10T05:00:38.659943+0000 mon.a (mon.0) 3132 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2021-02-10T05:00:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[29319]: cluster 2021-02-10T05:00:38.659980+0000 mon.a (mon.0) 3133 : cluster [DBG] osdmap e1395: 8 total, 8 up, 8 in 2021-02-10T05:00:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[35961]: cluster 2021-02-10T05:00:37.351092+0000 2021-02-10T05:00:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[35961]: osd.4 (osd.4) 2685 : cluster [DBG] 4.13 deep-scrub ok 2021-02-10T05:00:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[35961]: audit 2021-02-10T05:00:38.267776+0000 mon.a (mon.0) 3127 2021-02-10T05:00:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[35961]: : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T05:00:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T05:00:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[35961]: 10T05:00:38.269499+0000 mon.a (mon.0) 3128 : audit 2021-02-10T05:00:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[35961]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T05:00:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[35961]: cluster 2021-02-10T05:00:38. 2021-02-10T05:00:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[35961]: 269553+0000 osd.5 (osd.5) 216 : cluster 2021-02-10T05:00:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[35961]: [DBG] 6.4 scrub ok 2021-02-10T05:00:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[35961]: audit 2021-02-10T05:00 2021-02-10T05:00:39.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[35961]: :38.424128+0000 mon.c (mon.1) 2021-02-10T05:00:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[35961]: 177 : audit [DBG] from='client.? 172.21.15.85:0/1429822776' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-02-10T05:00:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[35961]: audit 2021-02-10T 2021-02-10T05:00:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[35961]: 05:00:38.603702+0000 mon.a (mon. 2021-02-10T05:00:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[35961]: 0) 3129 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2021-02-10T05:00:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T05:00:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[35961]: 10T05:00:38.603941+0000 mon.a ( 2021-02-10T05:00:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[35961]: mon.0) 3130 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2021-02-10T05:00:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[35961]: audit 2021- 2021-02-10T05:00:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[35961]: 02-10T05:00:38.659683+0000 2021-02-10T05:00:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[35961]: mon.a (mon.0) 3131 : audit [INF] 2021-02-10T05:00:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[35961]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2021-02-10T05:00:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[35961]: audit 2021-02-10T05:00:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[35961]: 2021-02-10T05:00:38.659943+0000 mon.a 2021-02-10T05:00:39.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[35961]: (mon.0) 3132 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2021-02-10T05:00:39.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[35961]: cluster 2021- 2021-02-10T05:00:39.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[35961]: 02-10T05:00:38.659980+0000 2021-02-10T05:00:39.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[35961]: mon.a (mon.0) 3133 : cluster [DBG] osdmap e1395: 8 total, 8 up, 8 in 2021-02-10T05:00:39.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:39 smithi085 conmon[35961]: 2021-02-10T05:00:39.534 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 1 1 2021-02-10T05:00:39.625 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 05:00:39 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T05:00:39.761 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:00:40.341 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:40 smithi085 conmon[29319]: cluster 2021-02-10T05:00:38.587064+0000 mgr.y (mgr.14140) 1570 : cluster [DBG] pgmap v2554: 172 pgs: 2 active+remapped+backfill_toofull, 170 active+clean; 26 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 40/1077 objects misplaced (3.714%) 2021-02-10T05:00:40.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:40 smithi085 conmon[29319]: cluster 2021-02-10T05:00:38.633989+0000 osd.7 (osd.7) 112 : cluster [DBG] 1.9 scrub ok 2021-02-10T05:00:40.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:40 smithi085 conmon[29319]: cluster 2021-02-10T05:00:39.148115+0000 osd.0 (osd.0) 16 : cluster [DBG] 6.10 scrub ok 2021-02-10T05:00:40.342 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:40 smithi085 conmon[29319]: cluster 2021-02-10T05:00:39.668015+0000 mon.a (mon.0) 3134 : cluster [DBG] osdmap e1396: 8 total, 8 up, 8 in 2021-02-10T05:00:40.342 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:40 smithi085 conmon[35961]: cluster 2021-02-10T05:00:38.587064+0000 mgr.y (mgr.14140) 1570 : cluster [DBG] pgmap v2554: 172 pgs: 2 active+remapped+backfill_toofull, 170 active+clean; 26 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 40/1077 objects misplaced (3.714%) 2021-02-10T05:00:40.342 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:40 smithi085 conmon[35961]: cluster 2021-02-10T05:00:38.633989+0000 osd.7 (osd.7) 112 : cluster [DBG] 1.9 scrub ok 2021-02-10T05:00:40.342 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:40 smithi085 conmon[35961]: cluster 2021-02-10T05:00:39.148115+0000 osd.0 (osd.0) 16 : cluster [DBG] 6.10 scrub ok 2021-02-10T05:00:40.342 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:40 smithi085 conmon[35961]: cluster 2021-02-10T05:00:39.668015+0000 mon.a (mon.0) 3134 : cluster [DBG] osdmap e1396: 8 total, 8 up, 8 in 2021-02-10T05:00:40.424 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:40 smithi164 conmon[29288]: cluster 2021-02-10T05:00:38.587064+0000 mgr.y (mgr.14140) 1570 2021-02-10T05:00:40.424 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:40 smithi164 conmon[29288]: : cluster [DBG] pgmap v2554: 172 pgs: 2 active+remapped+backfill_toofull, 170 active+clean; 26 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 40/1077 objects misplaced (3.714%) 2021-02-10T05:00:40.424 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:40 smithi164 conmon[29288]: cluster 2021-02-10T05:00:38.633989+0000 osd.7 (osd.7) 112 : cluster [DBG] 1.9 scrub ok 2021-02-10T05:00:40.424 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:40 smithi164 conmon[29288]: cluster 2021-02-10T05: 2021-02-10T05:00:40.424 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:40 smithi164 conmon[29288]: 00:39.148115+0000 osd.0 (osd.0) 16 : cluster [DBG] 6.10 scrub ok 2021-02-10T05:00:40.424 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:40 smithi164 conmon[29288]: cluster 2021 2021-02-10T05:00:40.424 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:40 smithi164 conmon[29288]: -02-10T05:00:39.668015+0000 mon.a (mon.0) 3134 : cluster 2021-02-10T05:00:40.424 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:40 smithi164 conmon[29288]: [DBG] osdmap e1396: 8 total, 8 up, 8 in 2021-02-10T05:00:41.438 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:41 smithi085 conmon[29319]: audit 2021-02-10T05:00:39.625010+0000 mgr.y (mgr.14140 2021-02-10T05:00:41.438 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:41 smithi085 conmon[29319]: ) 1571 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:00:41.438 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:41 smithi085 conmon[29319]: cluster 2021-02-10T05:00:39.728685+0000 osd.3 (osd.3) 58 : cluster [DBG] 5.13 scrub ok 2021-02-10T05:00:41.438 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:41 smithi085 conmon[29319]: cluster 2021-02-10T05:00:40.668484+0000 mon.a (mon.0) 3135 : cluster [DBG] osdmap e1397: 8 total, 8 up, 8 in 2021-02-10T05:00:41.438 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:41 smithi085 conmon[29319]: cluster 2021-02-10T05:00:40.669781+0000 osd.6 (osd.6) 201 : cluster [DBG] 4.1f starting backfill to osd.2 from (0'0,0'0] MAX to 1261'71 2021-02-10T05:00:41.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:41 smithi085 conmon[35961]: audit 2021-02-10T05:00:39.625010 2021-02-10T05:00:41.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:41 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1571 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:00:41.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:41 smithi085 conmon[35961]: cluster 2021-02-10T05:00:41.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:41 smithi085 conmon[35961]: 2021-02-10T05:00:39.728685+0000 osd.3 (osd.3 2021-02-10T05:00:41.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:41 smithi085 conmon[35961]: ) 58 : cluster [DBG] 5.13 scrub ok 2021-02-10T05:00:41.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:41 smithi085 conmon[35961]: cluster 2021 2021-02-10T05:00:41.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:41 smithi085 conmon[35961]: -02-10T05:00:40.668484+0000 2021-02-10T05:00:41.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:41 smithi085 conmon[35961]: mon.a (mon.0) 3135 : cluster [DBG] 2021-02-10T05:00:41.439 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:41 smithi085 conmon[35961]: osdmap e1397: 8 total, 8 up, 8 in 2021-02-10T05:00:41.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:41 smithi085 conmon[35961]: cluster 2021-02-10T05:00:41.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:41 smithi085 conmon[35961]: 2021-02-10T05:00: 2021-02-10T05:00:41.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:41 smithi085 conmon[35961]: 40.669781+0000 osd.6 (osd.6) 201 2021-02-10T05:00:41.440 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:41 smithi085 conmon[35961]: : cluster [DBG] 4.1f starting backfill to osd.2 from (0'0,0'0] MAX to 1261'71 2021-02-10T05:00:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:41 smithi164 conmon[29288]: audit 2021-02-10T05:00:39.625010+0000 2021-02-10T05:00:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:41 smithi164 conmon[29288]: mgr.y (mgr.14140) 1571 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:00:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:41 smithi164 conmon[29288]: cluster 2021-02-10T05:00:39.728685+0000 osd.3 (osd.3) 58 : cluster [DBG] 5.13 scrub ok 2021-02-10T05:00:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:41 smithi164 conmon[29288]: cluster 2021-02-10T05:00:40.668484+0000 mon.a (mon.0) 3135 : cluster [DBG] osdmap e1397: 8 total, 8 up, 8 in 2021-02-10T05:00:41.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:41 smithi164 conmon[29288]: cluster 2021-02-10T05:00:40.669781+0000 osd.6 (osd.6) 201 : cluster [DBG] 4.1f starting backfill to osd.2 from (0'0,0'0] MAX to 1261'71 2021-02-10T05:00:42.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[35961]: cluster 2021-02-10T05:00:40.192512+0000 osd.0 (osd.0) 2021-02-10T05:00:42.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[35961]: 17 : cluster [DBG] 5.3 deep-scrub ok 2021-02-10T05:00:42.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[35961]: cluster 2021-02-10T05:00:40.379845+0000 osd.4 (osd.4) 2686 : cluster [DBG] 6.1a scrub ok 2021-02-10T05:00:42.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[35961]: cluster 2021-02-10T05:00:40.587557+0000 mgr.y (mgr 2021-02-10T05:00:42.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[35961]: .14140) 1572 : cluster [DBG] pgmap v2557: 172 pgs: 1 remapped+peering, 1 peering, 2 active+remapped+backfill_toofull, 168 active+clean; 25 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 8.5 KiB/s rd, 511 B/s wr, 15 op/s; 40/1071 objects misplaced (3.735%); 52 B/s, 1 objects/s recovering 2021-02-10T05:00:42.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[35961]: cluster 2021-02-10T05:00:40.609147+0000 osd.7 2021-02-10T05:00:42.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[35961]: (osd.7) 113 : cluster [DBG] 6.5 scrub ok 2021-02-10T05:00:42.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[35961]: cluster 2021-02-10T05:00:40.671813 2021-02-10T05:00:42.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[35961]: +0000 osd.0 (osd.0) 18 : cluster [DBG] 4.7 starting backfill to osd.2 from (0'0,0'0] MAX to 1261'40 2021-02-10T05:00:42.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[35961]: cluster 2021-02-10T05:00: 2021-02-10T05:00:42.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[35961]: 41.159589+0000 mon.a (mon.0) 3136 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2021-02-10T05:00:42.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T05:00:42.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[35961]: 05:00:41.240250+0000 osd.2 (osd.2) 2059 : cluster 2021-02-10T05:00:42.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[35961]: [DBG] 2.1 scrub ok 2021-02-10T05:00:42.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[35961]: cluster 2021 2021-02-10T05:00:42.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[35961]: -02-10T05:00:41.669907+0000 mon.a (mon.0) 3137 : cluster [DBG] osdmap e1398: 8 total, 8 up, 8 in 2021-02-10T05:00:42.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[35961]: 2021-02-10T05:00:42.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T05:00:42.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[35961]: 05:00:41.695610+0000 osd.6 (osd.6) 202 : cluster [DBG] 2.22 scrub ok 2021-02-10T05:00:42.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[35961]: audit 2021-02-10T05 2021-02-10T05:00:42.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[35961]: :00:41.804186+0000 mon.a (mon.0) 3138 : audit 2021-02-10T05:00:42.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[35961]: [INF] from='client.? 172.21.15.85:0/2179973107' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-02-10T05:00:42.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[29319]: cluster 2021-02-10T05:00:40.192512+0000 osd.0 (osd.0) 17 2021-02-10T05:00:42.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[29319]: : cluster [DBG] 5.3 deep-scrub ok 2021-02-10T05:00:42.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[29319]: cluster 2021-02-10T05:00:40.379845+0000 osd.4 (osd.4) 2686 : cluster [DBG] 6.1a scrub ok 2021-02-10T05:00:42.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[29319]: cluster 2021-02-10T05:00:40.587557+0000 mgr.y (mgr.14140) 1572 : cluster [DBG] pgmap v2557: 172 pgs: 1 remapped+peering, 1 peering, 2 active+remapped+backfill_toofull, 168 active+clean; 25 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 8.5 KiB/s rd, 511 B/s wr, 15 op/s; 40/1071 objects misplaced (3.735%); 52 B/s, 1 objects/s recovering 2021-02-10T05:00:42.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[29319]: cluster 2021-02-10T05:00:40.609147+0000 osd.7 (osd.7) 113 : cluster [DBG] 6.5 scrub ok 2021-02-10T05:00:42.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T05:00:42.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[29319]: 10T05:00:40.671813+0000 osd.0 (osd.0) 18 : cluster [DBG] 4.7 starting backfill to osd.2 from (0'0,0'0] MAX to 1261'40 2021-02-10T05:00:42.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[29319]: cluster 2021-02-10T05 2021-02-10T05:00:42.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[29319]: :00:41.159589+0000 2021-02-10T05:00:42.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[29319]: mon.a (mon.0) 3136 : cluster [WRN] 2021-02-10T05:00:42.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[29319]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2021-02-10T05:00:42.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T05:00:42.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[29319]: 10T05:00:41.240250+0000 2021-02-10T05:00:42.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[29319]: osd.2 (osd.2) 2059 : cluster [DBG] 2021-02-10T05:00:42.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[29319]: 2.1 scrub ok 2021-02-10T05:00:42.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T05:00:42.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[29319]: T05:00:41.669907+0000 2021-02-10T05:00:42.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[29319]: mon.a (mon.0) 3137 : cluster [DBG] 2021-02-10T05:00:42.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[29319]: osdmap e1398: 8 total, 8 up, 8 in 2021-02-10T05:00:42.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T05:00:42.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[29319]: 05:00:41.695610+0000 osd.6 2021-02-10T05:00:42.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[29319]: (osd.6) 202 : cluster [DBG] 2.22 scrub ok 2021-02-10T05:00:42.445 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[29319]: 2021-02-10T05:00:42.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[29319]: audit 2021-02-10T 2021-02-10T05:00:42.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[29319]: 05:00:41.804186+0000 mon.a ( 2021-02-10T05:00:42.446 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:42 smithi085 conmon[29319]: mon.0) 3138 : audit [INF] from='client.? 172.21.15.85:0/2179973107' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-02-10T05:00:42.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:42 smithi164 conmon[29288]: cluster 2021-02-10T05:00:40.192512 2021-02-10T05:00:42.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:42 smithi164 conmon[29288]: +0000 osd.0 (osd.0) 17 : cluster [DBG] 5.3 deep-scrub ok 2021-02-10T05:00:42.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:42 smithi164 conmon[29288]: cluster 2021-02-10T05:00: 2021-02-10T05:00:42.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:42 smithi164 conmon[29288]: 40.379845+0000 osd.4 (osd.4) 2686 : cluster [DBG] 6.1a scrub ok 2021-02-10T05:00:42.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:42 smithi164 conmon[29288]: cluster 2021-02-10T05:00 2021-02-10T05:00:42.649 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:42 smithi164 conmon[29288]: :40.587557+0000 mgr.y (mgr.14140) 1572 : cluster [DBG] pgmap v2557: 172 pgs: 1 remapped+peering, 1 peering, 2 active+remapped+backfill_toofull, 168 active+clean; 25 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 8.5 KiB/s rd, 511 B/s wr, 15 op/s; 40/1071 objects misplaced (3.735%); 52 B/s, 1 objects/s recovering 2021-02-10T05:00:42.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:42 smithi164 conmon[29288]: cluster 2021-02-10T05:00:42.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:42 smithi164 conmon[29288]: 2021-02-10T05:00:40.609147+0000 osd.7 (osd.7) 113 2021-02-10T05:00:42.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:42 smithi164 conmon[29288]: : cluster [DBG] 6.5 scrub ok 2021-02-10T05:00:42.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:42 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T05:00:42.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:42 smithi164 conmon[29288]: 10T05:00:40.671813+0000 osd.0 ( 2021-02-10T05:00:42.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:42 smithi164 conmon[29288]: osd.0) 18 : cluster [DBG] 4.7 starting backfill to osd.2 from (0'0,0'0] MAX to 1261'40 2021-02-10T05:00:42.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:42 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T05:00:42.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:42 smithi164 conmon[29288]: 10T05:00:41.159589+0000 mon.a (mon. 2021-02-10T05:00:42.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:42 smithi164 conmon[29288]: 0) 3136 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2021-02-10T05:00:42.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:42 smithi164 conmon[29288]: cluster 2021-02-10T05:00: 2021-02-10T05:00:42.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:42 smithi164 conmon[29288]: 41.240250+0000 osd.2 (osd.2) 2059 : cluster [DBG] 2.1 scrub ok 2021-02-10T05:00:42.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:42 smithi164 conmon[29288]: cluster 2021-02-10T05:00:41. 2021-02-10T05:00:42.650 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:42 smithi164 conmon[29288]: 669907+0000 mon.a (mon.0) 3137 : cluster [DBG] osdmap e1398: 8 total, 8 up, 8 in 2021-02-10T05:00:42.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:42 smithi164 conmon[29288]: 2021-02-10T05:00:42.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:42 smithi164 conmon[29288]: cluster 2021-02-10T05:00 2021-02-10T05:00:42.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:42 smithi164 conmon[29288]: :41.695610+0000 osd.6 (osd.6) 202 : cluster [DBG] 2.22 scrub ok 2021-02-10T05:00:42.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:42 smithi164 conmon[29288]: audit 2021-02-10T05:00:41.804186+0000 mon.a ( 2021-02-10T05:00:42.651 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:42 smithi164 conmon[29288]: mon.0) 3138 : audit [INF] from='client.? 172.21.15.85:0/2179973107' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-02-10T05:00:42.680 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.1 to 1 (10000) 2021-02-10T05:00:43.801 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd primary-affinity 1 1 2021-02-10T05:00:43.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[35961]: cluster 2021-02-10T05:00:42.197572 2021-02-10T05:00:43.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[35961]: +0000 osd.0 (osd.0) 19 : cluster [DBG] 6.18 scrub ok 2021-02-10T05:00:43.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[35961]: cluster 2021-02-10T05:00:43.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[35961]: 2021-02-10T05:00:42.239880+0000 osd.2 (osd.2) 2060 : cluster 2021-02-10T05:00:43.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[35961]: [DBG] 3.4 scrub ok 2021-02-10T05:00:43.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T05:00:43.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[35961]: 10T05:00:42.588051+0000 mgr.y ( 2021-02-10T05:00:43.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[35961]: mgr.14140) 1573 : cluster [DBG] pgmap v2560: 172 pgs: 1 remapped+peering, 1 peering, 2 active+remapped+backfill_toofull, 168 active+clean; 25 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 767 B/s wr, 22 op/s; 40/1071 objects misplaced (3.735%); 78 B/s, 2 objects/s recovering 2021-02-10T05:00:43.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T05:00:43.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[35961]: 10T05:00:42.674143+0000 mon.a (mon. 2021-02-10T05:00:43.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[35961]: 0) 3139 : audit [INF] from='client.? 172.21.15.85:0/2179973107' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-02-10T05:00:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[35961]: cluster 2021-02-10T05:00:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[35961]: 2021-02-10T05:00:42.674210 2021-02-10T05:00:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[35961]: +0000 mon.a (mon.0) 3140 : cluster [DBG] 2021-02-10T05:00:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[35961]: osdmap e1399: 8 total, 8 up, 8 in 2021-02-10T05:00:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[35961]: cluster 2021-02-10T05:00:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[35961]: 2021-02-10T05:00:42.721343 2021-02-10T05:00:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[35961]: +0000 osd.3 (osd.3) 59 : cluster 2021-02-10T05:00:43.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[35961]: [DBG] 4.11 scrub ok 2021-02-10T05:00:43.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[29319]: cluster 2021-02-10T05:00:42.197572+0000 osd.0 (osd 2021-02-10T05:00:43.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[29319]: .0) 19 : cluster [DBG] 6.18 scrub ok 2021-02-10T05:00:43.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[29319]: cluster 2021-02-10T05:00:42.239880+0000 osd.2 (osd.2) 2021-02-10T05:00:43.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[29319]: 2060 : cluster [DBG] 3.4 scrub ok 2021-02-10T05:00:43.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[29319]: cluster 2021-02-10T05:00:42.588051+0000 mgr.y (mgr.14140) 1573 : cluster [DBG] pgmap v2560: 172 pgs: 1 remapped+peering, 1 peering, 2 active+remapped+backfill_toofull, 168 active+clean; 25 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 767 B/s wr, 22 op/s; 40/1071 objects misplaced (3.735%); 78 B/s, 2 objects/s recovering 2021-02-10T05:00:43.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[29319]: audit 2021-02-10T 2021-02-10T05:00:43.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[29319]: 05:00:42.674143+0000 mon.a (mon.0 2021-02-10T05:00:43.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[29319]: ) 3139 : audit [INF] from='client.? 172.21.15.85:0/2179973107' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-02-10T05:00:43.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[29319]: cluster 2021 2021-02-10T05:00:43.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[29319]: -02-10T05:00:42.674210+0000 2021-02-10T05:00:43.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[29319]: mon.a (mon.0) 3140 : cluster [DBG] 2021-02-10T05:00:43.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[29319]: osdmap e1399: 8 total, 8 up, 8 in 2021-02-10T05:00:43.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[29319]: cluster 2021- 2021-02-10T05:00:43.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[29319]: 02-10T05:00:42.721343+0000 2021-02-10T05:00:43.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[29319]: osd.3 (osd.3) 59 : cluster [DBG] 4.11 scrub ok 2021-02-10T05:00:43.944 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:43 smithi085 conmon[29319]: 2021-02-10T05:00:44.005 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:00:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:43 smithi164 conmon[29288]: cluster 2021-02-10T05:00:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:43 smithi164 conmon[29288]: 2021-02-10T05 2021-02-10T05:00:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:43 smithi164 conmon[29288]: :00:42.197572+0000 osd.0 (osd.0) 19 : cluster [DBG] 6.18 scrub ok 2021-02-10T05:00:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:43 smithi164 conmon[29288]: cluster 2021-02-10T05:00:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:43 smithi164 conmon[29288]: 2021-02-10T05:00:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:43 smithi164 conmon[29288]: 2021 2021-02-10T05:00:44.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:43 smithi164 conmon[29288]: -02-10T05:00:42 2021-02-10T05:00:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:43 smithi164 conmon[29288]: .239880+0000 osd.2 (osd.2) 2060 : cluster [DBG] 3.4 scrub ok 2021-02-10T05:00:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:43 smithi164 conmon[29288]: cluster 2021-02-10T05:00:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:43 smithi164 conmon[29288]: 2021-02-10T05:00:42.588051+0000 mgr.y 2021-02-10T05:00:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:43 smithi164 conmon[29288]: (mgr.14140) 1573 : cluster [DBG] pgmap v2560: 172 pgs: 1 remapped+peering, 1 peering, 2 active+remapped+backfill_toofull, 168 active+clean; 25 KiB data, 390 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 767 B/s wr, 22 op/s; 40/1071 objects misplaced (3.735%); 78 B/s, 2 objects/s recovering 2021-02-10T05:00:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:43 smithi164 conmon[29288]: audit 2021 2021-02-10T05:00:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:43 smithi164 conmon[29288]: -02-10T05:00:42.674143+0000 mon.a 2021-02-10T05:00:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:43 smithi164 conmon[29288]: (mon.0) 3139 : audit [INF] from='client.? 172.21.15.85:0/2179973107' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-02-10T05:00:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:43 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T05:00:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:43 smithi164 conmon[29288]: -10T05:00:42.674210+0000 mon.a ( 2021-02-10T05:00:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:43 smithi164 conmon[29288]: mon.0) 3140 : cluster [DBG] osdmap e1399: 8 total, 8 up, 8 in 2021-02-10T05:00:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:43 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T05:00:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:43 smithi164 conmon[29288]: 10T05:00:42.721343+0000 osd.3 (osd.3 2021-02-10T05:00:44.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:43 smithi164 conmon[29288]: ) 59 : cluster [DBG] 4.11 scrub ok 2021-02-10T05:00:44.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:44 smithi085 conmon[29319]: cluster 2021-02-10T05:00:43.223484 2021-02-10T05:00:44.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:44 smithi085 conmon[29319]: +0000 osd.0 (osd.0) 20 : cluster [DBG] 4.3 deep-scrub ok 2021-02-10T05:00:44.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:44 smithi085 conmon[29319]: cluster 2021-02-10T05:00:43.429020+0000 osd.4 (osd.4) 2687 : cluster [DBG] 6.2 scrub ok 2021-02-10T05:00:44.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:44 smithi085 conmon[29319]: cluster 2021-02-10T05:00:44.249460+0000 osd.5 (osd.5) 217 : cluster [DBG] 6.7 scrub ok 2021-02-10T05:00:44.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:44 smithi085 conmon[35961]: cluster 2021-02-10T05:00:43. 2021-02-10T05:00:44.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:44 smithi085 conmon[35961]: 223484+0000 osd.0 (osd.0) 20 : cluster [DBG] 4.3 deep-scrub ok 2021-02-10T05:00:44.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:44 smithi085 conmon[35961]: cluster 2021-02-10T05:00:43.429020+0000 osd.4 (osd.4) 2687 : cluster [DBG] 6.2 scrub ok 2021-02-10T05:00:44.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:44 smithi085 conmon[35961]: cluster 2021-02-10T05:00:44.249460+0000 osd.5 (osd.5) 217 : cluster [DBG] 6.7 scrub ok 2021-02-10T05:00:45.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:44 smithi164 conmon[29288]: cluster 2021-02-10T05:00:43.223484+0000 2021-02-10T05:00:45.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:44 smithi164 conmon[29288]: osd.0 (osd.0) 20 : cluster [DBG] 4.3 deep-scrub ok 2021-02-10T05:00:45.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:44 smithi164 conmon[29288]: cluster 2021-02-10T05:00:45.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:44 smithi164 conmon[29288]: 2021-02-10T05:00:43.429020+0000 osd.4 (osd.4 2021-02-10T05:00:45.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:44 smithi164 conmon[29288]: ) 2687 : cluster [DBG] 6.2 scrub ok 2021-02-10T05:00:45.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:44 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T05:00:45.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:44 smithi164 conmon[29288]: -10T05:00:44.249460+0000 osd.5 (osd.5 2021-02-10T05:00:45.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:44 smithi164 conmon[29288]: ) 217 : cluster [DBG] 6.7 scrub ok 2021-02-10T05:00:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:45 smithi085 conmon[29319]: cluster 2021-02-10T05:00:44.220319+0000 2021-02-10T05:00:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:45 smithi085 conmon[29319]: osd.2 (osd.2) 2061 : cluster [DBG] 4.1c deep-scrub ok 2021-02-10T05:00:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:45 smithi085 conmon[29319]: cluster 2021-02-10T05:00:44.588603+0000 mgr.y (mgr.14140) 1574 : cluster [DBG] pgmap v2562: 172 pgs: 1 remapped+peering, 2 active+remapped+backfill_toofull, 169 active+clean; 25 KiB data, 391 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 624 B/s wr, 19 op/s; 40/1071 objects misplaced (3.735%); 277 B/s, 4 objects/s recovering 2021-02-10T05:00:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:45 smithi085 conmon[29319]: cluster 2021-02-10T05:00:44.703853+0000 osd.6 (osd.6) 203 : cluster [DBG] 2.6 scrub ok 2021-02-10T05:00:45.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:45 smithi085 conmon[29319]: cluster 2021-02-10T05:00:45.146300+0000 osd.0 (osd.0) 21 : cluster [DBG] 6.16 scrub ok 2021-02-10T05:00:45.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:45 smithi085 conmon[35961]: cluster 2021-02-10T05:00:44 2021-02-10T05:00:45.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:45 smithi085 conmon[35961]: .220319+0000 osd.2 (osd.2) 2061 : cluster [DBG] 4.1c deep-scrub ok 2021-02-10T05:00:45.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:45 smithi085 conmon[35961]: cluster 2021-02-10T05:00:44.588603+0000 mgr.y (mgr.14140) 1574 : cluster [DBG] pgmap v2562: 172 pgs: 1 remapped+peering, 2 active+remapped+backfill_toofull, 169 active+clean; 25 KiB data, 391 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 624 B/s wr, 19 op/s; 40/1071 objects misplaced (3.735%); 277 B/s, 4 objects/s recovering 2021-02-10T05:00:45.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:45 smithi085 conmon[35961]: cluster 2021-02-10T05:00:44.703853+0000 osd.6 (osd.6 2021-02-10T05:00:45.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:45 smithi085 conmon[35961]: ) 203 : cluster [DBG] 2.6 scrub ok 2021-02-10T05:00:45.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:45 smithi085 conmon[35961]: cluster 2021-02-10T05:00:45.146300+0000 osd.0 (osd.0) 21 : cluster [DBG] 2021-02-10T05:00:45.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:45 smithi085 conmon[35961]: 6.16 scrub ok 2021-02-10T05:00:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:45 smithi164 conmon[29288]: cluster 2021-02-10T05:00:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:45 smithi164 conmon[29288]: 2021-02-10T05:00 2021-02-10T05:00:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:45 smithi164 conmon[29288]: :44.220319+0000 osd.2 (osd.2) 2061 : cluster [DBG] 4.1c deep-scrub ok 2021-02-10T05:00:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:45 smithi164 conmon[29288]: cluster 2021 2021-02-10T05:00:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:45 smithi164 conmon[29288]: -02-10T05:00:44.588603+0000 mgr.y (mgr 2021-02-10T05:00:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:45 smithi164 conmon[29288]: .14140) 1574 : cluster [DBG] pgmap v2562: 172 pgs: 1 remapped+peering, 2 active+remapped+backfill_toofull, 169 active+clean; 25 KiB data, 391 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 624 B/s wr, 19 op/s; 40/1071 objects misplaced (3.735%); 277 B/s, 4 objects/s recovering 2021-02-10T05:00:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:45 smithi164 conmon[29288]: cluster 2021 2021-02-10T05:00:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:45 smithi164 conmon[29288]: -02-10T05:00:44.703853+0000 2021-02-10T05:00:46.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:45 smithi164 conmon[29288]: osd.6 (osd.6) 203 : cluster [DBG] 2.6 scrub ok 2021-02-10T05:00:46.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:45 smithi164 conmon[29288]: cluster 2021 2021-02-10T05:00:46.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:45 smithi164 conmon[29288]: -02-10T05:00:45.146300+0000 2021-02-10T05:00:46.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:45 smithi164 conmon[29288]: osd.0 (osd.0) 21 : cluster [DBG] 6.16 scrub ok 2021-02-10T05:00:46.697 INFO:teuthology.orchestra.run.smithi085.stderr:set osd.1 primary-affinity to 1 (8655362) 2021-02-10T05:00:46.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:46 smithi085 conmon[29319]: cluster 2021-02-10T05:00:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:46 smithi085 conmon[29319]: 2021-02-10T 2021-02-10T05:00:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:46 smithi085 conmon[29319]: 05:00:45.170103+0000 osd.2 (osd.2) 2062 : cluster [DBG] 3.0 scrub ok 2021-02-10T05:00:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:46 smithi085 conmon[29319]: cluster 2021 2021-02-10T05:00:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:46 smithi085 conmon[29319]: -02-10T05:00:45.427081+0000 2021-02-10T05:00:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:46 smithi085 conmon[29319]: osd.4 (osd.4) 2688 : cluster [DBG] 2.d scrub ok 2021-02-10T05:00:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:46 smithi085 conmon[29319]: 2021-02-10T05:00:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:46 smithi085 conmon[29319]: cluster 2021-02-10T05:00: 2021-02-10T05:00:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:46 smithi085 conmon[29319]: 45.613170+0000 osd.3 (osd.3) 60 : cluster 2021-02-10T05:00:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:46 smithi085 conmon[29319]: [DBG] 2.a scrub ok 2021-02-10T05:00:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:46 smithi085 conmon[29319]: audit 2021-02-10T05:00:46.019533+0000 mon.a ( 2021-02-10T05:00:46.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:46 smithi085 conmon[29319]: mon.0) 3141 : audit [INF] from='client.? 172.21.15.85:0/875318818' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2021-02-10T05:00:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:46 smithi085 conmon[35961]: cluster 2021-02-10T05:00:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:46 smithi085 conmon[35961]: 2021-02-10T05:00:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:46 smithi085 conmon[35961]: 2021 2021-02-10T05:00:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:46 smithi085 conmon[35961]: -02-10T05:00:45 2021-02-10T05:00:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:46 smithi085 conmon[35961]: .170103+0000 osd.2 (osd.2) 2062 : cluster [DBG] 3.0 scrub ok 2021-02-10T05:00:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:46 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T05:00:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:46 smithi085 conmon[35961]: 10T05:00:45.427081+0000 osd.4 (osd. 2021-02-10T05:00:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:46 smithi085 conmon[35961]: 4) 2688 : cluster [DBG] 2.d scrub ok 2021-02-10T05:00:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:46 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T05:00:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:46 smithi085 conmon[35961]: -10T05:00:45.613170+0000 osd.3 2021-02-10T05:00:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:46 smithi085 conmon[35961]: (osd.3) 60 : cluster [DBG] 2.a scrub ok 2021-02-10T05:00:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:46 smithi085 conmon[35961]: audit 2021 2021-02-10T05:00:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:46 smithi085 conmon[35961]: -02-10T05:00:46 2021-02-10T05:00:46.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:46 smithi085 conmon[35961]: .019533+0000 mon.a (mon.0) 3141 : audit 2021-02-10T05:00:46.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:46 smithi085 conmon[35961]: [INF] from='client.? 172.21.15.85:0/875318818' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2021-02-10T05:00:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:46 smithi164 conmon[29288]: cluster 2021-02-10T05:00:45.170103+0000 osd.2 (osd.2) 2062 : 2021-02-10T05:00:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:46 smithi164 conmon[29288]: cluster [DBG] 3.0 scrub ok 2021-02-10T05:00:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:46 smithi164 conmon[29288]: cluster 2021-02-10T05:00:45.427081+0000 osd.4 (osd.4) 2688 : cluster [DBG] 2.d scrub ok 2021-02-10T05:00:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:46 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T05:00:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:46 smithi164 conmon[29288]: 05:00:45.613170+0000 osd.3 (osd. 2021-02-10T05:00:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:46 smithi164 conmon[29288]: 3) 60 : cluster [DBG] 2.a scrub ok 2021-02-10T05:00:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:46 smithi164 conmon[29288]: audit 2021-02- 2021-02-10T05:00:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:46 smithi164 conmon[29288]: 10T05:00:46.019533+0000 mon.a (mon.0 2021-02-10T05:00:47.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:46 smithi164 conmon[29288]: ) 3141 : audit [INF] from='client.? 172.21.15.85:0/875318818' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2021-02-10T05:00:47.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[49106]: debug 2021-02-10T05:00:47.160+0000 7f778961a700 -1 osd.2 pg_epoch: 1400 pg[4.7( v 1261'40 (1063'30,1261'40] local-lis/les=1398/1399 n=6 ec=55/55 lis/c=1398/1398 les/c/f=1399/1399/0 sis=1398) [2,5,0] r=0 lpr=1398 crt=1261'40 mlcod 0'0 active+clean] scrubber pg(4.7) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 5 (0x564a9faef8c0) 2021-02-10T05:00:47.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[29319]: cluster 2021-02-10T05:00:46.589133+0000 2021-02-10T05:00:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[29319]: mgr.y (mgr.14140) 1575 : cluster [DBG] pgmap v2563: 172 pgs: 2 active+remapped+backfill_toofull, 170 active+clean; 25 KiB data, 395 MiB used, 715 GiB / 715 GiB avail; 40/1065 objects misplaced (3.756%); 213 B/s, 0 objects/s recovering 2021-02-10T05:00:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[29319]: cluster 2021-02-10T05:00:46. 2021-02-10T05:00:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[29319]: 677458+0000 mon.a (mon.0) 3142 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2021-02-10T05:00:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[29319]: audit 2021-02-10T05: 2021-02-10T05:00:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[29319]: 00:46.685875+0000 mon.a (mon.0) 3143 : audit [INF] 2021-02-10T05:00:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[29319]: from='client.? 172.21.15.85:0/875318818' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2021-02-10T05:00:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[29319]: cluster 2021-02-10T05:00:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[29319]: 2021-02-10T05:00:46.686072+0000 mon.a (mon. 2021-02-10T05:00:47.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[29319]: 0) 3144 : cluster [DBG] osdmap e1400: 8 total, 8 up, 8 in 2021-02-10T05:00:47.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[35961]: cluster 2021-02-10T05:00:47.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[35961]: 2021-02-10T05:00:47.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[35961]: 2021 2021-02-10T05:00:47.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[35961]: - 2021-02-10T05:00:47.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[35961]: 02 2021-02-10T05:00:47.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[35961]: - 2021-02-10T05:00:47.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[35961]: 10 2021-02-10T05:00:47.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[35961]: T 2021-02-10T05:00:47.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[35961]: 05 2021-02-10T05:00:47.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[35961]: : 2021-02-10T05:00:47.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[35961]: 00 2021-02-10T05:00:47.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[35961]: : 2021-02-10T05:00:47.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[35961]: 46 2021-02-10T05:00:47.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[35961]: .589133+0000 mgr.y (mgr.14140) 2021-02-10T05:00:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[35961]: 1575 : cluster [DBG] pgmap v2563: 172 pgs: 2 active+remapped+backfill_toofull, 170 active+clean; 25 KiB data, 395 MiB used, 715 GiB / 715 GiB avail; 40/1065 objects misplaced (3.756%); 213 B/s, 0 objects/s recovering 2021-02-10T05:00:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T05:00:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[35961]: -10T05:00:46.677458 2021-02-10T05:00:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[35961]: +0000 mon.a (mon.0) 3142 : 2021-02-10T05:00:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[35961]: cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2021-02-10T05:00:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[35961]: audit 2021 2021-02-10T05:00:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[35961]: -02-10T05:00: 2021-02-10T05:00:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[35961]: 46.685875+0000 mon.a (mon. 2021-02-10T05:00:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[35961]: 0) 3143 : audit [INF] from='client.? 172.21.15.85:0/875318818' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2021-02-10T05:00:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[35961]: 2021-02-10T05:00:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T05:00:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[35961]: -10T05:00:46.686072 2021-02-10T05:00:47.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:47 smithi085 conmon[35961]: +0000 mon.a (mon.0) 3144 : cluster [DBG] osdmap e1400: 8 total, 8 up, 8 in 2021-02-10T05:00:47.980 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 2 1 2021-02-10T05:00:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:47 smithi164 conmon[29288]: cluster 2021-02-10T05:00:46. 2021-02-10T05:00:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:47 smithi164 conmon[29288]: 589133+0000 mgr.y (mgr.14140) 1575 : cluster [DBG] pgmap v2563: 172 pgs: 2 active+remapped+backfill_toofull, 170 active+clean; 25 KiB data, 395 MiB used, 715 GiB / 715 GiB avail; 40/1065 objects misplaced (3.756%); 213 B/s, 0 objects/s recovering 2021-02-10T05:00:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:47 smithi164 conmon[29288]: cluster 2021-02-10T05:00:46.677458 2021-02-10T05:00:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:47 smithi164 conmon[29288]: +0000 mon.a (mon.0) 3142 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2021-02-10T05:00:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:47 smithi164 conmon[29288]: audit 2021-02 2021-02-10T05:00:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:47 smithi164 conmon[29288]: -10T05:00:46.685875+0000 mon.a (mon.0) 3143 : audit [INF] from='client.? 172.21.15.85:0/875318818' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2021-02-10T05:00:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:47 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T05:00:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:47 smithi164 conmon[29288]: 10T05:00:46.686072+0000 mon.a (mon.0) 3144 : cluster 2021-02-10T05:00:48.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:47 smithi164 conmon[29288]: [DBG] osdmap e1400: 8 total, 8 up, 8 in 2021-02-10T05:00:48.198 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:00:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:48 smithi164 conmon[29288]: cluster 2021-02-10T05:00:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:48 smithi164 conmon[29288]: 2021-02-10T05 2021-02-10T05:00:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:48 smithi164 conmon[29288]: :00:47.690768+0000 mon.a (mon.0) 3145 : cluster [DBG] osdmap e1401: 8 total, 8 up, 8 in 2021-02-10T05:00:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:48 smithi164 conmon[29288]: cluster 2021-02-10T05:00:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:48 smithi164 conmon[29288]: 2021-02-10T05:00: 2021-02-10T05:00:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:48 smithi164 conmon[29288]: 48.661676+0000 mon.a (mon.0) 3146 : cluster 2021-02-10T05:00:49.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:48 smithi164 conmon[29288]: [DBG] osdmap e1402: 8 total, 8 up, 8 in 2021-02-10T05:00:49.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:48 smithi085 conmon[29319]: cluster 2021-02-10T05:00:47.690768+0000 2021-02-10T05:00:49.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:48 smithi085 conmon[29319]: mon.a (mon.0) 3145 : cluster [DBG] osdmap e1401: 8 total, 8 up, 8 in 2021-02-10T05:00:49.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:48 smithi085 conmon[29319]: cluster 2021-02-10T05:00:48.661676+0000 mon.a (mon.0) 3146 : cluster [DBG] osdmap e1402: 8 total, 8 up, 8 in 2021-02-10T05:00:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:48 smithi085 conmon[35961]: cluster 2021-02-10T05:00:47.690768+0000 mon.a (mon.0) 2021-02-10T05:00:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:48 smithi085 conmon[35961]: 3145 : cluster [DBG] osdmap e1401: 8 total, 8 up, 8 in 2021-02-10T05:00:49.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:48 smithi085 conmon[35961]: cluster 2021-02-10T05:00:48.661676+0000 mon.a (mon.0) 3146 : cluster [DBG] osdmap e1402: 8 total, 8 up, 8 in 2021-02-10T05:00:49.895 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 05:00:49 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T05:00:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:49 smithi164 conmon[29288]: cluster 2021-02-10T05:00:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:49 smithi164 conmon[29288]: 2021-02 2021-02-10T05:00:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:49 smithi164 conmon[29288]: -10T05:00:47.847213+0000 osd.1 (osd.1) 533 : cluster [DBG] 6.1 scrub ok 2021-02-10T05:00:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:49 smithi164 conmon[29288]: cluster 2021-02-10T05: 2021-02-10T05:00:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:49 smithi164 conmon[29288]: 00:48.589671+0000 mgr.y (mgr.14140) 1576 : cluster [DBG] pgmap v2566: 172 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 170 active+clean; 25 KiB data, 395 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 38 op/s; 7/1068 objects misplaced (0.655%); 52 B/s, 0 objects/s recovering 2021-02-10T05:00:49.895 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:49 smithi164 conmon[29288]: cluster 2021-02-10T05 2021-02-10T05:00:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:49 smithi164 conmon[29288]: :00:48.688016+0000 mon.a (mon.0) 2021-02-10T05:00:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:49 smithi164 conmon[29288]: 3147 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2021-02-10T05:00:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:49 smithi164 conmon[29288]: cluster 2021-02-10T05:00:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:49 smithi164 conmon[29288]: 2021-02-10T05:00:48. 2021-02-10T05:00:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:49 smithi164 conmon[29288]: 713650+0000 osd.6 (osd.6) 204 : cluster [DBG] 2021-02-10T05:00:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:49 smithi164 conmon[29288]: 2.11 scrub ok 2021-02-10T05:00:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:49 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T05:00:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:49 smithi164 conmon[29288]: 10T05:00:49.671358+0000 mon.a (mon. 2021-02-10T05:00:49.896 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:49 smithi164 conmon[29288]: 0) 3148 : cluster [DBG] osdmap e1403: 8 total, 8 up, 8 in 2021-02-10T05:00:50.081 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:49 smithi085 conmon[29319]: cluster 2021-02-10T05:00:47.847213+0000 osd.1 (osd 2021-02-10T05:00:50.081 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:49 smithi085 conmon[29319]: .1) 533 : cluster [DBG] 6.1 scrub ok 2021-02-10T05:00:50.081 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:49 smithi085 conmon[29319]: cluster 2021-02-10T05:00:48.589671+0000 mgr.y (mgr.14140) 1576 : cluster [DBG] pgmap v2566: 172 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 170 active+clean; 25 KiB data, 395 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 38 op/s; 7/1068 objects misplaced (0.655%); 52 B/s, 0 objects/s recovering 2021-02-10T05:00:50.081 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:49 smithi085 conmon[29319]: cluster 2021-02-10T05:00:48.688016+0000 mon.a (mon.0) 3147 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2021-02-10T05:00:50.081 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:49 smithi085 conmon[29319]: cluster 2021-02-10T05:00:48.713650+0000 osd.6 (osd.6) 204 : cluster [DBG] 2.11 scrub ok 2021-02-10T05:00:50.082 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:49 smithi085 conmon[29319]: cluster 2021-02-10T05:00:49.671358+0000 mon.a (mon.0) 3148 : cluster [DBG] osdmap e1403: 8 total, 8 up, 8 in 2021-02-10T05:00:50.082 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:49 smithi085 conmon[35961]: cluster 2021-02-10T05:00:47.847213 2021-02-10T05:00:50.082 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:49 smithi085 conmon[35961]: +0000 osd.1 (osd.1) 533 : cluster [DBG] 6.1 scrub ok 2021-02-10T05:00:50.082 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:49 smithi085 conmon[35961]: cluster 2021-02-10T05:00:48.589671+0000 mgr.y (mgr.14140) 1576 : cluster [DBG] pgmap v2566: 172 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 170 active+clean; 25 KiB data, 395 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 38 op/s; 7/1068 objects misplaced (0.655%); 52 B/s, 0 objects/s recovering 2021-02-10T05:00:50.082 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:49 smithi085 conmon[35961]: cluster 2021-02-10T05:00:48.688016+0000 mon.a (mon.0 2021-02-10T05:00:50.082 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:49 smithi085 conmon[35961]: ) 3147 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2021-02-10T05:00:50.082 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:49 smithi085 conmon[35961]: cluster 2021-02-10T05:00:50.082 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:49 smithi085 conmon[35961]: 2021- 2021-02-10T05:00:50.082 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:49 smithi085 conmon[35961]: 02-10T05:00:48 2021-02-10T05:00:50.082 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:49 smithi085 conmon[35961]: .713650+0000 osd.6 (osd 2021-02-10T05:00:50.082 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:49 smithi085 conmon[35961]: .6) 204 : cluster [DBG] 2021-02-10T05:00:50.083 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:49 smithi085 conmon[35961]: 2.11 scrub ok 2021-02-10T05:00:50.083 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:49 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T05:00:50.083 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:49 smithi085 conmon[35961]: -10T05:00:49.671358+0000 mon.a (mon. 2021-02-10T05:00:50.083 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:49 smithi085 conmon[35961]: 0) 3148 : cluster [DBG] osdmap e1403: 8 total, 8 up, 8 in 2021-02-10T05:00:50.763 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.2 to 1 (10000) 2021-02-10T05:00:51.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:50 smithi164 conmon[29288]: cluster 2021-02-10T05:00:49.493362+0000 osd.7 ( 2021-02-10T05:00:51.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:50 smithi164 conmon[29288]: osd.7) 114 : cluster [DBG] 6.3 deep-scrub ok 2021-02-10T05:00:51.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:50 smithi164 conmon[29288]: cluster 2021-02-10T05:00:49.556591+0000 osd.3 (osd.3) 61 : cluster [DBG] 6.1b deep-scrub ok 2021-02-10T05:00:51.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:50 smithi164 conmon[29288]: audit 2021-02-10T05:00:51.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:50 smithi164 conmon[29288]: 2021-02-10T05:00:49.635748+0000 mgr.y (mgr.14140) 1577 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:00:51.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:50 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T05:00:51.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:50 smithi164 conmon[29288]: 05:00:49.990302+0000 osd.7 (osd. 2021-02-10T05:00:51.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:50 smithi164 conmon[29288]: 7) 115 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:00:51.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:50 smithi164 conmon[29288]: cluster 2021- 2021-02-10T05:00:51.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:50 smithi164 conmon[29288]: 02-10T05:00:49.991344+0000 2021-02-10T05:00:51.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:50 smithi164 conmon[29288]: osd.7 (osd.7) 116 : cluster [DBG] 2021-02-10T05:00:51.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:50 smithi164 conmon[29288]: purged_snaps scrub ok 2021-02-10T05:00:51.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:50 smithi164 conmon[29288]: audit 2021- 2021-02-10T05:00:51.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:50 smithi164 conmon[29288]: 02-10T05:00:50.640105+0000 2021-02-10T05:00:51.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:50 smithi164 conmon[29288]: mon.c (mon.1) 178 : audit [INF] from='client.? 172.21.15.85:0/2999119091' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-02-10T05:00:51.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:50 smithi164 conmon[29288]: 2021-02-10T05:00:51.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:50 smithi164 conmon[29288]: audit 2021-02-10T05: 2021-02-10T05:00:51.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:50 smithi164 conmon[29288]: 00:50.640417+0000 mon.a (mon.0) 3149 2021-02-10T05:00:51.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:50 smithi164 conmon[29288]: : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-02-10T05:00:51.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[29319]: cluster 2021-02-10T05:00:49.493362+0000 osd.7 (osd.7) 114 : cluster [DBG] 6.3 deep-scrub ok 2021-02-10T05:00:51.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[29319]: cluster 2021-02-10T05:00:49.556591+0000 osd.3 (osd.3) 61 : cluster [DBG] 6.1b deep-scrub ok 2021-02-10T05:00:51.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[29319]: audit 2021-02-10T05:00:49.635748+0000 mgr.y (mgr.14140) 1577 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:00:51.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[29319]: cluster 2021-02-10T05:00:49.990302+0000 osd.7 (osd.7) 115 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:00:51.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[29319]: cluster 2021-02-10T05:00:49.991344+0000 osd.7 (osd.7) 116 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:00:51.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[29319]: audit 2021-02-10T05:00:50.640105+0000 mon.c (mon.1) 178 : audit [INF] from='client.? 172.21.15.85:0/2999119091' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-02-10T05:00:51.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[29319]: audit 2021-02-10T05:00:50.640417+0000 mon.a (mon.0) 3149 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-02-10T05:00:51.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[35961]: cluster 2021-02-10T05:00:51.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[35961]: 2021-02- 2021-02-10T05:00:51.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[35961]: 10T05:00:49.493362+0000 osd.7 (osd.7) 114 : cluster [DBG] 6.3 deep-scrub ok 2021-02-10T05:00:51.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[35961]: cluster 2021-02-10T05:00:51.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[35961]: 2021-02-10T05:00:51.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[35961]: 2021 2021-02-10T05:00:51.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[35961]: - 2021-02-10T05:00:51.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[35961]: 02-10T05:00:49 2021-02-10T05:00:51.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[35961]: .556591+0000 osd.3 (osd.3) 61 : cluster [DBG] 6.1b deep-scrub ok 2021-02-10T05:00:51.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[35961]: audit 2021-02-10T05:00:51.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T05:00:51.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[35961]: 05:00:49.635748+0000 mgr.y (mgr 2021-02-10T05:00:51.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[35961]: .14140) 1577 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:00:51.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[35961]: cluster 2021-02-10T05:00:51.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[35961]: 2021-02-10T05:00:49 2021-02-10T05:00:51.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[35961]: .990302+0000 osd.7 (osd.7) 115 2021-02-10T05:00:51.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[35961]: : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:00:51.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[35961]: cluster 2021 2021-02-10T05:00:51.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[35961]: -02-10T05:00:49.991344 2021-02-10T05:00:51.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[35961]: +0000 osd.7 (osd.7) 116 : cluster [DBG] 2021-02-10T05:00:51.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[35961]: purged_snaps scrub ok 2021-02-10T05:00:51.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[35961]: audit 2021-02-10T05:00:50.640105+0000 2021-02-10T05:00:51.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[35961]: mon.c (mon.1) 178 : audit [INF] from='client.? 172.21.15.85:0/2999119091' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-02-10T05:00:51.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[35961]: audit 2021-02-10T05:00:50.640417 2021-02-10T05:00:51.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:50 smithi085 conmon[35961]: +0000 mon.a (mon.0) 3149 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-02-10T05:00:51.634 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd primary-affinity 2 1 2021-02-10T05:00:51.860 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:00:51.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[29319]: cluster 2021-02-10T05:00:50.180242+0000 osd.0 (osd.0) 2021-02-10T05:00:51.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[29319]: 22 : cluster [DBG] 1.2 scrub ok 2021-02-10T05:00:51.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[29319]: cluster 2021-02-10T05:00:50.391901+0000 osd.4 (osd.4) 2689 : cluster [DBG] 2.b scrub ok 2021-02-10T05:00:51.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[29319]: cluster 2021-02-10T05:00:50.442026+0000 osd.7 (osd.7) 117 : cluster [DBG] 5.4 scrub ok 2021-02-10T05:00:51.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[29319]: cluster 2021-02-10T05:00:50.581039+0000 osd.3 (osd.3) 62 : cluster [DBG] 5.10 deep-scrub ok 2021-02-10T05:00:51.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[29319]: cluster 2021-02-10T05:00:50.590234+0000 mgr.y (mgr.14140) 1578 : cluster [DBG] pgmap v2569: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 255 B/s wr, 77 op/s; 7/1068 objects misplaced (0.655%); 0 B/s, 0 objects/s recovering 2021-02-10T05:00:51.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[29319]: audit 2021-02-10T05:00:50.757077+0000 mon.a (mon.0) 3150 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2021-02-10T05:00:51.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[29319]: cluster 2021-02-10T05:00:50.757150+0000 mon.a (mon.0) 3151 : cluster [DBG] osdmap e1404: 8 total, 8 up, 8 in 2021-02-10T05:00:51.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[29319]: cluster 2021-02-10T05:00:50.915673+0000 osd.1 (osd.1) 534 : cluster [DBG] 5.19 scrub ok 2021-02-10T05:00:51.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[29319]: cluster 2021-02-10T05:00:51.171588+0000 osd.5 (osd.5) 218 : cluster [DBG] 2.3 scrub ok 2021-02-10T05:00:51.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T05:00:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[35961]: -10T05:00:50.180242+0000 osd.0 (osd.0) 22 : cluster [DBG] 1.2 scrub ok 2021-02-10T05:00:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[35961]: cluster 2021-02-10T05:00:50.391901 2021-02-10T05:00:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2689 : cluster [DBG] 2021-02-10T05:00:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[35961]: 2.b scrub ok 2021-02-10T05:00:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[35961]: cluster 2021-02-10T05 2021-02-10T05:00:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[35961]: :00:50.442026+0000 osd.7 (osd.7) 2021-02-10T05:00:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[35961]: 117 : cluster [DBG] 5.4 scrub ok 2021-02-10T05:00:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T05:00:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[35961]: -10T05:00:50.581039+0000 2021-02-10T05:00:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[35961]: osd.3 (osd.3) 62 : cluster 2021-02-10T05:00:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[35961]: [DBG] 5.10 deep-scrub ok 2021-02-10T05:00:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[35961]: cluster 2021-02-10T05 2021-02-10T05:00:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[35961]: :00:50.590234+0000 mgr.y (mgr.14140 2021-02-10T05:00:51.943 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[35961]: ) 1578 : cluster [DBG] pgmap v2569: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 255 B/s wr, 77 op/s; 7/1068 objects misplaced (0.655%); 0 B/s, 0 objects/s recovering 2021-02-10T05:00:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[35961]: audit 2021-02-10T05 2021-02-10T05:00:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[35961]: :00:50.757077+0000 mon.a (mon. 2021-02-10T05:00:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[35961]: 0) 3150 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2021-02-10T05:00:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T05:00:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[35961]: 05:00:50.757150+0000 mon.a (mon 2021-02-10T05:00:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[35961]: .0) 3151 : cluster [DBG] osdmap e1404: 8 total, 8 up, 8 in 2021-02-10T05:00:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T05:00:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[35961]: 10T05:00:50.915673+0000 osd.1 2021-02-10T05:00:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[35961]: (osd 2021-02-10T05:00:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[35961]: .1) 534 : cluster 2021-02-10T05:00:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[35961]: [DBG] 5.19 scrub ok 2021-02-10T05:00:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[35961]: cluster 2021-02-10T05: 2021-02-10T05:00:51.944 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[35961]: 00:51.171588+0000 osd.5 (osd.5) 2021-02-10T05:00:51.945 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:51 smithi085 conmon[35961]: 218 : cluster [DBG] 2.3 scrub ok 2021-02-10T05:00:52.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:51 smithi164 conmon[29288]: cluster 2021-02-10T05:00:50.180242 2021-02-10T05:00:52.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:51 smithi164 conmon[29288]: +0000 osd.0 (osd.0) 22 : cluster [DBG] 1.2 scrub ok 2021-02-10T05:00:52.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:51 smithi164 conmon[29288]: cluster 2021-02-10T05:00:50.391901+0000 osd.4 (osd. 2021-02-10T05:00:52.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:51 smithi164 conmon[29288]: 4) 2689 : cluster [DBG] 2.b scrub ok 2021-02-10T05:00:52.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:51 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T05:00:52.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:51 smithi164 conmon[29288]: -10T05:00:50.442026+0000 osd.7 (osd.7) 117 : cluster [DBG] 5.4 scrub ok 2021-02-10T05:00:52.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:51 smithi164 conmon[29288]: cluster 2021-02-10T05:00:50.581039 2021-02-10T05:00:52.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:51 smithi164 conmon[29288]: +0000 osd.3 (osd.3) 62 : cluster [DBG] 2021-02-10T05:00:52.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:51 smithi164 conmon[29288]: 5.10 deep-scrub ok 2021-02-10T05:00:52.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:51 smithi164 conmon[29288]: cluster 2021-02-10T05:00:50. 2021-02-10T05:00:52.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:51 smithi164 conmon[29288]: 590234+0000 mgr.y (mgr.14140) 1578 : cluster [DBG] 2021-02-10T05:00:52.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:51 smithi164 conmon[29288]: pgmap v2569: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 255 B/s wr, 77 op/s; 7/1068 objects misplaced (0.655%); 0 B/s, 0 objects/s recovering 2021-02-10T05:00:52.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:51 smithi164 conmon[29288]: audit 2021-02-10T05:00:50.757077+0000 2021-02-10T05:00:52.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:51 smithi164 conmon[29288]: mon.a (mon.0) 3150 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2021-02-10T05:00:52.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:51 smithi164 conmon[29288]: cluster 2021- 2021-02-10T05:00:52.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:51 smithi164 conmon[29288]: 02-10T05:00:50.757150+0000 mon.a 2021-02-10T05:00:52.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:51 smithi164 conmon[29288]: (mon.0) 3151 : cluster [DBG] osdmap e1404: 8 total, 8 up, 8 in 2021-02-10T05:00:52.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:51 smithi164 conmon[29288]: cluster 2021- 2021-02-10T05:00:52.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:51 smithi164 conmon[29288]: 02-10T05:00:50.915673+0000 osd.1 2021-02-10T05:00:52.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:51 smithi164 conmon[29288]: (osd.1) 534 : cluster [DBG] 5.19 scrub ok 2021-02-10T05:00:52.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:51 smithi164 conmon[29288]: cluster 2021 2021-02-10T05:00:52.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:51 smithi164 conmon[29288]: -02-10T05:00:51.171588 2021-02-10T05:00:52.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:51 smithi164 conmon[29288]: +0000 osd.5 (osd.5) 218 2021-02-10T05:00:52.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:51 smithi164 conmon[29288]: : cluster [DBG] 2.3 scrub ok 2021-02-10T05:00:53.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:52 smithi164 conmon[29288]: cluster 2021-02-10T05:00:51.131039+0000 2021-02-10T05:00:53.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:52 smithi164 conmon[29288]: osd.2 (osd.2) 2063 : cluster [DBG] 4.7 scrub ok 2021-02-10T05:00:53.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:52 smithi164 conmon[29288]: cluster 2021-02-10T05: 2021-02-10T05:00:53.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:52 smithi164 conmon[29288]: 00:51.405224+0000 osd.4 (osd.4) 2690 : cluster [DBG] 2021-02-10T05:00:53.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:52 smithi164 conmon[29288]: 2.25 scrub ok 2021-02-10T05:00:53.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:52 smithi164 conmon[29288]: cluster 2021- 2021-02-10T05:00:53.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:52 smithi164 conmon[29288]: 02-10T05:00:51.792109+0000 2021-02-10T05:00:53.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:52 smithi164 conmon[29288]: osd.6 (osd.6) 205 : cluster [DBG] 2021-02-10T05:00:53.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:52 smithi164 conmon[29288]: 2.1a scrub ok 2021-02-10T05:00:53.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:52 smithi164 conmon[29288]: cluster 2021-02-10T05:00:52. 2021-02-10T05:00:53.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:52 smithi164 conmon[29288]: 153063+0000 osd.5 (osd.5) 219 : cluster 2021-02-10T05:00:53.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:52 smithi164 conmon[29288]: [DBG] 6.24 scrub ok 2021-02-10T05:00:53.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:52 smithi085 conmon[29319]: cluster 2021-02-10T05:00:51.131039 2021-02-10T05:00:53.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:52 smithi085 conmon[29319]: +0000 osd.2 (osd.2) 2063 : cluster [DBG] 4.7 scrub ok 2021-02-10T05:00:53.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:52 smithi085 conmon[29319]: cluster 2021-02-10T05:00:51.405224+0000 osd.4 (osd.4) 2690 : cluster [DBG] 2.25 scrub ok 2021-02-10T05:00:53.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:52 smithi085 conmon[29319]: cluster 2021-02-10T05:00:51.792109+0000 osd.6 (osd.6) 205 : cluster [DBG] 2.1a scrub ok 2021-02-10T05:00:53.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:52 smithi085 conmon[29319]: cluster 2021-02-10T05:00:53.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:52 smithi085 conmon[29319]: 2021-02-10T05:00:52.153063+0000 osd.5 (osd.5) 219 : cluster [DBG] 6.24 scrub ok 2021-02-10T05:00:53.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:52 smithi085 conmon[35961]: cluster 2021-02-10T05:00:51.131039+0000 2021-02-10T05:00:53.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:52 smithi085 conmon[35961]: osd.2 (osd.2) 2063 : cluster [DBG] 4.7 scrub ok 2021-02-10T05:00:53.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:52 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T05:00:53.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:52 smithi085 conmon[35961]: -10T05:00:51.405224+0000 osd.4 (osd 2021-02-10T05:00:53.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:52 smithi085 conmon[35961]: .4) 2690 : cluster [DBG] 2021-02-10T05:00:53.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:52 smithi085 conmon[35961]: 2.25 scrub ok 2021-02-10T05:00:53.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:52 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T05:00:53.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:52 smithi085 conmon[35961]: 05:00:51.792109+0000 osd.6 (osd 2021-02-10T05:00:53.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:52 smithi085 conmon[35961]: .6) 205 : cluster [DBG] 2021-02-10T05:00:53.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:52 smithi085 conmon[35961]: 2.1a scrub ok 2021-02-10T05:00:53.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:52 smithi085 conmon[35961]: cluster 2021 2021-02-10T05:00:53.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:52 smithi085 conmon[35961]: -02-10T05:00:52.153063+0000 2021-02-10T05:00:53.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:52 smithi085 conmon[35961]: osd.5 (osd.5) 219 : cluster 2021-02-10T05:00:53.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:52 smithi085 conmon[35961]: [DBG] 6.24 scrub ok 2021-02-10T05:00:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:53 smithi164 conmon[29288]: cluster 2021-02-10T05:00:52.231288+0000 osd.0 ( 2021-02-10T05:00:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:53 smithi164 conmon[29288]: osd.0) 23 : cluster [DBG] 3.6 deep-scrub ok 2021-02-10T05:00:54.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:53 smithi164 conmon[29288]: cluster 2021-02-10T05:00:52.428340+0000 osd.4 (osd.4) 2691 : cluster [DBG] 3.9 scrub ok 2021-02-10T05:00:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:53 smithi164 conmon[29288]: cluster 2021-02-10T05:00:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:53 smithi164 conmon[29288]: 2021-02-10T05:00:52.590776+0000 mgr.y (mgr.14140) 1579 : cluster [DBG] pgmap v2571: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 50 op/s; 7/1074 objects misplaced (0.652%) 2021-02-10T05:00:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:53 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T05:00:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:53 smithi164 conmon[29288]: 05:00:53.511387+0000 mon.a (mon 2021-02-10T05:00:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:53 smithi164 conmon[29288]: .0) 3152 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T05:00:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:53 smithi164 conmon[29288]: cluster 2021-02-10T05 2021-02-10T05:00:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:53 smithi164 conmon[29288]: :00:53.664696+0000 mon.a (mon.0) 3153 : 2021-02-10T05:00:54.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:53 smithi164 conmon[29288]: cluster [DBG] osdmap e1405: 8 total, 8 up, 8 in 2021-02-10T05:00:54.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:53 smithi085 conmon[29319]: cluster 2021-02-10T05:00:52.231288+0000 osd.0 (osd.0) 23 : cluster [DBG] 3.6 deep-scrub ok 2021-02-10T05:00:54.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:53 smithi085 conmon[29319]: cluster 2021-02-10T05:00:52.428340+0000 osd.4 (osd.4) 2691 : cluster [DBG] 3.9 scrub ok 2021-02-10T05:00:54.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:53 smithi085 conmon[29319]: cluster 2021-02-10T05:00:52.590776+0000 mgr.y (mgr.14140) 1579 : cluster [DBG] pgmap v2571: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 50 op/s; 7/1074 objects misplaced (0.652%) 2021-02-10T05:00:54.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:53 smithi085 conmon[29319]: audit 2021-02-10T05:00:53.511387+0000 mon.a (mon.0) 3152 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T05:00:54.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:53 smithi085 conmon[29319]: cluster 2021-02-10T05:00:53.664696+0000 mon.a (mon.0) 3153 : cluster [DBG] osdmap e1405: 8 total, 8 up, 8 in 2021-02-10T05:00:54.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:53 smithi085 conmon[35961]: cluster 2021-02-10T05:00:52.231288+0000 osd.0 (osd.0) 23 : cluster [DBG] 3.6 deep-scrub ok 2021-02-10T05:00:54.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:53 smithi085 conmon[35961]: cluster 2021-02-10T05:00:52.428340+0000 osd.4 (osd.4) 2691 : cluster [DBG] 3.9 scrub ok 2021-02-10T05:00:54.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:53 smithi085 conmon[35961]: cluster 2021-02-10T05:00:52.590776+0000 mgr.y (mgr.14140) 1579 : cluster [DBG] pgmap v2571: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 50 op/s; 7/1074 objects misplaced (0.652%) 2021-02-10T05:00:54.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:53 smithi085 conmon[35961]: audit 2021-02-10T05:00:53.511387+0000 mon.a (mon.0) 3152 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T05:00:54.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:53 smithi085 conmon[35961]: cluster 2021-02-10T05:00:53.664696+0000 mon.a (mon.0) 3153 : cluster [DBG] osdmap e1405: 8 total, 8 up, 8 in 2021-02-10T05:00:54.777 INFO:teuthology.orchestra.run.smithi085.stderr:set osd.2 primary-affinity to 1 (8655362) 2021-02-10T05:00:54.798 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.7 dump_blocked_ops 2021-02-10T05:00:55.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:54 smithi164 conmon[29288]: cluster 2021-02-10T05:00:53. 2021-02-10T05:00:55.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:54 smithi164 conmon[29288]: 237453+0000 osd.0 (osd.0) 24 : cluster [DBG] 3.1 scrub ok 2021-02-10T05:00:55.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:54 smithi164 conmon[29288]: cephadm 2021-02-10T05:00:53.512030+0000 mgr.y (mgr.14140) 1580 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T05:00:55.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:54 smithi164 conmon[29288]: cluster 2021-02-10T05: 2021-02-10T05:00:55.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:54 smithi164 conmon[29288]: 00:53.520901+0000 osd.7 2021-02-10T05:00:55.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:54 smithi164 conmon[29288]: (osd.7 2021-02-10T05:00:55.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:54 smithi164 conmon[29288]: ) 118 : cluster [DBG] 6.12 scrub ok 2021-02-10T05:00:55.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:54 smithi164 conmon[29288]: audit 2021 2021-02-10T05:00:55.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:54 smithi164 conmon[29288]: -02-10T05:00:53.882916+0000 2021-02-10T05:00:55.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:54 smithi164 conmon[29288]: mon.a (mon.0) 3154 : 2021-02-10T05:00:55.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:54 smithi164 conmon[29288]: audit [INF] from='client.? 172.21.15.85:0/3926941255' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2021-02-10T05:00:55.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:54 smithi164 conmon[29288]: cluster 2021-02-10T05:00:55.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:54 smithi164 conmon[29288]: 2021-02-10T05 2021-02-10T05:00:55.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:54 smithi164 conmon[29288]: :00:53.940437+0000 osd.1 2021-02-10T05:00:55.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:54 smithi164 conmon[29288]: (osd.1) 535 : cluster 2021-02-10T05:00:55.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:54 smithi164 conmon[29288]: [DBG] 4.2 scrub ok 2021-02-10T05:00:55.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:54 smithi164 conmon[29288]: audit 2021-02-10T05:00:55.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:54 smithi164 conmon[29288]: 2021-02-10T05: 2021-02-10T05:00:55.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:54 smithi164 conmon[29288]: 00:54.349455+0000 mon.a ( 2021-02-10T05:00:55.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:54 smithi164 conmon[29288]: mon.0) 3155 : audit [INF] 2021-02-10T05:00:55.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:54 smithi164 conmon[29288]: from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T05:00:55.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[35961]: cluster 2021-02-10T05:00:53.237453+0000 2021-02-10T05:00:55.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[35961]: osd.0 (osd.0) 24 : cluster [DBG] 3.1 scrub ok 2021-02-10T05:00:55.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[35961]: cephadm 2021-02-10T 2021-02-10T05:00:55.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[35961]: 05:00:53.512030+0000 mgr.y (mgr.14140) 1580 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T05:00:55.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[35961]: cluster 2021-02-10T05:00:55.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[35961]: 2021-02-10T05:00:53.520901 2021-02-10T05:00:55.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[35961]: +0000 osd.7 (osd.7) 118 : cluster [DBG] 2021-02-10T05:00:55.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[35961]: 6.12 scrub ok 2021-02-10T05:00:55.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[35961]: audit 2021-02-10T05:00:55.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[35961]: 2021-02-10T05:00:53.882916+0000 2021-02-10T05:00:55.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[35961]: mon.a (mon.0) 3154 : audit [INF] 2021-02-10T05:00:55.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/3926941255' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2021-02-10T05:00:55.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[35961]: cluster 2021- 2021-02-10T05:00:55.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[35961]: 02-10T05:00:53.940437+0000 2021-02-10T05:00:55.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[35961]: osd.1 (osd.1) 535 : cluster [DBG] 4.2 scrub ok 2021-02-10T05:00:55.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[35961]: 2021-02-10T05:00:55.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[35961]: audit 2021-02-10T05: 2021-02-10T05:00:55.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[35961]: 00:54.349455+0000 mon.a (mon.0 2021-02-10T05:00:55.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[35961]: ) 3155 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T05:00:55.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: cluster 2021-02-10T05:00:55.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: 2021-02-10T05:00:55.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: 2021 2021-02-10T05:00:55.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: - 2021-02-10T05:00:55.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: 02 2021-02-10T05:00:55.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: - 2021-02-10T05:00:55.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: 10 2021-02-10T05:00:55.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: T 2021-02-10T05:00:55.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: 05 2021-02-10T05:00:55.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: : 2021-02-10T05:00:55.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: 00 2021-02-10T05:00:55.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: : 2021-02-10T05:00:55.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: 53 2021-02-10T05:00:55.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: . 2021-02-10T05:00:55.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: 237453 2021-02-10T05:00:55.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: +0000 2021-02-10T05:00:55.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: 2021-02-10T05:00:55.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: osd.0 2021-02-10T05:00:55.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: ( 2021-02-10T05:00:55.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: osd 2021-02-10T05:00:55.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: . 2021-02-10T05:00:55.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: 0 2021-02-10T05:00:55.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: ) 2021-02-10T05:00:55.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: 24 2021-02-10T05:00:55.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: : 2021-02-10T05:00:55.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: cluster 2021-02-10T05:00:55.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: 2021-02-10T05:00:55.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: [DBG] 2021-02-10T05:00:55.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: 2021-02-10T05:00:55.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: 3.1 scrub ok 2021-02-10T05:00:55.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: 2021-02-10T05:00:55.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: cephadm 2021-02-10T05 2021-02-10T05:00:55.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: :00:53.512030+0000 mgr.y (mgr.14140) 1580 : cephadm [INF] Refreshing smithi164 facts 2021-02-10T05:00:55.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: cluster 2021- 2021-02-10T05:00:55.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: 02-10T05:00:53. 2021-02-10T05:00:55.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: 520901+0000 osd.7 (osd.7 2021-02-10T05:00:55.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: ) 118 : cluster [DBG] 6.12 scrub ok 2021-02-10T05:00:55.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: audit 2021 2021-02-10T05:00:55.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: -02-10T05:00:53 2021-02-10T05:00:55.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: .882916+0000 mon.a (mon.0 2021-02-10T05:00:55.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: ) 3154 : audit [INF] from='client.? 172.21.15.85:0/3926941255' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2021-02-10T05:00:55.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: cluster 2021 2021-02-10T05:00:55.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: -02-10T05:00: 2021-02-10T05:00:55.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: 53.940437+0000 osd.1 (osd 2021-02-10T05:00:55.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: .1) 535 : cluster [DBG] 2021-02-10T05:00:55.196 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: 4.2 scrub ok 2021-02-10T05:00:55.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: audit 2021 2021-02-10T05:00:55.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: -02-10T05:00:54 2021-02-10T05:00:55.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: .349455+0000 mon.a (mon.0 2021-02-10T05:00:55.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: ) 3155 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T05:00:55.197 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:54 smithi085 conmon[29319]: 2021-02-10T05:00:55.344 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T05:00:55.782 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 3 1 2021-02-10T05:00:55.988 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:00:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:55 smithi164 conmon[29288]: cluster 2021-02-10T05:00:54.273059+0000 2021-02-10T05:00:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:55 smithi164 conmon[29288]: osd.0 (osd.0) 25 : cluster [DBG] 4.b scrub ok 2021-02-10T05:00:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:55 smithi164 conmon[29288]: cluster 2021-02-10T05:00:54.587030+0000 osd.3 (osd.3) 63 : cluster [DBG] 6.11 deep-scrub ok 2021-02-10T05:00:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:55 smithi164 conmon[29288]: cluster 2021-02-10T05:00:54.591226+0000 mgr.y (mgr.14140) 1581 : cluster [DBG] pgmap v2573: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 7/1074 objects misplaced (0.652%) 2021-02-10T05:00:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:55 smithi164 conmon[29288]: audit 2021-02-10T05:00:54.766958+0000 mon.a (mon.0) 3156 : audit [INF] from='client.? 172.21.15.85:0/3926941255' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2021-02-10T05:00:56.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:55 smithi164 conmon[29288]: cluster 2021-02-10T05:00:54.767129+0000 mon.a (mon.0) 3157 : cluster [DBG] osdmap e1406: 8 total, 8 up, 8 in 2021-02-10T05:00:56.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:55 smithi085 conmon[29319]: cluster 2021-02-10T05:00:54.273059+0000 osd.0 (osd.0) 2021-02-10T05:00:56.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:55 smithi085 conmon[29319]: 25 : cluster [DBG] 4.b scrub ok 2021-02-10T05:00:56.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:55 smithi085 conmon[29319]: cluster 2021-02-10T05:00:54.587030+0000 osd.3 (osd.3) 63 : cluster [DBG] 6.11 deep-scrub ok 2021-02-10T05:00:56.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:55 smithi085 conmon[29319]: cluster 2021-02-10T05:00:54.591226+0000 mgr.y (mgr.14140) 1581 : cluster [DBG] pgmap v2573: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 7/1074 objects misplaced (0.652%) 2021-02-10T05:00:56.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:55 smithi085 conmon[29319]: audit 2021-02-10T05:00:54.766958+0000 mon.a 2021-02-10T05:00:56.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:55 smithi085 conmon[29319]: (mon.0) 3156 : audit [INF] from='client.? 172.21.15.85:0/3926941255' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2021-02-10T05:00:56.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:55 smithi085 conmon[29319]: cluster 2021-02-10T05:00:54.767129+0000 mon.a (mon. 2021-02-10T05:00:56.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:55 smithi085 conmon[29319]: 0) 3157 : cluster [DBG] osdmap e1406: 8 total, 8 up, 8 in 2021-02-10T05:00:56.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:55 smithi085 conmon[35961]: cluster 2021-02-10T05:00:54.273059 2021-02-10T05:00:56.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:55 smithi085 conmon[35961]: +0000 osd.0 (osd.0) 25 : cluster [DBG] 4.b scrub ok 2021-02-10T05:00:56.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:55 smithi085 conmon[35961]: cluster 2021-02-10T05:00:56.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:55 smithi085 conmon[35961]: 2021-02-10T05:00:54.587030+0000 osd.3 (osd.3) 63 : cluster 2021-02-10T05:00:56.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:55 smithi085 conmon[35961]: [DBG] 6.11 deep-scrub ok 2021-02-10T05:00:56.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:55 smithi085 conmon[35961]: cluster 2021-02-10T05:00:56.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:55 smithi085 conmon[35961]: 2021-02-10T05: 2021-02-10T05:00:56.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:55 smithi085 conmon[35961]: 00:54.591226+0000 mgr.y (mgr.14140) 1581 : 2021-02-10T05:00:56.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:55 smithi085 conmon[35961]: cluster [DBG] pgmap v2573: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 7/1074 objects misplaced (0.652%) 2021-02-10T05:00:56.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:55 smithi085 conmon[35961]: audit 2021-02 2021-02-10T05:00:56.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:55 smithi085 conmon[35961]: -10T05:00:54.766958+0000 mon.a 2021-02-10T05:00:56.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:55 smithi085 conmon[35961]: (mon.0) 3156 : audit [INF] 2021-02-10T05:00:56.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:55 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/3926941255' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2021-02-10T05:00:56.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:55 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T05:00:56.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:55 smithi085 conmon[35961]: -10T05:00:54.767129+0000 2021-02-10T05:00:56.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:55 smithi085 conmon[35961]: mon.a (mon.0) 3157 : cluster [DBG] 2021-02-10T05:00:56.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:55 smithi085 conmon[35961]: osdmap e1406: 8 total, 8 up, 8 in 2021-02-10T05:00:56.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:56 smithi085 conmon[29319]: cluster 2021-02-10T05:00:55.399042+0000 osd.4 (osd. 2021-02-10T05:00:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:56 smithi085 conmon[29319]: 4) 2692 : cluster [DBG] 4.9 scrub ok 2021-02-10T05:00:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:56 smithi085 conmon[29319]: cluster 2021-02-10T05:00:55.944387+0000 osd.1 (osd.1) 536 : cluster [DBG] 6.1d scrub ok 2021-02-10T05:00:56.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:56 smithi085 conmon[29319]: cluster 2021-02-10T05:00:56.111644+0000 osd.5 (osd.5) 220 : cluster [DBG] 2.16 scrub ok 2021-02-10T05:00:56.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:56 smithi085 conmon[35961]: cluster 2021-02-10T05:00:55.399042 2021-02-10T05:00:56.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:56 smithi085 conmon[35961]: +0000 osd.4 (osd.4) 2692 : cluster [DBG] 4.9 scrub ok 2021-02-10T05:00:56.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:56 smithi085 conmon[35961]: cluster 2021-02-10T05:00:55.944387+0000 2021-02-10T05:00:56.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:56 smithi085 conmon[35961]: osd.1 (osd.1) 536 : cluster [DBG] 6.1d scrub ok 2021-02-10T05:00:56.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:56 smithi085 conmon[35961]: cluster 2021-02-10T05:00:56.111644+0000 osd.5 (osd.5) 220 : 2021-02-10T05:00:56.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:56 smithi085 conmon[35961]: cluster [DBG] 2.16 scrub ok 2021-02-10T05:00:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:56 smithi164 conmon[29288]: cluster 2021-02-10T05:00:55.399042+0000 osd.4 (osd. 2021-02-10T05:00:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:56 smithi164 conmon[29288]: 4) 2692 : cluster [DBG] 4.9 scrub ok 2021-02-10T05:00:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:56 smithi164 conmon[29288]: cluster 2021-02-10T05:00:55.944387+0000 osd.1 (osd.1 2021-02-10T05:00:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:56 smithi164 conmon[29288]: ) 536 : cluster [DBG] 6.1d scrub ok 2021-02-10T05:00:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:56 smithi164 conmon[29288]: cluster 2021-02-10T05:00:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:56 smithi164 conmon[29288]: 2021-02-10T05:00:56.111644+0000 osd.5 (osd.5) 220 2021-02-10T05:00:57.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:56 smithi164 conmon[29288]: : cluster [DBG] 2.16 scrub ok 2021-02-10T05:00:57.780 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:57 smithi164 conmon[29288]: cluster 2021-02-10T05:00:56.290404+0000 osd.0 (osd.0) 26 2021-02-10T05:00:57.780 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:57 smithi164 conmon[29288]: : cluster [DBG] 4.6 scrub ok 2021-02-10T05:00:57.780 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:57 smithi164 conmon[29288]: cluster 2021-02-10T05:00:56.445271+0000 osd.4 (osd.4) 2693 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:00:57.780 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:57 smithi164 conmon[29288]: cluster 2021-02-10T05:00:56.446067+0000 osd.4 (osd.4) 2694 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:00:57.780 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:57 smithi164 conmon[29288]: cluster 2021-02-10T05:00:56.525204+0000 osd.7 (osd.7) 119 : cluster [DBG] 2.8 deep-scrub ok 2021-02-10T05:00:57.781 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:57 smithi164 conmon[29288]: cluster 2021-02-10T05:00:56.591737+0000 mgr.y (mgr.14140) 1582 : cluster [DBG] pgmap v2575: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 26 KiB data, 401 MiB used, 715 GiB / 715 GiB avail; 7/1083 objects misplaced (0.646%) 2021-02-10T05:00:57.781 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:57 smithi164 conmon[29288]: cluster 2021-02-10T05:00:56.752002+0000 osd.6 (osd.6) 206 : cluster [DBG] 2.14 deep-scrub ok 2021-02-10T05:00:57.781 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:57 smithi164 conmon[29288]: cluster 2021-02-10T05:00:56.778918+0000 mon.a (mon.0) 3158 : cluster [DBG] osdmap e1407: 8 total, 8 up, 8 in 2021-02-10T05:00:57.781 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:57 smithi164 conmon[29288]: cluster 2021-02-10T05:00:56 2021-02-10T05:00:57.781 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:57 smithi164 conmon[29288]: .876602+0000 osd.0 (osd.0) 27 : cluster 2021-02-10T05:00:57.781 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:57 smithi164 conmon[29288]: [DBG] purged_snaps scrub starts 2021-02-10T05:00:57.781 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:57 smithi164 conmon[29288]: cluster 2021-02-10T05 2021-02-10T05:00:57.781 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:57 smithi164 conmon[29288]: :00:56.879674+0000 osd.0 (osd.0 2021-02-10T05:00:57.781 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:57 smithi164 conmon[29288]: ) 28 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:00:58.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[29319]: cluster 2021-02-10T05:00: 2021-02-10T05:00:58.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[29319]: 56.290404+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.6 scrub ok 2021-02-10T05:00:58.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[29319]: cluster 2021-02-10T05:00:56.445271+0000 osd.4 (osd.4) 2693 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:00:58.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[29319]: cluster 2021-02-10T05:00:56.446067+0000 osd.4 (osd.4) 2694 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:00:58.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[29319]: cluster 2021-02-10T05:00:56.525204+0000 osd.7 (osd.7) 119 : 2021-02-10T05:00:58.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[29319]: cluster [DBG] 2.8 deep-scrub ok 2021-02-10T05:00:58.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T05:00:58.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[29319]: 10T05:00:56.591737+0000 mgr.y (mgr.14140) 1582 : cluster 2021-02-10T05:00:58.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[29319]: [DBG] pgmap v2575: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 26 KiB data, 401 MiB used, 715 GiB / 715 GiB avail; 7/1083 objects misplaced (0.646%) 2021-02-10T05:00:58.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T05:00:58.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[29319]: 10T05:00:56.752002+0000 osd.6 ( 2021-02-10T05:00:58.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[29319]: osd.6) 206 : cluster [DBG] 2.14 deep-scrub ok 2021-02-10T05:00:58.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[29319]: cluster 2021-02-10T05:00:58.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[29319]: 2021-02-10T05:00: 2021-02-10T05:00:58.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[29319]: 56.778918+0000 mon.a (mon.0) 3158 : 2021-02-10T05:00:58.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[29319]: cluster [DBG] osdmap e1407: 8 total, 8 up, 8 in 2021-02-10T05:00:58.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[29319]: cluster 2021-02-10T05:00:58.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[29319]: 2021-02-10T05:00:56.876602+0000 2021-02-10T05:00:58.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[29319]: osd.0 (osd.0) 27 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:00:58.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[29319]: 2021-02-10T05:00:58.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[29319]: cluster 2021-02-10T05 2021-02-10T05:00:58.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[29319]: :00:56.879674+0000 osd.0 (osd.0) 28 2021-02-10T05:00:58.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[29319]: : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:00:58.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[35961]: cluster 2021-02-10T05:00 2021-02-10T05:00:58.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[35961]: :56.290404+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.6 scrub ok 2021-02-10T05:00:58.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[35961]: cluster 2021-02-10T05:00:56.445271+0000 osd.4 (osd.4) 2693 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:00:58.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[35961]: cluster 2021-02-10T05:00:56.446067+0000 osd.4 (osd.4) 2694 : cluster 2021-02-10T05:00:58.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[35961]: [DBG] purged_snaps scrub ok 2021-02-10T05:00:58.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[35961]: cluster 2021-02-10T05:00:58.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[35961]: 2021-02-10T05:00:56 2021-02-10T05:00:58.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[35961]: .525204+0000 osd.7 (osd.7) 2021-02-10T05:00:58.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[35961]: 119 : cluster [DBG] 2.8 deep-scrub ok 2021-02-10T05:00:58.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[35961]: cluster 2021-02-10T05 2021-02-10T05:00:58.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[35961]: :00:56.591737+0000 mgr.y (mgr. 2021-02-10T05:00:58.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[35961]: 14140) 1582 : cluster [DBG] pgmap v2575: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 26 KiB data, 401 MiB used, 715 GiB / 715 GiB avail; 7/1083 objects misplaced (0.646%) 2021-02-10T05:00:58.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[35961]: 2021-02-10T05:00:58.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[35961]: cluster 2021-02-10T05:00:58.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[35961]: 2021-02-10T05:00:56.752002+0000 osd.6 (osd 2021-02-10T05:00:58.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[35961]: .6) 206 : cluster [DBG] 2.14 deep-scrub ok 2021-02-10T05:00:58.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T05:00:58.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[35961]: 10T05:00:56.778918+0000 mon.a (mon 2021-02-10T05:00:58.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[35961]: .0) 3158 : cluster [DBG] osdmap e1407: 8 total, 8 up, 8 in 2021-02-10T05:00:58.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T05:00:58.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[35961]: -10T05:00:56.876602+0000 osd.0 ( 2021-02-10T05:00:58.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[35961]: osd.0) 27 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:00:58.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T05:00:58.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[35961]: -10T05:00:56.879674+0000 osd.0 (osd.0 2021-02-10T05:00:58.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:57 smithi085 conmon[35961]: ) 28 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:00:58.674 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.3 to 1 (10000) 2021-02-10T05:00:59.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:58 smithi164 conmon[29288]: cluster 2021-02-10T05:00:59.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:58 smithi164 conmon[29288]: 2021-02- 2021-02-10T05:00:59.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:58 smithi164 conmon[29288]: 10T05:00:57.544592+0000 osd.7 (osd.7) 120 : cluster [DBG] 2.0 scrub ok 2021-02-10T05:00:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:58 smithi164 conmon[29288]: cluster 2021 2021-02-10T05:00:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:58 smithi164 conmon[29288]: -02-10T05:00:57.789932+0000 mon.a (mon.0) 3159 : cluster [DBG] osdmap e1408: 8 total, 8 up, 8 in 2021-02-10T05:00:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:58 smithi164 conmon[29288]: audit 2021- 2021-02-10T05:00:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:58 smithi164 conmon[29288]: 02-10T05:00:58.102449+0000 mon.a 2021-02-10T05:00:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:58 smithi164 conmon[29288]: (mon.0) 3160 : audit [INF] from='client.? 172.21.15.85:0/3917698872' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-02-10T05:00:59.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:58 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T05:00:59.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:58 smithi164 conmon[29288]: T05:00:58.137266+0000 osd.5 (osd.5) 221 : 2021-02-10T05:00:59.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:58 smithi164 conmon[29288]: cluster [DBG] 4.15 scrub ok 2021-02-10T05:00:59.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:58 smithi164 conmon[29288]: audit 2021-02 2021-02-10T05:00:59.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:58 smithi164 conmon[29288]: -10T05:00:58.668296+0000 mon.a (mon.0) 3161 : audit [INF] from='client.? 172.21.15.85:0/3917698872' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2021-02-10T05:00:59.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:58 smithi164 conmon[29288]: cluster 2021-02-10T05:00: 2021-02-10T05:00:59.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:58 smithi164 conmon[29288]: 58.668360+0000 mon.a (mon.0) 3162 : cluster [DBG] 2021-02-10T05:00:59.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:58 smithi164 conmon[29288]: osdmap e1409: 8 total, 8 up, 8 in 2021-02-10T05:00:59.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:58 smithi085 conmon[29319]: cluster 2021-02-10T05:00:57.544592 2021-02-10T05:00:59.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:58 smithi085 conmon[29319]: +0000 osd.7 (osd.7) 120 : cluster [DBG] 2.0 scrub ok 2021-02-10T05:00:59.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:58 smithi085 conmon[29319]: cluster 2021-02-10T05:00:57.789932+0000 mon.a (mon.0) 3159 : cluster [DBG] osdmap e1408: 8 total, 8 up, 8 in 2021-02-10T05:00:59.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:58 smithi085 conmon[29319]: audit 2021-02-10T05:00:58.102449+0000 mon.a (mon.0) 3160 : audit [INF] from='client.? 172.21.15.85:0/3917698872' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-02-10T05:00:59.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:58 smithi085 conmon[29319]: cluster 2021-02-10T05:00:58.137266+0000 osd.5 (osd.5) 221 : cluster [DBG] 4.15 scrub ok 2021-02-10T05:00:59.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:58 smithi085 conmon[29319]: audit 2021-02-10T05:00:58.668296+0000 mon.a (mon.0) 3161 : audit [INF] from='client.? 172.21.15.85:0/3917698872' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2021-02-10T05:00:59.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:58 smithi085 conmon[29319]: cluster 2021-02-10T05:00:58.668360+0000 mon.a (mon.0) 3162 : cluster [DBG] osdmap e1409: 8 total, 8 up, 8 in 2021-02-10T05:00:59.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:58 smithi085 conmon[35961]: cluster 2021-02-10T05:00 2021-02-10T05:00:59.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:58 smithi085 conmon[35961]: :57.544592+0000 osd.7 (osd.7) 120 : cluster [DBG] 2.0 scrub ok 2021-02-10T05:00:59.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:58 smithi085 conmon[35961]: cluster 2021- 2021-02-10T05:00:59.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:58 smithi085 conmon[35961]: 02-10T05:00:57 2021-02-10T05:00:59.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:58 smithi085 conmon[35961]: .789932+0000 mon.a (mon. 2021-02-10T05:00:59.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:58 smithi085 conmon[35961]: 0) 3159 : cluster [DBG] osdmap e1408: 8 total, 8 up, 8 in 2021-02-10T05:00:59.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:58 smithi085 conmon[35961]: 2021-02-10T05:00:59.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:58 smithi085 conmon[35961]: audit 2021-02-10T05:00:59.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:58 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T05:00:59.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:58 smithi085 conmon[35961]: 05:00:58.102449+0000 2021-02-10T05:00:59.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:58 smithi085 conmon[35961]: mon.a (mon.0) 2021-02-10T05:00:59.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:58 smithi085 conmon[35961]: 3160 : audit [INF] from='client.? 172.21.15.85:0/3917698872' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-02-10T05:00:59.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:58 smithi085 conmon[35961]: cluster 2021- 2021-02-10T05:00:59.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:58 smithi085 conmon[35961]: 02-10T05:00:58.137266+0000 2021-02-10T05:00:59.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:58 smithi085 conmon[35961]: osd.5 (osd.5) 221 : cluster [DBG] 4.15 scrub ok 2021-02-10T05:00:59.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:58 smithi085 conmon[35961]: 2021-02-10T05:00:59.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:58 smithi085 conmon[35961]: audit 2021-02-10T05:00: 2021-02-10T05:00:59.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:58 smithi085 conmon[35961]: 58.668296+0000 mon.a (mon.0) 3161 2021-02-10T05:00:59.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:58 smithi085 conmon[35961]: : audit [INF] from='client.? 172.21.15.85:0/3917698872' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2021-02-10T05:00:59.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:58 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T05:00:59.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:58 smithi085 conmon[35961]: T05:00:58.668360+0000 mon.a ( 2021-02-10T05:00:59.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:58 smithi085 conmon[35961]: mon.0) 3162 : cluster [DBG] osdmap e1409: 8 total, 8 up, 8 in 2021-02-10T05:00:59.636 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd primary-affinity 3 1 2021-02-10T05:00:59.881 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:00:59.941 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:59 smithi085 conmon[29319]: cluster 2021-02-10T05:00:58.592217+0000 mgr.y (mgr.14140) 1583 : cluster [DBG] pgmap v2578: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 26 KiB data, 401 MiB used, 715 GiB / 715 GiB avail; 7/1074 objects misplaced (0.652%) 2021-02-10T05:00:59.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:00:59 smithi085 conmon[29319]: cluster 2021-02-10T05:00:58.838390+0000 osd.6 (osd.6) 207 : cluster [DBG] 2.29 scrub ok 2021-02-10T05:00:59.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:59 smithi085 conmon[35961]: cluster 2021-02-10T05:00:58.592217+0000 mgr.y (mgr.14140) 1583 : cluster [DBG] pgmap v2578: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 26 KiB data, 401 MiB used, 715 GiB / 715 GiB avail; 7/1074 objects misplaced (0.652%) 2021-02-10T05:00:59.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:00:59 smithi085 conmon[35961]: cluster 2021-02-10T05:00:58.838390+0000 osd.6 (osd.6) 207 : cluster [DBG] 2.29 scrub ok 2021-02-10T05:01:00.144 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 05:00:59 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T05:01:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:59 smithi164 conmon[29288]: cluster 2021-02-10T05:00:58.592217+0000 mgr.y (mgr.14140) 1583 : cluster 2021-02-10T05:01:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:59 smithi164 conmon[29288]: [DBG] pgmap v2578: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 26 KiB data, 401 MiB used, 715 GiB / 715 GiB avail; 7/1074 objects misplaced (0.652%) 2021-02-10T05:01:00.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:00:59 smithi164 conmon[29288]: cluster 2021-02-10T05:00:58.838390+0000 osd.6 (osd.6) 207 : cluster [DBG] 2.29 scrub ok 2021-02-10T05:01:00.801 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:00 smithi085 conmon[29319]: cluster 2021-02-10T05:00:59.233482+0000 osd.2 (osd.2) 2021-02-10T05:01:00.802 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:00 smithi085 conmon[29319]: 2064 : cluster [DBG] 2.10 scrub ok 2021-02-10T05:01:00.802 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:00 smithi085 conmon[29319]: cluster 2021-02-10T05:00:59.306499+0000 osd.0 (osd.0) 29 : cluster [DBG] 6.9 scrub ok 2021-02-10T05:01:00.802 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:00 smithi085 conmon[29319]: cluster 2021-02-10T05:00:59.493588+0000 osd.7 (osd.7) 121 : cluster [DBG] 1.0 scrub ok 2021-02-10T05:01:00.802 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:00 smithi085 conmon[29319]: audit 2021-02-10T05:00:59.646003+0000 mgr.y (mgr.14140) 1584 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:01:00.802 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:00 smithi085 conmon[29319]: cluster 2021-02-10T05:00:59.792593+0000 osd.1 (osd.1) 537 : cluster [DBG] 4.12 scrub ok 2021-02-10T05:01:00.802 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:00 smithi085 conmon[29319]: cluster 2021-02-10T05:01:00.219491+0000 osd.3 (osd.3) 64 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:01:00.802 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:00 smithi085 conmon[29319]: cluster 2021-02-10T05:01:00.222449+0000 osd.3 (osd.3) 65 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:01:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: cluster 2021-02-10T05:01:01.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: 2021-02-10T05:01:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: 2021 2021-02-10T05:01:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: - 2021-02-10T05:01:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: 02 2021-02-10T05:01:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: - 2021-02-10T05:01:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: 10 2021-02-10T05:01:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: T 2021-02-10T05:01:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: 05 2021-02-10T05:01:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: : 2021-02-10T05:01:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: 00 2021-02-10T05:01:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: : 2021-02-10T05:01:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: 59 2021-02-10T05:01:01.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: . 2021-02-10T05:01:01.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: 233482 2021-02-10T05:01:01.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 2064 : cluster [DBG] 2.10 scrub ok 2021-02-10T05:01:01.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: 2021-02-10T05:01:01.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: cluster 2021-02-10T05: 2021-02-10T05:01:01.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: 00:59.306499+0000 osd.0 (osd.0) 29 : cluster [DBG] 6.9 scrub ok 2021-02-10T05:01:01.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: cluster 2021-02-10T05: 2021-02-10T05:01:01.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: 00:59.493588+0000 osd.7 (osd.7) 121 : 2021-02-10T05:01:01.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: cluster [DBG] 1.0 scrub ok 2021-02-10T05:01:01.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: audit 2021 2021-02-10T05:01:01.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: -02-10T05:00:59.646003+0000 2021-02-10T05:01:01.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: mgr.y (mgr.14140) 1584 : audit [DBG] 2021-02-10T05:01:01.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:01:01.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: cluster 2021-02-10T05:00 2021-02-10T05:01:01.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: :59.792593+0000 osd.1 (osd.1) 537 2021-02-10T05:01:01.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: : cluster [DBG] 4.12 scrub ok 2021-02-10T05:01:01.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: cluster 2021-02-10T05 2021-02-10T05:01:01.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: :01:00.219491+0000 osd.3 (osd.3 2021-02-10T05:01:01.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: ) 64 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:01:01.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T05:01:01.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: T05:01:00.222449+0000 osd.3 (osd. 2021-02-10T05:01:01.147 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:00 smithi164 conmon[29288]: 3) 65 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:01:01.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:00 smithi085 conmon[35961]: cluster 2021-02-10T05:00:59.233482 2021-02-10T05:01:01.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:00 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 2064 : cluster [DBG] 2.10 scrub ok 2021-02-10T05:01:01.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:00 smithi085 conmon[35961]: cluster 2021-02-10T05:00:59 2021-02-10T05:01:01.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:00 smithi085 conmon[35961]: .306499+0000 osd.0 (osd.0) 29 : cluster [DBG] 6.9 scrub ok 2021-02-10T05:01:01.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:00 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T05:01:01.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:00 smithi085 conmon[35961]: -10T05:00:59.493588+0000 osd.7 ( 2021-02-10T05:01:01.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:00 smithi085 conmon[35961]: osd.7) 121 : cluster [DBG] 1.0 scrub ok 2021-02-10T05:01:01.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:00 smithi085 conmon[35961]: audit 2021-02 2021-02-10T05:01:01.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:00 smithi085 conmon[35961]: -10T05:00:59.646003+0000 mgr.y (mgr. 2021-02-10T05:01:01.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:00 smithi085 conmon[35961]: 14140) 1584 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:01:01.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:00 smithi085 conmon[35961]: cluster 2021- 2021-02-10T05:01:01.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:00 smithi085 conmon[35961]: 02-10T05:00:59.792593+0000 osd.1 2021-02-10T05:01:01.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:00 smithi085 conmon[35961]: (osd.1) 537 : cluster [DBG] 4.12 scrub ok 2021-02-10T05:01:01.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:00 smithi085 conmon[35961]: 2021-02-10T05:01:01.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:00 smithi085 conmon[35961]: cluster 2021-02-10T05:01 2021-02-10T05:01:01.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:00 smithi085 conmon[35961]: :00.219491+0000 osd.3 (osd.3) 2021-02-10T05:01:01.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:00 smithi085 conmon[35961]: 64 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:01:01.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:00 smithi085 conmon[35961]: cluster 2021 2021-02-10T05:01:01.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:00 smithi085 conmon[35961]: -02-10T05:01:00.222449+0000 osd.3 (osd.3) 2021-02-10T05:01:01.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:00 smithi085 conmon[35961]: 65 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:01:01.804 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:01 smithi085 conmon[29319]: cluster 2021-02-10T05:01:00.592732+0000 2021-02-10T05:01:01.804 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:01 smithi085 conmon[29319]: mgr.y (mgr.14140) 1585 : cluster [DBG] pgmap v2580: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 401 MiB used, 715 GiB / 715 GiB avail; 7/1071 objects misplaced (0.654%) 2021-02-10T05:01:01.804 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:01 smithi085 conmon[29319]: cluster 2021-02-10T05:01:00.800784+0000 mon.a (mon.0) 3163 : cluster [DBG] osdmap e1410: 8 total, 8 up, 8 in 2021-02-10T05:01:01.804 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:01 smithi085 conmon[29319]: cluster 2021-02-10T05:01:01.220035+0000 osd.2 (osd.2) 2065 : cluster [DBG] 5.8 deep-scrub ok 2021-02-10T05:01:01.804 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:01 smithi085 conmon[35961]: cluster 2021-02-10T05:01:00.592732+0000 2021-02-10T05:01:01.804 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:01 smithi085 conmon[35961]: mgr.y (mgr.14140) 1585 : cluster [DBG] pgmap v2580: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 401 MiB used, 715 GiB / 715 GiB avail; 7/1071 objects misplaced (0.654%) 2021-02-10T05:01:01.805 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:01 smithi085 conmon[35961]: cluster 2021-02-10T05:01:00.800784+0000 mon.a (mon.0) 3163 : cluster [DBG] osdmap e1410: 8 total, 8 up, 8 in 2021-02-10T05:01:01.805 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:01 smithi085 conmon[35961]: cluster 2021-02-10T05:01:01.220035+0000 osd.2 (osd.2) 2065 : cluster [DBG] 5.8 deep-scrub ok 2021-02-10T05:01:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:01 smithi164 conmon[29288]: cluster 2021-02-10T05:01:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:01 smithi164 conmon[29288]: 2021-02-10T05:01:00 2021-02-10T05:01:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:01 smithi164 conmon[29288]: .592732+0000 mgr.y (mgr.14140) 1585 : cluster [DBG] pgmap v2580: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 401 MiB used, 715 GiB / 715 GiB avail; 7/1071 objects misplaced (0.654%) 2021-02-10T05:01:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:01 smithi164 conmon[29288]: cluster 2021-02-10T05:01:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:01 smithi164 conmon[29288]: 2021-02-10T05:01:00 2021-02-10T05:01:02.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:01 smithi164 conmon[29288]: .800784+0000 mon.a (mon.0) 3163 : 2021-02-10T05:01:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:01 smithi164 conmon[29288]: cluster [DBG] osdmap e1410: 8 total, 8 up, 8 in 2021-02-10T05:01:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:01 smithi164 conmon[29288]: cluster 2021-02-10T05:01:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:01 smithi164 conmon[29288]: 2021-02-10T05:01:01.220035 2021-02-10T05:01:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:01 smithi164 conmon[29288]: +0000 osd.2 (osd.2) 2065 : cluster [DBG] 2021-02-10T05:01:02.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:01 smithi164 conmon[29288]: 5.8 deep-scrub ok 2021-02-10T05:01:02.813 INFO:teuthology.orchestra.run.smithi085.stderr:set osd.3 primary-affinity to 1 (8655362) 2021-02-10T05:01:03.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:02 smithi164 conmon[29288]: cluster 2021-02-10T05:01:03.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:02 smithi164 conmon[29288]: 2021-02-10T05:01:03.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:02 smithi164 conmon[29288]: 2021-02-10T05:01:01.454360+0000 osd.4 (osd.4) 2695 : cluster 2021-02-10T05:01:03.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:02 smithi164 conmon[29288]: [DBG] 4.d scrub ok 2021-02-10T05:01:03.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:02 smithi164 conmon[29288]: cluster 2021- 2021-02-10T05:01:03.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:02 smithi164 conmon[29288]: 02-10T05:01:01.748099+0000 osd.6 (osd.6 2021-02-10T05:01:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:02 smithi164 conmon[29288]: ) 208 : cluster [DBG] 2.17 scrub ok 2021-02-10T05:01:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:02 smithi164 conmon[29288]: audit 2021-02 2021-02-10T05:01:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:02 smithi164 conmon[29288]: -10T05:01:01.971867+0000 mon.a (mon. 2021-02-10T05:01:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:02 smithi164 conmon[29288]: 0) 3164 : audit [INF] from='client.? 172.21.15.85:0/3209633411' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2021-02-10T05:01:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:02 smithi164 conmon[29288]: cluster 2021-02-10T05:01:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:02 smithi164 conmon[29288]: 2021-02-10T05:01:02.579739+0000 2021-02-10T05:01:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:02 smithi164 conmon[29288]: osd.1 (osd.1) 538 : cluster [DBG] 2021-02-10T05:01:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:02 smithi164 conmon[29288]: purged_snaps scrub starts 2021-02-10T05:01:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:02 smithi164 conmon[29288]: cluster 2021-02-10T05 2021-02-10T05:01:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:02 smithi164 conmon[29288]: :01:02.581093+0000 osd.1 (osd. 2021-02-10T05:01:03.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:02 smithi164 conmon[29288]: 1) 539 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:01:03.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:02 smithi085 conmon[29319]: cluster 2021-02-10T05:01:01.454360+0000 osd.4 2021-02-10T05:01:03.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:02 smithi085 conmon[29319]: (osd.4) 2695 : cluster [DBG] 4.d scrub ok 2021-02-10T05:01:03.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:02 smithi085 conmon[29319]: cluster 2021-02-10T05:01:01.748099+0000 osd.6 (osd.6) 208 : cluster [DBG] 2.17 scrub ok 2021-02-10T05:01:03.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:02 smithi085 conmon[29319]: audit 2021-02-10T05:01:01.971867+0000 mon.a (mon.0) 3164 : audit [INF] from='client.? 172.21.15.85:0/3209633411' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2021-02-10T05:01:03.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:02 smithi085 conmon[29319]: cluster 2021-02-10T05:01:02.579739+0000 osd.1 (osd.1) 538 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:01:03.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:02 smithi085 conmon[29319]: cluster 2021-02-10T05:01:02.581093+0000 osd.1 (osd.1) 539 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:01:03.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:02 smithi085 conmon[35961]: cluster 2021-02-10T05:01:01.454360+0000 osd.4 (osd.4) 2695 : cluster [DBG] 4.d scrub ok 2021-02-10T05:01:03.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:02 smithi085 conmon[35961]: cluster 2021-02-10T05:01:01.748099+0000 osd.6 (osd.6) 208 : cluster [DBG] 2.17 scrub ok 2021-02-10T05:01:03.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:02 smithi085 conmon[35961]: audit 2021-02-10T05:01:01.971867+0000 mon.a (mon.0) 3164 : audit [INF] from='client.? 172.21.15.85:0/3209633411' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2021-02-10T05:01:03.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:02 smithi085 conmon[35961]: cluster 2021-02-10T05:01:02.579739+0000 osd.1 (osd.1) 538 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:01:03.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:02 smithi085 conmon[35961]: cluster 2021-02-10T05:01:02.581093+0000 osd.1 (osd.1) 539 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:01:03.851 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 4 1 2021-02-10T05:01:04.066 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:01:04.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:03 smithi164 conmon[29288]: cluster 2021-02-10T05:01:02.348785+0000 osd.0 ( 2021-02-10T05:01:04.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:03 smithi164 conmon[29288]: osd.0) 30 : cluster [DBG] 4.16 scrub ok 2021-02-10T05:01:04.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:03 smithi164 conmon[29288]: cluster 2021-02-10T05:01:02.424679+0000 osd.7 (osd.7) 122 : cluster [DBG] 2.26 scrub ok 2021-02-10T05:01:04.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:03 smithi164 conmon[29288]: cluster 2021-02-10T05:01:02.593285+0000 2021-02-10T05:01:04.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:03 smithi164 conmon[29288]: mgr.y (mgr.14140) 1586 : cluster [DBG] pgmap v2582: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 401 MiB used, 715 GiB / 715 GiB avail; 7/1074 objects misplaced (0.652%) 2021-02-10T05:01:04.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:03 smithi164 conmon[29288]: audit 2021- 2021-02-10T05:01:04.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:03 smithi164 conmon[29288]: 02-10T05:01:02.809408+0000 mon.a (mon 2021-02-10T05:01:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:03 smithi164 conmon[29288]: .0) 3165 : audit [INF] from='client.? 172.21.15.85:0/3209633411' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2021-02-10T05:01:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:03 smithi164 conmon[29288]: cluster 2021-02-10T05:01:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:03 smithi164 conmon[29288]: 2021-02-10T05:01:02.809447 2021-02-10T05:01:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:03 smithi164 conmon[29288]: +0000 mon.a (mon.0) 3166 : cluster [DBG] 2021-02-10T05:01:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:03 smithi164 conmon[29288]: osdmap e1411: 8 total, 8 up, 8 in 2021-02-10T05:01:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:03 smithi164 conmon[29288]: cluster 2021-02-10T05 2021-02-10T05:01:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:03 smithi164 conmon[29288]: :01:03.129783+0000 osd.5 (osd.5) 2021-02-10T05:01:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:03 smithi164 conmon[29288]: 222 : cluster [DBG] 5.16 scrub ok 2021-02-10T05:01:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:03 smithi164 conmon[29288]: cluster 2021- 2021-02-10T05:01:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:03 smithi164 conmon[29288]: 02-10T05:01:03.622429+0000 osd.1 2021-02-10T05:01:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:03 smithi164 conmon[29288]: (osd.1) 540 : cluster [DBG] 5.12 deep-scrub ok 2021-02-10T05:01:04.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:03 smithi164 conmon[29288]: 2021-02-10T05:01:04.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[29319]: cluster 2021-02-10T05:01:02.348785 2021-02-10T05:01:04.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[29319]: +0000 osd.0 (osd.0) 30 : cluster [DBG] 4.16 scrub ok 2021-02-10T05:01:04.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[29319]: cluster 2021-02-10T05:01: 2021-02-10T05:01:04.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[29319]: 02.424679+0000 osd.7 (osd.7) 122 : cluster [DBG] 2.26 scrub ok 2021-02-10T05:01:04.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T05:01:04.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[29319]: -10T05:01:02.593285+0000 mgr.y (mgr 2021-02-10T05:01:04.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[29319]: .14140) 1586 : cluster [DBG] pgmap v2582: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 401 MiB used, 715 GiB / 715 GiB avail; 7/1074 objects misplaced (0.652%) 2021-02-10T05:01:04.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[29319]: audit 2021- 2021-02-10T05:01:04.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[29319]: 02-10T05:01:02.809408+0000 2021-02-10T05:01:04.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[29319]: mon.a (mon.0) 3165 : audit [INF] from='client.? 172.21.15.85:0/3209633411' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2021-02-10T05:01:04.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[29319]: 2021-02-10T05:01:04.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[29319]: cluster 2021- 2021-02-10T05:01:04.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[29319]: 02-10T05:01:02.809447+0000 mon.a (mon. 2021-02-10T05:01:04.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[29319]: 0) 3166 : cluster [DBG] osdmap e1411: 8 total, 8 up, 8 in 2021-02-10T05:01:04.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[29319]: cluster 2021-02-10T 2021-02-10T05:01:04.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[29319]: 05:01:03.129783+0000 osd.5 (osd 2021-02-10T05:01:04.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[29319]: .5) 222 : cluster [DBG] 5.16 scrub ok 2021-02-10T05:01:04.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T05:01:04.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[29319]: T05:01:03.622429+0000 osd.1 ( 2021-02-10T05:01:04.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[29319]: osd.1) 540 : cluster [DBG] 5.12 deep-scrub ok 2021-02-10T05:01:04.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[35961]: cluster 2021-02-10T05:01: 2021-02-10T05:01:04.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[35961]: 02.348785+0000 osd.0 (osd.0) 30 : cluster [DBG] 4.16 scrub ok 2021-02-10T05:01:04.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T05:01:04.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[35961]: 05:01:02.424679+0000 osd.7 (osd 2021-02-10T05:01:04.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[35961]: .7) 122 : cluster [DBG] 2.26 scrub ok 2021-02-10T05:01:04.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[35961]: cluster 2021-02-10T05:01:04.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[35961]: 2021 2021-02-10T05:01:04.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[35961]: - 2021-02-10T05:01:04.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[35961]: 02 2021-02-10T05:01:04.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[35961]: - 2021-02-10T05:01:04.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[35961]: 10 2021-02-10T05:01:04.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[35961]: T 2021-02-10T05:01:04.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[35961]: 05 2021-02-10T05:01:04.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[35961]: : 2021-02-10T05:01:04.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[35961]: 01 2021-02-10T05:01:04.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[35961]: :02.593285+0000 mgr.y (mgr.14140 2021-02-10T05:01:04.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[35961]: ) 1586 : cluster [DBG] pgmap v2582: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 401 MiB used, 715 GiB / 715 GiB avail; 7/1074 objects misplaced (0.652%) 2021-02-10T05:01:04.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[35961]: audit 2021-02-10T05:01:02 2021-02-10T05:01:04.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[35961]: .809408+0000 mon.a (mon.0) 3165 : audit 2021-02-10T05:01:04.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[35961]: [INF] from='client.? 172.21.15.85:0/3209633411' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2021-02-10T05:01:04.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[35961]: cluster 2021-02-10T05:01 2021-02-10T05:01:04.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[35961]: :02.809447+0000 mon.a (mon.0) 3166 2021-02-10T05:01:04.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[35961]: : cluster [DBG] osdmap e1411: 8 total, 8 up, 8 in 2021-02-10T05:01:04.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[35961]: cluster 2021-02-10T05 2021-02-10T05:01:04.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[35961]: :01:03.129783+0000 osd.5 (osd. 2021-02-10T05:01:04.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[35961]: 5) 222 : cluster [DBG] 5.16 scrub ok 2021-02-10T05:01:04.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T05:01:04.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[35961]: 05:01:03.622429+0000 osd.1 (osd. 2021-02-10T05:01:04.195 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:03 smithi085 conmon[35961]: 1) 540 : cluster [DBG] 5.12 deep-scrub ok 2021-02-10T05:01:04.941 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:04 smithi085 conmon[35961]: cluster 2021-02-10T05:01:03.324009 2021-02-10T05:01:04.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:04 smithi085 conmon[35961]: +0000 osd.0 (osd.0) 31 : cluster [DBG] 4.1e scrub ok 2021-02-10T05:01:04.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:04 smithi085 conmon[35961]: cluster 2021-02-10T05:01:03.557273+0000 osd.3 (osd.3) 66 : cluster [DBG] 6.1f scrub ok 2021-02-10T05:01:04.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:04 smithi085 conmon[35961]: cluster 2021-02-10T05:01:03.706674+0000 osd.6 (osd.6) 209 : cluster [DBG] 2.7 scrub ok 2021-02-10T05:01:04.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:04 smithi085 conmon[35961]: cluster 2021-02-10T05:01:04.606160+0000 osd.1 (osd.1) 541 : cluster [DBG] 3.0 scrub ok 2021-02-10T05:01:04.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:04 smithi085 conmon[29319]: cluster 2021-02-10T05:01:04.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:04 smithi085 conmon[29319]: 2021- 2021-02-10T05:01:04.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:04 smithi085 conmon[29319]: 02-10T05:01:03.324009+0000 osd.0 (osd.0) 31 : cluster [DBG] 4.1e scrub ok 2021-02-10T05:01:04.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:04 smithi085 conmon[29319]: cluster 2021-02-10T05:01 2021-02-10T05:01:04.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:04 smithi085 conmon[29319]: :03.557273+0000 osd.3 (osd.3) 66 : 2021-02-10T05:01:04.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:04 smithi085 conmon[29319]: cluster [DBG] 6.1f scrub ok 2021-02-10T05:01:04.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:04 smithi085 conmon[29319]: cluster 2021- 2021-02-10T05:01:04.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:04 smithi085 conmon[29319]: 02-10T05:01:03.706674+0000 2021-02-10T05:01:04.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:04 smithi085 conmon[29319]: osd.6 (osd.6) 209 : cluster [DBG] 2021-02-10T05:01:04.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:04 smithi085 conmon[29319]: 2.7 scrub ok 2021-02-10T05:01:04.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:04 smithi085 conmon[29319]: cluster 2021-02-10T05: 2021-02-10T05:01:04.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:04 smithi085 conmon[29319]: 01:04.606160+0000 osd.1 (osd.1 2021-02-10T05:01:04.943 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:04 smithi085 conmon[29319]: ) 541 : cluster [DBG] 3.0 scrub ok 2021-02-10T05:01:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:04 smithi164 conmon[29288]: cluster 2021-02-10T05:01:03.324009 2021-02-10T05:01:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:04 smithi164 conmon[29288]: +0000 osd.0 (osd.0) 31 : cluster [DBG] 4.1e scrub ok 2021-02-10T05:01:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:04 smithi164 conmon[29288]: cluster 2021-02-10T05:01:03.557273+0000 osd.3 (osd 2021-02-10T05:01:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:04 smithi164 conmon[29288]: .3) 66 : cluster [DBG] 6.1f scrub ok 2021-02-10T05:01:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:04 smithi164 conmon[29288]: cluster 2021-02-10T05:01:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:04 smithi164 conmon[29288]: 2021-02-10 2021-02-10T05:01:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:04 smithi164 conmon[29288]: T05:01:03.706674+0000 osd.6 (osd.6) 2021-02-10T05:01:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:04 smithi164 conmon[29288]: 209 : cluster [DBG] 2.7 scrub ok 2021-02-10T05:01:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:04 smithi164 conmon[29288]: cluster 2021 2021-02-10T05:01:05.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:04 smithi164 conmon[29288]: -02-10T05:01:04.606160 2021-02-10T05:01:05.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:04 smithi164 conmon[29288]: +0000 osd.1 (osd.1) 541 : cluster 2021-02-10T05:01:05.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:04 smithi164 conmon[29288]: [DBG] 3.0 scrub ok 2021-02-10T05:01:06.107 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T05:01:06.107 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [], 2021-02-10T05:01:06.107 INFO:teuthology.orchestra.run.smithi164.stdout: "complaint_time": 30, 2021-02-10T05:01:06.107 INFO:teuthology.orchestra.run.smithi164.stdout: "num_blocked_ops": 0 2021-02-10T05:01:06.107 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T05:01:06.132 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:05 smithi164 conmon[29288]: cluster 2021-02-10T05:01:04.517370+0000 osd.3 ( 2021-02-10T05:01:06.132 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:05 smithi164 conmon[29288]: osd.3) 67 : cluster [DBG] 4.14 scrub ok 2021-02-10T05:01:06.132 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:05 smithi164 conmon[29288]: cluster 2021-02-10T05:01:04.593853+0000 mgr.y (mgr.14140) 1587 : cluster [DBG] pgmap v2584: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 401 MiB used, 715 GiB / 715 GiB avail; 7/1074 objects misplaced (0.652%) 2021-02-10T05:01:06.132 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:05 smithi164 conmon[29288]: cluster 2021-02-10T05:01:05.066504+0000 osd.5 (osd.5) 223 : cluster [DBG] 5.1b scrub ok 2021-02-10T05:01:06.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:05 smithi085 conmon[35961]: cluster 2021-02-10T05:01:04.517370+0000 2021-02-10T05:01:06.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:05 smithi085 conmon[35961]: osd.3 (osd.3) 67 : cluster [DBG] 4.14 scrub ok 2021-02-10T05:01:06.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:05 smithi085 conmon[35961]: cluster 2021-02-10T05:01:04.593853+0000 mgr.y (mgr.14140) 1587 : cluster [DBG] pgmap v2584: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 401 MiB used, 715 GiB / 715 GiB avail; 7/1074 objects misplaced (0.652%) 2021-02-10T05:01:06.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:05 smithi085 conmon[35961]: cluster 2021-02-10T05:01:05.066504+0000 osd.5 (osd.5) 223 : cluster [DBG] 5.1b scrub ok 2021-02-10T05:01:06.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:05 smithi085 conmon[29319]: cluster 2021-02-10T05:01:04.517370+0000 osd.3 (osd.3 2021-02-10T05:01:06.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:05 smithi085 conmon[29319]: ) 67 : cluster [DBG] 4.14 scrub ok 2021-02-10T05:01:06.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:05 smithi085 conmon[29319]: cluster 2021-02-10T05:01:04.593853+0000 mgr.y (mgr.14140) 1587 : cluster [DBG] pgmap v2584: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 401 MiB used, 715 GiB / 715 GiB avail; 7/1074 objects misplaced (0.652%) 2021-02-10T05:01:06.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:05 smithi085 conmon[29319]: cluster 2021-02-10T05:01:05.066504+0000 osd.5 (osd.5) 223 : cluster [DBG] 5.1b scrub ok 2021-02-10T05:01:06.859 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.4 to 1 (10000) 2021-02-10T05:01:07.143 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:06 smithi164 conmon[29288]: cluster 2021-02-10T05:01:05.433936+0000 osd.4 2021-02-10T05:01:07.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:06 smithi164 conmon[29288]: (osd.4) 2696 : cluster [DBG] 4.1a scrub ok 2021-02-10T05:01:07.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:06 smithi164 conmon[29288]: audit 2021-02-10T05:01:06.196298+0000 mon.c (mon.1) 179 : audit [INF] from='client.? 172.21.15.85:0/2607052479' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-02-10T05:01:07.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:06 smithi164 conmon[29288]: audit 2021-02-10T05:01:06.196622+0000 2021-02-10T05:01:07.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:06 smithi164 conmon[29288]: mon.a (mon.0) 3167 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-02-10T05:01:07.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:06 smithi085 conmon[29319]: cluster 2021-02-10T05:01:05.433936+0000 osd.4 2021-02-10T05:01:07.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:06 smithi085 conmon[29319]: (osd.4) 2696 : cluster [DBG] 4.1a scrub ok 2021-02-10T05:01:07.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:06 smithi085 conmon[29319]: audit 2021-02-10T05:01:06.196298+0000 mon.c (mon.1) 179 : audit [INF] from='client.? 172.21.15.85:0/2607052479' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-02-10T05:01:07.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:06 smithi085 conmon[29319]: audit 2021-02-10T05:01:06. 2021-02-10T05:01:07.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:06 smithi085 conmon[29319]: 196622+0000 mon.a (mon.0) 3167 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-02-10T05:01:07.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:06 smithi085 conmon[29319]: 2021-02-10T05:01:07.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:06 smithi085 conmon[35961]: cluster 2021-02-10T05:01:05.433936+0000 2021-02-10T05:01:07.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:06 smithi085 conmon[35961]: osd.4 (osd.4) 2696 : cluster [DBG] 4.1a scrub ok 2021-02-10T05:01:07.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:06 smithi085 conmon[35961]: audit 2021-02-10T05:01:06.196298 2021-02-10T05:01:07.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:06 smithi085 conmon[35961]: +0000 mon.c (mon.1) 179 : audit [INF] from='client.? 172.21.15.85:0/2607052479' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-02-10T05:01:07.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:06 smithi085 conmon[35961]: audit 2021-02-10T05:01 2021-02-10T05:01:07.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:06 smithi085 conmon[35961]: :06.196622+0000 mon.a (mon.0) 3167 2021-02-10T05:01:07.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:06 smithi085 conmon[35961]: : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-02-10T05:01:07.777 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd primary-affinity 4 1 2021-02-10T05:01:07.984 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:01:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:07 smithi164 conmon[29288]: cluster 2021-02-10T05:01:06.594341+0000 mgr.y (mgr. 2021-02-10T05:01:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:07 smithi164 conmon[29288]: 14140) 1588 : cluster [DBG] pgmap v2585: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 7/1077 objects misplaced (0.650%) 2021-02-10T05:01:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:07 smithi164 conmon[29288]: audit 2021-02-10T05:01:06.854048+0000 mon.a (mon.0) 3168 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-02-10T05:01:08.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:07 smithi164 conmon[29288]: cluster 2021-02-10T05:01:06.854132+0000 mon.a (mon.0) 3169 : cluster [DBG] osdmap e1412: 8 total, 8 up, 8 in 2021-02-10T05:01:08.185 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:07 smithi085 conmon[35961]: cluster 2021-02-10T05:01:06.594341+0000 mgr.y (mgr.14140) 1588 : cluster [DBG] pgmap v2585: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 7/1077 objects misplaced (0.650%) 2021-02-10T05:01:08.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:07 smithi085 conmon[35961]: audit 2021-02-10T05:01:06.854048+0000 mon.a (mon.0) 3168 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-02-10T05:01:08.186 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:07 smithi085 conmon[35961]: cluster 2021-02-10T05:01:06.854132+0000 mon.a (mon.0) 3169 : cluster [DBG] osdmap e1412: 8 total, 8 up, 8 in 2021-02-10T05:01:08.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:07 smithi085 conmon[29319]: cluster 2021-02-10T05:01:06.594341+0000 mgr.y (mgr 2021-02-10T05:01:08.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:07 smithi085 conmon[29319]: .14140) 1588 : cluster [DBG] pgmap v2585: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 7/1077 objects misplaced (0.650%) 2021-02-10T05:01:08.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:07 smithi085 conmon[29319]: audit 2021-02-10T05:01:06.854048+0000 mon.a (mon.0) 3168 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-02-10T05:01:08.186 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:07 smithi085 conmon[29319]: cluster 2021-02-10T05:01:06.854132+0000 mon.a (mon.0) 3169 : cluster [DBG] osdmap e1412: 8 total, 8 up, 8 in 2021-02-10T05:01:09.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:08 smithi164 conmon[29288]: cluster 2021-02-10T05:01:07.683679+0000 2021-02-10T05:01:09.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:08 smithi164 conmon[29288]: osd.6 (osd.6) 210 : cluster [DBG] 1.7 scrub ok 2021-02-10T05:01:09.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:08 smithi164 conmon[29288]: cluster 2021-02-10T05:01:08.159825+0000 osd.2 (osd.2) 2066 : cluster [DBG] 2.1d scrub ok 2021-02-10T05:01:09.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:08 smithi164 conmon[29288]: cluster 2021-02-10T05:01:08.688008+0000 mon.a (mon.0) 3170 : cluster [DBG] osdmap e1413: 8 total, 8 up, 8 in 2021-02-10T05:01:09.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:08 smithi085 conmon[29319]: cluster 2021-02-10T05:01:07.683679+0000 osd.6 (osd.6) 210 : cluster [DBG] 1.7 scrub ok 2021-02-10T05:01:09.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:08 smithi085 conmon[29319]: cluster 2021-02-10T05:01:08.159825+0000 osd.2 (osd.2) 2066 : cluster [DBG] 2.1d scrub ok 2021-02-10T05:01:09.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:08 smithi085 conmon[29319]: cluster 2021-02-10T05:01:08.688008+0000 mon.a (mon.0) 3170 : cluster [DBG] osdmap e1413: 8 total, 8 up, 8 in 2021-02-10T05:01:09.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:08 smithi085 conmon[35961]: cluster 2021-02-10T05:01:07.683679+0000 osd.6 (osd.6) 210 : cluster [DBG] 1.7 scrub ok 2021-02-10T05:01:09.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:08 smithi085 conmon[35961]: cluster 2021-02-10T05:01:08.159825+0000 osd.2 (osd.2) 2066 : cluster [DBG] 2.1d scrub ok 2021-02-10T05:01:09.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:08 smithi085 conmon[35961]: cluster 2021-02-10T05:01:08.688008+0000 mon.a (mon.0) 3170 : cluster [DBG] osdmap e1413: 8 total, 8 up, 8 in 2021-02-10T05:01:10.144 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 05:01:09 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T05:01:10.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:09 smithi164 conmon[29288]: cluster 2021-02-10T05:01:08.413265+0000 osd.4 (osd.4) 2697 : cluster [DBG] 3.3 scrub ok 2021-02-10T05:01:10.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:09 smithi164 conmon[29288]: cluster 2021-02-10T05:01:08.437807+0000 2021-02-10T05:01:10.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:09 smithi164 conmon[29288]: osd.7 (osd.7) 123 : cluster [DBG] 2.20 scrub ok 2021-02-10T05:01:10.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:09 smithi164 conmon[29288]: cluster 2021-02-10T05:01:08.594967+0000 mgr.y (mgr.14140) 1589 : cluster [DBG] pgmap v2587: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 7/1071 objects misplaced (0.654%) 2021-02-10T05:01:10.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:09 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T05:01:10.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:09 smithi164 conmon[29288]: T05:01:08.767622+0000 osd.6 (osd.6) 211 : cluster 2021-02-10T05:01:10.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:09 smithi164 conmon[29288]: [DBG] 6.25 scrub ok 2021-02-10T05:01:10.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:09 smithi085 conmon[29319]: cluster 2021-02-10T05:01:08. 2021-02-10T05:01:10.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:09 smithi085 conmon[29319]: 413265+0000 osd.4 (osd.4) 2697 : cluster [DBG] 3.3 scrub ok 2021-02-10T05:01:10.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:09 smithi085 conmon[29319]: cluster 2021-02-10T05:01:08.437807+0000 osd.7 (osd.7) 123 : cluster [DBG] 2.20 scrub ok 2021-02-10T05:01:10.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:09 smithi085 conmon[29319]: cluster 2021-02-10T05:01:08.594967+0000 mgr.y (mgr.14140) 1589 : cluster [DBG] pgmap v2587: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 7/1071 objects misplaced (0.654%) 2021-02-10T05:01:10.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:09 smithi085 conmon[29319]: cluster 2021-02-10T05:01:08.767622+0000 osd.6 (osd.6) 211 : cluster [DBG] 6.25 scrub ok 2021-02-10T05:01:10.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:09 smithi085 conmon[35961]: cluster 2021-02-10T05:01:08.413265+0000 2021-02-10T05:01:10.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:09 smithi085 conmon[35961]: osd.4 (osd.4) 2697 : cluster [DBG] 3.3 scrub ok 2021-02-10T05:01:10.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:09 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T05:01:10.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:09 smithi085 conmon[35961]: 10T05:01:08.437807+0000 osd.7 (osd.7) 123 : 2021-02-10T05:01:10.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:09 smithi085 conmon[35961]: cluster [DBG] 2.20 scrub ok 2021-02-10T05:01:10.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:09 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T05:01:10.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:09 smithi085 conmon[35961]: 05:01:08.594967+0000 mgr.y (mgr 2021-02-10T05:01:10.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:09 smithi085 conmon[35961]: .14140) 1589 : cluster [DBG] pgmap v2587: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 402 MiB used, 715 GiB / 715 GiB avail; 7/1071 objects misplaced (0.654%) 2021-02-10T05:01:10.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:09 smithi085 conmon[35961]: cluster 2021- 2021-02-10T05:01:10.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:09 smithi085 conmon[35961]: 02-10T05:01:08.767622+0000 2021-02-10T05:01:10.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:09 smithi085 conmon[35961]: osd.6 (osd.6) 211 : cluster [DBG] 2021-02-10T05:01:10.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:09 smithi085 conmon[35961]: 6.25 scrub ok 2021-02-10T05:01:10.868 INFO:teuthology.orchestra.run.smithi085.stderr:set osd.4 primary-affinity to 1 (8655362) 2021-02-10T05:01:11.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:10 smithi164 conmon[29288]: cluster 2021-02-10T05:01:09.405860+0000 osd.0 (osd.0) 32 : cluster 2021-02-10T05:01:11.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:10 smithi164 conmon[29288]: [DBG] 6.0 scrub ok 2021-02-10T05:01:11.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:10 smithi164 conmon[29288]: audit 2021-02-10T05:01:09.656023+0000 mgr.y (mgr.14140) 1590 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:01:11.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:10 smithi164 conmon[29288]: cluster 2021-02-10T05:01:09.731112+0000 osd.1 ( 2021-02-10T05:01:11.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:10 smithi164 conmon[29288]: osd.1) 542 : cluster [DBG] 3.4 scrub ok 2021-02-10T05:01:11.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:10 smithi164 conmon[29288]: audit 2021-02-10T05:01:09.882729+0000 mon.a ( 2021-02-10T05:01:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:10 smithi164 conmon[29288]: mon.0) 3171 : audit [INF] from='client.? 172.21.15.85:0/2045103033' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2021-02-10T05:01:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:10 smithi164 conmon[29288]: cluster 2021-02-10T05:01:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:10 smithi164 conmon[29288]: 2021-02-10T05:01:10. 2021-02-10T05:01:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:10 smithi164 conmon[29288]: 121275+0000 osd.5 (osd.5) 224 2021-02-10T05:01:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:10 smithi164 conmon[29288]: : cluster [DBG] 2.18 scrub ok 2021-02-10T05:01:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:10 smithi164 conmon[29288]: cluster 2021 2021-02-10T05:01:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:10 smithi164 conmon[29288]: -02-10T05:01: 2021-02-10T05:01:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:10 smithi164 conmon[29288]: 10.713418+0000 osd.1 (osd.1) 543 2021-02-10T05:01:11.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:10 smithi164 conmon[29288]: : cluster [DBG] 1.3 scrub ok 2021-02-10T05:01:11.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[29319]: cluster 2021-02-10T05:01:09.405860+0000 2021-02-10T05:01:11.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[29319]: osd.0 (osd.0) 32 : cluster [DBG] 6.0 scrub ok 2021-02-10T05:01:11.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[29319]: audit 2021-02-10T05:01:09.656023+0000 mgr.y (mgr.14140) 1590 : audit 2021-02-10T05:01:11.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[29319]: [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:01:11.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T05:01:11.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[29319]: 10T05:01:09.731112+0000 osd.1 (osd.1) 542 : cluster 2021-02-10T05:01:11.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[29319]: [DBG] 3.4 scrub ok 2021-02-10T05:01:11.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[29319]: audit 2021- 2021-02-10T05:01:11.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[29319]: 02-10T05:01:09.882729+0000 mon.a ( 2021-02-10T05:01:11.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[29319]: mon.0) 3171 : audit [INF] from='client.? 172.21.15.85:0/2045103033' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2021-02-10T05:01:11.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[29319]: cluster 2021- 2021-02-10T05:01:11.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[29319]: 02-10T05:01:10.121275+0000 2021-02-10T05:01:11.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[29319]: osd.5 (osd.5) 224 : cluster [DBG] 2021-02-10T05:01:11.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[29319]: 2.18 scrub ok 2021-02-10T05:01:11.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[29319]: cluster 2021-02-10T05:01:11.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[29319]: 2021-02-10T05:01:10.713418+0000 osd.1 ( 2021-02-10T05:01:11.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[29319]: osd.1) 543 : cluster [DBG] 1.3 scrub ok 2021-02-10T05:01:11.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[35961]: cluster 2021-02-10T05:01:09.405860+0000 2021-02-10T05:01:11.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[35961]: osd.0 (osd.0) 32 : cluster [DBG] 6.0 scrub ok 2021-02-10T05:01:11.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[35961]: audit 2021 2021-02-10T05:01:11.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[35961]: -02-10T05:01:09.656023+0000 mgr.y (mgr.14140) 1590 2021-02-10T05:01:11.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[35961]: : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:01:11.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[35961]: cluster 2021-02-10T05:01:09 2021-02-10T05:01:11.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[35961]: .731112+0000 osd.1 (osd.1) 542 : cluster 2021-02-10T05:01:11.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[35961]: [DBG] 3.4 scrub ok 2021-02-10T05:01:11.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[35961]: audit 2021-02-10T05:01 2021-02-10T05:01:11.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[35961]: :09.882729+0000 mon.a (mon.0) 3171 : 2021-02-10T05:01:11.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[35961]: audit [INF] from='client.? 172.21.15.85:0/2045103033' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2021-02-10T05:01:11.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[35961]: cluster 2021-02-10T05:01:10 2021-02-10T05:01:11.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[35961]: .121275+0000 osd.5 (osd.5) 224 : cluster [DBG] 2021-02-10T05:01:11.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[35961]: 2.18 scrub ok 2021-02-10T05:01:11.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[35961]: cluster 2021-02-10T05:01:11.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[35961]: 2021-02-10T05:01:10.713418+0000 osd.1 (osd.1) 543 2021-02-10T05:01:11.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:10 smithi085 conmon[35961]: : cluster [DBG] 1.3 scrub ok 2021-02-10T05:01:11.854 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 5 1 2021-02-10T05:01:12.086 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:01:12.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:11 smithi164 conmon[29288]: cluster 2021-02-10T05:01:10.425821 2021-02-10T05:01:12.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:11 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2698 : cluster [DBG] 4.e scrub ok 2021-02-10T05:01:12.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:11 smithi164 conmon[29288]: cluster 2021-02-10T05:01:10.451707+0000 osd.0 (osd.0) 33 : 2021-02-10T05:01:12.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:11 smithi164 conmon[29288]: cluster [DBG] 2.1f scrub ok 2021-02-10T05:01:12.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:11 smithi164 conmon[29288]: cluster 2021-02-10T05:01:12.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:11 smithi164 conmon[29288]: 2021-02-10T05:01:10.595714 2021-02-10T05:01:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:11 smithi164 conmon[29288]: +0000 mgr.y (mgr.14140) 1591 2021-02-10T05:01:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:11 smithi164 conmon[29288]: : cluster [DBG] pgmap v2589: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 401 MiB used, 715 GiB / 715 GiB avail; 7/1071 objects misplaced (0.654%) 2021-02-10T05:01:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:11 smithi164 conmon[29288]: cluster 2021-02-10T05:01:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:11 smithi164 conmon[29288]: 2021-02 2021-02-10T05:01:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:11 smithi164 conmon[29288]: -10T05:01:10.791787+0000 osd.6 (osd.6) 212 : cluster [DBG] 5.1f scrub ok 2021-02-10T05:01:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:11 smithi164 conmon[29288]: 2021-02-10T05:01:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:11 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T05:01:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:11 smithi164 conmon[29288]: T05:01:10.863239+0000 mon.a (mon 2021-02-10T05:01:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:11 smithi164 conmon[29288]: .0) 3172 : audit [INF] from='client.? 172.21.15.85:0/2045103033' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2021-02-10T05:01:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:11 smithi164 conmon[29288]: cluster 2021 2021-02-10T05:01:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:11 smithi164 conmon[29288]: -02-10T05:01:10.863418+0000 2021-02-10T05:01:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:11 smithi164 conmon[29288]: mon.a (mon.0) 3173 : cluster [DBG] 2021-02-10T05:01:12.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:11 smithi164 conmon[29288]: osdmap e1414: 8 total, 8 up, 8 in 2021-02-10T05:01:12.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:11 smithi164 conmon[29288]: cluster 2021-02-10T05: 2021-02-10T05:01:12.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:11 smithi164 conmon[29288]: 01:11.674668+0000 osd.1 (osd.1 2021-02-10T05:01:12.146 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:11 smithi164 conmon[29288]: ) 544 : cluster [DBG] 2.9 scrub ok 2021-02-10T05:01:12.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[29319]: cluster 2021-02-10T05:01:10.425821+0000 2021-02-10T05:01:12.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[29319]: osd.4 (osd.4) 2698 : cluster [DBG] 4.e scrub ok 2021-02-10T05:01:12.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[29319]: cluster 2021-02-10T05:01:10.451707+0000 osd.0 (osd.0) 33 : cluster [DBG] 2.1f scrub ok 2021-02-10T05:01:12.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[29319]: cluster 2021-02-10T05:01:10.595714+0000 mgr.y (mgr.14140) 1591 : cluster [DBG] pgmap v2589: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 401 MiB used, 715 GiB / 715 GiB avail; 7/1071 objects misplaced (0.654%) 2021-02-10T05:01:12.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[29319]: cluster 2021-02-10T05:01:10.791787+0000 osd.6 (osd.6 2021-02-10T05:01:12.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[29319]: ) 212 : cluster [DBG] 5.1f scrub ok 2021-02-10T05:01:12.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[29319]: audit 2021-02 2021-02-10T05:01:12.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[29319]: -10T05:01:10.863239+0000 mon.a (mon.0 2021-02-10T05:01:12.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[29319]: ) 3172 : audit [INF] from='client.? 172.21.15.85:0/2045103033' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2021-02-10T05:01:12.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[29319]: cluster 2021-02-10T05:01:12.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[29319]: 2021-02-10T05:01:10.863418 2021-02-10T05:01:12.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[29319]: +0000 mon.a (mon.0) 3173 : cluster [DBG] osdmap e1414: 8 total, 8 up, 8 in 2021-02-10T05:01:12.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[29319]: 2021-02-10T05:01:12.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[29319]: cluster 2021-02-10T05:01: 2021-02-10T05:01:12.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[29319]: 11.674668+0000 osd.1 (osd.1) 2021-02-10T05:01:12.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[29319]: 544 : cluster [DBG] 2.9 scrub ok 2021-02-10T05:01:12.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[35961]: cluster 2021-02-10T05:01:10 2021-02-10T05:01:12.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[35961]: .425821+0000 osd.4 (osd.4) 2698 : cluster [DBG] 4.e scrub ok 2021-02-10T05:01:12.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[35961]: cluster 2021-02-10T05:01:10.451707+0000 osd.0 (osd.0) 2021-02-10T05:01:12.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[35961]: 33 : cluster [DBG] 2.1f scrub ok 2021-02-10T05:01:12.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T05:01:12.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[35961]: 05:01:10.595714+0000 mgr.y (mgr.14140) 1591 : cluster 2021-02-10T05:01:12.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[35961]: [DBG] pgmap v2589: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 401 MiB used, 715 GiB / 715 GiB avail; 7/1071 objects misplaced (0.654%) 2021-02-10T05:01:12.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[35961]: cluster 2021-02-10T05:01:10.791787+0000 2021-02-10T05:01:12.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[35961]: osd.6 (osd.6) 212 : cluster [DBG] 5.1f scrub ok 2021-02-10T05:01:12.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[35961]: audit 2021-02-10T05:01:10. 2021-02-10T05:01:12.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[35961]: 863239+0000 mon.a (mon.0) 3172 : audit [INF] from='client.? 172.21.15.85:0/2045103033' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2021-02-10T05:01:12.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[35961]: cluster 2021-02-10T05:01 2021-02-10T05:01:12.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[35961]: :10.863418+0000 mon.a (mon.0) 3173 : cluster [DBG] osdmap e1414: 8 total, 8 up, 8 in 2021-02-10T05:01:12.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[35961]: 2021-02-10T05:01:12.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[35961]: cluster 2021-02-10T05 2021-02-10T05:01:12.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:11 smithi085 conmon[35961]: :01:11.674668+0000 osd.1 (osd.1) 544 : cluster [DBG] 2.9 scrub ok 2021-02-10T05:01:12.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:12 smithi085 conmon[29319]: cluster 2021-02-10T05:01:11.456830+0000 osd.4 (osd.4) 2699 : cluster [DBG] 4.c scrub ok 2021-02-10T05:01:12.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:12 smithi085 conmon[29319]: cluster 2021-02-10T05:01:11.771183+0000 osd.6 (osd.6) 213 : cluster [DBG] 6.22 scrub ok 2021-02-10T05:01:12.942 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:12 smithi085 conmon[29319]: cluster 2021-02-10T05:01:12.177197+0000 osd.2 (osd.2) 2067 : cluster [DBG] 5.d scrub ok 2021-02-10T05:01:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:12 smithi085 conmon[35961]: cluster 2021-02-10T05:01:11.456830+0000 osd.4 (osd.4) 2699 : cluster [DBG] 4.c scrub ok 2021-02-10T05:01:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:12 smithi085 conmon[35961]: cluster 2021-02-10T05:01:11.771183+0000 osd.6 (osd.6) 213 : cluster [DBG] 6.22 scrub ok 2021-02-10T05:01:12.942 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:12 smithi085 conmon[35961]: cluster 2021-02-10T05:01:12.177197+0000 osd.2 (osd.2) 2067 : cluster [DBG] 5.d scrub ok 2021-02-10T05:01:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:12 smithi164 conmon[29288]: cluster 2021-02-10T05:01:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:12 smithi164 conmon[29288]: 2021-02-10T05: 2021-02-10T05:01:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:12 smithi164 conmon[29288]: 01:11.456830+0000 osd.4 (osd.4) 2699 : cluster [DBG] 4.c scrub ok 2021-02-10T05:01:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:12 smithi164 conmon[29288]: cluster 2021-02-10T05:01:11 2021-02-10T05:01:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:12 smithi164 conmon[29288]: .771183+0000 osd.6 (osd.6) 213 : cluster [DBG] 6.22 scrub ok 2021-02-10T05:01:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:12 smithi164 conmon[29288]: cluster 2021 2021-02-10T05:01:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:12 smithi164 conmon[29288]: -02-10T05:01:12.177197+0000 osd.2 (osd.2) 2067 : cluster 2021-02-10T05:01:13.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:12 smithi164 conmon[29288]: [DBG] 5.d scrub ok 2021-02-10T05:01:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:13 smithi164 conmon[29288]: cluster 2021-02-10T05:01:12.476550+0000 osd.4 (osd.4 2021-02-10T05:01:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:13 smithi164 conmon[29288]: ) 2700 : cluster [DBG] 4.18 scrub ok 2021-02-10T05:01:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:13 smithi164 conmon[29288]: cluster 2021-02-10T05:01:12.596270+0000 mgr.y (mgr.14140) 1592 : cluster [DBG] pgmap v2591: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 399 MiB used, 715 GiB / 715 GiB avail; 7/1071 objects misplaced (0.654%) 2021-02-10T05:01:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:13 smithi164 conmon[29288]: cluster 2021-02-10T05:01:13. 2021-02-10T05:01:14.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:13 smithi164 conmon[29288]: 666580+0000 osd.1 (osd.1) 545 : cluster [DBG] 6.20 scrub ok 2021-02-10T05:01:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:13 smithi164 conmon[29288]: cluster 2021-02-10T05:01:13.693853+0000 mon.a (mon 2021-02-10T05:01:14.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:13 smithi164 conmon[29288]: .0) 3174 : cluster [DBG] osdmap e1415: 8 total, 8 up, 8 in 2021-02-10T05:01:14.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:13 smithi085 conmon[29319]: cluster 2021-02-10T05:01:12.476550 2021-02-10T05:01:14.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:13 smithi085 conmon[29319]: +0000 osd.4 (osd.4) 2700 : cluster [DBG] 4.18 scrub ok 2021-02-10T05:01:14.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:13 smithi085 conmon[29319]: cluster 2021-02-10T05:01:12.596270+0000 mgr.y (mgr.14140) 1592 : cluster [DBG] pgmap v2591: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 399 MiB used, 715 GiB / 715 GiB avail; 7/1071 objects misplaced (0.654%) 2021-02-10T05:01:14.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:13 smithi085 conmon[29319]: cluster 2021-02-10T05:01:13.666580+0000 osd.1 (osd.1) 545 : cluster [DBG] 6.20 scrub ok 2021-02-10T05:01:14.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:13 smithi085 conmon[29319]: cluster 2021-02-10T05:01:13.693853+0000 mon.a (mon.0) 3174 : cluster [DBG] osdmap e1415: 8 total, 8 up, 8 in 2021-02-10T05:01:14.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:13 smithi085 conmon[35961]: cluster 2021-02-10T05:01: 2021-02-10T05:01:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:13 smithi085 conmon[35961]: 12.476550+0000 osd.4 (osd.4) 2700 : cluster [DBG] 4.18 scrub ok 2021-02-10T05:01:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:13 smithi085 conmon[35961]: cluster 2021-02-10T05:01:12.596270+0000 mgr.y (mgr.14140) 1592 : cluster [DBG] pgmap v2591: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 399 MiB used, 715 GiB / 715 GiB avail; 7/1071 objects misplaced (0.654%) 2021-02-10T05:01:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:13 smithi085 conmon[35961]: cluster 2021-02-10T05:01:13.666580+0000 osd.1 (osd.1) 545 : cluster [DBG] 6.20 scrub ok 2021-02-10T05:01:14.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:13 smithi085 conmon[35961]: cluster 2021-02-10T05:01:13.693853+0000 mon.a (mon.0) 3174 : cluster [DBG] osdmap e1415: 8 total, 8 up, 8 in 2021-02-10T05:01:14.885 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.5 to 1 (10000) 2021-02-10T05:01:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:14 smithi164 conmon[29288]: cluster 2021-02-10T05:01:13.514095+0000 osd.3 ( 2021-02-10T05:01:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:14 smithi164 conmon[29288]: osd.3) 68 : cluster [DBG] 6.b scrub ok 2021-02-10T05:01:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:14 smithi164 conmon[29288]: cluster 2021-02-10T05:01:13.618335+0000 osd.5 (osd.5) 225 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:01:15.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:14 smithi164 conmon[29288]: cluster 2021-02-10T05:01:13.619593+0000 2021-02-10T05:01:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:14 smithi164 conmon[29288]: osd.5 (osd.5) 226 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:01:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:14 smithi164 conmon[29288]: cluster 2021- 2021-02-10T05:01:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:14 smithi164 conmon[29288]: 02-10T05:01:14.117136+0000 osd.5 (osd. 2021-02-10T05:01:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:14 smithi164 conmon[29288]: 5) 227 : cluster [DBG] 6.19 scrub ok 2021-02-10T05:01:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:14 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T05:01:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:14 smithi164 conmon[29288]: T05:01:14.166206+0000 mon.a (mon 2021-02-10T05:01:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:14 smithi164 conmon[29288]: .0) 3175 : audit [INF] from='client.? 172.21.15.85:0/2057249998' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-02-10T05:01:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:14 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T05:01:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:14 smithi164 conmon[29288]: 05:01:14.505191+0000 osd.3 (osd.3) 2021-02-10T05:01:15.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:14 smithi164 conmon[29288]: 69 : cluster [DBG] 5.14 deep-scrub ok 2021-02-10T05:01:15.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:14 smithi085 conmon[29319]: cluster 2021-02-10T05:01:13.514095+0000 osd.3 (osd 2021-02-10T05:01:15.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:14 smithi085 conmon[29319]: .3) 68 : cluster [DBG] 6.b scrub ok 2021-02-10T05:01:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:14 smithi085 conmon[29319]: cluster 2021-02-10T05:01:13.618335+0000 osd.5 (osd.5) 225 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:01:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:14 smithi085 conmon[29319]: cluster 2021-02-10T05:01:13.619593+0000 osd.5 (osd.5) 226 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:01:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:14 smithi085 conmon[29319]: cluster 2021-02-10T05:01:14.117136+0000 osd.5 (osd.5) 227 : cluster [DBG] 6.19 scrub ok 2021-02-10T05:01:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:14 smithi085 conmon[29319]: audit 2021-02-10T05:01:14.166206+0000 mon.a (mon.0) 3175 : audit [INF] from='client.? 172.21.15.85:0/2057249998' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-02-10T05:01:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:14 smithi085 conmon[29319]: 2021-02-10T05:01:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:14 smithi085 conmon[29319]: cluster 2021-02-10T05:01: 2021-02-10T05:01:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:14 smithi085 conmon[29319]: 14.505191+0000 osd.3 (osd.3) 69 : cluster [DBG] 5.14 deep-scrub ok 2021-02-10T05:01:15.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:14 smithi085 conmon[29319]: 2021-02-10T05:01:15.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:14 smithi085 conmon[35961]: cluster 2021-02-10T05:01:13.514095+0000 osd.3 (osd 2021-02-10T05:01:15.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:14 smithi085 conmon[35961]: .3) 68 : cluster [DBG] 6.b scrub ok 2021-02-10T05:01:15.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:14 smithi085 conmon[35961]: cluster 2021-02-10T05:01:13.618335+0000 osd.5 (osd.5) 225 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:01:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:14 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T05:01:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:14 smithi085 conmon[35961]: 05:01:13.619593+0000 osd.5 (osd.5) 2021-02-10T05:01:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:14 smithi085 conmon[35961]: 226 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:01:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:14 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T05:01:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:14 smithi085 conmon[35961]: -10T05:01:14.117136+0000 2021-02-10T05:01:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:14 smithi085 conmon[35961]: osd.5 (osd.5) 227 : cluster 2021-02-10T05:01:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:14 smithi085 conmon[35961]: [DBG] 2021-02-10T05:01:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:14 smithi085 conmon[35961]: 6.19 scrub ok 2021-02-10T05:01:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:14 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T05:01:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:14 smithi085 conmon[35961]: 10T05:01:14.166206 2021-02-10T05:01:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:14 smithi085 conmon[35961]: +0000 mon.a (mon.0) 3175 2021-02-10T05:01:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:14 smithi085 conmon[35961]: : audit [INF] from='client.? 172.21.15.85:0/2057249998' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-02-10T05:01:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:14 smithi085 conmon[35961]: cluster 2021- 2021-02-10T05:01:15.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:14 smithi085 conmon[35961]: 02-10T05:01:14.505191+0000 2021-02-10T05:01:15.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:14 smithi085 conmon[35961]: osd.3 (osd.3) 69 : cluster 2021-02-10T05:01:15.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:14 smithi085 conmon[35961]: [DBG] 5.14 deep-scrub ok 2021-02-10T05:01:16.056 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd primary-affinity 5 1 2021-02-10T05:01:16.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:15 smithi164 conmon[29288]: cluster 2021-02-10T05:01:16.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:15 smithi164 conmon[29288]: 2021-02-10T05:01:16.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:15 smithi164 conmon[29288]: 2021-02-10T05:01:14.596694+0000 mgr.y (mgr.14140) 1593 : 2021-02-10T05:01:16.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:15 smithi164 conmon[29288]: cluster [DBG] pgmap v2593: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 397 MiB used, 715 GiB / 715 GiB avail; 7/1068 objects misplaced (0.655%) 2021-02-10T05:01:16.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:15 smithi164 conmon[29288]: audit 2021-02-10T05:01:16.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:15 smithi164 conmon[29288]: 2021-02-10T05:01: 2021-02-10T05:01:16.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:15 smithi164 conmon[29288]: 14.882239+0000 mon.a (mon.0) 3176 : audit [INF] from='client.? 172.21.15.85:0/2057249998' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-02-10T05:01:16.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:15 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T05:01:16.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:15 smithi164 conmon[29288]: -10T05:01:14.882286+0000 2021-02-10T05:01:16.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:15 smithi164 conmon[29288]: mon.a (mon.0) 3177 : cluster 2021-02-10T05:01:16.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:15 smithi164 conmon[29288]: [DBG] osdmap e1416: 8 total, 8 up, 8 in 2021-02-10T05:01:16.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:15 smithi164 conmon[29288]: cluster 2021-02-10T05:01:16.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:15 smithi164 conmon[29288]: 2021-02-10T05:01:15.093543+0000 2021-02-10T05:01:16.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:15 smithi164 conmon[29288]: osd.5 (osd.5) 228 : cluster [DBG] 3.5 scrub ok 2021-02-10T05:01:16.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:15 smithi164 conmon[29288]: cluster 2021-02-10T05:01:16.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:15 smithi164 conmon[29288]: 2021-02-10T05: 2021-02-10T05:01:16.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:15 smithi164 conmon[29288]: 01:15.648575+0000 osd.1 (osd.1) 546 2021-02-10T05:01:16.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:15 smithi164 conmon[29288]: : cluster [DBG] 2.4 scrub ok 2021-02-10T05:01:16.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[35961]: cluster 2021-02-10T05:01:14.596694+0000 mgr.y ( 2021-02-10T05:01:16.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[35961]: mgr.14140) 1593 : cluster [DBG] pgmap v2593: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 397 MiB used, 715 GiB / 715 GiB avail; 7/1068 objects misplaced (0.655%) 2021-02-10T05:01:16.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[35961]: audit 2021-02-10T05:01:14.882239+0000 2021-02-10T05:01:16.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[35961]: mon.a (mon.0) 3176 : audit [INF] from='client.? 172.21.15.85:0/2057249998' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-02-10T05:01:16.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[35961]: cluster 2021- 2021-02-10T05:01:16.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[35961]: 02-10T05:01:14.882286 2021-02-10T05:01:16.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[35961]: +0000 mon.a (mon.0) 3177 : 2021-02-10T05:01:16.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[35961]: cluster [DBG] osdmap e1416: 8 total, 8 up, 8 in 2021-02-10T05:01:16.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[35961]: cluster 2021 2021-02-10T05:01:16.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[35961]: -02-10T05:01:15 2021-02-10T05:01:16.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[35961]: .093543+0000 osd.5 (osd.5 2021-02-10T05:01:16.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[35961]: ) 228 : cluster [DBG] 3.5 scrub ok 2021-02-10T05:01:16.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[35961]: cluster 2021 2021-02-10T05:01:16.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[35961]: -02-10T05:01:15 2021-02-10T05:01:16.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[35961]: .648575+0000 osd.1 (osd.1 2021-02-10T05:01:16.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[35961]: ) 546 : cluster [DBG] 2.4 scrub ok 2021-02-10T05:01:16.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[35961]: 2021-02-10T05:01:16.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[29319]: cluster 2021-02-10T05:01:14.596694+0000 mgr.y (mgr.14140) 1593 : 2021-02-10T05:01:16.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[29319]: cluster [DBG] pgmap v2593: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 397 MiB used, 715 GiB / 715 GiB avail; 7/1068 objects misplaced (0.655%) 2021-02-10T05:01:16.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[29319]: audit 2021-02-10T05:01:14.882239+0000 mon.a (mon.0) 3176 : audit [INF] from='client.? 172.21.15.85:0/2057249998' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-02-10T05:01:16.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[29319]: cluster 2021-02-10T05:01:14.882286+0000 mon.a 2021-02-10T05:01:16.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[29319]: (mon.0) 3177 : cluster [DBG] 2021-02-10T05:01:16.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[29319]: osdmap e1416: 8 total, 8 up, 8 in 2021-02-10T05:01:16.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[29319]: cluster 2021- 2021-02-10T05:01:16.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[29319]: 02-10T05:01:15 2021-02-10T05:01:16.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[29319]: .093543+0000 osd.5 (osd. 2021-02-10T05:01:16.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[29319]: 5) 228 : cluster [DBG] 3.5 scrub ok 2021-02-10T05:01:16.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[29319]: 2021-02-10T05:01:16.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[29319]: cluster 2021-02-10T05:01:16.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[29319]: 2021-02-10T05 2021-02-10T05:01:16.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[29319]: :01:15.648575+0000 osd.1 2021-02-10T05:01:16.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[29319]: (osd.1) 546 : cluster 2021-02-10T05:01:16.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:15 smithi085 conmon[29319]: [DBG] 2.4 scrub ok 2021-02-10T05:01:16.258 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:01:17.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:16 smithi164 conmon[29288]: cluster 2021-02-10T05:01:15.872191+0000 osd.6 (osd.6) 214 : cluster [DBG] 3.a scrub ok 2021-02-10T05:01:17.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:16 smithi164 conmon[29288]: cluster 2021-02-10T05:01:16.461665+0000 osd.3 (osd.3) 70 : cluster [DBG] 5.0 scrub ok 2021-02-10T05:01:17.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:16 smithi164 conmon[29288]: cluster 2021-02-10T05:01:16.662446+0000 osd.1 (osd.1) 547 : cluster [DBG] 6.26 scrub ok 2021-02-10T05:01:17.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:16 smithi085 conmon[29319]: cluster 2021-02-10T05:01:15.872191+0000 osd.6 2021-02-10T05:01:17.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:16 smithi085 conmon[29319]: (osd.6) 214 : cluster [DBG] 3.a scrub ok 2021-02-10T05:01:17.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:16 smithi085 conmon[29319]: cluster 2021-02-10T05:01:16.461665+0000 osd.3 (osd.3) 70 : cluster [DBG] 5.0 scrub ok 2021-02-10T05:01:17.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:16 smithi085 conmon[29319]: cluster 2021-02-10T05:01:16.662446+0000 osd.1 (osd.1) 547 : cluster [DBG] 6.26 scrub ok 2021-02-10T05:01:17.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:16 smithi085 conmon[35961]: cluster 2021-02-10T05:01:15.872191+0000 osd.6 (osd.6 2021-02-10T05:01:17.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:16 smithi085 conmon[35961]: ) 214 : cluster [DBG] 3.a scrub ok 2021-02-10T05:01:17.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:16 smithi085 conmon[35961]: cluster 2021-02-10T05:01:16.461665+0000 osd.3 (osd.3) 70 : cluster [DBG] 5.0 scrub ok 2021-02-10T05:01:17.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:16 smithi085 conmon[35961]: cluster 2021-02-10T05:01:16.662446+0000 osd.1 (osd.1) 547 : cluster [DBG] 2021-02-10T05:01:17.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:16 smithi085 conmon[35961]: 6.26 scrub ok 2021-02-10T05:01:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:17 smithi164 conmon[29288]: cluster 2021-02-10T05:01:16.597321+0000 mgr.y 2021-02-10T05:01:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:17 smithi164 conmon[29288]: (mgr.14140) 1594 : cluster [DBG] pgmap v2595: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 7/1068 objects misplaced (0.655%) 2021-02-10T05:01:18.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:17 smithi164 conmon[29288]: cluster 2021-02-10T05:01:17.622757+0000 osd.1 (osd.1) 548 : cluster [DBG] 5.1d scrub ok 2021-02-10T05:01:18.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:17 smithi085 conmon[29319]: cluster 2021-02-10T05:01:16.597321+0000 mgr.y (mgr.14140) 1594 : cluster [DBG] pgmap v2595: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 7/1068 objects misplaced (0.655%) 2021-02-10T05:01:18.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:17 smithi085 conmon[29319]: cluster 2021-02-10T05:01:17.622757+0000 osd.1 (osd.1) 548 : cluster [DBG] 5.1d scrub ok 2021-02-10T05:01:18.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:17 smithi085 conmon[35961]: cluster 2021-02-10T05:01:16.597321+0000 mgr.y (mgr.14140) 1594 : cluster [DBG] pgmap v2595: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 396 MiB used, 715 GiB / 715 GiB avail; 7/1068 objects misplaced (0.655%) 2021-02-10T05:01:18.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:17 smithi085 conmon[35961]: cluster 2021-02-10T05:01:17.622757+0000 osd.1 (osd.1) 548 : cluster [DBG] 5.1d scrub ok 2021-02-10T05:01:18.843 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.7 dump_historic_ops 2021-02-10T05:01:18.904 INFO:teuthology.orchestra.run.smithi085.stderr:set osd.5 primary-affinity to 1 (8655362) 2021-02-10T05:01:19.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:18 smithi164 conmon[29288]: cluster 2021-02-10T05:01:19.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:18 smithi164 conmon[29288]: 2021-02-10 2021-02-10T05:01:19.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:18 smithi164 conmon[29288]: T05:01:17.475346+0000 osd.7 (osd.7) 124 : cluster [DBG] 6.15 scrub ok 2021-02-10T05:01:19.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:18 smithi164 conmon[29288]: cluster 2021-02-10T05:01:19.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:18 smithi164 conmon[29288]: 2021-02-10T05:01:19.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:18 smithi164 conmon[29288]: 2021-02-10T 2021-02-10T05:01:19.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:18 smithi164 conmon[29288]: 05:01:17.895582+0000 mon.a (mon.0) 3178 : cluster 2021-02-10T05:01:19.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:18 smithi164 conmon[29288]: [DBG] osdmap e1417: 8 total, 8 up, 8 in 2021-02-10T05:01:19.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:18 smithi164 conmon[29288]: audit 2021-02-10T05:01:19.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:18 smithi164 conmon[29288]: 2021- 2021-02-10T05:01:19.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:18 smithi164 conmon[29288]: 02-10T05:01:18.350678+0000 mon.a (mon.0) 2021-02-10T05:01:19.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:18 smithi164 conmon[29288]: 3179 : audit [INF] from='client.? 172.21.15.85:0/547323419' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2021-02-10T05:01:19.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:18 smithi085 conmon[29319]: cluster 2021-02-10T05:01:17.475346+0000 osd.7 (osd. 2021-02-10T05:01:19.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:18 smithi085 conmon[29319]: 7) 124 : cluster [DBG] 6.15 scrub ok 2021-02-10T05:01:19.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:18 smithi085 conmon[29319]: cluster 2021-02-10T05:01:17.895582+0000 mon.a (mon.0) 3178 : cluster [DBG] osdmap e1417: 8 total, 8 up, 8 in 2021-02-10T05:01:19.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:18 smithi085 conmon[29319]: audit 2021-02-10T05:01:18.350678+0000 mon.a (mon.0) 3179 : audit [INF] from='client.? 172.21.15.85:0/547323419' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2021-02-10T05:01:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:18 smithi085 conmon[35961]: cluster 2021-02-10T05:01:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:18 smithi085 conmon[35961]: 2021-02-10T05:01:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:18 smithi085 conmon[35961]: 2021 2021-02-10T05:01:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:18 smithi085 conmon[35961]: - 2021-02-10T05:01:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:18 smithi085 conmon[35961]: 02 2021-02-10T05:01:19.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:18 smithi085 conmon[35961]: -10T05:01: 2021-02-10T05:01:19.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:18 smithi085 conmon[35961]: 17.475346+0000 osd.7 (osd.7) 124 : cluster [DBG] 6.15 scrub ok 2021-02-10T05:01:19.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:18 smithi085 conmon[35961]: cluster 2021-02 2021-02-10T05:01:19.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:18 smithi085 conmon[35961]: -10T05:01:17.895582+0000 mon.a (mon 2021-02-10T05:01:19.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:18 smithi085 conmon[35961]: .0) 3178 : cluster [DBG] osdmap e1417: 8 total, 8 up, 8 in 2021-02-10T05:01:19.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:18 smithi085 conmon[35961]: audit 2021-02-10T05:01:19.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:18 smithi085 conmon[35961]: 2021-02-10T05:01:18.350678+0000 2021-02-10T05:01:19.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:18 smithi085 conmon[35961]: mon.a (mon.0) 3179 : audit [INF] 2021-02-10T05:01:19.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:18 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/547323419' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2021-02-10T05:01:19.583 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.b/config 2021-02-10T05:01:19.842 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 6 1 2021-02-10T05:01:20.064 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:01:20.144 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 05:01:19 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T05:01:20.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:19 smithi164 conmon[29288]: cluster 2021-02-10T05:01:18.519780+0000 osd.4 2021-02-10T05:01:20.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:19 smithi164 conmon[29288]: (osd.4) 2701 : cluster [DBG] 2.f deep-scrub ok 2021-02-10T05:01:20.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:19 smithi164 conmon[29288]: cluster 2021-02-10T05:01:18.597823+0000 mgr.y (mgr.14140) 1595 : cluster [DBG] pgmap v2597: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 393 MiB used, 715 GiB / 715 GiB avail; 7/1065 objects misplaced (0.657%) 2021-02-10T05:01:20.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:19 smithi164 conmon[29288]: audit 2021-02-10T05 2021-02-10T05:01:20.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:19 smithi164 conmon[29288]: :01:18.899610+0000 mon.a (mon.0) 3180 2021-02-10T05:01:20.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:19 smithi164 conmon[29288]: : audit [INF] from='client.? 172.21.15.85:0/547323419' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2021-02-10T05:01:20.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:19 smithi164 conmon[29288]: cluster 2021-02-10T05:01:20.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:19 smithi164 conmon[29288]: 2021-02-10T05:01:18.899671+0000 2021-02-10T05:01:20.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:19 smithi164 conmon[29288]: mon.a (mon.0) 3181 : cluster [DBG] osdmap e1418: 8 total, 8 up, 8 in 2021-02-10T05:01:20.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:19 smithi085 conmon[29319]: cluster 2021-02-10T05:01:18.519780+0000 osd.4 (osd.4) 2701 : cluster [DBG] 2021-02-10T05:01:20.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:19 smithi085 conmon[29319]: 2.f deep-scrub ok 2021-02-10T05:01:20.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:19 smithi085 conmon[29319]: cluster 2021-02-10T05:01:18.597823+0000 mgr.y (mgr.14140) 1595 : cluster [DBG] pgmap v2597: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 393 MiB used, 715 GiB / 715 GiB avail; 7/1065 objects misplaced (0.657%) 2021-02-10T05:01:20.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:19 smithi085 conmon[29319]: audit 2021-02-10T05:01:18.899610+0000 mon.a (mon.0) 3180 : audit [INF] from='client.? 172.21.15.85:0/547323419' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2021-02-10T05:01:20.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:19 smithi085 conmon[29319]: cluster 2021-02-10T05:01:18.899671+0000 mon.a (mon.0) 3181 : cluster [DBG] osdmap e1418: 8 total, 8 up, 8 in 2021-02-10T05:01:20.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:19 smithi085 conmon[35961]: cluster 2021-02-10T05:01:18.519780+0000 2021-02-10T05:01:20.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:19 smithi085 conmon[35961]: osd.4 (osd.4) 2701 : cluster [DBG] 2.f deep-scrub ok 2021-02-10T05:01:20.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:19 smithi085 conmon[35961]: cluster 2021 2021-02-10T05:01:20.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:19 smithi085 conmon[35961]: -02-10T05:01:18.597823+0000 mgr.y (mgr.14140) 1595 : cluster 2021-02-10T05:01:20.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:19 smithi085 conmon[35961]: [DBG] pgmap v2597: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 393 MiB used, 715 GiB / 715 GiB avail; 7/1065 objects misplaced (0.657%) 2021-02-10T05:01:20.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:19 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T05:01:20.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:19 smithi085 conmon[35961]: 10T05:01:18.899610+0000 mon.a ( 2021-02-10T05:01:20.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:19 smithi085 conmon[35961]: mon.0) 3180 : audit [INF] from='client.? 172.21.15.85:0/547323419' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2021-02-10T05:01:20.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:19 smithi085 conmon[35961]: cluster 2021-02-10T05:01:20.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:19 smithi085 conmon[35961]: 2021-02-10T05:01:18.899671 2021-02-10T05:01:20.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:19 smithi085 conmon[35961]: +0000 mon.a (mon.0) 3181 : cluster [DBG] 2021-02-10T05:01:20.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:19 smithi085 conmon[35961]: osdmap e1418: 8 total, 8 up, 8 in 2021-02-10T05:01:20.903 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:20 smithi085 conmon[35961]: audit 2021-02-10T05:01:19.666049+0000 2021-02-10T05:01:20.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:20 smithi085 conmon[35961]: mgr.y (mgr.14140) 1596 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:01:20.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:20 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T05:01:20.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:20 smithi085 conmon[35961]: T05:01:19.801197+0000 osd.6 (osd.6) 215 : cluster [DBG] 2021-02-10T05:01:20.904 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:20 smithi085 conmon[35961]: 6.23 scrub ok 2021-02-10T05:01:21.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:20 smithi085 conmon[29319]: audit 2021-02-10T05:01:19.666049+0000 mgr.y ( 2021-02-10T05:01:21.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:20 smithi085 conmon[29319]: mgr.14140) 1596 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:01:21.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:20 smithi085 conmon[29319]: cluster 2021-02-10T05:01:19.801197+0000 osd.6 (osd.6) 215 : cluster [DBG] 6.23 scrub ok 2021-02-10T05:01:21.192 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:20 smithi164 conmon[29288]: audit 2021-02-10T05:01:19.666049+0000 2021-02-10T05:01:21.192 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:20 smithi164 conmon[29288]: mgr.y (mgr.14140) 1596 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:01:21.192 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:20 smithi164 conmon[29288]: cluster 2021-02-10T05:01:19.801197+0000 osd.6 (osd.6) 215 : cluster [DBG] 6.23 scrub ok 2021-02-10T05:01:22.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:21 smithi164 conmon[29288]: cluster 2021-02-10T05:01:20.520971+0000 osd.3 (osd.3) 71 : cluster [DBG] 6.6 scrub ok 2021-02-10T05:01:22.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:21 smithi164 conmon[29288]: cluster 2021-02-10T05:01:20.598336+0000 mgr.y (mgr.14140) 1597 : cluster [DBG] pgmap v2599: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 391 MiB used, 715 GiB / 715 GiB avail; 7/1065 objects misplaced (0.657%) 2021-02-10T05:01:22.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:21 smithi164 conmon[29288]: cluster 2021-02-10T05:01:20.784053+0000 osd.6 (osd.6) 216 : cluster [DBG] 2.24 scrub ok 2021-02-10T05:01:22.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:21 smithi164 conmon[29288]: cluster 2021-02-10T05:01:21.610796+0000 osd.1 (osd.1) 549 : cluster [DBG] 2.15 scrub ok 2021-02-10T05:01:22.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:21 smithi085 conmon[29319]: cluster 2021-02-10T05:01:20 2021-02-10T05:01:22.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:21 smithi085 conmon[29319]: .520971+0000 osd.3 (osd.3) 71 : cluster [DBG] 6.6 scrub ok 2021-02-10T05:01:22.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:21 smithi085 conmon[29319]: cluster 2021-02-10T05:01:20.598336+0000 mgr.y (mgr.14140) 1597 : cluster [DBG] pgmap v2599: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 391 MiB used, 715 GiB / 715 GiB avail; 7/1065 objects misplaced (0.657%) 2021-02-10T05:01:22.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:21 smithi085 conmon[29319]: cluster 2021-02-10T05:01:20.784053+0000 osd.6 (osd.6) 216 : cluster [DBG] 2.24 scrub ok 2021-02-10T05:01:22.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:21 smithi085 conmon[29319]: cluster 2021-02-10T05:01:21.610796+0000 osd.1 (osd.1) 549 : cluster [DBG] 2.15 scrub ok 2021-02-10T05:01:22.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:21 smithi085 conmon[35961]: cluster 2021-02-10T05:01:20.520971+0000 2021-02-10T05:01:22.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:21 smithi085 conmon[35961]: osd.3 (osd.3) 71 : cluster [DBG] 6.6 scrub ok 2021-02-10T05:01:22.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:21 smithi085 conmon[35961]: cluster 2021-02-10T05:01:20.598336+0000 mgr.y (mgr.14140) 1597 : cluster [DBG] pgmap v2599: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 25 KiB data, 391 MiB used, 715 GiB / 715 GiB avail; 7/1065 objects misplaced (0.657%) 2021-02-10T05:01:22.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:21 smithi085 conmon[35961]: cluster 2021-02-10T05:01:20.784053+0000 osd.6 (osd.6) 216 : cluster [DBG] 2.24 scrub ok 2021-02-10T05:01:22.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:21 smithi085 conmon[35961]: cluster 2021-02-10T05:01:21.610796+0000 osd.1 (osd.1) 549 : cluster [DBG] 2.15 scrub ok 2021-02-10T05:01:22.922 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.6 to 1 (10000) 2021-02-10T05:01:23.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[29319]: cluster 2021-02-10T05:01:21.508056+0000 2021-02-10T05:01:23.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[29319]: osd.2 (osd.2) 2068 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:01:23.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[29319]: cluster 2021 2021-02-10T05:01:23.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[29319]: -02-10T05:01:21.509277+0000 osd.2 (osd. 2021-02-10T05:01:23.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[29319]: 2) 2069 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:01:23.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[29319]: cluster 2021-02-10T05: 2021-02-10T05:01:23.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[29319]: 01:21.826642+0000 osd.6 (osd.6) 217 : 2021-02-10T05:01:23.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[29319]: cluster [DBG] 4.a scrub ok 2021-02-10T05:01:23.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[29319]: cluster 2021-02-10 2021-02-10T05:01:23.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[29319]: T05:01:21.910975+0000 2021-02-10T05:01:23.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[29319]: mon.a (mon.0) 3182 : cluster [DBG] 2021-02-10T05:01:23.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[29319]: osdmap e1419: 8 total, 8 up, 8 in 2021-02-10T05:01:23.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[29319]: audit 2021-02-10T05 2021-02-10T05:01:23.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[29319]: :01:22.171362+0000 mon.c (mon.1) 180 : audit [INF] 2021-02-10T05:01:23.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[29319]: from='client.? 172.21.15.85:0/2068619320' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T05:01:23.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[29319]: audit 2021-02-10T05:01:23.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[29319]: 2021-02-10T05:01:22.171665+0000 mon.a (mon.0) 3183 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T05:01:23.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[35961]: cluster 2021-02-10T05:01:21.508056+0000 2021-02-10T05:01:23.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[35961]: osd.2 (osd.2) 2068 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:01:23.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[35961]: cluster 2021-02-10T05:01:23.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[35961]: 2021-02-10T05:01:21.509277+0000 osd.2 (osd.2) 2069 : 2021-02-10T05:01:23.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[35961]: cluster [DBG] purged_snaps scrub ok 2021-02-10T05:01:23.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T05:01:23.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[35961]: 10T05:01:21.826642 2021-02-10T05:01:23.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[35961]: +0000 osd.6 (osd.6) 217 : cluster 2021-02-10T05:01:23.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[35961]: [DBG] 4.a scrub ok 2021-02-10T05:01:23.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T05:01:23.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[35961]: T05:01:21.910975+0000 mon.a ( 2021-02-10T05:01:23.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[35961]: mon.0) 3182 : cluster [DBG] osdmap e1419: 8 total, 8 up, 8 in 2021-02-10T05:01:23.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[35961]: audit 2021-02 2021-02-10T05:01:23.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[35961]: -10T05:01:22.171362+0000 mon.c (mon. 2021-02-10T05:01:23.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[35961]: 1) 180 : audit [INF] from='client.? 172.21.15.85:0/2068619320' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T05:01:23.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[35961]: audit 2021 2021-02-10T05:01:23.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[35961]: -02-10T05:01:22.171665+0000 2021-02-10T05:01:23.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[35961]: mon.a (mon.0) 3183 : audit [INF] 2021-02-10T05:01:23.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:22 smithi085 conmon[35961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T05:01:23.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:22 smithi164 conmon[29288]: cluster 2021-02-10T05:01:23.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:22 smithi164 conmon[29288]: 2021-02-10T05:01:23.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:22 smithi164 conmon[29288]: 2021-02-10T05:01:21.508056+0000 osd.2 (osd.2) 2068 : 2021-02-10T05:01:23.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:22 smithi164 conmon[29288]: cluster [DBG] purged_snaps scrub starts 2021-02-10T05:01:23.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:22 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T05:01:23.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:22 smithi164 conmon[29288]: -10T05:01:21.509277+0000 osd.2 (osd.2) 2069 : cluster [DBG] 2021-02-10T05:01:23.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:22 smithi164 conmon[29288]: purged_snaps scrub ok 2021-02-10T05:01:23.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:22 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T05:01:23.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:22 smithi164 conmon[29288]: -10T05:01:21.826642+0000 osd.6 (osd.6) 217 : cluster [DBG] 2021-02-10T05:01:23.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:22 smithi164 conmon[29288]: 4.a scrub ok 2021-02-10T05:01:23.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:22 smithi164 conmon[29288]: cluster 2021-02-10T05:01:23.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:22 smithi164 conmon[29288]: 2021-02-10T05:01:21.910975+0000 mon.a (mon. 2021-02-10T05:01:23.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:22 smithi164 conmon[29288]: 0) 3182 : cluster [DBG] osdmap e1419: 8 total, 8 up, 8 in 2021-02-10T05:01:23.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:22 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T05:01:23.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:22 smithi164 conmon[29288]: 05:01:22.171362+0000 mon.c (mon.1) 180 2021-02-10T05:01:23.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:22 smithi164 conmon[29288]: : audit [INF] from='client.? 172.21.15.85:0/2068619320' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T05:01:23.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:22 smithi164 conmon[29288]: audit 2021-02-10T05:01:22.171665+0000 mon.a (mon.0) 3183 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T05:01:24.019 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd primary-affinity 6 1 2021-02-10T05:01:24.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[29319]: cluster 2021-02-10T05:01:22.529237+0000 osd.3 (osd.3) 72 : cluster [DBG] 4.10 deep-scrub ok 2021-02-10T05:01:24.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[29319]: cluster 2021-02-10T05:01:22.598789+0000 mgr.y (mgr.14140) 1598 : cluster [DBG] pgmap v2601: 172 pgs: 172 active+clean; 25 KiB data, 388 MiB used, 715 GiB / 715 GiB avail 2021-02-10T05:01:24.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[29319]: cluster 2021-02-10T05:01:22.908268+0000 mon.a (mon.0) 3184 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2021-02-10T05:01:24.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[29319]: cluster 2021-02-10T05:01:22.908301+0000 mon.a (mon.0) 3185 : cluster [INF] Cluster is now healthy 2021-02-10T05:01:24.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[29319]: audit 2021-02-10T05:01:22.914443+0000 mon.a (mon.0) 3186 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2021-02-10T05:01:24.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[29319]: cluster 2021-02-10T05:01:22.914574+0000 mon.a (mon.0) 3187 : cluster [DBG] osdmap e1420: 8 total, 8 up, 8 in 2021-02-10T05:01:24.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[29319]: cluster 2021-02-10T05:01:23.112613+0000 osd.5 (osd.5) 229 : cluster [DBG] 6.d scrub ok 2021-02-10T05:01:24.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[29319]: audit 2021-02-10T05:01:23.122241+0000 mon.a (mon.0) 3188 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T05:01:24.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[29319]: cluster 2021-02-10T05:01:23.691181+0000 mon.a (mon.0) 3189 : cluster [DBG] osdmap e1421: 8 total, 8 up, 8 in 2021-02-10T05:01:24.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[35961]: cluster 2021-02-10T05:01:22.529237 2021-02-10T05:01:24.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[35961]: +0000 osd.3 (osd.3) 72 : cluster [DBG] 4.10 deep-scrub ok 2021-02-10T05:01:24.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T05:01:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[35961]: 10T05:01:22.598789+0000 mgr.y (mgr.14140) 1598 : cluster [DBG] pgmap v2601: 172 pgs: 172 active+clean; 25 KiB data, 388 MiB used, 715 GiB / 715 GiB avail 2021-02-10T05:01:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[35961]: cluster 2021-02-10T05 2021-02-10T05:01:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[35961]: :01:22.908268+0000 mon.a (mon.0) 3184 : 2021-02-10T05:01:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[35961]: cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2021-02-10T05:01:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[35961]: cluster 2021-02-10T05:01:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[35961]: 2021-02-10T05:01:22.908301+0000 mon.a (mon.0 2021-02-10T05:01:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[35961]: ) 3185 : cluster [INF] Cluster is now healthy 2021-02-10T05:01:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[35961]: audit 2021-02-10T05:01: 2021-02-10T05:01:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[35961]: 22.914443+0000 mon.a (mon.0) 3186 : audit 2021-02-10T05:01:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[35961]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2021-02-10T05:01:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[35961]: cluster 2021-02-10T05:01:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[35961]: 2021-02-10T05:01:22.914574+0000 mon.a (mon.0) 3187 : cluster [DBG] 2021-02-10T05:01:24.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[35961]: osdmap e1420: 8 total, 8 up, 8 in 2021-02-10T05:01:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[35961]: cluster 2021-02-10T05:01:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[35961]: 2021-02-10T05:01:23.112613+0000 osd.5 ( 2021-02-10T05:01:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[35961]: osd.5) 229 : cluster [DBG] 6.d scrub ok 2021-02-10T05:01:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T05:01:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[35961]: 10T05:01:23.122241+0000 mon.a ( 2021-02-10T05:01:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[35961]: mon.0) 3188 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T05:01:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T05:01:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[35961]: 10T05:01:23.691181+0000 mon.a 2021-02-10T05:01:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[35961]: (mon.0) 3189 : cluster [DBG] 2021-02-10T05:01:24.194 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:23 smithi085 conmon[35961]: osdmap e1421: 8 total, 8 up, 8 in 2021-02-10T05:01:24.227 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:01:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:23 smithi164 conmon[29288]: cluster 2021-02-10T05:01:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:23 smithi164 conmon[29288]: 2021-02-10T05:01:22.529237+0000 osd.3 (osd.3) 72 : cluster [DBG] 4.10 deep-scrub ok 2021-02-10T05:01:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:23 smithi164 conmon[29288]: cluster 2021-02-10T05:01:22.598789+0000 2021-02-10T05:01:24.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:23 smithi164 conmon[29288]: mgr.y (mgr.14140) 1598 : cluster [DBG] pgmap v2601: 172 pgs: 172 active+clean; 25 KiB data, 388 MiB used, 715 GiB / 715 GiB avail 2021-02-10T05:01:24.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:23 smithi164 conmon[29288]: cluster 2021-02-10T05:01 2021-02-10T05:01:24.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:23 smithi164 conmon[29288]: :22.908268+0000 mon.a (mon.0) 3184 2021-02-10T05:01:24.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:23 smithi164 conmon[29288]: : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2021-02-10T05:01:24.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:23 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T05:01:24.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:23 smithi164 conmon[29288]: 05:01:22.908301+0000 mon.a ( 2021-02-10T05:01:24.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:23 smithi164 conmon[29288]: mon.0) 3185 : cluster [INF] Cluster is now healthy 2021-02-10T05:01:24.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:23 smithi164 conmon[29288]: audit 2021-02-10T05:01:24.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:23 smithi164 conmon[29288]: 2021-02-10T05:01 2021-02-10T05:01:24.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:23 smithi164 conmon[29288]: :22.914443+0000 mon.a (mon.0) 3186 2021-02-10T05:01:24.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:23 smithi164 conmon[29288]: : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2021-02-10T05:01:24.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:23 smithi164 conmon[29288]: cluster 2021-02-10T05:01:24.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:23 smithi164 conmon[29288]: 2021-02-10T05:01:22.914574+0000 2021-02-10T05:01:24.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:23 smithi164 conmon[29288]: mon.a (mon.0) 3187 : cluster [DBG] 2021-02-10T05:01:24.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:23 smithi164 conmon[29288]: osdmap e1420: 8 total, 8 up, 8 in 2021-02-10T05:01:24.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:23 smithi164 conmon[29288]: cluster 2021-02-10T05:01:23 2021-02-10T05:01:24.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:23 smithi164 conmon[29288]: .112613+0000 osd.5 (osd.5) 2021-02-10T05:01:24.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:23 smithi164 conmon[29288]: 229 : cluster [DBG] 6.d scrub ok 2021-02-10T05:01:24.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:23 smithi164 conmon[29288]: audit 2021-02-10 2021-02-10T05:01:24.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:23 smithi164 conmon[29288]: T05:01:23.122241+0000 mon.a (mon 2021-02-10T05:01:24.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:23 smithi164 conmon[29288]: .0) 3188 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' 2021-02-10T05:01:24.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:23 smithi164 conmon[29288]: cluster 2021- 2021-02-10T05:01:24.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:23 smithi164 conmon[29288]: 02-10T05:01:23.691181+0000 2021-02-10T05:01:24.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:23 smithi164 conmon[29288]: mon.a (mon.0) 3189 : cluster [DBG] osdmap e1421: 8 total, 8 up, 8 in 2021-02-10T05:01:24.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:23 smithi164 conmon[29288]: 2021-02-10T05:01:25.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:24 smithi085 conmon[29319]: cluster 2021-02-10T05:01:23.549822+0000 osd.4 (osd.4 2021-02-10T05:01:25.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:24 smithi085 conmon[29319]: ) 2702 : cluster [DBG] 4.1b scrub ok 2021-02-10T05:01:25.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:24 smithi085 conmon[29319]: cluster 2021-02-10T05:01:23.847364+0000 osd.6 (osd.6) 218 : cluster [DBG] 4.1f scrub ok 2021-02-10T05:01:25.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:24 smithi085 conmon[29319]: cluster 2021-02-10T05:01:24.361419+0000 osd.7 (osd.7) 125 : cluster [DBG] 5.1a deep-scrub ok 2021-02-10T05:01:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:24 smithi085 conmon[35961]: cluster 2021-02-10T05:01:23. 2021-02-10T05:01:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:24 smithi085 conmon[35961]: 549822+0000 osd.4 (osd.4) 2702 : cluster [DBG] 4.1b scrub ok 2021-02-10T05:01:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:24 smithi085 conmon[35961]: cluster 2021-02-10T05:01:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:24 smithi085 conmon[35961]: 2021-02-10T05:01:23.847364+0000 osd.6 (osd 2021-02-10T05:01:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:24 smithi085 conmon[35961]: .6) 218 : cluster [DBG] 4.1f scrub ok 2021-02-10T05:01:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:24 smithi085 conmon[35961]: cluster 2021-02-10T05:01:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:24 smithi085 conmon[35961]: 2021-02-10T05:01:24.361419+0000 2021-02-10T05:01:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:24 smithi085 conmon[35961]: osd.7 (osd.7) 125 : cluster [DBG] 2021-02-10T05:01:25.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:24 smithi085 conmon[35961]: 5.1a deep-scrub ok 2021-02-10T05:01:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:24 smithi164 conmon[29288]: cluster 2021-02-10T05:01:23.549822+0000 osd.4 (osd.4) 2702 : cluster 2021-02-10T05:01:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:24 smithi164 conmon[29288]: [DBG] 4.1b scrub ok 2021-02-10T05:01:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:24 smithi164 conmon[29288]: cluster 2021-02-10T05:01:23.847364+0000 osd.6 (osd.6) 218 : cluster [DBG] 4.1f scrub ok 2021-02-10T05:01:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:24 smithi164 conmon[29288]: cluster 2021-02-10T05:01:24.361419+0000 2021-02-10T05:01:25.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:24 smithi164 conmon[29288]: osd.7 (osd.7) 125 : cluster [DBG] 5.1a deep-scrub ok 2021-02-10T05:01:26.031 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:25 smithi085 conmon[29319]: cluster 2021-02-10T05:01:24.593818+0000 osd.4 (osd.4) 2703 : cluster [DBG] 6.17 scrub ok 2021-02-10T05:01:26.031 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:25 smithi085 conmon[29319]: 2021-02-10T05:01:26.031 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:25 smithi085 conmon[29319]: cluster 2021-02-10T05:01:24.599194+0000 mgr.y (mgr.14140) 1599 : cluster [DBG] pgmap v2604: 172 pgs: 172 active+clean; 26 KiB data, 387 MiB used, 715 GiB / 715 GiB avail 2021-02-10T05:01:26.031 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:25 smithi085 conmon[35961]: cluster 2021-02-10T05:01:24 2021-02-10T05:01:26.031 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:25 smithi085 conmon[35961]: .593818+0000 osd.4 (osd.4) 2703 : cluster [DBG] 6.17 scrub ok 2021-02-10T05:01:26.032 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:25 smithi085 conmon[35961]: cluster 2021-02-10T 2021-02-10T05:01:26.032 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:25 smithi085 conmon[35961]: 05:01:24.599194+0000 mgr.y (mgr.14140) 2021-02-10T05:01:26.032 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:25 smithi085 conmon[35961]: 1599 : cluster [DBG] pgmap v2604: 172 pgs: 172 active+clean; 26 KiB data, 387 MiB used, 715 GiB / 715 GiB avail 2021-02-10T05:01:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:25 smithi164 conmon[29288]: cluster 2021-02-10T05:01:24.593818+0000 2021-02-10T05:01:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:25 smithi164 conmon[29288]: osd.4 (osd.4) 2703 : cluster [DBG] 6.17 scrub ok 2021-02-10T05:01:26.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:25 smithi164 conmon[29288]: cluster 2021-02-10T05:01:24.599194+0000 mgr.y (mgr.14140) 1599 : cluster [DBG] pgmap v2604: 172 pgs: 172 active+clean; 26 KiB data, 387 MiB used, 715 GiB / 715 GiB avail 2021-02-10T05:01:26.937 INFO:teuthology.orchestra.run.smithi085.stderr:set osd.6 primary-affinity to 1 (8655362) 2021-02-10T05:01:27.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:26 smithi085 conmon[35961]: cluster 2021-02-10T05:01:25.905906+0000 osd.6 (osd.6) 219 : cluster [DBG] 4.5 deep-scrub ok 2021-02-10T05:01:27.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:26 smithi085 conmon[35961]: cluster 2021-02-10T05:01:26.325897+0000 osd.7 (osd.7) 126 : cluster [DBG] 5.9 scrub ok 2021-02-10T05:01:27.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:26 smithi085 conmon[35961]: audit 2021-02-10T05:01:26.471798+0000 mon.c (mon.1) 181 : audit [INF] from='client.? 172.21.15.85:0/2652646949' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T05:01:27.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:26 smithi085 conmon[35961]: audit 2021-02-10T05:01:26.472153+0000 mon.a (mon.0) 3190 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T05:01:27.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:26 smithi085 conmon[29319]: cluster 2021-02-10T05:01:25.905906+0000 osd.6 (osd.6) 219 : cluster [DBG] 4.5 deep-scrub ok 2021-02-10T05:01:27.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:26 smithi085 conmon[29319]: cluster 2021-02-10T05:01:26.325897+0000 osd.7 (osd.7) 126 : cluster [DBG] 5.9 scrub ok 2021-02-10T05:01:27.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:26 smithi085 conmon[29319]: audit 2021-02-10T05:01:26.471798+0000 mon.c (mon.1) 181 : audit [INF] from='client.? 172.21.15.85:0/2652646949' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T05:01:27.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:26 smithi085 conmon[29319]: audit 2021-02-10T05:01:26.472153+0000 mon.a (mon.0) 3190 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T05:01:27.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:26 smithi164 conmon[29288]: cluster 2021-02-10T05:01:25.905906+0000 osd.6 (osd.6) 219 : 2021-02-10T05:01:27.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:26 smithi164 conmon[29288]: cluster [DBG] 4.5 deep-scrub ok 2021-02-10T05:01:27.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:26 smithi164 conmon[29288]: cluster 2021-02-10T05:01:26.325897+0000 osd.7 (osd.7) 126 : cluster [DBG] 5.9 scrub ok 2021-02-10T05:01:27.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:26 smithi164 conmon[29288]: audit 2021-02 2021-02-10T05:01:27.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:26 smithi164 conmon[29288]: -10T05:01:26.471798+0000 mon.c (mon.1) 181 : audit [INF] from='client.? 172.21.15.85:0/2652646949' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T05:01:27.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:26 smithi164 conmon[29288]: audit 2021-02-10T05:01 2021-02-10T05:01:27.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:26 smithi164 conmon[29288]: :26.472153+0000 mon.a (mon.0) 3190 : audit 2021-02-10T05:01:27.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:26 smithi164 conmon[29288]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2021-02-10T05:01:27.908 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 7 1 2021-02-10T05:01:28.120 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:01:28.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:27 smithi085 conmon[29319]: cluster 2021-02-10T05:01:26.599650+0000 mgr.y (mgr.14140) 1600 : cluster [DBG] pgmap v2605: 172 pgs: 172 active+clean; 26 KiB data, 386 MiB used, 715 GiB / 715 GiB avail 2021-02-10T05:01:28.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:27 smithi085 conmon[29319]: audit 2021-02-10T05:01:26.931712+0000 mon.a (mon.0) 3191 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2021-02-10T05:01:28.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:27 smithi085 conmon[29319]: cluster 2021-02-10T05:01:26.931795+0000 mon.a (mon.0) 3192 : cluster [DBG] osdmap e1422: 8 total, 8 up, 8 in 2021-02-10T05:01:28.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:27 smithi085 conmon[29319]: cluster 2021-02-10T05:01:26.945675+0000 osd.2 (osd.2) 2070 : cluster [DBG] 5.a scrub ok 2021-02-10T05:01:28.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:27 smithi085 conmon[29319]: cluster 2021-02-10T05:01:27.434792+0000 osd.3 (osd.3) 73 : cluster [DBG] 6.c scrub ok 2021-02-10T05:01:28.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:27 smithi085 conmon[35961]: cluster 2021-02-10T05:01:26.599650+0000 mgr.y ( 2021-02-10T05:01:28.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:27 smithi085 conmon[35961]: mgr.14140) 1600 : cluster [DBG] pgmap v2605: 172 pgs: 172 active+clean; 26 KiB data, 386 MiB used, 715 GiB / 715 GiB avail 2021-02-10T05:01:28.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:27 smithi085 conmon[35961]: audit 2021-02-10T05:01:26.931712+0000 2021-02-10T05:01:28.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:27 smithi085 conmon[35961]: mon.a (mon.0) 3191 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2021-02-10T05:01:28.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:27 smithi085 conmon[35961]: cluster 2021 2021-02-10T05:01:28.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:27 smithi085 conmon[35961]: -02-10T05:01:26.931795+0000 mon.a (mon.0) 3192 : cluster [DBG] osdmap e1422: 8 total, 8 up, 8 in 2021-02-10T05:01:28.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:27 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T05:01:28.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:27 smithi085 conmon[35961]: 10T05:01:26.945675+0000 osd.2 (osd.2) 2021-02-10T05:01:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:27 smithi085 conmon[35961]: 2070 : cluster [DBG] 5.a scrub ok 2021-02-10T05:01:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:27 smithi085 conmon[35961]: cluster 2021-02-10T05:01:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:27 smithi085 conmon[35961]: 2021-02-10T05:01:27.434792+0000 osd.3 (osd.3 2021-02-10T05:01:28.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:27 smithi085 conmon[35961]: ) 73 : cluster [DBG] 6.c scrub ok 2021-02-10T05:01:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:27 smithi164 conmon[29288]: cluster 2021-02-10T05:01:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:27 smithi164 conmon[29288]: 2021-02-10T05:01: 2021-02-10T05:01:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:27 smithi164 conmon[29288]: 26.599650+0000 mgr.y (mgr.14140) 1600 : cluster [DBG] pgmap v2605: 172 pgs: 172 active+clean; 26 KiB data, 386 MiB used, 715 GiB / 715 GiB avail 2021-02-10T05:01:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:27 smithi164 conmon[29288]: audit 2021-02-10T05:01:28.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:27 smithi164 conmon[29288]: 2021-02-10T05:01: 2021-02-10T05:01:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:27 smithi164 conmon[29288]: 26.931712+0000 mon.a (mon.0) 3191 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2021-02-10T05:01:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:27 smithi164 conmon[29288]: cluster 2021-02-10T05 2021-02-10T05:01:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:27 smithi164 conmon[29288]: :01:26.931795+0000 mon.a (mon.0) 3192 : cluster [DBG] 2021-02-10T05:01:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:27 smithi164 conmon[29288]: osdmap e1422: 8 total, 8 up, 8 in 2021-02-10T05:01:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:27 smithi164 conmon[29288]: cluster 2021- 2021-02-10T05:01:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:27 smithi164 conmon[29288]: 02-10T05:01:26.945675+0000 osd.2 (osd.2) 2070 : cluster [DBG] 5.a scrub ok 2021-02-10T05:01:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:27 smithi164 conmon[29288]: 2021-02-10T05:01:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:27 smithi164 conmon[29288]: cluster 2021 2021-02-10T05:01:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:27 smithi164 conmon[29288]: -02-10T05:01:27.434792+0000 osd.3 (osd.3) 2021-02-10T05:01:28.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:27 smithi164 conmon[29288]: 73 : cluster [DBG] 6.c scrub ok 2021-02-10T05:01:29.191 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:28 smithi085 conmon[29319]: cluster 2021-02-10T05:01:27.657698+0000 osd.0 (osd.0) 34 2021-02-10T05:01:29.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:28 smithi085 conmon[29319]: : cluster [DBG] 2.13 scrub ok 2021-02-10T05:01:29.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:28 smithi085 conmon[29319]: cluster 2021-02-10T05:01:28.694097+0000 mon.a (mon.0) 3193 : cluster [DBG] osdmap e1423: 8 total, 8 up, 8 in 2021-02-10T05:01:29.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:28 smithi085 conmon[35961]: cluster 2021-02-10T05:01:27.657698+0000 2021-02-10T05:01:29.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:28 smithi085 conmon[35961]: osd.0 (osd.0) 34 : cluster [DBG] 2.13 scrub ok 2021-02-10T05:01:29.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:28 smithi085 conmon[35961]: cluster 2021- 2021-02-10T05:01:29.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:28 smithi085 conmon[35961]: 02-10T05:01:28.694097+0000 mon.a (mon.0) 3193 : cluster 2021-02-10T05:01:29.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:28 smithi085 conmon[35961]: [DBG] osdmap e1423: 8 total, 8 up, 8 in 2021-02-10T05:01:29.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:28 smithi164 conmon[29288]: cluster 2021-02-10T05:01:27.657698+0000 osd.0 (osd 2021-02-10T05:01:29.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:28 smithi164 conmon[29288]: .0) 34 : cluster [DBG] 2.13 scrub ok 2021-02-10T05:01:29.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:28 smithi164 conmon[29288]: cluster 2021-02-10T05:01:28.694097+0000 mon.a (mon.0) 3193 : cluster [DBG] osdmap e1423: 8 total, 8 up, 8 in 2021-02-10T05:01:29.678 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 05:01:29 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T05:01:29.945 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:29 smithi164 conmon[29288]: cluster 2021-02-10T05:01:28.600210+0000 mgr.y ( 2021-02-10T05:01:29.945 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:29 smithi164 conmon[29288]: mgr.14140) 1601 : cluster [DBG] pgmap v2607: 172 pgs: 172 active+clean; 25 KiB data, 386 MiB used, 715 GiB / 715 GiB avail 2021-02-10T05:01:29.945 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:29 smithi164 conmon[29288]: cluster 2021-02-10T05:01:29.377735+0000 osd.3 (osd.3) 74 : cluster [DBG] 2.23 scrub ok 2021-02-10T05:01:30.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:29 smithi085 conmon[35961]: cluster 2021-02-10T05:01:28.600210+0000 mgr.y (mgr.14140) 2021-02-10T05:01:30.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:29 smithi085 conmon[35961]: 1601 : cluster [DBG] pgmap v2607: 172 pgs: 172 active+clean; 25 KiB data, 386 MiB used, 715 GiB / 715 GiB avail 2021-02-10T05:01:30.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:29 smithi085 conmon[35961]: cluster 2021-02-10T05:01:29.377735+0000 osd.3 (osd.3) 74 : cluster 2021-02-10T05:01:30.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:29 smithi085 conmon[35961]: [DBG] 2.23 scrub ok 2021-02-10T05:01:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:29 smithi085 conmon[29319]: cluster 2021-02-10T05:01:28.600210+0000 mgr.y (mgr 2021-02-10T05:01:30.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:29 smithi085 conmon[29319]: .14140) 1601 : cluster [DBG] pgmap v2607: 172 pgs: 172 active+clean; 25 KiB data, 386 MiB used, 715 GiB / 715 GiB avail 2021-02-10T05:01:30.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:29 smithi085 conmon[29319]: cluster 2021-02-10T05:01:29.377735+0000 osd.3 (osd.3) 74 : cluster [DBG] 2.23 scrub ok 2021-02-10T05:01:30.809 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2021-02-10T05:01:30.809 INFO:teuthology.orchestra.run.smithi164.stdout: "size": 20, 2021-02-10T05:01:30.809 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 600, 2021-02-10T05:01:30.809 INFO:teuthology.orchestra.run.smithi164.stdout: "ops": [ 2021-02-10T05:01:30.809 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.809 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDScrubReserve(6.1b RELEASE e1135)", 2021-02-10T05:01:30.809 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:51:59.838977+0000", 2021-02-10T05:01:30.809 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 570.96708215399997, 2021-02-10T05:01:30.809 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00091688999999999996, 2021-02-10T05:01:30.810 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:01:30.810 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T05:01:30.810 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:01:30.810 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.810 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:01:30.810 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:59.838977+0000", 2021-02-10T05:01:30.810 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:01:30.810 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.810 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.810 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:01:30.810 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:59.838977+0000", 2021-02-10T05:01:30.810 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.3427999999999999e-05 2021-02-10T05:01:30.810 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.811 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.811 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:01:30.811 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:59.839010+0000", 2021-02-10T05:01:30.811 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.5780000000000004e-06 2021-02-10T05:01:30.811 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.811 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.811 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:01:30.811 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:59.839017+0000", 2021-02-10T05:01:30.811 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1941e-05 2021-02-10T05:01:30.811 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.811 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.811 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:01:30.811 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:59.839029+0000", 2021-02-10T05:01:30.812 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0549999999999996e-06 2021-02-10T05:01:30.812 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.812 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.812 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:01:30.812 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:59.839035+0000", 2021-02-10T05:01:30.812 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.7172999999999997e-05 2021-02-10T05:01:30.812 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.812 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.812 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:01:30.812 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:59.839082+0000", 2021-02-10T05:01:30.812 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0451999999999999e-05 2021-02-10T05:01:30.813 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.813 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.813 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2021-02-10T05:01:30.813 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:59.839093+0000", 2021-02-10T05:01:30.813 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00078892700000000003 2021-02-10T05:01:30.813 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.813 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.813 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:01:30.813 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:59.839881+0000", 2021-02-10T05:01:30.813 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.3559999999999999e-06 2021-02-10T05:01:30.813 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.813 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.814 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T05:01:30.814 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:59.839891+0000", 2021-02-10T05:01:30.814 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.9799999999999998e-06 2021-02-10T05:01:30.814 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.814 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.814 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:01:30.814 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:51:59.839894+0000", 2021-02-10T05:01:30.814 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00091688999999999996 2021-02-10T05:01:30.814 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.814 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:01:30.814 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.814 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.814 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.815 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_repop(client.14817.0:1822 7.8 e1150/1118 7:10dac65a:::smithi08575397-39:head v 1150'76, mlcod=0'0)", 2021-02-10T05:01:30.815 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:52:41.692373+0000", 2021-02-10T05:01:30.815 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 529.11368580199996, 2021-02-10T05:01:30.815 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.015071337000000001, 2021-02-10T05:01:30.815 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:01:30.815 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T05:01:30.815 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:01:30.815 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.815 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:01:30.815 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.692373+0000", 2021-02-10T05:01:30.815 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:01:30.815 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.815 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.816 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:01:30.816 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.692373+0000", 2021-02-10T05:01:30.816 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0074515270000000003 2021-02-10T05:01:30.816 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.816 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.816 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:01:30.816 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.699825+0000", 2021-02-10T05:01:30.816 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1279e-05 2021-02-10T05:01:30.816 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.816 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.816 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:01:30.816 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.699836+0000", 2021-02-10T05:01:30.817 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5341000000000001e-05 2021-02-10T05:01:30.817 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.817 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.817 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:01:30.817 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.699851+0000", 2021-02-10T05:01:30.817 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.2879999999999998e-06 2021-02-10T05:01:30.817 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.817 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.817 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:01:30.817 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.699861+0000", 2021-02-10T05:01:30.817 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.9645999999999998e-05 2021-02-10T05:01:30.817 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.817 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.818 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:01:30.818 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.699920+0000", 2021-02-10T05:01:30.818 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.5575000000000002e-05 2021-02-10T05:01:30.818 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.818 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.818 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T05:01:30.818 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.699946+0000", 2021-02-10T05:01:30.818 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0074493550000000004 2021-02-10T05:01:30.818 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.818 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.818 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "commit_sent", 2021-02-10T05:01:30.818 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.707395+0000", 2021-02-10T05:01:30.819 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9326000000000002e-05 2021-02-10T05:01:30.819 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.819 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.819 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:01:30.819 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:52:41.707445+0000", 2021-02-10T05:01:30.819 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.015071337000000001 2021-02-10T05:01:30.819 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.819 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:01:30.819 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.819 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.819 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.819 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDScrubReserve(4.15 REQUEST e1162)", 2021-02-10T05:01:30.819 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:53:10.268558+0000", 2021-02-10T05:01:30.820 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 500.537501464, 2021-02-10T05:01:30.820 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0040140230000000002, 2021-02-10T05:01:30.820 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:01:30.820 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T05:01:30.820 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:01:30.820 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.820 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:01:30.820 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:10.268558+0000", 2021-02-10T05:01:30.820 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:01:30.820 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.820 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.820 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:01:30.820 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:10.268558+0000", 2021-02-10T05:01:30.821 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.0046e-05 2021-02-10T05:01:30.821 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.821 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.821 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:01:30.821 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:10.268588+0000", 2021-02-10T05:01:30.821 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.3389999999999998e-06 2021-02-10T05:01:30.821 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.821 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.821 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:01:30.821 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:10.268594+0000", 2021-02-10T05:01:30.821 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7350000000000008e-06 2021-02-10T05:01:30.821 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.821 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.822 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:01:30.822 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:10.268604+0000", 2021-02-10T05:01:30.822 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.0740000000000004e-06 2021-02-10T05:01:30.822 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.822 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.822 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:01:30.822 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:10.268611+0000", 2021-02-10T05:01:30.822 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.6780000000000002e-05 2021-02-10T05:01:30.822 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.822 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.822 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:01:30.822 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:10.268668+0000", 2021-02-10T05:01:30.822 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.9339999999999995e-06 2021-02-10T05:01:30.823 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.823 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.823 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2021-02-10T05:01:30.823 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:10.268677+0000", 2021-02-10T05:01:30.823 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.003831119 2021-02-10T05:01:30.823 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.823 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.823 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:01:30.823 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:10.272508+0000", 2021-02-10T05:01:30.823 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5756999999999998e-05 2021-02-10T05:01:30.823 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.823 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.823 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T05:01:30.824 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:10.272523+0000", 2021-02-10T05:01:30.824 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.8238999999999997e-05 2021-02-10T05:01:30.824 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.824 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.824 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:01:30.824 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:10.272572+0000", 2021-02-10T05:01:30.824 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0040140230000000002 2021-02-10T05:01:30.824 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.824 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:01:30.824 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.824 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.824 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.825 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDScrubReserve(2.e REQUEST e1187)", 2021-02-10T05:01:30.825 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:53:41.711583+0000", 2021-02-10T05:01:30.825 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 469.09447626799999, 2021-02-10T05:01:30.825 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0079769279999999994, 2021-02-10T05:01:30.825 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:01:30.825 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T05:01:30.825 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:01:30.825 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.825 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:01:30.825 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:41.711583+0000", 2021-02-10T05:01:30.825 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:01:30.825 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.825 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.826 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:01:30.826 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:41.711583+0000", 2021-02-10T05:01:30.826 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5846e-05 2021-02-10T05:01:30.826 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.826 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.826 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:01:30.826 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:41.711599+0000", 2021-02-10T05:01:30.826 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.6509999999999999e-06 2021-02-10T05:01:30.826 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.826 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.826 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:01:30.826 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:41.711602+0000", 2021-02-10T05:01:30.827 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.8429999999999998e-06 2021-02-10T05:01:30.827 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.827 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.827 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:01:30.827 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:41.711608+0000", 2021-02-10T05:01:30.827 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.8400000000000002e-06 2021-02-10T05:01:30.827 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.827 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.827 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:01:30.827 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:41.711613+0000", 2021-02-10T05:01:30.827 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.8625999999999998e-05 2021-02-10T05:01:30.827 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.827 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.828 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:01:30.828 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:41.711662+0000", 2021-02-10T05:01:30.828 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.0850000000000003e-06 2021-02-10T05:01:30.828 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.828 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.828 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2021-02-10T05:01:30.828 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:41.711671+0000", 2021-02-10T05:01:30.828 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.007828425 2021-02-10T05:01:30.828 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.828 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.828 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:01:30.828 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:41.719499+0000", 2021-02-10T05:01:30.829 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4219000000000001e-05 2021-02-10T05:01:30.829 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.829 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.829 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T05:01:30.829 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:41.719513+0000", 2021-02-10T05:01:30.829 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.6393000000000001e-05 2021-02-10T05:01:30.829 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.829 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.829 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:01:30.829 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:53:41.719560+0000", 2021-02-10T05:01:30.829 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0079769279999999994 2021-02-10T05:01:30.829 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.829 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:01:30.829 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.830 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.830 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.830 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDScrubReserve(7.13 REQUEST e1233)", 2021-02-10T05:01:30.830 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:55:22.324141+0000", 2021-02-10T05:01:30.830 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 368.48191824600002, 2021-02-10T05:01:30.830 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001646387, 2021-02-10T05:01:30.830 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:01:30.830 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T05:01:30.830 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:01:30.830 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.830 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:01:30.830 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:22.324141+0000", 2021-02-10T05:01:30.830 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:01:30.831 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.831 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.831 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:01:30.831 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:22.324141+0000", 2021-02-10T05:01:30.831 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.6648e-05 2021-02-10T05:01:30.831 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.831 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.831 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:01:30.831 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:22.324167+0000", 2021-02-10T05:01:30.831 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.1410000000000004e-06 2021-02-10T05:01:30.831 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.831 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.832 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:01:30.832 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:22.324173+0000", 2021-02-10T05:01:30.832 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.4479999999999994e-06 2021-02-10T05:01:30.832 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.832 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.832 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:01:30.832 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:22.324181+0000", 2021-02-10T05:01:30.832 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.7630000000000003e-06 2021-02-10T05:01:30.832 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.832 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.832 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:01:30.832 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:22.324188+0000", 2021-02-10T05:01:30.832 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.1309000000000005e-05 2021-02-10T05:01:30.833 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.833 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.833 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:01:30.833 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:22.324249+0000", 2021-02-10T05:01:30.833 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.4956999999999999e-05 2021-02-10T05:01:30.833 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.833 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.833 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2021-02-10T05:01:30.833 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:22.324264+0000", 2021-02-10T05:01:30.833 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0014424869999999999 2021-02-10T05:01:30.833 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.833 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.834 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:01:30.834 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:22.325707+0000", 2021-02-10T05:01:30.834 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.3358999999999997e-05 2021-02-10T05:01:30.834 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.834 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.834 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T05:01:30.834 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:22.325740+0000", 2021-02-10T05:01:30.834 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.7274999999999997e-05 2021-02-10T05:01:30.834 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.834 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.834 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:01:30.834 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:22.325787+0000", 2021-02-10T05:01:30.835 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001646387 2021-02-10T05:01:30.835 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.835 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:01:30.835 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.835 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.835 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.835 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.24373.0:3882 5.9 5:93e5b521:::notify.7:head [watch ping cookie 94187770081792 gen 2] snapc 0=[] ondisk+write+known_if_redirected e1241)", 2021-02-10T05:01:30.835 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:55:38.037757+0000", 2021-02-10T05:01:30.835 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 352.76830189700001, 2021-02-10T05:01:30.835 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00085687000000000003, 2021-02-10T05:01:30.835 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:01:30.835 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T05:01:30.835 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T05:01:30.836 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.24373", 2021-02-10T05:01:30.836 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T05:01:30.836 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 3882 2021-02-10T05:01:30.836 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.836 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:01:30.836 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.836 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:01:30.836 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:38.037757+0000", 2021-02-10T05:01:30.836 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:01:30.836 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.836 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.836 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:01:30.837 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:38.037757+0000", 2021-02-10T05:01:30.837 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.3728000000000001e-05 2021-02-10T05:01:30.837 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.837 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.837 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:01:30.837 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:38.037781+0000", 2021-02-10T05:01:30.837 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.9559999999999997e-06 2021-02-10T05:01:30.837 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.837 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.837 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:01:30.837 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:38.037789+0000", 2021-02-10T05:01:30.837 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7234000000000001e-05 2021-02-10T05:01:30.837 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.838 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.838 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:01:30.838 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:38.037806+0000", 2021-02-10T05:01:30.838 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.695e-06 2021-02-10T05:01:30.838 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.838 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.838 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:01:30.838 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:38.037814+0000", 2021-02-10T05:01:30.838 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000580241 2021-02-10T05:01:30.838 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.838 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.838 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:01:30.839 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:38.038394+0000", 2021-02-10T05:01:30.839 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7225999999999994e-05 2021-02-10T05:01:30.839 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.839 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.839 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T05:01:30.839 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:38.038491+0000", 2021-02-10T05:01:30.839 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00012279000000000001 2021-02-10T05:01:30.839 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.839 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.839 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:01:30.839 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:55:38.038614+0000", 2021-02-10T05:01:30.839 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00085687000000000003 2021-02-10T05:01:30.839 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.840 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:01:30.840 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.840 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.840 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.840 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDScrubReserve(6.15 REJECT e1287)", 2021-02-10T05:01:30.840 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:57:02.371819+0000", 2021-02-10T05:01:30.840 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 268.43424016400002, 2021-02-10T05:01:30.840 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0019610510000000001, 2021-02-10T05:01:30.840 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:01:30.840 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T05:01:30.840 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:01:30.840 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.840 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:01:30.841 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:57:02.371819+0000", 2021-02-10T05:01:30.841 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:01:30.841 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.841 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.841 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:01:30.841 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:57:02.371819+0000", 2021-02-10T05:01:30.841 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.0576e-05 2021-02-10T05:01:30.841 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.841 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.841 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:01:30.841 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:57:02.371849+0000", 2021-02-10T05:01:30.841 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0789999999999997e-06 2021-02-10T05:01:30.841 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.841 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.842 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:01:30.842 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:57:02.371856+0000", 2021-02-10T05:01:30.842 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.9229999999999997e-06 2021-02-10T05:01:30.842 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.842 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.842 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:01:30.842 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:57:02.371865+0000", 2021-02-10T05:01:30.842 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.5520000000000004e-06 2021-02-10T05:01:30.842 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.842 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.842 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:01:30.842 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:57:02.371871+0000", 2021-02-10T05:01:30.842 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001897691 2021-02-10T05:01:30.843 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.843 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.843 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:01:30.843 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:57:02.373769+0000", 2021-02-10T05:01:30.843 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.7189999999999996e-06 2021-02-10T05:01:30.843 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.843 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.843 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T05:01:30.843 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:57:02.373778+0000", 2021-02-10T05:01:30.843 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.511e-06 2021-02-10T05:01:30.843 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.843 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.843 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:01:30.844 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:57:02.373780+0000", 2021-02-10T05:01:30.844 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0019610510000000001 2021-02-10T05:01:30.844 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.844 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:01:30.844 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.844 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.844 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.844 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.24373.0:4193 5.9 5:93e5b521:::notify.7:head [watch ping cookie 94187770081792 gen 2] snapc 0=[] ondisk+write+known_if_redirected e1298)", 2021-02-10T05:01:30.844 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:57:33.058884+0000", 2021-02-10T05:01:30.844 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 237.74717472200001, 2021-02-10T05:01:30.844 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00097157299999999999, 2021-02-10T05:01:30.844 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:01:30.844 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T05:01:30.845 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T05:01:30.845 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.24373", 2021-02-10T05:01:30.845 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T05:01:30.845 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 4193 2021-02-10T05:01:30.845 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.845 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:01:30.845 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.845 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:01:30.845 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:57:33.058884+0000", 2021-02-10T05:01:30.845 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:01:30.845 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.845 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.845 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:01:30.846 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:57:33.058884+0000", 2021-02-10T05:01:30.846 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.0902000000000003e-05 2021-02-10T05:01:30.846 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.846 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.846 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:01:30.846 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:57:33.058915+0000", 2021-02-10T05:01:30.846 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.3939999999999996e-06 2021-02-10T05:01:30.846 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.846 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.846 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:01:30.846 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:57:33.058924+0000", 2021-02-10T05:01:30.846 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.7626e-05 2021-02-10T05:01:30.847 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.847 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.847 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:01:30.847 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:57:33.058941+0000", 2021-02-10T05:01:30.847 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0685e-05 2021-02-10T05:01:30.847 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.847 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.847 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:01:30.847 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:57:33.058952+0000", 2021-02-10T05:01:30.847 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00069688200000000001 2021-02-10T05:01:30.847 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.847 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.848 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:01:30.848 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:57:33.059649+0000", 2021-02-10T05:01:30.848 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000104657 2021-02-10T05:01:30.848 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.848 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.848 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T05:01:30.848 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:57:33.059754+0000", 2021-02-10T05:01:30.848 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000102427 2021-02-10T05:01:30.848 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.848 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.848 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:01:30.848 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:57:33.059856+0000", 2021-02-10T05:01:30.848 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00097157299999999999 2021-02-10T05:01:30.849 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.849 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:01:30.849 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.849 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.849 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.849 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "pg_scan(get_digest 4.17 MIN-MIN e 1345/1344)", 2021-02-10T05:01:30.849 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:59:16.977072+0000", 2021-02-10T05:01:30.849 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 133.82898685699999, 2021-02-10T05:01:30.849 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.033474476000000003, 2021-02-10T05:01:30.849 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:01:30.849 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T05:01:30.849 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:01:30.850 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.850 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:01:30.850 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:59:16.977072+0000", 2021-02-10T05:01:30.850 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:01:30.850 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.850 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.850 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:01:30.850 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:59:16.977072+0000", 2021-02-10T05:01:30.850 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.1934e-05 2021-02-10T05:01:30.851 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.851 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.851 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:01:30.851 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:59:16.977094+0000", 2021-02-10T05:01:30.851 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.9230000000000001e-06 2021-02-10T05:01:30.851 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.851 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.851 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:01:30.851 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:59:16.977099+0000", 2021-02-10T05:01:30.851 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.6730000000000004e-06 2021-02-10T05:01:30.851 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.851 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.851 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:01:30.852 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:59:16.977109+0000", 2021-02-10T05:01:30.852 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.0299999999999999e-06 2021-02-10T05:01:30.852 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.852 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.852 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:01:30.852 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:59:16.977115+0000", 2021-02-10T05:01:30.852 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.6189000000000001e-05 2021-02-10T05:01:30.852 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.852 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.852 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:01:30.852 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:59:16.977161+0000", 2021-02-10T05:01:30.852 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.033190798000000001 2021-02-10T05:01:30.852 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.853 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.853 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T05:01:30.853 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:59:17.010352+0000", 2021-02-10T05:01:30.853 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00019492899999999999 2021-02-10T05:01:30.853 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.853 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.853 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:01:30.853 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:59:17.010547+0000", 2021-02-10T05:01:30.853 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.033474476000000003 2021-02-10T05:01:30.853 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.853 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:01:30.853 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.853 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.854 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.854 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "pg_backfill(progress 4.17 e 1345/1345 lb 4:e9187ab8:reshard::reshard.0000000000:head)", 2021-02-10T05:01:30.854 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T04:59:17.013652+0000", 2021-02-10T05:01:30.854 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 133.79240720999999, 2021-02-10T05:01:30.854 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.042304004999999999, 2021-02-10T05:01:30.854 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:01:30.854 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T05:01:30.854 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:01:30.854 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.854 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:01:30.854 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:59:17.013652+0000", 2021-02-10T05:01:30.854 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:01:30.855 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.855 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.855 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:01:30.855 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:59:17.013652+0000", 2021-02-10T05:01:30.855 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.5284000000000001e-05 2021-02-10T05:01:30.855 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.855 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.855 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:01:30.855 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:59:17.013677+0000", 2021-02-10T05:01:30.855 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.9790000000000005e-06 2021-02-10T05:01:30.855 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.856 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.856 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:01:30.856 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:59:17.013687+0000", 2021-02-10T05:01:30.856 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1096e-05 2021-02-10T05:01:30.856 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.856 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.856 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:01:30.856 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:59:17.013698+0000", 2021-02-10T05:01:30.856 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.7300000000000002e-06 2021-02-10T05:01:30.856 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.856 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.856 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:01:30.856 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:59:17.013704+0000", 2021-02-10T05:01:30.857 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00014804 2021-02-10T05:01:30.857 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.857 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.857 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:01:30.857 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:59:17.013852+0000", 2021-02-10T05:01:30.857 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.042022507000000001 2021-02-10T05:01:30.857 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.857 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.857 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T05:01:30.857 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:59:17.055875+0000", 2021-02-10T05:01:30.857 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1369e-05 2021-02-10T05:01:30.857 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.858 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.858 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:01:30.858 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T04:59:17.055956+0000", 2021-02-10T05:01:30.858 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.042304004999999999 2021-02-10T05:01:30.858 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.858 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:01:30.858 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.858 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.858 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.858 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDScrubReserve(4.0 REQUEST e1374)", 2021-02-10T05:01:30.858 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T05:00:02.803187+0000", 2021-02-10T05:01:30.858 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 88.002872476999997, 2021-02-10T05:01:30.858 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0040305640000000004, 2021-02-10T05:01:30.859 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:01:30.859 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T05:01:30.859 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:01:30.859 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.859 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:01:30.859 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:02.803187+0000", 2021-02-10T05:01:30.859 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:01:30.859 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.859 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.859 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:01:30.859 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:02.803187+0000", 2021-02-10T05:01:30.859 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5829999999999999e-05 2021-02-10T05:01:30.860 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.860 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.860 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:01:30.860 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:02.803202+0000", 2021-02-10T05:01:30.860 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.3479999999999997e-06 2021-02-10T05:01:30.860 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.860 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.860 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:01:30.860 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:02.803207+0000", 2021-02-10T05:01:30.860 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.7800000000000003e-06 2021-02-10T05:01:30.860 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.860 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.860 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:01:30.861 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:02.803214+0000", 2021-02-10T05:01:30.861 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6579999999999996e-06 2021-02-10T05:01:30.861 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.861 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.861 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:01:30.861 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:02.803220+0000", 2021-02-10T05:01:30.861 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.0460999999999997e-05 2021-02-10T05:01:30.861 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.861 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.861 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:01:30.861 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:02.803271+0000", 2021-02-10T05:01:30.861 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.1779999999999992e-06 2021-02-10T05:01:30.861 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.861 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.862 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2021-02-10T05:01:30.862 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:02.803280+0000", 2021-02-10T05:01:30.862 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.003904486 2021-02-10T05:01:30.862 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.862 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.862 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:01:30.862 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:02.807184+0000", 2021-02-10T05:01:30.862 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.8240000000000006e-06 2021-02-10T05:01:30.862 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.862 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.862 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T05:01:30.862 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:02.807193+0000", 2021-02-10T05:01:30.863 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.3998999999999998e-05 2021-02-10T05:01:30.863 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.863 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.863 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:01:30.863 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:02.807217+0000", 2021-02-10T05:01:30.863 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0040305640000000004 2021-02-10T05:01:30.863 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.863 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:01:30.863 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.863 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.863 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.863 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "replica_scrub(pg: 4.0,from:0'0,to:1261'84,epoch:1374/1332,start:4:00000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2021-02-10T05:01:30.863 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T05:00:02.809706+0000", 2021-02-10T05:01:30.864 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 87.996353380000002, 2021-02-10T05:01:30.864 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000840941, 2021-02-10T05:01:30.864 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:01:30.864 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "reached pg", 2021-02-10T05:01:30.864 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:01:30.864 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.864 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:01:30.864 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:02.809706+0000", 2021-02-10T05:01:30.864 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:01:30.864 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.864 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.864 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:01:30.865 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:02.809706+0000", 2021-02-10T05:01:30.865 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.3790000000000001e-05 2021-02-10T05:01:30.865 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.865 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.865 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:01:30.865 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:02.809729+0000", 2021-02-10T05:01:30.865 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.1930000000000001e-06 2021-02-10T05:01:30.865 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.865 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.865 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:01:30.865 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:02.809735+0000", 2021-02-10T05:01:30.865 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.2250000000000006e-06 2021-02-10T05:01:30.865 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.866 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.866 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:01:30.866 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:02.809744+0000", 2021-02-10T05:01:30.866 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.7230000000000004e-06 2021-02-10T05:01:30.866 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.866 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.866 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:01:30.866 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:02.809749+0000", 2021-02-10T05:01:30.866 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.8906000000000003e-05 2021-02-10T05:01:30.866 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.866 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.866 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:01:30.866 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:02.809788+0000", 2021-02-10T05:01:30.867 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00075910400000000003 2021-02-10T05:01:30.867 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.867 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.867 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:01:30.867 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:02.810547+0000", 2021-02-10T05:01:30.867 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000840941 2021-02-10T05:01:30.867 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.867 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:01:30.867 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.867 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.867 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.867 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.24373.0:4469 5.9 5:93e5b521:::notify.7:head [watch ping cookie 94187770081792 gen 3] snapc 0=[] ondisk+write+known_if_redirected e1378)", 2021-02-10T05:01:30.867 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T05:00:18.090883+0000", 2021-02-10T05:01:30.868 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 72.715175614000003, 2021-02-10T05:01:30.868 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001142768, 2021-02-10T05:01:30.868 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:01:30.868 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T05:01:30.868 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T05:01:30.868 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.24373", 2021-02-10T05:01:30.868 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T05:01:30.868 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 4469 2021-02-10T05:01:30.868 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.868 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:01:30.868 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.868 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:01:30.868 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:18.090883+0000", 2021-02-10T05:01:30.869 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:01:30.869 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.869 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.869 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:01:30.869 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:18.090883+0000", 2021-02-10T05:01:30.869 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.4793999999999998e-05 2021-02-10T05:01:30.869 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.869 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.869 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:01:30.869 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:18.090928+0000", 2021-02-10T05:01:30.869 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.6160000000000008e-06 2021-02-10T05:01:30.869 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.869 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.870 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:01:30.870 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:18.090938+0000", 2021-02-10T05:01:30.870 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8229000000000001e-05 2021-02-10T05:01:30.870 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.870 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.870 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:01:30.870 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:18.090956+0000", 2021-02-10T05:01:30.870 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.3680000000000008e-06 2021-02-10T05:01:30.870 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.870 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.870 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:01:30.870 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:18.090965+0000", 2021-02-10T05:01:30.871 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00085457199999999995 2021-02-10T05:01:30.871 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.871 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.871 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:01:30.871 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:18.091820+0000", 2021-02-10T05:01:30.871 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.4064e-05 2021-02-10T05:01:30.871 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.871 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.871 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T05:01:30.871 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:18.091904+0000", 2021-02-10T05:01:30.871 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000122125 2021-02-10T05:01:30.871 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.872 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.872 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:01:30.872 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:18.092026+0000", 2021-02-10T05:01:30.872 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001142768 2021-02-10T05:01:30.872 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.872 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:01:30.872 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.872 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.872 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.872 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "osd_op(client.24373.0:4488 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94187770081792 gen 4] snapc 0=[] ondisk+write+known_if_redirected e1390)", 2021-02-10T05:01:30.872 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T05:00:28.093386+0000", 2021-02-10T05:01:30.872 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 62.712672988000001, 2021-02-10T05:01:30.872 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0022147769999999998, 2021-02-10T05:01:30.873 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:01:30.873 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T05:01:30.873 INFO:teuthology.orchestra.run.smithi164.stdout: "client_info": { 2021-02-10T05:01:30.873 INFO:teuthology.orchestra.run.smithi164.stdout: "client": "client.24373", 2021-02-10T05:01:30.873 INFO:teuthology.orchestra.run.smithi164.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T05:01:30.873 INFO:teuthology.orchestra.run.smithi164.stdout: "tid": 4488 2021-02-10T05:01:30.873 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.873 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:01:30.873 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.873 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:01:30.873 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:28.093386+0000", 2021-02-10T05:01:30.873 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:01:30.873 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.873 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.874 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:01:30.874 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:28.093386+0000", 2021-02-10T05:01:30.874 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.0173000000000001e-05 2021-02-10T05:01:30.874 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.874 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.874 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:01:30.874 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:28.093416+0000", 2021-02-10T05:01:30.874 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.207e-06 2021-02-10T05:01:30.874 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.874 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.874 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:01:30.874 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:28.093422+0000", 2021-02-10T05:01:30.874 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1608e-05 2021-02-10T05:01:30.875 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.875 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.875 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:01:30.875 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:28.093434+0000", 2021-02-10T05:01:30.875 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.5539999999999997e-06 2021-02-10T05:01:30.875 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.875 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.875 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:01:30.875 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:28.093440+0000", 2021-02-10T05:01:30.875 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00066786999999999999 2021-02-10T05:01:30.875 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.875 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.876 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:01:30.876 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:28.094108+0000", 2021-02-10T05:01:30.876 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.7225999999999995e-05 2021-02-10T05:01:30.876 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.876 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.876 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T05:01:30.876 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:28.094195+0000", 2021-02-10T05:01:30.876 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001406139 2021-02-10T05:01:30.876 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.876 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.876 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:01:30.876 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:28.095601+0000", 2021-02-10T05:01:30.876 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0022147769999999998 2021-02-10T05:01:30.876 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.877 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:01:30.877 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.877 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.877 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.877 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDScrubReserve(5.1 REQUEST e1392)", 2021-02-10T05:01:30.877 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T05:00:31.265724+0000", 2021-02-10T05:01:30.877 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 59.540334700000002, 2021-02-10T05:01:30.877 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0026623340000000001, 2021-02-10T05:01:30.877 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:01:30.877 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T05:01:30.877 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:01:30.877 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.877 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:01:30.877 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:31.265724+0000", 2021-02-10T05:01:30.878 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:01:30.878 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.878 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.878 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:01:30.878 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:31.265724+0000", 2021-02-10T05:01:30.878 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.9026e-05 2021-02-10T05:01:30.878 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.878 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.878 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:01:30.878 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:31.265753+0000", 2021-02-10T05:01:30.878 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.2019999999999998e-06 2021-02-10T05:01:30.878 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.879 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.879 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:01:30.879 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:31.265761+0000", 2021-02-10T05:01:30.879 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0575e-05 2021-02-10T05:01:30.879 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.879 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.879 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:01:30.879 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:31.265771+0000", 2021-02-10T05:01:30.879 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0814e-05 2021-02-10T05:01:30.879 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.879 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.879 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:01:30.879 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:31.265782+0000", 2021-02-10T05:01:30.880 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00010477800000000001 2021-02-10T05:01:30.880 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.880 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.880 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:01:30.880 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:31.265887+0000", 2021-02-10T05:01:30.880 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.447e-06 2021-02-10T05:01:30.880 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.880 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.880 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2021-02-10T05:01:30.880 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:31.265894+0000", 2021-02-10T05:01:30.880 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0024535149999999999 2021-02-10T05:01:30.880 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.880 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.881 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:01:30.881 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:31.268348+0000", 2021-02-10T05:01:30.881 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.2810000000000003e-06 2021-02-10T05:01:30.881 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.881 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.881 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T05:01:30.881 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:31.268355+0000", 2021-02-10T05:01:30.881 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1696000000000003e-05 2021-02-10T05:01:30.881 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.881 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.881 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:01:30.881 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:31.268387+0000", 2021-02-10T05:01:30.882 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0026623340000000001 2021-02-10T05:01:30.882 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.882 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:01:30.882 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.882 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.882 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.882 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDScrubReserve(5.1e REQUEST e1393)", 2021-02-10T05:01:30.882 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T05:00:35.144874+0000", 2021-02-10T05:01:30.882 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 55.661184972000001, 2021-02-10T05:01:30.882 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001566472, 2021-02-10T05:01:30.882 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:01:30.882 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T05:01:30.882 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:01:30.883 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.883 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:01:30.883 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:35.144874+0000", 2021-02-10T05:01:30.883 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:01:30.883 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.883 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.883 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:01:30.883 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:35.144874+0000", 2021-02-10T05:01:30.883 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8005000000000001e-05 2021-02-10T05:01:30.883 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.883 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.883 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:01:30.883 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:35.144892+0000", 2021-02-10T05:01:30.884 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.2820000000000004e-06 2021-02-10T05:01:30.884 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.884 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.884 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:01:30.884 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:35.144897+0000", 2021-02-10T05:01:30.884 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.5220000000000001e-06 2021-02-10T05:01:30.884 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.884 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.884 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:01:30.884 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:35.144905+0000", 2021-02-10T05:01:30.884 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.87e-06 2021-02-10T05:01:30.884 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.884 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.885 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:01:30.885 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:35.144910+0000", 2021-02-10T05:01:30.885 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001474447 2021-02-10T05:01:30.885 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.885 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.885 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:01:30.885 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:35.146384+0000", 2021-02-10T05:01:30.885 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8834999999999998e-05 2021-02-10T05:01:30.885 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.885 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.885 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T05:01:30.885 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:35.146403+0000", 2021-02-10T05:01:30.885 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.7511000000000001e-05 2021-02-10T05:01:30.886 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.886 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.886 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:01:30.886 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:35.146441+0000", 2021-02-10T05:01:30.886 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001566472 2021-02-10T05:01:30.886 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.886 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:01:30.886 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.886 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.886 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.886 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDScrubReserve(4.11 REQUEST e1399)", 2021-02-10T05:01:30.886 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T05:00:42.717045+0000", 2021-02-10T05:01:30.886 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 48.089013637000001, 2021-02-10T05:01:30.887 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0016422400000000001, 2021-02-10T05:01:30.887 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:01:30.887 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T05:01:30.887 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:01:30.887 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.887 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:01:30.887 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:42.717045+0000", 2021-02-10T05:01:30.887 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:01:30.887 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.887 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.887 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:01:30.887 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:42.717045+0000", 2021-02-10T05:01:30.888 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.9975e-05 2021-02-10T05:01:30.888 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.888 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.888 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:01:30.888 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:42.717075+0000", 2021-02-10T05:01:30.888 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.8549999999999996e-06 2021-02-10T05:01:30.888 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.888 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.888 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:01:30.888 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:42.717082+0000", 2021-02-10T05:01:30.888 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.8789999999999998e-06 2021-02-10T05:01:30.888 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.888 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.889 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:01:30.889 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:42.717092+0000", 2021-02-10T05:01:30.889 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.1940000000000003e-06 2021-02-10T05:01:30.889 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.889 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.889 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:01:30.889 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:42.717098+0000", 2021-02-10T05:01:30.889 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.6128000000000001e-05 2021-02-10T05:01:30.889 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.889 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.889 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:01:30.889 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:42.717164+0000", 2021-02-10T05:01:30.890 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.1164e-05 2021-02-10T05:01:30.890 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.890 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.890 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2021-02-10T05:01:30.890 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:42.717176+0000", 2021-02-10T05:01:30.890 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001453317 2021-02-10T05:01:30.890 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.890 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.890 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:01:30.890 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:42.718629+0000", 2021-02-10T05:01:30.890 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2508e-05 2021-02-10T05:01:30.890 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.890 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.891 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T05:01:30.891 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:42.718641+0000", 2021-02-10T05:01:30.891 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 4.6220000000000001e-05 2021-02-10T05:01:30.891 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.891 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.891 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:01:30.891 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:42.718688+0000", 2021-02-10T05:01:30.891 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0016422400000000001 2021-02-10T05:01:30.891 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.891 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:01:30.891 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.891 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.891 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.892 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "pg_backfill(finish 4.1e e 1401/1401 lb MAX)", 2021-02-10T05:01:30.892 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T05:00:47.745232+0000", 2021-02-10T05:01:30.892 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 43.060827408999998, 2021-02-10T05:01:30.892 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0049651970000000002, 2021-02-10T05:01:30.892 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:01:30.892 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T05:01:30.892 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:01:30.892 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.892 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:01:30.892 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:47.745232+0000", 2021-02-10T05:01:30.892 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:01:30.892 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.892 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.893 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:01:30.893 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:47.745232+0000", 2021-02-10T05:01:30.893 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1819999999999997e-05 2021-02-10T05:01:30.893 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.893 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.893 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:01:30.893 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:47.745263+0000", 2021-02-10T05:01:30.893 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.8012999999999998e-05 2021-02-10T05:01:30.893 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.893 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.893 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:01:30.893 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:47.745282+0000", 2021-02-10T05:01:30.893 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.5238999999999999e-05 2021-02-10T05:01:30.894 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.894 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.894 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:01:30.894 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:47.745297+0000", 2021-02-10T05:01:30.894 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.452e-06 2021-02-10T05:01:30.894 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.894 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.894 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:01:30.894 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:47.745304+0000", 2021-02-10T05:01:30.894 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.3410000000000004e-05 2021-02-10T05:01:30.894 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.894 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.894 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:01:30.895 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:47.745368+0000", 2021-02-10T05:01:30.895 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0285999999999999e-05 2021-02-10T05:01:30.895 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.895 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.895 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2021-02-10T05:01:30.895 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:47.745378+0000", 2021-02-10T05:01:30.895 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0047449520000000002 2021-02-10T05:01:30.895 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.895 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.895 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:01:30.895 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:47.750123+0000", 2021-02-10T05:01:30.895 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.6065999999999998e-05 2021-02-10T05:01:30.895 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.896 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.896 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T05:01:30.896 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:47.750139+0000", 2021-02-10T05:01:30.896 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.7958999999999999e-05 2021-02-10T05:01:30.896 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.896 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.896 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:01:30.896 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:47.750197+0000", 2021-02-10T05:01:30.896 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0049651970000000002 2021-02-10T05:01:30.896 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.896 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:01:30.896 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.896 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.897 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.897 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDScrubReserve(4.15 REQUEST e1408)", 2021-02-10T05:01:30.897 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T05:00:58.131602+0000", 2021-02-10T05:01:30.897 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 32.674457048999997, 2021-02-10T05:01:30.897 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0025795269999999999, 2021-02-10T05:01:30.897 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:01:30.897 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T05:01:30.897 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:01:30.897 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.897 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:01:30.897 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:58.131602+0000", 2021-02-10T05:01:30.897 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:01:30.897 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.898 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.898 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:01:30.898 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:58.131602+0000", 2021-02-10T05:01:30.898 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.1047999999999998e-05 2021-02-10T05:01:30.898 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.898 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.898 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:01:30.898 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:58.131633+0000", 2021-02-10T05:01:30.898 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.8220000000000001e-06 2021-02-10T05:01:30.898 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.898 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.898 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:01:30.898 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:58.131640+0000", 2021-02-10T05:01:30.899 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0049e-05 2021-02-10T05:01:30.899 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.899 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.899 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:01:30.899 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:58.131650+0000", 2021-02-10T05:01:30.899 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 8.1159999999999999e-06 2021-02-10T05:01:30.899 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.899 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.899 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:01:30.899 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:58.131658+0000", 2021-02-10T05:01:30.899 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 5.3909999999999997e-05 2021-02-10T05:01:30.899 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.899 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.900 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:01:30.900 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:58.131712+0000", 2021-02-10T05:01:30.900 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.2771e-05 2021-02-10T05:01:30.900 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.900 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.900 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2021-02-10T05:01:30.900 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:58.131725+0000", 2021-02-10T05:01:30.900 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0019084639999999999 2021-02-10T05:01:30.900 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.900 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.900 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:01:30.900 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:58.133633+0000", 2021-02-10T05:01:30.900 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.8860000000000004e-06 2021-02-10T05:01:30.901 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.901 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.901 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T05:01:30.901 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:58.133641+0000", 2021-02-10T05:01:30.901 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.000540461 2021-02-10T05:01:30.901 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.901 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.901 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:01:30.901 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:58.134182+0000", 2021-02-10T05:01:30.901 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0025795269999999999 2021-02-10T05:01:30.901 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.901 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:01:30.901 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.901 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.902 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.902 INFO:teuthology.orchestra.run.smithi164.stdout: "description": "MOSDScrubReserve(2.29 REQUEST e1409)", 2021-02-10T05:01:30.902 INFO:teuthology.orchestra.run.smithi164.stdout: "initiated_at": "2021-02-10T05:00:58.835033+0000", 2021-02-10T05:01:30.902 INFO:teuthology.orchestra.run.smithi164.stdout: "age": 31.971025916999999, 2021-02-10T05:01:30.902 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001672694, 2021-02-10T05:01:30.902 INFO:teuthology.orchestra.run.smithi164.stdout: "type_data": { 2021-02-10T05:01:30.902 INFO:teuthology.orchestra.run.smithi164.stdout: "flag_point": "started", 2021-02-10T05:01:30.902 INFO:teuthology.orchestra.run.smithi164.stdout: "events": [ 2021-02-10T05:01:30.902 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.902 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "initiated", 2021-02-10T05:01:30.902 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:58.835033+0000", 2021-02-10T05:01:30.902 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0 2021-02-10T05:01:30.902 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.903 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.903 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "throttled", 2021-02-10T05:01:30.903 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:58.835033+0000", 2021-02-10T05:01:30.903 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 2.9830000000000001e-05 2021-02-10T05:01:30.903 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.903 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.903 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "header_read", 2021-02-10T05:01:30.903 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:58.835063+0000", 2021-02-10T05:01:30.903 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 6.5420000000000002e-06 2021-02-10T05:01:30.903 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.903 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.903 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "all_read", 2021-02-10T05:01:30.903 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:58.835070+0000", 2021-02-10T05:01:30.904 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0152999999999999e-05 2021-02-10T05:01:30.904 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.904 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.904 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "dispatched", 2021-02-10T05:01:30.904 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:58.835080+0000", 2021-02-10T05:01:30.904 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 7.328e-06 2021-02-10T05:01:30.904 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.904 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.904 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "queued_for_pg", 2021-02-10T05:01:30.904 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:58.835087+0000", 2021-02-10T05:01:30.904 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.00019738899999999999 2021-02-10T05:01:30.904 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.904 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.905 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:01:30.905 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:58.835284+0000", 2021-02-10T05:01:30.905 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 9.0440000000000001e-06 2021-02-10T05:01:30.905 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.905 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.905 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "op must wait for map", 2021-02-10T05:01:30.905 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:58.835293+0000", 2021-02-10T05:01:30.905 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.0013694429999999999 2021-02-10T05:01:30.905 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.905 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.905 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "reached_pg", 2021-02-10T05:01:30.905 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:58.836663+0000", 2021-02-10T05:01:30.906 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 1.0779e-05 2021-02-10T05:01:30.906 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.906 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.906 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "started", 2021-02-10T05:01:30.906 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:58.836674+0000", 2021-02-10T05:01:30.906 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 3.2186e-05 2021-02-10T05:01:30.906 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2021-02-10T05:01:30.906 INFO:teuthology.orchestra.run.smithi164.stdout: { 2021-02-10T05:01:30.906 INFO:teuthology.orchestra.run.smithi164.stdout: "event": "done", 2021-02-10T05:01:30.906 INFO:teuthology.orchestra.run.smithi164.stdout: "time": "2021-02-10T05:00:58.836706+0000", 2021-02-10T05:01:30.906 INFO:teuthology.orchestra.run.smithi164.stdout: "duration": 0.001672694 2021-02-10T05:01:30.906 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.906 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:01:30.907 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.907 INFO:teuthology.orchestra.run.smithi164.stdout: } 2021-02-10T05:01:30.907 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2021-02-10T05:01:30.907 INFO:teuthology.orchestra.run.smithi164.stdout:} 2021-02-10T05:01:30.958 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.7 to 1 (10000) 2021-02-10T05:01:31.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:30 smithi164 conmon[29288]: audit 2021-02-10T05:01:29.676983+0000 mgr.y (mgr.14140) 1602 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:01:31.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:30 smithi164 conmon[29288]: audit 2021-02-10T05:01:30.230895+0000 mon.a (mon.0) 3194 : audit [INF] from='client.? 172.21.15.85:0/3957673812' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T05:01:31.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:30 smithi164 conmon[29288]: cluster 2021-02-10T05:01:30.360093+0000 osd.3 (osd.3) 75 : 2021-02-10T05:01:31.145 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:30 smithi164 conmon[29288]: cluster [DBG] 2.28 scrub ok 2021-02-10T05:01:31.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:30 smithi085 conmon[29319]: audit 2021-02-10T05:01:29.676983+0000 mgr.y (mgr.14140 2021-02-10T05:01:31.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:30 smithi085 conmon[29319]: ) 1602 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:01:31.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:30 smithi085 conmon[29319]: audit 2021-02-10T05:01:30.230895+0000 mon.a (mon.0) 3194 : audit [INF] from='client.? 172.21.15.85:0/3957673812' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T05:01:31.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:30 smithi085 conmon[29319]: cluster 2021-02-10T05:01:30.360093+0000 osd.3 (osd.3) 75 : cluster [DBG] 2.28 scrub ok 2021-02-10T05:01:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:30 smithi085 conmon[35961]: audit 2021-02-10T05:01:29.676983 2021-02-10T05:01:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:30 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1602 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:01:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:30 smithi085 conmon[35961]: audit 2021-02-10T05: 2021-02-10T05:01:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:30 smithi085 conmon[35961]: 01:30.230895+0000 mon.a (mon.0 2021-02-10T05:01:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:30 smithi085 conmon[35961]: ) 3194 : audit [INF] from='client.? 172.21.15.85:0/3957673812' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T05:01:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:30 smithi085 conmon[35961]: cluster 2021-02-10T05:01:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:30 smithi085 conmon[35961]: 2021-02-10T05:01 2021-02-10T05:01:31.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:30 smithi085 conmon[35961]: :30.360093+0000 osd.3 (osd.3) 2021-02-10T05:01:31.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:30 smithi085 conmon[35961]: 75 : cluster [DBG] 2.28 scrub ok 2021-02-10T05:01:31.809 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd primary-affinity 7 1 2021-02-10T05:01:32.028 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:01:32.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:31 smithi085 conmon[35961]: cluster 2021-02-10T05:01:30.600705+0000 mgr.y (mgr.14140) 1603 : cluster [DBG] pgmap v2609: 172 pgs: 172 active+clean; 25 KiB data, 386 MiB used, 715 GiB / 715 GiB avail 2021-02-10T05:01:32.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:31 smithi085 conmon[35961]: audit 2021-02-10T05:01:30.951453+0000 mon.a (mon.0) 3195 : audit [INF] from='client.? 172.21.15.85:0/3957673812' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-02-10T05:01:32.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:31 smithi085 conmon[35961]: cluster 2021-02-10T05:01:30.951533+0000 mon.a (mon.0) 3196 : cluster [DBG] osdmap e1424: 8 total, 8 up, 8 in 2021-02-10T05:01:32.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:31 smithi085 conmon[35961]: cluster 2021-02-10T05:01:31.167512+0000 osd.5 (osd.5) 230 : cluster [DBG] 5.15 deep-scrub ok 2021-02-10T05:01:32.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:31 smithi085 conmon[29319]: cluster 2021-02-10T05:01:30.600705+0000 mgr.y (mgr.14140) 1603 : cluster [DBG] pgmap v2609: 172 pgs: 172 active+clean; 25 KiB data, 386 MiB used, 715 GiB / 715 GiB avail 2021-02-10T05:01:32.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:31 smithi085 conmon[29319]: audit 2021-02-10T05:01:30.951453+0000 mon.a (mon.0) 3195 : audit [INF] from='client.? 172.21.15.85:0/3957673812' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-02-10T05:01:32.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:31 smithi085 conmon[29319]: cluster 2021-02-10T05:01:30.951533+0000 mon.a (mon.0) 3196 : cluster [DBG] osdmap e1424: 8 total, 8 up, 8 in 2021-02-10T05:01:32.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:31 smithi085 conmon[29319]: cluster 2021-02-10T05:01:31.167512+0000 osd.5 (osd.5) 230 : cluster [DBG] 5.15 deep-scrub ok 2021-02-10T05:01:32.282 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:31 smithi164 conmon[29288]: cluster 2021-02-10T05:01:30.600705+0000 mgr.y (mgr 2021-02-10T05:01:32.282 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:31 smithi164 conmon[29288]: .14140) 1603 : cluster [DBG] pgmap v2609: 172 pgs: 172 active+clean; 25 KiB data, 386 MiB used, 715 GiB / 715 GiB avail 2021-02-10T05:01:32.282 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:31 smithi164 conmon[29288]: audit 2021-02-10T05:01:30.951453+0000 mon.a (mon.0) 3195 : audit [INF] from='client.? 172.21.15.85:0/3957673812' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-02-10T05:01:32.282 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:31 smithi164 conmon[29288]: cluster 2021-02-10T05:01:30.951533+0000 mon.a (mon.0) 3196 2021-02-10T05:01:32.282 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:31 smithi164 conmon[29288]: : cluster [DBG] osdmap e1424: 8 total, 8 up, 8 in 2021-02-10T05:01:32.282 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:31 smithi164 conmon[29288]: cluster 2021-02-10 2021-02-10T05:01:32.282 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:31 smithi164 conmon[29288]: T05:01:31.167512+0000 osd.5 (osd. 2021-02-10T05:01:32.282 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:31 smithi164 conmon[29288]: 5) 230 : cluster [DBG] 5.15 deep-scrub ok 2021-02-10T05:01:33.372 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:32 smithi085 conmon[29319]: cluster 2021-02-10T05:01:31.683330+0000 osd.0 (osd.0 2021-02-10T05:01:33.372 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:32 smithi085 conmon[29319]: ) 35 : cluster [DBG] 5.5 scrub ok 2021-02-10T05:01:33.372 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:32 smithi085 conmon[29319]: cluster 2021-02-10T05:01:31.716187+0000 osd.4 (osd.4) 2704 : cluster [DBG] 1.5 deep-scrub ok 2021-02-10T05:01:33.372 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:32 smithi085 conmon[29319]: cluster 2021-02-10T05:01:31.900921+0000 osd.6 (osd.6) 220 : cluster [DBG] 5.11 scrub ok 2021-02-10T05:01:33.372 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:32 smithi085 conmon[29319]: cluster 2021-02-10T05:01:32.305886+0000 osd.3 (osd.3) 76 : cluster [DBG] 5.18 scrub ok 2021-02-10T05:01:33.372 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:32 smithi085 conmon[29319]: cluster 2021-02-10T05:01:32.506827+0000 osd.1 (osd.1) 550 : cluster [DBG] 4.f scrub ok 2021-02-10T05:01:33.372 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:32 smithi085 conmon[35961]: cluster 2021-02-10T05:01:31.683330+0000 osd.0 ( 2021-02-10T05:01:33.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:32 smithi085 conmon[35961]: osd.0) 35 : cluster [DBG] 5.5 scrub ok 2021-02-10T05:01:33.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:32 smithi085 conmon[35961]: cluster 2021-02-10T05:01:31.716187+0000 osd.4 (osd.4) 2704 : cluster [DBG] 1.5 deep-scrub ok 2021-02-10T05:01:33.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:32 smithi085 conmon[35961]: 2021-02-10T05:01:33.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:32 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T05:01:33.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:32 smithi085 conmon[35961]: T05:01:31.900921+0000 osd.6 (osd.6) 220 : cluster 2021-02-10T05:01:33.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:32 smithi085 conmon[35961]: [DBG] 5.11 scrub ok 2021-02-10T05:01:33.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:32 smithi085 conmon[35961]: cluster 2021-02-10T05:01:33.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:32 smithi085 conmon[35961]: 2021-02-10T05:01:32 2021-02-10T05:01:33.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:32 smithi085 conmon[35961]: .305886+0000 osd.3 (osd.3) 76 : 2021-02-10T05:01:33.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:32 smithi085 conmon[35961]: cluster [DBG] 5.18 scrub ok 2021-02-10T05:01:33.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:32 smithi085 conmon[35961]: cluster 2021- 2021-02-10T05:01:33.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:32 smithi085 conmon[35961]: 02-10T05:01:32.506827+0000 2021-02-10T05:01:33.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:32 smithi085 conmon[35961]: osd.1 (osd.1) 550 : cluster [DBG] 4.f scrub ok 2021-02-10T05:01:33.373 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:32 smithi085 conmon[35961]: 2021-02-10T05:01:33.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:32 smithi164 conmon[29288]: cluster 2021-02-10T05:01:31. 2021-02-10T05:01:33.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:32 smithi164 conmon[29288]: 683330+0000 osd.0 (osd.0) 35 : cluster [DBG] 5.5 scrub ok 2021-02-10T05:01:33.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:32 smithi164 conmon[29288]: cluster 2021-02-10T05:01:31.716187+0000 osd.4 (osd.4) 2704 : cluster 2021-02-10T05:01:33.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:32 smithi164 conmon[29288]: [DBG] 1.5 deep-scrub ok 2021-02-10T05:01:33.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:32 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T05:01:33.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:32 smithi164 conmon[29288]: -10T05:01:31.900921+0000 osd.6 (osd.6) 220 : 2021-02-10T05:01:33.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:32 smithi164 conmon[29288]: cluster [DBG] 5.11 scrub ok 2021-02-10T05:01:33.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:32 smithi164 conmon[29288]: cluster 2021-02-10T 2021-02-10T05:01:33.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:32 smithi164 conmon[29288]: 05:01:32.305886+0000 osd.3 (osd. 2021-02-10T05:01:33.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:32 smithi164 conmon[29288]: 3) 76 : cluster [DBG] 5.18 scrub ok 2021-02-10T05:01:33.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:32 smithi164 conmon[29288]: cluster 2021-02-10T05:01:33.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:32 smithi164 conmon[29288]: 2021-02-10T05:01:32.506827+0000 osd.1 (osd.1) 550 : cluster 2021-02-10T05:01:33.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:32 smithi164 conmon[29288]: [DBG] 4.f scrub ok 2021-02-10T05:01:34.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:33 smithi164 conmon[29288]: cluster 2021-02-10T05:01:32. 2021-02-10T05:01:34.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:33 smithi164 conmon[29288]: 460753+0000 osd.6 (osd.6) 221 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:01:34.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:33 smithi164 conmon[29288]: cluster 2021-02-10T05:01: 2021-02-10T05:01:34.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:33 smithi164 conmon[29288]: 32.462146+0000 osd.6 (osd.6) 222 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:01:34.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:33 smithi164 conmon[29288]: cluster 2021-02-10T05: 2021-02-10T05:01:34.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:33 smithi164 conmon[29288]: 01:32.601275+0000 mgr.y (mgr.14140) 1604 : 2021-02-10T05:01:34.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:33 smithi164 conmon[29288]: cluster [DBG] pgmap v2611: 172 pgs: 172 active+clean; 25 KiB data, 386 MiB used, 715 GiB / 715 GiB avail 2021-02-10T05:01:34.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:33 smithi164 conmon[29288]: cluster 2021-02-10T05:01: 2021-02-10T05:01:34.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:33 smithi164 conmon[29288]: 32.957663+0000 mon.a (mon.0) 3197 : 2021-02-10T05:01:34.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:33 smithi164 conmon[29288]: cluster [DBG] osdmap e1425: 8 total, 8 up, 8 in 2021-02-10T05:01:34.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:33 smithi164 conmon[29288]: cluster 2021-02-10T05:01:34.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:33 smithi164 conmon[29288]: 2021-02-10T05:01:33. 2021-02-10T05:01:34.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:33 smithi164 conmon[29288]: 045807+0000 osd.2 (osd.2) 2021-02-10T05:01:34.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:33 smithi164 conmon[29288]: 2071 : cluster [DBG] 6.f scrub ok 2021-02-10T05:01:34.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:33 smithi164 conmon[29288]: cluster 2021 2021-02-10T05:01:34.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:33 smithi164 conmon[29288]: -02-10T05:01:33.225052+0000 osd.5 2021-02-10T05:01:34.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:33 smithi164 conmon[29288]: (osd.5) 231 : cluster [DBG] 5.7 scrub ok 2021-02-10T05:01:34.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:33 smithi164 conmon[29288]: 2021-02-10T05:01:34.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[29319]: cluster 2021-02-10T05:01:34.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[29319]: 2021-02-10T05:01:32 2021-02-10T05:01:34.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[29319]: .460753+0000 osd.6 (osd.6) 221 : cluster [DBG] purged_snaps scrub starts 2021-02-10T05:01:34.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[29319]: cluster 2021-02-10T05:01:32.462146+0000 2021-02-10T05:01:34.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[29319]: osd.6 (osd.6) 222 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:01:34.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[29319]: cluster 2021 2021-02-10T05:01:34.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[29319]: -02-10T05:01:32.601275+0000 2021-02-10T05:01:34.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[29319]: mgr.y (mgr.14140) 1604 : cluster [DBG] pgmap v2611: 172 pgs: 172 active+clean; 25 KiB data, 386 MiB used, 715 GiB / 715 GiB avail 2021-02-10T05:01:34.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[29319]: 2021-02-10T05:01:34.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[29319]: cluster 2021-02-10T05:01:32.957663 2021-02-10T05:01:34.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[29319]: +0000 mon.a (mon.0) 3197 : cluster [DBG] osdmap e1425: 8 total, 8 up, 8 in 2021-02-10T05:01:34.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[29319]: 2021-02-10T05:01:34.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[29319]: cluster 2021-02-10T05:01:34.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[29319]: 2021-02-10T05:01:33.045807+0000 osd.2 (osd.2) 2071 : cluster 2021-02-10T05:01:34.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[29319]: [DBG] 6.f scrub ok 2021-02-10T05:01:34.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[29319]: cluster 2021-02-10T05:01:33.225052+0000 2021-02-10T05:01:34.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[29319]: osd.5 (osd.5) 231 : cluster [DBG] 5.7 scrub ok 2021-02-10T05:01:34.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[35961]: cluster 2021-02-10T05:01:34.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[35961]: 2021-02- 2021-02-10T05:01:34.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[35961]: 10T05:01:32.460753+0000 osd.6 (osd.6) 221 : cluster [DBG] 2021-02-10T05:01:34.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[35961]: purged_snaps scrub starts 2021-02-10T05:01:34.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[35961]: cluster 2021-02-10T05:01:34.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T05:01:34.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[35961]: 05:01:32.462146+0000 osd.6 (osd 2021-02-10T05:01:34.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[35961]: .6) 222 : cluster [DBG] purged_snaps scrub ok 2021-02-10T05:01:34.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[35961]: cluster 2021 2021-02-10T05:01:34.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[35961]: -02-10T05:01:32.601275+0000 mgr.y 2021-02-10T05:01:34.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[35961]: (mgr.14140) 1604 : cluster [DBG] pgmap v2611: 172 pgs: 172 active+clean; 25 KiB data, 386 MiB used, 715 GiB / 715 GiB avail 2021-02-10T05:01:34.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[35961]: 2021-02-10T05:01:34.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[35961]: cluster 2021-02-10T05 2021-02-10T05:01:34.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[35961]: :01:32.957663+0000 mon.a (mon. 2021-02-10T05:01:34.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[35961]: 0) 3197 : cluster [DBG] osdmap e1425: 8 total, 8 up, 8 in 2021-02-10T05:01:34.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[35961]: cluster 2021-02-10T05:01:34.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[35961]: 2021-02-10T05:01:33.045807 2021-02-10T05:01:34.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 2071 : cluster 2021-02-10T05:01:34.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[35961]: [DBG] 6.f scrub ok 2021-02-10T05:01:34.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T05:01:34.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[35961]: T05:01:33.225052+0000 osd.5 ( 2021-02-10T05:01:34.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:33 smithi085 conmon[35961]: osd.5) 231 : cluster [DBG] 5.7 scrub ok 2021-02-10T05:01:34.968 INFO:teuthology.orchestra.run.smithi085.stderr:set osd.7 primary-affinity to 1 (8655362) 2021-02-10T05:01:35.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:34 smithi164 conmon[29288]: cluster 2021-02-10T05:01:35.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:34 smithi164 conmon[29288]: 2021-02-10T05 2021-02-10T05:01:35.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:34 smithi164 conmon[29288]: :01:34.083479+0000 osd.2 (osd.2) 2072 : cluster [DBG] 6.28 deep-scrub ok 2021-02-10T05:01:35.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:34 smithi164 conmon[29288]: audit 2021 2021-02-10T05:01:35.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:34 smithi164 conmon[29288]: -02-10T05:01:34.154604+0000 2021-02-10T05:01:35.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:34 smithi164 conmon[29288]: mon.c (mon.1) 182 : audit [INF] 2021-02-10T05:01:35.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:34 smithi164 conmon[29288]: from='client.? 172.21.15.85:0/2971085676' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T05:01:35.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:34 smithi164 conmon[29288]: audit 2021-02-10T05: 2021-02-10T05:01:35.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:34 smithi164 conmon[29288]: 01:34.154996+0000 mon.a (mon. 2021-02-10T05:01:35.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:34 smithi164 conmon[29288]: 0) 3198 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T05:01:35.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:34 smithi164 conmon[29288]: cluster 2021- 2021-02-10T05:01:35.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:34 smithi164 conmon[29288]: 02-10T05:01:34.239554+0000 2021-02-10T05:01:35.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:34 smithi164 conmon[29288]: osd.5 (osd.5) 232 : cluster [DBG] 2021-02-10T05:01:35.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:34 smithi164 conmon[29288]: 3.c scrub ok 2021-02-10T05:01:35.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:34 smithi085 conmon[29319]: cluster 2021-02-10T05:01:34.083479+0000 osd.2 (osd 2021-02-10T05:01:35.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:34 smithi085 conmon[29319]: .2) 2072 : cluster [DBG] 6.28 deep-scrub ok 2021-02-10T05:01:35.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:34 smithi085 conmon[29319]: audit 2021-02-10T05:01:34.154604+0000 mon.c (mon.1) 182 : audit [INF] from='client.? 172.21.15.85:0/2971085676' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T05:01:35.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:34 smithi085 conmon[29319]: audit 2021-02-10T05:01:34.154996+0000 mon.a (mon.0) 2021-02-10T05:01:35.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:34 smithi085 conmon[29319]: 3198 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T05:01:35.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:34 smithi085 conmon[29319]: cluster 2021-02-10T05:01:34. 2021-02-10T05:01:35.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:34 smithi085 conmon[29319]: 239554+0000 osd.5 (osd.5) 232 : 2021-02-10T05:01:35.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:34 smithi085 conmon[29319]: cluster [DBG] 3.c scrub ok 2021-02-10T05:01:35.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:34 smithi085 conmon[35961]: cluster 2021-02-10T05:01:34 2021-02-10T05:01:35.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:34 smithi085 conmon[35961]: .083479+0000 osd.2 (osd.2) 2072 : cluster [DBG] 6.28 deep-scrub ok 2021-02-10T05:01:35.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:34 smithi085 conmon[35961]: audit 2021-02-10T05: 2021-02-10T05:01:35.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:34 smithi085 conmon[35961]: 01:34.154604+0000 mon.c ( 2021-02-10T05:01:35.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:34 smithi085 conmon[35961]: mon.1) 182 : audit [INF] 2021-02-10T05:01:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:34 smithi085 conmon[35961]: from='client.? 172.21.15.85:0/2971085676' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T05:01:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:34 smithi085 conmon[35961]: audit 2021-02-10T05:01:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:34 smithi085 conmon[35961]: 2021-02-10T05 2021-02-10T05:01:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:34 smithi085 conmon[35961]: :01:34.154996+0000 mon.a (mon.0 2021-02-10T05:01:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:34 smithi085 conmon[35961]: ) 3198 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2021-02-10T05:01:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:34 smithi085 conmon[35961]: cluster 2021- 2021-02-10T05:01:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:34 smithi085 conmon[35961]: 02-10T05:01:34.239554+0000 2021-02-10T05:01:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:34 smithi085 conmon[35961]: osd.5 (osd.5) 232 : cluster [DBG] 3.c scrub ok 2021-02-10T05:01:35.443 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:34 smithi085 conmon[35961]: 2021-02-10T05:01:35.967 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd reweight 0 1 2021-02-10T05:01:36.181 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:01:36.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[29319]: cluster 2021-02-10T05:01:34. 2021-02-10T05:01:36.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[29319]: 601775+0000 mgr.y (mgr.14140) 1605 : cluster [DBG] pgmap v2613: 172 pgs: 172 active+clean; 25 KiB data, 386 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T05:01:36.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T05:01:36.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[29319]: 10T05:01:34.906902+0000 osd.6 (osd.6 2021-02-10T05:01:36.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[29319]: ) 223 : cluster [DBG] 6.27 scrub ok 2021-02-10T05:01:36.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[29319]: audit 2021- 2021-02-10T05:01:36.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[29319]: 02-10T05:01:34. 2021-02-10T05:01:36.382 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[29319]: 959701+0000 mon.a (mon.0) 3199 2021-02-10T05:01:36.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[29319]: : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2021-02-10T05:01:36.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[29319]: cluster 2021-02- 2021-02-10T05:01:36.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[29319]: 10T05:01:34.959774+0000 mon.a (mon.0) 3200 : 2021-02-10T05:01:36.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[29319]: cluster [DBG] osdmap e1426: 8 total, 8 up, 8 in 2021-02-10T05:01:36.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[29319]: cluster 2021-02-10T05:01:35.276707+0000 2021-02-10T05:01:36.383 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[29319]: osd.7 (osd.7) 127 : cluster [DBG] 2.1c deep-scrub ok 2021-02-10T05:01:36.383 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[35961]: cluster 2021-02-10T05:01:34.601775 2021-02-10T05:01:36.383 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[35961]: +0000 mgr.y (mgr.14140) 1605 : cluster [DBG] pgmap v2613: 172 pgs: 172 active+clean; 25 KiB data, 386 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T05:01:36.383 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[35961]: cluster 2021-02-10T05:01:36.383 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[35961]: 2021-02-10T05:01:34.906902+0000 osd.6 (osd.6 2021-02-10T05:01:36.383 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[35961]: ) 223 : cluster [DBG] 6.27 scrub ok 2021-02-10T05:01:36.383 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[35961]: audit 2021-02-10T05:01:36.384 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[35961]: 2021-02-10 2021-02-10T05:01:36.384 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[35961]: T05:01:34.959701+0000 mon.a ( 2021-02-10T05:01:36.384 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[35961]: mon.0) 3199 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2021-02-10T05:01:36.384 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[35961]: cluster 2021-02-10T05:01:36.384 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[35961]: 2021-02-10T05 2021-02-10T05:01:36.384 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[35961]: :01:34.959774+0000 mon.a (mon.0) 2021-02-10T05:01:36.384 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[35961]: 3200 : cluster [DBG] osdmap e1426: 8 total, 8 up, 8 in 2021-02-10T05:01:36.384 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[35961]: cluster 2021 2021-02-10T05:01:36.384 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[35961]: -02-10T05:01:35.276707 2021-02-10T05:01:36.384 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[35961]: +0000 osd.7 (osd.7) 127 : cluster [DBG] 2.1c deep-scrub ok 2021-02-10T05:01:36.384 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:35 smithi085 conmon[35961]: 2021-02-10T05:01:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:35 smithi164 conmon[29288]: cluster 2021-02-10T05:01:34.601775+0000 mgr.y (mgr.14140 2021-02-10T05:01:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:35 smithi164 conmon[29288]: ) 1605 : cluster [DBG] pgmap v2613: 172 pgs: 172 active+clean; 25 KiB data, 386 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-02-10T05:01:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:35 smithi164 conmon[29288]: cluster 2021-02-10T05:01:34.906902+0000 osd.6 (osd.6) 223 : cluster [DBG] 6.27 scrub ok 2021-02-10T05:01:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:35 smithi164 conmon[29288]: audit 2021-02-10T05:01:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:35 smithi164 conmon[29288]: 2021-02-10T05:01:34.959701+0000 mon.a (mon.0) 3199 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2021-02-10T05:01:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:35 smithi164 conmon[29288]: cluster 2021-02 2021-02-10T05:01:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:35 smithi164 conmon[29288]: -10T05:01:34.959774+0000 mon.a (mon. 2021-02-10T05:01:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:35 smithi164 conmon[29288]: 0) 3200 : cluster [DBG] osdmap e1426: 8 total, 8 up, 8 in 2021-02-10T05:01:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:35 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T05:01:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:35 smithi164 conmon[29288]: 10T05:01:35.276707+0000 osd.7 (osd 2021-02-10T05:01:36.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:35 smithi164 conmon[29288]: .7) 127 : cluster [DBG] 2.1c deep-scrub ok 2021-02-10T05:01:37.152 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:36 smithi085 conmon[29319]: cluster 2021-02-10T05:01:35.970509+0000 mon.a (mon.0) 3201 : cluster [DBG] osdmap e1427: 8 total, 8 up, 8 in 2021-02-10T05:01:37.152 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:36 smithi085 conmon[35961]: cluster 2021-02-10T05:01:35.970509+0000 mon.a (mon.0) 3201 : cluster [DBG] osdmap e1427: 8 total, 8 up, 8 in 2021-02-10T05:01:37.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:36 smithi164 conmon[29288]: cluster 2021-02-10T05:01:35.970509+0000 mon.a ( 2021-02-10T05:01:37.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:36 smithi164 conmon[29288]: mon.0) 3201 : cluster [DBG] osdmap e1427: 8 total, 8 up, 8 in 2021-02-10T05:01:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:37 smithi164 conmon[29288]: cluster 2021-02-10T05:01:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:37 smithi164 conmon[29288]: 2021-02-10T05:01:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:37 smithi164 conmon[29288]: 2021-02-10T05:01: 2021-02-10T05:01:38.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:37 smithi164 conmon[29288]: 36.602175+0000 mgr.y (mgr.14140) 1606 : cluster [DBG] pgmap v2616: 172 pgs: 2 peering, 170 active+clean; 25 KiB data, 386 MiB used, 715 GiB / 715 GiB avail; 905 B/s rd, 0 op/s 2021-02-10T05:01:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:37 smithi164 conmon[29288]: cluster 2021-02-10T05:01 2021-02-10T05:01:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:37 smithi164 conmon[29288]: :36.967384+0000 mon.a (mon.0) 3202 : 2021-02-10T05:01:38.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:37 smithi164 conmon[29288]: cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2021-02-10T05:01:38.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:37 smithi085 conmon[29319]: cluster 2021-02-10T05:01:36.602175+0000 mgr.y (mgr.14140 2021-02-10T05:01:38.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:37 smithi085 conmon[29319]: ) 1606 : cluster [DBG] pgmap v2616: 172 pgs: 2 peering, 170 active+clean; 25 KiB data, 386 MiB used, 715 GiB / 715 GiB avail; 905 B/s rd, 0 op/s 2021-02-10T05:01:38.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:37 smithi085 conmon[29319]: cluster 2021-02-10T05:01:36.967384+0000 mon.a (mon.0) 3202 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2021-02-10T05:01:38.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:37 smithi085 conmon[35961]: cluster 2021-02-10T05:01 2021-02-10T05:01:38.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:37 smithi085 conmon[35961]: :36.602175+0000 mgr.y (mgr.14140) 1606 : cluster [DBG] pgmap v2616: 172 pgs: 2 peering, 170 active+clean; 25 KiB data, 386 MiB used, 715 GiB / 715 GiB avail; 905 B/s rd, 0 op/s 2021-02-10T05:01:38.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:37 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T05:01:38.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:37 smithi085 conmon[35961]: 10T05:01:36.967384+0000 mon.a ( 2021-02-10T05:01:38.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:37 smithi085 conmon[35961]: mon.0) 3202 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2021-02-10T05:01:38.993 INFO:teuthology.orchestra.run.smithi085.stderr:reweighted osd.0 to 1 (10000) 2021-02-10T05:01:39.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:38 smithi164 conmon[29288]: cluster 2021-02-10T05:01:37.930856 2021-02-10T05:01:39.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:38 smithi164 conmon[29288]: +0000 osd.6 (osd.6) 224 : cluster [DBG] 5.2 scrub ok 2021-02-10T05:01:39.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:38 smithi164 conmon[29288]: cluster 2021-02-10T05:01:38.107244+0000 osd.2 (osd.2) 2073 : cluster [DBG] 2.c scrub ok 2021-02-10T05:01:39.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:38 smithi164 conmon[29288]: cluster 2021-02-10T05:01:38.219668+0000 osd.5 (osd.5) 233 : cluster [DBG] 2021-02-10T05:01:39.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:38 smithi164 conmon[29288]: 4.8 scrub ok 2021-02-10T05:01:39.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:38 smithi164 conmon[29288]: audit 2021-02-10T05:01:39.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:38 smithi164 conmon[29288]: 2021-02-10T05:01:38.269945+0000 mon.a (mon 2021-02-10T05:01:39.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:38 smithi164 conmon[29288]: .0) 3203 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T05:01:39.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:38 smithi164 conmon[29288]: audit 2021-02 2021-02-10T05:01:39.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:38 smithi164 conmon[29288]: -10T05:01:38.271434+0000 mon.a (mon 2021-02-10T05:01:39.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:38 smithi164 conmon[29288]: .0) 3204 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T05:01:39.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:38 smithi164 conmon[29288]: audit 2021-02-10T 2021-02-10T05:01:39.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:38 smithi164 conmon[29288]: 05:01:38.350384+0000 mon.c (mon. 2021-02-10T05:01:39.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:38 smithi164 conmon[29288]: 1) 183 : audit [INF] from='client.? 172.21.15.85:0/2246289459' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T05:01:39.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:38 smithi164 conmon[29288]: audit 2021-02-10T05: 2021-02-10T05:01:39.397 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:38 smithi164 conmon[29288]: 01:38.350694+0000 mon.a (mon.0) 3205 2021-02-10T05:01:39.398 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:38 smithi164 conmon[29288]: : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T05:01:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[29319]: cluster 2021-02-10T05:01:37.930856 2021-02-10T05:01:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[29319]: +0000 osd.6 (osd.6) 224 : cluster [DBG] 5.2 scrub ok 2021-02-10T05:01:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[29319]: cluster 2021-02-10T05:01:38.107244+0000 2021-02-10T05:01:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[29319]: osd.2 (osd.2) 2073 : cluster [DBG] 2.c scrub ok 2021-02-10T05:01:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[29319]: cluster 2021-02 2021-02-10T05:01:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[29319]: -10T05:01:38.219668+0000 osd.5 ( 2021-02-10T05:01:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[29319]: osd.5) 233 : cluster [DBG] 4.8 scrub ok 2021-02-10T05:01:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[29319]: audit 2021-02-10T05:01:39.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[29319]: 2021-02-10T05:01:38. 2021-02-10T05:01:39.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[29319]: 269945+0000 mon.a (mon.0) 3203 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T05:01:39.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[29319]: audit 2021-02-10T05:01:39.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[29319]: 2021-02-10T05:01:38. 2021-02-10T05:01:39.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[29319]: 271434+0000 mon.a (mon.0) 3204 : audit 2021-02-10T05:01:39.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[29319]: [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T05:01:39.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[29319]: audit 2021-02 2021-02-10T05:01:39.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[29319]: -10T05:01:38.350384+0000 mon.c (mon 2021-02-10T05:01:39.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[29319]: .1) 183 : audit [INF] from='client.? 172.21.15.85:0/2246289459' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T05:01:39.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[29319]: audit 2021-02-10T05:01:39.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[29319]: 2021-02- 2021-02-10T05:01:39.443 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[29319]: 10T05:01:38.350694+0000 mon.a ( 2021-02-10T05:01:39.444 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[29319]: mon.0) 3205 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T05:01:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[35961]: cluster 2021-02-10T05:01:37.930856+0000 2021-02-10T05:01:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[35961]: osd.6 (osd.6) 224 : cluster [DBG] 5.2 scrub ok 2021-02-10T05:01:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[35961]: cluster 2021-02-10 2021-02-10T05:01:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[35961]: T05:01:38.107244+0000 osd.2 (osd.2) 2073 : cluster 2021-02-10T05:01:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[35961]: [DBG] 2.c scrub ok 2021-02-10T05:01:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[35961]: cluster 2021-02-10T05 2021-02-10T05:01:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[35961]: :01:38.219668+0000 osd.5 (osd.5 2021-02-10T05:01:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[35961]: ) 233 : cluster [DBG] 4.8 scrub ok 2021-02-10T05:01:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[35961]: audit 2021- 2021-02-10T05:01:39.444 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[35961]: 02-10T05:01:38.269945+0000 mon.a 2021-02-10T05:01:39.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[35961]: (mon.0) 3203 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-02-10T05:01:39.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[35961]: audit 2021 2021-02-10T05:01:39.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[35961]: -02-10T05:01:38.271434+0000 2021-02-10T05:01:39.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[35961]: mon.a (mon.0) 3204 : audit [INF] from='mgr.14140 172.21.15.85:0/7' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-02-10T05:01:39.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[35961]: 2021-02-10T05:01:39.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[35961]: audit 2021-02-10T05:01:39.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[35961]: 2021-02-10T05:01:38.350384+0000 2021-02-10T05:01:39.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[35961]: mon.c (mon.1) 183 : audit [INF] from='client.? 172.21.15.85:0/2246289459' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T05:01:39.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[35961]: audit 2021 2021-02-10T05:01:39.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[35961]: -02-10T05:01:38.350694+0000 2021-02-10T05:01:39.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[35961]: mon.a (mon.0) 3205 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T05:01:39.445 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:38 smithi085 conmon[35961]: 2021-02-10T05:01:39.906 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph osd primary-affinity 0 1 2021-02-10T05:01:39.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 05:01:39 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T05:01:39.986 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:39 smithi164 conmon[29288]: cluster 2021-02-10T05:01:38.602795+0000 2021-02-10T05:01:39.986 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:39 smithi164 conmon[29288]: mgr.y (mgr.14140) 1607 : cluster [DBG] pgmap v2617: 172 pgs: 2 peering, 170 active+clean; 26 KiB data, 391 MiB used, 715 GiB / 715 GiB avail 2021-02-10T05:01:39.986 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:39 smithi164 conmon[29288]: audit 2021-02-10T05:01:38.988959+0000 2021-02-10T05:01:39.986 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:39 smithi164 conmon[29288]: mon.a (mon.0) 3206 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2021-02-10T05:01:39.986 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:39 smithi164 conmon[29288]: cluster 2021-02- 2021-02-10T05:01:39.987 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:39 smithi164 conmon[29288]: 10T05:01:38.989023+0000 mon.a (mon 2021-02-10T05:01:39.987 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:39 smithi164 conmon[29288]: .0) 3207 : cluster [DBG] osdmap e1428: 8 total, 8 up, 8 in 2021-02-10T05:01:39.987 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:39 smithi164 conmon[29288]: cluster 2021-02-10T05:01:39.987 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:39 smithi164 conmon[29288]: 2021-02-10T05:01:39.279218+0000 osd.7 2021-02-10T05:01:40.111 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:01:40.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:39 smithi085 conmon[35961]: cluster 2021-02-10T05:01:38.602795+0000 mgr.y (mgr.14140) 1607 : cluster [DBG] pgmap v2617: 172 pgs: 2 peering, 170 active+clean; 26 KiB data, 391 MiB used, 715 GiB / 715 GiB avail 2021-02-10T05:01:40.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:39 smithi085 conmon[35961]: audit 2021-02-10T05:01:38.988959+0000 mon.a (mon.0) 3206 : 2021-02-10T05:01:40.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:39 smithi085 conmon[35961]: audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2021-02-10T05:01:40.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:39 smithi085 conmon[35961]: cluster 2021 2021-02-10T05:01:40.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:39 smithi085 conmon[35961]: -02-10T05:01:38.989023+0000 mon.a (mon.0) 3207 : cluster 2021-02-10T05:01:40.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:39 smithi085 conmon[35961]: [DBG] osdmap e1428: 8 total, 8 up, 8 in 2021-02-10T05:01:40.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:39 smithi085 conmon[35961]: cluster 2021-02-10T05:01: 2021-02-10T05:01:40.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:39 smithi085 conmon[35961]: 39.279218+0000 osd.7 (osd.7) 128 : 2021-02-10T05:01:40.193 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:39 smithi085 conmon[35961]: cluster [DBG] 1.2 scrub ok 2021-02-10T05:01:40.193 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:39 smithi085 conmon[29319]: cluster 2021-02-10T05:01:38.602795+0000 mgr.y (mgr 2021-02-10T05:01:40.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:39 smithi085 conmon[29319]: .14140) 1607 : cluster [DBG] pgmap v2617: 172 pgs: 2 peering, 170 active+clean; 26 KiB data, 391 MiB used, 715 GiB / 715 GiB avail 2021-02-10T05:01:40.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:39 smithi085 conmon[29319]: audit 2021-02-10T05:01:38.988959+0000 mon.a (mon.0) 3206 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2021-02-10T05:01:40.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:39 smithi085 conmon[29319]: cluster 2021-02-10T05:01:38.989023 2021-02-10T05:01:40.194 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:39 smithi085 conmon[29319]: +0000 mon.a (mon.0) 3207 : cluster [DBG] 2021-02-10T05:01:40.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:39 smithi085 conmon[29319]: osdmap e1428: 8 total, 8 up, 8 in 2021-02-10T05:01:40.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:39 smithi085 conmon[29319]: cluster 2021-02-10T05:01:40.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:39 smithi085 conmon[29319]: 2021-02-10T05:01:39.279218 2021-02-10T05:01:40.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:39 smithi085 conmon[29319]: +0000 osd.7 (osd.7) 128 : cluster 2021-02-10T05:01:40.195 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:39 smithi085 conmon[29319]: [DBG] 1.2 scrub ok 2021-02-10T05:01:40.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:39 smithi164 conmon[29288]: (osd.7) 128 : cluster [DBG] 1.2 scrub ok 2021-02-10T05:01:41.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:40 smithi164 conmon[29288]: cluster 2021-02-10T05:01:39.683104 2021-02-10T05:01:41.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:40 smithi164 conmon[29288]: +0000 osd.4 (osd.4) 2705 : cluster [DBG] 5.1 deep-scrub ok 2021-02-10T05:01:41.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:40 smithi164 conmon[29288]: audit 2021-02-10T05:01:39.687767+0000 mgr.y 2021-02-10T05:01:41.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:40 smithi164 conmon[29288]: (mgr.14140) 1608 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:01:41.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:40 smithi164 conmon[29288]: cluster 2021-02-10T05:01:41.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:40 smithi164 conmon[29288]: 2021-02 2021-02-10T05:01:41.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:40 smithi164 conmon[29288]: -10T05:01:40.223013 2021-02-10T05:01:41.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:40 smithi164 conmon[29288]: +0000 osd.5 (osd.5) 234 2021-02-10T05:01:41.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:40 smithi164 conmon[29288]: : cluster [DBG] 2.5 deep-scrub ok 2021-02-10T05:01:41.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:40 smithi164 conmon[29288]: cluster 2021-02-10T05:01:41.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:40 smithi164 conmon[29288]: 2021-02-10T05:01:41.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:40 smithi164 conmon[29288]: 2021 2021-02-10T05:01:41.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:40 smithi164 conmon[29288]: -02-10T05:01:40 2021-02-10T05:01:41.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:40 smithi164 conmon[29288]: .296610+0000 osd.3 (osd.3) 77 : cluster [DBG] 3.8 scrub ok 2021-02-10T05:01:41.408 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:40 smithi085 conmon[29319]: cluster 2021-02-10T05:01:39.683104+0000 osd.4 (osd.4) 2705 : cluster [DBG] 5.1 deep-scrub ok 2021-02-10T05:01:41.408 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:40 smithi085 conmon[29319]: audit 2021-02-10T05:01:39.687767+0000 mgr.y (mgr.14140) 1608 : audit [DBG] from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:01:41.409 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:40 smithi085 conmon[29319]: cluster 2021-02-10T05:01:40.223013+0000 osd.5 (osd.5) 234 : cluster [DBG] 2.5 deep-scrub ok 2021-02-10T05:01:41.409 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:40 smithi085 conmon[29319]: cluster 2021-02-10T05:01:40.296610+0000 osd.3 (osd.3) 77 : cluster [DBG] 3.8 scrub ok 2021-02-10T05:01:41.409 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:40 smithi085 conmon[35961]: cluster 2021-02-10T05:01:41.409 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:40 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T05:01:41.409 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:40 smithi085 conmon[35961]: 05:01:39.683104+0000 osd.4 (osd.4) 2705 : cluster [DBG] 5.1 deep-scrub ok 2021-02-10T05:01:41.409 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:40 smithi085 conmon[35961]: audit 2021- 2021-02-10T05:01:41.409 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:40 smithi085 conmon[35961]: 02-10T05:01:39.687767+0000 mgr.y ( 2021-02-10T05:01:41.409 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:40 smithi085 conmon[35961]: mgr.14140) 1608 : audit [DBG] 2021-02-10T05:01:41.409 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:40 smithi085 conmon[35961]: from='client.34421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-02-10T05:01:41.409 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:40 smithi085 conmon[35961]: cluster 2021- 2021-02-10T05:01:41.409 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:40 smithi085 conmon[35961]: 02-10T05:01:40.223013 2021-02-10T05:01:41.409 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:40 smithi085 conmon[35961]: +0000 osd.5 (osd.5) 2021-02-10T05:01:41.410 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:40 smithi085 conmon[35961]: 234 : cluster [DBG] 2.5 deep-scrub ok 2021-02-10T05:01:41.410 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:40 smithi085 conmon[35961]: cluster 2021-02- 2021-02-10T05:01:41.410 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:40 smithi085 conmon[35961]: 10T05:01:40.296610 2021-02-10T05:01:41.410 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:40 smithi085 conmon[35961]: +0000 osd.3 (osd.3) 2021-02-10T05:01:41.410 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:40 smithi085 conmon[35961]: 77 : cluster [DBG] 3.8 scrub ok 2021-02-10T05:01:42.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:41 smithi164 conmon[29288]: cluster 2021-02-10T05:01:40.603287+0000 2021-02-10T05:01:42.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:41 smithi164 conmon[29288]: mgr.y (mgr.14140) 1609 : cluster [DBG] pgmap v2619: 172 pgs: 172 active+clean; 26 KiB data, 391 MiB used, 715 GiB / 715 GiB avail 2021-02-10T05:01:42.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:41 smithi164 conmon[29288]: cluster 2021-02-10T05:01:40.984870 2021-02-10T05:01:42.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:41 smithi164 conmon[29288]: +0000 mon.a (mon.0) 3208 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2021-02-10T05:01:42.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:41 smithi164 conmon[29288]: cluster 2021-02-10T05:01:42.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:41 smithi164 conmon[29288]: 2021-02-10T05:01 2021-02-10T05:01:42.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:41 smithi164 conmon[29288]: :40.984900+0000 mon.a (mon.0) 3209 : 2021-02-10T05:01:42.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:41 smithi164 conmon[29288]: cluster [INF] Cluster is now healthy 2021-02-10T05:01:42.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:41 smithi085 conmon[29319]: cluster 2021-02-10T05:01:40.603287+0000 mgr.y (mgr.14140) 1609 : 2021-02-10T05:01:42.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:41 smithi085 conmon[29319]: cluster [DBG] pgmap v2619: 172 pgs: 172 active+clean; 26 KiB data, 391 MiB used, 715 GiB / 715 GiB avail 2021-02-10T05:01:42.441 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:41 smithi085 conmon[29319]: cluster 2021-02-10T05:01:40.984870+0000 mon.a (mon.0) 3208 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2021-02-10T05:01:42.442 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:41 smithi085 conmon[29319]: cluster 2021-02-10T05:01:40.984900+0000 mon.a (mon.0) 3209 : cluster [INF] Cluster is now healthy 2021-02-10T05:01:42.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:41 smithi085 conmon[35961]: cluster 2021-02-10T05:01:40.603287+0000 mgr.y ( 2021-02-10T05:01:42.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:41 smithi085 conmon[35961]: mgr.14140) 1609 : cluster [DBG] pgmap v2619: 172 pgs: 172 active+clean; 26 KiB data, 391 MiB used, 715 GiB / 715 GiB avail 2021-02-10T05:01:42.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:41 smithi085 conmon[35961]: cluster 2021-02-10T05:01:40.984870+0000 mon.a (mon.0) 3208 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2021-02-10T05:01:42.442 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:41 smithi085 conmon[35961]: cluster 2021-02-10T05:01:40.984900+0000 mon.a (mon.0) 3209 : cluster [INF] Cluster is now healthy 2021-02-10T05:01:42.752 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.0 dump_ops_in_flight 2021-02-10T05:01:42.965 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:01:43.002 INFO:teuthology.orchestra.run.smithi085.stderr:set osd.0 primary-affinity to 1 (8655362) 2021-02-10T05:01:43.026 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:42 smithi085 conmon[35961]: cluster 2021-02-10T05:01:41.706526+0000 2021-02-10T05:01:43.027 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:42 smithi085 conmon[35961]: osd.4 (osd.4) 2706 : cluster [DBG] 6.1e scrub ok 2021-02-10T05:01:43.027 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:42 smithi085 conmon[35961]: audit 2021-02-10T05:01:42.094513+0000 mon.a 2021-02-10T05:01:43.027 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:42 smithi085 conmon[35961]: (mon.0) 3210 : audit [INF] from='client.? 172.21.15.85:0/3872940652' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T05:01:43.027 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:42 smithi085 conmon[29319]: cluster 2021-02-10T05:01:41.706526+0000 osd.4 2021-02-10T05:01:43.027 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:42 smithi085 conmon[29319]: (osd.4) 2706 : cluster [DBG] 6.1e scrub ok 2021-02-10T05:01:43.028 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:42 smithi085 conmon[29319]: audit 2021-02-10T05:01:42.094513+0000 mon.a (mon.0) 3210 : audit [INF] from='client.? 172.21.15.85:0/3872940652' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T05:01:43.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:42 smithi164 conmon[29288]: cluster 2021-02-10T05:01:41 2021-02-10T05:01:43.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:42 smithi164 conmon[29288]: .706526+0000 osd.4 (osd.4) 2706 : cluster [DBG] 6.1e scrub ok 2021-02-10T05:01:43.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:42 smithi164 conmon[29288]: audit 2021-02-10T05:01:42.094513+0000 mon.a (mon.0 2021-02-10T05:01:43.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:42 smithi164 conmon[29288]: ) 3210 : audit [INF] from='client.? 172.21.15.85:0/3872940652' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2021-02-10T05:01:44.300 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:43 smithi085 conmon[29319]: cluster 2021-02-10T05:01:42.603835+0000 2021-02-10T05:01:44.300 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:43 smithi085 conmon[29319]: mgr.y (mgr.14140) 1610 : cluster [DBG] pgmap v2620: 172 pgs: 172 active+clean; 26 KiB data, 391 MiB used, 715 GiB / 715 GiB avail 2021-02-10T05:01:44.300 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:43 smithi085 conmon[29319]: audit 2021-02-10T05:01:42.998210+0000 mon.a (mon.0) 3211 : audit [INF] from='client.? 172.21.15.85:0/3872940652' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2021-02-10T05:01:44.300 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:43 smithi085 conmon[29319]: cluster 2021-02-10T05:01:42.998301+0000 mon.a (mon.0) 3212 : cluster [DBG] osdmap e1429: 8 total, 8 up, 8 in 2021-02-10T05:01:44.300 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:43 smithi085 conmon[29319]: cluster 2021-02-10T05:01:43 2021-02-10T05:01:44.300 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:43 smithi085 conmon[29319]: .359992+0000 osd.3 (osd.3 2021-02-10T05:01:44.301 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:43 smithi085 conmon[29319]: ) 78 : cluster [DBG] 1.4 scrub ok 2021-02-10T05:01:44.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:43 smithi085 conmon[35961]: cluster 2021-02-10T05:01: 2021-02-10T05:01:44.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:43 smithi085 conmon[35961]: 42.603835+0000 mgr.y (mgr.14140) 1610 : cluster [DBG] pgmap v2620: 172 pgs: 172 active+clean; 26 KiB data, 391 MiB used, 715 GiB / 715 GiB avail 2021-02-10T05:01:44.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:43 smithi085 conmon[35961]: audit 2021-02- 2021-02-10T05:01:44.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:43 smithi085 conmon[35961]: 10T05:01:42.998210 2021-02-10T05:01:44.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:43 smithi085 conmon[35961]: +0000 mon.a (mon.0) 3211 2021-02-10T05:01:44.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:43 smithi085 conmon[35961]: : audit [INF] from='client.? 172.21.15.85:0/3872940652' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2021-02-10T05:01:44.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:43 smithi085 conmon[35961]: cluster 2021-02-10T05:01:44.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:43 smithi085 conmon[35961]: 2021-02-10T 2021-02-10T05:01:44.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:43 smithi085 conmon[35961]: 05:01:42.998301+0000 mon.a (mon 2021-02-10T05:01:44.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:43 smithi085 conmon[35961]: .0) 3212 : cluster [DBG] osdmap e1429: 8 total, 8 up, 8 in 2021-02-10T05:01:44.301 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:43 smithi085 conmon[35961]: cluster 2021-02-10T05:01:44.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:43 smithi085 conmon[35961]: 2021-02- 2021-02-10T05:01:44.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:43 smithi085 conmon[35961]: 10T05:01:43. 2021-02-10T05:01:44.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:43 smithi085 conmon[35961]: 359992+0000 osd.3 (osd.3) 2021-02-10T05:01:44.302 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:43 smithi085 conmon[35961]: 78 : cluster [DBG] 1.4 scrub ok 2021-02-10T05:01:44.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:43 smithi164 conmon[29288]: cluster 2021-02-10T05:01:44.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:43 smithi164 conmon[29288]: 2021-02-10T05:01:44.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:43 smithi164 conmon[29288]: 2021 2021-02-10T05:01:44.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:43 smithi164 conmon[29288]: -02-10T05 2021-02-10T05:01:44.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:43 smithi164 conmon[29288]: :01:42.603835+0000 mgr.y (mgr.14140) 1610 : cluster [DBG] pgmap v2620: 172 pgs: 172 active+clean; 26 KiB data, 391 MiB used, 715 GiB / 715 GiB avail 2021-02-10T05:01:44.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:43 smithi164 conmon[29288]: audit 2021-02-10T05:01:44.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:43 smithi164 conmon[29288]: 2021-02 2021-02-10T05:01:44.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:43 smithi164 conmon[29288]: -10T05:01:42.998210+0000 mon.a (mon.0 2021-02-10T05:01:44.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:43 smithi164 conmon[29288]: ) 3211 : audit [INF] from='client.? 172.21.15.85:0/3872940652' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2021-02-10T05:01:44.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:43 smithi164 conmon[29288]: cluster 2021-02-10T05:01:44.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:43 smithi164 conmon[29288]: 2021-02-10T05:01:42.998301+0000 mon.a 2021-02-10T05:01:44.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:43 smithi164 conmon[29288]: (mon.0) 3212 : cluster [DBG] osdmap e1429: 8 total, 8 up, 8 in 2021-02-10T05:01:44.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:43 smithi164 conmon[29288]: cluster 2021 2021-02-10T05:01:44.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:43 smithi164 conmon[29288]: -02-10T05:01:43.359992+0000 2021-02-10T05:01:44.395 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:43 smithi164 conmon[29288]: osd.3 (osd.3) 78 : cluster [DBG] 1.4 scrub ok 2021-02-10T05:01:44.396 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:43 smithi164 conmon[29288]: 2021-02-10T05:01:44.648 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2021-02-10T05:01:44.649 ERROR:teuthology.run_tasks:Manager failed: thrashosds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run_tasks.py", line 176, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_c1ea624123d412aff8b9d1430e36cb45fcab76b8/qa/tasks/thrashosds.py", line 214, in task thrash_proc.do_join() File "/home/teuthworker/src/git.ceph.com_ceph-c_c1ea624123d412aff8b9d1430e36cb45fcab76b8/qa/tasks/ceph_manager.py", line 729, in do_join self.sighup_thread.get() File "src/gevent/greenlet.py", line 683, in gevent._greenlet.Greenlet.get File "src/gevent/greenlet.py", line 317, in gevent._greenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 47, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_ceph-c_c1ea624123d412aff8b9d1430e36cb45fcab76b8/qa/tasks/ceph_manager.py", line 116, in wrapper return func(self) File "/home/teuthworker/src/git.ceph.com_ceph-c_c1ea624123d412aff8b9d1430e36cb45fcab76b8/qa/tasks/ceph_manager.py", line 1089, in do_sighup self.ceph_manager.signal_osd(osd, signal.SIGHUP, silent=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_c1ea624123d412aff8b9d1430e36cb45fcab76b8/qa/tasks/ceph_manager.py", line 2804, in signal_osd self.cluster).signal(sig, silent=silent) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/daemon/cephadmunit.py", line 123, in signal self.remote.sh(self.kill_cmd(sig)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/remote.py", line 259, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi085 with status 125: 'sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3' 2021-02-10T05:01:44.677 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2021-02-10T05:01:44.687 INFO:tasks.cephadm:Teardown begin 2021-02-10T05:01:44.687 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_c1ea624123d412aff8b9d1430e36cb45fcab76b8/qa/tasks/cephadm.py", line 1265, in task yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run_tasks.py", line 176, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_c1ea624123d412aff8b9d1430e36cb45fcab76b8/qa/tasks/thrashosds.py", line 214, in task thrash_proc.do_join() File "/home/teuthworker/src/git.ceph.com_ceph-c_c1ea624123d412aff8b9d1430e36cb45fcab76b8/qa/tasks/ceph_manager.py", line 729, in do_join self.sighup_thread.get() File "src/gevent/greenlet.py", line 683, in gevent._greenlet.Greenlet.get File "src/gevent/greenlet.py", line 317, in gevent._greenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 47, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_ceph-c_c1ea624123d412aff8b9d1430e36cb45fcab76b8/qa/tasks/ceph_manager.py", line 116, in wrapper return func(self) File "/home/teuthworker/src/git.ceph.com_ceph-c_c1ea624123d412aff8b9d1430e36cb45fcab76b8/qa/tasks/ceph_manager.py", line 1089, in do_sighup self.ceph_manager.signal_osd(osd, signal.SIGHUP, silent=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_c1ea624123d412aff8b9d1430e36cb45fcab76b8/qa/tasks/ceph_manager.py", line 2804, in signal_osd self.cluster).signal(sig, silent=silent) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/daemon/cephadmunit.py", line 123, in signal self.remote.sh(self.kill_cmd(sig)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/remote.py", line 259, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi085 with status 125: 'sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3' 2021-02-10T05:01:44.692 DEBUG:teuthology.orchestra.run.smithi085:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2021-02-10T05:01:44.773 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2021-02-10T05:01:45.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:45 smithi164 conmon[29288]: cluster 2021-02-10T05:01:43.995524+0000 osd.2 (osd.2) 2074 : cluster [DBG] 6.14 scrub ok 2021-02-10T05:01:45.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:45 smithi164 conmon[29288]: cluster 2021-02-10T05:01:44.330898+0000 osd.7 (osd.7) 129 : cluster [DBG] 1.9 scrub ok 2021-02-10T05:01:45.144 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:45 smithi164 conmon[29288]: cluster 2021-02-10T05:01:44.351644+0000 osd.3 (osd.3) 79 : cluster [DBG] 3.7 scrub ok 2021-02-10T05:01:45.191 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:45 smithi085 conmon[35961]: cluster 2021-02-10T05:01:43.995524 2021-02-10T05:01:45.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:45 smithi085 conmon[35961]: +0000 osd.2 (osd.2) 2074 : cluster [DBG] 6.14 scrub ok 2021-02-10T05:01:45.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:45 smithi085 conmon[35961]: cluster 2021-02-10T05:01:44.330898+0000 osd.7 (osd.7) 129 : cluster [DBG] 1.9 scrub ok 2021-02-10T05:01:45.192 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:45 smithi085 conmon[35961]: cluster 2021-02-10T05:01:44.351644+0000 osd.3 (osd.3) 79 : cluster [DBG] 3.7 scrub ok 2021-02-10T05:01:45.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:45 smithi085 conmon[29319]: cluster 2021-02-10T05:01:43.995524+0000 osd.2 (osd.2) 2074 : 2021-02-10T05:01:45.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:45 smithi085 conmon[29319]: cluster [DBG] 6.14 scrub ok 2021-02-10T05:01:45.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:45 smithi085 conmon[29319]: cluster 2021-02-10T05:01:44.330898+0000 osd.7 (osd.7) 129 : cluster [DBG] 1.9 scrub ok 2021-02-10T05:01:45.192 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:45 smithi085 conmon[29319]: cluster 2021-02-10T05:01:44.351644+0000 osd.3 (osd.3) 79 : cluster [DBG] 3.7 scrub ok 2021-02-10T05:01:45.283 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2021-02-10T05:01:45.283 DEBUG:teuthology.orchestra.run.smithi085:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2021-02-10T05:01:45.298 DEBUG:teuthology.orchestra.run.smithi164:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2021-02-10T05:01:45.316 INFO:tasks.cephadm:Stopping all daemons... 2021-02-10T05:01:45.316 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2021-02-10T05:01:45.316 DEBUG:teuthology.orchestra.run.smithi085:> sudo systemctl stop ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mon.a 2021-02-10T05:01:45.439 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T05:01:45.439 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [], 2021-02-10T05:01:45.439 INFO:teuthology.orchestra.run.smithi085.stdout: "num_ops": 0 2021-02-10T05:01:45.439 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T05:01:45.669 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:45 smithi085 systemd[1]: Stopping Ceph mon.a for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T05:01:45.670 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:45 smithi085 conmon[29319]: debug 2021-02-10T05:01:45.434+0000 7f9e16de3700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-02-10T05:01:45.670 INFO:journalctl@ceph.mon.a.smithi085.stdout:Feb 10 05:01:45 smithi085 conmon[29319]: debug 2021-02-10T05:01:45.434+0000 7f9e16de3700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2021-02-10T05:01:47.006 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.0 dump_blocked_ops 2021-02-10T05:01:47.026 DEBUG:teuthology.orchestra.run.smithi085:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mon.a.service' 2021-02-10T05:01:47.112 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-02-10T05:01:47.113 INFO:tasks.cephadm.mon.a:Stopped mon.a 2021-02-10T05:01:47.113 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2021-02-10T05:01:47.113 DEBUG:teuthology.orchestra.run.smithi085:> sudo systemctl stop ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mon.c 2021-02-10T05:01:47.254 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:01:47.295 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:47 smithi085 systemd[1]: Stopping Ceph mon.c for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T05:01:47.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:47 smithi085 conmon[35961]: debug 2021-02-10T05:01:47.399+0000 7fb40a535700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-02-10T05:01:47.640 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:47 smithi085 conmon[35961]: debug 2021-02-10T05:01:47.399+0000 7fb40a535700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2021-02-10T05:01:49.190 INFO:journalctl@ceph.mon.c.smithi085.stdout:Feb 10 05:01:48 smithi085 podman[226292]: dfe56a87fdf2454b954e8bcdbe47b0f656f99216cc4320f72f2274022cf8e9ee 2021-02-10T05:01:49.238 DEBUG:teuthology.orchestra.run.smithi085:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mon.c.service' 2021-02-10T05:01:49.282 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-02-10T05:01:49.282 INFO:tasks.cephadm.mon.c:Stopped mon.c 2021-02-10T05:01:49.282 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2021-02-10T05:01:49.282 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mon.b 2021-02-10T05:01:49.606 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T05:01:49.606 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [], 2021-02-10T05:01:49.606 INFO:teuthology.orchestra.run.smithi085.stdout: "complaint_time": 30, 2021-02-10T05:01:49.606 INFO:teuthology.orchestra.run.smithi085.stdout: "num_blocked_ops": 0 2021-02-10T05:01:49.606 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T05:01:49.644 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:49 smithi164 systemd[1]: Stopping Ceph mon.b for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T05:01:50.065 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 05:01:49 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T05:01:50.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:50 smithi164 conmon[29288]: debug 2021-02-10T05:01:50.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:50 smithi164 conmon[29288]: 2021-02-10T05:01:49.978+0000 7f9651e8e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-02-10T05:01:50.394 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:01:50 smithi164 conmon[29288]: debug 2021-02-10T05:01:50.063+0000 7f9651e8e700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2021-02-10T05:01:50.555 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c1ea624123d412aff8b9d1430e36cb45fcab76b8 shell --fsid 42224fce-6b57-11eb-8feb-001a4aab830c -- ceph daemon osd.0 dump_historic_ops 2021-02-10T05:01:50.790 INFO:teuthology.orchestra.run.smithi085.stderr:Inferring config /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/mon.a/config 2021-02-10T05:01:53.033 INFO:teuthology.orchestra.run.smithi085.stdout:{ 2021-02-10T05:01:53.033 INFO:teuthology.orchestra.run.smithi085.stdout: "size": 20, 2021-02-10T05:01:53.033 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 600, 2021-02-10T05:01:53.033 INFO:teuthology.orchestra.run.smithi085.stdout: "ops": [ 2021-02-10T05:01:53.033 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.033 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDScrubReserve(6.a REQUEST e1336)", 2021-02-10T05:01:53.033 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:59:02.274165+0000", 2021-02-10T05:01:53.033 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 170.75637258099999, 2021-02-10T05:01:53.033 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001713011, 2021-02-10T05:01:53.033 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T05:01:53.034 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T05:01:53.034 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T05:01:53.034 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.034 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T05:01:53.034 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:02.274165+0000", 2021-02-10T05:01:53.034 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T05:01:53.034 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.034 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.034 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T05:01:53.034 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:02.274165+0000", 2021-02-10T05:01:53.034 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.9925e-05 2021-02-10T05:01:53.034 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.035 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.035 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T05:01:53.035 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:02.274185+0000", 2021-02-10T05:01:53.035 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.3339999999999998e-06 2021-02-10T05:01:53.035 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.035 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.035 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T05:01:53.035 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:02.274191+0000", 2021-02-10T05:01:53.035 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.4455e-05 2021-02-10T05:01:53.035 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.035 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.035 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T05:01:53.035 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:02.274206+0000", 2021-02-10T05:01:53.035 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0429e-05 2021-02-10T05:01:53.036 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.036 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.036 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T05:01:53.036 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:02.274216+0000", 2021-02-10T05:01:53.036 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.702e-05 2021-02-10T05:01:53.036 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.036 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.036 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T05:01:53.036 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:02.274273+0000", 2021-02-10T05:01:53.036 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.5929999999999995e-06 2021-02-10T05:01:53.036 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.036 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.036 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op must wait for map", 2021-02-10T05:01:53.036 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:02.274283+0000", 2021-02-10T05:01:53.037 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0015330680000000001 2021-02-10T05:01:53.037 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.037 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.037 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T05:01:53.037 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:02.275816+0000", 2021-02-10T05:01:53.037 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1914e-05 2021-02-10T05:01:53.037 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.037 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.037 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T05:01:53.037 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:02.275828+0000", 2021-02-10T05:01:53.037 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.0272999999999998e-05 2021-02-10T05:01:53.037 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.038 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.038 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T05:01:53.038 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:02.275878+0000", 2021-02-10T05:01:53.038 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001713011 2021-02-10T05:01:53.038 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.038 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:01:53.038 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.038 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.038 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.038 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDScrubReserve(6.a RELEASE e1336)", 2021-02-10T05:01:53.039 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:59:02.274943+0000", 2021-02-10T05:01:53.039 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 170.75559500400001, 2021-02-10T05:01:53.039 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00096610900000000004, 2021-02-10T05:01:53.039 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T05:01:53.039 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T05:01:53.039 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T05:01:53.040 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.040 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T05:01:53.040 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:02.274943+0000", 2021-02-10T05:01:53.040 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T05:01:53.040 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.040 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.040 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T05:01:53.040 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:02.274943+0000", 2021-02-10T05:01:53.040 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.9619e-05 2021-02-10T05:01:53.040 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.040 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.041 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T05:01:53.041 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:02.274962+0000", 2021-02-10T05:01:53.041 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.1049999999999997e-06 2021-02-10T05:01:53.041 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.041 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.041 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T05:01:53.041 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:02.274966+0000", 2021-02-10T05:01:53.041 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.0299999999999999e-06 2021-02-10T05:01:53.041 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.041 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.041 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T05:01:53.041 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:02.274972+0000", 2021-02-10T05:01:53.041 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.7409999999999998e-06 2021-02-10T05:01:53.042 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.042 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.042 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T05:01:53.042 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:02.274976+0000", 2021-02-10T05:01:53.042 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.3995000000000002e-05 2021-02-10T05:01:53.042 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.042 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.042 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T05:01:53.042 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:02.275010+0000", 2021-02-10T05:01:53.042 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.6640000000000003e-06 2021-02-10T05:01:53.042 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.042 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.042 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting_for_map not empty", 2021-02-10T05:01:53.043 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:02.275016+0000", 2021-02-10T05:01:53.043 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00087796700000000001 2021-02-10T05:01:53.043 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.043 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.043 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T05:01:53.043 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:02.275894+0000", 2021-02-10T05:01:53.043 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.4690000000000003e-06 2021-02-10T05:01:53.043 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.043 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.044 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T05:01:53.044 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:02.275903+0000", 2021-02-10T05:01:53.044 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.519e-06 2021-02-10T05:01:53.044 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.044 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.044 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T05:01:53.044 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:02.275909+0000", 2021-02-10T05:01:53.044 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00096610900000000004 2021-02-10T05:01:53.044 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.044 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:01:53.044 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.044 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.044 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.045 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(4.17 1345/1344 [PushReplyOp(4:e9187ab8:reshard::reshard.0000000000:head)])", 2021-02-10T05:01:53.045 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:59:17.012385+0000", 2021-02-10T05:01:53.045 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 156.018152515, 2021-02-10T05:01:53.045 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00055793100000000001, 2021-02-10T05:01:53.045 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T05:01:53.045 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T05:01:53.045 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T05:01:53.045 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.045 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T05:01:53.045 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:17.012385+0000", 2021-02-10T05:01:53.045 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T05:01:53.046 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.046 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.046 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T05:01:53.046 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:17.012385+0000", 2021-02-10T05:01:53.046 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.0549999999999997e-05 2021-02-10T05:01:53.046 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.046 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.046 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T05:01:53.046 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:17.012416+0000", 2021-02-10T05:01:53.046 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.0079999999999999e-06 2021-02-10T05:01:53.046 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.046 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.046 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T05:01:53.047 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:17.012422+0000", 2021-02-10T05:01:53.047 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.8139999999999995e-06 2021-02-10T05:01:53.047 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.047 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.047 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T05:01:53.047 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:17.012432+0000", 2021-02-10T05:01:53.047 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.7389999999999999e-06 2021-02-10T05:01:53.047 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.047 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.047 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T05:01:53.047 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:17.012436+0000", 2021-02-10T05:01:53.047 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00013770399999999999 2021-02-10T05:01:53.047 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.048 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.048 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T05:01:53.048 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:17.012574+0000", 2021-02-10T05:01:53.048 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00036911600000000001 2021-02-10T05:01:53.048 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.048 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.048 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T05:01:53.048 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:17.012943+0000", 2021-02-10T05:01:53.048 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00055793100000000001 2021-02-10T05:01:53.048 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.048 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:01:53.048 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.048 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.049 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.049 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDScrubReserve(6.a REQUEST e1350)", 2021-02-10T05:01:53.049 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:59:30.138752+0000", 2021-02-10T05:01:53.049 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 142.89178531799999, 2021-02-10T05:01:53.049 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0016050369999999999, 2021-02-10T05:01:53.049 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T05:01:53.049 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T05:01:53.049 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T05:01:53.049 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.049 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T05:01:53.049 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:30.138752+0000", 2021-02-10T05:01:53.049 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T05:01:53.049 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.050 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.050 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T05:01:53.050 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:30.138752+0000", 2021-02-10T05:01:53.050 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.6727000000000001e-05 2021-02-10T05:01:53.050 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.050 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.050 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T05:01:53.050 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:30.138769+0000", 2021-02-10T05:01:53.050 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.9029999999999997e-06 2021-02-10T05:01:53.050 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.050 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.051 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T05:01:53.051 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:30.138773+0000", 2021-02-10T05:01:53.051 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.2589999999999997e-06 2021-02-10T05:01:53.051 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.051 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.051 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T05:01:53.051 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:30.138779+0000", 2021-02-10T05:01:53.051 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.5830000000000001e-06 2021-02-10T05:01:53.051 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.051 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.051 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T05:01:53.051 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:30.138787+0000", 2021-02-10T05:01:53.051 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.8128999999999999e-05 2021-02-10T05:01:53.052 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.052 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.052 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T05:01:53.052 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:30.138835+0000", 2021-02-10T05:01:53.052 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.4719999999999998e-06 2021-02-10T05:01:53.052 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.052 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.052 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op must wait for map", 2021-02-10T05:01:53.052 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:30.138842+0000", 2021-02-10T05:01:53.052 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0014528329999999999 2021-02-10T05:01:53.052 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.052 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.053 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T05:01:53.053 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:30.140295+0000", 2021-02-10T05:01:53.053 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2931999999999999e-05 2021-02-10T05:01:53.053 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.053 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.053 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T05:01:53.053 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:30.140308+0000", 2021-02-10T05:01:53.053 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9199000000000001e-05 2021-02-10T05:01:53.053 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.053 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.053 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T05:01:53.053 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:30.140357+0000", 2021-02-10T05:01:53.053 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0016050369999999999 2021-02-10T05:01:53.054 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.054 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:01:53.054 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.054 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.054 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.054 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDScrubReserve(6.a RELEASE e1350)", 2021-02-10T05:01:53.054 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:59:30.139269+0000", 2021-02-10T05:01:53.054 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 142.89126843400001, 2021-02-10T05:01:53.054 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001124671, 2021-02-10T05:01:53.054 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T05:01:53.054 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T05:01:53.054 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T05:01:53.054 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.055 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T05:01:53.055 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:30.139269+0000", 2021-02-10T05:01:53.055 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T05:01:53.055 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.055 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.055 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T05:01:53.055 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:30.139269+0000", 2021-02-10T05:01:53.055 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.8977000000000001e-05 2021-02-10T05:01:53.055 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.055 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.055 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T05:01:53.055 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:30.139288+0000", 2021-02-10T05:01:53.055 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.9880000000000004e-06 2021-02-10T05:01:53.056 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.056 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.056 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T05:01:53.056 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:30.139293+0000", 2021-02-10T05:01:53.056 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.8970000000000008e-06 2021-02-10T05:01:53.056 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.056 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.056 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T05:01:53.056 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:30.139301+0000", 2021-02-10T05:01:53.056 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.789e-06 2021-02-10T05:01:53.056 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.056 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.056 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T05:01:53.057 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:30.139307+0000", 2021-02-10T05:01:53.057 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.4378999999999997e-05 2021-02-10T05:01:53.057 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.057 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.057 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T05:01:53.057 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:30.139341+0000", 2021-02-10T05:01:53.057 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.8940000000000003e-06 2021-02-10T05:01:53.057 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.057 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.057 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "waiting_for_map not empty", 2021-02-10T05:01:53.057 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:30.139345+0000", 2021-02-10T05:01:53.057 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001035315 2021-02-10T05:01:53.057 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.058 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.058 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T05:01:53.058 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:30.140380+0000", 2021-02-10T05:01:53.058 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.8840000000000006e-06 2021-02-10T05:01:53.058 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.058 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.058 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T05:01:53.058 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:30.140390+0000", 2021-02-10T05:01:53.058 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.5480000000000002e-06 2021-02-10T05:01:53.058 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.058 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.058 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T05:01:53.058 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:30.140394+0000", 2021-02-10T05:01:53.059 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.001124671 2021-02-10T05:01:53.059 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.059 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:01:53.059 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.059 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.059 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.059 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDScrubReserve(5.8 REQUEST e1372)", 2021-02-10T05:01:53.059 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T04:59:59.236755+0000", 2021-02-10T05:01:53.059 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 113.793782432, 2021-02-10T05:01:53.059 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031274359999999999, 2021-02-10T05:01:53.059 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T05:01:53.059 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T05:01:53.059 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T05:01:53.060 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.060 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T05:01:53.060 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:59.236755+0000", 2021-02-10T05:01:53.060 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T05:01:53.060 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.060 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.060 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T05:01:53.060 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:59.236755+0000", 2021-02-10T05:01:53.060 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.9686e-05 2021-02-10T05:01:53.060 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.060 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.060 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T05:01:53.061 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:59.236785+0000", 2021-02-10T05:01:53.061 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.0399999999999998e-06 2021-02-10T05:01:53.061 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.061 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.061 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T05:01:53.061 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:59.236791+0000", 2021-02-10T05:01:53.061 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.6860000000000001e-06 2021-02-10T05:01:53.061 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.061 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.061 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T05:01:53.061 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:59.236801+0000", 2021-02-10T05:01:53.061 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.6629999999999993e-06 2021-02-10T05:01:53.061 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.062 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.062 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T05:01:53.062 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:59.236808+0000", 2021-02-10T05:01:53.062 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.4277999999999998e-05 2021-02-10T05:01:53.062 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.062 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.062 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T05:01:53.062 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:59.236863+0000", 2021-02-10T05:01:53.062 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.4900000000000006e-06 2021-02-10T05:01:53.062 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.062 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.062 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op must wait for map", 2021-02-10T05:01:53.062 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:59.236872+0000", 2021-02-10T05:01:53.062 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0029386630000000002 2021-02-10T05:01:53.063 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.063 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.063 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T05:01:53.063 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:59.239811+0000", 2021-02-10T05:01:53.063 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0061000000000001e-05 2021-02-10T05:01:53.063 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.063 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.063 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T05:01:53.063 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:59.239821+0000", 2021-02-10T05:01:53.063 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.1869e-05 2021-02-10T05:01:53.063 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.063 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.063 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T05:01:53.064 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T04:59:59.239883+0000", 2021-02-10T05:01:53.064 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0031274359999999999 2021-02-10T05:01:53.064 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.064 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:01:53.064 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.064 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.064 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.064 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_repop(osd.2.0:220 4.5 e1373/1363 4:a1987d85:::obj_delete_at_hint.0000000114:head v 1373'54, mlcod=0'0)", 2021-02-10T05:01:53.064 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T05:00:02.276576+0000", 2021-02-10T05:01:53.064 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 110.75396154800001, 2021-02-10T05:01:53.064 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00057761800000000001, 2021-02-10T05:01:53.064 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T05:01:53.064 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "commit sent; apply or cleanup", 2021-02-10T05:01:53.064 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T05:01:53.065 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.065 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T05:01:53.065 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:02.276576+0000", 2021-02-10T05:01:53.065 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T05:01:53.065 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.065 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.065 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T05:01:53.065 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:02.276576+0000", 2021-02-10T05:01:53.065 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.8682000000000001e-05 2021-02-10T05:01:53.065 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.065 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.065 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T05:01:53.066 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:02.276605+0000", 2021-02-10T05:01:53.066 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.5189999999999998e-06 2021-02-10T05:01:53.066 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.066 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.066 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T05:01:53.066 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:02.276611+0000", 2021-02-10T05:01:53.066 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.2639999999999997e-06 2021-02-10T05:01:53.066 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.066 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.066 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T05:01:53.066 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:02.276621+0000", 2021-02-10T05:01:53.066 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.8149999999999996e-06 2021-02-10T05:01:53.067 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.067 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.067 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T05:01:53.067 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:02.276625+0000", 2021-02-10T05:01:53.067 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000119449 2021-02-10T05:01:53.067 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.067 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.067 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T05:01:53.067 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:02.276745+0000", 2021-02-10T05:01:53.067 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.0496000000000002e-05 2021-02-10T05:01:53.067 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.067 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.067 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T05:01:53.068 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:02.276765+0000", 2021-02-10T05:01:53.068 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00034179700000000002 2021-02-10T05:01:53.068 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.068 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.068 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "commit_sent", 2021-02-10T05:01:53.068 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:02.277107+0000", 2021-02-10T05:01:53.068 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.6595999999999999e-05 2021-02-10T05:01:53.068 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.068 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.068 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T05:01:53.068 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:02.277154+0000", 2021-02-10T05:01:53.068 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00057761800000000001 2021-02-10T05:01:53.068 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.069 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:01:53.069 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.069 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.069 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.069 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDScrubReserve(4.0 REQUEST e1374)", 2021-02-10T05:01:53.069 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T05:00:02.803198+0000", 2021-02-10T05:01:53.069 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 110.227339969, 2021-02-10T05:01:53.069 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0021227170000000001, 2021-02-10T05:01:53.069 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T05:01:53.069 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T05:01:53.069 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T05:01:53.069 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.069 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T05:01:53.070 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:02.803198+0000", 2021-02-10T05:01:53.070 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T05:01:53.070 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.070 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.070 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T05:01:53.070 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:02.803198+0000", 2021-02-10T05:01:53.070 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.5069999999999999e-05 2021-02-10T05:01:53.070 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.070 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.070 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T05:01:53.070 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:02.803223+0000", 2021-02-10T05:01:53.070 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2551999999999999e-05 2021-02-10T05:01:53.071 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.071 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.071 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T05:01:53.071 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:02.803235+0000", 2021-02-10T05:01:53.071 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2337999999999999e-05 2021-02-10T05:01:53.071 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.071 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.071 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T05:01:53.071 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:02.803248+0000", 2021-02-10T05:01:53.071 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.2599999999999994e-06 2021-02-10T05:01:53.071 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.071 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.071 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T05:01:53.071 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:02.803257+0000", 2021-02-10T05:01:53.072 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.2896000000000002e-05 2021-02-10T05:01:53.072 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.072 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.072 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T05:01:53.072 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:02.803320+0000", 2021-02-10T05:01:53.072 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.4423000000000001e-05 2021-02-10T05:01:53.072 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.072 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.072 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op must wait for map", 2021-02-10T05:01:53.072 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:02.803334+0000", 2021-02-10T05:01:53.072 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0019152399999999999 2021-02-10T05:01:53.072 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.072 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.073 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T05:01:53.073 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:02.805249+0000", 2021-02-10T05:01:53.073 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.4581e-05 2021-02-10T05:01:53.073 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.073 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.073 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T05:01:53.073 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:02.805264+0000", 2021-02-10T05:01:53.073 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.6357000000000003e-05 2021-02-10T05:01:53.073 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.073 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.073 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T05:01:53.073 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:02.805320+0000", 2021-02-10T05:01:53.073 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0021227170000000001 2021-02-10T05:01:53.074 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.074 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:01:53.074 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.074 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.074 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.074 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDScrubReserve(6.26 REQUEST e1375)", 2021-02-10T05:01:53.074 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T05:00:03.673205+0000", 2021-02-10T05:01:53.074 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 109.35733241200001, 2021-02-10T05:01:53.074 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0030123459999999999, 2021-02-10T05:01:53.074 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T05:01:53.074 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T05:01:53.075 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T05:01:53.075 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.075 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T05:01:53.075 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:03.673205+0000", 2021-02-10T05:01:53.075 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T05:01:53.075 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.075 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.075 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T05:01:53.075 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:03.673205+0000", 2021-02-10T05:01:53.075 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.5553000000000002e-05 2021-02-10T05:01:53.076 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.076 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.076 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T05:01:53.076 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:03.673221+0000", 2021-02-10T05:01:53.076 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.3599999999999998e-06 2021-02-10T05:01:53.076 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.076 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.076 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T05:01:53.076 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:03.673225+0000", 2021-02-10T05:01:53.077 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.9499999999999998e-06 2021-02-10T05:01:53.077 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.077 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.077 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T05:01:53.077 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:03.673231+0000", 2021-02-10T05:01:53.077 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.8299999999999998e-06 2021-02-10T05:01:53.077 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.077 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.077 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T05:01:53.077 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:03.673238+0000", 2021-02-10T05:01:53.078 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.4360000000000002e-05 2021-02-10T05:01:53.078 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.078 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.078 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T05:01:53.078 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:03.673282+0000", 2021-02-10T05:01:53.078 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.5680000000000002e-06 2021-02-10T05:01:53.078 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.078 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.078 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op must wait for map", 2021-02-10T05:01:53.078 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:03.673290+0000", 2021-02-10T05:01:53.078 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0028733119999999998 2021-02-10T05:01:53.078 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.078 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.079 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T05:01:53.079 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:03.676163+0000", 2021-02-10T05:01:53.079 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0764000000000001e-05 2021-02-10T05:01:53.079 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.079 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.079 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T05:01:53.079 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:03.676174+0000", 2021-02-10T05:01:53.079 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.3649000000000002e-05 2021-02-10T05:01:53.079 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.079 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.079 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T05:01:53.079 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:03.676218+0000", 2021-02-10T05:01:53.080 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0030123459999999999 2021-02-10T05:01:53.080 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.080 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:01:53.080 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.080 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.080 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.080 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "pg_scan(get_digest 4.19 MIN-MIN e 1381/1344)", 2021-02-10T05:01:53.080 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T05:00:16.972364+0000", 2021-02-10T05:01:53.080 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 96.058173362999995, 2021-02-10T05:01:53.080 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00066968300000000002, 2021-02-10T05:01:53.080 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T05:01:53.080 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T05:01:53.080 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T05:01:53.081 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.081 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T05:01:53.081 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:16.972364+0000", 2021-02-10T05:01:53.081 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T05:01:53.081 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.081 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.081 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T05:01:53.081 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:16.972364+0000", 2021-02-10T05:01:53.081 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.1764000000000003e-05 2021-02-10T05:01:53.081 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.081 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.081 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T05:01:53.081 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:16.972396+0000", 2021-02-10T05:01:53.082 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.6980000000000004e-06 2021-02-10T05:01:53.082 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.082 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.082 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T05:01:53.082 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:16.972405+0000", 2021-02-10T05:01:53.082 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0883999999999999e-05 2021-02-10T05:01:53.082 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.082 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.082 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T05:01:53.082 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:16.972416+0000", 2021-02-10T05:01:53.082 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.404e-06 2021-02-10T05:01:53.082 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.082 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.083 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T05:01:53.083 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:16.972422+0000", 2021-02-10T05:01:53.083 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.8585000000000003e-05 2021-02-10T05:01:53.083 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.083 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.083 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T05:01:53.083 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:16.972471+0000", 2021-02-10T05:01:53.083 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.5764999999999999e-05 2021-02-10T05:01:53.083 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.083 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.083 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T05:01:53.083 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:16.972486+0000", 2021-02-10T05:01:53.083 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00054758300000000003 2021-02-10T05:01:53.083 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.084 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.084 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T05:01:53.084 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:16.973034+0000", 2021-02-10T05:01:53.084 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00066968300000000002 2021-02-10T05:01:53.084 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.084 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:01:53.084 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.084 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.084 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.084 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPushReply(4.16 1381/1344 [PushReplyOp(4:6a988a9b:reshard::reshard.0000000007:head)])", 2021-02-10T05:01:53.084 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T05:00:16.982870+0000", 2021-02-10T05:01:53.084 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 96.047667314999998, 2021-02-10T05:01:53.084 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00066498600000000001, 2021-02-10T05:01:53.084 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T05:01:53.085 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "reached pg", 2021-02-10T05:01:53.085 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T05:01:53.085 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.085 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T05:01:53.085 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:16.982870+0000", 2021-02-10T05:01:53.085 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T05:01:53.085 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.085 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.085 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T05:01:53.085 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:16.982870+0000", 2021-02-10T05:01:53.085 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.1915999999999997e-05 2021-02-10T05:01:53.085 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.085 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.085 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T05:01:53.086 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:16.982902+0000", 2021-02-10T05:01:53.086 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.7150000000000001e-06 2021-02-10T05:01:53.086 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.086 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.086 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T05:01:53.086 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:16.982909+0000", 2021-02-10T05:01:53.086 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1504e-05 2021-02-10T05:01:53.086 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.086 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.086 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T05:01:53.086 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:16.982920+0000", 2021-02-10T05:01:53.086 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00029116800000000001 2021-02-10T05:01:53.086 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.087 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.087 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T05:01:53.087 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:16.983212+0000", 2021-02-10T05:01:53.087 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.5104e-05 2021-02-10T05:01:53.087 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.087 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.087 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T05:01:53.087 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:16.983257+0000", 2021-02-10T05:01:53.087 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00027857899999999999 2021-02-10T05:01:53.087 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.087 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.087 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T05:01:53.087 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:16.983535+0000", 2021-02-10T05:01:53.088 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00066498600000000001 2021-02-10T05:01:53.088 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.088 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:01:53.088 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.088 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.088 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.088 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:4462 5.3 5:c609908c:::notify.5:head [watch ping cookie 94187770079488] snapc 0=[] ondisk+write+known_if_redirected e1378)", 2021-02-10T05:01:53.088 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T05:00:18.090809+0000", 2021-02-10T05:01:53.088 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 94.939729025999995, 2021-02-10T05:01:53.088 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00058754900000000004, 2021-02-10T05:01:53.088 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T05:01:53.088 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T05:01:53.088 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T05:01:53.089 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T05:01:53.089 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T05:01:53.089 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 4462 2021-02-10T05:01:53.089 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.089 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T05:01:53.089 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.089 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T05:01:53.089 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:18.090809+0000", 2021-02-10T05:01:53.089 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T05:01:53.089 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.089 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.089 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T05:01:53.089 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:18.090809+0000", 2021-02-10T05:01:53.090 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.6720000000000002e-05 2021-02-10T05:01:53.090 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.090 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.090 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T05:01:53.090 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:18.090835+0000", 2021-02-10T05:01:53.090 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.7370000000000001e-06 2021-02-10T05:01:53.090 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.090 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.090 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T05:01:53.090 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:18.090845+0000", 2021-02-10T05:01:53.090 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.5783000000000001e-05 2021-02-10T05:01:53.090 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.091 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.091 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T05:01:53.091 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:18.090861+0000", 2021-02-10T05:01:53.091 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.2439999999999993e-06 2021-02-10T05:01:53.091 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.091 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.091 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T05:01:53.091 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:18.090869+0000", 2021-02-10T05:01:53.091 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0002988 2021-02-10T05:01:53.091 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.091 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.091 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T05:01:53.091 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:18.091168+0000", 2021-02-10T05:01:53.091 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.0130000000000006e-05 2021-02-10T05:01:53.092 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.092 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.092 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T05:01:53.092 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:18.091258+0000", 2021-02-10T05:01:53.092 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000138135 2021-02-10T05:01:53.092 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.092 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.092 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T05:01:53.092 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:18.091396+0000", 2021-02-10T05:01:53.092 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00058754900000000004 2021-02-10T05:01:53.092 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.092 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:01:53.092 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.092 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.093 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.093 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:4486 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94187770079488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e1390)", 2021-02-10T05:01:53.093 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T05:00:28.093348+0000", 2021-02-10T05:01:53.093 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 84.937189885999999, 2021-02-10T05:01:53.093 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0024911009999999999, 2021-02-10T05:01:53.093 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T05:01:53.093 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T05:01:53.093 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T05:01:53.093 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T05:01:53.093 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T05:01:53.093 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 4486 2021-02-10T05:01:53.093 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.093 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T05:01:53.093 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.094 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T05:01:53.094 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:28.093348+0000", 2021-02-10T05:01:53.094 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T05:01:53.094 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.094 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.094 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T05:01:53.094 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:28.093348+0000", 2021-02-10T05:01:53.094 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.0528000000000003e-05 2021-02-10T05:01:53.094 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.094 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.094 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T05:01:53.094 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:28.093378+0000", 2021-02-10T05:01:53.095 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.5590000000000006e-06 2021-02-10T05:01:53.095 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.095 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.095 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T05:01:53.095 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:28.093387+0000", 2021-02-10T05:01:53.095 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2012e-05 2021-02-10T05:01:53.095 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.095 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.095 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T05:01:53.095 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:28.093399+0000", 2021-02-10T05:01:53.095 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.143e-06 2021-02-10T05:01:53.095 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.095 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.096 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T05:01:53.096 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:28.093403+0000", 2021-02-10T05:01:53.096 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.5191999999999998e-05 2021-02-10T05:01:53.096 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.096 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.096 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T05:01:53.096 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:28.093438+0000", 2021-02-10T05:01:53.096 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.3709999999999999e-05 2021-02-10T05:01:53.096 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.096 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.096 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T05:01:53.096 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:28.093492+0000", 2021-02-10T05:01:53.097 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0023469570000000002 2021-02-10T05:01:53.097 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.097 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.097 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T05:01:53.097 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:28.095839+0000", 2021-02-10T05:01:53.097 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0024911009999999999 2021-02-10T05:01:53.097 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.097 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:01:53.097 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.097 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.097 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.097 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "pg_scan(get_digest 4.1e MIN-MIN e 1400/1344)", 2021-02-10T05:01:53.098 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T05:00:46.983291+0000", 2021-02-10T05:01:53.098 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 66.047246199, 2021-02-10T05:01:53.098 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00054958199999999996, 2021-02-10T05:01:53.098 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T05:01:53.098 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T05:01:53.098 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T05:01:53.098 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.098 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T05:01:53.098 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:46.983291+0000", 2021-02-10T05:01:53.098 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T05:01:53.098 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.098 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.099 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T05:01:53.099 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:46.983291+0000", 2021-02-10T05:01:53.099 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.8337000000000001e-05 2021-02-10T05:01:53.099 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.099 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.099 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T05:01:53.099 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:46.983310+0000", 2021-02-10T05:01:53.099 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.3820000000000003e-06 2021-02-10T05:01:53.099 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.099 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.099 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T05:01:53.099 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:46.983315+0000", 2021-02-10T05:01:53.099 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.1480000000000006e-06 2021-02-10T05:01:53.099 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.100 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.100 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T05:01:53.100 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:46.983323+0000", 2021-02-10T05:01:53.100 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.653e-06 2021-02-10T05:01:53.100 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.100 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.100 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T05:01:53.100 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:46.983332+0000", 2021-02-10T05:01:53.100 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.8705999999999998e-05 2021-02-10T05:01:53.100 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.100 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.100 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T05:01:53.100 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:46.983381+0000", 2021-02-10T05:01:53.101 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.7839999999999998e-06 2021-02-10T05:01:53.101 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.101 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.101 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T05:01:53.101 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:46.983390+0000", 2021-02-10T05:01:53.101 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000450572 2021-02-10T05:01:53.101 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.101 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.101 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T05:01:53.101 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:46.983841+0000", 2021-02-10T05:01:53.101 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00054958199999999996 2021-02-10T05:01:53.101 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.101 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:01:53.101 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.102 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.102 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.102 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDPGPush(4.1e 1400/1344 [PushOp(4:7e82ab18:lc::lc.13:head, version: 63'22, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:7e82ab18:lc::lc.13:head@63'22, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2021-02-10T05:01:53.102 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T05:00:47.685786+0000", 2021-02-10T05:01:53.102 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 65.344751860000002, 2021-02-10T05:01:53.102 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00085815200000000005, 2021-02-10T05:01:53.102 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T05:01:53.102 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T05:01:53.102 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T05:01:53.102 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.102 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T05:01:53.103 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:47.685786+0000", 2021-02-10T05:01:53.103 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T05:01:53.103 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.103 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.103 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T05:01:53.103 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:47.685786+0000", 2021-02-10T05:01:53.103 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.5309999999999999e-05 2021-02-10T05:01:53.103 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.103 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.103 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T05:01:53.103 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:47.685821+0000", 2021-02-10T05:01:53.103 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.7825e-05 2021-02-10T05:01:53.103 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.103 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.104 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T05:01:53.104 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:47.685839+0000", 2021-02-10T05:01:53.104 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.6670999999999999e-05 2021-02-10T05:01:53.104 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.104 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.104 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T05:01:53.104 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:47.685866+0000", 2021-02-10T05:01:53.104 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2772000000000001e-05 2021-02-10T05:01:53.104 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.104 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.104 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T05:01:53.104 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:47.685878+0000", 2021-02-10T05:01:53.104 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000611021 2021-02-10T05:01:53.104 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.105 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.105 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T05:01:53.105 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:47.686489+0000", 2021-02-10T05:01:53.105 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.5199999999999997e-06 2021-02-10T05:01:53.105 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.105 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.105 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T05:01:53.105 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:47.686498+0000", 2021-02-10T05:01:53.105 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000146033 2021-02-10T05:01:53.105 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.105 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.105 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T05:01:53.105 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:47.686644+0000", 2021-02-10T05:01:53.106 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00085815200000000005 2021-02-10T05:01:53.106 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.106 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:01:53.106 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.106 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.106 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.106 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDScrubReserve(2.10 REQUEST e1409)", 2021-02-10T05:01:53.106 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T05:00:59.229081+0000", 2021-02-10T05:01:53.106 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 53.801456295999998, 2021-02-10T05:01:53.106 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0026567159999999999, 2021-02-10T05:01:53.106 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T05:01:53.106 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T05:01:53.107 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T05:01:53.107 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.107 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T05:01:53.107 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:59.229081+0000", 2021-02-10T05:01:53.107 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T05:01:53.107 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.107 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.107 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T05:01:53.107 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:59.229081+0000", 2021-02-10T05:01:53.107 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.9635e-05 2021-02-10T05:01:53.107 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.107 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.107 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T05:01:53.108 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:59.229111+0000", 2021-02-10T05:01:53.108 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.4420000000000003e-06 2021-02-10T05:01:53.108 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.108 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.108 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T05:01:53.108 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:59.229117+0000", 2021-02-10T05:01:53.108 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.6609999999999995e-06 2021-02-10T05:01:53.108 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.108 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.108 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T05:01:53.108 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:59.229127+0000", 2021-02-10T05:01:53.108 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.1609999999999997e-06 2021-02-10T05:01:53.108 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.109 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.109 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T05:01:53.109 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:59.229134+0000", 2021-02-10T05:01:53.109 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.6776e-05 2021-02-10T05:01:53.109 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.109 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.109 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T05:01:53.109 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:59.229191+0000", 2021-02-10T05:01:53.109 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.5690000000000005e-06 2021-02-10T05:01:53.109 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.109 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.109 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op must wait for map", 2021-02-10T05:01:53.110 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:59.229200+0000", 2021-02-10T05:01:53.110 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0024834829999999999 2021-02-10T05:01:53.110 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.110 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.110 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T05:01:53.110 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:59.231683+0000", 2021-02-10T05:01:53.110 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.2636999999999999e-05 2021-02-10T05:01:53.110 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.110 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.110 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T05:01:53.111 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:59.231696+0000", 2021-02-10T05:01:53.111 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.2351999999999997e-05 2021-02-10T05:01:53.111 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.111 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.111 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T05:01:53.111 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:00:59.231738+0000", 2021-02-10T05:01:53.111 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0026567159999999999 2021-02-10T05:01:53.111 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.111 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:01:53.111 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.111 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.111 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.111 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDScrubReserve(2.1d REQUEST e1412)", 2021-02-10T05:01:53.112 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T05:01:08.154870+0000", 2021-02-10T05:01:53.112 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 44.875668007999998, 2021-02-10T05:01:53.112 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0036193079999999999, 2021-02-10T05:01:53.112 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T05:01:53.112 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T05:01:53.112 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T05:01:53.112 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.112 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T05:01:53.112 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:08.154870+0000", 2021-02-10T05:01:53.112 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T05:01:53.112 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.112 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.112 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T05:01:53.113 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:08.154870+0000", 2021-02-10T05:01:53.113 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.0219e-05 2021-02-10T05:01:53.113 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.113 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.113 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T05:01:53.113 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:08.154900+0000", 2021-02-10T05:01:53.113 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.1689999999999997e-06 2021-02-10T05:01:53.113 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.113 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.113 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T05:01:53.113 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:08.154906+0000", 2021-02-10T05:01:53.114 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.7599999999999997e-06 2021-02-10T05:01:53.114 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.114 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.114 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T05:01:53.114 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:08.154916+0000", 2021-02-10T05:01:53.114 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.9819999999999997e-06 2021-02-10T05:01:53.115 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.115 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.115 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T05:01:53.115 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:08.154925+0000", 2021-02-10T05:01:53.115 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.8000000000000001e-05 2021-02-10T05:01:53.115 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.115 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.115 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T05:01:53.115 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:08.154973+0000", 2021-02-10T05:01:53.115 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 7.3030000000000002e-06 2021-02-10T05:01:53.115 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.115 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.116 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op must wait for map", 2021-02-10T05:01:53.116 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:08.154980+0000", 2021-02-10T05:01:53.116 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0034534729999999999 2021-02-10T05:01:53.116 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.116 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.116 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T05:01:53.116 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:08.158434+0000", 2021-02-10T05:01:53.116 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.1956999999999999e-05 2021-02-10T05:01:53.116 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.116 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.116 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T05:01:53.116 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:08.158446+0000", 2021-02-10T05:01:53.116 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 4.3445000000000002e-05 2021-02-10T05:01:53.117 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.117 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.117 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T05:01:53.117 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:08.158489+0000", 2021-02-10T05:01:53.117 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0036193079999999999 2021-02-10T05:01:53.117 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.117 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:01:53.117 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.117 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.117 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.117 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:4561 5.3 5:c609908c:::notify.5:head [watch ping cookie 94187770079488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e1414)", 2021-02-10T05:01:53.117 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T05:01:18.101695+0000", 2021-02-10T05:01:53.117 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 34.928842871000001, 2021-02-10T05:01:53.118 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00081085200000000004, 2021-02-10T05:01:53.118 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T05:01:53.118 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T05:01:53.118 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T05:01:53.118 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T05:01:53.118 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T05:01:53.118 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 4561 2021-02-10T05:01:53.118 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.118 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T05:01:53.118 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.118 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T05:01:53.119 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:18.101695+0000", 2021-02-10T05:01:53.119 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T05:01:53.119 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.119 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.119 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T05:01:53.119 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:18.101695+0000", 2021-02-10T05:01:53.119 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.9116000000000003e-05 2021-02-10T05:01:53.119 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.119 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.119 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T05:01:53.119 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:18.101734+0000", 2021-02-10T05:01:53.119 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.234e-05 2021-02-10T05:01:53.119 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.120 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.120 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T05:01:53.120 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:18.101746+0000", 2021-02-10T05:01:53.120 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.9025000000000002e-05 2021-02-10T05:01:53.120 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.120 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.120 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T05:01:53.120 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:18.101765+0000", 2021-02-10T05:01:53.120 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0189e-05 2021-02-10T05:01:53.120 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.120 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.120 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T05:01:53.120 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:18.101775+0000", 2021-02-10T05:01:53.121 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00053074399999999999 2021-02-10T05:01:53.121 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.121 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.121 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T05:01:53.121 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:18.102306+0000", 2021-02-10T05:01:53.121 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.9240000000000006e-05 2021-02-10T05:01:53.121 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.121 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.121 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T05:01:53.121 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:18.102395+0000", 2021-02-10T05:01:53.121 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000110198 2021-02-10T05:01:53.121 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.121 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.121 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T05:01:53.121 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:18.102506+0000", 2021-02-10T05:01:53.122 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.00081085200000000004 2021-02-10T05:01:53.122 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.122 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:01:53.122 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.122 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.122 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.122 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "osd_op(client.24373.0:4569 5.3 5:c609908c:::notify.5:head [watch ping cookie 94187770079488 gen 1] snapc 0=[] ondisk+write+known_if_redirected e1417)", 2021-02-10T05:01:53.122 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T05:01:23.102815+0000", 2021-02-10T05:01:53.122 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 29.927722839000001, 2021-02-10T05:01:53.122 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000796093, 2021-02-10T05:01:53.122 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T05:01:53.122 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T05:01:53.123 INFO:teuthology.orchestra.run.smithi085.stdout: "client_info": { 2021-02-10T05:01:53.123 INFO:teuthology.orchestra.run.smithi085.stdout: "client": "client.24373", 2021-02-10T05:01:53.123 INFO:teuthology.orchestra.run.smithi085.stdout: "client_addr": "172.21.15.85:0/1810546809", 2021-02-10T05:01:53.123 INFO:teuthology.orchestra.run.smithi085.stdout: "tid": 4569 2021-02-10T05:01:53.123 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.123 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T05:01:53.123 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.123 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T05:01:53.124 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:23.102815+0000", 2021-02-10T05:01:53.124 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T05:01:53.124 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.124 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.124 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T05:01:53.125 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:23.102815+0000", 2021-02-10T05:01:53.125 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 3.0267999999999999e-05 2021-02-10T05:01:53.125 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.125 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.125 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T05:01:53.125 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:23.102845+0000", 2021-02-10T05:01:53.125 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0769e-05 2021-02-10T05:01:53.125 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.125 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.125 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T05:01:53.126 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:23.102856+0000", 2021-02-10T05:01:53.126 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.4959e-05 2021-02-10T05:01:53.126 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.126 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.126 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T05:01:53.126 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:23.102871+0000", 2021-02-10T05:01:53.126 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.7609999999999996e-06 2021-02-10T05:01:53.126 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.126 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.126 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T05:01:53.126 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:23.102880+0000", 2021-02-10T05:01:53.126 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000498994 2021-02-10T05:01:53.126 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.126 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.127 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T05:01:53.127 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:23.103379+0000", 2021-02-10T05:01:53.127 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000108501 2021-02-10T05:01:53.127 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.127 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.127 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T05:01:53.127 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:23.103487+0000", 2021-02-10T05:01:53.127 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000123841 2021-02-10T05:01:53.127 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.127 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.127 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T05:01:53.127 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:23.103611+0000", 2021-02-10T05:01:53.127 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.000796093 2021-02-10T05:01:53.127 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.128 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:01:53.128 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.128 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.128 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.128 INFO:teuthology.orchestra.run.smithi085.stdout: "description": "MOSDScrubReserve(5.15 REQUEST e1424)", 2021-02-10T05:01:53.128 INFO:teuthology.orchestra.run.smithi085.stdout: "initiated_at": "2021-02-10T05:01:31.161187+0000", 2021-02-10T05:01:53.128 INFO:teuthology.orchestra.run.smithi085.stdout: "age": 21.869350190999999, 2021-02-10T05:01:53.128 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0020410129999999999, 2021-02-10T05:01:53.128 INFO:teuthology.orchestra.run.smithi085.stdout: "type_data": { 2021-02-10T05:01:53.128 INFO:teuthology.orchestra.run.smithi085.stdout: "flag_point": "started", 2021-02-10T05:01:53.128 INFO:teuthology.orchestra.run.smithi085.stdout: "events": [ 2021-02-10T05:01:53.128 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.128 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "initiated", 2021-02-10T05:01:53.128 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:31.161187+0000", 2021-02-10T05:01:53.129 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0 2021-02-10T05:01:53.129 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.129 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.129 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "throttled", 2021-02-10T05:01:53.129 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:31.161187+0000", 2021-02-10T05:01:53.129 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.9614000000000001e-05 2021-02-10T05:01:53.129 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.129 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.129 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "header_read", 2021-02-10T05:01:53.129 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:31.161217+0000", 2021-02-10T05:01:53.129 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 6.5420000000000002e-06 2021-02-10T05:01:53.129 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.129 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.130 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "all_read", 2021-02-10T05:01:53.130 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:31.161224+0000", 2021-02-10T05:01:53.130 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 1.0482999999999999e-05 2021-02-10T05:01:53.130 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.130 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.130 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "dispatched", 2021-02-10T05:01:53.130 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:31.161234+0000", 2021-02-10T05:01:53.130 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 9.2410000000000001e-06 2021-02-10T05:01:53.130 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.130 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.130 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "queued_for_pg", 2021-02-10T05:01:53.130 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:31.161243+0000", 2021-02-10T05:01:53.130 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 5.5288999999999999e-05 2021-02-10T05:01:53.131 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.131 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.131 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T05:01:53.131 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:31.161299+0000", 2021-02-10T05:01:53.131 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.2640000000000008e-06 2021-02-10T05:01:53.131 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.131 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.131 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "op must wait for map", 2021-02-10T05:01:53.131 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:31.161307+0000", 2021-02-10T05:01:53.131 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0018844700000000001 2021-02-10T05:01:53.131 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.131 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.131 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "reached_pg", 2021-02-10T05:01:53.131 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:31.163191+0000", 2021-02-10T05:01:53.132 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 8.0620000000000001e-06 2021-02-10T05:01:53.132 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.132 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.132 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "started", 2021-02-10T05:01:53.132 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:31.163199+0000", 2021-02-10T05:01:53.132 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 2.9048e-05 2021-02-10T05:01:53.132 INFO:teuthology.orchestra.run.smithi085.stdout: }, 2021-02-10T05:01:53.132 INFO:teuthology.orchestra.run.smithi085.stdout: { 2021-02-10T05:01:53.132 INFO:teuthology.orchestra.run.smithi085.stdout: "event": "done", 2021-02-10T05:01:53.132 INFO:teuthology.orchestra.run.smithi085.stdout: "time": "2021-02-10T05:01:31.163228+0000", 2021-02-10T05:01:53.132 INFO:teuthology.orchestra.run.smithi085.stdout: "duration": 0.0020410129999999999 2021-02-10T05:01:53.132 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.132 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:01:53.133 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.133 INFO:teuthology.orchestra.run.smithi085.stdout: } 2021-02-10T05:01:53.133 INFO:teuthology.orchestra.run.smithi085.stdout: ] 2021-02-10T05:01:53.133 INFO:teuthology.orchestra.run.smithi085.stdout:} 2021-02-10T05:02:00.144 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 05:01:59 smithi164 conmon[48560]: debug there is no tcmu-runner data avaliable 2021-02-10T05:02:01.057 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mon.b.service' 2021-02-10T05:02:01.086 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:02:00 smithi164 podman[97671]: 95a4c069d77f678a21ba47992165063ca09e7ba80462e765e4549b813a59920c 2021-02-10T05:02:01.086 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:02:01 smithi164 systemd[1]: ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mon.b.service: Succeeded. 2021-02-10T05:02:01.086 INFO:journalctl@ceph.mon.b.smithi164.stdout:Feb 10 05:02:01 smithi164 systemd[1]: Stopped Ceph mon.b for 42224fce-6b57-11eb-8feb-001a4aab830c. 2021-02-10T05:02:01.126 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-02-10T05:02:01.126 INFO:tasks.cephadm.mon.b:Stopped mon.b 2021-02-10T05:02:01.126 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2021-02-10T05:02:01.126 DEBUG:teuthology.orchestra.run.smithi085:> sudo systemctl stop ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mgr.y 2021-02-10T05:02:01.441 INFO:journalctl@ceph.mgr.y.smithi085.stdout:Feb 10 05:02:01 smithi085 systemd[1]: Stopping Ceph mgr.y for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T05:02:01.441 INFO:journalctl@ceph.mgr.y.smithi085.stdout:Feb 10 05:02:01 smithi085 conmon[29692]: debug 2021-02-10T05:02:03.086 DEBUG:teuthology.orchestra.run.smithi085:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mgr.y.service' 2021-02-10T05:02:03.123 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-02-10T05:02:03.123 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2021-02-10T05:02:03.123 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2021-02-10T05:02:03.124 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mgr.x 2021-02-10T05:02:03.569 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Feb 10 05:02:03 smithi164 systemd[1]: Stopping Ceph mgr.x for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T05:02:19.862 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mgr.x.service' 2021-02-10T05:02:19.894 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Feb 10 05:02:19 smithi164 podman[97809]: d2e6b5b121c998c03d5316f561dc6f4337773ef16b2ef8bd0673bc4fffb5a47e 2021-02-10T05:02:19.894 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Feb 10 05:02:19 smithi164 systemd[1]: ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mgr.x.service: Main process exited, code=exited, status=143/n/a 2021-02-10T05:02:19.894 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Feb 10 05:02:19 smithi164 systemd[1]: ceph-42224fce-6b57-11eb-8feb-001a4aab830c@mgr.x.service: Failed with result 'exit-code'. 2021-02-10T05:02:19.894 INFO:journalctl@ceph.mgr.x.smithi164.stdout:Feb 10 05:02:19 smithi164 systemd[1]: Stopped Ceph mgr.x for 42224fce-6b57-11eb-8feb-001a4aab830c. 2021-02-10T05:02:19.936 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-02-10T05:02:19.936 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2021-02-10T05:02:19.936 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2021-02-10T05:02:19.936 DEBUG:teuthology.orchestra.run.smithi085:> sudo systemctl stop ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.0 2021-02-10T05:02:20.269 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 05:02:19 smithi085 systemd[1]: Stopping Ceph osd.0 for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T05:02:20.269 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 05:02:20 smithi085 conmon[207560]: debug 2021-02-10T05:02:20.041+0000 7f26b3dd8700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T05:02:20.270 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 05:02:20 smithi085 conmon[207560]: debug 2021-02-10T05:02:20.041+0000 7f26b3dd8700 -1 osd.0 1429 *** Got signal Terminated *** 2021-02-10T05:02:20.270 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 05:02:20 smithi085 conmon[207560]: debug 2021-02-10T05:02:20.041+0000 7f26b3dd8700 -1 osd.0 1429 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-02-10T05:02:23.180 INFO:journalctl@ceph.osd.0.smithi085.stdout:Feb 10 05:02:22 smithi085 podman[227669]: 45efca45f6704f069274172006007b59e5dc24b241978723ce7baff2bdc0c00b 2021-02-10T05:02:25.921 DEBUG:teuthology.orchestra.run.smithi085:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.0.service' 2021-02-10T05:02:25.965 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-02-10T05:02:25.965 INFO:tasks.cephadm.osd.0:Stopped osd.0 2021-02-10T05:02:25.965 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2021-02-10T05:02:25.965 DEBUG:teuthology.orchestra.run.smithi085:> sudo systemctl stop ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.1 2021-02-10T05:02:26.192 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 05:02:26 smithi085 systemd[1]: Stopping Ceph osd.1 for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T05:02:26.192 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 05:02:26 smithi085 conmon[45443]: debug 2021-02-10T05:02:26.093+0000 7f71435f5700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T05:02:26.192 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 05:02:26 smithi085 conmon[45443]: debug 2021-02-10T05:02:26.093+0000 7f71435f5700 -1 osd.1 1429 *** Got signal Terminated *** 2021-02-10T05:02:26.192 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 05:02:26 smithi085 conmon[45443]: debug 2021-02-10T05:02:26.093+0000 7f71435f5700 -1 osd.1 1429 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-02-10T05:02:27.280 INFO:journalctl@ceph.osd.1.smithi085.stdout:Feb 10 05:02:26 smithi085 podman[228008]: Error: container 28e9e0ac35f0e85835b493e2043d79a65bfe9b0ce5a7af3e29df78dbe9f45eb0 does not exist in database: no such container 2021-02-10T05:02:28.400 DEBUG:teuthology.orchestra.run.smithi085:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.1.service' 2021-02-10T05:02:28.436 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-02-10T05:02:28.437 INFO:tasks.cephadm.osd.1:Stopped osd.1 2021-02-10T05:02:28.437 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2021-02-10T05:02:28.437 DEBUG:teuthology.orchestra.run.smithi085:> sudo systemctl stop ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.2 2021-02-10T05:02:28.644 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:28 smithi164 conmon[44544]: debug 2021-02-10T05:02:28.161+0000 7f64ac096700 -1 osd.7 pg_epoch: 1429 pg[2.b( empty local-lis/les=1426/1427 n=0 ec=48/48 lis/c=1426/1426 les/c/f=1427/1427/0 sis=1426) [7,4,5] r=0 lpr=1426 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.b) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 5 (0x55caba38ef20) 2021-02-10T05:02:28.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 05:02:28 smithi085 systemd[1]: Stopping Ceph osd.2 for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T05:02:28.691 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 05:02:28 smithi085 conmon[49106]: debug 2021-02-10T05:02:28.560+0000 7f77a8f1b700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T05:02:28.692 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 05:02:28 smithi085 conmon[49106]: debug 2021-02-10T05:02:28.560+0000 7f77a8f1b700 -1 osd.2 1429 *** Got signal Terminated *** 2021-02-10T05:02:28.692 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 05:02:28 smithi085 conmon[49106]: debug 2021-02-10T05:02:28.560+0000 7f77a8f1b700 -1 osd.2 1429 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-02-10T05:02:29.441 INFO:journalctl@ceph.osd.2.smithi085.stdout:Feb 10 05:02:29 smithi085 podman[228335]: 1d6622b188396745195b7d2f5979db49c62221e7a990092cdf96d2e398c22e07 2021-02-10T05:02:30.357 DEBUG:teuthology.orchestra.run.smithi085:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.2.service' 2021-02-10T05:02:30.393 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-02-10T05:02:30.393 INFO:tasks.cephadm.osd.2:Stopped osd.2 2021-02-10T05:02:30.393 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2021-02-10T05:02:30.393 DEBUG:teuthology.orchestra.run.smithi085:> sudo systemctl stop ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.3 2021-02-10T05:02:30.691 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 05:02:30 smithi085 systemd[1]: Stopping Ceph osd.3 for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T05:02:30.781 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 05:02:30 smithi085 conmon[147676]: debug 2021-02-10T05:02:30.515+0000 7fb04c597700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T05:02:30.781 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 05:02:30 smithi085 conmon[147676]: debug 2021-02-10T05:02:30.515+0000 7fb04c597700 -1 osd.3 1429 *** Got signal Terminated *** 2021-02-10T05:02:30.782 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 05:02:30 smithi085 conmon[147676]: debug 2021-02-10T05:02:30.515+0000 7fb04c597700 -1 osd.3 1429 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-02-10T05:02:31.436 INFO:journalctl@ceph.osd.3.smithi085.stdout:Feb 10 05:02:31 smithi085 podman[228660]: Error: container 63f0ddb1ac253738c578074d3651b1a5c0ae11f7b9d0e5968d8e739f894fd01f does not exist in database: no such container 2021-02-10T05:02:32.233 DEBUG:teuthology.orchestra.run.smithi085:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.3.service' 2021-02-10T05:02:32.269 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-02-10T05:02:32.269 INFO:tasks.cephadm.osd.3:Stopped osd.3 2021-02-10T05:02:32.269 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2021-02-10T05:02:32.270 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.4 2021-02-10T05:02:32.610 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 05:02:32 smithi164 systemd[1]: Stopping Ceph osd.4 for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T05:02:32.610 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 05:02:32 smithi164 conmon[33677]: debug 2021-02-10T05:02:32.608+0000 7fbec6a28700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T05:02:32.611 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 05:02:32 smithi164 conmon[33677]: debug 2021-02-10T05:02:32.608+0000 7fbec6a28700 -1 osd.4 1429 *** Got signal Terminated *** 2021-02-10T05:02:32.893 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 05:02:32 smithi164 conmon[33677]: debug 2021-02-10T05:02:32.618+0000 7fbec6a28700 -1 osd.4 1429 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-02-10T05:02:41.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:41 smithi164 conmon[37263]: debug 2021-02-10T05:02:41.083+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:02:42.393 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:42 smithi164 conmon[37263]: debug 2021-02-10T05:02:42.086+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:02:43.144 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:43 smithi164 conmon[37263]: debug 2021-02-10T05:02:43.092+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:02:43.144 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:43 smithi164 conmon[44544]: debug 2021-02-10T05:02:43.063+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:02:43.144 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:42 smithi164 conmon[40892]: debug 2021-02-10T05:02:42.852+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:02:44.090 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:43 smithi164 conmon[40892]: debug 2021-02-10T05:02:43.805+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:02:44.393 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:44 smithi164 conmon[37263]: debug 2021-02-10T05:02:44.094+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:02:44.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:44 smithi164 conmon[44544]: debug 2021-02-10T05:02:44.088+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:02:45.126 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:44 smithi164 conmon[40892]: debug 2021-02-10T05:02:44.771+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:02:45.126 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:45 smithi164 conmon[44544]: debug 2021-02-10T05:02:45.124+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:02:45.393 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:45 smithi164 conmon[37263]: debug 2021-02-10T05:02:45.125+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:02:46.080 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:45 smithi164 conmon[40892]: debug 2021-02-10T05:02:45.787+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:02:46.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:46 smithi164 conmon[37263]: debug 2021-02-10T05:02:46.137+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:02:46.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:46 smithi164 conmon[44544]: debug 2021-02-10T05:02:46.078+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:02:47.100 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:46 smithi164 conmon[40892]: debug 2021-02-10T05:02:46.834+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:02:47.100 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:47 smithi164 conmon[44544]: debug 2021-02-10T05:02:47.071+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:02:47.393 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:47 smithi164 conmon[37263]: debug 2021-02-10T05:02:47.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:47 smithi164 conmon[37263]: 2021-02-10T05:02:47.098+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:02:48.093 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:47 smithi164 conmon[40892]: debug 2021-02-10T05:02:47.793+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:02:48.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:48 smithi164 conmon[44544]: debug 2021-02-10T05:02:48.104+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:02:48.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:48 smithi164 conmon[37263]: debug 2021-02-10T05:02:48.091+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:02:48.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:48 smithi164 conmon[37263]: debug 2021-02-10T05:02:48.091+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:23.564321+0000 front 2021-02-10T05:02:23.564413+0000 (oldest deadline 2021-02-10T05:02:47.664447+0000) 2021-02-10T05:02:48.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:48 smithi164 conmon[37263]: 2021-02-10T05:02:49.088 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:48 smithi164 conmon[40892]: debug 2021-02-10T05:02:48.792+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:02:49.088 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:48 smithi164 conmon[40892]: debug 2021-02-10T05:02:48.792+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:02:49.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:49 smithi164 conmon[44544]: debug 2021-02-10T05:02:49.130+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:02:49.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:49 smithi164 conmon[37263]: debug 2021-02-10T05:02:49.086+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:02:49.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:49 smithi164 conmon[37263]: debug 2021-02-10T05:02:49.086+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:23.564321+0000 front 2021-02-10T05:02:23.564413+0000 (oldest deadline 2021-02-10T05:02:47.664447+0000) 2021-02-10T05:02:50.059 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:49 smithi164 conmon[40892]: debug 2021-02-10T05:02:49.747+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:02:50.060 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:49 smithi164 conmon[40892]: debug 2021-02-10T05:02:49.747+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:02:50.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:50 smithi164 conmon[37263]: debug 2021-02-10T05:02:50.057+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:02:50.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:50 smithi164 conmon[37263]: debug 2021-02-10T05:02:50.057+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:23.564321+0000 front 2021-02-10T05:02:23.564413+0000 (oldest deadline 2021-02-10T05:02:47.664447+0000) 2021-02-10T05:02:50.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:50 smithi164 conmon[44544]: debug 2021-02-10T05:02:50.130+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:02:50.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:50 smithi164 conmon[44544]: debug 2021-02-10T05:02:50.130+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:02:50.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:50 smithi164 conmon[44544]: debug 2021-02-10T05:02:50.130+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:02:51.072 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:50 smithi164 conmon[40892]: debug 2021-02-10T05:02:50.744+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:02:51.072 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:50 smithi164 conmon[40892]: debug 2021-02-10T05:02:50.744+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:02:51.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:51 smithi164 conmon[37263]: debug 2021-02-10T05:02:51.070+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:02:51.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:51 smithi164 conmon[37263]: debug 2021-02-10T05:02:51.070+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:23.564321+0000 front 2021-02-10T05:02:23.564413+0000 (oldest deadline 2021-02-10T05:02:47.664447+0000) 2021-02-10T05:02:51.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:51 smithi164 conmon[44544]: debug 2021-02-10T05:02:51.157+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:02:51.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:51 smithi164 conmon[44544]: debug 2021-02-10T05:02:51.157+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:02:51.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:51 smithi164 conmon[44544]: debug 2021-02-10T05:02:51.157+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:02:51.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:51 smithi164 conmon[44544]: debug 2021-02-10T05:02:51.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:51 smithi164 conmon[44544]: 2021-02-10T05:02:51.157+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:02:52.117 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:51 smithi164 conmon[40892]: debug 2021-02-10T05:02:51.785+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:02:52.117 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:51 smithi164 conmon[40892]: debug 2021-02-10T05:02:51.785+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:02:52.117 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:51 smithi164 conmon[40892]: debug 2021-02-10T05:02:51.785+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:02:52.117 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:51 smithi164 conmon[40892]: debug 2021-02-10T05:02:51.785+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:02:52.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:52 smithi164 conmon[37263]: debug 2021-02-10T05:02:52.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:52 smithi164 conmon[37263]: 2021-02-10T05:02:52.115+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:02:52.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:52 smithi164 conmon[37263]: debug 2021-02-10T05:02:52.115+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:23.564321+0000 front 2021-02-10T05:02:23.564413+0000 (oldest deadline 2021-02-10T05:02:47.664447+0000) 2021-02-10T05:02:52.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:52 smithi164 conmon[37263]: debug 2021-02-10T05:02:52.115+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:27.665225+0000 front 2021-02-10T05:02:27.665010+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:02:52.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:52 smithi164 conmon[37263]: debug 2021-02-10T05:02:52.115+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:27.665562+0000 front 2021-02-10T05:02:27.665151+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:02:52.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:52 smithi164 conmon[37263]: 2021-02-10T05:02:52.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:52 smithi164 conmon[44544]: debug 2021-02-10T05:02:52.179+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:02:52.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:52 smithi164 conmon[44544]: debug 2021-02-10T05:02:52.179+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:02:52.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:52 smithi164 conmon[44544]: debug 2021-02-10T05:02:52.179+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:02:52.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:52 smithi164 conmon[44544]: debug 2021-02-10T05:02:52.179+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:02:53.144 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:52 smithi164 conmon[40892]: debug 2021-02-10T05:02:52.795+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:02:53.144 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:52 smithi164 conmon[40892]: debug 2021-02-10T05:02:53.144 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:52 smithi164 conmon[40892]: 2021-02-10T05:02:52.795+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:02:53.144 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:52 smithi164 conmon[40892]: debug 2021-02-10T05:02:52.795+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:02:53.144 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:52 smithi164 conmon[40892]: debug 2021-02-10T05:02:52.795+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:02:53.144 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:52 smithi164 conmon[40892]: debug 2021-02-10T05:02:52.795+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:02:53.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:53 smithi164 conmon[37263]: debug 2021-02-10T05:02:53.162+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:02:53.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:53 smithi164 conmon[37263]: debug 2021-02-10T05:02:53.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:53 smithi164 conmon[37263]: 2021-02-10T05:02:53.162+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:23.564321+0000 front 2021-02-10T05:02:23.564413+0000 (oldest deadline 2021-02-10T05:02:47.664447+0000) 2021-02-10T05:02:53.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:53 smithi164 conmon[37263]: debug 2021-02-10T05:02:53.162+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:27.665225+0000 front 2021-02-10T05:02:27.665010+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:02:53.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:53 smithi164 conmon[37263]: debug 2021-02-10T05:02:53.162+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:27.665562+0000 front 2021-02-10T05:02:27.665151+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:02:53.644 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:53 smithi164 conmon[44544]: debug 2021-02-10T05:02:53.644 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:53 smithi164 conmon[44544]: 2021-02-10T05:02:53.190+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:02:53.644 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:53 smithi164 conmon[44544]: debug 2021-02-10T05:02:53.190+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:02:53.645 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:53 smithi164 conmon[44544]: debug 2021-02-10T05:02:53.190+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:02:53.645 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:53 smithi164 conmon[44544]: debug 2021-02-10T05:02:53.190+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:02:54.144 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:53 smithi164 conmon[40892]: debug 2021-02-10T05:02:54.144 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:53 smithi164 conmon[40892]: 2021-02-10T05:02:53.749+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:02:54.144 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:53 smithi164 conmon[40892]: debug 2021-02-10T05:02:53.749+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:02:54.144 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:53 smithi164 conmon[40892]: debug 2021-02-10T05:02:53.749+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:02:54.144 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:53 smithi164 conmon[40892]: debug 2021-02-10T05:02:53.749+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:02:54.144 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:53 smithi164 conmon[40892]: debug 2021-02-10T05:02:53.750+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:02:54.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:54 smithi164 conmon[37263]: debug 2021-02-10T05:02:54.167+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:02:54.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:54 smithi164 conmon[37263]: debug 2021-02-10T05:02:54.167+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:23.564321+0000 front 2021-02-10T05:02:23.564413+0000 (oldest deadline 2021-02-10T05:02:47.664447+0000) 2021-02-10T05:02:54.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:54 smithi164 conmon[37263]: debug 2021-02-10T05:02:54.167+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:27.665225+0000 front 2021-02-10T05:02:27.665010+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:02:54.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:54 smithi164 conmon[37263]: debug 2021-02-10T05:02:54.167+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:27.665562+0000 front 2021-02-10T05:02:27.665151+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:02:54.644 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:54 smithi164 conmon[44544]: debug 2021-02-10T05:02:54.233+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:02:54.644 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:54 smithi164 conmon[44544]: debug 2021-02-10T05:02:54.233+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:02:54.645 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:54 smithi164 conmon[44544]: debug 2021-02-10T05:02:54.233+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:02:54.645 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:54 smithi164 conmon[44544]: debug 2021-02-10T05:02:54.233+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:02:55.127 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:54 smithi164 conmon[40892]: debug 2021-02-10T05:02:54.789+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:02:55.127 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:54 smithi164 conmon[40892]: debug 2021-02-10T05:02:54.789+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:02:55.127 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:54 smithi164 conmon[40892]: debug 2021-02-10T05:02:54.789+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:02:55.127 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:54 smithi164 conmon[40892]: debug 2021-02-10T05:02:54.789+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:02:55.127 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:54 smithi164 conmon[40892]: debug 2021-02-10T05:02:54.789+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:02:55.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:55 smithi164 conmon[37263]: debug 2021-02-10T05:02:55.125+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:02:55.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:55 smithi164 conmon[37263]: debug 2021-02-10T05:02:55.125+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:23.564321+0000 front 2021-02-10T05:02:23.564413+0000 (oldest deadline 2021-02-10T05:02:47.664447+0000) 2021-02-10T05:02:55.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:55 smithi164 conmon[37263]: debug 2021-02-10T05:02:55.125+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:27.665225+0000 front 2021-02-10T05:02:27.665010+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:02:55.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:55 smithi164 conmon[37263]: debug 2021-02-10T05:02:55.125+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:27.665562+0000 front 2021-02-10T05:02:27.665151+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:02:55.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:55 smithi164 conmon[44544]: debug 2021-02-10T05:02:55.229+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:02:55.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:55 smithi164 conmon[44544]: debug 2021-02-10T05:02:55.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:55 smithi164 conmon[44544]: 2021-02-10T05:02:55.229+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:02:55.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:55 smithi164 conmon[44544]: debug 2021-02-10T05:02:55.229+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:02:55.396 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:55 smithi164 conmon[44544]: debug 2021-02-10T05:02:55.229+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:02:55.396 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:55 smithi164 conmon[44544]: debug 2021-02-10T05:02:55.229+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:02:56.096 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:55 smithi164 conmon[40892]: debug 2021-02-10T05:02:55.829+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:02:56.096 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:55 smithi164 conmon[40892]: debug 2021-02-10T05:02:55.829+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:02:56.096 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:55 smithi164 conmon[40892]: debug 2021-02-10T05:02:55.829+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:02:56.096 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:55 smithi164 conmon[40892]: debug 2021-02-10T05:02:55.829+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:02:56.096 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:55 smithi164 conmon[40892]: debug 2021-02-10T05:02:55.829+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:02:56.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:56 smithi164 conmon[37263]: debug 2021-02-10T05:02:56.094+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:02:56.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:56 smithi164 conmon[37263]: debug 2021-02-10T05:02:56.094+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:23.564321+0000 front 2021-02-10T05:02:23.564413+0000 (oldest deadline 2021-02-10T05:02:47.664447+0000) 2021-02-10T05:02:56.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:56 smithi164 conmon[37263]: debug 2021-02-10T05:02:56.094+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:27.665225+0000 front 2021-02-10T05:02:27.665010+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:02:56.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:56 smithi164 conmon[37263]: debug 2021-02-10T05:02:56.094+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:27.665562+0000 front 2021-02-10T05:02:27.665151+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:02:56.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:56 smithi164 conmon[37263]: debug 2021-02-10T05:02:56.094+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.766003+0000 front 2021-02-10T05:02:31.766025+0000 (oldest deadline 2021-02-10T05:02:55.865741+0000) 2021-02-10T05:02:56.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:56 smithi164 conmon[44544]: debug 2021-02-10T05:02:56.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:56 smithi164 conmon[44544]: 2021-02-10T05:02:56.182+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:02:56.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:56 smithi164 conmon[44544]: 2021-02-10T05:02:56.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:56 smithi164 conmon[44544]: debug 2021-02-10T05:02:56.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:56 smithi164 conmon[44544]: 2021-02-10T05:02:56.182+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:02:56.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:56 smithi164 conmon[44544]: 2021-02-10T05:02:56.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:56 smithi164 conmon[44544]: debug 2021-02-10T05:02:56.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:56 smithi164 conmon[44544]: 2021-02-10T05:02:56.182+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:02:56.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:56 smithi164 conmon[44544]: debug 2021-02-10T05:02:56.182+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:02:56.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:56 smithi164 conmon[44544]: debug 2021-02-10T05:02:56.182+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:02:57.082 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:56 smithi164 conmon[40892]: debug 2021-02-10T05:02:56.821+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:02:57.082 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:56 smithi164 conmon[40892]: debug 2021-02-10T05:02:56.821+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:02:57.082 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:56 smithi164 conmon[40892]: debug 2021-02-10T05:02:56.821+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:02:57.082 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:56 smithi164 conmon[40892]: debug 2021-02-10T05:02:56.821+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:02:57.082 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:56 smithi164 conmon[40892]: debug 2021-02-10T05:02:56.821+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:02:57.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:57 smithi164 conmon[37263]: debug 2021-02-10T05:02:57.079+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:02:57.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:57 smithi164 conmon[37263]: debug 2021-02-10T05:02:57.079+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:23.564321+0000 front 2021-02-10T05:02:23.564413+0000 (oldest deadline 2021-02-10T05:02:47.664447+0000) 2021-02-10T05:02:57.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:57 smithi164 conmon[37263]: debug 2021-02-10T05:02:57.079+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:27.665225+0000 front 2021-02-10T05:02:27.665010+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:02:57.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:57 smithi164 conmon[37263]: debug 2021-02-10T05:02:57.079+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:27.665562+0000 front 2021-02-10T05:02:27.665151+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:02:57.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:57 smithi164 conmon[37263]: debug 2021-02-10T05:02:57.079+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.766003+0000 front 2021-02-10T05:02:31.766025+0000 (oldest deadline 2021-02-10T05:02:55.865741+0000) 2021-02-10T05:02:57.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:57 smithi164 conmon[44544]: debug 2021-02-10T05:02:57.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:57 smithi164 conmon[44544]: 2021-02-10T05:02:57.155+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:02:57.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:57 smithi164 conmon[44544]: debug 2021-02-10T05:02:57.155+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:02:57.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:57 smithi164 conmon[44544]: debug 2021-02-10T05:02:57.155+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:02:57.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:57 smithi164 conmon[44544]: debug 2021-02-10T05:02:57.155+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:02:57.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:57 smithi164 conmon[44544]: debug 2021-02-10T05:02:57.155+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:02:58.097 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:57 smithi164 conmon[40892]: debug 2021-02-10T05:02:57.773+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:02:58.098 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:57 smithi164 conmon[40892]: debug 2021-02-10T05:02:57.773+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:02:58.098 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:57 smithi164 conmon[40892]: debug 2021-02-10T05:02:57.773+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:02:58.098 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:57 smithi164 conmon[40892]: debug 2021-02-10T05:02:57.773+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:02:58.098 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:57 smithi164 conmon[40892]: debug 2021-02-10T05:02:57.773+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:02:58.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:58 smithi164 conmon[37263]: debug 2021-02-10T05:02:58.096+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:02:58.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:58 smithi164 conmon[37263]: debug 2021-02-10T05:02:58.096+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:23.564321+0000 front 2021-02-10T05:02:23.564413+0000 (oldest deadline 2021-02-10T05:02:47.664447+0000) 2021-02-10T05:02:58.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:58 smithi164 conmon[37263]: debug 2021-02-10T05:02:58.096+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:27.665225+0000 front 2021-02-10T05:02:27.665010+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:02:58.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:58 smithi164 conmon[37263]: debug 2021-02-10T05:02:58.096+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:27.665562+0000 front 2021-02-10T05:02:27.665151+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:02:58.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:58 smithi164 conmon[37263]: debug 2021-02-10T05:02:58.096+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.766003+0000 front 2021-02-10T05:02:31.766025+0000 (oldest deadline 2021-02-10T05:02:55.865741+0000) 2021-02-10T05:02:58.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:58 smithi164 conmon[44544]: debug 2021-02-10T05:02:58.203+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:02:58.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:58 smithi164 conmon[44544]: debug 2021-02-10T05:02:58.203+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:02:58.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:58 smithi164 conmon[44544]: debug 2021-02-10T05:02:58.203+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:02:58.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:58 smithi164 conmon[44544]: debug 2021-02-10T05:02:58.204+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:02:58.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:58 smithi164 conmon[44544]: debug 2021-02-10T05:02:58.204+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:02:59.113 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:58 smithi164 conmon[40892]: debug 2021-02-10T05:02:58.747+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:02:59.113 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:58 smithi164 conmon[40892]: debug 2021-02-10T05:02:58.747+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:02:59.113 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:58 smithi164 conmon[40892]: debug 2021-02-10T05:02:58.747+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:02:59.113 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:58 smithi164 conmon[40892]: debug 2021-02-10T05:02:58.747+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:02:59.113 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:58 smithi164 conmon[40892]: debug 2021-02-10T05:02:58.748+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:02:59.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:59 smithi164 conmon[44544]: debug 2021-02-10T05:02:59.158+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:02:59.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:59 smithi164 conmon[44544]: debug 2021-02-10T05:02:59.158+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:02:59.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:59 smithi164 conmon[44544]: debug 2021-02-10T05:02:59.158+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:02:59.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:59 smithi164 conmon[44544]: debug 2021-02-10T05:02:59.158+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:02:59.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:02:59 smithi164 conmon[44544]: debug 2021-02-10T05:02:59.158+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:02:59.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:59 smithi164 conmon[37263]: debug 2021-02-10T05:02:59.112+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:02:59.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:59 smithi164 conmon[37263]: debug 2021-02-10T05:02:59.112+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:23.564321+0000 front 2021-02-10T05:02:23.564413+0000 (oldest deadline 2021-02-10T05:02:47.664447+0000) 2021-02-10T05:02:59.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:59 smithi164 conmon[37263]: debug 2021-02-10T05:02:59.112+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:27.665225+0000 front 2021-02-10T05:02:27.665010+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:02:59.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:59 smithi164 conmon[37263]: debug 2021-02-10T05:02:59.112+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:27.665562+0000 front 2021-02-10T05:02:27.665151+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:02:59.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:02:59 smithi164 conmon[37263]: debug 2021-02-10T05:02:59.112+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.766003+0000 front 2021-02-10T05:02:31.766025+0000 (oldest deadline 2021-02-10T05:02:55.865741+0000) 2021-02-10T05:03:00.126 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:59 smithi164 conmon[40892]: debug 2021-02-10T05:02:59.746+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:03:00.126 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:59 smithi164 conmon[40892]: debug 2021-02-10T05:02:59.746+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:03:00.126 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:59 smithi164 conmon[40892]: debug 2021-02-10T05:02:59.746+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:00.126 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:59 smithi164 conmon[40892]: debug 2021-02-10T05:02:59.746+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:00.126 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:02:59 smithi164 conmon[40892]: debug 2021-02-10T05:02:59.746+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:03:00.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:00 smithi164 conmon[37263]: debug 2021-02-10T05:03:00.125+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:03:00.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:00 smithi164 conmon[37263]: debug 2021-02-10T05:03:00.125+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:23.564321+0000 front 2021-02-10T05:02:23.564413+0000 (oldest deadline 2021-02-10T05:02:47.664447+0000) 2021-02-10T05:03:00.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:00 smithi164 conmon[37263]: debug 2021-02-10T05:03:00.125+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:27.665225+0000 front 2021-02-10T05:02:27.665010+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:00.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:00 smithi164 conmon[37263]: debug 2021-02-10T05:03:00.125+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:27.665562+0000 front 2021-02-10T05:02:27.665151+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:00.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:00 smithi164 conmon[37263]: debug 2021-02-10T05:03:00.125+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.766003+0000 front 2021-02-10T05:02:31.766025+0000 (oldest deadline 2021-02-10T05:02:55.865741+0000) 2021-02-10T05:03:00.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:00 smithi164 conmon[44544]: debug 2021-02-10T05:03:00.176+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:03:00.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:00 smithi164 conmon[44544]: debug 2021-02-10T05:03:00.176+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:00.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:00 smithi164 conmon[44544]: debug 2021-02-10T05:03:00.176+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:00.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:00 smithi164 conmon[44544]: debug 2021-02-10T05:03:00.176+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:03:00.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:00 smithi164 conmon[44544]: debug 2021-02-10T05:03:00.176+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:03:01.134 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:00 smithi164 conmon[40892]: debug 2021-02-10T05:03:00.734+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:03:01.134 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:00 smithi164 conmon[40892]: debug 2021-02-10T05:03:00.734+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:03:01.134 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:00 smithi164 conmon[40892]: debug 2021-02-10T05:03:00.734+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:01.134 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:00 smithi164 conmon[40892]: debug 2021-02-10T05:03:00.734+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:01.134 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:00 smithi164 conmon[40892]: debug 2021-02-10T05:03:00.734+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:03:01.134 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:01 smithi164 conmon[44544]: debug 2021-02-10T05:03:01.135 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:01 smithi164 conmon[44544]: 2021-02-10T05:03:01.133+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:03:01.135 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:01 smithi164 conmon[44544]: debug 2021-02-10T05:03:01.133+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:01.135 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:01 smithi164 conmon[44544]: debug 2021-02-10T05:03:01.133+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:01.135 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:01 smithi164 conmon[44544]: debug 2021-02-10T05:03:01.133+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:03:01.135 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:01 smithi164 conmon[44544]: debug 2021-02-10T05:03:01.133+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:03:01.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:01 smithi164 conmon[37263]: debug 2021-02-10T05:03:01.142+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:03:01.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:01 smithi164 conmon[37263]: debug 2021-02-10T05:03:01.142+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:23.564321+0000 front 2021-02-10T05:02:23.564413+0000 (oldest deadline 2021-02-10T05:02:47.664447+0000) 2021-02-10T05:03:01.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:01 smithi164 conmon[37263]: debug 2021-02-10T05:03:01.142+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:27.665225+0000 front 2021-02-10T05:02:27.665010+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:01.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:01 smithi164 conmon[37263]: debug 2021-02-10T05:03:01.142+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:27.665562+0000 front 2021-02-10T05:02:27.665151+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:01.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:01 smithi164 conmon[37263]: debug 2021-02-10T05:03:01.142+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.766003+0000 front 2021-02-10T05:02:31.766025+0000 (oldest deadline 2021-02-10T05:02:55.865741+0000) 2021-02-10T05:03:02.106 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:01 smithi164 conmon[40892]: debug 2021-02-10T05:03:01.757+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:03:02.106 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:01 smithi164 conmon[40892]: debug 2021-02-10T05:03:01.757+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:03:02.106 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:01 smithi164 conmon[40892]: debug 2021-02-10T05:03:01.757+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:02.106 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:01 smithi164 conmon[40892]: debug 2021-02-10T05:03:01.757+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:02.106 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:01 smithi164 conmon[40892]: debug 2021-02-10T05:03:01.757+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:03:02.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:02 smithi164 conmon[37263]: debug 2021-02-10T05:03:02.149+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:03:02.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:02 smithi164 conmon[37263]: debug 2021-02-10T05:03:02.149+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:23.564321+0000 front 2021-02-10T05:02:23.564413+0000 (oldest deadline 2021-02-10T05:02:47.664447+0000) 2021-02-10T05:03:02.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:02 smithi164 conmon[37263]: debug 2021-02-10T05:03:02.149+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:27.665225+0000 front 2021-02-10T05:02:27.665010+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:02.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:02 smithi164 conmon[37263]: debug 2021-02-10T05:03:02.149+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:27.665562+0000 front 2021-02-10T05:02:27.665151+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:02.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:02 smithi164 conmon[37263]: debug 2021-02-10T05:03:02.149+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.766003+0000 front 2021-02-10T05:02:31.766025+0000 (oldest deadline 2021-02-10T05:02:55.865741+0000) 2021-02-10T05:03:02.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:02 smithi164 conmon[44544]: debug 2021-02-10T05:03:02.105+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:03:02.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:02 smithi164 conmon[44544]: debug 2021-02-10T05:03:02.105+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:02.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:02 smithi164 conmon[44544]: debug 2021-02-10T05:03:02.105+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:02.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:02 smithi164 conmon[44544]: debug 2021-02-10T05:03:02.105+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:03:02.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:02 smithi164 conmon[44544]: debug 2021-02-10T05:03:02.105+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:03:03.126 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:02 smithi164 conmon[40892]: debug 2021-02-10T05:03:02.798+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:03:03.126 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:02 smithi164 conmon[40892]: debug 2021-02-10T05:03:02.798+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:03:03.126 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:02 smithi164 conmon[40892]: debug 2021-02-10T05:03:02.798+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:03.126 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:02 smithi164 conmon[40892]: debug 2021-02-10T05:03:02.798+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:03.126 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:02 smithi164 conmon[40892]: debug 2021-02-10T05:03:02.798+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:03:03.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:03 smithi164 conmon[44544]: debug 2021-02-10T05:03:03.125+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:03:03.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:03 smithi164 conmon[44544]: debug 2021-02-10T05:03:03.125+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:03.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:03 smithi164 conmon[44544]: debug 2021-02-10T05:03:03.125+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:03.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:03 smithi164 conmon[44544]: debug 2021-02-10T05:03:03.125+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:03:03.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:03 smithi164 conmon[44544]: debug 2021-02-10T05:03:03.125+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:03:03.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:03 smithi164 conmon[37263]: debug 2021-02-10T05:03:03.174+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:03:03.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:03 smithi164 conmon[37263]: debug 2021-02-10T05:03:03.174+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:23.564321+0000 front 2021-02-10T05:02:23.564413+0000 (oldest deadline 2021-02-10T05:02:47.664447+0000) 2021-02-10T05:03:03.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:03 smithi164 conmon[37263]: debug 2021-02-10T05:03:03.174+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:27.665225+0000 front 2021-02-10T05:02:27.665010+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:03.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:03 smithi164 conmon[37263]: debug 2021-02-10T05:03:03.174+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:27.665562+0000 front 2021-02-10T05:02:27.665151+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:03.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:03 smithi164 conmon[37263]: debug 2021-02-10T05:03:03.174+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.766003+0000 front 2021-02-10T05:02:31.766025+0000 (oldest deadline 2021-02-10T05:02:55.865741+0000) 2021-02-10T05:03:03.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:03 smithi164 conmon[37263]: debug 2021-02-10T05:03:03.174+0000 7f1c83a60700 -1 osd.5 1429 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24373.0:4691 5.15 5:a93a5511:::notify.2:head [watch ping cookie 94187755105536 gen 4] snapc 0=[] ondisk+write+known_if_redirected e1429) 2021-02-10T05:03:04.206 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:03 smithi164 conmon[40892]: debug 2021-02-10T05:03:03.929+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:03:04.206 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:03 smithi164 conmon[40892]: debug 2021-02-10T05:03:03.929+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:03:04.206 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:03 smithi164 conmon[40892]: debug 2021-02-10T05:03:03.929+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:04.206 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:03 smithi164 conmon[40892]: debug 2021-02-10T05:03:03.929+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:04.206 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:03 smithi164 conmon[40892]: debug 2021-02-10T05:03:03.929+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:03:04.207 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:04 smithi164 conmon[44544]: debug 2021-02-10T05:03:04.134+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:03:04.207 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:04 smithi164 conmon[44544]: debug 2021-02-10T05:03:04.134+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:04.207 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:04 smithi164 conmon[44544]: debug 2021-02-10T05:03:04.134+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:04.207 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:04 smithi164 conmon[44544]: debug 2021-02-10T05:03:04.134+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:03:04.207 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:04 smithi164 conmon[44544]: debug 2021-02-10T05:03:04.134+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:03:04.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:04 smithi164 conmon[37263]: debug 2021-02-10T05:03:04.205+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:03:04.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:04 smithi164 conmon[37263]: debug 2021-02-10T05:03:04.205+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:23.564321+0000 front 2021-02-10T05:02:23.564413+0000 (oldest deadline 2021-02-10T05:02:47.664447+0000) 2021-02-10T05:03:04.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:04 smithi164 conmon[37263]: debug 2021-02-10T05:03:04.205+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:27.665225+0000 front 2021-02-10T05:02:27.665010+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:04.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:04 smithi164 conmon[37263]: debug 2021-02-10T05:03:04.205+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:27.665562+0000 front 2021-02-10T05:02:27.665151+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:04.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:04 smithi164 conmon[37263]: debug 2021-02-10T05:03:04.205+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.766003+0000 front 2021-02-10T05:02:31.766025+0000 (oldest deadline 2021-02-10T05:02:55.865741+0000) 2021-02-10T05:03:04.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:04 smithi164 conmon[37263]: debug 2021-02-10T05:03:04.205+0000 7f1c83a60700 -1 osd.5 1429 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24373.0:4691 5.15 5:a93a5511:::notify.2:head [watch ping cookie 94187755105536 gen 4] snapc 0=[] ondisk+write+known_if_redirected e1429) 2021-02-10T05:03:05.144 INFO:journalctl@ceph.osd.4.smithi164.stdout:Feb 10 05:03:05 smithi164 podman[97947]: 3b782373f0f33ba28cd4572c18ce22973e7fe32d01f18e01ed897eaeaae6cdc0 2021-02-10T05:03:05.144 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:04 smithi164 conmon[40892]: debug 2021-02-10T05:03:04.889+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:03:05.144 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:04 smithi164 conmon[40892]: debug 2021-02-10T05:03:04.889+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:03:05.144 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:04 smithi164 conmon[40892]: debug 2021-02-10T05:03:04.889+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:05.144 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:04 smithi164 conmon[40892]: debug 2021-02-10T05:03:04.889+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:05.145 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:04 smithi164 conmon[40892]: debug 2021-02-10T05:03:04.889+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:03:05.145 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:05 smithi164 conmon[44544]: debug 2021-02-10T05:03:05.090+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:03:05.145 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:05 smithi164 conmon[44544]: debug 2021-02-10T05:03:05.090+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:05.145 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:05 smithi164 conmon[44544]: debug 2021-02-10T05:03:05.090+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:05.145 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:05 smithi164 conmon[44544]: debug 2021-02-10T05:03:05.145 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:05 smithi164 conmon[44544]: 2021-02-10T05:03:05.090+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:03:05.145 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:05 smithi164 conmon[44544]: debug 2021-02-10T05:03:05.090+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:03:05.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:05 smithi164 conmon[37263]: debug 2021-02-10T05:03:05.207+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:03:05.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:05 smithi164 conmon[37263]: debug 2021-02-10T05:03:05.207+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:23.564321+0000 front 2021-02-10T05:02:23.564413+0000 (oldest deadline 2021-02-10T05:02:47.664447+0000) 2021-02-10T05:03:05.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:05 smithi164 conmon[37263]: debug 2021-02-10T05:03:05.207+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:27.665225+0000 front 2021-02-10T05:02:27.665010+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:05.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:05 smithi164 conmon[37263]: debug 2021-02-10T05:03:05.207+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:27.665562+0000 front 2021-02-10T05:02:27.665151+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:05.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:05 smithi164 conmon[37263]: debug 2021-02-10T05:03:05.207+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.766003+0000 front 2021-02-10T05:02:31.766025+0000 (oldest deadline 2021-02-10T05:02:55.865741+0000) 2021-02-10T05:03:05.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:05 smithi164 conmon[37263]: debug 2021-02-10T05:03:05.207+0000 7f1c83a60700 -1 osd.5 1429 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24373.0:4691 5.15 5:a93a5511:::notify.2:head [watch ping cookie 94187755105536 gen 4] snapc 0=[] ondisk+write+known_if_redirected e1429) 2021-02-10T05:03:06.208 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:06 smithi164 conmon[37263]: debug 2021-02-10T05:03:06.207+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:03:06.208 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:06 smithi164 conmon[37263]: debug 2021-02-10T05:03:06.207+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:23.564321+0000 front 2021-02-10T05:02:23.564413+0000 (oldest deadline 2021-02-10T05:02:47.664447+0000) 2021-02-10T05:03:06.208 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:05 smithi164 conmon[40892]: debug 2021-02-10T05:03:05.936+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:03:06.208 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:05 smithi164 conmon[40892]: debug 2021-02-10T05:03:05.936+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:03:06.209 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:05 smithi164 conmon[40892]: debug 2021-02-10T05:03:05.936+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:06.209 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:05 smithi164 conmon[40892]: debug 2021-02-10T05:03:05.936+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:06.209 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:05 smithi164 conmon[40892]: debug 2021-02-10T05:03:05.936+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:03:06.209 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:06 smithi164 conmon[44544]: debug 2021-02-10T05:03:06.209 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:06 smithi164 conmon[44544]: 2021-02-10T05:03:06.127+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:03:06.209 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:06 smithi164 conmon[44544]: 2021-02-10T05:03:06.209 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:06 smithi164 conmon[44544]: debug 2021-02-10T05:03:06.127+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:06.209 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:06 smithi164 conmon[44544]: debug 2021-02-10T05:03:06.127+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:06.209 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:06 smithi164 conmon[44544]: debug 2021-02-10T05:03:06.127+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:03:06.210 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:06 smithi164 conmon[44544]: debug 2021-02-10T05:03:06.127+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:03:06.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:06 smithi164 conmon[37263]: debug 2021-02-10T05:03:06.207+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:27.665225+0000 front 2021-02-10T05:02:27.665010+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:06.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:06 smithi164 conmon[37263]: debug 2021-02-10T05:03:06.207+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:27.665562+0000 front 2021-02-10T05:02:27.665151+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:06.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:06 smithi164 conmon[37263]: debug 2021-02-10T05:03:06.207+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.766003+0000 front 2021-02-10T05:02:31.766025+0000 (oldest deadline 2021-02-10T05:02:55.865741+0000) 2021-02-10T05:03:06.644 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:06 smithi164 conmon[37263]: debug 2021-02-10T05:03:06.207+0000 7f1c83a60700 -1 osd.5 1429 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24373.0:4691 5.15 5:a93a5511:::notify.2:head [watch ping cookie 94187755105536 gen 4] snapc 0=[] ondisk+write+known_if_redirected e1429) 2021-02-10T05:03:07.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:07 smithi164 conmon[37263]: debug 2021-02-10T05:03:07.160+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:03:07.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:07 smithi164 conmon[37263]: debug 2021-02-10T05:03:07.160+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:23.564321+0000 front 2021-02-10T05:02:23.564413+0000 (oldest deadline 2021-02-10T05:02:47.664447+0000) 2021-02-10T05:03:07.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:07 smithi164 conmon[37263]: debug 2021-02-10T05:03:07.160+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:27.665225+0000 front 2021-02-10T05:02:27.665010+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:07.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:07 smithi164 conmon[37263]: debug 2021-02-10T05:03:07.160+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:27.665562+0000 front 2021-02-10T05:02:27.665151+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:07.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:07 smithi164 conmon[37263]: debug 2021-02-10T05:03:07.160+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.766003+0000 front 2021-02-10T05:02:31.766025+0000 (oldest deadline 2021-02-10T05:02:55.865741+0000) 2021-02-10T05:03:07.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:07 smithi164 conmon[37263]: debug 2021-02-10T05:03:07.160+0000 7f1c83a60700 -1 osd.5 1429 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24373.0:4691 5.15 5:a93a5511:::notify.2:head [watch ping cookie 94187755105536 gen 4] snapc 0=[] ondisk+write+known_if_redirected e1429) 2021-02-10T05:03:07.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:06 smithi164 conmon[40892]: debug 2021-02-10T05:03:06.949+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:03:07.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:06 smithi164 conmon[40892]: debug 2021-02-10T05:03:06.949+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:03:07.396 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:06 smithi164 conmon[40892]: debug 2021-02-10T05:03:06.949+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:07.396 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:06 smithi164 conmon[40892]: debug 2021-02-10T05:03:06.949+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:07.396 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:06 smithi164 conmon[40892]: debug 2021-02-10T05:03:06.949+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:03:07.396 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:07 smithi164 conmon[44544]: debug 2021-02-10T05:03:07.096+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:03:07.396 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:07 smithi164 conmon[44544]: debug 2021-02-10T05:03:07.096+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:07.396 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:07 smithi164 conmon[44544]: debug 2021-02-10T05:03:07.096+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:07.396 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:07 smithi164 conmon[44544]: debug 2021-02-10T05:03:07.096+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:03:07.396 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:07 smithi164 conmon[44544]: debug 2021-02-10T05:03:07.096+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:03:08.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:08 smithi164 conmon[37263]: debug 2021-02-10T05:03:08.192+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:03:08.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:08 smithi164 conmon[37263]: debug 2021-02-10T05:03:08.192+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:23.564321+0000 front 2021-02-10T05:02:23.564413+0000 (oldest deadline 2021-02-10T05:02:47.664447+0000) 2021-02-10T05:03:08.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:08 smithi164 conmon[37263]: debug 2021-02-10T05:03:08.192+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:27.665225+0000 front 2021-02-10T05:02:27.665010+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:08.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:08 smithi164 conmon[37263]: debug 2021-02-10T05:03:08.192+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:27.665562+0000 front 2021-02-10T05:02:27.665151+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:08.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:08 smithi164 conmon[37263]: debug 2021-02-10T05:03:08.192+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.766003+0000 front 2021-02-10T05:02:31.766025+0000 (oldest deadline 2021-02-10T05:02:55.865741+0000) 2021-02-10T05:03:08.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:07 smithi164 conmon[40892]: debug 2021-02-10T05:03:07.968+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:03:08.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:07 smithi164 conmon[40892]: debug 2021-02-10T05:03:07.969+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:03:08.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:07 smithi164 conmon[40892]: debug 2021-02-10T05:03:07.969+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:08.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:07 smithi164 conmon[40892]: debug 2021-02-10T05:03:07.969+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:08.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:07 smithi164 conmon[40892]: debug 2021-02-10T05:03:07.969+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:03:08.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:08 smithi164 conmon[44544]: debug 2021-02-10T05:03:08.086+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:03:08.396 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:08 smithi164 conmon[44544]: debug 2021-02-10T05:03:08.086+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:08.396 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:08 smithi164 conmon[44544]: debug 2021-02-10T05:03:08.086+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:08.396 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:08 smithi164 conmon[44544]: debug 2021-02-10T05:03:08.086+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:03:08.396 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:08 smithi164 conmon[44544]: debug 2021-02-10T05:03:08.086+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:03:09.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:09 smithi164 conmon[37263]: debug 2021-02-10T05:03:09.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:09 smithi164 conmon[37263]: 2021-02-10T05:03:09.221+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:03:09.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:09 smithi164 conmon[37263]: debug 2021-02-10T05:03:09.221+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:23.564321+0000 front 2021-02-10T05:02:23.564413+0000 (oldest deadline 2021-02-10T05:02:47.664447+0000) 2021-02-10T05:03:09.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:09 smithi164 conmon[37263]: debug 2021-02-10T05:03:09.221+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:27.665225+0000 front 2021-02-10T05:02:27.665010+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:09.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:09 smithi164 conmon[37263]: debug 2021-02-10T05:03:09.221+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:27.665562+0000 front 2021-02-10T05:02:27.665151+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:09.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:09 smithi164 conmon[37263]: debug 2021-02-10T05:03:09.221+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.766003+0000 front 2021-02-10T05:02:31.766025+0000 (oldest deadline 2021-02-10T05:02:55.865741+0000) 2021-02-10T05:03:09.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:08 smithi164 conmon[40892]: debug 2021-02-10T05:03:09.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:08 smithi164 conmon[40892]: 2021-02-10T05:03:08.991+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:03:09.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:08 smithi164 conmon[40892]: 2021-02-10T05:03:09.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:08 smithi164 conmon[40892]: debug 2021-02-10T05:03:08.991+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:03:09.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:08 smithi164 conmon[40892]: debug 2021-02-10T05:03:08.991+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:09.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:08 smithi164 conmon[40892]: debug 2021-02-10T05:03:08.991+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:09.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:08 smithi164 conmon[40892]: debug 2021-02-10T05:03:08.991+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:03:09.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:09 smithi164 conmon[44544]: debug 2021-02-10T05:03:09.104+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:03:09.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:09 smithi164 conmon[44544]: debug 2021-02-10T05:03:09.104+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:09.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:09 smithi164 conmon[44544]: debug 2021-02-10T05:03:09.104+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:09.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:09 smithi164 conmon[44544]: debug 2021-02-10T05:03:09.104+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:03:09.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:09 smithi164 conmon[44544]: debug 2021-02-10T05:03:09.104+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:03:10.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:10 smithi164 conmon[37263]: debug 2021-02-10T05:03:10.206+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:03:10.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:10 smithi164 conmon[37263]: debug 2021-02-10T05:03:10.206+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:23.564321+0000 front 2021-02-10T05:02:23.564413+0000 (oldest deadline 2021-02-10T05:02:47.664447+0000) 2021-02-10T05:03:10.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:10 smithi164 conmon[37263]: debug 2021-02-10T05:03:10.206+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:27.665225+0000 front 2021-02-10T05:02:27.665010+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:10.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:10 smithi164 conmon[37263]: debug 2021-02-10T05:03:10.206+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:27.665562+0000 front 2021-02-10T05:02:27.665151+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:10.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:10 smithi164 conmon[37263]: debug 2021-02-10T05:03:10.206+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.766003+0000 front 2021-02-10T05:02:31.766025+0000 (oldest deadline 2021-02-10T05:02:55.865741+0000) 2021-02-10T05:03:10.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:10 smithi164 conmon[40892]: debug 2021-02-10T05:03:10.021+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:03:10.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:10 smithi164 conmon[40892]: debug 2021-02-10T05:03:10.021+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:03:10.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:10 smithi164 conmon[40892]: debug 2021-02-10T05:03:10.021+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:10.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:10 smithi164 conmon[40892]: debug 2021-02-10T05:03:10.021+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:10.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:10 smithi164 conmon[40892]: debug 2021-02-10T05:03:10.021+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:03:10.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:10 smithi164 conmon[44544]: debug 2021-02-10T05:03:10.144+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:03:10.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:10 smithi164 conmon[44544]: debug 2021-02-10T05:03:10.144+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:10.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:10 smithi164 conmon[44544]: debug 2021-02-10T05:03:10.144+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:10.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:10 smithi164 conmon[44544]: debug 2021-02-10T05:03:10.144+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:03:10.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:10 smithi164 conmon[44544]: debug 2021-02-10T05:03:10.144+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:03:11.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:11 smithi164 conmon[37263]: debug 2021-02-10T05:03:11.217+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:03:11.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:11 smithi164 conmon[37263]: debug 2021-02-10T05:03:11.217+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:23.564321+0000 front 2021-02-10T05:02:23.564413+0000 (oldest deadline 2021-02-10T05:02:47.664447+0000) 2021-02-10T05:03:11.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:11 smithi164 conmon[37263]: debug 2021-02-10T05:03:11.217+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:27.665225+0000 front 2021-02-10T05:02:27.665010+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:11.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:11 smithi164 conmon[37263]: debug 2021-02-10T05:03:11.217+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:27.665562+0000 front 2021-02-10T05:02:27.665151+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:11.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:11 smithi164 conmon[37263]: debug 2021-02-10T05:03:11.217+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.766003+0000 front 2021-02-10T05:02:31.766025+0000 (oldest deadline 2021-02-10T05:02:55.865741+0000) 2021-02-10T05:03:11.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:10 smithi164 conmon[40892]: debug 2021-02-10T05:03:10.996+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:03:11.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:10 smithi164 conmon[40892]: debug 2021-02-10T05:03:10.996+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:03:11.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:10 smithi164 conmon[40892]: debug 2021-02-10T05:03:10.996+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:11.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:10 smithi164 conmon[40892]: debug 2021-02-10T05:03:10.996+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:11.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:10 smithi164 conmon[40892]: debug 2021-02-10T05:03:10.996+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:03:11.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:11 smithi164 conmon[44544]: debug 2021-02-10T05:03:11.181+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:03:11.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:11 smithi164 conmon[44544]: debug 2021-02-10T05:03:11.181+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:11.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:11 smithi164 conmon[44544]: debug 2021-02-10T05:03:11.181+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:11.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:11 smithi164 conmon[44544]: debug 2021-02-10T05:03:11.181+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:03:11.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:11 smithi164 conmon[44544]: debug 2021-02-10T05:03:11.181+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:03:12.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:12 smithi164 conmon[37263]: debug 2021-02-10T05:03:12.186+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:03:12.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:12 smithi164 conmon[37263]: debug 2021-02-10T05:03:12.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:12 smithi164 conmon[37263]: 2021-02-10T05:03:12.186+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:23.564321+0000 front 2021-02-10T05:02:23.564413+0000 (oldest deadline 2021-02-10T05:02:47.664447+0000) 2021-02-10T05:03:12.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:12 smithi164 conmon[37263]: debug 2021-02-10T05:03:12.186+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:27.665225+0000 front 2021-02-10T05:02:27.665010+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:12.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:12 smithi164 conmon[37263]: debug 2021-02-10T05:03:12.186+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:27.665562+0000 front 2021-02-10T05:02:27.665151+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:12.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:12 smithi164 conmon[37263]: debug 2021-02-10T05:03:12.186+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.766003+0000 front 2021-02-10T05:02:31.766025+0000 (oldest deadline 2021-02-10T05:02:55.865741+0000) 2021-02-10T05:03:12.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:12 smithi164 conmon[40892]: debug 2021-02-10T05:03:12.009+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:03:12.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:12 smithi164 conmon[40892]: debug 2021-02-10T05:03:12.009+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:03:12.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:12 smithi164 conmon[40892]: debug 2021-02-10T05:03:12.009+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:12.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:12 smithi164 conmon[40892]: debug 2021-02-10T05:03:12.009+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:12.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:12 smithi164 conmon[40892]: debug 2021-02-10T05:03:12.009+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:03:12.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:12 smithi164 conmon[44544]: debug 2021-02-10T05:03:12.193+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:03:12.396 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:12 smithi164 conmon[44544]: debug 2021-02-10T05:03:12.193+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:12.396 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:12 smithi164 conmon[44544]: 2021-02-10T05:03:12.396 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:12 smithi164 conmon[44544]: debug 2021-02-10T05:03:12.193+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:12.396 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:12 smithi164 conmon[44544]: debug 2021-02-10T05:03:12.193+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:03:12.396 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:12 smithi164 conmon[44544]: debug 2021-02-10T05:03:12.193+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:03:13.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:13 smithi164 conmon[37263]: debug 2021-02-10T05:03:13.212+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:03:13.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:13 smithi164 conmon[37263]: debug 2021-02-10T05:03:13.212+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:23.564321+0000 front 2021-02-10T05:02:23.564413+0000 (oldest deadline 2021-02-10T05:02:47.664447+0000) 2021-02-10T05:03:13.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:13 smithi164 conmon[37263]: debug 2021-02-10T05:03:13.212+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:27.665225+0000 front 2021-02-10T05:02:27.665010+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:13.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:13 smithi164 conmon[37263]: debug 2021-02-10T05:03:13.212+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:27.665562+0000 front 2021-02-10T05:02:27.665151+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:13.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:13 smithi164 conmon[37263]: debug 2021-02-10T05:03:13.212+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.766003+0000 front 2021-02-10T05:02:31.766025+0000 (oldest deadline 2021-02-10T05:02:55.865741+0000) 2021-02-10T05:03:13.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:13 smithi164 conmon[40892]: debug 2021-02-10T05:03:13.028+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:03:13.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:13 smithi164 conmon[40892]: debug 2021-02-10T05:03:13.028+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:03:13.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:13 smithi164 conmon[40892]: debug 2021-02-10T05:03:13.028+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:13.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:13 smithi164 conmon[40892]: debug 2021-02-10T05:03:13.028+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:13.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:13 smithi164 conmon[40892]: debug 2021-02-10T05:03:13.028+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:03:13.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:13 smithi164 conmon[44544]: debug 2021-02-10T05:03:13.169+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:03:13.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:13 smithi164 conmon[44544]: debug 2021-02-10T05:03:13.169+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:13.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:13 smithi164 conmon[44544]: debug 2021-02-10T05:03:13.169+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:13.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:13 smithi164 conmon[44544]: debug 2021-02-10T05:03:13.169+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:03:13.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:13 smithi164 conmon[44544]: debug 2021-02-10T05:03:13.169+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:03:13.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:13 smithi164 conmon[44544]: debug 2021-02-10T05:03:13.169+0000 7f64c490c700 -1 osd.7 1429 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24373.0:4709 5.9 5:93e5b521:::notify.7:head [watch ping cookie 94187770081792 gen 4] snapc 0=[] ondisk+write+known_if_redirected e1429) 2021-02-10T05:03:14.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:14 smithi164 conmon[37263]: debug 2021-02-10T05:03:14.214+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:03:14.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:14 smithi164 conmon[37263]: debug 2021-02-10T05:03:14.214+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:23.564321+0000 front 2021-02-10T05:02:23.564413+0000 (oldest deadline 2021-02-10T05:02:47.664447+0000) 2021-02-10T05:03:14.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:14 smithi164 conmon[37263]: debug 2021-02-10T05:03:14.214+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:27.665225+0000 front 2021-02-10T05:02:27.665010+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:14.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:14 smithi164 conmon[37263]: debug 2021-02-10T05:03:14.214+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:27.665562+0000 front 2021-02-10T05:02:27.665151+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:14.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:14 smithi164 conmon[37263]: debug 2021-02-10T05:03:14.214+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.766003+0000 front 2021-02-10T05:02:31.766025+0000 (oldest deadline 2021-02-10T05:02:55.865741+0000) 2021-02-10T05:03:14.396 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:14 smithi164 conmon[40892]: debug 2021-02-10T05:03:14.075+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:03:14.396 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:14 smithi164 conmon[40892]: debug 2021-02-10T05:03:14.075+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:03:14.396 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:14 smithi164 conmon[40892]: debug 2021-02-10T05:03:14.075+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:14.396 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:14 smithi164 conmon[40892]: debug 2021-02-10T05:03:14.075+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:14.396 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:14 smithi164 conmon[40892]: debug 2021-02-10T05:03:14.075+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:03:14.397 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:14 smithi164 conmon[44544]: debug 2021-02-10T05:03:14.125+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:03:14.397 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:14 smithi164 conmon[44544]: debug 2021-02-10T05:03:14.125+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:14.397 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:14 smithi164 conmon[44544]: debug 2021-02-10T05:03:14.125+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:14.397 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:14 smithi164 conmon[44544]: debug 2021-02-10T05:03:14.125+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:03:14.397 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:14 smithi164 conmon[44544]: debug 2021-02-10T05:03:14.125+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:03:14.397 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:14 smithi164 conmon[44544]: debug 2021-02-10T05:03:14.125+0000 7f64c490c700 -1 osd.7 1429 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24373.0:4709 5.9 5:93e5b521:::notify.7:head [watch ping cookie 94187770081792 gen 4] snapc 0=[] ondisk+write+known_if_redirected e1429) 2021-02-10T05:03:15.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:15 smithi164 conmon[37263]: debug 2021-02-10T05:03:15.227+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:03:15.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:15 smithi164 conmon[37263]: debug 2021-02-10T05:03:15.227+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:23.564321+0000 front 2021-02-10T05:02:23.564413+0000 (oldest deadline 2021-02-10T05:02:47.664447+0000) 2021-02-10T05:03:15.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:15 smithi164 conmon[37263]: debug 2021-02-10T05:03:15.227+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:27.665225+0000 front 2021-02-10T05:02:27.665010+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:15.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:15 smithi164 conmon[37263]: debug 2021-02-10T05:03:15.227+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:27.665562+0000 front 2021-02-10T05:02:27.665151+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:15.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:15 smithi164 conmon[37263]: debug 2021-02-10T05:03:15.227+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.766003+0000 front 2021-02-10T05:02:31.766025+0000 (oldest deadline 2021-02-10T05:02:55.865741+0000) 2021-02-10T05:03:15.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:15 smithi164 conmon[40892]: debug 2021-02-10T05:03:15.089+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:03:15.396 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:15 smithi164 conmon[40892]: debug 2021-02-10T05:03:15.089+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:03:15.396 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:15 smithi164 conmon[40892]: debug 2021-02-10T05:03:15.089+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:15.396 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:15 smithi164 conmon[40892]: debug 2021-02-10T05:03:15.089+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:15.396 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:15 smithi164 conmon[40892]: debug 2021-02-10T05:03:15.089+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:03:15.396 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:15 smithi164 conmon[44544]: debug 2021-02-10T05:03:15.125+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:03:15.396 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:15 smithi164 conmon[44544]: debug 2021-02-10T05:03:15.125+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:15.397 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:15 smithi164 conmon[44544]: debug 2021-02-10T05:03:15.125+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:15.397 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:15 smithi164 conmon[44544]: debug 2021-02-10T05:03:15.126+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:03:15.397 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:15 smithi164 conmon[44544]: debug 2021-02-10T05:03:15.126+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:03:15.397 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:15 smithi164 conmon[44544]: debug 2021-02-10T05:03:15.126+0000 7f64c490c700 -1 osd.7 1429 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24373.0:4709 5.9 5:93e5b521:::notify.7:head [watch ping cookie 94187770081792 gen 4] snapc 0=[] ondisk+write+known_if_redirected e1429) 2021-02-10T05:03:16.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:16 smithi164 conmon[37263]: debug 2021-02-10T05:03:16.200+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:03:16.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:16 smithi164 conmon[37263]: debug 2021-02-10T05:03:16.200+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:23.564321+0000 front 2021-02-10T05:02:23.564413+0000 (oldest deadline 2021-02-10T05:02:47.664447+0000) 2021-02-10T05:03:16.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:16 smithi164 conmon[37263]: debug 2021-02-10T05:03:16.200+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:27.665225+0000 front 2021-02-10T05:02:27.665010+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:16.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:16 smithi164 conmon[37263]: debug 2021-02-10T05:03:16.200+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:27.665562+0000 front 2021-02-10T05:02:27.665151+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:16.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:16 smithi164 conmon[37263]: debug 2021-02-10T05:03:16.200+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.766003+0000 front 2021-02-10T05:02:31.766025+0000 (oldest deadline 2021-02-10T05:02:55.865741+0000) 2021-02-10T05:03:16.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:16 smithi164 conmon[40892]: debug 2021-02-10T05:03:16.070+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:03:16.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:16 smithi164 conmon[40892]: debug 2021-02-10T05:03:16.070+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:03:16.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:16 smithi164 conmon[40892]: debug 2021-02-10T05:03:16.070+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:16.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:16 smithi164 conmon[40892]: debug 2021-02-10T05:03:16.070+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:16.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:16 smithi164 conmon[40892]: debug 2021-02-10T05:03:16.070+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:03:16.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:16 smithi164 conmon[44544]: debug 2021-02-10T05:03:16.141+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:03:16.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:16 smithi164 conmon[44544]: debug 2021-02-10T05:03:16.141+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:16.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:16 smithi164 conmon[44544]: 2021-02-10T05:03:16.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:16 smithi164 conmon[44544]: debug 2021-02-10T05:03:16.141+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:16.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:16 smithi164 conmon[44544]: debug 2021-02-10T05:03:16.141+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:03:16.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:16 smithi164 conmon[44544]: debug 2021-02-10T05:03:16.141+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:03:16.396 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:16 smithi164 conmon[44544]: debug 2021-02-10T05:03:16.141+0000 7f64c490c700 -1 osd.7 1429 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24373.0:4709 5.9 5:93e5b521:::notify.7:head [watch ping cookie 94187770081792 gen 4] snapc 0=[] ondisk+write+known_if_redirected e1429) 2021-02-10T05:03:17.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:17 smithi164 conmon[44544]: debug 2021-02-10T05:03:17.128+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:03:17.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:17 smithi164 conmon[44544]: debug 2021-02-10T05:03:17.128+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:17.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:17 smithi164 conmon[44544]: debug 2021-02-10T05:03:17.128+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:17.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:17 smithi164 conmon[44544]: debug 2021-02-10T05:03:17.128+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:03:17.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:17 smithi164 conmon[44544]: debug 2021-02-10T05:03:17.128+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:03:17.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:17 smithi164 conmon[44544]: debug 2021-02-10T05:03:17.128+0000 7f64c490c700 -1 osd.7 1429 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24373.0:4709 5.9 5:93e5b521:::notify.7:head [watch ping cookie 94187770081792 gen 4] snapc 0=[] ondisk+write+known_if_redirected e1429) 2021-02-10T05:03:17.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:17 smithi164 conmon[37263]: debug 2021-02-10T05:03:17.215+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:18.362462+0000 front 2021-02-10T05:02:18.363130+0000 (oldest deadline 2021-02-10T05:02:40.662679+0000) 2021-02-10T05:03:17.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:17 smithi164 conmon[37263]: debug 2021-02-10T05:03:17.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:17 smithi164 conmon[37263]: 2021-02-10T05:03:17.215+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:23.564321+0000 front 2021-02-10T05:02:23.564413+0000 (oldest deadline 2021-02-10T05:02:47.664447+0000) 2021-02-10T05:03:17.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:17 smithi164 conmon[37263]: debug 2021-02-10T05:03:17.215+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:27.665225+0000 front 2021-02-10T05:02:27.665010+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:17.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:17 smithi164 conmon[37263]: debug 2021-02-10T05:03:17.215+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:27.665562+0000 front 2021-02-10T05:02:27.665151+0000 (oldest deadline 2021-02-10T05:02:51.765091+0000) 2021-02-10T05:03:17.395 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:17 smithi164 conmon[37263]: debug 2021-02-10T05:03:17.215+0000 7f1c83a60700 -1 osd.5 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.766003+0000 front 2021-02-10T05:02:31.766025+0000 (oldest deadline 2021-02-10T05:02:55.865741+0000) 2021-02-10T05:03:17.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:17 smithi164 conmon[40892]: debug 2021-02-10T05:03:17.111+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:03:17.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:17 smithi164 conmon[40892]: debug 2021-02-10T05:03:17.111+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:03:17.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:17 smithi164 conmon[40892]: 2021-02-10T05:03:17.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:17 smithi164 conmon[40892]: debug 2021-02-10T05:03:17.111+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:17.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:17 smithi164 conmon[40892]: debug 2021-02-10T05:03:17.111+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:17.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:17 smithi164 conmon[40892]: debug 2021-02-10T05:03:17.111+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:03:17.906 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.4.service' 2021-02-10T05:03:17.970 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-02-10T05:03:17.971 INFO:tasks.cephadm.osd.4:Stopped osd.4 2021-02-10T05:03:17.971 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2021-02-10T05:03:17.971 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.5 2021-02-10T05:03:18.159 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:18 smithi164 systemd[1]: Stopping Ceph osd.5 for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T05:03:18.159 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:18 smithi164 conmon[37263]: debug 2021-02-10T05:03:18.122+0000 7f1c89ae9700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T05:03:18.159 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:18 smithi164 conmon[37263]: debug 2021-02-10T05:03:18.122+0000 7f1c89ae9700 -1 osd.5 1429 *** Got signal Terminated *** 2021-02-10T05:03:18.159 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:18 smithi164 conmon[37263]: debug 2021-02-10T05:03:18.159 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:18 smithi164 conmon[37263]: 2021-02-10T05:03:18.122+0000 7f1c89ae9700 -1 osd.5 1429 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-02-10T05:03:18.159 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:18 smithi164 conmon[40892]: debug 2021-02-10T05:03:18.134+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:03:18.159 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:18 smithi164 conmon[40892]: debug 2021-02-10T05:03:18.134+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:03:18.159 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:18 smithi164 conmon[40892]: debug 2021-02-10T05:03:18.134+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:18.159 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:18 smithi164 conmon[40892]: debug 2021-02-10T05:03:18.134+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:18.160 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:18 smithi164 conmon[40892]: debug 2021-02-10T05:03:18.134+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:03:18.160 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:18 smithi164 conmon[40892]: debug 2021-02-10T05:03:18.134+0000 7f3051d43700 -1 osd.6 1429 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24373.0:4716 5.1f 5:f95f44c2:::notify.0:head [watch ping cookie 94187755108992 gen 2] snapc 0=[] ondisk+write+known_if_redirected e1429) 2021-02-10T05:03:18.160 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:18 smithi164 conmon[44544]: debug 2021-02-10T05:03:18.136+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:03:18.160 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:18 smithi164 conmon[44544]: debug 2021-02-10T05:03:18.136+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:18.160 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:18 smithi164 conmon[44544]: debug 2021-02-10T05:03:18.136+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:18.160 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:18 smithi164 conmon[44544]: debug 2021-02-10T05:03:18.160 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:18 smithi164 conmon[44544]: 2021-02-10T05:03:18.136+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:03:18.160 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:18 smithi164 conmon[44544]: debug 2021-02-10T05:03:18.136+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:03:18.160 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:18 smithi164 conmon[44544]: debug 2021-02-10T05:03:18.136+0000 7f64c490c700 -1 osd.7 1429 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24373.0:4709 5.9 5:93e5b521:::notify.7:head [watch ping cookie 94187770081792 gen 4] snapc 0=[] ondisk+write+known_if_redirected e1429) 2021-02-10T05:03:19.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:19 smithi164 conmon[40892]: debug 2021-02-10T05:03:19.085+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:03:19.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:19 smithi164 conmon[40892]: debug 2021-02-10T05:03:19.085+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:03:19.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:19 smithi164 conmon[40892]: debug 2021-02-10T05:03:19.085+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:19.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:19 smithi164 conmon[40892]: debug 2021-02-10T05:03:19.085+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:19.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:19 smithi164 conmon[40892]: debug 2021-02-10T05:03:19.085+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:03:19.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:19 smithi164 conmon[40892]: debug 2021-02-10T05:03:19.085+0000 7f3051d43700 -1 osd.6 1429 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24373.0:4716 5.1f 5:f95f44c2:::notify.0:head [watch ping cookie 94187755108992 gen 2] snapc 0=[] ondisk+write+known_if_redirected e1429) 2021-02-10T05:03:19.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:19 smithi164 conmon[44544]: debug 2021-02-10T05:03:19.105+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:03:19.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:19 smithi164 conmon[44544]: debug 2021-02-10T05:03:19.105+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:19.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:19 smithi164 conmon[44544]: debug 2021-02-10T05:03:19.105+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:19.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:19 smithi164 conmon[44544]: debug 2021-02-10T05:03:19.105+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:03:19.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:19 smithi164 conmon[44544]: debug 2021-02-10T05:03:19.105+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:03:19.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:19 smithi164 conmon[44544]: debug 2021-02-10T05:03:19.105+0000 7f64c490c700 -1 osd.7 1429 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24373.0:4709 5.9 5:93e5b521:::notify.7:head [watch ping cookie 94187770081792 gen 4] snapc 0=[] ondisk+write+known_if_redirected e1429) 2021-02-10T05:03:20.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:20 smithi164 conmon[40892]: debug 2021-02-10T05:03:20.111+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:03:20.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:20 smithi164 conmon[40892]: debug 2021-02-10T05:03:20.111+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:03:20.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:20 smithi164 conmon[40892]: debug 2021-02-10T05:03:20.111+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:20.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:20 smithi164 conmon[40892]: debug 2021-02-10T05:03:20.111+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:20.396 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:20 smithi164 conmon[40892]: debug 2021-02-10T05:03:20.111+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:03:20.396 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:20 smithi164 conmon[40892]: debug 2021-02-10T05:03:20.111+0000 7f3051d43700 -1 osd.6 1429 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24373.0:4716 5.1f 5:f95f44c2:::notify.0:head [watch ping cookie 94187755108992 gen 2] snapc 0=[] ondisk+write+known_if_redirected e1429) 2021-02-10T05:03:20.396 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:20 smithi164 conmon[44544]: debug 2021-02-10T05:03:20.056+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:03:20.396 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:20 smithi164 conmon[44544]: debug 2021-02-10T05:03:20.056+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:20.396 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:20 smithi164 conmon[44544]: debug 2021-02-10T05:03:20.056+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:20.396 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:20 smithi164 conmon[44544]: debug 2021-02-10T05:03:20.056+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:03:20.397 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:20 smithi164 conmon[44544]: debug 2021-02-10T05:03:20.056+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:03:20.397 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:20 smithi164 conmon[44544]: debug 2021-02-10T05:03:20.056+0000 7f64c490c700 -1 osd.7 1429 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24373.0:4709 5.9 5:93e5b521:::notify.7:head [watch ping cookie 94187770081792 gen 4] snapc 0=[] ondisk+write+known_if_redirected e1429) 2021-02-10T05:03:21.394 INFO:journalctl@ceph.osd.5.smithi164.stdout:Feb 10 05:03:21 smithi164 podman[98265]: Error: container 83b8b71f692cc07a65829aa41106e6bbf42b120982d09937dee7fa17cab3ceb5 does not exist in database: no such container 2021-02-10T05:03:21.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:21 smithi164 conmon[40892]: debug 2021-02-10T05:03:21.130+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:03:21.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:21 smithi164 conmon[40892]: debug 2021-02-10T05:03:21.130+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:03:21.394 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:21 smithi164 conmon[40892]: debug 2021-02-10T05:03:21.130+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:21.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:21 smithi164 conmon[40892]: debug 2021-02-10T05:03:21.130+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:21.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:21 smithi164 conmon[40892]: debug 2021-02-10T05:03:21.130+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:03:21.395 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:21 smithi164 conmon[40892]: debug 2021-02-10T05:03:21.130+0000 7f3051d43700 -1 osd.6 1429 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24373.0:4716 5.1f 5:f95f44c2:::notify.0:head [watch ping cookie 94187755108992 gen 2] snapc 0=[] ondisk+write+known_if_redirected e1429) 2021-02-10T05:03:21.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:21 smithi164 conmon[44544]: debug 2021-02-10T05:03:21.028+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:03:21.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:21 smithi164 conmon[44544]: debug 2021-02-10T05:03:21.028+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:21.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:21 smithi164 conmon[44544]: debug 2021-02-10T05:03:21.028+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:21.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:21 smithi164 conmon[44544]: debug 2021-02-10T05:03:21.028+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:03:21.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:21 smithi164 conmon[44544]: debug 2021-02-10T05:03:21.028+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:03:21.395 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:21 smithi164 conmon[44544]: debug 2021-02-10T05:03:21.028+0000 7f64c490c700 -1 osd.7 1429 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24373.0:4709 5.9 5:93e5b521:::notify.7:head [watch ping cookie 94187770081792 gen 4] snapc 0=[] ondisk+write+known_if_redirected e1429) 2021-02-10T05:03:22.290 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:22 smithi164 conmon[40892]: debug 2021-02-10T05:03:22.161+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:03:22.291 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:22 smithi164 conmon[40892]: debug 2021-02-10T05:03:22.161+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:03:22.291 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:22 smithi164 conmon[40892]: debug 2021-02-10T05:03:22.161+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:22.291 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:22 smithi164 conmon[40892]: debug 2021-02-10T05:03:22.161+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:22.291 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:22 smithi164 conmon[40892]: debug 2021-02-10T05:03:22.161+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:03:22.291 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:22 smithi164 conmon[40892]: debug 2021-02-10T05:03:22.161+0000 7f3051d43700 -1 osd.6 1429 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24373.0:4716 5.1f 5:f95f44c2:::notify.0:head [watch ping cookie 94187755108992 gen 2] snapc 0=[] ondisk+write+known_if_redirected e1429) 2021-02-10T05:03:22.291 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:22 smithi164 conmon[44544]: debug 2021-02-10T05:03:22.034+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:03:22.291 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:22 smithi164 conmon[44544]: debug 2021-02-10T05:03:22.034+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:22.291 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:22 smithi164 conmon[44544]: debug 2021-02-10T05:03:22.034+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:22.291 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:22 smithi164 conmon[44544]: debug 2021-02-10T05:03:22.034+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:03:22.291 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:22 smithi164 conmon[44544]: debug 2021-02-10T05:03:22.034+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:03:22.291 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:22 smithi164 conmon[44544]: debug 2021-02-10T05:03:22.034+0000 7f64c490c700 -1 osd.7 1429 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24373.0:4709 5.9 5:93e5b521:::notify.7:head [watch ping cookie 94187770081792 gen 4] snapc 0=[] ondisk+write+known_if_redirected e1429) 2021-02-10T05:03:22.978 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.5.service' 2021-02-10T05:03:23.016 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-02-10T05:03:23.016 INFO:tasks.cephadm.osd.5:Stopped osd.5 2021-02-10T05:03:23.016 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2021-02-10T05:03:23.017 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.6 2021-02-10T05:03:23.355 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:23 smithi164 conmon[44544]: debug 2021-02-10T05:03:23.079+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:03:23.356 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:23 smithi164 conmon[44544]: debug 2021-02-10T05:03:23.079+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:23.356 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:23 smithi164 conmon[44544]: debug 2021-02-10T05:03:23.356 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:23 smithi164 conmon[44544]: 2021-02-10T05:03:23.079+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:23.356 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:23 smithi164 conmon[44544]: debug 2021-02-10T05:03:23.079+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:03:23.356 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:23 smithi164 conmon[44544]: debug 2021-02-10T05:03:23.079+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:03:23.356 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:23 smithi164 conmon[44544]: debug 2021-02-10T05:03:23.079+0000 7f64c490c700 -1 osd.7 1429 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24373.0:4709 5.9 5:93e5b521:::notify.7:head [watch ping cookie 94187770081792 gen 4] snapc 0=[] ondisk+write+known_if_redirected e1429) 2021-02-10T05:03:23.356 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:23 smithi164 systemd[1]: Stopping Ceph osd.6 for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T05:03:23.356 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:23 smithi164 conmon[40892]: debug 2021-02-10T05:03:23.124+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:16.856386+0000 front 2021-02-10T05:02:16.856257+0000 (oldest deadline 2021-02-10T05:02:42.756404+0000) 2021-02-10T05:03:23.356 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:23 smithi164 conmon[40892]: debug 2021-02-10T05:03:23.124+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:22.757200+0000 front 2021-02-10T05:02:22.757047+0000 (oldest deadline 2021-02-10T05:02:48.057135+0000) 2021-02-10T05:03:23.356 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:23 smithi164 conmon[40892]: debug 2021-02-10T05:03:23.124+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:28.057938+0000 front 2021-02-10T05:02:28.058278+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:23.356 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:23 smithi164 conmon[40892]: debug 2021-02-10T05:03:23.125+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:28.058148+0000 front 2021-02-10T05:02:28.058057+0000 (oldest deadline 2021-02-10T05:02:51.558072+0000) 2021-02-10T05:03:23.357 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:23 smithi164 conmon[40892]: debug 2021-02-10T05:03:23.125+0000 7f3051d43700 -1 osd.6 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:31.558950+0000 front 2021-02-10T05:02:31.558876+0000 (oldest deadline 2021-02-10T05:02:52.658838+0000) 2021-02-10T05:03:23.357 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:23 smithi164 conmon[40892]: debug 2021-02-10T05:03:23.125+0000 7f3051d43700 -1 osd.6 1429 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24373.0:4716 5.1f 5:f95f44c2:::notify.0:head [watch ping cookie 94187755108992 gen 2] snapc 0=[] ondisk+write+known_if_redirected e1429) 2021-02-10T05:03:23.357 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:23 smithi164 conmon[40892]: debug 2021-02-10T05:03:23.139+0000 7f30545c5700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T05:03:23.357 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:23 smithi164 conmon[40892]: debug 2021-02-10T05:03:23.139+0000 7f30545c5700 -1 osd.6 1429 *** Got signal Terminated *** 2021-02-10T05:03:23.357 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:23 smithi164 conmon[40892]: debug 2021-02-10T05:03:23.139+0000 7f30545c5700 -1 osd.6 1429 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-02-10T05:03:24.127 INFO:journalctl@ceph.osd.6.smithi164.stdout:Feb 10 05:03:23 smithi164 podman[98595]: 1e36602c1746b233e238a26006a89ca6fadb2049a4fc86d7a7ceac1f4214251c 2021-02-10T05:03:24.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:24 smithi164 conmon[44544]: debug 2021-02-10T05:03:24.126+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:03:24.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:24 smithi164 conmon[44544]: debug 2021-02-10T05:03:24.126+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:24.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:24 smithi164 conmon[44544]: debug 2021-02-10T05:03:24.126+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:24.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:24 smithi164 conmon[44544]: 2021-02-10T05:03:24.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:24 smithi164 conmon[44544]: debug 2021-02-10T05:03:24.126+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:03:24.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:24 smithi164 conmon[44544]: debug 2021-02-10T05:03:24.126+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:03:24.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:24 smithi164 conmon[44544]: debug 2021-02-10T05:03:24.126+0000 7f64c490c700 -1 osd.7 1429 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.24373.0:4709 5.9 5:93e5b521:::notify.7:head [watch ping cookie 94187770081792 gen 4] snapc 0=[] ondisk+write+known_if_redirected e1429) 2021-02-10T05:03:25.295 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.6.service' 2021-02-10T05:03:25.332 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-02-10T05:03:25.333 INFO:tasks.cephadm.osd.6:Stopped osd.6 2021-02-10T05:03:25.333 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2021-02-10T05:03:25.333 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.7 2021-02-10T05:03:25.457 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:25 smithi164 conmon[44544]: debug 2021-02-10T05:03:25.161+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6803 osd.0 since back 2021-02-10T05:02:17.095387+0000 front 2021-02-10T05:02:17.095552+0000 (oldest deadline 2021-02-10T05:02:42.395546+0000) 2021-02-10T05:03:25.457 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:25 smithi164 conmon[44544]: debug 2021-02-10T05:03:25.161+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6807 osd.1 since back 2021-02-10T05:02:25.197766+0000 front 2021-02-10T05:02:25.197745+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:25.457 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:25 smithi164 conmon[44544]: debug 2021-02-10T05:03:25.161+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6811 osd.2 since back 2021-02-10T05:02:25.197832+0000 front 2021-02-10T05:02:25.198626+0000 (oldest deadline 2021-02-10T05:02:49.897899+0000) 2021-02-10T05:03:25.457 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:25 smithi164 conmon[44544]: debug 2021-02-10T05:03:25.161+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.85:6815 osd.3 since back 2021-02-10T05:02:29.898683+0000 front 2021-02-10T05:02:29.898906+0000 (oldest deadline 2021-02-10T05:02:50.998610+0000) 2021-02-10T05:03:25.457 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:25 smithi164 conmon[44544]: debug 2021-02-10T05:03:25.161+0000 7f64c490c700 -1 osd.7 1429 heartbeat_check: no reply from 172.21.15.164:6802 osd.4 since back 2021-02-10T05:02:30.999488+0000 front 2021-02-10T05:02:30.999406+0000 (oldest deadline 2021-02-10T05:02:54.499378+0000) 2021-02-10T05:03:25.457 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:25 smithi164 conmon[44544]: debug 2021-02-10T05:03:25.161+0000 7f64c490c700 -1 osd.7 1429 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.24373.0:4709 5.9 5:93e5b521:::notify.7:head [watch ping cookie 94187770081792 gen 4] snapc 0=[] ondisk+write+known_if_redirected e1429) 2021-02-10T05:03:25.457 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:25 smithi164 systemd[1]: Stopping Ceph osd.7 for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T05:03:25.457 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:25 smithi164 conmon[44544]: debug 2021-02-10T05:03:25.455+0000 7f64cb997700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T05:03:25.458 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:25 smithi164 conmon[44544]: debug 2021-02-10T05:03:25.746 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:25 smithi164 conmon[44544]: 2021-02-10T05:03:25.455+0000 7f64cb997700 -1 osd.7 1429 *** Got signal Terminated *** 2021-02-10T05:03:25.746 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:25 smithi164 conmon[44544]: debug 2021-02-10T05:03:25.455+0000 7f64cb997700 -1 osd.7 1429 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-02-10T05:03:26.394 INFO:journalctl@ceph.osd.7.smithi164.stdout:Feb 10 05:03:26 smithi164 podman[98926]: 252dedc083dee9a5faec6c7c636d99e336556f8b9fc0edb2cc577542ca98fca1 2021-02-10T05:03:27.536 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@osd.7.service' 2021-02-10T05:03:27.573 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-02-10T05:03:27.574 INFO:tasks.cephadm.osd.7:Stopped osd.7 2021-02-10T05:03:27.574 INFO:tasks.cephadm.ceph.rgw.realm.zone.a:Stopping rgw.realm.zone.a... 2021-02-10T05:03:27.574 DEBUG:teuthology.orchestra.run.smithi085:> sudo systemctl stop ceph-42224fce-6b57-11eb-8feb-001a4aab830c@rgw.realm.zone.a 2021-02-10T05:03:27.941 INFO:journalctl@ceph.rgw.realm.zone.a.smithi085.stdout:Feb 10 05:03:27 smithi085 systemd[1]: Stopping Ceph rgw.realm.zone.a for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T05:03:27.942 INFO:journalctl@ceph.rgw.realm.zone.a.smithi085.stdout:Feb 10 05:03:27 smithi085 conmon[54650]: debug 2021-02-10T05:03:27.710+0000 7ffb97447700 -1 received signal: Terminated from /dev/init -- /usr/bin/radosgw -n client.rgw.realm.zone.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-02-10T05:03:27.942 INFO:journalctl@ceph.rgw.realm.zone.a.smithi085.stdout:Feb 10 05:03:27 smithi085 conmon[54650]: debug 2021-02-10T05:03:27.715+0000 7ffbac21c2c0 -1 shutting down 2021-02-10T05:03:38.276 DEBUG:teuthology.orchestra.run.smithi085:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@rgw.realm.zone.a.service' 2021-02-10T05:03:38.321 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-02-10T05:03:38.322 INFO:tasks.cephadm.ceph.rgw.realm.zone.a:Stopped rgw.realm.zone.a 2021-02-10T05:03:38.322 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2021-02-10T05:03:38.322 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-42224fce-6b57-11eb-8feb-001a4aab830c@prometheus.a 2021-02-10T05:03:38.644 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 05:03:38 smithi164 systemd[1]: Stopping Ceph prometheus.a for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T05:03:38.644 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 05:03:38 smithi164 conmon[53830]: level=warn ts=2021-02-10T05:03:38.461Z caller=main.go:524 msg="Received SIGTERM, exiting gracefully..." 2021-02-10T05:03:38.644 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 05:03:38 smithi164 conmon[53830]: level=info ts=2021-02-10T05:03:38.487Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2021-02-10T05:03:38.644 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 05:03:38 smithi164 conmon[53830]: level=info ts=2021-02-10T05:03:38.487Z caller=main.go:561 msg="Stopping notify discovery manager..." 2021-02-10T05:03:38.644 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 05:03:38 smithi164 conmon[53830]: level=info ts=2021-02-10T05:03:38.487Z caller=main.go:543 msg="Scrape discovery manager stopped" 2021-02-10T05:03:38.644 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 05:03:38 smithi164 conmon[53830]: level=info ts=2021-02-10T05:03:38.487Z caller=main.go:557 msg="Notify discovery manager stopped" 2021-02-10T05:03:38.644 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 05:03:38 smithi164 conmon[53830]: level=info ts=2021-02-10T05:03:38.487Z caller=main.go:583 msg="Stopping scrape manager..." 2021-02-10T05:03:38.644 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 05:03:38 smithi164 conmon[53830]: level=info ts=2021-02-10T05:03:38.503Z caller=main.go:577 msg="Scrape manager stopped" 2021-02-10T05:03:38.645 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 05:03:38 smithi164 conmon[53830]: level=info ts=2021-02-10T05:03:38.512Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2021-02-10T05:03:38.645 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 05:03:38 smithi164 conmon[53830]: level=info ts=2021-02-10T05:03:38.512Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2021-02-10T05:03:38.645 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 05:03:38 smithi164 conmon[53830]: level=info ts=2021-02-10T05:03:38.524Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2021-02-10T05:03:38.645 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 05:03:38 smithi164 conmon[53830]: level=info ts=2021-02-10T05:03:38.531Z caller=main.go:749 msg="Notifier manager stopped" 2021-02-10T05:03:38.645 INFO:journalctl@ceph.prometheus.a.smithi164.stdout:Feb 10 05:03:38 smithi164 conmon[53830]: level=info ts=2021-02-10T05:03:38.544Z caller=main.go:761 msg="See you next time!" 2021-02-10T05:03:39.198 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-42224fce-6b57-11eb-8feb-001a4aab830c@prometheus.a.service' 2021-02-10T05:03:39.233 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-02-10T05:03:39.233 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2021-02-10T05:03:39.233 DEBUG:teuthology.orchestra.run.smithi085:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2021-02-10T05:03:39.264 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2021-02-10T05:03:39.298 INFO:tasks.cephadm:Archiving crash dumps... 2021-02-10T05:03:39.300 DEBUG:teuthology.misc:Transferring archived files from smithi085:/var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/crash to /home/teuthworker/archive/sage-2021-02-09_22:53:38-rados:cephadm:thrash-wip-sage2-testing-2021-02-09-1332-distro-basic-smithi/5872147/remote/ubuntu@smithi085.front.sepia.ceph.com/crash 2021-02-10T05:03:39.301 DEBUG:teuthology.orchestra.run.smithi085:> sudo tar cz -f - -C /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/crash -- . 2021-02-10T05:03:39.370 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/crash to /home/teuthworker/archive/sage-2021-02-09_22:53:38-rados:cephadm:thrash-wip-sage2-testing-2021-02-09-1332-distro-basic-smithi/5872147/remote/ubuntu@smithi164.front.sepia.ceph.com/crash 2021-02-10T05:03:39.371 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar cz -f - -C /var/lib/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/crash -- . 2021-02-10T05:03:39.456 INFO:tasks.cephadm:Checking cluster log for badness... 2021-02-10T05:03:39.456 DEBUG:teuthology.orchestra.run.smithi085:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | head -n 1 2021-02-10T05:03:39.484 INFO:teuthology.orchestra.run.smithi085.stderr:grep: /var/log/ceph/42224fce-6b57-11eb-8feb-001a4aab830c/ceph.log: No such file or directory 2021-02-10T05:03:39.490 INFO:tasks.cephadm:Compressing logs... 2021-02-10T05:03:39.490 DEBUG:teuthology.orchestra.run.smithi085:> sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-10T05:03:39.533 DEBUG:teuthology.orchestra.run.smithi164:> sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-10T05:03:39.535 INFO:tasks.cephadm:Archiving logs... 2021-02-10T05:03:39.535 DEBUG:teuthology.misc:Transferring archived files from smithi085:/var/log/ceph to /home/teuthworker/archive/sage-2021-02-09_22:53:38-rados:cephadm:thrash-wip-sage2-testing-2021-02-09-1332-distro-basic-smithi/5872147/remote/ubuntu@smithi085.front.sepia.ceph.com/log 2021-02-10T05:03:39.536 DEBUG:teuthology.orchestra.run.smithi085:> sudo tar cz -f - -C /var/log/ceph -- . 2021-02-10T05:03:40.622 INFO:teuthology.orchestra.run.smithi085.stderr:tar: ./42224fce-6b57-11eb-8feb-001a4aab830c/ceph-mon.c.log: file changed as we read it 2021-02-10T05:03:43.345 INFO:teuthology.orchestra.run.smithi085.stderr:tar: ./42224fce-6b57-11eb-8feb-001a4aab830c/ceph-mon.a.log: file changed as we read it 2021-02-10T05:03:43.526 INFO:teuthology.orchestra.run.smithi085.stderr:tar: ./42224fce-6b57-11eb-8feb-001a4aab830c/ceph-client.rgw.realm.zone.a.log: File removed before we read it 2021-02-10T05:20:01.898 INFO:teuthology.orchestra.run.smithi085.stderr:tar: ./42224fce-6b57-11eb-8feb-001a4aab830c/ceph-osd.2.log: file changed as we read it 2021-02-10T05:30:25.714 INFO:teuthology.orchestra.run.smithi085.stderr:tar: ./42224fce-6b57-11eb-8feb-001a4aab830c: file changed as we read it 2021-02-10T05:30:26.067 INFO:teuthology.orchestra.run.smithi085.stderr:tar: .: file changed as we read it 2021-02-10T05:30:26.306 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/log/ceph to /home/teuthworker/archive/sage-2021-02-09_22:53:38-rados:cephadm:thrash-wip-sage2-testing-2021-02-09-1332-distro-basic-smithi/5872147/remote/ubuntu@smithi164.front.sepia.ceph.com/log 2021-02-10T05:30:26.322 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar cz -f - -C /var/log/ceph -- . 2021-02-10T06:55:47.497 INFO:teuthology.orchestra.run.smithi164.stderr:tar: ./42224fce-6b57-11eb-8feb-001a4aab830c/ceph-osd.6.log: file changed as we read it 2021-02-10T06:55:50.701 INFO:teuthology.orchestra.run.smithi164.stderr:tar: ./42224fce-6b57-11eb-8feb-001a4aab830c/ceph-mgr.x.log: File removed before we read it 2021-02-10T06:55:50.701 INFO:teuthology.orchestra.run.smithi164.stderr:tar: ./42224fce-6b57-11eb-8feb-001a4aab830c/ceph-osd.4.log: File removed before we read it 2021-02-10T06:55:50.701 INFO:teuthology.orchestra.run.smithi164.stderr:tar: ./42224fce-6b57-11eb-8feb-001a4aab830c/ceph-mon.b.log: File removed before we read it 2021-02-10T06:58:35.603 INFO:teuthology.orchestra.run.smithi164.stderr:tar: ./42224fce-6b57-11eb-8feb-001a4aab830c/ceph-osd.7.log: File removed before we read it 2021-02-10T07:01:29.400 INFO:teuthology.orchestra.run.smithi164.stderr:tar: ./42224fce-6b57-11eb-8feb-001a4aab830c/ceph-volume.log: File removed before we read it 2021-02-10T07:01:29.420 INFO:teuthology.orchestra.run.smithi164.stderr:tar: ./42224fce-6b57-11eb-8feb-001a4aab830c: file changed as we read it 2021-02-10T07:01:29.420 INFO:teuthology.orchestra.run.smithi164.stderr:tar: ./cephadm.log: File removed before we read it 2021-02-10T07:01:29.420 INFO:teuthology.orchestra.run.smithi164.stderr:tar: .: file changed as we read it 2021-02-10T07:01:29.442 INFO:tasks.cephadm:Removing cluster... 2021-02-10T07:01:29.442 DEBUG:teuthology.orchestra.run.smithi085:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 42224fce-6b57-11eb-8feb-001a4aab830c --force 2021-02-10T07:01:32.442 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 07:01:32 smithi085 systemd[1]: Stopping Ceph node-exporter.a for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T07:01:33.663 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 07:01:33 smithi085 podman[230609]: d7eb272b6262b1db7a15d101408d5d9f9125aa6495d73e8a65c61394e088d69d 2021-02-10T07:01:33.663 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 07:01:33 smithi085 systemd[1]: ceph-42224fce-6b57-11eb-8feb-001a4aab830c@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2021-02-10T07:01:33.663 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 07:01:33 smithi085 systemd[1]: ceph-42224fce-6b57-11eb-8feb-001a4aab830c@node-exporter.a.service: Failed with result 'exit-code'. 2021-02-10T07:01:33.664 INFO:journalctl@ceph.node-exporter.a.smithi085.stdout:Feb 10 07:01:33 smithi085 systemd[1]: Stopped Ceph node-exporter.a for 42224fce-6b57-11eb-8feb-001a4aab830c. 2021-02-10T07:01:33.942 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 07:01:33 smithi085 systemd[1]: Stopping Ceph alertmanager.a for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T07:01:33.942 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 07:01:33 smithi085 conmon[61668]: level=info ts=2021-02-10T07:01:33.754Z caller=main.go:536 msg="Received SIGTERM, exiting gracefully..." 2021-02-10T07:01:34.579 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 07:01:34 smithi085 podman[230741]: 6dd51e001cb902b709c2adf77ddc44b9a4beecc3f337400c5020afc4c2585882 2021-02-10T07:01:34.579 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 07:01:34 smithi085 systemd[1]: ceph-42224fce-6b57-11eb-8feb-001a4aab830c@alertmanager.a.service: Succeeded. 2021-02-10T07:01:34.579 INFO:journalctl@ceph.alertmanager.a.smithi085.stdout:Feb 10 07:01:34 smithi085 systemd[1]: Stopped Ceph alertmanager.a for 42224fce-6b57-11eb-8feb-001a4aab830c. 2021-02-10T07:01:34.914 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 42224fce-6b57-11eb-8feb-001a4aab830c --force 2021-02-10T07:01:37.145 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 07:01:37 smithi164 systemd[1]: Stopping Ceph iscsi.iscsi.a for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T07:01:37.644 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 07:01:37 smithi164 conmon[48560]: debug Shutdown received 2021-02-10T07:01:48.144 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 07:01:47 smithi164 podman[100698]: 2ce80b592708bdba5a7095821258bc41f772ede7b045dbb91391cce6763cd122 2021-02-10T07:01:48.145 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 07:01:48 smithi164 systemd[1]: ceph-42224fce-6b57-11eb-8feb-001a4aab830c@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2021-02-10T07:01:48.531 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 07:01:48 smithi164 bash[48232]: (process:7): GLib-GObject-CRITICAL **: 07:01:48.172: g_object_unref: assertion 'G_IS_OBJECT (object)' failed 2021-02-10T07:01:49.225 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 07:01:48 smithi164 bash[100793]: 0aa4c6de9d72d64d2c41b2369dd2289d044a970a6841bc6303349376b990b39a 2021-02-10T07:01:49.225 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 07:01:49 smithi164 systemd[1]: ceph-42224fce-6b57-11eb-8feb-001a4aab830c@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2021-02-10T07:01:49.225 INFO:journalctl@ceph.iscsi.iscsi.a.smithi164.stdout:Feb 10 07:01:49 smithi164 systemd[1]: Stopped Ceph iscsi.iscsi.a for 42224fce-6b57-11eb-8feb-001a4aab830c. 2021-02-10T07:01:49.521 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 07:01:49 smithi164 systemd[1]: Stopping Ceph node-exporter.b for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T07:01:50.078 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 07:01:49 smithi164 podman[100951]: e321ea0c60a542754dbd9e566d0f1b0ad9403f535811f3f308f4d33ed2472bc2 2021-02-10T07:01:50.079 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 07:01:49 smithi164 systemd[1]: ceph-42224fce-6b57-11eb-8feb-001a4aab830c@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2021-02-10T07:01:50.079 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 07:01:49 smithi164 systemd[1]: ceph-42224fce-6b57-11eb-8feb-001a4aab830c@node-exporter.b.service: Failed with result 'exit-code'. 2021-02-10T07:01:50.079 INFO:journalctl@ceph.node-exporter.b.smithi164.stdout:Feb 10 07:01:49 smithi164 systemd[1]: Stopped Ceph node-exporter.b for 42224fce-6b57-11eb-8feb-001a4aab830c. 2021-02-10T07:01:50.334 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 07:01:50 smithi164 systemd[1]: Stopping Ceph grafana.a for 42224fce-6b57-11eb-8feb-001a4aab830c... 2021-02-10T07:01:50.334 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 07:01:50 smithi164 conmon[52406]: t=2021-02-10T07:01:50+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2021-02-10T07:01:50.334 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 07:01:50 smithi164 conmon[52406]: t=2021-02-10T07:01:50+0000 lvl=info msg="Stopped PluginManager" logger=server reason="context canceled" 2021-02-10T07:01:50.334 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 07:01:50 smithi164 conmon[52406]: t=2021-02-10T07:01:50+0000 lvl=info msg="Stopped provisioningServiceImpl" logger=server reason="context canceled" 2021-02-10T07:01:50.334 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 07:01:50 smithi164 conmon[52406]: t=2021-02-10T07:01:50+0000 lvl=info msg="Stopped BackendPluginManager" logger=server reason="context canceled" 2021-02-10T07:01:50.334 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 07:01:50 smithi164 conmon[52406]: t=2021-02-10T07:01:50+0000 lvl=info msg="Stopped AlertEngine" logger=server reason="context canceled" 2021-02-10T07:01:50.334 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 07:01:50 smithi164 conmon[52406]: t=2021-02-10T07:01:50+0000 lvl=info msg="Stopped CleanUpService" logger=server reason="context canceled" 2021-02-10T07:01:50.334 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 07:01:50 smithi164 conmon[52406]: t=2021-02-10T07:01:50+0000 lvl=info msg="Stopped RemoteCache" logger=server reason="context canceled" 2021-02-10T07:01:50.334 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 07:01:50 smithi164 conmon[52406]: t=2021-02-10T07:01:50+0000 lvl=info msg="Stopped UserAuthTokenService" logger=server reason="context canceled" 2021-02-10T07:01:50.335 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 07:01:50 smithi164 conmon[52406]: t=2021-02-10T07:01:50+0000 lvl=info msg="Stopped UsageStatsService" logger=server reason="context canceled" 2021-02-10T07:01:50.335 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 07:01:50 smithi164 conmon[52406]: t=2021-02-10T07:01:50+0000 lvl=info msg="Stopped InternalMetricsService" logger=server reason="context canceled" 2021-02-10T07:01:50.335 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 07:01:50 smithi164 conmon[52406]: t=2021-02-10T07:01:50+0000 lvl=info msg="Stopped NotificationService" logger=server reason="context canceled" 2021-02-10T07:01:50.335 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 07:01:50 smithi164 conmon[52406]: t=2021-02-10T07:01:50+0000 lvl=info msg="Stopped Stream Manager" 2021-02-10T07:01:51.135 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 07:01:50 smithi164 podman[101076]: 0510ec24d30d8a6896ae8cc1991b7789a701c5ed9185e0334c264a4618fe3010 2021-02-10T07:01:51.136 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 07:01:51 smithi164 systemd[1]: ceph-42224fce-6b57-11eb-8feb-001a4aab830c@grafana.a.service: Succeeded. 2021-02-10T07:01:51.136 INFO:journalctl@ceph.grafana.a.smithi164.stdout:Feb 10 07:01:51 smithi164 systemd[1]: Stopped Ceph grafana.a for 42224fce-6b57-11eb-8feb-001a4aab830c. 2021-02-10T07:01:51.965 INFO:tasks.cephadm:Removing cephadm ... 2021-02-10T07:01:51.966 DEBUG:teuthology.orchestra.run.smithi085:> rm -rf /home/ubuntu/cephtest/cephadm 2021-02-10T07:01:51.985 DEBUG:teuthology.orchestra.run.smithi164:> rm -rf /home/ubuntu/cephtest/cephadm 2021-02-10T07:01:52.001 INFO:tasks.cephadm:Teardown complete 2021-02-10T07:01:52.001 DEBUG:teuthology.run_tasks:Unwinding manager install 2021-02-10T07:01:52.010 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/task/install/__init__.py", line 619, in task yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run_tasks.py", line 176, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_c1ea624123d412aff8b9d1430e36cb45fcab76b8/qa/tasks/thrashosds.py", line 214, in task thrash_proc.do_join() File "/home/teuthworker/src/git.ceph.com_ceph-c_c1ea624123d412aff8b9d1430e36cb45fcab76b8/qa/tasks/ceph_manager.py", line 729, in do_join self.sighup_thread.get() File "src/gevent/greenlet.py", line 683, in gevent._greenlet.Greenlet.get File "src/gevent/greenlet.py", line 317, in gevent._greenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 47, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_ceph-c_c1ea624123d412aff8b9d1430e36cb45fcab76b8/qa/tasks/ceph_manager.py", line 116, in wrapper return func(self) File "/home/teuthworker/src/git.ceph.com_ceph-c_c1ea624123d412aff8b9d1430e36cb45fcab76b8/qa/tasks/ceph_manager.py", line 1089, in do_sighup self.ceph_manager.signal_osd(osd, signal.SIGHUP, silent=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_c1ea624123d412aff8b9d1430e36cb45fcab76b8/qa/tasks/ceph_manager.py", line 2804, in signal_osd self.cluster).signal(sig, silent=silent) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/daemon/cephadmunit.py", line 123, in signal self.remote.sh(self.kill_cmd(sig)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/remote.py", line 259, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi085 with status 125: 'sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3' 2021-02-10T07:01:52.026 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2021-02-10T07:01:52.026 DEBUG:teuthology.orchestra.run.smithi085:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-02-10T07:01:52.029 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-02-10T07:01:52.118 DEBUG:teuthology.run_tasks:Unwinding manager clock 2021-02-10T07:01:52.299 INFO:teuthology.task.clock:Checking final clock skew... 2021-02-10T07:01:52.299 DEBUG:teuthology.orchestra.run.smithi085:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-10T07:01:52.326 INFO:teuthology.orchestra.run.smithi085.stderr:bash: ntpq: command not found 2021-02-10T07:01:52.349 INFO:teuthology.orchestra.run.smithi085.stdout:210 Number of sources = 4 2021-02-10T07:01:52.349 INFO:teuthology.orchestra.run.smithi085.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-02-10T07:01:52.349 INFO:teuthology.orchestra.run.smithi085.stdout:=============================================================================== 2021-02-10T07:01:52.349 INFO:teuthology.orchestra.run.smithi085.stdout:^- hv01.front.sepia.ceph.com 3 6 377 27 -1496us[-1496us] +/- 126ms 2021-02-10T07:01:52.349 INFO:teuthology.orchestra.run.smithi085.stdout:^* hv02.front.sepia.ceph.com 2 8 377 161 -43us[ -29us] +/- 39ms 2021-02-10T07:01:52.349 INFO:teuthology.orchestra.run.smithi085.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 +8121ns[+8121ns] +/- 55ms 2021-02-10T07:01:52.349 INFO:teuthology.orchestra.run.smithi085.stdout:^+ hv04.front.sepia.ceph.com 3 6 377 24 +90us[ +90us] +/- 49ms 2021-02-10T07:01:52.349 DEBUG:teuthology.orchestra.run.smithi164:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-10T07:01:52.363 INFO:teuthology.orchestra.run.smithi164.stderr:bash: ntpq: command not found 2021-02-10T07:01:52.445 INFO:teuthology.orchestra.run.smithi164.stdout:210 Number of sources = 4 2021-02-10T07:01:52.445 INFO:teuthology.orchestra.run.smithi164.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-02-10T07:01:52.445 INFO:teuthology.orchestra.run.smithi164.stdout:=============================================================================== 2021-02-10T07:01:52.445 INFO:teuthology.orchestra.run.smithi164.stdout:^- hv01.front.sepia.ceph.com 3 6 377 32 -1496us[-1496us] +/- 126ms 2021-02-10T07:01:52.446 INFO:teuthology.orchestra.run.smithi164.stdout:^* hv02.front.sepia.ceph.com 2 6 377 32 -69us[ -61us] +/- 41ms 2021-02-10T07:01:52.446 INFO:teuthology.orchestra.run.smithi164.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 +7185ns[+7185ns] +/- 55ms 2021-02-10T07:01:52.446 INFO:teuthology.orchestra.run.smithi164.stdout:^+ hv04.front.sepia.ceph.com 3 6 377 32 +84us[ +84us] +/- 49ms 2021-02-10T07:01:52.446 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2021-02-10T07:01:52.468 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2021-02-10T07:01:52.468 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2021-02-10T07:01:52.486 DEBUG:teuthology.orchestra.run.smithi085:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2021-02-10T07:01:52.673 DEBUG:teuthology.orchestra.run.smithi164:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2021-02-10T07:01:52.793 DEBUG:teuthology.orchestra.run.smithi085:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-02-10T07:01:52.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T07:01:52.825 DEBUG:teuthology.orchestra.run.smithi164:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-02-10T07:01:52.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T07:01:52.853 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2021-02-10T07:01:52.863 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2021-02-10T07:01:53.595 INFO:teuthology.task.internal:Duration was 10264.879252 seconds 2021-02-10T07:01:53.596 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2021-02-10T07:01:53.743 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2021-02-10T07:01:53.743 DEBUG:teuthology.orchestra.run.smithi085:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2021-02-10T07:01:53.746 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2021-02-10T07:01:53.825 INFO:teuthology.orchestra.run.smithi164.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-02-10T07:01:53.841 INFO:teuthology.orchestra.run.smithi085.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-02-10T07:01:54.250 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2021-02-10T07:01:54.250 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi085.front.sepia.ceph.com 2021-02-10T07:01:54.250 DEBUG:teuthology.orchestra.run.smithi085:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-02-10T07:01:54.448 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi164.front.sepia.ceph.com 2021-02-10T07:01:54.448 DEBUG:teuthology.orchestra.run.smithi164:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-02-10T07:01:54.567 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2021-02-10T07:01:54.568 DEBUG:teuthology.orchestra.run.smithi085:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-10T07:01:54.570 DEBUG:teuthology.orchestra.run.smithi164:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-10T07:01:54.737 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2021-02-10T07:01:54.751 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2021-02-10T07:01:54.751 DEBUG:teuthology.orchestra.run.smithi085:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-02-10T07:01:54.791 DEBUG:teuthology.orchestra.run.smithi164:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-02-10T07:01:54.844 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2021-02-10T07:01:54.861 DEBUG:teuthology.orchestra.run.smithi085:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2021-02-10T07:01:54.864 DEBUG:teuthology.orchestra.run.smithi164:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2021-02-10T07:01:54.896 INFO:teuthology.orchestra.run.smithi085.stdout:kernel.core_pattern = core 2021-02-10T07:01:54.901 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = core 2021-02-10T07:01:54.907 DEBUG:teuthology.orchestra.run.smithi085:> test -e /home/ubuntu/cephtest/archive/coredump 2021-02-10T07:01:54.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T07:01:54.960 DEBUG:teuthology.orchestra.run.smithi164:> test -e /home/ubuntu/cephtest/archive/coredump 2021-02-10T07:01:54.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T07:01:54.974 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2021-02-10T07:01:54.987 INFO:teuthology.task.internal:Transferring archived files... 2021-02-10T07:01:54.988 DEBUG:teuthology.misc:Transferring archived files from smithi085:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sage-2021-02-09_22:53:38-rados:cephadm:thrash-wip-sage2-testing-2021-02-09-1332-distro-basic-smithi/5872147/remote/smithi085 2021-02-10T07:01:54.989 DEBUG:teuthology.orchestra.run.smithi085:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2021-02-10T07:01:55.107 DEBUG:teuthology.misc:Transferring archived files from smithi164:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sage-2021-02-09_22:53:38-rados:cephadm:thrash-wip-sage2-testing-2021-02-09-1332-distro-basic-smithi/5872147/remote/smithi164 2021-02-10T07:01:55.109 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2021-02-10T07:01:55.184 INFO:teuthology.task.internal:Removing archive directory... 2021-02-10T07:01:55.184 DEBUG:teuthology.orchestra.run.smithi085:> rm -rf -- /home/ubuntu/cephtest/archive 2021-02-10T07:01:55.186 DEBUG:teuthology.orchestra.run.smithi164:> rm -rf -- /home/ubuntu/cephtest/archive 2021-02-10T07:01:55.234 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2021-02-10T07:01:55.248 INFO:teuthology.task.internal:Not uploading archives. 2021-02-10T07:01:55.249 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2021-02-10T07:01:55.264 INFO:teuthology.task.internal:Tidying up after the test... 2021-02-10T07:01:55.265 DEBUG:teuthology.orchestra.run.smithi085:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-02-10T07:01:55.268 DEBUG:teuthology.orchestra.run.smithi164:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-02-10T07:01:55.279 INFO:teuthology.orchestra.run.smithi085.stdout: 262163 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 07:01 /home/ubuntu/cephtest 2021-02-10T07:01:55.288 INFO:teuthology.orchestra.run.smithi164.stdout: 262163 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 07:01 /home/ubuntu/cephtest 2021-02-10T07:01:55.290 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2021-02-10T07:01:55.303 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi085 with status 125: 'sudo docker kill -s 1 ceph-42224fce-6b57-11eb-8feb-001a4aab830c-osd.3' 2021-02-10T07:01:55.303 INFO:teuthology.nuke:Checking targets against current locks 2021-02-10T07:01:55.377 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run.py", line 409, in main run_tasks(tasks=config['tasks'], ctx=fake_ctx) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run_tasks.py", line 204, in run_tasks raise SystemExit(1) SystemExit: 1 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/scripts/run.py", line 38, in main teuthology.run.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run.py", line 412, in main report_outcome(config, archive, fake_ctx.summary, fake_ctx) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run.py", line 269, in report_outcome nuke(fake_ctx, fake_ctx.lock) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/nuke/__init__.py", line 266, in nuke for unnuked in p: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/nuke/__init__.py", line 280, in nuke_one keep_logs = ctx.keep_logs File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/config.py", line 255, in __getattr__ raise AttributeError(name) AttributeError: keep_logs